2021-02-10T11:02:13.174 INFO:root:teuthology version: 1.1.0-14fad317 2021-02-10T11:02:13.175 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/khiremat-2021-02-10_10:13:45-fs-wip-khiremat-mgr-authid-bp-7-nautilus-distro-basic-gibba/5872720 --description fs/basic_functional/{begin clusters/1-mds-4-client-coloc conf/{client mds mon osd} mount/fuse objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{rhel_7} tasks/volumes} --name khiremat-2021-02-10_10:13:45-fs-wip-khiremat-mgr-authid-bp-7-nautilus-distro-basic-gibba --verbose --owner scheduled_khiremat@teuthology -- /home/teuthworker/archive/khiremat-2021-02-10_10:13:45-fs-wip-khiremat-mgr-authid-bp-7-nautilus-distro-basic-gibba/5872720/orig.config.yaml 2021-02-10T11:02:13.222 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2021-02-10T11:02:13.289 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/khiremat-2021-02-10_10:13:45-fs-wip-khiremat-mgr-authid-bp-7-nautilus-distro-basic-gibba/5872720 branch: wip-khiremat-mgr-authid-bp-7-nautilus description: fs/basic_functional/{begin clusters/1-mds-4-client-coloc conf/{client mds mon osd} mount/fuse objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{rhel_7} tasks/volumes} email: khiremat@redhat.com first_in_suite: false job_id: '5872720' kernel: kdb: true sha1: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: gibba name: khiremat-2021-02-10_10:13:45-fs-wip-khiremat-mgr-authid-bp-7-nautilus-distro-basic-gibba nuke-on-error: true openstack: - volumes: count: 4 size: 30 - machine: disk: 200 os_type: rhel os_version: '7.9' overrides: admin_socket: branch: wip-khiremat-mgr-authid-bp-7-nautilus ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 pid file: '' mds: debug mds: 20 debug ms: 1 mds bal frag: true mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 mgr: debug client: 10 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug filestore: 20 debug journal: 20 debug ms: 20 debug osd: 25 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(TOO_FEW_PGS\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - OSD full dropping all updates - OSD near full - pausewr flag - failsafe engaged, dropping updates - failsafe disengaged, no longer dropping - is full \(reached quota - POOL_FULL - POOL_BACKFILLFULL sha1: 962fac13b1a781b8c7d25c1eed50d93813b8f5f0 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: sha1: 962fac13b1a781b8c7d25c1eed50d93813b8f5f0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-khiremat-mgr-authid-bp-7-nautilus sha1: 962fac13b1a781b8c7d25c1eed50d93813b8f5f0 owner: scheduled_khiremat@teuthology priority: 50 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.1 - - mon.b - mon.c - mgr.x - mds.b - osd.4 - osd.5 - osd.6 - osd.7 - client.2 - client.3 sha1: 962fac13b1a781b8c7d25c1eed50d93813b8f5f0 sleep_before_teardown: 0 suite: fs suite_branch: wip-khiremat-mgr-authid-bp-7-nautilus suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_962fac13b1a781b8c7d25c1eed50d93813b8f5f0/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 962fac13b1a781b8c7d25c1eed50d93813b8f5f0 targets: gibba012.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBVcGlYALk4OspEafH8/Lx0pdzIft0iVC17OZulIeKaN1bVmyG4U/sfmwbmXgZGHPPrvbS1THXiFaWFuAH9p6q00b/d1A2Nolo5+s5U2itFFtDsrGLbMUF1u2zE3myTCQ1O1OT4a9Gw5b2xR9snaYROX22CW2NRwsLvFKZ8jisxqs8skGjHLu+Py7ce0O8h9d+iRfkTw+lqAt1CepBHLyTo2Sx3Q2R2VRTpDXZUKZWh0yVe7FMByIjG6YsmcVLMfN5jliHaQOfkuKiTGXuFfkdEJggUlpPMnuGIV89d4UfcIWOXGbAu/Gf6KPSISSQOvNNZehNEGXYuJ+x/DT/7Vgz gibba020.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCzcUFKpDP/1cUZh0gR/RKn7JfrfMtmdGkiIGToKQ0PjQjhcx4NpKQnjSid8quwUQ2p1kQ2U0fblQeO6HaeyVZMCFSqjtum1hwl15M1xmlCmYmVzM1ll9hfCGPeQf4tAcXvyGkLN+Q/nksjyYXUuFdyy284r+DzSiEDWvJRhANuea7IJtw7G9lhrL+7Qzli4lY5sQZf8zg/XTqnV/omc2n42gFr6q+RfviLlGGIrbDt05YkuA3Ee4L8b4JJh7ylG/Npl+jZBoDQJCNdPBEXan0doeWQB6dJg2hSANf/inH4nzvYNTNUQCYNaUk8kor492ImJLpZzK6cbso2I+KqH+UD tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell rpm: - python3-cephfs extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - ceph: null - ceph-fuse: null - cephfs_test_runner: fail_on_skip: false modules: - tasks.cephfs.test_volumes teuthology_branch: master timestamp: 2021-02-10_10:13:45 tube: gibba user: khiremat verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.gibba.29737 2021-02-10T11:02:13.290 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_962fac13b1a781b8c7d25c1eed50d93813b8f5f0/qa; will attempt to use it 2021-02-10T11:02:13.300 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_962fac13b1a781b8c7d25c1eed50d93813b8f5f0/qa/tasks 2021-02-10T11:02:13.300 INFO:teuthology.run_tasks:Running task internal.check_packages... 2021-02-10T11:02:13.306 INFO:teuthology.task.internal:Checking packages... 2021-02-10T11:02:13.363 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'basic' and ceph hash '962fac13b1a781b8c7d25c1eed50d93813b8f5f0' 2021-02-10T11:02:13.364 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2021-02-10T11:02:13.364 INFO:teuthology.packaging:ref: None 2021-02-10T11:02:13.364 INFO:teuthology.packaging:tag: None 2021-02-10T11:02:13.364 INFO:teuthology.packaging:branch: wip-khiremat-mgr-authid-bp-7-nautilus 2021-02-10T11:02:13.364 INFO:teuthology.packaging:sha1: 962fac13b1a781b8c7d25c1eed50d93813b8f5f0 2021-02-10T11:02:13.364 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F7%2Fx86_64&ref=wip-khiremat-mgr-authid-bp-7-nautilus 2021-02-10T11:02:13.539 INFO:teuthology.task.internal:Found packages for ceph version 14.2.16-280.g962fac13 2021-02-10T11:02:13.540 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2021-02-10T11:02:13.547 INFO:teuthology.task.internal:no buildpackages task found 2021-02-10T11:02:13.547 INFO:teuthology.run_tasks:Running task internal.save_config... 2021-02-10T11:02:13.600 INFO:teuthology.task.internal:Saving configuration 2021-02-10T11:02:13.615 INFO:teuthology.run_tasks:Running task internal.check_lock... 2021-02-10T11:02:13.715 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-02-10T11:02:13.736 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2021-02-10 10:57:15.485454', 'description': '/home/teuthworker/archive/khiremat-2021-02-10_10:13:45-fs-wip-khiremat-mgr-authid-bp-7-nautilus-distro-basic-gibba/5872720', 'locked_by': 'scheduled_khiremat@teuthology', 'up': True, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDSmbRUa+hQoWPp2ZpyDTJWAiGLklXga7D0Efjzk2y1xwJFvFUbLNB4sJhvBuWme94soiOPeKHhVecsqGvGern0=', 'os_version': '7.9', 'machine_type': 'gibba', 'vm_host': None, 'os_type': 'rhel', 'arch': 'x86_64', 'mac_address': None, 'name': 'gibba012.front.sepia.ceph.com'} 2021-02-10T11:02:13.755 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2021-02-10 10:57:15.485433', 'description': '/home/teuthworker/archive/khiremat-2021-02-10_10:13:45-fs-wip-khiremat-mgr-authid-bp-7-nautilus-distro-basic-gibba/5872720', 'locked_by': 'scheduled_khiremat@teuthology', 'up': True, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOotOd1mpOa7xwm83tEcRIHZ7+c3gbKcBC/I/oaUwLv8OUS0LlV3JUI2K1PME33lmPdiQ4f8lM7Cr3eTapOXjoI=', 'os_version': '7.9', 'machine_type': 'gibba', 'vm_host': None, 'os_type': 'rhel', 'arch': 'x86_64', 'mac_address': None, 'name': 'gibba020.front.sepia.ceph.com'} 2021-02-10T11:02:13.755 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2021-02-10T11:02:13.762 INFO:teuthology.task.internal:roles: ubuntu@gibba012.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.1'] 2021-02-10T11:02:13.762 INFO:teuthology.task.internal:roles: ubuntu@gibba020.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.2', 'client.3'] 2021-02-10T11:02:13.762 INFO:teuthology.run_tasks:Running task console_log... 2021-02-10T11:02:13.826 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f90e5f71620>, signals=[15]) 2021-02-10T11:02:13.827 INFO:teuthology.run_tasks:Running task internal.connect... 2021-02-10T11:02:13.833 INFO:teuthology.task.internal:Opening connections... 2021-02-10T11:02:13.833 DEBUG:teuthology.task.internal:connecting to ubuntu@gibba012.front.sepia.ceph.com 2021-02-10T11:02:13.835 DEBUG:teuthology.orchestra.connection:{'hostname': 'gibba012.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-02-10T11:02:13.905 DEBUG:teuthology.task.internal:connecting to ubuntu@gibba020.front.sepia.ceph.com 2021-02-10T11:02:13.906 DEBUG:teuthology.orchestra.connection:{'hostname': 'gibba020.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-02-10T11:02:13.972 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2021-02-10T11:02:13.979 DEBUG:teuthology.orchestra.run.gibba012:> uname -m 2021-02-10T11:02:14.355 INFO:teuthology.orchestra.run.gibba012.stdout:x86_64 2021-02-10T11:02:14.356 DEBUG:teuthology.orchestra.run.gibba012:> cat /etc/os-release 2021-02-10T11:02:14.690 INFO:teuthology.orchestra.run.gibba012.stdout:NAME="Red Hat Enterprise Linux Server" 2021-02-10T11:02:14.690 INFO:teuthology.orchestra.run.gibba012.stdout:VERSION="7.9 (Maipo)" 2021-02-10T11:02:14.690 INFO:teuthology.orchestra.run.gibba012.stdout:ID="rhel" 2021-02-10T11:02:14.690 INFO:teuthology.orchestra.run.gibba012.stdout:ID_LIKE="fedora" 2021-02-10T11:02:14.690 INFO:teuthology.orchestra.run.gibba012.stdout:VARIANT="Server" 2021-02-10T11:02:14.691 INFO:teuthology.orchestra.run.gibba012.stdout:VARIANT_ID="server" 2021-02-10T11:02:14.691 INFO:teuthology.orchestra.run.gibba012.stdout:VERSION_ID="7.9" 2021-02-10T11:02:14.691 INFO:teuthology.orchestra.run.gibba012.stdout:PRETTY_NAME="Employee SKU" 2021-02-10T11:02:14.691 INFO:teuthology.orchestra.run.gibba012.stdout:ANSI_COLOR="0;31" 2021-02-10T11:02:14.691 INFO:teuthology.orchestra.run.gibba012.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:7.9:GA:server" 2021-02-10T11:02:14.691 INFO:teuthology.orchestra.run.gibba012.stdout:HOME_URL="https://www.redhat.com/" 2021-02-10T11:02:14.691 INFO:teuthology.orchestra.run.gibba012.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2021-02-10T11:02:14.691 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:02:14.691 INFO:teuthology.orchestra.run.gibba012.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 7" 2021-02-10T11:02:14.691 INFO:teuthology.orchestra.run.gibba012.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=7.9 2021-02-10T11:02:14.691 INFO:teuthology.orchestra.run.gibba012.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2021-02-10T11:02:14.691 INFO:teuthology.orchestra.run.gibba012.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="7.9" 2021-02-10T11:02:14.692 INFO:teuthology.lock.ops:Updating gibba012.front.sepia.ceph.com on lock server 2021-02-10T11:02:14.710 DEBUG:teuthology.orchestra.run.gibba020:> uname -m 2021-02-10T11:02:15.060 INFO:teuthology.orchestra.run.gibba020.stdout:x86_64 2021-02-10T11:02:15.060 DEBUG:teuthology.orchestra.run.gibba020:> cat /etc/os-release 2021-02-10T11:02:15.395 INFO:teuthology.orchestra.run.gibba020.stdout:NAME="Red Hat Enterprise Linux Server" 2021-02-10T11:02:15.395 INFO:teuthology.orchestra.run.gibba020.stdout:VERSION="7.9 (Maipo)" 2021-02-10T11:02:15.395 INFO:teuthology.orchestra.run.gibba020.stdout:ID="rhel" 2021-02-10T11:02:15.395 INFO:teuthology.orchestra.run.gibba020.stdout:ID_LIKE="fedora" 2021-02-10T11:02:15.395 INFO:teuthology.orchestra.run.gibba020.stdout:VARIANT="Server" 2021-02-10T11:02:15.395 INFO:teuthology.orchestra.run.gibba020.stdout:VARIANT_ID="server" 2021-02-10T11:02:15.395 INFO:teuthology.orchestra.run.gibba020.stdout:VERSION_ID="7.9" 2021-02-10T11:02:15.395 INFO:teuthology.orchestra.run.gibba020.stdout:PRETTY_NAME="Employee SKU" 2021-02-10T11:02:15.395 INFO:teuthology.orchestra.run.gibba020.stdout:ANSI_COLOR="0;31" 2021-02-10T11:02:15.396 INFO:teuthology.orchestra.run.gibba020.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:7.9:GA:server" 2021-02-10T11:02:15.396 INFO:teuthology.orchestra.run.gibba020.stdout:HOME_URL="https://www.redhat.com/" 2021-02-10T11:02:15.396 INFO:teuthology.orchestra.run.gibba020.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2021-02-10T11:02:15.396 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:02:15.396 INFO:teuthology.orchestra.run.gibba020.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 7" 2021-02-10T11:02:15.396 INFO:teuthology.orchestra.run.gibba020.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=7.9 2021-02-10T11:02:15.396 INFO:teuthology.orchestra.run.gibba020.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2021-02-10T11:02:15.396 INFO:teuthology.orchestra.run.gibba020.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="7.9" 2021-02-10T11:02:15.396 INFO:teuthology.lock.ops:Updating gibba020.front.sepia.ceph.com on lock server 2021-02-10T11:02:15.411 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2021-02-10T11:02:15.421 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2021-02-10T11:02:15.458 INFO:teuthology.task.internal:Checking for old test directory... 2021-02-10T11:02:15.459 DEBUG:teuthology.orchestra.run.gibba012:> test '!' -e /home/ubuntu/cephtest 2021-02-10T11:02:15.463 DEBUG:teuthology.orchestra.run.gibba020:> test '!' -e /home/ubuntu/cephtest 2021-02-10T11:02:15.798 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2021-02-10T11:02:15.805 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2021-02-10T11:02:15.805 DEBUG:teuthology.orchestra.run.gibba012:> test -z $(ls -A /var/lib/ceph) 2021-02-10T11:02:15.808 DEBUG:teuthology.orchestra.run.gibba020:> test -z $(ls -A /var/lib/ceph) 2021-02-10T11:02:16.167 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2021-02-10T11:02:16.209 INFO:teuthology.run_tasks:Running task kernel... 2021-02-10T11:02:16.227 DEBUG:teuthology.task.kernel:normalized config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mgr.y': {'kdb': True, 'sha1': 'distro'}, 'mds.a': {'kdb': True, 'sha1': 'distro'}, 'osd.0': {'kdb': True, 'sha1': 'distro'}, 'osd.1': {'kdb': True, 'sha1': 'distro'}, 'osd.2': {'kdb': True, 'sha1': 'distro'}, 'osd.3': {'kdb': True, 'sha1': 'distro'}, 'client.0': {'kdb': True, 'sha1': 'distro'}, 'client.1': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}, 'mon.c': {'kdb': True, 'sha1': 'distro'}, 'mgr.x': {'kdb': True, 'sha1': 'distro'}, 'mds.b': {'kdb': True, 'sha1': 'distro'}, 'osd.4': {'kdb': True, 'sha1': 'distro'}, 'osd.5': {'kdb': True, 'sha1': 'distro'}, 'osd.6': {'kdb': True, 'sha1': 'distro'}, 'osd.7': {'kdb': True, 'sha1': 'distro'}, 'client.2': {'kdb': True, 'sha1': 'distro'}, 'client.3': {'kdb': True, 'sha1': 'distro'}} 2021-02-10T11:02:16.227 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2021-02-10T11:02:16.227 DEBUG:teuthology.orchestra.run.gibba012:> uname -r 2021-02-10T11:02:16.563 INFO:teuthology.orchestra.run.gibba012.stdout:3.10.0-1160.6.1.el7.x86_64 2021-02-10T11:02:16.563 INFO:teuthology.task.kernel:Running kernel on gibba012: 3.10.0-1160.6.1.el7.x86_64 2021-02-10T11:02:16.563 DEBUG:teuthology.orchestra.run.gibba012:> sudo yum install -y kernel 2021-02-10T11:02:17.816 INFO:teuthology.orchestra.run.gibba012.stdout:Loaded plugins: fastestmirror, langpacks, priorities, product-id, search- 2021-02-10T11:02:17.816 INFO:teuthology.orchestra.run.gibba012.stdout: : disabled-repos, subscription-manager 2021-02-10T11:02:17.985 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:02:17.985 INFO:teuthology.orchestra.run.gibba012.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2021-02-10T11:02:17.985 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:02:18.016 INFO:teuthology.orchestra.run.gibba012.stdout:Determining fastest mirrors 2021-02-10T11:02:18.094 INFO:teuthology.orchestra.run.gibba012.stdout: * epel: download-cc-rdu01.fedoraproject.org 2021-02-10T11:02:19.034 INFO:teuthology.orchestra.run.gibba012.stdout:Nothing to do 2021-02-10T11:02:19.048 DEBUG:teuthology.orchestra.run.gibba012:> echo no | sudo yum reinstall kernel || true 2021-02-10T11:02:19.500 INFO:teuthology.orchestra.run.gibba012.stdout:Loaded plugins: fastestmirror, langpacks, priorities, product-id, search- 2021-02-10T11:02:19.882 INFO:teuthology.orchestra.run.gibba012.stdout: : disabled-repos, subscription-manager 2021-02-10T11:02:19.884 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:02:19.884 INFO:teuthology.orchestra.run.gibba012.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2021-02-10T11:02:19.884 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:02:19.884 INFO:teuthology.orchestra.run.gibba012.stdout:Loading mirror speeds from cached hostfile 2021-02-10T11:02:19.884 INFO:teuthology.orchestra.run.gibba012.stdout: * epel: download-cc-rdu01.fedoraproject.org 2021-02-10T11:02:19.884 INFO:teuthology.orchestra.run.gibba012.stdout:Installed package kernel-3.10.0-1160.el7.x86_64 (from anaconda) not available. 2021-02-10T11:02:19.884 INFO:teuthology.orchestra.run.gibba012.stderr:Skipping the running kernel: kernel-3.10.0-1160.6.1.el7.x86_64 2021-02-10T11:02:19.885 INFO:teuthology.orchestra.run.gibba012.stderr:Error: Nothing to do 2021-02-10T11:02:19.885 DEBUG:teuthology.orchestra.run.gibba012:> rpm -q kernel --last | head -n 1 2021-02-10T11:02:20.328 INFO:teuthology.orchestra.run.gibba012.stdout:kernel-3.10.0-1160.6.1.el7.x86_64 Wed 11 Nov 2020 03:14:13 PM UTC 2021-02-10T11:02:20.330 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 3.10.0-1160.6.1.el7.x86_64 2021-02-10T11:02:20.330 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2021-02-10T11:02:20.330 DEBUG:teuthology.orchestra.run.gibba020:> uname -r 2021-02-10T11:02:20.671 INFO:teuthology.orchestra.run.gibba020.stdout:3.10.0-1160.6.1.el7.x86_64 2021-02-10T11:02:20.672 INFO:teuthology.task.kernel:Running kernel on gibba020: 3.10.0-1160.6.1.el7.x86_64 2021-02-10T11:02:20.672 DEBUG:teuthology.orchestra.run.gibba020:> sudo yum install -y kernel 2021-02-10T11:02:21.920 INFO:teuthology.orchestra.run.gibba020.stdout:Loaded plugins: fastestmirror, langpacks, priorities, product-id, search- 2021-02-10T11:02:21.920 INFO:teuthology.orchestra.run.gibba020.stdout: : disabled-repos, subscription-manager 2021-02-10T11:02:22.079 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:02:22.079 INFO:teuthology.orchestra.run.gibba020.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2021-02-10T11:02:22.080 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:02:22.105 INFO:teuthology.orchestra.run.gibba020.stdout:Determining fastest mirrors 2021-02-10T11:02:22.184 INFO:teuthology.orchestra.run.gibba020.stdout: * epel: download-cc-rdu01.fedoraproject.org 2021-02-10T11:02:23.246 INFO:teuthology.orchestra.run.gibba020.stdout:Nothing to do 2021-02-10T11:02:23.260 DEBUG:teuthology.orchestra.run.gibba020:> echo no | sudo yum reinstall kernel || true 2021-02-10T11:02:23.712 INFO:teuthology.orchestra.run.gibba020.stdout:Loaded plugins: fastestmirror, langpacks, priorities, product-id, search- 2021-02-10T11:02:23.713 INFO:teuthology.orchestra.run.gibba020.stdout: : disabled-repos, subscription-manager 2021-02-10T11:02:23.771 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:02:23.771 INFO:teuthology.orchestra.run.gibba020.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2021-02-10T11:02:23.771 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:02:23.826 INFO:teuthology.orchestra.run.gibba020.stderr:Skipping the running kernel: kernel-3.10.0-1160.6.1.el7.x86_64 2021-02-10T11:02:23.839 INFO:teuthology.orchestra.run.gibba020.stdout:Loading mirror speeds from cached hostfile 2021-02-10T11:02:23.839 INFO:teuthology.orchestra.run.gibba020.stdout: * epel: download-cc-rdu01.fedoraproject.org 2021-02-10T11:02:23.842 INFO:teuthology.orchestra.run.gibba020.stdout:Installed package kernel-3.10.0-1160.el7.x86_64 (from anaconda) not available. 2021-02-10T11:02:23.843 INFO:teuthology.orchestra.run.gibba020.stderr:Error: Nothing to do 2021-02-10T11:02:23.854 DEBUG:teuthology.orchestra.run.gibba020:> rpm -q kernel --last | head -n 1 2021-02-10T11:02:24.297 INFO:teuthology.orchestra.run.gibba020.stdout:kernel-3.10.0-1160.6.1.el7.x86_64 Wed 11 Nov 2020 03:14:13 PM UTC 2021-02-10T11:02:24.298 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 3.10.0-1160.6.1.el7.x86_64 2021-02-10T11:02:24.298 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2021-02-10T11:02:24.298 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2021-02-10T11:02:24.299 DEBUG:teuthology.orchestra.run.gibba012:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2021-02-10T11:02:24.664 INFO:teuthology.orchestra.run.gibba012.stdout:ttyS1 2021-02-10T11:02:24.676 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2021-02-10T11:02:24.677 DEBUG:teuthology.orchestra.run.gibba020:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2021-02-10T11:02:25.025 INFO:teuthology.orchestra.run.gibba020.stdout:ttyS1 2021-02-10T11:02:25.036 INFO:teuthology.run_tasks:Running task internal.base... 2021-02-10T11:02:25.044 INFO:teuthology.task.internal:Creating test directory... 2021-02-10T11:02:25.044 DEBUG:teuthology.orchestra.run.gibba012:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2021-02-10T11:02:25.048 DEBUG:teuthology.orchestra.run.gibba020:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2021-02-10T11:02:25.382 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2021-02-10T11:02:28.525 INFO:teuthology.run_tasks:Running task internal.archive... 2021-02-10T11:02:28.590 INFO:teuthology.task.internal:Creating archive directory... 2021-02-10T11:02:28.591 DEBUG:teuthology.orchestra.run.gibba012:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2021-02-10T11:02:28.595 DEBUG:teuthology.orchestra.run.gibba020:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2021-02-10T11:02:28.943 INFO:teuthology.run_tasks:Running task internal.coredump... 2021-02-10T11:02:28.953 INFO:teuthology.task.internal:Enabling coredump saving... 2021-02-10T11:02:28.953 DEBUG:teuthology.orchestra.run.gibba012:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2021-02-10T11:02:28.958 DEBUG:teuthology.orchestra.run.gibba020:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2021-02-10T11:02:29.298 INFO:teuthology.orchestra.run.gibba012.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-02-10T11:02:29.302 INFO:teuthology.orchestra.run.gibba020.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-02-10T11:02:29.307 INFO:teuthology.orchestra.run.gibba012.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-02-10T11:02:29.312 INFO:teuthology.orchestra.run.gibba020.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-02-10T11:02:29.313 INFO:teuthology.run_tasks:Running task internal.sudo... 2021-02-10T11:02:29.325 INFO:teuthology.task.internal:Configuring sudo... 2021-02-10T11:02:29.325 DEBUG:teuthology.orchestra.run.gibba012:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2021-02-10T11:02:29.329 DEBUG:teuthology.orchestra.run.gibba020:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2021-02-10T11:02:29.672 INFO:teuthology.run_tasks:Running task internal.syslog... 2021-02-10T11:02:29.683 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2021-02-10T11:02:29.684 DEBUG:teuthology.orchestra.run.gibba012:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2021-02-10T11:02:29.687 DEBUG:teuthology.orchestra.run.gibba020:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2021-02-10T11:02:30.020 DEBUG:teuthology.orchestra.run.gibba012:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2021-02-10T11:02:30.374 DEBUG:teuthology.orchestra.run.gibba012:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2021-02-10T11:02:30.719 DEBUG:teuthology.orchestra.run.gibba012:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2021-02-10T11:02:31.072 DEBUG:teuthology.orchestra.run.gibba012:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2021-02-10T11:02:31.415 DEBUG:teuthology.orchestra.run.gibba012:> set -ex 2021-02-10T11:02:31.415 DEBUG:teuthology.orchestra.run.gibba012:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2021-02-10T11:02:31.756 DEBUG:teuthology.orchestra.run.gibba020:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2021-02-10T11:02:32.111 DEBUG:teuthology.orchestra.run.gibba020:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2021-02-10T11:02:32.454 DEBUG:teuthology.orchestra.run.gibba020:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2021-02-10T11:02:32.807 DEBUG:teuthology.orchestra.run.gibba020:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2021-02-10T11:02:33.152 DEBUG:teuthology.orchestra.run.gibba020:> set -ex 2021-02-10T11:02:33.152 DEBUG:teuthology.orchestra.run.gibba020:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2021-02-10T11:02:33.494 DEBUG:teuthology.orchestra.run.gibba012:> sudo service rsyslog restart 2021-02-10T11:02:33.498 DEBUG:teuthology.orchestra.run.gibba020:> sudo service rsyslog restart 2021-02-10T11:02:33.871 INFO:teuthology.orchestra.run.gibba020.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2021-02-10T11:02:33.874 INFO:teuthology.orchestra.run.gibba012.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2021-02-10T11:02:33.888 INFO:teuthology.run_tasks:Running task internal.timer... 2021-02-10T11:02:33.899 INFO:teuthology.task.internal:Starting timer... 2021-02-10T11:02:33.899 INFO:teuthology.run_tasks:Running task pcp... 2021-02-10T11:02:33.988 INFO:teuthology.run_tasks:Running task selinux... 2021-02-10T11:02:34.026 DEBUG:teuthology.orchestra.run.gibba012:> sudo service auditd rotate 2021-02-10T11:02:34.391 INFO:teuthology.orchestra.run.gibba012.stdout:Rotating logs: [ OK ] 2021-02-10T11:02:34.392 DEBUG:teuthology.orchestra.run.gibba020:> sudo service auditd rotate 2021-02-10T11:02:34.748 INFO:teuthology.orchestra.run.gibba020.stdout:Rotating logs: [ OK ] 2021-02-10T11:02:34.749 DEBUG:teuthology.task.selinux:Getting current SELinux state 2021-02-10T11:02:34.749 DEBUG:teuthology.orchestra.run.gibba012:> /usr/sbin/getenforce 2021-02-10T11:02:35.087 INFO:teuthology.orchestra.run.gibba012.stdout:Permissive 2021-02-10T11:02:35.088 DEBUG:teuthology.orchestra.run.gibba020:> /usr/sbin/getenforce 2021-02-10T11:02:35.427 INFO:teuthology.orchestra.run.gibba020.stdout:Permissive 2021-02-10T11:02:35.427 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@gibba012.front.sepia.ceph.com': 'permissive', 'ubuntu@gibba020.front.sepia.ceph.com': 'permissive'} 2021-02-10T11:02:35.428 DEBUG:teuthology.orchestra.run.gibba012:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\)' 2021-02-10T11:02:35.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:02:35.770 DEBUG:teuthology.orchestra.run.gibba020:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\)' 2021-02-10T11:02:36.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:02:36.115 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2021-02-10T11:02:36.115 DEBUG:teuthology.orchestra.run.gibba012:> sudo /usr/sbin/setenforce permissive 2021-02-10T11:02:36.459 DEBUG:teuthology.orchestra.run.gibba020:> sudo /usr/sbin/setenforce permissive 2021-02-10T11:02:36.803 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2021-02-10T11:02:36.817 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master was just updated or references a specific commit; assuming it is current 2021-02-10T11:02:36.817 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to origin/master 2021-02-10T11:02:37.006 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}] 2021-02-10T11:02:37.009 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit gibba012.front.sepia.ceph.com,gibba020.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2021-02-10T11:02:37.784 INFO:teuthology.task.ansible.out:Using /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/ansible.cfg as config file 2021-02-10T11:02:39.844 INFO:teuthology.task.ansible.out:[WARNING]: file /home/teuthworker/src/git.ceph.com_git_ceph-cm- ansible_master/roles/testnode/tasks/setup-opensuse.yml is empty and had no tasks to include 2021-02-10T11:02:42.832 INFO:teuthology.task.ansible.out: PLAY [all] ********************************************************************* 2021-02-10T11:02:42.881 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:02:42 +0000 (0:00:00.169) 0:00:00.169 **** 2021-02-10T11:02:42.900 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:02:42 +0000 (0:00:00.019) 0:00:00.188 **** 2021-02-10T11:02:45.677 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [gibba012.front.sepia.ceph.com] 2021-02-10T11:02:45.759 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:02:45 +0000 (0:00:02.858) 0:00:03.047 **** 2021-02-10T11:02:45.771 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] 2021-02-10T11:02:45.843 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:02:45 +0000 (0:00:00.084) 0:00:03.132 **** 2021-02-10T11:02:46.331 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the sudo group.] ******************************** 2021-02-10T11:02:46.333 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2021-02-10T11:02:46.367 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:02:46 +0000 (0:00:00.523) 0:00:03.655 **** 2021-02-10T11:02:46.373 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2021-02-10T11:02:46.408 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:02:46 +0000 (0:00:00.040) 0:00:03.696 **** 2021-02-10T11:02:47.055 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the ansible user.] ****************************** 2021-02-10T11:02:47.057 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} ok: [gibba012.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2021-02-10T11:02:47.092 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:02:47 +0000 (0:00:00.684) 0:00:04.380 **** 2021-02-10T11:02:47.124 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:02:47 +0000 (0:00:00.031) 0:00:04.412 **** 2021-02-10T11:02:47.129 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Delete the ansible users password.] ******************** 2021-02-10T11:02:47.130 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:02:47.164 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:02:47 +0000 (0:00:00.039) 0:00:04.452 **** 2021-02-10T11:02:47.170 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:02:47.206 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:02:47 +0000 (0:00:00.042) 0:00:04.494 **** 2021-02-10T11:02:47.785 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Ensure includedir is present in sudoers.] ************** 2021-02-10T11:02:47.786 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-02-10T11:02:47.820 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:02:47 +0000 (0:00:00.614) 0:00:05.109 **** 2021-02-10T11:02:47.827 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-02-10T11:02:47.871 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:02:47 +0000 (0:00:00.050) 0:00:05.159 **** 2021-02-10T11:02:50.165 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the cephlab_sudo sudoers.d file.] *************** 2021-02-10T11:02:50.167 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => {"changed": false, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/cephlab_sudo", "secontext": "system_u:object_r:etc_runtime_t:s0", "size": 182, "state": "file", "uid": 0} 2021-02-10T11:02:50.204 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:02:50 +0000 (0:00:02.333) 0:00:07.492 **** 2021-02-10T11:02:50.211 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => {"changed": false, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/cephlab_sudo", "secontext": "system_u:object_r:etc_runtime_t:s0", "size": 182, "state": "file", "uid": 0} 2021-02-10T11:02:50.248 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:02:50 +0000 (0:00:00.043) 0:00:07.536 **** 2021-02-10T11:02:50.854 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Add authorized keys for the ansible user.] ************* 2021-02-10T11:02:50.855 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4 2021-02-10T11:02:50.857 INFO:teuthology.task.ansible.out:r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} ok: [gibba020.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.c 2021-02-10T11:02:50.858 INFO:teuthology.task.ansible.out:om\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@ak 2021-02-10T11:02:50.860 INFO:teuthology.task.ansible.out:lap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2021-02-10T11:02:50.990 INFO:teuthology.task.ansible.out: PLAY [teuthology] ************************************************************** skipping: no hosts matched 2021-02-10T11:02:50.999 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2021-02-10T11:02:51.183 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:02:51 +0000 (0:00:00.934) 0:00:08.471 **** 2021-02-10T11:02:51.218 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:02:51 +0000 (0:00:00.035) 0:00:08.506 **** 2021-02-10T11:02:52.268 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* 2021-02-10T11:02:52.269 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] 2021-02-10T11:02:52.355 INFO:teuthology.task.ansible.out:[WARNING]: Using run_once with the free strategy is not currently supported. This task will still be executed for every host in the inventory list. 2021-02-10T11:02:52.356 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:02:52 +0000 (0:00:01.136) 0:00:09.643 **** 2021-02-10T11:02:52.367 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] 2021-02-10T11:02:52.442 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:02:52 +0000 (0:00:00.087) 0:00:09.731 **** 2021-02-10T11:02:53.112 INFO:teuthology.task.ansible.out: TASK [users : Check keys_repo HEAD sha1] *************************************** changed: [gibba020.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.324703", "end": "2021-02-10 11:02:53.059675", "rc": 0, "start": "2021-02-10 11:02:52.734972", "stderr": "", "stderr_lines": [], "stdout": "df8946f08e870e2d2565b79099289ca48bd825f3", "stdout_lines": ["df8946f08e870e2d2565b79099289ca48bd825f3"]} 2021-02-10T11:02:53.113 INFO:teuthology.task.ansible.out:changed: [gibba012.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.319870", "end": "2021-02-10 11:02:53.060138", "rc": 0, "start": "2021-02-10 11:02:52.740268", "stderr": "", "stderr_lines": [], "stdout": "df8946f08e870e2d2565b79099289ca48bd825f3", "stdout_lines": ["df8946f08e870e2d2565b79099289ca48bd825f3"]} 2021-02-10T11:02:53.149 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:02:53 +0000 (0:00:00.706) 0:00:10.437 **** 2021-02-10T11:02:53.189 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:02:53 +0000 (0:00:00.039) 0:00:10.477 **** 2021-02-10T11:02:53.595 INFO:teuthology.task.ansible.out: TASK [users : Check host's /keys-repo-sha1 sentinel file] ********************** 2021-02-10T11:02:53.596 INFO:teuthology.task.ansible.out:changed: [gibba012.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.005867", "end": "2021-02-10 11:02:53.583675", "failed_when_result": false, "rc": 0, "start": "2021-02-10 11:02:53.577808", "stderr": "", "stderr_lines": [], "stdout": "46fd2a03b54c2ce7520f33746fa0586027b558c3", "stdout_lines": ["46fd2a03b54c2ce7520f33746fa0586027b558c3"]} 2021-02-10T11:02:53.641 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:02:53 +0000 (0:00:00.452) 0:00:10.930 **** 2021-02-10T11:02:53.649 INFO:teuthology.task.ansible.out:changed: [gibba020.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.005867", "end": "2021-02-10 11:02:53.627901", "failed_when_result": false, "rc": 0, "start": "2021-02-10 11:02:53.622034", "stderr": "", "stderr_lines": [], "stdout": "46fd2a03b54c2ce7520f33746fa0586027b558c3", "stdout_lines": ["46fd2a03b54c2ce7520f33746fa0586027b558c3"]} 2021-02-10T11:02:53.682 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:02:53 +0000 (0:00:00.040) 0:00:10.970 **** 2021-02-10T11:02:53.687 INFO:teuthology.task.ansible.out: TASK [users : Determine if we can skip users and pubkeys updates] ************** 2021-02-10T11:02:53.689 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:02:53.723 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:02:53 +0000 (0:00:00.041) 0:00:11.011 **** 2021-02-10T11:02:53.729 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:02:53.763 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:02:53 +0000 (0:00:00.040) 0:00:11.052 **** 2021-02-10T11:02:53.769 INFO:teuthology.task.ansible.out: TASK [users : Merge extra_admin_users into managed_admin_users] **************** skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:02:53.823 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:02:53 +0000 (0:00:00.059) 0:00:11.111 **** 2021-02-10T11:02:53.831 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:02:53.866 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:02:53 +0000 (0:00:00.043) 0:00:11.154 **** 2021-02-10T11:02:53.872 INFO:teuthology.task.ansible.out: TASK [users : Remove managed_admin_users from managed_users] ******************* 2021-02-10T11:02:53.873 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:02:53.907 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:02:53 +0000 (0:00:00.041) 0:00:11.196 **** 2021-02-10T11:02:53.915 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:02:53.950 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:02:53 +0000 (0:00:00.042) 0:00:11.238 **** 2021-02-10T11:02:53.956 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_users list] *********************************** 2021-02-10T11:02:53.958 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:02:53.992 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:02:53 +0000 (0:00:00.041) 0:00:11.280 **** 2021-02-10T11:02:53.999 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:02:54.033 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:02:54 +0000 (0:00:00.041) 0:00:11.321 **** 2021-02-10T11:02:54.039 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_admin_users list] ***************************** 2021-02-10T11:02:54.040 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:02:54.078 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:02:54 +0000 (0:00:00.044) 0:00:11.366 **** 2021-02-10T11:02:54.085 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:02:54.123 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:02:54 +0000 (0:00:00.045) 0:00:11.412 **** 2021-02-10T11:02:54.549 INFO:teuthology.task.ansible.out: TASK [users : Set UID_MIN to 1001] ********************************************* 2021-02-10T11:02:54.551 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-02-10T11:02:54.587 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:02:54 +0000 (0:00:00.463) 0:00:11.875 **** 2021-02-10T11:02:54.597 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-02-10T11:02:54.635 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:02:54 +0000 (0:00:00.048) 0:00:11.924 **** 2021-02-10T11:02:55.093 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ 2021-02-10T11:02:55.094 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2021-02-10T11:02:55.136 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2021-02-10T11:02:55.535 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2021-02-10T11:02:55.568 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2021-02-10T11:02:55.971 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2021-02-10T11:02:56.014 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2021-02-10T11:02:56.416 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1104} 2021-02-10T11:02:56.449 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1104} 2021-02-10T11:02:56.861 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1105} 2021-02-10T11:02:56.893 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1105} 2021-02-10T11:02:57.292 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1106} 2021-02-10T11:02:57.334 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1106} 2021-02-10T11:02:57.733 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1107} 2021-02-10T11:02:57.767 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1107} 2021-02-10T11:02:58.177 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/jcaratzas", "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "move_home": false, "name": "jcaratzas", "shell": "/bin/bash", "state": "present", "uid": 1108} 2021-02-10T11:02:58.210 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/jcaratzas", "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "move_home": false, "name": "jcaratzas", "shell": "/bin/bash", "state": "present", "uid": 1108} 2021-02-10T11:02:58.617 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1109} 2021-02-10T11:02:58.660 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1109} 2021-02-10T11:02:59.056 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1110} 2021-02-10T11:02:59.099 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1110} 2021-02-10T11:02:59.499 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1111} 2021-02-10T11:02:59.543 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1111} 2021-02-10T11:02:59.932 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/aemerson", "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1112} 2021-02-10T11:02:59.985 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/aemerson", "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1112} 2021-02-10T11:03:00.377 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1113, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1113} 2021-02-10T11:03:00.421 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1113, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1113} 2021-02-10T11:03:00.811 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1114} 2021-02-10T11:03:00.865 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1114} 2021-02-10T11:03:01.261 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1115} 2021-02-10T11:03:01.304 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1115} 2021-02-10T11:03:01.704 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1116} 2021-02-10T11:03:01.747 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1116} 2021-02-10T11:03:02.142 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1117} 2021-02-10T11:03:02.196 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1117} 2021-02-10T11:03:02.584 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1118, "groups": "sudo", "home": "/home/davidz", "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "move_home": false, "name": "davidz", "shell": "/bin/bash", "state": "present", "uid": 1118} 2021-02-10T11:03:02.637 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1118, "groups": "sudo", "home": "/home/davidz", "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "move_home": false, "name": "davidz", "shell": "/bin/bash", "state": "present", "uid": 1118} 2021-02-10T11:03:03.031 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/cmeno", "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "move_home": false, "name": "cmeno", "shell": "/bin/bash", "state": "present", "uid": 1119} 2021-02-10T11:03:03.074 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/cmeno", "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "move_home": false, "name": "cmeno", "shell": "/bin/bash", "state": "present", "uid": 1119} 2021-02-10T11:03:03.477 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1120} 2021-02-10T11:03:03.520 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1120} 2021-02-10T11:03:03.938 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1121, "groups": "sudo", "home": "/home/wusui", "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "move_home": false, "name": "wusui", "shell": "/bin/bash", "state": "present", "uid": 1121} 2021-02-10T11:03:03.972 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1121, "groups": "sudo", "home": "/home/wusui", "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "move_home": false, "name": "wusui", "shell": "/bin/bash", "state": "present", "uid": 1121} 2021-02-10T11:03:04.384 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1123} 2021-02-10T11:03:04.416 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1123} 2021-02-10T11:03:04.824 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1124} 2021-02-10T11:03:04.856 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1124} 2021-02-10T11:03:05.262 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1125} 2021-02-10T11:03:05.294 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1125} 2021-02-10T11:03:05.700 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1126} 2021-02-10T11:03:05.733 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1126} 2021-02-10T11:03:06.130 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1127} 2021-02-10T11:03:06.173 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1127} 2021-02-10T11:03:06.570 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1128} 2021-02-10T11:03:06.602 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1128} 2021-02-10T11:03:07.011 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1129} 2021-02-10T11:03:07.044 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1129} 2021-02-10T11:03:07.453 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1130} 2021-02-10T11:03:07.486 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1130} 2021-02-10T11:03:07.904 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1131} 2021-02-10T11:03:07.926 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1131} 2021-02-10T11:03:08.338 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1132} 2021-02-10T11:03:08.425 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1132} 2021-02-10T11:03:08.789 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1133, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1133} 2021-02-10T11:03:08.896 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1133, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1133} 2021-02-10T11:03:09.248 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/nwatkins", "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "move_home": false, "name": "nwatkins", "shell": "/bin/bash", "state": "present", "uid": 1134} 2021-02-10T11:03:09.344 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/nwatkins", "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "move_home": false, "name": "nwatkins", "shell": "/bin/bash", "state": "present", "uid": 1134} 2021-02-10T11:03:09.692 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1135, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1135} 2021-02-10T11:03:09.787 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1135, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1135} 2021-02-10T11:03:10.133 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1136} 2021-02-10T11:03:10.229 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1136} 2021-02-10T11:03:10.577 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1137, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1137} 2021-02-10T11:03:10.672 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1137, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1137} 2021-02-10T11:03:11.018 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/jdillaman", "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "move_home": false, "name": "jdillaman", "shell": "/bin/bash", "state": "present", "uid": 1138} 2021-02-10T11:03:11.112 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/jdillaman", "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "move_home": false, "name": "jdillaman", "shell": "/bin/bash", "state": "present", "uid": 1138} 2021-02-10T11:03:11.458 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1139} 2021-02-10T11:03:11.552 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1139} 2021-02-10T11:03:11.554 INFO:teuthology.task.ansible.out: 2021-02-10T11:03:11.891 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1140} 2021-02-10T11:03:11.987 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1140} 2021-02-10T11:03:12.336 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1141} 2021-02-10T11:03:12.431 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1141} 2021-02-10T11:03:12.775 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1142} 2021-02-10T11:03:12.869 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1142} 2021-02-10T11:03:13.224 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1143} 2021-02-10T11:03:13.308 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1143} 2021-02-10T11:03:13.673 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1144} 2021-02-10T11:03:13.757 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1144} 2021-02-10T11:03:14.112 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1145, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1145} 2021-02-10T11:03:14.197 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1145, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1145} 2021-02-10T11:03:14.560 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1146} 2021-02-10T11:03:14.634 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1146} 2021-02-10T11:03:15.006 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1147} 2021-02-10T11:03:15.073 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1147} 2021-02-10T11:03:15.448 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1148} 2021-02-10T11:03:15.512 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1148} 2021-02-10T11:03:15.877 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1149, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1149} 2021-02-10T11:03:15.972 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1149, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1149} 2021-02-10T11:03:16.317 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1150} 2021-02-10T11:03:16.412 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1150} 2021-02-10T11:03:16.755 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1151, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1151} 2021-02-10T11:03:16.850 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1151, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1151} 2021-02-10T11:03:17.193 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1152, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1152} 2021-02-10T11:03:17.287 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1152, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1152} 2021-02-10T11:03:17.630 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1153} 2021-02-10T11:03:17.725 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1153} 2021-02-10T11:03:18.073 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1154, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1154} 2021-02-10T11:03:18.170 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1154, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1154} 2021-02-10T11:03:18.519 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1155} 2021-02-10T11:03:18.606 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1155} 2021-02-10T11:03:18.953 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1156} 2021-02-10T11:03:19.047 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1156} 2021-02-10T11:03:19.390 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1157} 2021-02-10T11:03:19.485 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1157} 2021-02-10T11:03:19.831 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1158} 2021-02-10T11:03:19.937 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1158} 2021-02-10T11:03:20.275 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1159} 2021-02-10T11:03:20.371 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1159} 2021-02-10T11:03:20.714 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1160, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1160} 2021-02-10T11:03:20.810 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1160, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1160} 2021-02-10T11:03:21.154 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1161} 2021-02-10T11:03:21.249 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1161} 2021-02-10T11:03:21.591 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1162} 2021-02-10T11:03:21.685 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1162} 2021-02-10T11:03:22.029 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1163} 2021-02-10T11:03:22.123 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1163} 2021-02-10T11:03:22.471 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1164} 2021-02-10T11:03:22.565 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1164} 2021-02-10T11:03:22.922 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1140} 2021-02-10T11:03:23.006 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1140} 2021-02-10T11:03:23.352 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1165} 2021-02-10T11:03:23.448 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1165} 2021-02-10T11:03:23.793 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1166, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1166} 2021-02-10T11:03:23.888 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1166, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1166} 2021-02-10T11:03:24.223 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1167} 2021-02-10T11:03:24.333 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1167} 2021-02-10T11:03:24.668 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1168} 2021-02-10T11:03:24.773 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1168} 2021-02-10T11:03:25.106 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1169} 2021-02-10T11:03:25.210 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1169} 2021-02-10T11:03:25.545 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1170} 2021-02-10T11:03:25.650 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1170} 2021-02-10T11:03:25.983 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1171} 2021-02-10T11:03:26.090 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1171} 2021-02-10T11:03:26.424 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1172} 2021-02-10T11:03:26.529 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1172} 2021-02-10T11:03:26.863 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1173, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1173} 2021-02-10T11:03:26.972 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1173, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1173} 2021-02-10T11:03:27.305 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1174} 2021-02-10T11:03:27.421 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1174} 2021-02-10T11:03:27.743 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1175, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1175} 2021-02-10T11:03:27.858 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1175, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1175} 2021-02-10T11:03:28.190 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1176, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1176} 2021-02-10T11:03:28.295 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1176, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1176} 2021-02-10T11:03:28.622 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1177} 2021-02-10T11:03:28.728 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1177} 2021-02-10T11:03:29.077 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1178} 2021-02-10T11:03:29.181 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1178} 2021-02-10T11:03:29.182 INFO:teuthology.task.ansible.out: 2021-02-10T11:03:29.553 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1179} 2021-02-10T11:03:29.629 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1179} 2021-02-10T11:03:30.003 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1180} 2021-02-10T11:03:30.079 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1180} 2021-02-10T11:03:30.467 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1181} 2021-02-10T11:03:30.532 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1181} 2021-02-10T11:03:30.930 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1182} 2021-02-10T11:03:30.993 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1182} 2021-02-10T11:03:31.373 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1183, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1183} 2021-02-10T11:03:31.436 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1183, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1183} 2021-02-10T11:03:31.814 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1184, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1184} 2021-02-10T11:03:31.878 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1184, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1184} 2021-02-10T11:03:32.255 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/sidharthanup", "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "move_home": false, "name": "sidharthanup", "shell": "/bin/bash", "state": "present", "uid": 1185} 2021-02-10T11:03:32.319 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/sidharthanup", "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "move_home": false, "name": "sidharthanup", "shell": "/bin/bash", "state": "present", "uid": 1185} 2021-02-10T11:03:32.701 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/varsha", "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "move_home": false, "name": "varsha", "shell": "/bin/bash", "state": "present", "uid": 1186} 2021-02-10T11:03:32.764 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/varsha", "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "move_home": false, "name": "varsha", "shell": "/bin/bash", "state": "present", "uid": 1186} 2021-02-10T11:03:33.141 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1187} 2021-02-10T11:03:33.204 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1187} 2021-02-10T11:03:33.581 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1188} 2021-02-10T11:03:33.646 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1188} 2021-02-10T11:03:34.013 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1189} 2021-02-10T11:03:34.087 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1189} 2021-02-10T11:03:34.452 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1190} 2021-02-10T11:03:34.527 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1190} 2021-02-10T11:03:34.891 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1191} 2021-02-10T11:03:34.976 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1191} 2021-02-10T11:03:35.331 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1192} 2021-02-10T11:03:35.416 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1192} 2021-02-10T11:03:35.771 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1193} 2021-02-10T11:03:35.855 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1193} 2021-02-10T11:03:36.210 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1194} 2021-02-10T11:03:36.295 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1194} 2021-02-10T11:03:36.652 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1195} 2021-02-10T11:03:36.738 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1195} 2021-02-10T11:03:37.082 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "move_home": false, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "uid": 1196} 2021-02-10T11:03:37.176 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "move_home": false, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "uid": 1196} 2021-02-10T11:03:37.530 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "move_home": false, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "uid": 1197} 2021-02-10T11:03:37.614 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "move_home": false, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "uid": 1197} 2021-02-10T11:03:37.969 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "move_home": false, "name": "shyukri", "shell": "/bin/bash", "state": "present", "uid": 1198} 2021-02-10T11:03:37.970 INFO:teuthology.task.ansible.out: 2021-02-10T11:03:38.053 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "move_home": false, "name": "shyukri", "shell": "/bin/bash", "state": "present", "uid": 1198} 2021-02-10T11:03:38.399 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "move_home": false, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "uid": 1199} 2021-02-10T11:03:38.497 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "move_home": false, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "uid": 1199} 2021-02-10T11:03:38.841 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "move_home": false, "name": "rakesh", "shell": "/bin/bash", "state": "present", "uid": 1200} 2021-02-10T11:03:38.936 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "move_home": false, "name": "rakesh", "shell": "/bin/bash", "state": "present", "uid": 1200} 2021-02-10T11:03:39.280 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "move_home": false, "name": "ryg-", "shell": "/bin/bash", "state": "present", "uid": 1201} 2021-02-10T11:03:39.375 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "move_home": false, "name": "ryg-", "shell": "/bin/bash", "state": "present", "uid": 1201} 2021-02-10T11:03:39.376 INFO:teuthology.task.ansible.out: 2021-02-10T11:03:39.719 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1148} 2021-02-10T11:03:39.825 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1148} 2021-02-10T11:03:40.164 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/zdover", "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "move_home": false, "name": "zdover", "shell": "/bin/bash", "state": "present", "uid": 1202} 2021-02-10T11:03:40.282 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/zdover", "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "move_home": false, "name": "zdover", "shell": "/bin/bash", "state": "present", "uid": 1202} 2021-02-10T11:03:40.616 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/rraja", "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "move_home": false, "name": "rraja", "shell": "/bin/bash", "state": "present", "uid": 1203} 2021-02-10T11:03:40.731 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/rraja", "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "move_home": false, "name": "rraja", "shell": "/bin/bash", "state": "present", "uid": 1203} 2021-02-10T11:03:41.057 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/toabctl", "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "move_home": false, "name": "toabctl", "shell": "/bin/bash", "state": "present", "uid": 1204} 2021-02-10T11:03:41.174 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/toabctl", "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "move_home": false, "name": "toabctl", "shell": "/bin/bash", "state": "present", "uid": 1204} 2021-02-10T11:03:41.501 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/mgfritch", "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "move_home": false, "name": "mgfritch", "shell": "/bin/bash", "state": "present", "uid": 1205} 2021-02-10T11:03:41.618 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/mgfritch", "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "move_home": false, "name": "mgfritch", "shell": "/bin/bash", "state": "present", "uid": 1205} 2021-02-10T11:03:41.933 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1206, "groups": "sudo", "home": "/home/xiubli", "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "move_home": false, "name": "xiubli", "shell": "/bin/bash", "state": "present", "uid": 1206} 2021-02-10T11:03:42.059 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1206, "groups": "sudo", "home": "/home/xiubli", "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "move_home": false, "name": "xiubli", "shell": "/bin/bash", "state": "present", "uid": 1206} 2021-02-10T11:03:42.372 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1207, "groups": "sudo", "home": "/home/jafaj", "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "move_home": false, "name": "jafaj", "shell": "/bin/bash", "state": "present", "uid": 1207} 2021-02-10T11:03:42.503 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1207, "groups": "sudo", "home": "/home/jafaj", "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "move_home": false, "name": "jafaj", "shell": "/bin/bash", "state": "present", "uid": 1207} 2021-02-10T11:03:42.805 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1208, "groups": "sudo", "home": "/home/jschmid", "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "move_home": false, "name": "jschmid", "shell": "/bin/bash", "state": "present", "uid": 1208} 2021-02-10T11:03:42.942 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1208, "groups": "sudo", "home": "/home/jschmid", "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "move_home": false, "name": "jschmid", "shell": "/bin/bash", "state": "present", "uid": 1208} 2021-02-10T11:03:43.244 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1209, "groups": "sudo", "home": "/home/kachau", "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "move_home": false, "name": "kachau", "shell": "/bin/bash", "state": "present", "uid": 1209} 2021-02-10T11:03:43.380 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1209, "groups": "sudo", "home": "/home/kachau", "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "move_home": false, "name": "kachau", "shell": "/bin/bash", "state": "present", "uid": 1209} 2021-02-10T11:03:43.698 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1210, "groups": "sudo", "home": "/home/khiremat", "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "move_home": false, "name": "khiremat", "shell": "/bin/bash", "state": "present", "uid": 1210} 2021-02-10T11:03:43.823 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1210, "groups": "sudo", "home": "/home/khiremat", "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "move_home": false, "name": "khiremat", "shell": "/bin/bash", "state": "present", "uid": 1210} 2021-02-10T11:03:44.138 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1211, "groups": "sudo", "home": "/home/kapandya", "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "move_home": false, "name": "kapandya", "shell": "/bin/bash", "state": "present", "uid": 1211} 2021-02-10T11:03:44.253 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1211, "groups": "sudo", "home": "/home/kapandya", "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "move_home": false, "name": "kapandya", "shell": "/bin/bash", "state": "present", "uid": 1211} 2021-02-10T11:03:44.570 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1212, "groups": "sudo", "home": "/home/prasad", "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "move_home": false, "name": "prasad", "shell": "/bin/bash", "state": "present", "uid": 1212} 2021-02-10T11:03:44.698 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1212, "groups": "sudo", "home": "/home/prasad", "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "move_home": false, "name": "prasad", "shell": "/bin/bash", "state": "present", "uid": 1212} 2021-02-10T11:03:45.016 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1213, "groups": "sudo", "home": "/home/vikhyat", "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "move_home": false, "name": "vikhyat", "shell": "/bin/bash", "state": "present", "uid": 1213} 2021-02-10T11:03:45.133 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1213, "groups": "sudo", "home": "/home/vikhyat", "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "move_home": false, "name": "vikhyat", "shell": "/bin/bash", "state": "present", "uid": 1213} 2021-02-10T11:03:45.469 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1214, "groups": "sudo", "home": "/home/shraddhaag", "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "move_home": false, "name": "shraddhaag", "shell": "/bin/bash", "state": "present", "uid": 1214} 2021-02-10T11:03:45.586 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1214, "groups": "sudo", "home": "/home/shraddhaag", "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "move_home": false, "name": "shraddhaag", "shell": "/bin/bash", "state": "present", "uid": 1214} 2021-02-10T11:03:45.915 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1215, "groups": "sudo", "home": "/home/gkyratsas", "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "move_home": false, "name": "gkyratsas", "shell": "/bin/bash", "state": "present", "uid": 1215} 2021-02-10T11:03:46.032 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1215, "groups": "sudo", "home": "/home/gkyratsas", "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "move_home": false, "name": "gkyratsas", "shell": "/bin/bash", "state": "present", "uid": 1215} 2021-02-10T11:03:46.356 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1216, "groups": "sudo", "home": "/home/prateek", "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "move_home": false, "name": "prateek", "shell": "/bin/bash", "state": "present", "uid": 1216} 2021-02-10T11:03:46.484 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1216, "groups": "sudo", "home": "/home/prateek", "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "move_home": false, "name": "prateek", "shell": "/bin/bash", "state": "present", "uid": 1216} 2021-02-10T11:03:46.797 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1217, "groups": "sudo", "home": "/home/jelopez", "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "move_home": false, "name": "jelopez", "shell": "/bin/bash", "state": "present", "uid": 1217} 2021-02-10T11:03:46.912 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1217, "groups": "sudo", "home": "/home/jelopez", "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "move_home": false, "name": "jelopez", "shell": "/bin/bash", "state": "present", "uid": 1217} 2021-02-10T11:03:47.237 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1218, "groups": "sudo", "home": "/home/dcain", "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "move_home": false, "name": "dcain", "shell": "/bin/bash", "state": "present", "uid": 1218} 2021-02-10T11:03:47.353 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1218, "groups": "sudo", "home": "/home/dcain", "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "move_home": false, "name": "dcain", "shell": "/bin/bash", "state": "present", "uid": 1218} 2021-02-10T11:03:47.680 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1219, "groups": "sudo", "home": "/home/sunny", "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "move_home": false, "name": "sunny", "shell": "/bin/bash", "state": "present", "uid": 1219} 2021-02-10T11:03:47.796 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1219, "groups": "sudo", "home": "/home/sunny", "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "move_home": false, "name": "sunny", "shell": "/bin/bash", "state": "present", "uid": 1219} 2021-02-10T11:03:48.119 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1220, "groups": "sudo", "home": "/home/xuxuehan", "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "move_home": false, "name": "xuxuehan", "shell": "/bin/bash", "state": "present", "uid": 1220} 2021-02-10T11:03:48.244 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1220, "groups": "sudo", "home": "/home/xuxuehan", "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "move_home": false, "name": "xuxuehan", "shell": "/bin/bash", "state": "present", "uid": 1220} 2021-02-10T11:03:48.561 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1221, "groups": "sudo", "home": "/home/aclewett", "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "move_home": false, "name": "aclewett", "shell": "/bin/bash", "state": "present", "uid": 1221} 2021-02-10T11:03:48.687 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1221, "groups": "sudo", "home": "/home/aclewett", "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "move_home": false, "name": "aclewett", "shell": "/bin/bash", "state": "present", "uid": 1221} 2021-02-10T11:03:49.002 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1222, "groups": "sudo", "home": "/home/cblum", "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "move_home": false, "name": "cblum", "shell": "/bin/bash", "state": "present", "uid": 1222} 2021-02-10T11:03:49.127 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1222, "groups": "sudo", "home": "/home/cblum", "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "move_home": false, "name": "cblum", "shell": "/bin/bash", "state": "present", "uid": 1222} 2021-02-10T11:03:49.440 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1223, "groups": "sudo", "home": "/home/mlatimer", "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "move_home": false, "name": "mlatimer", "shell": "/bin/bash", "state": "present", "uid": 1223} 2021-02-10T11:03:49.567 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1223, "groups": "sudo", "home": "/home/mlatimer", "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "move_home": false, "name": "mlatimer", "shell": "/bin/bash", "state": "present", "uid": 1223} 2021-02-10T11:03:49.871 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1224, "groups": "sudo", "home": "/home/shyamsundarr", "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "move_home": false, "name": "shyamsundarr", "shell": "/bin/bash", "state": "present", "uid": 1224} 2021-02-10T11:03:49.997 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1224, "groups": "sudo", "home": "/home/shyamsundarr", "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "move_home": false, "name": "shyamsundarr", "shell": "/bin/bash", "state": "present", "uid": 1224} 2021-02-10T11:03:50.313 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1225, "groups": "sudo", "home": "/home/shuzhenyi", "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "move_home": false, "name": "shuzhenyi", "shell": "/bin/bash", "state": "present", "uid": 1225} 2021-02-10T11:03:50.438 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1225, "groups": "sudo", "home": "/home/shuzhenyi", "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "move_home": false, "name": "shuzhenyi", "shell": "/bin/bash", "state": "present", "uid": 1225} 2021-02-10T11:03:50.750 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1226, "groups": "sudo", "home": "/home/mdipalma", "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "move_home": false, "name": "mdipalma", "shell": "/bin/bash", "state": "present", "uid": 1226} 2021-02-10T11:03:50.879 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1226, "groups": "sudo", "home": "/home/mdipalma", "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "move_home": false, "name": "mdipalma", "shell": "/bin/bash", "state": "present", "uid": 1226} 2021-02-10T11:03:51.194 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1227, "groups": "sudo", "home": "/home/kchang", "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "move_home": false, "name": "kchang", "shell": "/bin/bash", "state": "present", "uid": 1227} 2021-02-10T11:03:51.313 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1227, "groups": "sudo", "home": "/home/kchang", "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "move_home": false, "name": "kchang", "shell": "/bin/bash", "state": "present", "uid": 1227} 2021-02-10T11:03:51.629 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1228, "groups": "sudo", "home": "/home/vereddy", "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "move_home": false, "name": "vereddy", "shell": "/bin/bash", "state": "present", "uid": 1228} 2021-02-10T11:03:51.755 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1228, "groups": "sudo", "home": "/home/vereddy", "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "move_home": false, "name": "vereddy", "shell": "/bin/bash", "state": "present", "uid": 1228} 2021-02-10T11:03:52.069 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1229, "groups": "sudo", "home": "/home/benhanokh", "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "move_home": false, "name": "benhanokh", "shell": "/bin/bash", "state": "present", "uid": 1229} 2021-02-10T11:03:52.195 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1229, "groups": "sudo", "home": "/home/benhanokh", "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "move_home": false, "name": "benhanokh", "shell": "/bin/bash", "state": "present", "uid": 1229} 2021-02-10T11:03:52.508 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1230, "groups": "sudo", "home": "/home/jsalomon", "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "move_home": false, "name": "jsalomon", "shell": "/bin/bash", "state": "present", "uid": 1230} 2021-02-10T11:03:52.636 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1230, "groups": "sudo", "home": "/home/jsalomon", "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "move_home": false, "name": "jsalomon", "shell": "/bin/bash", "state": "present", "uid": 1230} 2021-02-10T11:03:52.949 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1231, "groups": "sudo", "home": "/home/jolmomar", "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "move_home": false, "name": "jolmomar", "shell": "/bin/bash", "state": "present", "uid": 1231} 2021-02-10T11:03:53.074 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1231, "groups": "sudo", "home": "/home/jolmomar", "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "move_home": false, "name": "jolmomar", "shell": "/bin/bash", "state": "present", "uid": 1231} 2021-02-10T11:03:53.398 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1232, "groups": "sudo", "home": "/home/hmunjulu", "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "move_home": false, "name": "hmunjulu", "shell": "/bin/bash", "state": "present", "uid": 1232} 2021-02-10T11:03:53.526 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1232, "groups": "sudo", "home": "/home/hmunjulu", "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "move_home": false, "name": "hmunjulu", "shell": "/bin/bash", "state": "present", "uid": 1232} 2021-02-10T11:03:53.855 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1233, "groups": "sudo", "home": "/home/julpark", "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "move_home": false, "name": "julpark", "shell": "/bin/bash", "state": "present", "uid": 1233} 2021-02-10T11:03:53.963 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1233, "groups": "sudo", "home": "/home/julpark", "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "move_home": false, "name": "julpark", "shell": "/bin/bash", "state": "present", "uid": 1233} 2021-02-10T11:03:54.305 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1235, "groups": "sudo", "home": "/home/pdhange", "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "move_home": false, "name": "pdhange", "shell": "/bin/bash", "state": "present", "uid": 1235} 2021-02-10T11:03:54.414 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1235, "groups": "sudo", "home": "/home/pdhange", "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "move_home": false, "name": "pdhange", "shell": "/bin/bash", "state": "present", "uid": 1235} 2021-02-10T11:03:55.201 INFO:teuthology.task.ansible.out:changed: [gibba020.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10103, "groups": "sudo", "home": "/home/dpivonka", "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "name": "dpivonka", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10102} 2021-02-10T11:03:55.255 INFO:teuthology.task.ansible.out:changed: [gibba012.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10103, "groups": "sudo", "home": "/home/dpivonka", "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "name": "dpivonka", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10102} 2021-02-10T11:03:56.009 INFO:teuthology.task.ansible.out:changed: [gibba020.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10104, "groups": "sudo", "home": "/home/adking", "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "name": "adking", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10103} 2021-02-10T11:03:56.031 INFO:teuthology.task.ansible.out:changed: [gibba012.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10104, "groups": "sudo", "home": "/home/adking", "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "name": "adking", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10103} 2021-02-10T11:03:56.713 INFO:teuthology.task.ansible.out:changed: [gibba020.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10105, "groups": "sudo", "home": "/home/amathuri", "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267"}, "name": "amathuri", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10104} 2021-02-10T11:03:56.860 INFO:teuthology.task.ansible.out:changed: [gibba012.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10105, "groups": "sudo", "home": "/home/amathuri", "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267"}, "name": "amathuri", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10104} 2021-02-10T11:03:57.534 INFO:teuthology.task.ansible.out:changed: [gibba020.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10106, "groups": "sudo", "home": "/home/nizamial09", "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "name": "nizamial09", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10105} 2021-02-10T11:03:57.653 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:03:57 +0000 (0:01:03.017) 0:01:14.941 **** 2021-02-10T11:03:57.662 INFO:teuthology.task.ansible.out:changed: [gibba012.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10106, "groups": "sudo", "home": "/home/nizamial09", "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "name": "nizamial09", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10105} 2021-02-10T11:03:57.747 INFO:teuthology.task.ansible.out: TASK [users : Create all users without sudo access.] *************************** 2021-02-10T11:03:57.779 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:03:57 +0000 (0:00:00.125) 0:01:15.067 **** 2021-02-10T11:03:57.812 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:03:57 +0000 (0:00:00.032) 0:01:15.100 **** 2021-02-10T11:03:57.855 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:03:57 +0000 (0:00:00.042) 0:01:15.143 **** 2021-02-10T11:03:57.874 INFO:teuthology.task.ansible.out: TASK [users : Merge managed_users and managed_admin users] ********************* 2021-02-10T11:03:57.876 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "jcaratzas", "ovpn": "jcaratza@fedora r 2021-02-10T11:03:57.877 INFO:teuthology.task.ansible.out:T6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c 2021-02-10T11:03:57.879 INFO:teuthology.task.ansible.out:63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, {"name": "ivancich 2021-02-10T11:03:57.880 INFO:teuthology.task.ansible.out:", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiR 2021-02-10T11:03:57.881 INFO:teuthology.task.ansible.out:enVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vumr 2021-02-10T11:03:57.882 INFO:teuthology.task.ansible.out:ao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69m 2021-02-10T11:03:57.884 INFO:teuthology.task.ansible.out:h95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5 2021-02-10T11:03:57.884 INFO:teuthology.task.ansible.out:d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d84 2021-02-10T11:03:57.885 INFO:teuthology.task.ansible.out:1a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, {"name": "oliveiradan", "ovpn": "oliveiradan 2021-02-10T11:03:57.887 INFO:teuthology.task.ansible.out:@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOen 2021-02-10T11:03:57.888 INFO:teuthology.task.ansible.out:VV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9g 2021-02-10T11:03:57.889 INFO:teuthology.task.ansible.out:I46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, {"name": " 2021-02-10T11:03:57.891 INFO:teuthology.task.ansible.out:dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NIN 2021-02-10T11:03:57.892 INFO:teuthology.task.ansible.out:j0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa33958 2021-02-10T11:03:57.892 INFO:teuthology.task.ansible.out:62e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267"}, {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}]}, "changed": false} 2021-02-10T11:03:57.898 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "jcaratzas", "ovpn": "jcaratza@fedora r 2021-02-10T11:03:57.900 INFO:teuthology.task.ansible.out:T6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c 2021-02-10T11:03:57.901 INFO:teuthology.task.ansible.out:63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, {"name": "ivancich 2021-02-10T11:03:57.902 INFO:teuthology.task.ansible.out:", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiR 2021-02-10T11:03:57.904 INFO:teuthology.task.ansible.out:enVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vumr 2021-02-10T11:03:57.905 INFO:teuthology.task.ansible.out:ao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69m 2021-02-10T11:03:57.906 INFO:teuthology.task.ansible.out:h95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5 2021-02-10T11:03:57.906 INFO:teuthology.task.ansible.out:d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d84 2021-02-10T11:03:57.908 INFO:teuthology.task.ansible.out:1a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, {"name": "oliveiradan", "ovpn": "oliveiradan 2021-02-10T11:03:57.909 INFO:teuthology.task.ansible.out:@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOen 2021-02-10T11:03:57.910 INFO:teuthology.task.ansible.out:VV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9g 2021-02-10T11:03:57.912 INFO:teuthology.task.ansible.out:I46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, {"name": " 2021-02-10T11:03:57.913 INFO:teuthology.task.ansible.out:dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NIN 2021-02-10T11:03:57.915 INFO:teuthology.task.ansible.out:j0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa33958 2021-02-10T11:03:57.915 INFO:teuthology.task.ansible.out:62e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267"}, {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}]}, "changed": false} 2021-02-10T11:03:57.954 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:03:57 +0000 (0:00:00.099) 0:01:15.243 **** 2021-02-10T11:03:57.992 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:03:57 +0000 (0:00:00.037) 0:01:15.280 **** 2021-02-10T11:03:58.861 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* 2021-02-10T11:03:58.863 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com -> localhost] => {"after": "df8946f08e870e2d2565b79099289ca48bd825f3", "attempts": 1, "before": "df8946f08e870e2d2565b79099289ca48bd825f3", "changed": false, "remote_url_changed": false} 2021-02-10T11:03:58.896 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:03:58 +0000 (0:00:00.904) 0:01:16.184 **** 2021-02-10T11:03:58.903 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com -> localhost] => {"after": "df8946f08e870e2d2565b79099289ca48bd825f3", "attempts": 1, "before": "df8946f08e870e2d2565b79099289ca48bd825f3", "changed": false, "remote_url_changed": false} 2021-02-10T11:03:58.940 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:03:58 +0000 (0:00:00.044) 0:01:16.228 **** 2021-02-10T11:03:59.382 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-02-10T11:03:59.384 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDdkebJGXKQWBG/7wZG334P+EAdC+5VNMrjLjXhaZZbt3V9vKDQCodcU3zSOjMa/GDGg2WHQdO/vioCc6O/m61KTbgLDaaPSzHH16j/e1SaU8BIgKKU42AUBJzpPeaZ2wCC1i12Hq8ULwjEW29IDsnMortzlCSQmaTKc3DN2nCXZWQXA1rSiQ2AhIq5WNJY9S36lsg5tkex4xUajyNW4vPubLtkSv+WrzNK7ZJ8zfzP6eJEJ+Uj7CLdanBCgwoLGSPfGA5c/gyUT7mHibOUwvFB5sFVsozMPp2PXzUUIv70adJpOskLYfKY7TUSuaB4AUOPu6NVauzVIl3Tsn+uvHqvtn8KC3BPpyH7+JiFQWEt+aO+/ynprfoqCjktuf07SXo0ewoVYT2S++JEBa/N2hKRkzIoSPJ+Rvoazc8GZIjXpQRyhMdTV/LCSeyUTJ7lDQVrQVhUIHKnMm+hZmf4tLQKZvNAThO984CpY93ST5Ww07fjRLMXNZGuaz+zHKNexhWSPKr+ggChnZCOLl/F6zbDnBZoCaHR6hH4+QLz0K/FAgxN2Nn00RiOFkmIlzEioxMcwC/peTl3WzRfqPINdTftiTzcFi/c2QmlHhnxsueIwNhHOn4frkGkFmexdDrXRStERzSBJ13X4kWduPmQsn+bx5/7xwqC6M4bXpkYFOefOw== aschoen@redhat.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2021-02-10T11:03:59.424 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDdkebJGXKQWBG/7wZG334P+EAdC+5VNMrjLjXhaZZbt3V9vKDQCodcU3zSOjMa/GDGg2WHQdO/vioCc6O/m61KTbgLDaaPSzHH16j/e1SaU8BIgKKU42AUBJzpPeaZ2wCC1i12Hq8ULwjEW29IDsnMortzlCSQmaTKc3DN2nCXZWQXA1rSiQ2AhIq5WNJY9S36lsg5tkex4xUajyNW4vPubLtkSv+WrzNK7ZJ8zfzP6eJEJ+Uj7CLdanBCgwoLGSPfGA5c/gyUT7mHibOUwvFB5sFVsozMPp2PXzUUIv70adJpOskLYfKY7TUSuaB4AUOPu6NVauzVIl3Tsn+uvHqvtn8KC3BPpyH7+JiFQWEt+aO+/ynprfoqCjktuf07SXo0ewoVYT2S++JEBa/N2hKRkzIoSPJ+Rvoazc8GZIjXpQRyhMdTV/LCSeyUTJ7lDQVrQVhUIHKnMm+hZmf4tLQKZvNAThO984CpY93ST5Ww07fjRLMXNZGuaz+zHKNexhWSPKr+ggChnZCOLl/F6zbDnBZoCaHR6hH4+QLz0K/FAgxN2Nn00RiOFkmIlzEioxMcwC/peTl3WzRfqPINdTftiTzcFi/c2QmlHhnxsueIwNhHOn4frkGkFmexdDrXRStERzSBJ13X4kWduPmQsn+bx5/7xwqC6M4bXpkYFOefOw== aschoen@redhat.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2021-02-10T11:03:59.829 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2021-02-10T11:03:59.872 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2021-02-10T11:04:00.280 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2021-02-10T11:04:00.312 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2021-02-10T11:04:00.728 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZt 2021-02-10T11:04:00.730 INFO:teuthology.task.ansible.out:IEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOoIm6sOytKwqNf9FSQn0Olu72ZRDjAYeNFQVWTn2PIB ed25519-hplaptop", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "validate_certs": true} 2021-02-10T11:04:00.762 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZt 2021-02-10T11:04:00.763 INFO:teuthology.task.ansible.out:IEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOoIm6sOytKwqNf9FSQn0Olu72ZRDjAYeNFQVWTn2PIB ed25519-hplaptop", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "validate_certs": true} 2021-02-10T11:04:01.166 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2021-02-10T11:04:01.198 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2021-02-10T11:04:01.614 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o 2021-02-10T11:04:01.615 INFO:teuthology.task.ansible.out:8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} 2021-02-10T11:04:01.645 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o 2021-02-10T11:04:01.647 INFO:teuthology.task.ansible.out:8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} 2021-02-10T11:04:02.062 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2021-02-10T11:04:02.094 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2021-02-10T11:04:02.510 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCgaz5AsLB+hPwnHjlf6vktlUQ8iWsKA+hViok+j+uCm0U/iho7O7Qd8E/3tp9yBx10Wkvuz1mmf6v2AO7Qa8+SE1QQPmA+4xLp4xXCqlUGFjnzm/7uQcQZtlWdfBHzQ9jVyPAgUpnGVUtW8PhW0txe8MEtecOO2C3Snx0yuTOx6O11jNvT938QjOeRQWw3X2YFUtMIOiwEhKtqstBIGXlMdaQKfvdlbA9MjyYy6VfK1x8lVcu8umUf/z56YmMkC2iO117/obhdu1XsU9SmjM51wHMOUCXSVqH8E3BWu0yohy4QmZDkdFihyYa+hoS9dOqGC91dFaQc46xrdmV4wqg1 jcaratza@MacBook-Pro\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAV78mqEy/VkB2BthIE/m7iE5sbTjTRS0KuH7AB0KPvZBsDIw5cL8jKyxiaPC85+aOkBa4LrP0TUFBzN87Snt+PLMz2QHBOvlSTfUuRkHVo3vWJFzVuUSzzddkYds3hAxUriAHKv6CRSy1yFBxPQQHTSATUHKWqihMElbUCZrrrfH8A7J9vwixzCSLg4KbINjfa77rxefcZwugUqiYiVUWUSOb023f+QEaDm7OP+/uzz3SjF1Y0LMwa75yctttIfkRBKjY7uFCuZdKc3l2A/fdPxbFwlZmIox3LLngdTzHMVXGbVHqq9Ixo8FSAXiI1Buy2LTMoXMJwQYSvzsv7xNwujKVlUjoE9cOOIdzCC5AziwSiJB4+tNh73c5tRUzMMC98HLOhPJriSPvOt1pMEdWDX3/evcyxvRgJUKgYnmQG7n8v8O/CLyG0sAJDmGC3U9S/3fUDUjZIKWSsc85Q3xRaJwZJIS2VZwH0zG1Gmynih8PPq7VeDx7tKuNm4uTYT8= bigjust@desktop-jsho3l9.lan", "key_options": null, "keyfile": "/home/jcaratzas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcaratzas", "validate_certs": true} 2021-02-10T11:04:02.542 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCgaz5AsLB+hPwnHjlf6vktlUQ8iWsKA+hViok+j+uCm0U/iho7O7Qd8E/3tp9yBx10Wkvuz1mmf6v2AO7Qa8+SE1QQPmA+4xLp4xXCqlUGFjnzm/7uQcQZtlWdfBHzQ9jVyPAgUpnGVUtW8PhW0txe8MEtecOO2C3Snx0yuTOx6O11jNvT938QjOeRQWw3X2YFUtMIOiwEhKtqstBIGXlMdaQKfvdlbA9MjyYy6VfK1x8lVcu8umUf/z56YmMkC2iO117/obhdu1XsU9SmjM51wHMOUCXSVqH8E3BWu0yohy4QmZDkdFihyYa+hoS9dOqGC91dFaQc46xrdmV4wqg1 jcaratza@MacBook-Pro\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAV78mqEy/VkB2BthIE/m7iE5sbTjTRS0KuH7AB0KPvZBsDIw5cL8jKyxiaPC85+aOkBa4LrP0TUFBzN87Snt+PLMz2QHBOvlSTfUuRkHVo3vWJFzVuUSzzddkYds3hAxUriAHKv6CRSy1yFBxPQQHTSATUHKWqihMElbUCZrrrfH8A7J9vwixzCSLg4KbINjfa77rxefcZwugUqiYiVUWUSOb023f+QEaDm7OP+/uzz3SjF1Y0LMwa75yctttIfkRBKjY7uFCuZdKc3l2A/fdPxbFwlZmIox3LLngdTzHMVXGbVHqq9Ixo8FSAXiI1Buy2LTMoXMJwQYSvzsv7xNwujKVlUjoE9cOOIdzCC5AziwSiJB4+tNh73c5tRUzMMC98HLOhPJriSPvOt1pMEdWDX3/evcyxvRgJUKgYnmQG7n8v8O/CLyG0sAJDmGC3U9S/3fUDUjZIKWSsc85Q3xRaJwZJIS2VZwH0zG1Gmynih8PPq7VeDx7tKuNm4uTYT8= bigjust@desktop-jsho3l9.lan", "key_options": null, "keyfile": "/home/jcaratzas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcaratzas", "validate_certs": true} 2021-02-10T11:04:02.956 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} 2021-02-10T11:04:02.967 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:04:02.979 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} 2021-02-10T11:04:02.980 INFO:teuthology.task.ansible.out: 2021-02-10T11:04:02.990 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:04:03.410 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uTfMorb 2021-02-10T11:04:03.412 INFO:teuthology.task.ansible.out:nuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} 2021-02-10T11:04:03.432 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uTfMorb 2021-02-10T11:04:03.434 INFO:teuthology.task.ansible.out:nuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} 2021-02-10T11:04:03.858 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvG6+jM8aulc5CID/yE6AXPXczxq8tMgY+2sdo/ihWpgDZuFwGL4hwj8nyh5cTqZ2LEa4kRkUUIoH9avLCFN5pDBVvcj1/O06ZGnkq/hV6791UiPF5S5RGmz3MGxv0RDfjA0OLFav7u1s+ZJ6z2W1Tpy/W+g/fMYqbPc6rV+pm3Uo16+BWz/diZH8smqcty4chnnCy6gqOrcAxCZLpeyjX6Ydn3fnTLTAmhoQ9eUz7LjP4ZnFD5zL6jrSBlJ5WNTLevUzvS8eL7RuP8D3Qfn8ukIzPXG2vs594FhSc/HnvubKricvKMm0P0ki4Z0vM9VruQn7Tbeqdwpi1IfgyiAMX aemerson@work-yubi\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAv8/FJn7hz2WieoQFUGXS0bhwSwL1LCI9kkjWYNkWb8IJPrBetlkr5TsvHEKDPZGF5Sv8LsYA6njPnbOGyAHpYlnv/Gz4Hwq3hLgmXCs6JGWBQv8NS+pjwnG6Nh6JfFI0ANeg652Rk9Qt9uLYeKVKmHXXrVZAD4NMvJGfEvneQzuwwpylUmV80WAdaQw/H0EP4NScZhNhknwAs/+KPFZH+RFoUj/Cb3poIFjwyviIgyal5hjAEsoczCW3hAla/9IyC2VDS027WPiPFmBFaYK5lYYLStBQdRx+6amH76XScLFy0FEPLL5qiN0ClAptvh1oIURgipdXZwaxpf06AXG8qw== aemerson@electric", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemers 2021-02-10T11:04:03.859 INFO:teuthology.task.ansible.out:on", "validate_certs": true} 2021-02-10T11:04:03.869 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:04:03.869 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:04:03.870 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvG6+jM8aulc5CID/yE6AXPXczxq8tMgY+2sdo/ihWpgDZuFwGL4hwj8nyh5cTqZ2LEa4kRkUUIoH9avLCFN5pDBVvcj1/O06ZGnkq/hV6791UiPF5S5RGmz3MGxv0RDfjA0OLFav7u1s+ZJ6z2W1Tpy/W+g/fMYqbPc6rV+pm3Uo16+BWz/diZH8smqcty4chnnCy6gqOrcAxCZLpeyjX6Ydn3fnTLTAmhoQ9eUz7LjP4ZnFD5zL6jrSBlJ5WNTLevUzvS8eL7RuP8D3Qfn8ukIzPXG2vs594FhSc/HnvubKricvKMm0P0ki4Z0vM9VruQn7Tbeqdwpi1IfgyiAMX aemerson@work-yubi\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAv8/FJn7hz2WieoQFUGXS0bhwSwL1LCI9kkjWYNkWb8IJPrBetlkr5TsvHEKDPZGF5Sv8LsYA6njPnbOGyAHpYlnv/Gz4Hwq3hLgmXCs6JGWBQv8NS+pjwnG6Nh6JfFI0ANeg652Rk9Qt9uLYeKVKmHXXrVZAD4NMvJGfEvneQzuwwpylUmV80WAdaQw/H0EP4NScZhNhknwAs/+KPFZH+RFoUj/Cb3poIFjwyviIgyal5hjAEsoczCW3hAla/9IyC2VDS027WPiPFmBFaYK5lYYLStBQdRx+6amH76XScLFy0FEPLL5qiN0ClAptvh1oIURgipdXZwaxpf06AXG8qw== aemerson@electric", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemers 2021-02-10T11:04:03.872 INFO:teuthology.task.ansible.out:on", "validate_certs": true} 2021-02-10T11:04:03.881 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:04:03.892 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:04:04.317 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2021-02-10T11:04:04.328 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2021-02-10T11:04:04.754 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+2vtSo 2021-02-10T11:04:04.756 INFO:teuthology.task.ansible.out:AWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2021-02-10T11:04:04.776 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+2vtSo 2021-02-10T11:04:04.778 INFO:teuthology.task.ansible.out:AWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2021-02-10T11:04:05.192 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2021-02-10T11:04:05.213 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2021-02-10T11:04:05.629 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyOWnAdvzh5zR2ts9e8g0wG4wZ0uNr0YJUCncAOTJB0L8xkQU76jWzX3BRNU5Wk8VtLvpJAQ61P7UXyzgGWSO9FHZjD3RDrc2V9uh/lqRKmx1COtDvcart/JwzwHlsWdrYdcXvEaAiFrZJ6x1jJAjIL6tvMO7bqBz4YNruKpGWsQ6p5P/M5iIEPMj1oKanLoULXqReC6eYFTw3LNAXjberPY5GxYwaKS0HuzySRvSblShyA85jawTSlEbsOSnMqqGu5vGJiC415UFQQ5qKBtzrgAjlUgMbaqGTnTmBYn7L751gIWRetgWqwuHgfIe2ZGM3mmh0DX8E03DUbEt4UxyR dzafman@TrustyTahr\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDtSwGzH7V5jzppggL0+nYlBSM1HqGjkInwxutra5iEyFoBuV73MNSBk3HrvIVGN13+PCdHf4Oy8ywbhC/4yKbxS1NG76d/GU8ijZFuBlEwxwHZpcThpSTuYrNx7FJVp8Yjl45DtD5s8FMI/0YZZhLlIqT6QCTxlTIPXDb6zefEWvAVziE0bmhRpe4B6AOLOGTB9fivQr5YzxhfaSS5Vu4U1x5gmosouSJ+aMFrQWLqa/eoCy8t9PnxdMjlHjHCZ7Qv1JfpaAPw3qOK30GrVeXR/hPN3aFodTuIJpzdynjk5HO6/VKW5+mWBO7pmQqn9LNQEjADXbLh7PG//nOONbH dzafman@ubuntu16", "key_options": null, "keyfile": "/home/davidz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "davidz", "validate_certs": true} 2021-02-10T11:04:05.651 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyOWnAdvzh5zR2ts9e8g0wG4wZ0uNr0YJUCncAOTJB0L8xkQU76jWzX3BRNU5Wk8VtLvpJAQ61P7UXyzgGWSO9FHZjD3RDrc2V9uh/lqRKmx1COtDvcart/JwzwHlsWdrYdcXvEaAiFrZJ6x1jJAjIL6tvMO7bqBz4YNruKpGWsQ6p5P/M5iIEPMj1oKanLoULXqReC6eYFTw3LNAXjberPY5GxYwaKS0HuzySRvSblShyA85jawTSlEbsOSnMqqGu5vGJiC415UFQQ5qKBtzrgAjlUgMbaqGTnTmBYn7L751gIWRetgWqwuHgfIe2ZGM3mmh0DX8E03DUbEt4UxyR dzafman@TrustyTahr\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDtSwGzH7V5jzppggL0+nYlBSM1HqGjkInwxutra5iEyFoBuV73MNSBk3HrvIVGN13+PCdHf4Oy8ywbhC/4yKbxS1NG76d/GU8ijZFuBlEwxwHZpcThpSTuYrNx7FJVp8Yjl45DtD5s8FMI/0YZZhLlIqT6QCTxlTIPXDb6zefEWvAVziE0bmhRpe4B6AOLOGTB9fivQr5YzxhfaSS5Vu4U1x5gmosouSJ+aMFrQWLqa/eoCy8t9PnxdMjlHjHCZ7Qv1JfpaAPw3qOK30GrVeXR/hPN3aFodTuIJpzdynjk5HO6/VKW5+mWBO7pmQqn9LNQEjADXbLh7PG//nOONbH dzafman@ubuntu16", "key_options": null, "keyfile": "/home/davidz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "davidz", "validate_certs": true} 2021-02-10T11:04:06.078 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "key": "ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBAHT7CHaceRLB5mJv9SYKog1wBXsavzH3G3WCBEvVyK76JIslDudZm2ycJv66r+VlGaD8sFI5Mb/KKFMKcKesLRX+wHqZg9MlkoJgm7WGoRQ5I0M1Vt7TL3uJkVCaGZHtO7BXqzTx53C1AaJ1HN+DG2MGy5mm6tHwL901UVqG4EzjBnjxA== christinameno@boxtop", "key_options": null, "keyfile": "/home/cmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cmeno", "validate_certs": true} 2021-02-10T11:04:06.089 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "key": "ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBAHT7CHaceRLB5mJv9SYKog1wBXsavzH3G3WCBEvVyK76JIslDudZm2ycJv66r+VlGaD8sFI5Mb/KKFMKcKesLRX+wHqZg9MlkoJgm7WGoRQ5I0M1Vt7TL3uJkVCaGZHtO7BXqzTx53C1AaJ1HN+DG2MGy5mm6tHwL901UVqG4EzjBnjxA== christinameno@boxtop", "key_options": null, "keyfile": "/home/cmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cmeno", "validate_certs": true} 2021-02-10T11:04:06.517 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2021-02-10T11:04:06.528 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2021-02-10T11:04:06.965 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsTZI+b02boZv8jVLHQoFlloD2WF/3xT+ZfZxzwddL7Q6eqo7j8i21LAbLAMWXYMyhiClrloDeHfCZ42gG5bqSThF8LoCw+GspF6SahTI0lmptGu69/OeHqqCpW1DnsWB3wWqfi90XbQJrpg3cdRaqOCmkGVJrqfFWzff8Zg9IuojIH/aO9drCTRGa1pHHCyEdW9ZNeYFwrMjfP+wJi/mC/ABAoXyDy0NWT0uIA+36abNg2/ptok90rdan0r72hjFEz/1hpupKWri+VXAepiyDMuSgJm9uJCXDFaNTn3jocdX+sbVtltThQ2OBxe1T60vRLbXx3lvhxxZ1fUkcP/FF wusui@ubuntu\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD2UzyfBk1nZ8SLCtW9UjorMeOo4u9AM/YL8ZI74GdprR+odERZR17/QfwT5vmLVNa51+CQt1HcGodlQYfR/8n/OUl/gJyVagBuU2zgmLPO3tOor5RBfms5MZ2VZ9Vm3+WrCO/vrAq/5HVyGbQsKiQHXkIRSL2NIgfKRTOVTNmuTSXtZNDgdWGIBmcOFQv+Im9CRopoI8ZtJLQj7xfxtGtUas7FbWTcf0o92+xvWuGWq5WQgmZ8MUIfvBLYLmicuVfsWjZ1YsC1PNhuhkwrW7MhaD71Ej3ll243tG1QD7eAmo+psIp8phlU5s/5FeA5y5OzDZ2T963oHGhq018TYO0b wusui@teuthology", "key_options": null, "keyfile": "/home/wusui/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "wusui", "validate_certs": true} 2021-02-10T11:04:06.965 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsTZI+b02boZv8jVLHQoFlloD2WF/3xT+ZfZxzwddL7Q6eqo7j8i21LAbLAMWXYMyhiClrloDeHfCZ42gG5bqSThF8LoCw+GspF6SahTI0lmptGu69/OeHqqCpW1DnsWB3wWqfi90XbQJrpg3cdRaqOCmkGVJrqfFWzff8Zg9IuojIH/aO9drCTRGa1pHHCyEdW9ZNeYFwrMjfP+wJi/mC/ABAoXyDy0NWT0uIA+36abNg2/ptok90rdan0r72hjFEz/1hpupKWri+VXAepiyDMuSgJm9uJCXDFaNTn3jocdX+sbVtltThQ2OBxe1T60vRLbXx3lvhxxZ1fUkcP/FF wusui@ubuntu\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD2UzyfBk1nZ8SLCtW9UjorMeOo4u9AM/YL8ZI74GdprR+odERZR17/QfwT5vmLVNa51+CQt1HcGodlQYfR/8n/OUl/gJyVagBuU2zgmLPO3tOor5RBfms5MZ2VZ9Vm3+WrCO/vrAq/5HVyGbQsKiQHXkIRSL2NIgfKRTOVTNmuTSXtZNDgdWGIBmcOFQv+Im9CRopoI8ZtJLQj7xfxtGtUas7FbWTcf0o92+xvWuGWq5WQgmZ8MUIfvBLYLmicuVfsWjZ1YsC1PNhuhkwrW7MhaD71Ej3ll243tG1QD7eAmo+psIp8phlU5s/5FeA5y5OzDZ2T963oHGhq018TYO0b wusui@teuthology", "key_options": null, "keyfile": "/home/wusui/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "wusui", "validate_certs": true} 2021-02-10T11:04:07.407 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1Z9C 2021-02-10T11:04:07.408 INFO:teuthology.task.ansible.out:kxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, 2021-02-10T11:04:07.410 INFO:teuthology.task.ansible.out:"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1Z9CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} 2021-02-10T11:04:07.857 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkRhoBYB 2021-02-10T11:04:07.859 INFO:teuthology.task.ansible.out:0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} 2021-02-10T11:04:07.868 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkRhoBYB 2021-02-10T11:04:07.870 INFO:teuthology.task.ansible.out:0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} 2021-02-10T11:04:08.303 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2021-02-10T11:04:08.325 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2021-02-10T11:04:08.751 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2021-02-10T11:04:08.773 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2021-02-10T11:04:09.188 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7gAlIQv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwmih46T 2021-02-10T11:04:09.189 INFO:teuthology.task.ansible.out:RJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2021-02-10T11:04:09.209 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7gAlIQv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwmih46T 2021-02-10T11:04:09.211 INFO:teuthology.task.ansible.out:RJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2021-02-10T11:04:09.634 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2021-02-10T11:04:09.645 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2021-02-10T11:04:10.080 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2021-02-10T11:04:10.091 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2021-02-10T11:04:10.518 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2021-02-10T11:04:10.529 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2021-02-10T11:04:10.954 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2021-02-10T11:04:10.976 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2021-02-10T11:04:11.401 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2021-02-10T11:04:11.412 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2021-02-10T11:04:11.848 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2CfQvU 2021-02-10T11:04:11.849 INFO:teuthology.task.ansible.out:xUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} 2021-02-10T11:04:11.859 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2CfQvU 2021-02-10T11:04:11.860 INFO:teuthology.task.ansible.out:xUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} 2021-02-10T11:04:12.283 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaetIvR7YgpCq142fBQtOYsJrOMW+Wxu1beNXPJ8qzT5MxD05sdKntVE0DPveUydaeI8/vHeNjXDKN8mt1eO7Lf/rDKHOW/Y/36bUfQAODru9sdNCEJfSPgq1y6+X1ISljvLBXOOl+nQXlx3n5OjThNQbmdthgCT2Fk4hEoK7gn4+ol6foTkTdUXIBam8LuPU6f1KIG6ae1oAVFWLeavAbvmxUTmVl04+hKJWSPnvZxWupaPuiHvsImaQ7GwBjKhzL1wSMly0+pa2zbVYZKkLBA7Fx6fzP7jcEieOOPM1YU0jR5JF+WVzeRFJ1V9n75yOw6S4ef/X/zwWx42cdfx3N nwatkins@martini\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDRivdSX6cbPrdMrjz1oYgn0hb8VQo+QU7wD+AcCQi/CH06ny+2QjSr7bfdpgQ81Wgwqt9+e3KcLD5qdt3tneC8sgSAR5QUQrWKhPejrqdV+3990k8k0KoKIOZ9MGFc1FBv5IpuT/2d535iHJb0ZEg8mFKqKDw71sBEoQT4WElatf5DeVC507hP7GnrhZgtTPcrVddOLkmmqTQtLE1zvo9Y/eel8O2JMImMd6mdVBIcFQ1alU1wV4jLdMpagoPelvVTVeNkGRnOgvvQac5RXtGSMP1R5PoGz+1FgRhDCtdTXRfOdtrlJPtUjYiGPpIoKIUymlvQGjUrrK3WiQIIkIfDtFFu48EN4drjgQpCEQ2D0NMKEmk74mjx3CJDOF1Nj/2Fa1kjECiz4EM7sR8dDG5cW5Y8XsDoomgugM+rsOH9kWFFClHUJmdlNGWLhJnu9UQHLYZjQw3zT0F+FXUsUxKJUVTUdxQx5HBO9QFsSe13NPDnX64+SQ2VVStt8oQAvFO+6260zbux0RVaJM5dTNwnIHD/uI3PeR1z2b+iKhyj/9VP2uBRpGydLw/etQu/lS2K30S+kZIDAp0sqwXH89bdq3AJBipcryrnW1Ov9K5Mfl1UlV95/MivrnOFISWN1F9XR1kuIIYdCkI7hq/FHZ9gd84AjTiERfxRv8VCIHBVEQ== nwatkins@redhat.com", "key_options": null, "keyfile": "/home/nwatkins/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nwatkins", "validate_certs": true} 2021-02-10T11:04:12.294 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaetIvR7YgpCq142fBQtOYsJrOMW+Wxu1beNXPJ8qzT5MxD05sdKntVE0DPveUydaeI8/vHeNjXDKN8mt1eO7Lf/rDKHOW/Y/36bUfQAODru9sdNCEJfSPgq1y6+X1ISljvLBXOOl+nQXlx3n5OjThNQbmdthgCT2Fk4hEoK7gn4+ol6foTkTdUXIBam8LuPU6f1KIG6ae1oAVFWLeavAbvmxUTmVl04+hKJWSPnvZxWupaPuiHvsImaQ7GwBjKhzL1wSMly0+pa2zbVYZKkLBA7Fx6fzP7jcEieOOPM1YU0jR5JF+WVzeRFJ1V9n75yOw6S4ef/X/zwWx42cdfx3N nwatkins@martini\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDRivdSX6cbPrdMrjz1oYgn0hb8VQo+QU7wD+AcCQi/CH06ny+2QjSr7bfdpgQ81Wgwqt9+e3KcLD5qdt3tneC8sgSAR5QUQrWKhPejrqdV+3990k8k0KoKIOZ9MGFc1FBv5IpuT/2d535iHJb0ZEg8mFKqKDw71sBEoQT4WElatf5DeVC507hP7GnrhZgtTPcrVddOLkmmqTQtLE1zvo9Y/eel8O2JMImMd6mdVBIcFQ1alU1wV4jLdMpagoPelvVTVeNkGRnOgvvQac5RXtGSMP1R5PoGz+1FgRhDCtdTXRfOdtrlJPtUjYiGPpIoKIUymlvQGjUrrK3WiQIIkIfDtFFu48EN4drjgQpCEQ2D0NMKEmk74mjx3CJDOF1Nj/2Fa1kjECiz4EM7sR8dDG5cW5Y8XsDoomgugM+rsOH9kWFFClHUJmdlNGWLhJnu9UQHLYZjQw3zT0F+FXUsUxKJUVTUdxQx5HBO9QFsSe13NPDnX64+SQ2VVStt8oQAvFO+6260zbux0RVaJM5dTNwnIHD/uI3PeR1z2b+iKhyj/9VP2uBRpGydLw/etQu/lS2K30S+kZIDAp0sqwXH89bdq3AJBipcryrnW1Ov9K5Mfl1UlV95/MivrnOFISWN1F9XR1kuIIYdCkI7hq/FHZ9gd84AjTiERfxRv8VCIHBVEQ== nwatkins@redhat.com", "key_options": null, "keyfile": "/home/nwatkins/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nwatkins", "validate_certs": true} 2021-02-10T11:04:12.732 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2021-02-10T11:04:12.743 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2021-02-10T11:04:13.182 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2021-02-10T11:04:13.183 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2021-02-10T11:04:13.630 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2021-02-10T11:04:13.631 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2021-02-10T11:04:14.069 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA296in+8Sr9slF8hrzAV3dNz0q94Q4FlFaMYw08II5PQm6HOpabGOGOwOud/PPeh3lMUHhs45N5WnbgdfQdzz9lpDoNBcqxeACdMicdECMHtYWf2GuFUdTAnPBFVl8wFW5JcziQYJePUcAUJRXGGfdMLMppAi8VEuhMw/z3jz89UN+zf51Y9eT3rgAVSCR+Zn47kite4OB2S1aVMEcZBE0fPnh60i+rpN26s80fh3wG2N9JrPBEvst6LbL0aEixrlRUtoFHZuWNCIjntVMANa+idpuWSxN2dzfcoxpGLF1TFzqx0moznp0PnDs7ZAFFS1/EkeSCgBflyhPV6csG3VNw== jdillaman@ceph1.example.com", "key_options": null, "keyfile": "/home/jdillaman/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jdillaman", "validate_certs": true} 2021-02-10T11:04:14.253 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA296in+8Sr9slF8hrzAV3dNz0q94Q4FlFaMYw08II5PQm6HOpabGOGOwOud/PPeh3lMUHhs45N5WnbgdfQdzz9lpDoNBcqxeACdMicdECMHtYWf2GuFUdTAnPBFVl8wFW5JcziQYJePUcAUJRXGGfdMLMppAi8VEuhMw/z3jz89UN+zf51Y9eT3rgAVSCR+Zn47kite4OB2S1aVMEcZBE0fPnh60i+rpN26s80fh3wG2N9JrPBEvst6LbL0aEixrlRUtoFHZuWNCIjntVMANa+idpuWSxN2dzfcoxpGLF1TFzqx0moznp0PnDs7ZAFFS1/EkeSCgBflyhPV6csG3VNw== jdillaman@ceph1.example.com", "key_options": null, "keyfile": "/home/jdillaman/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jdillaman", "validate_certs": true} 2021-02-10T11:04:14.519 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2021-02-10T11:04:14.531 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2021-02-10T11:04:14.957 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-02-10T11:04:14.978 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-02-10T11:04:15.403 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2021-02-10T11:04:15.424 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2021-02-10T11:04:15.841 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIKepJjIovBTCkdVyvAdptljhw9bYcwtIqV/ojdbpeuJtxDo9UyrEkNoLeVpZgOcrbgrkOE7AEixo3qB9PbD8z5NRUleV7wipSh3QwrDmSylSVReJKOwYdxUKkzesZYBGIGYzAbTKqcVzPUWCeW314+xmv90XkO/+1vgbC5uoXfbfnaiDHQimBom8QV5Egr+EhhcuCqMEqaVSvjMr1T0e6IL9gUens6Mys6czI37go6QvmgEzkBbDJJ64PsiOOFHVNk4DJVJ1c7TRtsnpIHM1YVFjsH1AnAL6UufbgNSFf5dziUB/Bz815P1/jsOGpHkUKE37DbHU5xtLGl3MmimMt4ndcG6EggBGEYBwfEWPRAsZs3qngssrNH5W0CzbysLaoovzq5VqS8SdLMu5XrRLBEB0uWWzLq2bobYymk3t/oby5zrHJV78xAuLmldqjvlPZSdkgwWNv9YBLkteBVr2HhjtuashJ6amNPbA4kCGhXMpeRhfvfqs31CBEKbTV+A8ODWbWAoQxLCcqS24x0yMi73N0UzuqDTaJMcE8ODJvOxcLNaV688QDbrC/UehpX/Bp/DM7fLHj4YhkRpZxkkN5w2kb/bBbWya4DrffhHKqn9CH1z5ynj0237PhTwCXjtRn0JzpeRgN8OraFgR2H3uPc9XNqlkFxxOBK0lRngbxzw== owasserm@redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2021-02-10T11:04:15.873 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIKepJjIovBTCkdVyvAdptljhw9bYcwtIqV/ojdbpeuJtxDo9UyrEkNoLeVpZgOcrbgrkOE7AEixo3qB9PbD8z5NRUleV7wipSh3QwrDmSylSVReJKOwYdxUKkzesZYBGIGYzAbTKqcVzPUWCeW314+xmv90XkO/+1vgbC5uoXfbfnaiDHQimBom8QV5Egr+EhhcuCqMEqaVSvjMr1T0e6IL9gUens6Mys6czI37go6QvmgEzkBbDJJ64PsiOOFHVNk4DJVJ1c7TRtsnpIHM1YVFjsH1AnAL6UufbgNSFf5dziUB/Bz815P1/jsOGpHkUKE37DbHU5xtLGl3MmimMt4ndcG6EggBGEYBwfEWPRAsZs3qngssrNH5W0CzbysLaoovzq5VqS8SdLMu5XrRLBEB0uWWzLq2bobYymk3t/oby5zrHJV78xAuLmldqjvlPZSdkgwWNv9YBLkteBVr2HhjtuashJ6amNPbA4kCGhXMpeRhfvfqs31CBEKbTV+A8ODWbWAoQxLCcqS24x0yMi73N0UzuqDTaJMcE8ODJvOxcLNaV688QDbrC/UehpX/Bp/DM7fLHj4YhkRpZxkkN5w2kb/bBbWya4DrffhHKqn9CH1z5ynj0237PhTwCXjtRn0JzpeRgN8OraFgR2H3uPc9XNqlkFxxOBK0lRngbxzw== owasserm@redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2021-02-10T11:04:16.288 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6AKY1O 2021-02-10T11:04:16.290 INFO:teuthology.task.ansible.out:MJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2021-02-10T11:04:16.310 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6AKY1O 2021-02-10T11:04:16.311 INFO:teuthology.task.ansible.out:MJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2021-02-10T11:04:16.725 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2RuqiZyKz 2021-02-10T11:04:16.726 INFO:teuthology.task.ansible.out:2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2021-02-10T11:04:16.746 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2RuqiZyKz 2021-02-10T11:04:16.748 INFO:teuthology.task.ansible.out:2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2021-02-10T11:04:17.164 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2021-02-10T11:04:17.207 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2021-02-10T11:04:17.603 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2021-02-10T11:04:17.645 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2021-02-10T11:04:18.050 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2021-02-10T11:04:18.083 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2021-02-10T11:04:18.489 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": nul 2021-02-10T11:04:18.491 INFO:teuthology.task.ansible.out:l, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-02-10T11:04:18.532 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": nul 2021-02-10T11:04:18.534 INFO:teuthology.task.ansible.out:l, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-02-10T11:04:18.938 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+zmfjEf 2021-02-10T11:04:18.939 INFO:teuthology.task.ansible.out:dVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbBmS+dT 2021-02-10T11:04:18.940 INFO:teuthology.task.ansible.out:zNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2021-02-10T11:04:18.980 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+zmfjEf 2021-02-10T11:04:18.981 INFO:teuthology.task.ansible.out:dVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbBmS+dT 2021-02-10T11:04:18.983 INFO:teuthology.task.ansible.out:zNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2021-02-10T11:04:19.398 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2021-02-10T11:04:19.466 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2021-02-10T11:04:19.843 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2021-02-10T11:04:19.922 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2021-02-10T11:04:20.311 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2021-02-10T11:04:20.385 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2021-02-10T11:04:20.771 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2021-02-10T11:04:20.854 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2021-02-10T11:04:21.218 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2021-02-10T11:04:21.313 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2021-02-10T11:04:21.658 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6NWn9EzQJoeXkUbaNclYcR9Gga7HFhhXMGxdo/9ypYwlEzQTA6FnEreG3s1EmlQ4hofuc0kMhrjh0YeJZMrETWMoA0SmoX+6iR26UWVtOTk7i1v2TqkTY7llmQneHqQgmZEBXK0jF1HVP5kHY5dhagtcpfsGY5dc0eGUjfqAyh/xrCxciOE/JQ03tDcKIc5eOQIqv46JqnUS90Zr/Cov2qGKKHTa/VO1a2viOmDC/uHi5xXvII1Kh9rHp1oZ2Wm3fiGgT8xZaTJFFhWpdq3AdKfkEaV93yQcCdaUsGBHwDHcPOhxpBZpqZvcCR3I/tigDav4QViB1PRN7JnTIWCIznfrQK3r2Frl6BVFDvydwgRVS43ctEKPOJj8ELMUWX+fpjsKrA9Kn7nUU35ti7qIWKg1Cnz3lzVA29TsEg10E+nT4SVKZtRSWw/2AX0bsfxfSY30yjw5PP+bjnVh/bOOkqR2UkPBj4eeOKMD8nV1an79xP0eH+t36INaJhq6qwxE= nojha@nojha-mac", "key_options": null, "keyfile": "/home/ 2021-02-10T11:04:21.659 INFO:teuthology.task.ansible.out:nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} 2021-02-10T11:04:21.752 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6NWn9EzQJoeXkUbaNclYcR9Gga7HFhhXMGxdo/9ypYwlEzQTA6FnEreG3s1EmlQ4hofuc0kMhrjh0YeJZMrETWMoA0SmoX+6iR26UWVtOTk7i1v2TqkTY7llmQneHqQgmZEBXK0jF1HVP5kHY5dhagtcpfsGY5dc0eGUjfqAyh/xrCxciOE/JQ03tDcKIc5eOQIqv46JqnUS90Zr/Cov2qGKKHTa/VO1a2viOmDC/uHi5xXvII1Kh9rHp1oZ2Wm3fiGgT8xZaTJFFhWpdq3AdKfkEaV93yQcCdaUsGBHwDHcPOhxpBZpqZvcCR3I/tigDav4QViB1PRN7JnTIWCIznfrQK3r2Frl6BVFDvydwgRVS43ctEKPOJj8ELMUWX+fpjsKrA9Kn7nUU35ti7qIWKg1Cnz3lzVA29TsEg10E+nT4SVKZtRSWw/2AX0bsfxfSY30yjw5PP+bjnVh/bOOkqR2UkPBj4eeOKMD8nV1an79xP0eH+t36INaJhq6qwxE= nojha@nojha-mac", "key_options": null, "keyfile": "/home/ 2021-02-10T11:04:21.753 INFO:teuthology.task.ansible.out:nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} 2021-02-10T11:04:22.096 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9 ifed@37526", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2021-02-10T11:04:22.191 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9 ifed@37526", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2021-02-10T11:04:22.545 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCnOq6OHTRJuv7n0GZygBue/AE2q47pFM3KyrvmZxPRNmwtXn2r3ngHQQyc6DLZdFIv9Yn0VWu1KB/ryCdN3Xy7HAxBfHdKjtQ4vvfsWUbQF2JjhEjrsl+usgpqK1Tz6yuL4uAtFgLTN9o6gcH2bsQ15w+rEHCiYVzaGp7Cj+y+/VPBRIEHETSUZxwqiDH57uFTHxJrBIn6yhCorbn5QOjfmHUGmkxl3Geph++LbuQeha5F/DHaylMulsYZZBDEhIG8G2coDmVz8Cp1gw0kBWv63JHKnXaJboGXty2Zj36O/wdZKTE+7cV6JhxfJczhZBTb2cImIhspGwdHfnFgBoXz myoungwon@fg2server", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2021-02-10T11:04:22.650 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCnOq6OHTRJuv7n0GZygBue/AE2q47pFM3KyrvmZxPRNmwtXn2r3ngHQQyc6DLZdFIv9Yn0VWu1KB/ryCdN3Xy7HAxBfHdKjtQ4vvfsWUbQF2JjhEjrsl+usgpqK1Tz6yuL4uAtFgLTN9o6gcH2bsQ15w+rEHCiYVzaGp7Cj+y+/VPBRIEHETSUZxwqiDH57uFTHxJrBIn6yhCorbn5QOjfmHUGmkxl3Geph++LbuQeha5F/DHaylMulsYZZBDEhIG8G2coDmVz8Cp1gw0kBWv63JHKnXaJboGXty2Zj36O/wdZKTE+7cV6JhxfJczhZBTb2cImIhspGwdHfnFgBoXz myoungwon@fg2server", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2021-02-10T11:04:22.986 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "g 2021-02-10T11:04:22.987 INFO:teuthology.task.ansible.out:abrioux", "validate_certs": true} 2021-02-10T11:04:23.092 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "g 2021-02-10T11:04:23.093 INFO:teuthology.task.ansible.out:abrioux", "validate_certs": true} 2021-02-10T11:04:23.426 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2021-02-10T11:04:23.541 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2021-02-10T11:04:23.867 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2021-02-10T11:04:23.984 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2021-02-10T11:04:24.310 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS33Kkk 2021-02-10T11:04:24.311 INFO:teuthology.task.ansible.out:e6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} 2021-02-10T11:04:24.436 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS33Kkk 2021-02-10T11:04:24.438 INFO:teuthology.task.ansible.out:e6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} 2021-02-10T11:04:24.750 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2021-02-10T11:04:24.892 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2021-02-10T11:04:25.193 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2021-02-10T11:04:25.339 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2021-02-10T11:04:25.631 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2021-02-10T11:04:25.789 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2021-02-10T11:04:26.070 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-02-10T11:04:26.239 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-02-10T11:04:26.512 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2021-02-10T11:04:26.681 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2021-02-10T11:04:26.953 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} 2021-02-10T11:04:27.144 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} 2021-02-10T11:04:27.396 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} 2021-02-10T11:04:27.586 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} 2021-02-10T11:04:27.837 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2021-02-10T11:04:28.026 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2021-02-10T11:04:28.292 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} 2021-02-10T11:04:28.471 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} 2021-02-10T11:04:28.736 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2021-02-10T11:04:28.908 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2021-02-10T11:04:29.172 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2021-02-10T11:04:29.361 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2021-02-10T11:04:29.615 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2021-02-10T11:04:29.806 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2021-02-10T11:04:30.069 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2021-02-10T11:04:30.249 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2021-02-10T11:04:30.512 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2021-02-10T11:04:30.692 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2021-02-10T11:04:30.945 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDklgWHrL9TMaMxIyN9MxVR6nR3xh9zfj0iR63/yOzaslX0WoVBk7CYTsRlH2GFVctvixVwrWbQgMaKe49utal/3FA6sMl8S2DTrtgC1a/exUTvU0U/+ZhyUlkyjQfRgtYOUwodNupL5NsC8VZqZshgmgXC34uye3bplx3VwvpUt9r6QxJN3/9Dbi6HxudhDCLfmvROhwQYHaFF+GglcbyjAt3uvsuwXVKBd1/vMQwbcUo9/zQlrnmWBRuqTNkXGrZauWigoi/hyuWYQ3MUbRc3LVpJncMv7BEtcj8fCVvPB09SFee3nSeTtFkm01GvRcYd3HBz/YL4Rz92wXl9VIxqs8aaJBzy+MD0NO496Rfqtz6YuZ1m7AxDIccDW+OpcPgYwaUKFXNTuVLWZXhkS0p4swefcKWRlQFF3BLOebN/Kr9jq1MFa+5pX7Gj4WF5uCIX0/fx40KVZNhfl3qywBy/d5MKwXO7PCHMiGcu2FIXdZ334MxdgiIVoX54PpRP4hs= smanjara@localhost.localdomain", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2021-02-10T11:04:31.147 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDklgWHrL9TMaMxIyN9MxVR6nR3xh9zfj0iR63/yOzaslX0WoVBk7CYTsRlH2GFVctvixVwrWbQgMaKe49utal/3FA6sMl8S2DTrtgC1a/exUTvU0U/+ZhyUlkyjQfRgtYOUwodNupL5NsC8VZqZshgmgXC34uye3bplx3VwvpUt9r6QxJN3/9Dbi6HxudhDCLfmvROhwQYHaFF+GglcbyjAt3uvsuwXVKBd1/vMQwbcUo9/zQlrnmWBRuqTNkXGrZauWigoi/hyuWYQ3MUbRc3LVpJncMv7BEtcj8fCVvPB09SFee3nSeTtFkm01GvRcYd3HBz/YL4Rz92wXl9VIxqs8aaJBzy+MD0NO496Rfqtz6YuZ1m7AxDIccDW+OpcPgYwaUKFXNTuVLWZXhkS0p4swefcKWRlQFF3BLOebN/Kr9jq1MFa+5pX7Gj4WF5uCIX0/fx40KVZNhfl3qywBy/d5MKwXO7PCHMiGcu2FIXdZ334MxdgiIVoX54PpRP4hs= smanjara@localhost.localdomain", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2021-02-10T11:04:31.390 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2021-02-10T11:04:31.590 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2021-02-10T11:04:31.832 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2021-02-10T11:04:32.033 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2021-02-10T11:04:32.284 INFO:teuthology.task.ansible.out:changed: [gibba012.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDQu/CuhCIFVGCLbA0cjn+qqcwX12cRcFA1sHyJtkj6Ltfmge8FnfgAsex/LaPJwbtbBiVl9xRcazHNfBaDw5DxMRSJF+e7WTydZC2Jm4qIj4D3Lr+imGPrJTK+kjN0QXRXknGsNn36oRF/I41cGqYRGwawFGSPZBHuav+ehDk1doTrbDlll4zMUWm2iXTcBIcJo4NR1LoE2c9/MPfbH7jKm5ke7aS0ets+EANEplkeClfCVAashtIF+Bz6a9qk4F2n0baLT/s3IQkXCwwO2fxxY03x8gWiB1n4E7sousEB+J1Skl/z0bwmaiY0C+raPoV/FyZxQuh0ocvdU7fnQbKuNoGE/Usfm9Y3g8eH81b/k4fMJL2lqXHkPft6zUV9bJsp3v1iGEkIf2+ER/QLczteNz6kw/uTPQctMEDXM17GEnBXISVrQcPSW3i93eFwWacs8iT3G4bjEQvpxSBQrhDdCCoxy8bVWqkVeDWw5u2q/3WL9Ov1NrY+G4/DYb2qONNQds0vTHCGvemU5GNjtaeCbojkZ7Z3KOeAY8080kKrsO7fve+ZMACCIFD+se8kF7NN4ZJASSqq9lJpBWvtbc0O8jCcVGNuN7exXjrcbiN1L/9HkARV+KEJU3jbXObAhAnfAaeRWBwmTfZodCu+fDd/Jor6l36tjD4kC4/CPrDtyQ== alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2021-02-10T11:04:32.473 INFO:teuthology.task.ansible.out:changed: [gibba020.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDQu/CuhCIFVGCLbA0cjn+qqcwX12cRcFA1sHyJtkj6Ltfmge8FnfgAsex/LaPJwbtbBiVl9xRcazHNfBaDw5DxMRSJF+e7WTydZC2Jm4qIj4D3Lr+imGPrJTK+kjN0QXRXknGsNn36oRF/I41cGqYRGwawFGSPZBHuav+ehDk1doTrbDlll4zMUWm2iXTcBIcJo4NR1LoE2c9/MPfbH7jKm5ke7aS0ets+EANEplkeClfCVAashtIF+Bz6a9qk4F2n0baLT/s3IQkXCwwO2fxxY03x8gWiB1n4E7sousEB+J1Skl/z0bwmaiY0C+raPoV/FyZxQuh0ocvdU7fnQbKuNoGE/Usfm9Y3g8eH81b/k4fMJL2lqXHkPft6zUV9bJsp3v1iGEkIf2+ER/QLczteNz6kw/uTPQctMEDXM17GEnBXISVrQcPSW3i93eFwWacs8iT3G4bjEQvpxSBQrhDdCCoxy8bVWqkVeDWw5u2q/3WL9Ov1NrY+G4/DYb2qONNQds0vTHCGvemU5GNjtaeCbojkZ7Z3KOeAY8080kKrsO7fve+ZMACCIFD+se8kF7NN4ZJASSqq9lJpBWvtbc0O8jCcVGNuN7exXjrcbiN1L/9HkARV+KEJU3jbXObAhAnfAaeRWBwmTfZodCu+fDd/Jor6l36tjD4kC4/CPrDtyQ== alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2021-02-10T11:04:32.724 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2021-02-10T11:04:32.924 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2021-02-10T11:04:33.164 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2021-02-10T11:04:33.363 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2021-02-10T11:04:33.613 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDp0xz0ml2XuoQe70TxVeTo3Ib5xMinDrYNJ5eq5XYNa8ZLGcEByeDK4myzixTloAo31Rb/yDyGqv6kQ6nFChihO2aab3Ni/bsMOaJat1scrznaIYXGvLvw5KYx/qBDUxb3qW0vO0N/bB3dUPS1l 2021-02-10T11:04:33.614 INFO:teuthology.task.ansible.out:XTJaVf+KuljUk7v38ov2gsPWlofRYZNkPGQfUKXFEXc39OVj+Ex5LkehH/Z1LwjRYJa7jAdSm8hE5m8m6WborrzAA4/iFuLbhtM/Bqm4boSqa7koBVux422QX/HblSQ2KAeT1GFOS7xdcO6ndyp6mIpCZWunTjZwSLyK20NvIV/AzdAro4GZWi0D34PPtbSG344ZT1dSLeDRhYVkDjS6uutrmw9C+Ns1bo8s1ydhxL2XEsU14SWRWlbyrMF9PXyJOsaNiBNWGwc9lTly0ZlmPnN3l0AC3RIKOstkEYmI+TMj21r1tltz48e+6dH1an51wwnhNRYiODd3hbS67tmIg05BuSU5+whJ93wzXw2rC4FLE= ylifshit@localhost.localdomain", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} 2021-02-10T11:04:33.811 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDp0xz0ml2XuoQe70TxVeTo3Ib5xMinDrYNJ5eq5XYNa8ZLGcEByeDK4myzixTloAo31Rb/yDyGqv6kQ6nFChihO2aab3Ni/bsMOaJat1scrznaIYXGvLvw5KYx/qBDUxb3qW0vO0N/bB3dUPS1l 2021-02-10T11:04:33.813 INFO:teuthology.task.ansible.out:XTJaVf+KuljUk7v38ov2gsPWlofRYZNkPGQfUKXFEXc39OVj+Ex5LkehH/Z1LwjRYJa7jAdSm8hE5m8m6WborrzAA4/iFuLbhtM/Bqm4boSqa7koBVux422QX/HblSQ2KAeT1GFOS7xdcO6ndyp6mIpCZWunTjZwSLyK20NvIV/AzdAro4GZWi0D34PPtbSG344ZT1dSLeDRhYVkDjS6uutrmw9C+Ns1bo8s1ydhxL2XEsU14SWRWlbyrMF9PXyJOsaNiBNWGwc9lTly0ZlmPnN3l0AC3RIKOstkEYmI+TMj21r1tltz48e+6dH1an51wwnhNRYiODd3hbS67tmIg05BuSU5+whJ93wzXw2rC4FLE= ylifshit@localhost.localdomain", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} 2021-02-10T11:04:34.052 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2021-02-10T11:04:34.260 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2021-02-10T11:04:34.491 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2021-02-10T11:04:34.699 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2021-02-10T11:04:34.935 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2021-02-10T11:04:35.147 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2021-02-10T11:04:35.376 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC1OVCo0wMITAp8PNUqsTRPFz0SuBL0r1RzKFJUoxQm7gsP9qttZUQRlfiuZ7YRzkuGLOEBngPuIcV995o0lZ0+LVHA8jxWEJamb6sM9YgnonbGKDHfl4lT8LJd4YtjcWnMGvUUdPiPgIkfrpxXtx+RzJ7eCTmgRjAJziZrRh3AqXuKty+s9MDnmK8EQ2smpLdkMtZG5W3U2PiVaGgq2FIIOAoqr58xGqFgZ4tIpvWFpNxVsDZXIg2NdC+ouuKQvuJRWq9eERRHwq26CR+kDFNZeYSyhYKxNBqauLCsAoXNFCJzrdX5Oi9Ox4lJTsEtYX+SMziVipQS54F7MnRAUccukQ86IbXz8C5Ok2EzNo6smwRZ6UPKRvebWjIvfbeR/iwPRnrWcNgjGcaTR6yhlRJh8pBkW84nyY/bJa4WXsoNLgutdsYkjl18KKhbBldjdf0ovfL7FVgqYfAo/wpQYJJeMebP5QAczxvZIegjHRrAvbs/nbQTmuAkes3WL1hhBjSTwyIbWCzAuBJk8MNTmeZkVBvub7XoIyIySppfoJC/1a+ZsxGkzAlgrIxXPYrrBzTtXd/9YRiIgCYx3UtfdbPkE6hT6GblH6ST6QESfWyyvfGxwlOmu+F+icRyVVJbgM9T/PD2W6I04u2/1mcfoHwaQGdwvbbBkbjWzm+zwNSisQ== sanupkri@redhat.com", "key_options": null, "keyfile": "/home/sidharthanup/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sidharthanup", "validate_certs": true} 2021-02-10T11:04:35.596 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC1OVCo0wMITAp8PNUqsTRPFz0SuBL0r1RzKFJUoxQm7gsP9qttZUQRlfiuZ7YRzkuGLOEBngPuIcV995o0lZ0+LVHA8jxWEJamb6sM9YgnonbGKDHfl4lT8LJd4YtjcWnMGvUUdPiPgIkfrpxXtx+RzJ7eCTmgRjAJziZrRh3AqXuKty+s9MDnmK8EQ2smpLdkMtZG5W3U2PiVaGgq2FIIOAoqr58xGqFgZ4tIpvWFpNxVsDZXIg2NdC+ouuKQvuJRWq9eERRHwq26CR+kDFNZeYSyhYKxNBqauLCsAoXNFCJzrdX5Oi9Ox4lJTsEtYX+SMziVipQS54F7MnRAUccukQ86IbXz8C5Ok2EzNo6smwRZ6UPKRvebWjIvfbeR/iwPRnrWcNgjGcaTR6yhlRJh8pBkW84nyY/bJa4WXsoNLgutdsYkjl18KKhbBldjdf0ovfL7FVgqYfAo/wpQYJJeMebP5QAczxvZIegjHRrAvbs/nbQTmuAkes3WL1hhBjSTwyIbWCzAuBJk8MNTmeZkVBvub7XoIyIySppfoJC/1a+ZsxGkzAlgrIxXPYrrBzTtXd/9YRiIgCYx3UtfdbPkE6hT6GblH6ST6QESfWyyvfGxwlOmu+F+icRyVVJbgM9T/PD2W6I04u2/1mcfoHwaQGdwvbbBkbjWzm+zwNSisQ== sanupkri@redhat.com", "key_options": null, "keyfile": "/home/sidharthanup/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sidharthanup", "validate_certs": true} 2021-02-10T11:04:35.825 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkeYXWjU7wVdODC32U40QRFgFH7ZEuEsOeWWaKx3Y5EBuAcwkTHiF9Bq/2mrDu+c9W46S8L38Ogc9GZ0DcCXtN/Z4z8iKOIIi5U1GzvwdbKuiAPCd8bdYP0vo0MOIRgylJo5hcnB5BCwCXRN1hmci1ES5UbSeQg0rydMiyYXHZ64AT47+qVUpn/mVP6PVw00pPqt16l9YxhbPFG74Y/WnY4FyrqXGQJfv0tcP7+UR7GQdFiI4QQV3ZUSRHRLWGcmoiiOmGimLhtzEmP4Jzy4gF4wg9q8jyX54NjfxDRZs+YuaoqqWKz/HSz5ic7W2RGKHrRby2N9wNu19nmC1NzIn3 varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCW58OUR3pniaYgvr9xAExeNm4eMxgaJfFbEfEYPztTWv/xxqqihBYCeFltfgWbu7ERTy+hVNo1PJWyyczjl5Ivc3FdRmhykosygs7X74PPh02c7Em4MsSTr7SSPNN20rkJbWqiYMMOBbqbiZ7JXiSUNVilLHgZqZBEQc5Rf5I1PyBLHHJ2ZruPKnDPHCRLdwJ+MdXy2haKmGrQLhtzIKaDzh8VkfojL454k0aHCIh34Z0oBdP9757mNuYJUVEkKZF5t2oML84lWGMgwZxUWN2yO9ALTti4tNH1fXA6NipS9ztE+3hEJe1fWXEoUQDX9JNu4qEHTeCz0secQMkyZVDZSQk8yDUQxQ76yiett37sQB1WQR/3fclcahmqQ4dUS7M3Onyz6k6zia896PBK6G48TBUbFa1rw5Hx7WfNZluDMkv4YxkQvfz7XVzZx2/lhs45Kjh1UR9n7LaW/NID98vokcvASw1N6Bk/6pOd/45iPyEJemwovnQLs6ecwlMIikM= varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcQ8xMjwVf+KiFIQARcbFT+wg4NHysBRrFxk3Bx5X2k2hGKcEYR2tY/ApHZYfJ6me3LUvJEXg2c/6xa6ibVTUp/90l4y0cF6RmF5uB45J0xuBYoFJd8xJkVrhDIlZix4aFfb7TrmJEqjleBrkTwam7Q9aE3lss16nVOxU4hiViYiCANJ45wobK6tz7y5btfzrHE/iYpwniDvzLwK7EpkEK336VVd6f9wV0vG+ErEFxnNVxheXLLm05LuvdVQvFqJcTFrc7/GZ54yKg5IgwVwXqSAyOXIhgduUqk+oN1ciA7jnSqdiOKQPS/VmYkbhw2YP85l8wGVreIYUmOG19QEyVLB82oB9KS1LezM2KCH61XW22+85dHf4PvuL9DULhGt6wBCvpO10OHxw0JvHXbEgBPhiyd9j2yJmcQozbRPth/sMUvLKNXOGabZhhkPBYL7xZICHc+UwxrXs0h/j9jOjLyHzBAWMomF2lzgbvtzjcbxJ1a5oclJ6IW8yHJllkOh0= varsha@localhost.localdo 2021-02-10T11:04:35.827 INFO:teuthology.task.ansible.out:main", "key_options": null, "keyfile": "/home/varsha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "varsha", "validate_certs": true} 2021-02-10T11:04:36.043 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkeYXWjU7wVdODC32U40QRFgFH7ZEuEsOeWWaKx3Y5EBuAcwkTHiF9Bq/2mrDu+c9W46S8L38Ogc9GZ0DcCXtN/Z4z8iKOIIi5U1GzvwdbKuiAPCd8bdYP0vo0MOIRgylJo5hcnB5BCwCXRN1hmci1ES5UbSeQg0rydMiyYXHZ64AT47+qVUpn/mVP6PVw00pPqt16l9YxhbPFG74Y/WnY4FyrqXGQJfv0tcP7+UR7GQdFiI4QQV3ZUSRHRLWGcmoiiOmGimLhtzEmP4Jzy4gF4wg9q8jyX54NjfxDRZs+YuaoqqWKz/HSz5ic7W2RGKHrRby2N9wNu19nmC1NzIn3 varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCW58OUR3pniaYgvr9xAExeNm4eMxgaJfFbEfEYPztTWv/xxqqihBYCeFltfgWbu7ERTy+hVNo1PJWyyczjl5Ivc3FdRmhykosygs7X74PPh02c7Em4MsSTr7SSPNN20rkJbWqiYMMOBbqbiZ7JXiSUNVilLHgZqZBEQc5Rf5I1PyBLHHJ2ZruPKnDPHCRLdwJ+MdXy2haKmGrQLhtzIKaDzh8VkfojL454k0aHCIh34Z0oBdP9757mNuYJUVEkKZF5t2oML84lWGMgwZxUWN2yO9ALTti4tNH1fXA6NipS9ztE+3hEJe1fWXEoUQDX9JNu4qEHTeCz0secQMkyZVDZSQk8yDUQxQ76yiett37sQB1WQR/3fclcahmqQ4dUS7M3Onyz6k6zia896PBK6G48TBUbFa1rw5Hx7WfNZluDMkv4YxkQvfz7XVzZx2/lhs45Kjh1UR9n7LaW/NID98vokcvASw1N6Bk/6pOd/45iPyEJemwovnQLs6ecwlMIikM= varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcQ8xMjwVf+KiFIQARcbFT+wg4NHysBRrFxk3Bx5X2k2hGKcEYR2tY/ApHZYfJ6me3LUvJEXg2c/6xa6ibVTUp/90l4y0cF6RmF5uB45J0xuBYoFJd8xJkVrhDIlZix4aFfb7TrmJEqjleBrkTwam7Q9aE3lss16nVOxU4hiViYiCANJ45wobK6tz7y5btfzrHE/iYpwniDvzLwK7EpkEK336VVd6f9wV0vG+ErEFxnNVxheXLLm05LuvdVQvFqJcTFrc7/GZ54yKg5IgwVwXqSAyOXIhgduUqk+oN1ciA7jnSqdiOKQPS/VmYkbhw2YP85l8wGVreIYUmOG19QEyVLB82oB9KS1LezM2KCH61XW22+85dHf4PvuL9DULhGt6wBCvpO10OHxw0JvHXbEgBPhiyd9j2yJmcQozbRPth/sMUvLKNXOGabZhhkPBYL7xZICHc+UwxrXs0h/j9jOjLyHzBAWMomF2lzgbvtzjcbxJ1a5oclJ6IW8yHJllkOh0= varsha@localhost.localdo 2021-02-10T11:04:36.045 INFO:teuthology.task.ansible.out:main", "key_options": null, "keyfile": "/home/varsha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "varsha", "validate_certs": true} 2021-02-10T11:04:36.273 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfbsmXq2 2021-02-10T11:04:36.274 INFO:teuthology.task.ansible.out:z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} 2021-02-10T11:04:36.493 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfbsmXq2 2021-02-10T11:04:36.494 INFO:teuthology.task.ansible.out:z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} 2021-02-10T11:04:36.711 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDj6DxTZNodVnwJZrnSZ/jv9Cw4uYLb8MGpccmOvtwl93sVBvqq597AYnBqYiV4VBbKU3GVFbT41zkplia1iCfXoddttCOyWmEKviBXRnAsd5f80dxL0pfzrBJvCaH4NAfYgVw9JFWsvT981b+nB6vum4Vz8Dw0PM5GPnVUjsZNFkpmDPUBL4yHPmMCGP2WtUurJVD3Gm9t72XlG6Dl/vB9Qw/i4IY1SoCCbuhhNgE7IN8JXeGmVJbRwDeUlP+Q8Ti7iedV7YB/W1opt6eT5EPzX8NcxTc0c7rrDgsUWjFQYRSup0GLK9ofdHs0E2RbPPO5OSc5Z2+B1IEV7fZeyJwAyHwUgwx7sSGy7aROdvinaj/hOBglcgISyQ2CY7LTV8NGWJquo7yRCJXhZVIHfgqfBcrgWdNbr+vNBXnWe5PCtLhV8ryHPSJ/7Ru+7voQxgvwux5SdZwxXTbWDKni1f6/8bNrhRx6bmLK9xuYAgt2FhlIivI7L5kQF4etcBiaf68= deepika@x1cabon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdqbYcbJYtgCrAbLo7PcS82caXxcT8cpX0RkULaFXfaiUDs8Foy6Yf/XoEHIdpgw8kX6qBdydYXxti6HFhYDNsL5KleKu/yA+bVKlvri39CRNkNxFHAUL2CRBT5Hefj1AJQBm/BKVEN4/eb05OlJJ808Qb/aNycot/YwgpNvUIaO2yTMA0meq82wNdZmUcmyLB9QXz4QInTcoAsxq70KxX95YC938rTWpbg1hM8gzLkJJt/OD/Z7TMYv/Sc5KGOFqo930AmObQ5KPmffkWGYbLT65TxlCULnvIr63lWHdGTj5mYjib/v9ncu24az9GYRWYcicV0B/def+KVrrIp3Mz ideepika@teuthology", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2021-02-10T11:04:36.712 INFO:teuthology.task.ansible.out: 2021-02-10T11:04:36.925 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDj6DxTZNodVnwJZrnSZ/jv9Cw4uYLb8MGpccmOvtwl93sVBvqq597AYnBqYiV4VBbKU3GVFbT41zkplia1iCfXoddttCOyWmEKviBXRnAsd5f80dxL0pfzrBJvCaH4NAfYgVw9JFWsvT981b+nB6vum4Vz8Dw0PM5GPnVUjsZNFkpmDPUBL4yHPmMCGP2WtUurJVD3Gm9t72XlG6Dl/vB9Qw/i4IY1SoCCbuhhNgE7IN8JXeGmVJbRwDeUlP+Q8Ti7iedV7YB/W1opt6eT5EPzX8NcxTc0c7rrDgsUWjFQYRSup0GLK9ofdHs0E2RbPPO5OSc5Z2+B1IEV7fZeyJwAyHwUgwx7sSGy7aROdvinaj/hOBglcgISyQ2CY7LTV8NGWJquo7yRCJXhZVIHfgqfBcrgWdNbr+vNBXnWe5PCtLhV8ryHPSJ/7Ru+7voQxgvwux5SdZwxXTbWDKni1f6/8bNrhRx6bmLK9xuYAgt2FhlIivI7L5kQF4etcBiaf68= deepika@x1cabon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdqbYcbJYtgCrAbLo7PcS82caXxcT8cpX0RkULaFXfaiUDs8Foy6Yf/XoEHIdpgw8kX6qBdydYXxti6HFhYDNsL5KleKu/yA+bVKlvri39CRNkNxFHAUL2CRBT5Hefj1AJQBm/BKVEN4/eb05OlJJ808Qb/aNycot/YwgpNvUIaO2yTMA0meq82wNdZmUcmyLB9QXz4QInTcoAsxq70KxX95YC938rTWpbg1hM8gzLkJJt/OD/Z7TMYv/Sc5KGOFqo930AmObQ5KPmffkWGYbLT65TxlCULnvIr63lWHdGTj5mYjib/v9ncu24az9GYRWYcicV0B/def+KVrrIp3Mz ideepika@teuthology", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2021-02-10T11:04:37.156 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2021-02-10T11:04:37.375 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2021-02-10T11:04:37.635 INFO:teuthology.task.ansible.out:changed: [gibba012.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrGYBGPDQqn+2jnxKhDurSrAJvTT8dZMYGVCt6NbjpFhDFYtyvrF8Jv5NKBQynG18LwscN7a57L0Oq0e4jOEPRz//CDcAfP+L6p45L4BdUzTysN4+rQ7waMnIiYp7VjSswhGH/BnYIVKwmDqzM2fo1w/oLIT6mgwUYqAdceeswAkXBt7jgagL8ox3E4oHBqER7f60AHdTviuW953t5luVQiMjyF2fOwZuo2ycufIzBee0CwS+xyRuJQwdgD3HL4Qs2vtBCX+H8n3wC6A9JrMA5tzCDr5VZKBJZURIniwvaXfJodjO4SvBL5jAfXoMqriv8ZWHbap2fk82YE8/uPqNvSISjixJEBgQinklZ+iAOYm5tfPbhPM/8ErpN/yJ0EoGRn+nj6X6nAjXAIk3AENopeqtM7mLZ26RFIuEV5ld7eqUvAfiy6Hk2LH9Q1Z4CXaxD0gDGzUxgdAaaAgLAR4btWi2n7Tb7ZNDdOfctg2ya9ftzZBOHvCaVXSpJCry/GMU= skoduri@localhost.localdomain", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2021-02-10T11:04:37.812 INFO:teuthology.task.ansible.out:changed: [gibba020.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrGYBGPDQqn+2jnxKhDurSrAJvTT8dZMYGVCt6NbjpFhDFYtyvrF8Jv5NKBQynG18LwscN7a57L0Oq0e4jOEPRz//CDcAfP+L6p45L4BdUzTysN4+rQ7waMnIiYp7VjSswhGH/BnYIVKwmDqzM2fo1w/oLIT6mgwUYqAdceeswAkXBt7jgagL8ox3E4oHBqER7f60AHdTviuW953t5luVQiMjyF2fOwZuo2ycufIzBee0CwS+xyRuJQwdgD3HL4Qs2vtBCX+H8n3wC6A9JrMA5tzCDr5VZKBJZURIniwvaXfJodjO4SvBL5jAfXoMqriv8ZWHbap2fk82YE8/uPqNvSISjixJEBgQinklZ+iAOYm5tfPbhPM/8ErpN/yJ0EoGRn+nj6X6nAjXAIk3AENopeqtM7mLZ26RFIuEV5ld7eqUvAfiy6Hk2LH9Q1Z4CXaxD0gDGzUxgdAaaAgLAR4btWi2n7Tb7ZNDdOfctg2ya9ftzZBOHvCaVXSpJCry/GMU= skoduri@localhost.localdomain", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2021-02-10T11:04:38.083 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} 2021-02-10T11:04:38.253 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} 2021-02-10T11:04:38.526 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} 2021-02-10T11:04:38.704 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} 2021-02-10T11:04:38.976 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} 2021-02-10T11:04:39.143 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} 2021-02-10T11:04:39.426 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} 2021-02-10T11:04:39.593 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} 2021-02-10T11:04:39.864 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDS3HkgHb6z+gi4tN5rBL/N+WLJgN8RtUokt0YnRjpor2epRbmqgmEFRbRFqjLK9LiTuLxmkwol1/G56AJms9ixue33M6jVs73nc259LDxn2qj/NCtxeU7xdjSDuhKyItgM0yuiFjWOKSYTghNtwoR26PX7SQVbycD4NN1ZuwUbdMQA31VrLvQR7d0lYKlis2auH2Xb6WifotoEyszZz1KYRwzJvOHtHku/jAyNmTD/8kAsuCjEgUPSyyehAfdWn59tY2gM7aFyj/dtRoAghekv5gkov9zx+zjN6D2LSN7e+DRti05iJMifA3nfmdj7f8aHzXDtsDL0XNkmmT3g9N2OQ8WL4ESOFASXPPj9JGUROealgnaQ+Y64evDClhbt2L7FLbgj5sgKb+XYhpGnAiC+v3t1Rq3HZrstOYf2qSYguIo+zW9gHwzJDqH5YNCpSF4ZDCfkJ94ngl5ptQmZ/FcrH2hBs2cMoyK9EyDpcI9hBeACTzu19kaB6DrOC+UTvBgYL2sNbCvi+caIqhENJICZ21rI4u4EZqu7PVb+Y3CLgSjTEgclGYBZoxlrT/X2QHrIfZF7lLGeeh0bureedCrBg4b61SL7ZU3Me9x1iWOZMSJgVvg+qPSrzZiHbA46k8i0af5Q+geL1BuuGy76zbYd/KR0KiNZnXjdlHyRz76rpQ== kamoltatsirivadhna@Ksirivad-MacBook-Pro.loca 2021-02-10T11:04:39.865 INFO:teuthology.task.ansible.out:l", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} 2021-02-10T11:04:40.033 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDS3HkgHb6z+gi4tN5rBL/N+WLJgN8RtUokt0YnRjpor2epRbmqgmEFRbRFqjLK9LiTuLxmkwol1/G56AJms9ixue33M6jVs73nc259LDxn2qj/NCtxeU7xdjSDuhKyItgM0yuiFjWOKSYTghNtwoR26PX7SQVbycD4NN1ZuwUbdMQA31VrLvQR7d0lYKlis2auH2Xb6WifotoEyszZz1KYRwzJvOHtHku/jAyNmTD/8kAsuCjEgUPSyyehAfdWn59tY2gM7aFyj/dtRoAghekv5gkov9zx+zjN6D2LSN7e+DRti05iJMifA3nfmdj7f8aHzXDtsDL0XNkmmT3g9N2OQ8WL4ESOFASXPPj9JGUROealgnaQ+Y64evDClhbt2L7FLbgj5sgKb+XYhpGnAiC+v3t1Rq3HZrstOYf2qSYguIo+zW9gHwzJDqH5YNCpSF4ZDCfkJ94ngl5ptQmZ/FcrH2hBs2cMoyK9EyDpcI9hBeACTzu19kaB6DrOC+UTvBgYL2sNbCvi+caIqhENJICZ21rI4u4EZqu7PVb+Y3CLgSjTEgclGYBZoxlrT/X2QHrIfZF7lLGeeh0bureedCrBg4b61SL7ZU3Me9x1iWOZMSJgVvg+qPSrzZiHbA46k8i0af5Q+geL1BuuGy76zbYd/KR0KiNZnXjdlHyRz76rpQ== kamoltatsirivadhna@Ksirivad-MacBook-Pro.loca 2021-02-10T11:04:40.035 INFO:teuthology.task.ansible.out:l", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} 2021-02-10T11:04:40.305 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2021-02-10T11:04:40.481 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2021-02-10T11:04:40.753 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} 2021-02-10T11:04:40.919 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} 2021-02-10T11:04:41.191 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2021-02-10T11:04:41.359 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2021-02-10T11:04:41.641 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} 2021-02-10T11:04:41.809 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} 2021-02-10T11:04:42.083 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} 2021-02-10T11:04:42.252 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} 2021-02-10T11:04:42.524 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} 2021-02-10T11:04:42.691 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} 2021-02-10T11:04:42.963 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", " 2021-02-10T11:04:42.965 INFO:teuthology.task.ansible.out:manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-02-10T11:04:43.143 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", " 2021-02-10T11:04:43.145 INFO:teuthology.task.ansible.out:manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-02-10T11:04:43.416 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDOJg56HsyNa5vHHb52G6O5vtv9I2tdlNMjFlG3SVCCu/bVZwVeGzp1ekuNGU10owmnIvhlSldMjLn2wwTwiDjbvYJVVCPvzK63vCUcMevRsvr9EyU16JdAgrmrGMyuYYEb+wh8XWI9Dc72o84fMaBYFSSer1S9nUBL+KVLVhIe7culgADimrDgDVh2t7u2JHBI0m0nKFGJZNcIyYf+GKOM+n7SaQXIYIytvSteU5BvUIKPsDd4z5lVlWiKO3kG6jF2RnKFV6KWBWI9m8YGItdGu25mXELXN9MOlApZF3AoGTmCE2Web7uDVGvO6rz6b+dc7RXTOlbWIhtMoRqQf6mz zac.dover@gmail.com", "key_options": null, "keyfile": "/home/zdover/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zdover", "validate_certs": true} 2021-02-10T11:04:43.584 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDOJg56HsyNa5vHHb52G6O5vtv9I2tdlNMjFlG3SVCCu/bVZwVeGzp1ekuNGU10owmnIvhlSldMjLn2wwTwiDjbvYJVVCPvzK63vCUcMevRsvr9EyU16JdAgrmrGMyuYYEb+wh8XWI9Dc72o84fMaBYFSSer1S9nUBL+KVLVhIe7culgADimrDgDVh2t7u2JHBI0m0nKFGJZNcIyYf+GKOM+n7SaQXIYIytvSteU5BvUIKPsDd4z5lVlWiKO3kG6jF2RnKFV6KWBWI9m8YGItdGu25mXELXN9MOlApZF3AoGTmCE2Web7uDVGvO6rz6b+dc7RXTOlbWIhtMoRqQf6mz zac.dover@gmail.com", "key_options": null, "keyfile": "/home/zdover/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zdover", "validate_certs": true} 2021-02-10T11:04:43.858 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC4zwWf+kMEDgaSOrX7AlD+UCOgf4/fRiorQVJqC8GfMNynfiFVZaThngwIrwtQROfXHbzspi+AcRvDLewY25mXyuxVgOfSbuSIEqiUNvzcVq9IItI1J58xeDBpIktAsoHHS6JENV56fAT6krzRAQLnSa1k2b0pyKuPBzFSeNgiaQMhjrCLAaCi9yLsdKpnWgXhyfrdqvN2r/Ctts/zlTPqtRYdvdIL/p/l+vg7Q8S3OhuAGtimaEq3RE70xadaPbGhsLISMTrbfYXbxw84KKZpNYwO4zmQ2ztcvC9D7A/8ov3+A5Jq42G+pq7ORuix2NoTWtADMiwvNe1HyzS/7cX/BuXLFssRY1bzRvp+x2ExZ7b+bFx+TE/RduRh0eyDNLg7SyL83aeTr2Usa+mPoCAMKed9XtYbRnUu7ox4wmfeAgx6jM9uG5uQfauJ1XpDddrVBG8KYBvSd0BM2cyRdcAXqwxg9AVPZcdE7sbBk44CGVhw+ziZ3e3cLBTO4y44KRQre+1Bhe/Ze8OTQWLFugl/XagFlJE/L6fGUIPuFojmQvlw5dBZwtUURnxsu8yHm3iIsj4y8nR1zHMj+zuMk6rF3ZW0epPAECAdq4gG8X2wv3HTq0vLBkKpi1KMAmntEtcVwF0fzMt5XYqRd4jceFgonAAipJNDknHgi6yaUzBYmQ== rraja@redhat.com", "key_options": null, "keyfile": "/home/rraja/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rraja", "validate_certs": true} 2021-02-10T11:04:44.028 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC4zwWf+kMEDgaSOrX7AlD+UCOgf4/fRiorQVJqC8GfMNynfiFVZaThngwIrwtQROfXHbzspi+AcRvDLewY25mXyuxVgOfSbuSIEqiUNvzcVq9IItI1J58xeDBpIktAsoHHS6JENV56fAT6krzRAQLnSa1k2b0pyKuPBzFSeNgiaQMhjrCLAaCi9yLsdKpnWgXhyfrdqvN2r/Ctts/zlTPqtRYdvdIL/p/l+vg7Q8S3OhuAGtimaEq3RE70xadaPbGhsLISMTrbfYXbxw84KKZpNYwO4zmQ2ztcvC9D7A/8ov3+A5Jq42G+pq7ORuix2NoTWtADMiwvNe1HyzS/7cX/BuXLFssRY1bzRvp+x2ExZ7b+bFx+TE/RduRh0eyDNLg7SyL83aeTr2Usa+mPoCAMKed9XtYbRnUu7ox4wmfeAgx6jM9uG5uQfauJ1XpDddrVBG8KYBvSd0BM2cyRdcAXqwxg9AVPZcdE7sbBk44CGVhw+ziZ3e3cLBTO4y44KRQre+1Bhe/Ze8OTQWLFugl/XagFlJE/L6fGUIPuFojmQvlw5dBZwtUURnxsu8yHm3iIsj4y8nR1zHMj+zuMk6rF3ZW0epPAECAdq4gG8X2wv3HTq0vLBkKpi1KMAmntEtcVwF0fzMt5XYqRd4jceFgonAAipJNDknHgi6yaUzBYmQ== rraja@redhat.com", "key_options": null, "keyfile": "/home/rraja/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rraja", "validate_certs": true} 2021-02-10T11:04:44.300 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2mEyxfkW1u84u5BocLCAXESj2qObWkxUd20cmPolhJYlIWLOafa69Dw5A9CjFBp0U6Jx5d0JWcwmAc5ZeffQ7t7vzmtHCZnpKkyZhN9bu6vuCh4N+I643SI/K7tdbB2E8ZE6uI7ORMAan5+t0bNSVolUVq6X1w7dIA5El54K10WNdiwgRKW6e3anNbJ3rBLGEk7gawZqDDOqTvVQmOBGWek3BjSReDvg3rkmTQLxJkiKdxnx4/s7gzH1xTpsS/OludIuSxF0BK064ZcT1t09qcjnT4Os1inV2GfYswt+UhpgqrHqXjEs0Z3y4Oy77QBKL1rDf93CP4a4+eJjqXe+v tbechtold@suse.com", "key_options": null, "keyfile": "/home/toabctl/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "toabctl", "validate_certs": true} 2021-02-10T11:04:44.478 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2mEyxfkW1u84u5BocLCAXESj2qObWkxUd20cmPolhJYlIWLOafa69Dw5A9CjFBp0U6Jx5d0JWcwmAc5ZeffQ7t7vzmtHCZnpKkyZhN9bu6vuCh4N+I643SI/K7tdbB2E8ZE6uI7ORMAan5+t0bNSVolUVq6X1w7dIA5El54K10WNdiwgRKW6e3anNbJ3rBLGEk7gawZqDDOqTvVQmOBGWek3BjSReDvg3rkmTQLxJkiKdxnx4/s7gzH1xTpsS/OludIuSxF0BK064ZcT1t09qcjnT4Os1inV2GfYswt+UhpgqrHqXjEs0Z3y4Oy77QBKL1rDf93CP4a4+eJjqXe+v tbechtold@suse.com", "key_options": null, "keyfile": "/home/toabctl/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "toabctl", "validate_certs": true} 2021-02-10T11:04:44.752 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCitWIN7jm5KEmB6jqcA+ufKRLodxkJ1GIdaaueCdeXdMHgWNH9FUDCUJk5dcA11MxQO15DRH/cdhb3QHPGiMv4XrBeK6HHYwGttzkcUEz1EDUhpowOrnFETbPa7K39ObmRcgdvoGXOSrNZ4mvptCTz9Xb0MF+Gcg5iJ4g+qZzIVa2/ueecYY8ZorbaH4m/NWUub6Rf01Q04HfxTvKrUFswrbCoE/Aigz/Oh88RiS854XoTvV7rE1tKbyKkaungwMAaQSNnZkq4hi8nbp6pzTxJLV7N/ZYMhVpdxSbWcghFg6cy7aeoCzZMN97WcDhpdfbhRT0dbqCkMmHV6t0usDDnOss2v68UCzkxuA1Hj4uy8rRHjXBtYqI6xsrwCjvmgP05B14TpmFdG+3GdRHx2+QlhnGj7nuPf1KoWW/f3tD85DXoEmVl6RJ4KTrMeEVlOn3kdMLVH0y2hIqPTYMP5cMVNi0EdfeoBdd9kxMKMvoSVBAgmLzGVv17o1lNvVS6V1kyt7gME2PfNR2Np+U1cJVJfUJRTm28mHXBVoDXG9s0zv8+LpQ/gVPS+vsPAfxWWYGW8ZmEV3ZorDOZycSPybjHkPFzDAdUhPG0Db0o+F9ug+22F7pvifsdDZ42zqGGLbyzfIQE3AtyI7LR7Sc5tpN9rKTeSBYJ0yNcDKEj5FUHDw== mfritch@suse.com", "key_options": null, "keyfile": "/home/mgfritch/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mgfritch", "validate_certs": true} 2021-02-10T11:04:44.919 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCitWIN7jm5KEmB6jqcA+ufKRLodxkJ1GIdaaueCdeXdMHgWNH9FUDCUJk5dcA11MxQO15DRH/cdhb3QHPGiMv4XrBeK6HHYwGttzkcUEz1EDUhpowOrnFETbPa7K39ObmRcgdvoGXOSrNZ4mvptCTz9Xb0MF+Gcg5iJ4g+qZzIVa2/ueecYY8ZorbaH4m/NWUub6Rf01Q04HfxTvKrUFswrbCoE/Aigz/Oh88RiS854XoTvV7rE1tKbyKkaungwMAaQSNnZkq4hi8nbp6pzTxJLV7N/ZYMhVpdxSbWcghFg6cy7aeoCzZMN97WcDhpdfbhRT0dbqCkMmHV6t0usDDnOss2v68UCzkxuA1Hj4uy8rRHjXBtYqI6xsrwCjvmgP05B14TpmFdG+3GdRHx2+QlhnGj7nuPf1KoWW/f3tD85DXoEmVl6RJ4KTrMeEVlOn3kdMLVH0y2hIqPTYMP5cMVNi0EdfeoBdd9kxMKMvoSVBAgmLzGVv17o1lNvVS6V1kyt7gME2PfNR2Np+U1cJVJfUJRTm28mHXBVoDXG9s0zv8+LpQ/gVPS+vsPAfxWWYGW8ZmEV3ZorDOZycSPybjHkPFzDAdUhPG0Db0o+F9ug+22F7pvifsdDZ42zqGGLbyzfIQE3AtyI7LR7Sc5tpN9rKTeSBYJ0yNcDKEj5FUHDw== mfritch@suse.com", "key_options": null, "keyfile": "/home/mgfritch/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mgfritch", "validate_certs": true} 2021-02-10T11:04:45.193 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5bCXQ/B16AKl2oevC93R0U+3OrTyd3XEypouJy4LjIzXLX/Go1C1ViERwDAeHmqiUTqJUL+KZ0kMo8U7zlhNBk9PucbMZCqdqx5hkiCAo4y65lEXmuPGimRU2mK9TpcDTLWoyrD/EEhwzT6gmw3ytVTxi7qAb0tQUCQB+pW4+Zr5u6HZzgu9kLGxp7zsGtMA84Zqm5ql72mGuS0bUN1ajm8LWBetFuzOOW6YGfnKt29cLV2h1itVewA9DKM5G70B91eK4LjoNddCkqVA8L+v6LMQNU2pejUzPap98hmFUpQMNxlWQ0+B7UjfbkHWe+mIsSkdr3FfCfqWvsL0RiHcZ root@lxbceph0", "key_options": null, "keyfile": "/home/xiubli/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiubli", "validate_certs": true} 2021-02-10T11:04:45.371 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5bCXQ/B16AKl2oevC93R0U+3OrTyd3XEypouJy4LjIzXLX/Go1C1ViERwDAeHmqiUTqJUL+KZ0kMo8U7zlhNBk9PucbMZCqdqx5hkiCAo4y65lEXmuPGimRU2mK9TpcDTLWoyrD/EEhwzT6gmw3ytVTxi7qAb0tQUCQB+pW4+Zr5u6HZzgu9kLGxp7zsGtMA84Zqm5ql72mGuS0bUN1ajm8LWBetFuzOOW6YGfnKt29cLV2h1itVewA9DKM5G70B91eK4LjoNddCkqVA8L+v6LMQNU2pejUzPap98hmFUpQMNxlWQ0+B7UjfbkHWe+mIsSkdr3FfCfqWvsL0RiHcZ root@lxbceph0", "key_options": null, "keyfile": "/home/xiubli/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiubli", "validate_certs": true} 2021-02-10T11:04:45.642 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDT+6B5Fr99p8xe9Zo5x8ULcdeA6ne727iVJp/xaguGwGHj6AhuK+R/CjznnHk0yA+6kiblNzjvbnsuzO8T9c8Ayv5KU07MxP1y1pymdYBpEh3NTXmhB6QOTCNTQbT14PL6MVgzK17SqarFCYXoEGuyK9jaBdN9QM76vhEaXc/2Osg+d8boJTkOk7lOPHbzeOCbQU3d+jX0VFOgAQS5OFz3Ug/X7nB0D0LN8bt5FlCZiGO95Iwls/SyPDbVXc9ot8oh4ME7kd7sElokBdYdXPauibRxWKjSDn+hV6kZfSzqwnMFaZ1TC+Av6zNsBuZucNSp3/Ws3T5i9i+21eyS4+qVE8IiX2aVKYuC9afb2lPLbdG2Iy1CqiaeGPkZfiGT9P8iQPOkqae45859UzFAd/XzKjBcmus3lSArKG1EyZl03HQ5P2J60pErCXq8W9mPjxAze2ReLRXJVEgTXnmdhCptcg6PZ4d/k4BEddbdnoX6/BEgFxLEJ64HpTeTVcDXHpipCpxFSRvgyPH/wQGKAvt8AfKBhmeZs7LuyGOhJiOon4DDaShHpE9wTnJ5qlnyg44+u6NCwelGPMAkHKEO4OgcjSsyXm+HYUpxG2s4poloIJzvU6oa3dvJsMZFhcb8CsW75sBYE+YngS/1fg1WU88ZOdaa1yF4F87pVZycpZqY1w== jan@jfsuselaptop", "key_options": null, "keyfile": "/home/jafaj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jafaj", "validate_certs": true} 2021-02-10T11:04:45.815 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDT+6B5Fr99p8xe9Zo5x8ULcdeA6ne727iVJp/xaguGwGHj6AhuK+R/CjznnHk0yA+6kiblNzjvbnsuzO8T9c8Ayv5KU07MxP1y1pymdYBpEh3NTXmhB6QOTCNTQbT14PL6MVgzK17SqarFCYXoEGuyK9jaBdN9QM76vhEaXc/2Osg+d8boJTkOk7lOPHbzeOCbQU3d+jX0VFOgAQS5OFz3Ug/X7nB0D0LN8bt5FlCZiGO95Iwls/SyPDbVXc9ot8oh4ME7kd7sElokBdYdXPauibRxWKjSDn+hV6kZfSzqwnMFaZ1TC+Av6zNsBuZucNSp3/Ws3T5i9i+21eyS4+qVE8IiX2aVKYuC9afb2lPLbdG2Iy1CqiaeGPkZfiGT9P8iQPOkqae45859UzFAd/XzKjBcmus3lSArKG1EyZl03HQ5P2J60pErCXq8W9mPjxAze2ReLRXJVEgTXnmdhCptcg6PZ4d/k4BEddbdnoX6/BEgFxLEJ64HpTeTVcDXHpipCpxFSRvgyPH/wQGKAvt8AfKBhmeZs7LuyGOhJiOon4DDaShHpE9wTnJ5qlnyg44+u6NCwelGPMAkHKEO4OgcjSsyXm+HYUpxG2s4poloIJzvU6oa3dvJsMZFhcb8CsW75sBYE+YngS/1fg1WU88ZOdaa1yF4F87pVZycpZqY1w== jan@jfsuselaptop", "key_options": null, "keyfile": "/home/jafaj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jafaj", "validate_certs": true} 2021-02-10T11:04:46.100 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo8wE/b2/EDwV018ENfcENVfXSF7vnXAwBSzNVScgxdqr2DVKTWDzhoI0m5CXTqI2HLnJVFDTgN3t5/fFcmwmSAP8ic8wU1JznSPZXN7qEQrcEsU/cwHeuzyutibg1ItTWWtoXYV8SFnVom8NF3UwXF8I+Vj6VF1EStWCcClgPjLbiOm8vIgaouXtbUIFv25GpadXyUtwiHEq1NKUPLgka8Usm8ibyS9hY9KmNWlD9AfmnwAImc8Urn2dErcsfAltxoGUHCI7ZBHGpzg+6dpynNR+giJMrjwWzREaCHmiJX3g6jmHuuY54eSOs6cxqi3E4jNHVWXPe/tEF3l0klEvl jschmid@suse.com", "key_options": null, "keyfile": "/home/jschmid/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jschmid", "validate_certs": true} 2021-02-10T11:04:46.257 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo8wE/b2/EDwV018ENfcENVfXSF7vnXAwBSzNVScgxdqr2DVKTWDzhoI0m5CXTqI2HLnJVFDTgN3t5/fFcmwmSAP8ic8wU1JznSPZXN7qEQrcEsU/cwHeuzyutibg1ItTWWtoXYV8SFnVom8NF3UwXF8I+Vj6VF1EStWCcClgPjLbiOm8vIgaouXtbUIFv25GpadXyUtwiHEq1NKUPLgka8Usm8ibyS9hY9KmNWlD9AfmnwAImc8Urn2dErcsfAltxoGUHCI7ZBHGpzg+6dpynNR+giJMrjwWzREaCHmiJX3g6jmHuuY54eSOs6cxqi3E4jNHVWXPe/tEF3l0klEvl jschmid@suse.com", "key_options": null, "keyfile": "/home/jschmid/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jschmid", "validate_certs": true} 2021-02-10T11:04:46.540 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDnQfkkxYWLhMdoQ4RsZxWTXBpbzsssgeRfvV5q+n0mW60dxQ+9rjDXUTsRkmh45B5EhNWlKwyIUcQ5HpgJk6XQvYU6MPboWjriCTngmoUbwNIjOjyazkBbxfbwUiRDb+x1EyVm4Kt0rgj3nhqqxf3krqO2SlKgsVaY9b7+NTfshXb9T73/RRLtvwUOFkjp2B+w0SQs7Ka1GenmHSY/6xP25Mb8q9QW/2nuf1mV9rTd5l2ZHREzgG7gQFhjgUq3w6zCCfcCGwrPb4omqd9xiYLDw8l1vfQ7UPbuOqK1rFM67r7JQDG3yyJjCdWgRUI/ysbbXj0fyT+QOPZGK618V8yZPVlq5loQzYI39zAWZU9L9X9bkQCsJfjU5Yo9f/zOaecnWfUxC84NKpbG7k6XACdUIB9JGGtL6RPtgBU+k6Ts7EemIUCyMYm/GX+PYU3GnmmdqW9yBeyZoldwDCWLq05vjLXj+R3J2UrMl5x+fhP+KEFd1ZgNc7PV+CF35/ux8UE= kachau@localhost.localdomain", "key_options": null, "keyfile": "/home/kachau/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kachau", "validate_certs": true} 2021-02-10T11:04:46.699 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDnQfkkxYWLhMdoQ4RsZxWTXBpbzsssgeRfvV5q+n0mW60dxQ+9rjDXUTsRkmh45B5EhNWlKwyIUcQ5HpgJk6XQvYU6MPboWjriCTngmoUbwNIjOjyazkBbxfbwUiRDb+x1EyVm4Kt0rgj3nhqqxf3krqO2SlKgsVaY9b7+NTfshXb9T73/RRLtvwUOFkjp2B+w0SQs7Ka1GenmHSY/6xP25Mb8q9QW/2nuf1mV9rTd5l2ZHREzgG7gQFhjgUq3w6zCCfcCGwrPb4omqd9xiYLDw8l1vfQ7UPbuOqK1rFM67r7JQDG3yyJjCdWgRUI/ysbbXj0fyT+QOPZGK618V8yZPVlq5loQzYI39zAWZU9L9X9bkQCsJfjU5Yo9f/zOaecnWfUxC84NKpbG7k6XACdUIB9JGGtL6RPtgBU+k6Ts7EemIUCyMYm/GX+PYU3GnmmdqW9yBeyZoldwDCWLq05vjLXj+R3J2UrMl5x+fhP+KEFd1ZgNc7PV+CF35/ux8UE= kachau@localhost.localdomain", "key_options": null, "keyfile": "/home/kachau/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kachau", "validate_certs": true} 2021-02-10T11:04:46.995 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQujcXnq9qOCmTWs44GwKOw48BxZGBmo9AeFk3KPRx33Cz5hS7UHPYvJy03KdoMySgxJtuxsOSvjJSPxpXesTtArO5giW8RLyuVsu9q5j/GhPKyABttuGQyexiRokrHQFLVifzgqwsUARwHarWH16Oa1n6fZPFNqSH56c872zS4Pwqgkzx99NKRWh/B+fIk8VmDzzP1qvQAnUXDeTeTOzapAL+8fSNTp3QnhmZbvCHYwUxfSqJyzq1wBL+517WrvzvX7yCY9B0wzm72sIO4daV6UDyAL1B72QpB04vEZZ/skwS/0jmokfBn43HHCMPy/Mxywfi54alFJzIu4pti30V kravishankar@localhost.localdomain", "key_options": null, "keyfile": "/home/khiremat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "khiremat", "validate_certs": true} 2021-02-10T11:04:47.152 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQujcXnq9qOCmTWs44GwKOw48BxZGBmo9AeFk3KPRx33Cz5hS7UHPYvJy03KdoMySgxJtuxsOSvjJSPxpXesTtArO5giW8RLyuVsu9q5j/GhPKyABttuGQyexiRokrHQFLVifzgqwsUARwHarWH16Oa1n6fZPFNqSH56c872zS4Pwqgkzx99NKRWh/B+fIk8VmDzzP1qvQAnUXDeTeTOzapAL+8fSNTp3QnhmZbvCHYwUxfSqJyzq1wBL+517WrvzvX7yCY9B0wzm72sIO4daV6UDyAL1B72QpB04vEZZ/skwS/0jmokfBn43HHCMPy/Mxywfi54alFJzIu4pti30V kravishankar@localhost.localdomain", "key_options": null, "keyfile": "/home/khiremat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "khiremat", "validate_certs": true} 2021-02-10T11:04:47.433 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCoqsDnRqAeNPi5xiIfFl2i2zqcwLTvcAso3R9+QAfSChw4+wcQzmMV8uQz1jt8UReefg2QL6bwvcQPj5Zw7ADqUjfxm1K0ezgH6RrtHgRZ1SVseTvJUvzmkDZ2gr2ToZAJYsaV7gBLJLZ7TOiQ6jzrtitiKqJZdpZfzadjShecY1TdMo8bSS7XfbRcwri40u82TRM825B4L//QuhMI7akPqa3iJbGEAQPhkGLI3kz19ryVG1UwvAkkt7TQ68zz+piXSmB2CMRti51shT4NuAOq6Qe9y2Obicf88ItkYxvySMQ5jWi2bo7Zhwuq9jUH8lK6I6PHrYc8LOsR9gE/hqkXxTQrkOXEDtHN1mr/KKLrXO0GeBdgh8Gj4gcwgd1RUCE+T9q09WO9j7c0piWK9Fs1+q1FjH06/AfmP4qob1TlJMHhtPyOp+lk2XvABP+sqW4vyrgmXu5vJU+Hgq3X9KE7bKh4WkCem24Np4uA+irhpvZmIoHyINsYW46lGfwJvM8= kapandya@localhost.localdomain", "key_options": null, "keyfile": "/home/kapandya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kapandya", "validate_certs": true} 2021-02-10T11:04:47.590 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCoqsDnRqAeNPi5xiIfFl2i2zqcwLTvcAso3R9+QAfSChw4+wcQzmMV8uQz1jt8UReefg2QL6bwvcQPj5Zw7ADqUjfxm1K0ezgH6RrtHgRZ1SVseTvJUvzmkDZ2gr2ToZAJYsaV7gBLJLZ7TOiQ6jzrtitiKqJZdpZfzadjShecY1TdMo8bSS7XfbRcwri40u82TRM825B4L//QuhMI7akPqa3iJbGEAQPhkGLI3kz19ryVG1UwvAkkt7TQ68zz+piXSmB2CMRti51shT4NuAOq6Qe9y2Obicf88ItkYxvySMQ5jWi2bo7Zhwuq9jUH8lK6I6PHrYc8LOsR9gE/hqkXxTQrkOXEDtHN1mr/KKLrXO0GeBdgh8Gj4gcwgd1RUCE+T9q09WO9j7c0piWK9Fs1+q1FjH06/AfmP4qob1TlJMHhtPyOp+lk2XvABP+sqW4vyrgmXu5vJU+Hgq3X9KE7bKh4WkCem24Np4uA+irhpvZmIoHyINsYW46lGfwJvM8= kapandya@localhost.localdomain", "key_options": null, "keyfile": "/home/kapandya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kapandya", "validate_certs": true} 2021-02-10T11:04:47.870 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNEEz49M5KdLcp0cbKrE9iihrH5PEhsPgnAxac+aQwvnTstOOlUOyR+TvHiRa7sgXHuFX/UXo+k3Qmhs5U/sMZWZmmN3Jb3fKT78lSBbGnByclfPoGVTTLOHxwo+RFE79Sp3e30oshJ3DwqYgiX6JjM8B2Y7d5Tv0Ss5uEZswvmCEjdTLDSsoxrsOjVxBpuArOfCxes8PnKDdq21mWldpP66iwFel2GgOGSc85iVHKcOE2OpVMazMzXoMSMFoOpCULKjFfXBuDIqkXwJmtLJaiCgcVmTR56KXh7JpZ7vAeJ9ttv9qgOckqKdI3jEjeDT+TrdLlRWghBE7FRkm0vmun prasad.krishnan@macprasad", "key_options": null, "keyfile": "/home/prasad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prasad", "validate_certs": true} 2021-02-10T11:04:48.028 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNEEz49M5KdLcp0cbKrE9iihrH5PEhsPgnAxac+aQwvnTstOOlUOyR+TvHiRa7sgXHuFX/UXo+k3Qmhs5U/sMZWZmmN3Jb3fKT78lSBbGnByclfPoGVTTLOHxwo+RFE79Sp3e30oshJ3DwqYgiX6JjM8B2Y7d5Tv0Ss5uEZswvmCEjdTLDSsoxrsOjVxBpuArOfCxes8PnKDdq21mWldpP66iwFel2GgOGSc85iVHKcOE2OpVMazMzXoMSMFoOpCULKjFfXBuDIqkXwJmtLJaiCgcVmTR56KXh7JpZ7vAeJ9ttv9qgOckqKdI3jEjeDT+TrdLlRWghBE7FRkm0vmun prasad.krishnan@macprasad", "key_options": null, "keyfile": "/home/prasad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prasad", "validate_certs": true} 2021-02-10T11:04:48.314 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCu9CW99ySPXSKQpLbefu1A0KqbUQIzvUOJizIu4azEgdlOqUnOMR/GC6jgOdiVL5mNb/U5bg7yQuEHgPAtmPRotvbI6/FSwIAR9+GgiuKSOJCa4s7QzWbHwv9g7wS2SLURw8UERSzlLlT8KQyGj+GkBhToWp4PTXC7te61c8XJNGF6giLmIAky4SH8uPI1zeLwwviJQomjFWZZ1RTbcRJxvGPjPtuUKDlF7QgXTLE9ZgsNopusC6PfRd7N8Udkl2OYPAzA+Jun0IRnQyjRXG3c2sYXQR8YYGICsxvnm4l81vKdxRL34K7LqBUHQXUcoCjJy7+/2/1dca/AMHgi4sZwgeYQ/U9jZg5jbm7o4MpDUFsS8W6OO/ZPiuYTswGBNRXhPmbGFnfa9dBXBS0p+buHF8PcTSLH0Dp2nhHkK1ehGmBFVPw6nPj3wXA9BKuNc+g7AiNUyRKyCWroFJ0VwUQGbI6ZMccO2cI7QpYmIqPCl3am/vUSNq4j0m/iURR7+t0= vikhyat@redhat", "key_options": null, "keyfile": "/home/vikhyat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vikhyat", "validate_certs": true} 2021-02-10T11:04:48.471 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCu9CW99ySPXSKQpLbefu1A0KqbUQIzvUOJizIu4azEgdlOqUnOMR/GC6jgOdiVL5mNb/U5bg7yQuEHgPAtmPRotvbI6/FSwIAR9+GgiuKSOJCa4s7QzWbHwv9g7wS2SLURw8UERSzlLlT8KQyGj+GkBhToWp4PTXC7te61c8XJNGF6giLmIAky4SH8uPI1zeLwwviJQomjFWZZ1RTbcRJxvGPjPtuUKDlF7QgXTLE9ZgsNopusC6PfRd7N8Udkl2OYPAzA+Jun0IRnQyjRXG3c2sYXQR8YYGICsxvnm4l81vKdxRL34K7LqBUHQXUcoCjJy7+/2/1dca/AMHgi4sZwgeYQ/U9jZg5jbm7o4MpDUFsS8W6OO/ZPiuYTswGBNRXhPmbGFnfa9dBXBS0p+buHF8PcTSLH0Dp2nhHkK1ehGmBFVPw6nPj3wXA9BKuNc+g7AiNUyRKyCWroFJ0VwUQGbI6ZMccO2cI7QpYmIqPCl3am/vUSNq4j0m/iURR7+t0= vikhyat@redhat", "key_options": null, "keyfile": "/home/vikhyat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vikhyat", "validate_certs": true} 2021-02-10T11:04:48.762 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCipD0d1tI/Cff3AeOZTQ9eb6pZ/NOCAOw7MZG/JgpCs8+T3WqiUEYefsb2PuSa85vV2GC7kq0M9ux7+U+vZZf05geYiU2S9eTucLEu2+YE+RaAvIK+ZB134/33JSPGIVubXgGWZw/ZFCEYXmYk3kyEg1YDIk/BEaoG7PsDiSB4j0W7/WiV2bEEdSlTXt5C1/kH0qn4H9wQMiBSpdMgdaRWw60HjObDd6LASeUVgabbyclypravZdpYBnits4ayoN8VG1aZe4Qhm1SzP/CfxAAYfxo9NbqSBWJmn3iSN8UKHhCFrRCTaYic3z3usr8784VsA7z/LxK0fs6xjhoxRTa3 shraddha.agrawal000@gmail.com", "key_options": null, "keyfile": "/home/shraddhaag/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shraddhaag", "validate_certs": true} 2021-02-10T11:04:48.763 INFO:teuthology.task.ansible.out: 2021-02-10T11:04:48.920 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCipD0d1tI/Cff3AeOZTQ9eb6pZ/NOCAOw7MZG/JgpCs8+T3WqiUEYefsb2PuSa85vV2GC7kq0M9ux7+U+vZZf05geYiU2S9eTucLEu2+YE+RaAvIK+ZB134/33JSPGIVubXgGWZw/ZFCEYXmYk3kyEg1YDIk/BEaoG7PsDiSB4j0W7/WiV2bEEdSlTXt5C1/kH0qn4H9wQMiBSpdMgdaRWw60HjObDd6LASeUVgabbyclypravZdpYBnits4ayoN8VG1aZe4Qhm1SzP/CfxAAYfxo9NbqSBWJmn3iSN8UKHhCFrRCTaYic3z3usr8784VsA7z/LxK0fs6xjhoxRTa3 shraddha.agrawal000@gmail.com", "key_options": null, "keyfile": "/home/shraddhaag/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shraddhaag", "validate_certs": true} 2021-02-10T11:04:49.200 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpE5V9JG1WhkHSYqjqaijBb1MLyjcKQlmj5WQ8w5832n6x9J4yDO8wI4qmiX3XeiXPlmW6XjAq5keyukrQe/yVI6NDXzgRxaHDmNj1AR73gyUVqRXpJprrqqsnLHawIRovrBD/HNhTv/2G8xG3BHakac1+qj7tZe4Jjx0aG1HbRhhJ5Q98EgvTkDAhktnlvF9OXXC1WI7QirHx0pRHa40SYyFLfOQDbQSCzb1TAAY5NGn3SwK3s2mWeoEsAJpCLrkNfSTbR3Z0mylAJUn+PayqhJegjwKmLaONjx8ZHXN970JHgojBBGOD30J/NnJQ7N6eRl+ekjpHmLAYGbe2oCwD gkyratsas@suse.com", "key_options": null, "keyfile": "/home/gkyratsas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gkyratsas", "validate_certs": true} 2021-02-10T11:04:49.357 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpE5V9JG1WhkHSYqjqaijBb1MLyjcKQlmj5WQ8w5832n6x9J4yDO8wI4qmiX3XeiXPlmW6XjAq5keyukrQe/yVI6NDXzgRxaHDmNj1AR73gyUVqRXpJprrqqsnLHawIRovrBD/HNhTv/2G8xG3BHakac1+qj7tZe4Jjx0aG1HbRhhJ5Q98EgvTkDAhktnlvF9OXXC1WI7QirHx0pRHa40SYyFLfOQDbQSCzb1TAAY5NGn3SwK3s2mWeoEsAJpCLrkNfSTbR3Z0mylAJUn+PayqhJegjwKmLaONjx8ZHXN970JHgojBBGOD30J/NnJQ7N6eRl+ekjpHmLAYGbe2oCwD gkyratsas@suse.com", "key_options": null, "keyfile": "/home/gkyratsas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gkyratsas", "validate_certs": true} 2021-02-10T11:04:49.638 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC9sZe9p48s6RSpluK1R97X9tsN3tWiidPbdyXvyMUXifYQb3nth9wMWUakrVBp+QGEkqALLIf3so3MfgbRvYexzNfHvRT7Lxn8D/isiCpWa4aLM/0alprV+RI38dR0mI3IVb3aIEYmgp2IY4yzybWTHCmS1VUrUJJ0pOfXOssq0xgo3qeaSQBHYRKasqR8GWAD5MWpG6/kg6gxI7o0aHVVTXy5ADB7NS1LEZr06xxNm4exeYvl9TzXrbGZWf2gC8ZpXhmFHI80A+TKEZRU6wmcju7G69nyEEF8/ebcBKmZB4eTsDN7qR7lc9dxKxiiZcJXc/MWy4bSu69NhOeuq2xx7bdexS8OHKdgYH9gsJolu5zgN20IHzaRgswO3XIe3nh2Ud8wntwBCwmJ9bf5YByITGjpOhntkvWAP3AXkQ1B4HXCdlUNW+4pMexDe2D05LHOwxcS8RVIH6ELu8rWe/QBA+DFcyZz4xE55pbfVt3FCEFrAxUn5qkyzszwPMjtK4XAnn9kmnw2B0k6XLZOZd+n7W8ege6wih+gwfDhYMZX0qzi3HdAK7nk5vEOFibCztLaogzfc6fdFRx8OIWJQXqx5I1641FU3UE6sVxku4IMDTq2GVyUWKdSN6y8gBiDPVe3xu4qM4xghSnaAmfB45IrcNvibBjikNva2FHt8lcutQ== prateek@prateek-Dell", "key_options": null, "keyfile": "/home/prateek/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prateek", "validate_certs": true} 2021-02-10T11:04:49.805 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC9sZe9p48s6RSpluK1R97X9tsN3tWiidPbdyXvyMUXifYQb3nth9wMWUakrVBp+QGEkqALLIf3so3MfgbRvYexzNfHvRT7Lxn8D/isiCpWa4aLM/0alprV+RI38dR0mI3IVb3aIEYmgp2IY4yzybWTHCmS1VUrUJJ0pOfXOssq0xgo3qeaSQBHYRKasqR8GWAD5MWpG6/kg6gxI7o0aHVVTXy5ADB7NS1LEZr06xxNm4exeYvl9TzXrbGZWf2gC8ZpXhmFHI80A+TKEZRU6wmcju7G69nyEEF8/ebcBKmZB4eTsDN7qR7lc9dxKxiiZcJXc/MWy4bSu69NhOeuq2xx7bdexS8OHKdgYH9gsJolu5zgN20IHzaRgswO3XIe3nh2Ud8wntwBCwmJ9bf5YByITGjpOhntkvWAP3AXkQ1B4HXCdlUNW+4pMexDe2D05LHOwxcS8RVIH6ELu8rWe/QBA+DFcyZz4xE55pbfVt3FCEFrAxUn5qkyzszwPMjtK4XAnn9kmnw2B0k6XLZOZd+n7W8ege6wih+gwfDhYMZX0qzi3HdAK7nk5vEOFibCztLaogzfc6fdFRx8OIWJQXqx5I1641FU3UE6sVxku4IMDTq2GVyUWKdSN6y8gBiDPVe3xu4qM4xghSnaAmfB45IrcNvibBjikNva2FHt8lcutQ== prateek@prateek-Dell", "key_options": null, "keyfile": "/home/prateek/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prateek", "validate_certs": true} 2021-02-10T11:04:50.087 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFmmTp8+YDXU27xnSPxhWwouslpkvp1eULVp63taoXceDOnH8EMWv0wiEjN/P+Fcf/UB+iU8bELyHubZiqnOUr9f2pGX4jkJ6Yty1p5BUK4my7Si5J0/JawEDVQm4HpFWMKUVKYumvz3UFuBoINHVTqr1cLIDAt7mdeKfJqFB45AqxDUIPP4d8iznNCw+QO2CMxk0o/wMNHbygZZrHAiDngXQxwhe6oR17RfolhREy0JjHgojRjxgagpG7O8WW3OPI5Outnz5Pvraas5q7w+b3xbB9vdTObW/Nii5VOpTdIsFzBvp46hLoRSIcB8pRLUrxPJBPOY7G0zbopDRLB2zLNGKIH2zxEHJOeY928uUbWT22b9mGu/L95FFY7hmeRAicCUeixMTgXgczeQlHDn5eqQzLI3PNP7JX/0dauCD27GRKsDUnpIDj9RqgCQDVueIQQMC6QMa9dYQmps0r9Ljp8R84WjYFRP9oOI+POOzbgmRO0JCVnPqz9MKyGVby9/WFBHcvG6xmwACez00oqe62z+rhcavGXViX5CxGMV6vUTXugBa4ZkXlzs+djzYWfZ33ZKVeRIwj2Pu1PRTGzB8e2BaKqJMN0FVtzpV4hom4Y/9vIPyDpjRjNuKzrkYW1M8PxBxn+EJUPMLhr3jFMCo6HSxHoFXy8BzZoFkZWqap3w== JCL@IT-MINI-JCL.local", "key_options": null, "keyfile": "/home/jelopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jelopez", "validate_certs": true} 2021-02-10T11:04:50.256 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFmmTp8+YDXU27xnSPxhWwouslpkvp1eULVp63taoXceDOnH8EMWv0wiEjN/P+Fcf/UB+iU8bELyHubZiqnOUr9f2pGX4jkJ6Yty1p5BUK4my7Si5J0/JawEDVQm4HpFWMKUVKYumvz3UFuBoINHVTqr1cLIDAt7mdeKfJqFB45AqxDUIPP4d8iznNCw+QO2CMxk0o/wMNHbygZZrHAiDngXQxwhe6oR17RfolhREy0JjHgojRjxgagpG7O8WW3OPI5Outnz5Pvraas5q7w+b3xbB9vdTObW/Nii5VOpTdIsFzBvp46hLoRSIcB8pRLUrxPJBPOY7G0zbopDRLB2zLNGKIH2zxEHJOeY928uUbWT22b9mGu/L95FFY7hmeRAicCUeixMTgXgczeQlHDn5eqQzLI3PNP7JX/0dauCD27GRKsDUnpIDj9RqgCQDVueIQQMC6QMa9dYQmps0r9Ljp8R84WjYFRP9oOI+POOzbgmRO0JCVnPqz9MKyGVby9/WFBHcvG6xmwACez00oqe62z+rhcavGXViX5CxGMV6vUTXugBa4ZkXlzs+djzYWfZ33ZKVeRIwj2Pu1PRTGzB8e2BaKqJMN0FVtzpV4hom4Y/9vIPyDpjRjNuKzrkYW1M8PxBxn+EJUPMLhr3jFMCo6HSxHoFXy8BzZoFkZWqap3w== JCL@IT-MINI-JCL.local", "key_options": null, "keyfile": "/home/jelopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jelopez", "validate_certs": true} 2021-02-10T11:04:50.535 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdbJDYY43Zu0EKAGgrSKEobyXIxdhT0AeTT3yBoNY+YdYDScimJsQzK4hKNJIn1xC3klr0dSnsqeJru+SA4GMhbdcsmVyudhutCw75R/vFoRHOwaeVS2SMiBAtDbAYGQ3dh6LapkmXeLURcycNbebUV4HM9XsYqnuL8J2XyY4tRKgHrAd6OQw4jsXDOxwTRlHM822N/2kgv8iEwiXAJoUoymbr8mnfSxuuaH0oiKBwuaK1wuaDrlJyXkXNslPq89/1f8hJfjixY2dQdW7Xk5m3t5ptQxWENpw79OMRJ4cDW4G7LrtFONom5KQ+9TBZJIhNgJobkqI4qwinrWF8qMW5 dcain@fullsteam", "key_options": null, "keyfile": "/home/dcain/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dcain", "validate_certs": true} 2021-02-10T11:04:50.705 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdbJDYY43Zu0EKAGgrSKEobyXIxdhT0AeTT3yBoNY+YdYDScimJsQzK4hKNJIn1xC3klr0dSnsqeJru+SA4GMhbdcsmVyudhutCw75R/vFoRHOwaeVS2SMiBAtDbAYGQ3dh6LapkmXeLURcycNbebUV4HM9XsYqnuL8J2XyY4tRKgHrAd6OQw4jsXDOxwTRlHM822N/2kgv8iEwiXAJoUoymbr8mnfSxuuaH0oiKBwuaK1wuaDrlJyXkXNslPq89/1f8hJfjixY2dQdW7Xk5m3t5ptQxWENpw79OMRJ4cDW4G7LrtFONom5KQ+9TBZJIhNgJobkqI4qwinrWF8qMW5 dcain@fullsteam", "key_options": null, "keyfile": "/home/dcain/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dcain", "validate_certs": true} 2021-02-10T11:04:51.008 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDs2+6XwRPdiDHT+63Yab00VBulg96/OLEORtNUvm7TDRMpcx4OD2oOnbsE4rG3dOdXNgT17H15c6MWFX2a2uC2h1j4hJTa+dG567HrYXRHG60Kwg5Sviiuq/ETPoH/r3OSvHs0HJ4Y9qYPQ5j0k7gG1XgIY3QHDWNRsXiyRXP9FEfoaOKABDTUoLPjqRX+4ExJd1Nz6DLxqIlre8wfqF5L1l12vec8JU3HB5NSRa87HxKFtLLIAtw7B9H/HggqX5wWVEnXq2qN4o6OOePhaO5cjWAZxOtT+AJTyHkeyaWLwIGOHBVZS5BKti6jLc+PxQxAflYlu9X3xh/2FlA+9H+ABZwMHE3ruUfTVbviYBbtccMCvkzcvFgnvaUCHjd7Iv/T8/JHNfBgFgigcX0Vei0+PZQ5oC5b7YpKT241NA8DCGZNgr1KuL9LazkvbkbpDcaHUOrkWsmSW1RRMC6ffYkbAI9QiBJFpImDAWP1u+1yCGhzPKBfGNt91GcLCD6ML90= sunny@devBox", "key_options": null, "keyfile": "/home/sunny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunny", "validate_certs": true} 2021-02-10T11:04:51.149 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDs2+6XwRPdiDHT+63Yab00VBulg96/OLEORtNUvm7TDRMpcx4OD2oOnbsE4rG3dOdXNgT17H15c6MWFX2a2uC2h1j4hJTa+dG567HrYXRHG60Kwg5Sviiuq/ETPoH/r3OSvHs0HJ4Y9qYPQ5j0k7gG1XgIY3QHDWNRsXiyRXP9FEfoaOKABDTUoLPjqRX+4ExJd1Nz6DLxqIlre8wfqF5L1l12vec8JU3HB5NSRa87HxKFtLLIAtw7B9H/HggqX5wWVEnXq2qN4o6OOePhaO5cjWAZxOtT+AJTyHkeyaWLwIGOHBVZS5BKti6jLc+PxQxAflYlu9X3xh/2FlA+9H+ABZwMHE3ruUfTVbviYBbtccMCvkzcvFgnvaUCHjd7Iv/T8/JHNfBgFgigcX0Vei0+PZQ5oC5b7YpKT241NA8DCGZNgr1KuL9LazkvbkbpDcaHUOrkWsmSW1RRMC6ffYkbAI9QiBJFpImDAWP1u+1yCGhzPKBfGNt91GcLCD6ML90= sunny@devBox", "key_options": null, "keyfile": "/home/sunny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunny", "validate_certs": true} 2021-02-10T11:04:51.468 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4pIcNiZKVIfT9GxyTe/B8G2GyHKxrAle3iExEzLH5bBCKtBrottX1y66KBzQSsrAb0z9h+A+uxlswlY6VM/gNos9VRB6x4NRB4ADY1Bku+JsKs5aXwRB1zz2ueZhrFN9hJwMl1heLICx85ksHBdm0q+JPdUAV+/LBdr0H/IjWveV34WL3Awf7QikzyKhzUciMB2gN9PJXTW0bl8PP56y60ys++L3nMSRJ+15uPcWl8MS3D2vABII0Tu+3PsxzrTTNodElyQ6G/fwyFfWhdp7fkidjR7mDDx8NyWfCl2rKejXv9LnVC0wb3tFoQqvwdz6shN4E51EIEGXHcLUeRpBhL2pFOqIZ0qfnmLIQVZ88RKMTgo42CgtmS5UzxCLOyKOh9GxF7DDEajTinJFJJVbNJ5xF494H2geSnmP1o1AecGsGbCZ4ndhS9unSfIZLAyv1J3wZlAlMsvxXgY7yrh1srUiASVyKK8jba1IT+v3TXR8kQUCNsLKKbvuX/l0IXME= xuxuehan@bogon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBd4eSizc2zAkWDy3mFZUhELO2wXDgkVxDNM/Nw2zT1li3L+6YOjFfdT4NOxpkg5laLd54bdKb5GVRA6iKqAO+aqo53xHHJGC8OY5eJaUPjtpypgIWO4Qzbos2ZzsC+nX5jvi8O9nM+JkabSW5iFv3gGPTqHixFYkxK2NIBa/UtXFOhmb1wZ6CG5Q+WObCosBUEMWokC8ntCADfhi6qDsO2fol6mYUhYEE/0nhQ5uLRrVIM5QM5dhrQVEeAs4KcYpmPDw6q6AKgx/f5zObh5OSJFTVsgx2cqzdQVJVDDx2OD2QVRtZ9fgvI2EGmKSwe1TUQOdqa5bmsVIp2Jfi8MC1 xuxuehan@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwIokXSvPYP3TRlSw0SBgmMFW5jVltGf4c6CqjIW4KKCgsL4c4Vs0FztRyDGc9p0krZUoGJCZHErVToXm6/lSq8JlfrlLX4X4yh37fUifWqc2UTJGeH8kLaP3WEt9vT8eQ9h8t2GqlMlGQU8KzVAdC+c5IwrbGRsS6S7D09/y0rklkhdJ6nuj8cssN/2TzLF9YIvVcFZHdyznk84eB7TgpPKD88MOpTnEgWFlD4vAWEuHpkuxgj1KI9LscgSFcW9/RxLeRxcV0GypZ8jPmczoCnin/ZOstCTpbcmjLd 2021-02-10T11:04:51.470 INFO:teuthology.task.ansible.out:+LW1kgGXXQSkf09iHeJ/cyCw6ooZOkynVwwmI9Uz2ZaMedCUBpG0bKclIerJ9jn+CoTs3U3Orhd9+6MVtCHX9VNJbVa4q0KCS1sp5S3HDrYkgUyawUxdYorrLZcqfRHWWo1luiridmuQdr8lQ6dvNpz4rhsCrAG6S1eeaC/s+o0Rzpv1YxU64JzquZ6WNR+W3U6Bg3iYkSoDCK9byf0rsptyL8= luqixi@luqixideMacBook-Air.local", "key_options": null, "keyfile": "/home/xuxuehan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xuxuehan", "validate_certs": true} 2021-02-10T11:04:51.609 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4pIcNiZKVIfT9GxyTe/B8G2GyHKxrAle3iExEzLH5bBCKtBrottX1y66KBzQSsrAb0z9h+A+uxlswlY6VM/gNos9VRB6x4NRB4ADY1Bku+JsKs5aXwRB1zz2ueZhrFN9hJwMl1heLICx85ksHBdm0q+JPdUAV+/LBdr0H/IjWveV34WL3Awf7QikzyKhzUciMB2gN9PJXTW0bl8PP56y60ys++L3nMSRJ+15uPcWl8MS3D2vABII0Tu+3PsxzrTTNodElyQ6G/fwyFfWhdp7fkidjR7mDDx8NyWfCl2rKejXv9LnVC0wb3tFoQqvwdz6shN4E51EIEGXHcLUeRpBhL2pFOqIZ0qfnmLIQVZ88RKMTgo42CgtmS5UzxCLOyKOh9GxF7DDEajTinJFJJVbNJ5xF494H2geSnmP1o1AecGsGbCZ4ndhS9unSfIZLAyv1J3wZlAlMsvxXgY7yrh1srUiASVyKK8jba1IT+v3TXR8kQUCNsLKKbvuX/l0IXME= xuxuehan@bogon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBd4eSizc2zAkWDy3mFZUhELO2wXDgkVxDNM/Nw2zT1li3L+6YOjFfdT4NOxpkg5laLd54bdKb5GVRA6iKqAO+aqo53xHHJGC8OY5eJaUPjtpypgIWO4Qzbos2ZzsC+nX5jvi8O9nM+JkabSW5iFv3gGPTqHixFYkxK2NIBa/UtXFOhmb1wZ6CG5Q+WObCosBUEMWokC8ntCADfhi6qDsO2fol6mYUhYEE/0nhQ5uLRrVIM5QM5dhrQVEeAs4KcYpmPDw6q6AKgx/f5zObh5OSJFTVsgx2cqzdQVJVDDx2OD2QVRtZ9fgvI2EGmKSwe1TUQOdqa5bmsVIp2Jfi8MC1 xuxuehan@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwIokXSvPYP3TRlSw0SBgmMFW5jVltGf4c6CqjIW4KKCgsL4c4Vs0FztRyDGc9p0krZUoGJCZHErVToXm6/lSq8JlfrlLX4X4yh37fUifWqc2UTJGeH8kLaP3WEt9vT8eQ9h8t2GqlMlGQU8KzVAdC+c5IwrbGRsS6S7D09/y0rklkhdJ6nuj8cssN/2TzLF9YIvVcFZHdyznk84eB7TgpPKD88MOpTnEgWFlD4vAWEuHpkuxgj1KI9LscgSFcW9/RxLeRxcV0GypZ8jPmczoCnin/ZOstCTpbcmjLd 2021-02-10T11:04:51.610 INFO:teuthology.task.ansible.out:+LW1kgGXXQSkf09iHeJ/cyCw6ooZOkynVwwmI9Uz2ZaMedCUBpG0bKclIerJ9jn+CoTs3U3Orhd9+6MVtCHX9VNJbVa4q0KCS1sp5S3HDrYkgUyawUxdYorrLZcqfRHWWo1luiridmuQdr8lQ6dvNpz4rhsCrAG6S1eeaC/s+o0Rzpv1YxU64JzquZ6WNR+W3U6Bg3iYkSoDCK9byf0rsptyL8= luqixi@luqixideMacBook-Air.local", "key_options": null, "keyfile": "/home/xuxuehan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xuxuehan", "validate_certs": true} 2021-02-10T11:04:51.917 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDMOuxxTZyxz/TEVA+ODgtLrNcSsUhKE64BZg/Ub96LLffQXQc9SYT/0YI4wxu2vSsky05PnVSON7jqGxunCt0+AdRqxXhW4ttmRIOz8rq8SiMnnLayV34nIF1G0aYTkgY4ej4+VmSVIHU3bb5mp/Q3D0hxQwDPdgOJWNyW9oYerHbnx6lVgchvsZHB9VO09GNHcAsyUz4p+xBf9iR7eS74p4dZmUbth/oBBqpOa4NgNr69iyZ9a/pNDLxaVQoIpDFrevk5UU5nUHQq2frFNxF/Y2Ja1EBXCavvPF/m5v3J3xNEu3SI/GbLFCPKCKeBDpXJbe1QqTBKJjMl9qGmKEJ3 aclewett@mac", "key_options": null, "keyfile": "/home/aclewett/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aclewett", "validate_certs": true} 2021-02-10T11:04:52.056 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDMOuxxTZyxz/TEVA+ODgtLrNcSsUhKE64BZg/Ub96LLffQXQc9SYT/0YI4wxu2vSsky05PnVSON7jqGxunCt0+AdRqxXhW4ttmRIOz8rq8SiMnnLayV34nIF1G0aYTkgY4ej4+VmSVIHU3bb5mp/Q3D0hxQwDPdgOJWNyW9oYerHbnx6lVgchvsZHB9VO09GNHcAsyUz4p+xBf9iR7eS74p4dZmUbth/oBBqpOa4NgNr69iyZ9a/pNDLxaVQoIpDFrevk5UU5nUHQq2frFNxF/Y2Ja1EBXCavvPF/m5v3J3xNEu3SI/GbLFCPKCKeBDpXJbe1QqTBKJjMl9qGmKEJ3 aclewett@mac", "key_options": null, "keyfile": "/home/aclewett/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aclewett", "validate_certs": true} 2021-02-10T11:04:52.361 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdTLDprr3DXtsgiqrB/miglJwSkY7BAv+OW9sC0ZHQf8Y4dFoUtr7q9fd6W7ehKL6aMjya/8eQSMPRoEIZFD8f2KPCuE74Wgwgv2cpAkFmLjcPtjixqLo3LGFCyEJWGLQxjlthdzx59mpCWgBsEJQmlKP36JIYjILIMtXtsETIUGFa8QPIn9sMdIgutPPf3yB/6DHcR7C6yIAuvbJactEsiCt4C8+uJinvCii3PzYOMAEd+mictPGk8nOCQFZrI+ZmwTz4UbJylxFqnX8lhhoPIUEfNAXlb3lkZWH1LaFOxarbmupuPYcg2rT0+kFtyNr3/mAx0FO01jADYooqcW1Z cardno:000604936806", "key_options": null, "keyfile": "/home/cblum/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cblum", "validate_certs": true} 2021-02-10T11:04:52.498 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdTLDprr3DXtsgiqrB/miglJwSkY7BAv+OW9sC0ZHQf8Y4dFoUtr7q9fd6W7ehKL6aMjya/8eQSMPRoEIZFD8f2KPCuE74Wgwgv2cpAkFmLjcPtjixqLo3LGFCyEJWGLQxjlthdzx59mpCWgBsEJQmlKP36JIYjILIMtXtsETIUGFa8QPIn9sMdIgutPPf3yB/6DHcR7C6yIAuvbJactEsiCt4C8+uJinvCii3PzYOMAEd+mictPGk8nOCQFZrI+ZmwTz4UbJylxFqnX8lhhoPIUEfNAXlb3lkZWH1LaFOxarbmupuPYcg2rT0+kFtyNr3/mAx0FO01jADYooqcW1Z cardno:000604936806", "key_options": null, "keyfile": "/home/cblum/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cblum", "validate_certs": true} 2021-02-10T11:04:52.811 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDPUCIVfNQOOG+uAeKWo7tLAdgGO/oQu4exTE3XLfyTVdd8u6pbQCmIDFEnbGgMtlYuWJlCJtxJIzEZJ5HIwf+po3y7kb7HxEeWnEb9DsqJB4nCRE8fHnNLgF3bh9gvE7sOZdNXEKqtsnGcaaS81oXQ6QgkFGQzlOBOqxj7jIvad7AtNWD3WbJi/uTVDsWcukU51tn/aEW+hHMD7Y5QJkLH7O0PrCeEZCUCWoiMP7sNwACBym0DsqX+POkGJOaGx9EeNhGVA4B6o3xYXJPqmFbgMWMfG3/Y3SiheZjxzySMQ7FKW4Q817NaWqbjtLZQsSZw7JNqNFkcdTzVZwwCOJ1AAecajpniKeQLnyiWQ3Z/EkSGGwtDj67wo7uqT85tpOyLfYhQm6Zp+xSh+I6s0YJ0xO/huLKzeT5D1r5vVMk2VdcLD3CCxXUwZgaMUxYAC2Wq6mpIkxVZFQLbGuA/dWkQbFsrpbf7ES+kh+uzp4R6dQ29OjXbVcSfF2vyIuA+VUKoyj370YOZQhs50yuAJ4BSHlEgoJdRzi3XUX48h45XHiZcCY3X8LTX6a+0KAIgqeAWdFhRgQtAxoRYjhOxC3k/boBc8sEQ3o4pkxnhtE5Y/tnM2CPkPMX9HWYirNf9GyZTYcR/5rDf4ETQ18G68WLaNK758OuUPjSkwttcyLd4dw== mlatimer@suse.com", "key_options": null, "keyfile": "/home/mlatimer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mlatimer", "validate_certs": true} 2021-02-10T11:04:52.936 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDPUCIVfNQOOG+uAeKWo7tLAdgGO/oQu4exTE3XLfyTVdd8u6pbQCmIDFEnbGgMtlYuWJlCJtxJIzEZJ5HIwf+po3y7kb7HxEeWnEb9DsqJB4nCRE8fHnNLgF3bh9gvE7sOZdNXEKqtsnGcaaS81oXQ6QgkFGQzlOBOqxj7jIvad7AtNWD3WbJi/uTVDsWcukU51tn/aEW+hHMD7Y5QJkLH7O0PrCeEZCUCWoiMP7sNwACBym0DsqX+POkGJOaGx9EeNhGVA4B6o3xYXJPqmFbgMWMfG3/Y3SiheZjxzySMQ7FKW4Q817NaWqbjtLZQsSZw7JNqNFkcdTzVZwwCOJ1AAecajpniKeQLnyiWQ3Z/EkSGGwtDj67wo7uqT85tpOyLfYhQm6Zp+xSh+I6s0YJ0xO/huLKzeT5D1r5vVMk2VdcLD3CCxXUwZgaMUxYAC2Wq6mpIkxVZFQLbGuA/dWkQbFsrpbf7ES+kh+uzp4R6dQ29OjXbVcSfF2vyIuA+VUKoyj370YOZQhs50yuAJ4BSHlEgoJdRzi3XUX48h45XHiZcCY3X8LTX6a+0KAIgqeAWdFhRgQtAxoRYjhOxC3k/boBc8sEQ3o4pkxnhtE5Y/tnM2CPkPMX9HWYirNf9GyZTYcR/5rDf4ETQ18G68WLaNK758OuUPjSkwttcyLd4dw== mlatimer@suse.com", "key_options": null, "keyfile": "/home/mlatimer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mlatimer", "validate_certs": true} 2021-02-10T11:04:53.250 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCYh28a9R3a7jpRfPqVAL9EcD+S6KkZzKTfCmRn4s8Fg39kUZsgnWUZQ3jUxKSeBA00te7WMR8CsIEzulm7f+/egNuPcvzNI/SaaCDtVMaxqlEuW85ll8uBQi3s4SLmOu9oS+5oid4cqTrGTcG6uhl29oWS/53SxoGh+T+y6GesO2cYRuTpcUHxGMvIgZe/bKtVGdP1y7rQd62e8emb9UupMBxaHpkYj9AaMuLjovHbBgecWbe4PlzPWNzl+eITVk2d6CxxIem4/bw9VHkRjQZlhL2nbwRgtczLYhlaZAd4ePEtsqA+7nTDjTDhXXHVIf1pDAsJbZEdgQCSGH3FzxtrQ2hZJrH/NIKdHk/OtI06e7POSGj7PcHzTvMjQSFnTpJXinxwzn0lvFSlmABN/00lIE0p2ZXfPs6dgWJ17JnTq6dMUoTiGhfKphPHyfSXJCzfa2VB+lQQ5eAaa9cRXy9pLmWGHXVGz2hsBMtyYmTjR/wr8kdTzeK+Jk4opBtGgcxDustg59b9x/jvgoc3HCdtnkqLTug2KmutB6kb+4KnVO07GsSp0RmNtDmCRcXs9YN/eia9ICtIGDWPc3YDRppsw1Vktza/+ZW5vGzIKILZTo4SNGcg19zQw4Qxd9eLkXfB66vE2CQvA90CGo7mcye+UWjjXLr7i0cDEszAloQ9aQ== srangana@redhat.com", "key_options": null, "keyfile": "/home/shyamsundarr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyamsundarr", "validate_certs": true} 2021-02-10T11:04:53.377 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCYh28a9R3a7jpRfPqVAL9EcD+S6KkZzKTfCmRn4s8Fg39kUZsgnWUZQ3jUxKSeBA00te7WMR8CsIEzulm7f+/egNuPcvzNI/SaaCDtVMaxqlEuW85ll8uBQi3s4SLmOu9oS+5oid4cqTrGTcG6uhl29oWS/53SxoGh+T+y6GesO2cYRuTpcUHxGMvIgZe/bKtVGdP1y7rQd62e8emb9UupMBxaHpkYj9AaMuLjovHbBgecWbe4PlzPWNzl+eITVk2d6CxxIem4/bw9VHkRjQZlhL2nbwRgtczLYhlaZAd4ePEtsqA+7nTDjTDhXXHVIf1pDAsJbZEdgQCSGH3FzxtrQ2hZJrH/NIKdHk/OtI06e7POSGj7PcHzTvMjQSFnTpJXinxwzn0lvFSlmABN/00lIE0p2ZXfPs6dgWJ17JnTq6dMUoTiGhfKphPHyfSXJCzfa2VB+lQQ5eAaa9cRXy9pLmWGHXVGz2hsBMtyYmTjR/wr8kdTzeK+Jk4opBtGgcxDustg59b9x/jvgoc3HCdtnkqLTug2KmutB6kb+4KnVO07GsSp0RmNtDmCRcXs9YN/eia9ICtIGDWPc3YDRppsw1Vktza/+ZW5vGzIKILZTo4SNGcg19zQw4Qxd9eLkXfB66vE2CQvA90CGo7mcye+UWjjXLr7i0cDEszAloQ9aQ== srangana@redhat.com", "key_options": null, "keyfile": "/home/shyamsundarr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyamsundarr", "validate_certs": true} 2021-02-10T11:04:53.691 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRZyOnj6D9tCw5o4+ujpos5Uj1O0XVvea798DsXPRh8bEE1pLkARQJLv4iW6LsG3x/CXruahmwT+6C/VHppIsbgtpadMWdCTPYrsjctPK0/N9nr4kQEL6XeK3/GHFnwqYcc+LySpNNfbKrgN4qWOwdZBhQjSV7K7FJd4ZYSm4wp6TVbvKKp7o+JdeM2zhstfC9AcCtBv/TpxixFFq/+y8/Z0PwqN1YyHduDOr1dr1DWn4ZzrkhiTYfPOltx0jweD6zTJoTAJDqae7kBA+3XH3dUGX3n7Tchsa3bnIKVz9sLQvH/vPro61I+Xig6MQh77JsylNvpwb1LLOEzDf7dxl bigwhite@wrjj37deMacBook-Pro-2.local", "key_options": null, "keyfile": "/home/shuzhenyi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shuzhenyi", "validate_certs": true} 2021-02-10T11:04:53.839 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRZyOnj6D9tCw5o4+ujpos5Uj1O0XVvea798DsXPRh8bEE1pLkARQJLv4iW6LsG3x/CXruahmwT+6C/VHppIsbgtpadMWdCTPYrsjctPK0/N9nr4kQEL6XeK3/GHFnwqYcc+LySpNNfbKrgN4qWOwdZBhQjSV7K7FJd4ZYSm4wp6TVbvKKp7o+JdeM2zhstfC9AcCtBv/TpxixFFq/+y8/Z0PwqN1YyHduDOr1dr1DWn4ZzrkhiTYfPOltx0jweD6zTJoTAJDqae7kBA+3XH3dUGX3n7Tchsa3bnIKVz9sLQvH/vPro61I+Xig6MQh77JsylNvpwb1LLOEzDf7dxl bigwhite@wrjj37deMacBook-Pro-2.local", "key_options": null, "keyfile": "/home/shuzhenyi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shuzhenyi", "validate_certs": true} 2021-02-10T11:04:54.135 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEcI3X9mNxtHRSvi2gVFavu21kRCWDd1erXW3d/1zWnoUvRxe9ft2xA1c3Bmo067GqhFudPcXoYJvz3atAGR9k3n1UTtTJe95XRZ1Zmn/boi6mn7I2oRVFpW/KjxaiM3iN2SHKknMtcs6qRTioapYCU7u924oc0izqPBvkrC73cfVF4emH1ERWEcQBTxWub6N6Mn3xaDmJ5xnGlNvDY70CC9osoa2VRNJM7GU9eUQClHirPbSooDvvVKfTGD7YfMH76j6PdDR/a/cgNeiMevtMbeFGhu1SOEHb1ihCNoGq2vQFPPWwwTCzMI05XuUEc4iaxw/kbGBMpWbmIvswIMQrRwgwLZk67GGpf/dEeutI6WseeNKNuHnTzzx43AW9G2coGq0RKE7OuA5Cqj0CO7Kcf5UA4AqIIroLETVBrrEV6qzpkD74QBZXPl7gbSZIyxLpvtQoMfSSzN+Mj184LmQoURCqIT+yM5gddF5/S5I+rpw81vBj5M9ZskyzVQAo0H8= michelledipalma@mdipalma-mac", "key_options": null, "keyfile": "/home/mdipalma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mdipalma", "validate_certs": true} 2021-02-10T11:04:54.293 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEcI3X9mNxtHRSvi2gVFavu21kRCWDd1erXW3d/1zWnoUvRxe9ft2xA1c3Bmo067GqhFudPcXoYJvz3atAGR9k3n1UTtTJe95XRZ1Zmn/boi6mn7I2oRVFpW/KjxaiM3iN2SHKknMtcs6qRTioapYCU7u924oc0izqPBvkrC73cfVF4emH1ERWEcQBTxWub6N6Mn3xaDmJ5xnGlNvDY70CC9osoa2VRNJM7GU9eUQClHirPbSooDvvVKfTGD7YfMH76j6PdDR/a/cgNeiMevtMbeFGhu1SOEHb1ihCNoGq2vQFPPWwwTCzMI05XuUEc4iaxw/kbGBMpWbmIvswIMQrRwgwLZk67GGpf/dEeutI6WseeNKNuHnTzzx43AW9G2coGq0RKE7OuA5Cqj0CO7Kcf5UA4AqIIroLETVBrrEV6qzpkD74QBZXPl7gbSZIyxLpvtQoMfSSzN+Mj184LmQoURCqIT+yM5gddF5/S5I+rpw81vBj5M9ZskyzVQAo0H8= michelledipalma@mdipalma-mac", "key_options": null, "keyfile": "/home/mdipalma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mdipalma", "validate_certs": true} 2021-02-10T11:04:54.578 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+tqgRGdlbb7NMefZVPCtbzVZBapxmmPcKaUtqT7E/9AKRoHuroZQ7KJQfO15POpIfEja7BugBt5F95aW2Bd9/xhulzC4/JjiZYc6ST5Na03BrcudI3YcktoGR8Mm7ZVzSojvm0SKtLLHM5Su17JpCC7BitMdao1mETlUOklEXc6pgbr9XHOayo1zeuos8DB90iA8TQ2soR/POD1Rix4GKniCHXz10+iXQFYPsif1HsQ3rh8Vi1M0cTq98bMSmfsKODil8KIxf+lvLCL4GAgDYEyxnstiTFh5GAcIMOOo2dl1fDb/Oa981CxWU+xxJD+M7AEJVeyAqqAnShvZU6y1L kiefer@kiefer-uchi", "key_options": null, "keyfile": "/home/kchang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchang", "validate_certs": true} 2021-02-10T11:04:54.746 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+tqgRGdlbb7NMefZVPCtbzVZBapxmmPcKaUtqT7E/9AKRoHuroZQ7KJQfO15POpIfEja7BugBt5F95aW2Bd9/xhulzC4/JjiZYc6ST5Na03BrcudI3YcktoGR8Mm7ZVzSojvm0SKtLLHM5Su17JpCC7BitMdao1mETlUOklEXc6pgbr9XHOayo1zeuos8DB90iA8TQ2soR/POD1Rix4GKniCHXz10+iXQFYPsif1HsQ3rh8Vi1M0cTq98bMSmfsKODil8KIxf+lvLCL4GAgDYEyxnstiTFh5GAcIMOOo2dl1fDb/Oa981CxWU+xxJD+M7AEJVeyAqqAnShvZU6y1L kiefer@kiefer-uchi", "key_options": null, "keyfile": "/home/kchang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchang", "validate_certs": true} 2021-02-10T11:04:55.031 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDjj3bbbLvd4IHeFrSJyM9N6uKXA7ElQrzfcisKRGjUPON8IsAWkXQeAnHoBsIQv7tRGudXOrK+tbfv7crLI0/CxL+o0cCVW8bxuTEtdz07ZFsND9cppCc7Yq+wHsg7nQ/n0QfxnlYc0T9pATXUoT+aW4c3NQoDbsupLSkNs1C+G/MEBzz6iH6tgrv43OAOUccfQbC+IYx4RIKL5MteEjVrZanNELIG+jsYe+CZLdsONk1nR8+xKQw19Sm/jkKdl1K9NXoVPEBQmFXrYl6G+7tCdWeU3Vq24lM1X5p1mVspcG6FyyctcuP/8+zhL2L208DrXlOVPcqHR/4FEf3Wmk89 vereddy@vereddy.remote.csb", "key_options": null, "keyfile": "/home/vereddy/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vereddy", "validate_certs": true} 2021-02-10T11:04:55.190 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDjj3bbbLvd4IHeFrSJyM9N6uKXA7ElQrzfcisKRGjUPON8IsAWkXQeAnHoBsIQv7tRGudXOrK+tbfv7crLI0/CxL+o0cCVW8bxuTEtdz07ZFsND9cppCc7Yq+wHsg7nQ/n0QfxnlYc0T9pATXUoT+aW4c3NQoDbsupLSkNs1C+G/MEBzz6iH6tgrv43OAOUccfQbC+IYx4RIKL5MteEjVrZanNELIG+jsYe+CZLdsONk1nR8+xKQw19Sm/jkKdl1K9NXoVPEBQmFXrYl6G+7tCdWeU3Vq24lM1X5p1mVspcG6FyyctcuP/8+zhL2L208DrXlOVPcqHR/4FEf3Wmk89 vereddy@vereddy.remote.csb", "key_options": null, "keyfile": "/home/vereddy/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vereddy", "validate_certs": true} 2021-02-10T11:04:55.483 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+EkUbnaeC7Dqtq2N4/dX9v+WGlrgRS2ygHhlbZy4KHbd6+Q9T5NAWtqnFwqcbFpYYUBwMvz8zLo6HYN6gNskRN0qTiFKPQdtnydF0Tx1KHrdGGcX3G2lxyGH5P8nrqReXAPPvRiVc2/Mj7APCTmY2JftkSS1qYcTleeueivVyho6z6uD1AyC7OXANhtn7kGeIY/NvVKBhDr+3TuPmpPwPCQMcJUyZWCHYyp47v2vzNBy2Etp7WjGwOnfZ5ULiJGjH6PCXx0YfpmTUOPVkNbV6Om+0YJ3fcLlmQ/2nsdsWfOKvoAr+1uzO81h7ZBX9LVEitj8YU3WeSCV2vqS2VvOpq5NyHXWGq+ZkOCfhy4Z9MvMG+6hqp4p4z1JMpKsZ191kP7eAQua41GkTSQPlEfF+um1UV6ZjNzhB68utOXyGDoDcxct9oOdnhhjwu5W4/0t9iBHuMn02iHRYVrUWOH2FXVysR0p5x/L2xdvNPiTgFAZ2eN+EFWmdan4e9mAAw3M= gbenhano@localhost.localdomain", "key_options": null, "keyfile": "/home/benhanokh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "benhanokh", "validate_certs": true} 2021-02-10T11:04:55.631 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+EkUbnaeC7Dqtq2N4/dX9v+WGlrgRS2ygHhlbZy4KHbd6+Q9T5NAWtqnFwqcbFpYYUBwMvz8zLo6HYN6gNskRN0qTiFKPQdtnydF0Tx1KHrdGGcX3G2lxyGH5P8nrqReXAPPvRiVc2/Mj7APCTmY2JftkSS1qYcTleeueivVyho6z6uD1AyC7OXANhtn7kGeIY/NvVKBhDr+3TuPmpPwPCQMcJUyZWCHYyp47v2vzNBy2Etp7WjGwOnfZ5ULiJGjH6PCXx0YfpmTUOPVkNbV6Om+0YJ3fcLlmQ/2nsdsWfOKvoAr+1uzO81h7ZBX9LVEitj8YU3WeSCV2vqS2VvOpq5NyHXWGq+ZkOCfhy4Z9MvMG+6hqp4p4z1JMpKsZ191kP7eAQua41GkTSQPlEfF+um1UV6ZjNzhB68utOXyGDoDcxct9oOdnhhjwu5W4/0t9iBHuMn02iHRYVrUWOH2FXVysR0p5x/L2xdvNPiTgFAZ2eN+EFWmdan4e9mAAw3M= gbenhano@localhost.localdomain", "key_options": null, "keyfile": "/home/benhanokh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "benhanokh", "validate_certs": true} 2021-02-10T11:04:55.925 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBYtdHKBXQcXQFkEW8fSwlFK/xZM16VcFY288ABI/DPfejTStHhxiRvmlsWzMy0Ar5WadZ8FU1ptjA3MKBEY1Do1r1Ul369k3zHCvxXyg9QGGMNPYfHmnNeoF4GEgoJpuounPs0mkLmrPTR+cR+/XuaZZt38nsad8M/zJetnolvqCWpcV87D7+Rt8F32J6kGDknC4ikEe8I0n7F+goOmJhlgkqegbKEeJ/EMnOItwVZHFwaC+sxOq4IlYp2f1aZMKGAtr5+ICwYH1SMCcvnp3qWL8O//t2tvguZgQxzIs6Nf5GaxwdQkh4CvjbWiCkIbGFJ57YICS6k++KMXMbEnwh jsalomon@Josh-laptop.tlv.redhat.com", "key_options": null, "keyfile": "/home/jsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jsalomon", "validate_certs": true} 2021-02-10T11:04:56.072 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBYtdHKBXQcXQFkEW8fSwlFK/xZM16VcFY288ABI/DPfejTStHhxiRvmlsWzMy0Ar5WadZ8FU1ptjA3MKBEY1Do1r1Ul369k3zHCvxXyg9QGGMNPYfHmnNeoF4GEgoJpuounPs0mkLmrPTR+cR+/XuaZZt38nsad8M/zJetnolvqCWpcV87D7+Rt8F32J6kGDknC4ikEe8I0n7F+goOmJhlgkqegbKEeJ/EMnOItwVZHFwaC+sxOq4IlYp2f1aZMKGAtr5+ICwYH1SMCcvnp3qWL8O//t2tvguZgQxzIs6Nf5GaxwdQkh4CvjbWiCkIbGFJ57YICS6k++KMXMbEnwh jsalomon@Josh-laptop.tlv.redhat.com", "key_options": null, "keyfile": "/home/jsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jsalomon", "validate_certs": true} 2021-02-10T11:04:56.365 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9ltUNitnn0FNwMeiGkBnozpBN5ziVR2MreaqkfqTQY2TgI24559xuILyy+TnAKumhemZ8i8nIQ876Iso85OzIXbUbRM45hpL/bMQX7pWyMIRNeFKrCDwf18OT0DPc8aeDOAFQ6YoN0j4awUENjdZuuAW80cfcjoQO3m7ORF8wBF0IvMWwPDIQQrz+Li31ECgXg6VC7K3e+BNHlfNQ/2/ahS0+V4J6JYFsylZL0r2tydhDMd6rR8QA9c9BgeBz7+KWobGtiZVe7KReTzJ0O38PxNd8br3aS50j5BiN63gch8eFxPAROaUMnuiavu6OTDy4/aeL3Zc5WpUy3VyyumWf jolmomar@localhost.localdomain", "key_options": null, "keyfile": "/home/jolmomar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jolmomar", "validate_certs": true} 2021-02-10T11:04:56.512 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9ltUNitnn0FNwMeiGkBnozpBN5ziVR2MreaqkfqTQY2TgI24559xuILyy+TnAKumhemZ8i8nIQ876Iso85OzIXbUbRM45hpL/bMQX7pWyMIRNeFKrCDwf18OT0DPc8aeDOAFQ6YoN0j4awUENjdZuuAW80cfcjoQO3m7ORF8wBF0IvMWwPDIQQrz+Li31ECgXg6VC7K3e+BNHlfNQ/2/ahS0+V4J6JYFsylZL0r2tydhDMd6rR8QA9c9BgeBz7+KWobGtiZVe7KReTzJ0O38PxNd8br3aS50j5BiN63gch8eFxPAROaUMnuiavu6OTDy4/aeL3Zc5WpUy3VyyumWf jolmomar@localhost.localdomain", "key_options": null, "keyfile": "/home/jolmomar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jolmomar", "validate_certs": true} 2021-02-10T11:04:56.814 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQKiUAlB8dxudAY60M7jnIJvs9yb2pNaT+BVv1Vk1kMs0XjPjcw9cyT3DtQ1xGLF367phRi/cfQbXOYbHG+uyBzQ4EEnBFvziktWRx4mfeWYxKjkjbfkVfpKPsxztqCbwm6+NYzMxGK2cExN7jUHnghHAjfA0IUpVVg9peBD3kUveLwS+UxuJAnuj+G4480LS6GDyLrhFEXU7uOIIlW4S9oLnU6vJL6GAzwyLE4ijTA8Ek03SJ01Ey/gpU9AqnQNp9u12l/7sJely6sI9pAuIzmS8Ht324hexxSWN5sFjXgLvLfFK83ziDBdrFBr0WXnWsFdRpHs2CwSq1JTkByqbN hmunjulu@hmunjulu.remote.csb", "key_options": null, "keyfile": "/home/hmunjulu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hmunjulu", "validate_certs": true} 2021-02-10T11:04:56.971 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQKiUAlB8dxudAY60M7jnIJvs9yb2pNaT+BVv1Vk1kMs0XjPjcw9cyT3DtQ1xGLF367phRi/cfQbXOYbHG+uyBzQ4EEnBFvziktWRx4mfeWYxKjkjbfkVfpKPsxztqCbwm6+NYzMxGK2cExN7jUHnghHAjfA0IUpVVg9peBD3kUveLwS+UxuJAnuj+G4480LS6GDyLrhFEXU7uOIIlW4S9oLnU6vJL6GAzwyLE4ijTA8Ek03SJ01Ey/gpU9AqnQNp9u12l/7sJely6sI9pAuIzmS8Ht324hexxSWN5sFjXgLvLfFK83ziDBdrFBr0WXnWsFdRpHs2CwSq1JTkByqbN hmunjulu@hmunjulu.remote.csb", "key_options": null, "keyfile": "/home/hmunjulu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hmunjulu", "validate_certs": true} 2021-02-10T11:04:57.255 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoefxOfeArToGucfWG7y1tyLF2I1J/IkBSlmxxEJyHyDUB6ElXjYIJReoiPfmB0xLEmINpmt8QA4XN/8UbR1r9LFOexlZZbX1LSbihi11LXCvfIJWqkHVwIyKHRTu/duMrntBFVENJ0LVf0GSWVTtllXaDA+fMkp6/ra/hlRHQbx19UXy/9+Pogj8Xl+SntnMGiHlX8FFq2A2tUqhdQSf/1GW1HfUIa5WtwdEqIpKpAgMM7sOW9i3QAmLAMo8HiKrArSiy7jrRUODCqsTJsfifxEbmnOS0ZtmDh0eeWL/Dyqbz99Rg4WbFNeK8oyZYr7D6bEAhd2az2A+wycPx4Nq7 julpark@julpark.remote.csb", "key_options": null, "keyfile": "/home/julpark/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "julpark", "validate_certs": true} 2021-02-10T11:04:57.413 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoefxOfeArToGucfWG7y1tyLF2I1J/IkBSlmxxEJyHyDUB6ElXjYIJReoiPfmB0xLEmINpmt8QA4XN/8UbR1r9LFOexlZZbX1LSbihi11LXCvfIJWqkHVwIyKHRTu/duMrntBFVENJ0LVf0GSWVTtllXaDA+fMkp6/ra/hlRHQbx19UXy/9+Pogj8Xl+SntnMGiHlX8FFq2A2tUqhdQSf/1GW1HfUIa5WtwdEqIpKpAgMM7sOW9i3QAmLAMo8HiKrArSiy7jrRUODCqsTJsfifxEbmnOS0ZtmDh0eeWL/Dyqbz99Rg4WbFNeK8oyZYr7D6bEAhd2az2A+wycPx4Nq7 julpark@julpark.remote.csb", "key_options": null, "keyfile": "/home/julpark/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "julpark", "validate_certs": true} 2021-02-10T11:04:57.704 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD03otezqmbR7v9dq62mWH5Kkg692jdmDZk6Ve4vux0A0YhxBuXDCyBWLE1Z4lLNxQahvK0pGsgokp+Uk+Q6aU+ZnF/E2jLp6YWerFXIHY3M7byObJx9+QuVBfNI0bZBkV3+vSjhSCGOYU+TrTHsKZqhsTcejZjmL3gFLmWfU6dTh0sX05qz97cdJl3CMPhCIR3KjtkiPygj417/N7wet/Efv1t79TvyS7dtK1oz+zRLEpKi/fZjvpXb8xec7as72Y0XexsWMDyZJ/N8PijTQ5L6EfLMV7ae/bWGzNAKsoC1dHVY7q/Q4pzktC6zU0Rc9sDRMtrgFg+q+04BfveSLfxTb9tqf7SbSvh7IAG1eKiEuDT/uy3T6Uiw+CiZ0eIqYzZscO/aDjUArGbS8eyaycybdYCBK9Jz38i0mYKkXg855yoVuIExr6MXieYwyVoVW8gbx1bAslaafrzra2dkAFByOJ0sbosPFRgR2+boUSxD6IaE+ALgno4T/0CI2RMiH0= prashant@cephdev", "key_options": null, "keyfile": "/home/pdhange/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdhange", "validate_certs": true} 2021-02-10T11:04:57.851 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD03otezqmbR7v9dq62mWH5Kkg692jdmDZk6Ve4vux0A0YhxBuXDCyBWLE1Z4lLNxQahvK0pGsgokp+Uk+Q6aU+ZnF/E2jLp6YWerFXIHY3M7byObJx9+QuVBfNI0bZBkV3+vSjhSCGOYU+TrTHsKZqhsTcejZjmL3gFLmWfU6dTh0sX05qz97cdJl3CMPhCIR3KjtkiPygj417/N7wet/Efv1t79TvyS7dtK1oz+zRLEpKi/fZjvpXb8xec7as72Y0XexsWMDyZJ/N8PijTQ5L6EfLMV7ae/bWGzNAKsoC1dHVY7q/Q4pzktC6zU0Rc9sDRMtrgFg+q+04BfveSLfxTb9tqf7SbSvh7IAG1eKiEuDT/uy3T6Uiw+CiZ0eIqYzZscO/aDjUArGbS8eyaycybdYCBK9Jz38i0mYKkXg855yoVuIExr6MXieYwyVoVW8gbx1bAslaafrzra2dkAFByOJ0sbosPFRgR2+boUSxD6IaE+ALgno4T/0CI2RMiH0= prashant@cephdev", "key_options": null, "keyfile": "/home/pdhange/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdhange", "validate_certs": true} 2021-02-10T11:04:58.147 INFO:teuthology.task.ansible.out:changed: [gibba012.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDg7xW5+Mk7pG5MjLxOc7GTvDTCPBJj2L3MEQZIapmmo6t8uPODwDewXncSzrKDH6zFVE4TTTD9fJdu+BhyT+jCFgXrEOMMabj7vJsg0wO0SWqr+4NECFPhD9y4ewfYiewBeXisclA5Bto91v93AtYpZBCdmPYSi0BEcPVc6S3iBUD0PzIrL62I/kBAtm0f/bScRiT7XvQJXvkbEObW2yS2XhUv2qgq1VlFvO1jl3B85KMTHVb8+LrKpQWOKfRfxJ74c51y6tnEjZLFO6Xn0JevqOhd8gPQIXueFZOM0g0pNp1mbtGaaBlEIibbrzo2p9kzsdvJeHMq1XuMIi1/Qyj9 dpivonka@dhcp-41-144.bos.redhat.com", "key_options": null, "keyfile": "/home/dpivonka/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dpivonka", "validate_certs": true} 2021-02-10T11:04:58.293 INFO:teuthology.task.ansible.out:changed: [gibba020.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDg7xW5+Mk7pG5MjLxOc7GTvDTCPBJj2L3MEQZIapmmo6t8uPODwDewXncSzrKDH6zFVE4TTTD9fJdu+BhyT+jCFgXrEOMMabj7vJsg0wO0SWqr+4NECFPhD9y4ewfYiewBeXisclA5Bto91v93AtYpZBCdmPYSi0BEcPVc6S3iBUD0PzIrL62I/kBAtm0f/bScRiT7XvQJXvkbEObW2yS2XhUv2qgq1VlFvO1jl3B85KMTHVb8+LrKpQWOKfRfxJ74c51y6tnEjZLFO6Xn0JevqOhd8gPQIXueFZOM0g0pNp1mbtGaaBlEIibbrzo2p9kzsdvJeHMq1XuMIi1/Qyj9 dpivonka@dhcp-41-144.bos.redhat.com", "key_options": null, "keyfile": "/home/dpivonka/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dpivonka", "validate_certs": true} 2021-02-10T11:04:58.586 INFO:teuthology.task.ansible.out:changed: [gibba012.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0m+8wna4RnrOcuCUj971EBe4cFiTXVkEWkIsUX3yeCUwMaS/ZjaoOjcdYX7cYpCp1ilo4FSO5BsochU0RiMbQEwLGcE0zB2q1ac8+yYKCuWWyknig2t4bOReu5OrXvl6mj/9R+p7clFlYhwIoNG5ZZ/2eTbo3nH7gWgaOKaPbTtvEGg/5R/qjQKg8OTC8oix6kLsaNTisQJ78Zd7b4ukg9bO0jlq/2UhQOBM072sU1/kJpWTkcA0xxm3nsC4rWPwapgz7UwBmMdyx116GZgWdfQecTLKO7b8pyJw21npk2ntUmeMi7WPZGX6VI3IpaAhBjW3KiATAb8JbAQh8F9bmejkw6x7zrP3hYKQOg3A+U8DKRRo/Ad+f0zz9gjC42DUvMEOr09RBdxZcEAPY7H/YRWEqbA2zPX3Gfth4b7/letzM3m9ToUvVfSt1CdjaZD+gIia9pCgDMb+j1j2w9reG4F7Us+W/1f5pYCyFyrm4XO3AdoYM9/FOAOR3vffTIQU= adking@localhost.localdomain", "key_options": null, "keyfile": "/home/adking/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "adking", "validate_certs": true} 2021-02-10T11:04:58.734 INFO:teuthology.task.ansible.out:changed: [gibba020.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0m+8wna4RnrOcuCUj971EBe4cFiTXVkEWkIsUX3yeCUwMaS/ZjaoOjcdYX7cYpCp1ilo4FSO5BsochU0RiMbQEwLGcE0zB2q1ac8+yYKCuWWyknig2t4bOReu5OrXvl6mj/9R+p7clFlYhwIoNG5ZZ/2eTbo3nH7gWgaOKaPbTtvEGg/5R/qjQKg8OTC8oix6kLsaNTisQJ78Zd7b4ukg9bO0jlq/2UhQOBM072sU1/kJpWTkcA0xxm3nsC4rWPwapgz7UwBmMdyx116GZgWdfQecTLKO7b8pyJw21npk2ntUmeMi7WPZGX6VI3IpaAhBjW3KiATAb8JbAQh8F9bmejkw6x7zrP3hYKQOg3A+U8DKRRo/Ad+f0zz9gjC42DUvMEOr09RBdxZcEAPY7H/YRWEqbA2zPX3Gfth4b7/letzM3m9ToUvVfSt1CdjaZD+gIia9pCgDMb+j1j2w9reG4F7Us+W/1f5pYCyFyrm4XO3AdoYM9/FOAOR3vffTIQU= adking@localhost.localdomain", "key_options": null, "keyfile": "/home/adking/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "adking", "validate_certs": true} 2021-02-10T11:04:59.038 INFO:teuthology.task.ansible.out:changed: [gibba012.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCs6q2b5NcUsNXl+w1Mrc7idNvGdwWkL0XRX5zTw22UNpQyqPkSrRTFKi4NxoY8ucv8JoLnaJL7hYSRHKTh7SqH78uPLZX0FLtoHvpYLgUgJzzJ5AOW5uekIhCMCm+3egmFnEBwFcNxtGnU2es02ByXNEXRxt4/LQ0f8K6lMqxVRSKYcltvWzlA7WFFdTnYMunbCYOdpw88RXm2pYjGIHRBnie7+pFiFSS6K/mkzQcfLiJmahgLSaYslI0ho1vQd8xfNLmZ700C5F4VTUlT/mJK/dI2LF3OfaPTsyqKNzOchrjLFx5b6dhZ4WZyqu7VKsHNm+xHZwT8VP3LPGSVVBdmEsiCsmV4g4QTI5ssZLi7siZDpaFp72nVAIO1i6kBGuiF3or+J4LmIh8ZRKzTc0DRDOw7S+IkWu1fv2UhYg83z6wrz4HeczI+bnolFFWW6HgGV3MFaUeRTRvrbxw2DPoEGQ+cjP0wsjyEhxlScwdh8W8IbVpNl+7mBe45YGUH2hg83Lst5QQxD4PHelriu40moWRHFKCnKfNesb4L1uqu29o8f+/PgpLdrWE/P5zIKe2ReWJyScYAI7jTMlFw+xgf+0TKryk44NMy90l3ei0umlx4qsFbbEv+lhzAALrUJ/da7fsQw7eIVykmcscBeYr99/YSYk4MGdk4jVCiHsQkdw== amathuri@redhat.com", "key_options": null, "keyfile": "/home/amathuri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amathuri", "validate_certs": true} 2021-02-10T11:04:59.173 INFO:teuthology.task.ansible.out:changed: [gibba020.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCs6q2b5NcUsNXl+w1Mrc7idNvGdwWkL0XRX5zTw22UNpQyqPkSrRTFKi4NxoY8ucv8JoLnaJL7hYSRHKTh7SqH78uPLZX0FLtoHvpYLgUgJzzJ5AOW5uekIhCMCm+3egmFnEBwFcNxtGnU2es02ByXNEXRxt4/LQ0f8K6lMqxVRSKYcltvWzlA7WFFdTnYMunbCYOdpw88RXm2pYjGIHRBnie7+pFiFSS6K/mkzQcfLiJmahgLSaYslI0ho1vQd8xfNLmZ700C5F4VTUlT/mJK/dI2LF3OfaPTsyqKNzOchrjLFx5b6dhZ4WZyqu7VKsHNm+xHZwT8VP3LPGSVVBdmEsiCsmV4g4QTI5ssZLi7siZDpaFp72nVAIO1i6kBGuiF3or+J4LmIh8ZRKzTc0DRDOw7S+IkWu1fv2UhYg83z6wrz4HeczI+bnolFFWW6HgGV3MFaUeRTRvrbxw2DPoEGQ+cjP0wsjyEhxlScwdh8W8IbVpNl+7mBe45YGUH2hg83Lst5QQxD4PHelriu40moWRHFKCnKfNesb4L1uqu29o8f+/PgpLdrWE/P5zIKe2ReWJyScYAI7jTMlFw+xgf+0TKryk44NMy90l3ei0umlx4qsFbbEv+lhzAALrUJ/da7fsQw7eIVykmcscBeYr99/YSYk4MGdk4jVCiHsQkdw== amathuri@redhat.com", "key_options": null, "keyfile": "/home/amathuri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amathuri", "validate_certs": true} 2021-02-10T11:04:59.501 INFO:teuthology.task.ansible.out:changed: [gibba012.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDr6H7hmtGCUrO3GppgMHwbmA4XT/Vipz/uLG/N8h6ZxtaLQKpCefRpdKcFlOdDwxkTmT5q9vmXQ8nl95lTGq+eeMxl4ho104ZD9e9AIYmgYqFy7/qNP8gkqduIPlypLL9m5rElTcGd7zqfMqgljMvAmnXLtZNJlIuFtszVU5WPI1QCwfQ3t5aHF6j66bpVoN9uWItVGB1LApQz9TtViWsQ9Hl8X4YauJToO+TkuSX+H/OGH7ZC6CPOzNZb3b3vF/jLMnTzXl86PJs1se4DjKkRGLVCVbr2WhLwQeZ8TEn3Hg1Gn5X8zNM5cTo6U8SnXBAe9ZYbuyfvghrNLdlZpmN+8l0/l5zC3z+N+sEzklXwHw4MQLF+KSloVFd9NN/kyenufTZAyp8VxllZ9TgdC+CPORIWyzmu1elOtYo7LNuE6CIh/Zm9B54SO6hVHhVoOCqe/dIyjeHJA0WKIZblK1XZEvWuu0E2T3jhxfe1GICyTukeB8WoBQH9DZ1yzMPt59E= nia@redhat.com", "key_options": null, "keyfile": "/home/nizamial09/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nizamial09", "validate_certs": true} 2021-02-10T11:04:59.596 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:04:59 +0000 (0:01:00.656) 0:02:16.884 **** 2021-02-10T11:04:59.627 INFO:teuthology.task.ansible.out:changed: [gibba020.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDr6H7hmtGCUrO3GppgMHwbmA4XT/Vipz/uLG/N8h6ZxtaLQKpCefRpdKcFlOdDwxkTmT5q9vmXQ8nl95lTGq+eeMxl4ho104ZD9e9AIYmgYqFy7/qNP8gkqduIPlypLL9m5rElTcGd7zqfMqgljMvAmnXLtZNJlIuFtszVU5WPI1QCwfQ3t5aHF6j66bpVoN9uWItVGB1LApQz9TtViWsQ9Hl8X4YauJToO+TkuSX+H/OGH7ZC6CPOzNZb3b3vF/jLMnTzXl86PJs1se4DjKkRGLVCVbr2WhLwQeZ8TEn3Hg1Gn5X8zNM5cTo6U8SnXBAe9ZYbuyfvghrNLdlZpmN+8l0/l5zC3z+N+sEzklXwHw4MQLF+KSloVFd9NN/kyenufTZAyp8VxllZ9TgdC+CPORIWyzmu1elOtYo7LNuE6CIh/Zm9B54SO6hVHhVoOCqe/dIyjeHJA0WKIZblK1XZEvWuu0E2T3jhxfe1GICyTukeB8WoBQH9DZ1yzMPt59E= nia@redhat.com", "key_options": null, "keyfile": "/home/nizamial09/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nizamial09", "validate_certs": true} 2021-02-10T11:04:59.674 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-02-10T11:04:59.675 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:04:59.675 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmi 2021-02-10T11:04:59.677 INFO:teuthology.task.ansible.out:ck@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:04:59.711 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:04:59 +0000 (0:00:00.114) 0:02:16.999 **** 2021-02-10T11:04:59.718 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:04:59.719 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:04:59.720 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "skip_reason": "Conditional result was False"} skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:04:59.755 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:04:59.766 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:04:59.768 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:04:59.777 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:04:59.788 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:04:59.789 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:04:59.800 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:04:59.811 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:04:59.812 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:00.158 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} 2021-02-10T11:05:00.169 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:00.180 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:00.255 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} 2021-02-10T11:05:00.267 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:00.278 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:00.614 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} 2021-02-10T11:05:00.719 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} 2021-02-10T11:05:01.055 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} 2021-02-10T11:05:01.067 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.078 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.079 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.089 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.101 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.112 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.114 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.123 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.135 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.135 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.146 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.157 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} 2021-02-10T11:05:01.158 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.169 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.170 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.172 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.182 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.183 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.194 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.194 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.195 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.206 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.206 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.207 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.217 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.218 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.219 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.229 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.230 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.230 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.242 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.243 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.253 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.253 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.265 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.266 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.277 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.277 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.279 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.289 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.289 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.300 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.300 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.302 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.312 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.313 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.313 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.315 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.324 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.325 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.326 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.336 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.338 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.349 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.350 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.361 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.372 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.373 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.374 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "skip_reason": "Conditional result was False"} skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.385 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.385 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.387 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.397 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.397 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.399 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.410 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.411 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.412 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.422 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.423 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.424 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.434 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.434 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.436 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.446 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.447 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.448 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.458 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.459 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.460 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.472 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.473 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.474 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.484 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.485 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.486 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.497 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.497 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.509 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.509 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.511 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.521 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.522 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.533 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.533 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.535 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.545 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.545 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.547 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.557 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.558 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.559 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.571 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.571 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.583 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.584 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.585 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.596 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.597 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.598 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.608 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.609 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.610 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.622 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.623 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.624 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.636 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.637 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.638 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.648 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.649 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.660 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.660 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.661 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "skip_reason": "Conditional result was False"} skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.672 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.673 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.674 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.685 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.685 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.696 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.696 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.698 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.708 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.710 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.720 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.720 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.722 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.732 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.732 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.733 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.744 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "skip_reason": "Conditional result was False"} skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.744 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.746 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.756 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.756 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.767 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.767 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.769 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.780 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.780 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.792 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.792 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.793 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.804 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.805 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.806 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.816 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.817 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.818 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.828 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.829 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.830 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.841 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.841 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.852 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.853 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.854 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.864 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.865 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.866 INFO:teuthology.task.ansible.out: skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "skip_reason": "Conditional result was False"} skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.877 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.878 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.888 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.889 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.890 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.890 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.901 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.902 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.913 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.913 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.915 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "skip_reason": "Conditional result was False"} skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.925 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.926 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.936 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.937 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.938 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.948 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.948 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.950 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.960 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.960 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.962 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.972 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.972 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.974 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.984 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.985 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.996 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.996 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:01.998 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:02.008 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:02.008 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:02.009 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:02.019 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:02.020 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:02.021 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "skip_reason": "Conditional result was False"} skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:02.032 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:02.033 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:02.044 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:02.045 INFO:teuthology.task.ansible.out: 2021-02-10T11:05:02.063 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:02.064 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:02.065 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:02.100 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:05:02 +0000 (0:00:02.389) 0:02:19.388 **** 2021-02-10T11:05:02.107 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:02.107 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:02.109 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:02.109 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:02.110 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "skip_reason": "Conditional result was False"} skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:02.110 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:02.122 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:02.123 INFO:teuthology.task.ansible.out: TASK [users : Filter the revoked_users list] *********************************** 2021-02-10T11:05:02.123 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:02.156 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:05:02 +0000 (0:00:00.056) 0:02:19.444 **** 2021-02-10T11:05:02.163 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-02-10T11:05:02.164 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:02.165 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "skip_reason": "Conditional result was False"} skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:02.166 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:02.217 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:05:02 +0000 (0:00:00.060) 0:02:19.505 **** 2021-02-10T11:05:02.245 INFO:teuthology.task.ansible.out: TASK [users : Filter the revoked_users list] *********************************** 2021-02-10T11:05:02.246 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:02.278 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:05:02 +0000 (0:00:00.060) 0:02:19.566 **** 2021-02-10T11:05:02.627 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-10T11:05:02.629 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} 2021-02-10T11:05:02.755 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} 2021-02-10T11:05:03.061 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2021-02-10T11:05:03.187 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2021-02-10T11:05:03.502 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2021-02-10T11:05:03.629 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2021-02-10T11:05:03.930 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2021-02-10T11:05:04.055 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2021-02-10T11:05:04.369 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2021-02-10T11:05:04.495 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2021-02-10T11:05:04.809 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2021-02-10T11:05:04.924 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2021-02-10T11:05:05.250 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2021-02-10T11:05:05.366 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2021-02-10T11:05:05.684 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "name": "gmeno", "state": "absent"} 2021-02-10T11:05:05.804 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "name": "gmeno", "state": "absent"} 2021-02-10T11:05:06.112 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item=alfredodeza) => {"ansible_loop_var": "item", "changed": false, "item": "alfredodeza", "name": "alfredodeza", "state": "absent"} 2021-02-10T11:05:06.250 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item=alfredodeza) => {"ansible_loop_var": "item", "changed": false, "item": "alfredodeza", "name": "alfredodeza", "state": "absent"} 2021-02-10T11:05:06.554 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} 2021-02-10T11:05:06.691 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} 2021-02-10T11:05:06.995 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2021-02-10T11:05:07.126 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2021-02-10T11:05:07.432 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} 2021-02-10T11:05:07.570 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} 2021-02-10T11:05:07.875 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2021-02-10T11:05:08.004 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2021-02-10T11:05:08.310 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2021-02-10T11:05:08.311 INFO:teuthology.task.ansible.out: 2021-02-10T11:05:08.447 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2021-02-10T11:05:08.750 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2021-02-10T11:05:08.886 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2021-02-10T11:05:09.187 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2021-02-10T11:05:09.322 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2021-02-10T11:05:09.615 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} 2021-02-10T11:05:09.751 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} 2021-02-10T11:05:10.055 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2021-02-10T11:05:10.183 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2021-02-10T11:05:10.488 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2021-02-10T11:05:10.625 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2021-02-10T11:05:10.919 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2021-02-10T11:05:11.057 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2021-02-10T11:05:11.364 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2021-02-10T11:05:11.490 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2021-02-10T11:05:11.802 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "name": "loic", "state": "absent"} 2021-02-10T11:05:11.929 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "name": "loic", "state": "absent"} 2021-02-10T11:05:12.242 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2021-02-10T11:05:12.370 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2021-02-10T11:05:12.676 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "name": "jj", "state": "absent"} 2021-02-10T11:05:12.813 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "name": "jj", "state": "absent"} 2021-02-10T11:05:13.108 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "name": "amarangone", "state": "absent"} 2021-02-10T11:05:13.247 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "name": "amarangone", "state": "absent"} 2021-02-10T11:05:13.552 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "name": "oprypin", "state": "absent"} 2021-02-10T11:05:13.688 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "name": "oprypin", "state": "absent"} 2021-02-10T11:05:13.993 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "name": "adamyanova", "state": "absent"} 2021-02-10T11:05:13.994 INFO:teuthology.task.ansible.out: 2021-02-10T11:05:14.130 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "name": "adamyanova", "state": "absent"} 2021-02-10T11:05:14.421 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "name": "sbillah", "state": "absent"} 2021-02-10T11:05:14.567 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "name": "sbillah", "state": "absent"} 2021-02-10T11:05:14.860 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "name": "onyb", "state": "absent"} 2021-02-10T11:05:15.006 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "name": "onyb", "state": "absent"} 2021-02-10T11:05:15.293 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "name": "jwilliamson", "state": "absent"} 2021-02-10T11:05:15.441 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "name": "jwilliamson", "state": "absent"} 2021-02-10T11:05:15.734 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "name": "kmroz", "state": "absent"} 2021-02-10T11:05:15.882 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "name": "kmroz", "state": "absent"} 2021-02-10T11:05:16.164 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "name": "shehbazj", "state": "absent"} 2021-02-10T11:05:16.310 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "name": "shehbazj", "state": "absent"} 2021-02-10T11:05:16.603 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "name": "abhishekvrshny", "state": "absent"} 2021-02-10T11:05:16.750 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "name": "abhishekvrshny", "state": "absent"} 2021-02-10T11:05:17.042 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "name": "asheplyakov", "state": "absent"} 2021-02-10T11:05:17.192 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "name": "asheplyakov", "state": "absent"} 2021-02-10T11:05:17.483 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "name": "liupan", "state": "absent"} 2021-02-10T11:05:17.630 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "name": "liupan", "state": "absent"} 2021-02-10T11:05:17.911 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "name": "adeza", "state": "absent"} 2021-02-10T11:05:18.067 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "name": "adeza", "state": "absent"} 2021-02-10T11:05:18.069 INFO:teuthology.task.ansible.out: 2021-02-10T11:05:18.352 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item=pranith) => {"ansible_loop_var": "item", "changed": false, "item": "pranith", "name": "pranith", "state": "absent"} 2021-02-10T11:05:18.498 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item=pranith) => {"ansible_loop_var": "item", "changed": false, "item": "pranith", "name": "pranith", "state": "absent"} 2021-02-10T11:05:19.107 INFO:teuthology.task.ansible.out:changed: [gibba012.front.sepia.ceph.com] => (item=dorinda) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "dorinda", "name": "dorinda", "remove": false, "state": "absent"} 2021-02-10T11:05:19.327 INFO:teuthology.task.ansible.out:changed: [gibba020.front.sepia.ceph.com] => (item=dorinda) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "dorinda", "name": "dorinda", "remove": false, "state": "absent"} 2021-02-10T11:05:19.857 INFO:teuthology.task.ansible.out:changed: [gibba012.front.sepia.ceph.com] => (item=zyan) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "zyan", "name": "zyan", "remove": false, "state": "absent"} 2021-02-10T11:05:19.983 INFO:teuthology.task.ansible.out:changed: [gibba020.front.sepia.ceph.com] => (item=zyan) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "zyan", "name": "zyan", "remove": false, "state": "absent"} 2021-02-10T11:05:20.288 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2021-02-10T11:05:20.425 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2021-02-10T11:05:20.727 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2021-02-10T11:05:20.863 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2021-02-10T11:05:21.167 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2021-02-10T11:05:21.292 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2021-02-10T11:05:21.606 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2021-02-10T11:05:21.732 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2021-02-10T11:05:22.096 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2021-02-10T11:05:22.171 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2021-02-10T11:05:22.567 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2021-02-10T11:05:22.610 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2021-02-10T11:05:23.040 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2021-02-10T11:05:23.052 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2021-02-10T11:05:23.498 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "name": "fche", "state": "absent"} 2021-02-10T11:05:23.499 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "name": "fche", "state": "absent"} 2021-02-10T11:05:23.945 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item=jbainbri) => {"ansible_loop_var": "item", "changed": false, "item": "jbainbri", "name": "jbainbri", "state": "absent"} 2021-02-10T11:05:23.946 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item=jbainbri) => {"ansible_loop_var": "item", "changed": false, "item": "jbainbri", "name": "jbainbri", "state": "absent"} 2021-02-10T11:05:24.382 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item=kdhananj) => {"ansible_loop_var": "item", "changed": false, "item": "kdhananj", "name": "kdhananj", "state": "absent"} 2021-02-10T11:05:24.405 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item=kdhananj) => {"ansible_loop_var": "item", "changed": false, "item": "kdhananj", "name": "kdhananj", "state": "absent"} 2021-02-10T11:05:24.480 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:05:24 +0000 (0:00:22.202) 0:02:41.768 **** 2021-02-10T11:05:24.517 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:05:24 +0000 (0:00:00.036) 0:02:41.805 **** 2021-02-10T11:05:27.538 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** 2021-02-10T11:05:27.540 INFO:teuthology.task.ansible.out:changed: [gibba020.front.sepia.ceph.com] => {"changed": true, "checksum": "5ebc9fcf8859a9c716ab10089a7bec7e44d29368", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "md5sum": "ee06ed012f51c2e3764171d4b9180349", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_runtime_t:s0", "size": 40, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1612955124.5431685-54287835125138/source", "state": "file", "uid": 0} changed: [gibba012.front.sepia.ceph.com] => {"changed": true, "checksum": "5ebc9fcf8859a9c716ab10089a7bec7e44d29368", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "md5sum": "ee06ed012f51c2e3764171d4b9180349", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_runtime_t:s0", "size": 40, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1612955124.5129805-274867221988540/source", "state": "file", "uid": 0} 2021-02-10T11:05:27.582 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:05:27 +0000 (0:00:03.064) 0:02:44.870 **** 2021-02-10T11:05:27.583 INFO:teuthology.task.ansible.out: 2021-02-10T11:05:27.627 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:05:27 +0000 (0:00:00.044) 0:02:44.915 **** 2021-02-10T11:05:27.633 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2021-02-10T11:05:27.633 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => { "msg": "Host gibba012.front.sepia.ceph.com is running RedHat 7.9 (Maipo)" } 2021-02-10T11:05:27.671 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:05:27 +0000 (0:00:00.044) 0:02:44.959 **** 2021-02-10T11:05:27.677 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => { "msg": "Host gibba020.front.sepia.ceph.com is running RedHat 7.9 (Maipo)" } 2021-02-10T11:05:27.714 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:05:27 +0000 (0:00:00.043) 0:02:45.002 **** 2021-02-10T11:05:27.734 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2021-02-10T11:05:27.736 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/redhat_7.yml) => {"ansible_facts": {"rhsm_repos": ["rhel-7-server-rpms", "rhel-7-server-optional-rpms", "rhel-7-server-extras-rpms", "rhel-ha-for-rhel-7-server-rpms"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/redhat_7.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/redhat_7.yml"} 2021-02-10T11:05:27.769 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:05:27 +0000 (0:00:00.055) 0:02:45.058 **** 2021-02-10T11:05:27.776 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/redhat_7.yml) => {"ansible_facts": {"rhsm_repos": ["rhel-7-server-rpms", "rhel-7-server-optional-rpms", "rhel-7-server-extras-rpms", "rhel-ha-for-rhel-7-server-rpms"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/redhat_7.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/redhat_7.yml"} 2021-02-10T11:05:27.821 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:05:27 +0000 (0:00:00.051) 0:02:45.109 **** 2021-02-10T11:05:27.830 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2021-02-10T11:05:27.831 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:27.870 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:05:27 +0000 (0:00:00.048) 0:02:45.158 **** 2021-02-10T11:05:27.876 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:27.913 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:05:27 +0000 (0:00:00.043) 0:02:45.201 **** 2021-02-10T11:05:28.398 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** 2021-02-10T11:05:28.400 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => {"changed": false, "cmd": "timedatectl | grep -E \"Time ?zone\" | sed -e \"s/.*: \\(.*\\) (.*/\\1/\"", "delta": "0:00:00.065121", "end": "2021-02-10 11:05:28.381015", "rc": 0, "start": "2021-02-10 11:05:28.315894", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2021-02-10T11:05:28.441 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:05:28 +0000 (0:00:00.527) 0:02:45.729 **** 2021-02-10T11:05:28.450 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => {"changed": false, "cmd": "timedatectl | grep -E \"Time ?zone\" | sed -e \"s/.*: \\(.*\\) (.*/\\1/\"", "delta": "0:00:00.067090", "end": "2021-02-10 11:05:28.423476", "rc": 0, "start": "2021-02-10 11:05:28.356386", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2021-02-10T11:05:28.489 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:05:28 +0000 (0:00:00.048) 0:02:45.777 **** 2021-02-10T11:05:28.497 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2021-02-10T11:05:28.499 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:28.543 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:05:28 +0000 (0:00:00.053) 0:02:45.831 **** 2021-02-10T11:05:28.550 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:28.606 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:05:28 +0000 (0:00:00.062) 0:02:45.894 **** 2021-02-10T11:05:28.614 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** 2021-02-10T11:05:28.616 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:28.663 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:05:28 +0000 (0:00:00.057) 0:02:45.952 **** 2021-02-10T11:05:28.672 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:28.712 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:05:28 +0000 (0:00:00.048) 0:02:46.000 **** 2021-02-10T11:05:28.718 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** 2021-02-10T11:05:28.724 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:28.758 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:05:28 +0000 (0:00:00.046) 0:02:46.046 **** 2021-02-10T11:05:28.765 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:28.803 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:05:28 +0000 (0:00:00.044) 0:02:46.091 **** 2021-02-10T11:05:28.809 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** 2021-02-10T11:05:28.811 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:28.844 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:05:28 +0000 (0:00:00.041) 0:02:46.133 **** 2021-02-10T11:05:28.851 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:28.888 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:05:28 +0000 (0:00:00.043) 0:02:46.176 **** 2021-02-10T11:05:28.894 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** 2021-02-10T11:05:28.896 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:28.935 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:05:28 +0000 (0:00:00.047) 0:02:46.223 **** 2021-02-10T11:05:28.946 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:29.007 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:05:29 +0000 (0:00:00.071) 0:02:46.295 **** 2021-02-10T11:05:29.013 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2021-02-10T11:05:29.015 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => {"ansible_facts": {"entitlements_path": "/etc/ansible/secrets/entitlements.yml"}, "changed": false} 2021-02-10T11:05:29.072 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:05:29 +0000 (0:00:00.065) 0:02:46.360 **** 2021-02-10T11:05:29.084 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => {"ansible_facts": {"entitlements_path": "/etc/ansible/secrets/entitlements.yml"}, "changed": false} 2021-02-10T11:05:29.135 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:05:29 +0000 (0:00:00.063) 0:02:46.423 **** 2021-02-10T11:05:29.144 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2021-02-10T11:05:29.145 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} ok: [gibba012.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-02-10T11:05:29.184 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:05:29 +0000 (0:00:00.048) 0:02:46.472 **** 2021-02-10T11:05:29.193 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-02-10T11:05:29.195 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-02-10T11:05:29.232 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:05:29 +0000 (0:00:00.048) 0:02:46.521 **** 2021-02-10T11:05:29.239 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** 2021-02-10T11:05:29.241 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => {"ansible_facts": {"have_entitlements": true}, "changed": false} 2021-02-10T11:05:29.278 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:05:29 +0000 (0:00:00.046) 0:02:46.567 **** 2021-02-10T11:05:29.291 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => {"ansible_facts": {"have_entitlements": true}, "changed": false} 2021-02-10T11:05:29.325 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:05:29 +0000 (0:00:00.046) 0:02:46.614 **** 2021-02-10T11:05:31.159 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** 2021-02-10T11:05:31.160 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["katello-ca-consumer-satellite.front.sepia.ceph.com-1.0-3.noarch providing /root/.ansible/tmp/ansible-moduletmp-1612955130.34-4JefAh/katello-ca-consumer-latest.noarch6xVwYc.rpm is already installed"]} 2021-02-10T11:05:31.195 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:05:31 +0000 (0:00:01.869) 0:02:48.483 **** 2021-02-10T11:05:31.224 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["katello-ca-consumer-satellite.front.sepia.ceph.com-1.0-3.noarch providing /root/.ansible/tmp/ansible-moduletmp-1612955130.33-4nlDgH/katello-ca-consumer-latest.noarchIIk4YN.rpm is already installed"]} 2021-02-10T11:05:31.267 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:05:31 +0000 (0:00:00.072) 0:02:48.556 **** 2021-02-10T11:05:32.021 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2021-02-10T11:05:32.023 INFO:teuthology.task.ansible.out:fatal: [gibba020.front.sepia.ceph.com]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} ...ignoring 2021-02-10T11:05:32.072 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:05:32 +0000 (0:00:00.803) 0:02:49.359 **** 2021-02-10T11:05:32.091 INFO:teuthology.task.ansible.out:fatal: [gibba012.front.sepia.ceph.com]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-02-10T11:05:32.092 INFO:teuthology.task.ansible.out:...ignoring 2021-02-10T11:05:32.128 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:05:32 +0000 (0:00:00.056) 0:02:49.416 **** 2021-02-10T11:05:32.135 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2021-02-10T11:05:32.136 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => {"ansible_facts": {"rhsm_registered": false}, "changed": false} 2021-02-10T11:05:32.172 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:05:32 +0000 (0:00:00.043) 0:02:49.460 **** 2021-02-10T11:05:32.178 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => {"ansible_facts": {"rhsm_registered": false}, "changed": false} 2021-02-10T11:05:32.215 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:05:32 +0000 (0:00:00.042) 0:02:49.503 **** 2021-02-10T11:05:36.896 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** 2021-02-10T11:05:36.897 INFO:teuthology.task.ansible.out:changed: [gibba012.front.sepia.ceph.com] => {"attempts": 1, "censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2021-02-10T11:05:36.929 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:05:36 +0000 (0:00:04.714) 0:02:54.217 **** 2021-02-10T11:05:36.968 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** 2021-02-10T11:05:36.970 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => {"ansible_facts": {"rhsm_registered": true}, "changed": false} 2021-02-10T11:05:37.003 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:05:37 +0000 (0:00:00.073) 0:02:54.291 **** 2021-02-10T11:05:37.681 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** 2021-02-10T11:05:37.682 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => {"changed": false, "cmd": "subscription-manager release --list | grep -E '[0-9]'", "delta": "0:00:00.223735", "end": "2021-02-10 11:05:37.662329", "failed_when_result": false, "rc": 0, "start": "2021-02-10 11:05:37.438594", "stderr": "", "stderr_lines": [], "stdout": "7.4\n7.5\n7.6\n7.7\n7.8\n7.9\n7Server\n8\n8.0\n8.1\n8.2\n8.3", "stdout_lines": ["7.4", "7.5", "7.6", "7.7", "7.8", "7.9", "7Server", "8", "8.0", "8.1", "8.2", "8.3"]} 2021-02-10T11:05:37.714 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:05:37 +0000 (0:00:00.711) 0:02:55.002 **** 2021-02-10T11:05:37.751 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** 2021-02-10T11:05:37.753 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:37.785 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:05:37 +0000 (0:00:00.071) 0:02:55.073 **** 2021-02-10T11:05:37.822 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2021-02-10T11:05:37.824 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:37.868 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:05:37 +0000 (0:00:00.083) 0:02:55.157 **** 2021-02-10T11:05:37.898 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* 2021-02-10T11:05:37.900 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-02-10T11:05:37.931 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:05:37 +0000 (0:00:00.062) 0:02:55.220 **** 2021-02-10T11:05:37.970 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2021-02-10T11:05:37.972 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:38.017 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:05:38 +0000 (0:00:00.085) 0:02:55.306 **** 2021-02-10T11:05:38.140 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** 2021-02-10T11:05:38.141 INFO:teuthology.task.ansible.out:changed: [gibba020.front.sepia.ceph.com] => {"attempts": 1, "censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2021-02-10T11:05:38.176 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:05:38 +0000 (0:00:00.158) 0:02:55.464 **** 2021-02-10T11:05:38.221 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** 2021-02-10T11:05:38.222 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => {"ansible_facts": {"rhsm_registered": true}, "changed": false} 2021-02-10T11:05:38.257 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:05:38 +0000 (0:00:00.081) 0:02:55.545 **** 2021-02-10T11:05:38.936 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** 2021-02-10T11:05:38.937 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => {"changed": false, "cmd": "subscription-manager release --list | grep -E '[0-9]'", "delta": "0:00:00.220656", "end": "2021-02-10 11:05:38.913918", "failed_when_result": false, "rc": 0, "start": "2021-02-10 11:05:38.693262", "stderr": "", "stderr_lines": [], "stdout": "7.4\n7.5\n7.6\n7.7\n7.8\n7.9\n7Server\n8\n8.0\n8.1\n8.2\n8.3", "stdout_lines": ["7.4", "7.5", "7.6", "7.7", "7.8", "7.9", "7Server", "8", "8.0", "8.1", "8.2", "8.3"]} 2021-02-10T11:05:38.970 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:05:38 +0000 (0:00:00.712) 0:02:56.258 **** 2021-02-10T11:05:39.007 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** 2021-02-10T11:05:39.008 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:39.047 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:05:39 +0000 (0:00:00.076) 0:02:56.335 **** 2021-02-10T11:05:39.077 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:39.110 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:05:39 +0000 (0:00:00.063) 0:02:56.398 **** 2021-02-10T11:05:39.137 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* 2021-02-10T11:05:39.138 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-02-10T11:05:39.176 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:05:39 +0000 (0:00:00.066) 0:02:56.465 **** 2021-02-10T11:05:39.214 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2021-02-10T11:05:39.215 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:39.247 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:05:39 +0000 (0:00:00.070) 0:02:56.535 **** 2021-02-10T11:05:39.778 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* 2021-02-10T11:05:39.780 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => {"changed": false, "cmd": "subscription-manager repos --list | grep -B4 'Enabled:.*1' | grep 'Repo ID:' | sed -e 's/Repo ID:\\s*\\(.*\\)/\\1/' | sort", "delta": "0:00:01.297769", "end": "2021-02-10 11:05:39.762709", "rc": 0, "start": "2021-02-10 11:05:38.464940", "stderr": "", "stderr_lines": [], "stdout": "rhel-7-server-rpms", "stdout_lines": ["rhel-7-server-rpms"]} 2021-02-10T11:05:39.814 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:05:39 +0000 (0:00:00.567) 0:02:57.102 **** 2021-02-10T11:05:39.851 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ 2021-02-10T11:05:39.853 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => {"ansible_facts": {"repo_list": ["rhel-7-server-rpms"]}, "changed": false} 2021-02-10T11:05:39.895 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:05:39 +0000 (0:00:00.081) 0:02:57.184 **** 2021-02-10T11:05:39.937 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2021-02-10T11:05:39.938 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:39.971 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:05:39 +0000 (0:00:00.075) 0:02:57.259 **** 2021-02-10T11:05:40.011 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** 2021-02-10T11:05:40.012 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => {"ansible_facts": {"replace_repos": true}, "changed": false} 2021-02-10T11:05:40.051 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:05:40 +0000 (0:00:00.079) 0:02:57.339 **** 2021-02-10T11:05:40.100 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2021-02-10T11:05:40.102 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => {"ansible_facts": {"replace_repos": true}, "changed": false} 2021-02-10T11:05:40.135 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:05:40 +0000 (0:00:00.084) 0:02:57.423 **** 2021-02-10T11:05:41.076 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* 2021-02-10T11:05:41.077 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => {"changed": false, "cmd": "subscription-manager repos --list | grep -B4 'Enabled:.*1' | grep 'Repo ID:' | sed -e 's/Repo ID:\\s*\\(.*\\)/\\1/' | sort", "delta": "0:00:01.359535", "end": "2021-02-10 11:05:41.057297", "rc": 0, "start": "2021-02-10 11:05:39.697762", "stderr": "", "stderr_lines": [], "stdout": "rhel-7-server-rpms", "stdout_lines": ["rhel-7-server-rpms"]} 2021-02-10T11:05:41.115 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:05:41 +0000 (0:00:00.980) 0:02:58.404 **** 2021-02-10T11:05:41.164 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ 2021-02-10T11:05:41.166 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => {"ansible_facts": {"repo_list": ["rhel-7-server-rpms"]}, "changed": false} 2021-02-10T11:05:41.202 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:05:41 +0000 (0:00:00.086) 0:02:58.490 **** 2021-02-10T11:05:41.240 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2021-02-10T11:05:41.242 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:41.279 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:05:41 +0000 (0:00:00.077) 0:02:58.567 **** 2021-02-10T11:05:41.327 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** 2021-02-10T11:05:41.329 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => {"ansible_facts": {"replace_repos": true}, "changed": false} 2021-02-10T11:05:41.366 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:05:41 +0000 (0:00:00.086) 0:02:58.654 **** 2021-02-10T11:05:41.420 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2021-02-10T11:05:41.422 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => {"ansible_facts": {"replace_repos": true}, "changed": false} 2021-02-10T11:05:41.461 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:05:41 +0000 (0:00:00.095) 0:02:58.749 **** 2021-02-10T11:05:42.809 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** 2021-02-10T11:05:42.811 INFO:teuthology.task.ansible.out:changed: [gibba012.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2021-02-10T11:05:42.843 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:05:42 +0000 (0:00:01.382) 0:03:00.132 **** 2021-02-10T11:05:43.641 INFO:teuthology.task.ansible.out:changed: [gibba020.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2021-02-10T11:05:43.677 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:05:43 +0000 (0:00:00.833) 0:03:00.965 **** 2021-02-10T11:05:45.338 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2021-02-10T11:05:45.340 INFO:teuthology.task.ansible.out:changed: [gibba012.front.sepia.ceph.com] => {"changed": true, "cmd": ["subscription-manager", "repos", "--enable=rhel-7-server-rpms", "--enable=rhel-7-server-optional-rpms", "--enable=rhel-7-server-extras-rpms", "--enable=rhel-ha-for-rhel-7-server-rpms"], "delta": "0:00:02.010656", "end": "2021-02-10 11:05:45.323956", "rc": 0, "start": "2021-02-10 11:05:43.313300", "stderr": "", "stderr_lines": [], "stdout": "Repository 'rhel-7-server-rpms' is enabled for this system.\nRepository 'rhel-7-server-optional-rpms' is enabled for this system.\nRepository 'rhel-ha-for-rhel-7-server-rpms' is enabled for this system.\nRepository 'rhel-7-server-extras-rpms' is enabled for this system.", "stdout_lines": ["Repository 'rhel-7-server-rpms' is enabled for this system.", "Repository 'rhel-7-server-optional-rpms' is enabled for this system.", "Repository 'rhel-ha-for-rhel-7-server-rpms' is enabled for this system.", "Repository 'rhel-7-server-extras-rpms' is enabled for this system."]} 2021-02-10T11:05:45.374 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:05:45 +0000 (0:00:01.697) 0:03:02.662 **** 2021-02-10T11:05:45.848 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2021-02-10T11:05:45.849 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => {"changed": false, "path": "/etc/yum.repos.d/cd.repo", "state": "absent"} 2021-02-10T11:05:45.881 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:05:45 +0000 (0:00:00.507) 0:03:03.170 **** 2021-02-10T11:05:46.149 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2021-02-10T11:05:46.151 INFO:teuthology.task.ansible.out:changed: [gibba020.front.sepia.ceph.com] => {"changed": true, "cmd": ["subscription-manager", "repos", "--enable=rhel-7-server-rpms", "--enable=rhel-7-server-optional-rpms", "--enable=rhel-7-server-extras-rpms", "--enable=rhel-ha-for-rhel-7-server-rpms"], "delta": "0:00:02.010945", "end": "2021-02-10 11:05:46.136341", "rc": 0, "start": "2021-02-10 11:05:44.125396", "stderr": "", "stderr_lines": [], "stdout": "Repository 'rhel-7-server-rpms' is enabled for this system.\nRepository 'rhel-7-server-optional-rpms' is enabled for this system.\nRepository 'rhel-ha-for-rhel-7-server-rpms' is enabled for this system.\nRepository 'rhel-7-server-extras-rpms' is enabled for this system.", "stdout_lines": ["Repository 'rhel-7-server-rpms' is enabled for this system.", "Repository 'rhel-7-server-optional-rpms' is enabled for this system.", "Repository 'rhel-ha-for-rhel-7-server-rpms' is enabled for this system.", "Repository 'rhel-7-server-extras-rpms' is enabled for this system."]} 2021-02-10T11:05:46.188 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:05:46 +0000 (0:00:00.306) 0:03:03.476 **** 2021-02-10T11:05:46.343 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2021-02-10T11:05:46.344 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-02-10T11:05:46.379 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:05:46 +0000 (0:00:00.191) 0:03:03.667 **** 2021-02-10T11:05:46.643 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2021-02-10T11:05:46.645 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => {"changed": false, "path": "/etc/yum.repos.d/cd.repo", "state": "absent"} 2021-02-10T11:05:46.687 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:05:46 +0000 (0:00:00.308) 0:03:03.976 **** 2021-02-10T11:05:47.146 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2021-02-10T11:05:47.148 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-02-10T11:05:47.194 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:05:47 +0000 (0:00:00.506) 0:03:04.482 **** 2021-02-10T11:05:48.338 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2021-02-10T11:05:48.338 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "86c0d58bde7ad4cfaafe592d2fefd92c920c42ca", "dest": "/etc/yum.repos.d/epel.repo", "gid": 0, "group": "root", "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 236, "state": "file", "uid": 0} 2021-02-10T11:05:49.216 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "86c0d58bde7ad4cfaafe592d2fefd92c920c42ca", "dest": "/etc/yum.repos.d/epel.repo", "gid": 0, "group": "root", "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 236, "state": "file", "uid": 0} 2021-02-10T11:05:50.249 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a1bad5509ed35b895c2061841236b1cfe10619ae", "dest": "/etc/yum.repos.d/epel-testing.repo", "gid": 0, "group": "root", "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 262, "state": "file", "uid": 0} 2021-02-10T11:05:50.285 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:05:50 +0000 (0:00:03.090) 0:03:07.573 **** 2021-02-10T11:05:51.121 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a1bad5509ed35b895c2061841236b1cfe10619ae", "dest": "/etc/yum.repos.d/epel-testing.repo", "gid": 0, "group": "root", "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 262, "state": "file", "uid": 0} 2021-02-10T11:05:51.164 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:05:51 +0000 (0:00:00.879) 0:03:08.452 **** 2021-02-10T11:05:52.227 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2021-02-10T11:05:52.229 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "73c387451a4fa299c9cb53e25b182ae50e8b8563", "dest": "/etc/yum.repos.d/epel-mirrorlist", "gid": 0, "group": "root", "item": "epel-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 418, "state": "file", "uid": 0} 2021-02-10T11:05:53.100 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "73c387451a4fa299c9cb53e25b182ae50e8b8563", "dest": "/etc/yum.repos.d/epel-mirrorlist", "gid": 0, "group": "root", "item": "epel-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 418, "state": "file", "uid": 0} 2021-02-10T11:05:54.132 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d25e11bf71dfd21a39ace708c03408897a4f263", "dest": "/etc/yum.repos.d/epel-testing-mirrorlist", "gid": 0, "group": "root", "item": "epel-testing-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 474, "state": "file", "uid": 0} 2021-02-10T11:05:54.174 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:05:54 +0000 (0:00:03.010) 0:03:11.463 **** 2021-02-10T11:05:54.212 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2021-02-10T11:05:54.214 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:54.245 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:05:54 +0000 (0:00:00.070) 0:03:11.533 **** 2021-02-10T11:05:54.272 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:54.305 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:05:54 +0000 (0:00:00.060) 0:03:11.594 **** 2021-02-10T11:05:54.333 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** 2021-02-10T11:05:54.334 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:54.387 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:05:54 +0000 (0:00:00.082) 0:03:11.676 **** 2021-02-10T11:05:54.414 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2021-02-10T11:05:54.416 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:54.447 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:05:54 +0000 (0:00:00.060) 0:03:11.736 **** 2021-02-10T11:05:54.476 INFO:teuthology.task.ansible.out: TASK [common : Mask sleep units] *********************************************** 2021-02-10T11:05:54.477 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item=sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "sleep.target", "skip_reason": "Conditional result was False"} 2021-02-10T11:05:54.487 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item=suspend.target) => {"ansible_loop_var": "item", "changed": false, "item": "suspend.target", "skip_reason": "Conditional result was False"} 2021-02-10T11:05:54.489 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item=hibernate.target) => {"ansible_loop_var": "item", "changed": false, "item": "hibernate.target", "skip_reason": "Conditional result was False"} 2021-02-10T11:05:54.498 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item=hybrid-sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "hybrid-sleep.target", "skip_reason": "Conditional result was False"} 2021-02-10T11:05:54.534 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:05:54 +0000 (0:00:00.086) 0:03:11.822 **** 2021-02-10T11:05:54.562 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-02-10T11:05:54.563 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:54.599 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:05:54 +0000 (0:00:00.065) 0:03:11.887 **** 2021-02-10T11:05:54.621 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2021-02-10T11:05:54.622 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:54.655 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:05:54 +0000 (0:00:00.055) 0:03:11.943 **** 2021-02-10T11:05:54.682 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2021-02-10T11:05:54.683 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:54.717 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:05:54 +0000 (0:00:00.061) 0:03:12.005 **** 2021-02-10T11:05:54.744 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:54.777 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:05:54 +0000 (0:00:00.060) 0:03:12.066 **** 2021-02-10T11:05:54.815 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:54.852 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:05:54 +0000 (0:00:00.074) 0:03:12.140 **** 2021-02-10T11:05:55.017 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2021-02-10T11:05:55.018 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d25e11bf71dfd21a39ace708c03408897a4f263", "dest": "/etc/yum.repos.d/epel-testing-mirrorlist", "gid": 0, "group": "root", "item": "epel-testing-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 474, "state": "file", "uid": 0} 2021-02-10T11:05:55.056 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:05:55 +0000 (0:00:00.204) 0:03:12.345 **** 2021-02-10T11:05:55.094 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2021-02-10T11:05:55.095 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:55.134 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:05:55 +0000 (0:00:00.077) 0:03:12.423 **** 2021-02-10T11:05:55.172 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-02-10T11:05:55.173 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:55.211 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:05:55 +0000 (0:00:00.076) 0:03:12.499 **** 2021-02-10T11:05:55.239 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** 2021-02-10T11:05:55.240 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:55.279 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:05:55 +0000 (0:00:00.067) 0:03:12.567 **** 2021-02-10T11:05:55.307 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2021-02-10T11:05:55.308 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:55.341 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:05:55 +0000 (0:00:00.062) 0:03:12.629 **** 2021-02-10T11:05:55.370 INFO:teuthology.task.ansible.out: TASK [common : Mask sleep units] *********************************************** 2021-02-10T11:05:55.372 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item=sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "sleep.target", "skip_reason": "Conditional result was False"} 2021-02-10T11:05:55.381 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item=suspend.target) => {"ansible_loop_var": "item", "changed": false, "item": "suspend.target", "skip_reason": "Conditional result was False"} 2021-02-10T11:05:55.382 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item=hibernate.target) => {"ansible_loop_var": "item", "changed": false, "item": "hibernate.target", "skip_reason": "Conditional result was False"} 2021-02-10T11:05:55.383 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item=hybrid-sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "hybrid-sleep.target", "skip_reason": "Conditional result was False"} 2021-02-10T11:05:55.416 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:05:55 +0000 (0:00:00.075) 0:03:12.705 **** 2021-02-10T11:05:55.444 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-02-10T11:05:55.445 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:55.486 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:05:55 +0000 (0:00:00.069) 0:03:12.775 **** 2021-02-10T11:05:55.514 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** 2021-02-10T11:05:55.514 INFO:teuthology.task.ansible.out:changed: [gibba012.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.226136", "end": "2021-02-10 11:05:55.500559", "rc": 0, "start": "2021-02-10 11:05:55.274423", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-10T11:05:55.560 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:05:55 +0000 (0:00:00.073) 0:03:12.849 **** 2021-02-10T11:05:55.568 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2021-02-10T11:05:55.569 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:55.606 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:05:55 +0000 (0:00:00.045) 0:03:12.894 **** 2021-02-10T11:05:55.633 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2021-02-10T11:05:55.635 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:55.675 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:05:55 +0000 (0:00:00.069) 0:03:12.963 **** 2021-02-10T11:05:55.701 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2021-02-10T11:05:55.703 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:55.734 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:05:55 +0000 (0:00:00.059) 0:03:13.023 **** 2021-02-10T11:05:55.768 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2021-02-10T11:05:55.770 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:55.801 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:05:55 +0000 (0:00:00.066) 0:03:13.089 **** 2021-02-10T11:05:56.384 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2021-02-10T11:05:56.385 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["krb5-workstation-1.15.1-50.el7.x86_64 providing krb5-workstation is already installed"]} 2021-02-10T11:05:56.420 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:05:56 +0000 (0:00:00.618) 0:03:13.708 **** 2021-02-10T11:05:56.421 INFO:teuthology.task.ansible.out: 2021-02-10T11:05:56.449 INFO:teuthology.task.ansible.out: 2021-02-10T11:05:56.450 INFO:teuthology.task.ansible.out:TASK [common : Clean apt cache] ************************************************ skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:56.486 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:05:56 +0000 (0:00:00.066) 0:03:13.774 **** 2021-02-10T11:05:56.514 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** 2021-02-10T11:05:56.515 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:56.549 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:05:56 +0000 (0:00:00.062) 0:03:13.837 **** 2021-02-10T11:05:56.586 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** 2021-02-10T11:05:56.588 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:56.621 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:05:56 +0000 (0:00:00.072) 0:03:13.910 **** 2021-02-10T11:05:56.651 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** 2021-02-10T11:05:56.653 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:56.685 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:05:56 +0000 (0:00:00.064) 0:03:13.974 **** 2021-02-10T11:05:57.513 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** 2021-02-10T11:05:57.515 INFO:teuthology.task.ansible.out:changed: [gibba020.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:01.273328", "end": "2021-02-10 11:05:57.500587", "rc": 0, "start": "2021-02-10 11:05:56.227259", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-10T11:05:57.548 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:05:57 +0000 (0:00:00.862) 0:03:14.836 **** 2021-02-10T11:05:58.353 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2021-02-10T11:05:58.355 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["krb5-workstation-1.15.1-50.el7.x86_64 providing krb5-workstation is already installed"]} 2021-02-10T11:05:58.389 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:05:58 +0000 (0:00:00.841) 0:03:15.678 **** 2021-02-10T11:05:58.417 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2021-02-10T11:05:58.418 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:58.451 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:05:58 +0000 (0:00:00.062) 0:03:15.740 **** 2021-02-10T11:05:58.478 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** 2021-02-10T11:05:58.479 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:58.513 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:05:58 +0000 (0:00:00.061) 0:03:15.801 **** 2021-02-10T11:05:58.539 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** 2021-02-10T11:05:58.541 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:58.573 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:05:58 +0000 (0:00:00.060) 0:03:15.862 **** 2021-02-10T11:05:58.603 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:05:58.604 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* ok: [gibba012.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "secontext": "system_u:object_r:krb5_conf_t:s0", "size": 131, "state": "file", "uid": 0} 2021-02-10T11:05:58.636 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:05:58 +0000 (0:00:00.062) 0:03:15.925 **** 2021-02-10T11:05:58.668 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:05:58 +0000 (0:00:00.031) 0:03:15.956 **** 2021-02-10T11:06:00.577 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "secontext": "system_u:object_r:krb5_conf_t:s0", "size": 131, "state": "file", "uid": 0} 2021-02-10T11:06:00.612 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:06:00 +0000 (0:00:01.943) 0:03:17.900 **** 2021-02-10T11:06:26.417 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** ok: [gibba012.front.sepia.ceph.com] => {"changed": false, "changes": {"installed": [], "updated": []}, "msg": "", "rc": 0, "results": ["All packages providing smartmontools are up to date", ""]} 2021-02-10T11:06:26.452 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:06:26 +0000 (0:00:25.840) 0:03:43.741 **** 2021-02-10T11:06:28.379 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => {"changed": false, "changes": {"installed": [], "updated": []}, "msg": "", "rc": 0, "results": ["All packages providing smartmontools are up to date", ""]} 2021-02-10T11:06:28.411 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:06:28 +0000 (0:00:01.958) 0:03:45.700 **** 2021-02-10T11:06:28.517 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2021-02-10T11:06:28.519 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "secontext": "system_u:object_r:bin_t:s0", "size": 2716736, "state": "file", "uid": 0} 2021-02-10T11:06:30.388 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "secontext": "system_u:object_r:bin_t:s0", "size": 2716736, "state": "file", "uid": 0} 2021-02-10T11:06:30.431 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "secontext": "system_u:object_r:bin_t:s0", "size": 1952038, "state": "file", "uid": 0} 2021-02-10T11:06:32.275 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "secontext": "system_u:object_r:bin_t:s0", "size": 1952038, "state": "file", "uid": 0} 2021-02-10T11:06:32.463 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "secontext": "system_u:object_r:bin_t:s0", "size": 580432, "state": "file", "uid": 0} 2021-02-10T11:06:32.499 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:06:32 +0000 (0:00:04.087) 0:03:49.787 **** 2021-02-10T11:06:32.934 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* 2021-02-10T11:06:32.935 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "secontext": "system_u:object_r:bin_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-02-10T11:06:32.970 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:06:32 +0000 (0:00:00.471) 0:03:50.259 **** 2021-02-10T11:06:34.178 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2021-02-10T11:06:34.179 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "secontext": "system_u:object_r:bin_t:s0", "size": 580432, "state": "file", "uid": 0} 2021-02-10T11:06:34.217 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:06:34 +0000 (0:00:01.246) 0:03:51.505 **** 2021-02-10T11:06:34.655 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* 2021-02-10T11:06:34.655 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "secontext": "system_u:object_r:bin_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-02-10T11:06:34.688 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:06:34 +0000 (0:00:00.471) 0:03:51.977 **** 2021-02-10T11:06:35.593 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2021-02-10T11:06:35.593 INFO:teuthology.task.ansible.out:changed: [gibba012.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": true, "checksum": "00d0404b545463be4a65f475e0870e07a48f35b6", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "md5sum": "14041bf2666d8f8d277567958c7e9f69", "mode": "0755", "owner": "root", "secontext": "system_u:object_r:bin_t:s0", "size": 13355, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1612955192.9947026-40280879729077/source", "state": "file", "uid": 0} 2021-02-10T11:06:37.470 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 8602, "state": "file", "uid": 0} 2021-02-10T11:06:37.751 INFO:teuthology.task.ansible.out:changed: [gibba020.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": true, "checksum": "00d0404b545463be4a65f475e0870e07a48f35b6", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "md5sum": "14041bf2666d8f8d277567958c7e9f69", "mode": "0755", "owner": "root", "secontext": "system_u:object_r:bin_t:s0", "size": 13355, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1612955194.7120407-26003352683244/source", "state": "file", "uid": 0} 2021-02-10T11:06:39.350 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 3605, "state": "file", "uid": 0} 2021-02-10T11:06:39.389 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:06:39 +0000 (0:00:04.700) 0:03:56.677 **** 2021-02-10T11:06:39.416 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** 2021-02-10T11:06:39.418 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:06:39.449 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:06:39 +0000 (0:00:00.060) 0:03:56.738 **** 2021-02-10T11:06:39.487 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** ok: [gibba012.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib64/nagios/plugins", "nrpe_group": "nrpe", "nrpe_packages": ["nagios-common", "nrpe", "nagios-plugins", "nagios-plugins-load"], "nrpe_service_name": "nrpe", "nrpe_user": "nrpe"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/yum_systems.yml"], "changed": false} 2021-02-10T11:06:39.521 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:06:39 +0000 (0:00:00.071) 0:03:56.809 **** 2021-02-10T11:06:39.643 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2021-02-10T11:06:39.645 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 8602, "state": "file", "uid": 0} 2021-02-10T11:06:39.973 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* changed: [gibba012.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2021-02-10T11:06:40.005 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:06:40 +0000 (0:00:00.484) 0:03:57.294 **** 2021-02-10T11:06:40.053 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2021-02-10T11:06:40.055 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-02-10T11:06:40.075 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-02-10T11:06:40.086 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-02-10T11:06:40.096 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-02-10T11:06:40.132 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:06:40 +0000 (0:00:00.126) 0:03:57.420 **** 2021-02-10T11:06:41.524 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2021-02-10T11:06:41.525 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 3605, "state": "file", "uid": 0} 2021-02-10T11:06:41.561 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:06:41 +0000 (0:00:01.429) 0:03:58.850 **** 2021-02-10T11:06:41.588 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** 2021-02-10T11:06:41.589 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:06:41.625 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:06:41 +0000 (0:00:00.064) 0:03:58.914 **** 2021-02-10T11:06:41.662 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** 2021-02-10T11:06:41.663 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib64/nagios/plugins", "nrpe_group": "nrpe", "nrpe_packages": ["nagios-common", "nrpe", "nagios-plugins", "nagios-plugins-load"], "nrpe_service_name": "nrpe", "nrpe_user": "nrpe"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/yum_systems.yml"], "changed": false} 2021-02-10T11:06:41.695 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:06:41 +0000 (0:00:00.069) 0:03:58.984 **** 2021-02-10T11:06:42.138 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2021-02-10T11:06:42.140 INFO:teuthology.task.ansible.out:changed: [gibba020.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2021-02-10T11:06:42.182 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:06:42 +0000 (0:00:00.486) 0:03:59.470 **** 2021-02-10T11:06:42.231 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2021-02-10T11:06:42.232 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-02-10T11:06:42.243 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-02-10T11:06:42.254 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-02-10T11:06:42.265 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-02-10T11:06:42.299 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:06:42 +0000 (0:00:00.117) 0:03:59.587 **** 2021-02-10T11:06:46.413 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2021-02-10T11:06:46.415 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => {"changed": false, "changes": {"installed": [], "updated": []}, "msg": "", "rc": 0, "results": ["All packages providing nagios-common are up to date", "All packages providing nrpe are up to date", "All packages providing nagios-plugins are up to date", "All packages providing nagios-plugins-load are up to date", ""]} 2021-02-10T11:06:46.457 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:06:46 +0000 (0:00:04.158) 0:04:03.746 **** 2021-02-10T11:06:46.485 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu/Fedora)] ********** 2021-02-10T11:06:46.486 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:06:46.539 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:06:46 +0000 (0:00:00.081) 0:04:03.828 **** 2021-02-10T11:06:47.615 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2021-02-10T11:06:47.616 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => {"changed": false, "changes": {"installed": [], "updated": []}, "msg": "", "rc": 0, "results": ["All packages providing nagios-common are up to date", "All packages providing nrpe are up to date", "All packages providing nagios-plugins are up to date", "All packages providing nagios-plugins-load are up to date", ""]} 2021-02-10T11:06:47.650 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:06:47 +0000 (0:00:01.110) 0:04:04.938 **** 2021-02-10T11:06:47.677 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu/Fedora)] ********** 2021-02-10T11:06:47.678 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:06:47.712 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:06:47 +0000 (0:00:00.061) 0:04:05.000 **** 2021-02-10T11:06:48.466 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** ok: [gibba012.front.sepia.ceph.com] => {"changed": false, "checksum": "b423b985ccbb69b65a711d372645a5f704b30e8b", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "secontext": "system_u:object_r:etc_t:s0", "size": 178, "state": "file", "uid": 0} 2021-02-10T11:06:48.500 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:06:48 +0000 (0:00:00.787) 0:04:05.788 **** 2021-02-10T11:06:49.622 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => {"changed": false, "checksum": "b423b985ccbb69b65a711d372645a5f704b30e8b", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "secontext": "system_u:object_r:etc_t:s0", "size": 178, "state": "file", "uid": 0} 2021-02-10T11:06:49.660 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:06:49 +0000 (0:00:01.160) 0:04:06.948 **** 2021-02-10T11:06:50.419 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2021-02-10T11:06:50.421 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib64/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib64/nagios/plugins/check_mem.sh", "secontext": "system_u:object_r:nagios_unconfined_plugin_exec_t:s0", "size": 1964, "state": "file", "uid": 0} 2021-02-10T11:06:50.454 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:06:50 +0000 (0:00:00.794) 0:04:07.743 **** 2021-02-10T11:06:50.493 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2021-02-10T11:06:50.495 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "DAEMON_OPTS", "skip_reason": "Conditional result was False"} 2021-02-10T11:06:50.505 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "NRPE_OPTS", "skip_reason": "Conditional result was False"} 2021-02-10T11:06:50.555 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:06:50 +0000 (0:00:00.100) 0:04:07.843 **** 2021-02-10T11:06:51.015 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* 2021-02-10T11:06:51.017 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-02-10T11:06:51.053 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:06:51 +0000 (0:00:00.498) 0:04:08.341 **** 2021-02-10T11:06:51.547 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** 2021-02-10T11:06:51.549 INFO:teuthology.task.ansible.out:fatal: [gibba012.front.sepia.ceph.com]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} ...ignoring 2021-02-10T11:06:51.588 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:06:51 +0000 (0:00:00.534) 0:04:08.876 **** 2021-02-10T11:06:51.608 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2021-02-10T11:06:51.608 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib64/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib64/nagios/plugins/check_mem.sh", "secontext": "system_u:object_r:nagios_unconfined_plugin_exec_t:s0", "size": 1964, "state": "file", "uid": 0} 2021-02-10T11:06:51.644 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:06:51 +0000 (0:00:00.056) 0:04:08.932 **** 2021-02-10T11:06:51.652 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** 2021-02-10T11:06:51.652 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:06:51.702 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:06:51 +0000 (0:00:00.058) 0:04:08.991 **** 2021-02-10T11:06:51.711 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2021-02-10T11:06:51.713 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "DAEMON_OPTS", "skip_reason": "Conditional result was False"} skipping: [gibba020.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "NRPE_OPTS", "skip_reason": "Conditional result was False"} 2021-02-10T11:06:51.753 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:06:51 +0000 (0:00:00.050) 0:04:09.041 **** 2021-02-10T11:06:52.228 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* 2021-02-10T11:06:52.230 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-02-10T11:06:52.264 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:06:52 +0000 (0:00:00.511) 0:04:09.553 **** 2021-02-10T11:06:52.741 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** 2021-02-10T11:06:52.743 INFO:teuthology.task.ansible.out:fatal: [gibba020.front.sepia.ceph.com]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} ...ignoring 2021-02-10T11:06:52.776 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:06:52 +0000 (0:00:00.510) 0:04:10.064 **** 2021-02-10T11:06:52.804 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** 2021-02-10T11:06:52.805 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:06:52.839 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:06:52 +0000 (0:00:00.063) 0:04:10.127 **** 2021-02-10T11:06:53.634 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* 2021-02-10T11:06:53.636 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => {"changed": false, "checksum": "aa9150d4582f919d7a8504c01c8d86f05e3eeb5d", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "secontext": "system_u:object_r:nrpe_etc_t:s0", "size": 1184, "state": "file", "uid": 0} 2021-02-10T11:06:53.677 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:06:53 +0000 (0:00:00.838) 0:04:10.965 **** 2021-02-10T11:06:54.754 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => {"changed": false, "checksum": "aa9150d4582f919d7a8504c01c8d86f05e3eeb5d", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "secontext": "system_u:object_r:nrpe_etc_t:s0", "size": 1184, "state": "file", "uid": 0} 2021-02-10T11:06:54.791 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:06:54 +0000 (0:00:01.113) 0:04:12.079 **** 2021-02-10T11:06:54.798 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2021-02-10T11:06:54.800 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nrpe", "state": "started", "status": {"ActiveEnterTimestamp": "Wed 2021-02-10 11:01:33 UTC", "ActiveEnterTimestampMonotonic": "21764543", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "time-sync.target systemd-journald.socket system.slice local-fs.target network.target var-run.mount network-online.target nss-lookup.target basic.target", "AllowIsolate": "no", "AmbientCapabilities": "0", "AssertResult": "yes", "AssertTimestamp": "Wed 2021-02-10 11:01:33 UTC", "AssertTimestampMonotonic": "21741195", "Before": "multi-user.target xdm.service shutdown.target getty@tty1.service", "BlockIOAccounting": "no", "BlockIOWeight": "18446744073709551615", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "18446744073709551615", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "18446744073709551615", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Wed 2021-02-10 11:01:33 UTC", "ConditionTimestampMonotonic": "21741195", "Conflicts": "nrpe.socket shutdown.target", "ControlGroup": "/system.slice/nrpe.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Nagios Remote Program Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "EnvironmentFile": "/etc/sysconfig/nrpe (ignore_errors=no)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1188", "ExecMainStartTimestamp": "Wed 2021-02-10 11:01:33 UTC", "ExecMainStartTimestampMonotonic": "21764395", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /e 2021-02-10T11:06:54.801 INFO:teuthology.task.ansible.out:tc/nagios/nrpe.cfg -f $NRPE_SSL_OPT ; ignore_errors=no ; start_time=[Wed 2021-02-10 11:01:33 UTC] ; stop_time=[n/a] ; pid=1188 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nrpe/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/nrpe.service", "Group": "nrpe", "GuessMainPID": "yes", "IOScheduling": "0", "Id": "nrpe.service", "IgnoreOnIsolate": "no", "IgnoreOnSnapshot": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Wed 2021-02-10 11:01:33 UTC", "InactiveExitTimestampMonotonic": "21764543", "JobTimeoutAction": "none", "JobTimeoutUSec": "0", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "18446744073709551615", "LimitCORE": "18446744073709551615", "LimitCPU": "18446744073709551615", "LimitDATA": "18446744073709551615", "LimitFSIZE": "18446744073709551615", "LimitLOCKS": "18446744073709551615", "LimitMEMLOCK": "65536", "LimitMSGQUEUE": "819200", "LimitNICE": "0", "LimitNOFILE": "4096", "LimitNPROC": "127560", "LimitRSS": "18446744073709551615", "LimitRTPRIO": "0", "LimitRTTIME": "18446744073709551615", "LimitSIGPENDING": "127560", "LimitSTACK": "18446744073709551615", "LoadState": "loaded", "MainPID": "1188", "MemoryAccounting": "no", "MemoryCurrent": "18446744073709551615", "MemoryLimit": "18446744073709551615", "MountFlags": "0", "Names": "nrpe.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "ProtectHome": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "Requires": "basic.target system.slice", "Restart": "no", "RestartUSec": "100ms", "Re 2021-02-10T11:06:54.803 INFO:teuthology.task.ansible.out:sult": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitInterval": "10000000", "StartupBlockIOWeight": "18446744073709551615", "StartupCPUShares": "18446744073709551615", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "no", "TasksCurrent": "18446744073709551615", "TasksMax": "18446744073709551615", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "enabled", "User": "nrpe", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Wed 2021-02-10 11:01:33 UTC", "WatchdogTimestampMonotonic": "21764484", "WatchdogUSec": "0"}} 2021-02-10T11:06:54.836 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:06:54 +0000 (0:00:00.045) 0:04:12.124 **** 2021-02-10T11:06:55.295 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* 2021-02-10T11:06:55.296 INFO:teuthology.task.ansible.out:changed: [gibba012.front.sepia.ceph.com] => {"changed": true, "cmd": ["getenforce"], "delta": "0:00:00.001374", "end": "2021-02-10 11:06:55.278806", "rc": 0, "start": "2021-02-10 11:06:55.277432", "stderr": "", "stderr_lines": [], "stdout": "Permissive", "stdout_lines": ["Permissive"]} 2021-02-10T11:06:55.329 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:06:55 +0000 (0:00:00.493) 0:04:12.618 **** 2021-02-10T11:06:55.336 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2021-02-10T11:06:55.337 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nrpe", "state": "started", "status": {"ActiveEnterTimestamp": "Wed 2021-02-10 11:01:31 UTC", "ActiveEnterTimestampMonotonic": "20042155", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "system.slice var-run.mount time-sync.target basic.target local-fs.target network.target network-online.target nss-lookup.target systemd-journald.socket", "AllowIsolate": "no", "AmbientCapabilities": "0", "AssertResult": "yes", "AssertTimestamp": "Wed 2021-02-10 11:01:31 UTC", "AssertTimestampMonotonic": "19958171", "Before": "multi-user.target xdm.service shutdown.target getty@tty1.service", "BlockIOAccounting": "no", "BlockIOWeight": "18446744073709551615", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "18446744073709551615", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "18446744073709551615", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Wed 2021-02-10 11:01:31 UTC", "ConditionTimestampMonotonic": "19958170", "Conflicts": "shutdown.target nrpe.socket", "ControlGroup": "/system.slice/nrpe.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Nagios Remote Program Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "EnvironmentFile": "/etc/sysconfig/nrpe (ignore_errors=no)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1210", "ExecMainStartTimestamp": "Wed 2021-02-10 11:01:31 UTC", "ExecMainStartTimestampMonotonic": "20041954", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /e 2021-02-10T11:06:55.339 INFO:teuthology.task.ansible.out:tc/nagios/nrpe.cfg -f $NRPE_SSL_OPT ; ignore_errors=no ; start_time=[Wed 2021-02-10 11:01:31 UTC] ; stop_time=[n/a] ; pid=1210 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nrpe/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/nrpe.service", "Group": "nrpe", "GuessMainPID": "yes", "IOScheduling": "0", "Id": "nrpe.service", "IgnoreOnIsolate": "no", "IgnoreOnSnapshot": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Wed 2021-02-10 11:01:31 UTC", "InactiveExitTimestampMonotonic": "20042155", "JobTimeoutAction": "none", "JobTimeoutUSec": "0", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "18446744073709551615", "LimitCORE": "18446744073709551615", "LimitCPU": "18446744073709551615", "LimitDATA": "18446744073709551615", "LimitFSIZE": "18446744073709551615", "LimitLOCKS": "18446744073709551615", "LimitMEMLOCK": "65536", "LimitMSGQUEUE": "819200", "LimitNICE": "0", "LimitNOFILE": "4096", "LimitNPROC": "127560", "LimitRSS": "18446744073709551615", "LimitRTPRIO": "0", "LimitRTTIME": "18446744073709551615", "LimitSIGPENDING": "127560", "LimitSTACK": "18446744073709551615", "LoadState": "loaded", "MainPID": "1210", "MemoryAccounting": "no", "MemoryCurrent": "18446744073709551615", "MemoryLimit": "18446744073709551615", "MountFlags": "0", "Names": "nrpe.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "ProtectHome": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "Requires": "system.slice basic.target", "Restart": "no", "RestartUSec": "100ms", "Re 2021-02-10T11:06:55.340 INFO:teuthology.task.ansible.out:sult": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitInterval": "10000000", "StartupBlockIOWeight": "18446744073709551615", "StartupCPUShares": "18446744073709551615", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "no", "TasksCurrent": "18446744073709551615", "TasksMax": "18446744073709551615", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "enabled", "User": "nrpe", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Wed 2021-02-10 11:01:31 UTC", "WatchdogTimestampMonotonic": "20042088", "WatchdogUSec": "0"}} 2021-02-10T11:06:55.373 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:06:55 +0000 (0:00:00.043) 0:04:12.661 **** 2021-02-10T11:06:55.820 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* 2021-02-10T11:06:55.822 INFO:teuthology.task.ansible.out:changed: [gibba020.front.sepia.ceph.com] => {"changed": true, "cmd": ["getenforce"], "delta": "0:00:00.001339", "end": "2021-02-10 11:06:55.808379", "rc": 0, "start": "2021-02-10 11:06:55.807040", "stderr": "", "stderr_lines": [], "stdout": "Permissive", "stdout_lines": ["Permissive"]} 2021-02-10T11:06:55.855 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:06:55 +0000 (0:00:00.482) 0:04:13.144 **** 2021-02-10T11:06:56.150 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Install SELinux dependencies] **************************** 2021-02-10T11:06:56.152 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["libsemanage-python-2.5-14.el7.x86_64 providing libsemanage-python is already installed", "policycoreutils-python-2.5-34.el7.x86_64 providing policycoreutils-python is already installed"]} 2021-02-10T11:06:56.185 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:06:56 +0000 (0:00:00.329) 0:04:13.473 **** 2021-02-10T11:06:56.707 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["libsemanage-python-2.5-14.el7.x86_64 providing libsemanage-python is already installed", "policycoreutils-python-2.5-34.el7.x86_64 providing policycoreutils-python is already installed"]} 2021-02-10T11:06:56.746 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:06:56 +0000 (0:00:00.560) 0:04:14.034 **** 2021-02-10T11:07:00.677 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* 2021-02-10T11:07:00.678 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => {"changed": false, "changes": {"installed": [], "updated": []}, "msg": "", "rc": 0, "results": ["All packages providing selinux-policy-targeted are up to date", ""]} 2021-02-10T11:07:00.713 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:07:00 +0000 (0:00:03.966) 0:04:18.001 **** 2021-02-10T11:07:01.241 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => {"changed": false, "changes": {"installed": [], "updated": []}, "msg": "", "rc": 0, "results": ["All packages providing selinux-policy-targeted are up to date", ""]} 2021-02-10T11:07:01.278 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:07:01 +0000 (0:00:00.564) 0:04:18.566 **** 2021-02-10T11:07:01.758 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** 2021-02-10T11:07:01.759 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => {"changed": false, "name": "nagios_run_sudo", "persistent": true, "state": true} 2021-02-10T11:07:01.792 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:07:01 +0000 (0:00:00.514) 0:04:19.081 **** 2021-02-10T11:07:02.163 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => {"changed": false, "name": "nagios_run_sudo", "persistent": true, "state": true} 2021-02-10T11:07:02.204 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:07:02 +0000 (0:00:00.411) 0:04:19.492 **** 2021-02-10T11:07:07.908 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** 2021-02-10T11:07:07.910 INFO:teuthology.task.ansible.out:changed: [gibba012.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-r", "nrpe"], "delta": "0:00:05.669780", "end": "2021-02-10 11:07:07.890342", "failed_when_result": false, "rc": 0, "start": "2021-02-10 11:07:02.220562", "stderr": "", "stderr_lines": [], "stdout": "libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority).", "stdout_lines": ["libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority)."]} 2021-02-10T11:07:07.954 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:07:07 +0000 (0:00:05.749) 0:04:25.242 **** 2021-02-10T11:07:08.306 INFO:teuthology.task.ansible.out:changed: [gibba020.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-r", "nrpe"], "delta": "0:00:05.644122", "end": "2021-02-10 11:07:08.291068", "failed_when_result": false, "rc": 0, "start": "2021-02-10 11:07:02.646946", "stderr": "", "stderr_lines": [], "stdout": "libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority).", "stdout_lines": ["libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority)."]} 2021-02-10T11:07:08.341 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:07:08 +0000 (0:00:00.387) 0:04:25.629 **** 2021-02-10T11:07:09.913 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** 2021-02-10T11:07:09.915 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => {"changed": false, "checksum": "99d9b2ea41fb2d6613898129718064833d2de428", "dest": "/tmp/nrpe.te", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/tmp/nrpe.te", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 934, "state": "file", "uid": 0} 2021-02-10T11:07:09.948 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:07:09 +0000 (0:00:01.606) 0:04:27.236 **** 2021-02-10T11:07:10.273 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => {"changed": false, "checksum": "99d9b2ea41fb2d6613898129718064833d2de428", "dest": "/tmp/nrpe.te", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/tmp/nrpe.te", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 934, "state": "file", "uid": 0} 2021-02-10T11:07:10.308 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:07:10 +0000 (0:00:00.359) 0:04:27.596 **** 2021-02-10T11:07:10.409 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** 2021-02-10T11:07:10.410 INFO:teuthology.task.ansible.out:changed: [gibba012.front.sepia.ceph.com] => {"changed": true, "cmd": ["checkmodule", "-M", "-m", "-o", "/tmp/nrpe.mod", "/tmp/nrpe.te"], "delta": "0:00:00.014112", "end": "2021-02-10 11:07:10.391279", "rc": 0, "start": "2021-02-10 11:07:10.377167", "stderr": "", "stderr_lines": [], "stdout": "checkmodule: loading policy configuration from /tmp/nrpe.te\ncheckmodule: policy configuration loaded\ncheckmodule: writing binary representation (version 19) to /tmp/nrpe.mod", "stdout_lines": ["checkmodule: loading policy configuration from /tmp/nrpe.te", "checkmodule: policy configuration loaded", "checkmodule: writing binary representation (version 19) to /tmp/nrpe.mod"]} 2021-02-10T11:07:10.468 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:07:10 +0000 (0:00:00.159) 0:04:27.756 **** 2021-02-10T11:07:10.767 INFO:teuthology.task.ansible.out:changed: [gibba020.front.sepia.ceph.com] => {"changed": true, "cmd": ["checkmodule", "-M", "-m", "-o", "/tmp/nrpe.mod", "/tmp/nrpe.te"], "delta": "0:00:00.014266", "end": "2021-02-10 11:07:10.755291", "rc": 0, "start": "2021-02-10 11:07:10.741025", "stderr": "", "stderr_lines": [], "stdout": "checkmodule: loading policy configuration from /tmp/nrpe.te\ncheckmodule: policy configuration loaded\ncheckmodule: writing binary representation (version 19) to /tmp/nrpe.mod", "stdout_lines": ["checkmodule: loading policy configuration from /tmp/nrpe.te", "checkmodule: policy configuration loaded", "checkmodule: writing binary representation (version 19) to /tmp/nrpe.mod"]} 2021-02-10T11:07:10.801 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:07:10 +0000 (0:00:00.333) 0:04:28.089 **** 2021-02-10T11:07:10.926 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** 2021-02-10T11:07:10.926 INFO:teuthology.task.ansible.out:changed: [gibba012.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule_package", "-o", "/tmp/nrpe.pp", "-m", "/tmp/nrpe.mod"], "delta": "0:00:00.012111", "end": "2021-02-10 11:07:10.911484", "rc": 0, "start": "2021-02-10 11:07:10.899373", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-10T11:07:10.974 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:07:10 +0000 (0:00:00.173) 0:04:28.263 **** 2021-02-10T11:07:11.252 INFO:teuthology.task.ansible.out:changed: [gibba020.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule_package", "-o", "/tmp/nrpe.pp", "-m", "/tmp/nrpe.mod"], "delta": "0:00:00.004851", "end": "2021-02-10 11:07:11.240375", "rc": 0, "start": "2021-02-10 11:07:11.235524", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-10T11:07:11.292 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:07:11 +0000 (0:00:00.317) 0:04:28.580 **** 2021-02-10T11:07:17.039 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** 2021-02-10T11:07:17.041 INFO:teuthology.task.ansible.out:changed: [gibba012.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-i", "/tmp/nrpe.pp"], "delta": "0:00:05.612403", "end": "2021-02-10 11:07:17.019271", "rc": 0, "start": "2021-02-10 11:07:11.406868", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-10T11:07:17.075 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:07:17 +0000 (0:00:05.783) 0:04:34.364 **** 2021-02-10T11:07:17.306 INFO:teuthology.task.ansible.out:changed: [gibba020.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-i", "/tmp/nrpe.pp"], "delta": "0:00:05.572222", "end": "2021-02-10 11:07:17.294162", "rc": 0, "start": "2021-02-10 11:07:11.721940", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-10T11:07:17.340 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:07:17 +0000 (0:00:00.264) 0:04:34.629 **** 2021-02-10T11:07:17.558 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** 2021-02-10T11:07:17.559 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => {"changed": false, "path": "/tmp/nrpe.*", "state": "absent"} 2021-02-10T11:07:17.593 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:07:17 +0000 (0:00:00.252) 0:04:34.881 **** 2021-02-10T11:07:17.631 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2021-02-10T11:07:17.632 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => {"ansible_facts": {"ceph_dependency_packages_to_remove": ["boost-random", "boost-program-options", "leveldb", "xmlstarlet", "python-jinja2", "python-ceph", "python-flask", "python-requests", "boost-random", "python-urllib3", "python-babel", "hdparm", "python-markupsafe", "python-werkzeug", "python-itsdangerous"], "ceph_packages_to_remove": ["ceph", "ceph-base", "ceph-selinux", "ceph-common", "ceph-debuginfo", "ceph-release", "libcephfs1", "ceph-radosgw", "python-ceph", "python-rados", "python-rbd", "python-cephfs", "librbd1", "librados2", "mod_fastcgi"], "ntp_service_name": "ntpd", "packages_to_remove": ["device-mapper-multipath"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "sshd"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/yum_systems.yml"], "changed": false} 2021-02-10T11:07:17.666 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:07:17 +0000 (0:00:00.073) 0:04:34.954 **** 2021-02-10T11:07:17.715 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2021-02-10T11:07:17.717 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2021-02-10T11:07:17.750 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:07:17 +0000 (0:00:00.084) 0:04:35.039 **** 2021-02-10T11:07:17.789 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2021-02-10T11:07:17.791 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/redhat_7.yml) => {"ansible_facts": {"common_yum_repos": {"lab-extras": {"baseurl": "http://{{ mirror_host }}/lab-extras/rhel7/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}, "rhel-7-fcgi-ceph": {"baseurl": "http://{{ gitbuilder_host }}/mod_fastcgi-rpm-rhel7-x86_64-basic/ref/master/", "enabled": 1, "gpgcheck": 0, "name": "RHEL 7 Local fastcgi Repo"}}, "epel_packages": ["cryptopp-devel", "cryptopp", "dbench", "fcgi", "fuse-sshfs", "perl-JSON-XS", "leveldb", "xmlstarlet"], "nfs_service": "nfs-server", "packages": ["@core", "@base", "yum-plugin-priorities", "yum-plugin-fastestmirror", "redhat-lsb", "sysstat", "gdb", "git-all", "python-configobj", "gcc-c++", "libedit", "openssl098e", "boost-thread", "xfsprogs", "xfsprogs-devel", "gdisk", "parted", "libgcrypt", "fuse", "fuse-libs", "lvm2", "openssl", "libuuid", "btrfs-progs", "attr", "valgrind", "python-nose", "mpich", "ant", "lsof", "iozone", "libtool", "automake", "gettext", "libuuid-devel", "libacl-devel", "bc", "xfsdump", "blktrace", "numpy", "python-matplotlib", "qemu-kvm", "usbredir", "genisoimage", "httpd", "httpd-devel", "httpd-tools", "mod_ssl", "mod_fastcgi-2.4.7-1.ceph.el7", "libev-devel", "perl-XML-Twig", "java-1.6.0-openjdk-devel", "junit4", "nfs-utils", "ncurses-devel", "python-devel", "python-virtualenv", "perl-CPAN", "podman", "podman-docker", "python3"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/redhat_7.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/redhat_7.yml"} 2021-02-10T11:07:17.803 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** 2021-02-10T11:07:17.804 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => {"changed": false, "path": "/tmp/nrpe.*", "state": "absent"} 2021-02-10T11:07:17.837 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:07:17 +0000 (0:00:00.086) 0:04:35.125 **** 2021-02-10T11:07:17.870 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:07:17 +0000 (0:00:00.033) 0:04:35.159 **** 2021-02-10T11:07:17.888 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2021-02-10T11:07:17.889 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2021-02-10T11:07:17.926 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:07:17 +0000 (0:00:00.055) 0:04:35.214 **** 2021-02-10T11:07:17.932 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2021-02-10T11:07:17.933 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => {"ansible_facts": {"ceph_dependency_packages_to_remove": ["boost-random", "boost-program-options", "leveldb", "xmlstarlet", "python-jinja2", "python-ceph", "python-flask", "python-requests", "boost-random", "python-urllib3", "python-babel", "hdparm", "python-markupsafe", "python-werkzeug", "python-itsdangerous"], "ceph_packages_to_remove": ["ceph", "ceph-base", "ceph-selinux", "ceph-common", "ceph-debuginfo", "ceph-release", "libcephfs1", "ceph-radosgw", "python-ceph", "python-rados", "python-rbd", "python-cephfs", "librbd1", "librados2", "mod_fastcgi"], "ntp_service_name": "ntpd", "packages_to_remove": ["device-mapper-multipath"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "sshd"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/yum_systems.yml"], "changed": false} 2021-02-10T11:07:17.971 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:07:17 +0000 (0:00:00.044) 0:04:35.259 **** 2021-02-10T11:07:18.014 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2021-02-10T11:07:18.016 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2021-02-10T11:07:18.061 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:07:18 +0000 (0:00:00.090) 0:04:35.349 **** 2021-02-10T11:07:18.109 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2021-02-10T11:07:18.111 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/redhat_7.yml) => {"ansible_facts": {"common_yum_repos": {"lab-extras": {"baseurl": "http://{{ mirror_host }}/lab-extras/rhel7/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}, "rhel-7-fcgi-ceph": {"baseurl": "http://{{ gitbuilder_host }}/mod_fastcgi-rpm-rhel7-x86_64-basic/ref/master/", "enabled": 1, "gpgcheck": 0, "name": "RHEL 7 Local fastcgi Repo"}}, "epel_packages": ["cryptopp-devel", "cryptopp", "dbench", "fcgi", "fuse-sshfs", "perl-JSON-XS", "leveldb", "xmlstarlet"], "nfs_service": "nfs-server", "packages": ["@core", "@base", "yum-plugin-priorities", "yum-plugin-fastestmirror", "redhat-lsb", "sysstat", "gdb", "git-all", "python-configobj", "gcc-c++", "libedit", "openssl098e", "boost-thread", "xfsprogs", "xfsprogs-devel", "gdisk", "parted", "libgcrypt", "fuse", "fuse-libs", "lvm2", "openssl", "libuuid", "btrfs-progs", "attr", "valgrind", "python-nose", "mpich", "ant", "lsof", "iozone", "libtool", "automake", "gettext", "libuuid-devel", "libacl-devel", "bc", "xfsdump", "blktrace", "numpy", "python-matplotlib", "qemu-kvm", "usbredir", "genisoimage", "httpd", "httpd-devel", "httpd-tools", "mod_ssl", "mod_fastcgi-2.4.7-1.ceph.el7", "libev-devel", "perl-XML-Twig", "java-1.6.0-openjdk-devel", "junit4", "nfs-utils", "ncurses-devel", "python-devel", "python-virtualenv", "perl-CPAN", "podman", "podman-docker", "python3"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/redhat_7.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/redhat_7.yml"} 2021-02-10T11:07:18.144 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:07:18 +0000 (0:00:00.083) 0:04:35.432 **** 2021-02-10T11:07:18.187 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2021-02-10T11:07:18.187 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2021-02-10T11:07:18.222 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:07:18 +0000 (0:00:00.078) 0:04:35.511 **** 2021-02-10T11:07:18.377 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** 2021-02-10T11:07:18.379 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2021-02-10T11:07:18.418 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:07:18 +0000 (0:00:00.195) 0:04:35.707 **** 2021-02-10T11:07:18.669 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2021-02-10T11:07:18.705 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:07:18 +0000 (0:00:00.286) 0:04:35.993 **** 2021-02-10T11:07:18.872 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2021-02-10T11:07:18.873 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => {"changed": false, "gid": 1236, "name": "ubuntu", "state": "present", "system": false} 2021-02-10T11:07:18.908 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:07:18 +0000 (0:00:00.202) 0:04:36.196 **** 2021-02-10T11:07:18.937 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2021-02-10T11:07:18.939 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:07:18.976 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:07:18 +0000 (0:00:00.068) 0:04:36.265 **** 2021-02-10T11:07:19.006 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2021-02-10T11:07:19.008 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:07:19.040 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:07:19 +0000 (0:00:00.063) 0:04:36.328 **** 2021-02-10T11:07:19.153 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2021-02-10T11:07:19.155 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => {"changed": false, "gid": 1236, "name": "ubuntu", "state": "present", "system": false} 2021-02-10T11:07:19.197 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:07:19 +0000 (0:00:00.157) 0:04:36.486 **** 2021-02-10T11:07:19.224 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2021-02-10T11:07:19.226 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:07:19.257 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:07:19 +0000 (0:00:00.059) 0:04:36.546 **** 2021-02-10T11:07:19.284 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2021-02-10T11:07:19.285 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:07:19.317 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:07:19 +0000 (0:00:00.059) 0:04:36.606 **** 2021-02-10T11:07:19.507 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** 2021-02-10T11:07:19.509 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/bash", "state": "present", "uid": 10101} 2021-02-10T11:07:19.543 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:07:19 +0000 (0:00:00.225) 0:04:36.831 **** 2021-02-10T11:07:19.544 INFO:teuthology.task.ansible.out: 2021-02-10T11:07:19.793 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/bash", "state": "present", "uid": 10101} 2021-02-10T11:07:19.831 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:07:19 +0000 (0:00:00.287) 0:04:37.119 **** 2021-02-10T11:07:21.568 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2021-02-10T11:07:21.570 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 112, "state": "file", "uid": 0} 2021-02-10T11:07:21.606 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:07:21 +0000 (0:00:01.774) 0:04:38.894 **** 2021-02-10T11:07:21.636 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* 2021-02-10T11:07:21.637 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "gibba012"}, "changed": false} 2021-02-10T11:07:21.670 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:07:21 +0000 (0:00:00.064) 0:04:38.958 **** 2021-02-10T11:07:21.826 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2021-02-10T11:07:21.826 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 112, "state": "file", "uid": 0} 2021-02-10T11:07:21.867 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:07:21 +0000 (0:00:00.196) 0:04:39.155 **** 2021-02-10T11:07:21.906 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* 2021-02-10T11:07:21.907 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "gibba020"}, "changed": false} 2021-02-10T11:07:21.944 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:07:21 +0000 (0:00:00.077) 0:04:39.232 **** 2021-02-10T11:07:22.796 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ 2021-02-10T11:07:22.798 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "", "ansible_fqdn": "gibba012", "ansible_hostname": "gibba012", "ansible_nodename": "gibba012"}, "changed": false, "name": "gibba012"} ok: [gibba020.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "", "ansible_fqdn": "gibba020", "ansible_hostname": "gibba020", "ansible_nodename": "gibba020"}, "changed": false, "name": "gibba020"} 2021-02-10T11:07:22.830 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:07:22 +0000 (0:00:00.885) 0:04:40.118 **** 2021-02-10T11:07:22.862 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:07:22 +0000 (0:00:00.031) 0:04:40.150 **** 2021-02-10T11:07:24.763 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* 2021-02-10T11:07:24.764 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => {"changed": false, "checksum": "a5e550b2aa74f6355e926d70d73a50ad9ab4b4be", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "secontext": "system_u:object_r:etc_t:s0", "size": 1043, "state": "file", "uid": 0} 2021-02-10T11:07:24.804 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:07:24 +0000 (0:00:01.942) 0:04:42.092 **** 2021-02-10T11:07:24.816 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => {"changed": false, "checksum": "a5e550b2aa74f6355e926d70d73a50ad9ab4b4be", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "secontext": "system_u:object_r:etc_t:s0", "size": 1043, "state": "file", "uid": 0} 2021-02-10T11:07:24.864 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:07:24 +0000 (0:00:00.059) 0:04:42.152 **** 2021-02-10T11:07:26.722 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** 2021-02-10T11:07:26.724 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "secontext": "system_u:object_r:etc_t:s0", "size": 570, "state": "file", "uid": 0} 2021-02-10T11:07:26.768 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:07:26 +0000 (0:00:01.903) 0:04:44.056 **** 2021-02-10T11:07:26.800 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "secontext": "system_u:object_r:etc_t:s0", "size": 570, "state": "file", "uid": 0} 2021-02-10T11:07:26.846 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:07:26 +0000 (0:00:00.078) 0:04:44.134 **** 2021-02-10T11:07:27.572 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** 2021-02-10T11:07:27.573 INFO:teuthology.task.ansible.out:changed: [gibba012.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2021-02-10T11:07:27.612 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:07:27 +0000 (0:00:00.766) 0:04:44.901 **** 2021-02-10T11:07:27.620 INFO:teuthology.task.ansible.out:changed: [gibba020.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2021-02-10T11:07:27.656 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:07:27 +0000 (0:00:00.043) 0:04:44.944 **** 2021-02-10T11:07:29.584 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* 2021-02-10T11:07:29.585 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => {"changed": false, "checksum": "68d0b8ee2f88a77895f257aa69f32a42e47f455b", "dest": "/etc/security/limits.d/remote.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.d/remote.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 116, "state": "file", "uid": 0} 2021-02-10T11:07:29.640 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:07:29 +0000 (0:00:01.984) 0:04:46.929 **** 2021-02-10T11:07:29.648 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => {"changed": false, "checksum": "68d0b8ee2f88a77895f257aa69f32a42e47f455b", "dest": "/etc/security/limits.d/remote.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.d/remote.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 116, "state": "file", "uid": 0} 2021-02-10T11:07:29.684 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:07:29 +0000 (0:00:00.043) 0:04:46.972 **** 2021-02-10T11:07:30.113 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-02-10T11:07:30.114 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => {"changed": false, "dest": "/etc/fuse.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 0, "state": "file", "uid": 0} 2021-02-10T11:07:30.156 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:07:30 +0000 (0:00:00.471) 0:04:47.444 **** 2021-02-10T11:07:30.167 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => {"changed": false, "dest": "/etc/fuse.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 0, "state": "file", "uid": 0} 2021-02-10T11:07:30.210 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:07:30 +0000 (0:00:00.054) 0:04:47.499 **** 2021-02-10T11:07:30.611 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2021-02-10T11:07:30.613 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => {"changed": false, "gid": 10102, "name": "kvm", "state": "present", "system": false} 2021-02-10T11:07:30.658 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:07:30 +0000 (0:00:00.446) 0:04:47.946 **** 2021-02-10T11:07:30.677 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => {"changed": false, "gid": 10102, "name": "kvm", "state": "present", "system": false} 2021-02-10T11:07:30.734 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:07:30 +0000 (0:00:00.076) 0:04:48.022 **** 2021-02-10T11:07:31.145 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-02-10T11:07:31.146 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1236, "groups": "kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2021-02-10T11:07:31.208 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:07:31 +0000 (0:00:00.474) 0:04:48.497 **** 2021-02-10T11:07:31.217 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1236, "groups": "kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2021-02-10T11:07:31.257 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:07:31 +0000 (0:00:00.048) 0:04:48.545 **** 2021-02-10T11:07:33.249 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2021-02-10T11:07:33.250 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => {"changed": false, "checksum": "be3223c8b0917bde02657a5f9ddd95277cc3f812", "dest": "/etc/sudoers", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers", "secontext": "system_u:object_r:etc_t:s0", "size": 1808, "state": "file", "uid": 0} 2021-02-10T11:07:33.302 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:07:33 +0000 (0:00:02.045) 0:04:50.591 **** 2021-02-10T11:07:33.310 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => {"changed": false, "checksum": "be3223c8b0917bde02657a5f9ddd95277cc3f812", "dest": "/etc/sudoers", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers", "secontext": "system_u:object_r:etc_t:s0", "size": 1808, "state": "file", "uid": 0} 2021-02-10T11:07:33.360 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:07:33 +0000 (0:00:00.056) 0:04:50.648 **** 2021-02-10T11:07:35.316 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-02-10T11:07:35.318 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => {"changed": false, "checksum": "12a2e17405b9e00ea2061e25929905d08e5fe19f", "dest": "/etc/security/limits.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 854, "state": "file", "uid": 0} 2021-02-10T11:07:35.363 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:07:35 +0000 (0:00:02.003) 0:04:52.651 **** 2021-02-10T11:07:35.374 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => {"changed": false, "checksum": "12a2e17405b9e00ea2061e25929905d08e5fe19f", "dest": "/etc/security/limits.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 854, "state": "file", "uid": 0} 2021-02-10T11:07:35.415 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:07:35 +0000 (0:00:00.051) 0:04:52.703 **** 2021-02-10T11:07:37.302 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-02-10T11:07:37.303 INFO:teuthology.task.ansible.out:[WARNING]: Consider using the yum, dnf or zypper module rather than running 'rpm'. If you need to use command because yum, dnf or zypper is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. 2021-02-10T11:07:37.304 INFO:teuthology.task.ansible.out:changed: [gibba020.front.sepia.ceph.com] => {"changed": true, "cmd": ["rpm", "--rebuilddb"], "delta": "0:00:01.481590", "end": "2021-02-10 11:07:37.285968", "rc": 0, "start": "2021-02-10 11:07:35.804378", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-10T11:07:37.342 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:07:37 +0000 (0:00:01.927) 0:04:54.630 **** 2021-02-10T11:07:37.380 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** 2021-02-10T11:07:37.382 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:07:37.417 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:07:37 +0000 (0:00:00.075) 0:04:54.706 **** 2021-02-10T11:07:37.424 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-02-10T11:07:37.425 INFO:teuthology.task.ansible.out:changed: [gibba012.front.sepia.ceph.com] => {"changed": true, "cmd": ["rpm", "--rebuilddb"], "delta": "0:00:01.553857", "end": "2021-02-10 11:07:37.401908", "rc": 0, "start": "2021-02-10 11:07:35.848051", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-10T11:07:37.461 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:07:37 +0000 (0:00:00.043) 0:04:54.749 **** 2021-02-10T11:07:37.489 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** 2021-02-10T11:07:37.491 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:07:37.523 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:07:37 +0000 (0:00:00.062) 0:04:54.812 **** 2021-02-10T11:07:39.367 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2021-02-10T11:07:39.368 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'key': 'rhel-7-fcgi-ceph', 'value': {'name': 'RHEL 7 Local fastcgi Repo', 'baseurl': 'http://gitbuilder.ceph.com/mod_fastcgi-rpm-rhel7-x86_64-basic/ref/master/', 'enabled': 1, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "e6aba9c9ab05e27255ed430f065502af284a869b", "dest": "/etc/yum.repos.d/rhel-7-fcgi-ceph.repo", "gid": 0, "group": "root", "item": {"key": "rhel-7-fcgi-ceph", "value": {"baseurl": "http://gitbuilder.ceph.com/mod_fastcgi-rpm-rhel7-x86_64-basic/ref/master/", "enabled": 1, "gpgcheck": 0, "name": "RHEL 7 Local fastcgi Repo"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/rhel-7-fcgi-ceph.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 245, "state": "file", "uid": 0} 2021-02-10T11:07:39.464 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'key': 'rhel-7-fcgi-ceph', 'value': {'name': 'RHEL 7 Local fastcgi Repo', 'baseurl': 'http://gitbuilder.ceph.com/mod_fastcgi-rpm-rhel7-x86_64-basic/ref/master/', 'enabled': 1, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "e6aba9c9ab05e27255ed430f065502af284a869b", "dest": "/etc/yum.repos.d/rhel-7-fcgi-ceph.repo", "gid": 0, "group": "root", "item": {"key": "rhel-7-fcgi-ceph", "value": {"baseurl": "http://gitbuilder.ceph.com/mod_fastcgi-rpm-rhel7-x86_64-basic/ref/master/", "enabled": 1, "gpgcheck": 0, "name": "RHEL 7 Local fastcgi Repo"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/rhel-7-fcgi-ceph.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 245, "state": "file", "uid": 0} 2021-02-10T11:07:41.256 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'key': 'lab-extras', 'value': {'name': 'lab-extras', 'baseurl': 'http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/', 'enabled': 1, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "e3b44e00721d55ceb436a2b7c0ec58cd5bc39dc9", "dest": "/etc/yum.repos.d/lab-extras.repo", "gid": 0, "group": "root", "item": {"key": "lab-extras", "value": {"baseurl": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/lab-extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 207, "state": "file", "uid": 0} 2021-02-10T11:07:41.292 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:07:41 +0000 (0:00:03.768) 0:04:58.580 **** 2021-02-10T11:07:41.321 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2021-02-10T11:07:41.357 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:07:41 +0000 (0:00:00.064) 0:04:58.645 **** 2021-02-10T11:07:41.363 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2021-02-10T11:07:41.364 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'key': 'lab-extras', 'value': {'name': 'lab-extras', 'baseurl': 'http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/', 'enabled': 1, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "e3b44e00721d55ceb436a2b7c0ec58cd5bc39dc9", "dest": "/etc/yum.repos.d/lab-extras.repo", "gid": 0, "group": "root", "item": {"key": "lab-extras", "value": {"baseurl": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/lab-extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 207, "state": "file", "uid": 0} 2021-02-10T11:07:41.401 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:07:41 +0000 (0:00:00.043) 0:04:58.689 **** 2021-02-10T11:07:41.408 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** 2021-02-10T11:07:41.443 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:07:41 +0000 (0:00:00.042) 0:04:58.732 **** 2021-02-10T11:07:41.450 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2021-02-10T11:07:41.487 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:07:41 +0000 (0:00:00.043) 0:04:58.775 **** 2021-02-10T11:07:41.494 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** 2021-02-10T11:07:41.495 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:07:41.530 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:07:41 +0000 (0:00:00.043) 0:04:58.818 **** 2021-02-10T11:07:41.537 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** 2021-02-10T11:07:41.572 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:07:41 +0000 (0:00:00.042) 0:04:58.861 **** 2021-02-10T11:07:41.579 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2021-02-10T11:07:41.579 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2021-02-10T11:07:41.580 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2021-02-10T11:07:41.624 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:07:41 +0000 (0:00:00.051) 0:04:58.913 **** 2021-02-10T11:07:41.631 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** 2021-02-10T11:07:41.632 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:07:41.668 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:07:41 +0000 (0:00:00.043) 0:04:58.956 **** 2021-02-10T11:07:41.674 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** 2021-02-10T11:07:41.676 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:07:41.719 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:07:41 +0000 (0:00:00.050) 0:04:59.007 **** 2021-02-10T11:07:41.719 INFO:teuthology.task.ansible.out: 2021-02-10T11:07:41.728 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2021-02-10T11:07:41.729 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2021-02-10T11:07:41.730 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2021-02-10T11:07:41.772 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:07:41 +0000 (0:00:00.054) 0:04:59.061 **** 2021-02-10T11:07:41.779 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-10T11:07:41.779 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:07:41.815 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:07:41 +0000 (0:00:00.042) 0:04:59.104 **** 2021-02-10T11:07:41.822 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** 2021-02-10T11:07:41.823 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:07:41.859 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:07:41 +0000 (0:00:00.043) 0:04:59.147 **** 2021-02-10T11:07:41.887 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-10T11:07:41.888 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:07:41.922 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:07:41 +0000 (0:00:00.063) 0:04:59.210 **** 2021-02-10T11:07:42.699 INFO:teuthology.task.ansible.out: 2021-02-10T11:07:42.700 INFO:teuthology.task.ansible.out:TASK [testnode : Ensure yum_utils is present.] ********************************* ok: [gibba020.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["yum-utils-1.1.31-54.el7_8.noarch providing yum-utils is already installed"]} 2021-02-10T11:07:42.734 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:07:42 +0000 (0:00:00.811) 0:05:00.022 **** 2021-02-10T11:07:42.762 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["yum-utils-1.1.31-54.el7_8.noarch providing yum-utils is already installed"]} 2021-02-10T11:07:42.797 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:07:42 +0000 (0:00:00.062) 0:05:00.085 **** 2021-02-10T11:07:43.825 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** 2021-02-10T11:07:43.826 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => {"changed": false, "cmd": ["yum-complete-transaction", "--cleanup-only"], "delta": "0:00:00.634362", "end": "2021-02-10 11:07:43.807556", "rc": 0, "start": "2021-02-10 11:07:43.173194", "stderr": "", "stderr_lines": [], "stdout": "Loaded plugins: fastestmirror, langpacks, priorities, product-id, subscription-\n : manager\nLoading mirror speeds from cached hostfile\n * epel: download-cc-rdu01.fedoraproject.org\nNo unfinished transactions left.", "stdout_lines": ["Loaded plugins: fastestmirror, langpacks, priorities, product-id, subscription-", " : manager", "Loading mirror speeds from cached hostfile", " * epel: download-cc-rdu01.fedoraproject.org", "No unfinished transactions left."]} 2021-02-10T11:07:43.860 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:07:43 +0000 (0:00:01.062) 0:05:01.148 **** 2021-02-10T11:07:43.877 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => {"changed": false, "cmd": ["yum-complete-transaction", "--cleanup-only"], "delta": "0:00:00.617762", "end": "2021-02-10 11:07:43.856362", "rc": 0, "start": "2021-02-10 11:07:43.238600", "stderr": "", "stderr_lines": [], "stdout": "Loaded plugins: fastestmirror, langpacks, priorities, product-id, subscription-\n : manager\nLoading mirror speeds from cached hostfile\n * epel: download-cc-rdu01.fedoraproject.org\nNo unfinished transactions left.", "stdout_lines": ["Loaded plugins: fastestmirror, langpacks, priorities, product-id, subscription-", " : manager", "Loading mirror speeds from cached hostfile", " * epel: download-cc-rdu01.fedoraproject.org", "No unfinished transactions left."]} 2021-02-10T11:07:43.914 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:07:43 +0000 (0:00:00.053) 0:05:01.202 **** 2021-02-10T11:07:44.330 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2021-02-10T11:07:44.330 INFO:teuthology.task.ansible.out:fatal: [gibba020.front.sepia.ceph.com]: FAILED! => {"changed": false, "cmd": ["rpm", "-q", "ceph-debuginfo"], "delta": "0:00:00.017894", "end": "2021-02-10 11:07:44.316565", "msg": "non-zero return code", "rc": 1, "start": "2021-02-10 11:07:44.298671", "stderr": "", "stderr_lines": [], "stdout": "package ceph-debuginfo is not installed", "stdout_lines": ["package ceph-debuginfo is not installed"]} ...ignoring 2021-02-10T11:07:44.375 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:07:44 +0000 (0:00:00.461) 0:05:01.663 **** 2021-02-10T11:07:44.394 INFO:teuthology.task.ansible.out:fatal: [gibba012.front.sepia.ceph.com]: FAILED! => {"changed": false, "cmd": ["rpm", "-q", "ceph-debuginfo"], "delta": "0:00:00.017760", "end": "2021-02-10 11:07:44.372334", "msg": "non-zero return code", "rc": 1, "start": "2021-02-10 11:07:44.354574", "stderr": "", "stderr_lines": [], "stdout": "package ceph-debuginfo is not installed", "stdout_lines": ["package ceph-debuginfo is not installed"]} ...ignoring 2021-02-10T11:07:44.429 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:07:44 +0000 (0:00:00.053) 0:05:01.717 **** 2021-02-10T11:07:44.436 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2021-02-10T11:07:44.437 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:07:44.471 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:07:44 +0000 (0:00:00.042) 0:05:01.760 **** 2021-02-10T11:07:44.478 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:07:44.513 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:07:44 +0000 (0:00:00.041) 0:05:01.802 **** 2021-02-10T11:07:53.673 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2021-02-10T11:07:53.673 INFO:teuthology.task.ansible.out:changed: [gibba020.front.sepia.ceph.com] => {"changed": true, "changes": {"removed": ["librbd1", "librados2", "mod_fastcgi"]}, "msg": "", "rc": 0, "results": ["ceph is not installed", "ceph-base is not installed", "ceph-selinux is not installed", "ceph-common is not installed", "ceph-debuginfo is not installed", "ceph-release is not installed", "libcephfs1 is not installed", "ceph-radosgw is not installed", "python-ceph is not installed", "python-rados is not installed", "python-rbd is not installed", "python-cephfs is not installed", "Loaded plugins: fastestmirror, langpacks, priorities, product-id, search-\n : disabled-repos, subscription-manager\nResolving Dependencies\n--> Running transaction check\n---> Package librados2.x86_64 1:10.2.5-4.el7 will be erased\n--> Processing Dependency: librados.so.2()(64bit) for package: 10:qemu-img-1.5.3-175.el7_9.1.x86_64\n--> Processing Dependency: librados.so.2()(64bit) for package: 10:qemu-kvm-common-1.5.3-175.el7_9.1.x86_64\n--> Processing Dependency: librados.so.2()(64bit) for package: 10:qemu-kvm-1.5.3-175.el7_9.1.x86_64\n---> Package librbd1.x86_64 1:10.2.5-4.el7 will be erased\n---> Package mod_fastcgi.x86_64 0:2.4.7-1.ceph.el7 will be erased\n--> Running transaction check\n---> Package qemu-img.x86_64 10:1.5.3-175.el7_9.1 will be erased\n---> Package qemu-kvm.x86_64 10:1.5.3-175.el7_9.1 will be erased\n---> Package qemu-kvm-common.x86_64 10:1.5.3-175.el7_9.1 will be erased\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nRemoving:\n librados2 x86_64 1:10.2.5-4.el7 @rhel-7-server-rpms 5.5 M\n librbd1 x86_64 1:10.2.5-4.el7 @rhel-7-server-rpms 7.4 M\n mod_fastcgi x86_64 2.4.7-1.ceph.el7 @rhel-7- 2021-02-10T11:07:53.675 INFO:teuthology.task.ansible.out:fcgi-ceph 213 k\nRemoving for dependencies:\n qemu-img x86_64 10:1.5.3-175.el7_9.1 @rhel-7-server-rpms 2.3 M\n qemu-kvm x86_64 10:1.5.3-175.el7_9.1 @rhel-7-server-rpms 5.7 M\n qemu-kvm-common x86_64 10:1.5.3-175.el7_9.1 @rhel-7-server-rpms 755 k\n\nTransaction Summary\n================================================================================\nRemove 3 Packages (+3 Dependent packages)\n\nInstalled size: 22 M\nDownloading packages:\nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Erasing : 10:qemu-kvm-1.5.3-175.el7_9.1.x86_64 1/6 \n Erasing : 10:qemu-img-1.5.3-175.el7_9.1.x86_64 2/6 \n Erasing : 10:qemu-kvm-common-1.5.3-175.el7_9.1.x86_64 3/6 \n Erasing : 1:librbd1-10.2.5-4.el7.x86_64 4/6 \n Erasing : 1:librados2-10.2.5-4.el7.x86_64 5/6 \n Erasing : mod_fastcgi-2.4.7-1.ceph.el7.x86_64 6/6 \nwarning: file /var/run/mod_fastcgi: remove failed: No such file or directory\nLoaded plugins: fastestmirror, langpacks, priorities, product-id, subscription-\n : manager\n Verifying : 1:librbd1-10.2.5-4.el7.x86_64 1/6 \n Verifying : 10:qemu-kvm-1.5.3-175.el7_9.1.x86_64 2/6 \n Verifying : mod_fastcgi-2.4.7-1.ceph.el7.x86_64 3/6 \n Verifying : 10:qemu-kvm-common-1.5.3-175.el7_9.1.x86_64 4/6 \n Verifying : 1:librados2-10.2.5-4.el7.x86_64 5/6 \n Verifying : 10:qemu-img-1.5.3-175.el7_9.1.x86_64 6/6 \n\nRemoved:\n librados2.x86_64 1:10.2.5-4.el7 librbd1.x86_64 1:10.2.5-4.el7 \n mod_fastcgi.x86_64 0:2.4.7-1.ceph.el7 \n\nDependency Removed:\n qemu-img.x86_64 10:1.5.3-175.el7_9.1 2021-02-10T11:07:53.677 INFO:teuthology.task.ansible.out:\n qemu-kvm.x86_64 10:1.5.3-175.el7_9.1 \n qemu-kvm-common.x86_64 10:1.5.3-175.el7_9.1 \n\nComplete!\n"]} 2021-02-10T11:07:53.712 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:07:53 +0000 (0:00:09.198) 0:05:11.000 **** 2021-02-10T11:07:54.465 INFO:teuthology.task.ansible.out:changed: [gibba012.front.sepia.ceph.com] => {"changed": true, "changes": {"removed": ["librbd1", "librados2", "mod_fastcgi"]}, "msg": "", "rc": 0, "results": ["ceph is not installed", "ceph-base is not installed", "ceph-selinux is not installed", "ceph-common is not installed", "ceph-debuginfo is not installed", "ceph-release is not installed", "libcephfs1 is not installed", "ceph-radosgw is not installed", "python-ceph is not installed", "python-rados is not installed", "python-rbd is not installed", "python-cephfs is not installed", "Loaded plugins: fastestmirror, langpacks, priorities, product-id, search-\n : disabled-repos, subscription-manager\nResolving Dependencies\n--> Running transaction check\n---> Package librados2.x86_64 1:10.2.5-4.el7 will be erased\n--> Processing Dependency: librados.so.2()(64bit) for package: 10:qemu-img-1.5.3-175.el7_9.1.x86_64\n--> Processing Dependency: librados.so.2()(64bit) for package: 10:qemu-kvm-common-1.5.3-175.el7_9.1.x86_64\n--> Processing Dependency: librados.so.2()(64bit) for package: 10:qemu-kvm-1.5.3-175.el7_9.1.x86_64\n---> Package librbd1.x86_64 1:10.2.5-4.el7 will be erased\n---> Package mod_fastcgi.x86_64 0:2.4.7-1.ceph.el7 will be erased\n--> Running transaction check\n---> Package qemu-img.x86_64 10:1.5.3-175.el7_9.1 will be erased\n---> Package qemu-kvm.x86_64 10:1.5.3-175.el7_9.1 will be erased\n---> Package qemu-kvm-common.x86_64 10:1.5.3-175.el7_9.1 will be erased\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nRemoving:\n librados2 x86_64 1:10.2.5-4.el7 @rhel-7-server-rpms 5.5 M\n librbd1 x86_64 1:10.2.5-4.el7 @rhel-7-server-rpms 7.4 M\n mod_fastcgi x86_64 2.4.7-1.ceph.el7 @rhel-7- 2021-02-10T11:07:54.466 INFO:teuthology.task.ansible.out:fcgi-ceph 213 k\nRemoving for dependencies:\n qemu-img x86_64 10:1.5.3-175.el7_9.1 @rhel-7-server-rpms 2.3 M\n qemu-kvm x86_64 10:1.5.3-175.el7_9.1 @rhel-7-server-rpms 5.7 M\n qemu-kvm-common x86_64 10:1.5.3-175.el7_9.1 @rhel-7-server-rpms 755 k\n\nTransaction Summary\n================================================================================\nRemove 3 Packages (+3 Dependent packages)\n\nInstalled size: 22 M\nDownloading packages:\nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Erasing : 10:qemu-kvm-1.5.3-175.el7_9.1.x86_64 1/6 \n Erasing : 10:qemu-img-1.5.3-175.el7_9.1.x86_64 2/6 \n Erasing : 10:qemu-kvm-common-1.5.3-175.el7_9.1.x86_64 3/6 \n Erasing : 1:librbd1-10.2.5-4.el7.x86_64 4/6 \n Erasing : 1:librados2-10.2.5-4.el7.x86_64 5/6 \n Erasing : mod_fastcgi-2.4.7-1.ceph.el7.x86_64 6/6 \nwarning: file /var/run/mod_fastcgi: remove failed: No such file or directory\nLoaded plugins: fastestmirror, langpacks, priorities, product-id, subscription-\n : manager\n Verifying : 1:librbd1-10.2.5-4.el7.x86_64 1/6 \n Verifying : 10:qemu-kvm-1.5.3-175.el7_9.1.x86_64 2/6 \n Verifying : mod_fastcgi-2.4.7-1.ceph.el7.x86_64 3/6 \n Verifying : 10:qemu-kvm-common-1.5.3-175.el7_9.1.x86_64 4/6 \n Verifying : 1:librados2-10.2.5-4.el7.x86_64 5/6 \n Verifying : 10:qemu-img-1.5.3-175.el7_9.1.x86_64 6/6 \n\nRemoved:\n librados2.x86_64 1:10.2.5-4.el7 librbd1.x86_64 1:10.2.5-4.el7 \n mod_fastcgi.x86_64 0:2.4.7-1.ceph.el7 \n\nDependency Removed:\n qemu-img.x86_64 10:1.5.3-175.el7_9.1 2021-02-10T11:07:54.468 INFO:teuthology.task.ansible.out:\n qemu-kvm.x86_64 10:1.5.3-175.el7_9.1 \n qemu-kvm-common.x86_64 10:1.5.3-175.el7_9.1 \n\nComplete!\n"]} 2021-02-10T11:07:54.502 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:07:54 +0000 (0:00:00.790) 0:05:11.790 **** 2021-02-10T11:07:58.203 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2021-02-10T11:07:58.205 INFO:teuthology.task.ansible.out:changed: [gibba020.front.sepia.ceph.com] => {"changed": true, "changes": {"removed": ["boost-random", "leveldb", "xmlstarlet", "boost-random"]}, "msg": "", "rc": 0, "results": ["boost-program-options is not installed", "python-jinja2 is not installed", "python-ceph is not installed", "python-flask is not installed", "python-requests is not installed", "python-urllib3 is not installed", "python-babel is not installed", "hdparm is not installed", "python-markupsafe is not installed", "python-werkzeug is not installed", "python-itsdangerous is not installed", "Loaded plugins: fastestmirror, langpacks, priorities, product-id, search-\n : disabled-repos, subscription-manager\nResolving Dependencies\n--> Running transaction check\n---> Package boost-random.x86_64 0:1.53.0-28.el7 will be erased\n---> Package leveldb.x86_64 0:1.12.0-11.el7 will be erased\n---> Package xmlstarlet.x86_64 0:1.6.1-1.el7 will be erased\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nRemoving:\n boost-random x86_64 1.53.0-28.el7 @rhel-7-server-rpms 24 k\n leveldb x86_64 1.12.0-11.el7 @epel 439 k\n xmlstarlet x86_64 1.6.1-1.el7 @epel 192 k\n\nTransaction Summary\n================================================================================\nRemove 3 Packages\n\nInstalled size: 655 k\nDownloading packages:\nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Erasing : xmlstarlet-1.6.1-1.el7.x86_64 1/3 \n Erasing : boost-random-1.53.0-28.el7.x86_64 2/3 \n Erasing : leveldb-1.12.0-11.el7.x86_64 2021-02-10T11:07:58.206 INFO:teuthology.task.ansible.out: 3/3 \nLoaded plugins: fastestmirror, langpacks, priorities, product-id, subscription-\n : manager\n Verifying : leveldb-1.12.0-11.el7.x86_64 1/3 \n Verifying : boost-random-1.53.0-28.el7.x86_64 2/3 \n Verifying : xmlstarlet-1.6.1-1.el7.x86_64 3/3 \n\nRemoved:\n boost-random.x86_64 0:1.53.0-28.el7 leveldb.x86_64 0:1.12.0-11.el7 \n xmlstarlet.x86_64 0:1.6.1-1.el7 \n\nComplete!\n"]} 2021-02-10T11:07:58.241 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:07:58 +0000 (0:00:03.738) 0:05:15.529 **** 2021-02-10T11:07:59.100 INFO:teuthology.task.ansible.out:changed: [gibba012.front.sepia.ceph.com] => {"changed": true, "changes": {"removed": ["boost-random", "leveldb", "xmlstarlet", "boost-random"]}, "msg": "", "rc": 0, "results": ["boost-program-options is not installed", "python-jinja2 is not installed", "python-ceph is not installed", "python-flask is not installed", "python-requests is not installed", "python-urllib3 is not installed", "python-babel is not installed", "hdparm is not installed", "python-markupsafe is not installed", "python-werkzeug is not installed", "python-itsdangerous is not installed", "Loaded plugins: fastestmirror, langpacks, priorities, product-id, search-\n : disabled-repos, subscription-manager\nResolving Dependencies\n--> Running transaction check\n---> Package boost-random.x86_64 0:1.53.0-28.el7 will be erased\n---> Package leveldb.x86_64 0:1.12.0-11.el7 will be erased\n---> Package xmlstarlet.x86_64 0:1.6.1-1.el7 will be erased\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nRemoving:\n boost-random x86_64 1.53.0-28.el7 @rhel-7-server-rpms 24 k\n leveldb x86_64 1.12.0-11.el7 @epel 439 k\n xmlstarlet x86_64 1.6.1-1.el7 @epel 192 k\n\nTransaction Summary\n================================================================================\nRemove 3 Packages\n\nInstalled size: 655 k\nDownloading packages:\nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Erasing : xmlstarlet-1.6.1-1.el7.x86_64 1/3 \n Erasing : boost-random-1.53.0-28.el7.x86_64 2/3 \n Erasing : leveldb-1.12.0-11.el7.x86_64 2021-02-10T11:07:59.101 INFO:teuthology.task.ansible.out: 3/3 \nLoaded plugins: fastestmirror, langpacks, priorities, product-id, subscription-\n : manager\n Verifying : leveldb-1.12.0-11.el7.x86_64 1/3 \n Verifying : boost-random-1.53.0-28.el7.x86_64 2/3 \n Verifying : xmlstarlet-1.6.1-1.el7.x86_64 3/3 \n\nRemoved:\n boost-random.x86_64 0:1.53.0-28.el7 leveldb.x86_64 0:1.12.0-11.el7 \n xmlstarlet.x86_64 0:1.6.1-1.el7 \n\nComplete!\n"]} 2021-02-10T11:07:59.136 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:07:59 +0000 (0:00:00.894) 0:05:16.424 **** 2021-02-10T11:08:14.140 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-02-10T11:08:14.140 INFO:teuthology.task.ansible.out:changed: [gibba020.front.sepia.ceph.com] => {"changed": true, "changes": {"installed": ["qemu-kvm", "mod_fastcgi-2.4.7-1.ceph.el7"]}, "msg": "", "rc": 0, "results": ["yum-plugin-priorities-1.1.31-54.el7_8.noarch providing yum-plugin-priorities is already installed", "yum-plugin-fastestmirror-1.1.31-54.el7_8.noarch providing yum-plugin-fastestmirror is already installed", "redhat-lsb-4.1-27.el7.x86_64 providing redhat-lsb is already installed", "sysstat-10.1.5-19.el7.x86_64 providing sysstat is already installed", "gdb-7.6.1-120.el7.x86_64 providing gdb is already installed", "git-all-1.8.3.1-23.el7_8.noarch providing git-all is already installed", "python-configobj-4.7.2-7.el7.noarch providing python-configobj is already installed", "gcc-c++-4.8.5-44.el7.x86_64 providing gcc-c++ is already installed", "libedit-3.0-12.20121213cvs.el7.x86_64 providing libedit is already installed", "openssl098e-0.9.8e-29.el7_2.3.x86_64 providing openssl098e is already installed", "boost-thread-1.53.0-28.el7.x86_64 providing boost-thread is already installed", "xfsprogs-4.5.0-22.el7.x86_64 providing xfsprogs is already installed", "xfsprogs-devel-4.5.0-22.el7.x86_64 providing xfsprogs-devel is already installed", "gdisk-0.8.10-3.el7.x86_64 providing gdisk is already installed", "parted-3.1-32.el7.x86_64 providing parted is already installed", "libgcrypt-1.5.3-14.el7.x86_64 providing libgcrypt is already installed", "fuse-2.9.2-11.el7.x86_64 providing fuse is already installed", "fuse-libs-2.9.2-11.el7.x86_64 providing fuse-libs is already installed", "7:lvm2-2.02.187-6.el7.x86_64 providing lvm2 is already installed", "1:openssl-1.0.2k-19.el7.x86_64 providing openssl is already installed", "libuuid-2.23.2-65.el7.x86_64 providing libuuid is already installed", "btrfs-progs-4.9.1-1.el7.x86_64 providing btrfs-progs is already installed", "attr-2.4.46-13.el7.x86_64 providing attr is already installed", "1:valgrind-3.15.0-11.el7.x86_64 providing valgrind is already installed", "python-nose-1 2021-02-10T11:08:14.142 INFO:teuthology.task.ansible.out:.3.7-1.el7.noarch providing python-nose is already installed", "mpich-3.0-3.0.4-10.el7.x86_64 providing mpich is already installed", "ant-1.9.4-2.el7.noarch providing ant is already installed", "lsof-4.87-6.el7.x86_64 providing lsof is already installed", "iozone-3.424-2_ceph.el7.x86_64 providing iozone is already installed", "libtool-2.4.2-22.el7_3.x86_64 providing libtool is already installed", "automake-1.13.4-3.el7.noarch providing automake is already installed", "gettext-0.19.8.1-3.el7.x86_64 providing gettext is already installed", "libuuid-devel-2.23.2-65.el7.x86_64 providing libuuid-devel is already installed", "libacl-devel-2.2.51-15.el7.x86_64 providing libacl-devel is already installed", "bc-1.06.95-13.el7.x86_64 providing bc is already installed", "xfsdump-3.1.7-1.el7.x86_64 providing xfsdump is already installed", "blktrace-1.0.5-9.el7.x86_64 providing blktrace is already installed", "1:numpy-1.7.1-13.el7.x86_64 providing numpy is already installed", "python-matplotlib-1.2.0-16.el7.x86_64 providing python-matplotlib is already installed", "usbredir-0.7.1-3.el7.x86_64 providing usbredir is already installed", "genisoimage-1.1.11-25.el7.x86_64 providing genisoimage is already installed", "httpd-2.4.6-97.el7_9.x86_64 providing httpd is already installed", "httpd-devel-2.4.6-97.el7_9.x86_64 providing httpd-devel is already installed", "httpd-tools-2.4.6-97.el7_9.x86_64 providing httpd-tools is already installed", "1:mod_ssl-2.4.6-97.el7_9.x86_64 providing mod_ssl is already installed", "libev-devel-4.15-7.el7.x86_64 providing libev-devel is already installed", "perl-XML-Twig-3.44-2.el7.noarch providing perl-XML-Twig is already installed", "1:java-1.6.0-openjdk-devel-1.6.0.41-1.13.13.1.el7_3.x86_64 providing java-1.6.0-openjdk-devel is already installed", "junit-4.11-8.el7.noarch providing junit4 is already installed", "1:nfs-utils-1.3.0-0.68.el7.x86_64 providing nfs-utils is already installed", "ncurses-devel-5.9-14.20130511.el7_4.x86_64 providing ncurses-d 2021-02-10T11:08:14.143 INFO:teuthology.task.ansible.out:evel is already installed", "python-devel-2.7.5-90.el7.x86_64 providing python-devel is already installed", "python-virtualenv-15.1.0-4.el7_8.noarch providing python-virtualenv is already installed", "perl-CPAN-1.9800-297.el7.noarch providing perl-CPAN is already installed", "podman-1.6.4-26.el7_9.x86_64 providing podman is already installed", "podman-docker-1.6.4-26.el7_9.noarch providing podman-docker is already installed", "python3-3.6.8-18.el7.x86_64 providing python3 is already installed", "Loaded plugins: fastestmirror, langpacks, priorities, product-id, search-\n : disabled-repos, subscription-manager\nLoading mirror speeds from cached hostfile\n * epel: download-cc-rdu01.fedoraproject.org\nResolving Dependencies\n--> Running transaction check\n---> Package mod_fastcgi.x86_64 0:2.4.7-1.ceph.el7 will be installed\n---> Package qemu-kvm.x86_64 10:1.5.3-175.el7_9.3 will be installed\n--> Processing Dependency: qemu-kvm-common = 10:1.5.3-175.el7_9.3 for package: 10:qemu-kvm-1.5.3-175.el7_9.3.x86_64\n--> Processing Dependency: qemu-img = 10:1.5.3-175.el7_9.3 for package: 10:qemu-kvm-1.5.3-175.el7_9.3.x86_64\n--> Processing Dependency: librbd.so.1()(64bit) for package: 10:qemu-kvm-1.5.3-175.el7_9.3.x86_64\n--> Processing Dependency: librados.so.2()(64bit) for package: 10:qemu-kvm-1.5.3-175.el7_9.3.x86_64\n--> Running transaction check\n---> Package librados2.x86_64 1:10.2.5-4.el7 will be installed\n--> Processing Dependency: libboost_random-mt.so.1.53.0()(64bit) for package: 1:librados2-10.2.5-4.el7.x86_64\n---> Package librbd1.x86_64 1:10.2.5-4.el7 will be installed\n---> Package qemu-img.x86_64 10:1.5.3-175.el7_9.3 will be installed\n---> Package qemu-kvm-common.x86_64 10:1.5.3-175.el7_9.3 will be installed\n--> Running transaction check\n---> Package boost-random.x86_64 0:1.53.0-28.el7 will be installed\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n P 2021-02-10T11:08:14.144 INFO:teuthology.task.ansible.out:ackage Arch Version Repository Size\n================================================================================\nInstalling:\n mod_fastcgi x86_64 2.4.7-1.ceph.el7 rhel-7-fcgi-ceph 70 k\n qemu-kvm x86_64 10:1.5.3-175.el7_9.3 rhel-7-server-rpms 1.9 M\nInstalling for dependencies:\n boost-random x86_64 1.53.0-28.el7 rhel-7-server-rpms 39 k\n librados2 x86_64 1:10.2.5-4.el7 rhel-7-server-rpms 1.8 M\n librbd1 x86_64 1:10.2.5-4.el7 rhel-7-server-rpms 2.4 M\n qemu-img x86_64 10:1.5.3-175.el7_9.3 rhel-7-server-rpms 704 k\n qemu-kvm-common x86_64 10:1.5.3-175.el7_9.3 rhel-7-server-rpms 440 k\n\nTransaction Summary\n================================================================================\nInstall 2 Packages (+5 Dependent packages)\n\nTotal download size: 7.3 M\nInstalled size: 22 M\nDownloading packages:\n--------------------------------------------------------------------------------\nTotal 31 MB/s | 7.3 MB 00:00 \nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Installing : boost-random-1.53.0-28.el7.x86_64 1/7 \n Installing : 1:librados2-10.2.5-4.el7.x86_64 2/7 \n Installing : 1:librbd1-10.2.5-4.el7.x86_64 3/7 \n Installing : 10:qemu-img-1.5.3-175.el7_9.3.x86_64 4/7 \n Installing : 10:qemu-kvm-common-1.5.3-175.el7_9.3.x86_64 5/7 \n Installing : 10:qemu-kvm-1.5.3-175.el7_9.3.x86_64 6/7 \n Installing : mod_fastcgi-2.4.7-1.ceph.el7.x86_64 7/7 \nLoaded plugins: fastestmirror, langpacks, priorities, product-id, subscription-\n : manager\n Verifying : 1:librbd1-10.2.5-4.el7.x86_64 2021-02-10T11:08:14.146 INFO:teuthology.task.ansible.out: 1/7 \n Verifying : mod_fastcgi-2.4.7-1.ceph.el7.x86_64 2/7 \n Verifying : 10:qemu-img-1.5.3-175.el7_9.3.x86_64 3/7 \n Verifying : boost-random-1.53.0-28.el7.x86_64 4/7 \n Verifying : 1:librados2-10.2.5-4.el7.x86_64 5/7 \n Verifying : 10:qemu-kvm-1.5.3-175.el7_9.3.x86_64 6/7 \n Verifying : 10:qemu-kvm-common-1.5.3-175.el7_9.3.x86_64 7/7 \n\nInstalled:\n mod_fastcgi.x86_64 0:2.4.7-1.ceph.el7 qemu-kvm.x86_64 10:1.5.3-175.el7_9.3 \n\nDependency Installed:\n boost-random.x86_64 0:1.53.0-28.el7 \n librados2.x86_64 1:10.2.5-4.el7 \n librbd1.x86_64 1:10.2.5-4.el7 \n qemu-img.x86_64 10:1.5.3-175.el7_9.3 \n qemu-kvm-common.x86_64 10:1.5.3-175.el7_9.3 \n\nComplete!\n"]} 2021-02-10T11:08:14.176 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:14 +0000 (0:00:15.040) 0:05:31.465 **** 2021-02-10T11:08:15.393 INFO:teuthology.task.ansible.out:changed: [gibba012.front.sepia.ceph.com] => {"changed": true, "changes": {"installed": ["qemu-kvm", "mod_fastcgi-2.4.7-1.ceph.el7"]}, "msg": "", "rc": 0, "results": ["yum-plugin-priorities-1.1.31-54.el7_8.noarch providing yum-plugin-priorities is already installed", "yum-plugin-fastestmirror-1.1.31-54.el7_8.noarch providing yum-plugin-fastestmirror is already installed", "redhat-lsb-4.1-27.el7.x86_64 providing redhat-lsb is already installed", "sysstat-10.1.5-19.el7.x86_64 providing sysstat is already installed", "gdb-7.6.1-120.el7.x86_64 providing gdb is already installed", "git-all-1.8.3.1-23.el7_8.noarch providing git-all is already installed", "python-configobj-4.7.2-7.el7.noarch providing python-configobj is already installed", "gcc-c++-4.8.5-44.el7.x86_64 providing gcc-c++ is already installed", "libedit-3.0-12.20121213cvs.el7.x86_64 providing libedit is already installed", "openssl098e-0.9.8e-29.el7_2.3.x86_64 providing openssl098e is already installed", "boost-thread-1.53.0-28.el7.x86_64 providing boost-thread is already installed", "xfsprogs-4.5.0-22.el7.x86_64 providing xfsprogs is already installed", "xfsprogs-devel-4.5.0-22.el7.x86_64 providing xfsprogs-devel is already installed", "gdisk-0.8.10-3.el7.x86_64 providing gdisk is already installed", "parted-3.1-32.el7.x86_64 providing parted is already installed", "libgcrypt-1.5.3-14.el7.x86_64 providing libgcrypt is already installed", "fuse-2.9.2-11.el7.x86_64 providing fuse is already installed", "fuse-libs-2.9.2-11.el7.x86_64 providing fuse-libs is already installed", "7:lvm2-2.02.187-6.el7.x86_64 providing lvm2 is already installed", "1:openssl-1.0.2k-19.el7.x86_64 providing openssl is already installed", "libuuid-2.23.2-65.el7.x86_64 providing libuuid is already installed", "btrfs-progs-4.9.1-1.el7.x86_64 providing btrfs-progs is already installed", "attr-2.4.46-13.el7.x86_64 providing attr is already installed", "1:valgrind-3.15.0-11.el7.x86_64 providing valgrind is already installed", "python-nose-1 2021-02-10T11:08:15.394 INFO:teuthology.task.ansible.out:.3.7-1.el7.noarch providing python-nose is already installed", "mpich-3.0-3.0.4-10.el7.x86_64 providing mpich is already installed", "ant-1.9.4-2.el7.noarch providing ant is already installed", "lsof-4.87-6.el7.x86_64 providing lsof is already installed", "iozone-3.424-2_ceph.el7.x86_64 providing iozone is already installed", "libtool-2.4.2-22.el7_3.x86_64 providing libtool is already installed", "automake-1.13.4-3.el7.noarch providing automake is already installed", "gettext-0.19.8.1-3.el7.x86_64 providing gettext is already installed", "libuuid-devel-2.23.2-65.el7.x86_64 providing libuuid-devel is already installed", "libacl-devel-2.2.51-15.el7.x86_64 providing libacl-devel is already installed", "bc-1.06.95-13.el7.x86_64 providing bc is already installed", "xfsdump-3.1.7-1.el7.x86_64 providing xfsdump is already installed", "blktrace-1.0.5-9.el7.x86_64 providing blktrace is already installed", "1:numpy-1.7.1-13.el7.x86_64 providing numpy is already installed", "python-matplotlib-1.2.0-16.el7.x86_64 providing python-matplotlib is already installed", "usbredir-0.7.1-3.el7.x86_64 providing usbredir is already installed", "genisoimage-1.1.11-25.el7.x86_64 providing genisoimage is already installed", "httpd-2.4.6-97.el7_9.x86_64 providing httpd is already installed", "httpd-devel-2.4.6-97.el7_9.x86_64 providing httpd-devel is already installed", "httpd-tools-2.4.6-97.el7_9.x86_64 providing httpd-tools is already installed", "1:mod_ssl-2.4.6-97.el7_9.x86_64 providing mod_ssl is already installed", "libev-devel-4.15-7.el7.x86_64 providing libev-devel is already installed", "perl-XML-Twig-3.44-2.el7.noarch providing perl-XML-Twig is already installed", "1:java-1.6.0-openjdk-devel-1.6.0.41-1.13.13.1.el7_3.x86_64 providing java-1.6.0-openjdk-devel is already installed", "junit-4.11-8.el7.noarch providing junit4 is already installed", "1:nfs-utils-1.3.0-0.68.el7.x86_64 providing nfs-utils is already installed", "ncurses-devel-5.9-14.20130511.el7_4.x86_64 providing ncurses-d 2021-02-10T11:08:15.395 INFO:teuthology.task.ansible.out:evel is already installed", "python-devel-2.7.5-90.el7.x86_64 providing python-devel is already installed", "python-virtualenv-15.1.0-4.el7_8.noarch providing python-virtualenv is already installed", "perl-CPAN-1.9800-297.el7.noarch providing perl-CPAN is already installed", "podman-1.6.4-26.el7_9.x86_64 providing podman is already installed", "podman-docker-1.6.4-26.el7_9.noarch providing podman-docker is already installed", "python3-3.6.8-18.el7.x86_64 providing python3 is already installed", "Loaded plugins: fastestmirror, langpacks, priorities, product-id, search-\n : disabled-repos, subscription-manager\nLoading mirror speeds from cached hostfile\n * epel: download-cc-rdu01.fedoraproject.org\nResolving Dependencies\n--> Running transaction check\n---> Package mod_fastcgi.x86_64 0:2.4.7-1.ceph.el7 will be installed\n---> Package qemu-kvm.x86_64 10:1.5.3-175.el7_9.3 will be installed\n--> Processing Dependency: qemu-kvm-common = 10:1.5.3-175.el7_9.3 for package: 10:qemu-kvm-1.5.3-175.el7_9.3.x86_64\n--> Processing Dependency: qemu-img = 10:1.5.3-175.el7_9.3 for package: 10:qemu-kvm-1.5.3-175.el7_9.3.x86_64\n--> Processing Dependency: librbd.so.1()(64bit) for package: 10:qemu-kvm-1.5.3-175.el7_9.3.x86_64\n--> Processing Dependency: librados.so.2()(64bit) for package: 10:qemu-kvm-1.5.3-175.el7_9.3.x86_64\n--> Running transaction check\n---> Package librados2.x86_64 1:10.2.5-4.el7 will be installed\n--> Processing Dependency: libboost_random-mt.so.1.53.0()(64bit) for package: 1:librados2-10.2.5-4.el7.x86_64\n---> Package librbd1.x86_64 1:10.2.5-4.el7 will be installed\n---> Package qemu-img.x86_64 10:1.5.3-175.el7_9.3 will be installed\n---> Package qemu-kvm-common.x86_64 10:1.5.3-175.el7_9.3 will be installed\n--> Running transaction check\n---> Package boost-random.x86_64 0:1.53.0-28.el7 will be installed\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n P 2021-02-10T11:08:15.397 INFO:teuthology.task.ansible.out:ackage Arch Version Repository Size\n================================================================================\nInstalling:\n mod_fastcgi x86_64 2.4.7-1.ceph.el7 rhel-7-fcgi-ceph 70 k\n qemu-kvm x86_64 10:1.5.3-175.el7_9.3 rhel-7-server-rpms 1.9 M\nInstalling for dependencies:\n boost-random x86_64 1.53.0-28.el7 rhel-7-server-rpms 39 k\n librados2 x86_64 1:10.2.5-4.el7 rhel-7-server-rpms 1.8 M\n librbd1 x86_64 1:10.2.5-4.el7 rhel-7-server-rpms 2.4 M\n qemu-img x86_64 10:1.5.3-175.el7_9.3 rhel-7-server-rpms 704 k\n qemu-kvm-common x86_64 10:1.5.3-175.el7_9.3 rhel-7-server-rpms 440 k\n\nTransaction Summary\n================================================================================\nInstall 2 Packages (+5 Dependent packages)\n\nTotal download size: 7.3 M\nInstalled size: 22 M\nDownloading packages:\n--------------------------------------------------------------------------------\nTotal 33 MB/s | 7.3 MB 00:00 \nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Installing : boost-random-1.53.0-28.el7.x86_64 1/7 \n Installing : 1:librados2-10.2.5-4.el7.x86_64 2/7 \n Installing : 1:librbd1-10.2.5-4.el7.x86_64 3/7 \n Installing : 10:qemu-img-1.5.3-175.el7_9.3.x86_64 4/7 \n Installing : 10:qemu-kvm-common-1.5.3-175.el7_9.3.x86_64 5/7 \n Installing : 10:qemu-kvm-1.5.3-175.el7_9.3.x86_64 6/7 \n Installing : mod_fastcgi-2.4.7-1.ceph.el7.x86_64 7/7 \nLoaded plugins: fastestmirror, langpacks, priorities, product-id, subscription-\n : manager\n Verifying : 1:librbd1-10.2.5-4.el7.x86_64 2021-02-10T11:08:15.398 INFO:teuthology.task.ansible.out: 1/7 \n Verifying : mod_fastcgi-2.4.7-1.ceph.el7.x86_64 2/7 \n Verifying : 10:qemu-img-1.5.3-175.el7_9.3.x86_64 3/7 \n Verifying : boost-random-1.53.0-28.el7.x86_64 4/7 \n Verifying : 1:librados2-10.2.5-4.el7.x86_64 5/7 \n Verifying : 10:qemu-kvm-1.5.3-175.el7_9.3.x86_64 6/7 \n Verifying : 10:qemu-kvm-common-1.5.3-175.el7_9.3.x86_64 7/7 \n\nInstalled:\n mod_fastcgi.x86_64 0:2.4.7-1.ceph.el7 qemu-kvm.x86_64 10:1.5.3-175.el7_9.3 \n\nDependency Installed:\n boost-random.x86_64 0:1.53.0-28.el7 \n librados2.x86_64 1:10.2.5-4.el7 \n librbd1.x86_64 1:10.2.5-4.el7 \n qemu-img.x86_64 10:1.5.3-175.el7_9.3 \n qemu-kvm-common.x86_64 10:1.5.3-175.el7_9.3 \n\nComplete!\n"]} 2021-02-10T11:08:15.427 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:15 +0000 (0:00:01.250) 0:05:32.715 **** 2021-02-10T11:08:18.504 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2021-02-10T11:08:18.505 INFO:teuthology.task.ansible.out:changed: [gibba020.front.sepia.ceph.com] => {"changed": true, "changes": {"installed": ["leveldb", "xmlstarlet"]}, "msg": "", "rc": 0, "results": ["cryptopp-devel-5.6.2-10.el7.x86_64 providing cryptopp-devel is already installed", "cryptopp-5.6.2-10.el7.x86_64 providing cryptopp is already installed", "dbench-4.0-10.el7.x86_64 providing dbench is already installed", "fcgi-2.4.0-25.el7.x86_64 providing fcgi is already installed", "fuse-sshfs-2.10-1.el7.x86_64 providing fuse-sshfs is already installed", "1:perl-JSON-XS-3.01-2.el7.x86_64 providing perl-JSON-XS is already installed", "Loaded plugins: fastestmirror, langpacks, priorities, product-id, search-\n : disabled-repos, subscription-manager\nLoading mirror speeds from cached hostfile\n * epel: download-cc-rdu01.fedoraproject.org\nResolving Dependencies\n--> Running transaction check\n---> Package leveldb.x86_64 0:1.12.0-11.el7 will be installed\n---> Package xmlstarlet.x86_64 0:1.6.1-1.el7 will be installed\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nInstalling:\n leveldb x86_64 1.12.0-11.el7 epel 161 k\n xmlstarlet x86_64 1.6.1-1.el7 epel 62 k\n\nTransaction Summary\n================================================================================\nInstall 2 Packages\n\nTotal download size: 223 k\nInstalled size: 631 k\nDownloading packages:\n--------------------------------------------------------------------------------\nTotal 5.7 MB/s | 223 kB 00:00 \nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Installing : xmlstarlet-1.6.1-1.el7.x86_64 2021-02-10T11:08:18.506 INFO:teuthology.task.ansible.out: 1/2 \n Installing : leveldb-1.12.0-11.el7.x86_64 2/2 \nLoaded plugins: fastestmirror, langpacks, priorities, product-id, subscription-\n : manager\n Verifying : leveldb-1.12.0-11.el7.x86_64 1/2 \n Verifying : xmlstarlet-1.6.1-1.el7.x86_64 2/2 \n\nInstalled:\n leveldb.x86_64 0:1.12.0-11.el7 xmlstarlet.x86_64 0:1.6.1-1.el7 \n\nComplete!\n"]} 2021-02-10T11:08:18.544 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:18 +0000 (0:00:03.116) 0:05:35.832 **** 2021-02-10T11:08:19.420 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** ok: [gibba020.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["device-mapper-multipath is not installed"]} 2021-02-10T11:08:19.454 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:19 +0000 (0:00:00.910) 0:05:36.743 **** 2021-02-10T11:08:19.493 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-02-10T11:08:19.494 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:19.540 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:19 +0000 (0:00:00.085) 0:05:36.828 **** 2021-02-10T11:08:20.051 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2021-02-10T11:08:20.052 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => {"changed": false, "enabled": false, "name": "firewalld", "state": "stopped", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "basic.target dbus.service system.slice polkit.service", "AllowIsolate": "no", "AmbientCapabilities": "0", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "network-pre.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "18446744073709551615", "BusName": "org.fedoraproject.FirewallD1", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "18446744073709551615", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "18446744073709551615", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "Conflicts": "iptables.service ipset.service ip6tables.service ebtables.service shutdown.target", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "firewalld - dynamic firewall daemon", "DevicePolicy": "auto", "Documentation": "man:firewalld(1)", "EnvironmentFile": "/etc/sysconfig/firewalld (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/firewalld ; argv[]=/usr/sbin/firewalld --nofork --nopid $FIREWALLD_ARGS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/firewalld.service", "GuessMainPID": "yes", "IOScheduling": "0", "Id": "firewalld.service", "Ign 2021-02-10T11:08:20.053 INFO:teuthology.task.ansible.out:oreOnIsolate": "no", "IgnoreOnSnapshot": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobTimeoutAction": "none", "JobTimeoutUSec": "0", "KillMode": "mixed", "KillSignal": "15", "LimitAS": "18446744073709551615", "LimitCORE": "18446744073709551615", "LimitCPU": "18446744073709551615", "LimitDATA": "18446744073709551615", "LimitFSIZE": "18446744073709551615", "LimitLOCKS": "18446744073709551615", "LimitMEMLOCK": "65536", "LimitMSGQUEUE": "819200", "LimitNICE": "0", "LimitNOFILE": "4096", "LimitNPROC": "127560", "LimitRSS": "18446744073709551615", "LimitRTPRIO": "0", "LimitRTTIME": "18446744073709551615", "LimitSIGPENDING": "127560", "LimitSTACK": "18446744073709551615", "LoadState": "loaded", "MainPID": "0", "MemoryAccounting": "no", "MemoryCurrent": "18446744073709551615", "MemoryLimit": "18446744073709551615", "MountFlags": "0", "Names": "firewalld.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "ProtectHome": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "Requires": "basic.target system.slice", "Restart": "no", "RestartUSec": "100ms", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "null", "StandardInput": "null", "StandardOutput": "null", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitInterval": "10000000", "StartupBlockIOWeight": "18446744073709551615", "StartupCPUShares": "18446744073709551615", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", 2021-02-10T11:08:20.055 INFO:teuthology.task.ansible.out:"TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "no", "TasksCurrent": "18446744073709551615", "TasksMax": "18446744073709551615", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "dbus", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "disabled", "Wants": "network-pre.target", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2021-02-10T11:08:20.086 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:20 +0000 (0:00:00.546) 0:05:37.374 **** 2021-02-10T11:08:20.178 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** changed: [gibba012.front.sepia.ceph.com] => {"changed": true, "changes": {"installed": ["leveldb", "xmlstarlet"]}, "msg": "", "rc": 0, "results": ["cryptopp-devel-5.6.2-10.el7.x86_64 providing cryptopp-devel is already installed", "cryptopp-5.6.2-10.el7.x86_64 providing cryptopp is already installed", "dbench-4.0-10.el7.x86_64 providing dbench is already installed", "fcgi-2.4.0-25.el7.x86_64 providing fcgi is already installed", "fuse-sshfs-2.10-1.el7.x86_64 providing fuse-sshfs is already installed", "1:perl-JSON-XS-3.01-2.el7.x86_64 providing perl-JSON-XS is already installed", "Loaded plugins: fastestmirror, langpacks, priorities, product-id, search-\n : disabled-repos, subscription-manager\nLoading mirror speeds from cached hostfile\n * epel: download-cc-rdu01.fedoraproject.org\nResolving Dependencies\n--> Running transaction check\n---> Package leveldb.x86_64 0:1.12.0-11.el7 will be installed\n---> Package xmlstarlet.x86_64 0:1.6.1-1.el7 will be installed\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nInstalling:\n leveldb x86_64 1.12.0-11.el7 epel 161 k\n xmlstarlet x86_64 1.6.1-1.el7 epel 62 k\n\nTransaction Summary\n================================================================================\nInstall 2 Packages\n\nTotal download size: 223 k\nInstalled size: 631 k\nDownloading packages:\n--------------------------------------------------------------------------------\nTotal 5.8 MB/s | 223 kB 00:00 \nRunning transaction check\nRunning transaction test\nTransaction test succeede 2021-02-10T11:08:20.179 INFO:teuthology.task.ansible.out:d\nRunning transaction\n Installing : xmlstarlet-1.6.1-1.el7.x86_64 1/2 \n Installing : leveldb-1.12.0-11.el7.x86_64 2/2 \nLoaded plugins: fastestmirror, langpacks, priorities, product-id, subscription-\n : manager\n Verifying : leveldb-1.12.0-11.el7.x86_64 1/2 \n Verifying : xmlstarlet-1.6.1-1.el7.x86_64 2/2 \n\nInstalled:\n leveldb.x86_64 0:1.12.0-11.el7 xmlstarlet.x86_64 0:1.6.1-1.el7 \n\nComplete!\n"]} 2021-02-10T11:08:20.211 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:20 +0000 (0:00:00.125) 0:05:37.500 **** 2021-02-10T11:08:20.580 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ 2021-02-10T11:08:20.582 INFO:teuthology.task.ansible.out:fatal: [gibba020.front.sepia.ceph.com]: FAILED! => {"changed": false, "msg": "Could not find the requested service iptables: host"} ...ignoring 2021-02-10T11:08:20.615 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:20 +0000 (0:00:00.403) 0:05:37.903 **** 2021-02-10T11:08:21.096 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2021-02-10T11:08:21.097 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["device-mapper-multipath is not installed"]} 2021-02-10T11:08:21.132 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:21 +0000 (0:00:00.517) 0:05:38.421 **** 2021-02-10T11:08:21.170 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-02-10T11:08:21.172 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:21.205 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:21 +0000 (0:00:00.072) 0:05:38.493 **** 2021-02-10T11:08:21.611 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** 2021-02-10T11:08:21.613 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => {"changed": false, "configfile": "/etc/selinux/config", "msg": "", "policy": "targeted", "reboot_required": false, "state": "permissive"} 2021-02-10T11:08:21.653 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:21 +0000 (0:00:00.448) 0:05:38.942 **** 2021-02-10T11:08:21.681 INFO:teuthology.task.ansible.out: TASK [testnode : Install abrt] ************************************************* 2021-02-10T11:08:21.682 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:21.716 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:21 +0000 (0:00:00.063) 0:05:39.005 **** 2021-02-10T11:08:21.723 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2021-02-10T11:08:21.724 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => {"changed": false, "enabled": false, "name": "firewalld", "state": "stopped", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "system.slice dbus.service polkit.service basic.target", "AllowIsolate": "no", "AmbientCapabilities": "0", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "shutdown.target network-pre.target", "BlockIOAccounting": "no", "BlockIOWeight": "18446744073709551615", "BusName": "org.fedoraproject.FirewallD1", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "18446744073709551615", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "18446744073709551615", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "Conflicts": "ipset.service ip6tables.service shutdown.target ebtables.service iptables.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "firewalld - dynamic firewall daemon", "DevicePolicy": "auto", "Documentation": "man:firewalld(1)", "EnvironmentFile": "/etc/sysconfig/firewalld (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/firewalld ; argv[]=/usr/sbin/firewalld --nofork --nopid $FIREWALLD_ARGS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/firewalld.service", "GuessMainPID": "yes", "IOScheduling": "0", "Id": "firewalld.service", "Ign 2021-02-10T11:08:21.726 INFO:teuthology.task.ansible.out:oreOnIsolate": "no", "IgnoreOnSnapshot": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobTimeoutAction": "none", "JobTimeoutUSec": "0", "KillMode": "mixed", "KillSignal": "15", "LimitAS": "18446744073709551615", "LimitCORE": "18446744073709551615", "LimitCPU": "18446744073709551615", "LimitDATA": "18446744073709551615", "LimitFSIZE": "18446744073709551615", "LimitLOCKS": "18446744073709551615", "LimitMEMLOCK": "65536", "LimitMSGQUEUE": "819200", "LimitNICE": "0", "LimitNOFILE": "4096", "LimitNPROC": "127560", "LimitRSS": "18446744073709551615", "LimitRTPRIO": "0", "LimitRTTIME": "18446744073709551615", "LimitSIGPENDING": "127560", "LimitSTACK": "18446744073709551615", "LoadState": "loaded", "MainPID": "0", "MemoryAccounting": "no", "MemoryCurrent": "18446744073709551615", "MemoryLimit": "18446744073709551615", "MountFlags": "0", "Names": "firewalld.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "ProtectHome": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "Requires": "system.slice basic.target", "Restart": "no", "RestartUSec": "100ms", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "null", "StandardInput": "null", "StandardOutput": "null", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitInterval": "10000000", "StartupBlockIOWeight": "18446744073709551615", "StartupCPUShares": "18446744073709551615", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", 2021-02-10T11:08:21.727 INFO:teuthology.task.ansible.out:"TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "no", "TasksCurrent": "18446744073709551615", "TasksMax": "18446744073709551615", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "dbus", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "disabled", "Wants": "network-pre.target", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2021-02-10T11:08:21.773 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:21 +0000 (0:00:00.056) 0:05:39.061 **** 2021-02-10T11:08:21.782 INFO:teuthology.task.ansible.out: TASK [testnode : Enable abrt-auto-reporting] *********************************** 2021-02-10T11:08:21.783 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:21.829 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:21 +0000 (0:00:00.054) 0:05:39.116 **** 2021-02-10T11:08:21.857 INFO:teuthology.task.ansible.out: TASK [testnode : Set OpenGPGCheck in abrt-action-save-package-data.conf] ******* 2021-02-10T11:08:21.859 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:21.891 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:21 +0000 (0:00:00.063) 0:05:39.179 **** 2021-02-10T11:08:21.918 INFO:teuthology.task.ansible.out: TASK [testnode : Set ProcessUnpackaged in abrt-action-save-package-data.conf] *** 2021-02-10T11:08:21.920 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:21.954 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:21 +0000 (0:00:00.062) 0:05:39.242 **** 2021-02-10T11:08:21.995 INFO:teuthology.task.ansible.out: TASK [testnode : Restart abrtd] ************************************************ 2021-02-10T11:08:21.997 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:22.029 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:22 +0000 (0:00:00.075) 0:05:39.318 **** 2021-02-10T11:08:22.057 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:22.093 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:22 +0000 (0:00:00.063) 0:05:39.381 **** 2021-02-10T11:08:22.120 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2021-02-10T11:08:22.164 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:22 +0000 (0:00:00.071) 0:05:39.452 **** 2021-02-10T11:08:22.191 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:22.225 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:22 +0000 (0:00:00.060) 0:05:39.513 **** 2021-02-10T11:08:22.252 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* 2021-02-10T11:08:22.253 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:22.286 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:22 +0000 (0:00:00.061) 0:05:39.574 **** 2021-02-10T11:08:22.293 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ 2021-02-10T11:08:22.294 INFO:teuthology.task.ansible.out:fatal: [gibba012.front.sepia.ceph.com]: FAILED! => {"changed": false, "msg": "Could not find the requested service iptables: host"} ...ignoring 2021-02-10T11:08:22.329 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:22 +0000 (0:00:00.042) 0:05:39.617 **** 2021-02-10T11:08:22.335 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* 2021-02-10T11:08:22.337 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} skipping: [gibba020.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2021-02-10T11:08:22.372 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:22 +0000 (0:00:00.042) 0:05:39.660 **** 2021-02-10T11:08:22.409 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2021-02-10T11:08:22.411 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:22.446 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:22 +0000 (0:00:00.074) 0:05:39.734 **** 2021-02-10T11:08:22.479 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2021-02-10T11:08:22.511 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:22 +0000 (0:00:00.065) 0:05:39.800 **** 2021-02-10T11:08:22.550 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** 2021-02-10T11:08:22.551 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:22.584 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:22 +0000 (0:00:00.072) 0:05:39.872 **** 2021-02-10T11:08:22.611 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** 2021-02-10T11:08:22.612 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:22.649 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:22 +0000 (0:00:00.065) 0:05:39.938 **** 2021-02-10T11:08:22.678 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** 2021-02-10T11:08:22.679 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:22.715 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:22 +0000 (0:00:00.065) 0:05:40.003 **** 2021-02-10T11:08:22.742 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-02-10T11:08:22.744 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:22.779 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:22 +0000 (0:00:00.064) 0:05:40.068 **** 2021-02-10T11:08:22.806 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2021-02-10T11:08:22.808 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:22.840 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:22 +0000 (0:00:00.060) 0:05:40.128 **** 2021-02-10T11:08:22.847 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** 2021-02-10T11:08:22.848 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => {"changed": false, "configfile": "/etc/selinux/config", "msg": "", "policy": "targeted", "reboot_required": false, "state": "permissive"} 2021-02-10T11:08:22.884 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:22 +0000 (0:00:00.043) 0:05:40.172 **** 2021-02-10T11:08:22.891 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-02-10T11:08:22.892 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:22.936 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:22 +0000 (0:00:00.051) 0:05:40.224 **** 2021-02-10T11:08:22.944 INFO:teuthology.task.ansible.out: TASK [testnode : Install abrt] ************************************************* 2021-02-10T11:08:22.945 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:22.983 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:22 +0000 (0:00:00.047) 0:05:40.271 **** 2021-02-10T11:08:22.990 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* 2021-02-10T11:08:22.991 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:23.027 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:23 +0000 (0:00:00.043) 0:05:40.315 **** 2021-02-10T11:08:23.033 INFO:teuthology.task.ansible.out: TASK [testnode : Enable abrt-auto-reporting] *********************************** 2021-02-10T11:08:23.035 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:23.071 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:23 +0000 (0:00:00.044) 0:05:40.360 **** 2021-02-10T11:08:23.078 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2021-02-10T11:08:23.079 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:23.112 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:23 +0000 (0:00:00.041) 0:05:40.401 **** 2021-02-10T11:08:23.119 INFO:teuthology.task.ansible.out: TASK [testnode : Set OpenGPGCheck in abrt-action-save-package-data.conf] ******* 2021-02-10T11:08:23.120 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:23.154 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:23 +0000 (0:00:00.041) 0:05:40.443 **** 2021-02-10T11:08:23.162 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2021-02-10T11:08:23.164 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:23.197 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:23 +0000 (0:00:00.042) 0:05:40.486 **** 2021-02-10T11:08:23.204 INFO:teuthology.task.ansible.out: TASK [testnode : Set ProcessUnpackaged in abrt-action-save-package-data.conf] *** 2021-02-10T11:08:23.206 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:23.239 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:23 +0000 (0:00:00.041) 0:05:40.527 **** 2021-02-10T11:08:23.245 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2021-02-10T11:08:23.246 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:23.281 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:23 +0000 (0:00:00.042) 0:05:40.570 **** 2021-02-10T11:08:23.287 INFO:teuthology.task.ansible.out: TASK [testnode : Restart abrtd] ************************************************ 2021-02-10T11:08:23.289 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:23.323 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:23 +0000 (0:00:00.041) 0:05:40.611 **** 2021-02-10T11:08:23.330 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** 2021-02-10T11:08:23.331 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:23.371 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:23 +0000 (0:00:00.047) 0:05:40.659 **** 2021-02-10T11:08:23.378 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** 2021-02-10T11:08:23.380 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:23.413 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:23 +0000 (0:00:00.042) 0:05:40.701 **** 2021-02-10T11:08:23.420 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2021-02-10T11:08:23.422 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:23.455 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:23 +0000 (0:00:00.041) 0:05:40.743 **** 2021-02-10T11:08:23.461 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2021-02-10T11:08:23.496 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:23 +0000 (0:00:00.040) 0:05:40.784 **** 2021-02-10T11:08:23.504 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** 2021-02-10T11:08:23.505 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:23.539 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:23 +0000 (0:00:00.043) 0:05:40.827 **** 2021-02-10T11:08:23.545 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2021-02-10T11:08:23.547 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:23.579 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:23 +0000 (0:00:00.040) 0:05:40.868 **** 2021-02-10T11:08:23.586 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2021-02-10T11:08:23.587 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:23.620 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:23 +0000 (0:00:00.040) 0:05:40.909 **** 2021-02-10T11:08:23.627 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* 2021-02-10T11:08:23.628 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:23.661 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:23 +0000 (0:00:00.040) 0:05:40.949 **** 2021-02-10T11:08:23.668 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-02-10T11:08:23.670 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:23.703 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:23 +0000 (0:00:00.042) 0:05:40.991 **** 2021-02-10T11:08:23.709 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* 2021-02-10T11:08:23.709 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2021-02-10T11:08:23.711 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2021-02-10T11:08:23.744 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:23 +0000 (0:00:00.041) 0:05:41.033 **** 2021-02-10T11:08:23.751 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2021-02-10T11:08:23.752 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:23.784 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:23 +0000 (0:00:00.039) 0:05:41.072 **** 2021-02-10T11:08:23.790 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2021-02-10T11:08:23.792 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:23.826 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:23 +0000 (0:00:00.041) 0:05:41.114 **** 2021-02-10T11:08:23.832 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-02-10T11:08:23.833 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:23.871 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:23 +0000 (0:00:00.045) 0:05:41.159 **** 2021-02-10T11:08:23.878 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2021-02-10T11:08:23.912 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:23 +0000 (0:00:00.040) 0:05:41.200 **** 2021-02-10T11:08:23.918 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2021-02-10T11:08:23.920 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:23.955 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:23 +0000 (0:00:00.043) 0:05:41.244 **** 2021-02-10T11:08:23.963 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** 2021-02-10T11:08:23.964 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:23.999 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:23 +0000 (0:00:00.043) 0:05:41.287 **** 2021-02-10T11:08:24.005 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-02-10T11:08:24.007 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:24.040 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:24 +0000 (0:00:00.041) 0:05:41.328 **** 2021-02-10T11:08:24.046 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** 2021-02-10T11:08:24.048 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:24.082 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:24 +0000 (0:00:00.042) 0:05:41.371 **** 2021-02-10T11:08:24.091 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-02-10T11:08:24.092 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:24.125 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:24 +0000 (0:00:00.042) 0:05:41.413 **** 2021-02-10T11:08:24.132 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** 2021-02-10T11:08:24.132 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:24.168 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:24 +0000 (0:00:00.042) 0:05:41.456 **** 2021-02-10T11:08:24.175 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2021-02-10T11:08:24.176 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:24.210 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:24 +0000 (0:00:00.042) 0:05:41.498 **** 2021-02-10T11:08:24.217 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-02-10T11:08:24.218 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:24.253 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:24 +0000 (0:00:00.042) 0:05:41.541 **** 2021-02-10T11:08:24.259 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2021-02-10T11:08:24.261 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:24.295 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:24 +0000 (0:00:00.042) 0:05:41.584 **** 2021-02-10T11:08:24.302 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2021-02-10T11:08:24.303 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:24.337 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:24 +0000 (0:00:00.041) 0:05:41.625 **** 2021-02-10T11:08:24.344 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-10T11:08:24.346 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:24.380 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:24 +0000 (0:00:00.042) 0:05:41.668 **** 2021-02-10T11:08:24.386 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-02-10T11:08:24.388 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:24.422 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:24 +0000 (0:00:00.041) 0:05:41.710 **** 2021-02-10T11:08:24.428 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2021-02-10T11:08:24.430 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:24.464 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:24 +0000 (0:00:00.042) 0:05:41.752 **** 2021-02-10T11:08:24.470 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* 2021-02-10T11:08:24.472 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:24.505 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:24 +0000 (0:00:00.040) 0:05:41.793 **** 2021-02-10T11:08:24.512 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-02-10T11:08:24.514 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:24.547 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:24 +0000 (0:00:00.042) 0:05:41.835 **** 2021-02-10T11:08:24.555 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2021-02-10T11:08:24.556 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:24.590 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:24 +0000 (0:00:00.042) 0:05:41.878 **** 2021-02-10T11:08:24.597 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-02-10T11:08:24.598 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:24.632 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:24 +0000 (0:00:00.042) 0:05:41.920 **** 2021-02-10T11:08:24.638 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2021-02-10T11:08:24.640 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:24.673 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:24 +0000 (0:00:00.041) 0:05:41.962 **** 2021-02-10T11:08:24.685 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-02-10T11:08:24.686 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:24.724 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:24 +0000 (0:00:00.050) 0:05:42.012 **** 2021-02-10T11:08:24.730 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2021-02-10T11:08:24.731 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:24.767 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:24 +0000 (0:00:00.043) 0:05:42.055 **** 2021-02-10T11:08:24.773 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-02-10T11:08:24.775 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:24.808 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:24 +0000 (0:00:00.041) 0:05:42.097 **** 2021-02-10T11:08:24.814 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** 2021-02-10T11:08:24.816 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:24.849 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:24 +0000 (0:00:00.040) 0:05:42.137 **** 2021-02-10T11:08:24.855 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-02-10T11:08:24.857 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:24.892 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:24 +0000 (0:00:00.043) 0:05:42.180 **** 2021-02-10T11:08:24.899 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2021-02-10T11:08:24.900 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:24.934 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:24 +0000 (0:00:00.041) 0:05:42.222 **** 2021-02-10T11:08:24.942 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-02-10T11:08:24.944 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:24.977 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:24 +0000 (0:00:00.043) 0:05:42.265 **** 2021-02-10T11:08:24.983 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** 2021-02-10T11:08:24.984 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:25.019 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:25 +0000 (0:00:00.042) 0:05:42.307 **** 2021-02-10T11:08:25.047 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2021-02-10T11:08:25.049 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:25.081 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:25 +0000 (0:00:00.061) 0:05:42.369 **** 2021-02-10T11:08:25.109 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-02-10T11:08:25.110 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:25.141 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:25 +0000 (0:00:00.060) 0:05:42.429 **** 2021-02-10T11:08:25.168 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2021-02-10T11:08:25.169 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:25.203 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:25 +0000 (0:00:00.062) 0:05:42.492 **** 2021-02-10T11:08:25.230 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-02-10T11:08:25.232 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:25.264 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:25 +0000 (0:00:00.060) 0:05:42.552 **** 2021-02-10T11:08:25.291 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2021-02-10T11:08:25.293 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:25.325 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:25 +0000 (0:00:00.060) 0:05:42.613 **** 2021-02-10T11:08:25.351 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-02-10T11:08:25.353 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:25.386 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:25 +0000 (0:00:00.061) 0:05:42.675 **** 2021-02-10T11:08:25.414 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-02-10T11:08:25.416 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:25.449 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:25 +0000 (0:00:00.062) 0:05:42.737 **** 2021-02-10T11:08:25.477 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:25.511 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:25 +0000 (0:00:00.061) 0:05:42.799 **** 2021-02-10T11:08:25.538 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2021-02-10T11:08:25.539 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:25.572 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:25 +0000 (0:00:00.061) 0:05:42.860 **** 2021-02-10T11:08:25.611 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-10T11:08:25.612 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:25.649 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:25 +0000 (0:00:00.076) 0:05:42.937 **** 2021-02-10T11:08:25.677 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2021-02-10T11:08:25.678 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:25.710 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:25 +0000 (0:00:00.061) 0:05:42.999 **** 2021-02-10T11:08:25.739 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-02-10T11:08:25.741 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:25.775 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:25 +0000 (0:00:00.065) 0:05:43.064 **** 2021-02-10T11:08:25.803 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-02-10T11:08:25.804 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:25.841 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:25 +0000 (0:00:00.065) 0:05:43.129 **** 2021-02-10T11:08:25.879 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-02-10T11:08:25.880 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:25.916 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:25 +0000 (0:00:00.075) 0:05:43.204 **** 2021-02-10T11:08:25.944 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-02-10T11:08:25.945 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:25.979 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:25 +0000 (0:00:00.063) 0:05:43.267 **** 2021-02-10T11:08:26.007 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-02-10T11:08:26.008 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:26.047 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:26 +0000 (0:00:00.068) 0:05:43.336 **** 2021-02-10T11:08:26.078 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-02-10T11:08:26.079 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:26.123 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:26 +0000 (0:00:00.076) 0:05:43.412 **** 2021-02-10T11:08:26.900 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-02-10T11:08:26.901 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => {"changed": false, "checksum": "0b949a4622a315acfc90571fddbd013b4dc0f5f9", "dest": "/etc/exports", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/exports", "secontext": "system_u:object_r:exports_t:s0", "size": 578, "state": "file", "uid": 0} 2021-02-10T11:08:26.935 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:26 +0000 (0:00:00.811) 0:05:44.223 **** 2021-02-10T11:08:27.447 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-02-10T11:08:27.448 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nfs-server", "status": {"ActiveEnterTimestamp": "Wed 2021-02-10 11:01:32 UTC", "ActiveEnterTimestampMonotonic": "20320516", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "-.mount nfs-idmapd.service gssproxy.service system.slice rpcbind.socket proc-fs-nfsd.mount nfs-config.service systemd-journald.socket tmp.mount nfs-mountd.service local-fs.target network-online.target rpc-gssd.service rpc-statd.service", "AllowIsolate": "no", "AmbientCapabilities": "0", "AssertResult": "yes", "AssertTimestamp": "Wed 2021-02-10 11:01:31 UTC", "AssertTimestampMonotonic": "20121363", "Before": "rpc-statd-notify.service", "BlockIOAccounting": "no", "BlockIOWeight": "18446744073709551615", "BoundBy": "nfs-idmapd.service nfs-mountd.service", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "18446744073709551615", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "18446744073709551615", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Wed 2021-02-10 11:01:31 UTC", "ConditionTimestampMonotonic": "20121363", "ControlGroup": "/system.slice/nfs-server.service", "ControlPID": "0", "DefaultDependencies": "no", "Delegate": "no", "Description": "NFS server and services", "DevicePolicy": "auto", "DropInPaths": "/run/systemd/generator/nfs-server.service.d/order-with-mounts.conf", "EnvironmentFile": "/run/sysconfig/nfs-utils (ignore_errors=yes)", "ExecMainCode": "1", "ExecMainExitTimestamp": "Wed 2021-02-10 11:01:32 UTC", "ExecMainExitTimestampMonotonic": "20306546", "ExecMainPID": "1263", "ExecMainStartTimestamp": "Wed 2021-02-10 11:01:32 UTC", "ExecMainStartTimestampMonotonic": "20196973", "ExecMainStatus": "0", "ExecReload": "{ path=/usr/sbin/exportfs ; argv[]=/usr/sbin/exportfs -r ; ignore_errors=yes 2021-02-10T11:08:27.449 INFO:teuthology.task.ansible.out: ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/rpc.nfsd ; argv[]=/usr/sbin/rpc.nfsd $RPCNFSDARGS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStartPost": "{ path=/bin/sh ; argv[]=/bin/sh -c if systemctl -q is-active gssproxy; then systemctl reload gssproxy ; fi ; ignore_errors=yes ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStartPre": "{ path=/usr/sbin/exportfs ; argv[]=/usr/sbin/exportfs -r ; ignore_errors=yes ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStop": "{ path=/usr/sbin/rpc.nfsd ; argv[]=/usr/sbin/rpc.nfsd 0 ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/usr/sbin/exportfs ; argv[]=/usr/sbin/exportfs -f ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/nfs-server.service", "GuessMainPID": "yes", "IOScheduling": "0", "Id": "nfs-server.service", "IgnoreOnIsolate": "no", "IgnoreOnSnapshot": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Wed 2021-02-10 11:01:31 UTC", "InactiveExitTimestampMonotonic": "20121955", "JobTimeoutAction": "none", "JobTimeoutUSec": "0", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "18446744073709551615", "LimitCORE": "18446744073709551615", "LimitCPU": "18446744073709551615", "LimitDATA": "18446744073709551615", "LimitFSIZE": "18446744073709551615", "LimitLOCKS": "18446744073709551615", "LimitMEMLOCK": "65536", "LimitMSGQUEUE": "819200", "LimitNICE": "0", "LimitNOFILE": "4096", "LimitNPROC": "127560", "LimitRSS": "18446744073709551615", "LimitRTPRIO": "0", "LimitRTTIME": "18446744073709551615", "LimitSIGPENDING": "127560", "LimitSTACK": "18446744073709551615", "LoadState": "lo 2021-02-10T11:08:27.451 INFO:teuthology.task.ansible.out:aded", "MainPID": "0", "MemoryAccounting": "no", "MemoryCurrent": "18446744073709551615", "MemoryLimit": "18446744073709551615", "MountFlags": "0", "Names": "nfs-server.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "ProtectHome": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "yes", "Requires": "nfs-mountd.service -.mount network.target system.slice proc-fs-nfsd.mount", "RequiresMountsFor": "/tmp", "Restart": "no", "RestartUSec": "100ms", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitInterval": "10000000", "StartupBlockIOWeight": "18446744073709551615", "StartupCPUShares": "18446744073709551615", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "exited", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "no", "TasksCurrent": "18446744073709551615", "TasksMax": "18446744073709551615", "TimeoutStartUSec": "0", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "oneshot", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "enabled", "WantedBy": "multi-user.target", "Wants": "rpcbind.socket auth-rpcgss-module.service nfs-config.service rpc-statd.service network-online.target nfs-idmapd.service rpc-statd-notify.service", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2021-02-10T11:08:27.484 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:27 +0000 (0:00:00.549) 0:05:44.773 **** 2021-02-10T11:08:27.524 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-02-10T11:08:27.525 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:27.563 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:27 +0000 (0:00:00.078) 0:05:44.851 **** 2021-02-10T11:08:27.590 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:27.625 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:27 +0000 (0:00:00.062) 0:05:44.913 **** 2021-02-10T11:08:27.664 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:27.698 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:27 +0000 (0:00:00.073) 0:05:44.986 **** 2021-02-10T11:08:27.725 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-02-10T11:08:27.726 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:27.759 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:27 +0000 (0:00:00.061) 0:05:45.048 **** 2021-02-10T11:08:27.787 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-02-10T11:08:27.788 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:27.822 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:27 +0000 (0:00:00.062) 0:05:45.110 **** 2021-02-10T11:08:27.849 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-02-10T11:08:27.850 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:27.883 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:27 +0000 (0:00:00.060) 0:05:45.171 **** 2021-02-10T11:08:27.909 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-02-10T11:08:27.911 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:27.943 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:27 +0000 (0:00:00.059) 0:05:45.231 **** 2021-02-10T11:08:27.969 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2021-02-10T11:08:27.970 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:28.002 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:28 +0000 (0:00:00.059) 0:05:45.291 **** 2021-02-10T11:08:28.029 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2021-02-10T11:08:28.031 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:28.062 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:28 +0000 (0:00:00.060) 0:05:45.351 **** 2021-02-10T11:08:28.069 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-02-10T11:08:28.070 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => {"changed": false, "checksum": "0b949a4622a315acfc90571fddbd013b4dc0f5f9", "dest": "/etc/exports", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/exports", "secontext": "system_u:object_r:exports_t:s0", "size": 578, "state": "file", "uid": 0} 2021-02-10T11:08:28.106 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:28 +0000 (0:00:00.043) 0:05:45.394 **** 2021-02-10T11:08:28.112 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-02-10T11:08:28.113 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:28.148 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:28 +0000 (0:00:00.041) 0:05:45.436 **** 2021-02-10T11:08:28.174 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-02-10T11:08:28.175 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:28.209 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:28 +0000 (0:00:00.061) 0:05:45.497 **** 2021-02-10T11:08:28.238 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:28.239 INFO:teuthology.task.ansible.out: 2021-02-10T11:08:28.275 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:28 +0000 (0:00:00.065) 0:05:45.563 **** 2021-02-10T11:08:28.302 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* 2021-02-10T11:08:28.303 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:28.336 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:28 +0000 (0:00:00.061) 0:05:45.624 **** 2021-02-10T11:08:28.363 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2021-02-10T11:08:28.365 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:28.396 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:28 +0000 (0:00:00.060) 0:05:45.685 **** 2021-02-10T11:08:28.423 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2021-02-10T11:08:28.425 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:28.457 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:28 +0000 (0:00:00.060) 0:05:45.745 **** 2021-02-10T11:08:28.501 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2021-02-10T11:08:28.502 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:28.533 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:28 +0000 (0:00:00.076) 0:05:45.822 **** 2021-02-10T11:08:28.561 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:28.597 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:28 +0000 (0:00:00.063) 0:05:45.885 **** 2021-02-10T11:08:28.616 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-02-10T11:08:28.617 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nfs-server", "status": {"ActiveEnterTimestamp": "Wed 2021-02-10 11:01:33 UTC", "ActiveEnterTimestampMonotonic": "22044823", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "gssproxy.service local-fs.target nfs-config.service rpc-gssd.service tmp.mount rpc-statd.service -.mount nfs-mountd.service systemd-journald.socket nfs-idmapd.service proc-fs-nfsd.mount rpcbind.socket network-online.target system.slice", "AllowIsolate": "no", "AmbientCapabilities": "0", "AssertResult": "yes", "AssertTimestamp": "Wed 2021-02-10 11:01:33 UTC", "AssertTimestampMonotonic": "21908422", "Before": "rpc-statd-notify.service", "BlockIOAccounting": "no", "BlockIOWeight": "18446744073709551615", "BoundBy": "nfs-idmapd.service nfs-mountd.service", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "18446744073709551615", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "18446744073709551615", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Wed 2021-02-10 11:01:33 UTC", "ConditionTimestampMonotonic": "21908422", "ControlGroup": "/system.slice/nfs-server.service", "ControlPID": "0", "DefaultDependencies": "no", "Delegate": "no", "Description": "NFS server and services", "DevicePolicy": "auto", "DropInPaths": "/run/systemd/generator/nfs-server.service.d/order-with-mounts.conf", "EnvironmentFile": "/run/sysconfig/nfs-utils (ignore_errors=yes)", "ExecMainCode": "1", "ExecMainExitTimestamp": "Wed 2021-02-10 11:01:33 UTC", "ExecMainExitTimestampMonotonic": "22027120", "ExecMainPID": "1250", "ExecMainStartTimestamp": "Wed 2021-02-10 11:01:33 UTC", "ExecMainStartTimestampMonotonic": "21947291", "ExecMainStatus": "0", "ExecReload": "{ path=/usr/sbin/exportfs ; argv[]=/usr/sbin/exportfs -r ; ignore_errors=yes 2021-02-10T11:08:28.618 INFO:teuthology.task.ansible.out: ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/rpc.nfsd ; argv[]=/usr/sbin/rpc.nfsd $RPCNFSDARGS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStartPost": "{ path=/bin/sh ; argv[]=/bin/sh -c if systemctl -q is-active gssproxy; then systemctl reload gssproxy ; fi ; ignore_errors=yes ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStartPre": "{ path=/usr/sbin/exportfs ; argv[]=/usr/sbin/exportfs -r ; ignore_errors=yes ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStop": "{ path=/usr/sbin/rpc.nfsd ; argv[]=/usr/sbin/rpc.nfsd 0 ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/usr/sbin/exportfs ; argv[]=/usr/sbin/exportfs -f ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/nfs-server.service", "GuessMainPID": "yes", "IOScheduling": "0", "Id": "nfs-server.service", "IgnoreOnIsolate": "no", "IgnoreOnSnapshot": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Wed 2021-02-10 11:01:33 UTC", "InactiveExitTimestampMonotonic": "21908998", "JobTimeoutAction": "none", "JobTimeoutUSec": "0", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "18446744073709551615", "LimitCORE": "18446744073709551615", "LimitCPU": "18446744073709551615", "LimitDATA": "18446744073709551615", "LimitFSIZE": "18446744073709551615", "LimitLOCKS": "18446744073709551615", "LimitMEMLOCK": "65536", "LimitMSGQUEUE": "819200", "LimitNICE": "0", "LimitNOFILE": "4096", "LimitNPROC": "127560", "LimitRSS": "18446744073709551615", "LimitRTPRIO": "0", "LimitRTTIME": "18446744073709551615", "LimitSIGPENDING": "127560", "LimitSTACK": "18446744073709551615", "LoadState": "lo 2021-02-10T11:08:28.619 INFO:teuthology.task.ansible.out:aded", "MainPID": "0", "MemoryAccounting": "no", "MemoryCurrent": "18446744073709551615", "MemoryLimit": "18446744073709551615", "MountFlags": "0", "Names": "nfs-server.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "ProtectHome": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "yes", "Requires": "network.target -.mount nfs-mountd.service proc-fs-nfsd.mount system.slice", "RequiresMountsFor": "/tmp", "Restart": "no", "RestartUSec": "100ms", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitInterval": "10000000", "StartupBlockIOWeight": "18446744073709551615", "StartupCPUShares": "18446744073709551615", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "exited", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "no", "TasksCurrent": "18446744073709551615", "TasksMax": "18446744073709551615", "TimeoutStartUSec": "0", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "oneshot", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "enabled", "WantedBy": "multi-user.target", "Wants": "nfs-idmapd.service rpc-statd.service rpcbind.socket network-online.target auth-rpcgss-module.service rpc-statd-notify.service nfs-config.service", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2021-02-10T11:08:28.654 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:28 +0000 (0:00:00.056) 0:05:45.942 **** 2021-02-10T11:08:28.661 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2021-02-10T11:08:28.661 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:28.698 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:28 +0000 (0:00:00.043) 0:05:45.986 **** 2021-02-10T11:08:28.703 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-02-10T11:08:28.705 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:28.739 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:28 +0000 (0:00:00.041) 0:05:46.027 **** 2021-02-10T11:08:28.745 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2021-02-10T11:08:28.746 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:28.779 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:28 +0000 (0:00:00.039) 0:05:46.067 **** 2021-02-10T11:08:28.786 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-02-10T11:08:28.786 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:28.822 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:28 +0000 (0:00:00.042) 0:05:46.110 **** 2021-02-10T11:08:28.828 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2021-02-10T11:08:28.829 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:28.864 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:28 +0000 (0:00:00.041) 0:05:46.152 **** 2021-02-10T11:08:28.870 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:28.906 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:28 +0000 (0:00:00.042) 0:05:46.194 **** 2021-02-10T11:08:28.912 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-10T11:08:28.912 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2021-02-10T11:08:28.947 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:28 +0000 (0:00:00.041) 0:05:46.235 **** 2021-02-10T11:08:28.954 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-02-10T11:08:28.956 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:28.992 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:28 +0000 (0:00:00.044) 0:05:46.280 **** 2021-02-10T11:08:28.998 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2021-02-10T11:08:28.999 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2021-02-10T11:08:29.010 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2021-02-10T11:08:29.021 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2021-02-10T11:08:29.022 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Check for NVMe drive] ***************************************** skipping: [gibba020.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} skipping: [gibba020.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2021-02-10T11:08:29.056 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:29 +0000 (0:00:00.064) 0:05:46.344 **** 2021-02-10T11:08:29.062 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2021-02-10T11:08:29.063 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2021-02-10T11:08:29.108 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:29 +0000 (0:00:00.052) 0:05:46.397 **** 2021-02-10T11:08:29.115 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-02-10T11:08:29.116 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:29.150 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:29 +0000 (0:00:00.041) 0:05:46.439 **** 2021-02-10T11:08:29.178 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-02-10T11:08:29.180 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:29.211 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:29 +0000 (0:00:00.060) 0:05:46.499 **** 2021-02-10T11:08:29.240 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2021-02-10T11:08:29.242 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:29.273 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:29 +0000 (0:00:00.062) 0:05:46.562 **** 2021-02-10T11:08:29.301 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2021-02-10T11:08:29.302 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:29.337 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:29 +0000 (0:00:00.063) 0:05:46.625 **** 2021-02-10T11:08:29.343 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** changed: [gibba020.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.004236", "end": "2021-02-10 11:08:29.308209", "rc": 0, "start": "2021-02-10 11:08:29.303973", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/virtualenv/bin/teuthology-lock"]} 2021-02-10T11:08:29.385 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:29 +0000 (0:00:00.048) 0:05:46.673 **** 2021-02-10T11:08:29.391 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-02-10T11:08:29.392 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:29.427 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:29 +0000 (0:00:00.041) 0:05:46.715 **** 2021-02-10T11:08:29.433 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2021-02-10T11:08:29.434 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:29.472 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:29 +0000 (0:00:00.045) 0:05:46.761 **** 2021-02-10T11:08:29.478 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-02-10T11:08:29.479 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:29.514 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:29 +0000 (0:00:00.041) 0:05:46.802 **** 2021-02-10T11:08:29.522 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2021-02-10T11:08:29.523 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:29.559 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:29 +0000 (0:00:00.045) 0:05:46.847 **** 2021-02-10T11:08:29.567 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** 2021-02-10T11:08:29.568 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:29.603 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:29 +0000 (0:00:00.044) 0:05:46.892 **** 2021-02-10T11:08:29.610 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2021-02-10T11:08:29.611 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:29.644 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:29 +0000 (0:00:00.040) 0:05:46.933 **** 2021-02-10T11:08:29.651 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* 2021-02-10T11:08:29.653 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:29.688 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:29 +0000 (0:00:00.043) 0:05:46.976 **** 2021-02-10T11:08:29.694 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2021-02-10T11:08:29.695 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:29.729 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:29 +0000 (0:00:00.040) 0:05:47.017 **** 2021-02-10T11:08:29.735 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2021-02-10T11:08:29.736 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:29.770 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:29 +0000 (0:00:00.041) 0:05:47.058 **** 2021-02-10T11:08:29.799 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:29.833 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:29 +0000 (0:00:00.062) 0:05:47.121 **** 2021-02-10T11:08:29.860 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:29.897 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:29 +0000 (0:00:00.064) 0:05:47.185 **** 2021-02-10T11:08:29.924 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2021-02-10T11:08:29.926 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:29.961 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:29 +0000 (0:00:00.063) 0:05:47.249 **** 2021-02-10T11:08:29.987 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2021-02-10T11:08:29.989 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:30.025 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:30 +0000 (0:00:00.063) 0:05:47.313 **** 2021-02-10T11:08:30.056 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2021-02-10T11:08:30.057 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:30.089 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:30 +0000 (0:00:00.064) 0:05:47.378 **** 2021-02-10T11:08:30.126 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2021-02-10T11:08:30.128 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:30.167 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:30 +0000 (0:00:00.077) 0:05:47.455 **** 2021-02-10T11:08:30.205 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-10T11:08:30.206 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2021-02-10T11:08:30.239 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:30 +0000 (0:00:00.071) 0:05:47.527 **** 2021-02-10T11:08:30.300 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2021-02-10T11:08:30.301 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2021-02-10T11:08:30.312 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} skipping: [gibba012.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2021-02-10T11:08:30.323 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2021-02-10T11:08:30.334 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2021-02-10T11:08:30.335 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2021-02-10T11:08:30.347 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2021-02-10T11:08:30.388 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:30 +0000 (0:00:00.149) 0:05:47.677 **** 2021-02-10T11:08:30.587 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* 2021-02-10T11:08:30.589 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["gdisk-0.8.10-3.el7.x86_64 providing gdisk is already installed", "7:device-mapper-1.02.170-6.el7.x86_64 providing device-mapper is already installed"]} 2021-02-10T11:08:30.623 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:30 +0000 (0:00:00.234) 0:05:47.911 **** 2021-02-10T11:08:30.630 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2021-02-10T11:08:30.630 INFO:teuthology.task.ansible.out:changed: [gibba012.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.003662", "end": "2021-02-10 11:08:30.589485", "rc": 0, "start": "2021-02-10 11:08:30.585823", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/virtualenv/bin/teuthology-lock"]} 2021-02-10T11:08:30.673 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:30 +0000 (0:00:00.049) 0:05:47.961 **** 2021-02-10T11:08:30.680 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-02-10T11:08:30.680 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item={'block_used': 13742185, 'uuid': 'aeb0602f-c6fc-4ca1-8920-0340080ffa5a', 'size_total': 984369086464, 'block_total': 240324484, 'mount': '/', 'block_available': 226582299, 'size_available': 928081096704, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,seclabel,relatime,data=ordered', 'device': '/dev/sda1', 'inode_used': 90165, 'block_size': 4096, 'inode_available': 60964811}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 226582299, "block_size": 4096, "block_total": 240324484, "block_used": 13742185, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60964811, "inode_total": 61054976, "inode_used": 90165, "mount": "/", "options": "rw,seclabel,relatime,data=ordered", "size_available": 928081096704, "size_total": 984369086464, "uuid": "aeb0602f-c6fc-4ca1-8920-0340080ffa5a"}} 2021-02-10T11:08:30.718 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:30 +0000 (0:00:00.045) 0:05:48.007 **** 2021-02-10T11:08:30.729 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2021-02-10T11:08:30.730 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:30.772 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:30 +0000 (0:00:00.053) 0:05:48.060 **** 2021-02-10T11:08:30.811 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:30.849 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:30 +0000 (0:00:00.077) 0:05:48.137 **** 2021-02-10T11:08:30.879 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2021-02-10T11:08:30.880 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:30.926 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:30 +0000 (0:00:00.077) 0:05:48.214 **** 2021-02-10T11:08:30.955 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:31.007 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:31 +0000 (0:00:00.080) 0:05:48.295 **** 2021-02-10T11:08:31.174 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2021-02-10T11:08:31.175 INFO:teuthology.task.ansible.out:changed: [gibba020.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.004486", "end": "2021-02-10 11:08:31.161115", "rc": 0, "start": "2021-02-10 11:08:31.156629", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-10T11:08:31.209 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:31 +0000 (0:00:00.202) 0:05:48.498 **** 2021-02-10T11:08:31.249 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2021-02-10T11:08:31.251 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'block_used': 13742185, 'uuid': 'aeb0602f-c6fc-4ca1-8920-0340080ffa5a', 'size_total': 984369086464, 'block_total': 240324484, 'mount': '/', 'block_available': 226582299, 'size_available': 928081096704, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,seclabel,relatime,data=ordered', 'device': '/dev/sda1', 'inode_used': 90165, 'block_size': 4096, 'inode_available': 60964811}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226582299, "block_size": 4096, "block_total": 240324484, "block_used": 13742185, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60964811, "inode_total": 61054976, "inode_used": 90165, "mount": "/", "options": "rw,seclabel,relatime,data=ordered", "size_available": 928081096704, "size_total": 984369086464, "uuid": "aeb0602f-c6fc-4ca1-8920-0340080ffa5a"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:31.286 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:31 +0000 (0:00:00.076) 0:05:48.574 **** 2021-02-10T11:08:31.313 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2021-02-10T11:08:31.347 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:31 +0000 (0:00:00.061) 0:05:48.635 **** 2021-02-10T11:08:31.396 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-02-10T11:08:31.396 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'deadline', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-TOSHIBA_MG04ACA100N_Y9I8K38OF6XF', 'wwn-0x50000399aba8166e'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': 'aeb0602f-c6fc-4ca1-8920-0340080ffa5a', 'links': {'masters': [], 'labels': [], 'ids': ['ata-TOSHIBA_MG04ACA100N_Y9I8K38OF6XF-part1', 'wwn-0x50000399aba8166e-part1'], 'uuids': ['aeb0602f-c6fc-4ca1-8920-0340080ffa5a']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation Cannon Lake PCH SATA AHCI Controller (rev 10)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x50000399aba8166e', 'model': 'TOSHIBA MG04ACA1', 'serial': 'Y9I8K38OF6XF', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation Cannon Lake PCH SATA AHCI Controller (rev 10)", "links": {"ids": ["ata-TOSHIBA_MG04ACA100N_Y9I8K38OF6XF", "wwn-0x50000399aba8166e"], "labels": [], "masters": [], "uuids": []}, "model": "TOSHIBA MG04ACA1", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-TOSHIBA_MG04ACA100N_Y9I8K38OF6XF-part1", "wwn-0x50000399aba8166e-part1"], "labels": [], "masters": [], "uuids": ["aeb0602f-c6fc-4ca1-8920-0340080ffa5a"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "aeb0602f-c6fc-4ca1-8920-0340080ffa5a"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Y9I8K38OF6XF", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x50000399aba8166e"}}, "skip_reason": "Conditiona 2021-02-10T11:08:31.398 INFO:teuthology.task.ansible.out:l result was False"} 2021-02-10T11:08:31.844 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* 2021-02-10T11:08:31.846 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["gdisk-0.8.10-3.el7.x86_64 providing gdisk is already installed", "7:device-mapper-1.02.170-6.el7.x86_64 providing device-mapper is already installed"]} 2021-02-10T11:08:31.878 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:31 +0000 (0:00:00.530) 0:05:49.166 **** 2021-02-10T11:08:31.926 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-02-10T11:08:31.927 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item={'block_used': 13742195, 'uuid': 'aeb0602f-c6fc-4ca1-8920-0340080ffa5a', 'size_total': 984369086464, 'block_total': 240324484, 'mount': '/', 'block_available': 226582289, 'size_available': 928081055744, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,seclabel,relatime,data=ordered', 'device': '/dev/sda1', 'inode_used': 90165, 'block_size': 4096, 'inode_available': 60964811}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 226582289, "block_size": 4096, "block_total": 240324484, "block_used": 13742195, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60964811, "inode_total": 61054976, "inode_used": 90165, "mount": "/", "options": "rw,seclabel,relatime,data=ordered", "size_available": 928081055744, "size_total": 984369086464, "uuid": "aeb0602f-c6fc-4ca1-8920-0340080ffa5a"}} 2021-02-10T11:08:31.961 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:31 +0000 (0:00:00.083) 0:05:49.249 **** 2021-02-10T11:08:32.414 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2021-02-10T11:08:32.415 INFO:teuthology.task.ansible.out:changed: [gibba012.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.004439", "end": "2021-02-10 11:08:32.394696", "rc": 0, "start": "2021-02-10 11:08:32.390257", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-10T11:08:32.458 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:32 +0000 (0:00:00.497) 0:05:49.746 **** 2021-02-10T11:08:32.506 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2021-02-10T11:08:32.508 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'block_used': 13742195, 'uuid': 'aeb0602f-c6fc-4ca1-8920-0340080ffa5a', 'size_total': 984369086464, 'block_total': 240324484, 'mount': '/', 'block_available': 226582289, 'size_available': 928081055744, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,seclabel,relatime,data=ordered', 'device': '/dev/sda1', 'inode_used': 90165, 'block_size': 4096, 'inode_available': 60964811}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226582289, "block_size": 4096, "block_total": 240324484, "block_used": 13742195, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60964811, "inode_total": 61054976, "inode_used": 90165, "mount": "/", "options": "rw,seclabel,relatime,data=ordered", "size_available": 928081055744, "size_total": 984369086464, "uuid": "aeb0602f-c6fc-4ca1-8920-0340080ffa5a"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:32.542 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:32 +0000 (0:00:00.084) 0:05:49.831 **** 2021-02-10T11:08:32.569 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2021-02-10T11:08:32.604 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:32 +0000 (0:00:00.061) 0:05:49.892 **** 2021-02-10T11:08:32.654 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-02-10T11:08:32.654 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'deadline', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-TOSHIBA_MG04ACA100N_Y9I5K317F6XF', 'wwn-0x50000399ab901d77'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': 'aeb0602f-c6fc-4ca1-8920-0340080ffa5a', 'links': {'masters': [], 'labels': [], 'ids': ['ata-TOSHIBA_MG04ACA100N_Y9I5K317F6XF-part1', 'wwn-0x50000399ab901d77-part1'], 'uuids': ['aeb0602f-c6fc-4ca1-8920-0340080ffa5a']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation Cannon Lake PCH SATA AHCI Controller (rev 10)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x50000399ab901d77', 'model': 'TOSHIBA MG04ACA1', 'serial': 'Y9I5K317F6XF', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation Cannon Lake PCH SATA AHCI Controller (rev 10)", "links": {"ids": ["ata-TOSHIBA_MG04ACA100N_Y9I5K317F6XF", "wwn-0x50000399ab901d77"], "labels": [], "masters": [], "uuids": []}, "model": "TOSHIBA MG04ACA1", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-TOSHIBA_MG04ACA100N_Y9I5K317F6XF-part1", "wwn-0x50000399ab901d77-part1"], "labels": [], "masters": [], "uuids": ["aeb0602f-c6fc-4ca1-8920-0340080ffa5a"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "aeb0602f-c6fc-4ca1-8920-0340080ffa5a"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Y9I5K317F6XF", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x50000399ab901d77"}}, "skip_reason": "Conditiona 2021-02-10T11:08:32.656 INFO:teuthology.task.ansible.out:l result was False"} 2021-02-10T11:08:33.187 INFO:teuthology.task.ansible.out:changed: [gibba020.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '732585168', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-UTf90Z-F7kQ-SW19-grje-bhZG-r4Qp-jTk0BH', 'nvme-INTEL_SSDPEL1K375GA_PHKM920000CH375A', 'nvme-nvme.8086-50484b4d393230303030434833373541-494e54454c2053534450454c314b3337354741-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation NVMe Datacenter SSD [Optane]', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'holders': ['vg_nvme-lv_1', 'vg_nvme-lv_2', 'vg_nvme-lv_3', 'vg_nvme-lv_4', 'vg_nvme-lv_5'], 'partitions': {}, 'model': 'INTEL SSDPEL1K375GA', 'serial': 'PHKM920000CH375A', 'size': '349.32 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.347038", "end": "2021-02-10 11:08:33.162305", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_1", "vg_nvme-lv_2", "vg_nvme-lv_3", "vg_nvme-lv_4", "vg_nvme-lv_5"], "host": "Non-Volatile memory controller: Intel Corporation NVMe Datacenter SSD [Optane]", "links": {"ids": ["lvm-pv-uuid-UTf90Z-F7kQ-SW19-grje-bhZG-r4Qp-jTk0BH", "nvme-INTEL_SSDPEL1K375GA_PHKM920000CH375A", "nvme-nvme.8086-50484b4d393230303030434833373541-494e54454c2053534450454c314b3337354741-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEL1K375GA", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "732585168", "sectorsize": "512", "serial": "PHKM920000CH375A", "size": "349.32 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-10 11:08:31.815267", "stderr": "", "stderr_lines": [], "stdout": "Creating new 2021-02-10T11:08:33.188 INFO:teuthology.task.ansible.out:GPT entries.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-02-10T11:08:34.509 INFO:teuthology.task.ansible.out:changed: [gibba012.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '732585168', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-TgviFY-pbWs-rYNQ-JVbX-o7TT-DeZ2-hTMdUa', 'nvme-INTEL_SSDPEL1K375GA_PHKM913400C6375A', 'nvme-nvme.8086-50484b4d393133343030433633373541-494e54454c2053534450454c314b3337354741-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation NVMe Datacenter SSD [Optane]', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'holders': ['vg_nvme-lv_1', 'vg_nvme-lv_2', 'vg_nvme-lv_3', 'vg_nvme-lv_4', 'vg_nvme-lv_5'], 'partitions': {}, 'model': 'INTEL SSDPEL1K375GA', 'serial': 'PHKM913400C6375A', 'size': '349.32 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.240260", "end": "2021-02-10 11:08:34.491599", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_1", "vg_nvme-lv_2", "vg_nvme-lv_3", "vg_nvme-lv_4", "vg_nvme-lv_5"], "host": "Non-Volatile memory controller: Intel Corporation NVMe Datacenter SSD [Optane]", "links": {"ids": ["lvm-pv-uuid-TgviFY-pbWs-rYNQ-JVbX-o7TT-DeZ2-hTMdUa", "nvme-INTEL_SSDPEL1K375GA_PHKM913400C6375A", "nvme-nvme.8086-50484b4d393133343030433633373541-494e54454c2053534450454c314b3337354741-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEL1K375GA", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "732585168", "sectorsize": "512", "serial": "PHKM913400C6375A", "size": "349.32 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-10 11:08:33.251339", "stderr": "", "stderr_lines": [], "stdout": "Creating new 2021-02-10T11:08:34.510 INFO:teuthology.task.ansible.out:GPT entries.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-02-10T11:08:34.678 INFO:teuthology.task.ansible.out:changed: [gibba020.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '27262976', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-1iI8W1KuZDhA7NB0ykPcf7epce1lWxiXSOsS6IER6Krp2E0W216sbj6KUc6ZXX51'], 'uuids': ['78feb474-4059-4855-9029-01ae24e17264']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'holders': [], 'partitions': {}, 'model': None, 'serial': 'PHKM920000CH375A', 'size': '13.00 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.057376", "end": "2021-02-10 11:08:34.661462", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-1iI8W1KuZDhA7NB0ykPcf7epce1lWxiXSOsS6IER6Krp2E0W216sbj6KUc6ZXX51"], "labels": [], "masters": [], "uuids": ["78feb474-4059-4855-9029-01ae24e17264"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "27262976", "sectorsize": "512", "serial": "PHKM920000CH375A", "size": "13.00 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-10 11:08:33.604086", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-02-10T11:08:35.999 INFO:teuthology.task.ansible.out:changed: [gibba012.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '27262976', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-rX1Wdkn4r179wnV8K3m3gHkuRAufJkfqafwQrcvbf15oiGrap8xTBIzt781Zkzdb'], 'uuids': ['30f647b4-4e38-4bc5-bb25-a7082be31f10']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'holders': [], 'partitions': {}, 'model': None, 'serial': 'PHKM913400C6375A', 'size': '13.00 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.051438", "end": "2021-02-10 11:08:35.980843", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-rX1Wdkn4r179wnV8K3m3gHkuRAufJkfqafwQrcvbf15oiGrap8xTBIzt781Zkzdb"], "labels": [], "masters": [], "uuids": ["30f647b4-4e38-4bc5-bb25-a7082be31f10"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "27262976", "sectorsize": "512", "serial": "PHKM913400C6375A", "size": "13.00 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-10 11:08:34.929405", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-02-10T11:08:36.216 INFO:teuthology.task.ansible.out:changed: [gibba020.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '176160768', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-1iI8W1KuZDhA7NB0ykPcf7epce1lWxiXBI2yRfOgQbCwsVrhXLAbhhhvVBf6AQFh'], 'uuids': ['bca6b2bb-bdc4-42cb-a7d3-f818a49ae324']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'holders': [], 'partitions': {}, 'model': None, 'serial': 'PHKM920000CH375A', 'size': '84.00 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.049285", "end": "2021-02-10 11:08:36.143971", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-1iI8W1KuZDhA7NB0ykPcf7epce1lWxiXBI2yRfOgQbCwsVrhXLAbhhhvVBf6AQFh"], "labels": [], "masters": [], "uuids": ["bca6b2bb-bdc4-42cb-a7d3-f818a49ae324"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "176160768", "sectorsize": "512", "serial": "PHKM920000CH375A", "size": "84.00 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-10 11:08:35.094686", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-02-10T11:08:37.520 INFO:teuthology.task.ansible.out:changed: [gibba012.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '176160768', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-rX1Wdkn4r179wnV8K3m3gHkuRAufJkfq7Ex3IRus0HXGfhvO69mbNuhlYYRDHzeF'], 'uuids': ['64555546-2be2-4eff-9105-5122ee71c576']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'holders': [], 'partitions': {}, 'model': None, 'serial': 'PHKM913400C6375A', 'size': '84.00 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.067217", "end": "2021-02-10 11:08:37.497622", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-rX1Wdkn4r179wnV8K3m3gHkuRAufJkfq7Ex3IRus0HXGfhvO69mbNuhlYYRDHzeF"], "labels": [], "masters": [], "uuids": ["64555546-2be2-4eff-9105-5122ee71c576"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "176160768", "sectorsize": "512", "serial": "PHKM913400C6375A", "size": "84.00 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-10 11:08:36.430405", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-02-10T11:08:37.658 INFO:teuthology.task.ansible.out:changed: [gibba020.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '176160768', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-1iI8W1KuZDhA7NB0ykPcf7epce1lWxiX5gxikBbXLCeUvcQubWePKkRF10XvvPlH'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'holders': [], 'partitions': {}, 'model': None, 'serial': 'PHKM920000CH375A', 'size': '84.00 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.057478", "end": "2021-02-10 11:08:37.636759", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-1iI8W1KuZDhA7NB0ykPcf7epce1lWxiX5gxikBbXLCeUvcQubWePKkRF10XvvPlH"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "176160768", "sectorsize": "512", "serial": "PHKM920000CH375A", "size": "84.00 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-10 11:08:36.579281", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-02-10T11:08:39.014 INFO:teuthology.task.ansible.out:changed: [gibba012.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '176160768', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-rX1Wdkn4r179wnV8K3m3gHkuRAufJkfqjghkJlKyxyXym8zojAR2BNQ2t45YWB3D'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'holders': [], 'partitions': {}, 'model': None, 'serial': 'PHKM913400C6375A', 'size': '84.00 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.057964", "end": "2021-02-10 11:08:38.990215", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-rX1Wdkn4r179wnV8K3m3gHkuRAufJkfqjghkJlKyxyXym8zojAR2BNQ2t45YWB3D"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "176160768", "sectorsize": "512", "serial": "PHKM913400C6375A", "size": "84.00 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-10 11:08:37.932251", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-02-10T11:08:39.150 INFO:teuthology.task.ansible.out:changed: [gibba020.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '176160768', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-1iI8W1KuZDhA7NB0ykPcf7epce1lWxiXrujKutTmeo1utN1vCSQ45PGnMiM4AnY9'], 'uuids': ['5bd94596-b24d-4de5-a718-fae6243802e2']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'holders': [], 'partitions': {}, 'model': None, 'serial': 'PHKM920000CH375A', 'size': '84.00 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.055682", "end": "2021-02-10 11:08:39.128737", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-1iI8W1KuZDhA7NB0ykPcf7epce1lWxiXrujKutTmeo1utN1vCSQ45PGnMiM4AnY9"], "labels": [], "masters": [], "uuids": ["5bd94596-b24d-4de5-a718-fae6243802e2"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "176160768", "sectorsize": "512", "serial": "PHKM920000CH375A", "size": "84.00 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-10 11:08:38.073055", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-02-10T11:08:40.510 INFO:teuthology.task.ansible.out:changed: [gibba012.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '176160768', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-rX1Wdkn4r179wnV8K3m3gHkuRAufJkfqnI8qjPHQfVHQUZu4kbsv7DYRPkE5A2YF'], 'uuids': ['81a8d804-d67f-4342-b4ae-b629aca46b67']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'holders': [], 'partitions': {}, 'model': None, 'serial': 'PHKM913400C6375A', 'size': '84.00 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.050762", "end": "2021-02-10 11:08:40.482420", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-rX1Wdkn4r179wnV8K3m3gHkuRAufJkfqnI8qjPHQfVHQUZu4kbsv7DYRPkE5A2YF"], "labels": [], "masters": [], "uuids": ["81a8d804-d67f-4342-b4ae-b629aca46b67"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "176160768", "sectorsize": "512", "serial": "PHKM913400C6375A", "size": "84.00 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-10 11:08:39.431658", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-02-10T11:08:40.658 INFO:teuthology.task.ansible.out:changed: [gibba020.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '176160768', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-1iI8W1KuZDhA7NB0ykPcf7epce1lWxiXFXc8PMf0sNTGdwAbhHmV1KDYrjfK5re8'], 'uuids': ['3ec86837-bfe2-460c-984d-d491c2071f57']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'holders': [], 'partitions': {}, 'model': None, 'serial': 'PHKM920000CH375A', 'size': '84.00 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.057711", "end": "2021-02-10 11:08:40.637642", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-1iI8W1KuZDhA7NB0ykPcf7epce1lWxiXFXc8PMf0sNTGdwAbhHmV1KDYrjfK5re8"], "labels": [], "masters": [], "uuids": ["3ec86837-bfe2-460c-984d-d491c2071f57"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "176160768", "sectorsize": "512", "serial": "PHKM920000CH375A", "size": "84.00 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-10 11:08:39.579931", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-02-10T11:08:40.700 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:40 +0000 (0:00:08.095) 0:05:57.988 **** 2021-02-10T11:08:40.748 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-02-10T11:08:40.749 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'deadline', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-TOSHIBA_MG04ACA100N_Y9I8K38OF6XF', 'wwn-0x50000399aba8166e'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': 'aeb0602f-c6fc-4ca1-8920-0340080ffa5a', 'links': {'masters': [], 'labels': [], 'ids': ['ata-TOSHIBA_MG04ACA100N_Y9I8K38OF6XF-part1', 'wwn-0x50000399aba8166e-part1'], 'uuids': ['aeb0602f-c6fc-4ca1-8920-0340080ffa5a']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation Cannon Lake PCH SATA AHCI Controller (rev 10)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x50000399aba8166e', 'model': 'TOSHIBA MG04ACA1', 'serial': 'Y9I8K38OF6XF', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation Cannon Lake PCH SATA AHCI Controller (rev 10)", "links": {"ids": ["ata-TOSHIBA_MG04ACA100N_Y9I8K38OF6XF", "wwn-0x50000399aba8166e"], "labels": [], "masters": [], "uuids": []}, "model": "TOSHIBA MG04ACA1", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-TOSHIBA_MG04ACA100N_Y9I8K38OF6XF-part1", "wwn-0x50000399aba8166e-part1"], "labels": [], "masters": [], "uuids": ["aeb0602f-c6fc-4ca1-8920-0340080ffa5a"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "aeb0602f-c6fc-4ca1-8920-0340080ffa5a"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Y9I8K38OF6XF", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x50000399aba8166e"}}, "skip_reason": "Conditiona 2021-02-10T11:08:40.750 INFO:teuthology.task.ansible.out:l result was False"} 2021-02-10T11:08:41.209 INFO:teuthology.task.ansible.out:changed: [gibba020.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '732585168', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-UTf90Z-F7kQ-SW19-grje-bhZG-r4Qp-jTk0BH', 'nvme-INTEL_SSDPEL1K375GA_PHKM920000CH375A', 'nvme-nvme.8086-50484b4d393230303030434833373541-494e54454c2053534450454c314b3337354741-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation NVMe Datacenter SSD [Optane]', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'holders': ['vg_nvme-lv_1', 'vg_nvme-lv_2', 'vg_nvme-lv_3', 'vg_nvme-lv_4', 'vg_nvme-lv_5'], 'partitions': {}, 'model': 'INTEL SSDPEL1K375GA', 'serial': 'PHKM920000CH375A', 'size': '349.32 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.025575", "end": "2021-02-10 11:08:41.189267", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_1", "vg_nvme-lv_2", "vg_nvme-lv_3", "vg_nvme-lv_4", "vg_nvme-lv_5"], "host": "Non-Volatile memory controller: Intel Corporation NVMe Datacenter SSD [Optane]", "links": {"ids": ["lvm-pv-uuid-UTf90Z-F7kQ-SW19-grje-bhZG-r4Qp-jTk0BH", "nvme-INTEL_SSDPEL1K375GA_PHKM920000CH375A", "nvme-nvme.8086-50484b4d393230303030434833373541-494e54454c2053534450454c314b3337354741-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEL1K375GA", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "732585168", "sectorsize": "512", "serial": "PHKM920000CH375A", "size": "349.32 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-10 11:08:41.163692", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) 2021-02-10T11:08:41.210 INFO:teuthology.task.ansible.out:copied, 0.0234043 s, 4.9 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.0234043 s, 4.9 GB/s"], "stdout": "", "stdout_lines": []} 2021-02-10T11:08:41.721 INFO:teuthology.task.ansible.out:changed: [gibba020.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '27262976', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-1iI8W1KuZDhA7NB0ykPcf7epce1lWxiXSOsS6IER6Krp2E0W216sbj6KUc6ZXX51'], 'uuids': ['78feb474-4059-4855-9029-01ae24e17264']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'holders': [], 'partitions': {}, 'model': None, 'serial': 'PHKM920000CH375A', 'size': '13.00 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.079963", "end": "2021-02-10 11:08:41.706385", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-1iI8W1KuZDhA7NB0ykPcf7epce1lWxiXSOsS6IER6Krp2E0W216sbj6KUc6ZXX51"], "labels": [], "masters": [], "uuids": ["78feb474-4059-4855-9029-01ae24e17264"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "27262976", "sectorsize": "512", "serial": "PHKM920000CH375A", "size": "13.00 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-10 11:08:41.626422", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) copied, 0.0779135 s, 1.5 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.0779135 s, 1.5 GB/s"], "stdout": "", "stdout_lines": []} 2021-02-10T11:08:42.015 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-02-10T11:08:42.016 INFO:teuthology.task.ansible.out:changed: [gibba012.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '176160768', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-rX1Wdkn4r179wnV8K3m3gHkuRAufJkfqEtQdddKbRnNwdg629x98XvOE02jJo8qY'], 'uuids': ['c1a9a2bc-574e-429c-801f-610f6e5fe2c7']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'holders': [], 'partitions': {}, 'model': None, 'serial': 'PHKM913400C6375A', 'size': '84.00 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.054197", "end": "2021-02-10 11:08:41.998727", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-rX1Wdkn4r179wnV8K3m3gHkuRAufJkfqEtQdddKbRnNwdg629x98XvOE02jJo8qY"], "labels": [], "masters": [], "uuids": ["c1a9a2bc-574e-429c-801f-610f6e5fe2c7"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "176160768", "sectorsize": "512", "serial": "PHKM913400C6375A", "size": "84.00 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-10 11:08:40.944530", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-02-10T11:08:42.061 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:42 +0000 (0:00:01.361) 0:05:59.350 **** 2021-02-10T11:08:42.111 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-02-10T11:08:42.112 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'deadline', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-TOSHIBA_MG04ACA100N_Y9I5K317F6XF', 'wwn-0x50000399ab901d77'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': 'aeb0602f-c6fc-4ca1-8920-0340080ffa5a', 'links': {'masters': [], 'labels': [], 'ids': ['ata-TOSHIBA_MG04ACA100N_Y9I5K317F6XF-part1', 'wwn-0x50000399ab901d77-part1'], 'uuids': ['aeb0602f-c6fc-4ca1-8920-0340080ffa5a']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation Cannon Lake PCH SATA AHCI Controller (rev 10)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x50000399ab901d77', 'model': 'TOSHIBA MG04ACA1', 'serial': 'Y9I5K317F6XF', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation Cannon Lake PCH SATA AHCI Controller (rev 10)", "links": {"ids": ["ata-TOSHIBA_MG04ACA100N_Y9I5K317F6XF", "wwn-0x50000399ab901d77"], "labels": [], "masters": [], "uuids": []}, "model": "TOSHIBA MG04ACA1", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-TOSHIBA_MG04ACA100N_Y9I5K317F6XF-part1", "wwn-0x50000399ab901d77-part1"], "labels": [], "masters": [], "uuids": ["aeb0602f-c6fc-4ca1-8920-0340080ffa5a"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "aeb0602f-c6fc-4ca1-8920-0340080ffa5a"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Y9I5K317F6XF", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x50000399ab901d77"}}, "skip_reason": "Conditiona 2021-02-10T11:08:42.113 INFO:teuthology.task.ansible.out:l result was False"} 2021-02-10T11:08:42.240 INFO:teuthology.task.ansible.out:changed: [gibba020.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '176160768', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-1iI8W1KuZDhA7NB0ykPcf7epce1lWxiXBI2yRfOgQbCwsVrhXLAbhhhvVBf6AQFh'], 'uuids': ['bca6b2bb-bdc4-42cb-a7d3-f818a49ae324']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'holders': [], 'partitions': {}, 'model': None, 'serial': 'PHKM920000CH375A', 'size': '84.00 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.079541", "end": "2021-02-10 11:08:42.218244", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-1iI8W1KuZDhA7NB0ykPcf7epce1lWxiXBI2yRfOgQbCwsVrhXLAbhhhvVBf6AQFh"], "labels": [], "masters": [], "uuids": ["bca6b2bb-bdc4-42cb-a7d3-f818a49ae324"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "176160768", "sectorsize": "512", "serial": "PHKM920000CH375A", "size": "84.00 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-10 11:08:42.138703", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) copied, 0.0774721 s, 1.5 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.0774721 s, 1.5 GB/s"], "stdout": "", "stdout_lines": []} 2021-02-10T11:08:42.577 INFO:teuthology.task.ansible.out:changed: [gibba012.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '732585168', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-TgviFY-pbWs-rYNQ-JVbX-o7TT-DeZ2-hTMdUa', 'nvme-INTEL_SSDPEL1K375GA_PHKM913400C6375A', 'nvme-nvme.8086-50484b4d393133343030433633373541-494e54454c2053534450454c314b3337354741-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation NVMe Datacenter SSD [Optane]', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'holders': ['vg_nvme-lv_1', 'vg_nvme-lv_2', 'vg_nvme-lv_3', 'vg_nvme-lv_4', 'vg_nvme-lv_5'], 'partitions': {}, 'model': 'INTEL SSDPEL1K375GA', 'serial': 'PHKM913400C6375A', 'size': '349.32 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.026231", "end": "2021-02-10 11:08:42.552761", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_1", "vg_nvme-lv_2", "vg_nvme-lv_3", "vg_nvme-lv_4", "vg_nvme-lv_5"], "host": "Non-Volatile memory controller: Intel Corporation NVMe Datacenter SSD [Optane]", "links": {"ids": ["lvm-pv-uuid-TgviFY-pbWs-rYNQ-JVbX-o7TT-DeZ2-hTMdUa", "nvme-INTEL_SSDPEL1K375GA_PHKM913400C6375A", "nvme-nvme.8086-50484b4d393133343030433633373541-494e54454c2053534450454c314b3337354741-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEL1K375GA", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "732585168", "sectorsize": "512", "serial": "PHKM913400C6375A", "size": "349.32 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-10 11:08:42.526530", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) 2021-02-10T11:08:42.578 INFO:teuthology.task.ansible.out:copied, 0.0240752 s, 4.8 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.0240752 s, 4.8 GB/s"], "stdout": "", "stdout_lines": []} 2021-02-10T11:08:42.735 INFO:teuthology.task.ansible.out:changed: [gibba020.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '176160768', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-1iI8W1KuZDhA7NB0ykPcf7epce1lWxiX5gxikBbXLCeUvcQubWePKkRF10XvvPlH'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'holders': [], 'partitions': {}, 'model': None, 'serial': 'PHKM920000CH375A', 'size': '84.00 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.062190", "end": "2021-02-10 11:08:42.711678", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-1iI8W1KuZDhA7NB0ykPcf7epce1lWxiX5gxikBbXLCeUvcQubWePKkRF10XvvPlH"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "176160768", "sectorsize": "512", "serial": "PHKM920000CH375A", "size": "84.00 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-10 11:08:42.649488", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) copied, 0.0600716 s, 1.9 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.0600716 s, 1.9 GB/s"], "stdout": "", "stdout_lines": []} 2021-02-10T11:08:43.085 INFO:teuthology.task.ansible.out:changed: [gibba012.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '27262976', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-rX1Wdkn4r179wnV8K3m3gHkuRAufJkfqafwQrcvbf15oiGrap8xTBIzt781Zkzdb'], 'uuids': ['30f647b4-4e38-4bc5-bb25-a7082be31f10']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'holders': [], 'partitions': {}, 'model': None, 'serial': 'PHKM913400C6375A', 'size': '13.00 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.081298", "end": "2021-02-10 11:08:43.065905", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-rX1Wdkn4r179wnV8K3m3gHkuRAufJkfqafwQrcvbf15oiGrap8xTBIzt781Zkzdb"], "labels": [], "masters": [], "uuids": ["30f647b4-4e38-4bc5-bb25-a7082be31f10"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "27262976", "sectorsize": "512", "serial": "PHKM913400C6375A", "size": "13.00 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-10 11:08:42.984607", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) copied, 0.0792192 s, 1.5 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.0792192 s, 1.5 GB/s"], "stdout": "", "stdout_lines": []} 2021-02-10T11:08:43.253 INFO:teuthology.task.ansible.out:changed: [gibba020.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '176160768', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-1iI8W1KuZDhA7NB0ykPcf7epce1lWxiXrujKutTmeo1utN1vCSQ45PGnMiM4AnY9'], 'uuids': ['5bd94596-b24d-4de5-a718-fae6243802e2']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'holders': [], 'partitions': {}, 'model': None, 'serial': 'PHKM920000CH375A', 'size': '84.00 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.079598", "end": "2021-02-10 11:08:43.231536", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-1iI8W1KuZDhA7NB0ykPcf7epce1lWxiXrujKutTmeo1utN1vCSQ45PGnMiM4AnY9"], "labels": [], "masters": [], "uuids": ["5bd94596-b24d-4de5-a718-fae6243802e2"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "176160768", "sectorsize": "512", "serial": "PHKM920000CH375A", "size": "84.00 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-10 11:08:43.151938", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) copied, 0.0775141 s, 1.5 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.0775141 s, 1.5 GB/s"], "stdout": "", "stdout_lines": []} 2021-02-10T11:08:43.601 INFO:teuthology.task.ansible.out:changed: [gibba012.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '176160768', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-rX1Wdkn4r179wnV8K3m3gHkuRAufJkfq7Ex3IRus0HXGfhvO69mbNuhlYYRDHzeF'], 'uuids': ['64555546-2be2-4eff-9105-5122ee71c576']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'holders': [], 'partitions': {}, 'model': None, 'serial': 'PHKM913400C6375A', 'size': '84.00 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.079804", "end": "2021-02-10 11:08:43.583736", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-rX1Wdkn4r179wnV8K3m3gHkuRAufJkfq7Ex3IRus0HXGfhvO69mbNuhlYYRDHzeF"], "labels": [], "masters": [], "uuids": ["64555546-2be2-4eff-9105-5122ee71c576"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "176160768", "sectorsize": "512", "serial": "PHKM913400C6375A", "size": "84.00 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-10 11:08:43.503932", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) copied, 0.0777125 s, 1.5 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.0777125 s, 1.5 GB/s"], "stdout": "", "stdout_lines": []} 2021-02-10T11:08:43.759 INFO:teuthology.task.ansible.out:changed: [gibba020.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '176160768', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-1iI8W1KuZDhA7NB0ykPcf7epce1lWxiXFXc8PMf0sNTGdwAbhHmV1KDYrjfK5re8'], 'uuids': ['3ec86837-bfe2-460c-984d-d491c2071f57']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'holders': [], 'partitions': {}, 'model': None, 'serial': 'PHKM920000CH375A', 'size': '84.00 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.083187", "end": "2021-02-10 11:08:43.744572", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-1iI8W1KuZDhA7NB0ykPcf7epce1lWxiXFXc8PMf0sNTGdwAbhHmV1KDYrjfK5re8"], "labels": [], "masters": [], "uuids": ["3ec86837-bfe2-460c-984d-d491c2071f57"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "176160768", "sectorsize": "512", "serial": "PHKM920000CH375A", "size": "84.00 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-10 11:08:43.661385", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) copied, 0.0810265 s, 1.4 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.0810265 s, 1.4 GB/s"], "stdout": "", "stdout_lines": []} 2021-02-10T11:08:43.800 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:43 +0000 (0:00:01.738) 0:06:01.089 **** 2021-02-10T11:08:44.123 INFO:teuthology.task.ansible.out:changed: [gibba012.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '176160768', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-rX1Wdkn4r179wnV8K3m3gHkuRAufJkfqjghkJlKyxyXym8zojAR2BNQ2t45YWB3D'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'holders': [], 'partitions': {}, 'model': None, 'serial': 'PHKM913400C6375A', 'size': '84.00 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.081003", "end": "2021-02-10 11:08:44.105583", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-rX1Wdkn4r179wnV8K3m3gHkuRAufJkfqjghkJlKyxyXym8zojAR2BNQ2t45YWB3D"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "176160768", "sectorsize": "512", "serial": "PHKM913400C6375A", "size": "84.00 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-10 11:08:44.024580", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) copied, 0.0789208 s, 1.5 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.0789208 s, 1.5 GB/s"], "stdout": "", "stdout_lines": []} 2021-02-10T11:08:44.334 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2021-02-10T11:08:44.335 INFO:teuthology.task.ansible.out:changed: [gibba020.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.088888", "end": "2021-02-10 11:08:44.315441", "rc": 0, "start": "2021-02-10 11:08:44.226553", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-10T11:08:44.370 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:44 +0000 (0:00:00.569) 0:06:01.658 **** 2021-02-10T11:08:44.638 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-02-10T11:08:44.639 INFO:teuthology.task.ansible.out:changed: [gibba012.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '176160768', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-rX1Wdkn4r179wnV8K3m3gHkuRAufJkfqnI8qjPHQfVHQUZu4kbsv7DYRPkE5A2YF'], 'uuids': ['81a8d804-d67f-4342-b4ae-b629aca46b67']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'holders': [], 'partitions': {}, 'model': None, 'serial': 'PHKM913400C6375A', 'size': '84.00 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.081314", "end": "2021-02-10 11:08:44.616475", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-rX1Wdkn4r179wnV8K3m3gHkuRAufJkfqnI8qjPHQfVHQUZu4kbsv7DYRPkE5A2YF"], "labels": [], "masters": [], "uuids": ["81a8d804-d67f-4342-b4ae-b629aca46b67"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "176160768", "sectorsize": "512", "serial": "PHKM913400C6375A", "size": "84.00 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-10 11:08:44.535161", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) copied, 0.0791302 s, 1.5 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.0791302 s, 1.5 GB/s"], "stdout": "", "stdout_lines": []} 2021-02-10T11:08:44.841 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2021-02-10T11:08:44.842 INFO:teuthology.task.ansible.out:changed: [gibba020.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.030574", "end": "2021-02-10 11:08:44.825367", "rc": 0, "start": "2021-02-10 11:08:44.794793", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-10T11:08:44.877 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:44 +0000 (0:00:00.506) 0:06:02.165 **** 2021-02-10T11:08:44.904 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2021-02-10T11:08:44.940 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:44 +0000 (0:00:00.063) 0:06:02.228 **** 2021-02-10T11:08:44.967 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2021-02-10T11:08:44.968 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:45.002 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:45 +0000 (0:00:00.062) 0:06:02.291 **** 2021-02-10T11:08:45.030 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2021-02-10T11:08:45.031 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:45.063 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:45 +0000 (0:00:00.060) 0:06:02.351 **** 2021-02-10T11:08:45.090 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-02-10T11:08:45.091 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:45.127 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:45 +0000 (0:00:00.064) 0:06:02.416 **** 2021-02-10T11:08:45.144 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-02-10T11:08:45.145 INFO:teuthology.task.ansible.out:changed: [gibba012.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '176160768', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-rX1Wdkn4r179wnV8K3m3gHkuRAufJkfqEtQdddKbRnNwdg629x98XvOE02jJo8qY'], 'uuids': ['c1a9a2bc-574e-429c-801f-610f6e5fe2c7']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'holders': [], 'partitions': {}, 'model': None, 'serial': 'PHKM913400C6375A', 'size': '84.00 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.081391", "end": "2021-02-10 11:08:45.125834", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-rX1Wdkn4r179wnV8K3m3gHkuRAufJkfqEtQdddKbRnNwdg629x98XvOE02jJo8qY"], "labels": [], "masters": [], "uuids": ["c1a9a2bc-574e-429c-801f-610f6e5fe2c7"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "176160768", "sectorsize": "512", "serial": "PHKM913400C6375A", "size": "84.00 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-10 11:08:45.044443", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) copied, 0.0792902 s, 1.5 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.0792902 s, 1.5 GB/s"], "stdout": "", "stdout_lines": []} 2021-02-10T11:08:45.184 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:45 +0000 (0:00:00.056) 0:06:02.472 **** 2021-02-10T11:08:45.191 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2021-02-10T11:08:45.192 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:45.226 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:45 +0000 (0:00:00.041) 0:06:02.514 **** 2021-02-10T11:08:45.672 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2021-02-10T11:08:45.673 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-02-10T11:08:45.707 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:45 +0000 (0:00:00.481) 0:06:02.996 **** 2021-02-10T11:08:45.716 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2021-02-10T11:08:45.716 INFO:teuthology.task.ansible.out:changed: [gibba012.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.068668", "end": "2021-02-10 11:08:45.676522", "rc": 0, "start": "2021-02-10 11:08:45.607854", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-10T11:08:45.754 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:45 +0000 (0:00:00.046) 0:06:03.042 **** 2021-02-10T11:08:45.761 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-02-10T11:08:45.763 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'block_used': 13742185, 'uuid': 'aeb0602f-c6fc-4ca1-8920-0340080ffa5a', 'size_total': 984369086464, 'block_total': 240324484, 'mount': '/', 'block_available': 226582299, 'size_available': 928081096704, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,seclabel,relatime,data=ordered', 'device': '/dev/sda1', 'inode_used': 90165, 'block_size': 4096, 'inode_available': 60964811}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226582299, "block_size": 4096, "block_total": 240324484, "block_used": 13742185, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60964811, "inode_total": 61054976, "inode_used": 90165, "mount": "/", "options": "rw,seclabel,relatime,data=ordered", "size_available": 928081096704, "size_total": 984369086464, "uuid": "aeb0602f-c6fc-4ca1-8920-0340080ffa5a"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:45.807 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:45 +0000 (0:00:00.053) 0:06:03.096 **** 2021-02-10T11:08:45.836 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:45.874 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:45 +0000 (0:00:00.066) 0:06:03.162 **** 2021-02-10T11:08:45.912 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:45.947 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:45 +0000 (0:00:00.073) 0:06:03.235 **** 2021-02-10T11:08:45.974 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-10T11:08:45.975 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:46.008 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:46 +0000 (0:00:00.060) 0:06:03.296 **** 2021-02-10T11:08:46.035 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2021-02-10T11:08:46.036 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:46.068 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:46 +0000 (0:00:00.060) 0:06:03.356 **** 2021-02-10T11:08:46.095 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2021-02-10T11:08:46.096 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:46.129 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:46 +0000 (0:00:00.060) 0:06:03.417 **** 2021-02-10T11:08:46.159 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:46.193 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:46 +0000 (0:00:00.064) 0:06:03.481 **** 2021-02-10T11:08:46.231 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2021-02-10T11:08:46.233 INFO:teuthology.task.ansible.out:changed: [gibba012.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.032490", "end": "2021-02-10 11:08:46.213935", "rc": 0, "start": "2021-02-10 11:08:46.181445", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-10T11:08:46.269 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:46 +0000 (0:00:00.075) 0:06:03.557 **** 2021-02-10T11:08:46.297 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2021-02-10T11:08:46.337 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:46 +0000 (0:00:00.067) 0:06:03.625 **** 2021-02-10T11:08:46.364 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2021-02-10T11:08:46.366 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:46.403 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:46 +0000 (0:00:00.065) 0:06:03.691 **** 2021-02-10T11:08:46.430 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2021-02-10T11:08:46.432 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:46.465 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:46 +0000 (0:00:00.061) 0:06:03.753 **** 2021-02-10T11:08:46.492 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-02-10T11:08:46.493 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:46.531 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:46 +0000 (0:00:00.065) 0:06:03.819 **** 2021-02-10T11:08:46.558 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2021-02-10T11:08:46.560 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:46.597 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:46 +0000 (0:00:00.066) 0:06:03.886 **** 2021-02-10T11:08:47.034 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** 2021-02-10T11:08:47.035 INFO:teuthology.task.ansible.out:changed: [gibba020.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2021-02-10T11:08:47.037 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ ok: [gibba012.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-02-10T11:08:47.072 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:47 +0000 (0:00:00.475) 0:06:04.361 **** 2021-02-10T11:08:47.105 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:47 +0000 (0:00:00.031) 0:06:04.393 **** 2021-02-10T11:08:47.121 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-02-10T11:08:47.122 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'block_used': 13742195, 'uuid': 'aeb0602f-c6fc-4ca1-8920-0340080ffa5a', 'size_total': 984369086464, 'block_total': 240324484, 'mount': '/', 'block_available': 226582289, 'size_available': 928081055744, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,seclabel,relatime,data=ordered', 'device': '/dev/sda1', 'inode_used': 90165, 'block_size': 4096, 'inode_available': 60964811}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226582289, "block_size": 4096, "block_total": 240324484, "block_used": 13742195, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60964811, "inode_total": 61054976, "inode_used": 90165, "mount": "/", "options": "rw,seclabel,relatime,data=ordered", "size_available": 928081055744, "size_total": 984369086464, "uuid": "aeb0602f-c6fc-4ca1-8920-0340080ffa5a"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:47.158 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:47 +0000 (0:00:00.053) 0:06:04.446 **** 2021-02-10T11:08:47.186 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2021-02-10T11:08:47.187 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:47.219 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:47 +0000 (0:00:00.061) 0:06:04.508 **** 2021-02-10T11:08:47.257 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-10T11:08:47.258 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:47.291 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:47 +0000 (0:00:00.071) 0:06:04.579 **** 2021-02-10T11:08:47.319 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-10T11:08:47.320 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:47.354 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:47 +0000 (0:00:00.062) 0:06:04.642 **** 2021-02-10T11:08:47.381 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2021-02-10T11:08:47.383 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:47.416 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:47 +0000 (0:00:00.062) 0:06:04.704 **** 2021-02-10T11:08:47.443 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2021-02-10T11:08:47.444 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:47.476 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:47 +0000 (0:00:00.059) 0:06:04.764 **** 2021-02-10T11:08:47.503 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2021-02-10T11:08:47.504 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:47.538 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:47 +0000 (0:00:00.061) 0:06:04.826 **** 2021-02-10T11:08:47.927 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2021-02-10T11:08:47.928 INFO:teuthology.task.ansible.out:changed: [gibba020.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '84G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "84G", "vg": "vg_nvme"}}, "msg": ""} 2021-02-10T11:08:48.126 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** 2021-02-10T11:08:48.128 INFO:teuthology.task.ansible.out:changed: [gibba012.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2021-02-10T11:08:48.162 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:48 +0000 (0:00:00.624) 0:06:05.450 **** 2021-02-10T11:08:48.445 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2021-02-10T11:08:48.447 INFO:teuthology.task.ansible.out:changed: [gibba020.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '84G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "84G", "vg": "vg_nvme"}}, "msg": ""} 2021-02-10T11:08:48.728 INFO:teuthology.task.ansible.out:changed: [gibba012.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '84G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "84G", "vg": "vg_nvme"}}, "msg": ""} 2021-02-10T11:08:48.970 INFO:teuthology.task.ansible.out:changed: [gibba020.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '84G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "84G", "vg": "vg_nvme"}}, "msg": ""} 2021-02-10T11:08:49.260 INFO:teuthology.task.ansible.out:changed: [gibba012.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '84G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "84G", "vg": "vg_nvme"}}, "msg": ""} 2021-02-10T11:08:49.500 INFO:teuthology.task.ansible.out:changed: [gibba020.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '84G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "84G", "vg": "vg_nvme"}}, "msg": ""} 2021-02-10T11:08:49.791 INFO:teuthology.task.ansible.out:changed: [gibba012.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '84G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "84G", "vg": "vg_nvme"}}, "msg": ""} 2021-02-10T11:08:50.032 INFO:teuthology.task.ansible.out:changed: [gibba020.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '13G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "13G", "vg": "vg_nvme"}}, "msg": ""} 2021-02-10T11:08:50.071 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:50 +0000 (0:00:01.908) 0:06:07.359 **** 2021-02-10T11:08:50.318 INFO:teuthology.task.ansible.out:changed: [gibba012.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '84G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "84G", "vg": "vg_nvme"}}, "msg": ""} 2021-02-10T11:08:50.519 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-02-10T11:08:50.521 INFO:teuthology.task.ansible.out:changed: [gibba020.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2021-02-10T11:08:50.553 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:50 +0000 (0:00:00.482) 0:06:07.841 **** 2021-02-10T11:08:50.841 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2021-02-10T11:08:50.843 INFO:teuthology.task.ansible.out:changed: [gibba012.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '13G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "13G", "vg": "vg_nvme"}}, "msg": ""} 2021-02-10T11:08:50.878 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:50 +0000 (0:00:00.325) 0:06:08.166 **** 2021-02-10T11:08:51.029 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2021-02-10T11:08:51.030 INFO:teuthology.task.ansible.out:changed: [gibba020.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '84G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "84G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-02-10T11:08:51.334 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-02-10T11:08:51.335 INFO:teuthology.task.ansible.out:changed: [gibba012.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2021-02-10T11:08:51.368 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:51 +0000 (0:00:00.490) 0:06:08.656 **** 2021-02-10T11:08:51.450 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2021-02-10T11:08:51.451 INFO:teuthology.task.ansible.out:changed: [gibba020.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '84G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "84G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-02-10T11:08:51.829 INFO:teuthology.task.ansible.out:changed: [gibba012.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '84G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "84G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-02-10T11:08:51.881 INFO:teuthology.task.ansible.out:changed: [gibba020.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '84G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "84G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-02-10T11:08:52.258 INFO:teuthology.task.ansible.out:changed: [gibba012.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '84G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "84G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-02-10T11:08:52.323 INFO:teuthology.task.ansible.out:changed: [gibba020.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '84G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "84G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-02-10T11:08:52.334 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '13G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "13G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:52.372 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:52 +0000 (0:00:01.003) 0:06:09.660 **** 2021-02-10T11:08:52.399 INFO:teuthology.task.ansible.out: TASK [testnode : Create filesystems] ******************************************* 2021-02-10T11:08:52.400 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:52.434 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:52 +0000 (0:00:00.062) 0:06:09.722 **** 2021-02-10T11:08:52.463 INFO:teuthology.task.ansible.out: TASK [testnode : Mount filesystems] ******************************************** 2021-02-10T11:08:52.464 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:52.501 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:52 +0000 (0:00:00.066) 0:06:09.789 **** 2021-02-10T11:08:52.689 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2021-02-10T11:08:52.689 INFO:teuthology.task.ansible.out:changed: [gibba012.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '84G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "84G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-02-10T11:08:52.948 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** 2021-02-10T11:08:52.950 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "secontext": "unconfined_u:object_r:var_lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-02-10T11:08:52.982 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:52 +0000 (0:00:00.481) 0:06:10.271 **** 2021-02-10T11:08:53.020 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2021-02-10T11:08:53.021 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:53.054 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:53 +0000 (0:00:00.072) 0:06:10.343 **** 2021-02-10T11:08:53.135 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2021-02-10T11:08:53.136 INFO:teuthology.task.ansible.out:changed: [gibba012.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '84G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "84G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-02-10T11:08:53.147 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '13G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "13G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:53.183 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:53 +0000 (0:00:00.128) 0:06:10.471 **** 2021-02-10T11:08:53.212 INFO:teuthology.task.ansible.out: TASK [testnode : Create filesystems] ******************************************* 2021-02-10T11:08:53.213 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:53.245 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:53 +0000 (0:00:00.061) 0:06:10.533 **** 2021-02-10T11:08:53.282 INFO:teuthology.task.ansible.out: TASK [testnode : Mount filesystems] ******************************************** 2021-02-10T11:08:53.284 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:53.321 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:53 +0000 (0:00:00.076) 0:06:10.610 **** 2021-02-10T11:08:53.767 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** 2021-02-10T11:08:53.769 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "secontext": "unconfined_u:object_r:var_lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-02-10T11:08:53.801 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:53 +0000 (0:00:00.479) 0:06:11.089 **** 2021-02-10T11:08:53.828 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* 2021-02-10T11:08:53.829 INFO:teuthology.task.ansible.out:changed: [gibba020.front.sepia.ceph.com] => {"changed": true} TASK [testnode : Set xfs_opts on newer OSes] *********************************** skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:53.862 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:53 +0000 (0:00:00.061) 0:06:11.150 **** 2021-02-10T11:08:53.896 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:53 +0000 (0:00:00.033) 0:06:11.184 **** 2021-02-10T11:08:54.341 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* changed: [gibba012.front.sepia.ceph.com] => {"changed": true} 2021-02-10T11:08:54.377 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:54 +0000 (0:00:00.480) 0:06:11.665 **** 2021-02-10T11:08:54.698 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* 2021-02-10T11:08:54.699 INFO:teuthology.task.ansible.out:changed: [gibba020.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2021-02-10T11:08:54.733 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:54 +0000 (0:00:00.356) 0:06:12.021 **** 2021-02-10T11:08:54.760 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd] ****************************************** 2021-02-10T11:08:54.761 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:54.793 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:54 +0000 (0:00:00.060) 0:06:12.082 **** 2021-02-10T11:08:54.820 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd conf file] ******************************** 2021-02-10T11:08:54.822 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:54.854 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:54 +0000 (0:00:00.060) 0:06:12.142 **** 2021-02-10T11:08:54.880 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** 2021-02-10T11:08:54.882 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:54.921 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:54 +0000 (0:00:00.067) 0:06:12.209 **** 2021-02-10T11:08:54.928 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* 2021-02-10T11:08:54.929 INFO:teuthology.task.ansible.out:changed: [gibba012.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2021-02-10T11:08:54.963 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:54 +0000 (0:00:00.041) 0:06:12.251 **** 2021-02-10T11:08:54.971 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** 2021-02-10T11:08:54.973 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:55.006 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:55 +0000 (0:00:00.042) 0:06:12.294 **** 2021-02-10T11:08:55.012 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd] ****************************************** 2021-02-10T11:08:55.014 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:55.051 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:55 +0000 (0:00:00.045) 0:06:12.339 **** 2021-02-10T11:08:55.058 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** 2021-02-10T11:08:55.060 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:55.097 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:55 +0000 (0:00:00.045) 0:06:12.385 **** 2021-02-10T11:08:55.104 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd conf file] ******************************** 2021-02-10T11:08:55.105 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:55.141 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:55 +0000 (0:00:00.044) 0:06:12.429 **** 2021-02-10T11:08:55.148 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** 2021-02-10T11:08:55.149 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:55.183 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:55 +0000 (0:00:00.041) 0:06:12.471 **** 2021-02-10T11:08:55.189 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** 2021-02-10T11:08:55.191 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:55.224 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:55 +0000 (0:00:00.040) 0:06:12.512 **** 2021-02-10T11:08:55.230 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** 2021-02-10T11:08:55.231 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:55.266 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:55 +0000 (0:00:00.042) 0:06:12.555 **** 2021-02-10T11:08:55.273 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** 2021-02-10T11:08:55.273 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:55.309 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:55 +0000 (0:00:00.042) 0:06:12.597 **** 2021-02-10T11:08:55.315 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* 2021-02-10T11:08:55.316 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:55.351 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:55 +0000 (0:00:00.041) 0:06:12.639 **** 2021-02-10T11:08:55.356 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** 2021-02-10T11:08:55.358 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:55.391 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:55 +0000 (0:00:00.040) 0:06:12.680 **** 2021-02-10T11:08:55.397 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** 2021-02-10T11:08:55.398 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:55.432 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:55 +0000 (0:00:00.041) 0:06:12.721 **** 2021-02-10T11:08:55.439 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** 2021-02-10T11:08:55.439 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:55.474 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:55 +0000 (0:00:00.041) 0:06:12.762 **** 2021-02-10T11:08:55.480 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* 2021-02-10T11:08:55.482 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:55.515 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:55 +0000 (0:00:00.041) 0:06:12.803 **** 2021-02-10T11:08:55.521 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** 2021-02-10T11:08:55.521 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:55.556 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:55 +0000 (0:00:00.040) 0:06:12.844 **** 2021-02-10T11:08:55.563 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** 2021-02-10T11:08:55.564 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:55.598 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:55 +0000 (0:00:00.041) 0:06:12.886 **** 2021-02-10T11:08:55.604 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* 2021-02-10T11:08:55.606 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:55.642 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:55 +0000 (0:00:00.043) 0:06:12.930 **** 2021-02-10T11:08:55.669 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** 2021-02-10T11:08:55.670 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:55.703 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:55 +0000 (0:00:00.060) 0:06:12.991 **** 2021-02-10T11:08:55.730 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:55.762 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:55 +0000 (0:00:00.059) 0:06:13.050 **** 2021-02-10T11:08:55.790 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** 2021-02-10T11:08:55.791 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:55.823 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:55 +0000 (0:00:00.060) 0:06:13.111 **** 2021-02-10T11:08:56.432 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** ok: [gibba020.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["ntp-4.2.6p5-29.el7_8.2.x86_64 providing ntp is already installed"]} 2021-02-10T11:08:56.469 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:56 +0000 (0:00:00.646) 0:06:13.757 **** 2021-02-10T11:08:56.511 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2021-02-10T11:08:56.512 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:56.545 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:56 +0000 (0:00:00.075) 0:06:13.833 **** 2021-02-10T11:08:56.572 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2021-02-10T11:08:56.573 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:56.606 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:56 +0000 (0:00:00.061) 0:06:13.894 **** 2021-02-10T11:08:56.633 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** 2021-02-10T11:08:56.634 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["ntp-4.2.6p5-29.el7_8.2.x86_64 providing ntp is already installed"]} 2021-02-10T11:08:56.668 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:56 +0000 (0:00:00.062) 0:06:13.957 **** 2021-02-10T11:08:56.695 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2021-02-10T11:08:56.698 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:56.731 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:56 +0000 (0:00:00.062) 0:06:14.019 **** 2021-02-10T11:08:56.758 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2021-02-10T11:08:56.760 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:56.792 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:56 +0000 (0:00:00.060) 0:06:14.080 **** 2021-02-10T11:08:58.534 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ 2021-02-10T11:08:58.536 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => {"changed": false, "checksum": "645fd5116197cd9da73b328bc11bf633a40d80e1", "dest": "/etc/ntp.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/ntp.conf", "secontext": "system_u:object_r:net_conf_t:s0", "size": 2767, "state": "file", "uid": 0} 2021-02-10T11:08:58.574 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:58 +0000 (0:00:01.782) 0:06:15.863 **** 2021-02-10T11:08:58.603 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:58.637 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:58 +0000 (0:00:00.062) 0:06:15.925 **** 2021-02-10T11:08:58.717 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ 2021-02-10T11:08:58.719 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => {"changed": false, "checksum": "645fd5116197cd9da73b328bc11bf633a40d80e1", "dest": "/etc/ntp.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/ntp.conf", "secontext": "system_u:object_r:net_conf_t:s0", "size": 2767, "state": "file", "uid": 0} 2021-02-10T11:08:58.757 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:58 +0000 (0:00:00.120) 0:06:16.045 **** 2021-02-10T11:08:58.785 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** 2021-02-10T11:08:58.786 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:58.818 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:58 +0000 (0:00:00.061) 0:06:16.106 **** 2021-02-10T11:08:59.174 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2021-02-10T11:08:59.175 INFO:teuthology.task.ansible.out:changed: [gibba020.front.sepia.ceph.com] => {"changed": true, "enabled": true, "name": "ntpd", "state": "started", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "system.slice -.mount syslog.target tmp.mount ntpdate.service basic.target systemd-journald.socket sntp.service", "AllowIsolate": "no", "AmbientCapabilities": "0", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "shutdown.target multi-user.target chronyd.service", "BlockIOAccounting": "no", "BlockIOWeight": "18446744073709551615", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "18446744073709551615", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "18446744073709551615", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "ConflictedBy": "chronyd.service", "Conflicts": "shutdown.target", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Network Time Service", "DevicePolicy": "auto", "EnvironmentFile": "/etc/sysconfig/ntpd (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/sbin/ntpd ; argv[]=/usr/sbin/ntpd -u ntp:ntp $OPTIONS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/ntpd.service", "GuessMainPID": "yes", "IOScheduling": "0", "Id": "ntpd.service", "IgnoreOnIsolate": "no", "IgnoreOnSnapshot": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobTimeoutAction": "none", "JobTimeoutUSec": "0", "KillMode": "control-group", "KillSignal": "15", "Limi 2021-02-10T11:08:59.176 INFO:teuthology.task.ansible.out:tAS": "18446744073709551615", "LimitCORE": "18446744073709551615", "LimitCPU": "18446744073709551615", "LimitDATA": "18446744073709551615", "LimitFSIZE": "18446744073709551615", "LimitLOCKS": "18446744073709551615", "LimitMEMLOCK": "65536", "LimitMSGQUEUE": "819200", "LimitNICE": "0", "LimitNOFILE": "4096", "LimitNPROC": "127560", "LimitRSS": "18446744073709551615", "LimitRTPRIO": "0", "LimitRTTIME": "18446744073709551615", "LimitSIGPENDING": "127560", "LimitSTACK": "18446744073709551615", "LoadState": "loaded", "MainPID": "0", "MemoryAccounting": "no", "MemoryCurrent": "18446744073709551615", "MemoryLimit": "18446744073709551615", "MountFlags": "0", "Names": "ntpd.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "ProtectHome": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "Requires": "basic.target system.slice -.mount", "RequiresMountsFor": "/var/tmp", "Restart": "no", "RestartUSec": "100ms", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitInterval": "10000000", "StartupBlockIOWeight": "18446744073709551615", "StartupCPUShares": "18446744073709551615", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "no", "TasksCurrent": "18446744073709551615", "TasksMax": "18446744073709551615", "TimeoutStartUSec": "1min 30s", "Timeou 2021-02-10T11:08:59.177 INFO:teuthology.task.ansible.out:tStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "enabled", "WantedBy": "multi-user.target", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2021-02-10T11:08:59.212 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:59 +0000 (0:00:00.393) 0:06:16.500 **** 2021-02-10T11:08:59.239 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-10T11:08:59.241 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2021-02-10T11:08:59.289 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:59 +0000 (0:00:00.076) 0:06:16.577 **** 2021-02-10T11:08:59.319 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-10T11:08:59.320 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:08:59.354 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:59 +0000 (0:00:00.064) 0:06:16.642 **** 2021-02-10T11:08:59.369 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2021-02-10T11:08:59.370 INFO:teuthology.task.ansible.out:changed: [gibba012.front.sepia.ceph.com] => {"changed": true, "enabled": true, "name": "ntpd", "state": "started", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "-.mount ntpdate.service tmp.mount systemd-journald.socket basic.target sntp.service system.slice syslog.target", "AllowIsolate": "no", "AmbientCapabilities": "0", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "shutdown.target multi-user.target chronyd.service", "BlockIOAccounting": "no", "BlockIOWeight": "18446744073709551615", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "18446744073709551615", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "18446744073709551615", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "ConflictedBy": "chronyd.service", "Conflicts": "shutdown.target", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Network Time Service", "DevicePolicy": "auto", "EnvironmentFile": "/etc/sysconfig/ntpd (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/sbin/ntpd ; argv[]=/usr/sbin/ntpd -u ntp:ntp $OPTIONS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/ntpd.service", "GuessMainPID": "yes", "IOScheduling": "0", "Id": "ntpd.service", "IgnoreOnIsolate": "no", "IgnoreOnSnapshot": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobTimeoutAction": "none", "JobTimeoutUSec": "0", "KillMode": "control-group", "KillSignal": "15", "Limi 2021-02-10T11:08:59.372 INFO:teuthology.task.ansible.out:tAS": "18446744073709551615", "LimitCORE": "18446744073709551615", "LimitCPU": "18446744073709551615", "LimitDATA": "18446744073709551615", "LimitFSIZE": "18446744073709551615", "LimitLOCKS": "18446744073709551615", "LimitMEMLOCK": "65536", "LimitMSGQUEUE": "819200", "LimitNICE": "0", "LimitNOFILE": "4096", "LimitNPROC": "127560", "LimitRSS": "18446744073709551615", "LimitRTPRIO": "0", "LimitRTTIME": "18446744073709551615", "LimitSIGPENDING": "127560", "LimitSTACK": "18446744073709551615", "LoadState": "loaded", "MainPID": "0", "MemoryAccounting": "no", "MemoryCurrent": "18446744073709551615", "MemoryLimit": "18446744073709551615", "MountFlags": "0", "Names": "ntpd.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "ProtectHome": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "Requires": "system.slice basic.target -.mount", "RequiresMountsFor": "/var/tmp", "Restart": "no", "RestartUSec": "100ms", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitInterval": "10000000", "StartupBlockIOWeight": "18446744073709551615", "StartupCPUShares": "18446744073709551615", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "no", "TasksCurrent": "18446744073709551615", "TasksMax": "18446744073709551615", "TimeoutStartUSec": "1min 30s", "Timeou 2021-02-10T11:08:59.373 INFO:teuthology.task.ansible.out:tStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "enabled", "WantedBy": "multi-user.target", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2021-02-10T11:08:59.407 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:59 +0000 (0:00:00.053) 0:06:16.695 **** 2021-02-10T11:08:59.434 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-10T11:08:59.435 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2021-02-10T11:08:59.469 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:08:59 +0000 (0:00:00.062) 0:06:16.758 **** 2021-02-10T11:08:59.587 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Wednesday 10 February 2021 11:08:59 +0000 (0:00:00.064) 0:06:16.822 **** 2021-02-10T11:09:00.511 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2021-02-10T11:09:00.513 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["python2-pip-8.1.2-14.el7.noarch providing python-pip is already installed"]} 2021-02-10T11:09:00.547 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:09:00 +0000 (0:00:01.013) 0:06:17.835 **** 2021-02-10T11:09:00.574 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* 2021-02-10T11:09:00.575 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:09:00.609 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:09:00 +0000 (0:00:00.061) 0:06:17.897 **** 2021-02-10T11:09:00.636 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** 2021-02-10T11:09:00.638 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:09:00.676 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:09:00 +0000 (0:00:00.065) 0:06:17.963 **** 2021-02-10T11:09:00.681 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2021-02-10T11:09:00.681 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["python2-pip-8.1.2-14.el7.noarch providing python-pip is already installed"]} 2021-02-10T11:09:00.718 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:09:00 +0000 (0:00:00.044) 0:06:18.007 **** 2021-02-10T11:09:01.016 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* 2021-02-10T11:09:01.018 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:09:01.049 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:09:01 +0000 (0:00:00.330) 0:06:18.338 **** 2021-02-10T11:09:01.077 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** 2021-02-10T11:09:01.079 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:09:01.115 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:09:01 +0000 (0:00:00.065) 0:06:18.403 **** 2021-02-10T11:09:01.143 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** ok: [gibba020.front.sepia.ceph.com] => {"changed": false, "gid": 1236, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 4096, "state": "directory", "uid": 1000} 2021-02-10T11:09:01.174 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:09:01 +0000 (0:00:00.058) 0:06:18.462 **** 2021-02-10T11:09:01.572 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => {"changed": false, "gid": 1236, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 4096, "state": "directory", "uid": 1000} 2021-02-10T11:09:01.611 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:09:01 +0000 (0:00:00.437) 0:06:18.900 **** 2021-02-10T11:09:03.103 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** ok: [gibba020.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1236, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 73, "state": "file", "uid": 1000} 2021-02-10T11:09:03.140 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:09:03 +0000 (0:00:01.528) 0:06:20.428 **** 2021-02-10T11:09:03.551 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1236, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 73, "state": "file", "uid": 1000} 2021-02-10T11:09:03.586 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:09:03 +0000 (0:00:00.446) 0:06:20.874 **** 2021-02-10T11:09:04.440 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* 2021-02-10T11:09:04.442 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => {"changed": false, "cmd": ["/bin/pip", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "You are using pip version 8.1.2, however version 21.0.1 is available.\nYou should consider upgrading via the 'pip install --upgrade pip' command.\n", "stderr_lines": ["You are using pip version 8.1.2, however version 21.0.1 is available.", "You should consider upgrading via the 'pip install --upgrade pip' command."], "stdout": "Requirement already satisfied (use --upgrade to upgrade): remoto>=0.0.35 in /usr/lib/python2.7/site-packages\nRequirement already satisfied (use --upgrade to upgrade): execnet in /usr/lib/python2.7/site-packages (from remoto>=0.0.35)\nRequirement already satisfied (use --upgrade to upgrade): apipkg>=1.4 in /usr/lib/python2.7/site-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied (use --upgrade to upgrade): remoto>=0.0.35 in /usr/lib/python2.7/site-packages", "Requirement already satisfied (use --upgrade to upgrade): execnet in /usr/lib/python2.7/site-packages (from remoto>=0.0.35)", "Requirement already satisfied (use --upgrade to upgrade): apipkg>=1.4 in /usr/lib/python2.7/site-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2021-02-10T11:09:04.474 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:09:04 +0000 (0:00:00.887) 0:06:21.762 **** 2021-02-10T11:09:04.500 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:09:04.534 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:09:04 +0000 (0:00:00.060) 0:06:21.822 **** 2021-02-10T11:09:04.560 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ 2021-02-10T11:09:04.562 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp1s0f0"}, "changed": false} 2021-02-10T11:09:04.593 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:09:04 +0000 (0:00:00.059) 0:06:21.882 **** 2021-02-10T11:09:04.833 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* 2021-02-10T11:09:04.835 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => {"changed": false, "cmd": ["/bin/pip", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "You are using pip version 8.1.2, however version 21.0.1 is available.\nYou should consider upgrading via the 'pip install --upgrade pip' command.\n", "stderr_lines": ["You are using pip version 8.1.2, however version 21.0.1 is available.", "You should consider upgrading via the 'pip install --upgrade pip' command."], "stdout": "Requirement already satisfied (use --upgrade to upgrade): remoto>=0.0.35 in /usr/lib/python2.7/site-packages\nRequirement already satisfied (use --upgrade to upgrade): execnet in /usr/lib/python2.7/site-packages (from remoto>=0.0.35)\nRequirement already satisfied (use --upgrade to upgrade): apipkg>=1.4 in /usr/lib/python2.7/site-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied (use --upgrade to upgrade): remoto>=0.0.35 in /usr/lib/python2.7/site-packages", "Requirement already satisfied (use --upgrade to upgrade): execnet in /usr/lib/python2.7/site-packages (from remoto>=0.0.35)", "Requirement already satisfied (use --upgrade to upgrade): apipkg>=1.4 in /usr/lib/python2.7/site-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2021-02-10T11:09:04.869 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:09:04 +0000 (0:00:00.276) 0:06:22.158 **** 2021-02-10T11:09:04.896 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* 2021-02-10T11:09:04.898 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:09:04.930 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:09:04 +0000 (0:00:00.060) 0:06:22.218 **** 2021-02-10T11:09:04.957 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ 2021-02-10T11:09:04.958 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp1s0f0"}, "changed": false} 2021-02-10T11:09:04.991 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:09:04 +0000 (0:00:00.060) 0:06:22.278 **** 2021-02-10T11:09:05.038 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** 2021-02-10T11:09:05.040 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2021-02-10T11:09:05.072 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:09:05 +0000 (0:00:00.081) 0:06:22.360 **** 2021-02-10T11:09:05.100 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:09:05.139 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:09:05 +0000 (0:00:00.066) 0:06:22.427 **** 2021-02-10T11:09:05.168 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** 2021-02-10T11:09:05.170 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2021-02-10T11:09:05.202 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:09:05 +0000 (0:00:00.063) 0:06:22.490 **** 2021-02-10T11:09:05.229 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** 2021-02-10T11:09:05.231 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:09:05.263 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:09:05 +0000 (0:00:00.060) 0:06:22.551 **** 2021-02-10T11:09:05.471 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** ok: [gibba012.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2021-02-10T11:09:05.505 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:09:05 +0000 (0:00:00.242) 0:06:22.793 **** 2021-02-10T11:09:05.544 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:09:05.582 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:09:05 +0000 (0:00:00.076) 0:06:22.870 **** 2021-02-10T11:09:05.609 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** ok: [gibba012.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2021-02-10T11:09:05.644 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:09:05 +0000 (0:00:00.062) 0:06:22.933 **** 2021-02-10T11:09:05.682 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** 2021-02-10T11:09:05.683 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:09:05.717 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:09:05 +0000 (0:00:00.072) 0:06:23.005 **** 2021-02-10T11:09:05.724 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** 2021-02-10T11:09:05.724 INFO:teuthology.task.ansible.out:changed: [gibba020.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2021-02-10T11:09:05.764 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:09:05 +0000 (0:00:00.047) 0:06:23.053 **** 2021-02-10T11:09:06.162 INFO:teuthology.task.ansible.out:changed: [gibba012.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2021-02-10T11:09:06.200 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:09:06 +0000 (0:00:00.435) 0:06:23.488 **** 2021-02-10T11:09:06.217 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* 2021-02-10T11:09:06.219 INFO:teuthology.task.ansible.out:changed: [gibba020.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2021-02-10T11:09:06.257 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:09:06 +0000 (0:00:00.056) 0:06:23.545 **** 2021-02-10T11:09:06.284 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* 2021-02-10T11:09:06.285 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:09:06.319 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:09:06 +0000 (0:00:00.062) 0:06:23.608 **** 2021-02-10T11:09:06.358 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2021-02-10T11:09:06.359 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} 2021-02-10T11:09:06.370 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} 2021-02-10T11:09:06.372 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} skipping: [gibba020.front.sepia.ceph.com] => (item=enp1s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp1s0f1", "skip_reason": "Conditional result was False"} 2021-02-10T11:09:06.382 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item=enp1s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp1s0f0", "skip_reason": "Conditional result was False"} 2021-02-10T11:09:06.418 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:09:06 +0000 (0:00:00.098) 0:06:23.706 **** 2021-02-10T11:09:06.456 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** 2021-02-10T11:09:06.457 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-02-10T11:09:06.458 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-02-10T11:09:06.468 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-02-10T11:09:06.469 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp1s0f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp1s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-02-10T11:09:06.480 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp1s0f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp1s0f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-02-10T11:09:06.516 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:09:06 +0000 (0:00:00.097) 0:06:23.804 **** 2021-02-10T11:09:06.542 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:09:06.586 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:09:06 +0000 (0:00:00.070) 0:06:23.874 **** 2021-02-10T11:09:06.614 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** 2021-02-10T11:09:06.615 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:09:06.660 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:09:06 +0000 (0:00:00.073) 0:06:23.948 **** 2021-02-10T11:09:06.667 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* 2021-02-10T11:09:06.667 INFO:teuthology.task.ansible.out:changed: [gibba012.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2021-02-10T11:09:06.710 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:09:06 +0000 (0:00:00.049) 0:06:23.997 **** 2021-02-10T11:09:06.718 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** 2021-02-10T11:09:06.719 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:09:06.757 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:09:06 +0000 (0:00:00.047) 0:06:24.045 **** 2021-02-10T11:09:06.764 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* 2021-02-10T11:09:06.766 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:09:06.802 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:09:06 +0000 (0:00:00.045) 0:06:24.090 **** 2021-02-10T11:09:06.808 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* 2021-02-10T11:09:06.810 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:09:06.845 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:09:06 +0000 (0:00:00.043) 0:06:24.134 **** 2021-02-10T11:09:06.853 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2021-02-10T11:09:06.854 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} skipping: [gibba012.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} skipping: [gibba012.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} 2021-02-10T11:09:06.866 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item=enp1s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp1s0f1", "skip_reason": "Conditional result was False"} 2021-02-10T11:09:06.868 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item=enp1s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp1s0f0", "skip_reason": "Conditional result was False"} 2021-02-10T11:09:06.906 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:09:06 +0000 (0:00:00.060) 0:06:24.194 **** 2021-02-10T11:09:06.914 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2021-02-10T11:09:06.915 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:09:06.955 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:09:06 +0000 (0:00:00.049) 0:06:24.243 **** 2021-02-10T11:09:06.963 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** 2021-02-10T11:09:06.964 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-02-10T11:09:06.965 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-02-10T11:09:06.966 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [gibba012.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp1s0f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp1s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-02-10T11:09:06.977 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp1s0f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp1s0f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-02-10T11:09:06.979 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2021-02-10T11:09:06.980 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:09:07.014 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:09:07 +0000 (0:00:00.058) 0:06:24.302 **** 2021-02-10T11:09:07.051 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:09:07 +0000 (0:00:00.036) 0:06:24.339 **** 2021-02-10T11:09:07.057 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-10T11:09:07.058 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:09:07.096 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:09:07 +0000 (0:00:00.045) 0:06:24.384 **** 2021-02-10T11:09:07.103 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** 2021-02-10T11:09:07.105 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:09:07.139 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:09:07 +0000 (0:00:00.043) 0:06:24.427 **** 2021-02-10T11:09:07.146 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** 2021-02-10T11:09:07.147 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:09:07.182 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:09:07 +0000 (0:00:00.042) 0:06:24.470 **** 2021-02-10T11:09:07.188 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2021-02-10T11:09:07.189 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:09:07.225 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:09:07 +0000 (0:00:00.043) 0:06:24.513 **** 2021-02-10T11:09:07.231 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** 2021-02-10T11:09:07.233 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:09:07.270 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:09:07 +0000 (0:00:00.045) 0:06:24.559 **** 2021-02-10T11:09:07.298 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* 2021-02-10T11:09:07.300 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:09:07.334 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:09:07 +0000 (0:00:00.063) 0:06:24.623 **** 2021-02-10T11:09:07.362 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:09:07.401 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:09:07 +0000 (0:00:00.066) 0:06:24.690 **** 2021-02-10T11:09:07.432 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2021-02-10T11:09:07.433 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:09:07.470 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:09:07 +0000 (0:00:00.069) 0:06:24.759 **** 2021-02-10T11:09:07.508 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** 2021-02-10T11:09:07.509 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:09:07.554 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:09:07 +0000 (0:00:00.083) 0:06:24.842 **** 2021-02-10T11:09:07.582 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2021-02-10T11:09:07.582 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:09:07.622 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:09:07 +0000 (0:00:00.068) 0:06:24.910 **** 2021-02-10T11:09:07.681 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2021-02-10T11:09:07.683 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.005060", "end": "2021-02-10 11:09:07.659908", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2021-02-10 11:09:07.654848", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-10T11:09:08.083 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.004846", "end": "2021-02-10 11:09:08.058155", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2021-02-10 11:09:08.053309", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-10T11:09:08.116 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.002204", "end": "2021-02-10 11:09:08.093646", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2021-02-10 11:09:08.091442", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-10T11:09:08.512 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.002216", "end": "2021-02-10 11:09:08.491721", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2021-02-10 11:09:08.489505", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-10T11:09:08.545 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.002111", "end": "2021-02-10 11:09:08.520293", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2021-02-10 11:09:08.518182", "stderr": "find: ‘/etc/ceph’: No such file or directory", "stderr_lines": ["find: ‘/etc/ceph’: No such file or directory"], "stdout": "", "stdout_lines": []} 2021-02-10T11:09:08.939 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.002095", "end": "2021-02-10 11:09:08.919444", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2021-02-10 11:09:08.917349", "stderr": "find: ‘/etc/ceph’: No such file or directory", "stderr_lines": ["find: ‘/etc/ceph’: No such file or directory"], "stdout": "", "stdout_lines": []} 2021-02-10T11:09:08.983 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.014098", "end": "2021-02-10 11:09:08.958700", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2021-02-10 11:09:08.944602", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-10T11:09:09.021 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:09:09 +0000 (0:00:01.398) 0:06:26.309 **** 2021-02-10T11:09:09.048 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2021-02-10T11:09:09.092 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:09:09 +0000 (0:00:00.070) 0:06:26.380 **** 2021-02-10T11:09:09.402 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2021-02-10T11:09:09.403 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.028006", "end": "2021-02-10 11:09:09.375726", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2021-02-10 11:09:09.347720", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-10T11:09:09.441 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:09:09 +0000 (0:00:00.349) 0:06:26.730 **** 2021-02-10T11:09:09.469 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2021-02-10T11:09:09.502 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:09:09 +0000 (0:00:00.060) 0:06:26.790 **** 2021-02-10T11:09:09.541 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** 2021-02-10T11:09:09.542 INFO:teuthology.task.ansible.out:changed: [gibba020.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_runtime_t:s0", "size": 0, "state": "file", "uid": 0} 2021-02-10T11:09:09.947 INFO:teuthology.task.ansible.out:changed: [gibba012.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_runtime_t:s0", "size": 0, "state": "file", "uid": 0} 2021-02-10T11:09:10.063 INFO:teuthology.task.ansible.out: PLAY [testnodes,senta,vossi,jenkins_slaves] ************************************ 2021-02-10T11:09:10.098 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:09:10 +0000 (0:00:00.595) 0:06:27.386 **** 2021-02-10T11:09:10.130 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:09:10 +0000 (0:00:00.032) 0:06:27.418 **** 2021-02-10T11:09:11.206 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* 2021-02-10T11:09:11.208 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] 2021-02-10T11:09:11.281 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:09:11 +0000 (0:00:01.150) 0:06:28.569 **** 2021-02-10T11:09:11.308 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-02-10T11:09:11.309 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:09:11.343 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:09:11 +0000 (0:00:00.061) 0:06:28.631 **** 2021-02-10T11:09:11.380 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-02-10T11:09:11.381 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => {"ansible_facts": {"package_manager": "yum"}, "changed": false} 2021-02-10T11:09:11.414 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:09:11 +0000 (0:00:00.070) 0:06:28.702 **** 2021-02-10T11:09:11.451 INFO:teuthology.task.ansible.out: TASK [container-host : Including distro specific variables] ******************** 2021-02-10T11:09:11.453 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml) => {"ansible_facts": {"container_packages": ["podman", "podman-docker"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml"} 2021-02-10T11:09:11.486 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:09:11 +0000 (0:00:00.072) 0:06:28.775 **** 2021-02-10T11:09:12.215 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [gibba020.front.sepia.ceph.com] 2021-02-10T11:09:12.291 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:09:12 +0000 (0:00:00.804) 0:06:29.579 **** 2021-02-10T11:09:12.319 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-02-10T11:09:12.320 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:09:12.353 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:09:12 +0000 (0:00:00.062) 0:06:29.642 **** 2021-02-10T11:09:12.383 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-02-10T11:09:12.383 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => {"ansible_facts": {"package_manager": "yum"}, "changed": false} 2021-02-10T11:09:12.417 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:09:12 +0000 (0:00:00.063) 0:06:29.705 **** 2021-02-10T11:09:12.455 INFO:teuthology.task.ansible.out: TASK [container-host : Including distro specific variables] ******************** 2021-02-10T11:09:12.456 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml) => {"ansible_facts": {"container_packages": ["podman", "podman-docker"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml"} 2021-02-10T11:09:12.490 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:09:12 +0000 (0:00:00.073) 0:06:29.778 **** 2021-02-10T11:09:22.267 INFO:teuthology.task.ansible.out: TASK [container-host : Install container packages] ***************************** 2021-02-10T11:09:22.269 INFO:teuthology.task.ansible.out:changed: [gibba012.front.sepia.ceph.com] => {"changed": true, "changes": {"installed": [], "updated": [["podman", "1.6.4-27.el7_9.x86_64 from rhel-7-server-extras-rpms"], ["podman-docker", "1.6.4-27.el7_9.noarch from rhel-7-server-extras-rpms"]]}, "msg": "", "rc": 0, "results": ["Loaded plugins: fastestmirror, langpacks, priorities, product-id, search-\n : disabled-repos, subscription-manager\nLoading mirror speeds from cached hostfile\n * epel: download-cc-rdu01.fedoraproject.org\nResolving Dependencies\n--> Running transaction check\n---> Package podman.x86_64 0:1.6.4-26.el7_9 will be updated\n---> Package podman.x86_64 0:1.6.4-27.el7_9 will be an update\n---> Package podman-docker.noarch 0:1.6.4-26.el7_9 will be updated\n---> Package podman-docker.noarch 0:1.6.4-27.el7_9 will be an update\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nUpdating:\n podman x86_64 1.6.4-27.el7_9 rhel-7-server-extras-rpms 13 M\n podman-docker noarch 1.6.4-27.el7_9 rhel-7-server-extras-rpms 31 k\n\nTransaction Summary\n================================================================================\nUpgrade 2 Packages\n\nTotal download size: 13 M\nDownloading packages:\nDelta RPMs disabled because /usr/bin/applydeltarpm not installed.\n--------------------------------------------------------------------------------\nTotal 68 MB/s | 13 MB 00:00 \nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Updating : podman-1.6.4-27.el7_9.x86_64 1/4 \n Updating : podman-docker-1.6.4-27.el7_9.noarch 2/4 \n Cleanup : podman-docke 2021-02-10T11:09:22.270 INFO:teuthology.task.ansible.out:r-1.6.4-26.el7_9.noarch 3/4 \n Cleanup : podman-1.6.4-26.el7_9.x86_64 4/4 \nLoaded plugins: fastestmirror, langpacks, priorities, product-id, subscription-\n : manager\n Verifying : podman-docker-1.6.4-27.el7_9.noarch 1/4 \n Verifying : podman-1.6.4-27.el7_9.x86_64 2/4 \n Verifying : podman-docker-1.6.4-26.el7_9.noarch 3/4 \n Verifying : podman-1.6.4-26.el7_9.x86_64 4/4 \n\nUpdated:\n podman.x86_64 0:1.6.4-27.el7_9 podman-docker.noarch 0:1.6.4-27.el7_9 \n\nComplete!\n"]} 2021-02-10T11:09:22.303 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:09:22 +0000 (0:00:09.812) 0:06:39.591 **** 2021-02-10T11:09:22.330 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-02-10T11:09:22.332 INFO:teuthology.task.ansible.out:ok: [gibba012.front.sepia.ceph.com] => {"ansible_facts": {"container_service_conf": "/etc/containers/registries.conf"}, "changed": false} 2021-02-10T11:09:22.365 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:09:22 +0000 (0:00:00.062) 0:06:39.654 **** 2021-02-10T11:09:22.392 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-02-10T11:09:22.394 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:09:22.428 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:09:22 +0000 (0:00:00.062) 0:06:39.717 **** 2021-02-10T11:09:22.894 INFO:teuthology.task.ansible.out: TASK [container-host : Install container packages] ***************************** 2021-02-10T11:09:22.895 INFO:teuthology.task.ansible.out:changed: [gibba020.front.sepia.ceph.com] => {"changed": true, "changes": {"installed": [], "updated": [["podman", "1.6.4-27.el7_9.x86_64 from rhel-7-server-extras-rpms"], ["podman-docker", "1.6.4-27.el7_9.noarch from rhel-7-server-extras-rpms"]]}, "msg": "", "rc": 0, "results": ["Loaded plugins: fastestmirror, langpacks, priorities, product-id, search-\n : disabled-repos, subscription-manager\nLoading mirror speeds from cached hostfile\n * epel: download-cc-rdu01.fedoraproject.org\nResolving Dependencies\n--> Running transaction check\n---> Package podman.x86_64 0:1.6.4-26.el7_9 will be updated\n---> Package podman.x86_64 0:1.6.4-27.el7_9 will be an update\n---> Package podman-docker.noarch 0:1.6.4-26.el7_9 will be updated\n---> Package podman-docker.noarch 0:1.6.4-27.el7_9 will be an update\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nUpdating:\n podman x86_64 1.6.4-27.el7_9 rhel-7-server-extras-rpms 13 M\n podman-docker noarch 1.6.4-27.el7_9 rhel-7-server-extras-rpms 31 k\n\nTransaction Summary\n================================================================================\nUpgrade 2 Packages\n\nTotal download size: 13 M\nDownloading packages:\nDelta RPMs disabled because /usr/bin/applydeltarpm not installed.\n--------------------------------------------------------------------------------\nTotal 72 MB/s | 13 MB 00:00 \nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Updating : podman-1.6.4-27.el7_9.x86_64 1/4 \n Updating : podman-docker-1.6.4-27.el7_9.noarch 2/4 \n Cleanup : podman-docke 2021-02-10T11:09:22.897 INFO:teuthology.task.ansible.out:r-1.6.4-26.el7_9.noarch 3/4 \n Cleanup : podman-1.6.4-26.el7_9.x86_64 4/4 \nLoaded plugins: fastestmirror, langpacks, priorities, product-id, subscription-\n : manager\n Verifying : podman-docker-1.6.4-27.el7_9.noarch 1/4 \n Verifying : podman-1.6.4-27.el7_9.x86_64 2/4 \n Verifying : podman-docker-1.6.4-26.el7_9.noarch 3/4 \n Verifying : podman-1.6.4-26.el7_9.x86_64 4/4 \n\nUpdated:\n podman.x86_64 0:1.6.4-27.el7_9 podman-docker.noarch 0:1.6.4-27.el7_9 \n\nComplete!\n"]} TASK [container-host : Create container_mirror_cert_paths] ********************* changed: [gibba012.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "secontext": "unconfined_u:object_r:cert_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-02-10T11:09:22.932 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:09:22 +0000 (0:00:00.503) 0:06:40.220 **** 2021-02-10T11:09:22.960 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-02-10T11:09:22.961 INFO:teuthology.task.ansible.out:ok: [gibba020.front.sepia.ceph.com] => {"ansible_facts": {"container_service_conf": "/etc/containers/registries.conf"}, "changed": false} 2021-02-10T11:09:22.994 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:09:22 +0000 (0:00:00.062) 0:06:40.283 **** 2021-02-10T11:09:23.021 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-02-10T11:09:23.022 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:09:23.056 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:09:23 +0000 (0:00:00.062) 0:06:40.345 **** 2021-02-10T11:09:23.347 INFO:teuthology.task.ansible.out: TASK [container-host : Create container_mirror_cert_paths] ********************* changed: [gibba012.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-02-10T11:09:23.386 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:09:23 +0000 (0:00:00.329) 0:06:40.675 **** 2021-02-10T11:09:23.530 INFO:teuthology.task.ansible.out:changed: [gibba020.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "secontext": "unconfined_u:object_r:cert_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-02-10T11:09:23.972 INFO:teuthology.task.ansible.out:changed: [gibba020.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-02-10T11:09:24.010 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:09:24 +0000 (0:00:00.623) 0:06:41.298 **** 2021-02-10T11:09:26.013 INFO:teuthology.task.ansible.out: TASK [container-host : Copy docker-mirror.front.sepia.ceph.com:5000 self-signed cert] *** 2021-02-10T11:09:26.013 INFO:teuthology.task.ansible.out:changed: [gibba012.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:cert_t:s0", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1612955363.4350183-130200548141386/source", "state": "file", "uid": 0} 2021-02-10T11:09:26.650 INFO:teuthology.task.ansible.out:changed: [gibba020.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:cert_t:s0", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1612955364.0533996-139066643866482/source", "state": "file", "uid": 0} 2021-02-10T11:09:28.618 INFO:teuthology.task.ansible.out:changed: [gibba012.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1612955366.022818-48206870154781/source", "state": "file", "uid": 0} 2021-02-10T11:09:28.653 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:09:28 +0000 (0:00:04.643) 0:06:45.941 **** 2021-02-10T11:09:29.291 INFO:teuthology.task.ansible.out:changed: [gibba020.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1612955366.6554823-13819795341605/source", "state": "file", "uid": 0} 2021-02-10T11:09:29.330 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:09:29 +0000 (0:00:00.676) 0:06:46.618 **** 2021-02-10T11:09:31.283 INFO:teuthology.task.ansible.out: TASK [container-host : Install registries-conf-ctl] **************************** changed: [gibba012.front.sepia.ceph.com] => {"changed": true, "cmd": ["/bin/pip2", "install", "-U", "git+https://github.com/sebastian-philipp/registries-conf-ctl"], "name": ["git+https://github.com/sebastian-philipp/registries-conf-ctl"], "requirements": null, "state": "latest", "stderr": "You are using pip version 8.1.2, however version 21.0.1 is available.\nYou should consider upgrading via the 'pip install --upgrade pip' command.\n", "stderr_lines": ["You are using pip version 8.1.2, however version 21.0.1 is available.", "You should consider upgrading via the 'pip install --upgrade pip' command."], "stdout": "Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl\n Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-quzqgm-build\nCollecting toml (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl\nCollecting docopt (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz\nCollecting typing (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/3b/c0/e44213fcb799eac02881e2485724ba5b0914600bc9df6ed922e364fdc059/typing-3.7.4.3-py2-none-any.whl\nInstalling collected packages: toml, docopt, typing, registries-conf-ctl\n Running setup.py install for docopt: started\n Running setup.py install for docopt: finished with status 'done'\n Running setup.py install for registries-conf-ctl: started\n Running setup.py install for registries-conf-ctl: finished with status 'done'\nSuccessfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2 typing-3.7.4.3\n", "stdout_lines": ["Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl", " Cloning 2021-02-10T11:09:31.284 INFO:teuthology.task.ansible.out:https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-quzqgm-build", "Collecting toml (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl", "Collecting docopt (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packages/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz", "Collecting typing (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packages/3b/c0/e44213fcb799eac02881e2485724ba5b0914600bc9df6ed922e364fdc059/typing-3.7.4.3-py2-none-any.whl", "Installing collected packages: toml, docopt, typing, registries-conf-ctl", " Running setup.py install for docopt: started", " Running setup.py install for docopt: finished with status 'done'", " Running setup.py install for registries-conf-ctl: started", " Running setup.py install for registries-conf-ctl: finished with status 'done'", "Successfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2 typing-3.7.4.3"], "version": null, "virtualenv": null} 2021-02-10T11:09:31.321 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:09:31 +0000 (0:00:01.990) 0:06:48.609 **** 2021-02-10T11:09:31.349 INFO:teuthology.task.ansible.out: TASK [container-host : Check for docker's daemon.json] ************************* 2021-02-10T11:09:31.350 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:09:31.394 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:09:31 +0000 (0:00:00.073) 0:06:48.682 **** 2021-02-10T11:09:31.422 INFO:teuthology.task.ansible.out: TASK [container-host : Create /etc/containers/registries.conf if necessary] **** 2021-02-10T11:09:31.424 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:09:31.459 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:09:31 +0000 (0:00:00.065) 0:06:48.748 **** 2021-02-10T11:09:31.907 INFO:teuthology.task.ansible.out: TASK [container-host : Install registries-conf-ctl] **************************** 2021-02-10T11:09:31.908 INFO:teuthology.task.ansible.out:changed: [gibba020.front.sepia.ceph.com] => {"changed": true, "cmd": ["/bin/pip2", "install", "-U", "git+https://github.com/sebastian-philipp/registries-conf-ctl"], "name": ["git+https://github.com/sebastian-philipp/registries-conf-ctl"], "requirements": null, "state": "latest", "stderr": "You are using pip version 8.1.2, however version 21.0.1 is available.\nYou should consider upgrading via the 'pip install --upgrade pip' command.\n", "stderr_lines": ["You are using pip version 8.1.2, however version 21.0.1 is available.", "You should consider upgrading via the 'pip install --upgrade pip' command."], "stdout": "Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl\n Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-UEPxsi-build\nCollecting toml (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl\nCollecting docopt (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz\nCollecting typing (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/3b/c0/e44213fcb799eac02881e2485724ba5b0914600bc9df6ed922e364fdc059/typing-3.7.4.3-py2-none-any.whl\nInstalling collected packages: toml, docopt, typing, registries-conf-ctl\n Running setup.py install for docopt: started\n Running setup.py install for docopt: finished with status 'done'\n Running setup.py install for registries-conf-ctl: started\n Running setup.py install for registries-conf-ctl: finished with status 'done'\nSuccessfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2 typing-3.7.4.3\n", "stdout_lines": ["Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl", " Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-UEPxsi-build", "Co 2021-02-10T11:09:31.910 INFO:teuthology.task.ansible.out:llecting toml (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl", "Collecting docopt (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packages/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz", "Collecting typing (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packages/3b/c0/e44213fcb799eac02881e2485724ba5b0914600bc9df6ed922e364fdc059/typing-3.7.4.3-py2-none-any.whl", "Installing collected packages: toml, docopt, typing, registries-conf-ctl", " Running setup.py install for docopt: started", " Running setup.py install for docopt: finished with status 'done'", " Running setup.py install for registries-conf-ctl: started", " Running setup.py install for registries-conf-ctl: finished with status 'done'", "Successfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2 typing-3.7.4.3"], "version": null, "virtualenv": null} 2021-02-10T11:09:31.948 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:09:31 +0000 (0:00:00.488) 0:06:49.236 **** 2021-02-10T11:09:31.975 INFO:teuthology.task.ansible.out: TASK [container-host : Check for docker's daemon.json] ************************* 2021-02-10T11:09:31.976 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:09:31.977 INFO:teuthology.task.ansible.out: TASK [container-host : Add local docker.io registry mirror] ******************** changed: [gibba012.front.sepia.ceph.com] => {"changed": true, "cmd": ["registries-conf-ctl", "add-mirror", "docker.io", "docker-mirror.front.sepia.ceph.com:5000"], "delta": "0:00:00.070395", "end": "2021-02-10 11:09:31.966011", "rc": 0, "start": "2021-02-10 11:09:31.895616", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-10T11:09:32.019 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:09:32 +0000 (0:00:00.071) 0:06:49.307 **** 2021-02-10T11:09:32.056 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:09:32 +0000 (0:00:00.037) 0:06:49.345 **** 2021-02-10T11:09:32.063 INFO:teuthology.task.ansible.out: TASK [container-host : Restart docker service] ********************************* 2021-02-10T11:09:32.064 INFO:teuthology.task.ansible.out:skipping: [gibba012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:09:32.103 INFO:teuthology.task.ansible.out: TASK [container-host : Create /etc/containers/registries.conf if necessary] **** 2021-02-10T11:09:32.104 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:09:32.159 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:09:32 +0000 (0:00:00.103) 0:06:49.448 **** 2021-02-10T11:09:32.679 INFO:teuthology.task.ansible.out: TASK [container-host : Add local docker.io registry mirror] ******************** 2021-02-10T11:09:32.680 INFO:teuthology.task.ansible.out:changed: [gibba020.front.sepia.ceph.com] => {"changed": true, "cmd": ["registries-conf-ctl", "add-mirror", "docker.io", "docker-mirror.front.sepia.ceph.com:5000"], "delta": "0:00:00.068945", "end": "2021-02-10 11:09:32.662938", "rc": 0, "start": "2021-02-10 11:09:32.593993", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-10T11:09:32.713 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:09:32 +0000 (0:00:00.553) 0:06:50.002 **** 2021-02-10T11:09:32.740 INFO:teuthology.task.ansible.out: TASK [container-host : Restart docker service] ********************************* 2021-02-10T11:09:32.742 INFO:teuthology.task.ansible.out:skipping: [gibba020.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:09:32.838 INFO:teuthology.task.ansible.out: PLAY [cobbler] ***************************************************************** 2021-02-10T11:09:32.838 INFO:teuthology.task.ansible.out:skipping: no hosts matched 2021-02-10T11:09:32.842 INFO:teuthology.task.ansible.out: PLAY [paddles] ***************************************************************** 2021-02-10T11:09:32.842 INFO:teuthology.task.ansible.out:skipping: no hosts matched 2021-02-10T11:09:32.846 INFO:teuthology.task.ansible.out: 2021-02-10T11:09:32.848 INFO:teuthology.task.ansible.out:PLAY [pulpito] ***************************************************************** skipping: no hosts matched PLAY RECAP ********************************************************************* gibba012.front.sepia.ceph.com : ok=141 changed=45 unreachable=0 failed=0 skipped=169 rescued=0 ignored=4 gibba020.front.sepia.ceph.com : ok=141 changed=45 unreachable=0 failed=0 skipped=169 rescued=0 ignored=4 Wednesday 10 February 2021 11:09:32 +0000 (0:00:00.133) 0:06:50.135 **** =============================================================================== 2021-02-10T11:09:32.849 INFO:teuthology.task.ansible.out:users : Create all admin users with sudo access. ----------------------- 63.02s users : Update authorized_keys using the keys repo --------------------- 60.66s common : Make sure smartmontools is installed -------------------------- 25.84s users : Remove revoked users ------------------------------------------- 22.20s testnode : Install packages -------------------------------------------- 15.04s container-host : Install container packages ----------------------------- 9.81s testnode : Ensure ceph packages are not present. ------------------------ 9.20s testnode : Zap all non-root disks --------------------------------------- 8.10s common : nrpe - Load SELinux policy package ----------------------------- 5.78s common : nrpe - Remove SELinux policy package --------------------------- 5.75s common : Register with subscription-manager. ---------------------------- 4.71s common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/. --- 4.70s container-host : Copy docker-mirror.front.sepia.ceph.com:5000 self-signed cert --- 4.64s common : Install nrpe package and dependencies (RHEL/CentOS) ------------ 4.16s common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/. --- 4.09s common : nrpe - Ensure SELinux policy is up to date --------------------- 3.97s testnode : Configure common additional repos in /etc/yum.repos.d/ ------- 3.77s testnode : Ensure ceph dependency packages are not present. ------------- 3.74s testnode : Install epel packages ---------------------------------------- 3.12s common : Configure epel repos in /etc/yum.repos.d/ ---------------------- 3.09s 2021-02-10T11:09:33.179 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@gibba012.front.sepia.ceph.com'), Remote(name='ubuntu@gibba020.front.sepia.ceph.com')] 2021-02-10T11:09:33.180 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-02-10T11:09:33.181 DEBUG:teuthology.orchestra.connection:{'hostname': 'gibba012.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-02-10T11:09:33.250 DEBUG:teuthology.orchestra.run.gibba012:> true 2021-02-10T11:09:33.740 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-02-10T11:09:33.741 DEBUG:teuthology.orchestra.connection:{'hostname': 'gibba020.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-02-10T11:09:33.806 DEBUG:teuthology.orchestra.run.gibba020:> true 2021-02-10T11:09:34.301 INFO:teuthology.run_tasks:Running task clock... 2021-02-10T11:09:34.312 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2021-02-10T11:09:34.312 INFO:teuthology.orchestra.run:Running command with timeout 360 2021-02-10T11:09:34.314 DEBUG:teuthology.orchestra.run.gibba012:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-02-10T11:09:34.665 INFO:teuthology.orchestra.run.gibba012.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2021-02-10T11:09:41.704 INFO:teuthology.orchestra.run.gibba012.stdout:ntpd: time slew -0.001613s 2021-02-10T11:09:41.715 INFO:teuthology.orchestra.run.gibba012.stderr:Failed to start ntp.service: Unit not found. 2021-02-10T11:09:41.760 INFO:teuthology.orchestra.run.gibba012.stdout: remote refid st t when poll reach delay offset jitter 2021-02-10T11:09:41.760 INFO:teuthology.orchestra.run.gibba012.stdout:============================================================================== 2021-02-10T11:09:41.761 INFO:teuthology.orchestra.run.gibba012.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-02-10T11:09:41.761 INFO:teuthology.orchestra.run.gibba012.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-02-10T11:09:41.761 INFO:teuthology.orchestra.run.gibba012.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-02-10T11:09:41.761 INFO:teuthology.orchestra.run.gibba012.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-02-10T11:09:41.761 INFO:teuthology.orchestra.run:Running command with timeout 360 2021-02-10T11:09:41.762 DEBUG:teuthology.orchestra.run.gibba020:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-02-10T11:09:42.113 INFO:teuthology.orchestra.run.gibba020.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2021-02-10T11:09:49.152 INFO:teuthology.orchestra.run.gibba020.stdout:ntpd: time slew -0.002365s 2021-02-10T11:09:49.164 INFO:teuthology.orchestra.run.gibba020.stderr:Failed to start ntp.service: Unit not found. 2021-02-10T11:09:49.198 INFO:teuthology.orchestra.run.gibba020.stdout: remote refid st t when poll reach delay offset jitter 2021-02-10T11:09:49.198 INFO:teuthology.orchestra.run.gibba020.stdout:============================================================================== 2021-02-10T11:09:49.198 INFO:teuthology.orchestra.run.gibba020.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-02-10T11:09:49.198 INFO:teuthology.orchestra.run.gibba020.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-02-10T11:09:49.198 INFO:teuthology.orchestra.run.gibba020.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-02-10T11:09:49.199 INFO:teuthology.orchestra.run.gibba020.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-02-10T11:09:49.199 INFO:teuthology.run_tasks:Running task install... 2021-02-10T11:09:49.207 DEBUG:teuthology.task.install:project ceph 2021-02-10T11:09:49.207 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'sha1': '962fac13b1a781b8c7d25c1eed50d93813b8f5f0'}} 2021-02-10T11:09:49.207 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell'], 'rpm': ['python3-cephfs']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel']}, 'sha1': '962fac13b1a781b8c7d25c1eed50d93813b8f5f0'} 2021-02-10T11:09:49.207 INFO:teuthology.task.install:Using flavor: basic 2021-02-10T11:09:49.211 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python-ceph', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-cloud', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-ssh', 'ceph-fuse', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python-ceph', 'rbd-fuse']} 2021-02-10T11:09:49.212 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell'], 'rpm': ['python3-cephfs']} 2021-02-10T11:09:49.212 DEBUG:teuthology.orchestra.run.gibba012:> sudo lsb_release -is 2021-02-10T11:09:49.574 INFO:teuthology.orchestra.run.gibba012.stdout:RedHatEnterpriseServer 2021-02-10T11:09:49.574 DEBUG:teuthology.misc:System to be installed: RedHatEnterpriseServer 2021-02-10T11:09:49.575 DEBUG:teuthology.orchestra.run.gibba020:> sudo lsb_release -is 2021-02-10T11:09:49.575 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell'], 'rpm': ['python3-cephfs']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel']}, 'extras': None, 'flavor': 'basic', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '962fac13b1a781b8c7d25c1eed50d93813b8f5f0', 'tag': None, 'wait_for_package': False} 2021-02-10T11:09:49.576 DEBUG:teuthology.orchestra.run.gibba012:> sudo lsb_release -is 2021-02-10T11:09:49.929 INFO:teuthology.orchestra.run.gibba012.stdout:RedHatEnterpriseServer 2021-02-10T11:09:49.930 DEBUG:teuthology.misc:System to be installed: RedHatEnterpriseServer 2021-02-10T11:09:49.930 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F7%2Fx86_64&sha1=962fac13b1a781b8c7d25c1eed50d93813b8f5f0 2021-02-10T11:09:49.935 INFO:teuthology.orchestra.run.gibba020.stdout:RedHatEnterpriseServer 2021-02-10T11:09:49.935 DEBUG:teuthology.misc:System to be installed: RedHatEnterpriseServer 2021-02-10T11:09:49.935 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell'], 'rpm': ['python3-cephfs']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel']}, 'extras': None, 'flavor': 'basic', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '962fac13b1a781b8c7d25c1eed50d93813b8f5f0', 'tag': None, 'wait_for_package': False} 2021-02-10T11:09:49.936 DEBUG:teuthology.orchestra.run.gibba020:> sudo lsb_release -is 2021-02-10T11:09:50.098 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-khiremat-mgr-authid-bp-7-nautilus/962fac13b1a781b8c7d25c1eed50d93813b8f5f0/centos/7/flavors/default/ 2021-02-10T11:09:50.098 INFO:teuthology.task.install.rpm:Package version is 14.2.16-280.g962fac13 2021-02-10T11:09:50.289 INFO:teuthology.orchestra.run.gibba020.stdout:RedHatEnterpriseServer 2021-02-10T11:09:50.290 DEBUG:teuthology.misc:System to be installed: RedHatEnterpriseServer 2021-02-10T11:09:50.290 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F7%2Fx86_64&sha1=962fac13b1a781b8c7d25c1eed50d93813b8f5f0 2021-02-10T11:09:50.345 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-khiremat-mgr-authid-bp-7-nautilus/962fac13b1a781b8c7d25c1eed50d93813b8f5f0/centos/7/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-khiremat-mgr-authid-bp-7-nautilus/962fac13b1a781b8c7d25c1eed50d93813b8f5f0/centos/7/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-khiremat-mgr-authid-bp-7-nautilus/962fac13b1a781b8c7d25c1eed50d93813b8f5f0/centos/7/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2021-02-10T11:09:50.346 DEBUG:teuthology.orchestra.run.gibba012:> set -ex 2021-02-10T11:09:50.346 DEBUG:teuthology.orchestra.run.gibba012:> sudo dd of=/etc/yum.repos.d/ceph.repo 2021-02-10T11:09:50.454 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-khiremat-mgr-authid-bp-7-nautilus/962fac13b1a781b8c7d25c1eed50d93813b8f5f0/centos/7/flavors/default/ 2021-02-10T11:09:50.454 INFO:teuthology.task.install.rpm:Package version is 14.2.16-280.g962fac13 2021-02-10T11:09:50.690 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-cloud, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-ssh, ceph-fuse, libcephfs2, libcephfs-devel, librados2, librbd1, python-ceph, rbd-fuse, python3-cephfs, bison, flex, elfutils-libelf-devel, openssl-devel, bison, flex, elfutils-libelf-devel, openssl-devel, bison, flex, elfutils-libelf-devel, openssl-devel on remote rpm x86_64 2021-02-10T11:09:50.690 DEBUG:teuthology.orchestra.run.gibba012:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/962fac13b1a781b8c7d25c1eed50d93813b8f5f0/;g' /etc/yum.repos.d/ceph.repo ; fi 2021-02-10T11:09:50.707 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-khiremat-mgr-authid-bp-7-nautilus/962fac13b1a781b8c7d25c1eed50d93813b8f5f0/centos/7/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-khiremat-mgr-authid-bp-7-nautilus/962fac13b1a781b8c7d25c1eed50d93813b8f5f0/centos/7/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-khiremat-mgr-authid-bp-7-nautilus/962fac13b1a781b8c7d25c1eed50d93813b8f5f0/centos/7/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2021-02-10T11:09:50.708 DEBUG:teuthology.orchestra.run.gibba020:> set -ex 2021-02-10T11:09:50.708 DEBUG:teuthology.orchestra.run.gibba020:> sudo dd of=/etc/yum.repos.d/ceph.repo 2021-02-10T11:09:51.036 DEBUG:teuthology.orchestra.run.gibba012:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2021-02-10T11:09:51.051 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-cloud, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-ssh, ceph-fuse, libcephfs2, libcephfs-devel, librados2, librbd1, python-ceph, rbd-fuse, python3-cephfs, bison, flex, elfutils-libelf-devel, openssl-devel, bison, flex, elfutils-libelf-devel, openssl-devel, bison, flex, elfutils-libelf-devel, openssl-devel, bison, flex, elfutils-libelf-devel, openssl-devel on remote rpm x86_64 2021-02-10T11:09:51.052 DEBUG:teuthology.orchestra.run.gibba020:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/962fac13b1a781b8c7d25c1eed50d93813b8f5f0/;g' /etc/yum.repos.d/ceph.repo ; fi 2021-02-10T11:09:51.388 DEBUG:teuthology.orchestra.run.gibba012:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2021-02-10T11:09:51.398 DEBUG:teuthology.orchestra.run.gibba020:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2021-02-10T11:09:51.730 INFO:teuthology.orchestra.run.gibba012.stdout:check_obsoletes = 1 2021-02-10T11:09:51.731 DEBUG:teuthology.orchestra.run.gibba012:> sudo yum clean all 2021-02-10T11:09:51.750 DEBUG:teuthology.orchestra.run.gibba020:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2021-02-10T11:09:52.093 INFO:teuthology.orchestra.run.gibba020.stdout:check_obsoletes = 1 2021-02-10T11:09:52.094 DEBUG:teuthology.orchestra.run.gibba020:> sudo yum clean all 2021-02-10T11:09:52.180 INFO:teuthology.orchestra.run.gibba012.stdout:Loaded plugins: fastestmirror, langpacks, priorities, product-id, search- 2021-02-10T11:09:52.180 INFO:teuthology.orchestra.run.gibba012.stdout: : disabled-repos, subscription-manager 2021-02-10T11:09:52.539 INFO:teuthology.orchestra.run.gibba012.stdout:Cleaning repos: ceph ceph-noarch ceph-source epel lab-extras rhel-7-fcgi-ceph 2021-02-10T11:09:52.539 INFO:teuthology.orchestra.run.gibba012.stdout: : rhel-7-server-extras-rpms rhel-7-server-optional-rpms 2021-02-10T11:09:52.540 INFO:teuthology.orchestra.run.gibba012.stdout: : rhel-7-server-rpms rhel-ha-for-rhel-7-server-rpms 2021-02-10T11:09:52.541 INFO:teuthology.orchestra.run.gibba020.stdout:Loaded plugins: fastestmirror, langpacks, priorities, product-id, search- 2021-02-10T11:09:52.542 INFO:teuthology.orchestra.run.gibba020.stdout: : disabled-repos, subscription-manager 2021-02-10T11:09:52.703 INFO:teuthology.orchestra.run.gibba012.stdout:Cleaning up list of fastest mirrors 2021-02-10T11:09:52.715 DEBUG:teuthology.orchestra.run.gibba012:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-cloud ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-ssh ceph-fuse libcephfs2 libcephfs-devel librados2 librbd1 python-ceph rbd-fuse python3-cephfs bison flex elfutils-libelf-devel openssl-devel bison flex elfutils-libelf-devel openssl-devel bison flex elfutils-libelf-devel openssl-devel bison flex elfutils-libelf-devel openssl-devel 2021-02-10T11:09:52.802 INFO:teuthology.orchestra.run.gibba020.stdout:Cleaning repos: ceph ceph-noarch ceph-source epel lab-extras rhel-7-fcgi-ceph 2021-02-10T11:09:52.802 INFO:teuthology.orchestra.run.gibba020.stdout: : rhel-7-server-extras-rpms rhel-7-server-optional-rpms 2021-02-10T11:09:52.802 INFO:teuthology.orchestra.run.gibba020.stdout: : rhel-7-server-rpms rhel-ha-for-rhel-7-server-rpms 2021-02-10T11:09:52.953 INFO:teuthology.orchestra.run.gibba020.stdout:Cleaning up list of fastest mirrors 2021-02-10T11:09:52.965 DEBUG:teuthology.orchestra.run.gibba020:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-cloud ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-ssh ceph-fuse libcephfs2 libcephfs-devel librados2 librbd1 python-ceph rbd-fuse python3-cephfs bison flex elfutils-libelf-devel openssl-devel bison flex elfutils-libelf-devel openssl-devel bison flex elfutils-libelf-devel openssl-devel bison flex elfutils-libelf-devel openssl-devel 2021-02-10T11:09:53.171 INFO:teuthology.orchestra.run.gibba012.stdout:Loaded plugins: fastestmirror, langpacks, priorities, product-id, search- 2021-02-10T11:09:53.171 INFO:teuthology.orchestra.run.gibba012.stdout: : disabled-repos, subscription-manager 2021-02-10T11:09:53.441 INFO:teuthology.orchestra.run.gibba012.stdout:Determining fastest mirrors 2021-02-10T11:09:53.452 INFO:teuthology.orchestra.run.gibba020.stdout:Loaded plugins: fastestmirror, langpacks, priorities, product-id, search- 2021-02-10T11:09:53.452 INFO:teuthology.orchestra.run.gibba020.stdout: : disabled-repos, subscription-manager 2021-02-10T11:09:53.722 INFO:teuthology.orchestra.run.gibba020.stdout:Determining fastest mirrors 2021-02-10T11:09:56.446 INFO:teuthology.orchestra.run.gibba012.stdout: * epel: download-cc-rdu01.fedoraproject.org 2021-02-10T11:09:56.728 INFO:teuthology.orchestra.run.gibba020.stdout: * epel: download-cc-rdu01.fedoraproject.org 2021-02-10T11:10:22.478 INFO:teuthology.orchestra.run.gibba012.stdout:29 packages excluded due to repository priority protections 2021-02-10T11:10:26.790 INFO:teuthology.orchestra.run.gibba020.stdout:29 packages excluded due to repository priority protections 2021-02-10T11:10:39.278 INFO:teuthology.orchestra.run.gibba012.stdout:Resolving Dependencies 2021-02-10T11:10:39.278 INFO:teuthology.orchestra.run.gibba012.stdout:--> Running transaction check 2021-02-10T11:10:39.279 INFO:teuthology.orchestra.run.gibba012.stdout:---> Package bison.x86_64 0:3.0.4-2.el7 will be installed 2021-02-10T11:10:39.284 INFO:teuthology.orchestra.run.gibba012.stdout:---> Package ceph.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:10:39.285 INFO:teuthology.orchestra.run.gibba012.stdout:--> Processing Dependency: ceph-mon = 2:14.2.16-280.g962fac13.el7 for package: 2:ceph-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:10:39.286 INFO:teuthology.orchestra.run.gibba012.stdout:--> Processing Dependency: ceph-osd = 2:14.2.16-280.g962fac13.el7 for package: 2:ceph-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:10:39.287 INFO:teuthology.orchestra.run.gibba012.stdout:--> Processing Dependency: ceph-mds = 2:14.2.16-280.g962fac13.el7 for package: 2:ceph-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:10:39.288 INFO:teuthology.orchestra.run.gibba012.stdout:---> Package ceph-fuse.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:10:39.295 INFO:teuthology.orchestra.run.gibba012.stdout:---> Package ceph-mgr.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:10:39.297 INFO:teuthology.orchestra.run.gibba012.stdout:--> Processing Dependency: ceph-base = 2:14.2.16-280.g962fac13.el7 for package: 2:ceph-mgr-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:10:39.299 INFO:teuthology.orchestra.run.gibba012.stdout:--> Processing Dependency: python-requests for package: 2:ceph-mgr-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:10:39.302 INFO:teuthology.orchestra.run.gibba012.stdout:--> Processing Dependency: python-pecan for package: 2:ceph-mgr-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:10:39.302 INFO:teuthology.orchestra.run.gibba012.stdout:--> Processing Dependency: python-cherrypy for package: 2:ceph-mgr-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:10:39.303 INFO:teuthology.orchestra.run.gibba012.stdout:--> Processing Dependency: python-werkzeug for package: 2:ceph-mgr-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:10:39.304 INFO:teuthology.orchestra.run.gibba012.stdout:--> Processing Dependency: python-bcrypt for package: 2:ceph-mgr-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:10:39.305 INFO:teuthology.orchestra.run.gibba012.stdout:---> Package ceph-mgr-dashboard.noarch 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:10:39.305 INFO:teuthology.orchestra.run.gibba012.stdout:--> Processing Dependency: ceph-grafana-dashboards = 2:14.2.16-280.g962fac13.el7 for package: 2:ceph-mgr-dashboard-14.2.16-280.g962fac13.el7.noarch 2021-02-10T11:10:39.306 INFO:teuthology.orchestra.run.gibba012.stdout:--> Processing Dependency: python-routes for package: 2:ceph-mgr-dashboard-14.2.16-280.g962fac13.el7.noarch 2021-02-10T11:10:39.307 INFO:teuthology.orchestra.run.gibba012.stdout:---> Package ceph-mgr-diskprediction-cloud.noarch 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:10:39.307 INFO:teuthology.orchestra.run.gibba012.stdout:---> Package ceph-mgr-diskprediction-local.noarch 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:10:39.308 INFO:teuthology.orchestra.run.gibba012.stdout:--> Processing Dependency: scipy for package: 2:ceph-mgr-diskprediction-local-14.2.16-280.g962fac13.el7.noarch 2021-02-10T11:10:39.309 INFO:teuthology.orchestra.run.gibba012.stdout:---> Package ceph-mgr-rook.noarch 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:10:39.309 INFO:teuthology.orchestra.run.gibba012.stdout:--> Processing Dependency: python-kubernetes for package: 2:ceph-mgr-rook-14.2.16-280.g962fac13.el7.noarch 2021-02-10T11:10:39.310 INFO:teuthology.orchestra.run.gibba012.stdout:---> Package ceph-mgr-ssh.noarch 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:10:39.310 INFO:teuthology.orchestra.run.gibba012.stdout:--> Processing Dependency: python-remoto for package: 2:ceph-mgr-ssh-14.2.16-280.g962fac13.el7.noarch 2021-02-10T11:10:39.311 INFO:teuthology.orchestra.run.gibba012.stdout:---> Package ceph-radosgw.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:10:39.314 INFO:teuthology.orchestra.run.gibba012.stdout:--> Processing Dependency: ceph-selinux = 2:14.2.16-280.g962fac13.el7 for package: 2:ceph-radosgw-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:10:39.314 INFO:teuthology.orchestra.run.gibba012.stdout:--> Processing Dependency: librgw2 = 2:14.2.16-280.g962fac13.el7 for package: 2:ceph-radosgw-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:10:39.315 INFO:teuthology.orchestra.run.gibba012.stdout:--> Processing Dependency: librabbitmq.so.4()(64bit) for package: 2:ceph-radosgw-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:10:39.317 INFO:teuthology.orchestra.run.gibba012.stdout:--> Processing Dependency: liboath.so.0()(64bit) for package: 2:ceph-radosgw-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:10:39.318 INFO:teuthology.orchestra.run.gibba012.stdout:---> Package ceph-test.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:10:39.321 INFO:teuthology.orchestra.run.gibba012.stdout:--> Processing Dependency: ceph-common = 2:14.2.16-280.g962fac13.el7 for package: 2:ceph-test-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:10:39.322 INFO:teuthology.orchestra.run.gibba012.stdout:--> Processing Dependency: jq for package: 2:ceph-test-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:10:39.323 INFO:teuthology.orchestra.run.gibba012.stdout:--> Processing Dependency: socat for package: 2:ceph-test-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:10:39.324 INFO:teuthology.orchestra.run.gibba012.stdout:--> Processing Dependency: libradosstriper.so.1()(64bit) for package: 2:ceph-test-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:10:39.325 INFO:teuthology.orchestra.run.gibba012.stdout:---> Package elfutils-libelf-devel.x86_64 0:0.176-5.el7 will be installed 2021-02-10T11:10:39.326 INFO:teuthology.orchestra.run.gibba012.stdout:--> Processing Dependency: pkgconfig(zlib) for package: elfutils-libelf-devel-0.176-5.el7.x86_64 2021-02-10T11:10:39.331 INFO:teuthology.orchestra.run.gibba012.stdout:---> Package flex.x86_64 0:2.5.37-6.el7 will be installed 2021-02-10T11:10:39.331 INFO:teuthology.orchestra.run.gibba012.stdout:---> Package libcephfs-devel.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:10:39.331 INFO:teuthology.orchestra.run.gibba012.stdout:--> Processing Dependency: librados-devel = 2:14.2.16-280.g962fac13.el7 for package: 2:libcephfs-devel-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:10:39.332 INFO:teuthology.orchestra.run.gibba012.stdout:---> Package libcephfs2.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:10:39.333 INFO:teuthology.orchestra.run.gibba012.stdout:---> Package librados2.x86_64 1:10.2.5-4.el7 will be updated 2021-02-10T11:10:39.333 INFO:teuthology.orchestra.run.gibba012.stdout:---> Package librados2.x86_64 2:14.2.16-280.g962fac13.el7 will be an update 2021-02-10T11:10:39.335 INFO:teuthology.orchestra.run.gibba012.stdout:--> Processing Dependency: liblttng-ust.so.0()(64bit) for package: 2:librados2-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:10:39.336 INFO:teuthology.orchestra.run.gibba012.stdout:---> Package librbd1.x86_64 1:10.2.5-4.el7 will be updated 2021-02-10T11:10:39.336 INFO:teuthology.orchestra.run.gibba012.stdout:---> Package librbd1.x86_64 2:14.2.16-280.g962fac13.el7 will be an update 2021-02-10T11:10:39.336 INFO:teuthology.orchestra.run.gibba012.stdout:---> Package openssl-devel.x86_64 1:1.0.2k-21.el7_9 will be installed 2021-02-10T11:10:39.337 INFO:teuthology.orchestra.run.gibba012.stdout:--> Processing Dependency: openssl-libs(x86-64) = 1:1.0.2k-21.el7_9 for package: 1:openssl-devel-1.0.2k-21.el7_9.x86_64 2021-02-10T11:10:39.353 INFO:teuthology.orchestra.run.gibba012.stdout:--> Processing Dependency: krb5-devel(x86-64) for package: 1:openssl-devel-1.0.2k-21.el7_9.x86_64 2021-02-10T11:10:39.359 INFO:teuthology.orchestra.run.gibba012.stdout:---> Package python-ceph-compat.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:10:39.359 INFO:teuthology.orchestra.run.gibba012.stdout:--> Processing Dependency: python-rados = 2:14.2.16-280.g962fac13.el7 for package: 2:python-ceph-compat-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:10:39.360 INFO:teuthology.orchestra.run.gibba012.stdout:--> Processing Dependency: python-cephfs = 2:14.2.16-280.g962fac13.el7 for package: 2:python-ceph-compat-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:10:39.360 INFO:teuthology.orchestra.run.gibba012.stdout:--> Processing Dependency: python-rgw = 2:14.2.16-280.g962fac13.el7 for package: 2:python-ceph-compat-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:10:39.361 INFO:teuthology.orchestra.run.gibba012.stdout:--> Processing Dependency: python-rbd = 2:14.2.16-280.g962fac13.el7 for package: 2:python-ceph-compat-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:10:39.362 INFO:teuthology.orchestra.run.gibba012.stdout:---> Package python3-cephfs.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:10:39.363 INFO:teuthology.orchestra.run.gibba012.stdout:--> Processing Dependency: python3-ceph-argparse = 2:14.2.16-280.g962fac13.el7 for package: 2:python3-cephfs-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:10:39.364 INFO:teuthology.orchestra.run.gibba012.stdout:--> Processing Dependency: python3-rados = 2:14.2.16-280.g962fac13.el7 for package: 2:python3-cephfs-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:10:39.365 INFO:teuthology.orchestra.run.gibba012.stdout:---> Package rbd-fuse.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:10:39.366 INFO:teuthology.orchestra.run.gibba012.stdout:--> Running transaction check 2021-02-10T11:10:39.366 INFO:teuthology.orchestra.run.gibba012.stdout:---> Package ceph-base.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:10:39.370 INFO:teuthology.orchestra.run.gibba012.stdout:---> Package ceph-common.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:10:39.372 INFO:teuthology.orchestra.run.gibba012.stdout:--> Processing Dependency: python-ceph-argparse = 2:14.2.16-280.g962fac13.el7 for package: 2:ceph-common-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:10:39.373 INFO:teuthology.orchestra.run.gibba012.stdout:--> Processing Dependency: python-prettytable for package: 2:ceph-common-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:10:39.374 INFO:teuthology.orchestra.run.gibba012.stdout:--> Processing Dependency: librdkafka.so.1()(64bit) for package: 2:ceph-common-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:10:39.375 INFO:teuthology.orchestra.run.gibba012.stdout:--> Processing Dependency: libbabeltrace-ctf.so.1()(64bit) for package: 2:ceph-common-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:10:39.376 INFO:teuthology.orchestra.run.gibba012.stdout:--> Processing Dependency: libbabeltrace.so.1()(64bit) for package: 2:ceph-common-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:10:39.376 INFO:teuthology.orchestra.run.gibba012.stdout:---> Package ceph-grafana-dashboards.noarch 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:10:39.376 INFO:teuthology.orchestra.run.gibba012.stdout:---> Package ceph-mds.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:10:39.377 INFO:teuthology.orchestra.run.gibba012.stdout:---> Package ceph-mon.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:10:39.377 INFO:teuthology.orchestra.run.gibba012.stdout:---> Package ceph-osd.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:10:39.379 INFO:teuthology.orchestra.run.gibba012.stdout:---> Package ceph-selinux.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:10:39.381 INFO:teuthology.orchestra.run.gibba012.stdout:---> Package jq.x86_64 0:1.6-2.el7 will be installed 2021-02-10T11:10:39.381 INFO:teuthology.orchestra.run.gibba012.stdout:--> Processing Dependency: libonig.so.5()(64bit) for package: jq-1.6-2.el7.x86_64 2021-02-10T11:10:39.382 INFO:teuthology.orchestra.run.gibba012.stdout:---> Package krb5-devel.x86_64 0:1.15.1-50.el7 will be installed 2021-02-10T11:10:39.384 INFO:teuthology.orchestra.run.gibba012.stdout:--> Processing Dependency: libverto-devel for package: krb5-devel-1.15.1-50.el7.x86_64 2021-02-10T11:10:39.386 INFO:teuthology.orchestra.run.gibba012.stdout:--> Processing Dependency: libselinux-devel for package: krb5-devel-1.15.1-50.el7.x86_64 2021-02-10T11:10:39.391 INFO:teuthology.orchestra.run.gibba012.stdout:--> Processing Dependency: libcom_err-devel for package: krb5-devel-1.15.1-50.el7.x86_64 2021-02-10T11:10:39.402 INFO:teuthology.orchestra.run.gibba012.stdout:--> Processing Dependency: keyutils-libs-devel for package: krb5-devel-1.15.1-50.el7.x86_64 2021-02-10T11:10:39.404 INFO:teuthology.orchestra.run.gibba012.stdout:---> Package liboath.x86_64 0:2.6.2-1.el7 will be installed 2021-02-10T11:10:39.404 INFO:teuthology.orchestra.run.gibba012.stdout:---> Package librabbitmq.x86_64 0:0.8.0-3.el7 will be installed 2021-02-10T11:10:39.404 INFO:teuthology.orchestra.run.gibba012.stdout:---> Package librados-devel.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:10:39.404 INFO:teuthology.orchestra.run.gibba012.stdout:---> Package libradosstriper1.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:10:39.405 INFO:teuthology.orchestra.run.gibba012.stdout:---> Package librgw2.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:10:39.406 INFO:teuthology.orchestra.run.gibba012.stdout:---> Package lttng-ust.x86_64 0:2.4.1-4.el7 will be installed 2021-02-10T11:10:39.406 INFO:teuthology.orchestra.run.gibba012.stdout:--> Processing Dependency: liburcu-bp.so.1()(64bit) for package: lttng-ust-2.4.1-4.el7.x86_64 2021-02-10T11:10:39.407 INFO:teuthology.orchestra.run.gibba012.stdout:--> Processing Dependency: liburcu-cds.so.1()(64bit) for package: lttng-ust-2.4.1-4.el7.x86_64 2021-02-10T11:10:39.407 INFO:teuthology.orchestra.run.gibba012.stdout:---> Package openssl-libs.x86_64 1:1.0.2k-19.el7 will be updated 2021-02-10T11:10:39.408 INFO:teuthology.orchestra.run.gibba012.stdout:--> Processing Dependency: openssl-libs(x86-64) = 1:1.0.2k-19.el7 for package: 1:openssl-1.0.2k-19.el7.x86_64 2021-02-10T11:10:39.409 INFO:teuthology.orchestra.run.gibba012.stdout:---> Package openssl-libs.x86_64 1:1.0.2k-21.el7_9 will be an update 2021-02-10T11:10:39.410 INFO:teuthology.orchestra.run.gibba012.stdout:---> Package python-cephfs.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:10:39.410 INFO:teuthology.orchestra.run.gibba012.stdout:---> Package python-cherrypy.noarch 0:3.2.2-4.el7 will be installed 2021-02-10T11:10:39.426 INFO:teuthology.orchestra.run.gibba012.stdout:---> Package python-pecan.noarch 0:0.4.5-2.el7 will be installed 2021-02-10T11:10:39.426 INFO:teuthology.orchestra.run.gibba012.stdout:--> Processing Dependency: python-mako >= 0.4.0 for package: python-pecan-0.4.5-2.el7.noarch 2021-02-10T11:10:39.427 INFO:teuthology.orchestra.run.gibba012.stdout:--> Processing Dependency: python-simplegeneric >= 0.8 for package: python-pecan-0.4.5-2.el7.noarch 2021-02-10T11:10:39.427 INFO:teuthology.orchestra.run.gibba012.stdout:--> Processing Dependency: python-webob >= 1.2 for package: python-pecan-0.4.5-2.el7.noarch 2021-02-10T11:10:39.429 INFO:teuthology.orchestra.run.gibba012.stdout:--> Processing Dependency: python-webtest >= 1.3.1 for package: python-pecan-0.4.5-2.el7.noarch 2021-02-10T11:10:39.429 INFO:teuthology.orchestra.run.gibba012.stdout:--> Processing Dependency: python-singledispatch for package: python-pecan-0.4.5-2.el7.noarch 2021-02-10T11:10:39.430 INFO:teuthology.orchestra.run.gibba012.stdout:---> Package python-rados.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:10:39.431 INFO:teuthology.orchestra.run.gibba012.stdout:---> Package python-rbd.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:10:39.431 INFO:teuthology.orchestra.run.gibba012.stdout:---> Package python-requests.noarch 0:2.6.0-10.el7 will be installed 2021-02-10T11:10:39.431 INFO:teuthology.orchestra.run.gibba012.stdout:--> Processing Dependency: python-urllib3 >= 1.10.2-1 for package: python-requests-2.6.0-10.el7.noarch 2021-02-10T11:10:39.433 INFO:teuthology.orchestra.run.gibba012.stdout:---> Package python-rgw.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:10:39.433 INFO:teuthology.orchestra.run.gibba012.stdout:---> Package python-routes.noarch 0:1.13-2.el7 will be installed 2021-02-10T11:10:39.433 INFO:teuthology.orchestra.run.gibba012.stdout:--> Processing Dependency: python-repoze-lru for package: python-routes-1.13-2.el7.noarch 2021-02-10T11:10:39.434 INFO:teuthology.orchestra.run.gibba012.stdout:---> Package python-werkzeug.noarch 0:0.9.1-2.el7 will be installed 2021-02-10T11:10:39.434 INFO:teuthology.orchestra.run.gibba012.stdout:---> Package python2-bcrypt.x86_64 0:3.1.6-2.el7 will be installed 2021-02-10T11:10:39.434 INFO:teuthology.orchestra.run.gibba012.stdout:--> Processing Dependency: python2-six for package: python2-bcrypt-3.1.6-2.el7.x86_64 2021-02-10T11:10:39.435 INFO:teuthology.orchestra.run.gibba012.stdout:---> Package python2-kubernetes.noarch 1:11.0.0-6.el7 will be installed 2021-02-10T11:10:39.435 INFO:teuthology.orchestra.run.gibba012.stdout:--> Processing Dependency: python-certifi for package: 1:python2-kubernetes-11.0.0-6.el7.noarch 2021-02-10T11:10:39.436 INFO:teuthology.orchestra.run.gibba012.stdout:--> Processing Dependency: python-google-auth for package: 1:python2-kubernetes-11.0.0-6.el7.noarch 2021-02-10T11:10:39.437 INFO:teuthology.orchestra.run.gibba012.stdout:--> Processing Dependency: python-requests-oauthlib for package: 1:python2-kubernetes-11.0.0-6.el7.noarch 2021-02-10T11:10:39.437 INFO:teuthology.orchestra.run.gibba012.stdout:--> Processing Dependency: python-websocket-client for package: 1:python2-kubernetes-11.0.0-6.el7.noarch 2021-02-10T11:10:39.440 INFO:teuthology.orchestra.run.gibba012.stdout:---> Package python2-remoto.noarch 0:1.0.0-1.el7 will be installed 2021-02-10T11:10:39.441 INFO:teuthology.orchestra.run.gibba012.stdout:--> Processing Dependency: python2-execnet >= 1.2.0 for package: python2-remoto-1.0.0-1.el7.noarch 2021-02-10T11:10:39.442 INFO:teuthology.orchestra.run.gibba012.stdout:---> Package python3-ceph-argparse.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:10:39.442 INFO:teuthology.orchestra.run.gibba012.stdout:---> Package python3-rados.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:10:39.442 INFO:teuthology.orchestra.run.gibba012.stdout:---> Package scipy.x86_64 0:0.12.1-6.el7 will be installed 2021-02-10T11:10:39.444 INFO:teuthology.orchestra.run.gibba012.stdout:--> Processing Dependency: f2py for package: scipy-0.12.1-6.el7.x86_64 2021-02-10T11:10:39.445 INFO:teuthology.orchestra.run.gibba012.stdout:--> Processing Dependency: libumfpack.so.5()(64bit) for package: scipy-0.12.1-6.el7.x86_64 2021-02-10T11:10:39.447 INFO:teuthology.orchestra.run.gibba012.stdout:--> Processing Dependency: libamd.so.2()(64bit) for package: scipy-0.12.1-6.el7.x86_64 2021-02-10T11:10:39.447 INFO:teuthology.orchestra.run.gibba012.stdout:---> Package socat.x86_64 0:1.7.3.2-2.el7 will be installed 2021-02-10T11:10:39.448 INFO:teuthology.orchestra.run.gibba012.stdout:---> Package zlib-devel.x86_64 0:1.2.7-19.el7_9 will be installed 2021-02-10T11:10:39.448 INFO:teuthology.orchestra.run.gibba012.stdout:--> Processing Dependency: zlib = 1.2.7-19.el7_9 for package: zlib-devel-1.2.7-19.el7_9.x86_64 2021-02-10T11:10:39.450 INFO:teuthology.orchestra.run.gibba012.stdout:--> Running transaction check 2021-02-10T11:10:39.451 INFO:teuthology.orchestra.run.gibba012.stdout:---> Package keyutils-libs-devel.x86_64 0:1.5.8-3.el7 will be installed 2021-02-10T11:10:39.451 INFO:teuthology.orchestra.run.gibba012.stdout:---> Package libbabeltrace.x86_64 0:1.2.4-3.el7 will be installed 2021-02-10T11:10:39.452 INFO:teuthology.orchestra.run.gibba012.stdout:---> Package libcom_err-devel.x86_64 0:1.42.9-19.el7 will be installed 2021-02-10T11:10:39.452 INFO:teuthology.orchestra.run.gibba012.stdout:---> Package librdkafka.x86_64 0:0.11.5-1.el7 will be installed 2021-02-10T11:10:39.453 INFO:teuthology.orchestra.run.gibba012.stdout:---> Package libselinux-devel.x86_64 0:2.5-15.el7 will be installed 2021-02-10T11:10:39.453 INFO:teuthology.orchestra.run.gibba012.stdout:--> Processing Dependency: libsepol-devel(x86-64) >= 2.5-10 for package: libselinux-devel-2.5-15.el7.x86_64 2021-02-10T11:10:39.455 INFO:teuthology.orchestra.run.gibba012.stdout:--> Processing Dependency: pkgconfig(libsepol) for package: libselinux-devel-2.5-15.el7.x86_64 2021-02-10T11:10:39.455 INFO:teuthology.orchestra.run.gibba012.stdout:--> Processing Dependency: pkgconfig(libpcre) for package: libselinux-devel-2.5-15.el7.x86_64 2021-02-10T11:10:39.459 INFO:teuthology.orchestra.run.gibba012.stdout:---> Package libverto-devel.x86_64 0:0.2.5-4.el7 will be installed 2021-02-10T11:10:39.460 INFO:teuthology.orchestra.run.gibba012.stdout:---> Package numpy-f2py.x86_64 1:1.7.1-13.el7 will be installed 2021-02-10T11:10:39.460 INFO:teuthology.orchestra.run.gibba012.stdout:---> Package oniguruma.x86_64 0:6.8.2-1.el7 will be installed 2021-02-10T11:10:39.460 INFO:teuthology.orchestra.run.gibba012.stdout:---> Package openssl.x86_64 1:1.0.2k-19.el7 will be updated 2021-02-10T11:10:39.461 INFO:teuthology.orchestra.run.gibba012.stdout:---> Package openssl.x86_64 1:1.0.2k-21.el7_9 will be an update 2021-02-10T11:10:39.462 INFO:teuthology.orchestra.run.gibba012.stdout:---> Package python-ceph-argparse.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:10:39.463 INFO:teuthology.orchestra.run.gibba012.stdout:---> Package python-execnet.noarch 0:1.7.1-3.el7 will be installed 2021-02-10T11:10:39.463 INFO:teuthology.orchestra.run.gibba012.stdout:--> Processing Dependency: python-apipkg for package: python-execnet-1.7.1-3.el7.noarch 2021-02-10T11:10:39.463 INFO:teuthology.orchestra.run.gibba012.stdout:---> Package python-mako.noarch 0:0.8.1-2.el7 will be installed 2021-02-10T11:10:39.463 INFO:teuthology.orchestra.run.gibba012.stdout:--> Processing Dependency: python-markupsafe for package: python-mako-0.8.1-2.el7.noarch 2021-02-10T11:10:39.464 INFO:teuthology.orchestra.run.gibba012.stdout:--> Processing Dependency: python-beaker for package: python-mako-0.8.1-2.el7.noarch 2021-02-10T11:10:39.465 INFO:teuthology.orchestra.run.gibba012.stdout:---> Package python-prettytable.noarch 0:0.7.2-3.el7 will be installed 2021-02-10T11:10:39.465 INFO:teuthology.orchestra.run.gibba012.stdout:---> Package python-repoze-lru.noarch 0:0.4-3.el7 will be installed 2021-02-10T11:10:39.465 INFO:teuthology.orchestra.run.gibba012.stdout:---> Package python-simplegeneric.noarch 0:0.8-7.el7 will be installed 2021-02-10T11:10:39.465 INFO:teuthology.orchestra.run.gibba012.stdout:---> Package python-singledispatch.noarch 0:3.4.0.2-2.el7 will be installed 2021-02-10T11:10:39.465 INFO:teuthology.orchestra.run.gibba012.stdout:---> Package python-urllib3.noarch 0:1.10.2-7.el7 will be installed 2021-02-10T11:10:39.466 INFO:teuthology.orchestra.run.gibba012.stdout:---> Package python-webob.noarch 0:1.2.3-7.el7 will be installed 2021-02-10T11:10:39.466 INFO:teuthology.orchestra.run.gibba012.stdout:---> Package python-websocket-client.noarch 0:0.56.0-3.git3c25814.el7 will be installed 2021-02-10T11:10:39.466 INFO:teuthology.orchestra.run.gibba012.stdout:---> Package python-webtest.noarch 0:1.3.4-6.el7 will be installed 2021-02-10T11:10:39.467 INFO:teuthology.orchestra.run.gibba012.stdout:---> Package python2-certifi.noarch 0:2018.10.15-5.el7 will be installed 2021-02-10T11:10:39.467 INFO:teuthology.orchestra.run.gibba012.stdout:---> Package python2-google-auth.noarch 1:1.1.1-5.el7 will be installed 2021-02-10T11:10:39.467 INFO:teuthology.orchestra.run.gibba012.stdout:--> Processing Dependency: python-cachetools for package: 1:python2-google-auth-1.1.1-5.el7.noarch 2021-02-10T11:10:39.467 INFO:teuthology.orchestra.run.gibba012.stdout:--> Processing Dependency: python2-pyasn1-modules for package: 1:python2-google-auth-1.1.1-5.el7.noarch 2021-02-10T11:10:39.468 INFO:teuthology.orchestra.run.gibba012.stdout:--> Processing Dependency: python2-rsa for package: 1:python2-google-auth-1.1.1-5.el7.noarch 2021-02-10T11:10:39.469 INFO:teuthology.orchestra.run.gibba012.stdout:---> Package python2-requests-oauthlib.noarch 0:0.8.0-5.el7 will be installed 2021-02-10T11:10:39.469 INFO:teuthology.orchestra.run.gibba012.stdout:--> Processing Dependency: python2-oauthlib for package: python2-requests-oauthlib-0.8.0-5.el7.noarch 2021-02-10T11:10:39.469 INFO:teuthology.orchestra.run.gibba012.stdout:---> Package python2-six.noarch 0:1.9.0-0.el7 will be installed 2021-02-10T11:10:39.470 INFO:teuthology.orchestra.run.gibba012.stdout:---> Package suitesparse.x86_64 0:4.0.2-10.el7 will be installed 2021-02-10T11:10:39.470 INFO:teuthology.orchestra.run.gibba012.stdout:--> Processing Dependency: libtbbmalloc.so.2()(64bit) for package: suitesparse-4.0.2-10.el7.x86_64 2021-02-10T11:10:39.472 INFO:teuthology.orchestra.run.gibba012.stdout:--> Processing Dependency: libtbb.so.2()(64bit) for package: suitesparse-4.0.2-10.el7.x86_64 2021-02-10T11:10:39.472 INFO:teuthology.orchestra.run.gibba012.stdout:---> Package userspace-rcu.x86_64 0:0.7.16-1.el7 will be installed 2021-02-10T11:10:39.472 INFO:teuthology.orchestra.run.gibba012.stdout:---> Package zlib.x86_64 0:1.2.7-18.el7 will be updated 2021-02-10T11:10:39.474 INFO:teuthology.orchestra.run.gibba012.stdout:---> Package zlib.x86_64 0:1.2.7-19.el7_9 will be an update 2021-02-10T11:10:39.474 INFO:teuthology.orchestra.run.gibba012.stdout:--> Running transaction check 2021-02-10T11:10:39.474 INFO:teuthology.orchestra.run.gibba012.stdout:---> Package libsepol-devel.x86_64 0:2.5-10.el7 will be installed 2021-02-10T11:10:39.474 INFO:teuthology.orchestra.run.gibba012.stdout:---> Package pcre-devel.x86_64 0:8.32-17.el7 will be installed 2021-02-10T11:10:39.475 INFO:teuthology.orchestra.run.gibba012.stdout:---> Package python-beaker.noarch 0:1.5.4-10.el7 will be installed 2021-02-10T11:10:39.475 INFO:teuthology.orchestra.run.gibba012.stdout:--> Processing Dependency: python-paste for package: python-beaker-1.5.4-10.el7.noarch 2021-02-10T11:10:39.476 INFO:teuthology.orchestra.run.gibba012.stdout:---> Package python-cachetools.noarch 0:1.0.3-1.el7 will be installed 2021-02-10T11:10:39.476 INFO:teuthology.orchestra.run.gibba012.stdout:---> Package python-markupsafe.x86_64 0:0.11-10.el7 will be installed 2021-02-10T11:10:39.476 INFO:teuthology.orchestra.run.gibba012.stdout:---> Package python2-apipkg.noarch 0:1.5-5.el7 will be installed 2021-02-10T11:10:39.476 INFO:teuthology.orchestra.run.gibba012.stdout:---> Package python2-oauthlib.noarch 0:2.0.1-8.el7 will be installed 2021-02-10T11:10:39.476 INFO:teuthology.orchestra.run.gibba012.stdout:--> Processing Dependency: python-jwcrypto for package: python2-oauthlib-2.0.1-8.el7.noarch 2021-02-10T11:10:39.477 INFO:teuthology.orchestra.run.gibba012.stdout:---> Package python2-pyasn1-modules.noarch 0:0.1.9-7.el7 will be installed 2021-02-10T11:10:39.477 INFO:teuthology.orchestra.run.gibba012.stdout:---> Package python2-rsa.noarch 0:3.4.2-2.el7 will be installed 2021-02-10T11:10:39.477 INFO:teuthology.orchestra.run.gibba012.stdout:---> Package tbb.x86_64 0:4.1-9.20130314.el7 will be installed 2021-02-10T11:10:39.478 INFO:teuthology.orchestra.run.gibba012.stdout:--> Running transaction check 2021-02-10T11:10:39.478 INFO:teuthology.orchestra.run.gibba012.stdout:---> Package python-jwcrypto.noarch 0:0.4.2-1.el7 will be installed 2021-02-10T11:10:39.478 INFO:teuthology.orchestra.run.gibba012.stdout:---> Package python-paste.noarch 0:1.7.5.1-9.20111221hg1498.el7 will be installed 2021-02-10T11:10:39.478 INFO:teuthology.orchestra.run.gibba012.stdout:--> Processing Dependency: python-tempita for package: python-paste-1.7.5.1-9.20111221hg1498.el7.noarch 2021-02-10T11:10:39.479 INFO:teuthology.orchestra.run.gibba012.stdout:--> Running transaction check 2021-02-10T11:10:39.479 INFO:teuthology.orchestra.run.gibba012.stdout:---> Package python-tempita.noarch 0:0.5.1-6.el7 will be installed 2021-02-10T11:10:40.043 INFO:teuthology.orchestra.run.gibba012.stdout:--> Finished Dependency Resolution 2021-02-10T11:10:40.098 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:10:40.099 INFO:teuthology.orchestra.run.gibba012.stdout:Dependencies Resolved 2021-02-10T11:10:40.107 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:10:40.108 INFO:teuthology.orchestra.run.gibba012.stdout:================================================================================ 2021-02-10T11:10:40.108 INFO:teuthology.orchestra.run.gibba012.stdout: Package Arch Version Repository Size 2021-02-10T11:10:40.108 INFO:teuthology.orchestra.run.gibba012.stdout:================================================================================ 2021-02-10T11:10:40.108 INFO:teuthology.orchestra.run.gibba012.stdout:Installing: 2021-02-10T11:10:40.108 INFO:teuthology.orchestra.run.gibba012.stdout: bison x86_64 3.0.4-2.el7 rhel-7-server-rpms 674 k 2021-02-10T11:10:40.108 INFO:teuthology.orchestra.run.gibba012.stdout: ceph x86_64 2:14.2.16-280.g962fac13.el7 ceph 2.0 k 2021-02-10T11:10:40.108 INFO:teuthology.orchestra.run.gibba012.stdout: ceph-fuse x86_64 2:14.2.16-280.g962fac13.el7 ceph 571 k 2021-02-10T11:10:40.108 INFO:teuthology.orchestra.run.gibba012.stdout: ceph-mgr x86_64 2:14.2.16-280.g962fac13.el7 ceph 1.8 M 2021-02-10T11:10:40.108 INFO:teuthology.orchestra.run.gibba012.stdout: ceph-mgr-dashboard noarch 2:14.2.16-280.g962fac13.el7 ceph-noarch 4.0 M 2021-02-10T11:10:40.108 INFO:teuthology.orchestra.run.gibba012.stdout: ceph-mgr-diskprediction-cloud 2021-02-10T11:10:40.108 INFO:teuthology.orchestra.run.gibba012.stdout: noarch 2:14.2.16-280.g962fac13.el7 ceph-noarch 83 k 2021-02-10T11:10:40.109 INFO:teuthology.orchestra.run.gibba012.stdout: ceph-mgr-diskprediction-local 2021-02-10T11:10:40.109 INFO:teuthology.orchestra.run.gibba012.stdout: noarch 2:14.2.16-280.g962fac13.el7 ceph-noarch 1.1 M 2021-02-10T11:10:40.109 INFO:teuthology.orchestra.run.gibba012.stdout: ceph-mgr-rook noarch 2:14.2.16-280.g962fac13.el7 ceph-noarch 21 k 2021-02-10T11:10:40.109 INFO:teuthology.orchestra.run.gibba012.stdout: ceph-mgr-ssh noarch 2:14.2.16-280.g962fac13.el7 ceph-noarch 21 k 2021-02-10T11:10:40.109 INFO:teuthology.orchestra.run.gibba012.stdout: ceph-radosgw x86_64 2:14.2.16-280.g962fac13.el7 ceph 5.8 M 2021-02-10T11:10:40.109 INFO:teuthology.orchestra.run.gibba012.stdout: ceph-test x86_64 2:14.2.16-280.g962fac13.el7 ceph 34 M 2021-02-10T11:10:40.109 INFO:teuthology.orchestra.run.gibba012.stdout: elfutils-libelf-devel 2021-02-10T11:10:40.109 INFO:teuthology.orchestra.run.gibba012.stdout: x86_64 0.176-5.el7 rhel-7-server-rpms 40 k 2021-02-10T11:10:40.109 INFO:teuthology.orchestra.run.gibba012.stdout: flex x86_64 2.5.37-6.el7 rhel-7-server-rpms 293 k 2021-02-10T11:10:40.109 INFO:teuthology.orchestra.run.gibba012.stdout: libcephfs-devel x86_64 2:14.2.16-280.g962fac13.el7 ceph 17 k 2021-02-10T11:10:40.109 INFO:teuthology.orchestra.run.gibba012.stdout: libcephfs2 x86_64 2:14.2.16-280.g962fac13.el7 ceph 500 k 2021-02-10T11:10:40.110 INFO:teuthology.orchestra.run.gibba012.stdout: openssl-devel x86_64 1:1.0.2k-21.el7_9 rhel-7-server-rpms 1.5 M 2021-02-10T11:10:40.110 INFO:teuthology.orchestra.run.gibba012.stdout: python-ceph-compat x86_64 2:14.2.16-280.g962fac13.el7 ceph 2.2 k 2021-02-10T11:10:40.110 INFO:teuthology.orchestra.run.gibba012.stdout: python3-cephfs x86_64 2:14.2.16-280.g962fac13.el7 ceph 151 k 2021-02-10T11:10:40.110 INFO:teuthology.orchestra.run.gibba012.stdout: rbd-fuse x86_64 2:14.2.16-280.g962fac13.el7 ceph 69 k 2021-02-10T11:10:40.110 INFO:teuthology.orchestra.run.gibba012.stdout:Updating: 2021-02-10T11:10:40.110 INFO:teuthology.orchestra.run.gibba012.stdout: librados2 x86_64 2:14.2.16-280.g962fac13.el7 ceph 3.4 M 2021-02-10T11:10:40.110 INFO:teuthology.orchestra.run.gibba012.stdout: librbd1 x86_64 2:14.2.16-280.g962fac13.el7 ceph 1.7 M 2021-02-10T11:10:40.110 INFO:teuthology.orchestra.run.gibba012.stdout:Installing for dependencies: 2021-02-10T11:10:40.110 INFO:teuthology.orchestra.run.gibba012.stdout: ceph-base x86_64 2:14.2.16-280.g962fac13.el7 ceph 5.4 M 2021-02-10T11:10:40.110 INFO:teuthology.orchestra.run.gibba012.stdout: ceph-common x86_64 2:14.2.16-280.g962fac13.el7 ceph 19 M 2021-02-10T11:10:40.111 INFO:teuthology.orchestra.run.gibba012.stdout: ceph-grafana-dashboards 2021-02-10T11:10:40.111 INFO:teuthology.orchestra.run.gibba012.stdout: noarch 2:14.2.16-280.g962fac13.el7 ceph-noarch 19 k 2021-02-10T11:10:40.111 INFO:teuthology.orchestra.run.gibba012.stdout: ceph-mds x86_64 2:14.2.16-280.g962fac13.el7 ceph 1.9 M 2021-02-10T11:10:40.111 INFO:teuthology.orchestra.run.gibba012.stdout: ceph-mon x86_64 2:14.2.16-280.g962fac13.el7 ceph 4.3 M 2021-02-10T11:10:40.111 INFO:teuthology.orchestra.run.gibba012.stdout: ceph-osd x86_64 2:14.2.16-280.g962fac13.el7 ceph 16 M 2021-02-10T11:10:40.111 INFO:teuthology.orchestra.run.gibba012.stdout: ceph-selinux x86_64 2:14.2.16-280.g962fac13.el7 ceph 20 k 2021-02-10T11:10:40.111 INFO:teuthology.orchestra.run.gibba012.stdout: jq x86_64 1.6-2.el7 epel 167 k 2021-02-10T11:10:40.111 INFO:teuthology.orchestra.run.gibba012.stdout: keyutils-libs-devel 2021-02-10T11:10:40.111 INFO:teuthology.orchestra.run.gibba012.stdout: x86_64 1.5.8-3.el7 rhel-7-server-rpms 37 k 2021-02-10T11:10:40.111 INFO:teuthology.orchestra.run.gibba012.stdout: krb5-devel x86_64 1.15.1-50.el7 rhel-7-server-rpms 273 k 2021-02-10T11:10:40.111 INFO:teuthology.orchestra.run.gibba012.stdout: libbabeltrace x86_64 1.2.4-3.el7 epel 147 k 2021-02-10T11:10:40.111 INFO:teuthology.orchestra.run.gibba012.stdout: libcom_err-devel x86_64 1.42.9-19.el7 rhel-7-server-rpms 32 k 2021-02-10T11:10:40.112 INFO:teuthology.orchestra.run.gibba012.stdout: liboath x86_64 2.6.2-1.el7 epel 51 k 2021-02-10T11:10:40.112 INFO:teuthology.orchestra.run.gibba012.stdout: librabbitmq x86_64 0.8.0-3.el7 rhel-7-server-rpms 37 k 2021-02-10T11:10:40.112 INFO:teuthology.orchestra.run.gibba012.stdout: librados-devel x86_64 2:14.2.16-280.g962fac13.el7 ceph 86 k 2021-02-10T11:10:40.112 INFO:teuthology.orchestra.run.gibba012.stdout: libradosstriper1 x86_64 2:14.2.16-280.g962fac13.el7 ceph 364 k 2021-02-10T11:10:40.112 INFO:teuthology.orchestra.run.gibba012.stdout: librdkafka x86_64 0.11.5-1.el7 epel 368 k 2021-02-10T11:10:40.113 INFO:teuthology.orchestra.run.gibba012.stdout: librgw2 x86_64 2:14.2.16-280.g962fac13.el7 ceph 5.1 M 2021-02-10T11:10:40.113 INFO:teuthology.orchestra.run.gibba012.stdout: libselinux-devel x86_64 2.5-15.el7 rhel-7-server-rpms 187 k 2021-02-10T11:10:40.113 INFO:teuthology.orchestra.run.gibba012.stdout: libsepol-devel x86_64 2.5-10.el7 rhel-7-server-rpms 77 k 2021-02-10T11:10:40.113 INFO:teuthology.orchestra.run.gibba012.stdout: libverto-devel x86_64 0.2.5-4.el7 rhel-7-server-rpms 12 k 2021-02-10T11:10:40.113 INFO:teuthology.orchestra.run.gibba012.stdout: lttng-ust x86_64 2.4.1-4.el7 epel 176 k 2021-02-10T11:10:40.113 INFO:teuthology.orchestra.run.gibba012.stdout: numpy-f2py x86_64 1:1.7.1-13.el7 rhel-7-server-rpms 206 k 2021-02-10T11:10:40.113 INFO:teuthology.orchestra.run.gibba012.stdout: oniguruma x86_64 6.8.2-1.el7 epel 181 k 2021-02-10T11:10:40.113 INFO:teuthology.orchestra.run.gibba012.stdout: pcre-devel x86_64 8.32-17.el7 rhel-7-server-rpms 480 k 2021-02-10T11:10:40.114 INFO:teuthology.orchestra.run.gibba012.stdout: python-beaker noarch 1.5.4-10.el7 rhel-7-server-rpms 80 k 2021-02-10T11:10:40.114 INFO:teuthology.orchestra.run.gibba012.stdout: python-cachetools noarch 1.0.3-1.el7 epel 22 k 2021-02-10T11:10:40.114 INFO:teuthology.orchestra.run.gibba012.stdout: python-ceph-argparse 2021-02-10T11:10:40.114 INFO:teuthology.orchestra.run.gibba012.stdout: x86_64 2:14.2.16-280.g962fac13.el7 ceph 35 k 2021-02-10T11:10:40.114 INFO:teuthology.orchestra.run.gibba012.stdout: python-cephfs x86_64 2:14.2.16-280.g962fac13.el7 ceph 113 k 2021-02-10T11:10:40.114 INFO:teuthology.orchestra.run.gibba012.stdout: python-cherrypy noarch 3.2.2-4.el7 rhel-7-server-rpms 422 k 2021-02-10T11:10:40.114 INFO:teuthology.orchestra.run.gibba012.stdout: python-execnet noarch 1.7.1-3.el7 epel 260 k 2021-02-10T11:10:40.114 INFO:teuthology.orchestra.run.gibba012.stdout: python-jwcrypto noarch 0.4.2-1.el7 rhel-7-server-rpms 57 k 2021-02-10T11:10:40.114 INFO:teuthology.orchestra.run.gibba012.stdout: python-mako noarch 0.8.1-2.el7 rhel-7-server-rpms 307 k 2021-02-10T11:10:40.114 INFO:teuthology.orchestra.run.gibba012.stdout: python-markupsafe x86_64 0.11-10.el7 rhel-7-server-rpms 25 k 2021-02-10T11:10:40.114 INFO:teuthology.orchestra.run.gibba012.stdout: python-paste noarch 1.7.5.1-9.20111221hg1498.el7 2021-02-10T11:10:40.114 INFO:teuthology.orchestra.run.gibba012.stdout: rhel-7-server-rpms 866 k 2021-02-10T11:10:40.115 INFO:teuthology.orchestra.run.gibba012.stdout: python-pecan noarch 0.4.5-2.el7 epel 255 k 2021-02-10T11:10:40.115 INFO:teuthology.orchestra.run.gibba012.stdout: python-prettytable noarch 0.7.2-3.el7 rhel-7-server-rpms 37 k 2021-02-10T11:10:40.115 INFO:teuthology.orchestra.run.gibba012.stdout: python-rados x86_64 2:14.2.16-280.g962fac13.el7 ceph 199 k 2021-02-10T11:10:40.115 INFO:teuthology.orchestra.run.gibba012.stdout: python-rbd x86_64 2:14.2.16-280.g962fac13.el7 ceph 194 k 2021-02-10T11:10:40.115 INFO:teuthology.orchestra.run.gibba012.stdout: python-repoze-lru noarch 0.4-3.el7 epel 13 k 2021-02-10T11:10:40.115 INFO:teuthology.orchestra.run.gibba012.stdout: python-requests noarch 2.6.0-10.el7 rhel-7-server-rpms 95 k 2021-02-10T11:10:40.115 INFO:teuthology.orchestra.run.gibba012.stdout: python-rgw x86_64 2:14.2.16-280.g962fac13.el7 ceph 77 k 2021-02-10T11:10:40.115 INFO:teuthology.orchestra.run.gibba012.stdout: python-routes noarch 1.13-2.el7 epel 640 k 2021-02-10T11:10:40.115 INFO:teuthology.orchestra.run.gibba012.stdout: python-simplegeneric 2021-02-10T11:10:40.115 INFO:teuthology.orchestra.run.gibba012.stdout: noarch 0.8-7.el7 epel 12 k 2021-02-10T11:10:40.115 INFO:teuthology.orchestra.run.gibba012.stdout: python-singledispatch 2021-02-10T11:10:40.115 INFO:teuthology.orchestra.run.gibba012.stdout: noarch 3.4.0.2-2.el7 epel 18 k 2021-02-10T11:10:40.115 INFO:teuthology.orchestra.run.gibba012.stdout: python-tempita noarch 0.5.1-6.el7 rhel-7-server-rpms 33 k 2021-02-10T11:10:40.116 INFO:teuthology.orchestra.run.gibba012.stdout: python-urllib3 noarch 1.10.2-7.el7 rhel-7-server-rpms 103 k 2021-02-10T11:10:40.116 INFO:teuthology.orchestra.run.gibba012.stdout: python-webob noarch 1.2.3-7.el7 rhel-7-server-optional-rpms 2021-02-10T11:10:40.116 INFO:teuthology.orchestra.run.gibba012.stdout: 202 k 2021-02-10T11:10:40.116 INFO:teuthology.orchestra.run.gibba012.stdout: python-websocket-client 2021-02-10T11:10:40.116 INFO:teuthology.orchestra.run.gibba012.stdout: noarch 0.56.0-3.git3c25814.el7 rhel-7-server-extras-rpms 2021-02-10T11:10:40.116 INFO:teuthology.orchestra.run.gibba012.stdout: 66 k 2021-02-10T11:10:40.116 INFO:teuthology.orchestra.run.gibba012.stdout: python-webtest noarch 1.3.4-6.el7 rhel-7-server-optional-rpms 2021-02-10T11:10:40.116 INFO:teuthology.orchestra.run.gibba012.stdout: 102 k 2021-02-10T11:10:40.116 INFO:teuthology.orchestra.run.gibba012.stdout: python-werkzeug noarch 0.9.1-2.el7 rhel-7-server-extras-rpms 2021-02-10T11:10:40.116 INFO:teuthology.orchestra.run.gibba012.stdout: 562 k 2021-02-10T11:10:40.116 INFO:teuthology.orchestra.run.gibba012.stdout: python2-apipkg noarch 1.5-5.el7 epel 17 k 2021-02-10T11:10:40.116 INFO:teuthology.orchestra.run.gibba012.stdout: python2-bcrypt x86_64 3.1.6-2.el7 epel 39 k 2021-02-10T11:10:40.117 INFO:teuthology.orchestra.run.gibba012.stdout: python2-certifi noarch 2018.10.15-5.el7 epel 12 k 2021-02-10T11:10:40.117 INFO:teuthology.orchestra.run.gibba012.stdout: python2-google-auth 2021-02-10T11:10:40.117 INFO:teuthology.orchestra.run.gibba012.stdout: noarch 1:1.1.1-5.el7 epel 76 k 2021-02-10T11:10:40.117 INFO:teuthology.orchestra.run.gibba012.stdout: python2-kubernetes noarch 1:11.0.0-6.el7 epel 1.1 M 2021-02-10T11:10:40.117 INFO:teuthology.orchestra.run.gibba012.stdout: python2-oauthlib noarch 2.0.1-8.el7 rhel-7-server-rpms 146 k 2021-02-10T11:10:40.117 INFO:teuthology.orchestra.run.gibba012.stdout: python2-pyasn1-modules 2021-02-10T11:10:40.117 INFO:teuthology.orchestra.run.gibba012.stdout: noarch 0.1.9-7.el7 rhel-7-server-optional-rpms 2021-02-10T11:10:40.117 INFO:teuthology.orchestra.run.gibba012.stdout: 59 k 2021-02-10T11:10:40.117 INFO:teuthology.orchestra.run.gibba012.stdout: python2-remoto noarch 1.0.0-1.el7 epel 42 k 2021-02-10T11:10:40.117 INFO:teuthology.orchestra.run.gibba012.stdout: python2-requests-oauthlib 2021-02-10T11:10:40.117 INFO:teuthology.orchestra.run.gibba012.stdout: noarch 0.8.0-5.el7 rhel-7-server-rpms 35 k 2021-02-10T11:10:40.117 INFO:teuthology.orchestra.run.gibba012.stdout: python2-rsa noarch 3.4.2-2.el7 epel 68 k 2021-02-10T11:10:40.117 INFO:teuthology.orchestra.run.gibba012.stdout: python2-six noarch 1.9.0-0.el7 epel 2.9 k 2021-02-10T11:10:40.118 INFO:teuthology.orchestra.run.gibba012.stdout: python3-ceph-argparse 2021-02-10T11:10:40.118 INFO:teuthology.orchestra.run.gibba012.stdout: x86_64 2:14.2.16-280.g962fac13.el7 ceph 35 k 2021-02-10T11:10:40.118 INFO:teuthology.orchestra.run.gibba012.stdout: python3-rados x86_64 2:14.2.16-280.g962fac13.el7 ceph 317 k 2021-02-10T11:10:40.118 INFO:teuthology.orchestra.run.gibba012.stdout: scipy x86_64 0.12.1-6.el7 rhel-7-server-rpms 9.3 M 2021-02-10T11:10:40.118 INFO:teuthology.orchestra.run.gibba012.stdout: socat x86_64 1.7.3.2-2.el7 rhel-7-server-rpms 290 k 2021-02-10T11:10:40.118 INFO:teuthology.orchestra.run.gibba012.stdout: suitesparse x86_64 4.0.2-10.el7 rhel-7-server-rpms 928 k 2021-02-10T11:10:40.118 INFO:teuthology.orchestra.run.gibba012.stdout: tbb x86_64 4.1-9.20130314.el7 rhel-7-server-rpms 124 k 2021-02-10T11:10:40.118 INFO:teuthology.orchestra.run.gibba012.stdout: userspace-rcu x86_64 0.7.16-1.el7 epel 73 k 2021-02-10T11:10:40.118 INFO:teuthology.orchestra.run.gibba012.stdout: zlib-devel x86_64 1.2.7-19.el7_9 rhel-7-server-rpms 50 k 2021-02-10T11:10:40.118 INFO:teuthology.orchestra.run.gibba012.stdout:Updating for dependencies: 2021-02-10T11:10:40.118 INFO:teuthology.orchestra.run.gibba012.stdout: openssl x86_64 1:1.0.2k-21.el7_9 rhel-7-server-rpms 493 k 2021-02-10T11:10:40.119 INFO:teuthology.orchestra.run.gibba012.stdout: openssl-libs x86_64 1:1.0.2k-21.el7_9 rhel-7-server-rpms 1.2 M 2021-02-10T11:10:40.119 INFO:teuthology.orchestra.run.gibba012.stdout: zlib x86_64 1.2.7-19.el7_9 rhel-7-server-rpms 90 k 2021-02-10T11:10:40.119 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:10:40.119 INFO:teuthology.orchestra.run.gibba012.stdout:Transaction Summary 2021-02-10T11:10:40.119 INFO:teuthology.orchestra.run.gibba012.stdout:================================================================================ 2021-02-10T11:10:40.119 INFO:teuthology.orchestra.run.gibba012.stdout:Install 19 Packages (+70 Dependent packages) 2021-02-10T11:10:40.119 INFO:teuthology.orchestra.run.gibba012.stdout:Upgrade 2 Packages (+ 3 Dependent packages) 2021-02-10T11:10:40.119 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:10:40.120 INFO:teuthology.orchestra.run.gibba012.stdout:Total download size: 130 M 2021-02-10T11:10:40.120 INFO:teuthology.orchestra.run.gibba012.stdout:Downloading packages: 2021-02-10T11:10:40.120 INFO:teuthology.orchestra.run.gibba012.stdout:Delta RPMs disabled because /usr/bin/applydeltarpm not installed. 2021-02-10T11:10:43.532 INFO:teuthology.orchestra.run.gibba020.stdout:Resolving Dependencies 2021-02-10T11:10:43.533 INFO:teuthology.orchestra.run.gibba020.stdout:--> Running transaction check 2021-02-10T11:10:43.533 INFO:teuthology.orchestra.run.gibba020.stdout:---> Package bison.x86_64 0:3.0.4-2.el7 will be installed 2021-02-10T11:10:43.539 INFO:teuthology.orchestra.run.gibba020.stdout:---> Package ceph.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:10:43.540 INFO:teuthology.orchestra.run.gibba020.stdout:--> Processing Dependency: ceph-mon = 2:14.2.16-280.g962fac13.el7 for package: 2:ceph-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:10:43.541 INFO:teuthology.orchestra.run.gibba020.stdout:--> Processing Dependency: ceph-osd = 2:14.2.16-280.g962fac13.el7 for package: 2:ceph-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:10:43.542 INFO:teuthology.orchestra.run.gibba020.stdout:--> Processing Dependency: ceph-mds = 2:14.2.16-280.g962fac13.el7 for package: 2:ceph-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:10:43.543 INFO:teuthology.orchestra.run.gibba020.stdout:---> Package ceph-fuse.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:10:43.550 INFO:teuthology.orchestra.run.gibba020.stdout:---> Package ceph-mgr.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:10:43.552 INFO:teuthology.orchestra.run.gibba020.stdout:--> Processing Dependency: ceph-base = 2:14.2.16-280.g962fac13.el7 for package: 2:ceph-mgr-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:10:43.554 INFO:teuthology.orchestra.run.gibba020.stdout:--> Processing Dependency: python-requests for package: 2:ceph-mgr-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:10:43.557 INFO:teuthology.orchestra.run.gibba020.stdout:--> Processing Dependency: python-pecan for package: 2:ceph-mgr-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:10:43.558 INFO:teuthology.orchestra.run.gibba020.stdout:--> Processing Dependency: python-cherrypy for package: 2:ceph-mgr-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:10:43.558 INFO:teuthology.orchestra.run.gibba020.stdout:--> Processing Dependency: python-werkzeug for package: 2:ceph-mgr-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:10:43.559 INFO:teuthology.orchestra.run.gibba020.stdout:--> Processing Dependency: python-bcrypt for package: 2:ceph-mgr-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:10:43.560 INFO:teuthology.orchestra.run.gibba020.stdout:---> Package ceph-mgr-dashboard.noarch 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:10:43.560 INFO:teuthology.orchestra.run.gibba020.stdout:--> Processing Dependency: ceph-grafana-dashboards = 2:14.2.16-280.g962fac13.el7 for package: 2:ceph-mgr-dashboard-14.2.16-280.g962fac13.el7.noarch 2021-02-10T11:10:43.561 INFO:teuthology.orchestra.run.gibba020.stdout:--> Processing Dependency: python-routes for package: 2:ceph-mgr-dashboard-14.2.16-280.g962fac13.el7.noarch 2021-02-10T11:10:43.562 INFO:teuthology.orchestra.run.gibba020.stdout:---> Package ceph-mgr-diskprediction-cloud.noarch 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:10:43.562 INFO:teuthology.orchestra.run.gibba020.stdout:---> Package ceph-mgr-diskprediction-local.noarch 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:10:43.563 INFO:teuthology.orchestra.run.gibba020.stdout:--> Processing Dependency: scipy for package: 2:ceph-mgr-diskprediction-local-14.2.16-280.g962fac13.el7.noarch 2021-02-10T11:10:43.564 INFO:teuthology.orchestra.run.gibba020.stdout:---> Package ceph-mgr-rook.noarch 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:10:43.564 INFO:teuthology.orchestra.run.gibba020.stdout:--> Processing Dependency: python-kubernetes for package: 2:ceph-mgr-rook-14.2.16-280.g962fac13.el7.noarch 2021-02-10T11:10:43.565 INFO:teuthology.orchestra.run.gibba020.stdout:---> Package ceph-mgr-ssh.noarch 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:10:43.565 INFO:teuthology.orchestra.run.gibba020.stdout:--> Processing Dependency: python-remoto for package: 2:ceph-mgr-ssh-14.2.16-280.g962fac13.el7.noarch 2021-02-10T11:10:43.565 INFO:teuthology.orchestra.run.gibba020.stdout:---> Package ceph-radosgw.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:10:43.569 INFO:teuthology.orchestra.run.gibba020.stdout:--> Processing Dependency: ceph-selinux = 2:14.2.16-280.g962fac13.el7 for package: 2:ceph-radosgw-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:10:43.570 INFO:teuthology.orchestra.run.gibba020.stdout:--> Processing Dependency: librgw2 = 2:14.2.16-280.g962fac13.el7 for package: 2:ceph-radosgw-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:10:43.571 INFO:teuthology.orchestra.run.gibba020.stdout:--> Processing Dependency: librabbitmq.so.4()(64bit) for package: 2:ceph-radosgw-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:10:43.572 INFO:teuthology.orchestra.run.gibba020.stdout:--> Processing Dependency: liboath.so.0()(64bit) for package: 2:ceph-radosgw-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:10:43.573 INFO:teuthology.orchestra.run.gibba020.stdout:---> Package ceph-test.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:10:43.576 INFO:teuthology.orchestra.run.gibba020.stdout:--> Processing Dependency: ceph-common = 2:14.2.16-280.g962fac13.el7 for package: 2:ceph-test-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:10:43.577 INFO:teuthology.orchestra.run.gibba020.stdout:--> Processing Dependency: jq for package: 2:ceph-test-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:10:43.578 INFO:teuthology.orchestra.run.gibba020.stdout:--> Processing Dependency: socat for package: 2:ceph-test-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:10:43.579 INFO:teuthology.orchestra.run.gibba020.stdout:--> Processing Dependency: libradosstriper.so.1()(64bit) for package: 2:ceph-test-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:10:43.580 INFO:teuthology.orchestra.run.gibba020.stdout:---> Package elfutils-libelf-devel.x86_64 0:0.176-5.el7 will be installed 2021-02-10T11:10:43.581 INFO:teuthology.orchestra.run.gibba020.stdout:--> Processing Dependency: pkgconfig(zlib) for package: elfutils-libelf-devel-0.176-5.el7.x86_64 2021-02-10T11:10:43.585 INFO:teuthology.orchestra.run.gibba020.stdout:---> Package flex.x86_64 0:2.5.37-6.el7 will be installed 2021-02-10T11:10:43.586 INFO:teuthology.orchestra.run.gibba020.stdout:---> Package libcephfs-devel.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:10:43.586 INFO:teuthology.orchestra.run.gibba020.stdout:--> Processing Dependency: librados-devel = 2:14.2.16-280.g962fac13.el7 for package: 2:libcephfs-devel-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:10:43.587 INFO:teuthology.orchestra.run.gibba020.stdout:---> Package libcephfs2.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:10:43.588 INFO:teuthology.orchestra.run.gibba020.stdout:---> Package librados2.x86_64 1:10.2.5-4.el7 will be updated 2021-02-10T11:10:43.588 INFO:teuthology.orchestra.run.gibba020.stdout:---> Package librados2.x86_64 2:14.2.16-280.g962fac13.el7 will be an update 2021-02-10T11:10:43.589 INFO:teuthology.orchestra.run.gibba020.stdout:--> Processing Dependency: liblttng-ust.so.0()(64bit) for package: 2:librados2-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:10:43.590 INFO:teuthology.orchestra.run.gibba020.stdout:---> Package librbd1.x86_64 1:10.2.5-4.el7 will be updated 2021-02-10T11:10:43.590 INFO:teuthology.orchestra.run.gibba020.stdout:---> Package librbd1.x86_64 2:14.2.16-280.g962fac13.el7 will be an update 2021-02-10T11:10:43.591 INFO:teuthology.orchestra.run.gibba020.stdout:---> Package openssl-devel.x86_64 1:1.0.2k-21.el7_9 will be installed 2021-02-10T11:10:43.591 INFO:teuthology.orchestra.run.gibba020.stdout:--> Processing Dependency: openssl-libs(x86-64) = 1:1.0.2k-21.el7_9 for package: 1:openssl-devel-1.0.2k-21.el7_9.x86_64 2021-02-10T11:10:43.603 INFO:teuthology.orchestra.run.gibba020.stdout:--> Processing Dependency: krb5-devel(x86-64) for package: 1:openssl-devel-1.0.2k-21.el7_9.x86_64 2021-02-10T11:10:43.609 INFO:teuthology.orchestra.run.gibba020.stdout:---> Package python-ceph-compat.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:10:43.609 INFO:teuthology.orchestra.run.gibba020.stdout:--> Processing Dependency: python-rados = 2:14.2.16-280.g962fac13.el7 for package: 2:python-ceph-compat-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:10:43.610 INFO:teuthology.orchestra.run.gibba020.stdout:--> Processing Dependency: python-cephfs = 2:14.2.16-280.g962fac13.el7 for package: 2:python-ceph-compat-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:10:43.610 INFO:teuthology.orchestra.run.gibba020.stdout:--> Processing Dependency: python-rgw = 2:14.2.16-280.g962fac13.el7 for package: 2:python-ceph-compat-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:10:43.611 INFO:teuthology.orchestra.run.gibba020.stdout:--> Processing Dependency: python-rbd = 2:14.2.16-280.g962fac13.el7 for package: 2:python-ceph-compat-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:10:43.612 INFO:teuthology.orchestra.run.gibba020.stdout:---> Package python3-cephfs.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:10:43.613 INFO:teuthology.orchestra.run.gibba020.stdout:--> Processing Dependency: python3-ceph-argparse = 2:14.2.16-280.g962fac13.el7 for package: 2:python3-cephfs-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:10:43.614 INFO:teuthology.orchestra.run.gibba020.stdout:--> Processing Dependency: python3-rados = 2:14.2.16-280.g962fac13.el7 for package: 2:python3-cephfs-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:10:43.614 INFO:teuthology.orchestra.run.gibba020.stdout:---> Package rbd-fuse.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:10:43.615 INFO:teuthology.orchestra.run.gibba020.stdout:--> Running transaction check 2021-02-10T11:10:43.615 INFO:teuthology.orchestra.run.gibba020.stdout:---> Package ceph-base.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:10:43.619 INFO:teuthology.orchestra.run.gibba020.stdout:---> Package ceph-common.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:10:43.621 INFO:teuthology.orchestra.run.gibba020.stdout:--> Processing Dependency: python-ceph-argparse = 2:14.2.16-280.g962fac13.el7 for package: 2:ceph-common-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:10:43.622 INFO:teuthology.orchestra.run.gibba020.stdout:--> Processing Dependency: python-prettytable for package: 2:ceph-common-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:10:43.623 INFO:teuthology.orchestra.run.gibba020.stdout:--> Processing Dependency: librdkafka.so.1()(64bit) for package: 2:ceph-common-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:10:43.624 INFO:teuthology.orchestra.run.gibba020.stdout:--> Processing Dependency: libbabeltrace-ctf.so.1()(64bit) for package: 2:ceph-common-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:10:43.625 INFO:teuthology.orchestra.run.gibba020.stdout:--> Processing Dependency: libbabeltrace.so.1()(64bit) for package: 2:ceph-common-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:10:43.625 INFO:teuthology.orchestra.run.gibba020.stdout:---> Package ceph-grafana-dashboards.noarch 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:10:43.625 INFO:teuthology.orchestra.run.gibba020.stdout:---> Package ceph-mds.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:10:43.625 INFO:teuthology.orchestra.run.gibba020.stdout:---> Package ceph-mon.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:10:43.626 INFO:teuthology.orchestra.run.gibba020.stdout:---> Package ceph-osd.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:10:43.628 INFO:teuthology.orchestra.run.gibba020.stdout:---> Package ceph-selinux.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:10:43.630 INFO:teuthology.orchestra.run.gibba020.stdout:---> Package jq.x86_64 0:1.6-2.el7 will be installed 2021-02-10T11:10:43.630 INFO:teuthology.orchestra.run.gibba020.stdout:--> Processing Dependency: libonig.so.5()(64bit) for package: jq-1.6-2.el7.x86_64 2021-02-10T11:10:43.630 INFO:teuthology.orchestra.run.gibba020.stdout:---> Package krb5-devel.x86_64 0:1.15.1-50.el7 will be installed 2021-02-10T11:10:43.633 INFO:teuthology.orchestra.run.gibba020.stdout:--> Processing Dependency: libverto-devel for package: krb5-devel-1.15.1-50.el7.x86_64 2021-02-10T11:10:43.635 INFO:teuthology.orchestra.run.gibba020.stdout:--> Processing Dependency: libselinux-devel for package: krb5-devel-1.15.1-50.el7.x86_64 2021-02-10T11:10:43.640 INFO:teuthology.orchestra.run.gibba020.stdout:--> Processing Dependency: libcom_err-devel for package: krb5-devel-1.15.1-50.el7.x86_64 2021-02-10T11:10:43.650 INFO:teuthology.orchestra.run.gibba020.stdout:--> Processing Dependency: keyutils-libs-devel for package: krb5-devel-1.15.1-50.el7.x86_64 2021-02-10T11:10:43.652 INFO:teuthology.orchestra.run.gibba020.stdout:---> Package liboath.x86_64 0:2.6.2-1.el7 will be installed 2021-02-10T11:10:43.652 INFO:teuthology.orchestra.run.gibba020.stdout:---> Package librabbitmq.x86_64 0:0.8.0-3.el7 will be installed 2021-02-10T11:10:43.652 INFO:teuthology.orchestra.run.gibba020.stdout:---> Package librados-devel.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:10:43.653 INFO:teuthology.orchestra.run.gibba020.stdout:---> Package libradosstriper1.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:10:43.653 INFO:teuthology.orchestra.run.gibba020.stdout:---> Package librgw2.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:10:43.654 INFO:teuthology.orchestra.run.gibba020.stdout:---> Package lttng-ust.x86_64 0:2.4.1-4.el7 will be installed 2021-02-10T11:10:43.655 INFO:teuthology.orchestra.run.gibba020.stdout:--> Processing Dependency: liburcu-bp.so.1()(64bit) for package: lttng-ust-2.4.1-4.el7.x86_64 2021-02-10T11:10:43.655 INFO:teuthology.orchestra.run.gibba020.stdout:--> Processing Dependency: liburcu-cds.so.1()(64bit) for package: lttng-ust-2.4.1-4.el7.x86_64 2021-02-10T11:10:43.655 INFO:teuthology.orchestra.run.gibba020.stdout:---> Package openssl-libs.x86_64 1:1.0.2k-19.el7 will be updated 2021-02-10T11:10:43.656 INFO:teuthology.orchestra.run.gibba020.stdout:--> Processing Dependency: openssl-libs(x86-64) = 1:1.0.2k-19.el7 for package: 1:openssl-1.0.2k-19.el7.x86_64 2021-02-10T11:10:43.657 INFO:teuthology.orchestra.run.gibba020.stdout:---> Package openssl-libs.x86_64 1:1.0.2k-21.el7_9 will be an update 2021-02-10T11:10:43.658 INFO:teuthology.orchestra.run.gibba020.stdout:---> Package python-cephfs.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:10:43.658 INFO:teuthology.orchestra.run.gibba020.stdout:---> Package python-cherrypy.noarch 0:3.2.2-4.el7 will be installed 2021-02-10T11:10:43.664 INFO:teuthology.orchestra.run.gibba020.stdout:---> Package python-pecan.noarch 0:0.4.5-2.el7 will be installed 2021-02-10T11:10:43.664 INFO:teuthology.orchestra.run.gibba020.stdout:--> Processing Dependency: python-mako >= 0.4.0 for package: python-pecan-0.4.5-2.el7.noarch 2021-02-10T11:10:43.665 INFO:teuthology.orchestra.run.gibba020.stdout:--> Processing Dependency: python-simplegeneric >= 0.8 for package: python-pecan-0.4.5-2.el7.noarch 2021-02-10T11:10:43.665 INFO:teuthology.orchestra.run.gibba020.stdout:--> Processing Dependency: python-webob >= 1.2 for package: python-pecan-0.4.5-2.el7.noarch 2021-02-10T11:10:43.666 INFO:teuthology.orchestra.run.gibba020.stdout:--> Processing Dependency: python-webtest >= 1.3.1 for package: python-pecan-0.4.5-2.el7.noarch 2021-02-10T11:10:43.667 INFO:teuthology.orchestra.run.gibba020.stdout:--> Processing Dependency: python-singledispatch for package: python-pecan-0.4.5-2.el7.noarch 2021-02-10T11:10:43.668 INFO:teuthology.orchestra.run.gibba020.stdout:---> Package python-rados.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:10:43.668 INFO:teuthology.orchestra.run.gibba020.stdout:---> Package python-rbd.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:10:43.668 INFO:teuthology.orchestra.run.gibba020.stdout:---> Package python-requests.noarch 0:2.6.0-10.el7 will be installed 2021-02-10T11:10:43.669 INFO:teuthology.orchestra.run.gibba020.stdout:--> Processing Dependency: python-urllib3 >= 1.10.2-1 for package: python-requests-2.6.0-10.el7.noarch 2021-02-10T11:10:43.670 INFO:teuthology.orchestra.run.gibba020.stdout:---> Package python-rgw.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:10:43.671 INFO:teuthology.orchestra.run.gibba020.stdout:---> Package python-routes.noarch 0:1.13-2.el7 will be installed 2021-02-10T11:10:43.671 INFO:teuthology.orchestra.run.gibba020.stdout:--> Processing Dependency: python-repoze-lru for package: python-routes-1.13-2.el7.noarch 2021-02-10T11:10:43.671 INFO:teuthology.orchestra.run.gibba020.stdout:---> Package python-werkzeug.noarch 0:0.9.1-2.el7 will be installed 2021-02-10T11:10:43.671 INFO:teuthology.orchestra.run.gibba020.stdout:---> Package python2-bcrypt.x86_64 0:3.1.6-2.el7 will be installed 2021-02-10T11:10:43.672 INFO:teuthology.orchestra.run.gibba020.stdout:--> Processing Dependency: python2-six for package: python2-bcrypt-3.1.6-2.el7.x86_64 2021-02-10T11:10:43.672 INFO:teuthology.orchestra.run.gibba020.stdout:---> Package python2-kubernetes.noarch 1:11.0.0-6.el7 will be installed 2021-02-10T11:10:43.673 INFO:teuthology.orchestra.run.gibba020.stdout:--> Processing Dependency: python-certifi for package: 1:python2-kubernetes-11.0.0-6.el7.noarch 2021-02-10T11:10:43.674 INFO:teuthology.orchestra.run.gibba020.stdout:--> Processing Dependency: python-google-auth for package: 1:python2-kubernetes-11.0.0-6.el7.noarch 2021-02-10T11:10:43.674 INFO:teuthology.orchestra.run.gibba020.stdout:--> Processing Dependency: python-requests-oauthlib for package: 1:python2-kubernetes-11.0.0-6.el7.noarch 2021-02-10T11:10:43.675 INFO:teuthology.orchestra.run.gibba020.stdout:--> Processing Dependency: python-websocket-client for package: 1:python2-kubernetes-11.0.0-6.el7.noarch 2021-02-10T11:10:43.678 INFO:teuthology.orchestra.run.gibba020.stdout:---> Package python2-remoto.noarch 0:1.0.0-1.el7 will be installed 2021-02-10T11:10:43.678 INFO:teuthology.orchestra.run.gibba020.stdout:--> Processing Dependency: python2-execnet >= 1.2.0 for package: python2-remoto-1.0.0-1.el7.noarch 2021-02-10T11:10:43.679 INFO:teuthology.orchestra.run.gibba020.stdout:---> Package python3-ceph-argparse.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:10:43.679 INFO:teuthology.orchestra.run.gibba020.stdout:---> Package python3-rados.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:10:43.679 INFO:teuthology.orchestra.run.gibba020.stdout:---> Package scipy.x86_64 0:0.12.1-6.el7 will be installed 2021-02-10T11:10:43.681 INFO:teuthology.orchestra.run.gibba020.stdout:--> Processing Dependency: f2py for package: scipy-0.12.1-6.el7.x86_64 2021-02-10T11:10:43.682 INFO:teuthology.orchestra.run.gibba020.stdout:--> Processing Dependency: libumfpack.so.5()(64bit) for package: scipy-0.12.1-6.el7.x86_64 2021-02-10T11:10:43.684 INFO:teuthology.orchestra.run.gibba020.stdout:--> Processing Dependency: libamd.so.2()(64bit) for package: scipy-0.12.1-6.el7.x86_64 2021-02-10T11:10:43.684 INFO:teuthology.orchestra.run.gibba020.stdout:---> Package socat.x86_64 0:1.7.3.2-2.el7 will be installed 2021-02-10T11:10:43.685 INFO:teuthology.orchestra.run.gibba020.stdout:---> Package zlib-devel.x86_64 0:1.2.7-19.el7_9 will be installed 2021-02-10T11:10:43.685 INFO:teuthology.orchestra.run.gibba020.stdout:--> Processing Dependency: zlib = 1.2.7-19.el7_9 for package: zlib-devel-1.2.7-19.el7_9.x86_64 2021-02-10T11:10:43.687 INFO:teuthology.orchestra.run.gibba020.stdout:--> Running transaction check 2021-02-10T11:10:43.688 INFO:teuthology.orchestra.run.gibba020.stdout:---> Package keyutils-libs-devel.x86_64 0:1.5.8-3.el7 will be installed 2021-02-10T11:10:43.688 INFO:teuthology.orchestra.run.gibba020.stdout:---> Package libbabeltrace.x86_64 0:1.2.4-3.el7 will be installed 2021-02-10T11:10:43.689 INFO:teuthology.orchestra.run.gibba020.stdout:---> Package libcom_err-devel.x86_64 0:1.42.9-19.el7 will be installed 2021-02-10T11:10:43.689 INFO:teuthology.orchestra.run.gibba020.stdout:---> Package librdkafka.x86_64 0:0.11.5-1.el7 will be installed 2021-02-10T11:10:43.690 INFO:teuthology.orchestra.run.gibba020.stdout:---> Package libselinux-devel.x86_64 0:2.5-15.el7 will be installed 2021-02-10T11:10:43.690 INFO:teuthology.orchestra.run.gibba020.stdout:--> Processing Dependency: libsepol-devel(x86-64) >= 2.5-10 for package: libselinux-devel-2.5-15.el7.x86_64 2021-02-10T11:10:43.691 INFO:teuthology.orchestra.run.gibba020.stdout:--> Processing Dependency: pkgconfig(libsepol) for package: libselinux-devel-2.5-15.el7.x86_64 2021-02-10T11:10:43.692 INFO:teuthology.orchestra.run.gibba020.stdout:--> Processing Dependency: pkgconfig(libpcre) for package: libselinux-devel-2.5-15.el7.x86_64 2021-02-10T11:10:43.696 INFO:teuthology.orchestra.run.gibba020.stdout:---> Package libverto-devel.x86_64 0:0.2.5-4.el7 will be installed 2021-02-10T11:10:43.696 INFO:teuthology.orchestra.run.gibba020.stdout:---> Package numpy-f2py.x86_64 1:1.7.1-13.el7 will be installed 2021-02-10T11:10:43.697 INFO:teuthology.orchestra.run.gibba020.stdout:---> Package oniguruma.x86_64 0:6.8.2-1.el7 will be installed 2021-02-10T11:10:43.697 INFO:teuthology.orchestra.run.gibba020.stdout:---> Package openssl.x86_64 1:1.0.2k-19.el7 will be updated 2021-02-10T11:10:43.698 INFO:teuthology.orchestra.run.gibba020.stdout:---> Package openssl.x86_64 1:1.0.2k-21.el7_9 will be an update 2021-02-10T11:10:43.699 INFO:teuthology.orchestra.run.gibba020.stdout:---> Package python-ceph-argparse.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:10:43.699 INFO:teuthology.orchestra.run.gibba020.stdout:---> Package python-execnet.noarch 0:1.7.1-3.el7 will be installed 2021-02-10T11:10:43.699 INFO:teuthology.orchestra.run.gibba020.stdout:--> Processing Dependency: python-apipkg for package: python-execnet-1.7.1-3.el7.noarch 2021-02-10T11:10:43.700 INFO:teuthology.orchestra.run.gibba020.stdout:---> Package python-mako.noarch 0:0.8.1-2.el7 will be installed 2021-02-10T11:10:43.700 INFO:teuthology.orchestra.run.gibba020.stdout:--> Processing Dependency: python-markupsafe for package: python-mako-0.8.1-2.el7.noarch 2021-02-10T11:10:43.700 INFO:teuthology.orchestra.run.gibba020.stdout:--> Processing Dependency: python-beaker for package: python-mako-0.8.1-2.el7.noarch 2021-02-10T11:10:43.702 INFO:teuthology.orchestra.run.gibba020.stdout:---> Package python-prettytable.noarch 0:0.7.2-3.el7 will be installed 2021-02-10T11:10:43.702 INFO:teuthology.orchestra.run.gibba020.stdout:---> Package python-repoze-lru.noarch 0:0.4-3.el7 will be installed 2021-02-10T11:10:43.702 INFO:teuthology.orchestra.run.gibba020.stdout:---> Package python-simplegeneric.noarch 0:0.8-7.el7 will be installed 2021-02-10T11:10:43.702 INFO:teuthology.orchestra.run.gibba020.stdout:---> Package python-singledispatch.noarch 0:3.4.0.2-2.el7 will be installed 2021-02-10T11:10:43.702 INFO:teuthology.orchestra.run.gibba020.stdout:---> Package python-urllib3.noarch 0:1.10.2-7.el7 will be installed 2021-02-10T11:10:43.702 INFO:teuthology.orchestra.run.gibba020.stdout:---> Package python-webob.noarch 0:1.2.3-7.el7 will be installed 2021-02-10T11:10:43.702 INFO:teuthology.orchestra.run.gibba020.stdout:---> Package python-websocket-client.noarch 0:0.56.0-3.git3c25814.el7 will be installed 2021-02-10T11:10:43.703 INFO:teuthology.orchestra.run.gibba020.stdout:---> Package python-webtest.noarch 0:1.3.4-6.el7 will be installed 2021-02-10T11:10:43.703 INFO:teuthology.orchestra.run.gibba020.stdout:---> Package python2-certifi.noarch 0:2018.10.15-5.el7 will be installed 2021-02-10T11:10:43.703 INFO:teuthology.orchestra.run.gibba020.stdout:---> Package python2-google-auth.noarch 1:1.1.1-5.el7 will be installed 2021-02-10T11:10:43.703 INFO:teuthology.orchestra.run.gibba020.stdout:--> Processing Dependency: python-cachetools for package: 1:python2-google-auth-1.1.1-5.el7.noarch 2021-02-10T11:10:43.703 INFO:teuthology.orchestra.run.gibba020.stdout:--> Processing Dependency: python2-pyasn1-modules for package: 1:python2-google-auth-1.1.1-5.el7.noarch 2021-02-10T11:10:43.704 INFO:teuthology.orchestra.run.gibba020.stdout:--> Processing Dependency: python2-rsa for package: 1:python2-google-auth-1.1.1-5.el7.noarch 2021-02-10T11:10:43.705 INFO:teuthology.orchestra.run.gibba020.stdout:---> Package python2-requests-oauthlib.noarch 0:0.8.0-5.el7 will be installed 2021-02-10T11:10:43.705 INFO:teuthology.orchestra.run.gibba020.stdout:--> Processing Dependency: python2-oauthlib for package: python2-requests-oauthlib-0.8.0-5.el7.noarch 2021-02-10T11:10:43.706 INFO:teuthology.orchestra.run.gibba020.stdout:---> Package python2-six.noarch 0:1.9.0-0.el7 will be installed 2021-02-10T11:10:43.706 INFO:teuthology.orchestra.run.gibba020.stdout:---> Package suitesparse.x86_64 0:4.0.2-10.el7 will be installed 2021-02-10T11:10:43.706 INFO:teuthology.orchestra.run.gibba020.stdout:--> Processing Dependency: libtbbmalloc.so.2()(64bit) for package: suitesparse-4.0.2-10.el7.x86_64 2021-02-10T11:10:43.708 INFO:teuthology.orchestra.run.gibba020.stdout:--> Processing Dependency: libtbb.so.2()(64bit) for package: suitesparse-4.0.2-10.el7.x86_64 2021-02-10T11:10:43.708 INFO:teuthology.orchestra.run.gibba020.stdout:---> Package userspace-rcu.x86_64 0:0.7.16-1.el7 will be installed 2021-02-10T11:10:43.708 INFO:teuthology.orchestra.run.gibba020.stdout:---> Package zlib.x86_64 0:1.2.7-18.el7 will be updated 2021-02-10T11:10:43.710 INFO:teuthology.orchestra.run.gibba020.stdout:---> Package zlib.x86_64 0:1.2.7-19.el7_9 will be an update 2021-02-10T11:10:43.710 INFO:teuthology.orchestra.run.gibba020.stdout:--> Running transaction check 2021-02-10T11:10:43.710 INFO:teuthology.orchestra.run.gibba020.stdout:---> Package libsepol-devel.x86_64 0:2.5-10.el7 will be installed 2021-02-10T11:10:43.710 INFO:teuthology.orchestra.run.gibba020.stdout:---> Package pcre-devel.x86_64 0:8.32-17.el7 will be installed 2021-02-10T11:10:43.711 INFO:teuthology.orchestra.run.gibba020.stdout:---> Package python-beaker.noarch 0:1.5.4-10.el7 will be installed 2021-02-10T11:10:43.711 INFO:teuthology.orchestra.run.gibba020.stdout:--> Processing Dependency: python-paste for package: python-beaker-1.5.4-10.el7.noarch 2021-02-10T11:10:43.712 INFO:teuthology.orchestra.run.gibba020.stdout:---> Package python-cachetools.noarch 0:1.0.3-1.el7 will be installed 2021-02-10T11:10:43.712 INFO:teuthology.orchestra.run.gibba020.stdout:---> Package python-markupsafe.x86_64 0:0.11-10.el7 will be installed 2021-02-10T11:10:43.712 INFO:teuthology.orchestra.run.gibba020.stdout:---> Package python2-apipkg.noarch 0:1.5-5.el7 will be installed 2021-02-10T11:10:43.713 INFO:teuthology.orchestra.run.gibba020.stdout:---> Package python2-oauthlib.noarch 0:2.0.1-8.el7 will be installed 2021-02-10T11:10:43.713 INFO:teuthology.orchestra.run.gibba020.stdout:--> Processing Dependency: python-jwcrypto for package: python2-oauthlib-2.0.1-8.el7.noarch 2021-02-10T11:10:43.713 INFO:teuthology.orchestra.run.gibba020.stdout:---> Package python2-pyasn1-modules.noarch 0:0.1.9-7.el7 will be installed 2021-02-10T11:10:43.713 INFO:teuthology.orchestra.run.gibba020.stdout:---> Package python2-rsa.noarch 0:3.4.2-2.el7 will be installed 2021-02-10T11:10:43.713 INFO:teuthology.orchestra.run.gibba020.stdout:---> Package tbb.x86_64 0:4.1-9.20130314.el7 will be installed 2021-02-10T11:10:43.715 INFO:teuthology.orchestra.run.gibba020.stdout:--> Running transaction check 2021-02-10T11:10:43.715 INFO:teuthology.orchestra.run.gibba020.stdout:---> Package python-jwcrypto.noarch 0:0.4.2-1.el7 will be installed 2021-02-10T11:10:43.715 INFO:teuthology.orchestra.run.gibba020.stdout:---> Package python-paste.noarch 0:1.7.5.1-9.20111221hg1498.el7 will be installed 2021-02-10T11:10:43.715 INFO:teuthology.orchestra.run.gibba020.stdout:--> Processing Dependency: python-tempita for package: python-paste-1.7.5.1-9.20111221hg1498.el7.noarch 2021-02-10T11:10:43.715 INFO:teuthology.orchestra.run.gibba020.stdout:--> Running transaction check 2021-02-10T11:10:43.715 INFO:teuthology.orchestra.run.gibba020.stdout:---> Package python-tempita.noarch 0:0.5.1-6.el7 will be installed 2021-02-10T11:10:44.297 INFO:teuthology.orchestra.run.gibba020.stdout:--> Finished Dependency Resolution 2021-02-10T11:10:44.352 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:10:44.353 INFO:teuthology.orchestra.run.gibba020.stdout:Dependencies Resolved 2021-02-10T11:10:44.361 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:10:44.361 INFO:teuthology.orchestra.run.gibba020.stdout:================================================================================ 2021-02-10T11:10:44.361 INFO:teuthology.orchestra.run.gibba020.stdout: Package Arch Version Repository Size 2021-02-10T11:10:44.362 INFO:teuthology.orchestra.run.gibba020.stdout:================================================================================ 2021-02-10T11:10:44.362 INFO:teuthology.orchestra.run.gibba020.stdout:Installing: 2021-02-10T11:10:44.362 INFO:teuthology.orchestra.run.gibba020.stdout: bison x86_64 3.0.4-2.el7 rhel-7-server-rpms 674 k 2021-02-10T11:10:44.362 INFO:teuthology.orchestra.run.gibba020.stdout: ceph x86_64 2:14.2.16-280.g962fac13.el7 ceph 2.0 k 2021-02-10T11:10:44.362 INFO:teuthology.orchestra.run.gibba020.stdout: ceph-fuse x86_64 2:14.2.16-280.g962fac13.el7 ceph 571 k 2021-02-10T11:10:44.362 INFO:teuthology.orchestra.run.gibba020.stdout: ceph-mgr x86_64 2:14.2.16-280.g962fac13.el7 ceph 1.8 M 2021-02-10T11:10:44.362 INFO:teuthology.orchestra.run.gibba020.stdout: ceph-mgr-dashboard noarch 2:14.2.16-280.g962fac13.el7 ceph-noarch 4.0 M 2021-02-10T11:10:44.362 INFO:teuthology.orchestra.run.gibba020.stdout: ceph-mgr-diskprediction-cloud 2021-02-10T11:10:44.363 INFO:teuthology.orchestra.run.gibba020.stdout: noarch 2:14.2.16-280.g962fac13.el7 ceph-noarch 83 k 2021-02-10T11:10:44.363 INFO:teuthology.orchestra.run.gibba020.stdout: ceph-mgr-diskprediction-local 2021-02-10T11:10:44.363 INFO:teuthology.orchestra.run.gibba020.stdout: noarch 2:14.2.16-280.g962fac13.el7 ceph-noarch 1.1 M 2021-02-10T11:10:44.363 INFO:teuthology.orchestra.run.gibba020.stdout: ceph-mgr-rook noarch 2:14.2.16-280.g962fac13.el7 ceph-noarch 21 k 2021-02-10T11:10:44.363 INFO:teuthology.orchestra.run.gibba020.stdout: ceph-mgr-ssh noarch 2:14.2.16-280.g962fac13.el7 ceph-noarch 21 k 2021-02-10T11:10:44.363 INFO:teuthology.orchestra.run.gibba020.stdout: ceph-radosgw x86_64 2:14.2.16-280.g962fac13.el7 ceph 5.8 M 2021-02-10T11:10:44.363 INFO:teuthology.orchestra.run.gibba020.stdout: ceph-test x86_64 2:14.2.16-280.g962fac13.el7 ceph 34 M 2021-02-10T11:10:44.363 INFO:teuthology.orchestra.run.gibba020.stdout: elfutils-libelf-devel 2021-02-10T11:10:44.363 INFO:teuthology.orchestra.run.gibba020.stdout: x86_64 0.176-5.el7 rhel-7-server-rpms 40 k 2021-02-10T11:10:44.363 INFO:teuthology.orchestra.run.gibba020.stdout: flex x86_64 2.5.37-6.el7 rhel-7-server-rpms 293 k 2021-02-10T11:10:44.363 INFO:teuthology.orchestra.run.gibba020.stdout: libcephfs-devel x86_64 2:14.2.16-280.g962fac13.el7 ceph 17 k 2021-02-10T11:10:44.363 INFO:teuthology.orchestra.run.gibba020.stdout: libcephfs2 x86_64 2:14.2.16-280.g962fac13.el7 ceph 500 k 2021-02-10T11:10:44.363 INFO:teuthology.orchestra.run.gibba020.stdout: openssl-devel x86_64 1:1.0.2k-21.el7_9 rhel-7-server-rpms 1.5 M 2021-02-10T11:10:44.364 INFO:teuthology.orchestra.run.gibba020.stdout: python-ceph-compat x86_64 2:14.2.16-280.g962fac13.el7 ceph 2.2 k 2021-02-10T11:10:44.364 INFO:teuthology.orchestra.run.gibba020.stdout: python3-cephfs x86_64 2:14.2.16-280.g962fac13.el7 ceph 151 k 2021-02-10T11:10:44.364 INFO:teuthology.orchestra.run.gibba020.stdout: rbd-fuse x86_64 2:14.2.16-280.g962fac13.el7 ceph 69 k 2021-02-10T11:10:44.364 INFO:teuthology.orchestra.run.gibba020.stdout:Updating: 2021-02-10T11:10:44.364 INFO:teuthology.orchestra.run.gibba020.stdout: librados2 x86_64 2:14.2.16-280.g962fac13.el7 ceph 3.4 M 2021-02-10T11:10:44.364 INFO:teuthology.orchestra.run.gibba020.stdout: librbd1 x86_64 2:14.2.16-280.g962fac13.el7 ceph 1.7 M 2021-02-10T11:10:44.364 INFO:teuthology.orchestra.run.gibba020.stdout:Installing for dependencies: 2021-02-10T11:10:44.364 INFO:teuthology.orchestra.run.gibba020.stdout: ceph-base x86_64 2:14.2.16-280.g962fac13.el7 ceph 5.4 M 2021-02-10T11:10:44.364 INFO:teuthology.orchestra.run.gibba020.stdout: ceph-common x86_64 2:14.2.16-280.g962fac13.el7 ceph 19 M 2021-02-10T11:10:44.364 INFO:teuthology.orchestra.run.gibba020.stdout: ceph-grafana-dashboards 2021-02-10T11:10:44.364 INFO:teuthology.orchestra.run.gibba020.stdout: noarch 2:14.2.16-280.g962fac13.el7 ceph-noarch 19 k 2021-02-10T11:10:44.364 INFO:teuthology.orchestra.run.gibba020.stdout: ceph-mds x86_64 2:14.2.16-280.g962fac13.el7 ceph 1.9 M 2021-02-10T11:10:44.364 INFO:teuthology.orchestra.run.gibba020.stdout: ceph-mon x86_64 2:14.2.16-280.g962fac13.el7 ceph 4.3 M 2021-02-10T11:10:44.365 INFO:teuthology.orchestra.run.gibba020.stdout: ceph-osd x86_64 2:14.2.16-280.g962fac13.el7 ceph 16 M 2021-02-10T11:10:44.365 INFO:teuthology.orchestra.run.gibba020.stdout: ceph-selinux x86_64 2:14.2.16-280.g962fac13.el7 ceph 20 k 2021-02-10T11:10:44.365 INFO:teuthology.orchestra.run.gibba020.stdout: jq x86_64 1.6-2.el7 epel 167 k 2021-02-10T11:10:44.365 INFO:teuthology.orchestra.run.gibba020.stdout: keyutils-libs-devel 2021-02-10T11:10:44.365 INFO:teuthology.orchestra.run.gibba020.stdout: x86_64 1.5.8-3.el7 rhel-7-server-rpms 37 k 2021-02-10T11:10:44.365 INFO:teuthology.orchestra.run.gibba020.stdout: krb5-devel x86_64 1.15.1-50.el7 rhel-7-server-rpms 273 k 2021-02-10T11:10:44.365 INFO:teuthology.orchestra.run.gibba020.stdout: libbabeltrace x86_64 1.2.4-3.el7 epel 147 k 2021-02-10T11:10:44.365 INFO:teuthology.orchestra.run.gibba020.stdout: libcom_err-devel x86_64 1.42.9-19.el7 rhel-7-server-rpms 32 k 2021-02-10T11:10:44.365 INFO:teuthology.orchestra.run.gibba020.stdout: liboath x86_64 2.6.2-1.el7 epel 51 k 2021-02-10T11:10:44.365 INFO:teuthology.orchestra.run.gibba020.stdout: librabbitmq x86_64 0.8.0-3.el7 rhel-7-server-rpms 37 k 2021-02-10T11:10:44.365 INFO:teuthology.orchestra.run.gibba020.stdout: librados-devel x86_64 2:14.2.16-280.g962fac13.el7 ceph 86 k 2021-02-10T11:10:44.365 INFO:teuthology.orchestra.run.gibba020.stdout: libradosstriper1 x86_64 2:14.2.16-280.g962fac13.el7 ceph 364 k 2021-02-10T11:10:44.365 INFO:teuthology.orchestra.run.gibba020.stdout: librdkafka x86_64 0.11.5-1.el7 epel 368 k 2021-02-10T11:10:44.366 INFO:teuthology.orchestra.run.gibba020.stdout: librgw2 x86_64 2:14.2.16-280.g962fac13.el7 ceph 5.1 M 2021-02-10T11:10:44.366 INFO:teuthology.orchestra.run.gibba020.stdout: libselinux-devel x86_64 2.5-15.el7 rhel-7-server-rpms 187 k 2021-02-10T11:10:44.366 INFO:teuthology.orchestra.run.gibba020.stdout: libsepol-devel x86_64 2.5-10.el7 rhel-7-server-rpms 77 k 2021-02-10T11:10:44.366 INFO:teuthology.orchestra.run.gibba020.stdout: libverto-devel x86_64 0.2.5-4.el7 rhel-7-server-rpms 12 k 2021-02-10T11:10:44.366 INFO:teuthology.orchestra.run.gibba020.stdout: lttng-ust x86_64 2.4.1-4.el7 epel 176 k 2021-02-10T11:10:44.366 INFO:teuthology.orchestra.run.gibba020.stdout: numpy-f2py x86_64 1:1.7.1-13.el7 rhel-7-server-rpms 206 k 2021-02-10T11:10:44.366 INFO:teuthology.orchestra.run.gibba020.stdout: oniguruma x86_64 6.8.2-1.el7 epel 181 k 2021-02-10T11:10:44.366 INFO:teuthology.orchestra.run.gibba020.stdout: pcre-devel x86_64 8.32-17.el7 rhel-7-server-rpms 480 k 2021-02-10T11:10:44.366 INFO:teuthology.orchestra.run.gibba020.stdout: python-beaker noarch 1.5.4-10.el7 rhel-7-server-rpms 80 k 2021-02-10T11:10:44.366 INFO:teuthology.orchestra.run.gibba020.stdout: python-cachetools noarch 1.0.3-1.el7 epel 22 k 2021-02-10T11:10:44.366 INFO:teuthology.orchestra.run.gibba020.stdout: python-ceph-argparse 2021-02-10T11:10:44.366 INFO:teuthology.orchestra.run.gibba020.stdout: x86_64 2:14.2.16-280.g962fac13.el7 ceph 35 k 2021-02-10T11:10:44.367 INFO:teuthology.orchestra.run.gibba020.stdout: python-cephfs x86_64 2:14.2.16-280.g962fac13.el7 ceph 113 k 2021-02-10T11:10:44.367 INFO:teuthology.orchestra.run.gibba020.stdout: python-cherrypy noarch 3.2.2-4.el7 rhel-7-server-rpms 422 k 2021-02-10T11:10:44.367 INFO:teuthology.orchestra.run.gibba020.stdout: python-execnet noarch 1.7.1-3.el7 epel 260 k 2021-02-10T11:10:44.367 INFO:teuthology.orchestra.run.gibba020.stdout: python-jwcrypto noarch 0.4.2-1.el7 rhel-7-server-rpms 57 k 2021-02-10T11:10:44.367 INFO:teuthology.orchestra.run.gibba020.stdout: python-mako noarch 0.8.1-2.el7 rhel-7-server-rpms 307 k 2021-02-10T11:10:44.367 INFO:teuthology.orchestra.run.gibba020.stdout: python-markupsafe x86_64 0.11-10.el7 rhel-7-server-rpms 25 k 2021-02-10T11:10:44.367 INFO:teuthology.orchestra.run.gibba020.stdout: python-paste noarch 1.7.5.1-9.20111221hg1498.el7 2021-02-10T11:10:44.367 INFO:teuthology.orchestra.run.gibba020.stdout: rhel-7-server-rpms 866 k 2021-02-10T11:10:44.367 INFO:teuthology.orchestra.run.gibba020.stdout: python-pecan noarch 0.4.5-2.el7 epel 255 k 2021-02-10T11:10:44.367 INFO:teuthology.orchestra.run.gibba020.stdout: python-prettytable noarch 0.7.2-3.el7 rhel-7-server-rpms 37 k 2021-02-10T11:10:44.367 INFO:teuthology.orchestra.run.gibba020.stdout: python-rados x86_64 2:14.2.16-280.g962fac13.el7 ceph 199 k 2021-02-10T11:10:44.367 INFO:teuthology.orchestra.run.gibba020.stdout: python-rbd x86_64 2:14.2.16-280.g962fac13.el7 ceph 194 k 2021-02-10T11:10:44.367 INFO:teuthology.orchestra.run.gibba020.stdout: python-repoze-lru noarch 0.4-3.el7 epel 13 k 2021-02-10T11:10:44.368 INFO:teuthology.orchestra.run.gibba020.stdout: python-requests noarch 2.6.0-10.el7 rhel-7-server-rpms 95 k 2021-02-10T11:10:44.368 INFO:teuthology.orchestra.run.gibba020.stdout: python-rgw x86_64 2:14.2.16-280.g962fac13.el7 ceph 77 k 2021-02-10T11:10:44.368 INFO:teuthology.orchestra.run.gibba020.stdout: python-routes noarch 1.13-2.el7 epel 640 k 2021-02-10T11:10:44.368 INFO:teuthology.orchestra.run.gibba020.stdout: python-simplegeneric 2021-02-10T11:10:44.368 INFO:teuthology.orchestra.run.gibba020.stdout: noarch 0.8-7.el7 epel 12 k 2021-02-10T11:10:44.368 INFO:teuthology.orchestra.run.gibba020.stdout: python-singledispatch 2021-02-10T11:10:44.368 INFO:teuthology.orchestra.run.gibba020.stdout: noarch 3.4.0.2-2.el7 epel 18 k 2021-02-10T11:10:44.368 INFO:teuthology.orchestra.run.gibba020.stdout: python-tempita noarch 0.5.1-6.el7 rhel-7-server-rpms 33 k 2021-02-10T11:10:44.368 INFO:teuthology.orchestra.run.gibba020.stdout: python-urllib3 noarch 1.10.2-7.el7 rhel-7-server-rpms 103 k 2021-02-10T11:10:44.368 INFO:teuthology.orchestra.run.gibba020.stdout: python-webob noarch 1.2.3-7.el7 rhel-7-server-optional-rpms 2021-02-10T11:10:44.368 INFO:teuthology.orchestra.run.gibba020.stdout: 202 k 2021-02-10T11:10:44.368 INFO:teuthology.orchestra.run.gibba020.stdout: python-websocket-client 2021-02-10T11:10:44.368 INFO:teuthology.orchestra.run.gibba020.stdout: noarch 0.56.0-3.git3c25814.el7 rhel-7-server-extras-rpms 2021-02-10T11:10:44.369 INFO:teuthology.orchestra.run.gibba020.stdout: 66 k 2021-02-10T11:10:44.369 INFO:teuthology.orchestra.run.gibba020.stdout: python-webtest noarch 1.3.4-6.el7 rhel-7-server-optional-rpms 2021-02-10T11:10:44.369 INFO:teuthology.orchestra.run.gibba020.stdout: 102 k 2021-02-10T11:10:44.369 INFO:teuthology.orchestra.run.gibba020.stdout: python-werkzeug noarch 0.9.1-2.el7 rhel-7-server-extras-rpms 2021-02-10T11:10:44.369 INFO:teuthology.orchestra.run.gibba020.stdout: 562 k 2021-02-10T11:10:44.369 INFO:teuthology.orchestra.run.gibba020.stdout: python2-apipkg noarch 1.5-5.el7 epel 17 k 2021-02-10T11:10:44.369 INFO:teuthology.orchestra.run.gibba020.stdout: python2-bcrypt x86_64 3.1.6-2.el7 epel 39 k 2021-02-10T11:10:44.369 INFO:teuthology.orchestra.run.gibba020.stdout: python2-certifi noarch 2018.10.15-5.el7 epel 12 k 2021-02-10T11:10:44.369 INFO:teuthology.orchestra.run.gibba020.stdout: python2-google-auth 2021-02-10T11:10:44.369 INFO:teuthology.orchestra.run.gibba020.stdout: noarch 1:1.1.1-5.el7 epel 76 k 2021-02-10T11:10:44.369 INFO:teuthology.orchestra.run.gibba020.stdout: python2-kubernetes noarch 1:11.0.0-6.el7 epel 1.1 M 2021-02-10T11:10:44.369 INFO:teuthology.orchestra.run.gibba020.stdout: python2-oauthlib noarch 2.0.1-8.el7 rhel-7-server-rpms 146 k 2021-02-10T11:10:44.369 INFO:teuthology.orchestra.run.gibba020.stdout: python2-pyasn1-modules 2021-02-10T11:10:44.370 INFO:teuthology.orchestra.run.gibba020.stdout: noarch 0.1.9-7.el7 rhel-7-server-optional-rpms 2021-02-10T11:10:44.370 INFO:teuthology.orchestra.run.gibba020.stdout: 59 k 2021-02-10T11:10:44.370 INFO:teuthology.orchestra.run.gibba020.stdout: python2-remoto noarch 1.0.0-1.el7 epel 42 k 2021-02-10T11:10:44.370 INFO:teuthology.orchestra.run.gibba020.stdout: python2-requests-oauthlib 2021-02-10T11:10:44.370 INFO:teuthology.orchestra.run.gibba020.stdout: noarch 0.8.0-5.el7 rhel-7-server-rpms 35 k 2021-02-10T11:10:44.370 INFO:teuthology.orchestra.run.gibba020.stdout: python2-rsa noarch 3.4.2-2.el7 epel 68 k 2021-02-10T11:10:44.370 INFO:teuthology.orchestra.run.gibba020.stdout: python2-six noarch 1.9.0-0.el7 epel 2.9 k 2021-02-10T11:10:44.370 INFO:teuthology.orchestra.run.gibba020.stdout: python3-ceph-argparse 2021-02-10T11:10:44.370 INFO:teuthology.orchestra.run.gibba020.stdout: x86_64 2:14.2.16-280.g962fac13.el7 ceph 35 k 2021-02-10T11:10:44.370 INFO:teuthology.orchestra.run.gibba020.stdout: python3-rados x86_64 2:14.2.16-280.g962fac13.el7 ceph 317 k 2021-02-10T11:10:44.370 INFO:teuthology.orchestra.run.gibba020.stdout: scipy x86_64 0.12.1-6.el7 rhel-7-server-rpms 9.3 M 2021-02-10T11:10:44.370 INFO:teuthology.orchestra.run.gibba020.stdout: socat x86_64 1.7.3.2-2.el7 rhel-7-server-rpms 290 k 2021-02-10T11:10:44.371 INFO:teuthology.orchestra.run.gibba020.stdout: suitesparse x86_64 4.0.2-10.el7 rhel-7-server-rpms 928 k 2021-02-10T11:10:44.371 INFO:teuthology.orchestra.run.gibba020.stdout: tbb x86_64 4.1-9.20130314.el7 rhel-7-server-rpms 124 k 2021-02-10T11:10:44.371 INFO:teuthology.orchestra.run.gibba020.stdout: userspace-rcu x86_64 0.7.16-1.el7 epel 73 k 2021-02-10T11:10:44.371 INFO:teuthology.orchestra.run.gibba020.stdout: zlib-devel x86_64 1.2.7-19.el7_9 rhel-7-server-rpms 50 k 2021-02-10T11:10:44.371 INFO:teuthology.orchestra.run.gibba020.stdout:Updating for dependencies: 2021-02-10T11:10:44.371 INFO:teuthology.orchestra.run.gibba020.stdout: openssl x86_64 1:1.0.2k-21.el7_9 rhel-7-server-rpms 493 k 2021-02-10T11:10:44.371 INFO:teuthology.orchestra.run.gibba020.stdout: openssl-libs x86_64 1:1.0.2k-21.el7_9 rhel-7-server-rpms 1.2 M 2021-02-10T11:10:44.371 INFO:teuthology.orchestra.run.gibba020.stdout: zlib x86_64 1.2.7-19.el7_9 rhel-7-server-rpms 90 k 2021-02-10T11:10:44.371 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:10:44.371 INFO:teuthology.orchestra.run.gibba020.stdout:Transaction Summary 2021-02-10T11:10:44.371 INFO:teuthology.orchestra.run.gibba020.stdout:================================================================================ 2021-02-10T11:10:44.371 INFO:teuthology.orchestra.run.gibba020.stdout:Install 19 Packages (+70 Dependent packages) 2021-02-10T11:10:44.372 INFO:teuthology.orchestra.run.gibba020.stdout:Upgrade 2 Packages (+ 3 Dependent packages) 2021-02-10T11:10:44.372 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:10:44.372 INFO:teuthology.orchestra.run.gibba020.stdout:Total download size: 130 M 2021-02-10T11:10:44.372 INFO:teuthology.orchestra.run.gibba020.stdout:Downloading packages: 2021-02-10T11:10:44.372 INFO:teuthology.orchestra.run.gibba020.stdout:Delta RPMs disabled because /usr/bin/applydeltarpm not installed. 2021-02-10T11:10:44.593 INFO:teuthology.orchestra.run.gibba012.stdout:-------------------------------------------------------------------------------- 2021-02-10T11:10:44.593 INFO:teuthology.orchestra.run.gibba012.stdout:Total 29 MB/s | 130 MB 00:04 2021-02-10T11:10:44.628 INFO:teuthology.orchestra.run.gibba012.stdout:Running transaction check 2021-02-10T11:10:44.758 INFO:teuthology.orchestra.run.gibba012.stdout:Running transaction test 2021-02-10T11:10:45.289 INFO:teuthology.orchestra.run.gibba012.stdout:Transaction test succeeded 2021-02-10T11:10:45.289 INFO:teuthology.orchestra.run.gibba012.stdout:Running transaction 2021-02-10T11:10:46.716 INFO:teuthology.orchestra.run.gibba012.stdout: Updating : zlib-1.2.7-19.el7_9.x86_64 1/99 2021-02-10T11:10:47.076 INFO:teuthology.orchestra.run.gibba012.stdout: Updating : 1:openssl-libs-1.0.2k-21.el7_9.x86_64 2/99 2021-02-10T11:10:47.369 INFO:teuthology.orchestra.run.gibba012.stdout: Installing : liboath-2.6.2-1.el7.x86_64 3/99 2021-02-10T11:10:47.635 INFO:teuthology.orchestra.run.gibba012.stdout: Installing : librabbitmq-0.8.0-3.el7.x86_64 4/99 2021-02-10T11:10:47.918 INFO:teuthology.orchestra.run.gibba012.stdout: Installing : zlib-devel-1.2.7-19.el7_9.x86_64 5/99 2021-02-10T11:10:48.137 INFO:teuthology.orchestra.run.gibba012.stdout: Installing : python2-six-1.9.0-0.el7.noarch 6/99 2021-02-10T11:10:49.624 INFO:teuthology.orchestra.run.gibba012.stdout: Installing : 2:python-ceph-argparse-14.2.16-280.g962fac13.el7.x86_64 7/99 2021-02-10T11:10:49.624 INFO:teuthology.orchestra.run.gibba012.stdout: Installing : python-werkzeug-0.9.1-2.el7.noarch 8/99 2021-02-10T11:10:49.624 INFO:teuthology.orchestra.run.gibba012.stdout: Installing : python-cherrypy-3.2.2-4.el7.noarch 9/99 2021-02-10T11:10:49.625 INFO:teuthology.orchestra.run.gibba012.stdout: Installing : python-webob-1.2.3-7.el7.noarch 10/99 2021-02-10T11:10:49.732 INFO:teuthology.orchestra.run.gibba012.stdout: Installing : python-urllib3-1.10.2-7.el7.noarch 11/99 2021-02-10T11:10:50.051 INFO:teuthology.orchestra.run.gibba012.stdout: Installing : python-requests-2.6.0-10.el7.noarch 12/99 2021-02-10T11:10:50.314 INFO:teuthology.orchestra.run.gibba012.stdout: Installing : python-webtest-1.3.4-6.el7.noarch 13/99 2021-02-10T11:10:50.729 INFO:teuthology.orchestra.run.gibba012.stdout: Installing : python2-bcrypt-3.1.6-2.el7.x86_64 14/99 2021-02-10T11:10:51.087 INFO:teuthology.orchestra.run.gibba012.stdout: Installing : socat-1.7.3.2-2.el7.x86_64 15/99 2021-02-10T11:10:51.488 INFO:teuthology.orchestra.run.gibba012.stdout: Installing : librdkafka-0.11.5-1.el7.x86_64 16/99 2021-02-10T11:10:51.755 INFO:teuthology.orchestra.run.gibba020.stdout:-------------------------------------------------------------------------------- 2021-02-10T11:10:51.755 INFO:teuthology.orchestra.run.gibba020.stdout:Total 18 MB/s | 130 MB 00:07 2021-02-10T11:10:51.789 INFO:teuthology.orchestra.run.gibba020.stdout:Running transaction check 2021-02-10T11:10:51.817 INFO:teuthology.orchestra.run.gibba012.stdout: Installing : oniguruma-6.8.2-1.el7.x86_64 17/99 2021-02-10T11:10:51.918 INFO:teuthology.orchestra.run.gibba020.stdout:Running transaction test 2021-02-10T11:10:52.134 INFO:teuthology.orchestra.run.gibba012.stdout: Installing : jq-1.6-2.el7.x86_64 18/99 2021-02-10T11:10:52.396 INFO:teuthology.orchestra.run.gibba020.stdout:Transaction test succeeded 2021-02-10T11:10:52.396 INFO:teuthology.orchestra.run.gibba020.stdout:Running transaction 2021-02-10T11:10:52.508 INFO:teuthology.orchestra.run.gibba012.stdout: Installing : tbb-4.1-9.20130314.el7.x86_64 19/99 2021-02-10T11:10:52.925 INFO:teuthology.orchestra.run.gibba012.stdout: Installing : suitesparse-4.0.2-10.el7.x86_64 20/99 2021-02-10T11:10:53.082 INFO:teuthology.orchestra.run.gibba020.stdout: Updating : zlib-1.2.7-19.el7_9.x86_64 1/99 2021-02-10T11:10:53.292 INFO:teuthology.orchestra.run.gibba012.stdout: Installing : python-tempita-0.5.1-6.el7.noarch 21/99 2021-02-10T11:10:53.378 INFO:teuthology.orchestra.run.gibba020.stdout: Updating : 1:openssl-libs-1.0.2k-21.el7_9.x86_64 2/99 2021-02-10T11:10:53.640 INFO:teuthology.orchestra.run.gibba012.stdout: Installing : python-paste-1.7.5.1-9.20111221hg1498.el7.noarch 22/99 2021-02-10T11:10:53.677 INFO:teuthology.orchestra.run.gibba020.stdout: Installing : liboath-2.6.2-1.el7.x86_64 3/99 2021-02-10T11:10:53.913 INFO:teuthology.orchestra.run.gibba012.stdout: Installing : python-beaker-1.5.4-10.el7.noarch 23/99 2021-02-10T11:10:53.953 INFO:teuthology.orchestra.run.gibba020.stdout: Installing : librabbitmq-0.8.0-3.el7.x86_64 4/99 2021-02-10T11:10:54.169 INFO:teuthology.orchestra.run.gibba012.stdout: Installing : python-cachetools-1.0.3-1.el7.noarch 24/99 2021-02-10T11:10:54.260 INFO:teuthology.orchestra.run.gibba020.stdout: Installing : zlib-devel-1.2.7-19.el7_9.x86_64 5/99 2021-02-10T11:10:54.444 INFO:teuthology.orchestra.run.gibba012.stdout: Installing : 2:ceph-grafana-dashboards-14.2.16-280.g962fac13.el7.noar 25/99 2021-02-10T11:10:54.472 INFO:teuthology.orchestra.run.gibba020.stdout: Installing : python2-six-1.9.0-0.el7.noarch 6/99 2021-02-10T11:10:54.774 INFO:teuthology.orchestra.run.gibba012.stdout: Installing : libbabeltrace-1.2.4-3.el7.x86_64 26/99 2021-02-10T11:10:54.785 INFO:teuthology.orchestra.run.gibba020.stdout: Installing : 2:python-ceph-argparse-14.2.16-280.g962fac13.el7.x86_64 7/99 2021-02-10T11:10:55.076 INFO:teuthology.orchestra.run.gibba012.stdout: Installing : python-websocket-client-0.56.0-3.git3c25814.el7.noarch 27/99 2021-02-10T11:10:55.134 INFO:teuthology.orchestra.run.gibba020.stdout: Installing : python-werkzeug-0.9.1-2.el7.noarch 8/99 2021-02-10T11:10:55.390 INFO:teuthology.orchestra.run.gibba012.stdout: Installing : libcom_err-devel-1.42.9-19.el7.x86_64 28/99 2021-02-10T11:10:55.447 INFO:teuthology.orchestra.run.gibba020.stdout: Installing : python-cherrypy-3.2.2-4.el7.noarch 9/99 2021-02-10T11:10:55.648 INFO:teuthology.orchestra.run.gibba012.stdout: Installing : 2:python3-ceph-argparse-14.2.16-280.g962fac13.el7.x86_64 29/99 2021-02-10T11:10:55.738 INFO:teuthology.orchestra.run.gibba020.stdout: Installing : python-webob-1.2.3-7.el7.noarch 10/99 2021-02-10T11:10:56.024 INFO:teuthology.orchestra.run.gibba020.stdout: Installing : python-urllib3-1.10.2-7.el7.noarch 11/99 2021-02-10T11:10:56.187 INFO:teuthology.orchestra.run.gibba012.stdout: Installing : python2-certifi-2018.10.15-5.el7.noarch 30/99 2021-02-10T11:10:56.328 INFO:teuthology.orchestra.run.gibba020.stdout: Installing : python-requests-2.6.0-10.el7.noarch 12/99 2021-02-10T11:10:56.512 INFO:teuthology.orchestra.run.gibba012.stdout: Installing : pcre-devel-8.32-17.el7.x86_64 31/99 2021-02-10T11:10:56.574 INFO:teuthology.orchestra.run.gibba020.stdout: Installing : python-webtest-1.3.4-6.el7.noarch 13/99 2021-02-10T11:10:56.790 INFO:teuthology.orchestra.run.gibba012.stdout: Installing : python-prettytable-0.7.2-3.el7.noarch 32/99 2021-02-10T11:10:56.911 INFO:teuthology.orchestra.run.gibba020.stdout: Installing : python2-bcrypt-3.1.6-2.el7.x86_64 14/99 2021-02-10T11:10:57.059 INFO:teuthology.orchestra.run.gibba012.stdout: Installing : python2-pyasn1-modules-0.1.9-7.el7.noarch 33/99 2021-02-10T11:10:57.246 INFO:teuthology.orchestra.run.gibba020.stdout: Installing : socat-1.7.3.2-2.el7.x86_64 15/99 2021-02-10T11:10:57.342 INFO:teuthology.orchestra.run.gibba012.stdout: Installing : python-jwcrypto-0.4.2-1.el7.noarch 34/99 2021-02-10T11:10:57.512 INFO:teuthology.orchestra.run.gibba020.stdout: Installing : librdkafka-0.11.5-1.el7.x86_64 16/99 2021-02-10T11:10:57.664 INFO:teuthology.orchestra.run.gibba012.stdout: Installing : python2-oauthlib-2.0.1-8.el7.noarch 35/99 2021-02-10T11:10:57.785 INFO:teuthology.orchestra.run.gibba020.stdout: Installing : oniguruma-6.8.2-1.el7.x86_64 17/99 2021-02-10T11:10:57.970 INFO:teuthology.orchestra.run.gibba012.stdout: Installing : python2-requests-oauthlib-0.8.0-5.el7.noarch 36/99 2021-02-10T11:10:58.042 INFO:teuthology.orchestra.run.gibba020.stdout: Installing : jq-1.6-2.el7.x86_64 18/99 2021-02-10T11:10:58.330 INFO:teuthology.orchestra.run.gibba012.stdout: Installing : python2-rsa-3.4.2-2.el7.noarch 37/99 2021-02-10T11:10:58.401 INFO:teuthology.orchestra.run.gibba020.stdout: Installing : tbb-4.1-9.20130314.el7.x86_64 19/99 2021-02-10T11:10:58.750 INFO:teuthology.orchestra.run.gibba020.stdout: Installing : suitesparse-4.0.2-10.el7.x86_64 20/99 2021-02-10T11:10:59.093 INFO:teuthology.orchestra.run.gibba020.stdout: Installing : python-tempita-0.5.1-6.el7.noarch 21/99 2021-02-10T11:10:59.277 INFO:teuthology.orchestra.run.gibba012.stdout: Installing : 1:python2-google-auth-1.1.1-5.el7.noarch 38/99 2021-02-10T11:10:59.406 INFO:teuthology.orchestra.run.gibba020.stdout: Installing : python-paste-1.7.5.1-9.20111221hg1498.el7.noarch 22/99 2021-02-10T11:10:59.620 INFO:teuthology.orchestra.run.gibba012.stdout: Installing : 1:python2-kubernetes-11.0.0-6.el7.noarch 39/99 2021-02-10T11:10:59.647 INFO:teuthology.orchestra.run.gibba020.stdout: Installing : python-beaker-1.5.4-10.el7.noarch 23/99 2021-02-10T11:10:59.864 INFO:teuthology.orchestra.run.gibba012.stdout: Installing : python-singledispatch-3.4.0.2-2.el7.noarch 40/99 2021-02-10T11:10:59.895 INFO:teuthology.orchestra.run.gibba020.stdout: Installing : python-cachetools-1.0.3-1.el7.noarch 24/99 2021-02-10T11:11:00.146 INFO:teuthology.orchestra.run.gibba012.stdout: Installing : keyutils-libs-devel-1.5.8-3.el7.x86_64 41/99 2021-02-10T11:11:00.179 INFO:teuthology.orchestra.run.gibba020.stdout: Installing : 2:ceph-grafana-dashboards-14.2.16-280.g962fac13.el7.noar 25/99 2021-02-10T11:11:00.402 INFO:teuthology.orchestra.run.gibba012.stdout: Installing : python-simplegeneric-0.8-7.el7.noarch 42/99 2021-02-10T11:11:00.476 INFO:teuthology.orchestra.run.gibba020.stdout: Installing : libbabeltrace-1.2.4-3.el7.x86_64 26/99 2021-02-10T11:11:00.762 INFO:teuthology.orchestra.run.gibba012.stdout: Installing : userspace-rcu-0.7.16-1.el7.x86_64 43/99 2021-02-10T11:11:00.778 INFO:teuthology.orchestra.run.gibba020.stdout: Installing : python-websocket-client-0.56.0-3.git3c25814.el7.noarch 27/99 2021-02-10T11:11:01.070 INFO:teuthology.orchestra.run.gibba020.stdout: Installing : libcom_err-devel-1.42.9-19.el7.x86_64 28/99 2021-02-10T11:11:01.299 INFO:teuthology.orchestra.run.gibba020.stdout: Installing : 2:python3-ceph-argparse-14.2.16-280.g962fac13.el7.x86_64 29/99 2021-02-10T11:11:01.304 INFO:teuthology.orchestra.run.gibba012.stdout: Installing : lttng-ust-2.4.1-4.el7.x86_64 44/99 2021-02-10T11:11:01.646 INFO:teuthology.orchestra.run.gibba020.stdout: Installing : python2-certifi-2018.10.15-5.el7.noarch 30/99 2021-02-10T11:11:01.709 INFO:teuthology.orchestra.run.gibba012.stdout: Updating : 2:librados2-14.2.16-280.g962fac13.el7.x86_64 45/99 2021-02-10T11:11:01.921 INFO:teuthology.orchestra.run.gibba020.stdout: Installing : pcre-devel-8.32-17.el7.x86_64 31/99 2021-02-10T11:11:02.040 INFO:teuthology.orchestra.run.gibba012.stdout: Installing : 2:libcephfs2-14.2.16-280.g962fac13.el7.x86_64 46/99 2021-02-10T11:11:02.183 INFO:teuthology.orchestra.run.gibba020.stdout: Installing : python-prettytable-0.7.2-3.el7.noarch 32/99 2021-02-10T11:11:02.419 INFO:teuthology.orchestra.run.gibba012.stdout: Installing : 2:python-rados-14.2.16-280.g962fac13.el7.x86_64 47/99 2021-02-10T11:11:02.445 INFO:teuthology.orchestra.run.gibba020.stdout: Installing : python2-pyasn1-modules-0.1.9-7.el7.noarch 33/99 2021-02-10T11:11:02.727 INFO:teuthology.orchestra.run.gibba020.stdout: Installing : python-jwcrypto-0.4.2-1.el7.noarch 34/99 2021-02-10T11:11:03.007 INFO:teuthology.orchestra.run.gibba020.stdout: Installing : python2-oauthlib-2.0.1-8.el7.noarch 35/99 2021-02-10T11:11:03.042 INFO:teuthology.orchestra.run.gibba012.stdout: Updating : 2:librbd1-14.2.16-280.g962fac13.el7.x86_64 48/99 2021-02-10T11:11:03.289 INFO:teuthology.orchestra.run.gibba020.stdout: Installing : python2-requests-oauthlib-0.8.0-5.el7.noarch 36/99 2021-02-10T11:11:03.317 INFO:teuthology.orchestra.run.gibba012.stdout: Installing : 2:librgw2-14.2.16-280.g962fac13.el7.x86_64 49/99 2021-02-10T11:11:03.596 INFO:teuthology.orchestra.run.gibba020.stdout: Installing : python2-rsa-3.4.2-2.el7.noarch 37/99 2021-02-10T11:11:03.635 INFO:teuthology.orchestra.run.gibba012.stdout: Installing : 2:python-rgw-14.2.16-280.g962fac13.el7.x86_64 50/99 2021-02-10T11:11:03.955 INFO:teuthology.orchestra.run.gibba012.stdout: Installing : 2:python-rbd-14.2.16-280.g962fac13.el7.x86_64 51/99 2021-02-10T11:11:04.271 INFO:teuthology.orchestra.run.gibba012.stdout: Installing : 2:python-cephfs-14.2.16-280.g962fac13.el7.x86_64 52/99 2021-02-10T11:11:04.501 INFO:teuthology.orchestra.run.gibba020.stdout: Installing : 1:python2-google-auth-1.1.1-5.el7.noarch 38/99 2021-02-10T11:11:04.788 INFO:teuthology.orchestra.run.gibba020.stdout: Installing : 1:python2-kubernetes-11.0.0-6.el7.noarch 39/99 2021-02-10T11:11:05.025 INFO:teuthology.orchestra.run.gibba020.stdout: Installing : python-singledispatch-3.4.0.2-2.el7.noarch 40/99 2021-02-10T11:11:05.505 INFO:teuthology.orchestra.run.gibba020.stdout: Installing : keyutils-libs-devel-1.5.8-3.el7.x86_64 41/99 2021-02-10T11:11:05.513 INFO:teuthology.orchestra.run.gibba020.stdout: Installing : python-simplegeneric-0.8-7.el7.noarch 42/99 2021-02-10T11:11:05.814 INFO:teuthology.orchestra.run.gibba020.stdout: Installing : userspace-rcu-0.7.16-1.el7.x86_64 43/99 2021-02-10T11:11:06.095 INFO:teuthology.orchestra.run.gibba012.stdout: Installing : 2:libradosstriper1-14.2.16-280.g962fac13.el7.x86_64 53/99 2021-02-10T11:11:06.339 INFO:teuthology.orchestra.run.gibba020.stdout: Installing : lttng-ust-2.4.1-4.el7.x86_64 44/99 2021-02-10T11:11:06.674 INFO:teuthology.orchestra.run.gibba020.stdout: Updating : 2:librados2-14.2.16-280.g962fac13.el7.x86_64 45/99 2021-02-10T11:11:06.962 INFO:teuthology.orchestra.run.gibba012.stdout: Installing : 2:ceph-common-14.2.16-280.g962fac13.el7.x86_64 54/99 2021-02-10T11:11:07.088 INFO:teuthology.orchestra.run.gibba020.stdout: Installing : 2:libcephfs2-14.2.16-280.g962fac13.el7.x86_64 46/99 2021-02-10T11:11:07.353 INFO:teuthology.orchestra.run.gibba020.stdout: Installing : 2:python-rados-14.2.16-280.g962fac13.el7.x86_64 47/99 2021-02-10T11:11:07.383 INFO:teuthology.orchestra.run.gibba012.stdout: Installing : 2:ceph-base-14.2.16-280.g962fac13.el7.x86_64 55/99 2021-02-10T11:11:07.980 INFO:teuthology.orchestra.run.gibba020.stdout: Updating : 2:librbd1-14.2.16-280.g962fac13.el7.x86_64 48/99 2021-02-10T11:11:08.228 INFO:teuthology.orchestra.run.gibba020.stdout: Installing : 2:librgw2-14.2.16-280.g962fac13.el7.x86_64 49/99 2021-02-10T11:11:08.512 INFO:teuthology.orchestra.run.gibba020.stdout: Installing : 2:python-rgw-14.2.16-280.g962fac13.el7.x86_64 50/99 2021-02-10T11:11:08.782 INFO:teuthology.orchestra.run.gibba020.stdout: Installing : 2:python-rbd-14.2.16-280.g962fac13.el7.x86_64 51/99 2021-02-10T11:11:09.074 INFO:teuthology.orchestra.run.gibba020.stdout: Installing : 2:python-cephfs-14.2.16-280.g962fac13.el7.x86_64 52/99 2021-02-10T11:11:10.846 INFO:teuthology.orchestra.run.gibba020.stdout: Installing : 2:libradosstriper1-14.2.16-280.g962fac13.el7.x86_64 53/99 2021-02-10T11:11:11.605 INFO:teuthology.orchestra.run.gibba020.stdout: Installing : 2:ceph-common-14.2.16-280.g962fac13.el7.x86_64 54/99 2021-02-10T11:11:11.988 INFO:teuthology.orchestra.run.gibba020.stdout: Installing : 2:ceph-base-14.2.16-280.g962fac13.el7.x86_64 55/99 2021-02-10T11:11:14.800 INFO:teuthology.orchestra.run.gibba012.stdout: Installing : 2:ceph-selinux-14.2.16-280.g962fac13.el7.x86_64 56/99 2021-02-10T11:11:15.245 INFO:teuthology.orchestra.run.gibba012.stdout: Installing : 2:ceph-osd-14.2.16-280.g962fac13.el7.x86_64 57/99 2021-02-10T11:11:15.838 INFO:teuthology.orchestra.run.gibba012.stdout: Installing : 2:ceph-mds-14.2.16-280.g962fac13.el7.x86_64 58/99 2021-02-10T11:11:17.153 INFO:teuthology.orchestra.run.gibba012.stdout: Installing : 2:ceph-mon-14.2.16-280.g962fac13.el7.x86_64 59/99 2021-02-10T11:11:17.485 INFO:teuthology.orchestra.run.gibba012.stdout: Installing : 2:librados-devel-14.2.16-280.g962fac13.el7.x86_64 60/99 2021-02-10T11:11:17.758 INFO:teuthology.orchestra.run.gibba012.stdout: Installing : 2:python3-rados-14.2.16-280.g962fac13.el7.x86_64 61/99 2021-02-10T11:11:18.096 INFO:teuthology.orchestra.run.gibba012.stdout: Installing : python-repoze-lru-0.4-3.el7.noarch 62/99 2021-02-10T11:11:18.379 INFO:teuthology.orchestra.run.gibba012.stdout: Installing : python-routes-1.13-2.el7.noarch 63/99 2021-02-10T11:11:18.693 INFO:teuthology.orchestra.run.gibba012.stdout: Installing : python-markupsafe-0.11-10.el7.x86_64 64/99 2021-02-10T11:11:19.075 INFO:teuthology.orchestra.run.gibba012.stdout: Installing : python-mako-0.8.1-2.el7.noarch 65/99 2021-02-10T11:11:19.243 INFO:teuthology.orchestra.run.gibba020.stdout: Installing : 2:ceph-selinux-14.2.16-280.g962fac13.el7.x86_64 56/99 2021-02-10T11:11:19.594 INFO:teuthology.orchestra.run.gibba012.stdout: Installing : python-pecan-0.4.5-2.el7.noarch 66/99 2021-02-10T11:11:19.649 INFO:teuthology.orchestra.run.gibba020.stdout: Installing : 2:ceph-osd-14.2.16-280.g962fac13.el7.x86_64 57/99 2021-02-10T11:11:19.972 INFO:teuthology.orchestra.run.gibba012.stdout: Installing : 2:ceph-mgr-14.2.16-280.g962fac13.el7.x86_64 67/99 2021-02-10T11:11:20.336 INFO:teuthology.orchestra.run.gibba012.stdout: Installing : python2-apipkg-1.5-5.el7.noarch 68/99 2021-02-10T11:11:20.648 INFO:teuthology.orchestra.run.gibba020.stdout: Installing : 2:ceph-mds-14.2.16-280.g962fac13.el7.x86_64 58/99 2021-02-10T11:11:20.682 INFO:teuthology.orchestra.run.gibba012.stdout: Installing : python-execnet-1.7.1-3.el7.noarch 69/99 2021-02-10T11:11:20.938 INFO:teuthology.orchestra.run.gibba020.stdout: Installing : 2:ceph-mon-14.2.16-280.g962fac13.el7.x86_64 59/99 2021-02-10T11:11:21.074 INFO:teuthology.orchestra.run.gibba012.stdout: Installing : python2-remoto-1.0.0-1.el7.noarch 70/99 2021-02-10T11:11:21.220 INFO:teuthology.orchestra.run.gibba020.stdout: Installing : 2:librados-devel-14.2.16-280.g962fac13.el7.x86_64 60/99 2021-02-10T11:11:21.343 INFO:teuthology.orchestra.run.gibba012.stdout: Installing : libverto-devel-0.2.5-4.el7.x86_64 71/99 2021-02-10T11:11:21.435 INFO:teuthology.orchestra.run.gibba020.stdout: Installing : 2:python3-rados-14.2.16-280.g962fac13.el7.x86_64 61/99 2021-02-10T11:11:21.764 INFO:teuthology.orchestra.run.gibba020.stdout: Installing : python-repoze-lru-0.4-3.el7.noarch 62/99 2021-02-10T11:11:21.795 INFO:teuthology.orchestra.run.gibba012.stdout: Installing : libsepol-devel-2.5-10.el7.x86_64 72/99 2021-02-10T11:11:22.015 INFO:teuthology.orchestra.run.gibba020.stdout: Installing : python-routes-1.13-2.el7.noarch 63/99 2021-02-10T11:11:22.110 INFO:teuthology.orchestra.run.gibba012.stdout: Installing : libselinux-devel-2.5-15.el7.x86_64 73/99 2021-02-10T11:11:22.302 INFO:teuthology.orchestra.run.gibba020.stdout: Installing : python-markupsafe-0.11-10.el7.x86_64 64/99 2021-02-10T11:11:22.491 INFO:teuthology.orchestra.run.gibba012.stdout: Installing : krb5-devel-1.15.1-50.el7.x86_64 74/99 2021-02-10T11:11:22.659 INFO:teuthology.orchestra.run.gibba020.stdout: Installing : python-mako-0.8.1-2.el7.noarch 65/99 2021-02-10T11:11:23.137 INFO:teuthology.orchestra.run.gibba020.stdout: Installing : python-pecan-0.4.5-2.el7.noarch 66/99 2021-02-10T11:11:23.478 INFO:teuthology.orchestra.run.gibba020.stdout: Installing : 2:ceph-mgr-14.2.16-280.g962fac13.el7.x86_64 67/99 2021-02-10T11:11:23.766 INFO:teuthology.orchestra.run.gibba012.stdout: Installing : 1:numpy-f2py-1.7.1-13.el7.x86_64 75/99 2021-02-10T11:11:23.842 INFO:teuthology.orchestra.run.gibba020.stdout: Installing : python2-apipkg-1.5-5.el7.noarch 68/99 2021-02-10T11:11:24.189 INFO:teuthology.orchestra.run.gibba020.stdout: Installing : python-execnet-1.7.1-3.el7.noarch 69/99 2021-02-10T11:11:24.446 INFO:teuthology.orchestra.run.gibba012.stdout: Installing : scipy-0.12.1-6.el7.x86_64 76/99 2021-02-10T11:11:24.513 INFO:teuthology.orchestra.run.gibba020.stdout: Installing : python2-remoto-1.0.0-1.el7.noarch 70/99 2021-02-10T11:11:24.765 INFO:teuthology.orchestra.run.gibba020.stdout: Installing : libverto-devel-0.2.5-4.el7.x86_64 71/99 2021-02-10T11:11:25.010 INFO:teuthology.orchestra.run.gibba012.stdout: Installing : 2:ceph-mgr-diskprediction-local-14.2.16-280.g962fac13.el 77/99 2021-02-10T11:11:25.381 INFO:teuthology.orchestra.run.gibba012.stdout: Installing : 1:openssl-devel-1.0.2k-21.el7_9.x86_64 78/99 2021-02-10T11:11:25.675 INFO:teuthology.orchestra.run.gibba020.stdout: Installing : libsepol-devel-2.5-10.el7.x86_64 72/99 2021-02-10T11:11:25.680 INFO:teuthology.orchestra.run.gibba012.stdout: Installing : 2:ceph-mgr-ssh-14.2.16-280.g962fac13.el7.noarch 79/99 2021-02-10T11:11:25.974 INFO:teuthology.orchestra.run.gibba020.stdout: Installing : libselinux-devel-2.5-15.el7.x86_64 73/99 2021-02-10T11:11:25.985 INFO:teuthology.orchestra.run.gibba012.stdout: Installing : 2:ceph-mgr-diskprediction-cloud-14.2.16-280.g962fac13.el 80/99 2021-02-10T11:11:26.223 INFO:teuthology.orchestra.run.gibba012.stdout: Installing : 2:ceph-14.2.16-280.g962fac13.el7.x86_64 81/99 2021-02-10T11:11:26.297 INFO:teuthology.orchestra.run.gibba020.stdout: Installing : krb5-devel-1.15.1-50.el7.x86_64 74/99 2021-02-10T11:11:27.028 INFO:teuthology.orchestra.run.gibba012.stdout: Installing : 2:ceph-mgr-rook-14.2.16-280.g962fac13.el7.noarch 82/99 2021-02-10T11:11:27.440 INFO:teuthology.orchestra.run.gibba012.stdout: Installing : 2:ceph-mgr-dashboard-14.2.16-280.g962fac13.el7.noarch 83/99 2021-02-10T11:11:27.607 INFO:teuthology.orchestra.run.gibba020.stdout: Installing : 1:numpy-f2py-1.7.1-13.el7.x86_64 75/99 2021-02-10T11:11:27.739 INFO:teuthology.orchestra.run.gibba012.stdout: Installing : 2:python3-cephfs-14.2.16-280.g962fac13.el7.x86_64 84/99 2021-02-10T11:11:28.228 INFO:teuthology.orchestra.run.gibba020.stdout: Installing : scipy-0.12.1-6.el7.x86_64 76/99 2021-02-10T11:11:28.402 INFO:teuthology.orchestra.run.gibba012.stdout: Installing : 2:libcephfs-devel-14.2.16-280.g962fac13.el7.x86_64 85/99 2021-02-10T11:11:28.746 INFO:teuthology.orchestra.run.gibba020.stdout: Installing : 2:ceph-mgr-diskprediction-local-14.2.16-280.g962fac13.el 77/99 2021-02-10T11:11:29.035 INFO:teuthology.orchestra.run.gibba020.stdout: Installing : 1:openssl-devel-1.0.2k-21.el7_9.x86_64 78/99 2021-02-10T11:11:29.286 INFO:teuthology.orchestra.run.gibba020.stdout: Installing : 2:ceph-mgr-ssh-14.2.16-280.g962fac13.el7.noarch 79/99 2021-02-10T11:11:29.583 INFO:teuthology.orchestra.run.gibba020.stdout: Installing : 2:ceph-mgr-diskprediction-cloud-14.2.16-280.g962fac13.el 80/99 2021-02-10T11:11:29.835 INFO:teuthology.orchestra.run.gibba020.stdout: Installing : 2:ceph-14.2.16-280.g962fac13.el7.x86_64 81/99 2021-02-10T11:11:30.537 INFO:teuthology.orchestra.run.gibba020.stdout: Installing : 2:ceph-mgr-rook-14.2.16-280.g962fac13.el7.noarch 82/99 2021-02-10T11:11:30.888 INFO:teuthology.orchestra.run.gibba020.stdout: Installing : 2:ceph-mgr-dashboard-14.2.16-280.g962fac13.el7.noarch 83/99 2021-02-10T11:11:31.161 INFO:teuthology.orchestra.run.gibba012.stdout: Installing : 2:ceph-radosgw-14.2.16-280.g962fac13.el7.x86_64 86/99 2021-02-10T11:11:31.162 INFO:teuthology.orchestra.run.gibba020.stdout: Installing : 2:python3-cephfs-14.2.16-280.g962fac13.el7.x86_64 84/99 2021-02-10T11:11:31.467 INFO:teuthology.orchestra.run.gibba012.stdout: Installing : 2:ceph-test-14.2.16-280.g962fac13.el7.x86_64 87/99 2021-02-10T11:11:31.849 INFO:teuthology.orchestra.run.gibba020.stdout: Installing : 2:libcephfs-devel-14.2.16-280.g962fac13.el7.x86_64 85/99 2021-02-10T11:11:31.942 INFO:teuthology.orchestra.run.gibba012.stdout: Installing : 2:python-ceph-compat-14.2.16-280.g962fac13.el7.x86_64 88/99 2021-02-10T11:11:32.217 INFO:teuthology.orchestra.run.gibba012.stdout: Installing : 2:rbd-fuse-14.2.16-280.g962fac13.el7.x86_64 89/99 2021-02-10T11:11:32.481 INFO:teuthology.orchestra.run.gibba012.stdout: Installing : 2:ceph-fuse-14.2.16-280.g962fac13.el7.x86_64 90/99 2021-02-10T11:11:32.900 INFO:teuthology.orchestra.run.gibba012.stdout: Installing : elfutils-libelf-devel-0.176-5.el7.x86_64 91/99 2021-02-10T11:11:33.303 INFO:teuthology.orchestra.run.gibba012.stdout: Updating : 1:openssl-1.0.2k-21.el7_9.x86_64 92/99 2021-02-10T11:11:33.683 INFO:teuthology.orchestra.run.gibba012.stdout: Installing : bison-3.0.4-2.el7.x86_64 93/99 2021-02-10T11:11:34.036 INFO:teuthology.orchestra.run.gibba012.stdout: Installing : flex-2.5.37-6.el7.x86_64 94/99 2021-02-10T11:11:34.366 INFO:teuthology.orchestra.run.gibba012.stdout: Cleanup : 1:openssl-1.0.2k-19.el7.x86_64 95/99 2021-02-10T11:11:34.579 INFO:teuthology.orchestra.run.gibba020.stdout: Installing : 2:ceph-radosgw-14.2.16-280.g962fac13.el7.x86_64 86/99 2021-02-10T11:11:34.734 INFO:teuthology.orchestra.run.gibba012.stdout: Cleanup : 1:openssl-libs-1.0.2k-19.el7.x86_64 96/99 2021-02-10T11:11:34.840 INFO:teuthology.orchestra.run.gibba020.stdout: Installing : 2:ceph-test-14.2.16-280.g962fac13.el7.x86_64 87/99 2021-02-10T11:11:35.049 INFO:teuthology.orchestra.run.gibba012.stdout: Cleanup : 1:librbd1-10.2.5-4.el7.x86_64 97/99 2021-02-10T11:11:35.052 INFO:teuthology.orchestra.run.gibba020.stdout: Installing : 2:python-ceph-compat-14.2.16-280.g962fac13.el7.x86_64 88/99 2021-02-10T11:11:35.300 INFO:teuthology.orchestra.run.gibba012.stdout: Cleanup : 1:librados2-10.2.5-4.el7.x86_64 98/99 2021-02-10T11:11:35.675 INFO:teuthology.orchestra.run.gibba020.stdout: Installing : 2:rbd-fuse-14.2.16-280.g962fac13.el7.x86_64 89/99 2021-02-10T11:11:35.714 INFO:teuthology.orchestra.run.gibba012.stdout: Cleanup : zlib-1.2.7-18.el7.x86_64 99/99 2021-02-10T11:11:35.715 INFO:teuthology.orchestra.run.gibba012.stdout:Loaded plugins: fastestmirror, langpacks, priorities, product-id, subscription- 2021-02-10T11:11:35.715 INFO:teuthology.orchestra.run.gibba012.stdout: : manager 2021-02-10T11:11:35.904 INFO:teuthology.orchestra.run.gibba020.stdout: Installing : 2:ceph-fuse-14.2.16-280.g962fac13.el7.x86_64 90/99 2021-02-10T11:11:36.237 INFO:teuthology.orchestra.run.gibba020.stdout: Installing : elfutils-libelf-devel-0.176-5.el7.x86_64 91/99 2021-02-10T11:11:36.572 INFO:teuthology.orchestra.run.gibba012.stdout: Verifying : 2:libradosstriper1-14.2.16-280.g962fac13.el7.x86_64 1/99 2021-02-10T11:11:36.653 INFO:teuthology.orchestra.run.gibba020.stdout: Updating : 1:openssl-1.0.2k-21.el7_9.x86_64 92/99 2021-02-10T11:11:36.974 INFO:teuthology.orchestra.run.gibba020.stdout: Installing : bison-3.0.4-2.el7.x86_64 93/99 2021-02-10T11:11:37.225 INFO:teuthology.orchestra.run.gibba020.stdout: Installing : flex-2.5.37-6.el7.x86_64 94/99 2021-02-10T11:11:37.487 INFO:teuthology.orchestra.run.gibba020.stdout: Cleanup : 1:openssl-1.0.2k-19.el7.x86_64 95/99 2021-02-10T11:11:37.507 INFO:teuthology.orchestra.run.gibba012.stdout: Verifying : 2:ceph-selinux-14.2.16-280.g962fac13.el7.x86_64 2/99 2021-02-10T11:11:37.616 INFO:teuthology.orchestra.run.gibba012.stdout: Verifying : 2:python-rgw-14.2.16-280.g962fac13.el7.x86_64 3/99 2021-02-10T11:11:37.717 INFO:teuthology.orchestra.run.gibba012.stdout: Verifying : suitesparse-4.0.2-10.el7.x86_64 4/99 2021-02-10T11:11:37.829 INFO:teuthology.orchestra.run.gibba020.stdout: Cleanup : 1:openssl-libs-1.0.2k-19.el7.x86_64 96/99 2021-02-10T11:11:37.851 INFO:teuthology.orchestra.run.gibba012.stdout: Verifying : jq-1.6-2.el7.x86_64 5/99 2021-02-10T11:11:37.976 INFO:teuthology.orchestra.run.gibba012.stdout: Verifying : python-routes-1.13-2.el7.noarch 6/99 2021-02-10T11:11:38.078 INFO:teuthology.orchestra.run.gibba020.stdout: Cleanup : 1:librbd1-10.2.5-4.el7.x86_64 97/99 2021-02-10T11:11:38.127 INFO:teuthology.orchestra.run.gibba012.stdout: Verifying : 1:numpy-f2py-1.7.1-13.el7.x86_64 7/99 2021-02-10T11:11:38.260 INFO:teuthology.orchestra.run.gibba012.stdout: Verifying : python-urllib3-1.10.2-7.el7.noarch 8/99 2021-02-10T11:11:38.337 INFO:teuthology.orchestra.run.gibba020.stdout: Cleanup : 1:librados2-10.2.5-4.el7.x86_64 98/99 2021-02-10T11:11:38.419 INFO:teuthology.orchestra.run.gibba012.stdout: Verifying : libsepol-devel-2.5-10.el7.x86_64 9/99 2021-02-10T11:11:38.545 INFO:teuthology.orchestra.run.gibba012.stdout: Verifying : python-mako-0.8.1-2.el7.noarch 10/99 2021-02-10T11:11:38.678 INFO:teuthology.orchestra.run.gibba012.stdout: Verifying : libverto-devel-0.2.5-4.el7.x86_64 11/99 2021-02-10T11:11:38.743 INFO:teuthology.orchestra.run.gibba020.stdout: Cleanup : zlib-1.2.7-18.el7.x86_64 99/99 2021-02-10T11:11:38.744 INFO:teuthology.orchestra.run.gibba020.stdout:Loaded plugins: fastestmirror, langpacks, priorities, product-id, subscription- 2021-02-10T11:11:38.744 INFO:teuthology.orchestra.run.gibba020.stdout: : manager 2021-02-10T11:11:38.779 INFO:teuthology.orchestra.run.gibba012.stdout: Verifying : 2:libcephfs2-14.2.16-280.g962fac13.el7.x86_64 12/99 2021-02-10T11:11:38.862 INFO:teuthology.orchestra.run.gibba012.stdout: Verifying : 2:python-rbd-14.2.16-280.g962fac13.el7.x86_64 13/99 2021-02-10T11:11:38.988 INFO:teuthology.orchestra.run.gibba012.stdout: Verifying : 1:openssl-1.0.2k-21.el7_9.x86_64 14/99 2021-02-10T11:11:39.072 INFO:teuthology.orchestra.run.gibba012.stdout: Verifying : python2-apipkg-1.5-5.el7.noarch 15/99 2021-02-10T11:11:39.147 INFO:teuthology.orchestra.run.gibba012.stdout: Verifying : python-markupsafe-0.11-10.el7.x86_64 16/99 2021-02-10T11:11:39.264 INFO:teuthology.orchestra.run.gibba012.stdout: Verifying : python-webob-1.2.3-7.el7.noarch 17/99 2021-02-10T11:11:39.349 INFO:teuthology.orchestra.run.gibba012.stdout: Verifying : python-repoze-lru-0.4-3.el7.noarch 18/99 2021-02-10T11:11:39.462 INFO:teuthology.orchestra.run.gibba020.stdout: Verifying : 2:libradosstriper1-14.2.16-280.g962fac13.el7.x86_64 1/99 2021-02-10T11:11:39.483 INFO:teuthology.orchestra.run.gibba012.stdout: Verifying : 1:openssl-libs-1.0.2k-21.el7_9.x86_64 19/99 2021-02-10T11:11:39.587 INFO:teuthology.orchestra.run.gibba020.stdout: Verifying : 2:ceph-selinux-14.2.16-280.g962fac13.el7.x86_64 2/99 2021-02-10T11:11:39.617 INFO:teuthology.orchestra.run.gibba012.stdout: Verifying : python-requests-2.6.0-10.el7.noarch 20/99 2021-02-10T11:11:39.704 INFO:teuthology.orchestra.run.gibba020.stdout: Verifying : 2:python-rgw-14.2.16-280.g962fac13.el7.x86_64 3/99 2021-02-10T11:11:39.744 INFO:teuthology.orchestra.run.gibba012.stdout: Verifying : python-cherrypy-3.2.2-4.el7.noarch 21/99 2021-02-10T11:11:39.805 INFO:teuthology.orchestra.run.gibba020.stdout: Verifying : suitesparse-4.0.2-10.el7.x86_64 4/99 2021-02-10T11:11:39.859 INFO:teuthology.orchestra.run.gibba012.stdout: Verifying : python-beaker-1.5.4-10.el7.noarch 22/99 2021-02-10T11:11:39.889 INFO:teuthology.orchestra.run.gibba020.stdout: Verifying : jq-1.6-2.el7.x86_64 5/99 2021-02-10T11:11:39.969 INFO:teuthology.orchestra.run.gibba012.stdout: Verifying : 1:python2-kubernetes-11.0.0-6.el7.noarch 23/99 2021-02-10T11:11:39.981 INFO:teuthology.orchestra.run.gibba020.stdout: Verifying : python-routes-1.13-2.el7.noarch 6/99 2021-02-10T11:11:40.082 INFO:teuthology.orchestra.run.gibba020.stdout: Verifying : 1:numpy-f2py-1.7.1-13.el7.x86_64 7/99 2021-02-10T11:11:40.110 INFO:teuthology.orchestra.run.gibba012.stdout: Verifying : userspace-rcu-0.7.16-1.el7.x86_64 24/99 2021-02-10T11:11:40.182 INFO:teuthology.orchestra.run.gibba020.stdout: Verifying : python-urllib3-1.10.2-7.el7.noarch 8/99 2021-02-10T11:11:40.227 INFO:teuthology.orchestra.run.gibba012.stdout: Verifying : 2:ceph-osd-14.2.16-280.g962fac13.el7.x86_64 25/99 2021-02-10T11:11:40.319 INFO:teuthology.orchestra.run.gibba020.stdout: Verifying : libsepol-devel-2.5-10.el7.x86_64 9/99 2021-02-10T11:11:40.369 INFO:teuthology.orchestra.run.gibba012.stdout: Verifying : socat-1.7.3.2-2.el7.x86_64 26/99 2021-02-10T11:11:40.486 INFO:teuthology.orchestra.run.gibba012.stdout: Verifying : python2-remoto-1.0.0-1.el7.noarch 27/99 2021-02-10T11:11:40.628 INFO:teuthology.orchestra.run.gibba012.stdout: Verifying : python-simplegeneric-0.8-7.el7.noarch 28/99 2021-02-10T11:11:40.737 INFO:teuthology.orchestra.run.gibba012.stdout: Verifying : keyutils-libs-devel-1.5.8-3.el7.x86_64 29/99 2021-02-10T11:11:40.829 INFO:teuthology.orchestra.run.gibba012.stdout: Verifying : 2:ceph-mgr-diskprediction-cloud-14.2.16-280.g962fac13.el 30/99 2021-02-10T11:11:40.909 INFO:teuthology.orchestra.run.gibba020.stdout: Verifying : python-mako-0.8.1-2.el7.noarch 10/99 2021-02-10T11:11:40.938 INFO:teuthology.orchestra.run.gibba012.stdout: Verifying : 2:ceph-fuse-14.2.16-280.g962fac13.el7.x86_64 31/99 2021-02-10T11:11:41.018 INFO:teuthology.orchestra.run.gibba020.stdout: Verifying : libverto-devel-0.2.5-4.el7.x86_64 11/99 2021-02-10T11:11:41.039 INFO:teuthology.orchestra.run.gibba012.stdout: Verifying : 2:ceph-14.2.16-280.g962fac13.el7.x86_64 32/99 2021-02-10T11:11:41.106 INFO:teuthology.orchestra.run.gibba012.stdout: Verifying : python-singledispatch-3.4.0.2-2.el7.noarch 33/99 2021-02-10T11:11:41.143 INFO:teuthology.orchestra.run.gibba020.stdout: Verifying : 2:libcephfs2-14.2.16-280.g962fac13.el7.x86_64 12/99 2021-02-10T11:11:41.231 INFO:teuthology.orchestra.run.gibba012.stdout: Verifying : python2-requests-oauthlib-0.8.0-5.el7.noarch 34/99 2021-02-10T11:11:41.252 INFO:teuthology.orchestra.run.gibba020.stdout: Verifying : 2:python-rbd-14.2.16-280.g962fac13.el7.x86_64 13/99 2021-02-10T11:11:41.315 INFO:teuthology.orchestra.run.gibba012.stdout: Verifying : 2:ceph-mds-14.2.16-280.g962fac13.el7.x86_64 35/99 2021-02-10T11:11:41.369 INFO:teuthology.orchestra.run.gibba020.stdout: Verifying : 1:openssl-1.0.2k-21.el7_9.x86_64 14/99 2021-02-10T11:11:41.443 INFO:teuthology.orchestra.run.gibba012.stdout: Verifying : python-paste-1.7.5.1-9.20111221hg1498.el7.noarch 36/99 2021-02-10T11:11:41.503 INFO:teuthology.orchestra.run.gibba020.stdout: Verifying : python2-apipkg-1.5-5.el7.noarch 15/99 2021-02-10T11:11:41.550 INFO:teuthology.orchestra.run.gibba012.stdout: Verifying : python2-rsa-3.4.2-2.el7.noarch 37/99 2021-02-10T11:11:41.611 INFO:teuthology.orchestra.run.gibba020.stdout: Verifying : python-markupsafe-0.11-10.el7.x86_64 16/99 2021-02-10T11:11:41.659 INFO:teuthology.orchestra.run.gibba012.stdout: Verifying : 2:ceph-mgr-ssh-14.2.16-280.g962fac13.el7.noarch 38/99 2021-02-10T11:11:41.720 INFO:teuthology.orchestra.run.gibba020.stdout: Verifying : python-webob-1.2.3-7.el7.noarch 17/99 2021-02-10T11:11:41.801 INFO:teuthology.orchestra.run.gibba012.stdout: Verifying : python-jwcrypto-0.4.2-1.el7.noarch 39/99 2021-02-10T11:11:41.829 INFO:teuthology.orchestra.run.gibba020.stdout: Verifying : python-repoze-lru-0.4-3.el7.noarch 18/99 2021-02-10T11:11:41.918 INFO:teuthology.orchestra.run.gibba012.stdout: Verifying : python-werkzeug-0.9.1-2.el7.noarch 40/99 2021-02-10T11:11:41.929 INFO:teuthology.orchestra.run.gibba020.stdout: Verifying : 1:openssl-libs-1.0.2k-21.el7_9.x86_64 19/99 2021-02-10T11:11:42.030 INFO:teuthology.orchestra.run.gibba020.stdout: Verifying : python-requests-2.6.0-10.el7.noarch 20/99 2021-02-10T11:11:42.060 INFO:teuthology.orchestra.run.gibba012.stdout: Verifying : elfutils-libelf-devel-0.176-5.el7.x86_64 41/99 2021-02-10T11:11:42.106 INFO:teuthology.orchestra.run.gibba020.stdout: Verifying : python-cherrypy-3.2.2-4.el7.noarch 21/99 2021-02-10T11:11:42.177 INFO:teuthology.orchestra.run.gibba012.stdout: Verifying : python2-pyasn1-modules-0.1.9-7.el7.noarch 42/99 2021-02-10T11:11:42.190 INFO:teuthology.orchestra.run.gibba020.stdout: Verifying : python-beaker-1.5.4-10.el7.noarch 22/99 2021-02-10T11:11:42.282 INFO:teuthology.orchestra.run.gibba020.stdout: Verifying : 1:python2-kubernetes-11.0.0-6.el7.noarch 23/99 2021-02-10T11:11:42.374 INFO:teuthology.orchestra.run.gibba020.stdout: Verifying : userspace-rcu-0.7.16-1.el7.x86_64 24/99 2021-02-10T11:11:42.396 INFO:teuthology.orchestra.run.gibba012.stdout: Verifying : python-prettytable-0.7.2-3.el7.noarch 43/99 2021-02-10T11:11:42.466 INFO:teuthology.orchestra.run.gibba020.stdout: Verifying : 2:ceph-osd-14.2.16-280.g962fac13.el7.x86_64 25/99 2021-02-10T11:11:42.505 INFO:teuthology.orchestra.run.gibba012.stdout: Verifying : librabbitmq-0.8.0-3.el7.x86_64 44/99 2021-02-10T11:11:42.550 INFO:teuthology.orchestra.run.gibba020.stdout: Verifying : socat-1.7.3.2-2.el7.x86_64 26/99 2021-02-10T11:11:42.589 INFO:teuthology.orchestra.run.gibba012.stdout: Verifying : 2:ceph-base-14.2.16-280.g962fac13.el7.x86_64 45/99 2021-02-10T11:11:42.642 INFO:teuthology.orchestra.run.gibba020.stdout: Verifying : python2-remoto-1.0.0-1.el7.noarch 27/99 2021-02-10T11:11:42.714 INFO:teuthology.orchestra.run.gibba012.stdout: Verifying : scipy-0.12.1-6.el7.x86_64 46/99 2021-02-10T11:11:42.743 INFO:teuthology.orchestra.run.gibba020.stdout: Verifying : python-simplegeneric-0.8-7.el7.noarch 28/99 2021-02-10T11:11:42.798 INFO:teuthology.orchestra.run.gibba012.stdout: Verifying : 2:python-ceph-argparse-14.2.16-280.g962fac13.el7.x86_64 47/99 2021-02-10T11:11:42.868 INFO:teuthology.orchestra.run.gibba020.stdout: Verifying : keyutils-libs-devel-1.5.8-3.el7.x86_64 29/99 2021-02-10T11:11:42.915 INFO:teuthology.orchestra.run.gibba012.stdout: Verifying : libselinux-devel-2.5-15.el7.x86_64 48/99 2021-02-10T11:11:42.977 INFO:teuthology.orchestra.run.gibba020.stdout: Verifying : 2:ceph-mgr-diskprediction-cloud-14.2.16-280.g962fac13.el 30/99 2021-02-10T11:11:43.008 INFO:teuthology.orchestra.run.gibba012.stdout: Verifying : zlib-1.2.7-19.el7_9.x86_64 49/99 2021-02-10T11:11:43.086 INFO:teuthology.orchestra.run.gibba020.stdout: Verifying : 2:ceph-fuse-14.2.16-280.g962fac13.el7.x86_64 31/99 2021-02-10T11:11:43.108 INFO:teuthology.orchestra.run.gibba012.stdout: Verifying : python2-oauthlib-2.0.1-8.el7.noarch 50/99 2021-02-10T11:11:43.203 INFO:teuthology.orchestra.run.gibba020.stdout: Verifying : 2:ceph-14.2.16-280.g962fac13.el7.x86_64 32/99 2021-02-10T11:11:43.251 INFO:teuthology.orchestra.run.gibba012.stdout: Verifying : pcre-devel-8.32-17.el7.x86_64 51/99 2021-02-10T11:11:43.312 INFO:teuthology.orchestra.run.gibba020.stdout: Verifying : python-singledispatch-3.4.0.2-2.el7.noarch 33/99 2021-02-10T11:11:43.368 INFO:teuthology.orchestra.run.gibba012.stdout: Verifying : 2:ceph-radosgw-14.2.16-280.g962fac13.el7.x86_64 52/99 2021-02-10T11:11:43.441 INFO:teuthology.orchestra.run.gibba020.stdout: Verifying : python2-requests-oauthlib-0.8.0-5.el7.noarch 34/99 2021-02-10T11:11:43.510 INFO:teuthology.orchestra.run.gibba012.stdout: Verifying : 2:python-cephfs-14.2.16-280.g962fac13.el7.x86_64 53/99 2021-02-10T11:11:43.554 INFO:teuthology.orchestra.run.gibba020.stdout: Verifying : 2:ceph-mds-14.2.16-280.g962fac13.el7.x86_64 35/99 2021-02-10T11:11:43.627 INFO:teuthology.orchestra.run.gibba012.stdout: Verifying : 2:libcephfs-devel-14.2.16-280.g962fac13.el7.x86_64 54/99 2021-02-10T11:11:43.647 INFO:teuthology.orchestra.run.gibba020.stdout: Verifying : python-paste-1.7.5.1-9.20111221hg1498.el7.noarch 36/99 2021-02-10T11:11:43.731 INFO:teuthology.orchestra.run.gibba020.stdout: Verifying : python2-rsa-3.4.2-2.el7.noarch 37/99 2021-02-10T11:11:43.786 INFO:teuthology.orchestra.run.gibba012.stdout: Verifying : 2:python-rados-14.2.16-280.g962fac13.el7.x86_64 55/99 2021-02-10T11:11:43.814 INFO:teuthology.orchestra.run.gibba020.stdout: Verifying : 2:ceph-mgr-ssh-14.2.16-280.g962fac13.el7.noarch 38/99 2021-02-10T11:11:43.911 INFO:teuthology.orchestra.run.gibba012.stdout: Verifying : 2:librados2-14.2.16-280.g962fac13.el7.x86_64 56/99 2021-02-10T11:11:43.914 INFO:teuthology.orchestra.run.gibba020.stdout: Verifying : python-jwcrypto-0.4.2-1.el7.noarch 39/99 2021-02-10T11:11:44.007 INFO:teuthology.orchestra.run.gibba020.stdout: Verifying : python-werkzeug-0.9.1-2.el7.noarch 40/99 2021-02-10T11:11:44.053 INFO:teuthology.orchestra.run.gibba012.stdout: Verifying : librdkafka-0.11.5-1.el7.x86_64 57/99 2021-02-10T11:11:44.099 INFO:teuthology.orchestra.run.gibba020.stdout: Verifying : elfutils-libelf-devel-0.176-5.el7.x86_64 41/99 2021-02-10T11:11:44.154 INFO:teuthology.orchestra.run.gibba012.stdout: Verifying : flex-2.5.37-6.el7.x86_64 58/99 2021-02-10T11:11:44.183 INFO:teuthology.orchestra.run.gibba020.stdout: Verifying : python2-pyasn1-modules-0.1.9-7.el7.noarch 42/99 2021-02-10T11:11:44.230 INFO:teuthology.orchestra.run.gibba012.stdout: Verifying : python2-certifi-2018.10.15-5.el7.noarch 59/99 2021-02-10T11:11:44.258 INFO:teuthology.orchestra.run.gibba020.stdout: Verifying : python-prettytable-0.7.2-3.el7.noarch 43/99 2021-02-10T11:11:44.347 INFO:teuthology.orchestra.run.gibba012.stdout: Verifying : 2:python3-ceph-argparse-14.2.16-280.g962fac13.el7.x86_64 60/99 2021-02-10T11:11:44.359 INFO:teuthology.orchestra.run.gibba020.stdout: Verifying : librabbitmq-0.8.0-3.el7.x86_64 44/99 2021-02-10T11:11:44.439 INFO:teuthology.orchestra.run.gibba012.stdout: Verifying : 2:ceph-mgr-14.2.16-280.g962fac13.el7.x86_64 61/99 2021-02-10T11:11:44.459 INFO:teuthology.orchestra.run.gibba020.stdout: Verifying : 2:ceph-base-14.2.16-280.g962fac13.el7.x86_64 45/99 2021-02-10T11:11:44.531 INFO:teuthology.orchestra.run.gibba012.stdout: Verifying : 2:librgw2-14.2.16-280.g962fac13.el7.x86_64 62/99 2021-02-10T11:11:44.569 INFO:teuthology.orchestra.run.gibba020.stdout: Verifying : scipy-0.12.1-6.el7.x86_64 46/99 2021-02-10T11:11:44.648 INFO:teuthology.orchestra.run.gibba012.stdout: Verifying : krb5-devel-1.15.1-50.el7.x86_64 63/99 2021-02-10T11:11:44.686 INFO:teuthology.orchestra.run.gibba020.stdout: Verifying : 2:python-ceph-argparse-14.2.16-280.g962fac13.el7.x86_64 47/99 2021-02-10T11:11:44.740 INFO:teuthology.orchestra.run.gibba012.stdout: Verifying : 2:ceph-test-14.2.16-280.g962fac13.el7.x86_64 64/99 2021-02-10T11:11:44.795 INFO:teuthology.orchestra.run.gibba020.stdout: Verifying : libselinux-devel-2.5-15.el7.x86_64 48/99 2021-02-10T11:11:44.832 INFO:teuthology.orchestra.run.gibba012.stdout: Verifying : python-webtest-1.3.4-6.el7.noarch 65/99 2021-02-10T11:11:44.912 INFO:teuthology.orchestra.run.gibba020.stdout: Verifying : zlib-1.2.7-19.el7_9.x86_64 49/99 2021-02-10T11:11:44.966 INFO:teuthology.orchestra.run.gibba012.stdout: Verifying : libcom_err-devel-1.42.9-19.el7.x86_64 66/99 2021-02-10T11:11:45.029 INFO:teuthology.orchestra.run.gibba020.stdout: Verifying : python2-oauthlib-2.0.1-8.el7.noarch 50/99 2021-02-10T11:11:45.076 INFO:teuthology.orchestra.run.gibba012.stdout: Verifying : 2:rbd-fuse-14.2.16-280.g962fac13.el7.x86_64 67/99 2021-02-10T11:11:45.138 INFO:teuthology.orchestra.run.gibba020.stdout: Verifying : pcre-devel-8.32-17.el7.x86_64 51/99 2021-02-10T11:11:45.218 INFO:teuthology.orchestra.run.gibba012.stdout: Verifying : 2:librbd1-14.2.16-280.g962fac13.el7.x86_64 68/99 2021-02-10T11:11:45.238 INFO:teuthology.orchestra.run.gibba020.stdout: Verifying : 2:ceph-radosgw-14.2.16-280.g962fac13.el7.x86_64 52/99 2021-02-10T11:11:45.322 INFO:teuthology.orchestra.run.gibba020.stdout: Verifying : 2:python-cephfs-14.2.16-280.g962fac13.el7.x86_64 53/99 2021-02-10T11:11:45.335 INFO:teuthology.orchestra.run.gibba012.stdout: Verifying : liboath-2.6.2-1.el7.x86_64 69/99 2021-02-10T11:11:45.485 INFO:teuthology.orchestra.run.gibba012.stdout: Verifying : python2-six-1.9.0-0.el7.noarch 70/99 2021-02-10T11:11:45.525 INFO:teuthology.orchestra.run.gibba020.stdout: Verifying : 2:libcephfs-devel-14.2.16-280.g962fac13.el7.x86_64 54/99 2021-02-10T11:11:45.602 INFO:teuthology.orchestra.run.gibba012.stdout: Verifying : 2:ceph-mgr-diskprediction-local-14.2.16-280.g962fac13.el 71/99 2021-02-10T11:11:45.761 INFO:teuthology.orchestra.run.gibba012.stdout: Verifying : python2-bcrypt-3.1.6-2.el7.x86_64 72/99 2021-02-10T11:11:45.887 INFO:teuthology.orchestra.run.gibba012.stdout: Verifying : python-websocket-client-0.56.0-3.git3c25814.el7.noarch 73/99 2021-02-10T11:11:45.917 INFO:teuthology.orchestra.run.gibba020.stdout: Verifying : 2:python-rados-14.2.16-280.g962fac13.el7.x86_64 55/99 2021-02-10T11:11:46.021 INFO:teuthology.orchestra.run.gibba012.stdout: Verifying : 2:ceph-mgr-rook-14.2.16-280.g962fac13.el7.noarch 74/99 2021-02-10T11:11:46.026 INFO:teuthology.orchestra.run.gibba020.stdout: Verifying : 2:librados2-14.2.16-280.g962fac13.el7.x86_64 56/99 2021-02-10T11:11:46.113 INFO:teuthology.orchestra.run.gibba012.stdout: Verifying : 2:librados-devel-14.2.16-280.g962fac13.el7.x86_64 75/99 2021-02-10T11:11:46.143 INFO:teuthology.orchestra.run.gibba020.stdout: Verifying : librdkafka-0.11.5-1.el7.x86_64 57/99 2021-02-10T11:11:46.205 INFO:teuthology.orchestra.run.gibba012.stdout: Verifying : libbabeltrace-1.2.4-3.el7.x86_64 76/99 2021-02-10T11:11:46.260 INFO:teuthology.orchestra.run.gibba020.stdout: Verifying : flex-2.5.37-6.el7.x86_64 58/99 2021-02-10T11:11:46.322 INFO:teuthology.orchestra.run.gibba012.stdout: Verifying : 2:ceph-mgr-dashboard-14.2.16-280.g962fac13.el7.noarch 77/99 2021-02-10T11:11:46.370 INFO:teuthology.orchestra.run.gibba020.stdout: Verifying : python2-certifi-2018.10.15-5.el7.noarch 59/99 2021-02-10T11:11:46.398 INFO:teuthology.orchestra.run.gibba012.stdout: Verifying : 2:python3-rados-14.2.16-280.g962fac13.el7.x86_64 78/99 2021-02-10T11:11:46.478 INFO:teuthology.orchestra.run.gibba020.stdout: Verifying : 2:python3-ceph-argparse-14.2.16-280.g962fac13.el7.x86_64 60/99 2021-02-10T11:11:46.481 INFO:teuthology.orchestra.run.gibba012.stdout: Verifying : python-pecan-0.4.5-2.el7.noarch 79/99 2021-02-10T11:11:46.596 INFO:teuthology.orchestra.run.gibba020.stdout: Verifying : 2:ceph-mgr-14.2.16-280.g962fac13.el7.x86_64 61/99 2021-02-10T11:11:46.607 INFO:teuthology.orchestra.run.gibba012.stdout: Verifying : 2:python-ceph-compat-14.2.16-280.g962fac13.el7.x86_64 80/99 2021-02-10T11:11:46.699 INFO:teuthology.orchestra.run.gibba012.stdout: Verifying : 2:ceph-common-14.2.16-280.g962fac13.el7.x86_64 81/99 2021-02-10T11:11:46.704 INFO:teuthology.orchestra.run.gibba020.stdout: Verifying : 2:librgw2-14.2.16-280.g962fac13.el7.x86_64 62/99 2021-02-10T11:11:46.805 INFO:teuthology.orchestra.run.gibba020.stdout: Verifying : krb5-devel-1.15.1-50.el7.x86_64 63/99 2021-02-10T11:11:46.832 INFO:teuthology.orchestra.run.gibba012.stdout: Verifying : 2:python3-cephfs-14.2.16-280.g962fac13.el7.x86_64 82/99 2021-02-10T11:11:46.889 INFO:teuthology.orchestra.run.gibba020.stdout: Verifying : 2:ceph-test-14.2.16-280.g962fac13.el7.x86_64 64/99 2021-02-10T11:11:46.933 INFO:teuthology.orchestra.run.gibba012.stdout: Verifying : 2:ceph-grafana-dashboards-14.2.16-280.g962fac13.el7.noar 83/99 2021-02-10T11:11:46.972 INFO:teuthology.orchestra.run.gibba020.stdout: Verifying : python-webtest-1.3.4-6.el7.noarch 65/99 2021-02-10T11:11:47.043 INFO:teuthology.orchestra.run.gibba012.stdout: Verifying : python-execnet-1.7.1-3.el7.noarch 84/99 2021-02-10T11:11:47.056 INFO:teuthology.orchestra.run.gibba020.stdout: Verifying : libcom_err-devel-1.42.9-19.el7.x86_64 66/99 2021-02-10T11:11:47.148 INFO:teuthology.orchestra.run.gibba020.stdout: Verifying : 2:rbd-fuse-14.2.16-280.g962fac13.el7.x86_64 67/99 2021-02-10T11:11:47.193 INFO:teuthology.orchestra.run.gibba012.stdout: Verifying : lttng-ust-2.4.1-4.el7.x86_64 85/99 2021-02-10T11:11:47.257 INFO:teuthology.orchestra.run.gibba020.stdout: Verifying : 2:librbd1-14.2.16-280.g962fac13.el7.x86_64 68/99 2021-02-10T11:11:47.383 INFO:teuthology.orchestra.run.gibba020.stdout: Verifying : liboath-2.6.2-1.el7.x86_64 69/99 2021-02-10T11:11:47.475 INFO:teuthology.orchestra.run.gibba020.stdout: Verifying : python2-six-1.9.0-0.el7.noarch 70/99 2021-02-10T11:11:47.567 INFO:teuthology.orchestra.run.gibba020.stdout: Verifying : 2:ceph-mgr-diskprediction-local-14.2.16-280.g962fac13.el 71/99 2021-02-10T11:11:47.659 INFO:teuthology.orchestra.run.gibba020.stdout: Verifying : python2-bcrypt-3.1.6-2.el7.x86_64 72/99 2021-02-10T11:11:47.759 INFO:teuthology.orchestra.run.gibba020.stdout: Verifying : python-websocket-client-0.56.0-3.git3c25814.el7.noarch 73/99 2021-02-10T11:11:47.877 INFO:teuthology.orchestra.run.gibba020.stdout: Verifying : 2:ceph-mgr-rook-14.2.16-280.g962fac13.el7.noarch 74/99 2021-02-10T11:11:47.985 INFO:teuthology.orchestra.run.gibba020.stdout: Verifying : 2:librados-devel-14.2.16-280.g962fac13.el7.x86_64 75/99 2021-02-10T11:11:48.095 INFO:teuthology.orchestra.run.gibba020.stdout: Verifying : libbabeltrace-1.2.4-3.el7.x86_64 76/99 2021-02-10T11:11:48.186 INFO:teuthology.orchestra.run.gibba012.stdout: Verifying : 2:ceph-mon-14.2.16-280.g962fac13.el7.x86_64 86/99 2021-02-10T11:11:48.212 INFO:teuthology.orchestra.run.gibba020.stdout: Verifying : 2:ceph-mgr-dashboard-14.2.16-280.g962fac13.el7.noarch 77/99 2021-02-10T11:11:48.303 INFO:teuthology.orchestra.run.gibba012.stdout: Verifying : bison-3.0.4-2.el7.x86_64 87/99 2021-02-10T11:11:48.321 INFO:teuthology.orchestra.run.gibba020.stdout: Verifying : 2:python3-rados-14.2.16-280.g962fac13.el7.x86_64 78/99 2021-02-10T11:11:48.413 INFO:teuthology.orchestra.run.gibba020.stdout: Verifying : python-pecan-0.4.5-2.el7.noarch 79/99 2021-02-10T11:11:48.445 INFO:teuthology.orchestra.run.gibba012.stdout: Verifying : 1:openssl-devel-1.0.2k-21.el7_9.x86_64 88/99 2021-02-10T11:11:48.497 INFO:teuthology.orchestra.run.gibba020.stdout: Verifying : 2:python-ceph-compat-14.2.16-280.g962fac13.el7.x86_64 80/99 2021-02-10T11:11:48.554 INFO:teuthology.orchestra.run.gibba012.stdout: Verifying : python-cachetools-1.0.3-1.el7.noarch 89/99 2021-02-10T11:11:48.581 INFO:teuthology.orchestra.run.gibba020.stdout: Verifying : 2:ceph-common-14.2.16-280.g962fac13.el7.x86_64 81/99 2021-02-10T11:11:48.673 INFO:teuthology.orchestra.run.gibba020.stdout: Verifying : 2:python3-cephfs-14.2.16-280.g962fac13.el7.x86_64 82/99 2021-02-10T11:11:48.679 INFO:teuthology.orchestra.run.gibba012.stdout: Verifying : python-tempita-0.5.1-6.el7.noarch 90/99 2021-02-10T11:11:48.765 INFO:teuthology.orchestra.run.gibba020.stdout: Verifying : 2:ceph-grafana-dashboards-14.2.16-280.g962fac13.el7.noar 83/99 2021-02-10T11:11:48.780 INFO:teuthology.orchestra.run.gibba012.stdout: Verifying : zlib-devel-1.2.7-19.el7_9.x86_64 91/99 2021-02-10T11:11:48.864 INFO:teuthology.orchestra.run.gibba012.stdout: Verifying : 1:python2-google-auth-1.1.1-5.el7.noarch 92/99 2021-02-10T11:11:48.866 INFO:teuthology.orchestra.run.gibba020.stdout: Verifying : python-execnet-1.7.1-3.el7.noarch 84/99 2021-02-10T11:11:48.958 INFO:teuthology.orchestra.run.gibba020.stdout: Verifying : lttng-ust-2.4.1-4.el7.x86_64 85/99 2021-02-10T11:11:49.166 INFO:teuthology.orchestra.run.gibba012.stdout: Verifying : tbb-4.1-9.20130314.el7.x86_64 93/99 2021-02-10T11:11:49.166 INFO:teuthology.orchestra.run.gibba012.stdout: Verifying : oniguruma-6.8.2-1.el7.x86_64 94/99 2021-02-10T11:11:49.166 INFO:teuthology.orchestra.run.gibba012.stdout: Verifying : 1:librbd1-10.2.5-4.el7.x86_64 95/99 2021-02-10T11:11:49.166 INFO:teuthology.orchestra.run.gibba012.stdout: Verifying : zlib-1.2.7-18.el7.x86_64 96/99 2021-02-10T11:11:49.167 INFO:teuthology.orchestra.run.gibba012.stdout: Verifying : 1:openssl-libs-1.0.2k-19.el7.x86_64 97/99 2021-02-10T11:11:49.167 INFO:teuthology.orchestra.run.gibba012.stdout: Verifying : 1:openssl-1.0.2k-19.el7.x86_64 98/99 2021-02-10T11:11:49.167 INFO:teuthology.orchestra.run.gibba020.stdout: Verifying : 2:ceph-mon-14.2.16-280.g962fac13.el7.x86_64 86/99 2021-02-10T11:11:49.167 INFO:teuthology.orchestra.run.gibba020.stdout: Verifying : bison-3.0.4-2.el7.x86_64 87/99 2021-02-10T11:11:49.251 INFO:teuthology.orchestra.run.gibba020.stdout: Verifying : 1:openssl-devel-1.0.2k-21.el7_9.x86_64 88/99 2021-02-10T11:11:49.341 INFO:teuthology.orchestra.run.gibba012.stdout: Verifying : 1:librados2-10.2.5-4.el7.x86_64 99/99 2021-02-10T11:11:49.341 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:11:49.341 INFO:teuthology.orchestra.run.gibba012.stdout:Installed: 2021-02-10T11:11:49.341 INFO:teuthology.orchestra.run.gibba012.stdout: bison.x86_64 0:3.0.4-2.el7 2021-02-10T11:11:49.341 INFO:teuthology.orchestra.run.gibba012.stdout: ceph.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:11:49.341 INFO:teuthology.orchestra.run.gibba012.stdout: ceph-fuse.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:11:49.341 INFO:teuthology.orchestra.run.gibba012.stdout: ceph-mgr.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:11:49.341 INFO:teuthology.orchestra.run.gibba012.stdout: ceph-mgr-dashboard.noarch 2:14.2.16-280.g962fac13.el7 2021-02-10T11:11:49.341 INFO:teuthology.orchestra.run.gibba012.stdout: ceph-mgr-diskprediction-cloud.noarch 2:14.2.16-280.g962fac13.el7 2021-02-10T11:11:49.342 INFO:teuthology.orchestra.run.gibba012.stdout: ceph-mgr-diskprediction-local.noarch 2:14.2.16-280.g962fac13.el7 2021-02-10T11:11:49.342 INFO:teuthology.orchestra.run.gibba012.stdout: ceph-mgr-rook.noarch 2:14.2.16-280.g962fac13.el7 2021-02-10T11:11:49.342 INFO:teuthology.orchestra.run.gibba012.stdout: ceph-mgr-ssh.noarch 2:14.2.16-280.g962fac13.el7 2021-02-10T11:11:49.342 INFO:teuthology.orchestra.run.gibba012.stdout: ceph-radosgw.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:11:49.342 INFO:teuthology.orchestra.run.gibba012.stdout: ceph-test.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:11:49.342 INFO:teuthology.orchestra.run.gibba012.stdout: elfutils-libelf-devel.x86_64 0:0.176-5.el7 2021-02-10T11:11:49.342 INFO:teuthology.orchestra.run.gibba012.stdout: flex.x86_64 0:2.5.37-6.el7 2021-02-10T11:11:49.342 INFO:teuthology.orchestra.run.gibba012.stdout: libcephfs-devel.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:11:49.342 INFO:teuthology.orchestra.run.gibba012.stdout: libcephfs2.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:11:49.342 INFO:teuthology.orchestra.run.gibba012.stdout: openssl-devel.x86_64 1:1.0.2k-21.el7_9 2021-02-10T11:11:49.342 INFO:teuthology.orchestra.run.gibba012.stdout: python-ceph-compat.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:11:49.343 INFO:teuthology.orchestra.run.gibba012.stdout: python3-cephfs.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:11:49.343 INFO:teuthology.orchestra.run.gibba012.stdout: rbd-fuse.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:11:49.343 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:11:49.343 INFO:teuthology.orchestra.run.gibba012.stdout:Dependency Installed: 2021-02-10T11:11:49.343 INFO:teuthology.orchestra.run.gibba012.stdout: ceph-base.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:11:49.343 INFO:teuthology.orchestra.run.gibba012.stdout: ceph-common.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:11:49.343 INFO:teuthology.orchestra.run.gibba012.stdout: ceph-grafana-dashboards.noarch 2:14.2.16-280.g962fac13.el7 2021-02-10T11:11:49.343 INFO:teuthology.orchestra.run.gibba012.stdout: ceph-mds.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:11:49.343 INFO:teuthology.orchestra.run.gibba012.stdout: ceph-mon.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:11:49.343 INFO:teuthology.orchestra.run.gibba012.stdout: ceph-osd.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:11:49.343 INFO:teuthology.orchestra.run.gibba012.stdout: ceph-selinux.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:11:49.343 INFO:teuthology.orchestra.run.gibba012.stdout: jq.x86_64 0:1.6-2.el7 2021-02-10T11:11:49.343 INFO:teuthology.orchestra.run.gibba012.stdout: keyutils-libs-devel.x86_64 0:1.5.8-3.el7 2021-02-10T11:11:49.344 INFO:teuthology.orchestra.run.gibba012.stdout: krb5-devel.x86_64 0:1.15.1-50.el7 2021-02-10T11:11:49.344 INFO:teuthology.orchestra.run.gibba012.stdout: libbabeltrace.x86_64 0:1.2.4-3.el7 2021-02-10T11:11:49.344 INFO:teuthology.orchestra.run.gibba012.stdout: libcom_err-devel.x86_64 0:1.42.9-19.el7 2021-02-10T11:11:49.344 INFO:teuthology.orchestra.run.gibba012.stdout: liboath.x86_64 0:2.6.2-1.el7 2021-02-10T11:11:49.344 INFO:teuthology.orchestra.run.gibba012.stdout: librabbitmq.x86_64 0:0.8.0-3.el7 2021-02-10T11:11:49.344 INFO:teuthology.orchestra.run.gibba012.stdout: librados-devel.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:11:49.344 INFO:teuthology.orchestra.run.gibba012.stdout: libradosstriper1.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:11:49.344 INFO:teuthology.orchestra.run.gibba012.stdout: librdkafka.x86_64 0:0.11.5-1.el7 2021-02-10T11:11:49.344 INFO:teuthology.orchestra.run.gibba012.stdout: librgw2.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:11:49.344 INFO:teuthology.orchestra.run.gibba012.stdout: libselinux-devel.x86_64 0:2.5-15.el7 2021-02-10T11:11:49.344 INFO:teuthology.orchestra.run.gibba012.stdout: libsepol-devel.x86_64 0:2.5-10.el7 2021-02-10T11:11:49.344 INFO:teuthology.orchestra.run.gibba012.stdout: libverto-devel.x86_64 0:0.2.5-4.el7 2021-02-10T11:11:49.345 INFO:teuthology.orchestra.run.gibba012.stdout: lttng-ust.x86_64 0:2.4.1-4.el7 2021-02-10T11:11:49.345 INFO:teuthology.orchestra.run.gibba012.stdout: numpy-f2py.x86_64 1:1.7.1-13.el7 2021-02-10T11:11:49.345 INFO:teuthology.orchestra.run.gibba012.stdout: oniguruma.x86_64 0:6.8.2-1.el7 2021-02-10T11:11:49.345 INFO:teuthology.orchestra.run.gibba012.stdout: pcre-devel.x86_64 0:8.32-17.el7 2021-02-10T11:11:49.345 INFO:teuthology.orchestra.run.gibba012.stdout: python-beaker.noarch 0:1.5.4-10.el7 2021-02-10T11:11:49.345 INFO:teuthology.orchestra.run.gibba012.stdout: python-cachetools.noarch 0:1.0.3-1.el7 2021-02-10T11:11:49.345 INFO:teuthology.orchestra.run.gibba012.stdout: python-ceph-argparse.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:11:49.345 INFO:teuthology.orchestra.run.gibba012.stdout: python-cephfs.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:11:49.345 INFO:teuthology.orchestra.run.gibba012.stdout: python-cherrypy.noarch 0:3.2.2-4.el7 2021-02-10T11:11:49.345 INFO:teuthology.orchestra.run.gibba012.stdout: python-execnet.noarch 0:1.7.1-3.el7 2021-02-10T11:11:49.345 INFO:teuthology.orchestra.run.gibba012.stdout: python-jwcrypto.noarch 0:0.4.2-1.el7 2021-02-10T11:11:49.345 INFO:teuthology.orchestra.run.gibba012.stdout: python-mako.noarch 0:0.8.1-2.el7 2021-02-10T11:11:49.346 INFO:teuthology.orchestra.run.gibba012.stdout: python-markupsafe.x86_64 0:0.11-10.el7 2021-02-10T11:11:49.346 INFO:teuthology.orchestra.run.gibba012.stdout: python-paste.noarch 0:1.7.5.1-9.20111221hg1498.el7 2021-02-10T11:11:49.346 INFO:teuthology.orchestra.run.gibba012.stdout: python-pecan.noarch 0:0.4.5-2.el7 2021-02-10T11:11:49.346 INFO:teuthology.orchestra.run.gibba012.stdout: python-prettytable.noarch 0:0.7.2-3.el7 2021-02-10T11:11:49.346 INFO:teuthology.orchestra.run.gibba012.stdout: python-rados.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:11:49.346 INFO:teuthology.orchestra.run.gibba012.stdout: python-rbd.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:11:49.346 INFO:teuthology.orchestra.run.gibba012.stdout: python-repoze-lru.noarch 0:0.4-3.el7 2021-02-10T11:11:49.346 INFO:teuthology.orchestra.run.gibba012.stdout: python-requests.noarch 0:2.6.0-10.el7 2021-02-10T11:11:49.346 INFO:teuthology.orchestra.run.gibba012.stdout: python-rgw.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:11:49.346 INFO:teuthology.orchestra.run.gibba012.stdout: python-routes.noarch 0:1.13-2.el7 2021-02-10T11:11:49.346 INFO:teuthology.orchestra.run.gibba012.stdout: python-simplegeneric.noarch 0:0.8-7.el7 2021-02-10T11:11:49.346 INFO:teuthology.orchestra.run.gibba012.stdout: python-singledispatch.noarch 0:3.4.0.2-2.el7 2021-02-10T11:11:49.347 INFO:teuthology.orchestra.run.gibba012.stdout: python-tempita.noarch 0:0.5.1-6.el7 2021-02-10T11:11:49.347 INFO:teuthology.orchestra.run.gibba012.stdout: python-urllib3.noarch 0:1.10.2-7.el7 2021-02-10T11:11:49.347 INFO:teuthology.orchestra.run.gibba012.stdout: python-webob.noarch 0:1.2.3-7.el7 2021-02-10T11:11:49.347 INFO:teuthology.orchestra.run.gibba012.stdout: python-websocket-client.noarch 0:0.56.0-3.git3c25814.el7 2021-02-10T11:11:49.347 INFO:teuthology.orchestra.run.gibba012.stdout: python-webtest.noarch 0:1.3.4-6.el7 2021-02-10T11:11:49.347 INFO:teuthology.orchestra.run.gibba012.stdout: python-werkzeug.noarch 0:0.9.1-2.el7 2021-02-10T11:11:49.347 INFO:teuthology.orchestra.run.gibba012.stdout: python2-apipkg.noarch 0:1.5-5.el7 2021-02-10T11:11:49.347 INFO:teuthology.orchestra.run.gibba012.stdout: python2-bcrypt.x86_64 0:3.1.6-2.el7 2021-02-10T11:11:49.347 INFO:teuthology.orchestra.run.gibba012.stdout: python2-certifi.noarch 0:2018.10.15-5.el7 2021-02-10T11:11:49.347 INFO:teuthology.orchestra.run.gibba012.stdout: python2-google-auth.noarch 1:1.1.1-5.el7 2021-02-10T11:11:49.347 INFO:teuthology.orchestra.run.gibba012.stdout: python2-kubernetes.noarch 1:11.0.0-6.el7 2021-02-10T11:11:49.347 INFO:teuthology.orchestra.run.gibba012.stdout: python2-oauthlib.noarch 0:2.0.1-8.el7 2021-02-10T11:11:49.348 INFO:teuthology.orchestra.run.gibba012.stdout: python2-pyasn1-modules.noarch 0:0.1.9-7.el7 2021-02-10T11:11:49.348 INFO:teuthology.orchestra.run.gibba012.stdout: python2-remoto.noarch 0:1.0.0-1.el7 2021-02-10T11:11:49.348 INFO:teuthology.orchestra.run.gibba012.stdout: python2-requests-oauthlib.noarch 0:0.8.0-5.el7 2021-02-10T11:11:49.348 INFO:teuthology.orchestra.run.gibba012.stdout: python2-rsa.noarch 0:3.4.2-2.el7 2021-02-10T11:11:49.348 INFO:teuthology.orchestra.run.gibba012.stdout: python2-six.noarch 0:1.9.0-0.el7 2021-02-10T11:11:49.348 INFO:teuthology.orchestra.run.gibba012.stdout: python3-ceph-argparse.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:11:49.348 INFO:teuthology.orchestra.run.gibba012.stdout: python3-rados.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:11:49.348 INFO:teuthology.orchestra.run.gibba012.stdout: scipy.x86_64 0:0.12.1-6.el7 2021-02-10T11:11:49.348 INFO:teuthology.orchestra.run.gibba012.stdout: socat.x86_64 0:1.7.3.2-2.el7 2021-02-10T11:11:49.348 INFO:teuthology.orchestra.run.gibba012.stdout: suitesparse.x86_64 0:4.0.2-10.el7 2021-02-10T11:11:49.348 INFO:teuthology.orchestra.run.gibba012.stdout: tbb.x86_64 0:4.1-9.20130314.el7 2021-02-10T11:11:49.348 INFO:teuthology.orchestra.run.gibba012.stdout: userspace-rcu.x86_64 0:0.7.16-1.el7 2021-02-10T11:11:49.349 INFO:teuthology.orchestra.run.gibba012.stdout: zlib-devel.x86_64 0:1.2.7-19.el7_9 2021-02-10T11:11:49.349 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:11:49.349 INFO:teuthology.orchestra.run.gibba012.stdout:Updated: 2021-02-10T11:11:49.349 INFO:teuthology.orchestra.run.gibba012.stdout: librados2.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:11:49.349 INFO:teuthology.orchestra.run.gibba012.stdout: librbd1.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:11:49.349 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:11:49.349 INFO:teuthology.orchestra.run.gibba012.stdout:Dependency Updated: 2021-02-10T11:11:49.349 INFO:teuthology.orchestra.run.gibba012.stdout: openssl.x86_64 1:1.0.2k-21.el7_9 openssl-libs.x86_64 1:1.0.2k-21.el7_9 2021-02-10T11:11:49.349 INFO:teuthology.orchestra.run.gibba012.stdout: zlib.x86_64 0:1.2.7-19.el7_9 2021-02-10T11:11:49.349 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:11:49.349 INFO:teuthology.orchestra.run.gibba012.stdout:Complete! 2021-02-10T11:11:49.385 INFO:teuthology.orchestra.run.gibba020.stdout: Verifying : python-cachetools-1.0.3-1.el7.noarch 89/99 2021-02-10T11:11:49.468 DEBUG:teuthology.parallel:result is None 2021-02-10T11:11:49.493 INFO:teuthology.orchestra.run.gibba020.stdout: Verifying : python-tempita-0.5.1-6.el7.noarch 90/99 2021-02-10T11:11:49.610 INFO:teuthology.orchestra.run.gibba020.stdout: Verifying : zlib-devel-1.2.7-19.el7_9.x86_64 91/99 2021-02-10T11:11:49.719 INFO:teuthology.orchestra.run.gibba020.stdout: Verifying : 1:python2-google-auth-1.1.1-5.el7.noarch 92/99 2021-02-10T11:11:49.829 INFO:teuthology.orchestra.run.gibba020.stdout: Verifying : tbb-4.1-9.20130314.el7.x86_64 93/99 2021-02-10T11:11:49.938 INFO:teuthology.orchestra.run.gibba020.stdout: Verifying : oniguruma-6.8.2-1.el7.x86_64 94/99 2021-02-10T11:11:49.939 INFO:teuthology.orchestra.run.gibba020.stdout: Verifying : 1:librbd1-10.2.5-4.el7.x86_64 95/99 2021-02-10T11:11:49.939 INFO:teuthology.orchestra.run.gibba020.stdout: Verifying : zlib-1.2.7-18.el7.x86_64 96/99 2021-02-10T11:11:49.939 INFO:teuthology.orchestra.run.gibba020.stdout: Verifying : 1:openssl-libs-1.0.2k-19.el7.x86_64 97/99 2021-02-10T11:11:49.939 INFO:teuthology.orchestra.run.gibba020.stdout: Verifying : 1:openssl-1.0.2k-19.el7.x86_64 98/99 2021-02-10T11:11:50.172 INFO:teuthology.orchestra.run.gibba020.stdout: Verifying : 1:librados2-10.2.5-4.el7.x86_64 99/99 2021-02-10T11:11:50.172 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:11:50.172 INFO:teuthology.orchestra.run.gibba020.stdout:Installed: 2021-02-10T11:11:50.173 INFO:teuthology.orchestra.run.gibba020.stdout: bison.x86_64 0:3.0.4-2.el7 2021-02-10T11:11:50.173 INFO:teuthology.orchestra.run.gibba020.stdout: ceph.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:11:50.173 INFO:teuthology.orchestra.run.gibba020.stdout: ceph-fuse.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:11:50.173 INFO:teuthology.orchestra.run.gibba020.stdout: ceph-mgr.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:11:50.173 INFO:teuthology.orchestra.run.gibba020.stdout: ceph-mgr-dashboard.noarch 2:14.2.16-280.g962fac13.el7 2021-02-10T11:11:50.173 INFO:teuthology.orchestra.run.gibba020.stdout: ceph-mgr-diskprediction-cloud.noarch 2:14.2.16-280.g962fac13.el7 2021-02-10T11:11:50.173 INFO:teuthology.orchestra.run.gibba020.stdout: ceph-mgr-diskprediction-local.noarch 2:14.2.16-280.g962fac13.el7 2021-02-10T11:11:50.173 INFO:teuthology.orchestra.run.gibba020.stdout: ceph-mgr-rook.noarch 2:14.2.16-280.g962fac13.el7 2021-02-10T11:11:50.173 INFO:teuthology.orchestra.run.gibba020.stdout: ceph-mgr-ssh.noarch 2:14.2.16-280.g962fac13.el7 2021-02-10T11:11:50.173 INFO:teuthology.orchestra.run.gibba020.stdout: ceph-radosgw.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:11:50.173 INFO:teuthology.orchestra.run.gibba020.stdout: ceph-test.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:11:50.173 INFO:teuthology.orchestra.run.gibba020.stdout: elfutils-libelf-devel.x86_64 0:0.176-5.el7 2021-02-10T11:11:50.173 INFO:teuthology.orchestra.run.gibba020.stdout: flex.x86_64 0:2.5.37-6.el7 2021-02-10T11:11:50.174 INFO:teuthology.orchestra.run.gibba020.stdout: libcephfs-devel.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:11:50.174 INFO:teuthology.orchestra.run.gibba020.stdout: libcephfs2.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:11:50.174 INFO:teuthology.orchestra.run.gibba020.stdout: openssl-devel.x86_64 1:1.0.2k-21.el7_9 2021-02-10T11:11:50.174 INFO:teuthology.orchestra.run.gibba020.stdout: python-ceph-compat.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:11:50.174 INFO:teuthology.orchestra.run.gibba020.stdout: python3-cephfs.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:11:50.174 INFO:teuthology.orchestra.run.gibba020.stdout: rbd-fuse.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:11:50.174 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:11:50.174 INFO:teuthology.orchestra.run.gibba020.stdout:Dependency Installed: 2021-02-10T11:11:50.174 INFO:teuthology.orchestra.run.gibba020.stdout: ceph-base.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:11:50.174 INFO:teuthology.orchestra.run.gibba020.stdout: ceph-common.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:11:50.174 INFO:teuthology.orchestra.run.gibba020.stdout: ceph-grafana-dashboards.noarch 2:14.2.16-280.g962fac13.el7 2021-02-10T11:11:50.174 INFO:teuthology.orchestra.run.gibba020.stdout: ceph-mds.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:11:50.174 INFO:teuthology.orchestra.run.gibba020.stdout: ceph-mon.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:11:50.175 INFO:teuthology.orchestra.run.gibba020.stdout: ceph-osd.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:11:50.175 INFO:teuthology.orchestra.run.gibba020.stdout: ceph-selinux.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:11:50.175 INFO:teuthology.orchestra.run.gibba020.stdout: jq.x86_64 0:1.6-2.el7 2021-02-10T11:11:50.175 INFO:teuthology.orchestra.run.gibba020.stdout: keyutils-libs-devel.x86_64 0:1.5.8-3.el7 2021-02-10T11:11:50.175 INFO:teuthology.orchestra.run.gibba020.stdout: krb5-devel.x86_64 0:1.15.1-50.el7 2021-02-10T11:11:50.175 INFO:teuthology.orchestra.run.gibba020.stdout: libbabeltrace.x86_64 0:1.2.4-3.el7 2021-02-10T11:11:50.175 INFO:teuthology.orchestra.run.gibba020.stdout: libcom_err-devel.x86_64 0:1.42.9-19.el7 2021-02-10T11:11:50.175 INFO:teuthology.orchestra.run.gibba020.stdout: liboath.x86_64 0:2.6.2-1.el7 2021-02-10T11:11:50.175 INFO:teuthology.orchestra.run.gibba020.stdout: librabbitmq.x86_64 0:0.8.0-3.el7 2021-02-10T11:11:50.175 INFO:teuthology.orchestra.run.gibba020.stdout: librados-devel.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:11:50.175 INFO:teuthology.orchestra.run.gibba020.stdout: libradosstriper1.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:11:50.175 INFO:teuthology.orchestra.run.gibba020.stdout: librdkafka.x86_64 0:0.11.5-1.el7 2021-02-10T11:11:50.175 INFO:teuthology.orchestra.run.gibba020.stdout: librgw2.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:11:50.176 INFO:teuthology.orchestra.run.gibba020.stdout: libselinux-devel.x86_64 0:2.5-15.el7 2021-02-10T11:11:50.176 INFO:teuthology.orchestra.run.gibba020.stdout: libsepol-devel.x86_64 0:2.5-10.el7 2021-02-10T11:11:50.176 INFO:teuthology.orchestra.run.gibba020.stdout: libverto-devel.x86_64 0:0.2.5-4.el7 2021-02-10T11:11:50.176 INFO:teuthology.orchestra.run.gibba020.stdout: lttng-ust.x86_64 0:2.4.1-4.el7 2021-02-10T11:11:50.176 INFO:teuthology.orchestra.run.gibba020.stdout: numpy-f2py.x86_64 1:1.7.1-13.el7 2021-02-10T11:11:50.176 INFO:teuthology.orchestra.run.gibba020.stdout: oniguruma.x86_64 0:6.8.2-1.el7 2021-02-10T11:11:50.176 INFO:teuthology.orchestra.run.gibba020.stdout: pcre-devel.x86_64 0:8.32-17.el7 2021-02-10T11:11:50.176 INFO:teuthology.orchestra.run.gibba020.stdout: python-beaker.noarch 0:1.5.4-10.el7 2021-02-10T11:11:50.176 INFO:teuthology.orchestra.run.gibba020.stdout: python-cachetools.noarch 0:1.0.3-1.el7 2021-02-10T11:11:50.176 INFO:teuthology.orchestra.run.gibba020.stdout: python-ceph-argparse.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:11:50.176 INFO:teuthology.orchestra.run.gibba020.stdout: python-cephfs.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:11:50.176 INFO:teuthology.orchestra.run.gibba020.stdout: python-cherrypy.noarch 0:3.2.2-4.el7 2021-02-10T11:11:50.176 INFO:teuthology.orchestra.run.gibba020.stdout: python-execnet.noarch 0:1.7.1-3.el7 2021-02-10T11:11:50.177 INFO:teuthology.orchestra.run.gibba020.stdout: python-jwcrypto.noarch 0:0.4.2-1.el7 2021-02-10T11:11:50.177 INFO:teuthology.orchestra.run.gibba020.stdout: python-mako.noarch 0:0.8.1-2.el7 2021-02-10T11:11:50.177 INFO:teuthology.orchestra.run.gibba020.stdout: python-markupsafe.x86_64 0:0.11-10.el7 2021-02-10T11:11:50.177 INFO:teuthology.orchestra.run.gibba020.stdout: python-paste.noarch 0:1.7.5.1-9.20111221hg1498.el7 2021-02-10T11:11:50.177 INFO:teuthology.orchestra.run.gibba020.stdout: python-pecan.noarch 0:0.4.5-2.el7 2021-02-10T11:11:50.177 INFO:teuthology.orchestra.run.gibba020.stdout: python-prettytable.noarch 0:0.7.2-3.el7 2021-02-10T11:11:50.177 INFO:teuthology.orchestra.run.gibba020.stdout: python-rados.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:11:50.177 INFO:teuthology.orchestra.run.gibba020.stdout: python-rbd.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:11:50.177 INFO:teuthology.orchestra.run.gibba020.stdout: python-repoze-lru.noarch 0:0.4-3.el7 2021-02-10T11:11:50.177 INFO:teuthology.orchestra.run.gibba020.stdout: python-requests.noarch 0:2.6.0-10.el7 2021-02-10T11:11:50.177 INFO:teuthology.orchestra.run.gibba020.stdout: python-rgw.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:11:50.177 INFO:teuthology.orchestra.run.gibba020.stdout: python-routes.noarch 0:1.13-2.el7 2021-02-10T11:11:50.177 INFO:teuthology.orchestra.run.gibba020.stdout: python-simplegeneric.noarch 0:0.8-7.el7 2021-02-10T11:11:50.178 INFO:teuthology.orchestra.run.gibba020.stdout: python-singledispatch.noarch 0:3.4.0.2-2.el7 2021-02-10T11:11:50.178 INFO:teuthology.orchestra.run.gibba020.stdout: python-tempita.noarch 0:0.5.1-6.el7 2021-02-10T11:11:50.178 INFO:teuthology.orchestra.run.gibba020.stdout: python-urllib3.noarch 0:1.10.2-7.el7 2021-02-10T11:11:50.178 INFO:teuthology.orchestra.run.gibba020.stdout: python-webob.noarch 0:1.2.3-7.el7 2021-02-10T11:11:50.178 INFO:teuthology.orchestra.run.gibba020.stdout: python-websocket-client.noarch 0:0.56.0-3.git3c25814.el7 2021-02-10T11:11:50.178 INFO:teuthology.orchestra.run.gibba020.stdout: python-webtest.noarch 0:1.3.4-6.el7 2021-02-10T11:11:50.178 INFO:teuthology.orchestra.run.gibba020.stdout: python-werkzeug.noarch 0:0.9.1-2.el7 2021-02-10T11:11:50.178 INFO:teuthology.orchestra.run.gibba020.stdout: python2-apipkg.noarch 0:1.5-5.el7 2021-02-10T11:11:50.178 INFO:teuthology.orchestra.run.gibba020.stdout: python2-bcrypt.x86_64 0:3.1.6-2.el7 2021-02-10T11:11:50.179 INFO:teuthology.orchestra.run.gibba020.stdout: python2-certifi.noarch 0:2018.10.15-5.el7 2021-02-10T11:11:50.179 INFO:teuthology.orchestra.run.gibba020.stdout: python2-google-auth.noarch 1:1.1.1-5.el7 2021-02-10T11:11:50.179 INFO:teuthology.orchestra.run.gibba020.stdout: python2-kubernetes.noarch 1:11.0.0-6.el7 2021-02-10T11:11:50.179 INFO:teuthology.orchestra.run.gibba020.stdout: python2-oauthlib.noarch 0:2.0.1-8.el7 2021-02-10T11:11:50.179 INFO:teuthology.orchestra.run.gibba020.stdout: python2-pyasn1-modules.noarch 0:0.1.9-7.el7 2021-02-10T11:11:50.179 INFO:teuthology.orchestra.run.gibba020.stdout: python2-remoto.noarch 0:1.0.0-1.el7 2021-02-10T11:11:50.179 INFO:teuthology.orchestra.run.gibba020.stdout: python2-requests-oauthlib.noarch 0:0.8.0-5.el7 2021-02-10T11:11:50.179 INFO:teuthology.orchestra.run.gibba020.stdout: python2-rsa.noarch 0:3.4.2-2.el7 2021-02-10T11:11:50.179 INFO:teuthology.orchestra.run.gibba020.stdout: python2-six.noarch 0:1.9.0-0.el7 2021-02-10T11:11:50.180 INFO:teuthology.orchestra.run.gibba020.stdout: python3-ceph-argparse.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:11:50.180 INFO:teuthology.orchestra.run.gibba020.stdout: python3-rados.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:11:50.180 INFO:teuthology.orchestra.run.gibba020.stdout: scipy.x86_64 0:0.12.1-6.el7 2021-02-10T11:11:50.180 INFO:teuthology.orchestra.run.gibba020.stdout: socat.x86_64 0:1.7.3.2-2.el7 2021-02-10T11:11:50.180 INFO:teuthology.orchestra.run.gibba020.stdout: suitesparse.x86_64 0:4.0.2-10.el7 2021-02-10T11:11:50.180 INFO:teuthology.orchestra.run.gibba020.stdout: tbb.x86_64 0:4.1-9.20130314.el7 2021-02-10T11:11:50.180 INFO:teuthology.orchestra.run.gibba020.stdout: userspace-rcu.x86_64 0:0.7.16-1.el7 2021-02-10T11:11:50.180 INFO:teuthology.orchestra.run.gibba020.stdout: zlib-devel.x86_64 0:1.2.7-19.el7_9 2021-02-10T11:11:50.180 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:11:50.181 INFO:teuthology.orchestra.run.gibba020.stdout:Updated: 2021-02-10T11:11:50.181 INFO:teuthology.orchestra.run.gibba020.stdout: librados2.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:11:50.181 INFO:teuthology.orchestra.run.gibba020.stdout: librbd1.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:11:50.181 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:11:50.181 INFO:teuthology.orchestra.run.gibba020.stdout:Dependency Updated: 2021-02-10T11:11:50.181 INFO:teuthology.orchestra.run.gibba020.stdout: openssl.x86_64 1:1.0.2k-21.el7_9 openssl-libs.x86_64 1:1.0.2k-21.el7_9 2021-02-10T11:11:50.182 INFO:teuthology.orchestra.run.gibba020.stdout: zlib.x86_64 0:1.2.7-19.el7_9 2021-02-10T11:11:50.182 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:11:50.182 INFO:teuthology.orchestra.run.gibba020.stdout:Complete! 2021-02-10T11:11:50.301 DEBUG:teuthology.parallel:result is None 2021-02-10T11:11:50.302 DEBUG:teuthology.orchestra.run.gibba012:> sudo lsb_release -is 2021-02-10T11:11:50.674 INFO:teuthology.orchestra.run.gibba012.stdout:RedHatEnterpriseServer 2021-02-10T11:11:51.131 DEBUG:teuthology.misc:System to be installed: RedHatEnterpriseServer 2021-02-10T11:11:51.131 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F7%2Fx86_64&sha1=962fac13b1a781b8c7d25c1eed50d93813b8f5f0 2021-02-10T11:11:51.311 DEBUG:teuthology.orchestra.run.gibba012:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2021-02-10T11:11:51.663 INFO:teuthology.orchestra.run.gibba012.stdout:14.2.16-280.g962fac13.el7 2021-02-10T11:11:51.663 INFO:teuthology.packaging:The installed version of ceph is 14.2.16-280.g962fac13.el7 2021-02-10T11:11:51.663 INFO:teuthology.task.install:The correct ceph version 14.2.16-280.g962fac13 is installed. 2021-02-10T11:11:51.664 DEBUG:teuthology.orchestra.run.gibba020:> sudo lsb_release -is 2021-02-10T11:11:52.034 INFO:teuthology.orchestra.run.gibba020.stdout:RedHatEnterpriseServer 2021-02-10T11:11:52.035 DEBUG:teuthology.misc:System to be installed: RedHatEnterpriseServer 2021-02-10T11:11:52.036 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F7%2Fx86_64&sha1=962fac13b1a781b8c7d25c1eed50d93813b8f5f0 2021-02-10T11:11:52.200 DEBUG:teuthology.orchestra.run.gibba020:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2021-02-10T11:11:52.554 INFO:teuthology.orchestra.run.gibba020.stdout:14.2.16-280.g962fac13.el7 2021-02-10T11:11:52.554 INFO:teuthology.packaging:The installed version of ceph is 14.2.16-280.g962fac13.el7 2021-02-10T11:11:52.554 INFO:teuthology.task.install:The correct ceph version 14.2.16-280.g962fac13 is installed. 2021-02-10T11:11:52.555 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2021-02-10T11:11:52.561 DEBUG:teuthology.orchestra.run.gibba012:> set -ex 2021-02-10T11:11:52.561 DEBUG:teuthology.orchestra.run.gibba012:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2021-02-10T11:11:52.907 DEBUG:teuthology.orchestra.run.gibba020:> set -ex 2021-02-10T11:11:52.907 DEBUG:teuthology.orchestra.run.gibba020:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2021-02-10T11:11:53.257 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2021-02-10T11:11:53.265 DEBUG:teuthology.orchestra.run.gibba012:> set -ex 2021-02-10T11:11:53.265 DEBUG:teuthology.orchestra.run.gibba012:> sudo dd of=/usr/bin/daemon-helper 2021-02-10T11:11:53.612 DEBUG:teuthology.orchestra.run.gibba012:> sudo chmod a=rx -- /usr/bin/daemon-helper 2021-02-10T11:11:53.961 DEBUG:teuthology.orchestra.run.gibba020:> set -ex 2021-02-10T11:11:53.961 DEBUG:teuthology.orchestra.run.gibba020:> sudo dd of=/usr/bin/daemon-helper 2021-02-10T11:11:54.307 DEBUG:teuthology.orchestra.run.gibba020:> sudo chmod a=rx -- /usr/bin/daemon-helper 2021-02-10T11:11:54.652 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2021-02-10T11:11:54.653 DEBUG:teuthology.orchestra.run.gibba012:> set -ex 2021-02-10T11:11:54.653 DEBUG:teuthology.orchestra.run.gibba012:> sudo dd of=/usr/bin/adjust-ulimits 2021-02-10T11:11:54.999 DEBUG:teuthology.orchestra.run.gibba012:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2021-02-10T11:11:55.341 DEBUG:teuthology.orchestra.run.gibba020:> set -ex 2021-02-10T11:11:55.341 DEBUG:teuthology.orchestra.run.gibba020:> sudo dd of=/usr/bin/adjust-ulimits 2021-02-10T11:11:55.719 DEBUG:teuthology.orchestra.run.gibba020:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2021-02-10T11:11:56.059 INFO:teuthology.run_tasks:Running task ceph... 2021-02-10T11:11:56.136 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2021-02-10T11:11:56.136 DEBUG:teuthology.orchestra.run.gibba012:> sudo chmod 777 /var/log/ceph 2021-02-10T11:11:56.140 DEBUG:teuthology.orchestra.run.gibba020:> sudo chmod 777 /var/log/ceph 2021-02-10T11:11:56.483 INFO:tasks.ceph:Disabling ceph logrotate... 2021-02-10T11:11:56.484 DEBUG:teuthology.orchestra.run.gibba012:> sudo rm -f -- /etc/logrotate.d/ceph 2021-02-10T11:11:56.487 DEBUG:teuthology.orchestra.run.gibba020:> sudo rm -f -- /etc/logrotate.d/ceph 2021-02-10T11:11:56.831 INFO:tasks.ceph:Creating extra log directories... 2021-02-10T11:11:56.831 DEBUG:teuthology.orchestra.run.gibba012:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2021-02-10T11:11:56.834 DEBUG:teuthology.orchestra.run.gibba020:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2021-02-10T11:11:57.177 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2021-02-10T11:11:57.180 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2021-02-10T11:11:57.181 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2021-02-10T11:11:57.182 DEBUG:teuthology.orchestra.run.gibba012:> set -ex 2021-02-10T11:11:57.182 DEBUG:teuthology.orchestra.run.gibba012:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2021-02-10T11:11:57.516 DEBUG:teuthology.orchestra.run.gibba012:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2021-02-10T11:11:58.311 DEBUG:teuthology.orchestra.run.gibba012:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2021-02-10T11:11:58.655 DEBUG:teuthology.orchestra.run.gibba020:> set -ex 2021-02-10T11:11:58.655 DEBUG:teuthology.orchestra.run.gibba020:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2021-02-10T11:11:58.989 DEBUG:teuthology.orchestra.run.gibba020:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2021-02-10T11:11:59.374 DEBUG:teuthology.orchestra.run.gibba020:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2021-02-10T11:11:59.717 INFO:tasks.ceph:Creating ceph cluster ceph... 2021-02-10T11:11:59.717 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': ''}, 'mds': {'debug mds': 20, 'debug ms': 1, 'mds bal frag': True, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180}, 'mgr': {'debug client': 10, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(TOO_FEW_PGS\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'OSD full dropping all updates', 'OSD near full', 'pausewr flag', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping', 'is full \\(reached quota', 'POOL_FULL', 'POOL_BACKFILLFULL'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2021-02-10T11:11:59.717 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/khiremat-2021-02-10_10:13:45-fs-wip-khiremat-mgr-authid-bp-7-nautilus-distro-basic-gibba/5872720', 'branch': 'wip-khiremat-mgr-authid-bp-7-nautilus', 'description': 'fs/basic_functional/{begin clusters/1-mds-4-client-coloc conf/{client mds mon osd} mount/fuse objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{rhel_7} tasks/volumes}', 'email': 'khiremat@redhat.com', 'first_in_suite': False, 'job_id': '5872720', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'gibba', 'name': 'khiremat-2021-02-10_10:13:45-fs-wip-khiremat-mgr-authid-bp-7-nautilus-distro-basic-gibba', 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 30}}, {'machine': {'disk': 200}}], 'os_type': 'rhel', 'os_version': '7.9', 'overrides': {'admin_socket': {'branch': 'wip-khiremat-mgr-authid-bp-7-nautilus'}, 'ceph': {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': ''}, 'mds': {'debug mds': 20, 'debug ms': 1, 'mds bal frag': True, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180}, 'mgr': {'debug client': 10, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(TOO_FEW_PGS\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'OSD full dropping all updates', 'OSD near full', 'pausewr flag', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping', 'is full \\(reached quota', 'POOL_FULL', 'POOL_BACKFILLFULL'], 'sha1': '962fac13b1a781b8c7d25c1eed50d93813b8f5f0'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'sha1': '962fac13b1a781b8c7d25c1eed50d93813b8f5f0'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-khiremat-mgr-authid-bp-7-nautilus', 'sha1': '962fac13b1a781b8c7d25c1eed50d93813b8f5f0'}}, 'owner': 'scheduled_khiremat@teuthology', 'priority': 50, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.1'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.2', 'client.3']], 'sha1': '962fac13b1a781b8c7d25c1eed50d93813b8f5f0', 'sleep_before_teardown': 0, 'suite': 'fs', 'suite_branch': 'wip-khiremat-mgr-authid-bp-7-nautilus', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_962fac13b1a781b8c7d25c1eed50d93813b8f5f0/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '962fac13b1a781b8c7d25c1eed50d93813b8f5f0', 'targets': {'gibba012.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBVcGlYALk4OspEafH8/Lx0pdzIft0iVC17OZulIeKaN1bVmyG4U/sfmwbmXgZGHPPrvbS1THXiFaWFuAH9p6q00b/d1A2Nolo5+s5U2itFFtDsrGLbMUF1u2zE3myTCQ1O1OT4a9Gw5b2xR9snaYROX22CW2NRwsLvFKZ8jisxqs8skGjHLu+Py7ce0O8h9d+iRfkTw+lqAt1CepBHLyTo2Sx3Q2R2VRTpDXZUKZWh0yVe7FMByIjG6YsmcVLMfN5jliHaQOfkuKiTGXuFfkdEJggUlpPMnuGIV89d4UfcIWOXGbAu/Gf6KPSISSQOvNNZehNEGXYuJ+x/DT/7Vgz', 'gibba020.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCzcUFKpDP/1cUZh0gR/RKn7JfrfMtmdGkiIGToKQ0PjQjhcx4NpKQnjSid8quwUQ2p1kQ2U0fblQeO6HaeyVZMCFSqjtum1hwl15M1xmlCmYmVzM1ll9hfCGPeQf4tAcXvyGkLN+Q/nksjyYXUuFdyy284r+DzSiEDWvJRhANuea7IJtw7G9lhrL+7Qzli4lY5sQZf8zg/XTqnV/omc2n42gFr6q+RfviLlGGIrbDt05YkuA3Ee4L8b4JJh7ylG/Npl+jZBoDQJCNdPBEXan0doeWQB6dJg2hSANf/inH4nzvYNTNUQCYNaUk8kor492ImJLpZzK6cbso2I+KqH+UD'}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell'], 'rpm': ['python3-cephfs']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel']}, 'sha1': '962fac13b1a781b8c7d25c1eed50d93813b8f5f0'}}, {'ceph': None}, {'ceph-fuse': None}, {'cephfs_test_runner': {'fail_on_skip': False, 'modules': ['tasks.cephfs.test_volumes']}}], 'teuthology_branch': 'master', 'timestamp': '2021-02-10_10:13:45', 'tube': 'gibba', 'user': 'khiremat', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.gibba.29737'} 2021-02-10T11:11:59.718 DEBUG:teuthology.orchestra.run.gibba012:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2021-02-10T11:11:59.721 DEBUG:teuthology.orchestra.run.gibba020:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2021-02-10T11:12:00.055 DEBUG:teuthology.orchestra.run.gibba012:> sudo install -d -m0777 -- /var/run/ceph 2021-02-10T11:12:00.058 DEBUG:teuthology.orchestra.run.gibba020:> sudo install -d -m0777 -- /var/run/ceph 2021-02-10T11:12:00.402 DEBUG:teuthology.orchestra.run.gibba012:> set -ex 2021-02-10T11:12:00.402 DEBUG:teuthology.orchestra.run.gibba012:> dd if=/scratch_devs of=/dev/stdout 2021-02-10T11:12:00.736 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2021-02-10T11:12:00.736 DEBUG:teuthology.orchestra.run.gibba012:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2021-02-10T11:12:01.070 INFO:teuthology.orchestra.run.gibba012.stdout: File: ‘/dev/vg_nvme/lv_1’ -> ‘../dm-0’ 2021-02-10T11:12:01.070 INFO:teuthology.orchestra.run.gibba012.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-02-10T11:12:01.070 INFO:teuthology.orchestra.run.gibba012.stdout:Device: 5h/5d Inode: 92610 Links: 1 2021-02-10T11:12:01.070 INFO:teuthology.orchestra.run.gibba012.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-02-10T11:12:01.070 INFO:teuthology.orchestra.run.gibba012.stdout:Context: system_u:object_r:device_t:s0 2021-02-10T11:12:01.070 INFO:teuthology.orchestra.run.gibba012.stdout:Access: 2021-02-10 11:11:28.604733118 +0000 2021-02-10T11:12:01.070 INFO:teuthology.orchestra.run.gibba012.stdout:Modify: 2021-02-10 11:08:48.703548815 +0000 2021-02-10T11:12:01.070 INFO:teuthology.orchestra.run.gibba012.stdout:Change: 2021-02-10 11:08:48.703548815 +0000 2021-02-10T11:12:01.070 INFO:teuthology.orchestra.run.gibba012.stdout: Birth: - 2021-02-10T11:12:01.079 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:12:01.079 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:12:01.079 INFO:teuthology.orchestra.run.gibba012.stderr:512 bytes (512 B) copied, 0.000169181 s, 3.0 MB/s 2021-02-10T11:12:01.081 DEBUG:teuthology.orchestra.run.gibba012:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2021-02-10T11:12:01.414 INFO:teuthology.orchestra.run.gibba012.stdout: File: ‘/dev/vg_nvme/lv_2’ -> ‘../dm-1’ 2021-02-10T11:12:01.415 INFO:teuthology.orchestra.run.gibba012.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-02-10T11:12:01.415 INFO:teuthology.orchestra.run.gibba012.stdout:Device: 5h/5d Inode: 96443 Links: 1 2021-02-10T11:12:01.415 INFO:teuthology.orchestra.run.gibba012.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-02-10T11:12:01.415 INFO:teuthology.orchestra.run.gibba012.stdout:Context: system_u:object_r:device_t:s0 2021-02-10T11:12:01.415 INFO:teuthology.orchestra.run.gibba012.stdout:Access: 2021-02-10 11:11:28.605733118 +0000 2021-02-10T11:12:01.415 INFO:teuthology.orchestra.run.gibba012.stdout:Modify: 2021-02-10 11:08:49.244539691 +0000 2021-02-10T11:12:01.415 INFO:teuthology.orchestra.run.gibba012.stdout:Change: 2021-02-10 11:08:49.244539691 +0000 2021-02-10T11:12:01.415 INFO:teuthology.orchestra.run.gibba012.stdout: Birth: - 2021-02-10T11:12:01.424 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:12:01.424 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:12:01.424 INFO:teuthology.orchestra.run.gibba012.stderr:512 bytes (512 B) copied, 0.000204709 s, 2.5 MB/s 2021-02-10T11:12:01.426 DEBUG:teuthology.orchestra.run.gibba012:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2021-02-10T11:12:01.760 INFO:teuthology.orchestra.run.gibba012.stdout: File: ‘/dev/vg_nvme/lv_3’ -> ‘../dm-2’ 2021-02-10T11:12:01.760 INFO:teuthology.orchestra.run.gibba012.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-02-10T11:12:01.760 INFO:teuthology.orchestra.run.gibba012.stdout:Device: 5h/5d Inode: 91965 Links: 1 2021-02-10T11:12:01.760 INFO:teuthology.orchestra.run.gibba012.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-02-10T11:12:01.760 INFO:teuthology.orchestra.run.gibba012.stdout:Context: system_u:object_r:device_t:s0 2021-02-10T11:12:01.760 INFO:teuthology.orchestra.run.gibba012.stdout:Access: 2021-02-10 11:11:28.605733118 +0000 2021-02-10T11:12:01.760 INFO:teuthology.orchestra.run.gibba012.stdout:Modify: 2021-02-10 11:08:49.767530871 +0000 2021-02-10T11:12:01.760 INFO:teuthology.orchestra.run.gibba012.stdout:Change: 2021-02-10 11:08:49.767530871 +0000 2021-02-10T11:12:01.760 INFO:teuthology.orchestra.run.gibba012.stdout: Birth: - 2021-02-10T11:12:01.769 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:12:01.769 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:12:01.769 INFO:teuthology.orchestra.run.gibba012.stderr:512 bytes (512 B) copied, 0.000192742 s, 2.7 MB/s 2021-02-10T11:12:01.771 DEBUG:teuthology.orchestra.run.gibba012:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2021-02-10T11:12:02.104 INFO:teuthology.orchestra.run.gibba012.stdout: File: ‘/dev/vg_nvme/lv_4’ -> ‘../dm-3’ 2021-02-10T11:12:02.105 INFO:teuthology.orchestra.run.gibba012.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-02-10T11:12:02.105 INFO:teuthology.orchestra.run.gibba012.stdout:Device: 5h/5d Inode: 92645 Links: 1 2021-02-10T11:12:02.105 INFO:teuthology.orchestra.run.gibba012.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-02-10T11:12:02.105 INFO:teuthology.orchestra.run.gibba012.stdout:Context: system_u:object_r:device_t:s0 2021-02-10T11:12:02.105 INFO:teuthology.orchestra.run.gibba012.stdout:Access: 2021-02-10 11:11:28.605733118 +0000 2021-02-10T11:12:02.105 INFO:teuthology.orchestra.run.gibba012.stdout:Modify: 2021-02-10 11:08:50.299521900 +0000 2021-02-10T11:12:02.105 INFO:teuthology.orchestra.run.gibba012.stdout:Change: 2021-02-10 11:08:50.299521900 +0000 2021-02-10T11:12:02.105 INFO:teuthology.orchestra.run.gibba012.stdout: Birth: - 2021-02-10T11:12:02.113 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:12:02.114 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:12:02.114 INFO:teuthology.orchestra.run.gibba012.stderr:512 bytes (512 B) copied, 0.000215246 s, 2.4 MB/s 2021-02-10T11:12:02.116 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2021-02-10T11:12:02.116 DEBUG:teuthology.orchestra.run.gibba020:> set -ex 2021-02-10T11:12:02.116 DEBUG:teuthology.orchestra.run.gibba020:> dd if=/scratch_devs of=/dev/stdout 2021-02-10T11:12:02.452 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2021-02-10T11:12:02.452 DEBUG:teuthology.orchestra.run.gibba020:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2021-02-10T11:12:02.784 INFO:teuthology.orchestra.run.gibba020.stdout: File: ‘/dev/vg_nvme/lv_1’ -> ‘../dm-0’ 2021-02-10T11:12:02.785 INFO:teuthology.orchestra.run.gibba020.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-02-10T11:12:02.785 INFO:teuthology.orchestra.run.gibba020.stdout:Device: 5h/5d Inode: 102495 Links: 1 2021-02-10T11:12:02.785 INFO:teuthology.orchestra.run.gibba020.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-02-10T11:12:02.785 INFO:teuthology.orchestra.run.gibba020.stdout:Context: system_u:object_r:device_t:s0 2021-02-10T11:12:02.785 INFO:teuthology.orchestra.run.gibba020.stdout:Access: 2021-02-10 11:11:32.017753365 +0000 2021-02-10T11:12:02.785 INFO:teuthology.orchestra.run.gibba020.stdout:Modify: 2021-02-10 11:08:47.901322493 +0000 2021-02-10T11:12:02.785 INFO:teuthology.orchestra.run.gibba020.stdout:Change: 2021-02-10 11:08:47.901322493 +0000 2021-02-10T11:12:02.785 INFO:teuthology.orchestra.run.gibba020.stdout: Birth: - 2021-02-10T11:12:02.794 INFO:teuthology.orchestra.run.gibba020.stderr:1+0 records in 2021-02-10T11:12:02.794 INFO:teuthology.orchestra.run.gibba020.stderr:1+0 records out 2021-02-10T11:12:02.794 INFO:teuthology.orchestra.run.gibba020.stderr:512 bytes (512 B) copied, 0.000212168 s, 2.4 MB/s 2021-02-10T11:12:02.796 DEBUG:teuthology.orchestra.run.gibba020:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2021-02-10T11:12:03.129 INFO:teuthology.orchestra.run.gibba020.stdout: File: ‘/dev/vg_nvme/lv_2’ -> ‘../dm-1’ 2021-02-10T11:12:03.130 INFO:teuthology.orchestra.run.gibba020.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-02-10T11:12:03.130 INFO:teuthology.orchestra.run.gibba020.stdout:Device: 5h/5d Inode: 89294 Links: 1 2021-02-10T11:12:03.130 INFO:teuthology.orchestra.run.gibba020.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-02-10T11:12:03.130 INFO:teuthology.orchestra.run.gibba020.stdout:Context: system_u:object_r:device_t:s0 2021-02-10T11:12:03.130 INFO:teuthology.orchestra.run.gibba020.stdout:Access: 2021-02-10 11:11:32.017753365 +0000 2021-02-10T11:12:03.130 INFO:teuthology.orchestra.run.gibba020.stdout:Modify: 2021-02-10 11:08:48.415314404 +0000 2021-02-10T11:12:03.130 INFO:teuthology.orchestra.run.gibba020.stdout:Change: 2021-02-10 11:08:48.415314404 +0000 2021-02-10T11:12:03.130 INFO:teuthology.orchestra.run.gibba020.stdout: Birth: - 2021-02-10T11:12:03.139 INFO:teuthology.orchestra.run.gibba020.stderr:1+0 records in 2021-02-10T11:12:03.140 INFO:teuthology.orchestra.run.gibba020.stderr:1+0 records out 2021-02-10T11:12:03.140 INFO:teuthology.orchestra.run.gibba020.stderr:512 bytes (512 B) copied, 0.000206817 s, 2.5 MB/s 2021-02-10T11:12:03.142 DEBUG:teuthology.orchestra.run.gibba020:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2021-02-10T11:12:03.474 INFO:teuthology.orchestra.run.gibba020.stdout: File: ‘/dev/vg_nvme/lv_3’ -> ‘../dm-2’ 2021-02-10T11:12:03.475 INFO:teuthology.orchestra.run.gibba020.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-02-10T11:12:03.475 INFO:teuthology.orchestra.run.gibba020.stdout:Device: 5h/5d Inode: 91815 Links: 1 2021-02-10T11:12:03.475 INFO:teuthology.orchestra.run.gibba020.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-02-10T11:12:03.475 INFO:teuthology.orchestra.run.gibba020.stdout:Context: system_u:object_r:device_t:s0 2021-02-10T11:12:03.475 INFO:teuthology.orchestra.run.gibba020.stdout:Access: 2021-02-10 11:11:32.017753365 +0000 2021-02-10T11:12:03.475 INFO:teuthology.orchestra.run.gibba020.stdout:Modify: 2021-02-10 11:08:48.951305970 +0000 2021-02-10T11:12:03.475 INFO:teuthology.orchestra.run.gibba020.stdout:Change: 2021-02-10 11:08:48.951305970 +0000 2021-02-10T11:12:03.475 INFO:teuthology.orchestra.run.gibba020.stdout: Birth: - 2021-02-10T11:12:03.484 INFO:teuthology.orchestra.run.gibba020.stderr:1+0 records in 2021-02-10T11:12:03.484 INFO:teuthology.orchestra.run.gibba020.stderr:1+0 records out 2021-02-10T11:12:03.484 INFO:teuthology.orchestra.run.gibba020.stderr:512 bytes (512 B) copied, 0.000177769 s, 2.9 MB/s 2021-02-10T11:12:03.486 DEBUG:teuthology.orchestra.run.gibba020:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2021-02-10T11:12:03.820 INFO:teuthology.orchestra.run.gibba020.stdout: File: ‘/dev/vg_nvme/lv_4’ -> ‘../dm-3’ 2021-02-10T11:12:03.820 INFO:teuthology.orchestra.run.gibba020.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-02-10T11:12:03.820 INFO:teuthology.orchestra.run.gibba020.stdout:Device: 5h/5d Inode: 89323 Links: 1 2021-02-10T11:12:03.820 INFO:teuthology.orchestra.run.gibba020.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-02-10T11:12:03.820 INFO:teuthology.orchestra.run.gibba020.stdout:Context: system_u:object_r:device_t:s0 2021-02-10T11:12:03.820 INFO:teuthology.orchestra.run.gibba020.stdout:Access: 2021-02-10 11:11:32.017753365 +0000 2021-02-10T11:12:03.820 INFO:teuthology.orchestra.run.gibba020.stdout:Modify: 2021-02-10 11:08:49.474297740 +0000 2021-02-10T11:12:03.820 INFO:teuthology.orchestra.run.gibba020.stdout:Change: 2021-02-10 11:08:49.474297740 +0000 2021-02-10T11:12:03.820 INFO:teuthology.orchestra.run.gibba020.stdout: Birth: - 2021-02-10T11:12:03.829 INFO:teuthology.orchestra.run.gibba020.stderr:1+0 records in 2021-02-10T11:12:03.829 INFO:teuthology.orchestra.run.gibba020.stderr:1+0 records out 2021-02-10T11:12:03.829 INFO:teuthology.orchestra.run.gibba020.stderr:512 bytes (512 B) copied, 0.000205645 s, 2.5 MB/s 2021-02-10T11:12:03.831 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2021-02-10T11:12:03.831 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@gibba012.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@gibba020.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2021-02-10T11:12:03.831 INFO:tasks.ceph:Generating config... 2021-02-10T11:12:03.833 INFO:tasks.ceph:[client] client mount timeout = 600 2021-02-10T11:12:03.833 INFO:tasks.ceph:[client] debug client = 20 2021-02-10T11:12:03.834 INFO:tasks.ceph:[client] debug ms = 1 2021-02-10T11:12:03.834 INFO:tasks.ceph:[client] pid file = 2021-02-10T11:12:03.834 INFO:tasks.ceph:[mds] debug mds = 20 2021-02-10T11:12:03.834 INFO:tasks.ceph:[mds] debug ms = 1 2021-02-10T11:12:03.834 INFO:tasks.ceph:[mds] mds bal frag = True 2021-02-10T11:12:03.834 INFO:tasks.ceph:[mds] mds bal fragment size max = 10000 2021-02-10T11:12:03.834 INFO:tasks.ceph:[mds] mds bal merge size = 5 2021-02-10T11:12:03.834 INFO:tasks.ceph:[mds] mds bal split bits = 3 2021-02-10T11:12:03.834 INFO:tasks.ceph:[mds] mds bal split size = 100 2021-02-10T11:12:03.834 INFO:tasks.ceph:[mds] mds debug frag = True 2021-02-10T11:12:03.834 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2021-02-10T11:12:03.834 INFO:tasks.ceph:[mds] mds op complaint time = 180 2021-02-10T11:12:03.834 INFO:tasks.ceph:[mds] mds verify scatter = True 2021-02-10T11:12:03.835 INFO:tasks.ceph:[mds] osd op complaint time = 180 2021-02-10T11:12:03.835 INFO:tasks.ceph:[mgr] debug client = 10 2021-02-10T11:12:03.835 INFO:tasks.ceph:[mgr] debug mgr = 20 2021-02-10T11:12:03.835 INFO:tasks.ceph:[mgr] debug ms = 1 2021-02-10T11:12:03.835 INFO:tasks.ceph:[mon] debug mon = 20 2021-02-10T11:12:03.835 INFO:tasks.ceph:[mon] debug ms = 1 2021-02-10T11:12:03.835 INFO:tasks.ceph:[mon] debug paxos = 20 2021-02-10T11:12:03.835 INFO:tasks.ceph:[mon] mon op complaint time = 120 2021-02-10T11:12:03.835 INFO:tasks.ceph:[osd] bdev async discard = True 2021-02-10T11:12:03.835 INFO:tasks.ceph:[osd] bdev enable discard = True 2021-02-10T11:12:03.835 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2021-02-10T11:12:03.835 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2021-02-10T11:12:03.835 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2021-02-10T11:12:03.835 INFO:tasks.ceph:[osd] debug bluefs = 20 2021-02-10T11:12:03.836 INFO:tasks.ceph:[osd] debug bluestore = 20 2021-02-10T11:12:03.836 INFO:tasks.ceph:[osd] debug filestore = 20 2021-02-10T11:12:03.836 INFO:tasks.ceph:[osd] debug journal = 20 2021-02-10T11:12:03.836 INFO:tasks.ceph:[osd] debug ms = 20 2021-02-10T11:12:03.836 INFO:tasks.ceph:[osd] debug osd = 25 2021-02-10T11:12:03.836 INFO:tasks.ceph:[osd] debug rocksdb = 10 2021-02-10T11:12:03.836 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2021-02-10T11:12:03.836 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2021-02-10T11:12:03.836 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2021-02-10T11:12:03.836 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2021-02-10T11:12:03.836 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2021-02-10T11:12:03.836 INFO:tasks.ceph:[osd] osd op complaint time = 180 2021-02-10T11:12:03.837 INFO:tasks.ceph:Setting up mon.a... 2021-02-10T11:12:03.837 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2021-02-10T11:12:04.251 INFO:teuthology.orchestra.run.gibba012.stdout:creating /etc/ceph/ceph.keyring 2021-02-10T11:12:04.253 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2021-02-10T11:12:04.621 DEBUG:teuthology.orchestra.run.gibba012:> sudo chmod 0644 /etc/ceph/ceph.keyring 2021-02-10T11:12:04.964 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.2.112'), ('mon.b', '172.21.2.120'), ('mon.c', '[v2:172.21.2.120:3301,v1:172.21.2.120:6790]')] 2021-02-10T11:12:04.964 DEBUG:teuthology.orchestra.run.gibba012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool --create --clobber --enable-all-features --add a 172.21.2.112 --add b 172.21.2.120 --addv c '[v2:172.21.2.120:3301,v1:172.21.2.120:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2021-02-10T11:12:05.329 INFO:teuthology.orchestra.run.gibba012.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2021-02-10T11:12:05.329 INFO:teuthology.orchestra.run.gibba012.stdout:monmaptool: generated fsid f14f004e-0699-4584-ad81-3754258493de 2021-02-10T11:12:05.329 INFO:teuthology.orchestra.run.gibba012.stdout:epoch 0 2021-02-10T11:12:05.329 INFO:teuthology.orchestra.run.gibba012.stdout:fsid f14f004e-0699-4584-ad81-3754258493de 2021-02-10T11:12:05.329 INFO:teuthology.orchestra.run.gibba012.stdout:last_changed 2021-02-10 11:12:05.330424 2021-02-10T11:12:05.329 INFO:teuthology.orchestra.run.gibba012.stdout:created 2021-02-10 11:12:05.330424 2021-02-10T11:12:05.329 INFO:teuthology.orchestra.run.gibba012.stdout:min_mon_release 0 (unknown) 2021-02-10T11:12:05.329 INFO:teuthology.orchestra.run.gibba012.stdout:0: [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] mon.a 2021-02-10T11:12:05.329 INFO:teuthology.orchestra.run.gibba012.stdout:1: [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] mon.b 2021-02-10T11:12:05.330 INFO:teuthology.orchestra.run.gibba012.stdout:2: [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] mon.c 2021-02-10T11:12:05.330 INFO:teuthology.orchestra.run.gibba012.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2021-02-10T11:12:05.331 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID f14f004e-0699-4584-ad81-3754258493de... 2021-02-10T11:12:05.332 DEBUG:teuthology.orchestra.run.gibba012:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2021-02-10T11:12:05.335 DEBUG:teuthology.orchestra.run.gibba020:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2021-02-10T11:12:05.692 INFO:teuthology.orchestra.run.gibba012.stdout:[global] 2021-02-10T11:12:05.693 INFO:teuthology.orchestra.run.gibba012.stdout: chdir = "" 2021-02-10T11:12:05.693 INFO:teuthology.orchestra.run.gibba012.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2021-02-10T11:12:05.693 INFO:teuthology.orchestra.run.gibba012.stdout: auth supported = cephx 2021-02-10T11:12:05.693 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:12:05.693 INFO:teuthology.orchestra.run.gibba012.stdout: filestore xattr use omap = true 2021-02-10T11:12:05.693 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:12:05.693 INFO:teuthology.orchestra.run.gibba012.stdout: mon clock drift allowed = 1.000 2021-02-10T11:12:05.693 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:12:05.693 INFO:teuthology.orchestra.run.gibba012.stdout: osd crush chooseleaf type = 0 2021-02-10T11:12:05.693 INFO:teuthology.orchestra.run.gibba012.stdout: auth debug = true 2021-02-10T11:12:05.693 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:12:05.694 INFO:teuthology.orchestra.run.gibba012.stdout: ms die on old message = true 2021-02-10T11:12:05.694 INFO:teuthology.orchestra.run.gibba012.stdout: ms die on bug = true 2021-02-10T11:12:05.694 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:12:05.694 INFO:teuthology.orchestra.run.gibba012.stdout: mon max pg per osd = 10000 # >= luminous 2021-02-10T11:12:05.694 INFO:teuthology.orchestra.run.gibba012.stdout: mon pg warn max object skew = 0 2021-02-10T11:12:05.694 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:12:05.694 INFO:teuthology.orchestra.run.gibba012.stdout: osd pool default size = 2 2021-02-10T11:12:05.694 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:12:05.694 INFO:teuthology.orchestra.run.gibba012.stdout: mon osd allow primary affinity = true 2021-02-10T11:12:05.694 INFO:teuthology.orchestra.run.gibba012.stdout: mon osd allow pg remap = true 2021-02-10T11:12:05.694 INFO:teuthology.orchestra.run.gibba012.stdout: mon warn on legacy crush tunables = false 2021-02-10T11:12:05.694 INFO:teuthology.orchestra.run.gibba012.stdout: mon warn on crush straw calc version zero = false 2021-02-10T11:12:05.694 INFO:teuthology.orchestra.run.gibba012.stdout: mon warn on no sortbitwise = false 2021-02-10T11:12:05.695 INFO:teuthology.orchestra.run.gibba012.stdout: mon warn on osd down out interval zero = false 2021-02-10T11:12:05.695 INFO:teuthology.orchestra.run.gibba012.stdout: mon warn on too few osds = false 2021-02-10T11:12:05.695 INFO:teuthology.orchestra.run.gibba012.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2021-02-10T11:12:05.695 INFO:teuthology.orchestra.run.gibba012.stdout: mon_warn_on_pool_no_redundancy = false 2021-02-10T11:12:05.695 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:12:05.695 INFO:teuthology.orchestra.run.gibba012.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2021-02-10T11:12:05.695 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:12:05.695 INFO:teuthology.orchestra.run.gibba012.stdout: osd default data pool replay window = 5 2021-02-10T11:12:05.695 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:12:05.695 INFO:teuthology.orchestra.run.gibba012.stdout: mon allow pool delete = true 2021-02-10T11:12:05.695 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:12:05.695 INFO:teuthology.orchestra.run.gibba012.stdout: mon cluster log file level = debug 2021-02-10T11:12:05.695 INFO:teuthology.orchestra.run.gibba012.stdout: debug asserts on shutdown = true 2021-02-10T11:12:05.696 INFO:teuthology.orchestra.run.gibba012.stdout: mon health detail to clog = false 2021-02-10T11:12:05.696 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:12:05.696 INFO:teuthology.orchestra.run.gibba012.stdout: # we see this fail in qa on *nautilus*; bump up retries 2021-02-10T11:12:05.696 INFO:teuthology.orchestra.run.gibba012.stdout: mon_client_directed_command_retry = 4 2021-02-10T11:12:05.696 INFO:teuthology.orchestra.run.gibba012.stdout: mon host = "172.21.2.112,172.21.2.120,[v2:172.21.2.120:3301,v1:172.21.2.120:6790]" 2021-02-10T11:12:05.696 INFO:teuthology.orchestra.run.gibba012.stdout: fsid = f14f004e-0699-4584-ad81-3754258493de 2021-02-10T11:12:05.696 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:12:05.696 INFO:teuthology.orchestra.run.gibba012.stdout:[osd] 2021-02-10T11:12:05.696 INFO:teuthology.orchestra.run.gibba012.stdout: osd journal size = 100 2021-02-10T11:12:05.696 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:12:05.696 INFO:teuthology.orchestra.run.gibba012.stdout: osd scrub load threshold = 5.0 2021-02-10T11:12:05.696 INFO:teuthology.orchestra.run.gibba012.stdout: osd scrub max interval = 600 2021-02-10T11:12:05.696 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:12:05.696 INFO:teuthology.orchestra.run.gibba012.stdout: osd recover clone overlap = true 2021-02-10T11:12:05.697 INFO:teuthology.orchestra.run.gibba012.stdout: osd recovery max chunk = 1048576 2021-02-10T11:12:05.697 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:12:05.697 INFO:teuthology.orchestra.run.gibba012.stdout: osd debug shutdown = true 2021-02-10T11:12:05.697 INFO:teuthology.orchestra.run.gibba012.stdout: osd debug op order = true 2021-02-10T11:12:05.697 INFO:teuthology.orchestra.run.gibba012.stdout: osd debug verify stray on activate = true 2021-02-10T11:12:05.697 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:12:05.697 INFO:teuthology.orchestra.run.gibba012.stdout: osd open classes on start = true 2021-02-10T11:12:05.697 INFO:teuthology.orchestra.run.gibba012.stdout: osd debug pg log writeout = true 2021-02-10T11:12:05.697 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:12:05.697 INFO:teuthology.orchestra.run.gibba012.stdout: osd deep scrub update digest min age = 30 2021-02-10T11:12:05.697 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:12:05.697 INFO:teuthology.orchestra.run.gibba012.stdout: osd map max advance = 10 2021-02-10T11:12:05.697 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:12:05.698 INFO:teuthology.orchestra.run.gibba012.stdout: journal zero on create = true 2021-02-10T11:12:05.698 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:12:05.698 INFO:teuthology.orchestra.run.gibba012.stdout: filestore ondisk finisher threads = 3 2021-02-10T11:12:05.698 INFO:teuthology.orchestra.run.gibba012.stdout: filestore apply finisher threads = 3 2021-02-10T11:12:05.698 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:12:05.698 INFO:teuthology.orchestra.run.gibba012.stdout: bdev debug aio = true 2021-02-10T11:12:05.698 INFO:teuthology.orchestra.run.gibba012.stdout: osd debug misdirected ops = true 2021-02-10T11:12:05.698 INFO:teuthology.orchestra.run.gibba012.stdout: bdev async discard = True 2021-02-10T11:12:05.698 INFO:teuthology.orchestra.run.gibba012.stdout: bdev enable discard = True 2021-02-10T11:12:05.698 INFO:teuthology.orchestra.run.gibba012.stdout: bluestore allocator = bitmap 2021-02-10T11:12:05.698 INFO:teuthology.orchestra.run.gibba012.stdout: bluestore block size = 96636764160 2021-02-10T11:12:05.698 INFO:teuthology.orchestra.run.gibba012.stdout: bluestore fsck on mount = True 2021-02-10T11:12:05.698 INFO:teuthology.orchestra.run.gibba012.stdout: debug bluefs = 20 2021-02-10T11:12:05.699 INFO:teuthology.orchestra.run.gibba012.stdout: debug bluestore = 20 2021-02-10T11:12:05.699 INFO:teuthology.orchestra.run.gibba012.stdout: debug filestore = 20 2021-02-10T11:12:05.699 INFO:teuthology.orchestra.run.gibba012.stdout: debug journal = 20 2021-02-10T11:12:05.699 INFO:teuthology.orchestra.run.gibba012.stdout: debug ms = 20 2021-02-10T11:12:05.699 INFO:teuthology.orchestra.run.gibba012.stdout: debug osd = 25 2021-02-10T11:12:05.699 INFO:teuthology.orchestra.run.gibba012.stdout: debug rocksdb = 10 2021-02-10T11:12:05.699 INFO:teuthology.orchestra.run.gibba012.stdout: mon osd backfillfull_ratio = 0.85 2021-02-10T11:12:05.699 INFO:teuthology.orchestra.run.gibba012.stdout: mon osd full ratio = 0.9 2021-02-10T11:12:05.699 INFO:teuthology.orchestra.run.gibba012.stdout: mon osd nearfull ratio = 0.8 2021-02-10T11:12:05.699 INFO:teuthology.orchestra.run.gibba012.stdout: osd failsafe full ratio = 0.95 2021-02-10T11:12:05.699 INFO:teuthology.orchestra.run.gibba012.stdout: osd objectstore = bluestore 2021-02-10T11:12:05.699 INFO:teuthology.orchestra.run.gibba012.stdout: osd op complaint time = 180 2021-02-10T11:12:05.699 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:12:05.699 INFO:teuthology.orchestra.run.gibba012.stdout:[mgr] 2021-02-10T11:12:05.700 INFO:teuthology.orchestra.run.gibba012.stdout: debug ms = 1 2021-02-10T11:12:05.700 INFO:teuthology.orchestra.run.gibba012.stdout: debug mgr = 20 2021-02-10T11:12:05.700 INFO:teuthology.orchestra.run.gibba012.stdout: debug mon = 20 2021-02-10T11:12:05.700 INFO:teuthology.orchestra.run.gibba012.stdout: debug auth = 20 2021-02-10T11:12:05.700 INFO:teuthology.orchestra.run.gibba012.stdout: mon reweight min pgs per osd = 4 2021-02-10T11:12:05.700 INFO:teuthology.orchestra.run.gibba012.stdout: mon reweight min bytes per osd = 10 2021-02-10T11:12:05.700 INFO:teuthology.orchestra.run.gibba012.stdout: debug client = 10 2021-02-10T11:12:05.700 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:12:05.700 INFO:teuthology.orchestra.run.gibba012.stdout:[mon] 2021-02-10T11:12:05.700 INFO:teuthology.orchestra.run.gibba012.stdout: debug ms = 1 2021-02-10T11:12:05.700 INFO:teuthology.orchestra.run.gibba012.stdout: debug mon = 20 2021-02-10T11:12:05.700 INFO:teuthology.orchestra.run.gibba012.stdout: debug paxos = 20 2021-02-10T11:12:05.700 INFO:teuthology.orchestra.run.gibba012.stdout: debug auth = 20 2021-02-10T11:12:05.700 INFO:teuthology.orchestra.run.gibba012.stdout: mon data avail warn = 5 2021-02-10T11:12:05.701 INFO:teuthology.orchestra.run.gibba012.stdout: mon mgr mkfs grace = 240 2021-02-10T11:12:05.701 INFO:teuthology.orchestra.run.gibba012.stdout: mon reweight min pgs per osd = 4 2021-02-10T11:12:05.701 INFO:teuthology.orchestra.run.gibba012.stdout: mon osd reporter subtree level = osd 2021-02-10T11:12:05.701 INFO:teuthology.orchestra.run.gibba012.stdout: mon osd prime pg temp = true 2021-02-10T11:12:05.701 INFO:teuthology.orchestra.run.gibba012.stdout: mon reweight min bytes per osd = 10 2021-02-10T11:12:05.701 INFO:teuthology.orchestra.run.gibba012.stdout: mon op complaint time = 120 2021-02-10T11:12:05.701 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:12:05.701 INFO:teuthology.orchestra.run.gibba012.stdout:[client] 2021-02-10T11:12:05.701 INFO:teuthology.orchestra.run.gibba012.stdout: rgw cache enabled = true 2021-02-10T11:12:05.701 INFO:teuthology.orchestra.run.gibba012.stdout: rgw enable ops log = true 2021-02-10T11:12:05.701 INFO:teuthology.orchestra.run.gibba012.stdout: rgw enable usage log = true 2021-02-10T11:12:05.701 INFO:teuthology.orchestra.run.gibba012.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2021-02-10T11:12:05.701 INFO:teuthology.orchestra.run.gibba012.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2021-02-10T11:12:05.702 INFO:teuthology.orchestra.run.gibba012.stdout: client mount timeout = 600 2021-02-10T11:12:05.702 INFO:teuthology.orchestra.run.gibba012.stdout: debug client = 20 2021-02-10T11:12:05.702 INFO:teuthology.orchestra.run.gibba012.stdout: debug ms = 1 2021-02-10T11:12:05.702 INFO:teuthology.orchestra.run.gibba012.stdout: pid file = "" 2021-02-10T11:12:05.702 INFO:teuthology.orchestra.run.gibba012.stdout:[mon.a] 2021-02-10T11:12:05.702 INFO:teuthology.orchestra.run.gibba012.stdout:[mon.b] 2021-02-10T11:12:05.702 INFO:teuthology.orchestra.run.gibba012.stdout:[mon.c] 2021-02-10T11:12:05.702 INFO:teuthology.orchestra.run.gibba012.stdout:[mds.a] 2021-02-10T11:12:05.702 INFO:teuthology.orchestra.run.gibba012.stdout:[mds.b] 2021-02-10T11:12:05.702 INFO:teuthology.orchestra.run.gibba012.stdout:[mds] 2021-02-10T11:12:05.702 INFO:teuthology.orchestra.run.gibba012.stdout: debug mds = 20 2021-02-10T11:12:05.702 INFO:teuthology.orchestra.run.gibba012.stdout: debug ms = 1 2021-02-10T11:12:05.702 INFO:teuthology.orchestra.run.gibba012.stdout: mds bal frag = True 2021-02-10T11:12:05.702 INFO:teuthology.orchestra.run.gibba012.stdout: mds bal fragment size max = 10000 2021-02-10T11:12:05.703 INFO:teuthology.orchestra.run.gibba012.stdout: mds bal merge size = 5 2021-02-10T11:12:05.703 INFO:teuthology.orchestra.run.gibba012.stdout: mds bal split bits = 3 2021-02-10T11:12:05.703 INFO:teuthology.orchestra.run.gibba012.stdout: mds bal split size = 100 2021-02-10T11:12:05.703 INFO:teuthology.orchestra.run.gibba012.stdout: mds debug frag = True 2021-02-10T11:12:05.703 INFO:teuthology.orchestra.run.gibba012.stdout: mds debug scatterstat = True 2021-02-10T11:12:05.703 INFO:teuthology.orchestra.run.gibba012.stdout: mds op complaint time = 180 2021-02-10T11:12:05.703 INFO:teuthology.orchestra.run.gibba012.stdout: mds verify scatter = True 2021-02-10T11:12:05.703 INFO:teuthology.orchestra.run.gibba012.stdout: osd op complaint time = 180 2021-02-10T11:12:05.704 INFO:teuthology.orchestra.run.gibba020.stdout:[global] 2021-02-10T11:12:05.704 INFO:teuthology.orchestra.run.gibba020.stdout: chdir = "" 2021-02-10T11:12:05.704 INFO:teuthology.orchestra.run.gibba020.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2021-02-10T11:12:05.704 INFO:teuthology.orchestra.run.gibba020.stdout: auth supported = cephx 2021-02-10T11:12:05.704 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:12:05.704 INFO:teuthology.orchestra.run.gibba020.stdout: filestore xattr use omap = true 2021-02-10T11:12:05.704 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:12:05.705 INFO:teuthology.orchestra.run.gibba020.stdout: mon clock drift allowed = 1.000 2021-02-10T11:12:05.705 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:12:05.705 INFO:teuthology.orchestra.run.gibba020.stdout: osd crush chooseleaf type = 0 2021-02-10T11:12:05.705 INFO:teuthology.orchestra.run.gibba020.stdout: auth debug = true 2021-02-10T11:12:05.705 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:12:05.705 INFO:teuthology.orchestra.run.gibba020.stdout: ms die on old message = true 2021-02-10T11:12:05.705 INFO:teuthology.orchestra.run.gibba020.stdout: ms die on bug = true 2021-02-10T11:12:05.705 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:12:05.705 INFO:teuthology.orchestra.run.gibba020.stdout: mon max pg per osd = 10000 # >= luminous 2021-02-10T11:12:05.705 INFO:teuthology.orchestra.run.gibba020.stdout: mon pg warn max object skew = 0 2021-02-10T11:12:05.705 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:12:05.705 INFO:teuthology.orchestra.run.gibba020.stdout: osd pool default size = 2 2021-02-10T11:12:05.705 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:12:05.705 INFO:teuthology.orchestra.run.gibba020.stdout: mon osd allow primary affinity = true 2021-02-10T11:12:05.706 INFO:teuthology.orchestra.run.gibba020.stdout: mon osd allow pg remap = true 2021-02-10T11:12:05.706 INFO:teuthology.orchestra.run.gibba020.stdout: mon warn on legacy crush tunables = false 2021-02-10T11:12:05.706 INFO:teuthology.orchestra.run.gibba020.stdout: mon warn on crush straw calc version zero = false 2021-02-10T11:12:05.706 INFO:teuthology.orchestra.run.gibba020.stdout: mon warn on no sortbitwise = false 2021-02-10T11:12:05.706 INFO:teuthology.orchestra.run.gibba020.stdout: mon warn on osd down out interval zero = false 2021-02-10T11:12:05.706 INFO:teuthology.orchestra.run.gibba020.stdout: mon warn on too few osds = false 2021-02-10T11:12:05.706 INFO:teuthology.orchestra.run.gibba020.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2021-02-10T11:12:05.706 INFO:teuthology.orchestra.run.gibba020.stdout: mon_warn_on_pool_no_redundancy = false 2021-02-10T11:12:05.706 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:12:05.707 INFO:teuthology.orchestra.run.gibba020.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2021-02-10T11:12:05.707 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:12:05.707 INFO:teuthology.orchestra.run.gibba020.stdout: osd default data pool replay window = 5 2021-02-10T11:12:05.707 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:12:05.708 INFO:teuthology.orchestra.run.gibba020.stdout: mon allow pool delete = true 2021-02-10T11:12:05.708 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:12:05.708 INFO:teuthology.orchestra.run.gibba020.stdout: mon cluster log file level = debug 2021-02-10T11:12:05.708 INFO:teuthology.orchestra.run.gibba020.stdout: debug asserts on shutdown = true 2021-02-10T11:12:05.708 INFO:teuthology.orchestra.run.gibba020.stdout: mon health detail to clog = false 2021-02-10T11:12:05.708 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:12:05.708 INFO:teuthology.orchestra.run.gibba020.stdout: # we see this fail in qa on *nautilus*; bump up retries 2021-02-10T11:12:05.708 INFO:teuthology.orchestra.run.gibba020.stdout: mon_client_directed_command_retry = 4 2021-02-10T11:12:05.708 INFO:teuthology.orchestra.run.gibba020.stdout: mon host = "172.21.2.112,172.21.2.120,[v2:172.21.2.120:3301,v1:172.21.2.120:6790]" 2021-02-10T11:12:05.708 INFO:teuthology.orchestra.run.gibba020.stdout: fsid = f14f004e-0699-4584-ad81-3754258493de 2021-02-10T11:12:05.709 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:12:05.709 INFO:teuthology.orchestra.run.gibba020.stdout:[osd] 2021-02-10T11:12:05.709 INFO:teuthology.orchestra.run.gibba020.stdout: osd journal size = 100 2021-02-10T11:12:05.709 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:12:05.709 INFO:teuthology.orchestra.run.gibba020.stdout: osd scrub load threshold = 5.0 2021-02-10T11:12:05.709 INFO:teuthology.orchestra.run.gibba020.stdout: osd scrub max interval = 600 2021-02-10T11:12:05.709 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:12:05.709 INFO:teuthology.orchestra.run.gibba020.stdout: osd recover clone overlap = true 2021-02-10T11:12:05.709 INFO:teuthology.orchestra.run.gibba020.stdout: osd recovery max chunk = 1048576 2021-02-10T11:12:05.709 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:12:05.709 INFO:teuthology.orchestra.run.gibba020.stdout: osd debug shutdown = true 2021-02-10T11:12:05.709 INFO:teuthology.orchestra.run.gibba020.stdout: osd debug op order = true 2021-02-10T11:12:05.709 INFO:teuthology.orchestra.run.gibba020.stdout: osd debug verify stray on activate = true 2021-02-10T11:12:05.710 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:12:05.710 INFO:teuthology.orchestra.run.gibba020.stdout: osd open classes on start = true 2021-02-10T11:12:05.710 INFO:teuthology.orchestra.run.gibba020.stdout: osd debug pg log writeout = true 2021-02-10T11:12:05.710 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:12:05.710 INFO:teuthology.orchestra.run.gibba020.stdout: osd deep scrub update digest min age = 30 2021-02-10T11:12:05.710 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:12:05.710 INFO:teuthology.orchestra.run.gibba020.stdout: osd map max advance = 10 2021-02-10T11:12:05.710 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:12:05.710 INFO:teuthology.orchestra.run.gibba020.stdout: journal zero on create = true 2021-02-10T11:12:05.710 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:12:05.710 INFO:teuthology.orchestra.run.gibba020.stdout: filestore ondisk finisher threads = 3 2021-02-10T11:12:05.710 INFO:teuthology.orchestra.run.gibba020.stdout: filestore apply finisher threads = 3 2021-02-10T11:12:05.710 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:12:05.710 INFO:teuthology.orchestra.run.gibba020.stdout: bdev debug aio = true 2021-02-10T11:12:05.711 INFO:teuthology.orchestra.run.gibba020.stdout: osd debug misdirected ops = true 2021-02-10T11:12:05.711 INFO:teuthology.orchestra.run.gibba020.stdout: bdev async discard = True 2021-02-10T11:12:05.711 INFO:teuthology.orchestra.run.gibba020.stdout: bdev enable discard = True 2021-02-10T11:12:05.711 INFO:teuthology.orchestra.run.gibba020.stdout: bluestore allocator = bitmap 2021-02-10T11:12:05.712 INFO:teuthology.orchestra.run.gibba020.stdout: bluestore block size = 96636764160 2021-02-10T11:12:05.712 INFO:teuthology.orchestra.run.gibba020.stdout: bluestore fsck on mount = True 2021-02-10T11:12:05.712 INFO:teuthology.orchestra.run.gibba020.stdout: debug bluefs = 20 2021-02-10T11:12:05.712 INFO:teuthology.orchestra.run.gibba020.stdout: debug bluestore = 20 2021-02-10T11:12:05.712 INFO:teuthology.orchestra.run.gibba020.stdout: debug filestore = 20 2021-02-10T11:12:05.712 INFO:teuthology.orchestra.run.gibba020.stdout: debug journal = 20 2021-02-10T11:12:05.712 INFO:teuthology.orchestra.run.gibba020.stdout: debug ms = 20 2021-02-10T11:12:05.712 INFO:teuthology.orchestra.run.gibba020.stdout: debug osd = 25 2021-02-10T11:12:05.712 INFO:teuthology.orchestra.run.gibba020.stdout: debug rocksdb = 10 2021-02-10T11:12:05.713 INFO:teuthology.orchestra.run.gibba020.stdout: mon osd backfillfull_ratio = 0.85 2021-02-10T11:12:05.713 INFO:teuthology.orchestra.run.gibba020.stdout: mon osd full ratio = 0.9 2021-02-10T11:12:05.713 INFO:teuthology.orchestra.run.gibba020.stdout: mon osd nearfull ratio = 0.8 2021-02-10T11:12:05.713 INFO:teuthology.orchestra.run.gibba020.stdout: osd failsafe full ratio = 0.95 2021-02-10T11:12:05.713 INFO:teuthology.orchestra.run.gibba020.stdout: osd objectstore = bluestore 2021-02-10T11:12:05.713 INFO:teuthology.orchestra.run.gibba020.stdout: osd op complaint time = 180 2021-02-10T11:12:05.713 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:12:05.713 INFO:teuthology.orchestra.run.gibba020.stdout:[mgr] 2021-02-10T11:12:05.713 INFO:teuthology.orchestra.run.gibba020.stdout: debug ms = 1 2021-02-10T11:12:05.713 INFO:teuthology.orchestra.run.gibba020.stdout: debug mgr = 20 2021-02-10T11:12:05.713 INFO:teuthology.orchestra.run.gibba020.stdout: debug mon = 20 2021-02-10T11:12:05.713 INFO:teuthology.orchestra.run.gibba020.stdout: debug auth = 20 2021-02-10T11:12:05.713 INFO:teuthology.orchestra.run.gibba020.stdout: mon reweight min pgs per osd = 4 2021-02-10T11:12:05.714 INFO:teuthology.orchestra.run.gibba020.stdout: mon reweight min bytes per osd = 10 2021-02-10T11:12:05.714 INFO:teuthology.orchestra.run.gibba020.stdout: debug client = 10 2021-02-10T11:12:05.714 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:12:05.714 INFO:teuthology.orchestra.run.gibba020.stdout:[mon] 2021-02-10T11:12:05.714 INFO:teuthology.orchestra.run.gibba020.stdout: debug ms = 1 2021-02-10T11:12:05.714 INFO:teuthology.orchestra.run.gibba020.stdout: debug mon = 20 2021-02-10T11:12:05.714 INFO:teuthology.orchestra.run.gibba020.stdout: debug paxos = 20 2021-02-10T11:12:05.714 INFO:teuthology.orchestra.run.gibba020.stdout: debug auth = 20 2021-02-10T11:12:05.714 INFO:teuthology.orchestra.run.gibba020.stdout: mon data avail warn = 5 2021-02-10T11:12:05.714 INFO:teuthology.orchestra.run.gibba020.stdout: mon mgr mkfs grace = 240 2021-02-10T11:12:05.715 INFO:teuthology.orchestra.run.gibba020.stdout: mon reweight min pgs per osd = 4 2021-02-10T11:12:05.716 INFO:teuthology.orchestra.run.gibba020.stdout: mon osd reporter subtree level = osd 2021-02-10T11:12:05.716 INFO:teuthology.orchestra.run.gibba020.stdout: mon osd prime pg temp = true 2021-02-10T11:12:05.716 INFO:teuthology.orchestra.run.gibba020.stdout: mon reweight min bytes per osd = 10 2021-02-10T11:12:05.716 INFO:teuthology.orchestra.run.gibba020.stdout: mon op complaint time = 120 2021-02-10T11:12:05.716 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:12:05.716 INFO:teuthology.orchestra.run.gibba020.stdout:[client] 2021-02-10T11:12:05.716 INFO:teuthology.orchestra.run.gibba020.stdout: rgw cache enabled = true 2021-02-10T11:12:05.716 INFO:teuthology.orchestra.run.gibba020.stdout: rgw enable ops log = true 2021-02-10T11:12:05.716 INFO:teuthology.orchestra.run.gibba020.stdout: rgw enable usage log = true 2021-02-10T11:12:05.716 INFO:teuthology.orchestra.run.gibba020.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2021-02-10T11:12:05.716 INFO:teuthology.orchestra.run.gibba020.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2021-02-10T11:12:05.716 INFO:teuthology.orchestra.run.gibba020.stdout: client mount timeout = 600 2021-02-10T11:12:05.717 INFO:teuthology.orchestra.run.gibba020.stdout: debug client = 20 2021-02-10T11:12:05.717 INFO:teuthology.orchestra.run.gibba020.stdout: debug ms = 1 2021-02-10T11:12:05.717 INFO:teuthology.orchestra.run.gibba020.stdout: pid file = "" 2021-02-10T11:12:05.717 INFO:teuthology.orchestra.run.gibba020.stdout:[mon.a] 2021-02-10T11:12:05.717 INFO:teuthology.orchestra.run.gibba020.stdout:[mon.b] 2021-02-10T11:12:05.717 INFO:teuthology.orchestra.run.gibba020.stdout:[mon.c] 2021-02-10T11:12:05.717 INFO:teuthology.orchestra.run.gibba020.stdout:[mds.a] 2021-02-10T11:12:05.717 INFO:teuthology.orchestra.run.gibba020.stdout:[mds.b] 2021-02-10T11:12:05.717 INFO:teuthology.orchestra.run.gibba020.stdout:[mds] 2021-02-10T11:12:05.717 INFO:teuthology.orchestra.run.gibba020.stdout: debug mds = 20 2021-02-10T11:12:05.717 INFO:teuthology.orchestra.run.gibba020.stdout: debug ms = 1 2021-02-10T11:12:05.717 INFO:teuthology.orchestra.run.gibba020.stdout: mds bal frag = True 2021-02-10T11:12:05.718 INFO:teuthology.orchestra.run.gibba020.stdout: mds bal fragment size max = 10000 2021-02-10T11:12:05.718 INFO:teuthology.orchestra.run.gibba020.stdout: mds bal merge size = 5 2021-02-10T11:12:05.718 INFO:teuthology.orchestra.run.gibba020.stdout: mds bal split bits = 3 2021-02-10T11:12:05.718 INFO:teuthology.orchestra.run.gibba020.stdout: mds bal split size = 100 2021-02-10T11:12:05.718 INFO:teuthology.orchestra.run.gibba020.stdout: mds debug frag = True 2021-02-10T11:12:05.718 INFO:teuthology.orchestra.run.gibba020.stdout: mds debug scatterstat = True 2021-02-10T11:12:05.718 INFO:teuthology.orchestra.run.gibba020.stdout: mds op complaint time = 180 2021-02-10T11:12:05.718 INFO:teuthology.orchestra.run.gibba020.stdout: mds verify scatter = True 2021-02-10T11:12:05.718 INFO:teuthology.orchestra.run.gibba020.stdout: osd op complaint time = 180 2021-02-10T11:12:05.721 INFO:tasks.ceph:Creating admin key on mon.a... 2021-02-10T11:12:05.721 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2021-02-10T11:12:06.097 INFO:tasks.ceph:Copying monmap to all nodes... 2021-02-10T11:12:06.433 DEBUG:teuthology.orchestra.remote:gibba012:/etc/ceph/ceph.keyring is 206B 2021-02-10T11:12:07.108 DEBUG:teuthology.orchestra.remote:gibba012:/home/ubuntu/cephtest/ceph.monmap is 373B 2021-02-10T11:12:07.449 INFO:tasks.ceph:Sending monmap to node ubuntu@gibba012.front.sepia.ceph.com 2021-02-10T11:12:07.449 DEBUG:teuthology.orchestra.run.gibba012:> set -ex 2021-02-10T11:12:07.449 DEBUG:teuthology.orchestra.run.gibba012:> sudo dd of=/etc/ceph/ceph.keyring 2021-02-10T11:12:07.449 DEBUG:teuthology.orchestra.run.gibba012:> sudo chmod 0644 /etc/ceph/ceph.keyring 2021-02-10T11:12:07.800 DEBUG:teuthology.orchestra.run.gibba012:> set -ex 2021-02-10T11:12:07.800 DEBUG:teuthology.orchestra.run.gibba012:> dd of=/home/ubuntu/cephtest/ceph.monmap 2021-02-10T11:12:08.134 INFO:tasks.ceph:Sending monmap to node ubuntu@gibba020.front.sepia.ceph.com 2021-02-10T11:12:08.134 DEBUG:teuthology.orchestra.run.gibba020:> set -ex 2021-02-10T11:12:08.134 DEBUG:teuthology.orchestra.run.gibba020:> sudo dd of=/etc/ceph/ceph.keyring 2021-02-10T11:12:08.134 DEBUG:teuthology.orchestra.run.gibba020:> sudo chmod 0644 /etc/ceph/ceph.keyring 2021-02-10T11:12:08.487 DEBUG:teuthology.orchestra.run.gibba020:> set -ex 2021-02-10T11:12:08.528 DEBUG:teuthology.orchestra.run.gibba020:> dd of=/home/ubuntu/cephtest/ceph.monmap 2021-02-10T11:12:08.861 INFO:tasks.ceph:Setting up mon nodes... 2021-02-10T11:12:08.862 INFO:tasks.ceph:Setting up mgr nodes... 2021-02-10T11:12:08.862 DEBUG:teuthology.orchestra.run.gibba012:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2021-02-10T11:12:09.238 INFO:teuthology.orchestra.run.gibba012.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2021-02-10T11:12:09.241 DEBUG:teuthology.orchestra.run.gibba020:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2021-02-10T11:12:09.676 INFO:teuthology.orchestra.run.gibba020.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2021-02-10T11:12:09.679 INFO:tasks.ceph:Setting up mds nodes... 2021-02-10T11:12:09.679 DEBUG:teuthology.orchestra.run.gibba012:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2021-02-10T11:12:10.056 INFO:teuthology.orchestra.run.gibba012.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2021-02-10T11:12:10.058 DEBUG:teuthology.orchestra.run.gibba012:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2021-02-10T11:12:10.401 DEBUG:teuthology.orchestra.run.gibba020:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2021-02-10T11:12:10.779 INFO:teuthology.orchestra.run.gibba020.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2021-02-10T11:12:10.781 DEBUG:teuthology.orchestra.run.gibba020:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2021-02-10T11:12:11.123 INFO:tasks.ceph_client:Setting up client nodes... 2021-02-10T11:12:11.123 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2021-02-10T11:12:11.491 INFO:teuthology.orchestra.run.gibba012.stdout:creating /etc/ceph/ceph.client.0.keyring 2021-02-10T11:12:11.503 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2021-02-10T11:12:11.869 INFO:teuthology.orchestra.run.gibba012.stdout:creating /etc/ceph/ceph.client.1.keyring 2021-02-10T11:12:11.880 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2021-02-10T11:12:12.247 INFO:teuthology.orchestra.run.gibba020.stdout:creating /etc/ceph/ceph.client.2.keyring 2021-02-10T11:12:12.259 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.3 /etc/ceph/ceph.client.3.keyring && sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2021-02-10T11:12:12.625 INFO:teuthology.orchestra.run.gibba020.stdout:creating /etc/ceph/ceph.client.3.keyring 2021-02-10T11:12:12.637 INFO:tasks.ceph:Running mkfs on osd nodes... 2021-02-10T11:12:12.637 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@gibba012.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@gibba020.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2021-02-10T11:12:12.638 DEBUG:teuthology.orchestra.run.gibba012:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2021-02-10T11:12:12.980 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2021-02-10T11:12:12.981 INFO:tasks.ceph:role: osd.0 2021-02-10T11:12:12.981 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@gibba012.front.sepia.ceph.com 2021-02-10T11:12:12.981 DEBUG:teuthology.orchestra.run.gibba012:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2021-02-10T11:12:13.373 INFO:teuthology.orchestra.run.gibba012.stdout:Discarding blocks...Done. 2021-02-10T11:12:13.374 INFO:teuthology.orchestra.run.gibba012.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5505024 blks 2021-02-10T11:12:13.374 INFO:teuthology.orchestra.run.gibba012.stdout: = sectsz=512 attr=2, projid32bit=1 2021-02-10T11:12:13.374 INFO:teuthology.orchestra.run.gibba012.stdout: = crc=1 finobt=0, sparse=0 2021-02-10T11:12:13.374 INFO:teuthology.orchestra.run.gibba012.stdout:data = bsize=4096 blocks=22020096, imaxpct=25 2021-02-10T11:12:13.374 INFO:teuthology.orchestra.run.gibba012.stdout: = sunit=0 swidth=0 blks 2021-02-10T11:12:13.374 INFO:teuthology.orchestra.run.gibba012.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2021-02-10T11:12:13.374 INFO:teuthology.orchestra.run.gibba012.stdout:log =internal log bsize=4096 blocks=10752, version=2 2021-02-10T11:12:13.374 INFO:teuthology.orchestra.run.gibba012.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-02-10T11:12:13.374 INFO:teuthology.orchestra.run.gibba012.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-02-10T11:12:13.375 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@gibba012.front.sepia.ceph.com -o noatime 2021-02-10T11:12:13.375 DEBUG:teuthology.orchestra.run.gibba012:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2021-02-10T11:12:13.735 DEBUG:teuthology.orchestra.run.gibba012:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2021-02-10T11:12:14.090 DEBUG:teuthology.orchestra.run.gibba012:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2021-02-10T11:12:14.445 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2021-02-10T11:12:14.445 INFO:tasks.ceph:role: osd.1 2021-02-10T11:12:14.445 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@gibba012.front.sepia.ceph.com 2021-02-10T11:12:14.445 DEBUG:teuthology.orchestra.run.gibba012:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2021-02-10T11:12:14.835 INFO:teuthology.orchestra.run.gibba012.stdout:Discarding blocks...Done. 2021-02-10T11:12:14.836 INFO:teuthology.orchestra.run.gibba012.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5505024 blks 2021-02-10T11:12:14.836 INFO:teuthology.orchestra.run.gibba012.stdout: = sectsz=512 attr=2, projid32bit=1 2021-02-10T11:12:14.836 INFO:teuthology.orchestra.run.gibba012.stdout: = crc=1 finobt=0, sparse=0 2021-02-10T11:12:14.836 INFO:teuthology.orchestra.run.gibba012.stdout:data = bsize=4096 blocks=22020096, imaxpct=25 2021-02-10T11:12:14.836 INFO:teuthology.orchestra.run.gibba012.stdout: = sunit=0 swidth=0 blks 2021-02-10T11:12:14.836 INFO:teuthology.orchestra.run.gibba012.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2021-02-10T11:12:14.836 INFO:teuthology.orchestra.run.gibba012.stdout:log =internal log bsize=4096 blocks=10752, version=2 2021-02-10T11:12:14.836 INFO:teuthology.orchestra.run.gibba012.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-02-10T11:12:14.836 INFO:teuthology.orchestra.run.gibba012.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-02-10T11:12:14.837 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@gibba012.front.sepia.ceph.com -o noatime 2021-02-10T11:12:14.837 DEBUG:teuthology.orchestra.run.gibba012:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2021-02-10T11:12:15.194 DEBUG:teuthology.orchestra.run.gibba012:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2021-02-10T11:12:15.542 DEBUG:teuthology.orchestra.run.gibba012:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2021-02-10T11:12:15.887 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2021-02-10T11:12:15.887 INFO:tasks.ceph:role: osd.2 2021-02-10T11:12:15.887 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@gibba012.front.sepia.ceph.com 2021-02-10T11:12:15.887 DEBUG:teuthology.orchestra.run.gibba012:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2021-02-10T11:12:16.272 INFO:teuthology.orchestra.run.gibba012.stdout:Discarding blocks...Done. 2021-02-10T11:12:16.272 INFO:teuthology.orchestra.run.gibba012.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5505024 blks 2021-02-10T11:12:16.273 INFO:teuthology.orchestra.run.gibba012.stdout: = sectsz=512 attr=2, projid32bit=1 2021-02-10T11:12:16.273 INFO:teuthology.orchestra.run.gibba012.stdout: = crc=1 finobt=0, sparse=0 2021-02-10T11:12:16.273 INFO:teuthology.orchestra.run.gibba012.stdout:data = bsize=4096 blocks=22020096, imaxpct=25 2021-02-10T11:12:16.273 INFO:teuthology.orchestra.run.gibba012.stdout: = sunit=0 swidth=0 blks 2021-02-10T11:12:16.273 INFO:teuthology.orchestra.run.gibba012.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2021-02-10T11:12:16.273 INFO:teuthology.orchestra.run.gibba012.stdout:log =internal log bsize=4096 blocks=10752, version=2 2021-02-10T11:12:16.273 INFO:teuthology.orchestra.run.gibba012.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-02-10T11:12:16.273 INFO:teuthology.orchestra.run.gibba012.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-02-10T11:12:16.274 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@gibba012.front.sepia.ceph.com -o noatime 2021-02-10T11:12:16.274 DEBUG:teuthology.orchestra.run.gibba012:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2021-02-10T11:12:16.629 DEBUG:teuthology.orchestra.run.gibba012:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2021-02-10T11:12:16.974 DEBUG:teuthology.orchestra.run.gibba012:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2021-02-10T11:12:17.317 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2021-02-10T11:12:17.317 INFO:tasks.ceph:role: osd.3 2021-02-10T11:12:17.317 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@gibba012.front.sepia.ceph.com 2021-02-10T11:12:17.317 DEBUG:teuthology.orchestra.run.gibba012:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2021-02-10T11:12:17.708 INFO:teuthology.orchestra.run.gibba012.stdout:Discarding blocks...Done. 2021-02-10T11:12:17.708 INFO:teuthology.orchestra.run.gibba012.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5505024 blks 2021-02-10T11:12:17.708 INFO:teuthology.orchestra.run.gibba012.stdout: = sectsz=512 attr=2, projid32bit=1 2021-02-10T11:12:17.708 INFO:teuthology.orchestra.run.gibba012.stdout: = crc=1 finobt=0, sparse=0 2021-02-10T11:12:17.708 INFO:teuthology.orchestra.run.gibba012.stdout:data = bsize=4096 blocks=22020096, imaxpct=25 2021-02-10T11:12:17.708 INFO:teuthology.orchestra.run.gibba012.stdout: = sunit=0 swidth=0 blks 2021-02-10T11:12:17.708 INFO:teuthology.orchestra.run.gibba012.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2021-02-10T11:12:17.708 INFO:teuthology.orchestra.run.gibba012.stdout:log =internal log bsize=4096 blocks=10752, version=2 2021-02-10T11:12:17.709 INFO:teuthology.orchestra.run.gibba012.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-02-10T11:12:17.709 INFO:teuthology.orchestra.run.gibba012.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-02-10T11:12:17.709 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@gibba012.front.sepia.ceph.com -o noatime 2021-02-10T11:12:17.710 DEBUG:teuthology.orchestra.run.gibba012:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2021-02-10T11:12:18.064 DEBUG:teuthology.orchestra.run.gibba012:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2021-02-10T11:12:18.410 DEBUG:teuthology.orchestra.run.gibba012:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2021-02-10T11:12:18.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:12:18.780 7fe277425a80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2021-02-10T11:12:18.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:12:18.780 7fe277425a80 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2021-02-10T11:12:18.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:12:18.780 7fe277425a80 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2021-02-10T11:12:18.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:12:18.780 7fe277425a80 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2021-02-10T11:12:18.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:12:18.780 7fe277425a80 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2021-02-10T11:12:18.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:12:18.781 7fe277425a80 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2021-02-10T11:12:21.074 DEBUG:teuthology.orchestra.run.gibba012:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2021-02-10T11:12:21.416 DEBUG:teuthology.orchestra.run.gibba012:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2021-02-10T11:12:21.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:12:21.785 7f284383da80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2021-02-10T11:12:21.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:12:21.785 7f284383da80 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2021-02-10T11:12:21.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:12:21.785 7f284383da80 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2021-02-10T11:12:21.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:12:21.785 7f284383da80 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2021-02-10T11:12:21.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:12:21.785 7f284383da80 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2021-02-10T11:12:21.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:12:21.786 7f284383da80 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2021-02-10T11:12:24.074 DEBUG:teuthology.orchestra.run.gibba012:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2021-02-10T11:12:24.417 DEBUG:teuthology.orchestra.run.gibba012:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2021-02-10T11:12:24.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:12:24.786 7fb558c39a80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2021-02-10T11:12:24.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:12:24.786 7fb558c39a80 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2021-02-10T11:12:24.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:12:24.786 7fb558c39a80 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2021-02-10T11:12:24.786 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:12:24.786 7fb558c39a80 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2021-02-10T11:12:24.786 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:12:24.786 7fb558c39a80 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2021-02-10T11:12:24.786 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:12:24.787 7fb558c39a80 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2021-02-10T11:12:27.079 DEBUG:teuthology.orchestra.run.gibba012:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2021-02-10T11:12:27.423 DEBUG:teuthology.orchestra.run.gibba012:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2021-02-10T11:12:27.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:12:27.794 7f8864c4da80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2021-02-10T11:12:27.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:12:27.794 7f8864c4da80 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2021-02-10T11:12:27.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:12:27.794 7f8864c4da80 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2021-02-10T11:12:27.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:12:27.794 7f8864c4da80 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2021-02-10T11:12:27.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:12:27.794 7f8864c4da80 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2021-02-10T11:12:27.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:12:27.795 7f8864c4da80 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2021-02-10T11:12:29.719 DEBUG:teuthology.orchestra.run.gibba012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:12:29.722 DEBUG:teuthology.orchestra.run.gibba020:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:12:30.087 DEBUG:teuthology.orchestra.run.gibba012:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2021-02-10T11:12:30.430 DEBUG:teuthology.orchestra.run.gibba020:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2021-02-10T11:12:30.774 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2021-02-10T11:12:30.774 INFO:tasks.ceph:role: osd.4 2021-02-10T11:12:30.774 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@gibba020.front.sepia.ceph.com 2021-02-10T11:12:30.774 DEBUG:teuthology.orchestra.run.gibba020:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2021-02-10T11:12:31.161 INFO:teuthology.orchestra.run.gibba020.stdout:Discarding blocks...Done. 2021-02-10T11:12:31.161 INFO:teuthology.orchestra.run.gibba020.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5505024 blks 2021-02-10T11:12:31.161 INFO:teuthology.orchestra.run.gibba020.stdout: = sectsz=512 attr=2, projid32bit=1 2021-02-10T11:12:31.161 INFO:teuthology.orchestra.run.gibba020.stdout: = crc=1 finobt=0, sparse=0 2021-02-10T11:12:31.162 INFO:teuthology.orchestra.run.gibba020.stdout:data = bsize=4096 blocks=22020096, imaxpct=25 2021-02-10T11:12:31.162 INFO:teuthology.orchestra.run.gibba020.stdout: = sunit=0 swidth=0 blks 2021-02-10T11:12:31.162 INFO:teuthology.orchestra.run.gibba020.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2021-02-10T11:12:31.162 INFO:teuthology.orchestra.run.gibba020.stdout:log =internal log bsize=4096 blocks=10752, version=2 2021-02-10T11:12:31.162 INFO:teuthology.orchestra.run.gibba020.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-02-10T11:12:31.162 INFO:teuthology.orchestra.run.gibba020.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-02-10T11:12:31.163 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@gibba020.front.sepia.ceph.com -o noatime 2021-02-10T11:12:31.163 DEBUG:teuthology.orchestra.run.gibba020:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2021-02-10T11:12:31.522 DEBUG:teuthology.orchestra.run.gibba020:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2021-02-10T11:12:31.868 DEBUG:teuthology.orchestra.run.gibba020:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2021-02-10T11:12:32.212 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2021-02-10T11:12:32.212 INFO:tasks.ceph:role: osd.5 2021-02-10T11:12:32.212 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@gibba020.front.sepia.ceph.com 2021-02-10T11:12:32.212 DEBUG:teuthology.orchestra.run.gibba020:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2021-02-10T11:12:32.599 INFO:teuthology.orchestra.run.gibba020.stdout:Discarding blocks...Done. 2021-02-10T11:12:32.600 INFO:teuthology.orchestra.run.gibba020.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5505024 blks 2021-02-10T11:12:32.600 INFO:teuthology.orchestra.run.gibba020.stdout: = sectsz=512 attr=2, projid32bit=1 2021-02-10T11:12:32.600 INFO:teuthology.orchestra.run.gibba020.stdout: = crc=1 finobt=0, sparse=0 2021-02-10T11:12:32.600 INFO:teuthology.orchestra.run.gibba020.stdout:data = bsize=4096 blocks=22020096, imaxpct=25 2021-02-10T11:12:32.600 INFO:teuthology.orchestra.run.gibba020.stdout: = sunit=0 swidth=0 blks 2021-02-10T11:12:32.600 INFO:teuthology.orchestra.run.gibba020.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2021-02-10T11:12:32.600 INFO:teuthology.orchestra.run.gibba020.stdout:log =internal log bsize=4096 blocks=10752, version=2 2021-02-10T11:12:32.600 INFO:teuthology.orchestra.run.gibba020.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-02-10T11:12:32.600 INFO:teuthology.orchestra.run.gibba020.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-02-10T11:12:32.601 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@gibba020.front.sepia.ceph.com -o noatime 2021-02-10T11:12:32.601 DEBUG:teuthology.orchestra.run.gibba020:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2021-02-10T11:12:32.967 DEBUG:teuthology.orchestra.run.gibba020:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2021-02-10T11:12:33.312 DEBUG:teuthology.orchestra.run.gibba020:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2021-02-10T11:12:33.656 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2021-02-10T11:12:33.656 INFO:tasks.ceph:role: osd.6 2021-02-10T11:12:33.656 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@gibba020.front.sepia.ceph.com 2021-02-10T11:12:33.656 DEBUG:teuthology.orchestra.run.gibba020:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2021-02-10T11:12:34.049 INFO:teuthology.orchestra.run.gibba020.stdout:Discarding blocks...Done. 2021-02-10T11:12:34.049 INFO:teuthology.orchestra.run.gibba020.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5505024 blks 2021-02-10T11:12:34.049 INFO:teuthology.orchestra.run.gibba020.stdout: = sectsz=512 attr=2, projid32bit=1 2021-02-10T11:12:34.049 INFO:teuthology.orchestra.run.gibba020.stdout: = crc=1 finobt=0, sparse=0 2021-02-10T11:12:34.049 INFO:teuthology.orchestra.run.gibba020.stdout:data = bsize=4096 blocks=22020096, imaxpct=25 2021-02-10T11:12:34.049 INFO:teuthology.orchestra.run.gibba020.stdout: = sunit=0 swidth=0 blks 2021-02-10T11:12:34.049 INFO:teuthology.orchestra.run.gibba020.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2021-02-10T11:12:34.049 INFO:teuthology.orchestra.run.gibba020.stdout:log =internal log bsize=4096 blocks=10752, version=2 2021-02-10T11:12:34.049 INFO:teuthology.orchestra.run.gibba020.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-02-10T11:12:34.050 INFO:teuthology.orchestra.run.gibba020.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-02-10T11:12:34.050 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@gibba020.front.sepia.ceph.com -o noatime 2021-02-10T11:12:34.051 DEBUG:teuthology.orchestra.run.gibba020:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2021-02-10T11:12:34.408 DEBUG:teuthology.orchestra.run.gibba020:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2021-02-10T11:12:34.753 DEBUG:teuthology.orchestra.run.gibba020:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2021-02-10T11:12:35.096 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2021-02-10T11:12:35.096 INFO:tasks.ceph:role: osd.7 2021-02-10T11:12:35.096 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@gibba020.front.sepia.ceph.com 2021-02-10T11:12:35.096 DEBUG:teuthology.orchestra.run.gibba020:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2021-02-10T11:12:35.485 INFO:teuthology.orchestra.run.gibba020.stdout:Discarding blocks...Done. 2021-02-10T11:12:35.485 INFO:teuthology.orchestra.run.gibba020.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5505024 blks 2021-02-10T11:12:35.485 INFO:teuthology.orchestra.run.gibba020.stdout: = sectsz=512 attr=2, projid32bit=1 2021-02-10T11:12:35.485 INFO:teuthology.orchestra.run.gibba020.stdout: = crc=1 finobt=0, sparse=0 2021-02-10T11:12:35.485 INFO:teuthology.orchestra.run.gibba020.stdout:data = bsize=4096 blocks=22020096, imaxpct=25 2021-02-10T11:12:35.485 INFO:teuthology.orchestra.run.gibba020.stdout: = sunit=0 swidth=0 blks 2021-02-10T11:12:35.485 INFO:teuthology.orchestra.run.gibba020.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2021-02-10T11:12:35.485 INFO:teuthology.orchestra.run.gibba020.stdout:log =internal log bsize=4096 blocks=10752, version=2 2021-02-10T11:12:35.485 INFO:teuthology.orchestra.run.gibba020.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-02-10T11:12:35.486 INFO:teuthology.orchestra.run.gibba020.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-02-10T11:12:35.486 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@gibba020.front.sepia.ceph.com -o noatime 2021-02-10T11:12:35.487 DEBUG:teuthology.orchestra.run.gibba020:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2021-02-10T11:12:35.843 DEBUG:teuthology.orchestra.run.gibba020:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2021-02-10T11:12:36.194 DEBUG:teuthology.orchestra.run.gibba020:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2021-02-10T11:12:36.571 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:12:36.572 7faf9086da80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2021-02-10T11:12:36.572 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:12:36.572 7faf9086da80 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2021-02-10T11:12:36.572 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:12:36.573 7faf9086da80 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2021-02-10T11:12:36.572 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:12:36.573 7faf9086da80 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2021-02-10T11:12:36.572 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:12:36.573 7faf9086da80 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2021-02-10T11:12:36.572 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:12:36.573 7faf9086da80 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2021-02-10T11:12:38.867 DEBUG:teuthology.orchestra.run.gibba020:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2021-02-10T11:12:39.210 DEBUG:teuthology.orchestra.run.gibba020:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2021-02-10T11:12:39.581 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:12:39.582 7f388439ca80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2021-02-10T11:12:39.581 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:12:39.582 7f388439ca80 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2021-02-10T11:12:39.581 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:12:39.582 7f388439ca80 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2021-02-10T11:12:39.582 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:12:39.582 7f388439ca80 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2021-02-10T11:12:39.582 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:12:39.582 7f388439ca80 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2021-02-10T11:12:39.582 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:12:39.583 7f388439ca80 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2021-02-10T11:12:41.877 DEBUG:teuthology.orchestra.run.gibba020:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2021-02-10T11:12:42.221 DEBUG:teuthology.orchestra.run.gibba020:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2021-02-10T11:12:42.589 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:12:42.590 7ff9b42bba80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2021-02-10T11:12:42.589 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:12:42.590 7ff9b42bba80 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2021-02-10T11:12:42.590 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:12:42.591 7ff9b42bba80 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2021-02-10T11:12:42.590 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:12:42.591 7ff9b42bba80 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2021-02-10T11:12:42.590 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:12:42.591 7ff9b42bba80 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2021-02-10T11:12:42.590 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:12:42.591 7ff9b42bba80 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2021-02-10T11:12:44.881 DEBUG:teuthology.orchestra.run.gibba020:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2021-02-10T11:12:45.224 DEBUG:teuthology.orchestra.run.gibba020:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2021-02-10T11:12:45.594 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:12:45.595 7fe5939f6a80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2021-02-10T11:12:45.594 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:12:45.595 7fe5939f6a80 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2021-02-10T11:12:45.594 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:12:45.595 7fe5939f6a80 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2021-02-10T11:12:45.594 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:12:45.595 7fe5939f6a80 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2021-02-10T11:12:45.594 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:12:45.595 7fe5939f6a80 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2021-02-10T11:12:45.595 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:12:45.596 7fe5939f6a80 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2021-02-10T11:12:47.891 DEBUG:teuthology.orchestra.run.gibba020:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2021-02-10T11:12:48.233 INFO:tasks.ceph:Reading keys from all nodes... 2021-02-10T11:12:48.234 DEBUG:teuthology.orchestra.run.gibba012:> mktemp 2021-02-10T11:12:48.569 INFO:teuthology.orchestra.run.gibba012.stdout:/tmp/tmp.ZVOCT6DOA7 2021-02-10T11:12:48.570 DEBUG:teuthology.orchestra.run.gibba012:> sudo cp /var/lib/ceph/mgr/ceph-y/keyring /tmp/tmp.ZVOCT6DOA7 2021-02-10T11:12:48.914 DEBUG:teuthology.orchestra.run.gibba012:> sudo chmod 0666 /tmp/tmp.ZVOCT6DOA7 2021-02-10T11:12:49.592 DEBUG:teuthology.orchestra.remote:gibba012:/tmp/tmp.ZVOCT6DOA7 is 56B 2021-02-10T11:12:49.932 DEBUG:teuthology.orchestra.run.gibba012:> rm -fr /tmp/tmp.ZVOCT6DOA7 2021-02-10T11:12:50.265 DEBUG:teuthology.orchestra.run.gibba012:> mktemp 2021-02-10T11:12:50.600 INFO:teuthology.orchestra.run.gibba012.stdout:/tmp/tmp.Aiol69q19z 2021-02-10T11:12:50.600 DEBUG:teuthology.orchestra.run.gibba012:> sudo cp /var/lib/ceph/mds/ceph-a/keyring /tmp/tmp.Aiol69q19z 2021-02-10T11:12:50.945 DEBUG:teuthology.orchestra.run.gibba012:> sudo chmod 0666 /tmp/tmp.Aiol69q19z 2021-02-10T11:12:51.624 DEBUG:teuthology.orchestra.remote:gibba012:/tmp/tmp.Aiol69q19z is 56B 2021-02-10T11:12:51.964 DEBUG:teuthology.orchestra.run.gibba012:> rm -fr /tmp/tmp.Aiol69q19z 2021-02-10T11:12:52.298 DEBUG:teuthology.orchestra.run.gibba012:> mktemp 2021-02-10T11:12:52.633 INFO:teuthology.orchestra.run.gibba012.stdout:/tmp/tmp.eeqm9oT9gk 2021-02-10T11:12:52.633 DEBUG:teuthology.orchestra.run.gibba012:> sudo cp /var/lib/ceph/osd/ceph-0/keyring /tmp/tmp.eeqm9oT9gk 2021-02-10T11:12:52.977 DEBUG:teuthology.orchestra.run.gibba012:> sudo chmod 0666 /tmp/tmp.eeqm9oT9gk 2021-02-10T11:12:53.658 DEBUG:teuthology.orchestra.remote:gibba012:/tmp/tmp.eeqm9oT9gk is 56B 2021-02-10T11:12:53.998 DEBUG:teuthology.orchestra.run.gibba012:> rm -fr /tmp/tmp.eeqm9oT9gk 2021-02-10T11:12:54.331 DEBUG:teuthology.orchestra.run.gibba012:> mktemp 2021-02-10T11:12:54.665 INFO:teuthology.orchestra.run.gibba012.stdout:/tmp/tmp.rUyMRPTx1e 2021-02-10T11:12:54.728 DEBUG:teuthology.orchestra.run.gibba012:> sudo cp /var/lib/ceph/osd/ceph-1/keyring /tmp/tmp.rUyMRPTx1e 2021-02-10T11:12:55.070 DEBUG:teuthology.orchestra.run.gibba012:> sudo chmod 0666 /tmp/tmp.rUyMRPTx1e 2021-02-10T11:12:55.749 DEBUG:teuthology.orchestra.remote:gibba012:/tmp/tmp.rUyMRPTx1e is 56B 2021-02-10T11:12:56.087 DEBUG:teuthology.orchestra.run.gibba012:> rm -fr /tmp/tmp.rUyMRPTx1e 2021-02-10T11:12:56.598 DEBUG:teuthology.orchestra.run.gibba012:> mktemp 2021-02-10T11:12:56.932 INFO:teuthology.orchestra.run.gibba012.stdout:/tmp/tmp.P5M23TaZj2 2021-02-10T11:12:56.932 DEBUG:teuthology.orchestra.run.gibba012:> sudo cp /var/lib/ceph/osd/ceph-2/keyring /tmp/tmp.P5M23TaZj2 2021-02-10T11:12:57.275 DEBUG:teuthology.orchestra.run.gibba012:> sudo chmod 0666 /tmp/tmp.P5M23TaZj2 2021-02-10T11:12:57.983 DEBUG:teuthology.orchestra.remote:gibba012:/tmp/tmp.P5M23TaZj2 is 56B 2021-02-10T11:12:58.351 DEBUG:teuthology.orchestra.run.gibba012:> rm -fr /tmp/tmp.P5M23TaZj2 2021-02-10T11:12:58.694 DEBUG:teuthology.orchestra.run.gibba012:> mktemp 2021-02-10T11:12:59.032 INFO:teuthology.orchestra.run.gibba012.stdout:/tmp/tmp.x4QOnsxNg6 2021-02-10T11:12:59.032 DEBUG:teuthology.orchestra.run.gibba012:> sudo cp /var/lib/ceph/osd/ceph-3/keyring /tmp/tmp.x4QOnsxNg6 2021-02-10T11:12:59.386 DEBUG:teuthology.orchestra.run.gibba012:> sudo chmod 0666 /tmp/tmp.x4QOnsxNg6 2021-02-10T11:13:00.069 DEBUG:teuthology.orchestra.remote:gibba012:/tmp/tmp.x4QOnsxNg6 is 56B 2021-02-10T11:13:00.137 DEBUG:teuthology.orchestra.run.gibba012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:13:00.140 DEBUG:teuthology.orchestra.run.gibba020:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:13:00.408 DEBUG:teuthology.orchestra.run.gibba012:> rm -fr /tmp/tmp.x4QOnsxNg6 2021-02-10T11:13:00.742 DEBUG:teuthology.orchestra.run.gibba020:> mktemp 2021-02-10T11:13:01.077 INFO:teuthology.orchestra.run.gibba020.stdout:/tmp/tmp.Bct1zeZq11 2021-02-10T11:13:01.077 DEBUG:teuthology.orchestra.run.gibba020:> sudo cp /var/lib/ceph/mgr/ceph-x/keyring /tmp/tmp.Bct1zeZq11 2021-02-10T11:13:01.422 DEBUG:teuthology.orchestra.run.gibba020:> sudo chmod 0666 /tmp/tmp.Bct1zeZq11 2021-02-10T11:13:02.102 DEBUG:teuthology.orchestra.remote:gibba020:/tmp/tmp.Bct1zeZq11 is 56B 2021-02-10T11:13:02.441 DEBUG:teuthology.orchestra.run.gibba020:> rm -fr /tmp/tmp.Bct1zeZq11 2021-02-10T11:13:02.775 DEBUG:teuthology.orchestra.run.gibba020:> mktemp 2021-02-10T11:13:03.109 INFO:teuthology.orchestra.run.gibba020.stdout:/tmp/tmp.01po5BCqG0 2021-02-10T11:13:03.110 DEBUG:teuthology.orchestra.run.gibba020:> sudo cp /var/lib/ceph/mds/ceph-b/keyring /tmp/tmp.01po5BCqG0 2021-02-10T11:13:03.453 DEBUG:teuthology.orchestra.run.gibba020:> sudo chmod 0666 /tmp/tmp.01po5BCqG0 2021-02-10T11:13:04.133 DEBUG:teuthology.orchestra.remote:gibba020:/tmp/tmp.01po5BCqG0 is 56B 2021-02-10T11:13:04.473 DEBUG:teuthology.orchestra.run.gibba020:> rm -fr /tmp/tmp.01po5BCqG0 2021-02-10T11:13:04.807 DEBUG:teuthology.orchestra.run.gibba020:> mktemp 2021-02-10T11:13:05.140 INFO:teuthology.orchestra.run.gibba020.stdout:/tmp/tmp.gXHqwlwpKB 2021-02-10T11:13:05.141 DEBUG:teuthology.orchestra.run.gibba020:> sudo cp /var/lib/ceph/osd/ceph-4/keyring /tmp/tmp.gXHqwlwpKB 2021-02-10T11:13:05.485 DEBUG:teuthology.orchestra.run.gibba020:> sudo chmod 0666 /tmp/tmp.gXHqwlwpKB 2021-02-10T11:13:06.167 DEBUG:teuthology.orchestra.remote:gibba020:/tmp/tmp.gXHqwlwpKB is 56B 2021-02-10T11:13:06.507 DEBUG:teuthology.orchestra.run.gibba020:> rm -fr /tmp/tmp.gXHqwlwpKB 2021-02-10T11:13:06.841 DEBUG:teuthology.orchestra.run.gibba020:> mktemp 2021-02-10T11:13:07.175 INFO:teuthology.orchestra.run.gibba020.stdout:/tmp/tmp.VttJAmPZMc 2021-02-10T11:13:07.175 DEBUG:teuthology.orchestra.run.gibba020:> sudo cp /var/lib/ceph/osd/ceph-5/keyring /tmp/tmp.VttJAmPZMc 2021-02-10T11:13:07.520 DEBUG:teuthology.orchestra.run.gibba020:> sudo chmod 0666 /tmp/tmp.VttJAmPZMc 2021-02-10T11:13:08.199 DEBUG:teuthology.orchestra.remote:gibba020:/tmp/tmp.VttJAmPZMc is 56B 2021-02-10T11:13:08.538 DEBUG:teuthology.orchestra.run.gibba020:> rm -fr /tmp/tmp.VttJAmPZMc 2021-02-10T11:13:08.871 DEBUG:teuthology.orchestra.run.gibba020:> mktemp 2021-02-10T11:13:09.206 INFO:teuthology.orchestra.run.gibba020.stdout:/tmp/tmp.FVaFUEZynQ 2021-02-10T11:13:09.206 DEBUG:teuthology.orchestra.run.gibba020:> sudo cp /var/lib/ceph/osd/ceph-6/keyring /tmp/tmp.FVaFUEZynQ 2021-02-10T11:13:09.550 DEBUG:teuthology.orchestra.run.gibba020:> sudo chmod 0666 /tmp/tmp.FVaFUEZynQ 2021-02-10T11:13:10.229 DEBUG:teuthology.orchestra.remote:gibba020:/tmp/tmp.FVaFUEZynQ is 56B 2021-02-10T11:13:10.568 DEBUG:teuthology.orchestra.run.gibba020:> rm -fr /tmp/tmp.FVaFUEZynQ 2021-02-10T11:13:10.902 DEBUG:teuthology.orchestra.run.gibba020:> mktemp 2021-02-10T11:13:11.237 INFO:teuthology.orchestra.run.gibba020.stdout:/tmp/tmp.61EU2tAvJR 2021-02-10T11:13:11.237 DEBUG:teuthology.orchestra.run.gibba020:> sudo cp /var/lib/ceph/osd/ceph-7/keyring /tmp/tmp.61EU2tAvJR 2021-02-10T11:13:11.580 DEBUG:teuthology.orchestra.run.gibba020:> sudo chmod 0666 /tmp/tmp.61EU2tAvJR 2021-02-10T11:13:12.258 DEBUG:teuthology.orchestra.remote:gibba020:/tmp/tmp.61EU2tAvJR is 56B 2021-02-10T11:13:12.598 DEBUG:teuthology.orchestra.run.gibba020:> rm -fr /tmp/tmp.61EU2tAvJR 2021-02-10T11:13:13.271 DEBUG:teuthology.orchestra.remote:gibba012:/etc/ceph/ceph.client.0.keyring is 59B 2021-02-10T11:13:13.950 DEBUG:teuthology.orchestra.remote:gibba012:/etc/ceph/ceph.client.1.keyring is 59B 2021-02-10T11:13:14.626 DEBUG:teuthology.orchestra.remote:gibba020:/etc/ceph/ceph.client.2.keyring is 59B 2021-02-10T11:13:15.305 DEBUG:teuthology.orchestra.remote:gibba020:/etc/ceph/ceph.client.3.keyring is 59B 2021-02-10T11:13:15.644 INFO:tasks.ceph:Adding keys to all mons... 2021-02-10T11:13:15.644 DEBUG:teuthology.orchestra.run.gibba012:> sudo tee -a /etc/ceph/ceph.keyring 2021-02-10T11:13:15.649 DEBUG:teuthology.orchestra.run.gibba020:> sudo tee -a /etc/ceph/ceph.keyring 2021-02-10T11:13:15.987 INFO:teuthology.orchestra.run.gibba012.stdout:[mgr.y] 2021-02-10T11:13:15.987 INFO:teuthology.orchestra.run.gibba012.stdout: key = AQCJvyNgQ1tTDhAASwBssDy+Rz31ujw8fNlWow== 2021-02-10T11:13:15.987 INFO:teuthology.orchestra.run.gibba012.stdout:[mds.a] 2021-02-10T11:13:15.987 INFO:teuthology.orchestra.run.gibba012.stdout: key = AQCKvyNgPkV4AxAAEmVNaghdccP6lWd8hvQnxw== 2021-02-10T11:13:15.987 INFO:teuthology.orchestra.run.gibba012.stdout:[osd.0] 2021-02-10T11:13:15.988 INFO:teuthology.orchestra.run.gibba012.stdout: key = AQCSvyNgXcKPLhAAP/Bj2F+06DCtPCzQDNq+cQ== 2021-02-10T11:13:15.988 INFO:teuthology.orchestra.run.gibba012.stdout:[osd.1] 2021-02-10T11:13:15.988 INFO:teuthology.orchestra.run.gibba012.stdout: key = AQCVvyNghfbcLhAA66PrRwR2M7o6VTF5UhU/Fg== 2021-02-10T11:13:15.988 INFO:teuthology.orchestra.run.gibba012.stdout:[osd.2] 2021-02-10T11:13:15.988 INFO:teuthology.orchestra.run.gibba012.stdout: key = AQCYvyNgwiPqLhAA66BVkRsgaUlGpAsQ15Nf8Q== 2021-02-10T11:13:15.988 INFO:teuthology.orchestra.run.gibba012.stdout:[osd.3] 2021-02-10T11:13:15.988 INFO:teuthology.orchestra.run.gibba012.stdout: key = AQCbvyNgqBZrLxAAERPe/1L3SXAEzI3d9ReHHg== 2021-02-10T11:13:15.988 INFO:teuthology.orchestra.run.gibba012.stdout:[mgr.x] 2021-02-10T11:13:15.988 INFO:teuthology.orchestra.run.gibba012.stdout: key = AQCJvyNgTPVwKBAAjduiBE8+RSXBv82tdPecWg== 2021-02-10T11:13:15.988 INFO:teuthology.orchestra.run.gibba012.stdout:[mds.b] 2021-02-10T11:13:15.988 INFO:teuthology.orchestra.run.gibba012.stdout: key = AQCKvyNguxONLhAAHIpry+R89MAwdhFJyEhVCw== 2021-02-10T11:13:15.988 INFO:teuthology.orchestra.run.gibba012.stdout:[osd.4] 2021-02-10T11:13:15.988 INFO:teuthology.orchestra.run.gibba012.stdout: key = AQCkvyNgDQsxIhAAHmVjVYnx20N4SDmKqvbcuA== 2021-02-10T11:13:15.989 INFO:teuthology.orchestra.run.gibba012.stdout:[osd.5] 2021-02-10T11:13:15.989 INFO:teuthology.orchestra.run.gibba012.stdout: key = AQCnvyNg8ITFIhAADd4aKybAV7btoGPx3kCg3A== 2021-02-10T11:13:15.989 INFO:teuthology.orchestra.run.gibba012.stdout:[osd.6] 2021-02-10T11:13:15.989 INFO:teuthology.orchestra.run.gibba012.stdout: key = AQCqvyNgxOhDIxAAzXixEa/nlDc1xXrg0aeLwg== 2021-02-10T11:13:15.989 INFO:teuthology.orchestra.run.gibba012.stdout:[osd.7] 2021-02-10T11:13:15.989 INFO:teuthology.orchestra.run.gibba012.stdout: key = AQCtvyNgxSqGIxAACxJQlwP7DpiVVCZHA3zn1A== 2021-02-10T11:13:15.989 INFO:teuthology.orchestra.run.gibba012.stdout:[client.0] 2021-02-10T11:13:15.989 INFO:teuthology.orchestra.run.gibba012.stdout: key = AQCLvyNgDkNiHRAAygU8Za4PBusTqCgwAYWTiQ== 2021-02-10T11:13:15.989 INFO:teuthology.orchestra.run.gibba012.stdout:[client.1] 2021-02-10T11:13:15.989 INFO:teuthology.orchestra.run.gibba012.stdout: key = AQCLvyNgxmLqMxAA1MlSDJ9LH9u4f2sL7mm8/Q== 2021-02-10T11:13:15.989 INFO:teuthology.orchestra.run.gibba012.stdout:[client.2] 2021-02-10T11:13:15.989 INFO:teuthology.orchestra.run.gibba012.stdout: key = AQCMvyNgp8veDhAAniSVHYPtwwTog/pjbaRQmw== 2021-02-10T11:13:15.989 INFO:teuthology.orchestra.run.gibba012.stdout:[client.3] 2021-02-10T11:13:15.989 INFO:teuthology.orchestra.run.gibba012.stdout: key = AQCMvyNgQEVoJRAApHcWdm1sldIxu+heyD4HdQ== 2021-02-10T11:13:15.990 INFO:teuthology.orchestra.run.gibba020.stdout:[mgr.y] 2021-02-10T11:13:15.991 INFO:teuthology.orchestra.run.gibba020.stdout: key = AQCJvyNgQ1tTDhAASwBssDy+Rz31ujw8fNlWow== 2021-02-10T11:13:15.991 INFO:teuthology.orchestra.run.gibba020.stdout:[mds.a] 2021-02-10T11:13:15.991 INFO:teuthology.orchestra.run.gibba020.stdout: key = AQCKvyNgPkV4AxAAEmVNaghdccP6lWd8hvQnxw== 2021-02-10T11:13:15.991 INFO:teuthology.orchestra.run.gibba020.stdout:[osd.0] 2021-02-10T11:13:15.991 INFO:teuthology.orchestra.run.gibba020.stdout: key = AQCSvyNgXcKPLhAAP/Bj2F+06DCtPCzQDNq+cQ== 2021-02-10T11:13:15.991 INFO:teuthology.orchestra.run.gibba020.stdout:[osd.1] 2021-02-10T11:13:15.991 INFO:teuthology.orchestra.run.gibba020.stdout: key = AQCVvyNghfbcLhAA66PrRwR2M7o6VTF5UhU/Fg== 2021-02-10T11:13:15.991 INFO:teuthology.orchestra.run.gibba020.stdout:[osd.2] 2021-02-10T11:13:15.991 INFO:teuthology.orchestra.run.gibba020.stdout: key = AQCYvyNgwiPqLhAA66BVkRsgaUlGpAsQ15Nf8Q== 2021-02-10T11:13:15.991 INFO:teuthology.orchestra.run.gibba020.stdout:[osd.3] 2021-02-10T11:13:15.991 INFO:teuthology.orchestra.run.gibba020.stdout: key = AQCbvyNgqBZrLxAAERPe/1L3SXAEzI3d9ReHHg== 2021-02-10T11:13:15.991 INFO:teuthology.orchestra.run.gibba020.stdout:[mgr.x] 2021-02-10T11:13:15.992 INFO:teuthology.orchestra.run.gibba020.stdout: key = AQCJvyNgTPVwKBAAjduiBE8+RSXBv82tdPecWg== 2021-02-10T11:13:15.992 INFO:teuthology.orchestra.run.gibba020.stdout:[mds.b] 2021-02-10T11:13:15.992 INFO:teuthology.orchestra.run.gibba020.stdout: key = AQCKvyNguxONLhAAHIpry+R89MAwdhFJyEhVCw== 2021-02-10T11:13:15.992 INFO:teuthology.orchestra.run.gibba020.stdout:[osd.4] 2021-02-10T11:13:15.992 INFO:teuthology.orchestra.run.gibba020.stdout: key = AQCkvyNgDQsxIhAAHmVjVYnx20N4SDmKqvbcuA== 2021-02-10T11:13:15.992 INFO:teuthology.orchestra.run.gibba020.stdout:[osd.5] 2021-02-10T11:13:15.992 INFO:teuthology.orchestra.run.gibba020.stdout: key = AQCnvyNg8ITFIhAADd4aKybAV7btoGPx3kCg3A== 2021-02-10T11:13:15.992 INFO:teuthology.orchestra.run.gibba020.stdout:[osd.6] 2021-02-10T11:13:15.992 INFO:teuthology.orchestra.run.gibba020.stdout: key = AQCqvyNgxOhDIxAAzXixEa/nlDc1xXrg0aeLwg== 2021-02-10T11:13:15.992 INFO:teuthology.orchestra.run.gibba020.stdout:[osd.7] 2021-02-10T11:13:15.992 INFO:teuthology.orchestra.run.gibba020.stdout: key = AQCtvyNgxSqGIxAACxJQlwP7DpiVVCZHA3zn1A== 2021-02-10T11:13:15.992 INFO:teuthology.orchestra.run.gibba020.stdout:[client.0] 2021-02-10T11:13:15.992 INFO:teuthology.orchestra.run.gibba020.stdout: key = AQCLvyNgDkNiHRAAygU8Za4PBusTqCgwAYWTiQ== 2021-02-10T11:13:15.992 INFO:teuthology.orchestra.run.gibba020.stdout:[client.1] 2021-02-10T11:13:15.993 INFO:teuthology.orchestra.run.gibba020.stdout: key = AQCLvyNgxmLqMxAA1MlSDJ9LH9u4f2sL7mm8/Q== 2021-02-10T11:13:15.993 INFO:teuthology.orchestra.run.gibba020.stdout:[client.2] 2021-02-10T11:13:15.993 INFO:teuthology.orchestra.run.gibba020.stdout: key = AQCMvyNgp8veDhAAniSVHYPtwwTog/pjbaRQmw== 2021-02-10T11:13:15.993 INFO:teuthology.orchestra.run.gibba020.stdout:[client.3] 2021-02-10T11:13:15.993 INFO:teuthology.orchestra.run.gibba020.stdout: key = AQCMvyNgQEVoJRAApHcWdm1sldIxu+heyD4HdQ== 2021-02-10T11:13:15.994 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2021-02-10T11:13:15.997 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2021-02-10T11:13:16.371 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2021-02-10T11:13:16.375 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2021-02-10T11:13:16.750 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-02-10T11:13:16.753 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-02-10T11:13:17.128 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-02-10T11:13:17.131 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-02-10T11:13:17.505 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-02-10T11:13:17.508 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-02-10T11:13:17.883 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-02-10T11:13:17.885 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-02-10T11:13:18.259 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2021-02-10T11:13:18.263 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2021-02-10T11:13:18.637 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2021-02-10T11:13:18.640 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2021-02-10T11:13:19.020 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-02-10T11:13:19.043 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-02-10T11:13:19.421 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-02-10T11:13:19.425 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-02-10T11:13:19.799 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-02-10T11:13:19.802 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-02-10T11:13:20.174 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-02-10T11:13:20.178 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-02-10T11:13:20.552 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-02-10T11:13:20.555 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-02-10T11:13:20.931 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-02-10T11:13:20.934 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-02-10T11:13:21.310 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-02-10T11:13:21.314 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-02-10T11:13:21.690 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-02-10T11:13:21.693 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-02-10T11:13:22.068 INFO:tasks.ceph:Running mkfs on mon nodes... 2021-02-10T11:13:22.068 DEBUG:teuthology.orchestra.run.gibba012:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2021-02-10T11:13:22.411 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2021-02-10T11:13:22.792 DEBUG:teuthology.orchestra.run.gibba012:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2021-02-10T11:13:23.135 DEBUG:teuthology.orchestra.run.gibba020:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2021-02-10T11:13:23.480 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2021-02-10T11:13:23.861 DEBUG:teuthology.orchestra.run.gibba020:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2021-02-10T11:13:24.205 DEBUG:teuthology.orchestra.run.gibba020:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2021-02-10T11:13:24.556 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2021-02-10T11:13:24.949 DEBUG:teuthology.orchestra.run.gibba020:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2021-02-10T11:13:25.304 DEBUG:teuthology.orchestra.run.gibba012:> rm -- /home/ubuntu/cephtest/ceph.monmap 2021-02-10T11:13:25.316 DEBUG:teuthology.orchestra.run.gibba020:> rm -- /home/ubuntu/cephtest/ceph.monmap 2021-02-10T11:13:25.653 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2021-02-10T11:13:25.654 INFO:tasks.ceph.mon.a:Restarting daemon 2021-02-10T11:13:25.654 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2021-02-10T11:13:25.662 INFO:tasks.ceph.mon.a:Started 2021-02-10T11:13:25.662 INFO:tasks.ceph.mon.b:Restarting daemon 2021-02-10T11:13:25.662 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2021-02-10T11:13:25.666 INFO:tasks.ceph.mon.b:Started 2021-02-10T11:13:25.666 INFO:tasks.ceph.mon.c:Restarting daemon 2021-02-10T11:13:25.666 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2021-02-10T11:13:25.670 INFO:tasks.ceph.mon.c:Started 2021-02-10T11:13:25.670 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2021-02-10T11:13:25.671 INFO:tasks.ceph.mgr.y:Restarting daemon 2021-02-10T11:13:25.671 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2021-02-10T11:13:25.678 INFO:tasks.ceph.mgr.y:Started 2021-02-10T11:13:25.678 INFO:tasks.ceph.mgr.x:Restarting daemon 2021-02-10T11:13:25.678 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2021-02-10T11:13:25.683 INFO:tasks.ceph.mgr.x:Started 2021-02-10T11:13:25.683 INFO:tasks.ceph:Setting crush tunables to default 2021-02-10T11:13:25.684 DEBUG:teuthology.orchestra.run.gibba012:> sudo ceph --cluster ceph osd crush tunables default 2021-02-10T11:13:26.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:26.074 7fd2ba49f700 1 Processor -- start 2021-02-10T11:13:26.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:26.074 7fd2ba49f700 1 -- start start 2021-02-10T11:13:26.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:26.074 7fd2ba49f700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd2b40cff00 0x7fd2b41b39d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:26.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:26.074 7fd2ba49f700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fd2b40d1bf0 0x7fd2b40ef620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:26.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:26.074 7fd2ba49f700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2b40f0b20 con 0x7fd2b40d17f0 2021-02-10T11:13:26.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:26.074 7fd2ba49f700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2b41b3ff0 con 0x7fd2b40d1bf0 2021-02-10T11:13:26.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:26.074 7fd2ba49f700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2b40f20b0 con 0x7fd2b40cff00 2021-02-10T11:13:26.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:26.075 7fd2b8abc700 1 -- >> v2:172.21.2.112:3300/0 conn(0x7fd2b40d1bf0 msgr2=0x7fd2b40ef620 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.2.112:3300/0 2021-02-10T11:13:26.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:26.075 7fd2b8abc700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fd2b40d1bf0 0x7fd2b40ef620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-02-10T11:13:26.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:26.075 7fd2b3fff700 1 -- >> v1:172.21.2.112:6789/0 conn(0x7fd2b40d17f0 legacy=0x7fd2b41b31e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.2.112:6789/0 2021-02-10T11:13:26.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:26.075 7fd2b37fe700 1 -- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd2b40cff00 msgr2=0x7fd2b41b39d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.2.120:3301/0 2021-02-10T11:13:26.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:26.075 7fd2b37fe700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd2b40cff00 0x7fd2b41b39d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-02-10T11:13:26.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:26.276 7fd2b8abc700 1 -- >> v2:172.21.2.112:3300/0 conn(0x7fd2b40d1bf0 msgr2=0x7fd2b40ef620 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.2.112:3300/0 2021-02-10T11:13:26.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:26.276 7fd2b3fff700 1 -- >> v1:172.21.2.112:6789/0 conn(0x7fd2b40d17f0 legacy=0x7fd2b41b31e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.2.112:6789/0 2021-02-10T11:13:26.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:26.276 7fd2b8abc700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fd2b40d1bf0 0x7fd2b40ef620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2021-02-10T11:13:26.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:26.276 7fd2b37fe700 1 -- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd2b40cff00 msgr2=0x7fd2b41b39d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.2.120:3301/0 2021-02-10T11:13:26.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:26.276 7fd2b37fe700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd2b40cff00 0x7fd2b41b39d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2021-02-10T11:13:26.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:26.677 7fd2b3fff700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fd2b40d17f0 0x7fd2b41b31e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:41130/0 (socket says 172.21.2.112:41130) 2021-02-10T11:13:26.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:26.677 7fd2b3fff700 1 -- 172.21.2.112:0/2030533620 learned_addr learned my addr 172.21.2.112:0/2030533620 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:13:26.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:26.677 7fd2b8abc700 1 --2- 172.21.2.112:0/2030533620 >> v2:172.21.2.112:3300/0 conn(0x7fd2b40d1bf0 0x7fd2b40ef620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:26.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:26.677 7fd2b37fe700 1 --2- 172.21.2.112:0/2030533620 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd2b40cff00 0x7fd2b41b39d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:26.695 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:26.677 7fd2b8abc700 1 -- 172.21.2.112:0/2030533620 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd2b40cff00 msgr2=0x7fd2b41b39d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:13:26.695 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:26.677 7fd2b8abc700 1 --2- 172.21.2.112:0/2030533620 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd2b40cff00 0x7fd2b41b39d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:26.695 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:26.677 7fd2b8abc700 1 -- 172.21.2.112:0/2030533620 >> v1:172.21.2.112:6789/0 conn(0x7fd2b40d17f0 legacy=0x7fd2b41b31e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:26.695 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:26.677 7fd2b8abc700 1 -- 172.21.2.112:0/2030533620 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2b40f22a0 con 0x7fd2b40d1bf0 2021-02-10T11:13:26.695 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:26.677 7fd2b2ffd700 1 -- 172.21.2.112:0/2030533620 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2217309931 0 0) 0x7fd2a4003600 con 0x7fd2b40d17f0 2021-02-10T11:13:26.695 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:26.677 7fd2b37fe700 1 --2- 172.21.2.112:0/2030533620 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd2b40cff00 0x7fd2b41b39d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:13:26.695 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:26.678 7fd2b8abc700 1 --2- 172.21.2.112:0/2030533620 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd2b40d1bf0 0x7fd2b40ef620 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fd2ac006a70 tx=0x7fd2ac009ed0).ready entity=mon.0 client_cookie=35c62c6874030a2a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:13:26.695 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:26.678 7fd2b2ffd700 1 -- 172.21.2.112:0/2030533620 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd2ac0038a0 con 0x7fd2b40d1bf0 2021-02-10T11:13:26.695 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:26.678 7fd2b2ffd700 1 -- 172.21.2.112:0/2030533620 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2ac003ba0 con 0x7fd2b40d1bf0 2021-02-10T11:13:26.695 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:26.678 7fd2b2ffd700 1 -- 172.21.2.112:0/2030533620 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd2ac0062a0 con 0x7fd2b40d1bf0 2021-02-10T11:13:26.695 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:26.678 7fd2ba49f700 1 -- 172.21.2.112:0/2030533620 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd2b40d1bf0 msgr2=0x7fd2b40ef620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:26.695 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:26.678 7fd2ba49f700 1 --2- 172.21.2.112:0/2030533620 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd2b40d1bf0 0x7fd2b40ef620 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fd2ac006a70 tx=0x7fd2ac009ed0).stop 2021-02-10T11:13:26.695 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:26.679 7fd2ba49f700 1 -- 172.21.2.112:0/2030533620 shutdown_connections 2021-02-10T11:13:26.695 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:26.679 7fd2ba49f700 1 --2- 172.21.2.112:0/2030533620 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd2b40d1bf0 0x7fd2b40ef620 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:26.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:26.679 7fd2ba49f700 1 --2- 172.21.2.112:0/2030533620 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd2b40cff00 0x7fd2b41b39d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:26.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:26.679 7fd2ba49f700 1 -- 172.21.2.112:0/2030533620 >> 172.21.2.112:0/2030533620 conn(0x7fd2b40fde40 msgr2=0x7fd2b40ce390 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:13:26.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:26.679 7fd2ba49f700 1 -- 172.21.2.112:0/2030533620 shutdown_connections 2021-02-10T11:13:26.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:26.679 7fd2ba49f700 1 -- 172.21.2.112:0/2030533620 wait complete. 2021-02-10T11:13:26.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:26.694 7fd2ba49f700 1 Processor -- start 2021-02-10T11:13:26.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:26.694 7fd2ba49f700 1 -- start start 2021-02-10T11:13:26.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:26.695 7fd2ba49f700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd2b419ffe0 0x7fd2b41a03e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:26.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:26.695 7fd2ba49f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd2b4190340 0x7fd2b4190740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:26.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:26.695 7fd2ba49f700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd2b4190d60 0x7fd2b4191160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:26.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:26.695 7fd2ba49f700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2b4191850 con 0x7fd2b4190340 2021-02-10T11:13:26.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:26.695 7fd2ba49f700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2b40c3c30 con 0x7fd2b419ffe0 2021-02-10T11:13:26.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:26.695 7fd2ba49f700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2b4191d50 con 0x7fd2b4190d60 2021-02-10T11:13:26.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:26.695 7fd2b37fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd2b4190340 0x7fd2b4190740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:26.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:26.695 7fd2b37fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd2b4190340 0x7fd2b4190740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:41672/0 (socket says 172.21.2.112:41672) 2021-02-10T11:13:26.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:26.695 7fd2b37fe700 1 -- 172.21.2.112:0/3799804768 learned_addr learned my addr 172.21.2.112:0/3799804768 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:13:26.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:26.695 7fd2b3fff700 1 --2- 172.21.2.112:0/3799804768 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd2b419ffe0 0x7fd2b41a03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:26.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:26.695 7fd2b8abc700 1 --2- 172.21.2.112:0/3799804768 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd2b4190d60 0x7fd2b4191160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:26.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:26.696 7fd2b37fe700 1 -- 172.21.2.112:0/3799804768 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd2b4190d60 msgr2=0x7fd2b4191160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:26.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:26.696 7fd2b37fe700 1 --2- 172.21.2.112:0/3799804768 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd2b4190d60 0x7fd2b4191160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:26.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:26.696 7fd2b37fe700 1 -- 172.21.2.112:0/3799804768 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd2b419ffe0 msgr2=0x7fd2b41a03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:26.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:26.696 7fd2b37fe700 1 --2- 172.21.2.112:0/3799804768 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd2b419ffe0 0x7fd2b41a03e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:26.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:26.696 7fd2b37fe700 1 -- 172.21.2.112:0/3799804768 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2b4191f70 con 0x7fd2b4190340 2021-02-10T11:13:26.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:26.696 7fd2b8abc700 1 --2- 172.21.2.112:0/3799804768 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd2b4190d60 0x7fd2b4191160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:13:26.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:26.696 7fd2b3fff700 1 --2- 172.21.2.112:0/3799804768 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd2b419ffe0 0x7fd2b41a03e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:13:26.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:26.696 7fd2b37fe700 1 --2- 172.21.2.112:0/3799804768 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd2b4190340 0x7fd2b4190740 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fd2a8006a50 tx=0x7fd2a800b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:13:26.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:26.696 7fd2b0ff9700 1 -- 172.21.2.112:0/3799804768 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd2a80009b0 con 0x7fd2b4190340 2021-02-10T11:13:26.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:26.696 7fd2b0ff9700 1 -- 172.21.2.112:0/3799804768 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2a800bd80 con 0x7fd2b4190340 2021-02-10T11:13:26.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:26.696 7fd2b0ff9700 1 -- 172.21.2.112:0/3799804768 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd2a8003380 con 0x7fd2b4190340 2021-02-10T11:13:26.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:26.696 7fd2ba49f700 1 -- 172.21.2.112:0/3799804768 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2b4191f70 con 0x7fd2b4190340 2021-02-10T11:13:26.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:26.696 7fd2ba49f700 1 -- 172.21.2.112:0/3799804768 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2b4191f70 con 0x7fd2b4190340 2021-02-10T11:13:26.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:26.697 7fd2b0ff9700 1 -- 172.21.2.112:0/3799804768 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 1) v1 ==== 190+0+0 (secure 0 0 0) 0x7fd2a8006030 con 0x7fd2b4190340 2021-02-10T11:13:26.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:26.697 7fd2b0ff9700 1 -- 172.21.2.112:0/3799804768 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fd2a80036f0 con 0x7fd2b4190340 2021-02-10T11:13:26.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:26.726 7fd2ba49f700 1 -- 172.21.2.112:0/3799804768 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2b4103690 con 0x7fd2b4190340 2021-02-10T11:13:26.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:26.728 7fd2b0ff9700 1 -- 172.21.2.112:0/3799804768 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+66016 (secure 0 0 0) 0x7fd2a8006a90 con 0x7fd2b4190340 2021-02-10T11:13:26.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:26.798 7fd2ba49f700 1 -- 172.21.2.112:0/3799804768 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"profile": "default", "prefix": "osd crush tunables"} v 0) v1 -- 0x7fd2b4103690 con 0x7fd2b4190340 2021-02-10T11:13:26.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:26.801 7fd2b0ff9700 1 -- 172.21.2.112:0/3799804768 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"profile": "default", "prefix": "osd crush tunables"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7fd2a8000940 con 0x7fd2b4190340 2021-02-10T11:13:26.800 INFO:teuthology.orchestra.run.gibba012.stderr:adjusted tunables profile to default 2021-02-10T11:13:26.801 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:26.802 7fd2ba49f700 1 -- 172.21.2.112:0/3799804768 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd2b4190340 msgr2=0x7fd2b4190740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:26.801 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:26.802 7fd2ba49f700 1 --2- 172.21.2.112:0/3799804768 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd2b4190340 0x7fd2b4190740 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fd2a8006a50 tx=0x7fd2a800b990).stop 2021-02-10T11:13:26.801 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:26.802 7fd2ba49f700 1 -- 172.21.2.112:0/3799804768 shutdown_connections 2021-02-10T11:13:26.801 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:26.802 7fd2ba49f700 1 --2- 172.21.2.112:0/3799804768 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd2b4190d60 0x7fd2b4191160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:26.802 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:26.802 7fd2ba49f700 1 --2- 172.21.2.112:0/3799804768 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd2b4190340 0x7fd2b4190740 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:26.802 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:26.802 7fd2ba49f700 1 --2- 172.21.2.112:0/3799804768 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd2b419ffe0 0x7fd2b41a03e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:26.802 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:26.802 7fd2ba49f700 1 -- 172.21.2.112:0/3799804768 >> 172.21.2.112:0/3799804768 conn(0x7fd2b40a1690 msgr2=0x7fd2b40a94c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:13:26.802 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:26.803 7fd2ba49f700 1 -- 172.21.2.112:0/3799804768 shutdown_connections 2021-02-10T11:13:26.802 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:26.803 7fd2ba49f700 1 -- 172.21.2.112:0/3799804768 wait complete. 2021-02-10T11:13:26.809 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2021-02-10T11:13:26.810 DEBUG:teuthology.orchestra.run.gibba012:> mktemp 2021-02-10T11:13:27.142 INFO:teuthology.orchestra.run.gibba012.stdout:/tmp/tmp.G7TRga0ldj 2021-02-10T11:13:27.143 DEBUG:teuthology.orchestra.run.gibba012:> sudo cp /var/lib/ceph/osd/ceph-0/fsid /tmp/tmp.G7TRga0ldj 2021-02-10T11:13:27.486 DEBUG:teuthology.orchestra.run.gibba012:> sudo chmod 0666 /tmp/tmp.G7TRga0ldj 2021-02-10T11:13:28.165 DEBUG:teuthology.orchestra.remote:gibba012:/tmp/tmp.G7TRga0ldj is 37B 2021-02-10T11:13:28.504 DEBUG:teuthology.orchestra.run.gibba012:> rm -fr /tmp/tmp.G7TRga0ldj 2021-02-10T11:13:28.839 DEBUG:teuthology.orchestra.run.gibba012:> mktemp 2021-02-10T11:13:29.172 INFO:teuthology.orchestra.run.gibba012.stdout:/tmp/tmp.DPMsOlWTeg 2021-02-10T11:13:29.173 DEBUG:teuthology.orchestra.run.gibba012:> sudo cp /var/lib/ceph/osd/ceph-1/fsid /tmp/tmp.DPMsOlWTeg 2021-02-10T11:13:29.517 DEBUG:teuthology.orchestra.run.gibba012:> sudo chmod 0666 /tmp/tmp.DPMsOlWTeg 2021-02-10T11:13:30.197 DEBUG:teuthology.orchestra.remote:gibba012:/tmp/tmp.DPMsOlWTeg is 37B 2021-02-10T11:13:30.507 DEBUG:teuthology.orchestra.run.gibba012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:13:30.511 DEBUG:teuthology.orchestra.run.gibba020:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:13:30.536 DEBUG:teuthology.orchestra.run.gibba012:> rm -fr /tmp/tmp.DPMsOlWTeg 2021-02-10T11:13:30.843 DEBUG:teuthology.orchestra.run.gibba012:> mktemp 2021-02-10T11:13:31.178 INFO:teuthology.orchestra.run.gibba012.stdout:/tmp/tmp.ar2fTPMNmi 2021-02-10T11:13:31.178 DEBUG:teuthology.orchestra.run.gibba012:> sudo cp /var/lib/ceph/osd/ceph-2/fsid /tmp/tmp.ar2fTPMNmi 2021-02-10T11:13:31.520 DEBUG:teuthology.orchestra.run.gibba012:> sudo chmod 0666 /tmp/tmp.ar2fTPMNmi 2021-02-10T11:13:32.198 DEBUG:teuthology.orchestra.remote:gibba012:/tmp/tmp.ar2fTPMNmi is 37B 2021-02-10T11:13:32.538 DEBUG:teuthology.orchestra.run.gibba012:> rm -fr /tmp/tmp.ar2fTPMNmi 2021-02-10T11:13:32.872 DEBUG:teuthology.orchestra.run.gibba012:> mktemp 2021-02-10T11:13:33.205 INFO:teuthology.orchestra.run.gibba012.stdout:/tmp/tmp.T1O8vtCTfu 2021-02-10T11:13:33.205 DEBUG:teuthology.orchestra.run.gibba012:> sudo cp /var/lib/ceph/osd/ceph-3/fsid /tmp/tmp.T1O8vtCTfu 2021-02-10T11:13:33.550 DEBUG:teuthology.orchestra.run.gibba012:> sudo chmod 0666 /tmp/tmp.T1O8vtCTfu 2021-02-10T11:13:34.231 DEBUG:teuthology.orchestra.remote:gibba012:/tmp/tmp.T1O8vtCTfu is 37B 2021-02-10T11:13:34.570 DEBUG:teuthology.orchestra.run.gibba012:> rm -fr /tmp/tmp.T1O8vtCTfu 2021-02-10T11:13:34.905 DEBUG:teuthology.orchestra.run.gibba020:> mktemp 2021-02-10T11:13:35.238 INFO:teuthology.orchestra.run.gibba020.stdout:/tmp/tmp.CAIQ8Uk3g0 2021-02-10T11:13:35.239 DEBUG:teuthology.orchestra.run.gibba020:> sudo cp /var/lib/ceph/osd/ceph-4/fsid /tmp/tmp.CAIQ8Uk3g0 2021-02-10T11:13:35.583 DEBUG:teuthology.orchestra.run.gibba020:> sudo chmod 0666 /tmp/tmp.CAIQ8Uk3g0 2021-02-10T11:13:36.263 DEBUG:teuthology.orchestra.remote:gibba020:/tmp/tmp.CAIQ8Uk3g0 is 37B 2021-02-10T11:13:36.602 DEBUG:teuthology.orchestra.run.gibba020:> rm -fr /tmp/tmp.CAIQ8Uk3g0 2021-02-10T11:13:36.937 DEBUG:teuthology.orchestra.run.gibba020:> mktemp 2021-02-10T11:13:37.271 INFO:teuthology.orchestra.run.gibba020.stdout:/tmp/tmp.t8Mbj3rKs8 2021-02-10T11:13:37.272 DEBUG:teuthology.orchestra.run.gibba020:> sudo cp /var/lib/ceph/osd/ceph-5/fsid /tmp/tmp.t8Mbj3rKs8 2021-02-10T11:13:37.615 DEBUG:teuthology.orchestra.run.gibba020:> sudo chmod 0666 /tmp/tmp.t8Mbj3rKs8 2021-02-10T11:13:38.295 DEBUG:teuthology.orchestra.remote:gibba020:/tmp/tmp.t8Mbj3rKs8 is 37B 2021-02-10T11:13:38.638 DEBUG:teuthology.orchestra.run.gibba020:> rm -fr /tmp/tmp.t8Mbj3rKs8 2021-02-10T11:13:38.972 DEBUG:teuthology.orchestra.run.gibba020:> mktemp 2021-02-10T11:13:39.305 INFO:teuthology.orchestra.run.gibba020.stdout:/tmp/tmp.UmxRDh95JQ 2021-02-10T11:13:39.306 DEBUG:teuthology.orchestra.run.gibba020:> sudo cp /var/lib/ceph/osd/ceph-6/fsid /tmp/tmp.UmxRDh95JQ 2021-02-10T11:13:39.650 DEBUG:teuthology.orchestra.run.gibba020:> sudo chmod 0666 /tmp/tmp.UmxRDh95JQ 2021-02-10T11:13:40.331 DEBUG:teuthology.orchestra.remote:gibba020:/tmp/tmp.UmxRDh95JQ is 37B 2021-02-10T11:13:40.670 DEBUG:teuthology.orchestra.run.gibba020:> rm -fr /tmp/tmp.UmxRDh95JQ 2021-02-10T11:13:41.005 DEBUG:teuthology.orchestra.run.gibba020:> mktemp 2021-02-10T11:13:41.337 INFO:teuthology.orchestra.run.gibba020.stdout:/tmp/tmp.pVTAWfzR6P 2021-02-10T11:13:41.338 DEBUG:teuthology.orchestra.run.gibba020:> sudo cp /var/lib/ceph/osd/ceph-7/fsid /tmp/tmp.pVTAWfzR6P 2021-02-10T11:13:41.681 DEBUG:teuthology.orchestra.run.gibba020:> sudo chmod 0666 /tmp/tmp.pVTAWfzR6P 2021-02-10T11:13:42.361 DEBUG:teuthology.orchestra.remote:gibba020:/tmp/tmp.pVTAWfzR6P is 37B 2021-02-10T11:13:42.700 DEBUG:teuthology.orchestra.run.gibba020:> rm -fr /tmp/tmp.pVTAWfzR6P 2021-02-10T11:13:43.034 DEBUG:teuthology.orchestra.run.gibba020:> sudo ceph --cluster ceph osd new 95444b8f-6370-4a1d-ae5f-df62651b77da 0 2021-02-10T11:13:43.459 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:43.444 7f5aaa8ff700 1 Processor -- start 2021-02-10T11:13:43.459 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:43.444 7f5aaa8ff700 1 -- start start 2021-02-10T11:13:43.460 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:43.444 7f5aaa8ff700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5aa40cff00 0x7f5aa41b39d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:43.460 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:43.444 7f5aaa8ff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f5aa40d1bf0 0x7f5aa40ef620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:43.460 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:43.444 7f5aaa8ff700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5aa40f0b20 con 0x7f5aa40d17f0 2021-02-10T11:13:43.460 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:43.444 7f5aaa8ff700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5aa41b3ff0 con 0x7f5aa40d1bf0 2021-02-10T11:13:43.460 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:43.444 7f5aaa8ff700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5aa40f20b0 con 0x7f5aa40cff00 2021-02-10T11:13:43.460 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:43.445 7f5aa37fe700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5aa40cff00 0x7f5aa41b39d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:43.460 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:43.445 7f5aa3fff700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7f5aa40d17f0 0x7f5aa41b31e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.120:47908/0 (socket says 172.21.2.120:47908) 2021-02-10T11:13:43.460 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:43.445 7f5aa3fff700 1 -- 172.21.2.120:0/3450762408 learned_addr learned my addr 172.21.2.120:0/3450762408 (peer_addr_for_me v1:172.21.2.120:0/0) 2021-02-10T11:13:43.460 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:43.445 7f5aa37fe700 1 -- 172.21.2.120:0/3450762408 >> v2:172.21.2.112:3300/0 conn(0x7f5aa40d1bf0 msgr2=0x7f5aa40ef620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:13:43.460 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:43.445 7f5aa37fe700 1 --2- 172.21.2.120:0/3450762408 >> v2:172.21.2.112:3300/0 conn(0x7f5aa40d1bf0 0x7f5aa40ef620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:13:43.460 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:43.445 7f5aa37fe700 1 -- 172.21.2.120:0/3450762408 >> v1:172.21.2.120:6789/0 conn(0x7f5aa40d17f0 legacy=0x7f5aa41b31e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:43.461 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:43.445 7f5aa2ffd700 1 -- 172.21.2.120:0/3450762408 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3561655059 0 0) 0x7f5a940009b0 con 0x7f5aa40d17f0 2021-02-10T11:13:43.461 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:43.445 7f5aa37fe700 1 -- 172.21.2.120:0/3450762408 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5aa40f22a0 con 0x7f5aa40cff00 2021-02-10T11:13:43.461 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:43.445 7f5aa37fe700 1 --2- 172.21.2.120:0/3450762408 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5aa40cff00 0x7f5aa41b39d0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f5a98006650 tx=0x7f5a98009a80).ready entity=mon.2 client_cookie=d0d2e04b45bedf04 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:13:43.461 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:43.445 7f5aa2ffd700 1 -- 172.21.2.120:0/3450762408 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5a9800c0c0 con 0x7f5aa40cff00 2021-02-10T11:13:43.461 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:43.445 7f5aa2ffd700 1 -- 172.21.2.120:0/3450762408 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a9800c340 con 0x7f5aa40cff00 2021-02-10T11:13:43.461 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:43.445 7f5aa2ffd700 1 -- 172.21.2.120:0/3450762408 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5a9800c860 con 0x7f5aa40cff00 2021-02-10T11:13:43.461 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:43.445 7f5aaa8ff700 1 -- 172.21.2.120:0/3450762408 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5aa40cff00 msgr2=0x7f5aa41b39d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:43.461 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:43.445 7f5aaa8ff700 1 --2- 172.21.2.120:0/3450762408 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5aa40cff00 0x7f5aa41b39d0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f5a98006650 tx=0x7f5a98009a80).stop 2021-02-10T11:13:43.461 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:43.446 7f5aaa8ff700 1 -- 172.21.2.120:0/3450762408 shutdown_connections 2021-02-10T11:13:43.461 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:43.446 7f5aaa8ff700 1 --2- 172.21.2.120:0/3450762408 >> v2:172.21.2.112:3300/0 conn(0x7f5aa40d1bf0 0x7f5aa40ef620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:13:43.461 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:43.446 7f5aaa8ff700 1 --2- 172.21.2.120:0/3450762408 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5aa40cff00 0x7f5aa41b39d0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:43.461 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:43.446 7f5aaa8ff700 1 -- 172.21.2.120:0/3450762408 >> 172.21.2.120:0/3450762408 conn(0x7f5aa40fde40 msgr2=0x7f5aa40ce390 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:13:43.461 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:43.446 7f5aaa8ff700 1 -- 172.21.2.120:0/3450762408 shutdown_connections 2021-02-10T11:13:43.462 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:43.446 7f5aaa8ff700 1 -- 172.21.2.120:0/3450762408 wait complete. 2021-02-10T11:13:43.462 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:43.461 7f5aaa8ff700 1 Processor -- start 2021-02-10T11:13:43.462 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:43.461 7f5aaa8ff700 1 -- start start 2021-02-10T11:13:43.463 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:43.461 7f5aaa8ff700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5aa41960f0 0x7f5aa419b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:43.464 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:43.461 7f5aaa8ff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5aa418ecc0 0x7f5aa4195740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:43.464 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:43.461 7f5aaa8ff700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5aa419c280 0x7f5aa419c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:43.464 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:43.461 7f5aaa8ff700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5aa418f0c0 con 0x7f5aa418ecc0 2021-02-10T11:13:43.464 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:43.461 7f5aaa8ff700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5aa4195d60 con 0x7f5aa419c280 2021-02-10T11:13:43.464 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:43.461 7f5aaa8ff700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5aa419cca0 con 0x7f5aa41960f0 2021-02-10T11:13:43.464 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:43.462 7f5aa3fff700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5aa41960f0 0x7f5aa419b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:43.464 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:43.462 7f5aa3fff700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5aa41960f0 0x7f5aa419b6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3301/0 says I am v2:172.21.2.120:34626/0 (socket says 172.21.2.120:34626) 2021-02-10T11:13:43.464 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:43.462 7f5aa3fff700 1 -- 172.21.2.120:0/3969693842 learned_addr learned my addr 172.21.2.120:0/3969693842 (peer_addr_for_me v2:172.21.2.120:0/0) 2021-02-10T11:13:43.464 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:43.462 7f5aa8f1c700 1 --2- 172.21.2.120:0/3969693842 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5aa419c280 0x7f5aa419c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:43.464 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:43.462 7f5aa3fff700 1 -- 172.21.2.120:0/3969693842 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5aa419c280 msgr2=0x7f5aa419c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:43.465 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:43.462 7f5aa3fff700 1 --2- 172.21.2.120:0/3969693842 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5aa419c280 0x7f5aa419c680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:43.465 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:43.462 7f5aa3fff700 1 -- 172.21.2.120:0/3969693842 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5aa418ecc0 msgr2=0x7f5aa4195740 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:13:43.465 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:43.462 7f5aa3fff700 1 --2- 172.21.2.120:0/3969693842 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5aa418ecc0 0x7f5aa4195740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:13:43.465 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:43.462 7f5aa3fff700 1 -- 172.21.2.120:0/3969693842 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5aa419cec0 con 0x7f5aa41960f0 2021-02-10T11:13:43.465 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:43.462 7f5aa3fff700 1 --2- 172.21.2.120:0/3969693842 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5aa41960f0 0x7f5aa419b6a0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f5a94001c60 tx=0x7f5a9400a8d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:13:43.465 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:43.462 7f5aa0ff9700 1 -- 172.21.2.120:0/3969693842 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5a9400acc0 con 0x7f5aa41960f0 2021-02-10T11:13:43.465 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:43.462 7f5aa0ff9700 1 -- 172.21.2.120:0/3969693842 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a94004350 con 0x7f5aa41960f0 2021-02-10T11:13:43.465 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:43.462 7f5aa0ff9700 1 -- 172.21.2.120:0/3969693842 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5a94004830 con 0x7f5aa41960f0 2021-02-10T11:13:43.465 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:43.462 7f5aaa8ff700 1 -- 172.21.2.120:0/3969693842 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5aa419cec0 con 0x7f5aa41960f0 2021-02-10T11:13:43.465 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:43.462 7f5aaa8ff700 1 -- 172.21.2.120:0/3969693842 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5aa419cec0 con 0x7f5aa41960f0 2021-02-10T11:13:43.465 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:43.463 7f5aa0ff9700 1 -- 172.21.2.120:0/3969693842 <== mon.2 v2:172.21.2.120:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5a94000dd0 con 0x7f5aa41960f0 2021-02-10T11:13:43.465 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:43.463 7f5aa0ff9700 1 --2- 172.21.2.120:0/3969693842 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5a840186c0 0x7f5a8401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:43.465 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:43.463 7f5aa0ff9700 1 -- 172.21.2.120:0/3969693842 <== mon.2 v2:172.21.2.120:3301/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 754+0+0 (secure 0 0 0) 0x7f5a940314b0 con 0x7f5aa41960f0 2021-02-10T11:13:43.466 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:43.463 7f5aa37fe700 1 --2- 172.21.2.120:0/3969693842 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5a840186c0 0x7f5a8401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:43.466 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:43.464 7f5aa37fe700 1 --2- 172.21.2.120:0/3969693842 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5a840186c0 0x7f5a8401abc0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:13:43.493 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:43.494 7f5aaa8ff700 1 -- 172.21.2.120:0/3969693842 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5aa419bd50 con 0x7f5aa41960f0 2021-02-10T11:13:43.495 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:43.497 7f5aa0ff9700 1 -- 172.21.2.120:0/3969693842 <== mon.2 v2:172.21.2.120:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5a9401f4c0 con 0x7f5aa41960f0 2021-02-10T11:13:43.632 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:43.634 7f5aaa8ff700 1 -- 172.21.2.120:0/3969693842 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_command({"prefix": "osd new", "uuid": "95444b8f-6370-4a1d-ae5f-df62651b77da", "id": 0} v 0) v1 -- 0x7f5aa419bd50 con 0x7f5aa41960f0 2021-02-10T11:13:43.639 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:43.640 7f5aa0ff9700 1 -- 172.21.2.120:0/3969693842 <== mon.2 v2:172.21.2.120:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "95444b8f-6370-4a1d-ae5f-df62651b77da", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7f5a94000dd0 con 0x7f5aa41960f0 2021-02-10T11:13:43.640 INFO:teuthology.orchestra.run.gibba020.stdout:0 2021-02-10T11:13:43.641 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:43.643 7f5aaa8ff700 1 -- 172.21.2.120:0/3969693842 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5a840186c0 msgr2=0x7f5a8401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:43.641 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:43.643 7f5aaa8ff700 1 --2- 172.21.2.120:0/3969693842 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5a840186c0 0x7f5a8401abc0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:43.642 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:43.643 7f5aaa8ff700 1 -- 172.21.2.120:0/3969693842 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5aa41960f0 msgr2=0x7f5aa419b6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:43.642 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:43.643 7f5aaa8ff700 1 --2- 172.21.2.120:0/3969693842 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5aa41960f0 0x7f5aa419b6a0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f5a94001c60 tx=0x7f5a9400a8d0).stop 2021-02-10T11:13:43.642 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:43.643 7f5aaa8ff700 1 -- 172.21.2.120:0/3969693842 shutdown_connections 2021-02-10T11:13:43.642 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:43.643 7f5aaa8ff700 1 --2- 172.21.2.120:0/3969693842 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5a840186c0 0x7f5a8401abc0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:43.642 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:43.643 7f5aaa8ff700 1 --2- 172.21.2.120:0/3969693842 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5aa419c280 0x7f5aa419c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:43.642 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:43.643 7f5aaa8ff700 1 --2- 172.21.2.120:0/3969693842 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5aa418ecc0 0x7f5aa4195740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:13:43.642 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:43.643 7f5aaa8ff700 1 --2- 172.21.2.120:0/3969693842 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5aa41960f0 0x7f5aa419b6a0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:43.642 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:43.643 7f5aaa8ff700 1 -- 172.21.2.120:0/3969693842 >> 172.21.2.120:0/3969693842 conn(0x7f5aa419f7e0 msgr2=0x7f5aa418c240 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:13:43.643 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:43.643 7f5aaa8ff700 1 -- 172.21.2.120:0/3969693842 shutdown_connections 2021-02-10T11:13:43.643 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:43.643 7f5aaa8ff700 1 -- 172.21.2.120:0/3969693842 wait complete. 2021-02-10T11:13:43.649 DEBUG:teuthology.orchestra.run.gibba020:> sudo ceph --cluster ceph osd new a23e217e-0d8e-486e-be07-712adfabd262 1 2021-02-10T11:13:44.075 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.060 7fe52a222700 1 Processor -- start 2021-02-10T11:13:44.075 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.060 7fe52a222700 1 -- start start 2021-02-10T11:13:44.075 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.060 7fe52a222700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fe5240d17f0 0x7fe5241b31e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:44.075 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.060 7fe52a222700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fe5240cff00 0x7fe5240ef620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:44.075 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.060 7fe52a222700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5240f0b20 con 0x7fe5240d1bf0 2021-02-10T11:13:44.076 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.060 7fe52a222700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5241b3ff0 con 0x7fe5240d17f0 2021-02-10T11:13:44.076 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.060 7fe52a222700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5240f20b0 con 0x7fe5240cff00 2021-02-10T11:13:44.076 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.061 7fe52883f700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fe5240cff00 0x7fe5240ef620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:44.076 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.061 7fe52883f700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fe5240cff00 0x7fe5240ef620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3300/0 says I am v2:172.21.2.120:43856/0 (socket says 172.21.2.120:43856) 2021-02-10T11:13:44.076 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.061 7fe52883f700 1 -- 172.21.2.120:0/2007314210 learned_addr learned my addr 172.21.2.120:0/2007314210 (peer_addr_for_me v2:172.21.2.120:0/0) 2021-02-10T11:13:44.076 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.061 7fe52883f700 1 -- 172.21.2.120:0/2007314210 >> v2:172.21.2.112:3300/0 conn(0x7fe5240d17f0 msgr2=0x7fe5241b31e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:13:44.076 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.061 7fe52883f700 1 --2- 172.21.2.120:0/2007314210 >> v2:172.21.2.112:3300/0 conn(0x7fe5240d17f0 0x7fe5241b31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:13:44.076 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.061 7fe52883f700 1 -- 172.21.2.120:0/2007314210 >> v1:172.21.2.112:6789/0 conn(0x7fe5240d1bf0 legacy=0x7fe5241b3800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:13:44.076 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.061 7fe52883f700 1 -- 172.21.2.120:0/2007314210 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5240f22d0 con 0x7fe5240cff00 2021-02-10T11:13:44.076 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.061 7fe52883f700 1 --2- 172.21.2.120:0/2007314210 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe5240cff00 0x7fe5240ef620 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fe51c006f00 tx=0x7fe51c009f10).ready entity=mon.1 client_cookie=b02c311909e8be20 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:13:44.076 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.061 7fe522ffd700 1 -- 172.21.2.120:0/2007314210 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe51c00b3b0 con 0x7fe5240cff00 2021-02-10T11:13:44.076 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.061 7fe522ffd700 1 -- 172.21.2.120:0/2007314210 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe51c00b630 con 0x7fe5240cff00 2021-02-10T11:13:44.077 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.061 7fe522ffd700 1 -- 172.21.2.120:0/2007314210 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe51c00bb10 con 0x7fe5240cff00 2021-02-10T11:13:44.077 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.061 7fe52a222700 1 -- 172.21.2.120:0/2007314210 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe5240cff00 msgr2=0x7fe5240ef620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:44.077 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.061 7fe52a222700 1 --2- 172.21.2.120:0/2007314210 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe5240cff00 0x7fe5240ef620 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fe51c006f00 tx=0x7fe51c009f10).stop 2021-02-10T11:13:44.077 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.062 7fe52a222700 1 -- 172.21.2.120:0/2007314210 shutdown_connections 2021-02-10T11:13:44.077 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.062 7fe52a222700 1 --2- 172.21.2.120:0/2007314210 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe5240cff00 0x7fe5240ef620 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:44.077 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.062 7fe52a222700 1 --2- 172.21.2.120:0/2007314210 >> v2:172.21.2.112:3300/0 conn(0x7fe5240d17f0 0x7fe5241b31e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:13:44.077 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.062 7fe52a222700 1 -- 172.21.2.120:0/2007314210 >> 172.21.2.120:0/2007314210 conn(0x7fe5240fde40 msgr2=0x7fe5240ce390 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:13:44.077 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.062 7fe52a222700 1 -- 172.21.2.120:0/2007314210 shutdown_connections 2021-02-10T11:13:44.077 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.062 7fe52a222700 1 -- 172.21.2.120:0/2007314210 wait complete. 2021-02-10T11:13:44.077 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.077 7fe52a222700 1 Processor -- start 2021-02-10T11:13:44.077 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.077 7fe52a222700 1 -- start start 2021-02-10T11:13:44.079 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.077 7fe52a222700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe524107f60 0x7fe52409e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:44.079 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.077 7fe52a222700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe52418f470 0x7fe52418f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:44.079 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.077 7fe52a222700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe52418fe90 0x7fe524106a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:44.079 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.077 7fe52a222700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe524107a60 con 0x7fe52418f470 2021-02-10T11:13:44.080 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.077 7fe52a222700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe524108360 con 0x7fe52418fe90 2021-02-10T11:13:44.080 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.077 7fe52a222700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe524107090 con 0x7fe524107f60 2021-02-10T11:13:44.080 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.077 7fe52883f700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe52418fe90 0x7fe524106a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:44.080 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.077 7fe52883f700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe52418fe90 0x7fe524106a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3300/0 says I am v2:172.21.2.120:43860/0 (socket says 172.21.2.120:43860) 2021-02-10T11:13:44.080 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.077 7fe523fff700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe524107f60 0x7fe52409e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:44.080 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.077 7fe52883f700 1 -- 172.21.2.120:0/2275556914 learned_addr learned my addr 172.21.2.120:0/2275556914 (peer_addr_for_me v2:172.21.2.120:0/0) 2021-02-10T11:13:44.080 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.078 7fe5237fe700 1 --2- 172.21.2.120:0/2275556914 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe52418f470 0x7fe52418f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:44.080 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.078 7fe52883f700 1 -- 172.21.2.120:0/2275556914 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe524107f60 msgr2=0x7fe52409e0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:44.080 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.078 7fe52883f700 1 --2- 172.21.2.120:0/2275556914 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe524107f60 0x7fe52409e0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:44.080 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.078 7fe52883f700 1 -- 172.21.2.120:0/2275556914 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe52418f470 msgr2=0x7fe52418f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:44.081 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.078 7fe52883f700 1 --2- 172.21.2.120:0/2275556914 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe52418f470 0x7fe52418f870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:44.081 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.078 7fe52883f700 1 -- 172.21.2.120:0/2275556914 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5241072b0 con 0x7fe52418fe90 2021-02-10T11:13:44.081 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.078 7fe523fff700 1 --2- 172.21.2.120:0/2275556914 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe524107f60 0x7fe52409e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:13:44.081 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.078 7fe5237fe700 1 --2- 172.21.2.120:0/2275556914 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe52418f470 0x7fe52418f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:13:44.081 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.078 7fe52883f700 1 --2- 172.21.2.120:0/2275556914 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe52418fe90 0x7fe524106a70 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7fe51c009390 tx=0x7fe51c00bd40).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:13:44.081 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.078 7fe520ff9700 1 -- 172.21.2.120:0/2275556914 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe51c000cc0 con 0x7fe52418fe90 2021-02-10T11:13:44.081 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.078 7fe520ff9700 1 -- 172.21.2.120:0/2275556914 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe51c003030 con 0x7fe52418fe90 2021-02-10T11:13:44.081 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.078 7fe52a222700 1 -- 172.21.2.120:0/2275556914 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5241072b0 con 0x7fe52418fe90 2021-02-10T11:13:44.081 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.078 7fe520ff9700 1 -- 172.21.2.120:0/2275556914 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe51c001480 con 0x7fe52418fe90 2021-02-10T11:13:44.081 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.078 7fe52a222700 1 -- 172.21.2.120:0/2275556914 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5240b7370 con 0x7fe52418fe90 2021-02-10T11:13:44.081 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.079 7fe520ff9700 1 -- 172.21.2.120:0/2275556914 <== mon.1 v2:172.21.2.120:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe51c005820 con 0x7fe52418fe90 2021-02-10T11:13:44.082 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.079 7fe520ff9700 1 --2- 172.21.2.120:0/2275556914 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe504018730 0x7fe50401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:44.082 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.079 7fe520ff9700 1 -- 172.21.2.120:0/2275556914 <== mon.1 v2:172.21.2.120:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 857+0+0 (secure 0 0 0) 0x7fe51c031890 con 0x7fe52418fe90 2021-02-10T11:13:44.082 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.079 7fe523fff700 1 --2- 172.21.2.120:0/2275556914 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe504018730 0x7fe50401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:44.082 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.079 7fe523fff700 1 --2- 172.21.2.120:0/2275556914 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe504018730 0x7fe50401ac10 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:13:44.109 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.111 7fe52a222700 1 -- 172.21.2.120:0/2275556914 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe524103760 con 0x7fe52418fe90 2021-02-10T11:13:44.112 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.113 7fe520ff9700 1 -- 172.21.2.120:0/2275556914 <== mon.1 v2:172.21.2.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe51c009450 con 0x7fe52418fe90 2021-02-10T11:13:44.249 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.251 7fe52a222700 1 -- 172.21.2.120:0/2275556914 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_command({"prefix": "osd new", "uuid": "a23e217e-0d8e-486e-be07-712adfabd262", "id": 1} v 0) v1 -- 0x7fe5240f0b20 con 0x7fe52418fe90 2021-02-10T11:13:44.254 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.256 7fe520ff9700 1 -- 172.21.2.120:0/2275556914 <== mon.1 v2:172.21.2.120:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "a23e217e-0d8e-486e-be07-712adfabd262", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7fe51c005820 con 0x7fe52418fe90 2021-02-10T11:13:44.256 INFO:teuthology.orchestra.run.gibba020.stdout:1 2021-02-10T11:13:44.258 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.260 7fe52a222700 1 -- 172.21.2.120:0/2275556914 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe504018730 msgr2=0x7fe50401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:44.259 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.260 7fe52a222700 1 --2- 172.21.2.120:0/2275556914 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe504018730 0x7fe50401ac10 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:44.259 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.260 7fe52a222700 1 -- 172.21.2.120:0/2275556914 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe52418fe90 msgr2=0x7fe524106a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:44.259 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.260 7fe52a222700 1 --2- 172.21.2.120:0/2275556914 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe52418fe90 0x7fe524106a70 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7fe51c009390 tx=0x7fe51c00bd40).stop 2021-02-10T11:13:44.259 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.260 7fe52a222700 1 -- 172.21.2.120:0/2275556914 shutdown_connections 2021-02-10T11:13:44.259 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.260 7fe52a222700 1 --2- 172.21.2.120:0/2275556914 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe504018730 0x7fe50401ac10 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:44.259 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.260 7fe52a222700 1 --2- 172.21.2.120:0/2275556914 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe52418fe90 0x7fe524106a70 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:44.259 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.260 7fe52a222700 1 --2- 172.21.2.120:0/2275556914 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe52418f470 0x7fe52418f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:44.259 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.260 7fe52a222700 1 --2- 172.21.2.120:0/2275556914 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe524107f60 0x7fe52409e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:44.259 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.260 7fe52a222700 1 -- 172.21.2.120:0/2275556914 >> 172.21.2.120:0/2275556914 conn(0x7fe5241a3800 msgr2=0x7fe524092c10 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:13:44.259 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.260 7fe52a222700 1 -- 172.21.2.120:0/2275556914 shutdown_connections 2021-02-10T11:13:44.259 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.260 7fe52a222700 1 -- 172.21.2.120:0/2275556914 wait complete. 2021-02-10T11:13:44.266 DEBUG:teuthology.orchestra.run.gibba020:> sudo ceph --cluster ceph osd new 026afa8a-c0f1-4b35-9a76-53c00fe0c2fc 2 2021-02-10T11:13:44.693 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.678 7f4a35348700 1 Processor -- start 2021-02-10T11:13:44.693 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.678 7f4a35348700 1 -- start start 2021-02-10T11:13:44.693 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.678 7f4a35348700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f4a300d17f0 0x7f4a301b31e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:44.694 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.678 7f4a35348700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f4a300cff00 0x7f4a300ef620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:44.694 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.678 7f4a35348700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a300f0b20 con 0x7f4a300d1bf0 2021-02-10T11:13:44.694 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.678 7f4a35348700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a301b3ff0 con 0x7f4a300d17f0 2021-02-10T11:13:44.694 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.678 7f4a35348700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a300f20b0 con 0x7f4a300cff00 2021-02-10T11:13:44.694 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.678 7f4a2f87e700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f4a300cff00 0x7f4a300ef620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:44.694 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.678 7f4a2f87e700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f4a300cff00 0x7f4a300ef620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3300/0 says I am v2:172.21.2.120:43868/0 (socket says 172.21.2.120:43868) 2021-02-10T11:13:44.694 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.678 7f4a2f87e700 1 -- 172.21.2.120:0/1637516963 learned_addr learned my addr 172.21.2.120:0/1637516963 (peer_addr_for_me v2:172.21.2.120:0/0) 2021-02-10T11:13:44.694 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.679 7f4a2f87e700 1 -- 172.21.2.120:0/1637516963 >> v2:172.21.2.112:3300/0 conn(0x7f4a300d17f0 msgr2=0x7f4a301b31e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:13:44.694 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.679 7f4a2f87e700 1 --2- 172.21.2.120:0/1637516963 >> v2:172.21.2.112:3300/0 conn(0x7f4a300d17f0 0x7f4a301b31e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:13:44.694 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.679 7f4a2f87e700 1 -- 172.21.2.120:0/1637516963 >> v1:172.21.2.112:6789/0 conn(0x7f4a300d1bf0 legacy=0x7f4a301b3800 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:13:44.694 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.679 7f4a2f87e700 1 -- 172.21.2.120:0/1637516963 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a300f22d0 con 0x7f4a300cff00 2021-02-10T11:13:44.695 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.679 7f4a2f87e700 1 --2- 172.21.2.120:0/1637516963 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4a300cff00 0x7f4a300ef620 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f4a28006f00 tx=0x7f4a28009f10).ready entity=mon.1 client_cookie=32cb52cdba2610f3 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:13:44.695 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.679 7f4a2e07b700 1 -- 172.21.2.120:0/1637516963 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4a2800b3b0 con 0x7f4a300cff00 2021-02-10T11:13:44.695 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.679 7f4a2e07b700 1 -- 172.21.2.120:0/1637516963 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a2800b630 con 0x7f4a300cff00 2021-02-10T11:13:44.695 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.679 7f4a2e07b700 1 -- 172.21.2.120:0/1637516963 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4a2800bb50 con 0x7f4a300cff00 2021-02-10T11:13:44.695 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.679 7f4a35348700 1 -- 172.21.2.120:0/1637516963 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4a300cff00 msgr2=0x7f4a300ef620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:44.695 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.679 7f4a35348700 1 --2- 172.21.2.120:0/1637516963 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4a300cff00 0x7f4a300ef620 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f4a28006f00 tx=0x7f4a28009f10).stop 2021-02-10T11:13:44.695 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.680 7f4a35348700 1 -- 172.21.2.120:0/1637516963 shutdown_connections 2021-02-10T11:13:44.695 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.680 7f4a35348700 1 --2- 172.21.2.120:0/1637516963 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4a300cff00 0x7f4a300ef620 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:44.696 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.680 7f4a35348700 1 --2- 172.21.2.120:0/1637516963 >> v2:172.21.2.112:3300/0 conn(0x7f4a300d17f0 0x7f4a301b31e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:13:44.696 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.680 7f4a35348700 1 -- 172.21.2.120:0/1637516963 >> 172.21.2.120:0/1637516963 conn(0x7f4a300fde40 msgr2=0x7f4a300ce390 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:13:44.696 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.680 7f4a35348700 1 -- 172.21.2.120:0/1637516963 shutdown_connections 2021-02-10T11:13:44.696 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.680 7f4a35348700 1 -- 172.21.2.120:0/1637516963 wait complete. 2021-02-10T11:13:44.698 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.695 7f4a35348700 1 Processor -- start 2021-02-10T11:13:44.699 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.695 7f4a35348700 1 -- start start 2021-02-10T11:13:44.699 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.696 7f4a35348700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4a301960f0 0x7f4a3019b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:44.699 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.696 7f4a35348700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4a3018ecc0 0x7f4a30195740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:44.699 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.696 7f4a35348700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4a3019c280 0x7f4a3019c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:44.699 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.696 7f4a35348700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a3018f0c0 con 0x7f4a3019c280 2021-02-10T11:13:44.699 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.696 7f4a35348700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a30195d60 con 0x7f4a3018ecc0 2021-02-10T11:13:44.699 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.696 7f4a35348700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a3019cca0 con 0x7f4a301960f0 2021-02-10T11:13:44.699 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.696 7f4a2f07d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4a301960f0 0x7f4a3019b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:44.699 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.696 7f4a2e87c700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4a3018ecc0 0x7f4a30195740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:44.699 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.696 7f4a2f07d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4a301960f0 0x7f4a3019b6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3301/0 says I am v2:172.21.2.120:34650/0 (socket says 172.21.2.120:34650) 2021-02-10T11:13:44.699 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.696 7f4a2f07d700 1 -- 172.21.2.120:0/4066937555 learned_addr learned my addr 172.21.2.120:0/4066937555 (peer_addr_for_me v2:172.21.2.120:0/0) 2021-02-10T11:13:44.699 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.696 7f4a2f87e700 1 --2- 172.21.2.120:0/4066937555 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4a3019c280 0x7f4a3019c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:44.700 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.696 7f4a2e87c700 1 -- 172.21.2.120:0/4066937555 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4a301960f0 msgr2=0x7f4a3019b6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:44.700 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.696 7f4a2e87c700 1 --2- 172.21.2.120:0/4066937555 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4a301960f0 0x7f4a3019b6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:44.700 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.696 7f4a2e87c700 1 -- 172.21.2.120:0/4066937555 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4a3019c280 msgr2=0x7f4a3019c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:44.700 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.696 7f4a2e87c700 1 --2- 172.21.2.120:0/4066937555 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4a3019c280 0x7f4a3019c680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:44.700 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.697 7f4a2e87c700 1 -- 172.21.2.120:0/4066937555 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a3019cec0 con 0x7f4a3018ecc0 2021-02-10T11:13:44.701 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.697 7f4a2f07d700 1 --2- 172.21.2.120:0/4066937555 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4a301960f0 0x7f4a3019b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:13:44.701 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.697 7f4a2e87c700 1 --2- 172.21.2.120:0/4066937555 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4a3018ecc0 0x7f4a30195740 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f4a20009fa0 tx=0x7f4a2000b840).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:13:44.701 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.697 7f4a27fff700 1 -- 172.21.2.120:0/4066937555 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4a200009b0 con 0x7f4a3018ecc0 2021-02-10T11:13:44.702 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.697 7f4a27fff700 1 -- 172.21.2.120:0/4066937555 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a2000bc30 con 0x7f4a3018ecc0 2021-02-10T11:13:44.702 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.697 7f4a35348700 1 -- 172.21.2.120:0/4066937555 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a3019cec0 con 0x7f4a3018ecc0 2021-02-10T11:13:44.702 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.697 7f4a27fff700 1 -- 172.21.2.120:0/4066937555 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4a20002470 con 0x7f4a3018ecc0 2021-02-10T11:13:44.702 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.697 7f4a35348700 1 -- 172.21.2.120:0/4066937555 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a3019cec0 con 0x7f4a3018ecc0 2021-02-10T11:13:44.702 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.697 7f4a27fff700 1 -- 172.21.2.120:0/4066937555 <== mon.1 v2:172.21.2.120:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4a20006230 con 0x7f4a3018ecc0 2021-02-10T11:13:44.702 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.698 7f4a27fff700 1 --2- 172.21.2.120:0/4066937555 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4a14018730 0x7f4a1401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:44.702 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.698 7f4a27fff700 1 -- 172.21.2.120:0/4066937555 <== mon.1 v2:172.21.2.120:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 960+0+0 (secure 0 0 0) 0x7f4a20031390 con 0x7f4a3018ecc0 2021-02-10T11:13:44.702 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.698 7f4a2f07d700 1 --2- 172.21.2.120:0/4066937555 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4a14018730 0x7f4a1401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:44.702 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.698 7f4a2f07d700 1 --2- 172.21.2.120:0/4066937555 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4a14018730 0x7f4a1401ac10 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:13:44.727 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.728 7f4a35348700 1 -- 172.21.2.120:0/4066937555 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a3019bd50 con 0x7f4a3018ecc0 2021-02-10T11:13:44.729 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.731 7f4a27fff700 1 -- 172.21.2.120:0/4066937555 <== mon.1 v2:172.21.2.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4a20006020 con 0x7f4a3018ecc0 2021-02-10T11:13:44.867 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.868 7f4a35348700 1 -- 172.21.2.120:0/4066937555 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_command({"prefix": "osd new", "uuid": "026afa8a-c0f1-4b35-9a76-53c00fe0c2fc", "id": 2} v 0) v1 -- 0x7f4a3019bd50 con 0x7f4a3018ecc0 2021-02-10T11:13:44.872 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.874 7f4a27fff700 1 -- 172.21.2.120:0/4066937555 <== mon.1 v2:172.21.2.120:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "026afa8a-c0f1-4b35-9a76-53c00fe0c2fc", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7f4a20006230 con 0x7f4a3018ecc0 2021-02-10T11:13:44.874 INFO:teuthology.orchestra.run.gibba020.stdout:2 2021-02-10T11:13:44.876 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.877 7f4a35348700 1 -- 172.21.2.120:0/4066937555 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4a14018730 msgr2=0x7f4a1401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:44.876 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.877 7f4a35348700 1 --2- 172.21.2.120:0/4066937555 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4a14018730 0x7f4a1401ac10 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:44.876 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.877 7f4a35348700 1 -- 172.21.2.120:0/4066937555 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4a3018ecc0 msgr2=0x7f4a30195740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:44.876 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.877 7f4a35348700 1 --2- 172.21.2.120:0/4066937555 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4a3018ecc0 0x7f4a30195740 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f4a20009fa0 tx=0x7f4a2000b840).stop 2021-02-10T11:13:44.877 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.878 7f4a35348700 1 -- 172.21.2.120:0/4066937555 shutdown_connections 2021-02-10T11:13:44.877 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.878 7f4a35348700 1 --2- 172.21.2.120:0/4066937555 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4a14018730 0x7f4a1401ac10 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:44.877 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.878 7f4a35348700 1 --2- 172.21.2.120:0/4066937555 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4a3018ecc0 0x7f4a30195740 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:44.877 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.878 7f4a35348700 1 --2- 172.21.2.120:0/4066937555 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4a3019c280 0x7f4a3019c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:44.877 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.878 7f4a35348700 1 --2- 172.21.2.120:0/4066937555 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4a301960f0 0x7f4a3019b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:44.877 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.878 7f4a35348700 1 -- 172.21.2.120:0/4066937555 >> 172.21.2.120:0/4066937555 conn(0x7f4a3019f7e0 msgr2=0x7f4a3018c240 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:13:44.877 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.878 7f4a35348700 1 -- 172.21.2.120:0/4066937555 shutdown_connections 2021-02-10T11:13:44.877 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:44.878 7f4a35348700 1 -- 172.21.2.120:0/4066937555 wait complete. 2021-02-10T11:13:44.884 DEBUG:teuthology.orchestra.run.gibba020:> sudo ceph --cluster ceph osd new 58795682-e00d-4173-bb84-f7075ca89a11 3 2021-02-10T11:13:45.312 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.295 7f438444c700 1 Processor -- start 2021-02-10T11:13:45.312 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.295 7f438444c700 1 -- start start 2021-02-10T11:13:45.312 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.296 7f438444c700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f437c0d17f0 0x7f437c1b31e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:45.312 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.296 7f438444c700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f437c0d1bf0 0x7f437c1b3800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:45.312 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.296 7f438444c700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f437c1b3e20 0x7f437c0ceef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:45.312 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.296 7f438444c700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f437c0e5eb0 con 0x7f437c0d17f0 2021-02-10T11:13:45.312 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.296 7f438444c700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f437c0f0b20 con 0x7f437c0d1bf0 2021-02-10T11:13:45.312 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.296 7f438444c700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f437c0cf510 con 0x7f437c1b3e20 2021-02-10T11:13:45.312 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.296 7f4381a67700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f437c0d1bf0 0x7f437c1b3800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:45.313 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.296 7f4382a69700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f437c1b3e20 0x7f437c0ceef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:45.313 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.296 7f4381a67700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f437c0d1bf0 0x7f437c1b3800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3300/0 says I am v2:172.21.2.120:43878/0 (socket says 172.21.2.120:43878) 2021-02-10T11:13:45.313 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.296 7f4381a67700 1 -- 172.21.2.120:0/2903289817 learned_addr learned my addr 172.21.2.120:0/2903289817 (peer_addr_for_me v2:172.21.2.120:0/0) 2021-02-10T11:13:45.313 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.297 7f4381a67700 1 -- 172.21.2.120:0/2903289817 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f437c1b3e20 msgr2=0x7f437c0ceef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:13:45.313 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.297 7f4381a67700 1 --2- 172.21.2.120:0/2903289817 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f437c1b3e20 0x7f437c0ceef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:45.313 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.297 7f4381a67700 1 -- 172.21.2.120:0/2903289817 >> v2:172.21.2.112:3300/0 conn(0x7f437c0d17f0 msgr2=0x7f437c1b31e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:13:45.313 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.297 7f4382268700 1 --2- 172.21.2.120:0/2903289817 >> v2:172.21.2.112:3300/0 conn(0x7f437c0d17f0 0x7f437c1b31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:45.313 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.297 7f4381a67700 1 --2- 172.21.2.120:0/2903289817 >> v2:172.21.2.112:3300/0 conn(0x7f437c0d17f0 0x7f437c1b31e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:45.313 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.297 7f4381a67700 1 -- 172.21.2.120:0/2903289817 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f437c0cf6f0 con 0x7f437c0d1bf0 2021-02-10T11:13:45.313 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.297 7f4382a69700 1 --2- 172.21.2.120:0/2903289817 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f437c1b3e20 0x7f437c0ceef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:13:45.313 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.297 7f4382268700 1 --2- 172.21.2.120:0/2903289817 >> v2:172.21.2.112:3300/0 conn(0x7f437c0d17f0 0x7f437c1b31e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:13:45.313 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.297 7f4381a67700 1 --2- 172.21.2.120:0/2903289817 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f437c0d1bf0 0x7f437c1b3800 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f4378006700 tx=0x7f4378009b80).ready entity=mon.1 client_cookie=b6ed676b5c8eb067 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:13:45.313 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.297 7f4381266700 1 -- 172.21.2.120:0/2903289817 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4378006340 con 0x7f437c0d1bf0 2021-02-10T11:13:45.314 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.297 7f4381266700 1 -- 172.21.2.120:0/2903289817 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f437800c0c0 con 0x7f437c0d1bf0 2021-02-10T11:13:45.314 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.297 7f4381266700 1 -- 172.21.2.120:0/2903289817 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f437800c520 con 0x7f437c0d1bf0 2021-02-10T11:13:45.314 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.297 7f438444c700 1 -- 172.21.2.120:0/2903289817 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f437c0d1bf0 msgr2=0x7f437c1b3800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:45.314 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.297 7f438444c700 1 --2- 172.21.2.120:0/2903289817 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f437c0d1bf0 0x7f437c1b3800 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f4378006700 tx=0x7f4378009b80).stop 2021-02-10T11:13:45.314 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.298 7f438444c700 1 -- 172.21.2.120:0/2903289817 shutdown_connections 2021-02-10T11:13:45.314 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.298 7f438444c700 1 --2- 172.21.2.120:0/2903289817 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f437c1b3e20 0x7f437c0ceef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:45.314 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.298 7f438444c700 1 --2- 172.21.2.120:0/2903289817 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f437c0d1bf0 0x7f437c1b3800 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:45.314 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.298 7f438444c700 1 --2- 172.21.2.120:0/2903289817 >> v2:172.21.2.112:3300/0 conn(0x7f437c0d17f0 0x7f437c1b31e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:45.314 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.298 7f438444c700 1 -- 172.21.2.120:0/2903289817 >> 172.21.2.120:0/2903289817 conn(0x7f437c0fde40 msgr2=0x7f437c0ce390 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:13:45.314 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.298 7f438444c700 1 -- 172.21.2.120:0/2903289817 shutdown_connections 2021-02-10T11:13:45.314 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.298 7f438444c700 1 -- 172.21.2.120:0/2903289817 wait complete. 2021-02-10T11:13:45.314 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.312 7f438444c700 1 Processor -- start 2021-02-10T11:13:45.315 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.313 7f438444c700 1 -- start start 2021-02-10T11:13:45.315 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.313 7f438444c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f437c193150 0x7f437c0be6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:45.315 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.313 7f438444c700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f437c0becf0 0x7f437c097d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:45.315 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.313 7f438444c700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f437c098320 0x7f437c0b1e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:45.315 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.313 7f438444c700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f437c192c50 con 0x7f437c193150 2021-02-10T11:13:45.315 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.313 7f438444c700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f437c192960 con 0x7f437c098320 2021-02-10T11:13:45.315 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.313 7f438444c700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f437c098720 con 0x7f437c0becf0 2021-02-10T11:13:45.315 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.313 7f4382a69700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f437c098320 0x7f437c0b1e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:45.315 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.313 7f4381a67700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f437c0becf0 0x7f437c097d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:45.315 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.313 7f4382a69700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f437c098320 0x7f437c0b1e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3300/0 says I am v2:172.21.2.120:43882/0 (socket says 172.21.2.120:43882) 2021-02-10T11:13:45.315 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.313 7f4382a69700 1 -- 172.21.2.120:0/1313552655 learned_addr learned my addr 172.21.2.120:0/1313552655 (peer_addr_for_me v2:172.21.2.120:0/0) 2021-02-10T11:13:45.316 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.314 7f4382a69700 1 -- 172.21.2.120:0/1313552655 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f437c0becf0 msgr2=0x7f437c097d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:45.317 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.314 7f4382a69700 1 --2- 172.21.2.120:0/1313552655 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f437c0becf0 0x7f437c097d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:45.317 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.314 7f4382a69700 1 -- 172.21.2.120:0/1313552655 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f437c193150 msgr2=0x7f437c0be6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:45.317 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.314 7f4382a69700 1 --2- 172.21.2.120:0/1313552655 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f437c193150 0x7f437c0be6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:13:45.317 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.314 7f4382a69700 1 -- 172.21.2.120:0/1313552655 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f437c0b2490 con 0x7f437c098320 2021-02-10T11:13:45.317 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.314 7f4382a69700 1 --2- 172.21.2.120:0/1313552655 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f437c098320 0x7f437c0b1e70 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f437400b970 tx=0x7f437400bd10).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:13:45.317 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.314 7f436effd700 1 -- 172.21.2.120:0/1313552655 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4374000f90 con 0x7f437c098320 2021-02-10T11:13:45.317 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.314 7f436effd700 1 -- 172.21.2.120:0/1313552655 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4374001270 con 0x7f437c098320 2021-02-10T11:13:45.317 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.314 7f436effd700 1 -- 172.21.2.120:0/1313552655 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f43740014e0 con 0x7f437c098320 2021-02-10T11:13:45.317 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.314 7f438444c700 1 -- 172.21.2.120:0/1313552655 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f437c0b2490 con 0x7f437c098320 2021-02-10T11:13:45.317 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.314 7f438444c700 1 -- 172.21.2.120:0/1313552655 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f437c0b2490 con 0x7f437c098320 2021-02-10T11:13:45.317 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.315 7f436effd700 1 -- 172.21.2.120:0/1313552655 <== mon.1 v2:172.21.2.120:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4374001450 con 0x7f437c098320 2021-02-10T11:13:45.318 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.315 7f436effd700 1 --2- 172.21.2.120:0/1313552655 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f43640186c0 0x7f436401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:45.318 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.315 7f436effd700 1 -- 172.21.2.120:0/1313552655 <== mon.1 v2:172.21.2.120:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1063+0+0 (secure 0 0 0) 0x7f4374031840 con 0x7f437c098320 2021-02-10T11:13:45.318 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.315 7f4382268700 1 --2- 172.21.2.120:0/1313552655 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f43640186c0 0x7f436401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:45.318 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.315 7f4382268700 1 --2- 172.21.2.120:0/1313552655 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f43640186c0 0x7f436401abc0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:13:45.345 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.346 7f438444c700 1 -- 172.21.2.120:0/1313552655 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f437c098720 con 0x7f437c098320 2021-02-10T11:13:45.347 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.349 7f436effd700 1 -- 172.21.2.120:0/1313552655 <== mon.1 v2:172.21.2.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4374006af0 con 0x7f437c098320 2021-02-10T11:13:45.484 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.485 7f438444c700 1 -- 172.21.2.120:0/1313552655 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_command({"prefix": "osd new", "uuid": "58795682-e00d-4173-bb84-f7075ca89a11", "id": 3} v 0) v1 -- 0x7f437c098720 con 0x7f437c098320 2021-02-10T11:13:45.489 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.491 7f436effd700 1 -- 172.21.2.120:0/1313552655 <== mon.1 v2:172.21.2.120:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "58795682-e00d-4173-bb84-f7075ca89a11", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f43740013c0 con 0x7f437c098320 2021-02-10T11:13:45.491 INFO:teuthology.orchestra.run.gibba020.stdout:3 2021-02-10T11:13:45.493 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.494 7f438444c700 1 -- 172.21.2.120:0/1313552655 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f43640186c0 msgr2=0x7f436401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:45.493 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.494 7f438444c700 1 --2- 172.21.2.120:0/1313552655 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f43640186c0 0x7f436401abc0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:45.493 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.494 7f438444c700 1 -- 172.21.2.120:0/1313552655 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f437c098320 msgr2=0x7f437c0b1e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:45.493 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.494 7f438444c700 1 --2- 172.21.2.120:0/1313552655 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f437c098320 0x7f437c0b1e70 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f437400b970 tx=0x7f437400bd10).stop 2021-02-10T11:13:45.493 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.494 7f438444c700 1 -- 172.21.2.120:0/1313552655 shutdown_connections 2021-02-10T11:13:45.493 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.494 7f438444c700 1 --2- 172.21.2.120:0/1313552655 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f43640186c0 0x7f436401abc0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:45.493 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.494 7f438444c700 1 --2- 172.21.2.120:0/1313552655 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f437c098320 0x7f437c0b1e70 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:45.493 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.494 7f438444c700 1 --2- 172.21.2.120:0/1313552655 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f437c0becf0 0x7f437c097d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:45.494 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.494 7f438444c700 1 --2- 172.21.2.120:0/1313552655 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f437c193150 0x7f437c0be6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:13:45.494 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.494 7f438444c700 1 -- 172.21.2.120:0/1313552655 >> 172.21.2.120:0/1313552655 conn(0x7f437c0a1da0 msgr2=0x7f437c09a280 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:13:45.494 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.494 7f438444c700 1 -- 172.21.2.120:0/1313552655 shutdown_connections 2021-02-10T11:13:45.494 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.494 7f438444c700 1 -- 172.21.2.120:0/1313552655 wait complete. 2021-02-10T11:13:45.501 DEBUG:teuthology.orchestra.run.gibba020:> sudo ceph --cluster ceph osd new 6763f8d6-53dc-4e45-bdf6-a6eabcf81e39 4 2021-02-10T11:13:45.927 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.912 7f82ca6d9700 1 Processor -- start 2021-02-10T11:13:45.927 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.912 7f82ca6d9700 1 -- start start 2021-02-10T11:13:45.927 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.912 7f82ca6d9700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f82c40d17f0 0x7f82c41b31e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:45.927 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.912 7f82ca6d9700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82c40f0b20 con 0x7f82c40d1bf0 2021-02-10T11:13:45.927 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.912 7f82ca6d9700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82c41b3ff0 con 0x7f82c40cff00 2021-02-10T11:13:45.927 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.912 7f82ca6d9700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82c40e5eb0 con 0x7f82c40d17f0 2021-02-10T11:13:45.927 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.913 7f82c8cf6700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7f82c40cff00 0x7f82c40ef620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.120:47954/0 (socket says 172.21.2.120:47954) 2021-02-10T11:13:45.927 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.913 7f82c8cf6700 1 -- 172.21.2.120:0/4025843263 learned_addr learned my addr 172.21.2.120:0/4025843263 (peer_addr_for_me v1:172.21.2.120:0/0) 2021-02-10T11:13:45.927 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.913 7f82c3fff700 1 --2- 172.21.2.120:0/4025843263 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f82c40d17f0 0x7f82c41b31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:45.928 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.913 7f82c2ffd700 1 -- 172.21.2.120:0/4025843263 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1759284647 0 0) 0x7f82bc001500 con 0x7f82c40cff00 2021-02-10T11:13:45.928 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.913 7f82c2ffd700 1 -- 172.21.2.120:0/4025843263 --> v1:172.21.2.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82ac002fe0 con 0x7f82c40cff00 2021-02-10T11:13:45.928 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.913 7f82c3fff700 1 -- 172.21.2.120:0/4025843263 >> v1:172.21.2.120:6789/0 conn(0x7f82c40cff00 legacy=0x7f82c40ef620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:45.928 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.913 7f82c3fff700 1 -- 172.21.2.120:0/4025843263 >> v1:172.21.2.112:6789/0 conn(0x7f82c40d1bf0 legacy=0x7f82c41b3800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:13:45.928 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.913 7f82c3fff700 1 -- 172.21.2.120:0/4025843263 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82c40f20b0 con 0x7f82c40d17f0 2021-02-10T11:13:45.928 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.913 7f82c3fff700 1 --2- 172.21.2.120:0/4025843263 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f82c40d17f0 0x7f82c41b31e0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f82b4006650 tx=0x7f82b4009a80).ready entity=mon.2 client_cookie=84f65ab37fa26651 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:13:45.928 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.913 7f82c2ffd700 1 -- 172.21.2.120:0/4025843263 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f82b40009b0 con 0x7f82c40d17f0 2021-02-10T11:13:45.928 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.913 7f82c2ffd700 1 -- 172.21.2.120:0/4025843263 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82b400c0c0 con 0x7f82c40d17f0 2021-02-10T11:13:45.928 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.913 7f82c2ffd700 1 -- 172.21.2.120:0/4025843263 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f82b400c5e0 con 0x7f82c40d17f0 2021-02-10T11:13:45.928 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.914 7f82ca6d9700 1 -- 172.21.2.120:0/4025843263 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f82c40d17f0 msgr2=0x7f82c41b31e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:45.928 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.914 7f82ca6d9700 1 --2- 172.21.2.120:0/4025843263 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f82c40d17f0 0x7f82c41b31e0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f82b4006650 tx=0x7f82b4009a80).stop 2021-02-10T11:13:45.928 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.914 7f82ca6d9700 1 -- 172.21.2.120:0/4025843263 shutdown_connections 2021-02-10T11:13:45.928 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.914 7f82ca6d9700 1 --2- 172.21.2.120:0/4025843263 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f82c40d17f0 0x7f82c41b31e0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:45.928 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.914 7f82ca6d9700 1 -- 172.21.2.120:0/4025843263 >> 172.21.2.120:0/4025843263 conn(0x7f82c40fde40 msgr2=0x7f82c40ce390 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:13:45.929 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.914 7f82ca6d9700 1 -- 172.21.2.120:0/4025843263 shutdown_connections 2021-02-10T11:13:45.929 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.914 7f82ca6d9700 1 -- 172.21.2.120:0/4025843263 wait complete. 2021-02-10T11:13:45.931 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.929 7f82ca6d9700 1 Processor -- start 2021-02-10T11:13:45.931 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.929 7f82ca6d9700 1 -- start start 2021-02-10T11:13:45.931 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.929 7f82ca6d9700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f82c41ae510 0x7f82c4097900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:45.931 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.929 7f82ca6d9700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f82c40a57e0 0x7f82c4097f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:45.931 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.929 7f82ca6d9700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f82c408ce00 0x7f82c40b2e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:45.931 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.929 7f82ca6d9700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82c408c880 con 0x7f82c408ce00 2021-02-10T11:13:45.931 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.929 7f82ca6d9700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82c4098540 con 0x7f82c40a57e0 2021-02-10T11:13:45.931 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.929 7f82ca6d9700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82c40b3470 con 0x7f82c41ae510 2021-02-10T11:13:45.931 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.929 7f82c37fe700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f82c40a57e0 0x7f82c4097f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:45.931 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.929 7f82c3fff700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f82c41ae510 0x7f82c4097900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:45.931 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.929 7f82c37fe700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f82c40a57e0 0x7f82c4097f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3300/0 says I am v2:172.21.2.120:43896/0 (socket says 172.21.2.120:43896) 2021-02-10T11:13:45.932 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.929 7f82c3fff700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f82c41ae510 0x7f82c4097900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3301/0 says I am v2:172.21.2.120:34670/0 (socket says 172.21.2.120:34670) 2021-02-10T11:13:45.932 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.929 7f82c37fe700 1 -- 172.21.2.120:0/3294584261 learned_addr learned my addr 172.21.2.120:0/3294584261 (peer_addr_for_me v2:172.21.2.120:0/0) 2021-02-10T11:13:45.932 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.930 7f82c37fe700 1 -- 172.21.2.120:0/3294584261 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f82c41ae510 msgr2=0x7f82c4097900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:45.932 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.930 7f82c37fe700 1 --2- 172.21.2.120:0/3294584261 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f82c41ae510 0x7f82c4097900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:45.932 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.930 7f82c37fe700 1 -- 172.21.2.120:0/3294584261 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f82c408ce00 msgr2=0x7f82c40b2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:45.932 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.930 7f82c8cf6700 1 --2- 172.21.2.120:0/3294584261 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f82c408ce00 0x7f82c40b2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:45.932 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.930 7f82c37fe700 1 --2- 172.21.2.120:0/3294584261 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f82c408ce00 0x7f82c40b2e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:45.932 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.930 7f82c37fe700 1 -- 172.21.2.120:0/3294584261 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82c40b3690 con 0x7f82c40a57e0 2021-02-10T11:13:45.932 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.930 7f82c3fff700 1 --2- 172.21.2.120:0/3294584261 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f82c41ae510 0x7f82c4097900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:13:45.932 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.930 7f82c8cf6700 1 --2- 172.21.2.120:0/3294584261 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f82c408ce00 0x7f82c40b2e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:13:45.932 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.930 7f82c37fe700 1 --2- 172.21.2.120:0/3294584261 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f82c40a57e0 0x7f82c4097f20 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f82b800b5b0 tx=0x7f82b800b950).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:13:45.932 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.930 7f82c0ff9700 1 -- 172.21.2.120:0/3294584261 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f82b80009b0 con 0x7f82c40a57e0 2021-02-10T11:13:45.932 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.931 7f82c0ff9700 1 -- 172.21.2.120:0/3294584261 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82b800bd40 con 0x7f82c40a57e0 2021-02-10T11:13:45.933 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.931 7f82c0ff9700 1 -- 172.21.2.120:0/3294584261 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f82b8006210 con 0x7f82c40a57e0 2021-02-10T11:13:45.933 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.931 7f82ca6d9700 1 -- 172.21.2.120:0/3294584261 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82c40b3690 con 0x7f82c40a57e0 2021-02-10T11:13:45.933 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.931 7f82ca6d9700 1 -- 172.21.2.120:0/3294584261 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82c40b3c20 con 0x7f82c40a57e0 2021-02-10T11:13:45.933 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.931 7f82c0ff9700 1 -- 172.21.2.120:0/3294584261 <== mon.1 v2:172.21.2.120:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f82b8006020 con 0x7f82c40a57e0 2021-02-10T11:13:45.933 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.931 7f82c0ff9700 1 --2- 172.21.2.120:0/3294584261 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f82a40186c0 0x7f82a401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:45.933 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.931 7f82c0ff9700 1 -- 172.21.2.120:0/3294584261 <== mon.1 v2:172.21.2.120:3300/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1166+0+0 (secure 0 0 0) 0x7f82b8031420 con 0x7f82c40a57e0 2021-02-10T11:13:45.933 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.932 7f82c3fff700 1 --2- 172.21.2.120:0/3294584261 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f82a40186c0 0x7f82a401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:45.933 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.932 7f82c3fff700 1 --2- 172.21.2.120:0/3294584261 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f82a40186c0 0x7f82a401abc0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:13:45.961 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.963 7f82ca6d9700 1 -- 172.21.2.120:0/3294584261 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82c408d200 con 0x7f82c40a57e0 2021-02-10T11:13:45.963 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:45.965 7f82c0ff9700 1 -- 172.21.2.120:0/3294584261 <== mon.1 v2:172.21.2.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f82b80009b0 con 0x7f82c40a57e0 2021-02-10T11:13:46.100 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:46.102 7f82ca6d9700 1 -- 172.21.2.120:0/3294584261 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_command({"prefix": "osd new", "uuid": "6763f8d6-53dc-4e45-bdf6-a6eabcf81e39", "id": 4} v 0) v1 -- 0x7f82c408d200 con 0x7f82c40a57e0 2021-02-10T11:13:46.104 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:46.106 7f82c0ff9700 1 -- 172.21.2.120:0/3294584261 <== mon.1 v2:172.21.2.120:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "6763f8d6-53dc-4e45-bdf6-a6eabcf81e39", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7f82b8006000 con 0x7f82c40a57e0 2021-02-10T11:13:46.107 INFO:teuthology.orchestra.run.gibba020.stdout:4 2021-02-10T11:13:46.109 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:46.110 7f82ca6d9700 1 -- 172.21.2.120:0/3294584261 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f82a40186c0 msgr2=0x7f82a401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:46.109 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:46.110 7f82ca6d9700 1 --2- 172.21.2.120:0/3294584261 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f82a40186c0 0x7f82a401abc0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:46.109 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:46.110 7f82ca6d9700 1 -- 172.21.2.120:0/3294584261 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f82c40a57e0 msgr2=0x7f82c4097f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:46.109 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:46.110 7f82ca6d9700 1 --2- 172.21.2.120:0/3294584261 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f82c40a57e0 0x7f82c4097f20 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f82b800b5b0 tx=0x7f82b800b950).stop 2021-02-10T11:13:46.109 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:46.110 7f82ca6d9700 1 -- 172.21.2.120:0/3294584261 shutdown_connections 2021-02-10T11:13:46.109 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:46.110 7f82ca6d9700 1 --2- 172.21.2.120:0/3294584261 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f82a40186c0 0x7f82a401abc0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:46.109 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:46.110 7f82ca6d9700 1 --2- 172.21.2.120:0/3294584261 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f82c40a57e0 0x7f82c4097f20 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:46.109 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:46.110 7f82ca6d9700 1 --2- 172.21.2.120:0/3294584261 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f82c408ce00 0x7f82c40b2e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:46.110 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:46.110 7f82ca6d9700 1 --2- 172.21.2.120:0/3294584261 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f82c41ae510 0x7f82c4097900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:46.110 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:46.110 7f82ca6d9700 1 -- 172.21.2.120:0/3294584261 >> 172.21.2.120:0/3294584261 conn(0x7f82c40974c0 msgr2=0x7f82c41aeab0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:13:46.110 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:46.111 7f82ca6d9700 1 -- 172.21.2.120:0/3294584261 shutdown_connections 2021-02-10T11:13:46.110 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:46.111 7f82ca6d9700 1 -- 172.21.2.120:0/3294584261 wait complete. 2021-02-10T11:13:46.117 DEBUG:teuthology.orchestra.run.gibba020:> sudo ceph --cluster ceph osd new 5d0e4dd6-0cb0-4bf8-8314-626acaa1b544 5 2021-02-10T11:13:46.543 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:46.528 7f940048d700 1 Processor -- start 2021-02-10T11:13:46.543 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:46.528 7f940048d700 1 -- start start 2021-02-10T11:13:46.543 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:46.529 7f940048d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f93f80d17f0 0x7f93f81b31e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:46.544 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:46.529 7f940048d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f93f80cff00 0x7f93f80ef620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:46.544 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:46.529 7f940048d700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93f80f0b20 con 0x7f93f80d1bf0 2021-02-10T11:13:46.544 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:46.529 7f940048d700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93f80e5eb0 con 0x7f93f80cff00 2021-02-10T11:13:46.544 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:46.529 7f940048d700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93f81b3ff0 con 0x7f93f80d17f0 2021-02-10T11:13:46.544 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:46.529 7f93fe2a9700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f93f80d17f0 0x7f93f81b31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:46.544 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:46.529 7f93fdaa8700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7f93f80d1bf0 0x7f93f81b3800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.120:47966/0 (socket says 172.21.2.120:47966) 2021-02-10T11:13:46.544 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:46.529 7f93fdaa8700 1 -- 172.21.2.120:0/2799502847 learned_addr learned my addr 172.21.2.120:0/2799502847 (peer_addr_for_me v1:172.21.2.120:0/0) 2021-02-10T11:13:46.544 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:46.530 7f93fe2a9700 1 -- 172.21.2.120:0/2799502847 >> v2:172.21.2.112:3300/0 conn(0x7f93f80cff00 msgr2=0x7f93f80ef620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:13:46.544 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:46.530 7f93fe2a9700 1 --2- 172.21.2.120:0/2799502847 >> v2:172.21.2.112:3300/0 conn(0x7f93f80cff00 0x7f93f80ef620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:13:46.544 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:46.530 7f93fe2a9700 1 -- 172.21.2.120:0/2799502847 >> v1:172.21.2.120:6789/0 conn(0x7f93f80d1bf0 legacy=0x7f93f81b3800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:46.544 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:46.530 7f93fe2a9700 1 -- 172.21.2.120:0/2799502847 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93f80f20b0 con 0x7f93f80d17f0 2021-02-10T11:13:46.544 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:46.530 7f93fe2a9700 1 --2- 172.21.2.120:0/2799502847 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f93f80d17f0 0x7f93f81b31e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f93f4006650 tx=0x7f93f4009a00).ready entity=mon.2 client_cookie=5871558d8f8b84b8 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:13:46.544 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:46.530 7f93fd2a7700 1 -- 172.21.2.120:0/2799502847 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f93f40009b0 con 0x7f93f80d17f0 2021-02-10T11:13:46.545 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:46.530 7f93fd2a7700 1 -- 172.21.2.120:0/2799502847 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93f400c0c0 con 0x7f93f80d17f0 2021-02-10T11:13:46.545 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:46.530 7f93fd2a7700 1 -- 172.21.2.120:0/2799502847 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f93f400c5a0 con 0x7f93f80d17f0 2021-02-10T11:13:46.545 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:46.530 7f940048d700 1 -- 172.21.2.120:0/2799502847 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f93f80d17f0 msgr2=0x7f93f81b31e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:46.545 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:46.530 7f940048d700 1 --2- 172.21.2.120:0/2799502847 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f93f80d17f0 0x7f93f81b31e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f93f4006650 tx=0x7f93f4009a00).stop 2021-02-10T11:13:46.545 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:46.531 7f940048d700 1 -- 172.21.2.120:0/2799502847 shutdown_connections 2021-02-10T11:13:46.545 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:46.531 7f940048d700 1 --2- 172.21.2.120:0/2799502847 >> v2:172.21.2.112:3300/0 conn(0x7f93f80cff00 0x7f93f80ef620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:13:46.545 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:46.531 7f940048d700 1 --2- 172.21.2.120:0/2799502847 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f93f80d17f0 0x7f93f81b31e0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:46.545 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:46.531 7f940048d700 1 -- 172.21.2.120:0/2799502847 >> 172.21.2.120:0/2799502847 conn(0x7f93f80fde40 msgr2=0x7f93f80ce390 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:13:46.545 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:46.531 7f940048d700 1 -- 172.21.2.120:0/2799502847 shutdown_connections 2021-02-10T11:13:46.545 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:46.531 7f940048d700 1 -- 172.21.2.120:0/2799502847 wait complete. 2021-02-10T11:13:46.547 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:46.545 7f940048d700 1 Processor -- start 2021-02-10T11:13:46.547 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:46.545 7f940048d700 1 -- start start 2021-02-10T11:13:46.547 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:46.546 7f940048d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f93f81ae510 0x7f93f8097900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:46.547 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:46.546 7f940048d700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f93f80a57e0 0x7f93f8097f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:46.548 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:46.546 7f940048d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f93f808ce00 0x7f93f80b2e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:46.548 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:46.546 7f940048d700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93f808c880 con 0x7f93f81ae510 2021-02-10T11:13:46.548 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:46.546 7f940048d700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93f8098540 con 0x7f93f80a57e0 2021-02-10T11:13:46.548 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:46.546 7f940048d700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93f80b3470 con 0x7f93f808ce00 2021-02-10T11:13:46.548 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:46.546 7f93fdaa8700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f93f80a57e0 0x7f93f8097f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:46.548 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:46.546 7f93fdaa8700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f93f80a57e0 0x7f93f8097f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3300/0 says I am v2:172.21.2.120:43910/0 (socket says 172.21.2.120:43910) 2021-02-10T11:13:46.548 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:46.546 7f93fdaa8700 1 -- 172.21.2.120:0/819671468 learned_addr learned my addr 172.21.2.120:0/819671468 (peer_addr_for_me v2:172.21.2.120:0/0) 2021-02-10T11:13:46.548 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:46.546 7f93feaaa700 1 --2- 172.21.2.120:0/819671468 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f93f808ce00 0x7f93f80b2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:46.548 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:46.546 7f93fe2a9700 1 --2- 172.21.2.120:0/819671468 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f93f81ae510 0x7f93f8097900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:46.548 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:46.546 7f93fdaa8700 1 -- 172.21.2.120:0/819671468 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f93f808ce00 msgr2=0x7f93f80b2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:46.548 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:46.547 7f93fdaa8700 1 --2- 172.21.2.120:0/819671468 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f93f808ce00 0x7f93f80b2e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:46.548 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:46.547 7f93fdaa8700 1 -- 172.21.2.120:0/819671468 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f93f81ae510 msgr2=0x7f93f8097900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:46.548 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:46.547 7f93fdaa8700 1 --2- 172.21.2.120:0/819671468 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f93f81ae510 0x7f93f8097900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:46.549 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:46.547 7f93fdaa8700 1 -- 172.21.2.120:0/819671468 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93f80b3690 con 0x7f93f80a57e0 2021-02-10T11:13:46.549 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:46.547 7f93fe2a9700 1 --2- 172.21.2.120:0/819671468 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f93f81ae510 0x7f93f8097900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:13:46.549 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:46.547 7f93fdaa8700 1 --2- 172.21.2.120:0/819671468 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f93f80a57e0 0x7f93f8097f20 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f93ec001c60 tx=0x7f93ec00a8d0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:13:46.549 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:46.547 7f93eaffd700 1 -- 172.21.2.120:0/819671468 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f93ec00acc0 con 0x7f93f80a57e0 2021-02-10T11:13:46.549 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:46.547 7f93eaffd700 1 -- 172.21.2.120:0/819671468 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93ec004140 con 0x7f93f80a57e0 2021-02-10T11:13:46.549 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:46.547 7f93eaffd700 1 -- 172.21.2.120:0/819671468 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f93ec004620 con 0x7f93f80a57e0 2021-02-10T11:13:46.549 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:46.547 7f940048d700 1 -- 172.21.2.120:0/819671468 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93f80b3690 con 0x7f93f80a57e0 2021-02-10T11:13:46.549 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:46.547 7f940048d700 1 -- 172.21.2.120:0/819671468 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93f80b3c20 con 0x7f93f80a57e0 2021-02-10T11:13:46.549 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:46.548 7f93eaffd700 1 -- 172.21.2.120:0/819671468 <== mon.1 v2:172.21.2.120:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f93ec004af0 con 0x7f93f80a57e0 2021-02-10T11:13:46.549 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:46.548 7f93eaffd700 1 --2- 172.21.2.120:0/819671468 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f93e00186c0 0x7f93e001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:46.549 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:46.548 7f93eaffd700 1 -- 172.21.2.120:0/819671468 <== mon.1 v2:172.21.2.120:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1269+0+0 (secure 0 0 0) 0x7f93ec0314c0 con 0x7f93f80a57e0 2021-02-10T11:13:46.549 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:46.548 7f93fe2a9700 1 --2- 172.21.2.120:0/819671468 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f93e00186c0 0x7f93e001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:46.549 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:46.548 7f93fe2a9700 1 --2- 172.21.2.120:0/819671468 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f93e00186c0 0x7f93e001abc0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:13:46.577 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:46.579 7f940048d700 1 -- 172.21.2.120:0/819671468 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93f808d200 con 0x7f93f80a57e0 2021-02-10T11:13:46.580 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:46.581 7f93eaffd700 1 -- 172.21.2.120:0/819671468 <== mon.1 v2:172.21.2.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f93ec0048e0 con 0x7f93f80a57e0 2021-02-10T11:13:46.719 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:46.720 7f940048d700 1 -- 172.21.2.120:0/819671468 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_command({"prefix": "osd new", "uuid": "5d0e4dd6-0cb0-4bf8-8314-626acaa1b544", "id": 5} v 0) v1 -- 0x7f93f808d200 con 0x7f93f80a57e0 2021-02-10T11:13:46.723 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:46.725 7f93eaffd700 1 -- 172.21.2.120:0/819671468 <== mon.1 v2:172.21.2.120:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "5d0e4dd6-0cb0-4bf8-8314-626acaa1b544", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7f93ec004ad0 con 0x7f93f80a57e0 2021-02-10T11:13:46.726 INFO:teuthology.orchestra.run.gibba020.stdout:5 2021-02-10T11:13:46.728 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:46.729 7f940048d700 1 -- 172.21.2.120:0/819671468 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f93e00186c0 msgr2=0x7f93e001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:46.728 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:46.729 7f940048d700 1 --2- 172.21.2.120:0/819671468 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f93e00186c0 0x7f93e001abc0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:46.728 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:46.729 7f940048d700 1 -- 172.21.2.120:0/819671468 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f93f80a57e0 msgr2=0x7f93f8097f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:46.728 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:46.729 7f940048d700 1 --2- 172.21.2.120:0/819671468 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f93f80a57e0 0x7f93f8097f20 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f93ec001c60 tx=0x7f93ec00a8d0).stop 2021-02-10T11:13:46.728 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:46.729 7f940048d700 1 -- 172.21.2.120:0/819671468 shutdown_connections 2021-02-10T11:13:46.728 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:46.729 7f940048d700 1 --2- 172.21.2.120:0/819671468 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f93e00186c0 0x7f93e001abc0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:46.728 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:46.729 7f940048d700 1 --2- 172.21.2.120:0/819671468 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f93f80a57e0 0x7f93f8097f20 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:46.728 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:46.729 7f940048d700 1 --2- 172.21.2.120:0/819671468 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f93f808ce00 0x7f93f80b2e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:46.728 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:46.729 7f940048d700 1 --2- 172.21.2.120:0/819671468 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f93f81ae510 0x7f93f8097900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:46.729 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:46.729 7f940048d700 1 -- 172.21.2.120:0/819671468 >> 172.21.2.120:0/819671468 conn(0x7f93f80974c0 msgr2=0x7f93f81aeab0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:13:46.729 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:46.729 7f940048d700 1 -- 172.21.2.120:0/819671468 shutdown_connections 2021-02-10T11:13:46.729 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:46.729 7f940048d700 1 -- 172.21.2.120:0/819671468 wait complete. 2021-02-10T11:13:46.736 DEBUG:teuthology.orchestra.run.gibba020:> sudo ceph --cluster ceph osd new b010e187-4541-4b00-80dd-f948f6ab8f15 6 2021-02-10T11:13:47.162 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.147 7f7d6bb2e700 1 Processor -- start 2021-02-10T11:13:47.162 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.147 7f7d6bb2e700 1 -- start start 2021-02-10T11:13:47.162 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.147 7f7d6bb2e700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f7d640cff00 0x7f7d641b39d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:47.162 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.147 7f7d6bb2e700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7d640d1bf0 0x7f7d640ef620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:47.162 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.147 7f7d6bb2e700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d640f0b20 con 0x7f7d640d17f0 2021-02-10T11:13:47.162 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.147 7f7d6bb2e700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d641b3ff0 con 0x7f7d640cff00 2021-02-10T11:13:47.162 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.147 7f7d6bb2e700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d640f20b0 con 0x7f7d640d1bf0 2021-02-10T11:13:47.162 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.148 7f7d69149700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f7d640cff00 0x7f7d641b39d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:47.162 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.148 7f7d6a14b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7d640d1bf0 0x7f7d640ef620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:47.163 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.148 7f7d6a14b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7d640d1bf0 0x7f7d640ef620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3301/0 says I am v2:172.21.2.120:34692/0 (socket says 172.21.2.120:34692) 2021-02-10T11:13:47.163 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.148 7f7d69149700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f7d640cff00 0x7f7d641b39d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3300/0 says I am v2:172.21.2.120:43916/0 (socket says 172.21.2.120:43916) 2021-02-10T11:13:47.163 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.148 7f7d6a14b700 1 -- 172.21.2.120:0/3646519198 learned_addr learned my addr 172.21.2.120:0/3646519198 (peer_addr_for_me v2:172.21.2.120:0/0) 2021-02-10T11:13:47.163 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.148 7f7d6a14b700 1 -- 172.21.2.120:0/3646519198 >> v2:172.21.2.120:3300/0 conn(0x7f7d640cff00 msgr2=0x7f7d641b39d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:13:47.163 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.148 7f7d6a14b700 1 --2- 172.21.2.120:0/3646519198 >> v2:172.21.2.120:3300/0 conn(0x7f7d640cff00 0x7f7d641b39d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:47.163 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.148 7f7d6a14b700 1 -- 172.21.2.120:0/3646519198 >> v1:172.21.2.120:6789/0 conn(0x7f7d640d17f0 legacy=0x7f7d641b31e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:47.163 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.148 7f7d6a14b700 1 -- 172.21.2.120:0/3646519198 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d640f22d0 con 0x7f7d640d1bf0 2021-02-10T11:13:47.163 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.148 7f7d6a14b700 1 --2- 172.21.2.120:0/3646519198 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7d640d1bf0 0x7f7d640ef620 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f7d5c006f00 tx=0x7f7d5c009f10).ready entity=mon.2 client_cookie=f9abdb219cd39011 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:13:47.163 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.149 7f7d68948700 1 -- 172.21.2.120:0/3646519198 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7d5c00b3b0 con 0x7f7d640d1bf0 2021-02-10T11:13:47.163 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.149 7f7d68948700 1 -- 172.21.2.120:0/3646519198 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d5c00b730 con 0x7f7d640d1bf0 2021-02-10T11:13:47.163 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.149 7f7d68948700 1 -- 172.21.2.120:0/3646519198 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7d5c00bc50 con 0x7f7d640d1bf0 2021-02-10T11:13:47.163 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.149 7f7d6bb2e700 1 -- 172.21.2.120:0/3646519198 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7d640d1bf0 msgr2=0x7f7d640ef620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:47.163 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.149 7f7d6bb2e700 1 --2- 172.21.2.120:0/3646519198 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7d640d1bf0 0x7f7d640ef620 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f7d5c006f00 tx=0x7f7d5c009f10).stop 2021-02-10T11:13:47.164 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.149 7f7d6bb2e700 1 -- 172.21.2.120:0/3646519198 shutdown_connections 2021-02-10T11:13:47.164 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.149 7f7d6bb2e700 1 --2- 172.21.2.120:0/3646519198 >> v2:172.21.2.120:3300/0 conn(0x7f7d640cff00 0x7f7d641b39d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:47.164 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.149 7f7d6bb2e700 1 --2- 172.21.2.120:0/3646519198 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7d640d1bf0 0x7f7d640ef620 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:47.164 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.149 7f7d6bb2e700 1 -- 172.21.2.120:0/3646519198 >> 172.21.2.120:0/3646519198 conn(0x7f7d640fde40 msgr2=0x7f7d640ce390 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:13:47.164 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.150 7f7d6bb2e700 1 -- 172.21.2.120:0/3646519198 shutdown_connections 2021-02-10T11:13:47.164 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.150 7f7d6bb2e700 1 -- 172.21.2.120:0/3646519198 wait complete. 2021-02-10T11:13:47.166 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.164 7f7d6bb2e700 1 Processor -- start 2021-02-10T11:13:47.166 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.164 7f7d6bb2e700 1 -- start start 2021-02-10T11:13:47.166 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.164 7f7d6bb2e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7d641a17c0 0x7f7d640acf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:47.166 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.164 7f7d6bb2e700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7d640ad560 0x7f7d641a0230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:47.166 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.164 7f7d6bb2e700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7d640ad960 0x7f7d641a0850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:47.167 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.164 7f7d6bb2e700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d641b2e10 con 0x7f7d641a17c0 2021-02-10T11:13:47.167 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.164 7f7d6bb2e700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d641a1320 con 0x7f7d640ad960 2021-02-10T11:13:47.167 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.164 7f7d6bb2e700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d641a0e70 con 0x7f7d640ad560 2021-02-10T11:13:47.167 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.164 7f7d6a14b700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7d640ad960 0x7f7d641a0850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:47.167 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.165 7f7d6a14b700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7d640ad960 0x7f7d641a0850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3300/0 says I am v2:172.21.2.120:43920/0 (socket says 172.21.2.120:43920) 2021-02-10T11:13:47.167 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.165 7f7d6a14b700 1 -- 172.21.2.120:0/1015207606 learned_addr learned my addr 172.21.2.120:0/1015207606 (peer_addr_for_me v2:172.21.2.120:0/0) 2021-02-10T11:13:47.167 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.165 7f7d69149700 1 --2- 172.21.2.120:0/1015207606 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7d640ad560 0x7f7d641a0230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:47.167 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.165 7f7d6a14b700 1 -- 172.21.2.120:0/1015207606 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7d640ad560 msgr2=0x7f7d641a0230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:47.167 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.165 7f7d6a14b700 1 --2- 172.21.2.120:0/1015207606 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7d640ad560 0x7f7d641a0230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:47.167 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.165 7f7d6a14b700 1 -- 172.21.2.120:0/1015207606 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7d641a17c0 msgr2=0x7f7d640acf40 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:13:47.167 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.165 7f7d6a14b700 1 --2- 172.21.2.120:0/1015207606 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7d641a17c0 0x7f7d640acf40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:13:47.167 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.165 7f7d6a14b700 1 -- 172.21.2.120:0/1015207606 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d641a1090 con 0x7f7d640ad960 2021-02-10T11:13:47.168 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.165 7f7d6a14b700 1 --2- 172.21.2.120:0/1015207606 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7d640ad960 0x7f7d641a0850 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f7d5c00b9b0 tx=0x7f7d5c00bd50).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:13:47.168 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.165 7f7d5a7fc700 1 -- 172.21.2.120:0/1015207606 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7d5c005820 con 0x7f7d640ad960 2021-02-10T11:13:47.168 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.165 7f7d5a7fc700 1 -- 172.21.2.120:0/1015207606 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d5c005a60 con 0x7f7d640ad960 2021-02-10T11:13:47.168 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.165 7f7d6bb2e700 1 -- 172.21.2.120:0/1015207606 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d641a1090 con 0x7f7d640ad960 2021-02-10T11:13:47.168 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.165 7f7d5a7fc700 1 -- 172.21.2.120:0/1015207606 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7d5c006030 con 0x7f7d640ad960 2021-02-10T11:13:47.168 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.166 7f7d6bb2e700 1 -- 172.21.2.120:0/1015207606 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d641a1090 con 0x7f7d640ad960 2021-02-10T11:13:47.168 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.166 7f7d5a7fc700 1 -- 172.21.2.120:0/1015207606 <== mon.1 v2:172.21.2.120:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7d5c005c40 con 0x7f7d640ad960 2021-02-10T11:13:47.168 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.166 7f7d5a7fc700 1 --2- 172.21.2.120:0/1015207606 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7d4c018730 0x7f7d4c01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:47.168 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.166 7f7d5a7fc700 1 -- 172.21.2.120:0/1015207606 <== mon.1 v2:172.21.2.120:3300/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1372+0+0 (secure 0 0 0) 0x7f7d5c031cb0 con 0x7f7d640ad960 2021-02-10T11:13:47.168 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.166 7f7d6994a700 1 --2- 172.21.2.120:0/1015207606 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7d4c018730 0x7f7d4c01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:47.169 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.167 7f7d6994a700 1 --2- 172.21.2.120:0/1015207606 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7d4c018730 0x7f7d4c01ac10 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:13:47.196 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.198 7f7d6bb2e700 1 -- 172.21.2.120:0/1015207606 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d64105630 con 0x7f7d640ad960 2021-02-10T11:13:47.198 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.200 7f7d5a7fc700 1 -- 172.21.2.120:0/1015207606 <== mon.1 v2:172.21.2.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7d5c009450 con 0x7f7d640ad960 2021-02-10T11:13:47.333 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.335 7f7d6bb2e700 1 -- 172.21.2.120:0/1015207606 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_command({"prefix": "osd new", "uuid": "b010e187-4541-4b00-80dd-f948f6ab8f15", "id": 6} v 0) v1 -- 0x7f7d64105630 con 0x7f7d640ad960 2021-02-10T11:13:47.338 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.340 7f7d5a7fc700 1 -- 172.21.2.120:0/1015207606 <== mon.1 v2:172.21.2.120:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b010e187-4541-4b00-80dd-f948f6ab8f15", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7f7d5c005a60 con 0x7f7d640ad960 2021-02-10T11:13:47.340 INFO:teuthology.orchestra.run.gibba020.stdout:6 2021-02-10T11:13:47.342 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.343 7f7d6bb2e700 1 -- 172.21.2.120:0/1015207606 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7d4c018730 msgr2=0x7f7d4c01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:47.342 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.343 7f7d6bb2e700 1 --2- 172.21.2.120:0/1015207606 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7d4c018730 0x7f7d4c01ac10 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:47.342 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.343 7f7d6bb2e700 1 -- 172.21.2.120:0/1015207606 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7d640ad960 msgr2=0x7f7d641a0850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:47.342 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.343 7f7d6bb2e700 1 --2- 172.21.2.120:0/1015207606 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7d640ad960 0x7f7d641a0850 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f7d5c00b9b0 tx=0x7f7d5c00bd50).stop 2021-02-10T11:13:47.342 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.344 7f7d6bb2e700 1 -- 172.21.2.120:0/1015207606 shutdown_connections 2021-02-10T11:13:47.342 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.344 7f7d6bb2e700 1 --2- 172.21.2.120:0/1015207606 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7d4c018730 0x7f7d4c01ac10 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:47.342 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.344 7f7d6bb2e700 1 --2- 172.21.2.120:0/1015207606 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7d640ad960 0x7f7d641a0850 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:47.343 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.344 7f7d6bb2e700 1 --2- 172.21.2.120:0/1015207606 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7d640ad560 0x7f7d641a0230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:47.343 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.344 7f7d6bb2e700 1 --2- 172.21.2.120:0/1015207606 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7d641a17c0 0x7f7d640acf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:13:47.343 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.344 7f7d6bb2e700 1 -- 172.21.2.120:0/1015207606 >> 172.21.2.120:0/1015207606 conn(0x7f7d641ba190 msgr2=0x7f7d64097800 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:13:47.343 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.344 7f7d6bb2e700 1 -- 172.21.2.120:0/1015207606 shutdown_connections 2021-02-10T11:13:47.343 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.344 7f7d6bb2e700 1 -- 172.21.2.120:0/1015207606 wait complete. 2021-02-10T11:13:47.350 DEBUG:teuthology.orchestra.run.gibba020:> sudo ceph --cluster ceph osd new 671e2772-33b7-497e-8e44-2178bd850bcf 7 2021-02-10T11:13:47.780 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.764 7f7b070a5700 1 Processor -- start 2021-02-10T11:13:47.780 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.764 7f7b070a5700 1 -- start start 2021-02-10T11:13:47.780 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.765 7f7b070a5700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7b000d17f0 0x7f7b001b31e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:47.780 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.765 7f7b070a5700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f7b000cff00 0x7f7b000ef620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:47.780 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.765 7f7b070a5700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b000f0b20 con 0x7f7b000d1bf0 2021-02-10T11:13:47.780 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.765 7f7b070a5700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b000e5eb0 con 0x7f7b000cff00 2021-02-10T11:13:47.781 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.765 7f7b070a5700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b001b3ff0 con 0x7f7b000d17f0 2021-02-10T11:13:47.781 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.765 7f7b04ec1700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7b000d17f0 0x7f7b001b31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:47.781 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.765 7f7b04ec1700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7b000d17f0 0x7f7b001b31e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3301/0 says I am v2:172.21.2.120:34704/0 (socket says 172.21.2.120:34704) 2021-02-10T11:13:47.781 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.765 7f7b04ec1700 1 -- 172.21.2.120:0/1789309125 learned_addr learned my addr 172.21.2.120:0/1789309125 (peer_addr_for_me v2:172.21.2.120:0/0) 2021-02-10T11:13:47.781 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.765 7f7b056c2700 1 --2- 172.21.2.120:0/1789309125 >> v2:172.21.2.112:3300/0 conn(0x7f7b000cff00 0x7f7b000ef620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:47.781 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.765 7f7b04ec1700 1 -- 172.21.2.120:0/1789309125 >> v2:172.21.2.112:3300/0 conn(0x7f7b000cff00 msgr2=0x7f7b000ef620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:13:47.781 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.765 7f7b04ec1700 1 --2- 172.21.2.120:0/1789309125 >> v2:172.21.2.112:3300/0 conn(0x7f7b000cff00 0x7f7b000ef620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:47.781 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.765 7f7b04ec1700 1 -- 172.21.2.120:0/1789309125 >> v1:172.21.2.112:6789/0 conn(0x7f7b000d1bf0 legacy=0x7f7b001b3800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:47.781 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.765 7f7b04ec1700 1 -- 172.21.2.120:0/1789309125 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b000f20b0 con 0x7f7b000d17f0 2021-02-10T11:13:47.781 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.766 7f7b04ec1700 1 --2- 172.21.2.120:0/1789309125 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7b000d17f0 0x7f7b001b31e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f7ae8006700 tx=0x7f7ae8009b80).ready entity=mon.2 client_cookie=cd88d3ca6df13b5a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:13:47.781 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.766 7f7aff7fe700 1 -- 172.21.2.120:0/1789309125 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7ae800c0c0 con 0x7f7b000d17f0 2021-02-10T11:13:47.781 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.766 7f7aff7fe700 1 -- 172.21.2.120:0/1789309125 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ae800c340 con 0x7f7b000d17f0 2021-02-10T11:13:47.782 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.766 7f7aff7fe700 1 -- 172.21.2.120:0/1789309125 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7ae800c860 con 0x7f7b000d17f0 2021-02-10T11:13:47.782 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.766 7f7b070a5700 1 -- 172.21.2.120:0/1789309125 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7b000d17f0 msgr2=0x7f7b001b31e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:47.782 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.766 7f7b070a5700 1 --2- 172.21.2.120:0/1789309125 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7b000d17f0 0x7f7b001b31e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f7ae8006700 tx=0x7f7ae8009b80).stop 2021-02-10T11:13:47.782 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.767 7f7b070a5700 1 -- 172.21.2.120:0/1789309125 shutdown_connections 2021-02-10T11:13:47.782 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.767 7f7b070a5700 1 --2- 172.21.2.120:0/1789309125 >> v2:172.21.2.112:3300/0 conn(0x7f7b000cff00 0x7f7b000ef620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:47.782 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.767 7f7b070a5700 1 --2- 172.21.2.120:0/1789309125 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7b000d17f0 0x7f7b001b31e0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:47.782 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.767 7f7b070a5700 1 -- 172.21.2.120:0/1789309125 >> 172.21.2.120:0/1789309125 conn(0x7f7b000fde40 msgr2=0x7f7b000ce390 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:13:47.782 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.767 7f7b070a5700 1 -- 172.21.2.120:0/1789309125 shutdown_connections 2021-02-10T11:13:47.782 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.767 7f7b070a5700 1 -- 172.21.2.120:0/1789309125 wait complete. 2021-02-10T11:13:47.782 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.781 7f7b070a5700 1 Processor -- start 2021-02-10T11:13:47.782 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.781 7f7b070a5700 1 -- start start 2021-02-10T11:13:47.784 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.782 7f7b070a5700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7b001a49b0 0x7f7b0008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:47.784 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.782 7f7b070a5700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7b0009e080 0x7f7b000c3100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:47.784 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.782 7f7b070a5700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7b0009e480 0x7f7b000c3720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:47.784 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.782 7f7b070a5700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b000c2b80 con 0x7f7b001a49b0 2021-02-10T11:13:47.784 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.782 7f7b070a5700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b000bee50 con 0x7f7b0009e080 2021-02-10T11:13:47.784 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.782 7f7b070a5700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b000c13f0 con 0x7f7b0009e480 2021-02-10T11:13:47.784 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.782 7f7afffff700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7b0009e080 0x7f7b000c3100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:47.784 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.782 7f7afffff700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7b0009e080 0x7f7b000c3100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3300/0 says I am v2:172.21.2.120:43934/0 (socket says 172.21.2.120:43934) 2021-02-10T11:13:47.784 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.782 7f7b056c2700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7b0009e480 0x7f7b000c3720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:47.784 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.782 7f7afffff700 1 -- 172.21.2.120:0/2938844374 learned_addr learned my addr 172.21.2.120:0/2938844374 (peer_addr_for_me v2:172.21.2.120:0/0) 2021-02-10T11:13:47.785 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.782 7f7b04ec1700 1 --2- 172.21.2.120:0/2938844374 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7b001a49b0 0x7f7b0008a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:47.785 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.782 7f7afffff700 1 -- 172.21.2.120:0/2938844374 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7b0009e480 msgr2=0x7f7b000c3720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:47.785 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.782 7f7afffff700 1 --2- 172.21.2.120:0/2938844374 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7b0009e480 0x7f7b000c3720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:47.785 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.782 7f7afffff700 1 -- 172.21.2.120:0/2938844374 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7b001a49b0 msgr2=0x7f7b0008a400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:47.785 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.782 7f7afffff700 1 --2- 172.21.2.120:0/2938844374 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7b001a49b0 0x7f7b0008a400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:47.785 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.782 7f7afffff700 1 -- 172.21.2.120:0/2938844374 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b000c1610 con 0x7f7b0009e080 2021-02-10T11:13:47.785 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.783 7f7afffff700 1 --2- 172.21.2.120:0/2938844374 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7b0009e080 0x7f7b000c3100 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f7af0009940 tx=0x7f7af000bcf0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:13:47.785 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.783 7f7afd7fa700 1 -- 172.21.2.120:0/2938844374 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7af000c2f0 con 0x7f7b0009e080 2021-02-10T11:13:47.785 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.783 7f7afd7fa700 1 -- 172.21.2.120:0/2938844374 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7af000c570 con 0x7f7b0009e080 2021-02-10T11:13:47.785 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.783 7f7b070a5700 1 -- 172.21.2.120:0/2938844374 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b000c1610 con 0x7f7b0009e080 2021-02-10T11:13:47.785 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.783 7f7afd7fa700 1 -- 172.21.2.120:0/2938844374 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7af000ca90 con 0x7f7b0009e080 2021-02-10T11:13:47.785 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.783 7f7b070a5700 1 -- 172.21.2.120:0/2938844374 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b000c1610 con 0x7f7b0009e080 2021-02-10T11:13:47.785 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.783 7f7afd7fa700 1 -- 172.21.2.120:0/2938844374 <== mon.1 v2:172.21.2.120:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7af0003020 con 0x7f7b0009e080 2021-02-10T11:13:47.786 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.783 7f7afd7fa700 1 --2- 172.21.2.120:0/2938844374 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7ae4018730 0x7f7ae401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:47.786 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.783 7f7afd7fa700 1 -- 172.21.2.120:0/2938844374 <== mon.1 v2:172.21.2.120:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1475+0+0 (secure 0 0 0) 0x7f7af0031a10 con 0x7f7b0009e080 2021-02-10T11:13:47.786 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.784 7f7b04ec1700 1 --2- 172.21.2.120:0/2938844374 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7ae4018730 0x7f7ae401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:47.786 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.784 7f7b04ec1700 1 --2- 172.21.2.120:0/2938844374 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7ae4018730 0x7f7ae401ac10 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:13:47.812 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.813 7f7b070a5700 1 -- 172.21.2.120:0/2938844374 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b00103690 con 0x7f7b0009e080 2021-02-10T11:13:47.814 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.816 7f7afd7fa700 1 -- 172.21.2.120:0/2938844374 <== mon.1 v2:172.21.2.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7af000be30 con 0x7f7b0009e080 2021-02-10T11:13:47.954 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.955 7f7b070a5700 1 -- 172.21.2.120:0/2938844374 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_command({"prefix": "osd new", "uuid": "671e2772-33b7-497e-8e44-2178bd850bcf", "id": 7} v 0) v1 -- 0x7f7b000f0b20 con 0x7f7b0009e080 2021-02-10T11:13:48.045 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.960 7f7afd7fa700 1 -- 172.21.2.120:0/2938844374 <== mon.1 v2:172.21.2.120:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "671e2772-33b7-497e-8e44-2178bd850bcf", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7f7af0002fb0 con 0x7f7b0009e080 2021-02-10T11:13:48.046 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.963 7f7b070a5700 1 -- 172.21.2.120:0/2938844374 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7ae4018730 msgr2=0x7f7ae401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:48.046 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.963 7f7b070a5700 1 --2- 172.21.2.120:0/2938844374 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7ae4018730 0x7f7ae401ac10 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:48.046 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.963 7f7b070a5700 1 -- 172.21.2.120:0/2938844374 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7b0009e080 msgr2=0x7f7b000c3100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:48.046 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.963 7f7b070a5700 1 --2- 172.21.2.120:0/2938844374 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7b0009e080 0x7f7b000c3100 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f7af0009940 tx=0x7f7af000bcf0).stop 2021-02-10T11:13:48.046 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.964 7f7b070a5700 1 -- 172.21.2.120:0/2938844374 shutdown_connections 2021-02-10T11:13:48.046 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.964 7f7b070a5700 1 --2- 172.21.2.120:0/2938844374 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7ae4018730 0x7f7ae401ac10 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:48.046 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.964 7f7b070a5700 1 --2- 172.21.2.120:0/2938844374 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7b0009e080 0x7f7b000c3100 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:48.046 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.964 7f7b070a5700 1 --2- 172.21.2.120:0/2938844374 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7b0009e480 0x7f7b000c3720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:48.046 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.964 7f7b070a5700 1 --2- 172.21.2.120:0/2938844374 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7b001a49b0 0x7f7b0008a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:48.046 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.964 7f7b070a5700 1 -- 172.21.2.120:0/2938844374 >> 172.21.2.120:0/2938844374 conn(0x7f7b000a1690 msgr2=0x7f7b000a94c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:13:48.046 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.964 7f7b070a5700 1 -- 172.21.2.120:0/2938844374 shutdown_connections 2021-02-10T11:13:48.047 INFO:teuthology.orchestra.run.gibba020.stderr:2021-02-10 11:13:47.964 7f7b070a5700 1 -- 172.21.2.120:0/2938844374 wait complete. 2021-02-10T11:13:48.047 INFO:teuthology.orchestra.run.gibba020.stdout:7 2021-02-10T11:13:48.047 INFO:tasks.ceph.osd.0:Restarting daemon 2021-02-10T11:13:48.047 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2021-02-10T11:13:48.054 INFO:tasks.ceph.osd.0:Started 2021-02-10T11:13:48.054 INFO:tasks.ceph.osd.1:Restarting daemon 2021-02-10T11:13:48.055 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2021-02-10T11:13:48.063 INFO:tasks.ceph.osd.1:Started 2021-02-10T11:13:48.064 INFO:tasks.ceph.osd.2:Restarting daemon 2021-02-10T11:13:48.064 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2021-02-10T11:13:48.073 INFO:tasks.ceph.osd.2:Started 2021-02-10T11:13:48.073 INFO:tasks.ceph.osd.3:Restarting daemon 2021-02-10T11:13:48.073 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2021-02-10T11:13:48.081 INFO:tasks.ceph.osd.3:Started 2021-02-10T11:13:48.081 INFO:tasks.ceph.osd.4:Restarting daemon 2021-02-10T11:13:48.081 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2021-02-10T11:13:48.086 INFO:tasks.ceph.osd.4:Started 2021-02-10T11:13:48.087 INFO:tasks.ceph.osd.5:Restarting daemon 2021-02-10T11:13:48.087 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2021-02-10T11:13:48.091 INFO:tasks.ceph.osd.5:Started 2021-02-10T11:13:48.091 INFO:tasks.ceph.osd.6:Restarting daemon 2021-02-10T11:13:48.091 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2021-02-10T11:13:48.097 INFO:tasks.ceph.osd.6:Started 2021-02-10T11:13:48.097 INFO:tasks.ceph.osd.7:Restarting daemon 2021-02-10T11:13:48.097 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2021-02-10T11:13:48.103 INFO:tasks.ceph.osd.7:Started 2021-02-10T11:13:48.103 INFO:tasks.ceph:Waiting for OSDs to come up 2021-02-10T11:13:48.913 DEBUG:teuthology.orchestra.run.gibba012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2021-02-10T11:13:48.970 INFO:tasks.ceph.osd.1.gibba012.stderr:2021-02-10 11:13:48.972 7f62547d2a80 -1 Falling back to public interface 2021-02-10T11:13:48.971 INFO:tasks.ceph.osd.1.gibba012.stderr:2021-02-10 11:13:48.972 7f624d8f5700 -1 PosixStack listen unable to listen on v1:0.0.0.0:6801/0: (98) Address already in use 2021-02-10T11:13:48.971 INFO:tasks.ceph.osd.2.gibba012.stderr:2021-02-10 11:13:48.972 7f2eba034a80 -1 Falling back to public interface 2021-02-10T11:13:48.985 INFO:tasks.ceph.osd.0.gibba012.stderr:2021-02-10 11:13:48.986 7fd7a91c6a80 -1 Falling back to public interface 2021-02-10T11:13:48.985 INFO:tasks.ceph.osd.0.gibba012.stderr:2021-02-10 11:13:48.987 7fd7a22e9700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6816/0: (98) Address already in use 2021-02-10T11:13:48.985 INFO:tasks.ceph.osd.3.gibba012.stderr:2021-02-10 11:13:48.986 7f187b3e9a80 -1 Falling back to public interface 2021-02-10T11:13:49.007 INFO:tasks.ceph.osd.5.gibba020.stderr:2021-02-10 11:13:49.009 7fd8de976a80 -1 Falling back to public interface 2021-02-10T11:13:49.007 INFO:tasks.ceph.osd.5.gibba020.stderr:2021-02-10 11:13:49.009 7fd8d7a99700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6803/0: (98) Address already in use 2021-02-10T11:13:49.008 INFO:tasks.ceph.osd.7.gibba020.stderr:2021-02-10 11:13:49.009 7f2da56bba80 -1 Falling back to public interface 2021-02-10T11:13:49.008 INFO:tasks.ceph.osd.4.gibba020.stderr:2021-02-10 11:13:49.009 7f7819588a80 -1 Falling back to public interface 2021-02-10T11:13:49.009 INFO:tasks.ceph.osd.6.gibba020.stderr:2021-02-10 11:13:49.011 7fa2cbec2a80 -1 Falling back to public interface 2021-02-10T11:13:49.336 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:49.317 7feddfc71700 1 Processor -- start 2021-02-10T11:13:49.336 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:49.317 7feddfc71700 1 -- start start 2021-02-10T11:13:49.337 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:49.318 7feddfc71700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fedd80d17f0 0x7fedd81b31e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:49.337 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:49.318 7feddfc71700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fedd80cff00 0x7fedd80ef620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:49.337 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:49.318 7feddfc71700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedd80f0b20 con 0x7fedd80d1bf0 2021-02-10T11:13:49.337 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:49.318 7feddfc71700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fedd80e5eb0 con 0x7fedd80cff00 2021-02-10T11:13:49.337 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:49.318 7feddfc71700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedd81b3ff0 con 0x7fedd80d17f0 2021-02-10T11:13:49.337 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:49.318 7feddda8d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fedd80d17f0 0x7fedd81b31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:49.338 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:49.318 7fedde28e700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fedd80cff00 0x7fedd80ef620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:49.338 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:49.318 7fedde28e700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fedd80cff00 0x7fedd80ef620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3300/0 says I am v2:172.21.2.112:43986/0 (socket says 172.21.2.112:43986) 2021-02-10T11:13:49.338 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:49.318 7feddda8d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fedd80d17f0 0x7fedd81b31e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3301/0 says I am v2:172.21.2.112:41388/0 (socket says 172.21.2.112:41388) 2021-02-10T11:13:49.338 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:49.318 7feddd28c700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7fedd80d1bf0 0x7fedd81b3800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:34726/0 (socket says 172.21.2.112:34726) 2021-02-10T11:13:49.338 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:49.318 7feddda8d700 1 -- 172.21.2.112:0/712398362 learned_addr learned my addr 172.21.2.112:0/712398362 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:13:49.339 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:49.319 7fedde28e700 1 -- 172.21.2.112:0/712398362 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fedd80d17f0 msgr2=0x7fedd81b31e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:13:49.339 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:49.319 7fedde28e700 1 --2- 172.21.2.112:0/712398362 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fedd80d17f0 0x7fedd81b31e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:49.339 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:49.319 7fedde28e700 1 -- 172.21.2.112:0/712398362 >> v1:172.21.2.120:6789/0 conn(0x7fedd80d1bf0 legacy=0x7fedd81b3800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:49.339 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:49.319 7fedde28e700 1 -- 172.21.2.112:0/712398362 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedd80f20b0 con 0x7fedd80cff00 2021-02-10T11:13:49.339 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:49.319 7feddda8d700 1 --2- 172.21.2.112:0/712398362 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fedd80d17f0 0x7fedd81b31e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:13:49.339 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:49.319 7fedde28e700 1 --2- 172.21.2.112:0/712398362 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fedd80cff00 0x7fedd80ef620 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7fedd0006e50 tx=0x7fedd0009e70).ready entity=mon.1 client_cookie=533f24ace0067cff server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:13:49.339 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:49.319 7feddca8b700 1 -- 172.21.2.112:0/712398362 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fedd0006a20 con 0x7fedd80cff00 2021-02-10T11:13:49.339 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:49.319 7feddca8b700 1 -- 172.21.2.112:0/712398362 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedd0006790 con 0x7fedd80cff00 2021-02-10T11:13:49.339 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:49.319 7feddca8b700 1 -- 172.21.2.112:0/712398362 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fedd000b700 con 0x7fedd80cff00 2021-02-10T11:13:49.339 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:49.319 7feddfc71700 1 -- 172.21.2.112:0/712398362 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fedd80cff00 msgr2=0x7fedd80ef620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:49.340 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:49.319 7feddfc71700 1 --2- 172.21.2.112:0/712398362 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fedd80cff00 0x7fedd80ef620 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7fedd0006e50 tx=0x7fedd0009e70).stop 2021-02-10T11:13:49.340 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:49.320 7feddfc71700 1 -- 172.21.2.112:0/712398362 shutdown_connections 2021-02-10T11:13:49.340 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:49.320 7feddfc71700 1 --2- 172.21.2.112:0/712398362 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fedd80cff00 0x7fedd80ef620 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:49.340 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:49.320 7feddfc71700 1 --2- 172.21.2.112:0/712398362 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fedd80d17f0 0x7fedd81b31e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:49.340 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:49.320 7feddfc71700 1 -- 172.21.2.112:0/712398362 >> 172.21.2.112:0/712398362 conn(0x7fedd80fde40 msgr2=0x7fedd80ce390 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:13:49.340 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:49.320 7feddfc71700 1 -- 172.21.2.112:0/712398362 shutdown_connections 2021-02-10T11:13:49.340 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:49.320 7feddfc71700 1 -- 172.21.2.112:0/712398362 wait complete. 2021-02-10T11:13:49.340 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:49.335 7feddfc71700 1 Processor -- start 2021-02-10T11:13:49.340 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:49.335 7feddfc71700 1 -- start start 2021-02-10T11:13:49.340 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:49.336 7feddfc71700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fedd81a48f0 0x7fedd80aca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:49.340 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:49.336 7feddfc71700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fedd809a8d0 0x7fedd81039c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:49.341 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:49.336 7feddfc71700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fedd8103fe0 0x7fedd8104d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:49.341 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:49.336 7feddfc71700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedd80ad0f0 con 0x7fedd8103fe0 2021-02-10T11:13:49.341 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:49.336 7feddfc71700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedd809ae80 con 0x7fedd81a48f0 2021-02-10T11:13:49.341 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:49.336 7feddfc71700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedd81043e0 con 0x7fedd809a8d0 2021-02-10T11:13:49.341 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:49.336 7feddda8d700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fedd81a48f0 0x7fedd80aca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:49.341 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:49.336 7fedde28e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fedd8103fe0 0x7fedd8104d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:49.342 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:49.336 7feddd28c700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fedd809a8d0 0x7fedd81039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:49.342 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:49.336 7feddda8d700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fedd81a48f0 0x7fedd80aca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3300/0 says I am v2:172.21.2.112:43990/0 (socket says 172.21.2.112:43990) 2021-02-10T11:13:49.342 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:49.336 7feddd28c700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fedd809a8d0 0x7fedd81039c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3301/0 says I am v2:172.21.2.112:41394/0 (socket says 172.21.2.112:41394) 2021-02-10T11:13:49.342 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:49.336 7feddda8d700 1 -- 172.21.2.112:0/679876111 learned_addr learned my addr 172.21.2.112:0/679876111 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:13:49.342 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:49.336 7fedde28e700 1 -- 172.21.2.112:0/679876111 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fedd809a8d0 msgr2=0x7fedd81039c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:49.342 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:49.336 7fedde28e700 1 --2- 172.21.2.112:0/679876111 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fedd809a8d0 0x7fedd81039c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:49.342 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:49.336 7fedde28e700 1 -- 172.21.2.112:0/679876111 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fedd81a48f0 msgr2=0x7fedd80aca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:49.342 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:49.336 7fedde28e700 1 --2- 172.21.2.112:0/679876111 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fedd81a48f0 0x7fedd80aca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:49.342 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:49.336 7fedde28e700 1 -- 172.21.2.112:0/679876111 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedd8105390 con 0x7fedd8103fe0 2021-02-10T11:13:49.343 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:49.336 7feddda8d700 1 --2- 172.21.2.112:0/679876111 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fedd81a48f0 0x7fedd80aca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:13:49.343 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:49.337 7fedde28e700 1 --2- 172.21.2.112:0/679876111 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fedd8103fe0 0x7fedd8104d70 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7fedd000b860 tx=0x7fedd000bc00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:13:49.343 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:49.337 7fedca7fc700 1 -- 172.21.2.112:0/679876111 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fedd0001130 con 0x7fedd8103fe0 2021-02-10T11:13:49.343 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:49.337 7fedca7fc700 1 -- 172.21.2.112:0/679876111 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedd0001430 con 0x7fedd8103fe0 2021-02-10T11:13:49.343 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:49.337 7feddfc71700 1 -- 172.21.2.112:0/679876111 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedd8105390 con 0x7fedd8103fe0 2021-02-10T11:13:49.343 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:49.337 7fedca7fc700 1 -- 172.21.2.112:0/679876111 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fedd0001910 con 0x7fedd8103fe0 2021-02-10T11:13:49.343 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:49.337 7feddfc71700 1 -- 172.21.2.112:0/679876111 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedd8105390 con 0x7fedd8103fe0 2021-02-10T11:13:49.343 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:49.337 7fedca7fc700 1 -- 172.21.2.112:0/679876111 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fedd0001130 con 0x7fedd8103fe0 2021-02-10T11:13:49.343 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:49.337 7fedca7fc700 1 --2- 172.21.2.112:0/679876111 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fedc00186c0 0x7fedc001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:49.343 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:49.337 7fedca7fc700 1 -- 172.21.2.112:0/679876111 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1578+0+0 (secure 0 0 0) 0x7fedd0031b00 con 0x7fedd8103fe0 2021-02-10T11:13:49.343 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:49.338 7feddda8d700 1 --2- 172.21.2.112:0/679876111 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fedc00186c0 0x7fedc001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:49.344 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:49.338 7feddda8d700 1 --2- 172.21.2.112:0/679876111 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fedc00186c0 0x7fedc001abc0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:13:49.366 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:49.367 7feddfc71700 1 -- 172.21.2.112:0/679876111 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedd8103790 con 0x7fedd8103fe0 2021-02-10T11:13:49.369 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:49.370 7fedca7fc700 1 -- 172.21.2.112:0/679876111 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fedd0000df0 con 0x7fedd8103fe0 2021-02-10T11:13:49.497 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:49.499 7feddfc71700 1 -- 172.21.2.112:0/679876111 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fedd8103790 con 0x7fedd8103fe0 2021-02-10T11:13:49.498 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:49.499 7fedca7fc700 1 -- 172.21.2.112:0/679876111 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+5464 (secure 0 0 0) 0x7fedd0008fa0 con 0x7fedd8103fe0 2021-02-10T11:13:49.498 INFO:teuthology.misc.health.gibba012.stdout: 2021-02-10T11:13:49.498 INFO:teuthology.misc.health.gibba012.stdout:{"epoch":10,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:13:47.956945","last_up_change":"0.000000","last_in_change":"0.000000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":":/0","cluster_addr":":/0","heartbeat_back_addr":":/0","heartbeat_front_addr":":/0","state":["exists","new"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":":/0","cluster_addr":":/0","heartbeat_back_addr":":/0","heartbeat_front_addr":":/0","state":["exists","new"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":":/0","cluster_addr":":/0","heartbeat_back_addr":":/0","heartbeat_front_addr":":/0","state":["exists","new"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":":/0","cluster_addr":":/0","heartbeat_back_addr":":/0","heartbeat_front_addr":":/0","state":["exists","new"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":":/0","cluster_addr":":/0","heartbeat_back_addr":":/0","heartbeat_front_addr":":/0","state":["exists","new"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":":/0","cluster_addr":":/0","heartbeat_back_addr":":/0","heartbeat_front_addr":":/0","state":["exists","new"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":":/0","cluster_addr":":/0","heartbeat_back_addr":":/0","heartbeat_front_addr":":/0","state":["exists","new"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":":/0","cluster_addr":":/0","heartbeat_back_addr":":/0","heartbeat_front_addr":":/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:13:49.500 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:49.501 7feddfc71700 1 -- 172.21.2.112:0/679876111 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fedc00186c0 msgr2=0x7fedc001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:49.500 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:49.501 7feddfc71700 1 --2- 172.21.2.112:0/679876111 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fedc00186c0 0x7fedc001abc0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:49.501 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:49.501 7feddfc71700 1 -- 172.21.2.112:0/679876111 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fedd8103fe0 msgr2=0x7fedd8104d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:49.501 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:49.501 7feddfc71700 1 --2- 172.21.2.112:0/679876111 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fedd8103fe0 0x7fedd8104d70 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7fedd000b860 tx=0x7fedd000bc00).stop 2021-02-10T11:13:49.501 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:49.501 7feddfc71700 1 -- 172.21.2.112:0/679876111 shutdown_connections 2021-02-10T11:13:49.501 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:49.501 7feddfc71700 1 --2- 172.21.2.112:0/679876111 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fedc00186c0 0x7fedc001abc0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:49.501 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:49.501 7feddfc71700 1 --2- 172.21.2.112:0/679876111 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fedd8103fe0 0x7fedd8104d70 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:49.501 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:49.501 7feddfc71700 1 --2- 172.21.2.112:0/679876111 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fedd809a8d0 0x7fedd81039c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:49.501 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:49.501 7feddfc71700 1 --2- 172.21.2.112:0/679876111 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fedd81a48f0 0x7fedd80aca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:49.502 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:49.501 7feddfc71700 1 -- 172.21.2.112:0/679876111 >> 172.21.2.112:0/679876111 conn(0x7fedd80c40c0 msgr2=0x7fedd80c2ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:13:49.502 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:49.501 7feddfc71700 1 -- 172.21.2.112:0/679876111 shutdown_connections 2021-02-10T11:13:49.502 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:49.501 7feddfc71700 1 -- 172.21.2.112:0/679876111 wait complete. 2021-02-10T11:13:49.507 DEBUG:teuthology.misc:0 of 8 OSDs are up 2021-02-10T11:13:50.005 INFO:tasks.ceph.osd.2.gibba012.stderr:2021-02-10 11:13:50.007 7f2eba034a80 -1 osd.2 0 log_to_monitors {default=true} 2021-02-10T11:13:50.010 INFO:tasks.ceph.osd.1.gibba012.stderr:2021-02-10 11:13:50.011 7f62547d2a80 -1 osd.1 0 log_to_monitors {default=true} 2021-02-10T11:13:50.028 INFO:tasks.ceph.osd.0.gibba012.stderr:2021-02-10 11:13:50.030 7fd7a91c6a80 -1 osd.0 0 log_to_monitors {default=true} 2021-02-10T11:13:50.033 INFO:tasks.ceph.osd.3.gibba012.stderr:2021-02-10 11:13:50.035 7f187b3e9a80 -1 osd.3 0 log_to_monitors {default=true} 2021-02-10T11:13:50.043 INFO:tasks.ceph.osd.5.gibba020.stderr:2021-02-10 11:13:50.044 7fd8de976a80 -1 osd.5 0 log_to_monitors {default=true} 2021-02-10T11:13:50.044 INFO:tasks.ceph.osd.4.gibba020.stderr:2021-02-10 11:13:50.046 7f7819588a80 -1 osd.4 0 log_to_monitors {default=true} 2021-02-10T11:13:50.046 INFO:tasks.ceph.osd.7.gibba020.stderr:2021-02-10 11:13:50.048 7f2da56bba80 -1 osd.7 0 log_to_monitors {default=true} 2021-02-10T11:13:50.048 INFO:tasks.ceph.osd.6.gibba020.stderr:2021-02-10 11:13:50.050 7fa2cbec2a80 -1 osd.6 0 log_to_monitors {default=true} 2021-02-10T11:13:51.779 INFO:tasks.ceph.osd.0.gibba012.stderr:2021-02-10 11:13:51.779 7fd78f83f700 -1 osd.0 0 waiting for initial osdmap 2021-02-10T11:13:51.779 INFO:tasks.ceph.osd.3.gibba012.stderr:2021-02-10 11:13:51.780 7f1861a62700 -1 osd.3 0 waiting for initial osdmap 2021-02-10T11:13:51.779 INFO:tasks.ceph.osd.1.gibba012.stderr:2021-02-10 11:13:51.780 7f623ae4b700 -1 osd.1 0 waiting for initial osdmap 2021-02-10T11:13:51.780 INFO:tasks.ceph.osd.7.gibba020.stderr:2021-02-10 11:13:51.780 7f2d8bd34700 -1 osd.7 0 waiting for initial osdmap 2021-02-10T11:13:51.780 INFO:tasks.ceph.osd.5.gibba020.stderr:2021-02-10 11:13:51.780 7fd8c4fef700 -1 osd.5 0 waiting for initial osdmap 2021-02-10T11:13:51.780 INFO:tasks.ceph.osd.4.gibba020.stderr:2021-02-10 11:13:51.781 7f77ffc01700 -1 osd.4 0 waiting for initial osdmap 2021-02-10T11:13:51.780 INFO:tasks.ceph.osd.2.gibba012.stderr:2021-02-10 11:13:51.781 7f2ea06ad700 -1 osd.2 0 waiting for initial osdmap 2021-02-10T11:13:51.781 INFO:tasks.ceph.osd.6.gibba020.stderr:2021-02-10 11:13:51.783 7fa2b253b700 -1 osd.6 0 waiting for initial osdmap 2021-02-10T11:13:51.782 INFO:tasks.ceph.osd.7.gibba020.stderr:2021-02-10 11:13:51.784 7f2d97f52700 -1 osd.7 12 set_numa_affinity unable to identify public interface 'enp1s0f0' numa node: (0) Success 2021-02-10T11:13:51.783 INFO:tasks.ceph.osd.0.gibba012.stderr:2021-02-10 11:13:51.784 7fd79ba5d700 -1 osd.0 12 set_numa_affinity unable to identify public interface 'enp1s0f0' numa node: (0) Success 2021-02-10T11:13:51.783 INFO:tasks.ceph.osd.3.gibba012.stderr:2021-02-10 11:13:51.784 7f186dc80700 -1 osd.3 12 set_numa_affinity unable to identify public interface 'enp1s0f0' numa node: (0) Success 2021-02-10T11:13:51.783 INFO:tasks.ceph.osd.1.gibba012.stderr:2021-02-10 11:13:51.784 7f6247069700 -1 osd.1 12 set_numa_affinity unable to identify public interface 'enp1s0f0' numa node: (0) Success 2021-02-10T11:13:51.783 INFO:tasks.ceph.osd.5.gibba020.stderr:2021-02-10 11:13:51.784 7fd8d120d700 -1 osd.5 12 set_numa_affinity unable to identify public interface 'enp1s0f0' numa node: (0) Success 2021-02-10T11:13:51.783 INFO:tasks.ceph.osd.4.gibba020.stderr:2021-02-10 11:13:51.784 7f780be1f700 -1 osd.4 12 set_numa_affinity unable to identify public interface 'enp1s0f0' numa node: (0) Success 2021-02-10T11:13:51.784 INFO:tasks.ceph.osd.2.gibba012.stderr:2021-02-10 11:13:51.785 7f2eac8cb700 -1 osd.2 12 set_numa_affinity unable to identify public interface 'enp1s0f0' numa node: (0) Success 2021-02-10T11:13:51.785 INFO:tasks.ceph.osd.6.gibba020.stderr:2021-02-10 11:13:51.786 7fa2be759700 -1 osd.6 12 set_numa_affinity unable to identify public interface 'enp1s0f0' numa node: (0) Success 2021-02-10T11:13:56.313 DEBUG:teuthology.orchestra.run.gibba012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2021-02-10T11:13:56.731 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:56.715 7f47f712c700 1 Processor -- start 2021-02-10T11:13:56.731 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:56.715 7f47f712c700 1 -- start start 2021-02-10T11:13:56.731 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:56.716 7f47f712c700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f47f00e5070 0x7f47f01b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:56.731 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:56.716 7f47f712c700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f47f01b3e40 0x7f47f007d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:56.731 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:56.716 7f47f712c700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f47f00e5470 0x7f47f007dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:56.731 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:56.716 7f47f712c700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47f007ed80 con 0x7f47f00e5070 2021-02-10T11:13:56.731 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:56.716 7f47f712c700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47f007e2d0 con 0x7f47f01b3e40 2021-02-10T11:13:56.731 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:56.716 7f47f712c700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47f007e4f0 con 0x7f47f00e5470 2021-02-10T11:13:56.731 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:56.716 7f47f4f48700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f47f00e5070 0x7f47f01b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:56.732 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:56.716 7f47f4f48700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f47f00e5070 0x7f47f01b3820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:41934/0 (socket says 172.21.2.112:41934) 2021-02-10T11:13:56.732 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:56.716 7f47f4f48700 1 -- 172.21.2.112:0/2493462968 learned_addr learned my addr 172.21.2.112:0/2493462968 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:13:56.732 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:56.717 7f47f4f48700 1 -- 172.21.2.112:0/2493462968 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f47f00e5470 msgr2=0x7f47f007dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:13:56.732 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:56.717 7f47f4f48700 1 --2- 172.21.2.112:0/2493462968 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f47f00e5470 0x7f47f007dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:13:56.732 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:56.717 7f47f4f48700 1 -- 172.21.2.112:0/2493462968 >> v2:172.21.2.120:3300/0 conn(0x7f47f01b3e40 msgr2=0x7f47f007d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:13:56.732 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:56.717 7f47f4f48700 1 --2- 172.21.2.112:0/2493462968 >> v2:172.21.2.120:3300/0 conn(0x7f47f01b3e40 0x7f47f007d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:13:56.732 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:56.717 7f47f4f48700 1 -- 172.21.2.112:0/2493462968 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47f007e710 con 0x7f47f00e5070 2021-02-10T11:13:56.732 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:56.717 7f47f4f48700 1 --2- 172.21.2.112:0/2493462968 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f47f00e5070 0x7f47f01b3820 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f47d8006700 tx=0x7f47d8009b80).ready entity=mon.0 client_cookie=afee04f3a758ca89 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:13:56.733 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:56.717 7f47ef7fe700 1 -- 172.21.2.112:0/2493462968 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f47d800c0c0 con 0x7f47f00e5070 2021-02-10T11:13:56.733 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:56.717 7f47ef7fe700 1 -- 172.21.2.112:0/2493462968 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47d800c340 con 0x7f47f00e5070 2021-02-10T11:13:56.733 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:56.717 7f47ef7fe700 1 -- 172.21.2.112:0/2493462968 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f47d800c860 con 0x7f47f00e5070 2021-02-10T11:13:56.733 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:56.717 7f47f712c700 1 -- 172.21.2.112:0/2493462968 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f47f00e5070 msgr2=0x7f47f01b3820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:56.733 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:56.717 7f47f712c700 1 --2- 172.21.2.112:0/2493462968 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f47f00e5070 0x7f47f01b3820 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f47d8006700 tx=0x7f47d8009b80).stop 2021-02-10T11:13:56.733 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:56.718 7f47f712c700 1 -- 172.21.2.112:0/2493462968 shutdown_connections 2021-02-10T11:13:56.733 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:56.718 7f47f712c700 1 --2- 172.21.2.112:0/2493462968 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f47f00e5470 0x7f47f007dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:13:56.733 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:56.718 7f47f712c700 1 --2- 172.21.2.112:0/2493462968 >> v2:172.21.2.120:3300/0 conn(0x7f47f01b3e40 0x7f47f007d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:13:56.733 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:56.718 7f47f712c700 1 --2- 172.21.2.112:0/2493462968 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f47f00e5070 0x7f47f01b3820 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:56.733 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:56.718 7f47f712c700 1 -- 172.21.2.112:0/2493462968 >> 172.21.2.112:0/2493462968 conn(0x7f47f00ce690 msgr2=0x7f47f00e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:13:56.733 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:56.718 7f47f712c700 1 -- 172.21.2.112:0/2493462968 shutdown_connections 2021-02-10T11:13:56.733 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:56.718 7f47f712c700 1 -- 172.21.2.112:0/2493462968 wait complete. 2021-02-10T11:13:56.735 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:56.733 7f47f712c700 1 Processor -- start 2021-02-10T11:13:56.735 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:56.733 7f47f712c700 1 -- start start 2021-02-10T11:13:56.735 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:56.733 7f47f712c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f47f0092150 0x7f47f0092550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:56.735 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:56.733 7f47f712c700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f47f00ae2f0 0x7f47f00ae6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:56.736 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:56.733 7f47f712c700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f47f00887c0 0x7f47f0088bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:56.736 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:56.733 7f47f712c700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47f00882c0 con 0x7f47f0092150 2021-02-10T11:13:56.736 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:56.733 7f47f712c700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47f0091e90 con 0x7f47f00887c0 2021-02-10T11:13:56.736 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:56.733 7f47f712c700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47f01a4ab0 con 0x7f47f00ae2f0 2021-02-10T11:13:56.736 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:56.733 7f47f4f48700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f47f0092150 0x7f47f0092550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:56.736 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:56.733 7f47f4f48700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f47f0092150 0x7f47f0092550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:41940/0 (socket says 172.21.2.112:41940) 2021-02-10T11:13:56.736 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:56.733 7f47f4f48700 1 -- 172.21.2.112:0/2207154855 learned_addr learned my addr 172.21.2.112:0/2207154855 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:13:56.736 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:56.734 7f47f4f48700 1 -- 172.21.2.112:0/2207154855 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f47f00ae2f0 msgr2=0x7f47f00ae6f0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:13:56.736 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:56.734 7f47f4f48700 1 --2- 172.21.2.112:0/2207154855 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f47f00ae2f0 0x7f47f00ae6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:13:56.736 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:56.734 7f47f4f48700 1 -- 172.21.2.112:0/2207154855 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f47f00887c0 msgr2=0x7f47f0088bc0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:13:56.736 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:56.734 7f47f4f48700 1 --2- 172.21.2.112:0/2207154855 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f47f00887c0 0x7f47f0088bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:13:56.736 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:56.734 7f47f4f48700 1 -- 172.21.2.112:0/2207154855 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47f0090980 con 0x7f47f0092150 2021-02-10T11:13:56.737 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:56.734 7f47f4f48700 1 --2- 172.21.2.112:0/2207154855 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f47f0092150 0x7f47f0092550 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f47d800b5e0 tx=0x7f47d800b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:13:56.737 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:56.734 7f47ed7fa700 1 -- 172.21.2.112:0/2207154855 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f47d800bd70 con 0x7f47f0092150 2021-02-10T11:13:56.737 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:56.734 7f47ed7fa700 1 -- 172.21.2.112:0/2207154855 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47d8002030 con 0x7f47f0092150 2021-02-10T11:13:56.737 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:56.734 7f47ed7fa700 1 -- 172.21.2.112:0/2207154855 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f47d8002550 con 0x7f47f0092150 2021-02-10T11:13:56.737 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:56.734 7f47f712c700 1 -- 172.21.2.112:0/2207154855 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47f0090980 con 0x7f47f0092150 2021-02-10T11:13:56.737 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:56.735 7f47f712c700 1 -- 172.21.2.112:0/2207154855 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47f0090980 con 0x7f47f0092150 2021-02-10T11:13:56.737 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:56.735 7f47ed7fa700 1 -- 172.21.2.112:0/2207154855 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f47d8000c90 con 0x7f47f0092150 2021-02-10T11:13:56.737 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:56.735 7f47ed7fa700 1 --2- 172.21.2.112:0/2207154855 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f47d40186c0 0x7f47d401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:56.737 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:56.735 7f47ed7fa700 1 -- 172.21.2.112:0/2207154855 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 4400+0+0 (secure 0 0 0) 0x7f47d80320e0 con 0x7f47f0092150 2021-02-10T11:13:56.737 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:56.736 7f47effff700 1 --2- 172.21.2.112:0/2207154855 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f47d40186c0 0x7f47d401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:56.737 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:56.736 7f47effff700 1 --2- 172.21.2.112:0/2207154855 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f47d40186c0 0x7f47d401abc0 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:13:56.765 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:56.766 7f47f712c700 1 -- 172.21.2.112:0/2207154855 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47f01a4ab0 con 0x7f47f0092150 2021-02-10T11:13:56.767 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:56.769 7f47ed7fa700 1 -- 172.21.2.112:0/2207154855 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f47d8030d30 con 0x7f47f0092150 2021-02-10T11:13:56.891 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:56.892 7f47f712c700 1 -- 172.21.2.112:0/2207154855 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f47f01a4ab0 con 0x7f47f0092150 2021-02-10T11:13:56.891 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:56.893 7f47ed7fa700 1 -- 172.21.2.112:0/2207154855 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v13) v1 ==== 74+0+9772 (secure 0 0 0) 0x7f47d8000c90 con 0x7f47f0092150 2021-02-10T11:13:56.892 INFO:teuthology.misc.health.gibba012.stdout: 2021-02-10T11:13:56.892 INFO:teuthology.misc.health.gibba012.stdout:{"epoch":13,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:13:52.780316","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:13:56.893 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:56.894 7f47f712c700 1 -- 172.21.2.112:0/2207154855 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f47d40186c0 msgr2=0x7f47d401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:56.893 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:56.894 7f47f712c700 1 --2- 172.21.2.112:0/2207154855 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f47d40186c0 0x7f47d401abc0 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:56.893 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:56.894 7f47f712c700 1 -- 172.21.2.112:0/2207154855 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f47f0092150 msgr2=0x7f47f0092550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:56.893 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:56.894 7f47f712c700 1 --2- 172.21.2.112:0/2207154855 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f47f0092150 0x7f47f0092550 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f47d800b5e0 tx=0x7f47d800b980).stop 2021-02-10T11:13:56.894 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:56.895 7f47f712c700 1 -- 172.21.2.112:0/2207154855 shutdown_connections 2021-02-10T11:13:56.894 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:56.895 7f47f712c700 1 --2- 172.21.2.112:0/2207154855 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f47d40186c0 0x7f47d401abc0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:56.894 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:56.895 7f47f712c700 1 --2- 172.21.2.112:0/2207154855 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f47f00887c0 0x7f47f0088bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:13:56.894 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:56.895 7f47f712c700 1 --2- 172.21.2.112:0/2207154855 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f47f00ae2f0 0x7f47f00ae6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:13:56.894 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:56.895 7f47f712c700 1 --2- 172.21.2.112:0/2207154855 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f47f0092150 0x7f47f0092550 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:56.894 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:56.895 7f47f712c700 1 -- 172.21.2.112:0/2207154855 >> 172.21.2.112:0/2207154855 conn(0x7f47f0097110 msgr2=0x7f47f0193340 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:13:56.894 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:56.895 7f47f712c700 1 -- 172.21.2.112:0/2207154855 shutdown_connections 2021-02-10T11:13:56.894 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:13:56.895 7f47f712c700 1 -- 172.21.2.112:0/2207154855 wait complete. 2021-02-10T11:13:56.900 DEBUG:teuthology.misc:8 of 8 OSDs are up 2021-02-10T11:13:56.901 INFO:tasks.ceph:Creating RBD pool 2021-02-10T11:13:56.901 DEBUG:teuthology.orchestra.run.gibba012:> sudo ceph --cluster ceph osd pool create rbd 8 2021-02-10T11:13:57.327 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:57.311 7fa18a3fe700 1 Processor -- start 2021-02-10T11:13:57.327 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:57.311 7fa18a3fe700 1 -- start start 2021-02-10T11:13:57.327 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:57.312 7fa18a3fe700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fa1841b0470 0x7fa1841b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:57.327 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:57.312 7fa18a3fe700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa1840f20e0 0x7fa1840f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:57.327 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:57.312 7fa18a3fe700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1840f0b50 con 0x7fa1840ef610 2021-02-10T11:13:57.327 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:57.312 7fa18a3fe700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1840f0820 con 0x7fa1841b0470 2021-02-10T11:13:57.327 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:57.312 7fa18a3fe700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1840f24e0 con 0x7fa1840f20e0 2021-02-10T11:13:57.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:57.312 7fa1837fe700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7fa1840ef610 0x7fa1840efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:34968/0 (socket says 172.21.2.112:34968) 2021-02-10T11:13:57.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:57.312 7fa1837fe700 1 -- 172.21.2.112:0/482823825 learned_addr learned my addr 172.21.2.112:0/482823825 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:13:57.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:57.313 7fa188a1b700 1 --2- 172.21.2.112:0/482823825 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa1840f20e0 0x7fa1840f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:57.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:57.313 7fa183fff700 1 --2- 172.21.2.112:0/482823825 >> v2:172.21.2.120:3300/0 conn(0x7fa1841b0470 0x7fa1841b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:57.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:57.313 7fa182ffd700 1 -- 172.21.2.112:0/482823825 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3181759644 0 0) 0x7fa178001190 con 0x7fa1840ef610 2021-02-10T11:13:57.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:57.313 7fa182ffd700 1 -- 172.21.2.112:0/482823825 --> v1:172.21.2.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa16c002fe0 con 0x7fa1840ef610 2021-02-10T11:13:57.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:57.313 7fa188a1b700 1 -- 172.21.2.112:0/482823825 >> v2:172.21.2.120:3300/0 conn(0x7fa1841b0470 msgr2=0x7fa1841b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:13:57.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:57.313 7fa188a1b700 1 --2- 172.21.2.112:0/482823825 >> v2:172.21.2.120:3300/0 conn(0x7fa1841b0470 0x7fa1841b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:57.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:57.313 7fa188a1b700 1 -- 172.21.2.112:0/482823825 >> v1:172.21.2.120:6789/0 conn(0x7fa1840ef610 legacy=0x7fa1840efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:57.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:57.313 7fa188a1b700 1 -- 172.21.2.112:0/482823825 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1840e52e0 con 0x7fa1840f20e0 2021-02-10T11:13:57.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:57.313 7fa183fff700 1 --2- 172.21.2.112:0/482823825 >> v2:172.21.2.120:3300/0 conn(0x7fa1841b0470 0x7fa1841b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:13:57.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:57.313 7fa188a1b700 1 --2- 172.21.2.112:0/482823825 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa1840f20e0 0x7fa1840f0200 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7fa17c006e50 tx=0x7fa17c009e70).ready entity=mon.2 client_cookie=5c8fee9c68b67448 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:13:57.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:57.313 7fa182ffd700 1 -- 172.21.2.112:0/482823825 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa17c00b510 con 0x7fa1840f20e0 2021-02-10T11:13:57.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:57.313 7fa182ffd700 1 -- 172.21.2.112:0/482823825 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa17c006790 con 0x7fa1840f20e0 2021-02-10T11:13:57.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:57.314 7fa18a3fe700 1 -- 172.21.2.112:0/482823825 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa1840f20e0 msgr2=0x7fa1840f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:57.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:57.314 7fa18a3fe700 1 --2- 172.21.2.112:0/482823825 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa1840f20e0 0x7fa1840f0200 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7fa17c006e50 tx=0x7fa17c009e70).stop 2021-02-10T11:13:57.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:57.313 7fa182ffd700 1 -- 172.21.2.112:0/482823825 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa17c00b940 con 0x7fa1840f20e0 2021-02-10T11:13:57.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:57.314 7fa18a3fe700 1 -- 172.21.2.112:0/482823825 shutdown_connections 2021-02-10T11:13:57.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:57.314 7fa18a3fe700 1 --2- 172.21.2.112:0/482823825 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa1840f20e0 0x7fa1840f0200 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:57.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:57.314 7fa18a3fe700 1 --2- 172.21.2.112:0/482823825 >> v2:172.21.2.120:3300/0 conn(0x7fa1841b0470 0x7fa1841b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:57.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:57.314 7fa18a3fe700 1 -- 172.21.2.112:0/482823825 >> 172.21.2.112:0/482823825 conn(0x7fa1840ce3e0 msgr2=0x7fa1840e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:13:57.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:57.314 7fa18a3fe700 1 -- 172.21.2.112:0/482823825 shutdown_connections 2021-02-10T11:13:57.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:57.314 7fa18a3fe700 1 -- 172.21.2.112:0/482823825 wait complete. 2021-02-10T11:13:57.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:57.329 7fa18a3fe700 1 Processor -- start 2021-02-10T11:13:57.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:57.329 7fa18a3fe700 1 -- start start 2021-02-10T11:13:57.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:57.329 7fa18a3fe700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa1840a0420 0x7fa1840a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:57.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:57.329 7fa18a3fe700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa184189d00 0x7fa18418a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:57.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:57.329 7fa18a3fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa18418a720 0x7fa1840b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:57.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:57.329 7fa18a3fe700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa184189780 con 0x7fa18418a720 2021-02-10T11:13:57.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:57.329 7fa18a3fe700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa18418ab20 con 0x7fa1840a0420 2021-02-10T11:13:57.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:57.329 7fa18a3fe700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1840a1140 con 0x7fa184189d00 2021-02-10T11:13:57.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:57.330 7fa188a1b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa18418a720 0x7fa1840b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:57.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:57.330 7fa188a1b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa18418a720 0x7fa1840b7be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:41956/0 (socket says 172.21.2.112:41956) 2021-02-10T11:13:57.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:57.330 7fa188a1b700 1 -- 172.21.2.112:0/2880532980 learned_addr learned my addr 172.21.2.112:0/2880532980 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:13:57.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:57.330 7fa183fff700 1 --2- 172.21.2.112:0/2880532980 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa1840a0420 0x7fa1840a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:57.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:57.330 7fa1837fe700 1 --2- 172.21.2.112:0/2880532980 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa184189d00 0x7fa18418a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:57.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:57.330 7fa188a1b700 1 -- 172.21.2.112:0/2880532980 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa184189d00 msgr2=0x7fa18418a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:57.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:57.330 7fa188a1b700 1 --2- 172.21.2.112:0/2880532980 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa184189d00 0x7fa18418a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:57.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:57.330 7fa188a1b700 1 -- 172.21.2.112:0/2880532980 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa1840a0420 msgr2=0x7fa1840a0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:57.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:57.330 7fa188a1b700 1 --2- 172.21.2.112:0/2880532980 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa1840a0420 0x7fa1840a0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:57.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:57.330 7fa188a1b700 1 -- 172.21.2.112:0/2880532980 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1840b8200 con 0x7fa18418a720 2021-02-10T11:13:57.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:57.330 7fa188a1b700 1 --2- 172.21.2.112:0/2880532980 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa18418a720 0x7fa1840b7be0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7fa17c009eb0 tx=0x7fa17c00bd60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:13:57.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:57.330 7fa180ff9700 1 -- 172.21.2.112:0/2880532980 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa17c009650 con 0x7fa18418a720 2021-02-10T11:13:57.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:57.331 7fa180ff9700 1 -- 172.21.2.112:0/2880532980 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa17c001070 con 0x7fa18418a720 2021-02-10T11:13:57.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:57.331 7fa180ff9700 1 -- 172.21.2.112:0/2880532980 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa17c001500 con 0x7fa18418a720 2021-02-10T11:13:57.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:57.331 7fa18a3fe700 1 -- 172.21.2.112:0/2880532980 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1840b8200 con 0x7fa18418a720 2021-02-10T11:13:57.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:57.331 7fa18a3fe700 1 -- 172.21.2.112:0/2880532980 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1840b8790 con 0x7fa18418a720 2021-02-10T11:13:57.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:57.331 7fa180ff9700 1 -- 172.21.2.112:0/2880532980 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa17c0056b0 con 0x7fa18418a720 2021-02-10T11:13:57.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:57.331 7fa180ff9700 1 --2- 172.21.2.112:0/2880532980 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa1640186c0 0x7fa16401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:57.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:57.331 7fa180ff9700 1 -- 172.21.2.112:0/2880532980 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 4400+0+0 (secure 0 0 0) 0x7fa17c032300 con 0x7fa18418a720 2021-02-10T11:13:57.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:57.332 7fa183fff700 1 --2- 172.21.2.112:0/2880532980 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa1640186c0 0x7fa16401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:57.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:57.332 7fa183fff700 1 --2- 172.21.2.112:0/2880532980 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa1640186c0 0x7fa16401abc0 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:13:57.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:57.362 7fa18a3fe700 1 -- 172.21.2.112:0/2880532980 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa184105ae0 con 0x7fa18418a720 2021-02-10T11:13:57.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:57.364 7fa180ff9700 1 -- 172.21.2.112:0/2880532980 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa17c02d750 con 0x7fa18418a720 2021-02-10T11:13:57.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:57.491 7fa18a3fe700 1 -- 172.21.2.112:0/2880532980 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "rbd"} v 0) v1 -- 0x7fa1840f0b50 con 0x7fa18418a720 2021-02-10T11:13:57.952 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:57.954 7fa180ff9700 1 -- 172.21.2.112:0/2880532980 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "rbd"}]=0 pool 'rbd' created v14) v1 ==== 109+0+0 (secure 0 0 0) 0x7fa17c02d750 con 0x7fa18418a720 2021-02-10T11:13:57.956 INFO:teuthology.orchestra.run.gibba012.stderr:pool 'rbd' created 2021-02-10T11:13:57.959 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:57.960 7fa18a3fe700 1 -- 172.21.2.112:0/2880532980 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa1640186c0 msgr2=0x7fa16401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:57.959 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:57.960 7fa18a3fe700 1 --2- 172.21.2.112:0/2880532980 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa1640186c0 0x7fa16401abc0 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:57.959 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:57.960 7fa18a3fe700 1 -- 172.21.2.112:0/2880532980 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa18418a720 msgr2=0x7fa1840b7be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:57.959 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:57.960 7fa18a3fe700 1 --2- 172.21.2.112:0/2880532980 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa18418a720 0x7fa1840b7be0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7fa17c009eb0 tx=0x7fa17c00bd60).stop 2021-02-10T11:13:57.959 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:57.960 7fa18a3fe700 1 -- 172.21.2.112:0/2880532980 shutdown_connections 2021-02-10T11:13:57.960 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:57.960 7fa18a3fe700 1 --2- 172.21.2.112:0/2880532980 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa1640186c0 0x7fa16401abc0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:57.960 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:57.960 7fa18a3fe700 1 --2- 172.21.2.112:0/2880532980 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa18418a720 0x7fa1840b7be0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:57.960 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:57.960 7fa18a3fe700 1 --2- 172.21.2.112:0/2880532980 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa184189d00 0x7fa18418a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:57.960 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:57.960 7fa18a3fe700 1 --2- 172.21.2.112:0/2880532980 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa1840a0420 0x7fa1840a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:57.960 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:57.960 7fa18a3fe700 1 -- 172.21.2.112:0/2880532980 >> 172.21.2.112:0/2880532980 conn(0x7fa184094e10 msgr2=0x7fa1840a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:13:57.960 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:57.960 7fa18a3fe700 1 -- 172.21.2.112:0/2880532980 shutdown_connections 2021-02-10T11:13:57.960 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:57.960 7fa18a3fe700 1 -- 172.21.2.112:0/2880532980 wait complete. 2021-02-10T11:13:57.967 DEBUG:teuthology.orchestra.run.gibba012:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2021-02-10T11:13:58.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:58.377 7f5a77f2c700 1 Processor -- start 2021-02-10T11:13:58.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:58.377 7f5a77f2c700 1 -- start start 2021-02-10T11:13:58.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:58.378 7f5a77f2c700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f5a700d0330 0x7f5a700ef630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:58.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:58.378 7f5a77f2c700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5a700d1c60 0x7f5a700efc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:58.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:58.378 7f5a77f2c700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a700f0b00 con 0x7f5a700d1860 2021-02-10T11:13:58.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:58.378 7f5a77f2c700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a700e5ec0 con 0x7f5a700d0330 2021-02-10T11:13:58.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:58.378 7f5a77f2c700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a701b0bd0 con 0x7f5a700d1c60 2021-02-10T11:13:58.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:58.378 7f5a75547700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f5a700d0330 0x7f5a700ef630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:58.393 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:58.378 7f5a75d48700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f5a700d1860 0x7f5a701b03e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:41422/0 (socket says 172.21.2.112:41422) 2021-02-10T11:13:58.393 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:58.378 7f5a75d48700 1 -- 172.21.2.112:0/4257980574 learned_addr learned my addr 172.21.2.112:0/4257980574 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:13:58.393 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:58.378 7f5a75547700 1 -- 172.21.2.112:0/4257980574 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5a700d1c60 msgr2=0x7f5a700efc50 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:13:58.393 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:58.378 7f5a75547700 1 --2- 172.21.2.112:0/4257980574 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5a700d1c60 0x7f5a700efc50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:13:58.393 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:58.378 7f5a75547700 1 -- 172.21.2.112:0/4257980574 >> v1:172.21.2.112:6789/0 conn(0x7f5a700d1860 legacy=0x7f5a701b03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:58.393 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:58.378 7f5a75547700 1 -- 172.21.2.112:0/4257980574 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a700f2090 con 0x7f5a700d0330 2021-02-10T11:13:58.393 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:58.378 7f5a75547700 1 --2- 172.21.2.112:0/4257980574 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5a700d0330 0x7f5a700ef630 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7f5a6c006650 tx=0x7f5a6c009a00).ready entity=mon.0 client_cookie=49d78e9826485247 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:13:58.393 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:58.379 7f5a74d46700 1 -- 172.21.2.112:0/4257980574 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5a6c009df0 con 0x7f5a700d0330 2021-02-10T11:13:58.393 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:58.379 7f5a74d46700 1 -- 172.21.2.112:0/4257980574 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a6c0009b0 con 0x7f5a700d0330 2021-02-10T11:13:58.393 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:58.379 7f5a74d46700 1 -- 172.21.2.112:0/4257980574 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5a6c009df0 con 0x7f5a700d0330 2021-02-10T11:13:58.393 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:58.379 7f5a77f2c700 1 -- 172.21.2.112:0/4257980574 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5a700d0330 msgr2=0x7f5a700ef630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:58.393 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:58.379 7f5a77f2c700 1 --2- 172.21.2.112:0/4257980574 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5a700d0330 0x7f5a700ef630 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7f5a6c006650 tx=0x7f5a6c009a00).stop 2021-02-10T11:13:58.393 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:58.379 7f5a77f2c700 1 -- 172.21.2.112:0/4257980574 shutdown_connections 2021-02-10T11:13:58.394 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:58.379 7f5a77f2c700 1 --2- 172.21.2.112:0/4257980574 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5a700d1c60 0x7f5a700efc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:13:58.394 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:58.379 7f5a77f2c700 1 --2- 172.21.2.112:0/4257980574 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5a700d0330 0x7f5a700ef630 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:58.394 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:58.379 7f5a77f2c700 1 -- 172.21.2.112:0/4257980574 >> 172.21.2.112:0/4257980574 conn(0x7f5a700ce440 msgr2=0x7f5a700d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:13:58.394 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:58.379 7f5a77f2c700 1 -- 172.21.2.112:0/4257980574 shutdown_connections 2021-02-10T11:13:58.394 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:58.379 7f5a77f2c700 1 -- 172.21.2.112:0/4257980574 wait complete. 2021-02-10T11:13:58.395 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:58.394 7f5a77f2c700 1 Processor -- start 2021-02-10T11:13:58.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:58.394 7f5a77f2c700 1 -- start start 2021-02-10T11:13:58.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:58.394 7f5a77f2c700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5a7019edd0 0x7f5a7019f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:58.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:58.394 7f5a77f2c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5a701aca30 0x7f5a701ace30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:58.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:58.394 7f5a77f2c700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5a7019d3b0 0x7f5a7019d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:58.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:58.394 7f5a77f2c700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a701ad450 con 0x7f5a701aca30 2021-02-10T11:13:58.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:58.394 7f5a77f2c700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a7019fc90 con 0x7f5a7019edd0 2021-02-10T11:13:58.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:58.394 7f5a77f2c700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a7019ddd0 con 0x7f5a7019d3b0 2021-02-10T11:13:58.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:58.394 7f5a75547700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5a701aca30 0x7f5a701ace30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:58.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:58.394 7f5a75547700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5a701aca30 0x7f5a701ace30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:41968/0 (socket says 172.21.2.112:41968) 2021-02-10T11:13:58.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:58.394 7f5a75547700 1 -- 172.21.2.112:0/441960679 learned_addr learned my addr 172.21.2.112:0/441960679 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:13:58.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:58.394 7f5a75547700 1 -- 172.21.2.112:0/441960679 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5a7019d3b0 msgr2=0x7f5a7019d7b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:13:58.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:58.394 7f5a75547700 1 --2- 172.21.2.112:0/441960679 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5a7019d3b0 0x7f5a7019d7b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:13:58.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:58.394 7f5a75547700 1 -- 172.21.2.112:0/441960679 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5a7019edd0 msgr2=0x7f5a7019f1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:58.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:58.394 7f5a75547700 1 --2- 172.21.2.112:0/441960679 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5a7019edd0 0x7f5a7019f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:13:58.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:58.394 7f5a75547700 1 -- 172.21.2.112:0/441960679 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a7019dff0 con 0x7f5a701aca30 2021-02-10T11:13:58.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:58.395 7f5a75547700 1 --2- 172.21.2.112:0/441960679 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5a701aca30 0x7f5a701ace30 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7f5a6c00b580 tx=0x7f5a6c00b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:13:58.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:58.395 7f5a627fc700 1 -- 172.21.2.112:0/441960679 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5a6c000c90 con 0x7f5a701aca30 2021-02-10T11:13:58.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:58.395 7f5a627fc700 1 -- 172.21.2.112:0/441960679 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a6c002030 con 0x7f5a701aca30 2021-02-10T11:13:58.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:58.395 7f5a77f2c700 1 -- 172.21.2.112:0/441960679 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a7019dff0 con 0x7f5a701aca30 2021-02-10T11:13:58.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:58.395 7f5a627fc700 1 -- 172.21.2.112:0/441960679 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5a6c002490 con 0x7f5a701aca30 2021-02-10T11:13:58.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:58.395 7f5a77f2c700 1 -- 172.21.2.112:0/441960679 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a7019dff0 con 0x7f5a701aca30 2021-02-10T11:13:58.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:58.396 7f5a627fc700 1 -- 172.21.2.112:0/441960679 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5a6c000c90 con 0x7f5a701aca30 2021-02-10T11:13:58.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:58.396 7f5a627fc700 1 --2- 172.21.2.112:0/441960679 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5a58018730 0x7f5a5801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:58.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:58.396 7f5a627fc700 1 -- 172.21.2.112:0/441960679 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 4742+0+0 (secure 0 0 0) 0x7f5a6c032510 con 0x7f5a701aca30 2021-02-10T11:13:58.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:58.396 7f5a75d48700 1 --2- 172.21.2.112:0/441960679 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5a58018730 0x7f5a5801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:58.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:58.397 7f5a75d48700 1 --2- 172.21.2.112:0/441960679 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5a58018730 0x7f5a5801abc0 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:13:58.426 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:58.428 7f5a77f2c700 1 -- 172.21.2.112:0/441960679 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a7019e8a0 con 0x7f5a701aca30 2021-02-10T11:13:58.429 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:58.431 7f5a627fc700 1 -- 172.21.2.112:0/441960679 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5a6c000940 con 0x7f5a701aca30 2021-02-10T11:13:58.554 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:58.555 7f5a77f2c700 1 -- 172.21.2.112:0/441960679 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd pool application enable", "app": "rbd", "pool": "rbd", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f5a7019e8a0 con 0x7f5a701aca30 2021-02-10T11:13:58.957 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:58.957 7f5a627fc700 1 -- 172.21.2.112:0/441960679 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "app": "rbd", "pool": "rbd", "yes_i_really_mean_it": true}]=0 enabled application 'rbd' on pool 'rbd' v15) v1 ==== 173+0+0 (secure 0 0 0) 0x7f5a6c000940 con 0x7f5a701aca30 2021-02-10T11:13:58.970 INFO:teuthology.orchestra.run.gibba012.stderr:enabled application 'rbd' on pool 'rbd' 2021-02-10T11:13:58.972 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:58.973 7f5a77f2c700 1 -- 172.21.2.112:0/441960679 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5a58018730 msgr2=0x7f5a5801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:58.972 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:58.973 7f5a77f2c700 1 --2- 172.21.2.112:0/441960679 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5a58018730 0x7f5a5801abc0 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:58.972 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:58.973 7f5a77f2c700 1 -- 172.21.2.112:0/441960679 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5a701aca30 msgr2=0x7f5a701ace30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:58.973 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:58.973 7f5a77f2c700 1 --2- 172.21.2.112:0/441960679 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5a701aca30 0x7f5a701ace30 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7f5a6c00b580 tx=0x7f5a6c00b920).stop 2021-02-10T11:13:58.973 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:58.973 7f5a77f2c700 1 -- 172.21.2.112:0/441960679 shutdown_connections 2021-02-10T11:13:58.973 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:58.973 7f5a77f2c700 1 --2- 172.21.2.112:0/441960679 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5a58018730 0x7f5a5801abc0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:58.973 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:58.973 7f5a77f2c700 1 --2- 172.21.2.112:0/441960679 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5a7019d3b0 0x7f5a7019d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:13:58.973 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:58.973 7f5a77f2c700 1 --2- 172.21.2.112:0/441960679 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5a701aca30 0x7f5a701ace30 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:58.973 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:58.973 7f5a77f2c700 1 --2- 172.21.2.112:0/441960679 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5a7019edd0 0x7f5a7019f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:13:58.973 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:58.973 7f5a77f2c700 1 -- 172.21.2.112:0/441960679 >> 172.21.2.112:0/441960679 conn(0x7f5a7019f870 msgr2=0x7f5a70094cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:13:58.973 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:58.973 7f5a77f2c700 1 -- 172.21.2.112:0/441960679 shutdown_connections 2021-02-10T11:13:58.973 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:58.973 7f5a77f2c700 1 -- 172.21.2.112:0/441960679 wait complete. 2021-02-10T11:13:58.980 INFO:tasks.ceph:Setting up CephFS filesystem... 2021-02-10T11:13:58.981 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:13:59.407 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:59.391 7f8c0eaa3700 1 Processor -- start 2021-02-10T11:13:59.407 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:59.391 7f8c0eaa3700 1 -- start start 2021-02-10T11:13:59.407 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:59.391 7f8c0eaa3700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8c080e5070 0x7f8c081b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:59.407 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:59.391 7f8c0eaa3700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f8c081b3e40 0x7f8c08080310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:59.407 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:59.391 7f8c0eaa3700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c0807ed80 con 0x7f8c080e5470 2021-02-10T11:13:59.407 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:59.391 7f8c0eaa3700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c080d1790 con 0x7f8c081b3e40 2021-02-10T11:13:59.407 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:59.391 7f8c0eaa3700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c080d1980 con 0x7f8c080e5070 2021-02-10T11:13:59.407 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:59.391 7f8c0d0c0700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f8c081b3e40 0x7f8c08080310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:59.407 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:59.391 7f8c0d0c0700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f8c081b3e40 0x7f8c08080310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:41978/0 (socket says 172.21.2.112:41978) 2021-02-10T11:13:59.407 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:59.391 7f8c0d0c0700 1 -- 172.21.2.112:0/3013004284 learned_addr learned my addr 172.21.2.112:0/3013004284 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:13:59.408 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:59.392 7f8c0d0c0700 1 -- 172.21.2.112:0/3013004284 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8c080e5070 msgr2=0x7f8c081b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:13:59.408 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:59.392 7f8c0d0c0700 1 --2- 172.21.2.112:0/3013004284 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8c080e5070 0x7f8c081b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:13:59.408 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:59.392 7f8c0d0c0700 1 -- 172.21.2.112:0/3013004284 >> v1:172.21.2.120:6789/0 conn(0x7f8c080e5470 legacy=0x7f8c0807d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:13:59.408 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:59.392 7f8c0d0c0700 1 -- 172.21.2.112:0/3013004284 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c080d1ba0 con 0x7f8c081b3e40 2021-02-10T11:13:59.408 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:59.392 7f8c0d0c0700 1 --2- 172.21.2.112:0/3013004284 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8c081b3e40 0x7f8c08080310 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7f8c00006f00 tx=0x7f8c00009f10).ready entity=mon.0 client_cookie=4b1e431ab31748cf server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:13:59.408 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:59.392 7f8c077fe700 1 -- 172.21.2.112:0/3013004284 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8c0000b3b0 con 0x7f8c081b3e40 2021-02-10T11:13:59.408 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:59.392 7f8c077fe700 1 -- 172.21.2.112:0/3013004284 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c0000b730 con 0x7f8c081b3e40 2021-02-10T11:13:59.408 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:59.392 7f8c077fe700 1 -- 172.21.2.112:0/3013004284 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8c0000bc50 con 0x7f8c081b3e40 2021-02-10T11:13:59.408 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:59.392 7f8c0eaa3700 1 -- 172.21.2.112:0/3013004284 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8c081b3e40 msgr2=0x7f8c08080310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:59.408 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:59.392 7f8c0eaa3700 1 --2- 172.21.2.112:0/3013004284 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8c081b3e40 0x7f8c08080310 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7f8c00006f00 tx=0x7f8c00009f10).stop 2021-02-10T11:13:59.408 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:59.393 7f8c0eaa3700 1 -- 172.21.2.112:0/3013004284 shutdown_connections 2021-02-10T11:13:59.408 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:59.393 7f8c0eaa3700 1 --2- 172.21.2.112:0/3013004284 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8c081b3e40 0x7f8c08080310 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:59.409 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:59.393 7f8c0eaa3700 1 --2- 172.21.2.112:0/3013004284 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8c080e5070 0x7f8c081b3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:13:59.409 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:59.393 7f8c0eaa3700 1 -- 172.21.2.112:0/3013004284 >> 172.21.2.112:0/3013004284 conn(0x7f8c080ce690 msgr2=0x7f8c080e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:13:59.409 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:59.393 7f8c0eaa3700 1 -- 172.21.2.112:0/3013004284 shutdown_connections 2021-02-10T11:13:59.409 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:59.393 7f8c0eaa3700 1 -- 172.21.2.112:0/3013004284 wait complete. 2021-02-10T11:13:59.410 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:59.407 7f8c0eaa3700 1 Processor -- start 2021-02-10T11:13:59.410 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:59.408 7f8c0eaa3700 1 -- start start 2021-02-10T11:13:59.410 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:59.408 7f8c0eaa3700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8c081aa290 0x7f8c081aa690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:59.410 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:59.408 7f8c0eaa3700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8c080b7270 0x7f8c080b7670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:59.411 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:59.408 7f8c0eaa3700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8c080b7c90 0x7f8c08084e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:59.411 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:59.408 7f8c0eaa3700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c080b8090 con 0x7f8c080b7c90 2021-02-10T11:13:59.411 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:59.408 7f8c0eaa3700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c081a9df0 con 0x7f8c080b7270 2021-02-10T11:13:59.411 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:59.408 7f8c0eaa3700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c0809cc30 con 0x7f8c081aa290 2021-02-10T11:13:59.411 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:59.408 7f8c0d0c0700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8c080b7c90 0x7f8c08084e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:59.411 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:59.408 7f8c0d0c0700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8c080b7c90 0x7f8c08084e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:41982/0 (socket says 172.21.2.112:41982) 2021-02-10T11:13:59.411 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:59.408 7f8c0d0c0700 1 -- 172.21.2.112:0/1545268098 learned_addr learned my addr 172.21.2.112:0/1545268098 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:13:59.411 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:59.408 7f8c0d0c0700 1 -- 172.21.2.112:0/1545268098 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8c081aa290 msgr2=0x7f8c081aa690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:59.411 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:59.408 7f8c0d0c0700 1 --2- 172.21.2.112:0/1545268098 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8c081aa290 0x7f8c081aa690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:13:59.411 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:59.408 7f8c0d0c0700 1 -- 172.21.2.112:0/1545268098 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8c080b7270 msgr2=0x7f8c080b7670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:59.411 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:59.408 7f8c0d0c0700 1 --2- 172.21.2.112:0/1545268098 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8c080b7270 0x7f8c080b7670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:13:59.411 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:59.408 7f8c0d0c0700 1 -- 172.21.2.112:0/1545268098 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c08085430 con 0x7f8c080b7c90 2021-02-10T11:13:59.411 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:59.409 7f8c0d0c0700 1 --2- 172.21.2.112:0/1545268098 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8c080b7c90 0x7f8c08084e10 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f8c00009390 tx=0x7f8c0000bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:13:59.412 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:59.409 7f8c057fa700 1 -- 172.21.2.112:0/1545268098 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8c00003030 con 0x7f8c080b7c90 2021-02-10T11:13:59.412 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:59.409 7f8c057fa700 1 -- 172.21.2.112:0/1545268098 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c000068e0 con 0x7f8c080b7c90 2021-02-10T11:13:59.412 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:59.409 7f8c057fa700 1 -- 172.21.2.112:0/1545268098 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8c0000c570 con 0x7f8c080b7c90 2021-02-10T11:13:59.412 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:59.409 7f8c0eaa3700 1 -- 172.21.2.112:0/1545268098 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c08085430 con 0x7f8c080b7c90 2021-02-10T11:13:59.412 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:59.409 7f8c0eaa3700 1 -- 172.21.2.112:0/1545268098 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c08085430 con 0x7f8c080b7c90 2021-02-10T11:13:59.412 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:59.410 7f8c057fa700 1 -- 172.21.2.112:0/1545268098 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8c00001270 con 0x7f8c080b7c90 2021-02-10T11:13:59.412 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:59.410 7f8c057fa700 1 --2- 172.21.2.112:0/1545268098 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8bec0186c0 0x7f8bec01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:59.412 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:59.410 7f8c057fa700 1 -- 172.21.2.112:0/1545268098 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f8c000325a0 con 0x7f8c080b7c90 2021-02-10T11:13:59.412 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:59.410 7f8c0c8bf700 1 --2- 172.21.2.112:0/1545268098 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8bec0186c0 0x7f8bec01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:59.412 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:59.411 7f8c0c8bf700 1 --2- 172.21.2.112:0/1545268098 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8bec0186c0 0x7f8bec01abc0 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:13:59.440 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:59.441 7f8c0eaa3700 1 -- 172.21.2.112:0/1545268098 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c080b8090 con 0x7f8c080b7c90 2021-02-10T11:13:59.443 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:59.443 7f8c057fa700 1 -- 172.21.2.112:0/1545268098 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8c0001c330 con 0x7f8c080b7c90 2021-02-10T11:13:59.572 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:59.571 7f8c0eaa3700 1 -- 172.21.2.112:0/1545268098 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8c080b8090 con 0x7f8c080b7c90 2021-02-10T11:13:59.572 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:59.572 7f8c057fa700 1 -- 172.21.2.112:0/1545268098 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v15) v1 ==== 74+0+11155 (secure 0 0 0) 0x7f8c0001c330 con 0x7f8c080b7c90 2021-02-10T11:13:59.573 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:13:59.573 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":15,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:13:58.954065","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:13:59.573 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:59.574 7f8c0eaa3700 1 -- 172.21.2.112:0/1545268098 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8bec0186c0 msgr2=0x7f8bec01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:59.574 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:59.574 7f8c0eaa3700 1 --2- 172.21.2.112:0/1545268098 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8bec0186c0 0x7f8bec01abc0 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:59.574 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:59.574 7f8c0eaa3700 1 -- 172.21.2.112:0/1545268098 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8c080b7c90 msgr2=0x7f8c08084e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:59.574 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:59.574 7f8c0eaa3700 1 --2- 172.21.2.112:0/1545268098 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8c080b7c90 0x7f8c08084e10 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f8c00009390 tx=0x7f8c0000bd40).stop 2021-02-10T11:13:59.574 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:59.574 7f8c0eaa3700 1 -- 172.21.2.112:0/1545268098 shutdown_connections 2021-02-10T11:13:59.574 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:59.574 7f8c0eaa3700 1 --2- 172.21.2.112:0/1545268098 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8bec0186c0 0x7f8bec01abc0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:59.574 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:59.574 7f8c0eaa3700 1 --2- 172.21.2.112:0/1545268098 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8c080b7270 0x7f8c080b7670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:13:59.574 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:59.574 7f8c0eaa3700 1 --2- 172.21.2.112:0/1545268098 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8c080b7c90 0x7f8c08084e10 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:59.574 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:59.574 7f8c0eaa3700 1 --2- 172.21.2.112:0/1545268098 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8c081aa290 0x7f8c081aa690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:13:59.574 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:59.574 7f8c0eaa3700 1 -- 172.21.2.112:0/1545268098 >> 172.21.2.112:0/1545268098 conn(0x7f8c0808fe10 msgr2=0x7f8c0819c020 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:13:59.574 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:59.574 7f8c0eaa3700 1 -- 172.21.2.112:0/1545268098 shutdown_connections 2021-02-10T11:13:59.574 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:59.574 7f8c0eaa3700 1 -- 172.21.2.112:0/1545268098 wait complete. 2021-02-10T11:13:59.582 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:13:57.508758', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2021-02-10T11:13:59.582 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:14:00.015 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:59.999 7fe211a18700 1 Processor -- start 2021-02-10T11:14:00.015 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:59.999 7fe211a18700 1 -- start start 2021-02-10T11:14:00.015 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:59.999 7fe211a18700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fe20c0f05f0 0x7fe20c0f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:00.015 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:59.999 7fe211a18700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe20c1b0c70 con 0x7fe20c1b0470 2021-02-10T11:14:00.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:59.999 7fe211a18700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe20c0f0b50 con 0x7fe20c1b0870 2021-02-10T11:14:00.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:13:59.999 7fe211a18700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe20c0e5ea0 con 0x7fe20c0f05f0 2021-02-10T11:14:00.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.000 7fe20bfff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fe20c0f05f0 0x7fe20c0f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:00.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.000 7fe20b7fe700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fe20c1b0470 0x7fe20c0ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:41450/0 (socket says 172.21.2.112:41450) 2021-02-10T11:14:00.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.000 7fe20b7fe700 1 -- 172.21.2.112:0/8283452 learned_addr learned my addr 172.21.2.112:0/8283452 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:14:00.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.000 7fe20bfff700 1 -- 172.21.2.112:0/8283452 >> v1:172.21.2.120:6789/0 conn(0x7fe20c1b0870 legacy=0x7fe20c0efe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:14:00.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.000 7fe20bfff700 1 -- 172.21.2.112:0/8283452 >> v1:172.21.2.112:6789/0 conn(0x7fe20c1b0470 legacy=0x7fe20c0ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:00.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.000 7fe20bfff700 1 -- 172.21.2.112:0/8283452 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe20c0e52e0 con 0x7fe20c0f05f0 2021-02-10T11:14:00.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.000 7fe20bfff700 1 --2- 172.21.2.112:0/8283452 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe20c0f05f0 0x7fe20c0f20e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7fe204006e30 tx=0x7fe204009e60).ready entity=mon.0 client_cookie=f47f79ec684110cb server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:00.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.000 7fe20a7fc700 1 -- 172.21.2.112:0/8283452 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe20400b300 con 0x7fe20c0f05f0 2021-02-10T11:14:00.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.001 7fe20a7fc700 1 -- 172.21.2.112:0/8283452 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe20400b580 con 0x7fe20c0f05f0 2021-02-10T11:14:00.017 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.001 7fe20a7fc700 1 -- 172.21.2.112:0/8283452 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe20400baa0 con 0x7fe20c0f05f0 2021-02-10T11:14:00.017 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.001 7fe211a18700 1 -- 172.21.2.112:0/8283452 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe20c0f05f0 msgr2=0x7fe20c0f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:00.017 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.001 7fe211a18700 1 --2- 172.21.2.112:0/8283452 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe20c0f05f0 0x7fe20c0f20e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7fe204006e30 tx=0x7fe204009e60).stop 2021-02-10T11:14:00.017 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.001 7fe211a18700 1 -- 172.21.2.112:0/8283452 shutdown_connections 2021-02-10T11:14:00.017 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.001 7fe211a18700 1 --2- 172.21.2.112:0/8283452 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe20c0f05f0 0x7fe20c0f20e0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:00.017 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.001 7fe211a18700 1 -- 172.21.2.112:0/8283452 >> 172.21.2.112:0/8283452 conn(0x7fe20c0ce3e0 msgr2=0x7fe20c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:00.017 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.001 7fe211a18700 1 -- 172.21.2.112:0/8283452 shutdown_connections 2021-02-10T11:14:00.017 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.001 7fe211a18700 1 -- 172.21.2.112:0/8283452 wait complete. 2021-02-10T11:14:00.019 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.016 7fe211a18700 1 Processor -- start 2021-02-10T11:14:00.019 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.016 7fe211a18700 1 -- start start 2021-02-10T11:14:00.019 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.016 7fe211a18700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe20c0a3b50 0x7fe20c1b3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:00.019 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.016 7fe211a18700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe20c1c25b0 0x7fe20c1b3d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:00.019 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.017 7fe211a18700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe20c1b4350 0x7fe20c1b18a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:00.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.017 7fe211a18700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe20c1b3190 con 0x7fe20c1c25b0 2021-02-10T11:14:00.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.017 7fe211a18700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe20c087750 con 0x7fe20c0a3b50 2021-02-10T11:14:00.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.017 7fe211a18700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe20c0a6bf0 con 0x7fe20c1b4350 2021-02-10T11:14:00.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.017 7fe20affd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe20c1c25b0 0x7fe20c1b3d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:00.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.017 7fe20affd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe20c1c25b0 0x7fe20c1b3d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:41996/0 (socket says 172.21.2.112:41996) 2021-02-10T11:14:00.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.017 7fe20affd700 1 -- 172.21.2.112:0/1096513608 learned_addr learned my addr 172.21.2.112:0/1096513608 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:14:00.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.017 7fe20affd700 1 -- 172.21.2.112:0/1096513608 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe20c1b4350 msgr2=0x7fe20c1b18a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:14:00.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.017 7fe20affd700 1 --2- 172.21.2.112:0/1096513608 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe20c1b4350 0x7fe20c1b18a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:00.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.017 7fe20affd700 1 -- 172.21.2.112:0/1096513608 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe20c0a3b50 msgr2=0x7fe20c1b3710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:00.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.017 7fe20affd700 1 --2- 172.21.2.112:0/1096513608 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe20c0a3b50 0x7fe20c1b3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:00.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.017 7fe20affd700 1 -- 172.21.2.112:0/1096513608 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe20c1b1ec0 con 0x7fe20c1c25b0 2021-02-10T11:14:00.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.017 7fe20affd700 1 --2- 172.21.2.112:0/1096513608 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe20c1c25b0 0x7fe20c1b3d30 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7fe1fc00b5b0 tx=0x7fe1fc00b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:00.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.017 7fe203fff700 1 -- 172.21.2.112:0/1096513608 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe1fc00bd40 con 0x7fe20c1c25b0 2021-02-10T11:14:00.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.017 7fe203fff700 1 -- 172.21.2.112:0/1096513608 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1fc000e20 con 0x7fe20c1c25b0 2021-02-10T11:14:00.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.018 7fe211a18700 1 -- 172.21.2.112:0/1096513608 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe20c1b1ec0 con 0x7fe20c1c25b0 2021-02-10T11:14:00.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.018 7fe203fff700 1 -- 172.21.2.112:0/1096513608 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe1fc0054a0 con 0x7fe20c1c25b0 2021-02-10T11:14:00.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.018 7fe211a18700 1 -- 172.21.2.112:0/1096513608 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe20c1b2450 con 0x7fe20c1c25b0 2021-02-10T11:14:00.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.018 7fe203fff700 1 -- 172.21.2.112:0/1096513608 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe1fc006030 con 0x7fe20c1c25b0 2021-02-10T11:14:00.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.018 7fe203fff700 1 --2- 172.21.2.112:0/1096513608 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe1f00186c0 0x7fe1f001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:00.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.018 7fe203fff700 1 -- 172.21.2.112:0/1096513608 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4753+0+0 (secure 0 0 0) 0x7fe1fc0320c0 con 0x7fe20c1c25b0 2021-02-10T11:14:00.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.019 7fe20b7fe700 1 --2- 172.21.2.112:0/1096513608 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe1f00186c0 0x7fe1f001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:00.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.019 7fe20b7fe700 1 --2- 172.21.2.112:0/1096513608 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe1f00186c0 0x7fe1f001abc0 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:00.049 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.049 7fe211a18700 1 -- 172.21.2.112:0/1096513608 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe20c07e6b0 con 0x7fe20c1c25b0 2021-02-10T11:14:00.051 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.052 7fe203fff700 1 -- 172.21.2.112:0/1096513608 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe1fc01dd30 con 0x7fe20c1c25b0 2021-02-10T11:14:00.180 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.181 7fe211a18700 1 -- 172.21.2.112:0/1096513608 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7fe20c0ce3e0 con 0x7fe20c1c25b0 2021-02-10T11:14:00.180 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.181 7fe203fff700 1 -- 172.21.2.112:0/1096513608 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v16) v1 ==== 92+0+10 (secure 0 0 0) 0x7fe1fc01dd30 con 0x7fe20c1c25b0 2021-02-10T11:14:00.181 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:14:00.183 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.183 7fe211a18700 1 -- 172.21.2.112:0/1096513608 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe1f00186c0 msgr2=0x7fe1f001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:00.183 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.183 7fe211a18700 1 --2- 172.21.2.112:0/1096513608 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe1f00186c0 0x7fe1f001abc0 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:00.183 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.183 7fe211a18700 1 -- 172.21.2.112:0/1096513608 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe20c1c25b0 msgr2=0x7fe20c1b3d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:00.183 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.183 7fe211a18700 1 --2- 172.21.2.112:0/1096513608 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe20c1c25b0 0x7fe20c1b3d30 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7fe1fc00b5b0 tx=0x7fe1fc00b950).stop 2021-02-10T11:14:00.183 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.183 7fe211a18700 1 -- 172.21.2.112:0/1096513608 shutdown_connections 2021-02-10T11:14:00.184 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.183 7fe211a18700 1 --2- 172.21.2.112:0/1096513608 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe1f00186c0 0x7fe1f001abc0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:00.184 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.183 7fe211a18700 1 --2- 172.21.2.112:0/1096513608 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe20c1b4350 0x7fe20c1b18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:00.184 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.183 7fe211a18700 1 --2- 172.21.2.112:0/1096513608 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe20c1c25b0 0x7fe20c1b3d30 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:00.184 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.183 7fe211a18700 1 --2- 172.21.2.112:0/1096513608 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe20c0a3b50 0x7fe20c1b3710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:00.184 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.183 7fe211a18700 1 -- 172.21.2.112:0/1096513608 >> 172.21.2.112:0/1096513608 conn(0x7fe20c0886d0 msgr2=0x7fe20c081c40 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:00.184 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.184 7fe211a18700 1 -- 172.21.2.112:0/1096513608 shutdown_connections 2021-02-10T11:14:00.184 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.184 7fe211a18700 1 -- 172.21.2.112:0/1096513608 wait complete. 2021-02-10T11:14:00.191 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --format=json-pretty osd lspools 2021-02-10T11:14:00.618 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.601 7efc7ce9d700 1 Processor -- start 2021-02-10T11:14:00.618 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.601 7efc7ce9d700 1 -- start start 2021-02-10T11:14:00.618 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.602 7efc7ce9d700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7efc780e5070 0x7efc781b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:00.618 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.602 7efc7ce9d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7efc780e5470 0x7efc78080310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:00.618 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.602 7efc7ce9d700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc7807ed80 con 0x7efc781b3e40 2021-02-10T11:14:00.618 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.602 7efc7ce9d700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc780d1790 con 0x7efc780e5470 2021-02-10T11:14:00.618 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.602 7efc7ce9d700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc780d19b0 con 0x7efc780e5070 2021-02-10T11:14:00.618 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.602 7efc77314700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7efc780e5470 0x7efc78080310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:00.619 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.602 7efc77314700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7efc780e5470 0x7efc78080310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:42004/0 (socket says 172.21.2.112:42004) 2021-02-10T11:14:00.619 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.602 7efc77314700 1 -- 172.21.2.112:0/3239424279 learned_addr learned my addr 172.21.2.112:0/3239424279 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:14:00.619 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.602 7efc77314700 1 -- 172.21.2.112:0/3239424279 >> v2:172.21.2.120:3300/0 conn(0x7efc780e5070 msgr2=0x7efc781b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:14:00.619 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.602 7efc77314700 1 --2- 172.21.2.112:0/3239424279 >> v2:172.21.2.120:3300/0 conn(0x7efc780e5070 0x7efc781b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:00.619 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.602 7efc77314700 1 -- 172.21.2.112:0/3239424279 >> v1:172.21.2.120:6789/0 conn(0x7efc781b3e40 legacy=0x7efc7807d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:14:00.619 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.602 7efc77314700 1 -- 172.21.2.112:0/3239424279 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc780d1bd0 con 0x7efc780e5470 2021-02-10T11:14:00.619 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.602 7efc77314700 1 --2- 172.21.2.112:0/3239424279 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efc780e5470 0x7efc78080310 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7efc70006f00 tx=0x7efc70009f10).ready entity=mon.0 client_cookie=6ccc72ff91f3b433 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:00.619 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.603 7efc75b11700 1 -- 172.21.2.112:0/3239424279 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efc7000b3b0 con 0x7efc780e5470 2021-02-10T11:14:00.619 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.603 7efc75b11700 1 -- 172.21.2.112:0/3239424279 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc7000b730 con 0x7efc780e5470 2021-02-10T11:14:00.619 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.603 7efc75b11700 1 -- 172.21.2.112:0/3239424279 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efc7000bc50 con 0x7efc780e5470 2021-02-10T11:14:00.619 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.603 7efc7ce9d700 1 -- 172.21.2.112:0/3239424279 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efc780e5470 msgr2=0x7efc78080310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:00.619 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.603 7efc7ce9d700 1 --2- 172.21.2.112:0/3239424279 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efc780e5470 0x7efc78080310 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7efc70006f00 tx=0x7efc70009f10).stop 2021-02-10T11:14:00.620 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.603 7efc7ce9d700 1 -- 172.21.2.112:0/3239424279 shutdown_connections 2021-02-10T11:14:00.620 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.603 7efc7ce9d700 1 --2- 172.21.2.112:0/3239424279 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efc780e5470 0x7efc78080310 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:00.620 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.603 7efc7ce9d700 1 --2- 172.21.2.112:0/3239424279 >> v2:172.21.2.120:3300/0 conn(0x7efc780e5070 0x7efc781b3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:00.620 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.603 7efc7ce9d700 1 -- 172.21.2.112:0/3239424279 >> 172.21.2.112:0/3239424279 conn(0x7efc780ce690 msgr2=0x7efc780e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:00.620 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.604 7efc7ce9d700 1 -- 172.21.2.112:0/3239424279 shutdown_connections 2021-02-10T11:14:00.620 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.604 7efc7ce9d700 1 -- 172.21.2.112:0/3239424279 wait complete. 2021-02-10T11:14:00.620 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.618 7efc7ce9d700 1 Processor -- start 2021-02-10T11:14:00.620 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.618 7efc7ce9d700 1 -- start start 2021-02-10T11:14:00.621 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.619 7efc7ce9d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efc781aa290 0x7efc781aa690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:00.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.619 7efc7ce9d700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7efc780b7270 0x7efc780b7670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:00.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.619 7efc7ce9d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efc780b7c90 0x7efc78084e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:00.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.619 7efc7ce9d700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc780b8090 con 0x7efc780b7c90 2021-02-10T11:14:00.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.619 7efc7ce9d700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc781a9df0 con 0x7efc780b7270 2021-02-10T11:14:00.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.619 7efc7ce9d700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc7809cc30 con 0x7efc781aa290 2021-02-10T11:14:00.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.619 7efc77314700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efc780b7c90 0x7efc78084e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:00.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.619 7efc77314700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efc780b7c90 0x7efc78084e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:42010/0 (socket says 172.21.2.112:42010) 2021-02-10T11:14:00.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.619 7efc77314700 1 -- 172.21.2.112:0/3803131178 learned_addr learned my addr 172.21.2.112:0/3803131178 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:14:00.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.619 7efc77314700 1 -- 172.21.2.112:0/3803131178 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efc781aa290 msgr2=0x7efc781aa690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:00.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.619 7efc77314700 1 --2- 172.21.2.112:0/3803131178 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efc781aa290 0x7efc781aa690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:00.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.619 7efc77314700 1 -- 172.21.2.112:0/3803131178 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7efc780b7270 msgr2=0x7efc780b7670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:00.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.619 7efc77314700 1 --2- 172.21.2.112:0/3803131178 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7efc780b7270 0x7efc780b7670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:00.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.619 7efc77314700 1 -- 172.21.2.112:0/3803131178 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc78085430 con 0x7efc780b7c90 2021-02-10T11:14:00.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.619 7efc77314700 1 --2- 172.21.2.112:0/3803131178 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efc780b7c90 0x7efc78084e10 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7efc70009390 tx=0x7efc7000bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:00.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.620 7efc6f7fe700 1 -- 172.21.2.112:0/3803131178 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efc70003030 con 0x7efc780b7c90 2021-02-10T11:14:00.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.620 7efc6f7fe700 1 -- 172.21.2.112:0/3803131178 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc700068e0 con 0x7efc780b7c90 2021-02-10T11:14:00.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.620 7efc6f7fe700 1 -- 172.21.2.112:0/3803131178 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efc7000c570 con 0x7efc780b7c90 2021-02-10T11:14:00.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.620 7efc7ce9d700 1 -- 172.21.2.112:0/3803131178 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc78085430 con 0x7efc780b7c90 2021-02-10T11:14:00.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.620 7efc7ce9d700 1 -- 172.21.2.112:0/3803131178 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc78085430 con 0x7efc780b7c90 2021-02-10T11:14:00.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.620 7efc6f7fe700 1 -- 172.21.2.112:0/3803131178 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7efc70001270 con 0x7efc780b7c90 2021-02-10T11:14:00.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.621 7efc6f7fe700 1 --2- 172.21.2.112:0/3803131178 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7efc5c0186c0 0x7efc5c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:00.624 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.621 7efc6f7fe700 1 -- 172.21.2.112:0/3803131178 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4753+0+0 (secure 0 0 0) 0x7efc700325c0 con 0x7efc780b7c90 2021-02-10T11:14:00.624 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.621 7efc76b13700 1 --2- 172.21.2.112:0/3803131178 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7efc5c0186c0 0x7efc5c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:00.624 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.621 7efc76b13700 1 --2- 172.21.2.112:0/3803131178 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7efc5c0186c0 0x7efc5c01abc0 crc :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:00.651 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.652 7efc7ce9d700 1 -- 172.21.2.112:0/3803131178 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc780b8090 con 0x7efc780b7c90 2021-02-10T11:14:00.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.654 7efc6f7fe700 1 -- 172.21.2.112:0/3803131178 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7efc70006c80 con 0x7efc780b7c90 2021-02-10T11:14:00.777 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.777 7efc7ce9d700 1 -- 172.21.2.112:0/3803131178 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7efc780b8090 con 0x7efc780b7c90 2021-02-10T11:14:00.778 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.778 7efc6f7fe700 1 -- 172.21.2.112:0/3803131178 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v16) v1 ==== 84+0+64 (secure 0 0 0) 0x7efc70001270 con 0x7efc780b7c90 2021-02-10T11:14:00.778 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:14:00.778 INFO:teuthology.orchestra.run.gibba012.stdout:[ 2021-02-10T11:14:00.778 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:14:00.778 INFO:teuthology.orchestra.run.gibba012.stdout: "poolnum": 1, 2021-02-10T11:14:00.779 INFO:teuthology.orchestra.run.gibba012.stdout: "poolname": "rbd" 2021-02-10T11:14:00.779 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:14:00.779 INFO:teuthology.orchestra.run.gibba012.stdout:] 2021-02-10T11:14:00.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.780 7efc7ce9d700 1 -- 172.21.2.112:0/3803131178 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7efc5c0186c0 msgr2=0x7efc5c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:00.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.780 7efc7ce9d700 1 --2- 172.21.2.112:0/3803131178 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7efc5c0186c0 0x7efc5c01abc0 crc :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:00.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.780 7efc7ce9d700 1 -- 172.21.2.112:0/3803131178 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efc780b7c90 msgr2=0x7efc78084e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:00.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.780 7efc7ce9d700 1 --2- 172.21.2.112:0/3803131178 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efc780b7c90 0x7efc78084e10 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7efc70009390 tx=0x7efc7000bd40).stop 2021-02-10T11:14:00.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.780 7efc7ce9d700 1 -- 172.21.2.112:0/3803131178 shutdown_connections 2021-02-10T11:14:00.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.780 7efc7ce9d700 1 --2- 172.21.2.112:0/3803131178 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7efc5c0186c0 0x7efc5c01abc0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:00.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.780 7efc7ce9d700 1 --2- 172.21.2.112:0/3803131178 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7efc780b7270 0x7efc780b7670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:00.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.780 7efc7ce9d700 1 --2- 172.21.2.112:0/3803131178 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efc780b7c90 0x7efc78084e10 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:00.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.780 7efc7ce9d700 1 --2- 172.21.2.112:0/3803131178 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efc781aa290 0x7efc781aa690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:00.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.780 7efc7ce9d700 1 -- 172.21.2.112:0/3803131178 >> 172.21.2.112:0/3803131178 conn(0x7efc7808fe10 msgr2=0x7efc7819c020 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:00.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.780 7efc7ce9d700 1 -- 172.21.2.112:0/3803131178 shutdown_connections 2021-02-10T11:14:00.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:00.780 7efc7ce9d700 1 -- 172.21.2.112:0/3803131178 wait complete. 2021-02-10T11:14:00.788 INFO:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2021-02-10T11:14:00.788 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_metadata 8 2021-02-10T11:14:00.900 DEBUG:teuthology.orchestra.run.gibba012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:14:00.903 DEBUG:teuthology.orchestra.run.gibba020:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:14:01.220 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:01.202 7f25a100b700 1 Processor -- start 2021-02-10T11:14:01.220 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:01.202 7f25a100b700 1 -- start start 2021-02-10T11:14:01.220 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:01.203 7f25a100b700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f259c1b0870 0x7f259c0f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:01.220 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:01.203 7f25a100b700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f259c0efe00 0x7f259c0f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:01.220 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:01.203 7f25a100b700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f259c1b0c70 con 0x7f259c1b0470 2021-02-10T11:14:01.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:01.203 7f25a100b700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f259c0e5ea0 con 0x7f259c0efe00 2021-02-10T11:14:01.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:01.203 7f25a100b700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f259c0f0b50 con 0x7f259c1b0870 2021-02-10T11:14:01.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:01.203 7f259b59e700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f259c0efe00 0x7f259c0f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:01.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:01.203 7f259b59e700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f259c0efe00 0x7f259c0f0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:42018/0 (socket says 172.21.2.112:42018) 2021-02-10T11:14:01.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:01.203 7f259b59e700 1 -- 172.21.2.112:0/584729196 learned_addr learned my addr 172.21.2.112:0/584729196 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:14:01.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:01.203 7f259b59e700 1 -- 172.21.2.112:0/584729196 >> v2:172.21.2.120:3300/0 conn(0x7f259c1b0870 msgr2=0x7f259c0f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:14:01.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:01.203 7f259b59e700 1 --2- 172.21.2.112:0/584729196 >> v2:172.21.2.120:3300/0 conn(0x7f259c1b0870 0x7f259c0f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:01.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:01.203 7f259b59e700 1 -- 172.21.2.112:0/584729196 >> v1:172.21.2.120:6789/0 conn(0x7f259c1b0470 legacy=0x7f259c0ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:14:01.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:01.203 7f259b59e700 1 -- 172.21.2.112:0/584729196 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f259c0f0820 con 0x7f259c0efe00 2021-02-10T11:14:01.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:01.203 7f259b59e700 1 --2- 172.21.2.112:0/584729196 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f259c0efe00 0x7f259c0f0200 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f2594006f00 tx=0x7f2594009f10).ready entity=mon.0 client_cookie=8579cbdd68895956 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:01.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:01.204 7f2599d9b700 1 -- 172.21.2.112:0/584729196 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f259400b3b0 con 0x7f259c0efe00 2021-02-10T11:14:01.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:01.204 7f2599d9b700 1 -- 172.21.2.112:0/584729196 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f259400b730 con 0x7f259c0efe00 2021-02-10T11:14:01.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:01.204 7f2599d9b700 1 -- 172.21.2.112:0/584729196 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f259400bb90 con 0x7f259c0efe00 2021-02-10T11:14:01.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:01.204 7f25a100b700 1 -- 172.21.2.112:0/584729196 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f259c0efe00 msgr2=0x7f259c0f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:01.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:01.204 7f25a100b700 1 --2- 172.21.2.112:0/584729196 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f259c0efe00 0x7f259c0f0200 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f2594006f00 tx=0x7f2594009f10).stop 2021-02-10T11:14:01.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:01.204 7f25a100b700 1 -- 172.21.2.112:0/584729196 shutdown_connections 2021-02-10T11:14:01.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:01.204 7f25a100b700 1 --2- 172.21.2.112:0/584729196 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f259c0efe00 0x7f259c0f0200 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:01.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:01.204 7f25a100b700 1 --2- 172.21.2.112:0/584729196 >> v2:172.21.2.120:3300/0 conn(0x7f259c1b0870 0x7f259c0f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:01.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:01.204 7f25a100b700 1 -- 172.21.2.112:0/584729196 >> 172.21.2.112:0/584729196 conn(0x7f259c0ce3e0 msgr2=0x7f259c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:01.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:01.204 7f25a100b700 1 -- 172.21.2.112:0/584729196 shutdown_connections 2021-02-10T11:14:01.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:01.204 7f25a100b700 1 -- 172.21.2.112:0/584729196 wait complete. 2021-02-10T11:14:01.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:01.219 7f25a100b700 1 Processor -- start 2021-02-10T11:14:01.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:01.219 7f25a100b700 1 -- start start 2021-02-10T11:14:01.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:01.219 7f25a100b700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f259c094fb0 0x7f259c09e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:01.223 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:01.219 7f25a100b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f259c0c2be0 0x7f259c0c2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:01.223 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:01.219 7f25a100b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f259c0c3600 0x7f259c0a9c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:01.223 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:01.219 7f25a100b700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f259c09ebf0 con 0x7f259c0c2be0 2021-02-10T11:14:01.223 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:01.219 7f25a100b700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f259c0aa2a0 con 0x7f259c094fb0 2021-02-10T11:14:01.223 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:01.219 7f25a100b700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f259c0aa4c0 con 0x7f259c0c3600 2021-02-10T11:14:01.223 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:01.219 7f259a59c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f259c0c2be0 0x7f259c0c2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:01.223 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:01.220 7f259a59c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f259c0c2be0 0x7f259c0c2fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:42024/0 (socket says 172.21.2.112:42024) 2021-02-10T11:14:01.223 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:01.220 7f259a59c700 1 -- 172.21.2.112:0/4261401089 learned_addr learned my addr 172.21.2.112:0/4261401089 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:14:01.223 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:01.220 7f259a59c700 1 -- 172.21.2.112:0/4261401089 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f259c0c3600 msgr2=0x7f259c0a9c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:01.223 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:01.220 7f259a59c700 1 --2- 172.21.2.112:0/4261401089 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f259c0c3600 0x7f259c0a9c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:01.223 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:01.220 7f259a59c700 1 -- 172.21.2.112:0/4261401089 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f259c094fb0 msgr2=0x7f259c09e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:01.223 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:01.220 7f259a59c700 1 --2- 172.21.2.112:0/4261401089 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f259c094fb0 0x7f259c09e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:01.223 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:01.220 7f259a59c700 1 -- 172.21.2.112:0/4261401089 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f259c0aa6e0 con 0x7f259c0c2be0 2021-02-10T11:14:01.223 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:01.220 7f259a59c700 1 --2- 172.21.2.112:0/4261401089 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f259c0c2be0 0x7f259c0c2fe0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f258c00b530 tx=0x7f258c00b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:01.224 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:01.220 7f25937fe700 1 -- 172.21.2.112:0/4261401089 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f258c006350 con 0x7f259c0c2be0 2021-02-10T11:14:01.224 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:01.220 7f25937fe700 1 -- 172.21.2.112:0/4261401089 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f258c00bcc0 con 0x7f259c0c2be0 2021-02-10T11:14:01.224 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:01.220 7f25a100b700 1 -- 172.21.2.112:0/4261401089 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f259c0aa6e0 con 0x7f259c0c2be0 2021-02-10T11:14:01.224 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:01.220 7f25a100b700 1 -- 172.21.2.112:0/4261401089 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f259c0aac70 con 0x7f259c0c2be0 2021-02-10T11:14:01.224 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:01.220 7f25937fe700 1 -- 172.21.2.112:0/4261401089 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f258c0020d0 con 0x7f259c0c2be0 2021-02-10T11:14:01.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:01.221 7f25937fe700 1 -- 172.21.2.112:0/4261401089 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f258c007000 con 0x7f259c0c2be0 2021-02-10T11:14:01.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:01.221 7f25937fe700 1 --2- 172.21.2.112:0/4261401089 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2580018730 0x7f258001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:01.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:01.221 7f25937fe700 1 -- 172.21.2.112:0/4261401089 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f258c032470 con 0x7f259c0c2be0 2021-02-10T11:14:01.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:01.221 7f259ad9d700 1 --2- 172.21.2.112:0/4261401089 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2580018730 0x7f258001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:01.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:01.222 7f259ad9d700 1 --2- 172.21.2.112:0/4261401089 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2580018730 0x7f258001ac10 crc :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:01.253 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:01.253 7f25a100b700 1 -- 172.21.2.112:0/4261401089 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f259c0a9750 con 0x7f259c0c2be0 2021-02-10T11:14:01.255 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:01.255 7f25937fe700 1 -- 172.21.2.112:0/4261401089 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f258c000940 con 0x7f259c0c2be0 2021-02-10T11:14:01.382 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:01.382 7f25a100b700 1 -- 172.21.2.112:0/4261401089 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"} v 0) v1 -- 0x7f259c0a9750 con 0x7f259c0c2be0 2021-02-10T11:14:01.971 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:01.972 7f25937fe700 1 -- 172.21.2.112:0/4261401089 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"}]=0 pool 'cephfs_metadata' created v17) v1 ==== 133+0+0 (secure 0 0 0) 0x7f258c007000 con 0x7f259c0c2be0 2021-02-10T11:14:01.999 INFO:teuthology.orchestra.run.gibba012.stderr:pool 'cephfs_metadata' created 2021-02-10T11:14:02.001 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:02.002 7f25a100b700 1 -- 172.21.2.112:0/4261401089 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2580018730 msgr2=0x7f258001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:02.001 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:02.002 7f25a100b700 1 --2- 172.21.2.112:0/4261401089 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2580018730 0x7f258001ac10 crc :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:02.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:02.002 7f25a100b700 1 -- 172.21.2.112:0/4261401089 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f259c0c2be0 msgr2=0x7f259c0c2fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:02.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:02.002 7f25a100b700 1 --2- 172.21.2.112:0/4261401089 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f259c0c2be0 0x7f259c0c2fe0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f258c00b530 tx=0x7f258c00b8d0).stop 2021-02-10T11:14:02.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:02.002 7f25a100b700 1 -- 172.21.2.112:0/4261401089 shutdown_connections 2021-02-10T11:14:02.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:02.002 7f25a100b700 1 --2- 172.21.2.112:0/4261401089 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2580018730 0x7f258001ac10 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:02.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:02.002 7f25a100b700 1 --2- 172.21.2.112:0/4261401089 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f259c0c3600 0x7f259c0a9c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:02.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:02.002 7f25a100b700 1 --2- 172.21.2.112:0/4261401089 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f259c0c2be0 0x7f259c0c2fe0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:02.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:02.002 7f25a100b700 1 --2- 172.21.2.112:0/4261401089 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f259c094fb0 0x7f259c09e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:02.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:02.002 7f25a100b700 1 -- 172.21.2.112:0/4261401089 >> 172.21.2.112:0/4261401089 conn(0x7f259c09f630 msgr2=0x7f259c0a0130 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:02.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:02.002 7f25a100b700 1 -- 172.21.2.112:0/4261401089 shutdown_connections 2021-02-10T11:14:02.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:02.002 7f25a100b700 1 -- 172.21.2.112:0/4261401089 wait complete. 2021-02-10T11:14:02.010 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_data 8 2021-02-10T11:14:02.438 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:02.422 7f046ab52700 1 Processor -- start 2021-02-10T11:14:02.438 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:02.422 7f046ab52700 1 -- start start 2021-02-10T11:14:02.438 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:02.422 7f046ab52700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f04641b0470 0x7f04641b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:02.438 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:02.422 7f046ab52700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f04640ef610 0x7f04640efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:02.438 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:02.422 7f046ab52700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04640f0b50 con 0x7f04640f0030 2021-02-10T11:14:02.438 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:02.422 7f046ab52700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04640e52e0 con 0x7f04641b0470 2021-02-10T11:14:02.439 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:02.422 7f046ab52700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04640e54d0 con 0x7f04640ef610 2021-02-10T11:14:02.439 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:02.423 7f046896e700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f04641b0470 0x7f04641b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:02.439 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:02.423 7f046896e700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f04641b0470 0x7f04641b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:42030/0 (socket says 172.21.2.112:42030) 2021-02-10T11:14:02.439 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:02.423 7f046896e700 1 -- 172.21.2.112:0/20902846 learned_addr learned my addr 172.21.2.112:0/20902846 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:14:02.439 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:02.423 7f046896e700 1 -- 172.21.2.112:0/20902846 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f04640ef610 msgr2=0x7f04640efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:14:02.439 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:02.423 7f046896e700 1 --2- 172.21.2.112:0/20902846 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f04640ef610 0x7f04640efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:02.439 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:02.423 7f046896e700 1 -- 172.21.2.112:0/20902846 >> v1:172.21.2.120:6789/0 conn(0x7f04640f0030 legacy=0x7f04640f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:14:02.439 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:02.423 7f046896e700 1 -- 172.21.2.112:0/20902846 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04640f0430 con 0x7f04641b0470 2021-02-10T11:14:02.439 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:02.423 7f046896e700 1 --2- 172.21.2.112:0/20902846 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f04641b0470 0x7f04641b0870 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f044c006700 tx=0x7f044c009b80).ready entity=mon.0 client_cookie=538a4185657f9d61 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:02.440 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:02.423 7f04637fe700 1 -- 172.21.2.112:0/20902846 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f044c00c0c0 con 0x7f04641b0470 2021-02-10T11:14:02.440 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:02.423 7f04637fe700 1 -- 172.21.2.112:0/20902846 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f044c00c400 con 0x7f04641b0470 2021-02-10T11:14:02.440 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:02.423 7f04637fe700 1 -- 172.21.2.112:0/20902846 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f044c00c860 con 0x7f04641b0470 2021-02-10T11:14:02.440 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:02.423 7f046ab52700 1 -- 172.21.2.112:0/20902846 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f04641b0470 msgr2=0x7f04641b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:02.440 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:02.423 7f046ab52700 1 --2- 172.21.2.112:0/20902846 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f04641b0470 0x7f04641b0870 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f044c006700 tx=0x7f044c009b80).stop 2021-02-10T11:14:02.440 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:02.424 7f046ab52700 1 -- 172.21.2.112:0/20902846 shutdown_connections 2021-02-10T11:14:02.440 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:02.424 7f046ab52700 1 --2- 172.21.2.112:0/20902846 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f04640ef610 0x7f04640efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:02.440 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:02.424 7f046ab52700 1 --2- 172.21.2.112:0/20902846 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f04641b0470 0x7f04641b0870 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:02.440 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:02.424 7f046ab52700 1 -- 172.21.2.112:0/20902846 >> 172.21.2.112:0/20902846 conn(0x7f04640ce3e0 msgr2=0x7f04640e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:02.440 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:02.424 7f046ab52700 1 -- 172.21.2.112:0/20902846 shutdown_connections 2021-02-10T11:14:02.440 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:02.424 7f046ab52700 1 -- 172.21.2.112:0/20902846 wait complete. 2021-02-10T11:14:02.442 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:02.439 7f046ab52700 1 Processor -- start 2021-02-10T11:14:02.442 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:02.439 7f046ab52700 1 -- start start 2021-02-10T11:14:02.442 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:02.439 7f046ab52700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f04640c36a0 0x7f04640c2b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:02.442 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:02.439 7f046ab52700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f04640c3190 0x7f04640b1d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:02.442 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:02.439 7f046ab52700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f04640b2370 0x7f04640b2d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:02.442 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:02.439 7f046ab52700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04640b27c0 con 0x7f04640b2370 2021-02-10T11:14:02.442 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:02.439 7f046ab52700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04640b3360 con 0x7f04640c36a0 2021-02-10T11:14:02.443 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:02.439 7f046ab52700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04640b3540 con 0x7f04640c3190 2021-02-10T11:14:02.443 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:02.440 7f046916f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f04640b2370 0x7f04640b2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:02.443 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:02.440 7f046916f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f04640b2370 0x7f04640b2d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:42034/0 (socket says 172.21.2.112:42034) 2021-02-10T11:14:02.443 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:02.440 7f046916f700 1 -- 172.21.2.112:0/2476029818 learned_addr learned my addr 172.21.2.112:0/2476029818 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:14:02.443 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:02.440 7f046916f700 1 -- 172.21.2.112:0/2476029818 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f04640c3190 msgr2=0x7f04640b1d50 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:14:02.443 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:02.440 7f046916f700 1 --2- 172.21.2.112:0/2476029818 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f04640c3190 0x7f04640b1d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:02.443 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:02.440 7f046916f700 1 -- 172.21.2.112:0/2476029818 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f04640c36a0 msgr2=0x7f04640c2b70 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:14:02.443 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:02.440 7f046916f700 1 --2- 172.21.2.112:0/2476029818 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f04640c36a0 0x7f04640c2b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:02.443 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:02.440 7f046916f700 1 -- 172.21.2.112:0/2476029818 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04640b3760 con 0x7f04640b2370 2021-02-10T11:14:02.443 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:02.440 7f046916f700 1 --2- 172.21.2.112:0/2476029818 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f04640b2370 0x7f04640b2d40 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f045c006ed0 tx=0x7f045c00b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:02.443 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:02.440 7f04617fa700 1 -- 172.21.2.112:0/2476029818 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f045c00bd50 con 0x7f04640b2370 2021-02-10T11:14:02.443 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:02.440 7f04617fa700 1 -- 172.21.2.112:0/2476029818 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f045c0009b0 con 0x7f04640b2370 2021-02-10T11:14:02.443 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:02.440 7f04617fa700 1 -- 172.21.2.112:0/2476029818 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f045c0062a0 con 0x7f04640b2370 2021-02-10T11:14:02.443 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:02.440 7f046ab52700 1 -- 172.21.2.112:0/2476029818 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04640b3760 con 0x7f04640b2370 2021-02-10T11:14:02.444 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:02.440 7f046ab52700 1 -- 172.21.2.112:0/2476029818 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04640b3760 con 0x7f04640b2370 2021-02-10T11:14:02.444 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:02.441 7f04617fa700 1 -- 172.21.2.112:0/2476029818 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f045c006020 con 0x7f04640b2370 2021-02-10T11:14:02.444 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:02.441 7f04617fa700 1 --2- 172.21.2.112:0/2476029818 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f04480186c0 0x7f044801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:02.444 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:02.441 7f04617fa700 1 -- 172.21.2.112:0/2476029818 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 5107+0+0 (secure 0 0 0) 0x7f045c0323b0 con 0x7f04640b2370 2021-02-10T11:14:02.444 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:02.441 7f046896e700 1 --2- 172.21.2.112:0/2476029818 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f04480186c0 0x7f044801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:02.444 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:02.442 7f046896e700 1 --2- 172.21.2.112:0/2476029818 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f04480186c0 0x7f044801abc0 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:02.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:02.472 7f046ab52700 1 -- 172.21.2.112:0/2476029818 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04640edd10 con 0x7f04640b2370 2021-02-10T11:14:02.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:02.475 7f04617fa700 1 -- 172.21.2.112:0/2476029818 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f045c030db0 con 0x7f04640b2370 2021-02-10T11:14:02.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:02.604 7f046ab52700 1 -- 172.21.2.112:0/2476029818 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"} v 0) v1 -- 0x7f04640edd10 con 0x7f04640b2370 2021-02-10T11:14:02.974 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:02.975 7f04617fa700 1 -- 172.21.2.112:0/2476029818 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"}]=0 pool 'cephfs_data' created v18) v1 ==== 125+0+0 (secure 0 0 0) 0x7f045c006020 con 0x7f04640b2370 2021-02-10T11:14:03.020 INFO:teuthology.orchestra.run.gibba012.stderr:pool 'cephfs_data' created 2021-02-10T11:14:03.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:03.022 7f046ab52700 1 -- 172.21.2.112:0/2476029818 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f04480186c0 msgr2=0x7f044801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:03.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:03.022 7f046ab52700 1 --2- 172.21.2.112:0/2476029818 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f04480186c0 0x7f044801abc0 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:03.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:03.022 7f046ab52700 1 -- 172.21.2.112:0/2476029818 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f04640b2370 msgr2=0x7f04640b2d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:03.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:03.022 7f046ab52700 1 --2- 172.21.2.112:0/2476029818 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f04640b2370 0x7f04640b2d40 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f045c006ed0 tx=0x7f045c00b960).stop 2021-02-10T11:14:03.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:03.022 7f046ab52700 1 -- 172.21.2.112:0/2476029818 shutdown_connections 2021-02-10T11:14:03.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:03.022 7f046ab52700 1 --2- 172.21.2.112:0/2476029818 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f04480186c0 0x7f044801abc0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:03.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:03.022 7f046ab52700 1 --2- 172.21.2.112:0/2476029818 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f04640b2370 0x7f04640b2d40 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:03.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:03.022 7f046ab52700 1 --2- 172.21.2.112:0/2476029818 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f04640c3190 0x7f04640b1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:03.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:03.022 7f046ab52700 1 --2- 172.21.2.112:0/2476029818 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f04640c36a0 0x7f04640c2b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:03.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:03.022 7f046ab52700 1 -- 172.21.2.112:0/2476029818 >> 172.21.2.112:0/2476029818 conn(0x7f046409f630 msgr2=0x7f04640a0130 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:03.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:03.023 7f046ab52700 1 -- 172.21.2.112:0/2476029818 shutdown_connections 2021-02-10T11:14:03.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:03.023 7f046ab52700 1 -- 172.21.2.112:0/2476029818 wait complete. 2021-02-10T11:14:03.030 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2021-02-10T11:14:03.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:03.442 7f814b269700 1 Processor -- start 2021-02-10T11:14:03.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:03.442 7f814b269700 1 -- start start 2021-02-10T11:14:03.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:03.443 7f814b269700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f81440d1870 0x7f81441b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:03.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:03.443 7f814b269700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f81441b0ae0 0x7f8144108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:03.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:03.443 7f814b269700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8144109a60 con 0x7f81440d1c70 2021-02-10T11:14:03.460 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:03.443 7f814b269700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81440e5ef0 con 0x7f81441b0ae0 2021-02-10T11:14:03.460 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:03.443 7f814b269700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8144109750 con 0x7f81440d1870 2021-02-10T11:14:03.460 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:03.443 7f8148884700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f81441b0ae0 0x7f8144108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:03.460 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:03.443 7f8148884700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f81441b0ae0 0x7f8144108940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:42042/0 (socket says 172.21.2.112:42042) 2021-02-10T11:14:03.460 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:03.443 7f8148884700 1 -- 172.21.2.112:0/1167183758 learned_addr learned my addr 172.21.2.112:0/1167183758 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:14:03.460 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:03.444 7f8149085700 1 --2- 172.21.2.112:0/1167183758 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f81440d1870 0x7f81441b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:03.460 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:03.444 7f8148884700 1 -- 172.21.2.112:0/1167183758 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f81440d1870 msgr2=0x7f81441b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:14:03.460 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:03.444 7f8148884700 1 --2- 172.21.2.112:0/1167183758 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f81440d1870 0x7f81441b04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:03.460 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:03.444 7f8148884700 1 -- 172.21.2.112:0/1167183758 >> v1:172.21.2.112:6789/0 conn(0x7f81440d1c70 legacy=0x7f8144108f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:03.460 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:03.444 7f813bfff700 1 -- 172.21.2.112:0/1167183758 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3993403446 0 0) 0x7f813c000f90 con 0x7f81440d1c70 2021-02-10T11:14:03.460 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:03.444 7f8148884700 1 -- 172.21.2.112:0/1167183758 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8144088020 con 0x7f81441b0ae0 2021-02-10T11:14:03.460 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:03.444 7f8149085700 1 --2- 172.21.2.112:0/1167183758 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f81440d1870 0x7f81441b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:14:03.460 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:03.444 7f8148884700 1 --2- 172.21.2.112:0/1167183758 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f81441b0ae0 0x7f8144108940 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f8134006700 tx=0x7f8134009b80).ready entity=mon.0 client_cookie=51b0345647b8bee5 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:03.461 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:03.444 7f813bfff700 1 -- 172.21.2.112:0/1167183758 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f813400c0c0 con 0x7f81441b0ae0 2021-02-10T11:14:03.461 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:03.444 7f813bfff700 1 -- 172.21.2.112:0/1167183758 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f813400c340 con 0x7f81441b0ae0 2021-02-10T11:14:03.461 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:03.444 7f813bfff700 1 -- 172.21.2.112:0/1167183758 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f813400c860 con 0x7f81441b0ae0 2021-02-10T11:14:03.461 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:03.444 7f814b269700 1 -- 172.21.2.112:0/1167183758 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f81441b0ae0 msgr2=0x7f8144108940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:03.461 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:03.444 7f814b269700 1 --2- 172.21.2.112:0/1167183758 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f81441b0ae0 0x7f8144108940 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f8134006700 tx=0x7f8134009b80).stop 2021-02-10T11:14:03.461 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:03.445 7f814b269700 1 -- 172.21.2.112:0/1167183758 shutdown_connections 2021-02-10T11:14:03.461 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:03.445 7f814b269700 1 --2- 172.21.2.112:0/1167183758 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f81441b0ae0 0x7f8144108940 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:03.461 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:03.445 7f814b269700 1 --2- 172.21.2.112:0/1167183758 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f81440d1870 0x7f81441b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:03.461 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:03.445 7f814b269700 1 -- 172.21.2.112:0/1167183758 >> 172.21.2.112:0/1167183758 conn(0x7f81440ce420 msgr2=0x7f81440d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:03.461 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:03.445 7f814b269700 1 -- 172.21.2.112:0/1167183758 shutdown_connections 2021-02-10T11:14:03.461 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:03.445 7f814b269700 1 -- 172.21.2.112:0/1167183758 wait complete. 2021-02-10T11:14:03.461 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:03.459 7f814b269700 1 Processor -- start 2021-02-10T11:14:03.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:03.459 7f814b269700 1 -- start start 2021-02-10T11:14:03.463 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:03.460 7f814b269700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f81440aa9e0 0x7f814418ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:03.463 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:03.460 7f814b269700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f814418d4d0 0x7f8144098df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:03.464 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:03.460 7f814b269700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8144099410 0x7f81441a6cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:03.464 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:03.460 7f814b269700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8144098870 con 0x7f81440aa9e0 2021-02-10T11:14:03.464 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:03.460 7f814b269700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81441a72e0 con 0x7f814418d4d0 2021-02-10T11:14:03.464 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:03.460 7f814b269700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81441a74c0 con 0x7f8144099410 2021-02-10T11:14:03.464 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:03.460 7f8149085700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f81440aa9e0 0x7f814418ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:03.464 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:03.460 7f8149085700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f81440aa9e0 0x7f814418ceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:42048/0 (socket says 172.21.2.112:42048) 2021-02-10T11:14:03.464 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:03.460 7f8149085700 1 -- 172.21.2.112:0/3702158296 learned_addr learned my addr 172.21.2.112:0/3702158296 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:14:03.464 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:03.461 7f8149085700 1 -- 172.21.2.112:0/3702158296 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8144099410 msgr2=0x7f81441a6cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:03.464 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:03.461 7f8149886700 1 --2- 172.21.2.112:0/3702158296 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8144099410 0x7f81441a6cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:03.464 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:03.461 7f8148884700 1 --2- 172.21.2.112:0/3702158296 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f814418d4d0 0x7f8144098df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:03.464 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:03.461 7f8149085700 1 --2- 172.21.2.112:0/3702158296 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8144099410 0x7f81441a6cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:03.464 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:03.461 7f8149085700 1 -- 172.21.2.112:0/3702158296 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f814418d4d0 msgr2=0x7f8144098df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:03.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:03.461 7f8149085700 1 --2- 172.21.2.112:0/3702158296 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f814418d4d0 0x7f8144098df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:03.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:03.461 7f8149085700 1 -- 172.21.2.112:0/3702158296 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81441a76e0 con 0x7f81440aa9e0 2021-02-10T11:14:03.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:03.461 7f8148884700 1 --2- 172.21.2.112:0/3702158296 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f814418d4d0 0x7f8144098df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:14:03.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:03.461 7f8149886700 1 --2- 172.21.2.112:0/3702158296 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8144099410 0x7f81441a6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:14:03.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:03.461 7f8149085700 1 --2- 172.21.2.112:0/3702158296 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f81440aa9e0 0x7f814418ceb0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f8140006f20 tx=0x7f814000b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:03.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:03.461 7f8139ffb700 1 -- 172.21.2.112:0/3702158296 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f81400009b0 con 0x7f81440aa9e0 2021-02-10T11:14:03.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:03.461 7f8139ffb700 1 -- 172.21.2.112:0/3702158296 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f814000bd80 con 0x7f81440aa9e0 2021-02-10T11:14:03.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:03.461 7f814b269700 1 -- 172.21.2.112:0/3702158296 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81441a76e0 con 0x7f81440aa9e0 2021-02-10T11:14:03.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:03.461 7f8139ffb700 1 -- 172.21.2.112:0/3702158296 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f81400022b0 con 0x7f81440aa9e0 2021-02-10T11:14:03.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:03.461 7f814b269700 1 -- 172.21.2.112:0/3702158296 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8144097c90 con 0x7f81440aa9e0 2021-02-10T11:14:03.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:03.462 7f8139ffb700 1 -- 172.21.2.112:0/3702158296 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8140000c70 con 0x7f81440aa9e0 2021-02-10T11:14:03.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:03.462 7f8139ffb700 1 --2- 172.21.2.112:0/3702158296 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f81240186c0 0x7f812401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:03.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:03.462 7f8139ffb700 1 -- 172.21.2.112:0/3702158296 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5457+0+0 (secure 0 0 0) 0x7f81400325f0 con 0x7f81440aa9e0 2021-02-10T11:14:03.466 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:03.462 7f8148884700 1 --2- 172.21.2.112:0/3702158296 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f81240186c0 0x7f812401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:03.466 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:03.463 7f8148884700 1 --2- 172.21.2.112:0/3702158296 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f81240186c0 0x7f812401abc0 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:03.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:03.493 7f814b269700 1 -- 172.21.2.112:0/3702158296 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81440828f0 con 0x7f81440aa9e0 2021-02-10T11:14:03.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:03.495 7f8139ffb700 1 -- 172.21.2.112:0/3702158296 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f81400009b0 con 0x7f81440aa9e0 2021-02-10T11:14:03.624 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:03.625 7f814b269700 1 -- 172.21.2.112:0/3702158296 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"} v 0) v1 -- 0x7f8144109a60 con 0x7f81440aa9e0 2021-02-10T11:14:03.625 INFO:tasks.ceph.mon.a.gibba012.stderr:2021-02-10 11:14:03.625 7fc3fbedd700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-02-10T11:14:03.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:03.629 7f8139ffb700 1 -- 172.21.2.112:0/3702158296 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"}]=0 new fs with metadata pool 2 and data pool 3 v2) v1 ==== 187+0+0 (secure 0 0 0) 0x7f814001f520 con 0x7f81440aa9e0 2021-02-10T11:14:03.631 INFO:teuthology.orchestra.run.gibba012.stderr:new fs with metadata pool 2 and data pool 3 2021-02-10T11:14:03.633 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:03.634 7f814b269700 1 -- 172.21.2.112:0/3702158296 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f81240186c0 msgr2=0x7f812401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:03.634 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:03.634 7f814b269700 1 --2- 172.21.2.112:0/3702158296 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f81240186c0 0x7f812401abc0 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:03.634 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:03.634 7f814b269700 1 -- 172.21.2.112:0/3702158296 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f81440aa9e0 msgr2=0x7f814418ceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:03.634 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:03.634 7f814b269700 1 --2- 172.21.2.112:0/3702158296 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f81440aa9e0 0x7f814418ceb0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f8140006f20 tx=0x7f814000b990).stop 2021-02-10T11:14:03.634 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:03.634 7f814b269700 1 -- 172.21.2.112:0/3702158296 shutdown_connections 2021-02-10T11:14:03.634 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:03.634 7f814b269700 1 --2- 172.21.2.112:0/3702158296 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f81240186c0 0x7f812401abc0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:03.634 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:03.634 7f814b269700 1 --2- 172.21.2.112:0/3702158296 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f814418d4d0 0x7f8144098df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:03.634 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:03.634 7f814b269700 1 --2- 172.21.2.112:0/3702158296 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8144099410 0x7f81441a6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:03.634 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:03.634 7f814b269700 1 --2- 172.21.2.112:0/3702158296 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f81440aa9e0 0x7f814418ceb0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:03.634 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:03.634 7f814b269700 1 -- 172.21.2.112:0/3702158296 >> 172.21.2.112:0/3702158296 conn(0x7f814419af60 msgr2=0x7f814418c280 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:03.634 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:03.634 7f814b269700 1 -- 172.21.2.112:0/3702158296 shutdown_connections 2021-02-10T11:14:03.634 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:03.634 7f814b269700 1 -- 172.21.2.112:0/3702158296 wait complete. 2021-02-10T11:14:03.641 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:14:04.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.050 7f170f88a700 1 Processor -- start 2021-02-10T11:14:04.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.050 7f170f88a700 1 -- start start 2021-02-10T11:14:04.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.051 7f170f88a700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f17080e5070 0x7f17081b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:04.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.051 7f170f88a700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f17080e5470 0x7f1708080310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:04.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.051 7f170f88a700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f170807ed80 con 0x7f17081b3e40 2021-02-10T11:14:04.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.051 7f170f88a700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17080d1790 con 0x7f17080e5470 2021-02-10T11:14:04.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.051 7f170f88a700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17080d19b0 con 0x7f17080e5070 2021-02-10T11:14:04.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.051 7f170dea7700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f17080e5470 0x7f1708080310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:04.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.051 7f170cea5700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7f17081b3e40 0x7f170807d690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:35074/0 (socket says 172.21.2.112:35074) 2021-02-10T11:14:04.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.051 7f170cea5700 1 -- 172.21.2.112:0/3366504281 learned_addr learned my addr 172.21.2.112:0/3366504281 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:14:04.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.051 7f170dea7700 1 --2- 172.21.2.112:0/3366504281 >> v2:172.21.2.112:3300/0 conn(0x7f17080e5470 0x7f1708080310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:42054/0 (socket says 172.21.2.112:42054) 2021-02-10T11:14:04.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.051 7f170d6a6700 1 --2- 172.21.2.112:0/3366504281 >> v2:172.21.2.120:3300/0 conn(0x7f17080e5070 0x7f17081b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:04.068 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.052 7f170dea7700 1 -- 172.21.2.112:0/3366504281 >> v2:172.21.2.120:3300/0 conn(0x7f17080e5070 msgr2=0x7f17081b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:14:04.068 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.052 7f170dea7700 1 --2- 172.21.2.112:0/3366504281 >> v2:172.21.2.120:3300/0 conn(0x7f17080e5070 0x7f17081b3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:04.068 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.051 7f16fbfff700 1 -- 172.21.2.112:0/3366504281 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2205970143 0 0) 0x7f16fc0009b0 con 0x7f17081b3e40 2021-02-10T11:14:04.068 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.052 7f170dea7700 1 -- 172.21.2.112:0/3366504281 >> v1:172.21.2.120:6789/0 conn(0x7f17081b3e40 legacy=0x7f170807d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:04.068 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.052 7f170dea7700 1 -- 172.21.2.112:0/3366504281 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17080d1bd0 con 0x7f17080e5470 2021-02-10T11:14:04.068 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.052 7f170dea7700 1 --2- 172.21.2.112:0/3366504281 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f17080e5470 0x7f1708080310 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f1700006e30 tx=0x7f1700009e60).ready entity=mon.0 client_cookie=b0ac045d17f79c4b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:04.068 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.052 7f16fbfff700 1 -- 172.21.2.112:0/3366504281 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1700006a50 con 0x7f17080e5470 2021-02-10T11:14:04.068 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.052 7f16fbfff700 1 -- 172.21.2.112:0/3366504281 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f170000b300 con 0x7f17080e5470 2021-02-10T11:14:04.068 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.052 7f16fbfff700 1 -- 172.21.2.112:0/3366504281 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f170000b7e0 con 0x7f17080e5470 2021-02-10T11:14:04.068 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.052 7f170f88a700 1 -- 172.21.2.112:0/3366504281 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f17080e5470 msgr2=0x7f1708080310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:04.068 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.052 7f170f88a700 1 --2- 172.21.2.112:0/3366504281 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f17080e5470 0x7f1708080310 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f1700006e30 tx=0x7f1700009e60).stop 2021-02-10T11:14:04.068 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.053 7f170f88a700 1 -- 172.21.2.112:0/3366504281 shutdown_connections 2021-02-10T11:14:04.068 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.053 7f170f88a700 1 --2- 172.21.2.112:0/3366504281 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f17080e5470 0x7f1708080310 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:04.068 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.053 7f170f88a700 1 --2- 172.21.2.112:0/3366504281 >> v2:172.21.2.120:3300/0 conn(0x7f17080e5070 0x7f17081b3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:04.069 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.053 7f170f88a700 1 -- 172.21.2.112:0/3366504281 >> 172.21.2.112:0/3366504281 conn(0x7f17080ce690 msgr2=0x7f17080e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:04.069 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.053 7f170f88a700 1 -- 172.21.2.112:0/3366504281 shutdown_connections 2021-02-10T11:14:04.069 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.053 7f170f88a700 1 -- 172.21.2.112:0/3366504281 wait complete. 2021-02-10T11:14:04.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.067 7f170f88a700 1 Processor -- start 2021-02-10T11:14:04.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.068 7f170f88a700 1 -- start start 2021-02-10T11:14:04.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.068 7f170f88a700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1708191bb0 0x7f17080a2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:04.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.068 7f170f88a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f17080a29a0 0x7f17080a2da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:04.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.068 7f170f88a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f17080a33c0 0x7f17080a03d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:04.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.068 7f170f88a700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17080adf90 con 0x7f17080a33c0 2021-02-10T11:14:04.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.068 7f170f88a700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17080a1e00 con 0x7f1708191bb0 2021-02-10T11:14:04.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.068 7f170f88a700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17080a09f0 con 0x7f17080a29a0 2021-02-10T11:14:04.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.068 7f170dea7700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f17080a33c0 0x7f17080a03d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:04.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.068 7f170d6a6700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1708191bb0 0x7f17080a2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:04.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.068 7f170dea7700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f17080a33c0 0x7f17080a03d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:42062/0 (socket says 172.21.2.112:42062) 2021-02-10T11:14:04.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.068 7f170dea7700 1 -- 172.21.2.112:0/3573780937 learned_addr learned my addr 172.21.2.112:0/3573780937 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:14:04.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.069 7f170cea5700 1 --2- 172.21.2.112:0/3573780937 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f17080a29a0 0x7f17080a2da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:04.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.069 7f170d6a6700 1 -- 172.21.2.112:0/3573780937 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f17080a29a0 msgr2=0x7f17080a2da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:04.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.069 7f170d6a6700 1 --2- 172.21.2.112:0/3573780937 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f17080a29a0 0x7f17080a2da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:04.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.069 7f170d6a6700 1 -- 172.21.2.112:0/3573780937 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f17080a33c0 msgr2=0x7f17080a03d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:04.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.069 7f170d6a6700 1 --2- 172.21.2.112:0/3573780937 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f17080a33c0 0x7f17080a03d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:04.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.069 7f170d6a6700 1 -- 172.21.2.112:0/3573780937 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17080a0bd0 con 0x7f1708191bb0 2021-02-10T11:14:04.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.069 7f170d6a6700 1 --2- 172.21.2.112:0/3573780937 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1708191bb0 0x7f17080a2380 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f1704009540 tx=0x7f170400b8e0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:04.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.069 7f16f9ffb700 1 -- 172.21.2.112:0/3573780937 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f170400bcd0 con 0x7f1708191bb0 2021-02-10T11:14:04.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.069 7f16f9ffb700 1 -- 172.21.2.112:0/3573780937 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1704006740 con 0x7f1708191bb0 2021-02-10T11:14:04.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.069 7f170f88a700 1 -- 172.21.2.112:0/3573780937 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17080a0bd0 con 0x7f1708191bb0 2021-02-10T11:14:04.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.069 7f16f9ffb700 1 -- 172.21.2.112:0/3573780937 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f17040009b0 con 0x7f1708191bb0 2021-02-10T11:14:04.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.069 7f170f88a700 1 -- 172.21.2.112:0/3573780937 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17080a1160 con 0x7f1708191bb0 2021-02-10T11:14:04.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.070 7f16f9ffb700 1 -- 172.21.2.112:0/3573780937 <== mon.1 v2:172.21.2.120:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f17040052e0 con 0x7f1708191bb0 2021-02-10T11:14:04.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.070 7f16f9ffb700 1 --2- 172.21.2.112:0/3573780937 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f16e80186c0 0x7f16e801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:04.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.070 7f16f9ffb700 1 -- 172.21.2.112:0/3573780937 <== mon.1 v2:172.21.2.120:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f1704032330 con 0x7f1708191bb0 2021-02-10T11:14:04.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.071 7f170cea5700 1 --2- 172.21.2.112:0/3573780937 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f16e80186c0 0x7f16e801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:04.073 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.071 7f170cea5700 1 --2- 172.21.2.112:0/3573780937 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f16e80186c0 0x7f16e801abc0 crc :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:04.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.101 7f170f88a700 1 -- 172.21.2.112:0/3573780937 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f170807ed80 con 0x7f1708191bb0 2021-02-10T11:14:04.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.105 7f16f9ffb700 1 -- 172.21.2.112:0/3573780937 <== mon.1 v2:172.21.2.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1704000c90 con 0x7f1708191bb0 2021-02-10T11:14:04.233 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.233 7f170f88a700 1 -- 172.21.2.112:0/3573780937 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f17080adf90 con 0x7f1708191bb0 2021-02-10T11:14:04.233 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.235 7f16f9ffb700 1 -- 172.21.2.112:0/3573780937 <== mon.1 v2:172.21.2.120:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) v1 ==== 74+0+14006 (secure 0 0 0) 0x7f17040052e0 con 0x7f1708191bb0 2021-02-10T11:14:04.236 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:14:04.236 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":19,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:14:03.627598","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:14:01.400797","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2021-02-10 11:14:02.622129","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:14:04.237 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.237 7f170f88a700 1 -- 172.21.2.112:0/3573780937 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f16e80186c0 msgr2=0x7f16e801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:04.237 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.237 7f170f88a700 1 --2- 172.21.2.112:0/3573780937 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f16e80186c0 0x7f16e801abc0 crc :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:04.237 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.238 7f170f88a700 1 -- 172.21.2.112:0/3573780937 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1708191bb0 msgr2=0x7f17080a2380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:04.237 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.238 7f170f88a700 1 --2- 172.21.2.112:0/3573780937 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1708191bb0 0x7f17080a2380 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f1704009540 tx=0x7f170400b8e0).stop 2021-02-10T11:14:04.238 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.238 7f170f88a700 1 -- 172.21.2.112:0/3573780937 shutdown_connections 2021-02-10T11:14:04.238 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.238 7f170f88a700 1 --2- 172.21.2.112:0/3573780937 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f16e80186c0 0x7f16e801abc0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:04.238 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.238 7f170f88a700 1 --2- 172.21.2.112:0/3573780937 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f17080a33c0 0x7f17080a03d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:04.238 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.238 7f170f88a700 1 --2- 172.21.2.112:0/3573780937 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f17080a29a0 0x7f17080a2da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:04.238 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.238 7f170f88a700 1 --2- 172.21.2.112:0/3573780937 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1708191bb0 0x7f17080a2380 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:04.238 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.238 7f170f88a700 1 -- 172.21.2.112:0/3573780937 >> 172.21.2.112:0/3573780937 conn(0x7f17081a31d0 msgr2=0x7f170808b830 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:04.238 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.238 7f170f88a700 1 -- 172.21.2.112:0/3573780937 shutdown_connections 2021-02-10T11:14:04.238 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.238 7f170f88a700 1 -- 172.21.2.112:0/3573780937 wait complete. 2021-02-10T11:14:04.246 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:14:04.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.654 7eff7f1e6700 1 Processor -- start 2021-02-10T11:14:04.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.655 7eff7f1e6700 1 -- start start 2021-02-10T11:14:04.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.655 7eff7f1e6700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7eff780e5070 0x7eff781b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:04.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.655 7eff7f1e6700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7eff781b3e40 0x7eff7807d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:04.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.655 7eff7f1e6700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff7807ed80 con 0x7eff780e5470 2021-02-10T11:14:04.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.655 7eff7f1e6700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff78080310 con 0x7eff780e5070 2021-02-10T11:14:04.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.655 7eff7f1e6700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff78080500 con 0x7eff781b3e40 2021-02-10T11:14:04.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.656 7eff7c801700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7eff781b3e40 0x7eff7807d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:04.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.656 7eff7c801700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7eff781b3e40 0x7eff7807d690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3301/0 says I am v2:172.21.2.112:41748/0 (socket says 172.21.2.112:41748) 2021-02-10T11:14:04.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.656 7eff7c801700 1 -- 172.21.2.112:0/1750530130 learned_addr learned my addr 172.21.2.112:0/1750530130 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:14:04.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.656 7eff7d002700 1 --2- 172.21.2.112:0/1750530130 >> v2:172.21.2.120:3300/0 conn(0x7eff780e5070 0x7eff781b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:04.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.656 7eff7d803700 1 --1- 172.21.2.112:0/1750530130 >> v1:172.21.2.120:6789/0 conn(0x7eff780e5470 0x7eff7807dcb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:35086/0 (socket says 172.21.2.112:35086) 2021-02-10T11:14:04.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.656 7eff7c801700 1 -- 172.21.2.112:0/1750530130 >> v2:172.21.2.120:3300/0 conn(0x7eff780e5070 msgr2=0x7eff781b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:14:04.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.656 7eff7c801700 1 --2- 172.21.2.112:0/1750530130 >> v2:172.21.2.120:3300/0 conn(0x7eff780e5070 0x7eff781b3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:04.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.656 7eff7c801700 1 -- 172.21.2.112:0/1750530130 >> v1:172.21.2.120:6789/0 conn(0x7eff780e5470 legacy=0x7eff7807dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:04.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.656 7eff7c801700 1 -- 172.21.2.112:0/1750530130 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff780806f0 con 0x7eff781b3e40 2021-02-10T11:14:04.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.656 7eff7c801700 1 --2- 172.21.2.112:0/1750530130 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7eff781b3e40 0x7eff7807d690 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7eff68006700 tx=0x7eff68009b80).ready entity=mon.2 client_cookie=95b4fe18e45aa355 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:04.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.657 7eff6ffff700 1 -- 172.21.2.112:0/1750530130 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7eff6800c0c0 con 0x7eff781b3e40 2021-02-10T11:14:04.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.657 7eff6ffff700 1 -- 172.21.2.112:0/1750530130 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff6800c440 con 0x7eff781b3e40 2021-02-10T11:14:04.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.657 7eff6ffff700 1 -- 172.21.2.112:0/1750530130 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7eff6800c960 con 0x7eff781b3e40 2021-02-10T11:14:04.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.657 7eff7f1e6700 1 -- 172.21.2.112:0/1750530130 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7eff781b3e40 msgr2=0x7eff7807d690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:04.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.657 7eff7f1e6700 1 --2- 172.21.2.112:0/1750530130 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7eff781b3e40 0x7eff7807d690 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7eff68006700 tx=0x7eff68009b80).stop 2021-02-10T11:14:04.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.657 7eff7f1e6700 1 -- 172.21.2.112:0/1750530130 shutdown_connections 2021-02-10T11:14:04.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.657 7eff7f1e6700 1 --2- 172.21.2.112:0/1750530130 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7eff781b3e40 0x7eff7807d690 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:04.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.657 7eff7f1e6700 1 --2- 172.21.2.112:0/1750530130 >> v2:172.21.2.120:3300/0 conn(0x7eff780e5070 0x7eff781b3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:04.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.657 7eff7f1e6700 1 -- 172.21.2.112:0/1750530130 >> 172.21.2.112:0/1750530130 conn(0x7eff780ce690 msgr2=0x7eff780e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:04.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.657 7eff7f1e6700 1 -- 172.21.2.112:0/1750530130 shutdown_connections 2021-02-10T11:14:04.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.658 7eff7f1e6700 1 -- 172.21.2.112:0/1750530130 wait complete. 2021-02-10T11:14:04.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.672 7eff7f1e6700 1 Processor -- start 2021-02-10T11:14:04.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.672 7eff7f1e6700 1 -- start start 2021-02-10T11:14:04.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.673 7eff7f1e6700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7eff7808d400 0x7eff78096c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:04.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.673 7eff7f1e6700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7eff780b1450 0x7eff780b1850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:04.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.673 7eff7f1e6700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7eff780b1e70 0x7eff781b1f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:04.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.673 7eff7f1e6700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff78097310 con 0x7eff780b1e70 2021-02-10T11:14:04.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.673 7eff7f1e6700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff7808d800 con 0x7eff7808d400 2021-02-10T11:14:04.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.673 7eff7f1e6700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff7808d9f0 con 0x7eff780b1450 2021-02-10T11:14:04.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.673 7eff7c801700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7eff780b1450 0x7eff780b1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:04.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.673 7eff7c801700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7eff780b1450 0x7eff780b1850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3301/0 says I am v2:172.21.2.112:41752/0 (socket says 172.21.2.112:41752) 2021-02-10T11:14:04.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.673 7eff7c801700 1 -- 172.21.2.112:0/1291767567 learned_addr learned my addr 172.21.2.112:0/1291767567 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:14:04.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.673 7eff7d803700 1 --2- 172.21.2.112:0/1291767567 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7eff780b1e70 0x7eff781b1f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:04.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.673 7eff7d002700 1 --2- 172.21.2.112:0/1291767567 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7eff7808d400 0x7eff78096c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:04.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.673 7eff7c801700 1 -- 172.21.2.112:0/1291767567 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7eff7808d400 msgr2=0x7eff78096c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:04.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.673 7eff7c801700 1 --2- 172.21.2.112:0/1291767567 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7eff7808d400 0x7eff78096c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:04.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.673 7eff7c801700 1 -- 172.21.2.112:0/1291767567 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7eff780b1e70 msgr2=0x7eff781b1f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:04.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.673 7eff7c801700 1 --2- 172.21.2.112:0/1291767567 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7eff780b1e70 0x7eff781b1f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:04.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.673 7eff7c801700 1 -- 172.21.2.112:0/1291767567 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff781b25f0 con 0x7eff780b1450 2021-02-10T11:14:04.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.673 7eff7c801700 1 --2- 172.21.2.112:0/1291767567 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7eff780b1450 0x7eff780b1850 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7eff6800b5e0 tx=0x7eff6800b980).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:04.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.674 7eff6dffb700 1 -- 172.21.2.112:0/1291767567 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7eff6800bd70 con 0x7eff780b1450 2021-02-10T11:14:04.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.674 7eff6dffb700 1 -- 172.21.2.112:0/1291767567 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff68002030 con 0x7eff780b1450 2021-02-10T11:14:04.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.674 7eff7f1e6700 1 -- 172.21.2.112:0/1291767567 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff781b25f0 con 0x7eff780b1450 2021-02-10T11:14:04.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.674 7eff7f1e6700 1 -- 172.21.2.112:0/1291767567 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff781b2b80 con 0x7eff780b1450 2021-02-10T11:14:04.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.674 7eff6dffb700 1 -- 172.21.2.112:0/1291767567 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7eff68002550 con 0x7eff780b1450 2021-02-10T11:14:04.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.674 7eff6dffb700 1 -- 172.21.2.112:0/1291767567 <== mon.2 v2:172.21.2.120:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7eff68000c90 con 0x7eff780b1450 2021-02-10T11:14:04.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.674 7eff6dffb700 1 --2- 172.21.2.112:0/1291767567 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7eff580186c0 0x7eff5801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:04.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.674 7eff6dffb700 1 -- 172.21.2.112:0/1291767567 <== mon.2 v2:172.21.2.120:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7eff68032540 con 0x7eff780b1450 2021-02-10T11:14:04.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.675 7eff7d002700 1 --2- 172.21.2.112:0/1291767567 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7eff580186c0 0x7eff5801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:04.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.675 7eff7d002700 1 --2- 172.21.2.112:0/1291767567 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7eff580186c0 0x7eff5801abc0 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:04.704 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.705 7eff7f1e6700 1 -- 172.21.2.112:0/1291767567 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff7807ed80 con 0x7eff780b1450 2021-02-10T11:14:04.707 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.708 7eff6dffb700 1 -- 172.21.2.112:0/1291767567 <== mon.2 v2:172.21.2.120:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7eff68000940 con 0x7eff780b1450 2021-02-10T11:14:04.837 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.837 7eff7f1e6700 1 -- 172.21.2.112:0/1291767567 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7eff7807ed80 con 0x7eff780b1450 2021-02-10T11:14:04.838 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.838 7eff6dffb700 1 -- 172.21.2.112:0/1291767567 <== mon.2 v2:172.21.2.120:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+13993 (secure 0 0 0) 0x7eff68000c90 con 0x7eff780b1450 2021-02-10T11:14:04.838 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:14:04.838 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":20,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:14:04.629689","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:14:01.400797","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2021-02-10 11:14:02.622129","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:14:04.838 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.839 7eff7f1e6700 1 -- 172.21.2.112:0/1291767567 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7eff580186c0 msgr2=0x7eff5801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:04.838 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.839 7eff7f1e6700 1 --2- 172.21.2.112:0/1291767567 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7eff580186c0 0x7eff5801abc0 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:04.839 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.839 7eff7f1e6700 1 -- 172.21.2.112:0/1291767567 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7eff780b1450 msgr2=0x7eff780b1850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:04.839 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.839 7eff7f1e6700 1 --2- 172.21.2.112:0/1291767567 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7eff780b1450 0x7eff780b1850 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7eff6800b5e0 tx=0x7eff6800b980).stop 2021-02-10T11:14:04.839 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.839 7eff7f1e6700 1 -- 172.21.2.112:0/1291767567 shutdown_connections 2021-02-10T11:14:04.839 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.839 7eff7f1e6700 1 --2- 172.21.2.112:0/1291767567 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7eff580186c0 0x7eff5801abc0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:04.839 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.839 7eff7f1e6700 1 --2- 172.21.2.112:0/1291767567 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7eff780b1e70 0x7eff781b1f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:04.840 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.839 7eff7f1e6700 1 --2- 172.21.2.112:0/1291767567 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7eff780b1450 0x7eff780b1850 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:04.840 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.839 7eff7f1e6700 1 --2- 172.21.2.112:0/1291767567 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7eff7808d400 0x7eff78096c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:04.840 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.839 7eff7f1e6700 1 -- 172.21.2.112:0/1291767567 >> 172.21.2.112:0/1291767567 conn(0x7eff781ac620 msgr2=0x7eff780896e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:04.840 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.840 7eff7f1e6700 1 -- 172.21.2.112:0/1291767567 shutdown_connections 2021-02-10T11:14:04.840 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:04.840 7eff7f1e6700 1 -- 172.21.2.112:0/1291767567 wait complete. 2021-02-10T11:14:04.848 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2021-02-10T11:14:05.278 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:05.260 7f1d7ebf4700 1 Processor -- start 2021-02-10T11:14:05.278 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:05.260 7f1d7ebf4700 1 -- start start 2021-02-10T11:14:05.278 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:05.261 7f1d7ebf4700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f1d781b0870 0x7f1d780f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:05.279 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:05.261 7f1d7ebf4700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1d780efe00 0x7f1d780f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:05.279 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:05.261 7f1d7ebf4700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d781b0c70 con 0x7f1d781b0470 2021-02-10T11:14:05.279 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:05.261 7f1d7ebf4700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d780f0b50 con 0x7f1d781b0870 2021-02-10T11:14:05.279 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:05.261 7f1d7ebf4700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d780f0820 con 0x7f1d780efe00 2021-02-10T11:14:05.279 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:05.261 7f1d7ca10700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f1d781b0470 0x7f1d780ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:41540/0 (socket says 172.21.2.112:41540) 2021-02-10T11:14:05.279 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:05.261 7f1d7ca10700 1 -- 172.21.2.112:0/3584335989 learned_addr learned my addr 172.21.2.112:0/3584335989 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:14:05.279 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:05.261 7f1d777fe700 1 -- 172.21.2.112:0/3584335989 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2931851908 0 0) 0x7f1d60001180 con 0x7f1d781b0470 2021-02-10T11:14:05.279 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:05.262 7f1d777fe700 1 -- 172.21.2.112:0/3584335989 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d64002fe0 con 0x7f1d781b0470 2021-02-10T11:14:05.279 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:05.262 7f1d7d211700 1 --2- 172.21.2.112:0/3584335989 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1d780efe00 0x7f1d780f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:05.279 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:05.262 7f1d77fff700 1 --2- 172.21.2.112:0/3584335989 >> v2:172.21.2.120:3300/0 conn(0x7f1d781b0870 0x7f1d780f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:05.279 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:05.262 7f1d777fe700 1 -- 172.21.2.112:0/3584335989 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 843212856 0 0) 0x7f1d60000ed0 con 0x7f1d781b0470 2021-02-10T11:14:05.279 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:05.262 7f1d777fe700 1 -- 172.21.2.112:0/3584335989 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1d780efe00 msgr2=0x7f1d780f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:14:05.279 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:05.262 7f1d777fe700 1 --2- 172.21.2.112:0/3584335989 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1d780efe00 0x7f1d780f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:05.280 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:05.262 7f1d777fe700 1 -- 172.21.2.112:0/3584335989 >> v2:172.21.2.120:3300/0 conn(0x7f1d781b0870 msgr2=0x7f1d780f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:14:05.280 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:05.262 7f1d777fe700 1 --2- 172.21.2.112:0/3584335989 >> v2:172.21.2.120:3300/0 conn(0x7f1d781b0870 0x7f1d780f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:05.280 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:05.262 7f1d777fe700 1 -- 172.21.2.112:0/3584335989 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d780e52e0 con 0x7f1d781b0470 2021-02-10T11:14:05.280 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:05.262 7f1d777fe700 1 -- 172.21.2.112:0/3584335989 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f1d600015e0 con 0x7f1d781b0470 2021-02-10T11:14:05.280 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:05.262 7f1d777fe700 1 -- 172.21.2.112:0/3584335989 >> v1:172.21.2.112:6789/0 conn(0x7f1d781b0470 legacy=0x7f1d780ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:05.280 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:05.262 7f1d777fe700 1 --2- 172.21.2.112:0/3584335989 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1d64003af0 0x7f1d64005f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:05.280 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:05.262 7f1d777fe700 1 --2- 172.21.2.112:0/3584335989 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1d64006800 0x7f1d64008d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:05.280 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:05.262 7f1d777fe700 1 --2- 172.21.2.112:0/3584335989 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1d640095a0 0x7f1d6400bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:05.280 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:05.262 7f1d777fe700 1 -- 172.21.2.112:0/3584335989 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d6400c370 con 0x7f1d64006800 2021-02-10T11:14:05.280 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:05.262 7f1d777fe700 1 -- 172.21.2.112:0/3584335989 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d64050560 con 0x7f1d640095a0 2021-02-10T11:14:05.280 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:05.262 7f1d777fe700 1 -- 172.21.2.112:0/3584335989 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d64050780 con 0x7f1d64003af0 2021-02-10T11:14:05.280 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:05.262 7f1d77fff700 1 --2- 172.21.2.112:0/3584335989 >> v2:172.21.2.120:3300/0 conn(0x7f1d781b0870 0x7f1d780f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:14:05.281 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:05.262 7f1d77fff700 1 --2- 172.21.2.112:0/3584335989 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1d64006800 0x7f1d64008d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:05.281 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:05.262 7f1d7d211700 1 --2- 172.21.2.112:0/3584335989 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1d640095a0 0x7f1d6400bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:05.281 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:05.262 7f1d7ca10700 1 --2- 172.21.2.112:0/3584335989 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1d64003af0 0x7f1d64005f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:05.281 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:05.263 7f1d77fff700 1 -- 172.21.2.112:0/3584335989 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1d64003af0 msgr2=0x7f1d64005f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:14:05.281 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:05.263 7f1d77fff700 1 --2- 172.21.2.112:0/3584335989 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1d64003af0 0x7f1d64005f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:05.281 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:05.263 7f1d77fff700 1 -- 172.21.2.112:0/3584335989 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1d640095a0 msgr2=0x7f1d6400bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:14:05.281 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:05.263 7f1d77fff700 1 --2- 172.21.2.112:0/3584335989 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1d640095a0 0x7f1d6400bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:05.281 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:05.263 7f1d77fff700 1 -- 172.21.2.112:0/3584335989 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d64050a20 con 0x7f1d64006800 2021-02-10T11:14:05.281 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:05.263 7f1d77fff700 1 --2- 172.21.2.112:0/3584335989 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1d64006800 0x7f1d64008d20 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f1d68006fa0 tx=0x7f1d6800b300).ready entity=mon.0 client_cookie=f204c246e4ce47e9 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:05.281 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:05.263 7f1d777fe700 1 -- 172.21.2.112:0/3584335989 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1d6800b6f0 con 0x7f1d64006800 2021-02-10T11:14:05.281 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:05.263 7f1d777fe700 1 -- 172.21.2.112:0/3584335989 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d6800b970 con 0x7f1d64006800 2021-02-10T11:14:05.281 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:05.263 7f1d777fe700 1 -- 172.21.2.112:0/3584335989 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1d6800bdd0 con 0x7f1d64006800 2021-02-10T11:14:05.281 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:05.263 7f1d7ebf4700 1 -- 172.21.2.112:0/3584335989 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1d64006800 msgr2=0x7f1d64008d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:05.282 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:05.263 7f1d7ebf4700 1 --2- 172.21.2.112:0/3584335989 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1d64006800 0x7f1d64008d20 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f1d68006fa0 tx=0x7f1d6800b300).stop 2021-02-10T11:14:05.282 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:05.263 7f1d7d211700 1 -- 172.21.2.112:0/3584335989 reap_dead start 2021-02-10T11:14:05.282 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:05.264 7f1d7ebf4700 1 -- 172.21.2.112:0/3584335989 shutdown_connections 2021-02-10T11:14:05.282 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:05.264 7f1d7ebf4700 1 -- 172.21.2.112:0/3584335989 >> 172.21.2.112:0/3584335989 conn(0x7f1d780ce3e0 msgr2=0x7f1d780e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:05.282 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:05.264 7f1d7ebf4700 1 -- 172.21.2.112:0/3584335989 shutdown_connections 2021-02-10T11:14:05.282 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:05.264 7f1d7ebf4700 1 -- 172.21.2.112:0/3584335989 wait complete. 2021-02-10T11:14:05.282 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:05.279 7f1d7ebf4700 1 Processor -- start 2021-02-10T11:14:05.282 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:05.279 7f1d7ebf4700 1 -- start start 2021-02-10T11:14:05.283 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:05.279 7f1d7ebf4700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1d7809e560 0x7f1d780a0130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:05.283 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:05.279 7f1d7ebf4700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1d7809e960 0x7f1d780a0750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:05.283 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:05.279 7f1d7ebf4700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1d780a0d70 0x7f1d780a1af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:05.284 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:05.279 7f1d7ebf4700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d780c38f0 con 0x7f1d7809e560 2021-02-10T11:14:05.284 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:05.279 7f1d7ebf4700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d780a1170 con 0x7f1d7809e960 2021-02-10T11:14:05.284 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:05.279 7f1d7ebf4700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d780a1390 con 0x7f1d780a0d70 2021-02-10T11:14:05.284 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:05.279 7f1d7ca10700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1d7809e560 0x7f1d780a0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:05.284 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:05.279 7f1d7ca10700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1d7809e560 0x7f1d780a0130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:42090/0 (socket says 172.21.2.112:42090) 2021-02-10T11:14:05.284 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:05.279 7f1d7ca10700 1 -- 172.21.2.112:0/4044664846 learned_addr learned my addr 172.21.2.112:0/4044664846 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:14:05.284 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:05.280 7f1d7ca10700 1 -- 172.21.2.112:0/4044664846 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1d780a0d70 msgr2=0x7f1d780a1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:05.284 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:05.280 7f1d7ca10700 1 --2- 172.21.2.112:0/4044664846 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1d780a0d70 0x7f1d780a1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:05.284 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:05.280 7f1d7ca10700 1 -- 172.21.2.112:0/4044664846 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1d7809e960 msgr2=0x7f1d780a0750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:05.284 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:05.280 7f1d7ca10700 1 --2- 172.21.2.112:0/4044664846 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1d7809e960 0x7f1d780a0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:05.284 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:05.280 7f1d7ca10700 1 -- 172.21.2.112:0/4044664846 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d780a2110 con 0x7f1d7809e560 2021-02-10T11:14:05.284 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:05.280 7f1d7ca10700 1 --2- 172.21.2.112:0/4044664846 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1d7809e560 0x7f1d780a0130 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f1d60001e10 tx=0x7f1d6000a950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:05.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:05.280 7f1d757fa700 1 -- 172.21.2.112:0/4044664846 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1d60001170 con 0x7f1d7809e560 2021-02-10T11:14:05.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:05.280 7f1d757fa700 1 -- 172.21.2.112:0/4044664846 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d600013d0 con 0x7f1d7809e560 2021-02-10T11:14:05.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:05.280 7f1d757fa700 1 -- 172.21.2.112:0/4044664846 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1d600070d0 con 0x7f1d7809e560 2021-02-10T11:14:05.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:05.280 7f1d7ebf4700 1 -- 172.21.2.112:0/4044664846 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d780a2110 con 0x7f1d7809e560 2021-02-10T11:14:05.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:05.280 7f1d7ebf4700 1 -- 172.21.2.112:0/4044664846 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d780a26a0 con 0x7f1d7809e560 2021-02-10T11:14:05.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:05.281 7f1d757fa700 1 -- 172.21.2.112:0/4044664846 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1d600043a0 con 0x7f1d7809e560 2021-02-10T11:14:05.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:05.281 7f1d757fa700 1 --2- 172.21.2.112:0/4044664846 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1d5c0186c0 0x7f1d5c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:05.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:05.281 7f1d757fa700 1 -- 172.21.2.112:0/4044664846 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f1d60032560 con 0x7f1d7809e560 2021-02-10T11:14:05.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:05.281 7f1d77fff700 1 --2- 172.21.2.112:0/4044664846 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1d5c0186c0 0x7f1d5c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:05.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:05.282 7f1d77fff700 1 --2- 172.21.2.112:0/4044664846 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1d5c0186c0 0x7f1d5c01abc0 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:05.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:05.311 7f1d7ebf4700 1 -- 172.21.2.112:0/4044664846 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d78105b30 con 0x7f1d7809e560 2021-02-10T11:14:05.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:05.313 7f1d757fa700 1 -- 172.21.2.112:0/4044664846 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1d60030d10 con 0x7f1d7809e560 2021-02-10T11:14:05.448 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:05.449 7f1d7ebf4700 1 -- 172.21.2.112:0/4044664846 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"} v 0) v1 -- 0x7f1d780f0b50 con 0x7f1d7809e560 2021-02-10T11:14:05.988 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:05.989 7f1d757fa700 1 -- 172.21.2.112:0/4044664846 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"}]=0 v3) v1 ==== 118+0+0 (secure 0 0 0) 0x7f1d600043a0 con 0x7f1d7809e560 2021-02-10T11:14:06.018 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:06.018 7f1d7ebf4700 1 -- 172.21.2.112:0/4044664846 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1d5c0186c0 msgr2=0x7f1d5c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:06.018 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:06.018 7f1d7ebf4700 1 --2- 172.21.2.112:0/4044664846 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1d5c0186c0 0x7f1d5c01abc0 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:06.018 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:06.018 7f1d7ebf4700 1 -- 172.21.2.112:0/4044664846 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1d7809e560 msgr2=0x7f1d780a0130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:06.018 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:06.018 7f1d7ebf4700 1 --2- 172.21.2.112:0/4044664846 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1d7809e560 0x7f1d780a0130 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f1d60001e10 tx=0x7f1d6000a950).stop 2021-02-10T11:14:06.019 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:06.018 7f1d7ebf4700 1 -- 172.21.2.112:0/4044664846 shutdown_connections 2021-02-10T11:14:06.019 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:06.018 7f1d7ebf4700 1 --2- 172.21.2.112:0/4044664846 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1d5c0186c0 0x7f1d5c01abc0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:06.019 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:06.018 7f1d7ebf4700 1 --2- 172.21.2.112:0/4044664846 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1d7809e960 0x7f1d780a0750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:06.019 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:06.018 7f1d7ebf4700 1 --2- 172.21.2.112:0/4044664846 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1d780a0d70 0x7f1d780a1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:06.019 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:06.018 7f1d7ebf4700 1 --2- 172.21.2.112:0/4044664846 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1d7809e560 0x7f1d780a0130 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:06.019 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:06.018 7f1d7ebf4700 1 -- 172.21.2.112:0/4044664846 >> 172.21.2.112:0/4044664846 conn(0x7f1d7809fc10 msgr2=0x7f1d78094ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:06.019 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:06.019 7f1d7ebf4700 1 -- 172.21.2.112:0/4044664846 shutdown_connections 2021-02-10T11:14:06.019 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:06.019 7f1d7ebf4700 1 -- 172.21.2.112:0/4044664846 wait complete. 2021-02-10T11:14:06.026 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:14:06.453 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:06.436 7fb9bc2c0700 1 Processor -- start 2021-02-10T11:14:06.453 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:06.436 7fb9bc2c0700 1 -- start start 2021-02-10T11:14:06.453 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:06.436 7fb9bc2c0700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fb9b40e5470 0x7fb9b407e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:06.453 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:06.436 7fb9bc2c0700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fb9b407f300 0x7fb9b407f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:06.453 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:06.436 7fb9bc2c0700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9b407ed80 con 0x7fb9b40e5070 2021-02-10T11:14:06.453 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:06.436 7fb9bc2c0700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9b41b4010 con 0x7fb9b40e5470 2021-02-10T11:14:06.453 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:06.436 7fb9bc2c0700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9b40d1790 con 0x7fb9b407f300 2021-02-10T11:14:06.453 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:06.436 7fb9b98db700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fb9b40e5470 0x7fb9b407e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:06.454 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:06.436 7fb9b98db700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fb9b40e5470 0x7fb9b407e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:42096/0 (socket says 172.21.2.112:42096) 2021-02-10T11:14:06.454 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:06.436 7fb9b98db700 1 -- 172.21.2.112:0/175809885 learned_addr learned my addr 172.21.2.112:0/175809885 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:14:06.454 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:06.437 7fb9ba8dd700 1 --2- 172.21.2.112:0/175809885 >> v2:172.21.2.120:3300/0 conn(0x7fb9b407f300 0x7fb9b407f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:06.454 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:06.437 7fb9b98db700 1 -- 172.21.2.112:0/175809885 >> v2:172.21.2.120:3300/0 conn(0x7fb9b407f300 msgr2=0x7fb9b407f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:14:06.454 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:06.437 7fb9b98db700 1 --2- 172.21.2.112:0/175809885 >> v2:172.21.2.120:3300/0 conn(0x7fb9b407f300 0x7fb9b407f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:06.454 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:06.437 7fb9b98db700 1 -- 172.21.2.112:0/175809885 >> v1:172.21.2.120:6789/0 conn(0x7fb9b40e5070 legacy=0x7fb9b41b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:06.454 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:06.437 7fb9b98db700 1 -- 172.21.2.112:0/175809885 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9b40d19b0 con 0x7fb9b40e5470 2021-02-10T11:14:06.454 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:06.437 7fb9b98db700 1 --2- 172.21.2.112:0/175809885 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb9b40e5470 0x7fb9b407e6a0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7fb9a4006700 tx=0x7fb9a4009b80).ready entity=mon.0 client_cookie=2df3f5bb9f44ad22 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:06.454 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:06.437 7fb9b90da700 1 -- 172.21.2.112:0/175809885 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb9a400c0c0 con 0x7fb9b40e5470 2021-02-10T11:14:06.454 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:06.437 7fb9b90da700 1 -- 172.21.2.112:0/175809885 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9a400c440 con 0x7fb9b40e5470 2021-02-10T11:14:06.454 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:06.437 7fb9b90da700 1 -- 172.21.2.112:0/175809885 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb9a400c960 con 0x7fb9b40e5470 2021-02-10T11:14:06.454 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:06.438 7fb9bc2c0700 1 -- 172.21.2.112:0/175809885 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb9b40e5470 msgr2=0x7fb9b407e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:06.455 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:06.438 7fb9bc2c0700 1 --2- 172.21.2.112:0/175809885 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb9b40e5470 0x7fb9b407e6a0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7fb9a4006700 tx=0x7fb9a4009b80).stop 2021-02-10T11:14:06.455 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:06.438 7fb9bc2c0700 1 -- 172.21.2.112:0/175809885 shutdown_connections 2021-02-10T11:14:06.455 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:06.438 7fb9bc2c0700 1 --2- 172.21.2.112:0/175809885 >> v2:172.21.2.120:3300/0 conn(0x7fb9b407f300 0x7fb9b407f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:06.455 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:06.438 7fb9bc2c0700 1 --2- 172.21.2.112:0/175809885 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb9b40e5470 0x7fb9b407e6a0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:06.455 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:06.438 7fb9bc2c0700 1 -- 172.21.2.112:0/175809885 >> 172.21.2.112:0/175809885 conn(0x7fb9b40ce690 msgr2=0x7fb9b40e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:06.455 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:06.438 7fb9bc2c0700 1 -- 172.21.2.112:0/175809885 shutdown_connections 2021-02-10T11:14:06.455 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:06.438 7fb9bc2c0700 1 -- 172.21.2.112:0/175809885 wait complete. 2021-02-10T11:14:06.457 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:06.454 7fb9bc2c0700 1 Processor -- start 2021-02-10T11:14:06.457 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:06.454 7fb9bc2c0700 1 -- start start 2021-02-10T11:14:06.457 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:06.454 7fb9bc2c0700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb9b4088a60 0x7fb9b40ad410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:06.457 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:06.454 7fb9bc2c0700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb9b40ada30 0x7fb9b41af190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:06.458 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:06.454 7fb9bc2c0700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb9b41a3460 0x7fb9b41af7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:06.458 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:06.454 7fb9bc2c0700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9b41a2ee0 con 0x7fb9b4088a60 2021-02-10T11:14:06.458 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:06.454 7fb9bc2c0700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9b41afdd0 con 0x7fb9b40ada30 2021-02-10T11:14:06.458 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:06.454 7fb9bc2c0700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9b40885c0 con 0x7fb9b41a3460 2021-02-10T11:14:06.458 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:06.454 7fb9ba0dc700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb9b4088a60 0x7fb9b40ad410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:06.458 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:06.454 7fb9ba0dc700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb9b4088a60 0x7fb9b40ad410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:42100/0 (socket says 172.21.2.112:42100) 2021-02-10T11:14:06.458 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:06.454 7fb9ba0dc700 1 -- 172.21.2.112:0/573087810 learned_addr learned my addr 172.21.2.112:0/573087810 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:14:06.458 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:06.455 7fb9b98db700 1 --2- 172.21.2.112:0/573087810 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb9b40ada30 0x7fb9b41af190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:06.458 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:06.455 7fb9ba8dd700 1 --2- 172.21.2.112:0/573087810 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb9b41a3460 0x7fb9b41af7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:06.458 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:06.455 7fb9ba0dc700 1 -- 172.21.2.112:0/573087810 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb9b41a3460 msgr2=0x7fb9b41af7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:06.458 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:06.455 7fb9ba0dc700 1 --2- 172.21.2.112:0/573087810 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb9b41a3460 0x7fb9b41af7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:06.458 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:06.455 7fb9ba0dc700 1 -- 172.21.2.112:0/573087810 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb9b40ada30 msgr2=0x7fb9b41af190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:06.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:06.455 7fb9ba0dc700 1 --2- 172.21.2.112:0/573087810 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb9b40ada30 0x7fb9b41af190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:06.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:06.455 7fb9ba0dc700 1 -- 172.21.2.112:0/573087810 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9b41a1a20 con 0x7fb9b4088a60 2021-02-10T11:14:06.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:06.455 7fb9ba8dd700 1 --2- 172.21.2.112:0/573087810 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb9b41a3460 0x7fb9b41af7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:14:06.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:06.455 7fb9ba0dc700 1 --2- 172.21.2.112:0/573087810 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb9b4088a60 0x7fb9b40ad410 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7fb9b0006ed0 tx=0x7fb9b000b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:06.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:06.455 7fb9aaffd700 1 -- 172.21.2.112:0/573087810 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb9b0006020 con 0x7fb9b4088a60 2021-02-10T11:14:06.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:06.455 7fb9aaffd700 1 -- 172.21.2.112:0/573087810 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9b00063a0 con 0x7fb9b4088a60 2021-02-10T11:14:06.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:06.455 7fb9aaffd700 1 -- 172.21.2.112:0/573087810 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb9b0005730 con 0x7fb9b4088a60 2021-02-10T11:14:06.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:06.455 7fb9bc2c0700 1 -- 172.21.2.112:0/573087810 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9b41a1a20 con 0x7fb9b4088a60 2021-02-10T11:14:06.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:06.455 7fb9bc2c0700 1 -- 172.21.2.112:0/573087810 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9b41a1fb0 con 0x7fb9b4088a60 2021-02-10T11:14:06.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:06.456 7fb9aaffd700 1 -- 172.21.2.112:0/573087810 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb9b0006020 con 0x7fb9b4088a60 2021-02-10T11:14:06.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:06.456 7fb9aaffd700 1 --2- 172.21.2.112:0/573087810 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb99c0186c0 0x7fb99c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:06.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:06.456 7fb9aaffd700 1 -- 172.21.2.112:0/573087810 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fb9b00323d0 con 0x7fb9b4088a60 2021-02-10T11:14:06.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:06.456 7fb9b98db700 1 --2- 172.21.2.112:0/573087810 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb99c0186c0 0x7fb99c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:06.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:06.456 7fb9b98db700 1 --2- 172.21.2.112:0/573087810 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb99c0186c0 0x7fb99c01abc0 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:06.485 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:06.486 7fb9bc2c0700 1 -- 172.21.2.112:0/573087810 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9b4107180 con 0x7fb9b4088a60 2021-02-10T11:14:06.488 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:06.488 7fb9aaffd700 1 -- 172.21.2.112:0/573087810 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb9b0030d00 con 0x7fb9b4088a60 2021-02-10T11:14:06.616 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:06.617 7fb9bc2c0700 1 -- 172.21.2.112:0/573087810 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb9b407ed80 con 0x7fb9b4088a60 2021-02-10T11:14:06.617 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:06.617 7fb9aaffd700 1 -- 172.21.2.112:0/573087810 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+1415 (secure 0 0 0) 0x7fb9b000bd50 con 0x7fb9b4088a60 2021-02-10T11:14:06.617 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 3 2021-02-10T11:14:06.618 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:14:06.618 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":3,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":3,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:14:03.627691","modified":"2021-02-10 11:14:05.984391","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[],"up":{},"failed":[],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2021-02-10T11:14:06.619 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:06.619 7fb9bc2c0700 1 -- 172.21.2.112:0/573087810 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb99c0186c0 msgr2=0x7fb99c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:06.619 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:06.619 7fb9bc2c0700 1 --2- 172.21.2.112:0/573087810 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb99c0186c0 0x7fb99c01abc0 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:06.619 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:06.619 7fb9bc2c0700 1 -- 172.21.2.112:0/573087810 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb9b4088a60 msgr2=0x7fb9b40ad410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:06.619 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:06.619 7fb9bc2c0700 1 --2- 172.21.2.112:0/573087810 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb9b4088a60 0x7fb9b40ad410 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7fb9b0006ed0 tx=0x7fb9b000b960).stop 2021-02-10T11:14:06.619 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:06.619 7fb9bc2c0700 1 -- 172.21.2.112:0/573087810 shutdown_connections 2021-02-10T11:14:06.619 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:06.619 7fb9bc2c0700 1 --2- 172.21.2.112:0/573087810 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb99c0186c0 0x7fb99c01abc0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:06.619 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:06.619 7fb9bc2c0700 1 --2- 172.21.2.112:0/573087810 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb9b40ada30 0x7fb9b41af190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:06.620 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:06.619 7fb9bc2c0700 1 --2- 172.21.2.112:0/573087810 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb9b41a3460 0x7fb9b41af7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:06.620 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:06.619 7fb9bc2c0700 1 --2- 172.21.2.112:0/573087810 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb9b4088a60 0x7fb9b40ad410 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:06.620 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:06.619 7fb9bc2c0700 1 -- 172.21.2.112:0/573087810 >> 172.21.2.112:0/573087810 conn(0x7fb9b419f5f0 msgr2=0x7fb9b41971c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:06.620 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:06.620 7fb9bc2c0700 1 -- 172.21.2.112:0/573087810 shutdown_connections 2021-02-10T11:14:06.620 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:06.620 7fb9bc2c0700 1 -- 172.21.2.112:0/573087810 wait complete. 2021-02-10T11:14:06.627 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:14:07.055 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.038 7f9806581700 1 Processor -- start 2021-02-10T11:14:07.055 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.038 7f9806581700 1 -- start start 2021-02-10T11:14:07.055 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.039 7f9806581700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f98000e5470 0x7f980007e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:07.055 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.039 7f9806581700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f980007ed80 con 0x7f980007f300 2021-02-10T11:14:07.055 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.039 7f9806581700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98001b4010 con 0x7f98000e5070 2021-02-10T11:14:07.055 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.039 7f9806581700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98000d1790 con 0x7f98000e5470 2021-02-10T11:14:07.055 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.039 7f9804b9e700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f980007f300 0x7f980007f700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:41572/0 (socket says 172.21.2.112:41572) 2021-02-10T11:14:07.055 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.039 7f9804b9e700 1 -- 172.21.2.112:0/2881156315 learned_addr learned my addr 172.21.2.112:0/2881156315 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:14:07.055 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.039 7f97ff7fe700 1 --2- 172.21.2.112:0/2881156315 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f98000e5470 0x7f980007e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:07.055 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.039 7f97feffd700 1 -- 172.21.2.112:0/2881156315 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4015366877 0 0) 0x7f97f80009b0 con 0x7f980007f300 2021-02-10T11:14:07.056 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.039 7f97feffd700 1 -- 172.21.2.112:0/2881156315 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97ec002fe0 con 0x7f980007f300 2021-02-10T11:14:07.056 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.039 7f97feffd700 1 -- 172.21.2.112:0/2881156315 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2768022753 0 0) 0x7f97f0001400 con 0x7f98000e5070 2021-02-10T11:14:07.056 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.039 7f97feffd700 1 -- 172.21.2.112:0/2881156315 --> v1:172.21.2.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97ec004370 con 0x7f98000e5070 2021-02-10T11:14:07.056 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.039 7f97ff7fe700 1 -- 172.21.2.112:0/2881156315 >> v1:172.21.2.120:6789/0 conn(0x7f98000e5070 legacy=0x7f98001b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:07.056 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.039 7f97ff7fe700 1 -- 172.21.2.112:0/2881156315 >> v1:172.21.2.112:6789/0 conn(0x7f980007f300 legacy=0x7f980007f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:07.056 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.039 7f97ff7fe700 1 -- 172.21.2.112:0/2881156315 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98000d1940 con 0x7f98000e5470 2021-02-10T11:14:07.056 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.040 7f97ff7fe700 1 --2- 172.21.2.112:0/2881156315 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f98000e5470 0x7f980007e6a0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f97e8006650 tx=0x7f97e8009a80).ready entity=mon.2 client_cookie=2fe1dd9d6bb25c71 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:07.056 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.040 7f97feffd700 1 -- 172.21.2.112:0/2881156315 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f97e80009b0 con 0x7f98000e5470 2021-02-10T11:14:07.056 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.040 7f97feffd700 1 -- 172.21.2.112:0/2881156315 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97e800c0c0 con 0x7f98000e5470 2021-02-10T11:14:07.056 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.040 7f97feffd700 1 -- 172.21.2.112:0/2881156315 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f97e800c520 con 0x7f98000e5470 2021-02-10T11:14:07.056 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.040 7f9806581700 1 -- 172.21.2.112:0/2881156315 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f98000e5470 msgr2=0x7f980007e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:07.056 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.040 7f9806581700 1 --2- 172.21.2.112:0/2881156315 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f98000e5470 0x7f980007e6a0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f97e8006650 tx=0x7f97e8009a80).stop 2021-02-10T11:14:07.056 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.041 7f9806581700 1 -- 172.21.2.112:0/2881156315 shutdown_connections 2021-02-10T11:14:07.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.041 7f9806581700 1 --2- 172.21.2.112:0/2881156315 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f98000e5470 0x7f980007e6a0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:07.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.041 7f9806581700 1 -- 172.21.2.112:0/2881156315 >> 172.21.2.112:0/2881156315 conn(0x7f98000ce690 msgr2=0x7f98000e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:07.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.041 7f9806581700 1 -- 172.21.2.112:0/2881156315 shutdown_connections 2021-02-10T11:14:07.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.041 7f9806581700 1 -- 172.21.2.112:0/2881156315 wait complete. 2021-02-10T11:14:07.058 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.055 7f9806581700 1 Processor -- start 2021-02-10T11:14:07.058 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.056 7f9806581700 1 -- start start 2021-02-10T11:14:07.058 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.056 7f9806581700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f980008d400 0x7f9800096c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:07.058 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.056 7f9806581700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f98000b1450 0x7f98000b1850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:07.059 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.056 7f9806581700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f98000b1e70 0x7f98001b1f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:07.059 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.056 7f9806581700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9800097310 con 0x7f98000b1450 2021-02-10T11:14:07.059 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.056 7f9806581700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f980008d800 con 0x7f98000b1e70 2021-02-10T11:14:07.059 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.056 7f9806581700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f980008d9f0 con 0x7f980008d400 2021-02-10T11:14:07.059 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.056 7f97ff7fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f98000b1450 0x7f98000b1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:07.059 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.056 7f97ff7fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f98000b1450 0x7f98000b1850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:42114/0 (socket says 172.21.2.112:42114) 2021-02-10T11:14:07.059 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.056 7f97ff7fe700 1 -- 172.21.2.112:0/812116267 learned_addr learned my addr 172.21.2.112:0/812116267 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:14:07.059 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.057 7f97ff7fe700 1 -- 172.21.2.112:0/812116267 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f980008d400 msgr2=0x7f9800096c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:07.059 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.057 7f97ff7fe700 1 --2- 172.21.2.112:0/812116267 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f980008d400 0x7f9800096c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:07.059 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.057 7f97ff7fe700 1 -- 172.21.2.112:0/812116267 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f98000b1e70 msgr2=0x7f98001b1f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:07.059 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.057 7f97ff7fe700 1 --2- 172.21.2.112:0/812116267 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f98000b1e70 0x7f98001b1f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:07.059 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.057 7f97ff7fe700 1 -- 172.21.2.112:0/812116267 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98001b25f0 con 0x7f98000b1450 2021-02-10T11:14:07.059 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.057 7f97ff7fe700 1 --2- 172.21.2.112:0/812116267 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f98000b1450 0x7f98000b1850 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f97e8009630 tx=0x7f97e800b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:07.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.057 7f97fcff9700 1 -- 172.21.2.112:0/812116267 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f97e80093a0 con 0x7f98000b1450 2021-02-10T11:14:07.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.057 7f97fcff9700 1 -- 172.21.2.112:0/812116267 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97e800bd70 con 0x7f98000b1450 2021-02-10T11:14:07.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.057 7f9806581700 1 -- 172.21.2.112:0/812116267 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98001b25f0 con 0x7f98000b1450 2021-02-10T11:14:07.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.057 7f97fcff9700 1 -- 172.21.2.112:0/812116267 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f97e80054d0 con 0x7f98000b1450 2021-02-10T11:14:07.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.057 7f9806581700 1 -- 172.21.2.112:0/812116267 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98001b2b80 con 0x7f98000b1450 2021-02-10T11:14:07.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.058 7f97fcff9700 1 -- 172.21.2.112:0/812116267 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f97e800a000 con 0x7f98000b1450 2021-02-10T11:14:07.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.058 7f97fcff9700 1 --2- 172.21.2.112:0/812116267 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f97e40186c0 0x7f97e401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:07.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.058 7f97fcff9700 1 -- 172.21.2.112:0/812116267 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f97e8032800 con 0x7f98000b1450 2021-02-10T11:14:07.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.058 7f97fffff700 1 --2- 172.21.2.112:0/812116267 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f97e40186c0 0x7f97e401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:07.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.059 7f97fffff700 1 --2- 172.21.2.112:0/812116267 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f97e40186c0 0x7f97e401abc0 crc :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:07.087 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.088 7f9806581700 1 -- 172.21.2.112:0/812116267 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f980007ed80 con 0x7f98000b1450 2021-02-10T11:14:07.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.091 7f97fcff9700 1 -- 172.21.2.112:0/812116267 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f97e8009340 con 0x7f98000b1450 2021-02-10T11:14:07.218 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.219 7f9806581700 1 -- 172.21.2.112:0/812116267 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9800097290 con 0x7f98000b1450 2021-02-10T11:14:07.219 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.220 7f97fcff9700 1 -- 172.21.2.112:0/812116267 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+13993 (secure 0 0 0) 0x7f97e800a000 con 0x7f98000b1450 2021-02-10T11:14:07.220 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:14:07.220 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":20,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:14:04.629689","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:14:01.400797","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2021-02-10 11:14:02.622129","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:14:07.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.221 7f9806581700 1 -- 172.21.2.112:0/812116267 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f97e40186c0 msgr2=0x7f97e401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:07.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.221 7f9806581700 1 --2- 172.21.2.112:0/812116267 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f97e40186c0 0x7f97e401abc0 crc :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:07.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.221 7f9806581700 1 -- 172.21.2.112:0/812116267 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f98000b1450 msgr2=0x7f98000b1850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:07.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.221 7f9806581700 1 --2- 172.21.2.112:0/812116267 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f98000b1450 0x7f98000b1850 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f97e8009630 tx=0x7f97e800b980).stop 2021-02-10T11:14:07.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.221 7f9806581700 1 -- 172.21.2.112:0/812116267 shutdown_connections 2021-02-10T11:14:07.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.221 7f9806581700 1 --2- 172.21.2.112:0/812116267 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f97e40186c0 0x7f97e401abc0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:07.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.221 7f9806581700 1 --2- 172.21.2.112:0/812116267 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f98000b1e70 0x7f98001b1f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:07.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.221 7f9806581700 1 --2- 172.21.2.112:0/812116267 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f98000b1450 0x7f98000b1850 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:07.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.221 7f9806581700 1 --2- 172.21.2.112:0/812116267 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f980008d400 0x7f9800096c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:07.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.221 7f9806581700 1 -- 172.21.2.112:0/812116267 >> 172.21.2.112:0/812116267 conn(0x7f98001ac620 msgr2=0x7f98000896e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:07.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.222 7f9806581700 1 -- 172.21.2.112:0/812116267 shutdown_connections 2021-02-10T11:14:07.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.222 7f9806581700 1 -- 172.21.2.112:0/812116267 wait complete. 2021-02-10T11:14:07.230 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2021-02-10T11:14:07.230 INFO:tasks.ceph.mds.a:Restarting daemon 2021-02-10T11:14:07.230 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2021-02-10T11:14:07.233 INFO:tasks.ceph.mds.a:Started 2021-02-10T11:14:07.233 INFO:tasks.ceph.mds.b:Restarting daemon 2021-02-10T11:14:07.233 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2021-02-10T11:14:07.236 INFO:tasks.ceph.mds.b:Started 2021-02-10T11:14:07.238 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:14:07.641 INFO:tasks.ceph.mds.b.gibba020.stdout:starting mds.b at 2021-02-10T11:14:07.641 INFO:tasks.ceph.mds.a.gibba012.stdout:starting mds.a at 2021-02-10T11:14:07.660 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.643 7f134cbb2700 1 Processor -- start 2021-02-10T11:14:07.661 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.643 7f134cbb2700 1 -- start start 2021-02-10T11:14:07.661 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.644 7f134cbb2700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f13480e5070 0x7f13481b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:07.661 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.644 7f134cbb2700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f13480e5470 0x7f134807d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:07.661 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.644 7f134cbb2700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f134807ed80 con 0x7f13481b3e40 2021-02-10T11:14:07.661 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.644 7f134cbb2700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1348080310 con 0x7f13480e5470 2021-02-10T11:14:07.661 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.644 7f134cbb2700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1348080500 con 0x7f13480e5070 2021-02-10T11:14:07.661 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.644 7f134605d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f13480e5470 0x7f134807d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:07.661 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.644 7f134705f700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7f13481b3e40 0x7f134807dcb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:35158/0 (socket says 172.21.2.112:35158) 2021-02-10T11:14:07.662 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.644 7f134685e700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f13480e5070 0x7f13481b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:07.662 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.644 7f134705f700 1 -- 172.21.2.112:0/1980537093 learned_addr learned my addr 172.21.2.112:0/1980537093 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:14:07.662 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.644 7f134605d700 1 -- 172.21.2.112:0/1980537093 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f13480e5070 msgr2=0x7f13481b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:14:07.662 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.644 7f134605d700 1 --2- 172.21.2.112:0/1980537093 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f13480e5070 0x7f13481b3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:07.662 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.644 7f134605d700 1 -- 172.21.2.112:0/1980537093 >> v1:172.21.2.120:6789/0 conn(0x7f13481b3e40 legacy=0x7f134807dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:07.662 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.644 7f134585c700 1 -- 172.21.2.112:0/1980537093 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3320332951 0 0) 0x7f13380010c0 con 0x7f13481b3e40 2021-02-10T11:14:07.662 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.645 7f134605d700 1 -- 172.21.2.112:0/1980537093 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13480806b0 con 0x7f13480e5470 2021-02-10T11:14:07.662 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.645 7f134685e700 1 --2- 172.21.2.112:0/1980537093 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f13480e5070 0x7f13481b3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:14:07.662 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.645 7f134605d700 1 --2- 172.21.2.112:0/1980537093 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f13480e5470 0x7f134807d690 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f13400069d0 tx=0x7f1340009e40).ready entity=mon.0 client_cookie=543e36dd3010918a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:07.663 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.645 7f134585c700 1 -- 172.21.2.112:0/1980537093 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f134000b510 con 0x7f13480e5470 2021-02-10T11:14:07.663 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.645 7f134585c700 1 -- 172.21.2.112:0/1980537093 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f134000b890 con 0x7f13480e5470 2021-02-10T11:14:07.663 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.645 7f134585c700 1 -- 172.21.2.112:0/1980537093 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f134000bd70 con 0x7f13480e5470 2021-02-10T11:14:07.663 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.645 7f134cbb2700 1 -- 172.21.2.112:0/1980537093 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f13480e5470 msgr2=0x7f134807d690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:07.663 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.645 7f134cbb2700 1 --2- 172.21.2.112:0/1980537093 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f13480e5470 0x7f134807d690 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f13400069d0 tx=0x7f1340009e40).stop 2021-02-10T11:14:07.663 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.646 7f134cbb2700 1 -- 172.21.2.112:0/1980537093 shutdown_connections 2021-02-10T11:14:07.663 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.646 7f134cbb2700 1 --2- 172.21.2.112:0/1980537093 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f13480e5470 0x7f134807d690 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:07.663 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.646 7f134cbb2700 1 --2- 172.21.2.112:0/1980537093 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f13480e5070 0x7f13481b3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:07.663 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.646 7f134cbb2700 1 -- 172.21.2.112:0/1980537093 >> 172.21.2.112:0/1980537093 conn(0x7f13480ce690 msgr2=0x7f13480e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:07.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.646 7f134cbb2700 1 -- 172.21.2.112:0/1980537093 shutdown_connections 2021-02-10T11:14:07.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.646 7f134cbb2700 1 -- 172.21.2.112:0/1980537093 wait complete. 2021-02-10T11:14:07.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.660 7f134cbb2700 1 Processor -- start 2021-02-10T11:14:07.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.660 7f134cbb2700 1 -- start start 2021-02-10T11:14:07.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.661 7f134cbb2700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f134808fca0 0x7f13480ad560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:07.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.661 7f134cbb2700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f13480adb80 0x7f1348190fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:07.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.661 7f134cbb2700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1348108010 0x7f13481915f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:07.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.661 7f134cbb2700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1348197270 con 0x7f134808fca0 2021-02-10T11:14:07.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.661 7f134cbb2700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1348191c10 con 0x7f13480adb80 2021-02-10T11:14:07.665 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.661 7f134cbb2700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1348191e30 con 0x7f1348108010 2021-02-10T11:14:07.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.661 7f134685e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f134808fca0 0x7f13480ad560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:07.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.661 7f134685e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f134808fca0 0x7f13480ad560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:42140/0 (socket says 172.21.2.112:42140) 2021-02-10T11:14:07.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.661 7f134685e700 1 -- 172.21.2.112:0/3788738504 learned_addr learned my addr 172.21.2.112:0/3788738504 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:14:07.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.661 7f134605d700 1 --2- 172.21.2.112:0/3788738504 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f13480adb80 0x7f1348190fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:07.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.661 7f134705f700 1 --2- 172.21.2.112:0/3788738504 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1348108010 0x7f13481915f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:07.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.662 7f134685e700 1 -- 172.21.2.112:0/3788738504 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1348108010 msgr2=0x7f13481915f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:07.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.662 7f134685e700 1 --2- 172.21.2.112:0/3788738504 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1348108010 0x7f13481915f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:07.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.662 7f134685e700 1 -- 172.21.2.112:0/3788738504 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f13480adb80 msgr2=0x7f1348190fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:07.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.662 7f134685e700 1 --2- 172.21.2.112:0/3788738504 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f13480adb80 0x7f1348190fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:07.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.662 7f134685e700 1 -- 172.21.2.112:0/3788738504 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1348192690 con 0x7f134808fca0 2021-02-10T11:14:07.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.662 7f134685e700 1 --2- 172.21.2.112:0/3788738504 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f134808fca0 0x7f13480ad560 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f1334006f40 tx=0x7f133400b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:07.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.662 7f133f7fe700 1 -- 172.21.2.112:0/3788738504 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1334006bd0 con 0x7f134808fca0 2021-02-10T11:14:07.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.662 7f133f7fe700 1 -- 172.21.2.112:0/3788738504 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f133400bd80 con 0x7f134808fca0 2021-02-10T11:14:07.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.662 7f133f7fe700 1 -- 172.21.2.112:0/3788738504 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1334002220 con 0x7f134808fca0 2021-02-10T11:14:07.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.662 7f134cbb2700 1 -- 172.21.2.112:0/3788738504 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1348192690 con 0x7f134808fca0 2021-02-10T11:14:07.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.662 7f134cbb2700 1 -- 172.21.2.112:0/3788738504 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1348192c20 con 0x7f134808fca0 2021-02-10T11:14:07.669 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.663 7f133f7fe700 1 -- 172.21.2.112:0/3788738504 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1334000c90 con 0x7f134808fca0 2021-02-10T11:14:07.669 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.663 7f133f7fe700 1 --2- 172.21.2.112:0/3788738504 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f13240186c0 0x7f132401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:07.669 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.663 7f133f7fe700 1 -- 172.21.2.112:0/3788738504 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f1334032650 con 0x7f134808fca0 2021-02-10T11:14:07.669 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.663 7f134605d700 1 --2- 172.21.2.112:0/3788738504 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f13240186c0 0x7f132401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:07.669 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.663 7f134605d700 1 --2- 172.21.2.112:0/3788738504 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f13240186c0 0x7f132401abc0 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:07.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.693 7f134cbb2700 1 -- 172.21.2.112:0/3788738504 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1348107180 con 0x7f134808fca0 2021-02-10T11:14:07.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.695 7f133f7fe700 1 -- 172.21.2.112:0/3788738504 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f13340009b0 con 0x7f134808fca0 2021-02-10T11:14:07.821 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.821 7f134cbb2700 1 -- 172.21.2.112:0/3788738504 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f134807ed80 con 0x7f134808fca0 2021-02-10T11:14:07.822 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.822 7f133f7fe700 1 -- 172.21.2.112:0/3788738504 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+13993 (secure 0 0 0) 0x7f1334000c90 con 0x7f134808fca0 2021-02-10T11:14:07.823 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:14:07.823 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":20,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:14:04.629689","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:14:01.400797","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2021-02-10 11:14:02.622129","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:14:07.824 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.824 7f134cbb2700 1 -- 172.21.2.112:0/3788738504 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f13240186c0 msgr2=0x7f132401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:07.824 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.824 7f134cbb2700 1 --2- 172.21.2.112:0/3788738504 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f13240186c0 0x7f132401abc0 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:07.824 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.824 7f134cbb2700 1 -- 172.21.2.112:0/3788738504 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f134808fca0 msgr2=0x7f13480ad560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:07.824 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.824 7f134cbb2700 1 --2- 172.21.2.112:0/3788738504 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f134808fca0 0x7f13480ad560 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f1334006f40 tx=0x7f133400b990).stop 2021-02-10T11:14:07.824 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.824 7f134cbb2700 1 -- 172.21.2.112:0/3788738504 shutdown_connections 2021-02-10T11:14:07.825 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.824 7f134cbb2700 1 --2- 172.21.2.112:0/3788738504 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f13240186c0 0x7f132401abc0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:07.825 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.824 7f134cbb2700 1 --2- 172.21.2.112:0/3788738504 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f13480adb80 0x7f1348190fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:07.825 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.824 7f134cbb2700 1 --2- 172.21.2.112:0/3788738504 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1348108010 0x7f13481915f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:07.825 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.824 7f134cbb2700 1 --2- 172.21.2.112:0/3788738504 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f134808fca0 0x7f13480ad560 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:07.825 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.824 7f134cbb2700 1 -- 172.21.2.112:0/3788738504 >> 172.21.2.112:0/3788738504 conn(0x7f13481986e0 msgr2=0x7f1348093c60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:07.825 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.824 7f134cbb2700 1 -- 172.21.2.112:0/3788738504 shutdown_connections 2021-02-10T11:14:07.825 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:07.824 7f134cbb2700 1 -- 172.21.2.112:0/3788738504 wait complete. 2021-02-10T11:14:07.833 INFO:tasks.ceph.ceph_manager.ceph:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:13:57.508758', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:14:01.400797', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:14:02.622129', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:14:07.834 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:14:08.272 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.250 7f8613a3d700 1 Processor -- start 2021-02-10T11:14:08.273 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.250 7f8613a3d700 1 -- start start 2021-02-10T11:14:08.273 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.251 7f8613a3d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f860c1b0470 0x7f860c1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:08.273 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.251 7f8613a3d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f860c0ef610 0x7f860c0efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:08.273 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.251 7f8613a3d700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f860c0f0030 0x7f860c0f0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:08.273 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.251 7f8613a3d700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f860c0e5ea0 con 0x7f860c1b0470 2021-02-10T11:14:08.273 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.251 7f8613a3d700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f860c0f0b50 con 0x7f860c0f0030 2021-02-10T11:14:08.273 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.251 7f8613a3d700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f860c0e52e0 con 0x7f860c0ef610 2021-02-10T11:14:08.273 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.251 7f8611859700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f860c1b0470 0x7f860c1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:08.273 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.251 7f8611859700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f860c1b0470 0x7f860c1b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:42148/0 (socket says 172.21.2.112:42148) 2021-02-10T11:14:08.273 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.251 7f8611859700 1 -- 172.21.2.112:0/2589370346 learned_addr learned my addr 172.21.2.112:0/2589370346 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:14:08.273 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.252 7f8611859700 1 -- 172.21.2.112:0/2589370346 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f860c0ef610 msgr2=0x7f860c0efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:14:08.274 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.252 7f8611859700 1 --2- 172.21.2.112:0/2589370346 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f860c0ef610 0x7f860c0efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:08.274 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.252 7f8611859700 1 -- 172.21.2.112:0/2589370346 >> v2:172.21.2.120:3300/0 conn(0x7f860c0f0030 msgr2=0x7f860c0f0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:14:08.274 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.252 7f8611859700 1 --2- 172.21.2.112:0/2589370346 >> v2:172.21.2.120:3300/0 conn(0x7f860c0f0030 0x7f860c0f0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:08.274 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.252 7f8611859700 1 -- 172.21.2.112:0/2589370346 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f860c0f10d0 con 0x7f860c1b0470 2021-02-10T11:14:08.274 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.252 7f8611859700 1 --2- 172.21.2.112:0/2589370346 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f860c1b0470 0x7f860c1b0870 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f8600006700 tx=0x7f8600009b00).ready entity=mon.0 client_cookie=ffb45506b1a70173 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:08.274 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.252 7f8610857700 1 -- 172.21.2.112:0/2589370346 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f860000c0c0 con 0x7f860c1b0470 2021-02-10T11:14:08.274 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.252 7f8610857700 1 -- 172.21.2.112:0/2589370346 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f860000c400 con 0x7f860c1b0470 2021-02-10T11:14:08.274 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.252 7f8610857700 1 -- 172.21.2.112:0/2589370346 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f860000c920 con 0x7f860c1b0470 2021-02-10T11:14:08.274 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.252 7f8613a3d700 1 -- 172.21.2.112:0/2589370346 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f860c1b0470 msgr2=0x7f860c1b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:08.274 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.252 7f8613a3d700 1 --2- 172.21.2.112:0/2589370346 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f860c1b0470 0x7f860c1b0870 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f8600006700 tx=0x7f8600009b00).stop 2021-02-10T11:14:08.274 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.254 7f8613a3d700 1 -- 172.21.2.112:0/2589370346 shutdown_connections 2021-02-10T11:14:08.274 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.254 7f8613a3d700 1 --2- 172.21.2.112:0/2589370346 >> v2:172.21.2.120:3300/0 conn(0x7f860c0f0030 0x7f860c0f0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:08.275 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.254 7f8613a3d700 1 --2- 172.21.2.112:0/2589370346 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f860c0ef610 0x7f860c0efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:08.275 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.254 7f8613a3d700 1 --2- 172.21.2.112:0/2589370346 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f860c1b0470 0x7f860c1b0870 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:08.275 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.254 7f8613a3d700 1 -- 172.21.2.112:0/2589370346 >> 172.21.2.112:0/2589370346 conn(0x7f860c0ce3e0 msgr2=0x7f860c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:08.275 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.254 7f8613a3d700 1 -- 172.21.2.112:0/2589370346 shutdown_connections 2021-02-10T11:14:08.275 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.254 7f8613a3d700 1 -- 172.21.2.112:0/2589370346 wait complete. 2021-02-10T11:14:08.275 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.268 7f8613a3d700 1 Processor -- start 2021-02-10T11:14:08.275 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.268 7f8613a3d700 1 -- start start 2021-02-10T11:14:08.275 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.268 7f8613a3d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f860c0b20b0 0x7f860c0ac970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:08.275 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.268 7f8613a3d700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f860c0b24b0 0x7f860c191dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:08.275 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.268 7f8613a3d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f860c0acf90 0x7f860c1923e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:08.275 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.268 7f8613a3d700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f860c0b1b30 con 0x7f860c0b20b0 2021-02-10T11:14:08.275 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.268 7f8613a3d700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f860c1a04e0 con 0x7f860c0b24b0 2021-02-10T11:14:08.275 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.268 7f8613a3d700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f860c1a06c0 con 0x7f860c0acf90 2021-02-10T11:14:08.276 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.268 7f8611859700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f860c0b20b0 0x7f860c0ac970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:08.276 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.269 7f8611859700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f860c0b20b0 0x7f860c0ac970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:42154/0 (socket says 172.21.2.112:42154) 2021-02-10T11:14:08.276 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.269 7f8611859700 1 -- 172.21.2.112:0/81524351 learned_addr learned my addr 172.21.2.112:0/81524351 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:14:08.276 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.269 7f8611058700 1 --2- 172.21.2.112:0/81524351 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f860c0b24b0 0x7f860c191dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:08.276 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.269 7f8611859700 1 -- 172.21.2.112:0/81524351 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f860c0acf90 msgr2=0x7f860c1923e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:08.276 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.269 7f8611859700 1 --2- 172.21.2.112:0/81524351 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f860c0acf90 0x7f860c1923e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:08.276 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.269 7f8611859700 1 -- 172.21.2.112:0/81524351 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f860c0b24b0 msgr2=0x7f860c191dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:08.277 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.269 7f8611859700 1 --2- 172.21.2.112:0/81524351 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f860c0b24b0 0x7f860c191dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:08.277 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.269 7f8611859700 1 -- 172.21.2.112:0/81524351 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f860c192be0 con 0x7f860c0b20b0 2021-02-10T11:14:08.278 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.269 7f8611058700 1 --2- 172.21.2.112:0/81524351 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f860c0b24b0 0x7f860c191dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:14:08.278 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.269 7f8611859700 1 --2- 172.21.2.112:0/81524351 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f860c0b20b0 0x7f860c0ac970 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f860000b660 tx=0x7f860000ba00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:08.278 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.269 7f85fe7fc700 1 -- 172.21.2.112:0/81524351 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8600009440 con 0x7f860c0b20b0 2021-02-10T11:14:08.278 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.269 7f85fe7fc700 1 -- 172.21.2.112:0/81524351 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8600000c90 con 0x7f860c0b20b0 2021-02-10T11:14:08.278 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.269 7f85fe7fc700 1 -- 172.21.2.112:0/81524351 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8600002030 con 0x7f860c0b20b0 2021-02-10T11:14:08.278 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.269 7f8613a3d700 1 -- 172.21.2.112:0/81524351 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f860c192be0 con 0x7f860c0b20b0 2021-02-10T11:14:08.278 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.270 7f8613a3d700 1 -- 172.21.2.112:0/81524351 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f860c192be0 con 0x7f860c0b20b0 2021-02-10T11:14:08.278 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.270 7f85fe7fc700 1 -- 172.21.2.112:0/81524351 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f86000093e0 con 0x7f860c0b20b0 2021-02-10T11:14:08.278 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.270 7f85fe7fc700 1 --2- 172.21.2.112:0/81524351 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f85f40186c0 0x7f85f401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:08.278 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.270 7f85fe7fc700 1 -- 172.21.2.112:0/81524351 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f86000325d0 con 0x7f860c0b20b0 2021-02-10T11:14:08.278 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.271 7f8611058700 1 --2- 172.21.2.112:0/81524351 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f85f40186c0 0x7f85f401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:08.278 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.271 7f8611058700 1 --2- 172.21.2.112:0/81524351 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f85f40186c0 0x7f85f401abc0 crc :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:08.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.301 7f8613a3d700 1 -- 172.21.2.112:0/81524351 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f860c105af0 con 0x7f860c0b20b0 2021-02-10T11:14:08.303 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.305 7f85fe7fc700 1 -- 172.21.2.112:0/81524351 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f86000090c0 con 0x7f860c0b20b0 2021-02-10T11:14:08.430 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.430 7f8613a3d700 1 -- 172.21.2.112:0/81524351 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f860c0f0b50 con 0x7f860c0b20b0 2021-02-10T11:14:08.430 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.430 7f85fe7fc700 1 -- 172.21.2.112:0/81524351 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v20) v1 ==== 92+0+10 (secure 0 0 0) 0x7f860000bdf0 con 0x7f860c0b20b0 2021-02-10T11:14:08.431 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:14:08.431 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.432 7f8613a3d700 1 -- 172.21.2.112:0/81524351 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f85f40186c0 msgr2=0x7f85f401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:08.431 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.432 7f8613a3d700 1 --2- 172.21.2.112:0/81524351 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f85f40186c0 0x7f85f401abc0 crc :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:08.431 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.432 7f8613a3d700 1 -- 172.21.2.112:0/81524351 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f860c0b20b0 msgr2=0x7f860c0ac970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:08.432 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.432 7f8613a3d700 1 --2- 172.21.2.112:0/81524351 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f860c0b20b0 0x7f860c0ac970 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f860000b660 tx=0x7f860000ba00).stop 2021-02-10T11:14:08.432 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.432 7f8613a3d700 1 -- 172.21.2.112:0/81524351 shutdown_connections 2021-02-10T11:14:08.432 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.432 7f8613a3d700 1 --2- 172.21.2.112:0/81524351 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f85f40186c0 0x7f85f401abc0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:08.432 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.432 7f8613a3d700 1 --2- 172.21.2.112:0/81524351 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f860c0b24b0 0x7f860c191dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:08.432 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.432 7f8613a3d700 1 --2- 172.21.2.112:0/81524351 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f860c0acf90 0x7f860c1923e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:08.432 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.432 7f8613a3d700 1 --2- 172.21.2.112:0/81524351 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f860c0b20b0 0x7f860c0ac970 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:08.433 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.432 7f8613a3d700 1 -- 172.21.2.112:0/81524351 >> 172.21.2.112:0/81524351 conn(0x7f860c0bf4e0 msgr2=0x7f860c0be3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:08.433 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.432 7f8613a3d700 1 -- 172.21.2.112:0/81524351 shutdown_connections 2021-02-10T11:14:08.433 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.432 7f8613a3d700 1 -- 172.21.2.112:0/81524351 wait complete. 2021-02-10T11:14:08.440 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:14:08.878 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.861 7f6e46879700 1 Processor -- start 2021-02-10T11:14:08.878 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.861 7f6e46879700 1 -- start start 2021-02-10T11:14:08.878 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.862 7f6e46879700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f6e401b0870 0x7f6e400f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:08.878 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.862 7f6e46879700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6e400efe00 0x7f6e400f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:08.878 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.862 7f6e46879700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e401b0c70 con 0x7f6e401b0470 2021-02-10T11:14:08.878 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.862 7f6e46879700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e400f0b50 con 0x7f6e401b0870 2021-02-10T11:14:08.878 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.862 7f6e46879700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e400f0820 con 0x7f6e400efe00 2021-02-10T11:14:08.878 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.862 7f6e3f7fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f6e401b0870 0x7f6e400f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:08.879 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.862 7f6e3f7fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f6e401b0870 0x7f6e400f20e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:42162/0 (socket says 172.21.2.112:42162) 2021-02-10T11:14:08.879 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.862 7f6e3f7fe700 1 -- 172.21.2.112:0/3667665590 learned_addr learned my addr 172.21.2.112:0/3667665590 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:14:08.879 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.862 7f6e44e96700 1 --2- 172.21.2.112:0/3667665590 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6e400efe00 0x7f6e400f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:08.879 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.863 7f6e3f7fe700 1 -- 172.21.2.112:0/3667665590 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6e400efe00 msgr2=0x7f6e400f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:14:08.879 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.863 7f6e3f7fe700 1 --2- 172.21.2.112:0/3667665590 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6e400efe00 0x7f6e400f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:08.879 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.863 7f6e3f7fe700 1 -- 172.21.2.112:0/3667665590 >> v1:172.21.2.120:6789/0 conn(0x7f6e401b0470 legacy=0x7f6e400ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:08.879 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.863 7f6e3f7fe700 1 -- 172.21.2.112:0/3667665590 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e400e52e0 con 0x7f6e401b0870 2021-02-10T11:14:08.879 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.863 7f6e3effd700 1 -- 172.21.2.112:0/3667665590 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 583716622 0 0) 0x7f6e30001080 con 0x7f6e401b0470 2021-02-10T11:14:08.879 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.863 7f6e3f7fe700 1 --2- 172.21.2.112:0/3667665590 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6e401b0870 0x7f6e400f20e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f6e34006700 tx=0x7f6e34009b00).ready entity=mon.0 client_cookie=c7b14c1c7335f310 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:08.879 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.863 7f6e3effd700 1 -- 172.21.2.112:0/3667665590 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6e3400c0c0 con 0x7f6e401b0870 2021-02-10T11:14:08.879 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.863 7f6e3effd700 1 -- 172.21.2.112:0/3667665590 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e3400c400 con 0x7f6e401b0870 2021-02-10T11:14:08.879 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.863 7f6e3effd700 1 -- 172.21.2.112:0/3667665590 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6e3400c920 con 0x7f6e401b0870 2021-02-10T11:14:08.880 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.863 7f6e46879700 1 -- 172.21.2.112:0/3667665590 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6e401b0870 msgr2=0x7f6e400f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:08.880 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.863 7f6e46879700 1 --2- 172.21.2.112:0/3667665590 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6e401b0870 0x7f6e400f20e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f6e34006700 tx=0x7f6e34009b00).stop 2021-02-10T11:14:08.880 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.864 7f6e46879700 1 -- 172.21.2.112:0/3667665590 shutdown_connections 2021-02-10T11:14:08.880 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.864 7f6e46879700 1 --2- 172.21.2.112:0/3667665590 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6e401b0870 0x7f6e400f20e0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:08.880 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.864 7f6e46879700 1 --2- 172.21.2.112:0/3667665590 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6e400efe00 0x7f6e400f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:08.880 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.864 7f6e46879700 1 -- 172.21.2.112:0/3667665590 >> 172.21.2.112:0/3667665590 conn(0x7f6e400ce3e0 msgr2=0x7f6e400e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:08.880 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.864 7f6e46879700 1 -- 172.21.2.112:0/3667665590 shutdown_connections 2021-02-10T11:14:08.880 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.864 7f6e46879700 1 -- 172.21.2.112:0/3667665590 wait complete. 2021-02-10T11:14:08.881 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.878 7f6e46879700 1 Processor -- start 2021-02-10T11:14:08.882 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.879 7f6e46879700 1 -- start start 2021-02-10T11:14:08.882 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.879 7f6e46879700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6e400a0420 0x7f6e400a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:08.882 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.879 7f6e46879700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6e40189d00 0x7f6e4018a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:08.882 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.879 7f6e46879700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6e4018a720 0x7f6e400b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:08.882 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.879 7f6e46879700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e40189780 con 0x7f6e4018a720 2021-02-10T11:14:08.882 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.879 7f6e46879700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e4018ab20 con 0x7f6e400a0420 2021-02-10T11:14:08.882 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.879 7f6e46879700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e400a1140 con 0x7f6e40189d00 2021-02-10T11:14:08.882 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.879 7f6e44e96700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6e4018a720 0x7f6e400b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:08.882 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.879 7f6e44e96700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6e4018a720 0x7f6e400b7be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:42170/0 (socket says 172.21.2.112:42170) 2021-02-10T11:14:08.882 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.879 7f6e44e96700 1 -- 172.21.2.112:0/2747777085 learned_addr learned my addr 172.21.2.112:0/2747777085 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:14:08.882 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.880 7f6e3ffff700 1 --2- 172.21.2.112:0/2747777085 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6e400a0420 0x7f6e400a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:08.882 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.880 7f6e3f7fe700 1 --2- 172.21.2.112:0/2747777085 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6e40189d00 0x7f6e4018a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:08.883 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.880 7f6e44e96700 1 -- 172.21.2.112:0/2747777085 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6e40189d00 msgr2=0x7f6e4018a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:08.883 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.880 7f6e44e96700 1 --2- 172.21.2.112:0/2747777085 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6e40189d00 0x7f6e4018a100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:08.883 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.880 7f6e44e96700 1 -- 172.21.2.112:0/2747777085 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6e400a0420 msgr2=0x7f6e400a0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:08.883 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.880 7f6e44e96700 1 --2- 172.21.2.112:0/2747777085 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6e400a0420 0x7f6e400a0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:08.883 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.880 7f6e44e96700 1 -- 172.21.2.112:0/2747777085 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e400b8200 con 0x7f6e4018a720 2021-02-10T11:14:08.883 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.880 7f6e3ffff700 1 --2- 172.21.2.112:0/2747777085 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6e400a0420 0x7f6e400a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:14:08.883 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.880 7f6e44e96700 1 --2- 172.21.2.112:0/2747777085 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6e4018a720 0x7f6e400b7be0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f6e3800b970 tx=0x7f6e3800bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:08.883 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.880 7f6e3cff9700 1 -- 172.21.2.112:0/2747777085 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6e38000f90 con 0x7f6e4018a720 2021-02-10T11:14:08.883 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.880 7f6e3cff9700 1 -- 172.21.2.112:0/2747777085 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e38006aa0 con 0x7f6e4018a720 2021-02-10T11:14:08.883 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.880 7f6e3cff9700 1 -- 172.21.2.112:0/2747777085 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6e380015f0 con 0x7f6e4018a720 2021-02-10T11:14:08.883 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.880 7f6e46879700 1 -- 172.21.2.112:0/2747777085 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e400b8200 con 0x7f6e4018a720 2021-02-10T11:14:08.883 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.880 7f6e46879700 1 -- 172.21.2.112:0/2747777085 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e400b8200 con 0x7f6e4018a720 2021-02-10T11:14:08.883 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.880 7f6e3cff9700 1 -- 172.21.2.112:0/2747777085 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6e38000e70 con 0x7f6e4018a720 2021-02-10T11:14:08.884 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.881 7f6e3cff9700 1 --2- 172.21.2.112:0/2747777085 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6e200186c0 0x7f6e2001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:08.884 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.881 7f6e3cff9700 1 -- 172.21.2.112:0/2747777085 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f6e38032810 con 0x7f6e4018a720 2021-02-10T11:14:08.884 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.881 7f6e3ffff700 1 --2- 172.21.2.112:0/2747777085 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6e200186c0 0x7f6e2001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:08.884 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.881 7f6e3ffff700 1 --2- 172.21.2.112:0/2747777085 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6e200186c0 0x7f6e2001abc0 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:08.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.912 7f6e46879700 1 -- 172.21.2.112:0/2747777085 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e40105ae0 con 0x7f6e4018a720 2021-02-10T11:14:08.914 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:08.915 7f6e3cff9700 1 -- 172.21.2.112:0/2747777085 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6e380066f0 con 0x7f6e4018a720 2021-02-10T11:14:09.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:09.042 7f6e46879700 1 -- 172.21.2.112:0/2747777085 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f6e400f0b50 con 0x7f6e4018a720 2021-02-10T11:14:09.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:09.042 7f6e3cff9700 1 -- 172.21.2.112:0/2747777085 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v20) v1 ==== 104+0+10 (secure 0 0 0) 0x7f6e38000e70 con 0x7f6e4018a720 2021-02-10T11:14:09.042 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:14:09.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:09.045 7f6e46879700 1 -- 172.21.2.112:0/2747777085 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6e200186c0 msgr2=0x7f6e2001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:09.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:09.045 7f6e46879700 1 --2- 172.21.2.112:0/2747777085 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6e200186c0 0x7f6e2001abc0 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:09.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:09.045 7f6e46879700 1 -- 172.21.2.112:0/2747777085 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6e4018a720 msgr2=0x7f6e400b7be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:09.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:09.045 7f6e46879700 1 --2- 172.21.2.112:0/2747777085 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6e4018a720 0x7f6e400b7be0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f6e3800b970 tx=0x7f6e3800bd10).stop 2021-02-10T11:14:09.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:09.045 7f6e46879700 1 -- 172.21.2.112:0/2747777085 shutdown_connections 2021-02-10T11:14:09.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:09.045 7f6e46879700 1 --2- 172.21.2.112:0/2747777085 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6e200186c0 0x7f6e2001abc0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:09.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:09.045 7f6e46879700 1 --2- 172.21.2.112:0/2747777085 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6e4018a720 0x7f6e400b7be0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:09.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:09.045 7f6e46879700 1 --2- 172.21.2.112:0/2747777085 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6e40189d00 0x7f6e4018a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:09.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:09.045 7f6e46879700 1 --2- 172.21.2.112:0/2747777085 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6e400a0420 0x7f6e400a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:09.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:09.045 7f6e46879700 1 -- 172.21.2.112:0/2747777085 >> 172.21.2.112:0/2747777085 conn(0x7f6e40094e10 msgr2=0x7f6e400a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:09.045 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:09.045 7f6e46879700 1 -- 172.21.2.112:0/2747777085 shutdown_connections 2021-02-10T11:14:09.045 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:09.045 7f6e46879700 1 -- 172.21.2.112:0/2747777085 wait complete. 2021-02-10T11:14:09.052 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:14:09.489 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:09.473 7fb4023fa700 1 Processor -- start 2021-02-10T11:14:09.489 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:09.473 7fb4023fa700 1 -- start start 2021-02-10T11:14:09.489 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:09.474 7fb4023fa700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fb3fc1b0470 0x7fb3fc1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:09.489 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:09.474 7fb4023fa700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb3fc0ef610 0x7fb3fc0efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:09.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:09.474 7fb4023fa700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fb3fc0f0030 0x7fb3fc0f0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:09.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:09.474 7fb4023fa700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3fc0e5ea0 con 0x7fb3fc1b0470 2021-02-10T11:14:09.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:09.474 7fb4023fa700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3fc0f0b50 con 0x7fb3fc0f0030 2021-02-10T11:14:09.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:09.474 7fb4023fa700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3fc0e52e0 con 0x7fb3fc0ef610 2021-02-10T11:14:09.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:09.474 7fb3fbfff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fb3fc1b0470 0x7fb3fc1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:09.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:09.474 7fb3fbfff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fb3fc1b0470 0x7fb3fc1b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:42174/0 (socket says 172.21.2.112:42174) 2021-02-10T11:14:09.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:09.474 7fb3fbfff700 1 -- 172.21.2.112:0/4238728261 learned_addr learned my addr 172.21.2.112:0/4238728261 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:14:09.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:09.474 7fb400a17700 1 --2- 172.21.2.112:0/4238728261 >> v2:172.21.2.120:3300/0 conn(0x7fb3fc0f0030 0x7fb3fc0f0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:09.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:09.474 7fb3fb7fe700 1 --2- 172.21.2.112:0/4238728261 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb3fc0ef610 0x7fb3fc0efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:09.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:09.474 7fb3fbfff700 1 -- 172.21.2.112:0/4238728261 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb3fc0ef610 msgr2=0x7fb3fc0efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:14:09.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:09.474 7fb3fbfff700 1 --2- 172.21.2.112:0/4238728261 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb3fc0ef610 0x7fb3fc0efa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:09.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:09.474 7fb3fbfff700 1 -- 172.21.2.112:0/4238728261 >> v2:172.21.2.120:3300/0 conn(0x7fb3fc0f0030 msgr2=0x7fb3fc0f0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:14:09.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:09.474 7fb3fbfff700 1 --2- 172.21.2.112:0/4238728261 >> v2:172.21.2.120:3300/0 conn(0x7fb3fc0f0030 0x7fb3fc0f0430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:09.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:09.474 7fb3fbfff700 1 -- 172.21.2.112:0/4238728261 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3fc0f10d0 con 0x7fb3fc1b0470 2021-02-10T11:14:09.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:09.474 7fb3fbfff700 1 --2- 172.21.2.112:0/4238728261 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb3fc1b0470 0x7fb3fc1b0870 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7fb3f4006a80 tx=0x7fb3f4009f00).ready entity=mon.0 client_cookie=9f17b8d0a070ce26 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:09.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:09.474 7fb3faffd700 1 -- 172.21.2.112:0/4238728261 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb3f400b5c0 con 0x7fb3fc1b0470 2021-02-10T11:14:09.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:09.474 7fb3faffd700 1 -- 172.21.2.112:0/4238728261 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3f400b800 con 0x7fb3fc1b0470 2021-02-10T11:14:09.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:09.475 7fb3faffd700 1 -- 172.21.2.112:0/4238728261 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb3f400bce0 con 0x7fb3fc1b0470 2021-02-10T11:14:09.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:09.475 7fb4023fa700 1 -- 172.21.2.112:0/4238728261 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb3fc1b0470 msgr2=0x7fb3fc1b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:09.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:09.475 7fb4023fa700 1 --2- 172.21.2.112:0/4238728261 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb3fc1b0470 0x7fb3fc1b0870 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7fb3f4006a80 tx=0x7fb3f4009f00).stop 2021-02-10T11:14:09.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:09.475 7fb4023fa700 1 -- 172.21.2.112:0/4238728261 shutdown_connections 2021-02-10T11:14:09.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:09.475 7fb4023fa700 1 --2- 172.21.2.112:0/4238728261 >> v2:172.21.2.120:3300/0 conn(0x7fb3fc0f0030 0x7fb3fc0f0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:09.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:09.475 7fb4023fa700 1 --2- 172.21.2.112:0/4238728261 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb3fc0ef610 0x7fb3fc0efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:09.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:09.475 7fb4023fa700 1 --2- 172.21.2.112:0/4238728261 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb3fc1b0470 0x7fb3fc1b0870 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:09.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:09.475 7fb4023fa700 1 -- 172.21.2.112:0/4238728261 >> 172.21.2.112:0/4238728261 conn(0x7fb3fc0ce3e0 msgr2=0x7fb3fc0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:09.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:09.475 7fb4023fa700 1 -- 172.21.2.112:0/4238728261 shutdown_connections 2021-02-10T11:14:09.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:09.475 7fb4023fa700 1 -- 172.21.2.112:0/4238728261 wait complete. 2021-02-10T11:14:09.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:09.490 7fb4023fa700 1 Processor -- start 2021-02-10T11:14:09.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:09.490 7fb4023fa700 1 -- start start 2021-02-10T11:14:09.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:09.490 7fb4023fa700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb3fc09b930 0x7fb3fc0b7c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:09.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:09.490 7fb4023fa700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb3fc0b8410 0x7fb3fc0b8810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:09.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:09.490 7fb4023fa700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb3fc0c1f80 0x7fb3fc0c2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:09.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:09.490 7fb4023fa700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3fc0c1a00 con 0x7fb3fc0c1f80 2021-02-10T11:14:09.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:09.490 7fb4023fa700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3fc0b8e30 con 0x7fb3fc09b930 2021-02-10T11:14:09.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:09.490 7fb4023fa700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3fc0b9050 con 0x7fb3fc0b8410 2021-02-10T11:14:09.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:09.490 7fb3fbfff700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb3fc09b930 0x7fb3fc0b7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:09.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:09.490 7fb3fb7fe700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb3fc0b8410 0x7fb3fc0b8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:09.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:09.490 7fb400a17700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb3fc0c1f80 0x7fb3fc0c2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:09.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:09.490 7fb3fbfff700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb3fc09b930 0x7fb3fc0b7c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3300/0 says I am v2:172.21.2.112:44462/0 (socket says 172.21.2.112:44462) 2021-02-10T11:14:09.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:09.490 7fb3fbfff700 1 -- 172.21.2.112:0/130697240 learned_addr learned my addr 172.21.2.112:0/130697240 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:14:09.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:09.491 7fb3fbfff700 1 -- 172.21.2.112:0/130697240 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb3fc0b8410 msgr2=0x7fb3fc0b8810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:09.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:09.491 7fb3fbfff700 1 --2- 172.21.2.112:0/130697240 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb3fc0b8410 0x7fb3fc0b8810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:09.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:09.491 7fb3fbfff700 1 -- 172.21.2.112:0/130697240 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb3fc0c1f80 msgr2=0x7fb3fc0c2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:09.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:09.491 7fb3fbfff700 1 --2- 172.21.2.112:0/130697240 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb3fc0c1f80 0x7fb3fc0c2380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:09.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:09.491 7fb3fbfff700 1 -- 172.21.2.112:0/130697240 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3fc0c29a0 con 0x7fb3fc09b930 2021-02-10T11:14:09.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:09.491 7fb3fb7fe700 1 --2- 172.21.2.112:0/130697240 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb3fc0b8410 0x7fb3fc0b8810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:14:09.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:09.491 7fb3fbfff700 1 --2- 172.21.2.112:0/130697240 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb3fc09b930 0x7fb3fc0b7c20 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7fb3f400b9c0 tx=0x7fb3f400bd60).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:09.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:09.491 7fb3f8ff9700 1 -- 172.21.2.112:0/130697240 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb3f4006240 con 0x7fb3fc09b930 2021-02-10T11:14:09.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:09.491 7fb3f8ff9700 1 -- 172.21.2.112:0/130697240 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3f4005660 con 0x7fb3fc09b930 2021-02-10T11:14:09.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:09.491 7fb4023fa700 1 -- 172.21.2.112:0/130697240 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3fc0c29a0 con 0x7fb3fc09b930 2021-02-10T11:14:09.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:09.491 7fb4023fa700 1 -- 172.21.2.112:0/130697240 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3fc0c2f30 con 0x7fb3fc09b930 2021-02-10T11:14:09.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:09.491 7fb3f8ff9700 1 -- 172.21.2.112:0/130697240 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb3f4005b40 con 0x7fb3fc09b930 2021-02-10T11:14:09.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:09.492 7fb3f8ff9700 1 -- 172.21.2.112:0/130697240 <== mon.1 v2:172.21.2.120:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb3f4006030 con 0x7fb3fc09b930 2021-02-10T11:14:09.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:09.492 7fb3f8ff9700 1 --2- 172.21.2.112:0/130697240 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb3dc0186c0 0x7fb3dc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:09.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:09.492 7fb3f8ff9700 1 -- 172.21.2.112:0/130697240 <== mon.1 v2:172.21.2.120:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fb3f4032990 con 0x7fb3fc09b930 2021-02-10T11:14:09.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:09.492 7fb3fb7fe700 1 --2- 172.21.2.112:0/130697240 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb3dc0186c0 0x7fb3dc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:09.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:09.492 7fb3fb7fe700 1 --2- 172.21.2.112:0/130697240 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb3dc0186c0 0x7fb3dc01abc0 crc :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:09.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:09.524 7fb4023fa700 1 -- 172.21.2.112:0/130697240 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3fc105b30 con 0x7fb3fc09b930 2021-02-10T11:14:09.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:09.526 7fb3f8ff9700 1 -- 172.21.2.112:0/130697240 <== mon.1 v2:172.21.2.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb3f4000d30 con 0x7fb3fc09b930 2021-02-10T11:14:09.652 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:09.652 7fb4023fa700 1 -- 172.21.2.112:0/130697240 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7fb3fc0f0b50 con 0x7fb3fc09b930 2021-02-10T11:14:09.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:09.654 7fb3f8ff9700 1 -- 172.21.2.112:0/130697240 <== mon.1 v2:172.21.2.120:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v20) v1 ==== 100+0+10 (secure 0 0 0) 0x7fb3f4006030 con 0x7fb3fc09b930 2021-02-10T11:14:09.655 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:14:09.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:09.657 7fb4023fa700 1 -- 172.21.2.112:0/130697240 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb3dc0186c0 msgr2=0x7fb3dc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:09.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:09.657 7fb4023fa700 1 --2- 172.21.2.112:0/130697240 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb3dc0186c0 0x7fb3dc01abc0 crc :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:09.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:09.657 7fb4023fa700 1 -- 172.21.2.112:0/130697240 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb3fc09b930 msgr2=0x7fb3fc0b7c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:09.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:09.657 7fb4023fa700 1 --2- 172.21.2.112:0/130697240 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb3fc09b930 0x7fb3fc0b7c20 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7fb3f400b9c0 tx=0x7fb3f400bd60).stop 2021-02-10T11:14:09.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:09.657 7fb4023fa700 1 -- 172.21.2.112:0/130697240 shutdown_connections 2021-02-10T11:14:09.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:09.657 7fb4023fa700 1 --2- 172.21.2.112:0/130697240 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb3dc0186c0 0x7fb3dc01abc0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:09.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:09.657 7fb4023fa700 1 --2- 172.21.2.112:0/130697240 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb3fc0c1f80 0x7fb3fc0c2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:09.657 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:09.657 7fb4023fa700 1 --2- 172.21.2.112:0/130697240 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb3fc0b8410 0x7fb3fc0b8810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:09.657 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:09.657 7fb4023fa700 1 --2- 172.21.2.112:0/130697240 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb3fc09b930 0x7fb3fc0b7c20 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:09.657 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:09.657 7fb4023fa700 1 -- 172.21.2.112:0/130697240 >> 172.21.2.112:0/130697240 conn(0x7fb3fc094e10 msgr2=0x7fb3fc0a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:09.657 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:09.657 7fb4023fa700 1 -- 172.21.2.112:0/130697240 shutdown_connections 2021-02-10T11:14:09.657 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:09.657 7fb4023fa700 1 -- 172.21.2.112:0/130697240 wait complete. 2021-02-10T11:14:09.664 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2021-02-10T11:14:09.664 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2021-02-10T11:14:09.665 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph mgr dump --format=json 2021-02-10T11:14:10.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:10.077 7f4708bfd700 1 Processor -- start 2021-02-10T11:14:10.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:10.077 7f4708bfd700 1 -- start start 2021-02-10T11:14:10.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:10.078 7f4708bfd700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f47040e5070 0x7f47041b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:10.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:10.078 7f4708bfd700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f47040e5470 0x7f470407d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:10.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:10.078 7f4708bfd700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f470407ed80 con 0x7f47041b3e40 2021-02-10T11:14:10.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:10.078 7f4708bfd700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4704080310 con 0x7f47040e5470 2021-02-10T11:14:10.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:10.078 7f4708bfd700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4704080500 con 0x7f47040e5070 2021-02-10T11:14:10.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:10.079 7f470205d700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f47040e5470 0x7f470407d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:10.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:10.079 7f470305f700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7f47041b3e40 0x7f470407dcb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:35212/0 (socket says 172.21.2.112:35212) 2021-02-10T11:14:10.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:10.079 7f470305f700 1 -- 172.21.2.112:0/4269658200 learned_addr learned my addr 172.21.2.112:0/4269658200 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:14:10.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:10.079 7f470285e700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f47040e5070 0x7f47041b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:10.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:10.079 7f470205d700 1 -- 172.21.2.112:0/4269658200 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f47040e5070 msgr2=0x7f47041b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:14:10.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:10.079 7f470205d700 1 --2- 172.21.2.112:0/4269658200 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f47040e5070 0x7f47041b3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:10.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:10.079 7f470205d700 1 -- 172.21.2.112:0/4269658200 >> v1:172.21.2.120:6789/0 conn(0x7f47041b3e40 legacy=0x7f470407dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:10.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:10.079 7f470205d700 1 -- 172.21.2.112:0/4269658200 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47040806f0 con 0x7f47040e5470 2021-02-10T11:14:10.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:10.079 7f470285e700 1 --2- 172.21.2.112:0/4269658200 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f47040e5070 0x7f47041b3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:14:10.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:10.079 7f470205d700 1 --2- 172.21.2.112:0/4269658200 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f47040e5470 0x7f470407d690 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f46ec006650 tx=0x7f46ec009a80).ready entity=mon.1 client_cookie=6802cf0480c0d7bc server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:10.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:10.081 7f470185c700 1 -- 172.21.2.112:0/4269658200 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f46ec0009b0 con 0x7f47040e5470 2021-02-10T11:14:10.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:10.081 7f470185c700 1 -- 172.21.2.112:0/4269658200 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46ec00c0c0 con 0x7f47040e5470 2021-02-10T11:14:10.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:10.081 7f470185c700 1 -- 172.21.2.112:0/4269658200 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f46ec00c5e0 con 0x7f47040e5470 2021-02-10T11:14:10.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:10.081 7f4708bfd700 1 -- 172.21.2.112:0/4269658200 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f47040e5470 msgr2=0x7f470407d690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:10.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:10.081 7f4708bfd700 1 --2- 172.21.2.112:0/4269658200 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f47040e5470 0x7f470407d690 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f46ec006650 tx=0x7f46ec009a80).stop 2021-02-10T11:14:10.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:10.081 7f4708bfd700 1 -- 172.21.2.112:0/4269658200 shutdown_connections 2021-02-10T11:14:10.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:10.081 7f4708bfd700 1 --2- 172.21.2.112:0/4269658200 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f47040e5470 0x7f470407d690 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:10.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:10.081 7f4708bfd700 1 --2- 172.21.2.112:0/4269658200 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f47040e5070 0x7f47041b3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:10.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:10.081 7f4708bfd700 1 -- 172.21.2.112:0/4269658200 >> 172.21.2.112:0/4269658200 conn(0x7f47040ce690 msgr2=0x7f47040e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:10.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:10.081 7f4708bfd700 1 -- 172.21.2.112:0/4269658200 shutdown_connections 2021-02-10T11:14:10.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:10.081 7f4708bfd700 1 -- 172.21.2.112:0/4269658200 wait complete. 2021-02-10T11:14:10.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:10.095 7f4708bfd700 1 Processor -- start 2021-02-10T11:14:10.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:10.095 7f4708bfd700 1 -- start start 2021-02-10T11:14:10.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:10.095 7f4708bfd700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f470408d400 0x7f4704081230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:10.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:10.095 7f4708bfd700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4704081850 0x7f47040acf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:10.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:10.095 7f4708bfd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4704081c50 0x7f47040ad5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:10.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:10.095 7f4708bfd700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4704082110 con 0x7f4704081c50 2021-02-10T11:14:10.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:10.095 7f4708bfd700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f470408d800 con 0x7f4704081850 2021-02-10T11:14:10.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:10.095 7f4708bfd700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47040adbc0 con 0x7f470408d400 2021-02-10T11:14:10.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:10.096 7f470305f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4704081c50 0x7f47040ad5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:10.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:10.096 7f470305f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4704081c50 0x7f47040ad5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:42198/0 (socket says 172.21.2.112:42198) 2021-02-10T11:14:10.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:10.096 7f470305f700 1 -- 172.21.2.112:0/1865181620 learned_addr learned my addr 172.21.2.112:0/1865181620 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:14:10.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:10.096 7f470305f700 1 -- 172.21.2.112:0/1865181620 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f470408d400 msgr2=0x7f4704081230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:10.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:10.096 7f470305f700 1 --2- 172.21.2.112:0/1865181620 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f470408d400 0x7f4704081230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:10.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:10.096 7f470305f700 1 -- 172.21.2.112:0/1865181620 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4704081850 msgr2=0x7f47040acf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:10.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:10.096 7f470205d700 1 --2- 172.21.2.112:0/1865181620 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4704081850 0x7f47040acf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:10.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:10.096 7f470305f700 1 --2- 172.21.2.112:0/1865181620 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4704081850 0x7f47040acf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:10.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:10.096 7f470305f700 1 -- 172.21.2.112:0/1865181620 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4704187fc0 con 0x7f4704081c50 2021-02-10T11:14:10.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:10.096 7f470305f700 1 --2- 172.21.2.112:0/1865181620 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4704081c50 0x7f47040ad5a0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f46fc00a8a0 tx=0x7f46fc00ac40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:10.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:10.096 7f46fb7fe700 1 -- 172.21.2.112:0/1865181620 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f46fc0070d0 con 0x7f4704081c50 2021-02-10T11:14:10.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:10.096 7f46fb7fe700 1 -- 172.21.2.112:0/1865181620 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46fc007350 con 0x7f4704081c50 2021-02-10T11:14:10.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:10.096 7f4708bfd700 1 -- 172.21.2.112:0/1865181620 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4704187fc0 con 0x7f4704081c50 2021-02-10T11:14:10.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:10.096 7f4708bfd700 1 -- 172.21.2.112:0/1865181620 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4704188550 con 0x7f4704081c50 2021-02-10T11:14:10.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:10.096 7f46fb7fe700 1 -- 172.21.2.112:0/1865181620 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f46fc004920 con 0x7f4704081c50 2021-02-10T11:14:10.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:10.098 7f46fb7fe700 1 -- 172.21.2.112:0/1865181620 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f46fc0046a0 con 0x7f4704081c50 2021-02-10T11:14:10.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:10.098 7f46fb7fe700 1 --2- 172.21.2.112:0/1865181620 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f46e8018730 0x7f46e801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:10.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:10.098 7f46fb7fe700 1 -- 172.21.2.112:0/1865181620 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f46fc032b50 con 0x7f4704081c50 2021-02-10T11:14:10.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:10.098 7f470285e700 1 --2- 172.21.2.112:0/1865181620 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f46e8018730 0x7f46e801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:10.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:10.098 7f470285e700 1 --2- 172.21.2.112:0/1865181620 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f46e8018730 0x7f46e801abc0 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:10.127 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:10.128 7f4708bfd700 1 -- 172.21.2.112:0/1865181620 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f470407ed80 con 0x7f4704081c50 2021-02-10T11:14:10.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:10.132 7f46fb7fe700 1 -- 172.21.2.112:0/1865181620 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f46fc001230 con 0x7f4704081c50 2021-02-10T11:14:10.293 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:10.293 7f4708bfd700 1 -- 172.21.2.112:0/1865181620 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f4704082090 con 0x7f4704081c50 2021-02-10T11:14:10.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:10.295 7f46fb7fe700 1 -- 172.21.2.112:0/1865181620 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v3) v1 ==== 74+0+70186 (secure 0 0 0) 0x7f46fc007350 con 0x7f4704081c50 2021-02-10T11:14:10.297 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:14:10.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:10.299 7f4708bfd700 1 -- 172.21.2.112:0/1865181620 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f46e8018730 msgr2=0x7f46e801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:10.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:10.299 7f4708bfd700 1 --2- 172.21.2.112:0/1865181620 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f46e8018730 0x7f46e801abc0 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:10.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:10.299 7f4708bfd700 1 -- 172.21.2.112:0/1865181620 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4704081c50 msgr2=0x7f47040ad5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:10.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:10.299 7f4708bfd700 1 --2- 172.21.2.112:0/1865181620 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4704081c50 0x7f47040ad5a0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f46fc00a8a0 tx=0x7f46fc00ac40).stop 2021-02-10T11:14:10.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:10.299 7f4708bfd700 1 -- 172.21.2.112:0/1865181620 shutdown_connections 2021-02-10T11:14:10.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:10.299 7f4708bfd700 1 --2- 172.21.2.112:0/1865181620 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f46e8018730 0x7f46e801abc0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:10.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:10.299 7f4708bfd700 1 --2- 172.21.2.112:0/1865181620 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4704081850 0x7f47040acf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:10.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:10.299 7f4708bfd700 1 --2- 172.21.2.112:0/1865181620 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4704081c50 0x7f47040ad5a0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:10.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:10.299 7f4708bfd700 1 --2- 172.21.2.112:0/1865181620 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f470408d400 0x7f4704081230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:10.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:10.299 7f4708bfd700 1 -- 172.21.2.112:0/1865181620 >> 172.21.2.112:0/1865181620 conn(0x7f47041ac620 msgr2=0x7f47040896e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:10.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:10.299 7f4708bfd700 1 -- 172.21.2.112:0/1865181620 shutdown_connections 2021-02-10T11:14:10.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:10.299 7f4708bfd700 1 -- 172.21.2.112:0/1865181620 wait complete. 2021-02-10T11:14:10.307 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":3,"active_gid":4100,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6800","nonce":34990},{"type":"v1","addr":"172.21.2.120:6801","nonce":34990}]},"active_addr":"172.21.2.120:6801/34990","active_change":"2021-02-10 11:13:27.751969","available":true,"standbys":[{"gid":4098,"name":"y","available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"ansible","can_run":true,"error_string":"","module_options":{"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_url":{"name":"server_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_server":{"name":"verify_server","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"none","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_iterations":{"name":"upmap_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimization iterations","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_cephfs":{"name":"FEATURE_TOGGLE_cephfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_iscsi":{"name":"FEATURE_TOGGLE_iscsi","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_mirroring":{"name":"FEATURE_TOGGLE_mirroring","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rbd":{"name":"FEATURE_TOGGLE_rbd","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rgw":{"name":"FEATURE_TOGGLE_rgw","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"int","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"deepsea","can_run":true,"error_string":"","module_options":{"salt_api_eauth":{"name":"salt_api_eauth","type":"str","level":"advanced","flags":0,"default_value":"sharedsecret","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"salt_api_password":{"name":"salt_api_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"salt_api_url":{"name":"salt_api_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"salt_api_username":{"name":"salt_api_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_cloud","can_run":true,"error_string":"","module_options":{"diskprediction_cert_context":{"name":"diskprediction_cert_context","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_default_authority":{"name":"diskprediction_default_authority","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_password":{"name":"diskprediction_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_port":{"name":"diskprediction_port","type":"str","level":"advanced","flags":0,"default_value":"31400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_retrieve_prediction_interval":{"name":"diskprediction_retrieve_prediction_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_server":{"name":"diskprediction_server","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_ssl_target_name_override":{"name":"diskprediction_ssl_target_name_override","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_metrics_interval":{"name":"diskprediction_upload_metrics_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_smart_interval":{"name":"diskprediction_upload_smart_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_user":{"name":"diskprediction_user","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{}},{"name":"iostat","can_run":true,"error_string":"","module_options":{}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator_cli","can_run":true,"error_string":"","module_options":{"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{}},{"name":"restful","can_run":true,"error_string":"","module_options":{"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"ssh","can_run":true,"error_string":"","module_options":{"inventory_cache_timeout_min":{"name":"inventory_cache_timeout_min","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"ansible","can_run":true,"error_string":"","module_options":{"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_url":{"name":"server_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_server":{"name":"verify_server","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"none","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_iterations":{"name":"upmap_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimization iterations","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_cephfs":{"name":"FEATURE_TOGGLE_cephfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_iscsi":{"name":"FEATURE_TOGGLE_iscsi","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_mirroring":{"name":"FEATURE_TOGGLE_mirroring","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rbd":{"name":"FEATURE_TOGGLE_rbd","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rgw":{"name":"FEATURE_TOGGLE_rgw","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"int","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"deepsea","can_run":true,"error_string":"","module_options":{"salt_api_eauth":{"name":"salt_api_eauth","type":"str","level":"advanced","flags":0,"default_value":"sharedsecret","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"salt_api_password":{"name":"salt_api_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"salt_api_url":{"name":"salt_api_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"salt_api_username":{"name":"salt_api_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_cloud","can_run":true,"error_string":"","module_options":{"diskprediction_cert_context":{"name":"diskprediction_cert_context","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_default_authority":{"name":"diskprediction_default_authority","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_password":{"name":"diskprediction_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_port":{"name":"diskprediction_port","type":"str","level":"advanced","flags":0,"default_value":"31400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_retrieve_prediction_interval":{"name":"diskprediction_retrieve_prediction_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_server":{"name":"diskprediction_server","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_ssl_target_name_override":{"name":"diskprediction_ssl_target_name_override","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_metrics_interval":{"name":"diskprediction_upload_metrics_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_smart_interval":{"name":"diskprediction_upload_smart_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_user":{"name":"diskprediction_user","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{}},{"name":"iostat","can_run":true,"error_string":"","module_options":{}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator_cli","can_run":true,"error_string":"","module_options":{"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{}},{"name":"restful","can_run":true,"error_string":"","module_options":{"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"ssh","can_run":true,"error_string":"","module_options":{"inventory_cache_timeout_min":{"name":"inventory_cache_timeout_min","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"]}} 2021-02-10T11:14:10.309 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2021-02-10T11:14:11.117 DEBUG:teuthology.orchestra.run.gibba012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2021-02-10T11:14:11.536 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:11.519 7f70cc64b700 1 Processor -- start 2021-02-10T11:14:11.536 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:11.519 7f70cc64b700 1 -- start start 2021-02-10T11:14:11.536 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:11.520 7f70cc64b700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f70c40e5070 0x7f70c41b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:11.537 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:11.520 7f70cc64b700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70c407ed80 con 0x7f70c41b3e40 2021-02-10T11:14:11.537 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:11.520 7f70cc64b700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70c407e670 con 0x7f70c40e5470 2021-02-10T11:14:11.537 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:11.520 7f70cc64b700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70c407e890 con 0x7f70c40e5070 2021-02-10T11:14:11.537 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:11.520 7f70c9c66700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f70c41b3e40 0x7f70c407d690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:41664/0 (socket says 172.21.2.112:41664) 2021-02-10T11:14:11.537 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:11.520 7f70c9c66700 1 -- 172.21.2.112:0/2199233377 learned_addr learned my addr 172.21.2.112:0/2199233377 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:14:11.537 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:11.521 7f70c9465700 1 -- 172.21.2.112:0/2199233377 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2756056197 0 0) 0x7f70c00010b0 con 0x7f70c41b3e40 2021-02-10T11:14:11.537 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:11.521 7f70c9465700 1 -- 172.21.2.112:0/2199233377 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70b0002fe0 con 0x7f70c41b3e40 2021-02-10T11:14:11.537 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:11.521 7f70ca467700 1 --2- 172.21.2.112:0/2199233377 >> v2:172.21.2.120:3300/0 conn(0x7f70c40e5070 0x7f70c41b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:11.537 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:11.521 7f70c9465700 1 -- 172.21.2.112:0/2199233377 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 800000627 0 0) 0x7f70c00010b0 con 0x7f70c41b3e40 2021-02-10T11:14:11.537 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:11.521 7f70c9465700 1 -- 172.21.2.112:0/2199233377 >> v2:172.21.2.120:3300/0 conn(0x7f70c40e5070 msgr2=0x7f70c41b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:14:11.537 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:11.521 7f70c9465700 1 --2- 172.21.2.112:0/2199233377 >> v2:172.21.2.120:3300/0 conn(0x7f70c40e5070 0x7f70c41b3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:11.538 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:11.521 7f70c9465700 1 -- 172.21.2.112:0/2199233377 >> v1:172.21.2.120:6789/0 conn(0x7f70c40e5470 legacy=0x7f70c407de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:11.538 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:11.521 7f70c9465700 1 -- 172.21.2.112:0/2199233377 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70c407eab0 con 0x7f70c41b3e40 2021-02-10T11:14:11.538 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:11.521 7f70c9465700 1 -- 172.21.2.112:0/2199233377 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f70c00015b0 con 0x7f70c41b3e40 2021-02-10T11:14:11.538 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:11.521 7f70c9465700 1 -- 172.21.2.112:0/2199233377 >> v1:172.21.2.112:6789/0 conn(0x7f70c41b3e40 legacy=0x7f70c407d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:11.538 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:11.521 7f70c9465700 1 --2- 172.21.2.112:0/2199233377 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f70b0003b20 0x7f70b0006000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:11.538 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:11.521 7f70c9465700 1 --2- 172.21.2.112:0/2199233377 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f70b00068a0 0x7f70b0008dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:11.538 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:11.521 7f70c9465700 1 --2- 172.21.2.112:0/2199233377 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f70b0009640 0x7f70b000bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:11.538 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:11.521 7f70c9465700 1 -- 172.21.2.112:0/2199233377 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70b000c3b0 con 0x7f70b0003b20 2021-02-10T11:14:11.538 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:11.521 7f70c9465700 1 -- 172.21.2.112:0/2199233377 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70b000c5d0 con 0x7f70b00068a0 2021-02-10T11:14:11.538 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:11.521 7f70c9465700 1 -- 172.21.2.112:0/2199233377 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70b000c7f0 con 0x7f70b0009640 2021-02-10T11:14:11.538 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:11.521 7f70ca467700 1 --2- 172.21.2.112:0/2199233377 >> v2:172.21.2.120:3300/0 conn(0x7f70c40e5070 0x7f70c41b3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:14:11.538 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:11.521 7f70c9c66700 1 --2- 172.21.2.112:0/2199233377 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f70b00068a0 0x7f70b0008dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:11.539 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:11.521 7f70ca467700 1 --2- 172.21.2.112:0/2199233377 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f70b0003b20 0x7f70b0006000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:11.539 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:11.521 7f70cac68700 1 --2- 172.21.2.112:0/2199233377 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f70b0009640 0x7f70b000bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:11.539 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:11.522 7f70c9c66700 1 -- 172.21.2.112:0/2199233377 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f70b0009640 msgr2=0x7f70b000bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:14:11.539 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:11.522 7f70c9c66700 1 --2- 172.21.2.112:0/2199233377 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f70b0009640 0x7f70b000bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:11.539 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:11.522 7f70c9c66700 1 -- 172.21.2.112:0/2199233377 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f70b0003b20 msgr2=0x7f70b0006000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:14:11.539 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:11.522 7f70c9c66700 1 --2- 172.21.2.112:0/2199233377 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f70b0003b20 0x7f70b0006000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:11.539 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:11.522 7f70c9c66700 1 -- 172.21.2.112:0/2199233377 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70b0050a30 con 0x7f70b00068a0 2021-02-10T11:14:11.539 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:11.522 7f70cac68700 1 -- 172.21.2.112:0/2199233377 reap_dead start 2021-02-10T11:14:11.539 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:11.522 7f70ca467700 1 --2- 172.21.2.112:0/2199233377 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f70b0003b20 0x7f70b0006000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:14:11.539 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:11.522 7f70c9c66700 1 --2- 172.21.2.112:0/2199233377 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f70b00068a0 0x7f70b0008dc0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f70c0001360 tx=0x7f70c0007cf0).ready entity=mon.1 client_cookie=316da6d66612b381 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:11.539 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:11.522 7f70c9465700 1 -- 172.21.2.112:0/2199233377 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f70c000a520 con 0x7f70b00068a0 2021-02-10T11:14:11.539 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:11.522 7f70c9465700 1 -- 172.21.2.112:0/2199233377 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70c000a860 con 0x7f70b00068a0 2021-02-10T11:14:11.539 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:11.522 7f70c9465700 1 -- 172.21.2.112:0/2199233377 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f70c000ad40 con 0x7f70b00068a0 2021-02-10T11:14:11.540 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:11.522 7f70cc64b700 1 -- 172.21.2.112:0/2199233377 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f70b00068a0 msgr2=0x7f70b0008dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:11.540 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:11.522 7f70cc64b700 1 --2- 172.21.2.112:0/2199233377 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f70b00068a0 0x7f70b0008dc0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f70c0001360 tx=0x7f70c0007cf0).stop 2021-02-10T11:14:11.540 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:11.523 7f70cc64b700 1 -- 172.21.2.112:0/2199233377 shutdown_connections 2021-02-10T11:14:11.540 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:11.523 7f70cc64b700 1 --2- 172.21.2.112:0/2199233377 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f70b00068a0 0x7f70b0008dc0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:11.540 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:11.523 7f70cc64b700 1 -- 172.21.2.112:0/2199233377 >> 172.21.2.112:0/2199233377 conn(0x7f70c40ce690 msgr2=0x7f70c40e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:11.540 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:11.523 7f70cc64b700 1 -- 172.21.2.112:0/2199233377 shutdown_connections 2021-02-10T11:14:11.540 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:11.523 7f70cc64b700 1 -- 172.21.2.112:0/2199233377 wait complete. 2021-02-10T11:14:11.540 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:11.538 7f70cc64b700 1 Processor -- start 2021-02-10T11:14:11.540 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:11.538 7f70cc64b700 1 -- start start 2021-02-10T11:14:11.541 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:11.538 7f70cc64b700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f70c41999f0 0x7f70c408cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:11.542 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:11.538 7f70cc64b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f70c40b1290 0x7f70c408d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:11.542 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:11.538 7f70cc64b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f70c408d840 0x7f70c408aef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:11.542 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:11.538 7f70cc64b700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70c408dc40 con 0x7f70c40b1290 2021-02-10T11:14:11.542 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:11.538 7f70cc64b700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70c408de60 con 0x7f70c41999f0 2021-02-10T11:14:11.542 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:11.538 7f70cc64b700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70c408b510 con 0x7f70c408d840 2021-02-10T11:14:11.542 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:11.538 7f70c9c66700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f70c40b1290 0x7f70c408d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:11.542 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:11.538 7f70c9c66700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f70c40b1290 0x7f70c408d220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:42214/0 (socket says 172.21.2.112:42214) 2021-02-10T11:14:11.542 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:11.539 7f70c9c66700 1 -- 172.21.2.112:0/2407154631 learned_addr learned my addr 172.21.2.112:0/2407154631 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:14:11.542 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:11.539 7f70cac68700 1 --2- 172.21.2.112:0/2407154631 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f70c408d840 0x7f70c408aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:11.543 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:11.539 7f70ca467700 1 --2- 172.21.2.112:0/2407154631 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f70c41999f0 0x7f70c408cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:11.543 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:11.539 7f70c9c66700 1 -- 172.21.2.112:0/2407154631 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f70c408d840 msgr2=0x7f70c408aef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:11.543 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:11.539 7f70c9c66700 1 --2- 172.21.2.112:0/2407154631 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f70c408d840 0x7f70c408aef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:11.543 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:11.539 7f70c9c66700 1 -- 172.21.2.112:0/2407154631 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f70c41999f0 msgr2=0x7f70c408cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:11.543 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:11.539 7f70c9c66700 1 --2- 172.21.2.112:0/2407154631 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f70c41999f0 0x7f70c408cc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:11.543 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:11.539 7f70c9c66700 1 -- 172.21.2.112:0/2407154631 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70c408b730 con 0x7f70c40b1290 2021-02-10T11:14:11.543 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:11.539 7f70cac68700 1 --2- 172.21.2.112:0/2407154631 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f70c408d840 0x7f70c408aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:14:11.543 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:11.539 7f70ca467700 1 --2- 172.21.2.112:0/2407154631 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f70c41999f0 0x7f70c408cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:14:11.543 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:11.539 7f70c9c66700 1 --2- 172.21.2.112:0/2407154631 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f70c40b1290 0x7f70c408d220 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f70c00085b0 tx=0x7f70c0006950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:11.543 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:11.539 7f70b6ffd700 1 -- 172.21.2.112:0/2407154631 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f70c0006d40 con 0x7f70c40b1290 2021-02-10T11:14:11.544 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:11.539 7f70b6ffd700 1 -- 172.21.2.112:0/2407154631 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70c0006fc0 con 0x7f70c40b1290 2021-02-10T11:14:11.544 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:11.539 7f70cc64b700 1 -- 172.21.2.112:0/2407154631 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70c408b730 con 0x7f70c40b1290 2021-02-10T11:14:11.544 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:11.539 7f70b6ffd700 1 -- 172.21.2.112:0/2407154631 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f70c000b350 con 0x7f70c40b1290 2021-02-10T11:14:11.544 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:11.540 7f70cc64b700 1 -- 172.21.2.112:0/2407154631 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70c408bcc0 con 0x7f70c40b1290 2021-02-10T11:14:11.544 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:11.540 7f70b6ffd700 1 -- 172.21.2.112:0/2407154631 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f70c0000d40 con 0x7f70c40b1290 2021-02-10T11:14:11.544 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:11.540 7f70b6ffd700 1 --2- 172.21.2.112:0/2407154631 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f70ac018700 0x7f70ac01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:11.544 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:11.540 7f70b6ffd700 1 -- 172.21.2.112:0/2407154631 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f70c0032760 con 0x7f70c40b1290 2021-02-10T11:14:11.544 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:11.541 7f70ca467700 1 --2- 172.21.2.112:0/2407154631 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f70ac018700 0x7f70ac01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:11.544 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:11.541 7f70ca467700 1 --2- 172.21.2.112:0/2407154631 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f70ac018700 0x7f70ac01abc0 crc :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:11.569 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:11.570 7f70cc64b700 1 -- 172.21.2.112:0/2407154631 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70c408c6d0 con 0x7f70c40b1290 2021-02-10T11:14:11.571 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:11.573 7f70b6ffd700 1 -- 172.21.2.112:0/2407154631 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f70c0006fc0 con 0x7f70c40b1290 2021-02-10T11:14:11.695 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:11.697 7f70cc64b700 1 -- 172.21.2.112:0/2407154631 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f70c408c6d0 con 0x7f70c40b1290 2021-02-10T11:14:11.696 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:11.697 7f70b6ffd700 1 -- 172.21.2.112:0/2407154631 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+13993 (secure 0 0 0) 0x7f70c0000d40 con 0x7f70c40b1290 2021-02-10T11:14:11.696 INFO:teuthology.misc.health.gibba012.stdout: 2021-02-10T11:14:11.696 INFO:teuthology.misc.health.gibba012.stdout:{"epoch":20,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:14:04.629689","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:14:01.400797","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2021-02-10 11:14:02.622129","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:14:11.698 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:11.699 7f70cc64b700 1 -- 172.21.2.112:0/2407154631 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f70ac018700 msgr2=0x7f70ac01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:11.698 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:11.699 7f70cc64b700 1 --2- 172.21.2.112:0/2407154631 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f70ac018700 0x7f70ac01abc0 crc :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:11.698 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:11.699 7f70cc64b700 1 -- 172.21.2.112:0/2407154631 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f70c40b1290 msgr2=0x7f70c408d220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:11.698 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:11.699 7f70cc64b700 1 --2- 172.21.2.112:0/2407154631 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f70c40b1290 0x7f70c408d220 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f70c00085b0 tx=0x7f70c0006950).stop 2021-02-10T11:14:11.698 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:11.699 7f70cc64b700 1 -- 172.21.2.112:0/2407154631 shutdown_connections 2021-02-10T11:14:11.698 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:11.699 7f70cc64b700 1 --2- 172.21.2.112:0/2407154631 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f70ac018700 0x7f70ac01abc0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:11.699 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:11.699 7f70cc64b700 1 --2- 172.21.2.112:0/2407154631 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f70c408d840 0x7f70c408aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:11.699 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:11.699 7f70cc64b700 1 --2- 172.21.2.112:0/2407154631 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f70c40b1290 0x7f70c408d220 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:11.699 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:11.699 7f70cc64b700 1 --2- 172.21.2.112:0/2407154631 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f70c41999f0 0x7f70c408cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:11.699 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:11.699 7f70cc64b700 1 -- 172.21.2.112:0/2407154631 >> 172.21.2.112:0/2407154631 conn(0x7f70c41a31a0 msgr2=0x7f70c418c8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:11.699 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:11.699 7f70cc64b700 1 -- 172.21.2.112:0/2407154631 shutdown_connections 2021-02-10T11:14:11.699 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:11.699 7f70cc64b700 1 -- 172.21.2.112:0/2407154631 wait complete. 2021-02-10T11:14:11.705 DEBUG:teuthology.misc:8 of 8 OSDs are up 2021-02-10T11:14:11.706 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:14:12.133 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.117 7fb7cdf13700 1 Processor -- start 2021-02-10T11:14:12.133 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.117 7fb7cdf13700 1 -- start start 2021-02-10T11:14:12.133 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.118 7fb7cdf13700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fb7c807e6a0 0x7fb7c80d1790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:12.133 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.118 7fb7cdf13700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7c807ed80 con 0x7fb7c80e5070 2021-02-10T11:14:12.133 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.118 7fb7cdf13700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7c807eaa0 con 0x7fb7c80e5470 2021-02-10T11:14:12.133 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.118 7fb7cdf13700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7c81b4080 con 0x7fb7c807e6a0 2021-02-10T11:14:12.133 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.118 7fb7c6ffd700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7fb7c80e5470 0x7fb7c807f300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:35250/0 (socket says 172.21.2.112:35250) 2021-02-10T11:14:12.133 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.118 7fb7c6ffd700 1 -- 172.21.2.112:0/142937336 learned_addr learned my addr 172.21.2.112:0/142937336 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:14:12.133 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.118 7fb7c7fff700 1 --2- 172.21.2.112:0/142937336 >> v2:172.21.2.120:3300/0 conn(0x7fb7c807e6a0 0x7fb7c80d1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:12.134 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.118 7fb7c67fc700 1 -- 172.21.2.112:0/142937336 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4165611834 0 0) 0x7fb7b8000f30 con 0x7fb7c80e5070 2021-02-10T11:14:12.134 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.118 7fb7c67fc700 1 -- 172.21.2.112:0/142937336 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7b4002fe0 con 0x7fb7c80e5070 2021-02-10T11:14:12.134 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.118 7fb7c67fc700 1 -- 172.21.2.112:0/142937336 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3960895609 0 0) 0x7fb7b00010b0 con 0x7fb7c80e5470 2021-02-10T11:14:12.134 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.118 7fb7c67fc700 1 -- 172.21.2.112:0/142937336 --> v1:172.21.2.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7b4003090 con 0x7fb7c80e5470 2021-02-10T11:14:12.134 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.118 7fb7c7fff700 1 -- 172.21.2.112:0/142937336 >> v1:172.21.2.120:6789/0 conn(0x7fb7c80e5470 legacy=0x7fb7c807f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:12.134 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.118 7fb7c7fff700 1 -- 172.21.2.112:0/142937336 >> v1:172.21.2.112:6789/0 conn(0x7fb7c80e5070 legacy=0x7fb7c81b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:12.134 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.118 7fb7c7fff700 1 -- 172.21.2.112:0/142937336 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7c80d1db0 con 0x7fb7c807e6a0 2021-02-10T11:14:12.134 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.119 7fb7c7fff700 1 --2- 172.21.2.112:0/142937336 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb7c807e6a0 0x7fb7c80d1790 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7fb7c0006e50 tx=0x7fb7c0009e70).ready entity=mon.1 client_cookie=864b8c2efa94e7e2 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:12.134 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.119 7fb7c67fc700 1 -- 172.21.2.112:0/142937336 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb7c000b510 con 0x7fb7c807e6a0 2021-02-10T11:14:12.134 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.119 7fb7c67fc700 1 -- 172.21.2.112:0/142937336 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7c0006790 con 0x7fb7c807e6a0 2021-02-10T11:14:12.134 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.119 7fb7c67fc700 1 -- 172.21.2.112:0/142937336 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb7c000b940 con 0x7fb7c807e6a0 2021-02-10T11:14:12.134 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.119 7fb7cdf13700 1 -- 172.21.2.112:0/142937336 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb7c807e6a0 msgr2=0x7fb7c80d1790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:12.134 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.119 7fb7cdf13700 1 --2- 172.21.2.112:0/142937336 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb7c807e6a0 0x7fb7c80d1790 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7fb7c0006e50 tx=0x7fb7c0009e70).stop 2021-02-10T11:14:12.135 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.119 7fb7cdf13700 1 -- 172.21.2.112:0/142937336 shutdown_connections 2021-02-10T11:14:12.135 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.119 7fb7cdf13700 1 --2- 172.21.2.112:0/142937336 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb7c807e6a0 0x7fb7c80d1790 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:12.135 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.119 7fb7cdf13700 1 -- 172.21.2.112:0/142937336 >> 172.21.2.112:0/142937336 conn(0x7fb7c80ce690 msgr2=0x7fb7c80e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:12.135 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.120 7fb7cdf13700 1 -- 172.21.2.112:0/142937336 shutdown_connections 2021-02-10T11:14:12.135 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.120 7fb7cdf13700 1 -- 172.21.2.112:0/142937336 wait complete. 2021-02-10T11:14:12.135 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.134 7fb7cdf13700 1 Processor -- start 2021-02-10T11:14:12.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.135 7fb7cdf13700 1 -- start start 2021-02-10T11:14:12.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.135 7fb7cdf13700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb7c808d400 0x7fb7c8096c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:12.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.135 7fb7cdf13700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb7c80b1450 0x7fb7c80b1850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:12.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.135 7fb7cdf13700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb7c80b1e70 0x7fb7c81b1f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:12.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.135 7fb7cdf13700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7c8097310 con 0x7fb7c808d400 2021-02-10T11:14:12.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.135 7fb7cdf13700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7c808d800 con 0x7fb7c80b1e70 2021-02-10T11:14:12.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.135 7fb7cdf13700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7c808d9f0 con 0x7fb7c80b1450 2021-02-10T11:14:12.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.135 7fb7c77fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb7c808d400 0x7fb7c8096c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:12.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.135 7fb7c77fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb7c808d400 0x7fb7c8096c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:42236/0 (socket says 172.21.2.112:42236) 2021-02-10T11:14:12.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.135 7fb7c77fe700 1 -- 172.21.2.112:0/2342921344 learned_addr learned my addr 172.21.2.112:0/2342921344 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:14:12.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.135 7fb7c6ffd700 1 --2- 172.21.2.112:0/2342921344 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb7c80b1450 0x7fb7c80b1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:12.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.135 7fb7c7fff700 1 --2- 172.21.2.112:0/2342921344 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb7c80b1e70 0x7fb7c81b1f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:12.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.135 7fb7c77fe700 1 -- 172.21.2.112:0/2342921344 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb7c80b1450 msgr2=0x7fb7c80b1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:12.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.135 7fb7c77fe700 1 --2- 172.21.2.112:0/2342921344 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb7c80b1450 0x7fb7c80b1850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:12.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.135 7fb7c77fe700 1 -- 172.21.2.112:0/2342921344 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb7c80b1e70 msgr2=0x7fb7c81b1f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:12.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.135 7fb7c77fe700 1 --2- 172.21.2.112:0/2342921344 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb7c80b1e70 0x7fb7c81b1f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:12.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.135 7fb7c77fe700 1 -- 172.21.2.112:0/2342921344 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7c81b25f0 con 0x7fb7c808d400 2021-02-10T11:14:12.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.135 7fb7c77fe700 1 --2- 172.21.2.112:0/2342921344 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb7c808d400 0x7fb7c8096c70 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7fb7b8006f10 tx=0x7fb7b800b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:12.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.136 7fb7bffff700 1 -- 172.21.2.112:0/2342921344 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb7b800bd80 con 0x7fb7c808d400 2021-02-10T11:14:12.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.136 7fb7bffff700 1 -- 172.21.2.112:0/2342921344 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7b8001120 con 0x7fb7c808d400 2021-02-10T11:14:12.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.136 7fb7cdf13700 1 -- 172.21.2.112:0/2342921344 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7c81b25f0 con 0x7fb7c808d400 2021-02-10T11:14:12.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.136 7fb7bffff700 1 -- 172.21.2.112:0/2342921344 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb7b80015c0 con 0x7fb7c808d400 2021-02-10T11:14:12.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.136 7fb7cdf13700 1 -- 172.21.2.112:0/2342921344 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7c81b25f0 con 0x7fb7c808d400 2021-02-10T11:14:12.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.136 7fb7bffff700 1 -- 172.21.2.112:0/2342921344 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb7b8000d60 con 0x7fb7c808d400 2021-02-10T11:14:12.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.136 7fb7bffff700 1 --2- 172.21.2.112:0/2342921344 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb7ac0186c0 0x7fb7ac01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:12.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.136 7fb7bffff700 1 -- 172.21.2.112:0/2342921344 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fb7b8032550 con 0x7fb7c808d400 2021-02-10T11:14:12.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.136 7fb7c6ffd700 1 --2- 172.21.2.112:0/2342921344 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb7ac0186c0 0x7fb7ac01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:12.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.137 7fb7c6ffd700 1 --2- 172.21.2.112:0/2342921344 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb7ac0186c0 0x7fb7ac01abc0 crc :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:12.165 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.167 7fb7cdf13700 1 -- 172.21.2.112:0/2342921344 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7c807ed80 con 0x7fb7c808d400 2021-02-10T11:14:12.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.169 7fb7bffff700 1 -- 172.21.2.112:0/2342921344 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb7b8006ba0 con 0x7fb7c808d400 2021-02-10T11:14:12.295 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.296 7fb7cdf13700 1 -- 172.21.2.112:0/2342921344 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb7c807ed80 con 0x7fb7c808d400 2021-02-10T11:14:12.296 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.297 7fb7bffff700 1 -- 172.21.2.112:0/2342921344 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+13993 (secure 0 0 0) 0x7fb7b8000d60 con 0x7fb7c808d400 2021-02-10T11:14:12.296 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:14:12.296 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":20,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:14:04.629689","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:14:01.400797","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2021-02-10 11:14:02.622129","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:14:12.297 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.298 7fb7cdf13700 1 -- 172.21.2.112:0/2342921344 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb7ac0186c0 msgr2=0x7fb7ac01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:12.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.298 7fb7cdf13700 1 --2- 172.21.2.112:0/2342921344 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb7ac0186c0 0x7fb7ac01abc0 crc :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:12.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.298 7fb7cdf13700 1 -- 172.21.2.112:0/2342921344 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb7c808d400 msgr2=0x7fb7c8096c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:12.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.298 7fb7cdf13700 1 --2- 172.21.2.112:0/2342921344 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb7c808d400 0x7fb7c8096c70 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7fb7b8006f10 tx=0x7fb7b800b990).stop 2021-02-10T11:14:12.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.299 7fb7cdf13700 1 -- 172.21.2.112:0/2342921344 shutdown_connections 2021-02-10T11:14:12.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.299 7fb7cdf13700 1 --2- 172.21.2.112:0/2342921344 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb7ac0186c0 0x7fb7ac01abc0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:12.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.299 7fb7cdf13700 1 --2- 172.21.2.112:0/2342921344 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb7c80b1e70 0x7fb7c81b1f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:12.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.299 7fb7cdf13700 1 --2- 172.21.2.112:0/2342921344 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb7c80b1450 0x7fb7c80b1850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:12.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.299 7fb7cdf13700 1 --2- 172.21.2.112:0/2342921344 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb7c808d400 0x7fb7c8096c70 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:12.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.299 7fb7cdf13700 1 -- 172.21.2.112:0/2342921344 >> 172.21.2.112:0/2342921344 conn(0x7fb7c81ac620 msgr2=0x7fb7c80896e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:12.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.299 7fb7cdf13700 1 -- 172.21.2.112:0/2342921344 shutdown_connections 2021-02-10T11:14:12.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.299 7fb7cdf13700 1 -- 172.21.2.112:0/2342921344 wait complete. 2021-02-10T11:14:12.307 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell osd.0 flush_pg_stats 2021-02-10T11:14:12.734 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.719 7f8237fe3700 1 Processor -- start 2021-02-10T11:14:12.734 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.719 7f8237fe3700 1 -- start start 2021-02-10T11:14:12.734 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.720 7f8237fe3700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f823007e6a0 0x7f82300d1790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:12.735 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.720 7f8237fe3700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f823007ed80 con 0x7f82300e5470 2021-02-10T11:14:12.735 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.720 7f8237fe3700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f823007eaa0 con 0x7f82300e5070 2021-02-10T11:14:12.735 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.720 7f8237fe3700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82301b4080 con 0x7f823007e6a0 2021-02-10T11:14:12.735 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.720 7f82355fe700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f82300e5470 0x7f823007f300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:41706/0 (socket says 172.21.2.112:41706) 2021-02-10T11:14:12.735 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.720 7f82355fe700 1 -- 172.21.2.112:0/3991625327 learned_addr learned my addr 172.21.2.112:0/3991625327 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:14:12.735 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.720 7f8236600700 1 --2- 172.21.2.112:0/3991625327 >> v2:172.21.2.112:3300/0 conn(0x7f823007e6a0 0x7f82300d1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:12.735 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.720 7f8236600700 1 -- 172.21.2.112:0/3991625327 >> v1:172.21.2.120:6789/0 conn(0x7f82300e5070 legacy=0x7f82301b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:12.735 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.720 7f8236600700 1 -- 172.21.2.112:0/3991625327 >> v1:172.21.2.112:6789/0 conn(0x7f82300e5470 legacy=0x7f823007f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:12.735 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.720 7f8234dfd700 1 -- 172.21.2.112:0/3991625327 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4292417616 0 0) 0x7f8220001180 con 0x7f82300e5470 2021-02-10T11:14:12.735 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.720 7f8236600700 1 -- 172.21.2.112:0/3991625327 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82300d1db0 con 0x7f823007e6a0 2021-02-10T11:14:12.735 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.720 7f8236600700 1 --2- 172.21.2.112:0/3991625327 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f823007e6a0 0x7f82300d1790 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f8228006e30 tx=0x7f8228009e60).ready entity=mon.0 client_cookie=923a45ac9681cae6 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:12.735 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.721 7f8234dfd700 1 -- 172.21.2.112:0/3991625327 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8228003030 con 0x7f823007e6a0 2021-02-10T11:14:12.736 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.721 7f8234dfd700 1 -- 172.21.2.112:0/3991625327 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f822800b300 con 0x7f823007e6a0 2021-02-10T11:14:12.736 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.721 7f8234dfd700 1 -- 172.21.2.112:0/3991625327 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f822800b820 con 0x7f823007e6a0 2021-02-10T11:14:12.736 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.721 7f8237fe3700 1 -- 172.21.2.112:0/3991625327 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f823007e6a0 msgr2=0x7f82300d1790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:12.736 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.721 7f8237fe3700 1 --2- 172.21.2.112:0/3991625327 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f823007e6a0 0x7f82300d1790 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f8228006e30 tx=0x7f8228009e60).stop 2021-02-10T11:14:12.736 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.721 7f8237fe3700 1 -- 172.21.2.112:0/3991625327 shutdown_connections 2021-02-10T11:14:12.736 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.721 7f8237fe3700 1 --2- 172.21.2.112:0/3991625327 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f823007e6a0 0x7f82300d1790 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:12.736 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.721 7f8237fe3700 1 -- 172.21.2.112:0/3991625327 >> 172.21.2.112:0/3991625327 conn(0x7f82300ce690 msgr2=0x7f82300e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:12.736 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.721 7f8237fe3700 1 -- 172.21.2.112:0/3991625327 shutdown_connections 2021-02-10T11:14:12.736 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.721 7f8237fe3700 1 -- 172.21.2.112:0/3991625327 wait complete. 2021-02-10T11:14:12.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.736 7f8237fe3700 1 Processor -- start 2021-02-10T11:14:12.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.736 7f8237fe3700 1 -- start start 2021-02-10T11:14:12.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.737 7f8237fe3700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f82301952f0 0x7f82300be2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:12.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.737 7f8237fe3700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f82301a35e0 0x7f82300be900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:12.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.737 7f8237fe3700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f82300bef20 0x7f82300c1a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:12.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.737 7f8237fe3700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82300bf4b0 con 0x7f82301952f0 2021-02-10T11:14:12.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.737 7f8237fe3700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82300bf6d0 con 0x7f82300bef20 2021-02-10T11:14:12.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.737 7f8237fe3700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82300c2080 con 0x7f82301a35e0 2021-02-10T11:14:12.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.737 7f8235dff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f82301952f0 0x7f82300be2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:12.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.737 7f8235dff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f82301952f0 0x7f82300be2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:42252/0 (socket says 172.21.2.112:42252) 2021-02-10T11:14:12.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.737 7f8235dff700 1 -- 172.21.2.112:0/2196604981 learned_addr learned my addr 172.21.2.112:0/2196604981 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:14:12.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.737 7f8236600700 1 --2- 172.21.2.112:0/2196604981 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f82300bef20 0x7f82300c1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:12.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.737 7f8235dff700 1 -- 172.21.2.112:0/2196604981 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f82301a35e0 msgr2=0x7f82300be900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:12.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.737 7f8235dff700 1 --2- 172.21.2.112:0/2196604981 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f82301a35e0 0x7f82300be900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:12.740 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.737 7f8235dff700 1 -- 172.21.2.112:0/2196604981 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f82300bef20 msgr2=0x7f82300c1a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:12.740 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.737 7f8235dff700 1 --2- 172.21.2.112:0/2196604981 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f82300bef20 0x7f82300c1a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:12.740 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.737 7f8235dff700 1 -- 172.21.2.112:0/2196604981 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82300c2260 con 0x7f82301952f0 2021-02-10T11:14:12.740 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.737 7f8236600700 1 --2- 172.21.2.112:0/2196604981 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f82300bef20 0x7f82300c1a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:14:12.740 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.737 7f8235dff700 1 --2- 172.21.2.112:0/2196604981 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f82301952f0 0x7f82300be2e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f822c00b530 tx=0x7f822c00b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:12.740 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.738 7f82267fc700 1 -- 172.21.2.112:0/2196604981 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f822c00bcc0 con 0x7f82301952f0 2021-02-10T11:14:12.740 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.738 7f82267fc700 1 -- 172.21.2.112:0/2196604981 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f822c006020 con 0x7f82301952f0 2021-02-10T11:14:12.740 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.738 7f82267fc700 1 -- 172.21.2.112:0/2196604981 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f822c0054a0 con 0x7f82301952f0 2021-02-10T11:14:12.740 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.738 7f8237fe3700 1 -- 172.21.2.112:0/2196604981 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82300c2260 con 0x7f82301952f0 2021-02-10T11:14:12.740 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.738 7f8237fe3700 1 -- 172.21.2.112:0/2196604981 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82300c2260 con 0x7f82301952f0 2021-02-10T11:14:12.740 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.738 7f82267fc700 1 -- 172.21.2.112:0/2196604981 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f822c006200 con 0x7f82301952f0 2021-02-10T11:14:12.740 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.738 7f82267fc700 1 --2- 172.21.2.112:0/2196604981 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f82180186c0 0x7f821801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:12.740 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.738 7f82267fc700 1 -- 172.21.2.112:0/2196604981 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f822c0325b0 con 0x7f82301952f0 2021-02-10T11:14:12.741 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.739 7f82355fe700 1 --2- 172.21.2.112:0/2196604981 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f82180186c0 0x7f821801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:12.741 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.739 7f82355fe700 1 --2- 172.21.2.112:0/2196604981 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f82180186c0 0x7f821801abc0 crc :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:12.768 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.769 7f8237fe3700 1 --2- 172.21.2.112:0/2196604981 >> [v2:172.21.2.112:6817/36030,v1:172.21.2.112:6818/36030] conn(0x7f82300c45b0 0x7f82300c2e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:12.768 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.769 7f8237fe3700 1 -- 172.21.2.112:0/2196604981 --> [v2:172.21.2.112:6817/36030,v1:172.21.2.112:6818/36030] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8230083e60 con 0x7f82300c45b0 2021-02-10T11:14:12.768 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.769 7f8236600700 1 --2- 172.21.2.112:0/2196604981 >> [v2:172.21.2.112:6817/36030,v1:172.21.2.112:6818/36030] conn(0x7f82300c45b0 0x7f82300c2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:12.769 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.770 7f8236600700 1 --2- 172.21.2.112:0/2196604981 >> [v2:172.21.2.112:6817/36030,v1:172.21.2.112:6818/36030] conn(0x7f82300c45b0 0x7f82300c2e50 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:12.769 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.770 7f82267fc700 1 -- 172.21.2.112:0/2196604981 <== osd.0 v2:172.21.2.112:6817/36030 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+5151 (crc 0 0 0) 0x7f82280094d0 con 0x7f82300c45b0 2021-02-10T11:14:12.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.777 7f8237fe3700 1 -- 172.21.2.112:0/2196604981 --> [v2:172.21.2.112:6817/36030,v1:172.21.2.112:6818/36030] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f8230083e60 con 0x7f82300c45b0 2021-02-10T11:14:12.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.777 7f82267fc700 1 -- 172.21.2.112:0/2196604981 <== osd.0 v2:172.21.2.112:6817/36030 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f82280094d0 con 0x7f82300c45b0 2021-02-10T11:14:12.776 INFO:teuthology.orchestra.run.gibba012.stdout:55834574854 2021-02-10T11:14:12.777 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.778 7f8237fe3700 1 -- 172.21.2.112:0/2196604981 >> [v2:172.21.2.112:6817/36030,v1:172.21.2.112:6818/36030] conn(0x7f82300c45b0 msgr2=0x7f82300c2e50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:12.777 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.778 7f8237fe3700 1 --2- 172.21.2.112:0/2196604981 >> [v2:172.21.2.112:6817/36030,v1:172.21.2.112:6818/36030] conn(0x7f82300c45b0 0x7f82300c2e50 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:12.777 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.778 7f8237fe3700 1 -- 172.21.2.112:0/2196604981 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f82180186c0 msgr2=0x7f821801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:12.777 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.778 7f8237fe3700 1 --2- 172.21.2.112:0/2196604981 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f82180186c0 0x7f821801abc0 crc :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:12.777 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.779 7f8237fe3700 1 -- 172.21.2.112:0/2196604981 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f82301952f0 msgr2=0x7f82300be2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:12.778 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.779 7f8237fe3700 1 --2- 172.21.2.112:0/2196604981 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f82301952f0 0x7f82300be2e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f822c00b530 tx=0x7f822c00b8d0).stop 2021-02-10T11:14:12.778 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.779 7f8236600700 1 -- 172.21.2.112:0/2196604981 reap_dead start 2021-02-10T11:14:12.778 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.779 7f8237fe3700 1 -- 172.21.2.112:0/2196604981 shutdown_connections 2021-02-10T11:14:12.778 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.779 7f8237fe3700 1 -- 172.21.2.112:0/2196604981 >> 172.21.2.112:0/2196604981 conn(0x7f8230085a70 msgr2=0x7f82300a5790 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:12.778 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.779 7f8237fe3700 1 -- 172.21.2.112:0/2196604981 shutdown_connections 2021-02-10T11:14:12.778 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:12.779 7f8237fe3700 1 -- 172.21.2.112:0/2196604981 wait complete. 2021-02-10T11:14:12.785 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell osd.1 flush_pg_stats 2021-02-10T11:14:13.210 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.195 7fa10ef59700 1 Processor -- start 2021-02-10T11:14:13.210 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.195 7fa10ef59700 1 -- start start 2021-02-10T11:14:13.210 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.195 7fa10ef59700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa1080e5070 0x7fa1081b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:13.210 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.195 7fa10ef59700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fa1081b3e40 0x7fa108080310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:13.210 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.195 7fa10ef59700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa10807ed80 con 0x7fa1080e5470 2021-02-10T11:14:13.211 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.195 7fa10ef59700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1080d1790 con 0x7fa1081b3e40 2021-02-10T11:14:13.211 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.195 7fa10ef59700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1080d1980 con 0x7fa1080e5070 2021-02-10T11:14:13.211 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.196 7fa107fff700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fa1080e5470 0x7fa10807d690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:41724/0 (socket says 172.21.2.112:41724) 2021-02-10T11:14:13.211 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.196 7fa107fff700 1 -- 172.21.2.112:0/1019677679 learned_addr learned my addr 172.21.2.112:0/1019677679 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:14:13.211 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.196 7fa10d576700 1 --2- 172.21.2.112:0/1019677679 >> v2:172.21.2.120:3300/0 conn(0x7fa1081b3e40 0x7fa108080310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:13.211 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.196 7fa10cd75700 1 --2- 172.21.2.112:0/1019677679 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa1080e5070 0x7fa1081b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:13.211 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.196 7fa10d576700 1 -- 172.21.2.112:0/1019677679 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa1080e5070 msgr2=0x7fa1081b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:14:13.211 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.196 7fa10d576700 1 --2- 172.21.2.112:0/1019677679 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa1080e5070 0x7fa1081b3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:13.211 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.196 7fa10d576700 1 -- 172.21.2.112:0/1019677679 >> v1:172.21.2.112:6789/0 conn(0x7fa1080e5470 legacy=0x7fa10807d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:13.211 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.196 7fa1077fe700 1 -- 172.21.2.112:0/1019677679 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 750751218 0 0) 0x7fa0f8000ce0 con 0x7fa1080e5470 2021-02-10T11:14:13.211 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.196 7fa10d576700 1 -- 172.21.2.112:0/1019677679 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1080d1ba0 con 0x7fa1081b3e40 2021-02-10T11:14:13.211 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.196 7fa10cd75700 1 --2- 172.21.2.112:0/1019677679 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa1080e5070 0x7fa1081b3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:14:13.211 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.196 7fa10d576700 1 --2- 172.21.2.112:0/1019677679 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa1081b3e40 0x7fa108080310 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7fa100006e50 tx=0x7fa100009e70).ready entity=mon.1 client_cookie=7ad8da9d7a8f9048 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:13.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.196 7fa1077fe700 1 -- 172.21.2.112:0/1019677679 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa100006a20 con 0x7fa1081b3e40 2021-02-10T11:14:13.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.196 7fa1077fe700 1 -- 172.21.2.112:0/1019677679 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa100006790 con 0x7fa1081b3e40 2021-02-10T11:14:13.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.196 7fa1077fe700 1 -- 172.21.2.112:0/1019677679 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa10000b700 con 0x7fa1081b3e40 2021-02-10T11:14:13.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.196 7fa10ef59700 1 -- 172.21.2.112:0/1019677679 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa1081b3e40 msgr2=0x7fa108080310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:13.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.196 7fa10ef59700 1 --2- 172.21.2.112:0/1019677679 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa1081b3e40 0x7fa108080310 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7fa100006e50 tx=0x7fa100009e70).stop 2021-02-10T11:14:13.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.197 7fa10ef59700 1 -- 172.21.2.112:0/1019677679 shutdown_connections 2021-02-10T11:14:13.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.197 7fa10ef59700 1 --2- 172.21.2.112:0/1019677679 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa1081b3e40 0x7fa108080310 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:13.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.197 7fa10ef59700 1 --2- 172.21.2.112:0/1019677679 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa1080e5070 0x7fa1081b3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:13.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.197 7fa10ef59700 1 -- 172.21.2.112:0/1019677679 >> 172.21.2.112:0/1019677679 conn(0x7fa1080ce690 msgr2=0x7fa1080e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:13.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.197 7fa10ef59700 1 -- 172.21.2.112:0/1019677679 shutdown_connections 2021-02-10T11:14:13.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.197 7fa10ef59700 1 -- 172.21.2.112:0/1019677679 wait complete. 2021-02-10T11:14:13.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.212 7fa10ef59700 1 Processor -- start 2021-02-10T11:14:13.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.212 7fa10ef59700 1 -- start start 2021-02-10T11:14:13.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.212 7fa10ef59700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa1080b4ad0 0x7fa1080a78d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:13.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.212 7fa10ef59700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa1080a7ef0 0x7fa1081a6d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:13.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.212 7fa10ef59700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa1080a82f0 0x7fa1081a73b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:13.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.212 7fa10ef59700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1080aa3c0 con 0x7fa1080a82f0 2021-02-10T11:14:13.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.212 7fa10ef59700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1080a88c0 con 0x7fa1080b4ad0 2021-02-10T11:14:13.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.212 7fa10ef59700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1080b4ed0 con 0x7fa1080a7ef0 2021-02-10T11:14:13.216 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.213 7fa10d576700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa1080a82f0 0x7fa1081a73b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:13.216 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.213 7fa10d576700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa1080a82f0 0x7fa1081a73b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:42268/0 (socket says 172.21.2.112:42268) 2021-02-10T11:14:13.216 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.213 7fa10d576700 1 -- 172.21.2.112:0/2362605933 learned_addr learned my addr 172.21.2.112:0/2362605933 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:14:13.216 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.213 7fa10cd75700 1 --2- 172.21.2.112:0/2362605933 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa1080b4ad0 0x7fa1080a78d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:13.216 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.213 7fa107fff700 1 --2- 172.21.2.112:0/2362605933 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa1080a7ef0 0x7fa1081a6d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:13.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.213 7fa10d576700 1 -- 172.21.2.112:0/2362605933 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa1080a7ef0 msgr2=0x7fa1081a6d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:13.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.213 7fa10d576700 1 --2- 172.21.2.112:0/2362605933 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa1080a7ef0 0x7fa1081a6d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:13.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.213 7fa10d576700 1 -- 172.21.2.112:0/2362605933 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa1080b4ad0 msgr2=0x7fa1080a78d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:13.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.213 7fa10d576700 1 --2- 172.21.2.112:0/2362605933 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa1080b4ad0 0x7fa1080a78d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:13.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.213 7fa10d576700 1 -- 172.21.2.112:0/2362605933 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1081a79d0 con 0x7fa1080a82f0 2021-02-10T11:14:13.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.213 7fa10d576700 1 --2- 172.21.2.112:0/2362605933 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa1080a82f0 0x7fa1081a73b0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7fa100006e60 tx=0x7fa10000bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:13.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.213 7fa1057fa700 1 -- 172.21.2.112:0/2362605933 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa100000df0 con 0x7fa1080a82f0 2021-02-10T11:14:13.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.214 7fa10ef59700 1 -- 172.21.2.112:0/2362605933 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1081a79d0 con 0x7fa1080a82f0 2021-02-10T11:14:13.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.214 7fa1057fa700 1 -- 172.21.2.112:0/2362605933 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa100000df0 con 0x7fa1080a82f0 2021-02-10T11:14:13.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.214 7fa1057fa700 1 -- 172.21.2.112:0/2362605933 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa1000011a0 con 0x7fa1080a82f0 2021-02-10T11:14:13.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.214 7fa10ef59700 1 -- 172.21.2.112:0/2362605933 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1080a9130 con 0x7fa1080a82f0 2021-02-10T11:14:13.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.214 7fa1057fa700 1 -- 172.21.2.112:0/2362605933 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa100009130 con 0x7fa1080a82f0 2021-02-10T11:14:13.218 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.214 7fa1057fa700 1 --2- 172.21.2.112:0/2362605933 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa0ec0186c0 0x7fa0ec01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:13.218 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.214 7fa1057fa700 1 -- 172.21.2.112:0/2362605933 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fa100032a80 con 0x7fa1080a82f0 2021-02-10T11:14:13.218 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.215 7fa10cd75700 1 --2- 172.21.2.112:0/2362605933 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa0ec0186c0 0x7fa0ec01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:13.218 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.215 7fa10cd75700 1 --2- 172.21.2.112:0/2362605933 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa0ec0186c0 0x7fa0ec01abc0 crc :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:13.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.245 7fa10ef59700 1 --2- 172.21.2.112:0/2362605933 >> [v2:172.21.2.112:6800/36029,v1:172.21.2.112:6802/36029] conn(0x7fa1080c45a0 0x7fa1080a99c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:13.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.245 7fa10ef59700 1 -- 172.21.2.112:0/2362605933 --> [v2:172.21.2.112:6800/36029,v1:172.21.2.112:6802/36029] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa1080a9130 con 0x7fa1080c45a0 2021-02-10T11:14:13.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.245 7fa107fff700 1 --2- 172.21.2.112:0/2362605933 >> [v2:172.21.2.112:6800/36029,v1:172.21.2.112:6802/36029] conn(0x7fa1080c45a0 0x7fa1080a99c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:13.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.246 7fa107fff700 1 --2- 172.21.2.112:0/2362605933 >> [v2:172.21.2.112:6800/36029,v1:172.21.2.112:6802/36029] conn(0x7fa1080c45a0 0x7fa1080a99c0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:13.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.246 7fa1057fa700 1 -- 172.21.2.112:0/2362605933 <== osd.1 v2:172.21.2.112:6800/36029 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+5151 (crc 0 0 0) 0x7fa0f8006550 con 0x7fa1080c45a0 2021-02-10T11:14:13.252 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.253 7fa10ef59700 1 -- 172.21.2.112:0/2362605933 --> [v2:172.21.2.112:6800/36029,v1:172.21.2.112:6802/36029] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fa1081a79d0 con 0x7fa1080c45a0 2021-02-10T11:14:13.252 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.253 7fa1057fa700 1 -- 172.21.2.112:0/2362605933 <== osd.1 v2:172.21.2.112:6800/36029 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fa0f8006550 con 0x7fa1080c45a0 2021-02-10T11:14:13.252 INFO:teuthology.orchestra.run.gibba012.stdout:55834574853 2021-02-10T11:14:13.253 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.254 7fa10ef59700 1 -- 172.21.2.112:0/2362605933 >> [v2:172.21.2.112:6800/36029,v1:172.21.2.112:6802/36029] conn(0x7fa1080c45a0 msgr2=0x7fa1080a99c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:13.253 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.254 7fa10ef59700 1 --2- 172.21.2.112:0/2362605933 >> [v2:172.21.2.112:6800/36029,v1:172.21.2.112:6802/36029] conn(0x7fa1080c45a0 0x7fa1080a99c0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:13.253 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.254 7fa10ef59700 1 -- 172.21.2.112:0/2362605933 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa0ec0186c0 msgr2=0x7fa0ec01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:13.253 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.254 7fa10ef59700 1 --2- 172.21.2.112:0/2362605933 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa0ec0186c0 0x7fa0ec01abc0 crc :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:13.253 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.254 7fa10ef59700 1 -- 172.21.2.112:0/2362605933 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa1080a82f0 msgr2=0x7fa1081a73b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:13.254 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.254 7fa10ef59700 1 --2- 172.21.2.112:0/2362605933 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa1080a82f0 0x7fa1081a73b0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7fa100006e60 tx=0x7fa10000bcf0).stop 2021-02-10T11:14:13.254 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.254 7fa10d576700 1 -- 172.21.2.112:0/2362605933 reap_dead start 2021-02-10T11:14:13.254 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.255 7fa10ef59700 1 -- 172.21.2.112:0/2362605933 shutdown_connections 2021-02-10T11:14:13.254 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.255 7fa10ef59700 1 -- 172.21.2.112:0/2362605933 >> 172.21.2.112:0/2362605933 conn(0x7fa1080b5560 msgr2=0x7fa1080a6200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:13.254 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.255 7fa10ef59700 1 -- 172.21.2.112:0/2362605933 shutdown_connections 2021-02-10T11:14:13.254 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.255 7fa10ef59700 1 -- 172.21.2.112:0/2362605933 wait complete. 2021-02-10T11:14:13.261 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell osd.2 flush_pg_stats 2021-02-10T11:14:13.686 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.671 7f91740b9700 1 Processor -- start 2021-02-10T11:14:13.686 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.671 7f91740b9700 1 -- start start 2021-02-10T11:14:13.686 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.672 7f91740b9700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f916c0e5470 0x7f916c07e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:13.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.672 7f91740b9700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f916c07ed80 con 0x7f916c07f300 2021-02-10T11:14:13.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.672 7f91740b9700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f916c1b4010 con 0x7f916c0e5070 2021-02-10T11:14:13.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.672 7f91740b9700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f916c0d1790 con 0x7f916c0e5470 2021-02-10T11:14:13.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.672 7f91726d6700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f916c07f300 0x7f916c07f700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:41744/0 (socket says 172.21.2.112:41744) 2021-02-10T11:14:13.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.672 7f91726d6700 1 -- 172.21.2.112:0/3474212327 learned_addr learned my addr 172.21.2.112:0/3474212327 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:14:13.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.672 7f91716d4700 1 --2- 172.21.2.112:0/3474212327 >> v2:172.21.2.112:3300/0 conn(0x7f916c0e5470 0x7f916c07e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:13.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.672 7f9170ed3700 1 -- 172.21.2.112:0/3474212327 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2220053313 0 0) 0x7f9164001500 con 0x7f916c07f300 2021-02-10T11:14:13.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.672 7f9170ed3700 1 -- 172.21.2.112:0/3474212327 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9158002fe0 con 0x7f916c07f300 2021-02-10T11:14:13.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.673 7f91716d4700 1 -- 172.21.2.112:0/3474212327 >> v1:172.21.2.120:6789/0 conn(0x7f916c0e5070 legacy=0x7f916c1b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:14:13.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.673 7f91716d4700 1 -- 172.21.2.112:0/3474212327 >> v1:172.21.2.112:6789/0 conn(0x7f916c07f300 legacy=0x7f916c07f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:13.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.673 7f91716d4700 1 -- 172.21.2.112:0/3474212327 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f916c0d19b0 con 0x7f916c0e5470 2021-02-10T11:14:13.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.673 7f91716d4700 1 --2- 172.21.2.112:0/3474212327 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f916c0e5470 0x7f916c07e6a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f9160006650 tx=0x7f9160009a80).ready entity=mon.0 client_cookie=9316b57fa37317c6 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:13.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.673 7f9170ed3700 1 -- 172.21.2.112:0/3474212327 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f916000c0c0 con 0x7f916c0e5470 2021-02-10T11:14:13.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.673 7f9170ed3700 1 -- 172.21.2.112:0/3474212327 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f916000c340 con 0x7f916c0e5470 2021-02-10T11:14:13.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.673 7f9170ed3700 1 -- 172.21.2.112:0/3474212327 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f916000c860 con 0x7f916c0e5470 2021-02-10T11:14:13.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.673 7f91740b9700 1 -- 172.21.2.112:0/3474212327 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f916c0e5470 msgr2=0x7f916c07e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:13.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.673 7f91740b9700 1 --2- 172.21.2.112:0/3474212327 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f916c0e5470 0x7f916c07e6a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f9160006650 tx=0x7f9160009a80).stop 2021-02-10T11:14:13.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.674 7f91740b9700 1 -- 172.21.2.112:0/3474212327 shutdown_connections 2021-02-10T11:14:13.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.674 7f91740b9700 1 --2- 172.21.2.112:0/3474212327 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f916c0e5470 0x7f916c07e6a0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:13.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.674 7f91740b9700 1 -- 172.21.2.112:0/3474212327 >> 172.21.2.112:0/3474212327 conn(0x7f916c0ce690 msgr2=0x7f916c0e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:13.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.674 7f91740b9700 1 -- 172.21.2.112:0/3474212327 shutdown_connections 2021-02-10T11:14:13.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.674 7f91740b9700 1 -- 172.21.2.112:0/3474212327 wait complete. 2021-02-10T11:14:13.690 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.688 7f91740b9700 1 Processor -- start 2021-02-10T11:14:13.690 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.688 7f91740b9700 1 -- start start 2021-02-10T11:14:13.690 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.689 7f91740b9700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f916c0b34e0 0x7f916c085340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:13.690 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.689 7f91740b9700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f916c0bf560 0x7f916c19c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:13.690 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.689 7f91740b9700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f916c19cfd0 0x7f916c0b4320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:13.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.689 7f91740b9700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f916c0b4ed0 con 0x7f916c0b34e0 2021-02-10T11:14:13.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.689 7f91740b9700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f916c0b38e0 con 0x7f916c0bf560 2021-02-10T11:14:13.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.689 7f91740b9700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f916c0b3ac0 con 0x7f916c19cfd0 2021-02-10T11:14:13.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.689 7f9171ed5700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f916c0b34e0 0x7f916c085340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:13.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.689 7f9171ed5700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f916c0b34e0 0x7f916c085340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:42286/0 (socket says 172.21.2.112:42286) 2021-02-10T11:14:13.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.689 7f9171ed5700 1 -- 172.21.2.112:0/2816256257 learned_addr learned my addr 172.21.2.112:0/2816256257 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:14:13.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.689 7f9171ed5700 1 -- 172.21.2.112:0/2816256257 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f916c19cfd0 msgr2=0x7f916c0b4320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:13.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.689 7f9171ed5700 1 --2- 172.21.2.112:0/2816256257 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f916c19cfd0 0x7f916c0b4320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:13.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.689 7f9171ed5700 1 -- 172.21.2.112:0/2816256257 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f916c0bf560 msgr2=0x7f916c19c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:13.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.689 7f9171ed5700 1 --2- 172.21.2.112:0/2816256257 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f916c0bf560 0x7f916c19c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:13.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.689 7f9171ed5700 1 -- 172.21.2.112:0/2816256257 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f916c0b4940 con 0x7f916c0b34e0 2021-02-10T11:14:13.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.689 7f9171ed5700 1 --2- 172.21.2.112:0/2816256257 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f916c0b34e0 0x7f916c085340 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f916800b5b0 tx=0x7f916800b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:13.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.690 7f915e7fc700 1 -- 172.21.2.112:0/2816256257 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f916800bd40 con 0x7f916c0b34e0 2021-02-10T11:14:13.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.690 7f915e7fc700 1 -- 172.21.2.112:0/2816256257 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9168006240 con 0x7f916c0b34e0 2021-02-10T11:14:13.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.690 7f915e7fc700 1 -- 172.21.2.112:0/2816256257 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9168005720 con 0x7f916c0b34e0 2021-02-10T11:14:13.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.690 7f91740b9700 1 -- 172.21.2.112:0/2816256257 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f916c0b4940 con 0x7f916c0b34e0 2021-02-10T11:14:13.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.690 7f91740b9700 1 -- 172.21.2.112:0/2816256257 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f916c0b4940 con 0x7f916c0b34e0 2021-02-10T11:14:13.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.691 7f915e7fc700 1 -- 172.21.2.112:0/2816256257 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f91680054a0 con 0x7f916c0b34e0 2021-02-10T11:14:13.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.691 7f915e7fc700 1 --2- 172.21.2.112:0/2816256257 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f91540186c0 0x7f915401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:13.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.691 7f915e7fc700 1 -- 172.21.2.112:0/2816256257 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f91680325a0 con 0x7f916c0b34e0 2021-02-10T11:14:13.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.692 7f91716d4700 1 --2- 172.21.2.112:0/2816256257 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f91540186c0 0x7f915401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:13.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.692 7f91716d4700 1 --2- 172.21.2.112:0/2816256257 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f91540186c0 0x7f915401abc0 crc :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:13.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.722 7f91740b9700 1 --2- 172.21.2.112:0/2816256257 >> [v2:172.21.2.112:6801/36031,v1:172.21.2.112:6803/36031] conn(0x7f916c0c45a0 0x7f916c0b5740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:13.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.722 7f91740b9700 1 -- 172.21.2.112:0/2816256257 --> [v2:172.21.2.112:6801/36031,v1:172.21.2.112:6803/36031] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f916c083e50 con 0x7f916c0c45a0 2021-02-10T11:14:13.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.722 7f91726d6700 1 --2- 172.21.2.112:0/2816256257 >> [v2:172.21.2.112:6801/36031,v1:172.21.2.112:6803/36031] conn(0x7f916c0c45a0 0x7f916c0b5740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:13.721 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.722 7f91726d6700 1 --2- 172.21.2.112:0/2816256257 >> [v2:172.21.2.112:6801/36031,v1:172.21.2.112:6803/36031] conn(0x7f916c0c45a0 0x7f916c0b5740 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:13.721 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.723 7f915e7fc700 1 -- 172.21.2.112:0/2816256257 <== osd.2 v2:172.21.2.112:6801/36031 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+5151 (crc 0 0 0) 0x7f9164007b20 con 0x7f916c0c45a0 2021-02-10T11:14:13.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.730 7f91740b9700 1 -- 172.21.2.112:0/2816256257 --> [v2:172.21.2.112:6801/36031,v1:172.21.2.112:6803/36031] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f916c083e50 con 0x7f916c0c45a0 2021-02-10T11:14:13.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.730 7f915e7fc700 1 -- 172.21.2.112:0/2816256257 <== osd.2 v2:172.21.2.112:6801/36031 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f9164007b20 con 0x7f916c0c45a0 2021-02-10T11:14:13.729 INFO:teuthology.orchestra.run.gibba012.stdout:55834574854 2021-02-10T11:14:13.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.731 7f91740b9700 1 -- 172.21.2.112:0/2816256257 >> [v2:172.21.2.112:6801/36031,v1:172.21.2.112:6803/36031] conn(0x7f916c0c45a0 msgr2=0x7f916c0b5740 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:13.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.731 7f91740b9700 1 --2- 172.21.2.112:0/2816256257 >> [v2:172.21.2.112:6801/36031,v1:172.21.2.112:6803/36031] conn(0x7f916c0c45a0 0x7f916c0b5740 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:13.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.731 7f91740b9700 1 -- 172.21.2.112:0/2816256257 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f91540186c0 msgr2=0x7f915401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:13.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.731 7f91740b9700 1 --2- 172.21.2.112:0/2816256257 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f91540186c0 0x7f915401abc0 crc :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:13.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.731 7f91740b9700 1 -- 172.21.2.112:0/2816256257 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f916c0b34e0 msgr2=0x7f916c085340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:13.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.731 7f91740b9700 1 --2- 172.21.2.112:0/2816256257 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f916c0b34e0 0x7f916c085340 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f916800b5b0 tx=0x7f916800b950).stop 2021-02-10T11:14:13.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.731 7f91726d6700 1 -- 172.21.2.112:0/2816256257 reap_dead start 2021-02-10T11:14:13.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.731 7f91740b9700 1 -- 172.21.2.112:0/2816256257 shutdown_connections 2021-02-10T11:14:13.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.731 7f91740b9700 1 -- 172.21.2.112:0/2816256257 >> 172.21.2.112:0/2816256257 conn(0x7f916c0badc0 msgr2=0x7f916c0a78d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:13.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.732 7f91740b9700 1 -- 172.21.2.112:0/2816256257 shutdown_connections 2021-02-10T11:14:13.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:13.732 7f91740b9700 1 -- 172.21.2.112:0/2816256257 wait complete. 2021-02-10T11:14:13.738 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell osd.3 flush_pg_stats 2021-02-10T11:14:14.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.148 7ff56060d700 1 Processor -- start 2021-02-10T11:14:14.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.148 7ff56060d700 1 -- start start 2021-02-10T11:14:14.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.149 7ff56060d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff55807e6a0 0x7ff5580d1790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:14.164 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.149 7ff56060d700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff55807ed80 con 0x7ff5580e5070 2021-02-10T11:14:14.164 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.149 7ff56060d700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff55807eb10 con 0x7ff5580e5470 2021-02-10T11:14:14.164 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.149 7ff56060d700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5581b4080 con 0x7ff55807e6a0 2021-02-10T11:14:14.164 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.149 7ff55e429700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7ff5580e5070 0x7ff5581b3820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:41756/0 (socket says 172.21.2.112:41756) 2021-02-10T11:14:14.164 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.149 7ff55e429700 1 -- 172.21.2.112:0/2164248755 learned_addr learned my addr 172.21.2.112:0/2164248755 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:14:14.164 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.149 7ff55ec2a700 1 --2- 172.21.2.112:0/2164248755 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff55807e6a0 0x7ff5580d1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:14.164 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.149 7ff55d427700 1 -- 172.21.2.112:0/2164248755 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 157876032 0 0) 0x7ff54c0009b0 con 0x7ff5580e5070 2021-02-10T11:14:14.164 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.149 7ff55d427700 1 -- 172.21.2.112:0/2164248755 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff544002fe0 con 0x7ff5580e5070 2021-02-10T11:14:14.164 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.149 7ff55d427700 1 -- 172.21.2.112:0/2164248755 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2551011389 0 0) 0x7ff554001080 con 0x7ff5580e5470 2021-02-10T11:14:14.164 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.149 7ff55d427700 1 -- 172.21.2.112:0/2164248755 --> v1:172.21.2.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff544003210 con 0x7ff5580e5470 2021-02-10T11:14:14.164 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.149 7ff55ec2a700 1 -- 172.21.2.112:0/2164248755 >> v1:172.21.2.120:6789/0 conn(0x7ff5580e5470 legacy=0x7ff55807f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:14.164 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.149 7ff55ec2a700 1 -- 172.21.2.112:0/2164248755 >> v1:172.21.2.112:6789/0 conn(0x7ff5580e5070 legacy=0x7ff5581b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:14.164 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.150 7ff55ec2a700 1 -- 172.21.2.112:0/2164248755 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5580d1db0 con 0x7ff55807e6a0 2021-02-10T11:14:14.165 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.150 7ff55ec2a700 1 --2- 172.21.2.112:0/2164248755 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff55807e6a0 0x7ff5580d1790 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7ff550006e50 tx=0x7ff550009e70).ready entity=mon.2 client_cookie=3b83f60dbb0f0a65 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:14.165 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.150 7ff55d427700 1 -- 172.21.2.112:0/2164248755 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff55000b510 con 0x7ff55807e6a0 2021-02-10T11:14:14.165 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.150 7ff55d427700 1 -- 172.21.2.112:0/2164248755 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff550006790 con 0x7ff55807e6a0 2021-02-10T11:14:14.165 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.150 7ff55d427700 1 -- 172.21.2.112:0/2164248755 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff55000b940 con 0x7ff55807e6a0 2021-02-10T11:14:14.165 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.150 7ff56060d700 1 -- 172.21.2.112:0/2164248755 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff55807e6a0 msgr2=0x7ff5580d1790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:14.165 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.150 7ff56060d700 1 --2- 172.21.2.112:0/2164248755 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff55807e6a0 0x7ff5580d1790 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7ff550006e50 tx=0x7ff550009e70).stop 2021-02-10T11:14:14.165 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.151 7ff56060d700 1 -- 172.21.2.112:0/2164248755 shutdown_connections 2021-02-10T11:14:14.165 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.151 7ff56060d700 1 --2- 172.21.2.112:0/2164248755 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff55807e6a0 0x7ff5580d1790 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:14.165 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.151 7ff56060d700 1 -- 172.21.2.112:0/2164248755 >> 172.21.2.112:0/2164248755 conn(0x7ff5580ce690 msgr2=0x7ff5580e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:14.165 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.151 7ff56060d700 1 -- 172.21.2.112:0/2164248755 shutdown_connections 2021-02-10T11:14:14.165 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.151 7ff56060d700 1 -- 172.21.2.112:0/2164248755 wait complete. 2021-02-10T11:14:14.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.165 7ff56060d700 1 Processor -- start 2021-02-10T11:14:14.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.165 7ff56060d700 1 -- start start 2021-02-10T11:14:14.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.166 7ff56060d700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff558082520 0x7ff558087ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:14.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.166 7ff56060d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff5580880d0 0x7ff5580b1d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:14.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.166 7ff56060d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff5580884d0 0x7ff5580b2370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:14.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.166 7ff56060d700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5580b17d0 con 0x7ff5580880d0 2021-02-10T11:14:14.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.166 7ff56060d700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5580b2990 con 0x7ff558082520 2021-02-10T11:14:14.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.166 7ff56060d700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5580b2b70 con 0x7ff5580884d0 2021-02-10T11:14:14.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.166 7ff55dc28700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff5580880d0 0x7ff5580b1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:14.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.166 7ff55dc28700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff5580880d0 0x7ff5580b1d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:42298/0 (socket says 172.21.2.112:42298) 2021-02-10T11:14:14.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.166 7ff55dc28700 1 -- 172.21.2.112:0/1750919311 learned_addr learned my addr 172.21.2.112:0/1750919311 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:14:14.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.166 7ff55dc28700 1 -- 172.21.2.112:0/1750919311 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff5580884d0 msgr2=0x7ff5580b2370 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:14:14.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.166 7ff55dc28700 1 --2- 172.21.2.112:0/1750919311 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff5580884d0 0x7ff5580b2370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:14.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.166 7ff55dc28700 1 -- 172.21.2.112:0/1750919311 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff558082520 msgr2=0x7ff558087ab0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:14:14.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.166 7ff55dc28700 1 --2- 172.21.2.112:0/1750919311 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff558082520 0x7ff558087ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:14.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.166 7ff55dc28700 1 -- 172.21.2.112:0/1750919311 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff558088a70 con 0x7ff5580880d0 2021-02-10T11:14:14.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.166 7ff55dc28700 1 --2- 172.21.2.112:0/1750919311 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff5580880d0 0x7ff5580b1d50 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7ff554006f00 tx=0x7ff55400b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:14.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.167 7ff54affd700 1 -- 172.21.2.112:0/1750919311 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff55400bd30 con 0x7ff5580880d0 2021-02-10T11:14:14.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.167 7ff54affd700 1 -- 172.21.2.112:0/1750919311 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff554000e20 con 0x7ff5580880d0 2021-02-10T11:14:14.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.167 7ff54affd700 1 -- 172.21.2.112:0/1750919311 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff5540054f0 con 0x7ff5580880d0 2021-02-10T11:14:14.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.167 7ff56060d700 1 -- 172.21.2.112:0/1750919311 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff558088a70 con 0x7ff5580880d0 2021-02-10T11:14:14.169 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.167 7ff56060d700 1 -- 172.21.2.112:0/1750919311 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff558088fd0 con 0x7ff5580880d0 2021-02-10T11:14:14.169 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.167 7ff54affd700 1 -- 172.21.2.112:0/1750919311 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff554006030 con 0x7ff5580880d0 2021-02-10T11:14:14.169 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.167 7ff54affd700 1 --2- 172.21.2.112:0/1750919311 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff5400186c0 0x7ff54001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:14.169 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.167 7ff54affd700 1 -- 172.21.2.112:0/1750919311 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7ff554032590 con 0x7ff5580880d0 2021-02-10T11:14:14.169 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.168 7ff55e429700 1 --2- 172.21.2.112:0/1750919311 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff5400186c0 0x7ff54001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:14.169 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.168 7ff55e429700 1 --2- 172.21.2.112:0/1750919311 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff5400186c0 0x7ff54001abc0 crc :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:14.197 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.198 7ff56060d700 1 --2- 172.21.2.112:0/1750919311 >> [v2:172.21.2.112:6816/36028,v1:172.21.2.112:6819/36028] conn(0x7ff5580c45a0 0x7ff5580a0fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:14.197 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.198 7ff56060d700 1 -- 172.21.2.112:0/1750919311 --> [v2:172.21.2.112:6816/36028,v1:172.21.2.112:6819/36028] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff5580b2990 con 0x7ff5580c45a0 2021-02-10T11:14:14.198 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.199 7ff55ec2a700 1 --2- 172.21.2.112:0/1750919311 >> [v2:172.21.2.112:6816/36028,v1:172.21.2.112:6819/36028] conn(0x7ff5580c45a0 0x7ff5580a0fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:14.198 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.199 7ff55ec2a700 1 --2- 172.21.2.112:0/1750919311 >> [v2:172.21.2.112:6816/36028,v1:172.21.2.112:6819/36028] conn(0x7ff5580c45a0 0x7ff5580a0fb0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:14.198 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.200 7ff54affd700 1 -- 172.21.2.112:0/1750919311 <== osd.3 v2:172.21.2.112:6816/36028 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+5151 (crc 0 0 0) 0x7ff550006ce0 con 0x7ff5580c45a0 2021-02-10T11:14:14.205 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.207 7ff56060d700 1 -- 172.21.2.112:0/1750919311 --> [v2:172.21.2.112:6816/36028,v1:172.21.2.112:6819/36028] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ff558088fd0 con 0x7ff5580c45a0 2021-02-10T11:14:14.205 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.207 7ff54affd700 1 -- 172.21.2.112:0/1750919311 <== osd.3 v2:172.21.2.112:6816/36028 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7ff550006830 con 0x7ff5580c45a0 2021-02-10T11:14:14.206 INFO:teuthology.orchestra.run.gibba012.stdout:55834574856 2021-02-10T11:14:14.207 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.208 7ff56060d700 1 -- 172.21.2.112:0/1750919311 >> [v2:172.21.2.112:6816/36028,v1:172.21.2.112:6819/36028] conn(0x7ff5580c45a0 msgr2=0x7ff5580a0fb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:14.207 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.208 7ff56060d700 1 --2- 172.21.2.112:0/1750919311 >> [v2:172.21.2.112:6816/36028,v1:172.21.2.112:6819/36028] conn(0x7ff5580c45a0 0x7ff5580a0fb0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:14.207 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.208 7ff56060d700 1 -- 172.21.2.112:0/1750919311 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff5400186c0 msgr2=0x7ff54001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:14.207 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.208 7ff56060d700 1 --2- 172.21.2.112:0/1750919311 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff5400186c0 0x7ff54001abc0 crc :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:14.207 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.208 7ff56060d700 1 -- 172.21.2.112:0/1750919311 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff5580880d0 msgr2=0x7ff5580b1d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:14.207 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.208 7ff56060d700 1 --2- 172.21.2.112:0/1750919311 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff5580880d0 0x7ff5580b1d50 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7ff554006f00 tx=0x7ff55400b940).stop 2021-02-10T11:14:14.208 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.208 7ff55ec2a700 1 -- 172.21.2.112:0/1750919311 reap_dead start 2021-02-10T11:14:14.208 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.208 7ff56060d700 1 -- 172.21.2.112:0/1750919311 shutdown_connections 2021-02-10T11:14:14.208 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.208 7ff56060d700 1 -- 172.21.2.112:0/1750919311 >> 172.21.2.112:0/1750919311 conn(0x7ff558098740 msgr2=0x7ff5580c3590 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:14.208 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.209 7ff56060d700 1 -- 172.21.2.112:0/1750919311 shutdown_connections 2021-02-10T11:14:14.209 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.209 7ff56060d700 1 -- 172.21.2.112:0/1750919311 wait complete. 2021-02-10T11:14:14.215 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell osd.4 flush_pg_stats 2021-02-10T11:14:14.640 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.626 7f7493d98700 1 Processor -- start 2021-02-10T11:14:14.641 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.626 7f7493d98700 1 -- start start 2021-02-10T11:14:14.641 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.626 7f7493d98700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f748c07e6a0 0x7f748c0d1790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:14.641 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.626 7f7493d98700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f748c07ed80 con 0x7f748c0e5470 2021-02-10T11:14:14.641 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.626 7f7493d98700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f748c07eaa0 con 0x7f748c0e5070 2021-02-10T11:14:14.641 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.626 7f7493d98700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f748c1b4080 con 0x7f748c07e6a0 2021-02-10T11:14:14.641 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.626 7f74913b3700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f748c0e5470 0x7f748c07f300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:41770/0 (socket says 172.21.2.112:41770) 2021-02-10T11:14:14.641 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.626 7f74913b3700 1 -- 172.21.2.112:0/1513481941 learned_addr learned my addr 172.21.2.112:0/1513481941 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:14:14.641 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.626 7f74923b5700 1 --2- 172.21.2.112:0/1513481941 >> v2:172.21.2.120:3300/0 conn(0x7f748c07e6a0 0x7f748c0d1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:14.641 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.627 7f7490bb2700 1 -- 172.21.2.112:0/1513481941 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 880248531 0 0) 0x7f7488000ce0 con 0x7f748c0e5470 2021-02-10T11:14:14.641 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.627 7f7490bb2700 1 -- 172.21.2.112:0/1513481941 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7478002fe0 con 0x7f748c0e5470 2021-02-10T11:14:14.641 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.627 7f7490bb2700 1 -- 172.21.2.112:0/1513481941 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2768371503 0 0) 0x7f7480000c60 con 0x7f748c0e5070 2021-02-10T11:14:14.642 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.627 7f7490bb2700 1 -- 172.21.2.112:0/1513481941 --> v1:172.21.2.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7478004370 con 0x7f748c0e5070 2021-02-10T11:14:14.642 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.627 7f74923b5700 1 -- 172.21.2.112:0/1513481941 >> v1:172.21.2.120:6789/0 conn(0x7f748c0e5070 legacy=0x7f748c1b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:14.642 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.627 7f74923b5700 1 -- 172.21.2.112:0/1513481941 >> v1:172.21.2.112:6789/0 conn(0x7f748c0e5470 legacy=0x7f748c07f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:14.642 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.627 7f74923b5700 1 -- 172.21.2.112:0/1513481941 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f748c0d1db0 con 0x7f748c07e6a0 2021-02-10T11:14:14.642 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.627 7f74923b5700 1 --2- 172.21.2.112:0/1513481941 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f748c07e6a0 0x7f748c0d1790 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f7484006e50 tx=0x7f7484009e70).ready entity=mon.1 client_cookie=40d40ba7bf435e03 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:14.642 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.627 7f7490bb2700 1 -- 172.21.2.112:0/1513481941 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f748400b510 con 0x7f748c07e6a0 2021-02-10T11:14:14.642 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.627 7f7490bb2700 1 -- 172.21.2.112:0/1513481941 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7484006790 con 0x7f748c07e6a0 2021-02-10T11:14:14.642 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.627 7f7490bb2700 1 -- 172.21.2.112:0/1513481941 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f748400b940 con 0x7f748c07e6a0 2021-02-10T11:14:14.642 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.627 7f7493d98700 1 -- 172.21.2.112:0/1513481941 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f748c07e6a0 msgr2=0x7f748c0d1790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:14.642 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.627 7f7493d98700 1 --2- 172.21.2.112:0/1513481941 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f748c07e6a0 0x7f748c0d1790 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f7484006e50 tx=0x7f7484009e70).stop 2021-02-10T11:14:14.642 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.628 7f7493d98700 1 -- 172.21.2.112:0/1513481941 shutdown_connections 2021-02-10T11:14:14.642 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.628 7f7493d98700 1 --2- 172.21.2.112:0/1513481941 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f748c07e6a0 0x7f748c0d1790 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:14.642 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.628 7f7493d98700 1 -- 172.21.2.112:0/1513481941 >> 172.21.2.112:0/1513481941 conn(0x7f748c0ce690 msgr2=0x7f748c0e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:14.643 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.628 7f7493d98700 1 -- 172.21.2.112:0/1513481941 shutdown_connections 2021-02-10T11:14:14.643 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.628 7f7493d98700 1 -- 172.21.2.112:0/1513481941 wait complete. 2021-02-10T11:14:14.644 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.642 7f7493d98700 1 Processor -- start 2021-02-10T11:14:14.644 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.643 7f7493d98700 1 -- start start 2021-02-10T11:14:14.644 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.643 7f7493d98700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f748c0bd700 0x7f748c0bdb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:14.644 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.643 7f7493d98700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f748c0a4b10 0x7f748c0a4f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:14.644 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.643 7f7493d98700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f748c0a5530 0x7f748c0a5930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:14.645 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.643 7f7493d98700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f748c0be230 con 0x7f748c0a5530 2021-02-10T11:14:14.645 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.643 7f7493d98700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f748c0be450 con 0x7f748c0bd700 2021-02-10T11:14:14.645 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.643 7f7493d98700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f748c0be670 con 0x7f748c0a4b10 2021-02-10T11:14:14.645 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.643 7f74923b5700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f748c0a5530 0x7f748c0a5930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:14.645 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.643 7f74923b5700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f748c0a5530 0x7f748c0a5930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:42316/0 (socket says 172.21.2.112:42316) 2021-02-10T11:14:14.645 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.643 7f74923b5700 1 -- 172.21.2.112:0/2955865256 learned_addr learned my addr 172.21.2.112:0/2955865256 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:14:14.645 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.643 7f74923b5700 1 -- 172.21.2.112:0/2955865256 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f748c0a4b10 msgr2=0x7f748c0a4f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:14.645 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.643 7f74923b5700 1 --2- 172.21.2.112:0/2955865256 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f748c0a4b10 0x7f748c0a4f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:14.645 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.643 7f74923b5700 1 -- 172.21.2.112:0/2955865256 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f748c0bd700 msgr2=0x7f748c0bdb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:14.645 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.643 7f74923b5700 1 --2- 172.21.2.112:0/2955865256 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f748c0bd700 0x7f748c0bdb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:14.645 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.643 7f74923b5700 1 -- 172.21.2.112:0/2955865256 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f748c0be890 con 0x7f748c0a5530 2021-02-10T11:14:14.645 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.644 7f74923b5700 1 --2- 172.21.2.112:0/2955865256 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f748c0a5530 0x7f748c0a5930 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f7484006c50 tx=0x7f748400bd20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:14.645 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.644 7f747e7fc700 1 -- 172.21.2.112:0/2955865256 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f74840093d0 con 0x7f748c0a5530 2021-02-10T11:14:14.645 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.644 7f747e7fc700 1 -- 172.21.2.112:0/2955865256 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f748400be60 con 0x7f748c0a5530 2021-02-10T11:14:14.646 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.644 7f747e7fc700 1 -- 172.21.2.112:0/2955865256 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f748400c4e0 con 0x7f748c0a5530 2021-02-10T11:14:14.646 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.644 7f7493d98700 1 -- 172.21.2.112:0/2955865256 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f748c0be890 con 0x7f748c0a5530 2021-02-10T11:14:14.646 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.644 7f7493d98700 1 -- 172.21.2.112:0/2955865256 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f748c0be890 con 0x7f748c0a5530 2021-02-10T11:14:14.646 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.645 7f747e7fc700 1 -- 172.21.2.112:0/2955865256 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7484002ef0 con 0x7f748c0a5530 2021-02-10T11:14:14.646 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.645 7f747e7fc700 1 --2- 172.21.2.112:0/2955865256 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f74740186c0 0x7f747401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:14.646 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.645 7f747e7fc700 1 -- 172.21.2.112:0/2955865256 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f74840327a0 con 0x7f748c0a5530 2021-02-10T11:14:14.646 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.645 7f7491bb4700 1 --2- 172.21.2.112:0/2955865256 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f74740186c0 0x7f747401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:14.646 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.647 7f7491bb4700 1 --2- 172.21.2.112:0/2955865256 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f74740186c0 0x7f747401abc0 crc :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:14.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.676 7f7493d98700 1 --2- 172.21.2.112:0/2955865256 >> [v2:172.21.2.120:6810/36388,v1:172.21.2.120:6813/36388] conn(0x7f748c0bf480 0x7f748c0627c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:14.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.676 7f7493d98700 1 -- 172.21.2.112:0/2955865256 --> [v2:172.21.2.120:6810/36388,v1:172.21.2.120:6813/36388] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f748c07ed80 con 0x7f748c0bf480 2021-02-10T11:14:14.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.677 7f74913b3700 1 --2- 172.21.2.112:0/2955865256 >> [v2:172.21.2.120:6810/36388,v1:172.21.2.120:6813/36388] conn(0x7f748c0bf480 0x7f748c0627c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:14.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.678 7f74913b3700 1 --2- 172.21.2.112:0/2955865256 >> [v2:172.21.2.120:6810/36388,v1:172.21.2.120:6813/36388] conn(0x7f748c0bf480 0x7f748c0627c0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:14.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.678 7f747e7fc700 1 -- 172.21.2.112:0/2955865256 <== osd.4 v2:172.21.2.120:6810/36388 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+5151 (crc 0 0 0) 0x7f748800b050 con 0x7f748c0bf480 2021-02-10T11:14:14.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.684 7f7493d98700 1 -- 172.21.2.112:0/2955865256 --> [v2:172.21.2.120:6810/36388,v1:172.21.2.120:6813/36388] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f748c0c3aa0 con 0x7f748c0bf480 2021-02-10T11:14:14.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.685 7f747e7fc700 1 -- 172.21.2.112:0/2955865256 <== osd.4 v2:172.21.2.120:6810/36388 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f748800b150 con 0x7f748c0bf480 2021-02-10T11:14:14.684 INFO:teuthology.orchestra.run.gibba012.stdout:55834574856 2021-02-10T11:14:14.685 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.686 7f7493d98700 1 -- 172.21.2.112:0/2955865256 >> [v2:172.21.2.120:6810/36388,v1:172.21.2.120:6813/36388] conn(0x7f748c0bf480 msgr2=0x7f748c0627c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:14.685 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.686 7f7493d98700 1 --2- 172.21.2.112:0/2955865256 >> [v2:172.21.2.120:6810/36388,v1:172.21.2.120:6813/36388] conn(0x7f748c0bf480 0x7f748c0627c0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:14.685 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.686 7f7493d98700 1 -- 172.21.2.112:0/2955865256 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f74740186c0 msgr2=0x7f747401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:14.685 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.686 7f7493d98700 1 --2- 172.21.2.112:0/2955865256 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f74740186c0 0x7f747401abc0 crc :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:14.685 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.686 7f7493d98700 1 -- 172.21.2.112:0/2955865256 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f748c0a5530 msgr2=0x7f748c0a5930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:14.685 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.686 7f7493d98700 1 --2- 172.21.2.112:0/2955865256 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f748c0a5530 0x7f748c0a5930 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f7484006c50 tx=0x7f748400bd20).stop 2021-02-10T11:14:14.685 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.686 7f74923b5700 1 -- 172.21.2.112:0/2955865256 reap_dead start 2021-02-10T11:14:14.685 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.686 7f7493d98700 1 -- 172.21.2.112:0/2955865256 shutdown_connections 2021-02-10T11:14:14.685 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.686 7f7493d98700 1 -- 172.21.2.112:0/2955865256 >> 172.21.2.112:0/2955865256 conn(0x7f748c0c0cf0 msgr2=0x7f748c0a70a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:14.685 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.686 7f7493d98700 1 -- 172.21.2.112:0/2955865256 shutdown_connections 2021-02-10T11:14:14.686 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:14.686 7f7493d98700 1 -- 172.21.2.112:0/2955865256 wait complete. 2021-02-10T11:14:14.692 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell osd.5 flush_pg_stats 2021-02-10T11:14:15.118 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.101 7fcbb571c700 1 Processor -- start 2021-02-10T11:14:15.118 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.102 7fcbb571c700 1 -- start start 2021-02-10T11:14:15.118 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.102 7fcbb571c700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcbb00e5070 0x7fcbb01b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:15.118 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.102 7fcbb571c700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbb007ed80 con 0x7fcbb01b3e40 2021-02-10T11:14:15.118 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.102 7fcbb571c700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbb007e670 con 0x7fcbb00e5470 2021-02-10T11:14:15.118 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.102 7fcbb571c700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbb007e940 con 0x7fcbb00e5070 2021-02-10T11:14:15.118 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.102 7fcbafac0700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fcbb01b3e40 0x7fcbb007de80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:41784/0 (socket says 172.21.2.112:41784) 2021-02-10T11:14:15.118 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.102 7fcbafac0700 1 -- 172.21.2.112:0/3639387803 learned_addr learned my addr 172.21.2.112:0/3639387803 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:14:15.119 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.103 7fcbae2bd700 1 -- 172.21.2.112:0/3639387803 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2820177834 0 0) 0x7fcba8000f90 con 0x7fcbb01b3e40 2021-02-10T11:14:15.119 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.103 7fcbae2bd700 1 -- 172.21.2.112:0/3639387803 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb9c002fe0 con 0x7fcbb01b3e40 2021-02-10T11:14:15.119 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.103 7fcbaf2bf700 1 --2- 172.21.2.112:0/3639387803 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcbb00e5070 0x7fcbb01b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:15.119 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.103 7fcbae2bd700 1 -- 172.21.2.112:0/3639387803 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2552600907 0 0) 0x7fcba8001460 con 0x7fcbb01b3e40 2021-02-10T11:14:15.119 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.103 7fcbae2bd700 1 -- 172.21.2.112:0/3639387803 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcbb00e5070 msgr2=0x7fcbb01b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:14:15.119 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.103 7fcbae2bd700 1 --2- 172.21.2.112:0/3639387803 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcbb00e5070 0x7fcbb01b3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:15.119 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.103 7fcbae2bd700 1 -- 172.21.2.112:0/3639387803 >> v1:172.21.2.120:6789/0 conn(0x7fcbb00e5470 legacy=0x7fcbb007d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:15.119 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.103 7fcbae2bd700 1 -- 172.21.2.112:0/3639387803 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbb007eb30 con 0x7fcbb01b3e40 2021-02-10T11:14:15.119 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.103 7fcbae2bd700 1 -- 172.21.2.112:0/3639387803 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7fcba80019e0 con 0x7fcbb01b3e40 2021-02-10T11:14:15.119 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.103 7fcbae2bd700 1 -- 172.21.2.112:0/3639387803 >> v1:172.21.2.112:6789/0 conn(0x7fcbb01b3e40 legacy=0x7fcbb007de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:15.119 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.103 7fcbae2bd700 1 --2- 172.21.2.112:0/3639387803 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcb9c003b90 0x7fcb9c005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:15.119 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.103 7fcbae2bd700 1 --2- 172.21.2.112:0/3639387803 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcb9c006790 0x7fcb9c008c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:15.119 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.103 7fcbae2bd700 1 --2- 172.21.2.112:0/3639387803 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcb9c009510 0x7fcb9c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:15.119 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.103 7fcbae2bd700 1 -- 172.21.2.112:0/3639387803 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb9c00c2c0 con 0x7fcb9c006790 2021-02-10T11:14:15.120 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.103 7fcbae2bd700 1 -- 172.21.2.112:0/3639387803 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb9c00c4e0 con 0x7fcb9c009510 2021-02-10T11:14:15.120 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.103 7fcbae2bd700 1 -- 172.21.2.112:0/3639387803 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb9c00c700 con 0x7fcb9c003b90 2021-02-10T11:14:15.120 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.103 7fcbaeabe700 1 --2- 172.21.2.112:0/3639387803 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcb9c006790 0x7fcb9c008c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:15.120 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.103 7fcbaf2bf700 1 --2- 172.21.2.112:0/3639387803 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcb9c003b90 0x7fcb9c005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:15.120 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.104 7fcbafac0700 1 --2- 172.21.2.112:0/3639387803 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcb9c009510 0x7fcb9c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:15.120 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.104 7fcbaeabe700 1 -- 172.21.2.112:0/3639387803 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcb9c003b90 msgr2=0x7fcb9c005fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:14:15.120 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.104 7fcbaeabe700 1 --2- 172.21.2.112:0/3639387803 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcb9c003b90 0x7fcb9c005fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:15.120 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.104 7fcbaeabe700 1 -- 172.21.2.112:0/3639387803 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcb9c009510 msgr2=0x7fcb9c00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:14:15.120 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.104 7fcbaeabe700 1 --2- 172.21.2.112:0/3639387803 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcb9c009510 0x7fcb9c00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:15.120 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.104 7fcbaeabe700 1 -- 172.21.2.112:0/3639387803 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb9c050970 con 0x7fcb9c006790 2021-02-10T11:14:15.120 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.104 7fcbafac0700 1 --2- 172.21.2.112:0/3639387803 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcb9c009510 0x7fcb9c00ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:14:15.120 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.104 7fcbaeabe700 1 --2- 172.21.2.112:0/3639387803 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcb9c006790 0x7fcb9c008c70 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7fcba0006ac0 tx=0x7fcba0009af0).ready entity=mon.0 client_cookie=bfb86717c811e6c9 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:15.120 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.104 7fcbae2bd700 1 -- 172.21.2.112:0/3639387803 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcba000a280 con 0x7fcb9c006790 2021-02-10T11:14:15.120 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.104 7fcbae2bd700 1 -- 172.21.2.112:0/3639387803 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcba000a580 con 0x7fcb9c006790 2021-02-10T11:14:15.121 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.104 7fcbae2bd700 1 -- 172.21.2.112:0/3639387803 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcba000aa60 con 0x7fcb9c006790 2021-02-10T11:14:15.121 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.104 7fcbb571c700 1 -- 172.21.2.112:0/3639387803 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcb9c006790 msgr2=0x7fcb9c008c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:15.121 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.104 7fcbb571c700 1 --2- 172.21.2.112:0/3639387803 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcb9c006790 0x7fcb9c008c70 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7fcba0006ac0 tx=0x7fcba0009af0).stop 2021-02-10T11:14:15.121 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.104 7fcbafac0700 1 -- 172.21.2.112:0/3639387803 reap_dead start 2021-02-10T11:14:15.121 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.105 7fcbb571c700 1 -- 172.21.2.112:0/3639387803 shutdown_connections 2021-02-10T11:14:15.121 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.105 7fcbb571c700 1 -- 172.21.2.112:0/3639387803 >> 172.21.2.112:0/3639387803 conn(0x7fcbb00ce690 msgr2=0x7fcbb00e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:15.121 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.105 7fcbb571c700 1 -- 172.21.2.112:0/3639387803 shutdown_connections 2021-02-10T11:14:15.121 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.105 7fcbb571c700 1 -- 172.21.2.112:0/3639387803 wait complete. 2021-02-10T11:14:15.121 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.119 7fcbb571c700 1 Processor -- start 2021-02-10T11:14:15.121 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.119 7fcbb571c700 1 -- start start 2021-02-10T11:14:15.123 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.120 7fcbb571c700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcbb008f500 0x7fcbb019f900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:15.123 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.120 7fcbb571c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcbb008fd90 0x7fcbb0090190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:15.123 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.120 7fcbb571c700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcbb00907b0 0x7fcbb0094370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:15.123 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.120 7fcbb571c700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbb008f900 con 0x7fcbb008fd90 2021-02-10T11:14:15.123 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.120 7fcbb571c700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbb019ff20 con 0x7fcbb008f500 2021-02-10T11:14:15.123 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.120 7fcbb571c700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbb01a0140 con 0x7fcbb00907b0 2021-02-10T11:14:15.123 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.120 7fcbaeabe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcbb008fd90 0x7fcbb0090190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:15.123 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.120 7fcbaeabe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcbb008fd90 0x7fcbb0090190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:42334/0 (socket says 172.21.2.112:42334) 2021-02-10T11:14:15.123 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.120 7fcbaeabe700 1 -- 172.21.2.112:0/2553898933 learned_addr learned my addr 172.21.2.112:0/2553898933 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:14:15.124 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.120 7fcbaeabe700 1 -- 172.21.2.112:0/2553898933 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcbb00907b0 msgr2=0x7fcbb0094370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:15.124 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.120 7fcbaeabe700 1 --2- 172.21.2.112:0/2553898933 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcbb00907b0 0x7fcbb0094370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:15.124 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.120 7fcbaeabe700 1 -- 172.21.2.112:0/2553898933 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcbb008f500 msgr2=0x7fcbb019f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:15.124 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.120 7fcbaeabe700 1 --2- 172.21.2.112:0/2553898933 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcbb008f500 0x7fcbb019f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:15.124 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.120 7fcbaeabe700 1 -- 172.21.2.112:0/2553898933 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbb0094990 con 0x7fcbb008fd90 2021-02-10T11:14:15.124 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.121 7fcbaeabe700 1 --2- 172.21.2.112:0/2553898933 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcbb008fd90 0x7fcbb0090190 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7fcba000c590 tx=0x7fcba000c930).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:15.124 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.121 7fcba7fff700 1 -- 172.21.2.112:0/2553898933 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcba000cd20 con 0x7fcbb008fd90 2021-02-10T11:14:15.124 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.121 7fcba7fff700 1 -- 172.21.2.112:0/2553898933 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcba0000f20 con 0x7fcbb008fd90 2021-02-10T11:14:15.124 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.121 7fcba7fff700 1 -- 172.21.2.112:0/2553898933 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcba0001440 con 0x7fcbb008fd90 2021-02-10T11:14:15.124 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.121 7fcbb571c700 1 -- 172.21.2.112:0/2553898933 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbb0094990 con 0x7fcbb008fd90 2021-02-10T11:14:15.124 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.121 7fcbb571c700 1 -- 172.21.2.112:0/2553898933 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbb0094f20 con 0x7fcbb008fd90 2021-02-10T11:14:15.124 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.122 7fcba7fff700 1 -- 172.21.2.112:0/2553898933 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fcba00013d0 con 0x7fcbb008fd90 2021-02-10T11:14:15.124 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.122 7fcba7fff700 1 --2- 172.21.2.112:0/2553898933 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcb940186c0 0x7fcb9401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:15.124 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.122 7fcba7fff700 1 -- 172.21.2.112:0/2553898933 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fcba0032570 con 0x7fcbb008fd90 2021-02-10T11:14:15.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.122 7fcbaf2bf700 1 --2- 172.21.2.112:0/2553898933 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcb940186c0 0x7fcb9401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:15.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.122 7fcbaf2bf700 1 --2- 172.21.2.112:0/2553898933 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcb940186c0 0x7fcb9401abc0 crc :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:15.150 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.152 7fcbb571c700 1 --2- 172.21.2.112:0/2553898933 >> [v2:172.21.2.120:6804/36387,v1:172.21.2.120:6806/36387] conn(0x7fcbb00c45a0 0x7fcbb00627c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:15.150 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.152 7fcbb571c700 1 -- 172.21.2.112:0/2553898933 --> [v2:172.21.2.120:6804/36387,v1:172.21.2.120:6806/36387] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcbb0094f20 con 0x7fcbb00c45a0 2021-02-10T11:14:15.150 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.152 7fcbafac0700 1 --2- 172.21.2.112:0/2553898933 >> [v2:172.21.2.120:6804/36387,v1:172.21.2.120:6806/36387] conn(0x7fcbb00c45a0 0x7fcbb00627c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:15.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.152 7fcbafac0700 1 --2- 172.21.2.112:0/2553898933 >> [v2:172.21.2.120:6804/36387,v1:172.21.2.120:6806/36387] conn(0x7fcbb00c45a0 0x7fcbb00627c0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:15.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.153 7fcba7fff700 1 -- 172.21.2.112:0/2553898933 <== osd.5 v2:172.21.2.120:6804/36387 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+5151 (crc 0 0 0) 0x7fcba8007cc0 con 0x7fcbb00c45a0 2021-02-10T11:14:15.158 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.160 7fcbb571c700 1 -- 172.21.2.112:0/2553898933 --> [v2:172.21.2.120:6804/36387,v1:172.21.2.120:6806/36387] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fcbb0094f20 con 0x7fcbb00c45a0 2021-02-10T11:14:15.159 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.160 7fcba7fff700 1 -- 172.21.2.112:0/2553898933 <== osd.5 v2:172.21.2.120:6804/36387 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fcba8007cc0 con 0x7fcbb00c45a0 2021-02-10T11:14:15.159 INFO:teuthology.orchestra.run.gibba012.stdout:55834574855 2021-02-10T11:14:15.159 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.161 7fcbb571c700 1 -- 172.21.2.112:0/2553898933 >> [v2:172.21.2.120:6804/36387,v1:172.21.2.120:6806/36387] conn(0x7fcbb00c45a0 msgr2=0x7fcbb00627c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:15.160 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.161 7fcbb571c700 1 --2- 172.21.2.112:0/2553898933 >> [v2:172.21.2.120:6804/36387,v1:172.21.2.120:6806/36387] conn(0x7fcbb00c45a0 0x7fcbb00627c0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:15.160 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.161 7fcbb571c700 1 -- 172.21.2.112:0/2553898933 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcb940186c0 msgr2=0x7fcb9401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:15.160 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.161 7fcbb571c700 1 --2- 172.21.2.112:0/2553898933 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcb940186c0 0x7fcb9401abc0 crc :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:15.160 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.161 7fcbb571c700 1 -- 172.21.2.112:0/2553898933 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcbb008fd90 msgr2=0x7fcbb0090190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:15.160 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.161 7fcbb571c700 1 --2- 172.21.2.112:0/2553898933 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcbb008fd90 0x7fcbb0090190 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7fcba000c590 tx=0x7fcba000c930).stop 2021-02-10T11:14:15.160 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.161 7fcbafac0700 1 -- 172.21.2.112:0/2553898933 reap_dead start 2021-02-10T11:14:15.160 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.161 7fcbb571c700 1 -- 172.21.2.112:0/2553898933 shutdown_connections 2021-02-10T11:14:15.160 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.161 7fcbb571c700 1 -- 172.21.2.112:0/2553898933 >> 172.21.2.112:0/2553898933 conn(0x7fcbb00b5560 msgr2=0x7fcbb00a6200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:15.161 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.162 7fcbb571c700 1 -- 172.21.2.112:0/2553898933 shutdown_connections 2021-02-10T11:14:15.161 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.162 7fcbb571c700 1 -- 172.21.2.112:0/2553898933 wait complete. 2021-02-10T11:14:15.167 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell osd.6 flush_pg_stats 2021-02-10T11:14:15.593 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.578 7fd493a22700 1 Processor -- start 2021-02-10T11:14:15.593 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.578 7fd493a22700 1 -- start start 2021-02-10T11:14:15.593 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.578 7fd493a22700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fd48c07e6a0 0x7fd48c0d1790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:15.593 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.578 7fd493a22700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd48c07ed80 con 0x7fd48c0e5470 2021-02-10T11:14:15.594 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.578 7fd493a22700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd48c07eaa0 con 0x7fd48c0e5070 2021-02-10T11:14:15.594 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.578 7fd493a22700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd48c1b4080 con 0x7fd48c07e6a0 2021-02-10T11:14:15.594 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.578 7fd49103d700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fd48c0e5470 0x7fd48c07f300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:41806/0 (socket says 172.21.2.112:41806) 2021-02-10T11:14:15.594 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.578 7fd49103d700 1 -- 172.21.2.112:0/2971145951 learned_addr learned my addr 172.21.2.112:0/2971145951 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:14:15.594 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.578 7fd49203f700 1 --2- 172.21.2.112:0/2971145951 >> v2:172.21.2.112:3300/0 conn(0x7fd48c07e6a0 0x7fd48c0d1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:15.594 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.579 7fd49083c700 1 -- 172.21.2.112:0/2971145951 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 315977094 0 0) 0x7fd480001180 con 0x7fd48c0e5470 2021-02-10T11:14:15.594 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.579 7fd49083c700 1 -- 172.21.2.112:0/2971145951 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd478002fe0 con 0x7fd48c0e5470 2021-02-10T11:14:15.594 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.579 7fd49203f700 1 -- 172.21.2.112:0/2971145951 >> v1:172.21.2.120:6789/0 conn(0x7fd48c0e5070 legacy=0x7fd48c1b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:14:15.594 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.579 7fd49203f700 1 -- 172.21.2.112:0/2971145951 >> v1:172.21.2.112:6789/0 conn(0x7fd48c0e5470 legacy=0x7fd48c07f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:15.594 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.579 7fd49203f700 1 -- 172.21.2.112:0/2971145951 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd48c0d1db0 con 0x7fd48c07e6a0 2021-02-10T11:14:15.594 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.579 7fd49203f700 1 --2- 172.21.2.112:0/2971145951 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd48c07e6a0 0x7fd48c0d1790 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7fd484006e30 tx=0x7fd484009e60).ready entity=mon.0 client_cookie=d816043f3a8d26ff server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:15.595 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.579 7fd49083c700 1 -- 172.21.2.112:0/2971145951 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd484006a50 con 0x7fd48c07e6a0 2021-02-10T11:14:15.595 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.579 7fd49083c700 1 -- 172.21.2.112:0/2971145951 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd48400b300 con 0x7fd48c07e6a0 2021-02-10T11:14:15.595 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.579 7fd49083c700 1 -- 172.21.2.112:0/2971145951 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd48400b820 con 0x7fd48c07e6a0 2021-02-10T11:14:15.595 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.579 7fd493a22700 1 -- 172.21.2.112:0/2971145951 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd48c07e6a0 msgr2=0x7fd48c0d1790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:15.595 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.579 7fd493a22700 1 --2- 172.21.2.112:0/2971145951 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd48c07e6a0 0x7fd48c0d1790 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7fd484006e30 tx=0x7fd484009e60).stop 2021-02-10T11:14:15.595 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.580 7fd493a22700 1 -- 172.21.2.112:0/2971145951 shutdown_connections 2021-02-10T11:14:15.595 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.580 7fd493a22700 1 --2- 172.21.2.112:0/2971145951 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd48c07e6a0 0x7fd48c0d1790 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:15.595 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.580 7fd493a22700 1 -- 172.21.2.112:0/2971145951 >> 172.21.2.112:0/2971145951 conn(0x7fd48c0ce690 msgr2=0x7fd48c0e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:15.595 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.580 7fd493a22700 1 -- 172.21.2.112:0/2971145951 shutdown_connections 2021-02-10T11:14:15.595 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.580 7fd493a22700 1 -- 172.21.2.112:0/2971145951 wait complete. 2021-02-10T11:14:15.597 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.595 7fd493a22700 1 Processor -- start 2021-02-10T11:14:15.597 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.595 7fd493a22700 1 -- start start 2021-02-10T11:14:15.597 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.595 7fd493a22700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd48c0a2b70 0x7fd48c0c0590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:15.597 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.595 7fd493a22700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd48c0c0bb0 0x7fd48c0a3850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:15.597 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.595 7fd493a22700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd48c0c0fb0 0x7fd48c0a3e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:15.598 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.595 7fd493a22700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd48c0a2f70 con 0x7fd48c0c0fb0 2021-02-10T11:14:15.598 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.595 7fd493a22700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd48c0a1a40 con 0x7fd48c0c0bb0 2021-02-10T11:14:15.598 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.595 7fd493a22700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd48c0a1c60 con 0x7fd48c0a2b70 2021-02-10T11:14:15.598 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.596 7fd49203f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd48c0c0fb0 0x7fd48c0a3e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:15.598 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.596 7fd49203f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd48c0c0fb0 0x7fd48c0a3e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:42354/0 (socket says 172.21.2.112:42354) 2021-02-10T11:14:15.598 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.596 7fd49203f700 1 -- 172.21.2.112:0/81110860 learned_addr learned my addr 172.21.2.112:0/81110860 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:14:15.598 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.596 7fd49203f700 1 -- 172.21.2.112:0/81110860 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd48c0a2b70 msgr2=0x7fd48c0c0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:15.598 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.596 7fd49203f700 1 --2- 172.21.2.112:0/81110860 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd48c0a2b70 0x7fd48c0c0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:15.598 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.596 7fd49203f700 1 -- 172.21.2.112:0/81110860 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd48c0c0bb0 msgr2=0x7fd48c0a3850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:15.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.596 7fd49203f700 1 --2- 172.21.2.112:0/81110860 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd48c0c0bb0 0x7fd48c0a3850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:15.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.596 7fd49203f700 1 -- 172.21.2.112:0/81110860 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd48c0a4490 con 0x7fd48c0c0fb0 2021-02-10T11:14:15.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.596 7fd49203f700 1 --2- 172.21.2.112:0/81110860 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd48c0c0fb0 0x7fd48c0a3e70 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7fd48400b970 tx=0x7fd48400bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:15.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.596 7fd47e7fc700 1 -- 172.21.2.112:0/81110860 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd484009380 con 0x7fd48c0c0fb0 2021-02-10T11:14:15.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.597 7fd47e7fc700 1 -- 172.21.2.112:0/81110860 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd484005820 con 0x7fd48c0c0fb0 2021-02-10T11:14:15.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.597 7fd493a22700 1 -- 172.21.2.112:0/81110860 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd48c0a4490 con 0x7fd48c0c0fb0 2021-02-10T11:14:15.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.597 7fd47e7fc700 1 -- 172.21.2.112:0/81110860 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd484005d40 con 0x7fd48c0c0fb0 2021-02-10T11:14:15.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.597 7fd493a22700 1 -- 172.21.2.112:0/81110860 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd48c0a4490 con 0x7fd48c0c0fb0 2021-02-10T11:14:15.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.597 7fd47e7fc700 1 -- 172.21.2.112:0/81110860 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd484005cd0 con 0x7fd48c0c0fb0 2021-02-10T11:14:15.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.597 7fd47e7fc700 1 --2- 172.21.2.112:0/81110860 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd474018730 0x7fd47401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:15.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.598 7fd47e7fc700 1 -- 172.21.2.112:0/81110860 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fd484005ac0 con 0x7fd48c0c0fb0 2021-02-10T11:14:15.600 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.598 7fd49183e700 1 --2- 172.21.2.112:0/81110860 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd474018730 0x7fd47401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:15.600 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.598 7fd49183e700 1 --2- 172.21.2.112:0/81110860 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd474018730 0x7fd47401ac10 crc :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:15.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.628 7fd493a22700 1 --2- 172.21.2.112:0/81110860 >> [v2:172.21.2.120:6824/36385,v1:172.21.2.120:6826/36385] conn(0x7fd48c0c45a0 0x7fd48c0627c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:15.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.628 7fd493a22700 1 -- 172.21.2.112:0/81110860 --> [v2:172.21.2.120:6824/36385,v1:172.21.2.120:6826/36385] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd48c0a4d50 con 0x7fd48c0c45a0 2021-02-10T11:14:15.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.629 7fd49103d700 1 --2- 172.21.2.112:0/81110860 >> [v2:172.21.2.120:6824/36385,v1:172.21.2.120:6826/36385] conn(0x7fd48c0c45a0 0x7fd48c0627c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:15.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.630 7fd49103d700 1 --2- 172.21.2.112:0/81110860 >> [v2:172.21.2.120:6824/36385,v1:172.21.2.120:6826/36385] conn(0x7fd48c0c45a0 0x7fd48c0627c0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:15.629 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.631 7fd47e7fc700 1 -- 172.21.2.112:0/81110860 <== osd.6 v2:172.21.2.120:6824/36385 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+5151 (crc 0 0 0) 0x7fd480007c40 con 0x7fd48c0c45a0 2021-02-10T11:14:15.634 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.636 7fd493a22700 1 -- 172.21.2.112:0/81110860 --> [v2:172.21.2.120:6824/36385,v1:172.21.2.120:6826/36385] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fd48c083da0 con 0x7fd48c0c45a0 2021-02-10T11:14:15.635 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.637 7fd47e7fc700 1 -- 172.21.2.112:0/81110860 <== osd.6 v2:172.21.2.120:6824/36385 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fd4800078f0 con 0x7fd48c0c45a0 2021-02-10T11:14:15.636 INFO:teuthology.orchestra.run.gibba012.stdout:55834574856 2021-02-10T11:14:15.636 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.638 7fd493a22700 1 -- 172.21.2.112:0/81110860 >> [v2:172.21.2.120:6824/36385,v1:172.21.2.120:6826/36385] conn(0x7fd48c0c45a0 msgr2=0x7fd48c0627c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:15.637 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.638 7fd493a22700 1 --2- 172.21.2.112:0/81110860 >> [v2:172.21.2.120:6824/36385,v1:172.21.2.120:6826/36385] conn(0x7fd48c0c45a0 0x7fd48c0627c0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:15.637 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.638 7fd493a22700 1 -- 172.21.2.112:0/81110860 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd474018730 msgr2=0x7fd47401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:15.637 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.638 7fd493a22700 1 --2- 172.21.2.112:0/81110860 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd474018730 0x7fd47401ac10 crc :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:15.637 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.638 7fd493a22700 1 -- 172.21.2.112:0/81110860 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd48c0c0fb0 msgr2=0x7fd48c0a3e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:15.637 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.638 7fd493a22700 1 --2- 172.21.2.112:0/81110860 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd48c0c0fb0 0x7fd48c0a3e70 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7fd48400b970 tx=0x7fd48400bd10).stop 2021-02-10T11:14:15.637 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.638 7fd49203f700 1 -- 172.21.2.112:0/81110860 reap_dead start 2021-02-10T11:14:15.637 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.638 7fd493a22700 1 -- 172.21.2.112:0/81110860 shutdown_connections 2021-02-10T11:14:15.637 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.638 7fd493a22700 1 -- 172.21.2.112:0/81110860 >> 172.21.2.112:0/81110860 conn(0x7fd48c0bde70 msgr2=0x7fd48c0c3970 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:15.637 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.638 7fd493a22700 1 -- 172.21.2.112:0/81110860 shutdown_connections 2021-02-10T11:14:15.637 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:15.638 7fd493a22700 1 -- 172.21.2.112:0/81110860 wait complete. 2021-02-10T11:14:15.645 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell osd.7 flush_pg_stats 2021-02-10T11:14:16.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.055 7fd26a229700 1 Processor -- start 2021-02-10T11:14:16.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.055 7fd26a229700 1 -- start start 2021-02-10T11:14:16.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.055 7fd26a229700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fd2640e5470 0x7fd26407e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:16.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.055 7fd26a229700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd26407ed80 con 0x7fd26407f300 2021-02-10T11:14:16.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.055 7fd26a229700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2641b4010 con 0x7fd2640e5070 2021-02-10T11:14:16.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.055 7fd26a229700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2640d1790 con 0x7fd2640e5470 2021-02-10T11:14:16.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.056 7fd268846700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fd26407f300 0x7fd26407f700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:41822/0 (socket says 172.21.2.112:41822) 2021-02-10T11:14:16.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.056 7fd268846700 1 -- 172.21.2.112:0/1985199133 learned_addr learned my addr 172.21.2.112:0/1985199133 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:14:16.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.056 7fd2637fe700 1 --2- 172.21.2.112:0/1985199133 >> v2:172.21.2.120:3300/0 conn(0x7fd2640e5470 0x7fd26407e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:16.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.056 7fd262ffd700 1 -- 172.21.2.112:0/1985199133 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 394376842 0 0) 0x7fd25c001500 con 0x7fd26407f300 2021-02-10T11:14:16.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.056 7fd262ffd700 1 -- 172.21.2.112:0/1985199133 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd24c002fe0 con 0x7fd26407f300 2021-02-10T11:14:16.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.056 7fd262ffd700 1 -- 172.21.2.112:0/1985199133 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3632562632 0 0) 0x7fd254000c60 con 0x7fd2640e5070 2021-02-10T11:14:16.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.056 7fd262ffd700 1 -- 172.21.2.112:0/1985199133 --> v1:172.21.2.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd24c004370 con 0x7fd2640e5070 2021-02-10T11:14:16.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.056 7fd2637fe700 1 -- 172.21.2.112:0/1985199133 >> v1:172.21.2.120:6789/0 conn(0x7fd2640e5070 legacy=0x7fd2641b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:16.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.056 7fd2637fe700 1 -- 172.21.2.112:0/1985199133 >> v1:172.21.2.112:6789/0 conn(0x7fd26407f300 legacy=0x7fd26407f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:16.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.056 7fd2637fe700 1 -- 172.21.2.112:0/1985199133 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2640d19b0 con 0x7fd2640e5470 2021-02-10T11:14:16.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.057 7fd2637fe700 1 --2- 172.21.2.112:0/1985199133 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd2640e5470 0x7fd26407e6a0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7fd258006650 tx=0x7fd258009a80).ready entity=mon.1 client_cookie=b7343252aad9ce1b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:16.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.057 7fd262ffd700 1 -- 172.21.2.112:0/1985199133 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd25800c0c0 con 0x7fd2640e5470 2021-02-10T11:14:16.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.057 7fd262ffd700 1 -- 172.21.2.112:0/1985199133 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd25800c3c0 con 0x7fd2640e5470 2021-02-10T11:14:16.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.057 7fd262ffd700 1 -- 172.21.2.112:0/1985199133 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd25800c810 con 0x7fd2640e5470 2021-02-10T11:14:16.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.057 7fd26a229700 1 -- 172.21.2.112:0/1985199133 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd2640e5470 msgr2=0x7fd26407e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:16.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.057 7fd26a229700 1 --2- 172.21.2.112:0/1985199133 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd2640e5470 0x7fd26407e6a0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7fd258006650 tx=0x7fd258009a80).stop 2021-02-10T11:14:16.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.057 7fd26a229700 1 -- 172.21.2.112:0/1985199133 shutdown_connections 2021-02-10T11:14:16.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.057 7fd26a229700 1 --2- 172.21.2.112:0/1985199133 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd2640e5470 0x7fd26407e6a0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:16.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.057 7fd26a229700 1 -- 172.21.2.112:0/1985199133 >> 172.21.2.112:0/1985199133 conn(0x7fd2640ce690 msgr2=0x7fd2640e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:16.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.057 7fd26a229700 1 -- 172.21.2.112:0/1985199133 shutdown_connections 2021-02-10T11:14:16.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.057 7fd26a229700 1 -- 172.21.2.112:0/1985199133 wait complete. 2021-02-10T11:14:16.074 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.072 7fd26a229700 1 Processor -- start 2021-02-10T11:14:16.074 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.072 7fd26a229700 1 -- start start 2021-02-10T11:14:16.074 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.072 7fd26a229700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd2640bd700 0x7fd2640bdb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:16.074 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.072 7fd26a229700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd2640a4b10 0x7fd2640a4f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:16.074 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.072 7fd26a229700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd2640a5530 0x7fd2640a5930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:16.074 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.072 7fd26a229700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2640be230 con 0x7fd2640a4b10 2021-02-10T11:14:16.074 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.072 7fd26a229700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2640be450 con 0x7fd2640bd700 2021-02-10T11:14:16.074 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.072 7fd26a229700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2640be670 con 0x7fd2640a5530 2021-02-10T11:14:16.075 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.073 7fd263fff700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd2640bd700 0x7fd2640bdb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:16.075 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.073 7fd2637fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd2640a4b10 0x7fd2640a4f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:16.075 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.073 7fd263fff700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd2640bd700 0x7fd2640bdb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3300/0 says I am v2:172.21.2.112:44644/0 (socket says 172.21.2.112:44644) 2021-02-10T11:14:16.075 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.073 7fd2637fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd2640a4b10 0x7fd2640a4f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:42366/0 (socket says 172.21.2.112:42366) 2021-02-10T11:14:16.075 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.073 7fd263fff700 1 -- 172.21.2.112:0/2577210325 learned_addr learned my addr 172.21.2.112:0/2577210325 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:14:16.075 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.073 7fd268846700 1 --2- 172.21.2.112:0/2577210325 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd2640a5530 0x7fd2640a5930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:16.075 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.073 7fd2637fe700 1 -- 172.21.2.112:0/2577210325 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd2640a5530 msgr2=0x7fd2640a5930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:16.075 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.073 7fd2637fe700 1 --2- 172.21.2.112:0/2577210325 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd2640a5530 0x7fd2640a5930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:16.075 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.073 7fd2637fe700 1 -- 172.21.2.112:0/2577210325 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd2640bd700 msgr2=0x7fd2640bdb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:16.075 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.073 7fd2637fe700 1 --2- 172.21.2.112:0/2577210325 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd2640bd700 0x7fd2640bdb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:16.075 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.073 7fd2637fe700 1 -- 172.21.2.112:0/2577210325 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2640be890 con 0x7fd2640a4b10 2021-02-10T11:14:16.075 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.073 7fd2637fe700 1 --2- 172.21.2.112:0/2577210325 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd2640a4b10 0x7fd2640a4f10 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7fd258009240 tx=0x7fd25800b910).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:16.075 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.073 7fd260ff9700 1 -- 172.21.2.112:0/2577210325 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd2580093a0 con 0x7fd2640a4b10 2021-02-10T11:14:16.076 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.073 7fd260ff9700 1 -- 172.21.2.112:0/2577210325 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd25800bd00 con 0x7fd2640a4b10 2021-02-10T11:14:16.076 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.073 7fd26a229700 1 -- 172.21.2.112:0/2577210325 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2640be890 con 0x7fd2640a4b10 2021-02-10T11:14:16.076 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.074 7fd260ff9700 1 -- 172.21.2.112:0/2577210325 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd258005560 con 0x7fd2640a4b10 2021-02-10T11:14:16.076 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.074 7fd26a229700 1 -- 172.21.2.112:0/2577210325 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2640be890 con 0x7fd2640a4b10 2021-02-10T11:14:16.076 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.074 7fd260ff9700 1 -- 172.21.2.112:0/2577210325 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd258009280 con 0x7fd2640a4b10 2021-02-10T11:14:16.076 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.074 7fd260ff9700 1 --2- 172.21.2.112:0/2577210325 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd2440186d0 0x7fd24401abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:16.076 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.074 7fd260ff9700 1 -- 172.21.2.112:0/2577210325 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fd258032550 con 0x7fd2640a4b10 2021-02-10T11:14:16.076 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.075 7fd263fff700 1 --2- 172.21.2.112:0/2577210325 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd2440186d0 0x7fd24401abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:16.076 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.075 7fd263fff700 1 --2- 172.21.2.112:0/2577210325 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd2440186d0 0x7fd24401abd0 crc :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:16.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.105 7fd26a229700 1 --2- 172.21.2.112:0/2577210325 >> [v2:172.21.2.120:6802/36386,v1:172.21.2.120:6803/36386] conn(0x7fd2640c4580 0x7fd2640627c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:16.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.105 7fd26a229700 1 -- 172.21.2.112:0/2577210325 --> [v2:172.21.2.120:6802/36386,v1:172.21.2.120:6803/36386] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd26407ed80 con 0x7fd2640c4580 2021-02-10T11:14:16.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.105 7fd268846700 1 --2- 172.21.2.112:0/2577210325 >> [v2:172.21.2.120:6802/36386,v1:172.21.2.120:6803/36386] conn(0x7fd2640c4580 0x7fd2640627c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:16.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.105 7fd268846700 1 --2- 172.21.2.112:0/2577210325 >> [v2:172.21.2.120:6802/36386,v1:172.21.2.120:6803/36386] conn(0x7fd2640c4580 0x7fd2640627c0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:16.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.106 7fd260ff9700 1 -- 172.21.2.112:0/2577210325 <== osd.7 v2:172.21.2.120:6802/36386 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+5151 (crc 0 0 0) 0x7fd25c000cc0 con 0x7fd2640c4580 2021-02-10T11:14:16.111 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.113 7fd26a229700 1 -- 172.21.2.112:0/2577210325 --> [v2:172.21.2.120:6802/36386,v1:172.21.2.120:6803/36386] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fd2640c3aa0 con 0x7fd2640c4580 2021-02-10T11:14:16.112 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.113 7fd260ff9700 1 -- 172.21.2.112:0/2577210325 <== osd.7 v2:172.21.2.120:6802/36386 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fd25c000cc0 con 0x7fd2640c4580 2021-02-10T11:14:16.112 INFO:teuthology.orchestra.run.gibba012.stdout:55834574855 2021-02-10T11:14:16.113 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.114 7fd26a229700 1 -- 172.21.2.112:0/2577210325 >> [v2:172.21.2.120:6802/36386,v1:172.21.2.120:6803/36386] conn(0x7fd2640c4580 msgr2=0x7fd2640627c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:16.113 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.114 7fd26a229700 1 --2- 172.21.2.112:0/2577210325 >> [v2:172.21.2.120:6802/36386,v1:172.21.2.120:6803/36386] conn(0x7fd2640c4580 0x7fd2640627c0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:16.113 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.114 7fd26a229700 1 -- 172.21.2.112:0/2577210325 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd2440186d0 msgr2=0x7fd24401abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:16.113 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.114 7fd26a229700 1 --2- 172.21.2.112:0/2577210325 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd2440186d0 0x7fd24401abd0 crc :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:16.114 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.114 7fd26a229700 1 -- 172.21.2.112:0/2577210325 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd2640a4b10 msgr2=0x7fd2640a4f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:16.114 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.114 7fd26a229700 1 --2- 172.21.2.112:0/2577210325 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd2640a4b10 0x7fd2640a4f10 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7fd258009240 tx=0x7fd25800b910).stop 2021-02-10T11:14:16.114 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.114 7fd268846700 1 -- 172.21.2.112:0/2577210325 reap_dead start 2021-02-10T11:14:16.114 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.115 7fd26a229700 1 -- 172.21.2.112:0/2577210325 shutdown_connections 2021-02-10T11:14:16.114 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.115 7fd26a229700 1 -- 172.21.2.112:0/2577210325 >> 172.21.2.112:0/2577210325 conn(0x7fd2640c0cf0 msgr2=0x7fd2640a70a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:16.114 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.115 7fd26a229700 1 -- 172.21.2.112:0/2577210325 shutdown_connections 2021-02-10T11:14:16.114 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.115 7fd26a229700 1 -- 172.21.2.112:0/2577210325 wait complete. 2021-02-10T11:14:16.121 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd last-stat-seq osd.0 2021-02-10T11:14:16.546 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.530 7f1a6ff8f700 1 Processor -- start 2021-02-10T11:14:16.546 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.531 7f1a6ff8f700 1 -- start start 2021-02-10T11:14:16.546 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.531 7f1a6ff8f700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f1a680e5470 0x7f1a6807e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:16.546 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.531 7f1a6ff8f700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a6807ed80 con 0x7f1a680e5070 2021-02-10T11:14:16.546 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.531 7f1a6ff8f700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a681b4010 con 0x7f1a6807f300 2021-02-10T11:14:16.546 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.531 7f1a6ff8f700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a680d1790 con 0x7f1a680e5470 2021-02-10T11:14:16.546 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.531 7f1a6d5aa700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f1a680e5470 0x7f1a6807e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:16.546 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.531 7f1a6ddab700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f1a680e5070 0x7f1a681b3820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:41836/0 (socket says 172.21.2.112:41836) 2021-02-10T11:14:16.546 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.531 7f1a6ddab700 1 -- 172.21.2.112:0/2775095073 learned_addr learned my addr 172.21.2.112:0/2775095073 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:14:16.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.532 7f1a6cda9700 1 -- 172.21.2.112:0/2775095073 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2176806333 0 0) 0x7f1a64000ce0 con 0x7f1a680e5070 2021-02-10T11:14:16.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.532 7f1a6cda9700 1 -- 172.21.2.112:0/2775095073 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a54002fe0 con 0x7f1a680e5070 2021-02-10T11:14:16.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.532 7f1a6d5aa700 1 -- 172.21.2.112:0/2775095073 >> v1:172.21.2.120:6789/0 conn(0x7f1a6807f300 legacy=0x7f1a6807f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:14:16.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.532 7f1a6d5aa700 1 -- 172.21.2.112:0/2775095073 >> v1:172.21.2.112:6789/0 conn(0x7f1a680e5070 legacy=0x7f1a681b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:16.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.532 7f1a6d5aa700 1 -- 172.21.2.112:0/2775095073 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a680d19b0 con 0x7f1a680e5470 2021-02-10T11:14:16.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.532 7f1a6d5aa700 1 --2- 172.21.2.112:0/2775095073 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1a680e5470 0x7f1a6807e6a0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f1a5c006650 tx=0x7f1a5c009a80).ready entity=mon.0 client_cookie=c06f976bbac11d74 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:16.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.532 7f1a6cda9700 1 -- 172.21.2.112:0/2775095073 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1a5c0009b0 con 0x7f1a680e5470 2021-02-10T11:14:16.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.532 7f1a6cda9700 1 -- 172.21.2.112:0/2775095073 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a5c00c0c0 con 0x7f1a680e5470 2021-02-10T11:14:16.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.532 7f1a6cda9700 1 -- 172.21.2.112:0/2775095073 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1a5c00c520 con 0x7f1a680e5470 2021-02-10T11:14:16.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.532 7f1a6ff8f700 1 -- 172.21.2.112:0/2775095073 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1a680e5470 msgr2=0x7f1a6807e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:16.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.532 7f1a6ff8f700 1 --2- 172.21.2.112:0/2775095073 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1a680e5470 0x7f1a6807e6a0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f1a5c006650 tx=0x7f1a5c009a80).stop 2021-02-10T11:14:16.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.533 7f1a6ff8f700 1 -- 172.21.2.112:0/2775095073 shutdown_connections 2021-02-10T11:14:16.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.533 7f1a6ff8f700 1 --2- 172.21.2.112:0/2775095073 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1a680e5470 0x7f1a6807e6a0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:16.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.533 7f1a6ff8f700 1 -- 172.21.2.112:0/2775095073 >> 172.21.2.112:0/2775095073 conn(0x7f1a680ce690 msgr2=0x7f1a680e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:16.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.533 7f1a6ff8f700 1 -- 172.21.2.112:0/2775095073 shutdown_connections 2021-02-10T11:14:16.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.533 7f1a6ff8f700 1 -- 172.21.2.112:0/2775095073 wait complete. 2021-02-10T11:14:16.550 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.547 7f1a6ff8f700 1 Processor -- start 2021-02-10T11:14:16.550 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.548 7f1a6ff8f700 1 -- start start 2021-02-10T11:14:16.550 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.548 7f1a6ff8f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1a680be580 0x7f1a680be980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:16.550 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.548 7f1a6ff8f700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1a68097ed0 0x7f1a680982d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:16.550 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.548 7f1a6ff8f700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1a68099180 0x7f1a68099580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:16.550 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.548 7f1a6ff8f700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a680988f0 con 0x7f1a680be580 2021-02-10T11:14:16.550 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.548 7f1a6ff8f700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a680c1110 con 0x7f1a68097ed0 2021-02-10T11:14:16.550 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.548 7f1a6ff8f700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a68099ba0 con 0x7f1a68099180 2021-02-10T11:14:16.550 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.548 7f1a6ddab700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1a680be580 0x7f1a680be980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:16.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.548 7f1a6ddab700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1a680be580 0x7f1a680be980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:42382/0 (socket says 172.21.2.112:42382) 2021-02-10T11:14:16.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.548 7f1a6ddab700 1 -- 172.21.2.112:0/1144174134 learned_addr learned my addr 172.21.2.112:0/1144174134 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:14:16.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.549 7f1a6ddab700 1 -- 172.21.2.112:0/1144174134 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1a68099180 msgr2=0x7f1a68099580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:16.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.549 7f1a6ddab700 1 --2- 172.21.2.112:0/1144174134 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1a68099180 0x7f1a68099580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:16.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.549 7f1a6ddab700 1 -- 172.21.2.112:0/1144174134 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1a68097ed0 msgr2=0x7f1a680982d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:16.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.549 7f1a6ddab700 1 --2- 172.21.2.112:0/1144174134 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1a68097ed0 0x7f1a680982d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:16.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.549 7f1a6ddab700 1 -- 172.21.2.112:0/1144174134 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a68099dc0 con 0x7f1a680be580 2021-02-10T11:14:16.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.549 7f1a6ddab700 1 --2- 172.21.2.112:0/1144174134 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1a680be580 0x7f1a680be980 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f1a6400b5c0 tx=0x7f1a6400b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:16.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.549 7f1a5a7fc700 1 -- 172.21.2.112:0/1144174134 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1a6400bd50 con 0x7f1a680be580 2021-02-10T11:14:16.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.549 7f1a5a7fc700 1 -- 172.21.2.112:0/1144174134 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a64006280 con 0x7f1a680be580 2021-02-10T11:14:16.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.549 7f1a5a7fc700 1 -- 172.21.2.112:0/1144174134 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1a64004620 con 0x7f1a680be580 2021-02-10T11:14:16.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.549 7f1a6ff8f700 1 -- 172.21.2.112:0/1144174134 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a68099dc0 con 0x7f1a680be580 2021-02-10T11:14:16.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.549 7f1a6ff8f700 1 -- 172.21.2.112:0/1144174134 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a68099dc0 con 0x7f1a680be580 2021-02-10T11:14:16.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.550 7f1a5a7fc700 1 -- 172.21.2.112:0/1144174134 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1a640043a0 con 0x7f1a680be580 2021-02-10T11:14:16.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.550 7f1a5a7fc700 1 --2- 172.21.2.112:0/1144174134 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1a500186c0 0x7f1a5001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:16.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.550 7f1a5a7fc700 1 -- 172.21.2.112:0/1144174134 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f1a640325b0 con 0x7f1a680be580 2021-02-10T11:14:16.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.550 7f1a6d5aa700 1 --2- 172.21.2.112:0/1144174134 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1a500186c0 0x7f1a5001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:16.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.551 7f1a6d5aa700 1 --2- 172.21.2.112:0/1144174134 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1a500186c0 0x7f1a5001abc0 crc :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:16.579 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.581 7f1a6ff8f700 1 -- 172.21.2.112:0/1144174134 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a68098c50 con 0x7f1a680be580 2021-02-10T11:14:16.582 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.583 7f1a5a7fc700 1 -- 172.21.2.112:0/1144174134 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1a6402df10 con 0x7f1a680be580 2021-02-10T11:14:16.721 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.723 7f1a6ff8f700 1 -- 172.21.2.112:0/1144174134 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f1a68098c50 con 0x7f1a680be580 2021-02-10T11:14:16.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.723 7f1a5a7fc700 1 -- 172.21.2.112:0/1144174134 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f1a6402df10 con 0x7f1a680be580 2021-02-10T11:14:16.722 INFO:teuthology.orchestra.run.gibba012.stdout:55834574855 2021-02-10T11:14:16.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.725 7f1a6ff8f700 1 -- 172.21.2.112:0/1144174134 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1a500186c0 msgr2=0x7f1a5001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:16.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.725 7f1a6ff8f700 1 --2- 172.21.2.112:0/1144174134 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1a500186c0 0x7f1a5001abc0 crc :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:16.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.725 7f1a6ff8f700 1 -- 172.21.2.112:0/1144174134 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1a680be580 msgr2=0x7f1a680be980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:16.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.725 7f1a6ff8f700 1 --2- 172.21.2.112:0/1144174134 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1a680be580 0x7f1a680be980 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f1a6400b5c0 tx=0x7f1a6400b960).stop 2021-02-10T11:14:16.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.725 7f1a6ff8f700 1 -- 172.21.2.112:0/1144174134 shutdown_connections 2021-02-10T11:14:16.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.725 7f1a6ff8f700 1 --2- 172.21.2.112:0/1144174134 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1a500186c0 0x7f1a5001abc0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:16.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.725 7f1a6ff8f700 1 --2- 172.21.2.112:0/1144174134 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1a68097ed0 0x7f1a680982d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:16.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.725 7f1a6ff8f700 1 --2- 172.21.2.112:0/1144174134 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1a68099180 0x7f1a68099580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:16.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.725 7f1a6ff8f700 1 --2- 172.21.2.112:0/1144174134 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1a680be580 0x7f1a680be980 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:16.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.725 7f1a6ff8f700 1 -- 172.21.2.112:0/1144174134 >> 172.21.2.112:0/1144174134 conn(0x7f1a680bde70 msgr2=0x7f1a680c3970 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:16.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.725 7f1a6ff8f700 1 -- 172.21.2.112:0/1144174134 shutdown_connections 2021-02-10T11:14:16.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:16.726 7f1a6ff8f700 1 -- 172.21.2.112:0/1144174134 wait complete. 2021-02-10T11:14:16.732 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574855 for osd.0 2021-02-10T11:14:16.732 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd last-stat-seq osd.1 2021-02-10T11:14:17.157 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.141 7f781d87a700 1 Processor -- start 2021-02-10T11:14:17.157 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.141 7f781d87a700 1 -- start start 2021-02-10T11:14:17.157 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.141 7f781d87a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f78180e5070 0x7f78181b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:17.157 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.141 7f781d87a700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f78180e5470 0x7f781807d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:17.157 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.141 7f781d87a700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f781807ed80 con 0x7f78181b3e40 2021-02-10T11:14:17.157 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.141 7f781d87a700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7818080310 con 0x7f78180e5470 2021-02-10T11:14:17.157 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.141 7f781d87a700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7818080500 con 0x7f78180e5070 2021-02-10T11:14:17.157 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.142 7f7816d73700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f78180e5470 0x7f781807d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:17.158 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.142 7f7817574700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f78180e5070 0x7f78181b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:17.158 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.142 7f7817574700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f78180e5070 0x7f78181b3820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3301/0 says I am v2:172.21.2.112:42070/0 (socket says 172.21.2.112:42070) 2021-02-10T11:14:17.158 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.142 7f7816d73700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f78180e5470 0x7f781807d690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3300/0 says I am v2:172.21.2.112:44668/0 (socket says 172.21.2.112:44668) 2021-02-10T11:14:17.158 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.142 7f7817574700 1 -- 172.21.2.112:0/3572431364 learned_addr learned my addr 172.21.2.112:0/3572431364 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:14:17.158 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.143 7f7816d73700 1 -- 172.21.2.112:0/3572431364 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f78180e5070 msgr2=0x7f78181b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:14:17.158 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.143 7f7816d73700 1 --2- 172.21.2.112:0/3572431364 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f78180e5070 0x7f78181b3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:17.158 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.143 7f7816d73700 1 -- 172.21.2.112:0/3572431364 >> v1:172.21.2.120:6789/0 conn(0x7f78181b3e40 legacy=0x7f781807dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:17.158 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.143 7f7816d73700 1 -- 172.21.2.112:0/3572431364 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78180806f0 con 0x7f78180e5470 2021-02-10T11:14:17.158 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.143 7f7816d73700 1 --2- 172.21.2.112:0/3572431364 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f78180e5470 0x7f781807d690 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f780c006650 tx=0x7f780c009a80).ready entity=mon.1 client_cookie=70264176fe908cae server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:17.158 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.143 7f7816572700 1 -- 172.21.2.112:0/3572431364 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f780c00c0c0 con 0x7f78180e5470 2021-02-10T11:14:17.158 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.143 7f7816572700 1 -- 172.21.2.112:0/3572431364 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f780c00c340 con 0x7f78180e5470 2021-02-10T11:14:17.158 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.143 7f7816572700 1 -- 172.21.2.112:0/3572431364 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f780c00c860 con 0x7f78180e5470 2021-02-10T11:14:17.159 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.144 7f781d87a700 1 -- 172.21.2.112:0/3572431364 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f78180e5470 msgr2=0x7f781807d690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:17.159 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.144 7f781d87a700 1 --2- 172.21.2.112:0/3572431364 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f78180e5470 0x7f781807d690 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f780c006650 tx=0x7f780c009a80).stop 2021-02-10T11:14:17.159 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.144 7f781d87a700 1 -- 172.21.2.112:0/3572431364 shutdown_connections 2021-02-10T11:14:17.159 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.144 7f781d87a700 1 --2- 172.21.2.112:0/3572431364 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f78180e5470 0x7f781807d690 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:17.159 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.144 7f781d87a700 1 --2- 172.21.2.112:0/3572431364 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f78180e5070 0x7f78181b3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:17.159 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.144 7f781d87a700 1 -- 172.21.2.112:0/3572431364 >> 172.21.2.112:0/3572431364 conn(0x7f78180ce690 msgr2=0x7f78180e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:17.159 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.144 7f781d87a700 1 -- 172.21.2.112:0/3572431364 shutdown_connections 2021-02-10T11:14:17.159 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.144 7f781d87a700 1 -- 172.21.2.112:0/3572431364 wait complete. 2021-02-10T11:14:17.160 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.159 7f781d87a700 1 Processor -- start 2021-02-10T11:14:17.161 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.159 7f781d87a700 1 -- start start 2021-02-10T11:14:17.161 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.159 7f781d87a700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f781808f500 0x7f781819f900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:17.161 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.159 7f781d87a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f781808fd90 0x7f7818090190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:17.161 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.160 7f781d87a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f78180907b0 0x7f7818094370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:17.161 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.160 7f781d87a700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f781808f900 con 0x7f78180907b0 2021-02-10T11:14:17.161 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.160 7f781d87a700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f781819ff20 con 0x7f781808f500 2021-02-10T11:14:17.161 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.160 7f781d87a700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78181a0140 con 0x7f781808fd90 2021-02-10T11:14:17.161 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.160 7f7817d75700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f78180907b0 0x7f7818094370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:17.161 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.160 7f7817d75700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f78180907b0 0x7f7818094370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:42396/0 (socket says 172.21.2.112:42396) 2021-02-10T11:14:17.161 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.160 7f7817d75700 1 -- 172.21.2.112:0/603937039 learned_addr learned my addr 172.21.2.112:0/603937039 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:14:17.161 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.160 7f7816d73700 1 --2- 172.21.2.112:0/603937039 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f781808fd90 0x7f7818090190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:17.161 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.160 7f7817d75700 1 -- 172.21.2.112:0/603937039 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f781808fd90 msgr2=0x7f7818090190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:17.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.160 7f7817d75700 1 --2- 172.21.2.112:0/603937039 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f781808fd90 0x7f7818090190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:17.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.160 7f7817d75700 1 -- 172.21.2.112:0/603937039 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f781808f500 msgr2=0x7f781819f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:17.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.160 7f7817574700 1 --2- 172.21.2.112:0/603937039 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f781808f500 0x7f781819f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:17.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.160 7f7817d75700 1 --2- 172.21.2.112:0/603937039 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f781808f500 0x7f781819f900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:17.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.160 7f7817d75700 1 -- 172.21.2.112:0/603937039 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7818094990 con 0x7f78180907b0 2021-02-10T11:14:17.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.160 7f7816d73700 1 --2- 172.21.2.112:0/603937039 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f781808fd90 0x7f7818090190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:14:17.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.160 7f7817574700 1 --2- 172.21.2.112:0/603937039 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f781808f500 0x7f781819f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:14:17.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.160 7f7817d75700 1 --2- 172.21.2.112:0/603937039 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f78180907b0 0x7f7818094370 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f7810009940 tx=0x7f781000bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:17.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.161 7f77fffff700 1 -- 172.21.2.112:0/603937039 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7810000ce0 con 0x7f78180907b0 2021-02-10T11:14:17.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.161 7f77fffff700 1 -- 172.21.2.112:0/603937039 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f781000c2f0 con 0x7f78180907b0 2021-02-10T11:14:17.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.161 7f77fffff700 1 -- 172.21.2.112:0/603937039 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f781000c810 con 0x7f78180907b0 2021-02-10T11:14:17.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.161 7f781d87a700 1 -- 172.21.2.112:0/603937039 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7818094990 con 0x7f78180907b0 2021-02-10T11:14:17.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.161 7f781d87a700 1 -- 172.21.2.112:0/603937039 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7818094f20 con 0x7f78180907b0 2021-02-10T11:14:17.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.162 7f77fffff700 1 -- 172.21.2.112:0/603937039 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7810001220 con 0x7f78180907b0 2021-02-10T11:14:17.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.162 7f77fffff700 1 --2- 172.21.2.112:0/603937039 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f77f80186c0 0x7f77f801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:17.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.162 7f77fffff700 1 -- 172.21.2.112:0/603937039 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f7810032900 con 0x7f78180907b0 2021-02-10T11:14:17.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.162 7f7817574700 1 --2- 172.21.2.112:0/603937039 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f77f80186c0 0x7f77f801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:17.164 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.163 7f7817574700 1 --2- 172.21.2.112:0/603937039 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f77f80186c0 0x7f77f801abc0 crc :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:17.190 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.192 7f781d87a700 1 -- 172.21.2.112:0/603937039 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7818093e40 con 0x7f78180907b0 2021-02-10T11:14:17.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.194 7f77fffff700 1 -- 172.21.2.112:0/603937039 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7810000cc0 con 0x7f78180907b0 2021-02-10T11:14:17.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.332 7f781d87a700 1 -- 172.21.2.112:0/603937039 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f7818093e40 con 0x7f78180907b0 2021-02-10T11:14:17.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.332 7f77fffff700 1 -- 172.21.2.112:0/603937039 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f781001f6a0 con 0x7f78180907b0 2021-02-10T11:14:17.331 INFO:teuthology.orchestra.run.gibba012.stdout:55834574854 2021-02-10T11:14:17.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.334 7f781d87a700 1 -- 172.21.2.112:0/603937039 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f77f80186c0 msgr2=0x7f77f801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:17.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.334 7f781d87a700 1 --2- 172.21.2.112:0/603937039 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f77f80186c0 0x7f77f801abc0 crc :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:17.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.334 7f781d87a700 1 -- 172.21.2.112:0/603937039 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f78180907b0 msgr2=0x7f7818094370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:17.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.334 7f781d87a700 1 --2- 172.21.2.112:0/603937039 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f78180907b0 0x7f7818094370 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f7810009940 tx=0x7f781000bcf0).stop 2021-02-10T11:14:17.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.334 7f781d87a700 1 -- 172.21.2.112:0/603937039 shutdown_connections 2021-02-10T11:14:17.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.334 7f781d87a700 1 --2- 172.21.2.112:0/603937039 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f77f80186c0 0x7f77f801abc0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:17.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.334 7f781d87a700 1 --2- 172.21.2.112:0/603937039 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f78180907b0 0x7f7818094370 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:17.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.334 7f781d87a700 1 --2- 172.21.2.112:0/603937039 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f781808fd90 0x7f7818090190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:17.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.334 7f781d87a700 1 --2- 172.21.2.112:0/603937039 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f781808f500 0x7f781819f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:17.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.334 7f781d87a700 1 -- 172.21.2.112:0/603937039 >> 172.21.2.112:0/603937039 conn(0x7f78180b5560 msgr2=0x7f78180a6200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:17.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.335 7f781d87a700 1 -- 172.21.2.112:0/603937039 shutdown_connections 2021-02-10T11:14:17.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.335 7f781d87a700 1 -- 172.21.2.112:0/603937039 wait complete. 2021-02-10T11:14:17.341 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574854 for osd.1 2021-02-10T11:14:17.341 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd last-stat-seq osd.2 2021-02-10T11:14:17.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.755 7f81c06b0700 1 Processor -- start 2021-02-10T11:14:17.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.755 7f81c06b0700 1 -- start start 2021-02-10T11:14:17.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.756 7f81c06b0700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f81b80e5070 0x7f81b81b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:17.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.756 7f81c06b0700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f81b81b3e40 0x7f81b807d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:17.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.756 7f81c06b0700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81b807ed80 con 0x7f81b80e5470 2021-02-10T11:14:17.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.756 7f81c06b0700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81b8080310 con 0x7f81b80e5070 2021-02-10T11:14:17.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.756 7f81c06b0700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81b8080500 con 0x7f81b81b3e40 2021-02-10T11:14:17.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.756 7f81be4cc700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f81b80e5070 0x7f81b81b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:17.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.756 7f81beccd700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f81b80e5470 0x7f81b807dcb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:41866/0 (socket says 172.21.2.112:41866) 2021-02-10T11:14:17.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.756 7f81beccd700 1 -- 172.21.2.112:0/3132986636 learned_addr learned my addr 172.21.2.112:0/3132986636 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:14:17.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.756 7f81bdccb700 1 --2- 172.21.2.112:0/3132986636 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f81b81b3e40 0x7f81b807d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:17.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.757 7f81be4cc700 1 -- 172.21.2.112:0/3132986636 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f81b81b3e40 msgr2=0x7f81b807d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:14:17.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.757 7f81be4cc700 1 --2- 172.21.2.112:0/3132986636 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f81b81b3e40 0x7f81b807d690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:17.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.757 7f81be4cc700 1 -- 172.21.2.112:0/3132986636 >> v1:172.21.2.112:6789/0 conn(0x7f81b80e5470 legacy=0x7f81b807dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:17.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.757 7f81bd4ca700 1 -- 172.21.2.112:0/3132986636 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1643153644 0 0) 0x7f81b0001500 con 0x7f81b80e5470 2021-02-10T11:14:17.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.757 7f81be4cc700 1 -- 172.21.2.112:0/3132986636 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81b80806f0 con 0x7f81b80e5070 2021-02-10T11:14:17.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.757 7f81bdccb700 1 --2- 172.21.2.112:0/3132986636 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f81b81b3e40 0x7f81b807d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:14:17.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.757 7f81be4cc700 1 --2- 172.21.2.112:0/3132986636 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f81b80e5070 0x7f81b81b3820 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f81a8006650 tx=0x7f81a8009a80).ready entity=mon.0 client_cookie=e824a23d599fdce3 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:17.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.757 7f81bd4ca700 1 -- 172.21.2.112:0/3132986636 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f81a80009b0 con 0x7f81b80e5070 2021-02-10T11:14:17.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.757 7f81bd4ca700 1 -- 172.21.2.112:0/3132986636 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81a800c0c0 con 0x7f81b80e5070 2021-02-10T11:14:17.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.757 7f81bd4ca700 1 -- 172.21.2.112:0/3132986636 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f81a800c5e0 con 0x7f81b80e5070 2021-02-10T11:14:17.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.757 7f81c06b0700 1 -- 172.21.2.112:0/3132986636 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f81b80e5070 msgr2=0x7f81b81b3820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:17.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.757 7f81c06b0700 1 --2- 172.21.2.112:0/3132986636 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f81b80e5070 0x7f81b81b3820 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f81a8006650 tx=0x7f81a8009a80).stop 2021-02-10T11:14:17.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.758 7f81c06b0700 1 -- 172.21.2.112:0/3132986636 shutdown_connections 2021-02-10T11:14:17.774 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.758 7f81c06b0700 1 --2- 172.21.2.112:0/3132986636 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f81b81b3e40 0x7f81b807d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:17.774 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.758 7f81c06b0700 1 --2- 172.21.2.112:0/3132986636 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f81b80e5070 0x7f81b81b3820 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:17.774 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.758 7f81c06b0700 1 -- 172.21.2.112:0/3132986636 >> 172.21.2.112:0/3132986636 conn(0x7f81b80ce690 msgr2=0x7f81b80e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:17.774 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.758 7f81c06b0700 1 -- 172.21.2.112:0/3132986636 shutdown_connections 2021-02-10T11:14:17.774 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.758 7f81c06b0700 1 -- 172.21.2.112:0/3132986636 wait complete. 2021-02-10T11:14:17.774 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.773 7f81c06b0700 1 Processor -- start 2021-02-10T11:14:17.774 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.773 7f81c06b0700 1 -- start start 2021-02-10T11:14:17.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.774 7f81c06b0700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f81b80b4ad0 0x7f81b80a78d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:17.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.774 7f81c06b0700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f81b80a7ef0 0x7f81b81a6d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:17.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.774 7f81c06b0700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f81b80a82f0 0x7f81b81a73b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:17.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.774 7f81c06b0700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81b80aa3c0 con 0x7f81b80b4ad0 2021-02-10T11:14:17.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.774 7f81c06b0700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81b80a88c0 con 0x7f81b80a82f0 2021-02-10T11:14:17.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.774 7f81c06b0700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81b80b4ed0 con 0x7f81b80a7ef0 2021-02-10T11:14:17.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.774 7f81be4cc700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f81b80b4ad0 0x7f81b80a78d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:17.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.774 7f81be4cc700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f81b80b4ad0 0x7f81b80a78d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:42406/0 (socket says 172.21.2.112:42406) 2021-02-10T11:14:17.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.774 7f81be4cc700 1 -- 172.21.2.112:0/463544535 learned_addr learned my addr 172.21.2.112:0/463544535 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:14:17.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.774 7f81beccd700 1 --2- 172.21.2.112:0/463544535 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f81b80a82f0 0x7f81b81a73b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:17.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.774 7f81bdccb700 1 --2- 172.21.2.112:0/463544535 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f81b80a7ef0 0x7f81b81a6d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:17.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.774 7f81be4cc700 1 -- 172.21.2.112:0/463544535 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f81b80a7ef0 msgr2=0x7f81b81a6d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:17.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.774 7f81be4cc700 1 --2- 172.21.2.112:0/463544535 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f81b80a7ef0 0x7f81b81a6d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:17.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.774 7f81be4cc700 1 -- 172.21.2.112:0/463544535 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f81b80a82f0 msgr2=0x7f81b81a73b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:17.777 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.774 7f81be4cc700 1 --2- 172.21.2.112:0/463544535 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f81b80a82f0 0x7f81b81a73b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:17.777 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.774 7f81be4cc700 1 -- 172.21.2.112:0/463544535 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81b81a79d0 con 0x7f81b80b4ad0 2021-02-10T11:14:17.777 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.774 7f81beccd700 1 --2- 172.21.2.112:0/463544535 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f81b80a82f0 0x7f81b81a73b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:14:17.777 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.774 7f81be4cc700 1 --2- 172.21.2.112:0/463544535 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f81b80b4ad0 0x7f81b80a78d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f81a8009630 tx=0x7f81a800b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:17.777 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.775 7f81aeffd700 1 -- 172.21.2.112:0/463544535 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f81a80093a0 con 0x7f81b80b4ad0 2021-02-10T11:14:17.777 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.775 7f81aeffd700 1 -- 172.21.2.112:0/463544535 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81a800bd70 con 0x7f81b80b4ad0 2021-02-10T11:14:17.777 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.775 7f81aeffd700 1 -- 172.21.2.112:0/463544535 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f81a8005560 con 0x7f81b80b4ad0 2021-02-10T11:14:17.777 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.775 7f81c06b0700 1 -- 172.21.2.112:0/463544535 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81b81a79d0 con 0x7f81b80b4ad0 2021-02-10T11:14:17.777 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.775 7f81c06b0700 1 -- 172.21.2.112:0/463544535 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81b80a9130 con 0x7f81b80b4ad0 2021-02-10T11:14:17.777 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.775 7f81aeffd700 1 -- 172.21.2.112:0/463544535 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f81a800a000 con 0x7f81b80b4ad0 2021-02-10T11:14:17.777 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.775 7f81aeffd700 1 --2- 172.21.2.112:0/463544535 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f81a00186c0 0x7f81a001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:17.777 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.775 7f81aeffd700 1 -- 172.21.2.112:0/463544535 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f81a80325b0 con 0x7f81b80b4ad0 2021-02-10T11:14:17.777 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.776 7f81bdccb700 1 --2- 172.21.2.112:0/463544535 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f81a00186c0 0x7f81a001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:17.778 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.776 7f81bdccb700 1 --2- 172.21.2.112:0/463544535 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f81a00186c0 0x7f81a001abc0 crc :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:17.804 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.806 7f81c06b0700 1 -- 172.21.2.112:0/463544535 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81b8084a90 con 0x7f81b80b4ad0 2021-02-10T11:14:17.807 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.808 7f81aeffd700 1 -- 172.21.2.112:0/463544535 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f81a8009340 con 0x7f81b80b4ad0 2021-02-10T11:14:17.949 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.950 7f81c06b0700 1 -- 172.21.2.112:0/463544535 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f81b8084a90 con 0x7f81b80b4ad0 2021-02-10T11:14:17.949 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.950 7f81aeffd700 1 -- 172.21.2.112:0/463544535 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f81a800a000 con 0x7f81b80b4ad0 2021-02-10T11:14:17.949 INFO:teuthology.orchestra.run.gibba012.stdout:55834574855 2021-02-10T11:14:17.951 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.952 7f81c06b0700 1 -- 172.21.2.112:0/463544535 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f81a00186c0 msgr2=0x7f81a001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:17.951 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.952 7f81c06b0700 1 --2- 172.21.2.112:0/463544535 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f81a00186c0 0x7f81a001abc0 crc :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:17.951 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.952 7f81c06b0700 1 -- 172.21.2.112:0/463544535 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f81b80b4ad0 msgr2=0x7f81b80a78d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:17.951 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.952 7f81c06b0700 1 --2- 172.21.2.112:0/463544535 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f81b80b4ad0 0x7f81b80a78d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f81a8009630 tx=0x7f81a800b980).stop 2021-02-10T11:14:17.951 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.953 7f81c06b0700 1 -- 172.21.2.112:0/463544535 shutdown_connections 2021-02-10T11:14:17.951 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.953 7f81c06b0700 1 --2- 172.21.2.112:0/463544535 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f81a00186c0 0x7f81a001abc0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:17.951 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.953 7f81c06b0700 1 --2- 172.21.2.112:0/463544535 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f81b80a82f0 0x7f81b81a73b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:17.951 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.953 7f81c06b0700 1 --2- 172.21.2.112:0/463544535 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f81b80a7ef0 0x7f81b81a6d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:17.952 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.953 7f81c06b0700 1 --2- 172.21.2.112:0/463544535 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f81b80b4ad0 0x7f81b80a78d0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:17.952 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.953 7f81c06b0700 1 -- 172.21.2.112:0/463544535 >> 172.21.2.112:0/463544535 conn(0x7f81b80b5560 msgr2=0x7f81b80a6200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:17.952 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.953 7f81c06b0700 1 -- 172.21.2.112:0/463544535 shutdown_connections 2021-02-10T11:14:17.952 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:17.953 7f81c06b0700 1 -- 172.21.2.112:0/463544535 wait complete. 2021-02-10T11:14:17.959 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574855 for osd.2 2021-02-10T11:14:17.959 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd last-stat-seq osd.3 2021-02-10T11:14:18.384 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.369 7f41b89fb700 1 Processor -- start 2021-02-10T11:14:18.384 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.369 7f41b89fb700 1 -- start start 2021-02-10T11:14:18.384 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.369 7f41b89fb700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f41b40e5070 0x7f41b41b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:18.384 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.369 7f41b89fb700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41b407ed80 con 0x7f41b41b3e40 2021-02-10T11:14:18.384 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.369 7f41b89fb700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41b407e670 con 0x7f41b40e5470 2021-02-10T11:14:18.384 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.369 7f41b89fb700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41b407e890 con 0x7f41b40e5070 2021-02-10T11:14:18.384 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.369 7f41b261c700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f41b40e5070 0x7f41b41b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:18.385 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.369 7f41b1e1b700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f41b41b3e40 0x7f41b407d690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:41878/0 (socket says 172.21.2.112:41878) 2021-02-10T11:14:18.385 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.369 7f41b1e1b700 1 -- 172.21.2.112:0/1565379761 learned_addr learned my addr 172.21.2.112:0/1565379761 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:14:18.385 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.370 7f41b161a700 1 -- 172.21.2.112:0/1565379761 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3437387901 0 0) 0x7f41a80010b0 con 0x7f41b41b3e40 2021-02-10T11:14:18.385 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.370 7f41b161a700 1 -- 172.21.2.112:0/1565379761 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f419c002fe0 con 0x7f41b41b3e40 2021-02-10T11:14:18.385 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.370 7f41b261c700 1 -- 172.21.2.112:0/1565379761 >> v1:172.21.2.120:6789/0 conn(0x7f41b40e5470 legacy=0x7f41b407de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:14:18.385 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.370 7f41b261c700 1 -- 172.21.2.112:0/1565379761 >> v1:172.21.2.112:6789/0 conn(0x7f41b41b3e40 legacy=0x7f41b407d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:18.385 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.370 7f41b261c700 1 -- 172.21.2.112:0/1565379761 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41b407eab0 con 0x7f41b40e5070 2021-02-10T11:14:18.385 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.370 7f41b261c700 1 --2- 172.21.2.112:0/1565379761 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f41b40e5070 0x7f41b41b3820 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f41a4006650 tx=0x7f41a4009a80).ready entity=mon.0 client_cookie=696c1920ca1c9946 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:18.385 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.370 7f41b161a700 1 -- 172.21.2.112:0/1565379761 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f41a40009b0 con 0x7f41b40e5070 2021-02-10T11:14:18.385 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.370 7f41b161a700 1 -- 172.21.2.112:0/1565379761 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41a400c0c0 con 0x7f41b40e5070 2021-02-10T11:14:18.385 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.370 7f41b161a700 1 -- 172.21.2.112:0/1565379761 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f41a400c5e0 con 0x7f41b40e5070 2021-02-10T11:14:18.385 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.371 7f41b89fb700 1 -- 172.21.2.112:0/1565379761 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f41b40e5070 msgr2=0x7f41b41b3820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:18.385 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.371 7f41b89fb700 1 --2- 172.21.2.112:0/1565379761 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f41b40e5070 0x7f41b41b3820 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f41a4006650 tx=0x7f41a4009a80).stop 2021-02-10T11:14:18.386 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.371 7f41b89fb700 1 -- 172.21.2.112:0/1565379761 shutdown_connections 2021-02-10T11:14:18.386 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.371 7f41b89fb700 1 --2- 172.21.2.112:0/1565379761 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f41b40e5070 0x7f41b41b3820 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:18.386 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.371 7f41b89fb700 1 -- 172.21.2.112:0/1565379761 >> 172.21.2.112:0/1565379761 conn(0x7f41b40ce690 msgr2=0x7f41b40e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:18.386 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.371 7f41b89fb700 1 -- 172.21.2.112:0/1565379761 shutdown_connections 2021-02-10T11:14:18.386 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.371 7f41b89fb700 1 -- 172.21.2.112:0/1565379761 wait complete. 2021-02-10T11:14:18.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.386 7f41b89fb700 1 Processor -- start 2021-02-10T11:14:18.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.386 7f41b89fb700 1 -- start start 2021-02-10T11:14:18.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.386 7f41b89fb700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f41b40be580 0x7f41b40be980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:18.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.386 7f41b89fb700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f41b4097ed0 0x7f41b40982d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:18.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.386 7f41b89fb700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f41b4099180 0x7f41b4099580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:18.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.386 7f41b89fb700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41b40988f0 con 0x7f41b4097ed0 2021-02-10T11:14:18.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.386 7f41b89fb700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41b40c1110 con 0x7f41b40be580 2021-02-10T11:14:18.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.386 7f41b89fb700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41b4099ba0 con 0x7f41b4099180 2021-02-10T11:14:18.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.387 7f41b1e1b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f41b4097ed0 0x7f41b40982d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:18.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.387 7f41b1e1b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f41b4097ed0 0x7f41b40982d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:42422/0 (socket says 172.21.2.112:42422) 2021-02-10T11:14:18.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.387 7f41b1e1b700 1 -- 172.21.2.112:0/717064719 learned_addr learned my addr 172.21.2.112:0/717064719 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:14:18.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.387 7f41b1e1b700 1 -- 172.21.2.112:0/717064719 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f41b4099180 msgr2=0x7f41b4099580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:18.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.387 7f41b1e1b700 1 --2- 172.21.2.112:0/717064719 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f41b4099180 0x7f41b4099580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:18.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.387 7f41b1e1b700 1 -- 172.21.2.112:0/717064719 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f41b40be580 msgr2=0x7f41b40be980 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:14:18.389 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.387 7f41b1e1b700 1 --2- 172.21.2.112:0/717064719 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f41b40be580 0x7f41b40be980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:18.389 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.387 7f41b1e1b700 1 -- 172.21.2.112:0/717064719 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41b4099dc0 con 0x7f41b4097ed0 2021-02-10T11:14:18.389 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.387 7f41b1e1b700 1 --2- 172.21.2.112:0/717064719 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f41b4097ed0 0x7f41b40982d0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f41a8001c60 tx=0x7f41a800a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:18.389 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.387 7f41a2ffd700 1 -- 172.21.2.112:0/717064719 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f41a800acc0 con 0x7f41b4097ed0 2021-02-10T11:14:18.389 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.387 7f41a2ffd700 1 -- 172.21.2.112:0/717064719 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41a8004350 con 0x7f41b4097ed0 2021-02-10T11:14:18.389 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.387 7f41a2ffd700 1 -- 172.21.2.112:0/717064719 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f41a8004870 con 0x7f41b4097ed0 2021-02-10T11:14:18.389 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.387 7f41b89fb700 1 -- 172.21.2.112:0/717064719 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41b4099dc0 con 0x7f41b4097ed0 2021-02-10T11:14:18.389 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.387 7f41b89fb700 1 -- 172.21.2.112:0/717064719 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41b4099dc0 con 0x7f41b4097ed0 2021-02-10T11:14:18.389 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.388 7f41a2ffd700 1 -- 172.21.2.112:0/717064719 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f41a8007f30 con 0x7f41b4097ed0 2021-02-10T11:14:18.389 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.388 7f41a2ffd700 1 --2- 172.21.2.112:0/717064719 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f41980186c0 0x7f419801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:18.390 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.388 7f41a2ffd700 1 -- 172.21.2.112:0/717064719 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f41a8032520 con 0x7f41b4097ed0 2021-02-10T11:14:18.390 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.389 7f41b261c700 1 --2- 172.21.2.112:0/717064719 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f41980186c0 0x7f419801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:18.390 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.389 7f41b261c700 1 --2- 172.21.2.112:0/717064719 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f41980186c0 0x7f419801abc0 crc :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:18.418 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.419 7f41b89fb700 1 -- 172.21.2.112:0/717064719 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41b4098c50 con 0x7f41b4097ed0 2021-02-10T11:14:18.420 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.422 7f41a2ffd700 1 -- 172.21.2.112:0/717064719 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f41a8004140 con 0x7f41b4097ed0 2021-02-10T11:14:18.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.562 7f41b89fb700 1 -- 172.21.2.112:0/717064719 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f41b4098c50 con 0x7f41b4097ed0 2021-02-10T11:14:18.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.562 7f41a2ffd700 1 -- 172.21.2.112:0/717064719 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f41a8007f30 con 0x7f41b4097ed0 2021-02-10T11:14:18.561 INFO:teuthology.orchestra.run.gibba012.stdout:55834574856 2021-02-10T11:14:18.563 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.564 7f41b89fb700 1 -- 172.21.2.112:0/717064719 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f41980186c0 msgr2=0x7f419801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:18.563 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.564 7f41b89fb700 1 --2- 172.21.2.112:0/717064719 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f41980186c0 0x7f419801abc0 crc :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:18.563 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.564 7f41b89fb700 1 -- 172.21.2.112:0/717064719 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f41b4097ed0 msgr2=0x7f41b40982d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:18.563 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.564 7f41b89fb700 1 --2- 172.21.2.112:0/717064719 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f41b4097ed0 0x7f41b40982d0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f41a8001c60 tx=0x7f41a800a8d0).stop 2021-02-10T11:14:18.563 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.564 7f41b89fb700 1 -- 172.21.2.112:0/717064719 shutdown_connections 2021-02-10T11:14:18.563 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.564 7f41b89fb700 1 --2- 172.21.2.112:0/717064719 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f41980186c0 0x7f419801abc0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:18.564 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.565 7f41b89fb700 1 --2- 172.21.2.112:0/717064719 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f41b4099180 0x7f41b4099580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:18.564 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.565 7f41b89fb700 1 --2- 172.21.2.112:0/717064719 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f41b4097ed0 0x7f41b40982d0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:18.564 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.565 7f41b89fb700 1 --2- 172.21.2.112:0/717064719 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f41b40be580 0x7f41b40be980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:18.564 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.565 7f41b89fb700 1 -- 172.21.2.112:0/717064719 >> 172.21.2.112:0/717064719 conn(0x7f41b40bde70 msgr2=0x7f41b40c3970 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:18.564 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.565 7f41b89fb700 1 -- 172.21.2.112:0/717064719 shutdown_connections 2021-02-10T11:14:18.564 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.565 7f41b89fb700 1 -- 172.21.2.112:0/717064719 wait complete. 2021-02-10T11:14:18.571 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.3 2021-02-10T11:14:18.572 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd last-stat-seq osd.4 2021-02-10T11:14:18.995 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.980 7f3c7d82f700 1 Processor -- start 2021-02-10T11:14:18.995 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.980 7f3c7d82f700 1 -- start start 2021-02-10T11:14:18.995 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.981 7f3c7d82f700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f3c780e5070 0x7f3c781b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:18.995 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.981 7f3c7d82f700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f3c780e5470 0x7f3c78080310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:18.995 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.981 7f3c7d82f700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c7807ed80 con 0x7f3c781b3e40 2021-02-10T11:14:18.995 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.981 7f3c7d82f700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c780d1790 con 0x7f3c780e5470 2021-02-10T11:14:18.995 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.981 7f3c7d82f700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c780d19b0 con 0x7f3c780e5070 2021-02-10T11:14:18.995 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.981 7f3c77d75700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f3c780e5470 0x7f3c78080310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:18.995 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.981 7f3c77d75700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f3c780e5470 0x7f3c78080310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:42428/0 (socket says 172.21.2.112:42428) 2021-02-10T11:14:18.996 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.981 7f3c77d75700 1 -- 172.21.2.112:0/2941696535 learned_addr learned my addr 172.21.2.112:0/2941696535 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:14:18.996 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.981 7f3c77d75700 1 -- 172.21.2.112:0/2941696535 >> v2:172.21.2.120:3300/0 conn(0x7f3c780e5070 msgr2=0x7f3c781b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:14:18.996 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.981 7f3c77d75700 1 --2- 172.21.2.112:0/2941696535 >> v2:172.21.2.120:3300/0 conn(0x7f3c780e5070 0x7f3c781b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:18.996 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.981 7f3c77d75700 1 -- 172.21.2.112:0/2941696535 >> v1:172.21.2.120:6789/0 conn(0x7f3c781b3e40 legacy=0x7f3c7807d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:14:18.996 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.981 7f3c77d75700 1 -- 172.21.2.112:0/2941696535 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c780d1bd0 con 0x7f3c780e5470 2021-02-10T11:14:18.996 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.981 7f3c77d75700 1 --2- 172.21.2.112:0/2941696535 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3c780e5470 0x7f3c78080310 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f3c70006f00 tx=0x7f3c70009f10).ready entity=mon.0 client_cookie=f7148f5f71531faf server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:18.996 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.982 7f3c76572700 1 -- 172.21.2.112:0/2941696535 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3c7000b3b0 con 0x7f3c780e5470 2021-02-10T11:14:18.996 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.982 7f3c76572700 1 -- 172.21.2.112:0/2941696535 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c7000b630 con 0x7f3c780e5470 2021-02-10T11:14:18.996 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.982 7f3c76572700 1 -- 172.21.2.112:0/2941696535 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3c7000bb50 con 0x7f3c780e5470 2021-02-10T11:14:18.996 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.982 7f3c7d82f700 1 -- 172.21.2.112:0/2941696535 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3c780e5470 msgr2=0x7f3c78080310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:18.996 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.982 7f3c7d82f700 1 --2- 172.21.2.112:0/2941696535 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3c780e5470 0x7f3c78080310 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f3c70006f00 tx=0x7f3c70009f10).stop 2021-02-10T11:14:18.997 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.982 7f3c7d82f700 1 -- 172.21.2.112:0/2941696535 shutdown_connections 2021-02-10T11:14:18.997 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.982 7f3c7d82f700 1 --2- 172.21.2.112:0/2941696535 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3c780e5470 0x7f3c78080310 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:18.997 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.982 7f3c7d82f700 1 --2- 172.21.2.112:0/2941696535 >> v2:172.21.2.120:3300/0 conn(0x7f3c780e5070 0x7f3c781b3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:18.997 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.982 7f3c7d82f700 1 -- 172.21.2.112:0/2941696535 >> 172.21.2.112:0/2941696535 conn(0x7f3c780ce690 msgr2=0x7f3c780e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:18.997 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.982 7f3c7d82f700 1 -- 172.21.2.112:0/2941696535 shutdown_connections 2021-02-10T11:14:18.997 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.982 7f3c7d82f700 1 -- 172.21.2.112:0/2941696535 wait complete. 2021-02-10T11:14:18.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.997 7f3c7d82f700 1 Processor -- start 2021-02-10T11:14:18.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.997 7f3c7d82f700 1 -- start start 2021-02-10T11:14:18.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.997 7f3c7d82f700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3c781a2fa0 0x7f3c780914f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:18.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.997 7f3c7d82f700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3c781a33a0 0x7f3c78091b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:18.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.997 7f3c7d82f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3c780921c0 0x7f3c780925c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:18.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.997 7f3c7d82f700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c78092c60 con 0x7f3c780921c0 2021-02-10T11:14:18.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.997 7f3c7d82f700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c78092e80 con 0x7f3c781a33a0 2021-02-10T11:14:18.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.997 7f3c7d82f700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c780930a0 con 0x7f3c781a2fa0 2021-02-10T11:14:18.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.997 7f3c77d75700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3c780921c0 0x7f3c780925c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:18.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.997 7f3c77d75700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3c780921c0 0x7f3c780925c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:42436/0 (socket says 172.21.2.112:42436) 2021-02-10T11:14:18.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.997 7f3c77d75700 1 -- 172.21.2.112:0/1277872832 learned_addr learned my addr 172.21.2.112:0/1277872832 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:14:18.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.998 7f3c77d75700 1 -- 172.21.2.112:0/1277872832 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3c781a2fa0 msgr2=0x7f3c780914f0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:14:19.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.998 7f3c77d75700 1 --2- 172.21.2.112:0/1277872832 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3c781a2fa0 0x7f3c780914f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:19.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.998 7f3c77d75700 1 -- 172.21.2.112:0/1277872832 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3c781a33a0 msgr2=0x7f3c78091b10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:14:19.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.998 7f3c77d75700 1 --2- 172.21.2.112:0/1277872832 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3c781a33a0 0x7f3c78091b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:19.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.998 7f3c77d75700 1 -- 172.21.2.112:0/1277872832 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c780932c0 con 0x7f3c780921c0 2021-02-10T11:14:19.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.998 7f3c77d75700 1 --2- 172.21.2.112:0/1277872832 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3c780921c0 0x7f3c780925c0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f3c70009390 tx=0x7f3c7000bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:19.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.998 7f3c6ffff700 1 -- 172.21.2.112:0/1277872832 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3c70003030 con 0x7f3c780921c0 2021-02-10T11:14:19.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.998 7f3c6ffff700 1 -- 172.21.2.112:0/1277872832 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c700068e0 con 0x7f3c780921c0 2021-02-10T11:14:19.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.998 7f3c6ffff700 1 -- 172.21.2.112:0/1277872832 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3c7000c570 con 0x7f3c780921c0 2021-02-10T11:14:19.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.998 7f3c7d82f700 1 -- 172.21.2.112:0/1277872832 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c780932c0 con 0x7f3c780921c0 2021-02-10T11:14:19.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.998 7f3c7d82f700 1 -- 172.21.2.112:0/1277872832 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c780ba3b0 con 0x7f3c780921c0 2021-02-10T11:14:19.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.999 7f3c6ffff700 1 -- 172.21.2.112:0/1277872832 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3c70001270 con 0x7f3c780921c0 2021-02-10T11:14:19.001 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.999 7f3c6ffff700 1 --2- 172.21.2.112:0/1277872832 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3c5c0186c0 0x7f3c5c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:19.001 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.999 7f3c6ffff700 1 -- 172.21.2.112:0/1277872832 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f3c700328c0 con 0x7f3c780921c0 2021-02-10T11:14:19.001 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:18.999 7f3c77574700 1 --2- 172.21.2.112:0/1277872832 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3c5c0186c0 0x7f3c5c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:19.001 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:19.000 7f3c77574700 1 --2- 172.21.2.112:0/1277872832 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3c5c0186c0 0x7f3c5c01abc0 crc :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:19.029 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:19.030 7f3c7d82f700 1 -- 172.21.2.112:0/1277872832 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c78083da0 con 0x7f3c780921c0 2021-02-10T11:14:19.032 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:19.033 7f3c6ffff700 1 -- 172.21.2.112:0/1277872832 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3c70009430 con 0x7f3c780921c0 2021-02-10T11:14:19.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:19.172 7f3c7d82f700 1 -- 172.21.2.112:0/1277872832 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f3c7807ed80 con 0x7f3c780921c0 2021-02-10T11:14:19.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:19.173 7f3c6ffff700 1 -- 172.21.2.112:0/1277872832 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f3c7001f630 con 0x7f3c780921c0 2021-02-10T11:14:19.172 INFO:teuthology.orchestra.run.gibba012.stdout:55834574856 2021-02-10T11:14:19.173 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:19.175 7f3c7d82f700 1 -- 172.21.2.112:0/1277872832 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3c5c0186c0 msgr2=0x7f3c5c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:19.173 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:19.175 7f3c7d82f700 1 --2- 172.21.2.112:0/1277872832 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3c5c0186c0 0x7f3c5c01abc0 crc :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:19.173 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:19.175 7f3c7d82f700 1 -- 172.21.2.112:0/1277872832 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3c780921c0 msgr2=0x7f3c780925c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:19.173 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:19.175 7f3c7d82f700 1 --2- 172.21.2.112:0/1277872832 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3c780921c0 0x7f3c780925c0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f3c70009390 tx=0x7f3c7000bd40).stop 2021-02-10T11:14:19.174 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:19.175 7f3c7d82f700 1 -- 172.21.2.112:0/1277872832 shutdown_connections 2021-02-10T11:14:19.174 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:19.175 7f3c7d82f700 1 --2- 172.21.2.112:0/1277872832 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3c5c0186c0 0x7f3c5c01abc0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:19.174 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:19.175 7f3c7d82f700 1 --2- 172.21.2.112:0/1277872832 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3c781a33a0 0x7f3c78091b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:19.174 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:19.175 7f3c7d82f700 1 --2- 172.21.2.112:0/1277872832 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3c780921c0 0x7f3c780925c0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:19.174 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:19.175 7f3c7d82f700 1 --2- 172.21.2.112:0/1277872832 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3c781a2fa0 0x7f3c780914f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:19.174 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:19.175 7f3c7d82f700 1 -- 172.21.2.112:0/1277872832 >> 172.21.2.112:0/1277872832 conn(0x7f3c780bde70 msgr2=0x7f3c780c3970 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:19.174 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:19.175 7f3c7d82f700 1 -- 172.21.2.112:0/1277872832 shutdown_connections 2021-02-10T11:14:19.175 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:19.175 7f3c7d82f700 1 -- 172.21.2.112:0/1277872832 wait complete. 2021-02-10T11:14:19.181 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.4 2021-02-10T11:14:19.182 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd last-stat-seq osd.5 2021-02-10T11:14:19.608 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:19.592 7ff02ac06700 1 Processor -- start 2021-02-10T11:14:19.608 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:19.592 7ff02ac06700 1 -- start start 2021-02-10T11:14:19.609 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:19.593 7ff02ac06700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7ff0240e5070 0x7ff0241b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:19.609 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:19.593 7ff02ac06700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff0240e5470 0x7ff024080310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:19.609 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:19.593 7ff02ac06700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff02407ed80 con 0x7ff0241b3e40 2021-02-10T11:14:19.609 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:19.593 7ff02ac06700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0240d1790 con 0x7ff0240e5070 2021-02-10T11:14:19.609 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:19.593 7ff02ac06700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0240d19b0 con 0x7ff0240e5470 2021-02-10T11:14:19.609 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:19.594 7ff028a22700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7ff0240e5070 0x7ff0241b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:19.609 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:19.594 7ff028a22700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7ff0240e5070 0x7ff0241b3820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3300/0 says I am v2:172.21.2.112:44724/0 (socket says 172.21.2.112:44724) 2021-02-10T11:14:19.609 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:19.594 7ff028a22700 1 -- 172.21.2.112:0/911114301 learned_addr learned my addr 172.21.2.112:0/911114301 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:14:19.609 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:19.594 7ff029223700 1 --2- 172.21.2.112:0/911114301 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff0240e5470 0x7ff024080310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:19.609 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:19.594 7ff029223700 1 -- 172.21.2.112:0/911114301 >> v2:172.21.2.120:3300/0 conn(0x7ff0240e5070 msgr2=0x7ff0241b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:14:19.609 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:19.594 7ff029223700 1 --2- 172.21.2.112:0/911114301 >> v2:172.21.2.120:3300/0 conn(0x7ff0240e5070 0x7ff0241b3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:19.609 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:19.594 7ff029223700 1 -- 172.21.2.112:0/911114301 >> v1:172.21.2.120:6789/0 conn(0x7ff0241b3e40 legacy=0x7ff02407d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:19.610 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:19.594 7ff029223700 1 -- 172.21.2.112:0/911114301 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0240d1bd0 con 0x7ff0240e5470 2021-02-10T11:14:19.610 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:19.594 7ff0237fe700 1 -- 172.21.2.112:0/911114301 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3547470007 0 0) 0x7ff014000fe0 con 0x7ff0241b3e40 2021-02-10T11:14:19.610 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:19.595 7ff029223700 1 --2- 172.21.2.112:0/911114301 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff0240e5470 0x7ff024080310 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7ff01c006ad0 tx=0x7ff01c009ad0).ready entity=mon.2 client_cookie=6a41608e2b5006b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:19.610 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:19.595 7ff0237fe700 1 -- 172.21.2.112:0/911114301 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff01c0066a0 con 0x7ff0240e5470 2021-02-10T11:14:19.610 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:19.595 7ff0237fe700 1 -- 172.21.2.112:0/911114301 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff01c006410 con 0x7ff0240e5470 2021-02-10T11:14:19.610 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:19.595 7ff0237fe700 1 -- 172.21.2.112:0/911114301 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff01c0024d0 con 0x7ff0240e5470 2021-02-10T11:14:19.610 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:19.595 7ff02ac06700 1 -- 172.21.2.112:0/911114301 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff0240e5470 msgr2=0x7ff024080310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:19.610 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:19.595 7ff02ac06700 1 --2- 172.21.2.112:0/911114301 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff0240e5470 0x7ff024080310 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7ff01c006ad0 tx=0x7ff01c009ad0).stop 2021-02-10T11:14:19.610 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:19.596 7ff02ac06700 1 -- 172.21.2.112:0/911114301 shutdown_connections 2021-02-10T11:14:19.610 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:19.596 7ff02ac06700 1 --2- 172.21.2.112:0/911114301 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff0240e5470 0x7ff024080310 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:19.610 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:19.596 7ff02ac06700 1 --2- 172.21.2.112:0/911114301 >> v2:172.21.2.120:3300/0 conn(0x7ff0240e5070 0x7ff0241b3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:19.610 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:19.596 7ff02ac06700 1 -- 172.21.2.112:0/911114301 >> 172.21.2.112:0/911114301 conn(0x7ff0240ce690 msgr2=0x7ff0240e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:19.611 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:19.596 7ff02ac06700 1 -- 172.21.2.112:0/911114301 shutdown_connections 2021-02-10T11:14:19.611 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:19.596 7ff02ac06700 1 -- 172.21.2.112:0/911114301 wait complete. 2021-02-10T11:14:19.612 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:19.610 7ff02ac06700 1 Processor -- start 2021-02-10T11:14:19.612 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:19.610 7ff02ac06700 1 -- start start 2021-02-10T11:14:19.612 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:19.611 7ff02ac06700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff0240be120 0x7ff0240be720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:19.612 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:19.611 7ff02ac06700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff0240896d0 0x7ff024198b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:19.613 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:19.611 7ff02ac06700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff024199150 0x7ff0240ba680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:19.613 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:19.611 7ff02ac06700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0240baca0 con 0x7ff024199150 2021-02-10T11:14:19.613 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:19.611 7ff02ac06700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0240baec0 con 0x7ff0240896d0 2021-02-10T11:14:19.613 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:19.611 7ff02ac06700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff024089ad0 con 0x7ff0240be120 2021-02-10T11:14:19.613 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:19.611 7ff029223700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff024199150 0x7ff0240ba680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:19.613 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:19.611 7ff029223700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff024199150 0x7ff0240ba680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:42448/0 (socket says 172.21.2.112:42448) 2021-02-10T11:14:19.613 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:19.611 7ff029223700 1 -- 172.21.2.112:0/1496348400 learned_addr learned my addr 172.21.2.112:0/1496348400 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:14:19.613 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:19.611 7ff029223700 1 -- 172.21.2.112:0/1496348400 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff0240be120 msgr2=0x7ff0240be720 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:14:19.613 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:19.611 7ff029223700 1 --2- 172.21.2.112:0/1496348400 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff0240be120 0x7ff0240be720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:19.613 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:19.611 7ff029223700 1 -- 172.21.2.112:0/1496348400 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff0240896d0 msgr2=0x7ff024198b30 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:14:19.613 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:19.611 7ff029223700 1 --2- 172.21.2.112:0/1496348400 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff0240896d0 0x7ff024198b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:19.613 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:19.611 7ff029223700 1 -- 172.21.2.112:0/1496348400 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff024089cf0 con 0x7ff024199150 2021-02-10T11:14:19.613 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:19.611 7ff029223700 1 --2- 172.21.2.112:0/1496348400 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff024199150 0x7ff0240ba680 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7ff01c009670 tx=0x7ff01c00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:19.614 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:19.612 7ff0217fa700 1 -- 172.21.2.112:0/1496348400 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff01c009020 con 0x7ff024199150 2021-02-10T11:14:19.614 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:19.612 7ff0217fa700 1 -- 172.21.2.112:0/1496348400 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff01c00bd80 con 0x7ff024199150 2021-02-10T11:14:19.614 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:19.612 7ff0217fa700 1 -- 172.21.2.112:0/1496348400 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff01c002300 con 0x7ff024199150 2021-02-10T11:14:19.614 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:19.612 7ff02ac06700 1 -- 172.21.2.112:0/1496348400 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff024089cf0 con 0x7ff024199150 2021-02-10T11:14:19.614 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:19.612 7ff02ac06700 1 -- 172.21.2.112:0/1496348400 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0240bf0e0 con 0x7ff024199150 2021-02-10T11:14:19.614 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:19.612 7ff0217fa700 1 -- 172.21.2.112:0/1496348400 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff01c002020 con 0x7ff024199150 2021-02-10T11:14:19.614 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:19.612 7ff0217fa700 1 --2- 172.21.2.112:0/1496348400 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff0080186c0 0x7ff00801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:19.614 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:19.612 7ff0217fa700 1 -- 172.21.2.112:0/1496348400 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7ff01c032590 con 0x7ff024199150 2021-02-10T11:14:19.614 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:19.613 7ff028a22700 1 --2- 172.21.2.112:0/1496348400 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff0080186c0 0x7ff00801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:19.614 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:19.613 7ff028a22700 1 --2- 172.21.2.112:0/1496348400 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff0080186c0 0x7ff00801abc0 crc :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:19.641 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:19.642 7ff02ac06700 1 -- 172.21.2.112:0/1496348400 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff024107180 con 0x7ff024199150 2021-02-10T11:14:19.643 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:19.645 7ff0217fa700 1 -- 172.21.2.112:0/1496348400 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff01c016660 con 0x7ff024199150 2021-02-10T11:14:19.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:19.785 7ff02ac06700 1 -- 172.21.2.112:0/1496348400 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7ff02407ed80 con 0x7ff024199150 2021-02-10T11:14:19.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:19.786 7ff0217fa700 1 -- 172.21.2.112:0/1496348400 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7ff01c016660 con 0x7ff024199150 2021-02-10T11:14:19.785 INFO:teuthology.orchestra.run.gibba012.stdout:55834574855 2021-02-10T11:14:19.786 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:19.787 7ff02ac06700 1 -- 172.21.2.112:0/1496348400 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff0080186c0 msgr2=0x7ff00801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:19.786 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:19.787 7ff02ac06700 1 --2- 172.21.2.112:0/1496348400 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff0080186c0 0x7ff00801abc0 crc :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:19.786 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:19.787 7ff02ac06700 1 -- 172.21.2.112:0/1496348400 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff024199150 msgr2=0x7ff0240ba680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:19.786 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:19.787 7ff02ac06700 1 --2- 172.21.2.112:0/1496348400 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff024199150 0x7ff0240ba680 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7ff01c009670 tx=0x7ff01c00b990).stop 2021-02-10T11:14:19.787 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:19.788 7ff02ac06700 1 -- 172.21.2.112:0/1496348400 shutdown_connections 2021-02-10T11:14:19.787 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:19.788 7ff02ac06700 1 --2- 172.21.2.112:0/1496348400 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff0080186c0 0x7ff00801abc0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:19.787 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:19.788 7ff02ac06700 1 --2- 172.21.2.112:0/1496348400 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff0240896d0 0x7ff024198b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:19.787 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:19.788 7ff02ac06700 1 --2- 172.21.2.112:0/1496348400 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff024199150 0x7ff0240ba680 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:19.787 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:19.788 7ff02ac06700 1 --2- 172.21.2.112:0/1496348400 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff0240be120 0x7ff0240be720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:19.787 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:19.788 7ff02ac06700 1 -- 172.21.2.112:0/1496348400 >> 172.21.2.112:0/1496348400 conn(0x7ff024098740 msgr2=0x7ff0240c3590 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:19.787 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:19.788 7ff02ac06700 1 -- 172.21.2.112:0/1496348400 shutdown_connections 2021-02-10T11:14:19.787 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:19.788 7ff02ac06700 1 -- 172.21.2.112:0/1496348400 wait complete. 2021-02-10T11:14:19.796 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.5 2021-02-10T11:14:19.796 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd last-stat-seq osd.6 2021-02-10T11:14:20.220 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.206 7f300d3f8700 1 Processor -- start 2021-02-10T11:14:20.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.206 7f300d3f8700 1 -- start start 2021-02-10T11:14:20.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.206 7f300d3f8700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f30080e5070 0x7f30081b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:20.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.206 7f300d3f8700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f30081b3e40 0x7f300807d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:20.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.206 7f300d3f8700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f30080e5470 0x7f300807dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:20.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.206 7f300d3f8700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f300807ed80 con 0x7f30081b3e40 2021-02-10T11:14:20.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.206 7f300d3f8700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f300807e2d0 con 0x7f30080e5070 2021-02-10T11:14:20.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.206 7f300d3f8700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f300807e4f0 con 0x7f30080e5470 2021-02-10T11:14:20.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.207 7f300687c700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f30081b3e40 0x7f300807d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:20.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.207 7f300687c700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f30081b3e40 0x7f300807d690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:42454/0 (socket says 172.21.2.112:42454) 2021-02-10T11:14:20.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.207 7f300687c700 1 -- 172.21.2.112:0/1170985416 learned_addr learned my addr 172.21.2.112:0/1170985416 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:14:20.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.207 7f300687c700 1 -- 172.21.2.112:0/1170985416 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f30080e5470 msgr2=0x7f300807dcb0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:14:20.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.207 7f300687c700 1 --2- 172.21.2.112:0/1170985416 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f30080e5470 0x7f300807dcb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:20.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.207 7f300687c700 1 -- 172.21.2.112:0/1170985416 >> v2:172.21.2.120:3300/0 conn(0x7f30080e5070 msgr2=0x7f30081b3820 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:14:20.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.207 7f300687c700 1 --2- 172.21.2.112:0/1170985416 >> v2:172.21.2.120:3300/0 conn(0x7f30080e5070 0x7f30081b3820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:20.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.207 7f300687c700 1 -- 172.21.2.112:0/1170985416 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f300807e710 con 0x7f30081b3e40 2021-02-10T11:14:20.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.207 7f300687c700 1 --2- 172.21.2.112:0/1170985416 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f30081b3e40 0x7f300807d690 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f2ff8006700 tx=0x7f2ff8009b80).ready entity=mon.0 client_cookie=b959990fed3c39f0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:20.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.207 7f300607b700 1 -- 172.21.2.112:0/1170985416 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2ff800c0c0 con 0x7f30081b3e40 2021-02-10T11:14:20.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.207 7f300607b700 1 -- 172.21.2.112:0/1170985416 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ff800c440 con 0x7f30081b3e40 2021-02-10T11:14:20.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.207 7f300607b700 1 -- 172.21.2.112:0/1170985416 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2ff800c920 con 0x7f30081b3e40 2021-02-10T11:14:20.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.207 7f300d3f8700 1 -- 172.21.2.112:0/1170985416 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f30081b3e40 msgr2=0x7f300807d690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:20.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.207 7f300d3f8700 1 --2- 172.21.2.112:0/1170985416 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f30081b3e40 0x7f300807d690 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f2ff8006700 tx=0x7f2ff8009b80).stop 2021-02-10T11:14:20.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.208 7f300d3f8700 1 -- 172.21.2.112:0/1170985416 shutdown_connections 2021-02-10T11:14:20.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.208 7f300d3f8700 1 --2- 172.21.2.112:0/1170985416 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f30080e5470 0x7f300807dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:20.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.208 7f300d3f8700 1 --2- 172.21.2.112:0/1170985416 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f30081b3e40 0x7f300807d690 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:20.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.208 7f300d3f8700 1 --2- 172.21.2.112:0/1170985416 >> v2:172.21.2.120:3300/0 conn(0x7f30080e5070 0x7f30081b3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:20.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.208 7f300d3f8700 1 -- 172.21.2.112:0/1170985416 >> 172.21.2.112:0/1170985416 conn(0x7f30080ce690 msgr2=0x7f30080e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:20.223 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.208 7f300d3f8700 1 -- 172.21.2.112:0/1170985416 shutdown_connections 2021-02-10T11:14:20.223 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.208 7f300d3f8700 1 -- 172.21.2.112:0/1170985416 wait complete. 2021-02-10T11:14:20.224 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.222 7f300d3f8700 1 Processor -- start 2021-02-10T11:14:20.224 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.223 7f300d3f8700 1 -- start start 2021-02-10T11:14:20.224 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.223 7f300d3f8700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3008082140 0x7f30080937b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:20.224 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.223 7f300d3f8700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3008094350 0x7f3008094770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:20.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.223 7f300d3f8700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3008094d90 0x7f30080956e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:20.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.223 7f300d3f8700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3008093dd0 con 0x7f3008094d90 2021-02-10T11:14:20.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.223 7f300d3f8700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3008093ff0 con 0x7f3008094350 2021-02-10T11:14:20.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.223 7f300d3f8700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3008095190 con 0x7f3008082140 2021-02-10T11:14:20.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.223 7f300787e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3008094d90 0x7f30080956e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:20.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.223 7f300787e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3008094d90 0x7f30080956e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:42464/0 (socket says 172.21.2.112:42464) 2021-02-10T11:14:20.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.223 7f300787e700 1 -- 172.21.2.112:0/2214387155 learned_addr learned my addr 172.21.2.112:0/2214387155 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:14:20.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.224 7f300787e700 1 -- 172.21.2.112:0/2214387155 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3008082140 msgr2=0x7f30080937b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:20.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.224 7f300787e700 1 --2- 172.21.2.112:0/2214387155 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3008082140 0x7f30080937b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:20.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.224 7f300787e700 1 -- 172.21.2.112:0/2214387155 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3008094350 msgr2=0x7f3008094770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:20.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.224 7f300787e700 1 --2- 172.21.2.112:0/2214387155 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3008094350 0x7f3008094770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:20.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.224 7f300787e700 1 -- 172.21.2.112:0/2214387155 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30080953b0 con 0x7f3008094d90 2021-02-10T11:14:20.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.224 7f300787e700 1 --2- 172.21.2.112:0/2214387155 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3008094d90 0x7f30080956e0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f3000009960 tx=0x7f300000bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:20.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.224 7f2ffffff700 1 -- 172.21.2.112:0/2214387155 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3000003030 con 0x7f3008094d90 2021-02-10T11:14:20.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.224 7f2ffffff700 1 -- 172.21.2.112:0/2214387155 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3000001310 con 0x7f3008094d90 2021-02-10T11:14:20.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.224 7f2ffffff700 1 -- 172.21.2.112:0/2214387155 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3000001830 con 0x7f3008094d90 2021-02-10T11:14:20.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.224 7f300d3f8700 1 -- 172.21.2.112:0/2214387155 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30080953b0 con 0x7f3008094d90 2021-02-10T11:14:20.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.224 7f300d3f8700 1 -- 172.21.2.112:0/2214387155 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3008095ff0 con 0x7f3008094d90 2021-02-10T11:14:20.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.225 7f2ffffff700 1 -- 172.21.2.112:0/2214387155 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f30000014f0 con 0x7f3008094d90 2021-02-10T11:14:20.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.225 7f2ffffff700 1 --2- 172.21.2.112:0/2214387155 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2fec0186c0 0x7f2fec01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:20.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.225 7f2ffffff700 1 -- 172.21.2.112:0/2214387155 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f3000032970 con 0x7f3008094d90 2021-02-10T11:14:20.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.225 7f300707d700 1 --2- 172.21.2.112:0/2214387155 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2fec0186c0 0x7f2fec01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:20.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.226 7f300707d700 1 --2- 172.21.2.112:0/2214387155 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2fec0186c0 0x7f2fec01abc0 crc :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:20.255 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.256 7f300d3f8700 1 -- 172.21.2.112:0/2214387155 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3008107150 con 0x7f3008094d90 2021-02-10T11:14:20.257 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.259 7f2ffffff700 1 -- 172.21.2.112:0/2214387155 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f300002cfc0 con 0x7f3008094d90 2021-02-10T11:14:20.394 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.396 7f300d3f8700 1 -- 172.21.2.112:0/2214387155 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f300807ed80 con 0x7f3008094d90 2021-02-10T11:14:20.395 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.396 7f2ffffff700 1 -- 172.21.2.112:0/2214387155 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f300002cfc0 con 0x7f3008094d90 2021-02-10T11:14:20.395 INFO:teuthology.orchestra.run.gibba012.stdout:55834574857 2021-02-10T11:14:20.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.398 7f300d3f8700 1 -- 172.21.2.112:0/2214387155 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2fec0186c0 msgr2=0x7f2fec01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:20.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.398 7f300d3f8700 1 --2- 172.21.2.112:0/2214387155 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2fec0186c0 0x7f2fec01abc0 crc :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:20.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.398 7f300d3f8700 1 -- 172.21.2.112:0/2214387155 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3008094d90 msgr2=0x7f30080956e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:20.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.398 7f300d3f8700 1 --2- 172.21.2.112:0/2214387155 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3008094d90 0x7f30080956e0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f3000009960 tx=0x7f300000bcf0).stop 2021-02-10T11:14:20.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.399 7f300d3f8700 1 -- 172.21.2.112:0/2214387155 shutdown_connections 2021-02-10T11:14:20.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.399 7f300d3f8700 1 --2- 172.21.2.112:0/2214387155 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2fec0186c0 0x7f2fec01abc0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:20.398 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.399 7f300d3f8700 1 --2- 172.21.2.112:0/2214387155 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3008094350 0x7f3008094770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:20.398 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.399 7f300d3f8700 1 --2- 172.21.2.112:0/2214387155 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3008094d90 0x7f30080956e0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:20.398 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.399 7f300d3f8700 1 --2- 172.21.2.112:0/2214387155 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3008082140 0x7f30080937b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:20.398 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.399 7f300d3f8700 1 -- 172.21.2.112:0/2214387155 >> 172.21.2.112:0/2214387155 conn(0x7f30080a6c50 msgr2=0x7f30080a6230 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:20.399 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.399 7f300d3f8700 1 -- 172.21.2.112:0/2214387155 shutdown_connections 2021-02-10T11:14:20.399 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.399 7f300d3f8700 1 -- 172.21.2.112:0/2214387155 wait complete. 2021-02-10T11:14:20.405 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574857 for osd.6 2021-02-10T11:14:20.405 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd last-stat-seq osd.7 2021-02-10T11:14:20.830 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.814 7f4630252700 1 Processor -- start 2021-02-10T11:14:20.831 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.814 7f4630252700 1 -- start start 2021-02-10T11:14:20.831 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.815 7f4630252700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f46280e5470 0x7f462807e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:20.831 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.815 7f4630252700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f462807f300 0x7f462807f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:20.831 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.815 7f4630252700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f462807ed80 con 0x7f46280e5070 2021-02-10T11:14:20.831 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.815 7f4630252700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46281b4010 con 0x7f46280e5470 2021-02-10T11:14:20.831 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.815 7f4630252700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46280d1790 con 0x7f462807f300 2021-02-10T11:14:20.831 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.815 7f462e06e700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f46280e5070 0x7f46281b3820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:41930/0 (socket says 172.21.2.112:41930) 2021-02-10T11:14:20.831 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.815 7f462e06e700 1 -- 172.21.2.112:0/1096000708 learned_addr learned my addr 172.21.2.112:0/1096000708 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:14:20.831 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.815 7f462d86d700 1 --2- 172.21.2.112:0/1096000708 >> v2:172.21.2.112:3300/0 conn(0x7f46280e5470 0x7f462807e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:20.831 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.816 7f462d86d700 1 -- 172.21.2.112:0/1096000708 >> v2:172.21.2.120:3300/0 conn(0x7f462807f300 msgr2=0x7f462807f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:14:20.831 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.816 7f462d86d700 1 --2- 172.21.2.112:0/1096000708 >> v2:172.21.2.120:3300/0 conn(0x7f462807f300 0x7f462807f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:20.831 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.816 7f462d86d700 1 -- 172.21.2.112:0/1096000708 >> v1:172.21.2.112:6789/0 conn(0x7f46280e5070 legacy=0x7f46281b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:20.831 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.816 7f462d86d700 1 -- 172.21.2.112:0/1096000708 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46280d19b0 con 0x7f46280e5470 2021-02-10T11:14:20.832 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.816 7f462d86d700 1 --2- 172.21.2.112:0/1096000708 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f46280e5470 0x7f462807e6a0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f461c006650 tx=0x7f461c009a80).ready entity=mon.0 client_cookie=902af444276bc714 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:20.832 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.816 7f462d06c700 1 -- 172.21.2.112:0/1096000708 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f461c00c0c0 con 0x7f46280e5470 2021-02-10T11:14:20.832 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.816 7f462d06c700 1 -- 172.21.2.112:0/1096000708 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f461c00c3c0 con 0x7f46280e5470 2021-02-10T11:14:20.832 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.816 7f462d06c700 1 -- 172.21.2.112:0/1096000708 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f461c00c820 con 0x7f46280e5470 2021-02-10T11:14:20.832 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.816 7f4630252700 1 -- 172.21.2.112:0/1096000708 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f46280e5470 msgr2=0x7f462807e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:20.832 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.816 7f4630252700 1 --2- 172.21.2.112:0/1096000708 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f46280e5470 0x7f462807e6a0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f461c006650 tx=0x7f461c009a80).stop 2021-02-10T11:14:20.832 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.817 7f4630252700 1 -- 172.21.2.112:0/1096000708 shutdown_connections 2021-02-10T11:14:20.832 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.817 7f4630252700 1 --2- 172.21.2.112:0/1096000708 >> v2:172.21.2.120:3300/0 conn(0x7f462807f300 0x7f462807f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:20.832 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.817 7f4630252700 1 --2- 172.21.2.112:0/1096000708 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f46280e5470 0x7f462807e6a0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:20.832 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.817 7f4630252700 1 -- 172.21.2.112:0/1096000708 >> 172.21.2.112:0/1096000708 conn(0x7f46280ce690 msgr2=0x7f46280e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:20.832 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.817 7f4630252700 1 -- 172.21.2.112:0/1096000708 shutdown_connections 2021-02-10T11:14:20.832 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.817 7f4630252700 1 -- 172.21.2.112:0/1096000708 wait complete. 2021-02-10T11:14:20.832 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.832 7f4630252700 1 Processor -- start 2021-02-10T11:14:20.833 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.832 7f4630252700 1 -- start start 2021-02-10T11:14:20.835 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.832 7f4630252700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f46280be580 0x7f46280be980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:20.835 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.832 7f4630252700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4628097ed0 0x7f46280982d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:20.836 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.832 7f4630252700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4628099180 0x7f4628099580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:20.836 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.832 7f4630252700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46280988f0 con 0x7f4628099180 2021-02-10T11:14:20.836 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.832 7f4630252700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46280c1110 con 0x7f46280be580 2021-02-10T11:14:20.836 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.832 7f4630252700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4628099ba0 con 0x7f4628097ed0 2021-02-10T11:14:20.836 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.833 7f462e86f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4628099180 0x7f4628099580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:20.836 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.833 7f462e86f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4628099180 0x7f4628099580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:42476/0 (socket says 172.21.2.112:42476) 2021-02-10T11:14:20.836 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.833 7f462e86f700 1 -- 172.21.2.112:0/3399711197 learned_addr learned my addr 172.21.2.112:0/3399711197 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:14:20.836 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.833 7f462e06e700 1 --2- 172.21.2.112:0/3399711197 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f46280be580 0x7f46280be980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:20.836 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.833 7f462d86d700 1 --2- 172.21.2.112:0/3399711197 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4628097ed0 0x7f46280982d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:20.837 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.833 7f462e86f700 1 -- 172.21.2.112:0/3399711197 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4628097ed0 msgr2=0x7f46280982d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:20.837 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.833 7f462e86f700 1 --2- 172.21.2.112:0/3399711197 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4628097ed0 0x7f46280982d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:20.837 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.833 7f462e86f700 1 -- 172.21.2.112:0/3399711197 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f46280be580 msgr2=0x7f46280be980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:20.837 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.833 7f462e86f700 1 --2- 172.21.2.112:0/3399711197 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f46280be580 0x7f46280be980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:20.837 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.833 7f462e86f700 1 -- 172.21.2.112:0/3399711197 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4628099dc0 con 0x7f4628099180 2021-02-10T11:14:20.837 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.833 7f462d86d700 1 --2- 172.21.2.112:0/3399711197 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4628097ed0 0x7f46280982d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:14:20.837 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.833 7f462e06e700 1 --2- 172.21.2.112:0/3399711197 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f46280be580 0x7f46280be980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:14:20.837 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.833 7f462e86f700 1 --2- 172.21.2.112:0/3399711197 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4628099180 0x7f4628099580 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f4620009980 tx=0x7f462000bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:20.837 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.834 7f461affd700 1 -- 172.21.2.112:0/3399711197 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4620003020 con 0x7f4628099180 2021-02-10T11:14:20.837 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.834 7f461affd700 1 -- 172.21.2.112:0/3399711197 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4620005a80 con 0x7f4628099180 2021-02-10T11:14:20.838 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.834 7f461affd700 1 -- 172.21.2.112:0/3399711197 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4620006030 con 0x7f4628099180 2021-02-10T11:14:20.838 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.834 7f4630252700 1 -- 172.21.2.112:0/3399711197 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4628099dc0 con 0x7f4628099180 2021-02-10T11:14:20.838 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.834 7f4630252700 1 -- 172.21.2.112:0/3399711197 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4628099dc0 con 0x7f4628099180 2021-02-10T11:14:20.838 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.836 7f461affd700 1 -- 172.21.2.112:0/3399711197 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4620005a80 con 0x7f4628099180 2021-02-10T11:14:20.839 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.836 7f461affd700 1 --2- 172.21.2.112:0/3399711197 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f46100186c0 0x7f461001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:20.839 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.836 7f461affd700 1 -- 172.21.2.112:0/3399711197 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f46200328b0 con 0x7f4628099180 2021-02-10T11:14:20.839 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.836 7f462e06e700 1 --2- 172.21.2.112:0/3399711197 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f46100186c0 0x7f461001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:20.840 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.836 7f462e06e700 1 --2- 172.21.2.112:0/3399711197 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f46100186c0 0x7f461001abc0 crc :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:20.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.865 7f4630252700 1 -- 172.21.2.112:0/3399711197 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4628098c50 con 0x7f4628099180 2021-02-10T11:14:20.866 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:20.867 7f461affd700 1 -- 172.21.2.112:0/3399711197 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4620005870 con 0x7f4628099180 2021-02-10T11:14:21.005 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:21.007 7f4630252700 1 -- 172.21.2.112:0/3399711197 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f4628098c50 con 0x7f4628099180 2021-02-10T11:14:21.006 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:21.007 7f461affd700 1 -- 172.21.2.112:0/3399711197 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f4620007f00 con 0x7f4628099180 2021-02-10T11:14:21.006 INFO:teuthology.orchestra.run.gibba012.stdout:55834574856 2021-02-10T11:14:21.008 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:21.009 7f4630252700 1 -- 172.21.2.112:0/3399711197 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f46100186c0 msgr2=0x7f461001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:21.008 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:21.009 7f4630252700 1 --2- 172.21.2.112:0/3399711197 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f46100186c0 0x7f461001abc0 crc :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:21.008 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:21.009 7f4630252700 1 -- 172.21.2.112:0/3399711197 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4628099180 msgr2=0x7f4628099580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:21.008 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:21.009 7f4630252700 1 --2- 172.21.2.112:0/3399711197 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4628099180 0x7f4628099580 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f4620009980 tx=0x7f462000bcd0).stop 2021-02-10T11:14:21.008 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:21.009 7f4630252700 1 -- 172.21.2.112:0/3399711197 shutdown_connections 2021-02-10T11:14:21.009 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:21.009 7f4630252700 1 --2- 172.21.2.112:0/3399711197 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f46100186c0 0x7f461001abc0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:21.009 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:21.009 7f4630252700 1 --2- 172.21.2.112:0/3399711197 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4628099180 0x7f4628099580 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:21.009 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:21.009 7f4630252700 1 --2- 172.21.2.112:0/3399711197 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4628097ed0 0x7f46280982d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:21.009 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:21.009 7f4630252700 1 --2- 172.21.2.112:0/3399711197 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f46280be580 0x7f46280be980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:21.009 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:21.009 7f4630252700 1 -- 172.21.2.112:0/3399711197 >> 172.21.2.112:0/3399711197 conn(0x7f46280bde70 msgr2=0x7f46280c3970 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:21.009 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:21.010 7f4630252700 1 -- 172.21.2.112:0/3399711197 shutdown_connections 2021-02-10T11:14:21.010 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:21.010 7f4630252700 1 -- 172.21.2.112:0/3399711197 wait complete. 2021-02-10T11:14:21.015 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574856 for osd.7 2021-02-10T11:14:21.016 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2021-02-10T11:14:21.016 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph pg dump --format=json 2021-02-10T11:14:21.442 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:21.426 7f04fb04b700 1 Processor -- start 2021-02-10T11:14:21.442 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:21.426 7f04fb04b700 1 -- start start 2021-02-10T11:14:21.442 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:21.427 7f04fb04b700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f04f40e5070 0x7f04f41b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:21.442 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:21.427 7f04fb04b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f04f40e5470 0x7f04f4080310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:21.442 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:21.427 7f04fb04b700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04f407ed80 con 0x7f04f41b3e40 2021-02-10T11:14:21.442 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:21.427 7f04fb04b700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04f40d1790 con 0x7f04f40e5070 2021-02-10T11:14:21.443 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:21.427 7f04fb04b700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04f40d19b0 con 0x7f04f40e5470 2021-02-10T11:14:21.443 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:21.427 7f04f3fff700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7f04f41b3e40 0x7f04f407d690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:35500/0 (socket says 172.21.2.112:35500) 2021-02-10T11:14:21.443 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:21.427 7f04f3fff700 1 -- 172.21.2.112:0/1018296288 learned_addr learned my addr 172.21.2.112:0/1018296288 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:14:21.443 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:21.428 7f04f8e67700 1 --2- 172.21.2.112:0/1018296288 >> v2:172.21.2.120:3300/0 conn(0x7f04f40e5070 0x7f04f41b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:21.443 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:21.428 7f04f9668700 1 --2- 172.21.2.112:0/1018296288 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f04f40e5470 0x7f04f4080310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:21.443 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:21.428 7f04f37fe700 1 -- 172.21.2.112:0/1018296288 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2568497535 0 0) 0x7f04e4001430 con 0x7f04f41b3e40 2021-02-10T11:14:21.443 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:21.428 7f04f37fe700 1 -- 172.21.2.112:0/1018296288 --> v1:172.21.2.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04e0002fe0 con 0x7f04f41b3e40 2021-02-10T11:14:21.443 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:21.428 7f04f8e67700 1 -- 172.21.2.112:0/1018296288 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f04f40e5470 msgr2=0x7f04f4080310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:14:21.443 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:21.428 7f04f8e67700 1 --2- 172.21.2.112:0/1018296288 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f04f40e5470 0x7f04f4080310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:21.443 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:21.428 7f04f8e67700 1 -- 172.21.2.112:0/1018296288 >> v1:172.21.2.120:6789/0 conn(0x7f04f41b3e40 legacy=0x7f04f407d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:21.443 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:21.428 7f04f8e67700 1 -- 172.21.2.112:0/1018296288 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04f40d1bd0 con 0x7f04f40e5070 2021-02-10T11:14:21.443 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:21.428 7f04f9668700 1 --2- 172.21.2.112:0/1018296288 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f04f40e5470 0x7f04f4080310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:14:21.443 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:21.428 7f04f8e67700 1 --2- 172.21.2.112:0/1018296288 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f04f40e5070 0x7f04f41b3820 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f04dc006650 tx=0x7f04dc009a80).ready entity=mon.1 client_cookie=3a2542ac5fe44a4e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:21.444 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:21.429 7f04f37fe700 1 -- 172.21.2.112:0/1018296288 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f04dc006390 con 0x7f04f40e5070 2021-02-10T11:14:21.444 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:21.429 7f04f37fe700 1 -- 172.21.2.112:0/1018296288 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04dc006740 con 0x7f04f40e5070 2021-02-10T11:14:21.444 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:21.429 7f04f37fe700 1 -- 172.21.2.112:0/1018296288 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f04dc006390 con 0x7f04f40e5070 2021-02-10T11:14:21.444 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:21.429 7f04fb04b700 1 -- 172.21.2.112:0/1018296288 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f04f40e5070 msgr2=0x7f04f41b3820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:21.444 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:21.429 7f04fb04b700 1 --2- 172.21.2.112:0/1018296288 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f04f40e5070 0x7f04f41b3820 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f04dc006650 tx=0x7f04dc009a80).stop 2021-02-10T11:14:21.444 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:21.430 7f04fb04b700 1 -- 172.21.2.112:0/1018296288 shutdown_connections 2021-02-10T11:14:21.444 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:21.430 7f04fb04b700 1 --2- 172.21.2.112:0/1018296288 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f04f40e5470 0x7f04f4080310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:21.444 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:21.430 7f04fb04b700 1 --2- 172.21.2.112:0/1018296288 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f04f40e5070 0x7f04f41b3820 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:21.444 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:21.430 7f04fb04b700 1 -- 172.21.2.112:0/1018296288 >> 172.21.2.112:0/1018296288 conn(0x7f04f40ce690 msgr2=0x7f04f40e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:21.444 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:21.430 7f04fb04b700 1 -- 172.21.2.112:0/1018296288 shutdown_connections 2021-02-10T11:14:21.444 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:21.430 7f04fb04b700 1 -- 172.21.2.112:0/1018296288 wait complete. 2021-02-10T11:14:21.446 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:21.444 7f04fb04b700 1 Processor -- start 2021-02-10T11:14:21.446 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:21.444 7f04fb04b700 1 -- start start 2021-02-10T11:14:21.446 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:21.445 7f04fb04b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f04f41918e0 0x7f04f4189c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:21.446 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:21.445 7f04fb04b700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f04f418a2a0 0x7f04f418a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:21.446 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:21.445 7f04fb04b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f04f418acc0 0x7f04f4190070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:21.446 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:21.445 7f04fb04b700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04f41a2f00 con 0x7f04f418acc0 2021-02-10T11:14:21.446 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:21.445 7f04fb04b700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04f4191ce0 con 0x7f04f418a2a0 2021-02-10T11:14:21.446 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:21.445 7f04fb04b700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04f4191f00 con 0x7f04f41918e0 2021-02-10T11:14:21.447 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:21.445 7f04f9668700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f04f418acc0 0x7f04f4190070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:21.447 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:21.445 7f04f9668700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f04f418acc0 0x7f04f4190070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:42488/0 (socket says 172.21.2.112:42488) 2021-02-10T11:14:21.447 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:21.445 7f04f9668700 1 -- 172.21.2.112:0/2208989782 learned_addr learned my addr 172.21.2.112:0/2208989782 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:14:21.447 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:21.445 7f04f9668700 1 -- 172.21.2.112:0/2208989782 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f04f41918e0 msgr2=0x7f04f4189c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:21.447 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:21.445 7f04f9668700 1 --2- 172.21.2.112:0/2208989782 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f04f41918e0 0x7f04f4189c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:21.447 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:21.445 7f04f3fff700 1 --2- 172.21.2.112:0/2208989782 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f04f418a2a0 0x7f04f418a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:21.447 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:21.445 7f04f9668700 1 -- 172.21.2.112:0/2208989782 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f04f418a2a0 msgr2=0x7f04f418a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:21.447 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:21.445 7f04f9668700 1 --2- 172.21.2.112:0/2208989782 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f04f418a2a0 0x7f04f418a6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:21.447 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:21.445 7f04f9668700 1 -- 172.21.2.112:0/2208989782 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04f4190690 con 0x7f04f418acc0 2021-02-10T11:14:21.447 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:21.445 7f04f9668700 1 --2- 172.21.2.112:0/2208989782 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f04f418acc0 0x7f04f4190070 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f04ec00b5e0 tx=0x7f04ec00b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:21.447 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:21.446 7f04f17fa700 1 -- 172.21.2.112:0/2208989782 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f04ec000c10 con 0x7f04f418acc0 2021-02-10T11:14:21.447 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:21.446 7f04f17fa700 1 -- 172.21.2.112:0/2208989782 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04ec00bd70 con 0x7f04f418acc0 2021-02-10T11:14:21.448 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:21.446 7f04f17fa700 1 -- 172.21.2.112:0/2208989782 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f04ec002560 con 0x7f04f418acc0 2021-02-10T11:14:21.448 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:21.446 7f04fb04b700 1 -- 172.21.2.112:0/2208989782 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04f4190690 con 0x7f04f418acc0 2021-02-10T11:14:21.448 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:21.446 7f04fb04b700 1 -- 172.21.2.112:0/2208989782 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04f4190c20 con 0x7f04f418acc0 2021-02-10T11:14:21.448 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:21.446 7f04f17fa700 1 -- 172.21.2.112:0/2208989782 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f04ec005330 con 0x7f04f418acc0 2021-02-10T11:14:21.448 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:21.447 7f04f17fa700 1 --2- 172.21.2.112:0/2208989782 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f04d80186c0 0x7f04d801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:21.448 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:21.447 7f04f17fa700 1 -- 172.21.2.112:0/2208989782 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f04ec032430 con 0x7f04f418acc0 2021-02-10T11:14:21.448 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:21.447 7f04f8e67700 1 --2- 172.21.2.112:0/2208989782 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f04d80186c0 0x7f04d801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:21.448 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:21.448 7f04f8e67700 1 --2- 172.21.2.112:0/2208989782 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f04d80186c0 0x7f04d801abc0 crc :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:21.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:21.477 7f04fb04b700 1 -- 172.21.2.112:0/2208989782 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04f4189750 con 0x7f04f418acc0 2021-02-10T11:14:21.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:21.479 7f04f17fa700 1 -- 172.21.2.112:0/2208989782 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f04ec000bc0 con 0x7f04f418acc0 2021-02-10T11:14:21.612 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:21.613 7f04fb04b700 1 -- 172.21.2.112:0/2208989782 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "pg dump", "target": ["mgr", ""], "format": "json"}) v1 -- 0x7f04f4190c20 con 0x7f04d80186c0 2021-02-10T11:14:21.614 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:21.616 7f04f17fa700 1 -- 172.21.2.112:0/2208989782 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 dumped all) v1 ==== 18+0+77104 (crc 0 0 0) 0x7f04dc000940 con 0x7f04d80186c0 2021-02-10T11:14:21.617 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:14:21.617 INFO:teuthology.orchestra.run.gibba012.stderr:dumped all 2021-02-10T11:14:21.617 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:21.617 7f04fb04b700 1 -- 172.21.2.112:0/2208989782 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f04d80186c0 msgr2=0x7f04d801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:21.617 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:21.617 7f04fb04b700 1 --2- 172.21.2.112:0/2208989782 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f04d80186c0 0x7f04d801abc0 crc :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:21.618 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:21.617 7f04fb04b700 1 -- 172.21.2.112:0/2208989782 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f04f418acc0 msgr2=0x7f04f4190070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:21.618 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:21.617 7f04fb04b700 1 --2- 172.21.2.112:0/2208989782 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f04f418acc0 0x7f04f4190070 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f04ec00b5e0 tx=0x7f04ec00b980).stop 2021-02-10T11:14:21.618 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:21.618 7f04fb04b700 1 -- 172.21.2.112:0/2208989782 shutdown_connections 2021-02-10T11:14:21.618 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:21.618 7f04fb04b700 1 --2- 172.21.2.112:0/2208989782 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f04d80186c0 0x7f04d801abc0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:21.618 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:21.618 7f04fb04b700 1 --2- 172.21.2.112:0/2208989782 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f04f418a2a0 0x7f04f418a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:21.618 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:21.618 7f04fb04b700 1 --2- 172.21.2.112:0/2208989782 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f04f418acc0 0x7f04f4190070 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:21.618 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:21.618 7f04fb04b700 1 --2- 172.21.2.112:0/2208989782 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f04f41918e0 0x7f04f4189c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:21.618 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:21.618 7f04fb04b700 1 -- 172.21.2.112:0/2208989782 >> 172.21.2.112:0/2208989782 conn(0x7f04f41a31d0 msgr2=0x7f04f408b830 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:21.619 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:21.618 7f04fb04b700 1 -- 172.21.2.112:0/2208989782 shutdown_connections 2021-02-10T11:14:21.619 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:21.618 7f04fb04b700 1 -- 172.21.2.112:0/2208989782 wait complete. 2021-02-10T11:14:21.624 INFO:teuthology.orchestra.run.gibba012.stdout:{"pg_ready":true,"pg_map":{"version":37,"stamp":"2021-02-10 11:14:19.765537","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2286,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"kb":754974720,"kb_used":8390336,"kb_used_data":1664,"kb_used_omap":0,"kb_used_meta":8388608,"kb_avail":746584384,"statfs":{"total":773094113280,"available":764502409216,"internally_reserved":8589934592,"allocated":1703936,"data_stored":554356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8589934592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":760,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":33,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001170"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":"13","reported_epoch":"19","state":"active+clean","last_fresh":"2021-02-10 11:14:04.036250","last_change":"2021-02-10 11:13:58.963167","last_active":"2021-02-10 11:14:04.036250","last_peered":"2021-02-10 11:14:04.036250","last_clean":"2021-02-10 11:14:04.036250","last_became_active":"2021-02-10 11:13:58.962972","last_became_peered":"2021-02-10 11:13:58.962972","last_unstale":"2021-02-10 11:14:04.036250","last_undegraded":"2021-02-10 11:14:04.036250","last_fullsized":"2021-02-10 11:14:04.036250","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:13:57.948661","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:13:57.948661","last_clean_scrub_stamp":"2021-02-10 11:13:57.948661","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"20'8","reported_seq":"18","reported_epoch":"20","state":"active+clean","last_fresh":"2021-02-10 11:14:12.018750","last_change":"2021-02-10 11:14:02.981415","last_active":"2021-02-10 11:14:12.018750","last_peered":"2021-02-10 11:14:12.018750","last_clean":"2021-02-10 11:14:12.018750","last_became_active":"2021-02-10 11:14:02.981155","last_became_peered":"2021-02-10 11:14:02.981155","last_unstale":"2021-02-10 11:14:12.018750","last_undegraded":"2021-02-10 11:14:12.018750","last_fullsized":"2021-02-10 11:14:12.018750","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:14:01.967074","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:14:01.967074","last_clean_scrub_stamp":"2021-02-10 11:14:01.967074","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1526,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":"9","reported_epoch":"19","state":"active+clean","last_fresh":"2021-02-10 11:14:03.634114","last_change":"2021-02-10 11:14:03.633742","last_active":"2021-02-10 11:14:03.634114","last_peered":"2021-02-10 11:14:03.634114","last_clean":"2021-02-10 11:14:03.634114","last_became_active":"2021-02-10 11:14:03.633500","last_became_peered":"2021-02-10 11:14:03.633500","last_unstale":"2021-02-10 11:14:03.634114","last_undegraded":"2021-02-10 11:14:03.634114","last_fullsized":"2021-02-10 11:14:03.634114","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:14:02.972497","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:14:02.972497","last_clean_scrub_stamp":"2021-02-10 11:14:02.972497","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":"13","reported_epoch":"19","state":"active+clean","last_fresh":"2021-02-10 11:14:04.036246","last_change":"2021-02-10 11:13:58.964263","last_active":"2021-02-10 11:14:04.036246","last_peered":"2021-02-10 11:14:04.036246","last_clean":"2021-02-10 11:14:04.036246","last_became_active":"2021-02-10 11:13:58.964050","last_became_peered":"2021-02-10 11:13:58.964050","last_unstale":"2021-02-10 11:14:04.036246","last_undegraded":"2021-02-10 11:14:04.036246","last_fullsized":"2021-02-10 11:14:04.036246","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:13:57.948661","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:13:57.948661","last_clean_scrub_stamp":"2021-02-10 11:13:57.948661","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"20'6","reported_seq":"16","reported_epoch":"20","state":"active+clean","last_fresh":"2021-02-10 11:14:12.019059","last_change":"2021-02-10 11:14:02.981415","last_active":"2021-02-10 11:14:12.019059","last_peered":"2021-02-10 11:14:12.019059","last_clean":"2021-02-10 11:14:12.019059","last_became_active":"2021-02-10 11:14:02.981199","last_became_peered":"2021-02-10 11:14:02.981199","last_unstale":"2021-02-10 11:14:12.019059","last_undegraded":"2021-02-10 11:14:12.019059","last_fullsized":"2021-02-10 11:14:12.019059","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:14:01.967074","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:14:01.967074","last_clean_scrub_stamp":"2021-02-10 11:14:01.967074","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":160,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":"9","reported_epoch":"19","state":"active+clean","last_fresh":"2021-02-10 11:14:03.635393","last_change":"2021-02-10 11:14:03.634918","last_active":"2021-02-10 11:14:03.635393","last_peered":"2021-02-10 11:14:03.635393","last_clean":"2021-02-10 11:14:03.635393","last_became_active":"2021-02-10 11:14:03.634656","last_became_peered":"2021-02-10 11:14:03.634656","last_unstale":"2021-02-10 11:14:03.635393","last_undegraded":"2021-02-10 11:14:03.635393","last_fullsized":"2021-02-10 11:14:03.635393","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:14:02.972497","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:14:02.972497","last_clean_scrub_stamp":"2021-02-10 11:14:02.972497","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":"9","reported_epoch":"19","state":"active+clean","last_fresh":"2021-02-10 11:14:03.635153","last_change":"2021-02-10 11:14:03.634696","last_active":"2021-02-10 11:14:03.635153","last_peered":"2021-02-10 11:14:03.635153","last_clean":"2021-02-10 11:14:03.635153","last_became_active":"2021-02-10 11:14:03.634454","last_became_peered":"2021-02-10 11:14:03.634454","last_unstale":"2021-02-10 11:14:03.635153","last_undegraded":"2021-02-10 11:14:03.635153","last_fullsized":"2021-02-10 11:14:03.635153","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:14:02.972497","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:14:02.972497","last_clean_scrub_stamp":"2021-02-10 11:14:02.972497","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"13","reported_epoch":"19","state":"active+clean","last_fresh":"2021-02-10 11:14:03.631573","last_change":"2021-02-10 11:13:58.964235","last_active":"2021-02-10 11:14:03.631573","last_peered":"2021-02-10 11:14:03.631573","last_clean":"2021-02-10 11:14:03.631573","last_became_active":"2021-02-10 11:13:58.963951","last_became_peered":"2021-02-10 11:13:58.963951","last_unstale":"2021-02-10 11:14:03.631573","last_undegraded":"2021-02-10 11:14:03.631573","last_fullsized":"2021-02-10 11:14:03.631573","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:13:57.948661","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:13:57.948661","last_clean_scrub_stamp":"2021-02-10 11:13:57.948661","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"20'4","reported_seq":"14","reported_epoch":"20","state":"active+clean","last_fresh":"2021-02-10 11:14:12.017717","last_change":"2021-02-10 11:14:02.981301","last_active":"2021-02-10 11:14:12.017717","last_peered":"2021-02-10 11:14:12.017717","last_clean":"2021-02-10 11:14:12.017717","last_became_active":"2021-02-10 11:14:02.981054","last_became_peered":"2021-02-10 11:14:02.981054","last_unstale":"2021-02-10 11:14:12.017717","last_undegraded":"2021-02-10 11:14:12.017717","last_fullsized":"2021-02-10 11:14:12.017717","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:14:01.967074","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:14:01.967074","last_clean_scrub_stamp":"2021-02-10 11:14:01.967074","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":"9","reported_epoch":"19","state":"active+clean","last_fresh":"2021-02-10 11:14:03.633661","last_change":"2021-02-10 11:14:03.633144","last_active":"2021-02-10 11:14:03.633661","last_peered":"2021-02-10 11:14:03.633661","last_clean":"2021-02-10 11:14:03.633661","last_became_active":"2021-02-10 11:14:03.632958","last_became_peered":"2021-02-10 11:14:03.632958","last_unstale":"2021-02-10 11:14:03.633661","last_undegraded":"2021-02-10 11:14:03.633661","last_fullsized":"2021-02-10 11:14:03.633661","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:14:02.972497","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:14:02.972497","last_clean_scrub_stamp":"2021-02-10 11:14:02.972497","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":"12","reported_epoch":"19","state":"active+clean","last_fresh":"2021-02-10 11:14:03.632497","last_change":"2021-02-10 11:13:58.962265","last_active":"2021-02-10 11:14:03.632497","last_peered":"2021-02-10 11:14:03.632497","last_clean":"2021-02-10 11:14:03.632497","last_became_active":"2021-02-10 11:13:58.961934","last_became_peered":"2021-02-10 11:13:58.961934","last_unstale":"2021-02-10 11:14:03.632497","last_undegraded":"2021-02-10 11:14:03.632497","last_fullsized":"2021-02-10 11:14:03.632497","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:13:57.948661","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:13:57.948661","last_clean_scrub_stamp":"2021-02-10 11:13:57.948661","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'7","reported_seq":"17","reported_epoch":"20","state":"active+clean","last_fresh":"2021-02-10 11:14:12.018867","last_change":"2021-02-10 11:14:02.981151","last_active":"2021-02-10 11:14:12.018867","last_peered":"2021-02-10 11:14:12.018867","last_clean":"2021-02-10 11:14:12.018867","last_became_active":"2021-02-10 11:14:02.980919","last_became_peered":"2021-02-10 11:14:02.980919","last_unstale":"2021-02-10 11:14:12.018867","last_undegraded":"2021-02-10 11:14:12.018867","last_fullsized":"2021-02-10 11:14:12.018867","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:14:01.967074","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:14:01.967074","last_clean_scrub_stamp":"2021-02-10 11:14:01.967074","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":566,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"13","reported_epoch":"19","state":"active+clean","last_fresh":"2021-02-10 11:14:03.633764","last_change":"2021-02-10 11:13:58.964324","last_active":"2021-02-10 11:14:03.633764","last_peered":"2021-02-10 11:14:03.633764","last_clean":"2021-02-10 11:14:03.633764","last_became_active":"2021-02-10 11:13:58.964097","last_became_peered":"2021-02-10 11:13:58.964097","last_unstale":"2021-02-10 11:14:03.633764","last_undegraded":"2021-02-10 11:14:03.633764","last_fullsized":"2021-02-10 11:14:03.633764","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:13:57.948661","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:13:57.948661","last_clean_scrub_stamp":"2021-02-10 11:13:57.948661","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"20'5","reported_seq":"15","reported_epoch":"20","state":"active+clean","last_fresh":"2021-02-10 11:14:12.018537","last_change":"2021-02-10 11:14:02.980847","last_active":"2021-02-10 11:14:12.018537","last_peered":"2021-02-10 11:14:12.018537","last_clean":"2021-02-10 11:14:12.018537","last_became_active":"2021-02-10 11:14:02.980600","last_became_peered":"2021-02-10 11:14:02.980600","last_unstale":"2021-02-10 11:14:12.018537","last_undegraded":"2021-02-10 11:14:12.018537","last_fullsized":"2021-02-10 11:14:12.018537","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:14:01.967074","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:14:01.967074","last_clean_scrub_stamp":"2021-02-10 11:14:01.967074","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":"9","reported_epoch":"19","state":"active+clean","last_fresh":"2021-02-10 11:14:03.634230","last_change":"2021-02-10 11:14:03.633963","last_active":"2021-02-10 11:14:03.634230","last_peered":"2021-02-10 11:14:03.634230","last_clean":"2021-02-10 11:14:03.634230","last_became_active":"2021-02-10 11:14:03.633731","last_became_peered":"2021-02-10 11:14:03.633731","last_unstale":"2021-02-10 11:14:03.634230","last_undegraded":"2021-02-10 11:14:03.634230","last_fullsized":"2021-02-10 11:14:03.634230","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:14:02.972497","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:14:02.972497","last_clean_scrub_stamp":"2021-02-10 11:14:02.972497","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"13","reported_epoch":"19","state":"active+clean","last_fresh":"2021-02-10 11:14:03.631523","last_change":"2021-02-10 11:13:58.964040","last_active":"2021-02-10 11:14:03.631523","last_peered":"2021-02-10 11:14:03.631523","last_clean":"2021-02-10 11:14:03.631523","last_became_active":"2021-02-10 11:13:58.963814","last_became_peered":"2021-02-10 11:13:58.963814","last_unstale":"2021-02-10 11:14:03.631523","last_undegraded":"2021-02-10 11:14:03.631523","last_fullsized":"2021-02-10 11:14:03.631523","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:13:57.948661","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:13:57.948661","last_clean_scrub_stamp":"2021-02-10 11:13:57.948661","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"20'1","reported_seq":"11","reported_epoch":"20","state":"active+clean","last_fresh":"2021-02-10 11:14:04.637024","last_change":"2021-02-10 11:14:02.980630","last_active":"2021-02-10 11:14:04.637024","last_peered":"2021-02-10 11:14:04.637024","last_clean":"2021-02-10 11:14:04.637024","last_became_active":"2021-02-10 11:14:02.980422","last_became_peered":"2021-02-10 11:14:02.980422","last_unstale":"2021-02-10 11:14:04.637024","last_undegraded":"2021-02-10 11:14:04.637024","last_fullsized":"2021-02-10 11:14:04.637024","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:14:01.967074","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:14:01.967074","last_clean_scrub_stamp":"2021-02-10 11:14:01.967074","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":"9","reported_epoch":"19","state":"active+clean","last_fresh":"2021-02-10 11:14:03.633714","last_change":"2021-02-10 11:14:03.633198","last_active":"2021-02-10 11:14:03.633714","last_peered":"2021-02-10 11:14:03.633714","last_clean":"2021-02-10 11:14:03.633714","last_became_active":"2021-02-10 11:14:03.633030","last_became_peered":"2021-02-10 11:14:03.633030","last_unstale":"2021-02-10 11:14:03.633714","last_undegraded":"2021-02-10 11:14:03.633714","last_fullsized":"2021-02-10 11:14:03.633714","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:14:02.972497","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:14:02.972497","last_clean_scrub_stamp":"2021-02-10 11:14:02.972497","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"10","reported_epoch":"19","state":"active+clean","last_fresh":"2021-02-10 11:14:03.633656","last_change":"2021-02-10 11:14:02.981096","last_active":"2021-02-10 11:14:03.633656","last_peered":"2021-02-10 11:14:03.633656","last_clean":"2021-02-10 11:14:03.633656","last_became_active":"2021-02-10 11:14:02.980847","last_became_peered":"2021-02-10 11:14:02.980847","last_unstale":"2021-02-10 11:14:03.633656","last_undegraded":"2021-02-10 11:14:03.633656","last_fullsized":"2021-02-10 11:14:03.633656","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:14:01.967074","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:14:01.967074","last_clean_scrub_stamp":"2021-02-10 11:14:01.967074","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"13","reported_epoch":"19","state":"active+clean","last_fresh":"2021-02-10 11:14:03.633750","last_change":"2021-02-10 11:13:58.964469","last_active":"2021-02-10 11:14:03.633750","last_peered":"2021-02-10 11:14:03.633750","last_clean":"2021-02-10 11:14:03.633750","last_became_active":"2021-02-10 11:13:58.964040","last_became_peered":"2021-02-10 11:13:58.964040","last_unstale":"2021-02-10 11:14:03.633750","last_undegraded":"2021-02-10 11:14:03.633750","last_fullsized":"2021-02-10 11:14:03.633750","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:13:57.948661","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:13:57.948661","last_clean_scrub_stamp":"2021-02-10 11:13:57.948661","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":"9","reported_epoch":"19","state":"active+clean","last_fresh":"2021-02-10 11:14:03.635206","last_change":"2021-02-10 11:14:03.634745","last_active":"2021-02-10 11:14:03.635206","last_peered":"2021-02-10 11:14:03.635206","last_clean":"2021-02-10 11:14:03.635206","last_became_active":"2021-02-10 11:14:03.634520","last_became_peered":"2021-02-10 11:14:03.634520","last_unstale":"2021-02-10 11:14:03.635206","last_undegraded":"2021-02-10 11:14:03.635206","last_fullsized":"2021-02-10 11:14:03.635206","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:14:02.972497","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:14:02.972497","last_clean_scrub_stamp":"2021-02-10 11:14:02.972497","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"20'8","reported_seq":"18","reported_epoch":"20","state":"active+clean","last_fresh":"2021-02-10 11:14:12.018971","last_change":"2021-02-10 11:14:02.981413","last_active":"2021-02-10 11:14:12.018971","last_peered":"2021-02-10 11:14:12.018971","last_clean":"2021-02-10 11:14:12.018971","last_became_active":"2021-02-10 11:14:02.981149","last_became_peered":"2021-02-10 11:14:02.981149","last_unstale":"2021-02-10 11:14:12.018971","last_undegraded":"2021-02-10 11:14:12.018971","last_fullsized":"2021-02-10 11:14:12.018971","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:14:01.967074","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:14:01.967074","last_clean_scrub_stamp":"2021-02-10 11:14:01.967074","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"13","reported_epoch":"19","state":"active+clean","last_fresh":"2021-02-10 11:14:03.632109","last_change":"2021-02-10 11:13:58.962834","last_active":"2021-02-10 11:14:03.632109","last_peered":"2021-02-10 11:14:03.632109","last_clean":"2021-02-10 11:14:03.632109","last_became_active":"2021-02-10 11:13:58.962533","last_became_peered":"2021-02-10 11:13:58.962533","last_unstale":"2021-02-10 11:14:03.632109","last_undegraded":"2021-02-10 11:14:03.632109","last_fullsized":"2021-02-10 11:14:03.632109","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:13:57.948661","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:13:57.948661","last_clean_scrub_stamp":"2021-02-10 11:13:57.948661","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"9","reported_epoch":"19","state":"active+clean","last_fresh":"2021-02-10 11:14:03.633678","last_change":"2021-02-10 11:14:03.633061","last_active":"2021-02-10 11:14:03.633678","last_peered":"2021-02-10 11:14:03.633678","last_clean":"2021-02-10 11:14:03.633678","last_became_active":"2021-02-10 11:14:03.632829","last_became_peered":"2021-02-10 11:14:03.632829","last_unstale":"2021-02-10 11:14:03.633678","last_undegraded":"2021-02-10 11:14:03.633678","last_fullsized":"2021-02-10 11:14:03.633678","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:14:02.972497","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:14:02.972497","last_clean_scrub_stamp":"2021-02-10 11:14:02.972497","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2286,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574856,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1048800,"kb_used_data":216,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323040,"statfs":{"total":96636764160,"available":95562792960,"internally_reserved":1073741824,"allocated":221184,"data_stored":69449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59199999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66500000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75700000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82199999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86299999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93999999999999995}]}]},{"osd":6,"up_from":13,"seq":55834574857,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1048792,"kb_used_data":208,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323048,"statfs":{"total":96636764160,"available":95562801152,"internally_reserved":1073741824,"allocated":212992,"data_stored":69289,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64500000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81100000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68300000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72299999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.753}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93000000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85499999999999998}]}]},{"osd":5,"up_from":13,"seq":55834574856,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1048788,"kb_used_data":204,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323052,"statfs":{"total":96636764160,"available":95562805248,"internally_reserved":1073741824,"allocated":208896,"data_stored":68757,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67100000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85699999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81699999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61699999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73499999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69499999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70399999999999996}]}]},{"osd":4,"up_from":13,"seq":55834574857,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1048784,"kb_used_data":200,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323056,"statfs":{"total":96636764160,"available":95562809344,"internally_reserved":1073741824,"allocated":204800,"data_stored":68723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55200000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63500000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85699999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95899999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77700000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73499999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80500000000000005}]}]},{"osd":0,"up_from":13,"seq":55834574856,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1048804,"kb_used_data":220,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323036,"statfs":{"total":96636764160,"available":95562788864,"internally_reserved":1073741824,"allocated":225280,"data_stored":70409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68500000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52100000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.016}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2509999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.224}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.135}]}]},{"osd":1,"up_from":13,"seq":55834574855,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1048796,"kb_used_data":212,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323044,"statfs":{"total":96636764160,"available":95562797056,"internally_reserved":1073741824,"allocated":217088,"data_stored":70249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39800000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51900000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54600000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0609999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.161}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.085}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.111}]}]},{"osd":2,"up_from":13,"seq":55834574856,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1048788,"kb_used_data":204,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323052,"statfs":{"total":96636764160,"available":95562805248,"internally_reserved":1073741824,"allocated":208896,"data_stored":68757,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.504}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49099999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65800000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.048}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91100000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.141}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95399999999999996}]}]},{"osd":3,"up_from":13,"seq":55834574857,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1048784,"kb_used_data":200,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323056,"statfs":{"total":96636764160,"available":95562809344,"internally_reserved":1073741824,"allocated":204800,"data_stored":68723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50700000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54700000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1950000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.27}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2330000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.262}]}]}]}} 2021-02-10T11:14:21.627 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph pg dump --format=json 2021-02-10T11:14:22.053 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.038 7fe28633b700 1 Processor -- start 2021-02-10T11:14:22.053 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.038 7fe28633b700 1 -- start start 2021-02-10T11:14:22.053 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.038 7fe28633b700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fe2800e5070 0x7fe2801b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:22.053 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.038 7fe28633b700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fe2800e5470 0x7fe280080310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:22.053 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.038 7fe28633b700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe28007ed80 con 0x7fe2801b3e40 2021-02-10T11:14:22.053 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.038 7fe28633b700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2800d1790 con 0x7fe2800e5470 2021-02-10T11:14:22.053 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.038 7fe28633b700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2800d19b0 con 0x7fe2800e5070 2021-02-10T11:14:22.053 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.039 7fe284958700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fe2800e5470 0x7fe280080310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:22.053 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.039 7fe284958700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fe2800e5470 0x7fe280080310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:42494/0 (socket says 172.21.2.112:42494) 2021-02-10T11:14:22.053 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.039 7fe284958700 1 -- 172.21.2.112:0/2061844956 learned_addr learned my addr 172.21.2.112:0/2061844956 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:14:22.054 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.039 7fe27ffff700 1 --2- 172.21.2.112:0/2061844956 >> v2:172.21.2.120:3300/0 conn(0x7fe2800e5070 0x7fe2801b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:22.054 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.039 7fe284958700 1 -- 172.21.2.112:0/2061844956 >> v2:172.21.2.120:3300/0 conn(0x7fe2800e5070 msgr2=0x7fe2801b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:14:22.054 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.039 7fe284958700 1 --2- 172.21.2.112:0/2061844956 >> v2:172.21.2.120:3300/0 conn(0x7fe2800e5070 0x7fe2801b3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:22.054 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.039 7fe284958700 1 -- 172.21.2.112:0/2061844956 >> v1:172.21.2.120:6789/0 conn(0x7fe2801b3e40 legacy=0x7fe28007d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:22.054 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.039 7fe284958700 1 -- 172.21.2.112:0/2061844956 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2800d1bd0 con 0x7fe2800e5470 2021-02-10T11:14:22.054 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.039 7fe284958700 1 --2- 172.21.2.112:0/2061844956 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe2800e5470 0x7fe280080310 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7fe278006f00 tx=0x7fe278009f10).ready entity=mon.0 client_cookie=314326841a1f63fc server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:22.054 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.040 7fe27effd700 1 -- 172.21.2.112:0/2061844956 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe27800b3b0 con 0x7fe2800e5470 2021-02-10T11:14:22.054 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.040 7fe27effd700 1 -- 172.21.2.112:0/2061844956 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe27800b730 con 0x7fe2800e5470 2021-02-10T11:14:22.054 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.040 7fe27effd700 1 -- 172.21.2.112:0/2061844956 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe27800bc50 con 0x7fe2800e5470 2021-02-10T11:14:22.054 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.040 7fe28633b700 1 -- 172.21.2.112:0/2061844956 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe2800e5470 msgr2=0x7fe280080310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:22.054 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.040 7fe28633b700 1 --2- 172.21.2.112:0/2061844956 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe2800e5470 0x7fe280080310 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7fe278006f00 tx=0x7fe278009f10).stop 2021-02-10T11:14:22.054 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.040 7fe28633b700 1 -- 172.21.2.112:0/2061844956 shutdown_connections 2021-02-10T11:14:22.054 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.040 7fe28633b700 1 --2- 172.21.2.112:0/2061844956 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe2800e5470 0x7fe280080310 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:22.055 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.040 7fe28633b700 1 --2- 172.21.2.112:0/2061844956 >> v2:172.21.2.120:3300/0 conn(0x7fe2800e5070 0x7fe2801b3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:22.055 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.040 7fe28633b700 1 -- 172.21.2.112:0/2061844956 >> 172.21.2.112:0/2061844956 conn(0x7fe2800ce690 msgr2=0x7fe2800e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:22.055 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.041 7fe28633b700 1 -- 172.21.2.112:0/2061844956 shutdown_connections 2021-02-10T11:14:22.055 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.041 7fe28633b700 1 -- 172.21.2.112:0/2061844956 wait complete. 2021-02-10T11:14:22.056 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.055 7fe28633b700 1 Processor -- start 2021-02-10T11:14:22.056 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.055 7fe28633b700 1 -- start start 2021-02-10T11:14:22.056 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.055 7fe28633b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe280191bb0 0x7fe2800a2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:22.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.055 7fe28633b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe2800a29a0 0x7fe2800a2da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:22.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.055 7fe28633b700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe2800a33c0 0x7fe2800a03d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:22.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.055 7fe28633b700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2800adf90 con 0x7fe280191bb0 2021-02-10T11:14:22.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.055 7fe28633b700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2800a1e00 con 0x7fe2800a33c0 2021-02-10T11:14:22.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.055 7fe28633b700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2800a09f0 con 0x7fe2800a29a0 2021-02-10T11:14:22.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.056 7fe27ffff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe280191bb0 0x7fe2800a2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:22.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.056 7fe27ffff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe280191bb0 0x7fe2800a2380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:42498/0 (socket says 172.21.2.112:42498) 2021-02-10T11:14:22.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.056 7fe27ffff700 1 -- 172.21.2.112:0/4260032412 learned_addr learned my addr 172.21.2.112:0/4260032412 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:14:22.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.056 7fe284958700 1 --2- 172.21.2.112:0/4260032412 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe2800a33c0 0x7fe2800a03d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:22.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.056 7fe27f7fe700 1 --2- 172.21.2.112:0/4260032412 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe2800a29a0 0x7fe2800a2da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:22.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.056 7fe27ffff700 1 -- 172.21.2.112:0/4260032412 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe2800a29a0 msgr2=0x7fe2800a2da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:22.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.056 7fe27ffff700 1 --2- 172.21.2.112:0/4260032412 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe2800a29a0 0x7fe2800a2da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:22.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.056 7fe27ffff700 1 -- 172.21.2.112:0/4260032412 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe2800a33c0 msgr2=0x7fe2800a03d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:22.058 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.056 7fe27ffff700 1 --2- 172.21.2.112:0/4260032412 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe2800a33c0 0x7fe2800a03d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:22.058 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.056 7fe27ffff700 1 -- 172.21.2.112:0/4260032412 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2800a0bd0 con 0x7fe280191bb0 2021-02-10T11:14:22.058 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.056 7fe27ffff700 1 --2- 172.21.2.112:0/4260032412 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe280191bb0 0x7fe2800a2380 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7fe270006f60 tx=0x7fe27000b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:22.058 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.056 7fe27cff9700 1 -- 172.21.2.112:0/4260032412 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe27000bd60 con 0x7fe280191bb0 2021-02-10T11:14:22.058 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.057 7fe27cff9700 1 -- 172.21.2.112:0/4260032412 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe270000c90 con 0x7fe280191bb0 2021-02-10T11:14:22.058 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.057 7fe27cff9700 1 -- 172.21.2.112:0/4260032412 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe270002220 con 0x7fe280191bb0 2021-02-10T11:14:22.058 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.057 7fe28633b700 1 -- 172.21.2.112:0/4260032412 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2800a0bd0 con 0x7fe280191bb0 2021-02-10T11:14:22.058 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.057 7fe28633b700 1 -- 172.21.2.112:0/4260032412 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2800a0bd0 con 0x7fe280191bb0 2021-02-10T11:14:22.058 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.057 7fe27cff9700 1 -- 172.21.2.112:0/4260032412 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe2700052e0 con 0x7fe280191bb0 2021-02-10T11:14:22.058 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.057 7fe27cff9700 1 --2- 172.21.2.112:0/4260032412 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe2600186c0 0x7fe26001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:22.058 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.057 7fe27cff9700 1 -- 172.21.2.112:0/4260032412 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fe270032780 con 0x7fe280191bb0 2021-02-10T11:14:22.058 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.058 7fe27f7fe700 1 --2- 172.21.2.112:0/4260032412 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe2600186c0 0x7fe26001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:22.059 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.059 7fe27f7fe700 1 --2- 172.21.2.112:0/4260032412 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe2600186c0 0x7fe26001abc0 crc :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:22.087 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.088 7fe28633b700 1 -- 172.21.2.112:0/4260032412 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2800a1e00 con 0x7fe280191bb0 2021-02-10T11:14:22.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.091 7fe27cff9700 1 -- 172.21.2.112:0/4260032412 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe270006bb0 con 0x7fe280191bb0 2021-02-10T11:14:22.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.223 7fe28633b700 1 -- 172.21.2.112:0/4260032412 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "pg dump", "target": ["mgr", ""], "format": "json"}) v1 -- 0x7fe2800a1e00 con 0x7fe2600186c0 2021-02-10T11:14:22.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.226 7fe27cff9700 1 -- 172.21.2.112:0/4260032412 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 dumped all) v1 ==== 18+0+77104 (crc 0 0 0) 0x7fe274006b60 con 0x7fe2600186c0 2021-02-10T11:14:22.229 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:14:22.230 INFO:teuthology.orchestra.run.gibba012.stderr:dumped all 2021-02-10T11:14:22.230 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.232 7fe28633b700 1 -- 172.21.2.112:0/4260032412 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe2600186c0 msgr2=0x7fe26001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:22.230 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.232 7fe28633b700 1 --2- 172.21.2.112:0/4260032412 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe2600186c0 0x7fe26001abc0 crc :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:22.231 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.232 7fe28633b700 1 -- 172.21.2.112:0/4260032412 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe280191bb0 msgr2=0x7fe2800a2380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:22.231 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.232 7fe28633b700 1 --2- 172.21.2.112:0/4260032412 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe280191bb0 0x7fe2800a2380 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7fe270006f60 tx=0x7fe27000b970).stop 2021-02-10T11:14:22.231 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.232 7fe28633b700 1 -- 172.21.2.112:0/4260032412 shutdown_connections 2021-02-10T11:14:22.231 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.232 7fe28633b700 1 --2- 172.21.2.112:0/4260032412 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe2600186c0 0x7fe26001abc0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:22.231 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.232 7fe28633b700 1 --2- 172.21.2.112:0/4260032412 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe2800a33c0 0x7fe2800a03d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:22.231 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.232 7fe28633b700 1 --2- 172.21.2.112:0/4260032412 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe2800a29a0 0x7fe2800a2da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:22.231 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.232 7fe28633b700 1 --2- 172.21.2.112:0/4260032412 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe280191bb0 0x7fe2800a2380 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:22.231 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.232 7fe28633b700 1 -- 172.21.2.112:0/4260032412 >> 172.21.2.112:0/4260032412 conn(0x7fe2801a31d0 msgr2=0x7fe28008b830 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:22.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.232 7fe28633b700 1 -- 172.21.2.112:0/4260032412 shutdown_connections 2021-02-10T11:14:22.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.233 7fe28633b700 1 -- 172.21.2.112:0/4260032412 wait complete. 2021-02-10T11:14:22.239 INFO:teuthology.orchestra.run.gibba012.stdout:{"pg_ready":true,"pg_map":{"version":38,"stamp":"2021-02-10 11:14:21.766199","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2286,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"kb":754974720,"kb_used":8390336,"kb_used_data":1664,"kb_used_omap":0,"kb_used_meta":8388608,"kb_avail":746584384,"statfs":{"total":773094113280,"available":764502409216,"internally_reserved":8589934592,"allocated":1703936,"data_stored":554356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8589934592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":760,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":33,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001833"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":"13","reported_epoch":"19","state":"active+clean","last_fresh":"2021-02-10 11:14:04.036250","last_change":"2021-02-10 11:13:58.963167","last_active":"2021-02-10 11:14:04.036250","last_peered":"2021-02-10 11:14:04.036250","last_clean":"2021-02-10 11:14:04.036250","last_became_active":"2021-02-10 11:13:58.962972","last_became_peered":"2021-02-10 11:13:58.962972","last_unstale":"2021-02-10 11:14:04.036250","last_undegraded":"2021-02-10 11:14:04.036250","last_fullsized":"2021-02-10 11:14:04.036250","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:13:57.948661","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:13:57.948661","last_clean_scrub_stamp":"2021-02-10 11:13:57.948661","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"20'8","reported_seq":"18","reported_epoch":"20","state":"active+clean","last_fresh":"2021-02-10 11:14:12.018750","last_change":"2021-02-10 11:14:02.981415","last_active":"2021-02-10 11:14:12.018750","last_peered":"2021-02-10 11:14:12.018750","last_clean":"2021-02-10 11:14:12.018750","last_became_active":"2021-02-10 11:14:02.981155","last_became_peered":"2021-02-10 11:14:02.981155","last_unstale":"2021-02-10 11:14:12.018750","last_undegraded":"2021-02-10 11:14:12.018750","last_fullsized":"2021-02-10 11:14:12.018750","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:14:01.967074","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:14:01.967074","last_clean_scrub_stamp":"2021-02-10 11:14:01.967074","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1526,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":"9","reported_epoch":"19","state":"active+clean","last_fresh":"2021-02-10 11:14:03.634114","last_change":"2021-02-10 11:14:03.633742","last_active":"2021-02-10 11:14:03.634114","last_peered":"2021-02-10 11:14:03.634114","last_clean":"2021-02-10 11:14:03.634114","last_became_active":"2021-02-10 11:14:03.633500","last_became_peered":"2021-02-10 11:14:03.633500","last_unstale":"2021-02-10 11:14:03.634114","last_undegraded":"2021-02-10 11:14:03.634114","last_fullsized":"2021-02-10 11:14:03.634114","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:14:02.972497","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:14:02.972497","last_clean_scrub_stamp":"2021-02-10 11:14:02.972497","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":"13","reported_epoch":"19","state":"active+clean","last_fresh":"2021-02-10 11:14:04.036246","last_change":"2021-02-10 11:13:58.964263","last_active":"2021-02-10 11:14:04.036246","last_peered":"2021-02-10 11:14:04.036246","last_clean":"2021-02-10 11:14:04.036246","last_became_active":"2021-02-10 11:13:58.964050","last_became_peered":"2021-02-10 11:13:58.964050","last_unstale":"2021-02-10 11:14:04.036246","last_undegraded":"2021-02-10 11:14:04.036246","last_fullsized":"2021-02-10 11:14:04.036246","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:13:57.948661","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:13:57.948661","last_clean_scrub_stamp":"2021-02-10 11:13:57.948661","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"20'6","reported_seq":"16","reported_epoch":"20","state":"active+clean","last_fresh":"2021-02-10 11:14:12.019059","last_change":"2021-02-10 11:14:02.981415","last_active":"2021-02-10 11:14:12.019059","last_peered":"2021-02-10 11:14:12.019059","last_clean":"2021-02-10 11:14:12.019059","last_became_active":"2021-02-10 11:14:02.981199","last_became_peered":"2021-02-10 11:14:02.981199","last_unstale":"2021-02-10 11:14:12.019059","last_undegraded":"2021-02-10 11:14:12.019059","last_fullsized":"2021-02-10 11:14:12.019059","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:14:01.967074","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:14:01.967074","last_clean_scrub_stamp":"2021-02-10 11:14:01.967074","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":160,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":"9","reported_epoch":"19","state":"active+clean","last_fresh":"2021-02-10 11:14:03.635393","last_change":"2021-02-10 11:14:03.634918","last_active":"2021-02-10 11:14:03.635393","last_peered":"2021-02-10 11:14:03.635393","last_clean":"2021-02-10 11:14:03.635393","last_became_active":"2021-02-10 11:14:03.634656","last_became_peered":"2021-02-10 11:14:03.634656","last_unstale":"2021-02-10 11:14:03.635393","last_undegraded":"2021-02-10 11:14:03.635393","last_fullsized":"2021-02-10 11:14:03.635393","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:14:02.972497","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:14:02.972497","last_clean_scrub_stamp":"2021-02-10 11:14:02.972497","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":"9","reported_epoch":"19","state":"active+clean","last_fresh":"2021-02-10 11:14:03.635153","last_change":"2021-02-10 11:14:03.634696","last_active":"2021-02-10 11:14:03.635153","last_peered":"2021-02-10 11:14:03.635153","last_clean":"2021-02-10 11:14:03.635153","last_became_active":"2021-02-10 11:14:03.634454","last_became_peered":"2021-02-10 11:14:03.634454","last_unstale":"2021-02-10 11:14:03.635153","last_undegraded":"2021-02-10 11:14:03.635153","last_fullsized":"2021-02-10 11:14:03.635153","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:14:02.972497","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:14:02.972497","last_clean_scrub_stamp":"2021-02-10 11:14:02.972497","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"13","reported_epoch":"19","state":"active+clean","last_fresh":"2021-02-10 11:14:03.631573","last_change":"2021-02-10 11:13:58.964235","last_active":"2021-02-10 11:14:03.631573","last_peered":"2021-02-10 11:14:03.631573","last_clean":"2021-02-10 11:14:03.631573","last_became_active":"2021-02-10 11:13:58.963951","last_became_peered":"2021-02-10 11:13:58.963951","last_unstale":"2021-02-10 11:14:03.631573","last_undegraded":"2021-02-10 11:14:03.631573","last_fullsized":"2021-02-10 11:14:03.631573","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:13:57.948661","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:13:57.948661","last_clean_scrub_stamp":"2021-02-10 11:13:57.948661","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"20'4","reported_seq":"14","reported_epoch":"20","state":"active+clean","last_fresh":"2021-02-10 11:14:12.017717","last_change":"2021-02-10 11:14:02.981301","last_active":"2021-02-10 11:14:12.017717","last_peered":"2021-02-10 11:14:12.017717","last_clean":"2021-02-10 11:14:12.017717","last_became_active":"2021-02-10 11:14:02.981054","last_became_peered":"2021-02-10 11:14:02.981054","last_unstale":"2021-02-10 11:14:12.017717","last_undegraded":"2021-02-10 11:14:12.017717","last_fullsized":"2021-02-10 11:14:12.017717","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:14:01.967074","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:14:01.967074","last_clean_scrub_stamp":"2021-02-10 11:14:01.967074","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":"9","reported_epoch":"19","state":"active+clean","last_fresh":"2021-02-10 11:14:03.633661","last_change":"2021-02-10 11:14:03.633144","last_active":"2021-02-10 11:14:03.633661","last_peered":"2021-02-10 11:14:03.633661","last_clean":"2021-02-10 11:14:03.633661","last_became_active":"2021-02-10 11:14:03.632958","last_became_peered":"2021-02-10 11:14:03.632958","last_unstale":"2021-02-10 11:14:03.633661","last_undegraded":"2021-02-10 11:14:03.633661","last_fullsized":"2021-02-10 11:14:03.633661","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:14:02.972497","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:14:02.972497","last_clean_scrub_stamp":"2021-02-10 11:14:02.972497","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":"12","reported_epoch":"19","state":"active+clean","last_fresh":"2021-02-10 11:14:03.632497","last_change":"2021-02-10 11:13:58.962265","last_active":"2021-02-10 11:14:03.632497","last_peered":"2021-02-10 11:14:03.632497","last_clean":"2021-02-10 11:14:03.632497","last_became_active":"2021-02-10 11:13:58.961934","last_became_peered":"2021-02-10 11:13:58.961934","last_unstale":"2021-02-10 11:14:03.632497","last_undegraded":"2021-02-10 11:14:03.632497","last_fullsized":"2021-02-10 11:14:03.632497","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:13:57.948661","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:13:57.948661","last_clean_scrub_stamp":"2021-02-10 11:13:57.948661","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'7","reported_seq":"17","reported_epoch":"20","state":"active+clean","last_fresh":"2021-02-10 11:14:12.018867","last_change":"2021-02-10 11:14:02.981151","last_active":"2021-02-10 11:14:12.018867","last_peered":"2021-02-10 11:14:12.018867","last_clean":"2021-02-10 11:14:12.018867","last_became_active":"2021-02-10 11:14:02.980919","last_became_peered":"2021-02-10 11:14:02.980919","last_unstale":"2021-02-10 11:14:12.018867","last_undegraded":"2021-02-10 11:14:12.018867","last_fullsized":"2021-02-10 11:14:12.018867","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:14:01.967074","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:14:01.967074","last_clean_scrub_stamp":"2021-02-10 11:14:01.967074","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":566,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"13","reported_epoch":"19","state":"active+clean","last_fresh":"2021-02-10 11:14:03.633764","last_change":"2021-02-10 11:13:58.964324","last_active":"2021-02-10 11:14:03.633764","last_peered":"2021-02-10 11:14:03.633764","last_clean":"2021-02-10 11:14:03.633764","last_became_active":"2021-02-10 11:13:58.964097","last_became_peered":"2021-02-10 11:13:58.964097","last_unstale":"2021-02-10 11:14:03.633764","last_undegraded":"2021-02-10 11:14:03.633764","last_fullsized":"2021-02-10 11:14:03.633764","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:13:57.948661","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:13:57.948661","last_clean_scrub_stamp":"2021-02-10 11:13:57.948661","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"20'5","reported_seq":"15","reported_epoch":"20","state":"active+clean","last_fresh":"2021-02-10 11:14:12.018537","last_change":"2021-02-10 11:14:02.980847","last_active":"2021-02-10 11:14:12.018537","last_peered":"2021-02-10 11:14:12.018537","last_clean":"2021-02-10 11:14:12.018537","last_became_active":"2021-02-10 11:14:02.980600","last_became_peered":"2021-02-10 11:14:02.980600","last_unstale":"2021-02-10 11:14:12.018537","last_undegraded":"2021-02-10 11:14:12.018537","last_fullsized":"2021-02-10 11:14:12.018537","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:14:01.967074","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:14:01.967074","last_clean_scrub_stamp":"2021-02-10 11:14:01.967074","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":"9","reported_epoch":"19","state":"active+clean","last_fresh":"2021-02-10 11:14:03.634230","last_change":"2021-02-10 11:14:03.633963","last_active":"2021-02-10 11:14:03.634230","last_peered":"2021-02-10 11:14:03.634230","last_clean":"2021-02-10 11:14:03.634230","last_became_active":"2021-02-10 11:14:03.633731","last_became_peered":"2021-02-10 11:14:03.633731","last_unstale":"2021-02-10 11:14:03.634230","last_undegraded":"2021-02-10 11:14:03.634230","last_fullsized":"2021-02-10 11:14:03.634230","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:14:02.972497","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:14:02.972497","last_clean_scrub_stamp":"2021-02-10 11:14:02.972497","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"13","reported_epoch":"19","state":"active+clean","last_fresh":"2021-02-10 11:14:03.631523","last_change":"2021-02-10 11:13:58.964040","last_active":"2021-02-10 11:14:03.631523","last_peered":"2021-02-10 11:14:03.631523","last_clean":"2021-02-10 11:14:03.631523","last_became_active":"2021-02-10 11:13:58.963814","last_became_peered":"2021-02-10 11:13:58.963814","last_unstale":"2021-02-10 11:14:03.631523","last_undegraded":"2021-02-10 11:14:03.631523","last_fullsized":"2021-02-10 11:14:03.631523","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:13:57.948661","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:13:57.948661","last_clean_scrub_stamp":"2021-02-10 11:13:57.948661","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"20'1","reported_seq":"11","reported_epoch":"20","state":"active+clean","last_fresh":"2021-02-10 11:14:04.637024","last_change":"2021-02-10 11:14:02.980630","last_active":"2021-02-10 11:14:04.637024","last_peered":"2021-02-10 11:14:04.637024","last_clean":"2021-02-10 11:14:04.637024","last_became_active":"2021-02-10 11:14:02.980422","last_became_peered":"2021-02-10 11:14:02.980422","last_unstale":"2021-02-10 11:14:04.637024","last_undegraded":"2021-02-10 11:14:04.637024","last_fullsized":"2021-02-10 11:14:04.637024","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:14:01.967074","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:14:01.967074","last_clean_scrub_stamp":"2021-02-10 11:14:01.967074","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":"9","reported_epoch":"19","state":"active+clean","last_fresh":"2021-02-10 11:14:03.633714","last_change":"2021-02-10 11:14:03.633198","last_active":"2021-02-10 11:14:03.633714","last_peered":"2021-02-10 11:14:03.633714","last_clean":"2021-02-10 11:14:03.633714","last_became_active":"2021-02-10 11:14:03.633030","last_became_peered":"2021-02-10 11:14:03.633030","last_unstale":"2021-02-10 11:14:03.633714","last_undegraded":"2021-02-10 11:14:03.633714","last_fullsized":"2021-02-10 11:14:03.633714","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:14:02.972497","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:14:02.972497","last_clean_scrub_stamp":"2021-02-10 11:14:02.972497","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"10","reported_epoch":"19","state":"active+clean","last_fresh":"2021-02-10 11:14:03.633656","last_change":"2021-02-10 11:14:02.981096","last_active":"2021-02-10 11:14:03.633656","last_peered":"2021-02-10 11:14:03.633656","last_clean":"2021-02-10 11:14:03.633656","last_became_active":"2021-02-10 11:14:02.980847","last_became_peered":"2021-02-10 11:14:02.980847","last_unstale":"2021-02-10 11:14:03.633656","last_undegraded":"2021-02-10 11:14:03.633656","last_fullsized":"2021-02-10 11:14:03.633656","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:14:01.967074","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:14:01.967074","last_clean_scrub_stamp":"2021-02-10 11:14:01.967074","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"13","reported_epoch":"19","state":"active+clean","last_fresh":"2021-02-10 11:14:03.633750","last_change":"2021-02-10 11:13:58.964469","last_active":"2021-02-10 11:14:03.633750","last_peered":"2021-02-10 11:14:03.633750","last_clean":"2021-02-10 11:14:03.633750","last_became_active":"2021-02-10 11:13:58.964040","last_became_peered":"2021-02-10 11:13:58.964040","last_unstale":"2021-02-10 11:14:03.633750","last_undegraded":"2021-02-10 11:14:03.633750","last_fullsized":"2021-02-10 11:14:03.633750","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:13:57.948661","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:13:57.948661","last_clean_scrub_stamp":"2021-02-10 11:13:57.948661","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":"9","reported_epoch":"19","state":"active+clean","last_fresh":"2021-02-10 11:14:03.635206","last_change":"2021-02-10 11:14:03.634745","last_active":"2021-02-10 11:14:03.635206","last_peered":"2021-02-10 11:14:03.635206","last_clean":"2021-02-10 11:14:03.635206","last_became_active":"2021-02-10 11:14:03.634520","last_became_peered":"2021-02-10 11:14:03.634520","last_unstale":"2021-02-10 11:14:03.635206","last_undegraded":"2021-02-10 11:14:03.635206","last_fullsized":"2021-02-10 11:14:03.635206","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:14:02.972497","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:14:02.972497","last_clean_scrub_stamp":"2021-02-10 11:14:02.972497","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"20'8","reported_seq":"18","reported_epoch":"20","state":"active+clean","last_fresh":"2021-02-10 11:14:12.018971","last_change":"2021-02-10 11:14:02.981413","last_active":"2021-02-10 11:14:12.018971","last_peered":"2021-02-10 11:14:12.018971","last_clean":"2021-02-10 11:14:12.018971","last_became_active":"2021-02-10 11:14:02.981149","last_became_peered":"2021-02-10 11:14:02.981149","last_unstale":"2021-02-10 11:14:12.018971","last_undegraded":"2021-02-10 11:14:12.018971","last_fullsized":"2021-02-10 11:14:12.018971","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:14:01.967074","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:14:01.967074","last_clean_scrub_stamp":"2021-02-10 11:14:01.967074","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"13","reported_epoch":"19","state":"active+clean","last_fresh":"2021-02-10 11:14:03.632109","last_change":"2021-02-10 11:13:58.962834","last_active":"2021-02-10 11:14:03.632109","last_peered":"2021-02-10 11:14:03.632109","last_clean":"2021-02-10 11:14:03.632109","last_became_active":"2021-02-10 11:13:58.962533","last_became_peered":"2021-02-10 11:13:58.962533","last_unstale":"2021-02-10 11:14:03.632109","last_undegraded":"2021-02-10 11:14:03.632109","last_fullsized":"2021-02-10 11:14:03.632109","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:13:57.948661","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:13:57.948661","last_clean_scrub_stamp":"2021-02-10 11:13:57.948661","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"9","reported_epoch":"19","state":"active+clean","last_fresh":"2021-02-10 11:14:03.633678","last_change":"2021-02-10 11:14:03.633061","last_active":"2021-02-10 11:14:03.633678","last_peered":"2021-02-10 11:14:03.633678","last_clean":"2021-02-10 11:14:03.633678","last_became_active":"2021-02-10 11:14:03.632829","last_became_peered":"2021-02-10 11:14:03.632829","last_unstale":"2021-02-10 11:14:03.633678","last_undegraded":"2021-02-10 11:14:03.633678","last_fullsized":"2021-02-10 11:14:03.633678","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:14:02.972497","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:14:02.972497","last_clean_scrub_stamp":"2021-02-10 11:14:02.972497","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2286,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574856,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1048800,"kb_used_data":216,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323040,"statfs":{"total":96636764160,"available":95562792960,"internally_reserved":1073741824,"allocated":221184,"data_stored":69449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59199999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66500000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75700000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82199999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86299999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93999999999999995}]}]},{"osd":6,"up_from":13,"seq":55834574857,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1048792,"kb_used_data":208,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323048,"statfs":{"total":96636764160,"available":95562801152,"internally_reserved":1073741824,"allocated":212992,"data_stored":69289,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64500000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81100000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68300000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72299999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.753}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93000000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85499999999999998}]}]},{"osd":5,"up_from":13,"seq":55834574856,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1048788,"kb_used_data":204,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323052,"statfs":{"total":96636764160,"available":95562805248,"internally_reserved":1073741824,"allocated":208896,"data_stored":68757,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67100000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85699999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81699999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61699999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73499999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69499999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70399999999999996}]}]},{"osd":4,"up_from":13,"seq":55834574857,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1048784,"kb_used_data":200,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323056,"statfs":{"total":96636764160,"available":95562809344,"internally_reserved":1073741824,"allocated":204800,"data_stored":68723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55200000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63500000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85699999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95899999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77700000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73499999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80500000000000005}]}]},{"osd":0,"up_from":13,"seq":55834574856,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1048804,"kb_used_data":220,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323036,"statfs":{"total":96636764160,"available":95562788864,"internally_reserved":1073741824,"allocated":225280,"data_stored":70409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68500000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52100000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.016}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2509999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.224}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.135}]}]},{"osd":1,"up_from":13,"seq":55834574855,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1048796,"kb_used_data":212,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323044,"statfs":{"total":96636764160,"available":95562797056,"internally_reserved":1073741824,"allocated":217088,"data_stored":70249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39800000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51900000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54600000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0609999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.161}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.085}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.111}]}]},{"osd":2,"up_from":13,"seq":55834574856,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1048788,"kb_used_data":204,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323052,"statfs":{"total":96636764160,"available":95562805248,"internally_reserved":1073741824,"allocated":208896,"data_stored":68757,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.504}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49099999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65800000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.048}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91100000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.141}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95399999999999996}]}]},{"osd":3,"up_from":13,"seq":55834574857,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1048784,"kb_used_data":200,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323056,"statfs":{"total":96636764160,"available":95562809344,"internally_reserved":1073741824,"allocated":204800,"data_stored":68723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50700000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54700000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1950000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.27}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2330000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.262}]}]}]}} 2021-02-10T11:14:22.242 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph status --format=json-pretty 2021-02-10T11:14:22.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.653 7efd78006700 1 Processor -- start 2021-02-10T11:14:22.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.653 7efd78006700 1 -- start start 2021-02-10T11:14:22.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.653 7efd78006700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efd701b0470 0x7efd701b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:22.669 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.654 7efd78006700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7efd700ef610 0x7efd700efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:22.669 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.654 7efd78006700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7efd700f0030 0x7efd700f0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:22.669 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.654 7efd78006700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd700e5ea0 con 0x7efd700ef610 2021-02-10T11:14:22.669 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.654 7efd78006700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd700f0b50 con 0x7efd700f0030 2021-02-10T11:14:22.669 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.654 7efd78006700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd700e52e0 con 0x7efd701b0470 2021-02-10T11:14:22.669 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.654 7efd75621700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7efd700ef610 0x7efd700efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:22.669 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.654 7efd75621700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7efd700ef610 0x7efd700efa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:42504/0 (socket says 172.21.2.112:42504) 2021-02-10T11:14:22.669 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.654 7efd75621700 1 -- 172.21.2.112:0/2704250767 learned_addr learned my addr 172.21.2.112:0/2704250767 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:14:22.669 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.654 7efd76623700 1 --2- 172.21.2.112:0/2704250767 >> v2:172.21.2.120:3300/0 conn(0x7efd700f0030 0x7efd700f0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:22.669 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.654 7efd75e22700 1 --2- 172.21.2.112:0/2704250767 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efd701b0470 0x7efd701b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:22.669 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.654 7efd75621700 1 -- 172.21.2.112:0/2704250767 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efd701b0470 msgr2=0x7efd701b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:14:22.669 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.654 7efd75621700 1 --2- 172.21.2.112:0/2704250767 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efd701b0470 0x7efd701b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:22.669 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.654 7efd75621700 1 -- 172.21.2.112:0/2704250767 >> v2:172.21.2.120:3300/0 conn(0x7efd700f0030 msgr2=0x7efd700f0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:14:22.670 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.654 7efd75621700 1 --2- 172.21.2.112:0/2704250767 >> v2:172.21.2.120:3300/0 conn(0x7efd700f0030 0x7efd700f0430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:22.670 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.654 7efd75621700 1 -- 172.21.2.112:0/2704250767 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd700f10d0 con 0x7efd700ef610 2021-02-10T11:14:22.670 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.654 7efd76623700 1 --2- 172.21.2.112:0/2704250767 >> v2:172.21.2.120:3300/0 conn(0x7efd700f0030 0x7efd700f0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:14:22.670 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.654 7efd75e22700 1 --2- 172.21.2.112:0/2704250767 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efd701b0470 0x7efd701b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:14:22.670 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.654 7efd75621700 1 --2- 172.21.2.112:0/2704250767 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efd700ef610 0x7efd700efa10 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7efd60006700 tx=0x7efd60009b80).ready entity=mon.0 client_cookie=6f5f7336898debbd server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:22.670 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.654 7efd74e20700 1 -- 172.21.2.112:0/2704250767 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efd60006340 con 0x7efd700ef610 2021-02-10T11:14:22.670 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.654 7efd74e20700 1 -- 172.21.2.112:0/2704250767 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd6000c0c0 con 0x7efd700ef610 2021-02-10T11:14:22.670 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.654 7efd74e20700 1 -- 172.21.2.112:0/2704250767 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efd6000c5e0 con 0x7efd700ef610 2021-02-10T11:14:22.670 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.655 7efd78006700 1 -- 172.21.2.112:0/2704250767 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efd700ef610 msgr2=0x7efd700efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:22.670 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.655 7efd78006700 1 --2- 172.21.2.112:0/2704250767 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efd700ef610 0x7efd700efa10 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7efd60006700 tx=0x7efd60009b80).stop 2021-02-10T11:14:22.670 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.656 7efd78006700 1 -- 172.21.2.112:0/2704250767 shutdown_connections 2021-02-10T11:14:22.670 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.656 7efd78006700 1 --2- 172.21.2.112:0/2704250767 >> v2:172.21.2.120:3300/0 conn(0x7efd700f0030 0x7efd700f0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:22.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.656 7efd78006700 1 --2- 172.21.2.112:0/2704250767 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efd700ef610 0x7efd700efa10 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:22.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.656 7efd78006700 1 --2- 172.21.2.112:0/2704250767 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efd701b0470 0x7efd701b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:22.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.656 7efd78006700 1 -- 172.21.2.112:0/2704250767 >> 172.21.2.112:0/2704250767 conn(0x7efd700ce3e0 msgr2=0x7efd700e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:22.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.656 7efd78006700 1 -- 172.21.2.112:0/2704250767 shutdown_connections 2021-02-10T11:14:22.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.656 7efd78006700 1 -- 172.21.2.112:0/2704250767 wait complete. 2021-02-10T11:14:22.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.670 7efd78006700 1 Processor -- start 2021-02-10T11:14:22.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.670 7efd78006700 1 -- start start 2021-02-10T11:14:22.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.671 7efd78006700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7efd70094240 0x7efd70189e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:22.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.671 7efd78006700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efd7018a450 0x7efd700b18b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:22.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.671 7efd78006700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efd700b1ed0 0x7efd7018b090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:22.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.671 7efd78006700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd7018a850 con 0x7efd7018a450 2021-02-10T11:14:22.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.671 7efd78006700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd70094640 con 0x7efd70094240 2021-02-10T11:14:22.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.671 7efd78006700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd70094ab0 con 0x7efd700b1ed0 2021-02-10T11:14:22.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.671 7efd75e22700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7efd70094240 0x7efd70189e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:22.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.671 7efd75e22700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7efd70094240 0x7efd70189e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3300/0 says I am v2:172.21.2.112:44788/0 (socket says 172.21.2.112:44788) 2021-02-10T11:14:22.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.671 7efd75e22700 1 -- 172.21.2.112:0/1109661115 learned_addr learned my addr 172.21.2.112:0/1109661115 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:14:22.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.671 7efd75621700 1 --2- 172.21.2.112:0/1109661115 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efd7018a450 0x7efd700b18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:22.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.671 7efd76623700 1 --2- 172.21.2.112:0/1109661115 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efd700b1ed0 0x7efd7018b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:22.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.672 7efd75e22700 1 -- 172.21.2.112:0/1109661115 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efd700b1ed0 msgr2=0x7efd7018b090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:22.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.672 7efd75e22700 1 --2- 172.21.2.112:0/1109661115 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efd700b1ed0 0x7efd7018b090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:22.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.672 7efd75e22700 1 -- 172.21.2.112:0/1109661115 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efd7018a450 msgr2=0x7efd700b18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:22.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.672 7efd75e22700 1 --2- 172.21.2.112:0/1109661115 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efd7018a450 0x7efd700b18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:22.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.672 7efd75e22700 1 -- 172.21.2.112:0/1109661115 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd701b4ba0 con 0x7efd70094240 2021-02-10T11:14:22.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.672 7efd75e22700 1 --2- 172.21.2.112:0/1109661115 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7efd70094240 0x7efd70189e30 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7efd6c006f20 tx=0x7efd6c00b990).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:22.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.672 7efd667fc700 1 -- 172.21.2.112:0/1109661115 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efd6c006bb0 con 0x7efd70094240 2021-02-10T11:14:22.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.672 7efd667fc700 1 -- 172.21.2.112:0/1109661115 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd6c00bd80 con 0x7efd70094240 2021-02-10T11:14:22.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.672 7efd667fc700 1 -- 172.21.2.112:0/1109661115 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efd6c002220 con 0x7efd70094240 2021-02-10T11:14:22.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.672 7efd78006700 1 -- 172.21.2.112:0/1109661115 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd701b4ba0 con 0x7efd70094240 2021-02-10T11:14:22.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.672 7efd78006700 1 -- 172.21.2.112:0/1109661115 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd701b4ba0 con 0x7efd70094240 2021-02-10T11:14:22.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.673 7efd667fc700 1 -- 172.21.2.112:0/1109661115 <== mon.1 v2:172.21.2.120:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7efd6c000c90 con 0x7efd70094240 2021-02-10T11:14:22.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.673 7efd667fc700 1 --2- 172.21.2.112:0/1109661115 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7efd580186c0 0x7efd5801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:22.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.673 7efd667fc700 1 -- 172.21.2.112:0/1109661115 <== mon.1 v2:172.21.2.120:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7efd6c0325b0 con 0x7efd70094240 2021-02-10T11:14:22.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.673 7efd75621700 1 --2- 172.21.2.112:0/1109661115 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7efd580186c0 0x7efd5801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:22.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.674 7efd75621700 1 --2- 172.21.2.112:0/1109661115 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7efd580186c0 0x7efd5801abc0 crc :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:22.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.703 7efd78006700 1 -- 172.21.2.112:0/1109661115 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd70105b50 con 0x7efd70094240 2021-02-10T11:14:22.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.706 7efd667fc700 1 -- 172.21.2.112:0/1109661115 <== mon.1 v2:172.21.2.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7efd6c016050 con 0x7efd70094240 2021-02-10T11:14:22.836 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.838 7efd78006700 1 -- 172.21.2.112:0/1109661115 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7efd70105b50 con 0x7efd70094240 2021-02-10T11:14:22.843 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.841 7efd667fc700 1 -- 172.21.2.112:0/1109661115 <== mon.1 v2:172.21.2.120:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+211028 (secure 0 0 0) 0x7efd6c016050 con 0x7efd70094240 2021-02-10T11:14:22.843 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.843 7efd78006700 1 -- 172.21.2.112:0/1109661115 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7efd580186c0 msgr2=0x7efd5801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:22.843 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.843 7efd78006700 1 --2- 172.21.2.112:0/1109661115 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7efd580186c0 0x7efd5801abc0 crc :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:22.843 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.843 7efd78006700 1 -- 172.21.2.112:0/1109661115 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7efd70094240 msgr2=0x7efd70189e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:22.843 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.843 7efd78006700 1 --2- 172.21.2.112:0/1109661115 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7efd70094240 0x7efd70189e30 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7efd6c006f20 tx=0x7efd6c00b990).stop 2021-02-10T11:14:22.843 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.843 7efd78006700 1 -- 172.21.2.112:0/1109661115 shutdown_connections 2021-02-10T11:14:22.843 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.843 7efd78006700 1 --2- 172.21.2.112:0/1109661115 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7efd580186c0 0x7efd5801abc0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:22.843 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.843 7efd78006700 1 --2- 172.21.2.112:0/1109661115 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efd700b1ed0 0x7efd7018b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:22.843 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.843 7efd78006700 1 --2- 172.21.2.112:0/1109661115 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efd7018a450 0x7efd700b18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:22.843 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.843 7efd78006700 1 --2- 172.21.2.112:0/1109661115 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7efd70094240 0x7efd70189e30 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:22.844 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.843 7efd78006700 1 -- 172.21.2.112:0/1109661115 >> 172.21.2.112:0/1109661115 conn(0x7efd7019f800 msgr2=0x7efd70105d80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:22.844 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.843 7efd78006700 1 -- 172.21.2.112:0/1109661115 shutdown_connections 2021-02-10T11:14:22.844 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:22.843 7efd78006700 1 -- 172.21.2.112:0/1109661115 wait complete. 2021-02-10T11:14:22.844 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:14:22.844 INFO:teuthology.orchestra.run.gibba012.stdout:{ 2021-02-10T11:14:22.844 INFO:teuthology.orchestra.run.gibba012.stdout: "fsid": "f14f004e-0699-4584-ad81-3754258493de", 2021-02-10T11:14:22.844 INFO:teuthology.orchestra.run.gibba012.stdout: "health": { 2021-02-10T11:14:22.844 INFO:teuthology.orchestra.run.gibba012.stdout: "checks": {}, 2021-02-10T11:14:22.844 INFO:teuthology.orchestra.run.gibba012.stdout: "status": "HEALTH_OK" 2021-02-10T11:14:22.845 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.845 INFO:teuthology.orchestra.run.gibba012.stdout: "election_epoch": 4, 2021-02-10T11:14:22.845 INFO:teuthology.orchestra.run.gibba012.stdout: "quorum": [ 2021-02-10T11:14:22.845 INFO:teuthology.orchestra.run.gibba012.stdout: 0, 2021-02-10T11:14:22.845 INFO:teuthology.orchestra.run.gibba012.stdout: 1, 2021-02-10T11:14:22.845 INFO:teuthology.orchestra.run.gibba012.stdout: 2 2021-02-10T11:14:22.845 INFO:teuthology.orchestra.run.gibba012.stdout: ], 2021-02-10T11:14:22.845 INFO:teuthology.orchestra.run.gibba012.stdout: "quorum_names": [ 2021-02-10T11:14:22.845 INFO:teuthology.orchestra.run.gibba012.stdout: "a", 2021-02-10T11:14:22.845 INFO:teuthology.orchestra.run.gibba012.stdout: "b", 2021-02-10T11:14:22.845 INFO:teuthology.orchestra.run.gibba012.stdout: "c" 2021-02-10T11:14:22.845 INFO:teuthology.orchestra.run.gibba012.stdout: ], 2021-02-10T11:14:22.846 INFO:teuthology.orchestra.run.gibba012.stdout: "quorum_age": 56, 2021-02-10T11:14:22.846 INFO:teuthology.orchestra.run.gibba012.stdout: "monmap": { 2021-02-10T11:14:22.846 INFO:teuthology.orchestra.run.gibba012.stdout: "epoch": 1, 2021-02-10T11:14:22.846 INFO:teuthology.orchestra.run.gibba012.stdout: "min_mon_release_name": "14", 2021-02-10T11:14:22.846 INFO:teuthology.orchestra.run.gibba012.stdout: "num_mons": 3 2021-02-10T11:14:22.846 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.846 INFO:teuthology.orchestra.run.gibba012.stdout: "osdmap": { 2021-02-10T11:14:22.846 INFO:teuthology.orchestra.run.gibba012.stdout: "osdmap": { 2021-02-10T11:14:22.846 INFO:teuthology.orchestra.run.gibba012.stdout: "epoch": 20, 2021-02-10T11:14:22.846 INFO:teuthology.orchestra.run.gibba012.stdout: "num_osds": 8, 2021-02-10T11:14:22.846 INFO:teuthology.orchestra.run.gibba012.stdout: "num_up_osds": 8, 2021-02-10T11:14:22.846 INFO:teuthology.orchestra.run.gibba012.stdout: "num_in_osds": 8, 2021-02-10T11:14:22.847 INFO:teuthology.orchestra.run.gibba012.stdout: "num_remapped_pgs": 0 2021-02-10T11:14:22.847 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:14:22.847 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.847 INFO:teuthology.orchestra.run.gibba012.stdout: "pgmap": { 2021-02-10T11:14:22.847 INFO:teuthology.orchestra.run.gibba012.stdout: "pgs_by_state": [ 2021-02-10T11:14:22.847 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:14:22.847 INFO:teuthology.orchestra.run.gibba012.stdout: "state_name": "active+clean", 2021-02-10T11:14:22.847 INFO:teuthology.orchestra.run.gibba012.stdout: "count": 24 2021-02-10T11:14:22.847 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:14:22.847 INFO:teuthology.orchestra.run.gibba012.stdout: ], 2021-02-10T11:14:22.847 INFO:teuthology.orchestra.run.gibba012.stdout: "num_pgs": 24, 2021-02-10T11:14:22.847 INFO:teuthology.orchestra.run.gibba012.stdout: "num_pools": 3, 2021-02-10T11:14:22.848 INFO:teuthology.orchestra.run.gibba012.stdout: "num_objects": 22, 2021-02-10T11:14:22.848 INFO:teuthology.orchestra.run.gibba012.stdout: "data_bytes": 2286, 2021-02-10T11:14:22.848 INFO:teuthology.orchestra.run.gibba012.stdout: "bytes_used": 8591704064, 2021-02-10T11:14:22.848 INFO:teuthology.orchestra.run.gibba012.stdout: "bytes_avail": 764502409216, 2021-02-10T11:14:22.848 INFO:teuthology.orchestra.run.gibba012.stdout: "bytes_total": 773094113280, 2021-02-10T11:14:22.848 INFO:teuthology.orchestra.run.gibba012.stdout: "write_bytes_sec": 1279, 2021-02-10T11:14:22.848 INFO:teuthology.orchestra.run.gibba012.stdout: "read_op_per_sec": 0, 2021-02-10T11:14:22.848 INFO:teuthology.orchestra.run.gibba012.stdout: "write_op_per_sec": 4 2021-02-10T11:14:22.848 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.848 INFO:teuthology.orchestra.run.gibba012.stdout: "fsmap": { 2021-02-10T11:14:22.848 INFO:teuthology.orchestra.run.gibba012.stdout: "epoch": 6, 2021-02-10T11:14:22.848 INFO:teuthology.orchestra.run.gibba012.stdout: "id": 1, 2021-02-10T11:14:22.848 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:14:22.849 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:14:22.849 INFO:teuthology.orchestra.run.gibba012.stdout: "max": 1, 2021-02-10T11:14:22.849 INFO:teuthology.orchestra.run.gibba012.stdout: "by_rank": [ 2021-02-10T11:14:22.849 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:14:22.849 INFO:teuthology.orchestra.run.gibba012.stdout: "filesystem_id": 1, 2021-02-10T11:14:22.849 INFO:teuthology.orchestra.run.gibba012.stdout: "rank": 0, 2021-02-10T11:14:22.849 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "a", 2021-02-10T11:14:22.849 INFO:teuthology.orchestra.run.gibba012.stdout: "status": "up:active", 2021-02-10T11:14:22.849 INFO:teuthology.orchestra.run.gibba012.stdout: "gid": 4272 2021-02-10T11:14:22.849 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:14:22.849 INFO:teuthology.orchestra.run.gibba012.stdout: ], 2021-02-10T11:14:22.849 INFO:teuthology.orchestra.run.gibba012.stdout: "up:standby": 1 2021-02-10T11:14:22.849 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.850 INFO:teuthology.orchestra.run.gibba012.stdout: "mgrmap": { 2021-02-10T11:14:22.850 INFO:teuthology.orchestra.run.gibba012.stdout: "epoch": 3, 2021-02-10T11:14:22.850 INFO:teuthology.orchestra.run.gibba012.stdout: "active_gid": 4100, 2021-02-10T11:14:22.850 INFO:teuthology.orchestra.run.gibba012.stdout: "active_name": "x", 2021-02-10T11:14:22.850 INFO:teuthology.orchestra.run.gibba012.stdout: "active_addrs": { 2021-02-10T11:14:22.850 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:14:22.850 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:14:22.850 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:14:22.850 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6800", 2021-02-10T11:14:22.850 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 34990 2021-02-10T11:14:22.850 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.850 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:14:22.851 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:14:22.851 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6801", 2021-02-10T11:14:22.851 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 34990 2021-02-10T11:14:22.851 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:14:22.851 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:14:22.851 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.851 INFO:teuthology.orchestra.run.gibba012.stdout: "active_addr": "172.21.2.120:6801/34990", 2021-02-10T11:14:22.851 INFO:teuthology.orchestra.run.gibba012.stdout: "active_change": "2021-02-10 11:13:27.751969", 2021-02-10T11:14:22.851 INFO:teuthology.orchestra.run.gibba012.stdout: "available": true, 2021-02-10T11:14:22.851 INFO:teuthology.orchestra.run.gibba012.stdout: "standbys": [ 2021-02-10T11:14:22.851 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:14:22.851 INFO:teuthology.orchestra.run.gibba012.stdout: "gid": 4098, 2021-02-10T11:14:22.851 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "y", 2021-02-10T11:14:22.852 INFO:teuthology.orchestra.run.gibba012.stdout: "available_modules": [ 2021-02-10T11:14:22.852 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:14:22.852 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "alerts", 2021-02-10T11:14:22.852 INFO:teuthology.orchestra.run.gibba012.stdout: "can_run": true, 2021-02-10T11:14:22.852 INFO:teuthology.orchestra.run.gibba012.stdout: "error_string": "", 2021-02-10T11:14:22.852 INFO:teuthology.orchestra.run.gibba012.stdout: "module_options": { 2021-02-10T11:14:22.852 INFO:teuthology.orchestra.run.gibba012.stdout: "interval": { 2021-02-10T11:14:22.852 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "interval", 2021-02-10T11:14:22.852 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "secs", 2021-02-10T11:14:22.852 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.852 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:14:22.852 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "60", 2021-02-10T11:14:22.852 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:22.853 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:22.853 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:22.853 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "How frequently to reexamine health status", 2021-02-10T11:14:22.853 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:22.853 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.853 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.853 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.853 INFO:teuthology.orchestra.run.gibba012.stdout: "smtp_destination": { 2021-02-10T11:14:22.853 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "smtp_destination", 2021-02-10T11:14:22.853 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:22.853 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.853 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:14:22.854 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "", 2021-02-10T11:14:22.854 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:22.854 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:22.854 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:22.854 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "Email address to send alerts to", 2021-02-10T11:14:22.854 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:22.854 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.854 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.854 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.854 INFO:teuthology.orchestra.run.gibba012.stdout: "smtp_from_name": { 2021-02-10T11:14:22.854 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "smtp_from_name", 2021-02-10T11:14:22.854 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:22.854 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.855 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:14:22.855 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "Ceph", 2021-02-10T11:14:22.855 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:22.855 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:22.855 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:22.855 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "Email From: name", 2021-02-10T11:14:22.855 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:22.855 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.855 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.855 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.855 INFO:teuthology.orchestra.run.gibba012.stdout: "smtp_host": { 2021-02-10T11:14:22.855 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "smtp_host", 2021-02-10T11:14:22.855 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:22.856 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.856 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:14:22.856 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "", 2021-02-10T11:14:22.856 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:22.856 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:22.856 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:22.856 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "SMTP server", 2021-02-10T11:14:22.856 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:22.856 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.856 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.856 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.856 INFO:teuthology.orchestra.run.gibba012.stdout: "smtp_password": { 2021-02-10T11:14:22.856 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "smtp_password", 2021-02-10T11:14:22.857 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:22.857 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.857 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:14:22.857 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "", 2021-02-10T11:14:22.857 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:22.857 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:22.857 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:22.857 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "Password to authenticate with", 2021-02-10T11:14:22.857 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:22.857 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.857 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.857 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.857 INFO:teuthology.orchestra.run.gibba012.stdout: "smtp_port": { 2021-02-10T11:14:22.858 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "smtp_port", 2021-02-10T11:14:22.858 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "int", 2021-02-10T11:14:22.858 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.858 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:14:22.858 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "465", 2021-02-10T11:14:22.858 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:22.858 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:22.858 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:22.858 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "SMTP port", 2021-02-10T11:14:22.858 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:22.858 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.858 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.859 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.859 INFO:teuthology.orchestra.run.gibba012.stdout: "smtp_sender": { 2021-02-10T11:14:22.859 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "smtp_sender", 2021-02-10T11:14:22.859 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:22.859 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.859 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:14:22.859 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "", 2021-02-10T11:14:22.859 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:22.859 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:22.859 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:22.859 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "SMTP envelope sender", 2021-02-10T11:14:22.859 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:22.859 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.860 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.860 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.860 INFO:teuthology.orchestra.run.gibba012.stdout: "smtp_ssl": { 2021-02-10T11:14:22.860 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "smtp_ssl", 2021-02-10T11:14:22.860 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "bool", 2021-02-10T11:14:22.860 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.860 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:14:22.860 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "True", 2021-02-10T11:14:22.860 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:22.860 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:22.860 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:22.860 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "Use SSL to connect to SMTP server", 2021-02-10T11:14:22.860 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:22.861 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.861 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.861 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.861 INFO:teuthology.orchestra.run.gibba012.stdout: "smtp_user": { 2021-02-10T11:14:22.861 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "smtp_user", 2021-02-10T11:14:22.861 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:22.861 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.861 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:14:22.861 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "", 2021-02-10T11:14:22.861 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:22.861 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:22.861 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:22.862 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "User to authenticate as", 2021-02-10T11:14:22.862 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:22.862 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.862 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.862 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:14:22.862 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:14:22.862 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.862 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:14:22.862 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "ansible", 2021-02-10T11:14:22.862 INFO:teuthology.orchestra.run.gibba012.stdout: "can_run": true, 2021-02-10T11:14:22.862 INFO:teuthology.orchestra.run.gibba012.stdout: "error_string": "", 2021-02-10T11:14:22.862 INFO:teuthology.orchestra.run.gibba012.stdout: "module_options": { 2021-02-10T11:14:22.863 INFO:teuthology.orchestra.run.gibba012.stdout: "password": { 2021-02-10T11:14:22.863 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "password", 2021-02-10T11:14:22.863 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:22.863 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.863 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:22.863 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "", 2021-02-10T11:14:22.863 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:22.863 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:22.863 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:22.863 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:22.863 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:22.863 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.864 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.864 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.864 INFO:teuthology.orchestra.run.gibba012.stdout: "server_url": { 2021-02-10T11:14:22.864 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "server_url", 2021-02-10T11:14:22.864 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:22.864 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.864 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:22.864 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "", 2021-02-10T11:14:22.864 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:22.864 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:22.864 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:22.864 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:22.864 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:22.865 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.865 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.865 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.865 INFO:teuthology.orchestra.run.gibba012.stdout: "username": { 2021-02-10T11:14:22.865 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "username", 2021-02-10T11:14:22.865 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:22.865 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.865 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:22.865 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "", 2021-02-10T11:14:22.865 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:22.865 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:22.865 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:22.865 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:22.866 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:22.866 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.866 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.866 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.866 INFO:teuthology.orchestra.run.gibba012.stdout: "verify_server": { 2021-02-10T11:14:22.866 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "verify_server", 2021-02-10T11:14:22.866 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:22.866 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.866 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:22.866 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "", 2021-02-10T11:14:22.866 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:22.866 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:22.867 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:22.867 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:22.867 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:22.867 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.867 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.867 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:14:22.867 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:14:22.867 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.867 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:14:22.867 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "balancer", 2021-02-10T11:14:22.867 INFO:teuthology.orchestra.run.gibba012.stdout: "can_run": true, 2021-02-10T11:14:22.867 INFO:teuthology.orchestra.run.gibba012.stdout: "error_string": "", 2021-02-10T11:14:22.867 INFO:teuthology.orchestra.run.gibba012.stdout: "module_options": { 2021-02-10T11:14:22.868 INFO:teuthology.orchestra.run.gibba012.stdout: "active": { 2021-02-10T11:14:22.868 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "active", 2021-02-10T11:14:22.868 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "bool", 2021-02-10T11:14:22.868 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.868 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:14:22.868 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "False", 2021-02-10T11:14:22.868 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:22.868 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:22.868 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:22.868 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "automatically balance PGs across cluster", 2021-02-10T11:14:22.868 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:22.868 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.869 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.869 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.869 INFO:teuthology.orchestra.run.gibba012.stdout: "begin_time": { 2021-02-10T11:14:22.869 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "begin_time", 2021-02-10T11:14:22.869 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:22.869 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.869 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:14:22.869 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "0000", 2021-02-10T11:14:22.869 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:22.869 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:22.869 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:22.869 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "beginning time of day to automatically balance", 2021-02-10T11:14:22.869 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "This is a time of day in the format HHMM.", 2021-02-10T11:14:22.870 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.870 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.870 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.870 INFO:teuthology.orchestra.run.gibba012.stdout: "begin_weekday": { 2021-02-10T11:14:22.870 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "begin_weekday", 2021-02-10T11:14:22.870 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "uint", 2021-02-10T11:14:22.870 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.870 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:14:22.870 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "0", 2021-02-10T11:14:22.870 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "0", 2021-02-10T11:14:22.870 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "7", 2021-02-10T11:14:22.870 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:22.871 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2021-02-10T11:14:22.871 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "0 or 7 = Sunday, 1 = Monday, etc.", 2021-02-10T11:14:22.871 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.871 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.871 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.871 INFO:teuthology.orchestra.run.gibba012.stdout: "crush_compat_max_iterations": { 2021-02-10T11:14:22.871 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "crush_compat_max_iterations", 2021-02-10T11:14:22.871 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "uint", 2021-02-10T11:14:22.871 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.871 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:14:22.871 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "25", 2021-02-10T11:14:22.871 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "1", 2021-02-10T11:14:22.871 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "250", 2021-02-10T11:14:22.872 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:22.872 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "maximum number of iterations to attempt optimization", 2021-02-10T11:14:22.872 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:22.872 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.872 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.872 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.872 INFO:teuthology.orchestra.run.gibba012.stdout: "crush_compat_metrics": { 2021-02-10T11:14:22.872 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "crush_compat_metrics", 2021-02-10T11:14:22.872 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:22.872 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.872 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:14:22.872 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "pgs,objects,bytes", 2021-02-10T11:14:22.872 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:22.873 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:22.873 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:22.873 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "metrics with which to calculate OSD utilization", 2021-02-10T11:14:22.873 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2021-02-10T11:14:22.873 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.873 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.873 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.873 INFO:teuthology.orchestra.run.gibba012.stdout: "crush_compat_step": { 2021-02-10T11:14:22.873 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "crush_compat_step", 2021-02-10T11:14:22.873 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "float", 2021-02-10T11:14:22.873 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.873 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:14:22.873 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "0.5", 2021-02-10T11:14:22.874 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "0.001", 2021-02-10T11:14:22.874 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "0.999", 2021-02-10T11:14:22.874 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:22.874 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "aggressiveness of optimization", 2021-02-10T11:14:22.874 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2021-02-10T11:14:22.874 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.874 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.874 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.874 INFO:teuthology.orchestra.run.gibba012.stdout: "end_time": { 2021-02-10T11:14:22.874 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "end_time", 2021-02-10T11:14:22.874 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:22.874 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.874 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:14:22.875 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "2400", 2021-02-10T11:14:22.875 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:22.875 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:22.875 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:22.875 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "ending time of day to automatically balance", 2021-02-10T11:14:22.875 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "This is a time of day in the format HHMM.", 2021-02-10T11:14:22.875 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.875 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.875 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.875 INFO:teuthology.orchestra.run.gibba012.stdout: "end_weekday": { 2021-02-10T11:14:22.875 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "end_weekday", 2021-02-10T11:14:22.875 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "uint", 2021-02-10T11:14:22.876 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.876 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:14:22.876 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "7", 2021-02-10T11:14:22.876 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "0", 2021-02-10T11:14:22.876 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "7", 2021-02-10T11:14:22.876 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:22.876 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2021-02-10T11:14:22.876 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "0 or 7 = Sunday, 1 = Monday, etc.", 2021-02-10T11:14:22.876 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.876 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.877 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.877 INFO:teuthology.orchestra.run.gibba012.stdout: "min_score": { 2021-02-10T11:14:22.877 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "min_score", 2021-02-10T11:14:22.877 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "float", 2021-02-10T11:14:22.877 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.877 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:14:22.877 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "0", 2021-02-10T11:14:22.877 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:22.877 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:22.877 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:22.878 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "minimum score, below which no optimization is attempted", 2021-02-10T11:14:22.878 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:22.878 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.878 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.878 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.878 INFO:teuthology.orchestra.run.gibba012.stdout: "mode": { 2021-02-10T11:14:22.878 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "mode", 2021-02-10T11:14:22.878 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:22.878 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.878 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:14:22.878 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "none", 2021-02-10T11:14:22.879 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:22.879 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:22.879 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [ 2021-02-10T11:14:22.879 INFO:teuthology.orchestra.run.gibba012.stdout: "crush-compat", 2021-02-10T11:14:22.879 INFO:teuthology.orchestra.run.gibba012.stdout: "none", 2021-02-10T11:14:22.879 INFO:teuthology.orchestra.run.gibba012.stdout: "upmap" 2021-02-10T11:14:22.879 INFO:teuthology.orchestra.run.gibba012.stdout: ], 2021-02-10T11:14:22.879 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "Balancer mode", 2021-02-10T11:14:22.879 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:22.879 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.879 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.879 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.879 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_ids": { 2021-02-10T11:14:22.880 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "pool_ids", 2021-02-10T11:14:22.880 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:22.880 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.880 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:14:22.880 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "", 2021-02-10T11:14:22.880 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:22.880 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:22.880 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:22.880 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "pools which the automatic balancing will be limited to", 2021-02-10T11:14:22.880 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:22.880 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.880 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.881 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.881 INFO:teuthology.orchestra.run.gibba012.stdout: "sleep_interval": { 2021-02-10T11:14:22.881 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "sleep_interval", 2021-02-10T11:14:22.881 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "secs", 2021-02-10T11:14:22.881 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.881 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:14:22.881 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "60", 2021-02-10T11:14:22.881 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:22.881 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:22.881 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:22.881 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "how frequently to wake up and attempt optimization", 2021-02-10T11:14:22.881 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:22.881 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.882 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.882 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.882 INFO:teuthology.orchestra.run.gibba012.stdout: "upmap_max_deviation": { 2021-02-10T11:14:22.882 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "upmap_max_deviation", 2021-02-10T11:14:22.882 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "int", 2021-02-10T11:14:22.882 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.882 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:14:22.882 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "5", 2021-02-10T11:14:22.882 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "1", 2021-02-10T11:14:22.882 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:22.882 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:22.883 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "deviation below which no optimization is attempted", 2021-02-10T11:14:22.883 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2021-02-10T11:14:22.883 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.883 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.883 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.883 INFO:teuthology.orchestra.run.gibba012.stdout: "upmap_max_iterations": { 2021-02-10T11:14:22.883 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "upmap_max_iterations", 2021-02-10T11:14:22.883 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "uint", 2021-02-10T11:14:22.883 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.883 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:14:22.883 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "10", 2021-02-10T11:14:22.883 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:22.883 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:22.884 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:22.884 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "maximum upmap optimization iterations", 2021-02-10T11:14:22.884 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:22.884 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.884 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.884 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:14:22.884 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:14:22.884 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.884 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:14:22.884 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "crash", 2021-02-10T11:14:22.884 INFO:teuthology.orchestra.run.gibba012.stdout: "can_run": true, 2021-02-10T11:14:22.884 INFO:teuthology.orchestra.run.gibba012.stdout: "error_string": "", 2021-02-10T11:14:22.885 INFO:teuthology.orchestra.run.gibba012.stdout: "module_options": { 2021-02-10T11:14:22.885 INFO:teuthology.orchestra.run.gibba012.stdout: "retain_interval": { 2021-02-10T11:14:22.885 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "retain_interval", 2021-02-10T11:14:22.885 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "secs", 2021-02-10T11:14:22.885 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.885 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:14:22.885 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "31536000", 2021-02-10T11:14:22.885 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:22.885 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:22.885 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:22.885 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "how long to retain crashes before pruning them", 2021-02-10T11:14:22.885 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:22.885 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.886 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.886 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.886 INFO:teuthology.orchestra.run.gibba012.stdout: "warn_recent_interval": { 2021-02-10T11:14:22.886 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "warn_recent_interval", 2021-02-10T11:14:22.886 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "secs", 2021-02-10T11:14:22.886 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.886 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:14:22.886 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "1209600", 2021-02-10T11:14:22.886 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:22.886 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:22.886 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:22.886 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "time interval in which to warn about recent crashes", 2021-02-10T11:14:22.886 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:22.887 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.887 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.887 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:14:22.887 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:14:22.887 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.887 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:14:22.887 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "dashboard", 2021-02-10T11:14:22.887 INFO:teuthology.orchestra.run.gibba012.stdout: "can_run": true, 2021-02-10T11:14:22.887 INFO:teuthology.orchestra.run.gibba012.stdout: "error_string": "", 2021-02-10T11:14:22.887 INFO:teuthology.orchestra.run.gibba012.stdout: "module_options": { 2021-02-10T11:14:22.887 INFO:teuthology.orchestra.run.gibba012.stdout: "ALERTMANAGER_API_HOST": { 2021-02-10T11:14:22.887 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "ALERTMANAGER_API_HOST", 2021-02-10T11:14:22.887 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:22.888 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.888 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:22.888 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "", 2021-02-10T11:14:22.888 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:22.888 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:22.888 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:22.888 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:22.888 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:22.888 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.888 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.888 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.888 INFO:teuthology.orchestra.run.gibba012.stdout: "AUDIT_API_ENABLED": { 2021-02-10T11:14:22.889 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "AUDIT_API_ENABLED", 2021-02-10T11:14:22.889 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "bool", 2021-02-10T11:14:22.889 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.889 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:22.889 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "False", 2021-02-10T11:14:22.889 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:22.889 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:22.889 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:22.889 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:22.889 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:22.889 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.890 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.890 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.890 INFO:teuthology.orchestra.run.gibba012.stdout: "AUDIT_API_LOG_PAYLOAD": { 2021-02-10T11:14:22.890 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2021-02-10T11:14:22.890 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "bool", 2021-02-10T11:14:22.890 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.890 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:22.890 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "True", 2021-02-10T11:14:22.891 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:22.891 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:22.891 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:22.891 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:22.891 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:22.891 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.891 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.891 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.891 INFO:teuthology.orchestra.run.gibba012.stdout: "ENABLE_BROWSABLE_API": { 2021-02-10T11:14:22.891 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "ENABLE_BROWSABLE_API", 2021-02-10T11:14:22.891 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "bool", 2021-02-10T11:14:22.891 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.892 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:22.892 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "True", 2021-02-10T11:14:22.892 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:22.892 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:22.892 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:22.892 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:22.892 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:22.892 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.892 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.892 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.892 INFO:teuthology.orchestra.run.gibba012.stdout: "FEATURE_TOGGLE_cephfs": { 2021-02-10T11:14:22.892 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "FEATURE_TOGGLE_cephfs", 2021-02-10T11:14:22.893 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "bool", 2021-02-10T11:14:22.893 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.893 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:22.893 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "True", 2021-02-10T11:14:22.893 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:22.893 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:22.893 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:22.893 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:22.893 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:22.893 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.893 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.893 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.893 INFO:teuthology.orchestra.run.gibba012.stdout: "FEATURE_TOGGLE_iscsi": { 2021-02-10T11:14:22.894 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "FEATURE_TOGGLE_iscsi", 2021-02-10T11:14:22.894 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "bool", 2021-02-10T11:14:22.894 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.894 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:22.894 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "True", 2021-02-10T11:14:22.894 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:22.894 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:22.894 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:22.894 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:22.894 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:22.894 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.894 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.895 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.895 INFO:teuthology.orchestra.run.gibba012.stdout: "FEATURE_TOGGLE_mirroring": { 2021-02-10T11:14:22.895 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "FEATURE_TOGGLE_mirroring", 2021-02-10T11:14:22.895 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "bool", 2021-02-10T11:14:22.895 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.895 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:22.895 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "True", 2021-02-10T11:14:22.895 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:22.895 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:22.895 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:22.895 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:22.895 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:22.895 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.896 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.896 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.896 INFO:teuthology.orchestra.run.gibba012.stdout: "FEATURE_TOGGLE_rbd": { 2021-02-10T11:14:22.896 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "FEATURE_TOGGLE_rbd", 2021-02-10T11:14:22.896 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "bool", 2021-02-10T11:14:22.896 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.896 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:22.896 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "True", 2021-02-10T11:14:22.896 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:22.896 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:22.896 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:22.896 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:22.896 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:22.896 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.897 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.897 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.897 INFO:teuthology.orchestra.run.gibba012.stdout: "FEATURE_TOGGLE_rgw": { 2021-02-10T11:14:22.897 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "FEATURE_TOGGLE_rgw", 2021-02-10T11:14:22.897 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "bool", 2021-02-10T11:14:22.897 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.897 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:22.897 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "True", 2021-02-10T11:14:22.897 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:22.897 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:22.897 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:22.897 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:22.897 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:22.898 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.898 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.898 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.898 INFO:teuthology.orchestra.run.gibba012.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2021-02-10T11:14:22.898 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2021-02-10T11:14:22.898 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:22.898 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.898 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:22.898 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "", 2021-02-10T11:14:22.898 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:22.898 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:22.899 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:22.899 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:22.899 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:22.899 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.899 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.899 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.899 INFO:teuthology.orchestra.run.gibba012.stdout: "GRAFANA_API_PASSWORD": { 2021-02-10T11:14:22.899 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "GRAFANA_API_PASSWORD", 2021-02-10T11:14:22.899 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:22.899 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.899 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:22.899 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "admin", 2021-02-10T11:14:22.899 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:22.900 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:22.900 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:22.900 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:22.900 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:22.900 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.900 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.900 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.900 INFO:teuthology.orchestra.run.gibba012.stdout: "GRAFANA_API_SSL_VERIFY": { 2021-02-10T11:14:22.900 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2021-02-10T11:14:22.900 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "bool", 2021-02-10T11:14:22.900 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.900 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:22.900 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "True", 2021-02-10T11:14:22.901 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:22.901 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:22.901 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:22.901 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:22.901 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:22.901 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.901 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.901 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.901 INFO:teuthology.orchestra.run.gibba012.stdout: "GRAFANA_API_URL": { 2021-02-10T11:14:22.902 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "GRAFANA_API_URL", 2021-02-10T11:14:22.902 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:22.902 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.902 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:22.902 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "", 2021-02-10T11:14:22.902 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:22.902 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:22.902 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:22.902 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:22.902 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:22.902 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.902 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.902 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.903 INFO:teuthology.orchestra.run.gibba012.stdout: "GRAFANA_API_USERNAME": { 2021-02-10T11:14:22.903 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "GRAFANA_API_USERNAME", 2021-02-10T11:14:22.903 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:22.903 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.903 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:22.903 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "admin", 2021-02-10T11:14:22.903 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:22.903 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:22.903 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:22.903 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:22.903 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:22.903 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.904 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.904 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.904 INFO:teuthology.orchestra.run.gibba012.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2021-02-10T11:14:22.904 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2021-02-10T11:14:22.904 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "bool", 2021-02-10T11:14:22.904 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.904 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:22.904 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "False", 2021-02-10T11:14:22.904 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:22.904 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:22.904 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:22.904 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:22.905 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:22.905 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.905 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.905 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.905 INFO:teuthology.orchestra.run.gibba012.stdout: "ISCSI_API_SSL_VERIFICATION": { 2021-02-10T11:14:22.905 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2021-02-10T11:14:22.905 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "bool", 2021-02-10T11:14:22.905 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.905 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:22.905 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "True", 2021-02-10T11:14:22.905 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:22.905 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:22.906 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:22.906 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:22.906 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:22.906 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.906 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.906 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.906 INFO:teuthology.orchestra.run.gibba012.stdout: "PROMETHEUS_API_HOST": { 2021-02-10T11:14:22.906 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "PROMETHEUS_API_HOST", 2021-02-10T11:14:22.906 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:22.906 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.906 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:22.906 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "", 2021-02-10T11:14:22.907 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:22.907 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:22.907 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:22.907 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:22.907 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:22.907 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.907 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.907 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.908 INFO:teuthology.orchestra.run.gibba012.stdout: "REST_REQUESTS_TIMEOUT": { 2021-02-10T11:14:22.908 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "REST_REQUESTS_TIMEOUT", 2021-02-10T11:14:22.908 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "int", 2021-02-10T11:14:22.908 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.908 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:22.908 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "45", 2021-02-10T11:14:22.908 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:22.908 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:22.908 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:22.908 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:22.908 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:22.908 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.908 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.909 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.909 INFO:teuthology.orchestra.run.gibba012.stdout: "RGW_API_ACCESS_KEY": { 2021-02-10T11:14:22.909 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "RGW_API_ACCESS_KEY", 2021-02-10T11:14:22.909 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:22.909 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.909 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:22.909 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "", 2021-02-10T11:14:22.909 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:22.909 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:22.909 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:22.909 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:22.909 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:22.909 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.910 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.910 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.910 INFO:teuthology.orchestra.run.gibba012.stdout: "RGW_API_ADMIN_RESOURCE": { 2021-02-10T11:14:22.910 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2021-02-10T11:14:22.910 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:22.910 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.910 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:22.910 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "admin", 2021-02-10T11:14:22.910 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:22.910 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:22.910 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:22.910 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:22.910 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:22.911 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.911 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.911 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.911 INFO:teuthology.orchestra.run.gibba012.stdout: "RGW_API_HOST": { 2021-02-10T11:14:22.911 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "RGW_API_HOST", 2021-02-10T11:14:22.911 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:22.911 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.911 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:22.911 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "", 2021-02-10T11:14:22.911 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:22.911 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:22.911 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:22.911 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:22.912 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:22.912 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.912 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.912 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.912 INFO:teuthology.orchestra.run.gibba012.stdout: "RGW_API_PORT": { 2021-02-10T11:14:22.912 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "RGW_API_PORT", 2021-02-10T11:14:22.912 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "int", 2021-02-10T11:14:22.912 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.912 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:22.912 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "80", 2021-02-10T11:14:22.912 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:22.912 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:22.912 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:22.913 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:22.913 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:22.913 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.913 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.913 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.913 INFO:teuthology.orchestra.run.gibba012.stdout: "RGW_API_SCHEME": { 2021-02-10T11:14:22.913 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "RGW_API_SCHEME", 2021-02-10T11:14:22.913 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:22.913 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.913 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:22.913 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "http", 2021-02-10T11:14:22.913 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:22.913 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:22.914 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:22.914 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:22.914 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:22.914 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.914 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.914 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.914 INFO:teuthology.orchestra.run.gibba012.stdout: "RGW_API_SECRET_KEY": { 2021-02-10T11:14:22.914 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "RGW_API_SECRET_KEY", 2021-02-10T11:14:22.914 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:22.914 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.914 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:22.914 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "", 2021-02-10T11:14:22.914 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:22.915 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:22.915 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:22.915 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:22.915 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:22.915 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.915 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.915 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.915 INFO:teuthology.orchestra.run.gibba012.stdout: "RGW_API_SSL_VERIFY": { 2021-02-10T11:14:22.915 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "RGW_API_SSL_VERIFY", 2021-02-10T11:14:22.915 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "bool", 2021-02-10T11:14:22.915 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.915 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:22.915 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "True", 2021-02-10T11:14:22.915 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:22.916 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:22.916 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:22.916 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:22.916 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:22.916 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.916 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.916 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.916 INFO:teuthology.orchestra.run.gibba012.stdout: "RGW_API_USER_ID": { 2021-02-10T11:14:22.916 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "RGW_API_USER_ID", 2021-02-10T11:14:22.916 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:22.916 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.916 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:22.917 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "", 2021-02-10T11:14:22.917 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:22.917 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:22.917 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:22.917 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:22.917 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:22.917 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.917 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.917 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.917 INFO:teuthology.orchestra.run.gibba012.stdout: "crt_file": { 2021-02-10T11:14:22.917 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "crt_file", 2021-02-10T11:14:22.917 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:22.917 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.918 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:22.918 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "", 2021-02-10T11:14:22.918 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:22.918 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:22.918 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:22.918 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:22.918 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:22.918 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.918 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.918 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.919 INFO:teuthology.orchestra.run.gibba012.stdout: "debug": { 2021-02-10T11:14:22.919 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "debug", 2021-02-10T11:14:22.919 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "bool", 2021-02-10T11:14:22.919 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.919 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:22.919 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "False", 2021-02-10T11:14:22.919 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:22.919 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:22.919 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:22.919 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "Enable/disable debug options", 2021-02-10T11:14:22.919 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:22.919 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.919 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.920 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.920 INFO:teuthology.orchestra.run.gibba012.stdout: "jwt_token_ttl": { 2021-02-10T11:14:22.920 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "jwt_token_ttl", 2021-02-10T11:14:22.920 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "int", 2021-02-10T11:14:22.920 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.920 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:22.920 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "28800", 2021-02-10T11:14:22.920 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:22.920 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:22.920 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:22.920 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:22.920 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:22.920 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.921 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.921 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.921 INFO:teuthology.orchestra.run.gibba012.stdout: "key_file": { 2021-02-10T11:14:22.921 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "key_file", 2021-02-10T11:14:22.921 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:22.921 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.921 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:22.921 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "", 2021-02-10T11:14:22.921 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:22.921 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:22.921 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:22.921 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:22.922 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:22.922 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.922 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.922 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.922 INFO:teuthology.orchestra.run.gibba012.stdout: "password": { 2021-02-10T11:14:22.922 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "password", 2021-02-10T11:14:22.922 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:22.922 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.922 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:22.922 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "", 2021-02-10T11:14:22.922 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:22.923 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:22.923 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:22.923 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:22.923 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:22.923 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.924 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.924 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.924 INFO:teuthology.orchestra.run.gibba012.stdout: "server_addr": { 2021-02-10T11:14:22.924 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "server_addr", 2021-02-10T11:14:22.924 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:22.924 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.924 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:22.925 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "::", 2021-02-10T11:14:22.925 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:22.925 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:22.925 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:22.925 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:22.925 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:22.925 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.926 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.926 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.926 INFO:teuthology.orchestra.run.gibba012.stdout: "server_port": { 2021-02-10T11:14:22.926 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "server_port", 2021-02-10T11:14:22.926 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "int", 2021-02-10T11:14:22.926 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.926 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:22.926 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "8080", 2021-02-10T11:14:22.927 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:22.927 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:22.927 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:22.927 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:22.927 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:22.927 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.927 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.927 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.927 INFO:teuthology.orchestra.run.gibba012.stdout: "ssl": { 2021-02-10T11:14:22.927 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "ssl", 2021-02-10T11:14:22.928 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "bool", 2021-02-10T11:14:22.928 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.928 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:22.928 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "True", 2021-02-10T11:14:22.928 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:22.928 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:22.928 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:22.928 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:22.928 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:22.928 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.928 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.928 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.929 INFO:teuthology.orchestra.run.gibba012.stdout: "ssl_server_port": { 2021-02-10T11:14:22.929 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "ssl_server_port", 2021-02-10T11:14:22.929 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "int", 2021-02-10T11:14:22.929 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.929 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:22.929 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "8443", 2021-02-10T11:14:22.929 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:22.930 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:22.930 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:22.930 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:22.930 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:22.930 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.930 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.930 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.930 INFO:teuthology.orchestra.run.gibba012.stdout: "standby_behaviour": { 2021-02-10T11:14:22.930 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "standby_behaviour", 2021-02-10T11:14:22.930 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:22.930 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.930 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:22.931 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "redirect", 2021-02-10T11:14:22.931 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:22.931 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:22.931 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [ 2021-02-10T11:14:22.931 INFO:teuthology.orchestra.run.gibba012.stdout: "error", 2021-02-10T11:14:22.931 INFO:teuthology.orchestra.run.gibba012.stdout: "redirect" 2021-02-10T11:14:22.931 INFO:teuthology.orchestra.run.gibba012.stdout: ], 2021-02-10T11:14:22.931 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:22.931 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:22.931 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.931 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.931 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.931 INFO:teuthology.orchestra.run.gibba012.stdout: "standby_error_status_code": { 2021-02-10T11:14:22.932 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "standby_error_status_code", 2021-02-10T11:14:22.932 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "int", 2021-02-10T11:14:22.932 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.932 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:22.932 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "500", 2021-02-10T11:14:22.932 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "400", 2021-02-10T11:14:22.932 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "599", 2021-02-10T11:14:22.932 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:22.932 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:22.932 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:22.932 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.932 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.932 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.933 INFO:teuthology.orchestra.run.gibba012.stdout: "url_prefix": { 2021-02-10T11:14:22.933 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "url_prefix", 2021-02-10T11:14:22.933 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:22.933 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.933 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:22.933 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "", 2021-02-10T11:14:22.933 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:22.933 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:22.933 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:22.933 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:22.933 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:22.933 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.934 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.934 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.934 INFO:teuthology.orchestra.run.gibba012.stdout: "username": { 2021-02-10T11:14:22.934 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "username", 2021-02-10T11:14:22.934 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:22.934 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.934 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:22.934 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "", 2021-02-10T11:14:22.934 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:22.934 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:22.934 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:22.934 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:22.935 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:22.935 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.935 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.935 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:14:22.935 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:14:22.935 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.935 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:14:22.935 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "deepsea", 2021-02-10T11:14:22.935 INFO:teuthology.orchestra.run.gibba012.stdout: "can_run": true, 2021-02-10T11:14:22.935 INFO:teuthology.orchestra.run.gibba012.stdout: "error_string": "", 2021-02-10T11:14:22.935 INFO:teuthology.orchestra.run.gibba012.stdout: "module_options": { 2021-02-10T11:14:22.935 INFO:teuthology.orchestra.run.gibba012.stdout: "salt_api_eauth": { 2021-02-10T11:14:22.935 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "salt_api_eauth", 2021-02-10T11:14:22.936 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:22.936 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.936 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:22.936 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "sharedsecret", 2021-02-10T11:14:22.936 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:22.936 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:22.936 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:22.936 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:22.936 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:22.936 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.936 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.936 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.936 INFO:teuthology.orchestra.run.gibba012.stdout: "salt_api_password": { 2021-02-10T11:14:22.937 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "salt_api_password", 2021-02-10T11:14:22.937 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:22.937 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.937 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:22.937 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "", 2021-02-10T11:14:22.937 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:22.937 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:22.937 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:22.937 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:22.937 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:22.937 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.937 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.937 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.938 INFO:teuthology.orchestra.run.gibba012.stdout: "salt_api_url": { 2021-02-10T11:14:22.938 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "salt_api_url", 2021-02-10T11:14:22.938 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:22.938 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.938 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:22.938 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "", 2021-02-10T11:14:22.938 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:22.938 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:22.938 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:22.938 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:22.938 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:22.938 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.939 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.939 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.939 INFO:teuthology.orchestra.run.gibba012.stdout: "salt_api_username": { 2021-02-10T11:14:22.939 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "salt_api_username", 2021-02-10T11:14:22.939 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:22.939 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.939 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:22.939 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "", 2021-02-10T11:14:22.939 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:22.939 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:22.939 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:22.939 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:22.939 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:22.940 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.940 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.940 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:14:22.940 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:14:22.940 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.940 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:14:22.940 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "devicehealth", 2021-02-10T11:14:22.940 INFO:teuthology.orchestra.run.gibba012.stdout: "can_run": true, 2021-02-10T11:14:22.940 INFO:teuthology.orchestra.run.gibba012.stdout: "error_string": "", 2021-02-10T11:14:22.940 INFO:teuthology.orchestra.run.gibba012.stdout: "module_options": { 2021-02-10T11:14:22.940 INFO:teuthology.orchestra.run.gibba012.stdout: "enable_monitoring": { 2021-02-10T11:14:22.941 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "enable_monitoring", 2021-02-10T11:14:22.941 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "bool", 2021-02-10T11:14:22.941 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.941 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:14:22.941 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "False", 2021-02-10T11:14:22.941 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:22.941 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:22.941 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:22.941 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "monitor device health metrics", 2021-02-10T11:14:22.941 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:22.941 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.941 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.941 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.942 INFO:teuthology.orchestra.run.gibba012.stdout: "mark_out_threshold": { 2021-02-10T11:14:22.942 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "mark_out_threshold", 2021-02-10T11:14:22.942 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "secs", 2021-02-10T11:14:22.942 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.942 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:14:22.942 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "2419200", 2021-02-10T11:14:22.942 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:22.942 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:22.942 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:22.942 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "automatically mark OSD if it may fail before this long", 2021-02-10T11:14:22.942 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:22.942 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.943 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.943 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.943 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_name": { 2021-02-10T11:14:22.943 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "pool_name", 2021-02-10T11:14:22.943 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:22.943 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.943 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:14:22.943 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "device_health_metrics", 2021-02-10T11:14:22.943 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:22.943 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:22.943 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:22.943 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "name of pool in which to store device health metrics", 2021-02-10T11:14:22.943 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:22.944 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.944 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.944 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.944 INFO:teuthology.orchestra.run.gibba012.stdout: "retention_period": { 2021-02-10T11:14:22.944 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "retention_period", 2021-02-10T11:14:22.944 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "secs", 2021-02-10T11:14:22.944 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.944 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:14:22.944 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "15552000", 2021-02-10T11:14:22.944 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:22.944 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:22.945 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:22.945 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "how long to retain device health metrics", 2021-02-10T11:14:22.945 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:22.945 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.945 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.945 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.945 INFO:teuthology.orchestra.run.gibba012.stdout: "scrape_frequency": { 2021-02-10T11:14:22.945 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "scrape_frequency", 2021-02-10T11:14:22.945 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "secs", 2021-02-10T11:14:22.945 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.945 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:14:22.945 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "86400", 2021-02-10T11:14:22.945 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:22.946 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:22.946 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:22.946 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "how frequently to scrape device health metrics", 2021-02-10T11:14:22.946 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:22.946 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.946 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.946 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.946 INFO:teuthology.orchestra.run.gibba012.stdout: "self_heal": { 2021-02-10T11:14:22.946 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "self_heal", 2021-02-10T11:14:22.946 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "bool", 2021-02-10T11:14:22.946 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.946 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:14:22.947 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "True", 2021-02-10T11:14:22.947 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:22.947 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:22.947 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:22.947 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "preemptively heal cluster around devices that may fail", 2021-02-10T11:14:22.947 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:22.947 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.947 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.947 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.947 INFO:teuthology.orchestra.run.gibba012.stdout: "sleep_interval": { 2021-02-10T11:14:22.947 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "sleep_interval", 2021-02-10T11:14:22.948 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "secs", 2021-02-10T11:14:22.948 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.948 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:14:22.948 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "600", 2021-02-10T11:14:22.948 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:22.948 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:22.948 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:22.948 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "how frequently to wake up and check device health", 2021-02-10T11:14:22.948 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:22.948 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.948 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.948 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.948 INFO:teuthology.orchestra.run.gibba012.stdout: "warn_threshold": { 2021-02-10T11:14:22.949 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "warn_threshold", 2021-02-10T11:14:22.949 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "secs", 2021-02-10T11:14:22.949 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.949 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:14:22.949 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "7257600", 2021-02-10T11:14:22.949 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:22.949 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:22.949 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:22.949 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "raise health warning if OSD may fail before this long", 2021-02-10T11:14:22.949 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:22.949 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.949 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.950 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:14:22.950 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:14:22.950 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.950 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:14:22.950 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "diskprediction_cloud", 2021-02-10T11:14:22.950 INFO:teuthology.orchestra.run.gibba012.stdout: "can_run": true, 2021-02-10T11:14:22.950 INFO:teuthology.orchestra.run.gibba012.stdout: "error_string": "", 2021-02-10T11:14:22.950 INFO:teuthology.orchestra.run.gibba012.stdout: "module_options": { 2021-02-10T11:14:22.950 INFO:teuthology.orchestra.run.gibba012.stdout: "diskprediction_cert_context": { 2021-02-10T11:14:22.950 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "diskprediction_cert_context", 2021-02-10T11:14:22.950 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:22.950 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.951 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:22.951 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "", 2021-02-10T11:14:22.951 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:22.951 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:22.951 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:22.951 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:22.951 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:22.951 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.951 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.951 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.951 INFO:teuthology.orchestra.run.gibba012.stdout: "diskprediction_default_authority": { 2021-02-10T11:14:22.952 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "diskprediction_default_authority", 2021-02-10T11:14:22.952 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:22.952 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.952 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:22.952 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "localhost", 2021-02-10T11:14:22.952 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:22.952 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:22.952 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:22.952 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:22.952 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:22.952 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.952 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.952 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.953 INFO:teuthology.orchestra.run.gibba012.stdout: "diskprediction_password": { 2021-02-10T11:14:22.953 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "diskprediction_password", 2021-02-10T11:14:22.953 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:22.953 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.953 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:22.953 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "", 2021-02-10T11:14:22.953 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:22.953 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:22.953 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:22.953 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:22.954 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:22.954 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.954 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.954 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.954 INFO:teuthology.orchestra.run.gibba012.stdout: "diskprediction_port": { 2021-02-10T11:14:22.954 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "diskprediction_port", 2021-02-10T11:14:22.954 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:22.954 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.954 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:22.954 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "31400", 2021-02-10T11:14:22.954 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:22.954 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:22.955 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:22.955 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:22.955 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:22.955 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.955 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.955 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.955 INFO:teuthology.orchestra.run.gibba012.stdout: "diskprediction_retrieve_prediction_interval": { 2021-02-10T11:14:22.955 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "diskprediction_retrieve_prediction_interval", 2021-02-10T11:14:22.955 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:22.955 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.955 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:22.955 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "43200", 2021-02-10T11:14:22.955 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:22.956 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:22.956 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:22.956 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:22.956 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:22.956 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.956 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.956 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.956 INFO:teuthology.orchestra.run.gibba012.stdout: "diskprediction_server": { 2021-02-10T11:14:22.956 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "diskprediction_server", 2021-02-10T11:14:22.956 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:22.956 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.956 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:22.956 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "", 2021-02-10T11:14:22.957 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:22.957 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:22.957 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:22.957 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:22.957 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:22.957 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.957 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.957 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.957 INFO:teuthology.orchestra.run.gibba012.stdout: "diskprediction_ssl_target_name_override": { 2021-02-10T11:14:22.957 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "diskprediction_ssl_target_name_override", 2021-02-10T11:14:22.957 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:22.957 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.957 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:22.958 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "localhost", 2021-02-10T11:14:22.958 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:22.958 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:22.958 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:22.958 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:22.958 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:22.958 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.958 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.958 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.958 INFO:teuthology.orchestra.run.gibba012.stdout: "diskprediction_upload_metrics_interval": { 2021-02-10T11:14:22.958 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "diskprediction_upload_metrics_interval", 2021-02-10T11:14:22.958 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:22.959 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.959 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:22.959 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "600", 2021-02-10T11:14:22.959 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:22.959 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:22.959 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:22.959 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:22.959 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:22.959 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.959 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.959 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.959 INFO:teuthology.orchestra.run.gibba012.stdout: "diskprediction_upload_smart_interval": { 2021-02-10T11:14:22.960 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "diskprediction_upload_smart_interval", 2021-02-10T11:14:22.960 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:22.960 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.960 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:22.960 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "43200", 2021-02-10T11:14:22.960 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:22.960 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:22.960 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:22.960 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:22.960 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:22.960 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.960 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.961 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.961 INFO:teuthology.orchestra.run.gibba012.stdout: "diskprediction_user": { 2021-02-10T11:14:22.961 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "diskprediction_user", 2021-02-10T11:14:22.961 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:22.961 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.961 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:22.961 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "", 2021-02-10T11:14:22.961 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:22.961 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:22.961 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:22.961 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:22.961 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:22.962 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.962 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.962 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.962 INFO:teuthology.orchestra.run.gibba012.stdout: "sleep_interval": { 2021-02-10T11:14:22.962 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "sleep_interval", 2021-02-10T11:14:22.962 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:22.962 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.962 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:22.962 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "600", 2021-02-10T11:14:22.962 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:22.962 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:22.963 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:22.963 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:22.963 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:22.963 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.963 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.963 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:14:22.963 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:14:22.963 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.963 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:14:22.963 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "diskprediction_local", 2021-02-10T11:14:22.963 INFO:teuthology.orchestra.run.gibba012.stdout: "can_run": true, 2021-02-10T11:14:22.963 INFO:teuthology.orchestra.run.gibba012.stdout: "error_string": "", 2021-02-10T11:14:22.963 INFO:teuthology.orchestra.run.gibba012.stdout: "module_options": { 2021-02-10T11:14:22.964 INFO:teuthology.orchestra.run.gibba012.stdout: "predict_interval": { 2021-02-10T11:14:22.964 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "predict_interval", 2021-02-10T11:14:22.964 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:22.964 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.964 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:22.964 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "86400", 2021-02-10T11:14:22.964 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:22.964 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:22.964 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:22.964 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:22.964 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:22.964 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.964 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.965 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.965 INFO:teuthology.orchestra.run.gibba012.stdout: "sleep_interval": { 2021-02-10T11:14:22.965 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "sleep_interval", 2021-02-10T11:14:22.965 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:22.965 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.965 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:22.965 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "600", 2021-02-10T11:14:22.965 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:22.965 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:22.965 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:22.965 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:22.965 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:22.966 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.966 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.966 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:14:22.966 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:14:22.966 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.966 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:14:22.966 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "influx", 2021-02-10T11:14:22.966 INFO:teuthology.orchestra.run.gibba012.stdout: "can_run": false, 2021-02-10T11:14:22.966 INFO:teuthology.orchestra.run.gibba012.stdout: "error_string": "influxdb python module not found", 2021-02-10T11:14:22.966 INFO:teuthology.orchestra.run.gibba012.stdout: "module_options": { 2021-02-10T11:14:22.966 INFO:teuthology.orchestra.run.gibba012.stdout: "batch_size": { 2021-02-10T11:14:22.967 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "batch_size", 2021-02-10T11:14:22.967 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:22.967 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.967 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:22.967 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "5000", 2021-02-10T11:14:22.967 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:22.967 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:22.967 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:22.967 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:22.967 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:22.967 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.967 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.967 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.968 INFO:teuthology.orchestra.run.gibba012.stdout: "database": { 2021-02-10T11:14:22.968 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "database", 2021-02-10T11:14:22.968 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:22.968 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.968 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:22.968 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "ceph", 2021-02-10T11:14:22.968 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:22.968 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:22.968 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:22.968 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:22.968 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:22.968 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.968 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.969 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.969 INFO:teuthology.orchestra.run.gibba012.stdout: "hostname": { 2021-02-10T11:14:22.969 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "hostname", 2021-02-10T11:14:22.969 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:22.969 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.969 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:22.969 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "None", 2021-02-10T11:14:22.969 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:22.969 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:22.969 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:22.969 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:22.969 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:22.970 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.970 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.970 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.970 INFO:teuthology.orchestra.run.gibba012.stdout: "interval": { 2021-02-10T11:14:22.970 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "interval", 2021-02-10T11:14:22.970 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:22.970 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.970 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:22.970 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "30", 2021-02-10T11:14:22.970 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:22.970 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:22.970 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:22.971 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:22.971 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:22.971 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.971 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.971 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.971 INFO:teuthology.orchestra.run.gibba012.stdout: "password": { 2021-02-10T11:14:22.971 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "password", 2021-02-10T11:14:22.971 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:22.971 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.971 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:22.971 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "None", 2021-02-10T11:14:22.971 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:22.972 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:22.972 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:22.972 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:22.972 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:22.972 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.972 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.972 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.972 INFO:teuthology.orchestra.run.gibba012.stdout: "port": { 2021-02-10T11:14:22.972 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "port", 2021-02-10T11:14:22.972 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:22.972 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.972 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:22.972 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "8086", 2021-02-10T11:14:22.973 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:22.973 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:22.973 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:22.973 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:22.973 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:22.973 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.973 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.973 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.973 INFO:teuthology.orchestra.run.gibba012.stdout: "ssl": { 2021-02-10T11:14:22.973 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "ssl", 2021-02-10T11:14:22.974 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:22.974 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.974 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:22.974 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "false", 2021-02-10T11:14:22.974 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:22.974 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:22.974 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:22.974 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:22.974 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:22.974 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.974 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.975 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.975 INFO:teuthology.orchestra.run.gibba012.stdout: "threads": { 2021-02-10T11:14:22.975 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "threads", 2021-02-10T11:14:22.975 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:22.975 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.975 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:22.975 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "5", 2021-02-10T11:14:22.975 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:22.975 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:22.975 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:22.975 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:22.975 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:22.975 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.976 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.976 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.976 INFO:teuthology.orchestra.run.gibba012.stdout: "username": { 2021-02-10T11:14:22.976 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "username", 2021-02-10T11:14:22.976 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:22.976 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.976 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:22.976 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "None", 2021-02-10T11:14:22.976 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:22.976 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:22.976 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:22.976 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:22.977 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:22.977 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.977 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.977 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.977 INFO:teuthology.orchestra.run.gibba012.stdout: "verify_ssl": { 2021-02-10T11:14:22.977 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "verify_ssl", 2021-02-10T11:14:22.977 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:22.977 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.977 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:22.977 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "true", 2021-02-10T11:14:22.977 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:22.977 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:22.977 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:22.978 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:22.978 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:22.978 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.978 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.978 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:14:22.978 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:14:22.978 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.978 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:14:22.978 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "insights", 2021-02-10T11:14:22.978 INFO:teuthology.orchestra.run.gibba012.stdout: "can_run": true, 2021-02-10T11:14:22.978 INFO:teuthology.orchestra.run.gibba012.stdout: "error_string": "", 2021-02-10T11:14:22.978 INFO:teuthology.orchestra.run.gibba012.stdout: "module_options": {} 2021-02-10T11:14:22.979 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.979 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:14:22.979 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "iostat", 2021-02-10T11:14:22.979 INFO:teuthology.orchestra.run.gibba012.stdout: "can_run": true, 2021-02-10T11:14:22.979 INFO:teuthology.orchestra.run.gibba012.stdout: "error_string": "", 2021-02-10T11:14:22.979 INFO:teuthology.orchestra.run.gibba012.stdout: "module_options": {} 2021-02-10T11:14:22.979 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.979 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:14:22.979 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "localpool", 2021-02-10T11:14:22.979 INFO:teuthology.orchestra.run.gibba012.stdout: "can_run": true, 2021-02-10T11:14:22.979 INFO:teuthology.orchestra.run.gibba012.stdout: "error_string": "", 2021-02-10T11:14:22.979 INFO:teuthology.orchestra.run.gibba012.stdout: "module_options": { 2021-02-10T11:14:22.979 INFO:teuthology.orchestra.run.gibba012.stdout: "failure_domain": { 2021-02-10T11:14:22.980 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "failure_domain", 2021-02-10T11:14:22.980 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:22.980 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.980 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:14:22.980 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "host", 2021-02-10T11:14:22.980 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:22.980 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:22.980 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:22.980 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "failure domain for any created local pool", 2021-02-10T11:14:22.980 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:22.980 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.980 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.981 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.981 INFO:teuthology.orchestra.run.gibba012.stdout: "min_size": { 2021-02-10T11:14:22.981 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "min_size", 2021-02-10T11:14:22.981 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "int", 2021-02-10T11:14:22.981 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.981 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:14:22.981 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "", 2021-02-10T11:14:22.981 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:22.981 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:22.981 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:22.981 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "default min_size for any created local pool", 2021-02-10T11:14:22.981 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:22.981 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.982 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.982 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.982 INFO:teuthology.orchestra.run.gibba012.stdout: "num_rep": { 2021-02-10T11:14:22.982 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "num_rep", 2021-02-10T11:14:22.982 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "int", 2021-02-10T11:14:22.982 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.982 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:14:22.982 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "3", 2021-02-10T11:14:22.982 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:22.982 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:22.982 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:22.982 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "default replica count for any created local pool", 2021-02-10T11:14:22.982 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:22.983 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.983 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.983 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.983 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num": { 2021-02-10T11:14:22.983 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "pg_num", 2021-02-10T11:14:22.983 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "int", 2021-02-10T11:14:22.983 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.983 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:14:22.983 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "128", 2021-02-10T11:14:22.983 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:22.983 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:22.983 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:22.984 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "default pg_num for any created local pool", 2021-02-10T11:14:22.984 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:22.984 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.984 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.984 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.984 INFO:teuthology.orchestra.run.gibba012.stdout: "prefix": { 2021-02-10T11:14:22.984 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "prefix", 2021-02-10T11:14:22.984 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:22.984 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.984 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:14:22.984 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "", 2021-02-10T11:14:22.984 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:22.985 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:22.985 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:22.985 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "name prefix for any created local pool", 2021-02-10T11:14:22.985 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:22.985 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.985 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.985 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.985 INFO:teuthology.orchestra.run.gibba012.stdout: "subtree": { 2021-02-10T11:14:22.985 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "subtree", 2021-02-10T11:14:22.985 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:22.985 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.985 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:14:22.986 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "rack", 2021-02-10T11:14:22.986 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:22.986 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:22.986 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:22.986 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "CRUSH level for which to create a local pool", 2021-02-10T11:14:22.986 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:22.986 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.986 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.986 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:14:22.986 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:14:22.986 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.986 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:14:22.986 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "orchestrator_cli", 2021-02-10T11:14:22.987 INFO:teuthology.orchestra.run.gibba012.stdout: "can_run": true, 2021-02-10T11:14:22.987 INFO:teuthology.orchestra.run.gibba012.stdout: "error_string": "", 2021-02-10T11:14:22.987 INFO:teuthology.orchestra.run.gibba012.stdout: "module_options": { 2021-02-10T11:14:22.987 INFO:teuthology.orchestra.run.gibba012.stdout: "orchestrator": { 2021-02-10T11:14:22.987 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "orchestrator", 2021-02-10T11:14:22.987 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:22.987 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.987 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:22.987 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "", 2021-02-10T11:14:22.987 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:22.987 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:22.987 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:22.987 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:22.988 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:22.988 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.988 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.988 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:14:22.988 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:14:22.988 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.988 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:14:22.988 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "pg_autoscaler", 2021-02-10T11:14:22.988 INFO:teuthology.orchestra.run.gibba012.stdout: "can_run": true, 2021-02-10T11:14:22.988 INFO:teuthology.orchestra.run.gibba012.stdout: "error_string": "", 2021-02-10T11:14:22.988 INFO:teuthology.orchestra.run.gibba012.stdout: "module_options": { 2021-02-10T11:14:22.988 INFO:teuthology.orchestra.run.gibba012.stdout: "sleep_interval": { 2021-02-10T11:14:22.988 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "sleep_interval", 2021-02-10T11:14:22.989 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:22.989 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.989 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:22.989 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "60", 2021-02-10T11:14:22.989 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:22.989 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:22.989 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:22.989 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:22.989 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:22.989 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.989 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.989 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:14:22.990 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:14:22.990 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.990 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:14:22.990 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "progress", 2021-02-10T11:14:22.990 INFO:teuthology.orchestra.run.gibba012.stdout: "can_run": true, 2021-02-10T11:14:22.990 INFO:teuthology.orchestra.run.gibba012.stdout: "error_string": "", 2021-02-10T11:14:22.990 INFO:teuthology.orchestra.run.gibba012.stdout: "module_options": { 2021-02-10T11:14:22.990 INFO:teuthology.orchestra.run.gibba012.stdout: "enabled": { 2021-02-10T11:14:22.990 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "enabled", 2021-02-10T11:14:22.990 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "bool", 2021-02-10T11:14:22.990 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.991 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:22.991 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "True", 2021-02-10T11:14:22.991 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:22.991 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:22.991 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:22.991 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:22.991 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:22.991 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.991 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.991 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.991 INFO:teuthology.orchestra.run.gibba012.stdout: "max_completed_events": { 2021-02-10T11:14:22.991 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "max_completed_events", 2021-02-10T11:14:22.991 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "int", 2021-02-10T11:14:22.992 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.992 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:14:22.992 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "50", 2021-02-10T11:14:22.992 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:22.992 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:22.992 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:22.992 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "number of past completed events to remember", 2021-02-10T11:14:22.992 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:22.992 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.992 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.992 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.992 INFO:teuthology.orchestra.run.gibba012.stdout: "persist_interval": { 2021-02-10T11:14:22.992 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "persist_interval", 2021-02-10T11:14:22.993 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "secs", 2021-02-10T11:14:22.993 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.993 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:14:22.993 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "5", 2021-02-10T11:14:22.993 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:22.993 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:22.993 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:22.993 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "how frequently to persist completed events", 2021-02-10T11:14:22.993 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:22.993 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.993 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.993 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:14:22.993 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:14:22.994 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.994 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:14:22.994 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "prometheus", 2021-02-10T11:14:22.994 INFO:teuthology.orchestra.run.gibba012.stdout: "can_run": true, 2021-02-10T11:14:22.994 INFO:teuthology.orchestra.run.gibba012.stdout: "error_string": "", 2021-02-10T11:14:22.994 INFO:teuthology.orchestra.run.gibba012.stdout: "module_options": { 2021-02-10T11:14:22.994 INFO:teuthology.orchestra.run.gibba012.stdout: "rbd_stats_pools": { 2021-02-10T11:14:22.994 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "rbd_stats_pools", 2021-02-10T11:14:22.994 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:22.994 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.994 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:22.995 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "", 2021-02-10T11:14:22.995 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:22.995 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:22.995 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:22.995 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:22.995 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:22.995 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.995 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.995 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.995 INFO:teuthology.orchestra.run.gibba012.stdout: "rbd_stats_pools_refresh_interval": { 2021-02-10T11:14:22.995 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "rbd_stats_pools_refresh_interval", 2021-02-10T11:14:22.995 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "int", 2021-02-10T11:14:22.996 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.996 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:22.996 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "300", 2021-02-10T11:14:22.996 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:22.996 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:22.996 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:22.996 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:22.996 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:22.996 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.996 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.996 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.996 INFO:teuthology.orchestra.run.gibba012.stdout: "scrape_interval": { 2021-02-10T11:14:22.996 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "scrape_interval", 2021-02-10T11:14:22.997 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:22.997 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.997 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:22.997 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "", 2021-02-10T11:14:22.997 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:22.997 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:22.997 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:22.997 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:22.997 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:22.997 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.997 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.997 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.997 INFO:teuthology.orchestra.run.gibba012.stdout: "server_addr": { 2021-02-10T11:14:22.998 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "server_addr", 2021-02-10T11:14:22.998 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:22.998 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.998 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:22.998 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "", 2021-02-10T11:14:22.998 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:22.998 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:22.998 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:22.998 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:22.998 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:22.998 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:22.998 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:22.999 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:22.999 INFO:teuthology.orchestra.run.gibba012.stdout: "server_port": { 2021-02-10T11:14:22.999 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "server_port", 2021-02-10T11:14:22.999 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:22.999 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:22.999 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:22.999 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "", 2021-02-10T11:14:22.999 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:22.999 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:22.999 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:22.999 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:22.999 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:22.999 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.000 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.000 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.000 INFO:teuthology.orchestra.run.gibba012.stdout: "stale_cache_strategy": { 2021-02-10T11:14:23.000 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "stale_cache_strategy", 2021-02-10T11:14:23.000 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:23.000 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.000 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.000 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "", 2021-02-10T11:14:23.000 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.000 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.000 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.000 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.000 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.001 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.001 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.001 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:14:23.001 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:14:23.001 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.001 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:14:23.001 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "rbd_support", 2021-02-10T11:14:23.001 INFO:teuthology.orchestra.run.gibba012.stdout: "can_run": true, 2021-02-10T11:14:23.001 INFO:teuthology.orchestra.run.gibba012.stdout: "error_string": "", 2021-02-10T11:14:23.001 INFO:teuthology.orchestra.run.gibba012.stdout: "module_options": {} 2021-02-10T11:14:23.001 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.001 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:14:23.001 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "restful", 2021-02-10T11:14:23.002 INFO:teuthology.orchestra.run.gibba012.stdout: "can_run": true, 2021-02-10T11:14:23.002 INFO:teuthology.orchestra.run.gibba012.stdout: "error_string": "", 2021-02-10T11:14:23.002 INFO:teuthology.orchestra.run.gibba012.stdout: "module_options": { 2021-02-10T11:14:23.002 INFO:teuthology.orchestra.run.gibba012.stdout: "key_file": { 2021-02-10T11:14:23.002 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "key_file", 2021-02-10T11:14:23.002 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:23.002 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.002 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.002 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "", 2021-02-10T11:14:23.002 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.002 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.002 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.003 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.003 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.003 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.003 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.003 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.003 INFO:teuthology.orchestra.run.gibba012.stdout: "server_addr": { 2021-02-10T11:14:23.003 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "server_addr", 2021-02-10T11:14:23.003 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:23.003 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.003 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.003 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "", 2021-02-10T11:14:23.003 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.003 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.004 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.004 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.004 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.004 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.004 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.004 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.004 INFO:teuthology.orchestra.run.gibba012.stdout: "server_port": { 2021-02-10T11:14:23.004 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "server_port", 2021-02-10T11:14:23.004 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:23.004 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.004 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.005 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "", 2021-02-10T11:14:23.005 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.005 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.005 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.005 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.005 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.005 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.005 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.005 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:14:23.005 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:14:23.005 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.005 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:14:23.005 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "rook", 2021-02-10T11:14:23.006 INFO:teuthology.orchestra.run.gibba012.stdout: "can_run": true, 2021-02-10T11:14:23.006 INFO:teuthology.orchestra.run.gibba012.stdout: "error_string": "", 2021-02-10T11:14:23.006 INFO:teuthology.orchestra.run.gibba012.stdout: "module_options": {} 2021-02-10T11:14:23.006 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.006 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:14:23.006 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "selftest", 2021-02-10T11:14:23.006 INFO:teuthology.orchestra.run.gibba012.stdout: "can_run": true, 2021-02-10T11:14:23.006 INFO:teuthology.orchestra.run.gibba012.stdout: "error_string": "", 2021-02-10T11:14:23.006 INFO:teuthology.orchestra.run.gibba012.stdout: "module_options": { 2021-02-10T11:14:23.006 INFO:teuthology.orchestra.run.gibba012.stdout: "roption1": { 2021-02-10T11:14:23.006 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "roption1", 2021-02-10T11:14:23.007 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:23.007 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.007 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.007 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "", 2021-02-10T11:14:23.007 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.007 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.007 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.007 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.007 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.007 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.007 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.007 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.007 INFO:teuthology.orchestra.run.gibba012.stdout: "roption2": { 2021-02-10T11:14:23.008 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "roption2", 2021-02-10T11:14:23.008 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:23.008 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.008 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.008 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "xyz", 2021-02-10T11:14:23.008 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.008 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.008 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.008 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.008 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.008 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.008 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.008 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.009 INFO:teuthology.orchestra.run.gibba012.stdout: "rwoption1": { 2021-02-10T11:14:23.009 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "rwoption1", 2021-02-10T11:14:23.009 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:23.009 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.009 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.009 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "", 2021-02-10T11:14:23.009 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.009 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.009 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.009 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.009 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.009 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.009 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.010 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.010 INFO:teuthology.orchestra.run.gibba012.stdout: "rwoption2": { 2021-02-10T11:14:23.010 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "rwoption2", 2021-02-10T11:14:23.010 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "int", 2021-02-10T11:14:23.010 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.010 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.010 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "", 2021-02-10T11:14:23.010 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.010 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.010 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.010 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.010 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.010 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.011 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.011 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.011 INFO:teuthology.orchestra.run.gibba012.stdout: "rwoption3": { 2021-02-10T11:14:23.011 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "rwoption3", 2021-02-10T11:14:23.011 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "float", 2021-02-10T11:14:23.011 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.011 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.011 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "", 2021-02-10T11:14:23.011 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.011 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.011 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.011 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.011 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.012 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.012 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.012 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.012 INFO:teuthology.orchestra.run.gibba012.stdout: "rwoption4": { 2021-02-10T11:14:23.012 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "rwoption4", 2021-02-10T11:14:23.012 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:23.012 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.012 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.012 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "", 2021-02-10T11:14:23.012 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.012 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.012 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.012 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.013 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.013 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.013 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.013 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.013 INFO:teuthology.orchestra.run.gibba012.stdout: "rwoption5": { 2021-02-10T11:14:23.013 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "rwoption5", 2021-02-10T11:14:23.013 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "bool", 2021-02-10T11:14:23.013 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.013 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.013 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "", 2021-02-10T11:14:23.013 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.013 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.013 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.014 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.014 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.014 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.014 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.014 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.014 INFO:teuthology.orchestra.run.gibba012.stdout: "rwoption6": { 2021-02-10T11:14:23.014 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "rwoption6", 2021-02-10T11:14:23.014 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "bool", 2021-02-10T11:14:23.014 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.014 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.014 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "True", 2021-02-10T11:14:23.014 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.015 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.015 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.015 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.015 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.015 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.015 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.015 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.015 INFO:teuthology.orchestra.run.gibba012.stdout: "testkey": { 2021-02-10T11:14:23.015 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "testkey", 2021-02-10T11:14:23.015 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:23.015 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.015 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.015 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "", 2021-02-10T11:14:23.016 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.016 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.016 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.016 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.016 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.016 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.016 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.016 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.016 INFO:teuthology.orchestra.run.gibba012.stdout: "testlkey": { 2021-02-10T11:14:23.016 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "testlkey", 2021-02-10T11:14:23.016 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:23.016 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.016 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.017 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "", 2021-02-10T11:14:23.017 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.017 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.017 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.017 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.017 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.017 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.017 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.017 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.017 INFO:teuthology.orchestra.run.gibba012.stdout: "testnewline": { 2021-02-10T11:14:23.017 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "testnewline", 2021-02-10T11:14:23.017 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:23.018 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.018 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.018 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "", 2021-02-10T11:14:23.018 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.018 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.018 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.018 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.018 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.018 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.018 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.018 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:14:23.018 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:14:23.018 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.019 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:14:23.019 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "ssh", 2021-02-10T11:14:23.019 INFO:teuthology.orchestra.run.gibba012.stdout: "can_run": true, 2021-02-10T11:14:23.019 INFO:teuthology.orchestra.run.gibba012.stdout: "error_string": "", 2021-02-10T11:14:23.019 INFO:teuthology.orchestra.run.gibba012.stdout: "module_options": { 2021-02-10T11:14:23.019 INFO:teuthology.orchestra.run.gibba012.stdout: "inventory_cache_timeout_min": { 2021-02-10T11:14:23.019 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "inventory_cache_timeout_min", 2021-02-10T11:14:23.019 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:23.019 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.019 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.019 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "", 2021-02-10T11:14:23.020 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.020 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.020 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.020 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.020 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.020 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.020 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.020 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.020 INFO:teuthology.orchestra.run.gibba012.stdout: "ssh_config_file": { 2021-02-10T11:14:23.020 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "ssh_config_file", 2021-02-10T11:14:23.020 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:23.020 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.021 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.021 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "", 2021-02-10T11:14:23.021 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.021 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.021 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.021 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.021 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.021 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.021 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.021 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:14:23.021 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:14:23.021 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.022 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:14:23.022 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "status", 2021-02-10T11:14:23.022 INFO:teuthology.orchestra.run.gibba012.stdout: "can_run": true, 2021-02-10T11:14:23.022 INFO:teuthology.orchestra.run.gibba012.stdout: "error_string": "", 2021-02-10T11:14:23.022 INFO:teuthology.orchestra.run.gibba012.stdout: "module_options": {} 2021-02-10T11:14:23.022 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.022 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:14:23.022 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "telegraf", 2021-02-10T11:14:23.022 INFO:teuthology.orchestra.run.gibba012.stdout: "can_run": true, 2021-02-10T11:14:23.022 INFO:teuthology.orchestra.run.gibba012.stdout: "error_string": "", 2021-02-10T11:14:23.022 INFO:teuthology.orchestra.run.gibba012.stdout: "module_options": { 2021-02-10T11:14:23.022 INFO:teuthology.orchestra.run.gibba012.stdout: "address": { 2021-02-10T11:14:23.022 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "address", 2021-02-10T11:14:23.023 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:23.023 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.023 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.023 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2021-02-10T11:14:23.023 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.023 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.023 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.023 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.023 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.023 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.023 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.023 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.024 INFO:teuthology.orchestra.run.gibba012.stdout: "interval": { 2021-02-10T11:14:23.024 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "interval", 2021-02-10T11:14:23.024 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "secs", 2021-02-10T11:14:23.024 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.024 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.024 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "15", 2021-02-10T11:14:23.024 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.024 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.024 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.024 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.024 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.024 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.024 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.025 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:14:23.025 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:14:23.025 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.025 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:14:23.025 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "telemetry", 2021-02-10T11:14:23.025 INFO:teuthology.orchestra.run.gibba012.stdout: "can_run": true, 2021-02-10T11:14:23.025 INFO:teuthology.orchestra.run.gibba012.stdout: "error_string": "", 2021-02-10T11:14:23.025 INFO:teuthology.orchestra.run.gibba012.stdout: "module_options": { 2021-02-10T11:14:23.025 INFO:teuthology.orchestra.run.gibba012.stdout: "channel_basic": { 2021-02-10T11:14:23.025 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "channel_basic", 2021-02-10T11:14:23.025 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "bool", 2021-02-10T11:14:23.025 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.026 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.026 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "True", 2021-02-10T11:14:23.026 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.026 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.026 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.026 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "Share basic cluster information (size, version)", 2021-02-10T11:14:23.026 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.026 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.026 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.026 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.026 INFO:teuthology.orchestra.run.gibba012.stdout: "channel_crash": { 2021-02-10T11:14:23.026 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "channel_crash", 2021-02-10T11:14:23.027 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "bool", 2021-02-10T11:14:23.027 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.027 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.027 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "True", 2021-02-10T11:14:23.027 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.027 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.027 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.027 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.027 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.027 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.027 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.027 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.027 INFO:teuthology.orchestra.run.gibba012.stdout: "channel_device": { 2021-02-10T11:14:23.028 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "channel_device", 2021-02-10T11:14:23.028 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "bool", 2021-02-10T11:14:23.028 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.028 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.028 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "True", 2021-02-10T11:14:23.028 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.028 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.028 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.028 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.028 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.028 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.028 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.029 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.029 INFO:teuthology.orchestra.run.gibba012.stdout: "channel_ident": { 2021-02-10T11:14:23.029 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "channel_ident", 2021-02-10T11:14:23.029 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "bool", 2021-02-10T11:14:23.029 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.029 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.029 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "False", 2021-02-10T11:14:23.029 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.029 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.029 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.029 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.029 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.029 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.030 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.030 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.030 INFO:teuthology.orchestra.run.gibba012.stdout: "contact": { 2021-02-10T11:14:23.030 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "contact", 2021-02-10T11:14:23.030 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:23.030 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.030 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.030 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "None", 2021-02-10T11:14:23.030 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.030 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.030 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.030 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.030 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.031 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.031 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.031 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.031 INFO:teuthology.orchestra.run.gibba012.stdout: "description": { 2021-02-10T11:14:23.031 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "description", 2021-02-10T11:14:23.031 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:23.031 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.031 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.031 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "None", 2021-02-10T11:14:23.031 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.031 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.031 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.031 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.032 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.032 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.032 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.032 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.032 INFO:teuthology.orchestra.run.gibba012.stdout: "device_url": { 2021-02-10T11:14:23.032 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "device_url", 2021-02-10T11:14:23.032 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:23.032 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.032 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.032 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "https://telemetry.ceph.com/device", 2021-02-10T11:14:23.032 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.032 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.032 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.033 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.033 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.033 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.033 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.033 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.033 INFO:teuthology.orchestra.run.gibba012.stdout: "enabled": { 2021-02-10T11:14:23.033 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "enabled", 2021-02-10T11:14:23.033 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "bool", 2021-02-10T11:14:23.033 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.033 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.033 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "False", 2021-02-10T11:14:23.033 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.034 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.034 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.034 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.034 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.034 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.034 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.034 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.034 INFO:teuthology.orchestra.run.gibba012.stdout: "interval": { 2021-02-10T11:14:23.034 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "interval", 2021-02-10T11:14:23.034 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "int", 2021-02-10T11:14:23.034 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.034 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.035 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "24", 2021-02-10T11:14:23.035 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "8", 2021-02-10T11:14:23.035 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.035 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.035 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.035 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.035 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.035 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.035 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.035 INFO:teuthology.orchestra.run.gibba012.stdout: "last_opt_revision": { 2021-02-10T11:14:23.035 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "last_opt_revision", 2021-02-10T11:14:23.035 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "int", 2021-02-10T11:14:23.035 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.036 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.036 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "1", 2021-02-10T11:14:23.036 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.036 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.036 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.036 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.036 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.036 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.036 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.036 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.036 INFO:teuthology.orchestra.run.gibba012.stdout: "leaderboard": { 2021-02-10T11:14:23.036 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "leaderboard", 2021-02-10T11:14:23.036 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "bool", 2021-02-10T11:14:23.037 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.037 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.037 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "False", 2021-02-10T11:14:23.037 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.037 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.037 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.037 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.037 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.037 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.037 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.037 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.037 INFO:teuthology.orchestra.run.gibba012.stdout: "organization": { 2021-02-10T11:14:23.038 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "organization", 2021-02-10T11:14:23.038 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:23.038 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.038 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.038 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "None", 2021-02-10T11:14:23.038 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.038 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.038 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.038 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.038 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.038 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.038 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.038 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.039 INFO:teuthology.orchestra.run.gibba012.stdout: "proxy": { 2021-02-10T11:14:23.039 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "proxy", 2021-02-10T11:14:23.039 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:23.039 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.039 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.039 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "None", 2021-02-10T11:14:23.039 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.039 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.039 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.039 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.039 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.040 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.040 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.040 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.040 INFO:teuthology.orchestra.run.gibba012.stdout: "url": { 2021-02-10T11:14:23.040 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "url", 2021-02-10T11:14:23.040 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:23.040 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.040 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.040 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "https://telemetry.ceph.com/report", 2021-02-10T11:14:23.040 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.040 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.040 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.040 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.041 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.041 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.041 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.041 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:14:23.041 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:14:23.041 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.041 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:14:23.041 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "test_orchestrator", 2021-02-10T11:14:23.041 INFO:teuthology.orchestra.run.gibba012.stdout: "can_run": true, 2021-02-10T11:14:23.041 INFO:teuthology.orchestra.run.gibba012.stdout: "error_string": "", 2021-02-10T11:14:23.041 INFO:teuthology.orchestra.run.gibba012.stdout: "module_options": {} 2021-02-10T11:14:23.041 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.041 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:14:23.042 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "volumes", 2021-02-10T11:14:23.042 INFO:teuthology.orchestra.run.gibba012.stdout: "can_run": true, 2021-02-10T11:14:23.042 INFO:teuthology.orchestra.run.gibba012.stdout: "error_string": "", 2021-02-10T11:14:23.042 INFO:teuthology.orchestra.run.gibba012.stdout: "module_options": { 2021-02-10T11:14:23.042 INFO:teuthology.orchestra.run.gibba012.stdout: "max_concurrent_clones": { 2021-02-10T11:14:23.042 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "max_concurrent_clones", 2021-02-10T11:14:23.042 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "int", 2021-02-10T11:14:23.042 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.042 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.042 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "4", 2021-02-10T11:14:23.042 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.043 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.043 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.043 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "Number of asynchronous cloner threads", 2021-02-10T11:14:23.043 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.043 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.043 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.043 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:14:23.043 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:14:23.043 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.043 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:14:23.043 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "zabbix", 2021-02-10T11:14:23.043 INFO:teuthology.orchestra.run.gibba012.stdout: "can_run": true, 2021-02-10T11:14:23.043 INFO:teuthology.orchestra.run.gibba012.stdout: "error_string": "", 2021-02-10T11:14:23.044 INFO:teuthology.orchestra.run.gibba012.stdout: "module_options": { 2021-02-10T11:14:23.044 INFO:teuthology.orchestra.run.gibba012.stdout: "identifier": { 2021-02-10T11:14:23.044 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "identifier", 2021-02-10T11:14:23.044 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:23.044 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.044 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.044 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "", 2021-02-10T11:14:23.044 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.044 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.044 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.044 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.044 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.044 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.045 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.045 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.045 INFO:teuthology.orchestra.run.gibba012.stdout: "interval": { 2021-02-10T11:14:23.045 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "interval", 2021-02-10T11:14:23.045 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "secs", 2021-02-10T11:14:23.045 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.045 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.045 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "60", 2021-02-10T11:14:23.045 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.045 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.045 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.045 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.046 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.046 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.046 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.046 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.046 INFO:teuthology.orchestra.run.gibba012.stdout: "zabbix_host": { 2021-02-10T11:14:23.046 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "zabbix_host", 2021-02-10T11:14:23.046 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:23.046 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.046 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.046 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "None", 2021-02-10T11:14:23.046 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.046 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.046 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.047 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.047 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.047 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.047 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.047 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.047 INFO:teuthology.orchestra.run.gibba012.stdout: "zabbix_port": { 2021-02-10T11:14:23.047 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "zabbix_port", 2021-02-10T11:14:23.047 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "int", 2021-02-10T11:14:23.047 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.047 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.047 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "10051", 2021-02-10T11:14:23.047 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.047 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.048 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.048 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.048 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.048 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.048 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.048 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.048 INFO:teuthology.orchestra.run.gibba012.stdout: "zabbix_sender": { 2021-02-10T11:14:23.048 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "zabbix_sender", 2021-02-10T11:14:23.048 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:23.048 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.048 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.048 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "/usr/bin/zabbix_sender", 2021-02-10T11:14:23.049 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.049 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.049 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.049 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.049 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.049 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.049 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.049 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:14:23.049 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:14:23.049 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:14:23.049 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:14:23.049 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:14:23.050 INFO:teuthology.orchestra.run.gibba012.stdout: ], 2021-02-10T11:14:23.050 INFO:teuthology.orchestra.run.gibba012.stdout: "modules": [ 2021-02-10T11:14:23.050 INFO:teuthology.orchestra.run.gibba012.stdout: "iostat", 2021-02-10T11:14:23.050 INFO:teuthology.orchestra.run.gibba012.stdout: "restful" 2021-02-10T11:14:23.050 INFO:teuthology.orchestra.run.gibba012.stdout: ], 2021-02-10T11:14:23.050 INFO:teuthology.orchestra.run.gibba012.stdout: "available_modules": [ 2021-02-10T11:14:23.050 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:14:23.050 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "alerts", 2021-02-10T11:14:23.050 INFO:teuthology.orchestra.run.gibba012.stdout: "can_run": true, 2021-02-10T11:14:23.050 INFO:teuthology.orchestra.run.gibba012.stdout: "error_string": "", 2021-02-10T11:14:23.050 INFO:teuthology.orchestra.run.gibba012.stdout: "module_options": { 2021-02-10T11:14:23.050 INFO:teuthology.orchestra.run.gibba012.stdout: "interval": { 2021-02-10T11:14:23.050 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "interval", 2021-02-10T11:14:23.051 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "secs", 2021-02-10T11:14:23.051 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.051 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:14:23.051 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "60", 2021-02-10T11:14:23.051 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.051 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.051 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.051 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "How frequently to reexamine health status", 2021-02-10T11:14:23.051 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.051 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.051 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.051 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.052 INFO:teuthology.orchestra.run.gibba012.stdout: "smtp_destination": { 2021-02-10T11:14:23.052 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "smtp_destination", 2021-02-10T11:14:23.052 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:23.052 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.052 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:14:23.052 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "", 2021-02-10T11:14:23.052 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.052 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.052 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.052 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "Email address to send alerts to", 2021-02-10T11:14:23.052 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.052 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.052 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.053 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.053 INFO:teuthology.orchestra.run.gibba012.stdout: "smtp_from_name": { 2021-02-10T11:14:23.053 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "smtp_from_name", 2021-02-10T11:14:23.053 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:23.053 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.053 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:14:23.053 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "Ceph", 2021-02-10T11:14:23.053 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.053 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.053 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.053 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "Email From: name", 2021-02-10T11:14:23.053 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.053 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.054 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.054 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.054 INFO:teuthology.orchestra.run.gibba012.stdout: "smtp_host": { 2021-02-10T11:14:23.054 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "smtp_host", 2021-02-10T11:14:23.054 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:23.054 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.054 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:14:23.054 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "", 2021-02-10T11:14:23.054 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.054 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.054 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.054 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "SMTP server", 2021-02-10T11:14:23.055 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.055 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.055 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.055 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.055 INFO:teuthology.orchestra.run.gibba012.stdout: "smtp_password": { 2021-02-10T11:14:23.055 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "smtp_password", 2021-02-10T11:14:23.055 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:23.055 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.055 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:14:23.055 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "", 2021-02-10T11:14:23.055 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.055 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.055 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.056 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "Password to authenticate with", 2021-02-10T11:14:23.056 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.056 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.056 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.056 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.056 INFO:teuthology.orchestra.run.gibba012.stdout: "smtp_port": { 2021-02-10T11:14:23.056 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "smtp_port", 2021-02-10T11:14:23.056 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "int", 2021-02-10T11:14:23.056 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.056 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:14:23.056 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "465", 2021-02-10T11:14:23.056 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.056 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.057 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.057 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "SMTP port", 2021-02-10T11:14:23.057 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.057 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.057 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.057 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.057 INFO:teuthology.orchestra.run.gibba012.stdout: "smtp_sender": { 2021-02-10T11:14:23.057 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "smtp_sender", 2021-02-10T11:14:23.057 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:23.057 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.057 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:14:23.057 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "", 2021-02-10T11:14:23.058 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.058 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.058 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.058 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "SMTP envelope sender", 2021-02-10T11:14:23.058 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.058 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.058 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.058 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.058 INFO:teuthology.orchestra.run.gibba012.stdout: "smtp_ssl": { 2021-02-10T11:14:23.058 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "smtp_ssl", 2021-02-10T11:14:23.058 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "bool", 2021-02-10T11:14:23.059 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.059 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:14:23.059 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "True", 2021-02-10T11:14:23.059 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.059 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.059 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.059 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "Use SSL to connect to SMTP server", 2021-02-10T11:14:23.059 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.059 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.059 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.059 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.059 INFO:teuthology.orchestra.run.gibba012.stdout: "smtp_user": { 2021-02-10T11:14:23.060 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "smtp_user", 2021-02-10T11:14:23.060 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:23.060 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.060 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:14:23.060 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "", 2021-02-10T11:14:23.060 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.060 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.060 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.060 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "User to authenticate as", 2021-02-10T11:14:23.060 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.060 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.060 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.060 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:14:23.061 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:14:23.061 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.061 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:14:23.061 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "ansible", 2021-02-10T11:14:23.061 INFO:teuthology.orchestra.run.gibba012.stdout: "can_run": true, 2021-02-10T11:14:23.061 INFO:teuthology.orchestra.run.gibba012.stdout: "error_string": "", 2021-02-10T11:14:23.061 INFO:teuthology.orchestra.run.gibba012.stdout: "module_options": { 2021-02-10T11:14:23.061 INFO:teuthology.orchestra.run.gibba012.stdout: "password": { 2021-02-10T11:14:23.061 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "password", 2021-02-10T11:14:23.061 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:23.061 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.061 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.061 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "", 2021-02-10T11:14:23.062 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.062 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.062 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.062 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.062 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.062 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.062 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.062 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.062 INFO:teuthology.orchestra.run.gibba012.stdout: "server_url": { 2021-02-10T11:14:23.062 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "server_url", 2021-02-10T11:14:23.062 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:23.062 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.063 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.063 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "", 2021-02-10T11:14:23.063 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.063 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.063 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.063 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.063 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.063 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.063 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.063 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.063 INFO:teuthology.orchestra.run.gibba012.stdout: "username": { 2021-02-10T11:14:23.063 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "username", 2021-02-10T11:14:23.064 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:23.064 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.064 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.064 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "", 2021-02-10T11:14:23.064 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.064 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.064 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.064 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.064 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.064 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.064 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.064 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.064 INFO:teuthology.orchestra.run.gibba012.stdout: "verify_server": { 2021-02-10T11:14:23.065 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "verify_server", 2021-02-10T11:14:23.065 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:23.065 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.065 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.065 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "", 2021-02-10T11:14:23.065 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.065 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.065 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.065 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.065 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.065 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.066 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.066 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:14:23.066 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:14:23.066 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.066 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:14:23.066 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "balancer", 2021-02-10T11:14:23.066 INFO:teuthology.orchestra.run.gibba012.stdout: "can_run": true, 2021-02-10T11:14:23.066 INFO:teuthology.orchestra.run.gibba012.stdout: "error_string": "", 2021-02-10T11:14:23.066 INFO:teuthology.orchestra.run.gibba012.stdout: "module_options": { 2021-02-10T11:14:23.066 INFO:teuthology.orchestra.run.gibba012.stdout: "active": { 2021-02-10T11:14:23.066 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "active", 2021-02-10T11:14:23.067 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "bool", 2021-02-10T11:14:23.067 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.067 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:14:23.067 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "False", 2021-02-10T11:14:23.067 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.067 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.067 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.067 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "automatically balance PGs across cluster", 2021-02-10T11:14:23.067 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.067 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.067 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.067 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.068 INFO:teuthology.orchestra.run.gibba012.stdout: "begin_time": { 2021-02-10T11:14:23.068 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "begin_time", 2021-02-10T11:14:23.068 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:23.068 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.068 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:14:23.068 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "0000", 2021-02-10T11:14:23.068 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.068 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.068 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.068 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "beginning time of day to automatically balance", 2021-02-10T11:14:23.068 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "This is a time of day in the format HHMM.", 2021-02-10T11:14:23.068 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.068 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.069 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.069 INFO:teuthology.orchestra.run.gibba012.stdout: "begin_weekday": { 2021-02-10T11:14:23.069 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "begin_weekday", 2021-02-10T11:14:23.069 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "uint", 2021-02-10T11:14:23.069 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.069 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:14:23.069 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "0", 2021-02-10T11:14:23.069 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "0", 2021-02-10T11:14:23.069 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "7", 2021-02-10T11:14:23.069 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.069 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2021-02-10T11:14:23.069 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "0 or 7 = Sunday, 1 = Monday, etc.", 2021-02-10T11:14:23.070 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.070 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.070 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.070 INFO:teuthology.orchestra.run.gibba012.stdout: "crush_compat_max_iterations": { 2021-02-10T11:14:23.070 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "crush_compat_max_iterations", 2021-02-10T11:14:23.070 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "uint", 2021-02-10T11:14:23.070 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.070 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:14:23.070 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "25", 2021-02-10T11:14:23.070 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "1", 2021-02-10T11:14:23.070 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "250", 2021-02-10T11:14:23.070 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.071 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "maximum number of iterations to attempt optimization", 2021-02-10T11:14:23.071 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.071 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.071 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.071 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.071 INFO:teuthology.orchestra.run.gibba012.stdout: "crush_compat_metrics": { 2021-02-10T11:14:23.071 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "crush_compat_metrics", 2021-02-10T11:14:23.071 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:23.071 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.071 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:14:23.071 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "pgs,objects,bytes", 2021-02-10T11:14:23.071 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.071 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.072 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.072 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "metrics with which to calculate OSD utilization", 2021-02-10T11:14:23.072 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2021-02-10T11:14:23.072 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.072 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.072 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.072 INFO:teuthology.orchestra.run.gibba012.stdout: "crush_compat_step": { 2021-02-10T11:14:23.072 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "crush_compat_step", 2021-02-10T11:14:23.072 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "float", 2021-02-10T11:14:23.072 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.072 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:14:23.073 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "0.5", 2021-02-10T11:14:23.073 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "0.001", 2021-02-10T11:14:23.073 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "0.999", 2021-02-10T11:14:23.073 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.073 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "aggressiveness of optimization", 2021-02-10T11:14:23.073 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2021-02-10T11:14:23.073 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.073 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.073 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.073 INFO:teuthology.orchestra.run.gibba012.stdout: "end_time": { 2021-02-10T11:14:23.073 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "end_time", 2021-02-10T11:14:23.073 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:23.074 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.074 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:14:23.074 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "2400", 2021-02-10T11:14:23.074 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.074 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.074 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.074 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "ending time of day to automatically balance", 2021-02-10T11:14:23.074 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "This is a time of day in the format HHMM.", 2021-02-10T11:14:23.074 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.074 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.074 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.074 INFO:teuthology.orchestra.run.gibba012.stdout: "end_weekday": { 2021-02-10T11:14:23.074 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "end_weekday", 2021-02-10T11:14:23.075 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "uint", 2021-02-10T11:14:23.075 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.075 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:14:23.075 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "7", 2021-02-10T11:14:23.075 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "0", 2021-02-10T11:14:23.075 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "7", 2021-02-10T11:14:23.075 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.075 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2021-02-10T11:14:23.075 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "0 or 7 = Sunday, 1 = Monday, etc.", 2021-02-10T11:14:23.075 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.075 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.075 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.076 INFO:teuthology.orchestra.run.gibba012.stdout: "min_score": { 2021-02-10T11:14:23.076 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "min_score", 2021-02-10T11:14:23.076 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "float", 2021-02-10T11:14:23.076 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.076 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:14:23.076 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "0", 2021-02-10T11:14:23.076 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.076 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.076 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.076 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "minimum score, below which no optimization is attempted", 2021-02-10T11:14:23.076 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.076 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.076 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.077 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.077 INFO:teuthology.orchestra.run.gibba012.stdout: "mode": { 2021-02-10T11:14:23.077 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "mode", 2021-02-10T11:14:23.077 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:23.077 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.077 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:14:23.077 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "none", 2021-02-10T11:14:23.077 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.077 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.077 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [ 2021-02-10T11:14:23.077 INFO:teuthology.orchestra.run.gibba012.stdout: "crush-compat", 2021-02-10T11:14:23.077 INFO:teuthology.orchestra.run.gibba012.stdout: "none", 2021-02-10T11:14:23.077 INFO:teuthology.orchestra.run.gibba012.stdout: "upmap" 2021-02-10T11:14:23.078 INFO:teuthology.orchestra.run.gibba012.stdout: ], 2021-02-10T11:14:23.078 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "Balancer mode", 2021-02-10T11:14:23.078 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.078 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.078 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.078 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.078 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_ids": { 2021-02-10T11:14:23.078 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "pool_ids", 2021-02-10T11:14:23.078 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:23.078 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.078 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:14:23.078 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "", 2021-02-10T11:14:23.078 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.079 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.079 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.079 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "pools which the automatic balancing will be limited to", 2021-02-10T11:14:23.079 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.079 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.079 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.079 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.079 INFO:teuthology.orchestra.run.gibba012.stdout: "sleep_interval": { 2021-02-10T11:14:23.079 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "sleep_interval", 2021-02-10T11:14:23.079 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "secs", 2021-02-10T11:14:23.079 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.079 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:14:23.079 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "60", 2021-02-10T11:14:23.080 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.080 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.080 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.080 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "how frequently to wake up and attempt optimization", 2021-02-10T11:14:23.080 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.080 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.080 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.080 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.080 INFO:teuthology.orchestra.run.gibba012.stdout: "upmap_max_deviation": { 2021-02-10T11:14:23.080 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "upmap_max_deviation", 2021-02-10T11:14:23.080 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "int", 2021-02-10T11:14:23.081 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.081 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:14:23.081 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "5", 2021-02-10T11:14:23.081 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "1", 2021-02-10T11:14:23.081 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.081 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.081 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "deviation below which no optimization is attempted", 2021-02-10T11:14:23.081 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2021-02-10T11:14:23.081 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.081 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.081 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.081 INFO:teuthology.orchestra.run.gibba012.stdout: "upmap_max_iterations": { 2021-02-10T11:14:23.081 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "upmap_max_iterations", 2021-02-10T11:14:23.082 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "uint", 2021-02-10T11:14:23.082 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.082 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:14:23.082 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "10", 2021-02-10T11:14:23.082 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.082 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.082 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.082 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "maximum upmap optimization iterations", 2021-02-10T11:14:23.082 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.082 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.082 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.082 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:14:23.082 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:14:23.083 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.083 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:14:23.083 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "crash", 2021-02-10T11:14:23.083 INFO:teuthology.orchestra.run.gibba012.stdout: "can_run": true, 2021-02-10T11:14:23.083 INFO:teuthology.orchestra.run.gibba012.stdout: "error_string": "", 2021-02-10T11:14:23.083 INFO:teuthology.orchestra.run.gibba012.stdout: "module_options": { 2021-02-10T11:14:23.083 INFO:teuthology.orchestra.run.gibba012.stdout: "retain_interval": { 2021-02-10T11:14:23.083 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "retain_interval", 2021-02-10T11:14:23.083 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "secs", 2021-02-10T11:14:23.083 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.083 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:14:23.083 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "31536000", 2021-02-10T11:14:23.084 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.084 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.084 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.084 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "how long to retain crashes before pruning them", 2021-02-10T11:14:23.084 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.084 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.084 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.084 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.084 INFO:teuthology.orchestra.run.gibba012.stdout: "warn_recent_interval": { 2021-02-10T11:14:23.084 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "warn_recent_interval", 2021-02-10T11:14:23.084 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "secs", 2021-02-10T11:14:23.084 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.084 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:14:23.085 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "1209600", 2021-02-10T11:14:23.085 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.085 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.085 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.085 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "time interval in which to warn about recent crashes", 2021-02-10T11:14:23.085 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.085 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.085 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.085 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:14:23.085 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:14:23.085 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.085 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:14:23.085 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "dashboard", 2021-02-10T11:14:23.086 INFO:teuthology.orchestra.run.gibba012.stdout: "can_run": true, 2021-02-10T11:14:23.086 INFO:teuthology.orchestra.run.gibba012.stdout: "error_string": "", 2021-02-10T11:14:23.086 INFO:teuthology.orchestra.run.gibba012.stdout: "module_options": { 2021-02-10T11:14:23.086 INFO:teuthology.orchestra.run.gibba012.stdout: "ALERTMANAGER_API_HOST": { 2021-02-10T11:14:23.086 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "ALERTMANAGER_API_HOST", 2021-02-10T11:14:23.086 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:23.086 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.086 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.086 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "", 2021-02-10T11:14:23.086 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.086 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.086 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.087 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.087 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.087 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.087 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.087 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.087 INFO:teuthology.orchestra.run.gibba012.stdout: "AUDIT_API_ENABLED": { 2021-02-10T11:14:23.087 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "AUDIT_API_ENABLED", 2021-02-10T11:14:23.087 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "bool", 2021-02-10T11:14:23.087 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.087 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.087 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "False", 2021-02-10T11:14:23.087 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.088 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.088 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.088 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.088 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.088 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.088 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.088 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.088 INFO:teuthology.orchestra.run.gibba012.stdout: "AUDIT_API_LOG_PAYLOAD": { 2021-02-10T11:14:23.088 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2021-02-10T11:14:23.089 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "bool", 2021-02-10T11:14:23.089 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.089 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.089 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "True", 2021-02-10T11:14:23.089 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.089 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.089 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.089 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.089 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.089 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.089 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.090 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.090 INFO:teuthology.orchestra.run.gibba012.stdout: "ENABLE_BROWSABLE_API": { 2021-02-10T11:14:23.090 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "ENABLE_BROWSABLE_API", 2021-02-10T11:14:23.090 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "bool", 2021-02-10T11:14:23.090 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.090 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.090 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "True", 2021-02-10T11:14:23.090 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.091 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.091 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.091 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.091 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.091 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.091 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.091 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.091 INFO:teuthology.orchestra.run.gibba012.stdout: "FEATURE_TOGGLE_cephfs": { 2021-02-10T11:14:23.091 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "FEATURE_TOGGLE_cephfs", 2021-02-10T11:14:23.091 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "bool", 2021-02-10T11:14:23.091 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.091 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.092 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "True", 2021-02-10T11:14:23.092 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.092 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.092 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.092 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.092 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.092 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.092 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.092 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.092 INFO:teuthology.orchestra.run.gibba012.stdout: "FEATURE_TOGGLE_iscsi": { 2021-02-10T11:14:23.092 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "FEATURE_TOGGLE_iscsi", 2021-02-10T11:14:23.093 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "bool", 2021-02-10T11:14:23.093 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.093 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.093 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "True", 2021-02-10T11:14:23.093 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.093 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.093 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.093 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.093 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.093 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.093 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.094 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.094 INFO:teuthology.orchestra.run.gibba012.stdout: "FEATURE_TOGGLE_mirroring": { 2021-02-10T11:14:23.094 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "FEATURE_TOGGLE_mirroring", 2021-02-10T11:14:23.094 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "bool", 2021-02-10T11:14:23.094 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.094 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.094 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "True", 2021-02-10T11:14:23.094 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.094 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.094 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.094 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.094 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.095 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.095 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.095 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.095 INFO:teuthology.orchestra.run.gibba012.stdout: "FEATURE_TOGGLE_rbd": { 2021-02-10T11:14:23.095 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "FEATURE_TOGGLE_rbd", 2021-02-10T11:14:23.095 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "bool", 2021-02-10T11:14:23.095 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.095 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.095 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "True", 2021-02-10T11:14:23.095 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.095 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.095 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.095 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.096 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.096 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.096 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.096 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.096 INFO:teuthology.orchestra.run.gibba012.stdout: "FEATURE_TOGGLE_rgw": { 2021-02-10T11:14:23.096 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "FEATURE_TOGGLE_rgw", 2021-02-10T11:14:23.096 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "bool", 2021-02-10T11:14:23.096 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.096 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.096 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "True", 2021-02-10T11:14:23.096 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.096 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.096 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.097 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.097 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.097 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.097 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.097 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.097 INFO:teuthology.orchestra.run.gibba012.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2021-02-10T11:14:23.097 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2021-02-10T11:14:23.097 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:23.097 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.097 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.097 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "", 2021-02-10T11:14:23.097 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.098 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.098 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.098 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.098 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.098 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.098 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.098 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.098 INFO:teuthology.orchestra.run.gibba012.stdout: "GRAFANA_API_PASSWORD": { 2021-02-10T11:14:23.098 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "GRAFANA_API_PASSWORD", 2021-02-10T11:14:23.098 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:23.098 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.098 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.099 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "admin", 2021-02-10T11:14:23.099 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.099 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.099 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.099 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.099 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.099 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.099 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.099 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.099 INFO:teuthology.orchestra.run.gibba012.stdout: "GRAFANA_API_SSL_VERIFY": { 2021-02-10T11:14:23.099 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2021-02-10T11:14:23.099 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "bool", 2021-02-10T11:14:23.099 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.100 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.100 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "True", 2021-02-10T11:14:23.100 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.100 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.100 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.100 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.100 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.100 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.100 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.100 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.100 INFO:teuthology.orchestra.run.gibba012.stdout: "GRAFANA_API_URL": { 2021-02-10T11:14:23.100 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "GRAFANA_API_URL", 2021-02-10T11:14:23.100 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:23.101 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.101 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.101 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "", 2021-02-10T11:14:23.101 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.101 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.101 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.101 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.101 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.101 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.101 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.101 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.101 INFO:teuthology.orchestra.run.gibba012.stdout: "GRAFANA_API_USERNAME": { 2021-02-10T11:14:23.101 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "GRAFANA_API_USERNAME", 2021-02-10T11:14:23.102 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:23.102 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.102 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.102 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "admin", 2021-02-10T11:14:23.102 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.102 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.102 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.102 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.102 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.102 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.102 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.103 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.103 INFO:teuthology.orchestra.run.gibba012.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2021-02-10T11:14:23.103 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2021-02-10T11:14:23.103 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "bool", 2021-02-10T11:14:23.103 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.103 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.103 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "False", 2021-02-10T11:14:23.103 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.103 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.103 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.103 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.103 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.103 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.104 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.104 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.104 INFO:teuthology.orchestra.run.gibba012.stdout: "ISCSI_API_SSL_VERIFICATION": { 2021-02-10T11:14:23.104 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2021-02-10T11:14:23.104 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "bool", 2021-02-10T11:14:23.104 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.104 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.104 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "True", 2021-02-10T11:14:23.104 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.104 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.104 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.104 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.104 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.105 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.105 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.105 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.105 INFO:teuthology.orchestra.run.gibba012.stdout: "PROMETHEUS_API_HOST": { 2021-02-10T11:14:23.105 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "PROMETHEUS_API_HOST", 2021-02-10T11:14:23.105 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:23.105 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.105 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.105 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "", 2021-02-10T11:14:23.105 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.105 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.105 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.106 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.106 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.106 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.106 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.106 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.106 INFO:teuthology.orchestra.run.gibba012.stdout: "REST_REQUESTS_TIMEOUT": { 2021-02-10T11:14:23.106 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "REST_REQUESTS_TIMEOUT", 2021-02-10T11:14:23.106 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "int", 2021-02-10T11:14:23.106 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.106 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.106 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "45", 2021-02-10T11:14:23.106 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.106 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.107 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.107 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.107 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.107 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.107 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.107 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.107 INFO:teuthology.orchestra.run.gibba012.stdout: "RGW_API_ACCESS_KEY": { 2021-02-10T11:14:23.107 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "RGW_API_ACCESS_KEY", 2021-02-10T11:14:23.107 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:23.107 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.107 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.107 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "", 2021-02-10T11:14:23.107 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.108 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.108 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.108 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.108 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.108 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.108 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.108 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.108 INFO:teuthology.orchestra.run.gibba012.stdout: "RGW_API_ADMIN_RESOURCE": { 2021-02-10T11:14:23.108 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2021-02-10T11:14:23.108 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:23.108 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.108 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.108 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "admin", 2021-02-10T11:14:23.109 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.109 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.109 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.109 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.109 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.109 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.109 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.109 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.109 INFO:teuthology.orchestra.run.gibba012.stdout: "RGW_API_HOST": { 2021-02-10T11:14:23.109 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "RGW_API_HOST", 2021-02-10T11:14:23.109 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:23.109 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.109 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.110 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "", 2021-02-10T11:14:23.110 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.110 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.110 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.110 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.110 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.110 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.110 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.110 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.110 INFO:teuthology.orchestra.run.gibba012.stdout: "RGW_API_PORT": { 2021-02-10T11:14:23.110 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "RGW_API_PORT", 2021-02-10T11:14:23.110 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "int", 2021-02-10T11:14:23.111 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.111 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.111 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "80", 2021-02-10T11:14:23.111 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.111 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.111 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.111 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.111 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.111 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.111 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.111 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.111 INFO:teuthology.orchestra.run.gibba012.stdout: "RGW_API_SCHEME": { 2021-02-10T11:14:23.111 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "RGW_API_SCHEME", 2021-02-10T11:14:23.112 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:23.112 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.112 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.112 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "http", 2021-02-10T11:14:23.112 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.112 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.112 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.112 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.112 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.112 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.112 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.112 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.112 INFO:teuthology.orchestra.run.gibba012.stdout: "RGW_API_SECRET_KEY": { 2021-02-10T11:14:23.113 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "RGW_API_SECRET_KEY", 2021-02-10T11:14:23.113 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:23.113 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.113 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.113 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "", 2021-02-10T11:14:23.113 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.113 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.113 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.113 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.113 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.113 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.113 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.113 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.114 INFO:teuthology.orchestra.run.gibba012.stdout: "RGW_API_SSL_VERIFY": { 2021-02-10T11:14:23.114 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "RGW_API_SSL_VERIFY", 2021-02-10T11:14:23.114 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "bool", 2021-02-10T11:14:23.114 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.114 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.114 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "True", 2021-02-10T11:14:23.114 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.114 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.114 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.114 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.114 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.114 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.115 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.115 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.115 INFO:teuthology.orchestra.run.gibba012.stdout: "RGW_API_USER_ID": { 2021-02-10T11:14:23.115 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "RGW_API_USER_ID", 2021-02-10T11:14:23.115 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:23.115 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.115 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.115 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "", 2021-02-10T11:14:23.116 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.116 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.116 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.116 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.116 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.116 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.116 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.116 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.116 INFO:teuthology.orchestra.run.gibba012.stdout: "crt_file": { 2021-02-10T11:14:23.116 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "crt_file", 2021-02-10T11:14:23.116 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:23.116 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.116 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.117 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "", 2021-02-10T11:14:23.117 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.117 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.117 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.117 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.117 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.117 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.117 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.117 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.117 INFO:teuthology.orchestra.run.gibba012.stdout: "debug": { 2021-02-10T11:14:23.117 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "debug", 2021-02-10T11:14:23.117 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "bool", 2021-02-10T11:14:23.117 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.118 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.118 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "False", 2021-02-10T11:14:23.118 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.118 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.118 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.118 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "Enable/disable debug options", 2021-02-10T11:14:23.118 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.118 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.118 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.118 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.118 INFO:teuthology.orchestra.run.gibba012.stdout: "jwt_token_ttl": { 2021-02-10T11:14:23.118 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "jwt_token_ttl", 2021-02-10T11:14:23.118 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "int", 2021-02-10T11:14:23.119 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.119 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.119 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "28800", 2021-02-10T11:14:23.119 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.119 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.119 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.119 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.119 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.119 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.119 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.119 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.119 INFO:teuthology.orchestra.run.gibba012.stdout: "key_file": { 2021-02-10T11:14:23.119 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "key_file", 2021-02-10T11:14:23.120 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:23.120 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.120 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.120 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "", 2021-02-10T11:14:23.120 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.120 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.120 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.120 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.120 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.120 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.120 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.120 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.120 INFO:teuthology.orchestra.run.gibba012.stdout: "password": { 2021-02-10T11:14:23.121 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "password", 2021-02-10T11:14:23.121 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:23.121 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.121 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.121 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "", 2021-02-10T11:14:23.121 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.121 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.121 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.121 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.121 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.121 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.121 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.121 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.122 INFO:teuthology.orchestra.run.gibba012.stdout: "server_addr": { 2021-02-10T11:14:23.122 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "server_addr", 2021-02-10T11:14:23.122 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:23.122 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.122 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.122 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "::", 2021-02-10T11:14:23.122 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.122 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.122 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.122 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.122 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.122 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.122 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.123 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.123 INFO:teuthology.orchestra.run.gibba012.stdout: "server_port": { 2021-02-10T11:14:23.123 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "server_port", 2021-02-10T11:14:23.123 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "int", 2021-02-10T11:14:23.123 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.123 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.123 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "8080", 2021-02-10T11:14:23.123 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.123 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.123 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.123 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.123 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.123 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.124 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.124 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.124 INFO:teuthology.orchestra.run.gibba012.stdout: "ssl": { 2021-02-10T11:14:23.124 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "ssl", 2021-02-10T11:14:23.124 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "bool", 2021-02-10T11:14:23.124 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.124 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.124 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "True", 2021-02-10T11:14:23.124 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.124 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.124 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.124 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.124 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.124 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.125 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.125 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.125 INFO:teuthology.orchestra.run.gibba012.stdout: "ssl_server_port": { 2021-02-10T11:14:23.125 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "ssl_server_port", 2021-02-10T11:14:23.125 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "int", 2021-02-10T11:14:23.125 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.125 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.125 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "8443", 2021-02-10T11:14:23.125 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.125 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.125 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.125 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.125 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.126 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.126 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.126 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.126 INFO:teuthology.orchestra.run.gibba012.stdout: "standby_behaviour": { 2021-02-10T11:14:23.126 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "standby_behaviour", 2021-02-10T11:14:23.126 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:23.126 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.126 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.126 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "redirect", 2021-02-10T11:14:23.126 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.126 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.126 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [ 2021-02-10T11:14:23.126 INFO:teuthology.orchestra.run.gibba012.stdout: "error", 2021-02-10T11:14:23.127 INFO:teuthology.orchestra.run.gibba012.stdout: "redirect" 2021-02-10T11:14:23.127 INFO:teuthology.orchestra.run.gibba012.stdout: ], 2021-02-10T11:14:23.127 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.127 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.127 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.127 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.127 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.127 INFO:teuthology.orchestra.run.gibba012.stdout: "standby_error_status_code": { 2021-02-10T11:14:23.127 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "standby_error_status_code", 2021-02-10T11:14:23.127 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "int", 2021-02-10T11:14:23.127 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.127 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.127 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "500", 2021-02-10T11:14:23.128 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "400", 2021-02-10T11:14:23.128 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "599", 2021-02-10T11:14:23.128 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.128 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.128 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.128 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.128 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.128 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.128 INFO:teuthology.orchestra.run.gibba012.stdout: "url_prefix": { 2021-02-10T11:14:23.128 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "url_prefix", 2021-02-10T11:14:23.128 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:23.128 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.128 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.129 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "", 2021-02-10T11:14:23.130 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.130 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.130 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.130 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.130 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.130 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.130 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.130 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.130 INFO:teuthology.orchestra.run.gibba012.stdout: "username": { 2021-02-10T11:14:23.130 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "username", 2021-02-10T11:14:23.130 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:23.130 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.131 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.131 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "", 2021-02-10T11:14:23.131 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.131 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.131 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.131 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.131 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.131 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.131 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.131 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:14:23.131 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:14:23.131 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.132 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:14:23.132 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "deepsea", 2021-02-10T11:14:23.132 INFO:teuthology.orchestra.run.gibba012.stdout: "can_run": true, 2021-02-10T11:14:23.132 INFO:teuthology.orchestra.run.gibba012.stdout: "error_string": "", 2021-02-10T11:14:23.132 INFO:teuthology.orchestra.run.gibba012.stdout: "module_options": { 2021-02-10T11:14:23.132 INFO:teuthology.orchestra.run.gibba012.stdout: "salt_api_eauth": { 2021-02-10T11:14:23.132 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "salt_api_eauth", 2021-02-10T11:14:23.132 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:23.132 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.134 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.134 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "sharedsecret", 2021-02-10T11:14:23.134 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.134 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.134 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.134 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.134 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.135 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.135 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.136 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.136 INFO:teuthology.orchestra.run.gibba012.stdout: "salt_api_password": { 2021-02-10T11:14:23.136 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "salt_api_password", 2021-02-10T11:14:23.136 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:23.136 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.136 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.136 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "", 2021-02-10T11:14:23.136 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.136 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.136 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.136 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.136 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.137 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.137 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.137 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.137 INFO:teuthology.orchestra.run.gibba012.stdout: "salt_api_url": { 2021-02-10T11:14:23.137 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "salt_api_url", 2021-02-10T11:14:23.137 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:23.137 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.137 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.137 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "", 2021-02-10T11:14:23.137 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.137 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.137 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.137 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.137 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.138 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.138 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.138 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.138 INFO:teuthology.orchestra.run.gibba012.stdout: "salt_api_username": { 2021-02-10T11:14:23.139 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "salt_api_username", 2021-02-10T11:14:23.139 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:23.139 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.139 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.139 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "", 2021-02-10T11:14:23.139 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.139 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.139 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.139 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.140 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.140 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.140 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.140 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:14:23.140 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:14:23.140 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.140 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:14:23.140 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "devicehealth", 2021-02-10T11:14:23.140 INFO:teuthology.orchestra.run.gibba012.stdout: "can_run": true, 2021-02-10T11:14:23.140 INFO:teuthology.orchestra.run.gibba012.stdout: "error_string": "", 2021-02-10T11:14:23.140 INFO:teuthology.orchestra.run.gibba012.stdout: "module_options": { 2021-02-10T11:14:23.140 INFO:teuthology.orchestra.run.gibba012.stdout: "enable_monitoring": { 2021-02-10T11:14:23.141 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "enable_monitoring", 2021-02-10T11:14:23.141 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "bool", 2021-02-10T11:14:23.141 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.141 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:14:23.141 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "False", 2021-02-10T11:14:23.141 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.141 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.141 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.141 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "monitor device health metrics", 2021-02-10T11:14:23.141 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.141 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.141 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.142 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.142 INFO:teuthology.orchestra.run.gibba012.stdout: "mark_out_threshold": { 2021-02-10T11:14:23.142 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "mark_out_threshold", 2021-02-10T11:14:23.142 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "secs", 2021-02-10T11:14:23.142 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.142 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:14:23.142 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "2419200", 2021-02-10T11:14:23.142 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.142 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.142 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.142 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "automatically mark OSD if it may fail before this long", 2021-02-10T11:14:23.142 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.143 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.143 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.143 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.143 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_name": { 2021-02-10T11:14:23.143 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "pool_name", 2021-02-10T11:14:23.143 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:23.143 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.143 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:14:23.143 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "device_health_metrics", 2021-02-10T11:14:23.143 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.143 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.143 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.144 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "name of pool in which to store device health metrics", 2021-02-10T11:14:23.144 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.144 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.144 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.144 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.144 INFO:teuthology.orchestra.run.gibba012.stdout: "retention_period": { 2021-02-10T11:14:23.144 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "retention_period", 2021-02-10T11:14:23.144 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "secs", 2021-02-10T11:14:23.144 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.145 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:14:23.146 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "15552000", 2021-02-10T11:14:23.146 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.146 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.146 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.146 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "how long to retain device health metrics", 2021-02-10T11:14:23.146 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.147 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.147 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.147 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.147 INFO:teuthology.orchestra.run.gibba012.stdout: "scrape_frequency": { 2021-02-10T11:14:23.147 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "scrape_frequency", 2021-02-10T11:14:23.148 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "secs", 2021-02-10T11:14:23.148 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.148 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:14:23.148 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "86400", 2021-02-10T11:14:23.148 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.148 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.149 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.149 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "how frequently to scrape device health metrics", 2021-02-10T11:14:23.149 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.149 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.149 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.149 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.150 INFO:teuthology.orchestra.run.gibba012.stdout: "self_heal": { 2021-02-10T11:14:23.150 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "self_heal", 2021-02-10T11:14:23.150 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "bool", 2021-02-10T11:14:23.150 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.150 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:14:23.151 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "True", 2021-02-10T11:14:23.151 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.151 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.151 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.151 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "preemptively heal cluster around devices that may fail", 2021-02-10T11:14:23.152 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.152 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.152 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.152 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.152 INFO:teuthology.orchestra.run.gibba012.stdout: "sleep_interval": { 2021-02-10T11:14:23.152 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "sleep_interval", 2021-02-10T11:14:23.153 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "secs", 2021-02-10T11:14:23.153 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.153 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:14:23.153 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "600", 2021-02-10T11:14:23.153 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.153 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.154 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.154 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "how frequently to wake up and check device health", 2021-02-10T11:14:23.154 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.154 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.154 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.155 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.155 INFO:teuthology.orchestra.run.gibba012.stdout: "warn_threshold": { 2021-02-10T11:14:23.155 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "warn_threshold", 2021-02-10T11:14:23.155 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "secs", 2021-02-10T11:14:23.155 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.155 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:14:23.156 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "7257600", 2021-02-10T11:14:23.156 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.156 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.156 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.156 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "raise health warning if OSD may fail before this long", 2021-02-10T11:14:23.157 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.157 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.157 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.157 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:14:23.157 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:14:23.157 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.158 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:14:23.158 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "diskprediction_cloud", 2021-02-10T11:14:23.158 INFO:teuthology.orchestra.run.gibba012.stdout: "can_run": true, 2021-02-10T11:14:23.158 INFO:teuthology.orchestra.run.gibba012.stdout: "error_string": "", 2021-02-10T11:14:23.158 INFO:teuthology.orchestra.run.gibba012.stdout: "module_options": { 2021-02-10T11:14:23.159 INFO:teuthology.orchestra.run.gibba012.stdout: "diskprediction_cert_context": { 2021-02-10T11:14:23.159 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "diskprediction_cert_context", 2021-02-10T11:14:23.159 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:23.159 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.160 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.160 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "", 2021-02-10T11:14:23.160 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.160 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.160 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.160 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.161 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.161 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.161 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.162 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.163 INFO:teuthology.orchestra.run.gibba012.stdout: "diskprediction_default_authority": { 2021-02-10T11:14:23.163 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "diskprediction_default_authority", 2021-02-10T11:14:23.163 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:23.163 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.163 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.163 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "localhost", 2021-02-10T11:14:23.163 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.163 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.163 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.163 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.164 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.164 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.164 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.164 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.164 INFO:teuthology.orchestra.run.gibba012.stdout: "diskprediction_password": { 2021-02-10T11:14:23.164 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "diskprediction_password", 2021-02-10T11:14:23.164 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:23.164 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.164 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.164 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "", 2021-02-10T11:14:23.164 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.165 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.165 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.165 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.165 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.165 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.166 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.166 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.166 INFO:teuthology.orchestra.run.gibba012.stdout: "diskprediction_port": { 2021-02-10T11:14:23.166 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "diskprediction_port", 2021-02-10T11:14:23.166 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:23.166 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.166 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.166 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "31400", 2021-02-10T11:14:23.166 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.167 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.167 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.167 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.167 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.167 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.167 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.167 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.167 INFO:teuthology.orchestra.run.gibba012.stdout: "diskprediction_retrieve_prediction_interval": { 2021-02-10T11:14:23.167 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "diskprediction_retrieve_prediction_interval", 2021-02-10T11:14:23.167 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:23.167 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.167 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.168 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "43200", 2021-02-10T11:14:23.168 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.168 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.168 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.168 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.168 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.168 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.168 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.168 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.168 INFO:teuthology.orchestra.run.gibba012.stdout: "diskprediction_server": { 2021-02-10T11:14:23.168 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "diskprediction_server", 2021-02-10T11:14:23.168 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:23.168 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.169 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.169 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "", 2021-02-10T11:14:23.169 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.169 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.169 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.169 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.169 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.169 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.169 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.169 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.169 INFO:teuthology.orchestra.run.gibba012.stdout: "diskprediction_ssl_target_name_override": { 2021-02-10T11:14:23.169 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "diskprediction_ssl_target_name_override", 2021-02-10T11:14:23.169 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:23.170 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.170 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.170 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "localhost", 2021-02-10T11:14:23.170 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.170 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.170 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.170 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.170 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.170 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.170 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.170 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.170 INFO:teuthology.orchestra.run.gibba012.stdout: "diskprediction_upload_metrics_interval": { 2021-02-10T11:14:23.171 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "diskprediction_upload_metrics_interval", 2021-02-10T11:14:23.171 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:23.171 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.171 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.171 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "600", 2021-02-10T11:14:23.171 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.171 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.171 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.171 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.171 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.171 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.171 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.171 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.171 INFO:teuthology.orchestra.run.gibba012.stdout: "diskprediction_upload_smart_interval": { 2021-02-10T11:14:23.172 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "diskprediction_upload_smart_interval", 2021-02-10T11:14:23.172 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:23.172 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.172 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.172 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "43200", 2021-02-10T11:14:23.172 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.172 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.172 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.172 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.172 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.172 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.172 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.172 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.173 INFO:teuthology.orchestra.run.gibba012.stdout: "diskprediction_user": { 2021-02-10T11:14:23.173 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "diskprediction_user", 2021-02-10T11:14:23.173 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:23.173 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.173 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.173 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "", 2021-02-10T11:14:23.173 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.173 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.173 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.173 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.173 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.173 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.173 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.174 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.174 INFO:teuthology.orchestra.run.gibba012.stdout: "sleep_interval": { 2021-02-10T11:14:23.174 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "sleep_interval", 2021-02-10T11:14:23.174 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:23.174 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.174 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.174 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "600", 2021-02-10T11:14:23.174 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.174 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.174 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.174 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.174 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.175 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.175 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.175 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:14:23.175 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:14:23.175 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.175 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:14:23.175 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "diskprediction_local", 2021-02-10T11:14:23.175 INFO:teuthology.orchestra.run.gibba012.stdout: "can_run": true, 2021-02-10T11:14:23.175 INFO:teuthology.orchestra.run.gibba012.stdout: "error_string": "", 2021-02-10T11:14:23.175 INFO:teuthology.orchestra.run.gibba012.stdout: "module_options": { 2021-02-10T11:14:23.175 INFO:teuthology.orchestra.run.gibba012.stdout: "predict_interval": { 2021-02-10T11:14:23.175 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "predict_interval", 2021-02-10T11:14:23.175 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:23.176 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.176 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.176 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "86400", 2021-02-10T11:14:23.176 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.176 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.176 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.176 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.176 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.176 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.176 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.176 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.176 INFO:teuthology.orchestra.run.gibba012.stdout: "sleep_interval": { 2021-02-10T11:14:23.177 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "sleep_interval", 2021-02-10T11:14:23.177 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:23.177 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.177 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.177 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "600", 2021-02-10T11:14:23.177 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.178 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.178 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.178 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.178 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.178 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.178 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.178 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:14:23.178 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:14:23.178 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.178 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:14:23.178 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "influx", 2021-02-10T11:14:23.178 INFO:teuthology.orchestra.run.gibba012.stdout: "can_run": false, 2021-02-10T11:14:23.178 INFO:teuthology.orchestra.run.gibba012.stdout: "error_string": "influxdb python module not found", 2021-02-10T11:14:23.179 INFO:teuthology.orchestra.run.gibba012.stdout: "module_options": { 2021-02-10T11:14:23.179 INFO:teuthology.orchestra.run.gibba012.stdout: "batch_size": { 2021-02-10T11:14:23.179 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "batch_size", 2021-02-10T11:14:23.179 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:23.179 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.179 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.179 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "5000", 2021-02-10T11:14:23.179 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.179 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.179 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.179 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.179 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.179 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.179 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.180 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.180 INFO:teuthology.orchestra.run.gibba012.stdout: "database": { 2021-02-10T11:14:23.180 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "database", 2021-02-10T11:14:23.180 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:23.180 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.180 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.180 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "ceph", 2021-02-10T11:14:23.180 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.180 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.180 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.180 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.180 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.180 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.181 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.181 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.181 INFO:teuthology.orchestra.run.gibba012.stdout: "hostname": { 2021-02-10T11:14:23.181 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "hostname", 2021-02-10T11:14:23.181 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:23.181 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.181 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.181 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "None", 2021-02-10T11:14:23.181 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.181 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.181 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.181 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.182 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.182 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.182 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.182 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.182 INFO:teuthology.orchestra.run.gibba012.stdout: "interval": { 2021-02-10T11:14:23.182 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "interval", 2021-02-10T11:14:23.182 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:23.182 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.182 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.182 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "30", 2021-02-10T11:14:23.182 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.182 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.182 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.183 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.183 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.183 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.183 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.183 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.183 INFO:teuthology.orchestra.run.gibba012.stdout: "password": { 2021-02-10T11:14:23.183 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "password", 2021-02-10T11:14:23.183 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:23.183 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.183 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.183 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "None", 2021-02-10T11:14:23.183 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.183 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.184 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.184 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.184 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.184 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.184 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.184 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.184 INFO:teuthology.orchestra.run.gibba012.stdout: "port": { 2021-02-10T11:14:23.184 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "port", 2021-02-10T11:14:23.184 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:23.184 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.184 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.184 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "8086", 2021-02-10T11:14:23.184 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.185 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.185 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.185 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.185 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.185 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.185 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.185 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.185 INFO:teuthology.orchestra.run.gibba012.stdout: "ssl": { 2021-02-10T11:14:23.185 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "ssl", 2021-02-10T11:14:23.185 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:23.185 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.185 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.185 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "false", 2021-02-10T11:14:23.186 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.186 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.186 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.186 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.186 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.186 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.186 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.186 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.186 INFO:teuthology.orchestra.run.gibba012.stdout: "threads": { 2021-02-10T11:14:23.186 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "threads", 2021-02-10T11:14:23.186 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:23.186 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.186 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.187 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "5", 2021-02-10T11:14:23.187 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.187 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.187 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.187 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.187 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.187 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.187 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.187 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.187 INFO:teuthology.orchestra.run.gibba012.stdout: "username": { 2021-02-10T11:14:23.187 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "username", 2021-02-10T11:14:23.187 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:23.187 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.188 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.188 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "None", 2021-02-10T11:14:23.188 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.188 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.188 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.188 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.188 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.188 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.188 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.188 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.188 INFO:teuthology.orchestra.run.gibba012.stdout: "verify_ssl": { 2021-02-10T11:14:23.188 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "verify_ssl", 2021-02-10T11:14:23.188 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:23.189 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.189 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.189 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "true", 2021-02-10T11:14:23.189 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.189 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.189 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.189 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.189 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.189 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.189 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.189 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:14:23.189 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:14:23.189 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.189 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:14:23.190 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "insights", 2021-02-10T11:14:23.190 INFO:teuthology.orchestra.run.gibba012.stdout: "can_run": true, 2021-02-10T11:14:23.190 INFO:teuthology.orchestra.run.gibba012.stdout: "error_string": "", 2021-02-10T11:14:23.190 INFO:teuthology.orchestra.run.gibba012.stdout: "module_options": {} 2021-02-10T11:14:23.190 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.190 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:14:23.190 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "iostat", 2021-02-10T11:14:23.190 INFO:teuthology.orchestra.run.gibba012.stdout: "can_run": true, 2021-02-10T11:14:23.190 INFO:teuthology.orchestra.run.gibba012.stdout: "error_string": "", 2021-02-10T11:14:23.190 INFO:teuthology.orchestra.run.gibba012.stdout: "module_options": {} 2021-02-10T11:14:23.190 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.190 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:14:23.190 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "localpool", 2021-02-10T11:14:23.191 INFO:teuthology.orchestra.run.gibba012.stdout: "can_run": true, 2021-02-10T11:14:23.191 INFO:teuthology.orchestra.run.gibba012.stdout: "error_string": "", 2021-02-10T11:14:23.191 INFO:teuthology.orchestra.run.gibba012.stdout: "module_options": { 2021-02-10T11:14:23.191 INFO:teuthology.orchestra.run.gibba012.stdout: "failure_domain": { 2021-02-10T11:14:23.191 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "failure_domain", 2021-02-10T11:14:23.191 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:23.191 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.191 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:14:23.191 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "host", 2021-02-10T11:14:23.191 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.191 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.191 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.191 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "failure domain for any created local pool", 2021-02-10T11:14:23.192 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.192 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.192 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.192 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.192 INFO:teuthology.orchestra.run.gibba012.stdout: "min_size": { 2021-02-10T11:14:23.192 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "min_size", 2021-02-10T11:14:23.192 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "int", 2021-02-10T11:14:23.192 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.192 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:14:23.192 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "", 2021-02-10T11:14:23.192 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.192 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.192 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.193 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "default min_size for any created local pool", 2021-02-10T11:14:23.193 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.193 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.193 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.193 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.193 INFO:teuthology.orchestra.run.gibba012.stdout: "num_rep": { 2021-02-10T11:14:23.193 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "num_rep", 2021-02-10T11:14:23.193 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "int", 2021-02-10T11:14:23.193 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.193 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:14:23.193 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "3", 2021-02-10T11:14:23.193 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.193 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.193 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.194 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "default replica count for any created local pool", 2021-02-10T11:14:23.194 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.194 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.194 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.194 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.194 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num": { 2021-02-10T11:14:23.194 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "pg_num", 2021-02-10T11:14:23.194 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "int", 2021-02-10T11:14:23.194 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.194 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:14:23.194 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "128", 2021-02-10T11:14:23.194 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.195 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.195 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.195 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "default pg_num for any created local pool", 2021-02-10T11:14:23.195 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.195 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.195 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.195 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.195 INFO:teuthology.orchestra.run.gibba012.stdout: "prefix": { 2021-02-10T11:14:23.195 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "prefix", 2021-02-10T11:14:23.195 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:23.195 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.195 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:14:23.195 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "", 2021-02-10T11:14:23.196 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.196 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.196 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.196 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "name prefix for any created local pool", 2021-02-10T11:14:23.196 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.196 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.196 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.196 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.196 INFO:teuthology.orchestra.run.gibba012.stdout: "subtree": { 2021-02-10T11:14:23.196 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "subtree", 2021-02-10T11:14:23.196 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:23.196 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.196 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:14:23.196 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "rack", 2021-02-10T11:14:23.197 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.197 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.197 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.197 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "CRUSH level for which to create a local pool", 2021-02-10T11:14:23.197 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.197 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.197 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.197 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:14:23.197 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:14:23.197 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.197 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:14:23.197 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "orchestrator_cli", 2021-02-10T11:14:23.198 INFO:teuthology.orchestra.run.gibba012.stdout: "can_run": true, 2021-02-10T11:14:23.198 INFO:teuthology.orchestra.run.gibba012.stdout: "error_string": "", 2021-02-10T11:14:23.198 INFO:teuthology.orchestra.run.gibba012.stdout: "module_options": { 2021-02-10T11:14:23.198 INFO:teuthology.orchestra.run.gibba012.stdout: "orchestrator": { 2021-02-10T11:14:23.198 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "orchestrator", 2021-02-10T11:14:23.198 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:23.198 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.198 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.198 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "", 2021-02-10T11:14:23.198 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.198 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.198 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.198 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.199 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.199 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.199 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.199 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:14:23.199 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:14:23.199 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.199 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:14:23.199 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "pg_autoscaler", 2021-02-10T11:14:23.199 INFO:teuthology.orchestra.run.gibba012.stdout: "can_run": true, 2021-02-10T11:14:23.199 INFO:teuthology.orchestra.run.gibba012.stdout: "error_string": "", 2021-02-10T11:14:23.199 INFO:teuthology.orchestra.run.gibba012.stdout: "module_options": { 2021-02-10T11:14:23.199 INFO:teuthology.orchestra.run.gibba012.stdout: "sleep_interval": { 2021-02-10T11:14:23.199 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "sleep_interval", 2021-02-10T11:14:23.199 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:23.200 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.200 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.200 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "60", 2021-02-10T11:14:23.200 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.200 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.200 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.200 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.200 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.200 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.200 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.200 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:14:23.200 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:14:23.200 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.201 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:14:23.201 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "progress", 2021-02-10T11:14:23.201 INFO:teuthology.orchestra.run.gibba012.stdout: "can_run": true, 2021-02-10T11:14:23.201 INFO:teuthology.orchestra.run.gibba012.stdout: "error_string": "", 2021-02-10T11:14:23.201 INFO:teuthology.orchestra.run.gibba012.stdout: "module_options": { 2021-02-10T11:14:23.201 INFO:teuthology.orchestra.run.gibba012.stdout: "enabled": { 2021-02-10T11:14:23.201 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "enabled", 2021-02-10T11:14:23.201 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "bool", 2021-02-10T11:14:23.201 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.201 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.201 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "True", 2021-02-10T11:14:23.201 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.201 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.202 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.202 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.202 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.202 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.202 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.202 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.202 INFO:teuthology.orchestra.run.gibba012.stdout: "max_completed_events": { 2021-02-10T11:14:23.202 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "max_completed_events", 2021-02-10T11:14:23.202 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "int", 2021-02-10T11:14:23.202 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.202 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:14:23.202 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "50", 2021-02-10T11:14:23.203 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.203 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.203 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.203 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "number of past completed events to remember", 2021-02-10T11:14:23.203 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.203 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.203 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.203 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.203 INFO:teuthology.orchestra.run.gibba012.stdout: "persist_interval": { 2021-02-10T11:14:23.203 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "persist_interval", 2021-02-10T11:14:23.203 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "secs", 2021-02-10T11:14:23.203 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.203 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:14:23.203 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "5", 2021-02-10T11:14:23.204 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.204 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.204 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.204 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "how frequently to persist completed events", 2021-02-10T11:14:23.204 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.204 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.204 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.204 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:14:23.204 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:14:23.204 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.204 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:14:23.204 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "prometheus", 2021-02-10T11:14:23.204 INFO:teuthology.orchestra.run.gibba012.stdout: "can_run": true, 2021-02-10T11:14:23.205 INFO:teuthology.orchestra.run.gibba012.stdout: "error_string": "", 2021-02-10T11:14:23.205 INFO:teuthology.orchestra.run.gibba012.stdout: "module_options": { 2021-02-10T11:14:23.205 INFO:teuthology.orchestra.run.gibba012.stdout: "rbd_stats_pools": { 2021-02-10T11:14:23.205 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "rbd_stats_pools", 2021-02-10T11:14:23.205 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:23.205 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.205 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.205 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "", 2021-02-10T11:14:23.205 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.205 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.205 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.205 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.205 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.206 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.206 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.206 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.206 INFO:teuthology.orchestra.run.gibba012.stdout: "rbd_stats_pools_refresh_interval": { 2021-02-10T11:14:23.206 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "rbd_stats_pools_refresh_interval", 2021-02-10T11:14:23.206 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "int", 2021-02-10T11:14:23.206 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.206 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.206 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "300", 2021-02-10T11:14:23.206 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.206 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.206 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.206 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.207 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.207 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.207 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.207 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.207 INFO:teuthology.orchestra.run.gibba012.stdout: "scrape_interval": { 2021-02-10T11:14:23.207 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "scrape_interval", 2021-02-10T11:14:23.207 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:23.207 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.207 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.207 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "", 2021-02-10T11:14:23.207 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.207 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.207 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.208 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.208 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.208 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.208 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.208 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.208 INFO:teuthology.orchestra.run.gibba012.stdout: "server_addr": { 2021-02-10T11:14:23.208 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "server_addr", 2021-02-10T11:14:23.208 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:23.208 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.208 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.208 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "", 2021-02-10T11:14:23.208 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.208 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.208 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.209 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.209 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.209 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.209 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.209 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.209 INFO:teuthology.orchestra.run.gibba012.stdout: "server_port": { 2021-02-10T11:14:23.209 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "server_port", 2021-02-10T11:14:23.209 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:23.209 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.209 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.209 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "", 2021-02-10T11:14:23.209 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.209 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.209 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.210 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.210 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.210 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.210 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.210 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.210 INFO:teuthology.orchestra.run.gibba012.stdout: "stale_cache_strategy": { 2021-02-10T11:14:23.210 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "stale_cache_strategy", 2021-02-10T11:14:23.210 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:23.210 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.210 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.210 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "", 2021-02-10T11:14:23.210 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.211 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.211 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.211 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.211 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.211 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.211 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.211 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:14:23.211 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:14:23.211 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.211 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:14:23.211 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "rbd_support", 2021-02-10T11:14:23.211 INFO:teuthology.orchestra.run.gibba012.stdout: "can_run": true, 2021-02-10T11:14:23.211 INFO:teuthology.orchestra.run.gibba012.stdout: "error_string": "", 2021-02-10T11:14:23.211 INFO:teuthology.orchestra.run.gibba012.stdout: "module_options": {} 2021-02-10T11:14:23.212 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.212 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:14:23.212 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "restful", 2021-02-10T11:14:23.212 INFO:teuthology.orchestra.run.gibba012.stdout: "can_run": true, 2021-02-10T11:14:23.212 INFO:teuthology.orchestra.run.gibba012.stdout: "error_string": "", 2021-02-10T11:14:23.212 INFO:teuthology.orchestra.run.gibba012.stdout: "module_options": { 2021-02-10T11:14:23.212 INFO:teuthology.orchestra.run.gibba012.stdout: "key_file": { 2021-02-10T11:14:23.212 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "key_file", 2021-02-10T11:14:23.212 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:23.212 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.212 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.212 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "", 2021-02-10T11:14:23.212 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.213 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.213 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.213 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.213 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.213 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.213 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.213 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.213 INFO:teuthology.orchestra.run.gibba012.stdout: "server_addr": { 2021-02-10T11:14:23.213 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "server_addr", 2021-02-10T11:14:23.213 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:23.213 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.213 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.214 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "", 2021-02-10T11:14:23.214 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.214 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.214 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.214 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.214 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.214 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.214 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.214 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.215 INFO:teuthology.orchestra.run.gibba012.stdout: "server_port": { 2021-02-10T11:14:23.215 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "server_port", 2021-02-10T11:14:23.215 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:23.215 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.215 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.215 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "", 2021-02-10T11:14:23.215 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.215 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.215 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.215 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.215 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.215 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.215 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.216 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:14:23.216 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:14:23.216 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.216 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:14:23.216 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "rook", 2021-02-10T11:14:23.216 INFO:teuthology.orchestra.run.gibba012.stdout: "can_run": true, 2021-02-10T11:14:23.216 INFO:teuthology.orchestra.run.gibba012.stdout: "error_string": "", 2021-02-10T11:14:23.216 INFO:teuthology.orchestra.run.gibba012.stdout: "module_options": {} 2021-02-10T11:14:23.216 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.216 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:14:23.216 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "selftest", 2021-02-10T11:14:23.216 INFO:teuthology.orchestra.run.gibba012.stdout: "can_run": true, 2021-02-10T11:14:23.216 INFO:teuthology.orchestra.run.gibba012.stdout: "error_string": "", 2021-02-10T11:14:23.217 INFO:teuthology.orchestra.run.gibba012.stdout: "module_options": { 2021-02-10T11:14:23.217 INFO:teuthology.orchestra.run.gibba012.stdout: "roption1": { 2021-02-10T11:14:23.217 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "roption1", 2021-02-10T11:14:23.217 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:23.217 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.217 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.217 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "", 2021-02-10T11:14:23.217 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.217 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.217 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.217 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.217 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.217 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.218 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.218 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.218 INFO:teuthology.orchestra.run.gibba012.stdout: "roption2": { 2021-02-10T11:14:23.218 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "roption2", 2021-02-10T11:14:23.218 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:23.218 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.218 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.218 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "xyz", 2021-02-10T11:14:23.218 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.218 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.218 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.218 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.219 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.219 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.219 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.219 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.219 INFO:teuthology.orchestra.run.gibba012.stdout: "rwoption1": { 2021-02-10T11:14:23.219 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "rwoption1", 2021-02-10T11:14:23.219 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:23.219 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.219 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.219 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "", 2021-02-10T11:14:23.219 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.219 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.219 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.220 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.220 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.220 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.220 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.220 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.220 INFO:teuthology.orchestra.run.gibba012.stdout: "rwoption2": { 2021-02-10T11:14:23.220 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "rwoption2", 2021-02-10T11:14:23.220 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "int", 2021-02-10T11:14:23.220 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.220 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.220 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "", 2021-02-10T11:14:23.220 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.220 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.220 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.221 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.221 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.221 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.221 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.221 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.221 INFO:teuthology.orchestra.run.gibba012.stdout: "rwoption3": { 2021-02-10T11:14:23.221 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "rwoption3", 2021-02-10T11:14:23.221 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "float", 2021-02-10T11:14:23.221 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.221 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.221 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "", 2021-02-10T11:14:23.221 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.221 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.222 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.222 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.222 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.222 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.222 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.222 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.222 INFO:teuthology.orchestra.run.gibba012.stdout: "rwoption4": { 2021-02-10T11:14:23.222 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "rwoption4", 2021-02-10T11:14:23.222 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:23.222 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.222 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.222 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "", 2021-02-10T11:14:23.223 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.223 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.223 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.223 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.223 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.223 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.223 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.223 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.223 INFO:teuthology.orchestra.run.gibba012.stdout: "rwoption5": { 2021-02-10T11:14:23.224 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "rwoption5", 2021-02-10T11:14:23.224 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "bool", 2021-02-10T11:14:23.224 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.224 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.224 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "", 2021-02-10T11:14:23.224 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.224 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.224 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.224 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.224 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.224 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.224 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.224 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.225 INFO:teuthology.orchestra.run.gibba012.stdout: "rwoption6": { 2021-02-10T11:14:23.225 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "rwoption6", 2021-02-10T11:14:23.225 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "bool", 2021-02-10T11:14:23.225 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.225 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.225 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "True", 2021-02-10T11:14:23.225 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.225 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.225 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.225 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.225 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.225 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.225 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.225 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.226 INFO:teuthology.orchestra.run.gibba012.stdout: "testkey": { 2021-02-10T11:14:23.226 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "testkey", 2021-02-10T11:14:23.226 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:23.226 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.226 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.226 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "", 2021-02-10T11:14:23.226 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.226 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.226 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.226 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.226 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.226 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.227 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.227 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.227 INFO:teuthology.orchestra.run.gibba012.stdout: "testlkey": { 2021-02-10T11:14:23.227 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "testlkey", 2021-02-10T11:14:23.227 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:23.227 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.227 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.227 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "", 2021-02-10T11:14:23.227 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.227 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.227 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.227 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.227 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.227 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.228 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.228 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.228 INFO:teuthology.orchestra.run.gibba012.stdout: "testnewline": { 2021-02-10T11:14:23.228 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "testnewline", 2021-02-10T11:14:23.228 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:23.228 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.228 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.228 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "", 2021-02-10T11:14:23.228 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.228 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.228 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.228 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.228 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.229 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.229 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.229 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:14:23.229 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:14:23.229 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.229 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:14:23.229 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "ssh", 2021-02-10T11:14:23.229 INFO:teuthology.orchestra.run.gibba012.stdout: "can_run": true, 2021-02-10T11:14:23.229 INFO:teuthology.orchestra.run.gibba012.stdout: "error_string": "", 2021-02-10T11:14:23.229 INFO:teuthology.orchestra.run.gibba012.stdout: "module_options": { 2021-02-10T11:14:23.229 INFO:teuthology.orchestra.run.gibba012.stdout: "inventory_cache_timeout_min": { 2021-02-10T11:14:23.229 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "inventory_cache_timeout_min", 2021-02-10T11:14:23.230 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:23.230 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.230 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.230 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "", 2021-02-10T11:14:23.230 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.230 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.230 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.230 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.230 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.230 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.231 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.231 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.231 INFO:teuthology.orchestra.run.gibba012.stdout: "ssh_config_file": { 2021-02-10T11:14:23.231 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "ssh_config_file", 2021-02-10T11:14:23.231 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:23.231 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.231 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.231 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "", 2021-02-10T11:14:23.231 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.231 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.231 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.231 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.232 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.232 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.232 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.232 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:14:23.232 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:14:23.232 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.232 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:14:23.232 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "status", 2021-02-10T11:14:23.232 INFO:teuthology.orchestra.run.gibba012.stdout: "can_run": true, 2021-02-10T11:14:23.232 INFO:teuthology.orchestra.run.gibba012.stdout: "error_string": "", 2021-02-10T11:14:23.232 INFO:teuthology.orchestra.run.gibba012.stdout: "module_options": {} 2021-02-10T11:14:23.232 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.232 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:14:23.232 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "telegraf", 2021-02-10T11:14:23.233 INFO:teuthology.orchestra.run.gibba012.stdout: "can_run": true, 2021-02-10T11:14:23.233 INFO:teuthology.orchestra.run.gibba012.stdout: "error_string": "", 2021-02-10T11:14:23.233 INFO:teuthology.orchestra.run.gibba012.stdout: "module_options": { 2021-02-10T11:14:23.233 INFO:teuthology.orchestra.run.gibba012.stdout: "address": { 2021-02-10T11:14:23.233 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "address", 2021-02-10T11:14:23.233 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:23.233 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.233 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.233 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2021-02-10T11:14:23.233 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.233 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.233 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.234 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.234 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.234 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.234 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.234 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.234 INFO:teuthology.orchestra.run.gibba012.stdout: "interval": { 2021-02-10T11:14:23.234 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "interval", 2021-02-10T11:14:23.234 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "secs", 2021-02-10T11:14:23.234 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.234 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.234 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "15", 2021-02-10T11:14:23.234 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.234 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.235 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.235 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.235 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.235 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.235 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.235 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:14:23.235 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:14:23.235 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.235 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:14:23.235 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "telemetry", 2021-02-10T11:14:23.235 INFO:teuthology.orchestra.run.gibba012.stdout: "can_run": true, 2021-02-10T11:14:23.235 INFO:teuthology.orchestra.run.gibba012.stdout: "error_string": "", 2021-02-10T11:14:23.235 INFO:teuthology.orchestra.run.gibba012.stdout: "module_options": { 2021-02-10T11:14:23.236 INFO:teuthology.orchestra.run.gibba012.stdout: "channel_basic": { 2021-02-10T11:14:23.236 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "channel_basic", 2021-02-10T11:14:23.236 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "bool", 2021-02-10T11:14:23.236 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.236 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.236 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "True", 2021-02-10T11:14:23.236 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.236 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.236 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.236 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "Share basic cluster information (size, version)", 2021-02-10T11:14:23.236 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.236 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.236 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.237 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.237 INFO:teuthology.orchestra.run.gibba012.stdout: "channel_crash": { 2021-02-10T11:14:23.237 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "channel_crash", 2021-02-10T11:14:23.237 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "bool", 2021-02-10T11:14:23.237 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.237 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.237 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "True", 2021-02-10T11:14:23.237 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.237 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.237 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.237 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.237 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.237 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.238 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.238 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.238 INFO:teuthology.orchestra.run.gibba012.stdout: "channel_device": { 2021-02-10T11:14:23.238 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "channel_device", 2021-02-10T11:14:23.238 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "bool", 2021-02-10T11:14:23.238 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.238 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.238 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "True", 2021-02-10T11:14:23.238 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.238 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.238 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.238 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.238 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.239 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.239 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.239 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.239 INFO:teuthology.orchestra.run.gibba012.stdout: "channel_ident": { 2021-02-10T11:14:23.239 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "channel_ident", 2021-02-10T11:14:23.239 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "bool", 2021-02-10T11:14:23.239 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.239 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.239 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "False", 2021-02-10T11:14:23.239 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.239 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.239 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.239 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.240 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.240 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.240 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.240 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.240 INFO:teuthology.orchestra.run.gibba012.stdout: "contact": { 2021-02-10T11:14:23.240 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "contact", 2021-02-10T11:14:23.240 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:23.240 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.240 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.240 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "None", 2021-02-10T11:14:23.240 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.240 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.240 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.240 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.241 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.241 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.241 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.241 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.241 INFO:teuthology.orchestra.run.gibba012.stdout: "description": { 2021-02-10T11:14:23.241 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "description", 2021-02-10T11:14:23.241 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:23.241 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.241 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.241 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "None", 2021-02-10T11:14:23.241 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.241 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.241 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.242 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.242 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.242 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.242 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.242 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.242 INFO:teuthology.orchestra.run.gibba012.stdout: "device_url": { 2021-02-10T11:14:23.242 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "device_url", 2021-02-10T11:14:23.242 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:23.242 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.242 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.242 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "https://telemetry.ceph.com/device", 2021-02-10T11:14:23.242 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.242 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.243 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.243 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.243 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.243 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.243 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.243 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.243 INFO:teuthology.orchestra.run.gibba012.stdout: "enabled": { 2021-02-10T11:14:23.243 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "enabled", 2021-02-10T11:14:23.243 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "bool", 2021-02-10T11:14:23.243 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.243 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.243 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "False", 2021-02-10T11:14:23.243 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.244 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.244 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.244 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.244 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.244 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.244 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.244 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.244 INFO:teuthology.orchestra.run.gibba012.stdout: "interval": { 2021-02-10T11:14:23.244 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "interval", 2021-02-10T11:14:23.244 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "int", 2021-02-10T11:14:23.244 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.244 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.244 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "24", 2021-02-10T11:14:23.244 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "8", 2021-02-10T11:14:23.245 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.245 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.245 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.245 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.245 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.245 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.245 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.245 INFO:teuthology.orchestra.run.gibba012.stdout: "last_opt_revision": { 2021-02-10T11:14:23.245 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "last_opt_revision", 2021-02-10T11:14:23.245 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "int", 2021-02-10T11:14:23.245 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.245 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.245 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "1", 2021-02-10T11:14:23.246 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.246 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.246 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.246 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.246 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.246 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.246 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.246 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.246 INFO:teuthology.orchestra.run.gibba012.stdout: "leaderboard": { 2021-02-10T11:14:23.246 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "leaderboard", 2021-02-10T11:14:23.246 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "bool", 2021-02-10T11:14:23.246 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.246 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.247 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "False", 2021-02-10T11:14:23.247 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.247 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.247 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.247 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.247 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.247 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.247 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.247 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.247 INFO:teuthology.orchestra.run.gibba012.stdout: "organization": { 2021-02-10T11:14:23.247 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "organization", 2021-02-10T11:14:23.247 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:23.247 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.247 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.248 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "None", 2021-02-10T11:14:23.248 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.248 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.248 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.248 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.248 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.248 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.248 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.248 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.248 INFO:teuthology.orchestra.run.gibba012.stdout: "proxy": { 2021-02-10T11:14:23.248 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "proxy", 2021-02-10T11:14:23.248 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:23.249 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.249 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.249 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "None", 2021-02-10T11:14:23.249 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.249 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.249 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.249 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.249 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.249 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.249 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.249 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.249 INFO:teuthology.orchestra.run.gibba012.stdout: "url": { 2021-02-10T11:14:23.249 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "url", 2021-02-10T11:14:23.249 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:23.250 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.250 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.250 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "https://telemetry.ceph.com/report", 2021-02-10T11:14:23.250 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.250 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.250 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.250 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.250 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.250 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.250 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.250 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:14:23.250 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:14:23.251 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.251 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:14:23.251 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "test_orchestrator", 2021-02-10T11:14:23.251 INFO:teuthology.orchestra.run.gibba012.stdout: "can_run": true, 2021-02-10T11:14:23.251 INFO:teuthology.orchestra.run.gibba012.stdout: "error_string": "", 2021-02-10T11:14:23.251 INFO:teuthology.orchestra.run.gibba012.stdout: "module_options": {} 2021-02-10T11:14:23.251 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.251 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:14:23.251 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "volumes", 2021-02-10T11:14:23.251 INFO:teuthology.orchestra.run.gibba012.stdout: "can_run": true, 2021-02-10T11:14:23.251 INFO:teuthology.orchestra.run.gibba012.stdout: "error_string": "", 2021-02-10T11:14:23.251 INFO:teuthology.orchestra.run.gibba012.stdout: "module_options": { 2021-02-10T11:14:23.251 INFO:teuthology.orchestra.run.gibba012.stdout: "max_concurrent_clones": { 2021-02-10T11:14:23.251 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "max_concurrent_clones", 2021-02-10T11:14:23.252 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "int", 2021-02-10T11:14:23.252 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.252 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.252 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "4", 2021-02-10T11:14:23.252 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.252 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.252 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.252 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "Number of asynchronous cloner threads", 2021-02-10T11:14:23.252 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.252 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.252 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.252 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:14:23.252 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:14:23.253 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.253 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:14:23.253 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "zabbix", 2021-02-10T11:14:23.253 INFO:teuthology.orchestra.run.gibba012.stdout: "can_run": true, 2021-02-10T11:14:23.253 INFO:teuthology.orchestra.run.gibba012.stdout: "error_string": "", 2021-02-10T11:14:23.253 INFO:teuthology.orchestra.run.gibba012.stdout: "module_options": { 2021-02-10T11:14:23.253 INFO:teuthology.orchestra.run.gibba012.stdout: "identifier": { 2021-02-10T11:14:23.253 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "identifier", 2021-02-10T11:14:23.253 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:23.253 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.253 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.253 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "", 2021-02-10T11:14:23.254 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.254 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.254 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.254 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.254 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.254 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.254 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.254 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.254 INFO:teuthology.orchestra.run.gibba012.stdout: "interval": { 2021-02-10T11:14:23.255 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "interval", 2021-02-10T11:14:23.255 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "secs", 2021-02-10T11:14:23.255 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.255 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.255 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "60", 2021-02-10T11:14:23.255 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.255 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.255 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.255 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.255 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.255 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.255 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.255 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.256 INFO:teuthology.orchestra.run.gibba012.stdout: "zabbix_host": { 2021-02-10T11:14:23.256 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "zabbix_host", 2021-02-10T11:14:23.256 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:23.256 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.256 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.256 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "None", 2021-02-10T11:14:23.256 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.256 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.256 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.256 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.256 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.257 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.257 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.257 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.257 INFO:teuthology.orchestra.run.gibba012.stdout: "zabbix_port": { 2021-02-10T11:14:23.257 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "zabbix_port", 2021-02-10T11:14:23.257 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "int", 2021-02-10T11:14:23.257 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.257 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.257 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "10051", 2021-02-10T11:14:23.257 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.257 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.257 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.258 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.258 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.258 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.258 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.258 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.258 INFO:teuthology.orchestra.run.gibba012.stdout: "zabbix_sender": { 2021-02-10T11:14:23.258 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "zabbix_sender", 2021-02-10T11:14:23.258 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "str", 2021-02-10T11:14:23.258 INFO:teuthology.orchestra.run.gibba012.stdout: "level": "advanced", 2021-02-10T11:14:23.258 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 0, 2021-02-10T11:14:23.258 INFO:teuthology.orchestra.run.gibba012.stdout: "default_value": "/usr/bin/zabbix_sender", 2021-02-10T11:14:23.258 INFO:teuthology.orchestra.run.gibba012.stdout: "min": "", 2021-02-10T11:14:23.259 INFO:teuthology.orchestra.run.gibba012.stdout: "max": "", 2021-02-10T11:14:23.259 INFO:teuthology.orchestra.run.gibba012.stdout: "enum_allowed": [], 2021-02-10T11:14:23.259 INFO:teuthology.orchestra.run.gibba012.stdout: "desc": "", 2021-02-10T11:14:23.259 INFO:teuthology.orchestra.run.gibba012.stdout: "long_desc": "", 2021-02-10T11:14:23.259 INFO:teuthology.orchestra.run.gibba012.stdout: "tags": [], 2021-02-10T11:14:23.259 INFO:teuthology.orchestra.run.gibba012.stdout: "see_also": [] 2021-02-10T11:14:23.259 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:14:23.259 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:14:23.259 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:14:23.260 INFO:teuthology.orchestra.run.gibba012.stdout: ], 2021-02-10T11:14:23.260 INFO:teuthology.orchestra.run.gibba012.stdout: "services": {}, 2021-02-10T11:14:23.260 INFO:teuthology.orchestra.run.gibba012.stdout: "always_on_modules": { 2021-02-10T11:14:23.260 INFO:teuthology.orchestra.run.gibba012.stdout: "nautilus": [ 2021-02-10T11:14:23.260 INFO:teuthology.orchestra.run.gibba012.stdout: "balancer", 2021-02-10T11:14:23.260 INFO:teuthology.orchestra.run.gibba012.stdout: "crash", 2021-02-10T11:14:23.260 INFO:teuthology.orchestra.run.gibba012.stdout: "devicehealth", 2021-02-10T11:14:23.260 INFO:teuthology.orchestra.run.gibba012.stdout: "orchestrator_cli", 2021-02-10T11:14:23.260 INFO:teuthology.orchestra.run.gibba012.stdout: "progress", 2021-02-10T11:14:23.260 INFO:teuthology.orchestra.run.gibba012.stdout: "rbd_support", 2021-02-10T11:14:23.260 INFO:teuthology.orchestra.run.gibba012.stdout: "status", 2021-02-10T11:14:23.260 INFO:teuthology.orchestra.run.gibba012.stdout: "volumes" 2021-02-10T11:14:23.260 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:14:23.260 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:14:23.261 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.261 INFO:teuthology.orchestra.run.gibba012.stdout: "servicemap": { 2021-02-10T11:14:23.261 INFO:teuthology.orchestra.run.gibba012.stdout: "epoch": 2, 2021-02-10T11:14:23.261 INFO:teuthology.orchestra.run.gibba012.stdout: "modified": "2021-02-10 11:14:19.765616", 2021-02-10T11:14:23.261 INFO:teuthology.orchestra.run.gibba012.stdout: "services": { 2021-02-10T11:14:23.261 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": { 2021-02-10T11:14:23.261 INFO:teuthology.orchestra.run.gibba012.stdout: "daemons": { 2021-02-10T11:14:23.261 INFO:teuthology.orchestra.run.gibba012.stdout: "summary": "", 2021-02-10T11:14:23.261 INFO:teuthology.orchestra.run.gibba012.stdout: "a": { 2021-02-10T11:14:23.261 INFO:teuthology.orchestra.run.gibba012.stdout: "start_epoch": 0, 2021-02-10T11:14:23.261 INFO:teuthology.orchestra.run.gibba012.stdout: "start_stamp": "0.000000", 2021-02-10T11:14:23.261 INFO:teuthology.orchestra.run.gibba012.stdout: "gid": 0, 2021-02-10T11:14:23.261 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": ":/0", 2021-02-10T11:14:23.262 INFO:teuthology.orchestra.run.gibba012.stdout: "metadata": {}, 2021-02-10T11:14:23.262 INFO:teuthology.orchestra.run.gibba012.stdout: "task_status": { 2021-02-10T11:14:23.262 INFO:teuthology.orchestra.run.gibba012.stdout: "scrub status": "idle" 2021-02-10T11:14:23.262 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:14:23.262 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:14:23.262 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:14:23.262 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:14:23.262 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:14:23.262 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:23.262 INFO:teuthology.orchestra.run.gibba012.stdout: "progress_events": {} 2021-02-10T11:14:23.262 INFO:teuthology.orchestra.run.gibba012.stdout:} 2021-02-10T11:14:23.265 INFO:tasks.ceph.ceph_manager.ceph:{'fsid': 'f14f004e-0699-4584-ad81-3754258493de', 'health': {'checks': {}, 'status': 'HEALTH_OK'}, 'election_epoch': 4, 'quorum': [0, 1, 2], 'quorum_names': ['a', 'b', 'c'], 'quorum_age': 56, 'monmap': {'epoch': 1, 'min_mon_release_name': '14', 'num_mons': 3}, 'osdmap': {'osdmap': {'epoch': 20, 'num_osds': 8, 'num_up_osds': 8, 'num_in_osds': 8, 'num_remapped_pgs': 0}}, 'pgmap': {'pgs_by_state': [{'state_name': 'active+clean', 'count': 24}], 'num_pgs': 24, 'num_pools': 3, 'num_objects': 22, 'data_bytes': 2286, 'bytes_used': 8591704064, 'bytes_avail': 764502409216, 'bytes_total': 773094113280, 'write_bytes_sec': 1279, 'read_op_per_sec': 0, 'write_op_per_sec': 4}, 'fsmap': {'epoch': 6, 'id': 1, 'up': 1, 'in': 1, 'max': 1, 'by_rank': [{'filesystem_id': 1, 'rank': 0, 'name': 'a', 'status': 'up:active', 'gid': 4272}], 'up:standby': 1}, 'mgrmap': {'epoch': 3, 'active_gid': 4100, 'active_name': 'x', 'active_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.120:6800', 'nonce': 34990}, {'type': 'v1', 'addr': '172.21.2.120:6801', 'nonce': 34990}]}, 'active_addr': '172.21.2.120:6801/34990', 'active_change': '2021-02-10 11:13:27.751969', 'available': True, 'standbys': [{'gid': 4098, 'name': 'y', 'available_modules': [{'name': 'alerts', 'can_run': True, 'error_string': '', 'module_options': {'interval': {'name': 'interval', 'type': 'secs', 'level': 'advanced', 'flags': 1, 'default_value': '60', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'How frequently to reexamine health status', 'long_desc': '', 'tags': [], 'see_also': []}, 'smtp_destination': {'name': 'smtp_destination', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'Email address to send alerts to', 'long_desc': '', 'tags': [], 'see_also': []}, 'smtp_from_name': {'name': 'smtp_from_name', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': 'Ceph', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'Email From: name', 'long_desc': '', 'tags': [], 'see_also': []}, 'smtp_host': {'name': 'smtp_host', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'SMTP server', 'long_desc': '', 'tags': [], 'see_also': []}, 'smtp_password': {'name': 'smtp_password', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'Password to authenticate with', 'long_desc': '', 'tags': [], 'see_also': []}, 'smtp_port': {'name': 'smtp_port', 'type': 'int', 'level': 'advanced', 'flags': 1, 'default_value': '465', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'SMTP port', 'long_desc': '', 'tags': [], 'see_also': []}, 'smtp_sender': {'name': 'smtp_sender', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'SMTP envelope sender', 'long_desc': '', 'tags': [], 'see_also': []}, 'smtp_ssl': {'name': 'smtp_ssl', 'type': 'bool', 'level': 'advanced', 'flags': 1, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'Use SSL to connect to SMTP server', 'long_desc': '', 'tags': [], 'see_also': []}, 'smtp_user': {'name': 'smtp_user', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'User to authenticate as', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'ansible', 'can_run': True, 'error_string': '', 'module_options': {'password': {'name': 'password', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'server_url': {'name': 'server_url', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'username': {'name': 'username', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'verify_server': {'name': 'verify_server', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'balancer', 'can_run': True, 'error_string': '', 'module_options': {'active': {'name': 'active', 'type': 'bool', 'level': 'advanced', 'flags': 1, 'default_value': 'False', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'automatically balance PGs across cluster', 'long_desc': '', 'tags': [], 'see_also': []}, 'begin_time': {'name': 'begin_time', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': '0000', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'beginning time of day to automatically balance', 'long_desc': 'This is a time of day in the format HHMM.', 'tags': [], 'see_also': []}, 'begin_weekday': {'name': 'begin_weekday', 'type': 'uint', 'level': 'advanced', 'flags': 1, 'default_value': '0', 'min': '0', 'max': '7', 'enum_allowed': [], 'desc': 'Restrict automatic balancing to this day of the week or later', 'long_desc': '0 or 7 = Sunday, 1 = Monday, etc.', 'tags': [], 'see_also': []}, 'crush_compat_max_iterations': {'name': 'crush_compat_max_iterations', 'type': 'uint', 'level': 'advanced', 'flags': 1, 'default_value': '25', 'min': '1', 'max': '250', 'enum_allowed': [], 'desc': 'maximum number of iterations to attempt optimization', 'long_desc': '', 'tags': [], 'see_also': []}, 'crush_compat_metrics': {'name': 'crush_compat_metrics', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': 'pgs,objects,bytes', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'metrics with which to calculate OSD utilization', 'long_desc': 'Value is a list of one or more of "pgs", "objects", or "bytes", and indicates which metrics to use to balance utilization.', 'tags': [], 'see_also': []}, 'crush_compat_step': {'name': 'crush_compat_step', 'type': 'float', 'level': 'advanced', 'flags': 1, 'default_value': '0.5', 'min': '0.001', 'max': '0.999', 'enum_allowed': [], 'desc': 'aggressiveness of optimization', 'long_desc': '.99 is very aggressive, .01 is less aggressive', 'tags': [], 'see_also': []}, 'end_time': {'name': 'end_time', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': '2400', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'ending time of day to automatically balance', 'long_desc': 'This is a time of day in the format HHMM.', 'tags': [], 'see_also': []}, 'end_weekday': {'name': 'end_weekday', 'type': 'uint', 'level': 'advanced', 'flags': 1, 'default_value': '7', 'min': '0', 'max': '7', 'enum_allowed': [], 'desc': 'Restrict automatic balancing to days of the week earlier than this', 'long_desc': '0 or 7 = Sunday, 1 = Monday, etc.', 'tags': [], 'see_also': []}, 'min_score': {'name': 'min_score', 'type': 'float', 'level': 'advanced', 'flags': 1, 'default_value': '0', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'minimum score, below which no optimization is attempted', 'long_desc': '', 'tags': [], 'see_also': []}, 'mode': {'name': 'mode', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': 'none', 'min': '', 'max': '', 'enum_allowed': ['crush-compat', 'none', 'upmap'], 'desc': 'Balancer mode', 'long_desc': '', 'tags': [], 'see_also': []}, 'pool_ids': {'name': 'pool_ids', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'pools which the automatic balancing will be limited to', 'long_desc': '', 'tags': [], 'see_also': []}, 'sleep_interval': {'name': 'sleep_interval', 'type': 'secs', 'level': 'advanced', 'flags': 1, 'default_value': '60', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'how frequently to wake up and attempt optimization', 'long_desc': '', 'tags': [], 'see_also': []}, 'upmap_max_deviation': {'name': 'upmap_max_deviation', 'type': 'int', 'level': 'advanced', 'flags': 1, 'default_value': '5', 'min': '1', 'max': '', 'enum_allowed': [], 'desc': 'deviation below which no optimization is attempted', 'long_desc': 'If the number of PGs are within this count then no optimization is attempted', 'tags': [], 'see_also': []}, 'upmap_max_iterations': {'name': 'upmap_max_iterations', 'type': 'uint', 'level': 'advanced', 'flags': 1, 'default_value': '10', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'maximum upmap optimization iterations', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'crash', 'can_run': True, 'error_string': '', 'module_options': {'retain_interval': {'name': 'retain_interval', 'type': 'secs', 'level': 'advanced', 'flags': 1, 'default_value': '31536000', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'how long to retain crashes before pruning them', 'long_desc': '', 'tags': [], 'see_also': []}, 'warn_recent_interval': {'name': 'warn_recent_interval', 'type': 'secs', 'level': 'advanced', 'flags': 1, 'default_value': '1209600', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'time interval in which to warn about recent crashes', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'dashboard', 'can_run': True, 'error_string': '', 'module_options': {'ALERTMANAGER_API_HOST': {'name': 'ALERTMANAGER_API_HOST', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'AUDIT_API_ENABLED': {'name': 'AUDIT_API_ENABLED', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'False', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'AUDIT_API_LOG_PAYLOAD': {'name': 'AUDIT_API_LOG_PAYLOAD', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'ENABLE_BROWSABLE_API': {'name': 'ENABLE_BROWSABLE_API', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'FEATURE_TOGGLE_cephfs': {'name': 'FEATURE_TOGGLE_cephfs', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'FEATURE_TOGGLE_iscsi': {'name': 'FEATURE_TOGGLE_iscsi', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'FEATURE_TOGGLE_mirroring': {'name': 'FEATURE_TOGGLE_mirroring', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'FEATURE_TOGGLE_rbd': {'name': 'FEATURE_TOGGLE_rbd', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'FEATURE_TOGGLE_rgw': {'name': 'FEATURE_TOGGLE_rgw', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE': {'name': 'GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'GRAFANA_API_PASSWORD': {'name': 'GRAFANA_API_PASSWORD', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'admin', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'GRAFANA_API_SSL_VERIFY': {'name': 'GRAFANA_API_SSL_VERIFY', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'GRAFANA_API_URL': {'name': 'GRAFANA_API_URL', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'GRAFANA_API_USERNAME': {'name': 'GRAFANA_API_USERNAME', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'admin', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'GRAFANA_UPDATE_DASHBOARDS': {'name': 'GRAFANA_UPDATE_DASHBOARDS', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'False', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'ISCSI_API_SSL_VERIFICATION': {'name': 'ISCSI_API_SSL_VERIFICATION', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'PROMETHEUS_API_HOST': {'name': 'PROMETHEUS_API_HOST', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'REST_REQUESTS_TIMEOUT': {'name': 'REST_REQUESTS_TIMEOUT', 'type': 'int', 'level': 'advanced', 'flags': 0, 'default_value': '45', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'RGW_API_ACCESS_KEY': {'name': 'RGW_API_ACCESS_KEY', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'RGW_API_ADMIN_RESOURCE': {'name': 'RGW_API_ADMIN_RESOURCE', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'admin', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'RGW_API_HOST': {'name': 'RGW_API_HOST', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'RGW_API_PORT': {'name': 'RGW_API_PORT', 'type': 'int', 'level': 'advanced', 'flags': 0, 'default_value': '80', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'RGW_API_SCHEME': {'name': 'RGW_API_SCHEME', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'http', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'RGW_API_SECRET_KEY': {'name': 'RGW_API_SECRET_KEY', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'RGW_API_SSL_VERIFY': {'name': 'RGW_API_SSL_VERIFY', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'RGW_API_USER_ID': {'name': 'RGW_API_USER_ID', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'crt_file': {'name': 'crt_file', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'debug': {'name': 'debug', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'False', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'Enable/disable debug options', 'long_desc': '', 'tags': [], 'see_also': []}, 'jwt_token_ttl': {'name': 'jwt_token_ttl', 'type': 'int', 'level': 'advanced', 'flags': 0, 'default_value': '28800', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'key_file': {'name': 'key_file', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'password': {'name': 'password', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'server_addr': {'name': 'server_addr', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '::', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'server_port': {'name': 'server_port', 'type': 'int', 'level': 'advanced', 'flags': 0, 'default_value': '8080', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'ssl': {'name': 'ssl', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'ssl_server_port': {'name': 'ssl_server_port', 'type': 'int', 'level': 'advanced', 'flags': 0, 'default_value': '8443', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'standby_behaviour': {'name': 'standby_behaviour', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'redirect', 'min': '', 'max': '', 'enum_allowed': ['error', 'redirect'], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'standby_error_status_code': {'name': 'standby_error_status_code', 'type': 'int', 'level': 'advanced', 'flags': 0, 'default_value': '500', 'min': '400', 'max': '599', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'url_prefix': {'name': 'url_prefix', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'username': {'name': 'username', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'deepsea', 'can_run': True, 'error_string': '', 'module_options': {'salt_api_eauth': {'name': 'salt_api_eauth', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'sharedsecret', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'salt_api_password': {'name': 'salt_api_password', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'salt_api_url': {'name': 'salt_api_url', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'salt_api_username': {'name': 'salt_api_username', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'devicehealth', 'can_run': True, 'error_string': '', 'module_options': {'enable_monitoring': {'name': 'enable_monitoring', 'type': 'bool', 'level': 'advanced', 'flags': 1, 'default_value': 'False', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'monitor device health metrics', 'long_desc': '', 'tags': [], 'see_also': []}, 'mark_out_threshold': {'name': 'mark_out_threshold', 'type': 'secs', 'level': 'advanced', 'flags': 1, 'default_value': '2419200', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'automatically mark OSD if it may fail before this long', 'long_desc': '', 'tags': [], 'see_also': []}, 'pool_name': {'name': 'pool_name', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': 'device_health_metrics', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'name of pool in which to store device health metrics', 'long_desc': '', 'tags': [], 'see_also': []}, 'retention_period': {'name': 'retention_period', 'type': 'secs', 'level': 'advanced', 'flags': 1, 'default_value': '15552000', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'how long to retain device health metrics', 'long_desc': '', 'tags': [], 'see_also': []}, 'scrape_frequency': {'name': 'scrape_frequency', 'type': 'secs', 'level': 'advanced', 'flags': 1, 'default_value': '86400', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'how frequently to scrape device health metrics', 'long_desc': '', 'tags': [], 'see_also': []}, 'self_heal': {'name': 'self_heal', 'type': 'bool', 'level': 'advanced', 'flags': 1, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'preemptively heal cluster around devices that may fail', 'long_desc': '', 'tags': [], 'see_also': []}, 'sleep_interval': {'name': 'sleep_interval', 'type': 'secs', 'level': 'advanced', 'flags': 1, 'default_value': '600', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'how frequently to wake up and check device health', 'long_desc': '', 'tags': [], 'see_also': []}, 'warn_threshold': {'name': 'warn_threshold', 'type': 'secs', 'level': 'advanced', 'flags': 1, 'default_value': '7257600', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'raise health warning if OSD may fail before this long', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'diskprediction_cloud', 'can_run': True, 'error_string': '', 'module_options': {'diskprediction_cert_context': {'name': 'diskprediction_cert_context', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'diskprediction_default_authority': {'name': 'diskprediction_default_authority', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'localhost', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'diskprediction_password': {'name': 'diskprediction_password', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'diskprediction_port': {'name': 'diskprediction_port', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '31400', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'diskprediction_retrieve_prediction_interval': {'name': 'diskprediction_retrieve_prediction_interval', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '43200', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'diskprediction_server': {'name': 'diskprediction_server', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'diskprediction_ssl_target_name_override': {'name': 'diskprediction_ssl_target_name_override', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'localhost', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'diskprediction_upload_metrics_interval': {'name': 'diskprediction_upload_metrics_interval', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '600', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'diskprediction_upload_smart_interval': {'name': 'diskprediction_upload_smart_interval', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '43200', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'diskprediction_user': {'name': 'diskprediction_user', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'sleep_interval': {'name': 'sleep_interval', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '600', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'diskprediction_local', 'can_run': True, 'error_string': '', 'module_options': {'predict_interval': {'name': 'predict_interval', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '86400', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'sleep_interval': {'name': 'sleep_interval', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '600', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'influx', 'can_run': False, 'error_string': 'influxdb python module not found', 'module_options': {'batch_size': {'name': 'batch_size', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '5000', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'database': {'name': 'database', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'ceph', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'hostname': {'name': 'hostname', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'None', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'interval': {'name': 'interval', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '30', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'password': {'name': 'password', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'None', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'port': {'name': 'port', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '8086', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'ssl': {'name': 'ssl', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'false', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'threads': {'name': 'threads', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '5', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'username': {'name': 'username', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'None', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'verify_ssl': {'name': 'verify_ssl', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'true', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'insights', 'can_run': True, 'error_string': '', 'module_options': {}}, {'name': 'iostat', 'can_run': True, 'error_string': '', 'module_options': {}}, {'name': 'localpool', 'can_run': True, 'error_string': '', 'module_options': {'failure_domain': {'name': 'failure_domain', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': 'host', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'failure domain for any created local pool', 'long_desc': '', 'tags': [], 'see_also': []}, 'min_size': {'name': 'min_size', 'type': 'int', 'level': 'advanced', 'flags': 1, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'default min_size for any created local pool', 'long_desc': '', 'tags': [], 'see_also': []}, 'num_rep': {'name': 'num_rep', 'type': 'int', 'level': 'advanced', 'flags': 1, 'default_value': '3', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'default replica count for any created local pool', 'long_desc': '', 'tags': [], 'see_also': []}, 'pg_num': {'name': 'pg_num', 'type': 'int', 'level': 'advanced', 'flags': 1, 'default_value': '128', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'default pg_num for any created local pool', 'long_desc': '', 'tags': [], 'see_also': []}, 'prefix': {'name': 'prefix', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'name prefix for any created local pool', 'long_desc': '', 'tags': [], 'see_also': []}, 'subtree': {'name': 'subtree', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': 'rack', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'CRUSH level for which to create a local pool', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'orchestrator_cli', 'can_run': True, 'error_string': '', 'module_options': {'orchestrator': {'name': 'orchestrator', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'pg_autoscaler', 'can_run': True, 'error_string': '', 'module_options': {'sleep_interval': {'name': 'sleep_interval', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '60', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'progress', 'can_run': True, 'error_string': '', 'module_options': {'enabled': {'name': 'enabled', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'max_completed_events': {'name': 'max_completed_events', 'type': 'int', 'level': 'advanced', 'flags': 1, 'default_value': '50', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'number of past completed events to remember', 'long_desc': '', 'tags': [], 'see_also': []}, 'persist_interval': {'name': 'persist_interval', 'type': 'secs', 'level': 'advanced', 'flags': 1, 'default_value': '5', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'how frequently to persist completed events', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'prometheus', 'can_run': True, 'error_string': '', 'module_options': {'rbd_stats_pools': {'name': 'rbd_stats_pools', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'rbd_stats_pools_refresh_interval': {'name': 'rbd_stats_pools_refresh_interval', 'type': 'int', 'level': 'advanced', 'flags': 0, 'default_value': '300', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'scrape_interval': {'name': 'scrape_interval', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'server_addr': {'name': 'server_addr', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'server_port': {'name': 'server_port', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'stale_cache_strategy': {'name': 'stale_cache_strategy', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'rbd_support', 'can_run': True, 'error_string': '', 'module_options': {}}, {'name': 'restful', 'can_run': True, 'error_string': '', 'module_options': {'key_file': {'name': 'key_file', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'server_addr': {'name': 'server_addr', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'server_port': {'name': 'server_port', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'rook', 'can_run': True, 'error_string': '', 'module_options': {}}, {'name': 'selftest', 'can_run': True, 'error_string': '', 'module_options': {'roption1': {'name': 'roption1', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'roption2': {'name': 'roption2', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'xyz', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'rwoption1': {'name': 'rwoption1', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'rwoption2': {'name': 'rwoption2', 'type': 'int', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'rwoption3': {'name': 'rwoption3', 'type': 'float', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'rwoption4': {'name': 'rwoption4', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'rwoption5': {'name': 'rwoption5', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'rwoption6': {'name': 'rwoption6', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'testkey': {'name': 'testkey', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'testlkey': {'name': 'testlkey', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'testnewline': {'name': 'testnewline', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'ssh', 'can_run': True, 'error_string': '', 'module_options': {'inventory_cache_timeout_min': {'name': 'inventory_cache_timeout_min', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'ssh_config_file': {'name': 'ssh_config_file', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'status', 'can_run': True, 'error_string': '', 'module_options': {}}, {'name': 'telegraf', 'can_run': True, 'error_string': '', 'module_options': {'address': {'name': 'address', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'unixgram:///tmp/telegraf.sock', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'interval': {'name': 'interval', 'type': 'secs', 'level': 'advanced', 'flags': 0, 'default_value': '15', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'telemetry', 'can_run': True, 'error_string': '', 'module_options': {'channel_basic': {'name': 'channel_basic', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'Share basic cluster information (size, version)', 'long_desc': '', 'tags': [], 'see_also': []}, 'channel_crash': {'name': 'channel_crash', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'channel_device': {'name': 'channel_device', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'channel_ident': {'name': 'channel_ident', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'False', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'contact': {'name': 'contact', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'None', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'description': {'name': 'description', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'None', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'device_url': {'name': 'device_url', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'https://telemetry.ceph.com/device', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'enabled': {'name': 'enabled', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'False', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'interval': {'name': 'interval', 'type': 'int', 'level': 'advanced', 'flags': 0, 'default_value': '24', 'min': '8', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'last_opt_revision': {'name': 'last_opt_revision', 'type': 'int', 'level': 'advanced', 'flags': 0, 'default_value': '1', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'leaderboard': {'name': 'leaderboard', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'False', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'organization': {'name': 'organization', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'None', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'proxy': {'name': 'proxy', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'None', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'url': {'name': 'url', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'https://telemetry.ceph.com/report', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'test_orchestrator', 'can_run': True, 'error_string': '', 'module_options': {}}, {'name': 'volumes', 'can_run': True, 'error_string': '', 'module_options': {'max_concurrent_clones': {'name': 'max_concurrent_clones', 'type': 'int', 'level': 'advanced', 'flags': 0, 'default_value': '4', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'Number of asynchronous cloner threads', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'zabbix', 'can_run': True, 'error_string': '', 'module_options': {'identifier': {'name': 'identifier', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'interval': {'name': 'interval', 'type': 'secs', 'level': 'advanced', 'flags': 0, 'default_value': '60', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'zabbix_host': {'name': 'zabbix_host', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'None', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'zabbix_port': {'name': 'zabbix_port', 'type': 'int', 'level': 'advanced', 'flags': 0, 'default_value': '10051', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'zabbix_sender': {'name': 'zabbix_sender', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '/usr/bin/zabbix_sender', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}]}], 'modules': ['iostat', 'restful'], 'available_modules': [{'name': 'alerts', 'can_run': True, 'error_string': '', 'module_options': {'interval': {'name': 'interval', 'type': 'secs', 'level': 'advanced', 'flags': 1, 'default_value': '60', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'How frequently to reexamine health status', 'long_desc': '', 'tags': [], 'see_also': []}, 'smtp_destination': {'name': 'smtp_destination', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'Email address to send alerts to', 'long_desc': '', 'tags': [], 'see_also': []}, 'smtp_from_name': {'name': 'smtp_from_name', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': 'Ceph', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'Email From: name', 'long_desc': '', 'tags': [], 'see_also': []}, 'smtp_host': {'name': 'smtp_host', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'SMTP server', 'long_desc': '', 'tags': [], 'see_also': []}, 'smtp_password': {'name': 'smtp_password', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'Password to authenticate with', 'long_desc': '', 'tags': [], 'see_also': []}, 'smtp_port': {'name': 'smtp_port', 'type': 'int', 'level': 'advanced', 'flags': 1, 'default_value': '465', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'SMTP port', 'long_desc': '', 'tags': [], 'see_also': []}, 'smtp_sender': {'name': 'smtp_sender', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'SMTP envelope sender', 'long_desc': '', 'tags': [], 'see_also': []}, 'smtp_ssl': {'name': 'smtp_ssl', 'type': 'bool', 'level': 'advanced', 'flags': 1, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'Use SSL to connect to SMTP server', 'long_desc': '', 'tags': [], 'see_also': []}, 'smtp_user': {'name': 'smtp_user', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'User to authenticate as', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'ansible', 'can_run': True, 'error_string': '', 'module_options': {'password': {'name': 'password', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'server_url': {'name': 'server_url', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'username': {'name': 'username', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'verify_server': {'name': 'verify_server', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'balancer', 'can_run': True, 'error_string': '', 'module_options': {'active': {'name': 'active', 'type': 'bool', 'level': 'advanced', 'flags': 1, 'default_value': 'False', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'automatically balance PGs across cluster', 'long_desc': '', 'tags': [], 'see_also': []}, 'begin_time': {'name': 'begin_time', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': '0000', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'beginning time of day to automatically balance', 'long_desc': 'This is a time of day in the format HHMM.', 'tags': [], 'see_also': []}, 'begin_weekday': {'name': 'begin_weekday', 'type': 'uint', 'level': 'advanced', 'flags': 1, 'default_value': '0', 'min': '0', 'max': '7', 'enum_allowed': [], 'desc': 'Restrict automatic balancing to this day of the week or later', 'long_desc': '0 or 7 = Sunday, 1 = Monday, etc.', 'tags': [], 'see_also': []}, 'crush_compat_max_iterations': {'name': 'crush_compat_max_iterations', 'type': 'uint', 'level': 'advanced', 'flags': 1, 'default_value': '25', 'min': '1', 'max': '250', 'enum_allowed': [], 'desc': 'maximum number of iterations to attempt optimization', 'long_desc': '', 'tags': [], 'see_also': []}, 'crush_compat_metrics': {'name': 'crush_compat_metrics', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': 'pgs,objects,bytes', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'metrics with which to calculate OSD utilization', 'long_desc': 'Value is a list of one or more of "pgs", "objects", or "bytes", and indicates which metrics to use to balance utilization.', 'tags': [], 'see_also': []}, 'crush_compat_step': {'name': 'crush_compat_step', 'type': 'float', 'level': 'advanced', 'flags': 1, 'default_value': '0.5', 'min': '0.001', 'max': '0.999', 'enum_allowed': [], 'desc': 'aggressiveness of optimization', 'long_desc': '.99 is very aggressive, .01 is less aggressive', 'tags': [], 'see_also': []}, 'end_time': {'name': 'end_time', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': '2400', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'ending time of day to automatically balance', 'long_desc': 'This is a time of day in the format HHMM.', 'tags': [], 'see_also': []}, 'end_weekday': {'name': 'end_weekday', 'type': 'uint', 'level': 'advanced', 'flags': 1, 'default_value': '7', 'min': '0', 'max': '7', 'enum_allowed': [], 'desc': 'Restrict automatic balancing to days of the week earlier than this', 'long_desc': '0 or 7 = Sunday, 1 = Monday, etc.', 'tags': [], 'see_also': []}, 'min_score': {'name': 'min_score', 'type': 'float', 'level': 'advanced', 'flags': 1, 'default_value': '0', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'minimum score, below which no optimization is attempted', 'long_desc': '', 'tags': [], 'see_also': []}, 'mode': {'name': 'mode', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': 'none', 'min': '', 'max': '', 'enum_allowed': ['crush-compat', 'none', 'upmap'], 'desc': 'Balancer mode', 'long_desc': '', 'tags': [], 'see_also': []}, 'pool_ids': {'name': 'pool_ids', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'pools which the automatic balancing will be limited to', 'long_desc': '', 'tags': [], 'see_also': []}, 'sleep_interval': {'name': 'sleep_interval', 'type': 'secs', 'level': 'advanced', 'flags': 1, 'default_value': '60', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'how frequently to wake up and attempt optimization', 'long_desc': '', 'tags': [], 'see_also': []}, 'upmap_max_deviation': {'name': 'upmap_max_deviation', 'type': 'int', 'level': 'advanced', 'flags': 1, 'default_value': '5', 'min': '1', 'max': '', 'enum_allowed': [], 'desc': 'deviation below which no optimization is attempted', 'long_desc': 'If the number of PGs are within this count then no optimization is attempted', 'tags': [], 'see_also': []}, 'upmap_max_iterations': {'name': 'upmap_max_iterations', 'type': 'uint', 'level': 'advanced', 'flags': 1, 'default_value': '10', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'maximum upmap optimization iterations', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'crash', 'can_run': True, 'error_string': '', 'module_options': {'retain_interval': {'name': 'retain_interval', 'type': 'secs', 'level': 'advanced', 'flags': 1, 'default_value': '31536000', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'how long to retain crashes before pruning them', 'long_desc': '', 'tags': [], 'see_also': []}, 'warn_recent_interval': {'name': 'warn_recent_interval', 'type': 'secs', 'level': 'advanced', 'flags': 1, 'default_value': '1209600', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'time interval in which to warn about recent crashes', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'dashboard', 'can_run': True, 'error_string': '', 'module_options': {'ALERTMANAGER_API_HOST': {'name': 'ALERTMANAGER_API_HOST', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'AUDIT_API_ENABLED': {'name': 'AUDIT_API_ENABLED', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'False', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'AUDIT_API_LOG_PAYLOAD': {'name': 'AUDIT_API_LOG_PAYLOAD', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'ENABLE_BROWSABLE_API': {'name': 'ENABLE_BROWSABLE_API', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'FEATURE_TOGGLE_cephfs': {'name': 'FEATURE_TOGGLE_cephfs', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'FEATURE_TOGGLE_iscsi': {'name': 'FEATURE_TOGGLE_iscsi', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'FEATURE_TOGGLE_mirroring': {'name': 'FEATURE_TOGGLE_mirroring', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'FEATURE_TOGGLE_rbd': {'name': 'FEATURE_TOGGLE_rbd', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'FEATURE_TOGGLE_rgw': {'name': 'FEATURE_TOGGLE_rgw', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE': {'name': 'GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'GRAFANA_API_PASSWORD': {'name': 'GRAFANA_API_PASSWORD', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'admin', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'GRAFANA_API_SSL_VERIFY': {'name': 'GRAFANA_API_SSL_VERIFY', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'GRAFANA_API_URL': {'name': 'GRAFANA_API_URL', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'GRAFANA_API_USERNAME': {'name': 'GRAFANA_API_USERNAME', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'admin', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'GRAFANA_UPDATE_DASHBOARDS': {'name': 'GRAFANA_UPDATE_DASHBOARDS', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'False', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'ISCSI_API_SSL_VERIFICATION': {'name': 'ISCSI_API_SSL_VERIFICATION', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'PROMETHEUS_API_HOST': {'name': 'PROMETHEUS_API_HOST', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'REST_REQUESTS_TIMEOUT': {'name': 'REST_REQUESTS_TIMEOUT', 'type': 'int', 'level': 'advanced', 'flags': 0, 'default_value': '45', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'RGW_API_ACCESS_KEY': {'name': 'RGW_API_ACCESS_KEY', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'RGW_API_ADMIN_RESOURCE': {'name': 'RGW_API_ADMIN_RESOURCE', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'admin', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'RGW_API_HOST': {'name': 'RGW_API_HOST', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'RGW_API_PORT': {'name': 'RGW_API_PORT', 'type': 'int', 'level': 'advanced', 'flags': 0, 'default_value': '80', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'RGW_API_SCHEME': {'name': 'RGW_API_SCHEME', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'http', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'RGW_API_SECRET_KEY': {'name': 'RGW_API_SECRET_KEY', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'RGW_API_SSL_VERIFY': {'name': 'RGW_API_SSL_VERIFY', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'RGW_API_USER_ID': {'name': 'RGW_API_USER_ID', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'crt_file': {'name': 'crt_file', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'debug': {'name': 'debug', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'False', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'Enable/disable debug options', 'long_desc': '', 'tags': [], 'see_also': []}, 'jwt_token_ttl': {'name': 'jwt_token_ttl', 'type': 'int', 'level': 'advanced', 'flags': 0, 'default_value': '28800', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'key_file': {'name': 'key_file', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'password': {'name': 'password', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'server_addr': {'name': 'server_addr', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '::', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'server_port': {'name': 'server_port', 'type': 'int', 'level': 'advanced', 'flags': 0, 'default_value': '8080', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'ssl': {'name': 'ssl', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'ssl_server_port': {'name': 'ssl_server_port', 'type': 'int', 'level': 'advanced', 'flags': 0, 'default_value': '8443', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'standby_behaviour': {'name': 'standby_behaviour', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'redirect', 'min': '', 'max': '', 'enum_allowed': ['error', 'redirect'], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'standby_error_status_code': {'name': 'standby_error_status_code', 'type': 'int', 'level': 'advanced', 'flags': 0, 'default_value': '500', 'min': '400', 'max': '599', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'url_prefix': {'name': 'url_prefix', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'username': {'name': 'username', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'deepsea', 'can_run': True, 'error_string': '', 'module_options': {'salt_api_eauth': {'name': 'salt_api_eauth', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'sharedsecret', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'salt_api_password': {'name': 'salt_api_password', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'salt_api_url': {'name': 'salt_api_url', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'salt_api_username': {'name': 'salt_api_username', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'devicehealth', 'can_run': True, 'error_string': '', 'module_options': {'enable_monitoring': {'name': 'enable_monitoring', 'type': 'bool', 'level': 'advanced', 'flags': 1, 'default_value': 'False', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'monitor device health metrics', 'long_desc': '', 'tags': [], 'see_also': []}, 'mark_out_threshold': {'name': 'mark_out_threshold', 'type': 'secs', 'level': 'advanced', 'flags': 1, 'default_value': '2419200', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'automatically mark OSD if it may fail before this long', 'long_desc': '', 'tags': [], 'see_also': []}, 'pool_name': {'name': 'pool_name', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': 'device_health_metrics', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'name of pool in which to store device health metrics', 'long_desc': '', 'tags': [], 'see_also': []}, 'retention_period': {'name': 'retention_period', 'type': 'secs', 'level': 'advanced', 'flags': 1, 'default_value': '15552000', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'how long to retain device health metrics', 'long_desc': '', 'tags': [], 'see_also': []}, 'scrape_frequency': {'name': 'scrape_frequency', 'type': 'secs', 'level': 'advanced', 'flags': 1, 'default_value': '86400', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'how frequently to scrape device health metrics', 'long_desc': '', 'tags': [], 'see_also': []}, 'self_heal': {'name': 'self_heal', 'type': 'bool', 'level': 'advanced', 'flags': 1, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'preemptively heal cluster around devices that may fail', 'long_desc': '', 'tags': [], 'see_also': []}, 'sleep_interval': {'name': 'sleep_interval', 'type': 'secs', 'level': 'advanced', 'flags': 1, 'default_value': '600', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'how frequently to wake up and check device health', 'long_desc': '', 'tags': [], 'see_also': []}, 'warn_threshold': {'name': 'warn_threshold', 'type': 'secs', 'level': 'advanced', 'flags': 1, 'default_value': '7257600', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'raise health warning if OSD may fail before this long', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'diskprediction_cloud', 'can_run': True, 'error_string': '', 'module_options': {'diskprediction_cert_context': {'name': 'diskprediction_cert_context', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'diskprediction_default_authority': {'name': 'diskprediction_default_authority', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'localhost', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'diskprediction_password': {'name': 'diskprediction_password', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'diskprediction_port': {'name': 'diskprediction_port', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '31400', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'diskprediction_retrieve_prediction_interval': {'name': 'diskprediction_retrieve_prediction_interval', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '43200', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'diskprediction_server': {'name': 'diskprediction_server', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'diskprediction_ssl_target_name_override': {'name': 'diskprediction_ssl_target_name_override', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'localhost', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'diskprediction_upload_metrics_interval': {'name': 'diskprediction_upload_metrics_interval', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '600', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'diskprediction_upload_smart_interval': {'name': 'diskprediction_upload_smart_interval', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '43200', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'diskprediction_user': {'name': 'diskprediction_user', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'sleep_interval': {'name': 'sleep_interval', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '600', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'diskprediction_local', 'can_run': True, 'error_string': '', 'module_options': {'predict_interval': {'name': 'predict_interval', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '86400', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'sleep_interval': {'name': 'sleep_interval', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '600', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'influx', 'can_run': False, 'error_string': 'influxdb python module not found', 'module_options': {'batch_size': {'name': 'batch_size', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '5000', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'database': {'name': 'database', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'ceph', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'hostname': {'name': 'hostname', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'None', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'interval': {'name': 'interval', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '30', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'password': {'name': 'password', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'None', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'port': {'name': 'port', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '8086', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'ssl': {'name': 'ssl', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'false', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'threads': {'name': 'threads', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '5', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'username': {'name': 'username', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'None', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'verify_ssl': {'name': 'verify_ssl', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'true', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'insights', 'can_run': True, 'error_string': '', 'module_options': {}}, {'name': 'iostat', 'can_run': True, 'error_string': '', 'module_options': {}}, {'name': 'localpool', 'can_run': True, 'error_string': '', 'module_options': {'failure_domain': {'name': 'failure_domain', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': 'host', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'failure domain for any created local pool', 'long_desc': '', 'tags': [], 'see_also': []}, 'min_size': {'name': 'min_size', 'type': 'int', 'level': 'advanced', 'flags': 1, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'default min_size for any created local pool', 'long_desc': '', 'tags': [], 'see_also': []}, 'num_rep': {'name': 'num_rep', 'type': 'int', 'level': 'advanced', 'flags': 1, 'default_value': '3', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'default replica count for any created local pool', 'long_desc': '', 'tags': [], 'see_also': []}, 'pg_num': {'name': 'pg_num', 'type': 'int', 'level': 'advanced', 'flags': 1, 'default_value': '128', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'default pg_num for any created local pool', 'long_desc': '', 'tags': [], 'see_also': []}, 'prefix': {'name': 'prefix', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'name prefix for any created local pool', 'long_desc': '', 'tags': [], 'see_also': []}, 'subtree': {'name': 'subtree', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': 'rack', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'CRUSH level for which to create a local pool', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'orchestrator_cli', 'can_run': True, 'error_string': '', 'module_options': {'orchestrator': {'name': 'orchestrator', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'pg_autoscaler', 'can_run': True, 'error_string': '', 'module_options': {'sleep_interval': {'name': 'sleep_interval', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '60', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'progress', 'can_run': True, 'error_string': '', 'module_options': {'enabled': {'name': 'enabled', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'max_completed_events': {'name': 'max_completed_events', 'type': 'int', 'level': 'advanced', 'flags': 1, 'default_value': '50', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'number of past completed events to remember', 'long_desc': '', 'tags': [], 'see_also': []}, 'persist_interval': {'name': 'persist_interval', 'type': 'secs', 'level': 'advanced', 'flags': 1, 'default_value': '5', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'how frequently to persist completed events', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'prometheus', 'can_run': True, 'error_string': '', 'module_options': {'rbd_stats_pools': {'name': 'rbd_stats_pools', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'rbd_stats_pools_refresh_interval': {'name': 'rbd_stats_pools_refresh_interval', 'type': 'int', 'level': 'advanced', 'flags': 0, 'default_value': '300', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'scrape_interval': {'name': 'scrape_interval', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'server_addr': {'name': 'server_addr', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'server_port': {'name': 'server_port', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'stale_cache_strategy': {'name': 'stale_cache_strategy', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'rbd_support', 'can_run': True, 'error_string': '', 'module_options': {}}, {'name': 'restful', 'can_run': True, 'error_string': '', 'module_options': {'key_file': {'name': 'key_file', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'server_addr': {'name': 'server_addr', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'server_port': {'name': 'server_port', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'rook', 'can_run': True, 'error_string': '', 'module_options': {}}, {'name': 'selftest', 'can_run': True, 'error_string': '', 'module_options': {'roption1': {'name': 'roption1', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'roption2': {'name': 'roption2', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'xyz', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'rwoption1': {'name': 'rwoption1', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'rwoption2': {'name': 'rwoption2', 'type': 'int', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'rwoption3': {'name': 'rwoption3', 'type': 'float', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'rwoption4': {'name': 'rwoption4', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'rwoption5': {'name': 'rwoption5', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'rwoption6': {'name': 'rwoption6', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'testkey': {'name': 'testkey', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'testlkey': {'name': 'testlkey', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'testnewline': {'name': 'testnewline', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'ssh', 'can_run': True, 'error_string': '', 'module_options': {'inventory_cache_timeout_min': {'name': 'inventory_cache_timeout_min', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'ssh_config_file': {'name': 'ssh_config_file', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'status', 'can_run': True, 'error_string': '', 'module_options': {}}, {'name': 'telegraf', 'can_run': True, 'error_string': '', 'module_options': {'address': {'name': 'address', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'unixgram:///tmp/telegraf.sock', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'interval': {'name': 'interval', 'type': 'secs', 'level': 'advanced', 'flags': 0, 'default_value': '15', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'telemetry', 'can_run': True, 'error_string': '', 'module_options': {'channel_basic': {'name': 'channel_basic', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'Share basic cluster information (size, version)', 'long_desc': '', 'tags': [], 'see_also': []}, 'channel_crash': {'name': 'channel_crash', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'channel_device': {'name': 'channel_device', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'channel_ident': {'name': 'channel_ident', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'False', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'contact': {'name': 'contact', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'None', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'description': {'name': 'description', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'None', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'device_url': {'name': 'device_url', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'https://telemetry.ceph.com/device', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'enabled': {'name': 'enabled', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'False', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'interval': {'name': 'interval', 'type': 'int', 'level': 'advanced', 'flags': 0, 'default_value': '24', 'min': '8', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'last_opt_revision': {'name': 'last_opt_revision', 'type': 'int', 'level': 'advanced', 'flags': 0, 'default_value': '1', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'leaderboard': {'name': 'leaderboard', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'False', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'organization': {'name': 'organization', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'None', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'proxy': {'name': 'proxy', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'None', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'url': {'name': 'url', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'https://telemetry.ceph.com/report', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'test_orchestrator', 'can_run': True, 'error_string': '', 'module_options': {}}, {'name': 'volumes', 'can_run': True, 'error_string': '', 'module_options': {'max_concurrent_clones': {'name': 'max_concurrent_clones', 'type': 'int', 'level': 'advanced', 'flags': 0, 'default_value': '4', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'Number of asynchronous cloner threads', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'zabbix', 'can_run': True, 'error_string': '', 'module_options': {'identifier': {'name': 'identifier', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'interval': {'name': 'interval', 'type': 'secs', 'level': 'advanced', 'flags': 0, 'default_value': '60', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'zabbix_host': {'name': 'zabbix_host', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'None', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'zabbix_port': {'name': 'zabbix_port', 'type': 'int', 'level': 'advanced', 'flags': 0, 'default_value': '10051', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'zabbix_sender': {'name': 'zabbix_sender', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '/usr/bin/zabbix_sender', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}], 'services': {}, 'always_on_modules': {'nautilus': ['balancer', 'crash', 'devicehealth', 'orchestrator_cli', 'progress', 'rbd_support', 'status', 'volumes']}}, 'servicemap': {'epoch': 2, 'modified': '2021-02-10 11:14:19.765616', 'services': {'mds': {'daemons': {'summary': '', 'a': {'start_epoch': 0, 'start_stamp': '0.000000', 'gid': 0, 'addr': ':/0', 'metadata': {}, 'task_status': {'scrub status': 'idle'}}}}}}, 'progress_events': {}} 2021-02-10T11:14:23.268 INFO:tasks.ceph.ceph_manager.ceph:clean! 2021-02-10T11:14:23.268 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2021-02-10T11:14:23.268 DEBUG:teuthology.orchestra.run.gibba012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph health 2021-02-10T11:14:23.686 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:23.670 7ff513210700 1 Processor -- start 2021-02-10T11:14:23.686 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:23.670 7ff513210700 1 -- start start 2021-02-10T11:14:23.686 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:23.671 7ff513210700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7ff50c1b0470 0x7ff50c1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:23.686 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:23.671 7ff513210700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff50c0ef610 0x7ff50c0efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:23.686 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:23.671 7ff513210700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff50c0f0b50 con 0x7ff50c0f0030 2021-02-10T11:14:23.686 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:23.671 7ff513210700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff50c0e52e0 con 0x7ff50c1b0470 2021-02-10T11:14:23.686 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:23.671 7ff513210700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff50c0e54d0 con 0x7ff50c0ef610 2021-02-10T11:14:23.686 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:23.671 7ff51082b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff50c0ef610 0x7ff50c0efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:23.686 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:23.671 7ff51102c700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7ff50c1b0470 0x7ff50c1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:23.687 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:23.671 7ff51182d700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7ff50c0f0030 0x7ff50c0f10d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:35536/0 (socket says 172.21.2.112:35536) 2021-02-10T11:14:23.687 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:23.671 7ff51082b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff50c0ef610 0x7ff50c0efa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3301/0 says I am v2:172.21.2.112:42198/0 (socket says 172.21.2.112:42198) 2021-02-10T11:14:23.687 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:23.671 7ff51182d700 1 -- 172.21.2.112:0/3558417335 learned_addr learned my addr 172.21.2.112:0/3558417335 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:14:23.687 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:23.672 7ff51102c700 1 -- 172.21.2.112:0/3558417335 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff50c0ef610 msgr2=0x7ff50c0efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:14:23.687 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:23.672 7ff51102c700 1 --2- 172.21.2.112:0/3558417335 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff50c0ef610 0x7ff50c0efa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:23.687 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:23.672 7ff51102c700 1 -- 172.21.2.112:0/3558417335 >> v1:172.21.2.120:6789/0 conn(0x7ff50c0f0030 legacy=0x7ff50c0f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:23.687 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:23.672 7ff51102c700 1 -- 172.21.2.112:0/3558417335 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff50c0f0430 con 0x7ff50c1b0470 2021-02-10T11:14:23.687 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:23.672 7ff51102c700 1 --2- 172.21.2.112:0/3558417335 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff50c1b0470 0x7ff50c1b0870 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7ff508006650 tx=0x7ff508009a80).ready entity=mon.1 client_cookie=9ca9b0219e30bc12 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:23.687 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:23.673 7ff503fff700 1 -- 172.21.2.112:0/3558417335 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff50800c0c0 con 0x7ff50c1b0470 2021-02-10T11:14:23.687 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:23.673 7ff503fff700 1 -- 172.21.2.112:0/3558417335 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff50800c340 con 0x7ff50c1b0470 2021-02-10T11:14:23.687 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:23.673 7ff503fff700 1 -- 172.21.2.112:0/3558417335 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff50800c860 con 0x7ff50c1b0470 2021-02-10T11:14:23.687 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:23.673 7ff513210700 1 -- 172.21.2.112:0/3558417335 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff50c1b0470 msgr2=0x7ff50c1b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:23.687 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:23.673 7ff513210700 1 --2- 172.21.2.112:0/3558417335 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff50c1b0470 0x7ff50c1b0870 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7ff508006650 tx=0x7ff508009a80).stop 2021-02-10T11:14:23.688 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:23.673 7ff513210700 1 -- 172.21.2.112:0/3558417335 shutdown_connections 2021-02-10T11:14:23.688 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:23.673 7ff513210700 1 --2- 172.21.2.112:0/3558417335 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff50c0ef610 0x7ff50c0efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:23.688 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:23.673 7ff513210700 1 --2- 172.21.2.112:0/3558417335 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff50c1b0470 0x7ff50c1b0870 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:23.688 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:23.673 7ff513210700 1 -- 172.21.2.112:0/3558417335 >> 172.21.2.112:0/3558417335 conn(0x7ff50c0ce3e0 msgr2=0x7ff50c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:23.688 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:23.673 7ff513210700 1 -- 172.21.2.112:0/3558417335 shutdown_connections 2021-02-10T11:14:23.688 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:23.673 7ff513210700 1 -- 172.21.2.112:0/3558417335 wait complete. 2021-02-10T11:14:23.690 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:23.688 7ff513210700 1 Processor -- start 2021-02-10T11:14:23.690 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:23.688 7ff513210700 1 -- start start 2021-02-10T11:14:23.690 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:23.688 7ff513210700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff50c0bfcc0 0x7ff50c106c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:23.690 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:23.688 7ff513210700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff50c0c00c0 0x7ff50c08e730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:23.690 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:23.688 7ff513210700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff50c107230 0x7ff50c08ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:23.690 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:23.688 7ff513210700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff50c08f430 con 0x7ff50c107230 2021-02-10T11:14:23.690 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:23.688 7ff513210700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff50c08f9b0 con 0x7ff50c0bfcc0 2021-02-10T11:14:23.690 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:23.688 7ff513210700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff50c08fb90 con 0x7ff50c0c00c0 2021-02-10T11:14:23.690 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:23.688 7ff51182d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff50c107230 0x7ff50c08ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:23.690 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:23.688 7ff51182d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff50c107230 0x7ff50c08ed50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:42522/0 (socket says 172.21.2.112:42522) 2021-02-10T11:14:23.690 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:23.688 7ff51182d700 1 -- 172.21.2.112:0/1824937240 learned_addr learned my addr 172.21.2.112:0/1824937240 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:14:23.690 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:23.689 7ff51102c700 1 --2- 172.21.2.112:0/1824937240 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff50c0bfcc0 0x7ff50c106c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:23.691 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:23.689 7ff51082b700 1 --2- 172.21.2.112:0/1824937240 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff50c0c00c0 0x7ff50c08e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:23.691 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:23.689 7ff51182d700 1 -- 172.21.2.112:0/1824937240 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff50c0c00c0 msgr2=0x7ff50c08e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:23.691 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:23.689 7ff51182d700 1 --2- 172.21.2.112:0/1824937240 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff50c0c00c0 0x7ff50c08e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:23.691 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:23.689 7ff51182d700 1 -- 172.21.2.112:0/1824937240 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff50c0bfcc0 msgr2=0x7ff50c106c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:23.691 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:23.689 7ff51182d700 1 --2- 172.21.2.112:0/1824937240 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff50c0bfcc0 0x7ff50c106c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:23.691 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:23.689 7ff51182d700 1 -- 172.21.2.112:0/1824937240 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff50c08fdb0 con 0x7ff50c107230 2021-02-10T11:14:23.691 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:23.689 7ff51182d700 1 --2- 172.21.2.112:0/1824937240 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff50c107230 0x7ff50c08ed50 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7ff504001ba0 tx=0x7ff50400ac80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:23.691 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:23.689 7ff501ffb700 1 -- 172.21.2.112:0/1824937240 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff504000ce0 con 0x7ff50c107230 2021-02-10T11:14:23.691 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:23.689 7ff501ffb700 1 -- 172.21.2.112:0/1824937240 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5040070d0 con 0x7ff50c107230 2021-02-10T11:14:23.691 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:23.689 7ff501ffb700 1 -- 172.21.2.112:0/1824937240 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff5040048b0 con 0x7ff50c107230 2021-02-10T11:14:23.691 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:23.689 7ff513210700 1 -- 172.21.2.112:0/1824937240 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff50c08fdb0 con 0x7ff50c107230 2021-02-10T11:14:23.691 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:23.689 7ff513210700 1 -- 172.21.2.112:0/1824937240 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff50c08fdb0 con 0x7ff50c107230 2021-02-10T11:14:23.691 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:23.690 7ff501ffb700 1 -- 172.21.2.112:0/1824937240 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff5040072b0 con 0x7ff50c107230 2021-02-10T11:14:23.692 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:23.690 7ff501ffb700 1 --2- 172.21.2.112:0/1824937240 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff4ec0186c0 0x7ff4ec01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:23.692 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:23.690 7ff501ffb700 1 -- 172.21.2.112:0/1824937240 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7ff504032900 con 0x7ff50c107230 2021-02-10T11:14:23.692 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:23.690 7ff51102c700 1 --2- 172.21.2.112:0/1824937240 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff4ec0186c0 0x7ff4ec01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:23.692 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:23.691 7ff51102c700 1 --2- 172.21.2.112:0/1824937240 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff4ec0186c0 0x7ff4ec01abc0 crc :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:23.719 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:23.720 7ff513210700 1 -- 172.21.2.112:0/1824937240 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff50c105a10 con 0x7ff50c107230 2021-02-10T11:14:23.721 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:23.723 7ff501ffb700 1 -- 172.21.2.112:0/1824937240 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff504000cc0 con 0x7ff50c107230 2021-02-10T11:14:23.854 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:23.856 7ff513210700 1 -- 172.21.2.112:0/1824937240 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "health"} v 0) v1 -- 0x7ff50c0f0b50 con 0x7ff50c107230 2021-02-10T11:14:23.854 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:23.856 7ff501ffb700 1 -- 172.21.2.112:0/1824937240 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "health"}]=0 v0) v1 ==== 54+0+10 (secure 0 0 0) 0x7ff5040071a0 con 0x7ff50c107230 2021-02-10T11:14:23.855 INFO:teuthology.misc.health.gibba012.stdout:HEALTH_OK 2021-02-10T11:14:23.856 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:23.858 7ff513210700 1 -- 172.21.2.112:0/1824937240 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff4ec0186c0 msgr2=0x7ff4ec01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:23.856 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:23.858 7ff513210700 1 --2- 172.21.2.112:0/1824937240 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff4ec0186c0 0x7ff4ec01abc0 crc :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:23.856 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:23.858 7ff513210700 1 -- 172.21.2.112:0/1824937240 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff50c107230 msgr2=0x7ff50c08ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:23.856 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:23.858 7ff513210700 1 --2- 172.21.2.112:0/1824937240 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff50c107230 0x7ff50c08ed50 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7ff504001ba0 tx=0x7ff50400ac80).stop 2021-02-10T11:14:23.857 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:23.858 7ff513210700 1 -- 172.21.2.112:0/1824937240 shutdown_connections 2021-02-10T11:14:23.857 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:23.858 7ff513210700 1 --2- 172.21.2.112:0/1824937240 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff4ec0186c0 0x7ff4ec01abc0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:23.857 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:23.858 7ff513210700 1 --2- 172.21.2.112:0/1824937240 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff50c107230 0x7ff50c08ed50 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:23.857 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:23.858 7ff513210700 1 --2- 172.21.2.112:0/1824937240 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff50c0c00c0 0x7ff50c08e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:23.857 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:23.858 7ff513210700 1 --2- 172.21.2.112:0/1824937240 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff50c0bfcc0 0x7ff50c106c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:23.857 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:23.858 7ff513210700 1 -- 172.21.2.112:0/1824937240 >> 172.21.2.112:0/1824937240 conn(0x7ff50c0a2ae0 msgr2=0x7ff50c08b200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:23.857 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:23.858 7ff513210700 1 -- 172.21.2.112:0/1824937240 shutdown_connections 2021-02-10T11:14:23.857 INFO:teuthology.misc.health.gibba012.stderr:2021-02-10 11:14:23.858 7ff513210700 1 -- 172.21.2.112:0/1824937240 wait complete. 2021-02-10T11:14:23.864 DEBUG:teuthology.misc:Ceph health: HEALTH_OK 2021-02-10T11:14:23.864 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:14:24.290 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.274 7fde210b1700 1 Processor -- start 2021-02-10T11:14:24.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.274 7fde210b1700 1 -- start start 2021-02-10T11:14:24.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.274 7fde210b1700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fde1c0e5070 0x7fde1c1b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:24.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.274 7fde210b1700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fde1c0e5470 0x7fde1c080310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:24.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.274 7fde210b1700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde1c07ed80 con 0x7fde1c1b3e40 2021-02-10T11:14:24.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.274 7fde210b1700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde1c0d1790 con 0x7fde1c0e5070 2021-02-10T11:14:24.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.274 7fde210b1700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde1c0d19b0 con 0x7fde1c0e5470 2021-02-10T11:14:24.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.275 7fde1a59c700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fde1c1b3e40 0x7fde1c07d690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:41992/0 (socket says 172.21.2.112:41992) 2021-02-10T11:14:24.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.275 7fde1a59c700 1 -- 172.21.2.112:0/894852424 learned_addr learned my addr 172.21.2.112:0/894852424 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:14:24.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.275 7fde19d9b700 1 -- 172.21.2.112:0/894852424 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3959070703 0 0) 0x7fde0c000ce0 con 0x7fde1c1b3e40 2021-02-10T11:14:24.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.275 7fde19d9b700 1 -- 172.21.2.112:0/894852424 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde08002fe0 con 0x7fde1c1b3e40 2021-02-10T11:14:24.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.275 7fde1ad9d700 1 --2- 172.21.2.112:0/894852424 >> v2:172.21.2.120:3300/0 conn(0x7fde1c0e5070 0x7fde1c1b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:24.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.275 7fde1b59e700 1 --2- 172.21.2.112:0/894852424 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fde1c0e5470 0x7fde1c080310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:24.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.275 7fde19d9b700 1 -- 172.21.2.112:0/894852424 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2496348098 0 0) 0x7fde0c0010e0 con 0x7fde1c1b3e40 2021-02-10T11:14:24.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.275 7fde19d9b700 1 -- 172.21.2.112:0/894852424 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fde1c0e5470 msgr2=0x7fde1c080310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:14:24.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.275 7fde19d9b700 1 --2- 172.21.2.112:0/894852424 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fde1c0e5470 0x7fde1c080310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:24.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.275 7fde19d9b700 1 -- 172.21.2.112:0/894852424 >> v2:172.21.2.120:3300/0 conn(0x7fde1c0e5070 msgr2=0x7fde1c1b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:14:24.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.275 7fde19d9b700 1 --2- 172.21.2.112:0/894852424 >> v2:172.21.2.120:3300/0 conn(0x7fde1c0e5070 0x7fde1c1b3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:24.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.276 7fde19d9b700 1 -- 172.21.2.112:0/894852424 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde1c0d1bd0 con 0x7fde1c1b3e40 2021-02-10T11:14:24.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.276 7fde19d9b700 1 -- 172.21.2.112:0/894852424 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7fde0c001660 con 0x7fde1c1b3e40 2021-02-10T11:14:24.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.276 7fde19d9b700 1 -- 172.21.2.112:0/894852424 >> v1:172.21.2.112:6789/0 conn(0x7fde1c1b3e40 legacy=0x7fde1c07d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:24.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.276 7fde19d9b700 1 --2- 172.21.2.112:0/894852424 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fde08003a60 0x7fde08005f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:24.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.276 7fde19d9b700 1 --2- 172.21.2.112:0/894852424 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fde0804a7d0 0x7fde0804cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:24.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.276 7fde19d9b700 1 --2- 172.21.2.112:0/894852424 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fde0804d590 0x7fde0804fa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:24.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.276 7fde19d9b700 1 -- 172.21.2.112:0/894852424 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde080502d0 con 0x7fde08003a60 2021-02-10T11:14:24.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.276 7fde19d9b700 1 -- 172.21.2.112:0/894852424 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde080504f0 con 0x7fde0804a7d0 2021-02-10T11:14:24.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.276 7fde19d9b700 1 -- 172.21.2.112:0/894852424 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde08050710 con 0x7fde0804d590 2021-02-10T11:14:24.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.276 7fde1ad9d700 1 --2- 172.21.2.112:0/894852424 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fde08003a60 0x7fde08005f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:24.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.276 7fde1a59c700 1 --2- 172.21.2.112:0/894852424 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fde0804a7d0 0x7fde0804cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:24.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.276 7fde1b59e700 1 --2- 172.21.2.112:0/894852424 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fde0804d590 0x7fde0804fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:24.326 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.276 7fde1ad9d700 1 -- 172.21.2.112:0/894852424 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fde0804d590 msgr2=0x7fde0804fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:14:24.326 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.276 7fde1ad9d700 1 --2- 172.21.2.112:0/894852424 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fde0804d590 0x7fde0804fa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:24.326 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.276 7fde1ad9d700 1 -- 172.21.2.112:0/894852424 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fde0804a7d0 msgr2=0x7fde0804cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:14:24.326 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.276 7fde1ad9d700 1 --2- 172.21.2.112:0/894852424 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fde0804a7d0 0x7fde0804cc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:24.326 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.276 7fde1ad9d700 1 -- 172.21.2.112:0/894852424 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde080509b0 con 0x7fde08003a60 2021-02-10T11:14:24.326 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.276 7fde1b59e700 1 --2- 172.21.2.112:0/894852424 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fde0804d590 0x7fde0804fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:14:24.326 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.276 7fde1ad9d700 1 --2- 172.21.2.112:0/894852424 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fde08003a60 0x7fde08005f60 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7fde0400c8f0 tx=0x7fde0400cc90).ready entity=mon.0 client_cookie=a111371fde31c002 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:24.326 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.277 7fde19d9b700 1 -- 172.21.2.112:0/894852424 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fde0400cdd0 con 0x7fde08003a60 2021-02-10T11:14:24.326 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.277 7fde19d9b700 1 -- 172.21.2.112:0/894852424 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde04009310 con 0x7fde08003a60 2021-02-10T11:14:24.326 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.277 7fde19d9b700 1 -- 172.21.2.112:0/894852424 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fde04009830 con 0x7fde08003a60 2021-02-10T11:14:24.326 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.277 7fde210b1700 1 -- 172.21.2.112:0/894852424 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fde08003a60 msgr2=0x7fde08005f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:24.326 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.277 7fde210b1700 1 --2- 172.21.2.112:0/894852424 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fde08003a60 0x7fde08005f60 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7fde0400c8f0 tx=0x7fde0400cc90).stop 2021-02-10T11:14:24.326 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.277 7fde1b59e700 1 -- 172.21.2.112:0/894852424 reap_dead start 2021-02-10T11:14:24.327 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.277 7fde210b1700 1 -- 172.21.2.112:0/894852424 shutdown_connections 2021-02-10T11:14:24.327 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.277 7fde210b1700 1 -- 172.21.2.112:0/894852424 >> 172.21.2.112:0/894852424 conn(0x7fde1c0ce690 msgr2=0x7fde1c0e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:24.327 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.277 7fde210b1700 1 -- 172.21.2.112:0/894852424 shutdown_connections 2021-02-10T11:14:24.327 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.277 7fde210b1700 1 -- 172.21.2.112:0/894852424 wait complete. 2021-02-10T11:14:24.327 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.292 7fde210b1700 1 Processor -- start 2021-02-10T11:14:24.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.292 7fde210b1700 1 -- start start 2021-02-10T11:14:24.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.292 7fde210b1700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fde1c194000 0x7fde1c19f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:24.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.292 7fde210b1700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fde1c19fde0 0x7fde1c09bed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:24.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.292 7fde210b1700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fde1c1a01e0 0x7fde1c0895c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:24.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.292 7fde210b1700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde1c194400 con 0x7fde1c1a01e0 2021-02-10T11:14:24.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.292 7fde210b1700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde1c09c4f0 con 0x7fde1c194000 2021-02-10T11:14:24.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.292 7fde210b1700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde1c089be0 con 0x7fde1c19fde0 2021-02-10T11:14:24.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.292 7fde1b59e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fde1c1a01e0 0x7fde1c0895c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:24.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.293 7fde1b59e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fde1c1a01e0 0x7fde1c0895c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:42542/0 (socket says 172.21.2.112:42542) 2021-02-10T11:14:24.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.293 7fde1b59e700 1 -- 172.21.2.112:0/1807897636 learned_addr learned my addr 172.21.2.112:0/1807897636 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:14:24.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.293 7fde1b59e700 1 -- 172.21.2.112:0/1807897636 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fde1c19fde0 msgr2=0x7fde1c09bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:24.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.293 7fde1b59e700 1 --2- 172.21.2.112:0/1807897636 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fde1c19fde0 0x7fde1c09bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:24.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.293 7fde1b59e700 1 -- 172.21.2.112:0/1807897636 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fde1c194000 msgr2=0x7fde1c19f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:24.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.293 7fde1b59e700 1 --2- 172.21.2.112:0/1807897636 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fde1c194000 0x7fde1c19f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:24.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.293 7fde1b59e700 1 -- 172.21.2.112:0/1807897636 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde1c089e00 con 0x7fde1c1a01e0 2021-02-10T11:14:24.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.293 7fde1b59e700 1 --2- 172.21.2.112:0/1807897636 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fde1c1a01e0 0x7fde1c0895c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7fde140099e0 tx=0x7fde1400bd20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:24.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.293 7fde137fe700 1 -- 172.21.2.112:0/1807897636 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fde14001400 con 0x7fde1c1a01e0 2021-02-10T11:14:24.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.293 7fde137fe700 1 -- 172.21.2.112:0/1807897636 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde1400be60 con 0x7fde1c1a01e0 2021-02-10T11:14:24.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.293 7fde137fe700 1 -- 172.21.2.112:0/1807897636 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fde14001940 con 0x7fde1c1a01e0 2021-02-10T11:14:24.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.293 7fde210b1700 1 -- 172.21.2.112:0/1807897636 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde1c089e00 con 0x7fde1c1a01e0 2021-02-10T11:14:24.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.293 7fde210b1700 1 -- 172.21.2.112:0/1807897636 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde1c089e00 con 0x7fde1c1a01e0 2021-02-10T11:14:24.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.294 7fde137fe700 1 -- 172.21.2.112:0/1807897636 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fde140011f0 con 0x7fde1c1a01e0 2021-02-10T11:14:24.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.294 7fde137fe700 1 --2- 172.21.2.112:0/1807897636 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fde00018730 0x7fde0001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:24.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.294 7fde137fe700 1 -- 172.21.2.112:0/1807897636 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fde14032b70 con 0x7fde1c1a01e0 2021-02-10T11:14:24.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.295 7fde1ad9d700 1 --2- 172.21.2.112:0/1807897636 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fde00018730 0x7fde0001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:24.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.295 7fde1ad9d700 1 --2- 172.21.2.112:0/1807897636 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fde00018730 0x7fde0001abc0 crc :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:24.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.324 7fde210b1700 1 -- 172.21.2.112:0/1807897636 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde1c09c710 con 0x7fde1c1a01e0 2021-02-10T11:14:24.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.327 7fde137fe700 1 -- 172.21.2.112:0/1807897636 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fde14006c40 con 0x7fde1c1a01e0 2021-02-10T11:14:24.449 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.451 7fde210b1700 1 -- 172.21.2.112:0/1807897636 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fde1c089090 con 0x7fde1c1a01e0 2021-02-10T11:14:24.449 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.451 7fde137fe700 1 -- 172.21.2.112:0/1807897636 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+13993 (secure 0 0 0) 0x7fde14001ef0 con 0x7fde1c1a01e0 2021-02-10T11:14:24.450 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:14:24.450 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":20,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:14:04.629689","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:14:01.400797","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2021-02-10 11:14:02.622129","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:14:24.451 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.453 7fde210b1700 1 -- 172.21.2.112:0/1807897636 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fde00018730 msgr2=0x7fde0001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:24.451 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.453 7fde210b1700 1 --2- 172.21.2.112:0/1807897636 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fde00018730 0x7fde0001abc0 crc :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:24.451 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.453 7fde210b1700 1 -- 172.21.2.112:0/1807897636 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fde1c1a01e0 msgr2=0x7fde1c0895c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:24.451 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.453 7fde210b1700 1 --2- 172.21.2.112:0/1807897636 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fde1c1a01e0 0x7fde1c0895c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7fde140099e0 tx=0x7fde1400bd20).stop 2021-02-10T11:14:24.452 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.453 7fde210b1700 1 -- 172.21.2.112:0/1807897636 shutdown_connections 2021-02-10T11:14:24.452 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.453 7fde210b1700 1 --2- 172.21.2.112:0/1807897636 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fde00018730 0x7fde0001abc0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:24.452 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.453 7fde210b1700 1 --2- 172.21.2.112:0/1807897636 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fde1c1a01e0 0x7fde1c0895c0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:24.452 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.453 7fde210b1700 1 --2- 172.21.2.112:0/1807897636 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fde1c19fde0 0x7fde1c09bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:24.453 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.453 7fde210b1700 1 --2- 172.21.2.112:0/1807897636 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fde1c194000 0x7fde1c19f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:24.453 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.453 7fde210b1700 1 -- 172.21.2.112:0/1807897636 >> 172.21.2.112:0/1807897636 conn(0x7fde1c092c20 msgr2=0x7fde1c195070 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:24.453 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.453 7fde210b1700 1 -- 172.21.2.112:0/1807897636 shutdown_connections 2021-02-10T11:14:24.453 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.453 7fde210b1700 1 -- 172.21.2.112:0/1807897636 wait complete. 2021-02-10T11:14:24.460 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:13:57.508758', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:14:01.400797', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:14:02.622129', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:14:24.460 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:14:24.889 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.874 7ff12e46b700 1 Processor -- start 2021-02-10T11:14:24.889 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.875 7ff12e46b700 1 -- start start 2021-02-10T11:14:24.889 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.875 7ff12e46b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff1281b0470 0x7ff1281b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:24.889 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.875 7ff12e46b700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7ff1280ef610 0x7ff1280efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:24.889 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.875 7ff12e46b700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1280f0b50 con 0x7ff1280f0030 2021-02-10T11:14:24.889 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.875 7ff12e46b700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1280e52e0 con 0x7ff1280ef610 2021-02-10T11:14:24.890 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.875 7ff12e46b700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1280e5490 con 0x7ff1281b0470 2021-02-10T11:14:24.890 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.875 7ff12ca88700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7ff1280f0030 0x7ff1280f10d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:42012/0 (socket says 172.21.2.112:42012) 2021-02-10T11:14:24.890 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.875 7ff12ca88700 1 -- 172.21.2.112:0/2153716340 learned_addr learned my addr 172.21.2.112:0/2153716340 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:14:24.890 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.875 7ff1277fe700 1 --2- 172.21.2.112:0/2153716340 >> v2:172.21.2.112:3300/0 conn(0x7ff1280ef610 0x7ff1280efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:24.890 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.876 7ff1277fe700 1 -- 172.21.2.112:0/2153716340 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff1281b0470 msgr2=0x7ff1281b0870 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:14:24.890 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.876 7ff1277fe700 1 --2- 172.21.2.112:0/2153716340 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff1281b0470 0x7ff1281b0870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:24.890 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.876 7ff1277fe700 1 -- 172.21.2.112:0/2153716340 >> v1:172.21.2.112:6789/0 conn(0x7ff1280f0030 legacy=0x7ff1280f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:24.890 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.876 7ff1277fe700 1 -- 172.21.2.112:0/2153716340 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1280f0430 con 0x7ff1280ef610 2021-02-10T11:14:24.890 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.876 7ff1277fe700 1 --2- 172.21.2.112:0/2153716340 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff1280ef610 0x7ff1280efa10 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7ff11c006650 tx=0x7ff11c009a00).ready entity=mon.0 client_cookie=89ef3541c7684276 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:24.890 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.876 7ff126ffd700 1 -- 172.21.2.112:0/2153716340 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff11c009df0 con 0x7ff1280ef610 2021-02-10T11:14:24.890 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.876 7ff126ffd700 1 -- 172.21.2.112:0/2153716340 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff11c0009b0 con 0x7ff1280ef610 2021-02-10T11:14:24.890 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.876 7ff126ffd700 1 -- 172.21.2.112:0/2153716340 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff11c00c2b0 con 0x7ff1280ef610 2021-02-10T11:14:24.890 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.876 7ff12e46b700 1 -- 172.21.2.112:0/2153716340 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff1280ef610 msgr2=0x7ff1280efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:24.891 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.876 7ff12e46b700 1 --2- 172.21.2.112:0/2153716340 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff1280ef610 0x7ff1280efa10 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7ff11c006650 tx=0x7ff11c009a00).stop 2021-02-10T11:14:24.891 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.876 7ff12e46b700 1 -- 172.21.2.112:0/2153716340 shutdown_connections 2021-02-10T11:14:24.891 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.876 7ff12e46b700 1 --2- 172.21.2.112:0/2153716340 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff1280ef610 0x7ff1280efa10 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:24.891 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.876 7ff12e46b700 1 --2- 172.21.2.112:0/2153716340 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff1281b0470 0x7ff1281b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:24.891 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.876 7ff12e46b700 1 -- 172.21.2.112:0/2153716340 >> 172.21.2.112:0/2153716340 conn(0x7ff1280ce3e0 msgr2=0x7ff1280e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:24.891 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.877 7ff12e46b700 1 -- 172.21.2.112:0/2153716340 shutdown_connections 2021-02-10T11:14:24.891 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.877 7ff12e46b700 1 -- 172.21.2.112:0/2153716340 wait complete. 2021-02-10T11:14:24.891 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.891 7ff12e46b700 1 Processor -- start 2021-02-10T11:14:24.892 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.891 7ff12e46b700 1 -- start start 2021-02-10T11:14:24.892 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.891 7ff12e46b700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff1280c36a0 0x7ff1280c2b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:24.892 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.891 7ff12e46b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff1280c3190 0x7ff1280b1d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:24.893 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.891 7ff12e46b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff1280b2370 0x7ff1280b2d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:24.893 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.891 7ff12e46b700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1280b27c0 con 0x7ff1280c3190 2021-02-10T11:14:24.893 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.891 7ff12e46b700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1280b3360 con 0x7ff1280c36a0 2021-02-10T11:14:24.893 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.891 7ff12e46b700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1280b3540 con 0x7ff1280b2370 2021-02-10T11:14:24.893 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.892 7ff1277fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff1280c3190 0x7ff1280b1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:24.893 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.892 7ff1277fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff1280c3190 0x7ff1280b1d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:42554/0 (socket says 172.21.2.112:42554) 2021-02-10T11:14:24.893 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.892 7ff1277fe700 1 -- 172.21.2.112:0/3582706887 learned_addr learned my addr 172.21.2.112:0/3582706887 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:14:24.893 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.892 7ff1277fe700 1 -- 172.21.2.112:0/3582706887 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff1280b2370 msgr2=0x7ff1280b2d40 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:14:24.893 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.892 7ff1277fe700 1 --2- 172.21.2.112:0/3582706887 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff1280b2370 0x7ff1280b2d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:24.893 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.892 7ff1277fe700 1 -- 172.21.2.112:0/3582706887 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff1280c36a0 msgr2=0x7ff1280c2b70 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:14:24.893 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.892 7ff1277fe700 1 --2- 172.21.2.112:0/3582706887 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff1280c36a0 0x7ff1280c2b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:24.893 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.892 7ff1277fe700 1 -- 172.21.2.112:0/3582706887 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1280b3760 con 0x7ff1280c3190 2021-02-10T11:14:24.893 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.892 7ff1277fe700 1 --2- 172.21.2.112:0/3582706887 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff1280c3190 0x7ff1280b1d50 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7ff11c0096a0 tx=0x7ff11c00b9b0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:24.893 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.892 7ff124ff9700 1 -- 172.21.2.112:0/3582706887 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff11c000940 con 0x7ff1280c3190 2021-02-10T11:14:24.894 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.892 7ff124ff9700 1 -- 172.21.2.112:0/3582706887 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff11c0093a0 con 0x7ff1280c3190 2021-02-10T11:14:24.894 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.892 7ff124ff9700 1 -- 172.21.2.112:0/3582706887 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff11c002220 con 0x7ff1280c3190 2021-02-10T11:14:24.894 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.892 7ff12e46b700 1 -- 172.21.2.112:0/3582706887 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1280b3760 con 0x7ff1280c3190 2021-02-10T11:14:24.894 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.892 7ff12e46b700 1 -- 172.21.2.112:0/3582706887 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1280b3cf0 con 0x7ff1280c3190 2021-02-10T11:14:24.894 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.893 7ff124ff9700 1 -- 172.21.2.112:0/3582706887 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff11c0026f0 con 0x7ff1280c3190 2021-02-10T11:14:24.894 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.893 7ff124ff9700 1 --2- 172.21.2.112:0/3582706887 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff1080186c0 0x7ff10801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:24.894 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.893 7ff124ff9700 1 -- 172.21.2.112:0/3582706887 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7ff11c032580 con 0x7ff1280c3190 2021-02-10T11:14:24.894 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.894 7ff127fff700 1 --2- 172.21.2.112:0/3582706887 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff1080186c0 0x7ff10801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:24.894 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.895 7ff127fff700 1 --2- 172.21.2.112:0/3582706887 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff1080186c0 0x7ff10801abc0 crc :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:24.924 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.925 7ff12e46b700 1 -- 172.21.2.112:0/3582706887 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1280edd10 con 0x7ff1280c3190 2021-02-10T11:14:24.925 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:24.927 7ff124ff9700 1 -- 172.21.2.112:0/3582706887 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff11c030d30 con 0x7ff1280c3190 2021-02-10T11:14:25.055 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:25.056 7ff12e46b700 1 -- 172.21.2.112:0/3582706887 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7ff1280e52e0 con 0x7ff1280c3190 2021-02-10T11:14:25.055 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:25.057 7ff124ff9700 1 -- 172.21.2.112:0/3582706887 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v20) v1 ==== 92+0+10 (secure 0 0 0) 0x7ff11c00bda0 con 0x7ff1280c3190 2021-02-10T11:14:25.056 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:14:25.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:25.059 7ff12e46b700 1 -- 172.21.2.112:0/3582706887 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff1080186c0 msgr2=0x7ff10801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:25.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:25.059 7ff12e46b700 1 --2- 172.21.2.112:0/3582706887 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff1080186c0 0x7ff10801abc0 crc :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:25.058 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:25.059 7ff12e46b700 1 -- 172.21.2.112:0/3582706887 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff1280c3190 msgr2=0x7ff1280b1d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:25.058 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:25.059 7ff12e46b700 1 --2- 172.21.2.112:0/3582706887 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff1280c3190 0x7ff1280b1d50 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7ff11c0096a0 tx=0x7ff11c00b9b0).stop 2021-02-10T11:14:25.058 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:25.059 7ff12e46b700 1 -- 172.21.2.112:0/3582706887 shutdown_connections 2021-02-10T11:14:25.058 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:25.059 7ff12e46b700 1 --2- 172.21.2.112:0/3582706887 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff1080186c0 0x7ff10801abc0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:25.058 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:25.059 7ff12e46b700 1 --2- 172.21.2.112:0/3582706887 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff1280b2370 0x7ff1280b2d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:25.058 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:25.059 7ff12e46b700 1 --2- 172.21.2.112:0/3582706887 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff1280c3190 0x7ff1280b1d50 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:25.058 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:25.059 7ff12e46b700 1 --2- 172.21.2.112:0/3582706887 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff1280c36a0 0x7ff1280c2b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:25.058 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:25.059 7ff12e46b700 1 -- 172.21.2.112:0/3582706887 >> 172.21.2.112:0/3582706887 conn(0x7ff12809f630 msgr2=0x7ff1280a0130 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:25.058 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:25.059 7ff12e46b700 1 -- 172.21.2.112:0/3582706887 shutdown_connections 2021-02-10T11:14:25.059 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:25.059 7ff12e46b700 1 -- 172.21.2.112:0/3582706887 wait complete. 2021-02-10T11:14:25.066 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:14:25.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:25.480 7ff1a26a3700 1 Processor -- start 2021-02-10T11:14:25.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:25.480 7ff1a26a3700 1 -- start start 2021-02-10T11:14:25.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:25.480 7ff1a26a3700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7ff19c1b0470 0x7ff19c1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:25.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:25.480 7ff1a26a3700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7ff19c0f20e0 0x7ff19c0f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:25.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:25.480 7ff1a26a3700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff19c0f0b50 con 0x7ff19c0ef610 2021-02-10T11:14:25.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:25.480 7ff1a26a3700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff19c0f0820 con 0x7ff19c1b0470 2021-02-10T11:14:25.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:25.480 7ff1a26a3700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff19c0f24e0 con 0x7ff19c0f20e0 2021-02-10T11:14:25.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:25.480 7ff19bfff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7ff19c1b0470 0x7ff19c1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:25.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:25.480 7ff19bfff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7ff19c1b0470 0x7ff19c1b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:42562/0 (socket says 172.21.2.112:42562) 2021-02-10T11:14:25.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:25.480 7ff19bfff700 1 -- 172.21.2.112:0/601814774 learned_addr learned my addr 172.21.2.112:0/601814774 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:14:25.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:25.481 7ff1a0cc0700 1 --2- 172.21.2.112:0/601814774 >> v2:172.21.2.120:3300/0 conn(0x7ff19c0f20e0 0x7ff19c0f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:25.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:25.481 7ff19bfff700 1 -- 172.21.2.112:0/601814774 >> v2:172.21.2.120:3300/0 conn(0x7ff19c0f20e0 msgr2=0x7ff19c0f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:14:25.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:25.481 7ff19bfff700 1 --2- 172.21.2.112:0/601814774 >> v2:172.21.2.120:3300/0 conn(0x7ff19c0f20e0 0x7ff19c0f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:25.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:25.481 7ff19bfff700 1 -- 172.21.2.112:0/601814774 >> v1:172.21.2.112:6789/0 conn(0x7ff19c0ef610 legacy=0x7ff19c0efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:25.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:25.481 7ff19bfff700 1 -- 172.21.2.112:0/601814774 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff19c0e52e0 con 0x7ff19c1b0470 2021-02-10T11:14:25.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:25.481 7ff19bfff700 1 --2- 172.21.2.112:0/601814774 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff19c1b0470 0x7ff19c1b0870 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7ff194006a80 tx=0x7ff194009f00).ready entity=mon.0 client_cookie=5df281aec077708d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:25.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:25.481 7ff19affd700 1 -- 172.21.2.112:0/601814774 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff19400b5c0 con 0x7ff19c1b0470 2021-02-10T11:14:25.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:25.481 7ff19affd700 1 -- 172.21.2.112:0/601814774 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff19400b8c0 con 0x7ff19c1b0470 2021-02-10T11:14:25.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:25.481 7ff19affd700 1 -- 172.21.2.112:0/601814774 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff19400bda0 con 0x7ff19c1b0470 2021-02-10T11:14:25.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:25.481 7ff1a26a3700 1 -- 172.21.2.112:0/601814774 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff19c1b0470 msgr2=0x7ff19c1b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:25.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:25.481 7ff1a26a3700 1 --2- 172.21.2.112:0/601814774 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff19c1b0470 0x7ff19c1b0870 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7ff194006a80 tx=0x7ff194009f00).stop 2021-02-10T11:14:25.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:25.482 7ff1a26a3700 1 -- 172.21.2.112:0/601814774 shutdown_connections 2021-02-10T11:14:25.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:25.482 7ff1a26a3700 1 --2- 172.21.2.112:0/601814774 >> v2:172.21.2.120:3300/0 conn(0x7ff19c0f20e0 0x7ff19c0f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:25.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:25.482 7ff1a26a3700 1 --2- 172.21.2.112:0/601814774 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff19c1b0470 0x7ff19c1b0870 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:25.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:25.482 7ff1a26a3700 1 -- 172.21.2.112:0/601814774 >> 172.21.2.112:0/601814774 conn(0x7ff19c0ce3e0 msgr2=0x7ff19c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:25.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:25.482 7ff1a26a3700 1 -- 172.21.2.112:0/601814774 shutdown_connections 2021-02-10T11:14:25.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:25.482 7ff1a26a3700 1 -- 172.21.2.112:0/601814774 wait complete. 2021-02-10T11:14:25.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:25.497 7ff1a26a3700 1 Processor -- start 2021-02-10T11:14:25.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:25.497 7ff1a26a3700 1 -- start start 2021-02-10T11:14:25.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:25.497 7ff1a26a3700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff19c190920 0x7ff19c0b81a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:25.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:25.497 7ff1a26a3700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff19c0b87c0 0x7ff19c0b70c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:25.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:25.497 7ff1a26a3700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff19c0b8bc0 0x7ff19c0b76e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:25.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:25.497 7ff1a26a3700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff19c190d20 con 0x7ff19c190920 2021-02-10T11:14:25.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:25.497 7ff1a26a3700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff19c0b5ad0 con 0x7ff19c0b8bc0 2021-02-10T11:14:25.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:25.497 7ff1a26a3700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff19c0b7d00 con 0x7ff19c0b87c0 2021-02-10T11:14:25.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:25.497 7ff19bfff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff19c190920 0x7ff19c0b81a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:25.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:25.497 7ff19bfff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff19c190920 0x7ff19c0b81a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:42572/0 (socket says 172.21.2.112:42572) 2021-02-10T11:14:25.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:25.497 7ff19bfff700 1 -- 172.21.2.112:0/537164251 learned_addr learned my addr 172.21.2.112:0/537164251 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:14:25.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:25.498 7ff1a0cc0700 1 --2- 172.21.2.112:0/537164251 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff19c0b8bc0 0x7ff19c0b76e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:25.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:25.498 7ff19b7fe700 1 --2- 172.21.2.112:0/537164251 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff19c0b87c0 0x7ff19c0b70c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:25.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:25.498 7ff19bfff700 1 -- 172.21.2.112:0/537164251 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff19c0b87c0 msgr2=0x7ff19c0b70c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:25.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:25.498 7ff19bfff700 1 --2- 172.21.2.112:0/537164251 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff19c0b87c0 0x7ff19c0b70c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:25.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:25.498 7ff19bfff700 1 -- 172.21.2.112:0/537164251 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff19c0b8bc0 msgr2=0x7ff19c0b76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:25.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:25.498 7ff19bfff700 1 --2- 172.21.2.112:0/537164251 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff19c0b8bc0 0x7ff19c0b76e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:25.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:25.498 7ff19bfff700 1 -- 172.21.2.112:0/537164251 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff19c0a4590 con 0x7ff19c190920 2021-02-10T11:14:25.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:25.498 7ff19b7fe700 1 --2- 172.21.2.112:0/537164251 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff19c0b87c0 0x7ff19c0b70c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:14:25.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:25.498 7ff1a0cc0700 1 --2- 172.21.2.112:0/537164251 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff19c0b8bc0 0x7ff19c0b76e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:14:25.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:25.498 7ff19bfff700 1 --2- 172.21.2.112:0/537164251 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff19c190920 0x7ff19c0b81a0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7ff194009450 tx=0x7ff19400bd20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:25.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:25.498 7ff198ff9700 1 -- 172.21.2.112:0/537164251 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff19400c2f0 con 0x7ff19c190920 2021-02-10T11:14:25.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:25.498 7ff198ff9700 1 -- 172.21.2.112:0/537164251 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff19400be60 con 0x7ff19c190920 2021-02-10T11:14:25.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:25.498 7ff198ff9700 1 -- 172.21.2.112:0/537164251 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff19400c830 con 0x7ff19c190920 2021-02-10T11:14:25.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:25.498 7ff1a26a3700 1 -- 172.21.2.112:0/537164251 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff19c0a4590 con 0x7ff19c190920 2021-02-10T11:14:25.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:25.498 7ff1a26a3700 1 -- 172.21.2.112:0/537164251 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff19c0a4b20 con 0x7ff19c190920 2021-02-10T11:14:25.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:25.499 7ff198ff9700 1 -- 172.21.2.112:0/537164251 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff194008020 con 0x7ff19c190920 2021-02-10T11:14:25.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:25.499 7ff198ff9700 1 --2- 172.21.2.112:0/537164251 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff17c0186c0 0x7ff17c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:25.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:25.499 7ff198ff9700 1 -- 172.21.2.112:0/537164251 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7ff194032920 con 0x7ff19c190920 2021-02-10T11:14:25.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:25.499 7ff19b7fe700 1 --2- 172.21.2.112:0/537164251 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff17c0186c0 0x7ff17c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:25.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:25.500 7ff19b7fe700 1 --2- 172.21.2.112:0/537164251 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff17c0186c0 0x7ff17c01abc0 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:25.528 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:25.530 7ff1a26a3700 1 -- 172.21.2.112:0/537164251 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff19c105ae0 con 0x7ff19c190920 2021-02-10T11:14:25.531 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:25.533 7ff198ff9700 1 -- 172.21.2.112:0/537164251 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff194000d30 con 0x7ff19c190920 2021-02-10T11:14:25.658 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:25.659 7ff1a26a3700 1 -- 172.21.2.112:0/537164251 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7ff19c0f0b50 con 0x7ff19c190920 2021-02-10T11:14:25.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:25.660 7ff198ff9700 1 -- 172.21.2.112:0/537164251 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v20) v1 ==== 104+0+10 (secure 0 0 0) 0x7ff194007fe0 con 0x7ff19c190920 2021-02-10T11:14:25.659 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:14:25.660 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:25.662 7ff1a26a3700 1 -- 172.21.2.112:0/537164251 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff17c0186c0 msgr2=0x7ff17c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:25.660 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:25.662 7ff1a26a3700 1 --2- 172.21.2.112:0/537164251 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff17c0186c0 0x7ff17c01abc0 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:25.660 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:25.662 7ff1a26a3700 1 -- 172.21.2.112:0/537164251 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff19c190920 msgr2=0x7ff19c0b81a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:25.660 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:25.662 7ff1a26a3700 1 --2- 172.21.2.112:0/537164251 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff19c190920 0x7ff19c0b81a0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7ff194009450 tx=0x7ff19400bd20).stop 2021-02-10T11:14:25.660 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:25.662 7ff1a26a3700 1 -- 172.21.2.112:0/537164251 shutdown_connections 2021-02-10T11:14:25.660 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:25.662 7ff1a26a3700 1 --2- 172.21.2.112:0/537164251 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff17c0186c0 0x7ff17c01abc0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:25.660 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:25.662 7ff1a26a3700 1 --2- 172.21.2.112:0/537164251 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff19c0b8bc0 0x7ff19c0b76e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:25.661 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:25.662 7ff1a26a3700 1 --2- 172.21.2.112:0/537164251 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff19c0b87c0 0x7ff19c0b70c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:25.661 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:25.662 7ff1a26a3700 1 --2- 172.21.2.112:0/537164251 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff19c190920 0x7ff19c0b81a0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:25.661 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:25.662 7ff1a26a3700 1 -- 172.21.2.112:0/537164251 >> 172.21.2.112:0/537164251 conn(0x7ff19c08ffe0 msgr2=0x7ff19c0bf050 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:25.661 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:25.662 7ff1a26a3700 1 -- 172.21.2.112:0/537164251 shutdown_connections 2021-02-10T11:14:25.661 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:25.662 7ff1a26a3700 1 -- 172.21.2.112:0/537164251 wait complete. 2021-02-10T11:14:25.669 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:14:26.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.082 7f812489b700 1 Processor -- start 2021-02-10T11:14:26.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.082 7f812489b700 1 -- start start 2021-02-10T11:14:26.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.083 7f812489b700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f81201b0470 0x7f81201b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:26.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.083 7f812489b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f81200ef610 0x7f81200efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:26.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.083 7f812489b700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81200f0b50 con 0x7f81200f0030 2021-02-10T11:14:26.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.083 7f812489b700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81200e52e0 con 0x7f81201b0470 2021-02-10T11:14:26.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.083 7f812489b700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81200e54d0 con 0x7f81200ef610 2021-02-10T11:14:26.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.084 7f811e61c700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f81201b0470 0x7f81201b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:26.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.084 7f811e61c700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f81201b0470 0x7f81201b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3300/0 says I am v2:172.21.2.112:44856/0 (socket says 172.21.2.112:44856) 2021-02-10T11:14:26.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.084 7f811e61c700 1 -- 172.21.2.112:0/915933171 learned_addr learned my addr 172.21.2.112:0/915933171 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:14:26.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.084 7f811de1b700 1 --2- 172.21.2.112:0/915933171 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f81200ef610 0x7f81200efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:26.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.085 7f811de1b700 1 -- 172.21.2.112:0/915933171 >> v2:172.21.2.120:3300/0 conn(0x7f81201b0470 msgr2=0x7f81201b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:14:26.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.085 7f811de1b700 1 --2- 172.21.2.112:0/915933171 >> v2:172.21.2.120:3300/0 conn(0x7f81201b0470 0x7f81201b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:26.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.085 7f811de1b700 1 -- 172.21.2.112:0/915933171 >> v1:172.21.2.120:6789/0 conn(0x7f81200f0030 legacy=0x7f81200f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:26.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.085 7f811de1b700 1 -- 172.21.2.112:0/915933171 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81200f0430 con 0x7f81200ef610 2021-02-10T11:14:26.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.085 7f811de1b700 1 --2- 172.21.2.112:0/915933171 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f81200ef610 0x7f81200efa10 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f8108006650 tx=0x7f8108009a80).ready entity=mon.2 client_cookie=d26ae35a85406ffc server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:26.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.085 7f811d61a700 1 -- 172.21.2.112:0/915933171 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f81080009b0 con 0x7f81200ef610 2021-02-10T11:14:26.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.085 7f811d61a700 1 -- 172.21.2.112:0/915933171 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f810800c0c0 con 0x7f81200ef610 2021-02-10T11:14:26.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.085 7f811d61a700 1 -- 172.21.2.112:0/915933171 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f810800c5e0 con 0x7f81200ef610 2021-02-10T11:14:26.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.085 7f812489b700 1 -- 172.21.2.112:0/915933171 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f81200ef610 msgr2=0x7f81200efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:26.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.085 7f812489b700 1 --2- 172.21.2.112:0/915933171 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f81200ef610 0x7f81200efa10 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f8108006650 tx=0x7f8108009a80).stop 2021-02-10T11:14:26.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.086 7f812489b700 1 -- 172.21.2.112:0/915933171 shutdown_connections 2021-02-10T11:14:26.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.086 7f812489b700 1 --2- 172.21.2.112:0/915933171 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f81200ef610 0x7f81200efa10 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:26.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.086 7f812489b700 1 --2- 172.21.2.112:0/915933171 >> v2:172.21.2.120:3300/0 conn(0x7f81201b0470 0x7f81201b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:26.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.086 7f812489b700 1 -- 172.21.2.112:0/915933171 >> 172.21.2.112:0/915933171 conn(0x7f81200ce3e0 msgr2=0x7f81200e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:26.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.086 7f812489b700 1 -- 172.21.2.112:0/915933171 shutdown_connections 2021-02-10T11:14:26.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.086 7f812489b700 1 -- 172.21.2.112:0/915933171 wait complete. 2021-02-10T11:14:26.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.100 7f812489b700 1 Processor -- start 2021-02-10T11:14:26.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.101 7f812489b700 1 -- start start 2021-02-10T11:14:26.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.101 7f812489b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f81200a0420 0x7f81200a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:26.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.101 7f812489b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8120189d00 0x7f812018a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:26.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.101 7f812489b700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f812018a720 0x7f81200b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:26.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.101 7f812489b700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8120189780 con 0x7f8120189d00 2021-02-10T11:14:26.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.101 7f812489b700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f812018ab20 con 0x7f812018a720 2021-02-10T11:14:26.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.101 7f812489b700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81200a1140 con 0x7f81200a0420 2021-02-10T11:14:26.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.101 7f811de1b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8120189d00 0x7f812018a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:26.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.101 7f811de1b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8120189d00 0x7f812018a100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:42582/0 (socket says 172.21.2.112:42582) 2021-02-10T11:14:26.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.101 7f811de1b700 1 -- 172.21.2.112:0/3670257154 learned_addr learned my addr 172.21.2.112:0/3670257154 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:14:26.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.101 7f811ee1d700 1 --2- 172.21.2.112:0/3670257154 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f812018a720 0x7f81200b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:26.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.101 7f811e61c700 1 --2- 172.21.2.112:0/3670257154 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f81200a0420 0x7f81200a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:26.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.102 7f811de1b700 1 -- 172.21.2.112:0/3670257154 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f81200a0420 msgr2=0x7f81200a0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:26.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.102 7f811de1b700 1 --2- 172.21.2.112:0/3670257154 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f81200a0420 0x7f81200a0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:26.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.102 7f811de1b700 1 -- 172.21.2.112:0/3670257154 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f812018a720 msgr2=0x7f81200b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:26.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.102 7f811de1b700 1 --2- 172.21.2.112:0/3670257154 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f812018a720 0x7f81200b7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:26.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.102 7f811de1b700 1 -- 172.21.2.112:0/3670257154 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81200b8200 con 0x7f8120189d00 2021-02-10T11:14:26.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.102 7f811ee1d700 1 --2- 172.21.2.112:0/3670257154 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f812018a720 0x7f81200b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:14:26.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.102 7f811de1b700 1 --2- 172.21.2.112:0/3670257154 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8120189d00 0x7f812018a100 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f8108009630 tx=0x7f810800b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:26.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.102 7f8116ffd700 1 -- 172.21.2.112:0/3670257154 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f81080093a0 con 0x7f8120189d00 2021-02-10T11:14:26.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.102 7f8116ffd700 1 -- 172.21.2.112:0/3670257154 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f810800bd70 con 0x7f8120189d00 2021-02-10T11:14:26.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.102 7f8116ffd700 1 -- 172.21.2.112:0/3670257154 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8108005560 con 0x7f8120189d00 2021-02-10T11:14:26.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.102 7f812489b700 1 -- 172.21.2.112:0/3670257154 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81200b8200 con 0x7f8120189d00 2021-02-10T11:14:26.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.102 7f812489b700 1 -- 172.21.2.112:0/3670257154 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81200b8790 con 0x7f8120189d00 2021-02-10T11:14:26.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.103 7f8116ffd700 1 -- 172.21.2.112:0/3670257154 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f810800a000 con 0x7f8120189d00 2021-02-10T11:14:26.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.103 7f8116ffd700 1 --2- 172.21.2.112:0/3670257154 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f81040186c0 0x7f810401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:26.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.103 7f8116ffd700 1 -- 172.21.2.112:0/3670257154 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f81080327a0 con 0x7f8120189d00 2021-02-10T11:14:26.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.103 7f811e61c700 1 --2- 172.21.2.112:0/3670257154 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f81040186c0 0x7f810401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:26.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.104 7f811e61c700 1 --2- 172.21.2.112:0/3670257154 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f81040186c0 0x7f810401abc0 crc :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:26.131 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.133 7f812489b700 1 -- 172.21.2.112:0/3670257154 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8120105ae0 con 0x7f8120189d00 2021-02-10T11:14:26.133 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.135 7f8116ffd700 1 -- 172.21.2.112:0/3670257154 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8108009340 con 0x7f8120189d00 2021-02-10T11:14:26.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.262 7f812489b700 1 -- 172.21.2.112:0/3670257154 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f81200f0b50 con 0x7f8120189d00 2021-02-10T11:14:26.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.262 7f8116ffd700 1 -- 172.21.2.112:0/3670257154 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v20) v1 ==== 100+0+10 (secure 0 0 0) 0x7f810800bd70 con 0x7f8120189d00 2021-02-10T11:14:26.261 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:14:26.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.264 7f812489b700 1 -- 172.21.2.112:0/3670257154 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f81040186c0 msgr2=0x7f810401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:26.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.264 7f812489b700 1 --2- 172.21.2.112:0/3670257154 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f81040186c0 0x7f810401abc0 crc :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:26.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.264 7f812489b700 1 -- 172.21.2.112:0/3670257154 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8120189d00 msgr2=0x7f812018a100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:26.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.264 7f812489b700 1 --2- 172.21.2.112:0/3670257154 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8120189d00 0x7f812018a100 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f8108009630 tx=0x7f810800b980).stop 2021-02-10T11:14:26.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.264 7f812489b700 1 -- 172.21.2.112:0/3670257154 shutdown_connections 2021-02-10T11:14:26.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.264 7f812489b700 1 --2- 172.21.2.112:0/3670257154 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f81040186c0 0x7f810401abc0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:26.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.264 7f812489b700 1 --2- 172.21.2.112:0/3670257154 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f812018a720 0x7f81200b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:26.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.264 7f812489b700 1 --2- 172.21.2.112:0/3670257154 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8120189d00 0x7f812018a100 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:26.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.264 7f812489b700 1 --2- 172.21.2.112:0/3670257154 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f81200a0420 0x7f81200a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:26.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.265 7f812489b700 1 -- 172.21.2.112:0/3670257154 >> 172.21.2.112:0/3670257154 conn(0x7f8120094e10 msgr2=0x7f81200a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:26.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.265 7f812489b700 1 -- 172.21.2.112:0/3670257154 shutdown_connections 2021-02-10T11:14:26.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.265 7f812489b700 1 -- 172.21.2.112:0/3670257154 wait complete. 2021-02-10T11:14:26.271 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:14:26.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.682 7f686eb9c700 1 Processor -- start 2021-02-10T11:14:26.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.682 7f686eb9c700 1 -- start start 2021-02-10T11:14:26.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.683 7f686eb9c700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f68680e5470 0x7f686807e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:26.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.683 7f686eb9c700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f686807f300 0x7f686807f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:26.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.683 7f686eb9c700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f686807ed80 con 0x7f68680e5070 2021-02-10T11:14:26.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.683 7f686eb9c700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68681b4010 con 0x7f686807f300 2021-02-10T11:14:26.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.683 7f686eb9c700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68680d1790 con 0x7f68680e5470 2021-02-10T11:14:26.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.683 7f686d1b9700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f686807f300 0x7f686807f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:26.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.683 7f686d1b9700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f686807f300 0x7f686807f700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:42588/0 (socket says 172.21.2.112:42588) 2021-02-10T11:14:26.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.683 7f686d1b9700 1 -- 172.21.2.112:0/2112997946 learned_addr learned my addr 172.21.2.112:0/2112997946 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:14:26.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.683 7f686d1b9700 1 -- 172.21.2.112:0/2112997946 >> v2:172.21.2.120:3300/0 conn(0x7f68680e5470 msgr2=0x7f686807e6a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:14:26.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.683 7f686d1b9700 1 --2- 172.21.2.112:0/2112997946 >> v2:172.21.2.120:3300/0 conn(0x7f68680e5470 0x7f686807e6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:26.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.683 7f686d1b9700 1 -- 172.21.2.112:0/2112997946 >> v1:172.21.2.120:6789/0 conn(0x7f68680e5070 legacy=0x7f68681b3820 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:14:26.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.683 7f686d1b9700 1 -- 172.21.2.112:0/2112997946 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68680d19b0 con 0x7f686807f300 2021-02-10T11:14:26.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.683 7f686d1b9700 1 --2- 172.21.2.112:0/2112997946 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f686807f300 0x7f686807f700 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f6860006b80 tx=0x7f6860009b90).ready entity=mon.0 client_cookie=a8915e961d6ed375 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:26.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.683 7f68677fe700 1 -- 172.21.2.112:0/2112997946 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f686000c0c0 con 0x7f686807f300 2021-02-10T11:14:26.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.683 7f68677fe700 1 -- 172.21.2.112:0/2112997946 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f686000c340 con 0x7f686807f300 2021-02-10T11:14:26.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.683 7f68677fe700 1 -- 172.21.2.112:0/2112997946 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f686000c820 con 0x7f686807f300 2021-02-10T11:14:26.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.683 7f686eb9c700 1 -- 172.21.2.112:0/2112997946 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f686807f300 msgr2=0x7f686807f700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:26.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.683 7f686eb9c700 1 --2- 172.21.2.112:0/2112997946 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f686807f300 0x7f686807f700 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f6860006b80 tx=0x7f6860009b90).stop 2021-02-10T11:14:26.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.684 7f686eb9c700 1 -- 172.21.2.112:0/2112997946 shutdown_connections 2021-02-10T11:14:26.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.684 7f686eb9c700 1 --2- 172.21.2.112:0/2112997946 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f686807f300 0x7f686807f700 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:26.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.684 7f686eb9c700 1 --2- 172.21.2.112:0/2112997946 >> v2:172.21.2.120:3300/0 conn(0x7f68680e5470 0x7f686807e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:26.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.684 7f686eb9c700 1 -- 172.21.2.112:0/2112997946 >> 172.21.2.112:0/2112997946 conn(0x7f68680ce690 msgr2=0x7f68680e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:26.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.684 7f686eb9c700 1 -- 172.21.2.112:0/2112997946 shutdown_connections 2021-02-10T11:14:26.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.684 7f686eb9c700 1 -- 172.21.2.112:0/2112997946 wait complete. 2021-02-10T11:14:26.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.698 7f686eb9c700 1 Processor -- start 2021-02-10T11:14:26.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.698 7f686eb9c700 1 -- start start 2021-02-10T11:14:26.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.699 7f686eb9c700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f68680afa10 0x7f68680afe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:26.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.699 7f686eb9c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f686808f6e0 0x7f686808fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:26.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.699 7f686eb9c700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f686819a830 0x7f686819ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:26.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.699 7f686eb9c700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f686819b250 con 0x7f686808f6e0 2021-02-10T11:14:26.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.699 7f686eb9c700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f686819b470 con 0x7f68680afa10 2021-02-10T11:14:26.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.699 7f686eb9c700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f686819b690 con 0x7f686819a830 2021-02-10T11:14:26.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.699 7f6867fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f686808f6e0 0x7f686808fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:26.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.699 7f6867fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f686808f6e0 0x7f686808fae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:42592/0 (socket says 172.21.2.112:42592) 2021-02-10T11:14:26.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.699 7f6867fff700 1 -- 172.21.2.112:0/587471601 learned_addr learned my addr 172.21.2.112:0/587471601 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:14:26.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.699 7f6867fff700 1 -- 172.21.2.112:0/587471601 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f686819a830 msgr2=0x7f686819ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:26.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.699 7f6867fff700 1 --2- 172.21.2.112:0/587471601 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f686819a830 0x7f686819ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:26.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.699 7f6867fff700 1 -- 172.21.2.112:0/587471601 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f68680afa10 msgr2=0x7f68680afe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:26.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.699 7f6867fff700 1 --2- 172.21.2.112:0/587471601 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f68680afa10 0x7f68680afe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:26.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.699 7f6867fff700 1 -- 172.21.2.112:0/587471601 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f686819b8b0 con 0x7f686808f6e0 2021-02-10T11:14:26.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.700 7f6867fff700 1 --2- 172.21.2.112:0/587471601 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f686808f6e0 0x7f686808fae0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f685800dae0 tx=0x7f685800de80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:26.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.700 7f68657fa700 1 -- 172.21.2.112:0/587471601 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f68580030d0 con 0x7f686808f6e0 2021-02-10T11:14:26.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.700 7f68657fa700 1 -- 172.21.2.112:0/587471601 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6858005850 con 0x7f686808f6e0 2021-02-10T11:14:26.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.700 7f68657fa700 1 -- 172.21.2.112:0/587471601 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6858005ca0 con 0x7f686808f6e0 2021-02-10T11:14:26.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.700 7f686eb9c700 1 -- 172.21.2.112:0/587471601 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f686819b8b0 con 0x7f686808f6e0 2021-02-10T11:14:26.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.700 7f686eb9c700 1 -- 172.21.2.112:0/587471601 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68681a6600 con 0x7f686808f6e0 2021-02-10T11:14:26.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.701 7f68657fa700 1 -- 172.21.2.112:0/587471601 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6858005a30 con 0x7f686808f6e0 2021-02-10T11:14:26.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.701 7f68657fa700 1 --2- 172.21.2.112:0/587471601 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f684c0186c0 0x7f684c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:26.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.701 7f68657fa700 1 -- 172.21.2.112:0/587471601 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f6858034b30 con 0x7f686808f6e0 2021-02-10T11:14:26.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.701 7f686c9b8700 1 --2- 172.21.2.112:0/587471601 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f684c0186c0 0x7f684c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:26.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.702 7f686c9b8700 1 --2- 172.21.2.112:0/587471601 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f684c0186c0 0x7f684c01abc0 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:26.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.732 7f686eb9c700 1 -- 172.21.2.112:0/587471601 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6868083da0 con 0x7f686808f6e0 2021-02-10T11:14:26.733 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.735 7f68657fa700 1 -- 172.21.2.112:0/587471601 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6858000fc0 con 0x7f686808f6e0 2021-02-10T11:14:26.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.865 7f686eb9c700 1 -- 172.21.2.112:0/587471601 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f686807ed80 con 0x7f686808f6e0 2021-02-10T11:14:26.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.866 7f68657fa700 1 -- 172.21.2.112:0/587471601 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+2097 (secure 0 0 0) 0x7f6858005a10 con 0x7f686808f6e0 2021-02-10T11:14:26.865 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:14:26.865 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4279,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.120:6835/2695560847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":2695560847},{"type":"v1","addr":"172.21.2.120:6835","nonce":2695560847}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":4}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:14:03.627691","modified":"2021-02-10 11:14:13.011189","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4272},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4272":{"gid":4272,"name":"a","rank":0,"incarnation":5,"state":"up:active","state_seq":3,"addr":"172.21.2.112:6833/2584920075","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":2584920075},{"type":"v1","addr":"172.21.2.112:6833","nonce":2584920075}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2021-02-10T11:14:26.865 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 6 2021-02-10T11:14:26.866 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.868 7f686eb9c700 1 -- 172.21.2.112:0/587471601 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f684c0186c0 msgr2=0x7f684c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:26.866 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.868 7f686eb9c700 1 --2- 172.21.2.112:0/587471601 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f684c0186c0 0x7f684c01abc0 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:26.866 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.868 7f686eb9c700 1 -- 172.21.2.112:0/587471601 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f686808f6e0 msgr2=0x7f686808fae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:26.866 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.868 7f686eb9c700 1 --2- 172.21.2.112:0/587471601 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f686808f6e0 0x7f686808fae0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f685800dae0 tx=0x7f685800de80).stop 2021-02-10T11:14:26.867 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.868 7f686eb9c700 1 -- 172.21.2.112:0/587471601 shutdown_connections 2021-02-10T11:14:26.867 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.868 7f686eb9c700 1 --2- 172.21.2.112:0/587471601 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f684c0186c0 0x7f684c01abc0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:26.867 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.868 7f686eb9c700 1 --2- 172.21.2.112:0/587471601 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f686819a830 0x7f686819ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:26.867 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.868 7f686eb9c700 1 --2- 172.21.2.112:0/587471601 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f686808f6e0 0x7f686808fae0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:26.867 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.868 7f686eb9c700 1 --2- 172.21.2.112:0/587471601 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f68680afa10 0x7f68680afe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:26.867 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.868 7f686eb9c700 1 -- 172.21.2.112:0/587471601 >> 172.21.2.112:0/587471601 conn(0x7f686819e060 msgr2=0x7f68681b20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:26.867 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.868 7f686eb9c700 1 -- 172.21.2.112:0/587471601 shutdown_connections 2021-02-10T11:14:26.867 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:26.868 7f686eb9c700 1 -- 172.21.2.112:0/587471601 wait complete. 2021-02-10T11:14:27.077 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 6, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:14:03.627691', 'modified': '2021-02-10 11:14:13.011189', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4272}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4272': {'gid': 4272, 'name': 'a', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.112:6833/2584920075', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.112:6832', 'nonce': 2584920075}, {'type': 'v1', 'addr': '172.21.2.112:6833', 'nonce': 2584920075}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:14:27.155 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:14:27.156 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok status 2021-02-10T11:14:27.561 INFO:teuthology.orchestra.run.gibba012.stdout:{ 2021-02-10T11:14:27.561 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", 2021-02-10T11:14:27.561 INFO:teuthology.orchestra.run.gibba012.stdout: "whoami": 0, 2021-02-10T11:14:27.561 INFO:teuthology.orchestra.run.gibba012.stdout: "id": 4272, 2021-02-10T11:14:27.561 INFO:teuthology.orchestra.run.gibba012.stdout: "want_state": "up:active", 2021-02-10T11:14:27.561 INFO:teuthology.orchestra.run.gibba012.stdout: "state": "up:active", 2021-02-10T11:14:27.562 INFO:teuthology.orchestra.run.gibba012.stdout: "rank_uptime": 15.5516799, 2021-02-10T11:14:27.562 INFO:teuthology.orchestra.run.gibba012.stdout: "mdsmap_epoch": 6, 2021-02-10T11:14:27.562 INFO:teuthology.orchestra.run.gibba012.stdout: "osdmap_epoch": 20, 2021-02-10T11:14:27.562 INFO:teuthology.orchestra.run.gibba012.stdout: "osdmap_epoch_barrier": 20, 2021-02-10T11:14:27.562 INFO:teuthology.orchestra.run.gibba012.stdout: "uptime": 19.918723026999999 2021-02-10T11:14:27.562 INFO:teuthology.orchestra.run.gibba012.stdout:} 2021-02-10T11:14:27.565 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", "whoami": 0, "id": 4272, "want_state": "up:active", "state": "up:active", "rank_uptime": 15.5516799, "mdsmap_epoch": 6, "osdmap_epoch": 20, "osdmap_epoch_barrier": 20, "uptime": 19.918723026999999 } 2021-02-10T11:14:27.566 INFO:teuthology.run_tasks:Running task ceph-fuse... 2021-02-10T11:14:27.582 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2021-02-10T11:14:27.583 INFO:tasks.ceph_fuse:config is None 2021-02-10T11:14:27.583 INFO:tasks.ceph_fuse:new config is {'client.0': None, 'client.1': None, 'client.2': None, 'client.3': None} 2021-02-10T11:14:27.583 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2021-02-10T11:14:27.586 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:14:28.015 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:27.998 7f6017445700 1 Processor -- start 2021-02-10T11:14:28.015 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:27.998 7f6017445700 1 -- start start 2021-02-10T11:14:28.015 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:27.999 7f6017445700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f60100e5470 0x7f601007e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:28.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:27.999 7f6017445700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f601007ed80 con 0x7f601007f300 2021-02-10T11:14:28.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:27.999 7f6017445700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60101b4010 con 0x7f60100e5070 2021-02-10T11:14:28.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:27.999 7f6017445700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60100d1790 con 0x7f60100e5470 2021-02-10T11:14:28.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:27.999 7f6015a62700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f601007f300 0x7f601007f700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:42062/0 (socket says 172.21.2.112:42062) 2021-02-10T11:14:28.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:27.999 7f6015a62700 1 -- 172.21.2.112:0/4041999868 learned_addr learned my addr 172.21.2.112:0/4041999868 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:14:28.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:27.999 7f6014a60700 1 --2- 172.21.2.112:0/4041999868 >> v2:172.21.2.120:3300/0 conn(0x7f60100e5470 0x7f601007e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:28.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:27.999 7f6003fff700 1 -- 172.21.2.112:0/4041999868 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1825052735 0 0) 0x7f6008001500 con 0x7f601007f300 2021-02-10T11:14:28.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:27.999 7f6003fff700 1 -- 172.21.2.112:0/4041999868 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ffc002fe0 con 0x7f601007f300 2021-02-10T11:14:28.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:27.999 7f6003fff700 1 -- 172.21.2.112:0/4041999868 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2225636007 0 0) 0x7f6008001270 con 0x7f601007f300 2021-02-10T11:14:28.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:27.999 7f6003fff700 1 -- 172.21.2.112:0/4041999868 >> v2:172.21.2.120:3300/0 conn(0x7f60100e5470 msgr2=0x7f601007e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:14:28.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:27.999 7f6003fff700 1 --2- 172.21.2.112:0/4041999868 >> v2:172.21.2.120:3300/0 conn(0x7f60100e5470 0x7f601007e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:28.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:27.999 7f6003fff700 1 -- 172.21.2.112:0/4041999868 >> v1:172.21.2.120:6789/0 conn(0x7f60100e5070 legacy=0x7f60101b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:28.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:27.999 7f6003fff700 1 -- 172.21.2.112:0/4041999868 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60100d19b0 con 0x7f601007f300 2021-02-10T11:14:28.017 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:27.999 7f6003fff700 1 -- 172.21.2.112:0/4041999868 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f6008001b50 con 0x7f601007f300 2021-02-10T11:14:28.017 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.000 7f6003fff700 1 -- 172.21.2.112:0/4041999868 >> v1:172.21.2.112:6789/0 conn(0x7f601007f300 legacy=0x7f601007f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:28.017 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.000 7f6003fff700 1 --2- 172.21.2.112:0/4041999868 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5ffc003a70 0x7f5ffc005f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:28.017 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.000 7f6003fff700 1 --2- 172.21.2.112:0/4041999868 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5ffc0067a0 0x7f5ffc008ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:28.017 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.000 7f6003fff700 1 --2- 172.21.2.112:0/4041999868 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5ffc0095a0 0x7f5ffc00bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:28.017 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.000 7f6003fff700 1 -- 172.21.2.112:0/4041999868 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ffc00c310 con 0x7f5ffc0067a0 2021-02-10T11:14:28.017 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.000 7f6003fff700 1 -- 172.21.2.112:0/4041999868 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ffc00c530 con 0x7f5ffc003a70 2021-02-10T11:14:28.017 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.000 7f6003fff700 1 -- 172.21.2.112:0/4041999868 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ffc050760 con 0x7f5ffc0095a0 2021-02-10T11:14:28.017 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.000 7f6014a60700 1 --2- 172.21.2.112:0/4041999868 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5ffc0067a0 0x7f5ffc008ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:28.017 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.000 7f6015261700 1 --2- 172.21.2.112:0/4041999868 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5ffc003a70 0x7f5ffc005f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:28.017 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.000 7f6015a62700 1 --2- 172.21.2.112:0/4041999868 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5ffc0095a0 0x7f5ffc00bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:28.017 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.000 7f6014a60700 1 -- 172.21.2.112:0/4041999868 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5ffc0095a0 msgr2=0x7f5ffc00bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:14:28.018 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.000 7f6014a60700 1 --2- 172.21.2.112:0/4041999868 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5ffc0095a0 0x7f5ffc00bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:28.018 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.000 7f6014a60700 1 -- 172.21.2.112:0/4041999868 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5ffc003a70 msgr2=0x7f5ffc005f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:14:28.018 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.000 7f6014a60700 1 --2- 172.21.2.112:0/4041999868 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5ffc003a70 0x7f5ffc005f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:28.018 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.000 7f6014a60700 1 -- 172.21.2.112:0/4041999868 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ffc050990 con 0x7f5ffc0067a0 2021-02-10T11:14:28.018 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.000 7f6015a62700 1 -- 172.21.2.112:0/4041999868 reap_dead start 2021-02-10T11:14:28.018 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.000 7f6014a60700 1 --2- 172.21.2.112:0/4041999868 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5ffc0067a0 0x7f5ffc008ca0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f600c00c8f0 tx=0x7f600c00cc90).ready entity=mon.0 client_cookie=b137517b08b24186 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:28.018 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.000 7f6003fff700 1 -- 172.21.2.112:0/4041999868 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f600c009310 con 0x7f5ffc0067a0 2021-02-10T11:14:28.018 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.000 7f6003fff700 1 -- 172.21.2.112:0/4041999868 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f600c009690 con 0x7f5ffc0067a0 2021-02-10T11:14:28.018 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.000 7f6003fff700 1 -- 172.21.2.112:0/4041999868 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f600c009ae0 con 0x7f5ffc0067a0 2021-02-10T11:14:28.018 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.001 7f6017445700 1 -- 172.21.2.112:0/4041999868 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5ffc0067a0 msgr2=0x7f5ffc008ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:28.018 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.001 7f6017445700 1 --2- 172.21.2.112:0/4041999868 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5ffc0067a0 0x7f5ffc008ca0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f600c00c8f0 tx=0x7f600c00cc90).stop 2021-02-10T11:14:28.018 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.001 7f6017445700 1 -- 172.21.2.112:0/4041999868 shutdown_connections 2021-02-10T11:14:28.018 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.001 7f6017445700 1 --2- 172.21.2.112:0/4041999868 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5ffc0067a0 0x7f5ffc008ca0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:28.019 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.001 7f6017445700 1 -- 172.21.2.112:0/4041999868 >> 172.21.2.112:0/4041999868 conn(0x7f60100ce690 msgr2=0x7f60100e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:28.019 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.001 7f6017445700 1 -- 172.21.2.112:0/4041999868 shutdown_connections 2021-02-10T11:14:28.019 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.001 7f6017445700 1 -- 172.21.2.112:0/4041999868 wait complete. 2021-02-10T11:14:28.019 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.016 7f6017445700 1 Processor -- start 2021-02-10T11:14:28.019 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.016 7f6017445700 1 -- start start 2021-02-10T11:14:28.019 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.017 7f6017445700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f601019e5a0 0x7f60101aeef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:28.019 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.017 7f6017445700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6010189ec0 0x7f6010093f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:28.019 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.017 7f6017445700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f60100b5130 0x7f60100b5530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:28.019 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.017 7f6017445700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6010094570 con 0x7f601019e5a0 2021-02-10T11:14:28.019 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.017 7f6017445700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f601019e9a0 con 0x7f60100b5130 2021-02-10T11:14:28.019 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.017 7f6017445700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60101af740 con 0x7f6010189ec0 2021-02-10T11:14:28.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.017 7f6015261700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f601019e5a0 0x7f60101aeef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:28.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.017 7f6015261700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f601019e5a0 0x7f60101aeef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:42608/0 (socket says 172.21.2.112:42608) 2021-02-10T11:14:28.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.017 7f6015261700 1 -- 172.21.2.112:0/3328600708 learned_addr learned my addr 172.21.2.112:0/3328600708 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:14:28.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.017 7f6014a60700 1 --2- 172.21.2.112:0/3328600708 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6010189ec0 0x7f6010093f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:28.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.017 7f6015a62700 1 --2- 172.21.2.112:0/3328600708 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f60100b5130 0x7f60100b5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:28.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.018 7f6014a60700 1 -- 172.21.2.112:0/3328600708 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f60100b5130 msgr2=0x7f60100b5530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:28.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.018 7f6014a60700 1 --2- 172.21.2.112:0/3328600708 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f60100b5130 0x7f60100b5530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:28.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.018 7f6014a60700 1 -- 172.21.2.112:0/3328600708 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f601019e5a0 msgr2=0x7f60101aeef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:28.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.018 7f6014a60700 1 --2- 172.21.2.112:0/3328600708 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f601019e5a0 0x7f60101aeef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:28.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.018 7f6014a60700 1 -- 172.21.2.112:0/3328600708 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f601018a880 con 0x7f6010189ec0 2021-02-10T11:14:28.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.018 7f6015a62700 1 --2- 172.21.2.112:0/3328600708 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f60100b5130 0x7f60100b5530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:14:28.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.018 7f6015261700 1 --2- 172.21.2.112:0/3328600708 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f601019e5a0 0x7f60101aeef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:14:28.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.018 7f6014a60700 1 --2- 172.21.2.112:0/3328600708 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6010189ec0 0x7f6010093f50 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7f600c009e50 tx=0x7f600c00b300).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:28.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.018 7f6001ffb700 1 -- 172.21.2.112:0/3328600708 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f600c00c770 con 0x7f6010189ec0 2021-02-10T11:14:28.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.018 7f6001ffb700 1 -- 172.21.2.112:0/3328600708 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f600c00ca70 con 0x7f6010189ec0 2021-02-10T11:14:28.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.018 7f6001ffb700 1 -- 172.21.2.112:0/3328600708 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f600c00b6f0 con 0x7f6010189ec0 2021-02-10T11:14:28.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.018 7f6017445700 1 -- 172.21.2.112:0/3328600708 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f601018a880 con 0x7f6010189ec0 2021-02-10T11:14:28.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.018 7f6017445700 1 -- 172.21.2.112:0/3328600708 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f601018a880 con 0x7f6010189ec0 2021-02-10T11:14:28.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.019 7f6001ffb700 1 -- 172.21.2.112:0/3328600708 <== mon.2 v2:172.21.2.120:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f600c00c6e0 con 0x7f6010189ec0 2021-02-10T11:14:28.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.019 7f6001ffb700 1 --2- 172.21.2.112:0/3328600708 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5ff00186d0 0x7f5ff001abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:28.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.019 7f6001ffb700 1 -- 172.21.2.112:0/3328600708 <== mon.2 v2:172.21.2.120:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f600c032730 con 0x7f6010189ec0 2021-02-10T11:14:28.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.019 7f6015261700 1 --2- 172.21.2.112:0/3328600708 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5ff00186d0 0x7f5ff001abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:28.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.019 7f6015261700 1 --2- 172.21.2.112:0/3328600708 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5ff00186d0 0x7f5ff001abd0 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:28.047 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.048 7f6017445700 1 -- 172.21.2.112:0/3328600708 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6010084b20 con 0x7f6010189ec0 2021-02-10T11:14:28.050 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.051 7f6001ffb700 1 -- 172.21.2.112:0/3328600708 <== mon.2 v2:172.21.2.120:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f600c000c90 con 0x7f6010189ec0 2021-02-10T11:14:28.174 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.176 7f6017445700 1 -- 172.21.2.112:0/3328600708 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f601007ed80 con 0x7f6010189ec0 2021-02-10T11:14:28.175 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.177 7f6001ffb700 1 -- 172.21.2.112:0/3328600708 <== mon.2 v2:172.21.2.120:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+13993 (secure 0 0 0) 0x7f600c00c6e0 con 0x7f6010189ec0 2021-02-10T11:14:28.178 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:14:28.178 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":20,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:14:04.629689","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:14:01.400797","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2021-02-10 11:14:02.622129","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:14:28.179 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.181 7f6017445700 1 -- 172.21.2.112:0/3328600708 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5ff00186d0 msgr2=0x7f5ff001abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:28.179 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.181 7f6017445700 1 --2- 172.21.2.112:0/3328600708 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5ff00186d0 0x7f5ff001abd0 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:28.179 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.181 7f6017445700 1 -- 172.21.2.112:0/3328600708 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6010189ec0 msgr2=0x7f6010093f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:28.179 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.181 7f6017445700 1 --2- 172.21.2.112:0/3328600708 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6010189ec0 0x7f6010093f50 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7f600c009e50 tx=0x7f600c00b300).stop 2021-02-10T11:14:28.180 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.181 7f6017445700 1 -- 172.21.2.112:0/3328600708 shutdown_connections 2021-02-10T11:14:28.180 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.181 7f6017445700 1 --2- 172.21.2.112:0/3328600708 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5ff00186d0 0x7f5ff001abd0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:28.180 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.181 7f6017445700 1 --2- 172.21.2.112:0/3328600708 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f60100b5130 0x7f60100b5530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:28.181 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.181 7f6017445700 1 --2- 172.21.2.112:0/3328600708 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6010189ec0 0x7f6010093f50 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:28.181 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.181 7f6017445700 1 --2- 172.21.2.112:0/3328600708 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f601019e5a0 0x7f60101aeef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:28.181 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.181 7f6017445700 1 -- 172.21.2.112:0/3328600708 >> 172.21.2.112:0/3328600708 conn(0x7f6010192010 msgr2=0x7f60100a0dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:28.181 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.181 7f6017445700 1 -- 172.21.2.112:0/3328600708 shutdown_connections 2021-02-10T11:14:28.181 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.181 7f6017445700 1 -- 172.21.2.112:0/3328600708 wait complete. 2021-02-10T11:14:28.188 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:13:57.508758', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:14:01.400797', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:14:02.622129', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:14:28.189 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:14:28.621 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.605 7fecd2898700 1 Processor -- start 2021-02-10T11:14:28.621 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.605 7fecd2898700 1 -- start start 2021-02-10T11:14:28.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.606 7fecd2898700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7feccc0f05f0 0x7feccc0f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:28.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.606 7fecd2898700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feccc1b0c70 con 0x7feccc1b0870 2021-02-10T11:14:28.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.606 7fecd2898700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feccc0f0b50 con 0x7feccc1b0470 2021-02-10T11:14:28.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.606 7fecd2898700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feccc0e5ea0 con 0x7feccc0f05f0 2021-02-10T11:14:28.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.606 7feccb7fe700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7feccc1b0870 0x7feccc0efe00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:42078/0 (socket says 172.21.2.112:42078) 2021-02-10T11:14:28.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.606 7feccb7fe700 1 -- 172.21.2.112:0/4053379978 learned_addr learned my addr 172.21.2.112:0/4053379978 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:14:28.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.606 7fecd0eb5700 1 --2- 172.21.2.112:0/4053379978 >> v2:172.21.2.120:3300/0 conn(0x7feccc0f05f0 0x7feccc0f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:28.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.606 7feccaffd700 1 -- 172.21.2.112:0/4053379978 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2512533534 0 0) 0x7fecc00010b0 con 0x7feccc1b0870 2021-02-10T11:14:28.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.606 7feccaffd700 1 -- 172.21.2.112:0/4053379978 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fecb4002fe0 con 0x7feccc1b0870 2021-02-10T11:14:28.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.607 7fecd0eb5700 1 -- 172.21.2.112:0/4053379978 >> v1:172.21.2.120:6789/0 conn(0x7feccc1b0470 legacy=0x7feccc0ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:28.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.607 7feccaffd700 1 -- 172.21.2.112:0/4053379978 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3223356140 0 0) 0x7fecc00010b0 con 0x7feccc1b0870 2021-02-10T11:14:28.624 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.607 7fecd0eb5700 1 -- 172.21.2.112:0/4053379978 >> v1:172.21.2.112:6789/0 conn(0x7feccc1b0870 legacy=0x7feccc0efe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:28.624 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.607 7fecd0eb5700 1 -- 172.21.2.112:0/4053379978 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feccc0e52e0 con 0x7feccc0f05f0 2021-02-10T11:14:28.624 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.607 7fecd0eb5700 1 --2- 172.21.2.112:0/4053379978 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7feccc0f05f0 0x7feccc0f20e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7fecc4006ad0 tx=0x7fecc4009a50).ready entity=mon.1 client_cookie=a0aeed7bf6198a35 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:28.624 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.607 7feccaffd700 1 -- 172.21.2.112:0/4053379978 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fecc40066a0 con 0x7feccc0f05f0 2021-02-10T11:14:28.624 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.607 7feccaffd700 1 -- 172.21.2.112:0/4053379978 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecc4006410 con 0x7feccc0f05f0 2021-02-10T11:14:28.624 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.607 7feccaffd700 1 -- 172.21.2.112:0/4053379978 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fecc4000c10 con 0x7feccc0f05f0 2021-02-10T11:14:28.624 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.607 7fecd2898700 1 -- 172.21.2.112:0/4053379978 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7feccc0f05f0 msgr2=0x7feccc0f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:28.624 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.607 7fecd2898700 1 --2- 172.21.2.112:0/4053379978 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7feccc0f05f0 0x7feccc0f20e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7fecc4006ad0 tx=0x7fecc4009a50).stop 2021-02-10T11:14:28.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.608 7fecd2898700 1 -- 172.21.2.112:0/4053379978 shutdown_connections 2021-02-10T11:14:28.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.608 7fecd2898700 1 --2- 172.21.2.112:0/4053379978 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7feccc0f05f0 0x7feccc0f20e0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:28.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.608 7fecd2898700 1 -- 172.21.2.112:0/4053379978 >> 172.21.2.112:0/4053379978 conn(0x7feccc0ce3e0 msgr2=0x7feccc0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:28.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.608 7fecd2898700 1 -- 172.21.2.112:0/4053379978 shutdown_connections 2021-02-10T11:14:28.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.608 7fecd2898700 1 -- 172.21.2.112:0/4053379978 wait complete. 2021-02-10T11:14:28.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.622 7fecd2898700 1 Processor -- start 2021-02-10T11:14:28.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.623 7fecd2898700 1 -- start start 2021-02-10T11:14:28.629 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.623 7fecd2898700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7feccc094fb0 0x7feccc09e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:28.629 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.623 7fecd2898700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7feccc0c2be0 0x7feccc0c2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:28.630 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.623 7fecd2898700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7feccc0c3600 0x7feccc0a9c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:28.630 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.623 7fecd2898700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feccc09ebf0 con 0x7feccc0c3600 2021-02-10T11:14:28.630 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.623 7fecd2898700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feccc0aa2a0 con 0x7feccc0c2be0 2021-02-10T11:14:28.630 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.623 7fecd2898700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feccc0aa4c0 con 0x7feccc094fb0 2021-02-10T11:14:28.630 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.623 7fecd0eb5700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7feccc0c3600 0x7feccc0a9c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:28.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.623 7feccb7fe700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7feccc0c2be0 0x7feccc0c2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:28.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.623 7fecd0eb5700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7feccc0c3600 0x7feccc0a9c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:42624/0 (socket says 172.21.2.112:42624) 2021-02-10T11:14:28.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.623 7fecd0eb5700 1 -- 172.21.2.112:0/993334762 learned_addr learned my addr 172.21.2.112:0/993334762 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:14:28.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.623 7feccb7fe700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7feccc0c2be0 0x7feccc0c2fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3300/0 says I am v2:172.21.2.112:44902/0 (socket says 172.21.2.112:44902) 2021-02-10T11:14:28.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.623 7feccbfff700 1 --2- 172.21.2.112:0/993334762 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7feccc094fb0 0x7feccc09e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:28.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.624 7fecd0eb5700 1 -- 172.21.2.112:0/993334762 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7feccc094fb0 msgr2=0x7feccc09e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:28.632 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.624 7fecd0eb5700 1 --2- 172.21.2.112:0/993334762 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7feccc094fb0 0x7feccc09e5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:28.632 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.624 7fecd0eb5700 1 -- 172.21.2.112:0/993334762 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7feccc0c2be0 msgr2=0x7feccc0c2fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:28.632 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.624 7fecd0eb5700 1 --2- 172.21.2.112:0/993334762 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7feccc0c2be0 0x7feccc0c2fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:28.632 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.624 7fecd0eb5700 1 -- 172.21.2.112:0/993334762 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feccc0aa6e0 con 0x7feccc0c3600 2021-02-10T11:14:28.632 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.624 7feccb7fe700 1 --2- 172.21.2.112:0/993334762 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7feccc0c2be0 0x7feccc0c2fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:14:28.632 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.624 7fecd0eb5700 1 --2- 172.21.2.112:0/993334762 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7feccc0c3600 0x7feccc0a9c80 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7fecc400a670 tx=0x7fecc400c990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:28.633 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.624 7fecc8ff9700 1 -- 172.21.2.112:0/993334762 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fecc40020d0 con 0x7feccc0c3600 2021-02-10T11:14:28.633 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.624 7fecc8ff9700 1 -- 172.21.2.112:0/993334762 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecc4000e00 con 0x7feccc0c3600 2021-02-10T11:14:28.633 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.624 7fecd2898700 1 -- 172.21.2.112:0/993334762 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feccc0aa6e0 con 0x7feccc0c3600 2021-02-10T11:14:28.633 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.624 7fecc8ff9700 1 -- 172.21.2.112:0/993334762 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fecc4002650 con 0x7feccc0c3600 2021-02-10T11:14:28.633 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.624 7fecd2898700 1 -- 172.21.2.112:0/993334762 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feccc0aa6e0 con 0x7feccc0c3600 2021-02-10T11:14:28.633 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.625 7fecc8ff9700 1 -- 172.21.2.112:0/993334762 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fecc4002350 con 0x7feccc0c3600 2021-02-10T11:14:28.634 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.625 7fecc8ff9700 1 --2- 172.21.2.112:0/993334762 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fecac0186d0 0x7fecac01abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:28.634 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.625 7fecc8ff9700 1 -- 172.21.2.112:0/993334762 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fecc4032780 con 0x7feccc0c3600 2021-02-10T11:14:28.634 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.626 7feccbfff700 1 --2- 172.21.2.112:0/993334762 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fecac0186d0 0x7fecac01abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:28.634 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.626 7feccbfff700 1 --2- 172.21.2.112:0/993334762 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fecac0186d0 0x7fecac01abd0 crc :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:28.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.655 7fecd2898700 1 -- 172.21.2.112:0/993334762 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feccc0edd10 con 0x7feccc0c3600 2021-02-10T11:14:28.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.657 7fecc8ff9700 1 -- 172.21.2.112:0/993334762 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fecc4006730 con 0x7feccc0c3600 2021-02-10T11:14:28.790 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.787 7fecd2898700 1 -- 172.21.2.112:0/993334762 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7feccc0e52e0 con 0x7feccc0c3600 2021-02-10T11:14:28.790 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.788 7fecc8ff9700 1 -- 172.21.2.112:0/993334762 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v20) v1 ==== 92+0+10 (secure 0 0 0) 0x7fecc400cd80 con 0x7feccc0c3600 2021-02-10T11:14:28.791 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.789 7fecd2898700 1 -- 172.21.2.112:0/993334762 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fecac0186d0 msgr2=0x7fecac01abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:28.791 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.789 7fecd2898700 1 --2- 172.21.2.112:0/993334762 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fecac0186d0 0x7fecac01abd0 crc :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:28.791 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.789 7fecd2898700 1 -- 172.21.2.112:0/993334762 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7feccc0c3600 msgr2=0x7feccc0a9c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:28.791 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.789 7fecd2898700 1 --2- 172.21.2.112:0/993334762 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7feccc0c3600 0x7feccc0a9c80 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7fecc400a670 tx=0x7fecc400c990).stop 2021-02-10T11:14:28.793 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.790 7fecd2898700 1 -- 172.21.2.112:0/993334762 shutdown_connections 2021-02-10T11:14:28.793 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.790 7fecd2898700 1 --2- 172.21.2.112:0/993334762 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fecac0186d0 0x7fecac01abd0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:28.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.790 7fecd2898700 1 --2- 172.21.2.112:0/993334762 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7feccc0c2be0 0x7feccc0c2fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:28.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.790 7fecd2898700 1 --2- 172.21.2.112:0/993334762 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7feccc0c3600 0x7feccc0a9c80 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:28.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.790 7fecd2898700 1 --2- 172.21.2.112:0/993334762 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7feccc094fb0 0x7feccc09e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:28.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.790 7fecd2898700 1 -- 172.21.2.112:0/993334762 >> 172.21.2.112:0/993334762 conn(0x7feccc09f630 msgr2=0x7feccc0a0130 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:28.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.790 7fecd2898700 1 -- 172.21.2.112:0/993334762 shutdown_connections 2021-02-10T11:14:28.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:28.790 7fecd2898700 1 -- 172.21.2.112:0/993334762 wait complete. 2021-02-10T11:14:28.796 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:14:28.800 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:14:29.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.217 7f05512cf700 1 Processor -- start 2021-02-10T11:14:29.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.217 7f05512cf700 1 -- start start 2021-02-10T11:14:29.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.218 7f05512cf700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f054c1b0870 0x7f054c0f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:29.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.218 7f05512cf700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f054c0efe00 0x7f054c0f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:29.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.218 7f05512cf700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f054c1b0c70 con 0x7f054c1b0470 2021-02-10T11:14:29.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.218 7f05512cf700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f054c0e5ea0 con 0x7f054c0efe00 2021-02-10T11:14:29.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.218 7f05512cf700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f054c0f0b50 con 0x7f054c1b0870 2021-02-10T11:14:29.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.218 7f054b87e700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f054c0efe00 0x7f054c0f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:29.233 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.218 7f054b87e700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f054c0efe00 0x7f054c0f0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:42632/0 (socket says 172.21.2.112:42632) 2021-02-10T11:14:29.233 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.218 7f054b87e700 1 -- 172.21.2.112:0/2030079956 learned_addr learned my addr 172.21.2.112:0/2030079956 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:14:29.233 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.218 7f054b87e700 1 -- 172.21.2.112:0/2030079956 >> v2:172.21.2.120:3300/0 conn(0x7f054c1b0870 msgr2=0x7f054c0f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:14:29.233 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.218 7f054b87e700 1 --2- 172.21.2.112:0/2030079956 >> v2:172.21.2.120:3300/0 conn(0x7f054c1b0870 0x7f054c0f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:29.233 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.218 7f054b87e700 1 -- 172.21.2.112:0/2030079956 >> v1:172.21.2.120:6789/0 conn(0x7f054c1b0470 legacy=0x7f054c0ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:14:29.233 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.218 7f054b87e700 1 -- 172.21.2.112:0/2030079956 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f054c0f0820 con 0x7f054c0efe00 2021-02-10T11:14:29.233 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.218 7f054b87e700 1 --2- 172.21.2.112:0/2030079956 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f054c0efe00 0x7f054c0f0200 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f053c006b80 tx=0x7f053c009b90).ready entity=mon.0 client_cookie=cea887b27a2af591 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:29.233 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.218 7f054a07b700 1 -- 172.21.2.112:0/2030079956 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f053c0022e0 con 0x7f054c0efe00 2021-02-10T11:14:29.233 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.219 7f054a07b700 1 -- 172.21.2.112:0/2030079956 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f053c002620 con 0x7f054c0efe00 2021-02-10T11:14:29.233 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.219 7f054a07b700 1 -- 172.21.2.112:0/2030079956 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f053c002b00 con 0x7f054c0efe00 2021-02-10T11:14:29.233 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.219 7f05512cf700 1 -- 172.21.2.112:0/2030079956 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f054c0efe00 msgr2=0x7f054c0f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:29.233 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.219 7f05512cf700 1 --2- 172.21.2.112:0/2030079956 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f054c0efe00 0x7f054c0f0200 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f053c006b80 tx=0x7f053c009b90).stop 2021-02-10T11:14:29.233 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.219 7f05512cf700 1 -- 172.21.2.112:0/2030079956 shutdown_connections 2021-02-10T11:14:29.234 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.219 7f05512cf700 1 --2- 172.21.2.112:0/2030079956 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f054c0efe00 0x7f054c0f0200 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:29.234 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.219 7f05512cf700 1 --2- 172.21.2.112:0/2030079956 >> v2:172.21.2.120:3300/0 conn(0x7f054c1b0870 0x7f054c0f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:29.234 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.219 7f05512cf700 1 -- 172.21.2.112:0/2030079956 >> 172.21.2.112:0/2030079956 conn(0x7f054c0ce3e0 msgr2=0x7f054c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:29.234 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.219 7f05512cf700 1 -- 172.21.2.112:0/2030079956 shutdown_connections 2021-02-10T11:14:29.234 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.219 7f05512cf700 1 -- 172.21.2.112:0/2030079956 wait complete. 2021-02-10T11:14:29.234 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.233 7f05512cf700 1 Processor -- start 2021-02-10T11:14:29.234 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.234 7f05512cf700 1 -- start start 2021-02-10T11:14:29.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.234 7f05512cf700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f054c09a2d0 0x7f054c19fed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:29.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.234 7f05512cf700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f054c09a6d0 0x7f054c0ee520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:29.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.234 7f05512cf700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f054c0eeb40 0x7f054c0bdc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:29.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.234 7f05512cf700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f054c0b1b60 con 0x7f054c09a6d0 2021-02-10T11:14:29.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.234 7f05512cf700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f054c099d50 con 0x7f054c09a2d0 2021-02-10T11:14:29.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.234 7f05512cf700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f054c1a04f0 con 0x7f054c0eeb40 2021-02-10T11:14:29.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.234 7f054a87c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f054c09a6d0 0x7f054c0ee520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:29.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.234 7f054a87c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f054c09a6d0 0x7f054c0ee520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:42638/0 (socket says 172.21.2.112:42638) 2021-02-10T11:14:29.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.234 7f054a87c700 1 -- 172.21.2.112:0/1982987126 learned_addr learned my addr 172.21.2.112:0/1982987126 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:14:29.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.234 7f054a87c700 1 -- 172.21.2.112:0/1982987126 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f054c0eeb40 msgr2=0x7f054c0bdc70 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:14:29.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.234 7f054a87c700 1 --2- 172.21.2.112:0/1982987126 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f054c0eeb40 0x7f054c0bdc70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:29.236 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.234 7f054a87c700 1 -- 172.21.2.112:0/1982987126 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f054c09a2d0 msgr2=0x7f054c19fed0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:14:29.236 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.234 7f054a87c700 1 --2- 172.21.2.112:0/1982987126 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f054c09a2d0 0x7f054c19fed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:29.236 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.234 7f054a87c700 1 -- 172.21.2.112:0/1982987126 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f054c1a06d0 con 0x7f054c09a6d0 2021-02-10T11:14:29.237 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.234 7f054a87c700 1 --2- 172.21.2.112:0/1982987126 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f054c09a6d0 0x7f054c0ee520 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f0540006f00 tx=0x7f054000b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:29.237 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.235 7f0533fff700 1 -- 172.21.2.112:0/1982987126 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0540006b90 con 0x7f054c09a6d0 2021-02-10T11:14:29.237 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.235 7f0533fff700 1 -- 172.21.2.112:0/1982987126 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0540000c90 con 0x7f054c09a6d0 2021-02-10T11:14:29.237 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.235 7f0533fff700 1 -- 172.21.2.112:0/1982987126 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0540002020 con 0x7f054c09a6d0 2021-02-10T11:14:29.238 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.235 7f05512cf700 1 -- 172.21.2.112:0/1982987126 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f054c1a06d0 con 0x7f054c09a6d0 2021-02-10T11:14:29.238 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.235 7f05512cf700 1 -- 172.21.2.112:0/1982987126 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f054c0be580 con 0x7f054c09a6d0 2021-02-10T11:14:29.238 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.235 7f0533fff700 1 -- 172.21.2.112:0/1982987126 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0540005310 con 0x7f054c09a6d0 2021-02-10T11:14:29.238 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.235 7f0533fff700 1 --2- 172.21.2.112:0/1982987126 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f052c0186c0 0x7f052c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:29.238 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.235 7f0533fff700 1 -- 172.21.2.112:0/1982987126 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f0540032580 con 0x7f054c09a6d0 2021-02-10T11:14:29.238 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.236 7f054b07d700 1 --2- 172.21.2.112:0/1982987126 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f052c0186c0 0x7f052c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:29.238 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.236 7f054b07d700 1 --2- 172.21.2.112:0/1982987126 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f052c0186c0 0x7f052c01abc0 crc :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:29.266 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.268 7f05512cf700 1 -- 172.21.2.112:0/1982987126 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f054c1a06d0 con 0x7f054c09a6d0 2021-02-10T11:14:29.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.270 7f0533fff700 1 -- 172.21.2.112:0/1982987126 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0540000960 con 0x7f054c09a6d0 2021-02-10T11:14:29.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.398 7f05512cf700 1 -- 172.21.2.112:0/1982987126 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f054c1a06d0 con 0x7f054c09a6d0 2021-02-10T11:14:29.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.398 7f0533fff700 1 -- 172.21.2.112:0/1982987126 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v20) v1 ==== 104+0+10 (secure 0 0 0) 0x7f0540005310 con 0x7f054c09a6d0 2021-02-10T11:14:29.397 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:14:29.399 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.400 7f05512cf700 1 -- 172.21.2.112:0/1982987126 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f052c0186c0 msgr2=0x7f052c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:29.399 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.400 7f05512cf700 1 --2- 172.21.2.112:0/1982987126 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f052c0186c0 0x7f052c01abc0 crc :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:29.399 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.400 7f05512cf700 1 -- 172.21.2.112:0/1982987126 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f054c09a6d0 msgr2=0x7f054c0ee520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:29.399 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.400 7f05512cf700 1 --2- 172.21.2.112:0/1982987126 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f054c09a6d0 0x7f054c0ee520 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f0540006f00 tx=0x7f054000b940).stop 2021-02-10T11:14:29.399 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.400 7f05512cf700 1 -- 172.21.2.112:0/1982987126 shutdown_connections 2021-02-10T11:14:29.399 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.400 7f05512cf700 1 --2- 172.21.2.112:0/1982987126 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f052c0186c0 0x7f052c01abc0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:29.399 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.400 7f05512cf700 1 --2- 172.21.2.112:0/1982987126 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f054c0eeb40 0x7f054c0bdc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:29.399 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.400 7f05512cf700 1 --2- 172.21.2.112:0/1982987126 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f054c09a6d0 0x7f054c0ee520 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:29.399 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.400 7f05512cf700 1 --2- 172.21.2.112:0/1982987126 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f054c09a2d0 0x7f054c19fed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:29.399 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.400 7f05512cf700 1 -- 172.21.2.112:0/1982987126 >> 172.21.2.112:0/1982987126 conn(0x7f054c09dfd0 msgr2=0x7f054c09abf0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:29.400 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.401 7f05512cf700 1 -- 172.21.2.112:0/1982987126 shutdown_connections 2021-02-10T11:14:29.400 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.401 7f05512cf700 1 -- 172.21.2.112:0/1982987126 wait complete. 2021-02-10T11:14:29.407 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:14:29.857 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.842 7f9688068700 1 Processor -- start 2021-02-10T11:14:29.857 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.842 7f9688068700 1 -- start start 2021-02-10T11:14:29.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.843 7f9688068700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f96801b0870 0x7f96800f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:29.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.843 7f9688068700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f96800efe00 0x7f96800f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:29.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.843 7f9688068700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96801b0c70 con 0x7f96801b0470 2021-02-10T11:14:29.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.843 7f9688068700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96800e5ea0 con 0x7f96800efe00 2021-02-10T11:14:29.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.843 7f9688068700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96800f0b50 con 0x7f96801b0870 2021-02-10T11:14:29.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.843 7f9686685700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f96800efe00 0x7f96800f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:29.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.843 7f9685683700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f96801b0870 0x7f96800f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:29.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.843 7f9686685700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f96800efe00 0x7f96800f0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:42646/0 (socket says 172.21.2.112:42646) 2021-02-10T11:14:29.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.843 7f9686685700 1 -- 172.21.2.112:0/1764707714 learned_addr learned my addr 172.21.2.112:0/1764707714 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:14:29.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.843 7f9685e84700 1 --1- 172.21.2.112:0/1764707714 >> v1:172.21.2.120:6789/0 conn(0x7f96801b0470 0x7f96800ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:35662/0 (socket says 172.21.2.112:35662) 2021-02-10T11:14:29.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.843 7f9686685700 1 -- 172.21.2.112:0/1764707714 >> v2:172.21.2.120:3300/0 conn(0x7f96801b0870 msgr2=0x7f96800f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:14:29.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.843 7f9686685700 1 --2- 172.21.2.112:0/1764707714 >> v2:172.21.2.120:3300/0 conn(0x7f96801b0870 0x7f96800f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:29.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.843 7f9686685700 1 -- 172.21.2.112:0/1764707714 >> v1:172.21.2.120:6789/0 conn(0x7f96801b0470 legacy=0x7f96800ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:29.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.843 7f9686685700 1 -- 172.21.2.112:0/1764707714 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96800f0820 con 0x7f96800efe00 2021-02-10T11:14:29.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.843 7f9686685700 1 --2- 172.21.2.112:0/1764707714 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f96800efe00 0x7f96800f0200 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f9678006b80 tx=0x7f9678009b90).ready entity=mon.0 client_cookie=9155bb9ae1caec6d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:29.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.844 7f9684e82700 1 -- 172.21.2.112:0/1764707714 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f96780022e0 con 0x7f96800efe00 2021-02-10T11:14:29.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.844 7f9684e82700 1 -- 172.21.2.112:0/1764707714 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9678002620 con 0x7f96800efe00 2021-02-10T11:14:29.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.844 7f9684e82700 1 -- 172.21.2.112:0/1764707714 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9678002a80 con 0x7f96800efe00 2021-02-10T11:14:29.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.844 7f9688068700 1 -- 172.21.2.112:0/1764707714 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f96800efe00 msgr2=0x7f96800f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:29.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.844 7f9688068700 1 --2- 172.21.2.112:0/1764707714 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f96800efe00 0x7f96800f0200 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f9678006b80 tx=0x7f9678009b90).stop 2021-02-10T11:14:29.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.844 7f9688068700 1 -- 172.21.2.112:0/1764707714 shutdown_connections 2021-02-10T11:14:29.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.844 7f9688068700 1 --2- 172.21.2.112:0/1764707714 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f96800efe00 0x7f96800f0200 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:29.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.844 7f9688068700 1 --2- 172.21.2.112:0/1764707714 >> v2:172.21.2.120:3300/0 conn(0x7f96801b0870 0x7f96800f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:29.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.844 7f9688068700 1 -- 172.21.2.112:0/1764707714 >> 172.21.2.112:0/1764707714 conn(0x7f96800ce3e0 msgr2=0x7f96800e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:29.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.844 7f9688068700 1 -- 172.21.2.112:0/1764707714 shutdown_connections 2021-02-10T11:14:29.860 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.844 7f9688068700 1 -- 172.21.2.112:0/1764707714 wait complete. 2021-02-10T11:14:29.860 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.859 7f9688068700 1 Processor -- start 2021-02-10T11:14:29.860 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.859 7f9688068700 1 -- start start 2021-02-10T11:14:29.862 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.859 7f9688068700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9680190920 0x7f96800b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:29.862 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.859 7f9688068700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f96800b8200 0x7f96800b8600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:29.862 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.859 7f9688068700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f96800c1d80 0x7f96800c2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:29.862 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.859 7f9688068700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9680190da0 con 0x7f96800b8200 2021-02-10T11:14:29.862 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.859 7f9688068700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96800b8c20 con 0x7f9680190920 2021-02-10T11:14:29.862 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.859 7f9688068700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96800b8e10 con 0x7f96800c1d80 2021-02-10T11:14:29.862 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.860 7f9685683700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f96800b8200 0x7f96800b8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:29.862 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.860 7f9685683700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f96800b8200 0x7f96800b8600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:42648/0 (socket says 172.21.2.112:42648) 2021-02-10T11:14:29.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.860 7f9685683700 1 -- 172.21.2.112:0/2659778843 learned_addr learned my addr 172.21.2.112:0/2659778843 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:14:29.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.860 7f9685e84700 1 --2- 172.21.2.112:0/2659778843 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9680190920 0x7f96800b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:29.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.860 7f9686685700 1 --2- 172.21.2.112:0/2659778843 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f96800c1d80 0x7f96800c2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:29.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.860 7f9685683700 1 -- 172.21.2.112:0/2659778843 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f96800c1d80 msgr2=0x7f96800c2180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:29.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.860 7f9685683700 1 --2- 172.21.2.112:0/2659778843 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f96800c1d80 0x7f96800c2180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:29.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.860 7f9685683700 1 -- 172.21.2.112:0/2659778843 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9680190920 msgr2=0x7f96800b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:29.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.860 7f9685683700 1 --2- 172.21.2.112:0/2659778843 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9680190920 0x7f96800b7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:29.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.860 7f9685683700 1 -- 172.21.2.112:0/2659778843 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96800c27d0 con 0x7f96800b8200 2021-02-10T11:14:29.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.860 7f9685e84700 1 --2- 172.21.2.112:0/2659778843 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9680190920 0x7f96800b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:14:29.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.860 7f9685683700 1 --2- 172.21.2.112:0/2659778843 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f96800b8200 0x7f96800b8600 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f967000b560 tx=0x7f967000b900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:29.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.860 7f96767fc700 1 -- 172.21.2.112:0/2659778843 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f967000bcf0 con 0x7f96800b8200 2021-02-10T11:14:29.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.860 7f96767fc700 1 -- 172.21.2.112:0/2659778843 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9670000c90 con 0x7f96800b8200 2021-02-10T11:14:29.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.860 7f96767fc700 1 -- 172.21.2.112:0/2659778843 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f96700022b0 con 0x7f96800b8200 2021-02-10T11:14:29.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.860 7f9688068700 1 -- 172.21.2.112:0/2659778843 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96800c27d0 con 0x7f96800b8200 2021-02-10T11:14:29.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.860 7f9688068700 1 -- 172.21.2.112:0/2659778843 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96800c27d0 con 0x7f96800b8200 2021-02-10T11:14:29.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.861 7f96767fc700 1 -- 172.21.2.112:0/2659778843 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9670006b00 con 0x7f96800b8200 2021-02-10T11:14:29.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.861 7f96767fc700 1 --2- 172.21.2.112:0/2659778843 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9668018730 0x7f966801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:29.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.861 7f96767fc700 1 -- 172.21.2.112:0/2659778843 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f9670032630 con 0x7f96800b8200 2021-02-10T11:14:29.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.861 7f9685e84700 1 --2- 172.21.2.112:0/2659778843 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9668018730 0x7f966801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:29.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.862 7f9685e84700 1 --2- 172.21.2.112:0/2659778843 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9668018730 0x7f966801abc0 crc :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:29.891 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.892 7f9688068700 1 -- 172.21.2.112:0/2659778843 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9680105ae0 con 0x7f96800b8200 2021-02-10T11:14:29.893 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:29.895 7f96767fc700 1 -- 172.21.2.112:0/2659778843 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9670000c90 con 0x7f96800b8200 2021-02-10T11:14:30.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:30.025 7f9688068700 1 -- 172.21.2.112:0/2659778843 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f9680105ae0 con 0x7f96800b8200 2021-02-10T11:14:30.024 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:30.025 7f96767fc700 1 -- 172.21.2.112:0/2659778843 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v20) v1 ==== 100+0+10 (secure 0 0 0) 0x7f9670006b00 con 0x7f96800b8200 2021-02-10T11:14:30.025 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:14:30.027 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:30.027 7f9688068700 1 -- 172.21.2.112:0/2659778843 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9668018730 msgr2=0x7f966801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:30.028 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:30.027 7f9688068700 1 --2- 172.21.2.112:0/2659778843 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9668018730 0x7f966801abc0 crc :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:30.028 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:30.027 7f9688068700 1 -- 172.21.2.112:0/2659778843 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f96800b8200 msgr2=0x7f96800b8600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:30.028 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:30.027 7f9688068700 1 --2- 172.21.2.112:0/2659778843 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f96800b8200 0x7f96800b8600 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f967000b560 tx=0x7f967000b900).stop 2021-02-10T11:14:30.028 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:30.027 7f9688068700 1 -- 172.21.2.112:0/2659778843 shutdown_connections 2021-02-10T11:14:30.029 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:30.027 7f9688068700 1 --2- 172.21.2.112:0/2659778843 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9668018730 0x7f966801abc0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:30.029 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:30.027 7f9688068700 1 --2- 172.21.2.112:0/2659778843 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f96800c1d80 0x7f96800c2180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:30.029 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:30.027 7f9688068700 1 --2- 172.21.2.112:0/2659778843 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f96800b8200 0x7f96800b8600 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:30.029 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:30.027 7f9688068700 1 --2- 172.21.2.112:0/2659778843 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9680190920 0x7f96800b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:30.029 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:30.027 7f9688068700 1 -- 172.21.2.112:0/2659778843 >> 172.21.2.112:0/2659778843 conn(0x7f9680094e10 msgr2=0x7f96800a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:30.030 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:30.027 7f9688068700 1 -- 172.21.2.112:0/2659778843 shutdown_connections 2021-02-10T11:14:30.030 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:30.027 7f9688068700 1 -- 172.21.2.112:0/2659778843 wait complete. 2021-02-10T11:14:30.036 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:14:30.037 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:14:30.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:30.459 7f08efc7e700 1 Processor -- start 2021-02-10T11:14:30.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:30.459 7f08efc7e700 1 -- start start 2021-02-10T11:14:30.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:30.460 7f08efc7e700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f08e807e6a0 0x7f08e80d1790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:30.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:30.460 7f08efc7e700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08e807ed80 con 0x7f08e80e5470 2021-02-10T11:14:30.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:30.460 7f08efc7e700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08e807eaa0 con 0x7f08e80e5070 2021-02-10T11:14:30.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:30.460 7f08efc7e700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08e81b4080 con 0x7f08e807e6a0 2021-02-10T11:14:30.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:30.460 7f08ed299700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f08e80e5470 0x7f08e807f300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:42120/0 (socket says 172.21.2.112:42120) 2021-02-10T11:14:30.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:30.460 7f08ed299700 1 -- 172.21.2.112:0/2480651979 learned_addr learned my addr 172.21.2.112:0/2480651979 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:14:30.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:30.460 7f08eca98700 1 -- 172.21.2.112:0/2480651979 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2100537519 0 0) 0x7f08dc000ce0 con 0x7f08e80e5470 2021-02-10T11:14:30.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:30.460 7f08eca98700 1 -- 172.21.2.112:0/2480651979 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08d4002fe0 con 0x7f08e80e5470 2021-02-10T11:14:30.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:30.461 7f08ee29b700 1 --2- 172.21.2.112:0/2480651979 >> v2:172.21.2.120:3300/0 conn(0x7f08e807e6a0 0x7f08e80d1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:30.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:30.461 7f08eca98700 1 -- 172.21.2.112:0/2480651979 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1364084255 0 0) 0x7f08dc001180 con 0x7f08e80e5470 2021-02-10T11:14:30.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:30.461 7f08eca98700 1 -- 172.21.2.112:0/2480651979 >> v2:172.21.2.120:3300/0 conn(0x7f08e807e6a0 msgr2=0x7f08e80d1790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:14:30.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:30.461 7f08eca98700 1 --2- 172.21.2.112:0/2480651979 >> v2:172.21.2.120:3300/0 conn(0x7f08e807e6a0 0x7f08e80d1790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:30.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:30.461 7f08eca98700 1 -- 172.21.2.112:0/2480651979 >> v1:172.21.2.120:6789/0 conn(0x7f08e80e5070 legacy=0x7f08e81b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:30.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:30.461 7f08eca98700 1 -- 172.21.2.112:0/2480651979 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08e80d1db0 con 0x7f08e80e5470 2021-02-10T11:14:30.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:30.461 7f08eca98700 1 -- 172.21.2.112:0/2480651979 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f08dc001680 con 0x7f08e80e5470 2021-02-10T11:14:30.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:30.461 7f08eca98700 1 -- 172.21.2.112:0/2480651979 >> v1:172.21.2.112:6789/0 conn(0x7f08e80e5470 legacy=0x7f08e807f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:30.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:30.461 7f08eca98700 1 --2- 172.21.2.112:0/2480651979 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f08d4003b10 0x7f08d4006010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:30.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:30.461 7f08eca98700 1 --2- 172.21.2.112:0/2480651979 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f08d4006820 0x7f08d4008d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:30.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:30.461 7f08eca98700 1 --2- 172.21.2.112:0/2480651979 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f08d4009640 0x7f08d400bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:30.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:30.461 7f08eca98700 1 -- 172.21.2.112:0/2480651979 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08d400c390 con 0x7f08d4009640 2021-02-10T11:14:30.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:30.461 7f08eca98700 1 -- 172.21.2.112:0/2480651979 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08d400c5b0 con 0x7f08d4003b10 2021-02-10T11:14:30.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:30.461 7f08eca98700 1 -- 172.21.2.112:0/2480651979 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08d400c7d0 con 0x7f08d4006820 2021-02-10T11:14:30.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:30.461 7f08ee29b700 1 --2- 172.21.2.112:0/2480651979 >> v2:172.21.2.120:3300/0 conn(0x7f08e807e6a0 0x7f08e80d1790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:14:30.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:30.461 7f08eda9a700 1 --2- 172.21.2.112:0/2480651979 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f08d4003b10 0x7f08d4006010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:30.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:30.461 7f08ee29b700 1 --2- 172.21.2.112:0/2480651979 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f08d4009640 0x7f08d400bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:30.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:30.461 7f08ed299700 1 --2- 172.21.2.112:0/2480651979 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f08d4006820 0x7f08d4008d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:30.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:30.462 7f08eda9a700 1 -- 172.21.2.112:0/2480651979 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f08d4006820 msgr2=0x7f08d4008d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:14:30.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:30.462 7f08eda9a700 1 --2- 172.21.2.112:0/2480651979 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f08d4006820 0x7f08d4008d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:30.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:30.462 7f08eda9a700 1 -- 172.21.2.112:0/2480651979 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f08d4009640 msgr2=0x7f08d400bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:14:30.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:30.462 7f08eda9a700 1 --2- 172.21.2.112:0/2480651979 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f08d4009640 0x7f08d400bb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:30.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:30.462 7f08eda9a700 1 -- 172.21.2.112:0/2480651979 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08d400ca70 con 0x7f08d4003b10 2021-02-10T11:14:30.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:30.462 7f08ee29b700 1 -- 172.21.2.112:0/2480651979 reap_dead start 2021-02-10T11:14:30.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:30.462 7f08eda9a700 1 --2- 172.21.2.112:0/2480651979 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f08d4003b10 0x7f08d4006010 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f08e404a660 tx=0x7f08e404da10).ready entity=mon.1 client_cookie=bfbada2c89aae9de server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:30.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:30.462 7f08eca98700 1 -- 172.21.2.112:0/2480651979 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f08e4000960 con 0x7f08d4003b10 2021-02-10T11:14:30.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:30.462 7f08eca98700 1 -- 172.21.2.112:0/2480651979 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08e40500c0 con 0x7f08d4003b10 2021-02-10T11:14:30.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:30.462 7f08eca98700 1 -- 172.21.2.112:0/2480651979 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f08e4050510 con 0x7f08d4003b10 2021-02-10T11:14:30.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:30.462 7f08efc7e700 1 -- 172.21.2.112:0/2480651979 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f08d4003b10 msgr2=0x7f08d4006010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:30.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:30.462 7f08efc7e700 1 --2- 172.21.2.112:0/2480651979 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f08d4003b10 0x7f08d4006010 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f08e404a660 tx=0x7f08e404da10).stop 2021-02-10T11:14:30.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:30.463 7f08efc7e700 1 -- 172.21.2.112:0/2480651979 shutdown_connections 2021-02-10T11:14:30.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:30.463 7f08efc7e700 1 --2- 172.21.2.112:0/2480651979 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f08d4003b10 0x7f08d4006010 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:30.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:30.463 7f08efc7e700 1 -- 172.21.2.112:0/2480651979 >> 172.21.2.112:0/2480651979 conn(0x7f08e80ce690 msgr2=0x7f08e80e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:30.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:30.463 7f08efc7e700 1 -- 172.21.2.112:0/2480651979 shutdown_connections 2021-02-10T11:14:30.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:30.463 7f08efc7e700 1 -- 172.21.2.112:0/2480651979 wait complete. 2021-02-10T11:14:30.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:30.478 7f08efc7e700 1 Processor -- start 2021-02-10T11:14:30.482 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:30.478 7f08efc7e700 1 -- start start 2021-02-10T11:14:30.482 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:30.478 7f08efc7e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f08e8194000 0x7f08e819f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:30.482 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:30.478 7f08efc7e700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f08e819fde0 0x7f08e809bed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:30.482 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:30.478 7f08efc7e700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f08e81a01e0 0x7f08e80895c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:30.482 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:30.478 7f08efc7e700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08e8194400 con 0x7f08e8194000 2021-02-10T11:14:30.482 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:30.478 7f08efc7e700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08e809c4f0 con 0x7f08e819fde0 2021-02-10T11:14:30.482 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:30.478 7f08efc7e700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08e8089be0 con 0x7f08e81a01e0 2021-02-10T11:14:30.483 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:30.478 7f08eda9a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f08e8194000 0x7f08e819f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:30.483 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:30.478 7f08eda9a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f08e8194000 0x7f08e819f680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:42668/0 (socket says 172.21.2.112:42668) 2021-02-10T11:14:30.483 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:30.478 7f08eda9a700 1 -- 172.21.2.112:0/622745736 learned_addr learned my addr 172.21.2.112:0/622745736 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:14:30.483 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:30.479 7f08eda9a700 1 -- 172.21.2.112:0/622745736 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f08e81a01e0 msgr2=0x7f08e80895c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:14:30.483 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:30.479 7f08eda9a700 1 --2- 172.21.2.112:0/622745736 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f08e81a01e0 0x7f08e80895c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:30.483 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:30.479 7f08eda9a700 1 -- 172.21.2.112:0/622745736 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f08e819fde0 msgr2=0x7f08e809bed0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:14:30.483 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:30.479 7f08eda9a700 1 --2- 172.21.2.112:0/622745736 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f08e819fde0 0x7f08e809bed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:30.483 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:30.479 7f08eda9a700 1 -- 172.21.2.112:0/622745736 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08e8089e00 con 0x7f08e8194000 2021-02-10T11:14:30.483 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:30.479 7f08eda9a700 1 --2- 172.21.2.112:0/622745736 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f08e8194000 0x7f08e819f680 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f08e404dfc0 tx=0x7f08e404f9d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:30.483 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:30.479 7f08da7fc700 1 -- 172.21.2.112:0/622745736 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f08e404fdc0 con 0x7f08e8194000 2021-02-10T11:14:30.483 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:30.479 7f08da7fc700 1 -- 172.21.2.112:0/622745736 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08e404a0c0 con 0x7f08e8194000 2021-02-10T11:14:30.483 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:30.479 7f08da7fc700 1 -- 172.21.2.112:0/622745736 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f08e4002210 con 0x7f08e8194000 2021-02-10T11:14:30.483 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:30.479 7f08efc7e700 1 -- 172.21.2.112:0/622745736 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08e8089e00 con 0x7f08e8194000 2021-02-10T11:14:30.484 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:30.479 7f08efc7e700 1 -- 172.21.2.112:0/622745736 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08e8089e00 con 0x7f08e8194000 2021-02-10T11:14:30.484 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:30.480 7f08da7fc700 1 -- 172.21.2.112:0/622745736 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f08e4002020 con 0x7f08e8194000 2021-02-10T11:14:30.484 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:30.480 7f08da7fc700 1 --2- 172.21.2.112:0/622745736 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f08d00186c0 0x7f08d001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:30.484 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:30.480 7f08da7fc700 1 -- 172.21.2.112:0/622745736 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f08e4076570 con 0x7f08e8194000 2021-02-10T11:14:30.484 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:30.480 7f08ed299700 1 --2- 172.21.2.112:0/622745736 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f08d00186c0 0x7f08d001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:30.484 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:30.481 7f08ed299700 1 --2- 172.21.2.112:0/622745736 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f08d00186c0 0x7f08d001abc0 crc :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:30.508 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:30.510 7f08efc7e700 1 -- 172.21.2.112:0/622745736 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08e809c710 con 0x7f08e8194000 2021-02-10T11:14:30.511 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:30.512 7f08da7fc700 1 -- 172.21.2.112:0/622745736 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f08e4000940 con 0x7f08e8194000 2021-02-10T11:14:30.639 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:30.641 7f08efc7e700 1 -- 172.21.2.112:0/622745736 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f08e8089090 con 0x7f08e8194000 2021-02-10T11:14:30.640 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:30.641 7f08da7fc700 1 -- 172.21.2.112:0/622745736 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+2097 (secure 0 0 0) 0x7f08e404fdc0 con 0x7f08e8194000 2021-02-10T11:14:30.640 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:14:30.641 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4279,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.120:6835/2695560847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":2695560847},{"type":"v1","addr":"172.21.2.120:6835","nonce":2695560847}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":4}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:14:03.627691","modified":"2021-02-10 11:14:13.011189","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4272},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4272":{"gid":4272,"name":"a","rank":0,"incarnation":5,"state":"up:active","state_seq":3,"addr":"172.21.2.112:6833/2584920075","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":2584920075},{"type":"v1","addr":"172.21.2.112:6833","nonce":2584920075}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2021-02-10T11:14:30.641 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 6 2021-02-10T11:14:30.642 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:30.643 7f08efc7e700 1 -- 172.21.2.112:0/622745736 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f08d00186c0 msgr2=0x7f08d001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:30.642 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:30.643 7f08efc7e700 1 --2- 172.21.2.112:0/622745736 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f08d00186c0 0x7f08d001abc0 crc :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:30.642 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:30.643 7f08efc7e700 1 -- 172.21.2.112:0/622745736 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f08e8194000 msgr2=0x7f08e819f680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:30.642 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:30.643 7f08efc7e700 1 --2- 172.21.2.112:0/622745736 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f08e8194000 0x7f08e819f680 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f08e404dfc0 tx=0x7f08e404f9d0).stop 2021-02-10T11:14:30.643 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:30.643 7f08efc7e700 1 -- 172.21.2.112:0/622745736 shutdown_connections 2021-02-10T11:14:30.643 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:30.643 7f08efc7e700 1 --2- 172.21.2.112:0/622745736 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f08d00186c0 0x7f08d001abc0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:30.643 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:30.643 7f08efc7e700 1 --2- 172.21.2.112:0/622745736 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f08e819fde0 0x7f08e809bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:30.643 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:30.643 7f08efc7e700 1 --2- 172.21.2.112:0/622745736 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f08e81a01e0 0x7f08e80895c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:30.643 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:30.643 7f08efc7e700 1 --2- 172.21.2.112:0/622745736 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f08e8194000 0x7f08e819f680 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:30.643 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:30.643 7f08efc7e700 1 -- 172.21.2.112:0/622745736 >> 172.21.2.112:0/622745736 conn(0x7f08e8092c20 msgr2=0x7f08e8195070 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:30.643 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:30.644 7f08efc7e700 1 -- 172.21.2.112:0/622745736 shutdown_connections 2021-02-10T11:14:30.643 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:30.644 7f08efc7e700 1 -- 172.21.2.112:0/622745736 wait complete. 2021-02-10T11:14:30.852 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 6, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:14:03.627691', 'modified': '2021-02-10 11:14:13.011189', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4272}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4272': {'gid': 4272, 'name': 'a', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.112:6833/2584920075', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.112:6832', 'nonce': 2584920075}, {'type': 'v1', 'addr': '172.21.2.112:6833', 'nonce': 2584920075}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:14:30.852 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:14:30.852 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok status 2021-02-10T11:14:31.251 INFO:teuthology.orchestra.run.gibba012.stdout:{ 2021-02-10T11:14:31.251 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", 2021-02-10T11:14:31.251 INFO:teuthology.orchestra.run.gibba012.stdout: "whoami": 0, 2021-02-10T11:14:31.251 INFO:teuthology.orchestra.run.gibba012.stdout: "id": 4272, 2021-02-10T11:14:31.251 INFO:teuthology.orchestra.run.gibba012.stdout: "want_state": "up:active", 2021-02-10T11:14:31.251 INFO:teuthology.orchestra.run.gibba012.stdout: "state": "up:active", 2021-02-10T11:14:31.251 INFO:teuthology.orchestra.run.gibba012.stdout: "rank_uptime": 19.241763697, 2021-02-10T11:14:31.252 INFO:teuthology.orchestra.run.gibba012.stdout: "mdsmap_epoch": 6, 2021-02-10T11:14:31.252 INFO:teuthology.orchestra.run.gibba012.stdout: "osdmap_epoch": 20, 2021-02-10T11:14:31.252 INFO:teuthology.orchestra.run.gibba012.stdout: "osdmap_epoch_barrier": 20, 2021-02-10T11:14:31.252 INFO:teuthology.orchestra.run.gibba012.stdout: "uptime": 23.60880684 2021-02-10T11:14:31.252 INFO:teuthology.orchestra.run.gibba012.stdout:} 2021-02-10T11:14:31.255 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", "whoami": 0, "id": 4272, "want_state": "up:active", "state": "up:active", "rank_uptime": 19.241763697, "mdsmap_epoch": 6, "osdmap_epoch": 20, "osdmap_epoch_barrier": 20, "uptime": 23.60880684 } 2021-02-10T11:14:31.255 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:14:31.255 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2021-02-10T11:14:31.255 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@gibba012.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2021-02-10T11:14:31.256 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:14:31.256 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:14:31.277 DEBUG:teuthology.orchestra.run.gibba012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:14:31.300 DEBUG:teuthology.orchestra.run.gibba020:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:14:31.593 DEBUG:teuthology.orchestra.run.gibba012:> sudo modprobe fuse 2021-02-10T11:14:31.984 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:14:31.985 DEBUG:teuthology.orchestra.run.gibba012:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:14:32.329 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:14:32.329 DEBUG:teuthology.orchestra.run.gibba012:> ls /sys/fs/fuse/connections 2021-02-10T11:14:32.663 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2021-02-10T11:14:32.664 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:14:32.666 DEBUG:teuthology.orchestra.run.gibba012:> sudo modprobe fuse 2021-02-10T11:14:33.012 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:14:33.012 DEBUG:teuthology.orchestra.run.gibba012:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:14:33.078 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba012.stderr:2021-02-10 11:14:33.080 7f67c7407f80 -1 init, newargv = 0x5587dc246100 newargc=9 2021-02-10T11:14:33.078 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba012.stderr:ceph-fuse[39881]: starting ceph client 2021-02-10T11:14:33.082 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba012.stderr:ceph-fuse[39881]: starting fuse 2021-02-10T11:14:33.355 INFO:teuthology.orchestra.run.gibba012.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:14:33.355 INFO:teuthology.orchestra.run.gibba012.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:14:33.356 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:14:33.357 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:14:33.357 DEBUG:teuthology.orchestra.run.gibba012:> ls /sys/fs/fuse/connections 2021-02-10T11:14:33.691 INFO:teuthology.orchestra.run.gibba012.stdout:42 2021-02-10T11:14:33.691 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2021-02-10T11:14:33.692 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:14:33.692 DEBUG:teuthology.orchestra.run.gibba012:> sudo python3 -c ' 2021-02-10T11:14:33.692 DEBUG:teuthology.orchestra.run.gibba012:> import glob 2021-02-10T11:14:33.692 DEBUG:teuthology.orchestra.run.gibba012:> import re 2021-02-10T11:14:33.692 DEBUG:teuthology.orchestra.run.gibba012:> import os 2021-02-10T11:14:33.692 DEBUG:teuthology.orchestra.run.gibba012:> import subprocess 2021-02-10T11:14:33.692 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:14:33.692 DEBUG:teuthology.orchestra.run.gibba012:> def find_socket(client_name): 2021-02-10T11:14:33.692 DEBUG:teuthology.orchestra.run.gibba012:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2021-02-10T11:14:33.692 DEBUG:teuthology.orchestra.run.gibba012:> files = glob.glob(asok_path) 2021-02-10T11:14:33.692 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:14:33.693 DEBUG:teuthology.orchestra.run.gibba012:> # Given a non-glob path, it better be there 2021-02-10T11:14:33.693 DEBUG:teuthology.orchestra.run.gibba012:> if "*" not in asok_path: 2021-02-10T11:14:33.693 DEBUG:teuthology.orchestra.run.gibba012:> assert(len(files) == 1) 2021-02-10T11:14:33.693 DEBUG:teuthology.orchestra.run.gibba012:> return files[0] 2021-02-10T11:14:33.693 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:14:33.693 DEBUG:teuthology.orchestra.run.gibba012:> for f in files: 2021-02-10T11:14:33.693 DEBUG:teuthology.orchestra.run.gibba012:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:14:33.693 DEBUG:teuthology.orchestra.run.gibba012:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:14:33.693 DEBUG:teuthology.orchestra.run.gibba012:> return f 2021-02-10T11:14:33.693 DEBUG:teuthology.orchestra.run.gibba012:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:14:33.693 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:14:33.693 DEBUG:teuthology.orchestra.run.gibba012:> print(find_socket("client.0")) 2021-02-10T11:14:33.693 DEBUG:teuthology.orchestra.run.gibba012:> ' 2021-02-10T11:14:34.067 INFO:teuthology.orchestra.run.gibba012.stdout:/var/run/ceph/ceph-client.0.39881.asok 2021-02-10T11:14:34.070 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.39881.asok 2021-02-10T11:14:34.070 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:14:34.070 DEBUG:teuthology.orchestra.run.gibba012:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.39881.asok status 2021-02-10T11:14:34.459 INFO:teuthology.orchestra.run.gibba012.stdout:{ 2021-02-10T11:14:34.459 INFO:teuthology.orchestra.run.gibba012.stdout: "metadata": { 2021-02-10T11:14:34.459 INFO:teuthology.orchestra.run.gibba012.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:14:34.459 INFO:teuthology.orchestra.run.gibba012.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:14:34.459 INFO:teuthology.orchestra.run.gibba012.stdout: "entity_id": "0", 2021-02-10T11:14:34.459 INFO:teuthology.orchestra.run.gibba012.stdout: "hostname": "gibba012", 2021-02-10T11:14:34.460 INFO:teuthology.orchestra.run.gibba012.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2021-02-10T11:14:34.460 INFO:teuthology.orchestra.run.gibba012.stdout: "pid": "39881", 2021-02-10T11:14:34.460 INFO:teuthology.orchestra.run.gibba012.stdout: "root": "/" 2021-02-10T11:14:34.460 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:34.460 INFO:teuthology.orchestra.run.gibba012.stdout: "dentry_count": 0, 2021-02-10T11:14:34.460 INFO:teuthology.orchestra.run.gibba012.stdout: "dentry_pinned_count": 0, 2021-02-10T11:14:34.460 INFO:teuthology.orchestra.run.gibba012.stdout: "id": 4509, 2021-02-10T11:14:34.460 INFO:teuthology.orchestra.run.gibba012.stdout: "inst": { 2021-02-10T11:14:34.460 INFO:teuthology.orchestra.run.gibba012.stdout: "name": { 2021-02-10T11:14:34.460 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "client", 2021-02-10T11:14:34.460 INFO:teuthology.orchestra.run.gibba012.stdout: "num": 4509 2021-02-10T11:14:34.460 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:34.460 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": { 2021-02-10T11:14:34.461 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:14:34.461 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:0", 2021-02-10T11:14:34.461 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 1994160521 2021-02-10T11:14:34.461 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:14:34.461 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:34.461 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": { 2021-02-10T11:14:34.461 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:14:34.461 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:0", 2021-02-10T11:14:34.461 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 1994160521 2021-02-10T11:14:34.461 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:34.461 INFO:teuthology.orchestra.run.gibba012.stdout: "inst_str": "client.4509 172.21.2.112:0/1994160521", 2021-02-10T11:14:34.461 INFO:teuthology.orchestra.run.gibba012.stdout: "addr_str": "172.21.2.112:0/1994160521", 2021-02-10T11:14:34.461 INFO:teuthology.orchestra.run.gibba012.stdout: "inode_count": 1, 2021-02-10T11:14:34.461 INFO:teuthology.orchestra.run.gibba012.stdout: "mds_epoch": 6, 2021-02-10T11:14:34.462 INFO:teuthology.orchestra.run.gibba012.stdout: "osd_epoch": 20, 2021-02-10T11:14:34.462 INFO:teuthology.orchestra.run.gibba012.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:14:34.462 INFO:teuthology.orchestra.run.gibba012.stdout: "blacklisted": false 2021-02-10T11:14:34.462 INFO:teuthology.orchestra.run.gibba012.stdout:} 2021-02-10T11:14:34.463 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:14:34.887 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:34.872 7f39bf726700 1 Processor -- start 2021-02-10T11:14:34.888 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:34.872 7f39bf726700 1 -- start start 2021-02-10T11:14:34.888 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:34.873 7f39bf726700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f39b80e5070 0x7f39b81b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:34.888 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:34.873 7f39bf726700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f39b80e5470 0x7f39b8080310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:34.888 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:34.873 7f39bf726700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39b807ed80 con 0x7f39b81b3e40 2021-02-10T11:14:34.888 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:34.873 7f39bf726700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39b80d1790 con 0x7f39b80e5470 2021-02-10T11:14:34.888 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:34.873 7f39bf726700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39b80d19b0 con 0x7f39b80e5070 2021-02-10T11:14:34.888 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:34.873 7f39bdd43700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f39b80e5470 0x7f39b8080310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:34.888 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:34.873 7f39bdd43700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f39b80e5470 0x7f39b8080310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:42686/0 (socket says 172.21.2.112:42686) 2021-02-10T11:14:34.888 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:34.873 7f39bdd43700 1 -- 172.21.2.112:0/2066351167 learned_addr learned my addr 172.21.2.112:0/2066351167 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:14:34.888 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:34.873 7f39bdd43700 1 -- 172.21.2.112:0/2066351167 >> v2:172.21.2.120:3300/0 conn(0x7f39b80e5070 msgr2=0x7f39b81b3820 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:14:34.888 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:34.873 7f39bdd43700 1 --2- 172.21.2.112:0/2066351167 >> v2:172.21.2.120:3300/0 conn(0x7f39b80e5070 0x7f39b81b3820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:34.888 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:34.873 7f39bdd43700 1 -- 172.21.2.112:0/2066351167 >> v1:172.21.2.112:6789/0 conn(0x7f39b81b3e40 legacy=0x7f39b807d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:34.889 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:34.873 7f39bdd43700 1 -- 172.21.2.112:0/2066351167 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39b80d1bd0 con 0x7f39b80e5470 2021-02-10T11:14:34.889 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:34.874 7f39bdd43700 1 --2- 172.21.2.112:0/2066351167 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f39b80e5470 0x7f39b8080310 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f39b0006f00 tx=0x7f39b0009e90).ready entity=mon.0 client_cookie=67aa4662193e30e8 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:34.889 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:34.874 7f39abfff700 1 -- 172.21.2.112:0/2066351167 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f39b000b300 con 0x7f39b80e5470 2021-02-10T11:14:34.889 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:34.874 7f39abfff700 1 -- 172.21.2.112:0/2066351167 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39b000b580 con 0x7f39b80e5470 2021-02-10T11:14:34.889 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:34.874 7f39abfff700 1 -- 172.21.2.112:0/2066351167 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f39b000b9d0 con 0x7f39b80e5470 2021-02-10T11:14:34.889 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:34.874 7f39bf726700 1 -- 172.21.2.112:0/2066351167 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f39b80e5470 msgr2=0x7f39b8080310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:34.889 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:34.874 7f39bf726700 1 --2- 172.21.2.112:0/2066351167 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f39b80e5470 0x7f39b8080310 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f39b0006f00 tx=0x7f39b0009e90).stop 2021-02-10T11:14:34.889 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:34.874 7f39bf726700 1 -- 172.21.2.112:0/2066351167 shutdown_connections 2021-02-10T11:14:34.889 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:34.874 7f39bf726700 1 --2- 172.21.2.112:0/2066351167 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f39b80e5470 0x7f39b8080310 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:34.889 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:34.874 7f39bf726700 1 --2- 172.21.2.112:0/2066351167 >> v2:172.21.2.120:3300/0 conn(0x7f39b80e5070 0x7f39b81b3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:34.889 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:34.874 7f39bf726700 1 -- 172.21.2.112:0/2066351167 >> 172.21.2.112:0/2066351167 conn(0x7f39b80ce690 msgr2=0x7f39b80e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:34.889 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:34.875 7f39bf726700 1 -- 172.21.2.112:0/2066351167 shutdown_connections 2021-02-10T11:14:34.889 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:34.875 7f39bf726700 1 -- 172.21.2.112:0/2066351167 wait complete. 2021-02-10T11:14:34.889 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:34.889 7f39bf726700 1 Processor -- start 2021-02-10T11:14:34.890 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:34.889 7f39bf726700 1 -- start start 2021-02-10T11:14:34.891 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:34.890 7f39bf726700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f39b809e340 0x7f39b8092f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:34.892 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:34.890 7f39bf726700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f39b8084cb0 0x7f39b80850b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:34.892 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:34.890 7f39bf726700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f39b80856d0 0x7f39b81a5920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:34.892 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:34.890 7f39bf726700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39b81aac70 con 0x7f39b809e340 2021-02-10T11:14:34.892 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:34.890 7f39bf726700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39b8085ad0 con 0x7f39b8084cb0 2021-02-10T11:14:34.892 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:34.890 7f39bf726700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39b8093580 con 0x7f39b80856d0 2021-02-10T11:14:34.892 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:34.890 7f39bd542700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f39b809e340 0x7f39b8092f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:34.892 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:34.890 7f39bd542700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f39b809e340 0x7f39b8092f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:42692/0 (socket says 172.21.2.112:42692) 2021-02-10T11:14:34.892 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:34.890 7f39bd542700 1 -- 172.21.2.112:0/90842743 learned_addr learned my addr 172.21.2.112:0/90842743 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:14:34.892 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:34.890 7f39bdd43700 1 --2- 172.21.2.112:0/90842743 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f39b80856d0 0x7f39b81a5920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:34.892 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:34.890 7f39bcd41700 1 --2- 172.21.2.112:0/90842743 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f39b8084cb0 0x7f39b80850b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:34.892 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:34.890 7f39bd542700 1 -- 172.21.2.112:0/90842743 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f39b80856d0 msgr2=0x7f39b81a5920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:34.892 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:34.890 7f39bd542700 1 --2- 172.21.2.112:0/90842743 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f39b80856d0 0x7f39b81a5920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:34.893 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:34.890 7f39bd542700 1 -- 172.21.2.112:0/90842743 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f39b8084cb0 msgr2=0x7f39b80850b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:34.893 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:34.890 7f39bd542700 1 --2- 172.21.2.112:0/90842743 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f39b8084cb0 0x7f39b80850b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:34.893 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:34.890 7f39bd542700 1 -- 172.21.2.112:0/90842743 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39b81a5f40 con 0x7f39b809e340 2021-02-10T11:14:34.893 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:34.890 7f39bd542700 1 --2- 172.21.2.112:0/90842743 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f39b809e340 0x7f39b8092f60 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f39ac00b560 tx=0x7f39ac00b900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:34.893 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:34.891 7f39a9ffb700 1 -- 172.21.2.112:0/90842743 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f39ac00bcf0 con 0x7f39b809e340 2021-02-10T11:14:34.893 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:34.891 7f39a9ffb700 1 -- 172.21.2.112:0/90842743 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39ac006030 con 0x7f39b809e340 2021-02-10T11:14:34.893 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:34.891 7f39a9ffb700 1 -- 172.21.2.112:0/90842743 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f39ac0054d0 con 0x7f39b809e340 2021-02-10T11:14:34.893 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:34.891 7f39bf726700 1 -- 172.21.2.112:0/90842743 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39b81a5f40 con 0x7f39b809e340 2021-02-10T11:14:34.893 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:34.891 7f39bf726700 1 -- 172.21.2.112:0/90842743 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39b81a64d0 con 0x7f39b809e340 2021-02-10T11:14:34.893 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:34.891 7f39a9ffb700 1 -- 172.21.2.112:0/90842743 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f39ac006210 con 0x7f39b809e340 2021-02-10T11:14:34.893 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:34.891 7f39a9ffb700 1 --2- 172.21.2.112:0/90842743 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f39980186c0 0x7f399801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:34.893 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:34.892 7f39a9ffb700 1 -- 172.21.2.112:0/90842743 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f39ac0325c0 con 0x7f39b809e340 2021-02-10T11:14:34.893 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:34.892 7f39bcd41700 1 --2- 172.21.2.112:0/90842743 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f39980186c0 0x7f399801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:34.893 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:34.892 7f39bcd41700 1 --2- 172.21.2.112:0/90842743 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f39980186c0 0x7f399801abc0 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:34.921 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:34.923 7f39bf726700 1 -- 172.21.2.112:0/90842743 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39b807ed80 con 0x7f39b809e340 2021-02-10T11:14:34.924 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:34.925 7f39a9ffb700 1 -- 172.21.2.112:0/90842743 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f39ac0009b0 con 0x7f39b809e340 2021-02-10T11:14:35.049 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:35.051 7f39bf726700 1 -- 172.21.2.112:0/90842743 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f39b81aabf0 con 0x7f39b809e340 2021-02-10T11:14:35.050 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:35.051 7f39a9ffb700 1 -- 172.21.2.112:0/90842743 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+13993 (secure 0 0 0) 0x7f39ac000c40 con 0x7f39b809e340 2021-02-10T11:14:35.050 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:14:35.050 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":20,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:14:04.629689","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:14:01.400797","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2021-02-10 11:14:02.622129","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:14:35.051 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:35.053 7f39bf726700 1 -- 172.21.2.112:0/90842743 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f39980186c0 msgr2=0x7f399801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:35.051 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:35.053 7f39bf726700 1 --2- 172.21.2.112:0/90842743 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f39980186c0 0x7f399801abc0 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:35.051 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:35.053 7f39bf726700 1 -- 172.21.2.112:0/90842743 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f39b809e340 msgr2=0x7f39b8092f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:35.051 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:35.053 7f39bf726700 1 --2- 172.21.2.112:0/90842743 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f39b809e340 0x7f39b8092f60 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f39ac00b560 tx=0x7f39ac00b900).stop 2021-02-10T11:14:35.052 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:35.053 7f39bf726700 1 -- 172.21.2.112:0/90842743 shutdown_connections 2021-02-10T11:14:35.052 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:35.053 7f39bf726700 1 --2- 172.21.2.112:0/90842743 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f39980186c0 0x7f399801abc0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:35.052 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:35.053 7f39bf726700 1 --2- 172.21.2.112:0/90842743 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f39b8084cb0 0x7f39b80850b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:35.052 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:35.053 7f39bf726700 1 --2- 172.21.2.112:0/90842743 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f39b80856d0 0x7f39b81a5920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:35.052 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:35.053 7f39bf726700 1 --2- 172.21.2.112:0/90842743 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f39b809e340 0x7f39b8092f60 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:35.053 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:35.053 7f39bf726700 1 -- 172.21.2.112:0/90842743 >> 172.21.2.112:0/90842743 conn(0x7f39b819b160 msgr2=0x7f39b8187eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:35.053 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:35.054 7f39bf726700 1 -- 172.21.2.112:0/90842743 shutdown_connections 2021-02-10T11:14:35.053 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:35.054 7f39bf726700 1 -- 172.21.2.112:0/90842743 wait complete. 2021-02-10T11:14:35.060 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:13:57.508758', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:14:01.400797', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:14:02.622129', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:14:35.061 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:14:35.489 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:35.475 7f2d776a1700 1 Processor -- start 2021-02-10T11:14:35.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:35.475 7f2d776a1700 1 -- start start 2021-02-10T11:14:35.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:35.475 7f2d776a1700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f2d701b0470 0x7f2d701b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:35.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:35.475 7f2d776a1700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f2d700f20e0 0x7f2d700f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:35.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:35.475 7f2d776a1700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d700f0b50 con 0x7f2d700ef610 2021-02-10T11:14:35.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:35.475 7f2d776a1700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d700f0820 con 0x7f2d701b0470 2021-02-10T11:14:35.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:35.475 7f2d776a1700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d700f24e0 con 0x7f2d700f20e0 2021-02-10T11:14:35.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:35.476 7f2d754bd700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f2d701b0470 0x7f2d701b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:35.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:35.476 7f2d754bd700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f2d701b0470 0x7f2d701b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:42700/0 (socket says 172.21.2.112:42700) 2021-02-10T11:14:35.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:35.476 7f2d754bd700 1 -- 172.21.2.112:0/3924974969 learned_addr learned my addr 172.21.2.112:0/3924974969 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:14:35.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:35.476 7f2d75cbe700 1 --2- 172.21.2.112:0/3924974969 >> v2:172.21.2.120:3300/0 conn(0x7f2d700f20e0 0x7f2d700f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:35.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:35.476 7f2d754bd700 1 -- 172.21.2.112:0/3924974969 >> v2:172.21.2.120:3300/0 conn(0x7f2d700f20e0 msgr2=0x7f2d700f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:14:35.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:35.476 7f2d754bd700 1 --2- 172.21.2.112:0/3924974969 >> v2:172.21.2.120:3300/0 conn(0x7f2d700f20e0 0x7f2d700f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:35.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:35.476 7f2d754bd700 1 -- 172.21.2.112:0/3924974969 >> v1:172.21.2.120:6789/0 conn(0x7f2d700ef610 legacy=0x7f2d700efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:14:35.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:35.476 7f2d754bd700 1 -- 172.21.2.112:0/3924974969 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d700e52e0 con 0x7f2d701b0470 2021-02-10T11:14:35.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:35.476 7f2d75cbe700 1 --2- 172.21.2.112:0/3924974969 >> v2:172.21.2.120:3300/0 conn(0x7f2d700f20e0 0x7f2d700f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:14:35.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:35.476 7f2d754bd700 1 --2- 172.21.2.112:0/3924974969 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2d701b0470 0x7f2d701b0870 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f2d64006700 tx=0x7f2d64009b80).ready entity=mon.0 client_cookie=450f0e7eaac63980 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:35.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:35.476 7f2d63fff700 1 -- 172.21.2.112:0/3924974969 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2d6400c0c0 con 0x7f2d701b0470 2021-02-10T11:14:35.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:35.476 7f2d63fff700 1 -- 172.21.2.112:0/3924974969 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d6400c340 con 0x7f2d701b0470 2021-02-10T11:14:35.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:35.476 7f2d63fff700 1 -- 172.21.2.112:0/3924974969 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2d6400c860 con 0x7f2d701b0470 2021-02-10T11:14:35.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:35.476 7f2d776a1700 1 -- 172.21.2.112:0/3924974969 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2d701b0470 msgr2=0x7f2d701b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:35.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:35.476 7f2d776a1700 1 --2- 172.21.2.112:0/3924974969 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2d701b0470 0x7f2d701b0870 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f2d64006700 tx=0x7f2d64009b80).stop 2021-02-10T11:14:35.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:35.477 7f2d776a1700 1 -- 172.21.2.112:0/3924974969 shutdown_connections 2021-02-10T11:14:35.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:35.477 7f2d776a1700 1 --2- 172.21.2.112:0/3924974969 >> v2:172.21.2.120:3300/0 conn(0x7f2d700f20e0 0x7f2d700f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:35.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:35.477 7f2d776a1700 1 --2- 172.21.2.112:0/3924974969 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2d701b0470 0x7f2d701b0870 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:35.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:35.477 7f2d776a1700 1 -- 172.21.2.112:0/3924974969 >> 172.21.2.112:0/3924974969 conn(0x7f2d700ce3e0 msgr2=0x7f2d700e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:35.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:35.477 7f2d776a1700 1 -- 172.21.2.112:0/3924974969 shutdown_connections 2021-02-10T11:14:35.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:35.477 7f2d776a1700 1 -- 172.21.2.112:0/3924974969 wait complete. 2021-02-10T11:14:35.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:35.492 7f2d776a1700 1 Processor -- start 2021-02-10T11:14:35.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:35.492 7f2d776a1700 1 -- start start 2021-02-10T11:14:35.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:35.492 7f2d776a1700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2d7008ece0 0x7f2d700a6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:35.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:35.492 7f2d776a1700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2d700a6860 0x7f2d700a6c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:35.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:35.492 7f2d776a1700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2d700a7280 0x7f2d700a4590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:35.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:35.492 7f2d776a1700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d700a5cc0 con 0x7f2d700a7280 2021-02-10T11:14:35.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:35.492 7f2d776a1700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d7008f0e0 con 0x7f2d7008ece0 2021-02-10T11:14:35.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:35.492 7f2d776a1700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d7008f2c0 con 0x7f2d700a6860 2021-02-10T11:14:35.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:35.492 7f2d75cbe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2d700a7280 0x7f2d700a4590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:35.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:35.492 7f2d754bd700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2d7008ece0 0x7f2d700a6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:35.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:35.492 7f2d75cbe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2d700a7280 0x7f2d700a4590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:42702/0 (socket says 172.21.2.112:42702) 2021-02-10T11:14:35.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:35.492 7f2d75cbe700 1 -- 172.21.2.112:0/2080912519 learned_addr learned my addr 172.21.2.112:0/2080912519 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:14:35.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:35.493 7f2d74cbc700 1 --2- 172.21.2.112:0/2080912519 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2d700a6860 0x7f2d700a6c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:35.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:35.493 7f2d75cbe700 1 -- 172.21.2.112:0/2080912519 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2d700a6860 msgr2=0x7f2d700a6c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:35.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:35.493 7f2d75cbe700 1 --2- 172.21.2.112:0/2080912519 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2d700a6860 0x7f2d700a6c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:35.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:35.493 7f2d75cbe700 1 -- 172.21.2.112:0/2080912519 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2d7008ece0 msgr2=0x7f2d700a6240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:35.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:35.493 7f2d75cbe700 1 --2- 172.21.2.112:0/2080912519 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2d7008ece0 0x7f2d700a6240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:35.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:35.493 7f2d75cbe700 1 -- 172.21.2.112:0/2080912519 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d700a4bb0 con 0x7f2d700a7280 2021-02-10T11:14:35.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:35.493 7f2d74cbc700 1 --2- 172.21.2.112:0/2080912519 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2d700a6860 0x7f2d700a6c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:14:35.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:35.493 7f2d75cbe700 1 --2- 172.21.2.112:0/2080912519 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2d700a7280 0x7f2d700a4590 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f2d68009980 tx=0x7f2d6800bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:35.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:35.493 7f2d61ffb700 1 -- 172.21.2.112:0/2080912519 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2d6800c2f0 con 0x7f2d700a7280 2021-02-10T11:14:35.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:35.493 7f2d61ffb700 1 -- 172.21.2.112:0/2080912519 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d68006790 con 0x7f2d700a7280 2021-02-10T11:14:35.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:35.493 7f2d61ffb700 1 -- 172.21.2.112:0/2080912519 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2d6800c7f0 con 0x7f2d700a7280 2021-02-10T11:14:35.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:35.493 7f2d776a1700 1 -- 172.21.2.112:0/2080912519 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d700a4bb0 con 0x7f2d700a7280 2021-02-10T11:14:35.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:35.493 7f2d776a1700 1 -- 172.21.2.112:0/2080912519 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d700a5140 con 0x7f2d700a7280 2021-02-10T11:14:35.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:35.494 7f2d61ffb700 1 -- 172.21.2.112:0/2080912519 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2d680011f0 con 0x7f2d700a7280 2021-02-10T11:14:35.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:35.494 7f2d61ffb700 1 --2- 172.21.2.112:0/2080912519 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2d500186c0 0x7f2d5001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:35.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:35.494 7f2d61ffb700 1 -- 172.21.2.112:0/2080912519 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f2d680328e0 con 0x7f2d700a7280 2021-02-10T11:14:35.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:35.494 7f2d754bd700 1 --2- 172.21.2.112:0/2080912519 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2d500186c0 0x7f2d5001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:35.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:35.495 7f2d754bd700 1 --2- 172.21.2.112:0/2080912519 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2d500186c0 0x7f2d5001abc0 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:35.524 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:35.525 7f2d776a1700 1 -- 172.21.2.112:0/2080912519 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d7019d1a0 con 0x7f2d700a7280 2021-02-10T11:14:35.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:35.528 7f2d61ffb700 1 -- 172.21.2.112:0/2080912519 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2d68000f40 con 0x7f2d700a7280 2021-02-10T11:14:35.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:35.656 7f2d776a1700 1 -- 172.21.2.112:0/2080912519 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f2d7019d1a0 con 0x7f2d700a7280 2021-02-10T11:14:35.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:35.656 7f2d61ffb700 1 -- 172.21.2.112:0/2080912519 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v20) v1 ==== 92+0+10 (secure 0 0 0) 0x7f2d680011f0 con 0x7f2d700a7280 2021-02-10T11:14:35.655 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:14:35.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:35.658 7f2d776a1700 1 -- 172.21.2.112:0/2080912519 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2d500186c0 msgr2=0x7f2d5001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:35.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:35.658 7f2d776a1700 1 --2- 172.21.2.112:0/2080912519 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2d500186c0 0x7f2d5001abc0 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:35.657 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:35.658 7f2d776a1700 1 -- 172.21.2.112:0/2080912519 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2d700a7280 msgr2=0x7f2d700a4590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:35.657 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:35.658 7f2d776a1700 1 --2- 172.21.2.112:0/2080912519 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2d700a7280 0x7f2d700a4590 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f2d68009980 tx=0x7f2d6800bcd0).stop 2021-02-10T11:14:35.657 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:35.658 7f2d776a1700 1 -- 172.21.2.112:0/2080912519 shutdown_connections 2021-02-10T11:14:35.657 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:35.659 7f2d776a1700 1 --2- 172.21.2.112:0/2080912519 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2d500186c0 0x7f2d5001abc0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:35.657 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:35.659 7f2d776a1700 1 --2- 172.21.2.112:0/2080912519 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2d700a7280 0x7f2d700a4590 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:35.657 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:35.659 7f2d776a1700 1 --2- 172.21.2.112:0/2080912519 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2d700a6860 0x7f2d700a6c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:35.658 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:35.659 7f2d776a1700 1 --2- 172.21.2.112:0/2080912519 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2d7008ece0 0x7f2d700a6240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:35.658 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:35.659 7f2d776a1700 1 -- 172.21.2.112:0/2080912519 >> 172.21.2.112:0/2080912519 conn(0x7f2d7008ffe0 msgr2=0x7f2d700bf050 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:35.658 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:35.659 7f2d776a1700 1 -- 172.21.2.112:0/2080912519 shutdown_connections 2021-02-10T11:14:35.658 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:35.659 7f2d776a1700 1 -- 172.21.2.112:0/2080912519 wait complete. 2021-02-10T11:14:35.665 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:14:36.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.078 7f1f4d38d700 1 Processor -- start 2021-02-10T11:14:36.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.078 7f1f4d38d700 1 -- start start 2021-02-10T11:14:36.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.079 7f1f4d38d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f1f481b0470 0x7f1f481b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:36.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.079 7f1f4d38d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1f480ef610 0x7f1f480efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:36.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.079 7f1f4d38d700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f480f0b50 con 0x7f1f480f0030 2021-02-10T11:14:36.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.079 7f1f4d38d700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f480e52e0 con 0x7f1f481b0470 2021-02-10T11:14:36.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.079 7f1f4d38d700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f480e54d0 con 0x7f1f480ef610 2021-02-10T11:14:36.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.079 7f1f4707d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f1f481b0470 0x7f1f481b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:36.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.079 7f1f4707d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f1f481b0470 0x7f1f481b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:42710/0 (socket says 172.21.2.112:42710) 2021-02-10T11:14:36.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.079 7f1f4707d700 1 -- 172.21.2.112:0/3867936648 learned_addr learned my addr 172.21.2.112:0/3867936648 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:14:36.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.079 7f1f4707d700 1 -- 172.21.2.112:0/3867936648 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1f480ef610 msgr2=0x7f1f480efa10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:14:36.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.079 7f1f4707d700 1 --2- 172.21.2.112:0/3867936648 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1f480ef610 0x7f1f480efa10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:36.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.079 7f1f4707d700 1 -- 172.21.2.112:0/3867936648 >> v1:172.21.2.120:6789/0 conn(0x7f1f480f0030 legacy=0x7f1f480f10d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:14:36.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.079 7f1f4707d700 1 -- 172.21.2.112:0/3867936648 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f480f0430 con 0x7f1f481b0470 2021-02-10T11:14:36.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.079 7f1f4707d700 1 --2- 172.21.2.112:0/3867936648 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1f481b0470 0x7f1f481b0870 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f1f38006700 tx=0x7f1f38009b80).ready entity=mon.0 client_cookie=8b5c0db6e3a2f664 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:36.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.079 7f1f4607b700 1 -- 172.21.2.112:0/3867936648 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1f3800c0c0 con 0x7f1f481b0470 2021-02-10T11:14:36.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.079 7f1f4607b700 1 -- 172.21.2.112:0/3867936648 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f3800c340 con 0x7f1f481b0470 2021-02-10T11:14:36.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.079 7f1f4607b700 1 -- 172.21.2.112:0/3867936648 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1f3800c7a0 con 0x7f1f481b0470 2021-02-10T11:14:36.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.079 7f1f4d38d700 1 -- 172.21.2.112:0/3867936648 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1f481b0470 msgr2=0x7f1f481b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:36.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.079 7f1f4d38d700 1 --2- 172.21.2.112:0/3867936648 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1f481b0470 0x7f1f481b0870 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f1f38006700 tx=0x7f1f38009b80).stop 2021-02-10T11:14:36.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.080 7f1f4d38d700 1 -- 172.21.2.112:0/3867936648 shutdown_connections 2021-02-10T11:14:36.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.080 7f1f4d38d700 1 --2- 172.21.2.112:0/3867936648 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1f480ef610 0x7f1f480efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:36.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.080 7f1f4d38d700 1 --2- 172.21.2.112:0/3867936648 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1f481b0470 0x7f1f481b0870 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:36.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.080 7f1f4d38d700 1 -- 172.21.2.112:0/3867936648 >> 172.21.2.112:0/3867936648 conn(0x7f1f480ce3e0 msgr2=0x7f1f480e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:36.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.080 7f1f4d38d700 1 -- 172.21.2.112:0/3867936648 shutdown_connections 2021-02-10T11:14:36.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.080 7f1f4d38d700 1 -- 172.21.2.112:0/3867936648 wait complete. 2021-02-10T11:14:36.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.095 7f1f4d38d700 1 Processor -- start 2021-02-10T11:14:36.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.095 7f1f4d38d700 1 -- start start 2021-02-10T11:14:36.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.095 7f1f4d38d700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1f480ad2e0 0x7f1f480aca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:36.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.095 7f1f4d38d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1f4819dcd0 0x7f1f480ad7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:36.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.095 7f1f4d38d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1f480addd0 0x7f1f480c2d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:36.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.095 7f1f4d38d700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f480a37f0 con 0x7f1f480addd0 2021-02-10T11:14:36.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.095 7f1f4d38d700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f480c2810 con 0x7f1f480ad2e0 2021-02-10T11:14:36.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.095 7f1f4d38d700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f4819e0d0 con 0x7f1f4819dcd0 2021-02-10T11:14:36.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.096 7f1f4787e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1f480addd0 0x7f1f480c2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:36.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.096 7f1f4787e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1f480addd0 0x7f1f480c2d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:42718/0 (socket says 172.21.2.112:42718) 2021-02-10T11:14:36.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.096 7f1f4787e700 1 -- 172.21.2.112:0/2404313108 learned_addr learned my addr 172.21.2.112:0/2404313108 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:14:36.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.096 7f1f4787e700 1 -- 172.21.2.112:0/2404313108 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1f4819dcd0 msgr2=0x7f1f480ad7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:36.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.096 7f1f4787e700 1 --2- 172.21.2.112:0/2404313108 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1f4819dcd0 0x7f1f480ad7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:36.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.096 7f1f4787e700 1 -- 172.21.2.112:0/2404313108 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1f480ad2e0 msgr2=0x7f1f480aca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:36.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.096 7f1f4787e700 1 --2- 172.21.2.112:0/2404313108 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1f480ad2e0 0x7f1f480aca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:36.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.096 7f1f4787e700 1 -- 172.21.2.112:0/2404313108 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f4819e2b0 con 0x7f1f480addd0 2021-02-10T11:14:36.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.096 7f1f4787e700 1 --2- 172.21.2.112:0/2404313108 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1f480addd0 0x7f1f480c2d90 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f1f40009940 tx=0x7f1f4000bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:36.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.096 7f1f3ffff700 1 -- 172.21.2.112:0/2404313108 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1f40003030 con 0x7f1f480addd0 2021-02-10T11:14:36.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.096 7f1f3ffff700 1 -- 172.21.2.112:0/2404313108 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f40001270 con 0x7f1f480addd0 2021-02-10T11:14:36.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.096 7f1f3ffff700 1 -- 172.21.2.112:0/2404313108 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1f400016d0 con 0x7f1f480addd0 2021-02-10T11:14:36.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.096 7f1f4d38d700 1 -- 172.21.2.112:0/2404313108 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f4819e2b0 con 0x7f1f480addd0 2021-02-10T11:14:36.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.096 7f1f4d38d700 1 -- 172.21.2.112:0/2404313108 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f4819e2b0 con 0x7f1f480addd0 2021-02-10T11:14:36.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.097 7f1f3ffff700 1 -- 172.21.2.112:0/2404313108 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1f40001450 con 0x7f1f480addd0 2021-02-10T11:14:36.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.097 7f1f3ffff700 1 --2- 172.21.2.112:0/2404313108 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1f2c01cad0 0x7f1f2c01eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:36.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.097 7f1f3ffff700 1 -- 172.21.2.112:0/2404313108 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f1f40032930 con 0x7f1f480addd0 2021-02-10T11:14:36.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.097 7f1f4707d700 1 --2- 172.21.2.112:0/2404313108 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1f2c01cad0 0x7f1f2c01eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:36.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.098 7f1f4707d700 1 --2- 172.21.2.112:0/2404313108 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1f2c01cad0 0x7f1f2c01eff0 crc :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:36.127 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.129 7f1f4d38d700 1 -- 172.21.2.112:0/2404313108 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f480edd10 con 0x7f1f480addd0 2021-02-10T11:14:36.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.131 7f1f3ffff700 1 -- 172.21.2.112:0/2404313108 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1f4000be30 con 0x7f1f480addd0 2021-02-10T11:14:36.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.260 7f1f4d38d700 1 -- 172.21.2.112:0/2404313108 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f1f480e52e0 con 0x7f1f480addd0 2021-02-10T11:14:36.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.261 7f1f3ffff700 1 -- 172.21.2.112:0/2404313108 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v20) v1 ==== 104+0+10 (secure 0 0 0) 0x7f1f40001450 con 0x7f1f480addd0 2021-02-10T11:14:36.260 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:14:36.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.262 7f1f4d38d700 1 -- 172.21.2.112:0/2404313108 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1f2c01cad0 msgr2=0x7f1f2c01eff0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:36.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.262 7f1f4d38d700 1 --2- 172.21.2.112:0/2404313108 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1f2c01cad0 0x7f1f2c01eff0 crc :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:36.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.262 7f1f4d38d700 1 -- 172.21.2.112:0/2404313108 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1f480addd0 msgr2=0x7f1f480c2d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:36.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.262 7f1f4d38d700 1 --2- 172.21.2.112:0/2404313108 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1f480addd0 0x7f1f480c2d90 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f1f40009940 tx=0x7f1f4000bcf0).stop 2021-02-10T11:14:36.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.263 7f1f4d38d700 1 -- 172.21.2.112:0/2404313108 shutdown_connections 2021-02-10T11:14:36.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.263 7f1f4d38d700 1 --2- 172.21.2.112:0/2404313108 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1f2c01cad0 0x7f1f2c01eff0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:36.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.263 7f1f4d38d700 1 --2- 172.21.2.112:0/2404313108 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1f480addd0 0x7f1f480c2d90 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:36.262 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.263 7f1f4d38d700 1 --2- 172.21.2.112:0/2404313108 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1f4819dcd0 0x7f1f480ad7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:36.262 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.263 7f1f4d38d700 1 --2- 172.21.2.112:0/2404313108 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1f480ad2e0 0x7f1f480aca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:36.262 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.263 7f1f4d38d700 1 -- 172.21.2.112:0/2404313108 >> 172.21.2.112:0/2404313108 conn(0x7f1f480c40e0 msgr2=0x7f1f480f05a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:36.262 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.263 7f1f4d38d700 1 -- 172.21.2.112:0/2404313108 shutdown_connections 2021-02-10T11:14:36.262 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.263 7f1f4d38d700 1 -- 172.21.2.112:0/2404313108 wait complete. 2021-02-10T11:14:36.269 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:14:36.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.682 7f3ae59bb700 1 Processor -- start 2021-02-10T11:14:36.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.682 7f3ae59bb700 1 -- start start 2021-02-10T11:14:36.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.682 7f3ae59bb700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3ae01b0870 0x7f3ae00f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:36.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.682 7f3ae59bb700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ae01b0c70 con 0x7f3ae01b0470 2021-02-10T11:14:36.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.682 7f3ae59bb700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ae00f0b50 con 0x7f3ae00efe00 2021-02-10T11:14:36.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.682 7f3ae59bb700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ae00e5ea0 con 0x7f3ae01b0870 2021-02-10T11:14:36.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.682 7f3adf574700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f3ae01b0470 0x7f3ae00ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:42184/0 (socket says 172.21.2.112:42184) 2021-02-10T11:14:36.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.682 7f3adf574700 1 -- 172.21.2.112:0/262648092 learned_addr learned my addr 172.21.2.112:0/262648092 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:14:36.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.683 7f3aded73700 1 --2- 172.21.2.112:0/262648092 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3ae01b0870 0x7f3ae00f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:36.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.683 7f3ade572700 1 -- 172.21.2.112:0/262648092 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 249129676 0 0) 0x7f3ac80009b0 con 0x7f3ae01b0470 2021-02-10T11:14:36.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.683 7f3ade572700 1 -- 172.21.2.112:0/262648092 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3acc002fe0 con 0x7f3ae01b0470 2021-02-10T11:14:36.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.683 7f3ade572700 1 -- 172.21.2.112:0/262648092 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1621443209 0 0) 0x7f3ac8000e20 con 0x7f3ae01b0470 2021-02-10T11:14:36.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.683 7f3ade572700 1 -- 172.21.2.112:0/262648092 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3ae01b0870 msgr2=0x7f3ae00f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:14:36.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.683 7f3ade572700 1 --2- 172.21.2.112:0/262648092 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3ae01b0870 0x7f3ae00f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:36.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.683 7f3ade572700 1 -- 172.21.2.112:0/262648092 >> v1:172.21.2.120:6789/0 conn(0x7f3ae00efe00 legacy=0x7f3ae00f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:36.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.683 7f3ade572700 1 -- 172.21.2.112:0/262648092 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ae00e52e0 con 0x7f3ae01b0470 2021-02-10T11:14:36.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.683 7f3ade572700 1 -- 172.21.2.112:0/262648092 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f3ac8001530 con 0x7f3ae01b0470 2021-02-10T11:14:36.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.683 7f3aded73700 1 --2- 172.21.2.112:0/262648092 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3ae01b0870 0x7f3ae00f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:14:36.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.683 7f3ade572700 1 -- 172.21.2.112:0/262648092 >> v1:172.21.2.112:6789/0 conn(0x7f3ae01b0470 legacy=0x7f3ae00ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:36.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.683 7f3ade572700 1 --2- 172.21.2.112:0/262648092 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3acc003a60 0x7f3acc005e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:36.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.683 7f3ade572700 1 --2- 172.21.2.112:0/262648092 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3acc006730 0x7f3acc008c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:36.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.683 7f3ade572700 1 --2- 172.21.2.112:0/262648092 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3acc0094f0 0x7f3acc00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:36.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.683 7f3ade572700 1 -- 172.21.2.112:0/262648092 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3acc00c2c0 con 0x7f3acc0094f0 2021-02-10T11:14:36.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.683 7f3ade572700 1 -- 172.21.2.112:0/262648092 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3acc00c4e0 con 0x7f3acc006730 2021-02-10T11:14:36.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.683 7f3ade572700 1 -- 172.21.2.112:0/262648092 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3acc050710 con 0x7f3acc003a60 2021-02-10T11:14:36.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.683 7f3adfd75700 1 --2- 172.21.2.112:0/262648092 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3acc0094f0 0x7f3acc00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:36.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.683 7f3adf574700 1 --2- 172.21.2.112:0/262648092 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3acc003a60 0x7f3acc005e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:36.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.683 7f3adfd75700 1 -- 172.21.2.112:0/262648092 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3acc003a60 msgr2=0x7f3acc005e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:14:36.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.683 7f3adfd75700 1 --2- 172.21.2.112:0/262648092 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3acc003a60 0x7f3acc005e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:36.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.683 7f3aded73700 1 --2- 172.21.2.112:0/262648092 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3acc006730 0x7f3acc008c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:36.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.683 7f3adfd75700 1 -- 172.21.2.112:0/262648092 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3acc006730 msgr2=0x7f3acc008c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:14:36.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.683 7f3adfd75700 1 --2- 172.21.2.112:0/262648092 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3acc006730 0x7f3acc008c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:36.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.683 7f3adfd75700 1 -- 172.21.2.112:0/262648092 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3acc050940 con 0x7f3acc0094f0 2021-02-10T11:14:36.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.683 7f3aded73700 1 --2- 172.21.2.112:0/262648092 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3acc006730 0x7f3acc008c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:14:36.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.684 7f3adfd75700 1 --2- 172.21.2.112:0/262648092 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3acc0094f0 0x7f3acc00ba10 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f3ad8006e40 tx=0x7f3ad8009db0).ready entity=mon.0 client_cookie=fdf5f4b4c55531f5 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:36.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.684 7f3ade572700 1 -- 172.21.2.112:0/262648092 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3ad8006a60 con 0x7f3acc0094f0 2021-02-10T11:14:36.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.684 7f3ade572700 1 -- 172.21.2.112:0/262648092 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ad800b300 con 0x7f3acc0094f0 2021-02-10T11:14:36.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.684 7f3ade572700 1 -- 172.21.2.112:0/262648092 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3ad800b750 con 0x7f3acc0094f0 2021-02-10T11:14:36.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.684 7f3ae59bb700 1 -- 172.21.2.112:0/262648092 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3acc0094f0 msgr2=0x7f3acc00ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:36.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.684 7f3ae59bb700 1 --2- 172.21.2.112:0/262648092 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3acc0094f0 0x7f3acc00ba10 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f3ad8006e40 tx=0x7f3ad8009db0).stop 2021-02-10T11:14:36.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.684 7f3adfd75700 1 -- 172.21.2.112:0/262648092 reap_dead start 2021-02-10T11:14:36.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.684 7f3ae59bb700 1 -- 172.21.2.112:0/262648092 shutdown_connections 2021-02-10T11:14:36.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.684 7f3ae59bb700 1 -- 172.21.2.112:0/262648092 >> 172.21.2.112:0/262648092 conn(0x7f3ae00ce3e0 msgr2=0x7f3ae00e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:36.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.684 7f3ae59bb700 1 -- 172.21.2.112:0/262648092 shutdown_connections 2021-02-10T11:14:36.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.684 7f3ae59bb700 1 -- 172.21.2.112:0/262648092 wait complete. 2021-02-10T11:14:36.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.699 7f3ae59bb700 1 Processor -- start 2021-02-10T11:14:36.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.699 7f3ae59bb700 1 -- start start 2021-02-10T11:14:36.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.699 7f3ae59bb700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3ae009b930 0x7f3ae00b7c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:36.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.699 7f3ae59bb700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3ae00b8410 0x7f3ae00b8810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:36.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.699 7f3ae59bb700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3ae00c1f80 0x7f3ae00c2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:36.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.699 7f3ae59bb700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ae00c1a00 con 0x7f3ae009b930 2021-02-10T11:14:36.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.699 7f3ae59bb700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ae00b8e30 con 0x7f3ae00c1f80 2021-02-10T11:14:36.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.699 7f3ae59bb700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ae00b9050 con 0x7f3ae00b8410 2021-02-10T11:14:36.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.699 7f3adf574700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3ae009b930 0x7f3ae00b7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:36.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.699 7f3adf574700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3ae009b930 0x7f3ae00b7c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:42736/0 (socket says 172.21.2.112:42736) 2021-02-10T11:14:36.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.699 7f3adf574700 1 -- 172.21.2.112:0/862309437 learned_addr learned my addr 172.21.2.112:0/862309437 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:14:36.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.700 7f3adfd75700 1 --2- 172.21.2.112:0/862309437 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3ae00c1f80 0x7f3ae00c2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:36.704 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.700 7f3aded73700 1 --2- 172.21.2.112:0/862309437 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3ae00b8410 0x7f3ae00b8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:36.704 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.700 7f3adf574700 1 -- 172.21.2.112:0/862309437 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3ae00b8410 msgr2=0x7f3ae00b8810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:36.704 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.700 7f3adf574700 1 --2- 172.21.2.112:0/862309437 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3ae00b8410 0x7f3ae00b8810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:36.704 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.700 7f3adf574700 1 -- 172.21.2.112:0/862309437 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3ae00c1f80 msgr2=0x7f3ae00c2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:36.704 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.700 7f3adf574700 1 --2- 172.21.2.112:0/862309437 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3ae00c1f80 0x7f3ae00c2380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:36.704 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.700 7f3adf574700 1 -- 172.21.2.112:0/862309437 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ae00c29a0 con 0x7f3ae009b930 2021-02-10T11:14:36.704 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.700 7f3aded73700 1 --2- 172.21.2.112:0/862309437 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3ae00b8410 0x7f3ae00b8810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:14:36.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.700 7f3adf574700 1 --2- 172.21.2.112:0/862309437 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3ae009b930 0x7f3ae00b7c20 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f3ac800a540 tx=0x7f3ac800a8e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:36.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.700 7f3ad7fff700 1 -- 172.21.2.112:0/862309437 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3ac800acd0 con 0x7f3ae009b930 2021-02-10T11:14:36.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.700 7f3ad7fff700 1 -- 172.21.2.112:0/862309437 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ac8004350 con 0x7f3ae009b930 2021-02-10T11:14:36.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.700 7f3ae59bb700 1 -- 172.21.2.112:0/862309437 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ae00c29a0 con 0x7f3ae009b930 2021-02-10T11:14:36.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.701 7f3ad7fff700 1 -- 172.21.2.112:0/862309437 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3ac8004870 con 0x7f3ae009b930 2021-02-10T11:14:36.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.701 7f3ae59bb700 1 -- 172.21.2.112:0/862309437 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ae00c2f30 con 0x7f3ae009b930 2021-02-10T11:14:36.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.701 7f3ad7fff700 1 -- 172.21.2.112:0/862309437 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3ac8004530 con 0x7f3ae009b930 2021-02-10T11:14:36.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.701 7f3ad7fff700 1 --2- 172.21.2.112:0/862309437 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3ac40186c0 0x7f3ac401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:36.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.701 7f3ad7fff700 1 -- 172.21.2.112:0/862309437 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f3ac80325a0 con 0x7f3ae009b930 2021-02-10T11:14:36.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.702 7f3aded73700 1 --2- 172.21.2.112:0/862309437 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3ac40186c0 0x7f3ac401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:36.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.702 7f3aded73700 1 --2- 172.21.2.112:0/862309437 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3ac40186c0 0x7f3ac401abc0 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:36.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.732 7f3ae59bb700 1 -- 172.21.2.112:0/862309437 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ae0105b30 con 0x7f3ae009b930 2021-02-10T11:14:36.733 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.735 7f3ad7fff700 1 -- 172.21.2.112:0/862309437 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3ac8004b30 con 0x7f3ae009b930 2021-02-10T11:14:36.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.865 7f3ae59bb700 1 -- 172.21.2.112:0/862309437 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f3ae00f0b50 con 0x7f3ae009b930 2021-02-10T11:14:36.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.865 7f3ad7fff700 1 -- 172.21.2.112:0/862309437 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v20) v1 ==== 100+0+10 (secure 0 0 0) 0x7f3ac800acd0 con 0x7f3ae009b930 2021-02-10T11:14:36.864 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:14:36.865 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.867 7f3ae59bb700 1 -- 172.21.2.112:0/862309437 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3ac40186c0 msgr2=0x7f3ac401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:36.865 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.867 7f3ae59bb700 1 --2- 172.21.2.112:0/862309437 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3ac40186c0 0x7f3ac401abc0 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:36.865 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.867 7f3ae59bb700 1 -- 172.21.2.112:0/862309437 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3ae009b930 msgr2=0x7f3ae00b7c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:36.865 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.867 7f3ae59bb700 1 --2- 172.21.2.112:0/862309437 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3ae009b930 0x7f3ae00b7c20 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f3ac800a540 tx=0x7f3ac800a8e0).stop 2021-02-10T11:14:36.865 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.867 7f3ae59bb700 1 -- 172.21.2.112:0/862309437 shutdown_connections 2021-02-10T11:14:36.866 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.867 7f3ae59bb700 1 --2- 172.21.2.112:0/862309437 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3ac40186c0 0x7f3ac401abc0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:36.866 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.867 7f3ae59bb700 1 --2- 172.21.2.112:0/862309437 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3ae00c1f80 0x7f3ae00c2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:36.866 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.867 7f3ae59bb700 1 --2- 172.21.2.112:0/862309437 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3ae00b8410 0x7f3ae00b8810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:36.866 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.867 7f3ae59bb700 1 --2- 172.21.2.112:0/862309437 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3ae009b930 0x7f3ae00b7c20 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:36.866 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.867 7f3ae59bb700 1 -- 172.21.2.112:0/862309437 >> 172.21.2.112:0/862309437 conn(0x7f3ae0094e10 msgr2=0x7f3ae00a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:36.866 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.867 7f3ae59bb700 1 -- 172.21.2.112:0/862309437 shutdown_connections 2021-02-10T11:14:36.866 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:36.867 7f3ae59bb700 1 -- 172.21.2.112:0/862309437 wait complete. 2021-02-10T11:14:36.873 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:14:36.874 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:14:37.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:37.285 7fe4330b8700 1 Processor -- start 2021-02-10T11:14:37.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:37.285 7fe4330b8700 1 -- start start 2021-02-10T11:14:37.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:37.285 7fe4330b8700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fe42c0cff00 0x7fe42c1b39d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:37.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:37.285 7fe4330b8700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe42c0d1bf0 0x7fe42c0ef620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:37.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:37.285 7fe4330b8700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe42c0f0b20 con 0x7fe42c0d17f0 2021-02-10T11:14:37.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:37.285 7fe4330b8700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe42c1b3ff0 con 0x7fe42c0cff00 2021-02-10T11:14:37.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:37.285 7fe4330b8700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe42c0f20b0 con 0x7fe42c0d1bf0 2021-02-10T11:14:37.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:37.286 7fe4316d5700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe42c0d1bf0 0x7fe42c0ef620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:37.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:37.286 7fe4316d5700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe42c0d1bf0 0x7fe42c0ef620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3301/0 says I am v2:172.21.2.112:42426/0 (socket says 172.21.2.112:42426) 2021-02-10T11:14:37.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:37.286 7fe4316d5700 1 -- 172.21.2.112:0/379240563 learned_addr learned my addr 172.21.2.112:0/379240563 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:14:37.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:37.286 7fe42bfff700 1 --2- 172.21.2.112:0/379240563 >> v2:172.21.2.120:3300/0 conn(0x7fe42c0cff00 0x7fe42c1b39d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:37.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:37.286 7fe4316d5700 1 -- 172.21.2.112:0/379240563 >> v2:172.21.2.120:3300/0 conn(0x7fe42c0cff00 msgr2=0x7fe42c1b39d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:14:37.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:37.286 7fe4316d5700 1 --2- 172.21.2.112:0/379240563 >> v2:172.21.2.120:3300/0 conn(0x7fe42c0cff00 0x7fe42c1b39d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:37.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:37.286 7fe4316d5700 1 -- 172.21.2.112:0/379240563 >> v1:172.21.2.120:6789/0 conn(0x7fe42c0d17f0 legacy=0x7fe42c1b31e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:37.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:37.286 7fe4316d5700 1 -- 172.21.2.112:0/379240563 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe42c0f22d0 con 0x7fe42c0d1bf0 2021-02-10T11:14:37.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:37.286 7fe42bfff700 1 --2- 172.21.2.112:0/379240563 >> v2:172.21.2.120:3300/0 conn(0x7fe42c0cff00 0x7fe42c1b39d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:14:37.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:37.286 7fe4316d5700 1 --2- 172.21.2.112:0/379240563 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe42c0d1bf0 0x7fe42c0ef620 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7fe424006720 tx=0x7fe424009ba0).ready entity=mon.2 client_cookie=189475db9339bed0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:37.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:37.287 7fe42b7fe700 1 -- 172.21.2.112:0/379240563 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe424006370 con 0x7fe42c0d1bf0 2021-02-10T11:14:37.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:37.287 7fe42b7fe700 1 -- 172.21.2.112:0/379240563 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4240022e0 con 0x7fe42c0d1bf0 2021-02-10T11:14:37.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:37.287 7fe42b7fe700 1 -- 172.21.2.112:0/379240563 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe424002780 con 0x7fe42c0d1bf0 2021-02-10T11:14:37.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:37.287 7fe4330b8700 1 -- 172.21.2.112:0/379240563 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe42c0d1bf0 msgr2=0x7fe42c0ef620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:37.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:37.287 7fe4330b8700 1 --2- 172.21.2.112:0/379240563 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe42c0d1bf0 0x7fe42c0ef620 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7fe424006720 tx=0x7fe424009ba0).stop 2021-02-10T11:14:37.302 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:37.287 7fe4330b8700 1 -- 172.21.2.112:0/379240563 shutdown_connections 2021-02-10T11:14:37.302 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:37.287 7fe4330b8700 1 --2- 172.21.2.112:0/379240563 >> v2:172.21.2.120:3300/0 conn(0x7fe42c0cff00 0x7fe42c1b39d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:37.302 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:37.287 7fe4330b8700 1 --2- 172.21.2.112:0/379240563 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe42c0d1bf0 0x7fe42c0ef620 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:37.302 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:37.287 7fe4330b8700 1 -- 172.21.2.112:0/379240563 >> 172.21.2.112:0/379240563 conn(0x7fe42c0fde40 msgr2=0x7fe42c0ce390 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:37.302 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:37.287 7fe4330b8700 1 -- 172.21.2.112:0/379240563 shutdown_connections 2021-02-10T11:14:37.302 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:37.288 7fe4330b8700 1 -- 172.21.2.112:0/379240563 wait complete. 2021-02-10T11:14:37.303 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:37.302 7fe4330b8700 1 Processor -- start 2021-02-10T11:14:37.303 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:37.302 7fe4330b8700 1 -- start start 2021-02-10T11:14:37.303 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:37.302 7fe4330b8700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe42c1a7e80 0x7fe42c0ad650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:37.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:37.302 7fe4330b8700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe42c0adc70 0x7fe42c1b0fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:37.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:37.302 7fe4330b8700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe42c1b15c0 0x7fe42c1a5c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:37.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:37.302 7fe4330b8700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe42c1a7980 con 0x7fe42c1b15c0 2021-02-10T11:14:37.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:37.302 7fe4330b8700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe42c198920 con 0x7fe42c1a7e80 2021-02-10T11:14:37.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:37.302 7fe4330b8700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe42c1b19c0 con 0x7fe42c0adc70 2021-02-10T11:14:37.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:37.303 7fe4316d5700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe42c1b15c0 0x7fe42c1a5c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:37.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:37.303 7fe4316d5700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe42c1b15c0 0x7fe42c1a5c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:42746/0 (socket says 172.21.2.112:42746) 2021-02-10T11:14:37.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:37.303 7fe4316d5700 1 -- 172.21.2.112:0/2320734083 learned_addr learned my addr 172.21.2.112:0/2320734083 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:14:37.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:37.303 7fe4316d5700 1 -- 172.21.2.112:0/2320734083 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe42c0adc70 msgr2=0x7fe42c1b0fa0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:14:37.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:37.303 7fe4316d5700 1 --2- 172.21.2.112:0/2320734083 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe42c0adc70 0x7fe42c1b0fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:37.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:37.303 7fe4316d5700 1 -- 172.21.2.112:0/2320734083 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe42c1a7e80 msgr2=0x7fe42c0ad650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:37.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:37.303 7fe4316d5700 1 --2- 172.21.2.112:0/2320734083 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe42c1a7e80 0x7fe42c0ad650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:37.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:37.303 7fe4316d5700 1 -- 172.21.2.112:0/2320734083 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe42c1b1be0 con 0x7fe42c1b15c0 2021-02-10T11:14:37.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:37.303 7fe4316d5700 1 --2- 172.21.2.112:0/2320734083 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe42c1b15c0 0x7fe42c1a5c80 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7fe4240095c0 tx=0x7fe42400b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:37.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:37.303 7fe4297fa700 1 -- 172.21.2.112:0/2320734083 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe42400bd80 con 0x7fe42c1b15c0 2021-02-10T11:14:37.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:37.303 7fe4297fa700 1 -- 172.21.2.112:0/2320734083 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe424006240 con 0x7fe42c1b15c0 2021-02-10T11:14:37.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:37.303 7fe4330b8700 1 -- 172.21.2.112:0/2320734083 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe42c1b1be0 con 0x7fe42c1b15c0 2021-02-10T11:14:37.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:37.303 7fe4330b8700 1 -- 172.21.2.112:0/2320734083 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe42c1a6590 con 0x7fe42c1b15c0 2021-02-10T11:14:37.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:37.303 7fe4297fa700 1 -- 172.21.2.112:0/2320734083 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe424005520 con 0x7fe42c1b15c0 2021-02-10T11:14:37.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:37.304 7fe4297fa700 1 -- 172.21.2.112:0/2320734083 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe424005330 con 0x7fe42c1b15c0 2021-02-10T11:14:37.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:37.304 7fe4297fa700 1 --2- 172.21.2.112:0/2320734083 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe410018730 0x7fe41001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:37.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:37.304 7fe4297fa700 1 -- 172.21.2.112:0/2320734083 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fe4240327e0 con 0x7fe42c1b15c0 2021-02-10T11:14:37.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:37.304 7fe430ed4700 1 --2- 172.21.2.112:0/2320734083 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe410018730 0x7fe41001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:37.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:37.305 7fe430ed4700 1 --2- 172.21.2.112:0/2320734083 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe410018730 0x7fe41001abc0 crc :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:37.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:37.335 7fe4330b8700 1 -- 172.21.2.112:0/2320734083 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe42c1036a0 con 0x7fe42c1b15c0 2021-02-10T11:14:37.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:37.337 7fe4297fa700 1 -- 172.21.2.112:0/2320734083 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe4240090b0 con 0x7fe42c1b15c0 2021-02-10T11:14:37.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:37.466 7fe4330b8700 1 -- 172.21.2.112:0/2320734083 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe42c0f0b20 con 0x7fe42c1b15c0 2021-02-10T11:14:37.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:37.467 7fe4297fa700 1 -- 172.21.2.112:0/2320734083 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+2097 (secure 0 0 0) 0x7fe424006240 con 0x7fe42c1b15c0 2021-02-10T11:14:37.466 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:14:37.466 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4279,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.120:6835/2695560847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":2695560847},{"type":"v1","addr":"172.21.2.120:6835","nonce":2695560847}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":4}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:14:03.627691","modified":"2021-02-10 11:14:13.011189","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4272},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4272":{"gid":4272,"name":"a","rank":0,"incarnation":5,"state":"up:active","state_seq":3,"addr":"172.21.2.112:6833/2584920075","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":2584920075},{"type":"v1","addr":"172.21.2.112:6833","nonce":2584920075}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2021-02-10T11:14:37.466 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 6 2021-02-10T11:14:37.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:37.468 7fe4330b8700 1 -- 172.21.2.112:0/2320734083 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe410018730 msgr2=0x7fe41001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:37.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:37.468 7fe4330b8700 1 --2- 172.21.2.112:0/2320734083 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe410018730 0x7fe41001abc0 crc :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:37.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:37.468 7fe4330b8700 1 -- 172.21.2.112:0/2320734083 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe42c1b15c0 msgr2=0x7fe42c1a5c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:37.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:37.468 7fe4330b8700 1 --2- 172.21.2.112:0/2320734083 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe42c1b15c0 0x7fe42c1a5c80 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7fe4240095c0 tx=0x7fe42400b990).stop 2021-02-10T11:14:37.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:37.469 7fe4330b8700 1 -- 172.21.2.112:0/2320734083 shutdown_connections 2021-02-10T11:14:37.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:37.469 7fe4330b8700 1 --2- 172.21.2.112:0/2320734083 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe410018730 0x7fe41001abc0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:37.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:37.469 7fe4330b8700 1 --2- 172.21.2.112:0/2320734083 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe42c1b15c0 0x7fe42c1a5c80 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:37.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:37.469 7fe4330b8700 1 --2- 172.21.2.112:0/2320734083 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe42c0adc70 0x7fe42c1b0fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:37.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:37.469 7fe4330b8700 1 --2- 172.21.2.112:0/2320734083 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe42c1a7e80 0x7fe42c0ad650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:37.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:37.469 7fe4330b8700 1 -- 172.21.2.112:0/2320734083 >> 172.21.2.112:0/2320734083 conn(0x7fe42c108000 msgr2=0x7fe42c08ccc0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:37.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:37.469 7fe4330b8700 1 -- 172.21.2.112:0/2320734083 shutdown_connections 2021-02-10T11:14:37.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:37.469 7fe4330b8700 1 -- 172.21.2.112:0/2320734083 wait complete. 2021-02-10T11:14:37.678 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 6, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:14:03.627691', 'modified': '2021-02-10 11:14:13.011189', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4272}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4272': {'gid': 4272, 'name': 'a', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.112:6833/2584920075', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.112:6832', 'nonce': 2584920075}, {'type': 'v1', 'addr': '172.21.2.112:6833', 'nonce': 2584920075}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:14:37.678 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:14:37.678 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok status 2021-02-10T11:14:38.076 INFO:teuthology.orchestra.run.gibba012.stdout:{ 2021-02-10T11:14:38.076 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", 2021-02-10T11:14:38.076 INFO:teuthology.orchestra.run.gibba012.stdout: "whoami": 0, 2021-02-10T11:14:38.076 INFO:teuthology.orchestra.run.gibba012.stdout: "id": 4272, 2021-02-10T11:14:38.076 INFO:teuthology.orchestra.run.gibba012.stdout: "want_state": "up:active", 2021-02-10T11:14:38.076 INFO:teuthology.orchestra.run.gibba012.stdout: "state": "up:active", 2021-02-10T11:14:38.076 INFO:teuthology.orchestra.run.gibba012.stdout: "rank_uptime": 26.066606525000001, 2021-02-10T11:14:38.077 INFO:teuthology.orchestra.run.gibba012.stdout: "mdsmap_epoch": 6, 2021-02-10T11:14:38.077 INFO:teuthology.orchestra.run.gibba012.stdout: "osdmap_epoch": 20, 2021-02-10T11:14:38.077 INFO:teuthology.orchestra.run.gibba012.stdout: "osdmap_epoch_barrier": 20, 2021-02-10T11:14:38.077 INFO:teuthology.orchestra.run.gibba012.stdout: "uptime": 30.433649858999999 2021-02-10T11:14:38.077 INFO:teuthology.orchestra.run.gibba012.stdout:} 2021-02-10T11:14:38.080 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", "whoami": 0, "id": 4272, "want_state": "up:active", "state": "up:active", "rank_uptime": 26.066606525000001, "mdsmap_epoch": 6, "osdmap_epoch": 20, "osdmap_epoch_barrier": 20, "uptime": 30.433649858999999 } 2021-02-10T11:14:38.081 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:14:38.081 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2021-02-10T11:14:38.081 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.1 at ubuntu@gibba012.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2021-02-10T11:14:38.081 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:14:38.081 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:14:38.416 DEBUG:teuthology.orchestra.run.gibba012:> sudo modprobe fuse 2021-02-10T11:14:38.759 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:14:38.759 DEBUG:teuthology.orchestra.run.gibba012:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:14:39.103 INFO:teuthology.orchestra.run.gibba012.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:14:39.103 INFO:teuthology.orchestra.run.gibba012.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:14:39.104 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:14:39.104 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:14:39.105 DEBUG:teuthology.orchestra.run.gibba012:> ls /sys/fs/fuse/connections 2021-02-10T11:14:39.440 INFO:teuthology.orchestra.run.gibba012.stdout:42 2021-02-10T11:14:39.441 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [42] 2021-02-10T11:14:39.441 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.1 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:14:39.444 DEBUG:teuthology.orchestra.run.gibba012:> sudo modprobe fuse 2021-02-10T11:14:39.788 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:14:39.788 DEBUG:teuthology.orchestra.run.gibba012:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:14:39.853 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba012.stderr:2021-02-10 11:14:39.855 7f16b3319f80 -1 init, newargv = 0x5576aa0a7cb0 newargc=9 2021-02-10T11:14:39.854 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba012.stderr:ceph-fuse[40499]: starting ceph client 2021-02-10T11:14:39.859 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba012.stderr:ceph-fuse[40499]: starting fuse 2021-02-10T11:14:40.130 INFO:teuthology.orchestra.run.gibba012.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:14:40.131 INFO:teuthology.orchestra.run.gibba012.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:14:40.131 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:14:40.132 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:14:40.132 DEBUG:teuthology.orchestra.run.gibba012:> ls /sys/fs/fuse/connections 2021-02-10T11:14:40.466 INFO:teuthology.orchestra.run.gibba012.stdout:42 2021-02-10T11:14:40.467 INFO:teuthology.orchestra.run.gibba012.stdout:43 2021-02-10T11:14:40.467 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42, 43] 2021-02-10T11:14:40.467 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:14:40.467 DEBUG:teuthology.orchestra.run.gibba012:> sudo python3 -c ' 2021-02-10T11:14:40.467 DEBUG:teuthology.orchestra.run.gibba012:> import glob 2021-02-10T11:14:40.467 DEBUG:teuthology.orchestra.run.gibba012:> import re 2021-02-10T11:14:40.468 DEBUG:teuthology.orchestra.run.gibba012:> import os 2021-02-10T11:14:40.468 DEBUG:teuthology.orchestra.run.gibba012:> import subprocess 2021-02-10T11:14:40.468 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:14:40.468 DEBUG:teuthology.orchestra.run.gibba012:> def find_socket(client_name): 2021-02-10T11:14:40.468 DEBUG:teuthology.orchestra.run.gibba012:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2021-02-10T11:14:40.468 DEBUG:teuthology.orchestra.run.gibba012:> files = glob.glob(asok_path) 2021-02-10T11:14:40.468 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:14:40.468 DEBUG:teuthology.orchestra.run.gibba012:> # Given a non-glob path, it better be there 2021-02-10T11:14:40.468 DEBUG:teuthology.orchestra.run.gibba012:> if "*" not in asok_path: 2021-02-10T11:14:40.468 DEBUG:teuthology.orchestra.run.gibba012:> assert(len(files) == 1) 2021-02-10T11:14:40.468 DEBUG:teuthology.orchestra.run.gibba012:> return files[0] 2021-02-10T11:14:40.468 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:14:40.468 DEBUG:teuthology.orchestra.run.gibba012:> for f in files: 2021-02-10T11:14:40.468 DEBUG:teuthology.orchestra.run.gibba012:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:14:40.469 DEBUG:teuthology.orchestra.run.gibba012:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:14:40.469 DEBUG:teuthology.orchestra.run.gibba012:> return f 2021-02-10T11:14:40.469 DEBUG:teuthology.orchestra.run.gibba012:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:14:40.469 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:14:40.469 DEBUG:teuthology.orchestra.run.gibba012:> print(find_socket("client.1")) 2021-02-10T11:14:40.469 DEBUG:teuthology.orchestra.run.gibba012:> ' 2021-02-10T11:14:40.829 INFO:teuthology.orchestra.run.gibba012.stdout:/var/run/ceph/ceph-client.1.40499.asok 2021-02-10T11:14:40.832 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.40499.asok 2021-02-10T11:14:40.832 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:14:40.832 DEBUG:teuthology.orchestra.run.gibba012:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.40499.asok status 2021-02-10T11:14:41.220 INFO:teuthology.orchestra.run.gibba012.stdout:{ 2021-02-10T11:14:41.220 INFO:teuthology.orchestra.run.gibba012.stdout: "metadata": { 2021-02-10T11:14:41.220 INFO:teuthology.orchestra.run.gibba012.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:14:41.220 INFO:teuthology.orchestra.run.gibba012.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:14:41.220 INFO:teuthology.orchestra.run.gibba012.stdout: "entity_id": "1", 2021-02-10T11:14:41.220 INFO:teuthology.orchestra.run.gibba012.stdout: "hostname": "gibba012", 2021-02-10T11:14:41.220 INFO:teuthology.orchestra.run.gibba012.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2021-02-10T11:14:41.221 INFO:teuthology.orchestra.run.gibba012.stdout: "pid": "40499", 2021-02-10T11:14:41.221 INFO:teuthology.orchestra.run.gibba012.stdout: "root": "/" 2021-02-10T11:14:41.221 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:41.221 INFO:teuthology.orchestra.run.gibba012.stdout: "dentry_count": 0, 2021-02-10T11:14:41.221 INFO:teuthology.orchestra.run.gibba012.stdout: "dentry_pinned_count": 0, 2021-02-10T11:14:41.221 INFO:teuthology.orchestra.run.gibba012.stdout: "id": 4548, 2021-02-10T11:14:41.221 INFO:teuthology.orchestra.run.gibba012.stdout: "inst": { 2021-02-10T11:14:41.221 INFO:teuthology.orchestra.run.gibba012.stdout: "name": { 2021-02-10T11:14:41.221 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "client", 2021-02-10T11:14:41.221 INFO:teuthology.orchestra.run.gibba012.stdout: "num": 4548 2021-02-10T11:14:41.221 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:41.222 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": { 2021-02-10T11:14:41.222 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:14:41.222 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:0", 2021-02-10T11:14:41.222 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 535623389 2021-02-10T11:14:41.222 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:14:41.222 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:41.222 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": { 2021-02-10T11:14:41.222 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:14:41.222 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:0", 2021-02-10T11:14:41.222 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 535623389 2021-02-10T11:14:41.222 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:14:41.222 INFO:teuthology.orchestra.run.gibba012.stdout: "inst_str": "client.4548 172.21.2.112:0/535623389", 2021-02-10T11:14:41.222 INFO:teuthology.orchestra.run.gibba012.stdout: "addr_str": "172.21.2.112:0/535623389", 2021-02-10T11:14:41.223 INFO:teuthology.orchestra.run.gibba012.stdout: "inode_count": 1, 2021-02-10T11:14:41.223 INFO:teuthology.orchestra.run.gibba012.stdout: "mds_epoch": 6, 2021-02-10T11:14:41.223 INFO:teuthology.orchestra.run.gibba012.stdout: "osd_epoch": 20, 2021-02-10T11:14:41.223 INFO:teuthology.orchestra.run.gibba012.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:14:41.223 INFO:teuthology.orchestra.run.gibba012.stdout: "blacklisted": false 2021-02-10T11:14:41.223 INFO:teuthology.orchestra.run.gibba012.stdout:} 2021-02-10T11:14:41.225 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:14:41.649 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:41.634 7fd30fcd5700 1 Processor -- start 2021-02-10T11:14:41.649 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:41.634 7fd30fcd5700 1 -- start start 2021-02-10T11:14:41.649 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:41.634 7fd30fcd5700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fd3080e5070 0x7fd3081b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:41.649 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:41.634 7fd30fcd5700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fd3081b3e40 0x7fd30807d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:41.649 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:41.634 7fd30fcd5700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd30807ed80 con 0x7fd3080e5470 2021-02-10T11:14:41.649 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:41.634 7fd30fcd5700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd308080310 con 0x7fd3080e5070 2021-02-10T11:14:41.649 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:41.634 7fd30fcd5700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd308080530 con 0x7fd3081b3e40 2021-02-10T11:14:41.650 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:41.634 7fd30e2f2700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fd3080e5470 0x7fd30807dcb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:42230/0 (socket says 172.21.2.112:42230) 2021-02-10T11:14:41.650 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:41.635 7fd30e2f2700 1 -- 172.21.2.112:0/2728827449 learned_addr learned my addr 172.21.2.112:0/2728827449 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:14:41.650 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:41.635 7fd30daf1700 1 --2- 172.21.2.112:0/2728827449 >> v2:172.21.2.112:3300/0 conn(0x7fd3080e5070 0x7fd3081b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:41.650 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:41.635 7fd30d2f0700 1 --2- 172.21.2.112:0/2728827449 >> v2:172.21.2.120:3300/0 conn(0x7fd3081b3e40 0x7fd30807d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:41.650 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:41.635 7fd30caef700 1 -- 172.21.2.112:0/2728827449 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2035625158 0 0) 0x7fd300000f90 con 0x7fd3080e5470 2021-02-10T11:14:41.650 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:41.635 7fd30caef700 1 -- 172.21.2.112:0/2728827449 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2f4002fe0 con 0x7fd3080e5470 2021-02-10T11:14:41.650 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:41.635 7fd30daf1700 1 -- 172.21.2.112:0/2728827449 >> v2:172.21.2.120:3300/0 conn(0x7fd3081b3e40 msgr2=0x7fd30807d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:14:41.650 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:41.635 7fd30daf1700 1 --2- 172.21.2.112:0/2728827449 >> v2:172.21.2.120:3300/0 conn(0x7fd3081b3e40 0x7fd30807d690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:41.650 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:41.635 7fd30daf1700 1 -- 172.21.2.112:0/2728827449 >> v1:172.21.2.112:6789/0 conn(0x7fd3080e5470 legacy=0x7fd30807dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:41.650 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:41.635 7fd30daf1700 1 -- 172.21.2.112:0/2728827449 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd308080750 con 0x7fd3080e5070 2021-02-10T11:14:41.650 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:41.635 7fd30daf1700 1 --2- 172.21.2.112:0/2728827449 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd3080e5070 0x7fd3081b3820 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7fd2fc006650 tx=0x7fd2fc009a80).ready entity=mon.0 client_cookie=6ac935b9d4a929e6 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:41.651 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:41.636 7fd30caef700 1 -- 172.21.2.112:0/2728827449 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd2fc0009b0 con 0x7fd3080e5070 2021-02-10T11:14:41.651 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:41.636 7fd30caef700 1 -- 172.21.2.112:0/2728827449 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2fc00c0c0 con 0x7fd3080e5070 2021-02-10T11:14:41.651 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:41.636 7fd30caef700 1 -- 172.21.2.112:0/2728827449 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd2fc00c5e0 con 0x7fd3080e5070 2021-02-10T11:14:41.651 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:41.636 7fd30fcd5700 1 -- 172.21.2.112:0/2728827449 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd3080e5070 msgr2=0x7fd3081b3820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:41.651 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:41.636 7fd30fcd5700 1 --2- 172.21.2.112:0/2728827449 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd3080e5070 0x7fd3081b3820 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7fd2fc006650 tx=0x7fd2fc009a80).stop 2021-02-10T11:14:41.651 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:41.636 7fd30fcd5700 1 -- 172.21.2.112:0/2728827449 shutdown_connections 2021-02-10T11:14:41.651 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:41.636 7fd30fcd5700 1 --2- 172.21.2.112:0/2728827449 >> v2:172.21.2.120:3300/0 conn(0x7fd3081b3e40 0x7fd30807d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:41.651 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:41.636 7fd30fcd5700 1 --2- 172.21.2.112:0/2728827449 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd3080e5070 0x7fd3081b3820 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:41.651 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:41.636 7fd30fcd5700 1 -- 172.21.2.112:0/2728827449 >> 172.21.2.112:0/2728827449 conn(0x7fd3080ce690 msgr2=0x7fd3080e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:41.651 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:41.637 7fd30fcd5700 1 -- 172.21.2.112:0/2728827449 shutdown_connections 2021-02-10T11:14:41.651 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:41.637 7fd30fcd5700 1 -- 172.21.2.112:0/2728827449 wait complete. 2021-02-10T11:14:41.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:41.651 7fd30fcd5700 1 Processor -- start 2021-02-10T11:14:41.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:41.651 7fd30fcd5700 1 -- start start 2021-02-10T11:14:41.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:41.651 7fd30fcd5700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd30808d400 0x7fd308096c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:41.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:41.651 7fd30fcd5700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd3080b1450 0x7fd3080b1850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:41.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:41.651 7fd30fcd5700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd3080b1e70 0x7fd3081b1f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:41.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:41.652 7fd30fcd5700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd308097310 con 0x7fd3080b1e70 2021-02-10T11:14:41.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:41.652 7fd30fcd5700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd30808d800 con 0x7fd30808d400 2021-02-10T11:14:41.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:41.652 7fd30fcd5700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd30808d9f0 con 0x7fd3080b1450 2021-02-10T11:14:41.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:41.652 7fd30e2f2700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd3080b1e70 0x7fd3081b1f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:41.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:41.652 7fd30e2f2700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd3080b1e70 0x7fd3081b1f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:42776/0 (socket says 172.21.2.112:42776) 2021-02-10T11:14:41.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:41.652 7fd30e2f2700 1 -- 172.21.2.112:0/558866260 learned_addr learned my addr 172.21.2.112:0/558866260 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:14:41.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:41.652 7fd30e2f2700 1 -- 172.21.2.112:0/558866260 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd3080b1450 msgr2=0x7fd3080b1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:41.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:41.652 7fd30e2f2700 1 --2- 172.21.2.112:0/558866260 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd3080b1450 0x7fd3080b1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:41.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:41.652 7fd30e2f2700 1 -- 172.21.2.112:0/558866260 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd30808d400 msgr2=0x7fd308096c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:41.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:41.652 7fd30e2f2700 1 --2- 172.21.2.112:0/558866260 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd30808d400 0x7fd308096c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:41.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:41.652 7fd30e2f2700 1 -- 172.21.2.112:0/558866260 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3081b25f0 con 0x7fd3080b1e70 2021-02-10T11:14:41.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:41.652 7fd30e2f2700 1 --2- 172.21.2.112:0/558866260 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd3080b1e70 0x7fd3081b1f60 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7fd30000a8f0 tx=0x7fd30000ac90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:41.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:41.653 7fd2fa7fc700 1 -- 172.21.2.112:0/558866260 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd300007d10 con 0x7fd3080b1e70 2021-02-10T11:14:41.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:41.653 7fd2fa7fc700 1 -- 172.21.2.112:0/558866260 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd300007160 con 0x7fd3080b1e70 2021-02-10T11:14:41.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:41.653 7fd30fcd5700 1 -- 172.21.2.112:0/558866260 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3081b25f0 con 0x7fd3080b1e70 2021-02-10T11:14:41.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:41.653 7fd2fa7fc700 1 -- 172.21.2.112:0/558866260 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd300004720 con 0x7fd3080b1e70 2021-02-10T11:14:41.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:41.653 7fd30fcd5700 1 -- 172.21.2.112:0/558866260 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3081b25f0 con 0x7fd3080b1e70 2021-02-10T11:14:41.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:41.654 7fd2fa7fc700 1 -- 172.21.2.112:0/558866260 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd300007340 con 0x7fd3080b1e70 2021-02-10T11:14:41.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:41.654 7fd2fa7fc700 1 --2- 172.21.2.112:0/558866260 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd2f0018700 0x7fd2f001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:41.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:41.654 7fd2fa7fc700 1 -- 172.21.2.112:0/558866260 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fd300032b00 con 0x7fd3080b1e70 2021-02-10T11:14:41.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:41.654 7fd30daf1700 1 --2- 172.21.2.112:0/558866260 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd2f0018700 0x7fd2f001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:41.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:41.655 7fd30daf1700 1 --2- 172.21.2.112:0/558866260 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd2f0018700 0x7fd2f001abc0 crc :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:41.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:41.684 7fd30fcd5700 1 -- 172.21.2.112:0/558866260 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd30807ed80 con 0x7fd3080b1e70 2021-02-10T11:14:41.686 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:41.687 7fd2fa7fc700 1 -- 172.21.2.112:0/558866260 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd300000d30 con 0x7fd3080b1e70 2021-02-10T11:14:41.810 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:41.812 7fd30fcd5700 1 -- 172.21.2.112:0/558866260 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd30807ed80 con 0x7fd3080b1e70 2021-02-10T11:14:41.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:41.813 7fd2fa7fc700 1 -- 172.21.2.112:0/558866260 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+13993 (secure 0 0 0) 0x7fd300005ef0 con 0x7fd3080b1e70 2021-02-10T11:14:41.811 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:14:41.812 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":20,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:14:04.629689","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:14:01.400797","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2021-02-10 11:14:02.622129","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:14:41.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:41.814 7fd30fcd5700 1 -- 172.21.2.112:0/558866260 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd2f0018700 msgr2=0x7fd2f001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:41.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:41.814 7fd30fcd5700 1 --2- 172.21.2.112:0/558866260 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd2f0018700 0x7fd2f001abc0 crc :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:41.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:41.814 7fd30fcd5700 1 -- 172.21.2.112:0/558866260 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd3080b1e70 msgr2=0x7fd3081b1f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:41.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:41.814 7fd30fcd5700 1 --2- 172.21.2.112:0/558866260 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd3080b1e70 0x7fd3081b1f60 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7fd30000a8f0 tx=0x7fd30000ac90).stop 2021-02-10T11:14:41.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:41.815 7fd30fcd5700 1 -- 172.21.2.112:0/558866260 shutdown_connections 2021-02-10T11:14:41.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:41.815 7fd30fcd5700 1 --2- 172.21.2.112:0/558866260 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd2f0018700 0x7fd2f001abc0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:41.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:41.815 7fd30fcd5700 1 --2- 172.21.2.112:0/558866260 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd3080b1e70 0x7fd3081b1f60 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:41.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:41.815 7fd30fcd5700 1 --2- 172.21.2.112:0/558866260 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd3080b1450 0x7fd3080b1850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:41.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:41.815 7fd30fcd5700 1 --2- 172.21.2.112:0/558866260 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd30808d400 0x7fd308096c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:41.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:41.815 7fd30fcd5700 1 -- 172.21.2.112:0/558866260 >> 172.21.2.112:0/558866260 conn(0x7fd3081ac620 msgr2=0x7fd3080896e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:41.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:41.815 7fd30fcd5700 1 -- 172.21.2.112:0/558866260 shutdown_connections 2021-02-10T11:14:41.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:41.815 7fd30fcd5700 1 -- 172.21.2.112:0/558866260 wait complete. 2021-02-10T11:14:41.821 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:13:57.508758', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:14:01.400797', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:14:02.622129', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:14:41.822 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:14:42.250 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.235 7f81a1e7f700 1 Processor -- start 2021-02-10T11:14:42.250 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.235 7f81a1e7f700 1 -- start start 2021-02-10T11:14:42.250 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.235 7f81a1e7f700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f819c1b0470 0x7f819c1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:42.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.235 7f81a1e7f700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f819c0ef610 0x7f819c0efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:42.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.235 7f81a1e7f700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f819c0f0b50 con 0x7f819c0f0030 2021-02-10T11:14:42.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.235 7f81a1e7f700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f819c0e52e0 con 0x7f819c1b0470 2021-02-10T11:14:42.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.235 7f81a1e7f700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f819c0e54d0 con 0x7f819c0ef610 2021-02-10T11:14:42.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.236 7f819bfff700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f819c0f0030 0x7f819c0f10d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:42246/0 (socket says 172.21.2.112:42246) 2021-02-10T11:14:42.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.236 7f819bfff700 1 -- 172.21.2.112:0/2232462679 learned_addr learned my addr 172.21.2.112:0/2232462679 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:14:42.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.236 7f819b7fe700 1 --2- 172.21.2.112:0/2232462679 >> v2:172.21.2.120:3300/0 conn(0x7f819c1b0470 0x7f819c1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:42.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.236 7f819affd700 1 --2- 172.21.2.112:0/2232462679 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f819c0ef610 0x7f819c0efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:42.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.236 7f819a7fc700 1 -- 172.21.2.112:0/2232462679 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1794773859 0 0) 0x7f8194001500 con 0x7f819c0f0030 2021-02-10T11:14:42.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.236 7f819a7fc700 1 -- 172.21.2.112:0/2232462679 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8188002fe0 con 0x7f819c0f0030 2021-02-10T11:14:42.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.236 7f819affd700 1 -- 172.21.2.112:0/2232462679 >> v2:172.21.2.120:3300/0 conn(0x7f819c1b0470 msgr2=0x7f819c1b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:14:42.252 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.236 7f819affd700 1 --2- 172.21.2.112:0/2232462679 >> v2:172.21.2.120:3300/0 conn(0x7f819c1b0470 0x7f819c1b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:42.252 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.236 7f819affd700 1 -- 172.21.2.112:0/2232462679 >> v1:172.21.2.112:6789/0 conn(0x7f819c0f0030 legacy=0x7f819c0f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:42.252 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.236 7f819affd700 1 -- 172.21.2.112:0/2232462679 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f819c0f0430 con 0x7f819c0ef610 2021-02-10T11:14:42.252 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.236 7f819b7fe700 1 --2- 172.21.2.112:0/2232462679 >> v2:172.21.2.120:3300/0 conn(0x7f819c1b0470 0x7f819c1b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:14:42.252 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.237 7f819affd700 1 --2- 172.21.2.112:0/2232462679 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f819c0ef610 0x7f819c0efa10 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7f818c006650 tx=0x7f818c009a80).ready entity=mon.2 client_cookie=7203553ea9975cc2 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:42.252 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.237 7f819a7fc700 1 -- 172.21.2.112:0/2232462679 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f818c00c0c0 con 0x7f819c0ef610 2021-02-10T11:14:42.252 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.237 7f819a7fc700 1 -- 172.21.2.112:0/2232462679 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f818c00c340 con 0x7f819c0ef610 2021-02-10T11:14:42.252 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.237 7f819a7fc700 1 -- 172.21.2.112:0/2232462679 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f818c0009b0 con 0x7f819c0ef610 2021-02-10T11:14:42.252 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.237 7f81a1e7f700 1 -- 172.21.2.112:0/2232462679 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f819c0ef610 msgr2=0x7f819c0efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:42.252 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.237 7f81a1e7f700 1 --2- 172.21.2.112:0/2232462679 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f819c0ef610 0x7f819c0efa10 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7f818c006650 tx=0x7f818c009a80).stop 2021-02-10T11:14:42.252 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.238 7f81a1e7f700 1 -- 172.21.2.112:0/2232462679 shutdown_connections 2021-02-10T11:14:42.252 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.238 7f81a1e7f700 1 --2- 172.21.2.112:0/2232462679 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f819c0ef610 0x7f819c0efa10 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:42.253 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.238 7f81a1e7f700 1 --2- 172.21.2.112:0/2232462679 >> v2:172.21.2.120:3300/0 conn(0x7f819c1b0470 0x7f819c1b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:42.253 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.238 7f81a1e7f700 1 -- 172.21.2.112:0/2232462679 >> 172.21.2.112:0/2232462679 conn(0x7f819c0ce3e0 msgr2=0x7f819c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:42.253 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.238 7f81a1e7f700 1 -- 172.21.2.112:0/2232462679 shutdown_connections 2021-02-10T11:14:42.253 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.238 7f81a1e7f700 1 -- 172.21.2.112:0/2232462679 wait complete. 2021-02-10T11:14:42.255 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.252 7f81a1e7f700 1 Processor -- start 2021-02-10T11:14:42.255 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.252 7f81a1e7f700 1 -- start start 2021-02-10T11:14:42.255 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.253 7f81a1e7f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f819c0a0420 0x7f819c0a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:42.255 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.253 7f81a1e7f700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f819c189d00 0x7f819c18a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:42.255 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.253 7f81a1e7f700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f819c18a720 0x7f819c0b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:42.255 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.253 7f81a1e7f700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f819c189780 con 0x7f819c0a0420 2021-02-10T11:14:42.255 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.253 7f81a1e7f700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f819c18ab20 con 0x7f819c18a720 2021-02-10T11:14:42.255 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.253 7f81a1e7f700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f819c0a1140 con 0x7f819c189d00 2021-02-10T11:14:42.255 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.253 7f819b7fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f819c0a0420 0x7f819c0a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:42.255 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.253 7f819b7fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f819c0a0420 0x7f819c0a0820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:42786/0 (socket says 172.21.2.112:42786) 2021-02-10T11:14:42.255 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.253 7f819b7fe700 1 -- 172.21.2.112:0/831569371 learned_addr learned my addr 172.21.2.112:0/831569371 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:14:42.255 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.254 7f819bfff700 1 --2- 172.21.2.112:0/831569371 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f819c18a720 0x7f819c0b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:42.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.254 7f819b7fe700 1 -- 172.21.2.112:0/831569371 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f819c189d00 msgr2=0x7f819c18a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:42.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.254 7f819b7fe700 1 --2- 172.21.2.112:0/831569371 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f819c189d00 0x7f819c18a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:42.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.254 7f819b7fe700 1 -- 172.21.2.112:0/831569371 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f819c18a720 msgr2=0x7f819c0b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:42.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.254 7f819b7fe700 1 --2- 172.21.2.112:0/831569371 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f819c18a720 0x7f819c0b7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:42.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.254 7f819b7fe700 1 -- 172.21.2.112:0/831569371 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f819c0b8200 con 0x7f819c0a0420 2021-02-10T11:14:42.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.254 7f819bfff700 1 --2- 172.21.2.112:0/831569371 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f819c18a720 0x7f819c0b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:14:42.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.254 7f819b7fe700 1 --2- 172.21.2.112:0/831569371 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f819c0a0420 0x7f819c0a0820 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f8184006f40 tx=0x7f818400b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:42.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.254 7f8193fff700 1 -- 172.21.2.112:0/831569371 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f818400bd80 con 0x7f819c0a0420 2021-02-10T11:14:42.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.254 7f8193fff700 1 -- 172.21.2.112:0/831569371 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8184000c90 con 0x7f819c0a0420 2021-02-10T11:14:42.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.254 7f8193fff700 1 -- 172.21.2.112:0/831569371 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f81840022b0 con 0x7f819c0a0420 2021-02-10T11:14:42.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.254 7f81a1e7f700 1 -- 172.21.2.112:0/831569371 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f819c0b8200 con 0x7f819c0a0420 2021-02-10T11:14:42.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.254 7f81a1e7f700 1 -- 172.21.2.112:0/831569371 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f819c0b8790 con 0x7f819c0a0420 2021-02-10T11:14:42.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.255 7f8193fff700 1 -- 172.21.2.112:0/831569371 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f818400a000 con 0x7f819c0a0420 2021-02-10T11:14:42.257 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.255 7f8193fff700 1 --2- 172.21.2.112:0/831569371 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f81800186c0 0x7f818001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:42.257 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.255 7f8193fff700 1 -- 172.21.2.112:0/831569371 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f8184032790 con 0x7f819c0a0420 2021-02-10T11:14:42.257 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.255 7f819affd700 1 --2- 172.21.2.112:0/831569371 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f81800186c0 0x7f818001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:42.257 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.256 7f819affd700 1 --2- 172.21.2.112:0/831569371 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f81800186c0 0x7f818001abc0 crc :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:42.284 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.285 7f81a1e7f700 1 -- 172.21.2.112:0/831569371 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f819c105ae0 con 0x7f819c0a0420 2021-02-10T11:14:42.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.288 7f8193fff700 1 -- 172.21.2.112:0/831569371 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8184000940 con 0x7f819c0a0420 2021-02-10T11:14:42.412 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.414 7f81a1e7f700 1 -- 172.21.2.112:0/831569371 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f819c0f0b50 con 0x7f819c0a0420 2021-02-10T11:14:42.413 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.414 7f8193fff700 1 -- 172.21.2.112:0/831569371 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v20) v1 ==== 92+0+10 (secure 0 0 0) 0x7f818400a000 con 0x7f819c0a0420 2021-02-10T11:14:42.413 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:14:42.414 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.416 7f81a1e7f700 1 -- 172.21.2.112:0/831569371 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f81800186c0 msgr2=0x7f818001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:42.414 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.416 7f81a1e7f700 1 --2- 172.21.2.112:0/831569371 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f81800186c0 0x7f818001abc0 crc :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:42.415 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.416 7f81a1e7f700 1 -- 172.21.2.112:0/831569371 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f819c0a0420 msgr2=0x7f819c0a0820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:42.415 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.416 7f81a1e7f700 1 --2- 172.21.2.112:0/831569371 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f819c0a0420 0x7f819c0a0820 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f8184006f40 tx=0x7f818400b990).stop 2021-02-10T11:14:42.415 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.416 7f81a1e7f700 1 -- 172.21.2.112:0/831569371 shutdown_connections 2021-02-10T11:14:42.415 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.416 7f81a1e7f700 1 --2- 172.21.2.112:0/831569371 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f81800186c0 0x7f818001abc0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:42.415 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.416 7f81a1e7f700 1 --2- 172.21.2.112:0/831569371 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f819c18a720 0x7f819c0b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:42.415 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.416 7f81a1e7f700 1 --2- 172.21.2.112:0/831569371 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f819c189d00 0x7f819c18a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:42.415 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.416 7f81a1e7f700 1 --2- 172.21.2.112:0/831569371 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f819c0a0420 0x7f819c0a0820 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:42.415 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.416 7f81a1e7f700 1 -- 172.21.2.112:0/831569371 >> 172.21.2.112:0/831569371 conn(0x7f819c094e10 msgr2=0x7f819c0a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:42.415 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.417 7f81a1e7f700 1 -- 172.21.2.112:0/831569371 shutdown_connections 2021-02-10T11:14:42.416 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.417 7f81a1e7f700 1 -- 172.21.2.112:0/831569371 wait complete. 2021-02-10T11:14:42.423 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:14:42.853 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.837 7f75887fb700 1 Processor -- start 2021-02-10T11:14:42.853 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.837 7f75887fb700 1 -- start start 2021-02-10T11:14:42.853 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.838 7f75887fb700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f75801b0870 0x7f75800f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:42.853 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.838 7f75887fb700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75801b0c70 con 0x7f75800efe00 2021-02-10T11:14:42.853 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.838 7f75887fb700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75800f0b50 con 0x7f75801b0470 2021-02-10T11:14:42.853 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.838 7f75887fb700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75800e5ea0 con 0x7f75801b0870 2021-02-10T11:14:42.854 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.838 7f7586e18700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f75800efe00 0x7f75800f0200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:42260/0 (socket says 172.21.2.112:42260) 2021-02-10T11:14:42.854 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.838 7f7586e18700 1 -- 172.21.2.112:0/53029614 learned_addr learned my addr 172.21.2.112:0/53029614 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:14:42.854 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.839 7f7585e16700 1 --2- 172.21.2.112:0/53029614 >> v2:172.21.2.120:3300/0 conn(0x7f75801b0870 0x7f75800f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:42.854 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.839 7f7585615700 1 -- 172.21.2.112:0/53029614 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 141914470 0 0) 0x7f7578001180 con 0x7f75800efe00 2021-02-10T11:14:42.854 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.839 7f7585615700 1 -- 172.21.2.112:0/53029614 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f756c002fe0 con 0x7f75800efe00 2021-02-10T11:14:42.854 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.839 7f7585615700 1 -- 172.21.2.112:0/53029614 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 990290210 0 0) 0x7f757c000c60 con 0x7f75801b0470 2021-02-10T11:14:42.854 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.839 7f7585615700 1 -- 172.21.2.112:0/53029614 --> v1:172.21.2.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f756c003210 con 0x7f75801b0470 2021-02-10T11:14:42.854 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.839 7f7585615700 1 -- 172.21.2.112:0/53029614 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3017325491 0 0) 0x7f7578001180 con 0x7f75800efe00 2021-02-10T11:14:42.854 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.839 7f7585e16700 1 -- 172.21.2.112:0/53029614 >> v1:172.21.2.120:6789/0 conn(0x7f75801b0470 legacy=0x7f75800ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:42.854 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.839 7f7585e16700 1 -- 172.21.2.112:0/53029614 >> v1:172.21.2.112:6789/0 conn(0x7f75800efe00 legacy=0x7f75800f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:42.854 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.839 7f7585e16700 1 -- 172.21.2.112:0/53029614 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75800e52e0 con 0x7f75801b0870 2021-02-10T11:14:42.854 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.839 7f7585e16700 1 --2- 172.21.2.112:0/53029614 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f75801b0870 0x7f75800f20e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f75740069d0 tx=0x7f7574009e40).ready entity=mon.1 client_cookie=528378a4a0be6849 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:42.855 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.840 7f7585615700 1 -- 172.21.2.112:0/53029614 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f757400b510 con 0x7f75801b0870 2021-02-10T11:14:42.855 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.840 7f7585615700 1 -- 172.21.2.112:0/53029614 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f757400b890 con 0x7f75801b0870 2021-02-10T11:14:42.855 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.840 7f7585615700 1 -- 172.21.2.112:0/53029614 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f757400bdb0 con 0x7f75801b0870 2021-02-10T11:14:42.855 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.840 7f75887fb700 1 -- 172.21.2.112:0/53029614 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f75801b0870 msgr2=0x7f75800f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:42.855 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.840 7f75887fb700 1 --2- 172.21.2.112:0/53029614 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f75801b0870 0x7f75800f20e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f75740069d0 tx=0x7f7574009e40).stop 2021-02-10T11:14:42.855 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.841 7f75887fb700 1 -- 172.21.2.112:0/53029614 shutdown_connections 2021-02-10T11:14:42.855 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.841 7f75887fb700 1 --2- 172.21.2.112:0/53029614 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f75801b0870 0x7f75800f20e0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:42.855 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.841 7f75887fb700 1 -- 172.21.2.112:0/53029614 >> 172.21.2.112:0/53029614 conn(0x7f75800ce3e0 msgr2=0x7f75800e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:42.855 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.841 7f75887fb700 1 -- 172.21.2.112:0/53029614 shutdown_connections 2021-02-10T11:14:42.855 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.841 7f75887fb700 1 -- 172.21.2.112:0/53029614 wait complete. 2021-02-10T11:14:42.857 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.855 7f75887fb700 1 Processor -- start 2021-02-10T11:14:42.857 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.855 7f75887fb700 1 -- start start 2021-02-10T11:14:42.857 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.856 7f75887fb700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f75800a0420 0x7f75800a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:42.857 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.856 7f75887fb700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7580189d00 0x7f758018a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:42.857 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.856 7f75887fb700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f758018a720 0x7f75800b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:42.857 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.856 7f75887fb700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7580189780 con 0x7f75800a0420 2021-02-10T11:14:42.857 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.856 7f75887fb700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f758018ab20 con 0x7f7580189d00 2021-02-10T11:14:42.857 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.856 7f75887fb700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75800a1140 con 0x7f758018a720 2021-02-10T11:14:42.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.856 7f7586617700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f75800a0420 0x7f75800a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:42.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.856 7f7586617700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f75800a0420 0x7f75800a0820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:42802/0 (socket says 172.21.2.112:42802) 2021-02-10T11:14:42.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.856 7f7586617700 1 -- 172.21.2.112:0/1609494024 learned_addr learned my addr 172.21.2.112:0/1609494024 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:14:42.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.856 7f7586617700 1 -- 172.21.2.112:0/1609494024 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f758018a720 msgr2=0x7f75800b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:42.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.856 7f7586617700 1 --2- 172.21.2.112:0/1609494024 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f758018a720 0x7f75800b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:42.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.856 7f7585e16700 1 --2- 172.21.2.112:0/1609494024 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7580189d00 0x7f758018a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:42.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.856 7f7586617700 1 -- 172.21.2.112:0/1609494024 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7580189d00 msgr2=0x7f758018a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:42.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.856 7f7586617700 1 --2- 172.21.2.112:0/1609494024 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7580189d00 0x7f758018a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:42.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.856 7f7586617700 1 -- 172.21.2.112:0/1609494024 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75800b8200 con 0x7f75800a0420 2021-02-10T11:14:42.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.856 7f7585e16700 1 --2- 172.21.2.112:0/1609494024 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7580189d00 0x7f758018a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:14:42.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.857 7f7586617700 1 --2- 172.21.2.112:0/1609494024 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f75800a0420 0x7f75800a0820 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f757c006f00 tx=0x7f757c00b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:42.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.857 7f7572ffd700 1 -- 172.21.2.112:0/1609494024 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f757c00bd30 con 0x7f75800a0420 2021-02-10T11:14:42.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.857 7f7572ffd700 1 -- 172.21.2.112:0/1609494024 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f757c000e20 con 0x7f75800a0420 2021-02-10T11:14:42.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.857 7f7572ffd700 1 -- 172.21.2.112:0/1609494024 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f757c0054f0 con 0x7f75800a0420 2021-02-10T11:14:42.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.857 7f75887fb700 1 -- 172.21.2.112:0/1609494024 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75800b8200 con 0x7f75800a0420 2021-02-10T11:14:42.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.857 7f75887fb700 1 -- 172.21.2.112:0/1609494024 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75800b8200 con 0x7f75800a0420 2021-02-10T11:14:42.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.858 7f7572ffd700 1 -- 172.21.2.112:0/1609494024 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f757c006240 con 0x7f75800a0420 2021-02-10T11:14:42.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.858 7f7572ffd700 1 --2- 172.21.2.112:0/1609494024 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f75680186d0 0x7f756801abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:42.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.858 7f7572ffd700 1 -- 172.21.2.112:0/1609494024 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f757c032830 con 0x7f75800a0420 2021-02-10T11:14:42.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.858 7f7585e16700 1 --2- 172.21.2.112:0/1609494024 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f75680186d0 0x7f756801abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:42.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.859 7f7585e16700 1 --2- 172.21.2.112:0/1609494024 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f75680186d0 0x7f756801abd0 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:42.887 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.888 7f75887fb700 1 -- 172.21.2.112:0/1609494024 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7580105ae0 con 0x7f75800a0420 2021-02-10T11:14:42.889 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:42.891 7f7572ffd700 1 -- 172.21.2.112:0/1609494024 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f757c006030 con 0x7f75800a0420 2021-02-10T11:14:43.017 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:43.019 7f75887fb700 1 -- 172.21.2.112:0/1609494024 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f7580105ae0 con 0x7f75800a0420 2021-02-10T11:14:43.018 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:43.019 7f7572ffd700 1 -- 172.21.2.112:0/1609494024 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v20) v1 ==== 104+0+10 (secure 0 0 0) 0x7f757c006240 con 0x7f75800a0420 2021-02-10T11:14:43.018 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:14:43.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:43.021 7f75887fb700 1 -- 172.21.2.112:0/1609494024 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f75680186d0 msgr2=0x7f756801abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:43.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:43.021 7f75887fb700 1 --2- 172.21.2.112:0/1609494024 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f75680186d0 0x7f756801abd0 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:43.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:43.021 7f75887fb700 1 -- 172.21.2.112:0/1609494024 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f75800a0420 msgr2=0x7f75800a0820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:43.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:43.021 7f75887fb700 1 --2- 172.21.2.112:0/1609494024 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f75800a0420 0x7f75800a0820 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f757c006f00 tx=0x7f757c00b940).stop 2021-02-10T11:14:43.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:43.022 7f75887fb700 1 -- 172.21.2.112:0/1609494024 shutdown_connections 2021-02-10T11:14:43.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:43.022 7f75887fb700 1 --2- 172.21.2.112:0/1609494024 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f75680186d0 0x7f756801abd0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:43.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:43.022 7f75887fb700 1 --2- 172.21.2.112:0/1609494024 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7580189d00 0x7f758018a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:43.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:43.022 7f75887fb700 1 --2- 172.21.2.112:0/1609494024 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f758018a720 0x7f75800b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:43.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:43.022 7f75887fb700 1 --2- 172.21.2.112:0/1609494024 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f75800a0420 0x7f75800a0820 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:43.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:43.022 7f75887fb700 1 -- 172.21.2.112:0/1609494024 >> 172.21.2.112:0/1609494024 conn(0x7f7580094e10 msgr2=0x7f75800a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:43.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:43.022 7f75887fb700 1 -- 172.21.2.112:0/1609494024 shutdown_connections 2021-02-10T11:14:43.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:43.022 7f75887fb700 1 -- 172.21.2.112:0/1609494024 wait complete. 2021-02-10T11:14:43.028 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:14:43.457 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:43.441 7f3e31a64700 1 Processor -- start 2021-02-10T11:14:43.457 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:43.441 7f3e31a64700 1 -- start start 2021-02-10T11:14:43.457 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:43.442 7f3e31a64700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f3e2c1b0470 0x7f3e2c1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:43.457 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:43.442 7f3e31a64700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e2c0f0b50 con 0x7f3e2c0ef610 2021-02-10T11:14:43.457 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:43.442 7f3e31a64700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e2c0f24e0 con 0x7f3e2c0f20e0 2021-02-10T11:14:43.457 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:43.442 7f3e31a64700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e2c0e5ea0 con 0x7f3e2c1b0470 2021-02-10T11:14:43.457 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:43.442 7f3e2affd700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f3e2c0ef610 0x7f3e2c0efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:42268/0 (socket says 172.21.2.112:42268) 2021-02-10T11:14:43.458 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:43.442 7f3e2affd700 1 -- 172.21.2.112:0/1157315719 learned_addr learned my addr 172.21.2.112:0/1157315719 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:14:43.458 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:43.442 7f3e2b7fe700 1 --2- 172.21.2.112:0/1157315719 >> v2:172.21.2.120:3300/0 conn(0x7f3e2c1b0470 0x7f3e2c1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:43.458 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:43.443 7f3e2a7fc700 1 -- 172.21.2.112:0/1157315719 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3798779325 0 0) 0x7f3e1c000ce0 con 0x7f3e2c0ef610 2021-02-10T11:14:43.458 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:43.443 7f3e2a7fc700 1 -- 172.21.2.112:0/1157315719 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e18002fe0 con 0x7f3e2c0ef610 2021-02-10T11:14:43.458 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:43.443 7f3e2b7fe700 1 -- 172.21.2.112:0/1157315719 >> v1:172.21.2.120:6789/0 conn(0x7f3e2c0f20e0 legacy=0x7f3e2c0f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:43.458 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:43.443 7f3e2b7fe700 1 -- 172.21.2.112:0/1157315719 >> v1:172.21.2.112:6789/0 conn(0x7f3e2c0ef610 legacy=0x7f3e2c0efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:43.458 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:43.443 7f3e2b7fe700 1 -- 172.21.2.112:0/1157315719 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e2c0e52e0 con 0x7f3e2c1b0470 2021-02-10T11:14:43.458 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:43.443 7f3e2b7fe700 1 --2- 172.21.2.112:0/1157315719 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3e2c1b0470 0x7f3e2c1b0870 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f3e14006650 tx=0x7f3e14009a80).ready entity=mon.1 client_cookie=bc537e3129b89d2d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:43.458 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:43.443 7f3e2a7fc700 1 -- 172.21.2.112:0/1157315719 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3e1400c0c0 con 0x7f3e2c1b0470 2021-02-10T11:14:43.458 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:43.443 7f3e2a7fc700 1 -- 172.21.2.112:0/1157315719 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e1400c340 con 0x7f3e2c1b0470 2021-02-10T11:14:43.458 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:43.443 7f3e2a7fc700 1 -- 172.21.2.112:0/1157315719 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3e1400c860 con 0x7f3e2c1b0470 2021-02-10T11:14:43.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:43.443 7f3e31a64700 1 -- 172.21.2.112:0/1157315719 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3e2c1b0470 msgr2=0x7f3e2c1b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:43.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:43.443 7f3e31a64700 1 --2- 172.21.2.112:0/1157315719 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3e2c1b0470 0x7f3e2c1b0870 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f3e14006650 tx=0x7f3e14009a80).stop 2021-02-10T11:14:43.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:43.444 7f3e31a64700 1 -- 172.21.2.112:0/1157315719 shutdown_connections 2021-02-10T11:14:43.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:43.444 7f3e31a64700 1 --2- 172.21.2.112:0/1157315719 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3e2c1b0470 0x7f3e2c1b0870 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:43.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:43.444 7f3e31a64700 1 -- 172.21.2.112:0/1157315719 >> 172.21.2.112:0/1157315719 conn(0x7f3e2c0ce3e0 msgr2=0x7f3e2c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:43.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:43.444 7f3e31a64700 1 -- 172.21.2.112:0/1157315719 shutdown_connections 2021-02-10T11:14:43.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:43.444 7f3e31a64700 1 -- 172.21.2.112:0/1157315719 wait complete. 2021-02-10T11:14:43.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:43.458 7f3e31a64700 1 Processor -- start 2021-02-10T11:14:43.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:43.459 7f3e31a64700 1 -- start start 2021-02-10T11:14:43.461 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:43.459 7f3e31a64700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3e2c094fb0 0x7f3e2c09e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:43.461 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:43.459 7f3e31a64700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3e2c0c2be0 0x7f3e2c0c2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:43.461 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:43.459 7f3e31a64700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3e2c0c3600 0x7f3e2c0a9c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:43.461 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:43.459 7f3e31a64700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e2c09ebf0 con 0x7f3e2c0c2be0 2021-02-10T11:14:43.461 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:43.459 7f3e31a64700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e2c0aa2a0 con 0x7f3e2c094fb0 2021-02-10T11:14:43.461 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:43.459 7f3e31a64700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e2c0aa4c0 con 0x7f3e2c0c3600 2021-02-10T11:14:43.461 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:43.459 7f3e2affd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3e2c0c2be0 0x7f3e2c0c2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:43.461 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:43.459 7f3e2affd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3e2c0c2be0 0x7f3e2c0c2fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:42814/0 (socket says 172.21.2.112:42814) 2021-02-10T11:14:43.461 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:43.459 7f3e2affd700 1 -- 172.21.2.112:0/3155846563 learned_addr learned my addr 172.21.2.112:0/3155846563 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:14:43.461 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:43.460 7f3e2affd700 1 -- 172.21.2.112:0/3155846563 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3e2c0c3600 msgr2=0x7f3e2c0a9c80 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:14:43.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:43.460 7f3e2affd700 1 --2- 172.21.2.112:0/3155846563 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3e2c0c3600 0x7f3e2c0a9c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:43.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:43.460 7f3e2affd700 1 -- 172.21.2.112:0/3155846563 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3e2c094fb0 msgr2=0x7f3e2c09e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:43.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:43.460 7f3e2affd700 1 --2- 172.21.2.112:0/3155846563 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3e2c094fb0 0x7f3e2c09e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:43.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:43.460 7f3e2affd700 1 -- 172.21.2.112:0/3155846563 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e2c0aa6e0 con 0x7f3e2c0c2be0 2021-02-10T11:14:43.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:43.460 7f3e2affd700 1 --2- 172.21.2.112:0/3155846563 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3e2c0c2be0 0x7f3e2c0c2fe0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f3e1c00a5d0 tx=0x7f3e1c00a970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:43.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:43.460 7f3e23fff700 1 -- 172.21.2.112:0/3155846563 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3e1c000ce0 con 0x7f3e2c0c2be0 2021-02-10T11:14:43.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:43.460 7f3e23fff700 1 -- 172.21.2.112:0/3155846563 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e1c0043a0 con 0x7f3e2c0c2be0 2021-02-10T11:14:43.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:43.460 7f3e31a64700 1 -- 172.21.2.112:0/3155846563 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e2c0aa6e0 con 0x7f3e2c0c2be0 2021-02-10T11:14:43.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:43.460 7f3e31a64700 1 -- 172.21.2.112:0/3155846563 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e2c0aac70 con 0x7f3e2c0c2be0 2021-02-10T11:14:43.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:43.460 7f3e23fff700 1 -- 172.21.2.112:0/3155846563 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3e1c004800 con 0x7f3e2c0c2be0 2021-02-10T11:14:43.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:43.461 7f3e23fff700 1 -- 172.21.2.112:0/3155846563 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3e1c004cd0 con 0x7f3e2c0c2be0 2021-02-10T11:14:43.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:43.461 7f3e23fff700 1 --2- 172.21.2.112:0/3155846563 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3e10018730 0x7f3e1001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:43.463 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:43.461 7f3e23fff700 1 -- 172.21.2.112:0/3155846563 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f3e1c032650 con 0x7f3e2c0c2be0 2021-02-10T11:14:43.463 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:43.461 7f3e2b7fe700 1 --2- 172.21.2.112:0/3155846563 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3e10018730 0x7f3e1001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:43.463 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:43.461 7f3e2b7fe700 1 --2- 172.21.2.112:0/3155846563 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3e10018730 0x7f3e1001abc0 crc :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:43.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:43.492 7f3e31a64700 1 -- 172.21.2.112:0/3155846563 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e2c0a9750 con 0x7f3e2c0c2be0 2021-02-10T11:14:43.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:43.494 7f3e23fff700 1 -- 172.21.2.112:0/3155846563 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3e1c004aa0 con 0x7f3e2c0c2be0 2021-02-10T11:14:43.619 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:43.621 7f3e31a64700 1 -- 172.21.2.112:0/3155846563 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f3e2c0a9750 con 0x7f3e2c0c2be0 2021-02-10T11:14:43.620 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:43.621 7f3e23fff700 1 -- 172.21.2.112:0/3155846563 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v20) v1 ==== 100+0+10 (secure 0 0 0) 0x7f3e1c0043a0 con 0x7f3e2c0c2be0 2021-02-10T11:14:43.620 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:14:43.621 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:43.623 7f3e31a64700 1 -- 172.21.2.112:0/3155846563 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3e10018730 msgr2=0x7f3e1001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:43.621 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:43.623 7f3e31a64700 1 --2- 172.21.2.112:0/3155846563 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3e10018730 0x7f3e1001abc0 crc :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:43.621 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:43.623 7f3e31a64700 1 -- 172.21.2.112:0/3155846563 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3e2c0c2be0 msgr2=0x7f3e2c0c2fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:43.621 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:43.623 7f3e31a64700 1 --2- 172.21.2.112:0/3155846563 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3e2c0c2be0 0x7f3e2c0c2fe0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f3e1c00a5d0 tx=0x7f3e1c00a970).stop 2021-02-10T11:14:43.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:43.623 7f3e31a64700 1 -- 172.21.2.112:0/3155846563 shutdown_connections 2021-02-10T11:14:43.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:43.623 7f3e31a64700 1 --2- 172.21.2.112:0/3155846563 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3e10018730 0x7f3e1001abc0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:43.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:43.623 7f3e31a64700 1 --2- 172.21.2.112:0/3155846563 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3e2c0c3600 0x7f3e2c0a9c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:43.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:43.623 7f3e31a64700 1 --2- 172.21.2.112:0/3155846563 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3e2c0c2be0 0x7f3e2c0c2fe0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:43.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:43.623 7f3e31a64700 1 --2- 172.21.2.112:0/3155846563 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3e2c094fb0 0x7f3e2c09e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:43.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:43.623 7f3e31a64700 1 -- 172.21.2.112:0/3155846563 >> 172.21.2.112:0/3155846563 conn(0x7f3e2c09f630 msgr2=0x7f3e2c0a0130 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:43.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:43.624 7f3e31a64700 1 -- 172.21.2.112:0/3155846563 shutdown_connections 2021-02-10T11:14:43.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:43.624 7f3e31a64700 1 -- 172.21.2.112:0/3155846563 wait complete. 2021-02-10T11:14:43.631 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:14:43.631 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:14:44.055 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:44.040 7f633b10d700 1 Processor -- start 2021-02-10T11:14:44.056 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:44.040 7f633b10d700 1 -- start start 2021-02-10T11:14:44.056 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:44.041 7f633b10d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f633407e6a0 0x7f63340d1790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:44.056 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:44.041 7f633b10d700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f633407ed80 con 0x7f63340e5070 2021-02-10T11:14:44.056 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:44.041 7f633b10d700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f633407eaa0 con 0x7f63340e5470 2021-02-10T11:14:44.056 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:44.041 7f633b10d700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63341b4080 con 0x7f633407e6a0 2021-02-10T11:14:44.056 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:44.041 7f632bfff700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7f63340e5470 0x7f633407f300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:35842/0 (socket says 172.21.2.112:35842) 2021-02-10T11:14:44.056 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:44.041 7f632bfff700 1 -- 172.21.2.112:0/2822640364 learned_addr learned my addr 172.21.2.112:0/2822640364 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:14:44.056 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:44.041 7f633972a700 1 --2- 172.21.2.112:0/2822640364 >> v2:172.21.2.112:3300/0 conn(0x7f633407e6a0 0x7f63340d1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:44.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:44.042 7f632b7fe700 1 -- 172.21.2.112:0/2822640364 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 40778061 0 0) 0x7f6324000f30 con 0x7f63340e5070 2021-02-10T11:14:44.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:44.042 7f632b7fe700 1 -- 172.21.2.112:0/2822640364 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f631c002fe0 con 0x7f63340e5070 2021-02-10T11:14:44.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:44.042 7f632b7fe700 1 -- 172.21.2.112:0/2822640364 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1022326768 0 0) 0x7f63300010b0 con 0x7f63340e5470 2021-02-10T11:14:44.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:44.042 7f632b7fe700 1 -- 172.21.2.112:0/2822640364 --> v1:172.21.2.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f631c004370 con 0x7f63340e5470 2021-02-10T11:14:44.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:44.042 7f633972a700 1 -- 172.21.2.112:0/2822640364 >> v1:172.21.2.120:6789/0 conn(0x7f63340e5470 legacy=0x7f633407f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:44.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:44.042 7f633972a700 1 -- 172.21.2.112:0/2822640364 >> v1:172.21.2.112:6789/0 conn(0x7f63340e5070 legacy=0x7f63341b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:44.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:44.042 7f633972a700 1 -- 172.21.2.112:0/2822640364 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63340d1db0 con 0x7f633407e6a0 2021-02-10T11:14:44.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:44.042 7f633972a700 1 --2- 172.21.2.112:0/2822640364 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f633407e6a0 0x7f63340d1790 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f632c006e30 tx=0x7f632c009e60).ready entity=mon.0 client_cookie=b7f7d0867d59e446 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:44.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:44.042 7f632b7fe700 1 -- 172.21.2.112:0/2822640364 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f632c006a50 con 0x7f633407e6a0 2021-02-10T11:14:44.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:44.042 7f632b7fe700 1 -- 172.21.2.112:0/2822640364 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f632c00b300 con 0x7f633407e6a0 2021-02-10T11:14:44.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:44.042 7f632b7fe700 1 -- 172.21.2.112:0/2822640364 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f632c00b820 con 0x7f633407e6a0 2021-02-10T11:14:44.058 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:44.042 7f633b10d700 1 -- 172.21.2.112:0/2822640364 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f633407e6a0 msgr2=0x7f63340d1790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:44.058 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:44.042 7f633b10d700 1 --2- 172.21.2.112:0/2822640364 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f633407e6a0 0x7f63340d1790 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f632c006e30 tx=0x7f632c009e60).stop 2021-02-10T11:14:44.058 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:44.043 7f633b10d700 1 -- 172.21.2.112:0/2822640364 shutdown_connections 2021-02-10T11:14:44.058 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:44.043 7f633b10d700 1 --2- 172.21.2.112:0/2822640364 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f633407e6a0 0x7f63340d1790 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:44.058 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:44.043 7f633b10d700 1 -- 172.21.2.112:0/2822640364 >> 172.21.2.112:0/2822640364 conn(0x7f63340ce690 msgr2=0x7f63340e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:44.058 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:44.043 7f633b10d700 1 -- 172.21.2.112:0/2822640364 shutdown_connections 2021-02-10T11:14:44.058 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:44.043 7f633b10d700 1 -- 172.21.2.112:0/2822640364 wait complete. 2021-02-10T11:14:44.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:44.057 7f633b10d700 1 Processor -- start 2021-02-10T11:14:44.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:44.058 7f633b10d700 1 -- start start 2021-02-10T11:14:44.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:44.058 7f633b10d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f63340ae8c0 0x7f6334096c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:44.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:44.058 7f633b10d700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6334081aa0 0x7f6334187eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:44.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:44.058 7f633b10d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6334081ea0 0x7f63341884d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:44.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:44.058 7f633b10d700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63340815a0 con 0x7f6334081ea0 2021-02-10T11:14:44.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:44.058 7f633b10d700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6334097290 con 0x7f6334081aa0 2021-02-10T11:14:44.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:44.058 7f633b10d700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6334081230 con 0x7f63340ae8c0 2021-02-10T11:14:44.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:44.058 7f633972a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6334081ea0 0x7f63341884d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:44.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:44.058 7f633972a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6334081ea0 0x7f63341884d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:42830/0 (socket says 172.21.2.112:42830) 2021-02-10T11:14:44.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:44.058 7f633972a700 1 -- 172.21.2.112:0/2892329837 learned_addr learned my addr 172.21.2.112:0/2892329837 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:14:44.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:44.058 7f6338f29700 1 --2- 172.21.2.112:0/2892329837 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f63340ae8c0 0x7f6334096c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:44.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:44.058 7f632bfff700 1 --2- 172.21.2.112:0/2892329837 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6334081aa0 0x7f6334187eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:44.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:44.059 7f633972a700 1 -- 172.21.2.112:0/2892329837 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f63340ae8c0 msgr2=0x7f6334096c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:44.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:44.059 7f633972a700 1 --2- 172.21.2.112:0/2892329837 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f63340ae8c0 0x7f6334096c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:44.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:44.059 7f633972a700 1 -- 172.21.2.112:0/2892329837 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6334081aa0 msgr2=0x7f6334187eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:44.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:44.059 7f633972a700 1 --2- 172.21.2.112:0/2892329837 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6334081aa0 0x7f6334187eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:44.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:44.059 7f633972a700 1 -- 172.21.2.112:0/2892329837 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f633408da00 con 0x7f6334081ea0 2021-02-10T11:14:44.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:44.059 7f633972a700 1 --2- 172.21.2.112:0/2892329837 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6334081ea0 0x7f63341884d0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f632c00b970 tx=0x7f632c00bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:44.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:44.059 7f63297fa700 1 -- 172.21.2.112:0/2892329837 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f632c009380 con 0x7f6334081ea0 2021-02-10T11:14:44.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:44.059 7f63297fa700 1 -- 172.21.2.112:0/2892329837 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f632c005820 con 0x7f6334081ea0 2021-02-10T11:14:44.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:44.059 7f633b10d700 1 -- 172.21.2.112:0/2892329837 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f633408da00 con 0x7f6334081ea0 2021-02-10T11:14:44.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:44.059 7f633b10d700 1 -- 172.21.2.112:0/2892329837 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f633408ced0 con 0x7f6334081ea0 2021-02-10T11:14:44.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:44.059 7f63297fa700 1 -- 172.21.2.112:0/2892329837 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f632c005c80 con 0x7f6334081ea0 2021-02-10T11:14:44.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:44.060 7f63297fa700 1 -- 172.21.2.112:0/2892329837 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f632c005a00 con 0x7f6334081ea0 2021-02-10T11:14:44.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:44.060 7f63297fa700 1 --2- 172.21.2.112:0/2892329837 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f63140186c0 0x7f631401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:44.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:44.060 7f63297fa700 1 -- 172.21.2.112:0/2892329837 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f632c0328b0 con 0x7f6334081ea0 2021-02-10T11:14:44.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:44.060 7f6338f29700 1 --2- 172.21.2.112:0/2892329837 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f63140186c0 0x7f631401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:44.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:44.061 7f6338f29700 1 --2- 172.21.2.112:0/2892329837 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f63140186c0 0x7f631401abc0 crc :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:44.088 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:44.090 7f633b10d700 1 -- 172.21.2.112:0/2892329837 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f633407ed80 con 0x7f6334081ea0 2021-02-10T11:14:44.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:44.093 7f63297fa700 1 -- 172.21.2.112:0/2892329837 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f632c02cf00 con 0x7f6334081ea0 2021-02-10T11:14:44.219 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:44.221 7f633b10d700 1 -- 172.21.2.112:0/2892329837 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6334081520 con 0x7f6334081ea0 2021-02-10T11:14:44.220 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:44.221 7f63297fa700 1 -- 172.21.2.112:0/2892329837 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+2097 (secure 0 0 0) 0x7f632c02cf00 con 0x7f6334081ea0 2021-02-10T11:14:44.220 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:14:44.220 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4279,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.120:6835/2695560847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":2695560847},{"type":"v1","addr":"172.21.2.120:6835","nonce":2695560847}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":4}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:14:03.627691","modified":"2021-02-10 11:14:13.011189","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4272},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4272":{"gid":4272,"name":"a","rank":0,"incarnation":5,"state":"up:active","state_seq":3,"addr":"172.21.2.112:6833/2584920075","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":2584920075},{"type":"v1","addr":"172.21.2.112:6833","nonce":2584920075}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2021-02-10T11:14:44.220 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 6 2021-02-10T11:14:44.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:44.223 7f633b10d700 1 -- 172.21.2.112:0/2892329837 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f63140186c0 msgr2=0x7f631401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:44.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:44.223 7f633b10d700 1 --2- 172.21.2.112:0/2892329837 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f63140186c0 0x7f631401abc0 crc :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:44.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:44.223 7f633b10d700 1 -- 172.21.2.112:0/2892329837 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6334081ea0 msgr2=0x7f63341884d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:44.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:44.223 7f633b10d700 1 --2- 172.21.2.112:0/2892329837 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6334081ea0 0x7f63341884d0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f632c00b970 tx=0x7f632c00bd10).stop 2021-02-10T11:14:44.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:44.223 7f633b10d700 1 -- 172.21.2.112:0/2892329837 shutdown_connections 2021-02-10T11:14:44.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:44.223 7f633b10d700 1 --2- 172.21.2.112:0/2892329837 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f63140186c0 0x7f631401abc0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:44.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:44.223 7f633b10d700 1 --2- 172.21.2.112:0/2892329837 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6334081aa0 0x7f6334187eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:44.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:44.224 7f633b10d700 1 --2- 172.21.2.112:0/2892329837 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6334081ea0 0x7f63341884d0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:44.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:44.224 7f633b10d700 1 --2- 172.21.2.112:0/2892329837 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f63340ae8c0 0x7f6334096c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:44.223 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:44.224 7f633b10d700 1 -- 172.21.2.112:0/2892329837 >> 172.21.2.112:0/2892329837 conn(0x7f633419b930 msgr2=0x7f63341a0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:44.223 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:44.224 7f633b10d700 1 -- 172.21.2.112:0/2892329837 shutdown_connections 2021-02-10T11:14:44.223 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:44.224 7f633b10d700 1 -- 172.21.2.112:0/2892329837 wait complete. 2021-02-10T11:14:44.431 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 6, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:14:03.627691', 'modified': '2021-02-10 11:14:13.011189', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4272}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4272': {'gid': 4272, 'name': 'a', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.112:6833/2584920075', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.112:6832', 'nonce': 2584920075}, {'type': 'v1', 'addr': '172.21.2.112:6833', 'nonce': 2584920075}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:14:44.432 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:14:44.432 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok status 2021-02-10T11:14:44.832 INFO:teuthology.orchestra.run.gibba012.stdout:{ 2021-02-10T11:14:44.832 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", 2021-02-10T11:14:44.832 INFO:teuthology.orchestra.run.gibba012.stdout: "whoami": 0, 2021-02-10T11:14:44.832 INFO:teuthology.orchestra.run.gibba012.stdout: "id": 4272, 2021-02-10T11:14:44.832 INFO:teuthology.orchestra.run.gibba012.stdout: "want_state": "up:active", 2021-02-10T11:14:44.833 INFO:teuthology.orchestra.run.gibba012.stdout: "state": "up:active", 2021-02-10T11:14:44.833 INFO:teuthology.orchestra.run.gibba012.stdout: "rank_uptime": 32.822657043, 2021-02-10T11:14:44.833 INFO:teuthology.orchestra.run.gibba012.stdout: "mdsmap_epoch": 6, 2021-02-10T11:14:44.833 INFO:teuthology.orchestra.run.gibba012.stdout: "osdmap_epoch": 20, 2021-02-10T11:14:44.833 INFO:teuthology.orchestra.run.gibba012.stdout: "osdmap_epoch_barrier": 20, 2021-02-10T11:14:44.833 INFO:teuthology.orchestra.run.gibba012.stdout: "uptime": 37.189700807999998 2021-02-10T11:14:44.833 INFO:teuthology.orchestra.run.gibba012.stdout:} 2021-02-10T11:14:44.836 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", "whoami": 0, "id": 4272, "want_state": "up:active", "state": "up:active", "rank_uptime": 32.822657043, "mdsmap_epoch": 6, "osdmap_epoch": 20, "osdmap_epoch_barrier": 20, "uptime": 37.189700807999998 } 2021-02-10T11:14:44.837 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:14:44.837 INFO:tasks.cephfs.fuse_mount:Client client.2 config is {} 2021-02-10T11:14:44.837 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.2 at ubuntu@gibba020.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.2... 2021-02-10T11:14:44.837 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:14:44.837 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:14:45.171 DEBUG:teuthology.orchestra.run.gibba020:> sudo modprobe fuse 2021-02-10T11:14:45.554 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:14:45.554 DEBUG:teuthology.orchestra.run.gibba020:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:14:45.898 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:14:45.898 DEBUG:teuthology.orchestra.run.gibba020:> ls /sys/fs/fuse/connections 2021-02-10T11:14:46.236 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2021-02-10T11:14:46.237 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.2 /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:14:46.275 DEBUG:teuthology.orchestra.run.gibba020:> sudo modprobe fuse 2021-02-10T11:14:46.616 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:14:46.616 DEBUG:teuthology.orchestra.run.gibba020:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:14:46.682 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.gibba020.stderr:ceph-fuse[37208]: starting ceph client 2021-02-10T11:14:46.682 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.gibba020.stderr:2021-02-10 11:14:46.684 7f2c0c1d4f80 -1 init, newargv = 0x55768bbf2aa0 newargc=9 2021-02-10T11:14:46.687 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.gibba020.stderr:ceph-fuse[37208]: starting fuse 2021-02-10T11:14:46.965 INFO:teuthology.orchestra.run.gibba020.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:14:46.965 INFO:teuthology.orchestra.run.gibba020.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:14:46.966 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:14:46.966 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:14:46.966 DEBUG:teuthology.orchestra.run.gibba020:> ls /sys/fs/fuse/connections 2021-02-10T11:14:47.312 INFO:teuthology.orchestra.run.gibba020.stdout:42 2021-02-10T11:14:47.312 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2021-02-10T11:14:47.312 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:14:47.313 DEBUG:teuthology.orchestra.run.gibba020:> sudo python3 -c ' 2021-02-10T11:14:47.313 DEBUG:teuthology.orchestra.run.gibba020:> import glob 2021-02-10T11:14:47.313 DEBUG:teuthology.orchestra.run.gibba020:> import re 2021-02-10T11:14:47.313 DEBUG:teuthology.orchestra.run.gibba020:> import os 2021-02-10T11:14:47.313 DEBUG:teuthology.orchestra.run.gibba020:> import subprocess 2021-02-10T11:14:47.313 DEBUG:teuthology.orchestra.run.gibba020:> 2021-02-10T11:14:47.313 DEBUG:teuthology.orchestra.run.gibba020:> def find_socket(client_name): 2021-02-10T11:14:47.313 DEBUG:teuthology.orchestra.run.gibba020:> asok_path = "/var/run/ceph/ceph-client.2.*.asok" 2021-02-10T11:14:47.313 DEBUG:teuthology.orchestra.run.gibba020:> files = glob.glob(asok_path) 2021-02-10T11:14:47.313 DEBUG:teuthology.orchestra.run.gibba020:> 2021-02-10T11:14:47.313 DEBUG:teuthology.orchestra.run.gibba020:> # Given a non-glob path, it better be there 2021-02-10T11:14:47.313 DEBUG:teuthology.orchestra.run.gibba020:> if "*" not in asok_path: 2021-02-10T11:14:47.313 DEBUG:teuthology.orchestra.run.gibba020:> assert(len(files) == 1) 2021-02-10T11:14:47.314 DEBUG:teuthology.orchestra.run.gibba020:> return files[0] 2021-02-10T11:14:47.314 DEBUG:teuthology.orchestra.run.gibba020:> 2021-02-10T11:14:47.314 DEBUG:teuthology.orchestra.run.gibba020:> for f in files: 2021-02-10T11:14:47.314 DEBUG:teuthology.orchestra.run.gibba020:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:14:47.314 DEBUG:teuthology.orchestra.run.gibba020:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:14:47.314 DEBUG:teuthology.orchestra.run.gibba020:> return f 2021-02-10T11:14:47.314 DEBUG:teuthology.orchestra.run.gibba020:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:14:47.314 DEBUG:teuthology.orchestra.run.gibba020:> 2021-02-10T11:14:47.314 DEBUG:teuthology.orchestra.run.gibba020:> print(find_socket("client.2")) 2021-02-10T11:14:47.314 DEBUG:teuthology.orchestra.run.gibba020:> ' 2021-02-10T11:14:47.692 INFO:teuthology.orchestra.run.gibba020.stdout:/var/run/ceph/ceph-client.2.37208.asok 2021-02-10T11:14:47.695 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.2.37208.asok 2021-02-10T11:14:47.696 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:14:47.696 DEBUG:teuthology.orchestra.run.gibba020:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.2.37208.asok status 2021-02-10T11:14:48.090 INFO:teuthology.orchestra.run.gibba020.stdout:{ 2021-02-10T11:14:48.091 INFO:teuthology.orchestra.run.gibba020.stdout: "metadata": { 2021-02-10T11:14:48.091 INFO:teuthology.orchestra.run.gibba020.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:14:48.091 INFO:teuthology.orchestra.run.gibba020.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:14:48.091 INFO:teuthology.orchestra.run.gibba020.stdout: "entity_id": "2", 2021-02-10T11:14:48.091 INFO:teuthology.orchestra.run.gibba020.stdout: "hostname": "gibba020", 2021-02-10T11:14:48.091 INFO:teuthology.orchestra.run.gibba020.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.2", 2021-02-10T11:14:48.092 INFO:teuthology.orchestra.run.gibba020.stdout: "pid": "37208", 2021-02-10T11:14:48.092 INFO:teuthology.orchestra.run.gibba020.stdout: "root": "/" 2021-02-10T11:14:48.092 INFO:teuthology.orchestra.run.gibba020.stdout: }, 2021-02-10T11:14:48.092 INFO:teuthology.orchestra.run.gibba020.stdout: "dentry_count": 0, 2021-02-10T11:14:48.092 INFO:teuthology.orchestra.run.gibba020.stdout: "dentry_pinned_count": 0, 2021-02-10T11:14:48.092 INFO:teuthology.orchestra.run.gibba020.stdout: "id": 4498, 2021-02-10T11:14:48.092 INFO:teuthology.orchestra.run.gibba020.stdout: "inst": { 2021-02-10T11:14:48.092 INFO:teuthology.orchestra.run.gibba020.stdout: "name": { 2021-02-10T11:14:48.092 INFO:teuthology.orchestra.run.gibba020.stdout: "type": "client", 2021-02-10T11:14:48.092 INFO:teuthology.orchestra.run.gibba020.stdout: "num": 4498 2021-02-10T11:14:48.092 INFO:teuthology.orchestra.run.gibba020.stdout: }, 2021-02-10T11:14:48.093 INFO:teuthology.orchestra.run.gibba020.stdout: "addr": { 2021-02-10T11:14:48.093 INFO:teuthology.orchestra.run.gibba020.stdout: "type": "v1", 2021-02-10T11:14:48.093 INFO:teuthology.orchestra.run.gibba020.stdout: "addr": "172.21.2.120:0", 2021-02-10T11:14:48.093 INFO:teuthology.orchestra.run.gibba020.stdout: "nonce": 4186863039 2021-02-10T11:14:48.093 INFO:teuthology.orchestra.run.gibba020.stdout: } 2021-02-10T11:14:48.093 INFO:teuthology.orchestra.run.gibba020.stdout: }, 2021-02-10T11:14:48.093 INFO:teuthology.orchestra.run.gibba020.stdout: "addr": { 2021-02-10T11:14:48.093 INFO:teuthology.orchestra.run.gibba020.stdout: "type": "v1", 2021-02-10T11:14:48.093 INFO:teuthology.orchestra.run.gibba020.stdout: "addr": "172.21.2.120:0", 2021-02-10T11:14:48.093 INFO:teuthology.orchestra.run.gibba020.stdout: "nonce": 4186863039 2021-02-10T11:14:48.093 INFO:teuthology.orchestra.run.gibba020.stdout: }, 2021-02-10T11:14:48.094 INFO:teuthology.orchestra.run.gibba020.stdout: "inst_str": "client.4498 172.21.2.120:0/4186863039", 2021-02-10T11:14:48.094 INFO:teuthology.orchestra.run.gibba020.stdout: "addr_str": "172.21.2.120:0/4186863039", 2021-02-10T11:14:48.094 INFO:teuthology.orchestra.run.gibba020.stdout: "inode_count": 1, 2021-02-10T11:14:48.094 INFO:teuthology.orchestra.run.gibba020.stdout: "mds_epoch": 6, 2021-02-10T11:14:48.094 INFO:teuthology.orchestra.run.gibba020.stdout: "osd_epoch": 20, 2021-02-10T11:14:48.094 INFO:teuthology.orchestra.run.gibba020.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:14:48.094 INFO:teuthology.orchestra.run.gibba020.stdout: "blacklisted": false 2021-02-10T11:14:48.094 INFO:teuthology.orchestra.run.gibba020.stdout:} 2021-02-10T11:14:48.096 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:14:48.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:48.507 7f4245108700 1 Processor -- start 2021-02-10T11:14:48.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:48.507 7f4245108700 1 -- start start 2021-02-10T11:14:48.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:48.508 7f4245108700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f42400e5070 0x7f42401b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:48.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:48.508 7f4245108700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f42401b3e40 0x7f424007d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:48.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:48.508 7f4245108700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f42400e5470 0x7f424007dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:48.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:48.508 7f4245108700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f424007ed80 con 0x7f42400e5070 2021-02-10T11:14:48.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:48.508 7f4245108700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f424007e2d0 con 0x7f42400e5470 2021-02-10T11:14:48.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:48.508 7f4245108700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f424007e4c0 con 0x7f42401b3e40 2021-02-10T11:14:48.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:48.508 7f423ed9d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f42400e5070 0x7f42401b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:48.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:48.508 7f423ed9d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f42400e5070 0x7f42401b3820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:42834/0 (socket says 172.21.2.112:42834) 2021-02-10T11:14:48.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:48.508 7f423ed9d700 1 -- 172.21.2.112:0/2930145825 learned_addr learned my addr 172.21.2.112:0/2930145825 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:14:48.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:48.508 7f423ed9d700 1 -- 172.21.2.112:0/2930145825 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f42401b3e40 msgr2=0x7f424007d690 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:14:48.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:48.508 7f423ed9d700 1 --2- 172.21.2.112:0/2930145825 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f42401b3e40 0x7f424007d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:48.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:48.508 7f423ed9d700 1 -- 172.21.2.112:0/2930145825 >> v2:172.21.2.120:3300/0 conn(0x7f42400e5470 msgr2=0x7f424007dcb0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:14:48.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:48.508 7f423ed9d700 1 --2- 172.21.2.112:0/2930145825 >> v2:172.21.2.120:3300/0 conn(0x7f42400e5470 0x7f424007dcb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:48.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:48.508 7f423ed9d700 1 -- 172.21.2.112:0/2930145825 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f424007e6e0 con 0x7f42400e5070 2021-02-10T11:14:48.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:48.508 7f423ed9d700 1 --2- 172.21.2.112:0/2930145825 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f42400e5070 0x7f42401b3820 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f4230006700 tx=0x7f4230009b80).ready entity=mon.0 client_cookie=f344cea687a6bb4c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:48.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:48.509 7f423dd9b700 1 -- 172.21.2.112:0/2930145825 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f423000c0c0 con 0x7f42400e5070 2021-02-10T11:14:48.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:48.509 7f423dd9b700 1 -- 172.21.2.112:0/2930145825 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f423000c340 con 0x7f42400e5070 2021-02-10T11:14:48.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:48.509 7f423dd9b700 1 -- 172.21.2.112:0/2930145825 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f423000c860 con 0x7f42400e5070 2021-02-10T11:14:48.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:48.509 7f4245108700 1 -- 172.21.2.112:0/2930145825 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f42400e5070 msgr2=0x7f42401b3820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:48.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:48.509 7f4245108700 1 --2- 172.21.2.112:0/2930145825 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f42400e5070 0x7f42401b3820 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f4230006700 tx=0x7f4230009b80).stop 2021-02-10T11:14:48.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:48.509 7f4245108700 1 -- 172.21.2.112:0/2930145825 shutdown_connections 2021-02-10T11:14:48.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:48.509 7f4245108700 1 --2- 172.21.2.112:0/2930145825 >> v2:172.21.2.120:3300/0 conn(0x7f42400e5470 0x7f424007dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:48.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:48.509 7f4245108700 1 --2- 172.21.2.112:0/2930145825 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f42401b3e40 0x7f424007d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:48.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:48.509 7f4245108700 1 --2- 172.21.2.112:0/2930145825 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f42400e5070 0x7f42401b3820 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:48.524 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:48.509 7f4245108700 1 -- 172.21.2.112:0/2930145825 >> 172.21.2.112:0/2930145825 conn(0x7f42400ce690 msgr2=0x7f42400e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:48.524 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:48.509 7f4245108700 1 -- 172.21.2.112:0/2930145825 shutdown_connections 2021-02-10T11:14:48.524 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:48.509 7f4245108700 1 -- 172.21.2.112:0/2930145825 wait complete. 2021-02-10T11:14:48.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:48.524 7f4245108700 1 Processor -- start 2021-02-10T11:14:48.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:48.524 7f4245108700 1 -- start start 2021-02-10T11:14:48.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:48.524 7f4245108700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4240192960 0x7f4240192d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:48.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:48.524 7f4245108700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f42400ad110 0x7f42400ad510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:48.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:48.524 7f4245108700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f42400adb30 0x7f424008e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:48.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:48.524 7f4245108700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42401a03e0 con 0x7f42400ad110 2021-02-10T11:14:48.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:48.524 7f4245108700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4240193380 con 0x7f42400adb30 2021-02-10T11:14:48.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:48.524 7f4245108700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f424008ea80 con 0x7f4240192960 2021-02-10T11:14:48.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:48.524 7f423e59c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f42400ad110 0x7f42400ad510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:48.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:48.524 7f423e59c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f42400ad110 0x7f42400ad510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:42842/0 (socket says 172.21.2.112:42842) 2021-02-10T11:14:48.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:48.524 7f423e59c700 1 -- 172.21.2.112:0/1955800302 learned_addr learned my addr 172.21.2.112:0/1955800302 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:14:48.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:48.525 7f423e59c700 1 -- 172.21.2.112:0/1955800302 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4240192960 msgr2=0x7f4240192d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:48.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:48.525 7f423e59c700 1 --2- 172.21.2.112:0/1955800302 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4240192960 0x7f4240192d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:48.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:48.525 7f423e59c700 1 -- 172.21.2.112:0/1955800302 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f42400adb30 msgr2=0x7f424008e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:48.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:48.525 7f423e59c700 1 --2- 172.21.2.112:0/1955800302 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f42400adb30 0x7f424008e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:48.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:48.525 7f423e59c700 1 -- 172.21.2.112:0/1955800302 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f424008eca0 con 0x7f42400ad110 2021-02-10T11:14:48.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:48.525 7f423e59c700 1 --2- 172.21.2.112:0/1955800302 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f42400ad110 0x7f42400ad510 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f422800b5c0 tx=0x7f422800b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:48.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:48.525 7f42377fe700 1 -- 172.21.2.112:0/1955800302 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f422800bd50 con 0x7f42400ad110 2021-02-10T11:14:48.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:48.525 7f42377fe700 1 -- 172.21.2.112:0/1955800302 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4228006030 con 0x7f42400ad110 2021-02-10T11:14:48.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:48.525 7f42377fe700 1 -- 172.21.2.112:0/1955800302 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f42280054d0 con 0x7f42400ad110 2021-02-10T11:14:48.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:48.525 7f4245108700 1 -- 172.21.2.112:0/1955800302 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f424008eca0 con 0x7f42400ad110 2021-02-10T11:14:48.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:48.525 7f4245108700 1 -- 172.21.2.112:0/1955800302 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f424008eca0 con 0x7f42400ad110 2021-02-10T11:14:48.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:48.526 7f42377fe700 1 -- 172.21.2.112:0/1955800302 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4228006210 con 0x7f42400ad110 2021-02-10T11:14:48.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:48.526 7f42377fe700 1 --2- 172.21.2.112:0/1955800302 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f42240186c0 0x7f422401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:48.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:48.526 7f42377fe700 1 -- 172.21.2.112:0/1955800302 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f42280323f0 con 0x7f42400ad110 2021-02-10T11:14:48.528 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:48.526 7f423ed9d700 1 --2- 172.21.2.112:0/1955800302 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f42240186c0 0x7f422401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:48.528 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:48.527 7f423ed9d700 1 --2- 172.21.2.112:0/1955800302 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f42240186c0 0x7f422401abc0 crc :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:48.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:48.558 7f4245108700 1 -- 172.21.2.112:0/1955800302 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4240107120 con 0x7f42400ad110 2021-02-10T11:14:48.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:48.561 7f42377fe700 1 -- 172.21.2.112:0/1955800302 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4228006b80 con 0x7f42400ad110 2021-02-10T11:14:48.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:48.685 7f4245108700 1 -- 172.21.2.112:0/1955800302 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4240107120 con 0x7f42400ad110 2021-02-10T11:14:48.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:48.685 7f42377fe700 1 -- 172.21.2.112:0/1955800302 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+13993 (secure 0 0 0) 0x7f4228006210 con 0x7f42400ad110 2021-02-10T11:14:48.689 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:14:48.689 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":20,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:14:04.629689","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:14:01.400797","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2021-02-10 11:14:02.622129","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:14:48.690 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:48.687 7f4245108700 1 -- 172.21.2.112:0/1955800302 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f42240186c0 msgr2=0x7f422401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:48.690 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:48.687 7f4245108700 1 --2- 172.21.2.112:0/1955800302 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f42240186c0 0x7f422401abc0 crc :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:48.690 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:48.687 7f4245108700 1 -- 172.21.2.112:0/1955800302 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f42400ad110 msgr2=0x7f42400ad510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:48.690 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:48.687 7f4245108700 1 --2- 172.21.2.112:0/1955800302 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f42400ad110 0x7f42400ad510 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f422800b5c0 tx=0x7f422800b960).stop 2021-02-10T11:14:48.690 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:48.687 7f4245108700 1 -- 172.21.2.112:0/1955800302 shutdown_connections 2021-02-10T11:14:48.690 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:48.687 7f4245108700 1 --2- 172.21.2.112:0/1955800302 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f42240186c0 0x7f422401abc0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:48.690 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:48.687 7f4245108700 1 --2- 172.21.2.112:0/1955800302 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f42400adb30 0x7f424008e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:48.690 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:48.687 7f4245108700 1 --2- 172.21.2.112:0/1955800302 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f42400ad110 0x7f42400ad510 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:48.690 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:48.687 7f4245108700 1 --2- 172.21.2.112:0/1955800302 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4240192960 0x7f4240192d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:48.690 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:48.687 7f4245108700 1 -- 172.21.2.112:0/1955800302 >> 172.21.2.112:0/1955800302 conn(0x7f42400891f0 msgr2=0x7f42401a0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:48.690 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:48.687 7f4245108700 1 -- 172.21.2.112:0/1955800302 shutdown_connections 2021-02-10T11:14:48.690 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:48.687 7f4245108700 1 -- 172.21.2.112:0/1955800302 wait complete. 2021-02-10T11:14:48.694 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:13:57.508758', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:14:01.400797', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:14:02.622129', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:14:48.694 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:14:49.124 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.107 7fc3d53bb700 1 Processor -- start 2021-02-10T11:14:49.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.107 7fc3d53bb700 1 -- start start 2021-02-10T11:14:49.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.108 7fc3d53bb700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fc3d01b0470 0x7fc3d01b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:49.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.108 7fc3d53bb700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fc3d00ef610 0x7fc3d00efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:49.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.108 7fc3d53bb700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc3d00f0030 0x7fc3d00f0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:49.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.108 7fc3d53bb700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3d00e5ea0 con 0x7fc3d00ef610 2021-02-10T11:14:49.126 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.108 7fc3d53bb700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3d00f0b50 con 0x7fc3d01b0470 2021-02-10T11:14:49.126 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.108 7fc3d53bb700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3d00e52e0 con 0x7fc3d00f0030 2021-02-10T11:14:49.126 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.108 7fc3ce87c700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fc3d00ef610 0x7fc3d00efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:49.126 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.108 7fc3ce87c700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fc3d00ef610 0x7fc3d00efa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:42848/0 (socket says 172.21.2.112:42848) 2021-02-10T11:14:49.126 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.108 7fc3ce87c700 1 -- 172.21.2.112:0/3848640836 learned_addr learned my addr 172.21.2.112:0/3848640836 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:14:49.126 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.108 7fc3ce87c700 1 -- 172.21.2.112:0/3848640836 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc3d00f0030 msgr2=0x7fc3d00f0430 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:14:49.126 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.108 7fc3ce87c700 1 --2- 172.21.2.112:0/3848640836 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc3d00f0030 0x7fc3d00f0430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:49.126 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.108 7fc3ce87c700 1 -- 172.21.2.112:0/3848640836 >> v2:172.21.2.120:3300/0 conn(0x7fc3d01b0470 msgr2=0x7fc3d01b0870 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:14:49.126 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.108 7fc3ce87c700 1 --2- 172.21.2.112:0/3848640836 >> v2:172.21.2.120:3300/0 conn(0x7fc3d01b0470 0x7fc3d01b0870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:49.127 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.108 7fc3ce87c700 1 -- 172.21.2.112:0/3848640836 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3d00f10d0 con 0x7fc3d00ef610 2021-02-10T11:14:49.127 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.109 7fc3ce87c700 1 --2- 172.21.2.112:0/3848640836 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc3d00ef610 0x7fc3d00efa10 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7fc3c0006700 tx=0x7fc3c0009b80).ready entity=mon.0 client_cookie=1dba3b28641b335a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:49.127 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.109 7fc3ce07b700 1 -- 172.21.2.112:0/3848640836 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc3c000c0c0 con 0x7fc3d00ef610 2021-02-10T11:14:49.127 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.109 7fc3ce07b700 1 -- 172.21.2.112:0/3848640836 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3c000c440 con 0x7fc3d00ef610 2021-02-10T11:14:49.127 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.109 7fc3ce07b700 1 -- 172.21.2.112:0/3848640836 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc3c000c960 con 0x7fc3d00ef610 2021-02-10T11:14:49.127 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.109 7fc3d53bb700 1 -- 172.21.2.112:0/3848640836 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc3d00ef610 msgr2=0x7fc3d00efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:49.127 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.109 7fc3d53bb700 1 --2- 172.21.2.112:0/3848640836 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc3d00ef610 0x7fc3d00efa10 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7fc3c0006700 tx=0x7fc3c0009b80).stop 2021-02-10T11:14:49.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.109 7fc3d53bb700 1 -- 172.21.2.112:0/3848640836 shutdown_connections 2021-02-10T11:14:49.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.109 7fc3d53bb700 1 --2- 172.21.2.112:0/3848640836 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc3d00f0030 0x7fc3d00f0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:49.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.109 7fc3d53bb700 1 --2- 172.21.2.112:0/3848640836 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc3d00ef610 0x7fc3d00efa10 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:49.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.109 7fc3d53bb700 1 --2- 172.21.2.112:0/3848640836 >> v2:172.21.2.120:3300/0 conn(0x7fc3d01b0470 0x7fc3d01b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:49.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.109 7fc3d53bb700 1 -- 172.21.2.112:0/3848640836 >> 172.21.2.112:0/3848640836 conn(0x7fc3d00ce3e0 msgr2=0x7fc3d00e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:49.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.109 7fc3d53bb700 1 -- 172.21.2.112:0/3848640836 shutdown_connections 2021-02-10T11:14:49.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.110 7fc3d53bb700 1 -- 172.21.2.112:0/3848640836 wait complete. 2021-02-10T11:14:49.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.124 7fc3d53bb700 1 Processor -- start 2021-02-10T11:14:49.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.124 7fc3d53bb700 1 -- start start 2021-02-10T11:14:49.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.124 7fc3d53bb700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc3d00b20b0 0x7fc3d00ac970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:49.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.124 7fc3d53bb700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc3d00b24b0 0x7fc3d0191dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:49.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.124 7fc3d53bb700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc3d00acf90 0x7fc3d01923e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:49.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.124 7fc3d53bb700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3d00b1b30 con 0x7fc3d00b24b0 2021-02-10T11:14:49.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.124 7fc3d53bb700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3d01a04e0 con 0x7fc3d00b20b0 2021-02-10T11:14:49.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.124 7fc3d53bb700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3d01a06c0 con 0x7fc3d00acf90 2021-02-10T11:14:49.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.125 7fc3ce87c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc3d00b24b0 0x7fc3d0191dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:49.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.125 7fc3ce87c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc3d00b24b0 0x7fc3d0191dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:42854/0 (socket says 172.21.2.112:42854) 2021-02-10T11:14:49.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.125 7fc3ce87c700 1 -- 172.21.2.112:0/3423463840 learned_addr learned my addr 172.21.2.112:0/3423463840 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:14:49.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.125 7fc3ce87c700 1 -- 172.21.2.112:0/3423463840 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc3d00acf90 msgr2=0x7fc3d01923e0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:14:49.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.125 7fc3ce87c700 1 --2- 172.21.2.112:0/3423463840 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc3d00acf90 0x7fc3d01923e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:49.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.125 7fc3ce87c700 1 -- 172.21.2.112:0/3423463840 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc3d00b20b0 msgr2=0x7fc3d00ac970 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:14:49.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.125 7fc3ce87c700 1 --2- 172.21.2.112:0/3423463840 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc3d00b20b0 0x7fc3d00ac970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:49.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.125 7fc3ce87c700 1 -- 172.21.2.112:0/3423463840 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3d0192be0 con 0x7fc3d00b24b0 2021-02-10T11:14:49.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.125 7fc3ce87c700 1 --2- 172.21.2.112:0/3423463840 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc3d00b24b0 0x7fc3d0191dc0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7fc3c000b5e0 tx=0x7fc3c000b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:49.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.125 7fc3c7fff700 1 -- 172.21.2.112:0/3423463840 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc3c000bd70 con 0x7fc3d00b24b0 2021-02-10T11:14:49.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.125 7fc3c7fff700 1 -- 172.21.2.112:0/3423463840 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3c0002030 con 0x7fc3d00b24b0 2021-02-10T11:14:49.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.125 7fc3c7fff700 1 -- 172.21.2.112:0/3423463840 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc3c0002550 con 0x7fc3d00b24b0 2021-02-10T11:14:49.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.125 7fc3d53bb700 1 -- 172.21.2.112:0/3423463840 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3d0192be0 con 0x7fc3d00b24b0 2021-02-10T11:14:49.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.125 7fc3d53bb700 1 -- 172.21.2.112:0/3423463840 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3d0193170 con 0x7fc3d00b24b0 2021-02-10T11:14:49.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.126 7fc3c7fff700 1 -- 172.21.2.112:0/3423463840 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc3c0000c90 con 0x7fc3d00b24b0 2021-02-10T11:14:49.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.126 7fc3c7fff700 1 --2- 172.21.2.112:0/3423463840 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc3b40186c0 0x7fc3b401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:49.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.126 7fc3c7fff700 1 -- 172.21.2.112:0/3423463840 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fc3c00325a0 con 0x7fc3d00b24b0 2021-02-10T11:14:49.131 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.126 7fc3cf07d700 1 --2- 172.21.2.112:0/3423463840 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc3b40186c0 0x7fc3b401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:49.131 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.127 7fc3cf07d700 1 --2- 172.21.2.112:0/3423463840 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc3b40186c0 0x7fc3b401abc0 crc :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:49.156 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.158 7fc3d53bb700 1 -- 172.21.2.112:0/3423463840 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3d0105af0 con 0x7fc3d00b24b0 2021-02-10T11:14:49.159 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.160 7fc3c7fff700 1 -- 172.21.2.112:0/3423463840 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc3c00009b0 con 0x7fc3d00b24b0 2021-02-10T11:14:49.284 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.286 7fc3d53bb700 1 -- 172.21.2.112:0/3423463840 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7fc3d00f0b50 con 0x7fc3d00b24b0 2021-02-10T11:14:49.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.286 7fc3c7fff700 1 -- 172.21.2.112:0/3423463840 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v20) v1 ==== 92+0+10 (secure 0 0 0) 0x7fc3c0000c90 con 0x7fc3d00b24b0 2021-02-10T11:14:49.285 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:14:49.287 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.288 7fc3d53bb700 1 -- 172.21.2.112:0/3423463840 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc3b40186c0 msgr2=0x7fc3b401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:49.287 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.288 7fc3d53bb700 1 --2- 172.21.2.112:0/3423463840 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc3b40186c0 0x7fc3b401abc0 crc :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:49.287 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.288 7fc3d53bb700 1 -- 172.21.2.112:0/3423463840 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc3d00b24b0 msgr2=0x7fc3d0191dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:49.287 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.288 7fc3d53bb700 1 --2- 172.21.2.112:0/3423463840 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc3d00b24b0 0x7fc3d0191dc0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7fc3c000b5e0 tx=0x7fc3c000b980).stop 2021-02-10T11:14:49.287 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.288 7fc3d53bb700 1 -- 172.21.2.112:0/3423463840 shutdown_connections 2021-02-10T11:14:49.288 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.288 7fc3d53bb700 1 --2- 172.21.2.112:0/3423463840 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc3b40186c0 0x7fc3b401abc0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:49.288 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.288 7fc3d53bb700 1 --2- 172.21.2.112:0/3423463840 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc3d00acf90 0x7fc3d01923e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:49.288 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.288 7fc3d53bb700 1 --2- 172.21.2.112:0/3423463840 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc3d00b24b0 0x7fc3d0191dc0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:49.288 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.288 7fc3d53bb700 1 --2- 172.21.2.112:0/3423463840 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc3d00b20b0 0x7fc3d00ac970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:49.288 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.288 7fc3d53bb700 1 -- 172.21.2.112:0/3423463840 >> 172.21.2.112:0/3423463840 conn(0x7fc3d00bf4e0 msgr2=0x7fc3d00be3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:49.288 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.289 7fc3d53bb700 1 -- 172.21.2.112:0/3423463840 shutdown_connections 2021-02-10T11:14:49.288 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.289 7fc3d53bb700 1 -- 172.21.2.112:0/3423463840 wait complete. 2021-02-10T11:14:49.295 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:14:49.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.709 7f78bd1e4700 1 Processor -- start 2021-02-10T11:14:49.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.709 7f78bd1e4700 1 -- start start 2021-02-10T11:14:49.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.709 7f78bd1e4700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f78b81b0470 0x7f78b81b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:49.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.709 7f78bd1e4700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f78b80f20e0 0x7f78b80f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:49.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.709 7f78bd1e4700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78b80f0b50 con 0x7f78b80ef610 2021-02-10T11:14:49.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.709 7f78bd1e4700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78b80f0820 con 0x7f78b81b0470 2021-02-10T11:14:49.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.709 7f78bd1e4700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78b80f24e0 con 0x7f78b80f20e0 2021-02-10T11:14:49.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.711 7f78b659c700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7f78b80ef610 0x7f78b80efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:35878/0 (socket says 172.21.2.112:35878) 2021-02-10T11:14:49.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.711 7f78b6d9d700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f78b81b0470 0x7f78b81b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:49.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.711 7f78b659c700 1 -- 172.21.2.112:0/3101988808 learned_addr learned my addr 172.21.2.112:0/3101988808 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:14:49.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.711 7f78b759e700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f78b80f20e0 0x7f78b80f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:49.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.711 7f78b759e700 1 -- 172.21.2.112:0/3101988808 >> v2:172.21.2.120:3300/0 conn(0x7f78b81b0470 msgr2=0x7f78b81b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:14:49.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.711 7f78b759e700 1 --2- 172.21.2.112:0/3101988808 >> v2:172.21.2.120:3300/0 conn(0x7f78b81b0470 0x7f78b81b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:49.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.711 7f78b759e700 1 -- 172.21.2.112:0/3101988808 >> v1:172.21.2.120:6789/0 conn(0x7f78b80ef610 legacy=0x7f78b80efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:49.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.711 7f78b759e700 1 -- 172.21.2.112:0/3101988808 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78b80e52e0 con 0x7f78b80f20e0 2021-02-10T11:14:49.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.711 7f78b6d9d700 1 --2- 172.21.2.112:0/3101988808 >> v2:172.21.2.120:3300/0 conn(0x7f78b81b0470 0x7f78b81b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:14:49.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.711 7f78b759e700 1 --2- 172.21.2.112:0/3101988808 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f78b80f20e0 0x7f78b80f0200 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f78b0006ad0 tx=0x7f78b0009ad0).ready entity=mon.2 client_cookie=174b4df244a7f925 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:49.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.712 7f78b5d9b700 1 -- 172.21.2.112:0/3101988808 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f78b0000c10 con 0x7f78b80f20e0 2021-02-10T11:14:49.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.712 7f78b5d9b700 1 -- 172.21.2.112:0/3101988808 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78b0006370 con 0x7f78b80f20e0 2021-02-10T11:14:49.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.712 7f78b5d9b700 1 -- 172.21.2.112:0/3101988808 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f78b00024d0 con 0x7f78b80f20e0 2021-02-10T11:14:49.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.712 7f78bd1e4700 1 -- 172.21.2.112:0/3101988808 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f78b80f20e0 msgr2=0x7f78b80f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:49.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.712 7f78bd1e4700 1 --2- 172.21.2.112:0/3101988808 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f78b80f20e0 0x7f78b80f0200 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f78b0006ad0 tx=0x7f78b0009ad0).stop 2021-02-10T11:14:49.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.713 7f78bd1e4700 1 -- 172.21.2.112:0/3101988808 shutdown_connections 2021-02-10T11:14:49.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.713 7f78bd1e4700 1 --2- 172.21.2.112:0/3101988808 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f78b80f20e0 0x7f78b80f0200 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:49.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.713 7f78bd1e4700 1 --2- 172.21.2.112:0/3101988808 >> v2:172.21.2.120:3300/0 conn(0x7f78b81b0470 0x7f78b81b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:49.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.713 7f78bd1e4700 1 -- 172.21.2.112:0/3101988808 >> 172.21.2.112:0/3101988808 conn(0x7f78b80ce3e0 msgr2=0x7f78b80e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:49.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.713 7f78bd1e4700 1 -- 172.21.2.112:0/3101988808 shutdown_connections 2021-02-10T11:14:49.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.713 7f78bd1e4700 1 -- 172.21.2.112:0/3101988808 wait complete. 2021-02-10T11:14:49.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.727 7f78bd1e4700 1 Processor -- start 2021-02-10T11:14:49.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.727 7f78bd1e4700 1 -- start start 2021-02-10T11:14:49.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.728 7f78bd1e4700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f78b80a0420 0x7f78b80a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:49.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.728 7f78bd1e4700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f78b8189d00 0x7f78b818a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:49.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.728 7f78bd1e4700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f78b818a720 0x7f78b80b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:49.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.728 7f78bd1e4700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78b8189780 con 0x7f78b8189d00 2021-02-10T11:14:49.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.728 7f78bd1e4700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78b818ab20 con 0x7f78b80a0420 2021-02-10T11:14:49.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.728 7f78bd1e4700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78b80a1140 con 0x7f78b818a720 2021-02-10T11:14:49.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.728 7f78b759e700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f78b818a720 0x7f78b80b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:49.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.728 7f78b659c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f78b8189d00 0x7f78b818a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:49.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.728 7f78b659c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f78b8189d00 0x7f78b818a100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:42866/0 (socket says 172.21.2.112:42866) 2021-02-10T11:14:49.735 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.728 7f78b759e700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f78b818a720 0x7f78b80b7be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3301/0 says I am v2:172.21.2.112:42550/0 (socket says 172.21.2.112:42550) 2021-02-10T11:14:49.735 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.728 7f78b659c700 1 -- 172.21.2.112:0/816359972 learned_addr learned my addr 172.21.2.112:0/816359972 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:14:49.735 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.728 7f78b6d9d700 1 --2- 172.21.2.112:0/816359972 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f78b80a0420 0x7f78b80a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:49.735 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.729 7f78b659c700 1 -- 172.21.2.112:0/816359972 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f78b818a720 msgr2=0x7f78b80b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:49.735 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.729 7f78b659c700 1 --2- 172.21.2.112:0/816359972 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f78b818a720 0x7f78b80b7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:49.735 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.729 7f78b659c700 1 -- 172.21.2.112:0/816359972 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f78b80a0420 msgr2=0x7f78b80a0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:49.735 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.729 7f78b659c700 1 --2- 172.21.2.112:0/816359972 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f78b80a0420 0x7f78b80a0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:49.736 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.729 7f78b659c700 1 -- 172.21.2.112:0/816359972 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78b80b8200 con 0x7f78b8189d00 2021-02-10T11:14:49.736 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.729 7f78b6d9d700 1 --2- 172.21.2.112:0/816359972 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f78b80a0420 0x7f78b80a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:14:49.736 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.729 7f78b659c700 1 --2- 172.21.2.112:0/816359972 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f78b8189d00 0x7f78b818a100 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f78a0007f90 tx=0x7f78a000aca0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:49.736 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.729 7f78af7fe700 1 -- 172.21.2.112:0/816359972 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f78a0007020 con 0x7f78b8189d00 2021-02-10T11:14:49.736 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.729 7f78af7fe700 1 -- 172.21.2.112:0/816359972 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78a0001280 con 0x7f78b8189d00 2021-02-10T11:14:49.736 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.729 7f78bd1e4700 1 -- 172.21.2.112:0/816359972 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78b80b8200 con 0x7f78b8189d00 2021-02-10T11:14:49.736 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.729 7f78af7fe700 1 -- 172.21.2.112:0/816359972 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f78a00048b0 con 0x7f78b8189d00 2021-02-10T11:14:49.736 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.729 7f78bd1e4700 1 -- 172.21.2.112:0/816359972 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78b80b8790 con 0x7f78b8189d00 2021-02-10T11:14:49.736 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.729 7f78af7fe700 1 -- 172.21.2.112:0/816359972 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f78a000ade0 con 0x7f78b8189d00 2021-02-10T11:14:49.736 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.730 7f78af7fe700 1 --2- 172.21.2.112:0/816359972 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f789c018730 0x7f789c01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:49.736 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.730 7f78af7fe700 1 -- 172.21.2.112:0/816359972 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f78a00329c0 con 0x7f78b8189d00 2021-02-10T11:14:49.737 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.730 7f78b6d9d700 1 --2- 172.21.2.112:0/816359972 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f789c018730 0x7f789c01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:49.737 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.730 7f78b6d9d700 1 --2- 172.21.2.112:0/816359972 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f789c018730 0x7f789c01ac10 crc :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:49.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.759 7f78bd1e4700 1 -- 172.21.2.112:0/816359972 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78b8105ae0 con 0x7f78b8189d00 2021-02-10T11:14:49.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.762 7f78af7fe700 1 -- 172.21.2.112:0/816359972 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f78a0007100 con 0x7f78b8189d00 2021-02-10T11:14:49.892 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.894 7f78bd1e4700 1 -- 172.21.2.112:0/816359972 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f78b80f0b50 con 0x7f78b8189d00 2021-02-10T11:14:49.893 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.894 7f78af7fe700 1 -- 172.21.2.112:0/816359972 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v20) v1 ==== 104+0+10 (secure 0 0 0) 0x7f78a0007310 con 0x7f78b8189d00 2021-02-10T11:14:49.893 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:14:49.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.896 7f78bd1e4700 1 -- 172.21.2.112:0/816359972 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f789c018730 msgr2=0x7f789c01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:49.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.896 7f78bd1e4700 1 --2- 172.21.2.112:0/816359972 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f789c018730 0x7f789c01ac10 crc :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:49.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.896 7f78bd1e4700 1 -- 172.21.2.112:0/816359972 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f78b8189d00 msgr2=0x7f78b818a100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:49.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.896 7f78bd1e4700 1 --2- 172.21.2.112:0/816359972 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f78b8189d00 0x7f78b818a100 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f78a0007f90 tx=0x7f78a000aca0).stop 2021-02-10T11:14:49.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.897 7f78bd1e4700 1 -- 172.21.2.112:0/816359972 shutdown_connections 2021-02-10T11:14:49.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.897 7f78bd1e4700 1 --2- 172.21.2.112:0/816359972 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f789c018730 0x7f789c01ac10 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:49.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.897 7f78bd1e4700 1 --2- 172.21.2.112:0/816359972 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f78b818a720 0x7f78b80b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:49.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.897 7f78bd1e4700 1 --2- 172.21.2.112:0/816359972 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f78b8189d00 0x7f78b818a100 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:49.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.897 7f78bd1e4700 1 --2- 172.21.2.112:0/816359972 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f78b80a0420 0x7f78b80a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:49.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.897 7f78bd1e4700 1 -- 172.21.2.112:0/816359972 >> 172.21.2.112:0/816359972 conn(0x7f78b8094e10 msgr2=0x7f78b80a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:49.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.897 7f78bd1e4700 1 -- 172.21.2.112:0/816359972 shutdown_connections 2021-02-10T11:14:49.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:49.897 7f78bd1e4700 1 -- 172.21.2.112:0/816359972 wait complete. 2021-02-10T11:14:49.903 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:14:50.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.317 7f6b6c626700 1 Processor -- start 2021-02-10T11:14:50.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.317 7f6b6c626700 1 -- start start 2021-02-10T11:14:50.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.317 7f6b6c626700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f6b641b0470 0x7f6b641b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:50.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.317 7f6b6c626700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b640f0b50 con 0x7f6b640ef610 2021-02-10T11:14:50.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.317 7f6b6c626700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b640f24e0 con 0x7f6b640f20e0 2021-02-10T11:14:50.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.317 7f6b6c626700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b640e5ea0 con 0x7f6b641b0470 2021-02-10T11:14:50.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.318 7f6b69c41700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f6b640ef610 0x7f6b640efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:42334/0 (socket says 172.21.2.112:42334) 2021-02-10T11:14:50.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.318 7f6b69c41700 1 -- 172.21.2.112:0/3149398298 learned_addr learned my addr 172.21.2.112:0/3149398298 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:14:50.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.318 7f6b69440700 1 -- 172.21.2.112:0/3149398298 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3967555705 0 0) 0x7f6b580010b0 con 0x7f6b640ef610 2021-02-10T11:14:50.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.318 7f6b69440700 1 -- 172.21.2.112:0/3149398298 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b50002fe0 con 0x7f6b640ef610 2021-02-10T11:14:50.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.318 7f6b6a442700 1 --2- 172.21.2.112:0/3149398298 >> v2:172.21.2.120:3300/0 conn(0x7f6b641b0470 0x7f6b641b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:50.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.318 7f6b69440700 1 -- 172.21.2.112:0/3149398298 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4038903699 0 0) 0x7f6b580010b0 con 0x7f6b640ef610 2021-02-10T11:14:50.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.318 7f6b69440700 1 -- 172.21.2.112:0/3149398298 >> v2:172.21.2.120:3300/0 conn(0x7f6b641b0470 msgr2=0x7f6b641b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:14:50.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.318 7f6b69440700 1 --2- 172.21.2.112:0/3149398298 >> v2:172.21.2.120:3300/0 conn(0x7f6b641b0470 0x7f6b641b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:50.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.318 7f6b69440700 1 -- 172.21.2.112:0/3149398298 >> v1:172.21.2.120:6789/0 conn(0x7f6b640f20e0 legacy=0x7f6b640f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:14:50.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.318 7f6b69440700 1 -- 172.21.2.112:0/3149398298 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b640e52e0 con 0x7f6b640ef610 2021-02-10T11:14:50.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.318 7f6b69440700 1 -- 172.21.2.112:0/3149398298 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f6b580015b0 con 0x7f6b640ef610 2021-02-10T11:14:50.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.318 7f6b69440700 1 -- 172.21.2.112:0/3149398298 >> v1:172.21.2.112:6789/0 conn(0x7f6b640ef610 legacy=0x7f6b640efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:50.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.318 7f6b69440700 1 --2- 172.21.2.112:0/3149398298 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6b50003ae0 0x7f6b50005f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:50.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.318 7f6b69440700 1 --2- 172.21.2.112:0/3149398298 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6b50006810 0x7f6b50008d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:50.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.318 7f6b69440700 1 --2- 172.21.2.112:0/3149398298 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6b50009630 0x7f6b5000bb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:50.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.318 7f6b69440700 1 -- 172.21.2.112:0/3149398298 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b5000c3a0 con 0x7f6b50009630 2021-02-10T11:14:50.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.318 7f6b69440700 1 -- 172.21.2.112:0/3149398298 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b5000c5c0 con 0x7f6b50006810 2021-02-10T11:14:50.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.318 7f6b69440700 1 -- 172.21.2.112:0/3149398298 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b5000c7e0 con 0x7f6b50003ae0 2021-02-10T11:14:50.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.318 7f6b6a442700 1 --2- 172.21.2.112:0/3149398298 >> v2:172.21.2.120:3300/0 conn(0x7f6b641b0470 0x7f6b641b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:14:50.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.319 7f6b6ac43700 1 --2- 172.21.2.112:0/3149398298 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6b50009630 0x7f6b5000bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:50.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.319 7f6b69c41700 1 --2- 172.21.2.112:0/3149398298 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6b50006810 0x7f6b50008d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:50.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.319 7f6b6a442700 1 --2- 172.21.2.112:0/3149398298 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6b50003ae0 0x7f6b50005f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:50.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.319 7f6b6ac43700 1 -- 172.21.2.112:0/3149398298 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6b50003ae0 msgr2=0x7f6b50005f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:14:50.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.319 7f6b6ac43700 1 --2- 172.21.2.112:0/3149398298 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6b50003ae0 0x7f6b50005f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:50.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.319 7f6b6ac43700 1 -- 172.21.2.112:0/3149398298 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6b50006810 msgr2=0x7f6b50008d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:14:50.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.319 7f6b6ac43700 1 --2- 172.21.2.112:0/3149398298 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6b50006810 0x7f6b50008d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:50.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.319 7f6b6ac43700 1 -- 172.21.2.112:0/3149398298 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b5000ca80 con 0x7f6b50009630 2021-02-10T11:14:50.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.319 7f6b6ac43700 1 -- 172.21.2.112:0/3149398298 reap_dead start 2021-02-10T11:14:50.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.319 7f6b6ac43700 1 --2- 172.21.2.112:0/3149398298 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6b50009630 0x7f6b5000bb50 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f6b5c04ae40 tx=0x7f6b5c04dda0).ready entity=mon.0 client_cookie=f78dd08fa0bb880b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:50.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.319 7f6b69440700 1 -- 172.21.2.112:0/3149398298 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6b5c001400 con 0x7f6b50009630 2021-02-10T11:14:50.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.319 7f6b69440700 1 -- 172.21.2.112:0/3149398298 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b5c003210 con 0x7f6b50009630 2021-02-10T11:14:50.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.319 7f6b69440700 1 -- 172.21.2.112:0/3149398298 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6b5c001780 con 0x7f6b50009630 2021-02-10T11:14:50.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.319 7f6b6c626700 1 -- 172.21.2.112:0/3149398298 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6b50009630 msgr2=0x7f6b5000bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:50.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.319 7f6b6c626700 1 --2- 172.21.2.112:0/3149398298 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6b50009630 0x7f6b5000bb50 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f6b5c04ae40 tx=0x7f6b5c04dda0).stop 2021-02-10T11:14:50.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.320 7f6b6c626700 1 -- 172.21.2.112:0/3149398298 shutdown_connections 2021-02-10T11:14:50.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.320 7f6b6c626700 1 --2- 172.21.2.112:0/3149398298 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6b50009630 0x7f6b5000bb50 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:50.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.320 7f6b6c626700 1 -- 172.21.2.112:0/3149398298 >> 172.21.2.112:0/3149398298 conn(0x7f6b640ce3e0 msgr2=0x7f6b640e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:50.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.320 7f6b6c626700 1 -- 172.21.2.112:0/3149398298 shutdown_connections 2021-02-10T11:14:50.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.320 7f6b6c626700 1 -- 172.21.2.112:0/3149398298 wait complete. 2021-02-10T11:14:50.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.335 7f6b6c626700 1 Processor -- start 2021-02-10T11:14:50.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.335 7f6b6c626700 1 -- start start 2021-02-10T11:14:50.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.335 7f6b6c626700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6b640b7db0 0x7f6b640b8260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:50.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.335 7f6b6c626700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6b640b8880 0x7f6b641a7450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:50.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.335 7f6b6c626700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6b640b8c80 0x7f6b641a7a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:50.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.335 7f6b6c626700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b640b79b0 con 0x7f6b640b7db0 2021-02-10T11:14:50.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.335 7f6b6c626700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b641a8090 con 0x7f6b640b8880 2021-02-10T11:14:50.339 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.335 7f6b6c626700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b641a82b0 con 0x7f6b640b8c80 2021-02-10T11:14:50.339 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.335 7f6b6a442700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6b640b7db0 0x7f6b640b8260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:50.339 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.335 7f6b6a442700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6b640b7db0 0x7f6b640b8260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:42884/0 (socket says 172.21.2.112:42884) 2021-02-10T11:14:50.339 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.335 7f6b6a442700 1 -- 172.21.2.112:0/3561296661 learned_addr learned my addr 172.21.2.112:0/3561296661 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:14:50.339 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.336 7f6b6a442700 1 -- 172.21.2.112:0/3561296661 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6b640b8c80 msgr2=0x7f6b641a7a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:50.339 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.336 7f6b6a442700 1 --2- 172.21.2.112:0/3561296661 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6b640b8c80 0x7f6b641a7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:50.339 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.336 7f6b6a442700 1 -- 172.21.2.112:0/3561296661 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6b640b8880 msgr2=0x7f6b641a7450 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:14:50.339 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.336 7f6b6a442700 1 --2- 172.21.2.112:0/3561296661 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6b640b8880 0x7f6b641a7450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:50.339 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.336 7f6b6a442700 1 -- 172.21.2.112:0/3561296661 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b64190c10 con 0x7f6b640b7db0 2021-02-10T11:14:50.339 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.336 7f6b6a442700 1 --2- 172.21.2.112:0/3561296661 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6b640b7db0 0x7f6b640b8260 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f6b60006f70 tx=0x7f6b6000b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:50.339 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.336 7f6b56ffd700 1 -- 172.21.2.112:0/3561296661 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6b6000bd50 con 0x7f6b640b7db0 2021-02-10T11:14:50.339 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.336 7f6b56ffd700 1 -- 172.21.2.112:0/3561296661 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b60006030 con 0x7f6b640b7db0 2021-02-10T11:14:50.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.336 7f6b56ffd700 1 -- 172.21.2.112:0/3561296661 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6b60005560 con 0x7f6b640b7db0 2021-02-10T11:14:50.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.336 7f6b6c626700 1 -- 172.21.2.112:0/3561296661 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b64190c10 con 0x7f6b640b7db0 2021-02-10T11:14:50.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.336 7f6b6c626700 1 -- 172.21.2.112:0/3561296661 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b640c1eb0 con 0x7f6b640b7db0 2021-02-10T11:14:50.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.337 7f6b56ffd700 1 -- 172.21.2.112:0/3561296661 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6b60006030 con 0x7f6b640b7db0 2021-02-10T11:14:50.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.337 7f6b56ffd700 1 --2- 172.21.2.112:0/3561296661 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6b4c0186c0 0x7f6b4c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:50.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.337 7f6b56ffd700 1 -- 172.21.2.112:0/3561296661 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f6b60032530 con 0x7f6b640b7db0 2021-02-10T11:14:50.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.337 7f6b69c41700 1 --2- 172.21.2.112:0/3561296661 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6b4c0186c0 0x7f6b4c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:50.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.338 7f6b69c41700 1 --2- 172.21.2.112:0/3561296661 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6b4c0186c0 0x7f6b4c01abc0 crc :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:50.366 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.367 7f6b6c626700 1 -- 172.21.2.112:0/3561296661 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b64105b30 con 0x7f6b640b7db0 2021-02-10T11:14:50.368 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.370 7f6b56ffd700 1 -- 172.21.2.112:0/3561296661 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6b60030ce0 con 0x7f6b640b7db0 2021-02-10T11:14:50.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.498 7f6b6c626700 1 -- 172.21.2.112:0/3561296661 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f6b64105b30 con 0x7f6b640b7db0 2021-02-10T11:14:50.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.498 7f6b56ffd700 1 -- 172.21.2.112:0/3561296661 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v20) v1 ==== 100+0+10 (secure 0 0 0) 0x7f6b6000bd50 con 0x7f6b640b7db0 2021-02-10T11:14:50.497 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:14:50.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.500 7f6b6c626700 1 -- 172.21.2.112:0/3561296661 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6b4c0186c0 msgr2=0x7f6b4c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:50.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.500 7f6b6c626700 1 --2- 172.21.2.112:0/3561296661 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6b4c0186c0 0x7f6b4c01abc0 crc :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:50.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.500 7f6b6c626700 1 -- 172.21.2.112:0/3561296661 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6b640b7db0 msgr2=0x7f6b640b8260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:50.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.500 7f6b6c626700 1 --2- 172.21.2.112:0/3561296661 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6b640b7db0 0x7f6b640b8260 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f6b60006f70 tx=0x7f6b6000b960).stop 2021-02-10T11:14:50.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.500 7f6b6c626700 1 -- 172.21.2.112:0/3561296661 shutdown_connections 2021-02-10T11:14:50.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.500 7f6b6c626700 1 --2- 172.21.2.112:0/3561296661 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6b4c0186c0 0x7f6b4c01abc0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:50.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.500 7f6b6c626700 1 --2- 172.21.2.112:0/3561296661 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6b640b8880 0x7f6b641a7450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:50.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.500 7f6b6c626700 1 --2- 172.21.2.112:0/3561296661 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6b640b8c80 0x7f6b641a7a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:50.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.500 7f6b6c626700 1 --2- 172.21.2.112:0/3561296661 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6b640b7db0 0x7f6b640b8260 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:50.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.500 7f6b6c626700 1 -- 172.21.2.112:0/3561296661 >> 172.21.2.112:0/3561296661 conn(0x7f6b64094e10 msgr2=0x7f6b640a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:50.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.500 7f6b6c626700 1 -- 172.21.2.112:0/3561296661 shutdown_connections 2021-02-10T11:14:50.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.500 7f6b6c626700 1 -- 172.21.2.112:0/3561296661 wait complete. 2021-02-10T11:14:50.507 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:14:50.507 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:14:50.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.917 7f7245540700 1 Processor -- start 2021-02-10T11:14:50.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.917 7f7245540700 1 -- start start 2021-02-10T11:14:50.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.918 7f7245540700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f72400cff00 0x7f72401b39d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:50.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.918 7f7245540700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f72400d1bf0 0x7f72400ef620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:50.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.918 7f7245540700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72400f0b20 con 0x7f72400d17f0 2021-02-10T11:14:50.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.918 7f7245540700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72401b3ff0 con 0x7f72400cff00 2021-02-10T11:14:50.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.918 7f7245540700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72400f20b0 con 0x7f72400d1bf0 2021-02-10T11:14:50.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.918 7f723eabe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f72400cff00 0x7f72401b39d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:50.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.918 7f723eabe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f72400cff00 0x7f72401b39d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:42892/0 (socket says 172.21.2.112:42892) 2021-02-10T11:14:50.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.918 7f723eabe700 1 -- 172.21.2.112:0/1633088016 learned_addr learned my addr 172.21.2.112:0/1633088016 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:14:50.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.918 7f723eabe700 1 -- 172.21.2.112:0/1633088016 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f72400d1bf0 msgr2=0x7f72400ef620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:14:50.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.918 7f723eabe700 1 --2- 172.21.2.112:0/1633088016 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f72400d1bf0 0x7f72400ef620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:50.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.918 7f723eabe700 1 -- 172.21.2.112:0/1633088016 >> v1:172.21.2.120:6789/0 conn(0x7f72400d17f0 legacy=0x7f72401b31e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:14:50.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.918 7f723eabe700 1 -- 172.21.2.112:0/1633088016 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72400f22d0 con 0x7f72400cff00 2021-02-10T11:14:50.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.918 7f723eabe700 1 --2- 172.21.2.112:0/1633088016 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f72400cff00 0x7f72401b39d0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f7230006700 tx=0x7f7230009b80).ready entity=mon.0 client_cookie=80529f9b5f90c2f9 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:50.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.919 7f723e2bd700 1 -- 172.21.2.112:0/1633088016 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f723000c0c0 con 0x7f72400cff00 2021-02-10T11:14:50.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.919 7f723e2bd700 1 -- 172.21.2.112:0/1633088016 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f723000c400 con 0x7f72400cff00 2021-02-10T11:14:50.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.919 7f723e2bd700 1 -- 172.21.2.112:0/1633088016 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f723000c860 con 0x7f72400cff00 2021-02-10T11:14:50.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.919 7f7245540700 1 -- 172.21.2.112:0/1633088016 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f72400cff00 msgr2=0x7f72401b39d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:50.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.919 7f7245540700 1 --2- 172.21.2.112:0/1633088016 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f72400cff00 0x7f72401b39d0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f7230006700 tx=0x7f7230009b80).stop 2021-02-10T11:14:50.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.919 7f7245540700 1 -- 172.21.2.112:0/1633088016 shutdown_connections 2021-02-10T11:14:50.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.919 7f7245540700 1 --2- 172.21.2.112:0/1633088016 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f72400cff00 0x7f72401b39d0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:50.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.919 7f7245540700 1 --2- 172.21.2.112:0/1633088016 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f72400d1bf0 0x7f72400ef620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:50.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.919 7f7245540700 1 -- 172.21.2.112:0/1633088016 >> 172.21.2.112:0/1633088016 conn(0x7f72400fde40 msgr2=0x7f72400ce390 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:50.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.919 7f7245540700 1 -- 172.21.2.112:0/1633088016 shutdown_connections 2021-02-10T11:14:50.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.919 7f7245540700 1 -- 172.21.2.112:0/1633088016 wait complete. 2021-02-10T11:14:50.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.934 7f7245540700 1 Processor -- start 2021-02-10T11:14:50.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.934 7f7245540700 1 -- start start 2021-02-10T11:14:50.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.934 7f7245540700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7240093110 0x7f7240189e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:50.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.934 7f7245540700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f724018a470 0x7f724018a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:50.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.934 7f7245540700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f724018ae90 0x7f724018dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:50.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.934 7f7245540700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72401898d0 con 0x7f7240093110 2021-02-10T11:14:50.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.934 7f7245540700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7240092c70 con 0x7f724018ae90 2021-02-10T11:14:50.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.934 7f7245540700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7240092e50 con 0x7f724018a470 2021-02-10T11:14:50.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.935 7f723f2bf700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7240093110 0x7f7240189e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:50.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.935 7f723f2bf700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7240093110 0x7f7240189e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:42898/0 (socket says 172.21.2.112:42898) 2021-02-10T11:14:50.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.935 7f723f2bf700 1 -- 172.21.2.112:0/3697909933 learned_addr learned my addr 172.21.2.112:0/3697909933 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:14:50.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.935 7f723eabe700 1 --2- 172.21.2.112:0/3697909933 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f724018a470 0x7f724018a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:50.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.935 7f723fac0700 1 --2- 172.21.2.112:0/3697909933 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f724018ae90 0x7f724018dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:50.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.935 7f723f2bf700 1 -- 172.21.2.112:0/3697909933 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f724018a470 msgr2=0x7f724018a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:50.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.935 7f723f2bf700 1 --2- 172.21.2.112:0/3697909933 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f724018a470 0x7f724018a870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:50.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.935 7f723f2bf700 1 -- 172.21.2.112:0/3697909933 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f724018ae90 msgr2=0x7f724018dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:50.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.935 7f723f2bf700 1 --2- 172.21.2.112:0/3697909933 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f724018ae90 0x7f724018dcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:50.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.935 7f723f2bf700 1 -- 172.21.2.112:0/3697909933 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f724018e2d0 con 0x7f7240093110 2021-02-10T11:14:50.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.935 7f723f2bf700 1 --2- 172.21.2.112:0/3697909933 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7240093110 0x7f7240189e50 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f722800b5b0 tx=0x7f722800b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:50.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.935 7f7237fff700 1 -- 172.21.2.112:0/3697909933 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f722800bd40 con 0x7f7240093110 2021-02-10T11:14:50.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.935 7f7237fff700 1 -- 172.21.2.112:0/3697909933 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7228000ce0 con 0x7f7240093110 2021-02-10T11:14:50.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.935 7f7237fff700 1 -- 172.21.2.112:0/3697909933 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f72280062a0 con 0x7f7240093110 2021-02-10T11:14:50.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.935 7f7245540700 1 -- 172.21.2.112:0/3697909933 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f724018e2d0 con 0x7f7240093110 2021-02-10T11:14:50.938 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.936 7f7245540700 1 -- 172.21.2.112:0/3697909933 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f724018e860 con 0x7f7240093110 2021-02-10T11:14:50.938 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.936 7f7237fff700 1 -- 172.21.2.112:0/3697909933 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7228008280 con 0x7f7240093110 2021-02-10T11:14:50.938 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.936 7f7237fff700 1 --2- 172.21.2.112:0/3697909933 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f72240186c0 0x7f722401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:50.938 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.936 7f7237fff700 1 -- 172.21.2.112:0/3697909933 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f7228032520 con 0x7f7240093110 2021-02-10T11:14:50.938 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.937 7f723eabe700 1 --2- 172.21.2.112:0/3697909933 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f72240186c0 0x7f722401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:50.938 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.937 7f723eabe700 1 --2- 172.21.2.112:0/3697909933 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f72240186c0 0x7f722401abc0 crc :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:50.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.968 7f7245540700 1 -- 172.21.2.112:0/3697909933 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72401036a0 con 0x7f7240093110 2021-02-10T11:14:50.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:50.971 7f7237fff700 1 -- 172.21.2.112:0/3697909933 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7228008070 con 0x7f7240093110 2021-02-10T11:14:51.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:51.097 7f7245540700 1 -- 172.21.2.112:0/3697909933 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f72400f0b20 con 0x7f7240093110 2021-02-10T11:14:51.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:51.097 7f7237fff700 1 -- 172.21.2.112:0/3697909933 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+2097 (secure 0 0 0) 0x7f722800bd40 con 0x7f7240093110 2021-02-10T11:14:51.096 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 6 2021-02-10T11:14:51.096 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:14:51.096 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4279,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.120:6835/2695560847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":2695560847},{"type":"v1","addr":"172.21.2.120:6835","nonce":2695560847}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":4}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:14:03.627691","modified":"2021-02-10 11:14:13.011189","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4272},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4272":{"gid":4272,"name":"a","rank":0,"incarnation":5,"state":"up:active","state_seq":3,"addr":"172.21.2.112:6833/2584920075","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":2584920075},{"type":"v1","addr":"172.21.2.112:6833","nonce":2584920075}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2021-02-10T11:14:51.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:51.099 7f7245540700 1 -- 172.21.2.112:0/3697909933 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f72240186c0 msgr2=0x7f722401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:51.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:51.099 7f7245540700 1 --2- 172.21.2.112:0/3697909933 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f72240186c0 0x7f722401abc0 crc :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:51.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:51.099 7f7245540700 1 -- 172.21.2.112:0/3697909933 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7240093110 msgr2=0x7f7240189e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:51.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:51.099 7f7245540700 1 --2- 172.21.2.112:0/3697909933 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7240093110 0x7f7240189e50 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f722800b5b0 tx=0x7f722800b950).stop 2021-02-10T11:14:51.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:51.100 7f7245540700 1 -- 172.21.2.112:0/3697909933 shutdown_connections 2021-02-10T11:14:51.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:51.100 7f7245540700 1 --2- 172.21.2.112:0/3697909933 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f72240186c0 0x7f722401abc0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:51.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:51.100 7f7245540700 1 --2- 172.21.2.112:0/3697909933 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f724018ae90 0x7f724018dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:51.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:51.100 7f7245540700 1 --2- 172.21.2.112:0/3697909933 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f724018a470 0x7f724018a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:51.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:51.100 7f7245540700 1 --2- 172.21.2.112:0/3697909933 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7240093110 0x7f7240189e50 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:51.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:51.100 7f7245540700 1 -- 172.21.2.112:0/3697909933 >> 172.21.2.112:0/3697909933 conn(0x7f72401a3780 msgr2=0x7f72401a5db0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:51.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:51.100 7f7245540700 1 -- 172.21.2.112:0/3697909933 shutdown_connections 2021-02-10T11:14:51.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:51.100 7f7245540700 1 -- 172.21.2.112:0/3697909933 wait complete. 2021-02-10T11:14:51.308 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 6, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:14:03.627691', 'modified': '2021-02-10 11:14:13.011189', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4272}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4272': {'gid': 4272, 'name': 'a', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.112:6833/2584920075', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.112:6832', 'nonce': 2584920075}, {'type': 'v1', 'addr': '172.21.2.112:6833', 'nonce': 2584920075}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:14:51.308 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:14:51.308 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok status 2021-02-10T11:14:51.709 INFO:teuthology.orchestra.run.gibba012.stdout:{ 2021-02-10T11:14:51.709 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", 2021-02-10T11:14:51.709 INFO:teuthology.orchestra.run.gibba012.stdout: "whoami": 0, 2021-02-10T11:14:51.709 INFO:teuthology.orchestra.run.gibba012.stdout: "id": 4272, 2021-02-10T11:14:51.709 INFO:teuthology.orchestra.run.gibba012.stdout: "want_state": "up:active", 2021-02-10T11:14:51.709 INFO:teuthology.orchestra.run.gibba012.stdout: "state": "up:active", 2021-02-10T11:14:51.709 INFO:teuthology.orchestra.run.gibba012.stdout: "rank_uptime": 39.699708747999999, 2021-02-10T11:14:51.709 INFO:teuthology.orchestra.run.gibba012.stdout: "mdsmap_epoch": 6, 2021-02-10T11:14:51.709 INFO:teuthology.orchestra.run.gibba012.stdout: "osdmap_epoch": 20, 2021-02-10T11:14:51.709 INFO:teuthology.orchestra.run.gibba012.stdout: "osdmap_epoch_barrier": 20, 2021-02-10T11:14:51.709 INFO:teuthology.orchestra.run.gibba012.stdout: "uptime": 44.066751916000001 2021-02-10T11:14:51.709 INFO:teuthology.orchestra.run.gibba012.stdout:} 2021-02-10T11:14:51.713 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", "whoami": 0, "id": 4272, "want_state": "up:active", "state": "up:active", "rank_uptime": 39.699708747999999, "mdsmap_epoch": 6, "osdmap_epoch": 20, "osdmap_epoch_barrier": 20, "uptime": 44.066751916000001 } 2021-02-10T11:14:51.713 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:14:51.713 INFO:tasks.cephfs.fuse_mount:Client client.3 config is {} 2021-02-10T11:14:51.713 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.3 at ubuntu@gibba020.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.3... 2021-02-10T11:14:51.713 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:14:51.713 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:14:52.049 DEBUG:teuthology.orchestra.run.gibba020:> sudo modprobe fuse 2021-02-10T11:14:52.393 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:14:52.393 DEBUG:teuthology.orchestra.run.gibba020:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:14:52.736 INFO:teuthology.orchestra.run.gibba020.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:14:52.736 INFO:teuthology.orchestra.run.gibba020.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:14:52.737 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:14:52.737 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:14:52.737 DEBUG:teuthology.orchestra.run.gibba020:> ls /sys/fs/fuse/connections 2021-02-10T11:14:53.072 INFO:teuthology.orchestra.run.gibba020.stdout:42 2021-02-10T11:14:53.072 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [42] 2021-02-10T11:14:53.072 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.3 /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:14:53.076 DEBUG:teuthology.orchestra.run.gibba020:> sudo modprobe fuse 2021-02-10T11:14:53.418 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:14:53.418 DEBUG:teuthology.orchestra.run.gibba020:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:14:53.484 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.gibba020.stderr:ceph-fuse[37508]: starting ceph client 2021-02-10T11:14:53.484 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.gibba020.stderr:2021-02-10 11:14:53.485 7fb23c01bf80 -1 init, newargv = 0x56413054b110 newargc=9 2021-02-10T11:14:53.489 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.gibba020.stderr:ceph-fuse[37508]: starting fuse 2021-02-10T11:14:53.761 INFO:teuthology.orchestra.run.gibba020.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:14:53.761 INFO:teuthology.orchestra.run.gibba020.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:14:53.762 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:14:53.762 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:14:53.762 DEBUG:teuthology.orchestra.run.gibba020:> ls /sys/fs/fuse/connections 2021-02-10T11:14:54.097 INFO:teuthology.orchestra.run.gibba020.stdout:42 2021-02-10T11:14:54.097 INFO:teuthology.orchestra.run.gibba020.stdout:43 2021-02-10T11:14:54.097 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42, 43] 2021-02-10T11:14:54.098 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:14:54.098 DEBUG:teuthology.orchestra.run.gibba020:> sudo python3 -c ' 2021-02-10T11:14:54.098 DEBUG:teuthology.orchestra.run.gibba020:> import glob 2021-02-10T11:14:54.098 DEBUG:teuthology.orchestra.run.gibba020:> import re 2021-02-10T11:14:54.098 DEBUG:teuthology.orchestra.run.gibba020:> import os 2021-02-10T11:14:54.099 DEBUG:teuthology.orchestra.run.gibba020:> import subprocess 2021-02-10T11:14:54.099 DEBUG:teuthology.orchestra.run.gibba020:> 2021-02-10T11:14:54.099 DEBUG:teuthology.orchestra.run.gibba020:> def find_socket(client_name): 2021-02-10T11:14:54.099 DEBUG:teuthology.orchestra.run.gibba020:> asok_path = "/var/run/ceph/ceph-client.3.*.asok" 2021-02-10T11:14:54.099 DEBUG:teuthology.orchestra.run.gibba020:> files = glob.glob(asok_path) 2021-02-10T11:14:54.099 DEBUG:teuthology.orchestra.run.gibba020:> 2021-02-10T11:14:54.099 DEBUG:teuthology.orchestra.run.gibba020:> # Given a non-glob path, it better be there 2021-02-10T11:14:54.099 DEBUG:teuthology.orchestra.run.gibba020:> if "*" not in asok_path: 2021-02-10T11:14:54.100 DEBUG:teuthology.orchestra.run.gibba020:> assert(len(files) == 1) 2021-02-10T11:14:54.100 DEBUG:teuthology.orchestra.run.gibba020:> return files[0] 2021-02-10T11:14:54.100 DEBUG:teuthology.orchestra.run.gibba020:> 2021-02-10T11:14:54.100 DEBUG:teuthology.orchestra.run.gibba020:> for f in files: 2021-02-10T11:14:54.100 DEBUG:teuthology.orchestra.run.gibba020:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:14:54.100 DEBUG:teuthology.orchestra.run.gibba020:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:14:54.101 DEBUG:teuthology.orchestra.run.gibba020:> return f 2021-02-10T11:14:54.101 DEBUG:teuthology.orchestra.run.gibba020:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:14:54.101 DEBUG:teuthology.orchestra.run.gibba020:> 2021-02-10T11:14:54.101 DEBUG:teuthology.orchestra.run.gibba020:> print(find_socket("client.3")) 2021-02-10T11:14:54.101 DEBUG:teuthology.orchestra.run.gibba020:> ' 2021-02-10T11:14:54.461 INFO:teuthology.orchestra.run.gibba020.stdout:/var/run/ceph/ceph-client.3.37508.asok 2021-02-10T11:14:54.464 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.3.37508.asok 2021-02-10T11:14:54.464 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:14:54.464 DEBUG:teuthology.orchestra.run.gibba020:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.3.37508.asok status 2021-02-10T11:14:54.852 INFO:teuthology.orchestra.run.gibba020.stdout:{ 2021-02-10T11:14:54.852 INFO:teuthology.orchestra.run.gibba020.stdout: "metadata": { 2021-02-10T11:14:54.852 INFO:teuthology.orchestra.run.gibba020.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:14:54.852 INFO:teuthology.orchestra.run.gibba020.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:14:54.852 INFO:teuthology.orchestra.run.gibba020.stdout: "entity_id": "3", 2021-02-10T11:14:54.852 INFO:teuthology.orchestra.run.gibba020.stdout: "hostname": "gibba020", 2021-02-10T11:14:54.852 INFO:teuthology.orchestra.run.gibba020.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.3", 2021-02-10T11:14:54.852 INFO:teuthology.orchestra.run.gibba020.stdout: "pid": "37508", 2021-02-10T11:14:54.852 INFO:teuthology.orchestra.run.gibba020.stdout: "root": "/" 2021-02-10T11:14:54.852 INFO:teuthology.orchestra.run.gibba020.stdout: }, 2021-02-10T11:14:54.852 INFO:teuthology.orchestra.run.gibba020.stdout: "dentry_count": 0, 2021-02-10T11:14:54.853 INFO:teuthology.orchestra.run.gibba020.stdout: "dentry_pinned_count": 0, 2021-02-10T11:14:54.853 INFO:teuthology.orchestra.run.gibba020.stdout: "id": 4358, 2021-02-10T11:14:54.853 INFO:teuthology.orchestra.run.gibba020.stdout: "inst": { 2021-02-10T11:14:54.853 INFO:teuthology.orchestra.run.gibba020.stdout: "name": { 2021-02-10T11:14:54.853 INFO:teuthology.orchestra.run.gibba020.stdout: "type": "client", 2021-02-10T11:14:54.853 INFO:teuthology.orchestra.run.gibba020.stdout: "num": 4358 2021-02-10T11:14:54.853 INFO:teuthology.orchestra.run.gibba020.stdout: }, 2021-02-10T11:14:54.853 INFO:teuthology.orchestra.run.gibba020.stdout: "addr": { 2021-02-10T11:14:54.853 INFO:teuthology.orchestra.run.gibba020.stdout: "type": "v1", 2021-02-10T11:14:54.853 INFO:teuthology.orchestra.run.gibba020.stdout: "addr": "172.21.2.120:0", 2021-02-10T11:14:54.853 INFO:teuthology.orchestra.run.gibba020.stdout: "nonce": 1992184211 2021-02-10T11:14:54.853 INFO:teuthology.orchestra.run.gibba020.stdout: } 2021-02-10T11:14:54.853 INFO:teuthology.orchestra.run.gibba020.stdout: }, 2021-02-10T11:14:54.854 INFO:teuthology.orchestra.run.gibba020.stdout: "addr": { 2021-02-10T11:14:54.854 INFO:teuthology.orchestra.run.gibba020.stdout: "type": "v1", 2021-02-10T11:14:54.854 INFO:teuthology.orchestra.run.gibba020.stdout: "addr": "172.21.2.120:0", 2021-02-10T11:14:54.854 INFO:teuthology.orchestra.run.gibba020.stdout: "nonce": 1992184211 2021-02-10T11:14:54.854 INFO:teuthology.orchestra.run.gibba020.stdout: }, 2021-02-10T11:14:54.854 INFO:teuthology.orchestra.run.gibba020.stdout: "inst_str": "client.4358 172.21.2.120:0/1992184211", 2021-02-10T11:14:54.854 INFO:teuthology.orchestra.run.gibba020.stdout: "addr_str": "172.21.2.120:0/1992184211", 2021-02-10T11:14:54.854 INFO:teuthology.orchestra.run.gibba020.stdout: "inode_count": 1, 2021-02-10T11:14:54.854 INFO:teuthology.orchestra.run.gibba020.stdout: "mds_epoch": 6, 2021-02-10T11:14:54.854 INFO:teuthology.orchestra.run.gibba020.stdout: "osd_epoch": 20, 2021-02-10T11:14:54.854 INFO:teuthology.orchestra.run.gibba020.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:14:54.854 INFO:teuthology.orchestra.run.gibba020.stdout: "blacklisted": false 2021-02-10T11:14:54.854 INFO:teuthology.orchestra.run.gibba020.stdout:} 2021-02-10T11:14:54.856 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:14:54.856 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:14:54.856 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:14:55.190 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:14:55.191 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:14:55.191 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:14:55.191 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:14:55.537 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:14:55.537 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:14:55.537 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:14:55.871 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:14:55.871 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:14:55.872 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:14:55.872 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:14:56.220 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:14:56.220 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:14:56.221 DEBUG:teuthology.orchestra.run.gibba020:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:14:56.556 INFO:teuthology.orchestra.run.gibba020.stdout:fuseblk 2021-02-10T11:14:56.557 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.2 2021-02-10T11:14:56.557 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:14:56.557 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:14:56.907 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:14:56.907 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:14:56.907 DEBUG:teuthology.orchestra.run.gibba020:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:14:57.242 INFO:teuthology.orchestra.run.gibba020.stdout:fuseblk 2021-02-10T11:14:57.242 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.3 2021-02-10T11:14:57.243 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:14:57.243 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:14:57.592 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2021-02-10T11:14:57.655 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:14:58.082 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.066 7febf3061700 1 Processor -- start 2021-02-10T11:14:58.082 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.066 7febf3061700 1 -- start start 2021-02-10T11:14:58.082 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.067 7febf3061700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7febec0d17f0 0x7febec1b31e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:58.082 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.067 7febf3061700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7febec0cff00 0x7febec0ef620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:58.082 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.067 7febf3061700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febec0f0b20 con 0x7febec0d1bf0 2021-02-10T11:14:58.082 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.067 7febf3061700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febec1b3ff0 con 0x7febec0d17f0 2021-02-10T11:14:58.083 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.067 7febf3061700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febec0f20b0 con 0x7febec0cff00 2021-02-10T11:14:58.083 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.067 7febf0e7d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7febec0d17f0 0x7febec1b31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:58.083 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.067 7febebfff700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7febec0d1bf0 0x7febec1b3800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:42368/0 (socket says 172.21.2.112:42368) 2021-02-10T11:14:58.083 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.067 7febebfff700 1 -- 172.21.2.112:0/876996750 learned_addr learned my addr 172.21.2.112:0/876996750 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:14:58.083 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.067 7febeb7fe700 1 -- 172.21.2.112:0/876996750 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 519643243 0 0) 0x7febdc000ce0 con 0x7febec0d1bf0 2021-02-10T11:14:58.083 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.068 7febeb7fe700 1 -- 172.21.2.112:0/876996750 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febd8002fe0 con 0x7febec0d1bf0 2021-02-10T11:14:58.084 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.068 7febf0e7d700 1 -- 172.21.2.112:0/876996750 >> v2:172.21.2.120:3300/0 conn(0x7febec0cff00 msgr2=0x7febec0ef620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:14:58.084 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.068 7febf0e7d700 1 --2- 172.21.2.112:0/876996750 >> v2:172.21.2.120:3300/0 conn(0x7febec0cff00 0x7febec0ef620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:58.084 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.068 7febf0e7d700 1 -- 172.21.2.112:0/876996750 >> v1:172.21.2.112:6789/0 conn(0x7febec0d1bf0 legacy=0x7febec1b3800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:58.084 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.068 7febf0e7d700 1 -- 172.21.2.112:0/876996750 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febec0f22d0 con 0x7febec0d17f0 2021-02-10T11:14:58.084 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.068 7febf0e7d700 1 --2- 172.21.2.112:0/876996750 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7febec0d17f0 0x7febec1b31e0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7febd4006650 tx=0x7febd4009a80).ready entity=mon.0 client_cookie=82f0fa3f29b8e511 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:58.084 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.068 7febeb7fe700 1 -- 172.21.2.112:0/876996750 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7febd40009b0 con 0x7febec0d17f0 2021-02-10T11:14:58.084 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.068 7febeb7fe700 1 -- 172.21.2.112:0/876996750 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febd400c0c0 con 0x7febec0d17f0 2021-02-10T11:14:58.085 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.068 7febeb7fe700 1 -- 172.21.2.112:0/876996750 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7febd400c550 con 0x7febec0d17f0 2021-02-10T11:14:58.085 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.068 7febf3061700 1 -- 172.21.2.112:0/876996750 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7febec0d17f0 msgr2=0x7febec1b31e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:58.085 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.068 7febf3061700 1 --2- 172.21.2.112:0/876996750 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7febec0d17f0 0x7febec1b31e0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7febd4006650 tx=0x7febd4009a80).stop 2021-02-10T11:14:58.085 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.069 7febf3061700 1 -- 172.21.2.112:0/876996750 shutdown_connections 2021-02-10T11:14:58.085 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.069 7febf3061700 1 --2- 172.21.2.112:0/876996750 >> v2:172.21.2.120:3300/0 conn(0x7febec0cff00 0x7febec0ef620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:58.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.069 7febf3061700 1 --2- 172.21.2.112:0/876996750 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7febec0d17f0 0x7febec1b31e0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:58.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.069 7febf3061700 1 -- 172.21.2.112:0/876996750 >> 172.21.2.112:0/876996750 conn(0x7febec0fde40 msgr2=0x7febec0ce390 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:58.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.069 7febf3061700 1 -- 172.21.2.112:0/876996750 shutdown_connections 2021-02-10T11:14:58.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.069 7febf3061700 1 -- 172.21.2.112:0/876996750 wait complete. 2021-02-10T11:14:58.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.083 7febf3061700 1 Processor -- start 2021-02-10T11:14:58.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.083 7febf3061700 1 -- start start 2021-02-10T11:14:58.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.084 7febf3061700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7febec1a7e80 0x7febec0ad650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:58.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.084 7febf3061700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7febec0adc70 0x7febec1b0fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:58.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.084 7febf3061700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7febec1b15c0 0x7febec1a5c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:58.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.084 7febf3061700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febec1a7980 con 0x7febec1b15c0 2021-02-10T11:14:58.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.084 7febf3061700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febec198920 con 0x7febec1a7e80 2021-02-10T11:14:58.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.084 7febf3061700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febec1b19c0 con 0x7febec0adc70 2021-02-10T11:14:58.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.084 7febf167e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7febec1b15c0 0x7febec1a5c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:58.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.084 7febf167e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7febec1b15c0 0x7febec1a5c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:42916/0 (socket says 172.21.2.112:42916) 2021-02-10T11:14:58.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.084 7febf167e700 1 -- 172.21.2.112:0/722706339 learned_addr learned my addr 172.21.2.112:0/722706339 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:14:58.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.084 7febf167e700 1 -- 172.21.2.112:0/722706339 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7febec0adc70 msgr2=0x7febec1b0fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:58.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.084 7febf167e700 1 --2- 172.21.2.112:0/722706339 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7febec0adc70 0x7febec1b0fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:58.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.084 7febf167e700 1 -- 172.21.2.112:0/722706339 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7febec1a7e80 msgr2=0x7febec0ad650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:58.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.084 7febf167e700 1 --2- 172.21.2.112:0/722706339 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7febec1a7e80 0x7febec0ad650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:58.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.084 7febf167e700 1 -- 172.21.2.112:0/722706339 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febec1b1be0 con 0x7febec1b15c0 2021-02-10T11:14:58.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.085 7febf167e700 1 --2- 172.21.2.112:0/722706339 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7febec1b15c0 0x7febec1a5c80 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7febe400b8c0 tx=0x7febe400bc60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:58.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.085 7febe97fa700 1 -- 172.21.2.112:0/722706339 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7febe4005af0 con 0x7febec1b15c0 2021-02-10T11:14:58.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.085 7febe97fa700 1 -- 172.21.2.112:0/722706339 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febe4005d30 con 0x7febec1b15c0 2021-02-10T11:14:58.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.085 7febf3061700 1 -- 172.21.2.112:0/722706339 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febec1b1be0 con 0x7febec1b15c0 2021-02-10T11:14:58.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.085 7febf3061700 1 -- 172.21.2.112:0/722706339 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febec1b1be0 con 0x7febec1b15c0 2021-02-10T11:14:58.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.085 7febe97fa700 1 -- 172.21.2.112:0/722706339 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7febe40062b0 con 0x7febec1b15c0 2021-02-10T11:14:58.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.086 7febe97fa700 1 -- 172.21.2.112:0/722706339 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7febe4006030 con 0x7febec1b15c0 2021-02-10T11:14:58.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.086 7febe97fa700 1 --2- 172.21.2.112:0/722706339 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7febd00186d0 0x7febd001abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:58.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.086 7febe97fa700 1 -- 172.21.2.112:0/722706339 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7febe4032b50 con 0x7febec1b15c0 2021-02-10T11:14:58.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.086 7febf0e7d700 1 --2- 172.21.2.112:0/722706339 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7febd00186d0 0x7febd001abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:58.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.086 7febf0e7d700 1 --2- 172.21.2.112:0/722706339 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7febd00186d0 0x7febd001abd0 crc :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:58.115 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.117 7febf3061700 1 -- 172.21.2.112:0/722706339 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febec1036a0 con 0x7febec1b15c0 2021-02-10T11:14:58.117 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.119 7febe97fa700 1 -- 172.21.2.112:0/722706339 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7febe4005aa0 con 0x7febec1b15c0 2021-02-10T11:14:58.241 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.243 7febf3061700 1 -- 172.21.2.112:0/722706339 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7febec1036a0 con 0x7febec1b15c0 2021-02-10T11:14:58.242 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.244 7febe97fa700 1 -- 172.21.2.112:0/722706339 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+13993 (secure 0 0 0) 0x7febe4006030 con 0x7febec1b15c0 2021-02-10T11:14:58.242 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:14:58.243 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":20,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:14:04.629689","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:14:01.400797","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2021-02-10 11:14:02.622129","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:14:58.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.245 7febf3061700 1 -- 172.21.2.112:0/722706339 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7febd00186d0 msgr2=0x7febd001abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:58.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.245 7febf3061700 1 --2- 172.21.2.112:0/722706339 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7febd00186d0 0x7febd001abd0 crc :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:58.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.245 7febf3061700 1 -- 172.21.2.112:0/722706339 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7febec1b15c0 msgr2=0x7febec1a5c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:58.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.245 7febf3061700 1 --2- 172.21.2.112:0/722706339 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7febec1b15c0 0x7febec1a5c80 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7febe400b8c0 tx=0x7febe400bc60).stop 2021-02-10T11:14:58.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.245 7febf3061700 1 -- 172.21.2.112:0/722706339 shutdown_connections 2021-02-10T11:14:58.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.246 7febf3061700 1 --2- 172.21.2.112:0/722706339 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7febd00186d0 0x7febd001abd0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:58.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.246 7febf3061700 1 --2- 172.21.2.112:0/722706339 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7febec1b15c0 0x7febec1a5c80 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:58.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.246 7febf3061700 1 --2- 172.21.2.112:0/722706339 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7febec0adc70 0x7febec1b0fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:58.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.246 7febf3061700 1 --2- 172.21.2.112:0/722706339 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7febec1a7e80 0x7febec0ad650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:58.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.246 7febf3061700 1 -- 172.21.2.112:0/722706339 >> 172.21.2.112:0/722706339 conn(0x7febec108000 msgr2=0x7febec08ccc0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:58.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.246 7febf3061700 1 -- 172.21.2.112:0/722706339 shutdown_connections 2021-02-10T11:14:58.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.246 7febf3061700 1 -- 172.21.2.112:0/722706339 wait complete. 2021-02-10T11:14:58.252 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:13:57.508758', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:14:01.400797', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:14:02.622129', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:14:58.252 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:14:58.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.666 7f3572c26700 1 Processor -- start 2021-02-10T11:14:58.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.667 7f3572c26700 1 -- start start 2021-02-10T11:14:58.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.667 7f3572c26700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f356c1b0870 0x7f356c0f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:58.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.667 7f3572c26700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f356c0efe00 0x7f356c0f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:58.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.667 7f3572c26700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f356c1b0c70 con 0x7f356c1b0470 2021-02-10T11:14:58.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.667 7f3572c26700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f356c0e5ea0 con 0x7f356c0efe00 2021-02-10T11:14:58.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.667 7f3572c26700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f356c0f0b50 con 0x7f356c1b0870 2021-02-10T11:14:58.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.667 7f3571243700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f356c0efe00 0x7f356c0f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:58.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.667 7f3571243700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f356c0efe00 0x7f356c0f0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:42920/0 (socket says 172.21.2.112:42920) 2021-02-10T11:14:58.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.667 7f3571243700 1 -- 172.21.2.112:0/3359982006 learned_addr learned my addr 172.21.2.112:0/3359982006 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:14:58.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.667 7f3570a42700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f356c1b0470 0x7f356c0ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:42382/0 (socket says 172.21.2.112:42382) 2021-02-10T11:14:58.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.668 7f356bfff700 1 --2- 172.21.2.112:0/3359982006 >> v2:172.21.2.120:3300/0 conn(0x7f356c1b0870 0x7f356c0f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:58.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.668 7f3571243700 1 -- 172.21.2.112:0/3359982006 >> v2:172.21.2.120:3300/0 conn(0x7f356c1b0870 msgr2=0x7f356c0f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:14:58.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.668 7f3571243700 1 --2- 172.21.2.112:0/3359982006 >> v2:172.21.2.120:3300/0 conn(0x7f356c1b0870 0x7f356c0f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:58.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.668 7f3571243700 1 -- 172.21.2.112:0/3359982006 >> v1:172.21.2.112:6789/0 conn(0x7f356c1b0470 legacy=0x7f356c0ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:58.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.668 7f3571243700 1 -- 172.21.2.112:0/3359982006 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f356c0f0820 con 0x7f356c0efe00 2021-02-10T11:14:58.684 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.668 7f3571243700 1 --2- 172.21.2.112:0/3359982006 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f356c0efe00 0x7f356c0f0200 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f3564006b80 tx=0x7f3564009b90).ready entity=mon.0 client_cookie=2c74953e8212d077 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:58.684 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.668 7f356b7fe700 1 -- 172.21.2.112:0/3359982006 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f35640022e0 con 0x7f356c0efe00 2021-02-10T11:14:58.684 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.668 7f356b7fe700 1 -- 172.21.2.112:0/3359982006 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35640009b0 con 0x7f356c0efe00 2021-02-10T11:14:58.684 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.668 7f356b7fe700 1 -- 172.21.2.112:0/3359982006 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f35640027e0 con 0x7f356c0efe00 2021-02-10T11:14:58.684 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.668 7f3572c26700 1 -- 172.21.2.112:0/3359982006 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f356c0efe00 msgr2=0x7f356c0f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:58.684 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.668 7f3572c26700 1 --2- 172.21.2.112:0/3359982006 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f356c0efe00 0x7f356c0f0200 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f3564006b80 tx=0x7f3564009b90).stop 2021-02-10T11:14:58.684 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.669 7f3572c26700 1 -- 172.21.2.112:0/3359982006 shutdown_connections 2021-02-10T11:14:58.684 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.669 7f3572c26700 1 --2- 172.21.2.112:0/3359982006 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f356c0efe00 0x7f356c0f0200 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:58.684 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.669 7f3572c26700 1 --2- 172.21.2.112:0/3359982006 >> v2:172.21.2.120:3300/0 conn(0x7f356c1b0870 0x7f356c0f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:58.684 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.669 7f3572c26700 1 -- 172.21.2.112:0/3359982006 >> 172.21.2.112:0/3359982006 conn(0x7f356c0ce3e0 msgr2=0x7f356c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:58.684 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.669 7f3572c26700 1 -- 172.21.2.112:0/3359982006 shutdown_connections 2021-02-10T11:14:58.684 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.669 7f3572c26700 1 -- 172.21.2.112:0/3359982006 wait complete. 2021-02-10T11:14:58.684 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.684 7f3572c26700 1 Processor -- start 2021-02-10T11:14:58.685 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.684 7f3572c26700 1 -- start start 2021-02-10T11:14:58.686 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.684 7f3572c26700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f356c105d10 0x7f356c09eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:58.686 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.684 7f3572c26700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f356c106110 0x7f356c09f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:58.686 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.684 7f3572c26700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f356c0b2a00 0x7f356c0b2e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:58.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.684 7f3572c26700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f356c09e560 con 0x7f356c0b2a00 2021-02-10T11:14:58.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.684 7f3572c26700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f356c09c090 con 0x7f356c105d10 2021-02-10T11:14:58.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.684 7f3572c26700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f356c0b3420 con 0x7f356c106110 2021-02-10T11:14:58.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.685 7f3571243700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f356c0b2a00 0x7f356c0b2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:58.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.685 7f3571243700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f356c0b2a00 0x7f356c0b2e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:42928/0 (socket says 172.21.2.112:42928) 2021-02-10T11:14:58.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.685 7f3571243700 1 -- 172.21.2.112:0/1362942498 learned_addr learned my addr 172.21.2.112:0/1362942498 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:14:58.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.685 7f3571243700 1 -- 172.21.2.112:0/1362942498 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f356c106110 msgr2=0x7f356c09f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:58.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.685 7f3571243700 1 --2- 172.21.2.112:0/1362942498 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f356c106110 0x7f356c09f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:58.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.685 7f3571243700 1 -- 172.21.2.112:0/1362942498 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f356c105d10 msgr2=0x7f356c09eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:58.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.685 7f3571243700 1 --2- 172.21.2.112:0/1362942498 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f356c105d10 0x7f356c09eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:58.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.685 7f3571243700 1 -- 172.21.2.112:0/1362942498 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f356c0b3640 con 0x7f356c0b2a00 2021-02-10T11:14:58.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.685 7f3571243700 1 --2- 172.21.2.112:0/1362942498 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f356c0b2a00 0x7f356c0b2e00 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f356400b630 tx=0x7f356400b9d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:58.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.685 7f35697fa700 1 -- 172.21.2.112:0/1362942498 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f356400bdc0 con 0x7f356c0b2a00 2021-02-10T11:14:58.689 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.685 7f35697fa700 1 -- 172.21.2.112:0/1362942498 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3564000b30 con 0x7f356c0b2a00 2021-02-10T11:14:58.689 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.685 7f3572c26700 1 -- 172.21.2.112:0/1362942498 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f356c0b3640 con 0x7f356c0b2a00 2021-02-10T11:14:58.689 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.685 7f3572c26700 1 -- 172.21.2.112:0/1362942498 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f356c0b3bd0 con 0x7f356c0b2a00 2021-02-10T11:14:58.689 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.686 7f35697fa700 1 -- 172.21.2.112:0/1362942498 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3564006430 con 0x7f356c0b2a00 2021-02-10T11:14:58.689 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.686 7f35697fa700 1 -- 172.21.2.112:0/1362942498 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3564006030 con 0x7f356c0b2a00 2021-02-10T11:14:58.689 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.686 7f35697fa700 1 --2- 172.21.2.112:0/1362942498 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f35500186c0 0x7f355001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:58.690 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.686 7f35697fa700 1 -- 172.21.2.112:0/1362942498 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f3564032590 con 0x7f356c0b2a00 2021-02-10T11:14:58.690 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.687 7f3570a42700 1 --2- 172.21.2.112:0/1362942498 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f35500186c0 0x7f355001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:58.690 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.687 7f3570a42700 1 --2- 172.21.2.112:0/1362942498 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f35500186c0 0x7f355001abc0 crc :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:58.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.716 7f3572c26700 1 -- 172.21.2.112:0/1362942498 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f356c105ae0 con 0x7f356c0b2a00 2021-02-10T11:14:58.717 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.719 7f35697fa700 1 -- 172.21.2.112:0/1362942498 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3564009480 con 0x7f356c0b2a00 2021-02-10T11:14:58.847 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.849 7f3572c26700 1 -- 172.21.2.112:0/1362942498 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f356c0f0b50 con 0x7f356c0b2a00 2021-02-10T11:14:58.847 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.849 7f35697fa700 1 -- 172.21.2.112:0/1362942498 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v20) v1 ==== 92+0+10 (secure 0 0 0) 0x7f356400bdc0 con 0x7f356c0b2a00 2021-02-10T11:14:58.848 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:14:58.849 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.851 7f3572c26700 1 -- 172.21.2.112:0/1362942498 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f35500186c0 msgr2=0x7f355001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:58.849 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.851 7f3572c26700 1 --2- 172.21.2.112:0/1362942498 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f35500186c0 0x7f355001abc0 crc :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:58.849 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.851 7f3572c26700 1 -- 172.21.2.112:0/1362942498 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f356c0b2a00 msgr2=0x7f356c0b2e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:58.849 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.851 7f3572c26700 1 --2- 172.21.2.112:0/1362942498 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f356c0b2a00 0x7f356c0b2e00 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f356400b630 tx=0x7f356400b9d0).stop 2021-02-10T11:14:58.850 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.851 7f3572c26700 1 -- 172.21.2.112:0/1362942498 shutdown_connections 2021-02-10T11:14:58.850 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.851 7f3572c26700 1 --2- 172.21.2.112:0/1362942498 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f35500186c0 0x7f355001abc0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:58.850 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.851 7f3572c26700 1 --2- 172.21.2.112:0/1362942498 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f356c0b2a00 0x7f356c0b2e00 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:58.850 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.851 7f3572c26700 1 --2- 172.21.2.112:0/1362942498 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f356c106110 0x7f356c09f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:58.850 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.851 7f3572c26700 1 --2- 172.21.2.112:0/1362942498 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f356c105d10 0x7f356c09eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:58.850 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.851 7f3572c26700 1 -- 172.21.2.112:0/1362942498 >> 172.21.2.112:0/1362942498 conn(0x7f356c0c36f0 msgr2=0x7f356c0bd640 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:58.850 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.851 7f3572c26700 1 -- 172.21.2.112:0/1362942498 shutdown_connections 2021-02-10T11:14:58.850 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:58.851 7f3572c26700 1 -- 172.21.2.112:0/1362942498 wait complete. 2021-02-10T11:14:58.858 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:14:59.287 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.271 7f36c3178700 1 Processor -- start 2021-02-10T11:14:59.287 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.271 7f36c3178700 1 -- start start 2021-02-10T11:14:59.287 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.272 7f36c3178700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f36bc1b0470 0x7f36bc1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:59.288 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.272 7f36c3178700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f36bc0ef610 0x7f36bc0efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:59.288 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.272 7f36c3178700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f36bc0f0030 0x7f36bc0f0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:59.288 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.272 7f36c3178700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36bc0e5ea0 con 0x7f36bc1b0470 2021-02-10T11:14:59.288 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.272 7f36c3178700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36bc0f0b50 con 0x7f36bc0f0030 2021-02-10T11:14:59.288 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.272 7f36c3178700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36bc0e52e0 con 0x7f36bc0ef610 2021-02-10T11:14:59.288 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.272 7f36c0f94700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f36bc1b0470 0x7f36bc1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:59.288 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.272 7f36c0f94700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f36bc1b0470 0x7f36bc1b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:42932/0 (socket says 172.21.2.112:42932) 2021-02-10T11:14:59.289 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.272 7f36c0f94700 1 -- 172.21.2.112:0/969630671 learned_addr learned my addr 172.21.2.112:0/969630671 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:14:59.289 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.272 7f36b3fff700 1 --2- 172.21.2.112:0/969630671 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f36bc0ef610 0x7f36bc0efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:59.289 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.272 7f36c1795700 1 --2- 172.21.2.112:0/969630671 >> v2:172.21.2.120:3300/0 conn(0x7f36bc0f0030 0x7f36bc0f0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:59.289 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.273 7f36b3fff700 1 -- 172.21.2.112:0/969630671 >> v2:172.21.2.120:3300/0 conn(0x7f36bc0f0030 msgr2=0x7f36bc0f0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:14:59.289 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.273 7f36b3fff700 1 --2- 172.21.2.112:0/969630671 >> v2:172.21.2.120:3300/0 conn(0x7f36bc0f0030 0x7f36bc0f0430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:59.289 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.273 7f36b3fff700 1 -- 172.21.2.112:0/969630671 >> v2:172.21.2.112:3300/0 conn(0x7f36bc1b0470 msgr2=0x7f36bc1b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:14:59.289 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.273 7f36b3fff700 1 --2- 172.21.2.112:0/969630671 >> v2:172.21.2.112:3300/0 conn(0x7f36bc1b0470 0x7f36bc1b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:59.290 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.273 7f36b3fff700 1 -- 172.21.2.112:0/969630671 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36bc0f10d0 con 0x7f36bc0ef610 2021-02-10T11:14:59.290 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.273 7f36c0f94700 1 --2- 172.21.2.112:0/969630671 >> v2:172.21.2.112:3300/0 conn(0x7f36bc1b0470 0x7f36bc1b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:14:59.290 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.273 7f36b3fff700 1 --2- 172.21.2.112:0/969630671 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f36bc0ef610 0x7f36bc0efa10 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f36ac006650 tx=0x7f36ac009a80).ready entity=mon.2 client_cookie=6911cc0a7a900e8e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:59.290 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.273 7f36b37fe700 1 -- 172.21.2.112:0/969630671 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f36ac0009b0 con 0x7f36bc0ef610 2021-02-10T11:14:59.290 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.273 7f36b37fe700 1 -- 172.21.2.112:0/969630671 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36ac00c0c0 con 0x7f36bc0ef610 2021-02-10T11:14:59.290 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.273 7f36b37fe700 1 -- 172.21.2.112:0/969630671 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f36ac00c5e0 con 0x7f36bc0ef610 2021-02-10T11:14:59.290 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.273 7f36c3178700 1 -- 172.21.2.112:0/969630671 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f36bc0ef610 msgr2=0x7f36bc0efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:59.290 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.273 7f36c3178700 1 --2- 172.21.2.112:0/969630671 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f36bc0ef610 0x7f36bc0efa10 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f36ac006650 tx=0x7f36ac009a80).stop 2021-02-10T11:14:59.290 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.274 7f36c3178700 1 -- 172.21.2.112:0/969630671 shutdown_connections 2021-02-10T11:14:59.290 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.274 7f36c3178700 1 --2- 172.21.2.112:0/969630671 >> v2:172.21.2.120:3300/0 conn(0x7f36bc0f0030 0x7f36bc0f0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:59.291 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.274 7f36c3178700 1 --2- 172.21.2.112:0/969630671 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f36bc0ef610 0x7f36bc0efa10 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:59.291 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.274 7f36c3178700 1 --2- 172.21.2.112:0/969630671 >> v2:172.21.2.112:3300/0 conn(0x7f36bc1b0470 0x7f36bc1b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:59.291 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.274 7f36c3178700 1 -- 172.21.2.112:0/969630671 >> 172.21.2.112:0/969630671 conn(0x7f36bc0ce3e0 msgr2=0x7f36bc0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:59.291 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.274 7f36c3178700 1 -- 172.21.2.112:0/969630671 shutdown_connections 2021-02-10T11:14:59.291 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.274 7f36c3178700 1 -- 172.21.2.112:0/969630671 wait complete. 2021-02-10T11:14:59.291 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.289 7f36c3178700 1 Processor -- start 2021-02-10T11:14:59.292 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.289 7f36c3178700 1 -- start start 2021-02-10T11:14:59.293 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.289 7f36c3178700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f36bc09e560 0x7f36bc0a0130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:59.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.289 7f36c3178700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f36bc09e960 0x7f36bc0a0750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:59.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.289 7f36c3178700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f36bc0a0d70 0x7f36bc0a1af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:59.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.289 7f36c3178700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36bc0c38f0 con 0x7f36bc09e560 2021-02-10T11:14:59.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.289 7f36c3178700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36bc0a1170 con 0x7f36bc0a0d70 2021-02-10T11:14:59.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.289 7f36c3178700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36bc0a1390 con 0x7f36bc09e960 2021-02-10T11:14:59.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.289 7f36b3fff700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f36bc09e960 0x7f36bc0a0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:59.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.289 7f36c0f94700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f36bc09e560 0x7f36bc0a0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:59.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.289 7f36c0f94700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f36bc09e560 0x7f36bc0a0130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:42938/0 (socket says 172.21.2.112:42938) 2021-02-10T11:14:59.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.289 7f36c0f94700 1 -- 172.21.2.112:0/2131735624 learned_addr learned my addr 172.21.2.112:0/2131735624 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:14:59.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.290 7f36c1795700 1 --2- 172.21.2.112:0/2131735624 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f36bc0a0d70 0x7f36bc0a1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:59.295 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.290 7f36c0f94700 1 -- 172.21.2.112:0/2131735624 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f36bc09e960 msgr2=0x7f36bc0a0750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:59.295 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.290 7f36c0f94700 1 --2- 172.21.2.112:0/2131735624 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f36bc09e960 0x7f36bc0a0750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:59.295 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.290 7f36c0f94700 1 -- 172.21.2.112:0/2131735624 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f36bc0a0d70 msgr2=0x7f36bc0a1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:59.295 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.290 7f36c0f94700 1 --2- 172.21.2.112:0/2131735624 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f36bc0a0d70 0x7f36bc0a1af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:59.296 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.290 7f36c0f94700 1 -- 172.21.2.112:0/2131735624 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36bc0a2110 con 0x7f36bc09e560 2021-02-10T11:14:59.296 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.290 7f36b3fff700 1 --2- 172.21.2.112:0/2131735624 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f36bc09e960 0x7f36bc0a0750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:14:59.296 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.290 7f36c0f94700 1 --2- 172.21.2.112:0/2131735624 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f36bc09e560 0x7f36bc0a0130 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f36b8006ee0 tx=0x7f36b800b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:59.296 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.290 7f36b17fa700 1 -- 172.21.2.112:0/2131735624 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f36b800bd50 con 0x7f36bc09e560 2021-02-10T11:14:59.296 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.290 7f36b17fa700 1 -- 172.21.2.112:0/2131735624 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36b80052e0 con 0x7f36bc09e560 2021-02-10T11:14:59.296 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.290 7f36c3178700 1 -- 172.21.2.112:0/2131735624 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36bc0a2110 con 0x7f36bc09e560 2021-02-10T11:14:59.296 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.290 7f36c3178700 1 -- 172.21.2.112:0/2131735624 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36bc0a26a0 con 0x7f36bc09e560 2021-02-10T11:14:59.296 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.290 7f36b17fa700 1 -- 172.21.2.112:0/2131735624 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f36b8005800 con 0x7f36bc09e560 2021-02-10T11:14:59.296 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.291 7f36b17fa700 1 -- 172.21.2.112:0/2131735624 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f36b80057d0 con 0x7f36bc09e560 2021-02-10T11:14:59.296 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.291 7f36b17fa700 1 --2- 172.21.2.112:0/2131735624 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f369c0186c0 0x7f369c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:59.296 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.291 7f36b17fa700 1 -- 172.21.2.112:0/2131735624 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f36b80327c0 con 0x7f36bc09e560 2021-02-10T11:14:59.297 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.291 7f36b3fff700 1 --2- 172.21.2.112:0/2131735624 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f369c0186c0 0x7f369c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:59.297 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.292 7f36b3fff700 1 --2- 172.21.2.112:0/2131735624 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f369c0186c0 0x7f369c01abc0 crc :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:59.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.321 7f36c3178700 1 -- 172.21.2.112:0/2131735624 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36bc105b30 con 0x7f36bc09e560 2021-02-10T11:14:59.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.324 7f36b17fa700 1 -- 172.21.2.112:0/2131735624 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f36b80009b0 con 0x7f36bc09e560 2021-02-10T11:14:59.450 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.451 7f36c3178700 1 -- 172.21.2.112:0/2131735624 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f36bc0f0b50 con 0x7f36bc09e560 2021-02-10T11:14:59.451 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.452 7f36b17fa700 1 -- 172.21.2.112:0/2131735624 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v20) v1 ==== 104+0+10 (secure 0 0 0) 0x7f36b800bd50 con 0x7f36bc09e560 2021-02-10T11:14:59.451 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:14:59.452 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.453 7f36c3178700 1 -- 172.21.2.112:0/2131735624 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f369c0186c0 msgr2=0x7f369c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:59.452 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.453 7f36c3178700 1 --2- 172.21.2.112:0/2131735624 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f369c0186c0 0x7f369c01abc0 crc :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:59.452 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.453 7f36c3178700 1 -- 172.21.2.112:0/2131735624 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f36bc09e560 msgr2=0x7f36bc0a0130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:59.452 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.453 7f36c3178700 1 --2- 172.21.2.112:0/2131735624 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f36bc09e560 0x7f36bc0a0130 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f36b8006ee0 tx=0x7f36b800b960).stop 2021-02-10T11:14:59.452 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.454 7f36c3178700 1 -- 172.21.2.112:0/2131735624 shutdown_connections 2021-02-10T11:14:59.452 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.454 7f36c3178700 1 --2- 172.21.2.112:0/2131735624 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f369c0186c0 0x7f369c01abc0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:59.452 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.454 7f36c3178700 1 --2- 172.21.2.112:0/2131735624 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f36bc0a0d70 0x7f36bc0a1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:59.453 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.454 7f36c3178700 1 --2- 172.21.2.112:0/2131735624 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f36bc09e960 0x7f36bc0a0750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:59.453 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.454 7f36c3178700 1 --2- 172.21.2.112:0/2131735624 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f36bc09e560 0x7f36bc0a0130 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:59.453 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.454 7f36c3178700 1 -- 172.21.2.112:0/2131735624 >> 172.21.2.112:0/2131735624 conn(0x7f36bc09fc10 msgr2=0x7f36bc094ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:59.453 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.454 7f36c3178700 1 -- 172.21.2.112:0/2131735624 shutdown_connections 2021-02-10T11:14:59.453 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.454 7f36c3178700 1 -- 172.21.2.112:0/2131735624 wait complete. 2021-02-10T11:14:59.460 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:14:59.890 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.875 7f8f6c50f700 1 Processor -- start 2021-02-10T11:14:59.890 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.875 7f8f6c50f700 1 -- start start 2021-02-10T11:14:59.890 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.876 7f8f6c50f700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f8f640f05f0 0x7f8f640f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:59.890 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.876 7f8f6c50f700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f641b0c70 con 0x7f8f641b0870 2021-02-10T11:14:59.890 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.876 7f8f6c50f700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f640f0b50 con 0x7f8f641b0470 2021-02-10T11:14:59.891 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.876 7f8f6c50f700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f640e5ea0 con 0x7f8f640f05f0 2021-02-10T11:14:59.891 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.876 7f8f69b2a700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f8f641b0870 0x7f8f640efe00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:42408/0 (socket says 172.21.2.112:42408) 2021-02-10T11:14:59.891 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.876 7f8f69b2a700 1 -- 172.21.2.112:0/2062893862 learned_addr learned my addr 172.21.2.112:0/2062893862 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:14:59.891 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.876 7f8f6ab2c700 1 --2- 172.21.2.112:0/2062893862 >> v2:172.21.2.120:3300/0 conn(0x7f8f640f05f0 0x7f8f640f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:59.891 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.876 7f8f69329700 1 -- 172.21.2.112:0/2062893862 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 366649989 0 0) 0x7f8f5c001060 con 0x7f8f641b0870 2021-02-10T11:14:59.891 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.876 7f8f69329700 1 -- 172.21.2.112:0/2062893862 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f50002fe0 con 0x7f8f641b0870 2021-02-10T11:14:59.891 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.876 7f8f6ab2c700 1 -- 172.21.2.112:0/2062893862 >> v1:172.21.2.120:6789/0 conn(0x7f8f641b0470 legacy=0x7f8f640ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:59.891 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.876 7f8f6ab2c700 1 -- 172.21.2.112:0/2062893862 >> v1:172.21.2.112:6789/0 conn(0x7f8f641b0870 legacy=0x7f8f640efe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:59.891 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.876 7f8f6ab2c700 1 -- 172.21.2.112:0/2062893862 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f640e52e0 con 0x7f8f640f05f0 2021-02-10T11:14:59.891 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.877 7f8f6ab2c700 1 --2- 172.21.2.112:0/2062893862 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8f640f05f0 0x7f8f640f20e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f8f60006ad0 tx=0x7f8f60009ad0).ready entity=mon.1 client_cookie=a012c4dbe02beb55 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:59.891 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.877 7f8f69329700 1 -- 172.21.2.112:0/2062893862 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8f600022e0 con 0x7f8f640f05f0 2021-02-10T11:14:59.891 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.877 7f8f69329700 1 -- 172.21.2.112:0/2062893862 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f60006410 con 0x7f8f640f05f0 2021-02-10T11:14:59.891 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.877 7f8f69329700 1 -- 172.21.2.112:0/2062893862 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8f60002560 con 0x7f8f640f05f0 2021-02-10T11:14:59.892 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.877 7f8f6c50f700 1 -- 172.21.2.112:0/2062893862 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8f640f05f0 msgr2=0x7f8f640f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:59.892 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.877 7f8f6c50f700 1 --2- 172.21.2.112:0/2062893862 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8f640f05f0 0x7f8f640f20e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f8f60006ad0 tx=0x7f8f60009ad0).stop 2021-02-10T11:14:59.892 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.877 7f8f6c50f700 1 -- 172.21.2.112:0/2062893862 shutdown_connections 2021-02-10T11:14:59.892 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.877 7f8f6c50f700 1 --2- 172.21.2.112:0/2062893862 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8f640f05f0 0x7f8f640f20e0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:59.892 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.877 7f8f6c50f700 1 -- 172.21.2.112:0/2062893862 >> 172.21.2.112:0/2062893862 conn(0x7f8f640ce3e0 msgr2=0x7f8f640e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:59.892 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.878 7f8f6c50f700 1 -- 172.21.2.112:0/2062893862 shutdown_connections 2021-02-10T11:14:59.892 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.878 7f8f6c50f700 1 -- 172.21.2.112:0/2062893862 wait complete. 2021-02-10T11:14:59.894 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.892 7f8f6c50f700 1 Processor -- start 2021-02-10T11:14:59.894 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.892 7f8f6c50f700 1 -- start start 2021-02-10T11:14:59.894 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.893 7f8f6c50f700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8f640b2840 0x7f8f641a4300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:59.894 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.893 7f8f6c50f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8f640b30d0 0x7f8f640b34d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:59.894 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.893 7f8f6c50f700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8f640b3af0 0x7f8f640f0580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:59.894 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.893 7f8f6c50f700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f640b2c40 con 0x7f8f640b30d0 2021-02-10T11:14:59.894 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.893 7f8f6c50f700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f641a4920 con 0x7f8f640b3af0 2021-02-10T11:14:59.894 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.893 7f8f6c50f700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f640f0ba0 con 0x7f8f640b2840 2021-02-10T11:14:59.894 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.893 7f8f69b2a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8f640b30d0 0x7f8f640b34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:59.894 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.893 7f8f69b2a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8f640b30d0 0x7f8f640b34d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:42952/0 (socket says 172.21.2.112:42952) 2021-02-10T11:14:59.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.893 7f8f69b2a700 1 -- 172.21.2.112:0/3639154120 learned_addr learned my addr 172.21.2.112:0/3639154120 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:14:59.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.893 7f8f69b2a700 1 -- 172.21.2.112:0/3639154120 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8f640b2840 msgr2=0x7f8f641a4300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:59.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.893 7f8f69b2a700 1 --2- 172.21.2.112:0/3639154120 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8f640b2840 0x7f8f641a4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:59.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.893 7f8f69b2a700 1 -- 172.21.2.112:0/3639154120 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8f640b3af0 msgr2=0x7f8f640f0580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:59.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.893 7f8f69b2a700 1 --2- 172.21.2.112:0/3639154120 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8f640b3af0 0x7f8f640f0580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:59.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.893 7f8f69b2a700 1 -- 172.21.2.112:0/3639154120 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f640f0dc0 con 0x7f8f640b30d0 2021-02-10T11:14:59.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.893 7f8f69b2a700 1 --2- 172.21.2.112:0/3639154120 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8f640b30d0 0x7f8f640b34d0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f8f5c009980 tx=0x7f8f5c00bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:59.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.893 7f8f5affd700 1 -- 172.21.2.112:0/3639154120 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8f5c006080 con 0x7f8f640b30d0 2021-02-10T11:14:59.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.894 7f8f5affd700 1 -- 172.21.2.112:0/3639154120 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f5c006380 con 0x7f8f640b30d0 2021-02-10T11:14:59.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.894 7f8f6c50f700 1 -- 172.21.2.112:0/3639154120 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f640f0dc0 con 0x7f8f640b30d0 2021-02-10T11:14:59.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.894 7f8f5affd700 1 -- 172.21.2.112:0/3639154120 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8f5c0049a0 con 0x7f8f640b30d0 2021-02-10T11:14:59.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.894 7f8f6c50f700 1 -- 172.21.2.112:0/3639154120 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f640f1350 con 0x7f8f640b30d0 2021-02-10T11:14:59.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.894 7f8f5affd700 1 -- 172.21.2.112:0/3639154120 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8f5c006560 con 0x7f8f640b30d0 2021-02-10T11:14:59.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.894 7f8f5affd700 1 --2- 172.21.2.112:0/3639154120 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8f4c0186d0 0x7f8f4c01abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:59.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.894 7f8f5affd700 1 -- 172.21.2.112:0/3639154120 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f8f5c032980 con 0x7f8f640b30d0 2021-02-10T11:14:59.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.895 7f8f6a32b700 1 --2- 172.21.2.112:0/3639154120 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8f4c0186d0 0x7f8f4c01abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:59.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.895 7f8f6a32b700 1 --2- 172.21.2.112:0/3639154120 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8f4c0186d0 0x7f8f4c01abd0 crc :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:59.924 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.925 7f8f6c50f700 1 -- 172.21.2.112:0/3639154120 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f640f18f0 con 0x7f8f640b30d0 2021-02-10T11:14:59.926 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:14:59.928 7f8f5affd700 1 -- 172.21.2.112:0/3639154120 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8f5c004c60 con 0x7f8f640b30d0 2021-02-10T11:15:00.055 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:00.056 7f8f6c50f700 1 -- 172.21.2.112:0/3639154120 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f8f640f0050 con 0x7f8f640b30d0 2021-02-10T11:15:00.055 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:00.057 7f8f5affd700 1 -- 172.21.2.112:0/3639154120 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v20) v1 ==== 100+0+10 (secure 0 0 0) 0x7f8f5c006380 con 0x7f8f640b30d0 2021-02-10T11:15:00.055 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:15:00.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:00.059 7f8f6c50f700 1 -- 172.21.2.112:0/3639154120 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8f4c0186d0 msgr2=0x7f8f4c01abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:00.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:00.059 7f8f6c50f700 1 --2- 172.21.2.112:0/3639154120 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8f4c0186d0 0x7f8f4c01abd0 crc :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:00.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:00.059 7f8f6c50f700 1 -- 172.21.2.112:0/3639154120 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8f640b30d0 msgr2=0x7f8f640b34d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:00.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:00.059 7f8f6c50f700 1 --2- 172.21.2.112:0/3639154120 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8f640b30d0 0x7f8f640b34d0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f8f5c009980 tx=0x7f8f5c00bcd0).stop 2021-02-10T11:15:00.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:00.059 7f8f6c50f700 1 -- 172.21.2.112:0/3639154120 shutdown_connections 2021-02-10T11:15:00.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:00.059 7f8f6c50f700 1 --2- 172.21.2.112:0/3639154120 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8f4c0186d0 0x7f8f4c01abd0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:00.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:00.059 7f8f6c50f700 1 --2- 172.21.2.112:0/3639154120 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8f640b3af0 0x7f8f640f0580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:00.058 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:00.059 7f8f6c50f700 1 --2- 172.21.2.112:0/3639154120 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8f640b30d0 0x7f8f640b34d0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:00.058 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:00.059 7f8f6c50f700 1 --2- 172.21.2.112:0/3639154120 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8f640b2840 0x7f8f641a4300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:00.058 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:00.059 7f8f6c50f700 1 -- 172.21.2.112:0/3639154120 >> 172.21.2.112:0/3639154120 conn(0x7f8f640c36f0 msgr2=0x7f8f64094c30 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:00.058 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:00.059 7f8f6c50f700 1 -- 172.21.2.112:0/3639154120 shutdown_connections 2021-02-10T11:15:00.058 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:00.059 7f8f6c50f700 1 -- 172.21.2.112:0/3639154120 wait complete. 2021-02-10T11:15:00.066 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:15:00.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:00.476 7f3648d4a700 1 Processor -- start 2021-02-10T11:15:00.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:00.476 7f3648d4a700 1 -- start start 2021-02-10T11:15:00.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:00.476 7f3648d4a700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f36440e5470 0x7f364407e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:00.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:00.476 7f3648d4a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f364407f300 0x7f364407f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:00.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:00.476 7f3648d4a700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f364407ed80 con 0x7f36440e5070 2021-02-10T11:15:00.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:00.476 7f3648d4a700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36441b4010 con 0x7f36440e5470 2021-02-10T11:15:00.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:00.476 7f3648d4a700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36440d1790 con 0x7f364407f300 2021-02-10T11:15:00.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:00.477 7f3643314700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f364407f300 0x7f364407f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:00.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:00.477 7f3643314700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f364407f300 0x7f364407f700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3301/0 says I am v2:172.21.2.112:42646/0 (socket says 172.21.2.112:42646) 2021-02-10T11:15:00.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:00.477 7f3643314700 1 -- 172.21.2.112:0/84694462 learned_addr learned my addr 172.21.2.112:0/84694462 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:15:00.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:00.477 7f3642312700 1 --2- 172.21.2.112:0/84694462 >> v2:172.21.2.120:3300/0 conn(0x7f36440e5470 0x7f364407e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:00.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:00.478 7f3642312700 1 -- 172.21.2.112:0/84694462 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f364407f300 msgr2=0x7f364407f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:15:00.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:00.478 7f3642312700 1 --2- 172.21.2.112:0/84694462 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f364407f300 0x7f364407f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:00.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:00.478 7f3642312700 1 -- 172.21.2.112:0/84694462 >> v1:172.21.2.120:6789/0 conn(0x7f36440e5070 legacy=0x7f36441b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:00.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:00.478 7f3642312700 1 -- 172.21.2.112:0/84694462 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36440d19b0 con 0x7f36440e5470 2021-02-10T11:15:00.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:00.478 7f3643314700 1 --2- 172.21.2.112:0/84694462 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f364407f300 0x7f364407f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:15:00.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:00.478 7f3642312700 1 --2- 172.21.2.112:0/84694462 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f36440e5470 0x7f364407e6a0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f362c006650 tx=0x7f362c009a80).ready entity=mon.1 client_cookie=f317fe1b8bfb51f0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:00.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:00.478 7f3641b11700 1 -- 172.21.2.112:0/84694462 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f362c00c0c0 con 0x7f36440e5470 2021-02-10T11:15:00.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:00.478 7f3641b11700 1 -- 172.21.2.112:0/84694462 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f362c00c340 con 0x7f36440e5470 2021-02-10T11:15:00.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:00.478 7f3641b11700 1 -- 172.21.2.112:0/84694462 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f362c00c860 con 0x7f36440e5470 2021-02-10T11:15:00.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:00.479 7f3648d4a700 1 -- 172.21.2.112:0/84694462 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f36440e5470 msgr2=0x7f364407e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:00.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:00.479 7f3648d4a700 1 --2- 172.21.2.112:0/84694462 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f36440e5470 0x7f364407e6a0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f362c006650 tx=0x7f362c009a80).stop 2021-02-10T11:15:00.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:00.479 7f3648d4a700 1 -- 172.21.2.112:0/84694462 shutdown_connections 2021-02-10T11:15:00.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:00.479 7f3648d4a700 1 --2- 172.21.2.112:0/84694462 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f36440e5470 0x7f364407e6a0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:00.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:00.479 7f3648d4a700 1 --2- 172.21.2.112:0/84694462 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f364407f300 0x7f364407f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:00.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:00.479 7f3648d4a700 1 -- 172.21.2.112:0/84694462 >> 172.21.2.112:0/84694462 conn(0x7f36440ce690 msgr2=0x7f36440e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:00.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:00.479 7f3648d4a700 1 -- 172.21.2.112:0/84694462 shutdown_connections 2021-02-10T11:15:00.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:00.479 7f3648d4a700 1 -- 172.21.2.112:0/84694462 wait complete. 2021-02-10T11:15:00.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:00.494 7f3648d4a700 1 Processor -- start 2021-02-10T11:15:00.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:00.494 7f3648d4a700 1 -- start start 2021-02-10T11:15:00.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:00.494 7f3648d4a700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f36441af5e0 0x7f36441b1d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:00.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:00.494 7f3648d4a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f36441b2360 0x7f364418f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:00.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:00.494 7f3648d4a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f364418f990 0x7f3644080750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:00.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:00.494 7f3648d4a700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3644080d70 con 0x7f364418f990 2021-02-10T11:15:00.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:00.494 7f3648d4a700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3644080f90 con 0x7f36441af5e0 2021-02-10T11:15:00.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:00.494 7f3648d4a700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36440811b0 con 0x7f36441b2360 2021-02-10T11:15:00.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:00.494 7f3643314700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f364418f990 0x7f3644080750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:00.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:00.494 7f3643314700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f364418f990 0x7f3644080750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:42968/0 (socket says 172.21.2.112:42968) 2021-02-10T11:15:00.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:00.494 7f3643314700 1 -- 172.21.2.112:0/3107832105 learned_addr learned my addr 172.21.2.112:0/3107832105 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:15:00.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:00.495 7f3643314700 1 -- 172.21.2.112:0/3107832105 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f36441b2360 msgr2=0x7f364418f370 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:15:00.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:00.495 7f3643314700 1 --2- 172.21.2.112:0/3107832105 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f36441b2360 0x7f364418f370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:00.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:00.495 7f3643314700 1 -- 172.21.2.112:0/3107832105 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f36441af5e0 msgr2=0x7f36441b1d40 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:15:00.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:00.495 7f3643314700 1 --2- 172.21.2.112:0/3107832105 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f36441af5e0 0x7f36441b1d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:00.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:00.495 7f3643314700 1 -- 172.21.2.112:0/3107832105 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36440b1390 con 0x7f364418f990 2021-02-10T11:15:00.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:00.495 7f3643314700 1 --2- 172.21.2.112:0/3107832105 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f364418f990 0x7f3644080750 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f363c009940 tx=0x7f363c00bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:00.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:00.495 7f363b7fe700 1 -- 172.21.2.112:0/3107832105 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f363c00c2f0 con 0x7f364418f990 2021-02-10T11:15:00.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:00.495 7f363b7fe700 1 -- 172.21.2.112:0/3107832105 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f363c006790 con 0x7f364418f990 2021-02-10T11:15:00.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:00.495 7f363b7fe700 1 -- 172.21.2.112:0/3107832105 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f363c00c760 con 0x7f364418f990 2021-02-10T11:15:00.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:00.495 7f3648d4a700 1 -- 172.21.2.112:0/3107832105 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36440b1390 con 0x7f364418f990 2021-02-10T11:15:00.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:00.495 7f3648d4a700 1 -- 172.21.2.112:0/3107832105 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36440b1390 con 0x7f364418f990 2021-02-10T11:15:00.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:00.496 7f363b7fe700 1 -- 172.21.2.112:0/3107832105 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f363c0011f0 con 0x7f364418f990 2021-02-10T11:15:00.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:00.496 7f363b7fe700 1 --2- 172.21.2.112:0/3107832105 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f36280186c0 0x7f362801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:00.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:00.496 7f363b7fe700 1 -- 172.21.2.112:0/3107832105 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f363c032b30 con 0x7f364418f990 2021-02-10T11:15:00.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:00.496 7f3642b13700 1 --2- 172.21.2.112:0/3107832105 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f36280186c0 0x7f362801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:00.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:00.497 7f3642b13700 1 --2- 172.21.2.112:0/3107832105 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f36280186c0 0x7f362801abc0 crc :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:00.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:00.527 7f3648d4a700 1 -- 172.21.2.112:0/3107832105 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36441b1810 con 0x7f364418f990 2021-02-10T11:15:00.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:00.529 7f363b7fe700 1 -- 172.21.2.112:0/3107832105 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f363c00be30 con 0x7f364418f990 2021-02-10T11:15:00.652 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:00.654 7f3648d4a700 1 -- 172.21.2.112:0/3107832105 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f36441b1810 con 0x7f364418f990 2021-02-10T11:15:00.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:00.655 7f363b7fe700 1 -- 172.21.2.112:0/3107832105 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+13993 (secure 0 0 0) 0x7f363c0011f0 con 0x7f364418f990 2021-02-10T11:15:00.653 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:15:00.654 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":20,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:14:04.629689","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:14:01.400797","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2021-02-10 11:14:02.622129","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:15:00.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:00.656 7f3648d4a700 1 -- 172.21.2.112:0/3107832105 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f36280186c0 msgr2=0x7f362801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:00.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:00.656 7f3648d4a700 1 --2- 172.21.2.112:0/3107832105 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f36280186c0 0x7f362801abc0 crc :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:00.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:00.656 7f3648d4a700 1 -- 172.21.2.112:0/3107832105 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f364418f990 msgr2=0x7f3644080750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:00.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:00.656 7f3648d4a700 1 --2- 172.21.2.112:0/3107832105 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f364418f990 0x7f3644080750 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f363c009940 tx=0x7f363c00bcf0).stop 2021-02-10T11:15:00.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:00.657 7f3648d4a700 1 -- 172.21.2.112:0/3107832105 shutdown_connections 2021-02-10T11:15:00.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:00.657 7f3648d4a700 1 --2- 172.21.2.112:0/3107832105 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f36280186c0 0x7f362801abc0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:00.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:00.657 7f3648d4a700 1 --2- 172.21.2.112:0/3107832105 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f364418f990 0x7f3644080750 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:00.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:00.657 7f3648d4a700 1 --2- 172.21.2.112:0/3107832105 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f36441b2360 0x7f364418f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:00.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:00.657 7f3648d4a700 1 --2- 172.21.2.112:0/3107832105 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f36441af5e0 0x7f36441b1d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:00.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:00.657 7f3648d4a700 1 -- 172.21.2.112:0/3107832105 >> 172.21.2.112:0/3107832105 conn(0x7f36441afc80 msgr2=0x7f3644192820 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:00.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:00.657 7f3648d4a700 1 -- 172.21.2.112:0/3107832105 shutdown_connections 2021-02-10T11:15:00.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:00.657 7f3648d4a700 1 -- 172.21.2.112:0/3107832105 wait complete. 2021-02-10T11:15:00.664 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:13:57.508758', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:14:01.400797', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:14:02.622129', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:15:00.665 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:15:01.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.079 7f17808dc700 1 Processor -- start 2021-02-10T11:15:01.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.079 7f17808dc700 1 -- start start 2021-02-10T11:15:01.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.079 7f17808dc700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f177c1b0470 0x7f177c1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:01.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.079 7f17808dc700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f177c0ef610 0x7f177c0efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:01.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.079 7f17808dc700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f177c0f0b50 con 0x7f177c0f0030 2021-02-10T11:15:01.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.079 7f17808dc700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f177c0e52e0 con 0x7f177c1b0470 2021-02-10T11:15:01.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.079 7f17808dc700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f177c0e5490 con 0x7f177c0ef610 2021-02-10T11:15:01.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.080 7f177a61c700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f177c1b0470 0x7f177c1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:01.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.080 7f177a61c700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f177c1b0470 0x7f177c1b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:42974/0 (socket says 172.21.2.112:42974) 2021-02-10T11:15:01.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.080 7f177a61c700 1 -- 172.21.2.112:0/1270927613 learned_addr learned my addr 172.21.2.112:0/1270927613 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:15:01.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.080 7f177a61c700 1 -- 172.21.2.112:0/1270927613 >> v2:172.21.2.120:3300/0 conn(0x7f177c0ef610 msgr2=0x7f177c0efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:15:01.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.080 7f177a61c700 1 --2- 172.21.2.112:0/1270927613 >> v2:172.21.2.120:3300/0 conn(0x7f177c0ef610 0x7f177c0efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:01.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.080 7f177a61c700 1 -- 172.21.2.112:0/1270927613 >> v1:172.21.2.112:6789/0 conn(0x7f177c0f0030 legacy=0x7f177c0f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:01.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.080 7f177a61c700 1 -- 172.21.2.112:0/1270927613 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f177c0f0430 con 0x7f177c1b0470 2021-02-10T11:15:01.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.080 7f177a61c700 1 --2- 172.21.2.112:0/1270927613 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f177c1b0470 0x7f177c1b0870 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f1764006700 tx=0x7f1764009b80).ready entity=mon.0 client_cookie=300e450c3175ce0f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:01.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.080 7f177961a700 1 -- 172.21.2.112:0/1270927613 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f176400c0c0 con 0x7f177c1b0470 2021-02-10T11:15:01.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.080 7f177961a700 1 -- 172.21.2.112:0/1270927613 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f176400c340 con 0x7f177c1b0470 2021-02-10T11:15:01.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.080 7f177961a700 1 -- 172.21.2.112:0/1270927613 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f176400c860 con 0x7f177c1b0470 2021-02-10T11:15:01.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.080 7f17808dc700 1 -- 172.21.2.112:0/1270927613 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f177c1b0470 msgr2=0x7f177c1b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:01.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.080 7f17808dc700 1 --2- 172.21.2.112:0/1270927613 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f177c1b0470 0x7f177c1b0870 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f1764006700 tx=0x7f1764009b80).stop 2021-02-10T11:15:01.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.081 7f17808dc700 1 -- 172.21.2.112:0/1270927613 shutdown_connections 2021-02-10T11:15:01.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.081 7f17808dc700 1 --2- 172.21.2.112:0/1270927613 >> v2:172.21.2.120:3300/0 conn(0x7f177c0ef610 0x7f177c0efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:01.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.081 7f17808dc700 1 --2- 172.21.2.112:0/1270927613 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f177c1b0470 0x7f177c1b0870 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:01.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.081 7f17808dc700 1 -- 172.21.2.112:0/1270927613 >> 172.21.2.112:0/1270927613 conn(0x7f177c0ce3e0 msgr2=0x7f177c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:01.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.081 7f17808dc700 1 -- 172.21.2.112:0/1270927613 shutdown_connections 2021-02-10T11:15:01.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.081 7f17808dc700 1 -- 172.21.2.112:0/1270927613 wait complete. 2021-02-10T11:15:01.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.096 7f17808dc700 1 Processor -- start 2021-02-10T11:15:01.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.096 7f17808dc700 1 -- start start 2021-02-10T11:15:01.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.096 7f17808dc700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f177c0c36a0 0x7f177c0c2b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:01.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.096 7f17808dc700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f177c0c3190 0x7f177c0b1d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:01.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.096 7f17808dc700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f177c0b2370 0x7f177c0b2d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:01.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.096 7f17808dc700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f177c0b27c0 con 0x7f177c0c3190 2021-02-10T11:15:01.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.096 7f17808dc700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f177c0b3360 con 0x7f177c0b2370 2021-02-10T11:15:01.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.096 7f17808dc700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f177c0b3540 con 0x7f177c0c36a0 2021-02-10T11:15:01.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.097 7f1779e1b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f177c0c3190 0x7f177c0b1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:01.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.097 7f1779e1b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f177c0c3190 0x7f177c0b1d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:42984/0 (socket says 172.21.2.112:42984) 2021-02-10T11:15:01.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.097 7f1779e1b700 1 -- 172.21.2.112:0/2567898575 learned_addr learned my addr 172.21.2.112:0/2567898575 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:15:01.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.097 7f1779e1b700 1 -- 172.21.2.112:0/2567898575 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f177c0c36a0 msgr2=0x7f177c0c2b70 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:15:01.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.097 7f1779e1b700 1 --2- 172.21.2.112:0/2567898575 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f177c0c36a0 0x7f177c0c2b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:01.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.097 7f1779e1b700 1 -- 172.21.2.112:0/2567898575 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f177c0b2370 msgr2=0x7f177c0b2d40 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:15:01.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.097 7f1779e1b700 1 --2- 172.21.2.112:0/2567898575 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f177c0b2370 0x7f177c0b2d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:01.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.097 7f1779e1b700 1 -- 172.21.2.112:0/2567898575 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f177c0b3760 con 0x7f177c0c3190 2021-02-10T11:15:01.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.097 7f1779e1b700 1 --2- 172.21.2.112:0/2567898575 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f177c0c3190 0x7f177c0b1d50 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f176c006f00 tx=0x7f176c00b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:01.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.097 7f1772ffd700 1 -- 172.21.2.112:0/2567898575 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f176c00bd30 con 0x7f177c0c3190 2021-02-10T11:15:01.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.097 7f1772ffd700 1 -- 172.21.2.112:0/2567898575 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f176c000c90 con 0x7f177c0c3190 2021-02-10T11:15:01.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.097 7f1772ffd700 1 -- 172.21.2.112:0/2567898575 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f176c0022a0 con 0x7f177c0c3190 2021-02-10T11:15:01.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.097 7f17808dc700 1 -- 172.21.2.112:0/2567898575 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f177c0b3760 con 0x7f177c0c3190 2021-02-10T11:15:01.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.097 7f17808dc700 1 -- 172.21.2.112:0/2567898575 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f177c0b3cf0 con 0x7f177c0c3190 2021-02-10T11:15:01.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.098 7f1772ffd700 1 -- 172.21.2.112:0/2567898575 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f176c005310 con 0x7f177c0c3190 2021-02-10T11:15:01.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.098 7f1772ffd700 1 --2- 172.21.2.112:0/2567898575 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f17600186c0 0x7f176001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:01.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.098 7f1772ffd700 1 -- 172.21.2.112:0/2567898575 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f176c032580 con 0x7f177c0c3190 2021-02-10T11:15:01.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.099 7f177a61c700 1 --2- 172.21.2.112:0/2567898575 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f17600186c0 0x7f176001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:01.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.099 7f177a61c700 1 --2- 172.21.2.112:0/2567898575 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f17600186c0 0x7f176001abc0 crc :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:01.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.129 7f17808dc700 1 -- 172.21.2.112:0/2567898575 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f177c0edd10 con 0x7f177c0c3190 2021-02-10T11:15:01.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.132 7f1772ffd700 1 -- 172.21.2.112:0/2567898575 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f176c025fb0 con 0x7f177c0c3190 2021-02-10T11:15:01.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.262 7f17808dc700 1 -- 172.21.2.112:0/2567898575 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f177c0e52e0 con 0x7f177c0c3190 2021-02-10T11:15:01.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.262 7f1772ffd700 1 -- 172.21.2.112:0/2567898575 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v20) v1 ==== 92+0+10 (secure 0 0 0) 0x7f176c025fb0 con 0x7f177c0c3190 2021-02-10T11:15:01.261 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:15:01.262 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.264 7f17808dc700 1 -- 172.21.2.112:0/2567898575 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f17600186c0 msgr2=0x7f176001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:01.262 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.264 7f17808dc700 1 --2- 172.21.2.112:0/2567898575 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f17600186c0 0x7f176001abc0 crc :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:01.262 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.264 7f17808dc700 1 -- 172.21.2.112:0/2567898575 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f177c0c3190 msgr2=0x7f177c0b1d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:01.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.264 7f17808dc700 1 --2- 172.21.2.112:0/2567898575 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f177c0c3190 0x7f177c0b1d50 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f176c006f00 tx=0x7f176c00b940).stop 2021-02-10T11:15:01.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.264 7f17808dc700 1 -- 172.21.2.112:0/2567898575 shutdown_connections 2021-02-10T11:15:01.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.264 7f17808dc700 1 --2- 172.21.2.112:0/2567898575 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f17600186c0 0x7f176001abc0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:01.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.264 7f17808dc700 1 --2- 172.21.2.112:0/2567898575 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f177c0b2370 0x7f177c0b2d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:01.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.264 7f17808dc700 1 --2- 172.21.2.112:0/2567898575 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f177c0c3190 0x7f177c0b1d50 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:01.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.264 7f17808dc700 1 --2- 172.21.2.112:0/2567898575 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f177c0c36a0 0x7f177c0c2b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:01.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.264 7f17808dc700 1 -- 172.21.2.112:0/2567898575 >> 172.21.2.112:0/2567898575 conn(0x7f177c09f630 msgr2=0x7f177c0a0130 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:01.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.264 7f17808dc700 1 -- 172.21.2.112:0/2567898575 shutdown_connections 2021-02-10T11:15:01.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.264 7f17808dc700 1 -- 172.21.2.112:0/2567898575 wait complete. 2021-02-10T11:15:01.270 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:15:01.665 DEBUG:teuthology.orchestra.run.gibba012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:15:01.668 DEBUG:teuthology.orchestra.run.gibba020:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:15:01.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.686 7fee86ac2700 1 Processor -- start 2021-02-10T11:15:01.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.686 7fee86ac2700 1 -- start start 2021-02-10T11:15:01.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.687 7fee86ac2700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fee801b0870 0x7fee800f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:01.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.687 7fee86ac2700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fee800efe00 0x7fee800f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:01.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.687 7fee86ac2700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee801b0c70 con 0x7fee801b0470 2021-02-10T11:15:01.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.687 7fee86ac2700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee800f0b50 con 0x7fee800efe00 2021-02-10T11:15:01.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.687 7fee86ac2700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee800f0820 con 0x7fee801b0870 2021-02-10T11:15:01.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.687 7fee848de700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7fee801b0470 0x7fee800ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:36008/0 (socket says 172.21.2.112:36008) 2021-02-10T11:15:01.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.687 7fee850df700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fee800efe00 0x7fee800f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:01.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.687 7fee848de700 1 -- 172.21.2.112:0/3720027949 learned_addr learned my addr 172.21.2.112:0/3720027949 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:15:01.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.687 7fee7ffff700 1 --2- 172.21.2.112:0/3720027949 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fee801b0870 0x7fee800f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:01.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.688 7fee7ffff700 1 -- 172.21.2.112:0/3720027949 >> v2:172.21.2.112:3300/0 conn(0x7fee800efe00 msgr2=0x7fee800f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:15:01.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.688 7fee7ffff700 1 --2- 172.21.2.112:0/3720027949 >> v2:172.21.2.112:3300/0 conn(0x7fee800efe00 0x7fee800f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:01.704 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.688 7fee7ffff700 1 -- 172.21.2.112:0/3720027949 >> v1:172.21.2.120:6789/0 conn(0x7fee801b0470 legacy=0x7fee800ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:01.704 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.688 7fee7ffff700 1 -- 172.21.2.112:0/3720027949 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee800e52e0 con 0x7fee801b0870 2021-02-10T11:15:01.704 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.688 7fee850df700 1 --2- 172.21.2.112:0/3720027949 >> v2:172.21.2.112:3300/0 conn(0x7fee800efe00 0x7fee800f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:15:01.704 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.688 7fee7ffff700 1 --2- 172.21.2.112:0/3720027949 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fee801b0870 0x7fee800f20e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7fee70006650 tx=0x7fee70009a80).ready entity=mon.2 client_cookie=3f5be270556fca4f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:01.704 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.688 7fee7f7fe700 1 -- 172.21.2.112:0/3720027949 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fee700009b0 con 0x7fee801b0870 2021-02-10T11:15:01.704 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.688 7fee7f7fe700 1 -- 172.21.2.112:0/3720027949 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee7000c0c0 con 0x7fee801b0870 2021-02-10T11:15:01.704 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.688 7fee7f7fe700 1 -- 172.21.2.112:0/3720027949 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fee7000c5e0 con 0x7fee801b0870 2021-02-10T11:15:01.704 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.688 7fee86ac2700 1 -- 172.21.2.112:0/3720027949 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fee801b0870 msgr2=0x7fee800f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:01.704 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.688 7fee86ac2700 1 --2- 172.21.2.112:0/3720027949 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fee801b0870 0x7fee800f20e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7fee70006650 tx=0x7fee70009a80).stop 2021-02-10T11:15:01.704 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.689 7fee86ac2700 1 -- 172.21.2.112:0/3720027949 shutdown_connections 2021-02-10T11:15:01.704 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.689 7fee86ac2700 1 --2- 172.21.2.112:0/3720027949 >> v2:172.21.2.112:3300/0 conn(0x7fee800efe00 0x7fee800f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:01.704 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.689 7fee86ac2700 1 --2- 172.21.2.112:0/3720027949 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fee801b0870 0x7fee800f20e0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:01.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.689 7fee86ac2700 1 -- 172.21.2.112:0/3720027949 >> 172.21.2.112:0/3720027949 conn(0x7fee800ce3e0 msgr2=0x7fee800e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:01.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.689 7fee86ac2700 1 -- 172.21.2.112:0/3720027949 shutdown_connections 2021-02-10T11:15:01.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.689 7fee86ac2700 1 -- 172.21.2.112:0/3720027949 wait complete. 2021-02-10T11:15:01.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.704 7fee86ac2700 1 Processor -- start 2021-02-10T11:15:01.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.704 7fee86ac2700 1 -- start start 2021-02-10T11:15:01.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.705 7fee86ac2700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fee800a0420 0x7fee800a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:01.707 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.705 7fee86ac2700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fee80189d00 0x7fee8018a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:01.707 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.705 7fee86ac2700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fee8018a720 0x7fee800b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:01.707 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.705 7fee86ac2700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee80189800 con 0x7fee80189d00 2021-02-10T11:15:01.707 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.705 7fee86ac2700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee8018ab20 con 0x7fee8018a720 2021-02-10T11:15:01.707 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.705 7fee86ac2700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee800a1140 con 0x7fee800a0420 2021-02-10T11:15:01.707 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.705 7fee7ffff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fee80189d00 0x7fee8018a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:01.707 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.705 7fee7ffff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fee80189d00 0x7fee8018a100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:42998/0 (socket says 172.21.2.112:42998) 2021-02-10T11:15:01.707 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.705 7fee7ffff700 1 -- 172.21.2.112:0/3006904723 learned_addr learned my addr 172.21.2.112:0/3006904723 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:15:01.707 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.705 7fee848de700 1 --2- 172.21.2.112:0/3006904723 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fee800a0420 0x7fee800a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:01.707 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.705 7fee850df700 1 --2- 172.21.2.112:0/3006904723 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fee8018a720 0x7fee800b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:01.707 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.706 7fee7ffff700 1 -- 172.21.2.112:0/3006904723 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fee800a0420 msgr2=0x7fee800a0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:01.708 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.706 7fee7ffff700 1 --2- 172.21.2.112:0/3006904723 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fee800a0420 0x7fee800a0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:01.708 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.706 7fee7ffff700 1 -- 172.21.2.112:0/3006904723 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fee8018a720 msgr2=0x7fee800b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:01.708 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.706 7fee7ffff700 1 --2- 172.21.2.112:0/3006904723 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fee8018a720 0x7fee800b7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:01.708 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.706 7fee7ffff700 1 -- 172.21.2.112:0/3006904723 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee800b8200 con 0x7fee80189d00 2021-02-10T11:15:01.708 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.706 7fee7ffff700 1 --2- 172.21.2.112:0/3006904723 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fee80189d00 0x7fee8018a100 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7fee70009650 tx=0x7fee7000b8e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:01.708 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.706 7fee7d7fa700 1 -- 172.21.2.112:0/3006904723 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fee700093a0 con 0x7fee80189d00 2021-02-10T11:15:01.708 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.706 7fee7d7fa700 1 -- 172.21.2.112:0/3006904723 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee70006070 con 0x7fee80189d00 2021-02-10T11:15:01.708 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.706 7fee7d7fa700 1 -- 172.21.2.112:0/3006904723 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fee700052e0 con 0x7fee80189d00 2021-02-10T11:15:01.708 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.706 7fee86ac2700 1 -- 172.21.2.112:0/3006904723 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee800b8200 con 0x7fee80189d00 2021-02-10T11:15:01.708 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.706 7fee86ac2700 1 -- 172.21.2.112:0/3006904723 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee800b8200 con 0x7fee80189d00 2021-02-10T11:15:01.708 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.707 7fee7d7fa700 1 -- 172.21.2.112:0/3006904723 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fee70009300 con 0x7fee80189d00 2021-02-10T11:15:01.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.707 7fee7d7fa700 1 --2- 172.21.2.112:0/3006904723 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fee640186c0 0x7fee6401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:01.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.707 7fee7d7fa700 1 -- 172.21.2.112:0/3006904723 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fee700327c0 con 0x7fee80189d00 2021-02-10T11:15:01.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.707 7fee848de700 1 --2- 172.21.2.112:0/3006904723 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fee640186c0 0x7fee6401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:01.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.708 7fee848de700 1 --2- 172.21.2.112:0/3006904723 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fee640186c0 0x7fee6401abc0 crc :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:01.734 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.736 7fee86ac2700 1 -- 172.21.2.112:0/3006904723 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee80105ae0 con 0x7fee80189d00 2021-02-10T11:15:01.737 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.739 7fee7d7fa700 1 -- 172.21.2.112:0/3006904723 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fee70006030 con 0x7fee80189d00 2021-02-10T11:15:01.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.870 7fee86ac2700 1 -- 172.21.2.112:0/3006904723 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7fee800f0b50 con 0x7fee80189d00 2021-02-10T11:15:01.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.870 7fee7d7fa700 1 -- 172.21.2.112:0/3006904723 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v20) v1 ==== 104+0+10 (secure 0 0 0) 0x7fee70000940 con 0x7fee80189d00 2021-02-10T11:15:01.869 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:15:01.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.872 7fee86ac2700 1 -- 172.21.2.112:0/3006904723 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fee640186c0 msgr2=0x7fee6401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:01.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.872 7fee86ac2700 1 --2- 172.21.2.112:0/3006904723 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fee640186c0 0x7fee6401abc0 crc :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:01.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.872 7fee86ac2700 1 -- 172.21.2.112:0/3006904723 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fee80189d00 msgr2=0x7fee8018a100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:01.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.873 7fee86ac2700 1 --2- 172.21.2.112:0/3006904723 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fee80189d00 0x7fee8018a100 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7fee70009650 tx=0x7fee7000b8e0).stop 2021-02-10T11:15:01.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.873 7fee86ac2700 1 -- 172.21.2.112:0/3006904723 shutdown_connections 2021-02-10T11:15:01.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.873 7fee86ac2700 1 --2- 172.21.2.112:0/3006904723 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fee640186c0 0x7fee6401abc0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:01.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.873 7fee86ac2700 1 --2- 172.21.2.112:0/3006904723 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fee8018a720 0x7fee800b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:01.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.873 7fee86ac2700 1 --2- 172.21.2.112:0/3006904723 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fee80189d00 0x7fee8018a100 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:01.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.873 7fee86ac2700 1 --2- 172.21.2.112:0/3006904723 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fee800a0420 0x7fee800a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:01.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.873 7fee86ac2700 1 -- 172.21.2.112:0/3006904723 >> 172.21.2.112:0/3006904723 conn(0x7fee80094e10 msgr2=0x7fee800a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:01.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.873 7fee86ac2700 1 -- 172.21.2.112:0/3006904723 shutdown_connections 2021-02-10T11:15:01.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:01.873 7fee86ac2700 1 -- 172.21.2.112:0/3006904723 wait complete. 2021-02-10T11:15:01.879 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:15:02.347 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.333 7f36a2e7e700 1 Processor -- start 2021-02-10T11:15:02.348 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.333 7f36a2e7e700 1 -- start start 2021-02-10T11:15:02.348 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.333 7f36a2e7e700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f369c1b0870 0x7f369c0f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:02.348 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.333 7f36a2e7e700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f369c0efe00 0x7f369c0f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:02.348 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.333 7f36a2e7e700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f369c1b0c70 con 0x7f369c1b0470 2021-02-10T11:15:02.348 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.333 7f36a2e7e700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f369c0f0b50 con 0x7f369c0efe00 2021-02-10T11:15:02.348 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.333 7f36a2e7e700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f369c0f0820 con 0x7f369c1b0870 2021-02-10T11:15:02.348 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.333 7f36a149b700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f369c0efe00 0x7f369c0f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:02.348 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.333 7f36a0c9a700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f369c1b0470 0x7f369c0ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:42464/0 (socket says 172.21.2.112:42464) 2021-02-10T11:15:02.348 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.333 7f36a0c9a700 1 -- 172.21.2.112:0/2969186121 learned_addr learned my addr 172.21.2.112:0/2969186121 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:15:02.349 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.334 7f369bfff700 1 --2- 172.21.2.112:0/2969186121 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f369c1b0870 0x7f369c0f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:02.349 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.334 7f36a149b700 1 -- 172.21.2.112:0/2969186121 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f369c1b0870 msgr2=0x7f369c0f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:15:02.349 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.334 7f36a149b700 1 --2- 172.21.2.112:0/2969186121 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f369c1b0870 0x7f369c0f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:02.349 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.334 7f36a149b700 1 -- 172.21.2.112:0/2969186121 >> v1:172.21.2.112:6789/0 conn(0x7f369c1b0470 legacy=0x7f369c0ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:02.349 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.334 7f36a149b700 1 -- 172.21.2.112:0/2969186121 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f369c0e52e0 con 0x7f369c0efe00 2021-02-10T11:15:02.349 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.334 7f369bfff700 1 --2- 172.21.2.112:0/2969186121 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f369c1b0870 0x7f369c0f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:15:02.349 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.334 7f36a149b700 1 --2- 172.21.2.112:0/2969186121 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f369c0efe00 0x7f369c0f0200 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f3694006ab0 tx=0x7f3694009b40).ready entity=mon.0 client_cookie=e4cbcdbced34ec75 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:02.349 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.334 7f369b7fe700 1 -- 172.21.2.112:0/2969186121 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f36940022e0 con 0x7f369c0efe00 2021-02-10T11:15:02.349 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.334 7f369b7fe700 1 -- 172.21.2.112:0/2969186121 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3694002560 con 0x7f369c0efe00 2021-02-10T11:15:02.349 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.334 7f369b7fe700 1 -- 172.21.2.112:0/2969186121 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3694002930 con 0x7f369c0efe00 2021-02-10T11:15:02.349 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.334 7f36a2e7e700 1 -- 172.21.2.112:0/2969186121 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f369c0efe00 msgr2=0x7f369c0f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:02.349 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.334 7f36a2e7e700 1 --2- 172.21.2.112:0/2969186121 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f369c0efe00 0x7f369c0f0200 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f3694006ab0 tx=0x7f3694009b40).stop 2021-02-10T11:15:02.350 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.335 7f36a2e7e700 1 -- 172.21.2.112:0/2969186121 shutdown_connections 2021-02-10T11:15:02.350 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.335 7f36a2e7e700 1 --2- 172.21.2.112:0/2969186121 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f369c0efe00 0x7f369c0f0200 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:02.350 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.335 7f36a2e7e700 1 --2- 172.21.2.112:0/2969186121 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f369c1b0870 0x7f369c0f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:02.350 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.335 7f36a2e7e700 1 -- 172.21.2.112:0/2969186121 >> 172.21.2.112:0/2969186121 conn(0x7f369c0ce3e0 msgr2=0x7f369c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:02.350 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.335 7f36a2e7e700 1 -- 172.21.2.112:0/2969186121 shutdown_connections 2021-02-10T11:15:02.350 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.335 7f36a2e7e700 1 -- 172.21.2.112:0/2969186121 wait complete. 2021-02-10T11:15:02.352 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.350 7f36a2e7e700 1 Processor -- start 2021-02-10T11:15:02.352 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.350 7f36a2e7e700 1 -- start start 2021-02-10T11:15:02.352 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.350 7f36a2e7e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f369c190920 0x7f369c08f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:02.352 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.350 7f36a2e7e700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f369c190d20 0x7f369c08fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:02.352 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.350 7f36a2e7e700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f369c193270 0x7f369c193670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:02.352 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.350 7f36a2e7e700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f369c090470 con 0x7f369c190920 2021-02-10T11:15:02.352 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.350 7f36a2e7e700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f369c193c90 con 0x7f369c190d20 2021-02-10T11:15:02.352 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.350 7f36a2e7e700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f369c193e70 con 0x7f369c193270 2021-02-10T11:15:02.352 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.351 7f36a0c9a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f369c190920 0x7f369c08f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:02.352 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.351 7f36a0c9a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f369c190920 0x7f369c08f600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:43008/0 (socket says 172.21.2.112:43008) 2021-02-10T11:15:02.352 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.351 7f36a0c9a700 1 -- 172.21.2.112:0/1176739540 learned_addr learned my addr 172.21.2.112:0/1176739540 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:15:02.353 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.351 7f36a149b700 1 --2- 172.21.2.112:0/1176739540 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f369c193270 0x7f369c193670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:02.353 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.351 7f36a0c9a700 1 -- 172.21.2.112:0/1176739540 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f369c193270 msgr2=0x7f369c193670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:02.353 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.351 7f369bfff700 1 --2- 172.21.2.112:0/1176739540 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f369c190d20 0x7f369c08fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:02.353 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.351 7f36a0c9a700 1 --2- 172.21.2.112:0/1176739540 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f369c193270 0x7f369c193670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:02.353 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.351 7f36a0c9a700 1 -- 172.21.2.112:0/1176739540 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f369c190d20 msgr2=0x7f369c08fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:02.353 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.351 7f36a0c9a700 1 --2- 172.21.2.112:0/1176739540 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f369c190d20 0x7f369c08fc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:02.353 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.351 7f36a0c9a700 1 -- 172.21.2.112:0/1176739540 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f369c194090 con 0x7f369c190920 2021-02-10T11:15:02.353 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.351 7f36a149b700 1 --2- 172.21.2.112:0/1176739540 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f369c193270 0x7f369c193670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:15:02.353 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.351 7f369bfff700 1 --2- 172.21.2.112:0/1176739540 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f369c190d20 0x7f369c08fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:15:02.353 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.351 7f36a0c9a700 1 --2- 172.21.2.112:0/1176739540 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f369c190920 0x7f369c08f600 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f368400a570 tx=0x7f368400a910).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:02.353 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.352 7f36997fa700 1 -- 172.21.2.112:0/1176739540 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f368400ad00 con 0x7f369c190920 2021-02-10T11:15:02.353 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.352 7f36997fa700 1 -- 172.21.2.112:0/1176739540 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3684001180 con 0x7f369c190920 2021-02-10T11:15:02.353 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.352 7f36a2e7e700 1 -- 172.21.2.112:0/1176739540 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f369c194090 con 0x7f369c190920 2021-02-10T11:15:02.354 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.352 7f36997fa700 1 -- 172.21.2.112:0/1176739540 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3684004590 con 0x7f369c190920 2021-02-10T11:15:02.354 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.352 7f36a2e7e700 1 -- 172.21.2.112:0/1176739540 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f369c1920b0 con 0x7f369c190920 2021-02-10T11:15:02.354 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.352 7f36997fa700 1 -- 172.21.2.112:0/1176739540 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f36840043a0 con 0x7f369c190920 2021-02-10T11:15:02.354 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.353 7f36997fa700 1 --2- 172.21.2.112:0/1176739540 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f36800186c0 0x7f368001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:02.354 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.353 7f36997fa700 1 -- 172.21.2.112:0/1176739540 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f3684032590 con 0x7f369c190920 2021-02-10T11:15:02.354 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.353 7f369bfff700 1 --2- 172.21.2.112:0/1176739540 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f36800186c0 0x7f368001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:02.354 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.354 7f369bfff700 1 --2- 172.21.2.112:0/1176739540 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f36800186c0 0x7f368001abc0 crc :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:02.381 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.383 7f36a2e7e700 1 -- 172.21.2.112:0/1176739540 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f369c105af0 con 0x7f369c190920 2021-02-10T11:15:02.383 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.385 7f36997fa700 1 -- 172.21.2.112:0/1176739540 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3684030d40 con 0x7f369c190920 2021-02-10T11:15:02.512 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.514 7f36a2e7e700 1 -- 172.21.2.112:0/1176739540 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f369c0f0b50 con 0x7f369c190920 2021-02-10T11:15:02.512 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.514 7f36997fa700 1 -- 172.21.2.112:0/1176739540 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v20) v1 ==== 100+0+10 (secure 0 0 0) 0x7f368400ad00 con 0x7f369c190920 2021-02-10T11:15:02.513 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:15:02.514 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.516 7f36a2e7e700 1 -- 172.21.2.112:0/1176739540 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f36800186c0 msgr2=0x7f368001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:02.514 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.516 7f36a2e7e700 1 --2- 172.21.2.112:0/1176739540 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f36800186c0 0x7f368001abc0 crc :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:02.514 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.516 7f36a2e7e700 1 -- 172.21.2.112:0/1176739540 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f369c190920 msgr2=0x7f369c08f600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:02.514 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.516 7f36a2e7e700 1 --2- 172.21.2.112:0/1176739540 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f369c190920 0x7f369c08f600 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f368400a570 tx=0x7f368400a910).stop 2021-02-10T11:15:02.515 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.516 7f36a2e7e700 1 -- 172.21.2.112:0/1176739540 shutdown_connections 2021-02-10T11:15:02.515 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.516 7f36a2e7e700 1 --2- 172.21.2.112:0/1176739540 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f36800186c0 0x7f368001abc0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:02.515 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.516 7f36a2e7e700 1 --2- 172.21.2.112:0/1176739540 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f369c190d20 0x7f369c08fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:02.515 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.516 7f36a2e7e700 1 --2- 172.21.2.112:0/1176739540 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f369c193270 0x7f369c193670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:02.515 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.516 7f36a2e7e700 1 --2- 172.21.2.112:0/1176739540 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f369c190920 0x7f369c08f600 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:02.515 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.516 7f36a2e7e700 1 -- 172.21.2.112:0/1176739540 >> 172.21.2.112:0/1176739540 conn(0x7f369c0bf4e0 msgr2=0x7f369c0be3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:02.515 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.516 7f36a2e7e700 1 -- 172.21.2.112:0/1176739540 shutdown_connections 2021-02-10T11:15:02.515 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.516 7f36a2e7e700 1 -- 172.21.2.112:0/1176739540 wait complete. 2021-02-10T11:15:02.523 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:15:02.947 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.932 7f3b9a0bc700 1 Processor -- start 2021-02-10T11:15:02.947 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.932 7f3b9a0bc700 1 -- start start 2021-02-10T11:15:02.947 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.932 7f3b9a0bc700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f3b940e5470 0x7f3b9407e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:02.947 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.932 7f3b9a0bc700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3b9407f300 0x7f3b9407f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:02.947 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.932 7f3b9a0bc700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b9407ed80 con 0x7f3b940e5070 2021-02-10T11:15:02.947 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.932 7f3b9a0bc700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b941b4010 con 0x7f3b940e5470 2021-02-10T11:15:02.947 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.932 7f3b9a0bc700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b940d1790 con 0x7f3b9407f300 2021-02-10T11:15:02.947 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.933 7f3b92ffd700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f3b940e5470 0x7f3b9407e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:02.947 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.933 7f3b93fff700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3b9407f300 0x7f3b9407f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:02.947 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.933 7f3b937fe700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7f3b940e5070 0x7f3b941b3820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:36036/0 (socket says 172.21.2.112:36036) 2021-02-10T11:15:02.947 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.933 7f3b937fe700 1 -- 172.21.2.112:0/759616351 learned_addr learned my addr 172.21.2.112:0/759616351 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:15:02.948 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.933 7f3b93fff700 1 -- 172.21.2.112:0/759616351 >> v2:172.21.2.120:3300/0 conn(0x7f3b940e5470 msgr2=0x7f3b9407e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:15:02.948 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.933 7f3b93fff700 1 --2- 172.21.2.112:0/759616351 >> v2:172.21.2.120:3300/0 conn(0x7f3b940e5470 0x7f3b9407e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:02.948 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.933 7f3b93fff700 1 -- 172.21.2.112:0/759616351 >> v1:172.21.2.120:6789/0 conn(0x7f3b940e5070 legacy=0x7f3b941b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:02.948 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.933 7f3b93fff700 1 -- 172.21.2.112:0/759616351 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b940d19b0 con 0x7f3b9407f300 2021-02-10T11:15:02.948 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.933 7f3b93fff700 1 --2- 172.21.2.112:0/759616351 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3b9407f300 0x7f3b9407f700 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f3b8c006e50 tx=0x7f3b8c009e70).ready entity=mon.2 client_cookie=aa795f5d62e52595 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:02.948 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.933 7f3b927fc700 1 -- 172.21.2.112:0/759616351 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3b8c00b510 con 0x7f3b9407f300 2021-02-10T11:15:02.948 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.933 7f3b927fc700 1 -- 172.21.2.112:0/759616351 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b8c006790 con 0x7f3b9407f300 2021-02-10T11:15:02.948 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.933 7f3b927fc700 1 -- 172.21.2.112:0/759616351 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3b8c00b940 con 0x7f3b9407f300 2021-02-10T11:15:02.948 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.934 7f3b9a0bc700 1 -- 172.21.2.112:0/759616351 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3b9407f300 msgr2=0x7f3b9407f700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:02.948 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.934 7f3b9a0bc700 1 --2- 172.21.2.112:0/759616351 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3b9407f300 0x7f3b9407f700 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f3b8c006e50 tx=0x7f3b8c009e70).stop 2021-02-10T11:15:02.948 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.934 7f3b9a0bc700 1 -- 172.21.2.112:0/759616351 shutdown_connections 2021-02-10T11:15:02.948 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.934 7f3b9a0bc700 1 --2- 172.21.2.112:0/759616351 >> v2:172.21.2.120:3300/0 conn(0x7f3b940e5470 0x7f3b9407e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:02.948 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.934 7f3b9a0bc700 1 --2- 172.21.2.112:0/759616351 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3b9407f300 0x7f3b9407f700 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:02.949 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.934 7f3b9a0bc700 1 -- 172.21.2.112:0/759616351 >> 172.21.2.112:0/759616351 conn(0x7f3b940ce690 msgr2=0x7f3b940e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:02.949 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.934 7f3b9a0bc700 1 -- 172.21.2.112:0/759616351 shutdown_connections 2021-02-10T11:15:02.952 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.934 7f3b9a0bc700 1 -- 172.21.2.112:0/759616351 wait complete. 2021-02-10T11:15:02.952 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.948 7f3b9a0bc700 1 Processor -- start 2021-02-10T11:15:02.952 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.949 7f3b9a0bc700 1 -- start start 2021-02-10T11:15:02.954 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.949 7f3b9a0bc700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3b940b1970 0x7f3b940b1d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:02.954 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.949 7f3b9a0bc700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3b9408acb0 0x7f3b9408b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:02.954 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.949 7f3b9a0bc700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3b9408b6d0 0x7f3b941aac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:02.954 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.949 7f3b9a0bc700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b9408bba0 con 0x7f3b9408acb0 2021-02-10T11:15:02.954 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.949 7f3b9a0bc700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b941ab280 con 0x7f3b9408b6d0 2021-02-10T11:15:02.955 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.949 7f3b9a0bc700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b941ab4a0 con 0x7f3b940b1970 2021-02-10T11:15:02.955 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.949 7f3b92ffd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3b9408acb0 0x7f3b9408b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:02.955 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.949 7f3b92ffd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3b9408acb0 0x7f3b9408b0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:43026/0 (socket says 172.21.2.112:43026) 2021-02-10T11:15:02.955 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.949 7f3b92ffd700 1 -- 172.21.2.112:0/1855305282 learned_addr learned my addr 172.21.2.112:0/1855305282 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:15:02.955 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.950 7f3b92ffd700 1 -- 172.21.2.112:0/1855305282 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3b940b1970 msgr2=0x7f3b940b1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:02.955 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.950 7f3b92ffd700 1 --2- 172.21.2.112:0/1855305282 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3b940b1970 0x7f3b940b1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:02.955 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.950 7f3b92ffd700 1 -- 172.21.2.112:0/1855305282 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3b9408b6d0 msgr2=0x7f3b941aac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:02.955 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.950 7f3b92ffd700 1 --2- 172.21.2.112:0/1855305282 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3b9408b6d0 0x7f3b941aac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:02.955 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.950 7f3b92ffd700 1 -- 172.21.2.112:0/1855305282 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b941ab6c0 con 0x7f3b9408acb0 2021-02-10T11:15:02.955 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.950 7f3b92ffd700 1 --2- 172.21.2.112:0/1855305282 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3b9408acb0 0x7f3b9408b0b0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f3b84006f60 tx=0x7f3b8400b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:02.955 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.950 7f3b8bfff700 1 -- 172.21.2.112:0/1855305282 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3b84006bb0 con 0x7f3b9408acb0 2021-02-10T11:15:02.955 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.950 7f3b8bfff700 1 -- 172.21.2.112:0/1855305282 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b84006340 con 0x7f3b9408acb0 2021-02-10T11:15:02.955 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.950 7f3b9a0bc700 1 -- 172.21.2.112:0/1855305282 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b941ab6c0 con 0x7f3b9408acb0 2021-02-10T11:15:02.956 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.950 7f3b9a0bc700 1 -- 172.21.2.112:0/1855305282 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b9408c0a0 con 0x7f3b9408acb0 2021-02-10T11:15:02.956 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.950 7f3b8bfff700 1 -- 172.21.2.112:0/1855305282 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3b84002030 con 0x7f3b9408acb0 2021-02-10T11:15:02.956 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.951 7f3b8bfff700 1 -- 172.21.2.112:0/1855305282 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3b84006b50 con 0x7f3b9408acb0 2021-02-10T11:15:02.956 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.951 7f3b8bfff700 1 --2- 172.21.2.112:0/1855305282 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3b780186c0 0x7f3b7801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:02.956 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.951 7f3b8bfff700 1 -- 172.21.2.112:0/1855305282 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f3b84032500 con 0x7f3b9408acb0 2021-02-10T11:15:02.956 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.951 7f3b937fe700 1 --2- 172.21.2.112:0/1855305282 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3b780186c0 0x7f3b7801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:02.956 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.951 7f3b937fe700 1 --2- 172.21.2.112:0/1855305282 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3b780186c0 0x7f3b7801abc0 crc :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:02.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.982 7f3b9a0bc700 1 -- 172.21.2.112:0/1855305282 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b94107180 con 0x7f3b9408acb0 2021-02-10T11:15:02.982 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:02.984 7f3b8bfff700 1 -- 172.21.2.112:0/1855305282 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3b84029860 con 0x7f3b9408acb0 2021-02-10T11:15:03.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:03.107 7f3b9a0bc700 1 -- 172.21.2.112:0/1855305282 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3b9408bb20 con 0x7f3b9408acb0 2021-02-10T11:15:03.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:03.108 7f3b8bfff700 1 -- 172.21.2.112:0/1855305282 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+13993 (secure 0 0 0) 0x7f3b84029860 con 0x7f3b9408acb0 2021-02-10T11:15:03.107 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:15:03.107 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":20,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:14:04.629689","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:14:01.400797","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2021-02-10 11:14:02.622129","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:15:03.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:03.110 7f3b9a0bc700 1 -- 172.21.2.112:0/1855305282 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3b780186c0 msgr2=0x7f3b7801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:03.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:03.110 7f3b9a0bc700 1 --2- 172.21.2.112:0/1855305282 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3b780186c0 0x7f3b7801abc0 crc :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:03.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:03.110 7f3b9a0bc700 1 -- 172.21.2.112:0/1855305282 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3b9408acb0 msgr2=0x7f3b9408b0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:03.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:03.110 7f3b9a0bc700 1 --2- 172.21.2.112:0/1855305282 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3b9408acb0 0x7f3b9408b0b0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f3b84006f60 tx=0x7f3b8400b970).stop 2021-02-10T11:15:03.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:03.110 7f3b9a0bc700 1 -- 172.21.2.112:0/1855305282 shutdown_connections 2021-02-10T11:15:03.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:03.110 7f3b9a0bc700 1 --2- 172.21.2.112:0/1855305282 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3b780186c0 0x7f3b7801abc0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:03.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:03.110 7f3b9a0bc700 1 --2- 172.21.2.112:0/1855305282 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3b9408b6d0 0x7f3b941aac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:03.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:03.110 7f3b9a0bc700 1 --2- 172.21.2.112:0/1855305282 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3b9408acb0 0x7f3b9408b0b0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:03.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:03.110 7f3b9a0bc700 1 --2- 172.21.2.112:0/1855305282 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3b940b1970 0x7f3b940b1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:03.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:03.110 7f3b9a0bc700 1 -- 172.21.2.112:0/1855305282 >> 172.21.2.112:0/1855305282 conn(0x7f3b941ac620 msgr2=0x7f3b940893d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:03.109 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:03.110 7f3b9a0bc700 1 -- 172.21.2.112:0/1855305282 shutdown_connections 2021-02-10T11:15:03.109 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:03.110 7f3b9a0bc700 1 -- 172.21.2.112:0/1855305282 wait complete. 2021-02-10T11:15:03.116 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:13:57.508758', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:14:01.400797', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:14:02.622129', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:15:03.117 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:15:03.546 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:03.531 7f063c64b700 1 Processor -- start 2021-02-10T11:15:03.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:03.531 7f063c64b700 1 -- start start 2021-02-10T11:15:03.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:03.532 7f063c64b700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f06341b0470 0x7f06341b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:03.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:03.532 7f063c64b700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f06340f20e0 0x7f06340f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:03.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:03.532 7f063c64b700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06340f0b50 con 0x7f06340ef610 2021-02-10T11:15:03.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:03.532 7f063c64b700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06340f0820 con 0x7f06341b0470 2021-02-10T11:15:03.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:03.532 7f063c64b700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06340f24e0 con 0x7f06340f20e0 2021-02-10T11:15:03.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:03.532 7f0639c66700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f06340ef610 0x7f06340efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:42492/0 (socket says 172.21.2.112:42492) 2021-02-10T11:15:03.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:03.532 7f0639c66700 1 -- 172.21.2.112:0/1278599938 learned_addr learned my addr 172.21.2.112:0/1278599938 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:15:03.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:03.532 7f063a467700 1 --2- 172.21.2.112:0/1278599938 >> v2:172.21.2.112:3300/0 conn(0x7f06341b0470 0x7f06341b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:03.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:03.532 7f063ac68700 1 --2- 172.21.2.112:0/1278599938 >> v2:172.21.2.120:3300/0 conn(0x7f06340f20e0 0x7f06340f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:03.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:03.532 7f0639465700 1 -- 172.21.2.112:0/1278599938 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1861641408 0 0) 0x7f06300009b0 con 0x7f06340ef610 2021-02-10T11:15:03.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:03.532 7f063ac68700 1 -- 172.21.2.112:0/1278599938 >> v2:172.21.2.112:3300/0 conn(0x7f06341b0470 msgr2=0x7f06341b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:15:03.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:03.532 7f063ac68700 1 --2- 172.21.2.112:0/1278599938 >> v2:172.21.2.112:3300/0 conn(0x7f06341b0470 0x7f06341b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:03.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:03.532 7f063ac68700 1 -- 172.21.2.112:0/1278599938 >> v1:172.21.2.112:6789/0 conn(0x7f06340ef610 legacy=0x7f06340efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:03.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:03.532 7f063ac68700 1 -- 172.21.2.112:0/1278599938 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06340e52e0 con 0x7f06340f20e0 2021-02-10T11:15:03.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:03.533 7f063ac68700 1 --2- 172.21.2.112:0/1278599938 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f06340f20e0 0x7f06340f0200 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f062c006e50 tx=0x7f062c009e70).ready entity=mon.1 client_cookie=8cff2a62d47545ca server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:03.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:03.533 7f0639465700 1 -- 172.21.2.112:0/1278599938 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f062c00b510 con 0x7f06340f20e0 2021-02-10T11:15:03.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:03.533 7f0639465700 1 -- 172.21.2.112:0/1278599938 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f062c006790 con 0x7f06340f20e0 2021-02-10T11:15:03.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:03.533 7f0639465700 1 -- 172.21.2.112:0/1278599938 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f062c00b940 con 0x7f06340f20e0 2021-02-10T11:15:03.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:03.533 7f063c64b700 1 -- 172.21.2.112:0/1278599938 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f06340f20e0 msgr2=0x7f06340f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:03.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:03.533 7f063c64b700 1 --2- 172.21.2.112:0/1278599938 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f06340f20e0 0x7f06340f0200 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f062c006e50 tx=0x7f062c009e70).stop 2021-02-10T11:15:03.549 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:03.534 7f063c64b700 1 -- 172.21.2.112:0/1278599938 shutdown_connections 2021-02-10T11:15:03.549 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:03.534 7f063c64b700 1 --2- 172.21.2.112:0/1278599938 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f06340f20e0 0x7f06340f0200 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:03.549 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:03.534 7f063c64b700 1 --2- 172.21.2.112:0/1278599938 >> v2:172.21.2.112:3300/0 conn(0x7f06341b0470 0x7f06341b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:03.549 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:03.534 7f063c64b700 1 -- 172.21.2.112:0/1278599938 >> 172.21.2.112:0/1278599938 conn(0x7f06340ce3e0 msgr2=0x7f06340e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:03.549 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:03.534 7f063c64b700 1 -- 172.21.2.112:0/1278599938 shutdown_connections 2021-02-10T11:15:03.549 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:03.534 7f063c64b700 1 -- 172.21.2.112:0/1278599938 wait complete. 2021-02-10T11:15:03.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:03.549 7f063c64b700 1 Processor -- start 2021-02-10T11:15:03.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:03.549 7f063c64b700 1 -- start start 2021-02-10T11:15:03.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:03.549 7f063c64b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0634190920 0x7f06340b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:03.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:03.549 7f063c64b700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f06340b8200 0x7f06340b8600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:03.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:03.549 7f063c64b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f06340c1d80 0x7f06340c2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:03.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:03.549 7f063c64b700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0634190d20 con 0x7f0634190920 2021-02-10T11:15:03.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:03.549 7f063c64b700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06340b8c20 con 0x7f06340b8200 2021-02-10T11:15:03.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:03.549 7f063c64b700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06340c27a0 con 0x7f06340c1d80 2021-02-10T11:15:03.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:03.549 7f063a467700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0634190920 0x7f06340b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:03.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:03.549 7f063a467700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0634190920 0x7f06340b7be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:43034/0 (socket says 172.21.2.112:43034) 2021-02-10T11:15:03.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:03.549 7f063a467700 1 -- 172.21.2.112:0/3864141211 learned_addr learned my addr 172.21.2.112:0/3864141211 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:15:03.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:03.549 7f063ac68700 1 --2- 172.21.2.112:0/3864141211 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f06340c1d80 0x7f06340c2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:03.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:03.550 7f0639c66700 1 --2- 172.21.2.112:0/3864141211 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f06340b8200 0x7f06340b8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:03.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:03.550 7f063a467700 1 -- 172.21.2.112:0/3864141211 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f06340c1d80 msgr2=0x7f06340c2180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:03.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:03.550 7f063a467700 1 --2- 172.21.2.112:0/3864141211 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f06340c1d80 0x7f06340c2180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:03.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:03.550 7f063a467700 1 -- 172.21.2.112:0/3864141211 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f06340b8200 msgr2=0x7f06340b8600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:03.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:03.550 7f063a467700 1 --2- 172.21.2.112:0/3864141211 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f06340b8200 0x7f06340b8600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:03.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:03.550 7f063a467700 1 -- 172.21.2.112:0/3864141211 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06340c29c0 con 0x7f0634190920 2021-02-10T11:15:03.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:03.550 7f063a467700 1 --2- 172.21.2.112:0/3864141211 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0634190920 0x7f06340b7be0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f0628006f60 tx=0x7f062800b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:03.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:03.550 7f0626ffd700 1 -- 172.21.2.112:0/3864141211 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f06280009b0 con 0x7f0634190920 2021-02-10T11:15:03.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:03.550 7f0626ffd700 1 -- 172.21.2.112:0/3864141211 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f062800bd60 con 0x7f0634190920 2021-02-10T11:15:03.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:03.550 7f0626ffd700 1 -- 172.21.2.112:0/3864141211 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0628002030 con 0x7f0634190920 2021-02-10T11:15:03.553 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:03.550 7f063c64b700 1 -- 172.21.2.112:0/3864141211 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06340c29c0 con 0x7f0634190920 2021-02-10T11:15:03.553 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:03.550 7f063c64b700 1 -- 172.21.2.112:0/3864141211 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06340c2f50 con 0x7f0634190920 2021-02-10T11:15:03.553 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:03.551 7f0626ffd700 1 -- 172.21.2.112:0/3864141211 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0628006b00 con 0x7f0634190920 2021-02-10T11:15:03.553 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:03.551 7f0626ffd700 1 --2- 172.21.2.112:0/3864141211 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f061c0186c0 0x7f061c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:03.553 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:03.551 7f0626ffd700 1 -- 172.21.2.112:0/3864141211 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f0628032570 con 0x7f0634190920 2021-02-10T11:15:03.553 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:03.552 7f0639c66700 1 --2- 172.21.2.112:0/3864141211 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f061c0186c0 0x7f061c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:03.553 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:03.552 7f0639c66700 1 --2- 172.21.2.112:0/3864141211 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f061c0186c0 0x7f061c01abc0 crc :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:03.580 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:03.582 7f063c64b700 1 -- 172.21.2.112:0/3864141211 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0634105ae0 con 0x7f0634190920 2021-02-10T11:15:03.582 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:03.584 7f0626ffd700 1 -- 172.21.2.112:0/3864141211 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f062802d0b0 con 0x7f0634190920 2021-02-10T11:15:03.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:03.711 7f063c64b700 1 -- 172.21.2.112:0/3864141211 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f06340f0b50 con 0x7f0634190920 2021-02-10T11:15:03.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:03.711 7f0626ffd700 1 -- 172.21.2.112:0/3864141211 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v20) v1 ==== 92+0+10 (secure 0 0 0) 0x7f062802ded0 con 0x7f0634190920 2021-02-10T11:15:03.710 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:15:03.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:03.713 7f063c64b700 1 -- 172.21.2.112:0/3864141211 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f061c0186c0 msgr2=0x7f061c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:03.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:03.713 7f063c64b700 1 --2- 172.21.2.112:0/3864141211 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f061c0186c0 0x7f061c01abc0 crc :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:03.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:03.713 7f063c64b700 1 -- 172.21.2.112:0/3864141211 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0634190920 msgr2=0x7f06340b7be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:03.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:03.713 7f063c64b700 1 --2- 172.21.2.112:0/3864141211 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0634190920 0x7f06340b7be0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f0628006f60 tx=0x7f062800b970).stop 2021-02-10T11:15:03.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:03.713 7f063c64b700 1 -- 172.21.2.112:0/3864141211 shutdown_connections 2021-02-10T11:15:03.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:03.713 7f063c64b700 1 --2- 172.21.2.112:0/3864141211 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f061c0186c0 0x7f061c01abc0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:03.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:03.713 7f063c64b700 1 --2- 172.21.2.112:0/3864141211 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f06340b8200 0x7f06340b8600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:03.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:03.713 7f063c64b700 1 --2- 172.21.2.112:0/3864141211 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f06340c1d80 0x7f06340c2180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:03.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:03.713 7f063c64b700 1 --2- 172.21.2.112:0/3864141211 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0634190920 0x7f06340b7be0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:03.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:03.713 7f063c64b700 1 -- 172.21.2.112:0/3864141211 >> 172.21.2.112:0/3864141211 conn(0x7f0634094e10 msgr2=0x7f06340a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:03.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:03.714 7f063c64b700 1 -- 172.21.2.112:0/3864141211 shutdown_connections 2021-02-10T11:15:03.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:03.714 7f063c64b700 1 -- 172.21.2.112:0/3864141211 wait complete. 2021-02-10T11:15:03.720 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:15:04.147 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.133 7fda6a910700 1 Processor -- start 2021-02-10T11:15:04.147 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.133 7fda6a910700 1 -- start start 2021-02-10T11:15:04.147 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.133 7fda6a910700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fda641b0470 0x7fda641b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:04.147 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.133 7fda6a910700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fda640ef610 0x7fda640efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:04.148 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.133 7fda6a910700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fda640f0030 0x7fda640f0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:04.148 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.133 7fda6a910700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda640e5ea0 con 0x7fda640ef610 2021-02-10T11:15:04.148 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.133 7fda6a910700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda640f0b50 con 0x7fda640f0030 2021-02-10T11:15:04.148 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.133 7fda6a910700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda640e52e0 con 0x7fda641b0470 2021-02-10T11:15:04.148 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.133 7fda637fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fda640ef610 0x7fda640efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:04.148 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.133 7fda637fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fda640ef610 0x7fda640efa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:43042/0 (socket says 172.21.2.112:43042) 2021-02-10T11:15:04.148 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.133 7fda637fe700 1 -- 172.21.2.112:0/2007589953 learned_addr learned my addr 172.21.2.112:0/2007589953 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:15:04.148 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.134 7fda637fe700 1 -- 172.21.2.112:0/2007589953 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fda641b0470 msgr2=0x7fda641b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:15:04.148 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.134 7fda637fe700 1 --2- 172.21.2.112:0/2007589953 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fda641b0470 0x7fda641b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:04.148 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.134 7fda637fe700 1 -- 172.21.2.112:0/2007589953 >> v2:172.21.2.120:3300/0 conn(0x7fda640f0030 msgr2=0x7fda640f0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:15:04.149 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.134 7fda637fe700 1 --2- 172.21.2.112:0/2007589953 >> v2:172.21.2.120:3300/0 conn(0x7fda640f0030 0x7fda640f0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:04.149 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.134 7fda637fe700 1 -- 172.21.2.112:0/2007589953 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda640f10d0 con 0x7fda640ef610 2021-02-10T11:15:04.149 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.134 7fda637fe700 1 --2- 172.21.2.112:0/2007589953 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fda640ef610 0x7fda640efa10 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7fda58006700 tx=0x7fda58009b80).ready entity=mon.0 client_cookie=dae8aecad8b7d8b7 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:04.149 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.134 7fda62ffd700 1 -- 172.21.2.112:0/2007589953 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fda5800c0c0 con 0x7fda640ef610 2021-02-10T11:15:04.149 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.134 7fda62ffd700 1 -- 172.21.2.112:0/2007589953 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda5800c400 con 0x7fda640ef610 2021-02-10T11:15:04.149 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.134 7fda62ffd700 1 -- 172.21.2.112:0/2007589953 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fda5800c8e0 con 0x7fda640ef610 2021-02-10T11:15:04.150 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.134 7fda6a910700 1 -- 172.21.2.112:0/2007589953 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fda640ef610 msgr2=0x7fda640efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:04.150 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.134 7fda6a910700 1 --2- 172.21.2.112:0/2007589953 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fda640ef610 0x7fda640efa10 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7fda58006700 tx=0x7fda58009b80).stop 2021-02-10T11:15:04.150 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.134 7fda6a910700 1 -- 172.21.2.112:0/2007589953 shutdown_connections 2021-02-10T11:15:04.150 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.135 7fda6a910700 1 --2- 172.21.2.112:0/2007589953 >> v2:172.21.2.120:3300/0 conn(0x7fda640f0030 0x7fda640f0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:04.150 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.135 7fda6a910700 1 --2- 172.21.2.112:0/2007589953 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fda640ef610 0x7fda640efa10 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:04.150 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.135 7fda6a910700 1 --2- 172.21.2.112:0/2007589953 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fda641b0470 0x7fda641b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:04.150 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.135 7fda6a910700 1 -- 172.21.2.112:0/2007589953 >> 172.21.2.112:0/2007589953 conn(0x7fda640ce3e0 msgr2=0x7fda640e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:04.150 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.135 7fda6a910700 1 -- 172.21.2.112:0/2007589953 shutdown_connections 2021-02-10T11:15:04.150 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.135 7fda6a910700 1 -- 172.21.2.112:0/2007589953 wait complete. 2021-02-10T11:15:04.152 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.149 7fda6a910700 1 Processor -- start 2021-02-10T11:15:04.152 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.149 7fda6a910700 1 -- start start 2021-02-10T11:15:04.152 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.150 7fda6a910700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fda64193230 0x7fda640a6a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:04.152 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.150 7fda6a910700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fda6408f5b0 0x7fda6408f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:04.152 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.150 7fda6a910700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fda6408e6a0 0x7fda6408eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:04.152 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.150 7fda6a910700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda6408ffd0 con 0x7fda6408f5b0 2021-02-10T11:15:04.152 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.150 7fda6a910700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda64090240 con 0x7fda6408e6a0 2021-02-10T11:15:04.152 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.150 7fda6a910700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda6408f0c0 con 0x7fda64193230 2021-02-10T11:15:04.152 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.150 7fda637fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fda6408f5b0 0x7fda6408f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:04.153 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.150 7fda637fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fda6408f5b0 0x7fda6408f9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:43048/0 (socket says 172.21.2.112:43048) 2021-02-10T11:15:04.153 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.150 7fda637fe700 1 -- 172.21.2.112:0/4209965752 learned_addr learned my addr 172.21.2.112:0/4209965752 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:15:04.153 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.150 7fda637fe700 1 -- 172.21.2.112:0/4209965752 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fda64193230 msgr2=0x7fda640a6a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:04.153 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.150 7fda637fe700 1 --2- 172.21.2.112:0/4209965752 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fda64193230 0x7fda640a6a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:04.153 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.150 7fda637fe700 1 -- 172.21.2.112:0/4209965752 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fda6408e6a0 msgr2=0x7fda6408eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:04.153 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.150 7fda637fe700 1 --2- 172.21.2.112:0/4209965752 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fda6408e6a0 0x7fda6408eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:04.153 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.150 7fda637fe700 1 -- 172.21.2.112:0/4209965752 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda6408f2a0 con 0x7fda6408f5b0 2021-02-10T11:15:04.153 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.151 7fda637fe700 1 --2- 172.21.2.112:0/4209965752 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fda6408f5b0 0x7fda6408f9b0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7fda58006420 tx=0x7fda5800b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:04.153 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.151 7fda60ff9700 1 -- 172.21.2.112:0/4209965752 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fda5800bd50 con 0x7fda6408f5b0 2021-02-10T11:15:04.153 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.151 7fda60ff9700 1 -- 172.21.2.112:0/4209965752 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda58009440 con 0x7fda6408f5b0 2021-02-10T11:15:04.153 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.151 7fda60ff9700 1 -- 172.21.2.112:0/4209965752 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fda580054f0 con 0x7fda6408f5b0 2021-02-10T11:15:04.153 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.151 7fda6a910700 1 -- 172.21.2.112:0/4209965752 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda6408f2a0 con 0x7fda6408f5b0 2021-02-10T11:15:04.154 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.151 7fda6a910700 1 -- 172.21.2.112:0/4209965752 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda641924f0 con 0x7fda6408f5b0 2021-02-10T11:15:04.154 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.152 7fda60ff9700 1 -- 172.21.2.112:0/4209965752 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fda580052e0 con 0x7fda6408f5b0 2021-02-10T11:15:04.154 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.152 7fda60ff9700 1 --2- 172.21.2.112:0/4209965752 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fda440186c0 0x7fda4401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:04.154 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.152 7fda60ff9700 1 -- 172.21.2.112:0/4209965752 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fda580325a0 con 0x7fda6408f5b0 2021-02-10T11:15:04.154 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.152 7fda63fff700 1 --2- 172.21.2.112:0/4209965752 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fda440186c0 0x7fda4401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:04.154 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.153 7fda63fff700 1 --2- 172.21.2.112:0/4209965752 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fda440186c0 0x7fda4401abc0 crc :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:04.181 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.183 7fda6a910700 1 -- 172.21.2.112:0/4209965752 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda640f0b50 con 0x7fda6408f5b0 2021-02-10T11:15:04.184 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.186 7fda60ff9700 1 -- 172.21.2.112:0/4209965752 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fda580090d0 con 0x7fda6408f5b0 2021-02-10T11:15:04.311 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.312 7fda6a910700 1 -- 172.21.2.112:0/4209965752 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7fda640f0b50 con 0x7fda6408f5b0 2021-02-10T11:15:04.311 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.313 7fda60ff9700 1 -- 172.21.2.112:0/4209965752 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v20) v1 ==== 104+0+10 (secure 0 0 0) 0x7fda5800bd50 con 0x7fda6408f5b0 2021-02-10T11:15:04.312 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:15:04.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.315 7fda6a910700 1 -- 172.21.2.112:0/4209965752 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fda440186c0 msgr2=0x7fda4401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:04.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.315 7fda6a910700 1 --2- 172.21.2.112:0/4209965752 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fda440186c0 0x7fda4401abc0 crc :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:04.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.315 7fda6a910700 1 -- 172.21.2.112:0/4209965752 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fda6408f5b0 msgr2=0x7fda6408f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:04.314 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.315 7fda6a910700 1 --2- 172.21.2.112:0/4209965752 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fda6408f5b0 0x7fda6408f9b0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7fda58006420 tx=0x7fda5800b960).stop 2021-02-10T11:15:04.314 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.315 7fda6a910700 1 -- 172.21.2.112:0/4209965752 shutdown_connections 2021-02-10T11:15:04.314 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.315 7fda6a910700 1 --2- 172.21.2.112:0/4209965752 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fda440186c0 0x7fda4401abc0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:04.314 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.315 7fda6a910700 1 --2- 172.21.2.112:0/4209965752 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fda6408e6a0 0x7fda6408eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:04.314 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.315 7fda6a910700 1 --2- 172.21.2.112:0/4209965752 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fda6408f5b0 0x7fda6408f9b0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:04.314 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.315 7fda6a910700 1 --2- 172.21.2.112:0/4209965752 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fda64193230 0x7fda640a6a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:04.314 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.315 7fda6a910700 1 -- 172.21.2.112:0/4209965752 >> 172.21.2.112:0/4209965752 conn(0x7fda640bf4e0 msgr2=0x7fda640be160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:04.315 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.315 7fda6a910700 1 -- 172.21.2.112:0/4209965752 shutdown_connections 2021-02-10T11:15:04.315 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.315 7fda6a910700 1 -- 172.21.2.112:0/4209965752 wait complete. 2021-02-10T11:15:04.321 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:15:04.751 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.736 7fe3c2222700 1 Processor -- start 2021-02-10T11:15:04.751 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.736 7fe3c2222700 1 -- start start 2021-02-10T11:15:04.751 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.737 7fe3c2222700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fe3bc1b0470 0x7fe3bc1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:04.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.737 7fe3c2222700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fe3bc0ef610 0x7fe3bc0efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:04.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.737 7fe3c2222700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3bc0f0b50 con 0x7fe3bc0f0030 2021-02-10T11:15:04.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.737 7fe3c2222700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3bc0e52e0 con 0x7fe3bc0ef610 2021-02-10T11:15:04.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.737 7fe3c2222700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3bc0e5490 con 0x7fe3bc1b0470 2021-02-10T11:15:04.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.737 7fe3bb7fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fe3bc0ef610 0x7fe3bc0efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:04.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.737 7fe3c083f700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fe3bc0f0030 0x7fe3bc0f10d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:42516/0 (socket says 172.21.2.112:42516) 2021-02-10T11:15:04.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.737 7fe3c083f700 1 -- 172.21.2.112:0/3450422832 learned_addr learned my addr 172.21.2.112:0/3450422832 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:15:04.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.737 7fe3bb7fe700 1 -- 172.21.2.112:0/3450422832 >> v2:172.21.2.120:3300/0 conn(0x7fe3bc1b0470 msgr2=0x7fe3bc1b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:15:04.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.737 7fe3bb7fe700 1 --2- 172.21.2.112:0/3450422832 >> v2:172.21.2.120:3300/0 conn(0x7fe3bc1b0470 0x7fe3bc1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:04.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.737 7fe3bb7fe700 1 -- 172.21.2.112:0/3450422832 >> v1:172.21.2.112:6789/0 conn(0x7fe3bc0f0030 legacy=0x7fe3bc0f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:04.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.738 7fe3bb7fe700 1 -- 172.21.2.112:0/3450422832 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3bc0f0430 con 0x7fe3bc0ef610 2021-02-10T11:15:04.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.738 7fe3bb7fe700 1 --2- 172.21.2.112:0/3450422832 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe3bc0ef610 0x7fe3bc0efa10 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7fe3b0006650 tx=0x7fe3b0009a00).ready entity=mon.0 client_cookie=a18cc51ebb924937 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:04.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.738 7fe3baffd700 1 -- 172.21.2.112:0/3450422832 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe3b0009df0 con 0x7fe3bc0ef610 2021-02-10T11:15:04.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.738 7fe3baffd700 1 -- 172.21.2.112:0/3450422832 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3b00009b0 con 0x7fe3bc0ef610 2021-02-10T11:15:04.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.738 7fe3baffd700 1 -- 172.21.2.112:0/3450422832 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe3b000c2b0 con 0x7fe3bc0ef610 2021-02-10T11:15:04.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.738 7fe3c2222700 1 -- 172.21.2.112:0/3450422832 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe3bc0ef610 msgr2=0x7fe3bc0efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:04.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.738 7fe3c2222700 1 --2- 172.21.2.112:0/3450422832 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe3bc0ef610 0x7fe3bc0efa10 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7fe3b0006650 tx=0x7fe3b0009a00).stop 2021-02-10T11:15:04.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.739 7fe3c2222700 1 -- 172.21.2.112:0/3450422832 shutdown_connections 2021-02-10T11:15:04.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.739 7fe3c2222700 1 --2- 172.21.2.112:0/3450422832 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe3bc0ef610 0x7fe3bc0efa10 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:04.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.739 7fe3c2222700 1 --2- 172.21.2.112:0/3450422832 >> v2:172.21.2.120:3300/0 conn(0x7fe3bc1b0470 0x7fe3bc1b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:04.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.739 7fe3c2222700 1 -- 172.21.2.112:0/3450422832 >> 172.21.2.112:0/3450422832 conn(0x7fe3bc0ce3e0 msgr2=0x7fe3bc0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:04.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.739 7fe3c2222700 1 -- 172.21.2.112:0/3450422832 shutdown_connections 2021-02-10T11:15:04.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.739 7fe3c2222700 1 -- 172.21.2.112:0/3450422832 wait complete. 2021-02-10T11:15:04.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.753 7fe3c2222700 1 Processor -- start 2021-02-10T11:15:04.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.754 7fe3c2222700 1 -- start start 2021-02-10T11:15:04.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.754 7fe3c2222700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe3bc0c36a0 0x7fe3bc0c2b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:04.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.754 7fe3c2222700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe3bc0c3190 0x7fe3bc0b1d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:04.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.754 7fe3c2222700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe3bc0b2370 0x7fe3bc0b2d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:04.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.754 7fe3c2222700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3bc0b27c0 con 0x7fe3bc0c3190 2021-02-10T11:15:04.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.754 7fe3c2222700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3bc0b3360 con 0x7fe3bc0c36a0 2021-02-10T11:15:04.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.754 7fe3c2222700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3bc0b3540 con 0x7fe3bc0b2370 2021-02-10T11:15:04.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.754 7fe3bb7fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe3bc0c3190 0x7fe3bc0b1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:04.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.754 7fe3bb7fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe3bc0c3190 0x7fe3bc0b1d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:43056/0 (socket says 172.21.2.112:43056) 2021-02-10T11:15:04.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.754 7fe3bb7fe700 1 -- 172.21.2.112:0/844437285 learned_addr learned my addr 172.21.2.112:0/844437285 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:15:04.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.755 7fe3bb7fe700 1 -- 172.21.2.112:0/844437285 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe3bc0b2370 msgr2=0x7fe3bc0b2d40 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:15:04.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.755 7fe3bb7fe700 1 --2- 172.21.2.112:0/844437285 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe3bc0b2370 0x7fe3bc0b2d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:04.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.755 7fe3bb7fe700 1 -- 172.21.2.112:0/844437285 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe3bc0c36a0 msgr2=0x7fe3bc0c2b70 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:15:04.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.755 7fe3bb7fe700 1 --2- 172.21.2.112:0/844437285 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe3bc0c36a0 0x7fe3bc0c2b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:04.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.755 7fe3bb7fe700 1 -- 172.21.2.112:0/844437285 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3bc0b3760 con 0x7fe3bc0c3190 2021-02-10T11:15:04.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.755 7fe3bb7fe700 1 --2- 172.21.2.112:0/844437285 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe3bc0c3190 0x7fe3bc0b1d50 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7fe3b000b580 tx=0x7fe3b000b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:04.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.755 7fe3b8ff9700 1 -- 172.21.2.112:0/844437285 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe3b0000c90 con 0x7fe3bc0c3190 2021-02-10T11:15:04.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.755 7fe3b8ff9700 1 -- 172.21.2.112:0/844437285 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3b0002030 con 0x7fe3bc0c3190 2021-02-10T11:15:04.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.755 7fe3b8ff9700 1 -- 172.21.2.112:0/844437285 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe3b0002550 con 0x7fe3bc0c3190 2021-02-10T11:15:04.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.755 7fe3c2222700 1 -- 172.21.2.112:0/844437285 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3bc0b3760 con 0x7fe3bc0c3190 2021-02-10T11:15:04.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.755 7fe3c2222700 1 -- 172.21.2.112:0/844437285 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3bc0b3760 con 0x7fe3bc0c3190 2021-02-10T11:15:04.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.756 7fe3b8ff9700 1 -- 172.21.2.112:0/844437285 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe3b0000c90 con 0x7fe3bc0c3190 2021-02-10T11:15:04.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.756 7fe3b8ff9700 1 --2- 172.21.2.112:0/844437285 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe39c0186c0 0x7fe39c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:04.757 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.756 7fe3b8ff9700 1 -- 172.21.2.112:0/844437285 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fe3b0032850 con 0x7fe3bc0c3190 2021-02-10T11:15:04.757 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.757 7fe3bbfff700 1 --2- 172.21.2.112:0/844437285 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe39c0186c0 0x7fe39c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:04.757 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.757 7fe3bbfff700 1 --2- 172.21.2.112:0/844437285 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe39c0186c0 0x7fe39c01abc0 crc :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:04.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.787 7fe3c2222700 1 -- 172.21.2.112:0/844437285 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3bc0edd10 con 0x7fe3bc0c3190 2021-02-10T11:15:04.787 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.789 7fe3b8ff9700 1 -- 172.21.2.112:0/844437285 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe3b0000940 con 0x7fe3bc0c3190 2021-02-10T11:15:04.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.917 7fe3c2222700 1 -- 172.21.2.112:0/844437285 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7fe3bc0e52e0 con 0x7fe3bc0c3190 2021-02-10T11:15:04.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.917 7fe3b8ff9700 1 -- 172.21.2.112:0/844437285 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v20) v1 ==== 100+0+10 (secure 0 0 0) 0x7fe3b0000940 con 0x7fe3bc0c3190 2021-02-10T11:15:04.916 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:15:04.917 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.919 7fe3c2222700 1 -- 172.21.2.112:0/844437285 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe39c0186c0 msgr2=0x7fe39c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:04.917 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.919 7fe3c2222700 1 --2- 172.21.2.112:0/844437285 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe39c0186c0 0x7fe39c01abc0 crc :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:04.918 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.919 7fe3c2222700 1 -- 172.21.2.112:0/844437285 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe3bc0c3190 msgr2=0x7fe3bc0b1d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:04.918 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.919 7fe3c2222700 1 --2- 172.21.2.112:0/844437285 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe3bc0c3190 0x7fe3bc0b1d50 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7fe3b000b580 tx=0x7fe3b000b920).stop 2021-02-10T11:15:04.918 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.920 7fe3c2222700 1 -- 172.21.2.112:0/844437285 shutdown_connections 2021-02-10T11:15:04.918 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.920 7fe3c2222700 1 --2- 172.21.2.112:0/844437285 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe39c0186c0 0x7fe39c01abc0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:04.918 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.920 7fe3c2222700 1 --2- 172.21.2.112:0/844437285 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe3bc0b2370 0x7fe3bc0b2d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:04.918 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.920 7fe3c2222700 1 --2- 172.21.2.112:0/844437285 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe3bc0c3190 0x7fe3bc0b1d50 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:04.918 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.920 7fe3c2222700 1 --2- 172.21.2.112:0/844437285 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe3bc0c36a0 0x7fe3bc0c2b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:04.918 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.920 7fe3c2222700 1 -- 172.21.2.112:0/844437285 >> 172.21.2.112:0/844437285 conn(0x7fe3bc09f630 msgr2=0x7fe3bc0a0130 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:04.919 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.920 7fe3c2222700 1 -- 172.21.2.112:0/844437285 shutdown_connections 2021-02-10T11:15:04.919 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:04.920 7fe3c2222700 1 -- 172.21.2.112:0/844437285 wait complete. 2021-02-10T11:15:04.927 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:15:05.352 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.337 7fc21fb3b700 1 Processor -- start 2021-02-10T11:15:05.352 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.337 7fc21fb3b700 1 -- start start 2021-02-10T11:15:05.352 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.338 7fc21fb3b700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fc2180e5470 0x7fc21807e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:05.353 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.338 7fc21fb3b700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fc21807f300 0x7fc21807f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:05.353 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.338 7fc21fb3b700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc21807ed80 con 0x7fc2180e5070 2021-02-10T11:15:05.353 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.338 7fc21fb3b700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2181b4010 con 0x7fc2180e5470 2021-02-10T11:15:05.353 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.338 7fc21fb3b700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2180d1790 con 0x7fc21807f300 2021-02-10T11:15:05.353 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.338 7fc21d156700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fc2180e5470 0x7fc21807e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:05.353 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.338 7fc21d156700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fc2180e5470 0x7fc21807e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:43066/0 (socket says 172.21.2.112:43066) 2021-02-10T11:15:05.353 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.338 7fc21d156700 1 -- 172.21.2.112:0/3745442420 learned_addr learned my addr 172.21.2.112:0/3745442420 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:15:05.353 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.338 7fc21d957700 1 --1- 172.21.2.112:0/3745442420 >> v1:172.21.2.112:6789/0 conn(0x7fc2180e5070 0x7fc2181b3820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:42526/0 (socket says 172.21.2.112:42526) 2021-02-10T11:15:05.353 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.338 7fc21e158700 1 --2- 172.21.2.112:0/3745442420 >> v2:172.21.2.120:3300/0 conn(0x7fc21807f300 0x7fc21807f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:05.353 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.338 7fc21d156700 1 -- 172.21.2.112:0/3745442420 >> v2:172.21.2.120:3300/0 conn(0x7fc21807f300 msgr2=0x7fc21807f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:15:05.353 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.338 7fc21d156700 1 --2- 172.21.2.112:0/3745442420 >> v2:172.21.2.120:3300/0 conn(0x7fc21807f300 0x7fc21807f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:05.353 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.338 7fc21d156700 1 -- 172.21.2.112:0/3745442420 >> v1:172.21.2.112:6789/0 conn(0x7fc2180e5070 legacy=0x7fc2181b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:05.353 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.338 7fc21d156700 1 -- 172.21.2.112:0/3745442420 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2180d19b0 con 0x7fc2180e5470 2021-02-10T11:15:05.354 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.339 7fc21d156700 1 --2- 172.21.2.112:0/3745442420 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc2180e5470 0x7fc21807e6a0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7fc214006700 tx=0x7fc214009b80).ready entity=mon.0 client_cookie=84fa861ea30eb443 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:05.354 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.339 7fc21c955700 1 -- 172.21.2.112:0/3745442420 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc21400c0c0 con 0x7fc2180e5470 2021-02-10T11:15:05.354 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.339 7fc21c955700 1 -- 172.21.2.112:0/3745442420 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc21400c340 con 0x7fc2180e5470 2021-02-10T11:15:05.354 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.339 7fc21c955700 1 -- 172.21.2.112:0/3745442420 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc21400c860 con 0x7fc2180e5470 2021-02-10T11:15:05.354 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.339 7fc21fb3b700 1 -- 172.21.2.112:0/3745442420 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc2180e5470 msgr2=0x7fc21807e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:05.354 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.339 7fc21fb3b700 1 --2- 172.21.2.112:0/3745442420 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc2180e5470 0x7fc21807e6a0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7fc214006700 tx=0x7fc214009b80).stop 2021-02-10T11:15:05.354 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.340 7fc21fb3b700 1 -- 172.21.2.112:0/3745442420 shutdown_connections 2021-02-10T11:15:05.354 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.340 7fc21fb3b700 1 --2- 172.21.2.112:0/3745442420 >> v2:172.21.2.120:3300/0 conn(0x7fc21807f300 0x7fc21807f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:05.354 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.340 7fc21fb3b700 1 --2- 172.21.2.112:0/3745442420 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc2180e5470 0x7fc21807e6a0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:05.354 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.340 7fc21fb3b700 1 -- 172.21.2.112:0/3745442420 >> 172.21.2.112:0/3745442420 conn(0x7fc2180ce690 msgr2=0x7fc2180e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:05.354 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.340 7fc21fb3b700 1 -- 172.21.2.112:0/3745442420 shutdown_connections 2021-02-10T11:15:05.354 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.340 7fc21fb3b700 1 -- 172.21.2.112:0/3745442420 wait complete. 2021-02-10T11:15:05.357 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.354 7fc21fb3b700 1 Processor -- start 2021-02-10T11:15:05.357 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.354 7fc21fb3b700 1 -- start start 2021-02-10T11:15:05.357 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.355 7fc21fb3b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc21808d400 0x7fc218081230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:05.357 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.355 7fc21fb3b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc218081850 0x7fc2180acf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:05.357 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.355 7fc21fb3b700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc218081c50 0x7fc2180ad5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:05.357 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.355 7fc21fb3b700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc218082110 con 0x7fc21808d400 2021-02-10T11:15:05.357 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.355 7fc21fb3b700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc21808d800 con 0x7fc218081c50 2021-02-10T11:15:05.357 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.355 7fc21fb3b700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2180adbc0 con 0x7fc218081850 2021-02-10T11:15:05.358 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.355 7fc21d957700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc21808d400 0x7fc218081230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:05.358 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.355 7fc21d957700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc21808d400 0x7fc218081230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:43072/0 (socket says 172.21.2.112:43072) 2021-02-10T11:15:05.358 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.355 7fc21d957700 1 -- 172.21.2.112:0/2181487589 learned_addr learned my addr 172.21.2.112:0/2181487589 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:15:05.358 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.355 7fc21d156700 1 --2- 172.21.2.112:0/2181487589 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc218081850 0x7fc2180acf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:05.358 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.355 7fc21e158700 1 --2- 172.21.2.112:0/2181487589 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc218081c50 0x7fc2180ad5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:05.358 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.356 7fc21d957700 1 -- 172.21.2.112:0/2181487589 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc218081850 msgr2=0x7fc2180acf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:05.358 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.356 7fc21d957700 1 --2- 172.21.2.112:0/2181487589 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc218081850 0x7fc2180acf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:05.358 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.356 7fc21d957700 1 -- 172.21.2.112:0/2181487589 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc218081c50 msgr2=0x7fc2180ad5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:05.358 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.356 7fc21d957700 1 --2- 172.21.2.112:0/2181487589 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc218081c50 0x7fc2180ad5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:05.358 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.356 7fc21d957700 1 -- 172.21.2.112:0/2181487589 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc218187fc0 con 0x7fc21808d400 2021-02-10T11:15:05.358 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.356 7fc21d156700 1 --2- 172.21.2.112:0/2181487589 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc218081850 0x7fc2180acf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:15:05.358 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.356 7fc21e158700 1 --2- 172.21.2.112:0/2181487589 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc218081c50 0x7fc2180ad5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:15:05.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.356 7fc21d957700 1 --2- 172.21.2.112:0/2181487589 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc21808d400 0x7fc218081230 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7fc20c00a590 tx=0x7fc20c00a930).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:05.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.356 7fc20a7fc700 1 -- 172.21.2.112:0/2181487589 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc20c000ef0 con 0x7fc21808d400 2021-02-10T11:15:05.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.356 7fc20a7fc700 1 -- 172.21.2.112:0/2181487589 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc20c00ad20 con 0x7fc21808d400 2021-02-10T11:15:05.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.356 7fc20a7fc700 1 -- 172.21.2.112:0/2181487589 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc20c004620 con 0x7fc21808d400 2021-02-10T11:15:05.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.356 7fc21fb3b700 1 -- 172.21.2.112:0/2181487589 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc218187fc0 con 0x7fc21808d400 2021-02-10T11:15:05.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.356 7fc21fb3b700 1 -- 172.21.2.112:0/2181487589 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc218187fc0 con 0x7fc21808d400 2021-02-10T11:15:05.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.357 7fc20a7fc700 1 -- 172.21.2.112:0/2181487589 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc20c0045b0 con 0x7fc21808d400 2021-02-10T11:15:05.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.357 7fc20a7fc700 1 --2- 172.21.2.112:0/2181487589 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc2000186c0 0x7fc20001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:05.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.357 7fc20a7fc700 1 -- 172.21.2.112:0/2181487589 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fc20c032530 con 0x7fc21808d400 2021-02-10T11:15:05.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.357 7fc21d156700 1 --2- 172.21.2.112:0/2181487589 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc2000186c0 0x7fc20001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:05.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.358 7fc21d156700 1 --2- 172.21.2.112:0/2181487589 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc2000186c0 0x7fc20001abc0 crc :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:05.386 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.388 7fc21fb3b700 1 -- 172.21.2.112:0/2181487589 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc21807ed80 con 0x7fc21808d400 2021-02-10T11:15:05.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.390 7fc20a7fc700 1 -- 172.21.2.112:0/2181487589 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc20c0043a0 con 0x7fc21808d400 2021-02-10T11:15:05.511 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.513 7fc21fb3b700 1 -- 172.21.2.112:0/2181487589 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc21807ed80 con 0x7fc21808d400 2021-02-10T11:15:05.512 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.513 7fc20a7fc700 1 -- 172.21.2.112:0/2181487589 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+13993 (secure 0 0 0) 0x7fc20c0045b0 con 0x7fc21808d400 2021-02-10T11:15:05.512 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:15:05.512 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":20,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:14:04.629689","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:14:01.400797","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2021-02-10 11:14:02.622129","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:15:05.513 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.515 7fc21fb3b700 1 -- 172.21.2.112:0/2181487589 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc2000186c0 msgr2=0x7fc20001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:05.513 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.515 7fc21fb3b700 1 --2- 172.21.2.112:0/2181487589 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc2000186c0 0x7fc20001abc0 crc :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:05.513 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.515 7fc21fb3b700 1 -- 172.21.2.112:0/2181487589 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc21808d400 msgr2=0x7fc218081230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:05.513 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.515 7fc21fb3b700 1 --2- 172.21.2.112:0/2181487589 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc21808d400 0x7fc218081230 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7fc20c00a590 tx=0x7fc20c00a930).stop 2021-02-10T11:15:05.514 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.515 7fc21fb3b700 1 -- 172.21.2.112:0/2181487589 shutdown_connections 2021-02-10T11:15:05.514 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.515 7fc21fb3b700 1 --2- 172.21.2.112:0/2181487589 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc2000186c0 0x7fc20001abc0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:05.514 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.515 7fc21fb3b700 1 --2- 172.21.2.112:0/2181487589 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc218081c50 0x7fc2180ad5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:05.514 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.515 7fc21fb3b700 1 --2- 172.21.2.112:0/2181487589 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc218081850 0x7fc2180acf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:05.514 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.515 7fc21fb3b700 1 --2- 172.21.2.112:0/2181487589 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc21808d400 0x7fc218081230 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:05.514 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.515 7fc21fb3b700 1 -- 172.21.2.112:0/2181487589 >> 172.21.2.112:0/2181487589 conn(0x7fc2181ac620 msgr2=0x7fc2180896e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:05.514 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.516 7fc21fb3b700 1 -- 172.21.2.112:0/2181487589 shutdown_connections 2021-02-10T11:15:05.515 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.516 7fc21fb3b700 1 -- 172.21.2.112:0/2181487589 wait complete. 2021-02-10T11:15:05.522 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:13:57.508758', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:14:01.400797', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:14:02.622129', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:15:05.522 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:15:05.956 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.938 7f1ddb9fd700 1 Processor -- start 2021-02-10T11:15:05.956 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.939 7f1ddb9fd700 1 -- start start 2021-02-10T11:15:05.956 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.939 7f1ddb9fd700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f1dd41b0470 0x7f1dd41b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:05.956 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.939 7f1ddb9fd700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1dd40f20e0 0x7f1dd40f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:05.957 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.939 7f1ddb9fd700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dd40f0b50 con 0x7f1dd40ef610 2021-02-10T11:15:05.957 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.939 7f1ddb9fd700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1dd40f0820 con 0x7f1dd41b0470 2021-02-10T11:15:05.957 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.939 7f1ddb9fd700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1dd40f24e0 con 0x7f1dd40f20e0 2021-02-10T11:15:05.957 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.940 7f1dda01a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1dd40f20e0 0x7f1dd40f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:05.957 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.940 7f1dda01a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1dd40f20e0 0x7f1dd40f0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3301/0 says I am v2:172.21.2.112:42760/0 (socket says 172.21.2.112:42760) 2021-02-10T11:15:05.957 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.940 7f1dda01a700 1 -- 172.21.2.112:0/1642613712 learned_addr learned my addr 172.21.2.112:0/1642613712 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:15:05.958 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.940 7f1dd9819700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f1dd41b0470 0x7f1dd41b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:05.958 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.940 7f1dda01a700 1 -- 172.21.2.112:0/1642613712 >> v2:172.21.2.120:3300/0 conn(0x7f1dd41b0470 msgr2=0x7f1dd41b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:15:05.958 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.940 7f1dda01a700 1 --2- 172.21.2.112:0/1642613712 >> v2:172.21.2.120:3300/0 conn(0x7f1dd41b0470 0x7f1dd41b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:05.958 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.940 7f1dda01a700 1 -- 172.21.2.112:0/1642613712 >> v1:172.21.2.120:6789/0 conn(0x7f1dd40ef610 legacy=0x7f1dd40efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:05.958 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.940 7f1dda01a700 1 -- 172.21.2.112:0/1642613712 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dd40e52e0 con 0x7f1dd40f20e0 2021-02-10T11:15:05.958 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.940 7f1dda01a700 1 --2- 172.21.2.112:0/1642613712 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1dd40f20e0 0x7f1dd40f0200 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f1dcc006aa0 tx=0x7f1dcc009f20).ready entity=mon.2 client_cookie=7fcb9a5620d6abd6 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:05.959 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.940 7f1dd8817700 1 -- 172.21.2.112:0/1642613712 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1dcc00b5c0 con 0x7f1dd40f20e0 2021-02-10T11:15:05.959 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.940 7f1dd8817700 1 -- 172.21.2.112:0/1642613712 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1dcc003030 con 0x7f1dd40f20e0 2021-02-10T11:15:05.959 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.940 7f1dd8817700 1 -- 172.21.2.112:0/1642613712 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1dcc00ba80 con 0x7f1dd40f20e0 2021-02-10T11:15:05.959 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.941 7f1ddb9fd700 1 -- 172.21.2.112:0/1642613712 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1dd40f20e0 msgr2=0x7f1dd40f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:05.959 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.941 7f1ddb9fd700 1 --2- 172.21.2.112:0/1642613712 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1dd40f20e0 0x7f1dd40f0200 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f1dcc006aa0 tx=0x7f1dcc009f20).stop 2021-02-10T11:15:05.960 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.941 7f1ddb9fd700 1 -- 172.21.2.112:0/1642613712 shutdown_connections 2021-02-10T11:15:05.960 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.941 7f1ddb9fd700 1 --2- 172.21.2.112:0/1642613712 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1dd40f20e0 0x7f1dd40f0200 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:05.960 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.941 7f1ddb9fd700 1 --2- 172.21.2.112:0/1642613712 >> v2:172.21.2.120:3300/0 conn(0x7f1dd41b0470 0x7f1dd41b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:05.960 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.941 7f1ddb9fd700 1 -- 172.21.2.112:0/1642613712 >> 172.21.2.112:0/1642613712 conn(0x7f1dd40ce3e0 msgr2=0x7f1dd40e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:05.961 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.941 7f1ddb9fd700 1 -- 172.21.2.112:0/1642613712 shutdown_connections 2021-02-10T11:15:05.961 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.941 7f1ddb9fd700 1 -- 172.21.2.112:0/1642613712 wait complete. 2021-02-10T11:15:05.961 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.955 7f1ddb9fd700 1 Processor -- start 2021-02-10T11:15:05.961 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.956 7f1ddb9fd700 1 -- start start 2021-02-10T11:15:05.961 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.956 7f1ddb9fd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1dd4105d10 0x7f1dd409f880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:05.962 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.956 7f1ddb9fd700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1dd40961f0 0x7f1dd40965f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:05.962 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.956 7f1ddb9fd700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1dd4106890 0x7f1dd4106c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:05.962 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.956 7f1ddb9fd700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dd4106110 con 0x7f1dd4105d10 2021-02-10T11:15:05.962 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.956 7f1ddb9fd700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dd4106380 con 0x7f1dd4106890 2021-02-10T11:15:05.962 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.956 7f1ddb9fd700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1dd40a6620 con 0x7f1dd40961f0 2021-02-10T11:15:05.962 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.956 7f1dd9819700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1dd4105d10 0x7f1dd409f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:05.963 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.956 7f1dd9819700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1dd4105d10 0x7f1dd409f880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:43082/0 (socket says 172.21.2.112:43082) 2021-02-10T11:15:05.963 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.956 7f1dd9819700 1 -- 172.21.2.112:0/2737763195 learned_addr learned my addr 172.21.2.112:0/2737763195 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:15:05.963 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.956 7f1dd9819700 1 -- 172.21.2.112:0/2737763195 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1dd40961f0 msgr2=0x7f1dd40965f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:05.963 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.956 7f1dd9819700 1 --2- 172.21.2.112:0/2737763195 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1dd40961f0 0x7f1dd40965f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:05.963 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.956 7f1dd9819700 1 -- 172.21.2.112:0/2737763195 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1dd4106890 msgr2=0x7f1dd4106c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:05.963 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.956 7f1dd9819700 1 --2- 172.21.2.112:0/2737763195 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1dd4106890 0x7f1dd4106c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:05.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.956 7f1dd9819700 1 -- 172.21.2.112:0/2737763195 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dd41072b0 con 0x7f1dd4105d10 2021-02-10T11:15:05.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.957 7f1dd9819700 1 --2- 172.21.2.112:0/2737763195 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1dd4105d10 0x7f1dd409f880 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7f1dc800b560 tx=0x7f1dc800b900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:05.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.957 7f1dc67fc700 1 -- 172.21.2.112:0/2737763195 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1dc800bcf0 con 0x7f1dd4105d10 2021-02-10T11:15:05.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.957 7f1dc67fc700 1 -- 172.21.2.112:0/2737763195 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1dc8000c90 con 0x7f1dd4105d10 2021-02-10T11:15:05.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.957 7f1dc67fc700 1 -- 172.21.2.112:0/2737763195 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1dc80022b0 con 0x7f1dd4105d10 2021-02-10T11:15:05.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.957 7f1ddb9fd700 1 -- 172.21.2.112:0/2737763195 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1dd41072b0 con 0x7f1dd4105d10 2021-02-10T11:15:05.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.957 7f1ddb9fd700 1 -- 172.21.2.112:0/2737763195 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1dd41072b0 con 0x7f1dd4105d10 2021-02-10T11:15:05.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.957 7f1dc67fc700 1 -- 172.21.2.112:0/2737763195 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1dc8006b00 con 0x7f1dd4105d10 2021-02-10T11:15:05.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.957 7f1dc67fc700 1 --2- 172.21.2.112:0/2737763195 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1dbc0186c0 0x7f1dbc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:05.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.957 7f1dc67fc700 1 -- 172.21.2.112:0/2737763195 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f1dc80323b0 con 0x7f1dd4105d10 2021-02-10T11:15:05.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.958 7f1dd9018700 1 --2- 172.21.2.112:0/2737763195 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1dbc0186c0 0x7f1dbc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:05.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.958 7f1dd9018700 1 --2- 172.21.2.112:0/2737763195 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1dbc0186c0 0x7f1dbc01abc0 crc :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:05.988 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.989 7f1ddb9fd700 1 -- 172.21.2.112:0/2737763195 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1dd4105ae0 con 0x7f1dd4105d10 2021-02-10T11:15:05.990 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:05.992 7f1dc67fc700 1 -- 172.21.2.112:0/2737763195 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1dc8030ce0 con 0x7f1dd4105d10 2021-02-10T11:15:06.118 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:06.120 7f1ddb9fd700 1 -- 172.21.2.112:0/2737763195 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f1dd40f0b50 con 0x7f1dd4105d10 2021-02-10T11:15:06.118 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:06.120 7f1dc67fc700 1 -- 172.21.2.112:0/2737763195 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v20) v1 ==== 92+0+10 (secure 0 0 0) 0x7f1dc800bcf0 con 0x7f1dd4105d10 2021-02-10T11:15:06.119 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:15:06.120 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:06.122 7f1ddb9fd700 1 -- 172.21.2.112:0/2737763195 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1dbc0186c0 msgr2=0x7f1dbc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:06.120 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:06.122 7f1ddb9fd700 1 --2- 172.21.2.112:0/2737763195 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1dbc0186c0 0x7f1dbc01abc0 crc :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:06.120 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:06.122 7f1ddb9fd700 1 -- 172.21.2.112:0/2737763195 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1dd4105d10 msgr2=0x7f1dd409f880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:06.120 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:06.122 7f1ddb9fd700 1 --2- 172.21.2.112:0/2737763195 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1dd4105d10 0x7f1dd409f880 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7f1dc800b560 tx=0x7f1dc800b900).stop 2021-02-10T11:15:06.120 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:06.122 7f1ddb9fd700 1 -- 172.21.2.112:0/2737763195 shutdown_connections 2021-02-10T11:15:06.120 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:06.122 7f1ddb9fd700 1 --2- 172.21.2.112:0/2737763195 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1dbc0186c0 0x7f1dbc01abc0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:06.121 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:06.122 7f1ddb9fd700 1 --2- 172.21.2.112:0/2737763195 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1dd4106890 0x7f1dd4106c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:06.121 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:06.122 7f1ddb9fd700 1 --2- 172.21.2.112:0/2737763195 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1dd40961f0 0x7f1dd40965f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:06.121 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:06.122 7f1ddb9fd700 1 --2- 172.21.2.112:0/2737763195 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1dd4105d10 0x7f1dd409f880 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:06.121 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:06.122 7f1ddb9fd700 1 -- 172.21.2.112:0/2737763195 >> 172.21.2.112:0/2737763195 conn(0x7f1dd4095a70 msgr2=0x7f1dd40bf050 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:06.121 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:06.122 7f1ddb9fd700 1 -- 172.21.2.112:0/2737763195 shutdown_connections 2021-02-10T11:15:06.121 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:06.122 7f1ddb9fd700 1 -- 172.21.2.112:0/2737763195 wait complete. 2021-02-10T11:15:06.128 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:15:06.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:06.542 7efefa010700 1 Processor -- start 2021-02-10T11:15:06.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:06.542 7efefa010700 1 -- start start 2021-02-10T11:15:06.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:06.542 7efefa010700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efef41b0470 0x7efef41b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:06.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:06.542 7efefa010700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7efef40ef610 0x7efef40efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:06.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:06.542 7efefa010700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efef40f0b50 con 0x7efef40f0030 2021-02-10T11:15:06.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:06.542 7efefa010700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efef40e52e0 con 0x7efef40ef610 2021-02-10T11:15:06.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:06.542 7efefa010700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efef40e5490 con 0x7efef41b0470 2021-02-10T11:15:06.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:06.543 7efef37fe700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efef41b0470 0x7efef41b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:06.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:06.543 7efef2ffd700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7efef40ef610 0x7efef40efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:06.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:06.543 7efef37fe700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efef41b0470 0x7efef41b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3301/0 says I am v2:172.21.2.112:42770/0 (socket says 172.21.2.112:42770) 2021-02-10T11:15:06.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:06.543 7efef2ffd700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7efef40ef610 0x7efef40efa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3300/0 says I am v2:172.21.2.112:45368/0 (socket says 172.21.2.112:45368) 2021-02-10T11:15:06.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:06.543 7efef37fe700 1 -- 172.21.2.112:0/4105788522 learned_addr learned my addr 172.21.2.112:0/4105788522 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:15:06.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:06.544 7efef37fe700 1 -- 172.21.2.112:0/4105788522 >> v2:172.21.2.120:3300/0 conn(0x7efef40ef610 msgr2=0x7efef40efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:15:06.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:06.544 7efef37fe700 1 --2- 172.21.2.112:0/4105788522 >> v2:172.21.2.120:3300/0 conn(0x7efef40ef610 0x7efef40efa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:06.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:06.544 7efef37fe700 1 -- 172.21.2.112:0/4105788522 >> v1:172.21.2.120:6789/0 conn(0x7efef40f0030 legacy=0x7efef40f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:06.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:06.544 7efef37fe700 1 -- 172.21.2.112:0/4105788522 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efef40f0430 con 0x7efef41b0470 2021-02-10T11:15:06.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:06.544 7efef37fe700 1 --2- 172.21.2.112:0/4105788522 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efef41b0470 0x7efef41b0870 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7efedc006700 tx=0x7efedc009b80).ready entity=mon.2 client_cookie=ca85d7a0080a947a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:06.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:06.544 7efef27fc700 1 -- 172.21.2.112:0/4105788522 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efedc00c0c0 con 0x7efef41b0470 2021-02-10T11:15:06.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:06.544 7efef27fc700 1 -- 172.21.2.112:0/4105788522 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efedc00c340 con 0x7efef41b0470 2021-02-10T11:15:06.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:06.544 7efef27fc700 1 -- 172.21.2.112:0/4105788522 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efedc0009b0 con 0x7efef41b0470 2021-02-10T11:15:06.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:06.544 7efefa010700 1 -- 172.21.2.112:0/4105788522 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efef41b0470 msgr2=0x7efef41b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:06.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:06.544 7efefa010700 1 --2- 172.21.2.112:0/4105788522 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efef41b0470 0x7efef41b0870 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7efedc006700 tx=0x7efedc009b80).stop 2021-02-10T11:15:06.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:06.545 7efefa010700 1 -- 172.21.2.112:0/4105788522 shutdown_connections 2021-02-10T11:15:06.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:06.545 7efefa010700 1 --2- 172.21.2.112:0/4105788522 >> v2:172.21.2.120:3300/0 conn(0x7efef40ef610 0x7efef40efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:06.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:06.545 7efefa010700 1 --2- 172.21.2.112:0/4105788522 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efef41b0470 0x7efef41b0870 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:06.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:06.545 7efefa010700 1 -- 172.21.2.112:0/4105788522 >> 172.21.2.112:0/4105788522 conn(0x7efef40ce3e0 msgr2=0x7efef40e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:06.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:06.545 7efefa010700 1 -- 172.21.2.112:0/4105788522 shutdown_connections 2021-02-10T11:15:06.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:06.545 7efefa010700 1 -- 172.21.2.112:0/4105788522 wait complete. 2021-02-10T11:15:06.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:06.559 7efefa010700 1 Processor -- start 2021-02-10T11:15:06.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:06.560 7efefa010700 1 -- start start 2021-02-10T11:15:06.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:06.560 7efefa010700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efef40a0420 0x7efef40a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:06.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:06.560 7efefa010700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efef4189d00 0x7efef418a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:06.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:06.560 7efefa010700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7efef418a720 0x7efef40b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:06.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:06.560 7efefa010700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efef4189800 con 0x7efef4189d00 2021-02-10T11:15:06.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:06.560 7efefa010700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efef418ab20 con 0x7efef418a720 2021-02-10T11:15:06.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:06.560 7efefa010700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efef40a1140 con 0x7efef40a0420 2021-02-10T11:15:06.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:06.560 7efef2ffd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efef4189d00 0x7efef418a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:06.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:06.560 7efef2ffd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efef4189d00 0x7efef418a100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:43094/0 (socket says 172.21.2.112:43094) 2021-02-10T11:15:06.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:06.560 7efef2ffd700 1 -- 172.21.2.112:0/2020339991 learned_addr learned my addr 172.21.2.112:0/2020339991 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:15:06.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:06.560 7efef37fe700 1 --2- 172.21.2.112:0/2020339991 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efef40a0420 0x7efef40a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:06.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:06.560 7efef3fff700 1 --2- 172.21.2.112:0/2020339991 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7efef418a720 0x7efef40b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:06.563 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:06.561 7efef37fe700 1 -- 172.21.2.112:0/2020339991 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7efef418a720 msgr2=0x7efef40b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:06.563 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:06.561 7efef37fe700 1 --2- 172.21.2.112:0/2020339991 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7efef418a720 0x7efef40b7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:06.563 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:06.561 7efef37fe700 1 -- 172.21.2.112:0/2020339991 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efef4189d00 msgr2=0x7efef418a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:06.563 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:06.561 7efef37fe700 1 --2- 172.21.2.112:0/2020339991 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efef4189d00 0x7efef418a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:06.563 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:06.561 7efef37fe700 1 -- 172.21.2.112:0/2020339991 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efef40b8200 con 0x7efef40a0420 2021-02-10T11:15:06.563 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:06.561 7efef3fff700 1 --2- 172.21.2.112:0/2020339991 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7efef418a720 0x7efef40b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:15:06.563 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:06.561 7efef2ffd700 1 --2- 172.21.2.112:0/2020339991 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efef4189d00 0x7efef418a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:15:06.563 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:06.561 7efef37fe700 1 --2- 172.21.2.112:0/2020339991 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efef40a0420 0x7efef40a0820 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7efedc00b530 tx=0x7efedc00b8d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:06.563 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:06.561 7efeebfff700 1 -- 172.21.2.112:0/2020339991 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efedc009440 con 0x7efef40a0420 2021-02-10T11:15:06.563 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:06.561 7efeebfff700 1 -- 172.21.2.112:0/2020339991 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efedc0052e0 con 0x7efef40a0420 2021-02-10T11:15:06.563 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:06.561 7efefa010700 1 -- 172.21.2.112:0/2020339991 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efef40b8200 con 0x7efef40a0420 2021-02-10T11:15:06.563 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:06.561 7efeebfff700 1 -- 172.21.2.112:0/2020339991 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efedc005740 con 0x7efef40a0420 2021-02-10T11:15:06.564 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:06.561 7efefa010700 1 -- 172.21.2.112:0/2020339991 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efef40b8790 con 0x7efef40a0420 2021-02-10T11:15:06.564 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:06.562 7efeebfff700 1 -- 172.21.2.112:0/2020339991 <== mon.2 v2:172.21.2.120:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7efedc009440 con 0x7efef40a0420 2021-02-10T11:15:06.564 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:06.562 7efeebfff700 1 --2- 172.21.2.112:0/2020339991 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7efed80186c0 0x7efed801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:06.564 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:06.562 7efeebfff700 1 -- 172.21.2.112:0/2020339991 <== mon.2 v2:172.21.2.120:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7efedc032350 con 0x7efef40a0420 2021-02-10T11:15:06.564 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:06.562 7efef2ffd700 1 --2- 172.21.2.112:0/2020339991 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7efed80186c0 0x7efed801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:06.564 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:06.563 7efef2ffd700 1 --2- 172.21.2.112:0/2020339991 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7efed80186c0 0x7efed801abc0 crc :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:06.591 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:06.592 7efefa010700 1 -- 172.21.2.112:0/2020339991 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efef418ab20 con 0x7efef40a0420 2021-02-10T11:15:06.594 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:06.596 7efeebfff700 1 -- 172.21.2.112:0/2020339991 <== mon.2 v2:172.21.2.120:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7efedc006030 con 0x7efef40a0420 2021-02-10T11:15:06.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:06.725 7efefa010700 1 -- 172.21.2.112:0/2020339991 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7efef418ab20 con 0x7efef40a0420 2021-02-10T11:15:06.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:06.725 7efeebfff700 1 -- 172.21.2.112:0/2020339991 <== mon.2 v2:172.21.2.120:3301/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v20) v1 ==== 104+0+10 (secure 0 0 0) 0x7efedc009440 con 0x7efef40a0420 2021-02-10T11:15:06.724 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:15:06.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:06.727 7efefa010700 1 -- 172.21.2.112:0/2020339991 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7efed80186c0 msgr2=0x7efed801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:06.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:06.727 7efefa010700 1 --2- 172.21.2.112:0/2020339991 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7efed80186c0 0x7efed801abc0 crc :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:06.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:06.727 7efefa010700 1 -- 172.21.2.112:0/2020339991 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efef40a0420 msgr2=0x7efef40a0820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:06.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:06.727 7efefa010700 1 --2- 172.21.2.112:0/2020339991 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efef40a0420 0x7efef40a0820 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7efedc00b530 tx=0x7efedc00b8d0).stop 2021-02-10T11:15:06.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:06.727 7efefa010700 1 -- 172.21.2.112:0/2020339991 shutdown_connections 2021-02-10T11:15:06.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:06.727 7efefa010700 1 --2- 172.21.2.112:0/2020339991 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7efed80186c0 0x7efed801abc0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:06.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:06.727 7efefa010700 1 --2- 172.21.2.112:0/2020339991 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7efef418a720 0x7efef40b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:06.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:06.727 7efefa010700 1 --2- 172.21.2.112:0/2020339991 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efef4189d00 0x7efef418a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:06.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:06.727 7efefa010700 1 --2- 172.21.2.112:0/2020339991 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efef40a0420 0x7efef40a0820 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:06.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:06.727 7efefa010700 1 -- 172.21.2.112:0/2020339991 >> 172.21.2.112:0/2020339991 conn(0x7efef4094e10 msgr2=0x7efef40a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:06.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:06.727 7efefa010700 1 -- 172.21.2.112:0/2020339991 shutdown_connections 2021-02-10T11:15:06.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:06.727 7efefa010700 1 -- 172.21.2.112:0/2020339991 wait complete. 2021-02-10T11:15:06.734 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:15:07.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.149 7f2d497fe700 1 Processor -- start 2021-02-10T11:15:07.164 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.149 7f2d497fe700 1 -- start start 2021-02-10T11:15:07.164 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.149 7f2d497fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f2d440f05f0 0x7f2d440f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:07.164 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.149 7f2d497fe700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d441b0c70 con 0x7f2d441b0870 2021-02-10T11:15:07.164 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.149 7f2d497fe700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d440f0b50 con 0x7f2d441b0470 2021-02-10T11:15:07.164 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.149 7f2d497fe700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d440e5ea0 con 0x7f2d440f05f0 2021-02-10T11:15:07.164 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.149 7f2d43d75700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f2d440f05f0 0x7f2d440f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:07.164 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.149 7f2d43d75700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f2d440f05f0 0x7f2d440f20e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:43102/0 (socket says 172.21.2.112:43102) 2021-02-10T11:15:07.164 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.149 7f2d43d75700 1 -- 172.21.2.112:0/2993544896 learned_addr learned my addr 172.21.2.112:0/2993544896 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:15:07.164 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.150 7f2d43d75700 1 -- 172.21.2.112:0/2993544896 >> v1:172.21.2.120:6789/0 conn(0x7f2d441b0470 legacy=0x7f2d440ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:15:07.164 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.150 7f2d43d75700 1 -- 172.21.2.112:0/2993544896 >> v1:172.21.2.112:6789/0 conn(0x7f2d441b0870 legacy=0x7f2d440efe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:07.165 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.150 7f2d43d75700 1 -- 172.21.2.112:0/2993544896 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d440e52e0 con 0x7f2d440f05f0 2021-02-10T11:15:07.165 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.150 7f2d43d75700 1 --2- 172.21.2.112:0/2993544896 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2d440f05f0 0x7f2d440f20e0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7f2d3c006f00 tx=0x7f2d3c009f10).ready entity=mon.0 client_cookie=a2126246419cedab server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:07.165 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.150 7f2d42572700 1 -- 172.21.2.112:0/2993544896 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2d3c003030 con 0x7f2d440f05f0 2021-02-10T11:15:07.165 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.150 7f2d42572700 1 -- 172.21.2.112:0/2993544896 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d3c00b3b0 con 0x7f2d440f05f0 2021-02-10T11:15:07.165 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.150 7f2d42572700 1 -- 172.21.2.112:0/2993544896 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2d3c00b8d0 con 0x7f2d440f05f0 2021-02-10T11:15:07.165 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.150 7f2d497fe700 1 -- 172.21.2.112:0/2993544896 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2d440f05f0 msgr2=0x7f2d440f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:07.165 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.150 7f2d497fe700 1 --2- 172.21.2.112:0/2993544896 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2d440f05f0 0x7f2d440f20e0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7f2d3c006f00 tx=0x7f2d3c009f10).stop 2021-02-10T11:15:07.165 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.151 7f2d497fe700 1 -- 172.21.2.112:0/2993544896 shutdown_connections 2021-02-10T11:15:07.165 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.151 7f2d497fe700 1 --2- 172.21.2.112:0/2993544896 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2d440f05f0 0x7f2d440f20e0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:07.165 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.151 7f2d497fe700 1 -- 172.21.2.112:0/2993544896 >> 172.21.2.112:0/2993544896 conn(0x7f2d440ce3e0 msgr2=0x7f2d440e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:07.165 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.151 7f2d497fe700 1 -- 172.21.2.112:0/2993544896 shutdown_connections 2021-02-10T11:15:07.165 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.151 7f2d497fe700 1 -- 172.21.2.112:0/2993544896 wait complete. 2021-02-10T11:15:07.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.166 7f2d497fe700 1 Processor -- start 2021-02-10T11:15:07.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.166 7f2d497fe700 1 -- start start 2021-02-10T11:15:07.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.166 7f2d497fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2d44087510 0x7f2d440aedc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:07.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.166 7f2d497fe700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2d441c68f0 0x7f2d441b1f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:07.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.166 7f2d497fe700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2d4408c4d0 0x7f2d441b18a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:07.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.167 7f2d497fe700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d440a2d50 con 0x7f2d44087510 2021-02-10T11:15:07.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.167 7f2d497fe700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d440a2f70 con 0x7f2d441c68f0 2021-02-10T11:15:07.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.167 7f2d497fe700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d440a3190 con 0x7f2d4408c4d0 2021-02-10T11:15:07.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.167 7f2d43574700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2d44087510 0x7f2d440aedc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:07.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.167 7f2d43574700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2d44087510 0x7f2d440aedc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:43106/0 (socket says 172.21.2.112:43106) 2021-02-10T11:15:07.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.167 7f2d43574700 1 -- 172.21.2.112:0/2393619631 learned_addr learned my addr 172.21.2.112:0/2393619631 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:15:07.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.167 7f2d43574700 1 -- 172.21.2.112:0/2393619631 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2d4408c4d0 msgr2=0x7f2d441b18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:07.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.167 7f2d43574700 1 --2- 172.21.2.112:0/2393619631 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2d4408c4d0 0x7f2d441b18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:07.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.167 7f2d43574700 1 -- 172.21.2.112:0/2393619631 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2d441c68f0 msgr2=0x7f2d441b1f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:07.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.167 7f2d43574700 1 --2- 172.21.2.112:0/2393619631 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2d441c68f0 0x7f2d441b1f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:07.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.167 7f2d43574700 1 -- 172.21.2.112:0/2393619631 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d440a33b0 con 0x7f2d44087510 2021-02-10T11:15:07.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.167 7f2d43574700 1 --2- 172.21.2.112:0/2393619631 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2d44087510 0x7f2d440aedc0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7f2d3400b530 tx=0x7f2d3400b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:07.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.167 7f2d2bfff700 1 -- 172.21.2.112:0/2393619631 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2d3400bcc0 con 0x7f2d44087510 2021-02-10T11:15:07.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.167 7f2d2bfff700 1 -- 172.21.2.112:0/2393619631 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d34000e20 con 0x7f2d44087510 2021-02-10T11:15:07.169 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.167 7f2d2bfff700 1 -- 172.21.2.112:0/2393619631 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2d34006210 con 0x7f2d44087510 2021-02-10T11:15:07.169 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.168 7f2d497fe700 1 -- 172.21.2.112:0/2393619631 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d440a33b0 con 0x7f2d44087510 2021-02-10T11:15:07.169 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.168 7f2d497fe700 1 -- 172.21.2.112:0/2393619631 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d440a33b0 con 0x7f2d44087510 2021-02-10T11:15:07.169 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.168 7f2d2bfff700 1 -- 172.21.2.112:0/2393619631 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2d34006020 con 0x7f2d44087510 2021-02-10T11:15:07.169 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.168 7f2d2bfff700 1 --2- 172.21.2.112:0/2393619631 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2d240186c0 0x7f2d2401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:07.169 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.168 7f2d2bfff700 1 -- 172.21.2.112:0/2393619631 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f2d340327e0 con 0x7f2d44087510 2021-02-10T11:15:07.169 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.169 7f2d42d73700 1 --2- 172.21.2.112:0/2393619631 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2d240186c0 0x7f2d2401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:07.169 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.169 7f2d42d73700 1 --2- 172.21.2.112:0/2393619631 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2d240186c0 0x7f2d2401abc0 crc :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:07.197 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.199 7f2d497fe700 1 -- 172.21.2.112:0/2393619631 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d4407e6b0 con 0x7f2d44087510 2021-02-10T11:15:07.200 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.202 7f2d2bfff700 1 -- 172.21.2.112:0/2393619631 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2d3401f520 con 0x7f2d44087510 2021-02-10T11:15:07.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.330 7f2d497fe700 1 -- 172.21.2.112:0/2393619631 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f2d4407e6b0 con 0x7f2d44087510 2021-02-10T11:15:07.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.330 7f2d2bfff700 1 -- 172.21.2.112:0/2393619631 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v20) v1 ==== 100+0+10 (secure 0 0 0) 0x7f2d34000e00 con 0x7f2d44087510 2021-02-10T11:15:07.329 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:15:07.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.332 7f2d497fe700 1 -- 172.21.2.112:0/2393619631 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2d240186c0 msgr2=0x7f2d2401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:07.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.332 7f2d497fe700 1 --2- 172.21.2.112:0/2393619631 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2d240186c0 0x7f2d2401abc0 crc :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:07.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.332 7f2d497fe700 1 -- 172.21.2.112:0/2393619631 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2d44087510 msgr2=0x7f2d440aedc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:07.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.332 7f2d497fe700 1 --2- 172.21.2.112:0/2393619631 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2d44087510 0x7f2d440aedc0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7f2d3400b530 tx=0x7f2d3400b8d0).stop 2021-02-10T11:15:07.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.332 7f2d497fe700 1 -- 172.21.2.112:0/2393619631 shutdown_connections 2021-02-10T11:15:07.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.332 7f2d497fe700 1 --2- 172.21.2.112:0/2393619631 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2d240186c0 0x7f2d2401abc0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:07.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.332 7f2d497fe700 1 --2- 172.21.2.112:0/2393619631 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2d441c68f0 0x7f2d441b1f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:07.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.332 7f2d497fe700 1 --2- 172.21.2.112:0/2393619631 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2d4408c4d0 0x7f2d441b18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:07.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.332 7f2d497fe700 1 --2- 172.21.2.112:0/2393619631 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2d44087510 0x7f2d440aedc0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:07.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.332 7f2d497fe700 1 -- 172.21.2.112:0/2393619631 >> 172.21.2.112:0/2393619631 conn(0x7f2d440886d0 msgr2=0x7f2d44081c40 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:07.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.333 7f2d497fe700 1 -- 172.21.2.112:0/2393619631 shutdown_connections 2021-02-10T11:15:07.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.333 7f2d497fe700 1 -- 172.21.2.112:0/2393619631 wait complete. 2021-02-10T11:15:07.394 INFO:tasks.cephfs_test_runner:Starting test: test_async_subvolume_rm (tasks.cephfs.test_volumes.TestVolumes) 2021-02-10T11:15:07.395 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestVolumes.test_async_subvolume_rm' 2021-02-10T11:15:07.823 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.807 7febd92e8700 1 Processor -- start 2021-02-10T11:15:07.823 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.807 7febd92e8700 1 -- start start 2021-02-10T11:15:07.823 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.808 7febd92e8700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7febd407e6a0 0x7febd40d1790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:07.823 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.808 7febd92e8700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febd407ed80 con 0x7febd40e5470 2021-02-10T11:15:07.823 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.808 7febd92e8700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febd407eb10 con 0x7febd40e5070 2021-02-10T11:15:07.823 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.808 7febd92e8700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febd41b4080 con 0x7febd407e6a0 2021-02-10T11:15:07.824 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.808 7febd287c700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7febd40e5470 0x7febd407f300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:42578/0 (socket says 172.21.2.112:42578) 2021-02-10T11:15:07.824 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.808 7febd287c700 1 -- 172.21.2.112:0/1780412241 learned_addr learned my addr 172.21.2.112:0/1780412241 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:15:07.824 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.808 7febd207b700 1 -- 172.21.2.112:0/1780412241 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2394023906 0 0) 0x7febc40009b0 con 0x7febd40e5470 2021-02-10T11:15:07.824 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.808 7febd387e700 1 --2- 172.21.2.112:0/1780412241 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7febd407e6a0 0x7febd40d1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:07.824 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.808 7febd207b700 1 -- 172.21.2.112:0/1780412241 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febc0002fe0 con 0x7febd40e5470 2021-02-10T11:15:07.824 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.808 7febd207b700 1 -- 172.21.2.112:0/1780412241 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2527446131 0 0) 0x7febc4001010 con 0x7febd40e5470 2021-02-10T11:15:07.824 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.809 7febd207b700 1 -- 172.21.2.112:0/1780412241 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7febd407e6a0 msgr2=0x7febd40d1790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:15:07.824 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.809 7febd207b700 1 --2- 172.21.2.112:0/1780412241 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7febd407e6a0 0x7febd40d1790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:07.824 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.809 7febd207b700 1 -- 172.21.2.112:0/1780412241 >> v1:172.21.2.120:6789/0 conn(0x7febd40e5070 legacy=0x7febd41b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:07.824 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.809 7febd207b700 1 -- 172.21.2.112:0/1780412241 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febd40d1db0 con 0x7febd40e5470 2021-02-10T11:15:07.824 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.809 7febd207b700 1 -- 172.21.2.112:0/1780412241 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7febc4001530 con 0x7febd40e5470 2021-02-10T11:15:07.824 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.809 7febd207b700 1 -- 172.21.2.112:0/1780412241 >> v1:172.21.2.112:6789/0 conn(0x7febd40e5470 legacy=0x7febd407f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:07.824 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.809 7febd207b700 1 --2- 172.21.2.112:0/1780412241 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7febc0003ac0 0x7febc0005ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:07.825 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.809 7febd207b700 1 --2- 172.21.2.112:0/1780412241 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7febc0006770 0x7febc0008c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:07.825 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.809 7febd207b700 1 --2- 172.21.2.112:0/1780412241 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7febc0009510 0x7febc000ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:07.825 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.809 7febd207b700 1 -- 172.21.2.112:0/1780412241 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febc000c2e0 con 0x7febc0006770 2021-02-10T11:15:07.825 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.809 7febd207b700 1 -- 172.21.2.112:0/1780412241 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febc000c500 con 0x7febc0009510 2021-02-10T11:15:07.825 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.809 7febd207b700 1 -- 172.21.2.112:0/1780412241 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febc0050730 con 0x7febc0003ac0 2021-02-10T11:15:07.825 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.809 7febd287c700 1 --2- 172.21.2.112:0/1780412241 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7febc0006770 0x7febc0008c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:07.825 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.809 7febd387e700 1 --2- 172.21.2.112:0/1780412241 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7febc0009510 0x7febc000ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:07.825 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.809 7febd307d700 1 --2- 172.21.2.112:0/1780412241 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7febc0003ac0 0x7febc0005ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:07.825 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.809 7febd287c700 1 -- 172.21.2.112:0/1780412241 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7febc0003ac0 msgr2=0x7febc0005ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:15:07.825 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.809 7febd287c700 1 --2- 172.21.2.112:0/1780412241 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7febc0003ac0 0x7febc0005ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:07.825 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.809 7febd287c700 1 -- 172.21.2.112:0/1780412241 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7febc0009510 msgr2=0x7febc000ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:15:07.825 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.809 7febd287c700 1 --2- 172.21.2.112:0/1780412241 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7febc0009510 0x7febc000ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:07.825 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.809 7febd287c700 1 -- 172.21.2.112:0/1780412241 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febc0050990 con 0x7febc0006770 2021-02-10T11:15:07.826 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.809 7febd287c700 1 --2- 172.21.2.112:0/1780412241 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7febc0006770 0x7febc0008c70 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7febc40017d0 tx=0x7febc4007d30).ready entity=mon.0 client_cookie=a4ca00b080c13abe server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:07.826 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.810 7febd207b700 1 -- 172.21.2.112:0/1780412241 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7febc4001010 con 0x7febc0006770 2021-02-10T11:15:07.826 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.810 7febd207b700 1 -- 172.21.2.112:0/1780412241 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febc400a310 con 0x7febc0006770 2021-02-10T11:15:07.826 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.810 7febd207b700 1 -- 172.21.2.112:0/1780412241 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7febc400a830 con 0x7febc0006770 2021-02-10T11:15:07.826 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.810 7febd92e8700 1 -- 172.21.2.112:0/1780412241 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7febc0006770 msgr2=0x7febc0008c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:07.826 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.810 7febd92e8700 1 --2- 172.21.2.112:0/1780412241 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7febc0006770 0x7febc0008c70 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7febc40017d0 tx=0x7febc4007d30).stop 2021-02-10T11:15:07.826 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.810 7febd387e700 1 -- 172.21.2.112:0/1780412241 reap_dead start 2021-02-10T11:15:07.826 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.810 7febd92e8700 1 -- 172.21.2.112:0/1780412241 shutdown_connections 2021-02-10T11:15:07.826 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.810 7febd92e8700 1 -- 172.21.2.112:0/1780412241 >> 172.21.2.112:0/1780412241 conn(0x7febd40ce690 msgr2=0x7febd40e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:07.826 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.810 7febd92e8700 1 -- 172.21.2.112:0/1780412241 shutdown_connections 2021-02-10T11:15:07.826 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.810 7febd92e8700 1 -- 172.21.2.112:0/1780412241 wait complete. 2021-02-10T11:15:07.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.825 7febd92e8700 1 Processor -- start 2021-02-10T11:15:07.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.825 7febd92e8700 1 -- start start 2021-02-10T11:15:07.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.825 7febd92e8700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7febd4187eb0 0x7febd40b7f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:07.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.825 7febd92e8700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7febd40b85a0 0x7febd40b89a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:07.829 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.825 7febd92e8700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7febd4095a20 0x7febd4095e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:07.829 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.825 7febd92e8700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febd4188330 con 0x7febd4187eb0 2021-02-10T11:15:07.829 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.825 7febd92e8700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febd418aa00 con 0x7febd40b85a0 2021-02-10T11:15:07.829 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.825 7febd92e8700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febd4096440 con 0x7febd4095a20 2021-02-10T11:15:07.829 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.826 7febd307d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7febd4187eb0 0x7febd40b7f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:07.829 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.826 7febd307d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7febd4187eb0 0x7febd40b7f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:43126/0 (socket says 172.21.2.112:43126) 2021-02-10T11:15:07.829 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.826 7febd307d700 1 -- 172.21.2.112:0/3054390986 learned_addr learned my addr 172.21.2.112:0/3054390986 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:15:07.829 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.826 7febd387e700 1 --2- 172.21.2.112:0/3054390986 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7febd4095a20 0x7febd4095e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:07.829 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.826 7febd287c700 1 --2- 172.21.2.112:0/3054390986 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7febd40b85a0 0x7febd40b89a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:07.829 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.826 7febd307d700 1 -- 172.21.2.112:0/3054390986 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7febd4095a20 msgr2=0x7febd4095e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:07.829 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.826 7febd307d700 1 --2- 172.21.2.112:0/3054390986 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7febd4095a20 0x7febd4095e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:07.830 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.826 7febd307d700 1 -- 172.21.2.112:0/3054390986 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7febd40b85a0 msgr2=0x7febd40b89a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:07.830 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.826 7febd307d700 1 --2- 172.21.2.112:0/3054390986 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7febd40b85a0 0x7febd40b89a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:07.830 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.826 7febd307d700 1 -- 172.21.2.112:0/3054390986 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febd4096660 con 0x7febd4187eb0 2021-02-10T11:15:07.830 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.826 7febd287c700 1 --2- 172.21.2.112:0/3054390986 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7febd40b85a0 0x7febd40b89a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:15:07.830 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.826 7febd307d700 1 --2- 172.21.2.112:0/3054390986 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7febd4187eb0 0x7febd40b7f80 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7febbc00b590 tx=0x7febbc00b930).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:07.830 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.826 7febcbfff700 1 -- 172.21.2.112:0/3054390986 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7febbc00bd20 con 0x7febd4187eb0 2021-02-10T11:15:07.830 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.826 7febcbfff700 1 -- 172.21.2.112:0/3054390986 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febbc000ca0 con 0x7febd4187eb0 2021-02-10T11:15:07.830 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.826 7febd92e8700 1 -- 172.21.2.112:0/3054390986 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febd4096660 con 0x7febd4187eb0 2021-02-10T11:15:07.830 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.826 7febd92e8700 1 -- 172.21.2.112:0/3054390986 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febd4096660 con 0x7febd4187eb0 2021-02-10T11:15:07.830 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.826 7febcbfff700 1 -- 172.21.2.112:0/3054390986 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7febbc002220 con 0x7febd4187eb0 2021-02-10T11:15:07.830 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.827 7febcbfff700 1 -- 172.21.2.112:0/3054390986 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7febbc0052e0 con 0x7febd4187eb0 2021-02-10T11:15:07.831 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.827 7febcbfff700 1 --2- 172.21.2.112:0/3054390986 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7febb8018730 0x7febb801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:07.831 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.827 7febcbfff700 1 -- 172.21.2.112:0/3054390986 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7febbc0327a0 con 0x7febd4187eb0 2021-02-10T11:15:07.831 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.828 7febd287c700 1 --2- 172.21.2.112:0/3054390986 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7febb8018730 0x7febb801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:07.831 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.828 7febd287c700 1 --2- 172.21.2.112:0/3054390986 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7febb8018730 0x7febb801ac10 crc :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:07.856 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.857 7febd92e8700 1 -- 172.21.2.112:0/3054390986 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febd418a9d0 con 0x7febd4187eb0 2021-02-10T11:15:07.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.861 7febcbfff700 1 -- 172.21.2.112:0/3054390986 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7febbc006bb0 con 0x7febd4187eb0 2021-02-10T11:15:07.994 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:07.996 7febd92e8700 1 -- 172.21.2.112:0/3054390986 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_async_subvolume_rm"]} v 0) v1 -- 0x7febd418a9d0 con 0x7febd4187eb0 2021-02-10T11:15:08.867 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:08.868 7febcbfff700 1 -- 172.21.2.112:0/3054390986 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_async_subvolume_rm"]}]=0 v71) v1 ==== 143+0+0 (secure 0 0 0) 0x7febbc000ca0 con 0x7febd4187eb0 2021-02-10T11:15:08.914 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:08.916 7febd92e8700 1 -- 172.21.2.112:0/3054390986 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7febb8018730 msgr2=0x7febb801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:08.914 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:08.916 7febd92e8700 1 --2- 172.21.2.112:0/3054390986 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7febb8018730 0x7febb801ac10 crc :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:08.914 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:08.916 7febd92e8700 1 -- 172.21.2.112:0/3054390986 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7febd4187eb0 msgr2=0x7febd40b7f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:08.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:08.916 7febd92e8700 1 --2- 172.21.2.112:0/3054390986 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7febd4187eb0 0x7febd40b7f80 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7febbc00b590 tx=0x7febbc00b930).stop 2021-02-10T11:15:08.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:08.916 7febd92e8700 1 -- 172.21.2.112:0/3054390986 shutdown_connections 2021-02-10T11:15:08.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:08.916 7febd92e8700 1 --2- 172.21.2.112:0/3054390986 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7febb8018730 0x7febb801ac10 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:08.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:08.916 7febd92e8700 1 --2- 172.21.2.112:0/3054390986 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7febd40b85a0 0x7febd40b89a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:08.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:08.916 7febd92e8700 1 --2- 172.21.2.112:0/3054390986 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7febd4095a20 0x7febd4095e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:08.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:08.916 7febd92e8700 1 --2- 172.21.2.112:0/3054390986 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7febd4187eb0 0x7febd40b7f80 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:08.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:08.916 7febd92e8700 1 -- 172.21.2.112:0/3054390986 >> 172.21.2.112:0/3054390986 conn(0x7febd41b2710 msgr2=0x7febd4084dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:08.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:08.916 7febd92e8700 1 -- 172.21.2.112:0/3054390986 shutdown_connections 2021-02-10T11:15:08.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:08.916 7febd92e8700 1 -- 172.21.2.112:0/3054390986 wait complete. 2021-02-10T11:15:08.922 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:15:09.357 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:09.340 7f850e1df700 1 Processor -- start 2021-02-10T11:15:09.357 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:09.341 7f850e1df700 1 -- start start 2021-02-10T11:15:09.357 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:09.341 7f850e1df700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f85081b0470 0x7f85081b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:09.358 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:09.341 7f850e1df700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f85080ef610 0x7f85080efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:09.358 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:09.341 7f850e1df700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85080f0b50 con 0x7f85080f0030 2021-02-10T11:15:09.358 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:09.341 7f850e1df700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85080e52e0 con 0x7f85081b0470 2021-02-10T11:15:09.358 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:09.341 7f850e1df700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85080e54d0 con 0x7f85080ef610 2021-02-10T11:15:09.358 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:09.341 7f8507fff700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f85080f0030 0x7f85080f10d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:42596/0 (socket says 172.21.2.112:42596) 2021-02-10T11:15:09.358 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:09.341 7f8507fff700 1 -- 172.21.2.112:0/3211124282 learned_addr learned my addr 172.21.2.112:0/3211124282 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:15:09.358 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:09.342 7f85067fc700 1 -- 172.21.2.112:0/3211124282 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1580224027 0 0) 0x7f8500001500 con 0x7f85080f0030 2021-02-10T11:15:09.358 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:09.342 7f8506ffd700 1 --2- 172.21.2.112:0/3211124282 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f85080ef610 0x7f85080efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:09.358 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:09.342 7f85067fc700 1 -- 172.21.2.112:0/3211124282 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84f4002fe0 con 0x7f85080f0030 2021-02-10T11:15:09.358 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:09.342 7f85077fe700 1 --2- 172.21.2.112:0/3211124282 >> v2:172.21.2.120:3300/0 conn(0x7f85081b0470 0x7f85081b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:09.358 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:09.342 7f85067fc700 1 -- 172.21.2.112:0/3211124282 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2169326795 0 0) 0x7f8500001500 con 0x7f85080f0030 2021-02-10T11:15:09.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:09.342 7f85067fc700 1 -- 172.21.2.112:0/3211124282 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f85080ef610 msgr2=0x7f85080efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:15:09.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:09.342 7f85067fc700 1 --2- 172.21.2.112:0/3211124282 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f85080ef610 0x7f85080efa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:09.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:09.342 7f85067fc700 1 -- 172.21.2.112:0/3211124282 >> v2:172.21.2.120:3300/0 conn(0x7f85081b0470 msgr2=0x7f85081b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:15:09.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:09.342 7f85067fc700 1 --2- 172.21.2.112:0/3211124282 >> v2:172.21.2.120:3300/0 conn(0x7f85081b0470 0x7f85081b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:09.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:09.342 7f85067fc700 1 -- 172.21.2.112:0/3211124282 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85080f0430 con 0x7f85080f0030 2021-02-10T11:15:09.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:09.342 7f85067fc700 1 -- 172.21.2.112:0/3211124282 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f8500001a00 con 0x7f85080f0030 2021-02-10T11:15:09.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:09.342 7f85067fc700 1 -- 172.21.2.112:0/3211124282 >> v1:172.21.2.112:6789/0 conn(0x7f85080f0030 legacy=0x7f85080f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:09.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:09.342 7f85067fc700 1 --2- 172.21.2.112:0/3211124282 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f84f4003ac0 0x7f84f4005ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:09.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:09.342 7f85067fc700 1 --2- 172.21.2.112:0/3211124282 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f84f404a780 0x7f84f404cc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:09.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:09.342 7f85067fc700 1 --2- 172.21.2.112:0/3211124282 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f84f404d520 0x7f84f404fa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:09.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:09.342 7f85067fc700 1 -- 172.21.2.112:0/3211124282 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84f40502f0 con 0x7f84f404d520 2021-02-10T11:15:09.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:09.342 7f85067fc700 1 -- 172.21.2.112:0/3211124282 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84f4050510 con 0x7f84f404a780 2021-02-10T11:15:09.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:09.342 7f85067fc700 1 -- 172.21.2.112:0/3211124282 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84f4050730 con 0x7f84f4003ac0 2021-02-10T11:15:09.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:09.343 7f8507fff700 1 --2- 172.21.2.112:0/3211124282 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f84f404d520 0x7f84f404fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:09.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:09.343 7f85077fe700 1 --2- 172.21.2.112:0/3211124282 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f84f4003ac0 0x7f84f4005ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:09.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:09.343 7f8506ffd700 1 --2- 172.21.2.112:0/3211124282 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f84f404a780 0x7f84f404cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:09.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:09.343 7f8507fff700 1 -- 172.21.2.112:0/3211124282 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f84f4003ac0 msgr2=0x7f84f4005ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:15:09.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:09.343 7f8507fff700 1 --2- 172.21.2.112:0/3211124282 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f84f4003ac0 0x7f84f4005ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:09.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:09.343 7f8507fff700 1 -- 172.21.2.112:0/3211124282 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f84f404a780 msgr2=0x7f84f404cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:15:09.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:09.343 7f8507fff700 1 --2- 172.21.2.112:0/3211124282 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f84f404a780 0x7f84f404cc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:09.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:09.343 7f8507fff700 1 -- 172.21.2.112:0/3211124282 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84f40509d0 con 0x7f84f404d520 2021-02-10T11:15:09.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:09.343 7f8507fff700 1 --2- 172.21.2.112:0/3211124282 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f84f404d520 0x7f84f404fa40 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7f8500007ce0 tx=0x7f850000a310).ready entity=mon.0 client_cookie=64e13aa8ea20dded server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:09.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:09.343 7f85067fc700 1 -- 172.21.2.112:0/3211124282 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8500007d70 con 0x7f84f404d520 2021-02-10T11:15:09.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:09.343 7f85067fc700 1 -- 172.21.2.112:0/3211124282 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85000070c0 con 0x7f84f404d520 2021-02-10T11:15:09.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:09.343 7f85067fc700 1 -- 172.21.2.112:0/3211124282 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f85000075e0 con 0x7f84f404d520 2021-02-10T11:15:09.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:09.343 7f850e1df700 1 -- 172.21.2.112:0/3211124282 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f84f404d520 msgr2=0x7f84f404fa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:09.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:09.343 7f850e1df700 1 --2- 172.21.2.112:0/3211124282 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f84f404d520 0x7f84f404fa40 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7f8500007ce0 tx=0x7f850000a310).stop 2021-02-10T11:15:09.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:09.343 7f8507fff700 1 -- 172.21.2.112:0/3211124282 reap_dead start 2021-02-10T11:15:09.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:09.344 7f850e1df700 1 -- 172.21.2.112:0/3211124282 shutdown_connections 2021-02-10T11:15:09.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:09.344 7f850e1df700 1 -- 172.21.2.112:0/3211124282 >> 172.21.2.112:0/3211124282 conn(0x7f85080ce3e0 msgr2=0x7f85080e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:09.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:09.344 7f850e1df700 1 -- 172.21.2.112:0/3211124282 shutdown_connections 2021-02-10T11:15:09.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:09.344 7f850e1df700 1 -- 172.21.2.112:0/3211124282 wait complete. 2021-02-10T11:15:09.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:09.359 7f850e1df700 1 Processor -- start 2021-02-10T11:15:09.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:09.359 7f850e1df700 1 -- start start 2021-02-10T11:15:09.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:09.359 7f850e1df700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f85080a0910 0x7f8508105d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:09.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:09.359 7f850e1df700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f85081a7a10 0x7f85081a7e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:09.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:09.359 7f850e1df700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8508093770 0x7f8508093b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:09.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:09.359 7f850e1df700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8508106390 con 0x7f8508093770 2021-02-10T11:15:09.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:09.359 7f850e1df700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8508094190 con 0x7f85080a0910 2021-02-10T11:15:09.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:09.359 7f850e1df700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85080943b0 con 0x7f85081a7a10 2021-02-10T11:15:09.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:09.360 7f8507fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8508093770 0x7f8508093b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:09.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:09.360 7f8507fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8508093770 0x7f8508093b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:43148/0 (socket says 172.21.2.112:43148) 2021-02-10T11:15:09.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:09.360 7f8507fff700 1 -- 172.21.2.112:0/2952414007 learned_addr learned my addr 172.21.2.112:0/2952414007 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:15:09.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:09.360 7f8507fff700 1 -- 172.21.2.112:0/2952414007 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f85081a7a10 msgr2=0x7f85081a7e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:09.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:09.360 7f8507fff700 1 --2- 172.21.2.112:0/2952414007 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f85081a7a10 0x7f85081a7e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:09.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:09.360 7f8507fff700 1 -- 172.21.2.112:0/2952414007 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f85080a0910 msgr2=0x7f8508105d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:09.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:09.360 7f8507fff700 1 --2- 172.21.2.112:0/2952414007 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f85080a0910 0x7f8508105d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:09.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:09.360 7f8507fff700 1 -- 172.21.2.112:0/2952414007 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8508106c30 con 0x7f8508093770 2021-02-10T11:15:09.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:09.360 7f8507fff700 1 --2- 172.21.2.112:0/2952414007 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8508093770 0x7f8508093b70 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f8500001be0 tx=0x7f850000aca0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:09.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:09.361 7f84fffff700 1 -- 172.21.2.112:0/2952414007 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8500001900 con 0x7f8508093770 2021-02-10T11:15:09.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:09.361 7f84fffff700 1 -- 172.21.2.112:0/2952414007 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8500004930 con 0x7f8508093770 2021-02-10T11:15:09.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:09.361 7f84fffff700 1 -- 172.21.2.112:0/2952414007 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8500007020 con 0x7f8508093770 2021-02-10T11:15:09.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:09.361 7f850e1df700 1 -- 172.21.2.112:0/2952414007 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8508106c30 con 0x7f8508093770 2021-02-10T11:15:09.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:09.361 7f850e1df700 1 -- 172.21.2.112:0/2952414007 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8508106c30 con 0x7f8508093770 2021-02-10T11:15:09.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:09.362 7f84fffff700 1 -- 172.21.2.112:0/2952414007 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f850000ade0 con 0x7f8508093770 2021-02-10T11:15:09.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:09.362 7f84fffff700 1 --2- 172.21.2.112:0/2952414007 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f84ec0186c0 0x7f84ec01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:09.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:09.362 7f84fffff700 1 -- 172.21.2.112:0/2952414007 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f85000328d0 con 0x7f8508093770 2021-02-10T11:15:09.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:09.362 7f85077fe700 1 --2- 172.21.2.112:0/2952414007 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f84ec0186c0 0x7f84ec01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:09.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:09.363 7f85077fe700 1 --2- 172.21.2.112:0/2952414007 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f84ec0186c0 0x7f84ec01abc0 crc :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:09.389 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:09.391 7f850e1df700 1 -- 172.21.2.112:0/2952414007 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8508105b40 con 0x7f8508093770 2021-02-10T11:15:09.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:09.394 7f84fffff700 1 -- 172.21.2.112:0/2952414007 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8500004720 con 0x7f8508093770 2021-02-10T11:15:09.542 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:09.544 7f850e1df700 1 -- 172.21.2.112:0/2952414007 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f85080f0b50 con 0x7f8508093770 2021-02-10T11:15:09.545 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:09.547 7f84fffff700 1 -- 172.21.2.112:0/2952414007 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v2)=0 v2) v1 ==== 122+0+0 (secure 0 0 0) 0x7f850000ade0 con 0x7f8508093770 2021-02-10T11:15:09.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:09.552 7f850e1df700 1 -- 172.21.2.112:0/2952414007 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f84ec0186c0 msgr2=0x7f84ec01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:09.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:09.552 7f850e1df700 1 --2- 172.21.2.112:0/2952414007 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f84ec0186c0 0x7f84ec01abc0 crc :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:09.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:09.552 7f850e1df700 1 -- 172.21.2.112:0/2952414007 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8508093770 msgr2=0x7f8508093b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:09.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:09.552 7f850e1df700 1 --2- 172.21.2.112:0/2952414007 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8508093770 0x7f8508093b70 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f8500001be0 tx=0x7f850000aca0).stop 2021-02-10T11:15:09.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:09.553 7f850e1df700 1 -- 172.21.2.112:0/2952414007 shutdown_connections 2021-02-10T11:15:09.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:09.553 7f850e1df700 1 --2- 172.21.2.112:0/2952414007 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f84ec0186c0 0x7f84ec01abc0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:09.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:09.553 7f850e1df700 1 --2- 172.21.2.112:0/2952414007 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8508093770 0x7f8508093b70 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:09.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:09.553 7f850e1df700 1 --2- 172.21.2.112:0/2952414007 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f85081a7a10 0x7f85081a7e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:09.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:09.553 7f850e1df700 1 --2- 172.21.2.112:0/2952414007 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f85080a0910 0x7f8508105d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:09.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:09.553 7f850e1df700 1 -- 172.21.2.112:0/2952414007 >> 172.21.2.112:0/2952414007 conn(0x7f8508095ef0 msgr2=0x7f8508094e40 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:09.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:09.553 7f850e1df700 1 -- 172.21.2.112:0/2952414007 shutdown_connections 2021-02-10T11:15:09.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:09.553 7f850e1df700 1 -- 172.21.2.112:0/2952414007 wait complete. 2021-02-10T11:15:09.559 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:15:09.560 DEBUG:teuthology.orchestra.run.gibba012:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:15:09.920 DEBUG:teuthology.orchestra.run.gibba020:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:15:10.271 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:15:10.272 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:15:10.272 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:15:10.272 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:15:10.606 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:15:10.606 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:15:10.607 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:15:10.607 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:15:10.607 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:15:10.940 INFO:teuthology.orchestra.run.gibba012.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.0’: Transport endpoint is not connected 2021-02-10T11:15:10.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:15:10.941 WARNING:tasks.cephfs.fuse_mount:Found stale moutn point at /home/ubuntu/cephtest/mnt.0 2021-02-10T11:15:10.941 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba012.front.sepia.ceph.com... 2021-02-10T11:15:10.941 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:15:10.942 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:15:11.339 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:15:11.340 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:15:11.340 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:15:11.673 INFO:teuthology.orchestra.run.gibba012.stdout:ext2/ext3 2021-02-10T11:15:11.674 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:15:11.674 INFO:teuthology.orchestra.run:waiting for 900 2021-02-10T11:15:11.674 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:15:11.674 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:15:12.023 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:15:12.024 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:15:12.024 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:15:12.357 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:15:12.357 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:15:12.358 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:15:12.358 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:15:12.358 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:15:12.694 INFO:teuthology.orchestra.run.gibba012.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: Transport endpoint is not connected 2021-02-10T11:15:12.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:15:12.694 WARNING:tasks.cephfs.fuse_mount:Found stale moutn point at /home/ubuntu/cephtest/mnt.1 2021-02-10T11:15:12.694 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba012.front.sepia.ceph.com... 2021-02-10T11:15:12.695 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:15:12.695 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:15:13.044 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:15:13.044 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:15:13.044 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:15:13.378 INFO:teuthology.orchestra.run.gibba012.stdout:ext2/ext3 2021-02-10T11:15:13.379 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:15:13.379 INFO:teuthology.orchestra.run:waiting for 900 2021-02-10T11:15:13.379 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:15:13.379 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:15:13.716 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:15:13.717 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:15:13.717 DEBUG:teuthology.orchestra.run.gibba020:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:15:14.052 INFO:teuthology.orchestra.run.gibba020.stdout:fuseblk 2021-02-10T11:15:14.052 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.2 2021-02-10T11:15:14.053 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:15:14.053 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:15:14.053 DEBUG:teuthology.orchestra.run.gibba020:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:15:14.386 INFO:teuthology.orchestra.run.gibba020.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: Transport endpoint is not connected 2021-02-10T11:15:14.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:15:14.387 WARNING:tasks.cephfs.fuse_mount:Found stale moutn point at /home/ubuntu/cephtest/mnt.2 2021-02-10T11:15:14.387 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba020.front.sepia.ceph.com... 2021-02-10T11:15:14.387 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:15:14.387 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:15:14.750 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:15:14.751 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:15:14.751 DEBUG:teuthology.orchestra.run.gibba020:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:15:15.085 INFO:teuthology.orchestra.run.gibba020.stdout:ext2/ext3 2021-02-10T11:15:15.086 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:15:15.086 INFO:teuthology.orchestra.run:waiting for 900 2021-02-10T11:15:15.086 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:15:15.086 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:15:15.430 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:15:15.431 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:15:15.431 DEBUG:teuthology.orchestra.run.gibba020:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:15:15.766 INFO:teuthology.orchestra.run.gibba020.stdout:fuseblk 2021-02-10T11:15:15.766 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.3 2021-02-10T11:15:15.767 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:15:15.767 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:15:15.767 DEBUG:teuthology.orchestra.run.gibba020:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:15:16.102 INFO:teuthology.orchestra.run.gibba020.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: Transport endpoint is not connected 2021-02-10T11:15:16.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:15:16.102 WARNING:tasks.cephfs.fuse_mount:Found stale moutn point at /home/ubuntu/cephtest/mnt.3 2021-02-10T11:15:16.103 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba020.front.sepia.ceph.com... 2021-02-10T11:15:16.103 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:15:16.103 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:15:16.460 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:15:16.461 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:15:16.461 DEBUG:teuthology.orchestra.run.gibba020:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:15:16.794 INFO:teuthology.orchestra.run.gibba020.stdout:ext2/ext3 2021-02-10T11:15:16.794 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:15:16.795 INFO:teuthology.orchestra.run:waiting for 900 2021-02-10T11:15:16.795 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:15:16.795 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:15:17.129 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json-pretty 2021-02-10T11:15:17.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:17.541 7fb5e7b52700 1 Processor -- start 2021-02-10T11:15:17.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:17.541 7fb5e7b52700 1 -- start start 2021-02-10T11:15:17.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:17.542 7fb5e7b52700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fb5e00e5470 0x7fb5e007e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:17.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:17.542 7fb5e7b52700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fb5e007f300 0x7fb5e007f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:17.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:17.542 7fb5e7b52700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5e007ed80 con 0x7fb5e00e5070 2021-02-10T11:15:17.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:17.542 7fb5e7b52700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5e01b4010 con 0x7fb5e00e5470 2021-02-10T11:15:17.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:17.542 7fb5e7b52700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5e00d1790 con 0x7fb5e007f300 2021-02-10T11:15:17.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:17.542 7fb5e516d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fb5e00e5470 0x7fb5e007e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:17.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:17.542 7fb5e596e700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fb5e00e5070 0x7fb5e01b3820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:42614/0 (socket says 172.21.2.112:42614) 2021-02-10T11:15:17.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:17.542 7fb5e596e700 1 -- 172.21.2.112:0/2334960687 learned_addr learned my addr 172.21.2.112:0/2334960687 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:15:17.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:17.542 7fb5e616f700 1 --2- 172.21.2.112:0/2334960687 >> v2:172.21.2.120:3300/0 conn(0x7fb5e007f300 0x7fb5e007f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:17.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:17.543 7fb5e516d700 1 -- 172.21.2.112:0/2334960687 >> v2:172.21.2.120:3300/0 conn(0x7fb5e007f300 msgr2=0x7fb5e007f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:15:17.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:17.543 7fb5e516d700 1 --2- 172.21.2.112:0/2334960687 >> v2:172.21.2.120:3300/0 conn(0x7fb5e007f300 0x7fb5e007f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:17.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:17.543 7fb5e516d700 1 -- 172.21.2.112:0/2334960687 >> v1:172.21.2.112:6789/0 conn(0x7fb5e00e5070 legacy=0x7fb5e01b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:17.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:17.543 7fb5e516d700 1 -- 172.21.2.112:0/2334960687 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5e00d19b0 con 0x7fb5e00e5470 2021-02-10T11:15:17.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:17.543 7fb5e516d700 1 --2- 172.21.2.112:0/2334960687 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb5e00e5470 0x7fb5e007e6a0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7fb5d0006650 tx=0x7fb5d0009a00).ready entity=mon.0 client_cookie=ec76915506a84642 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:17.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:17.543 7fb5e496c700 1 -- 172.21.2.112:0/2334960687 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb5d0009df0 con 0x7fb5e00e5470 2021-02-10T11:15:17.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:17.543 7fb5e496c700 1 -- 172.21.2.112:0/2334960687 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5d00009b0 con 0x7fb5e00e5470 2021-02-10T11:15:17.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:17.543 7fb5e496c700 1 -- 172.21.2.112:0/2334960687 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb5d000c2b0 con 0x7fb5e00e5470 2021-02-10T11:15:17.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:17.543 7fb5e7b52700 1 -- 172.21.2.112:0/2334960687 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb5e00e5470 msgr2=0x7fb5e007e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:17.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:17.543 7fb5e7b52700 1 --2- 172.21.2.112:0/2334960687 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb5e00e5470 0x7fb5e007e6a0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7fb5d0006650 tx=0x7fb5d0009a00).stop 2021-02-10T11:15:17.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:17.544 7fb5e7b52700 1 -- 172.21.2.112:0/2334960687 shutdown_connections 2021-02-10T11:15:17.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:17.544 7fb5e7b52700 1 --2- 172.21.2.112:0/2334960687 >> v2:172.21.2.120:3300/0 conn(0x7fb5e007f300 0x7fb5e007f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:17.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:17.544 7fb5e7b52700 1 --2- 172.21.2.112:0/2334960687 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb5e00e5470 0x7fb5e007e6a0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:17.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:17.544 7fb5e7b52700 1 -- 172.21.2.112:0/2334960687 >> 172.21.2.112:0/2334960687 conn(0x7fb5e00ce690 msgr2=0x7fb5e00e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:17.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:17.544 7fb5e7b52700 1 -- 172.21.2.112:0/2334960687 shutdown_connections 2021-02-10T11:15:17.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:17.544 7fb5e7b52700 1 -- 172.21.2.112:0/2334960687 wait complete. 2021-02-10T11:15:17.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:17.559 7fb5e7b52700 1 Processor -- start 2021-02-10T11:15:17.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:17.559 7fb5e7b52700 1 -- start start 2021-02-10T11:15:17.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:17.559 7fb5e7b52700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb5e008d400 0x7fb5e0081230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:17.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:17.559 7fb5e7b52700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb5e0081850 0x7fb5e00acf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:17.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:17.559 7fb5e7b52700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb5e0081c50 0x7fb5e00ad5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:17.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:17.559 7fb5e7b52700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5e0082110 con 0x7fb5e0081c50 2021-02-10T11:15:17.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:17.559 7fb5e7b52700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5e008d800 con 0x7fb5e008d400 2021-02-10T11:15:17.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:17.559 7fb5e7b52700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5e00adbc0 con 0x7fb5e0081850 2021-02-10T11:15:17.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:17.559 7fb5e616f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb5e0081c50 0x7fb5e00ad5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:17.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:17.559 7fb5e616f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb5e0081c50 0x7fb5e00ad5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:43162/0 (socket says 172.21.2.112:43162) 2021-02-10T11:15:17.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:17.559 7fb5e616f700 1 -- 172.21.2.112:0/1757868126 learned_addr learned my addr 172.21.2.112:0/1757868126 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:15:17.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:17.560 7fb5e616f700 1 -- 172.21.2.112:0/1757868126 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb5e0081850 msgr2=0x7fb5e00acf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:17.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:17.560 7fb5e616f700 1 --2- 172.21.2.112:0/1757868126 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb5e0081850 0x7fb5e00acf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:17.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:17.560 7fb5e616f700 1 -- 172.21.2.112:0/1757868126 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb5e008d400 msgr2=0x7fb5e0081230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:17.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:17.560 7fb5e616f700 1 --2- 172.21.2.112:0/1757868126 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb5e008d400 0x7fb5e0081230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:17.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:17.560 7fb5e616f700 1 -- 172.21.2.112:0/1757868126 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5e0187fc0 con 0x7fb5e0081c50 2021-02-10T11:15:17.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:17.560 7fb5e616f700 1 --2- 172.21.2.112:0/1757868126 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb5e0081c50 0x7fb5e00ad5a0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7fb5d8009980 tx=0x7fb5d800bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:17.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:17.560 7fb5d67fc700 1 -- 172.21.2.112:0/1757868126 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb5d800c2f0 con 0x7fb5e0081c50 2021-02-10T11:15:17.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:17.560 7fb5d67fc700 1 -- 172.21.2.112:0/1757868126 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5d800c5b0 con 0x7fb5e0081c50 2021-02-10T11:15:17.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:17.560 7fb5d67fc700 1 -- 172.21.2.112:0/1757868126 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb5d800cad0 con 0x7fb5e0081c50 2021-02-10T11:15:17.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:17.560 7fb5e7b52700 1 -- 172.21.2.112:0/1757868126 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5e0187fc0 con 0x7fb5e0081c50 2021-02-10T11:15:17.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:17.560 7fb5e7b52700 1 -- 172.21.2.112:0/1757868126 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5e0187fc0 con 0x7fb5e0081c50 2021-02-10T11:15:17.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:17.561 7fb5d67fc700 1 -- 172.21.2.112:0/1757868126 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb5d80011f0 con 0x7fb5e0081c50 2021-02-10T11:15:17.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:17.561 7fb5d67fc700 1 --2- 172.21.2.112:0/1757868126 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb5c80186c0 0x7fb5c801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:17.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:17.561 7fb5d67fc700 1 -- 172.21.2.112:0/1757868126 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fb5d8032930 con 0x7fb5e0081c50 2021-02-10T11:15:17.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:17.562 7fb5e596e700 1 --2- 172.21.2.112:0/1757868126 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb5c80186c0 0x7fb5c801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:17.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:17.562 7fb5e596e700 1 --2- 172.21.2.112:0/1757868126 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb5c80186c0 0x7fb5c801abc0 crc :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:17.590 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:17.592 7fb5e7b52700 1 -- 172.21.2.112:0/1757868126 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5e007ed80 con 0x7fb5e0081c50 2021-02-10T11:15:17.592 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:17.594 7fb5d67fc700 1 -- 172.21.2.112:0/1757868126 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb5d800be10 con 0x7fb5e0081c50 2021-02-10T11:15:17.717 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:17.718 7fb5e7b52700 1 -- 172.21.2.112:0/1757868126 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7fb5e007ed80 con 0x7fb5e0081c50 2021-02-10T11:15:17.717 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:17.719 7fb5d67fc700 1 -- 172.21.2.112:0/1757868126 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v20) v1 ==== 81+0+30360 (secure 0 0 0) 0x7fb5d80011f0 con 0x7fb5e0081c50 2021-02-10T11:15:17.718 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:15:17.718 INFO:teuthology.orchestra.run.gibba012.stdout:{ 2021-02-10T11:15:17.718 INFO:teuthology.orchestra.run.gibba012.stdout: "epoch": 20, 2021-02-10T11:15:17.718 INFO:teuthology.orchestra.run.gibba012.stdout: "fsid": "f14f004e-0699-4584-ad81-3754258493de", 2021-02-10T11:15:17.718 INFO:teuthology.orchestra.run.gibba012.stdout: "created": "2021-02-10 11:13:26.501424", 2021-02-10T11:15:17.718 INFO:teuthology.orchestra.run.gibba012.stdout: "modified": "2021-02-10 11:14:04.629689", 2021-02-10T11:15:17.718 INFO:teuthology.orchestra.run.gibba012.stdout: "last_up_change": "2021-02-10 11:13:52.780316", 2021-02-10T11:15:17.718 INFO:teuthology.orchestra.run.gibba012.stdout: "last_in_change": "2021-02-10 11:13:52.780316", 2021-02-10T11:15:17.719 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-02-10T11:15:17.719 INFO:teuthology.orchestra.run.gibba012.stdout: "flags_num": 5799936, 2021-02-10T11:15:17.719 INFO:teuthology.orchestra.run.gibba012.stdout: "flags_set": [ 2021-02-10T11:15:17.719 INFO:teuthology.orchestra.run.gibba012.stdout: "pglog_hardlimit", 2021-02-10T11:15:17.719 INFO:teuthology.orchestra.run.gibba012.stdout: "purged_snapdirs", 2021-02-10T11:15:17.719 INFO:teuthology.orchestra.run.gibba012.stdout: "recovery_deletes", 2021-02-10T11:15:17.719 INFO:teuthology.orchestra.run.gibba012.stdout: "sortbitwise" 2021-02-10T11:15:17.719 INFO:teuthology.orchestra.run.gibba012.stdout: ], 2021-02-10T11:15:17.719 INFO:teuthology.orchestra.run.gibba012.stdout: "crush_version": 4, 2021-02-10T11:15:17.719 INFO:teuthology.orchestra.run.gibba012.stdout: "full_ratio": 0.94999998807907104, 2021-02-10T11:15:17.719 INFO:teuthology.orchestra.run.gibba012.stdout: "backfillfull_ratio": 0.89999997615814209, 2021-02-10T11:15:17.719 INFO:teuthology.orchestra.run.gibba012.stdout: "nearfull_ratio": 0.85000002384185791, 2021-02-10T11:15:17.720 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_snapshot": "", 2021-02-10T11:15:17.720 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_max": 3, 2021-02-10T11:15:17.720 INFO:teuthology.orchestra.run.gibba012.stdout: "max_osd": 8, 2021-02-10T11:15:17.720 INFO:teuthology.orchestra.run.gibba012.stdout: "require_min_compat_client": "jewel", 2021-02-10T11:15:17.720 INFO:teuthology.orchestra.run.gibba012.stdout: "min_compat_client": "jewel", 2021-02-10T11:15:17.720 INFO:teuthology.orchestra.run.gibba012.stdout: "require_osd_release": "nautilus", 2021-02-10T11:15:17.720 INFO:teuthology.orchestra.run.gibba012.stdout: "pools": [ 2021-02-10T11:15:17.720 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:15:17.720 INFO:teuthology.orchestra.run.gibba012.stdout: "pool": 1, 2021-02-10T11:15:17.720 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_name": "rbd", 2021-02-10T11:15:17.720 INFO:teuthology.orchestra.run.gibba012.stdout: "create_time": "2021-02-10 11:13:57.508758", 2021-02-10T11:15:17.720 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:15:17.721 INFO:teuthology.orchestra.run.gibba012.stdout: "flags_names": "hashpspool", 2021-02-10T11:15:17.721 INFO:teuthology.orchestra.run.gibba012.stdout: "type": 1, 2021-02-10T11:15:17.721 INFO:teuthology.orchestra.run.gibba012.stdout: "size": 2, 2021-02-10T11:15:17.721 INFO:teuthology.orchestra.run.gibba012.stdout: "min_size": 1, 2021-02-10T11:15:17.721 INFO:teuthology.orchestra.run.gibba012.stdout: "crush_rule": 0, 2021-02-10T11:15:17.721 INFO:teuthology.orchestra.run.gibba012.stdout: "object_hash": 2, 2021-02-10T11:15:17.721 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:15:17.721 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num": 8, 2021-02-10T11:15:17.721 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num": 8, 2021-02-10T11:15:17.721 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num_target": 8, 2021-02-10T11:15:17.721 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_target": 8, 2021-02-10T11:15:17.722 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_pending": 8, 2021-02-10T11:15:17.722 INFO:teuthology.orchestra.run.gibba012.stdout: "last_pg_merge_meta": { 2021-02-10T11:15:17.722 INFO:teuthology.orchestra.run.gibba012.stdout: "source_pgid": "0.0", 2021-02-10T11:15:17.722 INFO:teuthology.orchestra.run.gibba012.stdout: "ready_epoch": 0, 2021-02-10T11:15:17.722 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_started": 0, 2021-02-10T11:15:17.722 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_clean": 0, 2021-02-10T11:15:17.722 INFO:teuthology.orchestra.run.gibba012.stdout: "source_version": "0'0", 2021-02-10T11:15:17.722 INFO:teuthology.orchestra.run.gibba012.stdout: "target_version": "0'0" 2021-02-10T11:15:17.722 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:17.722 INFO:teuthology.orchestra.run.gibba012.stdout: "last_change": "16", 2021-02-10T11:15:17.723 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend": "0", 2021-02-10T11:15:17.723 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:15:17.723 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:15:17.723 INFO:teuthology.orchestra.run.gibba012.stdout: "auid": 0, 2021-02-10T11:15:17.723 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:15:17.723 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_seq": 0, 2021-02-10T11:15:17.723 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_epoch": 0, 2021-02-10T11:15:17.723 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_snaps": [], 2021-02-10T11:15:17.724 INFO:teuthology.orchestra.run.gibba012.stdout: "removed_snaps": "[]", 2021-02-10T11:15:17.724 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_bytes": 0, 2021-02-10T11:15:17.724 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_objects": 0, 2021-02-10T11:15:17.724 INFO:teuthology.orchestra.run.gibba012.stdout: "tiers": [], 2021-02-10T11:15:17.724 INFO:teuthology.orchestra.run.gibba012.stdout: "tier_of": -1, 2021-02-10T11:15:17.724 INFO:teuthology.orchestra.run.gibba012.stdout: "read_tier": -1, 2021-02-10T11:15:17.724 INFO:teuthology.orchestra.run.gibba012.stdout: "write_tier": -1, 2021-02-10T11:15:17.724 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_mode": "none", 2021-02-10T11:15:17.724 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_bytes": 0, 2021-02-10T11:15:17.724 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_objects": 0, 2021-02-10T11:15:17.724 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:15:17.724 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:15:17.725 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:15:17.725 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_flush_age": 0, 2021-02-10T11:15:17.725 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_evict_age": 0, 2021-02-10T11:15:17.725 INFO:teuthology.orchestra.run.gibba012.stdout: "erasure_code_profile": "", 2021-02-10T11:15:17.725 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_params": { 2021-02-10T11:15:17.725 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "none" 2021-02-10T11:15:17.725 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:17.725 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_period": 0, 2021-02-10T11:15:17.725 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_count": 0, 2021-02-10T11:15:17.725 INFO:teuthology.orchestra.run.gibba012.stdout: "use_gmt_hitset": true, 2021-02-10T11:15:17.725 INFO:teuthology.orchestra.run.gibba012.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:15:17.725 INFO:teuthology.orchestra.run.gibba012.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:15:17.725 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:15:17.726 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:15:17.726 INFO:teuthology.orchestra.run.gibba012.stdout: "grade_table": [], 2021-02-10T11:15:17.726 INFO:teuthology.orchestra.run.gibba012.stdout: "stripe_width": 0, 2021-02-10T11:15:17.726 INFO:teuthology.orchestra.run.gibba012.stdout: "expected_num_objects": 0, 2021-02-10T11:15:17.726 INFO:teuthology.orchestra.run.gibba012.stdout: "fast_read": false, 2021-02-10T11:15:17.726 INFO:teuthology.orchestra.run.gibba012.stdout: "options": {}, 2021-02-10T11:15:17.726 INFO:teuthology.orchestra.run.gibba012.stdout: "application_metadata": { 2021-02-10T11:15:17.726 INFO:teuthology.orchestra.run.gibba012.stdout: "rbd": {} 2021-02-10T11:15:17.726 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:15:17.726 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:17.726 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:15:17.726 INFO:teuthology.orchestra.run.gibba012.stdout: "pool": 2, 2021-02-10T11:15:17.726 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_name": "cephfs_metadata", 2021-02-10T11:15:17.727 INFO:teuthology.orchestra.run.gibba012.stdout: "create_time": "2021-02-10 11:14:01.400797", 2021-02-10T11:15:17.727 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:15:17.727 INFO:teuthology.orchestra.run.gibba012.stdout: "flags_names": "hashpspool", 2021-02-10T11:15:17.727 INFO:teuthology.orchestra.run.gibba012.stdout: "type": 1, 2021-02-10T11:15:17.727 INFO:teuthology.orchestra.run.gibba012.stdout: "size": 2, 2021-02-10T11:15:17.727 INFO:teuthology.orchestra.run.gibba012.stdout: "min_size": 1, 2021-02-10T11:15:17.727 INFO:teuthology.orchestra.run.gibba012.stdout: "crush_rule": 0, 2021-02-10T11:15:17.727 INFO:teuthology.orchestra.run.gibba012.stdout: "object_hash": 2, 2021-02-10T11:15:17.727 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:15:17.727 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num": 8, 2021-02-10T11:15:17.727 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num": 8, 2021-02-10T11:15:17.727 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num_target": 8, 2021-02-10T11:15:17.728 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_target": 8, 2021-02-10T11:15:17.728 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_pending": 8, 2021-02-10T11:15:17.728 INFO:teuthology.orchestra.run.gibba012.stdout: "last_pg_merge_meta": { 2021-02-10T11:15:17.728 INFO:teuthology.orchestra.run.gibba012.stdout: "source_pgid": "0.0", 2021-02-10T11:15:17.728 INFO:teuthology.orchestra.run.gibba012.stdout: "ready_epoch": 0, 2021-02-10T11:15:17.728 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_started": 0, 2021-02-10T11:15:17.728 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_clean": 0, 2021-02-10T11:15:17.728 INFO:teuthology.orchestra.run.gibba012.stdout: "source_version": "0'0", 2021-02-10T11:15:17.728 INFO:teuthology.orchestra.run.gibba012.stdout: "target_version": "0'0" 2021-02-10T11:15:17.728 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:17.728 INFO:teuthology.orchestra.run.gibba012.stdout: "last_change": "19", 2021-02-10T11:15:17.728 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend": "0", 2021-02-10T11:15:17.728 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:15:17.728 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:15:17.729 INFO:teuthology.orchestra.run.gibba012.stdout: "auid": 0, 2021-02-10T11:15:17.729 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:15:17.729 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_seq": 0, 2021-02-10T11:15:17.729 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_epoch": 0, 2021-02-10T11:15:17.729 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_snaps": [], 2021-02-10T11:15:17.729 INFO:teuthology.orchestra.run.gibba012.stdout: "removed_snaps": "[]", 2021-02-10T11:15:17.729 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_bytes": 0, 2021-02-10T11:15:17.729 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_objects": 0, 2021-02-10T11:15:17.729 INFO:teuthology.orchestra.run.gibba012.stdout: "tiers": [], 2021-02-10T11:15:17.729 INFO:teuthology.orchestra.run.gibba012.stdout: "tier_of": -1, 2021-02-10T11:15:17.729 INFO:teuthology.orchestra.run.gibba012.stdout: "read_tier": -1, 2021-02-10T11:15:17.729 INFO:teuthology.orchestra.run.gibba012.stdout: "write_tier": -1, 2021-02-10T11:15:17.729 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_mode": "none", 2021-02-10T11:15:17.730 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_bytes": 0, 2021-02-10T11:15:17.730 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_objects": 0, 2021-02-10T11:15:17.730 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:15:17.730 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:15:17.730 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:15:17.730 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_flush_age": 0, 2021-02-10T11:15:17.730 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_evict_age": 0, 2021-02-10T11:15:17.730 INFO:teuthology.orchestra.run.gibba012.stdout: "erasure_code_profile": "", 2021-02-10T11:15:17.730 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_params": { 2021-02-10T11:15:17.730 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "none" 2021-02-10T11:15:17.730 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:17.730 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_period": 0, 2021-02-10T11:15:17.730 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_count": 0, 2021-02-10T11:15:17.731 INFO:teuthology.orchestra.run.gibba012.stdout: "use_gmt_hitset": true, 2021-02-10T11:15:17.731 INFO:teuthology.orchestra.run.gibba012.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:15:17.731 INFO:teuthology.orchestra.run.gibba012.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:15:17.731 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:15:17.731 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:15:17.731 INFO:teuthology.orchestra.run.gibba012.stdout: "grade_table": [], 2021-02-10T11:15:17.731 INFO:teuthology.orchestra.run.gibba012.stdout: "stripe_width": 0, 2021-02-10T11:15:17.731 INFO:teuthology.orchestra.run.gibba012.stdout: "expected_num_objects": 0, 2021-02-10T11:15:17.731 INFO:teuthology.orchestra.run.gibba012.stdout: "fast_read": false, 2021-02-10T11:15:17.731 INFO:teuthology.orchestra.run.gibba012.stdout: "options": { 2021-02-10T11:15:17.731 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_autoscale_bias": 4, 2021-02-10T11:15:17.731 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_min": 16, 2021-02-10T11:15:17.731 INFO:teuthology.orchestra.run.gibba012.stdout: "recovery_priority": 5 2021-02-10T11:15:17.731 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:17.732 INFO:teuthology.orchestra.run.gibba012.stdout: "application_metadata": { 2021-02-10T11:15:17.732 INFO:teuthology.orchestra.run.gibba012.stdout: "cephfs": { 2021-02-10T11:15:17.732 INFO:teuthology.orchestra.run.gibba012.stdout: "metadata": "cephfs" 2021-02-10T11:15:17.732 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:15:17.732 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:15:17.732 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:17.732 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:15:17.732 INFO:teuthology.orchestra.run.gibba012.stdout: "pool": 3, 2021-02-10T11:15:17.732 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_name": "cephfs_data", 2021-02-10T11:15:17.732 INFO:teuthology.orchestra.run.gibba012.stdout: "create_time": "2021-02-10 11:14:02.622129", 2021-02-10T11:15:17.732 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:15:17.732 INFO:teuthology.orchestra.run.gibba012.stdout: "flags_names": "hashpspool", 2021-02-10T11:15:17.732 INFO:teuthology.orchestra.run.gibba012.stdout: "type": 1, 2021-02-10T11:15:17.733 INFO:teuthology.orchestra.run.gibba012.stdout: "size": 2, 2021-02-10T11:15:17.733 INFO:teuthology.orchestra.run.gibba012.stdout: "min_size": 1, 2021-02-10T11:15:17.733 INFO:teuthology.orchestra.run.gibba012.stdout: "crush_rule": 0, 2021-02-10T11:15:17.733 INFO:teuthology.orchestra.run.gibba012.stdout: "object_hash": 2, 2021-02-10T11:15:17.733 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:15:17.733 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num": 8, 2021-02-10T11:15:17.733 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num": 8, 2021-02-10T11:15:17.733 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num_target": 8, 2021-02-10T11:15:17.733 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_target": 8, 2021-02-10T11:15:17.733 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_pending": 8, 2021-02-10T11:15:17.733 INFO:teuthology.orchestra.run.gibba012.stdout: "last_pg_merge_meta": { 2021-02-10T11:15:17.733 INFO:teuthology.orchestra.run.gibba012.stdout: "source_pgid": "0.0", 2021-02-10T11:15:17.733 INFO:teuthology.orchestra.run.gibba012.stdout: "ready_epoch": 0, 2021-02-10T11:15:17.733 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_started": 0, 2021-02-10T11:15:17.734 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_clean": 0, 2021-02-10T11:15:17.734 INFO:teuthology.orchestra.run.gibba012.stdout: "source_version": "0'0", 2021-02-10T11:15:17.734 INFO:teuthology.orchestra.run.gibba012.stdout: "target_version": "0'0" 2021-02-10T11:15:17.734 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:17.734 INFO:teuthology.orchestra.run.gibba012.stdout: "last_change": "20", 2021-02-10T11:15:17.734 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend": "0", 2021-02-10T11:15:17.734 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:15:17.734 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:15:17.734 INFO:teuthology.orchestra.run.gibba012.stdout: "auid": 0, 2021-02-10T11:15:17.734 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:15:17.734 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_seq": 0, 2021-02-10T11:15:17.734 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_epoch": 0, 2021-02-10T11:15:17.734 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_snaps": [], 2021-02-10T11:15:17.735 INFO:teuthology.orchestra.run.gibba012.stdout: "removed_snaps": "[]", 2021-02-10T11:15:17.735 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_bytes": 0, 2021-02-10T11:15:17.735 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_objects": 0, 2021-02-10T11:15:17.735 INFO:teuthology.orchestra.run.gibba012.stdout: "tiers": [], 2021-02-10T11:15:17.735 INFO:teuthology.orchestra.run.gibba012.stdout: "tier_of": -1, 2021-02-10T11:15:17.735 INFO:teuthology.orchestra.run.gibba012.stdout: "read_tier": -1, 2021-02-10T11:15:17.735 INFO:teuthology.orchestra.run.gibba012.stdout: "write_tier": -1, 2021-02-10T11:15:17.735 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_mode": "none", 2021-02-10T11:15:17.735 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_bytes": 0, 2021-02-10T11:15:17.735 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_objects": 0, 2021-02-10T11:15:17.735 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:15:17.735 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:15:17.735 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:15:17.736 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_flush_age": 0, 2021-02-10T11:15:17.736 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_evict_age": 0, 2021-02-10T11:15:17.736 INFO:teuthology.orchestra.run.gibba012.stdout: "erasure_code_profile": "", 2021-02-10T11:15:17.736 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_params": { 2021-02-10T11:15:17.736 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "none" 2021-02-10T11:15:17.736 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:17.736 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_period": 0, 2021-02-10T11:15:17.736 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_count": 0, 2021-02-10T11:15:17.736 INFO:teuthology.orchestra.run.gibba012.stdout: "use_gmt_hitset": true, 2021-02-10T11:15:17.736 INFO:teuthology.orchestra.run.gibba012.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:15:17.736 INFO:teuthology.orchestra.run.gibba012.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:15:17.736 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:15:17.736 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:15:17.737 INFO:teuthology.orchestra.run.gibba012.stdout: "grade_table": [], 2021-02-10T11:15:17.737 INFO:teuthology.orchestra.run.gibba012.stdout: "stripe_width": 0, 2021-02-10T11:15:17.737 INFO:teuthology.orchestra.run.gibba012.stdout: "expected_num_objects": 0, 2021-02-10T11:15:17.737 INFO:teuthology.orchestra.run.gibba012.stdout: "fast_read": false, 2021-02-10T11:15:17.737 INFO:teuthology.orchestra.run.gibba012.stdout: "options": {}, 2021-02-10T11:15:17.737 INFO:teuthology.orchestra.run.gibba012.stdout: "application_metadata": { 2021-02-10T11:15:17.737 INFO:teuthology.orchestra.run.gibba012.stdout: "cephfs": { 2021-02-10T11:15:17.737 INFO:teuthology.orchestra.run.gibba012.stdout: "data": "cephfs" 2021-02-10T11:15:17.737 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:15:17.737 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:15:17.737 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:15:17.737 INFO:teuthology.orchestra.run.gibba012.stdout: ], 2021-02-10T11:15:17.737 INFO:teuthology.orchestra.run.gibba012.stdout: "osds": [ 2021-02-10T11:15:17.737 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:15:17.738 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 0, 2021-02-10T11:15:17.738 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "95444b8f-6370-4a1d-ae5f-df62651b77da", 2021-02-10T11:15:17.738 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:15:17.738 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:15:17.738 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:15:17.738 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:15:17.738 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:15:17.738 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:15:17.738 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:15:17.738 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 18, 2021-02-10T11:15:17.738 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:15:17.738 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:15:17.739 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:15:17.739 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:15:17.739 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:15:17.739 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:15:17.739 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6817", 2021-02-10T11:15:17.739 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:15:17.739 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:17.739 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:15:17.739 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:15:17.739 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6818", 2021-02-10T11:15:17.739 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:15:17.739 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:15:17.739 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:15:17.739 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:17.740 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:15:17.740 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:15:17.740 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:15:17.740 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:15:17.740 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6820", 2021-02-10T11:15:17.740 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:15:17.740 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:17.740 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:15:17.740 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:15:17.740 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6822", 2021-02-10T11:15:17.740 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:15:17.740 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:15:17.740 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:15:17.741 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:17.741 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:15:17.741 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:15:17.741 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:15:17.741 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:15:17.741 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6828", 2021-02-10T11:15:17.741 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:15:17.741 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:17.741 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:15:17.741 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:15:17.741 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6830", 2021-02-10T11:15:17.741 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:15:17.741 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:15:17.741 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:15:17.742 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:17.742 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:15:17.742 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:15:17.742 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:15:17.742 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:15:17.742 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6824", 2021-02-10T11:15:17.742 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:15:17.742 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:17.742 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:15:17.742 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:15:17.742 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6826", 2021-02-10T11:15:17.742 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:15:17.742 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:15:17.743 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:15:17.743 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:17.743 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.112:6818/36030", 2021-02-10T11:15:17.743 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.112:6822/36030", 2021-02-10T11:15:17.743 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.112:6830/36030", 2021-02-10T11:15:17.743 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.112:6826/36030", 2021-02-10T11:15:17.743 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:15:17.743 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:15:17.743 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:15:17.743 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:15:17.743 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:17.743 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:15:17.743 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 1, 2021-02-10T11:15:17.744 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "a23e217e-0d8e-486e-be07-712adfabd262", 2021-02-10T11:15:17.744 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:15:17.744 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:15:17.744 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:15:17.744 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:15:17.744 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:15:17.744 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:15:17.744 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:15:17.744 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 18, 2021-02-10T11:15:17.744 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:15:17.744 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:15:17.744 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:15:17.744 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:15:17.745 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:15:17.745 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:15:17.745 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6800", 2021-02-10T11:15:17.745 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:15:17.745 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:17.745 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:15:17.745 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:15:17.745 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6802", 2021-02-10T11:15:17.745 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:15:17.745 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:15:17.745 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:15:17.745 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:17.745 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:15:17.746 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:15:17.746 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:15:17.746 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:15:17.746 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6804", 2021-02-10T11:15:17.746 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:15:17.746 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:17.746 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:15:17.746 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:15:17.746 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6806", 2021-02-10T11:15:17.746 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:15:17.746 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:15:17.746 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:15:17.746 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:17.747 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:15:17.747 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:15:17.747 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:15:17.747 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:15:17.747 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6812", 2021-02-10T11:15:17.747 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:15:17.747 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:17.747 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:15:17.747 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:15:17.747 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6814", 2021-02-10T11:15:17.747 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:15:17.747 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:15:17.747 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:15:17.748 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:17.748 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:15:17.748 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:15:17.748 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:15:17.748 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:15:17.748 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6808", 2021-02-10T11:15:17.748 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:15:17.748 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:17.748 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:15:17.748 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:15:17.748 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6810", 2021-02-10T11:15:17.748 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:15:17.748 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:15:17.749 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:15:17.749 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:17.749 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.112:6802/36029", 2021-02-10T11:15:17.749 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.112:6806/36029", 2021-02-10T11:15:17.749 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.112:6814/36029", 2021-02-10T11:15:17.749 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.112:6810/36029", 2021-02-10T11:15:17.749 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:15:17.749 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:15:17.749 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:15:17.749 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:15:17.749 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:17.749 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:15:17.749 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 2, 2021-02-10T11:15:17.750 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "026afa8a-c0f1-4b35-9a76-53c00fe0c2fc", 2021-02-10T11:15:17.750 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:15:17.750 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:15:17.750 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:15:17.750 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:15:17.750 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:15:17.750 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:15:17.750 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:15:17.750 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 17, 2021-02-10T11:15:17.750 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:15:17.750 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:15:17.750 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:15:17.750 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:15:17.751 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:15:17.751 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:15:17.751 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6801", 2021-02-10T11:15:17.751 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:15:17.751 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:17.751 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:15:17.751 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:15:17.751 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6803", 2021-02-10T11:15:17.751 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:15:17.751 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:15:17.751 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:15:17.751 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:17.751 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:15:17.752 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:15:17.752 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:15:17.752 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:15:17.752 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6805", 2021-02-10T11:15:17.752 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:15:17.752 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:17.752 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:15:17.752 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:15:17.752 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6807", 2021-02-10T11:15:17.752 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:15:17.752 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:15:17.752 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:15:17.752 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:17.753 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:15:17.753 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:15:17.753 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:15:17.753 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:15:17.753 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6813", 2021-02-10T11:15:17.753 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:15:17.753 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:17.753 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:15:17.753 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:15:17.753 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6815", 2021-02-10T11:15:17.753 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:15:17.753 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:15:17.753 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:15:17.754 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:17.754 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:15:17.754 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:15:17.754 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:15:17.754 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:15:17.754 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6809", 2021-02-10T11:15:17.754 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:15:17.754 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:17.754 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:15:17.754 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:15:17.754 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6811", 2021-02-10T11:15:17.754 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:15:17.754 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:15:17.755 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:15:17.755 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:17.755 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.112:6803/36031", 2021-02-10T11:15:17.755 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.112:6807/36031", 2021-02-10T11:15:17.755 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.112:6815/36031", 2021-02-10T11:15:17.755 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.112:6811/36031", 2021-02-10T11:15:17.755 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:15:17.755 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:15:17.755 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:15:17.755 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:15:17.755 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:17.755 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:15:17.755 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 3, 2021-02-10T11:15:17.756 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "58795682-e00d-4173-bb84-f7075ca89a11", 2021-02-10T11:15:17.756 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:15:17.756 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:15:17.756 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:15:17.756 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:15:17.756 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:15:17.756 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:15:17.756 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:15:17.756 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 18, 2021-02-10T11:15:17.756 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:15:17.756 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:15:17.756 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:15:17.757 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:15:17.757 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:15:17.757 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:15:17.757 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6816", 2021-02-10T11:15:17.757 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:15:17.757 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:17.757 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:15:17.757 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:15:17.757 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6819", 2021-02-10T11:15:17.757 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:15:17.757 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:15:17.757 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:15:17.757 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:17.758 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:15:17.758 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:15:17.758 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:15:17.758 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:15:17.758 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6821", 2021-02-10T11:15:17.758 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:15:17.758 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:17.758 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:15:17.758 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:15:17.758 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6823", 2021-02-10T11:15:17.758 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:15:17.758 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:15:17.758 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:15:17.759 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:17.759 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:15:17.759 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:15:17.759 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:15:17.759 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:15:17.759 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6829", 2021-02-10T11:15:17.759 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:15:17.759 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:17.759 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:15:17.759 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:15:17.759 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6831", 2021-02-10T11:15:17.759 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:15:17.759 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:15:17.760 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:15:17.760 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:17.760 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:15:17.760 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:15:17.760 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:15:17.760 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:15:17.760 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6825", 2021-02-10T11:15:17.760 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:15:17.761 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:17.761 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:15:17.761 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:15:17.761 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6827", 2021-02-10T11:15:17.761 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:15:17.761 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:15:17.761 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:15:17.761 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:17.761 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.112:6819/36028", 2021-02-10T11:15:17.762 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.112:6823/36028", 2021-02-10T11:15:17.762 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.112:6831/36028", 2021-02-10T11:15:17.762 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.112:6827/36028", 2021-02-10T11:15:17.762 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:15:17.762 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:15:17.762 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:15:17.762 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:15:17.762 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:17.762 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:15:17.762 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 4, 2021-02-10T11:15:17.762 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "6763f8d6-53dc-4e45-bdf6-a6eabcf81e39", 2021-02-10T11:15:17.763 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:15:17.763 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:15:17.763 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:15:17.763 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:15:17.763 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:15:17.763 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:15:17.763 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:15:17.763 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 18, 2021-02-10T11:15:17.763 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:15:17.763 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:15:17.763 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:15:17.763 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:15:17.763 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:15:17.764 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:15:17.764 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6810", 2021-02-10T11:15:17.764 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:15:17.764 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:17.764 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:15:17.764 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:15:17.764 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6813", 2021-02-10T11:15:17.764 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:15:17.764 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:15:17.764 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:15:17.764 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:17.764 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:15:17.765 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:15:17.765 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:15:17.765 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:15:17.765 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6816", 2021-02-10T11:15:17.765 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:15:17.765 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:17.765 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:15:17.765 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:15:17.765 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6819", 2021-02-10T11:15:17.765 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:15:17.765 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:15:17.765 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:15:17.765 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:17.766 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:15:17.766 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:15:17.766 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:15:17.766 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:15:17.766 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6825", 2021-02-10T11:15:17.766 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:15:17.766 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:17.766 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:15:17.766 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:15:17.766 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6827", 2021-02-10T11:15:17.766 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:15:17.766 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:15:17.767 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:15:17.767 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:17.767 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:15:17.767 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:15:17.767 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:15:17.767 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:15:17.767 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6821", 2021-02-10T11:15:17.767 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:15:17.767 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:17.767 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:15:17.767 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:15:17.768 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6823", 2021-02-10T11:15:17.768 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:15:17.768 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:15:17.768 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:15:17.768 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:17.768 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.120:6813/36388", 2021-02-10T11:15:17.768 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.120:6819/36388", 2021-02-10T11:15:17.768 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.120:6827/36388", 2021-02-10T11:15:17.768 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.120:6823/36388", 2021-02-10T11:15:17.768 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:15:17.768 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:15:17.768 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:15:17.768 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:15:17.769 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:17.769 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:15:17.769 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 5, 2021-02-10T11:15:17.769 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "5d0e4dd6-0cb0-4bf8-8314-626acaa1b544", 2021-02-10T11:15:17.769 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:15:17.769 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:15:17.769 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:15:17.769 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:15:17.769 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:15:17.769 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:15:17.770 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:15:17.770 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 18, 2021-02-10T11:15:17.770 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:15:17.770 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:15:17.770 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:15:17.770 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:15:17.770 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:15:17.770 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:15:17.770 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6804", 2021-02-10T11:15:17.770 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:15:17.770 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:17.770 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:15:17.770 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:15:17.771 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6806", 2021-02-10T11:15:17.771 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:15:17.771 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:15:17.771 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:15:17.771 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:17.771 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:15:17.771 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:15:17.771 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:15:17.771 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:15:17.771 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6809", 2021-02-10T11:15:17.771 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:15:17.771 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:17.771 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:15:17.772 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:15:17.772 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6812", 2021-02-10T11:15:17.772 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:15:17.772 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:15:17.772 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:15:17.772 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:17.772 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:15:17.772 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:15:17.772 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:15:17.772 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:15:17.772 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6820", 2021-02-10T11:15:17.772 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:15:17.772 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:17.773 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:15:17.773 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:15:17.773 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6822", 2021-02-10T11:15:17.773 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:15:17.773 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:15:17.773 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:15:17.773 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:17.773 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:15:17.773 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:15:17.773 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:15:17.773 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:15:17.773 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6815", 2021-02-10T11:15:17.773 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:15:17.774 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:17.774 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:15:17.774 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:15:17.774 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6818", 2021-02-10T11:15:17.774 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:15:17.774 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:15:17.774 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:15:17.774 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:17.774 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.120:6806/36387", 2021-02-10T11:15:17.774 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.120:6812/36387", 2021-02-10T11:15:17.774 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.120:6822/36387", 2021-02-10T11:15:17.774 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.120:6818/36387", 2021-02-10T11:15:17.774 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:15:17.775 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:15:17.775 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:15:17.775 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:15:17.775 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:17.775 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:15:17.775 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 6, 2021-02-10T11:15:17.775 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "b010e187-4541-4b00-80dd-f948f6ab8f15", 2021-02-10T11:15:17.775 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:15:17.775 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:15:17.775 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:15:17.775 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:15:17.775 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:15:17.775 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:15:17.775 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:15:17.776 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 17, 2021-02-10T11:15:17.776 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:15:17.776 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:15:17.776 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:15:17.776 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:15:17.776 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:15:17.776 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:15:17.776 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6824", 2021-02-10T11:15:17.776 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:15:17.776 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:17.776 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:15:17.776 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:15:17.776 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6826", 2021-02-10T11:15:17.777 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:15:17.777 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:15:17.777 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:15:17.777 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:17.777 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:15:17.777 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:15:17.777 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:15:17.777 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:15:17.777 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6828", 2021-02-10T11:15:17.777 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:15:17.777 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:17.777 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:15:17.778 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:15:17.778 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6829", 2021-02-10T11:15:17.778 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:15:17.778 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:15:17.778 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:15:17.778 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:17.778 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:15:17.778 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:15:17.778 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:15:17.778 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:15:17.778 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6832", 2021-02-10T11:15:17.779 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:15:17.779 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:17.779 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:15:17.779 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:15:17.779 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6833", 2021-02-10T11:15:17.779 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:15:17.779 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:15:17.779 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:15:17.779 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:17.779 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:15:17.779 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:15:17.780 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:15:17.780 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:15:17.780 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6830", 2021-02-10T11:15:17.780 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:15:17.780 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:17.780 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:15:17.780 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:15:17.780 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6831", 2021-02-10T11:15:17.780 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:15:17.780 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:15:17.780 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:15:17.780 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:17.780 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.120:6826/36385", 2021-02-10T11:15:17.781 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.120:6829/36385", 2021-02-10T11:15:17.781 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.120:6833/36385", 2021-02-10T11:15:17.781 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.120:6831/36385", 2021-02-10T11:15:17.781 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:15:17.781 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:15:17.781 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:15:17.781 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:15:17.781 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:17.781 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:15:17.781 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 7, 2021-02-10T11:15:17.781 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "671e2772-33b7-497e-8e44-2178bd850bcf", 2021-02-10T11:15:17.781 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:15:17.781 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:15:17.782 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:15:17.782 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:15:17.782 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:15:17.782 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:15:17.782 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:15:17.782 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 17, 2021-02-10T11:15:17.782 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:15:17.782 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:15:17.782 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:15:17.782 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:15:17.782 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:15:17.782 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:15:17.783 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6802", 2021-02-10T11:15:17.783 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:15:17.783 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:17.783 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:15:17.783 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:15:17.783 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6803", 2021-02-10T11:15:17.783 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:15:17.783 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:15:17.783 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:15:17.783 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:17.783 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:15:17.783 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:15:17.784 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:15:17.784 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:15:17.784 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6805", 2021-02-10T11:15:17.784 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:15:17.784 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:17.784 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:15:17.784 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:15:17.784 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6807", 2021-02-10T11:15:17.784 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:15:17.784 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:15:17.784 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:15:17.784 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:17.784 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:15:17.785 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:15:17.785 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:15:17.785 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:15:17.785 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6814", 2021-02-10T11:15:17.785 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:15:17.785 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:17.785 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:15:17.785 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:15:17.785 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6817", 2021-02-10T11:15:17.785 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:15:17.785 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:15:17.785 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:15:17.785 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:17.786 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:15:17.786 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:15:17.786 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:15:17.786 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:15:17.786 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6808", 2021-02-10T11:15:17.786 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:15:17.786 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:17.786 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:15:17.786 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:15:17.786 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6811", 2021-02-10T11:15:17.786 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:15:17.786 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:15:17.787 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:15:17.787 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:17.787 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.120:6803/36386", 2021-02-10T11:15:17.787 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.120:6807/36386", 2021-02-10T11:15:17.787 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.120:6817/36386", 2021-02-10T11:15:17.787 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.120:6811/36386", 2021-02-10T11:15:17.787 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:15:17.787 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:15:17.787 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:15:17.787 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:15:17.787 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:15:17.787 INFO:teuthology.orchestra.run.gibba012.stdout: ], 2021-02-10T11:15:17.787 INFO:teuthology.orchestra.run.gibba012.stdout: "osd_xinfo": [ 2021-02-10T11:15:17.788 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:15:17.788 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 0, 2021-02-10T11:15:17.788 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:15:17.788 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:15:17.788 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:15:17.788 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:15:17.788 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:15:17.788 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:17.788 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:15:17.788 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 1, 2021-02-10T11:15:17.788 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:15:17.788 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:15:17.788 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:15:17.788 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:15:17.789 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:15:17.789 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:17.789 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:15:17.789 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 2, 2021-02-10T11:15:17.789 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:15:17.789 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:15:17.789 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:15:17.789 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:15:17.789 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:15:17.789 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:17.789 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:15:17.789 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 3, 2021-02-10T11:15:17.789 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:15:17.790 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:15:17.790 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:15:17.790 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:15:17.790 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:15:17.790 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:17.790 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:15:17.790 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 4, 2021-02-10T11:15:17.790 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:15:17.790 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:15:17.790 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:15:17.790 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:15:17.790 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:15:17.790 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:17.791 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:15:17.791 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 5, 2021-02-10T11:15:17.791 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:15:17.791 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:15:17.791 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:15:17.791 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:15:17.791 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:15:17.791 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:17.791 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:15:17.791 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 6, 2021-02-10T11:15:17.791 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:15:17.791 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:15:17.791 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:15:17.792 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:15:17.792 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:15:17.792 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:17.792 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:15:17.792 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 7, 2021-02-10T11:15:17.792 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:15:17.792 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:15:17.792 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:15:17.792 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:15:17.792 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:15:17.792 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:15:17.792 INFO:teuthology.orchestra.run.gibba012.stdout: ], 2021-02-10T11:15:17.792 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_upmap": [], 2021-02-10T11:15:17.792 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_upmap_items": [], 2021-02-10T11:15:17.793 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_temp": [], 2021-02-10T11:15:17.793 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_temp": [], 2021-02-10T11:15:17.793 INFO:teuthology.orchestra.run.gibba012.stdout: "blacklist": {}, 2021-02-10T11:15:17.793 INFO:teuthology.orchestra.run.gibba012.stdout: "erasure_code_profiles": { 2021-02-10T11:15:17.793 INFO:teuthology.orchestra.run.gibba012.stdout: "default": { 2021-02-10T11:15:17.793 INFO:teuthology.orchestra.run.gibba012.stdout: "crush-failure-domain": "osd", 2021-02-10T11:15:17.793 INFO:teuthology.orchestra.run.gibba012.stdout: "k": "2", 2021-02-10T11:15:17.793 INFO:teuthology.orchestra.run.gibba012.stdout: "m": "1", 2021-02-10T11:15:17.793 INFO:teuthology.orchestra.run.gibba012.stdout: "plugin": "jerasure", 2021-02-10T11:15:17.793 INFO:teuthology.orchestra.run.gibba012.stdout: "ruleset-failure-domain": "osd", 2021-02-10T11:15:17.793 INFO:teuthology.orchestra.run.gibba012.stdout: "technique": "reed_sol_van" 2021-02-10T11:15:17.793 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:15:17.793 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:17.794 INFO:teuthology.orchestra.run.gibba012.stdout: "removed_snaps_queue": [], 2021-02-10T11:15:17.794 INFO:teuthology.orchestra.run.gibba012.stdout: "new_removed_snaps": [], 2021-02-10T11:15:17.794 INFO:teuthology.orchestra.run.gibba012.stdout: "new_purged_snaps": [], 2021-02-10T11:15:17.794 INFO:teuthology.orchestra.run.gibba012.stdout: "crush_node_flags": {}, 2021-02-10T11:15:17.794 INFO:teuthology.orchestra.run.gibba012.stdout: "device_class_flags": {} 2021-02-10T11:15:17.794 INFO:teuthology.orchestra.run.gibba012.stdout:} 2021-02-10T11:15:17.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:17.721 7fb5e7b52700 1 -- 172.21.2.112:0/1757868126 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb5c80186c0 msgr2=0x7fb5c801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:17.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:17.721 7fb5e7b52700 1 --2- 172.21.2.112:0/1757868126 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb5c80186c0 0x7fb5c801abc0 crc :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:17.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:17.721 7fb5e7b52700 1 -- 172.21.2.112:0/1757868126 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb5e0081c50 msgr2=0x7fb5e00ad5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:17.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:17.721 7fb5e7b52700 1 --2- 172.21.2.112:0/1757868126 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb5e0081c50 0x7fb5e00ad5a0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7fb5d8009980 tx=0x7fb5d800bcd0).stop 2021-02-10T11:15:17.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:17.721 7fb5e7b52700 1 -- 172.21.2.112:0/1757868126 shutdown_connections 2021-02-10T11:15:17.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:17.721 7fb5e7b52700 1 --2- 172.21.2.112:0/1757868126 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb5c80186c0 0x7fb5c801abc0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:17.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:17.721 7fb5e7b52700 1 --2- 172.21.2.112:0/1757868126 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb5e0081c50 0x7fb5e00ad5a0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:17.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:17.721 7fb5e7b52700 1 --2- 172.21.2.112:0/1757868126 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb5e0081850 0x7fb5e00acf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:17.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:17.721 7fb5e7b52700 1 --2- 172.21.2.112:0/1757868126 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb5e008d400 0x7fb5e0081230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:17.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:17.721 7fb5e7b52700 1 -- 172.21.2.112:0/1757868126 >> 172.21.2.112:0/1757868126 conn(0x7fb5e01ac620 msgr2=0x7fb5e00896e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:17.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:17.721 7fb5e7b52700 1 -- 172.21.2.112:0/1757868126 shutdown_connections 2021-02-10T11:15:17.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:17.721 7fb5e7b52700 1 -- 172.21.2.112:0/1757868126 wait complete. 2021-02-10T11:15:17.797 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:15:18.223 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.208 7f1fddf5a700 1 Processor -- start 2021-02-10T11:15:18.223 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.208 7f1fddf5a700 1 -- start start 2021-02-10T11:15:18.223 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.208 7f1fddf5a700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f1fd80d17f0 0x7f1fd81b31e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:18.223 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.208 7f1fddf5a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1fd80cff00 0x7f1fd80ef620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:18.223 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.208 7f1fddf5a700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fd80f0b20 con 0x7f1fd80d1bf0 2021-02-10T11:15:18.223 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.208 7f1fddf5a700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fd81b3ff0 con 0x7f1fd80d17f0 2021-02-10T11:15:18.224 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.208 7f1fddf5a700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fd80f20b0 con 0x7f1fd80cff00 2021-02-10T11:15:18.224 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.209 7f1fd6ffd700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f1fd80d1bf0 0x7f1fd81b3800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:42628/0 (socket says 172.21.2.112:42628) 2021-02-10T11:15:18.224 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.209 7f1fd6ffd700 1 -- 172.21.2.112:0/3157959525 learned_addr learned my addr 172.21.2.112:0/3157959525 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:15:18.224 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.209 7f1fd77fe700 1 --2- 172.21.2.112:0/3157959525 >> v2:172.21.2.120:3300/0 conn(0x7f1fd80d17f0 0x7f1fd81b31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:18.224 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.209 7f1fd7fff700 1 --2- 172.21.2.112:0/3157959525 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1fd80cff00 0x7f1fd80ef620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:18.224 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.209 7f1fd67fc700 1 -- 172.21.2.112:0/3157959525 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 502766437 0 0) 0x7f1fc8001100 con 0x7f1fd80d1bf0 2021-02-10T11:15:18.224 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.209 7f1fd67fc700 1 -- 172.21.2.112:0/3157959525 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fc4002fe0 con 0x7f1fd80d1bf0 2021-02-10T11:15:18.224 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.209 7f1fd7fff700 1 -- 172.21.2.112:0/3157959525 >> v2:172.21.2.120:3300/0 conn(0x7f1fd80d17f0 msgr2=0x7f1fd81b31e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:15:18.224 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.209 7f1fd67fc700 1 -- 172.21.2.112:0/3157959525 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 360068680 0 0) 0x7f1fc80010e0 con 0x7f1fd80d1bf0 2021-02-10T11:15:18.224 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.209 7f1fd7fff700 1 --2- 172.21.2.112:0/3157959525 >> v2:172.21.2.120:3300/0 conn(0x7f1fd80d17f0 0x7f1fd81b31e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:18.224 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.209 7f1fd7fff700 1 -- 172.21.2.112:0/3157959525 >> v1:172.21.2.112:6789/0 conn(0x7f1fd80d1bf0 legacy=0x7f1fd81b3800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:18.224 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.209 7f1fd7fff700 1 -- 172.21.2.112:0/3157959525 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fd80f22d0 con 0x7f1fd80cff00 2021-02-10T11:15:18.224 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.209 7f1fd77fe700 1 --2- 172.21.2.112:0/3157959525 >> v2:172.21.2.120:3300/0 conn(0x7f1fd80d17f0 0x7f1fd81b31e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:15:18.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.210 7f1fd7fff700 1 --2- 172.21.2.112:0/3157959525 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1fd80cff00 0x7f1fd80ef620 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f1fd0006e50 tx=0x7f1fd0009e70).ready entity=mon.2 client_cookie=98ab193eaf62f15b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:18.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.210 7f1fd67fc700 1 -- 172.21.2.112:0/3157959525 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1fd000b510 con 0x7f1fd80cff00 2021-02-10T11:15:18.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.210 7f1fd67fc700 1 -- 172.21.2.112:0/3157959525 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fd0006790 con 0x7f1fd80cff00 2021-02-10T11:15:18.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.210 7f1fd67fc700 1 -- 172.21.2.112:0/3157959525 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1fd000b940 con 0x7f1fd80cff00 2021-02-10T11:15:18.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.210 7f1fddf5a700 1 -- 172.21.2.112:0/3157959525 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1fd80cff00 msgr2=0x7f1fd80ef620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:18.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.210 7f1fddf5a700 1 --2- 172.21.2.112:0/3157959525 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1fd80cff00 0x7f1fd80ef620 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f1fd0006e50 tx=0x7f1fd0009e70).stop 2021-02-10T11:15:18.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.210 7f1fddf5a700 1 -- 172.21.2.112:0/3157959525 shutdown_connections 2021-02-10T11:15:18.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.210 7f1fddf5a700 1 --2- 172.21.2.112:0/3157959525 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1fd80cff00 0x7f1fd80ef620 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:18.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.210 7f1fddf5a700 1 --2- 172.21.2.112:0/3157959525 >> v2:172.21.2.120:3300/0 conn(0x7f1fd80d17f0 0x7f1fd81b31e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:18.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.210 7f1fddf5a700 1 -- 172.21.2.112:0/3157959525 >> 172.21.2.112:0/3157959525 conn(0x7f1fd80fde40 msgr2=0x7f1fd80ce390 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:18.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.211 7f1fddf5a700 1 -- 172.21.2.112:0/3157959525 shutdown_connections 2021-02-10T11:15:18.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.211 7f1fddf5a700 1 -- 172.21.2.112:0/3157959525 wait complete. 2021-02-10T11:15:18.227 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.225 7f1fddf5a700 1 Processor -- start 2021-02-10T11:15:18.227 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.226 7f1fddf5a700 1 -- start start 2021-02-10T11:15:18.227 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.226 7f1fddf5a700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1fd81a48f0 0x7f1fd80b85b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:18.227 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.226 7f1fddf5a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1fd80b9320 0x7f1fd809bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:18.228 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.226 7f1fddf5a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1fd8107bf0 0x7f1fd8107ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:18.228 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.226 7f1fddf5a700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fd80b8da0 con 0x7f1fd80b9320 2021-02-10T11:15:18.228 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.226 7f1fddf5a700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fd80c26a0 con 0x7f1fd81a48f0 2021-02-10T11:15:18.228 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.226 7f1fddf5a700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fd80c2880 con 0x7f1fd8107bf0 2021-02-10T11:15:18.228 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.226 7f1fd6ffd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1fd80b9320 0x7f1fd809bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:18.228 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.226 7f1fd6ffd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1fd80b9320 0x7f1fd809bbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:43174/0 (socket says 172.21.2.112:43174) 2021-02-10T11:15:18.228 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.226 7f1fd6ffd700 1 -- 172.21.2.112:0/2361551530 learned_addr learned my addr 172.21.2.112:0/2361551530 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:15:18.228 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.227 7f1fd6ffd700 1 -- 172.21.2.112:0/2361551530 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1fd8107bf0 msgr2=0x7f1fd8107ff0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:15:18.228 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.227 7f1fd6ffd700 1 --2- 172.21.2.112:0/2361551530 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1fd8107bf0 0x7f1fd8107ff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:18.228 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.227 7f1fd6ffd700 1 -- 172.21.2.112:0/2361551530 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1fd81a48f0 msgr2=0x7f1fd80b85b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:15:18.228 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.227 7f1fd6ffd700 1 --2- 172.21.2.112:0/2361551530 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1fd81a48f0 0x7f1fd80b85b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:18.228 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.227 7f1fd6ffd700 1 -- 172.21.2.112:0/2361551530 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fd80a6620 con 0x7f1fd80b9320 2021-02-10T11:15:18.228 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.227 7f1fd6ffd700 1 --2- 172.21.2.112:0/2361551530 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1fd80b9320 0x7f1fd809bbb0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7f1fc800a5d0 tx=0x7f1fc800a970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:18.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.227 7f1fcffff700 1 -- 172.21.2.112:0/2361551530 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1fc800ad60 con 0x7f1fd80b9320 2021-02-10T11:15:18.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.227 7f1fcffff700 1 -- 172.21.2.112:0/2361551530 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fc80043a0 con 0x7f1fd80b9320 2021-02-10T11:15:18.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.227 7f1fddf5a700 1 -- 172.21.2.112:0/2361551530 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fd80a6620 con 0x7f1fd80b9320 2021-02-10T11:15:18.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.227 7f1fcffff700 1 -- 172.21.2.112:0/2361551530 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1fc80048c0 con 0x7f1fd80b9320 2021-02-10T11:15:18.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.227 7f1fddf5a700 1 -- 172.21.2.112:0/2361551530 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fd80a6620 con 0x7f1fd80b9320 2021-02-10T11:15:18.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.228 7f1fcffff700 1 -- 172.21.2.112:0/2361551530 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1fc8001a50 con 0x7f1fd80b9320 2021-02-10T11:15:18.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.228 7f1fcffff700 1 --2- 172.21.2.112:0/2361551530 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1fbc0186c0 0x7f1fbc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:18.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.228 7f1fcffff700 1 -- 172.21.2.112:0/2361551530 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f1fc80327d0 con 0x7f1fd80b9320 2021-02-10T11:15:18.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.228 7f1fd77fe700 1 --2- 172.21.2.112:0/2361551530 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1fbc0186c0 0x7f1fbc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:18.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.229 7f1fd77fe700 1 --2- 172.21.2.112:0/2361551530 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1fbc0186c0 0x7f1fbc01abc0 crc :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:18.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.258 7f1fddf5a700 1 -- 172.21.2.112:0/2361551530 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fd81036a0 con 0x7f1fd80b9320 2021-02-10T11:15:18.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.261 7f1fcffff700 1 -- 172.21.2.112:0/2361551530 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1fc801f510 con 0x7f1fd80b9320 2021-02-10T11:15:18.389 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.391 7f1fddf5a700 1 -- 172.21.2.112:0/2361551530 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1fd80f0b20 con 0x7f1fd80b9320 2021-02-10T11:15:18.390 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.391 7f1fcffff700 1 -- 172.21.2.112:0/2361551530 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+2097 (secure 0 0 0) 0x7f1fc8001a50 con 0x7f1fd80b9320 2021-02-10T11:15:18.390 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:15:18.390 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4279,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.120:6835/2695560847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":2695560847},{"type":"v1","addr":"172.21.2.120:6835","nonce":2695560847}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":4}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:14:03.627691","modified":"2021-02-10 11:14:13.011189","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4272},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4272":{"gid":4272,"name":"a","rank":0,"incarnation":5,"state":"up:active","state_seq":3,"addr":"172.21.2.112:6833/2584920075","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":2584920075},{"type":"v1","addr":"172.21.2.112:6833","nonce":2584920075}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2021-02-10T11:15:18.390 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 6 2021-02-10T11:15:18.391 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.393 7f1fddf5a700 1 -- 172.21.2.112:0/2361551530 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1fbc0186c0 msgr2=0x7f1fbc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:18.391 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.393 7f1fddf5a700 1 --2- 172.21.2.112:0/2361551530 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1fbc0186c0 0x7f1fbc01abc0 crc :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:18.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.393 7f1fddf5a700 1 -- 172.21.2.112:0/2361551530 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1fd80b9320 msgr2=0x7f1fd809bbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:18.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.393 7f1fddf5a700 1 --2- 172.21.2.112:0/2361551530 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1fd80b9320 0x7f1fd809bbb0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7f1fc800a5d0 tx=0x7f1fc800a970).stop 2021-02-10T11:15:18.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.393 7f1fddf5a700 1 -- 172.21.2.112:0/2361551530 shutdown_connections 2021-02-10T11:15:18.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.393 7f1fddf5a700 1 --2- 172.21.2.112:0/2361551530 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1fbc0186c0 0x7f1fbc01abc0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:18.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.393 7f1fddf5a700 1 --2- 172.21.2.112:0/2361551530 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1fd8107bf0 0x7f1fd8107ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:18.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.393 7f1fddf5a700 1 --2- 172.21.2.112:0/2361551530 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1fd80b9320 0x7f1fd809bbb0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:18.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.393 7f1fddf5a700 1 --2- 172.21.2.112:0/2361551530 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1fd81a48f0 0x7f1fd80b85b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:18.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.393 7f1fddf5a700 1 -- 172.21.2.112:0/2361551530 >> 172.21.2.112:0/2361551530 conn(0x7f1fd809ca80 msgr2=0x7f1fd80c3c60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:18.393 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.394 7f1fddf5a700 1 -- 172.21.2.112:0/2361551530 shutdown_connections 2021-02-10T11:15:18.393 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.394 7f1fddf5a700 1 -- 172.21.2.112:0/2361551530 wait complete. 2021-02-10T11:15:18.400 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs fail cephfs 2021-02-10T11:15:18.826 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.811 7fa1ef2f2700 1 Processor -- start 2021-02-10T11:15:18.826 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.811 7fa1ef2f2700 1 -- start start 2021-02-10T11:15:18.826 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.811 7fa1ef2f2700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fa1e80e5470 0x7fa1e807e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:18.826 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.811 7fa1ef2f2700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fa1e807f300 0x7fa1e807f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:18.826 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.811 7fa1ef2f2700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1e807ed80 con 0x7fa1e80e5070 2021-02-10T11:15:18.827 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.811 7fa1ef2f2700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1e81b4010 con 0x7fa1e807f300 2021-02-10T11:15:18.827 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.811 7fa1ef2f2700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1e80d1790 con 0x7fa1e80e5470 2021-02-10T11:15:18.827 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.812 7fa1ed10e700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fa1e80e5070 0x7fa1e81b3820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:42642/0 (socket says 172.21.2.112:42642) 2021-02-10T11:15:18.827 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.812 7fa1ed10e700 1 -- 172.21.2.112:0/3267766822 learned_addr learned my addr 172.21.2.112:0/3267766822 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:15:18.827 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.812 7fa1ed90f700 1 --2- 172.21.2.112:0/3267766822 >> v2:172.21.2.112:3300/0 conn(0x7fa1e807f300 0x7fa1e807f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:18.827 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.812 7fa1ed90f700 1 -- 172.21.2.112:0/3267766822 >> v2:172.21.2.120:3300/0 conn(0x7fa1e80e5470 msgr2=0x7fa1e807e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:15:18.827 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.812 7fa1ed90f700 1 --2- 172.21.2.112:0/3267766822 >> v2:172.21.2.120:3300/0 conn(0x7fa1e80e5470 0x7fa1e807e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:18.827 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.812 7fa1ed90f700 1 -- 172.21.2.112:0/3267766822 >> v1:172.21.2.112:6789/0 conn(0x7fa1e80e5070 legacy=0x7fa1e81b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:18.827 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.812 7fa1ed90f700 1 -- 172.21.2.112:0/3267766822 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1e80d19b0 con 0x7fa1e807f300 2021-02-10T11:15:18.827 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.812 7fa1ed90f700 1 --2- 172.21.2.112:0/3267766822 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa1e807f300 0x7fa1e807f700 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7fa1e0006e30 tx=0x7fa1e0009e60).ready entity=mon.0 client_cookie=ff2984d900ff4998 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:18.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.812 7fa1dbfff700 1 -- 172.21.2.112:0/3267766822 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa1e0006a50 con 0x7fa1e807f300 2021-02-10T11:15:18.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.812 7fa1dbfff700 1 -- 172.21.2.112:0/3267766822 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1e000b300 con 0x7fa1e807f300 2021-02-10T11:15:18.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.812 7fa1dbfff700 1 -- 172.21.2.112:0/3267766822 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa1e000b820 con 0x7fa1e807f300 2021-02-10T11:15:18.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.812 7fa1ef2f2700 1 -- 172.21.2.112:0/3267766822 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa1e807f300 msgr2=0x7fa1e807f700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:18.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.812 7fa1ef2f2700 1 --2- 172.21.2.112:0/3267766822 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa1e807f300 0x7fa1e807f700 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7fa1e0006e30 tx=0x7fa1e0009e60).stop 2021-02-10T11:15:18.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.813 7fa1ef2f2700 1 -- 172.21.2.112:0/3267766822 shutdown_connections 2021-02-10T11:15:18.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.813 7fa1ef2f2700 1 --2- 172.21.2.112:0/3267766822 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa1e807f300 0x7fa1e807f700 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:18.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.813 7fa1ef2f2700 1 --2- 172.21.2.112:0/3267766822 >> v2:172.21.2.120:3300/0 conn(0x7fa1e80e5470 0x7fa1e807e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:18.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.813 7fa1ef2f2700 1 -- 172.21.2.112:0/3267766822 >> 172.21.2.112:0/3267766822 conn(0x7fa1e80ce690 msgr2=0x7fa1e80e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:18.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.813 7fa1ef2f2700 1 -- 172.21.2.112:0/3267766822 shutdown_connections 2021-02-10T11:15:18.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.813 7fa1ef2f2700 1 -- 172.21.2.112:0/3267766822 wait complete. 2021-02-10T11:15:18.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.828 7fa1ef2f2700 1 Processor -- start 2021-02-10T11:15:18.829 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.828 7fa1ef2f2700 1 -- start start 2021-02-10T11:15:18.830 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.828 7fa1ef2f2700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa1e80924d0 0x7fa1e819a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:18.830 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.828 7fa1ef2f2700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa1e819ae00 0x7fa1e80ba630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:18.830 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.828 7fa1ef2f2700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa1e80bac50 0x7fa1e80bb050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:18.830 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.828 7fa1ef2f2700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1e8092a60 con 0x7fa1e80bac50 2021-02-10T11:15:18.830 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.828 7fa1ef2f2700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1e80bb670 con 0x7fa1e819ae00 2021-02-10T11:15:18.831 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.828 7fa1ef2f2700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1e80932f0 con 0x7fa1e80924d0 2021-02-10T11:15:18.831 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.829 7fa1ed90f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa1e80bac50 0x7fa1e80bb050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:18.831 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.829 7fa1ed90f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa1e80bac50 0x7fa1e80bb050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:43190/0 (socket says 172.21.2.112:43190) 2021-02-10T11:15:18.831 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.829 7fa1ed90f700 1 -- 172.21.2.112:0/180155099 learned_addr learned my addr 172.21.2.112:0/180155099 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:15:18.831 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.829 7fa1ec90d700 1 --2- 172.21.2.112:0/180155099 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa1e819ae00 0x7fa1e80ba630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:18.831 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.829 7fa1ed10e700 1 --2- 172.21.2.112:0/180155099 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa1e80924d0 0x7fa1e819a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:18.831 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.829 7fa1ed90f700 1 -- 172.21.2.112:0/180155099 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa1e80924d0 msgr2=0x7fa1e819a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:18.831 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.829 7fa1ed90f700 1 --2- 172.21.2.112:0/180155099 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa1e80924d0 0x7fa1e819a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:18.831 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.829 7fa1ed90f700 1 -- 172.21.2.112:0/180155099 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa1e819ae00 msgr2=0x7fa1e80ba630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:18.832 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.829 7fa1ed90f700 1 --2- 172.21.2.112:0/180155099 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa1e819ae00 0x7fa1e80ba630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:18.832 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.829 7fa1ed90f700 1 -- 172.21.2.112:0/180155099 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1e8093510 con 0x7fa1e80bac50 2021-02-10T11:15:18.832 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.829 7fa1ed10e700 1 --2- 172.21.2.112:0/180155099 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa1e80924d0 0x7fa1e819a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:15:18.832 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.829 7fa1ec90d700 1 --2- 172.21.2.112:0/180155099 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa1e819ae00 0x7fa1e80ba630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:15:18.832 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.829 7fa1ed90f700 1 --2- 172.21.2.112:0/180155099 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa1e80bac50 0x7fa1e80bb050 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7fa1e000b970 tx=0x7fa1e000bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:18.832 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.829 7fa1d9ffb700 1 -- 172.21.2.112:0/180155099 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa1e0009380 con 0x7fa1e80bac50 2021-02-10T11:15:18.832 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.829 7fa1d9ffb700 1 -- 172.21.2.112:0/180155099 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1e0005820 con 0x7fa1e80bac50 2021-02-10T11:15:18.832 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.829 7fa1d9ffb700 1 -- 172.21.2.112:0/180155099 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa1e0005c80 con 0x7fa1e80bac50 2021-02-10T11:15:18.832 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.829 7fa1ef2f2700 1 -- 172.21.2.112:0/180155099 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1e8093510 con 0x7fa1e80bac50 2021-02-10T11:15:18.832 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.829 7fa1ef2f2700 1 -- 172.21.2.112:0/180155099 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1e8093aa0 con 0x7fa1e80bac50 2021-02-10T11:15:18.832 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.830 7fa1d9ffb700 1 -- 172.21.2.112:0/180155099 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa1e0005a00 con 0x7fa1e80bac50 2021-02-10T11:15:18.832 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.830 7fa1d9ffb700 1 --2- 172.21.2.112:0/180155099 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa1c80186c0 0x7fa1c801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:18.832 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.830 7fa1d9ffb700 1 -- 172.21.2.112:0/180155099 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fa1e0032990 con 0x7fa1e80bac50 2021-02-10T11:15:18.833 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.831 7fa1ed10e700 1 --2- 172.21.2.112:0/180155099 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa1c80186c0 0x7fa1c801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:18.833 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.831 7fa1ed10e700 1 --2- 172.21.2.112:0/180155099 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa1c80186c0 0x7fa1c801abc0 crc :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:18.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.861 7fa1ef2f2700 1 -- 172.21.2.112:0/180155099 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1e8083e50 con 0x7fa1e80bac50 2021-02-10T11:15:18.862 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.864 7fa1d9ffb700 1 -- 172.21.2.112:0/180155099 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa1e0009380 con 0x7fa1e80bac50 2021-02-10T11:15:18.988 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:18.990 7fa1ef2f2700 1 -- 172.21.2.112:0/180155099 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fa1e807ed80 con 0x7fa1e80bac50 2021-02-10T11:15:19.776 INFO:tasks.ceph.mon.a.gibba012.stderr:2021-02-10 11:15:19.778 7fc3fe6e2700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-02-10T11:15:19.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:19.783 7fa1d9ffb700 1 -- 172.21.2.112:0/180155099 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v7) v1 ==== 161+0+0 (secure 0 0 0) 0x7fa1e0005a00 con 0x7fa1e80bac50 2021-02-10T11:15:19.808 INFO:teuthology.orchestra.run.gibba012.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2021-02-10T11:15:19.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:19.813 7fa1ef2f2700 1 -- 172.21.2.112:0/180155099 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa1c80186c0 msgr2=0x7fa1c801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:19.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:19.813 7fa1ef2f2700 1 --2- 172.21.2.112:0/180155099 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa1c80186c0 0x7fa1c801abc0 crc :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:19.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:19.813 7fa1ef2f2700 1 -- 172.21.2.112:0/180155099 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa1e80bac50 msgr2=0x7fa1e80bb050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:19.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:19.813 7fa1ef2f2700 1 --2- 172.21.2.112:0/180155099 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa1e80bac50 0x7fa1e80bb050 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7fa1e000b970 tx=0x7fa1e000bd10).stop 2021-02-10T11:15:19.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:19.813 7fa1ef2f2700 1 -- 172.21.2.112:0/180155099 shutdown_connections 2021-02-10T11:15:19.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:19.813 7fa1ef2f2700 1 --2- 172.21.2.112:0/180155099 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa1c80186c0 0x7fa1c801abc0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:19.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:19.813 7fa1ef2f2700 1 --2- 172.21.2.112:0/180155099 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa1e819ae00 0x7fa1e80ba630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:19.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:19.813 7fa1ef2f2700 1 --2- 172.21.2.112:0/180155099 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa1e80bac50 0x7fa1e80bb050 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:19.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:19.813 7fa1ef2f2700 1 --2- 172.21.2.112:0/180155099 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa1e80924d0 0x7fa1e819a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:19.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:19.813 7fa1ef2f2700 1 -- 172.21.2.112:0/180155099 >> 172.21.2.112:0/180155099 conn(0x7fa1e80bdea0 msgr2=0x7fa1e8096b50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:19.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:19.814 7fa1ef2f2700 1 -- 172.21.2.112:0/180155099 shutdown_connections 2021-02-10T11:15:19.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:19.814 7fa1ef2f2700 1 -- 172.21.2.112:0/180155099 wait complete. 2021-02-10T11:15:19.820 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:15:19.827 INFO:tasks.ceph.mds.a.gibba012.stdout:starting mds.a at 2021-02-10T11:15:20.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.232 7f09fe881700 1 Processor -- start 2021-02-10T11:15:20.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.232 7f09fe881700 1 -- start start 2021-02-10T11:15:20.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.233 7f09fe881700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f09f80e5470 0x7f09f807e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:20.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.233 7f09fe881700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09f807ed80 con 0x7f09f807f300 2021-02-10T11:15:20.248 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.233 7f09fe881700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09f81b4010 con 0x7f09f80e5070 2021-02-10T11:15:20.248 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.233 7f09fe881700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09f80d1790 con 0x7f09f80e5470 2021-02-10T11:15:20.248 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.233 7f09fce9e700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f09f807f300 0x7f09f807f700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:42670/0 (socket says 172.21.2.112:42670) 2021-02-10T11:15:20.248 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.233 7f09fce9e700 1 -- 172.21.2.112:0/763414157 learned_addr learned my addr 172.21.2.112:0/763414157 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:15:20.248 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.233 7f09f77fe700 1 --2- 172.21.2.112:0/763414157 >> v2:172.21.2.112:3300/0 conn(0x7f09f80e5470 0x7f09f807e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:20.248 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.233 7f09f6ffd700 1 -- 172.21.2.112:0/763414157 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3049295617 0 0) 0x7f09f0001500 con 0x7f09f807f300 2021-02-10T11:15:20.248 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.233 7f09f6ffd700 1 -- 172.21.2.112:0/763414157 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09e0002fe0 con 0x7f09f807f300 2021-02-10T11:15:20.248 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.233 7f09f77fe700 1 -- 172.21.2.112:0/763414157 >> v1:172.21.2.120:6789/0 conn(0x7f09f80e5070 legacy=0x7f09f81b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:15:20.248 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.233 7f09f77fe700 1 -- 172.21.2.112:0/763414157 >> v1:172.21.2.112:6789/0 conn(0x7f09f807f300 legacy=0x7f09f807f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:20.248 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.233 7f09f77fe700 1 -- 172.21.2.112:0/763414157 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09f80d19b0 con 0x7f09f80e5470 2021-02-10T11:15:20.248 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.234 7f09f77fe700 1 --2- 172.21.2.112:0/763414157 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f09f80e5470 0x7f09f807e6a0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f09ec006650 tx=0x7f09ec009a80).ready entity=mon.0 client_cookie=49070855354d3913 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:20.248 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.234 7f09f6ffd700 1 -- 172.21.2.112:0/763414157 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f09ec0009b0 con 0x7f09f80e5470 2021-02-10T11:15:20.248 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.234 7f09f6ffd700 1 -- 172.21.2.112:0/763414157 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09ec00c0c0 con 0x7f09f80e5470 2021-02-10T11:15:20.249 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.234 7f09f6ffd700 1 -- 172.21.2.112:0/763414157 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f09ec00c5e0 con 0x7f09f80e5470 2021-02-10T11:15:20.249 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.234 7f09fe881700 1 -- 172.21.2.112:0/763414157 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f09f80e5470 msgr2=0x7f09f807e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:20.249 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.234 7f09fe881700 1 --2- 172.21.2.112:0/763414157 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f09f80e5470 0x7f09f807e6a0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f09ec006650 tx=0x7f09ec009a80).stop 2021-02-10T11:15:20.249 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.234 7f09fe881700 1 -- 172.21.2.112:0/763414157 shutdown_connections 2021-02-10T11:15:20.249 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.234 7f09fe881700 1 --2- 172.21.2.112:0/763414157 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f09f80e5470 0x7f09f807e6a0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:20.249 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.234 7f09fe881700 1 -- 172.21.2.112:0/763414157 >> 172.21.2.112:0/763414157 conn(0x7f09f80ce690 msgr2=0x7f09f80e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:20.249 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.234 7f09fe881700 1 -- 172.21.2.112:0/763414157 shutdown_connections 2021-02-10T11:15:20.249 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.234 7f09fe881700 1 -- 172.21.2.112:0/763414157 wait complete. 2021-02-10T11:15:20.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.249 7f09fe881700 1 Processor -- start 2021-02-10T11:15:20.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.249 7f09fe881700 1 -- start start 2021-02-10T11:15:20.252 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.250 7f09fe881700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f09f81aa290 0x7f09f81aa690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:20.252 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.250 7f09fe881700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f09f80b7270 0x7f09f80b7670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:20.252 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.250 7f09fe881700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f09f80b7c90 0x7f09f8084e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:20.252 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.250 7f09fe881700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09f80b8090 con 0x7f09f80b7270 2021-02-10T11:15:20.252 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.250 7f09fe881700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09f81a9df0 con 0x7f09f81aa290 2021-02-10T11:15:20.252 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.250 7f09fe881700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09f809cc30 con 0x7f09f80b7c90 2021-02-10T11:15:20.252 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.250 7f09f77fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f09f80b7270 0x7f09f80b7670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:20.252 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.250 7f09f77fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f09f80b7270 0x7f09f80b7670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:43216/0 (socket says 172.21.2.112:43216) 2021-02-10T11:15:20.252 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.250 7f09f77fe700 1 -- 172.21.2.112:0/3247384957 learned_addr learned my addr 172.21.2.112:0/3247384957 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:15:20.252 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.250 7f09f77fe700 1 -- 172.21.2.112:0/3247384957 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f09f80b7c90 msgr2=0x7f09f8084e10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:15:20.252 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.250 7f09f77fe700 1 --2- 172.21.2.112:0/3247384957 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f09f80b7c90 0x7f09f8084e10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:20.252 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.250 7f09f77fe700 1 -- 172.21.2.112:0/3247384957 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f09f81aa290 msgr2=0x7f09f81aa690 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:15:20.252 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.250 7f09f77fe700 1 --2- 172.21.2.112:0/3247384957 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f09f81aa290 0x7f09f81aa690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:20.253 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.250 7f09f77fe700 1 -- 172.21.2.112:0/3247384957 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09f8085430 con 0x7f09f80b7270 2021-02-10T11:15:20.253 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.250 7f09f77fe700 1 --2- 172.21.2.112:0/3247384957 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f09f80b7270 0x7f09f80b7670 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7f09ec009650 tx=0x7f09ec00b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:20.253 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.250 7f09f4ff9700 1 -- 172.21.2.112:0/3247384957 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f09ec0093a0 con 0x7f09f80b7270 2021-02-10T11:15:20.253 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.251 7f09f4ff9700 1 -- 172.21.2.112:0/3247384957 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09ec00bde0 con 0x7f09f80b7270 2021-02-10T11:15:20.253 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.251 7f09f4ff9700 1 -- 172.21.2.112:0/3247384957 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f09ec0052e0 con 0x7f09f80b7270 2021-02-10T11:15:20.253 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.251 7f09fe881700 1 -- 172.21.2.112:0/3247384957 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09f8085430 con 0x7f09f80b7270 2021-02-10T11:15:20.253 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.251 7f09fe881700 1 -- 172.21.2.112:0/3247384957 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09f8085430 con 0x7f09f80b7270 2021-02-10T11:15:20.253 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.251 7f09f4ff9700 1 -- 172.21.2.112:0/3247384957 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f09ec009300 con 0x7f09f80b7270 2021-02-10T11:15:20.253 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.251 7f09f4ff9700 1 --2- 172.21.2.112:0/3247384957 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f09d80186d0 0x7f09d801abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:20.253 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.251 7f09f4ff9700 1 -- 172.21.2.112:0/3247384957 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5659+0+0 (secure 0 0 0) 0x7f09ec032800 con 0x7f09f80b7270 2021-02-10T11:15:20.253 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.251 7f09f7fff700 1 --2- 172.21.2.112:0/3247384957 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f09d80186d0 0x7f09d801abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:20.253 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.252 7f09f7fff700 1 --2- 172.21.2.112:0/3247384957 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f09d80186d0 0x7f09d801abd0 crc :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:20.280 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.282 7f09fe881700 1 -- 172.21.2.112:0/3247384957 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09f80b8090 con 0x7f09f80b7270 2021-02-10T11:15:20.282 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.285 7f09f4ff9700 1 -- 172.21.2.112:0/3247384957 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f09ec0009c0 con 0x7f09f80b7270 2021-02-10T11:15:20.413 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.414 7f09fe881700 1 -- 172.21.2.112:0/3247384957 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f09f80b8090 con 0x7f09f80b7270 2021-02-10T11:15:20.413 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.415 7f09f4ff9700 1 -- 172.21.2.112:0/3247384957 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+1753 (secure 0 0 0) 0x7f09ec00bd50 con 0x7f09f80b7270 2021-02-10T11:15:20.414 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:15:20.414 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4279,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.120:6835/2695560847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":2695560847},{"type":"v1","addr":"172.21.2.120:6835","nonce":2695560847}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":4}],"filesystems":[{"mdsmap":{"epoch":7,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:14:03.627691","modified":"2021-02-10 11:15:19.778966","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":21,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2021-02-10T11:15:20.414 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 7 2021-02-10T11:15:20.416 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.416 7f09fe881700 1 -- 172.21.2.112:0/3247384957 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f09d80186d0 msgr2=0x7f09d801abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:20.416 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.416 7f09fe881700 1 --2- 172.21.2.112:0/3247384957 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f09d80186d0 0x7f09d801abd0 crc :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:20.416 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.416 7f09fe881700 1 -- 172.21.2.112:0/3247384957 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f09f80b7270 msgr2=0x7f09f80b7670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:20.416 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.416 7f09fe881700 1 --2- 172.21.2.112:0/3247384957 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f09f80b7270 0x7f09f80b7670 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7f09ec009650 tx=0x7f09ec00b960).stop 2021-02-10T11:15:20.416 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.417 7f09fe881700 1 -- 172.21.2.112:0/3247384957 shutdown_connections 2021-02-10T11:15:20.417 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.417 7f09fe881700 1 --2- 172.21.2.112:0/3247384957 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f09d80186d0 0x7f09d801abd0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:20.417 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.417 7f09fe881700 1 --2- 172.21.2.112:0/3247384957 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f09f80b7c90 0x7f09f8084e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:20.417 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.417 7f09fe881700 1 --2- 172.21.2.112:0/3247384957 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f09f80b7270 0x7f09f80b7670 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:20.417 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.417 7f09fe881700 1 --2- 172.21.2.112:0/3247384957 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f09f81aa290 0x7f09f81aa690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:20.417 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.417 7f09fe881700 1 -- 172.21.2.112:0/3247384957 >> 172.21.2.112:0/3247384957 conn(0x7f09f808fe10 msgr2=0x7f09f819c020 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:20.417 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.417 7f09fe881700 1 -- 172.21.2.112:0/3247384957 shutdown_connections 2021-02-10T11:15:20.418 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.417 7f09fe881700 1 -- 172.21.2.112:0/3247384957 wait complete. 2021-02-10T11:15:20.424 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2021-02-10T11:15:20.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.880 7fcf027b4700 1 Processor -- start 2021-02-10T11:15:20.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.880 7fcf027b4700 1 -- start start 2021-02-10T11:15:20.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.880 7fcf027b4700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fcefc0d17f0 0x7fcefc1b31e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:20.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.880 7fcf027b4700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fcefc0d1bf0 0x7fcefc1b3800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:20.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.880 7fcf027b4700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcefc1b3e20 0x7fcefc0ceef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:20.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.880 7fcf027b4700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcefc0e5eb0 con 0x7fcefc0d1bf0 2021-02-10T11:15:20.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.880 7fcf027b4700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcefc0f0b20 con 0x7fcefc0d17f0 2021-02-10T11:15:20.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.880 7fcf027b4700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcefc0cf510 con 0x7fcefc1b3e20 2021-02-10T11:15:20.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.881 7fcefbfff700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fcefc0d17f0 0x7fcefc1b31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:20.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.881 7fcefb7fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fcefc0d1bf0 0x7fcefc1b3800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:20.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.881 7fcefb7fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fcefc0d1bf0 0x7fcefc1b3800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:43220/0 (socket says 172.21.2.112:43220) 2021-02-10T11:15:20.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.881 7fcefb7fe700 1 -- 172.21.2.112:0/1207955105 learned_addr learned my addr 172.21.2.112:0/1207955105 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:15:20.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.881 7fcf00dd1700 1 --2- 172.21.2.112:0/1207955105 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcefc1b3e20 0x7fcefc0ceef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:20.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.881 7fcefb7fe700 1 -- 172.21.2.112:0/1207955105 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcefc1b3e20 msgr2=0x7fcefc0ceef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:15:20.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.881 7fcefb7fe700 1 --2- 172.21.2.112:0/1207955105 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcefc1b3e20 0x7fcefc0ceef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:20.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.881 7fcefb7fe700 1 -- 172.21.2.112:0/1207955105 >> v2:172.21.2.120:3300/0 conn(0x7fcefc0d17f0 msgr2=0x7fcefc1b31e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:15:20.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.881 7fcefb7fe700 1 --2- 172.21.2.112:0/1207955105 >> v2:172.21.2.120:3300/0 conn(0x7fcefc0d17f0 0x7fcefc1b31e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:20.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.881 7fcefb7fe700 1 -- 172.21.2.112:0/1207955105 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcefc0cf6f0 con 0x7fcefc0d1bf0 2021-02-10T11:15:20.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.881 7fcefb7fe700 1 --2- 172.21.2.112:0/1207955105 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcefc0d1bf0 0x7fcefc1b3800 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7fcee8006700 tx=0x7fcee8009b80).ready entity=mon.0 client_cookie=b2cdba3af5c74e0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:20.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.881 7fcefaffd700 1 -- 172.21.2.112:0/1207955105 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcee800c0c0 con 0x7fcefc0d1bf0 2021-02-10T11:15:20.897 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.881 7fcefaffd700 1 -- 172.21.2.112:0/1207955105 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcee800c340 con 0x7fcefc0d1bf0 2021-02-10T11:15:20.897 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.881 7fcefaffd700 1 -- 172.21.2.112:0/1207955105 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcee800c7a0 con 0x7fcefc0d1bf0 2021-02-10T11:15:20.897 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.881 7fcf027b4700 1 -- 172.21.2.112:0/1207955105 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcefc0d1bf0 msgr2=0x7fcefc1b3800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:20.897 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.881 7fcf027b4700 1 --2- 172.21.2.112:0/1207955105 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcefc0d1bf0 0x7fcefc1b3800 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7fcee8006700 tx=0x7fcee8009b80).stop 2021-02-10T11:15:20.897 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.882 7fcf027b4700 1 -- 172.21.2.112:0/1207955105 shutdown_connections 2021-02-10T11:15:20.897 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.882 7fcf027b4700 1 --2- 172.21.2.112:0/1207955105 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcefc1b3e20 0x7fcefc0ceef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:20.897 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.882 7fcf027b4700 1 --2- 172.21.2.112:0/1207955105 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcefc0d1bf0 0x7fcefc1b3800 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:20.897 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.882 7fcf027b4700 1 --2- 172.21.2.112:0/1207955105 >> v2:172.21.2.120:3300/0 conn(0x7fcefc0d17f0 0x7fcefc1b31e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:20.897 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.882 7fcf027b4700 1 -- 172.21.2.112:0/1207955105 >> 172.21.2.112:0/1207955105 conn(0x7fcefc0fde40 msgr2=0x7fcefc0ce390 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:20.897 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.882 7fcf027b4700 1 -- 172.21.2.112:0/1207955105 shutdown_connections 2021-02-10T11:15:20.897 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.882 7fcf027b4700 1 -- 172.21.2.112:0/1207955105 wait complete. 2021-02-10T11:15:20.897 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.897 7fcf027b4700 1 Processor -- start 2021-02-10T11:15:20.897 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.897 7fcf027b4700 1 -- start start 2021-02-10T11:15:20.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.897 7fcf027b4700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcefc19ffe0 0x7fcefc1a03e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:20.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.897 7fcf027b4700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcefc190340 0x7fcefc190740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:20.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.897 7fcf027b4700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcefc190d60 0x7fcefc191160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:20.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.897 7fcf027b4700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcefc191850 con 0x7fcefc190340 2021-02-10T11:15:20.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.897 7fcf027b4700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcefc0c3c30 con 0x7fcefc19ffe0 2021-02-10T11:15:20.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.897 7fcf027b4700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcefc191d50 con 0x7fcefc190d60 2021-02-10T11:15:20.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.897 7fcefb7fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcefc190340 0x7fcefc190740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:20.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.898 7fcf00dd1700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcefc190d60 0x7fcefc191160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:20.900 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.898 7fcefb7fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcefc190340 0x7fcefc190740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:43224/0 (socket says 172.21.2.112:43224) 2021-02-10T11:15:20.900 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.898 7fcefbfff700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcefc19ffe0 0x7fcefc1a03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:20.900 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.898 7fcefb7fe700 1 -- 172.21.2.112:0/1610688924 learned_addr learned my addr 172.21.2.112:0/1610688924 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:15:20.900 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.898 7fcefb7fe700 1 -- 172.21.2.112:0/1610688924 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcefc190d60 msgr2=0x7fcefc191160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:20.900 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.898 7fcefb7fe700 1 --2- 172.21.2.112:0/1610688924 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcefc190d60 0x7fcefc191160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:20.900 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.898 7fcefb7fe700 1 -- 172.21.2.112:0/1610688924 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcefc19ffe0 msgr2=0x7fcefc1a03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:20.900 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.898 7fcefb7fe700 1 --2- 172.21.2.112:0/1610688924 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcefc19ffe0 0x7fcefc1a03e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:20.900 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.898 7fcefb7fe700 1 -- 172.21.2.112:0/1610688924 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcefc191f70 con 0x7fcefc190340 2021-02-10T11:15:20.900 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.898 7fcefb7fe700 1 --2- 172.21.2.112:0/1610688924 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcefc190340 0x7fcefc190740 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7fcee800b580 tx=0x7fcee800b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:20.900 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.898 7fcef8ff9700 1 -- 172.21.2.112:0/1610688924 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcee800bd10 con 0x7fcefc190340 2021-02-10T11:15:20.900 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.898 7fcef8ff9700 1 -- 172.21.2.112:0/1610688924 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcee8006500 con 0x7fcefc190340 2021-02-10T11:15:20.900 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.898 7fcef8ff9700 1 -- 172.21.2.112:0/1610688924 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcee8002220 con 0x7fcefc190340 2021-02-10T11:15:20.900 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.898 7fcf027b4700 1 -- 172.21.2.112:0/1610688924 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcefc191f70 con 0x7fcefc190340 2021-02-10T11:15:20.901 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.898 7fcf027b4700 1 -- 172.21.2.112:0/1610688924 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcefc191f70 con 0x7fcefc190340 2021-02-10T11:15:20.901 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.899 7fcef8ff9700 1 -- 172.21.2.112:0/1610688924 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fcee8000c90 con 0x7fcefc190340 2021-02-10T11:15:20.901 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.899 7fcef8ff9700 1 --2- 172.21.2.112:0/1610688924 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fced8018730 0x7fced801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:20.901 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.899 7fcef8ff9700 1 -- 172.21.2.112:0/1610688924 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5659+0+0 (secure 0 0 0) 0x7fcee80326d0 con 0x7fcefc190340 2021-02-10T11:15:20.901 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.899 7fcefbfff700 1 --2- 172.21.2.112:0/1610688924 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fced8018730 0x7fced801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:20.901 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.899 7fcefbfff700 1 --2- 172.21.2.112:0/1610688924 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fced8018730 0x7fced801abc0 crc :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:20.928 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.930 7fcf027b4700 1 -- 172.21.2.112:0/1610688924 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcefc103690 con 0x7fcefc190340 2021-02-10T11:15:20.930 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:20.932 7fcef8ff9700 1 -- 172.21.2.112:0/1610688924 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fcee8006500 con 0x7fcefc190340 2021-02-10T11:15:21.058 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:21.060 7fcf027b4700 1 -- 172.21.2.112:0/1610688924 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fcefc103690 con 0x7fcefc190340 2021-02-10T11:15:21.787 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:21.788 7fcef8ff9700 1 -- 172.21.2.112:0/1610688924 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v8) v1 ==== 104+0+0 (secure 0 0 0) 0x7fcee8000c90 con 0x7fcefc190340 2021-02-10T11:15:21.830 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:21.832 7fcf027b4700 1 -- 172.21.2.112:0/1610688924 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fced8018730 msgr2=0x7fced801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:21.830 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:21.832 7fcf027b4700 1 --2- 172.21.2.112:0/1610688924 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fced8018730 0x7fced801abc0 crc :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:21.830 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:21.832 7fcf027b4700 1 -- 172.21.2.112:0/1610688924 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcefc190340 msgr2=0x7fcefc190740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:21.830 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:21.832 7fcf027b4700 1 --2- 172.21.2.112:0/1610688924 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcefc190340 0x7fcefc190740 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7fcee800b580 tx=0x7fcee800b920).stop 2021-02-10T11:15:21.831 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:21.832 7fcf027b4700 1 -- 172.21.2.112:0/1610688924 shutdown_connections 2021-02-10T11:15:21.831 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:21.832 7fcf027b4700 1 --2- 172.21.2.112:0/1610688924 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fced8018730 0x7fced801abc0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:21.831 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:21.832 7fcf027b4700 1 --2- 172.21.2.112:0/1610688924 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcefc190d60 0x7fcefc191160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:21.831 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:21.832 7fcf027b4700 1 --2- 172.21.2.112:0/1610688924 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcefc190340 0x7fcefc190740 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:21.831 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:21.832 7fcf027b4700 1 --2- 172.21.2.112:0/1610688924 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcefc19ffe0 0x7fcefc1a03e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:21.831 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:21.832 7fcf027b4700 1 -- 172.21.2.112:0/1610688924 >> 172.21.2.112:0/1610688924 conn(0x7fcefc0a1690 msgr2=0x7fcefc0a94c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:21.831 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:21.832 7fcf027b4700 1 -- 172.21.2.112:0/1610688924 shutdown_connections 2021-02-10T11:15:21.831 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:21.832 7fcf027b4700 1 -- 172.21.2.112:0/1610688924 wait complete. 2021-02-10T11:15:21.839 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2021-02-10T11:15:22.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:22.254 7f1e92259700 1 Processor -- start 2021-02-10T11:15:22.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:22.254 7f1e92259700 1 -- start start 2021-02-10T11:15:22.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:22.255 7f1e92259700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f1e8c0d1870 0x7f1e8c1b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:22.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:22.255 7f1e92259700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e8c0fe1c0 con 0x7f1e8c1b0ae0 2021-02-10T11:15:22.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:22.255 7f1e92259700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e8c109a60 con 0x7f1e8c0cff70 2021-02-10T11:15:22.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:22.255 7f1e92259700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e8c088020 con 0x7f1e8c0d1870 2021-02-10T11:15:22.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:22.255 7f1e8b7fe700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f1e8c1b0ae0 0x7f1e8c108940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:42694/0 (socket says 172.21.2.112:42694) 2021-02-10T11:15:22.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:22.255 7f1e8b7fe700 1 -- 172.21.2.112:0/2007668208 learned_addr learned my addr 172.21.2.112:0/2007668208 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:15:22.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:22.255 7f1e8bfff700 1 --2- 172.21.2.112:0/2007668208 >> v2:172.21.2.120:3300/0 conn(0x7f1e8c0d1870 0x7f1e8c1b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:22.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:22.255 7f1e8affd700 1 -- 172.21.2.112:0/2007668208 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 605686228 0 0) 0x7f1e800010b0 con 0x7f1e8c1b0ae0 2021-02-10T11:15:22.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:22.255 7f1e8affd700 1 -- 172.21.2.112:0/2007668208 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e74002fe0 con 0x7f1e8c1b0ae0 2021-02-10T11:15:22.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:22.256 7f1e8bfff700 1 -- 172.21.2.112:0/2007668208 >> v1:172.21.2.120:6789/0 conn(0x7f1e8c0cff70 legacy=0x7f1e8c109130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:22.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:22.256 7f1e8bfff700 1 -- 172.21.2.112:0/2007668208 >> v1:172.21.2.112:6789/0 conn(0x7f1e8c1b0ae0 legacy=0x7f1e8c108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:22.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:22.256 7f1e8affd700 1 -- 172.21.2.112:0/2007668208 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1961118172 0 0) 0x7f1e80001010 con 0x7f1e8c1b0ae0 2021-02-10T11:15:22.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:22.256 7f1e8bfff700 1 -- 172.21.2.112:0/2007668208 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e8c088210 con 0x7f1e8c0d1870 2021-02-10T11:15:22.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:22.256 7f1e8bfff700 1 --2- 172.21.2.112:0/2007668208 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1e8c0d1870 0x7f1e8c1b04c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f1e7c006650 tx=0x7f1e7c009a80).ready entity=mon.1 client_cookie=6ba7e1b18706c87b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:22.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:22.256 7f1e8affd700 1 -- 172.21.2.112:0/2007668208 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1e7c0009b0 con 0x7f1e8c0d1870 2021-02-10T11:15:22.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:22.256 7f1e8affd700 1 -- 172.21.2.112:0/2007668208 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e7c00c0c0 con 0x7f1e8c0d1870 2021-02-10T11:15:22.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:22.256 7f1e8affd700 1 -- 172.21.2.112:0/2007668208 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1e7c00c5a0 con 0x7f1e8c0d1870 2021-02-10T11:15:22.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:22.256 7f1e92259700 1 -- 172.21.2.112:0/2007668208 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1e8c0d1870 msgr2=0x7f1e8c1b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:22.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:22.256 7f1e92259700 1 --2- 172.21.2.112:0/2007668208 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1e8c0d1870 0x7f1e8c1b04c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f1e7c006650 tx=0x7f1e7c009a80).stop 2021-02-10T11:15:22.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:22.257 7f1e92259700 1 -- 172.21.2.112:0/2007668208 shutdown_connections 2021-02-10T11:15:22.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:22.257 7f1e92259700 1 --2- 172.21.2.112:0/2007668208 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1e8c0d1870 0x7f1e8c1b04c0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:22.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:22.257 7f1e92259700 1 -- 172.21.2.112:0/2007668208 >> 172.21.2.112:0/2007668208 conn(0x7f1e8c0ce420 msgr2=0x7f1e8c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:22.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:22.257 7f1e92259700 1 -- 172.21.2.112:0/2007668208 shutdown_connections 2021-02-10T11:15:22.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:22.257 7f1e92259700 1 -- 172.21.2.112:0/2007668208 wait complete. 2021-02-10T11:15:22.273 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:22.272 7f1e92259700 1 Processor -- start 2021-02-10T11:15:22.273 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:22.272 7f1e92259700 1 -- start start 2021-02-10T11:15:22.273 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:22.272 7f1e92259700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1e8c1a2930 0x7f1e8c1a1c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:22.273 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:22.272 7f1e92259700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1e8c1a5820 0x7f1e8c19a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:22.273 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:22.272 7f1e92259700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1e8c09d670 0x7f1e8c09da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:22.273 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:22.272 7f1e92259700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e8c19a970 con 0x7f1e8c09d670 2021-02-10T11:15:22.274 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:22.272 7f1e92259700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e8c1a23e0 con 0x7f1e8c1a2930 2021-02-10T11:15:22.274 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:22.272 7f1e92259700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e8c1a5380 con 0x7f1e8c1a5820 2021-02-10T11:15:22.274 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:22.272 7f1e90876700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1e8c09d670 0x7f1e8c09da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:22.274 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:22.272 7f1e90876700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1e8c09d670 0x7f1e8c09da70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:43240/0 (socket says 172.21.2.112:43240) 2021-02-10T11:15:22.274 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:22.272 7f1e90876700 1 -- 172.21.2.112:0/1771441608 learned_addr learned my addr 172.21.2.112:0/1771441608 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:15:22.274 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:22.273 7f1e8bfff700 1 --2- 172.21.2.112:0/1771441608 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1e8c1a2930 0x7f1e8c1a1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:22.274 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:22.273 7f1e8b7fe700 1 --2- 172.21.2.112:0/1771441608 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1e8c1a5820 0x7f1e8c19a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:22.274 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:22.273 7f1e90876700 1 -- 172.21.2.112:0/1771441608 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1e8c1a5820 msgr2=0x7f1e8c19a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:22.274 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:22.273 7f1e90876700 1 --2- 172.21.2.112:0/1771441608 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1e8c1a5820 0x7f1e8c19a2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:22.274 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:22.273 7f1e90876700 1 -- 172.21.2.112:0/1771441608 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1e8c1a2930 msgr2=0x7f1e8c1a1c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:22.274 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:22.273 7f1e90876700 1 --2- 172.21.2.112:0/1771441608 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1e8c1a2930 0x7f1e8c1a1c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:22.275 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:22.273 7f1e90876700 1 -- 172.21.2.112:0/1771441608 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e8c1a5c20 con 0x7f1e8c09d670 2021-02-10T11:15:22.275 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:22.273 7f1e8bfff700 1 --2- 172.21.2.112:0/1771441608 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1e8c1a2930 0x7f1e8c1a1c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:15:22.275 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:22.273 7f1e90876700 1 --2- 172.21.2.112:0/1771441608 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1e8c09d670 0x7f1e8c09da70 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7f1e8400b8e0 tx=0x7f1e8400bc80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:22.275 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:22.273 7f1e88ff9700 1 -- 172.21.2.112:0/1771441608 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1e8400c2f0 con 0x7f1e8c09d670 2021-02-10T11:15:22.275 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:22.273 7f1e88ff9700 1 -- 172.21.2.112:0/1771441608 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e8400c570 con 0x7f1e8c09d670 2021-02-10T11:15:22.275 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:22.273 7f1e92259700 1 -- 172.21.2.112:0/1771441608 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e8c1a5c20 con 0x7f1e8c09d670 2021-02-10T11:15:22.275 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:22.273 7f1e92259700 1 -- 172.21.2.112:0/1771441608 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e8c09e380 con 0x7f1e8c09d670 2021-02-10T11:15:22.275 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:22.273 7f1e88ff9700 1 -- 172.21.2.112:0/1771441608 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1e8400ca50 con 0x7f1e8c09d670 2021-02-10T11:15:22.275 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:22.274 7f1e88ff9700 1 -- 172.21.2.112:0/1771441608 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1e8400c750 con 0x7f1e8c09d670 2021-02-10T11:15:22.276 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:22.274 7f1e88ff9700 1 --2- 172.21.2.112:0/1771441608 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1e6c0186c0 0x7f1e6c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:22.276 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:22.274 7f1e88ff9700 1 -- 172.21.2.112:0/1771441608 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5659+0+0 (secure 0 0 0) 0x7f1e84032970 con 0x7f1e8c09d670 2021-02-10T11:15:22.276 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:22.274 7f1e8bfff700 1 --2- 172.21.2.112:0/1771441608 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1e6c0186c0 0x7f1e6c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:22.276 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:22.275 7f1e8bfff700 1 --2- 172.21.2.112:0/1771441608 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1e6c0186c0 0x7f1e6c01abc0 crc :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:22.303 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:22.305 7f1e92259700 1 -- 172.21.2.112:0/1771441608 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e8c1a23e0 con 0x7f1e8c09d670 2021-02-10T11:15:22.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:22.307 7f1e88ff9700 1 -- 172.21.2.112:0/1771441608 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1e8400c840 con 0x7f1e8c09d670 2021-02-10T11:15:22.431 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:22.433 7f1e92259700 1 -- 172.21.2.112:0/1771441608 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f1e8c0fe1c0 con 0x7f1e8c09d670 2021-02-10T11:15:22.790 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:22.792 7f1e88ff9700 1 -- 172.21.2.112:0/1771441608 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v22) v1 ==== 185+0+0 (secure 0 0 0) 0x7f1e8400c630 con 0x7f1e8c09d670 2021-02-10T11:15:22.798 INFO:teuthology.orchestra.run.gibba012.stderr:pool 'cephfs_metadata' removed 2021-02-10T11:15:22.800 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:22.802 7f1e92259700 1 -- 172.21.2.112:0/1771441608 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1e6c0186c0 msgr2=0x7f1e6c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:22.800 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:22.802 7f1e92259700 1 --2- 172.21.2.112:0/1771441608 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1e6c0186c0 0x7f1e6c01abc0 crc :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:22.800 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:22.802 7f1e92259700 1 -- 172.21.2.112:0/1771441608 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1e8c09d670 msgr2=0x7f1e8c09da70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:22.800 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:22.802 7f1e92259700 1 --2- 172.21.2.112:0/1771441608 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1e8c09d670 0x7f1e8c09da70 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7f1e8400b8e0 tx=0x7f1e8400bc80).stop 2021-02-10T11:15:22.800 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:22.802 7f1e92259700 1 -- 172.21.2.112:0/1771441608 shutdown_connections 2021-02-10T11:15:22.800 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:22.802 7f1e92259700 1 --2- 172.21.2.112:0/1771441608 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1e6c0186c0 0x7f1e6c01abc0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:22.801 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:22.802 7f1e92259700 1 --2- 172.21.2.112:0/1771441608 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1e8c09d670 0x7f1e8c09da70 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:22.801 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:22.802 7f1e92259700 1 --2- 172.21.2.112:0/1771441608 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1e8c1a5820 0x7f1e8c19a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:22.801 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:22.802 7f1e92259700 1 --2- 172.21.2.112:0/1771441608 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1e8c1a2930 0x7f1e8c1a1c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:22.801 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:22.802 7f1e92259700 1 -- 172.21.2.112:0/1771441608 >> 172.21.2.112:0/1771441608 conn(0x7f1e8c094ea0 msgr2=0x7f1e8c19acf0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:22.801 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:22.802 7f1e92259700 1 -- 172.21.2.112:0/1771441608 shutdown_connections 2021-02-10T11:15:22.801 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:22.802 7f1e92259700 1 -- 172.21.2.112:0/1771441608 wait complete. 2021-02-10T11:15:22.808 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2021-02-10T11:15:23.237 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:23.222 7fc0f5f08700 1 Processor -- start 2021-02-10T11:15:23.237 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:23.223 7fc0f5f08700 1 -- start start 2021-02-10T11:15:23.237 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:23.223 7fc0f5f08700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fc0f00d1870 0x7fc0f01b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:23.238 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:23.223 7fc0f5f08700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fc0f01b0ae0 0x7fc0f0108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:23.238 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:23.223 7fc0f5f08700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0f00fe1c0 con 0x7fc0f00d1c70 2021-02-10T11:15:23.238 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:23.223 7fc0f5f08700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0f0109a60 con 0x7fc0f01b0ae0 2021-02-10T11:15:23.238 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:23.223 7fc0f5f08700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0f0109750 con 0x7fc0f00d1870 2021-02-10T11:15:23.238 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:23.223 7fc0eeffd700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fc0f01b0ae0 0x7fc0f0108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:23.238 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:23.223 7fc0eeffd700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fc0f01b0ae0 0x7fc0f0108940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:43244/0 (socket says 172.21.2.112:43244) 2021-02-10T11:15:23.238 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:23.223 7fc0eeffd700 1 -- 172.21.2.112:0/3989617262 learned_addr learned my addr 172.21.2.112:0/3989617262 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:15:23.238 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:23.224 7fc0eeffd700 1 -- 172.21.2.112:0/3989617262 >> v2:172.21.2.120:3300/0 conn(0x7fc0f00d1870 msgr2=0x7fc0f01b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:15:23.238 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:23.224 7fc0eeffd700 1 --2- 172.21.2.112:0/3989617262 >> v2:172.21.2.120:3300/0 conn(0x7fc0f00d1870 0x7fc0f01b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:23.238 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:23.224 7fc0eeffd700 1 -- 172.21.2.112:0/3989617262 >> v1:172.21.2.120:6789/0 conn(0x7fc0f00d1c70 legacy=0x7fc0f0108f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:15:23.238 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:23.224 7fc0eeffd700 1 -- 172.21.2.112:0/3989617262 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0f0088020 con 0x7fc0f01b0ae0 2021-02-10T11:15:23.239 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:23.224 7fc0eeffd700 1 --2- 172.21.2.112:0/3989617262 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc0f01b0ae0 0x7fc0f0108940 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7fc0e0006700 tx=0x7fc0e0009b00).ready entity=mon.0 client_cookie=269a107ffefaefe5 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:23.239 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:23.224 7fc0ee7fc700 1 -- 172.21.2.112:0/3989617262 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc0e000c0c0 con 0x7fc0f01b0ae0 2021-02-10T11:15:23.239 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:23.224 7fc0ee7fc700 1 -- 172.21.2.112:0/3989617262 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0e000c400 con 0x7fc0f01b0ae0 2021-02-10T11:15:23.239 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:23.224 7fc0ee7fc700 1 -- 172.21.2.112:0/3989617262 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc0e000c920 con 0x7fc0f01b0ae0 2021-02-10T11:15:23.239 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:23.224 7fc0f5f08700 1 -- 172.21.2.112:0/3989617262 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc0f01b0ae0 msgr2=0x7fc0f0108940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:23.239 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:23.224 7fc0f5f08700 1 --2- 172.21.2.112:0/3989617262 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc0f01b0ae0 0x7fc0f0108940 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7fc0e0006700 tx=0x7fc0e0009b00).stop 2021-02-10T11:15:23.239 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:23.225 7fc0f5f08700 1 -- 172.21.2.112:0/3989617262 shutdown_connections 2021-02-10T11:15:23.239 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:23.225 7fc0f5f08700 1 --2- 172.21.2.112:0/3989617262 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc0f01b0ae0 0x7fc0f0108940 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:23.239 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:23.225 7fc0f5f08700 1 --2- 172.21.2.112:0/3989617262 >> v2:172.21.2.120:3300/0 conn(0x7fc0f00d1870 0x7fc0f01b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:23.239 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:23.225 7fc0f5f08700 1 -- 172.21.2.112:0/3989617262 >> 172.21.2.112:0/3989617262 conn(0x7fc0f00ce420 msgr2=0x7fc0f00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:23.239 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:23.225 7fc0f5f08700 1 -- 172.21.2.112:0/3989617262 shutdown_connections 2021-02-10T11:15:23.239 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:23.225 7fc0f5f08700 1 -- 172.21.2.112:0/3989617262 wait complete. 2021-02-10T11:15:23.241 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:23.239 7fc0f5f08700 1 Processor -- start 2021-02-10T11:15:23.241 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:23.240 7fc0f5f08700 1 -- start start 2021-02-10T11:15:23.241 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:23.240 7fc0f5f08700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc0f00b7d30 0x7fc0f00b8130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:23.241 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:23.240 7fc0f5f08700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc0f0095780 0x7fc0f0095b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:23.241 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:23.240 7fc0f5f08700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc0f00961a0 0x7fc0f00965a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:23.241 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:23.240 7fc0f5f08700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0f019f110 con 0x7fc0f00961a0 2021-02-10T11:15:23.241 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:23.240 7fc0f5f08700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0f01abb60 con 0x7fc0f00b7d30 2021-02-10T11:15:23.241 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:23.240 7fc0f5f08700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0f0194a70 con 0x7fc0f0095780 2021-02-10T11:15:23.242 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:23.240 7fc0effff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc0f00961a0 0x7fc0f00965a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:23.242 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:23.240 7fc0effff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc0f00961a0 0x7fc0f00965a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:43250/0 (socket says 172.21.2.112:43250) 2021-02-10T11:15:23.242 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:23.240 7fc0effff700 1 -- 172.21.2.112:0/1430127486 learned_addr learned my addr 172.21.2.112:0/1430127486 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:15:23.242 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:23.241 7fc0ef7fe700 1 --2- 172.21.2.112:0/1430127486 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc0f00b7d30 0x7fc0f00b8130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:23.242 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:23.241 7fc0eeffd700 1 --2- 172.21.2.112:0/1430127486 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc0f0095780 0x7fc0f0095b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:23.242 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:23.241 7fc0effff700 1 -- 172.21.2.112:0/1430127486 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc0f0095780 msgr2=0x7fc0f0095b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:23.242 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:23.241 7fc0effff700 1 --2- 172.21.2.112:0/1430127486 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc0f0095780 0x7fc0f0095b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:23.242 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:23.241 7fc0effff700 1 -- 172.21.2.112:0/1430127486 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc0f00b7d30 msgr2=0x7fc0f00b8130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:23.242 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:23.241 7fc0effff700 1 --2- 172.21.2.112:0/1430127486 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc0f00b7d30 0x7fc0f00b8130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:23.242 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:23.241 7fc0effff700 1 -- 172.21.2.112:0/1430127486 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0f0194c90 con 0x7fc0f00961a0 2021-02-10T11:15:23.242 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:23.241 7fc0ef7fe700 1 --2- 172.21.2.112:0/1430127486 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc0f00b7d30 0x7fc0f00b8130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:15:23.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:23.241 7fc0effff700 1 --2- 172.21.2.112:0/1430127486 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc0f00961a0 0x7fc0f00965a0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7fc0e8009940 tx=0x7fc0e800bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:23.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:23.241 7fc0e7fff700 1 -- 172.21.2.112:0/1430127486 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc0e8000f90 con 0x7fc0f00961a0 2021-02-10T11:15:23.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:23.241 7fc0e7fff700 1 -- 172.21.2.112:0/1430127486 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0e800c2f0 con 0x7fc0f00961a0 2021-02-10T11:15:23.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:23.241 7fc0e7fff700 1 -- 172.21.2.112:0/1430127486 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc0e800c810 con 0x7fc0f00961a0 2021-02-10T11:15:23.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:23.241 7fc0f5f08700 1 -- 172.21.2.112:0/1430127486 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0f0194c90 con 0x7fc0f00961a0 2021-02-10T11:15:23.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:23.241 7fc0f5f08700 1 -- 172.21.2.112:0/1430127486 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0f0194c90 con 0x7fc0f00961a0 2021-02-10T11:15:23.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:23.242 7fc0e7fff700 1 -- 172.21.2.112:0/1430127486 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc0e8001220 con 0x7fc0f00961a0 2021-02-10T11:15:23.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:23.242 7fc0e7fff700 1 --2- 172.21.2.112:0/1430127486 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc0d40186d0 0x7fc0d401abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:23.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:23.242 7fc0e7fff700 1 -- 172.21.2.112:0/1430127486 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5221+0+0 (secure 0 0 0) 0x7fc0e8032a20 con 0x7fc0f00961a0 2021-02-10T11:15:23.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:23.242 7fc0ef7fe700 1 --2- 172.21.2.112:0/1430127486 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc0d40186d0 0x7fc0d401abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:23.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:23.243 7fc0ef7fe700 1 --2- 172.21.2.112:0/1430127486 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc0d40186d0 0x7fc0d401abd0 crc :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:23.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:23.273 7fc0f5f08700 1 -- 172.21.2.112:0/1430127486 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0f01961b0 con 0x7fc0f00961a0 2021-02-10T11:15:23.273 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:23.275 7fc0e7fff700 1 -- 172.21.2.112:0/1430127486 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc0e800be30 con 0x7fc0f00961a0 2021-02-10T11:15:23.401 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:23.403 7fc0f5f08700 1 -- 172.21.2.112:0/1430127486 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fc0f01961b0 con 0x7fc0f00961a0 2021-02-10T11:15:23.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:23.797 7fc0e7fff700 1 -- 172.21.2.112:0/1430127486 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v23) v1 ==== 173+0+0 (secure 0 0 0) 0x7fc0e8001220 con 0x7fc0f00961a0 2021-02-10T11:15:23.818 INFO:teuthology.orchestra.run.gibba012.stderr:pool 'cephfs_data' removed 2021-02-10T11:15:23.819 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:23.822 7fc0f5f08700 1 -- 172.21.2.112:0/1430127486 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc0d40186d0 msgr2=0x7fc0d401abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:23.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:23.822 7fc0f5f08700 1 --2- 172.21.2.112:0/1430127486 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc0d40186d0 0x7fc0d401abd0 crc :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:23.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:23.822 7fc0f5f08700 1 -- 172.21.2.112:0/1430127486 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc0f00961a0 msgr2=0x7fc0f00965a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:23.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:23.822 7fc0f5f08700 1 --2- 172.21.2.112:0/1430127486 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc0f00961a0 0x7fc0f00965a0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7fc0e8009940 tx=0x7fc0e800bcf0).stop 2021-02-10T11:15:23.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:23.822 7fc0f5f08700 1 -- 172.21.2.112:0/1430127486 shutdown_connections 2021-02-10T11:15:23.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:23.822 7fc0f5f08700 1 --2- 172.21.2.112:0/1430127486 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc0d40186d0 0x7fc0d401abd0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:23.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:23.822 7fc0f5f08700 1 --2- 172.21.2.112:0/1430127486 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc0f00961a0 0x7fc0f00965a0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:23.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:23.822 7fc0f5f08700 1 --2- 172.21.2.112:0/1430127486 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc0f0095780 0x7fc0f0095b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:23.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:23.822 7fc0f5f08700 1 --2- 172.21.2.112:0/1430127486 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc0f00b7d30 0x7fc0f00b8130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:23.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:23.822 7fc0f5f08700 1 -- 172.21.2.112:0/1430127486 >> 172.21.2.112:0/1430127486 conn(0x7fc0f019f820 msgr2=0x7fc0f0082b10 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:23.821 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:23.822 7fc0f5f08700 1 -- 172.21.2.112:0/1430127486 shutdown_connections 2021-02-10T11:15:23.821 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:23.822 7fc0f5f08700 1 -- 172.21.2.112:0/1430127486 wait complete. 2021-02-10T11:15:23.828 INFO:tasks.ceph.mds.a:Restarting daemon 2021-02-10T11:15:23.828 INFO:tasks.ceph.mds.a:Stopping old one... 2021-02-10T11:15:23.828 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2021-02-10T11:15:23.828 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:15:23.829 INFO:tasks.ceph.mds.b:Restarting daemon 2021-02-10T11:15:23.829 INFO:tasks.ceph.mds.b:Stopping old one... 2021-02-10T11:15:23.829 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2021-02-10T11:15:23.829 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:15:23.835 INFO:tasks.ceph.mds.a:Stopped 2021-02-10T11:15:23.835 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2021-02-10T11:15:23.837 INFO:tasks.ceph.mds.b:Stopped 2021-02-10T11:15:23.838 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2021-02-10T11:15:23.876 INFO:tasks.ceph.mds.a:Started 2021-02-10T11:15:23.876 DEBUG:teuthology.parallel:result is None 2021-02-10T11:15:23.878 INFO:tasks.ceph.mds.b:Started 2021-02-10T11:15:23.879 DEBUG:teuthology.parallel:result is None 2021-02-10T11:15:23.879 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd blacklist clear 2021-02-10T11:15:24.281 INFO:tasks.ceph.mds.b.gibba020.stdout:starting mds.b at 2021-02-10T11:15:24.282 INFO:tasks.ceph.mds.a.gibba012.stdout:starting mds.a at 2021-02-10T11:15:24.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:24.289 7fe0335cf700 1 Processor -- start 2021-02-10T11:15:24.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:24.290 7fe0335cf700 1 -- start start 2021-02-10T11:15:24.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:24.290 7fe0335cf700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe02c0e5070 0x7fe02c1b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:24.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:24.290 7fe0335cf700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fe02c1b3e40 0x7fe02c080310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:24.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:24.290 7fe0335cf700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe02c07ed80 con 0x7fe02c0e5470 2021-02-10T11:15:24.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:24.290 7fe0335cf700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe02c0d1790 con 0x7fe02c1b3e40 2021-02-10T11:15:24.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:24.290 7fe0335cf700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe02c0d1980 con 0x7fe02c0e5070 2021-02-10T11:15:24.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:24.290 7fe031bec700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fe02c1b3e40 0x7fe02c080310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:24.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:24.290 7fe030bea700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fe02c0e5470 0x7fe02c07d690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:42734/0 (socket says 172.21.2.112:42734) 2021-02-10T11:15:24.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:24.290 7fe030bea700 1 -- 172.21.2.112:0/2213038367 learned_addr learned my addr 172.21.2.112:0/2213038367 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:15:24.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:24.290 7fe031bec700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fe02c1b3e40 0x7fe02c080310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:43270/0 (socket says 172.21.2.112:43270) 2021-02-10T11:15:24.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:24.290 7fe0313eb700 1 --2- 172.21.2.112:0/2213038367 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe02c0e5070 0x7fe02c1b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:24.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:24.290 7fe031bec700 1 -- 172.21.2.112:0/2213038367 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe02c0e5070 msgr2=0x7fe02c1b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:15:24.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:24.290 7fe031bec700 1 --2- 172.21.2.112:0/2213038367 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe02c0e5070 0x7fe02c1b3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:24.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:24.290 7fe031bec700 1 -- 172.21.2.112:0/2213038367 >> v1:172.21.2.112:6789/0 conn(0x7fe02c0e5470 legacy=0x7fe02c07d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:24.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:24.290 7fe031bec700 1 -- 172.21.2.112:0/2213038367 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe02c0d1ba0 con 0x7fe02c1b3e40 2021-02-10T11:15:24.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:24.291 7fe031bec700 1 --2- 172.21.2.112:0/2213038367 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe02c1b3e40 0x7fe02c080310 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7fe024006e30 tx=0x7fe024009e60).ready entity=mon.0 client_cookie=4b4ba47ef90bdd8b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:24.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:24.291 7fe023fff700 1 -- 172.21.2.112:0/2213038367 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe024006a50 con 0x7fe02c1b3e40 2021-02-10T11:15:24.306 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:24.291 7fe023fff700 1 -- 172.21.2.112:0/2213038367 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe02400b300 con 0x7fe02c1b3e40 2021-02-10T11:15:24.306 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:24.291 7fe023fff700 1 -- 172.21.2.112:0/2213038367 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe02400b820 con 0x7fe02c1b3e40 2021-02-10T11:15:24.306 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:24.291 7fe0335cf700 1 -- 172.21.2.112:0/2213038367 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe02c1b3e40 msgr2=0x7fe02c080310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:24.306 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:24.291 7fe0335cf700 1 --2- 172.21.2.112:0/2213038367 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe02c1b3e40 0x7fe02c080310 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7fe024006e30 tx=0x7fe024009e60).stop 2021-02-10T11:15:24.306 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:24.291 7fe0335cf700 1 -- 172.21.2.112:0/2213038367 shutdown_connections 2021-02-10T11:15:24.306 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:24.291 7fe0335cf700 1 --2- 172.21.2.112:0/2213038367 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe02c1b3e40 0x7fe02c080310 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:24.306 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:24.291 7fe0335cf700 1 --2- 172.21.2.112:0/2213038367 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe02c0e5070 0x7fe02c1b3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:24.306 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:24.291 7fe0335cf700 1 -- 172.21.2.112:0/2213038367 >> 172.21.2.112:0/2213038367 conn(0x7fe02c0ce690 msgr2=0x7fe02c0e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:24.306 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:24.292 7fe0335cf700 1 -- 172.21.2.112:0/2213038367 shutdown_connections 2021-02-10T11:15:24.306 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:24.292 7fe0335cf700 1 -- 172.21.2.112:0/2213038367 wait complete. 2021-02-10T11:15:24.308 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:24.306 7fe0335cf700 1 Processor -- start 2021-02-10T11:15:24.308 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:24.306 7fe0335cf700 1 -- start start 2021-02-10T11:15:24.308 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:24.307 7fe0335cf700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe02c090e20 0x7fe02c094ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:24.308 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:24.307 7fe0335cf700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe02c0849f0 0x7fe02c095100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:24.308 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:24.307 7fe0335cf700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe02c095720 0x7fe02c07f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:24.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:24.307 7fe0335cf700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe02c084fe0 con 0x7fe02c0849f0 2021-02-10T11:15:24.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:24.307 7fe0335cf700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe02c091350 con 0x7fe02c095720 2021-02-10T11:15:24.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:24.307 7fe0335cf700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe02c07f840 con 0x7fe02c090e20 2021-02-10T11:15:24.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:24.307 7fe030bea700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe02c0849f0 0x7fe02c095100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:24.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:24.307 7fe030bea700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe02c0849f0 0x7fe02c095100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:43274/0 (socket says 172.21.2.112:43274) 2021-02-10T11:15:24.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:24.307 7fe030bea700 1 -- 172.21.2.112:0/739502359 learned_addr learned my addr 172.21.2.112:0/739502359 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:15:24.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:24.307 7fe031bec700 1 --2- 172.21.2.112:0/739502359 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe02c095720 0x7fe02c07f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:24.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:24.307 7fe0313eb700 1 --2- 172.21.2.112:0/739502359 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe02c090e20 0x7fe02c094ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:24.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:24.307 7fe030bea700 1 -- 172.21.2.112:0/739502359 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe02c090e20 msgr2=0x7fe02c094ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:24.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:24.307 7fe030bea700 1 --2- 172.21.2.112:0/739502359 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe02c090e20 0x7fe02c094ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:24.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:24.307 7fe030bea700 1 -- 172.21.2.112:0/739502359 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe02c095720 msgr2=0x7fe02c07f220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:24.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:24.307 7fe030bea700 1 --2- 172.21.2.112:0/739502359 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe02c095720 0x7fe02c07f220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:24.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:24.307 7fe030bea700 1 -- 172.21.2.112:0/739502359 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe02c07fa60 con 0x7fe02c0849f0 2021-02-10T11:15:24.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:24.307 7fe0313eb700 1 --2- 172.21.2.112:0/739502359 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe02c090e20 0x7fe02c094ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:15:24.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:24.307 7fe030bea700 1 --2- 172.21.2.112:0/739502359 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe02c0849f0 0x7fe02c095100 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7fe01c001c60 tx=0x7fe01c00a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:24.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:24.308 7fe021ffb700 1 -- 172.21.2.112:0/739502359 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe01c00acc0 con 0x7fe02c0849f0 2021-02-10T11:15:24.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:24.308 7fe021ffb700 1 -- 172.21.2.112:0/739502359 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe01c004140 con 0x7fe02c0849f0 2021-02-10T11:15:24.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:24.308 7fe0335cf700 1 -- 172.21.2.112:0/739502359 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe02c07fa60 con 0x7fe02c0849f0 2021-02-10T11:15:24.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:24.308 7fe0335cf700 1 -- 172.21.2.112:0/739502359 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe02c07fff0 con 0x7fe02c0849f0 2021-02-10T11:15:24.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:24.308 7fe021ffb700 1 -- 172.21.2.112:0/739502359 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe01c004510 con 0x7fe02c0849f0 2021-02-10T11:15:24.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:24.308 7fe021ffb700 1 -- 172.21.2.112:0/739502359 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe01c004320 con 0x7fe02c0849f0 2021-02-10T11:15:24.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:24.308 7fe021ffb700 1 --2- 172.21.2.112:0/739502359 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe00c018730 0x7fe00c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:24.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:24.308 7fe021ffb700 1 -- 172.21.2.112:0/739502359 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 4839+0+0 (secure 0 0 0) 0x7fe01c032490 con 0x7fe02c0849f0 2021-02-10T11:15:24.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:24.308 7fe0313eb700 1 --2- 172.21.2.112:0/739502359 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe00c018730 0x7fe00c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:24.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:24.309 7fe0313eb700 1 --2- 172.21.2.112:0/739502359 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe00c018730 0x7fe00c01abc0 crc :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:24.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:24.340 7fe0335cf700 1 -- 172.21.2.112:0/739502359 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe02c107180 con 0x7fe02c0849f0 2021-02-10T11:15:24.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:24.342 7fe021ffb700 1 -- 172.21.2.112:0/739502359 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe01c000dd0 con 0x7fe02c0849f0 2021-02-10T11:15:24.481 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:24.483 7fe0335cf700 1 -- 172.21.2.112:0/739502359 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7fe02c084fe0 con 0x7fe02c0849f0 2021-02-10T11:15:24.800 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:24.802 7fe021ffb700 1 -- 172.21.2.112:0/739502359 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v24) v1 ==== 97+0+0 (secure 0 0 0) 0x7fe01c004140 con 0x7fe02c0849f0 2021-02-10T11:15:24.846 INFO:teuthology.orchestra.run.gibba012.stderr: removed all blacklist entries 2021-02-10T11:15:24.848 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:24.850 7fe0335cf700 1 -- 172.21.2.112:0/739502359 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe00c018730 msgr2=0x7fe00c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:24.848 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:24.850 7fe0335cf700 1 --2- 172.21.2.112:0/739502359 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe00c018730 0x7fe00c01abc0 crc :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:24.848 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:24.850 7fe0335cf700 1 -- 172.21.2.112:0/739502359 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe02c0849f0 msgr2=0x7fe02c095100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:24.848 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:24.850 7fe0335cf700 1 --2- 172.21.2.112:0/739502359 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe02c0849f0 0x7fe02c095100 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7fe01c001c60 tx=0x7fe01c00a8d0).stop 2021-02-10T11:15:24.849 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:24.850 7fe0335cf700 1 -- 172.21.2.112:0/739502359 shutdown_connections 2021-02-10T11:15:24.849 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:24.850 7fe0335cf700 1 --2- 172.21.2.112:0/739502359 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe00c018730 0x7fe00c01abc0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:24.849 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:24.850 7fe0335cf700 1 --2- 172.21.2.112:0/739502359 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe02c095720 0x7fe02c07f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:24.849 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:24.850 7fe0335cf700 1 --2- 172.21.2.112:0/739502359 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe02c0849f0 0x7fe02c095100 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:24.849 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:24.850 7fe0335cf700 1 --2- 172.21.2.112:0/739502359 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe02c090e20 0x7fe02c094ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:24.849 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:24.850 7fe0335cf700 1 -- 172.21.2.112:0/739502359 >> 172.21.2.112:0/739502359 conn(0x7fe02c107d80 msgr2=0x7fe02c1073b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:24.849 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:24.851 7fe0335cf700 1 -- 172.21.2.112:0/739502359 shutdown_connections 2021-02-10T11:15:24.849 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:24.851 7fe0335cf700 1 -- 172.21.2.112:0/739502359 wait complete. 2021-02-10T11:15:24.856 INFO:tasks.cephfs.cephfs_test_case:['0', '1', '2', '3'] 2021-02-10T11:15:24.856 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth ls --format=json-pretty 2021-02-10T11:15:25.282 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.267 7fc7aa6c5700 1 Processor -- start 2021-02-10T11:15:25.282 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.267 7fc7aa6c5700 1 -- start start 2021-02-10T11:15:25.282 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.267 7fc7aa6c5700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fc7a40e5070 0x7fc7a41b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:25.282 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.267 7fc7aa6c5700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc7a40e5470 0x7fc7a4080310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:25.282 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.267 7fc7aa6c5700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7a407ed80 con 0x7fc7a41b3e40 2021-02-10T11:15:25.283 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.267 7fc7aa6c5700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7a40d1790 con 0x7fc7a40e5070 2021-02-10T11:15:25.283 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.267 7fc7aa6c5700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7a40d19b0 con 0x7fc7a40e5470 2021-02-10T11:15:25.283 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.268 7fc7a8ce2700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc7a40e5470 0x7fc7a4080310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:25.283 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.268 7fc7a3fff700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fc7a40e5070 0x7fc7a41b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:25.283 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.268 7fc7a8ce2700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc7a40e5470 0x7fc7a4080310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3301/0 says I am v2:172.21.2.112:42964/0 (socket says 172.21.2.112:42964) 2021-02-10T11:15:25.283 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.268 7fc7a8ce2700 1 -- 172.21.2.112:0/3812992242 learned_addr learned my addr 172.21.2.112:0/3812992242 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:15:25.283 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.269 7fc7a8ce2700 1 -- 172.21.2.112:0/3812992242 >> v2:172.21.2.120:3300/0 conn(0x7fc7a40e5070 msgr2=0x7fc7a41b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:15:25.283 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.269 7fc7a8ce2700 1 --2- 172.21.2.112:0/3812992242 >> v2:172.21.2.120:3300/0 conn(0x7fc7a40e5070 0x7fc7a41b3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:25.283 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.269 7fc7a8ce2700 1 -- 172.21.2.112:0/3812992242 >> v1:172.21.2.120:6789/0 conn(0x7fc7a41b3e40 legacy=0x7fc7a407d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:25.283 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.269 7fc7a8ce2700 1 -- 172.21.2.112:0/3812992242 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7a40d1bd0 con 0x7fc7a40e5470 2021-02-10T11:15:25.283 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.269 7fc7a3fff700 1 --2- 172.21.2.112:0/3812992242 >> v2:172.21.2.120:3300/0 conn(0x7fc7a40e5070 0x7fc7a41b3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:15:25.283 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.269 7fc7a8ce2700 1 --2- 172.21.2.112:0/3812992242 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc7a40e5470 0x7fc7a4080310 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7fc79c006aa0 tx=0x7fc79c009f20).ready entity=mon.2 client_cookie=f2b162a7ecc647a4 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:25.283 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.269 7fc7a2ffd700 1 -- 172.21.2.112:0/3812992242 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc79c00b5c0 con 0x7fc7a40e5470 2021-02-10T11:15:25.283 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.269 7fc7a2ffd700 1 -- 172.21.2.112:0/3812992242 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc79c003030 con 0x7fc7a40e5470 2021-02-10T11:15:25.284 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.269 7fc7a2ffd700 1 -- 172.21.2.112:0/3812992242 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc79c00bb40 con 0x7fc7a40e5470 2021-02-10T11:15:25.284 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.269 7fc7aa6c5700 1 -- 172.21.2.112:0/3812992242 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc7a40e5470 msgr2=0x7fc7a4080310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:25.284 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.269 7fc7aa6c5700 1 --2- 172.21.2.112:0/3812992242 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc7a40e5470 0x7fc7a4080310 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7fc79c006aa0 tx=0x7fc79c009f20).stop 2021-02-10T11:15:25.284 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.270 7fc7aa6c5700 1 -- 172.21.2.112:0/3812992242 shutdown_connections 2021-02-10T11:15:25.284 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.270 7fc7aa6c5700 1 --2- 172.21.2.112:0/3812992242 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc7a40e5470 0x7fc7a4080310 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:25.284 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.270 7fc7aa6c5700 1 --2- 172.21.2.112:0/3812992242 >> v2:172.21.2.120:3300/0 conn(0x7fc7a40e5070 0x7fc7a41b3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:25.284 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.270 7fc7aa6c5700 1 -- 172.21.2.112:0/3812992242 >> 172.21.2.112:0/3812992242 conn(0x7fc7a40ce690 msgr2=0x7fc7a40e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:25.284 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.270 7fc7aa6c5700 1 -- 172.21.2.112:0/3812992242 shutdown_connections 2021-02-10T11:15:25.284 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.270 7fc7aa6c5700 1 -- 172.21.2.112:0/3812992242 wait complete. 2021-02-10T11:15:25.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.284 7fc7aa6c5700 1 Processor -- start 2021-02-10T11:15:25.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.285 7fc7aa6c5700 1 -- start start 2021-02-10T11:15:25.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.285 7fc7aa6c5700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc7a408d5a0 0x7fc7a40ad560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:25.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.285 7fc7aa6c5700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc7a40adb80 0x7fc7a41a1b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:25.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.285 7fc7aa6c5700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc7a41a21b0 0x7fc7a41a25b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:25.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.285 7fc7aa6c5700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7a41b2f70 con 0x7fc7a40adb80 2021-02-10T11:15:25.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.285 7fc7aa6c5700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7a408d9a0 con 0x7fc7a41a21b0 2021-02-10T11:15:25.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.285 7fc7aa6c5700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7a408dbc0 con 0x7fc7a408d5a0 2021-02-10T11:15:25.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.285 7fc7a37fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc7a40adb80 0x7fc7a41a1b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:25.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.285 7fc7a37fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc7a40adb80 0x7fc7a41a1b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:43286/0 (socket says 172.21.2.112:43286) 2021-02-10T11:15:25.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.285 7fc7a37fe700 1 -- 172.21.2.112:0/1022864165 learned_addr learned my addr 172.21.2.112:0/1022864165 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:15:25.287 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.286 7fc7a8ce2700 1 --2- 172.21.2.112:0/1022864165 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc7a41a21b0 0x7fc7a41a25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:25.287 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.286 7fc7a3fff700 1 --2- 172.21.2.112:0/1022864165 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc7a408d5a0 0x7fc7a40ad560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:25.287 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.286 7fc7a37fe700 1 -- 172.21.2.112:0/1022864165 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc7a408d5a0 msgr2=0x7fc7a40ad560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:25.287 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.286 7fc7a37fe700 1 --2- 172.21.2.112:0/1022864165 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc7a408d5a0 0x7fc7a40ad560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:25.287 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.286 7fc7a37fe700 1 -- 172.21.2.112:0/1022864165 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc7a41a21b0 msgr2=0x7fc7a41a25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:25.287 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.286 7fc7a37fe700 1 --2- 172.21.2.112:0/1022864165 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc7a41a21b0 0x7fc7a41a25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:25.287 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.286 7fc7a37fe700 1 -- 172.21.2.112:0/1022864165 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7a41a2bd0 con 0x7fc7a40adb80 2021-02-10T11:15:25.287 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.286 7fc7a8ce2700 1 --2- 172.21.2.112:0/1022864165 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc7a41a21b0 0x7fc7a41a25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:15:25.287 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.286 7fc7a3fff700 1 --2- 172.21.2.112:0/1022864165 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc7a408d5a0 0x7fc7a40ad560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:15:25.287 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.286 7fc7a37fe700 1 --2- 172.21.2.112:0/1022864165 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc7a40adb80 0x7fc7a41a1b90 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7fc79800a570 tx=0x7fc79800a910).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:25.287 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.286 7fc7a0ff9700 1 -- 172.21.2.112:0/1022864165 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc79800ad00 con 0x7fc7a40adb80 2021-02-10T11:15:25.287 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.286 7fc7a0ff9700 1 -- 172.21.2.112:0/1022864165 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc79800b070 con 0x7fc7a40adb80 2021-02-10T11:15:25.287 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.286 7fc7aa6c5700 1 -- 172.21.2.112:0/1022864165 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7a41a2bd0 con 0x7fc7a40adb80 2021-02-10T11:15:25.288 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.286 7fc7a0ff9700 1 -- 172.21.2.112:0/1022864165 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc79800b4d0 con 0x7fc7a40adb80 2021-02-10T11:15:25.288 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.286 7fc7aa6c5700 1 -- 172.21.2.112:0/1022864165 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7a41a3960 con 0x7fc7a40adb80 2021-02-10T11:15:25.288 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.287 7fc7a0ff9700 1 -- 172.21.2.112:0/1022864165 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc79800ad00 con 0x7fc7a40adb80 2021-02-10T11:15:25.288 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.287 7fc7a0ff9700 1 --2- 172.21.2.112:0/1022864165 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc7840186c0 0x7fc78401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:25.288 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.287 7fc7a0ff9700 1 -- 172.21.2.112:0/1022864165 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 4753+0+0 (secure 0 0 0) 0x7fc798032140 con 0x7fc7a40adb80 2021-02-10T11:15:25.288 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.287 7fc7a3fff700 1 --2- 172.21.2.112:0/1022864165 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc7840186c0 0x7fc78401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:25.288 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.287 7fc7a3fff700 1 --2- 172.21.2.112:0/1022864165 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc7840186c0 0x7fc78401abc0 crc :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:25.315 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.317 7fc7aa6c5700 1 -- 172.21.2.112:0/1022864165 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7a4107170 con 0x7fc7a40adb80 2021-02-10T11:15:25.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.319 7fc7a0ff9700 1 -- 172.21.2.112:0/1022864165 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc798000990 con 0x7fc7a40adb80 2021-02-10T11:15:25.449 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.451 7fc7aa6c5700 1 -- 172.21.2.112:0/1022864165 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fc7a407ed80 con 0x7fc7a40adb80 2021-02-10T11:15:25.449 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.451 7fc7a0ff9700 1 -- 172.21.2.112:0/1022864165 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v6) v1 ==== 80+0+5917 (secure 0 0 0) 0x7fc79800ad00 con 0x7fc7a40adb80 2021-02-10T11:15:25.450 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:15:25.450 INFO:teuthology.orchestra.run.gibba012.stdout:{ 2021-02-10T11:15:25.450 INFO:teuthology.orchestra.run.gibba012.stdout: "auth_dump": [ 2021-02-10T11:15:25.450 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:15:25.450 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "mds.a", 2021-02-10T11:15:25.450 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCKvyNgPkV4AxAAEmVNaghdccP6lWd8hvQnxw==", 2021-02-10T11:15:25.450 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:15:25.450 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:15:25.451 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:15:25.451 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:15:25.451 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:15:25.451 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:15:25.451 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:25.451 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:15:25.451 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "mds.b", 2021-02-10T11:15:25.451 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCKvyNguxONLhAAHIpry+R89MAwdhFJyEhVCw==", 2021-02-10T11:15:25.451 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:15:25.451 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:15:25.451 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:15:25.452 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:15:25.452 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:15:25.452 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:15:25.453 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:25.453 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:15:25.453 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.0", 2021-02-10T11:15:25.453 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCSvyNgXcKPLhAAP/Bj2F+06DCtPCzQDNq+cQ==", 2021-02-10T11:15:25.453 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:15:25.453 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:15:25.453 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:15:25.453 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:15:25.453 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:15:25.453 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:25.453 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:15:25.453 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.1", 2021-02-10T11:15:25.453 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCVvyNghfbcLhAA66PrRwR2M7o6VTF5UhU/Fg==", 2021-02-10T11:15:25.454 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:15:25.454 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:15:25.454 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:15:25.454 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:15:25.454 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:15:25.454 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:25.454 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:15:25.454 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.2", 2021-02-10T11:15:25.454 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCYvyNgwiPqLhAA66BVkRsgaUlGpAsQ15Nf8Q==", 2021-02-10T11:15:25.454 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:15:25.454 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:15:25.454 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:15:25.455 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:15:25.455 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:15:25.455 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:25.455 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:15:25.455 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.3", 2021-02-10T11:15:25.455 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCbvyNgqBZrLxAAERPe/1L3SXAEzI3d9ReHHg==", 2021-02-10T11:15:25.455 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:15:25.455 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:15:25.455 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:15:25.455 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:15:25.455 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:15:25.455 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:25.455 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:15:25.455 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.4", 2021-02-10T11:15:25.456 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCkvyNgDQsxIhAAHmVjVYnx20N4SDmKqvbcuA==", 2021-02-10T11:15:25.456 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:15:25.456 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:15:25.456 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:15:25.456 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:15:25.456 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:15:25.456 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:25.456 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:15:25.456 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.5", 2021-02-10T11:15:25.456 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCnvyNg8ITFIhAADd4aKybAV7btoGPx3kCg3A==", 2021-02-10T11:15:25.456 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:15:25.456 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:15:25.456 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:15:25.457 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:15:25.457 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:15:25.457 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:25.457 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:15:25.457 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.6", 2021-02-10T11:15:25.457 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCqvyNgxOhDIxAAzXixEa/nlDc1xXrg0aeLwg==", 2021-02-10T11:15:25.457 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:15:25.457 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:15:25.457 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:15:25.457 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:15:25.457 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:15:25.457 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:25.458 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:15:25.458 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.7", 2021-02-10T11:15:25.458 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCtvyNgxSqGIxAACxJQlwP7DpiVVCZHA3zn1A==", 2021-02-10T11:15:25.458 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:15:25.458 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:15:25.458 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:15:25.458 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:15:25.458 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:15:25.458 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:25.458 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:15:25.458 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.0", 2021-02-10T11:15:25.458 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCLvyNgDkNiHRAAygU8Za4PBusTqCgwAYWTiQ==", 2021-02-10T11:15:25.458 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:15:25.459 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:15:25.459 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow r", 2021-02-10T11:15:25.459 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow rw", 2021-02-10T11:15:25.459 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow rwx" 2021-02-10T11:15:25.459 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:15:25.459 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:25.459 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:15:25.459 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.1", 2021-02-10T11:15:25.459 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCLvyNgxmLqMxAA1MlSDJ9LH9u4f2sL7mm8/Q==", 2021-02-10T11:15:25.459 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:15:25.459 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:15:25.459 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow r", 2021-02-10T11:15:25.459 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow rw", 2021-02-10T11:15:25.460 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow rwx" 2021-02-10T11:15:25.460 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:15:25.460 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:25.460 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:15:25.460 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.2", 2021-02-10T11:15:25.460 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCMvyNgp8veDhAAniSVHYPtwwTog/pjbaRQmw==", 2021-02-10T11:15:25.460 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:15:25.460 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:15:25.460 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow r", 2021-02-10T11:15:25.460 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow rw", 2021-02-10T11:15:25.460 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow rwx" 2021-02-10T11:15:25.460 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:15:25.461 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:25.461 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:15:25.461 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.3", 2021-02-10T11:15:25.461 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCMvyNgQEVoJRAApHcWdm1sldIxu+heyD4HdQ==", 2021-02-10T11:15:25.461 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:15:25.461 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:15:25.461 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow r", 2021-02-10T11:15:25.461 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow rw", 2021-02-10T11:15:25.461 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow rwx" 2021-02-10T11:15:25.461 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:15:25.461 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:25.461 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:15:25.461 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.admin", 2021-02-10T11:15:25.461 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCGvyNg7jjJBRAAtot+ypUZOiTo1GbY+oVhVw==", 2021-02-10T11:15:25.462 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:15:25.462 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow *", 2021-02-10T11:15:25.462 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:15:25.462 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:15:25.462 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:15:25.462 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:15:25.462 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:25.462 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:15:25.462 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.bootstrap-mds", 2021-02-10T11:15:25.462 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQDWvyNgv8XmHRAAijMFw9aEdtfTdUsTCRlzAA==", 2021-02-10T11:15:25.462 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:15:25.462 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow profile bootstrap-mds" 2021-02-10T11:15:25.462 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:15:25.463 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:25.463 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:15:25.463 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.bootstrap-mgr", 2021-02-10T11:15:25.463 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQDWvyNg/c7mHRAAIGhmBNLAsHzcv9C0Dq4aVA==", 2021-02-10T11:15:25.463 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:15:25.463 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow profile bootstrap-mgr" 2021-02-10T11:15:25.463 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:15:25.463 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:25.463 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:15:25.463 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.bootstrap-osd", 2021-02-10T11:15:25.463 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQDWvyNg9NfmHRAA/4Q4W4z9DDzjOp3poPj2mQ==", 2021-02-10T11:15:25.463 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:15:25.464 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow profile bootstrap-osd" 2021-02-10T11:15:25.464 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:15:25.464 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:25.464 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:15:25.464 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.bootstrap-rbd", 2021-02-10T11:15:25.464 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQDWvyNgxQPnHRAAwWv3T/IxnXURBwx99v5jrA==", 2021-02-10T11:15:25.464 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:15:25.464 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow profile bootstrap-rbd" 2021-02-10T11:15:25.464 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:15:25.464 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:25.464 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:15:25.464 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.bootstrap-rbd-mirror", 2021-02-10T11:15:25.464 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQDWvyNg0wznHRAA4lCc5xZPaygvEOpR49n/CA==", 2021-02-10T11:15:25.464 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:15:25.465 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2021-02-10T11:15:25.465 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:15:25.465 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:25.465 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:15:25.465 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.bootstrap-rgw", 2021-02-10T11:15:25.465 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQDWvyNgwxXnHRAA2+HWeES7bO2syXSuRq7htg==", 2021-02-10T11:15:25.465 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:15:25.465 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow profile bootstrap-rgw" 2021-02-10T11:15:25.465 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:15:25.465 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:25.465 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:15:25.465 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "mgr.x", 2021-02-10T11:15:25.465 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCJvyNgTPVwKBAAjduiBE8+RSXBv82tdPecWg==", 2021-02-10T11:15:25.465 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:15:25.466 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow *", 2021-02-10T11:15:25.466 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow profile mgr", 2021-02-10T11:15:25.466 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:15:25.466 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:15:25.466 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:25.466 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:15:25.466 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "mgr.y", 2021-02-10T11:15:25.466 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCJvyNgQ1tTDhAASwBssDy+Rz31ujw8fNlWow==", 2021-02-10T11:15:25.466 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:15:25.466 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow *", 2021-02-10T11:15:25.466 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow profile mgr", 2021-02-10T11:15:25.466 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:15:25.466 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:15:25.467 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:15:25.467 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:15:25.467 INFO:teuthology.orchestra.run.gibba012.stdout:} 2021-02-10T11:15:25.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.453 7fc7aa6c5700 1 -- 172.21.2.112:0/1022864165 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc7840186c0 msgr2=0x7fc78401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:25.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.453 7fc7aa6c5700 1 --2- 172.21.2.112:0/1022864165 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc7840186c0 0x7fc78401abc0 crc :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:25.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.453 7fc7aa6c5700 1 -- 172.21.2.112:0/1022864165 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc7a40adb80 msgr2=0x7fc7a41a1b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:25.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.453 7fc7aa6c5700 1 --2- 172.21.2.112:0/1022864165 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc7a40adb80 0x7fc7a41a1b90 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7fc79800a570 tx=0x7fc79800a910).stop 2021-02-10T11:15:25.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.453 7fc7aa6c5700 1 -- 172.21.2.112:0/1022864165 shutdown_connections 2021-02-10T11:15:25.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.453 7fc7aa6c5700 1 --2- 172.21.2.112:0/1022864165 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc7840186c0 0x7fc78401abc0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:25.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.453 7fc7aa6c5700 1 --2- 172.21.2.112:0/1022864165 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc7a41a21b0 0x7fc7a41a25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:25.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.453 7fc7aa6c5700 1 --2- 172.21.2.112:0/1022864165 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc7a40adb80 0x7fc7a41a1b90 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:25.469 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.453 7fc7aa6c5700 1 --2- 172.21.2.112:0/1022864165 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc7a408d5a0 0x7fc7a40ad560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:25.469 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.453 7fc7aa6c5700 1 -- 172.21.2.112:0/1022864165 >> 172.21.2.112:0/1022864165 conn(0x7fc7a41ac620 msgr2=0x7fc7a40896e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:25.469 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.453 7fc7aa6c5700 1 -- 172.21.2.112:0/1022864165 shutdown_connections 2021-02-10T11:15:25.469 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.453 7fc7aa6c5700 1 -- 172.21.2.112:0/1022864165 wait complete. 2021-02-10T11:15:25.469 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth del client.bootstrap-mds 2021-02-10T11:15:25.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.881 7f17ed6b4700 1 Processor -- start 2021-02-10T11:15:25.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.881 7f17ed6b4700 1 -- start start 2021-02-10T11:15:25.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.882 7f17ed6b4700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f17e80e5470 0x7f17e807e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:25.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.882 7f17ed6b4700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f17e807f300 0x7f17e807f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:25.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.882 7f17ed6b4700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17e807ed80 con 0x7f17e80e5070 2021-02-10T11:15:25.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.882 7f17ed6b4700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17e81b4010 con 0x7f17e80e5470 2021-02-10T11:15:25.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.882 7f17ed6b4700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17e80d1790 con 0x7f17e807f300 2021-02-10T11:15:25.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.882 7f17e6abe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f17e80e5470 0x7f17e807e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:25.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.882 7f17e72bf700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f17e80e5070 0x7f17e81b3820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:42754/0 (socket says 172.21.2.112:42754) 2021-02-10T11:15:25.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.882 7f17e72bf700 1 -- 172.21.2.112:0/3803754116 learned_addr learned my addr 172.21.2.112:0/3803754116 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:15:25.897 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.882 7f17e7ac0700 1 --2- 172.21.2.112:0/3803754116 >> v2:172.21.2.120:3300/0 conn(0x7f17e807f300 0x7f17e807f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:25.897 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.882 7f17e7ac0700 1 -- 172.21.2.112:0/3803754116 >> v2:172.21.2.112:3300/0 conn(0x7f17e80e5470 msgr2=0x7f17e807e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:15:25.897 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.882 7f17e7ac0700 1 --2- 172.21.2.112:0/3803754116 >> v2:172.21.2.112:3300/0 conn(0x7f17e80e5470 0x7f17e807e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:25.897 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.882 7f17e7ac0700 1 -- 172.21.2.112:0/3803754116 >> v1:172.21.2.112:6789/0 conn(0x7f17e80e5070 legacy=0x7f17e81b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:25.897 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.882 7f17e7ac0700 1 -- 172.21.2.112:0/3803754116 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17e80d19b0 con 0x7f17e807f300 2021-02-10T11:15:25.897 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.883 7f17e7ac0700 1 --2- 172.21.2.112:0/3803754116 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f17e807f300 0x7f17e807f700 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f17e0006e50 tx=0x7f17e0009e70).ready entity=mon.1 client_cookie=f5604b3a34587c3e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:25.897 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.883 7f17e62bd700 1 -- 172.21.2.112:0/3803754116 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f17e0006a20 con 0x7f17e807f300 2021-02-10T11:15:25.897 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.883 7f17e62bd700 1 -- 172.21.2.112:0/3803754116 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17e0006790 con 0x7f17e807f300 2021-02-10T11:15:25.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.883 7f17e62bd700 1 -- 172.21.2.112:0/3803754116 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f17e0000f90 con 0x7f17e807f300 2021-02-10T11:15:25.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.883 7f17ed6b4700 1 -- 172.21.2.112:0/3803754116 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f17e807f300 msgr2=0x7f17e807f700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:25.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.883 7f17ed6b4700 1 --2- 172.21.2.112:0/3803754116 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f17e807f300 0x7f17e807f700 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f17e0006e50 tx=0x7f17e0009e70).stop 2021-02-10T11:15:25.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.883 7f17ed6b4700 1 -- 172.21.2.112:0/3803754116 shutdown_connections 2021-02-10T11:15:25.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.883 7f17ed6b4700 1 --2- 172.21.2.112:0/3803754116 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f17e807f300 0x7f17e807f700 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:25.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.883 7f17ed6b4700 1 --2- 172.21.2.112:0/3803754116 >> v2:172.21.2.112:3300/0 conn(0x7f17e80e5470 0x7f17e807e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:25.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.883 7f17ed6b4700 1 -- 172.21.2.112:0/3803754116 >> 172.21.2.112:0/3803754116 conn(0x7f17e80ce690 msgr2=0x7f17e80e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:25.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.883 7f17ed6b4700 1 -- 172.21.2.112:0/3803754116 shutdown_connections 2021-02-10T11:15:25.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.883 7f17ed6b4700 1 -- 172.21.2.112:0/3803754116 wait complete. 2021-02-10T11:15:25.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.898 7f17ed6b4700 1 Processor -- start 2021-02-10T11:15:25.900 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.898 7f17ed6b4700 1 -- start start 2021-02-10T11:15:25.900 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.898 7f17ed6b4700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f17e80b4ad0 0x7f17e80a78d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:25.900 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.898 7f17ed6b4700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f17e80a7ef0 0x7f17e81a6d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:25.900 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.898 7f17ed6b4700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f17e80a82f0 0x7f17e81a73b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:25.900 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.898 7f17ed6b4700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17e80aa3c0 con 0x7f17e80b4ad0 2021-02-10T11:15:25.900 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.898 7f17ed6b4700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17e80a88c0 con 0x7f17e80a82f0 2021-02-10T11:15:25.900 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.898 7f17ed6b4700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17e80b4ed0 con 0x7f17e80a7ef0 2021-02-10T11:15:25.901 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.899 7f17e72bf700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f17e80b4ad0 0x7f17e80a78d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:25.901 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.899 7f17e72bf700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f17e80b4ad0 0x7f17e80a78d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:43298/0 (socket says 172.21.2.112:43298) 2021-02-10T11:15:25.901 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.899 7f17e72bf700 1 -- 172.21.2.112:0/1682083175 learned_addr learned my addr 172.21.2.112:0/1682083175 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:15:25.901 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.899 7f17e7ac0700 1 --2- 172.21.2.112:0/1682083175 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f17e80a82f0 0x7f17e81a73b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:25.901 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.899 7f17e6abe700 1 --2- 172.21.2.112:0/1682083175 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f17e80a7ef0 0x7f17e81a6d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:25.901 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.899 7f17e72bf700 1 -- 172.21.2.112:0/1682083175 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f17e80a7ef0 msgr2=0x7f17e81a6d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:25.901 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.899 7f17e72bf700 1 --2- 172.21.2.112:0/1682083175 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f17e80a7ef0 0x7f17e81a6d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:25.901 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.899 7f17e72bf700 1 -- 172.21.2.112:0/1682083175 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f17e80a82f0 msgr2=0x7f17e81a73b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:25.901 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.899 7f17e72bf700 1 --2- 172.21.2.112:0/1682083175 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f17e80a82f0 0x7f17e81a73b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:25.901 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.899 7f17e72bf700 1 -- 172.21.2.112:0/1682083175 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17e81a79d0 con 0x7f17e80b4ad0 2021-02-10T11:15:25.901 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.899 7f17e72bf700 1 --2- 172.21.2.112:0/1682083175 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f17e80b4ad0 0x7f17e80a78d0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7f17d8001410 tx=0x7f17d800a950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:25.901 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.899 7f17dffff700 1 -- 172.21.2.112:0/1682083175 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f17d800ad40 con 0x7f17e80b4ad0 2021-02-10T11:15:25.901 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.900 7f17dffff700 1 -- 172.21.2.112:0/1682083175 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17d80009b0 con 0x7f17e80b4ad0 2021-02-10T11:15:25.901 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.900 7f17dffff700 1 -- 172.21.2.112:0/1682083175 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f17d8004620 con 0x7f17e80b4ad0 2021-02-10T11:15:25.902 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.900 7f17ed6b4700 1 -- 172.21.2.112:0/1682083175 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17e81a79d0 con 0x7f17e80b4ad0 2021-02-10T11:15:25.902 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.900 7f17ed6b4700 1 -- 172.21.2.112:0/1682083175 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17e80a9130 con 0x7f17e80b4ad0 2021-02-10T11:15:25.902 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.900 7f17dffff700 1 -- 172.21.2.112:0/1682083175 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f17d80043a0 con 0x7f17e80b4ad0 2021-02-10T11:15:25.902 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.900 7f17dffff700 1 --2- 172.21.2.112:0/1682083175 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f17cc0186c0 0x7f17cc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:25.902 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.900 7f17dffff700 1 -- 172.21.2.112:0/1682083175 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f17d80320c0 con 0x7f17e80b4ad0 2021-02-10T11:15:25.902 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.901 7f17e6abe700 1 --2- 172.21.2.112:0/1682083175 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f17cc0186c0 0x7f17cc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:25.902 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.901 7f17e6abe700 1 --2- 172.21.2.112:0/1682083175 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f17cc0186c0 0x7f17cc01abc0 crc :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:25.930 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.932 7f17ed6b4700 1 -- 172.21.2.112:0/1682083175 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17e8084a90 con 0x7f17e80b4ad0 2021-02-10T11:15:25.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:25.934 7f17dffff700 1 -- 172.21.2.112:0/1682083175 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f17d8030d20 con 0x7f17e80b4ad0 2021-02-10T11:15:26.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:26.063 7f17ed6b4700 1 -- 172.21.2.112:0/1682083175 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) v1 -- 0x7f17e807ed80 con 0x7f17e80b4ad0 2021-02-10T11:15:26.063 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:26.066 7f17dffff700 1 -- 172.21.2.112:0/1682083175 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 updated v7) v1 ==== 97+0+0 (secure 0 0 0) 0x7f17d801f710 con 0x7f17e80b4ad0 2021-02-10T11:15:26.064 INFO:teuthology.orchestra.run.gibba012.stderr:updated 2021-02-10T11:15:26.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:26.067 7f17ed6b4700 1 -- 172.21.2.112:0/1682083175 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f17cc0186c0 msgr2=0x7f17cc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:26.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:26.067 7f17ed6b4700 1 --2- 172.21.2.112:0/1682083175 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f17cc0186c0 0x7f17cc01abc0 crc :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:26.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:26.067 7f17ed6b4700 1 -- 172.21.2.112:0/1682083175 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f17e80b4ad0 msgr2=0x7f17e80a78d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:26.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:26.067 7f17ed6b4700 1 --2- 172.21.2.112:0/1682083175 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f17e80b4ad0 0x7f17e80a78d0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7f17d8001410 tx=0x7f17d800a950).stop 2021-02-10T11:15:26.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:26.067 7f17ed6b4700 1 -- 172.21.2.112:0/1682083175 shutdown_connections 2021-02-10T11:15:26.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:26.067 7f17ed6b4700 1 --2- 172.21.2.112:0/1682083175 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f17cc0186c0 0x7f17cc01abc0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:26.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:26.067 7f17ed6b4700 1 --2- 172.21.2.112:0/1682083175 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f17e80a82f0 0x7f17e81a73b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:26.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:26.067 7f17ed6b4700 1 --2- 172.21.2.112:0/1682083175 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f17e80a7ef0 0x7f17e81a6d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:26.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:26.067 7f17ed6b4700 1 --2- 172.21.2.112:0/1682083175 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f17e80b4ad0 0x7f17e80a78d0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:26.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:26.067 7f17ed6b4700 1 -- 172.21.2.112:0/1682083175 >> 172.21.2.112:0/1682083175 conn(0x7f17e80b5560 msgr2=0x7f17e80a6200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:26.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:26.068 7f17ed6b4700 1 -- 172.21.2.112:0/1682083175 shutdown_connections 2021-02-10T11:15:26.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:26.068 7f17ed6b4700 1 -- 172.21.2.112:0/1682083175 wait complete. 2021-02-10T11:15:26.074 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth del client.bootstrap-mgr 2021-02-10T11:15:26.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:26.484 7f21f4856700 1 Processor -- start 2021-02-10T11:15:26.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:26.484 7f21f4856700 1 -- start start 2021-02-10T11:15:26.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:26.484 7f21f4856700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f21f00e5070 0x7f21f01b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:26.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:26.484 7f21f4856700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f21f00e5470 0x7f21f0080310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:26.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:26.484 7f21f4856700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21f007ed80 con 0x7f21f01b3e40 2021-02-10T11:15:26.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:26.484 7f21f4856700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21f00d1790 con 0x7f21f00e5070 2021-02-10T11:15:26.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:26.484 7f21f4856700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21f00d19b0 con 0x7f21f00e5470 2021-02-10T11:15:26.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:26.485 7f21ee61c700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f21f00e5070 0x7f21f01b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:26.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:26.485 7f21eee1d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f21f00e5470 0x7f21f0080310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:26.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:26.485 7f21eee1d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f21f00e5470 0x7f21f0080310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3301/0 says I am v2:172.21.2.112:42988/0 (socket says 172.21.2.112:42988) 2021-02-10T11:15:26.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:26.485 7f21ee61c700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f21f00e5070 0x7f21f01b3820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3300/0 says I am v2:172.21.2.112:45586/0 (socket says 172.21.2.112:45586) 2021-02-10T11:15:26.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:26.485 7f21eee1d700 1 -- 172.21.2.112:0/2278236083 learned_addr learned my addr 172.21.2.112:0/2278236083 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:15:26.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:26.486 7f21ee61c700 1 -- 172.21.2.112:0/2278236083 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f21f00e5470 msgr2=0x7f21f0080310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:15:26.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:26.486 7f21ee61c700 1 --2- 172.21.2.112:0/2278236083 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f21f00e5470 0x7f21f0080310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:26.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:26.486 7f21ee61c700 1 -- 172.21.2.112:0/2278236083 >> v1:172.21.2.120:6789/0 conn(0x7f21f01b3e40 legacy=0x7f21f007d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:26.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:26.486 7f21ee61c700 1 -- 172.21.2.112:0/2278236083 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21f00d1bd0 con 0x7f21f00e5070 2021-02-10T11:15:26.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:26.486 7f21ee61c700 1 --2- 172.21.2.112:0/2278236083 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f21f00e5070 0x7f21f01b3820 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f21d8006650 tx=0x7f21d8009a80).ready entity=mon.1 client_cookie=2de13bc784b6e95 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:26.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:26.486 7f21ed61a700 1 -- 172.21.2.112:0/2278236083 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f21d800c0c0 con 0x7f21f00e5070 2021-02-10T11:15:26.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:26.486 7f21ed61a700 1 -- 172.21.2.112:0/2278236083 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21d800c340 con 0x7f21f00e5070 2021-02-10T11:15:26.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:26.486 7f21ed61a700 1 -- 172.21.2.112:0/2278236083 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f21d800c860 con 0x7f21f00e5070 2021-02-10T11:15:26.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:26.486 7f21f4856700 1 -- 172.21.2.112:0/2278236083 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f21f00e5070 msgr2=0x7f21f01b3820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:26.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:26.486 7f21f4856700 1 --2- 172.21.2.112:0/2278236083 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f21f00e5070 0x7f21f01b3820 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f21d8006650 tx=0x7f21d8009a80).stop 2021-02-10T11:15:26.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:26.487 7f21f4856700 1 -- 172.21.2.112:0/2278236083 shutdown_connections 2021-02-10T11:15:26.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:26.487 7f21f4856700 1 --2- 172.21.2.112:0/2278236083 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f21f00e5470 0x7f21f0080310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:26.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:26.487 7f21f4856700 1 --2- 172.21.2.112:0/2278236083 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f21f00e5070 0x7f21f01b3820 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:26.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:26.487 7f21f4856700 1 -- 172.21.2.112:0/2278236083 >> 172.21.2.112:0/2278236083 conn(0x7f21f00ce690 msgr2=0x7f21f00e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:26.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:26.487 7f21f4856700 1 -- 172.21.2.112:0/2278236083 shutdown_connections 2021-02-10T11:15:26.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:26.487 7f21f4856700 1 -- 172.21.2.112:0/2278236083 wait complete. 2021-02-10T11:15:26.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:26.501 7f21f4856700 1 Processor -- start 2021-02-10T11:15:26.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:26.502 7f21f4856700 1 -- start start 2021-02-10T11:15:26.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:26.502 7f21f4856700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f21f00bd700 0x7f21f00bdb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:26.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:26.502 7f21f4856700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f21f00a4b10 0x7f21f00a4f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:26.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:26.502 7f21f4856700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f21f00a5530 0x7f21f00a5930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:26.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:26.502 7f21f4856700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21f00be230 con 0x7f21f00a4b10 2021-02-10T11:15:26.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:26.502 7f21f4856700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21f00be450 con 0x7f21f00a5530 2021-02-10T11:15:26.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:26.502 7f21f4856700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21f00be670 con 0x7f21f00bd700 2021-02-10T11:15:26.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:26.502 7f21ede1b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f21f00a4b10 0x7f21f00a4f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:26.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:26.502 7f21ede1b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f21f00a4b10 0x7f21f00a4f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:43308/0 (socket says 172.21.2.112:43308) 2021-02-10T11:15:26.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:26.502 7f21ede1b700 1 -- 172.21.2.112:0/568545212 learned_addr learned my addr 172.21.2.112:0/568545212 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:15:26.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:26.502 7f21ede1b700 1 -- 172.21.2.112:0/568545212 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f21f00bd700 msgr2=0x7f21f00bdb00 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:15:26.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:26.502 7f21ede1b700 1 --2- 172.21.2.112:0/568545212 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f21f00bd700 0x7f21f00bdb00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:26.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:26.502 7f21ede1b700 1 -- 172.21.2.112:0/568545212 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f21f00a5530 msgr2=0x7f21f00a5930 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:15:26.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:26.502 7f21ede1b700 1 --2- 172.21.2.112:0/568545212 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f21f00a5530 0x7f21f00a5930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:26.505 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:26.502 7f21ede1b700 1 -- 172.21.2.112:0/568545212 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21f00be890 con 0x7f21f00a4b10 2021-02-10T11:15:26.505 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:26.502 7f21ede1b700 1 --2- 172.21.2.112:0/568545212 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f21f00a4b10 0x7f21f00a4f10 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7f21e0006ed0 tx=0x7f21e000b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:26.505 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:26.503 7f21e6ffd700 1 -- 172.21.2.112:0/568545212 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f21e0006240 con 0x7f21f00a4b10 2021-02-10T11:15:26.505 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:26.503 7f21e6ffd700 1 -- 172.21.2.112:0/568545212 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e00054b0 con 0x7f21f00a4b10 2021-02-10T11:15:26.505 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:26.503 7f21f4856700 1 -- 172.21.2.112:0/568545212 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21f00be890 con 0x7f21f00a4b10 2021-02-10T11:15:26.505 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:26.503 7f21f4856700 1 -- 172.21.2.112:0/568545212 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21f00be890 con 0x7f21f00a4b10 2021-02-10T11:15:26.505 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:26.503 7f21e6ffd700 1 -- 172.21.2.112:0/568545212 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f21e0005900 con 0x7f21f00a4b10 2021-02-10T11:15:26.505 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:26.503 7f21e6ffd700 1 -- 172.21.2.112:0/568545212 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f21e0006030 con 0x7f21f00a4b10 2021-02-10T11:15:26.505 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:26.503 7f21e6ffd700 1 --2- 172.21.2.112:0/568545212 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f21d4018730 0x7f21d401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:26.505 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:26.503 7f21e6ffd700 1 -- 172.21.2.112:0/568545212 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f21e00324c0 con 0x7f21f00a4b10 2021-02-10T11:15:26.505 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:26.504 7f21ee61c700 1 --2- 172.21.2.112:0/568545212 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f21d4018730 0x7f21d401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:26.505 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:26.504 7f21ee61c700 1 --2- 172.21.2.112:0/568545212 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f21d4018730 0x7f21d401ac10 crc :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:26.532 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:26.534 7f21f4856700 1 -- 172.21.2.112:0/568545212 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21f007ed80 con 0x7f21f00a4b10 2021-02-10T11:15:26.535 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:26.537 7f21e6ffd700 1 -- 172.21.2.112:0/568545212 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f21e0000de0 con 0x7f21f00a4b10 2021-02-10T11:15:26.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:26.666 7f21f4856700 1 -- 172.21.2.112:0/568545212 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) v1 -- 0x7f21f007ed80 con 0x7f21f00a4b10 2021-02-10T11:15:26.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:26.670 7f21e6ffd700 1 -- 172.21.2.112:0/568545212 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 updated v8) v1 ==== 97+0+0 (secure 0 0 0) 0x7f21e0006030 con 0x7f21f00a4b10 2021-02-10T11:15:26.672 INFO:teuthology.orchestra.run.gibba012.stderr:updated 2021-02-10T11:15:26.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:26.675 7f21f4856700 1 -- 172.21.2.112:0/568545212 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f21d4018730 msgr2=0x7f21d401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:26.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:26.675 7f21f4856700 1 --2- 172.21.2.112:0/568545212 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f21d4018730 0x7f21d401ac10 crc :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:26.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:26.675 7f21f4856700 1 -- 172.21.2.112:0/568545212 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f21f00a4b10 msgr2=0x7f21f00a4f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:26.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:26.675 7f21f4856700 1 --2- 172.21.2.112:0/568545212 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f21f00a4b10 0x7f21f00a4f10 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7f21e0006ed0 tx=0x7f21e000b960).stop 2021-02-10T11:15:26.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:26.675 7f21f4856700 1 -- 172.21.2.112:0/568545212 shutdown_connections 2021-02-10T11:15:26.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:26.676 7f21f4856700 1 --2- 172.21.2.112:0/568545212 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f21d4018730 0x7f21d401ac10 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:26.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:26.676 7f21f4856700 1 --2- 172.21.2.112:0/568545212 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f21f00a5530 0x7f21f00a5930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:26.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:26.676 7f21f4856700 1 --2- 172.21.2.112:0/568545212 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f21f00a4b10 0x7f21f00a4f10 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:26.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:26.676 7f21f4856700 1 --2- 172.21.2.112:0/568545212 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f21f00bd700 0x7f21f00bdb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:26.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:26.676 7f21f4856700 1 -- 172.21.2.112:0/568545212 >> 172.21.2.112:0/568545212 conn(0x7f21f00c0cf0 msgr2=0x7f21f00a70a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:26.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:26.676 7f21f4856700 1 -- 172.21.2.112:0/568545212 shutdown_connections 2021-02-10T11:15:26.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:26.676 7f21f4856700 1 -- 172.21.2.112:0/568545212 wait complete. 2021-02-10T11:15:26.682 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth del client.bootstrap-osd 2021-02-10T11:15:27.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.092 7f1959ff6700 1 Processor -- start 2021-02-10T11:15:27.159 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.092 7f1959ff6700 1 -- start start 2021-02-10T11:15:27.160 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.092 7f1959ff6700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f195407e6a0 0x7f19540d1790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:27.160 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.092 7f1959ff6700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f195407ed80 con 0x7f19540e5070 2021-02-10T11:15:27.160 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.092 7f1959ff6700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f195407eb10 con 0x7f19540e5470 2021-02-10T11:15:27.160 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.092 7f1959ff6700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19541b4080 con 0x7f195407e6a0 2021-02-10T11:15:27.160 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.092 7f19537fe700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f19540e5070 0x7f19541b3820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:42778/0 (socket says 172.21.2.112:42778) 2021-02-10T11:15:27.160 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.092 7f19537fe700 1 -- 172.21.2.112:0/296463980 learned_addr learned my addr 172.21.2.112:0/296463980 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:15:27.160 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.092 7f1952ffd700 1 --1- 172.21.2.112:0/296463980 >> v1:172.21.2.120:6789/0 conn(0x7f19540e5470 0x7f195407f300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:36336/0 (socket says 172.21.2.112:36336) 2021-02-10T11:15:27.161 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.092 7f1953fff700 1 --2- 172.21.2.112:0/296463980 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f195407e6a0 0x7f19540d1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:27.161 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.093 7f19527fc700 1 -- 172.21.2.112:0/296463980 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2103661946 0 0) 0x7f1944001180 con 0x7f19540e5070 2021-02-10T11:15:27.161 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.093 7f1953fff700 1 -- 172.21.2.112:0/296463980 >> v1:172.21.2.120:6789/0 conn(0x7f19540e5470 legacy=0x7f195407f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:27.161 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.093 7f1953fff700 1 -- 172.21.2.112:0/296463980 >> v1:172.21.2.112:6789/0 conn(0x7f19540e5070 legacy=0x7f19541b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:27.161 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.093 7f1953fff700 1 -- 172.21.2.112:0/296463980 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19540d1db0 con 0x7f195407e6a0 2021-02-10T11:15:27.161 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.093 7f1953fff700 1 --2- 172.21.2.112:0/296463980 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f195407e6a0 0x7f19540d1790 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f194c006e50 tx=0x7f194c009e70).ready entity=mon.2 client_cookie=1babe54f76085584 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:27.161 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.093 7f19527fc700 1 -- 172.21.2.112:0/296463980 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f194c00b510 con 0x7f195407e6a0 2021-02-10T11:15:27.161 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.093 7f19527fc700 1 -- 172.21.2.112:0/296463980 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f194c006790 con 0x7f195407e6a0 2021-02-10T11:15:27.161 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.093 7f19527fc700 1 -- 172.21.2.112:0/296463980 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f194c00b940 con 0x7f195407e6a0 2021-02-10T11:15:27.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.093 7f1959ff6700 1 -- 172.21.2.112:0/296463980 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f195407e6a0 msgr2=0x7f19540d1790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:27.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.093 7f1959ff6700 1 --2- 172.21.2.112:0/296463980 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f195407e6a0 0x7f19540d1790 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f194c006e50 tx=0x7f194c009e70).stop 2021-02-10T11:15:27.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.094 7f1959ff6700 1 -- 172.21.2.112:0/296463980 shutdown_connections 2021-02-10T11:15:27.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.094 7f1959ff6700 1 --2- 172.21.2.112:0/296463980 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f195407e6a0 0x7f19540d1790 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:27.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.094 7f1959ff6700 1 -- 172.21.2.112:0/296463980 >> 172.21.2.112:0/296463980 conn(0x7f19540ce690 msgr2=0x7f19540e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:27.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.094 7f1959ff6700 1 -- 172.21.2.112:0/296463980 shutdown_connections 2021-02-10T11:15:27.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.094 7f1959ff6700 1 -- 172.21.2.112:0/296463980 wait complete. 2021-02-10T11:15:27.164 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.109 7f1959ff6700 1 Processor -- start 2021-02-10T11:15:27.164 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.109 7f1959ff6700 1 -- start start 2021-02-10T11:15:27.164 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.109 7f1959ff6700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f19541c24c0 0x7f19540c37a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:27.165 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.109 7f1959ff6700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f19540c3dc0 0x7f19540a1ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:27.165 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.109 7f1959ff6700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f19540a2510 0x7f19540a0e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:27.165 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.109 7f1959ff6700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19540b9ef0 con 0x7f19540a2510 2021-02-10T11:15:27.165 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.109 7f1959ff6700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19540a2910 con 0x7f19540c3dc0 2021-02-10T11:15:27.165 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.109 7f1959ff6700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19540a2af0 con 0x7f19541c24c0 2021-02-10T11:15:27.166 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.110 7f1953fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f19540a2510 0x7f19540a0e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:27.166 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.110 7f1952ffd700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f19540c3dc0 0x7f19540a1ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:27.166 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.110 7f19537fe700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f19541c24c0 0x7f19540c37a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:27.166 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.110 7f1953fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f19540a2510 0x7f19540a0e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:43324/0 (socket says 172.21.2.112:43324) 2021-02-10T11:15:27.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.110 7f1953fff700 1 -- 172.21.2.112:0/3331352973 learned_addr learned my addr 172.21.2.112:0/3331352973 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:15:27.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.110 7f1953fff700 1 -- 172.21.2.112:0/3331352973 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f19541c24c0 msgr2=0x7f19540c37a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:27.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.110 7f1953fff700 1 --2- 172.21.2.112:0/3331352973 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f19541c24c0 0x7f19540c37a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:27.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.110 7f1953fff700 1 -- 172.21.2.112:0/3331352973 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f19540c3dc0 msgr2=0x7f19540a1ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:27.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.110 7f1953fff700 1 --2- 172.21.2.112:0/3331352973 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f19540c3dc0 0x7f19540a1ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:27.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.110 7f1953fff700 1 -- 172.21.2.112:0/3331352973 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19540a14b0 con 0x7f19540a2510 2021-02-10T11:15:27.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.110 7f1953fff700 1 --2- 172.21.2.112:0/3331352973 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f19540a2510 0x7f19540a0e90 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7f194c0099f0 tx=0x7f194c00bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:27.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.110 7f193bfff700 1 -- 172.21.2.112:0/3331352973 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f194c0093d0 con 0x7f19540a2510 2021-02-10T11:15:27.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.110 7f193bfff700 1 -- 172.21.2.112:0/3331352973 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f194c003030 con 0x7f19540a2510 2021-02-10T11:15:27.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.110 7f193bfff700 1 -- 172.21.2.112:0/3331352973 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f194c0013b0 con 0x7f19540a2510 2021-02-10T11:15:27.169 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.111 7f1959ff6700 1 -- 172.21.2.112:0/3331352973 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19540a14b0 con 0x7f19540a2510 2021-02-10T11:15:27.169 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.111 7f1959ff6700 1 -- 172.21.2.112:0/3331352973 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19540a14b0 con 0x7f19540a2510 2021-02-10T11:15:27.169 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.111 7f193bfff700 1 -- 172.21.2.112:0/3331352973 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f194c0093d0 con 0x7f19540a2510 2021-02-10T11:15:27.169 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.111 7f193bfff700 1 --2- 172.21.2.112:0/3331352973 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f19340186c0 0x7f193401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:27.170 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.111 7f193bfff700 1 -- 172.21.2.112:0/3331352973 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f194c032470 con 0x7f19540a2510 2021-02-10T11:15:27.170 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.111 7f19537fe700 1 --2- 172.21.2.112:0/3331352973 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f19340186c0 0x7f193401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:27.170 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.112 7f19537fe700 1 --2- 172.21.2.112:0/3331352973 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f19340186c0 0x7f193401abc0 crc :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:27.170 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.142 7f1959ff6700 1 -- 172.21.2.112:0/3331352973 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19540898d0 con 0x7f19540a2510 2021-02-10T11:15:27.170 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.145 7f193bfff700 1 -- 172.21.2.112:0/3331352973 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f194c009000 con 0x7f19540a2510 2021-02-10T11:15:27.274 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.276 7f1959ff6700 1 -- 172.21.2.112:0/3331352973 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) v1 -- 0x7f19540ce690 con 0x7f19540a2510 2021-02-10T11:15:27.279 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.280 7f193bfff700 1 -- 172.21.2.112:0/3331352973 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 updated v9) v1 ==== 97+0+0 (secure 0 0 0) 0x7f194c0093d0 con 0x7f19540a2510 2021-02-10T11:15:27.281 INFO:teuthology.orchestra.run.gibba012.stderr:updated 2021-02-10T11:15:27.283 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.285 7f1959ff6700 1 -- 172.21.2.112:0/3331352973 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f19340186c0 msgr2=0x7f193401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:27.283 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.285 7f1959ff6700 1 --2- 172.21.2.112:0/3331352973 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f19340186c0 0x7f193401abc0 crc :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:27.283 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.285 7f1959ff6700 1 -- 172.21.2.112:0/3331352973 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f19540a2510 msgr2=0x7f19540a0e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:27.283 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.285 7f1959ff6700 1 --2- 172.21.2.112:0/3331352973 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f19540a2510 0x7f19540a0e90 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7f194c0099f0 tx=0x7f194c00bd10).stop 2021-02-10T11:15:27.284 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.285 7f1959ff6700 1 -- 172.21.2.112:0/3331352973 shutdown_connections 2021-02-10T11:15:27.284 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.285 7f1959ff6700 1 --2- 172.21.2.112:0/3331352973 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f19340186c0 0x7f193401abc0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:27.284 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.285 7f1959ff6700 1 --2- 172.21.2.112:0/3331352973 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f19540c3dc0 0x7f19540a1ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:27.284 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.285 7f1959ff6700 1 --2- 172.21.2.112:0/3331352973 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f19540a2510 0x7f19540a0e90 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:27.284 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.285 7f1959ff6700 1 --2- 172.21.2.112:0/3331352973 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f19541c24c0 0x7f19540c37a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:27.284 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.285 7f1959ff6700 1 -- 172.21.2.112:0/3331352973 >> 172.21.2.112:0/3331352973 conn(0x7f19540bc2f0 msgr2=0x7f19540afea0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:27.284 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.286 7f1959ff6700 1 -- 172.21.2.112:0/3331352973 shutdown_connections 2021-02-10T11:15:27.284 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.286 7f1959ff6700 1 -- 172.21.2.112:0/3331352973 wait complete. 2021-02-10T11:15:27.292 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth del client.bootstrap-rbd 2021-02-10T11:15:27.717 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.702 7ffb558ec700 1 Processor -- start 2021-02-10T11:15:27.717 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.702 7ffb558ec700 1 -- start start 2021-02-10T11:15:27.718 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.702 7ffb558ec700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7ffb500e5070 0x7ffb501b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:27.718 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.702 7ffb558ec700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb5007ed80 con 0x7ffb501b3e40 2021-02-10T11:15:27.718 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.702 7ffb558ec700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb5007e670 con 0x7ffb500e5470 2021-02-10T11:15:27.718 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.702 7ffb558ec700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb5007e890 con 0x7ffb500e5070 2021-02-10T11:15:27.718 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.703 7ffb4ed73700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7ffb501b3e40 0x7ffb5007d690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:42790/0 (socket says 172.21.2.112:42790) 2021-02-10T11:15:27.718 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.703 7ffb4ed73700 1 -- 172.21.2.112:0/3258986419 learned_addr learned my addr 172.21.2.112:0/3258986419 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:15:27.718 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.703 7ffb4e572700 1 -- 172.21.2.112:0/3258986419 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3605867640 0 0) 0x7ffb400010b0 con 0x7ffb501b3e40 2021-02-10T11:15:27.718 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.703 7ffb4e572700 1 -- 172.21.2.112:0/3258986419 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb3c002fe0 con 0x7ffb501b3e40 2021-02-10T11:15:27.718 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.703 7ffb4f574700 1 --2- 172.21.2.112:0/3258986419 >> v2:172.21.2.120:3300/0 conn(0x7ffb500e5070 0x7ffb501b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:27.718 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.703 7ffb4e572700 1 -- 172.21.2.112:0/3258986419 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1604088423 0 0) 0x7ffb40001010 con 0x7ffb501b3e40 2021-02-10T11:15:27.718 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.703 7ffb4e572700 1 -- 172.21.2.112:0/3258986419 >> v2:172.21.2.120:3300/0 conn(0x7ffb500e5070 msgr2=0x7ffb501b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:15:27.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.703 7ffb4e572700 1 --2- 172.21.2.112:0/3258986419 >> v2:172.21.2.120:3300/0 conn(0x7ffb500e5070 0x7ffb501b3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:27.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.703 7ffb4e572700 1 -- 172.21.2.112:0/3258986419 >> v1:172.21.2.120:6789/0 conn(0x7ffb500e5470 legacy=0x7ffb5007de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:15:27.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.703 7ffb4e572700 1 -- 172.21.2.112:0/3258986419 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb5007eab0 con 0x7ffb501b3e40 2021-02-10T11:15:27.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.703 7ffb4e572700 1 -- 172.21.2.112:0/3258986419 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7ffb40001530 con 0x7ffb501b3e40 2021-02-10T11:15:27.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.703 7ffb4e572700 1 -- 172.21.2.112:0/3258986419 >> v1:172.21.2.112:6789/0 conn(0x7ffb501b3e40 legacy=0x7ffb5007d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:27.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.703 7ffb4e572700 1 --2- 172.21.2.112:0/3258986419 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ffb3c003a70 0x7ffb3c005f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:27.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.704 7ffb4e572700 1 --2- 172.21.2.112:0/3258986419 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ffb3c006780 0x7ffb3c008ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:27.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.704 7ffb4e572700 1 --2- 172.21.2.112:0/3258986419 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ffb3c0095a0 0x7ffb3c00bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:27.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.704 7ffb4e572700 1 -- 172.21.2.112:0/3258986419 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb3c00c310 con 0x7ffb3c003a70 2021-02-10T11:15:27.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.704 7ffb4e572700 1 -- 172.21.2.112:0/3258986419 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb3c00c530 con 0x7ffb3c0095a0 2021-02-10T11:15:27.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.704 7ffb4e572700 1 -- 172.21.2.112:0/3258986419 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb3c00c750 con 0x7ffb3c006780 2021-02-10T11:15:27.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.704 7ffb4f574700 1 --2- 172.21.2.112:0/3258986419 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ffb3c003a70 0x7ffb3c005f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:27.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.704 7ffb4ed73700 1 --2- 172.21.2.112:0/3258986419 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ffb3c006780 0x7ffb3c008ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:27.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.704 7ffb4fd75700 1 --2- 172.21.2.112:0/3258986419 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ffb3c0095a0 0x7ffb3c00bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:27.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.704 7ffb4f574700 1 -- 172.21.2.112:0/3258986419 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ffb3c006780 msgr2=0x7ffb3c008ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:15:27.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.704 7ffb4f574700 1 --2- 172.21.2.112:0/3258986419 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ffb3c006780 0x7ffb3c008ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:27.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.704 7ffb4f574700 1 -- 172.21.2.112:0/3258986419 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ffb3c0095a0 msgr2=0x7ffb3c00bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:15:27.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.704 7ffb4f574700 1 --2- 172.21.2.112:0/3258986419 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ffb3c0095a0 0x7ffb3c00bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:27.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.704 7ffb4fd75700 1 -- 172.21.2.112:0/3258986419 reap_dead start 2021-02-10T11:15:27.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.704 7ffb4f574700 1 -- 172.21.2.112:0/3258986419 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb3c0509c0 con 0x7ffb3c003a70 2021-02-10T11:15:27.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.704 7ffb4f574700 1 --2- 172.21.2.112:0/3258986419 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ffb3c003a70 0x7ffb3c005f00 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7ffb3800c470 tx=0x7ffb38006c20).ready entity=mon.0 client_cookie=5fe319daede1f342 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:27.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.704 7ffb4e572700 1 -- 172.21.2.112:0/3258986419 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffb380090d0 con 0x7ffb3c003a70 2021-02-10T11:15:27.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.704 7ffb4e572700 1 -- 172.21.2.112:0/3258986419 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb38009450 con 0x7ffb3c003a70 2021-02-10T11:15:27.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.704 7ffb4e572700 1 -- 172.21.2.112:0/3258986419 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffb38009930 con 0x7ffb3c003a70 2021-02-10T11:15:27.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.704 7ffb558ec700 1 -- 172.21.2.112:0/3258986419 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ffb3c003a70 msgr2=0x7ffb3c005f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:27.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.704 7ffb558ec700 1 --2- 172.21.2.112:0/3258986419 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ffb3c003a70 0x7ffb3c005f00 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7ffb3800c470 tx=0x7ffb38006c20).stop 2021-02-10T11:15:27.721 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.705 7ffb558ec700 1 -- 172.21.2.112:0/3258986419 shutdown_connections 2021-02-10T11:15:27.721 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.705 7ffb558ec700 1 --2- 172.21.2.112:0/3258986419 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ffb3c003a70 0x7ffb3c005f00 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:27.721 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.705 7ffb558ec700 1 -- 172.21.2.112:0/3258986419 >> 172.21.2.112:0/3258986419 conn(0x7ffb500ce690 msgr2=0x7ffb500e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:27.721 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.705 7ffb558ec700 1 -- 172.21.2.112:0/3258986419 shutdown_connections 2021-02-10T11:15:27.721 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.705 7ffb558ec700 1 -- 172.21.2.112:0/3258986419 wait complete. 2021-02-10T11:15:27.721 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.719 7ffb558ec700 1 Processor -- start 2021-02-10T11:15:27.721 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.719 7ffb558ec700 1 -- start start 2021-02-10T11:15:27.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.720 7ffb558ec700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ffb500b05e0 0x7ffb501a3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:27.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.720 7ffb558ec700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ffb500b09e0 0x7ffb500bcc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:27.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.720 7ffb558ec700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ffb501a42d0 0x7ffb500bd260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:27.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.720 7ffb558ec700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb500b0f00 con 0x7ffb500b05e0 2021-02-10T11:15:27.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.720 7ffb558ec700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb5008a430 con 0x7ffb500b09e0 2021-02-10T11:15:27.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.720 7ffb558ec700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb5008a650 con 0x7ffb501a42d0 2021-02-10T11:15:27.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.720 7ffb4f574700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ffb500b05e0 0x7ffb501a3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:27.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.720 7ffb4f574700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ffb500b05e0 0x7ffb501a3cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:43340/0 (socket says 172.21.2.112:43340) 2021-02-10T11:15:27.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.720 7ffb4f574700 1 -- 172.21.2.112:0/3439580186 learned_addr learned my addr 172.21.2.112:0/3439580186 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:15:27.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.720 7ffb4ed73700 1 --2- 172.21.2.112:0/3439580186 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ffb500b09e0 0x7ffb500bcc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:27.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.721 7ffb4fd75700 1 --2- 172.21.2.112:0/3439580186 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ffb501a42d0 0x7ffb500bd260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:27.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.721 7ffb4f574700 1 -- 172.21.2.112:0/3439580186 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ffb501a42d0 msgr2=0x7ffb500bd260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:27.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.721 7ffb4f574700 1 --2- 172.21.2.112:0/3439580186 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ffb501a42d0 0x7ffb500bd260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:27.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.721 7ffb4f574700 1 -- 172.21.2.112:0/3439580186 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ffb500b09e0 msgr2=0x7ffb500bcc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:27.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.721 7ffb4f574700 1 --2- 172.21.2.112:0/3439580186 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ffb500b09e0 0x7ffb500bcc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:27.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.721 7ffb4f574700 1 -- 172.21.2.112:0/3439580186 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb5008a870 con 0x7ffb500b05e0 2021-02-10T11:15:27.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.721 7ffb4f574700 1 --2- 172.21.2.112:0/3439580186 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ffb500b05e0 0x7ffb501a3cb0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7ffb38006fc0 tx=0x7ffb38009f70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:27.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.721 7ffb47fff700 1 -- 172.21.2.112:0/3439580186 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffb38006030 con 0x7ffb500b05e0 2021-02-10T11:15:27.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.721 7ffb47fff700 1 -- 172.21.2.112:0/3439580186 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb3800c770 con 0x7ffb500b05e0 2021-02-10T11:15:27.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.721 7ffb558ec700 1 -- 172.21.2.112:0/3439580186 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb5008a870 con 0x7ffb500b05e0 2021-02-10T11:15:27.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.721 7ffb47fff700 1 -- 172.21.2.112:0/3439580186 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffb3800cc90 con 0x7ffb500b05e0 2021-02-10T11:15:27.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.721 7ffb558ec700 1 -- 172.21.2.112:0/3439580186 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb5008ae00 con 0x7ffb500b05e0 2021-02-10T11:15:27.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.722 7ffb47fff700 1 -- 172.21.2.112:0/3439580186 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ffb38006030 con 0x7ffb500b05e0 2021-02-10T11:15:27.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.722 7ffb47fff700 1 --2- 172.21.2.112:0/3439580186 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ffb340186c0 0x7ffb3401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:27.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.722 7ffb47fff700 1 -- 172.21.2.112:0/3439580186 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 4753+0+0 (secure 0 0 0) 0x7ffb38032200 con 0x7ffb500b05e0 2021-02-10T11:15:27.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.723 7ffb4ed73700 1 --2- 172.21.2.112:0/3439580186 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ffb340186c0 0x7ffb3401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:27.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.723 7ffb4ed73700 1 --2- 172.21.2.112:0/3439580186 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ffb340186c0 0x7ffb3401abc0 crc :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:27.751 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.753 7ffb558ec700 1 -- 172.21.2.112:0/3439580186 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb50084ad0 con 0x7ffb500b05e0 2021-02-10T11:15:27.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.755 7ffb47fff700 1 -- 172.21.2.112:0/3439580186 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ffb38006740 con 0x7ffb500b05e0 2021-02-10T11:15:27.884 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.886 7ffb558ec700 1 -- 172.21.2.112:0/3439580186 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) v1 -- 0x7ffb5007ed80 con 0x7ffb500b05e0 2021-02-10T11:15:27.887 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.889 7ffb47fff700 1 -- 172.21.2.112:0/3439580186 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 updated v10) v1 ==== 97+0+0 (secure 0 0 0) 0x7ffb38006030 con 0x7ffb500b05e0 2021-02-10T11:15:27.891 INFO:teuthology.orchestra.run.gibba012.stderr:updated 2021-02-10T11:15:27.892 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.894 7ffb558ec700 1 -- 172.21.2.112:0/3439580186 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ffb340186c0 msgr2=0x7ffb3401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:27.892 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.894 7ffb558ec700 1 --2- 172.21.2.112:0/3439580186 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ffb340186c0 0x7ffb3401abc0 crc :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:27.892 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.894 7ffb558ec700 1 -- 172.21.2.112:0/3439580186 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ffb500b05e0 msgr2=0x7ffb501a3cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:27.893 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.894 7ffb558ec700 1 --2- 172.21.2.112:0/3439580186 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ffb500b05e0 0x7ffb501a3cb0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7ffb38006fc0 tx=0x7ffb38009f70).stop 2021-02-10T11:15:27.893 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.895 7ffb558ec700 1 -- 172.21.2.112:0/3439580186 shutdown_connections 2021-02-10T11:15:27.893 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.895 7ffb558ec700 1 --2- 172.21.2.112:0/3439580186 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ffb340186c0 0x7ffb3401abc0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:27.893 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.895 7ffb558ec700 1 --2- 172.21.2.112:0/3439580186 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ffb500b09e0 0x7ffb500bcc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:27.893 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.895 7ffb558ec700 1 --2- 172.21.2.112:0/3439580186 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ffb501a42d0 0x7ffb500bd260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:27.893 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.895 7ffb558ec700 1 --2- 172.21.2.112:0/3439580186 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ffb500b05e0 0x7ffb501a3cb0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:27.893 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.895 7ffb558ec700 1 -- 172.21.2.112:0/3439580186 >> 172.21.2.112:0/3439580186 conn(0x7ffb500b9c30 msgr2=0x7ffb500af040 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:27.893 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.895 7ffb558ec700 1 -- 172.21.2.112:0/3439580186 shutdown_connections 2021-02-10T11:15:27.894 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:27.895 7ffb558ec700 1 -- 172.21.2.112:0/3439580186 wait complete. 2021-02-10T11:15:27.901 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2021-02-10T11:15:28.326 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.312 7ff219b31700 1 Processor -- start 2021-02-10T11:15:28.327 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.312 7ff219b31700 1 -- start start 2021-02-10T11:15:28.327 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.312 7ff219b31700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff2140e5070 0x7ff2141b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:28.327 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.312 7ff219b31700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7ff2141b3e40 0x7ff214080310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:28.327 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.312 7ff219b31700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff21407ed80 con 0x7ff2140e5470 2021-02-10T11:15:28.327 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.312 7ff219b31700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2140d1790 con 0x7ff2141b3e40 2021-02-10T11:15:28.327 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.312 7ff219b31700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2140d1980 con 0x7ff2140e5070 2021-02-10T11:15:28.327 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.312 7ff212ffd700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7ff2140e5470 0x7ff21407d690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:36366/0 (socket says 172.21.2.112:36366) 2021-02-10T11:15:28.327 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.312 7ff213fff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7ff2141b3e40 0x7ff214080310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:28.327 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.312 7ff212ffd700 1 -- 172.21.2.112:0/3980258948 learned_addr learned my addr 172.21.2.112:0/3980258948 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:15:28.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.313 7ff2137fe700 1 --2- 172.21.2.112:0/3980258948 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff2140e5070 0x7ff2141b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:28.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.313 7ff213fff700 1 -- 172.21.2.112:0/3980258948 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff2140e5070 msgr2=0x7ff2141b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:15:28.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.313 7ff213fff700 1 --2- 172.21.2.112:0/3980258948 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff2140e5070 0x7ff2141b3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:28.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.313 7ff213fff700 1 -- 172.21.2.112:0/3980258948 >> v1:172.21.2.120:6789/0 conn(0x7ff2140e5470 legacy=0x7ff21407d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:28.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.313 7ff213fff700 1 -- 172.21.2.112:0/3980258948 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2140d1ba0 con 0x7ff2141b3e40 2021-02-10T11:15:28.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.313 7ff213fff700 1 --2- 172.21.2.112:0/3980258948 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff2141b3e40 0x7ff214080310 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7ff20c006e30 tx=0x7ff20c009e60).ready entity=mon.0 client_cookie=eedc41f94171441f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:28.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.313 7ff2127fc700 1 -- 172.21.2.112:0/3980258948 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff20c003030 con 0x7ff2141b3e40 2021-02-10T11:15:28.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.313 7ff2127fc700 1 -- 172.21.2.112:0/3980258948 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff20c00b300 con 0x7ff2141b3e40 2021-02-10T11:15:28.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.313 7ff2127fc700 1 -- 172.21.2.112:0/3980258948 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff20c00b7e0 con 0x7ff2141b3e40 2021-02-10T11:15:28.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.313 7ff219b31700 1 -- 172.21.2.112:0/3980258948 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff2141b3e40 msgr2=0x7ff214080310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:28.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.313 7ff219b31700 1 --2- 172.21.2.112:0/3980258948 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff2141b3e40 0x7ff214080310 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7ff20c006e30 tx=0x7ff20c009e60).stop 2021-02-10T11:15:28.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.314 7ff219b31700 1 -- 172.21.2.112:0/3980258948 shutdown_connections 2021-02-10T11:15:28.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.314 7ff219b31700 1 --2- 172.21.2.112:0/3980258948 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff2141b3e40 0x7ff214080310 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:28.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.314 7ff219b31700 1 --2- 172.21.2.112:0/3980258948 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff2140e5070 0x7ff2141b3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:28.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.314 7ff219b31700 1 -- 172.21.2.112:0/3980258948 >> 172.21.2.112:0/3980258948 conn(0x7ff2140ce690 msgr2=0x7ff2140e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:28.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.314 7ff219b31700 1 -- 172.21.2.112:0/3980258948 shutdown_connections 2021-02-10T11:15:28.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.314 7ff219b31700 1 -- 172.21.2.112:0/3980258948 wait complete. 2021-02-10T11:15:28.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.328 7ff219b31700 1 Processor -- start 2021-02-10T11:15:28.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.329 7ff219b31700 1 -- start start 2021-02-10T11:15:28.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.329 7ff219b31700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff214193b60 0x7ff2140bac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:28.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.329 7ff219b31700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff2140bb240 0x7ff2141a6d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:28.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.329 7ff219b31700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff2140bb640 0x7ff2141a7390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:28.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.329 7ff219b31700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff214193f60 con 0x7ff2140bb240 2021-02-10T11:15:28.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.329 7ff219b31700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2141a79b0 con 0x7ff2140bb640 2021-02-10T11:15:28.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.329 7ff219b31700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2141a7bd0 con 0x7ff214193b60 2021-02-10T11:15:28.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.329 7ff212ffd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff2140bb240 0x7ff2141a6d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:28.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.329 7ff212ffd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff2140bb240 0x7ff2141a6d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:43354/0 (socket says 172.21.2.112:43354) 2021-02-10T11:15:28.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.329 7ff212ffd700 1 -- 172.21.2.112:0/3197869808 learned_addr learned my addr 172.21.2.112:0/3197869808 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:15:28.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.330 7ff2137fe700 1 --2- 172.21.2.112:0/3197869808 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff214193b60 0x7ff2140bac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:28.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.330 7ff213fff700 1 --2- 172.21.2.112:0/3197869808 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff2140bb640 0x7ff2141a7390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:28.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.330 7ff212ffd700 1 -- 172.21.2.112:0/3197869808 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff214193b60 msgr2=0x7ff2140bac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:28.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.330 7ff212ffd700 1 --2- 172.21.2.112:0/3197869808 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff214193b60 0x7ff2140bac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:28.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.330 7ff212ffd700 1 -- 172.21.2.112:0/3197869808 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff2140bb640 msgr2=0x7ff2141a7390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:28.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.330 7ff212ffd700 1 --2- 172.21.2.112:0/3197869808 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff2140bb640 0x7ff2141a7390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:28.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.330 7ff212ffd700 1 -- 172.21.2.112:0/3197869808 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2140bc110 con 0x7ff2140bb240 2021-02-10T11:15:28.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.330 7ff213fff700 1 --2- 172.21.2.112:0/3197869808 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff2140bb640 0x7ff2141a7390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:15:28.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.330 7ff212ffd700 1 --2- 172.21.2.112:0/3197869808 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff2140bb240 0x7ff2141a6d70 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7ff204001c60 tx=0x7ff20400a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:28.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.330 7ff20bfff700 1 -- 172.21.2.112:0/3197869808 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff20400acc0 con 0x7ff2140bb240 2021-02-10T11:15:28.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.330 7ff20bfff700 1 -- 172.21.2.112:0/3197869808 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff204004140 con 0x7ff2140bb240 2021-02-10T11:15:28.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.330 7ff20bfff700 1 -- 172.21.2.112:0/3197869808 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff2040045a0 con 0x7ff2140bb240 2021-02-10T11:15:28.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.330 7ff219b31700 1 -- 172.21.2.112:0/3197869808 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2140bc110 con 0x7ff2140bb240 2021-02-10T11:15:28.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.330 7ff219b31700 1 -- 172.21.2.112:0/3197869808 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2140bc6a0 con 0x7ff2140bb240 2021-02-10T11:15:28.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.331 7ff20bfff700 1 -- 172.21.2.112:0/3197869808 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff204004320 con 0x7ff2140bb240 2021-02-10T11:15:28.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.331 7ff20bfff700 1 --2- 172.21.2.112:0/3197869808 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff1f80186c0 0x7ff1f801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:28.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.331 7ff20bfff700 1 -- 172.21.2.112:0/3197869808 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4753+0+0 (secure 0 0 0) 0x7ff204032270 con 0x7ff2140bb240 2021-02-10T11:15:28.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.331 7ff2137fe700 1 --2- 172.21.2.112:0/3197869808 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff1f80186c0 0x7ff1f801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:28.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.331 7ff2137fe700 1 --2- 172.21.2.112:0/3197869808 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff1f80186c0 0x7ff1f801abc0 crc :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:28.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.362 7ff219b31700 1 -- 172.21.2.112:0/3197869808 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2140bbbe0 con 0x7ff2140bb240 2021-02-10T11:15:28.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.365 7ff20bfff700 1 -- 172.21.2.112:0/3197869808 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff20402df00 con 0x7ff2140bb240 2021-02-10T11:15:28.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.493 7ff219b31700 1 -- 172.21.2.112:0/3197869808 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) v1 -- 0x7ff2140bbbe0 con 0x7ff2140bb240 2021-02-10T11:15:28.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.497 7ff20bfff700 1 -- 172.21.2.112:0/3197869808 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 updated v11) v1 ==== 104+0+0 (secure 0 0 0) 0x7ff20402df00 con 0x7ff2140bb240 2021-02-10T11:15:28.498 INFO:teuthology.orchestra.run.gibba012.stderr:updated 2021-02-10T11:15:28.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.501 7ff219b31700 1 -- 172.21.2.112:0/3197869808 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff1f80186c0 msgr2=0x7ff1f801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:28.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.501 7ff219b31700 1 --2- 172.21.2.112:0/3197869808 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff1f80186c0 0x7ff1f801abc0 crc :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:28.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.501 7ff219b31700 1 -- 172.21.2.112:0/3197869808 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff2140bb240 msgr2=0x7ff2141a6d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:28.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.501 7ff219b31700 1 --2- 172.21.2.112:0/3197869808 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff2140bb240 0x7ff2141a6d70 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7ff204001c60 tx=0x7ff20400a8d0).stop 2021-02-10T11:15:28.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.501 7ff219b31700 1 -- 172.21.2.112:0/3197869808 shutdown_connections 2021-02-10T11:15:28.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.501 7ff219b31700 1 --2- 172.21.2.112:0/3197869808 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff1f80186c0 0x7ff1f801abc0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:28.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.501 7ff219b31700 1 --2- 172.21.2.112:0/3197869808 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff2140bb640 0x7ff2141a7390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:28.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.501 7ff219b31700 1 --2- 172.21.2.112:0/3197869808 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff2140bb240 0x7ff2141a6d70 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:28.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.501 7ff219b31700 1 --2- 172.21.2.112:0/3197869808 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff214193b60 0x7ff2140bac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:28.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.502 7ff219b31700 1 -- 172.21.2.112:0/3197869808 >> 172.21.2.112:0/3197869808 conn(0x7ff2140b9620 msgr2=0x7ff2140af040 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:28.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.502 7ff219b31700 1 -- 172.21.2.112:0/3197869808 shutdown_connections 2021-02-10T11:15:28.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.502 7ff219b31700 1 -- 172.21.2.112:0/3197869808 wait complete. 2021-02-10T11:15:28.508 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth del client.bootstrap-rgw 2021-02-10T11:15:28.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.919 7fd78db4e700 1 Processor -- start 2021-02-10T11:15:28.935 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.919 7fd78db4e700 1 -- start start 2021-02-10T11:15:28.935 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.919 7fd78db4e700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fd78807e6a0 0x7fd7880d1790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:28.935 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.919 7fd78db4e700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd78807ed80 con 0x7fd7880e5070 2021-02-10T11:15:28.935 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.919 7fd78db4e700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd78807eaa0 con 0x7fd7880e5470 2021-02-10T11:15:28.935 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.919 7fd78db4e700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7881b4080 con 0x7fd78807e6a0 2021-02-10T11:15:28.935 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.920 7fd7877fe700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fd7880e5070 0x7fd7881b3820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:42820/0 (socket says 172.21.2.112:42820) 2021-02-10T11:15:28.935 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.920 7fd7877fe700 1 -- 172.21.2.112:0/3503348870 learned_addr learned my addr 172.21.2.112:0/3503348870 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:15:28.935 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.920 7fd7867fc700 1 -- 172.21.2.112:0/3503348870 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1088661607 0 0) 0x7fd770001180 con 0x7fd7880e5070 2021-02-10T11:15:28.935 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.920 7fd787fff700 1 --2- 172.21.2.112:0/3503348870 >> v2:172.21.2.120:3300/0 conn(0x7fd78807e6a0 0x7fd7880d1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:28.935 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.920 7fd7867fc700 1 -- 172.21.2.112:0/3503348870 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd774002fe0 con 0x7fd7880e5070 2021-02-10T11:15:28.935 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.920 7fd7867fc700 1 -- 172.21.2.112:0/3503348870 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4185656998 0 0) 0x7fd778000c60 con 0x7fd7880e5470 2021-02-10T11:15:28.935 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.920 7fd7867fc700 1 -- 172.21.2.112:0/3503348870 --> v1:172.21.2.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd774003210 con 0x7fd7880e5470 2021-02-10T11:15:28.935 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.920 7fd7867fc700 1 -- 172.21.2.112:0/3503348870 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3300194817 0 0) 0x7fd770001180 con 0x7fd7880e5070 2021-02-10T11:15:28.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.920 7fd787fff700 1 -- 172.21.2.112:0/3503348870 >> v1:172.21.2.120:6789/0 conn(0x7fd7880e5470 legacy=0x7fd78807f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:28.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.920 7fd787fff700 1 -- 172.21.2.112:0/3503348870 >> v1:172.21.2.112:6789/0 conn(0x7fd7880e5070 legacy=0x7fd7881b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:28.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.920 7fd787fff700 1 -- 172.21.2.112:0/3503348870 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7880d1db0 con 0x7fd78807e6a0 2021-02-10T11:15:28.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.920 7fd787fff700 1 --2- 172.21.2.112:0/3503348870 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd78807e6a0 0x7fd7880d1790 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7fd780006e50 tx=0x7fd780009df0).ready entity=mon.1 client_cookie=caed8fa2392ecb78 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:28.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.921 7fd7867fc700 1 -- 172.21.2.112:0/3503348870 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd78000b510 con 0x7fd78807e6a0 2021-02-10T11:15:28.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.921 7fd7867fc700 1 -- 172.21.2.112:0/3503348870 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd780003030 con 0x7fd78807e6a0 2021-02-10T11:15:28.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.921 7fd7867fc700 1 -- 172.21.2.112:0/3503348870 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd78000b7d0 con 0x7fd78807e6a0 2021-02-10T11:15:28.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.921 7fd78db4e700 1 -- 172.21.2.112:0/3503348870 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd78807e6a0 msgr2=0x7fd7880d1790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:28.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.921 7fd78db4e700 1 --2- 172.21.2.112:0/3503348870 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd78807e6a0 0x7fd7880d1790 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7fd780006e50 tx=0x7fd780009df0).stop 2021-02-10T11:15:28.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.922 7fd78db4e700 1 -- 172.21.2.112:0/3503348870 shutdown_connections 2021-02-10T11:15:28.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.922 7fd78db4e700 1 --2- 172.21.2.112:0/3503348870 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd78807e6a0 0x7fd7880d1790 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:28.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.922 7fd78db4e700 1 -- 172.21.2.112:0/3503348870 >> 172.21.2.112:0/3503348870 conn(0x7fd7880ce690 msgr2=0x7fd7880e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:28.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.922 7fd78db4e700 1 -- 172.21.2.112:0/3503348870 shutdown_connections 2021-02-10T11:15:28.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.922 7fd78db4e700 1 -- 172.21.2.112:0/3503348870 wait complete. 2021-02-10T11:15:28.938 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.937 7fd78db4e700 1 Processor -- start 2021-02-10T11:15:28.938 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.937 7fd78db4e700 1 -- start start 2021-02-10T11:15:28.938 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.937 7fd78db4e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd7880bd700 0x7fd7880bdb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:28.938 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.937 7fd78db4e700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd7880a4b10 0x7fd7880a4f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:28.938 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.937 7fd78db4e700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd7880a5530 0x7fd7880a5930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:28.939 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.937 7fd78db4e700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7880be230 con 0x7fd7880bd700 2021-02-10T11:15:28.939 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.937 7fd78db4e700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7880be450 con 0x7fd7880a5530 2021-02-10T11:15:28.939 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.937 7fd78db4e700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7880be670 con 0x7fd7880a4b10 2021-02-10T11:15:28.939 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.937 7fd7877fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd7880bd700 0x7fd7880bdb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:28.939 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.937 7fd7877fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd7880bd700 0x7fd7880bdb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:43364/0 (socket says 172.21.2.112:43364) 2021-02-10T11:15:28.939 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.937 7fd7877fe700 1 -- 172.21.2.112:0/834766247 learned_addr learned my addr 172.21.2.112:0/834766247 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:15:28.939 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.938 7fd7877fe700 1 -- 172.21.2.112:0/834766247 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd7880a4b10 msgr2=0x7fd7880a4f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:28.939 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.938 7fd7877fe700 1 --2- 172.21.2.112:0/834766247 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd7880a4b10 0x7fd7880a4f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:28.939 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.938 7fd7877fe700 1 -- 172.21.2.112:0/834766247 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd7880a5530 msgr2=0x7fd7880a5930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:28.939 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.938 7fd7877fe700 1 --2- 172.21.2.112:0/834766247 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd7880a5530 0x7fd7880a5930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:28.939 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.938 7fd7877fe700 1 -- 172.21.2.112:0/834766247 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7880be890 con 0x7fd7880bd700 2021-02-10T11:15:28.939 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.938 7fd7877fe700 1 --2- 172.21.2.112:0/834766247 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd7880bd700 0x7fd7880bdb00 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7fd770000b80 tx=0x7fd77000a980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:28.939 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.938 7fd77ffff700 1 -- 172.21.2.112:0/834766247 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd770001490 con 0x7fd7880bd700 2021-02-10T11:15:28.939 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.938 7fd77ffff700 1 -- 172.21.2.112:0/834766247 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7700070d0 con 0x7fd7880bd700 2021-02-10T11:15:28.940 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.938 7fd78db4e700 1 -- 172.21.2.112:0/834766247 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7880be890 con 0x7fd7880bd700 2021-02-10T11:15:28.940 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.938 7fd78db4e700 1 -- 172.21.2.112:0/834766247 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7880bee20 con 0x7fd7880bd700 2021-02-10T11:15:28.940 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.939 7fd77ffff700 1 -- 172.21.2.112:0/834766247 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd770004830 con 0x7fd7880bd700 2021-02-10T11:15:28.940 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.939 7fd77ffff700 1 -- 172.21.2.112:0/834766247 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd7700043a0 con 0x7fd7880bd700 2021-02-10T11:15:28.940 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.939 7fd77ffff700 1 --2- 172.21.2.112:0/834766247 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd76c0186c0 0x7fd76c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:28.940 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.939 7fd77ffff700 1 -- 172.21.2.112:0/834766247 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4753+0+0 (secure 0 0 0) 0x7fd770032070 con 0x7fd7880bd700 2021-02-10T11:15:28.940 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.940 7fd786ffd700 1 --2- 172.21.2.112:0/834766247 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd76c0186c0 0x7fd76c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:28.940 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.940 7fd786ffd700 1 --2- 172.21.2.112:0/834766247 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd76c0186c0 0x7fd76c01abc0 crc :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:28.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.969 7fd78db4e700 1 -- 172.21.2.112:0/834766247 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd78807ed80 con 0x7fd7880bd700 2021-02-10T11:15:28.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:28.972 7fd77ffff700 1 -- 172.21.2.112:0/834766247 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd770004af0 con 0x7fd7880bd700 2021-02-10T11:15:29.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:29.100 7fd78db4e700 1 -- 172.21.2.112:0/834766247 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) v1 -- 0x7fd78807ed80 con 0x7fd7880bd700 2021-02-10T11:15:29.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:29.104 7fd77ffff700 1 -- 172.21.2.112:0/834766247 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 updated v12) v1 ==== 97+0+0 (secure 0 0 0) 0x7fd7700043a0 con 0x7fd7880bd700 2021-02-10T11:15:29.105 INFO:teuthology.orchestra.run.gibba012.stderr:updated 2021-02-10T11:15:29.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:29.109 7fd78db4e700 1 -- 172.21.2.112:0/834766247 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd76c0186c0 msgr2=0x7fd76c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:29.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:29.109 7fd78db4e700 1 --2- 172.21.2.112:0/834766247 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd76c0186c0 0x7fd76c01abc0 crc :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:29.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:29.109 7fd78db4e700 1 -- 172.21.2.112:0/834766247 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd7880bd700 msgr2=0x7fd7880bdb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:29.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:29.109 7fd78db4e700 1 --2- 172.21.2.112:0/834766247 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd7880bd700 0x7fd7880bdb00 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7fd770000b80 tx=0x7fd77000a980).stop 2021-02-10T11:15:29.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:29.109 7fd78db4e700 1 -- 172.21.2.112:0/834766247 shutdown_connections 2021-02-10T11:15:29.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:29.109 7fd78db4e700 1 --2- 172.21.2.112:0/834766247 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd76c0186c0 0x7fd76c01abc0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:29.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:29.109 7fd78db4e700 1 --2- 172.21.2.112:0/834766247 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd7880a5530 0x7fd7880a5930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:29.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:29.109 7fd78db4e700 1 --2- 172.21.2.112:0/834766247 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd7880a4b10 0x7fd7880a4f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:29.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:29.109 7fd78db4e700 1 --2- 172.21.2.112:0/834766247 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd7880bd700 0x7fd7880bdb00 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:29.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:29.109 7fd78db4e700 1 -- 172.21.2.112:0/834766247 >> 172.21.2.112:0/834766247 conn(0x7fd7880c0cf0 msgr2=0x7fd7880a70a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:29.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:29.110 7fd78db4e700 1 -- 172.21.2.112:0/834766247 shutdown_connections 2021-02-10T11:15:29.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:29.110 7fd78db4e700 1 -- 172.21.2.112:0/834766247 wait complete. 2021-02-10T11:15:29.115 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:15:29.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:29.526 7f9c0f4c0700 1 Processor -- start 2021-02-10T11:15:29.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:29.526 7f9c0f4c0700 1 -- start start 2021-02-10T11:15:29.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:29.527 7f9c0f4c0700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f9c080e5070 0x7f9c081b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:29.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:29.527 7f9c0f4c0700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9c081b3e40 0x7f9c0807d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:29.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:29.527 7f9c0f4c0700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c0807ed80 con 0x7f9c080e5470 2021-02-10T11:15:29.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:29.527 7f9c0f4c0700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c08080310 con 0x7f9c080e5070 2021-02-10T11:15:29.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:29.527 7f9c0f4c0700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c08080500 con 0x7f9c081b3e40 2021-02-10T11:15:29.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:29.527 7f9c0d2dc700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f9c080e5070 0x7f9c081b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:29.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:29.527 7f9c0d2dc700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f9c080e5070 0x7f9c081b3820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:43374/0 (socket says 172.21.2.112:43374) 2021-02-10T11:15:29.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:29.527 7f9c0d2dc700 1 -- 172.21.2.112:0/278411093 learned_addr learned my addr 172.21.2.112:0/278411093 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:15:29.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:29.527 7f9c0cadb700 1 --2- 172.21.2.112:0/278411093 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9c081b3e40 0x7f9c0807d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:29.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:29.527 7f9c0d2dc700 1 -- 172.21.2.112:0/278411093 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9c081b3e40 msgr2=0x7f9c0807d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:15:29.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:29.527 7f9c0d2dc700 1 --2- 172.21.2.112:0/278411093 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9c081b3e40 0x7f9c0807d690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:29.549 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:29.527 7f9c0d2dc700 1 -- 172.21.2.112:0/278411093 >> v1:172.21.2.120:6789/0 conn(0x7f9c080e5470 legacy=0x7f9c0807dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:15:29.549 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:29.527 7f9c0d2dc700 1 -- 172.21.2.112:0/278411093 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c080806f0 con 0x7f9c080e5070 2021-02-10T11:15:29.549 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:29.527 7f9c0cadb700 1 --2- 172.21.2.112:0/278411093 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9c081b3e40 0x7f9c0807d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:15:29.549 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:29.528 7f9c0d2dc700 1 --2- 172.21.2.112:0/278411093 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9c080e5070 0x7f9c081b3820 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7f9bfc006700 tx=0x7f9bfc009b80).ready entity=mon.0 client_cookie=3aec2bede2c03c89 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:29.549 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:29.528 7f9bfbfff700 1 -- 172.21.2.112:0/278411093 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9bfc00c0c0 con 0x7f9c080e5070 2021-02-10T11:15:29.549 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:29.528 7f9bfbfff700 1 -- 172.21.2.112:0/278411093 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bfc00c340 con 0x7f9c080e5070 2021-02-10T11:15:29.549 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:29.528 7f9bfbfff700 1 -- 172.21.2.112:0/278411093 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9bfc00c820 con 0x7f9c080e5070 2021-02-10T11:15:29.549 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:29.528 7f9c0f4c0700 1 -- 172.21.2.112:0/278411093 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9c080e5070 msgr2=0x7f9c081b3820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:29.549 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:29.528 7f9c0f4c0700 1 --2- 172.21.2.112:0/278411093 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9c080e5070 0x7f9c081b3820 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7f9bfc006700 tx=0x7f9bfc009b80).stop 2021-02-10T11:15:29.549 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:29.528 7f9c0f4c0700 1 -- 172.21.2.112:0/278411093 shutdown_connections 2021-02-10T11:15:29.549 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:29.528 7f9c0f4c0700 1 --2- 172.21.2.112:0/278411093 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9c081b3e40 0x7f9c0807d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:29.549 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:29.528 7f9c0f4c0700 1 --2- 172.21.2.112:0/278411093 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9c080e5070 0x7f9c081b3820 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:29.549 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:29.528 7f9c0f4c0700 1 -- 172.21.2.112:0/278411093 >> 172.21.2.112:0/278411093 conn(0x7f9c080ce690 msgr2=0x7f9c080e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:29.550 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:29.529 7f9c0f4c0700 1 -- 172.21.2.112:0/278411093 shutdown_connections 2021-02-10T11:15:29.550 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:29.529 7f9c0f4c0700 1 -- 172.21.2.112:0/278411093 wait complete. 2021-02-10T11:15:29.550 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:29.543 7f9c0f4c0700 1 Processor -- start 2021-02-10T11:15:29.550 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:29.543 7f9c0f4c0700 1 -- start start 2021-02-10T11:15:29.550 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:29.543 7f9c0f4c0700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9c0818c700 0x7f9c08096c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:29.550 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:29.543 7f9c0f4c0700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9c080ae400 0x7f9c080ae800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:29.550 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:29.543 7f9c0f4c0700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9c08192380 0x7f9c08192780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:29.550 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:29.543 7f9c0f4c0700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c08097290 con 0x7f9c08192380 2021-02-10T11:15:29.550 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:29.543 7f9c0f4c0700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c08192da0 con 0x7f9c080ae400 2021-02-10T11:15:29.550 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:29.543 7f9c0f4c0700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c08192f80 con 0x7f9c0818c700 2021-02-10T11:15:29.550 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:29.544 7f9c0dadd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9c08192380 0x7f9c08192780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:29.550 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:29.544 7f9c0dadd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9c08192380 0x7f9c08192780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:43382/0 (socket says 172.21.2.112:43382) 2021-02-10T11:15:29.550 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:29.544 7f9c0dadd700 1 -- 172.21.2.112:0/1827314772 learned_addr learned my addr 172.21.2.112:0/1827314772 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:15:29.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:29.544 7f9c0dadd700 1 -- 172.21.2.112:0/1827314772 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9c0818c700 msgr2=0x7f9c08096c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:29.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:29.544 7f9c0dadd700 1 --2- 172.21.2.112:0/1827314772 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9c0818c700 0x7f9c08096c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:29.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:29.544 7f9c0dadd700 1 -- 172.21.2.112:0/1827314772 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9c080ae400 msgr2=0x7f9c080ae800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:29.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:29.544 7f9c0dadd700 1 --2- 172.21.2.112:0/1827314772 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9c080ae400 0x7f9c080ae800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:29.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:29.544 7f9c0dadd700 1 -- 172.21.2.112:0/1827314772 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c081931a0 con 0x7f9c08192380 2021-02-10T11:15:29.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:29.544 7f9c0dadd700 1 --2- 172.21.2.112:0/1827314772 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9c08192380 0x7f9c08192780 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7f9c00009940 tx=0x7f9c0000bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:29.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:29.544 7f9bf9ffb700 1 -- 172.21.2.112:0/1827314772 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9c00000f90 con 0x7f9c08192380 2021-02-10T11:15:29.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:29.544 7f9bf9ffb700 1 -- 172.21.2.112:0/1827314772 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c0000c2f0 con 0x7f9c08192380 2021-02-10T11:15:29.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:29.544 7f9bf9ffb700 1 -- 172.21.2.112:0/1827314772 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9c0000c750 con 0x7f9c08192380 2021-02-10T11:15:29.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:29.544 7f9c0f4c0700 1 -- 172.21.2.112:0/1827314772 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c081931a0 con 0x7f9c08192380 2021-02-10T11:15:29.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:29.544 7f9c0f4c0700 1 -- 172.21.2.112:0/1827314772 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c0807f510 con 0x7f9c08192380 2021-02-10T11:15:29.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:29.545 7f9bf9ffb700 1 -- 172.21.2.112:0/1827314772 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9c00001220 con 0x7f9c08192380 2021-02-10T11:15:29.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:29.545 7f9bf9ffb700 1 --2- 172.21.2.112:0/1827314772 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9be80186c0 0x7f9be801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:29.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:29.545 7f9bf9ffb700 1 -- 172.21.2.112:0/1827314772 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f9c00032410 con 0x7f9c08192380 2021-02-10T11:15:29.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:29.546 7f9c0d2dc700 1 --2- 172.21.2.112:0/1827314772 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9be80186c0 0x7f9be801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:29.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:29.546 7f9c0d2dc700 1 --2- 172.21.2.112:0/1827314772 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9be80186c0 0x7f9be801abc0 crc :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:29.575 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:29.577 7f9c0f4c0700 1 -- 172.21.2.112:0/1827314772 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c08107120 con 0x7f9c08192380 2021-02-10T11:15:29.577 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:29.579 7f9bf9ffb700 1 -- 172.21.2.112:0/1827314772 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9c00000f40 con 0x7f9c08192380 2021-02-10T11:15:29.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:29.701 7f9c0f4c0700 1 -- 172.21.2.112:0/1827314772 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9c0807ed80 con 0x7f9c08192380 2021-02-10T11:15:29.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:29.702 7f9bf9ffb700 1 -- 172.21.2.112:0/1827314772 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+11144 (secure 0 0 0) 0x7f9c0001f860 con 0x7f9c08192380 2021-02-10T11:15:29.701 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:15:29.701 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":25,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:15:28.288976","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:15:29.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:29.703 7f9c0f4c0700 1 -- 172.21.2.112:0/1827314772 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9be80186c0 msgr2=0x7f9be801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:29.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:29.703 7f9c0f4c0700 1 --2- 172.21.2.112:0/1827314772 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9be80186c0 0x7f9be801abc0 crc :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:29.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:29.703 7f9c0f4c0700 1 -- 172.21.2.112:0/1827314772 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9c08192380 msgr2=0x7f9c08192780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:29.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:29.703 7f9c0f4c0700 1 --2- 172.21.2.112:0/1827314772 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9c08192380 0x7f9c08192780 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7f9c00009940 tx=0x7f9c0000bcf0).stop 2021-02-10T11:15:29.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:29.704 7f9c0f4c0700 1 -- 172.21.2.112:0/1827314772 shutdown_connections 2021-02-10T11:15:29.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:29.704 7f9c0f4c0700 1 --2- 172.21.2.112:0/1827314772 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9be80186c0 0x7f9be801abc0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:29.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:29.704 7f9c0f4c0700 1 --2- 172.21.2.112:0/1827314772 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9c080ae400 0x7f9c080ae800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:29.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:29.704 7f9c0f4c0700 1 --2- 172.21.2.112:0/1827314772 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9c08192380 0x7f9c08192780 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:29.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:29.704 7f9c0f4c0700 1 --2- 172.21.2.112:0/1827314772 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9c0818c700 0x7f9c08096c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:29.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:29.704 7f9c0f4c0700 1 -- 172.21.2.112:0/1827314772 >> 172.21.2.112:0/1827314772 conn(0x7f9c080891f0 msgr2=0x7f9c081a0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:29.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:29.704 7f9c0f4c0700 1 -- 172.21.2.112:0/1827314772 shutdown_connections 2021-02-10T11:15:29.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:29.704 7f9c0f4c0700 1 -- 172.21.2.112:0/1827314772 wait complete. 2021-02-10T11:15:29.710 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:13:57.508758', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2021-02-10T11:15:29.711 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:15:30.153 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.134 7efe6da19700 1 Processor -- start 2021-02-10T11:15:30.154 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.134 7efe6da19700 1 -- start start 2021-02-10T11:15:30.154 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.134 7efe6da19700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efe681b0470 0x7efe681b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:30.154 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.134 7efe6da19700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7efe680ef610 0x7efe680efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:30.154 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.134 7efe6da19700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7efe680f0030 0x7efe680f0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:30.154 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.134 7efe6da19700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe680e5ea0 con 0x7efe680ef610 2021-02-10T11:15:30.154 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.134 7efe6da19700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe680f0b50 con 0x7efe680f0030 2021-02-10T11:15:30.154 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.134 7efe6da19700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe680e52e0 con 0x7efe681b0470 2021-02-10T11:15:30.154 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.134 7efe66ffd700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7efe680ef610 0x7efe680efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:30.155 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.134 7efe66ffd700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7efe680ef610 0x7efe680efa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:43388/0 (socket says 172.21.2.112:43388) 2021-02-10T11:15:30.155 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.134 7efe66ffd700 1 -- 172.21.2.112:0/1093179768 learned_addr learned my addr 172.21.2.112:0/1093179768 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:15:30.155 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.134 7efe67fff700 1 --2- 172.21.2.112:0/1093179768 >> v2:172.21.2.120:3300/0 conn(0x7efe680f0030 0x7efe680f0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:30.155 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.134 7efe677fe700 1 --2- 172.21.2.112:0/1093179768 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efe681b0470 0x7efe681b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:30.155 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.135 7efe66ffd700 1 -- 172.21.2.112:0/1093179768 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efe681b0470 msgr2=0x7efe681b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:15:30.155 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.135 7efe66ffd700 1 --2- 172.21.2.112:0/1093179768 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efe681b0470 0x7efe681b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:30.155 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.135 7efe66ffd700 1 -- 172.21.2.112:0/1093179768 >> v2:172.21.2.120:3300/0 conn(0x7efe680f0030 msgr2=0x7efe680f0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:15:30.155 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.135 7efe66ffd700 1 --2- 172.21.2.112:0/1093179768 >> v2:172.21.2.120:3300/0 conn(0x7efe680f0030 0x7efe680f0430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:30.155 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.135 7efe66ffd700 1 -- 172.21.2.112:0/1093179768 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe680f10d0 con 0x7efe680ef610 2021-02-10T11:15:30.155 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.135 7efe66ffd700 1 --2- 172.21.2.112:0/1093179768 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efe680ef610 0x7efe680efa10 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0x7efe60006a80 tx=0x7efe60009f00).ready entity=mon.0 client_cookie=ba467b4969790c02 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:30.155 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.135 7efe667fc700 1 -- 172.21.2.112:0/1093179768 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efe6000b5c0 con 0x7efe680ef610 2021-02-10T11:15:30.155 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.135 7efe667fc700 1 -- 172.21.2.112:0/1093179768 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe6000b800 con 0x7efe680ef610 2021-02-10T11:15:30.155 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.135 7efe667fc700 1 -- 172.21.2.112:0/1093179768 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efe6000bd20 con 0x7efe680ef610 2021-02-10T11:15:30.156 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.135 7efe6da19700 1 -- 172.21.2.112:0/1093179768 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efe680ef610 msgr2=0x7efe680efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:30.156 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.135 7efe6da19700 1 --2- 172.21.2.112:0/1093179768 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efe680ef610 0x7efe680efa10 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0x7efe60006a80 tx=0x7efe60009f00).stop 2021-02-10T11:15:30.156 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.136 7efe6da19700 1 -- 172.21.2.112:0/1093179768 shutdown_connections 2021-02-10T11:15:30.156 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.136 7efe6da19700 1 --2- 172.21.2.112:0/1093179768 >> v2:172.21.2.120:3300/0 conn(0x7efe680f0030 0x7efe680f0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:30.156 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.136 7efe6da19700 1 --2- 172.21.2.112:0/1093179768 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efe680ef610 0x7efe680efa10 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:30.156 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.136 7efe6da19700 1 --2- 172.21.2.112:0/1093179768 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efe681b0470 0x7efe681b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:30.156 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.136 7efe6da19700 1 -- 172.21.2.112:0/1093179768 >> 172.21.2.112:0/1093179768 conn(0x7efe680ce3e0 msgr2=0x7efe680e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:30.156 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.136 7efe6da19700 1 -- 172.21.2.112:0/1093179768 shutdown_connections 2021-02-10T11:15:30.156 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.136 7efe6da19700 1 -- 172.21.2.112:0/1093179768 wait complete. 2021-02-10T11:15:30.156 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.150 7efe6da19700 1 Processor -- start 2021-02-10T11:15:30.156 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.150 7efe6da19700 1 -- start start 2021-02-10T11:15:30.156 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.151 7efe6da19700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7efe6809b930 0x7efe680b7c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:30.157 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.151 7efe6da19700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efe680b8410 0x7efe680b8810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:30.157 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.151 7efe6da19700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efe680c1f80 0x7efe680c2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:30.157 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.151 7efe6da19700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe680c1a00 con 0x7efe680b8410 2021-02-10T11:15:30.157 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.151 7efe6da19700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe680b8e30 con 0x7efe6809b930 2021-02-10T11:15:30.157 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.151 7efe6da19700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe680b9050 con 0x7efe680c1f80 2021-02-10T11:15:30.157 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.151 7efe66ffd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efe680b8410 0x7efe680b8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:30.157 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.151 7efe66ffd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efe680b8410 0x7efe680b8810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:43394/0 (socket says 172.21.2.112:43394) 2021-02-10T11:15:30.157 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.151 7efe66ffd700 1 -- 172.21.2.112:0/2755435723 learned_addr learned my addr 172.21.2.112:0/2755435723 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:15:30.157 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.151 7efe67fff700 1 --2- 172.21.2.112:0/2755435723 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efe680c1f80 0x7efe680c2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:30.157 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.151 7efe677fe700 1 --2- 172.21.2.112:0/2755435723 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7efe6809b930 0x7efe680b7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:30.157 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.151 7efe66ffd700 1 -- 172.21.2.112:0/2755435723 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efe680c1f80 msgr2=0x7efe680c2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:30.157 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.151 7efe66ffd700 1 --2- 172.21.2.112:0/2755435723 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efe680c1f80 0x7efe680c2380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:30.157 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.151 7efe66ffd700 1 -- 172.21.2.112:0/2755435723 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7efe6809b930 msgr2=0x7efe680b7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:30.158 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.151 7efe66ffd700 1 --2- 172.21.2.112:0/2755435723 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7efe6809b930 0x7efe680b7c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:30.158 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.151 7efe66ffd700 1 -- 172.21.2.112:0/2755435723 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe680c29a0 con 0x7efe680b8410 2021-02-10T11:15:30.158 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.151 7efe66ffd700 1 --2- 172.21.2.112:0/2755435723 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efe680b8410 0x7efe680b8810 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7efe60006a90 tx=0x7efe6000bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:30.158 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.152 7efe4ffff700 1 -- 172.21.2.112:0/2755435723 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efe60000d30 con 0x7efe680b8410 2021-02-10T11:15:30.158 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.152 7efe4ffff700 1 -- 172.21.2.112:0/2755435723 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe60006710 con 0x7efe680b8410 2021-02-10T11:15:30.158 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.152 7efe4ffff700 1 -- 172.21.2.112:0/2755435723 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efe6000c4e0 con 0x7efe680b8410 2021-02-10T11:15:30.158 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.152 7efe6da19700 1 -- 172.21.2.112:0/2755435723 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe680c29a0 con 0x7efe680b8410 2021-02-10T11:15:30.158 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.152 7efe6da19700 1 -- 172.21.2.112:0/2755435723 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe680c29a0 con 0x7efe680b8410 2021-02-10T11:15:30.158 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.153 7efe4ffff700 1 -- 172.21.2.112:0/2755435723 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7efe60007000 con 0x7efe680b8410 2021-02-10T11:15:30.158 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.153 7efe4ffff700 1 --2- 172.21.2.112:0/2755435723 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7efe48018730 0x7efe4801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:30.158 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.153 7efe4ffff700 1 -- 172.21.2.112:0/2755435723 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4753+0+0 (secure 0 0 0) 0x7efe60032820 con 0x7efe680b8410 2021-02-10T11:15:30.158 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.153 7efe677fe700 1 --2- 172.21.2.112:0/2755435723 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7efe48018730 0x7efe4801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:30.158 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.153 7efe677fe700 1 --2- 172.21.2.112:0/2755435723 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7efe48018730 0x7efe4801ac10 crc :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:30.184 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.184 7efe6da19700 1 -- 172.21.2.112:0/2755435723 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe68105b30 con 0x7efe680b8410 2021-02-10T11:15:30.186 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.187 7efe4ffff700 1 -- 172.21.2.112:0/2755435723 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7efe6001f890 con 0x7efe680b8410 2021-02-10T11:15:30.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.314 7efe6da19700 1 -- 172.21.2.112:0/2755435723 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7efe68105b30 con 0x7efe680b8410 2021-02-10T11:15:30.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.315 7efe4ffff700 1 -- 172.21.2.112:0/2755435723 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v25) v1 ==== 92+0+10 (secure 0 0 0) 0x7efe60007000 con 0x7efe680b8410 2021-02-10T11:15:30.313 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:15:30.315 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.317 7efe6da19700 1 -- 172.21.2.112:0/2755435723 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7efe48018730 msgr2=0x7efe4801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:30.315 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.317 7efe6da19700 1 --2- 172.21.2.112:0/2755435723 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7efe48018730 0x7efe4801ac10 crc :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:30.315 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.317 7efe6da19700 1 -- 172.21.2.112:0/2755435723 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efe680b8410 msgr2=0x7efe680b8810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:30.315 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.317 7efe6da19700 1 --2- 172.21.2.112:0/2755435723 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efe680b8410 0x7efe680b8810 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7efe60006a90 tx=0x7efe6000bd10).stop 2021-02-10T11:15:30.315 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.317 7efe6da19700 1 -- 172.21.2.112:0/2755435723 shutdown_connections 2021-02-10T11:15:30.315 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.317 7efe6da19700 1 --2- 172.21.2.112:0/2755435723 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7efe48018730 0x7efe4801ac10 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:30.315 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.317 7efe6da19700 1 --2- 172.21.2.112:0/2755435723 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efe680c1f80 0x7efe680c2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:30.315 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.317 7efe6da19700 1 --2- 172.21.2.112:0/2755435723 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efe680b8410 0x7efe680b8810 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:30.315 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.317 7efe6da19700 1 --2- 172.21.2.112:0/2755435723 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7efe6809b930 0x7efe680b7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:30.315 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.317 7efe6da19700 1 -- 172.21.2.112:0/2755435723 >> 172.21.2.112:0/2755435723 conn(0x7efe68094e10 msgr2=0x7efe680a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:30.316 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.317 7efe6da19700 1 -- 172.21.2.112:0/2755435723 shutdown_connections 2021-02-10T11:15:30.316 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.317 7efe6da19700 1 -- 172.21.2.112:0/2755435723 wait complete. 2021-02-10T11:15:30.324 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --format=json-pretty osd lspools 2021-02-10T11:15:30.754 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.739 7f4288e98700 1 Processor -- start 2021-02-10T11:15:30.754 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.739 7f4288e98700 1 -- start start 2021-02-10T11:15:30.754 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.740 7f4288e98700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f42840e5070 0x7f42841b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:30.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.740 7f4288e98700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f428407ed80 con 0x7f42841b3e40 2021-02-10T11:15:30.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.740 7f4288e98700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f428407e670 con 0x7f42840e5470 2021-02-10T11:15:30.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.740 7f4288e98700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f428407e890 con 0x7f42840e5070 2021-02-10T11:15:30.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.740 7f4282312700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f42841b3e40 0x7f428407d690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:42866/0 (socket says 172.21.2.112:42866) 2021-02-10T11:15:30.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.740 7f4282312700 1 -- 172.21.2.112:0/451553445 learned_addr learned my addr 172.21.2.112:0/451553445 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:15:30.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.740 7f4282b13700 1 --2- 172.21.2.112:0/451553445 >> v2:172.21.2.120:3300/0 conn(0x7f42840e5070 0x7f42841b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:30.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.740 7f4281b11700 1 -- 172.21.2.112:0/451553445 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 830637173 0 0) 0x7f42740009b0 con 0x7f42841b3e40 2021-02-10T11:15:30.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.740 7f4281b11700 1 -- 172.21.2.112:0/451553445 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4270002fe0 con 0x7f42841b3e40 2021-02-10T11:15:30.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.741 7f4282b13700 1 -- 172.21.2.112:0/451553445 >> v1:172.21.2.120:6789/0 conn(0x7f42840e5470 legacy=0x7f428407de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:30.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.741 7f4282b13700 1 -- 172.21.2.112:0/451553445 >> v1:172.21.2.112:6789/0 conn(0x7f42841b3e40 legacy=0x7f428407d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:30.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.741 7f4282b13700 1 -- 172.21.2.112:0/451553445 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f428407eab0 con 0x7f42840e5070 2021-02-10T11:15:30.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.741 7f4282b13700 1 --2- 172.21.2.112:0/451553445 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f42840e5070 0x7f42841b3820 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f426c006650 tx=0x7f426c009a80).ready entity=mon.1 client_cookie=51bb7cb3910340d2 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:30.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.741 7f4281b11700 1 -- 172.21.2.112:0/451553445 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f426c00c0c0 con 0x7f42840e5070 2021-02-10T11:15:30.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.741 7f4281b11700 1 -- 172.21.2.112:0/451553445 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f426c00c440 con 0x7f42840e5070 2021-02-10T11:15:30.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.741 7f4281b11700 1 -- 172.21.2.112:0/451553445 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f426c00c960 con 0x7f42840e5070 2021-02-10T11:15:30.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.741 7f4288e98700 1 -- 172.21.2.112:0/451553445 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f42840e5070 msgr2=0x7f42841b3820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:30.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.741 7f4288e98700 1 --2- 172.21.2.112:0/451553445 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f42840e5070 0x7f42841b3820 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f426c006650 tx=0x7f426c009a80).stop 2021-02-10T11:15:30.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.742 7f4288e98700 1 -- 172.21.2.112:0/451553445 shutdown_connections 2021-02-10T11:15:30.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.742 7f4288e98700 1 --2- 172.21.2.112:0/451553445 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f42840e5070 0x7f42841b3820 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:30.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.742 7f4288e98700 1 -- 172.21.2.112:0/451553445 >> 172.21.2.112:0/451553445 conn(0x7f42840ce690 msgr2=0x7f42840e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:30.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.742 7f4288e98700 1 -- 172.21.2.112:0/451553445 shutdown_connections 2021-02-10T11:15:30.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.742 7f4288e98700 1 -- 172.21.2.112:0/451553445 wait complete. 2021-02-10T11:15:30.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.756 7f4288e98700 1 Processor -- start 2021-02-10T11:15:30.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.756 7f4288e98700 1 -- start start 2021-02-10T11:15:30.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.757 7f4288e98700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f42840b7a90 0x7f42840b7e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:30.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.757 7f4288e98700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4284084180 0x7f4284084580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:30.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.757 7f4288e98700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4284084ba0 0x7f4284084fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:30.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.757 7f4288e98700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f428409d0f0 con 0x7f42840b7a90 2021-02-10T11:15:30.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.757 7f4288e98700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4284191ed0 con 0x7f4284084180 2021-02-10T11:15:30.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.757 7f4288e98700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42841a9190 con 0x7f4284084ba0 2021-02-10T11:15:30.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.757 7f4282b13700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f42840b7a90 0x7f42840b7e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:30.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.757 7f4283314700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4284084ba0 0x7f4284084fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:30.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.757 7f4282b13700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f42840b7a90 0x7f42840b7e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:43410/0 (socket says 172.21.2.112:43410) 2021-02-10T11:15:30.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.757 7f4282b13700 1 -- 172.21.2.112:0/1640398953 learned_addr learned my addr 172.21.2.112:0/1640398953 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:15:30.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.757 7f4282312700 1 --2- 172.21.2.112:0/1640398953 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4284084180 0x7f4284084580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:30.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.757 7f4282b13700 1 -- 172.21.2.112:0/1640398953 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4284084ba0 msgr2=0x7f4284084fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:30.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.757 7f4282b13700 1 --2- 172.21.2.112:0/1640398953 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4284084ba0 0x7f4284084fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:30.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.757 7f4282b13700 1 -- 172.21.2.112:0/1640398953 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4284084180 msgr2=0x7f4284084580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:30.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.757 7f4282b13700 1 --2- 172.21.2.112:0/1640398953 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4284084180 0x7f4284084580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:30.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.757 7f4282b13700 1 -- 172.21.2.112:0/1640398953 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42841a93b0 con 0x7f42840b7a90 2021-02-10T11:15:30.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.758 7f4282b13700 1 --2- 172.21.2.112:0/1640398953 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f42840b7a90 0x7f42840b7e90 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7f426c00b5e0 tx=0x7f426c00b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:30.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.758 7f427b7fe700 1 -- 172.21.2.112:0/1640398953 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f426c00bd70 con 0x7f42840b7a90 2021-02-10T11:15:30.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.758 7f427b7fe700 1 -- 172.21.2.112:0/1640398953 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f426c006030 con 0x7f42840b7a90 2021-02-10T11:15:30.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.758 7f427b7fe700 1 -- 172.21.2.112:0/1640398953 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f426c0052e0 con 0x7f42840b7a90 2021-02-10T11:15:30.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.758 7f4288e98700 1 -- 172.21.2.112:0/1640398953 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42841a93b0 con 0x7f42840b7a90 2021-02-10T11:15:30.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.758 7f4288e98700 1 -- 172.21.2.112:0/1640398953 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42841a93b0 con 0x7f42840b7a90 2021-02-10T11:15:30.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.759 7f427b7fe700 1 -- 172.21.2.112:0/1640398953 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f426c0052e0 con 0x7f42840b7a90 2021-02-10T11:15:30.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.759 7f427b7fe700 1 --2- 172.21.2.112:0/1640398953 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f42680186d0 0x7f426801abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:30.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.759 7f427b7fe700 1 -- 172.21.2.112:0/1640398953 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f426c0322f0 con 0x7f42840b7a90 2021-02-10T11:15:30.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.759 7f4282312700 1 --2- 172.21.2.112:0/1640398953 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f42680186d0 0x7f426801abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:30.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.759 7f4282312700 1 --2- 172.21.2.112:0/1640398953 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f42680186d0 0x7f426801abd0 crc :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:30.787 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.789 7f4288e98700 1 -- 172.21.2.112:0/1640398953 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42841aa850 con 0x7f42840b7a90 2021-02-10T11:15:30.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.792 7f427b7fe700 1 -- 172.21.2.112:0/1640398953 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f426c000970 con 0x7f42840b7a90 2021-02-10T11:15:30.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.917 7f4288e98700 1 -- 172.21.2.112:0/1640398953 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f42841aa850 con 0x7f42840b7a90 2021-02-10T11:15:30.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.917 7f427b7fe700 1 -- 172.21.2.112:0/1640398953 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v25) v1 ==== 84+0+64 (secure 0 0 0) 0x7f426c00bd70 con 0x7f42840b7a90 2021-02-10T11:15:30.916 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:15:30.916 INFO:teuthology.orchestra.run.gibba012.stdout:[ 2021-02-10T11:15:30.916 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:15:30.916 INFO:teuthology.orchestra.run.gibba012.stdout: "poolnum": 1, 2021-02-10T11:15:30.916 INFO:teuthology.orchestra.run.gibba012.stdout: "poolname": "rbd" 2021-02-10T11:15:30.917 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:15:30.917 INFO:teuthology.orchestra.run.gibba012.stdout:] 2021-02-10T11:15:30.917 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.919 7f4288e98700 1 -- 172.21.2.112:0/1640398953 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f42680186d0 msgr2=0x7f426801abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:30.917 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.919 7f4288e98700 1 --2- 172.21.2.112:0/1640398953 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f42680186d0 0x7f426801abd0 crc :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:30.917 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.919 7f4288e98700 1 -- 172.21.2.112:0/1640398953 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f42840b7a90 msgr2=0x7f42840b7e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:30.917 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.919 7f4288e98700 1 --2- 172.21.2.112:0/1640398953 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f42840b7a90 0x7f42840b7e90 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7f426c00b5e0 tx=0x7f426c00b980).stop 2021-02-10T11:15:30.918 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.920 7f4288e98700 1 -- 172.21.2.112:0/1640398953 shutdown_connections 2021-02-10T11:15:30.918 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.920 7f4288e98700 1 --2- 172.21.2.112:0/1640398953 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f42680186d0 0x7f426801abd0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:30.918 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.920 7f4288e98700 1 --2- 172.21.2.112:0/1640398953 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4284084180 0x7f4284084580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:30.918 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.920 7f4288e98700 1 --2- 172.21.2.112:0/1640398953 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4284084ba0 0x7f4284084fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:30.918 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.920 7f4288e98700 1 --2- 172.21.2.112:0/1640398953 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f42840b7a90 0x7f42840b7e90 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:30.918 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.920 7f4288e98700 1 -- 172.21.2.112:0/1640398953 >> 172.21.2.112:0/1640398953 conn(0x7f42841b2dc0 msgr2=0x7f4284092c70 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:30.919 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.920 7f4288e98700 1 -- 172.21.2.112:0/1640398953 shutdown_connections 2021-02-10T11:15:30.919 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:30.920 7f4288e98700 1 -- 172.21.2.112:0/1640398953 wait complete. 2021-02-10T11:15:30.926 INFO:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2021-02-10T11:15:30.926 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_metadata 8 2021-02-10T11:15:31.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:31.343 7f6620842700 1 Processor -- start 2021-02-10T11:15:31.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:31.343 7f6620842700 1 -- start start 2021-02-10T11:15:31.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:31.344 7f6620842700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f661c1b0870 0x7f661c0f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:31.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:31.344 7f6620842700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f661c1b0c70 con 0x7f661c0efe00 2021-02-10T11:15:31.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:31.344 7f6620842700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f661c0f0b50 con 0x7f661c1b0470 2021-02-10T11:15:31.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:31.344 7f6620842700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f661c0e5ea0 con 0x7f661c1b0870 2021-02-10T11:15:31.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:31.344 7f661ae1d700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f661c0efe00 0x7f661c0f0200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:42880/0 (socket says 172.21.2.112:42880) 2021-02-10T11:15:31.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:31.344 7f661ae1d700 1 -- 172.21.2.112:0/3865231986 learned_addr learned my addr 172.21.2.112:0/3865231986 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:15:31.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:31.345 7f6619e1b700 1 --2- 172.21.2.112:0/3865231986 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f661c1b0870 0x7f661c0f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:31.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:31.345 7f661961a700 1 -- 172.21.2.112:0/3865231986 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3324887417 0 0) 0x7f6614001180 con 0x7f661c0efe00 2021-02-10T11:15:31.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:31.345 7f661961a700 1 -- 172.21.2.112:0/3865231986 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6608002fe0 con 0x7f661c0efe00 2021-02-10T11:15:31.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:31.345 7f661961a700 1 -- 172.21.2.112:0/3865231986 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2295766502 0 0) 0x7f6604000fe0 con 0x7f661c1b0470 2021-02-10T11:15:31.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:31.345 7f661961a700 1 -- 172.21.2.112:0/3865231986 --> v1:172.21.2.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6608003210 con 0x7f661c1b0470 2021-02-10T11:15:31.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:31.345 7f661961a700 1 -- 172.21.2.112:0/3865231986 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2531378973 0 0) 0x7f6614001180 con 0x7f661c0efe00 2021-02-10T11:15:31.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:31.345 7f661961a700 1 -- 172.21.2.112:0/3865231986 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f661c1b0870 msgr2=0x7f661c0f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:15:31.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:31.345 7f661961a700 1 --2- 172.21.2.112:0/3865231986 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f661c1b0870 0x7f661c0f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:31.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:31.345 7f661961a700 1 -- 172.21.2.112:0/3865231986 >> v1:172.21.2.120:6789/0 conn(0x7f661c1b0470 legacy=0x7f661c0ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:31.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:31.345 7f661961a700 1 -- 172.21.2.112:0/3865231986 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f661c0e52e0 con 0x7f661c0efe00 2021-02-10T11:15:31.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:31.345 7f661961a700 1 -- 172.21.2.112:0/3865231986 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f6614001680 con 0x7f661c0efe00 2021-02-10T11:15:31.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:31.345 7f661961a700 1 -- 172.21.2.112:0/3865231986 >> v1:172.21.2.112:6789/0 conn(0x7f661c0efe00 legacy=0x7f661c0f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:31.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:31.345 7f661961a700 1 --2- 172.21.2.112:0/3865231986 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6608003900 0x7f6608003d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:31.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:31.345 7f661961a700 1 --2- 172.21.2.112:0/3865231986 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6608006860 0x7f6608008d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:31.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:31.345 7f661961a700 1 --2- 172.21.2.112:0/3865231986 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6608009620 0x7f660800bb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:31.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:31.345 7f661961a700 1 -- 172.21.2.112:0/3865231986 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f660800c360 con 0x7f6608006860 2021-02-10T11:15:31.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:31.345 7f661961a700 1 -- 172.21.2.112:0/3865231986 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6608050550 con 0x7f6608003900 2021-02-10T11:15:31.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:31.345 7f661961a700 1 -- 172.21.2.112:0/3865231986 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6608050770 con 0x7f6608009620 2021-02-10T11:15:31.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:31.345 7f6619e1b700 1 --2- 172.21.2.112:0/3865231986 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f661c1b0870 0x7f661c0f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:15:31.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:31.345 7f661a61c700 1 --2- 172.21.2.112:0/3865231986 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6608003900 0x7f6608003d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:31.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:31.345 7f6619e1b700 1 --2- 172.21.2.112:0/3865231986 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6608006860 0x7f6608008d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:31.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:31.345 7f661ae1d700 1 --2- 172.21.2.112:0/3865231986 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6608009620 0x7f660800bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:31.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:31.346 7f6619e1b700 1 -- 172.21.2.112:0/3865231986 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6608009620 msgr2=0x7f660800bb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:15:31.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:31.346 7f6619e1b700 1 --2- 172.21.2.112:0/3865231986 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6608009620 0x7f660800bb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:31.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:31.346 7f6619e1b700 1 -- 172.21.2.112:0/3865231986 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6608003900 msgr2=0x7f6608003d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:15:31.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:31.346 7f6619e1b700 1 --2- 172.21.2.112:0/3865231986 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6608003900 0x7f6608003d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:31.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:31.346 7f6619e1b700 1 -- 172.21.2.112:0/3865231986 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6608050a10 con 0x7f6608006860 2021-02-10T11:15:31.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:31.346 7f6619e1b700 1 --2- 172.21.2.112:0/3865231986 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6608006860 0x7f6608008d20 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7f660c0063a0 tx=0x7f660c009aa0).ready entity=mon.0 client_cookie=40a62149b30f84a8 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:31.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:31.346 7f661961a700 1 -- 172.21.2.112:0/3865231986 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f660c00c340 con 0x7f6608006860 2021-02-10T11:15:31.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:31.346 7f661961a700 1 -- 172.21.2.112:0/3865231986 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f660c00c6c0 con 0x7f6608006860 2021-02-10T11:15:31.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:31.346 7f661961a700 1 -- 172.21.2.112:0/3865231986 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f660c00cbe0 con 0x7f6608006860 2021-02-10T11:15:31.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:31.346 7f6620842700 1 -- 172.21.2.112:0/3865231986 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6608006860 msgr2=0x7f6608008d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:31.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:31.346 7f6620842700 1 --2- 172.21.2.112:0/3865231986 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6608006860 0x7f6608008d20 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7f660c0063a0 tx=0x7f660c009aa0).stop 2021-02-10T11:15:31.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:31.346 7f661ae1d700 1 -- 172.21.2.112:0/3865231986 reap_dead start 2021-02-10T11:15:31.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:31.347 7f6620842700 1 -- 172.21.2.112:0/3865231986 shutdown_connections 2021-02-10T11:15:31.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:31.347 7f6620842700 1 -- 172.21.2.112:0/3865231986 >> 172.21.2.112:0/3865231986 conn(0x7f661c0ce3e0 msgr2=0x7f661c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:31.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:31.347 7f6620842700 1 -- 172.21.2.112:0/3865231986 shutdown_connections 2021-02-10T11:15:31.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:31.347 7f6620842700 1 -- 172.21.2.112:0/3865231986 wait complete. 2021-02-10T11:15:31.365 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:31.361 7f6620842700 1 Processor -- start 2021-02-10T11:15:31.365 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:31.361 7f6620842700 1 -- start start 2021-02-10T11:15:31.365 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:31.362 7f6620842700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f661c0924b0 0x7f661c1a3590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:31.365 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:31.362 7f6620842700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f661c1a3bb0 0x7f661c0bf920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:31.365 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:31.362 7f6620842700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f661c0bff40 0x7f661c0c0340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:31.365 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:31.362 7f6620842700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f661c09a0a0 con 0x7f661c0bff40 2021-02-10T11:15:31.365 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:31.362 7f6620842700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f661c0c0960 con 0x7f661c1a3bb0 2021-02-10T11:15:31.365 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:31.362 7f6620842700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f661c098420 con 0x7f661c0924b0 2021-02-10T11:15:31.365 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:31.362 7f661ae1d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f661c0bff40 0x7f661c0c0340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:31.365 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:31.362 7f661ae1d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f661c0bff40 0x7f661c0c0340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:43428/0 (socket says 172.21.2.112:43428) 2021-02-10T11:15:31.365 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:31.362 7f661ae1d700 1 -- 172.21.2.112:0/2400149857 learned_addr learned my addr 172.21.2.112:0/2400149857 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:15:31.365 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:31.363 7f661ae1d700 1 -- 172.21.2.112:0/2400149857 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f661c0924b0 msgr2=0x7f661c1a3590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:31.365 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:31.363 7f661a61c700 1 --2- 172.21.2.112:0/2400149857 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f661c0924b0 0x7f661c1a3590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:31.366 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:31.363 7f6619e1b700 1 --2- 172.21.2.112:0/2400149857 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f661c1a3bb0 0x7f661c0bf920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:31.366 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:31.363 7f661ae1d700 1 --2- 172.21.2.112:0/2400149857 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f661c0924b0 0x7f661c1a3590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:31.366 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:31.363 7f661ae1d700 1 -- 172.21.2.112:0/2400149857 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f661c1a3bb0 msgr2=0x7f661c0bf920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:31.366 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:31.363 7f661ae1d700 1 --2- 172.21.2.112:0/2400149857 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f661c1a3bb0 0x7f661c0bf920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:31.366 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:31.363 7f661ae1d700 1 -- 172.21.2.112:0/2400149857 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f661c098640 con 0x7f661c0bff40 2021-02-10T11:15:31.366 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:31.363 7f661a61c700 1 --2- 172.21.2.112:0/2400149857 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f661c0924b0 0x7f661c1a3590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:15:31.366 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:31.363 7f6619e1b700 1 --2- 172.21.2.112:0/2400149857 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f661c1a3bb0 0x7f661c0bf920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:15:31.366 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:31.363 7f661ae1d700 1 --2- 172.21.2.112:0/2400149857 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f661c0bff40 0x7f661c0c0340 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0x7f6614006fc0 tx=0x7f661400b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:31.366 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:31.363 7f6612ffd700 1 -- 172.21.2.112:0/2400149857 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f661400bd80 con 0x7f661c0bff40 2021-02-10T11:15:31.366 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:31.363 7f6612ffd700 1 -- 172.21.2.112:0/2400149857 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6614007620 con 0x7f661c0bff40 2021-02-10T11:15:31.366 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:31.363 7f6612ffd700 1 -- 172.21.2.112:0/2400149857 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6614007a80 con 0x7f661c0bff40 2021-02-10T11:15:31.366 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:31.363 7f6620842700 1 -- 172.21.2.112:0/2400149857 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f661c098640 con 0x7f661c0bff40 2021-02-10T11:15:31.366 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:31.363 7f6620842700 1 -- 172.21.2.112:0/2400149857 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f661c098bd0 con 0x7f661c0bff40 2021-02-10T11:15:31.367 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:31.364 7f6612ffd700 1 -- 172.21.2.112:0/2400149857 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6614006020 con 0x7f661c0bff40 2021-02-10T11:15:31.367 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:31.364 7f6612ffd700 1 --2- 172.21.2.112:0/2400149857 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f66000186c0 0x7f660001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:31.367 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:31.364 7f6612ffd700 1 -- 172.21.2.112:0/2400149857 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f6614032060 con 0x7f661c0bff40 2021-02-10T11:15:31.367 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:31.365 7f661a61c700 1 --2- 172.21.2.112:0/2400149857 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f66000186c0 0x7f660001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:31.367 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:31.365 7f661a61c700 1 --2- 172.21.2.112:0/2400149857 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f66000186c0 0x7f660001abc0 crc :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:31.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:31.394 7f6620842700 1 -- 172.21.2.112:0/2400149857 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f661c105b40 con 0x7f661c0bff40 2021-02-10T11:15:31.394 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:31.396 7f6612ffd700 1 -- 172.21.2.112:0/2400149857 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6614027eb0 con 0x7f661c0bff40 2021-02-10T11:15:31.520 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:31.522 7f6620842700 1 -- 172.21.2.112:0/2400149857 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"} v 0) v1 -- 0x7f661c0f0b50 con 0x7f661c0bff40 2021-02-10T11:15:32.086 DEBUG:teuthology.orchestra.run.gibba012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:15:32.089 DEBUG:teuthology.orchestra.run.gibba020:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:15:32.293 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:32.295 7f6612ffd700 1 -- 172.21.2.112:0/2400149857 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"}]=0 pool 'cephfs_metadata' created v26) v1 ==== 133+0+0 (secure 0 0 0) 0x7f6614006020 con 0x7f661c0bff40 2021-02-10T11:15:32.337 INFO:teuthology.orchestra.run.gibba012.stderr:pool 'cephfs_metadata' created 2021-02-10T11:15:32.339 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:32.341 7f6620842700 1 -- 172.21.2.112:0/2400149857 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f66000186c0 msgr2=0x7f660001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:32.339 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:32.341 7f6620842700 1 --2- 172.21.2.112:0/2400149857 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f66000186c0 0x7f660001abc0 crc :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:32.339 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:32.341 7f6620842700 1 -- 172.21.2.112:0/2400149857 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f661c0bff40 msgr2=0x7f661c0c0340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:32.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:32.341 7f6620842700 1 --2- 172.21.2.112:0/2400149857 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f661c0bff40 0x7f661c0c0340 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0x7f6614006fc0 tx=0x7f661400b990).stop 2021-02-10T11:15:32.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:32.341 7f6620842700 1 -- 172.21.2.112:0/2400149857 shutdown_connections 2021-02-10T11:15:32.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:32.341 7f6620842700 1 --2- 172.21.2.112:0/2400149857 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f66000186c0 0x7f660001abc0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:32.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:32.341 7f6620842700 1 --2- 172.21.2.112:0/2400149857 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f661c1a3bb0 0x7f661c0bf920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:32.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:32.341 7f6620842700 1 --2- 172.21.2.112:0/2400149857 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f661c0bff40 0x7f661c0c0340 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:32.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:32.341 7f6620842700 1 --2- 172.21.2.112:0/2400149857 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f661c0924b0 0x7f661c1a3590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:32.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:32.341 7f6620842700 1 -- 172.21.2.112:0/2400149857 >> 172.21.2.112:0/2400149857 conn(0x7f661c0b50c0 msgr2=0x7f661c092d90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:32.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:32.341 7f6620842700 1 -- 172.21.2.112:0/2400149857 shutdown_connections 2021-02-10T11:15:32.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:32.341 7f6620842700 1 -- 172.21.2.112:0/2400149857 wait complete. 2021-02-10T11:15:32.348 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_data 8 2021-02-10T11:15:32.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:32.761 7f16a233d700 1 Processor -- start 2021-02-10T11:15:32.777 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:32.761 7f16a233d700 1 -- start start 2021-02-10T11:15:32.777 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:32.761 7f16a233d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f169c1b0470 0x7f169c1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:32.777 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:32.761 7f16a233d700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f169c0ef610 0x7f169c0efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:32.777 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:32.761 7f16a233d700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f169c0f0b50 con 0x7f169c0f0030 2021-02-10T11:15:32.777 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:32.761 7f16a233d700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f169c0e52e0 con 0x7f169c0ef610 2021-02-10T11:15:32.777 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:32.761 7f16a233d700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f169c0e5490 con 0x7f169c1b0470 2021-02-10T11:15:32.777 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:32.761 7f16a095a700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f169c0f0030 0x7f169c0f10d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:42896/0 (socket says 172.21.2.112:42896) 2021-02-10T11:15:32.777 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:32.761 7f16a095a700 1 -- 172.21.2.112:0/1004906503 learned_addr learned my addr 172.21.2.112:0/1004906503 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:15:32.777 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:32.762 7f169affd700 1 -- 172.21.2.112:0/1004906503 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2704499574 0 0) 0x7f1694001500 con 0x7f169c0f0030 2021-02-10T11:15:32.777 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:32.762 7f169affd700 1 -- 172.21.2.112:0/1004906503 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1684002fe0 con 0x7f169c0f0030 2021-02-10T11:15:32.777 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:32.762 7f169b7fe700 1 --2- 172.21.2.112:0/1004906503 >> v2:172.21.2.120:3300/0 conn(0x7f169c0ef610 0x7f169c0efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:32.777 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:32.762 7f169bfff700 1 --2- 172.21.2.112:0/1004906503 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f169c1b0470 0x7f169c1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:32.778 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:32.762 7f169affd700 1 -- 172.21.2.112:0/1004906503 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 82441083 0 0) 0x7f1694001500 con 0x7f169c0f0030 2021-02-10T11:15:32.778 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:32.762 7f169affd700 1 -- 172.21.2.112:0/1004906503 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f169c1b0470 msgr2=0x7f169c1b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:15:32.778 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:32.762 7f169affd700 1 --2- 172.21.2.112:0/1004906503 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f169c1b0470 0x7f169c1b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:32.778 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:32.762 7f169affd700 1 -- 172.21.2.112:0/1004906503 >> v2:172.21.2.120:3300/0 conn(0x7f169c0ef610 msgr2=0x7f169c0efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:15:32.778 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:32.762 7f169affd700 1 --2- 172.21.2.112:0/1004906503 >> v2:172.21.2.120:3300/0 conn(0x7f169c0ef610 0x7f169c0efa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:32.778 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:32.762 7f169affd700 1 -- 172.21.2.112:0/1004906503 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f169c0f0430 con 0x7f169c0f0030 2021-02-10T11:15:32.778 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:32.762 7f169affd700 1 -- 172.21.2.112:0/1004906503 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f1694001a00 con 0x7f169c0f0030 2021-02-10T11:15:32.778 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:32.762 7f169affd700 1 -- 172.21.2.112:0/1004906503 >> v1:172.21.2.112:6789/0 conn(0x7f169c0f0030 legacy=0x7f169c0f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:32.778 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:32.762 7f169affd700 1 --2- 172.21.2.112:0/1004906503 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1684003a60 0x7f1684005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:32.778 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:32.762 7f169affd700 1 --2- 172.21.2.112:0/1004906503 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f168404a7a0 0x7f168404cc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:32.778 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:32.762 7f169affd700 1 --2- 172.21.2.112:0/1004906503 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f168404d510 0x7f168404fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:32.778 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:32.762 7f169affd700 1 -- 172.21.2.112:0/1004906503 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16840502e0 con 0x7f168404d510 2021-02-10T11:15:32.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:32.762 7f169affd700 1 -- 172.21.2.112:0/1004906503 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1684050500 con 0x7f1684003a60 2021-02-10T11:15:32.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:32.762 7f169affd700 1 -- 172.21.2.112:0/1004906503 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1684050720 con 0x7f168404a7a0 2021-02-10T11:15:32.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:32.763 7f16a095a700 1 --2- 172.21.2.112:0/1004906503 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f168404d510 0x7f168404fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:32.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:32.763 7f169bfff700 1 --2- 172.21.2.112:0/1004906503 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1684003a60 0x7f1684005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:32.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:32.763 7f16a095a700 1 -- 172.21.2.112:0/1004906503 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f168404a7a0 msgr2=0x7f168404cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:15:32.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:32.763 7f16a095a700 1 --2- 172.21.2.112:0/1004906503 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f168404a7a0 0x7f168404cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:32.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:32.763 7f16a095a700 1 -- 172.21.2.112:0/1004906503 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1684003a60 msgr2=0x7f1684005ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:15:32.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:32.763 7f16a095a700 1 --2- 172.21.2.112:0/1004906503 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1684003a60 0x7f1684005ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:32.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:32.763 7f16a095a700 1 -- 172.21.2.112:0/1004906503 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16840509c0 con 0x7f168404d510 2021-02-10T11:15:32.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:32.763 7f16a095a700 1 --2- 172.21.2.112:0/1004906503 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f168404d510 0x7f168404fa30 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0x7f1694000f00 tx=0x7f169400a310).ready entity=mon.0 client_cookie=ff81761e1ae1988a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:32.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:32.763 7f169affd700 1 -- 172.21.2.112:0/1004906503 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f16940070c0 con 0x7f168404d510 2021-02-10T11:15:32.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:32.763 7f169affd700 1 -- 172.21.2.112:0/1004906503 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1694007380 con 0x7f168404d510 2021-02-10T11:15:32.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:32.763 7f169affd700 1 -- 172.21.2.112:0/1004906503 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f169400ab90 con 0x7f168404d510 2021-02-10T11:15:32.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:32.763 7f16a233d700 1 -- 172.21.2.112:0/1004906503 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f168404d510 msgr2=0x7f168404fa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:32.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:32.763 7f16a233d700 1 --2- 172.21.2.112:0/1004906503 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f168404d510 0x7f168404fa30 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0x7f1694000f00 tx=0x7f169400a310).stop 2021-02-10T11:15:32.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:32.763 7f16a095a700 1 -- 172.21.2.112:0/1004906503 reap_dead start 2021-02-10T11:15:32.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:32.764 7f16a233d700 1 -- 172.21.2.112:0/1004906503 shutdown_connections 2021-02-10T11:15:32.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:32.764 7f16a233d700 1 -- 172.21.2.112:0/1004906503 >> 172.21.2.112:0/1004906503 conn(0x7f169c0ce3e0 msgr2=0x7f169c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:32.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:32.764 7f16a233d700 1 -- 172.21.2.112:0/1004906503 shutdown_connections 2021-02-10T11:15:32.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:32.764 7f16a233d700 1 -- 172.21.2.112:0/1004906503 wait complete. 2021-02-10T11:15:32.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:32.778 7f16a233d700 1 Processor -- start 2021-02-10T11:15:32.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:32.778 7f16a233d700 1 -- start start 2021-02-10T11:15:32.782 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:32.779 7f16a233d700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f169c09b930 0x7f169c0b7c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:32.782 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:32.779 7f16a233d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f169c0b8410 0x7f169c0b8810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:32.782 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:32.779 7f16a233d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f169c0c1f80 0x7f169c0c2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:32.782 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:32.779 7f16a233d700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f169c0c1a00 con 0x7f169c0c1f80 2021-02-10T11:15:32.782 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:32.779 7f16a233d700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f169c0b8e30 con 0x7f169c09b930 2021-02-10T11:15:32.782 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:32.779 7f16a233d700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f169c0b9050 con 0x7f169c0b8410 2021-02-10T11:15:32.782 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:32.779 7f16a095a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f169c0c1f80 0x7f169c0c2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:32.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:32.779 7f16a095a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f169c0c1f80 0x7f169c0c2380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:43448/0 (socket says 172.21.2.112:43448) 2021-02-10T11:15:32.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:32.779 7f16a095a700 1 -- 172.21.2.112:0/329630890 learned_addr learned my addr 172.21.2.112:0/329630890 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:15:32.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:32.779 7f169b7fe700 1 --2- 172.21.2.112:0/329630890 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f169c0b8410 0x7f169c0b8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:32.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:32.779 7f16a095a700 1 -- 172.21.2.112:0/329630890 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f169c0b8410 msgr2=0x7f169c0b8810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:32.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:32.779 7f169bfff700 1 --2- 172.21.2.112:0/329630890 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f169c09b930 0x7f169c0b7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:32.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:32.779 7f16a095a700 1 --2- 172.21.2.112:0/329630890 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f169c0b8410 0x7f169c0b8810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:32.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:32.779 7f16a095a700 1 -- 172.21.2.112:0/329630890 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f169c09b930 msgr2=0x7f169c0b7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:32.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:32.779 7f16a095a700 1 --2- 172.21.2.112:0/329630890 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f169c09b930 0x7f169c0b7c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:32.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:32.779 7f16a095a700 1 -- 172.21.2.112:0/329630890 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f169c0c29a0 con 0x7f169c0c1f80 2021-02-10T11:15:32.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:32.780 7f169bfff700 1 --2- 172.21.2.112:0/329630890 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f169c09b930 0x7f169c0b7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:15:32.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:32.779 7f169b7fe700 1 --2- 172.21.2.112:0/329630890 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f169c0b8410 0x7f169c0b8810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:15:32.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:32.780 7f16a095a700 1 --2- 172.21.2.112:0/329630890 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f169c0c1f80 0x7f169c0c2380 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7f16940012f0 tx=0x7f1694007250).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:32.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:32.780 7f1698ff9700 1 -- 172.21.2.112:0/329630890 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1694007390 con 0x7f169c0c1f80 2021-02-10T11:15:32.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:32.780 7f1698ff9700 1 -- 172.21.2.112:0/329630890 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1694007610 con 0x7f169c0c1f80 2021-02-10T11:15:32.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:32.780 7f1698ff9700 1 -- 172.21.2.112:0/329630890 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f16940049a0 con 0x7f169c0c1f80 2021-02-10T11:15:32.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:32.780 7f16a233d700 1 -- 172.21.2.112:0/329630890 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f169c0c29a0 con 0x7f169c0c1f80 2021-02-10T11:15:32.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:32.780 7f16a233d700 1 -- 172.21.2.112:0/329630890 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f169c0c29a0 con 0x7f169c0c1f80 2021-02-10T11:15:32.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:32.780 7f1698ff9700 1 -- 172.21.2.112:0/329630890 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1694004720 con 0x7f169c0c1f80 2021-02-10T11:15:32.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:32.781 7f1698ff9700 1 --2- 172.21.2.112:0/329630890 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f167c0186d0 0x7f167c01abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:32.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:32.781 7f1698ff9700 1 -- 172.21.2.112:0/329630890 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5107+0+0 (secure 0 0 0) 0x7f16940327f0 con 0x7f169c0c1f80 2021-02-10T11:15:32.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:32.781 7f169bfff700 1 --2- 172.21.2.112:0/329630890 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f167c0186d0 0x7f167c01abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:32.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:32.781 7f169bfff700 1 --2- 172.21.2.112:0/329630890 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f167c0186d0 0x7f167c01abd0 crc :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:32.809 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:32.811 7f16a233d700 1 -- 172.21.2.112:0/329630890 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f169c105b30 con 0x7f169c0c1f80 2021-02-10T11:15:32.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:32.814 7f1698ff9700 1 -- 172.21.2.112:0/329630890 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1694001ba0 con 0x7f169c0c1f80 2021-02-10T11:15:32.946 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:32.948 7f16a233d700 1 -- 172.21.2.112:0/329630890 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"} v 0) v1 -- 0x7f169c105b30 con 0x7f169c0c1f80 2021-02-10T11:15:33.296 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:33.298 7f1698ff9700 1 -- 172.21.2.112:0/329630890 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"}]=0 pool 'cephfs_data' created v27) v1 ==== 125+0+0 (secure 0 0 0) 0x7f1694004720 con 0x7f169c0c1f80 2021-02-10T11:15:33.311 INFO:teuthology.orchestra.run.gibba012.stderr:pool 'cephfs_data' created 2021-02-10T11:15:33.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:33.315 7f16a233d700 1 -- 172.21.2.112:0/329630890 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f167c0186d0 msgr2=0x7f167c01abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:33.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:33.315 7f16a233d700 1 --2- 172.21.2.112:0/329630890 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f167c0186d0 0x7f167c01abd0 crc :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:33.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:33.315 7f16a233d700 1 -- 172.21.2.112:0/329630890 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f169c0c1f80 msgr2=0x7f169c0c2380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:33.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:33.315 7f16a233d700 1 --2- 172.21.2.112:0/329630890 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f169c0c1f80 0x7f169c0c2380 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7f16940012f0 tx=0x7f1694007250).stop 2021-02-10T11:15:33.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:33.315 7f16a233d700 1 -- 172.21.2.112:0/329630890 shutdown_connections 2021-02-10T11:15:33.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:33.315 7f16a233d700 1 --2- 172.21.2.112:0/329630890 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f167c0186d0 0x7f167c01abd0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:33.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:33.315 7f16a233d700 1 --2- 172.21.2.112:0/329630890 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f169c0c1f80 0x7f169c0c2380 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:33.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:33.315 7f16a233d700 1 --2- 172.21.2.112:0/329630890 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f169c0b8410 0x7f169c0b8810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:33.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:33.315 7f16a233d700 1 --2- 172.21.2.112:0/329630890 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f169c09b930 0x7f169c0b7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:33.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:33.315 7f16a233d700 1 -- 172.21.2.112:0/329630890 >> 172.21.2.112:0/329630890 conn(0x7f169c094e10 msgr2=0x7f169c0a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:33.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:33.315 7f16a233d700 1 -- 172.21.2.112:0/329630890 shutdown_connections 2021-02-10T11:15:33.314 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:33.315 7f16a233d700 1 -- 172.21.2.112:0/329630890 wait complete. 2021-02-10T11:15:33.321 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2021-02-10T11:15:33.750 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:33.735 7f98f0cb2700 1 Processor -- start 2021-02-10T11:15:33.750 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:33.735 7f98f0cb2700 1 -- start start 2021-02-10T11:15:33.751 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:33.735 7f98f0cb2700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f98ec0d1870 0x7f98ec1b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:33.751 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:33.735 7f98f0cb2700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f98ec0cff70 0x7f98ec109130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:33.751 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:33.735 7f98f0cb2700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98ec109a60 con 0x7f98ec1b0ae0 2021-02-10T11:15:33.751 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:33.735 7f98f0cb2700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98ec109750 con 0x7f98ec0cff70 2021-02-10T11:15:33.751 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:33.735 7f98f0cb2700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98ec088020 con 0x7f98ec0d1870 2021-02-10T11:15:33.751 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:33.736 7f98ea05d700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f98ec1b0ae0 0x7f98ec108940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:42916/0 (socket says 172.21.2.112:42916) 2021-02-10T11:15:33.751 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:33.736 7f98ea05d700 1 -- 172.21.2.112:0/633166707 learned_addr learned my addr 172.21.2.112:0/633166707 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:15:33.751 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:33.736 7f98ea85e700 1 --2- 172.21.2.112:0/633166707 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f98ec0d1870 0x7f98ec1b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:33.751 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:33.736 7f98eb05f700 1 --2- 172.21.2.112:0/633166707 >> v2:172.21.2.120:3300/0 conn(0x7f98ec0cff70 0x7f98ec109130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:33.751 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:33.736 7f98e985c700 1 -- 172.21.2.112:0/633166707 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 573378841 0 0) 0x7f98dc0009b0 con 0x7f98ec1b0ae0 2021-02-10T11:15:33.751 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:33.736 7f98e985c700 1 -- 172.21.2.112:0/633166707 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98d8002fe0 con 0x7f98ec1b0ae0 2021-02-10T11:15:33.751 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:33.736 7f98ea85e700 1 -- 172.21.2.112:0/633166707 >> v2:172.21.2.120:3300/0 conn(0x7f98ec0cff70 msgr2=0x7f98ec109130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:15:33.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:33.736 7f98ea85e700 1 --2- 172.21.2.112:0/633166707 >> v2:172.21.2.120:3300/0 conn(0x7f98ec0cff70 0x7f98ec109130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:33.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:33.736 7f98ea85e700 1 -- 172.21.2.112:0/633166707 >> v1:172.21.2.112:6789/0 conn(0x7f98ec1b0ae0 legacy=0x7f98ec108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:33.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:33.736 7f98ea85e700 1 -- 172.21.2.112:0/633166707 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98ec088210 con 0x7f98ec0d1870 2021-02-10T11:15:33.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:33.736 7f98eb05f700 1 --2- 172.21.2.112:0/633166707 >> v2:172.21.2.120:3300/0 conn(0x7f98ec0cff70 0x7f98ec109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:15:33.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:33.737 7f98ea85e700 1 --2- 172.21.2.112:0/633166707 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f98ec0d1870 0x7f98ec1b04c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f98d4006650 tx=0x7f98d4009a80).ready entity=mon.2 client_cookie=a8b52ae7795597a9 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:33.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:33.737 7f98e985c700 1 -- 172.21.2.112:0/633166707 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f98d400c0c0 con 0x7f98ec0d1870 2021-02-10T11:15:33.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:33.737 7f98e985c700 1 -- 172.21.2.112:0/633166707 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98d400c3c0 con 0x7f98ec0d1870 2021-02-10T11:15:33.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:33.737 7f98e985c700 1 -- 172.21.2.112:0/633166707 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f98d40009b0 con 0x7f98ec0d1870 2021-02-10T11:15:33.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:33.737 7f98f0cb2700 1 -- 172.21.2.112:0/633166707 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f98ec0d1870 msgr2=0x7f98ec1b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:33.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:33.737 7f98f0cb2700 1 --2- 172.21.2.112:0/633166707 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f98ec0d1870 0x7f98ec1b04c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f98d4006650 tx=0x7f98d4009a80).stop 2021-02-10T11:15:33.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:33.738 7f98f0cb2700 1 -- 172.21.2.112:0/633166707 shutdown_connections 2021-02-10T11:15:33.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:33.738 7f98f0cb2700 1 --2- 172.21.2.112:0/633166707 >> v2:172.21.2.120:3300/0 conn(0x7f98ec0cff70 0x7f98ec109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:33.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:33.738 7f98f0cb2700 1 --2- 172.21.2.112:0/633166707 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f98ec0d1870 0x7f98ec1b04c0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:33.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:33.738 7f98f0cb2700 1 -- 172.21.2.112:0/633166707 >> 172.21.2.112:0/633166707 conn(0x7f98ec0ce420 msgr2=0x7f98ec0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:33.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:33.738 7f98f0cb2700 1 -- 172.21.2.112:0/633166707 shutdown_connections 2021-02-10T11:15:33.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:33.738 7f98f0cb2700 1 -- 172.21.2.112:0/633166707 wait complete. 2021-02-10T11:15:33.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:33.753 7f98f0cb2700 1 Processor -- start 2021-02-10T11:15:33.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:33.753 7f98f0cb2700 1 -- start start 2021-02-10T11:15:33.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:33.753 7f98f0cb2700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f98ec18db40 0x7f98ec18ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:33.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:33.753 7f98f0cb2700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f98ec18d490 0x7f98ec0aa190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:33.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:33.753 7f98f0cb2700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f98ec0aa7b0 0x7f98ec0837e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:33.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:33.753 7f98f0cb2700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98ec0942f0 con 0x7f98ec18d490 2021-02-10T11:15:33.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:33.753 7f98f0cb2700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98ec18e0c0 con 0x7f98ec0aa7b0 2021-02-10T11:15:33.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:33.753 7f98f0cb2700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98ec0aabb0 con 0x7f98ec18db40 2021-02-10T11:15:33.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:33.753 7f98ea05d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f98ec18d490 0x7f98ec0aa190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:33.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:33.753 7f98ea85e700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f98ec18db40 0x7f98ec18ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:33.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:33.753 7f98ea85e700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f98ec18db40 0x7f98ec18ce70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3301/0 says I am v2:172.21.2.112:43142/0 (socket says 172.21.2.112:43142) 2021-02-10T11:15:33.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:33.753 7f98ea85e700 1 -- 172.21.2.112:0/1911900140 learned_addr learned my addr 172.21.2.112:0/1911900140 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:15:33.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:33.753 7f98ea05d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f98ec18d490 0x7f98ec0aa190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:43462/0 (socket says 172.21.2.112:43462) 2021-02-10T11:15:33.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:33.754 7f98eb05f700 1 --2- 172.21.2.112:0/1911900140 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f98ec0aa7b0 0x7f98ec0837e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:33.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:33.754 7f98ea85e700 1 -- 172.21.2.112:0/1911900140 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f98ec0aa7b0 msgr2=0x7f98ec0837e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:33.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:33.754 7f98ea85e700 1 --2- 172.21.2.112:0/1911900140 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f98ec0aa7b0 0x7f98ec0837e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:33.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:33.754 7f98ea85e700 1 -- 172.21.2.112:0/1911900140 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f98ec18d490 msgr2=0x7f98ec0aa190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:33.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:33.754 7f98ea85e700 1 --2- 172.21.2.112:0/1911900140 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f98ec18d490 0x7f98ec0aa190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:33.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:33.754 7f98ea85e700 1 -- 172.21.2.112:0/1911900140 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98ec0aadd0 con 0x7f98ec18db40 2021-02-10T11:15:33.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:33.754 7f98ea85e700 1 --2- 172.21.2.112:0/1911900140 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f98ec18db40 0x7f98ec18ce70 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f98d4006610 tx=0x7f98d400b8e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:33.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:33.754 7f98e37fe700 1 -- 172.21.2.112:0/1911900140 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f98d400bcd0 con 0x7f98ec18db40 2021-02-10T11:15:33.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:33.754 7f98e37fe700 1 -- 172.21.2.112:0/1911900140 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98d40052e0 con 0x7f98ec18db40 2021-02-10T11:15:33.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:33.754 7f98e37fe700 1 -- 172.21.2.112:0/1911900140 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f98d40093a0 con 0x7f98ec18db40 2021-02-10T11:15:33.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:33.754 7f98f0cb2700 1 -- 172.21.2.112:0/1911900140 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98ec0aadd0 con 0x7f98ec18db40 2021-02-10T11:15:33.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:33.754 7f98f0cb2700 1 -- 172.21.2.112:0/1911900140 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98ec0aadd0 con 0x7f98ec18db40 2021-02-10T11:15:33.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:33.755 7f98e37fe700 1 -- 172.21.2.112:0/1911900140 <== mon.2 v2:172.21.2.120:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f98d4006030 con 0x7f98ec18db40 2021-02-10T11:15:33.757 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:33.755 7f98e37fe700 1 --2- 172.21.2.112:0/1911900140 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f98d00186c0 0x7f98d001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:33.757 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:33.755 7f98e37fe700 1 -- 172.21.2.112:0/1911900140 <== mon.2 v2:172.21.2.120:3301/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 5457+0+0 (secure 0 0 0) 0x7f98d4032390 con 0x7f98ec18db40 2021-02-10T11:15:33.757 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:33.755 7f98ea05d700 1 --2- 172.21.2.112:0/1911900140 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f98d00186c0 0x7f98d001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:33.757 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:33.755 7f98ea05d700 1 --2- 172.21.2.112:0/1911900140 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f98d00186c0 0x7f98d001abc0 crc :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:33.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:33.785 7f98f0cb2700 1 -- 172.21.2.112:0/1911900140 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98ec0828d0 con 0x7f98ec18db40 2021-02-10T11:15:33.786 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:33.788 7f98e37fe700 1 -- 172.21.2.112:0/1911900140 <== mon.2 v2:172.21.2.120:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f98d4030d30 con 0x7f98ec18db40 2021-02-10T11:15:33.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:33.918 7f98f0cb2700 1 -- 172.21.2.112:0/1911900140 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_command({"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"} v 0) v1 -- 0x7f98ec109a60 con 0x7f98ec18db40 2021-02-10T11:15:33.921 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:33.923 7f98e37fe700 1 -- 172.21.2.112:0/1911900140 <== mon.2 v2:172.21.2.120:3301/0 7 ==== mon_command_ack([{"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"}]=0 new fs with metadata pool 4 and data pool 5 v10) v1 ==== 187+0+0 (secure 0 0 0) 0x7f98d4006030 con 0x7f98ec18db40 2021-02-10T11:15:33.923 INFO:teuthology.orchestra.run.gibba012.stderr:new fs with metadata pool 4 and data pool 5 2021-02-10T11:15:33.925 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:33.927 7f98f0cb2700 1 -- 172.21.2.112:0/1911900140 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f98d00186c0 msgr2=0x7f98d001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:33.925 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:33.927 7f98f0cb2700 1 --2- 172.21.2.112:0/1911900140 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f98d00186c0 0x7f98d001abc0 crc :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:33.925 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:33.927 7f98f0cb2700 1 -- 172.21.2.112:0/1911900140 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f98ec18db40 msgr2=0x7f98ec18ce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:33.925 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:33.927 7f98f0cb2700 1 --2- 172.21.2.112:0/1911900140 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f98ec18db40 0x7f98ec18ce70 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f98d4006610 tx=0x7f98d400b8e0).stop 2021-02-10T11:15:33.925 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:33.927 7f98f0cb2700 1 -- 172.21.2.112:0/1911900140 shutdown_connections 2021-02-10T11:15:33.926 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:33.927 7f98f0cb2700 1 --2- 172.21.2.112:0/1911900140 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f98d00186c0 0x7f98d001abc0 crc :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:33.926 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:33.927 7f98f0cb2700 1 --2- 172.21.2.112:0/1911900140 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f98ec0aa7b0 0x7f98ec0837e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:33.926 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:33.927 7f98f0cb2700 1 --2- 172.21.2.112:0/1911900140 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f98ec18d490 0x7f98ec0aa190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:33.926 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:33.927 7f98f0cb2700 1 --2- 172.21.2.112:0/1911900140 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f98ec18db40 0x7f98ec18ce70 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:33.926 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:33.927 7f98f0cb2700 1 -- 172.21.2.112:0/1911900140 >> 172.21.2.112:0/1911900140 conn(0x7f98ec19c540 msgr2=0x7f98ec082b00 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:33.927 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:33.929 7f98f0cb2700 1 -- 172.21.2.112:0/1911900140 shutdown_connections 2021-02-10T11:15:33.927 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:33.929 7f98f0cb2700 1 -- 172.21.2.112:0/1911900140 wait complete. 2021-02-10T11:15:33.935 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:15:34.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.346 7f9f6f916700 1 Processor -- start 2021-02-10T11:15:34.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.346 7f9f6f916700 1 -- start start 2021-02-10T11:15:34.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.347 7f9f6f916700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f9f680e5070 0x7f9f681b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:34.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.347 7f9f6f916700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9f681b3e40 0x7f9f6807d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:34.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.347 7f9f6f916700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f9f680e5470 0x7f9f6807dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:34.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.347 7f9f6f916700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f6807ed80 con 0x7f9f680e5470 2021-02-10T11:15:34.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.347 7f9f6f916700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f6807e2d0 con 0x7f9f680e5070 2021-02-10T11:15:34.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.347 7f9f6f916700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f6807e4c0 con 0x7f9f681b3e40 2021-02-10T11:15:34.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.347 7f9f6df33700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f9f680e5470 0x7f9f6807dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:34.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.347 7f9f6df33700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f9f680e5470 0x7f9f6807dcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:43468/0 (socket says 172.21.2.112:43468) 2021-02-10T11:15:34.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.347 7f9f6df33700 1 -- 172.21.2.112:0/3695054810 learned_addr learned my addr 172.21.2.112:0/3695054810 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:15:34.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.347 7f9f6d732700 1 --2- 172.21.2.112:0/3695054810 >> v2:172.21.2.120:3300/0 conn(0x7f9f680e5070 0x7f9f681b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:34.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.347 7f9f6cf31700 1 --2- 172.21.2.112:0/3695054810 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9f681b3e40 0x7f9f6807d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:34.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.348 7f9f6df33700 1 -- 172.21.2.112:0/3695054810 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9f681b3e40 msgr2=0x7f9f6807d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:15:34.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.348 7f9f6df33700 1 --2- 172.21.2.112:0/3695054810 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9f681b3e40 0x7f9f6807d690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:34.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.348 7f9f6df33700 1 -- 172.21.2.112:0/3695054810 >> v2:172.21.2.120:3300/0 conn(0x7f9f680e5070 msgr2=0x7f9f681b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:15:34.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.348 7f9f6df33700 1 --2- 172.21.2.112:0/3695054810 >> v2:172.21.2.120:3300/0 conn(0x7f9f680e5070 0x7f9f681b3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:34.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.348 7f9f6df33700 1 -- 172.21.2.112:0/3695054810 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f6807e6e0 con 0x7f9f680e5470 2021-02-10T11:15:34.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.348 7f9f6df33700 1 --2- 172.21.2.112:0/3695054810 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9f680e5470 0x7f9f6807dcb0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7f9f60006f00 tx=0x7f9f60009f10).ready entity=mon.0 client_cookie=41d4a70b58f4344d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:34.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.348 7f9f5ffff700 1 -- 172.21.2.112:0/3695054810 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9f6000b3b0 con 0x7f9f680e5470 2021-02-10T11:15:34.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.348 7f9f5ffff700 1 -- 172.21.2.112:0/3695054810 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f6000b730 con 0x7f9f680e5470 2021-02-10T11:15:34.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.348 7f9f5ffff700 1 -- 172.21.2.112:0/3695054810 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9f6000bc50 con 0x7f9f680e5470 2021-02-10T11:15:34.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.348 7f9f6f916700 1 -- 172.21.2.112:0/3695054810 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9f680e5470 msgr2=0x7f9f6807dcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:34.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.348 7f9f6f916700 1 --2- 172.21.2.112:0/3695054810 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9f680e5470 0x7f9f6807dcb0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7f9f60006f00 tx=0x7f9f60009f10).stop 2021-02-10T11:15:34.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.349 7f9f6f916700 1 -- 172.21.2.112:0/3695054810 shutdown_connections 2021-02-10T11:15:34.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.349 7f9f6f916700 1 --2- 172.21.2.112:0/3695054810 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9f680e5470 0x7f9f6807dcb0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:34.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.349 7f9f6f916700 1 --2- 172.21.2.112:0/3695054810 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9f681b3e40 0x7f9f6807d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:34.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.349 7f9f6f916700 1 --2- 172.21.2.112:0/3695054810 >> v2:172.21.2.120:3300/0 conn(0x7f9f680e5070 0x7f9f681b3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:34.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.349 7f9f6f916700 1 -- 172.21.2.112:0/3695054810 >> 172.21.2.112:0/3695054810 conn(0x7f9f680ce690 msgr2=0x7f9f680e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:34.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.349 7f9f6f916700 1 -- 172.21.2.112:0/3695054810 shutdown_connections 2021-02-10T11:15:34.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.349 7f9f6f916700 1 -- 172.21.2.112:0/3695054810 wait complete. 2021-02-10T11:15:34.365 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.364 7f9f6f916700 1 Processor -- start 2021-02-10T11:15:34.366 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.364 7f9f6f916700 1 -- start start 2021-02-10T11:15:34.366 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.364 7f9f6f916700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9f68191bb0 0x7f9f680a2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:34.366 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.364 7f9f6f916700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9f680a29a0 0x7f9f680a2da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:34.366 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.364 7f9f6f916700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9f680a33c0 0x7f9f680a03d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:34.366 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.364 7f9f6f916700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f680adf90 con 0x7f9f680a33c0 2021-02-10T11:15:34.366 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.364 7f9f6f916700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f680a1e00 con 0x7f9f680a29a0 2021-02-10T11:15:34.367 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.364 7f9f6f916700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f680a09f0 con 0x7f9f68191bb0 2021-02-10T11:15:34.367 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.365 7f9f6d732700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9f68191bb0 0x7f9f680a2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:34.367 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.365 7f9f6d732700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9f68191bb0 0x7f9f680a2380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3301/0 says I am v2:172.21.2.112:43154/0 (socket says 172.21.2.112:43154) 2021-02-10T11:15:34.368 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.365 7f9f6d732700 1 -- 172.21.2.112:0/469739019 learned_addr learned my addr 172.21.2.112:0/469739019 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:15:34.368 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.365 7f9f6cf31700 1 --2- 172.21.2.112:0/469739019 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9f680a29a0 0x7f9f680a2da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:34.368 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.365 7f9f6df33700 1 --2- 172.21.2.112:0/469739019 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9f680a33c0 0x7f9f680a03d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:34.368 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.365 7f9f6d732700 1 -- 172.21.2.112:0/469739019 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9f680a29a0 msgr2=0x7f9f680a2da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:34.368 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.365 7f9f6d732700 1 --2- 172.21.2.112:0/469739019 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9f680a29a0 0x7f9f680a2da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:34.368 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.365 7f9f6d732700 1 -- 172.21.2.112:0/469739019 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9f680a33c0 msgr2=0x7f9f680a03d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:34.368 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.365 7f9f6d732700 1 --2- 172.21.2.112:0/469739019 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9f680a33c0 0x7f9f680a03d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:34.368 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.365 7f9f6d732700 1 -- 172.21.2.112:0/469739019 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f680a0bd0 con 0x7f9f68191bb0 2021-02-10T11:15:34.368 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.365 7f9f6d732700 1 --2- 172.21.2.112:0/469739019 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9f68191bb0 0x7f9f680a2380 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f9f64006f60 tx=0x7f9f6400b970).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:34.368 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.365 7f9f5dffb700 1 -- 172.21.2.112:0/469739019 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9f6400bd60 con 0x7f9f68191bb0 2021-02-10T11:15:34.368 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.365 7f9f5dffb700 1 -- 172.21.2.112:0/469739019 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f64000c90 con 0x7f9f68191bb0 2021-02-10T11:15:34.368 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.365 7f9f6f916700 1 -- 172.21.2.112:0/469739019 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f680a0bd0 con 0x7f9f68191bb0 2021-02-10T11:15:34.369 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.366 7f9f6f916700 1 -- 172.21.2.112:0/469739019 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f680a1160 con 0x7f9f68191bb0 2021-02-10T11:15:34.369 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.366 7f9f5dffb700 1 -- 172.21.2.112:0/469739019 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9f64002220 con 0x7f9f68191bb0 2021-02-10T11:15:34.369 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.366 7f9f5dffb700 1 -- 172.21.2.112:0/469739019 <== mon.2 v2:172.21.2.120:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9f640052e0 con 0x7f9f68191bb0 2021-02-10T11:15:34.369 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.366 7f9f5dffb700 1 --2- 172.21.2.112:0/469739019 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9f48018730 0x7f9f4801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:34.369 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.366 7f9f5dffb700 1 -- 172.21.2.112:0/469739019 <== mon.2 v2:172.21.2.120:3301/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f9f640327b0 con 0x7f9f68191bb0 2021-02-10T11:15:34.369 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.366 7f9f6cf31700 1 --2- 172.21.2.112:0/469739019 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9f48018730 0x7f9f4801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:34.369 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.367 7f9f6cf31700 1 --2- 172.21.2.112:0/469739019 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9f48018730 0x7f9f4801abc0 crc :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:34.395 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.397 7f9f6f916700 1 -- 172.21.2.112:0/469739019 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f6807ed80 con 0x7f9f68191bb0 2021-02-10T11:15:34.398 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.400 7f9f5dffb700 1 -- 172.21.2.112:0/469739019 <== mon.2 v2:172.21.2.120:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9f64000960 con 0x7f9f68191bb0 2021-02-10T11:15:34.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.523 7f9f6f916700 1 -- 172.21.2.112:0/469739019 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9f680adf90 con 0x7f9f68191bb0 2021-02-10T11:15:34.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.525 7f9f5dffb700 1 -- 172.21.2.112:0/469739019 <== mon.2 v2:172.21.2.120:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v28) v1 ==== 74+0+14006 (secure 0 0 0) 0x7f9f640052e0 con 0x7f9f68191bb0 2021-02-10T11:15:34.524 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:15:34.525 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":28,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:15:33.919688","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:15:31.539720","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":5,"pool_name":"cephfs_data","create_time":"2021-02-10 11:15:32.965643","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:15:34.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.528 7f9f6f916700 1 -- 172.21.2.112:0/469739019 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9f48018730 msgr2=0x7f9f4801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:34.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.528 7f9f6f916700 1 --2- 172.21.2.112:0/469739019 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9f48018730 0x7f9f4801abc0 crc :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:34.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.528 7f9f6f916700 1 -- 172.21.2.112:0/469739019 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9f68191bb0 msgr2=0x7f9f680a2380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:34.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.528 7f9f6f916700 1 --2- 172.21.2.112:0/469739019 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9f68191bb0 0x7f9f680a2380 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f9f64006f60 tx=0x7f9f6400b970).stop 2021-02-10T11:15:34.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.528 7f9f6f916700 1 -- 172.21.2.112:0/469739019 shutdown_connections 2021-02-10T11:15:34.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.528 7f9f6f916700 1 --2- 172.21.2.112:0/469739019 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9f48018730 0x7f9f4801abc0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:34.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.528 7f9f6f916700 1 --2- 172.21.2.112:0/469739019 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9f680a29a0 0x7f9f680a2da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:34.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.528 7f9f6f916700 1 --2- 172.21.2.112:0/469739019 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9f680a33c0 0x7f9f680a03d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:34.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.528 7f9f6f916700 1 --2- 172.21.2.112:0/469739019 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9f68191bb0 0x7f9f680a2380 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:34.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.528 7f9f6f916700 1 -- 172.21.2.112:0/469739019 >> 172.21.2.112:0/469739019 conn(0x7f9f681a31d0 msgr2=0x7f9f6808b830 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:34.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.528 7f9f6f916700 1 -- 172.21.2.112:0/469739019 shutdown_connections 2021-02-10T11:15:34.528 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.528 7f9f6f916700 1 -- 172.21.2.112:0/469739019 wait complete. 2021-02-10T11:15:34.535 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:15:34.961 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.947 7f74adcdf700 1 Processor -- start 2021-02-10T11:15:34.961 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.947 7f74adcdf700 1 -- start start 2021-02-10T11:15:34.962 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.947 7f74adcdf700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f74a80d17f0 0x7f74a81b31e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:34.962 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.947 7f74adcdf700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f74a80d1bf0 0x7f74a81b3800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:34.962 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.947 7f74adcdf700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f74a81b3e20 0x7f74a80ceef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:34.962 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.947 7f74adcdf700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74a80e5eb0 con 0x7f74a80d17f0 2021-02-10T11:15:34.962 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.947 7f74adcdf700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74a80f0b20 con 0x7f74a80d1bf0 2021-02-10T11:15:34.962 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.947 7f74adcdf700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74a80cf510 con 0x7f74a81b3e20 2021-02-10T11:15:34.962 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.948 7f74a7fff700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f74a81b3e20 0x7f74a80ceef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:34.962 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.948 7f74a6ffd700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f74a80d1bf0 0x7f74a81b3800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:34.962 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.948 7f74a77fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f74a80d17f0 0x7f74a81b31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:34.962 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.948 7f74a6ffd700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f74a80d1bf0 0x7f74a81b3800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3300/0 says I am v2:172.21.2.112:45760/0 (socket says 172.21.2.112:45760) 2021-02-10T11:15:34.962 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.948 7f74a7fff700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f74a81b3e20 0x7f74a80ceef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3301/0 says I am v2:172.21.2.112:43162/0 (socket says 172.21.2.112:43162) 2021-02-10T11:15:34.963 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.948 7f74a6ffd700 1 -- 172.21.2.112:0/1020491353 learned_addr learned my addr 172.21.2.112:0/1020491353 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:15:34.963 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.948 7f74a77fe700 1 -- 172.21.2.112:0/1020491353 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f74a81b3e20 msgr2=0x7f74a80ceef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:15:34.963 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.948 7f74a77fe700 1 --2- 172.21.2.112:0/1020491353 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f74a81b3e20 0x7f74a80ceef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:34.963 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.948 7f74a77fe700 1 -- 172.21.2.112:0/1020491353 >> v2:172.21.2.120:3300/0 conn(0x7f74a80d1bf0 msgr2=0x7f74a81b3800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:15:34.963 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.948 7f74a77fe700 1 --2- 172.21.2.112:0/1020491353 >> v2:172.21.2.120:3300/0 conn(0x7f74a80d1bf0 0x7f74a81b3800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:34.963 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.948 7f74a77fe700 1 -- 172.21.2.112:0/1020491353 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74a80cf6f0 con 0x7f74a80d17f0 2021-02-10T11:15:34.963 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.948 7f74a77fe700 1 --2- 172.21.2.112:0/1020491353 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f74a80d17f0 0x7f74a81b31e0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7f7498006650 tx=0x7f7498009a80).ready entity=mon.0 client_cookie=a6430bf7de83d3c6 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:34.963 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.949 7f74a67fc700 1 -- 172.21.2.112:0/1020491353 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f74980009b0 con 0x7f74a80d17f0 2021-02-10T11:15:34.963 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.949 7f74a67fc700 1 -- 172.21.2.112:0/1020491353 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f749800c0c0 con 0x7f74a80d17f0 2021-02-10T11:15:34.963 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.949 7f74a67fc700 1 -- 172.21.2.112:0/1020491353 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f749800c5e0 con 0x7f74a80d17f0 2021-02-10T11:15:34.963 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.949 7f74adcdf700 1 -- 172.21.2.112:0/1020491353 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f74a80d17f0 msgr2=0x7f74a81b31e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:34.963 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.949 7f74adcdf700 1 --2- 172.21.2.112:0/1020491353 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f74a80d17f0 0x7f74a81b31e0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7f7498006650 tx=0x7f7498009a80).stop 2021-02-10T11:15:34.963 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.949 7f74adcdf700 1 -- 172.21.2.112:0/1020491353 shutdown_connections 2021-02-10T11:15:34.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.949 7f74adcdf700 1 --2- 172.21.2.112:0/1020491353 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f74a81b3e20 0x7f74a80ceef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:34.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.949 7f74adcdf700 1 --2- 172.21.2.112:0/1020491353 >> v2:172.21.2.120:3300/0 conn(0x7f74a80d1bf0 0x7f74a81b3800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:34.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.949 7f74adcdf700 1 --2- 172.21.2.112:0/1020491353 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f74a80d17f0 0x7f74a81b31e0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:34.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.949 7f74adcdf700 1 -- 172.21.2.112:0/1020491353 >> 172.21.2.112:0/1020491353 conn(0x7f74a80fde40 msgr2=0x7f74a80ce390 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:34.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.949 7f74adcdf700 1 -- 172.21.2.112:0/1020491353 shutdown_connections 2021-02-10T11:15:34.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.949 7f74adcdf700 1 -- 172.21.2.112:0/1020491353 wait complete. 2021-02-10T11:15:34.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.964 7f74adcdf700 1 Processor -- start 2021-02-10T11:15:34.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.964 7f74adcdf700 1 -- start start 2021-02-10T11:15:34.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.964 7f74adcdf700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f74a8107ee0 0x7f74a808e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:34.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.964 7f74adcdf700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f74a808ee90 0x7f74a808f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:34.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.964 7f74adcdf700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f74a80b6d20 0x7f74a80b7120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:34.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.964 7f74adcdf700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74a80b67a0 con 0x7f74a8107ee0 2021-02-10T11:15:34.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.964 7f74adcdf700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74a81082e0 con 0x7f74a80b6d20 2021-02-10T11:15:34.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.964 7f74adcdf700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74a808f8b0 con 0x7f74a808ee90 2021-02-10T11:15:34.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.965 7f74a77fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f74a8107ee0 0x7f74a808e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:34.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.965 7f74a77fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f74a8107ee0 0x7f74a808e870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:43490/0 (socket says 172.21.2.112:43490) 2021-02-10T11:15:34.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.965 7f74a77fe700 1 -- 172.21.2.112:0/2152499919 learned_addr learned my addr 172.21.2.112:0/2152499919 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:15:34.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.965 7f74a7fff700 1 --2- 172.21.2.112:0/2152499919 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f74a80b6d20 0x7f74a80b7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:34.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.965 7f74a6ffd700 1 --2- 172.21.2.112:0/2152499919 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f74a808ee90 0x7f74a808f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:34.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.965 7f74a77fe700 1 -- 172.21.2.112:0/2152499919 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f74a808ee90 msgr2=0x7f74a808f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:34.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.965 7f74a77fe700 1 --2- 172.21.2.112:0/2152499919 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f74a808ee90 0x7f74a808f290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:34.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.965 7f74a77fe700 1 -- 172.21.2.112:0/2152499919 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f74a80b6d20 msgr2=0x7f74a80b7120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:34.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.965 7f74a77fe700 1 --2- 172.21.2.112:0/2152499919 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f74a80b6d20 0x7f74a80b7120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:34.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.965 7f74a77fe700 1 -- 172.21.2.112:0/2152499919 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74a80b7740 con 0x7f74a8107ee0 2021-02-10T11:15:34.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.965 7f74a6ffd700 1 --2- 172.21.2.112:0/2152499919 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f74a808ee90 0x7f74a808f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:15:34.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.965 7f74a77fe700 1 --2- 172.21.2.112:0/2152499919 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f74a8107ee0 0x7f74a808e870 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7f7498009630 tx=0x7f749800b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:34.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.965 7f748ffff700 1 -- 172.21.2.112:0/2152499919 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f74980093a0 con 0x7f74a8107ee0 2021-02-10T11:15:34.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.966 7f748ffff700 1 -- 172.21.2.112:0/2152499919 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f749800bd70 con 0x7f74a8107ee0 2021-02-10T11:15:34.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.966 7f748ffff700 1 -- 172.21.2.112:0/2152499919 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f74980054d0 con 0x7f74a8107ee0 2021-02-10T11:15:34.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.966 7f74adcdf700 1 -- 172.21.2.112:0/2152499919 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74a80b7740 con 0x7f74a8107ee0 2021-02-10T11:15:34.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.966 7f74adcdf700 1 -- 172.21.2.112:0/2152499919 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74a80b7cd0 con 0x7f74a8107ee0 2021-02-10T11:15:34.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.966 7f748ffff700 1 -- 172.21.2.112:0/2152499919 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f749800a000 con 0x7f74a8107ee0 2021-02-10T11:15:34.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.966 7f748ffff700 1 --2- 172.21.2.112:0/2152499919 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f74880186c0 0x7f748801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:34.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.966 7f748ffff700 1 -- 172.21.2.112:0/2152499919 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f7498032410 con 0x7f74a8107ee0 2021-02-10T11:15:34.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.967 7f74a6ffd700 1 --2- 172.21.2.112:0/2152499919 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f74880186c0 0x7f748801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:34.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.967 7f74a6ffd700 1 --2- 172.21.2.112:0/2152499919 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f74880186c0 0x7f748801abc0 crc :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:34.996 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:34.998 7f74adcdf700 1 -- 172.21.2.112:0/2152499919 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74a808f8b0 con 0x7f74a8107ee0 2021-02-10T11:15:34.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:35.000 7f748ffff700 1 -- 172.21.2.112:0/2152499919 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f749802ef10 con 0x7f74a8107ee0 2021-02-10T11:15:35.121 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:35.123 7f74adcdf700 1 -- 172.21.2.112:0/2152499919 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f74a808f8b0 con 0x7f74a8107ee0 2021-02-10T11:15:35.121 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:35.123 7f748ffff700 1 -- 172.21.2.112:0/2152499919 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) v1 ==== 74+0+13993 (secure 0 0 0) 0x7f749802ef10 con 0x7f74a8107ee0 2021-02-10T11:15:35.122 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:15:35.122 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":29,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:15:34.922144","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:15:31.539720","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":5,"pool_name":"cephfs_data","create_time":"2021-02-10 11:15:32.965643","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:15:35.123 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:35.125 7f74adcdf700 1 -- 172.21.2.112:0/2152499919 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f74880186c0 msgr2=0x7f748801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:35.124 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:35.125 7f74adcdf700 1 --2- 172.21.2.112:0/2152499919 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f74880186c0 0x7f748801abc0 crc :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:35.124 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:35.125 7f74adcdf700 1 -- 172.21.2.112:0/2152499919 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f74a8107ee0 msgr2=0x7f74a808e870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:35.124 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:35.125 7f74adcdf700 1 --2- 172.21.2.112:0/2152499919 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f74a8107ee0 0x7f74a808e870 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7f7498009630 tx=0x7f749800b980).stop 2021-02-10T11:15:35.124 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:35.125 7f74adcdf700 1 -- 172.21.2.112:0/2152499919 shutdown_connections 2021-02-10T11:15:35.124 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:35.125 7f74adcdf700 1 --2- 172.21.2.112:0/2152499919 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f74880186c0 0x7f748801abc0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:35.124 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:35.125 7f74adcdf700 1 --2- 172.21.2.112:0/2152499919 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f74a80b6d20 0x7f74a80b7120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:35.124 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:35.125 7f74adcdf700 1 --2- 172.21.2.112:0/2152499919 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f74a808ee90 0x7f74a808f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:35.124 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:35.125 7f74adcdf700 1 --2- 172.21.2.112:0/2152499919 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f74a8107ee0 0x7f74a808e870 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:35.124 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:35.125 7f74adcdf700 1 -- 172.21.2.112:0/2152499919 >> 172.21.2.112:0/2152499919 conn(0x7f74a809ca80 msgr2=0x7f74a80c3c60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:35.124 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:35.126 7f74adcdf700 1 -- 172.21.2.112:0/2152499919 shutdown_connections 2021-02-10T11:15:35.124 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:35.126 7f74adcdf700 1 -- 172.21.2.112:0/2152499919 wait complete. 2021-02-10T11:15:35.134 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2021-02-10T11:15:35.564 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:35.550 7f06c704b700 1 Processor -- start 2021-02-10T11:15:35.565 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:35.550 7f06c704b700 1 -- start start 2021-02-10T11:15:35.565 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:35.550 7f06c704b700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f06c01b0470 0x7f06c01b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:35.565 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:35.550 7f06c704b700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f06c00ef610 0x7f06c00efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:35.565 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:35.550 7f06c704b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f06c00f0030 0x7f06c00f0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:35.565 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:35.550 7f06c704b700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06c00e5ea0 con 0x7f06c00ef610 2021-02-10T11:15:35.565 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:35.550 7f06c704b700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06c00f0b50 con 0x7f06c01b0470 2021-02-10T11:15:35.565 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:35.550 7f06c704b700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06c00e52e0 con 0x7f06c00f0030 2021-02-10T11:15:35.565 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:35.551 7f06bffff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f06c00ef610 0x7f06c00efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:35.565 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:35.551 7f06bffff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f06c00ef610 0x7f06c00efa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:43498/0 (socket says 172.21.2.112:43498) 2021-02-10T11:15:35.565 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:35.551 7f06bffff700 1 -- 172.21.2.112:0/2772473063 learned_addr learned my addr 172.21.2.112:0/2772473063 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:15:35.565 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:35.551 7f06bffff700 1 -- 172.21.2.112:0/2772473063 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f06c00f0030 msgr2=0x7f06c00f0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:15:35.566 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:35.551 7f06bffff700 1 --2- 172.21.2.112:0/2772473063 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f06c00f0030 0x7f06c00f0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:35.566 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:35.551 7f06bffff700 1 -- 172.21.2.112:0/2772473063 >> v2:172.21.2.120:3300/0 conn(0x7f06c01b0470 msgr2=0x7f06c01b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:15:35.566 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:35.551 7f06bffff700 1 --2- 172.21.2.112:0/2772473063 >> v2:172.21.2.120:3300/0 conn(0x7f06c01b0470 0x7f06c01b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:35.566 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:35.551 7f06bffff700 1 -- 172.21.2.112:0/2772473063 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06c00f10d0 con 0x7f06c00ef610 2021-02-10T11:15:35.566 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:35.551 7f06bffff700 1 --2- 172.21.2.112:0/2772473063 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f06c00ef610 0x7f06c00efa10 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 rx=0x7f06b0006700 tx=0x7f06b0009b80).ready entity=mon.0 client_cookie=f5c0f652c7d77b36 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:35.566 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:35.551 7f06bf7fe700 1 -- 172.21.2.112:0/2772473063 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f06b000c0c0 con 0x7f06c00ef610 2021-02-10T11:15:35.566 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:35.551 7f06bf7fe700 1 -- 172.21.2.112:0/2772473063 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06b000c440 con 0x7f06c00ef610 2021-02-10T11:15:35.566 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:35.551 7f06bf7fe700 1 -- 172.21.2.112:0/2772473063 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f06b000c8a0 con 0x7f06c00ef610 2021-02-10T11:15:35.566 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:35.551 7f06c704b700 1 -- 172.21.2.112:0/2772473063 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f06c00ef610 msgr2=0x7f06c00efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:35.566 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:35.551 7f06c704b700 1 --2- 172.21.2.112:0/2772473063 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f06c00ef610 0x7f06c00efa10 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 rx=0x7f06b0006700 tx=0x7f06b0009b80).stop 2021-02-10T11:15:35.566 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:35.552 7f06c704b700 1 -- 172.21.2.112:0/2772473063 shutdown_connections 2021-02-10T11:15:35.566 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:35.552 7f06c704b700 1 --2- 172.21.2.112:0/2772473063 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f06c00f0030 0x7f06c00f0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:35.566 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:35.552 7f06c704b700 1 --2- 172.21.2.112:0/2772473063 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f06c00ef610 0x7f06c00efa10 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:35.567 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:35.552 7f06c704b700 1 --2- 172.21.2.112:0/2772473063 >> v2:172.21.2.120:3300/0 conn(0x7f06c01b0470 0x7f06c01b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:35.567 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:35.552 7f06c704b700 1 -- 172.21.2.112:0/2772473063 >> 172.21.2.112:0/2772473063 conn(0x7f06c00ce3e0 msgr2=0x7f06c00e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:35.567 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:35.552 7f06c704b700 1 -- 172.21.2.112:0/2772473063 shutdown_connections 2021-02-10T11:15:35.567 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:35.552 7f06c704b700 1 -- 172.21.2.112:0/2772473063 wait complete. 2021-02-10T11:15:35.568 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:35.567 7f06c704b700 1 Processor -- start 2021-02-10T11:15:35.568 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:35.567 7f06c704b700 1 -- start start 2021-02-10T11:15:35.569 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:35.568 7f06c704b700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f06c00bb980 0x7f06c019bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:35.569 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:35.568 7f06c704b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f06c019c140 0x7f06c00acf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:35.569 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:35.568 7f06c704b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f06c00ad570 0x7f06c00a78f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:35.569 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:35.568 7f06c704b700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06c00c3e00 con 0x7f06c019c140 2021-02-10T11:15:35.569 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:35.568 7f06c704b700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06c00ad970 con 0x7f06c00bb980 2021-02-10T11:15:35.569 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:35.568 7f06c704b700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06c00adcb0 con 0x7f06c00ad570 2021-02-10T11:15:35.569 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:35.568 7f06bffff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f06c019c140 0x7f06c00acf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:35.569 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:35.568 7f06bffff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f06c019c140 0x7f06c00acf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:43500/0 (socket says 172.21.2.112:43500) 2021-02-10T11:15:35.569 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:35.568 7f06bffff700 1 -- 172.21.2.112:0/3149695263 learned_addr learned my addr 172.21.2.112:0/3149695263 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:15:35.569 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:35.568 7f06c4e67700 1 --2- 172.21.2.112:0/3149695263 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f06c00bb980 0x7f06c019bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:35.569 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:35.568 7f06c5668700 1 --2- 172.21.2.112:0/3149695263 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f06c00ad570 0x7f06c00a78f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:35.569 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:35.568 7f06bffff700 1 -- 172.21.2.112:0/3149695263 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f06c00ad570 msgr2=0x7f06c00a78f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:35.569 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:35.568 7f06bffff700 1 --2- 172.21.2.112:0/3149695263 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f06c00ad570 0x7f06c00a78f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:35.570 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:35.568 7f06bffff700 1 -- 172.21.2.112:0/3149695263 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f06c00bb980 msgr2=0x7f06c019bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:35.570 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:35.568 7f06bffff700 1 --2- 172.21.2.112:0/3149695263 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f06c00bb980 0x7f06c019bb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:35.570 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:35.568 7f06bffff700 1 -- 172.21.2.112:0/3149695263 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06c00a7f10 con 0x7f06c019c140 2021-02-10T11:15:35.570 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:35.569 7f06c4e67700 1 --2- 172.21.2.112:0/3149695263 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f06c00bb980 0x7f06c019bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:15:35.570 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:35.569 7f06bffff700 1 --2- 172.21.2.112:0/3149695263 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f06c019c140 0x7f06c00acf50 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7f06b000b5e0 tx=0x7f06b000b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:35.570 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:35.569 7f06bd7fa700 1 -- 172.21.2.112:0/3149695263 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f06b000bd70 con 0x7f06c019c140 2021-02-10T11:15:35.570 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:35.569 7f06bd7fa700 1 -- 172.21.2.112:0/3149695263 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06b0002030 con 0x7f06c019c140 2021-02-10T11:15:35.570 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:35.569 7f06bd7fa700 1 -- 172.21.2.112:0/3149695263 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f06b0002550 con 0x7f06c019c140 2021-02-10T11:15:35.570 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:35.569 7f06c704b700 1 -- 172.21.2.112:0/3149695263 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06c00a7f10 con 0x7f06c019c140 2021-02-10T11:15:35.570 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:35.569 7f06c704b700 1 -- 172.21.2.112:0/3149695263 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06c00a7f10 con 0x7f06c019c140 2021-02-10T11:15:35.570 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:35.570 7f06bd7fa700 1 -- 172.21.2.112:0/3149695263 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f06b0000c90 con 0x7f06c019c140 2021-02-10T11:15:35.570 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:35.570 7f06bd7fa700 1 --2- 172.21.2.112:0/3149695263 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f06a40186c0 0x7f06a401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:35.570 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:35.570 7f06bd7fa700 1 -- 172.21.2.112:0/3149695263 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f06b00323c0 con 0x7f06c019c140 2021-02-10T11:15:35.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:35.571 7f06c4e67700 1 --2- 172.21.2.112:0/3149695263 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f06a40186c0 0x7f06a401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:35.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:35.571 7f06c4e67700 1 --2- 172.21.2.112:0/3149695263 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f06a40186c0 0x7f06a401abc0 crc :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:35.598 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:35.600 7f06c704b700 1 -- 172.21.2.112:0/3149695263 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06c0105af0 con 0x7f06c019c140 2021-02-10T11:15:35.600 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:35.603 7f06bd7fa700 1 -- 172.21.2.112:0/3149695263 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f06b0030cf0 con 0x7f06c019c140 2021-02-10T11:15:35.734 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:35.736 7f06c704b700 1 -- 172.21.2.112:0/3149695263 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"} v 0) v1 -- 0x7f06c00f0b50 con 0x7f06c019c140 2021-02-10T11:15:36.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:36.314 7f06bd7fa700 1 -- 172.21.2.112:0/3149695263 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"}]=0 v13) v1 ==== 118+0+0 (secure 0 0 0) 0x7f06b0030cf0 con 0x7f06c019c140 2021-02-10T11:15:36.355 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:36.357 7f06c704b700 1 -- 172.21.2.112:0/3149695263 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f06a40186c0 msgr2=0x7f06a401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:36.355 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:36.357 7f06c704b700 1 --2- 172.21.2.112:0/3149695263 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f06a40186c0 0x7f06a401abc0 crc :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:36.356 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:36.357 7f06c704b700 1 -- 172.21.2.112:0/3149695263 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f06c019c140 msgr2=0x7f06c00acf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:36.356 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:36.357 7f06c704b700 1 --2- 172.21.2.112:0/3149695263 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f06c019c140 0x7f06c00acf50 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7f06b000b5e0 tx=0x7f06b000b980).stop 2021-02-10T11:15:36.356 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:36.357 7f06c704b700 1 -- 172.21.2.112:0/3149695263 shutdown_connections 2021-02-10T11:15:36.356 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:36.357 7f06c704b700 1 --2- 172.21.2.112:0/3149695263 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f06a40186c0 0x7f06a401abc0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:36.356 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:36.357 7f06c704b700 1 --2- 172.21.2.112:0/3149695263 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f06c00ad570 0x7f06c00a78f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:36.356 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:36.357 7f06c704b700 1 --2- 172.21.2.112:0/3149695263 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f06c019c140 0x7f06c00acf50 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:36.357 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:36.357 7f06c704b700 1 --2- 172.21.2.112:0/3149695263 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f06c00bb980 0x7f06c019bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:36.357 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:36.357 7f06c704b700 1 -- 172.21.2.112:0/3149695263 >> 172.21.2.112:0/3149695263 conn(0x7f06c00bf4e0 msgr2=0x7f06c00bfdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:36.357 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:36.358 7f06c704b700 1 -- 172.21.2.112:0/3149695263 shutdown_connections 2021-02-10T11:15:36.357 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:36.358 7f06c704b700 1 -- 172.21.2.112:0/3149695263 wait complete. 2021-02-10T11:15:36.363 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:15:36.788 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:36.773 7f93b838a700 1 Processor -- start 2021-02-10T11:15:36.788 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:36.773 7f93b838a700 1 -- start start 2021-02-10T11:15:36.788 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:36.774 7f93b838a700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f93b00e5470 0x7f93b007e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:36.788 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:36.774 7f93b838a700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f93b007f300 0x7f93b007f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:36.788 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:36.774 7f93b838a700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93b007ed80 con 0x7f93b00e5070 2021-02-10T11:15:36.788 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:36.774 7f93b838a700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93b01b4010 con 0x7f93b00e5470 2021-02-10T11:15:36.788 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:36.774 7f93b838a700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93b00d1790 con 0x7f93b007f300 2021-02-10T11:15:36.788 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:36.774 7f93b59a5700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f93b00e5470 0x7f93b007e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:36.788 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:36.774 7f93b59a5700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f93b00e5470 0x7f93b007e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:43508/0 (socket says 172.21.2.112:43508) 2021-02-10T11:15:36.788 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:36.774 7f93b59a5700 1 -- 172.21.2.112:0/2615964448 learned_addr learned my addr 172.21.2.112:0/2615964448 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:15:36.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:36.774 7f93b59a5700 1 -- 172.21.2.112:0/2615964448 >> v2:172.21.2.120:3300/0 conn(0x7f93b007f300 msgr2=0x7f93b007f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:15:36.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:36.774 7f93b59a5700 1 --2- 172.21.2.112:0/2615964448 >> v2:172.21.2.120:3300/0 conn(0x7f93b007f300 0x7f93b007f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:36.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:36.774 7f93b59a5700 1 -- 172.21.2.112:0/2615964448 >> v1:172.21.2.120:6789/0 conn(0x7f93b00e5070 legacy=0x7f93b01b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:15:36.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:36.774 7f93b59a5700 1 -- 172.21.2.112:0/2615964448 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93b00d19b0 con 0x7f93b00e5470 2021-02-10T11:15:36.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:36.774 7f93b59a5700 1 --2- 172.21.2.112:0/2615964448 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f93b00e5470 0x7f93b007e6a0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0x7f93ac006700 tx=0x7f93ac009b80).ready entity=mon.0 client_cookie=afcb3b95d03fbcdf server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:36.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:36.775 7f93b51a4700 1 -- 172.21.2.112:0/2615964448 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f93ac00c0c0 con 0x7f93b00e5470 2021-02-10T11:15:36.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:36.775 7f93b51a4700 1 -- 172.21.2.112:0/2615964448 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93ac00c340 con 0x7f93b00e5470 2021-02-10T11:15:36.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:36.775 7f93b51a4700 1 -- 172.21.2.112:0/2615964448 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f93ac00c880 con 0x7f93b00e5470 2021-02-10T11:15:36.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:36.775 7f93b838a700 1 -- 172.21.2.112:0/2615964448 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f93b00e5470 msgr2=0x7f93b007e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:36.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:36.775 7f93b838a700 1 --2- 172.21.2.112:0/2615964448 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f93b00e5470 0x7f93b007e6a0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0x7f93ac006700 tx=0x7f93ac009b80).stop 2021-02-10T11:15:36.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:36.775 7f93b838a700 1 -- 172.21.2.112:0/2615964448 shutdown_connections 2021-02-10T11:15:36.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:36.775 7f93b838a700 1 --2- 172.21.2.112:0/2615964448 >> v2:172.21.2.120:3300/0 conn(0x7f93b007f300 0x7f93b007f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:36.790 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:36.775 7f93b838a700 1 --2- 172.21.2.112:0/2615964448 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f93b00e5470 0x7f93b007e6a0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:36.790 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:36.775 7f93b838a700 1 -- 172.21.2.112:0/2615964448 >> 172.21.2.112:0/2615964448 conn(0x7f93b00ce690 msgr2=0x7f93b00e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:36.790 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:36.775 7f93b838a700 1 -- 172.21.2.112:0/2615964448 shutdown_connections 2021-02-10T11:15:36.790 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:36.776 7f93b838a700 1 -- 172.21.2.112:0/2615964448 wait complete. 2021-02-10T11:15:36.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:36.790 7f93b838a700 1 Processor -- start 2021-02-10T11:15:36.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:36.790 7f93b838a700 1 -- start start 2021-02-10T11:15:36.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:36.790 7f93b838a700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f93b01b19d0 0x7f93b008cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:36.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:36.790 7f93b838a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f93b01b0d00 0x7f93b01b1100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:36.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:36.790 7f93b838a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f93b01a2b50 0x7f93b01a2f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:36.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:36.791 7f93b838a700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93b01b1720 con 0x7f93b01b0d00 2021-02-10T11:15:36.793 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:36.791 7f93b838a700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93b008c470 con 0x7f93b01b19d0 2021-02-10T11:15:36.793 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:36.791 7f93b838a700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93b01b1f50 con 0x7f93b01a2b50 2021-02-10T11:15:36.793 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:36.791 7f93b59a5700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f93b01b0d00 0x7f93b01b1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:36.793 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:36.791 7f93b69a7700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f93b01a2b50 0x7f93b01a2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:36.793 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:36.791 7f93b59a5700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f93b01b0d00 0x7f93b01b1100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:43516/0 (socket says 172.21.2.112:43516) 2021-02-10T11:15:36.793 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:36.791 7f93b59a5700 1 -- 172.21.2.112:0/2823890055 learned_addr learned my addr 172.21.2.112:0/2823890055 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:15:36.793 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:36.791 7f93b61a6700 1 --2- 172.21.2.112:0/2823890055 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f93b01b19d0 0x7f93b008cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:36.793 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:36.791 7f93b59a5700 1 -- 172.21.2.112:0/2823890055 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f93b01a2b50 msgr2=0x7f93b01a2f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:36.793 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:36.791 7f93b59a5700 1 --2- 172.21.2.112:0/2823890055 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f93b01a2b50 0x7f93b01a2f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:36.793 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:36.791 7f93b59a5700 1 -- 172.21.2.112:0/2823890055 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f93b01b19d0 msgr2=0x7f93b008cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:36.793 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:36.791 7f93b59a5700 1 --2- 172.21.2.112:0/2823890055 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f93b01b19d0 0x7f93b008cd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:36.793 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:36.791 7f93b59a5700 1 -- 172.21.2.112:0/2823890055 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93b01a3570 con 0x7f93b01b0d00 2021-02-10T11:15:36.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:36.791 7f93b59a5700 1 --2- 172.21.2.112:0/2823890055 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f93b01b0d00 0x7f93b01b1100 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7f93ac009720 tx=0x7f93ac00b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:36.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:36.792 7f93a2ffd700 1 -- 172.21.2.112:0/2823890055 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f93ac0090c0 con 0x7f93b01b0d00 2021-02-10T11:15:36.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:36.792 7f93a2ffd700 1 -- 172.21.2.112:0/2823890055 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93ac002030 con 0x7f93b01b0d00 2021-02-10T11:15:36.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:36.792 7f93a2ffd700 1 -- 172.21.2.112:0/2823890055 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f93ac002550 con 0x7f93b01b0d00 2021-02-10T11:15:36.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:36.792 7f93b838a700 1 -- 172.21.2.112:0/2823890055 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93b01a3570 con 0x7f93b01b0d00 2021-02-10T11:15:36.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:36.792 7f93b838a700 1 -- 172.21.2.112:0/2823890055 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93b01a3570 con 0x7f93b01b0d00 2021-02-10T11:15:36.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:36.792 7f93a2ffd700 1 -- 172.21.2.112:0/2823890055 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f93ac000c90 con 0x7f93b01b0d00 2021-02-10T11:15:36.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:36.793 7f93a2ffd700 1 --2- 172.21.2.112:0/2823890055 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9398018730 0x7f939801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:36.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:36.793 7f93a2ffd700 1 -- 172.21.2.112:0/2823890055 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f93ac032640 con 0x7f93b01b0d00 2021-02-10T11:15:36.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:36.793 7f93b61a6700 1 --2- 172.21.2.112:0/2823890055 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9398018730 0x7f939801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:36.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:36.793 7f93b61a6700 1 --2- 172.21.2.112:0/2823890055 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9398018730 0x7f939801ac10 crc :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:36.822 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:36.824 7f93b838a700 1 -- 172.21.2.112:0/2823890055 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93b0083dd0 con 0x7f93b01b0d00 2021-02-10T11:15:36.824 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:36.826 7f93a2ffd700 1 -- 172.21.2.112:0/2823890055 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f93ac000c90 con 0x7f93b01b0d00 2021-02-10T11:15:36.953 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:36.954 7f93b838a700 1 -- 172.21.2.112:0/2823890055 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f93b007ed80 con 0x7f93b01b0d00 2021-02-10T11:15:36.953 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:36.955 7f93a2ffd700 1 -- 172.21.2.112:0/2823890055 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+2100 (secure 0 0 0) 0x7f93ac00bd50 con 0x7f93b01b0d00 2021-02-10T11:15:36.954 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 13 2021-02-10T11:15:36.954 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:15:36.954 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":13,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4627,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/2302110857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":2302110857},{"type":"v1","addr":"172.21.2.112:6833","nonce":2302110857}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":9}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:15:33.919772","modified":"2021-02-10 11:15:36.309405","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4427},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4427":{"gid":4427,"name":"b","rank":0,"incarnation":10,"state":"up:active","state_seq":4,"addr":"172.21.2.120:6835/3730022736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":3730022736},{"type":"v1","addr":"172.21.2.120:6835","nonce":3730022736}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2021-02-10T11:15:36.955 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:36.957 7f93b838a700 1 -- 172.21.2.112:0/2823890055 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9398018730 msgr2=0x7f939801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:36.955 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:36.957 7f93b838a700 1 --2- 172.21.2.112:0/2823890055 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9398018730 0x7f939801ac10 crc :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:36.956 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:36.957 7f93b838a700 1 -- 172.21.2.112:0/2823890055 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f93b01b0d00 msgr2=0x7f93b01b1100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:36.956 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:36.957 7f93b838a700 1 --2- 172.21.2.112:0/2823890055 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f93b01b0d00 0x7f93b01b1100 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7f93ac009720 tx=0x7f93ac00b960).stop 2021-02-10T11:15:36.956 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:36.957 7f93b838a700 1 -- 172.21.2.112:0/2823890055 shutdown_connections 2021-02-10T11:15:36.956 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:36.957 7f93b838a700 1 --2- 172.21.2.112:0/2823890055 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9398018730 0x7f939801ac10 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:36.956 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:36.957 7f93b838a700 1 --2- 172.21.2.112:0/2823890055 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f93b01a2b50 0x7f93b01a2f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:36.956 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:36.957 7f93b838a700 1 --2- 172.21.2.112:0/2823890055 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f93b01b0d00 0x7f93b01b1100 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:36.956 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:36.957 7f93b838a700 1 --2- 172.21.2.112:0/2823890055 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f93b01b19d0 0x7f93b008cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:36.956 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:36.957 7f93b838a700 1 -- 172.21.2.112:0/2823890055 >> 172.21.2.112:0/2823890055 conn(0x7f93b0093720 msgr2=0x7f93b0084b80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:36.956 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:36.957 7f93b838a700 1 -- 172.21.2.112:0/2823890055 shutdown_connections 2021-02-10T11:15:36.956 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:36.957 7f93b838a700 1 -- 172.21.2.112:0/2823890055 wait complete. 2021-02-10T11:15:36.964 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:15:37.390 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:37.376 7f2c6366f700 1 Processor -- start 2021-02-10T11:15:37.391 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:37.376 7f2c6366f700 1 -- start start 2021-02-10T11:15:37.391 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:37.377 7f2c6366f700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f2c5c0e5070 0x7f2c5c1b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:37.391 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:37.377 7f2c6366f700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f2c5c1b3e40 0x7f2c5c07d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:37.391 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:37.377 7f2c6366f700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2c5c0e5470 0x7f2c5c07dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:37.391 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:37.377 7f2c6366f700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c5c07ed80 con 0x7f2c5c1b3e40 2021-02-10T11:15:37.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:37.377 7f2c6366f700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c5c07e2d0 con 0x7f2c5c0e5070 2021-02-10T11:15:37.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:37.377 7f2c6366f700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c5c07e4f0 con 0x7f2c5c0e5470 2021-02-10T11:15:37.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:37.377 7f2c60c8a700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f2c5c1b3e40 0x7f2c5c07d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:37.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:37.377 7f2c60c8a700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f2c5c1b3e40 0x7f2c5c07d690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:43522/0 (socket says 172.21.2.112:43522) 2021-02-10T11:15:37.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:37.377 7f2c60c8a700 1 -- 172.21.2.112:0/2304205811 learned_addr learned my addr 172.21.2.112:0/2304205811 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:15:37.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:37.377 7f2c60c8a700 1 -- 172.21.2.112:0/2304205811 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2c5c0e5470 msgr2=0x7f2c5c07dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:15:37.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:37.377 7f2c60c8a700 1 --2- 172.21.2.112:0/2304205811 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2c5c0e5470 0x7f2c5c07dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:37.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:37.377 7f2c60c8a700 1 -- 172.21.2.112:0/2304205811 >> v2:172.21.2.120:3300/0 conn(0x7f2c5c0e5070 msgr2=0x7f2c5c1b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:15:37.393 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:37.377 7f2c60c8a700 1 --2- 172.21.2.112:0/2304205811 >> v2:172.21.2.120:3300/0 conn(0x7f2c5c0e5070 0x7f2c5c1b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:37.393 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:37.377 7f2c60c8a700 1 -- 172.21.2.112:0/2304205811 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c5c07e710 con 0x7f2c5c1b3e40 2021-02-10T11:15:37.393 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:37.377 7f2c60c8a700 1 --2- 172.21.2.112:0/2304205811 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2c5c1b3e40 0x7f2c5c07d690 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 rx=0x7f2c58006700 tx=0x7f2c58009b80).ready entity=mon.0 client_cookie=b731094b0bf15d2 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:37.393 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:37.378 7f2c4ffff700 1 -- 172.21.2.112:0/2304205811 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2c5800c0c0 con 0x7f2c5c1b3e40 2021-02-10T11:15:37.393 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:37.378 7f2c4ffff700 1 -- 172.21.2.112:0/2304205811 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c5800c440 con 0x7f2c5c1b3e40 2021-02-10T11:15:37.393 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:37.378 7f2c4ffff700 1 -- 172.21.2.112:0/2304205811 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2c5800c920 con 0x7f2c5c1b3e40 2021-02-10T11:15:37.393 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:37.378 7f2c6366f700 1 -- 172.21.2.112:0/2304205811 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2c5c1b3e40 msgr2=0x7f2c5c07d690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:37.394 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:37.378 7f2c6366f700 1 --2- 172.21.2.112:0/2304205811 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2c5c1b3e40 0x7f2c5c07d690 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 rx=0x7f2c58006700 tx=0x7f2c58009b80).stop 2021-02-10T11:15:37.394 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:37.378 7f2c6366f700 1 -- 172.21.2.112:0/2304205811 shutdown_connections 2021-02-10T11:15:37.394 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:37.378 7f2c6366f700 1 --2- 172.21.2.112:0/2304205811 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2c5c0e5470 0x7f2c5c07dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:37.394 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:37.378 7f2c6366f700 1 --2- 172.21.2.112:0/2304205811 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2c5c1b3e40 0x7f2c5c07d690 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:37.394 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:37.378 7f2c6366f700 1 --2- 172.21.2.112:0/2304205811 >> v2:172.21.2.120:3300/0 conn(0x7f2c5c0e5070 0x7f2c5c1b3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:37.394 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:37.378 7f2c6366f700 1 -- 172.21.2.112:0/2304205811 >> 172.21.2.112:0/2304205811 conn(0x7f2c5c0ce690 msgr2=0x7f2c5c0e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:37.394 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:37.379 7f2c6366f700 1 -- 172.21.2.112:0/2304205811 shutdown_connections 2021-02-10T11:15:37.394 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:37.379 7f2c6366f700 1 -- 172.21.2.112:0/2304205811 wait complete. 2021-02-10T11:15:37.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:37.393 7f2c6366f700 1 Processor -- start 2021-02-10T11:15:37.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:37.393 7f2c6366f700 1 -- start start 2021-02-10T11:15:37.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:37.393 7f2c6366f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2c5c0969a0 0x7f2c5c096110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:37.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:37.393 7f2c6366f700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2c5c08a460 0x7f2c5c1a1440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:37.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:37.393 7f2c6366f700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2c5c0b1440 0x7f2c5c0b1840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:37.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:37.393 7f2c6366f700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c5c0b1e60 con 0x7f2c5c0969a0 2021-02-10T11:15:37.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:37.393 7f2c6366f700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c5c0b2080 con 0x7f2c5c08a460 2021-02-10T11:15:37.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:37.393 7f2c6366f700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c5c0b22a0 con 0x7f2c5c0b1440 2021-02-10T11:15:37.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:37.394 7f2c6148b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2c5c0969a0 0x7f2c5c096110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:37.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:37.394 7f2c6148b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2c5c0969a0 0x7f2c5c096110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:43528/0 (socket says 172.21.2.112:43528) 2021-02-10T11:15:37.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:37.394 7f2c6148b700 1 -- 172.21.2.112:0/1483247431 learned_addr learned my addr 172.21.2.112:0/1483247431 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:15:37.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:37.394 7f2c6148b700 1 -- 172.21.2.112:0/1483247431 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2c5c0b1440 msgr2=0x7f2c5c0b1840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:37.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:37.394 7f2c6148b700 1 --2- 172.21.2.112:0/1483247431 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2c5c0b1440 0x7f2c5c0b1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:37.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:37.394 7f2c6148b700 1 -- 172.21.2.112:0/1483247431 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2c5c08a460 msgr2=0x7f2c5c1a1440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:37.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:37.394 7f2c6148b700 1 --2- 172.21.2.112:0/1483247431 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2c5c08a460 0x7f2c5c1a1440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:37.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:37.394 7f2c6148b700 1 -- 172.21.2.112:0/1483247431 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c5c08ae90 con 0x7f2c5c0969a0 2021-02-10T11:15:37.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:37.394 7f2c6148b700 1 --2- 172.21.2.112:0/1483247431 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2c5c0969a0 0x7f2c5c096110 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7f2c50006f00 tx=0x7f2c5000b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:37.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:37.394 7f2c4dffb700 1 -- 172.21.2.112:0/1483247431 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2c50006350 con 0x7f2c5c0969a0 2021-02-10T11:15:37.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:37.394 7f2c4dffb700 1 -- 172.21.2.112:0/1483247431 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c5000bdc0 con 0x7f2c5c0969a0 2021-02-10T11:15:37.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:37.394 7f2c6366f700 1 -- 172.21.2.112:0/1483247431 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c5c08ae90 con 0x7f2c5c0969a0 2021-02-10T11:15:37.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:37.394 7f2c4dffb700 1 -- 172.21.2.112:0/1483247431 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2c50002020 con 0x7f2c5c0969a0 2021-02-10T11:15:37.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:37.394 7f2c6366f700 1 -- 172.21.2.112:0/1483247431 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c5c08ae90 con 0x7f2c5c0969a0 2021-02-10T11:15:37.398 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:37.395 7f2c4dffb700 1 -- 172.21.2.112:0/1483247431 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2c50007020 con 0x7f2c5c0969a0 2021-02-10T11:15:37.398 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:37.395 7f2c4dffb700 1 --2- 172.21.2.112:0/1483247431 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2c3c018730 0x7f2c3c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:37.398 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:37.395 7f2c4dffb700 1 -- 172.21.2.112:0/1483247431 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f2c500326a0 con 0x7f2c5c0969a0 2021-02-10T11:15:37.398 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:37.395 7f2c60c8a700 1 --2- 172.21.2.112:0/1483247431 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2c3c018730 0x7f2c3c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:37.398 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:37.396 7f2c60c8a700 1 --2- 172.21.2.112:0/1483247431 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2c3c018730 0x7f2c3c01abc0 crc :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:37.425 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:37.427 7f2c6366f700 1 -- 172.21.2.112:0/1483247431 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c5c07ed80 con 0x7f2c5c0969a0 2021-02-10T11:15:37.427 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:37.429 7f2c4dffb700 1 -- 172.21.2.112:0/1483247431 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2c50006b90 con 0x7f2c5c0969a0 2021-02-10T11:15:37.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:37.553 7f2c6366f700 1 -- 172.21.2.112:0/1483247431 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2c5c07ed80 con 0x7f2c5c0969a0 2021-02-10T11:15:37.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:37.553 7f2c4dffb700 1 -- 172.21.2.112:0/1483247431 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) v1 ==== 74+0+13993 (secure 0 0 0) 0x7f2c5000bd30 con 0x7f2c5c0969a0 2021-02-10T11:15:37.552 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:15:37.552 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":29,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:15:34.922144","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:15:31.539720","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":5,"pool_name":"cephfs_data","create_time":"2021-02-10 11:15:32.965643","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:15:37.553 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:37.555 7f2c6366f700 1 -- 172.21.2.112:0/1483247431 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2c3c018730 msgr2=0x7f2c3c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:37.553 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:37.555 7f2c6366f700 1 --2- 172.21.2.112:0/1483247431 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2c3c018730 0x7f2c3c01abc0 crc :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:37.553 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:37.555 7f2c6366f700 1 -- 172.21.2.112:0/1483247431 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2c5c0969a0 msgr2=0x7f2c5c096110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:37.553 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:37.555 7f2c6366f700 1 --2- 172.21.2.112:0/1483247431 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2c5c0969a0 0x7f2c5c096110 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7f2c50006f00 tx=0x7f2c5000b940).stop 2021-02-10T11:15:37.554 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:37.555 7f2c6366f700 1 -- 172.21.2.112:0/1483247431 shutdown_connections 2021-02-10T11:15:37.554 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:37.555 7f2c6366f700 1 --2- 172.21.2.112:0/1483247431 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2c3c018730 0x7f2c3c01abc0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:37.554 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:37.555 7f2c6366f700 1 --2- 172.21.2.112:0/1483247431 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2c5c08a460 0x7f2c5c1a1440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:37.554 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:37.555 7f2c6366f700 1 --2- 172.21.2.112:0/1483247431 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2c5c0b1440 0x7f2c5c0b1840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:37.554 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:37.555 7f2c6366f700 1 --2- 172.21.2.112:0/1483247431 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2c5c0969a0 0x7f2c5c096110 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:37.554 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:37.555 7f2c6366f700 1 -- 172.21.2.112:0/1483247431 >> 172.21.2.112:0/1483247431 conn(0x7f2c5c1909e0 msgr2=0x7f2c5c0895a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:37.554 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:37.555 7f2c6366f700 1 -- 172.21.2.112:0/1483247431 shutdown_connections 2021-02-10T11:15:37.554 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:37.555 7f2c6366f700 1 -- 172.21.2.112:0/1483247431 wait complete. 2021-02-10T11:15:37.563 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:15:38.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:37.987 7f0bdd92f700 1 Processor -- start 2021-02-10T11:15:38.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:37.987 7f0bdd92f700 1 -- start start 2021-02-10T11:15:38.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:37.988 7f0bdd92f700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f0bd80e5070 0x7f0bd81b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:38.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:37.988 7f0bdd92f700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0bd81b3e40 0x7f0bd807d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:38.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:37.988 7f0bdd92f700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bd807ed80 con 0x7f0bd80e5470 2021-02-10T11:15:38.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:37.988 7f0bdd92f700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bd8080310 con 0x7f0bd80e5070 2021-02-10T11:15:38.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:37.988 7f0bdd92f700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bd8080500 con 0x7f0bd81b3e40 2021-02-10T11:15:38.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:37.988 7f0bd7574700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f0bd80e5070 0x7f0bd81b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:38.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:37.988 7f0bd7574700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f0bd80e5070 0x7f0bd81b3820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:43534/0 (socket says 172.21.2.112:43534) 2021-02-10T11:15:38.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:37.988 7f0bd7574700 1 -- 172.21.2.112:0/1808085288 learned_addr learned my addr 172.21.2.112:0/1808085288 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:15:38.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:37.988 7f0bd6d73700 1 --2- 172.21.2.112:0/1808085288 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0bd81b3e40 0x7f0bd807d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:38.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:37.988 7f0bd7574700 1 -- 172.21.2.112:0/1808085288 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0bd81b3e40 msgr2=0x7f0bd807d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:15:38.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:37.988 7f0bd7574700 1 --2- 172.21.2.112:0/1808085288 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0bd81b3e40 0x7f0bd807d690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:38.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:37.988 7f0bd7574700 1 -- 172.21.2.112:0/1808085288 >> v1:172.21.2.120:6789/0 conn(0x7f0bd80e5470 legacy=0x7f0bd807dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:38.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:37.989 7f0bd7574700 1 -- 172.21.2.112:0/1808085288 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bd80806f0 con 0x7f0bd80e5070 2021-02-10T11:15:38.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:37.989 7f0bd7574700 1 --2- 172.21.2.112:0/1808085288 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0bd80e5070 0x7f0bd81b3820 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0x7f0bc0006700 tx=0x7f0bc0009b80).ready entity=mon.0 client_cookie=2b170f5ac1866f28 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:38.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:37.989 7f0bd6572700 1 -- 172.21.2.112:0/1808085288 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0bc000c0c0 con 0x7f0bd80e5070 2021-02-10T11:15:38.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:37.989 7f0bd6572700 1 -- 172.21.2.112:0/1808085288 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bc000c440 con 0x7f0bd80e5070 2021-02-10T11:15:38.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:37.989 7f0bd6572700 1 -- 172.21.2.112:0/1808085288 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0bc000c960 con 0x7f0bd80e5070 2021-02-10T11:15:38.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:37.989 7f0bdd92f700 1 -- 172.21.2.112:0/1808085288 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0bd80e5070 msgr2=0x7f0bd81b3820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:38.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:37.989 7f0bdd92f700 1 --2- 172.21.2.112:0/1808085288 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0bd80e5070 0x7f0bd81b3820 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0x7f0bc0006700 tx=0x7f0bc0009b80).stop 2021-02-10T11:15:38.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:37.990 7f0bdd92f700 1 -- 172.21.2.112:0/1808085288 shutdown_connections 2021-02-10T11:15:38.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:37.990 7f0bdd92f700 1 --2- 172.21.2.112:0/1808085288 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0bd81b3e40 0x7f0bd807d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:38.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:37.990 7f0bdd92f700 1 --2- 172.21.2.112:0/1808085288 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0bd80e5070 0x7f0bd81b3820 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:38.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:37.990 7f0bdd92f700 1 -- 172.21.2.112:0/1808085288 >> 172.21.2.112:0/1808085288 conn(0x7f0bd80ce690 msgr2=0x7f0bd80e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:38.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:37.990 7f0bdd92f700 1 -- 172.21.2.112:0/1808085288 shutdown_connections 2021-02-10T11:15:38.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:37.990 7f0bdd92f700 1 -- 172.21.2.112:0/1808085288 wait complete. 2021-02-10T11:15:38.006 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.005 7f0bdd92f700 1 Processor -- start 2021-02-10T11:15:38.006 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.005 7f0bdd92f700 1 -- start start 2021-02-10T11:15:38.006 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.005 7f0bdd92f700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0bd818c860 0x7f0bd818cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:38.007 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.005 7f0bdd92f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0bd818d280 0x7f0bd81a9e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:38.007 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.005 7f0bdd92f700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0bd818d680 0x7f0bd81aa420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:38.007 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.005 7f0bdd92f700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bd808d4e0 con 0x7f0bd818d280 2021-02-10T11:15:38.007 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.005 7f0bdd92f700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bd81ab5e0 con 0x7f0bd818d680 2021-02-10T11:15:38.007 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.005 7f0bdd92f700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bd80a07c0 con 0x7f0bd818c860 2021-02-10T11:15:38.007 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.005 7f0bd6d73700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0bd818d280 0x7f0bd81a9e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:38.007 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.006 7f0bd6d73700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0bd818d280 0x7f0bd81a9e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:43544/0 (socket says 172.21.2.112:43544) 2021-02-10T11:15:38.007 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.006 7f0bd6d73700 1 -- 172.21.2.112:0/2328148877 learned_addr learned my addr 172.21.2.112:0/2328148877 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:15:38.007 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.006 7f0bd7574700 1 --2- 172.21.2.112:0/2328148877 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0bd818c860 0x7f0bd818cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:38.007 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.006 7f0bd7d75700 1 --2- 172.21.2.112:0/2328148877 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0bd818d680 0x7f0bd81aa420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:38.007 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.006 7f0bd6d73700 1 -- 172.21.2.112:0/2328148877 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0bd818c860 msgr2=0x7f0bd818cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:38.007 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.006 7f0bd6d73700 1 --2- 172.21.2.112:0/2328148877 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0bd818c860 0x7f0bd818cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:38.007 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.006 7f0bd6d73700 1 -- 172.21.2.112:0/2328148877 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0bd818d680 msgr2=0x7f0bd81aa420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:38.008 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.006 7f0bd6d73700 1 --2- 172.21.2.112:0/2328148877 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0bd818d680 0x7f0bd81aa420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:38.008 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.006 7f0bd6d73700 1 -- 172.21.2.112:0/2328148877 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bd80b5610 con 0x7f0bd818d280 2021-02-10T11:15:38.008 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.006 7f0bd7d75700 1 --2- 172.21.2.112:0/2328148877 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0bd818d680 0x7f0bd81aa420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:15:38.008 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.006 7f0bd6d73700 1 --2- 172.21.2.112:0/2328148877 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0bd818d280 0x7f0bd81a9e00 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7f0bc8006f00 tx=0x7f0bc800b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:38.008 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.006 7f0bcffff700 1 -- 172.21.2.112:0/2328148877 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0bc800bd30 con 0x7f0bd818d280 2021-02-10T11:15:38.008 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.006 7f0bcffff700 1 -- 172.21.2.112:0/2328148877 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bc8002030 con 0x7f0bd818d280 2021-02-10T11:15:38.008 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.006 7f0bdd92f700 1 -- 172.21.2.112:0/2328148877 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bd80b5610 con 0x7f0bd818d280 2021-02-10T11:15:38.008 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.007 7f0bcffff700 1 -- 172.21.2.112:0/2328148877 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0bc8002510 con 0x7f0bd818d280 2021-02-10T11:15:38.008 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.007 7f0bdd92f700 1 -- 172.21.2.112:0/2328148877 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bd81aad30 con 0x7f0bd818d280 2021-02-10T11:15:38.008 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.007 7f0bcffff700 1 -- 172.21.2.112:0/2328148877 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0bc8000c90 con 0x7f0bd818d280 2021-02-10T11:15:38.008 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.007 7f0bcffff700 1 --2- 172.21.2.112:0/2328148877 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0bbc0186c0 0x7f0bbc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:38.008 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.007 7f0bcffff700 1 -- 172.21.2.112:0/2328148877 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f0bc8032560 con 0x7f0bd818d280 2021-02-10T11:15:38.008 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.008 7f0bd7574700 1 --2- 172.21.2.112:0/2328148877 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0bbc0186c0 0x7f0bbc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:38.009 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.008 7f0bd7574700 1 --2- 172.21.2.112:0/2328148877 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0bbc0186c0 0x7f0bbc01abc0 crc :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:38.036 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.038 7f0bdd92f700 1 -- 172.21.2.112:0/2328148877 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bd8084a90 con 0x7f0bd818d280 2021-02-10T11:15:38.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.041 7f0bcffff700 1 -- 172.21.2.112:0/2328148877 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0bc8000940 con 0x7f0bd818d280 2021-02-10T11:15:38.170 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.172 7f0bdd92f700 1 -- 172.21.2.112:0/2328148877 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f0bd807ed80 con 0x7f0bd818d280 2021-02-10T11:15:38.174 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.176 7f0bcffff700 1 -- 172.21.2.112:0/2328148877 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v13) v1 ==== 184+0+0 (secure 0 0 0) 0x7f0bc8002210 con 0x7f0bd818d280 2021-02-10T11:15:38.177 INFO:teuthology.orchestra.run.gibba012.stderr:updated caps for client.0 2021-02-10T11:15:38.179 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.181 7f0bdd92f700 1 -- 172.21.2.112:0/2328148877 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0bbc0186c0 msgr2=0x7f0bbc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:38.179 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.181 7f0bdd92f700 1 --2- 172.21.2.112:0/2328148877 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0bbc0186c0 0x7f0bbc01abc0 crc :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:38.179 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.181 7f0bdd92f700 1 -- 172.21.2.112:0/2328148877 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0bd818d280 msgr2=0x7f0bd81a9e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:38.179 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.181 7f0bdd92f700 1 --2- 172.21.2.112:0/2328148877 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0bd818d280 0x7f0bd81a9e00 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7f0bc8006f00 tx=0x7f0bc800b940).stop 2021-02-10T11:15:38.179 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.181 7f0bdd92f700 1 -- 172.21.2.112:0/2328148877 shutdown_connections 2021-02-10T11:15:38.179 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.181 7f0bdd92f700 1 --2- 172.21.2.112:0/2328148877 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0bbc0186c0 0x7f0bbc01abc0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:38.179 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.181 7f0bdd92f700 1 --2- 172.21.2.112:0/2328148877 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0bd818d680 0x7f0bd81aa420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:38.179 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.181 7f0bdd92f700 1 --2- 172.21.2.112:0/2328148877 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0bd818d280 0x7f0bd81a9e00 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:38.179 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.181 7f0bdd92f700 1 --2- 172.21.2.112:0/2328148877 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0bd818c860 0x7f0bd818cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:38.180 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.181 7f0bdd92f700 1 -- 172.21.2.112:0/2328148877 >> 172.21.2.112:0/2328148877 conn(0x7f0bd809b760 msgr2=0x7f0bd8089670 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:38.180 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.181 7f0bdd92f700 1 -- 172.21.2.112:0/2328148877 shutdown_connections 2021-02-10T11:15:38.180 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.181 7f0bdd92f700 1 -- 172.21.2.112:0/2328148877 wait complete. 2021-02-10T11:15:38.187 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:15:38.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.611 7f06723b4700 1 Processor -- start 2021-02-10T11:15:38.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.611 7f06723b4700 1 -- start start 2021-02-10T11:15:38.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.611 7f06723b4700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f066c1b0470 0x7f066c1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:38.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.611 7f06723b4700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f066c0ef610 0x7f066c0efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:38.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.611 7f06723b4700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f066c0f0b50 con 0x7f066c0f0030 2021-02-10T11:15:38.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.611 7f06723b4700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f066c0e52e0 con 0x7f066c0ef610 2021-02-10T11:15:38.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.611 7f06723b4700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f066c0e5490 con 0x7f066c1b0470 2021-02-10T11:15:38.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.612 7f06709d1700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f066c0f0030 0x7f066c0f10d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:43010/0 (socket says 172.21.2.112:43010) 2021-02-10T11:15:38.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.612 7f06709d1700 1 -- 172.21.2.112:0/715966395 learned_addr learned my addr 172.21.2.112:0/715966395 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:15:38.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.612 7f066bfff700 1 --2- 172.21.2.112:0/715966395 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f066c1b0470 0x7f066c1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:38.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.612 7f066b7fe700 1 --2- 172.21.2.112:0/715966395 >> v2:172.21.2.120:3300/0 conn(0x7f066c0ef610 0x7f066c0efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:38.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.612 7f066affd700 1 -- 172.21.2.112:0/715966395 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2461227796 0 0) 0x7f0664001500 con 0x7f066c0f0030 2021-02-10T11:15:38.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.612 7f066affd700 1 -- 172.21.2.112:0/715966395 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0654002fe0 con 0x7f066c0f0030 2021-02-10T11:15:38.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.612 7f066bfff700 1 -- 172.21.2.112:0/715966395 >> v2:172.21.2.120:3300/0 conn(0x7f066c0ef610 msgr2=0x7f066c0efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:15:38.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.612 7f066bfff700 1 --2- 172.21.2.112:0/715966395 >> v2:172.21.2.120:3300/0 conn(0x7f066c0ef610 0x7f066c0efa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:38.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.612 7f066bfff700 1 -- 172.21.2.112:0/715966395 >> v1:172.21.2.112:6789/0 conn(0x7f066c0f0030 legacy=0x7f066c0f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:38.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.612 7f066bfff700 1 -- 172.21.2.112:0/715966395 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f066c0f0430 con 0x7f066c1b0470 2021-02-10T11:15:38.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.612 7f066b7fe700 1 --2- 172.21.2.112:0/715966395 >> v2:172.21.2.120:3300/0 conn(0x7f066c0ef610 0x7f066c0efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:15:38.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.612 7f066bfff700 1 --2- 172.21.2.112:0/715966395 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f066c1b0470 0x7f066c1b0870 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f065c006650 tx=0x7f065c009a80).ready entity=mon.2 client_cookie=eeb9d3e8992b01b7 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:38.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.613 7f066affd700 1 -- 172.21.2.112:0/715966395 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f065c00c0c0 con 0x7f066c1b0470 2021-02-10T11:15:38.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.613 7f066affd700 1 -- 172.21.2.112:0/715966395 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f065c00c340 con 0x7f066c1b0470 2021-02-10T11:15:38.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.613 7f066affd700 1 -- 172.21.2.112:0/715966395 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f065c00c860 con 0x7f066c1b0470 2021-02-10T11:15:38.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.613 7f06723b4700 1 -- 172.21.2.112:0/715966395 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f066c1b0470 msgr2=0x7f066c1b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:38.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.613 7f06723b4700 1 --2- 172.21.2.112:0/715966395 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f066c1b0470 0x7f066c1b0870 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f065c006650 tx=0x7f065c009a80).stop 2021-02-10T11:15:38.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.613 7f06723b4700 1 -- 172.21.2.112:0/715966395 shutdown_connections 2021-02-10T11:15:38.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.613 7f06723b4700 1 --2- 172.21.2.112:0/715966395 >> v2:172.21.2.120:3300/0 conn(0x7f066c0ef610 0x7f066c0efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:38.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.613 7f06723b4700 1 --2- 172.21.2.112:0/715966395 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f066c1b0470 0x7f066c1b0870 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:38.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.613 7f06723b4700 1 -- 172.21.2.112:0/715966395 >> 172.21.2.112:0/715966395 conn(0x7f066c0ce3e0 msgr2=0x7f066c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:38.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.613 7f06723b4700 1 -- 172.21.2.112:0/715966395 shutdown_connections 2021-02-10T11:15:38.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.613 7f06723b4700 1 -- 172.21.2.112:0/715966395 wait complete. 2021-02-10T11:15:38.630 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.628 7f06723b4700 1 Processor -- start 2021-02-10T11:15:38.630 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.628 7f06723b4700 1 -- start start 2021-02-10T11:15:38.630 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.628 7f06723b4700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f066c1a0480 0x7f066c08e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:38.630 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.628 7f06723b4700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f066c10a4b0 0x7f066c08ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:38.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.628 7f06723b4700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f066c18ebf0 0x7f066c18eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:38.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.628 7f06723b4700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f066c18f720 con 0x7f066c1a0480 2021-02-10T11:15:38.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.628 7f06723b4700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f066c18f940 con 0x7f066c18ebf0 2021-02-10T11:15:38.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.628 7f06723b4700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f066c18fb60 con 0x7f066c10a4b0 2021-02-10T11:15:38.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.629 7f066bfff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f066c1a0480 0x7f066c08e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:38.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.629 7f066bfff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f066c1a0480 0x7f066c08e840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:43550/0 (socket says 172.21.2.112:43550) 2021-02-10T11:15:38.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.629 7f066bfff700 1 -- 172.21.2.112:0/40239998 learned_addr learned my addr 172.21.2.112:0/40239998 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:15:38.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.629 7f066bfff700 1 -- 172.21.2.112:0/40239998 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f066c10a4b0 msgr2=0x7f066c08ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:38.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.629 7f066b7fe700 1 --2- 172.21.2.112:0/40239998 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f066c10a4b0 0x7f066c08ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:38.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.629 7f066bfff700 1 --2- 172.21.2.112:0/40239998 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f066c10a4b0 0x7f066c08ee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:38.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.629 7f066bfff700 1 -- 172.21.2.112:0/40239998 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f066c18ebf0 msgr2=0x7f066c18eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:38.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.629 7f06709d1700 1 --2- 172.21.2.112:0/40239998 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f066c18ebf0 0x7f066c18eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:38.632 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.629 7f066bfff700 1 --2- 172.21.2.112:0/40239998 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f066c18ebf0 0x7f066c18eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:38.632 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.629 7f066bfff700 1 -- 172.21.2.112:0/40239998 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f066c18fd80 con 0x7f066c1a0480 2021-02-10T11:15:38.632 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.629 7f066b7fe700 1 --2- 172.21.2.112:0/40239998 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f066c10a4b0 0x7f066c08ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:15:38.632 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.629 7f06709d1700 1 --2- 172.21.2.112:0/40239998 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f066c18ebf0 0x7f066c18eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:15:38.632 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.629 7f066bfff700 1 --2- 172.21.2.112:0/40239998 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f066c1a0480 0x7f066c08e840 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7f065c009650 tx=0x7f065c00b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:38.632 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.629 7f0668ff9700 1 -- 172.21.2.112:0/40239998 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f065c0093a0 con 0x7f066c1a0480 2021-02-10T11:15:38.632 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.629 7f0668ff9700 1 -- 172.21.2.112:0/40239998 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f065c00bde0 con 0x7f066c1a0480 2021-02-10T11:15:38.632 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.629 7f0668ff9700 1 -- 172.21.2.112:0/40239998 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f065c0052e0 con 0x7f066c1a0480 2021-02-10T11:15:38.632 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.629 7f06723b4700 1 -- 172.21.2.112:0/40239998 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f066c18fd80 con 0x7f066c1a0480 2021-02-10T11:15:38.632 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.630 7f06723b4700 1 -- 172.21.2.112:0/40239998 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f066c18fd80 con 0x7f066c1a0480 2021-02-10T11:15:38.632 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.630 7f0668ff9700 1 -- 172.21.2.112:0/40239998 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f065c009300 con 0x7f066c1a0480 2021-02-10T11:15:38.632 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.630 7f0668ff9700 1 --2- 172.21.2.112:0/40239998 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f064c0186c0 0x7f064c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:38.632 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.630 7f0668ff9700 1 -- 172.21.2.112:0/40239998 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f065c032540 con 0x7f066c1a0480 2021-02-10T11:15:38.633 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.631 7f066b7fe700 1 --2- 172.21.2.112:0/40239998 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f064c0186c0 0x7f064c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:38.633 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.631 7f066b7fe700 1 --2- 172.21.2.112:0/40239998 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f064c0186c0 0x7f064c01abc0 crc :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:38.660 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.661 7f06723b4700 1 -- 172.21.2.112:0/40239998 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f066c1059f0 con 0x7f066c1a0480 2021-02-10T11:15:38.662 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.664 7f0668ff9700 1 -- 172.21.2.112:0/40239998 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f065c02e370 con 0x7f066c1a0480 2021-02-10T11:15:38.793 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.795 7f06723b4700 1 -- 172.21.2.112:0/40239998 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f066c1059f0 con 0x7f066c1a0480 2021-02-10T11:15:38.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.798 7f0668ff9700 1 -- 172.21.2.112:0/40239998 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v14) v1 ==== 184+0+0 (secure 0 0 0) 0x7f065c009300 con 0x7f066c1a0480 2021-02-10T11:15:38.800 INFO:teuthology.orchestra.run.gibba012.stderr:updated caps for client.1 2021-02-10T11:15:38.802 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.804 7f06723b4700 1 -- 172.21.2.112:0/40239998 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f064c0186c0 msgr2=0x7f064c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:38.802 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.804 7f06723b4700 1 --2- 172.21.2.112:0/40239998 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f064c0186c0 0x7f064c01abc0 crc :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:38.802 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.804 7f06723b4700 1 -- 172.21.2.112:0/40239998 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f066c1a0480 msgr2=0x7f066c08e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:38.802 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.804 7f06723b4700 1 --2- 172.21.2.112:0/40239998 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f066c1a0480 0x7f066c08e840 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7f065c009650 tx=0x7f065c00b960).stop 2021-02-10T11:15:38.802 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.804 7f06723b4700 1 -- 172.21.2.112:0/40239998 shutdown_connections 2021-02-10T11:15:38.802 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.804 7f06723b4700 1 --2- 172.21.2.112:0/40239998 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f064c0186c0 0x7f064c01abc0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:38.802 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.804 7f06723b4700 1 --2- 172.21.2.112:0/40239998 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f066c18ebf0 0x7f066c18eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:38.802 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.804 7f06723b4700 1 --2- 172.21.2.112:0/40239998 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f066c10a4b0 0x7f066c08ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:38.802 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.804 7f06723b4700 1 --2- 172.21.2.112:0/40239998 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f066c1a0480 0x7f066c08e840 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:38.802 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.804 7f06723b4700 1 -- 172.21.2.112:0/40239998 >> 172.21.2.112:0/40239998 conn(0x7f066c09cd10 msgr2=0x7f066c093a20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:38.803 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.804 7f06723b4700 1 -- 172.21.2.112:0/40239998 shutdown_connections 2021-02-10T11:15:38.803 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:38.804 7f06723b4700 1 -- 172.21.2.112:0/40239998 wait complete. 2021-02-10T11:15:38.811 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.2 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:15:39.250 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.235 7fc2ef24f700 1 Processor -- start 2021-02-10T11:15:39.250 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.235 7fc2ef24f700 1 -- start start 2021-02-10T11:15:39.250 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.236 7fc2ef24f700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fc2e81b0870 0x7fc2e80f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:39.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.236 7fc2ef24f700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fc2e80efe00 0x7fc2e80f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:39.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.236 7fc2ef24f700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2e81b0c70 con 0x7fc2e81b0470 2021-02-10T11:15:39.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.236 7fc2ef24f700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2e80e5ea0 con 0x7fc2e80efe00 2021-02-10T11:15:39.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.236 7fc2ef24f700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2e80f0b50 con 0x7fc2e81b0870 2021-02-10T11:15:39.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.236 7fc2ed86c700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fc2e80efe00 0x7fc2e80f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:39.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.236 7fc2ed86c700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fc2e80efe00 0x7fc2e80f0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:43562/0 (socket says 172.21.2.112:43562) 2021-02-10T11:15:39.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.236 7fc2ed86c700 1 -- 172.21.2.112:0/3563428447 learned_addr learned my addr 172.21.2.112:0/3563428447 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:15:39.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.236 7fc2ed86c700 1 -- 172.21.2.112:0/3563428447 >> v2:172.21.2.120:3300/0 conn(0x7fc2e81b0870 msgr2=0x7fc2e80f20e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:15:39.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.236 7fc2ed86c700 1 --2- 172.21.2.112:0/3563428447 >> v2:172.21.2.120:3300/0 conn(0x7fc2e81b0870 0x7fc2e80f20e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:39.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.236 7fc2ed86c700 1 -- 172.21.2.112:0/3563428447 >> v1:172.21.2.120:6789/0 conn(0x7fc2e81b0470 legacy=0x7fc2e80ef610 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:15:39.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.236 7fc2ed86c700 1 -- 172.21.2.112:0/3563428447 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2e80f0820 con 0x7fc2e80efe00 2021-02-10T11:15:39.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.236 7fc2ed86c700 1 --2- 172.21.2.112:0/3563428447 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc2e80efe00 0x7fc2e80f0200 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0x7fc2e0006f00 tx=0x7fc2e0009f10).ready entity=mon.0 client_cookie=dc4084bcae96880c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:39.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.237 7fc2dbfff700 1 -- 172.21.2.112:0/3563428447 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc2e000b3b0 con 0x7fc2e80efe00 2021-02-10T11:15:39.252 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.237 7fc2dbfff700 1 -- 172.21.2.112:0/3563428447 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2e000b3b0 con 0x7fc2e80efe00 2021-02-10T11:15:39.252 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.237 7fc2dbfff700 1 -- 172.21.2.112:0/3563428447 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc2e000b630 con 0x7fc2e80efe00 2021-02-10T11:15:39.252 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.237 7fc2ef24f700 1 -- 172.21.2.112:0/3563428447 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc2e80efe00 msgr2=0x7fc2e80f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:39.252 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.237 7fc2ef24f700 1 --2- 172.21.2.112:0/3563428447 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc2e80efe00 0x7fc2e80f0200 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0x7fc2e0006f00 tx=0x7fc2e0009f10).stop 2021-02-10T11:15:39.252 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.237 7fc2ef24f700 1 -- 172.21.2.112:0/3563428447 shutdown_connections 2021-02-10T11:15:39.252 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.237 7fc2ef24f700 1 --2- 172.21.2.112:0/3563428447 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc2e80efe00 0x7fc2e80f0200 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:39.252 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.237 7fc2ef24f700 1 --2- 172.21.2.112:0/3563428447 >> v2:172.21.2.120:3300/0 conn(0x7fc2e81b0870 0x7fc2e80f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:39.252 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.237 7fc2ef24f700 1 -- 172.21.2.112:0/3563428447 >> 172.21.2.112:0/3563428447 conn(0x7fc2e80ce3e0 msgr2=0x7fc2e80e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:39.252 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.237 7fc2ef24f700 1 -- 172.21.2.112:0/3563428447 shutdown_connections 2021-02-10T11:15:39.252 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.238 7fc2ef24f700 1 -- 172.21.2.112:0/3563428447 wait complete. 2021-02-10T11:15:39.252 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.252 7fc2ef24f700 1 Processor -- start 2021-02-10T11:15:39.252 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.252 7fc2ef24f700 1 -- start start 2021-02-10T11:15:39.254 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.253 7fc2ef24f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc2e8099520 0x7fc2e8099920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:39.254 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.253 7fc2ef24f700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc2e8195af0 0x7fc2e8195ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:39.254 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.253 7fc2ef24f700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc2e80f0890 0x7fc2e80f0c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:39.254 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.253 7fc2ef24f700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2e81955f0 con 0x7fc2e8099520 2021-02-10T11:15:39.254 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.253 7fc2ef24f700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2e81b29a0 con 0x7fc2e8195af0 2021-02-10T11:15:39.255 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.253 7fc2ef24f700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2e80f12b0 con 0x7fc2e80f0890 2021-02-10T11:15:39.255 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.253 7fc2ed06b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc2e8099520 0x7fc2e8099920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:39.255 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.253 7fc2ed06b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc2e8099520 0x7fc2e8099920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:43564/0 (socket says 172.21.2.112:43564) 2021-02-10T11:15:39.255 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.253 7fc2ed06b700 1 -- 172.21.2.112:0/3848206085 learned_addr learned my addr 172.21.2.112:0/3848206085 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:15:39.255 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.253 7fc2ec86a700 1 --2- 172.21.2.112:0/3848206085 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc2e8195af0 0x7fc2e8195ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:39.255 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.253 7fc2ed86c700 1 --2- 172.21.2.112:0/3848206085 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc2e80f0890 0x7fc2e80f0c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:39.255 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.253 7fc2ed06b700 1 -- 172.21.2.112:0/3848206085 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc2e80f0890 msgr2=0x7fc2e80f0c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:39.255 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.253 7fc2ed06b700 1 --2- 172.21.2.112:0/3848206085 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc2e80f0890 0x7fc2e80f0c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:39.255 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.253 7fc2ed06b700 1 -- 172.21.2.112:0/3848206085 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc2e8195af0 msgr2=0x7fc2e8195ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:39.255 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.253 7fc2ed06b700 1 --2- 172.21.2.112:0/3848206085 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc2e8195af0 0x7fc2e8195ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:39.255 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.254 7fc2ed06b700 1 -- 172.21.2.112:0/3848206085 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2e80f14d0 con 0x7fc2e8099520 2021-02-10T11:15:39.255 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.254 7fc2ed06b700 1 --2- 172.21.2.112:0/3848206085 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc2e8099520 0x7fc2e8099920 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7fc2dc00b5b0 tx=0x7fc2dc00b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:39.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.254 7fc2d9ffb700 1 -- 172.21.2.112:0/3848206085 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc2dc00bd40 con 0x7fc2e8099520 2021-02-10T11:15:39.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.254 7fc2d9ffb700 1 -- 172.21.2.112:0/3848206085 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2dc006240 con 0x7fc2e8099520 2021-02-10T11:15:39.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.254 7fc2d9ffb700 1 -- 172.21.2.112:0/3848206085 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc2dc005720 con 0x7fc2e8099520 2021-02-10T11:15:39.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.254 7fc2ef24f700 1 -- 172.21.2.112:0/3848206085 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2e80f14d0 con 0x7fc2e8099520 2021-02-10T11:15:39.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.254 7fc2ef24f700 1 -- 172.21.2.112:0/3848206085 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2e80f1a60 con 0x7fc2e8099520 2021-02-10T11:15:39.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.255 7fc2d9ffb700 1 -- 172.21.2.112:0/3848206085 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc2dc0054a0 con 0x7fc2e8099520 2021-02-10T11:15:39.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.255 7fc2d9ffb700 1 --2- 172.21.2.112:0/3848206085 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc2c8018730 0x7fc2c801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:39.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.255 7fc2d9ffb700 1 -- 172.21.2.112:0/3848206085 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fc2dc0327e0 con 0x7fc2e8099520 2021-02-10T11:15:39.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.255 7fc2ec86a700 1 --2- 172.21.2.112:0/3848206085 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc2c8018730 0x7fc2c801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:39.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.255 7fc2ec86a700 1 --2- 172.21.2.112:0/3848206085 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc2c8018730 0x7fc2c801abc0 crc :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:39.284 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.286 7fc2ef24f700 1 -- 172.21.2.112:0/3848206085 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2e80edd30 con 0x7fc2e8099520 2021-02-10T11:15:39.287 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.289 7fc2d9ffb700 1 -- 172.21.2.112:0/3848206085 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc2dc000c60 con 0x7fc2e8099520 2021-02-10T11:15:39.422 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.424 7fc2ef24f700 1 -- 172.21.2.112:0/3848206085 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fc2e80e52e0 con 0x7fc2e8099520 2021-02-10T11:15:39.426 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.427 7fc2d9ffb700 1 -- 172.21.2.112:0/3848206085 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.2 v15) v1 ==== 184+0+0 (secure 0 0 0) 0x7fc2dc0054a0 con 0x7fc2e8099520 2021-02-10T11:15:39.429 INFO:teuthology.orchestra.run.gibba012.stderr:updated caps for client.2 2021-02-10T11:15:39.430 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.433 7fc2ef24f700 1 -- 172.21.2.112:0/3848206085 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc2c8018730 msgr2=0x7fc2c801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:39.430 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.433 7fc2ef24f700 1 --2- 172.21.2.112:0/3848206085 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc2c8018730 0x7fc2c801abc0 crc :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:39.431 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.433 7fc2ef24f700 1 -- 172.21.2.112:0/3848206085 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc2e8099520 msgr2=0x7fc2e8099920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:39.431 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.433 7fc2ef24f700 1 --2- 172.21.2.112:0/3848206085 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc2e8099520 0x7fc2e8099920 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7fc2dc00b5b0 tx=0x7fc2dc00b950).stop 2021-02-10T11:15:39.431 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.433 7fc2ef24f700 1 -- 172.21.2.112:0/3848206085 shutdown_connections 2021-02-10T11:15:39.431 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.433 7fc2ef24f700 1 --2- 172.21.2.112:0/3848206085 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc2c8018730 0x7fc2c801abc0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:39.432 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.433 7fc2ef24f700 1 --2- 172.21.2.112:0/3848206085 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc2e8195af0 0x7fc2e8195ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:39.432 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.433 7fc2ef24f700 1 --2- 172.21.2.112:0/3848206085 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc2e80f0890 0x7fc2e80f0c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:39.432 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.433 7fc2ef24f700 1 --2- 172.21.2.112:0/3848206085 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc2e8099520 0x7fc2e8099920 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:39.432 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.433 7fc2ef24f700 1 -- 172.21.2.112:0/3848206085 >> 172.21.2.112:0/3848206085 conn(0x7fc2e81a1590 msgr2=0x7fc2e81b2e60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:39.432 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.433 7fc2ef24f700 1 -- 172.21.2.112:0/3848206085 shutdown_connections 2021-02-10T11:15:39.432 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.433 7fc2ef24f700 1 -- 172.21.2.112:0/3848206085 wait complete. 2021-02-10T11:15:39.440 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.3 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:15:39.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.855 7f893db09700 1 Processor -- start 2021-02-10T11:15:39.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.855 7f893db09700 1 -- start start 2021-02-10T11:15:39.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.855 7f893db09700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f89381b0470 0x7f89381b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:39.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.855 7f893db09700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f89380ef610 0x7f89380efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:39.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.855 7f893db09700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f89380f0030 0x7f89380f0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:39.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.855 7f893db09700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89380e5ea0 con 0x7f89380f0030 2021-02-10T11:15:39.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.855 7f893db09700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89380f0b50 con 0x7f89380ef610 2021-02-10T11:15:39.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.855 7f893db09700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89380e52e0 con 0x7f89381b0470 2021-02-10T11:15:39.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.855 7f8937fff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f89380f0030 0x7f89380f0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:39.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.855 7f8937fff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f89380f0030 0x7f89380f0430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:43572/0 (socket says 172.21.2.112:43572) 2021-02-10T11:15:39.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.855 7f8937fff700 1 -- 172.21.2.112:0/344179253 learned_addr learned my addr 172.21.2.112:0/344179253 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:15:39.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.856 7f8937fff700 1 -- 172.21.2.112:0/344179253 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f89381b0470 msgr2=0x7f89381b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:15:39.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.856 7f8937fff700 1 --2- 172.21.2.112:0/344179253 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f89381b0470 0x7f89381b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:39.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.856 7f8937fff700 1 -- 172.21.2.112:0/344179253 >> v2:172.21.2.120:3300/0 conn(0x7f89380ef610 msgr2=0x7f89380efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:15:39.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.856 7f8937fff700 1 --2- 172.21.2.112:0/344179253 >> v2:172.21.2.120:3300/0 conn(0x7f89380ef610 0x7f89380efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:39.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.856 7f8937fff700 1 -- 172.21.2.112:0/344179253 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89380f10d0 con 0x7f89380f0030 2021-02-10T11:15:39.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.856 7f8937fff700 1 --2- 172.21.2.112:0/344179253 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f89380f0030 0x7f89380f0430 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7f8930006f00 tx=0x7f8930009f10).ready entity=mon.0 client_cookie=70efc36cd6510099 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:39.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.856 7f89367fc700 1 -- 172.21.2.112:0/344179253 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f893000b3b0 con 0x7f89380f0030 2021-02-10T11:15:39.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.856 7f89367fc700 1 -- 172.21.2.112:0/344179253 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f893000b730 con 0x7f89380f0030 2021-02-10T11:15:39.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.856 7f89367fc700 1 -- 172.21.2.112:0/344179253 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f893000bc10 con 0x7f89380f0030 2021-02-10T11:15:39.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.856 7f893db09700 1 -- 172.21.2.112:0/344179253 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f89380f0030 msgr2=0x7f89380f0430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:39.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.856 7f893db09700 1 --2- 172.21.2.112:0/344179253 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f89380f0030 0x7f89380f0430 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7f8930006f00 tx=0x7f8930009f10).stop 2021-02-10T11:15:39.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.857 7f893db09700 1 -- 172.21.2.112:0/344179253 shutdown_connections 2021-02-10T11:15:39.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.857 7f893db09700 1 --2- 172.21.2.112:0/344179253 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f89380f0030 0x7f89380f0430 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:39.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.857 7f893db09700 1 --2- 172.21.2.112:0/344179253 >> v2:172.21.2.120:3300/0 conn(0x7f89380ef610 0x7f89380efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:39.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.857 7f893db09700 1 --2- 172.21.2.112:0/344179253 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f89381b0470 0x7f89381b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:39.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.857 7f893db09700 1 -- 172.21.2.112:0/344179253 >> 172.21.2.112:0/344179253 conn(0x7f89380ce3e0 msgr2=0x7f89380e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:39.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.857 7f893db09700 1 -- 172.21.2.112:0/344179253 shutdown_connections 2021-02-10T11:15:39.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.857 7f893db09700 1 -- 172.21.2.112:0/344179253 wait complete. 2021-02-10T11:15:39.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.871 7f893db09700 1 Processor -- start 2021-02-10T11:15:39.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.872 7f893db09700 1 -- start start 2021-02-10T11:15:39.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.872 7f893db09700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f89381b6de0 0x7f89381b63f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:39.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.872 7f893db09700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f893818f3f0 0x7f893818f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:39.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.872 7f893db09700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f893818fe10 0x7f893818ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:39.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.872 7f893db09700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f893818d950 con 0x7f893818f3f0 2021-02-10T11:15:39.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.872 7f893db09700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89381b71e0 con 0x7f89381b6de0 2021-02-10T11:15:39.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.872 7f893db09700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89381b7400 con 0x7f893818fe10 2021-02-10T11:15:39.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.872 7f8936ffd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f893818f3f0 0x7f893818f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:39.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.872 7f8936ffd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f893818f3f0 0x7f893818f7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:43574/0 (socket says 172.21.2.112:43574) 2021-02-10T11:15:39.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.872 7f8936ffd700 1 -- 172.21.2.112:0/1192847618 learned_addr learned my addr 172.21.2.112:0/1192847618 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:15:39.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.872 7f8936ffd700 1 -- 172.21.2.112:0/1192847618 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f893818fe10 msgr2=0x7f893818ddf0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:15:39.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.872 7f8936ffd700 1 --2- 172.21.2.112:0/1192847618 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f893818fe10 0x7f893818ddf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:39.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.872 7f8936ffd700 1 -- 172.21.2.112:0/1192847618 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f89381b6de0 msgr2=0x7f89381b63f0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:15:39.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.872 7f8936ffd700 1 --2- 172.21.2.112:0/1192847618 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f89381b6de0 0x7f89381b63f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:39.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.872 7f8936ffd700 1 -- 172.21.2.112:0/1192847618 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8938190210 con 0x7f893818f3f0 2021-02-10T11:15:39.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.873 7f8936ffd700 1 --2- 172.21.2.112:0/1192847618 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f893818f3f0 0x7f893818f7f0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7f892c006f00 tx=0x7f892c00b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:39.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.873 7f891ffff700 1 -- 172.21.2.112:0/1192847618 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f892c0009b0 con 0x7f893818f3f0 2021-02-10T11:15:39.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.873 7f891ffff700 1 -- 172.21.2.112:0/1192847618 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f892c006350 con 0x7f893818f3f0 2021-02-10T11:15:39.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.873 7f891ffff700 1 -- 172.21.2.112:0/1192847618 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f892c000c90 con 0x7f893818f3f0 2021-02-10T11:15:39.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.873 7f893db09700 1 -- 172.21.2.112:0/1192847618 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8938190210 con 0x7f893818f3f0 2021-02-10T11:15:39.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.873 7f893db09700 1 -- 172.21.2.112:0/1192847618 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8938190210 con 0x7f893818f3f0 2021-02-10T11:15:39.875 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.874 7f891ffff700 1 -- 172.21.2.112:0/1192847618 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f892c007000 con 0x7f893818f3f0 2021-02-10T11:15:39.875 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.874 7f891ffff700 1 --2- 172.21.2.112:0/1192847618 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f89180186c0 0x7f891801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:39.875 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.874 7f891ffff700 1 -- 172.21.2.112:0/1192847618 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f892c032570 con 0x7f893818f3f0 2021-02-10T11:15:39.875 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.874 7f89377fe700 1 --2- 172.21.2.112:0/1192847618 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f89180186c0 0x7f891801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:39.875 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.875 7f89377fe700 1 --2- 172.21.2.112:0/1192847618 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f89180186c0 0x7f891801abc0 crc :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:39.903 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.905 7f893db09700 1 -- 172.21.2.112:0/1192847618 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89381059e0 con 0x7f893818f3f0 2021-02-10T11:15:39.905 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:39.907 7f891ffff700 1 -- 172.21.2.112:0/1192847618 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f892c026010 con 0x7f893818f3f0 2021-02-10T11:15:40.033 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:40.035 7f893db09700 1 -- 172.21.2.112:0/1192847618 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f89380f0b50 con 0x7f893818f3f0 2021-02-10T11:15:40.036 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:40.039 7f891ffff700 1 -- 172.21.2.112:0/1192847618 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.3 v16) v1 ==== 184+0+0 (secure 0 0 0) 0x7f892c007000 con 0x7f893818f3f0 2021-02-10T11:15:40.040 INFO:teuthology.orchestra.run.gibba012.stderr:updated caps for client.3 2021-02-10T11:15:40.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:40.044 7f893db09700 1 -- 172.21.2.112:0/1192847618 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f89180186c0 msgr2=0x7f891801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:40.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:40.044 7f893db09700 1 --2- 172.21.2.112:0/1192847618 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f89180186c0 0x7f891801abc0 crc :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:40.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:40.044 7f893db09700 1 -- 172.21.2.112:0/1192847618 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f893818f3f0 msgr2=0x7f893818f7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:40.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:40.044 7f893db09700 1 --2- 172.21.2.112:0/1192847618 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f893818f3f0 0x7f893818f7f0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7f892c006f00 tx=0x7f892c00b940).stop 2021-02-10T11:15:40.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:40.044 7f893db09700 1 -- 172.21.2.112:0/1192847618 shutdown_connections 2021-02-10T11:15:40.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:40.044 7f893db09700 1 --2- 172.21.2.112:0/1192847618 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f89180186c0 0x7f891801abc0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:40.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:40.044 7f893db09700 1 --2- 172.21.2.112:0/1192847618 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f893818fe10 0x7f893818ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:40.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:40.044 7f893db09700 1 --2- 172.21.2.112:0/1192847618 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f893818f3f0 0x7f893818f7f0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:40.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:40.044 7f893db09700 1 --2- 172.21.2.112:0/1192847618 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f89381b6de0 0x7f89381b63f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:40.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:40.044 7f893db09700 1 -- 172.21.2.112:0/1192847618 >> 172.21.2.112:0/1192847618 conn(0x7f89381a37a0 msgr2=0x7f893818c230 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:40.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:40.044 7f893db09700 1 -- 172.21.2.112:0/1192847618 shutdown_connections 2021-02-10T11:15:40.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:40.044 7f893db09700 1 -- 172.21.2.112:0/1192847618 wait complete. 2021-02-10T11:15:40.050 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:15:40.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:40.461 7f27fa3e9700 1 Processor -- start 2021-02-10T11:15:40.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:40.461 7f27fa3e9700 1 -- start start 2021-02-10T11:15:40.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:40.461 7f27fa3e9700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f27f40e5470 0x7f27f407e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:40.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:40.461 7f27fa3e9700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27f407ed80 con 0x7f27f407f300 2021-02-10T11:15:40.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:40.461 7f27fa3e9700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27f41b4010 con 0x7f27f40e5070 2021-02-10T11:15:40.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:40.461 7f27fa3e9700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27f40d1790 con 0x7f27f40e5470 2021-02-10T11:15:40.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:40.462 7f27f37fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f27f40e5470 0x7f27f407e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:40.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:40.462 7f27f8a06700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f27f407f300 0x7f27f407f700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:43048/0 (socket says 172.21.2.112:43048) 2021-02-10T11:15:40.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:40.462 7f27f8a06700 1 -- 172.21.2.112:0/3912049722 learned_addr learned my addr 172.21.2.112:0/3912049722 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:15:40.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:40.462 7f27f37fe700 1 --2- 172.21.2.112:0/3912049722 >> v2:172.21.2.112:3300/0 conn(0x7f27f40e5470 0x7f27f407e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:43582/0 (socket says 172.21.2.112:43582) 2021-02-10T11:15:40.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:40.462 7f27f37fe700 1 -- 172.21.2.112:0/3912049722 >> v1:172.21.2.120:6789/0 conn(0x7f27f40e5070 legacy=0x7f27f41b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:40.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:40.462 7f27f37fe700 1 -- 172.21.2.112:0/3912049722 >> v1:172.21.2.112:6789/0 conn(0x7f27f407f300 legacy=0x7f27f407f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:40.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:40.462 7f27f37fe700 1 -- 172.21.2.112:0/3912049722 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27f40d19b0 con 0x7f27f40e5470 2021-02-10T11:15:40.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:40.462 7f27f37fe700 1 --2- 172.21.2.112:0/3912049722 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f27f40e5470 0x7f27f407e6a0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7f27e8006650 tx=0x7f27e8009a80).ready entity=mon.0 client_cookie=4393040f4394253f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:40.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:40.462 7f27f2ffd700 1 -- 172.21.2.112:0/3912049722 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f27e80009b0 con 0x7f27f40e5470 2021-02-10T11:15:40.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:40.462 7f27f2ffd700 1 -- 172.21.2.112:0/3912049722 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27e800c0c0 con 0x7f27f40e5470 2021-02-10T11:15:40.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:40.462 7f27f2ffd700 1 -- 172.21.2.112:0/3912049722 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f27e800c510 con 0x7f27f40e5470 2021-02-10T11:15:40.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:40.462 7f27fa3e9700 1 -- 172.21.2.112:0/3912049722 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f27f40e5470 msgr2=0x7f27f407e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:40.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:40.463 7f27fa3e9700 1 --2- 172.21.2.112:0/3912049722 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f27f40e5470 0x7f27f407e6a0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7f27e8006650 tx=0x7f27e8009a80).stop 2021-02-10T11:15:40.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:40.463 7f27fa3e9700 1 -- 172.21.2.112:0/3912049722 shutdown_connections 2021-02-10T11:15:40.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:40.463 7f27fa3e9700 1 --2- 172.21.2.112:0/3912049722 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f27f40e5470 0x7f27f407e6a0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:40.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:40.463 7f27fa3e9700 1 -- 172.21.2.112:0/3912049722 >> 172.21.2.112:0/3912049722 conn(0x7f27f40ce690 msgr2=0x7f27f40e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:40.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:40.463 7f27fa3e9700 1 -- 172.21.2.112:0/3912049722 shutdown_connections 2021-02-10T11:15:40.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:40.463 7f27fa3e9700 1 -- 172.21.2.112:0/3912049722 wait complete. 2021-02-10T11:15:40.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:40.478 7f27fa3e9700 1 Processor -- start 2021-02-10T11:15:40.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:40.478 7f27fa3e9700 1 -- start start 2021-02-10T11:15:40.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:40.478 7f27fa3e9700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f27f4096380 0x7f27f4090ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:40.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:40.478 7f27fa3e9700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f27f4096780 0x7f27f4091100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:40.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:40.478 7f27fa3e9700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f27f4091720 0x7f27f4097ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:40.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:40.478 7f27fa3e9700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27f4108870 con 0x7f27f4096380 2021-02-10T11:15:40.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:40.478 7f27fa3e9700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27f4096b80 con 0x7f27f4091720 2021-02-10T11:15:40.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:40.478 7f27fa3e9700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27f4090560 con 0x7f27f4096780 2021-02-10T11:15:40.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:40.478 7f27f3fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f27f4096380 0x7f27f4090ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:40.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:40.479 7f27f3fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f27f4096380 0x7f27f4090ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:43588/0 (socket says 172.21.2.112:43588) 2021-02-10T11:15:40.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:40.479 7f27f8a06700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f27f4091720 0x7f27f4097ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:40.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:40.479 7f27f3fff700 1 -- 172.21.2.112:0/1218774929 learned_addr learned my addr 172.21.2.112:0/1218774929 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:15:40.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:40.479 7f27f37fe700 1 --2- 172.21.2.112:0/1218774929 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f27f4096780 0x7f27f4091100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:40.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:40.479 7f27f8a06700 1 -- 172.21.2.112:0/1218774929 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f27f4096780 msgr2=0x7f27f4091100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:40.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:40.479 7f27f8a06700 1 --2- 172.21.2.112:0/1218774929 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f27f4096780 0x7f27f4091100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:40.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:40.479 7f27f8a06700 1 -- 172.21.2.112:0/1218774929 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f27f4096380 msgr2=0x7f27f4090ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:40.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:40.479 7f27f8a06700 1 --2- 172.21.2.112:0/1218774929 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f27f4096380 0x7f27f4090ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:40.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:40.479 7f27f8a06700 1 -- 172.21.2.112:0/1218774929 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27f4108440 con 0x7f27f4091720 2021-02-10T11:15:40.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:40.479 7f27f8a06700 1 --2- 172.21.2.112:0/1218774929 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f27f4091720 0x7f27f4097ad0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f27ec007f70 tx=0x7f27ec00ac20).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:40.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:40.479 7f27f0ff9700 1 -- 172.21.2.112:0/1218774929 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f27ec001270 con 0x7f27f4091720 2021-02-10T11:15:40.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:40.479 7f27f0ff9700 1 -- 172.21.2.112:0/1218774929 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27ec007800 con 0x7f27f4091720 2021-02-10T11:15:40.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:40.479 7f27fa3e9700 1 -- 172.21.2.112:0/1218774929 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27f4108440 con 0x7f27f4091720 2021-02-10T11:15:40.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:40.479 7f27f0ff9700 1 -- 172.21.2.112:0/1218774929 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f27ec004b20 con 0x7f27f4091720 2021-02-10T11:15:40.481 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:40.479 7f27fa3e9700 1 -- 172.21.2.112:0/1218774929 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27f4108440 con 0x7f27f4091720 2021-02-10T11:15:40.481 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:40.480 7f27f0ff9700 1 -- 172.21.2.112:0/1218774929 <== mon.1 v2:172.21.2.120:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f27ec00ad60 con 0x7f27f4091720 2021-02-10T11:15:40.481 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:40.480 7f27f0ff9700 1 --2- 172.21.2.112:0/1218774929 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f27d40186c0 0x7f27d401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:40.481 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:40.480 7f27f0ff9700 1 -- 172.21.2.112:0/1218774929 <== mon.1 v2:172.21.2.120:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f27ec0326b0 con 0x7f27f4091720 2021-02-10T11:15:40.481 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:40.480 7f27f3fff700 1 --2- 172.21.2.112:0/1218774929 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f27d40186c0 0x7f27d401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:40.481 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:40.481 7f27f3fff700 1 --2- 172.21.2.112:0/1218774929 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f27d40186c0 0x7f27d401abc0 crc :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:40.509 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:40.511 7f27fa3e9700 1 -- 172.21.2.112:0/1218774929 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27f40891c0 con 0x7f27f4091720 2021-02-10T11:15:40.512 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:40.514 7f27f0ff9700 1 -- 172.21.2.112:0/1218774929 <== mon.1 v2:172.21.2.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f27ec000d30 con 0x7f27f4091720 2021-02-10T11:15:40.639 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:40.641 7f27fa3e9700 1 -- 172.21.2.112:0/1218774929 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f27f40ce690 con 0x7f27f4091720 2021-02-10T11:15:40.640 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:40.643 7f27f0ff9700 1 -- 172.21.2.112:0/1218774929 <== mon.1 v2:172.21.2.120:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+2100 (secure 0 0 0) 0x7f27ec00ad60 con 0x7f27f4091720 2021-02-10T11:15:40.642 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 13 2021-02-10T11:15:40.643 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:15:40.643 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":13,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4627,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/2302110857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":2302110857},{"type":"v1","addr":"172.21.2.112:6833","nonce":2302110857}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":9}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:15:33.919772","modified":"2021-02-10 11:15:36.309405","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4427},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4427":{"gid":4427,"name":"b","rank":0,"incarnation":10,"state":"up:active","state_seq":4,"addr":"172.21.2.120:6835/3730022736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":3730022736},{"type":"v1","addr":"172.21.2.120:6835","nonce":3730022736}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2021-02-10T11:15:40.644 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:40.645 7f27fa3e9700 1 -- 172.21.2.112:0/1218774929 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f27d40186c0 msgr2=0x7f27d401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:40.644 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:40.645 7f27fa3e9700 1 --2- 172.21.2.112:0/1218774929 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f27d40186c0 0x7f27d401abc0 crc :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:40.644 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:40.645 7f27fa3e9700 1 -- 172.21.2.112:0/1218774929 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f27f4091720 msgr2=0x7f27f4097ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:40.644 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:40.646 7f27fa3e9700 1 --2- 172.21.2.112:0/1218774929 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f27f4091720 0x7f27f4097ad0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f27ec007f70 tx=0x7f27ec00ac20).stop 2021-02-10T11:15:40.644 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:40.646 7f27fa3e9700 1 -- 172.21.2.112:0/1218774929 shutdown_connections 2021-02-10T11:15:40.644 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:40.646 7f27fa3e9700 1 --2- 172.21.2.112:0/1218774929 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f27d40186c0 0x7f27d401abc0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:40.644 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:40.646 7f27fa3e9700 1 --2- 172.21.2.112:0/1218774929 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f27f4091720 0x7f27f4097ad0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:40.644 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:40.646 7f27fa3e9700 1 --2- 172.21.2.112:0/1218774929 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f27f4096780 0x7f27f4091100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:40.644 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:40.646 7f27fa3e9700 1 --2- 172.21.2.112:0/1218774929 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f27f4096380 0x7f27f4090ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:40.644 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:40.646 7f27fa3e9700 1 -- 172.21.2.112:0/1218774929 >> 172.21.2.112:0/1218774929 conn(0x7f27f40b07f0 msgr2=0x7f27f4107250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:40.645 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:40.646 7f27fa3e9700 1 -- 172.21.2.112:0/1218774929 shutdown_connections 2021-02-10T11:15:40.645 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:40.646 7f27fa3e9700 1 -- 172.21.2.112:0/1218774929 wait complete. 2021-02-10T11:15:40.854 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 13, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:15:33.919772', 'modified': '2021-02-10 11:15:36.309405', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4427}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4427': {'gid': 4427, 'name': 'b', 'rank': 0, 'incarnation': 10, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.2.120:6835/3730022736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.120:6834', 'nonce': 3730022736}, {'type': 'v1', 'addr': '172.21.2.120:6835', 'nonce': 3730022736}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:15:40.854 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:15:40.854 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:15:41.253 INFO:teuthology.orchestra.run.gibba020.stdout:{ 2021-02-10T11:15:41.254 INFO:teuthology.orchestra.run.gibba020.stdout: "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", 2021-02-10T11:15:41.254 INFO:teuthology.orchestra.run.gibba020.stdout: "whoami": 0, 2021-02-10T11:15:41.254 INFO:teuthology.orchestra.run.gibba020.stdout: "id": 4427, 2021-02-10T11:15:41.254 INFO:teuthology.orchestra.run.gibba020.stdout: "want_state": "up:active", 2021-02-10T11:15:41.254 INFO:teuthology.orchestra.run.gibba020.stdout: "state": "up:active", 2021-02-10T11:15:41.254 INFO:teuthology.orchestra.run.gibba020.stdout: "rank_uptime": 7.3336943010000004, 2021-02-10T11:15:41.254 INFO:teuthology.orchestra.run.gibba020.stdout: "mdsmap_epoch": 13, 2021-02-10T11:15:41.254 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch": 29, 2021-02-10T11:15:41.254 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch_barrier": 29, 2021-02-10T11:15:41.254 INFO:teuthology.orchestra.run.gibba020.stdout: "uptime": 16.970804808 2021-02-10T11:15:41.254 INFO:teuthology.orchestra.run.gibba020.stdout:} 2021-02-10T11:15:41.258 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", "whoami": 0, "id": 4427, "want_state": "up:active", "state": "up:active", "rank_uptime": 7.3336943010000004, "mdsmap_epoch": 13, "osdmap_epoch": 29, "osdmap_epoch_barrier": 29, "uptime": 16.970804808 } 2021-02-10T11:15:41.258 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:15:41.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:41.671 7ff146404700 1 Processor -- start 2021-02-10T11:15:41.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:41.671 7ff146404700 1 -- start start 2021-02-10T11:15:41.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:41.672 7ff146404700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7ff1400e5470 0x7ff14007e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:41.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:41.672 7ff146404700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff14007f300 0x7ff14007f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:41.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:41.672 7ff146404700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff14007ed80 con 0x7ff1400e5070 2021-02-10T11:15:41.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:41.672 7ff146404700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1401b4010 con 0x7ff1400e5470 2021-02-10T11:15:41.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:41.672 7ff146404700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1400d1790 con 0x7ff14007f300 2021-02-10T11:15:41.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:41.672 7ff13ffff700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7ff1400e5070 0x7ff1401b3820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:43058/0 (socket says 172.21.2.112:43058) 2021-02-10T11:15:41.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:41.672 7ff13ffff700 1 -- 172.21.2.112:0/1160205769 learned_addr learned my addr 172.21.2.112:0/1160205769 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:15:41.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:41.672 7ff13effd700 1 -- 172.21.2.112:0/1160205769 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1062055116 0 0) 0x7ff130000ce0 con 0x7ff1400e5070 2021-02-10T11:15:41.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:41.672 7ff13effd700 1 -- 172.21.2.112:0/1160205769 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff128002fe0 con 0x7ff1400e5070 2021-02-10T11:15:41.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:41.672 7ff13f7fe700 1 --2- 172.21.2.112:0/1160205769 >> v2:172.21.2.120:3300/0 conn(0x7ff1400e5470 0x7ff14007e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:41.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:41.672 7ff144a21700 1 --2- 172.21.2.112:0/1160205769 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff14007f300 0x7ff14007f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:41.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:41.672 7ff13effd700 1 -- 172.21.2.112:0/1160205769 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2147648965 0 0) 0x7ff1300010e0 con 0x7ff1400e5070 2021-02-10T11:15:41.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:41.672 7ff13effd700 1 -- 172.21.2.112:0/1160205769 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff14007f300 msgr2=0x7ff14007f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:15:41.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:41.672 7ff13effd700 1 --2- 172.21.2.112:0/1160205769 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff14007f300 0x7ff14007f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:41.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:41.672 7ff13effd700 1 -- 172.21.2.112:0/1160205769 >> v2:172.21.2.120:3300/0 conn(0x7ff1400e5470 msgr2=0x7ff14007e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:15:41.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:41.672 7ff13effd700 1 --2- 172.21.2.112:0/1160205769 >> v2:172.21.2.120:3300/0 conn(0x7ff1400e5470 0x7ff14007e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:41.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:41.672 7ff13effd700 1 -- 172.21.2.112:0/1160205769 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1400d19b0 con 0x7ff1400e5070 2021-02-10T11:15:41.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:41.673 7ff13effd700 1 -- 172.21.2.112:0/1160205769 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7ff130001660 con 0x7ff1400e5070 2021-02-10T11:15:41.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:41.673 7ff13effd700 1 -- 172.21.2.112:0/1160205769 >> v1:172.21.2.112:6789/0 conn(0x7ff1400e5070 legacy=0x7ff1401b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:41.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:41.673 7ff13effd700 1 --2- 172.21.2.112:0/1160205769 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff128003af0 0x7ff128005f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:41.689 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:41.673 7ff13effd700 1 --2- 172.21.2.112:0/1160205769 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff1280067e0 0x7ff128008d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:41.689 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:41.673 7ff13effd700 1 --2- 172.21.2.112:0/1160205769 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff1280095a0 0x7ff12800bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:41.689 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:41.673 7ff13effd700 1 -- 172.21.2.112:0/1160205769 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff12800c370 con 0x7ff1280095a0 2021-02-10T11:15:41.689 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:41.673 7ff13effd700 1 -- 172.21.2.112:0/1160205769 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff128050560 con 0x7ff1280067e0 2021-02-10T11:15:41.689 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:41.673 7ff13effd700 1 -- 172.21.2.112:0/1160205769 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff128050780 con 0x7ff128003af0 2021-02-10T11:15:41.689 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:41.673 7ff13f7fe700 1 --2- 172.21.2.112:0/1160205769 >> v2:172.21.2.120:3300/0 conn(0x7ff1400e5470 0x7ff14007e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:15:41.689 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:41.673 7ff144a21700 1 --2- 172.21.2.112:0/1160205769 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff1280095a0 0x7ff12800bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:41.689 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:41.673 7ff13ffff700 1 --2- 172.21.2.112:0/1160205769 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff128003af0 0x7ff128005f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:41.689 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:41.673 7ff13f7fe700 1 --2- 172.21.2.112:0/1160205769 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff1280067e0 0x7ff128008d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:41.689 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:41.673 7ff144a21700 1 -- 172.21.2.112:0/1160205769 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff128003af0 msgr2=0x7ff128005f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:15:41.689 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:41.673 7ff144a21700 1 --2- 172.21.2.112:0/1160205769 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff128003af0 0x7ff128005f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:41.689 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:41.673 7ff144a21700 1 -- 172.21.2.112:0/1160205769 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff1280067e0 msgr2=0x7ff128008d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:15:41.689 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:41.673 7ff144a21700 1 --2- 172.21.2.112:0/1160205769 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff1280067e0 0x7ff128008d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:41.690 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:41.673 7ff144a21700 1 -- 172.21.2.112:0/1160205769 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff128050a20 con 0x7ff1280095a0 2021-02-10T11:15:41.690 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:41.673 7ff144a21700 1 --2- 172.21.2.112:0/1160205769 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff1280095a0 0x7ff12800bac0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7ff138006e50 tx=0x7ff138009e70).ready entity=mon.0 client_cookie=67299c931d282d8e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:41.690 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:41.673 7ff13effd700 1 -- 172.21.2.112:0/1160205769 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff13800b300 con 0x7ff1280095a0 2021-02-10T11:15:41.690 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:41.673 7ff13effd700 1 -- 172.21.2.112:0/1160205769 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff138003030 con 0x7ff1280095a0 2021-02-10T11:15:41.690 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:41.673 7ff13effd700 1 -- 172.21.2.112:0/1160205769 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff13800b8c0 con 0x7ff1280095a0 2021-02-10T11:15:41.690 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:41.673 7ff146404700 1 -- 172.21.2.112:0/1160205769 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff1280095a0 msgr2=0x7ff12800bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:41.690 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:41.673 7ff146404700 1 --2- 172.21.2.112:0/1160205769 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff1280095a0 0x7ff12800bac0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7ff138006e50 tx=0x7ff138009e70).stop 2021-02-10T11:15:41.690 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:41.673 7ff144a21700 1 -- 172.21.2.112:0/1160205769 reap_dead start 2021-02-10T11:15:41.690 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:41.674 7ff146404700 1 -- 172.21.2.112:0/1160205769 shutdown_connections 2021-02-10T11:15:41.690 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:41.674 7ff146404700 1 -- 172.21.2.112:0/1160205769 >> 172.21.2.112:0/1160205769 conn(0x7ff1400ce690 msgr2=0x7ff1400e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:41.690 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:41.674 7ff146404700 1 -- 172.21.2.112:0/1160205769 shutdown_connections 2021-02-10T11:15:41.690 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:41.674 7ff146404700 1 -- 172.21.2.112:0/1160205769 wait complete. 2021-02-10T11:15:41.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:41.689 7ff146404700 1 Processor -- start 2021-02-10T11:15:41.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:41.689 7ff146404700 1 -- start start 2021-02-10T11:15:41.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:41.689 7ff146404700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff140194000 0x7ff14019f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:41.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:41.689 7ff146404700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff14019fde0 0x7ff14009bed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:41.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:41.689 7ff146404700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff1401a01e0 0x7ff1400895c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:41.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:41.689 7ff146404700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff140194400 con 0x7ff1401a01e0 2021-02-10T11:15:41.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:41.689 7ff146404700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff14009c4f0 con 0x7ff14019fde0 2021-02-10T11:15:41.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:41.689 7ff146404700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff140089be0 con 0x7ff140194000 2021-02-10T11:15:41.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:41.689 7ff144a21700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff1401a01e0 0x7ff1400895c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:41.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:41.689 7ff144a21700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff1401a01e0 0x7ff1400895c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:43612/0 (socket says 172.21.2.112:43612) 2021-02-10T11:15:41.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:41.689 7ff144a21700 1 -- 172.21.2.112:0/2452804231 learned_addr learned my addr 172.21.2.112:0/2452804231 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:15:41.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:41.690 7ff144a21700 1 -- 172.21.2.112:0/2452804231 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff140194000 msgr2=0x7ff14019f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:41.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:41.690 7ff144a21700 1 --2- 172.21.2.112:0/2452804231 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff140194000 0x7ff14019f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:41.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:41.690 7ff144a21700 1 -- 172.21.2.112:0/2452804231 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff14019fde0 msgr2=0x7ff14009bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:41.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:41.690 7ff144a21700 1 --2- 172.21.2.112:0/2452804231 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff14019fde0 0x7ff14009bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:41.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:41.690 7ff144a21700 1 -- 172.21.2.112:0/2452804231 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff140089e00 con 0x7ff1401a01e0 2021-02-10T11:15:41.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:41.690 7ff144a21700 1 --2- 172.21.2.112:0/2452804231 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff1401a01e0 0x7ff1400895c0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0x7ff138009a00 tx=0x7ff13800bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:41.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:41.690 7ff13cff9700 1 -- 172.21.2.112:0/2452804231 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff1380093d0 con 0x7ff1401a01e0 2021-02-10T11:15:41.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:41.690 7ff13cff9700 1 -- 172.21.2.112:0/2452804231 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff138000e70 con 0x7ff1401a01e0 2021-02-10T11:15:41.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:41.690 7ff13cff9700 1 -- 172.21.2.112:0/2452804231 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff13800c570 con 0x7ff1401a01e0 2021-02-10T11:15:41.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:41.690 7ff146404700 1 -- 172.21.2.112:0/2452804231 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff140089e00 con 0x7ff1401a01e0 2021-02-10T11:15:41.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:41.690 7ff146404700 1 -- 172.21.2.112:0/2452804231 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1400b8230 con 0x7ff1401a01e0 2021-02-10T11:15:41.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:41.691 7ff13cff9700 1 -- 172.21.2.112:0/2452804231 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff1380056b0 con 0x7ff1401a01e0 2021-02-10T11:15:41.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:41.691 7ff13cff9700 1 --2- 172.21.2.112:0/2452804231 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff1200186c0 0x7ff12001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:41.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:41.691 7ff13cff9700 1 -- 172.21.2.112:0/2452804231 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7ff138032960 con 0x7ff1401a01e0 2021-02-10T11:15:41.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:41.691 7ff13ffff700 1 --2- 172.21.2.112:0/2452804231 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff1200186c0 0x7ff12001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:41.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:41.692 7ff13ffff700 1 --2- 172.21.2.112:0/2452804231 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff1200186c0 0x7ff12001abc0 crc :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:41.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:41.721 7ff146404700 1 -- 172.21.2.112:0/2452804231 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff140089090 con 0x7ff1401a01e0 2021-02-10T11:15:41.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:41.724 7ff13cff9700 1 -- 172.21.2.112:0/2452804231 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff13802f940 con 0x7ff1401a01e0 2021-02-10T11:15:41.851 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:41.853 7ff146404700 1 -- 172.21.2.112:0/2452804231 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff140089090 con 0x7ff1401a01e0 2021-02-10T11:15:41.851 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:41.853 7ff13cff9700 1 -- 172.21.2.112:0/2452804231 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) v1 ==== 74+0+13993 (secure 0 0 0) 0x7ff13802f940 con 0x7ff1401a01e0 2021-02-10T11:15:41.852 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:15:41.852 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":29,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:15:34.922144","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:15:31.539720","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":5,"pool_name":"cephfs_data","create_time":"2021-02-10 11:15:32.965643","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:15:41.853 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:41.855 7ff146404700 1 -- 172.21.2.112:0/2452804231 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff1200186c0 msgr2=0x7ff12001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:41.853 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:41.855 7ff146404700 1 --2- 172.21.2.112:0/2452804231 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff1200186c0 0x7ff12001abc0 crc :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:41.853 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:41.855 7ff146404700 1 -- 172.21.2.112:0/2452804231 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff1401a01e0 msgr2=0x7ff1400895c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:41.853 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:41.855 7ff146404700 1 --2- 172.21.2.112:0/2452804231 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff1401a01e0 0x7ff1400895c0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0x7ff138009a00 tx=0x7ff13800bd10).stop 2021-02-10T11:15:41.853 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:41.855 7ff146404700 1 -- 172.21.2.112:0/2452804231 shutdown_connections 2021-02-10T11:15:41.853 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:41.855 7ff146404700 1 --2- 172.21.2.112:0/2452804231 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff1200186c0 0x7ff12001abc0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:41.853 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:41.855 7ff146404700 1 --2- 172.21.2.112:0/2452804231 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff14019fde0 0x7ff14009bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:41.854 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:41.855 7ff146404700 1 --2- 172.21.2.112:0/2452804231 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff1401a01e0 0x7ff1400895c0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:41.854 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:41.855 7ff146404700 1 --2- 172.21.2.112:0/2452804231 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff140194000 0x7ff14019f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:41.854 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:41.855 7ff146404700 1 -- 172.21.2.112:0/2452804231 >> 172.21.2.112:0/2452804231 conn(0x7ff140092c20 msgr2=0x7ff140195070 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:41.854 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:41.855 7ff146404700 1 -- 172.21.2.112:0/2452804231 shutdown_connections 2021-02-10T11:15:41.854 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:41.855 7ff146404700 1 -- 172.21.2.112:0/2452804231 wait complete. 2021-02-10T11:15:41.862 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:13:57.508758', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:15:31.539720', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '28', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:15:32.965643', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '29', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:15:41.862 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:15:42.291 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.276 7f4678239700 1 Processor -- start 2021-02-10T11:15:42.291 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.276 7f4678239700 1 -- start start 2021-02-10T11:15:42.292 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.276 7f4678239700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f46701b0470 0x7f46701b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:42.292 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.277 7f4678239700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f46700ef610 0x7f46700efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:42.292 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.277 7f4678239700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46700f0b50 con 0x7f46700f0030 2021-02-10T11:15:42.292 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.277 7f4678239700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46700e52e0 con 0x7f46700ef610 2021-02-10T11:15:42.292 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.277 7f4678239700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46700e5490 con 0x7f46701b0470 2021-02-10T11:15:42.292 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.277 7f4675854700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f46700ef610 0x7f46700efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:42.293 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.277 7f4675854700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f46700ef610 0x7f46700efa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:43616/0 (socket says 172.21.2.112:43616) 2021-02-10T11:15:42.293 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.277 7f4676856700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f46700f0030 0x7f46700f10d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:43078/0 (socket says 172.21.2.112:43078) 2021-02-10T11:15:42.293 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.277 7f4675854700 1 -- 172.21.2.112:0/494181278 learned_addr learned my addr 172.21.2.112:0/494181278 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:15:42.293 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.277 7f4676055700 1 --2- 172.21.2.112:0/494181278 >> v2:172.21.2.120:3300/0 conn(0x7f46701b0470 0x7f46701b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:42.293 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.277 7f4675854700 1 -- 172.21.2.112:0/494181278 >> v2:172.21.2.120:3300/0 conn(0x7f46701b0470 msgr2=0x7f46701b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:15:42.293 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.277 7f4675854700 1 --2- 172.21.2.112:0/494181278 >> v2:172.21.2.120:3300/0 conn(0x7f46701b0470 0x7f46701b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:42.293 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.277 7f4675854700 1 -- 172.21.2.112:0/494181278 >> v1:172.21.2.112:6789/0 conn(0x7f46700f0030 legacy=0x7f46700f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:42.293 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.277 7f4675053700 1 -- 172.21.2.112:0/494181278 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 539094324 0 0) 0x7f4668001480 con 0x7f46700f0030 2021-02-10T11:15:42.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.277 7f4675854700 1 -- 172.21.2.112:0/494181278 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46700f0430 con 0x7f46700ef610 2021-02-10T11:15:42.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.278 7f4675854700 1 --2- 172.21.2.112:0/494181278 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f46700ef610 0x7f46700efa10 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0x7f466c006700 tx=0x7f466c009b00).ready entity=mon.0 client_cookie=97d17c8c154c4d7 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:42.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.278 7f4675053700 1 -- 172.21.2.112:0/494181278 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f466c00c0c0 con 0x7f46700ef610 2021-02-10T11:15:42.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.278 7f4675053700 1 -- 172.21.2.112:0/494181278 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f466c00c400 con 0x7f46700ef610 2021-02-10T11:15:42.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.278 7f4675053700 1 -- 172.21.2.112:0/494181278 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f466c00c920 con 0x7f46700ef610 2021-02-10T11:15:42.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.278 7f4678239700 1 -- 172.21.2.112:0/494181278 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f46700ef610 msgr2=0x7f46700efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:42.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.278 7f4678239700 1 --2- 172.21.2.112:0/494181278 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f46700ef610 0x7f46700efa10 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0x7f466c006700 tx=0x7f466c009b00).stop 2021-02-10T11:15:42.295 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.279 7f4678239700 1 -- 172.21.2.112:0/494181278 shutdown_connections 2021-02-10T11:15:42.295 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.279 7f4678239700 1 --2- 172.21.2.112:0/494181278 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f46700ef610 0x7f46700efa10 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:42.295 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.279 7f4678239700 1 --2- 172.21.2.112:0/494181278 >> v2:172.21.2.120:3300/0 conn(0x7f46701b0470 0x7f46701b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:42.295 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.279 7f4678239700 1 -- 172.21.2.112:0/494181278 >> 172.21.2.112:0/494181278 conn(0x7f46700ce3e0 msgr2=0x7f46700e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:42.295 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.279 7f4678239700 1 -- 172.21.2.112:0/494181278 shutdown_connections 2021-02-10T11:15:42.295 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.279 7f4678239700 1 -- 172.21.2.112:0/494181278 wait complete. 2021-02-10T11:15:42.295 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.293 7f4678239700 1 Processor -- start 2021-02-10T11:15:42.296 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.293 7f4678239700 1 -- start start 2021-02-10T11:15:42.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.294 7f4678239700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4670105d10 0x7f467009f880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:42.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.294 7f4678239700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f46700961f0 0x7f46700965f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:42.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.294 7f4678239700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4670106890 0x7f4670106c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:42.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.294 7f4678239700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4670106110 con 0x7f46700961f0 2021-02-10T11:15:42.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.294 7f4678239700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4670106380 con 0x7f4670105d10 2021-02-10T11:15:42.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.294 7f4678239700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46700a6620 con 0x7f4670106890 2021-02-10T11:15:42.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.294 7f4676055700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4670105d10 0x7f467009f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:42.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.294 7f4675854700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f46700961f0 0x7f46700965f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:42.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.294 7f4676055700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4670105d10 0x7f467009f880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3300/0 says I am v2:172.21.2.112:45900/0 (socket says 172.21.2.112:45900) 2021-02-10T11:15:42.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.294 7f4676055700 1 -- 172.21.2.112:0/2212633934 learned_addr learned my addr 172.21.2.112:0/2212633934 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:15:42.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.294 7f4676856700 1 --2- 172.21.2.112:0/2212633934 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4670106890 0x7f4670106c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:42.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.294 7f4675854700 1 -- 172.21.2.112:0/2212633934 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4670106890 msgr2=0x7f4670106c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:42.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.294 7f4675854700 1 --2- 172.21.2.112:0/2212633934 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4670106890 0x7f4670106c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:42.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.294 7f4675854700 1 -- 172.21.2.112:0/2212633934 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4670105d10 msgr2=0x7f467009f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:42.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.294 7f4675854700 1 --2- 172.21.2.112:0/2212633934 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4670105d10 0x7f467009f880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:42.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.294 7f4675854700 1 -- 172.21.2.112:0/2212633934 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46701072b0 con 0x7f46700961f0 2021-02-10T11:15:42.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.294 7f4676055700 1 --2- 172.21.2.112:0/2212633934 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4670105d10 0x7f467009f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:15:42.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.295 7f4675854700 1 --2- 172.21.2.112:0/2212633934 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f46700961f0 0x7f46700965f0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7f466c006620 tx=0x7f466c00b8e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:42.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.295 7f4662ffd700 1 -- 172.21.2.112:0/2212633934 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f466c002030 con 0x7f46700961f0 2021-02-10T11:15:42.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.295 7f4662ffd700 1 -- 172.21.2.112:0/2212633934 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f466c0023b0 con 0x7f46700961f0 2021-02-10T11:15:42.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.295 7f4678239700 1 -- 172.21.2.112:0/2212633934 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46701072b0 con 0x7f46700961f0 2021-02-10T11:15:42.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.295 7f4678239700 1 -- 172.21.2.112:0/2212633934 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4670107840 con 0x7f46700961f0 2021-02-10T11:15:42.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.295 7f4662ffd700 1 -- 172.21.2.112:0/2212633934 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f466c002890 con 0x7f46700961f0 2021-02-10T11:15:42.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.295 7f4662ffd700 1 -- 172.21.2.112:0/2212633934 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f466c002590 con 0x7f46700961f0 2021-02-10T11:15:42.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.296 7f4662ffd700 1 --2- 172.21.2.112:0/2212633934 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f46580186c0 0x7f465801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:42.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.296 7f4662ffd700 1 -- 172.21.2.112:0/2212633934 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f466c032480 con 0x7f46700961f0 2021-02-10T11:15:42.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.296 7f4676055700 1 --2- 172.21.2.112:0/2212633934 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f46580186c0 0x7f465801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:42.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.296 7f4676055700 1 --2- 172.21.2.112:0/2212633934 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f46580186c0 0x7f465801abc0 crc :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:42.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.326 7f4678239700 1 -- 172.21.2.112:0/2212633934 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4670105ae0 con 0x7f46700961f0 2021-02-10T11:15:42.327 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.329 7f4662ffd700 1 -- 172.21.2.112:0/2212633934 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f466c030c30 con 0x7f46700961f0 2021-02-10T11:15:42.455 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.457 7f4678239700 1 -- 172.21.2.112:0/2212633934 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f46700f0b50 con 0x7f46700961f0 2021-02-10T11:15:42.456 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.458 7f4662ffd700 1 -- 172.21.2.112:0/2212633934 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v29) v1 ==== 92+0+10 (secure 0 0 0) 0x7f466c002570 con 0x7f46700961f0 2021-02-10T11:15:42.456 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:15:42.458 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.460 7f4678239700 1 -- 172.21.2.112:0/2212633934 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f46580186c0 msgr2=0x7f465801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:42.458 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.460 7f4678239700 1 --2- 172.21.2.112:0/2212633934 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f46580186c0 0x7f465801abc0 crc :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:42.458 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.460 7f4678239700 1 -- 172.21.2.112:0/2212633934 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f46700961f0 msgr2=0x7f46700965f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:42.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.460 7f4678239700 1 --2- 172.21.2.112:0/2212633934 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f46700961f0 0x7f46700965f0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7f466c006620 tx=0x7f466c00b8e0).stop 2021-02-10T11:15:42.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.460 7f4678239700 1 -- 172.21.2.112:0/2212633934 shutdown_connections 2021-02-10T11:15:42.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.460 7f4678239700 1 --2- 172.21.2.112:0/2212633934 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f46580186c0 0x7f465801abc0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:42.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.460 7f4678239700 1 --2- 172.21.2.112:0/2212633934 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4670106890 0x7f4670106c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:42.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.460 7f4678239700 1 --2- 172.21.2.112:0/2212633934 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f46700961f0 0x7f46700965f0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:42.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.460 7f4678239700 1 --2- 172.21.2.112:0/2212633934 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4670105d10 0x7f467009f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:42.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.460 7f4678239700 1 -- 172.21.2.112:0/2212633934 >> 172.21.2.112:0/2212633934 conn(0x7f4670095a70 msgr2=0x7f46700bf050 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:42.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.460 7f4678239700 1 -- 172.21.2.112:0/2212633934 shutdown_connections 2021-02-10T11:15:42.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.460 7f4678239700 1 -- 172.21.2.112:0/2212633934 wait complete. 2021-02-10T11:15:42.466 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:15:42.894 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.880 7f7b1cfbf700 1 Processor -- start 2021-02-10T11:15:42.894 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.880 7f7b1cfbf700 1 -- start start 2021-02-10T11:15:42.894 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.880 7f7b1cfbf700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f7b181b0870 0x7f7b180f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:42.894 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.880 7f7b1cfbf700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7b180efe00 0x7f7b180f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:42.894 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.880 7f7b1cfbf700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b181b0c70 con 0x7f7b181b0470 2021-02-10T11:15:42.894 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.880 7f7b1cfbf700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b180f0b50 con 0x7f7b181b0870 2021-02-10T11:15:42.894 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.880 7f7b1cfbf700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b180f0820 con 0x7f7b180efe00 2021-02-10T11:15:42.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.880 7f7b1659c700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f7b181b0870 0x7f7b180f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:42.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.880 7f7b1659c700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f7b181b0870 0x7f7b180f20e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3300/0 says I am v2:172.21.2.112:45908/0 (socket says 172.21.2.112:45908) 2021-02-10T11:15:42.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.880 7f7b1659c700 1 -- 172.21.2.112:0/2914632481 learned_addr learned my addr 172.21.2.112:0/2914632481 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:15:42.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.880 7f7b1759e700 1 --2- 172.21.2.112:0/2914632481 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7b180efe00 0x7f7b180f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:42.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.881 7f7b1659c700 1 -- 172.21.2.112:0/2914632481 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7b180efe00 msgr2=0x7f7b180f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:15:42.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.881 7f7b1659c700 1 --2- 172.21.2.112:0/2914632481 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7b180efe00 0x7f7b180f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:42.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.881 7f7b1659c700 1 -- 172.21.2.112:0/2914632481 >> v1:172.21.2.120:6789/0 conn(0x7f7b181b0470 legacy=0x7f7b180ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:42.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.881 7f7b1659c700 1 -- 172.21.2.112:0/2914632481 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b180e52e0 con 0x7f7b181b0870 2021-02-10T11:15:42.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.881 7f7b1659c700 1 --2- 172.21.2.112:0/2914632481 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7b181b0870 0x7f7b180f20e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f7b08006700 tx=0x7f7b08009b80).ready entity=mon.1 client_cookie=145d0cd500770b1c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:42.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.881 7f7b15d9b700 1 -- 172.21.2.112:0/2914632481 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7b0800c0c0 con 0x7f7b181b0870 2021-02-10T11:15:42.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.881 7f7b15d9b700 1 -- 172.21.2.112:0/2914632481 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b0800c340 con 0x7f7b181b0870 2021-02-10T11:15:42.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.881 7f7b15d9b700 1 -- 172.21.2.112:0/2914632481 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7b0800c860 con 0x7f7b181b0870 2021-02-10T11:15:42.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.881 7f7b1cfbf700 1 -- 172.21.2.112:0/2914632481 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7b181b0870 msgr2=0x7f7b180f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:42.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.881 7f7b1cfbf700 1 --2- 172.21.2.112:0/2914632481 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7b181b0870 0x7f7b180f20e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f7b08006700 tx=0x7f7b08009b80).stop 2021-02-10T11:15:42.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.882 7f7b1cfbf700 1 -- 172.21.2.112:0/2914632481 shutdown_connections 2021-02-10T11:15:42.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.882 7f7b1cfbf700 1 --2- 172.21.2.112:0/2914632481 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7b181b0870 0x7f7b180f20e0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:42.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.882 7f7b1cfbf700 1 --2- 172.21.2.112:0/2914632481 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7b180efe00 0x7f7b180f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:42.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.882 7f7b1cfbf700 1 -- 172.21.2.112:0/2914632481 >> 172.21.2.112:0/2914632481 conn(0x7f7b180ce3e0 msgr2=0x7f7b180e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:42.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.882 7f7b1cfbf700 1 -- 172.21.2.112:0/2914632481 shutdown_connections 2021-02-10T11:15:42.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.882 7f7b1cfbf700 1 -- 172.21.2.112:0/2914632481 wait complete. 2021-02-10T11:15:42.897 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.896 7f7b1cfbf700 1 Processor -- start 2021-02-10T11:15:42.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.896 7f7b1cfbf700 1 -- start start 2021-02-10T11:15:42.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.897 7f7b1cfbf700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7b18105d10 0x7f7b1809ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:42.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.897 7f7b1cfbf700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7b18106110 0x7f7b180b2a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:42.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.897 7f7b1cfbf700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7b1809e0a0 0x7f7b180b3040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:42.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.897 7f7b1cfbf700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b1809e640 con 0x7f7b18106110 2021-02-10T11:15:42.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.897 7f7b1cfbf700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b181a1620 con 0x7f7b1809e0a0 2021-02-10T11:15:42.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.897 7f7b1cfbf700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b180b3a70 con 0x7f7b18105d10 2021-02-10T11:15:42.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.897 7f7b1659c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7b18106110 0x7f7b180b2a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:42.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.897 7f7b1659c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7b18106110 0x7f7b180b2a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:43634/0 (socket says 172.21.2.112:43634) 2021-02-10T11:15:42.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.897 7f7b1659c700 1 -- 172.21.2.112:0/756949618 learned_addr learned my addr 172.21.2.112:0/756949618 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:15:42.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.897 7f7b1659c700 1 -- 172.21.2.112:0/756949618 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7b18105d10 msgr2=0x7f7b1809ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:42.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.897 7f7b1659c700 1 --2- 172.21.2.112:0/756949618 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7b18105d10 0x7f7b1809ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:42.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.897 7f7b1759e700 1 --2- 172.21.2.112:0/756949618 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7b1809e0a0 0x7f7b180b3040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:42.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.897 7f7b1659c700 1 -- 172.21.2.112:0/756949618 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7b1809e0a0 msgr2=0x7f7b180b3040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:42.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.897 7f7b1659c700 1 --2- 172.21.2.112:0/756949618 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7b1809e0a0 0x7f7b180b3040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:42.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.897 7f7b1659c700 1 -- 172.21.2.112:0/756949618 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b180b3660 con 0x7f7b18106110 2021-02-10T11:15:42.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.898 7f7b1659c700 1 --2- 172.21.2.112:0/756949618 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7b18106110 0x7f7b180b2a20 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 rx=0x7f7b08009760 tx=0x7f7b0800b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:42.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.898 7f7b0f7fe700 1 -- 172.21.2.112:0/756949618 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7b08006240 con 0x7f7b18106110 2021-02-10T11:15:42.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.898 7f7b0f7fe700 1 -- 172.21.2.112:0/756949618 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b080064c0 con 0x7f7b18106110 2021-02-10T11:15:42.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.898 7f7b1cfbf700 1 -- 172.21.2.112:0/756949618 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b180b3660 con 0x7f7b18106110 2021-02-10T11:15:42.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.898 7f7b1cfbf700 1 -- 172.21.2.112:0/756949618 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b1809f4d0 con 0x7f7b18106110 2021-02-10T11:15:42.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.898 7f7b0f7fe700 1 -- 172.21.2.112:0/756949618 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7b080054d0 con 0x7f7b18106110 2021-02-10T11:15:42.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.899 7f7b0f7fe700 1 -- 172.21.2.112:0/756949618 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7b08006030 con 0x7f7b18106110 2021-02-10T11:15:42.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.899 7f7b0f7fe700 1 --2- 172.21.2.112:0/756949618 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7afc0186d0 0x7f7afc01abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:42.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.899 7f7b0f7fe700 1 -- 172.21.2.112:0/756949618 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f7b080327e0 con 0x7f7b18106110 2021-02-10T11:15:42.900 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.900 7f7b16d9d700 1 --2- 172.21.2.112:0/756949618 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7afc0186d0 0x7f7afc01abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:42.900 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.901 7f7b16d9d700 1 --2- 172.21.2.112:0/756949618 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7afc0186d0 0x7f7afc01abd0 crc :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:42.928 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.930 7f7b1cfbf700 1 -- 172.21.2.112:0/756949618 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b18105ae0 con 0x7f7b18106110 2021-02-10T11:15:42.930 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:42.933 7f7b0f7fe700 1 -- 172.21.2.112:0/756949618 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7b080063c0 con 0x7f7b18106110 2021-02-10T11:15:43.059 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:43.061 7f7b1cfbf700 1 -- 172.21.2.112:0/756949618 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f7b180f0b50 con 0x7f7b18106110 2021-02-10T11:15:43.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:43.062 7f7b0f7fe700 1 -- 172.21.2.112:0/756949618 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v29) v1 ==== 104+0+10 (secure 0 0 0) 0x7f7b0800bd80 con 0x7f7b18106110 2021-02-10T11:15:43.060 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:15:43.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:43.064 7f7b1cfbf700 1 -- 172.21.2.112:0/756949618 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7afc0186d0 msgr2=0x7f7afc01abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:43.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:43.064 7f7b1cfbf700 1 --2- 172.21.2.112:0/756949618 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7afc0186d0 0x7f7afc01abd0 crc :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:43.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:43.064 7f7b1cfbf700 1 -- 172.21.2.112:0/756949618 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7b18106110 msgr2=0x7f7b180b2a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:43.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:43.064 7f7b1cfbf700 1 --2- 172.21.2.112:0/756949618 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7b18106110 0x7f7b180b2a20 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 rx=0x7f7b08009760 tx=0x7f7b0800b990).stop 2021-02-10T11:15:43.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:43.064 7f7b1cfbf700 1 -- 172.21.2.112:0/756949618 shutdown_connections 2021-02-10T11:15:43.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:43.064 7f7b1cfbf700 1 --2- 172.21.2.112:0/756949618 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7afc0186d0 0x7f7afc01abd0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:43.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:43.064 7f7b1cfbf700 1 --2- 172.21.2.112:0/756949618 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7b1809e0a0 0x7f7b180b3040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:43.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:43.064 7f7b1cfbf700 1 --2- 172.21.2.112:0/756949618 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7b18106110 0x7f7b180b2a20 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:43.063 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:43.064 7f7b1cfbf700 1 --2- 172.21.2.112:0/756949618 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7b18105d10 0x7f7b1809ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:43.063 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:43.064 7f7b1cfbf700 1 -- 172.21.2.112:0/756949618 >> 172.21.2.112:0/756949618 conn(0x7f7b180c3c70 msgr2=0x7f7b180bd640 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:43.063 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:43.064 7f7b1cfbf700 1 -- 172.21.2.112:0/756949618 shutdown_connections 2021-02-10T11:15:43.063 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:43.064 7f7b1cfbf700 1 -- 172.21.2.112:0/756949618 wait complete. 2021-02-10T11:15:43.070 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:15:43.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:43.486 7f8ac09cd700 1 Processor -- start 2021-02-10T11:15:43.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:43.486 7f8ac09cd700 1 -- start start 2021-02-10T11:15:43.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:43.486 7f8ac09cd700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f8abc1b0470 0x7f8abc1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:43.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:43.486 7f8ac09cd700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8abc0f0b50 con 0x7f8abc0f20e0 2021-02-10T11:15:43.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:43.486 7f8ac09cd700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8abc0f24e0 con 0x7f8abc0ef610 2021-02-10T11:15:43.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:43.486 7f8ac09cd700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8abc0e5ea0 con 0x7f8abc1b0470 2021-02-10T11:15:43.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:43.487 7f8abae1d700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f8abc0f20e0 0x7f8abc0f0200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:43104/0 (socket says 172.21.2.112:43104) 2021-02-10T11:15:43.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:43.487 7f8abae1d700 1 -- 172.21.2.112:0/2647685257 learned_addr learned my addr 172.21.2.112:0/2647685257 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:15:43.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:43.487 7f8aba61c700 1 --2- 172.21.2.112:0/2647685257 >> v2:172.21.2.112:3300/0 conn(0x7f8abc1b0470 0x7f8abc1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:43.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:43.487 7f8ab961a700 1 -- 172.21.2.112:0/2647685257 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2094489734 0 0) 0x7f8ab4001500 con 0x7f8abc0f20e0 2021-02-10T11:15:43.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:43.487 7f8ab961a700 1 -- 172.21.2.112:0/2647685257 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8aa8002fe0 con 0x7f8abc0f20e0 2021-02-10T11:15:43.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:43.487 7f8aba61c700 1 -- 172.21.2.112:0/2647685257 >> v1:172.21.2.120:6789/0 conn(0x7f8abc0ef610 legacy=0x7f8abc0efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:43.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:43.487 7f8aba61c700 1 -- 172.21.2.112:0/2647685257 >> v1:172.21.2.112:6789/0 conn(0x7f8abc0f20e0 legacy=0x7f8abc0f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:43.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:43.487 7f8aba61c700 1 -- 172.21.2.112:0/2647685257 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8abc0e52e0 con 0x7f8abc1b0470 2021-02-10T11:15:43.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:43.487 7f8aba61c700 1 --2- 172.21.2.112:0/2647685257 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8abc1b0470 0x7f8abc1b0870 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7f8aa4006650 tx=0x7f8aa4009a80).ready entity=mon.0 client_cookie=5749fefc98e04a9f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:43.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:43.487 7f8ab961a700 1 -- 172.21.2.112:0/2647685257 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8aa400c0c0 con 0x7f8abc1b0470 2021-02-10T11:15:43.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:43.488 7f8ab961a700 1 -- 172.21.2.112:0/2647685257 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8aa400c340 con 0x7f8abc1b0470 2021-02-10T11:15:43.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:43.488 7f8ab961a700 1 -- 172.21.2.112:0/2647685257 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8aa400c820 con 0x7f8abc1b0470 2021-02-10T11:15:43.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:43.488 7f8ac09cd700 1 -- 172.21.2.112:0/2647685257 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8abc1b0470 msgr2=0x7f8abc1b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:43.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:43.488 7f8ac09cd700 1 --2- 172.21.2.112:0/2647685257 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8abc1b0470 0x7f8abc1b0870 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7f8aa4006650 tx=0x7f8aa4009a80).stop 2021-02-10T11:15:43.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:43.488 7f8ac09cd700 1 -- 172.21.2.112:0/2647685257 shutdown_connections 2021-02-10T11:15:43.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:43.488 7f8ac09cd700 1 --2- 172.21.2.112:0/2647685257 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8abc1b0470 0x7f8abc1b0870 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:43.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:43.488 7f8ac09cd700 1 -- 172.21.2.112:0/2647685257 >> 172.21.2.112:0/2647685257 conn(0x7f8abc0ce3e0 msgr2=0x7f8abc0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:43.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:43.489 7f8ac09cd700 1 -- 172.21.2.112:0/2647685257 shutdown_connections 2021-02-10T11:15:43.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:43.489 7f8ac09cd700 1 -- 172.21.2.112:0/2647685257 wait complete. 2021-02-10T11:15:43.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:43.503 7f8ac09cd700 1 Processor -- start 2021-02-10T11:15:43.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:43.503 7f8ac09cd700 1 -- start start 2021-02-10T11:15:43.505 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:43.503 7f8ac09cd700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8abc0b2840 0x7f8abc1a4300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:43.505 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:43.503 7f8ac09cd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8abc0b30d0 0x7f8abc0b34d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:43.505 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:43.503 7f8ac09cd700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8abc0b3af0 0x7f8abc0f0580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:43.505 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:43.503 7f8ac09cd700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8abc0b2c40 con 0x7f8abc0b30d0 2021-02-10T11:15:43.505 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:43.503 7f8ac09cd700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8abc1a4920 con 0x7f8abc0b3af0 2021-02-10T11:15:43.505 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:43.503 7f8ac09cd700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8abc0f0ba0 con 0x7f8abc0b2840 2021-02-10T11:15:43.505 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:43.504 7f8ab9e1b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8abc0b30d0 0x7f8abc0b34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:43.505 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:43.504 7f8ab9e1b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8abc0b30d0 0x7f8abc0b34d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:43648/0 (socket says 172.21.2.112:43648) 2021-02-10T11:15:43.505 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:43.504 7f8ab9e1b700 1 -- 172.21.2.112:0/2532846618 learned_addr learned my addr 172.21.2.112:0/2532846618 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:15:43.505 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:43.504 7f8ab9e1b700 1 -- 172.21.2.112:0/2532846618 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8abc0b2840 msgr2=0x7f8abc1a4300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:43.505 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:43.504 7f8ab9e1b700 1 --2- 172.21.2.112:0/2532846618 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8abc0b2840 0x7f8abc1a4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:43.505 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:43.504 7f8ab9e1b700 1 -- 172.21.2.112:0/2532846618 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8abc0b3af0 msgr2=0x7f8abc0f0580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:43.505 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:43.504 7f8ab9e1b700 1 --2- 172.21.2.112:0/2532846618 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8abc0b3af0 0x7f8abc0f0580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:43.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:43.504 7f8ab9e1b700 1 -- 172.21.2.112:0/2532846618 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8abc0f0dc0 con 0x7f8abc0b30d0 2021-02-10T11:15:43.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:43.504 7f8ab9e1b700 1 --2- 172.21.2.112:0/2532846618 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8abc0b30d0 0x7f8abc0b34d0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7f8aac006ed0 tx=0x7f8aac00b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:43.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:43.504 7f8ab2ffd700 1 -- 172.21.2.112:0/2532846618 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8aac006240 con 0x7f8abc0b30d0 2021-02-10T11:15:43.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:43.504 7f8ab2ffd700 1 -- 172.21.2.112:0/2532846618 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8aac0054b0 con 0x7f8abc0b30d0 2021-02-10T11:15:43.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:43.504 7f8ac09cd700 1 -- 172.21.2.112:0/2532846618 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8abc0f0dc0 con 0x7f8abc0b30d0 2021-02-10T11:15:43.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:43.504 7f8ac09cd700 1 -- 172.21.2.112:0/2532846618 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8abc0f0dc0 con 0x7f8abc0b30d0 2021-02-10T11:15:43.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:43.504 7f8ab2ffd700 1 -- 172.21.2.112:0/2532846618 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8aac005990 con 0x7f8abc0b30d0 2021-02-10T11:15:43.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:43.505 7f8ab2ffd700 1 -- 172.21.2.112:0/2532846618 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8aac006030 con 0x7f8abc0b30d0 2021-02-10T11:15:43.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:43.505 7f8ab2ffd700 1 --2- 172.21.2.112:0/2532846618 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8aa0018730 0x7f8aa001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:43.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:43.505 7f8ab2ffd700 1 -- 172.21.2.112:0/2532846618 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f8aac032640 con 0x7f8abc0b30d0 2021-02-10T11:15:43.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:43.505 7f8aba61c700 1 --2- 172.21.2.112:0/2532846618 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8aa0018730 0x7f8aa001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:43.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:43.506 7f8aba61c700 1 --2- 172.21.2.112:0/2532846618 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8aa0018730 0x7f8aa001ac10 crc :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:43.534 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:43.536 7f8ac09cd700 1 -- 172.21.2.112:0/2532846618 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8abc0f0050 con 0x7f8abc0b30d0 2021-02-10T11:15:43.536 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:43.539 7f8ab2ffd700 1 -- 172.21.2.112:0/2532846618 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8aac000de0 con 0x7f8abc0b30d0 2021-02-10T11:15:43.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:43.666 7f8ac09cd700 1 -- 172.21.2.112:0/2532846618 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f8abc0f0050 con 0x7f8abc0b30d0 2021-02-10T11:15:43.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:43.666 7f8ab2ffd700 1 -- 172.21.2.112:0/2532846618 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v29) v1 ==== 100+0+10 (secure 0 0 0) 0x7f8aac00bd50 con 0x7f8abc0b30d0 2021-02-10T11:15:43.665 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:15:43.666 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:43.668 7f8ac09cd700 1 -- 172.21.2.112:0/2532846618 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8aa0018730 msgr2=0x7f8aa001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:43.666 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:43.668 7f8ac09cd700 1 --2- 172.21.2.112:0/2532846618 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8aa0018730 0x7f8aa001ac10 crc :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:43.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:43.668 7f8ac09cd700 1 -- 172.21.2.112:0/2532846618 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8abc0b30d0 msgr2=0x7f8abc0b34d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:43.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:43.668 7f8ac09cd700 1 --2- 172.21.2.112:0/2532846618 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8abc0b30d0 0x7f8abc0b34d0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7f8aac006ed0 tx=0x7f8aac00b960).stop 2021-02-10T11:15:43.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:43.668 7f8ac09cd700 1 -- 172.21.2.112:0/2532846618 shutdown_connections 2021-02-10T11:15:43.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:43.668 7f8ac09cd700 1 --2- 172.21.2.112:0/2532846618 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8aa0018730 0x7f8aa001ac10 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:43.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:43.668 7f8ac09cd700 1 --2- 172.21.2.112:0/2532846618 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8abc0b3af0 0x7f8abc0f0580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:43.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:43.668 7f8ac09cd700 1 --2- 172.21.2.112:0/2532846618 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8abc0b30d0 0x7f8abc0b34d0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:43.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:43.668 7f8ac09cd700 1 --2- 172.21.2.112:0/2532846618 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8abc0b2840 0x7f8abc1a4300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:43.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:43.668 7f8ac09cd700 1 -- 172.21.2.112:0/2532846618 >> 172.21.2.112:0/2532846618 conn(0x7f8abc0c36f0 msgr2=0x7f8abc094c30 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:43.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:43.669 7f8ac09cd700 1 -- 172.21.2.112:0/2532846618 shutdown_connections 2021-02-10T11:15:43.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:43.669 7f8ac09cd700 1 -- 172.21.2.112:0/2532846618 wait complete. 2021-02-10T11:15:43.675 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:15:43.675 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:15:44.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:44.087 7f1dfd6e7700 1 Processor -- start 2021-02-10T11:15:44.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:44.087 7f1dfd6e7700 1 -- start start 2021-02-10T11:15:44.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:44.087 7f1dfd6e7700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f1df80e5070 0x7f1df81b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:44.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:44.087 7f1dfd6e7700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f1df80e5470 0x7f1df8080310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:44.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:44.087 7f1dfd6e7700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1df807ed80 con 0x7f1df81b3e40 2021-02-10T11:15:44.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:44.087 7f1dfd6e7700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1df80d1790 con 0x7f1df80e5070 2021-02-10T11:15:44.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:44.087 7f1dfd6e7700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1df80d19b0 con 0x7f1df80e5470 2021-02-10T11:15:44.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:44.087 7f1df72bf700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f1df80e5070 0x7f1df81b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:44.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:44.088 7f1df6abe700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f1df81b3e40 0x7f1df807d690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:43114/0 (socket says 172.21.2.112:43114) 2021-02-10T11:15:44.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:44.088 7f1df6abe700 1 -- 172.21.2.112:0/3200652030 learned_addr learned my addr 172.21.2.112:0/3200652030 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:15:44.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:44.088 7f1df72bf700 1 -- 172.21.2.112:0/3200652030 >> v2:172.21.2.120:3300/0 conn(0x7f1df80e5470 msgr2=0x7f1df8080310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:15:44.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:44.088 7f1df72bf700 1 --2- 172.21.2.112:0/3200652030 >> v2:172.21.2.120:3300/0 conn(0x7f1df80e5470 0x7f1df8080310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:44.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:44.088 7f1df72bf700 1 -- 172.21.2.112:0/3200652030 >> v1:172.21.2.112:6789/0 conn(0x7f1df81b3e40 legacy=0x7f1df807d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:44.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:44.088 7f1df62bd700 1 -- 172.21.2.112:0/3200652030 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1435943627 0 0) 0x7f1dec0010b0 con 0x7f1df81b3e40 2021-02-10T11:15:44.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:44.088 7f1df72bf700 1 -- 172.21.2.112:0/3200652030 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1df80d1bd0 con 0x7f1df80e5070 2021-02-10T11:15:44.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:44.088 7f1df72bf700 1 --2- 172.21.2.112:0/3200652030 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1df80e5070 0x7f1df81b3820 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7f1de8006650 tx=0x7f1de8009a80).ready entity=mon.0 client_cookie=1bd631ad275682dc server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:44.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:44.088 7f1df62bd700 1 -- 172.21.2.112:0/3200652030 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1de80009b0 con 0x7f1df80e5070 2021-02-10T11:15:44.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:44.088 7f1df62bd700 1 -- 172.21.2.112:0/3200652030 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1de800c0c0 con 0x7f1df80e5070 2021-02-10T11:15:44.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:44.088 7f1df62bd700 1 -- 172.21.2.112:0/3200652030 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1de800c5e0 con 0x7f1df80e5070 2021-02-10T11:15:44.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:44.088 7f1dfd6e7700 1 -- 172.21.2.112:0/3200652030 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1df80e5070 msgr2=0x7f1df81b3820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:44.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:44.088 7f1dfd6e7700 1 --2- 172.21.2.112:0/3200652030 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1df80e5070 0x7f1df81b3820 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7f1de8006650 tx=0x7f1de8009a80).stop 2021-02-10T11:15:44.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:44.089 7f1dfd6e7700 1 -- 172.21.2.112:0/3200652030 shutdown_connections 2021-02-10T11:15:44.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:44.089 7f1dfd6e7700 1 --2- 172.21.2.112:0/3200652030 >> v2:172.21.2.120:3300/0 conn(0x7f1df80e5470 0x7f1df8080310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:44.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:44.089 7f1dfd6e7700 1 --2- 172.21.2.112:0/3200652030 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1df80e5070 0x7f1df81b3820 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:44.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:44.089 7f1dfd6e7700 1 -- 172.21.2.112:0/3200652030 >> 172.21.2.112:0/3200652030 conn(0x7f1df80ce690 msgr2=0x7f1df80e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:44.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:44.089 7f1dfd6e7700 1 -- 172.21.2.112:0/3200652030 shutdown_connections 2021-02-10T11:15:44.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:44.089 7f1dfd6e7700 1 -- 172.21.2.112:0/3200652030 wait complete. 2021-02-10T11:15:44.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:44.104 7f1dfd6e7700 1 Processor -- start 2021-02-10T11:15:44.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:44.104 7f1dfd6e7700 1 -- start start 2021-02-10T11:15:44.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:44.104 7f1dfd6e7700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1df818c700 0x7f1df8096c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:44.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:44.104 7f1dfd6e7700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1df80ae400 0x7f1df80ae800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:44.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:44.104 7f1dfd6e7700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1df8192380 0x7f1df8192780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:44.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:44.104 7f1dfd6e7700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1df8097290 con 0x7f1df818c700 2021-02-10T11:15:44.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:44.104 7f1dfd6e7700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1df8192da0 con 0x7f1df8192380 2021-02-10T11:15:44.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:44.104 7f1dfd6e7700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1df8192f80 con 0x7f1df80ae400 2021-02-10T11:15:44.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:44.104 7f1df72bf700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1df818c700 0x7f1df8096c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:44.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:44.104 7f1df72bf700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1df818c700 0x7f1df8096c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:43654/0 (socket says 172.21.2.112:43654) 2021-02-10T11:15:44.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:44.104 7f1df72bf700 1 -- 172.21.2.112:0/172468177 learned_addr learned my addr 172.21.2.112:0/172468177 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:15:44.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:44.105 7f1df72bf700 1 -- 172.21.2.112:0/172468177 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1df80ae400 msgr2=0x7f1df80ae800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:44.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:44.105 7f1df72bf700 1 --2- 172.21.2.112:0/172468177 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1df80ae400 0x7f1df80ae800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:44.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:44.105 7f1df72bf700 1 -- 172.21.2.112:0/172468177 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1df8192380 msgr2=0x7f1df8192780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:44.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:44.105 7f1df72bf700 1 --2- 172.21.2.112:0/172468177 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1df8192380 0x7f1df8192780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:44.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:44.105 7f1df72bf700 1 -- 172.21.2.112:0/172468177 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1df81931a0 con 0x7f1df818c700 2021-02-10T11:15:44.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:44.105 7f1df72bf700 1 --2- 172.21.2.112:0/172468177 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1df818c700 0x7f1df8096c70 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0x7f1de8009650 tx=0x7f1de800b8e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:44.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:44.105 7f1ddffff700 1 -- 172.21.2.112:0/172468177 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1de80093a0 con 0x7f1df818c700 2021-02-10T11:15:44.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:44.105 7f1ddffff700 1 -- 172.21.2.112:0/172468177 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1de8006100 con 0x7f1df818c700 2021-02-10T11:15:44.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:44.105 7f1dfd6e7700 1 -- 172.21.2.112:0/172468177 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1df81931a0 con 0x7f1df818c700 2021-02-10T11:15:44.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:44.105 7f1ddffff700 1 -- 172.21.2.112:0/172468177 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1de80052e0 con 0x7f1df818c700 2021-02-10T11:15:44.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:44.105 7f1dfd6e7700 1 -- 172.21.2.112:0/172468177 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1df807f510 con 0x7f1df818c700 2021-02-10T11:15:44.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:44.106 7f1ddffff700 1 -- 172.21.2.112:0/172468177 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1de80052e0 con 0x7f1df818c700 2021-02-10T11:15:44.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:44.106 7f1ddffff700 1 --2- 172.21.2.112:0/172468177 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1dd80186c0 0x7f1dd801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:44.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:44.106 7f1ddffff700 1 -- 172.21.2.112:0/172468177 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f1de8032530 con 0x7f1df818c700 2021-02-10T11:15:44.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:44.107 7f1df6abe700 1 --2- 172.21.2.112:0/172468177 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1dd80186c0 0x7f1dd801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:44.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:44.108 7f1df6abe700 1 --2- 172.21.2.112:0/172468177 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1dd80186c0 0x7f1dd801abc0 crc :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:44.135 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:44.138 7f1dfd6e7700 1 -- 172.21.2.112:0/172468177 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1df8107120 con 0x7f1df818c700 2021-02-10T11:15:44.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:44.140 7f1ddffff700 1 -- 172.21.2.112:0/172468177 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1de802e310 con 0x7f1df818c700 2021-02-10T11:15:44.266 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:44.268 7f1dfd6e7700 1 -- 172.21.2.112:0/172468177 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1df807ed80 con 0x7f1df818c700 2021-02-10T11:15:44.267 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:44.269 7f1ddffff700 1 -- 172.21.2.112:0/172468177 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+2100 (secure 0 0 0) 0x7f1de802e310 con 0x7f1df818c700 2021-02-10T11:15:44.267 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:15:44.267 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":13,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4627,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/2302110857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":2302110857},{"type":"v1","addr":"172.21.2.112:6833","nonce":2302110857}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":9}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:15:33.919772","modified":"2021-02-10 11:15:36.309405","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4427},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4427":{"gid":4427,"name":"b","rank":0,"incarnation":10,"state":"up:active","state_seq":4,"addr":"172.21.2.120:6835/3730022736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":3730022736},{"type":"v1","addr":"172.21.2.120:6835","nonce":3730022736}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2021-02-10T11:15:44.267 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 13 2021-02-10T11:15:44.268 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:44.270 7f1dfd6e7700 1 -- 172.21.2.112:0/172468177 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1dd80186c0 msgr2=0x7f1dd801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:44.268 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:44.270 7f1dfd6e7700 1 --2- 172.21.2.112:0/172468177 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1dd80186c0 0x7f1dd801abc0 crc :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:44.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:44.271 7f1dfd6e7700 1 -- 172.21.2.112:0/172468177 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1df818c700 msgr2=0x7f1df8096c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:44.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:44.271 7f1dfd6e7700 1 --2- 172.21.2.112:0/172468177 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1df818c700 0x7f1df8096c70 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0x7f1de8009650 tx=0x7f1de800b8e0).stop 2021-02-10T11:15:44.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:44.271 7f1dfd6e7700 1 -- 172.21.2.112:0/172468177 shutdown_connections 2021-02-10T11:15:44.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:44.271 7f1dfd6e7700 1 --2- 172.21.2.112:0/172468177 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1dd80186c0 0x7f1dd801abc0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:44.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:44.271 7f1dfd6e7700 1 --2- 172.21.2.112:0/172468177 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1df8192380 0x7f1df8192780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:44.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:44.271 7f1dfd6e7700 1 --2- 172.21.2.112:0/172468177 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1df80ae400 0x7f1df80ae800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:44.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:44.271 7f1dfd6e7700 1 --2- 172.21.2.112:0/172468177 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1df818c700 0x7f1df8096c70 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:44.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:44.271 7f1dfd6e7700 1 -- 172.21.2.112:0/172468177 >> 172.21.2.112:0/172468177 conn(0x7f1df80891f0 msgr2=0x7f1df81a0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:44.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:44.271 7f1dfd6e7700 1 -- 172.21.2.112:0/172468177 shutdown_connections 2021-02-10T11:15:44.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:44.271 7f1dfd6e7700 1 -- 172.21.2.112:0/172468177 wait complete. 2021-02-10T11:15:44.478 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 13, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:15:33.919772', 'modified': '2021-02-10 11:15:36.309405', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4427}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4427': {'gid': 4427, 'name': 'b', 'rank': 0, 'incarnation': 10, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.2.120:6835/3730022736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.120:6834', 'nonce': 3730022736}, {'type': 'v1', 'addr': '172.21.2.120:6835', 'nonce': 3730022736}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:15:44.479 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:15:44.479 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:15:44.876 INFO:teuthology.orchestra.run.gibba020.stdout:{ 2021-02-10T11:15:44.876 INFO:teuthology.orchestra.run.gibba020.stdout: "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", 2021-02-10T11:15:44.876 INFO:teuthology.orchestra.run.gibba020.stdout: "whoami": 0, 2021-02-10T11:15:44.876 INFO:teuthology.orchestra.run.gibba020.stdout: "id": 4427, 2021-02-10T11:15:44.876 INFO:teuthology.orchestra.run.gibba020.stdout: "want_state": "up:active", 2021-02-10T11:15:44.876 INFO:teuthology.orchestra.run.gibba020.stdout: "state": "up:active", 2021-02-10T11:15:44.877 INFO:teuthology.orchestra.run.gibba020.stdout: "rank_uptime": 10.956393902, 2021-02-10T11:15:44.877 INFO:teuthology.orchestra.run.gibba020.stdout: "mdsmap_epoch": 13, 2021-02-10T11:15:44.877 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch": 29, 2021-02-10T11:15:44.877 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch_barrier": 29, 2021-02-10T11:15:44.877 INFO:teuthology.orchestra.run.gibba020.stdout: "uptime": 20.593506428000001 2021-02-10T11:15:44.877 INFO:teuthology.orchestra.run.gibba020.stdout:} 2021-02-10T11:15:44.880 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", "whoami": 0, "id": 4427, "want_state": "up:active", "state": "up:active", "rank_uptime": 10.956393902, "mdsmap_epoch": 13, "osdmap_epoch": 29, "osdmap_epoch_barrier": 29, "uptime": 20.593506428000001 } 2021-02-10T11:15:44.880 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:15:44.880 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2021-02-10T11:15:44.880 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@gibba012.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2021-02-10T11:15:44.881 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:15:44.881 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:15:45.217 DEBUG:teuthology.orchestra.run.gibba012:> sudo modprobe fuse 2021-02-10T11:15:45.562 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:15:45.562 DEBUG:teuthology.orchestra.run.gibba012:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:15:45.903 INFO:teuthology.orchestra.run.gibba012.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:15:45.903 INFO:teuthology.orchestra.run.gibba012.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:15:45.904 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:15:45.904 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:15:45.904 DEBUG:teuthology.orchestra.run.gibba012:> ls /sys/fs/fuse/connections 2021-02-10T11:15:46.239 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2021-02-10T11:15:46.239 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:15:46.242 DEBUG:teuthology.orchestra.run.gibba012:> sudo modprobe fuse 2021-02-10T11:15:46.584 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:15:46.584 DEBUG:teuthology.orchestra.run.gibba012:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:15:46.649 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba012.stderr:ceph-fuse[45043]: starting ceph client2021-02-10 11:15:46.651 7fbcf7431f80 -1 init, newargv = 0x556f8a250af0 newargc=9 2021-02-10T11:15:46.650 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba012.stderr: 2021-02-10T11:15:46.655 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba012.stderr:ceph-fuse[45043]: starting fuse 2021-02-10T11:15:46.925 INFO:teuthology.orchestra.run.gibba012.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:15:46.925 INFO:teuthology.orchestra.run.gibba012.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:15:46.926 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:15:46.926 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:15:46.926 DEBUG:teuthology.orchestra.run.gibba012:> ls /sys/fs/fuse/connections 2021-02-10T11:15:47.261 INFO:teuthology.orchestra.run.gibba012.stdout:42 2021-02-10T11:15:47.261 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2021-02-10T11:15:47.262 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:15:47.262 DEBUG:teuthology.orchestra.run.gibba012:> sudo python3 -c ' 2021-02-10T11:15:47.262 DEBUG:teuthology.orchestra.run.gibba012:> import glob 2021-02-10T11:15:47.262 DEBUG:teuthology.orchestra.run.gibba012:> import re 2021-02-10T11:15:47.262 DEBUG:teuthology.orchestra.run.gibba012:> import os 2021-02-10T11:15:47.262 DEBUG:teuthology.orchestra.run.gibba012:> import subprocess 2021-02-10T11:15:47.262 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:15:47.262 DEBUG:teuthology.orchestra.run.gibba012:> def find_socket(client_name): 2021-02-10T11:15:47.262 DEBUG:teuthology.orchestra.run.gibba012:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2021-02-10T11:15:47.263 DEBUG:teuthology.orchestra.run.gibba012:> files = glob.glob(asok_path) 2021-02-10T11:15:47.263 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:15:47.263 DEBUG:teuthology.orchestra.run.gibba012:> # Given a non-glob path, it better be there 2021-02-10T11:15:47.263 DEBUG:teuthology.orchestra.run.gibba012:> if "*" not in asok_path: 2021-02-10T11:15:47.263 DEBUG:teuthology.orchestra.run.gibba012:> assert(len(files) == 1) 2021-02-10T11:15:47.263 DEBUG:teuthology.orchestra.run.gibba012:> return files[0] 2021-02-10T11:15:47.263 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:15:47.263 DEBUG:teuthology.orchestra.run.gibba012:> for f in files: 2021-02-10T11:15:47.263 DEBUG:teuthology.orchestra.run.gibba012:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:15:47.263 DEBUG:teuthology.orchestra.run.gibba012:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:15:47.263 DEBUG:teuthology.orchestra.run.gibba012:> return f 2021-02-10T11:15:47.263 DEBUG:teuthology.orchestra.run.gibba012:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:15:47.263 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:15:47.263 DEBUG:teuthology.orchestra.run.gibba012:> print(find_socket("client.0")) 2021-02-10T11:15:47.264 DEBUG:teuthology.orchestra.run.gibba012:> ' 2021-02-10T11:15:47.624 INFO:teuthology.orchestra.run.gibba012.stdout:/var/run/ceph/ceph-client.0.45043.asok 2021-02-10T11:15:47.627 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.45043.asok 2021-02-10T11:15:47.627 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:15:47.627 DEBUG:teuthology.orchestra.run.gibba012:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.45043.asok status 2021-02-10T11:15:48.015 INFO:teuthology.orchestra.run.gibba012.stdout:{ 2021-02-10T11:15:48.015 INFO:teuthology.orchestra.run.gibba012.stdout: "metadata": { 2021-02-10T11:15:48.015 INFO:teuthology.orchestra.run.gibba012.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:15:48.015 INFO:teuthology.orchestra.run.gibba012.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:15:48.015 INFO:teuthology.orchestra.run.gibba012.stdout: "entity_id": "0", 2021-02-10T11:15:48.015 INFO:teuthology.orchestra.run.gibba012.stdout: "hostname": "gibba012", 2021-02-10T11:15:48.015 INFO:teuthology.orchestra.run.gibba012.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2021-02-10T11:15:48.016 INFO:teuthology.orchestra.run.gibba012.stdout: "pid": "45043", 2021-02-10T11:15:48.016 INFO:teuthology.orchestra.run.gibba012.stdout: "root": "/" 2021-02-10T11:15:48.016 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:48.016 INFO:teuthology.orchestra.run.gibba012.stdout: "dentry_count": 0, 2021-02-10T11:15:48.016 INFO:teuthology.orchestra.run.gibba012.stdout: "dentry_pinned_count": 0, 2021-02-10T11:15:48.016 INFO:teuthology.orchestra.run.gibba012.stdout: "id": 5004, 2021-02-10T11:15:48.016 INFO:teuthology.orchestra.run.gibba012.stdout: "inst": { 2021-02-10T11:15:48.016 INFO:teuthology.orchestra.run.gibba012.stdout: "name": { 2021-02-10T11:15:48.016 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "client", 2021-02-10T11:15:48.016 INFO:teuthology.orchestra.run.gibba012.stdout: "num": 5004 2021-02-10T11:15:48.016 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:48.016 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": { 2021-02-10T11:15:48.016 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:15:48.017 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:0", 2021-02-10T11:15:48.017 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 1949026975 2021-02-10T11:15:48.017 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:15:48.017 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:48.017 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": { 2021-02-10T11:15:48.017 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:15:48.017 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:0", 2021-02-10T11:15:48.017 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 1949026975 2021-02-10T11:15:48.017 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:48.017 INFO:teuthology.orchestra.run.gibba012.stdout: "inst_str": "client.5004 172.21.2.112:0/1949026975", 2021-02-10T11:15:48.017 INFO:teuthology.orchestra.run.gibba012.stdout: "addr_str": "172.21.2.112:0/1949026975", 2021-02-10T11:15:48.017 INFO:teuthology.orchestra.run.gibba012.stdout: "inode_count": 1, 2021-02-10T11:15:48.017 INFO:teuthology.orchestra.run.gibba012.stdout: "mds_epoch": 13, 2021-02-10T11:15:48.018 INFO:teuthology.orchestra.run.gibba012.stdout: "osd_epoch": 29, 2021-02-10T11:15:48.018 INFO:teuthology.orchestra.run.gibba012.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:15:48.018 INFO:teuthology.orchestra.run.gibba012.stdout: "blacklisted": false 2021-02-10T11:15:48.018 INFO:teuthology.orchestra.run.gibba012.stdout:} 2021-02-10T11:15:48.019 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:15:48.019 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:15:48.019 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:15:48.353 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:15:48.354 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:15:48.354 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:15:48.354 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:15:48.698 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:15:49.124 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.109 7f1329f1d700 1 Processor -- start 2021-02-10T11:15:49.124 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.109 7f1329f1d700 1 -- start start 2021-02-10T11:15:49.124 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.110 7f1329f1d700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f13240e5470 0x7f132407e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:49.124 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.110 7f1329f1d700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f132407ed80 con 0x7f13240e5070 2021-02-10T11:15:49.124 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.110 7f1329f1d700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13241b4010 con 0x7f132407f300 2021-02-10T11:15:49.124 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.110 7f1329f1d700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13240d1790 con 0x7f13240e5470 2021-02-10T11:15:49.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.110 7f13237fe700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f13240e5070 0x7f13241b3820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:43136/0 (socket says 172.21.2.112:43136) 2021-02-10T11:15:49.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.110 7f13237fe700 1 -- 172.21.2.112:0/1167379410 learned_addr learned my addr 172.21.2.112:0/1167379410 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:15:49.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.110 7f1322ffd700 1 --2- 172.21.2.112:0/1167379410 >> v2:172.21.2.120:3300/0 conn(0x7f13240e5470 0x7f132407e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:49.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.110 7f13227fc700 1 -- 172.21.2.112:0/1167379410 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1639952413 0 0) 0x7f130c001180 con 0x7f13240e5070 2021-02-10T11:15:49.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.110 7f13227fc700 1 -- 172.21.2.112:0/1167379410 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1310002fe0 con 0x7f13240e5070 2021-02-10T11:15:49.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.110 7f1322ffd700 1 -- 172.21.2.112:0/1167379410 >> v1:172.21.2.120:6789/0 conn(0x7f132407f300 legacy=0x7f132407f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:49.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.110 7f1322ffd700 1 -- 172.21.2.112:0/1167379410 >> v1:172.21.2.112:6789/0 conn(0x7f13240e5070 legacy=0x7f13241b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:49.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.111 7f1322ffd700 1 -- 172.21.2.112:0/1167379410 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13240d19b0 con 0x7f13240e5470 2021-02-10T11:15:49.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.111 7f1322ffd700 1 --2- 172.21.2.112:0/1167379410 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f13240e5470 0x7f132407e6a0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f1314006650 tx=0x7f1314009a80).ready entity=mon.1 client_cookie=90c6619c5d4dacc9 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:49.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.111 7f13227fc700 1 -- 172.21.2.112:0/1167379410 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f131400c0c0 con 0x7f13240e5470 2021-02-10T11:15:49.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.111 7f13227fc700 1 -- 172.21.2.112:0/1167379410 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f131400c340 con 0x7f13240e5470 2021-02-10T11:15:49.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.111 7f13227fc700 1 -- 172.21.2.112:0/1167379410 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f131400c860 con 0x7f13240e5470 2021-02-10T11:15:49.126 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.111 7f1329f1d700 1 -- 172.21.2.112:0/1167379410 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f13240e5470 msgr2=0x7f132407e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:49.126 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.111 7f1329f1d700 1 --2- 172.21.2.112:0/1167379410 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f13240e5470 0x7f132407e6a0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f1314006650 tx=0x7f1314009a80).stop 2021-02-10T11:15:49.126 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.112 7f1329f1d700 1 -- 172.21.2.112:0/1167379410 shutdown_connections 2021-02-10T11:15:49.126 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.112 7f1329f1d700 1 --2- 172.21.2.112:0/1167379410 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f13240e5470 0x7f132407e6a0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:49.126 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.112 7f1329f1d700 1 -- 172.21.2.112:0/1167379410 >> 172.21.2.112:0/1167379410 conn(0x7f13240ce690 msgr2=0x7f13240e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:49.126 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.112 7f1329f1d700 1 -- 172.21.2.112:0/1167379410 shutdown_connections 2021-02-10T11:15:49.126 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.112 7f1329f1d700 1 -- 172.21.2.112:0/1167379410 wait complete. 2021-02-10T11:15:49.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.127 7f1329f1d700 1 Processor -- start 2021-02-10T11:15:49.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.127 7f1329f1d700 1 -- start start 2021-02-10T11:15:49.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.127 7f1329f1d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f13241a2a90 0x7f13241aa0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:49.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.127 7f1329f1d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f13241aa700 0x7f13241a3a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:49.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.127 7f1329f1d700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f13240a55e0 0x7f13241a4040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:49.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.127 7f1329f1d700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13241a34a0 con 0x7f13241a2a90 2021-02-10T11:15:49.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.127 7f1329f1d700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13241a2e90 con 0x7f13240a55e0 2021-02-10T11:15:49.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.127 7f1329f1d700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13241a30b0 con 0x7f13241aa700 2021-02-10T11:15:49.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.127 7f13237fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f13241a2a90 0x7f13241aa0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:49.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.127 7f1322ffd700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f13241aa700 0x7f13241a3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:49.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.127 7f13237fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f13241a2a90 0x7f13241aa0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:43682/0 (socket says 172.21.2.112:43682) 2021-02-10T11:15:49.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.128 7f13237fe700 1 -- 172.21.2.112:0/4187326100 learned_addr learned my addr 172.21.2.112:0/4187326100 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:15:49.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.128 7f1323fff700 1 --2- 172.21.2.112:0/4187326100 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f13240a55e0 0x7f13241a4040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:49.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.128 7f13237fe700 1 -- 172.21.2.112:0/4187326100 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f13241aa700 msgr2=0x7f13241a3a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:49.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.128 7f13237fe700 1 --2- 172.21.2.112:0/4187326100 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f13241aa700 0x7f13241a3a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:49.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.128 7f13237fe700 1 -- 172.21.2.112:0/4187326100 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f13240a55e0 msgr2=0x7f13241a4040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:49.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.128 7f13237fe700 1 --2- 172.21.2.112:0/4187326100 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f13240a55e0 0x7f13241a4040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:49.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.128 7f13237fe700 1 -- 172.21.2.112:0/4187326100 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13241a4660 con 0x7f13241a2a90 2021-02-10T11:15:49.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.128 7f1323fff700 1 --2- 172.21.2.112:0/4187326100 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f13240a55e0 0x7f13241a4040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:15:49.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.128 7f13237fe700 1 --2- 172.21.2.112:0/4187326100 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f13241a2a90 0x7f13241aa0e0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0x7f130c00a4f0 tx=0x7f130c00a890).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:49.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.128 7f131bfff700 1 -- 172.21.2.112:0/4187326100 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f130c00ac80 con 0x7f13241a2a90 2021-02-10T11:15:49.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.128 7f131bfff700 1 -- 172.21.2.112:0/4187326100 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f130c0043a0 con 0x7f13241a2a90 2021-02-10T11:15:49.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.128 7f131bfff700 1 -- 172.21.2.112:0/4187326100 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f130c004800 con 0x7f13241a2a90 2021-02-10T11:15:49.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.128 7f1329f1d700 1 -- 172.21.2.112:0/4187326100 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13241a4660 con 0x7f13241a2a90 2021-02-10T11:15:49.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.128 7f1329f1d700 1 -- 172.21.2.112:0/4187326100 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13241a4bf0 con 0x7f13241a2a90 2021-02-10T11:15:49.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.129 7f131bfff700 1 -- 172.21.2.112:0/4187326100 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f130c004580 con 0x7f13241a2a90 2021-02-10T11:15:49.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.129 7f131bfff700 1 --2- 172.21.2.112:0/4187326100 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f13080186c0 0x7f130801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:49.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.129 7f131bfff700 1 -- 172.21.2.112:0/4187326100 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f130c0323d0 con 0x7f13241a2a90 2021-02-10T11:15:49.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.129 7f1322ffd700 1 --2- 172.21.2.112:0/4187326100 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f13080186c0 0x7f130801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:49.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.130 7f1322ffd700 1 --2- 172.21.2.112:0/4187326100 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f13080186c0 0x7f130801abc0 crc :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:49.158 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.160 7f1329f1d700 1 -- 172.21.2.112:0/4187326100 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1324083da0 con 0x7f13241a2a90 2021-02-10T11:15:49.160 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.162 7f131bfff700 1 -- 172.21.2.112:0/4187326100 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f130c000c90 con 0x7f13241a2a90 2021-02-10T11:15:49.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.287 7f1329f1d700 1 -- 172.21.2.112:0/4187326100 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f132407ed80 con 0x7f13241a2a90 2021-02-10T11:15:49.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.288 7f131bfff700 1 -- 172.21.2.112:0/4187326100 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) v1 ==== 74+0+13993 (secure 0 0 0) 0x7f130c004580 con 0x7f13241a2a90 2021-02-10T11:15:49.286 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:15:49.286 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":29,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:15:34.922144","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:15:31.539720","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":5,"pool_name":"cephfs_data","create_time":"2021-02-10 11:15:32.965643","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:15:49.288 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.290 7f1329f1d700 1 -- 172.21.2.112:0/4187326100 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f13080186c0 msgr2=0x7f130801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:49.289 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.290 7f1329f1d700 1 --2- 172.21.2.112:0/4187326100 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f13080186c0 0x7f130801abc0 crc :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:49.289 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.290 7f1329f1d700 1 -- 172.21.2.112:0/4187326100 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f13241a2a90 msgr2=0x7f13241aa0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:49.289 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.290 7f1329f1d700 1 --2- 172.21.2.112:0/4187326100 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f13241a2a90 0x7f13241aa0e0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0x7f130c00a4f0 tx=0x7f130c00a890).stop 2021-02-10T11:15:49.289 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.290 7f1329f1d700 1 -- 172.21.2.112:0/4187326100 shutdown_connections 2021-02-10T11:15:49.289 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.290 7f1329f1d700 1 --2- 172.21.2.112:0/4187326100 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f13080186c0 0x7f130801abc0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:49.289 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.290 7f1329f1d700 1 --2- 172.21.2.112:0/4187326100 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f13240a55e0 0x7f13241a4040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:49.289 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.290 7f1329f1d700 1 --2- 172.21.2.112:0/4187326100 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f13241aa700 0x7f13241a3a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:49.289 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.290 7f1329f1d700 1 --2- 172.21.2.112:0/4187326100 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f13241a2a90 0x7f13241aa0e0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:49.289 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.290 7f1329f1d700 1 -- 172.21.2.112:0/4187326100 >> 172.21.2.112:0/4187326100 conn(0x7f132408fe10 msgr2=0x7f132419c020 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:49.289 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.291 7f1329f1d700 1 -- 172.21.2.112:0/4187326100 shutdown_connections 2021-02-10T11:15:49.289 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.291 7f1329f1d700 1 -- 172.21.2.112:0/4187326100 wait complete. 2021-02-10T11:15:49.297 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:13:57.508758', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:15:31.539720', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '28', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:15:32.965643', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '29', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:15:49.297 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:15:49.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.712 7f538e14e700 1 Processor -- start 2021-02-10T11:15:49.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.712 7f538e14e700 1 -- start start 2021-02-10T11:15:49.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.712 7f538e14e700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f53881b0870 0x7f53880f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:49.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.712 7f538e14e700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53881b0c70 con 0x7f53881b0470 2021-02-10T11:15:49.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.712 7f538e14e700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53880f0b50 con 0x7f53880efe00 2021-02-10T11:15:49.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.712 7f538e14e700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53880e5ea0 con 0x7f53881b0870 2021-02-10T11:15:49.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.712 7f53877fe700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f53881b0470 0x7f53880ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:43150/0 (socket says 172.21.2.112:43150) 2021-02-10T11:15:49.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.712 7f53877fe700 1 -- 172.21.2.112:0/2968425103 learned_addr learned my addr 172.21.2.112:0/2968425103 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:15:49.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.713 7f53867fc700 1 -- 172.21.2.112:0/2968425103 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1603878585 0 0) 0x7f537c0010b0 con 0x7f53881b0470 2021-02-10T11:15:49.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.713 7f53867fc700 1 -- 172.21.2.112:0/2968425103 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5370002fe0 con 0x7f53881b0470 2021-02-10T11:15:49.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.713 7f5386ffd700 1 --2- 172.21.2.112:0/2968425103 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f53881b0870 0x7f53880f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:49.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.713 7f53867fc700 1 -- 172.21.2.112:0/2968425103 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2192153088 0 0) 0x7f537c0010b0 con 0x7f53881b0470 2021-02-10T11:15:49.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.713 7f53867fc700 1 -- 172.21.2.112:0/2968425103 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f53881b0870 msgr2=0x7f53880f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:15:49.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.713 7f53867fc700 1 --2- 172.21.2.112:0/2968425103 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f53881b0870 0x7f53880f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:49.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.713 7f53867fc700 1 -- 172.21.2.112:0/2968425103 >> v1:172.21.2.120:6789/0 conn(0x7f53880efe00 legacy=0x7f53880f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:49.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.713 7f53867fc700 1 -- 172.21.2.112:0/2968425103 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53880e52e0 con 0x7f53881b0470 2021-02-10T11:15:49.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.713 7f53867fc700 1 -- 172.21.2.112:0/2968425103 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f537c0015b0 con 0x7f53881b0470 2021-02-10T11:15:49.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.713 7f53867fc700 1 -- 172.21.2.112:0/2968425103 >> v1:172.21.2.112:6789/0 conn(0x7f53881b0470 legacy=0x7f53880ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:49.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.713 7f53867fc700 1 --2- 172.21.2.112:0/2968425103 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5370003aa0 0x7f5370005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:49.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.713 7f53867fc700 1 --2- 172.21.2.112:0/2968425103 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5370006790 0x7f5370008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:49.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.713 7f53867fc700 1 --2- 172.21.2.112:0/2968425103 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5370009590 0x7f537000ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:49.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.713 7f53867fc700 1 -- 172.21.2.112:0/2968425103 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f537000c2d0 con 0x7f5370003aa0 2021-02-10T11:15:49.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.713 7f53867fc700 1 -- 172.21.2.112:0/2968425103 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f537000c4f0 con 0x7f5370006790 2021-02-10T11:15:49.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.713 7f53867fc700 1 -- 172.21.2.112:0/2968425103 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f537000c710 con 0x7f5370009590 2021-02-10T11:15:49.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.713 7f53877fe700 1 --2- 172.21.2.112:0/2968425103 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5370003aa0 0x7f5370005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:49.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.713 7f5387fff700 1 --2- 172.21.2.112:0/2968425103 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5370009590 0x7f537000ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:49.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.713 7f5386ffd700 1 --2- 172.21.2.112:0/2968425103 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5370006790 0x7f5370008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:49.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.714 7f53877fe700 1 -- 172.21.2.112:0/2968425103 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5370009590 msgr2=0x7f537000ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:15:49.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.714 7f53877fe700 1 --2- 172.21.2.112:0/2968425103 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5370009590 0x7f537000ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:49.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.714 7f53877fe700 1 -- 172.21.2.112:0/2968425103 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5370006790 msgr2=0x7f5370008c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:15:49.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.714 7f53877fe700 1 --2- 172.21.2.112:0/2968425103 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5370006790 0x7f5370008c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:49.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.714 7f53877fe700 1 -- 172.21.2.112:0/2968425103 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f537000c9b0 con 0x7f5370003aa0 2021-02-10T11:15:49.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.714 7f53877fe700 1 --2- 172.21.2.112:0/2968425103 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5370003aa0 0x7f5370005ef0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0x7f537c000b50 tx=0x7f537c007d70).ready entity=mon.0 client_cookie=d195d0303eda66fe server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:49.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.714 7f53867fc700 1 -- 172.21.2.112:0/2968425103 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f537c00a310 con 0x7f5370003aa0 2021-02-10T11:15:49.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.714 7f53867fc700 1 -- 172.21.2.112:0/2968425103 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f537c00a690 con 0x7f5370003aa0 2021-02-10T11:15:49.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.714 7f53867fc700 1 -- 172.21.2.112:0/2968425103 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f537c00ab70 con 0x7f5370003aa0 2021-02-10T11:15:49.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.714 7f538e14e700 1 -- 172.21.2.112:0/2968425103 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5370003aa0 msgr2=0x7f5370005ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:49.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.714 7f538e14e700 1 --2- 172.21.2.112:0/2968425103 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5370003aa0 0x7f5370005ef0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0x7f537c000b50 tx=0x7f537c007d70).stop 2021-02-10T11:15:49.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.714 7f5387fff700 1 -- 172.21.2.112:0/2968425103 reap_dead start 2021-02-10T11:15:49.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.715 7f538e14e700 1 -- 172.21.2.112:0/2968425103 shutdown_connections 2021-02-10T11:15:49.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.715 7f538e14e700 1 -- 172.21.2.112:0/2968425103 >> 172.21.2.112:0/2968425103 conn(0x7f53880ce3e0 msgr2=0x7f53880e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:49.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.715 7f538e14e700 1 -- 172.21.2.112:0/2968425103 shutdown_connections 2021-02-10T11:15:49.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.715 7f538e14e700 1 -- 172.21.2.112:0/2968425103 wait complete. 2021-02-10T11:15:49.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.729 7f538e14e700 1 Processor -- start 2021-02-10T11:15:49.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.730 7f538e14e700 1 -- start start 2021-02-10T11:15:49.732 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.730 7f538e14e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f538809b930 0x7f53880b7c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:49.732 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.730 7f538e14e700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f53880b8410 0x7f53880b8810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:49.733 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.730 7f538e14e700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f53880c1f80 0x7f53880c2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:49.733 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.730 7f538e14e700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53880c1a00 con 0x7f538809b930 2021-02-10T11:15:49.733 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.730 7f538e14e700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53880b8e30 con 0x7f53880c1f80 2021-02-10T11:15:49.733 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.730 7f538e14e700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53880b9050 con 0x7f53880b8410 2021-02-10T11:15:49.733 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.730 7f53877fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f538809b930 0x7f53880b7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:49.733 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.730 7f53877fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f538809b930 0x7f53880b7c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:43700/0 (socket says 172.21.2.112:43700) 2021-02-10T11:15:49.733 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.730 7f53877fe700 1 -- 172.21.2.112:0/589386647 learned_addr learned my addr 172.21.2.112:0/589386647 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:15:49.733 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.730 7f53877fe700 1 -- 172.21.2.112:0/589386647 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f53880b8410 msgr2=0x7f53880b8810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:49.733 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.731 7f53877fe700 1 --2- 172.21.2.112:0/589386647 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f53880b8410 0x7f53880b8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:49.733 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.731 7f53877fe700 1 -- 172.21.2.112:0/589386647 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f53880c1f80 msgr2=0x7f53880c2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:49.733 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.731 7f53877fe700 1 --2- 172.21.2.112:0/589386647 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f53880c1f80 0x7f53880c2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:49.733 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.731 7f53877fe700 1 -- 172.21.2.112:0/589386647 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53880c29a0 con 0x7f538809b930 2021-02-10T11:15:49.733 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.731 7f53877fe700 1 --2- 172.21.2.112:0/589386647 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f538809b930 0x7f53880b7c20 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7f537c0012f0 tx=0x7f537c00a930).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:49.734 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.731 7f536ffff700 1 -- 172.21.2.112:0/589386647 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f537c007770 con 0x7f538809b930 2021-02-10T11:15:49.734 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.731 7f536ffff700 1 -- 172.21.2.112:0/589386647 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f537c00ad20 con 0x7f538809b930 2021-02-10T11:15:49.734 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.731 7f538e14e700 1 -- 172.21.2.112:0/589386647 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53880c29a0 con 0x7f538809b930 2021-02-10T11:15:49.734 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.731 7f538e14e700 1 -- 172.21.2.112:0/589386647 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53880c2f30 con 0x7f538809b930 2021-02-10T11:15:49.734 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.731 7f536ffff700 1 -- 172.21.2.112:0/589386647 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f537c004330 con 0x7f538809b930 2021-02-10T11:15:49.734 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.732 7f536ffff700 1 -- 172.21.2.112:0/589386647 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f537c004140 con 0x7f538809b930 2021-02-10T11:15:49.734 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.732 7f536ffff700 1 --2- 172.21.2.112:0/589386647 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5368018730 0x7f536801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:49.734 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.732 7f536ffff700 1 -- 172.21.2.112:0/589386647 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f537c0327c0 con 0x7f538809b930 2021-02-10T11:15:49.734 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.732 7f5386ffd700 1 --2- 172.21.2.112:0/589386647 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5368018730 0x7f536801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:49.734 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.733 7f5386ffd700 1 --2- 172.21.2.112:0/589386647 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5368018730 0x7f536801ac10 crc :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:49.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.762 7f538e14e700 1 -- 172.21.2.112:0/589386647 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5388105b30 con 0x7f538809b930 2021-02-10T11:15:49.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.765 7f536ffff700 1 -- 172.21.2.112:0/589386647 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f537c01f500 con 0x7f538809b930 2021-02-10T11:15:49.891 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.892 7f538e14e700 1 -- 172.21.2.112:0/589386647 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f53880f0b50 con 0x7f538809b930 2021-02-10T11:15:49.891 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.892 7f536ffff700 1 -- 172.21.2.112:0/589386647 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v29) v1 ==== 92+0+10 (secure 0 0 0) 0x7f537c004140 con 0x7f538809b930 2021-02-10T11:15:49.892 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:15:49.893 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.894 7f538e14e700 1 -- 172.21.2.112:0/589386647 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5368018730 msgr2=0x7f536801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:49.893 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.894 7f538e14e700 1 --2- 172.21.2.112:0/589386647 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5368018730 0x7f536801ac10 crc :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:49.893 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.894 7f538e14e700 1 -- 172.21.2.112:0/589386647 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f538809b930 msgr2=0x7f53880b7c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:49.893 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.894 7f538e14e700 1 --2- 172.21.2.112:0/589386647 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f538809b930 0x7f53880b7c20 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7f537c0012f0 tx=0x7f537c00a930).stop 2021-02-10T11:15:49.893 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.894 7f538e14e700 1 -- 172.21.2.112:0/589386647 shutdown_connections 2021-02-10T11:15:49.893 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.894 7f538e14e700 1 --2- 172.21.2.112:0/589386647 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5368018730 0x7f536801ac10 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:49.894 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.894 7f538e14e700 1 --2- 172.21.2.112:0/589386647 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f53880c1f80 0x7f53880c2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:49.894 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.894 7f538e14e700 1 --2- 172.21.2.112:0/589386647 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f53880b8410 0x7f53880b8810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:49.894 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.894 7f538e14e700 1 --2- 172.21.2.112:0/589386647 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f538809b930 0x7f53880b7c20 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:49.894 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.894 7f538e14e700 1 -- 172.21.2.112:0/589386647 >> 172.21.2.112:0/589386647 conn(0x7f5388094e10 msgr2=0x7f53880a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:49.894 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.895 7f538e14e700 1 -- 172.21.2.112:0/589386647 shutdown_connections 2021-02-10T11:15:49.894 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:49.895 7f538e14e700 1 -- 172.21.2.112:0/589386647 wait complete. 2021-02-10T11:15:49.901 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:15:50.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.315 7f737aa3d700 1 Processor -- start 2021-02-10T11:15:50.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.315 7f737aa3d700 1 -- start start 2021-02-10T11:15:50.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.315 7f737aa3d700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f73741b0870 0x7f73740f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:50.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.315 7f737aa3d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f73740efe00 0x7f73740f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:50.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.315 7f737aa3d700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73741b0c70 con 0x7f73741b0470 2021-02-10T11:15:50.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.315 7f737aa3d700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73740e5ea0 con 0x7f73740efe00 2021-02-10T11:15:50.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.315 7f737aa3d700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73740f0b50 con 0x7f73741b0870 2021-02-10T11:15:50.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.315 7f737905a700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f73740efe00 0x7f73740f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:50.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.315 7f737905a700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f73740efe00 0x7f73740f0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:43710/0 (socket says 172.21.2.112:43710) 2021-02-10T11:15:50.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.315 7f737905a700 1 -- 172.21.2.112:0/2138472 learned_addr learned my addr 172.21.2.112:0/2138472 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:15:50.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.316 7f737905a700 1 -- 172.21.2.112:0/2138472 >> v2:172.21.2.120:3300/0 conn(0x7f73741b0870 msgr2=0x7f73740f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:15:50.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.316 7f737905a700 1 --2- 172.21.2.112:0/2138472 >> v2:172.21.2.120:3300/0 conn(0x7f73741b0870 0x7f73740f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:50.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.316 7f737905a700 1 -- 172.21.2.112:0/2138472 >> v1:172.21.2.120:6789/0 conn(0x7f73741b0470 legacy=0x7f73740ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:15:50.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.316 7f737905a700 1 -- 172.21.2.112:0/2138472 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73740f0820 con 0x7f73740efe00 2021-02-10T11:15:50.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.316 7f737905a700 1 --2- 172.21.2.112:0/2138472 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f73740efe00 0x7f73740f0200 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0x7f736c006f00 tx=0x7f736c009f10).ready entity=mon.0 client_cookie=c88294ed47f7b1ab server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:50.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.316 7f73737fe700 1 -- 172.21.2.112:0/2138472 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f736c00b3b0 con 0x7f73740efe00 2021-02-10T11:15:50.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.316 7f73737fe700 1 -- 172.21.2.112:0/2138472 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f736c00b630 con 0x7f73740efe00 2021-02-10T11:15:50.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.316 7f73737fe700 1 -- 172.21.2.112:0/2138472 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f736c00bb50 con 0x7f73740efe00 2021-02-10T11:15:50.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.316 7f737aa3d700 1 -- 172.21.2.112:0/2138472 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f73740efe00 msgr2=0x7f73740f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:50.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.316 7f737aa3d700 1 --2- 172.21.2.112:0/2138472 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f73740efe00 0x7f73740f0200 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0x7f736c006f00 tx=0x7f736c009f10).stop 2021-02-10T11:15:50.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.317 7f737aa3d700 1 -- 172.21.2.112:0/2138472 shutdown_connections 2021-02-10T11:15:50.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.317 7f737aa3d700 1 --2- 172.21.2.112:0/2138472 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f73740efe00 0x7f73740f0200 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:50.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.317 7f737aa3d700 1 --2- 172.21.2.112:0/2138472 >> v2:172.21.2.120:3300/0 conn(0x7f73741b0870 0x7f73740f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:50.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.317 7f737aa3d700 1 -- 172.21.2.112:0/2138472 >> 172.21.2.112:0/2138472 conn(0x7f73740ce3e0 msgr2=0x7f73740e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:50.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.317 7f737aa3d700 1 -- 172.21.2.112:0/2138472 shutdown_connections 2021-02-10T11:15:50.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.317 7f737aa3d700 1 -- 172.21.2.112:0/2138472 wait complete. 2021-02-10T11:15:50.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.331 7f737aa3d700 1 Processor -- start 2021-02-10T11:15:50.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.331 7f737aa3d700 1 -- start start 2021-02-10T11:15:50.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.332 7f737aa3d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f73740c36a0 0x7f73740c2b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:50.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.332 7f737aa3d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f73740c3190 0x7f73740b1d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:50.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.332 7f737aa3d700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f73740b2370 0x7f73740b2d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:50.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.332 7f737aa3d700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73740b27c0 con 0x7f73740c3190 2021-02-10T11:15:50.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.332 7f737aa3d700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73740b3360 con 0x7f73740b2370 2021-02-10T11:15:50.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.332 7f737aa3d700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73740b3540 con 0x7f73740c36a0 2021-02-10T11:15:50.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.332 7f7373fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f73740c3190 0x7f73740b1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:50.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.332 7f7373fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f73740c3190 0x7f73740b1d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:43716/0 (socket says 172.21.2.112:43716) 2021-02-10T11:15:50.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.332 7f7373fff700 1 -- 172.21.2.112:0/672923487 learned_addr learned my addr 172.21.2.112:0/672923487 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:15:50.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.332 7f7373fff700 1 -- 172.21.2.112:0/672923487 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f73740c36a0 msgr2=0x7f73740c2b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:50.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.332 7f7373fff700 1 --2- 172.21.2.112:0/672923487 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f73740c36a0 0x7f73740c2b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:50.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.332 7f7373fff700 1 -- 172.21.2.112:0/672923487 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f73740b2370 msgr2=0x7f73740b2d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:50.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.332 7f7373fff700 1 --2- 172.21.2.112:0/672923487 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f73740b2370 0x7f73740b2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:50.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.332 7f7373fff700 1 -- 172.21.2.112:0/672923487 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73740b3760 con 0x7f73740c3190 2021-02-10T11:15:50.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.332 7f7373fff700 1 --2- 172.21.2.112:0/672923487 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f73740c3190 0x7f73740b1d50 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7f7364006f00 tx=0x7f736400b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:50.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.332 7f73717fa700 1 -- 172.21.2.112:0/672923487 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f73640009b0 con 0x7f73740c3190 2021-02-10T11:15:50.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.333 7f73717fa700 1 -- 172.21.2.112:0/672923487 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f736400bdc0 con 0x7f73740c3190 2021-02-10T11:15:50.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.333 7f73717fa700 1 -- 172.21.2.112:0/672923487 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7364002020 con 0x7f73740c3190 2021-02-10T11:15:50.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.333 7f737aa3d700 1 -- 172.21.2.112:0/672923487 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73740b3760 con 0x7f73740c3190 2021-02-10T11:15:50.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.333 7f737aa3d700 1 -- 172.21.2.112:0/672923487 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73740b3760 con 0x7f73740c3190 2021-02-10T11:15:50.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.333 7f73717fa700 1 -- 172.21.2.112:0/672923487 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7364005310 con 0x7f73740c3190 2021-02-10T11:15:50.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.333 7f73717fa700 1 --2- 172.21.2.112:0/672923487 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f73580186c0 0x7f735801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:50.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.333 7f73717fa700 1 -- 172.21.2.112:0/672923487 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f7364032580 con 0x7f73740c3190 2021-02-10T11:15:50.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.335 7f7378859700 1 --2- 172.21.2.112:0/672923487 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f73580186c0 0x7f735801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:50.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.335 7f7378859700 1 --2- 172.21.2.112:0/672923487 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f73580186c0 0x7f735801abc0 crc :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:50.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.365 7f737aa3d700 1 -- 172.21.2.112:0/672923487 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73740edd10 con 0x7f73740c3190 2021-02-10T11:15:50.366 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.368 7f73717fa700 1 -- 172.21.2.112:0/672923487 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7364006350 con 0x7f73740c3190 2021-02-10T11:15:50.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.497 7f737aa3d700 1 -- 172.21.2.112:0/672923487 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f73740e52e0 con 0x7f73740c3190 2021-02-10T11:15:50.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.497 7f73717fa700 1 -- 172.21.2.112:0/672923487 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v29) v1 ==== 104+0+10 (secure 0 0 0) 0x7f7364005310 con 0x7f73740c3190 2021-02-10T11:15:50.497 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:15:50.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.499 7f737aa3d700 1 -- 172.21.2.112:0/672923487 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f73580186c0 msgr2=0x7f735801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:50.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.499 7f737aa3d700 1 --2- 172.21.2.112:0/672923487 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f73580186c0 0x7f735801abc0 crc :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:50.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.499 7f737aa3d700 1 -- 172.21.2.112:0/672923487 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f73740c3190 msgr2=0x7f73740b1d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:50.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.499 7f737aa3d700 1 --2- 172.21.2.112:0/672923487 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f73740c3190 0x7f73740b1d50 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7f7364006f00 tx=0x7f736400b940).stop 2021-02-10T11:15:50.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.499 7f737aa3d700 1 -- 172.21.2.112:0/672923487 shutdown_connections 2021-02-10T11:15:50.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.499 7f737aa3d700 1 --2- 172.21.2.112:0/672923487 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f73580186c0 0x7f735801abc0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:50.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.499 7f737aa3d700 1 --2- 172.21.2.112:0/672923487 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f73740b2370 0x7f73740b2d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:50.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.499 7f737aa3d700 1 --2- 172.21.2.112:0/672923487 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f73740c3190 0x7f73740b1d50 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:50.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.499 7f737aa3d700 1 --2- 172.21.2.112:0/672923487 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f73740c36a0 0x7f73740c2b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:50.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.499 7f737aa3d700 1 -- 172.21.2.112:0/672923487 >> 172.21.2.112:0/672923487 conn(0x7f737409f630 msgr2=0x7f73740a0130 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:50.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.499 7f737aa3d700 1 -- 172.21.2.112:0/672923487 shutdown_connections 2021-02-10T11:15:50.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.500 7f737aa3d700 1 -- 172.21.2.112:0/672923487 wait complete. 2021-02-10T11:15:50.506 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:15:50.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.921 7fb482091700 1 Processor -- start 2021-02-10T11:15:50.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.921 7fb482091700 1 -- start start 2021-02-10T11:15:50.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.922 7fb482091700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fb47c1b0870 0x7fb47c0f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:50.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.922 7fb482091700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fb47c0efe00 0x7fb47c0f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:50.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.922 7fb482091700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb47c1b0c70 con 0x7fb47c1b0470 2021-02-10T11:15:50.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.922 7fb482091700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb47c0e5ea0 con 0x7fb47c0efe00 2021-02-10T11:15:50.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.922 7fb482091700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb47c0f0b50 con 0x7fb47c1b0870 2021-02-10T11:15:50.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.922 7fb47b7fe700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fb47c1b0470 0x7fb47c0ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:43184/0 (socket says 172.21.2.112:43184) 2021-02-10T11:15:50.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.922 7fb47b7fe700 1 -- 172.21.2.112:0/4176502321 learned_addr learned my addr 172.21.2.112:0/4176502321 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:15:50.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.922 7fb47bfff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fb47c0efe00 0x7fb47c0f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:50.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.922 7fb47bfff700 1 -- 172.21.2.112:0/4176502321 >> v2:172.21.2.120:3300/0 conn(0x7fb47c1b0870 msgr2=0x7fb47c0f20e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:15:50.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.922 7fb47bfff700 1 --2- 172.21.2.112:0/4176502321 >> v2:172.21.2.120:3300/0 conn(0x7fb47c1b0870 0x7fb47c0f20e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:50.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.922 7fb47bfff700 1 -- 172.21.2.112:0/4176502321 >> v1:172.21.2.112:6789/0 conn(0x7fb47c1b0470 legacy=0x7fb47c0ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:50.938 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.922 7fb47bfff700 1 -- 172.21.2.112:0/4176502321 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb47c0f0820 con 0x7fb47c0efe00 2021-02-10T11:15:50.938 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.923 7fb47bfff700 1 --2- 172.21.2.112:0/4176502321 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb47c0efe00 0x7fb47c0f0200 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7fb474006e30 tx=0x7fb474009e60).ready entity=mon.0 client_cookie=b90d3e4fc4b9be22 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:50.938 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.923 7fb47a7fc700 1 -- 172.21.2.112:0/4176502321 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb474006a50 con 0x7fb47c0efe00 2021-02-10T11:15:50.938 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.923 7fb47a7fc700 1 -- 172.21.2.112:0/4176502321 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb47400b300 con 0x7fb47c0efe00 2021-02-10T11:15:50.938 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.923 7fb47a7fc700 1 -- 172.21.2.112:0/4176502321 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb47400b820 con 0x7fb47c0efe00 2021-02-10T11:15:50.938 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.923 7fb482091700 1 -- 172.21.2.112:0/4176502321 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb47c0efe00 msgr2=0x7fb47c0f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:50.938 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.923 7fb482091700 1 --2- 172.21.2.112:0/4176502321 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb47c0efe00 0x7fb47c0f0200 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7fb474006e30 tx=0x7fb474009e60).stop 2021-02-10T11:15:50.938 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.923 7fb482091700 1 -- 172.21.2.112:0/4176502321 shutdown_connections 2021-02-10T11:15:50.938 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.923 7fb482091700 1 --2- 172.21.2.112:0/4176502321 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb47c0efe00 0x7fb47c0f0200 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:50.938 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.923 7fb482091700 1 --2- 172.21.2.112:0/4176502321 >> v2:172.21.2.120:3300/0 conn(0x7fb47c1b0870 0x7fb47c0f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:50.938 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.923 7fb482091700 1 -- 172.21.2.112:0/4176502321 >> 172.21.2.112:0/4176502321 conn(0x7fb47c0ce3e0 msgr2=0x7fb47c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:50.938 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.924 7fb482091700 1 -- 172.21.2.112:0/4176502321 shutdown_connections 2021-02-10T11:15:50.939 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.924 7fb482091700 1 -- 172.21.2.112:0/4176502321 wait complete. 2021-02-10T11:15:50.940 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.938 7fb482091700 1 Processor -- start 2021-02-10T11:15:50.940 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.938 7fb482091700 1 -- start start 2021-02-10T11:15:50.940 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.939 7fb482091700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb47c0c36a0 0x7fb47c0c2b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:50.940 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.939 7fb482091700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb47c0c3190 0x7fb47c0b1d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:50.941 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.939 7fb482091700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb47c0b2370 0x7fb47c0b2d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:50.941 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.939 7fb482091700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb47c0b27c0 con 0x7fb47c0c36a0 2021-02-10T11:15:50.941 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.939 7fb482091700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb47c0b3360 con 0x7fb47c0b2370 2021-02-10T11:15:50.941 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.939 7fb482091700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb47c0b3540 con 0x7fb47c0c3190 2021-02-10T11:15:50.941 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.939 7fb47b7fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb47c0c36a0 0x7fb47c0c2b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:50.941 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.939 7fb47b7fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb47c0c36a0 0x7fb47c0c2b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:43728/0 (socket says 172.21.2.112:43728) 2021-02-10T11:15:50.941 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.939 7fb47b7fe700 1 -- 172.21.2.112:0/3475212301 learned_addr learned my addr 172.21.2.112:0/3475212301 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:15:50.941 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.939 7fb47b7fe700 1 -- 172.21.2.112:0/3475212301 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb47c0c3190 msgr2=0x7fb47c0b1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:50.941 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.939 7fb47b7fe700 1 --2- 172.21.2.112:0/3475212301 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb47c0c3190 0x7fb47c0b1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:50.941 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.939 7fb47b7fe700 1 -- 172.21.2.112:0/3475212301 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb47c0b2370 msgr2=0x7fb47c0b2d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:50.941 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.939 7fb47b7fe700 1 --2- 172.21.2.112:0/3475212301 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb47c0b2370 0x7fb47c0b2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:50.941 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.939 7fb47b7fe700 1 -- 172.21.2.112:0/3475212301 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb47c0b3760 con 0x7fb47c0c36a0 2021-02-10T11:15:50.941 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.939 7fb47b7fe700 1 --2- 172.21.2.112:0/3475212301 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb47c0c36a0 0x7fb47c0c2b70 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 rx=0x7fb464001c60 tx=0x7fb46400a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:50.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.939 7fb473fff700 1 -- 172.21.2.112:0/3475212301 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb46400acc0 con 0x7fb47c0c36a0 2021-02-10T11:15:50.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.939 7fb473fff700 1 -- 172.21.2.112:0/3475212301 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb464004140 con 0x7fb47c0c36a0 2021-02-10T11:15:50.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.939 7fb482091700 1 -- 172.21.2.112:0/3475212301 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb47c0b3760 con 0x7fb47c0c36a0 2021-02-10T11:15:50.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.939 7fb473fff700 1 -- 172.21.2.112:0/3475212301 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb4640045a0 con 0x7fb47c0c36a0 2021-02-10T11:15:50.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.940 7fb482091700 1 -- 172.21.2.112:0/3475212301 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb47c0b3cf0 con 0x7fb47c0c36a0 2021-02-10T11:15:50.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.940 7fb473fff700 1 -- 172.21.2.112:0/3475212301 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb464004320 con 0x7fb47c0c36a0 2021-02-10T11:15:50.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.940 7fb473fff700 1 --2- 172.21.2.112:0/3475212301 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb4600186c0 0x7fb46001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:50.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.940 7fb473fff700 1 -- 172.21.2.112:0/3475212301 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fb4640323f0 con 0x7fb47c0c36a0 2021-02-10T11:15:50.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.941 7fb47affd700 1 --2- 172.21.2.112:0/3475212301 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb4600186c0 0x7fb46001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:50.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.941 7fb47affd700 1 --2- 172.21.2.112:0/3475212301 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb4600186c0 0x7fb46001abc0 crc :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:50.971 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.972 7fb482091700 1 -- 172.21.2.112:0/3475212301 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb47c0edd10 con 0x7fb47c0c36a0 2021-02-10T11:15:50.973 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:50.975 7fb473fff700 1 -- 172.21.2.112:0/3475212301 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb46402cef0 con 0x7fb47c0c36a0 2021-02-10T11:15:51.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:51.102 7fb482091700 1 -- 172.21.2.112:0/3475212301 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7fb47c0e52e0 con 0x7fb47c0c36a0 2021-02-10T11:15:51.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:51.102 7fb473fff700 1 -- 172.21.2.112:0/3475212301 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v29) v1 ==== 100+0+10 (secure 0 0 0) 0x7fb46402cef0 con 0x7fb47c0c36a0 2021-02-10T11:15:51.102 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:15:51.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:51.104 7fb482091700 1 -- 172.21.2.112:0/3475212301 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb4600186c0 msgr2=0x7fb46001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:51.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:51.104 7fb482091700 1 --2- 172.21.2.112:0/3475212301 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb4600186c0 0x7fb46001abc0 crc :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:51.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:51.104 7fb482091700 1 -- 172.21.2.112:0/3475212301 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb47c0c36a0 msgr2=0x7fb47c0c2b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:51.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:51.104 7fb482091700 1 --2- 172.21.2.112:0/3475212301 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb47c0c36a0 0x7fb47c0c2b70 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 rx=0x7fb464001c60 tx=0x7fb46400a8d0).stop 2021-02-10T11:15:51.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:51.105 7fb482091700 1 -- 172.21.2.112:0/3475212301 shutdown_connections 2021-02-10T11:15:51.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:51.105 7fb482091700 1 --2- 172.21.2.112:0/3475212301 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb4600186c0 0x7fb46001abc0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:51.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:51.105 7fb482091700 1 --2- 172.21.2.112:0/3475212301 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb47c0b2370 0x7fb47c0b2d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:51.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:51.105 7fb482091700 1 --2- 172.21.2.112:0/3475212301 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb47c0c3190 0x7fb47c0b1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:51.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:51.105 7fb482091700 1 --2- 172.21.2.112:0/3475212301 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb47c0c36a0 0x7fb47c0c2b70 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:51.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:51.105 7fb482091700 1 -- 172.21.2.112:0/3475212301 >> 172.21.2.112:0/3475212301 conn(0x7fb47c09f630 msgr2=0x7fb47c0a0130 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:51.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:51.105 7fb482091700 1 -- 172.21.2.112:0/3475212301 shutdown_connections 2021-02-10T11:15:51.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:51.105 7fb482091700 1 -- 172.21.2.112:0/3475212301 wait complete. 2021-02-10T11:15:51.112 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:15:51.112 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:15:51.538 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:51.521 7f3c2f702700 1 Processor -- start 2021-02-10T11:15:51.539 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:51.522 7f3c2f702700 1 -- start start 2021-02-10T11:15:51.539 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:51.522 7f3c2f702700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f3c280e5470 0x7f3c2807e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:51.539 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:51.522 7f3c2f702700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3c2807f300 0x7f3c2807f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:51.539 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:51.522 7f3c2f702700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c2807ed80 con 0x7f3c280e5070 2021-02-10T11:15:51.539 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:51.522 7f3c2f702700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c281b4010 con 0x7f3c280e5470 2021-02-10T11:15:51.539 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:51.522 7f3c2f702700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c280d1790 con 0x7f3c2807f300 2021-02-10T11:15:51.539 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:51.522 7f3c2d51e700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f3c280e5070 0x7f3c281b3820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:43198/0 (socket says 172.21.2.112:43198) 2021-02-10T11:15:51.540 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:51.522 7f3c2d51e700 1 -- 172.21.2.112:0/844179354 learned_addr learned my addr 172.21.2.112:0/844179354 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:15:51.540 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:51.522 7f3c2cd1d700 1 --2- 172.21.2.112:0/844179354 >> v2:172.21.2.112:3300/0 conn(0x7f3c280e5470 0x7f3c2807e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:51.540 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:51.522 7f3c2dd1f700 1 --2- 172.21.2.112:0/844179354 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3c2807f300 0x7f3c2807f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:51.540 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:51.523 7f3c1ffff700 1 -- 172.21.2.112:0/844179354 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2381562074 0 0) 0x7f3c20001500 con 0x7f3c280e5070 2021-02-10T11:15:51.540 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:51.523 7f3c1ffff700 1 -- 172.21.2.112:0/844179354 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c14002fe0 con 0x7f3c280e5070 2021-02-10T11:15:51.540 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:51.523 7f3c2cd1d700 1 -- 172.21.2.112:0/844179354 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3c2807f300 msgr2=0x7f3c2807f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:15:51.540 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:51.523 7f3c2cd1d700 1 --2- 172.21.2.112:0/844179354 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3c2807f300 0x7f3c2807f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:51.541 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:51.523 7f3c2cd1d700 1 -- 172.21.2.112:0/844179354 >> v1:172.21.2.112:6789/0 conn(0x7f3c280e5070 legacy=0x7f3c281b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:51.541 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:51.523 7f3c2cd1d700 1 -- 172.21.2.112:0/844179354 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c280d19b0 con 0x7f3c280e5470 2021-02-10T11:15:51.541 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:51.523 7f3c2cd1d700 1 --2- 172.21.2.112:0/844179354 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3c280e5470 0x7f3c2807e6a0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 rx=0x7f3c18006650 tx=0x7f3c18009a00).ready entity=mon.0 client_cookie=221904de28b9a872 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:51.541 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:51.523 7f3c1ffff700 1 -- 172.21.2.112:0/844179354 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3c180009b0 con 0x7f3c280e5470 2021-02-10T11:15:51.541 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:51.523 7f3c1ffff700 1 -- 172.21.2.112:0/844179354 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c1800c0c0 con 0x7f3c280e5470 2021-02-10T11:15:51.541 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:51.523 7f3c1ffff700 1 -- 172.21.2.112:0/844179354 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3c1800c5a0 con 0x7f3c280e5470 2021-02-10T11:15:51.541 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:51.523 7f3c2f702700 1 -- 172.21.2.112:0/844179354 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3c280e5470 msgr2=0x7f3c2807e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:51.541 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:51.523 7f3c2f702700 1 --2- 172.21.2.112:0/844179354 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3c280e5470 0x7f3c2807e6a0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 rx=0x7f3c18006650 tx=0x7f3c18009a00).stop 2021-02-10T11:15:51.541 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:51.524 7f3c2f702700 1 -- 172.21.2.112:0/844179354 shutdown_connections 2021-02-10T11:15:51.542 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:51.524 7f3c2f702700 1 --2- 172.21.2.112:0/844179354 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3c2807f300 0x7f3c2807f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:51.542 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:51.524 7f3c2f702700 1 --2- 172.21.2.112:0/844179354 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3c280e5470 0x7f3c2807e6a0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:51.542 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:51.524 7f3c2f702700 1 -- 172.21.2.112:0/844179354 >> 172.21.2.112:0/844179354 conn(0x7f3c280ce690 msgr2=0x7f3c280e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:51.542 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:51.524 7f3c2f702700 1 -- 172.21.2.112:0/844179354 shutdown_connections 2021-02-10T11:15:51.542 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:51.524 7f3c2f702700 1 -- 172.21.2.112:0/844179354 wait complete. 2021-02-10T11:15:51.542 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:51.538 7f3c2f702700 1 Processor -- start 2021-02-10T11:15:51.542 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:51.538 7f3c2f702700 1 -- start start 2021-02-10T11:15:51.542 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:51.539 7f3c2f702700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3c28191bb0 0x7f3c280a2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:51.542 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:51.539 7f3c2f702700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3c280a29a0 0x7f3c280a2da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:51.542 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:51.539 7f3c2f702700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3c280a33c0 0x7f3c280a03d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:51.542 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:51.539 7f3c2f702700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c280adf90 con 0x7f3c28191bb0 2021-02-10T11:15:51.543 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:51.539 7f3c2f702700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c280a1e00 con 0x7f3c280a33c0 2021-02-10T11:15:51.543 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:51.539 7f3c2f702700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c280a09f0 con 0x7f3c280a29a0 2021-02-10T11:15:51.543 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:51.539 7f3c2d51e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3c28191bb0 0x7f3c280a2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:51.543 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:51.539 7f3c2d51e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3c28191bb0 0x7f3c280a2380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:43742/0 (socket says 172.21.2.112:43742) 2021-02-10T11:15:51.543 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:51.539 7f3c2d51e700 1 -- 172.21.2.112:0/1108240100 learned_addr learned my addr 172.21.2.112:0/1108240100 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:15:51.543 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:51.539 7f3c2cd1d700 1 --2- 172.21.2.112:0/1108240100 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3c280a29a0 0x7f3c280a2da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:51.543 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:51.539 7f3c2dd1f700 1 --2- 172.21.2.112:0/1108240100 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3c280a33c0 0x7f3c280a03d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:51.543 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:51.539 7f3c2d51e700 1 -- 172.21.2.112:0/1108240100 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3c280a29a0 msgr2=0x7f3c280a2da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:51.543 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:51.539 7f3c2d51e700 1 --2- 172.21.2.112:0/1108240100 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3c280a29a0 0x7f3c280a2da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:51.546 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:51.539 7f3c2d51e700 1 -- 172.21.2.112:0/1108240100 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3c280a33c0 msgr2=0x7f3c280a03d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:51.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:51.539 7f3c2d51e700 1 --2- 172.21.2.112:0/1108240100 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3c280a33c0 0x7f3c280a03d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:51.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:51.539 7f3c2d51e700 1 -- 172.21.2.112:0/1108240100 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c280a0bd0 con 0x7f3c28191bb0 2021-02-10T11:15:51.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:51.540 7f3c2d51e700 1 --2- 172.21.2.112:0/1108240100 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3c28191bb0 0x7f3c280a2380 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7f3c20009980 tx=0x7f3c2000bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:51.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:51.540 7f3c1dffb700 1 -- 172.21.2.112:0/1108240100 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3c20006030 con 0x7f3c28191bb0 2021-02-10T11:15:51.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:51.540 7f3c1dffb700 1 -- 172.21.2.112:0/1108240100 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c200062b0 con 0x7f3c28191bb0 2021-02-10T11:15:51.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:51.540 7f3c2f702700 1 -- 172.21.2.112:0/1108240100 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c280a0bd0 con 0x7f3c28191bb0 2021-02-10T11:15:51.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:51.540 7f3c1dffb700 1 -- 172.21.2.112:0/1108240100 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3c20004720 con 0x7f3c28191bb0 2021-02-10T11:15:51.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:51.540 7f3c2f702700 1 -- 172.21.2.112:0/1108240100 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c280a1160 con 0x7f3c28191bb0 2021-02-10T11:15:51.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:51.540 7f3c1dffb700 1 -- 172.21.2.112:0/1108240100 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3c20006490 con 0x7f3c28191bb0 2021-02-10T11:15:51.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:51.540 7f3c1dffb700 1 --2- 172.21.2.112:0/1108240100 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3c080186d0 0x7f3c0801abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:51.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:51.540 7f3c1dffb700 1 -- 172.21.2.112:0/1108240100 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f3c20032bb0 con 0x7f3c28191bb0 2021-02-10T11:15:51.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:51.541 7f3c2cd1d700 1 --2- 172.21.2.112:0/1108240100 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3c080186d0 0x7f3c0801abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:51.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:51.541 7f3c2cd1d700 1 --2- 172.21.2.112:0/1108240100 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3c080186d0 0x7f3c0801abd0 crc :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:51.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:51.572 7f3c2f702700 1 -- 172.21.2.112:0/1108240100 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c2807ed80 con 0x7f3c28191bb0 2021-02-10T11:15:51.573 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:51.574 7f3c1dffb700 1 -- 172.21.2.112:0/1108240100 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3c200064a0 con 0x7f3c28191bb0 2021-02-10T11:15:51.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:51.703 7f3c2f702700 1 -- 172.21.2.112:0/1108240100 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3c280adf90 con 0x7f3c28191bb0 2021-02-10T11:15:51.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:51.704 7f3c1dffb700 1 -- 172.21.2.112:0/1108240100 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+2100 (secure 0 0 0) 0x7f3c20006290 con 0x7f3c28191bb0 2021-02-10T11:15:51.703 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:15:51.704 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":13,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4627,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/2302110857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":2302110857},{"type":"v1","addr":"172.21.2.112:6833","nonce":2302110857}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":9}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:15:33.919772","modified":"2021-02-10 11:15:36.309405","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4427},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4427":{"gid":4427,"name":"b","rank":0,"incarnation":10,"state":"up:active","state_seq":4,"addr":"172.21.2.120:6835/3730022736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":3730022736},{"type":"v1","addr":"172.21.2.120:6835","nonce":3730022736}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2021-02-10T11:15:51.704 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 13 2021-02-10T11:15:51.704 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:51.705 7f3c2f702700 1 -- 172.21.2.112:0/1108240100 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3c080186d0 msgr2=0x7f3c0801abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:51.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:51.705 7f3c2f702700 1 --2- 172.21.2.112:0/1108240100 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3c080186d0 0x7f3c0801abd0 crc :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:51.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:51.705 7f3c2f702700 1 -- 172.21.2.112:0/1108240100 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3c28191bb0 msgr2=0x7f3c280a2380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:51.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:51.705 7f3c2f702700 1 --2- 172.21.2.112:0/1108240100 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3c28191bb0 0x7f3c280a2380 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7f3c20009980 tx=0x7f3c2000bcd0).stop 2021-02-10T11:15:51.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:51.706 7f3c2f702700 1 -- 172.21.2.112:0/1108240100 shutdown_connections 2021-02-10T11:15:51.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:51.706 7f3c2f702700 1 --2- 172.21.2.112:0/1108240100 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3c080186d0 0x7f3c0801abd0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:51.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:51.706 7f3c2f702700 1 --2- 172.21.2.112:0/1108240100 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3c280a33c0 0x7f3c280a03d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:51.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:51.706 7f3c2f702700 1 --2- 172.21.2.112:0/1108240100 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3c280a29a0 0x7f3c280a2da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:51.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:51.706 7f3c2f702700 1 --2- 172.21.2.112:0/1108240100 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3c28191bb0 0x7f3c280a2380 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:51.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:51.706 7f3c2f702700 1 -- 172.21.2.112:0/1108240100 >> 172.21.2.112:0/1108240100 conn(0x7f3c281a31d0 msgr2=0x7f3c2808b830 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:51.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:51.706 7f3c2f702700 1 -- 172.21.2.112:0/1108240100 shutdown_connections 2021-02-10T11:15:51.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:51.706 7f3c2f702700 1 -- 172.21.2.112:0/1108240100 wait complete. 2021-02-10T11:15:51.915 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 13, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:15:33.919772', 'modified': '2021-02-10 11:15:36.309405', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4427}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4427': {'gid': 4427, 'name': 'b', 'rank': 0, 'incarnation': 10, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.2.120:6835/3730022736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.120:6834', 'nonce': 3730022736}, {'type': 'v1', 'addr': '172.21.2.120:6835', 'nonce': 3730022736}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:15:51.915 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:15:51.915 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:15:52.328 INFO:teuthology.orchestra.run.gibba020.stdout:{ 2021-02-10T11:15:52.328 INFO:teuthology.orchestra.run.gibba020.stdout: "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", 2021-02-10T11:15:52.329 INFO:teuthology.orchestra.run.gibba020.stdout: "whoami": 0, 2021-02-10T11:15:52.329 INFO:teuthology.orchestra.run.gibba020.stdout: "id": 4427, 2021-02-10T11:15:52.329 INFO:teuthology.orchestra.run.gibba020.stdout: "want_state": "up:active", 2021-02-10T11:15:52.329 INFO:teuthology.orchestra.run.gibba020.stdout: "state": "up:active", 2021-02-10T11:15:52.329 INFO:teuthology.orchestra.run.gibba020.stdout: "rank_uptime": 18.408872825, 2021-02-10T11:15:52.329 INFO:teuthology.orchestra.run.gibba020.stdout: "mdsmap_epoch": 13, 2021-02-10T11:15:52.329 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch": 29, 2021-02-10T11:15:52.329 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch_barrier": 29, 2021-02-10T11:15:52.329 INFO:teuthology.orchestra.run.gibba020.stdout: "uptime": 28.045983892999999 2021-02-10T11:15:52.329 INFO:teuthology.orchestra.run.gibba020.stdout:} 2021-02-10T11:15:52.332 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", "whoami": 0, "id": 4427, "want_state": "up:active", "state": "up:active", "rank_uptime": 18.408872825, "mdsmap_epoch": 13, "osdmap_epoch": 29, "osdmap_epoch_barrier": 29, "uptime": 28.045983892999999 } 2021-02-10T11:15:52.333 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:15:52.333 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2021-02-10T11:15:52.333 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.1 at ubuntu@gibba012.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2021-02-10T11:15:52.333 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:15:52.333 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:15:52.671 DEBUG:teuthology.orchestra.run.gibba012:> sudo modprobe fuse 2021-02-10T11:15:53.019 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:15:53.020 DEBUG:teuthology.orchestra.run.gibba012:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:15:53.363 INFO:teuthology.orchestra.run.gibba012.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:15:53.363 INFO:teuthology.orchestra.run.gibba012.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:15:53.364 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:15:53.364 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:15:53.365 DEBUG:teuthology.orchestra.run.gibba012:> ls /sys/fs/fuse/connections 2021-02-10T11:15:53.700 INFO:teuthology.orchestra.run.gibba012.stdout:42 2021-02-10T11:15:53.701 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [42] 2021-02-10T11:15:53.701 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.1 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:15:53.704 DEBUG:teuthology.orchestra.run.gibba012:> sudo modprobe fuse 2021-02-10T11:15:54.047 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:15:54.047 DEBUG:teuthology.orchestra.run.gibba012:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:15:54.112 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba012.stderr:ceph-fuse[45683]: starting ceph client 2021-02-10T11:15:54.112 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba012.stderr:2021-02-10 11:15:54.113 7ff0aeb4af80 -1 init, newargv = 0x55a814459130 newargc=9 2021-02-10T11:15:54.116 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba012.stderr:ceph-fuse[45683]: starting fuse 2021-02-10T11:15:54.389 INFO:teuthology.orchestra.run.gibba012.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:15:54.389 INFO:teuthology.orchestra.run.gibba012.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:15:54.390 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:15:54.390 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:15:54.390 DEBUG:teuthology.orchestra.run.gibba012:> ls /sys/fs/fuse/connections 2021-02-10T11:15:54.725 INFO:teuthology.orchestra.run.gibba012.stdout:42 2021-02-10T11:15:54.725 INFO:teuthology.orchestra.run.gibba012.stdout:43 2021-02-10T11:15:54.725 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42, 43] 2021-02-10T11:15:54.725 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:15:54.726 DEBUG:teuthology.orchestra.run.gibba012:> sudo python3 -c ' 2021-02-10T11:15:54.726 DEBUG:teuthology.orchestra.run.gibba012:> import glob 2021-02-10T11:15:54.726 DEBUG:teuthology.orchestra.run.gibba012:> import re 2021-02-10T11:15:54.726 DEBUG:teuthology.orchestra.run.gibba012:> import os 2021-02-10T11:15:54.726 DEBUG:teuthology.orchestra.run.gibba012:> import subprocess 2021-02-10T11:15:54.726 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:15:54.726 DEBUG:teuthology.orchestra.run.gibba012:> def find_socket(client_name): 2021-02-10T11:15:54.726 DEBUG:teuthology.orchestra.run.gibba012:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2021-02-10T11:15:54.726 DEBUG:teuthology.orchestra.run.gibba012:> files = glob.glob(asok_path) 2021-02-10T11:15:54.726 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:15:54.727 DEBUG:teuthology.orchestra.run.gibba012:> # Given a non-glob path, it better be there 2021-02-10T11:15:54.727 DEBUG:teuthology.orchestra.run.gibba012:> if "*" not in asok_path: 2021-02-10T11:15:54.727 DEBUG:teuthology.orchestra.run.gibba012:> assert(len(files) == 1) 2021-02-10T11:15:54.727 DEBUG:teuthology.orchestra.run.gibba012:> return files[0] 2021-02-10T11:15:54.727 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:15:54.727 DEBUG:teuthology.orchestra.run.gibba012:> for f in files: 2021-02-10T11:15:54.727 DEBUG:teuthology.orchestra.run.gibba012:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:15:54.727 DEBUG:teuthology.orchestra.run.gibba012:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:15:54.727 DEBUG:teuthology.orchestra.run.gibba012:> return f 2021-02-10T11:15:54.727 DEBUG:teuthology.orchestra.run.gibba012:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:15:54.727 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:15:54.728 DEBUG:teuthology.orchestra.run.gibba012:> print(find_socket("client.1")) 2021-02-10T11:15:54.728 DEBUG:teuthology.orchestra.run.gibba012:> ' 2021-02-10T11:15:55.089 INFO:teuthology.orchestra.run.gibba012.stdout:/var/run/ceph/ceph-client.1.45683.asok 2021-02-10T11:15:55.092 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.45683.asok 2021-02-10T11:15:55.092 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:15:55.092 DEBUG:teuthology.orchestra.run.gibba012:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.45683.asok status 2021-02-10T11:15:55.480 INFO:teuthology.orchestra.run.gibba012.stdout:{ 2021-02-10T11:15:55.480 INFO:teuthology.orchestra.run.gibba012.stdout: "metadata": { 2021-02-10T11:15:55.480 INFO:teuthology.orchestra.run.gibba012.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:15:55.481 INFO:teuthology.orchestra.run.gibba012.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:15:55.481 INFO:teuthology.orchestra.run.gibba012.stdout: "entity_id": "1", 2021-02-10T11:15:55.481 INFO:teuthology.orchestra.run.gibba012.stdout: "hostname": "gibba012", 2021-02-10T11:15:55.481 INFO:teuthology.orchestra.run.gibba012.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2021-02-10T11:15:55.481 INFO:teuthology.orchestra.run.gibba012.stdout: "pid": "45683", 2021-02-10T11:15:55.481 INFO:teuthology.orchestra.run.gibba012.stdout: "root": "/" 2021-02-10T11:15:55.481 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:55.481 INFO:teuthology.orchestra.run.gibba012.stdout: "dentry_count": 0, 2021-02-10T11:15:55.481 INFO:teuthology.orchestra.run.gibba012.stdout: "dentry_pinned_count": 0, 2021-02-10T11:15:55.481 INFO:teuthology.orchestra.run.gibba012.stdout: "id": 5049, 2021-02-10T11:15:55.481 INFO:teuthology.orchestra.run.gibba012.stdout: "inst": { 2021-02-10T11:15:55.481 INFO:teuthology.orchestra.run.gibba012.stdout: "name": { 2021-02-10T11:15:55.481 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "client", 2021-02-10T11:15:55.481 INFO:teuthology.orchestra.run.gibba012.stdout: "num": 5049 2021-02-10T11:15:55.482 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:55.482 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": { 2021-02-10T11:15:55.482 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:15:55.482 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:0", 2021-02-10T11:15:55.482 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 3735038355 2021-02-10T11:15:55.482 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:15:55.482 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:55.482 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": { 2021-02-10T11:15:55.482 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:15:55.482 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:0", 2021-02-10T11:15:55.482 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 3735038355 2021-02-10T11:15:55.482 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:15:55.482 INFO:teuthology.orchestra.run.gibba012.stdout: "inst_str": "client.5049 172.21.2.112:0/3735038355", 2021-02-10T11:15:55.483 INFO:teuthology.orchestra.run.gibba012.stdout: "addr_str": "172.21.2.112:0/3735038355", 2021-02-10T11:15:55.483 INFO:teuthology.orchestra.run.gibba012.stdout: "inode_count": 1, 2021-02-10T11:15:55.483 INFO:teuthology.orchestra.run.gibba012.stdout: "mds_epoch": 13, 2021-02-10T11:15:55.483 INFO:teuthology.orchestra.run.gibba012.stdout: "osd_epoch": 29, 2021-02-10T11:15:55.483 INFO:teuthology.orchestra.run.gibba012.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:15:55.483 INFO:teuthology.orchestra.run.gibba012.stdout: "blacklisted": false 2021-02-10T11:15:55.483 INFO:teuthology.orchestra.run.gibba012.stdout:} 2021-02-10T11:15:55.484 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:15:55.484 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:15:55.484 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:15:55.818 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:15:55.818 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:15:55.819 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:15:55.819 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:15:56.164 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2021-02-10T11:15:56.595 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:56.579 7f28bf891700 1 Processor -- start 2021-02-10T11:15:56.595 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:56.579 7f28bf891700 1 -- start start 2021-02-10T11:15:56.595 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:56.580 7f28bf891700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f28b80d1870 0x7f28b81b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:56.595 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:56.580 7f28bf891700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28b80fe1c0 con 0x7f28b80cff70 2021-02-10T11:15:56.595 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:56.580 7f28bf891700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28b8109a60 con 0x7f28b81b0ae0 2021-02-10T11:15:56.595 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:56.580 7f28bf891700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28b8088020 con 0x7f28b80d1870 2021-02-10T11:15:56.595 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:56.580 7f28bd6ad700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f28b80d1870 0x7f28b81b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:56.596 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:56.580 7f28bd6ad700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f28b80d1870 0x7f28b81b04c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:43762/0 (socket says 172.21.2.112:43762) 2021-02-10T11:15:56.596 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:56.580 7f28bd6ad700 1 -- 172.21.2.112:0/1657086825 learned_addr learned my addr 172.21.2.112:0/1657086825 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:15:56.596 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:56.580 7f28bd6ad700 1 -- 172.21.2.112:0/1657086825 >> v1:172.21.2.120:6789/0 conn(0x7f28b81b0ae0 legacy=0x7f28b8108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:56.596 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:56.580 7f28bd6ad700 1 -- 172.21.2.112:0/1657086825 >> v1:172.21.2.112:6789/0 conn(0x7f28b80cff70 legacy=0x7f28b8109130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:56.596 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:56.580 7f28bd6ad700 1 -- 172.21.2.112:0/1657086825 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28b8088210 con 0x7f28b80d1870 2021-02-10T11:15:56.596 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:56.580 7f28bd6ad700 1 --2- 172.21.2.112:0/1657086825 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f28b80d1870 0x7f28b81b04c0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 rx=0x7f28ac006700 tx=0x7f28ac009b80).ready entity=mon.0 client_cookie=81ea21d6dd00dc4e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:56.596 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:56.581 7f28abfff700 1 -- 172.21.2.112:0/1657086825 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f28ac00c0c0 con 0x7f28b80d1870 2021-02-10T11:15:56.596 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:56.581 7f28abfff700 1 -- 172.21.2.112:0/1657086825 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28ac00c340 con 0x7f28b80d1870 2021-02-10T11:15:56.596 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:56.581 7f28abfff700 1 -- 172.21.2.112:0/1657086825 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f28ac00c860 con 0x7f28b80d1870 2021-02-10T11:15:56.596 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:56.581 7f28bf891700 1 -- 172.21.2.112:0/1657086825 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f28b80d1870 msgr2=0x7f28b81b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:56.596 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:56.581 7f28bf891700 1 --2- 172.21.2.112:0/1657086825 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f28b80d1870 0x7f28b81b04c0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 rx=0x7f28ac006700 tx=0x7f28ac009b80).stop 2021-02-10T11:15:56.596 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:56.581 7f28bf891700 1 -- 172.21.2.112:0/1657086825 shutdown_connections 2021-02-10T11:15:56.597 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:56.581 7f28bf891700 1 --2- 172.21.2.112:0/1657086825 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f28b80d1870 0x7f28b81b04c0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:56.597 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:56.581 7f28bf891700 1 -- 172.21.2.112:0/1657086825 >> 172.21.2.112:0/1657086825 conn(0x7f28b80ce420 msgr2=0x7f28b80d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:56.597 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:56.581 7f28bf891700 1 -- 172.21.2.112:0/1657086825 shutdown_connections 2021-02-10T11:15:56.597 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:56.581 7f28bf891700 1 -- 172.21.2.112:0/1657086825 wait complete. 2021-02-10T11:15:56.597 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:56.596 7f28bf891700 1 Processor -- start 2021-02-10T11:15:56.597 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:56.596 7f28bf891700 1 -- start start 2021-02-10T11:15:56.598 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:56.597 7f28bf891700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f28b809c0e0 0x7f28b809cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:56.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:56.597 7f28bf891700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f28b809d240 0x7f28b809d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:56.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:56.597 7f28bf891700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f28b809dc60 0x7f28b8100d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:56.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:56.597 7f28bf891700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28b809b520 con 0x7f28b809d240 2021-02-10T11:15:56.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:56.597 7f28bf891700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28b809b740 con 0x7f28b809dc60 2021-02-10T11:15:56.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:56.597 7f28bf891700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28b81c68a0 con 0x7f28b809c0e0 2021-02-10T11:15:56.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:56.597 7f28bceac700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f28b809d240 0x7f28b809d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:56.600 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:56.597 7f28bceac700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f28b809d240 0x7f28b809d640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:43772/0 (socket says 172.21.2.112:43772) 2021-02-10T11:15:56.600 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:56.597 7f28bceac700 1 -- 172.21.2.112:0/3955274553 learned_addr learned my addr 172.21.2.112:0/3955274553 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:15:56.600 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:56.597 7f28bdeae700 1 --2- 172.21.2.112:0/3955274553 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f28b809dc60 0x7f28b8100d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:56.600 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:56.597 7f28bd6ad700 1 --2- 172.21.2.112:0/3955274553 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f28b809c0e0 0x7f28b809cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:56.600 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:56.597 7f28bceac700 1 -- 172.21.2.112:0/3955274553 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f28b809c0e0 msgr2=0x7f28b809cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:56.600 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:56.597 7f28bceac700 1 --2- 172.21.2.112:0/3955274553 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f28b809c0e0 0x7f28b809cc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:56.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:56.597 7f28bceac700 1 -- 172.21.2.112:0/3955274553 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f28b809dc60 msgr2=0x7f28b8100d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:56.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:56.597 7f28bceac700 1 --2- 172.21.2.112:0/3955274553 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f28b809dc60 0x7f28b8100d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:56.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:56.597 7f28bceac700 1 -- 172.21.2.112:0/3955274553 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28b81013a0 con 0x7f28b809d240 2021-02-10T11:15:56.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:56.597 7f28bceac700 1 --2- 172.21.2.112:0/3955274553 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f28b809d240 0x7f28b809d640 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7f28b4006ed0 tx=0x7f28b400b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:56.602 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:56.598 7f28a9ffb700 1 -- 172.21.2.112:0/3955274553 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f28b4006240 con 0x7f28b809d240 2021-02-10T11:15:56.602 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:56.598 7f28a9ffb700 1 -- 172.21.2.112:0/3955274553 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28b40054b0 con 0x7f28b809d240 2021-02-10T11:15:56.602 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:56.598 7f28a9ffb700 1 -- 172.21.2.112:0/3955274553 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f28b4005910 con 0x7f28b809d240 2021-02-10T11:15:56.602 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:56.598 7f28bf891700 1 -- 172.21.2.112:0/3955274553 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28b81013a0 con 0x7f28b809d240 2021-02-10T11:15:56.602 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:56.598 7f28bf891700 1 -- 172.21.2.112:0/3955274553 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28b81013a0 con 0x7f28b809d240 2021-02-10T11:15:56.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:56.598 7f28a9ffb700 1 -- 172.21.2.112:0/3955274553 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f28b4006030 con 0x7f28b809d240 2021-02-10T11:15:56.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:56.598 7f28a9ffb700 1 --2- 172.21.2.112:0/3955274553 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f28980186c0 0x7f289801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:56.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:56.598 7f28a9ffb700 1 -- 172.21.2.112:0/3955274553 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f28b40323b0 con 0x7f28b809d240 2021-02-10T11:15:56.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:56.599 7f28bd6ad700 1 --2- 172.21.2.112:0/3955274553 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f28980186c0 0x7f289801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:56.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:56.599 7f28bd6ad700 1 --2- 172.21.2.112:0/3955274553 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f28980186c0 0x7f289801abc0 crc :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:56.629 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:56.630 7f28bf891700 1 -- 172.21.2.112:0/3955274553 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28b8080650 con 0x7f28b809d240 2021-02-10T11:15:56.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:56.632 7f28a9ffb700 1 -- 172.21.2.112:0/3955274553 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f28b4000d00 con 0x7f28b809d240 2021-02-10T11:15:56.766 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:56.767 7f28bf891700 1 -- 172.21.2.112:0/3955274553 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f28b8080650 con 0x7f28b809d240 2021-02-10T11:15:56.839 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:56.840 7f28a9ffb700 1 -- 172.21.2.112:0/3955274553 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v14) v1 ==== 136+0+0 (secure 0 0 0) 0x7f28b4006030 con 0x7f28b809d240 2021-02-10T11:15:56.882 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:56.883 7f28bf891700 1 -- 172.21.2.112:0/3955274553 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f28980186c0 msgr2=0x7f289801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:56.883 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:56.883 7f28bf891700 1 --2- 172.21.2.112:0/3955274553 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f28980186c0 0x7f289801abc0 crc :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:56.883 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:56.883 7f28bf891700 1 -- 172.21.2.112:0/3955274553 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f28b809d240 msgr2=0x7f28b809d640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:56.883 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:56.883 7f28bf891700 1 --2- 172.21.2.112:0/3955274553 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f28b809d240 0x7f28b809d640 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7f28b4006ed0 tx=0x7f28b400b960).stop 2021-02-10T11:15:56.884 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:56.884 7f28bf891700 1 -- 172.21.2.112:0/3955274553 shutdown_connections 2021-02-10T11:15:56.884 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:56.884 7f28bf891700 1 --2- 172.21.2.112:0/3955274553 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f28980186c0 0x7f289801abc0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:56.884 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:56.884 7f28bf891700 1 --2- 172.21.2.112:0/3955274553 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f28b809dc60 0x7f28b8100d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:56.884 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:56.884 7f28bf891700 1 --2- 172.21.2.112:0/3955274553 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f28b809d240 0x7f28b809d640 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:56.884 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:56.884 7f28bf891700 1 --2- 172.21.2.112:0/3955274553 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f28b809c0e0 0x7f28b809cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:56.884 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:56.884 7f28bf891700 1 -- 172.21.2.112:0/3955274553 >> 172.21.2.112:0/3955274553 conn(0x7f28b809ba20 msgr2=0x7f28b80a7920 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:56.884 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:56.884 7f28bf891700 1 -- 172.21.2.112:0/3955274553 shutdown_connections 2021-02-10T11:15:56.884 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:56.884 7f28bf891700 1 -- 172.21.2.112:0/3955274553 wait complete. 2021-02-10T11:15:56.890 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs volume ls 2021-02-10T11:15:57.316 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.299 7f9d5ed42700 1 Processor -- start 2021-02-10T11:15:57.316 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.299 7f9d5ed42700 1 -- start start 2021-02-10T11:15:57.316 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.299 7f9d5ed42700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9d580e5070 0x7f9d581b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:57.316 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.299 7f9d5ed42700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f9d580e5470 0x7f9d5807d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:57.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.299 7f9d5ed42700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d5807ed80 con 0x7f9d581b3e40 2021-02-10T11:15:57.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.299 7f9d5ed42700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d58080310 con 0x7f9d580e5470 2021-02-10T11:15:57.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.299 7f9d5ed42700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d58080500 con 0x7f9d580e5070 2021-02-10T11:15:57.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.299 7f9d5d35f700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f9d581b3e40 0x7f9d5807dcb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:43242/0 (socket says 172.21.2.112:43242) 2021-02-10T11:15:57.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.299 7f9d5d35f700 1 -- 172.21.2.112:0/1573320570 learned_addr learned my addr 172.21.2.112:0/1573320570 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:15:57.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.300 7f9d577fe700 1 -- 172.21.2.112:0/1573320570 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1146525492 0 0) 0x7f9d50000c10 con 0x7f9d581b3e40 2021-02-10T11:15:57.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.300 7f9d577fe700 1 -- 172.21.2.112:0/1573320570 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d44002fe0 con 0x7f9d581b3e40 2021-02-10T11:15:57.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.300 7f9d57fff700 1 --2- 172.21.2.112:0/1573320570 >> v2:172.21.2.120:3300/0 conn(0x7f9d580e5470 0x7f9d5807d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:57.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.300 7f9d5cb5e700 1 --2- 172.21.2.112:0/1573320570 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9d580e5070 0x7f9d581b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:57.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.300 7f9d577fe700 1 -- 172.21.2.112:0/1573320570 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2620425492 0 0) 0x7f9d500010e0 con 0x7f9d581b3e40 2021-02-10T11:15:57.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.300 7f9d577fe700 1 -- 172.21.2.112:0/1573320570 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9d580e5070 msgr2=0x7f9d581b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:15:57.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.300 7f9d577fe700 1 --2- 172.21.2.112:0/1573320570 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9d580e5070 0x7f9d581b3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:57.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.300 7f9d577fe700 1 -- 172.21.2.112:0/1573320570 >> v2:172.21.2.120:3300/0 conn(0x7f9d580e5470 msgr2=0x7f9d5807d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:15:57.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.300 7f9d577fe700 1 --2- 172.21.2.112:0/1573320570 >> v2:172.21.2.120:3300/0 conn(0x7f9d580e5470 0x7f9d5807d690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:57.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.300 7f9d577fe700 1 -- 172.21.2.112:0/1573320570 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d580806f0 con 0x7f9d581b3e40 2021-02-10T11:15:57.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.300 7f9d577fe700 1 -- 172.21.2.112:0/1573320570 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f9d50001660 con 0x7f9d581b3e40 2021-02-10T11:15:57.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.300 7f9d577fe700 1 -- 172.21.2.112:0/1573320570 >> v1:172.21.2.112:6789/0 conn(0x7f9d581b3e40 legacy=0x7f9d5807dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:57.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.300 7f9d577fe700 1 --2- 172.21.2.112:0/1573320570 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9d44003b10 0x7f9d44006010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:57.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.300 7f9d577fe700 1 --2- 172.21.2.112:0/1573320570 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9d44006830 0x7f9d44008d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:57.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.300 7f9d577fe700 1 --2- 172.21.2.112:0/1573320570 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9d44009650 0x7f9d4400bb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:57.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.300 7f9d577fe700 1 -- 172.21.2.112:0/1573320570 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d4400c370 con 0x7f9d44003b10 2021-02-10T11:15:57.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.300 7f9d577fe700 1 -- 172.21.2.112:0/1573320570 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d44050560 con 0x7f9d44006830 2021-02-10T11:15:57.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.300 7f9d5cb5e700 1 --2- 172.21.2.112:0/1573320570 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9d44003b10 0x7f9d44006010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:57.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.300 7f9d577fe700 1 -- 172.21.2.112:0/1573320570 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d44050780 con 0x7f9d44009650 2021-02-10T11:15:57.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.301 7f9d57fff700 1 --2- 172.21.2.112:0/1573320570 >> v2:172.21.2.120:3300/0 conn(0x7f9d580e5470 0x7f9d5807d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:15:57.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.301 7f9d5d35f700 1 --2- 172.21.2.112:0/1573320570 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9d44009650 0x7f9d4400bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:57.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.301 7f9d5cb5e700 1 -- 172.21.2.112:0/1573320570 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9d44009650 msgr2=0x7f9d4400bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:15:57.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.301 7f9d5cb5e700 1 --2- 172.21.2.112:0/1573320570 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9d44009650 0x7f9d4400bb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:57.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.301 7f9d5cb5e700 1 -- 172.21.2.112:0/1573320570 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9d44006830 msgr2=0x7f9d44008d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:15:57.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.301 7f9d5cb5e700 1 --2- 172.21.2.112:0/1573320570 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9d44006830 0x7f9d44008d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:57.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.301 7f9d5cb5e700 1 -- 172.21.2.112:0/1573320570 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d44050a20 con 0x7f9d44003b10 2021-02-10T11:15:57.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.301 7f9d5cb5e700 1 --2- 172.21.2.112:0/1573320570 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9d44003b10 0x7f9d44006010 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7f9d40006a90 tx=0x7f9d40009a10).ready entity=mon.0 client_cookie=45d7e45eaa45ad0e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:57.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.301 7f9d577fe700 1 -- 172.21.2.112:0/1573320570 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9d400009b0 con 0x7f9d44003b10 2021-02-10T11:15:57.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.301 7f9d577fe700 1 -- 172.21.2.112:0/1573320570 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d4000c0c0 con 0x7f9d44003b10 2021-02-10T11:15:57.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.301 7f9d577fe700 1 -- 172.21.2.112:0/1573320570 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9d4000c520 con 0x7f9d44003b10 2021-02-10T11:15:57.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.301 7f9d5ed42700 1 -- 172.21.2.112:0/1573320570 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9d44003b10 msgr2=0x7f9d44006010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:57.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.301 7f9d5ed42700 1 --2- 172.21.2.112:0/1573320570 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9d44003b10 0x7f9d44006010 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7f9d40006a90 tx=0x7f9d40009a10).stop 2021-02-10T11:15:57.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.301 7f9d5d35f700 1 -- 172.21.2.112:0/1573320570 reap_dead start 2021-02-10T11:15:57.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.302 7f9d5ed42700 1 -- 172.21.2.112:0/1573320570 shutdown_connections 2021-02-10T11:15:57.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.302 7f9d5ed42700 1 -- 172.21.2.112:0/1573320570 >> 172.21.2.112:0/1573320570 conn(0x7f9d580ce690 msgr2=0x7f9d580e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:57.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.302 7f9d5ed42700 1 -- 172.21.2.112:0/1573320570 shutdown_connections 2021-02-10T11:15:57.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.302 7f9d5ed42700 1 -- 172.21.2.112:0/1573320570 wait complete. 2021-02-10T11:15:57.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.316 7f9d5ed42700 1 Processor -- start 2021-02-10T11:15:57.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.317 7f9d5ed42700 1 -- start start 2021-02-10T11:15:57.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.317 7f9d5ed42700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9d5808f500 0x7f9d5819f900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:57.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.317 7f9d5ed42700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9d5808fd90 0x7f9d58090190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:57.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.317 7f9d5ed42700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9d580907b0 0x7f9d58094370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:57.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.317 7f9d5ed42700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d5808f900 con 0x7f9d5808fd90 2021-02-10T11:15:57.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.317 7f9d5ed42700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d5819ff20 con 0x7f9d5808f500 2021-02-10T11:15:57.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.317 7f9d5ed42700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d581a0140 con 0x7f9d580907b0 2021-02-10T11:15:57.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.317 7f9d57fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9d5808fd90 0x7f9d58090190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:57.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.317 7f9d57fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9d5808fd90 0x7f9d58090190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:43788/0 (socket says 172.21.2.112:43788) 2021-02-10T11:15:57.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.317 7f9d57fff700 1 -- 172.21.2.112:0/394918120 learned_addr learned my addr 172.21.2.112:0/394918120 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:15:57.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.318 7f9d5d35f700 1 --2- 172.21.2.112:0/394918120 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9d580907b0 0x7f9d58094370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:57.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.318 7f9d57fff700 1 -- 172.21.2.112:0/394918120 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9d580907b0 msgr2=0x7f9d58094370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:57.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.318 7f9d57fff700 1 --2- 172.21.2.112:0/394918120 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9d580907b0 0x7f9d58094370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:57.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.318 7f9d57fff700 1 -- 172.21.2.112:0/394918120 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9d5808f500 msgr2=0x7f9d5819f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:57.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.318 7f9d57fff700 1 --2- 172.21.2.112:0/394918120 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9d5808f500 0x7f9d5819f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:57.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.318 7f9d57fff700 1 -- 172.21.2.112:0/394918120 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d58094990 con 0x7f9d5808fd90 2021-02-10T11:15:57.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.318 7f9d57fff700 1 --2- 172.21.2.112:0/394918120 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9d5808fd90 0x7f9d58090190 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 rx=0x7f9d480099c0 tx=0x7f9d4800bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:57.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.318 7f9d557fa700 1 -- 172.21.2.112:0/394918120 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9d48006240 con 0x7f9d5808fd90 2021-02-10T11:15:57.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.318 7f9d557fa700 1 -- 172.21.2.112:0/394918120 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d480064c0 con 0x7f9d5808fd90 2021-02-10T11:15:57.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.318 7f9d5ed42700 1 -- 172.21.2.112:0/394918120 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d58094990 con 0x7f9d5808fd90 2021-02-10T11:15:57.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.318 7f9d557fa700 1 -- 172.21.2.112:0/394918120 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9d48005850 con 0x7f9d5808fd90 2021-02-10T11:15:57.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.318 7f9d5ed42700 1 -- 172.21.2.112:0/394918120 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d58094f20 con 0x7f9d5808fd90 2021-02-10T11:15:57.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.319 7f9d557fa700 1 -- 172.21.2.112:0/394918120 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9d48005660 con 0x7f9d5808fd90 2021-02-10T11:15:57.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.319 7f9d557fa700 1 --2- 172.21.2.112:0/394918120 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9d3c0186c0 0x7f9d3c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:57.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.319 7f9d557fa700 1 -- 172.21.2.112:0/394918120 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f9d480327a0 con 0x7f9d5808fd90 2021-02-10T11:15:57.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.319 7f9d5cb5e700 1 --2- 172.21.2.112:0/394918120 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9d3c0186c0 0x7f9d3c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:57.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.320 7f9d5cb5e700 1 --2- 172.21.2.112:0/394918120 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9d3c0186c0 0x7f9d3c01abc0 crc :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:57.348 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.349 7f9d5ed42700 1 -- 172.21.2.112:0/394918120 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d580957b0 con 0x7f9d5808fd90 2021-02-10T11:15:57.350 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.352 7f9d557fa700 1 -- 172.21.2.112:0/394918120 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9d48006cb0 con 0x7f9d5808fd90 2021-02-10T11:15:57.486 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.488 7f9d5ed42700 1 -- 172.21.2.112:0/394918120 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs volume ls", "target": ["mgr", ""]}) v1 -- 0x7f9d5808f900 con 0x7f9d3c0186c0 2021-02-10T11:15:57.488 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.489 7f9d557fa700 1 -- 172.21.2.112:0/394918120 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+41 (crc 0 0 0) 0x7f9d40000940 con 0x7f9d3c0186c0 2021-02-10T11:15:57.488 INFO:teuthology.orchestra.run.gibba012.stdout:[ 2021-02-10T11:15:57.488 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:15:57.488 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "cephfs" 2021-02-10T11:15:57.488 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:15:57.489 INFO:teuthology.orchestra.run.gibba012.stdout:] 2021-02-10T11:15:57.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.490 7f9d5ed42700 1 -- 172.21.2.112:0/394918120 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9d3c0186c0 msgr2=0x7f9d3c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:57.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.490 7f9d5ed42700 1 --2- 172.21.2.112:0/394918120 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9d3c0186c0 0x7f9d3c01abc0 crc :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:57.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.490 7f9d5ed42700 1 -- 172.21.2.112:0/394918120 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9d5808fd90 msgr2=0x7f9d58090190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:57.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.490 7f9d5ed42700 1 --2- 172.21.2.112:0/394918120 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9d5808fd90 0x7f9d58090190 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 rx=0x7f9d480099c0 tx=0x7f9d4800bd40).stop 2021-02-10T11:15:57.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.490 7f9d5ed42700 1 -- 172.21.2.112:0/394918120 shutdown_connections 2021-02-10T11:15:57.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.490 7f9d5ed42700 1 --2- 172.21.2.112:0/394918120 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9d3c0186c0 0x7f9d3c01abc0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:57.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.490 7f9d5ed42700 1 --2- 172.21.2.112:0/394918120 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9d580907b0 0x7f9d58094370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:57.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.490 7f9d5ed42700 1 --2- 172.21.2.112:0/394918120 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9d5808fd90 0x7f9d58090190 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:57.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.490 7f9d5ed42700 1 --2- 172.21.2.112:0/394918120 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9d5808f500 0x7f9d5819f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:57.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.490 7f9d5ed42700 1 -- 172.21.2.112:0/394918120 >> 172.21.2.112:0/394918120 conn(0x7f9d580b5560 msgr2=0x7f9d580a6200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:57.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.490 7f9d5ed42700 1 -- 172.21.2.112:0/394918120 shutdown_connections 2021-02-10T11:15:57.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.491 7f9d5ed42700 1 -- 172.21.2.112:0/394918120 wait complete. 2021-02-10T11:15:57.498 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:15:57.928 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.912 7f556acfa700 1 Processor -- start 2021-02-10T11:15:57.928 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.912 7f556acfa700 1 -- start start 2021-02-10T11:15:57.928 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.912 7f556acfa700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f55641b0470 0x7f55641b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:57.928 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.912 7f556acfa700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f55640f20e0 0x7f55640f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:57.928 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.912 7f556acfa700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55640f0b50 con 0x7f55640ef610 2021-02-10T11:15:57.928 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.912 7f556acfa700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55640f0820 con 0x7f55641b0470 2021-02-10T11:15:57.928 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.912 7f556acfa700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55640f24e0 con 0x7f55640f20e0 2021-02-10T11:15:57.928 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.913 7f5569317700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f55640f20e0 0x7f55640f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:57.929 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.913 7f5568b16700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f55641b0470 0x7f55641b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:57.929 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.913 7f5569317700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f55640f20e0 0x7f55640f0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3301/0 says I am v2:172.21.2.112:43482/0 (socket says 172.21.2.112:43482) 2021-02-10T11:15:57.929 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.913 7f5569317700 1 -- 172.21.2.112:0/3394979720 learned_addr learned my addr 172.21.2.112:0/3394979720 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:15:57.929 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.914 7f55637fe700 1 -- 172.21.2.112:0/3394979720 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2907093699 0 0) 0x7f5554000fe0 con 0x7f55640ef610 2021-02-10T11:15:57.929 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.914 7f55637fe700 1 -- 172.21.2.112:0/3394979720 --> v1:172.21.2.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5550002fe0 con 0x7f55640ef610 2021-02-10T11:15:57.929 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.914 7f5568b16700 1 -- 172.21.2.112:0/3394979720 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f55640f20e0 msgr2=0x7f55640f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:15:57.929 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.914 7f5568b16700 1 --2- 172.21.2.112:0/3394979720 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f55640f20e0 0x7f55640f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:57.929 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.914 7f5568b16700 1 -- 172.21.2.112:0/3394979720 >> v1:172.21.2.120:6789/0 conn(0x7f55640ef610 legacy=0x7f55640efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:57.929 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.914 7f5568b16700 1 -- 172.21.2.112:0/3394979720 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55640e52e0 con 0x7f55641b0470 2021-02-10T11:15:57.929 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.914 7f5568b16700 1 --2- 172.21.2.112:0/3394979720 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f55641b0470 0x7f55641b0870 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f554c006650 tx=0x7f554c009a80).ready entity=mon.1 client_cookie=f1352b0d6e6a23f5 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:57.929 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.914 7f55637fe700 1 -- 172.21.2.112:0/3394979720 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f554c00c0c0 con 0x7f55641b0470 2021-02-10T11:15:57.929 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.914 7f55637fe700 1 -- 172.21.2.112:0/3394979720 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f554c00c340 con 0x7f55641b0470 2021-02-10T11:15:57.929 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.914 7f55637fe700 1 -- 172.21.2.112:0/3394979720 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f554c00c7a0 con 0x7f55641b0470 2021-02-10T11:15:57.929 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.914 7f556acfa700 1 -- 172.21.2.112:0/3394979720 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f55641b0470 msgr2=0x7f55641b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:57.930 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.914 7f556acfa700 1 --2- 172.21.2.112:0/3394979720 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f55641b0470 0x7f55641b0870 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f554c006650 tx=0x7f554c009a80).stop 2021-02-10T11:15:57.930 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.915 7f556acfa700 1 -- 172.21.2.112:0/3394979720 shutdown_connections 2021-02-10T11:15:57.930 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.915 7f556acfa700 1 --2- 172.21.2.112:0/3394979720 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f55640f20e0 0x7f55640f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:57.930 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.915 7f556acfa700 1 --2- 172.21.2.112:0/3394979720 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f55641b0470 0x7f55641b0870 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:57.930 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.915 7f556acfa700 1 -- 172.21.2.112:0/3394979720 >> 172.21.2.112:0/3394979720 conn(0x7f55640ce3e0 msgr2=0x7f55640e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:57.930 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.915 7f556acfa700 1 -- 172.21.2.112:0/3394979720 shutdown_connections 2021-02-10T11:15:57.930 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.915 7f556acfa700 1 -- 172.21.2.112:0/3394979720 wait complete. 2021-02-10T11:15:57.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.929 7f556acfa700 1 Processor -- start 2021-02-10T11:15:57.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.929 7f556acfa700 1 -- start start 2021-02-10T11:15:57.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.930 7f556acfa700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f55640a0420 0x7f55640a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:57.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.930 7f556acfa700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5564189d00 0x7f556418a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:57.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.930 7f556acfa700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f556418a720 0x7f55640b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:57.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.930 7f556acfa700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5564189800 con 0x7f5564189d00 2021-02-10T11:15:57.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.930 7f556acfa700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f556418ab20 con 0x7f556418a720 2021-02-10T11:15:57.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.930 7f556acfa700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55640a1140 con 0x7f55640a0420 2021-02-10T11:15:57.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.930 7f5563fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5564189d00 0x7f556418a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:57.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.930 7f5568b16700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f55640a0420 0x7f55640a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:57.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.930 7f5568b16700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f55640a0420 0x7f55640a0820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3301/0 says I am v2:172.21.2.112:43486/0 (socket says 172.21.2.112:43486) 2021-02-10T11:15:57.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.930 7f5563fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5564189d00 0x7f556418a100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:43802/0 (socket says 172.21.2.112:43802) 2021-02-10T11:15:57.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.930 7f5568b16700 1 -- 172.21.2.112:0/3479246650 learned_addr learned my addr 172.21.2.112:0/3479246650 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:15:57.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.930 7f5569317700 1 --2- 172.21.2.112:0/3479246650 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f556418a720 0x7f55640b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:57.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.930 7f5569317700 1 -- 172.21.2.112:0/3479246650 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f55640a0420 msgr2=0x7f55640a0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:57.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.930 7f5569317700 1 --2- 172.21.2.112:0/3479246650 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f55640a0420 0x7f55640a0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:57.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.930 7f5569317700 1 -- 172.21.2.112:0/3479246650 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5564189d00 msgr2=0x7f556418a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:57.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.930 7f5569317700 1 --2- 172.21.2.112:0/3479246650 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5564189d00 0x7f556418a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:57.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.930 7f5569317700 1 -- 172.21.2.112:0/3479246650 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55640b8200 con 0x7f556418a720 2021-02-10T11:15:57.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.931 7f5569317700 1 --2- 172.21.2.112:0/3479246650 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f556418a720 0x7f55640b7be0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f555c006ed0 tx=0x7f555c00b910).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:57.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.931 7f55617fa700 1 -- 172.21.2.112:0/3479246650 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f555c006b30 con 0x7f556418a720 2021-02-10T11:15:57.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.931 7f55617fa700 1 -- 172.21.2.112:0/3479246650 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f555c00bd00 con 0x7f556418a720 2021-02-10T11:15:57.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.931 7f55617fa700 1 -- 172.21.2.112:0/3479246650 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f555c002220 con 0x7f556418a720 2021-02-10T11:15:57.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.931 7f556acfa700 1 -- 172.21.2.112:0/3479246650 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55640b8200 con 0x7f556418a720 2021-02-10T11:15:57.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.931 7f556acfa700 1 -- 172.21.2.112:0/3479246650 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55640b8200 con 0x7f556418a720 2021-02-10T11:15:57.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.931 7f55617fa700 1 -- 172.21.2.112:0/3479246650 <== mon.1 v2:172.21.2.120:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f555c00a000 con 0x7f556418a720 2021-02-10T11:15:57.935 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.931 7f55617fa700 1 --2- 172.21.2.112:0/3479246650 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5548018730 0x7f554801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:57.935 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.931 7f55617fa700 1 -- 172.21.2.112:0/3479246650 <== mon.1 v2:172.21.2.120:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f555c0326d0 con 0x7f556418a720 2021-02-10T11:15:57.935 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.932 7f5568b16700 1 --2- 172.21.2.112:0/3479246650 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5548018730 0x7f554801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:57.935 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.932 7f5568b16700 1 --2- 172.21.2.112:0/3479246650 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5548018730 0x7f554801ac10 crc :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:57.961 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.962 7f556acfa700 1 -- 172.21.2.112:0/3479246650 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f556418ab20 con 0x7f556418a720 2021-02-10T11:15:57.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:57.965 7f55617fa700 1 -- 172.21.2.112:0/3479246650 <== mon.1 v2:172.21.2.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f555c006520 con 0x7f556418a720 2021-02-10T11:15:58.114 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:58.115 7f556acfa700 1 -- 172.21.2.112:0/3479246650 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f556418ab20 con 0x7f556418a720 2021-02-10T11:15:58.115 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:58.116 7f55617fa700 1 -- 172.21.2.112:0/3479246650 <== mon.1 v2:172.21.2.120:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v2)=0 v2) v1 ==== 122+0+0 (secure 0 0 0) 0x7f555c006520 con 0x7f556418a720 2021-02-10T11:15:58.119 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:58.120 7f556acfa700 1 -- 172.21.2.112:0/3479246650 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5548018730 msgr2=0x7f554801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:58.119 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:58.120 7f556acfa700 1 --2- 172.21.2.112:0/3479246650 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5548018730 0x7f554801ac10 crc :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:58.119 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:58.120 7f556acfa700 1 -- 172.21.2.112:0/3479246650 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f556418a720 msgr2=0x7f55640b7be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:58.119 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:58.120 7f556acfa700 1 --2- 172.21.2.112:0/3479246650 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f556418a720 0x7f55640b7be0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f555c006ed0 tx=0x7f555c00b910).stop 2021-02-10T11:15:58.120 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:58.121 7f556acfa700 1 -- 172.21.2.112:0/3479246650 shutdown_connections 2021-02-10T11:15:58.120 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:58.121 7f556acfa700 1 --2- 172.21.2.112:0/3479246650 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5548018730 0x7f554801ac10 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:58.120 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:58.121 7f556acfa700 1 --2- 172.21.2.112:0/3479246650 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f556418a720 0x7f55640b7be0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:58.120 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:58.121 7f556acfa700 1 --2- 172.21.2.112:0/3479246650 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5564189d00 0x7f556418a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:58.120 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:58.121 7f556acfa700 1 --2- 172.21.2.112:0/3479246650 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f55640a0420 0x7f55640a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:58.120 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:58.121 7f556acfa700 1 -- 172.21.2.112:0/3479246650 >> 172.21.2.112:0/3479246650 conn(0x7f5564094e10 msgr2=0x7f55640a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:58.120 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:58.121 7f556acfa700 1 -- 172.21.2.112:0/3479246650 shutdown_connections 2021-02-10T11:15:58.120 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:58.121 7f556acfa700 1 -- 172.21.2.112:0/3479246650 wait complete. 2021-02-10T11:15:58.127 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000348568 2021-02-10T11:15:58.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:58.541 7fcbf0445700 1 Processor -- start 2021-02-10T11:15:58.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:58.541 7fcbf0445700 1 -- start start 2021-02-10T11:15:58.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:58.542 7fcbf0445700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fcbe80f05f0 0x7fcbe80f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:58.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:58.542 7fcbf0445700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbe81b0c70 con 0x7fcbe81b0470 2021-02-10T11:15:58.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:58.542 7fcbf0445700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbe80f0b50 con 0x7fcbe81b0870 2021-02-10T11:15:58.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:58.542 7fcbf0445700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbe80e5ea0 con 0x7fcbe80f05f0 2021-02-10T11:15:58.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:58.542 7fcbeea62700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fcbe80f05f0 0x7fcbe80f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:58.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:58.542 7fcbeea62700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fcbe80f05f0 0x7fcbe80f20e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:43814/0 (socket says 172.21.2.112:43814) 2021-02-10T11:15:58.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:58.542 7fcbeea62700 1 -- 172.21.2.112:0/3048503603 learned_addr learned my addr 172.21.2.112:0/3048503603 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:15:58.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:58.542 7fcbee261700 1 --1- 172.21.2.112:0/3048503603 >> v1:172.21.2.112:6789/0 conn(0x7fcbe81b0470 0x7fcbe80ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:43272/0 (socket says 172.21.2.112:43272) 2021-02-10T11:15:58.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:58.542 7fcbeea62700 1 -- 172.21.2.112:0/3048503603 >> v1:172.21.2.120:6789/0 conn(0x7fcbe81b0870 legacy=0x7fcbe80efe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:15:58.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:58.542 7fcbeea62700 1 -- 172.21.2.112:0/3048503603 >> v1:172.21.2.112:6789/0 conn(0x7fcbe81b0470 legacy=0x7fcbe80ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:58.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:58.542 7fcbeea62700 1 -- 172.21.2.112:0/3048503603 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbe80e52e0 con 0x7fcbe80f05f0 2021-02-10T11:15:58.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:58.543 7fcbeea62700 1 --2- 172.21.2.112:0/3048503603 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcbe80f05f0 0x7fcbe80f20e0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 rx=0x7fcbe0006f00 tx=0x7fcbe0009f10).ready entity=mon.0 client_cookie=d9d38300b330d137 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:58.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:58.543 7fcbed25f700 1 -- 172.21.2.112:0/3048503603 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcbe000b3b0 con 0x7fcbe80f05f0 2021-02-10T11:15:58.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:58.543 7fcbed25f700 1 -- 172.21.2.112:0/3048503603 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbe000b630 con 0x7fcbe80f05f0 2021-02-10T11:15:58.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:58.543 7fcbed25f700 1 -- 172.21.2.112:0/3048503603 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcbe000bb10 con 0x7fcbe80f05f0 2021-02-10T11:15:58.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:58.543 7fcbf0445700 1 -- 172.21.2.112:0/3048503603 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcbe80f05f0 msgr2=0x7fcbe80f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:58.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:58.543 7fcbf0445700 1 --2- 172.21.2.112:0/3048503603 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcbe80f05f0 0x7fcbe80f20e0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 rx=0x7fcbe0006f00 tx=0x7fcbe0009f10).stop 2021-02-10T11:15:58.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:58.543 7fcbf0445700 1 -- 172.21.2.112:0/3048503603 shutdown_connections 2021-02-10T11:15:58.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:58.543 7fcbf0445700 1 --2- 172.21.2.112:0/3048503603 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcbe80f05f0 0x7fcbe80f20e0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:58.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:58.543 7fcbf0445700 1 -- 172.21.2.112:0/3048503603 >> 172.21.2.112:0/3048503603 conn(0x7fcbe80ce3e0 msgr2=0x7fcbe80e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:58.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:58.544 7fcbf0445700 1 -- 172.21.2.112:0/3048503603 shutdown_connections 2021-02-10T11:15:58.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:58.544 7fcbf0445700 1 -- 172.21.2.112:0/3048503603 wait complete. 2021-02-10T11:15:58.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:58.558 7fcbf0445700 1 Processor -- start 2021-02-10T11:15:58.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:58.558 7fcbf0445700 1 -- start start 2021-02-10T11:15:58.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:58.559 7fcbf0445700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcbe8082170 0x7fcbe80894a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:58.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:58.559 7fcbf0445700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcbe81b01a0 0x7fcbe81b05a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:58.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:58.559 7fcbf0445700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcbe81b1460 0x7fcbe81b1860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:58.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:58.559 7fcbf0445700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbe81bf2a0 con 0x7fcbe81b01a0 2021-02-10T11:15:58.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:58.559 7fcbf0445700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbe81b0ee0 con 0x7fcbe81b1460 2021-02-10T11:15:58.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:58.559 7fcbf0445700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbe81b60c0 con 0x7fcbe8082170 2021-02-10T11:15:58.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:58.559 7fcbeda60700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcbe81b01a0 0x7fcbe81b05a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:58.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:58.559 7fcbeda60700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcbe81b01a0 0x7fcbe81b05a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:43816/0 (socket says 172.21.2.112:43816) 2021-02-10T11:15:58.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:58.559 7fcbeda60700 1 -- 172.21.2.112:0/4172104621 learned_addr learned my addr 172.21.2.112:0/4172104621 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:15:58.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:58.559 7fcbeda60700 1 -- 172.21.2.112:0/4172104621 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcbe8082170 msgr2=0x7fcbe80894a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:58.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:58.559 7fcbeda60700 1 --2- 172.21.2.112:0/4172104621 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcbe8082170 0x7fcbe80894a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:58.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:58.559 7fcbeda60700 1 -- 172.21.2.112:0/4172104621 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcbe81b1460 msgr2=0x7fcbe81b1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:58.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:58.559 7fcbeda60700 1 --2- 172.21.2.112:0/4172104621 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcbe81b1460 0x7fcbe81b1860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:58.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:58.559 7fcbeda60700 1 -- 172.21.2.112:0/4172104621 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbe81b0bc0 con 0x7fcbe81b01a0 2021-02-10T11:15:58.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:58.559 7fcbeda60700 1 --2- 172.21.2.112:0/4172104621 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcbe81b01a0 0x7fcbe81b05a0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7fcbd800b530 tx=0x7fcbd800b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:58.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:58.560 7fcbdeffd700 1 -- 172.21.2.112:0/4172104621 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcbd800bcc0 con 0x7fcbe81b01a0 2021-02-10T11:15:58.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:58.560 7fcbdeffd700 1 -- 172.21.2.112:0/4172104621 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbd8006020 con 0x7fcbe81b01a0 2021-02-10T11:15:58.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:58.560 7fcbf0445700 1 -- 172.21.2.112:0/4172104621 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbe81b0bc0 con 0x7fcbe81b01a0 2021-02-10T11:15:58.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:58.560 7fcbf0445700 1 -- 172.21.2.112:0/4172104621 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbe81b0bc0 con 0x7fcbe81b01a0 2021-02-10T11:15:58.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:58.560 7fcbdeffd700 1 -- 172.21.2.112:0/4172104621 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcbd80054a0 con 0x7fcbe81b01a0 2021-02-10T11:15:58.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:58.561 7fcbdeffd700 1 -- 172.21.2.112:0/4172104621 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fcbd8006200 con 0x7fcbe81b01a0 2021-02-10T11:15:58.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:58.561 7fcbdeffd700 1 --2- 172.21.2.112:0/4172104621 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcbd0018730 0x7fcbd001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:58.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:58.561 7fcbdeffd700 1 -- 172.21.2.112:0/4172104621 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fcbd8032810 con 0x7fcbe81b01a0 2021-02-10T11:15:58.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:58.561 7fcbee261700 1 --2- 172.21.2.112:0/4172104621 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcbd0018730 0x7fcbd001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:58.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:58.562 7fcbee261700 1 --2- 172.21.2.112:0/4172104621 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcbd0018730 0x7fcbd001ac10 crc :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:58.591 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:58.592 7fcbf0445700 1 -- 172.21.2.112:0/4172104621 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbe81bf2a0 con 0x7fcbe81b01a0 2021-02-10T11:15:58.593 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:58.595 7fcbdeffd700 1 -- 172.21.2.112:0/4172104621 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fcbd8006230 con 0x7fcbe81b01a0 2021-02-10T11:15:58.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:58.720 7fcbf0445700 1 -- 172.21.2.112:0/4172104621 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348568", "target": ["mgr", ""]}) v1 -- 0x7fcbe81b0ee0 con 0x7fcbd0018730 2021-02-10T11:15:58.722 INFO:tasks.ceph.mgr.x.gibba020.stderr:2021-02-10 11:15:58.725 7f6382eee700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:15:58.722 INFO:tasks.ceph.mgr.x.gibba020.stderr:2021-02-10 11:15:58.725 7f6382eee700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:15:58.723 INFO:tasks.ceph.mgr.x.gibba020.stderr:2021-02-10 11:15:58.725 7f6382eee700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:15:58.723 INFO:tasks.ceph.mgr.x.gibba020.stderr:2021-02-10 11:15:58.725 7f6382eee700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:15:58.723 INFO:tasks.ceph.mgr.x.gibba020.stderr:2021-02-10 11:15:58.725 7f6382eee700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:15:58.754 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:58.756 7fcbdeffd700 1 -- 172.21.2.112:0/4172104621 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fcbe4000ea0 con 0x7fcbd0018730 2021-02-10T11:15:58.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:58.786 7fcbf0445700 1 -- 172.21.2.112:0/4172104621 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcbd0018730 msgr2=0x7fcbd001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:58.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:58.786 7fcbf0445700 1 --2- 172.21.2.112:0/4172104621 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcbd0018730 0x7fcbd001ac10 crc :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:58.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:58.786 7fcbf0445700 1 -- 172.21.2.112:0/4172104621 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcbe81b01a0 msgr2=0x7fcbe81b05a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:58.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:58.786 7fcbf0445700 1 --2- 172.21.2.112:0/4172104621 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcbe81b01a0 0x7fcbe81b05a0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7fcbd800b530 tx=0x7fcbd800b8d0).stop 2021-02-10T11:15:58.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:58.786 7fcbf0445700 1 -- 172.21.2.112:0/4172104621 shutdown_connections 2021-02-10T11:15:58.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:58.786 7fcbf0445700 1 --2- 172.21.2.112:0/4172104621 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcbd0018730 0x7fcbd001ac10 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:58.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:58.786 7fcbf0445700 1 --2- 172.21.2.112:0/4172104621 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcbe81b1460 0x7fcbe81b1860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:58.786 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:58.786 7fcbf0445700 1 --2- 172.21.2.112:0/4172104621 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcbe81b01a0 0x7fcbe81b05a0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:58.786 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:58.786 7fcbf0445700 1 --2- 172.21.2.112:0/4172104621 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcbe8082170 0x7fcbe80894a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:58.786 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:58.786 7fcbf0445700 1 -- 172.21.2.112:0/4172104621 >> 172.21.2.112:0/4172104621 conn(0x7fcbe8088d10 msgr2=0x7fcbe80afb80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:58.786 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:58.786 7fcbf0445700 1 -- 172.21.2.112:0/4172104621 shutdown_connections 2021-02-10T11:15:58.786 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:58.786 7fcbf0445700 1 -- 172.21.2.112:0/4172104621 wait complete. 2021-02-10T11:15:58.793 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000348568 2021-02-10T11:15:59.220 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:59.205 7fc000b98700 1 Processor -- start 2021-02-10T11:15:59.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:59.206 7fc000b98700 1 -- start start 2021-02-10T11:15:59.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:59.206 7fc000b98700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fbffc1b0470 0x7fbffc1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:59.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:59.206 7fc000b98700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fbffc0ef610 0x7fbffc0efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:59.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:59.206 7fc000b98700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbffc0f0030 0x7fbffc0f0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:59.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:59.206 7fc000b98700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbffc0e5ea0 con 0x7fbffc1b0470 2021-02-10T11:15:59.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:59.206 7fc000b98700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbffc0f0b50 con 0x7fbffc0ef610 2021-02-10T11:15:59.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:59.206 7fc000b98700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbffc0e52e0 con 0x7fbffc0f0030 2021-02-10T11:15:59.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:59.206 7fbffa85e700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fbffc1b0470 0x7fbffc1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:59.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:59.206 7fbffa85e700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fbffc1b0470 0x7fbffc1b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:43824/0 (socket says 172.21.2.112:43824) 2021-02-10T11:15:59.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:59.206 7fbffa85e700 1 -- 172.21.2.112:0/1485189070 learned_addr learned my addr 172.21.2.112:0/1485189070 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:15:59.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:59.206 7fbffa05d700 1 --2- 172.21.2.112:0/1485189070 >> v2:172.21.2.120:3300/0 conn(0x7fbffc0ef610 0x7fbffc0efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:59.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:59.206 7fbffb05f700 1 --2- 172.21.2.112:0/1485189070 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbffc0f0030 0x7fbffc0f0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:59.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:59.206 7fbffa85e700 1 -- 172.21.2.112:0/1485189070 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbffc0f0030 msgr2=0x7fbffc0f0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:15:59.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:59.207 7fbffa85e700 1 --2- 172.21.2.112:0/1485189070 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbffc0f0030 0x7fbffc0f0430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:59.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:59.207 7fbffa85e700 1 -- 172.21.2.112:0/1485189070 >> v2:172.21.2.120:3300/0 conn(0x7fbffc0ef610 msgr2=0x7fbffc0efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:15:59.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:59.207 7fbffa85e700 1 --2- 172.21.2.112:0/1485189070 >> v2:172.21.2.120:3300/0 conn(0x7fbffc0ef610 0x7fbffc0efa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:59.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:59.207 7fbffa85e700 1 -- 172.21.2.112:0/1485189070 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbffc0f10d0 con 0x7fbffc1b0470 2021-02-10T11:15:59.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:59.207 7fbffb05f700 1 --2- 172.21.2.112:0/1485189070 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbffc0f0030 0x7fbffc0f0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:15:59.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:59.207 7fbffa85e700 1 --2- 172.21.2.112:0/1485189070 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbffc1b0470 0x7fbffc1b0870 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 rx=0x7fbfec006700 tx=0x7fbfec009b80).ready entity=mon.0 client_cookie=27c8a2722b1ff70f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:59.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:59.207 7fbff985c700 1 -- 172.21.2.112:0/1485189070 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbfec006340 con 0x7fbffc1b0470 2021-02-10T11:15:59.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:59.207 7fbff985c700 1 -- 172.21.2.112:0/1485189070 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfec00c0c0 con 0x7fbffc1b0470 2021-02-10T11:15:59.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:59.207 7fbff985c700 1 -- 172.21.2.112:0/1485189070 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbfec00c510 con 0x7fbffc1b0470 2021-02-10T11:15:59.223 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:59.207 7fc000b98700 1 -- 172.21.2.112:0/1485189070 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbffc1b0470 msgr2=0x7fbffc1b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:59.223 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:59.207 7fc000b98700 1 --2- 172.21.2.112:0/1485189070 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbffc1b0470 0x7fbffc1b0870 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 rx=0x7fbfec006700 tx=0x7fbfec009b80).stop 2021-02-10T11:15:59.223 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:59.208 7fc000b98700 1 -- 172.21.2.112:0/1485189070 shutdown_connections 2021-02-10T11:15:59.223 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:59.208 7fc000b98700 1 --2- 172.21.2.112:0/1485189070 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbffc0f0030 0x7fbffc0f0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:59.223 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:59.208 7fc000b98700 1 --2- 172.21.2.112:0/1485189070 >> v2:172.21.2.120:3300/0 conn(0x7fbffc0ef610 0x7fbffc0efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:59.223 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:59.208 7fc000b98700 1 --2- 172.21.2.112:0/1485189070 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbffc1b0470 0x7fbffc1b0870 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:59.223 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:59.208 7fc000b98700 1 -- 172.21.2.112:0/1485189070 >> 172.21.2.112:0/1485189070 conn(0x7fbffc0ce3e0 msgr2=0x7fbffc0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:59.223 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:59.208 7fc000b98700 1 -- 172.21.2.112:0/1485189070 shutdown_connections 2021-02-10T11:15:59.223 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:59.208 7fc000b98700 1 -- 172.21.2.112:0/1485189070 wait complete. 2021-02-10T11:15:59.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:59.222 7fc000b98700 1 Processor -- start 2021-02-10T11:15:59.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:59.222 7fc000b98700 1 -- start start 2021-02-10T11:15:59.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:59.223 7fc000b98700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbffc0a0910 0x7fbffc105d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:59.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:59.223 7fc000b98700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbffc1a7a10 0x7fbffc1a7e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:59.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:59.223 7fc000b98700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fbffc093770 0x7fbffc093b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:59.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:59.223 7fc000b98700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbffc106390 con 0x7fbffc0a0910 2021-02-10T11:15:59.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:59.223 7fc000b98700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbffc094190 con 0x7fbffc093770 2021-02-10T11:15:59.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:59.223 7fc000b98700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbffc0943b0 con 0x7fbffc1a7a10 2021-02-10T11:15:59.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:59.223 7fbffa85e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbffc0a0910 0x7fbffc105d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:59.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:59.223 7fbffa85e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbffc0a0910 0x7fbffc105d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:43832/0 (socket says 172.21.2.112:43832) 2021-02-10T11:15:59.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:59.223 7fbffa85e700 1 -- 172.21.2.112:0/3153911529 learned_addr learned my addr 172.21.2.112:0/3153911529 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:15:59.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:59.223 7fbffb05f700 1 --2- 172.21.2.112:0/3153911529 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fbffc093770 0x7fbffc093b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:59.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:59.223 7fbffa05d700 1 --2- 172.21.2.112:0/3153911529 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbffc1a7a10 0x7fbffc1a7e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:59.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:59.223 7fbffa85e700 1 -- 172.21.2.112:0/3153911529 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbffc1a7a10 msgr2=0x7fbffc1a7e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:59.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:59.223 7fbffa85e700 1 --2- 172.21.2.112:0/3153911529 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbffc1a7a10 0x7fbffc1a7e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:59.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:59.223 7fbffa85e700 1 -- 172.21.2.112:0/3153911529 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fbffc093770 msgr2=0x7fbffc093b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:59.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:59.223 7fbffa85e700 1 --2- 172.21.2.112:0/3153911529 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fbffc093770 0x7fbffc093b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:59.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:59.223 7fbffa85e700 1 -- 172.21.2.112:0/3153911529 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbffc106c30 con 0x7fbffc0a0910 2021-02-10T11:15:59.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:59.223 7fbffa85e700 1 --2- 172.21.2.112:0/3153911529 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbffc0a0910 0x7fbffc105d70 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7fbfec00b580 tx=0x7fbfec00b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:59.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:59.224 7fbff37fe700 1 -- 172.21.2.112:0/3153911529 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbfec002030 con 0x7fbffc0a0910 2021-02-10T11:15:59.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:59.224 7fbff37fe700 1 -- 172.21.2.112:0/3153911529 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfec002370 con 0x7fbffc0a0910 2021-02-10T11:15:59.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:59.224 7fbff37fe700 1 -- 172.21.2.112:0/3153911529 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbfec002890 con 0x7fbffc0a0910 2021-02-10T11:15:59.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:59.224 7fc000b98700 1 -- 172.21.2.112:0/3153911529 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbffc106c30 con 0x7fbffc0a0910 2021-02-10T11:15:59.227 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:59.224 7fc000b98700 1 -- 172.21.2.112:0/3153911529 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbffc1071c0 con 0x7fbffc0a0910 2021-02-10T11:15:59.227 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:59.224 7fbff37fe700 1 -- 172.21.2.112:0/3153911529 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fbfec000c90 con 0x7fbffc0a0910 2021-02-10T11:15:59.227 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:59.224 7fbff37fe700 1 --2- 172.21.2.112:0/3153911529 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fbfe00186c0 0x7fbfe001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:59.227 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:59.224 7fbff37fe700 1 -- 172.21.2.112:0/3153911529 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fbfec032420 con 0x7fbffc0a0910 2021-02-10T11:15:59.227 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:59.225 7fbffa05d700 1 --2- 172.21.2.112:0/3153911529 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fbfe00186c0 0x7fbfe001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:59.227 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:59.225 7fbffa05d700 1 --2- 172.21.2.112:0/3153911529 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fbfe00186c0 0x7fbfe001abc0 crc :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:59.254 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:59.256 7fc000b98700 1 -- 172.21.2.112:0/3153911529 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbffc105b40 con 0x7fbffc0a0910 2021-02-10T11:15:59.257 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:59.258 7fbff37fe700 1 -- 172.21.2.112:0/3153911529 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fbfec000940 con 0x7fbffc0a0910 2021-02-10T11:15:59.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:59.389 7fc000b98700 1 -- 172.21.2.112:0/3153911529 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348568", "target": ["mgr", ""]}) v1 -- 0x7fbffc1071c0 con 0x7fbfe00186c0 2021-02-10T11:15:59.391 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:59.393 7fbff37fe700 1 -- 172.21.2.112:0/3153911529 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7fbfe4000940 con 0x7fbfe00186c0 2021-02-10T11:15:59.395 INFO:teuthology.orchestra.run.gibba012.stdout:/volumes/_nogroup/subvolume_0000000000348568/db929eab-f570-421f-9690-fdd85732f568 2021-02-10T11:15:59.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:59.397 7fc000b98700 1 -- 172.21.2.112:0/3153911529 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fbfe00186c0 msgr2=0x7fbfe001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:59.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:59.397 7fc000b98700 1 --2- 172.21.2.112:0/3153911529 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fbfe00186c0 0x7fbfe001abc0 crc :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:59.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:59.397 7fc000b98700 1 -- 172.21.2.112:0/3153911529 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbffc0a0910 msgr2=0x7fbffc105d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:59.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:59.397 7fc000b98700 1 --2- 172.21.2.112:0/3153911529 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbffc0a0910 0x7fbffc105d70 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7fbfec00b580 tx=0x7fbfec00b920).stop 2021-02-10T11:15:59.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:59.398 7fc000b98700 1 -- 172.21.2.112:0/3153911529 shutdown_connections 2021-02-10T11:15:59.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:59.398 7fc000b98700 1 --2- 172.21.2.112:0/3153911529 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fbfe00186c0 0x7fbfe001abc0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:59.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:59.398 7fc000b98700 1 --2- 172.21.2.112:0/3153911529 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fbffc093770 0x7fbffc093b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:59.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:59.398 7fc000b98700 1 --2- 172.21.2.112:0/3153911529 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbffc1a7a10 0x7fbffc1a7e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:59.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:59.398 7fc000b98700 1 --2- 172.21.2.112:0/3153911529 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbffc0a0910 0x7fbffc105d70 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:59.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:59.398 7fc000b98700 1 -- 172.21.2.112:0/3153911529 >> 172.21.2.112:0/3153911529 conn(0x7fbffc095ef0 msgr2=0x7fbffc094e40 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:59.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:59.398 7fc000b98700 1 -- 172.21.2.112:0/3153911529 shutdown_connections 2021-02-10T11:15:59.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:15:59.398 7fc000b98700 1 -- 172.21.2.112:0/3153911529 wait complete. 2021-02-10T11:15:59.405 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000348568 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000348568/db929eab-f570-421f-9690-fdd85732f568 2021-02-10T11:15:59.405 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:15:59.405 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:15:59.405 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:15:59.740 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:15:59.740 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:15:59.741 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348568/db929eab-f570-421f-9690-fdd85732f568/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:00.123 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:16:00.123 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:16:00.123 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0268286 s, 39.1 MB/s 2021-02-10T11:16:00.124 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:00.124 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:00.124 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:00.459 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:16:00.460 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:00.460 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348568/db929eab-f570-421f-9690-fdd85732f568/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:00.831 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:16:00.831 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:16:00.831 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0243483 s, 43.1 MB/s 2021-02-10T11:16:00.832 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:00.832 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:00.833 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:01.166 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:16:01.167 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:01.167 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348568/db929eab-f570-421f-9690-fdd85732f568/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:01.541 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:16:01.541 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:16:01.541 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0235973 s, 44.4 MB/s 2021-02-10T11:16:01.542 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:01.542 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:01.542 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:01.876 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:16:01.876 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:01.877 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348568/db929eab-f570-421f-9690-fdd85732f568/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:02.249 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:16:02.249 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:16:02.249 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0228559 s, 45.9 MB/s 2021-02-10T11:16:02.250 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:02.250 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:02.251 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:02.523 DEBUG:teuthology.orchestra.run.gibba012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:16:02.527 DEBUG:teuthology.orchestra.run.gibba020:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:16:02.586 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:16:02.586 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:02.586 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348568/db929eab-f570-421f-9690-fdd85732f568/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:02.896 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:16:02.896 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:16:02.896 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0218278 s, 48.0 MB/s 2021-02-10T11:16:02.897 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:02.897 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:02.897 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:03.231 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:16:03.231 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:03.232 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348568/db929eab-f570-421f-9690-fdd85732f568/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:03.608 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:16:03.608 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:16:03.608 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0250445 s, 41.9 MB/s 2021-02-10T11:16:03.609 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:03.609 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:03.609 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:03.944 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:16:03.944 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:03.945 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348568/db929eab-f570-421f-9690-fdd85732f568/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:04.317 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:16:04.317 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:16:04.317 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0246767 s, 42.5 MB/s 2021-02-10T11:16:04.318 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:04.319 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:04.319 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:04.653 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:16:04.654 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:04.654 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348568/db929eab-f570-421f-9690-fdd85732f568/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:05.030 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:16:05.030 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:16:05.030 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0259879 s, 40.3 MB/s 2021-02-10T11:16:05.031 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:05.032 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:05.032 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:05.365 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:16:05.366 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:05.366 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348568/db929eab-f570-421f-9690-fdd85732f568/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:05.738 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:16:05.738 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:16:05.739 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0224703 s, 46.7 MB/s 2021-02-10T11:16:05.739 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:05.740 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:05.740 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:06.073 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:16:06.074 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:06.074 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348568/db929eab-f570-421f-9690-fdd85732f568/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:06.450 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:16:06.450 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:16:06.450 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0261347 s, 40.1 MB/s 2021-02-10T11:16:06.451 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000348569 2021-02-10T11:16:06.881 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:06.865 7f262431f700 1 Processor -- start 2021-02-10T11:16:06.881 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:06.865 7f262431f700 1 -- start start 2021-02-10T11:16:06.881 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:06.866 7f262431f700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f261c1b0870 0x7f261c0f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:06.881 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:06.866 7f262431f700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f261c0efe00 0x7f261c0f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:06.881 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:06.866 7f262431f700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f261c1b0c70 con 0x7f261c1b0470 2021-02-10T11:16:06.882 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:06.866 7f262431f700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f261c0f0b50 con 0x7f261c1b0870 2021-02-10T11:16:06.882 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:06.866 7f262431f700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f261c0f0820 con 0x7f261c0efe00 2021-02-10T11:16:06.882 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:06.866 7f262193a700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f261c1b0870 0x7f261c0f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:06.882 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:06.866 7f262193a700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f261c1b0870 0x7f261c0f20e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:43848/0 (socket says 172.21.2.112:43848) 2021-02-10T11:16:06.882 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:06.866 7f262193a700 1 -- 172.21.2.112:0/668029365 learned_addr learned my addr 172.21.2.112:0/668029365 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:16:06.882 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:06.867 7f262293c700 1 --2- 172.21.2.112:0/668029365 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f261c0efe00 0x7f261c0f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:06.882 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:06.867 7f262193a700 1 -- 172.21.2.112:0/668029365 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f261c0efe00 msgr2=0x7f261c0f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:16:06.882 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:06.867 7f262193a700 1 --2- 172.21.2.112:0/668029365 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f261c0efe00 0x7f261c0f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:06.882 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:06.867 7f262193a700 1 -- 172.21.2.112:0/668029365 >> v1:172.21.2.120:6789/0 conn(0x7f261c1b0470 legacy=0x7f261c0ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:06.882 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:06.867 7f262193a700 1 -- 172.21.2.112:0/668029365 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f261c0e52e0 con 0x7f261c1b0870 2021-02-10T11:16:06.882 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:06.867 7f262193a700 1 --2- 172.21.2.112:0/668029365 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f261c1b0870 0x7f261c0f20e0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 rx=0x7f2618006700 tx=0x7f2618009b80).ready entity=mon.0 client_cookie=a566130d0b43df9b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:16:06.882 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:06.867 7f2621139700 1 -- 172.21.2.112:0/668029365 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f261800c0c0 con 0x7f261c1b0870 2021-02-10T11:16:06.883 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:06.867 7f2621139700 1 -- 172.21.2.112:0/668029365 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f261800c340 con 0x7f261c1b0870 2021-02-10T11:16:06.883 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:06.867 7f2621139700 1 -- 172.21.2.112:0/668029365 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f261800c860 con 0x7f261c1b0870 2021-02-10T11:16:06.883 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:06.867 7f262431f700 1 -- 172.21.2.112:0/668029365 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f261c1b0870 msgr2=0x7f261c0f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:06.883 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:06.867 7f262431f700 1 --2- 172.21.2.112:0/668029365 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f261c1b0870 0x7f261c0f20e0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 rx=0x7f2618006700 tx=0x7f2618009b80).stop 2021-02-10T11:16:06.883 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:06.868 7f262431f700 1 -- 172.21.2.112:0/668029365 shutdown_connections 2021-02-10T11:16:06.883 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:06.868 7f262431f700 1 --2- 172.21.2.112:0/668029365 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f261c1b0870 0x7f261c0f20e0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:06.883 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:06.868 7f262431f700 1 --2- 172.21.2.112:0/668029365 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f261c0efe00 0x7f261c0f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:06.883 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:06.868 7f262431f700 1 -- 172.21.2.112:0/668029365 >> 172.21.2.112:0/668029365 conn(0x7f261c0ce3e0 msgr2=0x7f261c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:16:06.883 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:06.868 7f262431f700 1 -- 172.21.2.112:0/668029365 shutdown_connections 2021-02-10T11:16:06.883 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:06.868 7f262431f700 1 -- 172.21.2.112:0/668029365 wait complete. 2021-02-10T11:16:06.885 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:06.883 7f262431f700 1 Processor -- start 2021-02-10T11:16:06.885 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:06.883 7f262431f700 1 -- start start 2021-02-10T11:16:06.885 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:06.883 7f262431f700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f261c08ece0 0x7f261c0a6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:06.885 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:06.883 7f262431f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f261c0a6860 0x7f261c0a6c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:06.885 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:06.883 7f262431f700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f261c0a7280 0x7f261c0a4590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:06.885 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:06.883 7f262431f700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f261c0a5cc0 con 0x7f261c0a6860 2021-02-10T11:16:06.885 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:06.883 7f262431f700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f261c08f0e0 con 0x7f261c0a7280 2021-02-10T11:16:06.885 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:06.883 7f262431f700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f261c08f2c0 con 0x7f261c08ece0 2021-02-10T11:16:06.885 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:06.884 7f262193a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f261c0a6860 0x7f261c0a6c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:06.885 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:06.884 7f262193a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f261c0a6860 0x7f261c0a6c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:43856/0 (socket says 172.21.2.112:43856) 2021-02-10T11:16:06.886 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:06.884 7f262193a700 1 -- 172.21.2.112:0/3794860358 learned_addr learned my addr 172.21.2.112:0/3794860358 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:16:06.886 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:06.884 7f262213b700 1 --2- 172.21.2.112:0/3794860358 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f261c08ece0 0x7f261c0a6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:06.886 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:06.884 7f262293c700 1 --2- 172.21.2.112:0/3794860358 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f261c0a7280 0x7f261c0a4590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:06.886 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:06.884 7f262193a700 1 -- 172.21.2.112:0/3794860358 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f261c08ece0 msgr2=0x7f261c0a6240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:06.886 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:06.884 7f262193a700 1 --2- 172.21.2.112:0/3794860358 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f261c08ece0 0x7f261c0a6240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:06.886 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:06.884 7f262193a700 1 -- 172.21.2.112:0/3794860358 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f261c0a7280 msgr2=0x7f261c0a4590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:06.886 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:06.884 7f262193a700 1 --2- 172.21.2.112:0/3794860358 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f261c0a7280 0x7f261c0a4590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:06.886 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:06.884 7f262193a700 1 -- 172.21.2.112:0/3794860358 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f261c0a4bb0 con 0x7f261c0a6860 2021-02-10T11:16:06.886 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:06.884 7f262293c700 1 --2- 172.21.2.112:0/3794860358 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f261c0a7280 0x7f261c0a4590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:16:06.886 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:06.884 7f262193a700 1 --2- 172.21.2.112:0/3794860358 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f261c0a6860 0x7f261c0a6c60 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7f261800b580 tx=0x7f261800b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:16:06.886 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:06.884 7f260effd700 1 -- 172.21.2.112:0/3794860358 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f261800bd10 con 0x7f261c0a6860 2021-02-10T11:16:06.886 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:06.884 7f260effd700 1 -- 172.21.2.112:0/3794860358 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2618002030 con 0x7f261c0a6860 2021-02-10T11:16:06.886 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:06.884 7f260effd700 1 -- 172.21.2.112:0/3794860358 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f26180024d0 con 0x7f261c0a6860 2021-02-10T11:16:06.887 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:06.884 7f262431f700 1 -- 172.21.2.112:0/3794860358 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f261c0a4bb0 con 0x7f261c0a6860 2021-02-10T11:16:06.887 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:06.884 7f262431f700 1 -- 172.21.2.112:0/3794860358 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f261c0a4bb0 con 0x7f261c0a6860 2021-02-10T11:16:06.887 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:06.885 7f260effd700 1 -- 172.21.2.112:0/3794860358 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2618002210 con 0x7f261c0a6860 2021-02-10T11:16:06.887 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:06.885 7f260effd700 1 --2- 172.21.2.112:0/3794860358 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f26040186c0 0x7f260401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:06.887 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:06.885 7f260effd700 1 -- 172.21.2.112:0/3794860358 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f2618032540 con 0x7f261c0a6860 2021-02-10T11:16:06.887 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:06.885 7f262213b700 1 --2- 172.21.2.112:0/3794860358 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f26040186c0 0x7f260401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:06.887 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:06.886 7f262213b700 1 --2- 172.21.2.112:0/3794860358 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f26040186c0 0x7f260401abc0 crc :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:16:06.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:06.916 7f262431f700 1 -- 172.21.2.112:0/3794860358 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f261c105ae0 con 0x7f261c0a6860 2021-02-10T11:16:06.917 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:06.919 7f260effd700 1 -- 172.21.2.112:0/3794860358 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f26180091e0 con 0x7f261c0a6860 2021-02-10T11:16:07.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:07.046 7f262431f700 1 -- 172.21.2.112:0/3794860358 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348569", "target": ["mgr", ""]}) v1 -- 0x7f261c0f0b50 con 0x7f26040186c0 2021-02-10T11:16:07.068 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:07.069 7f260effd700 1 -- 172.21.2.112:0/3794860358 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f2610000ce0 con 0x7f26040186c0 2021-02-10T11:16:07.078 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:07.079 7f262431f700 1 -- 172.21.2.112:0/3794860358 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f26040186c0 msgr2=0x7f260401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:07.079 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:07.079 7f262431f700 1 --2- 172.21.2.112:0/3794860358 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f26040186c0 0x7f260401abc0 crc :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:07.079 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:07.079 7f262431f700 1 -- 172.21.2.112:0/3794860358 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f261c0a6860 msgr2=0x7f261c0a6c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:07.079 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:07.079 7f262431f700 1 --2- 172.21.2.112:0/3794860358 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f261c0a6860 0x7f261c0a6c60 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7f261800b580 tx=0x7f261800b920).stop 2021-02-10T11:16:07.079 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:07.079 7f262431f700 1 -- 172.21.2.112:0/3794860358 shutdown_connections 2021-02-10T11:16:07.079 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:07.079 7f262431f700 1 --2- 172.21.2.112:0/3794860358 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f26040186c0 0x7f260401abc0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:07.079 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:07.079 7f262431f700 1 --2- 172.21.2.112:0/3794860358 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f261c0a7280 0x7f261c0a4590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:07.079 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:07.079 7f262431f700 1 --2- 172.21.2.112:0/3794860358 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f261c0a6860 0x7f261c0a6c60 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:07.079 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:07.079 7f262431f700 1 --2- 172.21.2.112:0/3794860358 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f261c08ece0 0x7f261c0a6240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:07.079 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:07.079 7f262431f700 1 -- 172.21.2.112:0/3794860358 >> 172.21.2.112:0/3794860358 conn(0x7f261c08ffe0 msgr2=0x7f261c0bf050 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:16:07.079 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:07.080 7f262431f700 1 -- 172.21.2.112:0/3794860358 shutdown_connections 2021-02-10T11:16:07.079 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:07.080 7f262431f700 1 -- 172.21.2.112:0/3794860358 wait complete. 2021-02-10T11:16:07.086 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000348569 2021-02-10T11:16:07.515 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:07.500 7f9520ad1700 1 Processor -- start 2021-02-10T11:16:07.515 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:07.500 7f9520ad1700 1 -- start start 2021-02-10T11:16:07.515 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:07.501 7f9520ad1700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f951c1b0470 0x7f951c1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:07.515 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:07.501 7f9520ad1700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f951c0f20e0 0x7f951c0f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:07.516 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:07.501 7f9520ad1700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f951c0f0b50 con 0x7f951c0ef610 2021-02-10T11:16:07.516 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:07.501 7f9520ad1700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f951c0f0820 con 0x7f951c0f20e0 2021-02-10T11:16:07.516 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:07.501 7f9520ad1700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f951c0f24e0 con 0x7f951c1b0470 2021-02-10T11:16:07.516 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:07.501 7f951b05f700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f951c0f20e0 0x7f951c0f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:07.516 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:07.501 7f951b05f700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f951c0f20e0 0x7f951c0f0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:43866/0 (socket says 172.21.2.112:43866) 2021-02-10T11:16:07.516 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:07.501 7f951b05f700 1 -- 172.21.2.112:0/1960199928 learned_addr learned my addr 172.21.2.112:0/1960199928 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:16:07.516 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:07.501 7f951b05f700 1 -- 172.21.2.112:0/1960199928 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f951c1b0470 msgr2=0x7f951c1b0870 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:16:07.517 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:07.501 7f951b05f700 1 --2- 172.21.2.112:0/1960199928 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f951c1b0470 0x7f951c1b0870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:16:07.517 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:07.501 7f951b05f700 1 -- 172.21.2.112:0/1960199928 >> v1:172.21.2.120:6789/0 conn(0x7f951c0ef610 legacy=0x7f951c0efa10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:16:07.517 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:07.501 7f951b05f700 1 -- 172.21.2.112:0/1960199928 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f951c0e52e0 con 0x7f951c0f20e0 2021-02-10T11:16:07.517 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:07.501 7f951b05f700 1 --2- 172.21.2.112:0/1960199928 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f951c0f20e0 0x7f951c0f0200 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 rx=0x7f9514006f00 tx=0x7f9514009f10).ready entity=mon.0 client_cookie=101efaa60337b82c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:16:07.517 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:07.502 7f951985c700 1 -- 172.21.2.112:0/1960199928 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f951400b3b0 con 0x7f951c0f20e0 2021-02-10T11:16:07.517 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:07.502 7f951985c700 1 -- 172.21.2.112:0/1960199928 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f951400b630 con 0x7f951c0f20e0 2021-02-10T11:16:07.517 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:07.502 7f951985c700 1 -- 172.21.2.112:0/1960199928 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f951400bb50 con 0x7f951c0f20e0 2021-02-10T11:16:07.517 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:07.502 7f9520ad1700 1 -- 172.21.2.112:0/1960199928 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f951c0f20e0 msgr2=0x7f951c0f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:07.517 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:07.502 7f9520ad1700 1 --2- 172.21.2.112:0/1960199928 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f951c0f20e0 0x7f951c0f0200 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 rx=0x7f9514006f00 tx=0x7f9514009f10).stop 2021-02-10T11:16:07.517 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:07.502 7f9520ad1700 1 -- 172.21.2.112:0/1960199928 shutdown_connections 2021-02-10T11:16:07.517 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:07.502 7f9520ad1700 1 --2- 172.21.2.112:0/1960199928 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f951c0f20e0 0x7f951c0f0200 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:07.518 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:07.502 7f9520ad1700 1 --2- 172.21.2.112:0/1960199928 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f951c1b0470 0x7f951c1b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:16:07.518 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:07.502 7f9520ad1700 1 -- 172.21.2.112:0/1960199928 >> 172.21.2.112:0/1960199928 conn(0x7f951c0ce3e0 msgr2=0x7f951c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:16:07.518 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:07.503 7f9520ad1700 1 -- 172.21.2.112:0/1960199928 shutdown_connections 2021-02-10T11:16:07.518 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:07.503 7f9520ad1700 1 -- 172.21.2.112:0/1960199928 wait complete. 2021-02-10T11:16:07.519 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:07.517 7f9520ad1700 1 Processor -- start 2021-02-10T11:16:07.520 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:07.517 7f9520ad1700 1 -- start start 2021-02-10T11:16:07.520 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:07.517 7f9520ad1700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f951c094fb0 0x7f951c09e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:07.520 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:07.517 7f9520ad1700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f951c0c2be0 0x7f951c0c2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:07.520 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:07.517 7f9520ad1700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f951c0c3600 0x7f951c0a9c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:07.520 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:07.517 7f9520ad1700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f951c09ebf0 con 0x7f951c0c2be0 2021-02-10T11:16:07.520 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:07.517 7f9520ad1700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f951c0aa2a0 con 0x7f951c094fb0 2021-02-10T11:16:07.520 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:07.517 7f9520ad1700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f951c0aa4c0 con 0x7f951c0c3600 2021-02-10T11:16:07.520 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:07.518 7f951a05d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f951c0c2be0 0x7f951c0c2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:07.520 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:07.518 7f951a05d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f951c0c2be0 0x7f951c0c2fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:43870/0 (socket says 172.21.2.112:43870) 2021-02-10T11:16:07.520 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:07.518 7f951a05d700 1 -- 172.21.2.112:0/2337365533 learned_addr learned my addr 172.21.2.112:0/2337365533 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:16:07.520 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:07.518 7f951a05d700 1 -- 172.21.2.112:0/2337365533 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f951c0c3600 msgr2=0x7f951c0a9c80 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:16:07.520 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:07.518 7f951a05d700 1 --2- 172.21.2.112:0/2337365533 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f951c0c3600 0x7f951c0a9c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:16:07.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:07.518 7f951a05d700 1 -- 172.21.2.112:0/2337365533 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f951c094fb0 msgr2=0x7f951c09e5d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:16:07.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:07.518 7f951a05d700 1 --2- 172.21.2.112:0/2337365533 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f951c094fb0 0x7f951c09e5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:16:07.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:07.518 7f951a05d700 1 -- 172.21.2.112:0/2337365533 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f951c0aa6e0 con 0x7f951c0c2be0 2021-02-10T11:16:07.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:07.518 7f951a05d700 1 --2- 172.21.2.112:0/2337365533 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f951c0c2be0 0x7f951c0c2fe0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7f950400b530 tx=0x7f950400b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:16:07.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:07.518 7f95137fe700 1 -- 172.21.2.112:0/2337365533 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f950400bcc0 con 0x7f951c0c2be0 2021-02-10T11:16:07.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:07.518 7f95137fe700 1 -- 172.21.2.112:0/2337365533 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9504006020 con 0x7f951c0c2be0 2021-02-10T11:16:07.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:07.519 7f9520ad1700 1 -- 172.21.2.112:0/2337365533 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f951c0aa6e0 con 0x7f951c0c2be0 2021-02-10T11:16:07.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:07.519 7f9520ad1700 1 -- 172.21.2.112:0/2337365533 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f951c0aac70 con 0x7f951c0c2be0 2021-02-10T11:16:07.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:07.519 7f95137fe700 1 -- 172.21.2.112:0/2337365533 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f95040054a0 con 0x7f951c0c2be0 2021-02-10T11:16:07.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:07.519 7f95137fe700 1 -- 172.21.2.112:0/2337365533 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9504006260 con 0x7f951c0c2be0 2021-02-10T11:16:07.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:07.519 7f95137fe700 1 --2- 172.21.2.112:0/2337365533 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9500018730 0x7f950001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:07.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:07.519 7f95137fe700 1 -- 172.21.2.112:0/2337365533 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f9504032690 con 0x7f951c0c2be0 2021-02-10T11:16:07.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:07.520 7f951a85e700 1 --2- 172.21.2.112:0/2337365533 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9500018730 0x7f950001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:07.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:07.520 7f951a85e700 1 --2- 172.21.2.112:0/2337365533 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9500018730 0x7f950001ac10 crc :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:16:07.549 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:07.550 7f9520ad1700 1 -- 172.21.2.112:0/2337365533 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f951c0a9750 con 0x7f951c0c2be0 2021-02-10T11:16:07.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:07.554 7f95137fe700 1 -- 172.21.2.112:0/2337365533 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9504006000 con 0x7f951c0c2be0 2021-02-10T11:16:07.679 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:07.680 7f9520ad1700 1 -- 172.21.2.112:0/2337365533 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348569", "target": ["mgr", ""]}) v1 -- 0x7f951c0aa2a0 con 0x7f9500018730 2021-02-10T11:16:07.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:07.684 7f95137fe700 1 -- 172.21.2.112:0/2337365533 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f950c006990 con 0x7f9500018730 2021-02-10T11:16:07.686 INFO:teuthology.orchestra.run.gibba012.stdout:/volumes/_nogroup/subvolume_0000000000348569/0cf87606-0319-467c-b698-9de3b7ff0d7a 2021-02-10T11:16:07.689 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:07.689 7f9520ad1700 1 -- 172.21.2.112:0/2337365533 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9500018730 msgr2=0x7f950001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:07.689 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:07.689 7f9520ad1700 1 --2- 172.21.2.112:0/2337365533 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9500018730 0x7f950001ac10 crc :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:07.689 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:07.689 7f9520ad1700 1 -- 172.21.2.112:0/2337365533 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f951c0c2be0 msgr2=0x7f951c0c2fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:07.689 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:07.689 7f9520ad1700 1 --2- 172.21.2.112:0/2337365533 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f951c0c2be0 0x7f951c0c2fe0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7f950400b530 tx=0x7f950400b8d0).stop 2021-02-10T11:16:07.689 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:07.690 7f9520ad1700 1 -- 172.21.2.112:0/2337365533 shutdown_connections 2021-02-10T11:16:07.689 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:07.690 7f9520ad1700 1 --2- 172.21.2.112:0/2337365533 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9500018730 0x7f950001ac10 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:07.690 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:07.690 7f9520ad1700 1 --2- 172.21.2.112:0/2337365533 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f951c0c3600 0x7f951c0a9c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:16:07.690 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:07.690 7f9520ad1700 1 --2- 172.21.2.112:0/2337365533 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f951c0c2be0 0x7f951c0c2fe0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:07.690 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:07.690 7f9520ad1700 1 --2- 172.21.2.112:0/2337365533 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f951c094fb0 0x7f951c09e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:16:07.690 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:07.690 7f9520ad1700 1 -- 172.21.2.112:0/2337365533 >> 172.21.2.112:0/2337365533 conn(0x7f951c09f630 msgr2=0x7f951c0a0130 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:16:07.690 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:07.690 7f9520ad1700 1 -- 172.21.2.112:0/2337365533 shutdown_connections 2021-02-10T11:16:07.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:07.690 7f9520ad1700 1 -- 172.21.2.112:0/2337365533 wait complete. 2021-02-10T11:16:07.697 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000348569 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000348569/0cf87606-0319-467c-b698-9de3b7ff0d7a 2021-02-10T11:16:07.697 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:07.697 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:07.697 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:08.034 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:16:08.034 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:08.034 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348569/0cf87606-0319-467c-b698-9de3b7ff0d7a/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:08.416 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:16:08.416 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:16:08.416 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0265352 s, 39.5 MB/s 2021-02-10T11:16:08.417 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:08.417 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:08.418 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:08.752 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:16:08.752 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:08.753 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348569/0cf87606-0319-467c-b698-9de3b7ff0d7a/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:09.127 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:16:09.127 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:16:09.127 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0235078 s, 44.6 MB/s 2021-02-10T11:16:09.129 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:09.129 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:09.129 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:09.462 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:16:09.520 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:09.521 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348569/0cf87606-0319-467c-b698-9de3b7ff0d7a/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:09.894 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:16:09.895 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:16:09.895 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0230571 s, 45.5 MB/s 2021-02-10T11:16:09.896 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:09.896 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:09.896 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:10.231 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:16:10.232 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:10.232 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348569/0cf87606-0319-467c-b698-9de3b7ff0d7a/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:10.605 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:16:10.605 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:16:10.605 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0232173 s, 45.2 MB/s 2021-02-10T11:16:10.606 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:10.606 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:10.606 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:10.940 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:16:10.940 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:10.941 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348569/0cf87606-0319-467c-b698-9de3b7ff0d7a/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:11.317 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:16:11.317 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:16:11.317 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0263391 s, 39.8 MB/s 2021-02-10T11:16:11.318 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:11.318 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:11.318 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:11.653 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:16:11.654 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:11.654 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348569/0cf87606-0319-467c-b698-9de3b7ff0d7a/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:12.029 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:16:12.030 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:16:12.030 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0250397 s, 41.9 MB/s 2021-02-10T11:16:12.031 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:12.031 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:12.031 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:12.370 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:16:12.370 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:12.370 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348569/0cf87606-0319-467c-b698-9de3b7ff0d7a/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:12.748 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:16:12.749 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:16:12.749 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0250007 s, 41.9 MB/s 2021-02-10T11:16:12.750 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:12.750 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:12.750 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:13.086 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:16:13.087 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:13.087 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348569/0cf87606-0319-467c-b698-9de3b7ff0d7a/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:13.463 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:16:13.463 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:16:13.464 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0233784 s, 44.9 MB/s 2021-02-10T11:16:13.464 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:13.465 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:13.465 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:13.801 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:16:13.802 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:13.802 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348569/0cf87606-0319-467c-b698-9de3b7ff0d7a/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:14.176 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:16:14.176 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:16:14.177 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0240039 s, 43.7 MB/s 2021-02-10T11:16:14.177 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:14.178 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:14.178 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:14.512 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:16:14.512 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:14.513 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348569/0cf87606-0319-467c-b698-9de3b7ff0d7a/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:14.886 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:16:14.887 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:16:14.887 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0250432 s, 41.9 MB/s 2021-02-10T11:16:14.890 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000348570 2021-02-10T11:16:15.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.305 7fb753f49700 1 Processor -- start 2021-02-10T11:16:15.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.305 7fb753f49700 1 -- start start 2021-02-10T11:16:15.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.306 7fb753f49700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fb74c1b0470 0x7fb74c1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:15.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.306 7fb753f49700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fb74c0f20e0 0x7fb74c0f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:15.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.306 7fb753f49700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb74c0f0b50 con 0x7fb74c0ef610 2021-02-10T11:16:15.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.306 7fb753f49700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb74c0f0820 con 0x7fb74c0f20e0 2021-02-10T11:16:15.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.306 7fb753f49700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb74c0f24e0 con 0x7fb74c1b0470 2021-02-10T11:16:15.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.306 7fb752566700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fb74c0f20e0 0x7fb74c0f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:15.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.306 7fb752566700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fb74c0f20e0 0x7fb74c0f0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:43876/0 (socket says 172.21.2.112:43876) 2021-02-10T11:16:15.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.306 7fb752566700 1 -- 172.21.2.112:0/3909819521 learned_addr learned my addr 172.21.2.112:0/3909819521 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:16:15.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.306 7fb752566700 1 -- 172.21.2.112:0/3909819521 >> v2:172.21.2.120:3300/0 conn(0x7fb74c1b0470 msgr2=0x7fb74c1b0870 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:16:15.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.306 7fb752566700 1 --2- 172.21.2.112:0/3909819521 >> v2:172.21.2.120:3300/0 conn(0x7fb74c1b0470 0x7fb74c1b0870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:16:15.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.306 7fb752566700 1 -- 172.21.2.112:0/3909819521 >> v1:172.21.2.120:6789/0 conn(0x7fb74c0ef610 legacy=0x7fb74c0efa10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:16:15.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.306 7fb752566700 1 -- 172.21.2.112:0/3909819521 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb74c0e52e0 con 0x7fb74c0f20e0 2021-02-10T11:16:15.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.306 7fb752566700 1 --2- 172.21.2.112:0/3909819521 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb74c0f20e0 0x7fb74c0f0200 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 rx=0x7fb744006f00 tx=0x7fb744009f10).ready entity=mon.0 client_cookie=38061908509352b9 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:16:15.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.307 7fb750d63700 1 -- 172.21.2.112:0/3909819521 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb74400b3b0 con 0x7fb74c0f20e0 2021-02-10T11:16:15.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.307 7fb750d63700 1 -- 172.21.2.112:0/3909819521 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb74400b630 con 0x7fb74c0f20e0 2021-02-10T11:16:15.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.307 7fb750d63700 1 -- 172.21.2.112:0/3909819521 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb74400bb50 con 0x7fb74c0f20e0 2021-02-10T11:16:15.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.307 7fb753f49700 1 -- 172.21.2.112:0/3909819521 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb74c0f20e0 msgr2=0x7fb74c0f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:15.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.307 7fb753f49700 1 --2- 172.21.2.112:0/3909819521 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb74c0f20e0 0x7fb74c0f0200 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 rx=0x7fb744006f00 tx=0x7fb744009f10).stop 2021-02-10T11:16:15.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.307 7fb753f49700 1 -- 172.21.2.112:0/3909819521 shutdown_connections 2021-02-10T11:16:15.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.307 7fb753f49700 1 --2- 172.21.2.112:0/3909819521 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb74c0f20e0 0x7fb74c0f0200 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:15.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.307 7fb753f49700 1 --2- 172.21.2.112:0/3909819521 >> v2:172.21.2.120:3300/0 conn(0x7fb74c1b0470 0x7fb74c1b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:16:15.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.307 7fb753f49700 1 -- 172.21.2.112:0/3909819521 >> 172.21.2.112:0/3909819521 conn(0x7fb74c0ce3e0 msgr2=0x7fb74c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:16:15.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.308 7fb753f49700 1 -- 172.21.2.112:0/3909819521 shutdown_connections 2021-02-10T11:16:15.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.308 7fb753f49700 1 -- 172.21.2.112:0/3909819521 wait complete. 2021-02-10T11:16:15.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.322 7fb753f49700 1 Processor -- start 2021-02-10T11:16:15.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.322 7fb753f49700 1 -- start start 2021-02-10T11:16:15.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.323 7fb753f49700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb74c094fb0 0x7fb74c09e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:15.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.323 7fb753f49700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb74c0c2be0 0x7fb74c0c2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:15.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.323 7fb753f49700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb74c0c3600 0x7fb74c0a9c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:15.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.323 7fb753f49700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb74c09ebf0 con 0x7fb74c0c2be0 2021-02-10T11:16:15.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.323 7fb753f49700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb74c0aa2a0 con 0x7fb74c094fb0 2021-02-10T11:16:15.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.323 7fb753f49700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb74c0aa4c0 con 0x7fb74c0c3600 2021-02-10T11:16:15.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.323 7fb751564700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb74c0c2be0 0x7fb74c0c2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:15.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.323 7fb751564700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb74c0c2be0 0x7fb74c0c2fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:43882/0 (socket says 172.21.2.112:43882) 2021-02-10T11:16:15.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.323 7fb751564700 1 -- 172.21.2.112:0/1385497424 learned_addr learned my addr 172.21.2.112:0/1385497424 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:16:15.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.323 7fb751d65700 1 --2- 172.21.2.112:0/1385497424 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb74c094fb0 0x7fb74c09e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:15.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.323 7fb752566700 1 --2- 172.21.2.112:0/1385497424 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb74c0c3600 0x7fb74c0a9c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:15.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.323 7fb751564700 1 -- 172.21.2.112:0/1385497424 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb74c0c3600 msgr2=0x7fb74c0a9c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:15.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.323 7fb751564700 1 --2- 172.21.2.112:0/1385497424 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb74c0c3600 0x7fb74c0a9c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:15.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.324 7fb751564700 1 -- 172.21.2.112:0/1385497424 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb74c094fb0 msgr2=0x7fb74c09e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:15.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.324 7fb751564700 1 --2- 172.21.2.112:0/1385497424 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb74c094fb0 0x7fb74c09e5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:15.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.324 7fb751564700 1 -- 172.21.2.112:0/1385497424 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb74c0aa6e0 con 0x7fb74c0c2be0 2021-02-10T11:16:15.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.324 7fb752566700 1 --2- 172.21.2.112:0/1385497424 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb74c0c3600 0x7fb74c0a9c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:16:15.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.324 7fb751564700 1 --2- 172.21.2.112:0/1385497424 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb74c0c2be0 0x7fb74c0c2fe0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7fb74800b5b0 tx=0x7fb74800b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:16:15.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.324 7fb73e7fc700 1 -- 172.21.2.112:0/1385497424 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb74800bd40 con 0x7fb74c0c2be0 2021-02-10T11:16:15.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.324 7fb73e7fc700 1 -- 172.21.2.112:0/1385497424 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb748000c60 con 0x7fb74c0c2be0 2021-02-10T11:16:15.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.324 7fb73e7fc700 1 -- 172.21.2.112:0/1385497424 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb7480062a0 con 0x7fb74c0c2be0 2021-02-10T11:16:15.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.324 7fb753f49700 1 -- 172.21.2.112:0/1385497424 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb74c0aa6e0 con 0x7fb74c0c2be0 2021-02-10T11:16:15.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.324 7fb753f49700 1 -- 172.21.2.112:0/1385497424 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb74c0aa6e0 con 0x7fb74c0c2be0 2021-02-10T11:16:15.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.325 7fb73e7fc700 1 -- 172.21.2.112:0/1385497424 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb748006020 con 0x7fb74c0c2be0 2021-02-10T11:16:15.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.325 7fb73e7fc700 1 --2- 172.21.2.112:0/1385497424 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb7340186c0 0x7fb73401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:15.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.325 7fb73e7fc700 1 -- 172.21.2.112:0/1385497424 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fb7480325c0 con 0x7fb74c0c2be0 2021-02-10T11:16:15.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.325 7fb751d65700 1 --2- 172.21.2.112:0/1385497424 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb7340186c0 0x7fb73401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:15.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.326 7fb751d65700 1 --2- 172.21.2.112:0/1385497424 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb7340186c0 0x7fb73401abc0 crc :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:16:15.353 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.356 7fb753f49700 1 -- 172.21.2.112:0/1385497424 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb74c0a9750 con 0x7fb74c0c2be0 2021-02-10T11:16:15.355 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.358 7fb73e7fc700 1 -- 172.21.2.112:0/1385497424 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb748030d70 con 0x7fb74c0c2be0 2021-02-10T11:16:15.481 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.484 7fb753f49700 1 -- 172.21.2.112:0/1385497424 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348570", "target": ["mgr", ""]}) v1 -- 0x7fb74c0a9750 con 0x7fb7340186c0 2021-02-10T11:16:15.508 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.510 7fb73e7fc700 1 -- 172.21.2.112:0/1385497424 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fb740000ce0 con 0x7fb7340186c0 2021-02-10T11:16:15.515 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.517 7fb753f49700 1 -- 172.21.2.112:0/1385497424 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb7340186c0 msgr2=0x7fb73401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:15.516 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.517 7fb753f49700 1 --2- 172.21.2.112:0/1385497424 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb7340186c0 0x7fb73401abc0 crc :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:15.516 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.517 7fb753f49700 1 -- 172.21.2.112:0/1385497424 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb74c0c2be0 msgr2=0x7fb74c0c2fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:15.516 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.517 7fb753f49700 1 --2- 172.21.2.112:0/1385497424 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb74c0c2be0 0x7fb74c0c2fe0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7fb74800b5b0 tx=0x7fb74800b950).stop 2021-02-10T11:16:15.516 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.518 7fb753f49700 1 -- 172.21.2.112:0/1385497424 shutdown_connections 2021-02-10T11:16:15.516 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.518 7fb753f49700 1 --2- 172.21.2.112:0/1385497424 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb7340186c0 0x7fb73401abc0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:15.516 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.518 7fb753f49700 1 --2- 172.21.2.112:0/1385497424 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb74c0c3600 0x7fb74c0a9c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:15.516 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.518 7fb753f49700 1 --2- 172.21.2.112:0/1385497424 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb74c0c2be0 0x7fb74c0c2fe0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:15.516 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.518 7fb753f49700 1 --2- 172.21.2.112:0/1385497424 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb74c094fb0 0x7fb74c09e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:15.516 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.518 7fb753f49700 1 -- 172.21.2.112:0/1385497424 >> 172.21.2.112:0/1385497424 conn(0x7fb74c09f630 msgr2=0x7fb74c0a0130 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:16:15.516 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.518 7fb753f49700 1 -- 172.21.2.112:0/1385497424 shutdown_connections 2021-02-10T11:16:15.517 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.518 7fb753f49700 1 -- 172.21.2.112:0/1385497424 wait complete. 2021-02-10T11:16:15.524 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000348570 2021-02-10T11:16:15.954 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.940 7f896bfcc700 1 Processor -- start 2021-02-10T11:16:15.954 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.940 7f896bfcc700 1 -- start start 2021-02-10T11:16:15.954 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.940 7f896bfcc700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f89641b0470 0x7f89641b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:15.954 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.940 7f896bfcc700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f89640ef610 0x7f89640efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:15.954 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.940 7f896bfcc700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f89640f0030 0x7f89640f0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:15.954 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.940 7f896bfcc700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89640e5ea0 con 0x7f89640f0030 2021-02-10T11:16:15.954 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.940 7f896bfcc700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89640f0b50 con 0x7f89640ef610 2021-02-10T11:16:15.955 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.940 7f896bfcc700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89640e52e0 con 0x7f89641b0470 2021-02-10T11:16:15.955 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.940 7f896a5e9700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f89640f0030 0x7f89640f0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:15.955 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.940 7f896a5e9700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f89640f0030 0x7f89640f0430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:43888/0 (socket says 172.21.2.112:43888) 2021-02-10T11:16:15.955 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.940 7f896a5e9700 1 -- 172.21.2.112:0/3356937414 learned_addr learned my addr 172.21.2.112:0/3356937414 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:16:15.955 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.940 7f89695e7700 1 --2- 172.21.2.112:0/3356937414 >> v2:172.21.2.120:3300/0 conn(0x7f89640ef610 0x7f89640efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:15.955 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.941 7f8969de8700 1 --2- 172.21.2.112:0/3356937414 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f89641b0470 0x7f89641b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:15.955 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.941 7f896a5e9700 1 -- 172.21.2.112:0/3356937414 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f89641b0470 msgr2=0x7f89641b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:16:15.955 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.941 7f896a5e9700 1 --2- 172.21.2.112:0/3356937414 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f89641b0470 0x7f89641b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:15.955 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.941 7f896a5e9700 1 -- 172.21.2.112:0/3356937414 >> v2:172.21.2.120:3300/0 conn(0x7f89640ef610 msgr2=0x7f89640efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:16:15.955 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.941 7f896a5e9700 1 --2- 172.21.2.112:0/3356937414 >> v2:172.21.2.120:3300/0 conn(0x7f89640ef610 0x7f89640efa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:15.955 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.941 7f896a5e9700 1 -- 172.21.2.112:0/3356937414 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89640f10d0 con 0x7f89640f0030 2021-02-10T11:16:15.955 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.941 7f896a5e9700 1 --2- 172.21.2.112:0/3356937414 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f89640f0030 0x7f89640f0430 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 rx=0x7f895c006f00 tx=0x7f895c009f10).ready entity=mon.0 client_cookie=2fa96ec6faaa8eb6 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:16:15.955 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.941 7f8968de6700 1 -- 172.21.2.112:0/3356937414 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f895c00b3b0 con 0x7f89640f0030 2021-02-10T11:16:15.956 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.941 7f8968de6700 1 -- 172.21.2.112:0/3356937414 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f895c00b630 con 0x7f89640f0030 2021-02-10T11:16:15.956 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.941 7f8968de6700 1 -- 172.21.2.112:0/3356937414 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f895c00ba90 con 0x7f89640f0030 2021-02-10T11:16:15.956 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.941 7f896bfcc700 1 -- 172.21.2.112:0/3356937414 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f89640f0030 msgr2=0x7f89640f0430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:15.956 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.941 7f896bfcc700 1 --2- 172.21.2.112:0/3356937414 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f89640f0030 0x7f89640f0430 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 rx=0x7f895c006f00 tx=0x7f895c009f10).stop 2021-02-10T11:16:15.956 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.942 7f896bfcc700 1 -- 172.21.2.112:0/3356937414 shutdown_connections 2021-02-10T11:16:15.956 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.942 7f896bfcc700 1 --2- 172.21.2.112:0/3356937414 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f89640f0030 0x7f89640f0430 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:15.956 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.942 7f896bfcc700 1 --2- 172.21.2.112:0/3356937414 >> v2:172.21.2.120:3300/0 conn(0x7f89640ef610 0x7f89640efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:15.956 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.942 7f896bfcc700 1 --2- 172.21.2.112:0/3356937414 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f89641b0470 0x7f89641b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:15.956 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.942 7f896bfcc700 1 -- 172.21.2.112:0/3356937414 >> 172.21.2.112:0/3356937414 conn(0x7f89640ce3e0 msgr2=0x7f89640e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:16:15.956 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.942 7f896bfcc700 1 -- 172.21.2.112:0/3356937414 shutdown_connections 2021-02-10T11:16:15.956 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.942 7f896bfcc700 1 -- 172.21.2.112:0/3356937414 wait complete. 2021-02-10T11:16:15.958 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.957 7f896bfcc700 1 Processor -- start 2021-02-10T11:16:15.958 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.957 7f896bfcc700 1 -- start start 2021-02-10T11:16:15.958 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.957 7f896bfcc700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f896409b930 0x7f89640b7c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:15.958 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.957 7f896bfcc700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f89640b8410 0x7f89640b8810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:15.958 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.957 7f896bfcc700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f89640c1f80 0x7f89640c2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:15.958 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.957 7f896bfcc700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89640c1a00 con 0x7f896409b930 2021-02-10T11:16:15.958 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.957 7f896bfcc700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89640b8e30 con 0x7f89640b8410 2021-02-10T11:16:15.958 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.957 7f896bfcc700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89640b9050 con 0x7f89640c1f80 2021-02-10T11:16:15.959 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.957 7f8969de8700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f896409b930 0x7f89640b7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:15.959 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.957 7f8969de8700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f896409b930 0x7f89640b7c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:43890/0 (socket says 172.21.2.112:43890) 2021-02-10T11:16:15.959 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.957 7f8969de8700 1 -- 172.21.2.112:0/3928015060 learned_addr learned my addr 172.21.2.112:0/3928015060 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:16:15.959 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.957 7f89695e7700 1 --2- 172.21.2.112:0/3928015060 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f89640b8410 0x7f89640b8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:15.959 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.958 7f896a5e9700 1 --2- 172.21.2.112:0/3928015060 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f89640c1f80 0x7f89640c2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:15.959 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.958 7f8969de8700 1 -- 172.21.2.112:0/3928015060 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f89640c1f80 msgr2=0x7f89640c2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:15.959 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.958 7f8969de8700 1 --2- 172.21.2.112:0/3928015060 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f89640c1f80 0x7f89640c2380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:15.959 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.958 7f8969de8700 1 -- 172.21.2.112:0/3928015060 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f89640b8410 msgr2=0x7f89640b8810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:15.959 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.958 7f8969de8700 1 --2- 172.21.2.112:0/3928015060 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f89640b8410 0x7f89640b8810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:15.959 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.958 7f8969de8700 1 -- 172.21.2.112:0/3928015060 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89640c29a0 con 0x7f896409b930 2021-02-10T11:16:15.959 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.958 7f8969de8700 1 --2- 172.21.2.112:0/3928015060 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f896409b930 0x7f89640b7c20 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7f895400b560 tx=0x7f895400b900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:16:15.959 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.958 7f895a7fc700 1 -- 172.21.2.112:0/3928015060 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f895400bcf0 con 0x7f896409b930 2021-02-10T11:16:15.959 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.958 7f895a7fc700 1 -- 172.21.2.112:0/3928015060 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8954000c90 con 0x7f896409b930 2021-02-10T11:16:15.960 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.958 7f896bfcc700 1 -- 172.21.2.112:0/3928015060 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89640c29a0 con 0x7f896409b930 2021-02-10T11:16:15.960 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.958 7f895a7fc700 1 -- 172.21.2.112:0/3928015060 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8954002220 con 0x7f896409b930 2021-02-10T11:16:15.960 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.958 7f896bfcc700 1 -- 172.21.2.112:0/3928015060 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89640c29a0 con 0x7f896409b930 2021-02-10T11:16:15.960 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.959 7f895a7fc700 1 -- 172.21.2.112:0/3928015060 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f89540052e0 con 0x7f896409b930 2021-02-10T11:16:15.960 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.959 7f895a7fc700 1 --2- 172.21.2.112:0/3928015060 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f894c018730 0x7f894c01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:15.960 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.959 7f895a7fc700 1 -- 172.21.2.112:0/3928015060 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f89540327d0 con 0x7f896409b930 2021-02-10T11:16:15.960 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.959 7f89695e7700 1 --2- 172.21.2.112:0/3928015060 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f894c018730 0x7f894c01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:15.960 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.960 7f89695e7700 1 --2- 172.21.2.112:0/3928015060 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f894c018730 0x7f894c01ac10 crc :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:16:15.988 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.990 7f896bfcc700 1 -- 172.21.2.112:0/3928015060 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8964105b30 con 0x7f896409b930 2021-02-10T11:16:15.990 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:15.993 7f895a7fc700 1 -- 172.21.2.112:0/3928015060 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8954000c90 con 0x7f896409b930 2021-02-10T11:16:16.119 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:16.121 7f896bfcc700 1 -- 172.21.2.112:0/3928015060 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348570", "target": ["mgr", ""]}) v1 -- 0x7f89640b8e30 con 0x7f894c018730 2021-02-10T11:16:16.122 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:16.124 7f895a7fc700 1 -- 172.21.2.112:0/3928015060 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f8960006a70 con 0x7f894c018730 2021-02-10T11:16:16.122 INFO:teuthology.orchestra.run.gibba012.stdout:/volumes/_nogroup/subvolume_0000000000348570/69aef623-8fb5-4551-83da-7fa973873571 2021-02-10T11:16:16.123 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:16.126 7f896bfcc700 1 -- 172.21.2.112:0/3928015060 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f894c018730 msgr2=0x7f894c01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:16.123 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:16.126 7f896bfcc700 1 --2- 172.21.2.112:0/3928015060 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f894c018730 0x7f894c01ac10 crc :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:16.123 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:16.126 7f896bfcc700 1 -- 172.21.2.112:0/3928015060 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f896409b930 msgr2=0x7f89640b7c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:16.123 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:16.126 7f896bfcc700 1 --2- 172.21.2.112:0/3928015060 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f896409b930 0x7f89640b7c20 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7f895400b560 tx=0x7f895400b900).stop 2021-02-10T11:16:16.124 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:16.126 7f896bfcc700 1 -- 172.21.2.112:0/3928015060 shutdown_connections 2021-02-10T11:16:16.124 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:16.126 7f896bfcc700 1 --2- 172.21.2.112:0/3928015060 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f894c018730 0x7f894c01ac10 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:16.124 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:16.126 7f896bfcc700 1 --2- 172.21.2.112:0/3928015060 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f89640b8410 0x7f89640b8810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:16.124 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:16.126 7f896bfcc700 1 --2- 172.21.2.112:0/3928015060 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f89640c1f80 0x7f89640c2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:16.124 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:16.126 7f896bfcc700 1 --2- 172.21.2.112:0/3928015060 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f896409b930 0x7f89640b7c20 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:16.124 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:16.126 7f896bfcc700 1 -- 172.21.2.112:0/3928015060 >> 172.21.2.112:0/3928015060 conn(0x7f8964094e10 msgr2=0x7f89640a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:16:16.124 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:16.126 7f896bfcc700 1 -- 172.21.2.112:0/3928015060 shutdown_connections 2021-02-10T11:16:16.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:16.126 7f896bfcc700 1 -- 172.21.2.112:0/3928015060 wait complete. 2021-02-10T11:16:16.131 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000348570 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000348570/69aef623-8fb5-4551-83da-7fa973873571 2021-02-10T11:16:16.131 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:16.132 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:16.132 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:16.466 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:16:16.467 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:16.467 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348570/69aef623-8fb5-4551-83da-7fa973873571/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:16.848 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:16:16.848 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:16:16.848 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0260724 s, 40.2 MB/s 2021-02-10T11:16:16.849 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:16.850 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:16.850 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:17.184 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:16:17.185 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:17.185 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348570/69aef623-8fb5-4551-83da-7fa973873571/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:17.563 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:16:17.563 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:16:17.563 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0286478 s, 36.6 MB/s 2021-02-10T11:16:17.564 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:17.564 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:17.564 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:17.899 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:16:17.900 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:17.900 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348570/69aef623-8fb5-4551-83da-7fa973873571/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:18.275 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:16:18.275 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:16:18.275 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0245217 s, 42.8 MB/s 2021-02-10T11:16:18.276 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:18.276 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:18.276 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:18.611 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:16:18.612 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:18.612 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348570/69aef623-8fb5-4551-83da-7fa973873571/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:18.986 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:16:18.987 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:16:18.987 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0246017 s, 42.6 MB/s 2021-02-10T11:16:18.988 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:18.988 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:18.988 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:19.323 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:16:19.323 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:19.323 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348570/69aef623-8fb5-4551-83da-7fa973873571/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:19.695 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:16:19.695 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:16:19.695 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0252092 s, 41.6 MB/s 2021-02-10T11:16:19.696 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:19.696 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:19.696 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:20.030 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:16:20.031 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:20.031 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348570/69aef623-8fb5-4551-83da-7fa973873571/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:20.405 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:16:20.405 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:16:20.406 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0245914 s, 42.6 MB/s 2021-02-10T11:16:20.406 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:20.407 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:20.407 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:20.741 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:16:20.741 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:20.742 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348570/69aef623-8fb5-4551-83da-7fa973873571/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:21.116 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:16:21.116 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:16:21.116 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0252334 s, 41.6 MB/s 2021-02-10T11:16:21.117 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:21.118 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:21.118 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:21.452 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:16:21.452 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:21.452 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348570/69aef623-8fb5-4551-83da-7fa973873571/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:21.827 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:16:21.827 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:16:21.827 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0254953 s, 41.1 MB/s 2021-02-10T11:16:21.828 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:21.828 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:21.829 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:22.163 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:16:22.164 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:22.164 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348570/69aef623-8fb5-4551-83da-7fa973873571/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:22.540 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:16:22.540 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:16:22.540 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0254711 s, 41.2 MB/s 2021-02-10T11:16:22.541 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:22.541 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:22.541 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:22.875 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:16:22.875 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:22.876 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348570/69aef623-8fb5-4551-83da-7fa973873571/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:23.252 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:16:23.252 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:16:23.252 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.025101 s, 41.8 MB/s 2021-02-10T11:16:23.253 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000348571 2021-02-10T11:16:23.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:23.668 7f120222b700 1 Processor -- start 2021-02-10T11:16:23.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:23.668 7f120222b700 1 -- start start 2021-02-10T11:16:23.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:23.668 7f120222b700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f11fc1b0470 0x7f11fc1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:23.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:23.668 7f120222b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f11fc0f20e0 0x7f11fc0f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:23.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:23.668 7f120222b700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11fc0f0b50 con 0x7f11fc0ef610 2021-02-10T11:16:23.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:23.668 7f120222b700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11fc0f0820 con 0x7f11fc1b0470 2021-02-10T11:16:23.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:23.668 7f120222b700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11fc0f24e0 con 0x7f11fc0f20e0 2021-02-10T11:16:23.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:23.669 7f11fb7fe700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f11fc0ef610 0x7f11fc0efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:43362/0 (socket says 172.21.2.112:43362) 2021-02-10T11:16:23.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:23.669 7f11fb7fe700 1 -- 172.21.2.112:0/3838946357 learned_addr learned my addr 172.21.2.112:0/3838946357 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:16:23.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:23.669 7f1200848700 1 --2- 172.21.2.112:0/3838946357 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f11fc0f20e0 0x7f11fc0f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:23.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:23.669 7f11fbfff700 1 --2- 172.21.2.112:0/3838946357 >> v2:172.21.2.120:3300/0 conn(0x7f11fc1b0470 0x7f11fc1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:23.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:23.669 7f11faffd700 1 -- 172.21.2.112:0/3838946357 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 671905139 0 0) 0x7f11f00009b0 con 0x7f11fc0ef610 2021-02-10T11:16:23.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:23.669 7f11faffd700 1 -- 172.21.2.112:0/3838946357 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11e4002fe0 con 0x7f11fc0ef610 2021-02-10T11:16:23.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:23.669 7f1200848700 1 -- 172.21.2.112:0/3838946357 >> v2:172.21.2.120:3300/0 conn(0x7f11fc1b0470 msgr2=0x7f11fc1b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:16:23.684 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:23.669 7f1200848700 1 --2- 172.21.2.112:0/3838946357 >> v2:172.21.2.120:3300/0 conn(0x7f11fc1b0470 0x7f11fc1b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:23.684 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:23.669 7f1200848700 1 -- 172.21.2.112:0/3838946357 >> v1:172.21.2.112:6789/0 conn(0x7f11fc0ef610 legacy=0x7f11fc0efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:23.684 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:23.669 7f1200848700 1 -- 172.21.2.112:0/3838946357 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11fc0e52e0 con 0x7f11fc0f20e0 2021-02-10T11:16:23.684 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:23.669 7f11fbfff700 1 --2- 172.21.2.112:0/3838946357 >> v2:172.21.2.120:3300/0 conn(0x7f11fc1b0470 0x7f11fc1b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:16:23.684 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:23.669 7f1200848700 1 --2- 172.21.2.112:0/3838946357 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f11fc0f20e0 0x7f11fc0f0200 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f11f4006e50 tx=0x7f11f4009e70).ready entity=mon.2 client_cookie=a40f189c25f7a691 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:16:23.684 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:23.670 7f11faffd700 1 -- 172.21.2.112:0/3838946357 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f11f4000f90 con 0x7f11fc0f20e0 2021-02-10T11:16:23.684 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:23.670 7f11faffd700 1 -- 172.21.2.112:0/3838946357 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11f400b300 con 0x7f11fc0f20e0 2021-02-10T11:16:23.684 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:23.670 7f11faffd700 1 -- 172.21.2.112:0/3838946357 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f11f400b820 con 0x7f11fc0f20e0 2021-02-10T11:16:23.684 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:23.670 7f120222b700 1 -- 172.21.2.112:0/3838946357 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f11fc0f20e0 msgr2=0x7f11fc0f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:23.684 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:23.670 7f120222b700 1 --2- 172.21.2.112:0/3838946357 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f11fc0f20e0 0x7f11fc0f0200 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f11f4006e50 tx=0x7f11f4009e70).stop 2021-02-10T11:16:23.684 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:23.670 7f120222b700 1 -- 172.21.2.112:0/3838946357 shutdown_connections 2021-02-10T11:16:23.684 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:23.670 7f120222b700 1 --2- 172.21.2.112:0/3838946357 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f11fc0f20e0 0x7f11fc0f0200 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:23.685 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:23.670 7f120222b700 1 --2- 172.21.2.112:0/3838946357 >> v2:172.21.2.120:3300/0 conn(0x7f11fc1b0470 0x7f11fc1b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:23.685 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:23.670 7f120222b700 1 -- 172.21.2.112:0/3838946357 >> 172.21.2.112:0/3838946357 conn(0x7f11fc0ce3e0 msgr2=0x7f11fc0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:16:23.685 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:23.670 7f120222b700 1 -- 172.21.2.112:0/3838946357 shutdown_connections 2021-02-10T11:16:23.685 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:23.670 7f120222b700 1 -- 172.21.2.112:0/3838946357 wait complete. 2021-02-10T11:16:23.685 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:23.685 7f120222b700 1 Processor -- start 2021-02-10T11:16:23.686 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:23.685 7f120222b700 1 -- start start 2021-02-10T11:16:23.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:23.686 7f120222b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f11fc109280 0x7f11fc109af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:23.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:23.686 7f120222b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f11fc10a110 0x7f11fc1a7440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:23.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:23.686 7f120222b700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f11fc10a510 0x7f11fc1a7a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:23.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:23.686 7f120222b700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11fc109680 con 0x7f11fc109280 2021-02-10T11:16:23.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:23.686 7f120222b700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11fc1a8080 con 0x7f11fc10a510 2021-02-10T11:16:23.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:23.686 7f120222b700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11fc1a82a0 con 0x7f11fc10a110 2021-02-10T11:16:23.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:23.686 7f11fbfff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f11fc109280 0x7f11fc109af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:23.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:23.686 7f11fbfff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f11fc109280 0x7f11fc109af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:43906/0 (socket says 172.21.2.112:43906) 2021-02-10T11:16:23.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:23.686 7f11fbfff700 1 -- 172.21.2.112:0/1939543208 learned_addr learned my addr 172.21.2.112:0/1939543208 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:16:23.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:23.686 7f1200848700 1 --2- 172.21.2.112:0/1939543208 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f11fc10a510 0x7f11fc1a7a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:23.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:23.686 7f11fb7fe700 1 --2- 172.21.2.112:0/1939543208 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f11fc10a110 0x7f11fc1a7440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:23.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:23.686 7f11fbfff700 1 -- 172.21.2.112:0/1939543208 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f11fc10a110 msgr2=0x7f11fc1a7440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:23.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:23.686 7f11fbfff700 1 --2- 172.21.2.112:0/1939543208 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f11fc10a110 0x7f11fc1a7440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:23.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:23.686 7f11fbfff700 1 -- 172.21.2.112:0/1939543208 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f11fc10a510 msgr2=0x7f11fc1a7a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:23.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:23.686 7f11fbfff700 1 --2- 172.21.2.112:0/1939543208 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f11fc10a510 0x7f11fc1a7a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:23.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:23.686 7f11fbfff700 1 -- 172.21.2.112:0/1939543208 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11fc090b90 con 0x7f11fc109280 2021-02-10T11:16:23.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:23.687 7f11fbfff700 1 --2- 172.21.2.112:0/1939543208 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f11fc109280 0x7f11fc109af0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7f11ec006f40 tx=0x7f11ec00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:16:23.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:23.687 7f11f8ff9700 1 -- 172.21.2.112:0/1939543208 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f11ec00bd80 con 0x7f11fc109280 2021-02-10T11:16:23.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:23.687 7f11f8ff9700 1 -- 172.21.2.112:0/1939543208 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11ec000c90 con 0x7f11fc109280 2021-02-10T11:16:23.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:23.687 7f120222b700 1 -- 172.21.2.112:0/1939543208 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11fc090b90 con 0x7f11fc109280 2021-02-10T11:16:23.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:23.687 7f120222b700 1 -- 172.21.2.112:0/1939543208 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11fc091120 con 0x7f11fc109280 2021-02-10T11:16:23.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:23.687 7f11f8ff9700 1 -- 172.21.2.112:0/1939543208 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f11ec0022b0 con 0x7f11fc109280 2021-02-10T11:16:23.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:23.687 7f11f8ff9700 1 -- 172.21.2.112:0/1939543208 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f11ec00a000 con 0x7f11fc109280 2021-02-10T11:16:23.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:23.687 7f11f8ff9700 1 --2- 172.21.2.112:0/1939543208 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f11dc0186c0 0x7f11dc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:23.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:23.687 7f11f8ff9700 1 -- 172.21.2.112:0/1939543208 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f11ec032800 con 0x7f11fc109280 2021-02-10T11:16:23.689 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:23.688 7f11fb7fe700 1 --2- 172.21.2.112:0/1939543208 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f11dc0186c0 0x7f11dc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:23.689 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:23.688 7f11fb7fe700 1 --2- 172.21.2.112:0/1939543208 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f11dc0186c0 0x7f11dc01abc0 crc :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:16:23.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:23.718 7f120222b700 1 -- 172.21.2.112:0/1939543208 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11fc105ae0 con 0x7f11fc109280 2021-02-10T11:16:23.718 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:23.720 7f11f8ff9700 1 -- 172.21.2.112:0/1939543208 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f11ec000990 con 0x7f11fc109280 2021-02-10T11:16:23.847 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:23.849 7f120222b700 1 -- 172.21.2.112:0/1939543208 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348571", "target": ["mgr", ""]}) v1 -- 0x7f11fc109680 con 0x7f11dc0186c0 2021-02-10T11:16:23.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:23.873 7f11f8ff9700 1 -- 172.21.2.112:0/1939543208 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f11f0000ec0 con 0x7f11dc0186c0 2021-02-10T11:16:23.880 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:23.882 7f120222b700 1 -- 172.21.2.112:0/1939543208 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f11dc0186c0 msgr2=0x7f11dc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:23.880 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:23.882 7f120222b700 1 --2- 172.21.2.112:0/1939543208 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f11dc0186c0 0x7f11dc01abc0 crc :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:23.880 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:23.882 7f120222b700 1 -- 172.21.2.112:0/1939543208 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f11fc109280 msgr2=0x7f11fc109af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:23.880 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:23.882 7f120222b700 1 --2- 172.21.2.112:0/1939543208 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f11fc109280 0x7f11fc109af0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7f11ec006f40 tx=0x7f11ec00b990).stop 2021-02-10T11:16:23.880 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:23.882 7f120222b700 1 -- 172.21.2.112:0/1939543208 shutdown_connections 2021-02-10T11:16:23.880 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:23.882 7f120222b700 1 --2- 172.21.2.112:0/1939543208 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f11dc0186c0 0x7f11dc01abc0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:23.881 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:23.882 7f120222b700 1 --2- 172.21.2.112:0/1939543208 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f11fc10a510 0x7f11fc1a7a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:23.881 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:23.882 7f120222b700 1 --2- 172.21.2.112:0/1939543208 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f11fc10a110 0x7f11fc1a7440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:23.881 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:23.882 7f120222b700 1 --2- 172.21.2.112:0/1939543208 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f11fc109280 0x7f11fc109af0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:23.881 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:23.882 7f120222b700 1 -- 172.21.2.112:0/1939543208 >> 172.21.2.112:0/1939543208 conn(0x7f11fc0bbf60 msgr2=0x7f11fc0b87c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:16:23.881 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:23.883 7f120222b700 1 -- 172.21.2.112:0/1939543208 shutdown_connections 2021-02-10T11:16:23.881 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:23.883 7f120222b700 1 -- 172.21.2.112:0/1939543208 wait complete. 2021-02-10T11:16:23.888 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000348571 2021-02-10T11:16:24.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:24.306 7f3c24383700 1 Processor -- start 2021-02-10T11:16:24.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:24.306 7f3c24383700 1 -- start start 2021-02-10T11:16:24.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:24.307 7f3c24383700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f3c1c1b0870 0x7f3c1c0f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:24.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:24.307 7f3c24383700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3c1c0efe00 0x7f3c1c0f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:24.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:24.307 7f3c24383700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c1c1b0c70 con 0x7f3c1c1b0470 2021-02-10T11:16:24.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:24.307 7f3c24383700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c1c0f0b50 con 0x7f3c1c1b0870 2021-02-10T11:16:24.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:24.307 7f3c24383700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c1c0d2fb0 con 0x7f3c1c0efe00 2021-02-10T11:16:24.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:24.307 7f3c2199e700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f3c1c1b0870 0x7f3c1c0f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:24.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:24.307 7f3c2219f700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7f3c1c1b0470 0x7f3c1c0ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:36932/0 (socket says 172.21.2.112:36932) 2021-02-10T11:16:24.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:24.307 7f3c2199e700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f3c1c1b0870 0x7f3c1c0f20e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3300/0 says I am v2:172.21.2.112:46194/0 (socket says 172.21.2.112:46194) 2021-02-10T11:16:24.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:24.307 7f3c2219f700 1 -- 172.21.2.112:0/659886240 learned_addr learned my addr 172.21.2.112:0/659886240 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:16:24.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:24.307 7f3c229a0700 1 --2- 172.21.2.112:0/659886240 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3c1c0efe00 0x7f3c1c0f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:24.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:24.307 7f3c2119d700 1 -- 172.21.2.112:0/659886240 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2224837195 0 0) 0x7f3c180010c0 con 0x7f3c1c1b0470 2021-02-10T11:16:24.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:24.307 7f3c2119d700 1 -- 172.21.2.112:0/659886240 --> v1:172.21.2.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c08002fe0 con 0x7f3c1c1b0470 2021-02-10T11:16:24.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:24.307 7f3c229a0700 1 -- 172.21.2.112:0/659886240 >> v2:172.21.2.120:3300/0 conn(0x7f3c1c1b0870 msgr2=0x7f3c1c0f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:16:24.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:24.307 7f3c229a0700 1 --2- 172.21.2.112:0/659886240 >> v2:172.21.2.120:3300/0 conn(0x7f3c1c1b0870 0x7f3c1c0f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:24.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:24.307 7f3c229a0700 1 -- 172.21.2.112:0/659886240 >> v1:172.21.2.120:6789/0 conn(0x7f3c1c1b0470 legacy=0x7f3c1c0ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:24.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:24.307 7f3c229a0700 1 -- 172.21.2.112:0/659886240 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c1c0d3330 con 0x7f3c1c0efe00 2021-02-10T11:16:24.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:24.308 7f3c229a0700 1 --2- 172.21.2.112:0/659886240 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3c1c0efe00 0x7f3c1c0f0200 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f3c14006e50 tx=0x7f3c14009e70).ready entity=mon.2 client_cookie=87ad69927528d290 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:16:24.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:24.308 7f3c2119d700 1 -- 172.21.2.112:0/659886240 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3c1400b510 con 0x7f3c1c0efe00 2021-02-10T11:16:24.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:24.308 7f3c2119d700 1 -- 172.21.2.112:0/659886240 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c14006790 con 0x7f3c1c0efe00 2021-02-10T11:16:24.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:24.308 7f3c24383700 1 -- 172.21.2.112:0/659886240 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3c1c0efe00 msgr2=0x7f3c1c0f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:24.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:24.308 7f3c24383700 1 --2- 172.21.2.112:0/659886240 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3c1c0efe00 0x7f3c1c0f0200 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f3c14006e50 tx=0x7f3c14009e70).stop 2021-02-10T11:16:24.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:24.308 7f3c2119d700 1 -- 172.21.2.112:0/659886240 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3c1400b940 con 0x7f3c1c0efe00 2021-02-10T11:16:24.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:24.309 7f3c24383700 1 -- 172.21.2.112:0/659886240 shutdown_connections 2021-02-10T11:16:24.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:24.309 7f3c24383700 1 --2- 172.21.2.112:0/659886240 >> v2:172.21.2.120:3300/0 conn(0x7f3c1c1b0870 0x7f3c1c0f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:24.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:24.309 7f3c24383700 1 --2- 172.21.2.112:0/659886240 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3c1c0efe00 0x7f3c1c0f0200 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:24.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:24.309 7f3c24383700 1 -- 172.21.2.112:0/659886240 >> 172.21.2.112:0/659886240 conn(0x7f3c1c0ce3e0 msgr2=0x7f3c1c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:16:24.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:24.309 7f3c24383700 1 -- 172.21.2.112:0/659886240 shutdown_connections 2021-02-10T11:16:24.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:24.309 7f3c24383700 1 -- 172.21.2.112:0/659886240 wait complete. 2021-02-10T11:16:24.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:24.324 7f3c24383700 1 Processor -- start 2021-02-10T11:16:24.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:24.324 7f3c24383700 1 -- start start 2021-02-10T11:16:24.327 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:24.324 7f3c24383700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3c1c0a0420 0x7f3c1c0a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:24.327 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:24.324 7f3c24383700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3c1c189d00 0x7f3c1c18a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:24.327 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:24.324 7f3c24383700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3c1c18a720 0x7f3c1c0b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:24.327 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:24.325 7f3c24383700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c1c189780 con 0x7f3c1c0a0420 2021-02-10T11:16:24.327 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:24.325 7f3c24383700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c1c18ab20 con 0x7f3c1c18a720 2021-02-10T11:16:24.327 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:24.325 7f3c24383700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c1c0a1140 con 0x7f3c1c189d00 2021-02-10T11:16:24.327 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:24.325 7f3c2219f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3c1c0a0420 0x7f3c1c0a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:24.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:24.325 7f3c2219f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3c1c0a0420 0x7f3c1c0a0820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:43922/0 (socket says 172.21.2.112:43922) 2021-02-10T11:16:24.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:24.325 7f3c2219f700 1 -- 172.21.2.112:0/974178242 learned_addr learned my addr 172.21.2.112:0/974178242 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:16:24.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:24.325 7f3c2219f700 1 -- 172.21.2.112:0/974178242 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3c1c189d00 msgr2=0x7f3c1c18a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:24.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:24.325 7f3c2199e700 1 --2- 172.21.2.112:0/974178242 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3c1c189d00 0x7f3c1c18a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:24.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:24.325 7f3c229a0700 1 --2- 172.21.2.112:0/974178242 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3c1c18a720 0x7f3c1c0b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:24.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:24.325 7f3c2219f700 1 --2- 172.21.2.112:0/974178242 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3c1c189d00 0x7f3c1c18a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:24.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:24.325 7f3c2219f700 1 -- 172.21.2.112:0/974178242 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3c1c18a720 msgr2=0x7f3c1c0b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:24.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:24.325 7f3c2219f700 1 --2- 172.21.2.112:0/974178242 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3c1c18a720 0x7f3c1c0b7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:24.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:24.325 7f3c2219f700 1 -- 172.21.2.112:0/974178242 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c1c0b8200 con 0x7f3c1c0a0420 2021-02-10T11:16:24.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:24.325 7f3c2199e700 1 --2- 172.21.2.112:0/974178242 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3c1c189d00 0x7f3c1c18a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:16:24.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:24.325 7f3c229a0700 1 --2- 172.21.2.112:0/974178242 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3c1c18a720 0x7f3c1c0b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:16:24.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:24.325 7f3c2219f700 1 --2- 172.21.2.112:0/974178242 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3c1c0a0420 0x7f3c1c0a0820 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 rx=0x7f3c18001450 tx=0x7f3c1800a930).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:16:24.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:24.326 7f3c12ffd700 1 -- 172.21.2.112:0/974178242 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3c1800ad20 con 0x7f3c1c0a0420 2021-02-10T11:16:24.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:24.326 7f3c12ffd700 1 -- 172.21.2.112:0/974178242 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c180075a0 con 0x7f3c1c0a0420 2021-02-10T11:16:24.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:24.326 7f3c12ffd700 1 -- 172.21.2.112:0/974178242 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3c18004510 con 0x7f3c1c0a0420 2021-02-10T11:16:24.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:24.326 7f3c24383700 1 -- 172.21.2.112:0/974178242 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c1c0b8200 con 0x7f3c1c0a0420 2021-02-10T11:16:24.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:24.326 7f3c24383700 1 -- 172.21.2.112:0/974178242 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c1c0b8790 con 0x7f3c1c0a0420 2021-02-10T11:16:24.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:24.326 7f3c12ffd700 1 -- 172.21.2.112:0/974178242 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3c18004320 con 0x7f3c1c0a0420 2021-02-10T11:16:24.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:24.326 7f3c12ffd700 1 --2- 172.21.2.112:0/974178242 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3c040186c0 0x7f3c0401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:24.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:24.326 7f3c12ffd700 1 -- 172.21.2.112:0/974178242 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f3c18032680 con 0x7f3c1c0a0420 2021-02-10T11:16:24.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:24.327 7f3c2199e700 1 --2- 172.21.2.112:0/974178242 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3c040186c0 0x7f3c0401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:24.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:24.327 7f3c2199e700 1 --2- 172.21.2.112:0/974178242 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3c040186c0 0x7f3c0401abc0 crc :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:16:24.352 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:24.354 7f3c24383700 1 -- 172.21.2.112:0/974178242 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c1c0a1360 con 0x7f3c1c0a0420 2021-02-10T11:16:24.354 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:24.357 7f3c12ffd700 1 -- 172.21.2.112:0/974178242 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3c18000e70 con 0x7f3c1c0a0420 2021-02-10T11:16:24.488 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:24.490 7f3c24383700 1 -- 172.21.2.112:0/974178242 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348571", "target": ["mgr", ""]}) v1 -- 0x7f3c1c0b9020 con 0x7f3c040186c0 2021-02-10T11:16:24.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:24.493 7f3c12ffd700 1 -- 172.21.2.112:0/974178242 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f3c0c006b10 con 0x7f3c040186c0 2021-02-10T11:16:24.491 INFO:teuthology.orchestra.run.gibba012.stdout:/volumes/_nogroup/subvolume_0000000000348571/52522976-64f4-4a25-9116-ea25e1ecd513 2021-02-10T11:16:24.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:24.495 7f3c24383700 1 -- 172.21.2.112:0/974178242 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3c040186c0 msgr2=0x7f3c0401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:24.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:24.495 7f3c24383700 1 --2- 172.21.2.112:0/974178242 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3c040186c0 0x7f3c0401abc0 crc :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:24.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:24.495 7f3c24383700 1 -- 172.21.2.112:0/974178242 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3c1c0a0420 msgr2=0x7f3c1c0a0820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:24.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:24.495 7f3c24383700 1 --2- 172.21.2.112:0/974178242 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3c1c0a0420 0x7f3c1c0a0820 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 rx=0x7f3c18001450 tx=0x7f3c1800a930).stop 2021-02-10T11:16:24.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:24.495 7f3c24383700 1 -- 172.21.2.112:0/974178242 shutdown_connections 2021-02-10T11:16:24.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:24.495 7f3c24383700 1 --2- 172.21.2.112:0/974178242 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3c040186c0 0x7f3c0401abc0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:24.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:24.495 7f3c24383700 1 --2- 172.21.2.112:0/974178242 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3c1c18a720 0x7f3c1c0b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:24.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:24.495 7f3c24383700 1 --2- 172.21.2.112:0/974178242 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3c1c189d00 0x7f3c1c18a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:24.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:24.495 7f3c24383700 1 --2- 172.21.2.112:0/974178242 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3c1c0a0420 0x7f3c1c0a0820 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:24.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:24.495 7f3c24383700 1 -- 172.21.2.112:0/974178242 >> 172.21.2.112:0/974178242 conn(0x7f3c1c094e10 msgr2=0x7f3c1c0a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:16:24.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:24.495 7f3c24383700 1 -- 172.21.2.112:0/974178242 shutdown_connections 2021-02-10T11:16:24.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:24.495 7f3c24383700 1 -- 172.21.2.112:0/974178242 wait complete. 2021-02-10T11:16:24.500 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000348571 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000348571/52522976-64f4-4a25-9116-ea25e1ecd513 2021-02-10T11:16:24.501 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:24.501 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:24.501 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:24.836 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:16:24.836 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:24.837 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348571/52522976-64f4-4a25-9116-ea25e1ecd513/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:25.218 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:16:25.218 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:16:25.218 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0284024 s, 36.9 MB/s 2021-02-10T11:16:25.219 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:25.219 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:25.219 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:25.553 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:16:25.554 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:25.554 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348571/52522976-64f4-4a25-9116-ea25e1ecd513/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:25.929 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:16:25.929 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:16:25.929 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0247743 s, 42.3 MB/s 2021-02-10T11:16:25.930 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:25.930 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:25.931 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:26.268 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:16:26.268 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:26.269 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348571/52522976-64f4-4a25-9116-ea25e1ecd513/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:26.644 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:16:26.644 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:16:26.644 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0247708 s, 42.3 MB/s 2021-02-10T11:16:26.646 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:26.646 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:26.646 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:26.981 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:16:26.981 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:26.982 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348571/52522976-64f4-4a25-9116-ea25e1ecd513/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:27.354 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:16:27.355 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:16:27.355 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0234106 s, 44.8 MB/s 2021-02-10T11:16:27.356 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:27.356 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:27.356 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:27.690 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:16:27.690 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:27.691 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348571/52522976-64f4-4a25-9116-ea25e1ecd513/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:28.065 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:16:28.065 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:16:28.065 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.024813 s, 42.3 MB/s 2021-02-10T11:16:28.066 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:28.066 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:28.066 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:28.401 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:16:28.401 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:28.401 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348571/52522976-64f4-4a25-9116-ea25e1ecd513/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:28.783 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:16:28.783 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:16:28.784 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0260067 s, 40.3 MB/s 2021-02-10T11:16:28.784 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:28.784 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:28.784 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:29.119 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:16:29.119 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:29.120 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348571/52522976-64f4-4a25-9116-ea25e1ecd513/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:29.492 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:16:29.492 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:16:29.492 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0258306 s, 40.6 MB/s 2021-02-10T11:16:29.493 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:29.494 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:29.494 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:29.830 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:16:29.831 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:29.831 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348571/52522976-64f4-4a25-9116-ea25e1ecd513/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:30.208 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:16:30.209 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:16:30.209 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.023309 s, 45.0 MB/s 2021-02-10T11:16:30.210 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:30.210 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:30.210 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:30.547 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:16:30.547 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:30.548 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348571/52522976-64f4-4a25-9116-ea25e1ecd513/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:30.924 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:16:30.925 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:16:30.925 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0252544 s, 41.5 MB/s 2021-02-10T11:16:30.926 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:30.926 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:30.926 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:31.263 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:16:31.264 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:31.264 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348571/52522976-64f4-4a25-9116-ea25e1ecd513/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:31.641 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:16:31.641 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:16:31.641 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0256353 s, 40.9 MB/s 2021-02-10T11:16:31.642 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000348572 2021-02-10T11:16:32.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.056 7fbb7b4ca700 1 Processor -- start 2021-02-10T11:16:32.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.056 7fbb7b4ca700 1 -- start start 2021-02-10T11:16:32.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.057 7fbb7b4ca700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbb741b0470 0x7fbb741b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:32.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.057 7fbb7b4ca700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb740f0b50 con 0x7fbb740f20e0 2021-02-10T11:16:32.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.057 7fbb7b4ca700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb740f24e0 con 0x7fbb740ef610 2021-02-10T11:16:32.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.057 7fbb7b4ca700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb740e5ea0 con 0x7fbb741b0470 2021-02-10T11:16:32.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.057 7fbb79ae7700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fbb740f20e0 0x7fbb740f0200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:43388/0 (socket says 172.21.2.112:43388) 2021-02-10T11:16:32.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.057 7fbb79ae7700 1 -- 172.21.2.112:0/4107593050 learned_addr learned my addr 172.21.2.112:0/4107593050 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:16:32.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.057 7fbb792e6700 1 --2- 172.21.2.112:0/4107593050 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbb741b0470 0x7fbb741b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:32.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.057 7fbb67fff700 1 -- 172.21.2.112:0/4107593050 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1524836637 0 0) 0x7fbb6c000c10 con 0x7fbb740f20e0 2021-02-10T11:16:32.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.057 7fbb67fff700 1 -- 172.21.2.112:0/4107593050 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb60002fe0 con 0x7fbb740f20e0 2021-02-10T11:16:32.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.057 7fbb67fff700 1 -- 172.21.2.112:0/4107593050 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3235280671 0 0) 0x7fbb68001370 con 0x7fbb740ef610 2021-02-10T11:16:32.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.057 7fbb67fff700 1 -- 172.21.2.112:0/4107593050 --> v1:172.21.2.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb60003210 con 0x7fbb740ef610 2021-02-10T11:16:32.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.058 7fbb792e6700 1 -- 172.21.2.112:0/4107593050 >> v1:172.21.2.120:6789/0 conn(0x7fbb740ef610 legacy=0x7fbb740efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:32.073 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.058 7fbb792e6700 1 -- 172.21.2.112:0/4107593050 >> v1:172.21.2.112:6789/0 conn(0x7fbb740f20e0 legacy=0x7fbb740f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:32.073 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.058 7fbb67fff700 1 -- 172.21.2.112:0/4107593050 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3071326764 0 0) 0x7fbb6c0010e0 con 0x7fbb740f20e0 2021-02-10T11:16:32.073 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.058 7fbb792e6700 1 -- 172.21.2.112:0/4107593050 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb740e52e0 con 0x7fbb741b0470 2021-02-10T11:16:32.073 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.058 7fbb792e6700 1 --2- 172.21.2.112:0/4107593050 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbb741b0470 0x7fbb741b0870 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7fbb70006650 tx=0x7fbb70009a80).ready entity=mon.2 client_cookie=abc19fd93ac6fd18 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:16:32.073 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.058 7fbb67fff700 1 -- 172.21.2.112:0/4107593050 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbb7000c0c0 con 0x7fbb741b0470 2021-02-10T11:16:32.073 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.058 7fbb67fff700 1 -- 172.21.2.112:0/4107593050 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb7000c340 con 0x7fbb741b0470 2021-02-10T11:16:32.073 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.058 7fbb67fff700 1 -- 172.21.2.112:0/4107593050 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbb7000c860 con 0x7fbb741b0470 2021-02-10T11:16:32.073 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.058 7fbb7b4ca700 1 -- 172.21.2.112:0/4107593050 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbb741b0470 msgr2=0x7fbb741b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:32.073 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.058 7fbb7b4ca700 1 --2- 172.21.2.112:0/4107593050 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbb741b0470 0x7fbb741b0870 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7fbb70006650 tx=0x7fbb70009a80).stop 2021-02-10T11:16:32.073 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.059 7fbb7b4ca700 1 -- 172.21.2.112:0/4107593050 shutdown_connections 2021-02-10T11:16:32.073 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.059 7fbb7b4ca700 1 --2- 172.21.2.112:0/4107593050 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbb741b0470 0x7fbb741b0870 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:32.073 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.059 7fbb7b4ca700 1 -- 172.21.2.112:0/4107593050 >> 172.21.2.112:0/4107593050 conn(0x7fbb740ce3e0 msgr2=0x7fbb740e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:16:32.073 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.059 7fbb7b4ca700 1 -- 172.21.2.112:0/4107593050 shutdown_connections 2021-02-10T11:16:32.074 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.059 7fbb7b4ca700 1 -- 172.21.2.112:0/4107593050 wait complete. 2021-02-10T11:16:32.074 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.073 7fbb7b4ca700 1 Processor -- start 2021-02-10T11:16:32.074 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.074 7fbb7b4ca700 1 -- start start 2021-02-10T11:16:32.075 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.074 7fbb7b4ca700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fbb74190920 0x7fbb740b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:32.075 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.074 7fbb7b4ca700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbb740b8200 0x7fbb740b8600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:32.075 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.074 7fbb7b4ca700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbb740c1d80 0x7fbb740c2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:32.075 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.074 7fbb7b4ca700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb74190d20 con 0x7fbb740b8200 2021-02-10T11:16:32.076 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.074 7fbb7b4ca700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb740b8c20 con 0x7fbb74190920 2021-02-10T11:16:32.076 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.074 7fbb7b4ca700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb740c27a0 con 0x7fbb740c1d80 2021-02-10T11:16:32.076 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.074 7fbb78ae5700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbb740b8200 0x7fbb740b8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:32.076 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.074 7fbb78ae5700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbb740b8200 0x7fbb740b8600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:43930/0 (socket says 172.21.2.112:43930) 2021-02-10T11:16:32.076 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.074 7fbb78ae5700 1 -- 172.21.2.112:0/4228197618 learned_addr learned my addr 172.21.2.112:0/4228197618 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:16:32.076 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.074 7fbb78ae5700 1 -- 172.21.2.112:0/4228197618 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbb740c1d80 msgr2=0x7fbb740c2180 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:16:32.076 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.074 7fbb78ae5700 1 --2- 172.21.2.112:0/4228197618 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbb740c1d80 0x7fbb740c2180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:16:32.076 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.074 7fbb78ae5700 1 -- 172.21.2.112:0/4228197618 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fbb74190920 msgr2=0x7fbb740b7be0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:16:32.076 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.074 7fbb78ae5700 1 --2- 172.21.2.112:0/4228197618 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fbb74190920 0x7fbb740b7be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:16:32.076 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.074 7fbb78ae5700 1 -- 172.21.2.112:0/4228197618 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb740c29c0 con 0x7fbb740b8200 2021-02-10T11:16:32.076 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.075 7fbb78ae5700 1 --2- 172.21.2.112:0/4228197618 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbb740b8200 0x7fbb740b8600 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 rx=0x7fbb68001330 tx=0x7fbb6800bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:16:32.076 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.075 7fbb65ffb700 1 -- 172.21.2.112:0/4228197618 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbb68000d30 con 0x7fbb740b8200 2021-02-10T11:16:32.077 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.075 7fbb65ffb700 1 -- 172.21.2.112:0/4228197618 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb68003030 con 0x7fbb740b8200 2021-02-10T11:16:32.077 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.075 7fbb65ffb700 1 -- 172.21.2.112:0/4228197618 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbb680015f0 con 0x7fbb740b8200 2021-02-10T11:16:32.077 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.075 7fbb7b4ca700 1 -- 172.21.2.112:0/4228197618 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb740c29c0 con 0x7fbb740b8200 2021-02-10T11:16:32.077 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.075 7fbb7b4ca700 1 -- 172.21.2.112:0/4228197618 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb740c29c0 con 0x7fbb740b8200 2021-02-10T11:16:32.077 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.075 7fbb65ffb700 1 -- 172.21.2.112:0/4228197618 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fbb68001370 con 0x7fbb740b8200 2021-02-10T11:16:32.077 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.075 7fbb65ffb700 1 --2- 172.21.2.112:0/4228197618 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fbb54018730 0x7fbb5401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:32.077 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.075 7fbb65ffb700 1 -- 172.21.2.112:0/4228197618 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fbb68032b00 con 0x7fbb740b8200 2021-02-10T11:16:32.077 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.076 7fbb792e6700 1 --2- 172.21.2.112:0/4228197618 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fbb54018730 0x7fbb5401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:32.077 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.076 7fbb792e6700 1 --2- 172.21.2.112:0/4228197618 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fbb54018730 0x7fbb5401abc0 crc :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:16:32.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.107 7fbb7b4ca700 1 -- 172.21.2.112:0/4228197618 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb74105ae0 con 0x7fbb740b8200 2021-02-10T11:16:32.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.109 7fbb65ffb700 1 -- 172.21.2.112:0/4228197618 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fbb6800be30 con 0x7fbb740b8200 2021-02-10T11:16:32.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.235 7fbb7b4ca700 1 -- 172.21.2.112:0/4228197618 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348572", "target": ["mgr", ""]}) v1 -- 0x7fbb74105ae0 con 0x7fbb54018730 2021-02-10T11:16:32.258 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.261 7fbb65ffb700 1 -- 172.21.2.112:0/4228197618 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fbb70000d10 con 0x7fbb54018730 2021-02-10T11:16:32.266 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.268 7fbb7b4ca700 1 -- 172.21.2.112:0/4228197618 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fbb54018730 msgr2=0x7fbb5401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:32.266 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.268 7fbb7b4ca700 1 --2- 172.21.2.112:0/4228197618 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fbb54018730 0x7fbb5401abc0 crc :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:32.266 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.268 7fbb7b4ca700 1 -- 172.21.2.112:0/4228197618 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbb740b8200 msgr2=0x7fbb740b8600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:32.266 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.268 7fbb7b4ca700 1 --2- 172.21.2.112:0/4228197618 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbb740b8200 0x7fbb740b8600 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 rx=0x7fbb68001330 tx=0x7fbb6800bcf0).stop 2021-02-10T11:16:32.266 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.269 7fbb7b4ca700 1 -- 172.21.2.112:0/4228197618 shutdown_connections 2021-02-10T11:16:32.266 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.269 7fbb7b4ca700 1 --2- 172.21.2.112:0/4228197618 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fbb54018730 0x7fbb5401abc0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:32.267 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.269 7fbb7b4ca700 1 --2- 172.21.2.112:0/4228197618 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbb740c1d80 0x7fbb740c2180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:16:32.267 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.269 7fbb7b4ca700 1 --2- 172.21.2.112:0/4228197618 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbb740b8200 0x7fbb740b8600 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:32.267 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.269 7fbb7b4ca700 1 --2- 172.21.2.112:0/4228197618 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fbb74190920 0x7fbb740b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:16:32.267 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.269 7fbb7b4ca700 1 -- 172.21.2.112:0/4228197618 >> 172.21.2.112:0/4228197618 conn(0x7fbb74094e10 msgr2=0x7fbb740a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:16:32.267 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.269 7fbb7b4ca700 1 -- 172.21.2.112:0/4228197618 shutdown_connections 2021-02-10T11:16:32.267 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.269 7fbb7b4ca700 1 -- 172.21.2.112:0/4228197618 wait complete. 2021-02-10T11:16:32.274 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000348572 2021-02-10T11:16:32.704 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.690 7fb07928c700 1 Processor -- start 2021-02-10T11:16:32.704 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.690 7fb07928c700 1 -- start start 2021-02-10T11:16:32.704 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.691 7fb07928c700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fb0741b0870 0x7fb0740f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:32.704 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.691 7fb07928c700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0741b0c70 con 0x7fb0740efe00 2021-02-10T11:16:32.704 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.691 7fb07928c700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0740f0b50 con 0x7fb0741b0470 2021-02-10T11:16:32.704 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.691 7fb07928c700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0740e5ea0 con 0x7fb0741b0870 2021-02-10T11:16:32.704 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.691 7fb07307d700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7fb0741b0470 0x7fb0740ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:36958/0 (socket says 172.21.2.112:36958) 2021-02-10T11:16:32.704 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.691 7fb07307d700 1 -- 172.21.2.112:0/3156719478 learned_addr learned my addr 172.21.2.112:0/3156719478 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:16:32.704 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.691 7fb07387e700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fb0740efe00 0x7fb0740f0200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:43402/0 (socket says 172.21.2.112:43402) 2021-02-10T11:16:32.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.691 7fb07287c700 1 --2- 172.21.2.112:0/3156719478 >> v2:172.21.2.120:3300/0 conn(0x7fb0741b0870 0x7fb0740f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:32.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.691 7fb07207b700 1 -- 172.21.2.112:0/3156719478 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3962857889 0 0) 0x7fb064000c10 con 0x7fb0740efe00 2021-02-10T11:16:32.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.691 7fb07207b700 1 -- 172.21.2.112:0/3156719478 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb05c002fe0 con 0x7fb0740efe00 2021-02-10T11:16:32.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.691 7fb07207b700 1 -- 172.21.2.112:0/3156719478 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4173261011 0 0) 0x7fb0680009b0 con 0x7fb0741b0470 2021-02-10T11:16:32.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.691 7fb07207b700 1 -- 172.21.2.112:0/3156719478 --> v1:172.21.2.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb05c0042c0 con 0x7fb0741b0470 2021-02-10T11:16:32.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.691 7fb07287c700 1 -- 172.21.2.112:0/3156719478 >> v1:172.21.2.120:6789/0 conn(0x7fb0741b0470 legacy=0x7fb0740ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:32.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.691 7fb07287c700 1 -- 172.21.2.112:0/3156719478 >> v1:172.21.2.112:6789/0 conn(0x7fb0740efe00 legacy=0x7fb0740f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:32.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.691 7fb07287c700 1 -- 172.21.2.112:0/3156719478 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0740e52e0 con 0x7fb0741b0870 2021-02-10T11:16:32.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.692 7fb07287c700 1 --2- 172.21.2.112:0/3156719478 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb0741b0870 0x7fb0740f20e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7fb06c0069d0 tx=0x7fb06c009dc0).ready entity=mon.1 client_cookie=6c7349ec58291b5a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:16:32.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.692 7fb07207b700 1 -- 172.21.2.112:0/3156719478 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb06c00b510 con 0x7fb0741b0870 2021-02-10T11:16:32.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.692 7fb07207b700 1 -- 172.21.2.112:0/3156719478 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb06c00b790 con 0x7fb0741b0870 2021-02-10T11:16:32.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.692 7fb07207b700 1 -- 172.21.2.112:0/3156719478 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb06c00bcb0 con 0x7fb0741b0870 2021-02-10T11:16:32.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.692 7fb07928c700 1 -- 172.21.2.112:0/3156719478 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb0741b0870 msgr2=0x7fb0740f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:32.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.692 7fb07928c700 1 --2- 172.21.2.112:0/3156719478 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb0741b0870 0x7fb0740f20e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7fb06c0069d0 tx=0x7fb06c009dc0).stop 2021-02-10T11:16:32.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.692 7fb07928c700 1 -- 172.21.2.112:0/3156719478 shutdown_connections 2021-02-10T11:16:32.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.692 7fb07928c700 1 --2- 172.21.2.112:0/3156719478 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb0741b0870 0x7fb0740f20e0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:32.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.692 7fb07928c700 1 -- 172.21.2.112:0/3156719478 >> 172.21.2.112:0/3156719478 conn(0x7fb0740ce3e0 msgr2=0x7fb0740e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:16:32.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.692 7fb07928c700 1 -- 172.21.2.112:0/3156719478 shutdown_connections 2021-02-10T11:16:32.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.692 7fb07928c700 1 -- 172.21.2.112:0/3156719478 wait complete. 2021-02-10T11:16:32.708 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.707 7fb07928c700 1 Processor -- start 2021-02-10T11:16:32.708 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.707 7fb07928c700 1 -- start start 2021-02-10T11:16:32.708 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.707 7fb07928c700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb0740a0420 0x7fb0740a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:32.708 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.707 7fb07928c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb074189d00 0x7fb07418a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:32.708 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.707 7fb07928c700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb07418a720 0x7fb0740b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:32.708 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.707 7fb07928c700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb074189780 con 0x7fb074189d00 2021-02-10T11:16:32.708 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.707 7fb07928c700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb07418ab20 con 0x7fb07418a720 2021-02-10T11:16:32.708 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.707 7fb07928c700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0740a1140 con 0x7fb0740a0420 2021-02-10T11:16:32.708 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.708 7fb07287c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb074189d00 0x7fb07418a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:32.708 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.708 7fb07287c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb074189d00 0x7fb07418a100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:43946/0 (socket says 172.21.2.112:43946) 2021-02-10T11:16:32.708 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.708 7fb07287c700 1 -- 172.21.2.112:0/2547480917 learned_addr learned my addr 172.21.2.112:0/2547480917 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:16:32.708 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.708 7fb07287c700 1 -- 172.21.2.112:0/2547480917 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb0740a0420 msgr2=0x7fb0740a0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:32.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.708 7fb07287c700 1 --2- 172.21.2.112:0/2547480917 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb0740a0420 0x7fb0740a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:16:32.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.708 7fb07287c700 1 -- 172.21.2.112:0/2547480917 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb07418a720 msgr2=0x7fb0740b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:32.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.708 7fb07387e700 1 --2- 172.21.2.112:0/2547480917 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb07418a720 0x7fb0740b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:32.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.708 7fb07287c700 1 --2- 172.21.2.112:0/2547480917 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb07418a720 0x7fb0740b7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:32.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.708 7fb07287c700 1 -- 172.21.2.112:0/2547480917 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0740b8200 con 0x7fb074189d00 2021-02-10T11:16:32.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.708 7fb07387e700 1 --2- 172.21.2.112:0/2547480917 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb07418a720 0x7fb0740b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:16:32.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.708 7fb07287c700 1 --2- 172.21.2.112:0/2547480917 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb074189d00 0x7fb07418a100 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 rx=0x7fb06c00b930 tx=0x7fb06c00bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:16:32.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.708 7fb05bfff700 1 -- 172.21.2.112:0/2547480917 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb06c006030 con 0x7fb074189d00 2021-02-10T11:16:32.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.708 7fb05bfff700 1 -- 172.21.2.112:0/2547480917 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb06c0063b0 con 0x7fb074189d00 2021-02-10T11:16:32.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.708 7fb07928c700 1 -- 172.21.2.112:0/2547480917 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0740b8200 con 0x7fb074189d00 2021-02-10T11:16:32.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.709 7fb05bfff700 1 -- 172.21.2.112:0/2547480917 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb06c005660 con 0x7fb074189d00 2021-02-10T11:16:32.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.709 7fb07928c700 1 -- 172.21.2.112:0/2547480917 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0740b8790 con 0x7fb074189d00 2021-02-10T11:16:32.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.709 7fb05bfff700 1 -- 172.21.2.112:0/2547480917 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb06c005660 con 0x7fb074189d00 2021-02-10T11:16:32.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.709 7fb05bfff700 1 --2- 172.21.2.112:0/2547480917 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb0540186c0 0x7fb05401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:32.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.709 7fb05bfff700 1 -- 172.21.2.112:0/2547480917 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fb06c032900 con 0x7fb074189d00 2021-02-10T11:16:32.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.710 7fb07307d700 1 --2- 172.21.2.112:0/2547480917 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb0540186c0 0x7fb05401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:32.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.710 7fb07307d700 1 --2- 172.21.2.112:0/2547480917 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb0540186c0 0x7fb05401abc0 crc :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:16:32.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.740 7fb07928c700 1 -- 172.21.2.112:0/2547480917 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb074105ae0 con 0x7fb074189d00 2021-02-10T11:16:32.740 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.743 7fb05bfff700 1 -- 172.21.2.112:0/2547480917 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb06c000d10 con 0x7fb074189d00 2021-02-10T11:16:32.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.873 7fb07928c700 1 -- 172.21.2.112:0/2547480917 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348572", "target": ["mgr", ""]}) v1 -- 0x7fb0740b9020 con 0x7fb0540186c0 2021-02-10T11:16:32.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.876 7fb05bfff700 1 -- 172.21.2.112:0/2547480917 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7fb068001530 con 0x7fb0540186c0 2021-02-10T11:16:32.877 INFO:teuthology.orchestra.run.gibba012.stdout:/volumes/_nogroup/subvolume_0000000000348572/9a416f49-b386-4cc3-9dc9-e165e70f21df 2021-02-10T11:16:32.879 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.882 7fb07928c700 1 -- 172.21.2.112:0/2547480917 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb0540186c0 msgr2=0x7fb05401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:32.879 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.882 7fb07928c700 1 --2- 172.21.2.112:0/2547480917 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb0540186c0 0x7fb05401abc0 crc :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:32.880 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.882 7fb07928c700 1 -- 172.21.2.112:0/2547480917 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb074189d00 msgr2=0x7fb07418a100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:32.880 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.882 7fb07928c700 1 --2- 172.21.2.112:0/2547480917 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb074189d00 0x7fb07418a100 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 rx=0x7fb06c00b930 tx=0x7fb06c00bcd0).stop 2021-02-10T11:16:32.880 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.882 7fb07928c700 1 -- 172.21.2.112:0/2547480917 shutdown_connections 2021-02-10T11:16:32.880 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.882 7fb07928c700 1 --2- 172.21.2.112:0/2547480917 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb0540186c0 0x7fb05401abc0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:32.880 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.882 7fb07928c700 1 --2- 172.21.2.112:0/2547480917 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb07418a720 0x7fb0740b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:32.880 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.882 7fb07928c700 1 --2- 172.21.2.112:0/2547480917 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb074189d00 0x7fb07418a100 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:32.880 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.882 7fb07928c700 1 --2- 172.21.2.112:0/2547480917 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb0740a0420 0x7fb0740a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:16:32.880 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.882 7fb07928c700 1 -- 172.21.2.112:0/2547480917 >> 172.21.2.112:0/2547480917 conn(0x7fb074094e10 msgr2=0x7fb0740a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:16:32.880 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.883 7fb07928c700 1 -- 172.21.2.112:0/2547480917 shutdown_connections 2021-02-10T11:16:32.881 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:32.883 7fb07928c700 1 -- 172.21.2.112:0/2547480917 wait complete. 2021-02-10T11:16:32.887 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000348572 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000348572/9a416f49-b386-4cc3-9dc9-e165e70f21df 2021-02-10T11:16:32.888 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:32.888 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:32.888 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:32.988 DEBUG:teuthology.orchestra.run.gibba012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:16:32.991 DEBUG:teuthology.orchestra.run.gibba020:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:16:33.223 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:16:33.224 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:33.224 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348572/9a416f49-b386-4cc3-9dc9-e165e70f21df/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:33.605 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:16:33.606 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:16:33.606 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.025518 s, 41.1 MB/s 2021-02-10T11:16:33.607 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:33.607 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:33.607 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:33.941 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:16:33.942 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:33.942 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348572/9a416f49-b386-4cc3-9dc9-e165e70f21df/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:34.323 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:16:34.323 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:16:34.323 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0301001 s, 34.8 MB/s 2021-02-10T11:16:34.324 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:34.324 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:34.324 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:34.659 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:16:34.660 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:34.660 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348572/9a416f49-b386-4cc3-9dc9-e165e70f21df/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:35.036 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:16:35.036 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:16:35.036 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0262227 s, 40.0 MB/s 2021-02-10T11:16:35.037 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:35.037 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:35.037 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:35.371 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:16:35.372 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:35.372 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348572/9a416f49-b386-4cc3-9dc9-e165e70f21df/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:35.748 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:16:35.748 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:16:35.748 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0263725 s, 39.8 MB/s 2021-02-10T11:16:35.749 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:35.749 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:35.749 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:36.083 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:16:36.083 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:36.083 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348572/9a416f49-b386-4cc3-9dc9-e165e70f21df/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:36.459 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:16:36.460 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:16:36.460 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0264086 s, 39.7 MB/s 2021-02-10T11:16:36.461 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:36.461 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:36.461 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:36.795 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:16:36.795 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:36.796 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348572/9a416f49-b386-4cc3-9dc9-e165e70f21df/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:37.168 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:16:37.169 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:16:37.169 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0230715 s, 45.4 MB/s 2021-02-10T11:16:37.170 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:37.170 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:37.170 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:37.504 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:16:37.504 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:37.505 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348572/9a416f49-b386-4cc3-9dc9-e165e70f21df/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:37.881 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:16:37.882 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:16:37.882 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0265169 s, 39.5 MB/s 2021-02-10T11:16:37.883 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:37.883 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:37.883 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:38.217 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:16:38.217 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:38.218 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348572/9a416f49-b386-4cc3-9dc9-e165e70f21df/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:38.591 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:16:38.591 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:16:38.591 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0231125 s, 45.4 MB/s 2021-02-10T11:16:38.592 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:38.592 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:38.592 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:38.927 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:16:38.927 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:38.928 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348572/9a416f49-b386-4cc3-9dc9-e165e70f21df/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:39.307 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:16:39.307 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:16:39.307 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0274063 s, 38.3 MB/s 2021-02-10T11:16:39.309 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:39.309 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:39.309 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:39.644 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:16:39.644 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:39.645 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348572/9a416f49-b386-4cc3-9dc9-e165e70f21df/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:40.016 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:16:40.017 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:16:40.017 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0230532 s, 45.5 MB/s 2021-02-10T11:16:40.018 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000348573 2021-02-10T11:16:40.448 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:40.433 7f6bed290700 1 Processor -- start 2021-02-10T11:16:40.449 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:40.433 7f6bed290700 1 -- start start 2021-02-10T11:16:40.449 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:40.434 7f6bed290700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6be81b0470 0x7f6be81b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:40.449 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:40.434 7f6bed290700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f6be80ef610 0x7f6be80efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:40.449 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:40.434 7f6bed290700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6be80f0b50 con 0x7f6be80f0030 2021-02-10T11:16:40.449 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:40.434 7f6bed290700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6be80e52e0 con 0x7f6be80ef610 2021-02-10T11:16:40.449 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:40.434 7f6bed290700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6be80e5490 con 0x7f6be81b0470 2021-02-10T11:16:40.449 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:40.435 7f6be687c700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f6be80ef610 0x7f6be80efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:40.449 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:40.435 7f6be707d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6be81b0470 0x7f6be81b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:40.449 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:40.435 7f6be707d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6be81b0470 0x7f6be81b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3301/0 says I am v2:172.21.2.112:43636/0 (socket says 172.21.2.112:43636) 2021-02-10T11:16:40.449 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:40.435 7f6be707d700 1 -- 172.21.2.112:0/1177780641 learned_addr learned my addr 172.21.2.112:0/1177780641 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:16:40.449 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:40.436 7f6be707d700 1 -- 172.21.2.112:0/1177780641 >> v2:172.21.2.120:3300/0 conn(0x7f6be80ef610 msgr2=0x7f6be80efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:16:40.450 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:40.436 7f6be707d700 1 --2- 172.21.2.112:0/1177780641 >> v2:172.21.2.120:3300/0 conn(0x7f6be80ef610 0x7f6be80efa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:40.450 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:40.436 7f6be607b700 1 -- 172.21.2.112:0/1177780641 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 722227992 0 0) 0x7f6be0001400 con 0x7f6be80f0030 2021-02-10T11:16:40.450 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:40.436 7f6be707d700 1 -- 172.21.2.112:0/1177780641 >> v1:172.21.2.120:6789/0 conn(0x7f6be80f0030 legacy=0x7f6be80f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:40.450 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:40.436 7f6be707d700 1 -- 172.21.2.112:0/1177780641 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6be80f0430 con 0x7f6be81b0470 2021-02-10T11:16:40.450 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:40.436 7f6be707d700 1 --2- 172.21.2.112:0/1177780641 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6be81b0470 0x7f6be81b0870 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f6bd0006700 tx=0x7f6bd0009b80).ready entity=mon.2 client_cookie=c01b567b852b87be server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:16:40.450 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:40.436 7f6be607b700 1 -- 172.21.2.112:0/1177780641 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6bd000c0c0 con 0x7f6be81b0470 2021-02-10T11:16:40.450 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:40.436 7f6be607b700 1 -- 172.21.2.112:0/1177780641 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bd000c340 con 0x7f6be81b0470 2021-02-10T11:16:40.450 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:40.436 7f6be607b700 1 -- 172.21.2.112:0/1177780641 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6bd000c860 con 0x7f6be81b0470 2021-02-10T11:16:40.451 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:40.436 7f6bed290700 1 -- 172.21.2.112:0/1177780641 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6be81b0470 msgr2=0x7f6be81b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:40.451 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:40.436 7f6bed290700 1 --2- 172.21.2.112:0/1177780641 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6be81b0470 0x7f6be81b0870 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f6bd0006700 tx=0x7f6bd0009b80).stop 2021-02-10T11:16:40.451 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:40.437 7f6bed290700 1 -- 172.21.2.112:0/1177780641 shutdown_connections 2021-02-10T11:16:40.451 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:40.437 7f6bed290700 1 --2- 172.21.2.112:0/1177780641 >> v2:172.21.2.120:3300/0 conn(0x7f6be80ef610 0x7f6be80efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:40.451 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:40.437 7f6bed290700 1 --2- 172.21.2.112:0/1177780641 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6be81b0470 0x7f6be81b0870 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:40.451 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:40.437 7f6bed290700 1 -- 172.21.2.112:0/1177780641 >> 172.21.2.112:0/1177780641 conn(0x7f6be80ce3e0 msgr2=0x7f6be80e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:16:40.451 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:40.437 7f6bed290700 1 -- 172.21.2.112:0/1177780641 shutdown_connections 2021-02-10T11:16:40.451 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:40.437 7f6bed290700 1 -- 172.21.2.112:0/1177780641 wait complete. 2021-02-10T11:16:40.453 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:40.452 7f6bed290700 1 Processor -- start 2021-02-10T11:16:40.453 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:40.452 7f6bed290700 1 -- start start 2021-02-10T11:16:40.453 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:40.452 7f6bed290700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6be8190920 0x7f6be80b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:40.454 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:40.452 7f6bed290700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6be80b8200 0x7f6be80b8600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:40.454 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:40.452 7f6bed290700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6be80c1d80 0x7f6be80c2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:40.454 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:40.452 7f6bed290700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6be8190d20 con 0x7f6be80b8200 2021-02-10T11:16:40.454 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:40.452 7f6bed290700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6be80b8c20 con 0x7f6be8190920 2021-02-10T11:16:40.454 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:40.452 7f6bed290700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6be80c27a0 con 0x7f6be80c1d80 2021-02-10T11:16:40.454 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:40.452 7f6be687c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6be80b8200 0x7f6be80b8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:40.454 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:40.452 7f6be687c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6be80b8200 0x7f6be80b8600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:43958/0 (socket says 172.21.2.112:43958) 2021-02-10T11:16:40.454 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:40.452 7f6be687c700 1 -- 172.21.2.112:0/3019832783 learned_addr learned my addr 172.21.2.112:0/3019832783 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:16:40.454 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:40.453 7f6be687c700 1 -- 172.21.2.112:0/3019832783 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6be80c1d80 msgr2=0x7f6be80c2180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:40.454 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:40.453 7f6be687c700 1 --2- 172.21.2.112:0/3019832783 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6be80c1d80 0x7f6be80c2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:16:40.454 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:40.453 7f6be687c700 1 -- 172.21.2.112:0/3019832783 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6be8190920 msgr2=0x7f6be80b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:40.454 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:40.453 7f6be707d700 1 --2- 172.21.2.112:0/3019832783 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6be8190920 0x7f6be80b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:40.454 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:40.453 7f6be687c700 1 --2- 172.21.2.112:0/3019832783 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6be8190920 0x7f6be80b7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:40.455 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:40.453 7f6be687c700 1 -- 172.21.2.112:0/3019832783 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6be80c29c0 con 0x7f6be80b8200 2021-02-10T11:16:40.455 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:40.453 7f6be707d700 1 --2- 172.21.2.112:0/3019832783 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6be8190920 0x7f6be80b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:16:40.455 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:40.453 7f6be687c700 1 --2- 172.21.2.112:0/3019832783 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6be80b8200 0x7f6be80b8600 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7f6bd800b560 tx=0x7f6bd800b900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:16:40.455 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:40.453 7f6bdffff700 1 -- 172.21.2.112:0/3019832783 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6bd800bcf0 con 0x7f6be80b8200 2021-02-10T11:16:40.455 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:40.453 7f6bdffff700 1 -- 172.21.2.112:0/3019832783 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bd8000c90 con 0x7f6be80b8200 2021-02-10T11:16:40.455 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:40.453 7f6bed290700 1 -- 172.21.2.112:0/3019832783 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6be80c29c0 con 0x7f6be80b8200 2021-02-10T11:16:40.455 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:40.453 7f6bdffff700 1 -- 172.21.2.112:0/3019832783 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6bd80022b0 con 0x7f6be80b8200 2021-02-10T11:16:40.455 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:40.453 7f6bed290700 1 -- 172.21.2.112:0/3019832783 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6be80c2f50 con 0x7f6be80b8200 2021-02-10T11:16:40.455 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:40.454 7f6bdffff700 1 -- 172.21.2.112:0/3019832783 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6bd80052e0 con 0x7f6be80b8200 2021-02-10T11:16:40.455 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:40.454 7f6bdffff700 1 --2- 172.21.2.112:0/3019832783 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6bcc0186c0 0x7f6bcc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:40.455 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:40.454 7f6bdffff700 1 -- 172.21.2.112:0/3019832783 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f6bd8032790 con 0x7f6be80b8200 2021-02-10T11:16:40.455 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:40.455 7f6be707d700 1 --2- 172.21.2.112:0/3019832783 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6bcc0186c0 0x7f6bcc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:40.456 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:40.455 7f6be707d700 1 --2- 172.21.2.112:0/3019832783 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6bcc0186c0 0x7f6bcc01abc0 crc :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:16:40.481 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:40.484 7f6bed290700 1 -- 172.21.2.112:0/3019832783 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6be8105ae0 con 0x7f6be80b8200 2021-02-10T11:16:40.484 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:40.486 7f6bdffff700 1 -- 172.21.2.112:0/3019832783 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6bd8000960 con 0x7f6be80b8200 2021-02-10T11:16:40.610 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:40.613 7f6bed290700 1 -- 172.21.2.112:0/3019832783 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348573", "target": ["mgr", ""]}) v1 -- 0x7f6be80c2f50 con 0x7f6bcc0186c0 2021-02-10T11:16:40.638 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:40.641 7f6bdffff700 1 -- 172.21.2.112:0/3019832783 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f6bd00009b0 con 0x7f6bcc0186c0 2021-02-10T11:16:40.644 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:40.646 7f6bed290700 1 -- 172.21.2.112:0/3019832783 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6bcc0186c0 msgr2=0x7f6bcc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:40.644 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:40.646 7f6bed290700 1 --2- 172.21.2.112:0/3019832783 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6bcc0186c0 0x7f6bcc01abc0 crc :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:40.644 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:40.646 7f6bed290700 1 -- 172.21.2.112:0/3019832783 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6be80b8200 msgr2=0x7f6be80b8600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:40.644 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:40.646 7f6bed290700 1 --2- 172.21.2.112:0/3019832783 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6be80b8200 0x7f6be80b8600 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7f6bd800b560 tx=0x7f6bd800b900).stop 2021-02-10T11:16:40.644 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:40.646 7f6bed290700 1 -- 172.21.2.112:0/3019832783 shutdown_connections 2021-02-10T11:16:40.644 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:40.646 7f6bed290700 1 --2- 172.21.2.112:0/3019832783 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6bcc0186c0 0x7f6bcc01abc0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:40.644 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:40.646 7f6bed290700 1 --2- 172.21.2.112:0/3019832783 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6be80c1d80 0x7f6be80c2180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:16:40.644 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:40.646 7f6bed290700 1 --2- 172.21.2.112:0/3019832783 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6be80b8200 0x7f6be80b8600 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:40.644 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:40.646 7f6bed290700 1 --2- 172.21.2.112:0/3019832783 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6be8190920 0x7f6be80b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:40.644 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:40.646 7f6bed290700 1 -- 172.21.2.112:0/3019832783 >> 172.21.2.112:0/3019832783 conn(0x7f6be8094e10 msgr2=0x7f6be80a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:16:40.645 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:40.647 7f6bed290700 1 -- 172.21.2.112:0/3019832783 shutdown_connections 2021-02-10T11:16:40.645 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:40.647 7f6bed290700 1 -- 172.21.2.112:0/3019832783 wait complete. 2021-02-10T11:16:40.652 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000348573 2021-02-10T11:16:41.083 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:41.068 7fef628fd700 1 Processor -- start 2021-02-10T11:16:41.083 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:41.068 7fef628fd700 1 -- start start 2021-02-10T11:16:41.083 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:41.068 7fef628fd700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fef5c1b0870 0x7fef5c0f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:41.083 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:41.068 7fef628fd700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef5c1b0c70 con 0x7fef5c0efe00 2021-02-10T11:16:41.083 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:41.068 7fef628fd700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef5c0f0b50 con 0x7fef5c1b0470 2021-02-10T11:16:41.083 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:41.068 7fef628fd700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef5c0e5ea0 con 0x7fef5c1b0870 2021-02-10T11:16:41.083 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:41.069 7fef60f1a700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fef5c0efe00 0x7fef5c0f0200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:43428/0 (socket says 172.21.2.112:43428) 2021-02-10T11:16:41.084 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:41.069 7fef60f1a700 1 -- 172.21.2.112:0/2550955821 learned_addr learned my addr 172.21.2.112:0/2550955821 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:16:41.084 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:41.069 7fef5affd700 1 -- 172.21.2.112:0/2550955821 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3096494135 0 0) 0x7fef54000f90 con 0x7fef5c0efe00 2021-02-10T11:16:41.084 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:41.069 7fef5b7fe700 1 --2- 172.21.2.112:0/2550955821 >> v2:172.21.2.120:3300/0 conn(0x7fef5c1b0870 0x7fef5c0f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:41.084 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:41.069 7fef5affd700 1 -- 172.21.2.112:0/2550955821 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef44002fe0 con 0x7fef5c0efe00 2021-02-10T11:16:41.084 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:41.069 7fef5affd700 1 -- 172.21.2.112:0/2550955821 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4120006144 0 0) 0x7fef54001500 con 0x7fef5c0efe00 2021-02-10T11:16:41.084 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:41.069 7fef5affd700 1 -- 172.21.2.112:0/2550955821 >> v2:172.21.2.120:3300/0 conn(0x7fef5c1b0870 msgr2=0x7fef5c0f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:16:41.084 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:41.069 7fef5affd700 1 --2- 172.21.2.112:0/2550955821 >> v2:172.21.2.120:3300/0 conn(0x7fef5c1b0870 0x7fef5c0f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:41.084 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:41.069 7fef5affd700 1 -- 172.21.2.112:0/2550955821 >> v1:172.21.2.120:6789/0 conn(0x7fef5c1b0470 legacy=0x7fef5c0ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:41.084 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:41.069 7fef5affd700 1 -- 172.21.2.112:0/2550955821 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef5c0e52e0 con 0x7fef5c0efe00 2021-02-10T11:16:41.084 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:41.070 7fef5affd700 1 -- 172.21.2.112:0/2550955821 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7fef54001a00 con 0x7fef5c0efe00 2021-02-10T11:16:41.084 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:41.070 7fef5affd700 1 -- 172.21.2.112:0/2550955821 >> v1:172.21.2.112:6789/0 conn(0x7fef5c0efe00 legacy=0x7fef5c0f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:41.084 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:41.070 7fef5affd700 1 --2- 172.21.2.112:0/2550955821 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fef44003a60 0x7fef44005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:41.085 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:41.070 7fef5affd700 1 --2- 172.21.2.112:0/2550955821 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fef44006790 0x7fef44008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:41.085 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:41.070 7fef5affd700 1 --2- 172.21.2.112:0/2550955821 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fef44009590 0x7fef4400bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:41.085 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:41.070 7fef5affd700 1 -- 172.21.2.112:0/2550955821 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef4400c300 con 0x7fef44003a60 2021-02-10T11:16:41.085 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:41.070 7fef5affd700 1 -- 172.21.2.112:0/2550955821 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef4400c520 con 0x7fef44009590 2021-02-10T11:16:41.085 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:41.070 7fef5affd700 1 -- 172.21.2.112:0/2550955821 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef4400c740 con 0x7fef44006790 2021-02-10T11:16:41.085 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:41.070 7fef5bfff700 1 --2- 172.21.2.112:0/2550955821 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fef44003a60 0x7fef44005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:41.085 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:41.070 7fef60f1a700 1 --2- 172.21.2.112:0/2550955821 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fef44009590 0x7fef4400bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:41.085 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:41.070 7fef5b7fe700 1 --2- 172.21.2.112:0/2550955821 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fef44006790 0x7fef44008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:41.085 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:41.070 7fef5bfff700 1 -- 172.21.2.112:0/2550955821 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fef44006790 msgr2=0x7fef44008c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:16:41.085 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:41.070 7fef5bfff700 1 --2- 172.21.2.112:0/2550955821 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fef44006790 0x7fef44008c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:41.085 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:41.070 7fef5bfff700 1 -- 172.21.2.112:0/2550955821 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fef44009590 msgr2=0x7fef4400bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:16:41.085 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:41.070 7fef5bfff700 1 --2- 172.21.2.112:0/2550955821 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fef44009590 0x7fef4400bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:41.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:41.070 7fef5bfff700 1 -- 172.21.2.112:0/2550955821 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef440509b0 con 0x7fef44003a60 2021-02-10T11:16:41.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:41.070 7fef60f1a700 1 -- 172.21.2.112:0/2550955821 reap_dead start 2021-02-10T11:16:41.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:41.070 7fef5bfff700 1 --2- 172.21.2.112:0/2550955821 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fef44003a60 0x7fef44005ef0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7fef4c006ab0 tx=0x7fef4c009b40).ready entity=mon.0 client_cookie=9223da2f9705c1ad server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:16:41.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:41.070 7fef5affd700 1 -- 172.21.2.112:0/2550955821 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fef4c00b2e0 con 0x7fef44003a60 2021-02-10T11:16:41.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:41.070 7fef5affd700 1 -- 172.21.2.112:0/2550955821 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef4c00b660 con 0x7fef44003a60 2021-02-10T11:16:41.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:41.070 7fef5affd700 1 -- 172.21.2.112:0/2550955821 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fef4c00bb40 con 0x7fef44003a60 2021-02-10T11:16:41.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:41.070 7fef628fd700 1 -- 172.21.2.112:0/2550955821 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fef44003a60 msgr2=0x7fef44005ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:41.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:41.070 7fef628fd700 1 --2- 172.21.2.112:0/2550955821 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fef44003a60 0x7fef44005ef0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7fef4c006ab0 tx=0x7fef4c009b40).stop 2021-02-10T11:16:41.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:41.071 7fef628fd700 1 -- 172.21.2.112:0/2550955821 shutdown_connections 2021-02-10T11:16:41.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:41.071 7fef628fd700 1 --2- 172.21.2.112:0/2550955821 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fef44003a60 0x7fef44005ef0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:41.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:41.071 7fef628fd700 1 -- 172.21.2.112:0/2550955821 >> 172.21.2.112:0/2550955821 conn(0x7fef5c0ce3e0 msgr2=0x7fef5c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:16:41.087 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:41.071 7fef628fd700 1 -- 172.21.2.112:0/2550955821 shutdown_connections 2021-02-10T11:16:41.087 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:41.071 7fef628fd700 1 -- 172.21.2.112:0/2550955821 wait complete. 2021-02-10T11:16:41.087 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:41.086 7fef628fd700 1 Processor -- start 2021-02-10T11:16:41.087 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:41.086 7fef628fd700 1 -- start start 2021-02-10T11:16:41.088 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:41.086 7fef628fd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fef5c09b930 0x7fef5c0b7c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:41.088 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:41.086 7fef628fd700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fef5c0b8410 0x7fef5c0b8810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:41.088 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:41.086 7fef628fd700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fef5c0c1f80 0x7fef5c0c2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:41.088 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:41.086 7fef628fd700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef5c0c1a00 con 0x7fef5c09b930 2021-02-10T11:16:41.088 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:41.086 7fef628fd700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef5c0b8e30 con 0x7fef5c0c1f80 2021-02-10T11:16:41.088 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:41.086 7fef628fd700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef5c0b9050 con 0x7fef5c0b8410 2021-02-10T11:16:41.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:41.086 7fef5bfff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fef5c09b930 0x7fef5c0b7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:41.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:41.086 7fef5bfff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fef5c09b930 0x7fef5c0b7c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:43978/0 (socket says 172.21.2.112:43978) 2021-02-10T11:16:41.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:41.086 7fef5bfff700 1 -- 172.21.2.112:0/973939670 learned_addr learned my addr 172.21.2.112:0/973939670 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:16:41.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:41.087 7fef5bfff700 1 -- 172.21.2.112:0/973939670 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fef5c0b8410 msgr2=0x7fef5c0b8810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:41.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:41.087 7fef5bfff700 1 --2- 172.21.2.112:0/973939670 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fef5c0b8410 0x7fef5c0b8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:16:41.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:41.087 7fef5bfff700 1 -- 172.21.2.112:0/973939670 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fef5c0c1f80 msgr2=0x7fef5c0c2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:41.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:41.087 7fef5bfff700 1 --2- 172.21.2.112:0/973939670 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fef5c0c1f80 0x7fef5c0c2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:16:41.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:41.087 7fef5bfff700 1 -- 172.21.2.112:0/973939670 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef5c0c29a0 con 0x7fef5c09b930 2021-02-10T11:16:41.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:41.087 7fef5bfff700 1 --2- 172.21.2.112:0/973939670 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fef5c09b930 0x7fef5c0b7c20 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0x7fef4c0093a0 tx=0x7fef4c00b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:16:41.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:41.087 7fef58ff9700 1 -- 172.21.2.112:0/973939670 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fef4c00bd30 con 0x7fef5c09b930 2021-02-10T11:16:41.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:41.087 7fef58ff9700 1 -- 172.21.2.112:0/973939670 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef4c000a10 con 0x7fef5c09b930 2021-02-10T11:16:41.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:41.087 7fef628fd700 1 -- 172.21.2.112:0/973939670 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef5c0c29a0 con 0x7fef5c09b930 2021-02-10T11:16:41.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:41.087 7fef58ff9700 1 -- 172.21.2.112:0/973939670 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fef4c0054a0 con 0x7fef5c09b930 2021-02-10T11:16:41.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:41.087 7fef628fd700 1 -- 172.21.2.112:0/973939670 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef5c0c2f30 con 0x7fef5c09b930 2021-02-10T11:16:41.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:41.088 7fef58ff9700 1 -- 172.21.2.112:0/973939670 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fef4c006750 con 0x7fef5c09b930 2021-02-10T11:16:41.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:41.088 7fef58ff9700 1 --2- 172.21.2.112:0/973939670 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fef3c018700 0x7fef3c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:41.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:41.088 7fef58ff9700 1 -- 172.21.2.112:0/973939670 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fef4c032610 con 0x7fef5c09b930 2021-02-10T11:16:41.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:41.088 7fef5b7fe700 1 --2- 172.21.2.112:0/973939670 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fef3c018700 0x7fef3c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:41.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:41.089 7fef5b7fe700 1 --2- 172.21.2.112:0/973939670 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fef3c018700 0x7fef3c01abc0 crc :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:16:41.115 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:41.118 7fef628fd700 1 -- 172.21.2.112:0/973939670 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef5c105b30 con 0x7fef5c09b930 2021-02-10T11:16:41.118 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:41.121 7fef58ff9700 1 -- 172.21.2.112:0/973939670 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fef4c01f4d0 con 0x7fef5c09b930 2021-02-10T11:16:41.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:41.249 7fef628fd700 1 -- 172.21.2.112:0/973939670 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348573", "target": ["mgr", ""]}) v1 -- 0x7fef5c0c2f30 con 0x7fef3c018700 2021-02-10T11:16:41.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:41.254 7fef58ff9700 1 -- 172.21.2.112:0/973939670 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7fef50006a20 con 0x7fef3c018700 2021-02-10T11:16:41.254 INFO:teuthology.orchestra.run.gibba012.stdout:/volumes/_nogroup/subvolume_0000000000348573/9cc1b412-40c9-4c22-b5d1-e1699d4fa778 2021-02-10T11:16:41.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:41.259 7fef628fd700 1 -- 172.21.2.112:0/973939670 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fef3c018700 msgr2=0x7fef3c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:41.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:41.259 7fef628fd700 1 --2- 172.21.2.112:0/973939670 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fef3c018700 0x7fef3c01abc0 crc :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:41.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:41.259 7fef628fd700 1 -- 172.21.2.112:0/973939670 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fef5c09b930 msgr2=0x7fef5c0b7c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:41.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:41.259 7fef628fd700 1 --2- 172.21.2.112:0/973939670 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fef5c09b930 0x7fef5c0b7c20 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0x7fef4c0093a0 tx=0x7fef4c00b940).stop 2021-02-10T11:16:41.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:41.259 7fef628fd700 1 -- 172.21.2.112:0/973939670 shutdown_connections 2021-02-10T11:16:41.257 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:41.259 7fef628fd700 1 --2- 172.21.2.112:0/973939670 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fef3c018700 0x7fef3c01abc0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:41.257 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:41.259 7fef628fd700 1 --2- 172.21.2.112:0/973939670 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fef5c0c1f80 0x7fef5c0c2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:16:41.257 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:41.259 7fef628fd700 1 --2- 172.21.2.112:0/973939670 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fef5c0b8410 0x7fef5c0b8810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:16:41.257 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:41.259 7fef628fd700 1 --2- 172.21.2.112:0/973939670 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fef5c09b930 0x7fef5c0b7c20 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:41.257 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:41.259 7fef628fd700 1 -- 172.21.2.112:0/973939670 >> 172.21.2.112:0/973939670 conn(0x7fef5c094e10 msgr2=0x7fef5c0a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:16:41.257 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:41.259 7fef628fd700 1 -- 172.21.2.112:0/973939670 shutdown_connections 2021-02-10T11:16:41.257 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:41.259 7fef628fd700 1 -- 172.21.2.112:0/973939670 wait complete. 2021-02-10T11:16:41.264 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000348573 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000348573/9cc1b412-40c9-4c22-b5d1-e1699d4fa778 2021-02-10T11:16:41.264 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:41.264 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:41.264 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:41.599 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:16:41.599 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:41.599 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348573/9cc1b412-40c9-4c22-b5d1-e1699d4fa778/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:41.977 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:16:41.978 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:16:41.978 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0227701 s, 46.1 MB/s 2021-02-10T11:16:41.979 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:41.979 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:41.979 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:42.314 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:16:42.315 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:42.315 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348573/9cc1b412-40c9-4c22-b5d1-e1699d4fa778/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:42.687 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:16:42.688 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:16:42.688 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0242095 s, 43.3 MB/s 2021-02-10T11:16:42.689 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:42.689 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:42.689 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:43.024 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:16:43.024 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:43.024 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348573/9cc1b412-40c9-4c22-b5d1-e1699d4fa778/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:43.398 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:16:43.399 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:16:43.399 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0243468 s, 43.1 MB/s 2021-02-10T11:16:43.400 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:43.400 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:43.400 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:43.735 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:16:43.735 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:43.735 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348573/9cc1b412-40c9-4c22-b5d1-e1699d4fa778/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:44.110 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:16:44.110 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:16:44.110 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0240709 s, 43.6 MB/s 2021-02-10T11:16:44.111 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:44.111 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:44.111 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:44.447 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:16:44.447 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:44.448 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348573/9cc1b412-40c9-4c22-b5d1-e1699d4fa778/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:44.820 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:16:44.820 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:16:44.820 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0245898 s, 42.6 MB/s 2021-02-10T11:16:44.822 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:44.822 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:44.822 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:45.157 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:16:45.157 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:45.158 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348573/9cc1b412-40c9-4c22-b5d1-e1699d4fa778/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:45.530 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:16:45.530 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:16:45.530 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0233505 s, 44.9 MB/s 2021-02-10T11:16:45.531 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:45.531 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:45.531 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:45.865 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:16:45.865 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:45.866 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348573/9cc1b412-40c9-4c22-b5d1-e1699d4fa778/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:46.240 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:16:46.240 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:16:46.240 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.024034 s, 43.6 MB/s 2021-02-10T11:16:46.242 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:46.242 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:46.242 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:46.577 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:16:46.577 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:46.577 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348573/9cc1b412-40c9-4c22-b5d1-e1699d4fa778/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:46.952 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:16:46.952 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:16:46.952 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0231861 s, 45.2 MB/s 2021-02-10T11:16:46.953 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:46.953 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:46.953 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:47.288 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:16:47.288 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:47.288 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348573/9cc1b412-40c9-4c22-b5d1-e1699d4fa778/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:47.664 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:16:47.664 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:16:47.664 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0263773 s, 39.8 MB/s 2021-02-10T11:16:47.665 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:47.665 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:47.665 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:47.999 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:16:47.999 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:47.999 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348573/9cc1b412-40c9-4c22-b5d1-e1699d4fa778/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:48.372 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:16:48.372 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:16:48.372 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0230184 s, 45.6 MB/s 2021-02-10T11:16:48.373 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000348574 2021-02-10T11:16:48.804 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:48.788 7ff7dc406700 1 Processor -- start 2021-02-10T11:16:48.804 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:48.788 7ff7dc406700 1 -- start start 2021-02-10T11:16:48.804 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:48.789 7ff7dc406700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff7d41b0470 0x7ff7d41b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:48.804 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:48.789 7ff7dc406700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7ff7d40ef610 0x7ff7d40efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:48.804 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:48.789 7ff7dc406700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7ff7d40f0030 0x7ff7d40f0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:48.804 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:48.789 7ff7dc406700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7d40e5ea0 con 0x7ff7d40f0030 2021-02-10T11:16:48.805 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:48.789 7ff7dc406700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7d40f0b50 con 0x7ff7d40ef610 2021-02-10T11:16:48.805 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:48.789 7ff7dc406700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7d40e52e0 con 0x7ff7d41b0470 2021-02-10T11:16:48.805 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:48.789 7ff7daa23700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7ff7d40f0030 0x7ff7d40f0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:48.805 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:48.789 7ff7daa23700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7ff7d40f0030 0x7ff7d40f0430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:43988/0 (socket says 172.21.2.112:43988) 2021-02-10T11:16:48.805 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:48.789 7ff7daa23700 1 -- 172.21.2.112:0/4119918552 learned_addr learned my addr 172.21.2.112:0/4119918552 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:16:48.805 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:48.789 7ff7daa23700 1 -- 172.21.2.112:0/4119918552 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff7d41b0470 msgr2=0x7ff7d41b0870 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:16:48.805 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:48.789 7ff7daa23700 1 --2- 172.21.2.112:0/4119918552 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff7d41b0470 0x7ff7d41b0870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:16:48.805 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:48.789 7ff7daa23700 1 -- 172.21.2.112:0/4119918552 >> v2:172.21.2.120:3300/0 conn(0x7ff7d40ef610 msgr2=0x7ff7d40efa10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:16:48.805 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:48.789 7ff7daa23700 1 --2- 172.21.2.112:0/4119918552 >> v2:172.21.2.120:3300/0 conn(0x7ff7d40ef610 0x7ff7d40efa10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:16:48.805 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:48.789 7ff7daa23700 1 -- 172.21.2.112:0/4119918552 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7d40f10d0 con 0x7ff7d40f0030 2021-02-10T11:16:48.805 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:48.789 7ff7daa23700 1 --2- 172.21.2.112:0/4119918552 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff7d40f0030 0x7ff7d40f0430 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7ff7cc006f00 tx=0x7ff7cc009f10).ready entity=mon.0 client_cookie=b9227472258f0bb1 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:16:48.805 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:48.789 7ff7d9220700 1 -- 172.21.2.112:0/4119918552 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff7cc00b3b0 con 0x7ff7d40f0030 2021-02-10T11:16:48.805 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:48.789 7ff7d9220700 1 -- 172.21.2.112:0/4119918552 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7cc00b630 con 0x7ff7d40f0030 2021-02-10T11:16:48.806 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:48.789 7ff7d9220700 1 -- 172.21.2.112:0/4119918552 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff7cc00bb50 con 0x7ff7d40f0030 2021-02-10T11:16:48.806 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:48.789 7ff7dc406700 1 -- 172.21.2.112:0/4119918552 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff7d40f0030 msgr2=0x7ff7d40f0430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:48.806 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:48.789 7ff7dc406700 1 --2- 172.21.2.112:0/4119918552 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff7d40f0030 0x7ff7d40f0430 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7ff7cc006f00 tx=0x7ff7cc009f10).stop 2021-02-10T11:16:48.806 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:48.790 7ff7dc406700 1 -- 172.21.2.112:0/4119918552 shutdown_connections 2021-02-10T11:16:48.806 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:48.790 7ff7dc406700 1 --2- 172.21.2.112:0/4119918552 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff7d40f0030 0x7ff7d40f0430 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:48.806 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:48.790 7ff7dc406700 1 --2- 172.21.2.112:0/4119918552 >> v2:172.21.2.120:3300/0 conn(0x7ff7d40ef610 0x7ff7d40efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:16:48.806 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:48.790 7ff7dc406700 1 --2- 172.21.2.112:0/4119918552 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff7d41b0470 0x7ff7d41b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:16:48.806 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:48.790 7ff7dc406700 1 -- 172.21.2.112:0/4119918552 >> 172.21.2.112:0/4119918552 conn(0x7ff7d40ce3e0 msgr2=0x7ff7d40e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:16:48.806 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:48.790 7ff7dc406700 1 -- 172.21.2.112:0/4119918552 shutdown_connections 2021-02-10T11:16:48.806 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:48.790 7ff7dc406700 1 -- 172.21.2.112:0/4119918552 wait complete. 2021-02-10T11:16:48.806 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:48.805 7ff7dc406700 1 Processor -- start 2021-02-10T11:16:48.806 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:48.805 7ff7dc406700 1 -- start start 2021-02-10T11:16:48.807 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:48.805 7ff7dc406700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff7d409e5d0 0x7ff7d40c2be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:48.807 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:48.805 7ff7dc406700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff7d409e9d0 0x7ff7d40c3200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:48.807 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:48.805 7ff7dc406700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff7d40a9c80 0x7ff7d40aa080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:48.807 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:48.805 7ff7dc406700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7d4095040 con 0x7ff7d40a9c80 2021-02-10T11:16:48.807 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:48.805 7ff7dc406700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7d40c3820 con 0x7ff7d409e5d0 2021-02-10T11:16:48.807 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:48.805 7ff7dc406700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7d40aa6a0 con 0x7ff7d409e9d0 2021-02-10T11:16:48.807 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:48.805 7ff7daa23700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff7d40a9c80 0x7ff7d40aa080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:48.807 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:48.805 7ff7daa23700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff7d40a9c80 0x7ff7d40aa080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:43990/0 (socket says 172.21.2.112:43990) 2021-02-10T11:16:48.807 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:48.805 7ff7daa23700 1 -- 172.21.2.112:0/3938465457 learned_addr learned my addr 172.21.2.112:0/3938465457 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:16:48.807 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:48.805 7ff7d9a21700 1 --2- 172.21.2.112:0/3938465457 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff7d409e9d0 0x7ff7d40c3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:48.807 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:48.805 7ff7da222700 1 --2- 172.21.2.112:0/3938465457 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff7d409e5d0 0x7ff7d40c2be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:48.807 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:48.805 7ff7daa23700 1 -- 172.21.2.112:0/3938465457 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff7d409e9d0 msgr2=0x7ff7d40c3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:48.807 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:48.805 7ff7daa23700 1 --2- 172.21.2.112:0/3938465457 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff7d409e9d0 0x7ff7d40c3200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:48.808 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:48.806 7ff7daa23700 1 -- 172.21.2.112:0/3938465457 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff7d409e5d0 msgr2=0x7ff7d40c2be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:48.808 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:48.806 7ff7daa23700 1 --2- 172.21.2.112:0/3938465457 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff7d409e5d0 0x7ff7d40c2be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:48.808 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:48.806 7ff7daa23700 1 -- 172.21.2.112:0/3938465457 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7d40aa8c0 con 0x7ff7d40a9c80 2021-02-10T11:16:48.808 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:48.806 7ff7daa23700 1 --2- 172.21.2.112:0/3938465457 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff7d40a9c80 0x7ff7d40aa080 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0x7ff7cc00b8e0 tx=0x7ff7cc00bc80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:16:48.808 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:48.806 7ff7c6ffd700 1 -- 172.21.2.112:0/3938465457 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff7cc00bdc0 con 0x7ff7d40a9c80 2021-02-10T11:16:48.808 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:48.806 7ff7c6ffd700 1 -- 172.21.2.112:0/3938465457 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7cc00c2f0 con 0x7ff7d40a9c80 2021-02-10T11:16:48.808 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:48.806 7ff7c6ffd700 1 -- 172.21.2.112:0/3938465457 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff7cc00c6e0 con 0x7ff7d40a9c80 2021-02-10T11:16:48.808 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:48.806 7ff7dc406700 1 -- 172.21.2.112:0/3938465457 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7d40aa8c0 con 0x7ff7d40a9c80 2021-02-10T11:16:48.808 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:48.806 7ff7dc406700 1 -- 172.21.2.112:0/3938465457 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7d40aa8c0 con 0x7ff7d40a9c80 2021-02-10T11:16:48.808 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:48.807 7ff7c6ffd700 1 -- 172.21.2.112:0/3938465457 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff7cc005820 con 0x7ff7d40a9c80 2021-02-10T11:16:48.808 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:48.807 7ff7c6ffd700 1 --2- 172.21.2.112:0/3938465457 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff7bc0186c0 0x7ff7bc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:48.808 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:48.807 7ff7c6ffd700 1 -- 172.21.2.112:0/3938465457 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7ff7cc0328a0 con 0x7ff7d40a9c80 2021-02-10T11:16:48.808 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:48.807 7ff7da222700 1 --2- 172.21.2.112:0/3938465457 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff7bc0186c0 0x7ff7bc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:48.809 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:48.807 7ff7da222700 1 --2- 172.21.2.112:0/3938465457 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff7bc0186c0 0x7ff7bc01abc0 crc :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:16:48.836 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:48.839 7ff7dc406700 1 -- 172.21.2.112:0/3938465457 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7d40edd10 con 0x7ff7d40a9c80 2021-02-10T11:16:48.838 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:48.841 7ff7c6ffd700 1 -- 172.21.2.112:0/3938465457 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff7cc000f90 con 0x7ff7d40a9c80 2021-02-10T11:16:48.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:48.969 7ff7dc406700 1 -- 172.21.2.112:0/3938465457 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348574", "target": ["mgr", ""]}) v1 -- 0x7ff7d40e52e0 con 0x7ff7bc0186c0 2021-02-10T11:16:48.993 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:48.996 7ff7c6ffd700 1 -- 172.21.2.112:0/3938465457 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7ff7c80009b0 con 0x7ff7bc0186c0 2021-02-10T11:16:48.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:49.002 7ff7dc406700 1 -- 172.21.2.112:0/3938465457 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff7bc0186c0 msgr2=0x7ff7bc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:48.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:49.002 7ff7dc406700 1 --2- 172.21.2.112:0/3938465457 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff7bc0186c0 0x7ff7bc01abc0 crc :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:49.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:49.002 7ff7dc406700 1 -- 172.21.2.112:0/3938465457 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff7d40a9c80 msgr2=0x7ff7d40aa080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:49.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:49.002 7ff7dc406700 1 --2- 172.21.2.112:0/3938465457 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff7d40a9c80 0x7ff7d40aa080 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0x7ff7cc00b8e0 tx=0x7ff7cc00bc80).stop 2021-02-10T11:16:49.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:49.002 7ff7dc406700 1 -- 172.21.2.112:0/3938465457 shutdown_connections 2021-02-10T11:16:49.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:49.002 7ff7dc406700 1 --2- 172.21.2.112:0/3938465457 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff7bc0186c0 0x7ff7bc01abc0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:49.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:49.002 7ff7dc406700 1 --2- 172.21.2.112:0/3938465457 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff7d40a9c80 0x7ff7d40aa080 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:49.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:49.002 7ff7dc406700 1 --2- 172.21.2.112:0/3938465457 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff7d409e9d0 0x7ff7d40c3200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:49.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:49.002 7ff7dc406700 1 --2- 172.21.2.112:0/3938465457 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff7d409e5d0 0x7ff7d40c2be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:49.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:49.002 7ff7dc406700 1 -- 172.21.2.112:0/3938465457 >> 172.21.2.112:0/3938465457 conn(0x7ff7d409f630 msgr2=0x7ff7d40a0130 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:16:49.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:49.002 7ff7dc406700 1 -- 172.21.2.112:0/3938465457 shutdown_connections 2021-02-10T11:16:49.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:49.002 7ff7dc406700 1 -- 172.21.2.112:0/3938465457 wait complete. 2021-02-10T11:16:49.007 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000348574 2021-02-10T11:16:49.438 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:49.424 7f06e1c4a700 1 Processor -- start 2021-02-10T11:16:49.438 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:49.424 7f06e1c4a700 1 -- start start 2021-02-10T11:16:49.438 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:49.424 7f06e1c4a700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f06dc1b0470 0x7f06dc1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:49.438 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:49.424 7f06e1c4a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f06dc0f20e0 0x7f06dc0f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:49.438 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:49.424 7f06e1c4a700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06dc0f0b50 con 0x7f06dc0ef610 2021-02-10T11:16:49.438 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:49.424 7f06e1c4a700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06dc0f0820 con 0x7f06dc1b0470 2021-02-10T11:16:49.438 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:49.424 7f06e1c4a700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06dc0f24e0 con 0x7f06dc0f20e0 2021-02-10T11:16:49.439 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:49.425 7f06db7fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f06dc1b0470 0x7f06dc1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:49.439 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:49.425 7f06db7fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f06dc1b0470 0x7f06dc1b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:43998/0 (socket says 172.21.2.112:43998) 2021-02-10T11:16:49.439 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:49.425 7f06db7fe700 1 -- 172.21.2.112:0/2805630584 learned_addr learned my addr 172.21.2.112:0/2805630584 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:16:49.439 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:49.425 7f06db7fe700 1 -- 172.21.2.112:0/2805630584 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f06dc0f20e0 msgr2=0x7f06dc0f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:16:49.439 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:49.425 7f06db7fe700 1 --2- 172.21.2.112:0/2805630584 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f06dc0f20e0 0x7f06dc0f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:16:49.439 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:49.425 7f06db7fe700 1 -- 172.21.2.112:0/2805630584 >> v1:172.21.2.120:6789/0 conn(0x7f06dc0ef610 legacy=0x7f06dc0efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:16:49.439 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:49.425 7f06db7fe700 1 -- 172.21.2.112:0/2805630584 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06dc0e52e0 con 0x7f06dc1b0470 2021-02-10T11:16:49.439 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:49.425 7f06db7fe700 1 --2- 172.21.2.112:0/2805630584 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f06dc1b0470 0x7f06dc1b0870 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7f06cc006700 tx=0x7f06cc009b80).ready entity=mon.0 client_cookie=46212af93e952582 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:16:49.439 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:49.425 7f06da7fc700 1 -- 172.21.2.112:0/2805630584 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f06cc00c0c0 con 0x7f06dc1b0470 2021-02-10T11:16:49.439 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:49.425 7f06da7fc700 1 -- 172.21.2.112:0/2805630584 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06cc00c340 con 0x7f06dc1b0470 2021-02-10T11:16:49.439 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:49.425 7f06da7fc700 1 -- 172.21.2.112:0/2805630584 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f06cc00c860 con 0x7f06dc1b0470 2021-02-10T11:16:49.439 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:49.425 7f06e1c4a700 1 -- 172.21.2.112:0/2805630584 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f06dc1b0470 msgr2=0x7f06dc1b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:49.439 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:49.425 7f06e1c4a700 1 --2- 172.21.2.112:0/2805630584 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f06dc1b0470 0x7f06dc1b0870 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7f06cc006700 tx=0x7f06cc009b80).stop 2021-02-10T11:16:49.440 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:49.426 7f06e1c4a700 1 -- 172.21.2.112:0/2805630584 shutdown_connections 2021-02-10T11:16:49.440 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:49.426 7f06e1c4a700 1 --2- 172.21.2.112:0/2805630584 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f06dc0f20e0 0x7f06dc0f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:16:49.440 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:49.426 7f06e1c4a700 1 --2- 172.21.2.112:0/2805630584 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f06dc1b0470 0x7f06dc1b0870 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:49.440 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:49.426 7f06e1c4a700 1 -- 172.21.2.112:0/2805630584 >> 172.21.2.112:0/2805630584 conn(0x7f06dc0ce3e0 msgr2=0x7f06dc0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:16:49.440 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:49.426 7f06e1c4a700 1 -- 172.21.2.112:0/2805630584 shutdown_connections 2021-02-10T11:16:49.440 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:49.426 7f06e1c4a700 1 -- 172.21.2.112:0/2805630584 wait complete. 2021-02-10T11:16:49.441 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:49.441 7f06e1c4a700 1 Processor -- start 2021-02-10T11:16:49.442 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:49.441 7f06e1c4a700 1 -- start start 2021-02-10T11:16:49.442 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:49.442 7f06e1c4a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f06dc0a6c90 0x7f06dc1924c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:49.442 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:49.442 7f06e1c4a700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f06dc192ae0 0x7f06dc192ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:49.442 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:49.442 7f06e1c4a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f06dc0919e0 0x7f06dc091de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:49.442 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:49.442 7f06e1c4a700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06dc19d2e0 con 0x7f06dc0a6c90 2021-02-10T11:16:49.442 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:49.442 7f06e1c4a700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06dc0914d0 con 0x7f06dc192ae0 2021-02-10T11:16:49.442 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:49.442 7f06e1c4a700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06dc0a6820 con 0x7f06dc0919e0 2021-02-10T11:16:49.442 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:49.442 7f06db7fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f06dc0a6c90 0x7f06dc1924c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:49.442 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:49.442 7f06db7fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f06dc0a6c90 0x7f06dc1924c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:44000/0 (socket says 172.21.2.112:44000) 2021-02-10T11:16:49.442 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:49.442 7f06db7fe700 1 -- 172.21.2.112:0/154405559 learned_addr learned my addr 172.21.2.112:0/154405559 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:16:49.442 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:49.442 7f06db7fe700 1 -- 172.21.2.112:0/154405559 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f06dc0919e0 msgr2=0x7f06dc091de0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:16:49.443 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:49.442 7f06db7fe700 1 --2- 172.21.2.112:0/154405559 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f06dc0919e0 0x7f06dc091de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:16:49.443 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:49.442 7f06db7fe700 1 -- 172.21.2.112:0/154405559 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f06dc192ae0 msgr2=0x7f06dc192ee0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:16:49.443 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:49.442 7f06db7fe700 1 --2- 172.21.2.112:0/154405559 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f06dc192ae0 0x7f06dc192ee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:16:49.443 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:49.442 7f06db7fe700 1 -- 172.21.2.112:0/154405559 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06dc092400 con 0x7f06dc0a6c90 2021-02-10T11:16:49.443 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:49.442 7f06db7fe700 1 --2- 172.21.2.112:0/154405559 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f06dc0a6c90 0x7f06dc1924c0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 rx=0x7f06cc00b650 tx=0x7f06cc00b9f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:16:49.443 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:49.442 7f06d3fff700 1 -- 172.21.2.112:0/154405559 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f06cc00bde0 con 0x7f06dc0a6c90 2021-02-10T11:16:49.443 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:49.442 7f06d3fff700 1 -- 172.21.2.112:0/154405559 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06cc0064d0 con 0x7f06dc0a6c90 2021-02-10T11:16:49.443 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:49.442 7f06d3fff700 1 -- 172.21.2.112:0/154405559 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f06cc0022b0 con 0x7f06dc0a6c90 2021-02-10T11:16:49.443 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:49.442 7f06e1c4a700 1 -- 172.21.2.112:0/154405559 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06dc092400 con 0x7f06dc0a6c90 2021-02-10T11:16:49.443 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:49.443 7f06e1c4a700 1 -- 172.21.2.112:0/154405559 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06dc092990 con 0x7f06dc0a6c90 2021-02-10T11:16:49.443 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:49.443 7f06d3fff700 1 -- 172.21.2.112:0/154405559 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f06cc0052e0 con 0x7f06dc0a6c90 2021-02-10T11:16:49.443 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:49.443 7f06d3fff700 1 --2- 172.21.2.112:0/154405559 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f06c00186c0 0x7f06c001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:49.443 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:49.443 7f06d3fff700 1 -- 172.21.2.112:0/154405559 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f06cc0323c0 con 0x7f06dc0a6c90 2021-02-10T11:16:49.444 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:49.444 7f06daffd700 1 --2- 172.21.2.112:0/154405559 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f06c00186c0 0x7f06c001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:49.444 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:49.444 7f06daffd700 1 --2- 172.21.2.112:0/154405559 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f06c00186c0 0x7f06c001abc0 crc :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:16:49.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:49.474 7f06e1c4a700 1 -- 172.21.2.112:0/154405559 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06dc0a67f0 con 0x7f06dc0a6c90 2021-02-10T11:16:49.473 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:49.476 7f06d3fff700 1 -- 172.21.2.112:0/154405559 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f06cc0244e0 con 0x7f06dc0a6c90 2021-02-10T11:16:49.602 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:49.605 7f06e1c4a700 1 -- 172.21.2.112:0/154405559 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348574", "target": ["mgr", ""]}) v1 -- 0x7f06dc092990 con 0x7f06c00186c0 2021-02-10T11:16:49.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:49.608 7f06d3fff700 1 -- 172.21.2.112:0/154405559 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f06c4006970 con 0x7f06c00186c0 2021-02-10T11:16:49.605 INFO:teuthology.orchestra.run.gibba012.stdout:/volumes/_nogroup/subvolume_0000000000348574/465544fd-51e0-42c2-b788-2bd8ca512916 2021-02-10T11:16:49.607 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:49.610 7f06e1c4a700 1 -- 172.21.2.112:0/154405559 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f06c00186c0 msgr2=0x7f06c001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:49.607 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:49.610 7f06e1c4a700 1 --2- 172.21.2.112:0/154405559 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f06c00186c0 0x7f06c001abc0 crc :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:49.607 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:49.610 7f06e1c4a700 1 -- 172.21.2.112:0/154405559 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f06dc0a6c90 msgr2=0x7f06dc1924c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:49.608 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:49.610 7f06e1c4a700 1 --2- 172.21.2.112:0/154405559 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f06dc0a6c90 0x7f06dc1924c0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 rx=0x7f06cc00b650 tx=0x7f06cc00b9f0).stop 2021-02-10T11:16:49.608 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:49.610 7f06e1c4a700 1 -- 172.21.2.112:0/154405559 shutdown_connections 2021-02-10T11:16:49.608 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:49.610 7f06e1c4a700 1 --2- 172.21.2.112:0/154405559 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f06c00186c0 0x7f06c001abc0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:49.608 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:49.610 7f06e1c4a700 1 --2- 172.21.2.112:0/154405559 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f06dc192ae0 0x7f06dc192ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:16:49.608 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:49.610 7f06e1c4a700 1 --2- 172.21.2.112:0/154405559 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f06dc0919e0 0x7f06dc091de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:16:49.608 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:49.610 7f06e1c4a700 1 --2- 172.21.2.112:0/154405559 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f06dc0a6c90 0x7f06dc1924c0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:49.608 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:49.610 7f06e1c4a700 1 -- 172.21.2.112:0/154405559 >> 172.21.2.112:0/154405559 conn(0x7f06dc0ac550 msgr2=0x7f06dc0be1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:16:49.608 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:49.610 7f06e1c4a700 1 -- 172.21.2.112:0/154405559 shutdown_connections 2021-02-10T11:16:49.608 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:49.610 7f06e1c4a700 1 -- 172.21.2.112:0/154405559 wait complete. 2021-02-10T11:16:49.615 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000348574 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000348574/465544fd-51e0-42c2-b788-2bd8ca512916 2021-02-10T11:16:49.615 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:49.615 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:49.616 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:49.950 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:16:49.950 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:49.950 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348574/465544fd-51e0-42c2-b788-2bd8ca512916/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:50.329 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:16:50.329 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:16:50.329 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0243321 s, 43.1 MB/s 2021-02-10T11:16:50.330 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:50.330 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:50.330 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:50.666 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:16:50.666 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:50.666 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348574/465544fd-51e0-42c2-b788-2bd8ca512916/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:51.041 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:16:51.041 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:16:51.041 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0254961 s, 41.1 MB/s 2021-02-10T11:16:51.042 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:51.042 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:51.043 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:51.377 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:16:51.377 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:51.378 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348574/465544fd-51e0-42c2-b788-2bd8ca512916/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:51.752 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:16:51.752 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:16:51.752 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0231708 s, 45.3 MB/s 2021-02-10T11:16:51.753 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:51.753 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:51.753 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:52.087 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:16:52.087 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:52.087 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348574/465544fd-51e0-42c2-b788-2bd8ca512916/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:52.462 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:16:52.462 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:16:52.462 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.022619 s, 46.4 MB/s 2021-02-10T11:16:52.463 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:52.463 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:52.464 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:52.799 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:16:52.799 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:52.800 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348574/465544fd-51e0-42c2-b788-2bd8ca512916/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:53.174 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:16:53.174 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:16:53.174 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0244569 s, 42.9 MB/s 2021-02-10T11:16:53.175 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:53.175 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:53.175 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:53.508 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:16:53.509 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:53.509 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348574/465544fd-51e0-42c2-b788-2bd8ca512916/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:53.885 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:16:53.886 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:16:53.886 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0248979 s, 42.1 MB/s 2021-02-10T11:16:53.887 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:53.887 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:53.887 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:54.220 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:16:54.221 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:54.221 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348574/465544fd-51e0-42c2-b788-2bd8ca512916/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:54.592 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:16:54.593 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:16:54.593 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0242058 s, 43.3 MB/s 2021-02-10T11:16:54.593 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:54.594 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:54.594 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:54.929 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:16:54.929 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:54.930 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348574/465544fd-51e0-42c2-b788-2bd8ca512916/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:55.308 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:16:55.308 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:16:55.308 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0271416 s, 38.6 MB/s 2021-02-10T11:16:55.309 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:55.309 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:55.309 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:55.643 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:16:55.643 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:55.643 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348574/465544fd-51e0-42c2-b788-2bd8ca512916/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:56.023 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:16:56.023 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:16:56.023 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0245714 s, 42.7 MB/s 2021-02-10T11:16:56.024 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:56.025 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:56.025 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:56.363 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:16:56.363 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:56.364 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348574/465544fd-51e0-42c2-b788-2bd8ca512916/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:56.741 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:16:56.742 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:16:56.742 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0237933 s, 44.1 MB/s 2021-02-10T11:16:56.743 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000348575 2021-02-10T11:16:57.173 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.158 7f146f919700 1 Processor -- start 2021-02-10T11:16:57.173 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.158 7f146f919700 1 -- start start 2021-02-10T11:16:57.173 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.158 7f146f919700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f14681b0470 0x7f14681b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:57.173 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.158 7f146f919700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f14680f20e0 0x7f14680f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:57.173 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.158 7f146f919700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14680f0b50 con 0x7f14680ef610 2021-02-10T11:16:57.173 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.158 7f146f919700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14680f0820 con 0x7f14680f20e0 2021-02-10T11:16:57.173 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.158 7f146f919700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14680f24e0 con 0x7f14681b0470 2021-02-10T11:16:57.174 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.160 7f146df36700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f14680f20e0 0x7f14680f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:57.174 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.160 7f146d735700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f14681b0470 0x7f14681b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:57.174 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.160 7f146cf34700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7f14680ef610 0x7f14680efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:37028/0 (socket says 172.21.2.112:37028) 2021-02-10T11:16:57.174 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.160 7f146cf34700 1 -- 172.21.2.112:0/2318361988 learned_addr learned my addr 172.21.2.112:0/2318361988 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:16:57.174 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.160 7f146d735700 1 -- 172.21.2.112:0/2318361988 >> v2:172.21.2.120:3300/0 conn(0x7f14680f20e0 msgr2=0x7f14680f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:16:57.174 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.160 7f146d735700 1 --2- 172.21.2.112:0/2318361988 >> v2:172.21.2.120:3300/0 conn(0x7f14680f20e0 0x7f14680f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:57.174 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.160 7f146d735700 1 -- 172.21.2.112:0/2318361988 >> v1:172.21.2.120:6789/0 conn(0x7f14680ef610 legacy=0x7f14680efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:57.174 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.160 7f146d735700 1 -- 172.21.2.112:0/2318361988 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14680e52e0 con 0x7f14681b0470 2021-02-10T11:16:57.174 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.160 7f146d735700 1 --2- 172.21.2.112:0/2318361988 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f14681b0470 0x7f14681b0870 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f1464006650 tx=0x7f1464009a80).ready entity=mon.2 client_cookie=283a78d58cbcdac7 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:16:57.174 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.160 7f145ffff700 1 -- 172.21.2.112:0/2318361988 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f146400c0c0 con 0x7f14681b0470 2021-02-10T11:16:57.174 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.161 7f145ffff700 1 -- 172.21.2.112:0/2318361988 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f146400c340 con 0x7f14681b0470 2021-02-10T11:16:57.174 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.161 7f145ffff700 1 -- 172.21.2.112:0/2318361988 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f146400c7a0 con 0x7f14681b0470 2021-02-10T11:16:57.175 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.161 7f146f919700 1 -- 172.21.2.112:0/2318361988 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f14681b0470 msgr2=0x7f14681b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:57.175 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.161 7f146f919700 1 --2- 172.21.2.112:0/2318361988 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f14681b0470 0x7f14681b0870 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f1464006650 tx=0x7f1464009a80).stop 2021-02-10T11:16:57.175 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.161 7f146f919700 1 -- 172.21.2.112:0/2318361988 shutdown_connections 2021-02-10T11:16:57.175 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.161 7f146f919700 1 --2- 172.21.2.112:0/2318361988 >> v2:172.21.2.120:3300/0 conn(0x7f14680f20e0 0x7f14680f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:57.175 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.161 7f146f919700 1 --2- 172.21.2.112:0/2318361988 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f14681b0470 0x7f14681b0870 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:57.175 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.161 7f146f919700 1 -- 172.21.2.112:0/2318361988 >> 172.21.2.112:0/2318361988 conn(0x7f14680ce3e0 msgr2=0x7f14680e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:16:57.175 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.161 7f146f919700 1 -- 172.21.2.112:0/2318361988 shutdown_connections 2021-02-10T11:16:57.175 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.161 7f146f919700 1 -- 172.21.2.112:0/2318361988 wait complete. 2021-02-10T11:16:57.177 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.176 7f146f919700 1 Processor -- start 2021-02-10T11:16:57.177 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.176 7f146f919700 1 -- start start 2021-02-10T11:16:57.177 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.177 7f146f919700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f14680b7c20 0x7f14680b8410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:57.177 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.177 7f146f919700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f14680b8a30 0x7f14680c1f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:57.177 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.177 7f146f919700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f14680c25a0 0x7f14680c29a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:57.177 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.177 7f146f919700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14680c1a80 con 0x7f14680b8a30 2021-02-10T11:16:57.177 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.177 7f146f919700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14680b8020 con 0x7f14680b7c20 2021-02-10T11:16:57.177 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.177 7f146f919700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f146809b9c0 con 0x7f14680c25a0 2021-02-10T11:16:57.177 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.177 7f146cf34700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f14680b8a30 0x7f14680c1f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:57.177 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.177 7f146d735700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f14680b7c20 0x7f14680b8410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:57.178 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.177 7f146cf34700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f14680b8a30 0x7f14680c1f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:44012/0 (socket says 172.21.2.112:44012) 2021-02-10T11:16:57.178 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.177 7f146cf34700 1 -- 172.21.2.112:0/3475102078 learned_addr learned my addr 172.21.2.112:0/3475102078 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:16:57.178 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.177 7f146df36700 1 --2- 172.21.2.112:0/3475102078 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f14680c25a0 0x7f14680c29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:57.178 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.177 7f146cf34700 1 -- 172.21.2.112:0/3475102078 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f14680c25a0 msgr2=0x7f14680c29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:57.178 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.177 7f146cf34700 1 --2- 172.21.2.112:0/3475102078 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f14680c25a0 0x7f14680c29a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:57.178 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.177 7f146cf34700 1 -- 172.21.2.112:0/3475102078 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f14680b7c20 msgr2=0x7f14680b8410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:57.178 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.177 7f146cf34700 1 --2- 172.21.2.112:0/3475102078 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f14680b7c20 0x7f14680b8410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:57.178 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.177 7f146cf34700 1 -- 172.21.2.112:0/3475102078 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14680b8e30 con 0x7f14680b8a30 2021-02-10T11:16:57.178 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.177 7f146df36700 1 --2- 172.21.2.112:0/3475102078 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f14680c25a0 0x7f14680c29a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:16:57.178 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.178 7f146cf34700 1 --2- 172.21.2.112:0/3475102078 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f14680b8a30 0x7f14680c1f80 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7f1458001c60 tx=0x7f145800a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:16:57.178 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.178 7f145dffb700 1 -- 172.21.2.112:0/3475102078 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f145800acc0 con 0x7f14680b8a30 2021-02-10T11:16:57.178 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.178 7f145dffb700 1 -- 172.21.2.112:0/3475102078 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1458004140 con 0x7f14680b8a30 2021-02-10T11:16:57.178 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.178 7f145dffb700 1 -- 172.21.2.112:0/3475102078 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1458004620 con 0x7f14680b8a30 2021-02-10T11:16:57.179 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.178 7f146f919700 1 -- 172.21.2.112:0/3475102078 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14680b8e30 con 0x7f14680b8a30 2021-02-10T11:16:57.179 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.178 7f146f919700 1 -- 172.21.2.112:0/3475102078 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14680c32b0 con 0x7f14680b8a30 2021-02-10T11:16:57.179 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.179 7f145dffb700 1 -- 172.21.2.112:0/3475102078 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1458004320 con 0x7f14680b8a30 2021-02-10T11:16:57.179 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.179 7f145dffb700 1 --2- 172.21.2.112:0/3475102078 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f14480186c0 0x7f144801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:57.179 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.179 7f145dffb700 1 -- 172.21.2.112:0/3475102078 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f1458032570 con 0x7f14680b8a30 2021-02-10T11:16:57.179 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.179 7f146d735700 1 --2- 172.21.2.112:0/3475102078 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f14480186c0 0x7f144801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:57.179 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.179 7f146d735700 1 --2- 172.21.2.112:0/3475102078 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f14480186c0 0x7f144801abc0 crc :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:16:57.205 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.208 7f146f919700 1 -- 172.21.2.112:0/3475102078 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1468105b30 con 0x7f14680b8a30 2021-02-10T11:16:57.208 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.211 7f145dffb700 1 -- 172.21.2.112:0/3475102078 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1458030d20 con 0x7f14680b8a30 2021-02-10T11:16:57.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.337 7f146f919700 1 -- 172.21.2.112:0/3475102078 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348575", "target": ["mgr", ""]}) v1 -- 0x7f14680b8e30 con 0x7f14480186c0 2021-02-10T11:16:57.358 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.361 7f145dffb700 1 -- 172.21.2.112:0/3475102078 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f1464009180 con 0x7f14480186c0 2021-02-10T11:16:57.367 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.370 7f146f919700 1 -- 172.21.2.112:0/3475102078 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f14480186c0 msgr2=0x7f144801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:57.367 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.370 7f146f919700 1 --2- 172.21.2.112:0/3475102078 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f14480186c0 0x7f144801abc0 crc :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:57.367 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.370 7f146f919700 1 -- 172.21.2.112:0/3475102078 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f14680b8a30 msgr2=0x7f14680c1f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:57.367 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.370 7f146f919700 1 --2- 172.21.2.112:0/3475102078 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f14680b8a30 0x7f14680c1f80 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7f1458001c60 tx=0x7f145800a8d0).stop 2021-02-10T11:16:57.368 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.370 7f146f919700 1 -- 172.21.2.112:0/3475102078 shutdown_connections 2021-02-10T11:16:57.368 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.370 7f146f919700 1 --2- 172.21.2.112:0/3475102078 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f14480186c0 0x7f144801abc0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:57.368 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.370 7f146f919700 1 --2- 172.21.2.112:0/3475102078 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f14680c25a0 0x7f14680c29a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:57.368 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.370 7f146f919700 1 --2- 172.21.2.112:0/3475102078 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f14680b8a30 0x7f14680c1f80 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:57.369 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.370 7f146f919700 1 --2- 172.21.2.112:0/3475102078 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f14680b7c20 0x7f14680b8410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:57.369 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.370 7f146f919700 1 -- 172.21.2.112:0/3475102078 >> 172.21.2.112:0/3475102078 conn(0x7f1468094e10 msgr2=0x7f14680a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:16:57.369 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.370 7f146f919700 1 -- 172.21.2.112:0/3475102078 shutdown_connections 2021-02-10T11:16:57.369 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.371 7f146f919700 1 -- 172.21.2.112:0/3475102078 wait complete. 2021-02-10T11:16:57.376 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000348575 2021-02-10T11:16:57.807 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.792 7f2af097b700 1 Processor -- start 2021-02-10T11:16:57.807 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.792 7f2af097b700 1 -- start start 2021-02-10T11:16:57.807 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.792 7f2af097b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2aec1b0470 0x7f2aec1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:57.807 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.792 7f2af097b700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f2aec0f20e0 0x7f2aec0f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:57.807 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.792 7f2af097b700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2aec0f0b50 con 0x7f2aec0ef610 2021-02-10T11:16:57.807 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.792 7f2af097b700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2aec0f0820 con 0x7f2aec0f20e0 2021-02-10T11:16:57.807 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.792 7f2af097b700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2aec0f24e0 con 0x7f2aec1b0470 2021-02-10T11:16:57.807 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.794 7f2aeae1d700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f2aec0f20e0 0x7f2aec0f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:57.807 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.794 7f2ae9e1b700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7f2aec0ef610 0x7f2aec0efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:37038/0 (socket says 172.21.2.112:37038) 2021-02-10T11:16:57.808 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.794 7f2aea61c700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2aec1b0470 0x7f2aec1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:57.808 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.794 7f2ae9e1b700 1 -- 172.21.2.112:0/3717863099 learned_addr learned my addr 172.21.2.112:0/3717863099 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:16:57.808 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.794 7f2aea61c700 1 -- 172.21.2.112:0/3717863099 >> v2:172.21.2.120:3300/0 conn(0x7f2aec0f20e0 msgr2=0x7f2aec0f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:16:57.808 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.794 7f2aea61c700 1 --2- 172.21.2.112:0/3717863099 >> v2:172.21.2.120:3300/0 conn(0x7f2aec0f20e0 0x7f2aec0f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:57.808 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.794 7f2aea61c700 1 -- 172.21.2.112:0/3717863099 >> v1:172.21.2.120:6789/0 conn(0x7f2aec0ef610 legacy=0x7f2aec0efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:57.808 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.794 7f2aea61c700 1 -- 172.21.2.112:0/3717863099 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2aec0e52e0 con 0x7f2aec1b0470 2021-02-10T11:16:57.808 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.794 7f2aeae1d700 1 --2- 172.21.2.112:0/3717863099 >> v2:172.21.2.120:3300/0 conn(0x7f2aec0f20e0 0x7f2aec0f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:16:57.808 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.794 7f2aea61c700 1 --2- 172.21.2.112:0/3717863099 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2aec1b0470 0x7f2aec1b0870 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f2adc006650 tx=0x7f2adc009a80).ready entity=mon.2 client_cookie=354c68e701c541be server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:16:57.808 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.795 7f2ae961a700 1 -- 172.21.2.112:0/3717863099 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2adc0009b0 con 0x7f2aec1b0470 2021-02-10T11:16:57.808 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.795 7f2ae961a700 1 -- 172.21.2.112:0/3717863099 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2adc00c0c0 con 0x7f2aec1b0470 2021-02-10T11:16:57.808 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.795 7f2ae961a700 1 -- 172.21.2.112:0/3717863099 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2adc00c5e0 con 0x7f2aec1b0470 2021-02-10T11:16:57.808 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.795 7f2af097b700 1 -- 172.21.2.112:0/3717863099 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2aec1b0470 msgr2=0x7f2aec1b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:57.809 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.795 7f2af097b700 1 --2- 172.21.2.112:0/3717863099 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2aec1b0470 0x7f2aec1b0870 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f2adc006650 tx=0x7f2adc009a80).stop 2021-02-10T11:16:57.809 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.796 7f2af097b700 1 -- 172.21.2.112:0/3717863099 shutdown_connections 2021-02-10T11:16:57.809 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.796 7f2af097b700 1 --2- 172.21.2.112:0/3717863099 >> v2:172.21.2.120:3300/0 conn(0x7f2aec0f20e0 0x7f2aec0f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:57.809 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.796 7f2af097b700 1 --2- 172.21.2.112:0/3717863099 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2aec1b0470 0x7f2aec1b0870 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:57.809 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.796 7f2af097b700 1 -- 172.21.2.112:0/3717863099 >> 172.21.2.112:0/3717863099 conn(0x7f2aec0ce3e0 msgr2=0x7f2aec0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:16:57.809 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.796 7f2af097b700 1 -- 172.21.2.112:0/3717863099 shutdown_connections 2021-02-10T11:16:57.809 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.796 7f2af097b700 1 -- 172.21.2.112:0/3717863099 wait complete. 2021-02-10T11:16:57.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.810 7f2af097b700 1 Processor -- start 2021-02-10T11:16:57.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.810 7f2af097b700 1 -- start start 2021-02-10T11:16:57.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.810 7f2af097b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2aec0b7c20 0x7f2aec0b8410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:57.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.810 7f2af097b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2aec0b8a30 0x7f2aec0c1f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:57.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.810 7f2af097b700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2aec0c25a0 0x7f2aec0c29a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:57.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.810 7f2af097b700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aec0c1a80 con 0x7f2aec0b8a30 2021-02-10T11:16:57.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.810 7f2af097b700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aec0b8020 con 0x7f2aec0c25a0 2021-02-10T11:16:57.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.810 7f2af097b700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2aec09b9c0 con 0x7f2aec0b7c20 2021-02-10T11:16:57.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.811 7f2ae9e1b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2aec0b8a30 0x7f2aec0c1f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:57.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.811 7f2ae9e1b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2aec0b8a30 0x7f2aec0c1f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:44022/0 (socket says 172.21.2.112:44022) 2021-02-10T11:16:57.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.811 7f2ae9e1b700 1 -- 172.21.2.112:0/3810706358 learned_addr learned my addr 172.21.2.112:0/3810706358 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:16:57.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.811 7f2aea61c700 1 --2- 172.21.2.112:0/3810706358 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2aec0b7c20 0x7f2aec0b8410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:57.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.811 7f2aeae1d700 1 --2- 172.21.2.112:0/3810706358 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2aec0c25a0 0x7f2aec0c29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:57.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.811 7f2ae9e1b700 1 -- 172.21.2.112:0/3810706358 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2aec0b7c20 msgr2=0x7f2aec0b8410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:57.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.811 7f2ae9e1b700 1 --2- 172.21.2.112:0/3810706358 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2aec0b7c20 0x7f2aec0b8410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:57.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.811 7f2ae9e1b700 1 -- 172.21.2.112:0/3810706358 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2aec0c25a0 msgr2=0x7f2aec0c29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:57.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.811 7f2ae9e1b700 1 --2- 172.21.2.112:0/3810706358 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2aec0c25a0 0x7f2aec0c29a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:57.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.811 7f2ae9e1b700 1 -- 172.21.2.112:0/3810706358 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2aec0b8e30 con 0x7f2aec0b8a30 2021-02-10T11:16:57.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.811 7f2aea61c700 1 --2- 172.21.2.112:0/3810706358 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2aec0b7c20 0x7f2aec0b8410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:16:57.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.811 7f2aeae1d700 1 --2- 172.21.2.112:0/3810706358 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2aec0c25a0 0x7f2aec0c29a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:16:57.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.811 7f2ae9e1b700 1 --2- 172.21.2.112:0/3810706358 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2aec0b8a30 0x7f2aec0c1f80 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 rx=0x7f2ad4001c60 tx=0x7f2ad400a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:16:57.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.811 7f2ae2ffd700 1 -- 172.21.2.112:0/3810706358 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2ad400acc0 con 0x7f2aec0b8a30 2021-02-10T11:16:57.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.812 7f2ae2ffd700 1 -- 172.21.2.112:0/3810706358 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ad4004140 con 0x7f2aec0b8a30 2021-02-10T11:16:57.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.812 7f2ae2ffd700 1 -- 172.21.2.112:0/3810706358 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2ad40045a0 con 0x7f2aec0b8a30 2021-02-10T11:16:57.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.812 7f2af097b700 1 -- 172.21.2.112:0/3810706358 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2aec0b8e30 con 0x7f2aec0b8a30 2021-02-10T11:16:57.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.812 7f2af097b700 1 -- 172.21.2.112:0/3810706358 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2aec0c32b0 con 0x7f2aec0b8a30 2021-02-10T11:16:57.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.812 7f2ae2ffd700 1 -- 172.21.2.112:0/3810706358 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2ad4004320 con 0x7f2aec0b8a30 2021-02-10T11:16:57.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.812 7f2ae2ffd700 1 --2- 172.21.2.112:0/3810706358 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2ad00186c0 0x7f2ad001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:57.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.812 7f2ae2ffd700 1 -- 172.21.2.112:0/3810706358 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f2ad4032500 con 0x7f2aec0b8a30 2021-02-10T11:16:57.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.813 7f2aea61c700 1 --2- 172.21.2.112:0/3810706358 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2ad00186c0 0x7f2ad001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:57.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.813 7f2aea61c700 1 --2- 172.21.2.112:0/3810706358 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2ad00186c0 0x7f2ad001abc0 crc :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:16:57.839 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.842 7f2af097b700 1 -- 172.21.2.112:0/3810706358 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2aec105b30 con 0x7f2aec0b8a30 2021-02-10T11:16:57.841 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.845 7f2ae2ffd700 1 -- 172.21.2.112:0/3810706358 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2ad40010b0 con 0x7f2aec0b8a30 2021-02-10T11:16:57.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.973 7f2af097b700 1 -- 172.21.2.112:0/3810706358 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348575", "target": ["mgr", ""]}) v1 -- 0x7f2aec105b30 con 0x7f2ad00186c0 2021-02-10T11:16:57.973 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.976 7f2ae2ffd700 1 -- 172.21.2.112:0/3810706358 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f2adc000940 con 0x7f2ad00186c0 2021-02-10T11:16:57.977 INFO:teuthology.orchestra.run.gibba012.stdout:/volumes/_nogroup/subvolume_0000000000348575/9ef01c90-e6ab-464b-b6f4-fe4d54dec568 2021-02-10T11:16:57.979 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.982 7f2af097b700 1 -- 172.21.2.112:0/3810706358 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2ad00186c0 msgr2=0x7f2ad001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:57.979 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.982 7f2af097b700 1 --2- 172.21.2.112:0/3810706358 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2ad00186c0 0x7f2ad001abc0 crc :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:57.979 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.982 7f2af097b700 1 -- 172.21.2.112:0/3810706358 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2aec0b8a30 msgr2=0x7f2aec0c1f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:57.979 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.982 7f2af097b700 1 --2- 172.21.2.112:0/3810706358 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2aec0b8a30 0x7f2aec0c1f80 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 rx=0x7f2ad4001c60 tx=0x7f2ad400a8d0).stop 2021-02-10T11:16:57.979 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.982 7f2af097b700 1 -- 172.21.2.112:0/3810706358 shutdown_connections 2021-02-10T11:16:57.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.982 7f2af097b700 1 --2- 172.21.2.112:0/3810706358 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2ad00186c0 0x7f2ad001abc0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:57.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.982 7f2af097b700 1 --2- 172.21.2.112:0/3810706358 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2aec0c25a0 0x7f2aec0c29a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:57.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.982 7f2af097b700 1 --2- 172.21.2.112:0/3810706358 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2aec0b8a30 0x7f2aec0c1f80 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:57.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.982 7f2af097b700 1 --2- 172.21.2.112:0/3810706358 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2aec0b7c20 0x7f2aec0b8410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:57.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.982 7f2af097b700 1 -- 172.21.2.112:0/3810706358 >> 172.21.2.112:0/3810706358 conn(0x7f2aec094e10 msgr2=0x7f2aec0a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:16:57.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.982 7f2af097b700 1 -- 172.21.2.112:0/3810706358 shutdown_connections 2021-02-10T11:16:57.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:16:57.982 7f2af097b700 1 -- 172.21.2.112:0/3810706358 wait complete. 2021-02-10T11:16:57.987 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000348575 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000348575/9ef01c90-e6ab-464b-b6f4-fe4d54dec568 2021-02-10T11:16:57.987 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:57.988 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:57.988 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:58.322 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:16:58.323 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:58.323 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348575/9ef01c90-e6ab-464b-b6f4-fe4d54dec568/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:58.703 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:16:58.703 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:16:58.703 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0250627 s, 41.8 MB/s 2021-02-10T11:16:58.704 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:58.704 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:58.704 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:59.038 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:16:59.038 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:59.039 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348575/9ef01c90-e6ab-464b-b6f4-fe4d54dec568/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:59.408 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:16:59.409 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:16:59.409 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.023283 s, 45.0 MB/s 2021-02-10T11:16:59.410 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:59.410 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:59.410 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:59.744 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:16:59.744 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:59.744 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348575/9ef01c90-e6ab-464b-b6f4-fe4d54dec568/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:00.117 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:17:00.117 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:17:00.117 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0233817 s, 44.8 MB/s 2021-02-10T11:17:00.118 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:00.118 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:00.118 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:00.454 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:17:00.454 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:00.454 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348575/9ef01c90-e6ab-464b-b6f4-fe4d54dec568/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:00.829 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:17:00.829 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:17:00.829 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.023866 s, 43.9 MB/s 2021-02-10T11:17:00.830 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:00.830 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:00.830 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:01.165 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:17:01.166 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:01.166 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348575/9ef01c90-e6ab-464b-b6f4-fe4d54dec568/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:01.542 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:17:01.543 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:17:01.543 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0265335 s, 39.5 MB/s 2021-02-10T11:17:01.544 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:01.544 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:01.544 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:01.877 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:17:01.878 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:01.878 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348575/9ef01c90-e6ab-464b-b6f4-fe4d54dec568/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:02.249 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:17:02.249 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:17:02.250 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0228254 s, 45.9 MB/s 2021-02-10T11:17:02.250 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:02.251 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:02.251 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:02.585 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:17:02.585 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:02.585 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348575/9ef01c90-e6ab-464b-b6f4-fe4d54dec568/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:02.961 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:17:02.961 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:17:02.961 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0253474 s, 41.4 MB/s 2021-02-10T11:17:02.962 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:02.962 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:02.962 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:03.295 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:17:03.295 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:03.295 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348575/9ef01c90-e6ab-464b-b6f4-fe4d54dec568/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:03.356 DEBUG:teuthology.orchestra.run.gibba012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:17:03.360 DEBUG:teuthology.orchestra.run.gibba020:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:17:03.669 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:17:03.669 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:17:03.669 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0238016 s, 44.1 MB/s 2021-02-10T11:17:03.670 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:03.670 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:03.670 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:04.044 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:17:04.044 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:04.044 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348575/9ef01c90-e6ab-464b-b6f4-fe4d54dec568/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:04.413 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:17:04.413 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:17:04.414 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0229748 s, 45.6 MB/s 2021-02-10T11:17:04.415 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:04.415 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:04.416 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:04.751 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:17:04.751 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:04.751 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348575/9ef01c90-e6ab-464b-b6f4-fe4d54dec568/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:05.125 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:17:05.125 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:17:05.125 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0233185 s, 45.0 MB/s 2021-02-10T11:17:05.126 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000348576 2021-02-10T11:17:05.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:05.542 7f5690cc8700 1 Processor -- start 2021-02-10T11:17:05.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:05.542 7f5690cc8700 1 -- start start 2021-02-10T11:17:05.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:05.543 7f5690cc8700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f568c1b0870 0x7f568c0f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:05.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:05.543 7f5690cc8700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f568c0efe00 0x7f568c0f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:05.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:05.543 7f5690cc8700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f568c1b0c70 con 0x7f568c1b0470 2021-02-10T11:17:05.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:05.543 7f5690cc8700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f568c0f0b50 con 0x7f568c1b0870 2021-02-10T11:17:05.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:05.543 7f5690cc8700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f568c0f0820 con 0x7f568c0efe00 2021-02-10T11:17:05.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:05.543 7f568a05d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f568c1b0870 0x7f568c0f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:05.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:05.543 7f568a05d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f568c1b0870 0x7f568c0f20e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:44032/0 (socket says 172.21.2.112:44032) 2021-02-10T11:17:05.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:05.543 7f568a85e700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f568c1b0470 0x7f568c0ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:43492/0 (socket says 172.21.2.112:43492) 2021-02-10T11:17:05.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:05.543 7f568a05d700 1 -- 172.21.2.112:0/2955855926 learned_addr learned my addr 172.21.2.112:0/2955855926 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:17:05.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:05.543 7f568a05d700 1 -- 172.21.2.112:0/2955855926 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f568c0efe00 msgr2=0x7f568c0f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:17:05.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:05.543 7f568985c700 1 -- 172.21.2.112:0/2955855926 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 542681233 0 0) 0x7f567c001100 con 0x7f568c1b0470 2021-02-10T11:17:05.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:05.543 7f568a05d700 1 --2- 172.21.2.112:0/2955855926 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f568c0efe00 0x7f568c0f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:17:05.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:05.543 7f568a05d700 1 -- 172.21.2.112:0/2955855926 >> v1:172.21.2.112:6789/0 conn(0x7f568c1b0470 legacy=0x7f568c0ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:05.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:05.543 7f568a05d700 1 -- 172.21.2.112:0/2955855926 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f568c0e52e0 con 0x7f568c1b0870 2021-02-10T11:17:05.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:05.544 7f568a05d700 1 --2- 172.21.2.112:0/2955855926 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f568c1b0870 0x7f568c0f20e0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 rx=0x7f5674006700 tx=0x7f5674009b80).ready entity=mon.0 client_cookie=b99283f99de5dfcf server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:17:05.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:05.544 7f568985c700 1 -- 172.21.2.112:0/2955855926 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f567400c0c0 con 0x7f568c1b0870 2021-02-10T11:17:05.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:05.544 7f568985c700 1 -- 172.21.2.112:0/2955855926 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f567400c440 con 0x7f568c1b0870 2021-02-10T11:17:05.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:05.544 7f568985c700 1 -- 172.21.2.112:0/2955855926 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f567400c920 con 0x7f568c1b0870 2021-02-10T11:17:05.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:05.544 7f5690cc8700 1 -- 172.21.2.112:0/2955855926 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f568c1b0870 msgr2=0x7f568c0f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:05.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:05.544 7f5690cc8700 1 --2- 172.21.2.112:0/2955855926 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f568c1b0870 0x7f568c0f20e0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 rx=0x7f5674006700 tx=0x7f5674009b80).stop 2021-02-10T11:17:05.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:05.544 7f5690cc8700 1 -- 172.21.2.112:0/2955855926 shutdown_connections 2021-02-10T11:17:05.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:05.544 7f5690cc8700 1 --2- 172.21.2.112:0/2955855926 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f568c0efe00 0x7f568c0f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:17:05.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:05.544 7f5690cc8700 1 --2- 172.21.2.112:0/2955855926 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f568c1b0870 0x7f568c0f20e0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:05.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:05.544 7f5690cc8700 1 -- 172.21.2.112:0/2955855926 >> 172.21.2.112:0/2955855926 conn(0x7f568c0ce3e0 msgr2=0x7f568c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:17:05.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:05.545 7f5690cc8700 1 -- 172.21.2.112:0/2955855926 shutdown_connections 2021-02-10T11:17:05.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:05.545 7f5690cc8700 1 -- 172.21.2.112:0/2955855926 wait complete. 2021-02-10T11:17:05.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:05.559 7f5690cc8700 1 Processor -- start 2021-02-10T11:17:05.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:05.559 7f5690cc8700 1 -- start start 2021-02-10T11:17:05.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:05.559 7f5690cc8700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f568c190920 0x7f568c08f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:05.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:05.559 7f5690cc8700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f568c190d20 0x7f568c08fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:05.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:05.559 7f5690cc8700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f568c193270 0x7f568c193670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:05.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:05.559 7f5690cc8700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f568c090470 con 0x7f568c190d20 2021-02-10T11:17:05.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:05.559 7f5690cc8700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f568c193c90 con 0x7f568c193270 2021-02-10T11:17:05.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:05.559 7f5690cc8700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f568c193e70 con 0x7f568c190920 2021-02-10T11:17:05.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:05.560 7f568a05d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f568c190d20 0x7f568c08fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:05.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:05.560 7f568a05d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f568c190d20 0x7f568c08fc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:44036/0 (socket says 172.21.2.112:44036) 2021-02-10T11:17:05.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:05.560 7f568a05d700 1 -- 172.21.2.112:0/1608151261 learned_addr learned my addr 172.21.2.112:0/1608151261 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:17:05.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:05.560 7f568a05d700 1 -- 172.21.2.112:0/1608151261 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f568c190920 msgr2=0x7f568c08f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:05.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:05.560 7f568b05f700 1 --2- 172.21.2.112:0/1608151261 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f568c193270 0x7f568c193670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:05.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:05.560 7f568a85e700 1 --2- 172.21.2.112:0/1608151261 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f568c190920 0x7f568c08f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:05.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:05.560 7f568a05d700 1 --2- 172.21.2.112:0/1608151261 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f568c190920 0x7f568c08f600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:05.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:05.560 7f568a05d700 1 -- 172.21.2.112:0/1608151261 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f568c193270 msgr2=0x7f568c193670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:05.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:05.560 7f568a05d700 1 --2- 172.21.2.112:0/1608151261 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f568c193270 0x7f568c193670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:05.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:05.560 7f568a05d700 1 -- 172.21.2.112:0/1608151261 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f568c194090 con 0x7f568c190d20 2021-02-10T11:17:05.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:05.560 7f568b05f700 1 --2- 172.21.2.112:0/1608151261 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f568c193270 0x7f568c193670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:17:05.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:05.560 7f568a85e700 1 --2- 172.21.2.112:0/1608151261 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f568c190920 0x7f568c08f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:17:05.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:05.560 7f568a05d700 1 --2- 172.21.2.112:0/1608151261 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f568c190d20 0x7f568c08fc20 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7f567400b5e0 tx=0x7f567400b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:17:05.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:05.561 7f56837fe700 1 -- 172.21.2.112:0/1608151261 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f567400bd70 con 0x7f568c190d20 2021-02-10T11:17:05.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:05.561 7f56837fe700 1 -- 172.21.2.112:0/1608151261 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5674002030 con 0x7f568c190d20 2021-02-10T11:17:05.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:05.561 7f56837fe700 1 -- 172.21.2.112:0/1608151261 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f56740024d0 con 0x7f568c190d20 2021-02-10T11:17:05.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:05.561 7f5690cc8700 1 -- 172.21.2.112:0/1608151261 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f568c194090 con 0x7f568c190d20 2021-02-10T11:17:05.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:05.561 7f5690cc8700 1 -- 172.21.2.112:0/1608151261 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f568c194090 con 0x7f568c190d20 2021-02-10T11:17:05.563 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:05.561 7f56837fe700 1 -- 172.21.2.112:0/1608151261 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5674000c90 con 0x7f568c190d20 2021-02-10T11:17:05.563 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:05.562 7f56837fe700 1 --2- 172.21.2.112:0/1608151261 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f56700186c0 0x7f567001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:05.563 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:05.562 7f56837fe700 1 -- 172.21.2.112:0/1608151261 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f56740324c0 con 0x7f568c190d20 2021-02-10T11:17:05.563 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:05.562 7f568a85e700 1 --2- 172.21.2.112:0/1608151261 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f56700186c0 0x7f567001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:05.563 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:05.563 7f568a85e700 1 --2- 172.21.2.112:0/1608151261 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f56700186c0 0x7f567001abc0 crc :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:17:05.590 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:05.593 7f5690cc8700 1 -- 172.21.2.112:0/1608151261 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f568c105af0 con 0x7f568c190d20 2021-02-10T11:17:05.592 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:05.595 7f56837fe700 1 -- 172.21.2.112:0/1608151261 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5674030df0 con 0x7f568c190d20 2021-02-10T11:17:05.718 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:05.721 7f5690cc8700 1 -- 172.21.2.112:0/1608151261 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348576", "target": ["mgr", ""]}) v1 -- 0x7f568c0f0b50 con 0x7f56700186c0 2021-02-10T11:17:05.744 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:05.747 7f56837fe700 1 -- 172.21.2.112:0/1608151261 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f567c0068a0 con 0x7f56700186c0 2021-02-10T11:17:05.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:05.754 7f5690cc8700 1 -- 172.21.2.112:0/1608151261 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f56700186c0 msgr2=0x7f567001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:05.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:05.754 7f5690cc8700 1 --2- 172.21.2.112:0/1608151261 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f56700186c0 0x7f567001abc0 crc :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:05.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:05.754 7f5690cc8700 1 -- 172.21.2.112:0/1608151261 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f568c190d20 msgr2=0x7f568c08fc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:05.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:05.754 7f5690cc8700 1 --2- 172.21.2.112:0/1608151261 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f568c190d20 0x7f568c08fc20 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7f567400b5e0 tx=0x7f567400b980).stop 2021-02-10T11:17:05.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:05.755 7f5690cc8700 1 -- 172.21.2.112:0/1608151261 shutdown_connections 2021-02-10T11:17:05.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:05.755 7f5690cc8700 1 --2- 172.21.2.112:0/1608151261 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f56700186c0 0x7f567001abc0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:05.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:05.755 7f5690cc8700 1 --2- 172.21.2.112:0/1608151261 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f568c193270 0x7f568c193670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:05.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:05.755 7f5690cc8700 1 --2- 172.21.2.112:0/1608151261 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f568c190d20 0x7f568c08fc20 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:05.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:05.755 7f5690cc8700 1 --2- 172.21.2.112:0/1608151261 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f568c190920 0x7f568c08f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:05.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:05.755 7f5690cc8700 1 -- 172.21.2.112:0/1608151261 >> 172.21.2.112:0/1608151261 conn(0x7f568c0bf4e0 msgr2=0x7f568c0be3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:17:05.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:05.755 7f5690cc8700 1 -- 172.21.2.112:0/1608151261 shutdown_connections 2021-02-10T11:17:05.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:05.755 7f5690cc8700 1 -- 172.21.2.112:0/1608151261 wait complete. 2021-02-10T11:17:05.760 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000348576 2021-02-10T11:17:06.189 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:06.175 7fd472e33700 1 Processor -- start 2021-02-10T11:17:06.189 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:06.175 7fd472e33700 1 -- start start 2021-02-10T11:17:06.189 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:06.176 7fd472e33700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd46c1b0470 0x7fd46c1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:06.189 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:06.176 7fd472e33700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fd46c0ef610 0x7fd46c0efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:06.189 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:06.176 7fd472e33700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd46c0f0b50 con 0x7fd46c0f0030 2021-02-10T11:17:06.189 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:06.176 7fd472e33700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd46c0e52e0 con 0x7fd46c0ef610 2021-02-10T11:17:06.189 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:06.176 7fd472e33700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd46c0e5490 con 0x7fd46c1b0470 2021-02-10T11:17:06.189 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:06.176 7fd471450700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fd46c0f0030 0x7fd46c0f10d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:43510/0 (socket says 172.21.2.112:43510) 2021-02-10T11:17:06.190 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:06.176 7fd471450700 1 -- 172.21.2.112:0/672654532 learned_addr learned my addr 172.21.2.112:0/672654532 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:17:06.190 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:06.176 7fd46bfff700 1 --2- 172.21.2.112:0/672654532 >> v2:172.21.2.112:3300/0 conn(0x7fd46c0ef610 0x7fd46c0efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:06.190 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:06.176 7fd46b7fe700 1 -- 172.21.2.112:0/672654532 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1125976869 0 0) 0x7fd464001500 con 0x7fd46c0f0030 2021-02-10T11:17:06.190 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:06.176 7fd46bfff700 1 -- 172.21.2.112:0/672654532 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd46c1b0470 msgr2=0x7fd46c1b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:17:06.190 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:06.176 7fd46bfff700 1 --2- 172.21.2.112:0/672654532 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd46c1b0470 0x7fd46c1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:17:06.190 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:06.176 7fd46bfff700 1 -- 172.21.2.112:0/672654532 >> v1:172.21.2.112:6789/0 conn(0x7fd46c0f0030 legacy=0x7fd46c0f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:06.190 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:06.176 7fd46bfff700 1 -- 172.21.2.112:0/672654532 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd46c0f0430 con 0x7fd46c0ef610 2021-02-10T11:17:06.190 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:06.177 7fd46bfff700 1 --2- 172.21.2.112:0/672654532 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd46c0ef610 0x7fd46c0efa10 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7fd454006650 tx=0x7fd454009a80).ready entity=mon.0 client_cookie=9c756106ca970d0e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:17:06.190 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:06.177 7fd46b7fe700 1 -- 172.21.2.112:0/672654532 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd4540009b0 con 0x7fd46c0ef610 2021-02-10T11:17:06.190 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:06.177 7fd46b7fe700 1 -- 172.21.2.112:0/672654532 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd45400c0c0 con 0x7fd46c0ef610 2021-02-10T11:17:06.190 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:06.177 7fd46b7fe700 1 -- 172.21.2.112:0/672654532 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd45400c5e0 con 0x7fd46c0ef610 2021-02-10T11:17:06.190 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:06.177 7fd472e33700 1 -- 172.21.2.112:0/672654532 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd46c0ef610 msgr2=0x7fd46c0efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:06.191 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:06.177 7fd472e33700 1 --2- 172.21.2.112:0/672654532 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd46c0ef610 0x7fd46c0efa10 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7fd454006650 tx=0x7fd454009a80).stop 2021-02-10T11:17:06.191 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:06.177 7fd472e33700 1 -- 172.21.2.112:0/672654532 shutdown_connections 2021-02-10T11:17:06.191 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:06.177 7fd472e33700 1 --2- 172.21.2.112:0/672654532 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd46c0ef610 0x7fd46c0efa10 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:06.191 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:06.178 7fd472e33700 1 --2- 172.21.2.112:0/672654532 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd46c1b0470 0x7fd46c1b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:17:06.191 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:06.178 7fd472e33700 1 -- 172.21.2.112:0/672654532 >> 172.21.2.112:0/672654532 conn(0x7fd46c0ce3e0 msgr2=0x7fd46c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:17:06.191 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:06.178 7fd472e33700 1 -- 172.21.2.112:0/672654532 shutdown_connections 2021-02-10T11:17:06.191 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:06.178 7fd472e33700 1 -- 172.21.2.112:0/672654532 wait complete. 2021-02-10T11:17:06.193 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:06.192 7fd472e33700 1 Processor -- start 2021-02-10T11:17:06.193 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:06.192 7fd472e33700 1 -- start start 2021-02-10T11:17:06.193 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:06.193 7fd472e33700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd46c0b20b0 0x7fd46c0ac970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:06.193 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:06.193 7fd472e33700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd46c0b24b0 0x7fd46c191dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:06.193 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:06.193 7fd472e33700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd46c0acf90 0x7fd46c1923e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:06.193 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:06.193 7fd472e33700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd46c0b1b30 con 0x7fd46c0acf90 2021-02-10T11:17:06.193 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:06.193 7fd472e33700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd46c1a04e0 con 0x7fd46c0b24b0 2021-02-10T11:17:06.193 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:06.193 7fd472e33700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd46c1a06c0 con 0x7fd46c0b20b0 2021-02-10T11:17:06.194 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:06.193 7fd471450700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd46c0acf90 0x7fd46c1923e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:06.194 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:06.193 7fd471450700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd46c0acf90 0x7fd46c1923e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:44050/0 (socket says 172.21.2.112:44050) 2021-02-10T11:17:06.194 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:06.193 7fd471450700 1 -- 172.21.2.112:0/3046727846 learned_addr learned my addr 172.21.2.112:0/3046727846 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:17:06.194 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:06.193 7fd471450700 1 -- 172.21.2.112:0/3046727846 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd46c0b20b0 msgr2=0x7fd46c0ac970 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:17:06.194 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:06.193 7fd471450700 1 --2- 172.21.2.112:0/3046727846 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd46c0b20b0 0x7fd46c0ac970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:17:06.194 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:06.193 7fd471450700 1 -- 172.21.2.112:0/3046727846 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd46c0b24b0 msgr2=0x7fd46c191dc0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:17:06.194 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:06.193 7fd471450700 1 --2- 172.21.2.112:0/3046727846 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd46c0b24b0 0x7fd46c191dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:17:06.194 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:06.193 7fd471450700 1 -- 172.21.2.112:0/3046727846 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd46c192be0 con 0x7fd46c0acf90 2021-02-10T11:17:06.194 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:06.193 7fd471450700 1 --2- 172.21.2.112:0/3046727846 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd46c0acf90 0x7fd46c1923e0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0x7fd464001bf0 tx=0x7fd46400ac50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:17:06.194 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:06.194 7fd4697fa700 1 -- 172.21.2.112:0/3046727846 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd4640070d0 con 0x7fd46c0acf90 2021-02-10T11:17:06.194 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:06.194 7fd4697fa700 1 -- 172.21.2.112:0/3046727846 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd464007350 con 0x7fd46c0acf90 2021-02-10T11:17:06.194 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:06.194 7fd4697fa700 1 -- 172.21.2.112:0/3046727846 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd464004720 con 0x7fd46c0acf90 2021-02-10T11:17:06.195 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:06.194 7fd472e33700 1 -- 172.21.2.112:0/3046727846 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd46c192be0 con 0x7fd46c0acf90 2021-02-10T11:17:06.195 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:06.194 7fd472e33700 1 -- 172.21.2.112:0/3046727846 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd46c192be0 con 0x7fd46c0acf90 2021-02-10T11:17:06.195 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:06.194 7fd4697fa700 1 -- 172.21.2.112:0/3046727846 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd464007530 con 0x7fd46c0acf90 2021-02-10T11:17:06.195 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:06.194 7fd4697fa700 1 --2- 172.21.2.112:0/3046727846 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd4500186d0 0x7fd45001abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:06.195 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:06.194 7fd4697fa700 1 -- 172.21.2.112:0/3046727846 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fd464032b20 con 0x7fd46c0acf90 2021-02-10T11:17:06.195 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:06.195 7fd470c4f700 1 --2- 172.21.2.112:0/3046727846 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd4500186d0 0x7fd45001abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:06.195 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:06.195 7fd470c4f700 1 --2- 172.21.2.112:0/3046727846 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd4500186d0 0x7fd45001abd0 crc :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:17:06.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:06.226 7fd472e33700 1 -- 172.21.2.112:0/3046727846 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd46c105af0 con 0x7fd46c0acf90 2021-02-10T11:17:06.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:06.228 7fd4697fa700 1 -- 172.21.2.112:0/3046727846 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd464000cc0 con 0x7fd46c0acf90 2021-02-10T11:17:06.351 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:06.354 7fd472e33700 1 -- 172.21.2.112:0/3046727846 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348576", "target": ["mgr", ""]}) v1 -- 0x7fd46c1a04e0 con 0x7fd4500186d0 2021-02-10T11:17:06.355 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:06.358 7fd4697fa700 1 -- 172.21.2.112:0/3046727846 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7fd45c006970 con 0x7fd4500186d0 2021-02-10T11:17:06.359 INFO:teuthology.orchestra.run.gibba012.stdout:/volumes/_nogroup/subvolume_0000000000348576/99a9371e-a654-447f-aec4-93ff2d8572ad 2021-02-10T11:17:06.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:06.364 7fd472e33700 1 -- 172.21.2.112:0/3046727846 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd4500186d0 msgr2=0x7fd45001abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:06.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:06.364 7fd472e33700 1 --2- 172.21.2.112:0/3046727846 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd4500186d0 0x7fd45001abd0 crc :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:06.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:06.364 7fd472e33700 1 -- 172.21.2.112:0/3046727846 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd46c0acf90 msgr2=0x7fd46c1923e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:06.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:06.364 7fd472e33700 1 --2- 172.21.2.112:0/3046727846 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd46c0acf90 0x7fd46c1923e0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0x7fd464001bf0 tx=0x7fd46400ac50).stop 2021-02-10T11:17:06.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:06.364 7fd472e33700 1 -- 172.21.2.112:0/3046727846 shutdown_connections 2021-02-10T11:17:06.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:06.364 7fd472e33700 1 --2- 172.21.2.112:0/3046727846 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd4500186d0 0x7fd45001abd0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:06.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:06.364 7fd472e33700 1 --2- 172.21.2.112:0/3046727846 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd46c0b24b0 0x7fd46c191dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:17:06.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:06.364 7fd472e33700 1 --2- 172.21.2.112:0/3046727846 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd46c0acf90 0x7fd46c1923e0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:06.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:06.364 7fd472e33700 1 --2- 172.21.2.112:0/3046727846 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd46c0b20b0 0x7fd46c0ac970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:17:06.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:06.364 7fd472e33700 1 -- 172.21.2.112:0/3046727846 >> 172.21.2.112:0/3046727846 conn(0x7fd46c0bf4e0 msgr2=0x7fd46c0be3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:17:06.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:06.364 7fd472e33700 1 -- 172.21.2.112:0/3046727846 shutdown_connections 2021-02-10T11:17:06.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:06.364 7fd472e33700 1 -- 172.21.2.112:0/3046727846 wait complete. 2021-02-10T11:17:06.369 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000348576 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000348576/99a9371e-a654-447f-aec4-93ff2d8572ad 2021-02-10T11:17:06.370 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:06.370 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:06.370 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:06.703 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:17:06.704 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:06.704 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348576/99a9371e-a654-447f-aec4-93ff2d8572ad/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:07.086 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:17:07.086 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:17:07.087 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0273232 s, 38.4 MB/s 2021-02-10T11:17:07.087 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:07.088 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:07.088 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:07.422 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:17:07.422 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:07.423 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348576/99a9371e-a654-447f-aec4-93ff2d8572ad/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:07.799 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:17:07.799 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:17:07.800 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0267454 s, 39.2 MB/s 2021-02-10T11:17:07.801 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:07.801 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:07.801 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:08.136 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:17:08.136 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:08.136 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348576/99a9371e-a654-447f-aec4-93ff2d8572ad/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:08.512 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:17:08.512 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:17:08.512 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0249025 s, 42.1 MB/s 2021-02-10T11:17:08.513 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:08.513 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:08.514 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:08.847 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:17:08.848 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:08.848 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348576/99a9371e-a654-447f-aec4-93ff2d8572ad/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:09.227 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:17:09.227 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:17:09.227 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.027109 s, 38.7 MB/s 2021-02-10T11:17:09.228 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:09.228 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:09.228 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:09.563 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:17:09.563 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:09.563 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348576/99a9371e-a654-447f-aec4-93ff2d8572ad/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:09.936 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:17:09.936 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:17:09.936 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0261997 s, 40.0 MB/s 2021-02-10T11:17:09.937 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:09.937 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:09.937 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:10.273 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:17:10.273 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:10.273 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348576/99a9371e-a654-447f-aec4-93ff2d8572ad/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:10.646 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:17:10.646 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:17:10.646 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0238694 s, 43.9 MB/s 2021-02-10T11:17:10.647 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:10.648 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:10.648 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:10.982 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:17:10.982 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:10.982 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348576/99a9371e-a654-447f-aec4-93ff2d8572ad/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:11.360 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:17:11.360 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:17:11.360 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0258555 s, 40.6 MB/s 2021-02-10T11:17:11.361 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:11.361 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:11.361 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:11.697 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:17:11.697 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:11.697 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348576/99a9371e-a654-447f-aec4-93ff2d8572ad/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:12.070 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:17:12.070 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:17:12.070 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0226803 s, 46.2 MB/s 2021-02-10T11:17:12.071 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:12.071 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:12.071 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:12.406 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:17:12.406 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:12.406 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348576/99a9371e-a654-447f-aec4-93ff2d8572ad/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:12.780 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:17:12.781 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:17:12.781 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0230842 s, 45.4 MB/s 2021-02-10T11:17:12.782 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:12.782 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:12.782 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:13.117 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:17:13.117 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:13.117 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348576/99a9371e-a654-447f-aec4-93ff2d8572ad/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:13.493 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:17:13.493 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:17:13.493 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0253937 s, 41.3 MB/s 2021-02-10T11:17:13.494 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000348577 2021-02-10T11:17:13.922 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:13.909 7ff1916b4700 1 Processor -- start 2021-02-10T11:17:13.922 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:13.909 7ff1916b4700 1 -- start start 2021-02-10T11:17:13.922 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:13.910 7ff1916b4700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff18c1b0470 0x7ff18c1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:13.922 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:13.910 7ff1916b4700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7ff18c0f20e0 0x7ff18c0f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:13.922 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:13.910 7ff1916b4700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff18c0f0b50 con 0x7ff18c0ef610 2021-02-10T11:17:13.922 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:13.910 7ff1916b4700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff18c0f0820 con 0x7ff18c0f20e0 2021-02-10T11:17:13.922 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:13.910 7ff1916b4700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff18c0f24e0 con 0x7ff18c1b0470 2021-02-10T11:17:13.923 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:13.910 7ff18bac0700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7ff18c0f20e0 0x7ff18c0f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:13.923 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:13.910 7ff18bac0700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7ff18c0f20e0 0x7ff18c0f0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:44058/0 (socket says 172.21.2.112:44058) 2021-02-10T11:17:13.923 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:13.910 7ff18bac0700 1 -- 172.21.2.112:0/1348744624 learned_addr learned my addr 172.21.2.112:0/1348744624 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:17:13.923 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:13.910 7ff18aabe700 1 --1- 172.21.2.112:0/1348744624 >> v1:172.21.2.112:6789/0 conn(0x7ff18c0ef610 0x7ff18c0efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:43520/0 (socket says 172.21.2.112:43520) 2021-02-10T11:17:13.923 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:13.910 7ff18bac0700 1 -- 172.21.2.112:0/1348744624 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff18c1b0470 msgr2=0x7ff18c1b0870 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:17:13.923 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:13.910 7ff18bac0700 1 --2- 172.21.2.112:0/1348744624 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff18c1b0470 0x7ff18c1b0870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:17:13.923 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:13.910 7ff18bac0700 1 -- 172.21.2.112:0/1348744624 >> v1:172.21.2.112:6789/0 conn(0x7ff18c0ef610 legacy=0x7ff18c0efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:13.923 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:13.910 7ff18a2bd700 1 -- 172.21.2.112:0/1348744624 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2133581372 0 0) 0x7ff17c0010b0 con 0x7ff18c0ef610 2021-02-10T11:17:13.923 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:13.910 7ff18bac0700 1 -- 172.21.2.112:0/1348744624 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff18c0e52e0 con 0x7ff18c0f20e0 2021-02-10T11:17:13.923 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:13.910 7ff18bac0700 1 --2- 172.21.2.112:0/1348744624 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff18c0f20e0 0x7ff18c0f0200 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 rx=0x7ff184006f00 tx=0x7ff184009e90).ready entity=mon.0 client_cookie=99cebc404019d575 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:17:13.923 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:13.910 7ff18a2bd700 1 -- 172.21.2.112:0/1348744624 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff18400b300 con 0x7ff18c0f20e0 2021-02-10T11:17:13.923 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:13.910 7ff18a2bd700 1 -- 172.21.2.112:0/1348744624 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff18400b580 con 0x7ff18c0f20e0 2021-02-10T11:17:13.923 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:13.910 7ff18a2bd700 1 -- 172.21.2.112:0/1348744624 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff18400b9d0 con 0x7ff18c0f20e0 2021-02-10T11:17:13.924 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:13.910 7ff1916b4700 1 -- 172.21.2.112:0/1348744624 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff18c0f20e0 msgr2=0x7ff18c0f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:13.924 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:13.910 7ff1916b4700 1 --2- 172.21.2.112:0/1348744624 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff18c0f20e0 0x7ff18c0f0200 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 rx=0x7ff184006f00 tx=0x7ff184009e90).stop 2021-02-10T11:17:13.924 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:13.911 7ff1916b4700 1 -- 172.21.2.112:0/1348744624 shutdown_connections 2021-02-10T11:17:13.924 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:13.911 7ff1916b4700 1 --2- 172.21.2.112:0/1348744624 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff18c0f20e0 0x7ff18c0f0200 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:13.924 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:13.911 7ff1916b4700 1 --2- 172.21.2.112:0/1348744624 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff18c1b0470 0x7ff18c1b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:17:13.924 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:13.911 7ff1916b4700 1 -- 172.21.2.112:0/1348744624 >> 172.21.2.112:0/1348744624 conn(0x7ff18c0ce3e0 msgr2=0x7ff18c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:17:13.924 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:13.911 7ff1916b4700 1 -- 172.21.2.112:0/1348744624 shutdown_connections 2021-02-10T11:17:13.924 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:13.911 7ff1916b4700 1 -- 172.21.2.112:0/1348744624 wait complete. 2021-02-10T11:17:13.925 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:13.925 7ff1916b4700 1 Processor -- start 2021-02-10T11:17:13.925 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:13.925 7ff1916b4700 1 -- start start 2021-02-10T11:17:13.926 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:13.926 7ff1916b4700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff18c190920 0x7ff18c08f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:13.926 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:13.926 7ff1916b4700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff18c190d20 0x7ff18c08fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:13.926 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:13.926 7ff1916b4700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff18c193270 0x7ff18c193670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:13.926 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:13.926 7ff1916b4700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff18c090470 con 0x7ff18c190920 2021-02-10T11:17:13.926 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:13.926 7ff1916b4700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff18c193c90 con 0x7ff18c190d20 2021-02-10T11:17:13.926 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:13.926 7ff1916b4700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff18c193e70 con 0x7ff18c193270 2021-02-10T11:17:13.926 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:13.926 7ff18b2bf700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff18c190920 0x7ff18c08f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:13.926 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:13.926 7ff18b2bf700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff18c190920 0x7ff18c08f600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:44066/0 (socket says 172.21.2.112:44066) 2021-02-10T11:17:13.926 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:13.926 7ff18b2bf700 1 -- 172.21.2.112:0/4033230022 learned_addr learned my addr 172.21.2.112:0/4033230022 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:17:13.926 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:13.926 7ff18b2bf700 1 -- 172.21.2.112:0/4033230022 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff18c193270 msgr2=0x7ff18c193670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:13.926 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:13.926 7ff18b2bf700 1 --2- 172.21.2.112:0/4033230022 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff18c193270 0x7ff18c193670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:17:13.926 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:13.926 7ff18b2bf700 1 -- 172.21.2.112:0/4033230022 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff18c190d20 msgr2=0x7ff18c08fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:13.927 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:13.926 7ff18b2bf700 1 --2- 172.21.2.112:0/4033230022 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff18c190d20 0x7ff18c08fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:17:13.927 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:13.926 7ff18b2bf700 1 -- 172.21.2.112:0/4033230022 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff18c194090 con 0x7ff18c190920 2021-02-10T11:17:13.927 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:13.927 7ff18b2bf700 1 --2- 172.21.2.112:0/4033230022 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff18c190920 0x7ff18c08f600 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7ff17400b5c0 tx=0x7ff17400b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:17:13.927 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:13.927 7ff183fff700 1 -- 172.21.2.112:0/4033230022 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff17400bd50 con 0x7ff18c190920 2021-02-10T11:17:13.927 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:13.927 7ff183fff700 1 -- 172.21.2.112:0/4033230022 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff174006030 con 0x7ff18c190920 2021-02-10T11:17:13.927 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:13.927 7ff183fff700 1 -- 172.21.2.112:0/4033230022 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff1740054d0 con 0x7ff18c190920 2021-02-10T11:17:13.927 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:13.927 7ff1916b4700 1 -- 172.21.2.112:0/4033230022 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff18c194090 con 0x7ff18c190920 2021-02-10T11:17:13.927 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:13.927 7ff1916b4700 1 -- 172.21.2.112:0/4033230022 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff18c1920b0 con 0x7ff18c190920 2021-02-10T11:17:13.927 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:13.927 7ff183fff700 1 -- 172.21.2.112:0/4033230022 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff174006210 con 0x7ff18c190920 2021-02-10T11:17:13.927 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:13.927 7ff183fff700 1 --2- 172.21.2.112:0/4033230022 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff1700186c0 0x7ff17001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:13.927 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:13.927 7ff183fff700 1 -- 172.21.2.112:0/4033230022 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7ff174032400 con 0x7ff18c190920 2021-02-10T11:17:13.927 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:13.928 7ff18aabe700 1 --2- 172.21.2.112:0/4033230022 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff1700186c0 0x7ff17001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:13.927 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:13.928 7ff18aabe700 1 --2- 172.21.2.112:0/4033230022 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff1700186c0 0x7ff17001abc0 crc :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:17:13.957 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:13.960 7ff1916b4700 1 -- 172.21.2.112:0/4033230022 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff18c105af0 con 0x7ff18c190920 2021-02-10T11:17:13.959 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:13.962 7ff183fff700 1 -- 172.21.2.112:0/4033230022 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff1740009b0 con 0x7ff18c190920 2021-02-10T11:17:14.085 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:14.088 7ff1916b4700 1 -- 172.21.2.112:0/4033230022 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348577", "target": ["mgr", ""]}) v1 -- 0x7ff18c1920b0 con 0x7ff1700186c0 2021-02-10T11:17:14.112 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:14.116 7ff183fff700 1 -- 172.21.2.112:0/4033230022 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7ff17c000940 con 0x7ff1700186c0 2021-02-10T11:17:14.119 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:14.121 7ff1916b4700 1 -- 172.21.2.112:0/4033230022 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff1700186c0 msgr2=0x7ff17001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:14.119 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:14.121 7ff1916b4700 1 --2- 172.21.2.112:0/4033230022 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff1700186c0 0x7ff17001abc0 crc :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:14.119 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:14.121 7ff1916b4700 1 -- 172.21.2.112:0/4033230022 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff18c190920 msgr2=0x7ff18c08f600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:14.119 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:14.121 7ff1916b4700 1 --2- 172.21.2.112:0/4033230022 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff18c190920 0x7ff18c08f600 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7ff17400b5c0 tx=0x7ff17400b960).stop 2021-02-10T11:17:14.119 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:14.121 7ff1916b4700 1 -- 172.21.2.112:0/4033230022 shutdown_connections 2021-02-10T11:17:14.120 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:14.121 7ff1916b4700 1 --2- 172.21.2.112:0/4033230022 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff1700186c0 0x7ff17001abc0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:14.120 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:14.121 7ff1916b4700 1 --2- 172.21.2.112:0/4033230022 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff18c190d20 0x7ff18c08fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:17:14.120 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:14.121 7ff1916b4700 1 --2- 172.21.2.112:0/4033230022 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff18c193270 0x7ff18c193670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:17:14.120 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:14.121 7ff1916b4700 1 --2- 172.21.2.112:0/4033230022 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff18c190920 0x7ff18c08f600 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:14.120 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:14.121 7ff1916b4700 1 -- 172.21.2.112:0/4033230022 >> 172.21.2.112:0/4033230022 conn(0x7ff18c0bf4e0 msgr2=0x7ff18c0be3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:17:14.120 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:14.121 7ff1916b4700 1 -- 172.21.2.112:0/4033230022 shutdown_connections 2021-02-10T11:17:14.121 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:14.121 7ff1916b4700 1 -- 172.21.2.112:0/4033230022 wait complete. 2021-02-10T11:17:14.127 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000348577 2021-02-10T11:17:14.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:14.543 7f0871b78700 1 Processor -- start 2021-02-10T11:17:14.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:14.543 7f0871b78700 1 -- start start 2021-02-10T11:17:14.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:14.544 7f0871b78700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f086c1b0470 0x7f086c1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:14.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:14.544 7f0871b78700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f086c0f0b50 con 0x7f086c0ef610 2021-02-10T11:17:14.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:14.544 7f0871b78700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f086c0f24e0 con 0x7f086c0f20e0 2021-02-10T11:17:14.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:14.544 7f0871b78700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f086c0e5ea0 con 0x7f086c1b0470 2021-02-10T11:17:14.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:14.544 7f086b7fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f086c1b0470 0x7f086c1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:14.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:14.544 7f086affd700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f086c0ef610 0x7f086c0efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:43534/0 (socket says 172.21.2.112:43534) 2021-02-10T11:17:14.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:14.544 7f086affd700 1 -- 172.21.2.112:0/3589142843 learned_addr learned my addr 172.21.2.112:0/3589142843 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:17:14.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:14.544 7f086a7fc700 1 -- 172.21.2.112:0/3589142843 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3235161698 0 0) 0x7f08600009b0 con 0x7f086c0ef610 2021-02-10T11:17:14.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:14.544 7f086a7fc700 1 -- 172.21.2.112:0/3589142843 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0854002fe0 con 0x7f086c0ef610 2021-02-10T11:17:14.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:14.545 7f086b7fe700 1 -- 172.21.2.112:0/3589142843 >> v1:172.21.2.120:6789/0 conn(0x7f086c0f20e0 legacy=0x7f086c0f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:14.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:14.545 7f086b7fe700 1 -- 172.21.2.112:0/3589142843 >> v1:172.21.2.112:6789/0 conn(0x7f086c0ef610 legacy=0x7f086c0efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:14.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:14.545 7f086b7fe700 1 -- 172.21.2.112:0/3589142843 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f086c0e52e0 con 0x7f086c1b0470 2021-02-10T11:17:14.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:14.545 7f086b7fe700 1 --2- 172.21.2.112:0/3589142843 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f086c1b0470 0x7f086c1b0870 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7f085c006650 tx=0x7f085c009a80).ready entity=mon.0 client_cookie=1197c315beb3bf1 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:17:14.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:14.545 7f086a7fc700 1 -- 172.21.2.112:0/3589142843 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f085c00c0c0 con 0x7f086c1b0470 2021-02-10T11:17:14.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:14.545 7f086a7fc700 1 -- 172.21.2.112:0/3589142843 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f085c00c340 con 0x7f086c1b0470 2021-02-10T11:17:14.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:14.545 7f086a7fc700 1 -- 172.21.2.112:0/3589142843 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f085c00c860 con 0x7f086c1b0470 2021-02-10T11:17:14.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:14.545 7f0871b78700 1 -- 172.21.2.112:0/3589142843 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f086c1b0470 msgr2=0x7f086c1b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:14.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:14.545 7f0871b78700 1 --2- 172.21.2.112:0/3589142843 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f086c1b0470 0x7f086c1b0870 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7f085c006650 tx=0x7f085c009a80).stop 2021-02-10T11:17:14.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:14.546 7f0871b78700 1 -- 172.21.2.112:0/3589142843 shutdown_connections 2021-02-10T11:17:14.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:14.546 7f0871b78700 1 --2- 172.21.2.112:0/3589142843 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f086c1b0470 0x7f086c1b0870 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:14.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:14.546 7f0871b78700 1 -- 172.21.2.112:0/3589142843 >> 172.21.2.112:0/3589142843 conn(0x7f086c0ce3e0 msgr2=0x7f086c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:17:14.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:14.546 7f0871b78700 1 -- 172.21.2.112:0/3589142843 shutdown_connections 2021-02-10T11:17:14.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:14.546 7f0871b78700 1 -- 172.21.2.112:0/3589142843 wait complete. 2021-02-10T11:17:14.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:14.561 7f0871b78700 1 Processor -- start 2021-02-10T11:17:14.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:14.561 7f0871b78700 1 -- start start 2021-02-10T11:17:14.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:14.561 7f0871b78700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f086c0a9690 0x7f086c0a9a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:14.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:14.561 7f0871b78700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f086c0aa0b0 0x7f086c0aa4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:14.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:14.561 7f0871b78700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f086c0f1800 0x7f086c0f1c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:14.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:14.561 7f0871b78700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f086c092350 con 0x7f086c0aa0b0 2021-02-10T11:17:14.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:14.561 7f0871b78700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f086c0a6540 con 0x7f086c0a9690 2021-02-10T11:17:14.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:14.561 7f0871b78700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f086c0a6720 con 0x7f086c0f1800 2021-02-10T11:17:14.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:14.561 7f086affd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f086c0aa0b0 0x7f086c0aa4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:14.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:14.561 7f086affd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f086c0aa0b0 0x7f086c0aa4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:44076/0 (socket says 172.21.2.112:44076) 2021-02-10T11:17:14.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:14.561 7f086affd700 1 -- 172.21.2.112:0/4246335424 learned_addr learned my addr 172.21.2.112:0/4246335424 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:17:14.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:14.562 7f086affd700 1 -- 172.21.2.112:0/4246335424 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f086c0f1800 msgr2=0x7f086c0f1c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:14.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:14.562 7f086affd700 1 --2- 172.21.2.112:0/4246335424 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f086c0f1800 0x7f086c0f1c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:17:14.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:14.562 7f086affd700 1 -- 172.21.2.112:0/4246335424 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f086c0a9690 msgr2=0x7f086c0a9a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:14.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:14.562 7f086affd700 1 --2- 172.21.2.112:0/4246335424 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f086c0a9690 0x7f086c0a9a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:17:14.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:14.562 7f086affd700 1 -- 172.21.2.112:0/4246335424 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f086c0a69e0 con 0x7f086c0aa0b0 2021-02-10T11:17:14.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:14.562 7f086affd700 1 --2- 172.21.2.112:0/4246335424 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f086c0aa0b0 0x7f086c0aa4b0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 rx=0x7f0860007650 tx=0x7f086000a910).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:17:14.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:14.562 7f0853fff700 1 -- 172.21.2.112:0/4246335424 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0860000ea0 con 0x7f086c0aa0b0 2021-02-10T11:17:14.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:14.562 7f0853fff700 1 -- 172.21.2.112:0/4246335424 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f086000ad00 con 0x7f086c0aa0b0 2021-02-10T11:17:14.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:14.562 7f0853fff700 1 -- 172.21.2.112:0/4246335424 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0860004540 con 0x7f086c0aa0b0 2021-02-10T11:17:14.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:14.562 7f0871b78700 1 -- 172.21.2.112:0/4246335424 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f086c0a69e0 con 0x7f086c0aa0b0 2021-02-10T11:17:14.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:14.563 7f0871b78700 1 -- 172.21.2.112:0/4246335424 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f086c0a69e0 con 0x7f086c0aa0b0 2021-02-10T11:17:14.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:14.563 7f0853fff700 1 -- 172.21.2.112:0/4246335424 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0860009030 con 0x7f086c0aa0b0 2021-02-10T11:17:14.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:14.563 7f0853fff700 1 --2- 172.21.2.112:0/4246335424 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f084c0186d0 0x7f084c01abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:14.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:14.563 7f0853fff700 1 -- 172.21.2.112:0/4246335424 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f0860032790 con 0x7f086c0aa0b0 2021-02-10T11:17:14.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:14.564 7f086b7fe700 1 --2- 172.21.2.112:0/4246335424 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f084c0186d0 0x7f084c01abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:14.563 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:14.564 7f086b7fe700 1 --2- 172.21.2.112:0/4246335424 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f084c0186d0 0x7f084c01abd0 crc :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:17:14.590 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:14.593 7f0871b78700 1 -- 172.21.2.112:0/4246335424 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f086c0edd10 con 0x7f086c0aa0b0 2021-02-10T11:17:14.592 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:14.596 7f0853fff700 1 -- 172.21.2.112:0/4246335424 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0860000de0 con 0x7f086c0aa0b0 2021-02-10T11:17:14.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:14.726 7f0871b78700 1 -- 172.21.2.112:0/4246335424 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348577", "target": ["mgr", ""]}) v1 -- 0x7f086c0e52e0 con 0x7f084c0186d0 2021-02-10T11:17:14.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:14.730 7f0853fff700 1 -- 172.21.2.112:0/4246335424 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f085c006060 con 0x7f084c0186d0 2021-02-10T11:17:14.730 INFO:teuthology.orchestra.run.gibba012.stdout:/volumes/_nogroup/subvolume_0000000000348577/f35d0189-3475-4731-a1c1-ea8eafaa4a61 2021-02-10T11:17:14.732 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:14.735 7f0871b78700 1 -- 172.21.2.112:0/4246335424 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f084c0186d0 msgr2=0x7f084c01abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:14.733 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:14.735 7f0871b78700 1 --2- 172.21.2.112:0/4246335424 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f084c0186d0 0x7f084c01abd0 crc :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:14.733 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:14.735 7f0871b78700 1 -- 172.21.2.112:0/4246335424 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f086c0aa0b0 msgr2=0x7f086c0aa4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:14.733 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:14.735 7f0871b78700 1 --2- 172.21.2.112:0/4246335424 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f086c0aa0b0 0x7f086c0aa4b0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 rx=0x7f0860007650 tx=0x7f086000a910).stop 2021-02-10T11:17:14.733 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:14.736 7f0871b78700 1 -- 172.21.2.112:0/4246335424 shutdown_connections 2021-02-10T11:17:14.733 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:14.736 7f0871b78700 1 --2- 172.21.2.112:0/4246335424 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f084c0186d0 0x7f084c01abd0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:14.733 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:14.736 7f0871b78700 1 --2- 172.21.2.112:0/4246335424 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f086c0f1800 0x7f086c0f1c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:17:14.733 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:14.736 7f0871b78700 1 --2- 172.21.2.112:0/4246335424 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f086c0aa0b0 0x7f086c0aa4b0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:14.733 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:14.736 7f0871b78700 1 --2- 172.21.2.112:0/4246335424 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f086c0a9690 0x7f086c0a9a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:17:14.733 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:14.736 7f0871b78700 1 -- 172.21.2.112:0/4246335424 >> 172.21.2.112:0/4246335424 conn(0x7f086c0c40e0 msgr2=0x7f086c094eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:17:14.734 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:14.736 7f0871b78700 1 -- 172.21.2.112:0/4246335424 shutdown_connections 2021-02-10T11:17:14.734 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:14.736 7f0871b78700 1 -- 172.21.2.112:0/4246335424 wait complete. 2021-02-10T11:17:14.741 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000348577 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000348577/f35d0189-3475-4731-a1c1-ea8eafaa4a61 2021-02-10T11:17:14.741 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:14.741 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:14.742 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:15.078 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:17:15.078 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:15.078 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348577/f35d0189-3475-4731-a1c1-ea8eafaa4a61/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:15.458 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:17:15.458 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:17:15.458 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0250005 s, 41.9 MB/s 2021-02-10T11:17:15.459 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:15.459 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:15.459 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:15.794 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:17:15.794 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:15.794 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348577/f35d0189-3475-4731-a1c1-ea8eafaa4a61/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:16.169 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:17:16.169 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:17:16.169 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0239688 s, 43.7 MB/s 2021-02-10T11:17:16.170 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:16.170 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:16.170 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:16.504 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:17:16.504 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:16.505 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348577/f35d0189-3475-4731-a1c1-ea8eafaa4a61/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:16.880 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:17:16.880 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:17:16.880 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0259199 s, 40.5 MB/s 2021-02-10T11:17:16.881 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:16.881 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:16.881 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:17.216 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:17:17.216 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:17.216 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348577/f35d0189-3475-4731-a1c1-ea8eafaa4a61/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:17.590 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:17:17.590 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:17:17.590 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0240659 s, 43.6 MB/s 2021-02-10T11:17:17.591 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:17.591 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:17.591 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:17.926 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:17:17.927 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:17.927 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348577/f35d0189-3475-4731-a1c1-ea8eafaa4a61/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:18.303 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:17:18.304 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:17:18.305 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0265924 s, 39.4 MB/s 2021-02-10T11:17:18.305 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:18.306 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:18.306 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:18.641 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:17:18.642 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:18.642 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348577/f35d0189-3475-4731-a1c1-ea8eafaa4a61/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:19.019 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:17:19.019 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:17:19.020 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0254376 s, 41.2 MB/s 2021-02-10T11:17:19.020 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:19.021 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:19.021 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:19.354 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:17:19.354 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:19.354 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348577/f35d0189-3475-4731-a1c1-ea8eafaa4a61/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:19.725 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:17:19.726 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:17:19.726 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0250171 s, 41.9 MB/s 2021-02-10T11:17:19.727 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:19.727 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:19.727 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:20.061 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:17:20.061 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:20.061 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348577/f35d0189-3475-4731-a1c1-ea8eafaa4a61/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:20.438 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:17:20.438 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:17:20.438 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.025446 s, 41.2 MB/s 2021-02-10T11:17:20.439 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:20.440 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:20.440 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:20.775 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:17:20.775 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:20.775 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348577/f35d0189-3475-4731-a1c1-ea8eafaa4a61/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:21.150 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:17:21.151 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:17:21.151 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0236399 s, 44.4 MB/s 2021-02-10T11:17:21.152 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:21.152 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:21.152 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:21.487 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:17:21.487 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:21.487 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348577/f35d0189-3475-4731-a1c1-ea8eafaa4a61/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:21.861 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:17:21.861 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:17:21.861 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0245601 s, 42.7 MB/s 2021-02-10T11:17:21.862 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000348578 2021-02-10T11:17:22.291 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.278 7f54390d8700 1 Processor -- start 2021-02-10T11:17:22.291 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.278 7f54390d8700 1 -- start start 2021-02-10T11:17:22.291 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.279 7f54390d8700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f54341b0870 0x7f54340f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:22.291 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.279 7f54390d8700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f54340efe00 0x7f54340f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:22.291 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.279 7f54390d8700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54341b0c70 con 0x7f54341b0470 2021-02-10T11:17:22.292 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.279 7f54390d8700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54340e5ea0 con 0x7f54340efe00 2021-02-10T11:17:22.292 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.279 7f54390d8700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54340f0b50 con 0x7f54341b0870 2021-02-10T11:17:22.292 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.279 7f543359e700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f54340efe00 0x7f54340f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:22.292 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.279 7f5432d9d700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f54341b0470 0x7f54340ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:43546/0 (socket says 172.21.2.112:43546) 2021-02-10T11:17:22.292 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.279 7f543359e700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f54340efe00 0x7f54340f0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:44086/0 (socket says 172.21.2.112:44086) 2021-02-10T11:17:22.292 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.279 7f5432d9d700 1 -- 172.21.2.112:0/4224912184 learned_addr learned my addr 172.21.2.112:0/4224912184 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:17:22.292 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.279 7f543259c700 1 --2- 172.21.2.112:0/4224912184 >> v2:172.21.2.120:3300/0 conn(0x7f54341b0870 0x7f54340f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:22.292 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.279 7f543359e700 1 -- 172.21.2.112:0/4224912184 >> v2:172.21.2.120:3300/0 conn(0x7f54341b0870 msgr2=0x7f54340f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:17:22.292 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.279 7f543359e700 1 --2- 172.21.2.112:0/4224912184 >> v2:172.21.2.120:3300/0 conn(0x7f54341b0870 0x7f54340f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:22.292 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.279 7f543359e700 1 -- 172.21.2.112:0/4224912184 >> v1:172.21.2.112:6789/0 conn(0x7f54341b0470 legacy=0x7f54340ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:22.292 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.279 7f543359e700 1 -- 172.21.2.112:0/4224912184 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54340f0820 con 0x7f54340efe00 2021-02-10T11:17:22.292 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.279 7f543359e700 1 --2- 172.21.2.112:0/4224912184 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f54340efe00 0x7f54340f0200 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 rx=0x7f5424006ab0 tx=0x7f5424009ac0).ready entity=mon.0 client_cookie=7783a18467b540a3 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:17:22.292 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.279 7f5431d9b700 1 -- 172.21.2.112:0/4224912184 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f54240066d0 con 0x7f54340efe00 2021-02-10T11:17:22.292 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.279 7f5431d9b700 1 -- 172.21.2.112:0/4224912184 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54240022e0 con 0x7f54340efe00 2021-02-10T11:17:22.293 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.279 7f5431d9b700 1 -- 172.21.2.112:0/4224912184 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f54240027c0 con 0x7f54340efe00 2021-02-10T11:17:22.293 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.280 7f54390d8700 1 -- 172.21.2.112:0/4224912184 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f54340efe00 msgr2=0x7f54340f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:22.293 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.280 7f54390d8700 1 --2- 172.21.2.112:0/4224912184 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f54340efe00 0x7f54340f0200 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 rx=0x7f5424006ab0 tx=0x7f5424009ac0).stop 2021-02-10T11:17:22.293 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.280 7f54390d8700 1 -- 172.21.2.112:0/4224912184 shutdown_connections 2021-02-10T11:17:22.293 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.280 7f54390d8700 1 --2- 172.21.2.112:0/4224912184 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f54340efe00 0x7f54340f0200 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:22.293 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.280 7f54390d8700 1 --2- 172.21.2.112:0/4224912184 >> v2:172.21.2.120:3300/0 conn(0x7f54341b0870 0x7f54340f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:22.293 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.280 7f54390d8700 1 -- 172.21.2.112:0/4224912184 >> 172.21.2.112:0/4224912184 conn(0x7f54340ce3e0 msgr2=0x7f54340e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:17:22.293 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.280 7f54390d8700 1 -- 172.21.2.112:0/4224912184 shutdown_connections 2021-02-10T11:17:22.293 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.280 7f54390d8700 1 -- 172.21.2.112:0/4224912184 wait complete. 2021-02-10T11:17:22.295 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.295 7f54390d8700 1 Processor -- start 2021-02-10T11:17:22.295 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.295 7f54390d8700 1 -- start start 2021-02-10T11:17:22.295 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.295 7f54390d8700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f54340a0420 0x7f54340a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:22.295 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.295 7f54390d8700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5434189d00 0x7f543418a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:22.295 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.295 7f54390d8700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f543418a720 0x7f54340b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:22.295 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.295 7f54390d8700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5434189800 con 0x7f54340a0420 2021-02-10T11:17:22.295 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.295 7f54390d8700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f543418ab20 con 0x7f5434189d00 2021-02-10T11:17:22.295 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.295 7f54390d8700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54340a1140 con 0x7f543418a720 2021-02-10T11:17:22.295 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.295 7f5432d9d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f54340a0420 0x7f54340a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:22.296 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.295 7f5432d9d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f54340a0420 0x7f54340a0820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:44092/0 (socket says 172.21.2.112:44092) 2021-02-10T11:17:22.296 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.295 7f543259c700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5434189d00 0x7f543418a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:22.296 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.295 7f5432d9d700 1 -- 172.21.2.112:0/2130730974 learned_addr learned my addr 172.21.2.112:0/2130730974 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:17:22.296 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.295 7f543359e700 1 --2- 172.21.2.112:0/2130730974 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f543418a720 0x7f54340b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:22.296 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.296 7f543259c700 1 -- 172.21.2.112:0/2130730974 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f543418a720 msgr2=0x7f54340b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:22.296 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.296 7f543259c700 1 --2- 172.21.2.112:0/2130730974 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f543418a720 0x7f54340b7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:22.296 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.296 7f543259c700 1 -- 172.21.2.112:0/2130730974 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f54340a0420 msgr2=0x7f54340a0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:22.296 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.296 7f543259c700 1 --2- 172.21.2.112:0/2130730974 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f54340a0420 0x7f54340a0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:22.296 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.296 7f543259c700 1 -- 172.21.2.112:0/2130730974 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54340b8200 con 0x7f5434189d00 2021-02-10T11:17:22.296 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.296 7f543259c700 1 --2- 172.21.2.112:0/2130730974 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5434189d00 0x7f543418a100 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f542c006dc0 tx=0x7f542c00bc90).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:17:22.296 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.296 7f54237fe700 1 -- 172.21.2.112:0/2130730974 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f542c009140 con 0x7f5434189d00 2021-02-10T11:17:22.296 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.296 7f54237fe700 1 -- 172.21.2.112:0/2130730974 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f542c006810 con 0x7f5434189d00 2021-02-10T11:17:22.296 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.296 7f54390d8700 1 -- 172.21.2.112:0/2130730974 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54340b8200 con 0x7f5434189d00 2021-02-10T11:17:22.297 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.296 7f54237fe700 1 -- 172.21.2.112:0/2130730974 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f542c00c570 con 0x7f5434189d00 2021-02-10T11:17:22.297 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.296 7f54390d8700 1 -- 172.21.2.112:0/2130730974 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54340b8790 con 0x7f5434189d00 2021-02-10T11:17:22.297 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.297 7f54237fe700 1 -- 172.21.2.112:0/2130730974 <== mon.1 v2:172.21.2.120:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f542c005660 con 0x7f5434189d00 2021-02-10T11:17:22.297 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.297 7f54237fe700 1 --2- 172.21.2.112:0/2130730974 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f54180186c0 0x7f541801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:22.297 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.297 7f54237fe700 1 -- 172.21.2.112:0/2130730974 <== mon.1 v2:172.21.2.120:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f542c032810 con 0x7f5434189d00 2021-02-10T11:17:22.297 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.297 7f5432d9d700 1 --2- 172.21.2.112:0/2130730974 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f54180186c0 0x7f541801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:22.297 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.298 7f5432d9d700 1 --2- 172.21.2.112:0/2130730974 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f54180186c0 0x7f541801abc0 crc :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:17:22.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.328 7f54390d8700 1 -- 172.21.2.112:0/2130730974 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5434105ae0 con 0x7f5434189d00 2021-02-10T11:17:22.327 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.331 7f54237fe700 1 -- 172.21.2.112:0/2130730974 <== mon.1 v2:172.21.2.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f542c0090f0 con 0x7f5434189d00 2021-02-10T11:17:22.453 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.456 7f54390d8700 1 -- 172.21.2.112:0/2130730974 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348578", "target": ["mgr", ""]}) v1 -- 0x7f54340b9020 con 0x7f54180186c0 2021-02-10T11:17:22.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.480 7f54237fe700 1 -- 172.21.2.112:0/2130730974 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f5428000c90 con 0x7f54180186c0 2021-02-10T11:17:22.486 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.489 7f54390d8700 1 -- 172.21.2.112:0/2130730974 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f54180186c0 msgr2=0x7f541801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:22.486 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.489 7f54390d8700 1 --2- 172.21.2.112:0/2130730974 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f54180186c0 0x7f541801abc0 crc :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:22.486 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.489 7f54390d8700 1 -- 172.21.2.112:0/2130730974 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5434189d00 msgr2=0x7f543418a100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:22.487 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.489 7f54390d8700 1 --2- 172.21.2.112:0/2130730974 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5434189d00 0x7f543418a100 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f542c006dc0 tx=0x7f542c00bc90).stop 2021-02-10T11:17:22.487 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.489 7f54390d8700 1 -- 172.21.2.112:0/2130730974 shutdown_connections 2021-02-10T11:17:22.487 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.489 7f54390d8700 1 --2- 172.21.2.112:0/2130730974 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f54180186c0 0x7f541801abc0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:22.487 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.489 7f54390d8700 1 --2- 172.21.2.112:0/2130730974 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5434189d00 0x7f543418a100 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:22.487 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.489 7f54390d8700 1 --2- 172.21.2.112:0/2130730974 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f543418a720 0x7f54340b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:22.487 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.489 7f54390d8700 1 --2- 172.21.2.112:0/2130730974 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f54340a0420 0x7f54340a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:22.487 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.489 7f54390d8700 1 -- 172.21.2.112:0/2130730974 >> 172.21.2.112:0/2130730974 conn(0x7f5434094e10 msgr2=0x7f54340a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:17:22.487 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.489 7f54390d8700 1 -- 172.21.2.112:0/2130730974 shutdown_connections 2021-02-10T11:17:22.487 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.489 7f54390d8700 1 -- 172.21.2.112:0/2130730974 wait complete. 2021-02-10T11:17:22.494 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000348578 2021-02-10T11:17:22.925 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.910 7fd413b31700 1 Processor -- start 2021-02-10T11:17:22.926 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.910 7fd413b31700 1 -- start start 2021-02-10T11:17:22.926 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.910 7fd413b31700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fd40c1b0870 0x7fd40c0f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:22.926 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.910 7fd413b31700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd40c0efe00 0x7fd40c0f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:22.926 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.910 7fd413b31700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd40c1b0c70 con 0x7fd40c1b0470 2021-02-10T11:17:22.926 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.910 7fd413b31700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd40c0f0b50 con 0x7fd40c1b0870 2021-02-10T11:17:22.926 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.910 7fd413b31700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd40c0f0820 con 0x7fd40c0efe00 2021-02-10T11:17:22.926 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.911 7fd41194d700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fd40c1b0470 0x7fd40c0ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:43560/0 (socket says 172.21.2.112:43560) 2021-02-10T11:17:22.926 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.911 7fd41194d700 1 -- 172.21.2.112:0/4078791552 learned_addr learned my addr 172.21.2.112:0/4078791552 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:17:22.926 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.911 7fd41114c700 1 --2- 172.21.2.112:0/4078791552 >> v2:172.21.2.120:3300/0 conn(0x7fd40c1b0870 0x7fd40c0f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:22.926 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.911 7fd41214e700 1 --2- 172.21.2.112:0/4078791552 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd40c0efe00 0x7fd40c0f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:22.926 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.911 7fd41094b700 1 -- 172.21.2.112:0/4078791552 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1706221443 0 0) 0x7fd400001180 con 0x7fd40c1b0470 2021-02-10T11:17:22.926 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.911 7fd41094b700 1 -- 172.21.2.112:0/4078791552 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3f8002fe0 con 0x7fd40c1b0470 2021-02-10T11:17:22.927 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.912 7fd41114c700 1 -- 172.21.2.112:0/4078791552 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd40c0efe00 msgr2=0x7fd40c0f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:17:22.927 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.912 7fd41114c700 1 --2- 172.21.2.112:0/4078791552 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd40c0efe00 0x7fd40c0f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:22.927 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.912 7fd41114c700 1 -- 172.21.2.112:0/4078791552 >> v1:172.21.2.112:6789/0 conn(0x7fd40c1b0470 legacy=0x7fd40c0ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:22.927 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.912 7fd41114c700 1 -- 172.21.2.112:0/4078791552 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd40c0e52e0 con 0x7fd40c1b0870 2021-02-10T11:17:22.927 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.912 7fd41114c700 1 --2- 172.21.2.112:0/4078791552 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd40c1b0870 0x7fd40c0f20e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7fd408006650 tx=0x7fd408009a80).ready entity=mon.1 client_cookie=de511811c2c59fbc server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:17:22.927 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.912 7fd41094b700 1 -- 172.21.2.112:0/4078791552 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd40800c0c0 con 0x7fd40c1b0870 2021-02-10T11:17:22.927 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.912 7fd41094b700 1 -- 172.21.2.112:0/4078791552 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd40800c340 con 0x7fd40c1b0870 2021-02-10T11:17:22.927 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.912 7fd41094b700 1 -- 172.21.2.112:0/4078791552 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd40800c860 con 0x7fd40c1b0870 2021-02-10T11:17:22.927 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.912 7fd413b31700 1 -- 172.21.2.112:0/4078791552 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd40c1b0870 msgr2=0x7fd40c0f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:22.927 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.912 7fd413b31700 1 --2- 172.21.2.112:0/4078791552 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd40c1b0870 0x7fd40c0f20e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7fd408006650 tx=0x7fd408009a80).stop 2021-02-10T11:17:22.927 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.913 7fd413b31700 1 -- 172.21.2.112:0/4078791552 shutdown_connections 2021-02-10T11:17:22.927 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.913 7fd413b31700 1 --2- 172.21.2.112:0/4078791552 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd40c0efe00 0x7fd40c0f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:22.927 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.913 7fd413b31700 1 --2- 172.21.2.112:0/4078791552 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd40c1b0870 0x7fd40c0f20e0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:22.928 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.913 7fd413b31700 1 -- 172.21.2.112:0/4078791552 >> 172.21.2.112:0/4078791552 conn(0x7fd40c0ce3e0 msgr2=0x7fd40c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:17:22.928 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.913 7fd413b31700 1 -- 172.21.2.112:0/4078791552 shutdown_connections 2021-02-10T11:17:22.928 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.913 7fd413b31700 1 -- 172.21.2.112:0/4078791552 wait complete. 2021-02-10T11:17:22.928 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.928 7fd413b31700 1 Processor -- start 2021-02-10T11:17:22.928 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.928 7fd413b31700 1 -- start start 2021-02-10T11:17:22.928 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.928 7fd413b31700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd40c0a0420 0x7fd40c0a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:22.928 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.928 7fd413b31700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd40c189d00 0x7fd40c18a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:22.928 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.928 7fd413b31700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd40c18a720 0x7fd40c0b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:22.928 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.928 7fd413b31700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd40c189800 con 0x7fd40c18a720 2021-02-10T11:17:22.928 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.928 7fd413b31700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd40c18ab20 con 0x7fd40c189d00 2021-02-10T11:17:22.928 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.928 7fd413b31700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd40c0a1140 con 0x7fd40c0a0420 2021-02-10T11:17:22.928 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.928 7fd41214e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd40c18a720 0x7fd40c0b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:22.928 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.928 7fd41214e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd40c18a720 0x7fd40c0b7be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:44103/0 (socket says 172.21.2.112:44103) 2021-02-10T11:17:22.928 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.928 7fd41214e700 1 -- 172.21.2.112:0/3277279889 learned_addr learned my addr 172.21.2.112:0/3277279889 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:17:22.929 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.929 7fd41214e700 1 -- 172.21.2.112:0/3277279889 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd40c0a0420 msgr2=0x7fd40c0a0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:22.929 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.929 7fd41214e700 1 --2- 172.21.2.112:0/3277279889 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd40c0a0420 0x7fd40c0a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:17:22.929 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.929 7fd41214e700 1 -- 172.21.2.112:0/3277279889 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd40c189d00 msgr2=0x7fd40c18a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:22.929 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.929 7fd41214e700 1 --2- 172.21.2.112:0/3277279889 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd40c189d00 0x7fd40c18a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:17:22.929 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.929 7fd41214e700 1 -- 172.21.2.112:0/3277279889 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd40c0b8200 con 0x7fd40c18a720 2021-02-10T11:17:22.930 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.929 7fd41214e700 1 --2- 172.21.2.112:0/3277279889 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd40c18a720 0x7fd40c0b7be0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 rx=0x7fd40400b970 tx=0x7fd40400bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:17:22.930 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.929 7fd3fe7fc700 1 -- 172.21.2.112:0/3277279889 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd404000f90 con 0x7fd40c18a720 2021-02-10T11:17:22.930 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.929 7fd3fe7fc700 1 -- 172.21.2.112:0/3277279889 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd404006aa0 con 0x7fd40c18a720 2021-02-10T11:17:22.930 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.929 7fd413b31700 1 -- 172.21.2.112:0/3277279889 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd40c0b8200 con 0x7fd40c18a720 2021-02-10T11:17:22.930 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.929 7fd3fe7fc700 1 -- 172.21.2.112:0/3277279889 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd404001680 con 0x7fd40c18a720 2021-02-10T11:17:22.930 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.929 7fd413b31700 1 -- 172.21.2.112:0/3277279889 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd40c0b8200 con 0x7fd40c18a720 2021-02-10T11:17:22.930 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.930 7fd3fe7fc700 1 -- 172.21.2.112:0/3277279889 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd404000df0 con 0x7fd40c18a720 2021-02-10T11:17:22.930 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.930 7fd3fe7fc700 1 --2- 172.21.2.112:0/3277279889 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd3f4018730 0x7fd3f401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:22.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.930 7fd3fe7fc700 1 -- 172.21.2.112:0/3277279889 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fd404032b40 con 0x7fd40c18a720 2021-02-10T11:17:22.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.930 7fd41194d700 1 --2- 172.21.2.112:0/3277279889 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd3f4018730 0x7fd3f401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:22.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.931 7fd41194d700 1 --2- 172.21.2.112:0/3277279889 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd3f4018730 0x7fd3f401ac10 crc :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:17:22.957 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.960 7fd413b31700 1 -- 172.21.2.112:0/3277279889 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd40c105ae0 con 0x7fd40c18a720 2021-02-10T11:17:22.960 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:22.963 7fd3fe7fc700 1 -- 172.21.2.112:0/3277279889 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd404001650 con 0x7fd40c18a720 2021-02-10T11:17:23.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:23.092 7fd413b31700 1 -- 172.21.2.112:0/3277279889 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348578", "target": ["mgr", ""]}) v1 -- 0x7fd40c105ae0 con 0x7fd3f4018730 2021-02-10T11:17:23.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:23.096 7fd3fe7fc700 1 -- 172.21.2.112:0/3277279889 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7fd4000077a0 con 0x7fd3f4018730 2021-02-10T11:17:23.096 INFO:teuthology.orchestra.run.gibba012.stdout:/volumes/_nogroup/subvolume_0000000000348578/b053f220-3eba-4240-b160-9dc38734693a 2021-02-10T11:17:23.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:23.101 7fd413b31700 1 -- 172.21.2.112:0/3277279889 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd3f4018730 msgr2=0x7fd3f401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:23.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:23.101 7fd413b31700 1 --2- 172.21.2.112:0/3277279889 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd3f4018730 0x7fd3f401ac10 crc :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:23.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:23.101 7fd413b31700 1 -- 172.21.2.112:0/3277279889 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd40c18a720 msgr2=0x7fd40c0b7be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:23.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:23.101 7fd413b31700 1 --2- 172.21.2.112:0/3277279889 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd40c18a720 0x7fd40c0b7be0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 rx=0x7fd40400b970 tx=0x7fd40400bd10).stop 2021-02-10T11:17:23.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:23.102 7fd413b31700 1 -- 172.21.2.112:0/3277279889 shutdown_connections 2021-02-10T11:17:23.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:23.102 7fd413b31700 1 --2- 172.21.2.112:0/3277279889 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd3f4018730 0x7fd3f401ac10 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:23.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:23.102 7fd413b31700 1 --2- 172.21.2.112:0/3277279889 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd40c189d00 0x7fd40c18a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:17:23.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:23.102 7fd413b31700 1 --2- 172.21.2.112:0/3277279889 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd40c18a720 0x7fd40c0b7be0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:23.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:23.102 7fd413b31700 1 --2- 172.21.2.112:0/3277279889 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd40c0a0420 0x7fd40c0a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:17:23.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:23.102 7fd413b31700 1 -- 172.21.2.112:0/3277279889 >> 172.21.2.112:0/3277279889 conn(0x7fd40c094e10 msgr2=0x7fd40c0a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:17:23.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:23.102 7fd413b31700 1 -- 172.21.2.112:0/3277279889 shutdown_connections 2021-02-10T11:17:23.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:23.102 7fd413b31700 1 -- 172.21.2.112:0/3277279889 wait complete. 2021-02-10T11:17:23.107 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000348578 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000348578/b053f220-3eba-4240-b160-9dc38734693a 2021-02-10T11:17:23.107 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:23.107 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:23.107 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:23.442 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:17:23.443 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:23.443 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348578/b053f220-3eba-4240-b160-9dc38734693a/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:23.826 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:17:24.145 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:17:24.145 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0256796 s, 40.8 MB/s 2021-02-10T11:17:24.147 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:24.147 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:24.147 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:24.481 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:17:24.481 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:24.481 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348578/b053f220-3eba-4240-b160-9dc38734693a/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:24.852 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:17:24.852 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:17:24.852 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0234376 s, 44.7 MB/s 2021-02-10T11:17:24.853 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:24.853 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:24.853 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:25.188 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:17:25.414 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:25.415 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348578/b053f220-3eba-4240-b160-9dc38734693a/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:25.790 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:17:25.790 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:17:25.790 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0231881 s, 45.2 MB/s 2021-02-10T11:17:25.791 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:25.791 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:25.791 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:26.126 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:17:26.127 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:26.127 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348578/b053f220-3eba-4240-b160-9dc38734693a/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:26.504 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:17:26.504 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:17:26.504 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0261707 s, 40.1 MB/s 2021-02-10T11:17:26.505 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:26.505 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:26.505 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:26.839 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:17:26.840 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:26.840 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348578/b053f220-3eba-4240-b160-9dc38734693a/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:27.216 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:17:27.216 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:17:27.216 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0237835 s, 44.1 MB/s 2021-02-10T11:17:27.218 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:27.218 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:27.218 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:27.555 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:17:27.555 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:27.555 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348578/b053f220-3eba-4240-b160-9dc38734693a/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:27.931 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:17:27.932 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:17:27.932 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.022926 s, 45.7 MB/s 2021-02-10T11:17:27.933 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:27.933 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:27.933 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:28.269 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:17:28.270 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:28.270 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348578/b053f220-3eba-4240-b160-9dc38734693a/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:28.652 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:17:28.652 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:17:28.653 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0264772 s, 39.6 MB/s 2021-02-10T11:17:28.653 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:28.654 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:28.654 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:28.989 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:17:28.990 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:28.990 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348578/b053f220-3eba-4240-b160-9dc38734693a/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:29.361 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:17:29.361 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:17:29.361 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0239421 s, 43.8 MB/s 2021-02-10T11:17:29.363 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:29.363 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:29.363 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:29.697 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:17:29.697 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:29.697 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348578/b053f220-3eba-4240-b160-9dc38734693a/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:30.071 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:17:30.072 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:17:30.072 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0244642 s, 42.9 MB/s 2021-02-10T11:17:30.073 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:30.073 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:30.073 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:30.406 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:17:30.406 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:30.406 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348578/b053f220-3eba-4240-b160-9dc38734693a/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:30.779 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:17:30.780 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:17:30.780 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0247591 s, 42.4 MB/s 2021-02-10T11:17:30.781 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000348579 2021-02-10T11:17:31.211 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.197 7f466ff4e700 1 Processor -- start 2021-02-10T11:17:31.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.197 7f466ff4e700 1 -- start start 2021-02-10T11:17:31.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.197 7f466ff4e700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f46681b0470 0x7f46681b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:31.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.197 7f466ff4e700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f46680ef610 0x7f46680efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:31.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.197 7f466ff4e700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46680f0b50 con 0x7f46680f0030 2021-02-10T11:17:31.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.197 7f466ff4e700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46680e52e0 con 0x7f46680ef610 2021-02-10T11:17:31.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.197 7f466ff4e700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46680e5490 con 0x7f46681b0470 2021-02-10T11:17:31.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.198 7f466d569700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f46680ef610 0x7f46680efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:31.213 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.198 7f466dd6a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f46681b0470 0x7f46681b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:31.213 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.198 7f466d569700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f46680ef610 0x7f46680efa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3300/0 says I am v2:172.21.2.112:46388/0 (socket says 172.21.2.112:46388) 2021-02-10T11:17:31.213 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.199 7f466d569700 1 -- 172.21.2.112:0/1023206785 learned_addr learned my addr 172.21.2.112:0/1023206785 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:17:31.213 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.199 7f466d569700 1 -- 172.21.2.112:0/1023206785 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f46681b0470 msgr2=0x7f46681b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:17:31.213 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.199 7f466d569700 1 --2- 172.21.2.112:0/1023206785 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f46681b0470 0x7f46681b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:31.213 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.199 7f466d569700 1 -- 172.21.2.112:0/1023206785 >> v1:172.21.2.120:6789/0 conn(0x7f46680f0030 legacy=0x7f46680f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:31.213 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.199 7f466d569700 1 -- 172.21.2.112:0/1023206785 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46680f0430 con 0x7f46680ef610 2021-02-10T11:17:31.213 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.199 7f466dd6a700 1 --2- 172.21.2.112:0/1023206785 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f46681b0470 0x7f46681b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:17:31.213 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.199 7f466d569700 1 --2- 172.21.2.112:0/1023206785 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f46680ef610 0x7f46680efa10 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f4664006700 tx=0x7f4664009b80).ready entity=mon.1 client_cookie=12b691c5d55eca15 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:17:31.213 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.199 7f466cd68700 1 -- 172.21.2.112:0/1023206785 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4664006340 con 0x7f46680ef610 2021-02-10T11:17:31.214 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.199 7f466cd68700 1 -- 172.21.2.112:0/1023206785 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f466400c0c0 con 0x7f46680ef610 2021-02-10T11:17:31.214 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.199 7f466cd68700 1 -- 172.21.2.112:0/1023206785 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f466400c5e0 con 0x7f46680ef610 2021-02-10T11:17:31.214 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.199 7f466ff4e700 1 -- 172.21.2.112:0/1023206785 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f46680ef610 msgr2=0x7f46680efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:31.214 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.199 7f466ff4e700 1 --2- 172.21.2.112:0/1023206785 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f46680ef610 0x7f46680efa10 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f4664006700 tx=0x7f4664009b80).stop 2021-02-10T11:17:31.214 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.200 7f466ff4e700 1 -- 172.21.2.112:0/1023206785 shutdown_connections 2021-02-10T11:17:31.214 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.200 7f466ff4e700 1 --2- 172.21.2.112:0/1023206785 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f46680ef610 0x7f46680efa10 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:31.214 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.200 7f466ff4e700 1 --2- 172.21.2.112:0/1023206785 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f46681b0470 0x7f46681b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:31.214 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.200 7f466ff4e700 1 -- 172.21.2.112:0/1023206785 >> 172.21.2.112:0/1023206785 conn(0x7f46680ce3e0 msgr2=0x7f46680e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:17:31.214 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.200 7f466ff4e700 1 -- 172.21.2.112:0/1023206785 shutdown_connections 2021-02-10T11:17:31.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.200 7f466ff4e700 1 -- 172.21.2.112:0/1023206785 wait complete. 2021-02-10T11:17:31.216 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.215 7f466ff4e700 1 Processor -- start 2021-02-10T11:17:31.216 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.215 7f466ff4e700 1 -- start start 2021-02-10T11:17:31.216 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.215 7f466ff4e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f46680a0420 0x7f46680a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:31.216 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.215 7f466ff4e700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4668189d00 0x7f466818a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:31.216 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.215 7f466ff4e700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f466818a720 0x7f46680b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:31.216 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.215 7f466ff4e700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4668189780 con 0x7f46680a0420 2021-02-10T11:17:31.216 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.215 7f466ff4e700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f466818ab20 con 0x7f466818a720 2021-02-10T11:17:31.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.215 7f466ff4e700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46680a1140 con 0x7f4668189d00 2021-02-10T11:17:31.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.216 7f466dd6a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f46680a0420 0x7f46680a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:31.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.216 7f466dd6a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f46680a0420 0x7f46680a0820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:44110/0 (socket says 172.21.2.112:44110) 2021-02-10T11:17:31.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.216 7f466dd6a700 1 -- 172.21.2.112:0/1495644565 learned_addr learned my addr 172.21.2.112:0/1495644565 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:17:31.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.216 7f466dd6a700 1 -- 172.21.2.112:0/1495644565 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4668189d00 msgr2=0x7f466818a100 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:17:31.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.216 7f466dd6a700 1 --2- 172.21.2.112:0/1495644565 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4668189d00 0x7f466818a100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:17:31.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.216 7f466dd6a700 1 -- 172.21.2.112:0/1495644565 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f466818a720 msgr2=0x7f46680b7be0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:17:31.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.216 7f466dd6a700 1 --2- 172.21.2.112:0/1495644565 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f466818a720 0x7f46680b7be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:17:31.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.216 7f466dd6a700 1 -- 172.21.2.112:0/1495644565 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46680b8200 con 0x7f46680a0420 2021-02-10T11:17:31.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.216 7f466dd6a700 1 --2- 172.21.2.112:0/1495644565 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f46680a0420 0x7f46680a0820 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7f465c006f40 tx=0x7f465c00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:17:31.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.216 7f465a7fc700 1 -- 172.21.2.112:0/1495644565 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f465c00bd80 con 0x7f46680a0420 2021-02-10T11:17:31.218 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.216 7f465a7fc700 1 -- 172.21.2.112:0/1495644565 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f465c000c90 con 0x7f46680a0420 2021-02-10T11:17:31.218 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.216 7f466ff4e700 1 -- 172.21.2.112:0/1495644565 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46680b8200 con 0x7f46680a0420 2021-02-10T11:17:31.218 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.216 7f465a7fc700 1 -- 172.21.2.112:0/1495644565 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f465c002220 con 0x7f46680a0420 2021-02-10T11:17:31.218 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.216 7f466ff4e700 1 -- 172.21.2.112:0/1495644565 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46680b8200 con 0x7f46680a0420 2021-02-10T11:17:31.218 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.217 7f465a7fc700 1 -- 172.21.2.112:0/1495644565 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f465c00a000 con 0x7f46680a0420 2021-02-10T11:17:31.218 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.217 7f465a7fc700 1 --2- 172.21.2.112:0/1495644565 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f46500186c0 0x7f465001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:31.218 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.217 7f465a7fc700 1 -- 172.21.2.112:0/1495644565 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f465c032530 con 0x7f46680a0420 2021-02-10T11:17:31.218 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.217 7f466d569700 1 --2- 172.21.2.112:0/1495644565 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f46500186c0 0x7f465001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:31.218 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.218 7f466d569700 1 --2- 172.21.2.112:0/1495644565 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f46500186c0 0x7f465001abc0 crc :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:17:31.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.247 7f466ff4e700 1 -- 172.21.2.112:0/1495644565 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4668105ae0 con 0x7f46680a0420 2021-02-10T11:17:31.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.250 7f465a7fc700 1 -- 172.21.2.112:0/1495644565 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f465c006b00 con 0x7f46680a0420 2021-02-10T11:17:31.374 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.377 7f466ff4e700 1 -- 172.21.2.112:0/1495644565 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348579", "target": ["mgr", ""]}) v1 -- 0x7f4668105ae0 con 0x7f46500186c0 2021-02-10T11:17:31.398 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.402 7f465a7fc700 1 -- 172.21.2.112:0/1495644565 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f4664009100 con 0x7f46500186c0 2021-02-10T11:17:31.407 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.410 7f466ff4e700 1 -- 172.21.2.112:0/1495644565 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f46500186c0 msgr2=0x7f465001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:31.407 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.410 7f466ff4e700 1 --2- 172.21.2.112:0/1495644565 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f46500186c0 0x7f465001abc0 crc :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:31.408 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.410 7f466ff4e700 1 -- 172.21.2.112:0/1495644565 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f46680a0420 msgr2=0x7f46680a0820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:31.408 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.410 7f466ff4e700 1 --2- 172.21.2.112:0/1495644565 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f46680a0420 0x7f46680a0820 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7f465c006f40 tx=0x7f465c00b990).stop 2021-02-10T11:17:31.408 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.411 7f466ff4e700 1 -- 172.21.2.112:0/1495644565 shutdown_connections 2021-02-10T11:17:31.408 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.411 7f466ff4e700 1 --2- 172.21.2.112:0/1495644565 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f46500186c0 0x7f465001abc0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:31.408 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.411 7f466ff4e700 1 --2- 172.21.2.112:0/1495644565 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f466818a720 0x7f46680b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:17:31.408 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.411 7f466ff4e700 1 --2- 172.21.2.112:0/1495644565 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4668189d00 0x7f466818a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:17:31.408 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.411 7f466ff4e700 1 --2- 172.21.2.112:0/1495644565 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f46680a0420 0x7f46680a0820 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:31.408 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.411 7f466ff4e700 1 -- 172.21.2.112:0/1495644565 >> 172.21.2.112:0/1495644565 conn(0x7f4668094e10 msgr2=0x7f46680a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:17:31.408 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.411 7f466ff4e700 1 -- 172.21.2.112:0/1495644565 shutdown_connections 2021-02-10T11:17:31.408 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.411 7f466ff4e700 1 -- 172.21.2.112:0/1495644565 wait complete. 2021-02-10T11:17:31.416 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000348579 2021-02-10T11:17:31.845 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.831 7f95a9d41700 1 Processor -- start 2021-02-10T11:17:31.845 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.831 7f95a9d41700 1 -- start start 2021-02-10T11:17:31.845 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.831 7f95a9d41700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f95a41b0870 0x7f95a40f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:31.846 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.831 7f95a9d41700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95a41b0c70 con 0x7f95a41b0470 2021-02-10T11:17:31.846 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.831 7f95a9d41700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95a40f0b50 con 0x7f95a40efe00 2021-02-10T11:17:31.846 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.831 7f95a9d41700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95a40e5ea0 con 0x7f95a41b0870 2021-02-10T11:17:31.846 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.832 7f95a37fe700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f95a41b0470 0x7f95a40ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:43580/0 (socket says 172.21.2.112:43580) 2021-02-10T11:17:31.846 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.832 7f95a37fe700 1 -- 172.21.2.112:0/2995619835 learned_addr learned my addr 172.21.2.112:0/2995619835 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:17:31.846 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.832 7f95a2ffd700 1 --2- 172.21.2.112:0/2995619835 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f95a41b0870 0x7f95a40f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:31.846 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.832 7f95a27fc700 1 -- 172.21.2.112:0/2995619835 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4101098560 0 0) 0x7f9598000ce0 con 0x7f95a41b0470 2021-02-10T11:17:31.846 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.832 7f95a27fc700 1 -- 172.21.2.112:0/2995619835 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f958c002fe0 con 0x7f95a41b0470 2021-02-10T11:17:31.846 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.832 7f95a27fc700 1 -- 172.21.2.112:0/2995619835 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 804963540 0 0) 0x7f959c001080 con 0x7f95a40efe00 2021-02-10T11:17:31.846 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.832 7f95a27fc700 1 -- 172.21.2.112:0/2995619835 --> v1:172.21.2.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f958c003210 con 0x7f95a40efe00 2021-02-10T11:17:31.846 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.832 7f95a27fc700 1 -- 172.21.2.112:0/2995619835 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 450583856 0 0) 0x7f9598001180 con 0x7f95a41b0470 2021-02-10T11:17:31.846 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.832 7f95a27fc700 1 -- 172.21.2.112:0/2995619835 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f95a41b0870 msgr2=0x7f95a40f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:17:31.847 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.832 7f95a27fc700 1 --2- 172.21.2.112:0/2995619835 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f95a41b0870 0x7f95a40f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:31.847 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.832 7f95a27fc700 1 -- 172.21.2.112:0/2995619835 >> v1:172.21.2.120:6789/0 conn(0x7f95a40efe00 legacy=0x7f95a40f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:31.847 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.832 7f95a27fc700 1 -- 172.21.2.112:0/2995619835 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95a40e52e0 con 0x7f95a41b0470 2021-02-10T11:17:31.847 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.832 7f95a27fc700 1 -- 172.21.2.112:0/2995619835 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f9598001680 con 0x7f95a41b0470 2021-02-10T11:17:31.847 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.832 7f95a27fc700 1 -- 172.21.2.112:0/2995619835 >> v1:172.21.2.112:6789/0 conn(0x7f95a41b0470 legacy=0x7f95a40ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:31.847 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.832 7f95a27fc700 1 --2- 172.21.2.112:0/2995619835 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f958c0037e0 0x7f958c003c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:31.847 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.832 7f95a27fc700 1 --2- 172.21.2.112:0/2995619835 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f958c0067e0 0x7f958c008c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:31.847 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.832 7f95a27fc700 1 --2- 172.21.2.112:0/2995619835 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f958c009510 0x7f958c00ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:31.847 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.832 7f95a27fc700 1 -- 172.21.2.112:0/2995619835 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f958c0502f0 con 0x7f958c0067e0 2021-02-10T11:17:31.847 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.832 7f95a27fc700 1 -- 172.21.2.112:0/2995619835 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f958c0504d0 con 0x7f958c009510 2021-02-10T11:17:31.847 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.832 7f95a27fc700 1 -- 172.21.2.112:0/2995619835 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f958c0506f0 con 0x7f958c0037e0 2021-02-10T11:17:31.848 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.832 7f95a2ffd700 1 --2- 172.21.2.112:0/2995619835 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f958c0067e0 0x7f958c008c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:31.848 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.833 7f95a2ffd700 1 -- 172.21.2.112:0/2995619835 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f958c0037e0 msgr2=0x7f958c003c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:17:31.848 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.833 7f95a2ffd700 1 --2- 172.21.2.112:0/2995619835 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f958c0037e0 0x7f958c003c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:17:31.848 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.833 7f95a2ffd700 1 -- 172.21.2.112:0/2995619835 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f958c009510 msgr2=0x7f958c00ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:17:31.848 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.833 7f95a2ffd700 1 --2- 172.21.2.112:0/2995619835 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f958c009510 0x7f958c00ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:17:31.848 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.833 7f95a2ffd700 1 -- 172.21.2.112:0/2995619835 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f958c050990 con 0x7f958c0067e0 2021-02-10T11:17:31.848 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.833 7f95a2ffd700 1 --2- 172.21.2.112:0/2995619835 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f958c0067e0 0x7f958c008c70 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7f9594009680 tx=0x7f9594009e30).ready entity=mon.0 client_cookie=10af3377f9fb9b74 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:17:31.848 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.833 7f95a27fc700 1 -- 172.21.2.112:0/2995619835 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f959400b510 con 0x7f958c0067e0 2021-02-10T11:17:31.848 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.833 7f95a27fc700 1 -- 172.21.2.112:0/2995619835 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9594006780 con 0x7f958c0067e0 2021-02-10T11:17:31.848 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.833 7f95a27fc700 1 -- 172.21.2.112:0/2995619835 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f959400b790 con 0x7f958c0067e0 2021-02-10T11:17:31.849 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.833 7f95a9d41700 1 -- 172.21.2.112:0/2995619835 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f958c0067e0 msgr2=0x7f958c008c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:31.849 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.833 7f95a9d41700 1 --2- 172.21.2.112:0/2995619835 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f958c0067e0 0x7f958c008c70 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7f9594009680 tx=0x7f9594009e30).stop 2021-02-10T11:17:31.849 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.833 7f95a3fff700 1 -- 172.21.2.112:0/2995619835 reap_dead start 2021-02-10T11:17:31.849 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.834 7f95a9d41700 1 -- 172.21.2.112:0/2995619835 shutdown_connections 2021-02-10T11:17:31.849 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.834 7f95a9d41700 1 -- 172.21.2.112:0/2995619835 >> 172.21.2.112:0/2995619835 conn(0x7f95a40ce3e0 msgr2=0x7f95a40e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:17:31.849 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.834 7f95a9d41700 1 -- 172.21.2.112:0/2995619835 shutdown_connections 2021-02-10T11:17:31.850 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.834 7f95a9d41700 1 -- 172.21.2.112:0/2995619835 wait complete. 2021-02-10T11:17:31.850 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.848 7f95a9d41700 1 Processor -- start 2021-02-10T11:17:31.850 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.848 7f95a9d41700 1 -- start start 2021-02-10T11:17:31.850 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.848 7f95a9d41700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f95a40bf920 0x7f95a40bfd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:31.850 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.849 7f95a9d41700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f95a40b6570 0x7f95a40b6970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:31.850 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.849 7f95a9d41700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f95a41a7430 0x7f95a41a7830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:31.851 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.849 7f95a9d41700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95a40c0440 con 0x7f95a40b6570 2021-02-10T11:17:31.851 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.849 7f95a9d41700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95a40924b0 con 0x7f95a41a7430 2021-02-10T11:17:31.851 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.849 7f95a9d41700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95a4092970 con 0x7f95a40bf920 2021-02-10T11:17:31.854 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.849 7f95a2ffd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f95a40b6570 0x7f95a40b6970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:31.854 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.849 7f95a2ffd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f95a40b6570 0x7f95a40b6970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:44134/0 (socket says 172.21.2.112:44134) 2021-02-10T11:17:31.854 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.849 7f95a2ffd700 1 -- 172.21.2.112:0/2287876128 learned_addr learned my addr 172.21.2.112:0/2287876128 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:17:31.854 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.849 7f95a2ffd700 1 -- 172.21.2.112:0/2287876128 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f95a40bf920 msgr2=0x7f95a40bfd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:31.854 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.849 7f95a2ffd700 1 --2- 172.21.2.112:0/2287876128 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f95a40bf920 0x7f95a40bfd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:17:31.854 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.849 7f95a3fff700 1 --2- 172.21.2.112:0/2287876128 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f95a41a7430 0x7f95a41a7830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:31.854 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.849 7f95a2ffd700 1 -- 172.21.2.112:0/2287876128 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f95a41a7430 msgr2=0x7f95a41a7830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:31.855 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.849 7f95a2ffd700 1 --2- 172.21.2.112:0/2287876128 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f95a41a7430 0x7f95a41a7830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:31.855 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.849 7f95a2ffd700 1 -- 172.21.2.112:0/2287876128 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95a41a7e50 con 0x7f95a40b6570 2021-02-10T11:17:31.855 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.849 7f95a3fff700 1 --2- 172.21.2.112:0/2287876128 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f95a41a7430 0x7f95a41a7830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:17:31.855 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.849 7f95a2ffd700 1 --2- 172.21.2.112:0/2287876128 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f95a40b6570 0x7f95a40b6970 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 rx=0x7f959400b8f0 tx=0x7f959400bc90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:17:31.855 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.850 7f958bfff700 1 -- 172.21.2.112:0/2287876128 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f959400c2f0 con 0x7f95a40b6570 2021-02-10T11:17:31.855 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.850 7f958bfff700 1 -- 172.21.2.112:0/2287876128 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95940093d0 con 0x7f95a40b6570 2021-02-10T11:17:31.855 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.850 7f958bfff700 1 -- 172.21.2.112:0/2287876128 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f959400c8f0 con 0x7f95a40b6570 2021-02-10T11:17:31.855 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.850 7f95a9d41700 1 -- 172.21.2.112:0/2287876128 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95a41a7e50 con 0x7f95a40b6570 2021-02-10T11:17:31.855 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.850 7f95a9d41700 1 -- 172.21.2.112:0/2287876128 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95a40c09c0 con 0x7f95a40b6570 2021-02-10T11:17:31.855 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.851 7f958bfff700 1 -- 172.21.2.112:0/2287876128 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9594001010 con 0x7f95a40b6570 2021-02-10T11:17:31.855 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.851 7f958bfff700 1 --2- 172.21.2.112:0/2287876128 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f95840186c0 0x7f958401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:31.855 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.851 7f958bfff700 1 -- 172.21.2.112:0/2287876128 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f95940328e0 con 0x7f95a40b6570 2021-02-10T11:17:31.855 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.851 7f95a37fe700 1 --2- 172.21.2.112:0/2287876128 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f95840186c0 0x7f958401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:31.856 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.851 7f95a37fe700 1 --2- 172.21.2.112:0/2287876128 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f95840186c0 0x7f958401abc0 crc :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:17:31.878 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.881 7f95a9d41700 1 -- 172.21.2.112:0/2287876128 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95a4105b40 con 0x7f95a40b6570 2021-02-10T11:17:31.880 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:31.884 7f958bfff700 1 -- 172.21.2.112:0/2287876128 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9594000d30 con 0x7f95a40b6570 2021-02-10T11:17:32.008 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:32.011 7f95a9d41700 1 -- 172.21.2.112:0/2287876128 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348579", "target": ["mgr", ""]}) v1 -- 0x7f95a40b6f90 con 0x7f95840186c0 2021-02-10T11:17:32.011 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:32.015 7f958bfff700 1 -- 172.21.2.112:0/2287876128 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f9598001530 con 0x7f95840186c0 2021-02-10T11:17:32.015 INFO:teuthology.orchestra.run.gibba012.stdout:/volumes/_nogroup/subvolume_0000000000348579/2a3feb63-21d9-40ab-b250-d5f6646f6c1e 2021-02-10T11:17:32.018 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:32.020 7f95a9d41700 1 -- 172.21.2.112:0/2287876128 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f95840186c0 msgr2=0x7f958401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:32.018 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:32.020 7f95a9d41700 1 --2- 172.21.2.112:0/2287876128 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f95840186c0 0x7f958401abc0 crc :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:32.018 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:32.020 7f95a9d41700 1 -- 172.21.2.112:0/2287876128 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f95a40b6570 msgr2=0x7f95a40b6970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:32.018 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:32.020 7f95a9d41700 1 --2- 172.21.2.112:0/2287876128 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f95a40b6570 0x7f95a40b6970 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 rx=0x7f959400b8f0 tx=0x7f959400bc90).stop 2021-02-10T11:17:32.018 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:32.021 7f95a9d41700 1 -- 172.21.2.112:0/2287876128 shutdown_connections 2021-02-10T11:17:32.018 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:32.021 7f95a9d41700 1 --2- 172.21.2.112:0/2287876128 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f95840186c0 0x7f958401abc0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:32.018 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:32.021 7f95a9d41700 1 --2- 172.21.2.112:0/2287876128 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f95a41a7430 0x7f95a41a7830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:32.018 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:32.021 7f95a9d41700 1 --2- 172.21.2.112:0/2287876128 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f95a40b6570 0x7f95a40b6970 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:32.018 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:32.021 7f95a9d41700 1 --2- 172.21.2.112:0/2287876128 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f95a40bf920 0x7f95a40bfd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:17:32.018 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:32.021 7f95a9d41700 1 -- 172.21.2.112:0/2287876128 >> 172.21.2.112:0/2287876128 conn(0x7f95a40b50c0 msgr2=0x7f95a4092d90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:17:32.018 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:32.021 7f95a9d41700 1 -- 172.21.2.112:0/2287876128 shutdown_connections 2021-02-10T11:17:32.019 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:32.021 7f95a9d41700 1 -- 172.21.2.112:0/2287876128 wait complete. 2021-02-10T11:17:32.025 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000348579 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000348579/2a3feb63-21d9-40ab-b250-d5f6646f6c1e 2021-02-10T11:17:32.026 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:32.026 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:32.026 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:32.360 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:17:32.361 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:32.361 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348579/2a3feb63-21d9-40ab-b250-d5f6646f6c1e/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:32.739 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:17:32.740 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:17:32.740 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0225022 s, 46.6 MB/s 2021-02-10T11:17:32.741 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:32.741 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:32.741 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:33.075 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:17:33.076 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:33.076 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348579/2a3feb63-21d9-40ab-b250-d5f6646f6c1e/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:33.449 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:17:33.449 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:17:33.449 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0239102 s, 43.9 MB/s 2021-02-10T11:17:33.450 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:33.451 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:33.451 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:33.724 DEBUG:teuthology.orchestra.run.gibba012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:17:33.727 DEBUG:teuthology.orchestra.run.gibba020:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:17:33.786 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:17:33.786 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:33.786 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348579/2a3feb63-21d9-40ab-b250-d5f6646f6c1e/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:34.102 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:17:34.102 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:17:34.102 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0259558 s, 40.4 MB/s 2021-02-10T11:17:34.103 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:34.103 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:34.103 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:34.476 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:17:34.477 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:34.477 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348579/2a3feb63-21d9-40ab-b250-d5f6646f6c1e/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:34.851 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:17:34.851 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:17:34.851 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0256088 s, 40.9 MB/s 2021-02-10T11:17:34.852 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:34.852 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:34.852 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:35.185 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:17:35.186 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:35.186 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348579/2a3feb63-21d9-40ab-b250-d5f6646f6c1e/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:35.561 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:17:35.561 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:17:35.561 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0251194 s, 41.7 MB/s 2021-02-10T11:17:35.562 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:35.563 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:35.563 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:35.897 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:17:35.898 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:35.898 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348579/2a3feb63-21d9-40ab-b250-d5f6646f6c1e/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:36.272 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:17:36.273 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:17:36.273 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0248407 s, 42.2 MB/s 2021-02-10T11:17:36.274 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:36.274 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:36.274 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:36.609 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:17:36.609 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:36.610 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348579/2a3feb63-21d9-40ab-b250-d5f6646f6c1e/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:36.987 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:17:36.987 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:17:36.987 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0262393 s, 40.0 MB/s 2021-02-10T11:17:36.988 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:36.988 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:36.988 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:37.322 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:17:37.322 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:37.322 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348579/2a3feb63-21d9-40ab-b250-d5f6646f6c1e/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:37.697 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:17:37.698 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:17:37.698 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0237277 s, 44.2 MB/s 2021-02-10T11:17:37.699 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:37.699 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:37.699 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:38.033 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:17:38.034 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:38.034 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348579/2a3feb63-21d9-40ab-b250-d5f6646f6c1e/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:38.409 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:17:38.410 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:17:38.410 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0256526 s, 40.9 MB/s 2021-02-10T11:17:38.411 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:38.411 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:38.412 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:38.745 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:17:38.746 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:38.746 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348579/2a3feb63-21d9-40ab-b250-d5f6646f6c1e/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:39.123 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:17:39.123 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:17:39.123 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0259042 s, 40.5 MB/s 2021-02-10T11:17:39.124 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000348580 2021-02-10T11:17:39.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:39.538 7f96e58c8700 1 Processor -- start 2021-02-10T11:17:39.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:39.538 7f96e58c8700 1 -- start start 2021-02-10T11:17:39.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:39.539 7f96e58c8700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f96e01b0870 0x7f96e00f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:39.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:39.539 7f96e58c8700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96e01b0c70 con 0x7f96e00efe00 2021-02-10T11:17:39.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:39.539 7f96e58c8700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96e00f0b50 con 0x7f96e01b0470 2021-02-10T11:17:39.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:39.539 7f96e58c8700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96e00e5ea0 con 0x7f96e01b0870 2021-02-10T11:17:39.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:39.539 7f96dfd75700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f96e00efe00 0x7f96e00f0200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:43600/0 (socket says 172.21.2.112:43600) 2021-02-10T11:17:39.553 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:39.539 7f96dfd75700 1 -- 172.21.2.112:0/1112496523 learned_addr learned my addr 172.21.2.112:0/1112496523 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:17:39.553 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:39.539 7f96de572700 1 -- 172.21.2.112:0/1112496523 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1609599192 0 0) 0x7f96d8001180 con 0x7f96e00efe00 2021-02-10T11:17:39.553 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:39.539 7f96de572700 1 -- 172.21.2.112:0/1112496523 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96cc002fe0 con 0x7f96e00efe00 2021-02-10T11:17:39.553 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:39.539 7f96ded73700 1 --2- 172.21.2.112:0/1112496523 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f96e01b0870 0x7f96e00f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:39.553 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:39.540 7f96de572700 1 -- 172.21.2.112:0/1112496523 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2235274337 0 0) 0x7f96d80010e0 con 0x7f96e00efe00 2021-02-10T11:17:39.553 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:39.540 7f96de572700 1 -- 172.21.2.112:0/1112496523 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f96e01b0870 msgr2=0x7f96e00f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:17:39.553 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:39.540 7f96de572700 1 --2- 172.21.2.112:0/1112496523 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f96e01b0870 0x7f96e00f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:39.553 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:39.540 7f96de572700 1 -- 172.21.2.112:0/1112496523 >> v1:172.21.2.120:6789/0 conn(0x7f96e01b0470 legacy=0x7f96e00ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:39.553 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:39.540 7f96de572700 1 -- 172.21.2.112:0/1112496523 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96e00e52e0 con 0x7f96e00efe00 2021-02-10T11:17:39.553 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:39.540 7f96de572700 1 -- 172.21.2.112:0/1112496523 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f96d8001600 con 0x7f96e00efe00 2021-02-10T11:17:39.553 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:39.540 7f96de572700 1 -- 172.21.2.112:0/1112496523 >> v1:172.21.2.112:6789/0 conn(0x7f96e00efe00 legacy=0x7f96e00f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:39.553 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:39.540 7f96de572700 1 --2- 172.21.2.112:0/1112496523 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f96cc003aa0 0x7f96cc005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:39.553 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:39.540 7f96de572700 1 --2- 172.21.2.112:0/1112496523 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f96cc006790 0x7f96cc008c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:39.554 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:39.540 7f96de572700 1 --2- 172.21.2.112:0/1112496523 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f96cc009500 0x7f96cc00ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:39.554 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:39.540 7f96de572700 1 -- 172.21.2.112:0/1112496523 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96cc00c2d0 con 0x7f96cc003aa0 2021-02-10T11:17:39.554 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:39.540 7f96de572700 1 -- 172.21.2.112:0/1112496523 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96cc00c4f0 con 0x7f96cc009500 2021-02-10T11:17:39.554 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:39.540 7f96de572700 1 -- 172.21.2.112:0/1112496523 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96cc00c710 con 0x7f96cc006790 2021-02-10T11:17:39.554 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:39.540 7f96df574700 1 --2- 172.21.2.112:0/1112496523 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f96cc003aa0 0x7f96cc005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:39.554 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:39.540 7f96ded73700 1 --2- 172.21.2.112:0/1112496523 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f96cc006790 0x7f96cc008c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:39.554 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:39.540 7f96dfd75700 1 --2- 172.21.2.112:0/1112496523 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f96cc009500 0x7f96cc00ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:39.554 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:39.540 7f96df574700 1 -- 172.21.2.112:0/1112496523 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f96cc006790 msgr2=0x7f96cc008c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:17:39.554 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:39.540 7f96df574700 1 --2- 172.21.2.112:0/1112496523 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f96cc006790 0x7f96cc008c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:39.554 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:39.540 7f96df574700 1 -- 172.21.2.112:0/1112496523 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f96cc009500 msgr2=0x7f96cc00ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:17:39.554 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:39.540 7f96df574700 1 --2- 172.21.2.112:0/1112496523 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f96cc009500 0x7f96cc00ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:39.554 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:39.540 7f96df574700 1 -- 172.21.2.112:0/1112496523 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96cc00c9b0 con 0x7f96cc003aa0 2021-02-10T11:17:39.555 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:39.540 7f96df574700 1 --2- 172.21.2.112:0/1112496523 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f96cc003aa0 0x7f96cc005ef0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 rx=0x7f96c8006f90 tx=0x7f96c804de70).ready entity=mon.0 client_cookie=661aac8329fe97d8 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:17:39.555 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:39.540 7f96de572700 1 -- 172.21.2.112:0/1112496523 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f96c8003030 con 0x7f96cc003aa0 2021-02-10T11:17:39.555 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:39.540 7f96de572700 1 -- 172.21.2.112:0/1112496523 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96c8006a50 con 0x7f96cc003aa0 2021-02-10T11:17:39.555 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:39.540 7f96de572700 1 -- 172.21.2.112:0/1112496523 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f96c804f700 con 0x7f96cc003aa0 2021-02-10T11:17:39.555 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:39.541 7f96e58c8700 1 -- 172.21.2.112:0/1112496523 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f96cc003aa0 msgr2=0x7f96cc005ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:39.555 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:39.541 7f96e58c8700 1 --2- 172.21.2.112:0/1112496523 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f96cc003aa0 0x7f96cc005ef0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 rx=0x7f96c8006f90 tx=0x7f96c804de70).stop 2021-02-10T11:17:39.555 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:39.541 7f96dfd75700 1 -- 172.21.2.112:0/1112496523 reap_dead start 2021-02-10T11:17:39.555 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:39.541 7f96e58c8700 1 -- 172.21.2.112:0/1112496523 shutdown_connections 2021-02-10T11:17:39.555 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:39.541 7f96e58c8700 1 -- 172.21.2.112:0/1112496523 >> 172.21.2.112:0/1112496523 conn(0x7f96e00ce3e0 msgr2=0x7f96e00e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:17:39.555 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:39.541 7f96e58c8700 1 -- 172.21.2.112:0/1112496523 shutdown_connections 2021-02-10T11:17:39.555 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:39.541 7f96e58c8700 1 -- 172.21.2.112:0/1112496523 wait complete. 2021-02-10T11:17:39.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:39.556 7f96e58c8700 1 Processor -- start 2021-02-10T11:17:39.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:39.556 7f96e58c8700 1 -- start start 2021-02-10T11:17:39.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:39.556 7f96e58c8700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f96e00b7db0 0x7f96e00b8260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:39.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:39.556 7f96e58c8700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f96e00b8880 0x7f96e01a7450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:39.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:39.556 7f96e58c8700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f96e00b8c80 0x7f96e01a7a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:39.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:39.556 7f96e58c8700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96e00b79b0 con 0x7f96e00b7db0 2021-02-10T11:17:39.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:39.556 7f96e58c8700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96e01a8090 con 0x7f96e00b8c80 2021-02-10T11:17:39.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:39.556 7f96e58c8700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96e01a82b0 con 0x7f96e00b8880 2021-02-10T11:17:39.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:39.556 7f96df574700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f96e00b7db0 0x7f96e00b8260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:39.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:39.556 7f96df574700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f96e00b7db0 0x7f96e00b8260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:44148/0 (socket says 172.21.2.112:44148) 2021-02-10T11:17:39.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:39.556 7f96df574700 1 -- 172.21.2.112:0/790680633 learned_addr learned my addr 172.21.2.112:0/790680633 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:17:39.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:39.557 7f96df574700 1 -- 172.21.2.112:0/790680633 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f96e00b8880 msgr2=0x7f96e01a7450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:39.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:39.557 7f96df574700 1 --2- 172.21.2.112:0/790680633 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f96e00b8880 0x7f96e01a7450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:17:39.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:39.557 7f96df574700 1 -- 172.21.2.112:0/790680633 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f96e00b8c80 msgr2=0x7f96e01a7a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:39.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:39.557 7f96df574700 1 --2- 172.21.2.112:0/790680633 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f96e00b8c80 0x7f96e01a7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:17:39.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:39.557 7f96df574700 1 -- 172.21.2.112:0/790680633 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96e0190c10 con 0x7f96e00b7db0 2021-02-10T11:17:39.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:39.557 7f96df574700 1 --2- 172.21.2.112:0/790680633 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f96e00b7db0 0x7f96e00b8260 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7f96c8006e60 tx=0x7f96c8051d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:17:39.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:39.557 7f96d7fff700 1 -- 172.21.2.112:0/790680633 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f96c804d030 con 0x7f96e00b7db0 2021-02-10T11:17:39.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:39.557 7f96d7fff700 1 -- 172.21.2.112:0/790680633 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96c804b300 con 0x7f96e00b7db0 2021-02-10T11:17:39.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:39.557 7f96d7fff700 1 -- 172.21.2.112:0/790680633 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f96c804b760 con 0x7f96e00b7db0 2021-02-10T11:17:39.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:39.557 7f96e58c8700 1 -- 172.21.2.112:0/790680633 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96e0190c10 con 0x7f96e00b7db0 2021-02-10T11:17:39.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:39.557 7f96e58c8700 1 -- 172.21.2.112:0/790680633 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96e00c1eb0 con 0x7f96e00b7db0 2021-02-10T11:17:39.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:39.558 7f96d7fff700 1 -- 172.21.2.112:0/790680633 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f96c804b4e0 con 0x7f96e00b7db0 2021-02-10T11:17:39.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:39.558 7f96d7fff700 1 --2- 172.21.2.112:0/790680633 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f96c40186c0 0x7f96c401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:39.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:39.558 7f96d7fff700 1 -- 172.21.2.112:0/790680633 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f96c8076930 con 0x7f96e00b7db0 2021-02-10T11:17:39.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:39.558 7f96ded73700 1 --2- 172.21.2.112:0/790680633 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f96c40186c0 0x7f96c401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:39.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:39.559 7f96ded73700 1 --2- 172.21.2.112:0/790680633 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f96c40186c0 0x7f96c401abc0 crc :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:17:39.586 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:39.589 7f96e58c8700 1 -- 172.21.2.112:0/790680633 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96e0105b30 con 0x7f96e00b7db0 2021-02-10T11:17:39.588 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:39.592 7f96d7fff700 1 -- 172.21.2.112:0/790680633 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f96c8063050 con 0x7f96e00b7db0 2021-02-10T11:17:39.718 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:39.721 7f96e58c8700 1 -- 172.21.2.112:0/790680633 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348580", "target": ["mgr", ""]}) v1 -- 0x7f96e00c1eb0 con 0x7f96c40186c0 2021-02-10T11:17:39.743 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:39.747 7f96d7fff700 1 -- 172.21.2.112:0/790680633 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f96d00069f0 con 0x7f96c40186c0 2021-02-10T11:17:39.751 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:39.754 7f96e58c8700 1 -- 172.21.2.112:0/790680633 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f96c40186c0 msgr2=0x7f96c401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:39.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:39.754 7f96e58c8700 1 --2- 172.21.2.112:0/790680633 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f96c40186c0 0x7f96c401abc0 crc :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:39.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:39.754 7f96e58c8700 1 -- 172.21.2.112:0/790680633 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f96e00b7db0 msgr2=0x7f96e00b8260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:39.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:39.754 7f96e58c8700 1 --2- 172.21.2.112:0/790680633 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f96e00b7db0 0x7f96e00b8260 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7f96c8006e60 tx=0x7f96c8051d10).stop 2021-02-10T11:17:39.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:39.754 7f96e58c8700 1 -- 172.21.2.112:0/790680633 shutdown_connections 2021-02-10T11:17:39.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:39.754 7f96e58c8700 1 --2- 172.21.2.112:0/790680633 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f96c40186c0 0x7f96c401abc0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:39.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:39.754 7f96e58c8700 1 --2- 172.21.2.112:0/790680633 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f96e00b8c80 0x7f96e01a7a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:17:39.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:39.754 7f96e58c8700 1 --2- 172.21.2.112:0/790680633 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f96e00b8880 0x7f96e01a7450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:17:39.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:39.754 7f96e58c8700 1 --2- 172.21.2.112:0/790680633 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f96e00b7db0 0x7f96e00b8260 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:39.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:39.754 7f96e58c8700 1 -- 172.21.2.112:0/790680633 >> 172.21.2.112:0/790680633 conn(0x7f96e0094e10 msgr2=0x7f96e00a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:17:39.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:39.755 7f96e58c8700 1 -- 172.21.2.112:0/790680633 shutdown_connections 2021-02-10T11:17:39.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:39.755 7f96e58c8700 1 -- 172.21.2.112:0/790680633 wait complete. 2021-02-10T11:17:39.760 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000348580 2021-02-10T11:17:40.187 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:40.175 7f853a5c3700 1 Processor -- start 2021-02-10T11:17:40.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:40.175 7f853a5c3700 1 -- start start 2021-02-10T11:17:40.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:40.175 7f853a5c3700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f85341b0870 0x7f85340f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:40.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:40.175 7f853a5c3700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f85340efe00 0x7f85340f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:40.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:40.175 7f853a5c3700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85341b0c70 con 0x7f85341b0470 2021-02-10T11:17:40.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:40.175 7f853a5c3700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85340e5ea0 con 0x7f85340efe00 2021-02-10T11:17:40.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:40.175 7f853a5c3700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85340f0b50 con 0x7f85341b0870 2021-02-10T11:17:40.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:40.175 7f8538be0700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f85340efe00 0x7f85340f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:40.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:40.175 7f8538be0700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f85340efe00 0x7f85340f0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:44158/0 (socket says 172.21.2.112:44158) 2021-02-10T11:17:40.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:40.175 7f8538be0700 1 -- 172.21.2.112:0/3623528858 learned_addr learned my addr 172.21.2.112:0/3623528858 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:17:40.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:40.176 7f8538be0700 1 -- 172.21.2.112:0/3623528858 >> v2:172.21.2.120:3300/0 conn(0x7f85341b0870 msgr2=0x7f85340f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:17:40.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:40.176 7f8538be0700 1 --2- 172.21.2.112:0/3623528858 >> v2:172.21.2.120:3300/0 conn(0x7f85341b0870 0x7f85340f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:17:40.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:40.176 7f8538be0700 1 -- 172.21.2.112:0/3623528858 >> v1:172.21.2.120:6789/0 conn(0x7f85341b0470 legacy=0x7f85340ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:17:40.189 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:40.176 7f8538be0700 1 -- 172.21.2.112:0/3623528858 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85340f0820 con 0x7f85340efe00 2021-02-10T11:17:40.189 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:40.176 7f8538be0700 1 --2- 172.21.2.112:0/3623528858 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f85340efe00 0x7f85340f0200 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 rx=0x7f852c006f00 tx=0x7f852c009f10).ready entity=mon.0 client_cookie=862f8b429e039de8 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:17:40.189 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:40.176 7f8532ffd700 1 -- 172.21.2.112:0/3623528858 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f852c00b3b0 con 0x7f85340efe00 2021-02-10T11:17:40.189 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:40.176 7f8532ffd700 1 -- 172.21.2.112:0/3623528858 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f852c00b630 con 0x7f85340efe00 2021-02-10T11:17:40.189 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:40.176 7f8532ffd700 1 -- 172.21.2.112:0/3623528858 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f852c00bb50 con 0x7f85340efe00 2021-02-10T11:17:40.189 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:40.176 7f853a5c3700 1 -- 172.21.2.112:0/3623528858 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f85340efe00 msgr2=0x7f85340f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:40.189 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:40.176 7f853a5c3700 1 --2- 172.21.2.112:0/3623528858 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f85340efe00 0x7f85340f0200 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 rx=0x7f852c006f00 tx=0x7f852c009f10).stop 2021-02-10T11:17:40.189 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:40.176 7f853a5c3700 1 -- 172.21.2.112:0/3623528858 shutdown_connections 2021-02-10T11:17:40.189 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:40.176 7f853a5c3700 1 --2- 172.21.2.112:0/3623528858 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f85340efe00 0x7f85340f0200 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:40.189 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:40.176 7f853a5c3700 1 --2- 172.21.2.112:0/3623528858 >> v2:172.21.2.120:3300/0 conn(0x7f85341b0870 0x7f85340f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:17:40.189 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:40.176 7f853a5c3700 1 -- 172.21.2.112:0/3623528858 >> 172.21.2.112:0/3623528858 conn(0x7f85340ce3e0 msgr2=0x7f85340e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:17:40.189 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:40.177 7f853a5c3700 1 -- 172.21.2.112:0/3623528858 shutdown_connections 2021-02-10T11:17:40.189 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:40.177 7f853a5c3700 1 -- 172.21.2.112:0/3623528858 wait complete. 2021-02-10T11:17:40.190 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:40.191 7f853a5c3700 1 Processor -- start 2021-02-10T11:17:40.190 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:40.191 7f853a5c3700 1 -- start start 2021-02-10T11:17:40.191 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:40.191 7f853a5c3700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f85340c36a0 0x7f85340c2b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:40.191 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:40.191 7f853a5c3700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f85340c3190 0x7f85340b1d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:40.191 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:40.191 7f853a5c3700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f85340b2370 0x7f85340b2d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:40.191 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:40.191 7f853a5c3700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85340b27c0 con 0x7f85340b2370 2021-02-10T11:17:40.191 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:40.191 7f853a5c3700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85340b3360 con 0x7f85340c36a0 2021-02-10T11:17:40.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:40.191 7f853a5c3700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85340b3540 con 0x7f85340c3190 2021-02-10T11:17:40.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:40.192 7f8538be0700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f85340b2370 0x7f85340b2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:40.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:40.192 7f8538be0700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f85340b2370 0x7f85340b2d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:44162/0 (socket says 172.21.2.112:44162) 2021-02-10T11:17:40.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:40.192 7f8538be0700 1 -- 172.21.2.112:0/4082776781 learned_addr learned my addr 172.21.2.112:0/4082776781 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:17:40.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:40.192 7f8538be0700 1 -- 172.21.2.112:0/4082776781 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f85340c3190 msgr2=0x7f85340b1d50 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:17:40.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:40.192 7f8538be0700 1 --2- 172.21.2.112:0/4082776781 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f85340c3190 0x7f85340b1d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:17:40.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:40.192 7f8538be0700 1 -- 172.21.2.112:0/4082776781 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f85340c36a0 msgr2=0x7f85340c2b70 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:17:40.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:40.192 7f8538be0700 1 --2- 172.21.2.112:0/4082776781 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f85340c36a0 0x7f85340c2b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:17:40.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:40.192 7f8538be0700 1 -- 172.21.2.112:0/4082776781 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85340b3760 con 0x7f85340b2370 2021-02-10T11:17:40.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:40.192 7f8538be0700 1 --2- 172.21.2.112:0/4082776781 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f85340b2370 0x7f85340b2d40 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7f852c00ba30 tx=0x7f852c00bdd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:17:40.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:40.192 7f8530ff9700 1 -- 172.21.2.112:0/4082776781 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f852c005820 con 0x7f85340b2370 2021-02-10T11:17:40.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:40.192 7f8530ff9700 1 -- 172.21.2.112:0/4082776781 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f852c005aa0 con 0x7f85340b2370 2021-02-10T11:17:40.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:40.192 7f8530ff9700 1 -- 172.21.2.112:0/4082776781 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f852c006030 con 0x7f85340b2370 2021-02-10T11:17:40.193 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:40.192 7f853a5c3700 1 -- 172.21.2.112:0/4082776781 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85340b3760 con 0x7f85340b2370 2021-02-10T11:17:40.193 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:40.192 7f853a5c3700 1 -- 172.21.2.112:0/4082776781 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85340b3cf0 con 0x7f85340b2370 2021-02-10T11:17:40.193 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:40.193 7f8530ff9700 1 -- 172.21.2.112:0/4082776781 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f852c005c80 con 0x7f85340b2370 2021-02-10T11:17:40.193 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:40.193 7f8530ff9700 1 --2- 172.21.2.112:0/4082776781 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f85140186c0 0x7f851401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:40.193 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:40.193 7f8530ff9700 1 -- 172.21.2.112:0/4082776781 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f852c0328e0 con 0x7f85340b2370 2021-02-10T11:17:40.193 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:40.193 7f8533fff700 1 --2- 172.21.2.112:0/4082776781 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f85140186c0 0x7f851401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:40.193 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:40.194 7f8533fff700 1 --2- 172.21.2.112:0/4082776781 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f85140186c0 0x7f851401abc0 crc :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:17:40.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:40.225 7f853a5c3700 1 -- 172.21.2.112:0/4082776781 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85340edd10 con 0x7f85340b2370 2021-02-10T11:17:40.224 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:40.228 7f8530ff9700 1 -- 172.21.2.112:0/4082776781 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f852c024880 con 0x7f85340b2370 2021-02-10T11:17:40.352 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:40.355 7f853a5c3700 1 -- 172.21.2.112:0/4082776781 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348580", "target": ["mgr", ""]}) v1 -- 0x7f85340b3cf0 con 0x7f85140186c0 2021-02-10T11:17:40.355 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:40.359 7f8530ff9700 1 -- 172.21.2.112:0/4082776781 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f8524000c10 con 0x7f85140186c0 2021-02-10T11:17:40.359 INFO:teuthology.orchestra.run.gibba012.stdout:/volumes/_nogroup/subvolume_0000000000348580/4b8e0918-72a6-4f17-8848-e6f2b33e008a 2021-02-10T11:17:40.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:40.364 7f853a5c3700 1 -- 172.21.2.112:0/4082776781 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f85140186c0 msgr2=0x7f851401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:40.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:40.364 7f853a5c3700 1 --2- 172.21.2.112:0/4082776781 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f85140186c0 0x7f851401abc0 crc :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:40.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:40.365 7f853a5c3700 1 -- 172.21.2.112:0/4082776781 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f85340b2370 msgr2=0x7f85340b2d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:40.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:40.365 7f853a5c3700 1 --2- 172.21.2.112:0/4082776781 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f85340b2370 0x7f85340b2d40 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7f852c00ba30 tx=0x7f852c00bdd0).stop 2021-02-10T11:17:40.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:40.365 7f853a5c3700 1 -- 172.21.2.112:0/4082776781 shutdown_connections 2021-02-10T11:17:40.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:40.365 7f853a5c3700 1 --2- 172.21.2.112:0/4082776781 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f85140186c0 0x7f851401abc0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:40.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:40.365 7f853a5c3700 1 --2- 172.21.2.112:0/4082776781 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f85340b2370 0x7f85340b2d40 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:40.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:40.365 7f853a5c3700 1 --2- 172.21.2.112:0/4082776781 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f85340c3190 0x7f85340b1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:17:40.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:40.365 7f853a5c3700 1 --2- 172.21.2.112:0/4082776781 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f85340c36a0 0x7f85340c2b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:17:40.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:40.365 7f853a5c3700 1 -- 172.21.2.112:0/4082776781 >> 172.21.2.112:0/4082776781 conn(0x7f853409f630 msgr2=0x7f85340a0130 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:17:40.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:40.365 7f853a5c3700 1 -- 172.21.2.112:0/4082776781 shutdown_connections 2021-02-10T11:17:40.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:40.365 7f853a5c3700 1 -- 172.21.2.112:0/4082776781 wait complete. 2021-02-10T11:17:40.369 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000348580 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000348580/4b8e0918-72a6-4f17-8848-e6f2b33e008a 2021-02-10T11:17:40.369 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:40.369 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:40.369 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:40.705 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:17:40.706 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:40.706 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348580/4b8e0918-72a6-4f17-8848-e6f2b33e008a/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:41.085 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:17:41.086 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:17:41.086 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0246271 s, 42.6 MB/s 2021-02-10T11:17:41.087 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:41.087 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:41.087 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:41.423 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:17:41.423 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:41.424 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348580/4b8e0918-72a6-4f17-8848-e6f2b33e008a/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:41.798 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:17:41.798 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:17:41.798 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0252793 s, 41.5 MB/s 2021-02-10T11:17:41.799 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:41.799 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:41.799 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:42.134 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:17:42.135 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:42.135 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348580/4b8e0918-72a6-4f17-8848-e6f2b33e008a/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:42.511 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:17:42.511 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:17:42.511 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0265141 s, 39.5 MB/s 2021-02-10T11:17:42.512 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:42.512 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:42.512 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:42.847 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:17:42.848 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:42.848 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348580/4b8e0918-72a6-4f17-8848-e6f2b33e008a/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:43.223 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:17:43.223 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:17:43.223 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0239467 s, 43.8 MB/s 2021-02-10T11:17:43.224 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:43.224 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:43.224 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:43.559 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:17:43.559 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:43.559 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348580/4b8e0918-72a6-4f17-8848-e6f2b33e008a/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:43.933 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:17:43.934 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:17:43.934 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0230158 s, 45.6 MB/s 2021-02-10T11:17:43.936 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:43.936 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:43.936 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:44.271 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:17:44.271 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:44.271 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348580/4b8e0918-72a6-4f17-8848-e6f2b33e008a/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:44.643 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:17:44.643 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:17:44.643 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0238986 s, 43.9 MB/s 2021-02-10T11:17:44.644 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:44.645 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:44.645 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:44.979 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:17:44.980 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:44.980 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348580/4b8e0918-72a6-4f17-8848-e6f2b33e008a/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:45.355 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:17:45.355 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:17:45.355 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0253666 s, 41.3 MB/s 2021-02-10T11:17:45.356 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:45.356 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:45.356 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:45.691 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:17:45.692 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:45.692 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348580/4b8e0918-72a6-4f17-8848-e6f2b33e008a/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:46.066 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:17:46.067 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:17:46.067 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0237767 s, 44.1 MB/s 2021-02-10T11:17:46.067 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:46.068 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:46.068 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:46.403 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:17:46.403 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:46.404 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348580/4b8e0918-72a6-4f17-8848-e6f2b33e008a/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:46.778 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:17:46.778 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:17:46.779 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0243045 s, 43.1 MB/s 2021-02-10T11:17:46.779 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:46.780 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:46.780 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:47.113 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:17:47.114 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:47.114 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348580/4b8e0918-72a6-4f17-8848-e6f2b33e008a/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:47.488 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:17:47.489 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:17:47.489 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0230551 s, 45.5 MB/s 2021-02-10T11:17:47.490 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000348581 2021-02-10T11:17:47.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:47.906 7f04cd6a2700 1 Processor -- start 2021-02-10T11:17:47.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:47.906 7f04cd6a2700 1 -- start start 2021-02-10T11:17:47.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:47.907 7f04cd6a2700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f04c81b0470 0x7f04c81b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:47.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:47.907 7f04cd6a2700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f04c80ef610 0x7f04c80efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:47.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:47.907 7f04cd6a2700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04c80f0b50 con 0x7f04c80f0030 2021-02-10T11:17:47.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:47.907 7f04cd6a2700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04c80e52e0 con 0x7f04c80ef610 2021-02-10T11:17:47.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:47.907 7f04cd6a2700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04c80e5490 con 0x7f04c81b0470 2021-02-10T11:17:47.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:47.907 7f04c7ac0700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f04c80f0030 0x7f04c80f10d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:43632/0 (socket says 172.21.2.112:43632) 2021-02-10T11:17:47.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:47.907 7f04c7ac0700 1 -- 172.21.2.112:0/362631795 learned_addr learned my addr 172.21.2.112:0/362631795 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:17:47.921 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:47.907 7f04c62bd700 1 -- 172.21.2.112:0/362631795 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2587145658 0 0) 0x7f04c0001500 con 0x7f04c80f0030 2021-02-10T11:17:47.921 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:47.907 7f04c62bd700 1 -- 172.21.2.112:0/362631795 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04b4002fe0 con 0x7f04c80f0030 2021-02-10T11:17:47.921 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:47.908 7f04c62bd700 1 -- 172.21.2.112:0/362631795 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 841289058 0 0) 0x7f04c0001460 con 0x7f04c80f0030 2021-02-10T11:17:47.921 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:47.908 7f04c62bd700 1 -- 172.21.2.112:0/362631795 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f04c81b0470 msgr2=0x7f04c81b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:17:47.921 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:47.908 7f04c72bf700 1 --2- 172.21.2.112:0/362631795 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f04c81b0470 0x7f04c81b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:47.921 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:47.908 7f04c6abe700 1 --2- 172.21.2.112:0/362631795 >> v2:172.21.2.120:3300/0 conn(0x7f04c80ef610 0x7f04c80efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:47.921 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:47.908 7f04c62bd700 1 --2- 172.21.2.112:0/362631795 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f04c81b0470 0x7f04c81b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:47.921 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:47.908 7f04c62bd700 1 -- 172.21.2.112:0/362631795 >> v2:172.21.2.120:3300/0 conn(0x7f04c80ef610 msgr2=0x7f04c80efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:17:47.921 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:47.908 7f04c62bd700 1 --2- 172.21.2.112:0/362631795 >> v2:172.21.2.120:3300/0 conn(0x7f04c80ef610 0x7f04c80efa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:47.921 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:47.908 7f04c62bd700 1 -- 172.21.2.112:0/362631795 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04c80f0430 con 0x7f04c80f0030 2021-02-10T11:17:47.921 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:47.908 7f04c62bd700 1 -- 172.21.2.112:0/362631795 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f04c0001980 con 0x7f04c80f0030 2021-02-10T11:17:47.921 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:47.908 7f04c62bd700 1 -- 172.21.2.112:0/362631795 >> v1:172.21.2.112:6789/0 conn(0x7f04c80f0030 legacy=0x7f04c80f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:47.921 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:47.908 7f04c62bd700 1 --2- 172.21.2.112:0/362631795 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f04b4003b90 0x7f04b4006040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:47.921 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:47.908 7f04c62bd700 1 --2- 172.21.2.112:0/362631795 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f04b40068e0 0x7f04b4008de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:47.922 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:47.908 7f04c62bd700 1 --2- 172.21.2.112:0/362631795 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f04b40096e0 0x7f04b400bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:47.922 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:47.908 7f04c62bd700 1 -- 172.21.2.112:0/362631795 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04b400c420 con 0x7f04b40068e0 2021-02-10T11:17:47.922 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:47.908 7f04c62bd700 1 -- 172.21.2.112:0/362631795 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04b400c640 con 0x7f04b4003b90 2021-02-10T11:17:47.922 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:47.908 7f04c62bd700 1 -- 172.21.2.112:0/362631795 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04b400c860 con 0x7f04b40096e0 2021-02-10T11:17:47.922 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:47.908 7f04c72bf700 1 --2- 172.21.2.112:0/362631795 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f04c81b0470 0x7f04c81b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:17:47.922 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:47.908 7f04c6abe700 1 --2- 172.21.2.112:0/362631795 >> v2:172.21.2.120:3300/0 conn(0x7f04c80ef610 0x7f04c80efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:17:47.922 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:47.908 7f04c6abe700 1 --2- 172.21.2.112:0/362631795 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f04b40068e0 0x7f04b4008de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:47.922 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:47.908 7f04c6abe700 1 -- 172.21.2.112:0/362631795 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f04b40096e0 msgr2=0x7f04b400bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:17:47.922 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:47.908 7f04c6abe700 1 --2- 172.21.2.112:0/362631795 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f04b40096e0 0x7f04b400bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:17:47.922 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:47.908 7f04c6abe700 1 -- 172.21.2.112:0/362631795 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f04b4003b90 msgr2=0x7f04b4006040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:17:47.922 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:47.908 7f04c6abe700 1 --2- 172.21.2.112:0/362631795 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f04b4003b90 0x7f04b4006040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:17:47.922 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:47.908 7f04c6abe700 1 -- 172.21.2.112:0/362631795 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04b400cb00 con 0x7f04b40068e0 2021-02-10T11:17:47.923 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:47.908 7f04c6abe700 1 --2- 172.21.2.112:0/362631795 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f04b40068e0 0x7f04b4008de0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7f04b0009970 tx=0x7f04b0009d10).ready entity=mon.0 client_cookie=d47c31cb7964f1c5 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:17:47.923 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:47.908 7f04c62bd700 1 -- 172.21.2.112:0/362631795 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f04b00065d0 con 0x7f04b40068e0 2021-02-10T11:17:47.923 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:47.908 7f04c62bd700 1 -- 172.21.2.112:0/362631795 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04b0000c90 con 0x7f04b40068e0 2021-02-10T11:17:47.923 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:47.908 7f04c62bd700 1 -- 172.21.2.112:0/362631795 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f04b0002220 con 0x7f04b40068e0 2021-02-10T11:17:47.923 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:47.909 7f04cd6a2700 1 -- 172.21.2.112:0/362631795 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f04b40068e0 msgr2=0x7f04b4008de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:47.923 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:47.909 7f04cd6a2700 1 --2- 172.21.2.112:0/362631795 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f04b40068e0 0x7f04b4008de0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7f04b0009970 tx=0x7f04b0009d10).stop 2021-02-10T11:17:47.923 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:47.909 7f04c7ac0700 1 -- 172.21.2.112:0/362631795 reap_dead start 2021-02-10T11:17:47.923 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:47.909 7f04cd6a2700 1 -- 172.21.2.112:0/362631795 shutdown_connections 2021-02-10T11:17:47.923 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:47.909 7f04cd6a2700 1 -- 172.21.2.112:0/362631795 >> 172.21.2.112:0/362631795 conn(0x7f04c80ce3e0 msgr2=0x7f04c80e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:17:47.923 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:47.909 7f04cd6a2700 1 -- 172.21.2.112:0/362631795 shutdown_connections 2021-02-10T11:17:47.923 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:47.909 7f04cd6a2700 1 -- 172.21.2.112:0/362631795 wait complete. 2021-02-10T11:17:47.923 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:47.923 7f04cd6a2700 1 Processor -- start 2021-02-10T11:17:47.925 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:47.923 7f04cd6a2700 1 -- start start 2021-02-10T11:17:47.925 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:47.924 7f04cd6a2700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f04c809b930 0x7f04c80b7c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:47.925 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:47.924 7f04cd6a2700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f04c80b8410 0x7f04c80b8810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:47.925 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:47.924 7f04cd6a2700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f04c80c1f80 0x7f04c80c2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:47.925 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:47.924 7f04cd6a2700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04c80c1a00 con 0x7f04c80c1f80 2021-02-10T11:17:47.925 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:47.924 7f04cd6a2700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04c80b8e30 con 0x7f04c809b930 2021-02-10T11:17:47.925 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:47.924 7f04cd6a2700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04c80b9050 con 0x7f04c80b8410 2021-02-10T11:17:47.925 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:47.924 7f04c7ac0700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f04c80c1f80 0x7f04c80c2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:47.925 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:47.924 7f04c7ac0700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f04c80c1f80 0x7f04c80c2380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:44182/0 (socket says 172.21.2.112:44182) 2021-02-10T11:17:47.926 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:47.924 7f04c7ac0700 1 -- 172.21.2.112:0/1119078710 learned_addr learned my addr 172.21.2.112:0/1119078710 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:17:47.926 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:47.924 7f04c7ac0700 1 -- 172.21.2.112:0/1119078710 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f04c80b8410 msgr2=0x7f04c80b8810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:47.926 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:47.924 7f04c7ac0700 1 --2- 172.21.2.112:0/1119078710 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f04c80b8410 0x7f04c80b8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:17:47.926 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:47.924 7f04c7ac0700 1 -- 172.21.2.112:0/1119078710 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f04c809b930 msgr2=0x7f04c80b7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:47.926 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:47.924 7f04c7ac0700 1 --2- 172.21.2.112:0/1119078710 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f04c809b930 0x7f04c80b7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:17:47.926 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:47.924 7f04c7ac0700 1 -- 172.21.2.112:0/1119078710 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04c80c29a0 con 0x7f04c80c1f80 2021-02-10T11:17:47.926 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:47.925 7f04c7ac0700 1 --2- 172.21.2.112:0/1119078710 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f04c80c1f80 0x7f04c80c2380 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 rx=0x7f04c00021f0 tx=0x7f04c000abd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:17:47.926 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:47.925 7f04bffff700 1 -- 172.21.2.112:0/1119078710 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f04c0001390 con 0x7f04c80c1f80 2021-02-10T11:17:47.926 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:47.925 7f04bffff700 1 -- 172.21.2.112:0/1119078710 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04c00070d0 con 0x7f04c80c1f80 2021-02-10T11:17:47.926 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:47.925 7f04cd6a2700 1 -- 172.21.2.112:0/1119078710 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04c80c29a0 con 0x7f04c80c1f80 2021-02-10T11:17:47.926 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:47.925 7f04cd6a2700 1 -- 172.21.2.112:0/1119078710 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04c80c2f30 con 0x7f04c80c1f80 2021-02-10T11:17:47.926 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:47.925 7f04bffff700 1 -- 172.21.2.112:0/1119078710 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f04c0004930 con 0x7f04c80c1f80 2021-02-10T11:17:47.927 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:47.926 7f04bffff700 1 -- 172.21.2.112:0/1119078710 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f04c00072b0 con 0x7f04c80c1f80 2021-02-10T11:17:47.927 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:47.926 7f04bffff700 1 --2- 172.21.2.112:0/1119078710 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f04ac0186c0 0x7f04ac01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:47.927 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:47.926 7f04bffff700 1 -- 172.21.2.112:0/1119078710 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f04c00328d0 con 0x7f04c80c1f80 2021-02-10T11:17:47.927 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:47.926 7f04c72bf700 1 --2- 172.21.2.112:0/1119078710 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f04ac0186c0 0x7f04ac01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:47.927 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:47.926 7f04c72bf700 1 --2- 172.21.2.112:0/1119078710 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f04ac0186c0 0x7f04ac01abc0 crc :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:17:47.954 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:47.957 7f04cd6a2700 1 -- 172.21.2.112:0/1119078710 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04c8105b30 con 0x7f04c80c1f80 2021-02-10T11:17:47.956 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:47.960 7f04bffff700 1 -- 172.21.2.112:0/1119078710 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f04c0004bd0 con 0x7f04c80c1f80 2021-02-10T11:17:48.083 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:48.086 7f04cd6a2700 1 -- 172.21.2.112:0/1119078710 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348581", "target": ["mgr", ""]}) v1 -- 0x7f04c80f0b50 con 0x7f04ac0186c0 2021-02-10T11:17:48.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:48.112 7f04bffff700 1 -- 172.21.2.112:0/1119078710 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f04b8000940 con 0x7f04ac0186c0 2021-02-10T11:17:48.116 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:48.119 7f04cd6a2700 1 -- 172.21.2.112:0/1119078710 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f04ac0186c0 msgr2=0x7f04ac01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:48.116 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:48.119 7f04cd6a2700 1 --2- 172.21.2.112:0/1119078710 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f04ac0186c0 0x7f04ac01abc0 crc :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:48.116 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:48.119 7f04cd6a2700 1 -- 172.21.2.112:0/1119078710 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f04c80c1f80 msgr2=0x7f04c80c2380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:48.116 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:48.119 7f04cd6a2700 1 --2- 172.21.2.112:0/1119078710 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f04c80c1f80 0x7f04c80c2380 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 rx=0x7f04c00021f0 tx=0x7f04c000abd0).stop 2021-02-10T11:17:48.116 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:48.120 7f04cd6a2700 1 -- 172.21.2.112:0/1119078710 shutdown_connections 2021-02-10T11:17:48.116 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:48.120 7f04cd6a2700 1 --2- 172.21.2.112:0/1119078710 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f04ac0186c0 0x7f04ac01abc0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:48.116 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:48.120 7f04cd6a2700 1 --2- 172.21.2.112:0/1119078710 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f04c80c1f80 0x7f04c80c2380 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:48.117 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:48.120 7f04cd6a2700 1 --2- 172.21.2.112:0/1119078710 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f04c80b8410 0x7f04c80b8810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:17:48.117 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:48.120 7f04cd6a2700 1 --2- 172.21.2.112:0/1119078710 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f04c809b930 0x7f04c80b7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:17:48.117 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:48.120 7f04cd6a2700 1 -- 172.21.2.112:0/1119078710 >> 172.21.2.112:0/1119078710 conn(0x7f04c8094e10 msgr2=0x7f04c80a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:17:48.117 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:48.120 7f04cd6a2700 1 -- 172.21.2.112:0/1119078710 shutdown_connections 2021-02-10T11:17:48.117 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:48.120 7f04cd6a2700 1 -- 172.21.2.112:0/1119078710 wait complete. 2021-02-10T11:17:48.124 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000348581 2021-02-10T11:17:48.555 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:48.542 7f171b193700 1 Processor -- start 2021-02-10T11:17:48.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:48.543 7f171b193700 1 -- start start 2021-02-10T11:17:48.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:48.543 7f171b193700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f17141b0470 0x7f17141b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:48.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:48.543 7f171b193700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f17140ef610 0x7f17140efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:48.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:48.543 7f171b193700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17140f0b50 con 0x7f17140f0030 2021-02-10T11:17:48.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:48.543 7f171b193700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17140e52e0 con 0x7f17141b0470 2021-02-10T11:17:48.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:48.543 7f171b193700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17140e5490 con 0x7f17140ef610 2021-02-10T11:17:48.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:48.543 7f1718faf700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f17141b0470 0x7f17141b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:48.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:48.543 7f17197b0700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f17140f0030 0x7f17140f10d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:43650/0 (socket says 172.21.2.112:43650) 2021-02-10T11:17:48.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:48.543 7f1718faf700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f17141b0470 0x7f17141b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:44188/0 (socket says 172.21.2.112:44188) 2021-02-10T11:17:48.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:48.543 7f1718faf700 1 -- 172.21.2.112:0/2083126026 learned_addr learned my addr 172.21.2.112:0/2083126026 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:17:48.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:48.544 7f1718faf700 1 -- 172.21.2.112:0/2083126026 >> v2:172.21.2.120:3300/0 conn(0x7f17140ef610 msgr2=0x7f17140efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:17:48.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:48.544 7f1718faf700 1 --2- 172.21.2.112:0/2083126026 >> v2:172.21.2.120:3300/0 conn(0x7f17140ef610 0x7f17140efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:17:48.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:48.544 7f1718faf700 1 -- 172.21.2.112:0/2083126026 >> v1:172.21.2.112:6789/0 conn(0x7f17140f0030 legacy=0x7f17140f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:48.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:48.544 7f1718faf700 1 -- 172.21.2.112:0/2083126026 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17140f0430 con 0x7f17141b0470 2021-02-10T11:17:48.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:48.544 7f1718faf700 1 --2- 172.21.2.112:0/2083126026 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f17141b0470 0x7f17141b0870 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 rx=0x7f16fc006700 tx=0x7f16fc009b80).ready entity=mon.0 client_cookie=913b98157a0f573b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:17:48.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:48.544 7f17137fe700 1 -- 172.21.2.112:0/2083126026 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f16fc00c0c0 con 0x7f17141b0470 2021-02-10T11:17:48.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:48.544 7f17137fe700 1 -- 172.21.2.112:0/2083126026 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16fc00c440 con 0x7f17141b0470 2021-02-10T11:17:48.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:48.544 7f17137fe700 1 -- 172.21.2.112:0/2083126026 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f16fc00c8a0 con 0x7f17141b0470 2021-02-10T11:17:48.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:48.544 7f171b193700 1 -- 172.21.2.112:0/2083126026 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f17141b0470 msgr2=0x7f17141b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:48.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:48.544 7f171b193700 1 --2- 172.21.2.112:0/2083126026 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f17141b0470 0x7f17141b0870 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 rx=0x7f16fc006700 tx=0x7f16fc009b80).stop 2021-02-10T11:17:48.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:48.545 7f171b193700 1 -- 172.21.2.112:0/2083126026 shutdown_connections 2021-02-10T11:17:48.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:48.545 7f171b193700 1 --2- 172.21.2.112:0/2083126026 >> v2:172.21.2.120:3300/0 conn(0x7f17140ef610 0x7f17140efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:17:48.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:48.545 7f171b193700 1 --2- 172.21.2.112:0/2083126026 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f17141b0470 0x7f17141b0870 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:48.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:48.545 7f171b193700 1 -- 172.21.2.112:0/2083126026 >> 172.21.2.112:0/2083126026 conn(0x7f17140ce3e0 msgr2=0x7f17140e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:17:48.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:48.545 7f171b193700 1 -- 172.21.2.112:0/2083126026 shutdown_connections 2021-02-10T11:17:48.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:48.545 7f171b193700 1 -- 172.21.2.112:0/2083126026 wait complete. 2021-02-10T11:17:48.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:48.559 7f171b193700 1 Processor -- start 2021-02-10T11:17:48.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:48.559 7f171b193700 1 -- start start 2021-02-10T11:17:48.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:48.560 7f171b193700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f17140bf910 0x7f17140bfd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:48.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:48.560 7f171b193700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f17140c0330 0x7f17140c0730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:48.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:48.560 7f171b193700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f17140c13d0 0x7f17140c17d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:48.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:48.560 7f171b193700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1714109930 con 0x7f17140bf910 2021-02-10T11:17:48.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:48.560 7f171b193700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1714109b50 con 0x7f17140c13d0 2021-02-10T11:17:48.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:48.560 7f171b193700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17140c1df0 con 0x7f17140c0330 2021-02-10T11:17:48.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:48.560 7f1718faf700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f17140bf910 0x7f17140bfd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:48.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:48.560 7f1718faf700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f17140bf910 0x7f17140bfd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:44194/0 (socket says 172.21.2.112:44194) 2021-02-10T11:17:48.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:48.560 7f1718faf700 1 -- 172.21.2.112:0/629178013 learned_addr learned my addr 172.21.2.112:0/629178013 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:17:48.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:48.561 7f17197b0700 1 --2- 172.21.2.112:0/629178013 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f17140c13d0 0x7f17140c17d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:48.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:48.561 7f1713fff700 1 --2- 172.21.2.112:0/629178013 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f17140c0330 0x7f17140c0730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:48.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:48.561 7f1718faf700 1 -- 172.21.2.112:0/629178013 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f17140c0330 msgr2=0x7f17140c0730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:48.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:48.561 7f1718faf700 1 --2- 172.21.2.112:0/629178013 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f17140c0330 0x7f17140c0730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:48.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:48.561 7f1718faf700 1 -- 172.21.2.112:0/629178013 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f17140c13d0 msgr2=0x7f17140c17d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:48.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:48.561 7f1718faf700 1 --2- 172.21.2.112:0/629178013 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f17140c13d0 0x7f17140c17d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:48.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:48.561 7f1718faf700 1 -- 172.21.2.112:0/629178013 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17140c1fd0 con 0x7f17140bf910 2021-02-10T11:17:48.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:48.561 7f17197b0700 1 --2- 172.21.2.112:0/629178013 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f17140c13d0 0x7f17140c17d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:17:48.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:48.561 7f1718faf700 1 --2- 172.21.2.112:0/629178013 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f17140bf910 0x7f17140bfd10 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7f16fc00b580 tx=0x7f16fc00b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:17:48.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:48.561 7f17117fa700 1 -- 172.21.2.112:0/629178013 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f16fc00bd10 con 0x7f17140bf910 2021-02-10T11:17:48.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:48.561 7f17117fa700 1 -- 172.21.2.112:0/629178013 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16fc002030 con 0x7f17140bf910 2021-02-10T11:17:48.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:48.561 7f171b193700 1 -- 172.21.2.112:0/629178013 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17140c1fd0 con 0x7f17140bf910 2021-02-10T11:17:48.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:48.561 7f171b193700 1 -- 172.21.2.112:0/629178013 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17140c2560 con 0x7f17140bf910 2021-02-10T11:17:48.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:48.561 7f17117fa700 1 -- 172.21.2.112:0/629178013 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f16fc002480 con 0x7f17140bf910 2021-02-10T11:17:48.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:48.562 7f17117fa700 1 -- 172.21.2.112:0/629178013 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f16fc000c90 con 0x7f17140bf910 2021-02-10T11:17:48.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:48.562 7f17117fa700 1 --2- 172.21.2.112:0/629178013 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f16f80186c0 0x7f16f801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:48.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:48.562 7f17117fa700 1 -- 172.21.2.112:0/629178013 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f16fc0327b0 con 0x7f17140bf910 2021-02-10T11:17:48.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:48.562 7f1713fff700 1 --2- 172.21.2.112:0/629178013 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f16f80186c0 0x7f16f801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:48.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:48.563 7f1713fff700 1 --2- 172.21.2.112:0/629178013 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f16f80186c0 0x7f16f801abc0 crc :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:17:48.589 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:48.593 7f171b193700 1 -- 172.21.2.112:0/629178013 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17140f0b50 con 0x7f17140bf910 2021-02-10T11:17:48.592 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:48.595 7f17117fa700 1 -- 172.21.2.112:0/629178013 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f16fc0009b0 con 0x7f17140bf910 2021-02-10T11:17:48.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:48.724 7f171b193700 1 -- 172.21.2.112:0/629178013 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348581", "target": ["mgr", ""]}) v1 -- 0x7f17140f0b50 con 0x7f16f80186c0 2021-02-10T11:17:48.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:48.727 7f17117fa700 1 -- 172.21.2.112:0/629178013 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f1704000cc0 con 0x7f16f80186c0 2021-02-10T11:17:48.724 INFO:teuthology.orchestra.run.gibba012.stdout:/volumes/_nogroup/subvolume_0000000000348581/ed14c146-d84c-44e4-ad60-7947529b6a38 2021-02-10T11:17:48.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:48.729 7f171b193700 1 -- 172.21.2.112:0/629178013 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f16f80186c0 msgr2=0x7f16f801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:48.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:48.729 7f171b193700 1 --2- 172.21.2.112:0/629178013 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f16f80186c0 0x7f16f801abc0 crc :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:48.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:48.729 7f171b193700 1 -- 172.21.2.112:0/629178013 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f17140bf910 msgr2=0x7f17140bfd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:48.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:48.729 7f171b193700 1 --2- 172.21.2.112:0/629178013 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f17140bf910 0x7f17140bfd10 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7f16fc00b580 tx=0x7f16fc00b920).stop 2021-02-10T11:17:48.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:48.729 7f171b193700 1 -- 172.21.2.112:0/629178013 shutdown_connections 2021-02-10T11:17:48.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:48.729 7f171b193700 1 --2- 172.21.2.112:0/629178013 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f16f80186c0 0x7f16f801abc0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:48.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:48.729 7f171b193700 1 --2- 172.21.2.112:0/629178013 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f17140c13d0 0x7f17140c17d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:48.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:48.729 7f171b193700 1 --2- 172.21.2.112:0/629178013 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f17140c0330 0x7f17140c0730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:48.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:48.729 7f171b193700 1 --2- 172.21.2.112:0/629178013 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f17140bf910 0x7f17140bfd10 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:48.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:48.729 7f171b193700 1 -- 172.21.2.112:0/629178013 >> 172.21.2.112:0/629178013 conn(0x7f17140bf4e0 msgr2=0x7f1714191190 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:17:48.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:48.729 7f171b193700 1 -- 172.21.2.112:0/629178013 shutdown_connections 2021-02-10T11:17:48.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:48.729 7f171b193700 1 -- 172.21.2.112:0/629178013 wait complete. 2021-02-10T11:17:48.734 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000348581 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000348581/ed14c146-d84c-44e4-ad60-7947529b6a38 2021-02-10T11:17:48.734 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:48.734 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:48.734 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:49.068 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:17:49.068 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:49.069 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348581/ed14c146-d84c-44e4-ad60-7947529b6a38/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:49.444 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:17:49.444 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:17:49.444 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0232501 s, 45.1 MB/s 2021-02-10T11:17:49.445 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:49.445 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:49.445 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:49.781 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:17:49.781 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:49.782 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348581/ed14c146-d84c-44e4-ad60-7947529b6a38/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:50.157 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:17:50.157 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:17:50.157 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0257959 s, 40.6 MB/s 2021-02-10T11:17:50.158 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:50.158 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:50.158 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:50.492 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:17:50.492 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:50.493 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348581/ed14c146-d84c-44e4-ad60-7947529b6a38/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:50.867 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:17:50.867 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:17:50.867 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.02534 s, 41.4 MB/s 2021-02-10T11:17:50.868 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:50.868 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:50.868 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:51.203 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:17:51.203 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:51.203 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348581/ed14c146-d84c-44e4-ad60-7947529b6a38/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:51.578 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:17:51.578 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:17:51.578 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0244471 s, 42.9 MB/s 2021-02-10T11:17:51.579 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:51.579 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:51.579 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:51.913 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:17:51.913 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:51.913 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348581/ed14c146-d84c-44e4-ad60-7947529b6a38/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:52.288 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:17:52.288 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:17:52.288 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.02487 s, 42.2 MB/s 2021-02-10T11:17:52.289 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:52.289 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:52.289 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:52.623 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:17:52.623 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:52.624 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348581/ed14c146-d84c-44e4-ad60-7947529b6a38/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:53.001 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:17:53.001 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:17:53.001 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0267336 s, 39.2 MB/s 2021-02-10T11:17:53.002 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:53.002 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:53.002 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:53.337 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:17:53.337 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:53.337 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348581/ed14c146-d84c-44e4-ad60-7947529b6a38/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:53.716 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:17:53.716 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:17:53.716 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0235345 s, 44.6 MB/s 2021-02-10T11:17:53.717 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:53.718 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:53.718 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:54.055 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:17:54.056 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:54.056 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348581/ed14c146-d84c-44e4-ad60-7947529b6a38/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:54.427 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:17:54.427 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:17:54.427 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0237417 s, 44.2 MB/s 2021-02-10T11:17:54.428 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:54.428 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:54.428 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:54.770 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:17:54.771 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:54.771 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348581/ed14c146-d84c-44e4-ad60-7947529b6a38/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:55.148 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:17:55.148 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:17:55.148 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0235676 s, 44.5 MB/s 2021-02-10T11:17:55.149 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:55.149 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:55.149 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:55.487 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:17:55.487 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:55.487 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348581/ed14c146-d84c-44e4-ad60-7947529b6a38/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:55.864 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:17:55.864 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:17:55.865 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0260151 s, 40.3 MB/s 2021-02-10T11:17:55.866 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000348582 2021-02-10T11:17:56.296 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.282 7f5e8ce6a700 1 Processor -- start 2021-02-10T11:17:56.296 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.282 7f5e8ce6a700 1 -- start start 2021-02-10T11:17:56.296 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.283 7f5e8ce6a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5e881b0470 0x7f5e881b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:56.296 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.283 7f5e8ce6a700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e880f0b50 con 0x7f5e880ef610 2021-02-10T11:17:56.296 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.283 7f5e8ce6a700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e880f24e0 con 0x7f5e880f20e0 2021-02-10T11:17:56.296 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.283 7f5e8ce6a700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e880e5ea0 con 0x7f5e881b0470 2021-02-10T11:17:56.296 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.283 7f5e86312700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f5e880ef610 0x7f5e880efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:43664/0 (socket says 172.21.2.112:43664) 2021-02-10T11:17:56.296 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.283 7f5e86312700 1 -- 172.21.2.112:0/1733678553 learned_addr learned my addr 172.21.2.112:0/1733678553 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:17:56.296 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.283 7f5e86b13700 1 --2- 172.21.2.112:0/1733678553 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5e881b0470 0x7f5e881b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:56.296 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.283 7f5e85b11700 1 -- 172.21.2.112:0/1733678553 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1750085451 0 0) 0x7f5e700010b0 con 0x7f5e880ef610 2021-02-10T11:17:56.296 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.283 7f5e85b11700 1 -- 172.21.2.112:0/1733678553 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e74002fe0 con 0x7f5e880ef610 2021-02-10T11:17:56.297 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.283 7f5e85b11700 1 -- 172.21.2.112:0/1733678553 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1291238466 0 0) 0x7f5e80001400 con 0x7f5e880f20e0 2021-02-10T11:17:56.297 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.283 7f5e85b11700 1 -- 172.21.2.112:0/1733678553 --> v1:172.21.2.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e74004370 con 0x7f5e880f20e0 2021-02-10T11:17:56.297 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.283 7f5e86b13700 1 -- 172.21.2.112:0/1733678553 >> v1:172.21.2.120:6789/0 conn(0x7f5e880f20e0 legacy=0x7f5e880f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:56.297 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.283 7f5e86b13700 1 -- 172.21.2.112:0/1733678553 >> v1:172.21.2.112:6789/0 conn(0x7f5e880ef610 legacy=0x7f5e880efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:56.297 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.283 7f5e86b13700 1 -- 172.21.2.112:0/1733678553 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e880e52e0 con 0x7f5e881b0470 2021-02-10T11:17:56.297 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.284 7f5e86b13700 1 --2- 172.21.2.112:0/1733678553 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5e881b0470 0x7f5e881b0870 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f5e78006650 tx=0x7f5e78009a80).ready entity=mon.2 client_cookie=e19f3e11b4ce9f20 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:17:56.297 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.284 7f5e85b11700 1 -- 172.21.2.112:0/1733678553 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5e7800c0c0 con 0x7f5e881b0470 2021-02-10T11:17:56.297 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.284 7f5e85b11700 1 -- 172.21.2.112:0/1733678553 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e7800c340 con 0x7f5e881b0470 2021-02-10T11:17:56.297 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.284 7f5e85b11700 1 -- 172.21.2.112:0/1733678553 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5e7800c860 con 0x7f5e881b0470 2021-02-10T11:17:56.297 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.284 7f5e8ce6a700 1 -- 172.21.2.112:0/1733678553 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5e881b0470 msgr2=0x7f5e881b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:56.297 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.284 7f5e8ce6a700 1 --2- 172.21.2.112:0/1733678553 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5e881b0470 0x7f5e881b0870 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f5e78006650 tx=0x7f5e78009a80).stop 2021-02-10T11:17:56.297 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.285 7f5e8ce6a700 1 -- 172.21.2.112:0/1733678553 shutdown_connections 2021-02-10T11:17:56.297 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.285 7f5e8ce6a700 1 --2- 172.21.2.112:0/1733678553 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5e881b0470 0x7f5e881b0870 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:56.297 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.285 7f5e8ce6a700 1 -- 172.21.2.112:0/1733678553 >> 172.21.2.112:0/1733678553 conn(0x7f5e880ce3e0 msgr2=0x7f5e880e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:17:56.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.285 7f5e8ce6a700 1 -- 172.21.2.112:0/1733678553 shutdown_connections 2021-02-10T11:17:56.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.285 7f5e8ce6a700 1 -- 172.21.2.112:0/1733678553 wait complete. 2021-02-10T11:17:56.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.299 7f5e8ce6a700 1 Processor -- start 2021-02-10T11:17:56.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.300 7f5e8ce6a700 1 -- start start 2021-02-10T11:17:56.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.300 7f5e8ce6a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5e880a0420 0x7f5e880a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:56.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.300 7f5e8ce6a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5e88189d00 0x7f5e8818a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:56.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.300 7f5e8ce6a700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5e8818a720 0x7f5e880b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:56.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.300 7f5e8ce6a700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e88189780 con 0x7f5e880a0420 2021-02-10T11:17:56.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.300 7f5e8ce6a700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e8818ab20 con 0x7f5e8818a720 2021-02-10T11:17:56.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.300 7f5e8ce6a700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e880a1140 con 0x7f5e88189d00 2021-02-10T11:17:56.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.300 7f5e86b13700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5e880a0420 0x7f5e880a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:56.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.300 7f5e86b13700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5e880a0420 0x7f5e880a0820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:44206/0 (socket says 172.21.2.112:44206) 2021-02-10T11:17:56.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.300 7f5e86b13700 1 -- 172.21.2.112:0/142014750 learned_addr learned my addr 172.21.2.112:0/142014750 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:17:56.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.300 7f5e86b13700 1 -- 172.21.2.112:0/142014750 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5e88189d00 msgr2=0x7f5e8818a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:56.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.300 7f5e86b13700 1 --2- 172.21.2.112:0/142014750 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5e88189d00 0x7f5e8818a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:17:56.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.300 7f5e86b13700 1 -- 172.21.2.112:0/142014750 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5e8818a720 msgr2=0x7f5e880b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:56.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.300 7f5e86b13700 1 --2- 172.21.2.112:0/142014750 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5e8818a720 0x7f5e880b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:17:56.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.300 7f5e86b13700 1 -- 172.21.2.112:0/142014750 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e880b8200 con 0x7f5e880a0420 2021-02-10T11:17:56.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.301 7f5e86b13700 1 --2- 172.21.2.112:0/142014750 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5e880a0420 0x7f5e880a0820 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7f5e78009630 tx=0x7f5e7800b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:17:56.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.301 7f5e7f7fe700 1 -- 172.21.2.112:0/142014750 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5e780093a0 con 0x7f5e880a0420 2021-02-10T11:17:56.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.301 7f5e7f7fe700 1 -- 172.21.2.112:0/142014750 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e7800bd70 con 0x7f5e880a0420 2021-02-10T11:17:56.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.301 7f5e7f7fe700 1 -- 172.21.2.112:0/142014750 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5e78005560 con 0x7f5e880a0420 2021-02-10T11:17:56.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.301 7f5e8ce6a700 1 -- 172.21.2.112:0/142014750 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e880b8200 con 0x7f5e880a0420 2021-02-10T11:17:56.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.301 7f5e8ce6a700 1 -- 172.21.2.112:0/142014750 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e880b8200 con 0x7f5e880a0420 2021-02-10T11:17:56.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.302 7f5e7f7fe700 1 -- 172.21.2.112:0/142014750 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5e7800a000 con 0x7f5e880a0420 2021-02-10T11:17:56.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.302 7f5e7f7fe700 1 --2- 172.21.2.112:0/142014750 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5e6c018730 0x7f5e6c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:56.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.302 7f5e7f7fe700 1 -- 172.21.2.112:0/142014750 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f5e780327d0 con 0x7f5e880a0420 2021-02-10T11:17:56.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.302 7f5e86312700 1 --2- 172.21.2.112:0/142014750 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5e6c018730 0x7f5e6c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:56.302 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.303 7f5e86312700 1 --2- 172.21.2.112:0/142014750 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5e6c018730 0x7f5e6c01abc0 crc :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:17:56.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.332 7f5e8ce6a700 1 -- 172.21.2.112:0/142014750 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e88105ae0 con 0x7f5e880a0420 2021-02-10T11:17:56.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.335 7f5e7f7fe700 1 -- 172.21.2.112:0/142014750 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5e7801f510 con 0x7f5e880a0420 2021-02-10T11:17:56.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.462 7f5e8ce6a700 1 -- 172.21.2.112:0/142014750 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348582", "target": ["mgr", ""]}) v1 -- 0x7f5e880f0b50 con 0x7f5e6c018730 2021-02-10T11:17:56.484 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.488 7f5e7f7fe700 1 -- 172.21.2.112:0/142014750 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f5e700009b0 con 0x7f5e6c018730 2021-02-10T11:17:56.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.496 7f5e8ce6a700 1 -- 172.21.2.112:0/142014750 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5e6c018730 msgr2=0x7f5e6c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:56.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.496 7f5e8ce6a700 1 --2- 172.21.2.112:0/142014750 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5e6c018730 0x7f5e6c01abc0 crc :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:56.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.496 7f5e8ce6a700 1 -- 172.21.2.112:0/142014750 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5e880a0420 msgr2=0x7f5e880a0820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:56.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.496 7f5e8ce6a700 1 --2- 172.21.2.112:0/142014750 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5e880a0420 0x7f5e880a0820 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7f5e78009630 tx=0x7f5e7800b980).stop 2021-02-10T11:17:56.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.496 7f5e8ce6a700 1 -- 172.21.2.112:0/142014750 shutdown_connections 2021-02-10T11:17:56.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.496 7f5e8ce6a700 1 --2- 172.21.2.112:0/142014750 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5e6c018730 0x7f5e6c01abc0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:56.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.496 7f5e8ce6a700 1 --2- 172.21.2.112:0/142014750 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5e8818a720 0x7f5e880b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:17:56.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.496 7f5e8ce6a700 1 --2- 172.21.2.112:0/142014750 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5e88189d00 0x7f5e8818a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:17:56.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.496 7f5e8ce6a700 1 --2- 172.21.2.112:0/142014750 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5e880a0420 0x7f5e880a0820 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:56.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.496 7f5e8ce6a700 1 -- 172.21.2.112:0/142014750 >> 172.21.2.112:0/142014750 conn(0x7f5e88094e10 msgr2=0x7f5e880a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:17:56.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.496 7f5e8ce6a700 1 -- 172.21.2.112:0/142014750 shutdown_connections 2021-02-10T11:17:56.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.496 7f5e8ce6a700 1 -- 172.21.2.112:0/142014750 wait complete. 2021-02-10T11:17:56.501 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000348582 2021-02-10T11:17:56.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.918 7ffb9b2bb700 1 Processor -- start 2021-02-10T11:17:56.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.918 7ffb9b2bb700 1 -- start start 2021-02-10T11:17:56.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.918 7ffb9b2bb700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7ffb941b0470 0x7ffb941b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:56.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.918 7ffb9b2bb700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ffb940ef610 0x7ffb940efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:56.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.918 7ffb9b2bb700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7ffb940f0030 0x7ffb940f0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:56.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.918 7ffb9b2bb700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb940e5ea0 con 0x7ffb941b0470 2021-02-10T11:17:56.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.918 7ffb9b2bb700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb940f0b50 con 0x7ffb940f0030 2021-02-10T11:17:56.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.918 7ffb9b2bb700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb940e52e0 con 0x7ffb940ef610 2021-02-10T11:17:56.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.918 7ffb990d7700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7ffb941b0470 0x7ffb941b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:56.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.918 7ffb990d7700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7ffb941b0470 0x7ffb941b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:44214/0 (socket says 172.21.2.112:44214) 2021-02-10T11:17:56.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.918 7ffb990d7700 1 -- 172.21.2.112:0/3719115586 learned_addr learned my addr 172.21.2.112:0/3719115586 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:17:56.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.919 7ffb998d8700 1 --2- 172.21.2.112:0/3719115586 >> v2:172.21.2.120:3300/0 conn(0x7ffb940f0030 0x7ffb940f0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:56.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.919 7ffb988d6700 1 --2- 172.21.2.112:0/3719115586 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ffb940ef610 0x7ffb940efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:56.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.919 7ffb990d7700 1 -- 172.21.2.112:0/3719115586 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ffb940ef610 msgr2=0x7ffb940efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:17:56.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.919 7ffb990d7700 1 --2- 172.21.2.112:0/3719115586 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ffb940ef610 0x7ffb940efa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:56.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.919 7ffb990d7700 1 -- 172.21.2.112:0/3719115586 >> v2:172.21.2.120:3300/0 conn(0x7ffb940f0030 msgr2=0x7ffb940f0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:17:56.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.919 7ffb990d7700 1 --2- 172.21.2.112:0/3719115586 >> v2:172.21.2.120:3300/0 conn(0x7ffb940f0030 0x7ffb940f0430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:56.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.919 7ffb990d7700 1 -- 172.21.2.112:0/3719115586 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb940f10d0 con 0x7ffb941b0470 2021-02-10T11:17:56.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.919 7ffb990d7700 1 --2- 172.21.2.112:0/3719115586 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ffb941b0470 0x7ffb941b0870 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 rx=0x7ffb90006700 tx=0x7ffb90009b80).ready entity=mon.0 client_cookie=54fab25259b9211d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:17:56.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.919 7ffb87fff700 1 -- 172.21.2.112:0/3719115586 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffb9000c0c0 con 0x7ffb941b0470 2021-02-10T11:17:56.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.919 7ffb87fff700 1 -- 172.21.2.112:0/3719115586 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb9000c340 con 0x7ffb941b0470 2021-02-10T11:17:56.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.919 7ffb87fff700 1 -- 172.21.2.112:0/3719115586 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffb9000c860 con 0x7ffb941b0470 2021-02-10T11:17:56.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.919 7ffb9b2bb700 1 -- 172.21.2.112:0/3719115586 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ffb941b0470 msgr2=0x7ffb941b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:56.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.919 7ffb9b2bb700 1 --2- 172.21.2.112:0/3719115586 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ffb941b0470 0x7ffb941b0870 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 rx=0x7ffb90006700 tx=0x7ffb90009b80).stop 2021-02-10T11:17:56.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.920 7ffb9b2bb700 1 -- 172.21.2.112:0/3719115586 shutdown_connections 2021-02-10T11:17:56.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.920 7ffb9b2bb700 1 --2- 172.21.2.112:0/3719115586 >> v2:172.21.2.120:3300/0 conn(0x7ffb940f0030 0x7ffb940f0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:56.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.920 7ffb9b2bb700 1 --2- 172.21.2.112:0/3719115586 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ffb940ef610 0x7ffb940efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:56.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.920 7ffb9b2bb700 1 --2- 172.21.2.112:0/3719115586 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ffb941b0470 0x7ffb941b0870 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:56.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.920 7ffb9b2bb700 1 -- 172.21.2.112:0/3719115586 >> 172.21.2.112:0/3719115586 conn(0x7ffb940ce3e0 msgr2=0x7ffb940e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:17:56.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.920 7ffb9b2bb700 1 -- 172.21.2.112:0/3719115586 shutdown_connections 2021-02-10T11:17:56.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.920 7ffb9b2bb700 1 -- 172.21.2.112:0/3719115586 wait complete. 2021-02-10T11:17:56.935 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.934 7ffb9b2bb700 1 Processor -- start 2021-02-10T11:17:56.935 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.935 7ffb9b2bb700 1 -- start start 2021-02-10T11:17:56.935 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.935 7ffb9b2bb700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ffb9409e560 0x7ffb940a0130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:56.935 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.935 7ffb9b2bb700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ffb9409e960 0x7ffb940a0750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:56.935 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.935 7ffb9b2bb700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ffb940a0d70 0x7ffb940a1af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:56.935 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.935 7ffb9b2bb700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb940c38f0 con 0x7ffb9409e560 2021-02-10T11:17:56.935 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.935 7ffb9b2bb700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb940a1170 con 0x7ffb940a0d70 2021-02-10T11:17:56.935 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.935 7ffb9b2bb700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb940a1390 con 0x7ffb9409e960 2021-02-10T11:17:56.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.935 7ffb990d7700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ffb9409e560 0x7ffb940a0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:56.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.935 7ffb990d7700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ffb9409e560 0x7ffb940a0130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:44216/0 (socket says 172.21.2.112:44216) 2021-02-10T11:17:56.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.935 7ffb990d7700 1 -- 172.21.2.112:0/753471601 learned_addr learned my addr 172.21.2.112:0/753471601 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:17:56.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.935 7ffb998d8700 1 --2- 172.21.2.112:0/753471601 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ffb940a0d70 0x7ffb940a1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:56.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.935 7ffb988d6700 1 --2- 172.21.2.112:0/753471601 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ffb9409e960 0x7ffb940a0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:56.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.936 7ffb990d7700 1 -- 172.21.2.112:0/753471601 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ffb9409e960 msgr2=0x7ffb940a0750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:56.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.936 7ffb990d7700 1 --2- 172.21.2.112:0/753471601 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ffb9409e960 0x7ffb940a0750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:56.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.936 7ffb990d7700 1 -- 172.21.2.112:0/753471601 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ffb940a0d70 msgr2=0x7ffb940a1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:56.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.936 7ffb990d7700 1 --2- 172.21.2.112:0/753471601 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ffb940a0d70 0x7ffb940a1af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:56.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.936 7ffb990d7700 1 -- 172.21.2.112:0/753471601 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb940a2110 con 0x7ffb9409e560 2021-02-10T11:17:56.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.936 7ffb990d7700 1 --2- 172.21.2.112:0/753471601 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ffb9409e560 0x7ffb940a0130 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7ffb9000b5e0 tx=0x7ffb9000b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:17:56.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.936 7ffb85ffb700 1 -- 172.21.2.112:0/753471601 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffb9000bd70 con 0x7ffb9409e560 2021-02-10T11:17:56.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.936 7ffb85ffb700 1 -- 172.21.2.112:0/753471601 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb90002030 con 0x7ffb9409e560 2021-02-10T11:17:56.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.936 7ffb85ffb700 1 -- 172.21.2.112:0/753471601 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffb90002550 con 0x7ffb9409e560 2021-02-10T11:17:56.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.936 7ffb9b2bb700 1 -- 172.21.2.112:0/753471601 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb940a2110 con 0x7ffb9409e560 2021-02-10T11:17:56.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.936 7ffb9b2bb700 1 -- 172.21.2.112:0/753471601 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb940a2110 con 0x7ffb9409e560 2021-02-10T11:17:56.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.937 7ffb85ffb700 1 -- 172.21.2.112:0/753471601 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ffb90000c90 con 0x7ffb9409e560 2021-02-10T11:17:56.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.937 7ffb85ffb700 1 --2- 172.21.2.112:0/753471601 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ffb740186c0 0x7ffb7401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:56.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.937 7ffb85ffb700 1 -- 172.21.2.112:0/753471601 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7ffb900327c0 con 0x7ffb9409e560 2021-02-10T11:17:56.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.937 7ffb988d6700 1 --2- 172.21.2.112:0/753471601 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ffb740186c0 0x7ffb7401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:56.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.938 7ffb988d6700 1 --2- 172.21.2.112:0/753471601 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ffb740186c0 0x7ffb7401abc0 crc :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:17:56.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.968 7ffb9b2bb700 1 -- 172.21.2.112:0/753471601 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb94105b30 con 0x7ffb9409e560 2021-02-10T11:17:56.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:56.970 7ffb85ffb700 1 -- 172.21.2.112:0/753471601 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ffb9001f500 con 0x7ffb9409e560 2021-02-10T11:17:57.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:57.098 7ffb9b2bb700 1 -- 172.21.2.112:0/753471601 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348582", "target": ["mgr", ""]}) v1 -- 0x7ffb940f0b50 con 0x7ffb740186c0 2021-02-10T11:17:57.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:57.102 7ffb85ffb700 1 -- 172.21.2.112:0/753471601 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7ffb88000960 con 0x7ffb740186c0 2021-02-10T11:17:57.102 INFO:teuthology.orchestra.run.gibba012.stdout:/volumes/_nogroup/subvolume_0000000000348582/edff43e1-94c0-4627-8538-577d6d62a768 2021-02-10T11:17:57.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:57.106 7ffb9b2bb700 1 -- 172.21.2.112:0/753471601 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ffb740186c0 msgr2=0x7ffb7401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:57.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:57.106 7ffb9b2bb700 1 --2- 172.21.2.112:0/753471601 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ffb740186c0 0x7ffb7401abc0 crc :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:57.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:57.106 7ffb9b2bb700 1 -- 172.21.2.112:0/753471601 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ffb9409e560 msgr2=0x7ffb940a0130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:57.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:57.106 7ffb9b2bb700 1 --2- 172.21.2.112:0/753471601 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ffb9409e560 0x7ffb940a0130 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7ffb9000b5e0 tx=0x7ffb9000b980).stop 2021-02-10T11:17:57.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:57.107 7ffb9b2bb700 1 -- 172.21.2.112:0/753471601 shutdown_connections 2021-02-10T11:17:57.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:57.107 7ffb9b2bb700 1 --2- 172.21.2.112:0/753471601 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ffb740186c0 0x7ffb7401abc0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:57.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:57.107 7ffb9b2bb700 1 --2- 172.21.2.112:0/753471601 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ffb940a0d70 0x7ffb940a1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:57.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:57.107 7ffb9b2bb700 1 --2- 172.21.2.112:0/753471601 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ffb9409e960 0x7ffb940a0750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:57.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:57.107 7ffb9b2bb700 1 --2- 172.21.2.112:0/753471601 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ffb9409e560 0x7ffb940a0130 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:57.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:57.107 7ffb9b2bb700 1 -- 172.21.2.112:0/753471601 >> 172.21.2.112:0/753471601 conn(0x7ffb9409fc10 msgr2=0x7ffb94094ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:17:57.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:57.107 7ffb9b2bb700 1 -- 172.21.2.112:0/753471601 shutdown_connections 2021-02-10T11:17:57.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:17:57.107 7ffb9b2bb700 1 -- 172.21.2.112:0/753471601 wait complete. 2021-02-10T11:17:57.111 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000348582 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000348582/edff43e1-94c0-4627-8538-577d6d62a768 2021-02-10T11:17:57.111 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:57.111 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:57.111 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:57.447 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:17:57.447 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:57.447 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348582/edff43e1-94c0-4627-8538-577d6d62a768/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:57.831 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:17:57.832 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:17:57.832 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0261001 s, 40.2 MB/s 2021-02-10T11:17:57.833 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:57.833 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:57.833 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:58.168 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:17:58.169 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:58.169 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348582/edff43e1-94c0-4627-8538-577d6d62a768/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:58.542 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:17:58.543 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:17:58.543 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0233185 s, 45.0 MB/s 2021-02-10T11:17:58.544 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:58.544 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:58.544 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:58.880 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:17:58.880 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:58.881 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348582/edff43e1-94c0-4627-8538-577d6d62a768/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:59.257 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:17:59.257 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:17:59.257 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0258118 s, 40.6 MB/s 2021-02-10T11:17:59.258 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:59.258 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:59.258 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:59.592 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:17:59.593 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:59.593 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348582/edff43e1-94c0-4627-8538-577d6d62a768/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:59.968 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:17:59.968 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:17:59.968 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.027216 s, 38.5 MB/s 2021-02-10T11:17:59.969 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:59.969 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:59.969 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:00.303 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:18:00.303 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:00.303 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348582/edff43e1-94c0-4627-8538-577d6d62a768/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:00.677 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:18:00.677 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:18:00.677 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0223912 s, 46.8 MB/s 2021-02-10T11:18:00.678 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:00.678 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:00.678 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:01.012 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:18:01.013 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:01.013 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348582/edff43e1-94c0-4627-8538-577d6d62a768/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:01.389 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:18:01.389 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:18:01.389 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0243325 s, 43.1 MB/s 2021-02-10T11:18:01.390 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:01.390 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:01.390 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:01.724 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:18:01.725 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:01.725 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348582/edff43e1-94c0-4627-8538-577d6d62a768/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:02.098 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:18:02.098 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:18:02.098 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0241045 s, 43.5 MB/s 2021-02-10T11:18:02.099 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:02.099 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:02.099 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:02.433 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:18:02.433 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:02.434 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348582/edff43e1-94c0-4627-8538-577d6d62a768/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:02.809 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:18:02.809 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:18:02.809 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0250227 s, 41.9 MB/s 2021-02-10T11:18:02.810 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:02.810 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:02.810 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:03.144 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:18:03.145 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:03.145 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348582/edff43e1-94c0-4627-8538-577d6d62a768/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:03.523 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:18:03.523 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:18:03.523 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0268078 s, 39.1 MB/s 2021-02-10T11:18:03.525 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:03.525 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:03.525 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:03.861 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:18:03.862 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:03.862 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348582/edff43e1-94c0-4627-8538-577d6d62a768/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:04.096 DEBUG:teuthology.orchestra.run.gibba012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:18:04.100 DEBUG:teuthology.orchestra.run.gibba020:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:18:04.239 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:18:04.239 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:18:04.239 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0268321 s, 39.1 MB/s 2021-02-10T11:18:04.241 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000348583 2021-02-10T11:18:04.670 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:04.657 7f05b0ef1700 1 Processor -- start 2021-02-10T11:18:04.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:04.658 7f05b0ef1700 1 -- start start 2021-02-10T11:18:04.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:04.658 7f05b0ef1700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f05ac1b0470 0x7f05ac1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:04.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:04.658 7f05b0ef1700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f05ac0ef610 0x7f05ac0efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:04.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:04.658 7f05b0ef1700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f05ac0f0030 0x7f05ac0f0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:04.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:04.658 7f05b0ef1700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05ac0e5ea0 con 0x7f05ac1b0470 2021-02-10T11:18:04.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:04.658 7f05b0ef1700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05ac0f0b50 con 0x7f05ac0ef610 2021-02-10T11:18:04.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:04.658 7f05b0ef1700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05ac0e52e0 con 0x7f05ac0f0030 2021-02-10T11:18:04.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:04.658 7f05aab13700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f05ac1b0470 0x7f05ac1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:04.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:04.658 7f05aab13700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f05ac1b0470 0x7f05ac1b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:44224/0 (socket says 172.21.2.112:44224) 2021-02-10T11:18:04.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:04.658 7f05aab13700 1 -- 172.21.2.112:0/2673434750 learned_addr learned my addr 172.21.2.112:0/2673434750 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:18:04.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:04.659 7f05aab13700 1 -- 172.21.2.112:0/2673434750 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f05ac0f0030 msgr2=0x7f05ac0f0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:18:04.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:04.659 7f05aa312700 1 --2- 172.21.2.112:0/2673434750 >> v2:172.21.2.120:3300/0 conn(0x7f05ac0ef610 0x7f05ac0efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:04.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:04.659 7f05ab314700 1 --2- 172.21.2.112:0/2673434750 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f05ac0f0030 0x7f05ac0f0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:04.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:04.659 7f05aab13700 1 --2- 172.21.2.112:0/2673434750 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f05ac0f0030 0x7f05ac0f0430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:04.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:04.659 7f05aab13700 1 -- 172.21.2.112:0/2673434750 >> v2:172.21.2.120:3300/0 conn(0x7f05ac0ef610 msgr2=0x7f05ac0efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:18:04.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:04.659 7f05aab13700 1 --2- 172.21.2.112:0/2673434750 >> v2:172.21.2.120:3300/0 conn(0x7f05ac0ef610 0x7f05ac0efa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:04.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:04.659 7f05aab13700 1 -- 172.21.2.112:0/2673434750 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05ac0f10d0 con 0x7f05ac1b0470 2021-02-10T11:18:04.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:04.659 7f05aa312700 1 --2- 172.21.2.112:0/2673434750 >> v2:172.21.2.120:3300/0 conn(0x7f05ac0ef610 0x7f05ac0efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:18:04.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:04.659 7f05ab314700 1 --2- 172.21.2.112:0/2673434750 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f05ac0f0030 0x7f05ac0f0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:18:04.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:04.659 7f05aab13700 1 --2- 172.21.2.112:0/2673434750 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f05ac1b0470 0x7f05ac1b0870 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 rx=0x7f059c006700 tx=0x7f059c009b80).ready entity=mon.0 client_cookie=e73a8091c6b75de0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:18:04.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:04.659 7f05a9b11700 1 -- 172.21.2.112:0/2673434750 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f059c00c0c0 con 0x7f05ac1b0470 2021-02-10T11:18:04.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:04.659 7f05a9b11700 1 -- 172.21.2.112:0/2673434750 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f059c00c400 con 0x7f05ac1b0470 2021-02-10T11:18:04.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:04.660 7f05a9b11700 1 -- 172.21.2.112:0/2673434750 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f059c00c920 con 0x7f05ac1b0470 2021-02-10T11:18:04.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:04.660 7f05b0ef1700 1 -- 172.21.2.112:0/2673434750 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f05ac1b0470 msgr2=0x7f05ac1b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:04.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:04.660 7f05b0ef1700 1 --2- 172.21.2.112:0/2673434750 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f05ac1b0470 0x7f05ac1b0870 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 rx=0x7f059c006700 tx=0x7f059c009b80).stop 2021-02-10T11:18:04.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:04.660 7f05b0ef1700 1 -- 172.21.2.112:0/2673434750 shutdown_connections 2021-02-10T11:18:04.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:04.660 7f05b0ef1700 1 --2- 172.21.2.112:0/2673434750 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f05ac0f0030 0x7f05ac0f0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:04.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:04.660 7f05b0ef1700 1 --2- 172.21.2.112:0/2673434750 >> v2:172.21.2.120:3300/0 conn(0x7f05ac0ef610 0x7f05ac0efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:04.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:04.660 7f05b0ef1700 1 --2- 172.21.2.112:0/2673434750 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f05ac1b0470 0x7f05ac1b0870 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:04.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:04.660 7f05b0ef1700 1 -- 172.21.2.112:0/2673434750 >> 172.21.2.112:0/2673434750 conn(0x7f05ac0ce3e0 msgr2=0x7f05ac0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:18:04.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:04.660 7f05b0ef1700 1 -- 172.21.2.112:0/2673434750 shutdown_connections 2021-02-10T11:18:04.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:04.660 7f05b0ef1700 1 -- 172.21.2.112:0/2673434750 wait complete. 2021-02-10T11:18:04.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:04.675 7f05b0ef1700 1 Processor -- start 2021-02-10T11:18:04.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:04.675 7f05b0ef1700 1 -- start start 2021-02-10T11:18:04.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:04.675 7f05b0ef1700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f05ac0b7db0 0x7f05ac0b8260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:04.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:04.675 7f05b0ef1700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f05ac0b8880 0x7f05ac1a7450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:04.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:04.675 7f05b0ef1700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f05ac0b8c80 0x7f05ac1a7a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:04.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:04.675 7f05b0ef1700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05ac0b79b0 con 0x7f05ac0b7db0 2021-02-10T11:18:04.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:04.675 7f05b0ef1700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05ac1a8090 con 0x7f05ac0b8880 2021-02-10T11:18:04.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:04.675 7f05b0ef1700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05ac1a82b0 con 0x7f05ac0b8c80 2021-02-10T11:18:04.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:04.675 7f05aab13700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f05ac0b7db0 0x7f05ac0b8260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:04.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:04.675 7f05aab13700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f05ac0b7db0 0x7f05ac0b8260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:44228/0 (socket says 172.21.2.112:44228) 2021-02-10T11:18:04.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:04.675 7f05aab13700 1 -- 172.21.2.112:0/2855057111 learned_addr learned my addr 172.21.2.112:0/2855057111 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:18:04.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:04.676 7f05aab13700 1 -- 172.21.2.112:0/2855057111 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f05ac0b8c80 msgr2=0x7f05ac1a7a70 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:18:04.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:04.676 7f05aab13700 1 --2- 172.21.2.112:0/2855057111 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f05ac0b8c80 0x7f05ac1a7a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:18:04.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:04.676 7f05aab13700 1 -- 172.21.2.112:0/2855057111 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f05ac0b8880 msgr2=0x7f05ac1a7450 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:18:04.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:04.676 7f05aab13700 1 --2- 172.21.2.112:0/2855057111 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f05ac0b8880 0x7f05ac1a7450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:18:04.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:04.676 7f05aab13700 1 -- 172.21.2.112:0/2855057111 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05ac190c10 con 0x7f05ac0b7db0 2021-02-10T11:18:04.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:04.676 7f05aab13700 1 --2- 172.21.2.112:0/2855057111 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f05ac0b7db0 0x7f05ac0b8260 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7f059c0066a0 tx=0x7f059c00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:18:04.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:04.676 7f059b7fe700 1 -- 172.21.2.112:0/2855057111 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f059c00bd80 con 0x7f05ac0b7db0 2021-02-10T11:18:04.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:04.676 7f059b7fe700 1 -- 172.21.2.112:0/2855057111 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f059c0052e0 con 0x7f05ac0b7db0 2021-02-10T11:18:04.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:04.676 7f05b0ef1700 1 -- 172.21.2.112:0/2855057111 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05ac190c10 con 0x7f05ac0b7db0 2021-02-10T11:18:04.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:04.676 7f05b0ef1700 1 -- 172.21.2.112:0/2855057111 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05ac0c1eb0 con 0x7f05ac0b7db0 2021-02-10T11:18:04.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:04.676 7f059b7fe700 1 -- 172.21.2.112:0/2855057111 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f059c005800 con 0x7f05ac0b7db0 2021-02-10T11:18:04.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:04.677 7f059b7fe700 1 -- 172.21.2.112:0/2855057111 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f059c006030 con 0x7f05ac0b7db0 2021-02-10T11:18:04.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:04.677 7f059b7fe700 1 --2- 172.21.2.112:0/2855057111 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f05900186d0 0x7f059001abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:04.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:04.677 7f059b7fe700 1 -- 172.21.2.112:0/2855057111 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f059c0327f0 con 0x7f05ac0b7db0 2021-02-10T11:18:04.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:04.677 7f05aa312700 1 --2- 172.21.2.112:0/2855057111 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f05900186d0 0x7f059001abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:04.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:04.678 7f05aa312700 1 --2- 172.21.2.112:0/2855057111 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f05900186d0 0x7f059001abd0 crc :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:18:04.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:04.708 7f05b0ef1700 1 -- 172.21.2.112:0/2855057111 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05ac105b30 con 0x7f05ac0b7db0 2021-02-10T11:18:04.707 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:04.711 7f059b7fe700 1 -- 172.21.2.112:0/2855057111 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f059c0052e0 con 0x7f05ac0b7db0 2021-02-10T11:18:04.834 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:04.837 7f05b0ef1700 1 -- 172.21.2.112:0/2855057111 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348583", "target": ["mgr", ""]}) v1 -- 0x7f05ac0c1eb0 con 0x7f05900186d0 2021-02-10T11:18:04.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:04.863 7f059b7fe700 1 -- 172.21.2.112:0/2855057111 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f05a00009b0 con 0x7f05900186d0 2021-02-10T11:18:04.867 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:04.870 7f05b0ef1700 1 -- 172.21.2.112:0/2855057111 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f05900186d0 msgr2=0x7f059001abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:04.867 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:04.870 7f05b0ef1700 1 --2- 172.21.2.112:0/2855057111 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f05900186d0 0x7f059001abd0 crc :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:04.867 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:04.870 7f05b0ef1700 1 -- 172.21.2.112:0/2855057111 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f05ac0b7db0 msgr2=0x7f05ac0b8260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:04.868 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:04.870 7f05b0ef1700 1 --2- 172.21.2.112:0/2855057111 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f05ac0b7db0 0x7f05ac0b8260 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7f059c0066a0 tx=0x7f059c00b990).stop 2021-02-10T11:18:04.868 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:04.870 7f05b0ef1700 1 -- 172.21.2.112:0/2855057111 shutdown_connections 2021-02-10T11:18:04.868 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:04.870 7f05b0ef1700 1 --2- 172.21.2.112:0/2855057111 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f05900186d0 0x7f059001abd0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:04.868 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:04.870 7f05b0ef1700 1 --2- 172.21.2.112:0/2855057111 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f05ac0b8880 0x7f05ac1a7450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:18:04.868 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:04.870 7f05b0ef1700 1 --2- 172.21.2.112:0/2855057111 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f05ac0b8c80 0x7f05ac1a7a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:18:04.868 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:04.870 7f05b0ef1700 1 --2- 172.21.2.112:0/2855057111 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f05ac0b7db0 0x7f05ac0b8260 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:04.868 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:04.870 7f05b0ef1700 1 -- 172.21.2.112:0/2855057111 >> 172.21.2.112:0/2855057111 conn(0x7f05ac094e10 msgr2=0x7f05ac0a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:18:04.868 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:04.871 7f05b0ef1700 1 -- 172.21.2.112:0/2855057111 shutdown_connections 2021-02-10T11:18:04.868 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:04.871 7f05b0ef1700 1 -- 172.21.2.112:0/2855057111 wait complete. 2021-02-10T11:18:04.875 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000348583 2021-02-10T11:18:05.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:05.292 7fc9d0606700 1 Processor -- start 2021-02-10T11:18:05.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:05.292 7fc9d0606700 1 -- start start 2021-02-10T11:18:05.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:05.292 7fc9d0606700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fc9c81b0870 0x7fc9c80f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:05.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:05.292 7fc9d0606700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc9c80efe00 0x7fc9c80f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:05.306 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:05.292 7fc9d0606700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9c81b0c70 con 0x7fc9c81b0470 2021-02-10T11:18:05.306 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:05.293 7fc9d0606700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9c80f0b50 con 0x7fc9c81b0870 2021-02-10T11:18:05.306 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:05.293 7fc9d0606700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9c80f0820 con 0x7fc9c80efe00 2021-02-10T11:18:05.306 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:05.293 7fc9cdc21700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fc9c81b0870 0x7fc9c80f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:05.306 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:05.293 7fc9cdc21700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fc9c81b0870 0x7fc9c80f20e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:44236/0 (socket says 172.21.2.112:44236) 2021-02-10T11:18:05.306 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:05.293 7fc9cdc21700 1 -- 172.21.2.112:0/3014033534 learned_addr learned my addr 172.21.2.112:0/3014033534 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:18:05.306 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:05.293 7fc9cdc21700 1 -- 172.21.2.112:0/3014033534 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc9c80efe00 msgr2=0x7fc9c80f0200 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:18:05.306 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:05.293 7fc9cdc21700 1 --2- 172.21.2.112:0/3014033534 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc9c80efe00 0x7fc9c80f0200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:18:05.306 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:05.293 7fc9cdc21700 1 -- 172.21.2.112:0/3014033534 >> v1:172.21.2.120:6789/0 conn(0x7fc9c81b0470 legacy=0x7fc9c80ef610 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:18:05.306 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:05.293 7fc9cdc21700 1 -- 172.21.2.112:0/3014033534 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9c80e52e0 con 0x7fc9c81b0870 2021-02-10T11:18:05.307 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:05.293 7fc9cdc21700 1 --2- 172.21.2.112:0/3014033534 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc9c81b0870 0x7fc9c80f20e0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 rx=0x7fc9bc006700 tx=0x7fc9bc009b80).ready entity=mon.0 client_cookie=b055344912653a2a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:18:05.307 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:05.293 7fc9cd420700 1 -- 172.21.2.112:0/3014033534 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc9bc00c0c0 con 0x7fc9c81b0870 2021-02-10T11:18:05.307 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:05.293 7fc9cd420700 1 -- 172.21.2.112:0/3014033534 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9bc00c440 con 0x7fc9c81b0870 2021-02-10T11:18:05.307 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:05.293 7fc9cd420700 1 -- 172.21.2.112:0/3014033534 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc9bc00c960 con 0x7fc9c81b0870 2021-02-10T11:18:05.307 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:05.293 7fc9d0606700 1 -- 172.21.2.112:0/3014033534 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc9c81b0870 msgr2=0x7fc9c80f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:05.307 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:05.293 7fc9d0606700 1 --2- 172.21.2.112:0/3014033534 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc9c81b0870 0x7fc9c80f20e0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 rx=0x7fc9bc006700 tx=0x7fc9bc009b80).stop 2021-02-10T11:18:05.307 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:05.294 7fc9d0606700 1 -- 172.21.2.112:0/3014033534 shutdown_connections 2021-02-10T11:18:05.307 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:05.294 7fc9d0606700 1 --2- 172.21.2.112:0/3014033534 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc9c81b0870 0x7fc9c80f20e0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:05.307 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:05.294 7fc9d0606700 1 --2- 172.21.2.112:0/3014033534 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc9c80efe00 0x7fc9c80f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:18:05.307 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:05.294 7fc9d0606700 1 -- 172.21.2.112:0/3014033534 >> 172.21.2.112:0/3014033534 conn(0x7fc9c80ce3e0 msgr2=0x7fc9c80e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:18:05.307 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:05.294 7fc9d0606700 1 -- 172.21.2.112:0/3014033534 shutdown_connections 2021-02-10T11:18:05.308 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:05.294 7fc9d0606700 1 -- 172.21.2.112:0/3014033534 wait complete. 2021-02-10T11:18:05.308 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:05.309 7fc9d0606700 1 Processor -- start 2021-02-10T11:18:05.308 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:05.309 7fc9d0606700 1 -- start start 2021-02-10T11:18:05.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:05.309 7fc9d0606700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc9c80b2840 0x7fc9c81a4300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:05.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:05.309 7fc9d0606700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc9c80b30d0 0x7fc9c80b34d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:05.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:05.309 7fc9d0606700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc9c80b3af0 0x7fc9c80f0580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:05.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:05.309 7fc9d0606700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9c80b2c40 con 0x7fc9c80b30d0 2021-02-10T11:18:05.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:05.309 7fc9d0606700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9c81a4920 con 0x7fc9c80b3af0 2021-02-10T11:18:05.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:05.309 7fc9d0606700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9c80f0ba0 con 0x7fc9c80b2840 2021-02-10T11:18:05.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:05.309 7fc9cdc21700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc9c80b30d0 0x7fc9c80b34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:05.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:05.309 7fc9cdc21700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc9c80b30d0 0x7fc9c80b34d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:44242/0 (socket says 172.21.2.112:44242) 2021-02-10T11:18:05.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:05.309 7fc9cdc21700 1 -- 172.21.2.112:0/495262998 learned_addr learned my addr 172.21.2.112:0/495262998 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:18:05.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:05.310 7fc9cdc21700 1 -- 172.21.2.112:0/495262998 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc9c80b2840 msgr2=0x7fc9c81a4300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:05.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:05.310 7fc9cdc21700 1 --2- 172.21.2.112:0/495262998 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc9c80b2840 0x7fc9c81a4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:18:05.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:05.310 7fc9cdc21700 1 -- 172.21.2.112:0/495262998 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc9c80b3af0 msgr2=0x7fc9c80f0580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:05.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:05.310 7fc9cdc21700 1 --2- 172.21.2.112:0/495262998 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc9c80b3af0 0x7fc9c80f0580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:18:05.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:05.310 7fc9cdc21700 1 -- 172.21.2.112:0/495262998 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9c80f0dc0 con 0x7fc9c80b30d0 2021-02-10T11:18:05.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:05.310 7fc9cdc21700 1 --2- 172.21.2.112:0/495262998 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc9c80b30d0 0x7fc9c80b34d0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7fc9bc00b5e0 tx=0x7fc9bc00b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:18:05.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:05.310 7fc9baffd700 1 -- 172.21.2.112:0/495262998 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc9bc00bd70 con 0x7fc9c80b30d0 2021-02-10T11:18:05.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:05.310 7fc9baffd700 1 -- 172.21.2.112:0/495262998 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9bc002030 con 0x7fc9c80b30d0 2021-02-10T11:18:05.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:05.310 7fc9d0606700 1 -- 172.21.2.112:0/495262998 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9c80f0dc0 con 0x7fc9c80b30d0 2021-02-10T11:18:05.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:05.310 7fc9d0606700 1 -- 172.21.2.112:0/495262998 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9c80f0dc0 con 0x7fc9c80b30d0 2021-02-10T11:18:05.311 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:05.310 7fc9baffd700 1 -- 172.21.2.112:0/495262998 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc9bc002550 con 0x7fc9c80b30d0 2021-02-10T11:18:05.311 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:05.310 7fc9baffd700 1 -- 172.21.2.112:0/495262998 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc9bc000c90 con 0x7fc9c80b30d0 2021-02-10T11:18:05.311 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:05.310 7fc9baffd700 1 --2- 172.21.2.112:0/495262998 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc9b0018730 0x7fc9b001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:05.311 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:05.310 7fc9baffd700 1 -- 172.21.2.112:0/495262998 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fc9bc0326b0 con 0x7fc9c80b30d0 2021-02-10T11:18:05.311 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:05.311 7fc9ce422700 1 --2- 172.21.2.112:0/495262998 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc9b0018730 0x7fc9b001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:05.311 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:05.311 7fc9ce422700 1 --2- 172.21.2.112:0/495262998 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc9b0018730 0x7fc9b001ac10 crc :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:18:05.339 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:05.342 7fc9d0606700 1 -- 172.21.2.112:0/495262998 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9c80f0050 con 0x7fc9c80b30d0 2021-02-10T11:18:05.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:05.345 7fc9baffd700 1 -- 172.21.2.112:0/495262998 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc9bc01f570 con 0x7fc9c80b30d0 2021-02-10T11:18:05.470 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:05.474 7fc9d0606700 1 -- 172.21.2.112:0/495262998 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348583", "target": ["mgr", ""]}) v1 -- 0x7fc9c80f0050 con 0x7fc9b0018730 2021-02-10T11:18:05.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:05.478 7fc9baffd700 1 -- 172.21.2.112:0/495262998 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7fc9c4006a10 con 0x7fc9b0018730 2021-02-10T11:18:05.478 INFO:teuthology.orchestra.run.gibba012.stdout:/volumes/_nogroup/subvolume_0000000000348583/f47eed6b-e152-422f-884b-23a446d33215 2021-02-10T11:18:05.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:05.484 7fc9d0606700 1 -- 172.21.2.112:0/495262998 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc9b0018730 msgr2=0x7fc9b001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:05.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:05.484 7fc9d0606700 1 --2- 172.21.2.112:0/495262998 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc9b0018730 0x7fc9b001ac10 crc :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:05.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:05.484 7fc9d0606700 1 -- 172.21.2.112:0/495262998 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc9c80b30d0 msgr2=0x7fc9c80b34d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:05.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:05.484 7fc9d0606700 1 --2- 172.21.2.112:0/495262998 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc9c80b30d0 0x7fc9c80b34d0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7fc9bc00b5e0 tx=0x7fc9bc00b980).stop 2021-02-10T11:18:05.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:05.484 7fc9d0606700 1 -- 172.21.2.112:0/495262998 shutdown_connections 2021-02-10T11:18:05.481 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:05.484 7fc9d0606700 1 --2- 172.21.2.112:0/495262998 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc9b0018730 0x7fc9b001ac10 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:05.481 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:05.484 7fc9d0606700 1 --2- 172.21.2.112:0/495262998 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc9c80b3af0 0x7fc9c80f0580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:18:05.481 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:05.484 7fc9d0606700 1 --2- 172.21.2.112:0/495262998 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc9c80b30d0 0x7fc9c80b34d0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:05.481 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:05.484 7fc9d0606700 1 --2- 172.21.2.112:0/495262998 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc9c80b2840 0x7fc9c81a4300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:18:05.481 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:05.484 7fc9d0606700 1 -- 172.21.2.112:0/495262998 >> 172.21.2.112:0/495262998 conn(0x7fc9c80c36f0 msgr2=0x7fc9c8094c30 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:18:05.481 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:05.484 7fc9d0606700 1 -- 172.21.2.112:0/495262998 shutdown_connections 2021-02-10T11:18:05.481 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:05.484 7fc9d0606700 1 -- 172.21.2.112:0/495262998 wait complete. 2021-02-10T11:18:05.488 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000348583 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000348583/f47eed6b-e152-422f-884b-23a446d33215 2021-02-10T11:18:05.488 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:05.488 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:05.488 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:05.823 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:18:05.823 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:05.823 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348583/f47eed6b-e152-422f-884b-23a446d33215/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:06.205 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:18:06.205 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:18:06.205 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.026424 s, 39.7 MB/s 2021-02-10T11:18:06.206 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:06.206 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:06.207 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:06.540 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:18:06.540 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:06.541 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348583/f47eed6b-e152-422f-884b-23a446d33215/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:06.915 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:18:06.915 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:18:06.915 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0240752 s, 43.6 MB/s 2021-02-10T11:18:06.916 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:06.916 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:06.916 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:07.251 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:18:07.252 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:07.252 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348583/f47eed6b-e152-422f-884b-23a446d33215/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:07.629 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:18:07.629 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:18:07.630 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0264034 s, 39.7 MB/s 2021-02-10T11:18:07.630 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:07.631 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:07.631 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:07.966 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:18:07.967 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:07.967 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348583/f47eed6b-e152-422f-884b-23a446d33215/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:08.346 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:18:08.346 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:18:08.346 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0269801 s, 38.9 MB/s 2021-02-10T11:18:08.347 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:08.347 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:08.347 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:08.683 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:18:08.684 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:08.684 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348583/f47eed6b-e152-422f-884b-23a446d33215/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:09.059 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:18:09.059 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:18:09.059 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0245199 s, 42.8 MB/s 2021-02-10T11:18:09.060 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:09.060 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:09.060 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:09.394 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:18:09.394 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:09.394 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348583/f47eed6b-e152-422f-884b-23a446d33215/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:09.766 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:18:09.766 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:18:09.766 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0259603 s, 40.4 MB/s 2021-02-10T11:18:09.767 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:09.767 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:09.767 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:10.103 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:18:10.103 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:10.103 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348583/f47eed6b-e152-422f-884b-23a446d33215/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:10.480 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:18:10.480 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:18:10.480 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0254769 s, 41.2 MB/s 2021-02-10T11:18:10.481 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:10.481 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:10.481 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:10.816 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:18:10.816 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:10.816 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348583/f47eed6b-e152-422f-884b-23a446d33215/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:11.193 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:18:11.210 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:18:11.210 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0258538 s, 40.6 MB/s 2021-02-10T11:18:11.211 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:11.211 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:11.211 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:11.546 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:18:11.546 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:11.547 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348583/f47eed6b-e152-422f-884b-23a446d33215/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:11.922 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:18:11.922 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:18:11.922 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.025342 s, 41.4 MB/s 2021-02-10T11:18:11.923 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:11.923 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:11.923 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:12.257 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:18:12.257 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:12.258 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348583/f47eed6b-e152-422f-884b-23a446d33215/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:12.630 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:18:12.630 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:18:12.630 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0231252 s, 45.3 MB/s 2021-02-10T11:18:12.631 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000348584 2021-02-10T11:18:13.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:13.048 7fc134a5a700 1 Processor -- start 2021-02-10T11:18:13.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:13.048 7fc134a5a700 1 -- start start 2021-02-10T11:18:13.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:13.048 7fc134a5a700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fc1301b0470 0x7fc1301b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:13.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:13.048 7fc134a5a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc1300ef610 0x7fc1300efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:13.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:13.049 7fc134a5a700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1300f0b50 con 0x7fc1300f0030 2021-02-10T11:18:13.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:13.049 7fc134a5a700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1300e52e0 con 0x7fc1301b0470 2021-02-10T11:18:13.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:13.049 7fc134a5a700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1300e54d0 con 0x7fc1300ef610 2021-02-10T11:18:13.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:13.049 7fc12e85e700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fc1301b0470 0x7fc1301b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:13.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:13.049 7fc12f05f700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fc1300f0030 0x7fc1300f10d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:43716/0 (socket says 172.21.2.112:43716) 2021-02-10T11:18:13.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:13.049 7fc12f05f700 1 -- 172.21.2.112:0/3436980244 learned_addr learned my addr 172.21.2.112:0/3436980244 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:18:13.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:13.049 7fc12e05d700 1 --2- 172.21.2.112:0/3436980244 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc1300ef610 0x7fc1300efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:13.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:13.049 7fc12e85e700 1 -- 172.21.2.112:0/3436980244 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc1300ef610 msgr2=0x7fc1300efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:18:13.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:13.049 7fc12e85e700 1 --2- 172.21.2.112:0/3436980244 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc1300ef610 0x7fc1300efa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:13.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:13.049 7fc12e85e700 1 -- 172.21.2.112:0/3436980244 >> v1:172.21.2.112:6789/0 conn(0x7fc1300f0030 legacy=0x7fc1300f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:13.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:13.049 7fc12e85e700 1 -- 172.21.2.112:0/3436980244 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1300f0430 con 0x7fc1301b0470 2021-02-10T11:18:13.063 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:13.049 7fc12e85e700 1 --2- 172.21.2.112:0/3436980244 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc1301b0470 0x7fc1301b0870 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7fc118006650 tx=0x7fc118009a80).ready entity=mon.0 client_cookie=2a97d0997fa187dc server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:18:13.063 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:13.050 7fc12d85c700 1 -- 172.21.2.112:0/3436980244 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc11800c0c0 con 0x7fc1301b0470 2021-02-10T11:18:13.063 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:13.050 7fc12d85c700 1 -- 172.21.2.112:0/3436980244 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc11800c340 con 0x7fc1301b0470 2021-02-10T11:18:13.063 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:13.050 7fc12d85c700 1 -- 172.21.2.112:0/3436980244 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc11800c860 con 0x7fc1301b0470 2021-02-10T11:18:13.063 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:13.050 7fc134a5a700 1 -- 172.21.2.112:0/3436980244 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc1301b0470 msgr2=0x7fc1301b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:13.063 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:13.050 7fc134a5a700 1 --2- 172.21.2.112:0/3436980244 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc1301b0470 0x7fc1301b0870 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7fc118006650 tx=0x7fc118009a80).stop 2021-02-10T11:18:13.063 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:13.051 7fc134a5a700 1 -- 172.21.2.112:0/3436980244 shutdown_connections 2021-02-10T11:18:13.063 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:13.051 7fc134a5a700 1 --2- 172.21.2.112:0/3436980244 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc1300ef610 0x7fc1300efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:13.063 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:13.051 7fc134a5a700 1 --2- 172.21.2.112:0/3436980244 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc1301b0470 0x7fc1301b0870 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:13.063 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:13.051 7fc134a5a700 1 -- 172.21.2.112:0/3436980244 >> 172.21.2.112:0/3436980244 conn(0x7fc1300ce3e0 msgr2=0x7fc1300e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:18:13.063 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:13.051 7fc134a5a700 1 -- 172.21.2.112:0/3436980244 shutdown_connections 2021-02-10T11:18:13.063 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:13.051 7fc134a5a700 1 -- 172.21.2.112:0/3436980244 wait complete. 2021-02-10T11:18:13.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:13.065 7fc134a5a700 1 Processor -- start 2021-02-10T11:18:13.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:13.065 7fc134a5a700 1 -- start start 2021-02-10T11:18:13.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:13.066 7fc134a5a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc1300a0420 0x7fc1300a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:13.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:13.066 7fc134a5a700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc130189d00 0x7fc13018a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:13.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:13.066 7fc134a5a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc13018a720 0x7fc1300b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:13.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:13.066 7fc134a5a700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc130189780 con 0x7fc1300a0420 2021-02-10T11:18:13.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:13.066 7fc134a5a700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc13018ab20 con 0x7fc130189d00 2021-02-10T11:18:13.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:13.066 7fc134a5a700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1300a1140 con 0x7fc13018a720 2021-02-10T11:18:13.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:13.066 7fc12e85e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc1300a0420 0x7fc1300a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:13.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:13.066 7fc12e85e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc1300a0420 0x7fc1300a0820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:44256/0 (socket says 172.21.2.112:44256) 2021-02-10T11:18:13.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:13.066 7fc12e85e700 1 -- 172.21.2.112:0/1241932857 learned_addr learned my addr 172.21.2.112:0/1241932857 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:18:13.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:13.066 7fc12e85e700 1 -- 172.21.2.112:0/1241932857 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc13018a720 msgr2=0x7fc1300b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:13.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:13.066 7fc12e85e700 1 --2- 172.21.2.112:0/1241932857 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc13018a720 0x7fc1300b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:18:13.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:13.067 7fc12e85e700 1 -- 172.21.2.112:0/1241932857 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc130189d00 msgr2=0x7fc13018a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:13.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:13.067 7fc12e85e700 1 --2- 172.21.2.112:0/1241932857 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc130189d00 0x7fc13018a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:18:13.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:13.067 7fc12e85e700 1 -- 172.21.2.112:0/1241932857 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1300b8200 con 0x7fc1300a0420 2021-02-10T11:18:13.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:13.067 7fc12e85e700 1 --2- 172.21.2.112:0/1241932857 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc1300a0420 0x7fc1300a0820 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 rx=0x7fc118009630 tx=0x7fc11800b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:18:13.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:13.067 7fc1277fe700 1 -- 172.21.2.112:0/1241932857 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc1180093a0 con 0x7fc1300a0420 2021-02-10T11:18:13.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:13.067 7fc1277fe700 1 -- 172.21.2.112:0/1241932857 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc11800bd70 con 0x7fc1300a0420 2021-02-10T11:18:13.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:13.067 7fc1277fe700 1 -- 172.21.2.112:0/1241932857 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc1180054d0 con 0x7fc1300a0420 2021-02-10T11:18:13.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:13.067 7fc134a5a700 1 -- 172.21.2.112:0/1241932857 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1300b8200 con 0x7fc1300a0420 2021-02-10T11:18:13.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:13.067 7fc134a5a700 1 -- 172.21.2.112:0/1241932857 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1300b8200 con 0x7fc1300a0420 2021-02-10T11:18:13.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:13.068 7fc1277fe700 1 -- 172.21.2.112:0/1241932857 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc118008240 con 0x7fc1300a0420 2021-02-10T11:18:13.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:13.068 7fc1277fe700 1 --2- 172.21.2.112:0/1241932857 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc1140186c0 0x7fc11401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:13.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:13.068 7fc1277fe700 1 -- 172.21.2.112:0/1241932857 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fc118032530 con 0x7fc1300a0420 2021-02-10T11:18:13.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:13.069 7fc12e05d700 1 --2- 172.21.2.112:0/1241932857 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc1140186c0 0x7fc11401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:13.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:13.069 7fc12e05d700 1 --2- 172.21.2.112:0/1241932857 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc1140186c0 0x7fc11401abc0 crc :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:18:13.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:13.098 7fc134a5a700 1 -- 172.21.2.112:0/1241932857 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc130105ae0 con 0x7fc1300a0420 2021-02-10T11:18:13.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:13.101 7fc1277fe700 1 -- 172.21.2.112:0/1241932857 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc118030ce0 con 0x7fc1300a0420 2021-02-10T11:18:13.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:13.226 7fc134a5a700 1 -- 172.21.2.112:0/1241932857 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348584", "target": ["mgr", ""]}) v1 -- 0x7fc130189780 con 0x7fc1140186c0 2021-02-10T11:18:13.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:13.250 7fc1277fe700 1 -- 172.21.2.112:0/1241932857 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fc1200009b0 con 0x7fc1140186c0 2021-02-10T11:18:13.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:13.259 7fc134a5a700 1 -- 172.21.2.112:0/1241932857 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc1140186c0 msgr2=0x7fc11401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:13.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:13.259 7fc134a5a700 1 --2- 172.21.2.112:0/1241932857 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc1140186c0 0x7fc11401abc0 crc :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:13.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:13.259 7fc134a5a700 1 -- 172.21.2.112:0/1241932857 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc1300a0420 msgr2=0x7fc1300a0820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:13.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:13.259 7fc134a5a700 1 --2- 172.21.2.112:0/1241932857 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc1300a0420 0x7fc1300a0820 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 rx=0x7fc118009630 tx=0x7fc11800b980).stop 2021-02-10T11:18:13.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:13.260 7fc134a5a700 1 -- 172.21.2.112:0/1241932857 shutdown_connections 2021-02-10T11:18:13.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:13.260 7fc134a5a700 1 --2- 172.21.2.112:0/1241932857 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc1140186c0 0x7fc11401abc0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:13.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:13.260 7fc134a5a700 1 --2- 172.21.2.112:0/1241932857 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc130189d00 0x7fc13018a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:18:13.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:13.260 7fc134a5a700 1 --2- 172.21.2.112:0/1241932857 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc13018a720 0x7fc1300b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:18:13.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:13.260 7fc134a5a700 1 --2- 172.21.2.112:0/1241932857 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc1300a0420 0x7fc1300a0820 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:13.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:13.260 7fc134a5a700 1 -- 172.21.2.112:0/1241932857 >> 172.21.2.112:0/1241932857 conn(0x7fc130094e10 msgr2=0x7fc1300a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:18:13.586 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:13.260 7fc134a5a700 1 -- 172.21.2.112:0/1241932857 shutdown_connections 2021-02-10T11:18:13.586 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:13.260 7fc134a5a700 1 -- 172.21.2.112:0/1241932857 wait complete. 2021-02-10T11:18:13.586 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000348584 2021-02-10T11:18:14.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:14.003 7f2dcffae700 1 Processor -- start 2021-02-10T11:18:14.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:14.003 7f2dcffae700 1 -- start start 2021-02-10T11:18:14.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:14.004 7f2dcffae700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f2dc80f05f0 0x7f2dc80f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:14.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:14.004 7f2dcffae700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dc81b0c70 con 0x7f2dc81b0470 2021-02-10T11:18:14.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:14.004 7f2dcffae700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dc80f0b50 con 0x7f2dc81b0870 2021-02-10T11:18:14.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:14.004 7f2dcffae700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2dc80e5ea0 con 0x7f2dc80f05f0 2021-02-10T11:18:14.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:14.004 7f2dce5cb700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f2dc80f05f0 0x7f2dc80f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:14.017 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:14.004 7f2dce5cb700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f2dc80f05f0 0x7f2dc80f20e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:44268/0 (socket says 172.21.2.112:44268) 2021-02-10T11:18:14.017 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:14.004 7f2dce5cb700 1 -- 172.21.2.112:0/2295183706 learned_addr learned my addr 172.21.2.112:0/2295183706 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:18:14.017 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:14.004 7f2dce5cb700 1 -- 172.21.2.112:0/2295183706 >> v1:172.21.2.120:6789/0 conn(0x7f2dc81b0870 legacy=0x7f2dc80efe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:18:14.017 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:14.004 7f2dce5cb700 1 -- 172.21.2.112:0/2295183706 >> v1:172.21.2.112:6789/0 conn(0x7f2dc81b0470 legacy=0x7f2dc80ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:14.017 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:14.004 7f2dce5cb700 1 -- 172.21.2.112:0/2295183706 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dc80e52e0 con 0x7f2dc80f05f0 2021-02-10T11:18:14.017 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:14.004 7f2dce5cb700 1 --2- 172.21.2.112:0/2295183706 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2dc80f05f0 0x7f2dc80f20e0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 rx=0x7f2dc0006f00 tx=0x7f2dc0009f10).ready entity=mon.0 client_cookie=54a1868b1e8b6296 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:18:14.017 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:14.005 7f2dccdc8700 1 -- 172.21.2.112:0/2295183706 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2dc000b3b0 con 0x7f2dc80f05f0 2021-02-10T11:18:14.017 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:14.005 7f2dccdc8700 1 -- 172.21.2.112:0/2295183706 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dc000b630 con 0x7f2dc80f05f0 2021-02-10T11:18:14.017 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:14.005 7f2dccdc8700 1 -- 172.21.2.112:0/2295183706 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2dc000bb50 con 0x7f2dc80f05f0 2021-02-10T11:18:14.017 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:14.005 7f2dcffae700 1 -- 172.21.2.112:0/2295183706 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2dc80f05f0 msgr2=0x7f2dc80f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:14.017 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:14.005 7f2dcffae700 1 --2- 172.21.2.112:0/2295183706 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2dc80f05f0 0x7f2dc80f20e0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 rx=0x7f2dc0006f00 tx=0x7f2dc0009f10).stop 2021-02-10T11:18:14.017 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:14.005 7f2dcffae700 1 -- 172.21.2.112:0/2295183706 shutdown_connections 2021-02-10T11:18:14.018 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:14.005 7f2dcffae700 1 --2- 172.21.2.112:0/2295183706 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2dc80f05f0 0x7f2dc80f20e0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:14.018 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:14.005 7f2dcffae700 1 -- 172.21.2.112:0/2295183706 >> 172.21.2.112:0/2295183706 conn(0x7f2dc80ce3e0 msgr2=0x7f2dc80e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:18:14.018 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:14.006 7f2dcffae700 1 -- 172.21.2.112:0/2295183706 shutdown_connections 2021-02-10T11:18:14.018 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:14.006 7f2dcffae700 1 -- 172.21.2.112:0/2295183706 wait complete. 2021-02-10T11:18:14.019 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:14.020 7f2dcffae700 1 Processor -- start 2021-02-10T11:18:14.019 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:14.020 7f2dcffae700 1 -- start start 2021-02-10T11:18:14.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:14.021 7f2dcffae700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2dc80a3b50 0x7f2dc81b3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:14.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:14.021 7f2dcffae700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2dc81c25b0 0x7f2dc81b3d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:14.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:14.021 7f2dcffae700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2dc81b4350 0x7f2dc81b18a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:14.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:14.021 7f2dcffae700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dc81b3190 con 0x7f2dc80a3b50 2021-02-10T11:18:14.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:14.021 7f2dcffae700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dc8087750 con 0x7f2dc81c25b0 2021-02-10T11:18:14.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:14.021 7f2dcffae700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2dc80a6bf0 con 0x7f2dc81b4350 2021-02-10T11:18:14.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:14.021 7f2dcddca700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2dc80a3b50 0x7f2dc81b3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:14.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:14.021 7f2dcddca700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2dc80a3b50 0x7f2dc81b3710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:44272/0 (socket says 172.21.2.112:44272) 2021-02-10T11:18:14.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:14.021 7f2dcddca700 1 -- 172.21.2.112:0/1292047606 learned_addr learned my addr 172.21.2.112:0/1292047606 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:18:14.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:14.021 7f2dcd5c9700 1 --2- 172.21.2.112:0/1292047606 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2dc81c25b0 0x7f2dc81b3d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:14.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:14.021 7f2dcddca700 1 -- 172.21.2.112:0/1292047606 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2dc81b4350 msgr2=0x7f2dc81b18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:14.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:14.021 7f2dcddca700 1 --2- 172.21.2.112:0/1292047606 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2dc81b4350 0x7f2dc81b18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:18:14.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:14.021 7f2dcddca700 1 -- 172.21.2.112:0/1292047606 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2dc81c25b0 msgr2=0x7f2dc81b3d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:14.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:14.021 7f2dcddca700 1 --2- 172.21.2.112:0/1292047606 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2dc81c25b0 0x7f2dc81b3d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:14.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:14.021 7f2dcddca700 1 -- 172.21.2.112:0/1292047606 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dc81b1ec0 con 0x7f2dc80a3b50 2021-02-10T11:18:14.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:14.021 7f2dcd5c9700 1 --2- 172.21.2.112:0/1292047606 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2dc81c25b0 0x7f2dc81b3d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:18:14.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:14.021 7f2dcddca700 1 --2- 172.21.2.112:0/1292047606 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2dc80a3b50 0x7f2dc81b3710 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7f2dbc006ed0 tx=0x7f2dbc00b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:18:14.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:14.022 7f2dba7fc700 1 -- 172.21.2.112:0/1292047606 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2dbc006020 con 0x7f2dc80a3b50 2021-02-10T11:18:14.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:14.022 7f2dba7fc700 1 -- 172.21.2.112:0/1292047606 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dbc0063a0 con 0x7f2dc80a3b50 2021-02-10T11:18:14.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:14.022 7f2dba7fc700 1 -- 172.21.2.112:0/1292047606 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2dbc005730 con 0x7f2dc80a3b50 2021-02-10T11:18:14.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:14.022 7f2dcffae700 1 -- 172.21.2.112:0/1292047606 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2dc81b1ec0 con 0x7f2dc80a3b50 2021-02-10T11:18:14.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:14.022 7f2dcffae700 1 -- 172.21.2.112:0/1292047606 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2dc81b1ec0 con 0x7f2dc80a3b50 2021-02-10T11:18:14.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:14.022 7f2dba7fc700 1 -- 172.21.2.112:0/1292047606 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2dbc006020 con 0x7f2dc80a3b50 2021-02-10T11:18:14.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:14.022 7f2dba7fc700 1 --2- 172.21.2.112:0/1292047606 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2db0018730 0x7f2db001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:14.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:14.022 7f2dba7fc700 1 -- 172.21.2.112:0/1292047606 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f2dbc032650 con 0x7f2dc80a3b50 2021-02-10T11:18:14.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:14.023 7f2dcd5c9700 1 --2- 172.21.2.112:0/1292047606 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2db0018730 0x7f2db001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:14.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:14.023 7f2dcd5c9700 1 --2- 172.21.2.112:0/1292047606 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2db0018730 0x7f2db001ac10 crc :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:18:14.050 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:14.054 7f2dcffae700 1 -- 172.21.2.112:0/1292047606 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2dc807e6b0 con 0x7f2dc80a3b50 2021-02-10T11:18:14.052 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:14.056 7f2dba7fc700 1 -- 172.21.2.112:0/1292047606 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2dbc0009b0 con 0x7f2dc80a3b50 2021-02-10T11:18:14.182 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:14.185 7f2dcffae700 1 -- 172.21.2.112:0/1292047606 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348584", "target": ["mgr", ""]}) v1 -- 0x7f2dc807e6b0 con 0x7f2db0018730 2021-02-10T11:18:14.185 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:14.189 7f2dba7fc700 1 -- 172.21.2.112:0/1292047606 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f2dc4006b20 con 0x7f2db0018730 2021-02-10T11:18:14.189 INFO:teuthology.orchestra.run.gibba012.stdout:/volumes/_nogroup/subvolume_0000000000348584/c75fd23b-66ac-47b5-8be8-e13237f757a4 2021-02-10T11:18:14.191 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:14.195 7f2dcffae700 1 -- 172.21.2.112:0/1292047606 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2db0018730 msgr2=0x7f2db001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:14.191 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:14.195 7f2dcffae700 1 --2- 172.21.2.112:0/1292047606 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2db0018730 0x7f2db001ac10 crc :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:14.191 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:14.195 7f2dcffae700 1 -- 172.21.2.112:0/1292047606 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2dc80a3b50 msgr2=0x7f2dc81b3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:14.191 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:14.195 7f2dcffae700 1 --2- 172.21.2.112:0/1292047606 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2dc80a3b50 0x7f2dc81b3710 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7f2dbc006ed0 tx=0x7f2dbc00b960).stop 2021-02-10T11:18:14.191 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:14.195 7f2dcffae700 1 -- 172.21.2.112:0/1292047606 shutdown_connections 2021-02-10T11:18:14.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:14.195 7f2dcffae700 1 --2- 172.21.2.112:0/1292047606 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2db0018730 0x7f2db001ac10 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:14.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:14.195 7f2dcffae700 1 --2- 172.21.2.112:0/1292047606 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2dc81c25b0 0x7f2dc81b3d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:14.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:14.195 7f2dcffae700 1 --2- 172.21.2.112:0/1292047606 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2dc81b4350 0x7f2dc81b18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:18:14.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:14.195 7f2dcffae700 1 --2- 172.21.2.112:0/1292047606 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2dc80a3b50 0x7f2dc81b3710 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:14.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:14.195 7f2dcffae700 1 -- 172.21.2.112:0/1292047606 >> 172.21.2.112:0/1292047606 conn(0x7f2dc80886d0 msgr2=0x7f2dc8081c40 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:18:14.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:14.195 7f2dcffae700 1 -- 172.21.2.112:0/1292047606 shutdown_connections 2021-02-10T11:18:14.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:14.195 7f2dcffae700 1 -- 172.21.2.112:0/1292047606 wait complete. 2021-02-10T11:18:14.199 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000348584 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000348584/c75fd23b-66ac-47b5-8be8-e13237f757a4 2021-02-10T11:18:14.199 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:14.199 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:14.199 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:14.534 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:18:14.534 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:14.535 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348584/c75fd23b-66ac-47b5-8be8-e13237f757a4/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:14.914 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:18:14.914 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:18:14.914 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0242322 s, 43.3 MB/s 2021-02-10T11:18:14.915 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:14.915 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:14.915 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:15.253 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:18:15.253 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:15.254 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348584/c75fd23b-66ac-47b5-8be8-e13237f757a4/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:15.629 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:18:15.629 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:18:15.629 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0227102 s, 46.2 MB/s 2021-02-10T11:18:15.630 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:15.630 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:15.630 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:15.974 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:18:15.976 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:15.976 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348584/c75fd23b-66ac-47b5-8be8-e13237f757a4/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:16.349 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:18:16.349 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:18:16.349 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0230272 s, 45.5 MB/s 2021-02-10T11:18:16.350 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:16.351 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:16.351 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:16.687 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:18:16.688 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:16.688 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348584/c75fd23b-66ac-47b5-8be8-e13237f757a4/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:17.062 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:18:17.063 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:18:17.063 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0244455 s, 42.9 MB/s 2021-02-10T11:18:17.064 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:17.064 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:17.064 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:17.398 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:18:17.398 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:17.399 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348584/c75fd23b-66ac-47b5-8be8-e13237f757a4/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:17.773 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:18:17.774 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:18:17.774 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0243498 s, 43.1 MB/s 2021-02-10T11:18:17.775 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:17.775 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:17.775 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:18.109 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:18:18.110 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:18.110 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348584/c75fd23b-66ac-47b5-8be8-e13237f757a4/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:18.485 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:18:18.485 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:18:18.485 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0246842 s, 42.5 MB/s 2021-02-10T11:18:18.486 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:18.486 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:18.486 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:18.822 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:18:18.822 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:18.822 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348584/c75fd23b-66ac-47b5-8be8-e13237f757a4/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:19.196 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:18:19.196 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:18:19.197 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0231071 s, 45.4 MB/s 2021-02-10T11:18:19.198 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:19.198 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:19.198 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:19.533 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:18:19.533 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:19.534 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348584/c75fd23b-66ac-47b5-8be8-e13237f757a4/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:19.906 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:18:19.906 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:18:19.906 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0253222 s, 41.4 MB/s 2021-02-10T11:18:19.907 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:19.907 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:19.907 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:20.242 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:18:20.242 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:20.242 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348584/c75fd23b-66ac-47b5-8be8-e13237f757a4/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:20.616 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:18:20.617 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:18:20.617 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0239023 s, 43.9 MB/s 2021-02-10T11:18:20.617 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:20.618 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:20.618 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:20.952 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:18:20.952 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:20.952 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348584/c75fd23b-66ac-47b5-8be8-e13237f757a4/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:21.328 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:18:21.328 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:18:21.328 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0257529 s, 40.7 MB/s 2021-02-10T11:18:21.329 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000348585 2021-02-10T11:18:21.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:21.746 7f7873b3e700 1 Processor -- start 2021-02-10T11:18:21.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:21.746 7f7873b3e700 1 -- start start 2021-02-10T11:18:21.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:21.746 7f7873b3e700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f786c1b0470 0x7f786c1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:21.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:21.746 7f7873b3e700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f786c0f0b50 con 0x7f786c0ef610 2021-02-10T11:18:21.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:21.746 7f7873b3e700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f786c0f24e0 con 0x7f786c0f20e0 2021-02-10T11:18:21.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:21.746 7f7873b3e700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f786c0e5ea0 con 0x7f786c1b0470 2021-02-10T11:18:21.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:21.747 7f787195a700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f786c1b0470 0x7f786c1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:21.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:21.747 7f7871159700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f786c0ef610 0x7f786c0efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:43740/0 (socket says 172.21.2.112:43740) 2021-02-10T11:18:21.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:21.747 7f787215b700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7f786c0f20e0 0x7f786c0f0200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:37298/0 (socket says 172.21.2.112:37298) 2021-02-10T11:18:21.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:21.747 7f7871159700 1 -- 172.21.2.112:0/1253095425 learned_addr learned my addr 172.21.2.112:0/1253095425 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:18:21.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:21.747 7f787195a700 1 -- 172.21.2.112:0/1253095425 >> v1:172.21.2.120:6789/0 conn(0x7f786c0f20e0 legacy=0x7f786c0f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:21.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:21.747 7f787195a700 1 -- 172.21.2.112:0/1253095425 >> v1:172.21.2.112:6789/0 conn(0x7f786c0ef610 legacy=0x7f786c0efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:21.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:21.747 7f7870958700 1 -- 172.21.2.112:0/1253095425 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2793654940 0 0) 0x7f785c0009b0 con 0x7f786c0ef610 2021-02-10T11:18:21.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:21.747 7f787195a700 1 -- 172.21.2.112:0/1253095425 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f786c0e52e0 con 0x7f786c1b0470 2021-02-10T11:18:21.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:21.747 7f787195a700 1 --2- 172.21.2.112:0/1253095425 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f786c1b0470 0x7f786c1b0870 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7f7868006650 tx=0x7f7868009a00).ready entity=mon.0 client_cookie=7f1afdba0da59209 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:18:21.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:21.747 7f7870958700 1 -- 172.21.2.112:0/1253095425 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7868009df0 con 0x7f786c1b0470 2021-02-10T11:18:21.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:21.747 7f7870958700 1 -- 172.21.2.112:0/1253095425 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78680009b0 con 0x7f786c1b0470 2021-02-10T11:18:21.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:21.747 7f7870958700 1 -- 172.21.2.112:0/1253095425 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f786800c2b0 con 0x7f786c1b0470 2021-02-10T11:18:21.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:21.747 7f7873b3e700 1 -- 172.21.2.112:0/1253095425 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f786c1b0470 msgr2=0x7f786c1b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:21.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:21.747 7f7873b3e700 1 --2- 172.21.2.112:0/1253095425 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f786c1b0470 0x7f786c1b0870 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7f7868006650 tx=0x7f7868009a00).stop 2021-02-10T11:18:21.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:21.748 7f7873b3e700 1 -- 172.21.2.112:0/1253095425 shutdown_connections 2021-02-10T11:18:21.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:21.748 7f7873b3e700 1 --2- 172.21.2.112:0/1253095425 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f786c1b0470 0x7f786c1b0870 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:21.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:21.748 7f7873b3e700 1 -- 172.21.2.112:0/1253095425 >> 172.21.2.112:0/1253095425 conn(0x7f786c0ce3e0 msgr2=0x7f786c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:18:21.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:21.748 7f7873b3e700 1 -- 172.21.2.112:0/1253095425 shutdown_connections 2021-02-10T11:18:21.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:21.748 7f7873b3e700 1 -- 172.21.2.112:0/1253095425 wait complete. 2021-02-10T11:18:21.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:21.762 7f7873b3e700 1 Processor -- start 2021-02-10T11:18:21.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:21.763 7f7873b3e700 1 -- start start 2021-02-10T11:18:21.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:21.763 7f7873b3e700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f786c0a1dd0 0x7f786c190920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:21.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:21.763 7f7873b3e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f786c08fd30 0x7f786c0effc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:21.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:21.763 7f7873b3e700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f786c0f0e90 0x7f786c0f1290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:21.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:21.763 7f7873b3e700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f786c0f06f0 con 0x7f786c08fd30 2021-02-10T11:18:21.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:21.763 7f7873b3e700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f786c090160 con 0x7f786c0f0e90 2021-02-10T11:18:21.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:21.763 7f7873b3e700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f786c0f18b0 con 0x7f786c0a1dd0 2021-02-10T11:18:21.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:21.763 7f7871159700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f786c08fd30 0x7f786c0effc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:21.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:21.763 7f787195a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f786c0a1dd0 0x7f786c190920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:21.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:21.763 7f787195a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f786c0a1dd0 0x7f786c190920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3301/0 says I am v2:172.21.2.112:43964/0 (socket says 172.21.2.112:43964) 2021-02-10T11:18:21.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:21.763 7f787195a700 1 -- 172.21.2.112:0/1128240463 learned_addr learned my addr 172.21.2.112:0/1128240463 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:18:21.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:21.763 7f787215b700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f786c0f0e90 0x7f786c0f1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:21.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:21.763 7f7871159700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f786c08fd30 0x7f786c0effc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:44284/0 (socket says 172.21.2.112:44284) 2021-02-10T11:18:21.763 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:21.764 7f7871159700 1 -- 172.21.2.112:0/1128240463 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f786c0a1dd0 msgr2=0x7f786c190920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:21.763 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:21.764 7f7871159700 1 --2- 172.21.2.112:0/1128240463 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f786c0a1dd0 0x7f786c190920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:21.763 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:21.764 7f7871159700 1 -- 172.21.2.112:0/1128240463 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f786c0f0e90 msgr2=0x7f786c0f1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:21.763 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:21.764 7f7871159700 1 --2- 172.21.2.112:0/1128240463 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f786c0f0e90 0x7f786c0f1290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:21.763 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:21.764 7f7871159700 1 -- 172.21.2.112:0/1128240463 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f786c0f1ad0 con 0x7f786c08fd30 2021-02-10T11:18:21.763 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:21.764 7f787195a700 1 --2- 172.21.2.112:0/1128240463 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f786c0a1dd0 0x7f786c190920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:18:21.763 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:21.764 7f787215b700 1 --2- 172.21.2.112:0/1128240463 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f786c0f0e90 0x7f786c0f1290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:18:21.763 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:21.764 7f7871159700 1 --2- 172.21.2.112:0/1128240463 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f786c08fd30 0x7f786c0effc0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 rx=0x7f785c000b80 tx=0x7f785c00a8f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:18:21.763 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:21.764 7f78627fc700 1 -- 172.21.2.112:0/1128240463 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f785c000de0 con 0x7f786c08fd30 2021-02-10T11:18:21.763 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:21.764 7f78627fc700 1 -- 172.21.2.112:0/1128240463 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f785c004140 con 0x7f786c08fd30 2021-02-10T11:18:21.763 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:21.764 7f7873b3e700 1 -- 172.21.2.112:0/1128240463 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f786c0f1ad0 con 0x7f786c08fd30 2021-02-10T11:18:21.763 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:21.764 7f78627fc700 1 -- 172.21.2.112:0/1128240463 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f785c004590 con 0x7f786c08fd30 2021-02-10T11:18:21.763 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:21.764 7f7873b3e700 1 -- 172.21.2.112:0/1128240463 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f786c0f1ad0 con 0x7f786c08fd30 2021-02-10T11:18:21.763 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:21.765 7f78627fc700 1 -- 172.21.2.112:0/1128240463 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f785c004320 con 0x7f786c08fd30 2021-02-10T11:18:21.764 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:21.765 7f78627fc700 1 --2- 172.21.2.112:0/1128240463 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7854018730 0x7f785401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:21.764 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:21.765 7f78627fc700 1 -- 172.21.2.112:0/1128240463 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f785c032580 con 0x7f786c08fd30 2021-02-10T11:18:21.764 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:21.765 7f787195a700 1 --2- 172.21.2.112:0/1128240463 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7854018730 0x7f785401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:21.764 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:21.766 7f787195a700 1 --2- 172.21.2.112:0/1128240463 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7854018730 0x7f785401ac10 crc :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:18:21.793 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:21.796 7f7873b3e700 1 -- 172.21.2.112:0/1128240463 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f786c0e52e0 con 0x7f786c08fd30 2021-02-10T11:18:21.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:21.799 7f78627fc700 1 -- 172.21.2.112:0/1128240463 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f785c004350 con 0x7f786c08fd30 2021-02-10T11:18:21.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:21.924 7f7873b3e700 1 -- 172.21.2.112:0/1128240463 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348585", "target": ["mgr", ""]}) v1 -- 0x7f786c0f23d0 con 0x7f7854018730 2021-02-10T11:18:21.947 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:21.951 7f78627fc700 1 -- 172.21.2.112:0/1128240463 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f7868006390 con 0x7f7854018730 2021-02-10T11:18:21.954 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:21.958 7f7873b3e700 1 -- 172.21.2.112:0/1128240463 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7854018730 msgr2=0x7f785401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:21.954 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:21.958 7f7873b3e700 1 --2- 172.21.2.112:0/1128240463 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7854018730 0x7f785401ac10 crc :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:21.954 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:21.958 7f7873b3e700 1 -- 172.21.2.112:0/1128240463 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f786c08fd30 msgr2=0x7f786c0effc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:21.954 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:21.958 7f7873b3e700 1 --2- 172.21.2.112:0/1128240463 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f786c08fd30 0x7f786c0effc0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 rx=0x7f785c000b80 tx=0x7f785c00a8f0).stop 2021-02-10T11:18:21.954 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:21.958 7f7873b3e700 1 -- 172.21.2.112:0/1128240463 shutdown_connections 2021-02-10T11:18:21.954 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:21.958 7f7873b3e700 1 --2- 172.21.2.112:0/1128240463 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7854018730 0x7f785401ac10 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:21.954 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:21.958 7f7873b3e700 1 --2- 172.21.2.112:0/1128240463 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f786c0f0e90 0x7f786c0f1290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:21.955 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:21.958 7f7873b3e700 1 --2- 172.21.2.112:0/1128240463 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f786c08fd30 0x7f786c0effc0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:21.955 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:21.958 7f7873b3e700 1 --2- 172.21.2.112:0/1128240463 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f786c0a1dd0 0x7f786c190920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:21.955 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:21.958 7f7873b3e700 1 -- 172.21.2.112:0/1128240463 >> 172.21.2.112:0/1128240463 conn(0x7f786c0958b0 msgr2=0x7f786c0bf040 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:18:21.955 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:21.958 7f7873b3e700 1 -- 172.21.2.112:0/1128240463 shutdown_connections 2021-02-10T11:18:21.955 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:21.958 7f7873b3e700 1 -- 172.21.2.112:0/1128240463 wait complete. 2021-02-10T11:18:21.962 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000348585 2021-02-10T11:18:22.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:22.379 7f75d31ac700 1 Processor -- start 2021-02-10T11:18:22.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:22.379 7f75d31ac700 1 -- start start 2021-02-10T11:18:22.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:22.380 7f75d31ac700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f75cc1b0870 0x7f75cc0f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:22.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:22.380 7f75d31ac700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75cc1b0c70 con 0x7f75cc0efe00 2021-02-10T11:18:22.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:22.380 7f75d31ac700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75cc0f0b50 con 0x7f75cc1b0470 2021-02-10T11:18:22.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:22.380 7f75d31ac700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75cc0e5ea0 con 0x7f75cc1b0870 2021-02-10T11:18:22.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:22.380 7f75cbfff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f75cc1b0870 0x7f75cc0f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:22.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:22.380 7f75d17c9700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f75cc0efe00 0x7f75cc0f0200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:43754/0 (socket says 172.21.2.112:43754) 2021-02-10T11:18:22.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:22.380 7f75d17c9700 1 -- 172.21.2.112:0/2208855461 learned_addr learned my addr 172.21.2.112:0/2208855461 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:18:22.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:22.380 7f75cb7fe700 1 -- 172.21.2.112:0/2208855461 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1366841086 0 0) 0x7f75c4000f90 con 0x7f75cc0efe00 2021-02-10T11:18:22.393 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:22.380 7f75cb7fe700 1 -- 172.21.2.112:0/2208855461 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75b8002fe0 con 0x7f75cc0efe00 2021-02-10T11:18:22.393 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:22.380 7f75cbfff700 1 -- 172.21.2.112:0/2208855461 >> v1:172.21.2.120:6789/0 conn(0x7f75cc1b0470 legacy=0x7f75cc0ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:18:22.393 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:22.380 7f75cbfff700 1 -- 172.21.2.112:0/2208855461 >> v1:172.21.2.112:6789/0 conn(0x7f75cc0efe00 legacy=0x7f75cc0f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:22.393 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:22.380 7f75cbfff700 1 -- 172.21.2.112:0/2208855461 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75cc0e52e0 con 0x7f75cc1b0870 2021-02-10T11:18:22.393 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:22.380 7f75cbfff700 1 --2- 172.21.2.112:0/2208855461 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f75cc1b0870 0x7f75cc0f20e0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 rx=0x7f75bc006650 tx=0x7f75bc009a80).ready entity=mon.0 client_cookie=b4d0a8a88aa1c0de server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:18:22.393 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:22.381 7f75cb7fe700 1 -- 172.21.2.112:0/2208855461 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f75bc0009b0 con 0x7f75cc1b0870 2021-02-10T11:18:22.393 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:22.381 7f75cb7fe700 1 -- 172.21.2.112:0/2208855461 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75bc00c0c0 con 0x7f75cc1b0870 2021-02-10T11:18:22.393 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:22.381 7f75cb7fe700 1 -- 172.21.2.112:0/2208855461 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f75bc00c510 con 0x7f75cc1b0870 2021-02-10T11:18:22.393 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:22.381 7f75d31ac700 1 -- 172.21.2.112:0/2208855461 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f75cc1b0870 msgr2=0x7f75cc0f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:22.393 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:22.381 7f75d31ac700 1 --2- 172.21.2.112:0/2208855461 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f75cc1b0870 0x7f75cc0f20e0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 rx=0x7f75bc006650 tx=0x7f75bc009a80).stop 2021-02-10T11:18:22.393 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:22.381 7f75d31ac700 1 -- 172.21.2.112:0/2208855461 shutdown_connections 2021-02-10T11:18:22.393 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:22.381 7f75d31ac700 1 --2- 172.21.2.112:0/2208855461 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f75cc1b0870 0x7f75cc0f20e0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:22.393 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:22.381 7f75d31ac700 1 -- 172.21.2.112:0/2208855461 >> 172.21.2.112:0/2208855461 conn(0x7f75cc0ce3e0 msgr2=0x7f75cc0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:18:22.393 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:22.381 7f75d31ac700 1 -- 172.21.2.112:0/2208855461 shutdown_connections 2021-02-10T11:18:22.394 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:22.381 7f75d31ac700 1 -- 172.21.2.112:0/2208855461 wait complete. 2021-02-10T11:18:22.395 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:22.396 7f75d31ac700 1 Processor -- start 2021-02-10T11:18:22.395 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:22.396 7f75d31ac700 1 -- start start 2021-02-10T11:18:22.395 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:22.396 7f75d31ac700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f75cc0b2840 0x7f75cc1a4300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:22.395 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:22.396 7f75d31ac700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f75cc0b30d0 0x7f75cc0b34d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:22.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:22.396 7f75d31ac700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f75cc0b3af0 0x7f75cc0f0580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:22.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:22.396 7f75d31ac700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75cc0b2c40 con 0x7f75cc0b3af0 2021-02-10T11:18:22.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:22.396 7f75d31ac700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75cc1a4920 con 0x7f75cc0b2840 2021-02-10T11:18:22.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:22.396 7f75d31ac700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75cc0f0ba0 con 0x7f75cc0b30d0 2021-02-10T11:18:22.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:22.397 7f75d17c9700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f75cc0b3af0 0x7f75cc0f0580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:22.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:22.397 7f75d17c9700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f75cc0b3af0 0x7f75cc0f0580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:44296/0 (socket says 172.21.2.112:44296) 2021-02-10T11:18:22.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:22.397 7f75d17c9700 1 -- 172.21.2.112:0/4202805507 learned_addr learned my addr 172.21.2.112:0/4202805507 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:18:22.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:22.397 7f75cbfff700 1 --2- 172.21.2.112:0/4202805507 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f75cc0b30d0 0x7f75cc0b34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:22.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:22.397 7f75d0fc8700 1 --2- 172.21.2.112:0/4202805507 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f75cc0b2840 0x7f75cc1a4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:22.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:22.397 7f75d17c9700 1 -- 172.21.2.112:0/4202805507 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f75cc0b30d0 msgr2=0x7f75cc0b34d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:22.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:22.397 7f75d17c9700 1 --2- 172.21.2.112:0/4202805507 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f75cc0b30d0 0x7f75cc0b34d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:22.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:22.397 7f75d17c9700 1 -- 172.21.2.112:0/4202805507 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f75cc0b2840 msgr2=0x7f75cc1a4300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:22.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:22.397 7f75d17c9700 1 --2- 172.21.2.112:0/4202805507 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f75cc0b2840 0x7f75cc1a4300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:22.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:22.397 7f75d17c9700 1 -- 172.21.2.112:0/4202805507 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75cc0f0dc0 con 0x7f75cc0b3af0 2021-02-10T11:18:22.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:22.397 7f75d0fc8700 1 --2- 172.21.2.112:0/4202805507 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f75cc0b2840 0x7f75cc1a4300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:18:22.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:22.397 7f75d17c9700 1 --2- 172.21.2.112:0/4202805507 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f75cc0b3af0 0x7f75cc0f0580 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7f75c400a940 tx=0x7f75c400ace0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:18:22.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:22.398 7f75c97fa700 1 -- 172.21.2.112:0/4202805507 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f75c40070d0 con 0x7f75cc0b3af0 2021-02-10T11:18:22.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:22.398 7f75c97fa700 1 -- 172.21.2.112:0/4202805507 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75c4007390 con 0x7f75cc0b3af0 2021-02-10T11:18:22.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:22.398 7f75d31ac700 1 -- 172.21.2.112:0/4202805507 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75cc0f0dc0 con 0x7f75cc0b3af0 2021-02-10T11:18:22.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:22.398 7f75d31ac700 1 -- 172.21.2.112:0/4202805507 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75cc0f1350 con 0x7f75cc0b3af0 2021-02-10T11:18:22.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:22.398 7f75c97fa700 1 -- 172.21.2.112:0/4202805507 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f75c4004910 con 0x7f75cc0b3af0 2021-02-10T11:18:22.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:22.398 7f75c97fa700 1 -- 172.21.2.112:0/4202805507 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f75c4004720 con 0x7f75cc0b3af0 2021-02-10T11:18:22.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:22.398 7f75c97fa700 1 --2- 172.21.2.112:0/4202805507 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f75b0018730 0x7f75b001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:22.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:22.398 7f75c97fa700 1 -- 172.21.2.112:0/4202805507 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f75c4032b20 con 0x7f75cc0b3af0 2021-02-10T11:18:22.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:22.399 7f75d0fc8700 1 --2- 172.21.2.112:0/4202805507 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f75b0018730 0x7f75b001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:22.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:22.399 7f75d0fc8700 1 --2- 172.21.2.112:0/4202805507 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f75b0018730 0x7f75b001ac10 crc :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:18:22.425 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:22.429 7f75d31ac700 1 -- 172.21.2.112:0/4202805507 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75cc0f0050 con 0x7f75cc0b3af0 2021-02-10T11:18:22.427 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:22.432 7f75c97fa700 1 -- 172.21.2.112:0/4202805507 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f75c4001270 con 0x7f75cc0b3af0 2021-02-10T11:18:22.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:22.562 7f75d31ac700 1 -- 172.21.2.112:0/4202805507 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348585", "target": ["mgr", ""]}) v1 -- 0x7f75cc0f1350 con 0x7f75b0018730 2021-02-10T11:18:22.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:22.566 7f75c97fa700 1 -- 172.21.2.112:0/4202805507 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f75b4006bc0 con 0x7f75b0018730 2021-02-10T11:18:22.565 INFO:teuthology.orchestra.run.gibba012.stdout:/volumes/_nogroup/subvolume_0000000000348585/a74b537b-ffd4-46f1-bb25-aa0babb27682 2021-02-10T11:18:22.567 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:22.571 7f75d31ac700 1 -- 172.21.2.112:0/4202805507 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f75b0018730 msgr2=0x7f75b001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:22.567 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:22.571 7f75d31ac700 1 --2- 172.21.2.112:0/4202805507 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f75b0018730 0x7f75b001ac10 crc :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:22.567 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:22.571 7f75d31ac700 1 -- 172.21.2.112:0/4202805507 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f75cc0b3af0 msgr2=0x7f75cc0f0580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:22.567 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:22.571 7f75d31ac700 1 --2- 172.21.2.112:0/4202805507 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f75cc0b3af0 0x7f75cc0f0580 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7f75c400a940 tx=0x7f75c400ace0).stop 2021-02-10T11:18:22.568 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:22.571 7f75d31ac700 1 -- 172.21.2.112:0/4202805507 shutdown_connections 2021-02-10T11:18:22.568 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:22.571 7f75d31ac700 1 --2- 172.21.2.112:0/4202805507 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f75b0018730 0x7f75b001ac10 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:22.568 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:22.571 7f75d31ac700 1 --2- 172.21.2.112:0/4202805507 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f75cc0b3af0 0x7f75cc0f0580 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:22.568 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:22.571 7f75d31ac700 1 --2- 172.21.2.112:0/4202805507 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f75cc0b30d0 0x7f75cc0b34d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:22.568 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:22.571 7f75d31ac700 1 --2- 172.21.2.112:0/4202805507 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f75cc0b2840 0x7f75cc1a4300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:22.568 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:22.571 7f75d31ac700 1 -- 172.21.2.112:0/4202805507 >> 172.21.2.112:0/4202805507 conn(0x7f75cc0c36f0 msgr2=0x7f75cc094c30 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:18:22.568 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:22.572 7f75d31ac700 1 -- 172.21.2.112:0/4202805507 shutdown_connections 2021-02-10T11:18:22.568 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:22.572 7f75d31ac700 1 -- 172.21.2.112:0/4202805507 wait complete. 2021-02-10T11:18:22.576 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000348585 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000348585/a74b537b-ffd4-46f1-bb25-aa0babb27682 2021-02-10T11:18:22.576 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:22.576 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:22.576 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:22.912 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:18:22.912 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:22.913 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348585/a74b537b-ffd4-46f1-bb25-aa0babb27682/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:23.291 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:18:23.291 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:18:23.291 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0238857 s, 43.9 MB/s 2021-02-10T11:18:23.292 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:23.292 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:23.292 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:23.626 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:18:23.626 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:23.626 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348585/a74b537b-ffd4-46f1-bb25-aa0babb27682/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:24.001 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:18:24.002 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:18:24.002 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0260041 s, 40.3 MB/s 2021-02-10T11:18:24.003 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:24.003 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:24.003 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:24.337 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:18:24.337 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:24.337 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348585/a74b537b-ffd4-46f1-bb25-aa0babb27682/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:24.711 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:18:24.711 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:18:24.711 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0251304 s, 41.7 MB/s 2021-02-10T11:18:24.712 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:24.712 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:24.713 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:25.048 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:18:25.049 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:25.049 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348585/a74b537b-ffd4-46f1-bb25-aa0babb27682/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:25.424 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:18:25.424 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:18:25.424 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0232992 s, 45.0 MB/s 2021-02-10T11:18:25.425 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:25.425 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:25.425 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:25.760 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:18:25.760 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:25.761 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348585/a74b537b-ffd4-46f1-bb25-aa0babb27682/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:26.136 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:18:26.136 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:18:26.137 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0253051 s, 41.4 MB/s 2021-02-10T11:18:26.137 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:26.138 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:26.138 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:26.473 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:18:26.473 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:26.473 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348585/a74b537b-ffd4-46f1-bb25-aa0babb27682/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:26.850 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:18:26.850 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:18:26.850 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0264041 s, 39.7 MB/s 2021-02-10T11:18:26.851 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:26.851 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:26.852 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:27.186 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:18:27.186 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:27.186 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348585/a74b537b-ffd4-46f1-bb25-aa0babb27682/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:27.562 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:18:27.562 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:18:27.563 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0248665 s, 42.2 MB/s 2021-02-10T11:18:27.563 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:27.564 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:27.564 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:27.897 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:18:27.898 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:27.898 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348585/a74b537b-ffd4-46f1-bb25-aa0babb27682/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:28.271 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:18:28.271 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:18:28.271 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0223636 s, 46.9 MB/s 2021-02-10T11:18:28.272 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:28.272 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:28.272 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:28.608 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:18:28.608 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:28.608 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348585/a74b537b-ffd4-46f1-bb25-aa0babb27682/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:28.983 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:18:28.983 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:18:28.983 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0249768 s, 42.0 MB/s 2021-02-10T11:18:28.984 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:28.984 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:28.984 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:29.318 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:18:29.318 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:29.319 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348585/a74b537b-ffd4-46f1-bb25-aa0babb27682/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:29.692 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:18:29.692 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:18:29.692 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0262406 s, 40.0 MB/s 2021-02-10T11:18:29.693 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000348586 2021-02-10T11:18:30.120 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.108 7f6295615700 1 Processor -- start 2021-02-10T11:18:30.120 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.108 7f6295615700 1 -- start start 2021-02-10T11:18:30.120 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.109 7f6295615700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f62901b0470 0x7f62901b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:30.121 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.109 7f6295615700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f62900f20e0 0x7f62900f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:30.121 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.109 7f6295615700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62900f0b50 con 0x7f62900ef610 2021-02-10T11:18:30.121 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.109 7f6295615700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62900f0820 con 0x7f62901b0470 2021-02-10T11:18:30.121 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.109 7f6295615700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62900f24e0 con 0x7f62900f20e0 2021-02-10T11:18:30.121 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.109 7f628f2bf700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f62901b0470 0x7f62901b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:30.121 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.109 7f628f2bf700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f62901b0470 0x7f62901b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:44304/0 (socket says 172.21.2.112:44304) 2021-02-10T11:18:30.121 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.109 7f628f2bf700 1 -- 172.21.2.112:0/3412662130 learned_addr learned my addr 172.21.2.112:0/3412662130 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:18:30.121 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.109 7f628f2bf700 1 -- 172.21.2.112:0/3412662130 >> v2:172.21.2.120:3300/0 conn(0x7f62900f20e0 msgr2=0x7f62900f0200 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:18:30.121 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.109 7f628f2bf700 1 --2- 172.21.2.112:0/3412662130 >> v2:172.21.2.120:3300/0 conn(0x7f62900f20e0 0x7f62900f0200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:18:30.121 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.109 7f628f2bf700 1 -- 172.21.2.112:0/3412662130 >> v1:172.21.2.120:6789/0 conn(0x7f62900ef610 legacy=0x7f62900efa10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:18:30.121 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.109 7f628f2bf700 1 -- 172.21.2.112:0/3412662130 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62900e52e0 con 0x7f62901b0470 2021-02-10T11:18:30.121 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.109 7f628f2bf700 1 --2- 172.21.2.112:0/3412662130 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f62901b0470 0x7f62901b0870 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 rx=0x7f6280006700 tx=0x7f6280009b80).ready entity=mon.0 client_cookie=7583ac10d91e30b3 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:18:30.122 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.109 7f628e2bd700 1 -- 172.21.2.112:0/3412662130 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f628000c0c0 con 0x7f62901b0470 2021-02-10T11:18:30.122 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.110 7f628e2bd700 1 -- 172.21.2.112:0/3412662130 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f628000c440 con 0x7f62901b0470 2021-02-10T11:18:30.122 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.110 7f628e2bd700 1 -- 172.21.2.112:0/3412662130 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f628000c920 con 0x7f62901b0470 2021-02-10T11:18:30.122 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.110 7f6295615700 1 -- 172.21.2.112:0/3412662130 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f62901b0470 msgr2=0x7f62901b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:30.122 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.110 7f6295615700 1 --2- 172.21.2.112:0/3412662130 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f62901b0470 0x7f62901b0870 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 rx=0x7f6280006700 tx=0x7f6280009b80).stop 2021-02-10T11:18:30.122 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.110 7f6295615700 1 -- 172.21.2.112:0/3412662130 shutdown_connections 2021-02-10T11:18:30.122 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.110 7f6295615700 1 --2- 172.21.2.112:0/3412662130 >> v2:172.21.2.120:3300/0 conn(0x7f62900f20e0 0x7f62900f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:18:30.122 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.110 7f6295615700 1 --2- 172.21.2.112:0/3412662130 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f62901b0470 0x7f62901b0870 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:30.122 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.110 7f6295615700 1 -- 172.21.2.112:0/3412662130 >> 172.21.2.112:0/3412662130 conn(0x7f62900ce3e0 msgr2=0x7f62900e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:18:30.122 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.110 7f6295615700 1 -- 172.21.2.112:0/3412662130 shutdown_connections 2021-02-10T11:18:30.122 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.110 7f6295615700 1 -- 172.21.2.112:0/3412662130 wait complete. 2021-02-10T11:18:30.124 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.124 7f6295615700 1 Processor -- start 2021-02-10T11:18:30.124 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.125 7f6295615700 1 -- start start 2021-02-10T11:18:30.124 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.125 7f6295615700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f62900b2840 0x7f62901a4300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:30.124 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.125 7f6295615700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f62900b30d0 0x7f62900b34d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:30.124 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.125 7f6295615700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f62900b3af0 0x7f62900f0580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:30.124 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.125 7f6295615700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62900b2c40 con 0x7f62900b30d0 2021-02-10T11:18:30.124 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.125 7f6295615700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62901a4920 con 0x7f62900b3af0 2021-02-10T11:18:30.124 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.125 7f6295615700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62900f0ba0 con 0x7f62900b2840 2021-02-10T11:18:30.124 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.125 7f628eabe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f62900b30d0 0x7f62900b34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:30.124 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.125 7f628eabe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f62900b30d0 0x7f62900b34d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:44308/0 (socket says 172.21.2.112:44308) 2021-02-10T11:18:30.124 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.125 7f628eabe700 1 -- 172.21.2.112:0/2178885023 learned_addr learned my addr 172.21.2.112:0/2178885023 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:18:30.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.126 7f628eabe700 1 -- 172.21.2.112:0/2178885023 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f62900b2840 msgr2=0x7f62901a4300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:30.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.126 7f628eabe700 1 --2- 172.21.2.112:0/2178885023 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f62900b2840 0x7f62901a4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:18:30.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.126 7f628eabe700 1 -- 172.21.2.112:0/2178885023 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f62900b3af0 msgr2=0x7f62900f0580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:30.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.126 7f628eabe700 1 --2- 172.21.2.112:0/2178885023 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f62900b3af0 0x7f62900f0580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:18:30.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.126 7f628eabe700 1 -- 172.21.2.112:0/2178885023 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62900f0dc0 con 0x7f62900b30d0 2021-02-10T11:18:30.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.126 7f628eabe700 1 --2- 172.21.2.112:0/2178885023 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f62900b30d0 0x7f62900b34d0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7f627800b5b0 tx=0x7f627800b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:18:30.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.126 7f6287fff700 1 -- 172.21.2.112:0/2178885023 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f627800bd40 con 0x7f62900b30d0 2021-02-10T11:18:30.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.126 7f6287fff700 1 -- 172.21.2.112:0/2178885023 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6278006240 con 0x7f62900b30d0 2021-02-10T11:18:30.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.126 7f6295615700 1 -- 172.21.2.112:0/2178885023 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62900f0dc0 con 0x7f62900b30d0 2021-02-10T11:18:30.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.126 7f6295615700 1 -- 172.21.2.112:0/2178885023 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62900f1350 con 0x7f62900b30d0 2021-02-10T11:18:30.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.126 7f6287fff700 1 -- 172.21.2.112:0/2178885023 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6278005720 con 0x7f62900b30d0 2021-02-10T11:18:30.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.127 7f6287fff700 1 -- 172.21.2.112:0/2178885023 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f62780054a0 con 0x7f62900b30d0 2021-02-10T11:18:30.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.127 7f6287fff700 1 --2- 172.21.2.112:0/2178885023 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f62740186c0 0x7f627401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:30.126 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.127 7f6287fff700 1 -- 172.21.2.112:0/2178885023 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f6278032590 con 0x7f62900b30d0 2021-02-10T11:18:30.126 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.127 7f628f2bf700 1 --2- 172.21.2.112:0/2178885023 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f62740186c0 0x7f627401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:30.126 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.128 7f628f2bf700 1 --2- 172.21.2.112:0/2178885023 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f62740186c0 0x7f627401abc0 crc :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:18:30.155 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.159 7f6295615700 1 -- 172.21.2.112:0/2178885023 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62900f18f0 con 0x7f62900b30d0 2021-02-10T11:18:30.157 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.161 7f6287fff700 1 -- 172.21.2.112:0/2178885023 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f62780009b0 con 0x7f62900b30d0 2021-02-10T11:18:30.284 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.288 7f6295615700 1 -- 172.21.2.112:0/2178885023 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348586", "target": ["mgr", ""]}) v1 -- 0x7f62900f1350 con 0x7f62740186c0 2021-02-10T11:18:30.311 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.315 7f6287fff700 1 -- 172.21.2.112:0/2178885023 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f6280000990 con 0x7f62740186c0 2021-02-10T11:18:30.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.321 7f6295615700 1 -- 172.21.2.112:0/2178885023 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f62740186c0 msgr2=0x7f627401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:30.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.321 7f6295615700 1 --2- 172.21.2.112:0/2178885023 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f62740186c0 0x7f627401abc0 crc :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:30.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.321 7f6295615700 1 -- 172.21.2.112:0/2178885023 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f62900b30d0 msgr2=0x7f62900b34d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:30.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.321 7f6295615700 1 --2- 172.21.2.112:0/2178885023 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f62900b30d0 0x7f62900b34d0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7f627800b5b0 tx=0x7f627800b950).stop 2021-02-10T11:18:30.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.321 7f6295615700 1 -- 172.21.2.112:0/2178885023 shutdown_connections 2021-02-10T11:18:30.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.321 7f6295615700 1 --2- 172.21.2.112:0/2178885023 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f62740186c0 0x7f627401abc0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:30.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.321 7f6295615700 1 --2- 172.21.2.112:0/2178885023 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f62900b3af0 0x7f62900f0580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:18:30.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.321 7f6295615700 1 --2- 172.21.2.112:0/2178885023 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f62900b30d0 0x7f62900b34d0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:30.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.321 7f6295615700 1 --2- 172.21.2.112:0/2178885023 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f62900b2840 0x7f62901a4300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:18:30.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.321 7f6295615700 1 -- 172.21.2.112:0/2178885023 >> 172.21.2.112:0/2178885023 conn(0x7f62900c36f0 msgr2=0x7f6290094c30 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:18:30.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.321 7f6295615700 1 -- 172.21.2.112:0/2178885023 shutdown_connections 2021-02-10T11:18:30.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.321 7f6295615700 1 -- 172.21.2.112:0/2178885023 wait complete. 2021-02-10T11:18:30.325 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000348586 2021-02-10T11:18:30.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.743 7fae5b20a700 1 Processor -- start 2021-02-10T11:18:30.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.743 7fae5b20a700 1 -- start start 2021-02-10T11:18:30.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.743 7fae5b20a700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fae541b0470 0x7fae541b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:30.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.743 7fae5b20a700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fae540ef610 0x7fae540efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:30.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.743 7fae5b20a700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae540f0b50 con 0x7fae540f0030 2021-02-10T11:18:30.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.743 7fae5b20a700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae540e52e0 con 0x7fae540ef610 2021-02-10T11:18:30.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.743 7fae5b20a700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae540e5490 con 0x7fae541b0470 2021-02-10T11:18:30.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.744 7fae58825700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fae540ef610 0x7fae540efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:30.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.744 7fae58825700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fae540ef610 0x7fae540efa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:44314/0 (socket says 172.21.2.112:44314) 2021-02-10T11:18:30.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.744 7fae58825700 1 -- 172.21.2.112:0/41296137 learned_addr learned my addr 172.21.2.112:0/41296137 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:18:30.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.744 7fae59026700 1 --2- 172.21.2.112:0/41296137 >> v2:172.21.2.120:3300/0 conn(0x7fae541b0470 0x7fae541b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:30.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.744 7fae58825700 1 -- 172.21.2.112:0/41296137 >> v2:172.21.2.120:3300/0 conn(0x7fae541b0470 msgr2=0x7fae541b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:18:30.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.744 7fae58825700 1 --2- 172.21.2.112:0/41296137 >> v2:172.21.2.120:3300/0 conn(0x7fae541b0470 0x7fae541b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:30.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.744 7fae58825700 1 -- 172.21.2.112:0/41296137 >> v1:172.21.2.120:6789/0 conn(0x7fae540f0030 legacy=0x7fae540f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:18:30.757 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.744 7fae58825700 1 -- 172.21.2.112:0/41296137 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae540f0430 con 0x7fae540ef610 2021-02-10T11:18:30.757 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.744 7fae58825700 1 --2- 172.21.2.112:0/41296137 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fae540ef610 0x7fae540efa10 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 rx=0x7fae50006700 tx=0x7fae50009b80).ready entity=mon.0 client_cookie=30d4445ebc58d433 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:18:30.757 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.744 7fae47fff700 1 -- 172.21.2.112:0/41296137 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fae5000c0c0 con 0x7fae540ef610 2021-02-10T11:18:30.757 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.744 7fae47fff700 1 -- 172.21.2.112:0/41296137 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae5000c440 con 0x7fae540ef610 2021-02-10T11:18:30.757 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.744 7fae47fff700 1 -- 172.21.2.112:0/41296137 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fae5000c920 con 0x7fae540ef610 2021-02-10T11:18:30.757 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.744 7fae5b20a700 1 -- 172.21.2.112:0/41296137 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fae540ef610 msgr2=0x7fae540efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:30.757 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.744 7fae5b20a700 1 --2- 172.21.2.112:0/41296137 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fae540ef610 0x7fae540efa10 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 rx=0x7fae50006700 tx=0x7fae50009b80).stop 2021-02-10T11:18:30.757 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.745 7fae5b20a700 1 -- 172.21.2.112:0/41296137 shutdown_connections 2021-02-10T11:18:30.757 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.745 7fae5b20a700 1 --2- 172.21.2.112:0/41296137 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fae540ef610 0x7fae540efa10 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:30.757 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.745 7fae5b20a700 1 --2- 172.21.2.112:0/41296137 >> v2:172.21.2.120:3300/0 conn(0x7fae541b0470 0x7fae541b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:30.757 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.745 7fae5b20a700 1 -- 172.21.2.112:0/41296137 >> 172.21.2.112:0/41296137 conn(0x7fae540ce3e0 msgr2=0x7fae540e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:18:30.757 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.745 7fae5b20a700 1 -- 172.21.2.112:0/41296137 shutdown_connections 2021-02-10T11:18:30.757 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.746 7fae5b20a700 1 -- 172.21.2.112:0/41296137 wait complete. 2021-02-10T11:18:30.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.760 7fae5b20a700 1 Processor -- start 2021-02-10T11:18:30.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.760 7fae5b20a700 1 -- start start 2021-02-10T11:18:30.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.760 7fae5b20a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fae54190920 0x7fae540b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:30.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.760 7fae5b20a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fae540b8200 0x7fae540b8600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:30.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.760 7fae5b20a700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fae540c1d80 0x7fae540c2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:30.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.760 7fae5b20a700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae54190da0 con 0x7fae54190920 2021-02-10T11:18:30.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.760 7fae5b20a700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae540b8c20 con 0x7fae540c1d80 2021-02-10T11:18:30.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.760 7fae5b20a700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae540b8e10 con 0x7fae540b8200 2021-02-10T11:18:30.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.761 7fae59026700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fae54190920 0x7fae540b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:30.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.761 7fae59026700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fae54190920 0x7fae540b7be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:44322/0 (socket says 172.21.2.112:44322) 2021-02-10T11:18:30.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.761 7fae59026700 1 -- 172.21.2.112:0/269012330 learned_addr learned my addr 172.21.2.112:0/269012330 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:18:30.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.761 7fae59827700 1 --2- 172.21.2.112:0/269012330 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fae540c1d80 0x7fae540c2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:30.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.761 7fae58825700 1 --2- 172.21.2.112:0/269012330 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fae540b8200 0x7fae540b8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:30.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.761 7fae59026700 1 -- 172.21.2.112:0/269012330 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fae540b8200 msgr2=0x7fae540b8600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:30.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.761 7fae59026700 1 --2- 172.21.2.112:0/269012330 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fae540b8200 0x7fae540b8600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:30.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.761 7fae59026700 1 -- 172.21.2.112:0/269012330 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fae540c1d80 msgr2=0x7fae540c2180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:30.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.761 7fae59026700 1 --2- 172.21.2.112:0/269012330 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fae540c1d80 0x7fae540c2180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:30.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.761 7fae59026700 1 -- 172.21.2.112:0/269012330 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae540c27d0 con 0x7fae54190920 2021-02-10T11:18:30.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.761 7fae59026700 1 --2- 172.21.2.112:0/269012330 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fae54190920 0x7fae540b7be0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7fae480099e0 tx=0x7fae4800bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:18:30.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.761 7fae45ffb700 1 -- 172.21.2.112:0/269012330 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fae4800c2f0 con 0x7fae54190920 2021-02-10T11:18:30.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.761 7fae45ffb700 1 -- 172.21.2.112:0/269012330 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae4800c630 con 0x7fae54190920 2021-02-10T11:18:30.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.761 7fae45ffb700 1 -- 172.21.2.112:0/269012330 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fae4800cb10 con 0x7fae54190920 2021-02-10T11:18:30.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.761 7fae5b20a700 1 -- 172.21.2.112:0/269012330 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae540c27d0 con 0x7fae54190920 2021-02-10T11:18:30.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.761 7fae5b20a700 1 -- 172.21.2.112:0/269012330 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae540c27d0 con 0x7fae54190920 2021-02-10T11:18:30.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.762 7fae45ffb700 1 -- 172.21.2.112:0/269012330 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fae48005660 con 0x7fae54190920 2021-02-10T11:18:30.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.762 7fae45ffb700 1 --2- 172.21.2.112:0/269012330 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fae340186d0 0x7fae3401abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:30.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.762 7fae45ffb700 1 -- 172.21.2.112:0/269012330 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fae48032980 con 0x7fae54190920 2021-02-10T11:18:30.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.762 7fae58825700 1 --2- 172.21.2.112:0/269012330 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fae340186d0 0x7fae3401abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:30.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.763 7fae58825700 1 --2- 172.21.2.112:0/269012330 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fae340186d0 0x7fae3401abd0 crc :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:18:30.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.793 7fae5b20a700 1 -- 172.21.2.112:0/269012330 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae54105ae0 con 0x7fae54190920 2021-02-10T11:18:30.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.796 7fae45ffb700 1 -- 172.21.2.112:0/269012330 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fae4800c2f0 con 0x7fae54190920 2021-02-10T11:18:30.921 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.925 7fae5b20a700 1 -- 172.21.2.112:0/269012330 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348586", "target": ["mgr", ""]}) v1 -- 0x7fae540f0b50 con 0x7fae340186d0 2021-02-10T11:18:30.925 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.930 7fae45ffb700 1 -- 172.21.2.112:0/269012330 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7fae500009b0 con 0x7fae340186d0 2021-02-10T11:18:30.929 INFO:teuthology.orchestra.run.gibba012.stdout:/volumes/_nogroup/subvolume_0000000000348586/a822214e-5576-466d-9561-db084764b343 2021-02-10T11:18:30.930 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.934 7fae5b20a700 1 -- 172.21.2.112:0/269012330 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fae340186d0 msgr2=0x7fae3401abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:30.930 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.934 7fae5b20a700 1 --2- 172.21.2.112:0/269012330 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fae340186d0 0x7fae3401abd0 crc :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:30.930 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.934 7fae5b20a700 1 -- 172.21.2.112:0/269012330 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fae54190920 msgr2=0x7fae540b7be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:30.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.934 7fae5b20a700 1 --2- 172.21.2.112:0/269012330 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fae54190920 0x7fae540b7be0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7fae480099e0 tx=0x7fae4800bd40).stop 2021-02-10T11:18:30.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.935 7fae5b20a700 1 -- 172.21.2.112:0/269012330 shutdown_connections 2021-02-10T11:18:30.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.935 7fae5b20a700 1 --2- 172.21.2.112:0/269012330 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fae340186d0 0x7fae3401abd0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:30.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.935 7fae5b20a700 1 --2- 172.21.2.112:0/269012330 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fae540c1d80 0x7fae540c2180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:30.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.935 7fae5b20a700 1 --2- 172.21.2.112:0/269012330 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fae540b8200 0x7fae540b8600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:30.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.935 7fae5b20a700 1 --2- 172.21.2.112:0/269012330 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fae54190920 0x7fae540b7be0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:30.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.935 7fae5b20a700 1 -- 172.21.2.112:0/269012330 >> 172.21.2.112:0/269012330 conn(0x7fae54094e10 msgr2=0x7fae540a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:18:30.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.935 7fae5b20a700 1 -- 172.21.2.112:0/269012330 shutdown_connections 2021-02-10T11:18:30.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:30.935 7fae5b20a700 1 -- 172.21.2.112:0/269012330 wait complete. 2021-02-10T11:18:30.939 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000348586 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000348586/a822214e-5576-466d-9561-db084764b343 2021-02-10T11:18:30.939 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:30.939 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:30.939 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:31.274 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:18:31.274 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:31.274 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348586/a822214e-5576-466d-9561-db084764b343/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:31.652 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:18:31.652 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:18:31.652 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.022445 s, 46.7 MB/s 2021-02-10T11:18:31.653 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:31.653 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:31.653 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:31.989 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:18:32.074 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:32.074 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348586/a822214e-5576-466d-9561-db084764b343/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:32.451 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:18:32.451 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:18:32.452 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0258164 s, 40.6 MB/s 2021-02-10T11:18:32.452 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:32.453 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:32.453 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:32.787 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:18:32.787 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:32.787 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348586/a822214e-5576-466d-9561-db084764b343/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:33.163 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:18:33.163 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:18:33.163 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0258448 s, 40.6 MB/s 2021-02-10T11:18:33.164 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:33.164 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:33.164 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:33.499 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:18:33.499 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:33.499 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348586/a822214e-5576-466d-9561-db084764b343/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:33.876 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:18:33.876 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:18:33.877 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0263293 s, 39.8 MB/s 2021-02-10T11:18:33.877 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:33.878 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:33.878 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:34.212 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:18:34.212 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:34.212 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348586/a822214e-5576-466d-9561-db084764b343/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:34.467 DEBUG:teuthology.orchestra.run.gibba012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:18:34.471 DEBUG:teuthology.orchestra.run.gibba020:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:18:34.585 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:18:34.585 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:18:34.585 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0256395 s, 40.9 MB/s 2021-02-10T11:18:34.586 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:34.586 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:34.586 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:34.920 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:18:34.920 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:34.921 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348586/a822214e-5576-466d-9561-db084764b343/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:35.296 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:18:35.296 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:18:35.296 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0247299 s, 42.4 MB/s 2021-02-10T11:18:35.297 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:35.297 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:35.297 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:35.631 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:18:35.632 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:35.632 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348586/a822214e-5576-466d-9561-db084764b343/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:36.004 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:18:36.004 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:18:36.004 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0233905 s, 44.8 MB/s 2021-02-10T11:18:36.005 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:36.005 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:36.005 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:36.340 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:18:36.340 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:36.340 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348586/a822214e-5576-466d-9561-db084764b343/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:36.718 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:18:36.718 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:18:36.718 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0270752 s, 38.7 MB/s 2021-02-10T11:18:36.720 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:36.720 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:36.720 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:37.055 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:18:37.055 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:37.055 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348586/a822214e-5576-466d-9561-db084764b343/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:37.431 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:18:37.431 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:18:37.431 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0243895 s, 43.0 MB/s 2021-02-10T11:18:37.432 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:37.432 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:37.432 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:37.770 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:18:37.770 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:37.770 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348586/a822214e-5576-466d-9561-db084764b343/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:38.147 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:18:38.147 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:18:38.147 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0235887 s, 44.5 MB/s 2021-02-10T11:18:38.148 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000348587 2021-02-10T11:18:38.583 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:38.571 7fb01ea63700 1 Processor -- start 2021-02-10T11:18:38.584 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:38.571 7fb01ea63700 1 -- start start 2021-02-10T11:18:38.584 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:38.572 7fb01ea63700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fb0181b0870 0x7fb0180f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:38.584 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:38.572 7fb01ea63700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb0180efe00 0x7fb0180f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:38.584 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:38.572 7fb01ea63700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0181b0c70 con 0x7fb0181b0470 2021-02-10T11:18:38.584 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:38.572 7fb01ea63700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0180f0b50 con 0x7fb0181b0870 2021-02-10T11:18:38.584 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:38.572 7fb01ea63700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0180f0820 con 0x7fb0180efe00 2021-02-10T11:18:38.584 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:38.572 7fb017fff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fb0181b0870 0x7fb0180f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:38.584 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:38.572 7fb01c87f700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fb0181b0470 0x7fb0180ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:43790/0 (socket says 172.21.2.112:43790) 2021-02-10T11:18:38.584 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:38.572 7fb01c87f700 1 -- 172.21.2.112:0/3721479429 learned_addr learned my addr 172.21.2.112:0/3721479429 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:18:38.584 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:38.572 7fb017fff700 1 -- 172.21.2.112:0/3721479429 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb0180efe00 msgr2=0x7fb0180f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:18:38.584 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:38.572 7fb017fff700 1 --2- 172.21.2.112:0/3721479429 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb0180efe00 0x7fb0180f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:18:38.584 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:38.572 7fb017fff700 1 -- 172.21.2.112:0/3721479429 >> v1:172.21.2.112:6789/0 conn(0x7fb0181b0470 legacy=0x7fb0180ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:38.584 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:38.572 7fb017fff700 1 -- 172.21.2.112:0/3721479429 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0180e52e0 con 0x7fb0181b0870 2021-02-10T11:18:38.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:38.572 7fb017fff700 1 --2- 172.21.2.112:0/3721479429 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb0181b0870 0x7fb0180f20e0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7fb008006650 tx=0x7fb008009a00).ready entity=mon.0 client_cookie=f4d237aa5744240c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:18:38.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:38.573 7fb0177fe700 1 -- 172.21.2.112:0/3721479429 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb008009df0 con 0x7fb0181b0870 2021-02-10T11:18:38.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:38.573 7fb0177fe700 1 -- 172.21.2.112:0/3721479429 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0080009b0 con 0x7fb0181b0870 2021-02-10T11:18:38.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:38.573 7fb0177fe700 1 -- 172.21.2.112:0/3721479429 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb00800c2b0 con 0x7fb0181b0870 2021-02-10T11:18:38.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:38.573 7fb01ea63700 1 -- 172.21.2.112:0/3721479429 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb0181b0870 msgr2=0x7fb0180f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:38.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:38.573 7fb01ea63700 1 --2- 172.21.2.112:0/3721479429 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb0181b0870 0x7fb0180f20e0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7fb008006650 tx=0x7fb008009a00).stop 2021-02-10T11:18:38.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:38.573 7fb01ea63700 1 -- 172.21.2.112:0/3721479429 shutdown_connections 2021-02-10T11:18:38.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:38.573 7fb01ea63700 1 --2- 172.21.2.112:0/3721479429 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb0180efe00 0x7fb0180f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:18:38.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:38.573 7fb01ea63700 1 --2- 172.21.2.112:0/3721479429 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb0181b0870 0x7fb0180f20e0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:38.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:38.573 7fb01ea63700 1 -- 172.21.2.112:0/3721479429 >> 172.21.2.112:0/3721479429 conn(0x7fb0180ce3e0 msgr2=0x7fb0180e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:18:38.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:38.573 7fb01ea63700 1 -- 172.21.2.112:0/3721479429 shutdown_connections 2021-02-10T11:18:38.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:38.573 7fb01ea63700 1 -- 172.21.2.112:0/3721479429 wait complete. 2021-02-10T11:18:38.587 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:38.588 7fb01ea63700 1 Processor -- start 2021-02-10T11:18:38.587 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:38.588 7fb01ea63700 1 -- start start 2021-02-10T11:18:38.588 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:38.588 7fb01ea63700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb018094fb0 0x7fb01809e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:38.588 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:38.588 7fb01ea63700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb0180c2be0 0x7fb0180c2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:38.588 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:38.588 7fb01ea63700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb0180c3600 0x7fb0180a9c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:38.588 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:38.588 7fb01ea63700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb01809ebf0 con 0x7fb0180c2be0 2021-02-10T11:18:38.588 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:38.588 7fb01ea63700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0180aa2a0 con 0x7fb018094fb0 2021-02-10T11:18:38.588 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:38.588 7fb01ea63700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0180aa4c0 con 0x7fb0180c3600 2021-02-10T11:18:38.588 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:38.589 7fb017fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb0180c2be0 0x7fb0180c2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:38.588 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:38.589 7fb017fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb0180c2be0 0x7fb0180c2fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:44338/0 (socket says 172.21.2.112:44338) 2021-02-10T11:18:38.588 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:38.589 7fb017fff700 1 -- 172.21.2.112:0/3559783683 learned_addr learned my addr 172.21.2.112:0/3559783683 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:18:38.588 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:38.589 7fb017fff700 1 -- 172.21.2.112:0/3559783683 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb0180c3600 msgr2=0x7fb0180a9c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:38.589 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:38.589 7fb017fff700 1 --2- 172.21.2.112:0/3559783683 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb0180c3600 0x7fb0180a9c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:18:38.589 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:38.589 7fb017fff700 1 -- 172.21.2.112:0/3559783683 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb018094fb0 msgr2=0x7fb01809e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:38.589 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:38.589 7fb017fff700 1 --2- 172.21.2.112:0/3559783683 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb018094fb0 0x7fb01809e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:18:38.589 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:38.589 7fb017fff700 1 -- 172.21.2.112:0/3559783683 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0180aa6e0 con 0x7fb0180c2be0 2021-02-10T11:18:38.589 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:38.589 7fb017fff700 1 --2- 172.21.2.112:0/3559783683 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb0180c2be0 0x7fb0180c2fe0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 rx=0x7fb0080096a0 tx=0x7fb00800b9b0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:18:38.589 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:38.589 7fb0157fa700 1 -- 172.21.2.112:0/3559783683 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb008000940 con 0x7fb0180c2be0 2021-02-10T11:18:38.589 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:38.589 7fb0157fa700 1 -- 172.21.2.112:0/3559783683 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0080093a0 con 0x7fb0180c2be0 2021-02-10T11:18:38.589 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:38.589 7fb0157fa700 1 -- 172.21.2.112:0/3559783683 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb008002220 con 0x7fb0180c2be0 2021-02-10T11:18:38.589 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:38.589 7fb01ea63700 1 -- 172.21.2.112:0/3559783683 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0180aa6e0 con 0x7fb0180c2be0 2021-02-10T11:18:38.589 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:38.590 7fb01ea63700 1 -- 172.21.2.112:0/3559783683 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0180aa6e0 con 0x7fb0180c2be0 2021-02-10T11:18:38.589 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:38.590 7fb0157fa700 1 -- 172.21.2.112:0/3559783683 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb008002020 con 0x7fb0180c2be0 2021-02-10T11:18:38.590 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:38.590 7fb0157fa700 1 --2- 172.21.2.112:0/3559783683 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7faffc0186c0 0x7faffc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:38.590 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:38.590 7fb0157fa700 1 -- 172.21.2.112:0/3559783683 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fb0080327e0 con 0x7fb0180c2be0 2021-02-10T11:18:38.590 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:38.591 7fb01c87f700 1 --2- 172.21.2.112:0/3559783683 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7faffc0186c0 0x7faffc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:38.590 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:38.592 7fb01c87f700 1 --2- 172.21.2.112:0/3559783683 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7faffc0186c0 0x7faffc01abc0 crc :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:18:38.617 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:38.621 7fb01ea63700 1 -- 172.21.2.112:0/3559783683 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0180edd10 con 0x7fb0180c2be0 2021-02-10T11:18:38.620 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:38.624 7fb0157fa700 1 -- 172.21.2.112:0/3559783683 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb008000c90 con 0x7fb0180c2be0 2021-02-10T11:18:38.747 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:38.751 7fb01ea63700 1 -- 172.21.2.112:0/3559783683 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348587", "target": ["mgr", ""]}) v1 -- 0x7fb01809ebf0 con 0x7faffc0186c0 2021-02-10T11:18:38.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:38.777 7fb0157fa700 1 -- 172.21.2.112:0/3559783683 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fb0000077d0 con 0x7faffc0186c0 2021-02-10T11:18:38.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:38.784 7fb01ea63700 1 -- 172.21.2.112:0/3559783683 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7faffc0186c0 msgr2=0x7faffc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:38.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:38.784 7fb01ea63700 1 --2- 172.21.2.112:0/3559783683 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7faffc0186c0 0x7faffc01abc0 crc :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:38.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:38.784 7fb01ea63700 1 -- 172.21.2.112:0/3559783683 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb0180c2be0 msgr2=0x7fb0180c2fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:38.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:38.784 7fb01ea63700 1 --2- 172.21.2.112:0/3559783683 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb0180c2be0 0x7fb0180c2fe0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 rx=0x7fb0080096a0 tx=0x7fb00800b9b0).stop 2021-02-10T11:18:38.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:38.784 7fb01ea63700 1 -- 172.21.2.112:0/3559783683 shutdown_connections 2021-02-10T11:18:38.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:38.784 7fb01ea63700 1 --2- 172.21.2.112:0/3559783683 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7faffc0186c0 0x7faffc01abc0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:38.782 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:38.784 7fb01ea63700 1 --2- 172.21.2.112:0/3559783683 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb0180c3600 0x7fb0180a9c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:18:38.782 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:38.784 7fb01ea63700 1 --2- 172.21.2.112:0/3559783683 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb0180c2be0 0x7fb0180c2fe0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:38.782 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:38.784 7fb01ea63700 1 --2- 172.21.2.112:0/3559783683 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb018094fb0 0x7fb01809e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:18:38.782 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:38.784 7fb01ea63700 1 -- 172.21.2.112:0/3559783683 >> 172.21.2.112:0/3559783683 conn(0x7fb01809f630 msgr2=0x7fb0180a0130 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:18:38.782 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:38.784 7fb01ea63700 1 -- 172.21.2.112:0/3559783683 shutdown_connections 2021-02-10T11:18:38.782 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:38.784 7fb01ea63700 1 -- 172.21.2.112:0/3559783683 wait complete. 2021-02-10T11:18:38.789 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000348587 2021-02-10T11:18:39.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:39.209 7fdcbe6e2700 1 Processor -- start 2021-02-10T11:18:39.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:39.209 7fdcbe6e2700 1 -- start start 2021-02-10T11:18:39.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:39.209 7fdcbe6e2700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fdcb81b0470 0x7fdcb81b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:39.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:39.209 7fdcbe6e2700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fdcb80ef610 0x7fdcb80efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:39.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:39.209 7fdcbe6e2700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdcb80f0030 0x7fdcb80f0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:39.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:39.209 7fdcbe6e2700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdcb80e5ea0 con 0x7fdcb81b0470 2021-02-10T11:18:39.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:39.209 7fdcbe6e2700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdcb80f0b50 con 0x7fdcb80ef610 2021-02-10T11:18:39.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:39.209 7fdcbe6e2700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcb80e52e0 con 0x7fdcb80f0030 2021-02-10T11:18:39.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:39.209 7fdcb7fff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fdcb81b0470 0x7fdcb81b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:39.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:39.209 7fdcb7fff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fdcb81b0470 0x7fdcb81b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:44342/0 (socket says 172.21.2.112:44342) 2021-02-10T11:18:39.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:39.209 7fdcb7fff700 1 -- 172.21.2.112:0/3387631402 learned_addr learned my addr 172.21.2.112:0/3387631402 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:18:39.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:39.209 7fdcb77fe700 1 --2- 172.21.2.112:0/3387631402 >> v2:172.21.2.120:3300/0 conn(0x7fdcb80ef610 0x7fdcb80efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:39.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:39.209 7fdcbccff700 1 --2- 172.21.2.112:0/3387631402 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdcb80f0030 0x7fdcb80f0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:39.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:39.210 7fdcb7fff700 1 -- 172.21.2.112:0/3387631402 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdcb80f0030 msgr2=0x7fdcb80f0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:18:39.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:39.210 7fdcb7fff700 1 --2- 172.21.2.112:0/3387631402 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdcb80f0030 0x7fdcb80f0430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:39.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:39.210 7fdcb7fff700 1 -- 172.21.2.112:0/3387631402 >> v2:172.21.2.120:3300/0 conn(0x7fdcb80ef610 msgr2=0x7fdcb80efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:18:39.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:39.210 7fdcb7fff700 1 --2- 172.21.2.112:0/3387631402 >> v2:172.21.2.120:3300/0 conn(0x7fdcb80ef610 0x7fdcb80efa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:39.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:39.210 7fdcb7fff700 1 -- 172.21.2.112:0/3387631402 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcb80f10d0 con 0x7fdcb81b0470 2021-02-10T11:18:39.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:39.210 7fdcbccff700 1 --2- 172.21.2.112:0/3387631402 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdcb80f0030 0x7fdcb80f0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:18:39.223 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:39.210 7fdcb7fff700 1 --2- 172.21.2.112:0/3387631402 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdcb81b0470 0x7fdcb81b0870 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7fdcb0006a80 tx=0x7fdcb0009f00).ready entity=mon.0 client_cookie=27b345df3693fb7b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:18:39.223 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:39.210 7fdcb6ffd700 1 -- 172.21.2.112:0/3387631402 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdcb000b5c0 con 0x7fdcb81b0470 2021-02-10T11:18:39.223 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:39.210 7fdcb6ffd700 1 -- 172.21.2.112:0/3387631402 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcb000b8c0 con 0x7fdcb81b0470 2021-02-10T11:18:39.223 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:39.210 7fdcb6ffd700 1 -- 172.21.2.112:0/3387631402 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdcb000bde0 con 0x7fdcb81b0470 2021-02-10T11:18:39.223 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:39.210 7fdcbe6e2700 1 -- 172.21.2.112:0/3387631402 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdcb81b0470 msgr2=0x7fdcb81b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:39.223 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:39.210 7fdcbe6e2700 1 --2- 172.21.2.112:0/3387631402 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdcb81b0470 0x7fdcb81b0870 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7fdcb0006a80 tx=0x7fdcb0009f00).stop 2021-02-10T11:18:39.223 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:39.211 7fdcbe6e2700 1 -- 172.21.2.112:0/3387631402 shutdown_connections 2021-02-10T11:18:39.223 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:39.211 7fdcbe6e2700 1 --2- 172.21.2.112:0/3387631402 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdcb80f0030 0x7fdcb80f0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:39.223 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:39.211 7fdcbe6e2700 1 --2- 172.21.2.112:0/3387631402 >> v2:172.21.2.120:3300/0 conn(0x7fdcb80ef610 0x7fdcb80efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:39.223 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:39.211 7fdcbe6e2700 1 --2- 172.21.2.112:0/3387631402 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdcb81b0470 0x7fdcb81b0870 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:39.223 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:39.211 7fdcbe6e2700 1 -- 172.21.2.112:0/3387631402 >> 172.21.2.112:0/3387631402 conn(0x7fdcb80ce3e0 msgr2=0x7fdcb80e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:18:39.223 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:39.211 7fdcbe6e2700 1 -- 172.21.2.112:0/3387631402 shutdown_connections 2021-02-10T11:18:39.223 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:39.211 7fdcbe6e2700 1 -- 172.21.2.112:0/3387631402 wait complete. 2021-02-10T11:18:39.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:39.225 7fdcbe6e2700 1 Processor -- start 2021-02-10T11:18:39.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:39.225 7fdcbe6e2700 1 -- start start 2021-02-10T11:18:39.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:39.226 7fdcbe6e2700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fdcb80b7c20 0x7fdcb80b8410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:39.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:39.226 7fdcbe6e2700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdcb80b8a30 0x7fdcb80c1f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:39.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:39.226 7fdcbe6e2700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdcb80c25a0 0x7fdcb80c29a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:39.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:39.226 7fdcbe6e2700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcb80c1a80 con 0x7fdcb80c25a0 2021-02-10T11:18:39.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:39.226 7fdcbe6e2700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcb80b8020 con 0x7fdcb80b7c20 2021-02-10T11:18:39.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:39.226 7fdcbe6e2700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcb809b9c0 con 0x7fdcb80b8a30 2021-02-10T11:18:39.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:39.226 7fdcbccff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdcb80c25a0 0x7fdcb80c29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:39.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:39.226 7fdcbccff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdcb80c25a0 0x7fdcb80c29a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:44346/0 (socket says 172.21.2.112:44346) 2021-02-10T11:18:39.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:39.226 7fdcbccff700 1 -- 172.21.2.112:0/1925298493 learned_addr learned my addr 172.21.2.112:0/1925298493 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:18:39.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:39.226 7fdcb7fff700 1 --2- 172.21.2.112:0/1925298493 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fdcb80b7c20 0x7fdcb80b8410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:39.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:39.226 7fdcb77fe700 1 --2- 172.21.2.112:0/1925298493 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdcb80b8a30 0x7fdcb80c1f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:39.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:39.227 7fdcbccff700 1 -- 172.21.2.112:0/1925298493 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdcb80b8a30 msgr2=0x7fdcb80c1f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:39.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:39.227 7fdcbccff700 1 --2- 172.21.2.112:0/1925298493 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdcb80b8a30 0x7fdcb80c1f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:39.227 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:39.227 7fdcbccff700 1 -- 172.21.2.112:0/1925298493 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fdcb80b7c20 msgr2=0x7fdcb80b8410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:39.227 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:39.227 7fdcbccff700 1 --2- 172.21.2.112:0/1925298493 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fdcb80b7c20 0x7fdcb80b8410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:39.227 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:39.227 7fdcbccff700 1 -- 172.21.2.112:0/1925298493 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcb80b8e30 con 0x7fdcb80c25a0 2021-02-10T11:18:39.227 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:39.227 7fdcb7fff700 1 --2- 172.21.2.112:0/1925298493 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fdcb80b7c20 0x7fdcb80b8410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:18:39.227 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:39.227 7fdcbccff700 1 --2- 172.21.2.112:0/1925298493 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdcb80c25a0 0x7fdcb80c29a0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 rx=0x7fdca8006f10 tx=0x7fdca800b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:18:39.227 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:39.227 7fdcb4ff9700 1 -- 172.21.2.112:0/1925298493 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdca800bd80 con 0x7fdcb80c25a0 2021-02-10T11:18:39.227 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:39.227 7fdcb4ff9700 1 -- 172.21.2.112:0/1925298493 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdca8006410 con 0x7fdcb80c25a0 2021-02-10T11:18:39.227 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:39.227 7fdcbe6e2700 1 -- 172.21.2.112:0/1925298493 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcb80b8e30 con 0x7fdcb80c25a0 2021-02-10T11:18:39.227 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:39.227 7fdcb4ff9700 1 -- 172.21.2.112:0/1925298493 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdca8002220 con 0x7fdcb80c25a0 2021-02-10T11:18:39.227 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:39.227 7fdcbe6e2700 1 -- 172.21.2.112:0/1925298493 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcb80b8e30 con 0x7fdcb80c25a0 2021-02-10T11:18:39.227 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:39.227 7fdcb4ff9700 1 -- 172.21.2.112:0/1925298493 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fdca800a030 con 0x7fdcb80c25a0 2021-02-10T11:18:39.227 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:39.228 7fdcb4ff9700 1 --2- 172.21.2.112:0/1925298493 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fdc98018730 0x7fdc9801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:39.227 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:39.228 7fdcb4ff9700 1 -- 172.21.2.112:0/1925298493 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fdca8032790 con 0x7fdcb80c25a0 2021-02-10T11:18:39.227 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:39.228 7fdcb7fff700 1 --2- 172.21.2.112:0/1925298493 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fdc98018730 0x7fdc9801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:39.228 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:39.228 7fdcb7fff700 1 --2- 172.21.2.112:0/1925298493 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fdc98018730 0x7fdc9801ac10 crc :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:18:39.255 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:39.259 7fdcbe6e2700 1 -- 172.21.2.112:0/1925298493 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcb8105b30 con 0x7fdcb80c25a0 2021-02-10T11:18:39.257 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:39.261 7fdcb4ff9700 1 -- 172.21.2.112:0/1925298493 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fdca80063b0 con 0x7fdcb80c25a0 2021-02-10T11:18:39.389 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:39.393 7fdcbe6e2700 1 -- 172.21.2.112:0/1925298493 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348587", "target": ["mgr", ""]}) v1 -- 0x7fdcb80f0b50 con 0x7fdc98018730 2021-02-10T11:18:39.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:39.397 7fdcb4ff9700 1 -- 172.21.2.112:0/1925298493 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7fdcb0000d30 con 0x7fdc98018730 2021-02-10T11:18:39.396 INFO:teuthology.orchestra.run.gibba012.stdout:/volumes/_nogroup/subvolume_0000000000348587/6b8b1b2a-d316-4292-8ea7-3e12fed9623e 2021-02-10T11:18:39.399 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:39.402 7fdcbe6e2700 1 -- 172.21.2.112:0/1925298493 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fdc98018730 msgr2=0x7fdc9801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:39.399 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:39.402 7fdcbe6e2700 1 --2- 172.21.2.112:0/1925298493 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fdc98018730 0x7fdc9801ac10 crc :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:39.399 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:39.402 7fdcbe6e2700 1 -- 172.21.2.112:0/1925298493 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdcb80c25a0 msgr2=0x7fdcb80c29a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:39.399 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:39.402 7fdcbe6e2700 1 --2- 172.21.2.112:0/1925298493 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdcb80c25a0 0x7fdcb80c29a0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 rx=0x7fdca8006f10 tx=0x7fdca800b990).stop 2021-02-10T11:18:39.399 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:39.403 7fdcbe6e2700 1 -- 172.21.2.112:0/1925298493 shutdown_connections 2021-02-10T11:18:39.399 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:39.403 7fdcbe6e2700 1 --2- 172.21.2.112:0/1925298493 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fdc98018730 0x7fdc9801ac10 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:39.399 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:39.403 7fdcbe6e2700 1 --2- 172.21.2.112:0/1925298493 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdcb80c25a0 0x7fdcb80c29a0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:39.399 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:39.403 7fdcbe6e2700 1 --2- 172.21.2.112:0/1925298493 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdcb80b8a30 0x7fdcb80c1f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:39.399 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:39.403 7fdcbe6e2700 1 --2- 172.21.2.112:0/1925298493 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fdcb80b7c20 0x7fdcb80b8410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:39.399 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:39.403 7fdcbe6e2700 1 -- 172.21.2.112:0/1925298493 >> 172.21.2.112:0/1925298493 conn(0x7fdcb8094e10 msgr2=0x7fdcb80a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:18:39.400 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:39.403 7fdcbe6e2700 1 -- 172.21.2.112:0/1925298493 shutdown_connections 2021-02-10T11:18:39.400 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:39.403 7fdcbe6e2700 1 -- 172.21.2.112:0/1925298493 wait complete. 2021-02-10T11:18:39.407 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000348587 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000348587/6b8b1b2a-d316-4292-8ea7-3e12fed9623e 2021-02-10T11:18:39.407 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:39.407 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:39.407 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:39.744 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:18:39.744 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:39.744 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348587/6b8b1b2a-d316-4292-8ea7-3e12fed9623e/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:40.124 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:18:40.124 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:18:40.124 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.025637 s, 40.9 MB/s 2021-02-10T11:18:40.125 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:40.125 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:40.125 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:40.460 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:18:40.460 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:40.461 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348587/6b8b1b2a-d316-4292-8ea7-3e12fed9623e/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:40.835 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:18:40.836 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:18:40.836 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.024447 s, 42.9 MB/s 2021-02-10T11:18:40.837 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:40.837 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:40.837 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:41.172 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:18:41.173 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:41.173 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348587/6b8b1b2a-d316-4292-8ea7-3e12fed9623e/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:41.549 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:18:41.550 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:18:41.550 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0253829 s, 41.3 MB/s 2021-02-10T11:18:41.551 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:41.551 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:41.551 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:41.886 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:18:41.887 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:41.887 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348587/6b8b1b2a-d316-4292-8ea7-3e12fed9623e/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:42.261 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:18:42.262 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:18:42.262 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0243924 s, 43.0 MB/s 2021-02-10T11:18:42.263 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:42.263 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:42.263 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:42.598 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:18:42.598 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:42.598 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348587/6b8b1b2a-d316-4292-8ea7-3e12fed9623e/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:42.974 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:18:42.974 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:18:42.974 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0247088 s, 42.4 MB/s 2021-02-10T11:18:42.975 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:42.975 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:42.975 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:43.310 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:18:43.310 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:43.310 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348587/6b8b1b2a-d316-4292-8ea7-3e12fed9623e/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:43.683 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:18:43.683 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:18:43.683 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0235051 s, 44.6 MB/s 2021-02-10T11:18:43.684 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:43.684 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:43.684 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:44.019 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:18:44.019 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:44.019 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348587/6b8b1b2a-d316-4292-8ea7-3e12fed9623e/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:44.392 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:18:44.392 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:18:44.392 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0237167 s, 44.2 MB/s 2021-02-10T11:18:44.393 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:44.393 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:44.393 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:44.726 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:18:44.727 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:44.727 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348587/6b8b1b2a-d316-4292-8ea7-3e12fed9623e/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:45.102 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:18:45.102 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:18:45.102 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0245925 s, 42.6 MB/s 2021-02-10T11:18:45.103 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:45.103 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:45.103 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:45.437 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:18:45.437 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:45.437 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348587/6b8b1b2a-d316-4292-8ea7-3e12fed9623e/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:45.813 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:18:45.813 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:18:45.813 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0259728 s, 40.4 MB/s 2021-02-10T11:18:45.814 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:45.814 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:45.814 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:46.149 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:18:46.149 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:46.149 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348587/6b8b1b2a-d316-4292-8ea7-3e12fed9623e/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:46.525 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:18:46.525 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:18:46.525 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0258294 s, 40.6 MB/s 2021-02-10T11:18:46.526 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000348588 2021-02-10T11:18:46.955 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:46.943 7f5a6d18a700 1 Processor -- start 2021-02-10T11:18:46.955 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:46.943 7f5a6d18a700 1 -- start start 2021-02-10T11:18:46.955 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:46.943 7f5a6d18a700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f5a681b0470 0x7f5a681b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:46.955 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:46.943 7f5a6d18a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5a680ef610 0x7f5a680efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:46.956 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:46.943 7f5a6d18a700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a680f0b50 con 0x7f5a680f0030 2021-02-10T11:18:46.956 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:46.943 7f5a6d18a700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a680e52e0 con 0x7f5a681b0470 2021-02-10T11:18:46.956 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:46.943 7f5a6d18a700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a680e54d0 con 0x7f5a680ef610 2021-02-10T11:18:46.956 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:46.943 7f5a66d9d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f5a681b0470 0x7f5a681b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:46.956 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:46.943 7f5a66d9d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f5a681b0470 0x7f5a681b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:44354/0 (socket says 172.21.2.112:44354) 2021-02-10T11:18:46.956 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:46.943 7f5a66d9d700 1 -- 172.21.2.112:0/2627486025 learned_addr learned my addr 172.21.2.112:0/2627486025 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:18:46.956 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:46.944 7f5a66d9d700 1 -- 172.21.2.112:0/2627486025 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5a680ef610 msgr2=0x7f5a680efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:18:46.956 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:46.944 7f5a66d9d700 1 --2- 172.21.2.112:0/2627486025 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5a680ef610 0x7f5a680efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:18:46.956 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:46.944 7f5a66d9d700 1 -- 172.21.2.112:0/2627486025 >> v1:172.21.2.120:6789/0 conn(0x7f5a680f0030 legacy=0x7f5a680f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:18:46.956 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:46.944 7f5a66d9d700 1 -- 172.21.2.112:0/2627486025 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a680f0430 con 0x7f5a681b0470 2021-02-10T11:18:46.956 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:46.944 7f5a66d9d700 1 --2- 172.21.2.112:0/2627486025 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5a681b0470 0x7f5a681b0870 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7f5a58006700 tx=0x7f5a58009b80).ready entity=mon.0 client_cookie=e1e174ffaa0fd42 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:18:46.956 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:46.944 7f5a65d9b700 1 -- 172.21.2.112:0/2627486025 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5a5800c0c0 con 0x7f5a681b0470 2021-02-10T11:18:46.956 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:46.944 7f5a65d9b700 1 -- 172.21.2.112:0/2627486025 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a5800c400 con 0x7f5a681b0470 2021-02-10T11:18:46.957 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:46.944 7f5a65d9b700 1 -- 172.21.2.112:0/2627486025 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5a5800c920 con 0x7f5a681b0470 2021-02-10T11:18:46.957 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:46.944 7f5a6d18a700 1 -- 172.21.2.112:0/2627486025 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5a681b0470 msgr2=0x7f5a681b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:46.957 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:46.944 7f5a6d18a700 1 --2- 172.21.2.112:0/2627486025 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5a681b0470 0x7f5a681b0870 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7f5a58006700 tx=0x7f5a58009b80).stop 2021-02-10T11:18:46.957 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:46.945 7f5a6d18a700 1 -- 172.21.2.112:0/2627486025 shutdown_connections 2021-02-10T11:18:46.957 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:46.945 7f5a6d18a700 1 --2- 172.21.2.112:0/2627486025 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5a680ef610 0x7f5a680efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:18:46.957 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:46.945 7f5a6d18a700 1 --2- 172.21.2.112:0/2627486025 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5a681b0470 0x7f5a681b0870 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:46.957 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:46.945 7f5a6d18a700 1 -- 172.21.2.112:0/2627486025 >> 172.21.2.112:0/2627486025 conn(0x7f5a680ce3e0 msgr2=0x7f5a680e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:18:46.957 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:46.945 7f5a6d18a700 1 -- 172.21.2.112:0/2627486025 shutdown_connections 2021-02-10T11:18:46.957 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:46.945 7f5a6d18a700 1 -- 172.21.2.112:0/2627486025 wait complete. 2021-02-10T11:18:46.959 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:46.960 7f5a6d18a700 1 Processor -- start 2021-02-10T11:18:46.959 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:46.960 7f5a6d18a700 1 -- start start 2021-02-10T11:18:46.959 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:46.960 7f5a6d18a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5a68094fb0 0x7f5a6809e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:46.959 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:46.960 7f5a6d18a700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5a680c2be0 0x7f5a680c2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:46.959 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:46.960 7f5a6d18a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5a680c3600 0x7f5a680a9c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:46.959 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:46.960 7f5a6d18a700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a6809ebf0 con 0x7f5a680c3600 2021-02-10T11:18:46.959 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:46.960 7f5a6d18a700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a680aa2a0 con 0x7f5a680c2be0 2021-02-10T11:18:46.959 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:46.960 7f5a6d18a700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a680aa4c0 con 0x7f5a68094fb0 2021-02-10T11:18:46.959 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:46.960 7f5a6759e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5a680c3600 0x7f5a680a9c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:46.959 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:46.960 7f5a6759e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5a680c3600 0x7f5a680a9c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:44362/0 (socket says 172.21.2.112:44362) 2021-02-10T11:18:46.960 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:46.960 7f5a6759e700 1 -- 172.21.2.112:0/4234540299 learned_addr learned my addr 172.21.2.112:0/4234540299 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:18:46.960 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:46.961 7f5a6759e700 1 -- 172.21.2.112:0/4234540299 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5a68094fb0 msgr2=0x7f5a6809e5d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:18:46.960 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:46.961 7f5a6759e700 1 --2- 172.21.2.112:0/4234540299 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5a68094fb0 0x7f5a6809e5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:18:46.960 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:46.961 7f5a6759e700 1 -- 172.21.2.112:0/4234540299 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5a680c2be0 msgr2=0x7f5a680c2fe0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:18:46.960 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:46.961 7f5a6759e700 1 --2- 172.21.2.112:0/4234540299 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5a680c2be0 0x7f5a680c2fe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:18:46.960 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:46.961 7f5a6759e700 1 -- 172.21.2.112:0/4234540299 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a680aa6e0 con 0x7f5a680c3600 2021-02-10T11:18:46.960 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:46.961 7f5a6759e700 1 --2- 172.21.2.112:0/4234540299 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5a680c3600 0x7f5a680a9c80 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 rx=0x7f5a60009940 tx=0x7f5a6000bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:18:46.960 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:46.961 7f5a5f7fe700 1 -- 172.21.2.112:0/4234540299 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5a60000f90 con 0x7f5a680c3600 2021-02-10T11:18:46.960 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:46.961 7f5a5f7fe700 1 -- 172.21.2.112:0/4234540299 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a6000c2f0 con 0x7f5a680c3600 2021-02-10T11:18:46.960 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:46.961 7f5a5f7fe700 1 -- 172.21.2.112:0/4234540299 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5a6000c7d0 con 0x7f5a680c3600 2021-02-10T11:18:46.960 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:46.961 7f5a6d18a700 1 -- 172.21.2.112:0/4234540299 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a680aa6e0 con 0x7f5a680c3600 2021-02-10T11:18:46.960 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:46.961 7f5a6d18a700 1 -- 172.21.2.112:0/4234540299 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a680aa6e0 con 0x7f5a680c3600 2021-02-10T11:18:46.960 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:46.962 7f5a5f7fe700 1 -- 172.21.2.112:0/4234540299 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5a60001220 con 0x7f5a680c3600 2021-02-10T11:18:46.961 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:46.962 7f5a5f7fe700 1 --2- 172.21.2.112:0/4234540299 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5a4c0186c0 0x7f5a4c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:46.961 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:46.962 7f5a5f7fe700 1 -- 172.21.2.112:0/4234540299 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f5a60032910 con 0x7f5a680c3600 2021-02-10T11:18:46.961 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:46.962 7f5a66d9d700 1 --2- 172.21.2.112:0/4234540299 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5a4c0186c0 0x7f5a4c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:46.961 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:46.963 7f5a66d9d700 1 --2- 172.21.2.112:0/4234540299 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5a4c0186c0 0x7f5a4c01abc0 crc :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:18:46.989 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:46.993 7f5a6d18a700 1 -- 172.21.2.112:0/4234540299 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a680edd10 con 0x7f5a680c3600 2021-02-10T11:18:46.992 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:46.996 7f5a5f7fe700 1 -- 172.21.2.112:0/4234540299 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5a6000be30 con 0x7f5a680c3600 2021-02-10T11:18:47.118 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:47.122 7f5a6d18a700 1 -- 172.21.2.112:0/4234540299 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348588", "target": ["mgr", ""]}) v1 -- 0x7f5a6809ebf0 con 0x7f5a4c0186c0 2021-02-10T11:18:47.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:47.148 7f5a5f7fe700 1 -- 172.21.2.112:0/4234540299 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f5a58000a50 con 0x7f5a4c0186c0 2021-02-10T11:18:47.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:47.155 7f5a6d18a700 1 -- 172.21.2.112:0/4234540299 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5a4c0186c0 msgr2=0x7f5a4c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:47.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:47.155 7f5a6d18a700 1 --2- 172.21.2.112:0/4234540299 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5a4c0186c0 0x7f5a4c01abc0 crc :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:47.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:47.155 7f5a6d18a700 1 -- 172.21.2.112:0/4234540299 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5a680c3600 msgr2=0x7f5a680a9c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:47.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:47.155 7f5a6d18a700 1 --2- 172.21.2.112:0/4234540299 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5a680c3600 0x7f5a680a9c80 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 rx=0x7f5a60009940 tx=0x7f5a6000bcf0).stop 2021-02-10T11:18:47.152 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:47.155 7f5a6d18a700 1 -- 172.21.2.112:0/4234540299 shutdown_connections 2021-02-10T11:18:47.152 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:47.155 7f5a6d18a700 1 --2- 172.21.2.112:0/4234540299 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5a4c0186c0 0x7f5a4c01abc0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:47.152 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:47.155 7f5a6d18a700 1 --2- 172.21.2.112:0/4234540299 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5a680c2be0 0x7f5a680c2fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:18:47.152 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:47.155 7f5a6d18a700 1 --2- 172.21.2.112:0/4234540299 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5a680c3600 0x7f5a680a9c80 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:47.152 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:47.155 7f5a6d18a700 1 --2- 172.21.2.112:0/4234540299 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5a68094fb0 0x7f5a6809e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:18:47.152 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:47.155 7f5a6d18a700 1 -- 172.21.2.112:0/4234540299 >> 172.21.2.112:0/4234540299 conn(0x7f5a6809f630 msgr2=0x7f5a680a0130 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:18:47.152 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:47.155 7f5a6d18a700 1 -- 172.21.2.112:0/4234540299 shutdown_connections 2021-02-10T11:18:47.152 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:47.155 7f5a6d18a700 1 -- 172.21.2.112:0/4234540299 wait complete. 2021-02-10T11:18:47.159 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000348588 2021-02-10T11:18:47.588 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:47.577 7f861e6b2700 1 Processor -- start 2021-02-10T11:18:47.588 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:47.577 7f861e6b2700 1 -- start start 2021-02-10T11:18:47.589 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:47.577 7f861e6b2700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f86181b0870 0x7f86180f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:47.589 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:47.577 7f861e6b2700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f86180efe00 0x7f86180f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:47.589 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:47.577 7f861e6b2700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86181b0c70 con 0x7f86181b0470 2021-02-10T11:18:47.589 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:47.577 7f861e6b2700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86180f0b50 con 0x7f86181b0870 2021-02-10T11:18:47.589 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:47.577 7f861e6b2700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86180f0820 con 0x7f86180efe00 2021-02-10T11:18:47.589 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:47.577 7f86177fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f86181b0870 0x7f86180f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:47.589 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:47.577 7f86177fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f86181b0870 0x7f86180f20e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:44368/0 (socket says 172.21.2.112:44368) 2021-02-10T11:18:47.589 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:47.577 7f86177fe700 1 -- 172.21.2.112:0/826180562 learned_addr learned my addr 172.21.2.112:0/826180562 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:18:47.589 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:47.578 7f86177fe700 1 -- 172.21.2.112:0/826180562 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f86180efe00 msgr2=0x7f86180f0200 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:18:47.589 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:47.578 7f86177fe700 1 --2- 172.21.2.112:0/826180562 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f86180efe00 0x7f86180f0200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:18:47.589 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:47.578 7f86177fe700 1 -- 172.21.2.112:0/826180562 >> v1:172.21.2.120:6789/0 conn(0x7f86181b0470 legacy=0x7f86180ef610 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:18:47.589 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:47.578 7f86177fe700 1 -- 172.21.2.112:0/826180562 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86180e52e0 con 0x7f86181b0870 2021-02-10T11:18:47.589 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:47.578 7f86177fe700 1 --2- 172.21.2.112:0/826180562 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f86181b0870 0x7f86180f20e0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7f860c006700 tx=0x7f860c009b80).ready entity=mon.0 client_cookie=930e84036fc255e6 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:18:47.590 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:47.578 7f8616ffd700 1 -- 172.21.2.112:0/826180562 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f860c00c0c0 con 0x7f86181b0870 2021-02-10T11:18:47.590 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:47.578 7f8616ffd700 1 -- 172.21.2.112:0/826180562 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f860c00c340 con 0x7f86181b0870 2021-02-10T11:18:47.590 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:47.578 7f8616ffd700 1 -- 172.21.2.112:0/826180562 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f860c00c860 con 0x7f86181b0870 2021-02-10T11:18:47.590 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:47.578 7f861e6b2700 1 -- 172.21.2.112:0/826180562 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f86181b0870 msgr2=0x7f86180f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:47.590 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:47.578 7f861e6b2700 1 --2- 172.21.2.112:0/826180562 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f86181b0870 0x7f86180f20e0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7f860c006700 tx=0x7f860c009b80).stop 2021-02-10T11:18:47.590 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:47.579 7f861e6b2700 1 -- 172.21.2.112:0/826180562 shutdown_connections 2021-02-10T11:18:47.590 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:47.579 7f861e6b2700 1 --2- 172.21.2.112:0/826180562 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f86181b0870 0x7f86180f20e0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:47.590 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:47.579 7f861e6b2700 1 --2- 172.21.2.112:0/826180562 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f86180efe00 0x7f86180f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:18:47.590 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:47.579 7f861e6b2700 1 -- 172.21.2.112:0/826180562 >> 172.21.2.112:0/826180562 conn(0x7f86180ce3e0 msgr2=0x7f86180e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:18:47.590 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:47.579 7f861e6b2700 1 -- 172.21.2.112:0/826180562 shutdown_connections 2021-02-10T11:18:47.590 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:47.579 7f861e6b2700 1 -- 172.21.2.112:0/826180562 wait complete. 2021-02-10T11:18:47.592 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:47.593 7f861e6b2700 1 Processor -- start 2021-02-10T11:18:47.592 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:47.593 7f861e6b2700 1 -- start start 2021-02-10T11:18:47.592 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:47.593 7f861e6b2700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f86180c36a0 0x7f86180c2b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:47.592 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:47.593 7f861e6b2700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f86180c3190 0x7f86180b1d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:47.592 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:47.593 7f861e6b2700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f86180b2370 0x7f86180b2d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:47.592 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:47.593 7f861e6b2700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86180b27c0 con 0x7f86180c3190 2021-02-10T11:18:47.592 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:47.593 7f861e6b2700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86180b3360 con 0x7f86180c36a0 2021-02-10T11:18:47.592 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:47.593 7f861e6b2700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86180b3540 con 0x7f86180b2370 2021-02-10T11:18:47.593 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:47.594 7f86177fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f86180c3190 0x7f86180b1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:47.593 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:47.594 7f86177fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f86180c3190 0x7f86180b1d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:44374/0 (socket says 172.21.2.112:44374) 2021-02-10T11:18:47.593 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:47.594 7f86177fe700 1 -- 172.21.2.112:0/2116728051 learned_addr learned my addr 172.21.2.112:0/2116728051 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:18:47.593 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:47.594 7f86177fe700 1 -- 172.21.2.112:0/2116728051 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f86180b2370 msgr2=0x7f86180b2d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:47.593 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:47.594 7f86177fe700 1 --2- 172.21.2.112:0/2116728051 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f86180b2370 0x7f86180b2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:18:47.593 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:47.594 7f86177fe700 1 -- 172.21.2.112:0/2116728051 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f86180c36a0 msgr2=0x7f86180c2b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:47.593 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:47.594 7f86177fe700 1 --2- 172.21.2.112:0/2116728051 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f86180c36a0 0x7f86180c2b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:18:47.593 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:47.594 7f86177fe700 1 -- 172.21.2.112:0/2116728051 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86180b3760 con 0x7f86180c3190 2021-02-10T11:18:47.593 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:47.594 7f86177fe700 1 --2- 172.21.2.112:0/2116728051 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f86180c3190 0x7f86180b1d50 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 rx=0x7f860c00b580 tx=0x7f860c00b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:18:47.593 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:47.594 7f8614ff9700 1 -- 172.21.2.112:0/2116728051 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f860c00bd10 con 0x7f86180c3190 2021-02-10T11:18:47.593 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:47.594 7f8614ff9700 1 -- 172.21.2.112:0/2116728051 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f860c002030 con 0x7f86180c3190 2021-02-10T11:18:47.593 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:47.594 7f861e6b2700 1 -- 172.21.2.112:0/2116728051 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86180b3760 con 0x7f86180c3190 2021-02-10T11:18:47.593 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:47.594 7f8614ff9700 1 -- 172.21.2.112:0/2116728051 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f860c002550 con 0x7f86180c3190 2021-02-10T11:18:47.594 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:47.594 7f861e6b2700 1 -- 172.21.2.112:0/2116728051 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86180b3cf0 con 0x7f86180c3190 2021-02-10T11:18:47.594 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:47.595 7f8614ff9700 1 -- 172.21.2.112:0/2116728051 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f860c000c90 con 0x7f86180c3190 2021-02-10T11:18:47.594 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:47.595 7f8614ff9700 1 --2- 172.21.2.112:0/2116728051 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f85f80186c0 0x7f85f801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:47.594 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:47.595 7f8614ff9700 1 -- 172.21.2.112:0/2116728051 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f860c032450 con 0x7f86180c3190 2021-02-10T11:18:47.594 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:47.595 7f8617fff700 1 --2- 172.21.2.112:0/2116728051 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f85f80186c0 0x7f85f801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:47.594 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:47.596 7f8617fff700 1 --2- 172.21.2.112:0/2116728051 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f85f80186c0 0x7f85f801abc0 crc :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:18:47.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:47.627 7f861e6b2700 1 -- 172.21.2.112:0/2116728051 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86180edd10 con 0x7f86180c3190 2021-02-10T11:18:47.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:47.630 7f8614ff9700 1 -- 172.21.2.112:0/2116728051 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f860c02df30 con 0x7f86180c3190 2021-02-10T11:18:47.754 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:47.758 7f861e6b2700 1 -- 172.21.2.112:0/2116728051 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348588", "target": ["mgr", ""]}) v1 -- 0x7f86180b27c0 con 0x7f85f80186c0 2021-02-10T11:18:47.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:47.762 7f8614ff9700 1 -- 172.21.2.112:0/2116728051 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f8608006a10 con 0x7f85f80186c0 2021-02-10T11:18:47.757 INFO:teuthology.orchestra.run.gibba012.stdout:/volumes/_nogroup/subvolume_0000000000348588/f11a264d-697c-4097-bdcb-0a1bba5b1638 2021-02-10T11:18:47.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:47.763 7f861e6b2700 1 -- 172.21.2.112:0/2116728051 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f85f80186c0 msgr2=0x7f85f801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:47.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:47.763 7f861e6b2700 1 --2- 172.21.2.112:0/2116728051 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f85f80186c0 0x7f85f801abc0 crc :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:47.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:47.763 7f861e6b2700 1 -- 172.21.2.112:0/2116728051 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f86180c3190 msgr2=0x7f86180b1d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:47.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:47.763 7f861e6b2700 1 --2- 172.21.2.112:0/2116728051 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f86180c3190 0x7f86180b1d50 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 rx=0x7f860c00b580 tx=0x7f860c00b920).stop 2021-02-10T11:18:47.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:47.763 7f861e6b2700 1 -- 172.21.2.112:0/2116728051 shutdown_connections 2021-02-10T11:18:47.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:47.763 7f861e6b2700 1 --2- 172.21.2.112:0/2116728051 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f85f80186c0 0x7f85f801abc0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:47.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:47.763 7f861e6b2700 1 --2- 172.21.2.112:0/2116728051 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f86180b2370 0x7f86180b2d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:18:47.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:47.763 7f861e6b2700 1 --2- 172.21.2.112:0/2116728051 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f86180c3190 0x7f86180b1d50 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:47.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:47.763 7f861e6b2700 1 --2- 172.21.2.112:0/2116728051 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f86180c36a0 0x7f86180c2b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:18:47.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:47.763 7f861e6b2700 1 -- 172.21.2.112:0/2116728051 >> 172.21.2.112:0/2116728051 conn(0x7f861809f630 msgr2=0x7f86180a0130 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:18:47.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:47.763 7f861e6b2700 1 -- 172.21.2.112:0/2116728051 shutdown_connections 2021-02-10T11:18:47.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:47.764 7f861e6b2700 1 -- 172.21.2.112:0/2116728051 wait complete. 2021-02-10T11:18:47.767 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000348588 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000348588/f11a264d-697c-4097-bdcb-0a1bba5b1638 2021-02-10T11:18:47.767 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:47.768 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:47.768 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:48.102 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:18:48.103 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:48.103 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348588/f11a264d-697c-4097-bdcb-0a1bba5b1638/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:48.485 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:18:48.485 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:18:48.485 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0256191 s, 40.9 MB/s 2021-02-10T11:18:48.486 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:48.486 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:48.486 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:48.821 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:18:48.821 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:48.821 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348588/f11a264d-697c-4097-bdcb-0a1bba5b1638/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:49.196 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:18:49.197 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:18:49.197 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0245984 s, 42.6 MB/s 2021-02-10T11:18:49.198 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:49.198 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:49.198 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:49.531 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:18:49.532 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:49.532 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348588/f11a264d-697c-4097-bdcb-0a1bba5b1638/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:49.904 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:18:49.904 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:18:49.904 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0250844 s, 41.8 MB/s 2021-02-10T11:18:49.905 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:49.905 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:49.905 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:50.239 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:18:50.240 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:50.240 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348588/f11a264d-697c-4097-bdcb-0a1bba5b1638/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:50.614 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:18:50.614 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:18:50.614 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0244499 s, 42.9 MB/s 2021-02-10T11:18:50.615 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:50.615 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:50.616 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:50.950 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:18:50.950 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:50.950 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348588/f11a264d-697c-4097-bdcb-0a1bba5b1638/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:51.326 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:18:51.327 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:18:51.327 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0265487 s, 39.5 MB/s 2021-02-10T11:18:51.328 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:51.328 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:51.328 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:51.664 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:18:51.664 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:51.665 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348588/f11a264d-697c-4097-bdcb-0a1bba5b1638/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:52.039 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:18:52.040 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:18:52.040 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.024026 s, 43.6 MB/s 2021-02-10T11:18:52.040 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:52.041 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:52.041 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:52.376 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:18:52.377 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:52.377 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348588/f11a264d-697c-4097-bdcb-0a1bba5b1638/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:52.751 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:18:52.751 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:18:52.751 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0241645 s, 43.4 MB/s 2021-02-10T11:18:52.752 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:52.752 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:52.752 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:53.087 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:18:53.088 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:53.088 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348588/f11a264d-697c-4097-bdcb-0a1bba5b1638/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:53.464 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:18:53.464 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:18:53.464 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0248071 s, 42.3 MB/s 2021-02-10T11:18:53.465 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:53.466 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:53.466 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:53.800 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:18:53.801 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:53.801 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348588/f11a264d-697c-4097-bdcb-0a1bba5b1638/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:54.174 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:18:54.174 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:18:54.175 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0219361 s, 47.8 MB/s 2021-02-10T11:18:54.175 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:54.176 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:54.176 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:54.509 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:18:54.510 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:54.510 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348588/f11a264d-697c-4097-bdcb-0a1bba5b1638/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:54.882 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:18:54.882 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:18:54.882 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0250911 s, 41.8 MB/s 2021-02-10T11:18:54.883 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000348589 2021-02-10T11:18:55.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.300 7f1f6bb46700 1 Processor -- start 2021-02-10T11:18:55.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.300 7f1f6bb46700 1 -- start start 2021-02-10T11:18:55.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.300 7f1f6bb46700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1f641b0870 0x7f1f640f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:55.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.300 7f1f6bb46700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f641b0c70 con 0x7f1f641b0470 2021-02-10T11:18:55.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.300 7f1f6bb46700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f640f0b50 con 0x7f1f640efe00 2021-02-10T11:18:55.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.300 7f1f6bb46700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f640e5ea0 con 0x7f1f641b0870 2021-02-10T11:18:55.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.300 7f1f69962700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f1f641b0470 0x7f1f640ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:43844/0 (socket says 172.21.2.112:43844) 2021-02-10T11:18:55.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.300 7f1f69962700 1 -- 172.21.2.112:0/1235522585 learned_addr learned my addr 172.21.2.112:0/1235522585 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:18:55.314 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.301 7f1f69161700 1 --2- 172.21.2.112:0/1235522585 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1f641b0870 0x7f1f640f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:55.314 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.301 7f1f68960700 1 -- 172.21.2.112:0/1235522585 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1316915202 0 0) 0x7f1f4c001060 con 0x7f1f641b0470 2021-02-10T11:18:55.314 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.301 7f1f68960700 1 -- 172.21.2.112:0/1235522585 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f50002fe0 con 0x7f1f641b0470 2021-02-10T11:18:55.314 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.301 7f1f68960700 1 -- 172.21.2.112:0/1235522585 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2507406648 0 0) 0x7f1f5c001080 con 0x7f1f640efe00 2021-02-10T11:18:55.314 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.301 7f1f68960700 1 -- 172.21.2.112:0/1235522585 --> v1:172.21.2.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f50003210 con 0x7f1f640efe00 2021-02-10T11:18:55.314 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.301 7f1f68960700 1 -- 172.21.2.112:0/1235522585 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3336503900 0 0) 0x7f1f4c001460 con 0x7f1f641b0470 2021-02-10T11:18:55.314 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.301 7f1f69161700 1 -- 172.21.2.112:0/1235522585 >> v1:172.21.2.120:6789/0 conn(0x7f1f640efe00 legacy=0x7f1f640f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:55.314 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.301 7f1f69161700 1 -- 172.21.2.112:0/1235522585 >> v1:172.21.2.112:6789/0 conn(0x7f1f641b0470 legacy=0x7f1f640ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:55.314 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.301 7f1f69161700 1 -- 172.21.2.112:0/1235522585 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f640e52e0 con 0x7f1f641b0870 2021-02-10T11:18:55.314 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.302 7f1f69161700 1 --2- 172.21.2.112:0/1235522585 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1f641b0870 0x7f1f640f20e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f1f54006650 tx=0x7f1f54009a80).ready entity=mon.2 client_cookie=72caecad85c6f0a5 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:18:55.314 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.302 7f1f68960700 1 -- 172.21.2.112:0/1235522585 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1f5400c0c0 con 0x7f1f641b0870 2021-02-10T11:18:55.314 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.302 7f1f68960700 1 -- 172.21.2.112:0/1235522585 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f5400c340 con 0x7f1f641b0870 2021-02-10T11:18:55.314 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.302 7f1f68960700 1 -- 172.21.2.112:0/1235522585 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1f5400c820 con 0x7f1f641b0870 2021-02-10T11:18:55.315 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.302 7f1f6bb46700 1 -- 172.21.2.112:0/1235522585 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1f641b0870 msgr2=0x7f1f640f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:55.315 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.302 7f1f6bb46700 1 --2- 172.21.2.112:0/1235522585 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1f641b0870 0x7f1f640f20e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f1f54006650 tx=0x7f1f54009a80).stop 2021-02-10T11:18:55.315 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.303 7f1f6bb46700 1 -- 172.21.2.112:0/1235522585 shutdown_connections 2021-02-10T11:18:55.315 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.303 7f1f6bb46700 1 --2- 172.21.2.112:0/1235522585 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1f641b0870 0x7f1f640f20e0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:55.315 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.303 7f1f6bb46700 1 -- 172.21.2.112:0/1235522585 >> 172.21.2.112:0/1235522585 conn(0x7f1f640ce3e0 msgr2=0x7f1f640e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:18:55.315 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.303 7f1f6bb46700 1 -- 172.21.2.112:0/1235522585 shutdown_connections 2021-02-10T11:18:55.315 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.303 7f1f6bb46700 1 -- 172.21.2.112:0/1235522585 wait complete. 2021-02-10T11:18:55.316 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.318 7f1f6bb46700 1 Processor -- start 2021-02-10T11:18:55.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.318 7f1f6bb46700 1 -- start start 2021-02-10T11:18:55.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.318 7f1f6bb46700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1f64190920 0x7f1f640b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:55.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.318 7f1f6bb46700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1f640b8200 0x7f1f640b8600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:55.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.318 7f1f6bb46700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1f640c1d80 0x7f1f640c2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:55.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.318 7f1f6bb46700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f64190d20 con 0x7f1f640c1d80 2021-02-10T11:18:55.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.318 7f1f6bb46700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f640b8c20 con 0x7f1f640b8200 2021-02-10T11:18:55.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.318 7f1f6bb46700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f640c27a0 con 0x7f1f64190920 2021-02-10T11:18:55.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.318 7f1f6a163700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1f640c1d80 0x7f1f640c2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:55.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.318 7f1f6a163700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1f640c1d80 0x7f1f640c2180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:44386/0 (socket says 172.21.2.112:44386) 2021-02-10T11:18:55.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.318 7f1f6a163700 1 -- 172.21.2.112:0/1350612763 learned_addr learned my addr 172.21.2.112:0/1350612763 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:18:55.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.319 7f1f6a163700 1 -- 172.21.2.112:0/1350612763 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1f64190920 msgr2=0x7f1f640b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:55.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.319 7f1f6a163700 1 --2- 172.21.2.112:0/1350612763 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1f64190920 0x7f1f640b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:18:55.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.319 7f1f6a163700 1 -- 172.21.2.112:0/1350612763 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1f640b8200 msgr2=0x7f1f640b8600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:55.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.319 7f1f69161700 1 --2- 172.21.2.112:0/1350612763 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1f640b8200 0x7f1f640b8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:55.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.319 7f1f6a163700 1 --2- 172.21.2.112:0/1350612763 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1f640b8200 0x7f1f640b8600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:55.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.319 7f1f6a163700 1 -- 172.21.2.112:0/1350612763 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f640c29c0 con 0x7f1f640c1d80 2021-02-10T11:18:55.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.319 7f1f69161700 1 --2- 172.21.2.112:0/1350612763 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1f640b8200 0x7f1f640b8600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:18:55.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.319 7f1f6a163700 1 --2- 172.21.2.112:0/1350612763 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1f640c1d80 0x7f1f640c2180 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 rx=0x7f1f5c00b530 tx=0x7f1f5c00b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:18:55.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.319 7f1f627fc700 1 -- 172.21.2.112:0/1350612763 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1f5c00bcc0 con 0x7f1f640c1d80 2021-02-10T11:18:55.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.319 7f1f627fc700 1 -- 172.21.2.112:0/1350612763 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f5c006020 con 0x7f1f640c1d80 2021-02-10T11:18:55.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.319 7f1f627fc700 1 -- 172.21.2.112:0/1350612763 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1f5c0054a0 con 0x7f1f640c1d80 2021-02-10T11:18:55.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.319 7f1f6bb46700 1 -- 172.21.2.112:0/1350612763 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f640c29c0 con 0x7f1f640c1d80 2021-02-10T11:18:55.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.319 7f1f6bb46700 1 -- 172.21.2.112:0/1350612763 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f640c29c0 con 0x7f1f640c1d80 2021-02-10T11:18:55.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.320 7f1f627fc700 1 -- 172.21.2.112:0/1350612763 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1f5c000bc0 con 0x7f1f640c1d80 2021-02-10T11:18:55.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.320 7f1f627fc700 1 --2- 172.21.2.112:0/1350612763 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1f48018730 0x7f1f4801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:55.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.320 7f1f627fc700 1 -- 172.21.2.112:0/1350612763 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f1f5c0327e0 con 0x7f1f640c1d80 2021-02-10T11:18:55.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.320 7f1f69962700 1 --2- 172.21.2.112:0/1350612763 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1f48018730 0x7f1f4801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:55.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.321 7f1f69962700 1 --2- 172.21.2.112:0/1350612763 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1f48018730 0x7f1f4801ac10 crc :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:18:55.346 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.350 7f1f6bb46700 1 -- 172.21.2.112:0/1350612763 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f64105ae0 con 0x7f1f640c1d80 2021-02-10T11:18:55.348 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.353 7f1f627fc700 1 -- 172.21.2.112:0/1350612763 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1f5c006000 con 0x7f1f640c1d80 2021-02-10T11:18:55.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.479 7f1f6bb46700 1 -- 172.21.2.112:0/1350612763 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348589", "target": ["mgr", ""]}) v1 -- 0x7f1f640c32c0 con 0x7f1f48018730 2021-02-10T11:18:55.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.507 7f1f627fc700 1 -- 172.21.2.112:0/1350612763 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f1f4c001060 con 0x7f1f48018730 2021-02-10T11:18:55.509 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.513 7f1f6bb46700 1 -- 172.21.2.112:0/1350612763 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1f48018730 msgr2=0x7f1f4801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:55.509 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.513 7f1f6bb46700 1 --2- 172.21.2.112:0/1350612763 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1f48018730 0x7f1f4801ac10 crc :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:55.509 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.513 7f1f6bb46700 1 -- 172.21.2.112:0/1350612763 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1f640c1d80 msgr2=0x7f1f640c2180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:55.510 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.513 7f1f6bb46700 1 --2- 172.21.2.112:0/1350612763 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1f640c1d80 0x7f1f640c2180 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 rx=0x7f1f5c00b530 tx=0x7f1f5c00b8d0).stop 2021-02-10T11:18:55.510 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.513 7f1f6bb46700 1 -- 172.21.2.112:0/1350612763 shutdown_connections 2021-02-10T11:18:55.510 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.513 7f1f6bb46700 1 --2- 172.21.2.112:0/1350612763 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1f48018730 0x7f1f4801ac10 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:55.510 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.513 7f1f6bb46700 1 --2- 172.21.2.112:0/1350612763 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1f640b8200 0x7f1f640b8600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:55.510 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.513 7f1f6bb46700 1 --2- 172.21.2.112:0/1350612763 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1f640c1d80 0x7f1f640c2180 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:55.510 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.513 7f1f6bb46700 1 --2- 172.21.2.112:0/1350612763 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1f64190920 0x7f1f640b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:18:55.510 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.513 7f1f6bb46700 1 -- 172.21.2.112:0/1350612763 >> 172.21.2.112:0/1350612763 conn(0x7f1f64094e10 msgr2=0x7f1f640a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:18:55.510 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.513 7f1f6bb46700 1 -- 172.21.2.112:0/1350612763 shutdown_connections 2021-02-10T11:18:55.510 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.513 7f1f6bb46700 1 -- 172.21.2.112:0/1350612763 wait complete. 2021-02-10T11:18:55.517 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000348589 2021-02-10T11:18:55.948 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.934 7f20eca8e700 1 Processor -- start 2021-02-10T11:18:55.948 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.934 7f20eca8e700 1 -- start start 2021-02-10T11:18:55.948 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.935 7f20eca8e700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f20e81b0870 0x7f20e80f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:55.948 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.935 7f20eca8e700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f20e80efe00 0x7f20e80f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:55.948 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.935 7f20eca8e700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20e81b0c70 con 0x7f20e81b0470 2021-02-10T11:18:55.949 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.935 7f20eca8e700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20e80f0b50 con 0x7f20e81b0870 2021-02-10T11:18:55.949 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.935 7f20eca8e700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20e80f0820 con 0x7f20e80efe00 2021-02-10T11:18:55.949 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.935 7f20e605d700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f20e81b0870 0x7f20e80f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:55.949 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.935 7f20e605d700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f20e81b0870 0x7f20e80f20e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3300/0 says I am v2:172.21.2.112:46674/0 (socket says 172.21.2.112:46674) 2021-02-10T11:18:55.949 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.935 7f20e605d700 1 -- 172.21.2.112:0/1560205218 learned_addr learned my addr 172.21.2.112:0/1560205218 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:18:55.949 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.935 7f20e705f700 1 --2- 172.21.2.112:0/1560205218 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f20e80efe00 0x7f20e80f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:55.949 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.936 7f20e605d700 1 -- 172.21.2.112:0/1560205218 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f20e80efe00 msgr2=0x7f20e80f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:18:55.949 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.936 7f20e605d700 1 --2- 172.21.2.112:0/1560205218 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f20e80efe00 0x7f20e80f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:55.949 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.936 7f20e605d700 1 -- 172.21.2.112:0/1560205218 >> v1:172.21.2.120:6789/0 conn(0x7f20e81b0470 legacy=0x7f20e80ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:55.949 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.936 7f20e605d700 1 -- 172.21.2.112:0/1560205218 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20e80e52e0 con 0x7f20e81b0870 2021-02-10T11:18:55.949 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.936 7f20e605d700 1 --2- 172.21.2.112:0/1560205218 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f20e81b0870 0x7f20e80f20e0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f20d8006700 tx=0x7f20d8009b80).ready entity=mon.1 client_cookie=64512b7808b6a7d5 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:18:55.949 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.937 7f20e585c700 1 -- 172.21.2.112:0/1560205218 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f20d800c0c0 con 0x7f20e81b0870 2021-02-10T11:18:55.950 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.937 7f20e585c700 1 -- 172.21.2.112:0/1560205218 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20d800c340 con 0x7f20e81b0870 2021-02-10T11:18:55.950 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.937 7f20e585c700 1 -- 172.21.2.112:0/1560205218 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f20d800c860 con 0x7f20e81b0870 2021-02-10T11:18:55.950 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.937 7f20eca8e700 1 -- 172.21.2.112:0/1560205218 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f20e81b0870 msgr2=0x7f20e80f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:55.950 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.937 7f20eca8e700 1 --2- 172.21.2.112:0/1560205218 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f20e81b0870 0x7f20e80f20e0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f20d8006700 tx=0x7f20d8009b80).stop 2021-02-10T11:18:55.950 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.937 7f20eca8e700 1 -- 172.21.2.112:0/1560205218 shutdown_connections 2021-02-10T11:18:55.950 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.937 7f20eca8e700 1 --2- 172.21.2.112:0/1560205218 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f20e81b0870 0x7f20e80f20e0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:55.950 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.937 7f20eca8e700 1 --2- 172.21.2.112:0/1560205218 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f20e80efe00 0x7f20e80f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:55.950 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.937 7f20eca8e700 1 -- 172.21.2.112:0/1560205218 >> 172.21.2.112:0/1560205218 conn(0x7f20e80ce3e0 msgr2=0x7f20e80e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:18:55.950 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.937 7f20eca8e700 1 -- 172.21.2.112:0/1560205218 shutdown_connections 2021-02-10T11:18:55.950 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.938 7f20eca8e700 1 -- 172.21.2.112:0/1560205218 wait complete. 2021-02-10T11:18:55.952 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.952 7f20eca8e700 1 Processor -- start 2021-02-10T11:18:55.952 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.952 7f20eca8e700 1 -- start start 2021-02-10T11:18:55.952 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.952 7f20eca8e700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f20e80a0420 0x7f20e80a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:55.952 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.952 7f20eca8e700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f20e8189d00 0x7f20e818a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:55.952 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.952 7f20eca8e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f20e818a720 0x7f20e80b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:55.952 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.952 7f20eca8e700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20e8189780 con 0x7f20e818a720 2021-02-10T11:18:55.952 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.952 7f20eca8e700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20e818ab20 con 0x7f20e8189d00 2021-02-10T11:18:55.952 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.952 7f20eca8e700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20e80a1140 con 0x7f20e80a0420 2021-02-10T11:18:55.952 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.952 7f20e705f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f20e818a720 0x7f20e80b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:55.952 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.952 7f20e705f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f20e818a720 0x7f20e80b7be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:44400/0 (socket says 172.21.2.112:44400) 2021-02-10T11:18:55.953 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.952 7f20e705f700 1 -- 172.21.2.112:0/2633249821 learned_addr learned my addr 172.21.2.112:0/2633249821 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:18:55.953 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.952 7f20e705f700 1 -- 172.21.2.112:0/2633249821 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f20e80a0420 msgr2=0x7f20e80a0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:55.953 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.952 7f20e705f700 1 --2- 172.21.2.112:0/2633249821 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f20e80a0420 0x7f20e80a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:18:55.953 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.953 7f20e705f700 1 -- 172.21.2.112:0/2633249821 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f20e8189d00 msgr2=0x7f20e818a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:55.953 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.953 7f20e705f700 1 --2- 172.21.2.112:0/2633249821 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f20e8189d00 0x7f20e818a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:18:55.953 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.953 7f20e705f700 1 -- 172.21.2.112:0/2633249821 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20e80b8200 con 0x7f20e818a720 2021-02-10T11:18:55.953 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.953 7f20e705f700 1 --2- 172.21.2.112:0/2633249821 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f20e818a720 0x7f20e80b7be0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7f20e000b970 tx=0x7f20e000bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:18:55.953 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.953 7f20df7fe700 1 -- 172.21.2.112:0/2633249821 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f20e0000f90 con 0x7f20e818a720 2021-02-10T11:18:55.953 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.953 7f20df7fe700 1 -- 172.21.2.112:0/2633249821 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20e0003030 con 0x7f20e818a720 2021-02-10T11:18:55.953 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.953 7f20eca8e700 1 -- 172.21.2.112:0/2633249821 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20e80b8200 con 0x7f20e818a720 2021-02-10T11:18:55.953 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.953 7f20df7fe700 1 -- 172.21.2.112:0/2633249821 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f20e0001400 con 0x7f20e818a720 2021-02-10T11:18:55.953 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.953 7f20eca8e700 1 -- 172.21.2.112:0/2633249821 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20e80b8200 con 0x7f20e818a720 2021-02-10T11:18:55.953 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.953 7f20df7fe700 1 -- 172.21.2.112:0/2633249821 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f20e0000eb0 con 0x7f20e818a720 2021-02-10T11:18:55.953 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.953 7f20df7fe700 1 --2- 172.21.2.112:0/2633249821 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f20cc018730 0x7f20cc01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:55.954 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.954 7f20df7fe700 1 -- 172.21.2.112:0/2633249821 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f20e00329d0 con 0x7f20e818a720 2021-02-10T11:18:55.954 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.954 7f20e685e700 1 --2- 172.21.2.112:0/2633249821 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f20cc018730 0x7f20cc01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:55.954 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.954 7f20e685e700 1 --2- 172.21.2.112:0/2633249821 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f20cc018730 0x7f20cc01ac10 crc :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:18:55.981 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.984 7f20eca8e700 1 -- 172.21.2.112:0/2633249821 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20e8105ae0 con 0x7f20e818a720 2021-02-10T11:18:55.983 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:55.987 7f20df7fe700 1 -- 172.21.2.112:0/2633249821 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f20e001f890 con 0x7f20e818a720 2021-02-10T11:18:56.111 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:56.114 7f20eca8e700 1 -- 172.21.2.112:0/2633249821 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348589", "target": ["mgr", ""]}) v1 -- 0x7f20e80f0b50 con 0x7f20cc018730 2021-02-10T11:18:56.114 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:56.118 7f20df7fe700 1 -- 172.21.2.112:0/2633249821 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f20d0006a10 con 0x7f20cc018730 2021-02-10T11:18:56.118 INFO:teuthology.orchestra.run.gibba012.stdout:/volumes/_nogroup/subvolume_0000000000348589/8d559078-3925-4ae1-aef0-738f2ffdf39d 2021-02-10T11:18:56.120 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:56.124 7f20eca8e700 1 -- 172.21.2.112:0/2633249821 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f20cc018730 msgr2=0x7f20cc01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:56.121 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:56.124 7f20eca8e700 1 --2- 172.21.2.112:0/2633249821 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f20cc018730 0x7f20cc01ac10 crc :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:56.121 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:56.124 7f20eca8e700 1 -- 172.21.2.112:0/2633249821 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f20e818a720 msgr2=0x7f20e80b7be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:56.121 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:56.124 7f20eca8e700 1 --2- 172.21.2.112:0/2633249821 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f20e818a720 0x7f20e80b7be0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7f20e000b970 tx=0x7f20e000bd10).stop 2021-02-10T11:18:56.121 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:56.124 7f20eca8e700 1 -- 172.21.2.112:0/2633249821 shutdown_connections 2021-02-10T11:18:56.121 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:56.124 7f20eca8e700 1 --2- 172.21.2.112:0/2633249821 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f20cc018730 0x7f20cc01ac10 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:56.121 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:56.124 7f20eca8e700 1 --2- 172.21.2.112:0/2633249821 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f20e8189d00 0x7f20e818a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:18:56.121 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:56.124 7f20eca8e700 1 --2- 172.21.2.112:0/2633249821 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f20e818a720 0x7f20e80b7be0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:56.121 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:56.124 7f20eca8e700 1 --2- 172.21.2.112:0/2633249821 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f20e80a0420 0x7f20e80a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:18:56.121 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:56.124 7f20eca8e700 1 -- 172.21.2.112:0/2633249821 >> 172.21.2.112:0/2633249821 conn(0x7f20e8094e10 msgr2=0x7f20e80a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:18:56.121 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:56.124 7f20eca8e700 1 -- 172.21.2.112:0/2633249821 shutdown_connections 2021-02-10T11:18:56.121 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:18:56.124 7f20eca8e700 1 -- 172.21.2.112:0/2633249821 wait complete. 2021-02-10T11:18:56.128 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000348589 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000348589/8d559078-3925-4ae1-aef0-738f2ffdf39d 2021-02-10T11:18:56.128 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:56.129 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:56.129 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:56.463 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:18:56.464 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:56.464 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348589/8d559078-3925-4ae1-aef0-738f2ffdf39d/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:56.844 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:18:56.844 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:18:56.845 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0245639 s, 42.7 MB/s 2021-02-10T11:18:56.846 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:56.846 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:56.846 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:57.181 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:18:57.182 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:57.182 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348589/8d559078-3925-4ae1-aef0-738f2ffdf39d/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:57.556 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:18:57.556 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:18:57.556 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0238315 s, 44.0 MB/s 2021-02-10T11:18:57.557 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:57.557 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:57.557 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:57.892 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:18:57.892 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:57.892 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348589/8d559078-3925-4ae1-aef0-738f2ffdf39d/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:58.268 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:18:58.268 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:18:58.268 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0242832 s, 43.2 MB/s 2021-02-10T11:18:58.269 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:58.269 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:58.269 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:58.604 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:18:58.604 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:58.604 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348589/8d559078-3925-4ae1-aef0-738f2ffdf39d/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:58.977 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:18:58.977 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:18:58.977 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0223923 s, 46.8 MB/s 2021-02-10T11:18:58.978 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:58.978 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:58.978 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:59.313 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:18:59.313 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:59.313 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348589/8d559078-3925-4ae1-aef0-738f2ffdf39d/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:59.684 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:18:59.684 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:18:59.684 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.024869 s, 42.2 MB/s 2021-02-10T11:18:59.685 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:59.685 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:59.686 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:00.022 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:19:00.022 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:00.022 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348589/8d559078-3925-4ae1-aef0-738f2ffdf39d/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:00.393 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:19:00.393 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:19:00.393 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0220071 s, 47.6 MB/s 2021-02-10T11:19:00.394 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:00.394 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:00.395 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:00.729 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:19:00.729 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:00.729 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348589/8d559078-3925-4ae1-aef0-738f2ffdf39d/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:01.105 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:19:01.105 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:19:01.105 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0256057 s, 41.0 MB/s 2021-02-10T11:19:01.106 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:01.106 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:01.106 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:01.440 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:19:01.441 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:01.441 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348589/8d559078-3925-4ae1-aef0-738f2ffdf39d/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:01.813 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:19:01.813 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:19:01.813 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0228368 s, 45.9 MB/s 2021-02-10T11:19:01.814 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:01.814 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:01.814 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:02.149 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:19:02.150 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:02.150 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348589/8d559078-3925-4ae1-aef0-738f2ffdf39d/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:02.524 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:19:02.524 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:19:02.525 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0237278 s, 44.2 MB/s 2021-02-10T11:19:02.525 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:02.526 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:02.526 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:02.860 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:19:02.861 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:02.861 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348589/8d559078-3925-4ae1-aef0-738f2ffdf39d/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:03.236 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:19:03.236 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:19:03.236 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0230188 s, 45.6 MB/s 2021-02-10T11:19:03.237 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000348590 2021-02-10T11:19:03.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:03.655 7f4d597de700 1 Processor -- start 2021-02-10T11:19:03.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:03.655 7f4d597de700 1 -- start start 2021-02-10T11:19:03.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:03.655 7f4d597de700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4d541b0870 0x7f4d540f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:03.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:03.655 7f4d597de700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f4d540efe00 0x7f4d540f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:03.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:03.655 7f4d597de700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d541b0c70 con 0x7f4d541b0470 2021-02-10T11:19:03.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:03.655 7f4d597de700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d540f0b50 con 0x7f4d540efe00 2021-02-10T11:19:03.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:03.655 7f4d597de700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d540f0820 con 0x7f4d541b0870 2021-02-10T11:19:03.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:03.655 7f4d53d75700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f4d540efe00 0x7f4d540f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:03.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:03.655 7f4d53d75700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f4d540efe00 0x7f4d540f0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:44410/0 (socket says 172.21.2.112:44410) 2021-02-10T11:19:03.669 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:03.655 7f4d53d75700 1 -- 172.21.2.112:0/4198043525 learned_addr learned my addr 172.21.2.112:0/4198043525 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:19:03.669 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:03.656 7f4d53d75700 1 -- 172.21.2.112:0/4198043525 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4d541b0870 msgr2=0x7f4d540f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:19:03.669 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:03.656 7f4d52d73700 1 --2- 172.21.2.112:0/4198043525 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4d541b0870 0x7f4d540f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:03.669 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:03.656 7f4d53d75700 1 --2- 172.21.2.112:0/4198043525 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4d541b0870 0x7f4d540f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:03.669 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:03.656 7f4d53d75700 1 -- 172.21.2.112:0/4198043525 >> v1:172.21.2.120:6789/0 conn(0x7f4d541b0470 legacy=0x7f4d540ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:03.669 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:03.656 7f4d53d75700 1 -- 172.21.2.112:0/4198043525 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d540e52e0 con 0x7f4d540efe00 2021-02-10T11:19:03.669 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:03.656 7f4d52d73700 1 --2- 172.21.2.112:0/4198043525 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4d541b0870 0x7f4d540f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:19:03.669 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:03.656 7f4d53d75700 1 --2- 172.21.2.112:0/4198043525 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4d540efe00 0x7f4d540f0200 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 rx=0x7f4d4c006f00 tx=0x7f4d4c009f10).ready entity=mon.0 client_cookie=4970e59780203639 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:19:03.669 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:03.656 7f4d52572700 1 -- 172.21.2.112:0/4198043525 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4d4c00b3b0 con 0x7f4d540efe00 2021-02-10T11:19:03.669 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:03.656 7f4d52572700 1 -- 172.21.2.112:0/4198043525 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d4c00b730 con 0x7f4d540efe00 2021-02-10T11:19:03.669 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:03.656 7f4d52572700 1 -- 172.21.2.112:0/4198043525 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4d4c00bc10 con 0x7f4d540efe00 2021-02-10T11:19:03.669 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:03.656 7f4d597de700 1 -- 172.21.2.112:0/4198043525 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4d540efe00 msgr2=0x7f4d540f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:03.669 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:03.656 7f4d597de700 1 --2- 172.21.2.112:0/4198043525 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4d540efe00 0x7f4d540f0200 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 rx=0x7f4d4c006f00 tx=0x7f4d4c009f10).stop 2021-02-10T11:19:03.669 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:03.657 7f4d597de700 1 -- 172.21.2.112:0/4198043525 shutdown_connections 2021-02-10T11:19:03.670 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:03.657 7f4d597de700 1 --2- 172.21.2.112:0/4198043525 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4d540efe00 0x7f4d540f0200 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:03.670 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:03.657 7f4d597de700 1 --2- 172.21.2.112:0/4198043525 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4d541b0870 0x7f4d540f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:03.670 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:03.657 7f4d597de700 1 -- 172.21.2.112:0/4198043525 >> 172.21.2.112:0/4198043525 conn(0x7f4d540ce3e0 msgr2=0x7f4d540e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:19:03.670 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:03.657 7f4d597de700 1 -- 172.21.2.112:0/4198043525 shutdown_connections 2021-02-10T11:19:03.670 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:03.657 7f4d597de700 1 -- 172.21.2.112:0/4198043525 wait complete. 2021-02-10T11:19:03.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:03.671 7f4d597de700 1 Processor -- start 2021-02-10T11:19:03.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:03.672 7f4d597de700 1 -- start start 2021-02-10T11:19:03.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:03.672 7f4d597de700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4d540bf910 0x7f4d540bfd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:03.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:03.672 7f4d597de700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4d540c0330 0x7f4d540c0730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:03.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:03.672 7f4d597de700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4d540c13d0 0x7f4d540c17d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:03.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:03.672 7f4d597de700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d54109930 con 0x7f4d540c13d0 2021-02-10T11:19:03.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:03.672 7f4d597de700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d54109b50 con 0x7f4d540bf910 2021-02-10T11:19:03.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:03.672 7f4d597de700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d540c1df0 con 0x7f4d540c0330 2021-02-10T11:19:03.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:03.672 7f4d53d75700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4d540c13d0 0x7f4d540c17d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:03.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:03.672 7f4d53d75700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4d540c13d0 0x7f4d540c17d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:44412/0 (socket says 172.21.2.112:44412) 2021-02-10T11:19:03.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:03.672 7f4d53d75700 1 -- 172.21.2.112:0/2162861297 learned_addr learned my addr 172.21.2.112:0/2162861297 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:19:03.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:03.672 7f4d53d75700 1 -- 172.21.2.112:0/2162861297 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4d540c0330 msgr2=0x7f4d540c0730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:03.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:03.672 7f4d53d75700 1 --2- 172.21.2.112:0/2162861297 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4d540c0330 0x7f4d540c0730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:19:03.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:03.673 7f4d53d75700 1 -- 172.21.2.112:0/2162861297 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4d540bf910 msgr2=0x7f4d540bfd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:03.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:03.673 7f4d53d75700 1 --2- 172.21.2.112:0/2162861297 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4d540bf910 0x7f4d540bfd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:19:03.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:03.673 7f4d53d75700 1 -- 172.21.2.112:0/2162861297 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d540c1fd0 con 0x7f4d540c13d0 2021-02-10T11:19:03.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:03.673 7f4d53d75700 1 --2- 172.21.2.112:0/2162861297 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4d540c13d0 0x7f4d540c17d0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7f4d4c009390 tx=0x7f4d4c00bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:19:03.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:03.673 7f4d3bfff700 1 -- 172.21.2.112:0/2162861297 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4d4c003030 con 0x7f4d540c13d0 2021-02-10T11:19:03.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:03.673 7f4d3bfff700 1 -- 172.21.2.112:0/2162861297 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d4c0068e0 con 0x7f4d540c13d0 2021-02-10T11:19:03.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:03.673 7f4d597de700 1 -- 172.21.2.112:0/2162861297 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d540c1fd0 con 0x7f4d540c13d0 2021-02-10T11:19:03.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:03.673 7f4d3bfff700 1 -- 172.21.2.112:0/2162861297 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4d4c00c570 con 0x7f4d540c13d0 2021-02-10T11:19:03.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:03.673 7f4d597de700 1 -- 172.21.2.112:0/2162861297 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d540c2560 con 0x7f4d540c13d0 2021-02-10T11:19:03.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:03.673 7f4d3bfff700 1 -- 172.21.2.112:0/2162861297 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4d4c005820 con 0x7f4d540c13d0 2021-02-10T11:19:03.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:03.673 7f4d3bfff700 1 --2- 172.21.2.112:0/2162861297 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4d340186c0 0x7f4d3401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:03.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:03.673 7f4d3bfff700 1 -- 172.21.2.112:0/2162861297 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f4d4c0068e0 con 0x7f4d540c13d0 2021-02-10T11:19:03.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:03.674 7f4d53574700 1 --2- 172.21.2.112:0/2162861297 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4d340186c0 0x7f4d3401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:03.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:03.674 7f4d53574700 1 --2- 172.21.2.112:0/2162861297 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4d340186c0 0x7f4d3401abc0 crc :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:19:03.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:03.705 7f4d597de700 1 -- 172.21.2.112:0/2162861297 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d540f0b50 con 0x7f4d540c13d0 2021-02-10T11:19:03.704 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:03.708 7f4d3bfff700 1 -- 172.21.2.112:0/2162861297 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4d4c009450 con 0x7f4d540c13d0 2021-02-10T11:19:03.832 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:03.835 7f4d597de700 1 -- 172.21.2.112:0/2162861297 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348590", "target": ["mgr", ""]}) v1 -- 0x7f4d540c2560 con 0x7f4d340186c0 2021-02-10T11:19:03.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:03.862 7f4d3bfff700 1 -- 172.21.2.112:0/2162861297 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f4d44006a10 con 0x7f4d340186c0 2021-02-10T11:19:03.865 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:03.868 7f4d597de700 1 -- 172.21.2.112:0/2162861297 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4d340186c0 msgr2=0x7f4d3401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:03.865 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:03.868 7f4d597de700 1 --2- 172.21.2.112:0/2162861297 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4d340186c0 0x7f4d3401abc0 crc :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:03.865 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:03.868 7f4d597de700 1 -- 172.21.2.112:0/2162861297 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4d540c13d0 msgr2=0x7f4d540c17d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:03.865 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:03.868 7f4d597de700 1 --2- 172.21.2.112:0/2162861297 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4d540c13d0 0x7f4d540c17d0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7f4d4c009390 tx=0x7f4d4c00bd40).stop 2021-02-10T11:19:03.866 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:03.868 7f4d597de700 1 -- 172.21.2.112:0/2162861297 shutdown_connections 2021-02-10T11:19:03.866 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:03.868 7f4d597de700 1 --2- 172.21.2.112:0/2162861297 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4d340186c0 0x7f4d3401abc0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:03.866 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:03.868 7f4d597de700 1 --2- 172.21.2.112:0/2162861297 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4d540c13d0 0x7f4d540c17d0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:03.866 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:03.868 7f4d597de700 1 --2- 172.21.2.112:0/2162861297 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4d540c0330 0x7f4d540c0730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:19:03.866 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:03.868 7f4d597de700 1 --2- 172.21.2.112:0/2162861297 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4d540bf910 0x7f4d540bfd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:19:03.866 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:03.868 7f4d597de700 1 -- 172.21.2.112:0/2162861297 >> 172.21.2.112:0/2162861297 conn(0x7f4d540bf4e0 msgr2=0x7f4d54191190 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:19:03.866 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:03.869 7f4d597de700 1 -- 172.21.2.112:0/2162861297 shutdown_connections 2021-02-10T11:19:03.866 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:03.869 7f4d597de700 1 -- 172.21.2.112:0/2162861297 wait complete. 2021-02-10T11:19:03.873 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000348590 2021-02-10T11:19:04.307 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:04.290 7efefce92700 1 Processor -- start 2021-02-10T11:19:04.307 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:04.290 7efefce92700 1 -- start start 2021-02-10T11:19:04.307 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:04.291 7efefce92700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7efef80f05f0 0x7efef80f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:04.307 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:04.291 7efefce92700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efef81b0c70 con 0x7efef81b0470 2021-02-10T11:19:04.307 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:04.291 7efefce92700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efef80f0b50 con 0x7efef81b0870 2021-02-10T11:19:04.307 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:04.291 7efefce92700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efef80e5ea0 con 0x7efef80f05f0 2021-02-10T11:19:04.307 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:04.291 7efef6b13700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7efef81b0470 0x7efef80ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:43882/0 (socket says 172.21.2.112:43882) 2021-02-10T11:19:04.307 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:04.291 7efef6b13700 1 -- 172.21.2.112:0/1960293374 learned_addr learned my addr 172.21.2.112:0/1960293374 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:19:04.307 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:04.291 7efef7314700 1 --2- 172.21.2.112:0/1960293374 >> v2:172.21.2.112:3300/0 conn(0x7efef80f05f0 0x7efef80f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:04.307 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:04.291 7efef5b11700 1 -- 172.21.2.112:0/1960293374 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2261160417 0 0) 0x7efee0000ce0 con 0x7efef81b0470 2021-02-10T11:19:04.308 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:04.291 7efef5b11700 1 -- 172.21.2.112:0/1960293374 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efee4002fe0 con 0x7efef81b0470 2021-02-10T11:19:04.308 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:04.291 7efef7314700 1 -- 172.21.2.112:0/1960293374 >> v1:172.21.2.120:6789/0 conn(0x7efef81b0870 legacy=0x7efef80efe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:04.308 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:04.291 7efef7314700 1 -- 172.21.2.112:0/1960293374 >> v1:172.21.2.112:6789/0 conn(0x7efef81b0470 legacy=0x7efef80ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:04.308 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:04.291 7efef7314700 1 -- 172.21.2.112:0/1960293374 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efef80e52e0 con 0x7efef80f05f0 2021-02-10T11:19:04.308 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:04.292 7efef7314700 1 --2- 172.21.2.112:0/1960293374 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efef80f05f0 0x7efef80f20e0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7efef0006e30 tx=0x7efef0009e60).ready entity=mon.0 client_cookie=963bacb676a3b5e6 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:19:04.308 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:04.292 7efef5b11700 1 -- 172.21.2.112:0/1960293374 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efef0006a50 con 0x7efef80f05f0 2021-02-10T11:19:04.308 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:04.292 7efef5b11700 1 -- 172.21.2.112:0/1960293374 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efef000b300 con 0x7efef80f05f0 2021-02-10T11:19:04.308 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:04.292 7efef5b11700 1 -- 172.21.2.112:0/1960293374 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efef000b820 con 0x7efef80f05f0 2021-02-10T11:19:04.308 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:04.292 7efefce92700 1 -- 172.21.2.112:0/1960293374 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efef80f05f0 msgr2=0x7efef80f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:04.308 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:04.292 7efefce92700 1 --2- 172.21.2.112:0/1960293374 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efef80f05f0 0x7efef80f20e0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7efef0006e30 tx=0x7efef0009e60).stop 2021-02-10T11:19:04.308 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:04.292 7efefce92700 1 -- 172.21.2.112:0/1960293374 shutdown_connections 2021-02-10T11:19:04.308 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:04.292 7efefce92700 1 --2- 172.21.2.112:0/1960293374 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efef80f05f0 0x7efef80f20e0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:04.308 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:04.292 7efefce92700 1 -- 172.21.2.112:0/1960293374 >> 172.21.2.112:0/1960293374 conn(0x7efef80ce3e0 msgr2=0x7efef80e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:19:04.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:04.293 7efefce92700 1 -- 172.21.2.112:0/1960293374 shutdown_connections 2021-02-10T11:19:04.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:04.293 7efefce92700 1 -- 172.21.2.112:0/1960293374 wait complete. 2021-02-10T11:19:04.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:04.310 7efefce92700 1 Processor -- start 2021-02-10T11:19:04.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:04.310 7efefce92700 1 -- start start 2021-02-10T11:19:04.311 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:04.310 7efefce92700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efef80c36a0 0x7efef80c2b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:04.311 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:04.310 7efefce92700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7efef80c3190 0x7efef80b1d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:04.311 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:04.310 7efefce92700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efef80b2370 0x7efef80b2d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:04.311 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:04.310 7efefce92700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efef80b27c0 con 0x7efef80b2370 2021-02-10T11:19:04.311 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:04.310 7efefce92700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efef80b3360 con 0x7efef80c3190 2021-02-10T11:19:04.311 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:04.310 7efefce92700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efef80b3540 con 0x7efef80c36a0 2021-02-10T11:19:04.311 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:04.311 7efef7314700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efef80b2370 0x7efef80b2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:04.311 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:04.311 7efef7314700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efef80b2370 0x7efef80b2d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:44428/0 (socket says 172.21.2.112:44428) 2021-02-10T11:19:04.311 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:04.311 7efef7314700 1 -- 172.21.2.112:0/4036376466 learned_addr learned my addr 172.21.2.112:0/4036376466 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:19:04.311 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:04.311 7efef6312700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7efef80c3190 0x7efef80b1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:04.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:04.311 7efef6b13700 1 --2- 172.21.2.112:0/4036376466 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efef80c36a0 0x7efef80c2b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:04.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:04.311 7efef7314700 1 -- 172.21.2.112:0/4036376466 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efef80c36a0 msgr2=0x7efef80c2b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:04.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:04.311 7efef7314700 1 --2- 172.21.2.112:0/4036376466 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efef80c36a0 0x7efef80c2b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:04.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:04.311 7efef7314700 1 -- 172.21.2.112:0/4036376466 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7efef80c3190 msgr2=0x7efef80b1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:04.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:04.311 7efef7314700 1 --2- 172.21.2.112:0/4036376466 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7efef80c3190 0x7efef80b1d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:04.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:04.311 7efef7314700 1 -- 172.21.2.112:0/4036376466 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efef80b3760 con 0x7efef80b2370 2021-02-10T11:19:04.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:04.311 7efef6312700 1 --2- 172.21.2.112:0/4036376466 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7efef80c3190 0x7efef80b1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:19:04.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:04.311 7efef6b13700 1 --2- 172.21.2.112:0/4036376466 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efef80c36a0 0x7efef80c2b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:19:04.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:04.311 7efef7314700 1 --2- 172.21.2.112:0/4036376466 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efef80b2370 0x7efef80b2d40 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 rx=0x7efef0001070 tx=0x7efef000bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:19:04.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:04.311 7efeef7fe700 1 -- 172.21.2.112:0/4036376466 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efef0001270 con 0x7efef80b2370 2021-02-10T11:19:04.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:04.312 7efeef7fe700 1 -- 172.21.2.112:0/4036376466 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efef0000d70 con 0x7efef80b2370 2021-02-10T11:19:04.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:04.312 7efefce92700 1 -- 172.21.2.112:0/4036376466 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efef80b3760 con 0x7efef80b2370 2021-02-10T11:19:04.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:04.312 7efeef7fe700 1 -- 172.21.2.112:0/4036376466 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efef0001720 con 0x7efef80b2370 2021-02-10T11:19:04.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:04.312 7efefce92700 1 -- 172.21.2.112:0/4036376466 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efef80b3760 con 0x7efef80b2370 2021-02-10T11:19:04.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:04.312 7efeef7fe700 1 -- 172.21.2.112:0/4036376466 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7efef00093d0 con 0x7efef80b2370 2021-02-10T11:19:04.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:04.312 7efeef7fe700 1 --2- 172.21.2.112:0/4036376466 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7efedc018730 0x7efedc01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:04.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:04.312 7efeef7fe700 1 -- 172.21.2.112:0/4036376466 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7efef00329c0 con 0x7efef80b2370 2021-02-10T11:19:04.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:04.312 7efef6b13700 1 --2- 172.21.2.112:0/4036376466 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7efedc018730 0x7efedc01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:04.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:04.313 7efef6b13700 1 --2- 172.21.2.112:0/4036376466 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7efedc018730 0x7efedc01ac10 crc :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:19:04.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:04.341 7efefce92700 1 -- 172.21.2.112:0/4036376466 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efef80edd10 con 0x7efef80b2370 2021-02-10T11:19:04.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:04.343 7efeef7fe700 1 -- 172.21.2.112:0/4036376466 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7efef0003020 con 0x7efef80b2370 2021-02-10T11:19:04.473 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:04.476 7efefce92700 1 -- 172.21.2.112:0/4036376466 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348590", "target": ["mgr", ""]}) v1 -- 0x7efef80edd10 con 0x7efedc018730 2021-02-10T11:19:04.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:04.480 7efeef7fe700 1 -- 172.21.2.112:0/4036376466 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7efee0006bc0 con 0x7efedc018730 2021-02-10T11:19:04.480 INFO:teuthology.orchestra.run.gibba012.stdout:/volumes/_nogroup/subvolume_0000000000348590/bc6264d9-f2af-4edd-9f13-98a51fb5c64b 2021-02-10T11:19:04.482 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:04.485 7efefce92700 1 -- 172.21.2.112:0/4036376466 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7efedc018730 msgr2=0x7efedc01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:04.483 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:04.485 7efefce92700 1 --2- 172.21.2.112:0/4036376466 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7efedc018730 0x7efedc01ac10 crc :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:04.483 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:04.486 7efefce92700 1 -- 172.21.2.112:0/4036376466 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efef80b2370 msgr2=0x7efef80b2d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:04.483 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:04.486 7efefce92700 1 --2- 172.21.2.112:0/4036376466 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efef80b2370 0x7efef80b2d40 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 rx=0x7efef0001070 tx=0x7efef000bd40).stop 2021-02-10T11:19:04.483 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:04.486 7efefce92700 1 -- 172.21.2.112:0/4036376466 shutdown_connections 2021-02-10T11:19:04.483 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:04.486 7efefce92700 1 --2- 172.21.2.112:0/4036376466 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7efedc018730 0x7efedc01ac10 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:04.483 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:04.486 7efefce92700 1 --2- 172.21.2.112:0/4036376466 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7efef80c3190 0x7efef80b1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:04.483 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:04.486 7efefce92700 1 --2- 172.21.2.112:0/4036376466 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efef80b2370 0x7efef80b2d40 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:04.483 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:04.486 7efefce92700 1 --2- 172.21.2.112:0/4036376466 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efef80c36a0 0x7efef80c2b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:04.483 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:04.486 7efefce92700 1 -- 172.21.2.112:0/4036376466 >> 172.21.2.112:0/4036376466 conn(0x7efef809f630 msgr2=0x7efef80a0130 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:19:04.483 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:04.486 7efefce92700 1 -- 172.21.2.112:0/4036376466 shutdown_connections 2021-02-10T11:19:04.483 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:04.486 7efefce92700 1 -- 172.21.2.112:0/4036376466 wait complete. 2021-02-10T11:19:04.490 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000348590 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000348590/bc6264d9-f2af-4edd-9f13-98a51fb5c64b 2021-02-10T11:19:04.491 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:04.491 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:04.491 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:04.826 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:19:04.827 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:04.827 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348590/bc6264d9-f2af-4edd-9f13-98a51fb5c64b/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:04.837 DEBUG:teuthology.orchestra.run.gibba012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:19:04.873 DEBUG:teuthology.orchestra.run.gibba020:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:19:05.204 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:19:05.204 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:19:05.204 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0232962 s, 45.0 MB/s 2021-02-10T11:19:05.205 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:05.205 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:05.205 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:05.580 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:19:05.580 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:05.581 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348590/bc6264d9-f2af-4edd-9f13-98a51fb5c64b/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:05.955 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:19:05.955 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:19:05.955 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0253692 s, 41.3 MB/s 2021-02-10T11:19:05.956 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:05.956 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:05.956 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:06.290 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:19:06.291 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:06.291 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348590/bc6264d9-f2af-4edd-9f13-98a51fb5c64b/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:06.667 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:19:06.667 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:19:06.667 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0261405 s, 40.1 MB/s 2021-02-10T11:19:06.668 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:06.669 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:06.669 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:07.003 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:19:07.003 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:07.003 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348590/bc6264d9-f2af-4edd-9f13-98a51fb5c64b/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:07.378 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:19:07.378 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:19:07.379 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0237169 s, 44.2 MB/s 2021-02-10T11:19:07.379 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:07.380 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:07.380 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:07.714 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:19:07.714 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:07.715 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348590/bc6264d9-f2af-4edd-9f13-98a51fb5c64b/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:08.091 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:19:08.091 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:19:08.091 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0248455 s, 42.2 MB/s 2021-02-10T11:19:08.092 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:08.093 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:08.093 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:08.428 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:19:08.428 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:08.428 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348590/bc6264d9-f2af-4edd-9f13-98a51fb5c64b/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:08.805 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:19:08.805 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:19:08.806 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.025054 s, 41.9 MB/s 2021-02-10T11:19:08.807 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:08.808 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:08.808 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:09.146 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:19:09.147 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:09.147 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348590/bc6264d9-f2af-4edd-9f13-98a51fb5c64b/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:09.526 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:19:09.526 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:19:09.526 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0248457 s, 42.2 MB/s 2021-02-10T11:19:09.527 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:09.528 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:09.528 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:09.865 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:19:09.865 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:09.865 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348590/bc6264d9-f2af-4edd-9f13-98a51fb5c64b/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:10.240 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:19:10.240 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:19:10.240 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0238103 s, 44.0 MB/s 2021-02-10T11:19:10.241 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:10.241 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:10.242 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:10.577 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:19:10.577 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:10.577 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348590/bc6264d9-f2af-4edd-9f13-98a51fb5c64b/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:10.953 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:19:10.953 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:19:10.953 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0236608 s, 44.3 MB/s 2021-02-10T11:19:10.954 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:10.954 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:10.954 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:11.290 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:19:11.290 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:11.291 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348590/bc6264d9-f2af-4edd-9f13-98a51fb5c64b/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:11.665 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:19:11.665 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:19:11.665 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0228903 s, 45.8 MB/s 2021-02-10T11:19:11.666 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000348591 2021-02-10T11:19:12.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.082 7f5d4e396700 1 Processor -- start 2021-02-10T11:19:12.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.082 7f5d4e396700 1 -- start start 2021-02-10T11:19:12.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.083 7f5d4e396700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5d481b0870 0x7f5d480f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:12.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.083 7f5d4e396700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f5d480efe00 0x7f5d480f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:12.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.083 7f5d4e396700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d481b0c70 con 0x7f5d481b0470 2021-02-10T11:19:12.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.083 7f5d4e396700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d480f0b50 con 0x7f5d480efe00 2021-02-10T11:19:12.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.083 7f5d4e396700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d480f0820 con 0x7f5d481b0870 2021-02-10T11:19:12.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.083 7f5d4c9b3700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f5d480efe00 0x7f5d480f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:12.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.083 7f5d4c9b3700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f5d480efe00 0x7f5d480f0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:44436/0 (socket says 172.21.2.112:44436) 2021-02-10T11:19:12.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.083 7f5d4c9b3700 1 -- 172.21.2.112:0/2842581593 learned_addr learned my addr 172.21.2.112:0/2842581593 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:19:12.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.083 7f5d47fff700 1 --1- 172.21.2.112:0/2842581593 >> v1:172.21.2.112:6789/0 conn(0x7f5d481b0470 0x7f5d480ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:43896/0 (socket says 172.21.2.112:43896) 2021-02-10T11:19:12.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.083 7f5d4c9b3700 1 -- 172.21.2.112:0/2842581593 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5d481b0870 msgr2=0x7f5d480f20e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:19:12.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.083 7f5d4c9b3700 1 --2- 172.21.2.112:0/2842581593 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5d481b0870 0x7f5d480f20e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:19:12.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.083 7f5d4c9b3700 1 -- 172.21.2.112:0/2842581593 >> v1:172.21.2.112:6789/0 conn(0x7f5d481b0470 legacy=0x7f5d480ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:12.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.083 7f5d4c9b3700 1 -- 172.21.2.112:0/2842581593 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d480e52e0 con 0x7f5d480efe00 2021-02-10T11:19:12.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.083 7f5d4c9b3700 1 --2- 172.21.2.112:0/2842581593 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5d480efe00 0x7f5d480f0200 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 rx=0x7f5d40006b80 tx=0x7f5d40009b90).ready entity=mon.0 client_cookie=af9c8c475f0be47e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:19:12.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.083 7f5d46ffd700 1 -- 172.21.2.112:0/2842581593 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5d400022e0 con 0x7f5d480efe00 2021-02-10T11:19:12.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.083 7f5d46ffd700 1 -- 172.21.2.112:0/2842581593 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d40002560 con 0x7f5d480efe00 2021-02-10T11:19:12.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.083 7f5d46ffd700 1 -- 172.21.2.112:0/2842581593 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5d40002a80 con 0x7f5d480efe00 2021-02-10T11:19:12.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.083 7f5d4e396700 1 -- 172.21.2.112:0/2842581593 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5d480efe00 msgr2=0x7f5d480f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:12.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.083 7f5d4e396700 1 --2- 172.21.2.112:0/2842581593 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5d480efe00 0x7f5d480f0200 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 rx=0x7f5d40006b80 tx=0x7f5d40009b90).stop 2021-02-10T11:19:12.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.084 7f5d4e396700 1 -- 172.21.2.112:0/2842581593 shutdown_connections 2021-02-10T11:19:12.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.084 7f5d4e396700 1 --2- 172.21.2.112:0/2842581593 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5d480efe00 0x7f5d480f0200 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:12.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.084 7f5d4e396700 1 --2- 172.21.2.112:0/2842581593 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5d481b0870 0x7f5d480f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:19:12.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.084 7f5d4e396700 1 -- 172.21.2.112:0/2842581593 >> 172.21.2.112:0/2842581593 conn(0x7f5d480ce3e0 msgr2=0x7f5d480e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:19:12.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.084 7f5d4e396700 1 -- 172.21.2.112:0/2842581593 shutdown_connections 2021-02-10T11:19:12.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.084 7f5d4e396700 1 -- 172.21.2.112:0/2842581593 wait complete. 2021-02-10T11:19:12.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.099 7f5d4e396700 1 Processor -- start 2021-02-10T11:19:12.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.099 7f5d4e396700 1 -- start start 2021-02-10T11:19:12.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.099 7f5d4e396700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5d480b20b0 0x7f5d480ac970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:12.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.099 7f5d4e396700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5d480b24b0 0x7f5d48191dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:12.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.099 7f5d4e396700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5d480acf90 0x7f5d481923e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:12.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.099 7f5d4e396700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d480b1b30 con 0x7f5d480b20b0 2021-02-10T11:19:12.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.099 7f5d4e396700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d481a04e0 con 0x7f5d480acf90 2021-02-10T11:19:12.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.099 7f5d4e396700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d481a06c0 con 0x7f5d480b24b0 2021-02-10T11:19:12.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.099 7f5d47fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5d480b20b0 0x7f5d480ac970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:12.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.099 7f5d47fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5d480b20b0 0x7f5d480ac970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:44438/0 (socket says 172.21.2.112:44438) 2021-02-10T11:19:12.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.099 7f5d47fff700 1 -- 172.21.2.112:0/333286362 learned_addr learned my addr 172.21.2.112:0/333286362 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:19:12.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.100 7f5d47fff700 1 -- 172.21.2.112:0/333286362 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5d480b24b0 msgr2=0x7f5d48191dc0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:19:12.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.100 7f5d47fff700 1 --2- 172.21.2.112:0/333286362 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5d480b24b0 0x7f5d48191dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:19:12.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.100 7f5d47fff700 1 -- 172.21.2.112:0/333286362 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5d480acf90 msgr2=0x7f5d481923e0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:19:12.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.100 7f5d47fff700 1 --2- 172.21.2.112:0/333286362 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5d480acf90 0x7f5d481923e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:19:12.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.100 7f5d47fff700 1 -- 172.21.2.112:0/333286362 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d48192be0 con 0x7f5d480b20b0 2021-02-10T11:19:12.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.100 7f5d47fff700 1 --2- 172.21.2.112:0/333286362 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5d480b20b0 0x7f5d480ac970 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7f5d3800a920 tx=0x7f5d3800acc0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:19:12.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.100 7f5d44ff9700 1 -- 172.21.2.112:0/333286362 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5d380070d0 con 0x7f5d480b20b0 2021-02-10T11:19:12.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.100 7f5d44ff9700 1 -- 172.21.2.112:0/333286362 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d38007310 con 0x7f5d480b20b0 2021-02-10T11:19:12.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.100 7f5d44ff9700 1 -- 172.21.2.112:0/333286362 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5d380046b0 con 0x7f5d480b20b0 2021-02-10T11:19:12.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.100 7f5d4e396700 1 -- 172.21.2.112:0/333286362 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d48192be0 con 0x7f5d480b20b0 2021-02-10T11:19:12.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.100 7f5d4e396700 1 -- 172.21.2.112:0/333286362 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d48193170 con 0x7f5d480b20b0 2021-02-10T11:19:12.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.100 7f5d44ff9700 1 -- 172.21.2.112:0/333286362 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5d3800ae00 con 0x7f5d480b20b0 2021-02-10T11:19:12.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.100 7f5d44ff9700 1 --2- 172.21.2.112:0/333286362 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5d2c018730 0x7f5d2c01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:12.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.100 7f5d44ff9700 1 -- 172.21.2.112:0/333286362 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f5d380329b0 con 0x7f5d480b20b0 2021-02-10T11:19:12.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.101 7f5d477fe700 1 --2- 172.21.2.112:0/333286362 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5d2c018730 0x7f5d2c01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:12.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.101 7f5d477fe700 1 --2- 172.21.2.112:0/333286362 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5d2c018730 0x7f5d2c01ac10 crc :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:19:12.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.133 7f5d4e396700 1 -- 172.21.2.112:0/333286362 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d48105af0 con 0x7f5d480b20b0 2021-02-10T11:19:12.131 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.135 7f5d44ff9700 1 -- 172.21.2.112:0/333286362 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5d38001150 con 0x7f5d480b20b0 2021-02-10T11:19:12.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.263 7f5d4e396700 1 -- 172.21.2.112:0/333286362 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348591", "target": ["mgr", ""]}) v1 -- 0x7f5d480b1b30 con 0x7f5d2c018730 2021-02-10T11:19:12.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.289 7f5d44ff9700 1 -- 172.21.2.112:0/333286362 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f5d300009b0 con 0x7f5d2c018730 2021-02-10T11:19:12.293 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.296 7f5d4e396700 1 -- 172.21.2.112:0/333286362 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5d2c018730 msgr2=0x7f5d2c01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:12.293 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.296 7f5d4e396700 1 --2- 172.21.2.112:0/333286362 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5d2c018730 0x7f5d2c01ac10 crc :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:12.293 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.296 7f5d4e396700 1 -- 172.21.2.112:0/333286362 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5d480b20b0 msgr2=0x7f5d480ac970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:12.293 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.296 7f5d4e396700 1 --2- 172.21.2.112:0/333286362 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5d480b20b0 0x7f5d480ac970 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7f5d3800a920 tx=0x7f5d3800acc0).stop 2021-02-10T11:19:12.293 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.296 7f5d4e396700 1 -- 172.21.2.112:0/333286362 shutdown_connections 2021-02-10T11:19:12.293 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.296 7f5d4e396700 1 --2- 172.21.2.112:0/333286362 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5d2c018730 0x7f5d2c01ac10 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:12.293 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.296 7f5d4e396700 1 --2- 172.21.2.112:0/333286362 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5d480acf90 0x7f5d481923e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:19:12.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.296 7f5d4e396700 1 --2- 172.21.2.112:0/333286362 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5d480b24b0 0x7f5d48191dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:19:12.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.296 7f5d4e396700 1 --2- 172.21.2.112:0/333286362 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5d480b20b0 0x7f5d480ac970 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:12.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.296 7f5d4e396700 1 -- 172.21.2.112:0/333286362 >> 172.21.2.112:0/333286362 conn(0x7f5d480bf4e0 msgr2=0x7f5d480be3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:19:12.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.297 7f5d4e396700 1 -- 172.21.2.112:0/333286362 shutdown_connections 2021-02-10T11:19:12.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.297 7f5d4e396700 1 -- 172.21.2.112:0/333286362 wait complete. 2021-02-10T11:19:12.301 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000348591 2021-02-10T11:19:12.732 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.718 7f8550a16700 1 Processor -- start 2021-02-10T11:19:12.732 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.719 7f8550a16700 1 -- start start 2021-02-10T11:19:12.732 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.719 7f8550a16700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f854c1b0870 0x7f854c0f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:12.732 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.719 7f8550a16700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f854c0efe00 0x7f854c0f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:12.732 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.719 7f8550a16700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f854c1b0c70 con 0x7f854c1b0470 2021-02-10T11:19:12.732 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.719 7f8550a16700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f854c0f0b50 con 0x7f854c1b0870 2021-02-10T11:19:12.732 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.719 7f8550a16700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f854c0f0820 con 0x7f854c0efe00 2021-02-10T11:19:12.732 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.719 7f854a61c700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f854c1b0470 0x7f854c0ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:43908/0 (socket says 172.21.2.112:43908) 2021-02-10T11:19:12.732 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.719 7f854a61c700 1 -- 172.21.2.112:0/1409417979 learned_addr learned my addr 172.21.2.112:0/1409417979 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:19:12.733 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.719 7f854ae1d700 1 --2- 172.21.2.112:0/1409417979 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f854c0efe00 0x7f854c0f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:12.733 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.719 7f8549e1b700 1 --2- 172.21.2.112:0/1409417979 >> v2:172.21.2.120:3300/0 conn(0x7f854c1b0870 0x7f854c0f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:12.733 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.720 7f854961a700 1 -- 172.21.2.112:0/1409417979 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 64049330 0 0) 0x7f853c0010b0 con 0x7f854c1b0470 2021-02-10T11:19:12.733 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.720 7f854961a700 1 -- 172.21.2.112:0/1409417979 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8538002fe0 con 0x7f854c1b0470 2021-02-10T11:19:12.733 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.720 7f8549e1b700 1 -- 172.21.2.112:0/1409417979 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f854c0efe00 msgr2=0x7f854c0f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:19:12.733 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.720 7f8549e1b700 1 --2- 172.21.2.112:0/1409417979 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f854c0efe00 0x7f854c0f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:12.733 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.720 7f8549e1b700 1 -- 172.21.2.112:0/1409417979 >> v1:172.21.2.112:6789/0 conn(0x7f854c1b0470 legacy=0x7f854c0ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:12.733 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.720 7f8549e1b700 1 -- 172.21.2.112:0/1409417979 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f854c0e52e0 con 0x7f854c1b0870 2021-02-10T11:19:12.733 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.720 7f854ae1d700 1 --2- 172.21.2.112:0/1409417979 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f854c0efe00 0x7f854c0f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:19:12.733 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.720 7f8549e1b700 1 --2- 172.21.2.112:0/1409417979 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f854c1b0870 0x7f854c0f20e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f8534006650 tx=0x7f8534009a80).ready entity=mon.1 client_cookie=6f642d18f60a8ebb server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:19:12.733 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.720 7f854961a700 1 -- 172.21.2.112:0/1409417979 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f853400c0c0 con 0x7f854c1b0870 2021-02-10T11:19:12.733 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.720 7f854961a700 1 -- 172.21.2.112:0/1409417979 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f853400c340 con 0x7f854c1b0870 2021-02-10T11:19:12.733 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.720 7f854961a700 1 -- 172.21.2.112:0/1409417979 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f853400c860 con 0x7f854c1b0870 2021-02-10T11:19:12.733 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.721 7f8550a16700 1 -- 172.21.2.112:0/1409417979 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f854c1b0870 msgr2=0x7f854c0f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:12.734 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.721 7f8550a16700 1 --2- 172.21.2.112:0/1409417979 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f854c1b0870 0x7f854c0f20e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f8534006650 tx=0x7f8534009a80).stop 2021-02-10T11:19:12.734 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.721 7f8550a16700 1 -- 172.21.2.112:0/1409417979 shutdown_connections 2021-02-10T11:19:12.734 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.721 7f8550a16700 1 --2- 172.21.2.112:0/1409417979 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f854c0efe00 0x7f854c0f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:12.734 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.721 7f8550a16700 1 --2- 172.21.2.112:0/1409417979 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f854c1b0870 0x7f854c0f20e0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:12.734 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.721 7f8550a16700 1 -- 172.21.2.112:0/1409417979 >> 172.21.2.112:0/1409417979 conn(0x7f854c0ce3e0 msgr2=0x7f854c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:19:12.734 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.721 7f8550a16700 1 -- 172.21.2.112:0/1409417979 shutdown_connections 2021-02-10T11:19:12.734 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.721 7f8550a16700 1 -- 172.21.2.112:0/1409417979 wait complete. 2021-02-10T11:19:12.735 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.736 7f8550a16700 1 Processor -- start 2021-02-10T11:19:12.736 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.736 7f8550a16700 1 -- start start 2021-02-10T11:19:12.736 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.736 7f8550a16700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f854c09f140 0x7f854c0c1940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:12.736 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.736 7f8550a16700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f854c0c1f60 0x7f854c09e300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:12.736 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.736 7f8550a16700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f854c09e920 0x7f854c0c2960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:12.736 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.736 7f8550a16700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f854c0c23e0 con 0x7f854c09e920 2021-02-10T11:19:12.736 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.736 7f8550a16700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f854c09ed20 con 0x7f854c0c1f60 2021-02-10T11:19:12.736 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.736 7f8550a16700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f854c105d10 con 0x7f854c09f140 2021-02-10T11:19:12.736 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.736 7f854ae1d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f854c09e920 0x7f854c0c2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:12.736 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.736 7f854ae1d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f854c09e920 0x7f854c0c2960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:44452/0 (socket says 172.21.2.112:44452) 2021-02-10T11:19:12.736 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.736 7f854ae1d700 1 -- 172.21.2.112:0/1328340902 learned_addr learned my addr 172.21.2.112:0/1328340902 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:19:12.736 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.736 7f854a61c700 1 --2- 172.21.2.112:0/1328340902 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f854c09f140 0x7f854c0c1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:12.736 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.736 7f8549e1b700 1 --2- 172.21.2.112:0/1328340902 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f854c0c1f60 0x7f854c09e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:12.736 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.737 7f854ae1d700 1 -- 172.21.2.112:0/1328340902 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f854c09f140 msgr2=0x7f854c0c1940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:12.737 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.737 7f854ae1d700 1 --2- 172.21.2.112:0/1328340902 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f854c09f140 0x7f854c0c1940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:12.737 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.737 7f854ae1d700 1 -- 172.21.2.112:0/1328340902 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f854c0c1f60 msgr2=0x7f854c09e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:12.737 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.737 7f854ae1d700 1 --2- 172.21.2.112:0/1328340902 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f854c0c1f60 0x7f854c09e300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:12.737 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.737 7f854ae1d700 1 -- 172.21.2.112:0/1328340902 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f854c1a4910 con 0x7f854c09e920 2021-02-10T11:19:12.737 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.737 7f854ae1d700 1 --2- 172.21.2.112:0/1328340902 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f854c09e920 0x7f854c0c2960 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7f854400b910 tx=0x7f854400bcb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:19:12.737 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.737 7f8542ffd700 1 -- 172.21.2.112:0/1328340902 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8544006aa0 con 0x7f854c09e920 2021-02-10T11:19:12.737 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.737 7f8542ffd700 1 -- 172.21.2.112:0/1328340902 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85440011f0 con 0x7f854c09e920 2021-02-10T11:19:12.737 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.737 7f8542ffd700 1 -- 172.21.2.112:0/1328340902 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8544001710 con 0x7f854c09e920 2021-02-10T11:19:12.737 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.737 7f8550a16700 1 -- 172.21.2.112:0/1328340902 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f854c1a4910 con 0x7f854c09e920 2021-02-10T11:19:12.737 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.737 7f8550a16700 1 -- 172.21.2.112:0/1328340902 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f854c1a4910 con 0x7f854c09e920 2021-02-10T11:19:12.737 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.738 7f8542ffd700 1 -- 172.21.2.112:0/1328340902 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f85440013d0 con 0x7f854c09e920 2021-02-10T11:19:12.737 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.738 7f8542ffd700 1 --2- 172.21.2.112:0/1328340902 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8530018730 0x7f853001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:12.737 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.738 7f8542ffd700 1 -- 172.21.2.112:0/1328340902 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f8544032b30 con 0x7f854c09e920 2021-02-10T11:19:12.737 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.738 7f854a61c700 1 --2- 172.21.2.112:0/1328340902 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8530018730 0x7f853001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:12.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.738 7f854a61c700 1 --2- 172.21.2.112:0/1328340902 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8530018730 0x7f853001ac10 crc :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:19:12.765 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.769 7f8550a16700 1 -- 172.21.2.112:0/1328340902 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f854c0f0b50 con 0x7f854c09e920 2021-02-10T11:19:12.768 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.771 7f8542ffd700 1 -- 172.21.2.112:0/1328340902 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8544006aa0 con 0x7f854c09e920 2021-02-10T11:19:12.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.901 7f8550a16700 1 -- 172.21.2.112:0/1328340902 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348591", "target": ["mgr", ""]}) v1 -- 0x7f854c0f0b50 con 0x7f8530018730 2021-02-10T11:19:12.901 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.906 7f8542ffd700 1 -- 172.21.2.112:0/1328340902 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f853c000ec0 con 0x7f8530018730 2021-02-10T11:19:12.905 INFO:teuthology.orchestra.run.gibba012.stdout:/volumes/_nogroup/subvolume_0000000000348591/934785f8-d9fe-4825-8a14-91a233462530 2021-02-10T11:19:12.907 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.910 7f8550a16700 1 -- 172.21.2.112:0/1328340902 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8530018730 msgr2=0x7f853001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:12.907 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.910 7f8550a16700 1 --2- 172.21.2.112:0/1328340902 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8530018730 0x7f853001ac10 crc :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:12.907 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.910 7f8550a16700 1 -- 172.21.2.112:0/1328340902 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f854c09e920 msgr2=0x7f854c0c2960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:12.907 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.910 7f8550a16700 1 --2- 172.21.2.112:0/1328340902 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f854c09e920 0x7f854c0c2960 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7f854400b910 tx=0x7f854400bcb0).stop 2021-02-10T11:19:12.907 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.910 7f8550a16700 1 -- 172.21.2.112:0/1328340902 shutdown_connections 2021-02-10T11:19:12.907 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.910 7f8550a16700 1 --2- 172.21.2.112:0/1328340902 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8530018730 0x7f853001ac10 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:12.907 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.910 7f8550a16700 1 --2- 172.21.2.112:0/1328340902 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f854c0c1f60 0x7f854c09e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:12.907 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.910 7f8550a16700 1 --2- 172.21.2.112:0/1328340902 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f854c09e920 0x7f854c0c2960 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:12.907 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.910 7f8550a16700 1 --2- 172.21.2.112:0/1328340902 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f854c09f140 0x7f854c0c1940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:12.907 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.910 7f8550a16700 1 -- 172.21.2.112:0/1328340902 >> 172.21.2.112:0/1328340902 conn(0x7f854c09fc10 msgr2=0x7f854c0b3250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:19:12.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.910 7f8550a16700 1 -- 172.21.2.112:0/1328340902 shutdown_connections 2021-02-10T11:19:12.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:12.910 7f8550a16700 1 -- 172.21.2.112:0/1328340902 wait complete. 2021-02-10T11:19:12.915 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000348591 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000348591/934785f8-d9fe-4825-8a14-91a233462530 2021-02-10T11:19:12.915 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:12.915 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:12.915 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:13.249 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:19:13.250 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:13.250 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348591/934785f8-d9fe-4825-8a14-91a233462530/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:13.631 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:19:13.631 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:19:13.631 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0248094 s, 42.3 MB/s 2021-02-10T11:19:13.632 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:13.632 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:13.632 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:13.967 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:19:13.968 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:13.968 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348591/934785f8-d9fe-4825-8a14-91a233462530/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:14.336 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:19:14.336 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:19:14.337 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0227896 s, 46.0 MB/s 2021-02-10T11:19:14.338 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:14.338 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:14.338 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:14.672 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:19:14.672 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:14.673 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348591/934785f8-d9fe-4825-8a14-91a233462530/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:15.049 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:19:15.049 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:19:15.050 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0261615 s, 40.1 MB/s 2021-02-10T11:19:15.050 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:15.051 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:15.051 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:15.385 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:19:15.385 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:15.385 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348591/934785f8-d9fe-4825-8a14-91a233462530/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:15.760 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:19:15.760 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:19:15.760 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.023653 s, 44.3 MB/s 2021-02-10T11:19:15.761 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:15.761 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:15.761 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:16.096 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:19:16.096 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:16.097 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348591/934785f8-d9fe-4825-8a14-91a233462530/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:16.471 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:19:16.471 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:19:16.471 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0245036 s, 42.8 MB/s 2021-02-10T11:19:16.472 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:16.472 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:16.472 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:16.806 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:19:16.806 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:16.807 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348591/934785f8-d9fe-4825-8a14-91a233462530/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:17.189 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:19:17.189 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:19:17.189 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0322055 s, 32.6 MB/s 2021-02-10T11:19:17.190 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:17.190 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:17.190 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:17.525 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:19:17.525 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:17.525 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348591/934785f8-d9fe-4825-8a14-91a233462530/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:17.901 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:19:17.901 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:19:17.901 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0257477 s, 40.7 MB/s 2021-02-10T11:19:17.902 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:17.903 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:17.903 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:18.238 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:19:18.238 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:18.239 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348591/934785f8-d9fe-4825-8a14-91a233462530/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:18.614 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:19:18.614 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:19:18.614 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0244484 s, 42.9 MB/s 2021-02-10T11:19:18.615 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:18.615 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:18.616 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:18.951 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:19:18.951 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:18.951 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348591/934785f8-d9fe-4825-8a14-91a233462530/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:19.322 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:19:19.323 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:19:19.323 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0231352 s, 45.3 MB/s 2021-02-10T11:19:19.324 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:19.324 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:19.324 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:19.659 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:19:19.659 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:19.659 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348591/934785f8-d9fe-4825-8a14-91a233462530/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:20.035 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:19:20.035 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:19:20.035 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0245127 s, 42.8 MB/s 2021-02-10T11:19:20.036 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000348592 2021-02-10T11:19:20.466 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:20.454 7f38aea9a700 1 Processor -- start 2021-02-10T11:19:20.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:20.454 7f38aea9a700 1 -- start start 2021-02-10T11:19:20.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:20.454 7f38aea9a700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f38a81b0470 0x7f38a81b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:20.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:20.454 7f38aea9a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f38a80f20e0 0x7f38a80f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:20.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:20.454 7f38aea9a700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38a80f0b50 con 0x7f38a80ef610 2021-02-10T11:19:20.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:20.454 7f38aea9a700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38a80f0820 con 0x7f38a81b0470 2021-02-10T11:19:20.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:20.454 7f38aea9a700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38a80f24e0 con 0x7f38a80f20e0 2021-02-10T11:19:20.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:20.455 7f38ac8b6700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f38a81b0470 0x7f38a81b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:20.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:20.455 7f38ac8b6700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f38a81b0470 0x7f38a81b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:44460/0 (socket says 172.21.2.112:44460) 2021-02-10T11:19:20.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:20.455 7f38ac8b6700 1 -- 172.21.2.112:0/1111552679 learned_addr learned my addr 172.21.2.112:0/1111552679 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:19:20.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:20.455 7f38a7fff700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f38a80ef610 0x7f38a80efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:43922/0 (socket says 172.21.2.112:43922) 2021-02-10T11:19:20.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:20.455 7f38ad0b7700 1 --2- 172.21.2.112:0/1111552679 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f38a80f20e0 0x7f38a80f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:20.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:20.455 7f38ac8b6700 1 -- 172.21.2.112:0/1111552679 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f38a80f20e0 msgr2=0x7f38a80f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:19:20.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:20.455 7f38ac8b6700 1 --2- 172.21.2.112:0/1111552679 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f38a80f20e0 0x7f38a80f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:20.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:20.455 7f38ac8b6700 1 -- 172.21.2.112:0/1111552679 >> v1:172.21.2.112:6789/0 conn(0x7f38a80ef610 legacy=0x7f38a80efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:20.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:20.455 7f38ac8b6700 1 -- 172.21.2.112:0/1111552679 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38a80e52e0 con 0x7f38a81b0470 2021-02-10T11:19:20.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:20.455 7f38ad0b7700 1 --2- 172.21.2.112:0/1111552679 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f38a80f20e0 0x7f38a80f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:19:20.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:20.455 7f38ac8b6700 1 --2- 172.21.2.112:0/1111552679 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f38a81b0470 0x7f38a81b0870 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7f3890006700 tx=0x7f3890009b80).ready entity=mon.0 client_cookie=8ec981026345f03b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:19:20.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:20.455 7f38a77fe700 1 -- 172.21.2.112:0/1111552679 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f389000c0c0 con 0x7f38a81b0470 2021-02-10T11:19:20.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:20.455 7f38a77fe700 1 -- 172.21.2.112:0/1111552679 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f389000c3c0 con 0x7f38a81b0470 2021-02-10T11:19:20.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:20.456 7f38a77fe700 1 -- 172.21.2.112:0/1111552679 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f389000c8e0 con 0x7f38a81b0470 2021-02-10T11:19:20.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:20.456 7f38aea9a700 1 -- 172.21.2.112:0/1111552679 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f38a81b0470 msgr2=0x7f38a81b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:20.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:20.456 7f38aea9a700 1 --2- 172.21.2.112:0/1111552679 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f38a81b0470 0x7f38a81b0870 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7f3890006700 tx=0x7f3890009b80).stop 2021-02-10T11:19:20.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:20.456 7f38aea9a700 1 -- 172.21.2.112:0/1111552679 shutdown_connections 2021-02-10T11:19:20.469 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:20.456 7f38aea9a700 1 --2- 172.21.2.112:0/1111552679 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f38a80f20e0 0x7f38a80f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:20.469 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:20.456 7f38aea9a700 1 --2- 172.21.2.112:0/1111552679 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f38a81b0470 0x7f38a81b0870 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:20.469 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:20.456 7f38aea9a700 1 -- 172.21.2.112:0/1111552679 >> 172.21.2.112:0/1111552679 conn(0x7f38a80ce3e0 msgr2=0x7f38a80e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:19:20.469 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:20.456 7f38aea9a700 1 -- 172.21.2.112:0/1111552679 shutdown_connections 2021-02-10T11:19:20.469 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:20.456 7f38aea9a700 1 -- 172.21.2.112:0/1111552679 wait complete. 2021-02-10T11:19:20.470 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:20.471 7f38aea9a700 1 Processor -- start 2021-02-10T11:19:20.470 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:20.471 7f38aea9a700 1 -- start start 2021-02-10T11:19:20.470 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:20.471 7f38aea9a700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f38a80c3580 0x7f38a80a7080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:20.470 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:20.471 7f38aea9a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f38a8105d10 0x7f38a8106110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:20.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:20.471 7f38aea9a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f38a8106730 0x7f38a80bff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:20.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:20.471 7f38aea9a700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38a80c3980 con 0x7f38a8106730 2021-02-10T11:19:20.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:20.471 7f38aea9a700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38a80c05b0 con 0x7f38a80c3580 2021-02-10T11:19:20.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:20.471 7f38aea9a700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38a80c07d0 con 0x7f38a8105d10 2021-02-10T11:19:20.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:20.471 7f38ad0b7700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f38a8106730 0x7f38a80bff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:20.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:20.471 7f38ad0b7700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f38a8106730 0x7f38a80bff90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:44466/0 (socket says 172.21.2.112:44466) 2021-02-10T11:19:20.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:20.471 7f38ad0b7700 1 -- 172.21.2.112:0/1329110738 learned_addr learned my addr 172.21.2.112:0/1329110738 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:19:20.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:20.472 7f38ad0b7700 1 -- 172.21.2.112:0/1329110738 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f38a8105d10 msgr2=0x7f38a8106110 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:19:20.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:20.472 7f38ad0b7700 1 --2- 172.21.2.112:0/1329110738 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f38a8105d10 0x7f38a8106110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:19:20.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:20.472 7f38ad0b7700 1 -- 172.21.2.112:0/1329110738 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f38a80c3580 msgr2=0x7f38a80a7080 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:19:20.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:20.472 7f38ad0b7700 1 --2- 172.21.2.112:0/1329110738 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f38a80c3580 0x7f38a80a7080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:19:20.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:20.472 7f38ad0b7700 1 -- 172.21.2.112:0/1329110738 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38a80c09f0 con 0x7f38a8106730 2021-02-10T11:19:20.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:20.472 7f38ad0b7700 1 --2- 172.21.2.112:0/1329110738 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f38a8106730 0x7f38a80bff90 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 rx=0x7f38a0009980 tx=0x7f38a000bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:19:20.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:20.472 7f38a57fa700 1 -- 172.21.2.112:0/1329110738 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f38a000c2f0 con 0x7f38a8106730 2021-02-10T11:19:20.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:20.472 7f38a57fa700 1 -- 172.21.2.112:0/1329110738 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38a0006790 con 0x7f38a8106730 2021-02-10T11:19:20.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:20.472 7f38a57fa700 1 -- 172.21.2.112:0/1329110738 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f38a000c870 con 0x7f38a8106730 2021-02-10T11:19:20.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:20.472 7f38aea9a700 1 -- 172.21.2.112:0/1329110738 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38a80c09f0 con 0x7f38a8106730 2021-02-10T11:19:20.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:20.472 7f38aea9a700 1 -- 172.21.2.112:0/1329110738 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38a81a70d0 con 0x7f38a8106730 2021-02-10T11:19:20.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:20.473 7f38a57fa700 1 -- 172.21.2.112:0/1329110738 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f38a00011f0 con 0x7f38a8106730 2021-02-10T11:19:20.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:20.473 7f38a57fa700 1 --2- 172.21.2.112:0/1329110738 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f388c0186c0 0x7f388c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:20.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:20.473 7f38a57fa700 1 -- 172.21.2.112:0/1329110738 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f38a0032970 con 0x7f38a8106730 2021-02-10T11:19:20.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:20.474 7f38ac8b6700 1 --2- 172.21.2.112:0/1329110738 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f388c0186c0 0x7f388c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:20.473 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:20.474 7f38ac8b6700 1 --2- 172.21.2.112:0/1329110738 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f388c0186c0 0x7f388c01abc0 crc :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:19:20.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:20.504 7f38aea9a700 1 -- 172.21.2.112:0/1329110738 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38a8105ae0 con 0x7f38a8106730 2021-02-10T11:19:20.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:20.507 7f38a57fa700 1 -- 172.21.2.112:0/1329110738 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f38a0000f40 con 0x7f38a8106730 2021-02-10T11:19:20.630 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:20.633 7f38aea9a700 1 -- 172.21.2.112:0/1329110738 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348592", "target": ["mgr", ""]}) v1 -- 0x7f38a80c3980 con 0x7f388c0186c0 2021-02-10T11:19:20.657 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:20.661 7f38a57fa700 1 -- 172.21.2.112:0/1329110738 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f3890009100 con 0x7f388c0186c0 2021-02-10T11:19:20.663 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:20.667 7f38aea9a700 1 -- 172.21.2.112:0/1329110738 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f388c0186c0 msgr2=0x7f388c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:20.663 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:20.667 7f38aea9a700 1 --2- 172.21.2.112:0/1329110738 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f388c0186c0 0x7f388c01abc0 crc :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:20.663 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:20.667 7f38aea9a700 1 -- 172.21.2.112:0/1329110738 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f38a8106730 msgr2=0x7f38a80bff90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:20.663 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:20.667 7f38aea9a700 1 --2- 172.21.2.112:0/1329110738 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f38a8106730 0x7f38a80bff90 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 rx=0x7f38a0009980 tx=0x7f38a000bcd0).stop 2021-02-10T11:19:20.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:20.667 7f38aea9a700 1 -- 172.21.2.112:0/1329110738 shutdown_connections 2021-02-10T11:19:20.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:20.667 7f38aea9a700 1 --2- 172.21.2.112:0/1329110738 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f388c0186c0 0x7f388c01abc0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:20.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:20.667 7f38aea9a700 1 --2- 172.21.2.112:0/1329110738 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f38a8106730 0x7f38a80bff90 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:20.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:20.667 7f38aea9a700 1 --2- 172.21.2.112:0/1329110738 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f38a8105d10 0x7f38a8106110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:19:20.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:20.667 7f38aea9a700 1 --2- 172.21.2.112:0/1329110738 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f38a80c3580 0x7f38a80a7080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:19:20.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:20.667 7f38aea9a700 1 -- 172.21.2.112:0/1329110738 >> 172.21.2.112:0/1329110738 conn(0x7f38a810a430 msgr2=0x7f38a80bf050 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:19:20.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:20.667 7f38aea9a700 1 -- 172.21.2.112:0/1329110738 shutdown_connections 2021-02-10T11:19:20.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:20.667 7f38aea9a700 1 -- 172.21.2.112:0/1329110738 wait complete. 2021-02-10T11:19:20.671 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000348592 2021-02-10T11:19:21.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:21.088 7fed42c8f700 1 Processor -- start 2021-02-10T11:19:21.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:21.088 7fed42c8f700 1 -- start start 2021-02-10T11:19:21.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:21.088 7fed42c8f700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fed3c1b0870 0x7fed3c0f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:21.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:21.088 7fed42c8f700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fed3c0efe00 0x7fed3c0f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:21.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:21.088 7fed42c8f700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed3c1b0c70 con 0x7fed3c1b0470 2021-02-10T11:19:21.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:21.088 7fed42c8f700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed3c0e5ea0 con 0x7fed3c0efe00 2021-02-10T11:19:21.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:21.088 7fed42c8f700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed3c0f0b50 con 0x7fed3c1b0870 2021-02-10T11:19:21.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:21.088 7fed412ac700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fed3c0efe00 0x7fed3c0f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:21.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:21.088 7fed412ac700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fed3c0efe00 0x7fed3c0f0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:44476/0 (socket says 172.21.2.112:44476) 2021-02-10T11:19:21.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:21.088 7fed412ac700 1 -- 172.21.2.112:0/3876006931 learned_addr learned my addr 172.21.2.112:0/3876006931 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:19:21.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:21.089 7fed412ac700 1 -- 172.21.2.112:0/3876006931 >> v2:172.21.2.120:3300/0 conn(0x7fed3c1b0870 msgr2=0x7fed3c0f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:19:21.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:21.089 7fed412ac700 1 --2- 172.21.2.112:0/3876006931 >> v2:172.21.2.120:3300/0 conn(0x7fed3c1b0870 0x7fed3c0f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:19:21.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:21.089 7fed412ac700 1 -- 172.21.2.112:0/3876006931 >> v1:172.21.2.120:6789/0 conn(0x7fed3c1b0470 legacy=0x7fed3c0ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:19:21.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:21.089 7fed412ac700 1 -- 172.21.2.112:0/3876006931 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed3c0f0820 con 0x7fed3c0efe00 2021-02-10T11:19:21.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:21.089 7fed412ac700 1 --2- 172.21.2.112:0/3876006931 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fed3c0efe00 0x7fed3c0f0200 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7fed34006f00 tx=0x7fed34009f10).ready entity=mon.0 client_cookie=f84458ed3d470df3 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:19:21.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:21.089 7fed3b7fe700 1 -- 172.21.2.112:0/3876006931 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fed3400b3b0 con 0x7fed3c0efe00 2021-02-10T11:19:21.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:21.089 7fed3b7fe700 1 -- 172.21.2.112:0/3876006931 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed3400b630 con 0x7fed3c0efe00 2021-02-10T11:19:21.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:21.089 7fed3b7fe700 1 -- 172.21.2.112:0/3876006931 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fed3400bb50 con 0x7fed3c0efe00 2021-02-10T11:19:21.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:21.089 7fed42c8f700 1 -- 172.21.2.112:0/3876006931 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fed3c0efe00 msgr2=0x7fed3c0f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:21.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:21.089 7fed42c8f700 1 --2- 172.21.2.112:0/3876006931 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fed3c0efe00 0x7fed3c0f0200 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7fed34006f00 tx=0x7fed34009f10).stop 2021-02-10T11:19:21.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:21.090 7fed42c8f700 1 -- 172.21.2.112:0/3876006931 shutdown_connections 2021-02-10T11:19:21.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:21.090 7fed42c8f700 1 --2- 172.21.2.112:0/3876006931 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fed3c0efe00 0x7fed3c0f0200 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:21.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:21.090 7fed42c8f700 1 --2- 172.21.2.112:0/3876006931 >> v2:172.21.2.120:3300/0 conn(0x7fed3c1b0870 0x7fed3c0f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:19:21.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:21.090 7fed42c8f700 1 -- 172.21.2.112:0/3876006931 >> 172.21.2.112:0/3876006931 conn(0x7fed3c0ce3e0 msgr2=0x7fed3c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:19:21.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:21.090 7fed42c8f700 1 -- 172.21.2.112:0/3876006931 shutdown_connections 2021-02-10T11:19:21.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:21.090 7fed42c8f700 1 -- 172.21.2.112:0/3876006931 wait complete. 2021-02-10T11:19:21.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:21.104 7fed42c8f700 1 Processor -- start 2021-02-10T11:19:21.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:21.104 7fed42c8f700 1 -- start start 2021-02-10T11:19:21.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:21.104 7fed42c8f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fed3c094fb0 0x7fed3c09e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:21.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:21.104 7fed42c8f700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fed3c0c2be0 0x7fed3c0c2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:21.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:21.104 7fed42c8f700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fed3c0c3600 0x7fed3c0a9c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:21.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:21.104 7fed42c8f700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed3c09ebf0 con 0x7fed3c094fb0 2021-02-10T11:19:21.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:21.104 7fed42c8f700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed3c0aa2a0 con 0x7fed3c0c3600 2021-02-10T11:19:21.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:21.104 7fed42c8f700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed3c0aa4c0 con 0x7fed3c0c2be0 2021-02-10T11:19:21.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:21.105 7fed40aab700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fed3c094fb0 0x7fed3c09e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:21.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:21.105 7fed40aab700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fed3c094fb0 0x7fed3c09e5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:44478/0 (socket says 172.21.2.112:44478) 2021-02-10T11:19:21.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:21.105 7fed40aab700 1 -- 172.21.2.112:0/1337210113 learned_addr learned my addr 172.21.2.112:0/1337210113 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:19:21.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:21.105 7fed40aab700 1 -- 172.21.2.112:0/1337210113 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fed3c0c2be0 msgr2=0x7fed3c0c2fe0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:19:21.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:21.105 7fed40aab700 1 --2- 172.21.2.112:0/1337210113 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fed3c0c2be0 0x7fed3c0c2fe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:19:21.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:21.105 7fed40aab700 1 -- 172.21.2.112:0/1337210113 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fed3c0c3600 msgr2=0x7fed3c0a9c80 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:19:21.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:21.105 7fed40aab700 1 --2- 172.21.2.112:0/1337210113 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fed3c0c3600 0x7fed3c0a9c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:19:21.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:21.105 7fed40aab700 1 -- 172.21.2.112:0/1337210113 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed3c0aa6e0 con 0x7fed3c094fb0 2021-02-10T11:19:21.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:21.105 7fed40aab700 1 --2- 172.21.2.112:0/1337210113 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fed3c094fb0 0x7fed3c09e5d0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 rx=0x7fed2c00b5b0 tx=0x7fed2c00b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:19:21.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:21.105 7fed397fa700 1 -- 172.21.2.112:0/1337210113 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fed2c00bd40 con 0x7fed3c094fb0 2021-02-10T11:19:21.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:21.105 7fed397fa700 1 -- 172.21.2.112:0/1337210113 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed2c000e20 con 0x7fed3c094fb0 2021-02-10T11:19:21.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:21.105 7fed42c8f700 1 -- 172.21.2.112:0/1337210113 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed3c0aa6e0 con 0x7fed3c094fb0 2021-02-10T11:19:21.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:21.105 7fed42c8f700 1 -- 172.21.2.112:0/1337210113 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed3c0aac70 con 0x7fed3c094fb0 2021-02-10T11:19:21.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:21.105 7fed397fa700 1 -- 172.21.2.112:0/1337210113 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fed2c006210 con 0x7fed3c094fb0 2021-02-10T11:19:21.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:21.106 7fed397fa700 1 -- 172.21.2.112:0/1337210113 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fed2c006020 con 0x7fed3c094fb0 2021-02-10T11:19:21.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:21.106 7fed397fa700 1 --2- 172.21.2.112:0/1337210113 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fed180186c0 0x7fed1801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:21.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:21.106 7fed397fa700 1 -- 172.21.2.112:0/1337210113 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fed2c0327f0 con 0x7fed3c094fb0 2021-02-10T11:19:21.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:21.107 7fed3bfff700 1 --2- 172.21.2.112:0/1337210113 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fed180186c0 0x7fed1801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:21.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:21.108 7fed3bfff700 1 --2- 172.21.2.112:0/1337210113 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fed180186c0 0x7fed1801abc0 crc :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:19:21.134 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:21.138 7fed42c8f700 1 -- 172.21.2.112:0/1337210113 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed3c0a9750 con 0x7fed3c094fb0 2021-02-10T11:19:21.136 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:21.140 7fed397fa700 1 -- 172.21.2.112:0/1337210113 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fed2c0009b0 con 0x7fed3c094fb0 2021-02-10T11:19:21.265 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:21.269 7fed42c8f700 1 -- 172.21.2.112:0/1337210113 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348592", "target": ["mgr", ""]}) v1 -- 0x7fed3c0aa2a0 con 0x7fed180186c0 2021-02-10T11:19:21.268 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:21.272 7fed397fa700 1 -- 172.21.2.112:0/1337210113 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7fed28006970 con 0x7fed180186c0 2021-02-10T11:19:21.268 INFO:teuthology.orchestra.run.gibba012.stdout:/volumes/_nogroup/subvolume_0000000000348592/a6266d21-c3c5-4ef2-9141-e3dae8605b13 2021-02-10T11:19:21.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:21.273 7fed42c8f700 1 -- 172.21.2.112:0/1337210113 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fed180186c0 msgr2=0x7fed1801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:21.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:21.273 7fed42c8f700 1 --2- 172.21.2.112:0/1337210113 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fed180186c0 0x7fed1801abc0 crc :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:21.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:21.273 7fed42c8f700 1 -- 172.21.2.112:0/1337210113 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fed3c094fb0 msgr2=0x7fed3c09e5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:21.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:21.273 7fed42c8f700 1 --2- 172.21.2.112:0/1337210113 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fed3c094fb0 0x7fed3c09e5d0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 rx=0x7fed2c00b5b0 tx=0x7fed2c00b950).stop 2021-02-10T11:19:21.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:21.274 7fed42c8f700 1 -- 172.21.2.112:0/1337210113 shutdown_connections 2021-02-10T11:19:21.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:21.274 7fed42c8f700 1 --2- 172.21.2.112:0/1337210113 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fed180186c0 0x7fed1801abc0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:21.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:21.274 7fed42c8f700 1 --2- 172.21.2.112:0/1337210113 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fed3c0c3600 0x7fed3c0a9c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:19:21.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:21.274 7fed42c8f700 1 --2- 172.21.2.112:0/1337210113 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fed3c0c2be0 0x7fed3c0c2fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:19:21.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:21.274 7fed42c8f700 1 --2- 172.21.2.112:0/1337210113 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fed3c094fb0 0x7fed3c09e5d0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:21.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:21.274 7fed42c8f700 1 -- 172.21.2.112:0/1337210113 >> 172.21.2.112:0/1337210113 conn(0x7fed3c09f630 msgr2=0x7fed3c0a0130 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:19:21.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:21.274 7fed42c8f700 1 -- 172.21.2.112:0/1337210113 shutdown_connections 2021-02-10T11:19:21.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:21.274 7fed42c8f700 1 -- 172.21.2.112:0/1337210113 wait complete. 2021-02-10T11:19:21.278 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000348592 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000348592/a6266d21-c3c5-4ef2-9141-e3dae8605b13 2021-02-10T11:19:21.278 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:21.279 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:21.279 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:21.613 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:19:21.613 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:21.614 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348592/a6266d21-c3c5-4ef2-9141-e3dae8605b13/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:21.992 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:19:21.993 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:19:21.993 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0237424 s, 44.2 MB/s 2021-02-10T11:19:21.994 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:21.994 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:21.994 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:22.327 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:19:22.327 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:22.328 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348592/a6266d21-c3c5-4ef2-9141-e3dae8605b13/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:22.702 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:19:22.703 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:19:22.703 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0235676 s, 44.5 MB/s 2021-02-10T11:19:22.704 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:22.704 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:22.704 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:23.040 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:19:23.040 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:23.041 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348592/a6266d21-c3c5-4ef2-9141-e3dae8605b13/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:23.416 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:19:23.416 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:19:23.416 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0241749 s, 43.4 MB/s 2021-02-10T11:19:23.417 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:23.417 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:23.417 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:23.751 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:19:23.752 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:23.752 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348592/a6266d21-c3c5-4ef2-9141-e3dae8605b13/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:24.128 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:19:24.128 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:19:24.128 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0259903 s, 40.3 MB/s 2021-02-10T11:19:24.129 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:24.129 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:24.129 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:24.463 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:19:24.464 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:24.464 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348592/a6266d21-c3c5-4ef2-9141-e3dae8605b13/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:24.836 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:19:24.836 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:19:24.836 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0230176 s, 45.6 MB/s 2021-02-10T11:19:24.837 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:24.837 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:24.837 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:25.173 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:19:25.173 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:25.173 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348592/a6266d21-c3c5-4ef2-9141-e3dae8605b13/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:25.547 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:19:25.548 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:19:25.548 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0246742 s, 42.5 MB/s 2021-02-10T11:19:25.549 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:25.549 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:25.549 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:25.884 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:19:25.884 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:25.885 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348592/a6266d21-c3c5-4ef2-9141-e3dae8605b13/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:26.259 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:19:26.259 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:19:26.259 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0234242 s, 44.8 MB/s 2021-02-10T11:19:26.260 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:26.260 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:26.260 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:26.594 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:19:26.594 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:26.595 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348592/a6266d21-c3c5-4ef2-9141-e3dae8605b13/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:26.967 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:19:26.968 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:19:26.968 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0232874 s, 45.0 MB/s 2021-02-10T11:19:26.969 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:26.969 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:26.969 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:27.303 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:19:27.303 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:27.303 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348592/a6266d21-c3c5-4ef2-9141-e3dae8605b13/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:27.679 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:19:27.679 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:19:27.679 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0255446 s, 41.0 MB/s 2021-02-10T11:19:27.680 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:27.680 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:27.680 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:28.014 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:19:28.014 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:28.014 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348592/a6266d21-c3c5-4ef2-9141-e3dae8605b13/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:28.389 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:19:28.390 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:19:28.390 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0243975 s, 43.0 MB/s 2021-02-10T11:19:28.391 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000348593 2021-02-10T11:19:28.819 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:28.807 7f57c9c01700 1 Processor -- start 2021-02-10T11:19:28.819 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:28.807 7f57c9c01700 1 -- start start 2021-02-10T11:19:28.819 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:28.807 7f57c9c01700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f57c40f05f0 0x7f57c40f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:28.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:28.807 7f57c9c01700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57c41b0c70 con 0x7f57c41b0470 2021-02-10T11:19:28.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:28.807 7f57c9c01700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57c40f0b50 con 0x7f57c41b0870 2021-02-10T11:19:28.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:28.807 7f57c9c01700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57c40e5ea0 con 0x7f57c40f05f0 2021-02-10T11:19:28.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:28.807 7f57c3fff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f57c40f05f0 0x7f57c40f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:28.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:28.807 7f57c3fff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f57c40f05f0 0x7f57c40f20e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:44490/0 (socket says 172.21.2.112:44490) 2021-02-10T11:19:28.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:28.807 7f57c3fff700 1 -- 172.21.2.112:0/1977932324 learned_addr learned my addr 172.21.2.112:0/1977932324 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:19:28.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:28.808 7f57c3fff700 1 -- 172.21.2.112:0/1977932324 >> v1:172.21.2.120:6789/0 conn(0x7f57c41b0870 legacy=0x7f57c40efe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:28.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:28.808 7f57c3fff700 1 -- 172.21.2.112:0/1977932324 >> v1:172.21.2.112:6789/0 conn(0x7f57c41b0470 legacy=0x7f57c40ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:28.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:28.808 7f57c3fff700 1 -- 172.21.2.112:0/1977932324 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57c40e52e0 con 0x7f57c40f05f0 2021-02-10T11:19:28.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:28.808 7f57c3fff700 1 --2- 172.21.2.112:0/1977932324 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f57c40f05f0 0x7f57c40f20e0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 rx=0x7f57bc006f00 tx=0x7f57bc009f10).ready entity=mon.0 client_cookie=358c449741a61485 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:19:28.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:28.808 7f57c27fc700 1 -- 172.21.2.112:0/1977932324 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f57bc00b3b0 con 0x7f57c40f05f0 2021-02-10T11:19:28.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:28.808 7f57c27fc700 1 -- 172.21.2.112:0/1977932324 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57bc00b730 con 0x7f57c40f05f0 2021-02-10T11:19:28.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:28.808 7f57c27fc700 1 -- 172.21.2.112:0/1977932324 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f57bc00bc10 con 0x7f57c40f05f0 2021-02-10T11:19:28.821 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:28.808 7f57c9c01700 1 -- 172.21.2.112:0/1977932324 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f57c40f05f0 msgr2=0x7f57c40f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:28.821 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:28.808 7f57c9c01700 1 --2- 172.21.2.112:0/1977932324 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f57c40f05f0 0x7f57c40f20e0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 rx=0x7f57bc006f00 tx=0x7f57bc009f10).stop 2021-02-10T11:19:28.821 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:28.810 7f57c9c01700 1 -- 172.21.2.112:0/1977932324 shutdown_connections 2021-02-10T11:19:28.821 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:28.810 7f57c9c01700 1 --2- 172.21.2.112:0/1977932324 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f57c40f05f0 0x7f57c40f20e0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:28.821 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:28.810 7f57c9c01700 1 -- 172.21.2.112:0/1977932324 >> 172.21.2.112:0/1977932324 conn(0x7f57c40ce3e0 msgr2=0x7f57c40e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:19:28.821 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:28.810 7f57c9c01700 1 -- 172.21.2.112:0/1977932324 shutdown_connections 2021-02-10T11:19:28.821 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:28.810 7f57c9c01700 1 -- 172.21.2.112:0/1977932324 wait complete. 2021-02-10T11:19:28.823 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:28.823 7f57c9c01700 1 Processor -- start 2021-02-10T11:19:28.823 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:28.824 7f57c9c01700 1 -- start start 2021-02-10T11:19:28.823 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:28.824 7f57c9c01700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f57c40a3b50 0x7f57c41b3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:28.823 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:28.824 7f57c9c01700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f57c41c25b0 0x7f57c41b3d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:28.823 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:28.824 7f57c9c01700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f57c41b4350 0x7f57c41b18a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:28.824 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:28.824 7f57c9c01700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57c41b3190 con 0x7f57c41c25b0 2021-02-10T11:19:28.824 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:28.824 7f57c9c01700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57c4087750 con 0x7f57c40a3b50 2021-02-10T11:19:28.824 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:28.824 7f57c9c01700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57c40a6bf0 con 0x7f57c41b4350 2021-02-10T11:19:28.824 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:28.824 7f57c2ffd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f57c41c25b0 0x7f57c41b3d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:28.824 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:28.824 7f57c2ffd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f57c41c25b0 0x7f57c41b3d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:44492/0 (socket says 172.21.2.112:44492) 2021-02-10T11:19:28.824 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:28.824 7f57c2ffd700 1 -- 172.21.2.112:0/907008887 learned_addr learned my addr 172.21.2.112:0/907008887 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:19:28.824 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:28.824 7f57c2ffd700 1 -- 172.21.2.112:0/907008887 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f57c41b4350 msgr2=0x7f57c41b18a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:19:28.824 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:28.824 7f57c2ffd700 1 --2- 172.21.2.112:0/907008887 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f57c41b4350 0x7f57c41b18a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:19:28.824 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:28.824 7f57c2ffd700 1 -- 172.21.2.112:0/907008887 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f57c40a3b50 msgr2=0x7f57c41b3710 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:19:28.824 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:28.824 7f57c2ffd700 1 --2- 172.21.2.112:0/907008887 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f57c40a3b50 0x7f57c41b3710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:19:28.824 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:28.824 7f57c2ffd700 1 -- 172.21.2.112:0/907008887 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57c41b1ec0 con 0x7f57c41c25b0 2021-02-10T11:19:28.824 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:28.825 7f57c2ffd700 1 --2- 172.21.2.112:0/907008887 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f57c41c25b0 0x7f57c41b3d30 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7f57b4006ed0 tx=0x7f57b400b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:19:28.824 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:28.825 7f57bbfff700 1 -- 172.21.2.112:0/907008887 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f57b4006240 con 0x7f57c41c25b0 2021-02-10T11:19:28.825 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:28.825 7f57bbfff700 1 -- 172.21.2.112:0/907008887 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57b40054b0 con 0x7f57c41c25b0 2021-02-10T11:19:28.825 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:28.825 7f57bbfff700 1 -- 172.21.2.112:0/907008887 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f57b4005990 con 0x7f57c41c25b0 2021-02-10T11:19:28.825 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:28.825 7f57c9c01700 1 -- 172.21.2.112:0/907008887 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57c41b1ec0 con 0x7f57c41c25b0 2021-02-10T11:19:28.825 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:28.825 7f57c9c01700 1 -- 172.21.2.112:0/907008887 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57c41b2450 con 0x7f57c41c25b0 2021-02-10T11:19:28.825 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:28.826 7f57bbfff700 1 -- 172.21.2.112:0/907008887 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f57b40056f0 con 0x7f57c41c25b0 2021-02-10T11:19:28.825 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:28.826 7f57bbfff700 1 --2- 172.21.2.112:0/907008887 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f57a80186c0 0x7f57a801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:28.825 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:28.826 7f57bbfff700 1 -- 172.21.2.112:0/907008887 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f57b4032440 con 0x7f57c41c25b0 2021-02-10T11:19:28.825 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:28.826 7f57c37fe700 1 --2- 172.21.2.112:0/907008887 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f57a80186c0 0x7f57a801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:28.825 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:28.827 7f57c37fe700 1 --2- 172.21.2.112:0/907008887 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f57a80186c0 0x7f57a801abc0 crc :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:19:28.854 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:28.858 7f57c9c01700 1 -- 172.21.2.112:0/907008887 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57c407e6b0 con 0x7f57c41c25b0 2021-02-10T11:19:28.856 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:28.861 7f57bbfff700 1 -- 172.21.2.112:0/907008887 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f57b401e8d0 con 0x7f57c41c25b0 2021-02-10T11:19:28.982 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:28.985 7f57c9c01700 1 -- 172.21.2.112:0/907008887 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348593", "target": ["mgr", ""]}) v1 -- 0x7f57c41b2450 con 0x7f57a80186c0 2021-02-10T11:19:29.007 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:29.012 7f57bbfff700 1 -- 172.21.2.112:0/907008887 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f57ac006db0 con 0x7f57a80186c0 2021-02-10T11:19:29.015 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:29.019 7f57c9c01700 1 -- 172.21.2.112:0/907008887 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f57a80186c0 msgr2=0x7f57a801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:29.015 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:29.019 7f57c9c01700 1 --2- 172.21.2.112:0/907008887 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f57a80186c0 0x7f57a801abc0 crc :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:29.015 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:29.020 7f57c9c01700 1 -- 172.21.2.112:0/907008887 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f57c41c25b0 msgr2=0x7f57c41b3d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:29.015 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:29.020 7f57c9c01700 1 --2- 172.21.2.112:0/907008887 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f57c41c25b0 0x7f57c41b3d30 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7f57b4006ed0 tx=0x7f57b400b960).stop 2021-02-10T11:19:29.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:29.020 7f57c9c01700 1 -- 172.21.2.112:0/907008887 shutdown_connections 2021-02-10T11:19:29.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:29.020 7f57c9c01700 1 --2- 172.21.2.112:0/907008887 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f57a80186c0 0x7f57a801abc0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:29.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:29.020 7f57c9c01700 1 --2- 172.21.2.112:0/907008887 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f57c41b4350 0x7f57c41b18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:19:29.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:29.020 7f57c9c01700 1 --2- 172.21.2.112:0/907008887 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f57c41c25b0 0x7f57c41b3d30 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:29.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:29.020 7f57c9c01700 1 --2- 172.21.2.112:0/907008887 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f57c40a3b50 0x7f57c41b3710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:19:29.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:29.020 7f57c9c01700 1 -- 172.21.2.112:0/907008887 >> 172.21.2.112:0/907008887 conn(0x7f57c40886d0 msgr2=0x7f57c4081c40 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:19:29.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:29.020 7f57c9c01700 1 -- 172.21.2.112:0/907008887 shutdown_connections 2021-02-10T11:19:29.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:29.020 7f57c9c01700 1 -- 172.21.2.112:0/907008887 wait complete. 2021-02-10T11:19:29.023 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000348593 2021-02-10T11:19:29.457 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:29.442 7f2cd2f77700 1 Processor -- start 2021-02-10T11:19:29.458 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:29.442 7f2cd2f77700 1 -- start start 2021-02-10T11:19:29.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:29.442 7f2cd2f77700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2ccc1b0870 0x7f2ccc0f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:29.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:29.442 7f2cd2f77700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f2ccc0efe00 0x7f2ccc0f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:29.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:29.442 7f2cd2f77700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ccc1b0c70 con 0x7f2ccc1b0470 2021-02-10T11:19:29.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:29.443 7f2cd2f77700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ccc0f0b50 con 0x7f2ccc0efe00 2021-02-10T11:19:29.460 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:29.443 7f2cd2f77700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ccc0f0820 con 0x7f2ccc1b0870 2021-02-10T11:19:29.460 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:29.443 7f2cd0d93700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f2ccc1b0470 0x7f2ccc0ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:43962/0 (socket says 172.21.2.112:43962) 2021-02-10T11:19:29.460 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:29.443 7f2cd0d93700 1 -- 172.21.2.112:0/3328471827 learned_addr learned my addr 172.21.2.112:0/3328471827 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:19:29.460 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:29.443 7f2ccb7fe700 1 -- 172.21.2.112:0/3328471827 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 567471703 0 0) 0x7f2cb4000ce0 con 0x7f2ccc1b0470 2021-02-10T11:19:29.461 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:29.443 7f2ccb7fe700 1 -- 172.21.2.112:0/3328471827 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2cb8002fe0 con 0x7f2ccc1b0470 2021-02-10T11:19:29.461 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:29.443 7f2cd1594700 1 --2- 172.21.2.112:0/3328471827 >> v2:172.21.2.120:3300/0 conn(0x7f2ccc0efe00 0x7f2ccc0f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:29.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:29.443 7f2ccbfff700 1 --2- 172.21.2.112:0/3328471827 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2ccc1b0870 0x7f2ccc0f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:29.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:29.443 7f2ccb7fe700 1 -- 172.21.2.112:0/3328471827 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 880992296 0 0) 0x7f2cb40010e0 con 0x7f2ccc1b0470 2021-02-10T11:19:29.463 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:29.443 7f2ccb7fe700 1 -- 172.21.2.112:0/3328471827 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2ccc1b0870 msgr2=0x7f2ccc0f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:19:29.463 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:29.443 7f2ccb7fe700 1 --2- 172.21.2.112:0/3328471827 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2ccc1b0870 0x7f2ccc0f20e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:29.463 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:29.443 7f2ccb7fe700 1 -- 172.21.2.112:0/3328471827 >> v2:172.21.2.120:3300/0 conn(0x7f2ccc0efe00 msgr2=0x7f2ccc0f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:19:29.464 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:29.443 7f2ccb7fe700 1 --2- 172.21.2.112:0/3328471827 >> v2:172.21.2.120:3300/0 conn(0x7f2ccc0efe00 0x7f2ccc0f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:29.464 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:29.443 7f2ccb7fe700 1 -- 172.21.2.112:0/3328471827 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ccc0e52e0 con 0x7f2ccc1b0470 2021-02-10T11:19:29.464 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:29.443 7f2ccb7fe700 1 -- 172.21.2.112:0/3328471827 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f2cb4001600 con 0x7f2ccc1b0470 2021-02-10T11:19:29.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:29.443 7f2ccb7fe700 1 -- 172.21.2.112:0/3328471827 >> v1:172.21.2.112:6789/0 conn(0x7f2ccc1b0470 legacy=0x7f2ccc0ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:29.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:29.443 7f2ccb7fe700 1 --2- 172.21.2.112:0/3328471827 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2cb8003a60 0x7f2cb8005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:29.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:29.443 7f2ccb7fe700 1 --2- 172.21.2.112:0/3328471827 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2cb8006790 0x7f2cb8008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:29.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:29.443 7f2ccb7fe700 1 --2- 172.21.2.112:0/3328471827 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2cb8009590 0x7f2cb800ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:29.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:29.443 7f2ccb7fe700 1 -- 172.21.2.112:0/3328471827 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cb800c2d0 con 0x7f2cb8003a60 2021-02-10T11:19:29.466 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:29.443 7f2ccb7fe700 1 -- 172.21.2.112:0/3328471827 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cb800c4f0 con 0x7f2cb8006790 2021-02-10T11:19:29.466 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:29.443 7f2cd0d93700 1 --2- 172.21.2.112:0/3328471827 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2cb8003a60 0x7f2cb8005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:29.466 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:29.443 7f2ccb7fe700 1 -- 172.21.2.112:0/3328471827 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cb80506e0 con 0x7f2cb8009590 2021-02-10T11:19:29.466 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:29.444 7f2ccbfff700 1 --2- 172.21.2.112:0/3328471827 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2cb8006790 0x7f2cb8008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:29.466 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:29.444 7f2cd0d93700 1 -- 172.21.2.112:0/3328471827 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2cb8009590 msgr2=0x7f2cb800ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:19:29.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:29.444 7f2cd1594700 1 --2- 172.21.2.112:0/3328471827 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2cb8009590 0x7f2cb800ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:29.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:29.444 7f2cd0d93700 1 --2- 172.21.2.112:0/3328471827 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2cb8009590 0x7f2cb800ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:29.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:29.444 7f2cd0d93700 1 -- 172.21.2.112:0/3328471827 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2cb8006790 msgr2=0x7f2cb8008c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:19:29.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:29.444 7f2cd0d93700 1 --2- 172.21.2.112:0/3328471827 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2cb8006790 0x7f2cb8008c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:29.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:29.444 7f2cd0d93700 1 -- 172.21.2.112:0/3328471827 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cb8050980 con 0x7f2cb8003a60 2021-02-10T11:19:29.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:29.444 7f2cd0d93700 1 --2- 172.21.2.112:0/3328471827 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2cb8003a60 0x7f2cb8005ef0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7f2cb40012f0 tx=0x7f2cb4007cb0).ready entity=mon.0 client_cookie=ea99110a23facdaa server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:19:29.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:29.444 7f2ccb7fe700 1 -- 172.21.2.112:0/3328471827 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2cb40070c0 con 0x7f2cb8003a60 2021-02-10T11:19:29.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:29.444 7f2ccb7fe700 1 -- 172.21.2.112:0/3328471827 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cb40043a0 con 0x7f2cb8003a60 2021-02-10T11:19:29.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:29.444 7f2ccb7fe700 1 -- 172.21.2.112:0/3328471827 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2cb40048c0 con 0x7f2cb8003a60 2021-02-10T11:19:29.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:29.444 7f2cd2f77700 1 -- 172.21.2.112:0/3328471827 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2cb8003a60 msgr2=0x7f2cb8005ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:29.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:29.444 7f2cd2f77700 1 --2- 172.21.2.112:0/3328471827 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2cb8003a60 0x7f2cb8005ef0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7f2cb40012f0 tx=0x7f2cb4007cb0).stop 2021-02-10T11:19:29.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:29.444 7f2cd1594700 1 -- 172.21.2.112:0/3328471827 reap_dead start 2021-02-10T11:19:29.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:29.444 7f2cd2f77700 1 -- 172.21.2.112:0/3328471827 shutdown_connections 2021-02-10T11:19:29.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:29.444 7f2cd2f77700 1 -- 172.21.2.112:0/3328471827 >> 172.21.2.112:0/3328471827 conn(0x7f2ccc0ce3e0 msgr2=0x7f2ccc0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:19:29.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:29.444 7f2cd2f77700 1 -- 172.21.2.112:0/3328471827 shutdown_connections 2021-02-10T11:19:29.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:29.444 7f2cd2f77700 1 -- 172.21.2.112:0/3328471827 wait complete. 2021-02-10T11:19:29.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:29.459 7f2cd2f77700 1 Processor -- start 2021-02-10T11:19:29.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:29.459 7f2cd2f77700 1 -- start start 2021-02-10T11:19:29.481 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:29.460 7f2cd2f77700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2ccc0924b0 0x7f2ccc1a3590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:29.481 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:29.460 7f2cd2f77700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2ccc1a3bb0 0x7f2ccc0bf920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:29.482 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:29.460 7f2cd2f77700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2ccc0bff40 0x7f2ccc0c0340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:29.482 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:29.460 7f2cd2f77700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ccc09a0a0 con 0x7f2ccc0924b0 2021-02-10T11:19:29.483 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:29.460 7f2cd2f77700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ccc0c0960 con 0x7f2ccc1a3bb0 2021-02-10T11:19:29.483 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:29.460 7f2cd2f77700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ccc098420 con 0x7f2ccc0bff40 2021-02-10T11:19:29.484 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:29.460 7f2cd0d93700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2ccc0924b0 0x7f2ccc1a3590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:29.484 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:29.460 7f2cd0d93700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2ccc0924b0 0x7f2ccc1a3590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:44512/0 (socket says 172.21.2.112:44512) 2021-02-10T11:19:29.486 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:29.460 7f2cd0d93700 1 -- 172.21.2.112:0/1458037007 learned_addr learned my addr 172.21.2.112:0/1458037007 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:19:29.486 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:29.460 7f2cd0d93700 1 -- 172.21.2.112:0/1458037007 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2ccc0bff40 msgr2=0x7f2ccc0c0340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:29.487 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:29.460 7f2cd0d93700 1 --2- 172.21.2.112:0/1458037007 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2ccc0bff40 0x7f2ccc0c0340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:19:29.487 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:29.460 7f2cd0d93700 1 -- 172.21.2.112:0/1458037007 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2ccc1a3bb0 msgr2=0x7f2ccc0bf920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:29.487 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:29.460 7f2cd0d93700 1 --2- 172.21.2.112:0/1458037007 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2ccc1a3bb0 0x7f2ccc0bf920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:19:29.488 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:29.460 7f2cd0d93700 1 -- 172.21.2.112:0/1458037007 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ccc098640 con 0x7f2ccc0924b0 2021-02-10T11:19:29.488 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:29.461 7f2cd0d93700 1 --2- 172.21.2.112:0/1458037007 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2ccc0924b0 0x7f2ccc1a3590 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 rx=0x7f2cb400b5c0 tx=0x7f2cb400b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:19:29.489 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:29.461 7f2cc97fa700 1 -- 172.21.2.112:0/1458037007 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2cb40075b0 con 0x7f2ccc0924b0 2021-02-10T11:19:29.489 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:29.461 7f2cc97fa700 1 -- 172.21.2.112:0/1458037007 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cb4000c90 con 0x7f2ccc0924b0 2021-02-10T11:19:29.489 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:29.461 7f2cc97fa700 1 -- 172.21.2.112:0/1458037007 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2cb4007bd0 con 0x7f2ccc0924b0 2021-02-10T11:19:29.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:29.461 7f2cd2f77700 1 -- 172.21.2.112:0/1458037007 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ccc098640 con 0x7f2ccc0924b0 2021-02-10T11:19:29.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:29.461 7f2cd2f77700 1 -- 172.21.2.112:0/1458037007 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ccc098bd0 con 0x7f2ccc0924b0 2021-02-10T11:19:29.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:29.461 7f2cc97fa700 1 -- 172.21.2.112:0/1458037007 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2cb4000c90 con 0x7f2ccc0924b0 2021-02-10T11:19:29.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:29.463 7f2cc97fa700 1 --2- 172.21.2.112:0/1458037007 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2cb00186c0 0x7f2cb001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:29.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:29.463 7f2cc97fa700 1 -- 172.21.2.112:0/1458037007 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f2cb4032590 con 0x7f2ccc0924b0 2021-02-10T11:19:29.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:29.463 7f2ccbfff700 1 --2- 172.21.2.112:0/1458037007 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2cb00186c0 0x7f2cb001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:29.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:29.463 7f2ccbfff700 1 --2- 172.21.2.112:0/1458037007 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2cb00186c0 0x7f2cb001abc0 crc :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:19:29.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:29.492 7f2cd2f77700 1 -- 172.21.2.112:0/1458037007 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ccc105b40 con 0x7f2ccc0924b0 2021-02-10T11:19:29.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:29.495 7f2cc97fa700 1 -- 172.21.2.112:0/1458037007 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2cb4030d40 con 0x7f2ccc0924b0 2021-02-10T11:19:29.619 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:29.623 7f2cd2f77700 1 -- 172.21.2.112:0/1458037007 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348593", "target": ["mgr", ""]}) v1 -- 0x7f2ccc098bd0 con 0x7f2cb00186c0 2021-02-10T11:19:29.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:29.628 7f2cc97fa700 1 -- 172.21.2.112:0/1458037007 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f2cbc0077a0 con 0x7f2cb00186c0 2021-02-10T11:19:29.627 INFO:teuthology.orchestra.run.gibba012.stdout:/volumes/_nogroup/subvolume_0000000000348593/25c8a093-c2b7-4bd5-a8e8-cca7ed8a181f 2021-02-10T11:19:29.629 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:29.633 7f2cd2f77700 1 -- 172.21.2.112:0/1458037007 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2cb00186c0 msgr2=0x7f2cb001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:29.629 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:29.633 7f2cd2f77700 1 --2- 172.21.2.112:0/1458037007 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2cb00186c0 0x7f2cb001abc0 crc :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:29.629 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:29.633 7f2cd2f77700 1 -- 172.21.2.112:0/1458037007 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2ccc0924b0 msgr2=0x7f2ccc1a3590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:29.629 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:29.633 7f2cd2f77700 1 --2- 172.21.2.112:0/1458037007 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2ccc0924b0 0x7f2ccc1a3590 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 rx=0x7f2cb400b5c0 tx=0x7f2cb400b960).stop 2021-02-10T11:19:29.629 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:29.633 7f2cd2f77700 1 -- 172.21.2.112:0/1458037007 shutdown_connections 2021-02-10T11:19:29.629 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:29.633 7f2cd2f77700 1 --2- 172.21.2.112:0/1458037007 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2cb00186c0 0x7f2cb001abc0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:29.630 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:29.633 7f2cd2f77700 1 --2- 172.21.2.112:0/1458037007 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2ccc1a3bb0 0x7f2ccc0bf920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:19:29.630 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:29.633 7f2cd2f77700 1 --2- 172.21.2.112:0/1458037007 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2ccc0bff40 0x7f2ccc0c0340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:19:29.630 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:29.633 7f2cd2f77700 1 --2- 172.21.2.112:0/1458037007 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2ccc0924b0 0x7f2ccc1a3590 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:29.630 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:29.633 7f2cd2f77700 1 -- 172.21.2.112:0/1458037007 >> 172.21.2.112:0/1458037007 conn(0x7f2ccc0b50c0 msgr2=0x7f2ccc092d90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:19:29.630 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:29.633 7f2cd2f77700 1 -- 172.21.2.112:0/1458037007 shutdown_connections 2021-02-10T11:19:29.630 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:29.633 7f2cd2f77700 1 -- 172.21.2.112:0/1458037007 wait complete. 2021-02-10T11:19:29.637 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000348593 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000348593/25c8a093-c2b7-4bd5-a8e8-cca7ed8a181f 2021-02-10T11:19:29.637 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:29.637 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:29.637 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:29.981 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:19:29.981 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:29.981 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348593/25c8a093-c2b7-4bd5-a8e8-cca7ed8a181f/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:30.358 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:19:30.359 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:19:30.359 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0222736 s, 47.1 MB/s 2021-02-10T11:19:30.359 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:30.360 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:30.360 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:30.696 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:19:30.696 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:30.696 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348593/25c8a093-c2b7-4bd5-a8e8-cca7ed8a181f/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:31.069 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:19:31.069 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:19:31.069 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0231892 s, 45.2 MB/s 2021-02-10T11:19:31.070 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:31.070 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:31.070 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:31.406 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:19:31.406 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:31.407 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348593/25c8a093-c2b7-4bd5-a8e8-cca7ed8a181f/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:31.782 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:19:31.782 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:19:31.783 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0236497 s, 44.3 MB/s 2021-02-10T11:19:31.783 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:31.784 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:31.784 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:32.118 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:19:32.118 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:32.118 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348593/25c8a093-c2b7-4bd5-a8e8-cca7ed8a181f/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:32.492 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:19:32.492 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:19:32.492 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0229503 s, 45.7 MB/s 2021-02-10T11:19:32.493 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:32.493 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:32.493 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:32.827 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:19:32.827 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:32.827 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348593/25c8a093-c2b7-4bd5-a8e8-cca7ed8a181f/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:33.202 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:19:33.202 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:19:33.202 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0233693 s, 44.9 MB/s 2021-02-10T11:19:33.203 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:33.203 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:33.203 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:33.538 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:19:33.538 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:33.539 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348593/25c8a093-c2b7-4bd5-a8e8-cca7ed8a181f/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:33.913 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:19:33.914 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:19:33.914 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0246397 s, 42.6 MB/s 2021-02-10T11:19:33.915 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:33.915 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:33.915 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:34.251 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:19:34.252 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:34.252 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348593/25c8a093-c2b7-4bd5-a8e8-cca7ed8a181f/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:34.626 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:19:34.830 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:19:34.830 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0253856 s, 41.3 MB/s 2021-02-10T11:19:34.831 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:34.831 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:34.831 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:35.167 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:19:35.167 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:35.167 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348593/25c8a093-c2b7-4bd5-a8e8-cca7ed8a181f/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:35.233 DEBUG:teuthology.orchestra.run.gibba012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:19:35.236 DEBUG:teuthology.orchestra.run.gibba020:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:19:35.542 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:19:35.542 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:19:35.542 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0241272 s, 43.5 MB/s 2021-02-10T11:19:35.543 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:35.544 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:35.544 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:35.919 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:19:35.919 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:35.919 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348593/25c8a093-c2b7-4bd5-a8e8-cca7ed8a181f/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:36.293 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:19:36.293 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:19:36.294 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0234414 s, 44.7 MB/s 2021-02-10T11:19:36.294 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:36.295 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:36.295 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:36.629 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:19:36.629 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:36.629 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348593/25c8a093-c2b7-4bd5-a8e8-cca7ed8a181f/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:37.003 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:19:37.004 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:19:37.004 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0248233 s, 42.2 MB/s 2021-02-10T11:19:37.005 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000348594 2021-02-10T11:19:37.435 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:37.424 7f57a50fa700 1 Processor -- start 2021-02-10T11:19:37.435 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:37.424 7f57a50fa700 1 -- start start 2021-02-10T11:19:37.435 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:37.425 7f57a50fa700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f57a01b0470 0x7f57a01b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:37.436 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:37.425 7f57a50fa700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f57a00ef610 0x7f57a00efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:37.436 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:37.425 7f57a50fa700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57a00f0b50 con 0x7f57a00f0030 2021-02-10T11:19:37.436 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:37.425 7f57a50fa700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57a00e52e0 con 0x7f57a01b0470 2021-02-10T11:19:37.436 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:37.425 7f57a50fa700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57a00e5490 con 0x7f57a00ef610 2021-02-10T11:19:37.436 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:37.425 7f579ed9d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f57a01b0470 0x7f57a01b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:37.436 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:37.425 7f579ed9d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f57a01b0470 0x7f57a01b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:44518/0 (socket says 172.21.2.112:44518) 2021-02-10T11:19:37.436 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:37.425 7f579ed9d700 1 -- 172.21.2.112:0/620736594 learned_addr learned my addr 172.21.2.112:0/620736594 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:19:37.436 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:37.425 7f579ed9d700 1 -- 172.21.2.112:0/620736594 >> v2:172.21.2.120:3300/0 conn(0x7f57a00ef610 msgr2=0x7f57a00efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:19:37.436 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:37.425 7f579ed9d700 1 --2- 172.21.2.112:0/620736594 >> v2:172.21.2.120:3300/0 conn(0x7f57a00ef610 0x7f57a00efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:19:37.436 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:37.425 7f579ed9d700 1 -- 172.21.2.112:0/620736594 >> v1:172.21.2.120:6789/0 conn(0x7f57a00f0030 legacy=0x7f57a00f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:19:37.436 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:37.425 7f579ed9d700 1 -- 172.21.2.112:0/620736594 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57a00f0430 con 0x7f57a01b0470 2021-02-10T11:19:37.437 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:37.425 7f579ed9d700 1 --2- 172.21.2.112:0/620736594 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f57a01b0470 0x7f57a01b0870 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7f5790006700 tx=0x7f5790009b00).ready entity=mon.0 client_cookie=e0a4db584be92800 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:19:37.437 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:37.425 7f579dd9b700 1 -- 172.21.2.112:0/620736594 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f579000c0c0 con 0x7f57a01b0470 2021-02-10T11:19:37.437 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:37.425 7f579dd9b700 1 -- 172.21.2.112:0/620736594 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f579000c400 con 0x7f57a01b0470 2021-02-10T11:19:37.437 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:37.425 7f579dd9b700 1 -- 172.21.2.112:0/620736594 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f579000c8e0 con 0x7f57a01b0470 2021-02-10T11:19:37.437 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:37.425 7f57a50fa700 1 -- 172.21.2.112:0/620736594 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f57a01b0470 msgr2=0x7f57a01b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:37.437 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:37.425 7f57a50fa700 1 --2- 172.21.2.112:0/620736594 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f57a01b0470 0x7f57a01b0870 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7f5790006700 tx=0x7f5790009b00).stop 2021-02-10T11:19:37.437 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:37.426 7f57a50fa700 1 -- 172.21.2.112:0/620736594 shutdown_connections 2021-02-10T11:19:37.437 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:37.426 7f57a50fa700 1 --2- 172.21.2.112:0/620736594 >> v2:172.21.2.120:3300/0 conn(0x7f57a00ef610 0x7f57a00efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:19:37.437 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:37.426 7f57a50fa700 1 --2- 172.21.2.112:0/620736594 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f57a01b0470 0x7f57a01b0870 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:37.437 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:37.426 7f57a50fa700 1 -- 172.21.2.112:0/620736594 >> 172.21.2.112:0/620736594 conn(0x7f57a00ce3e0 msgr2=0x7f57a00e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:19:37.437 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:37.426 7f57a50fa700 1 -- 172.21.2.112:0/620736594 shutdown_connections 2021-02-10T11:19:37.437 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:37.426 7f57a50fa700 1 -- 172.21.2.112:0/620736594 wait complete. 2021-02-10T11:19:37.439 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:37.440 7f57a50fa700 1 Processor -- start 2021-02-10T11:19:37.439 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:37.440 7f57a50fa700 1 -- start start 2021-02-10T11:19:37.439 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:37.441 7f57a50fa700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f57a00c36a0 0x7f57a00c2b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:37.439 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:37.441 7f57a50fa700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f57a00c3190 0x7f57a00b1d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:37.440 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:37.441 7f57a50fa700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f57a00b2370 0x7f57a00b2d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:37.440 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:37.441 7f57a50fa700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57a00b27c0 con 0x7f57a00c3190 2021-02-10T11:19:37.440 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:37.441 7f57a50fa700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57a00b3360 con 0x7f57a00b2370 2021-02-10T11:19:37.440 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:37.441 7f57a50fa700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57a00b3540 con 0x7f57a00c36a0 2021-02-10T11:19:37.440 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:37.441 7f579e59c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f57a00c3190 0x7f57a00b1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:37.440 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:37.441 7f579e59c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f57a00c3190 0x7f57a00b1d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:44524/0 (socket says 172.21.2.112:44524) 2021-02-10T11:19:37.440 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:37.441 7f579e59c700 1 -- 172.21.2.112:0/3380108330 learned_addr learned my addr 172.21.2.112:0/3380108330 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:19:37.440 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:37.441 7f579e59c700 1 -- 172.21.2.112:0/3380108330 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f57a00c36a0 msgr2=0x7f57a00c2b70 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:19:37.440 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:37.441 7f579e59c700 1 --2- 172.21.2.112:0/3380108330 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f57a00c36a0 0x7f57a00c2b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:19:37.440 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:37.441 7f579e59c700 1 -- 172.21.2.112:0/3380108330 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f57a00b2370 msgr2=0x7f57a00b2d40 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:19:37.440 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:37.441 7f579e59c700 1 --2- 172.21.2.112:0/3380108330 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f57a00b2370 0x7f57a00b2d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:19:37.441 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:37.441 7f579e59c700 1 -- 172.21.2.112:0/3380108330 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57a00b3760 con 0x7f57a00c3190 2021-02-10T11:19:37.441 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:37.441 7f579e59c700 1 --2- 172.21.2.112:0/3380108330 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f57a00c3190 0x7f57a00b1d50 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 rx=0x7f579400b5b0 tx=0x7f579400b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:19:37.441 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:37.441 7f578f7fe700 1 -- 172.21.2.112:0/3380108330 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f579400bd40 con 0x7f57a00c3190 2021-02-10T11:19:37.441 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:37.441 7f578f7fe700 1 -- 172.21.2.112:0/3380108330 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5794002080 con 0x7f57a00c3190 2021-02-10T11:19:37.441 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:37.441 7f57a50fa700 1 -- 172.21.2.112:0/3380108330 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57a00b3760 con 0x7f57a00c3190 2021-02-10T11:19:37.441 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:37.441 7f578f7fe700 1 -- 172.21.2.112:0/3380108330 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5794002520 con 0x7f57a00c3190 2021-02-10T11:19:37.441 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:37.441 7f57a50fa700 1 -- 172.21.2.112:0/3380108330 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57a00b3cf0 con 0x7f57a00c3190 2021-02-10T11:19:37.441 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:37.442 7f578f7fe700 1 -- 172.21.2.112:0/3380108330 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5794000c90 con 0x7f57a00c3190 2021-02-10T11:19:37.441 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:37.442 7f578f7fe700 1 --2- 172.21.2.112:0/3380108330 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f57840186c0 0x7f578401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:37.441 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:37.442 7f578f7fe700 1 -- 172.21.2.112:0/3380108330 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f5794032550 con 0x7f57a00c3190 2021-02-10T11:19:37.441 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:37.443 7f579ed9d700 1 --2- 172.21.2.112:0/3380108330 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f57840186c0 0x7f578401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:37.441 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:37.443 7f579ed9d700 1 --2- 172.21.2.112:0/3380108330 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f57840186c0 0x7f578401abc0 crc :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:19:37.470 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:37.474 7f57a50fa700 1 -- 172.21.2.112:0/3380108330 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57a00edd10 con 0x7f57a00c3190 2021-02-10T11:19:37.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:37.477 7f578f7fe700 1 -- 172.21.2.112:0/3380108330 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5794002df0 con 0x7f57a00c3190 2021-02-10T11:19:37.598 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:37.603 7f57a50fa700 1 -- 172.21.2.112:0/3380108330 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348594", "target": ["mgr", ""]}) v1 -- 0x7f57a00b27c0 con 0x7f57840186c0 2021-02-10T11:19:37.624 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:37.629 7f578f7fe700 1 -- 172.21.2.112:0/3380108330 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f5790009440 con 0x7f57840186c0 2021-02-10T11:19:37.632 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:37.636 7f57a50fa700 1 -- 172.21.2.112:0/3380108330 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f57840186c0 msgr2=0x7f578401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:37.632 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:37.636 7f57a50fa700 1 --2- 172.21.2.112:0/3380108330 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f57840186c0 0x7f578401abc0 crc :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:37.632 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:37.636 7f57a50fa700 1 -- 172.21.2.112:0/3380108330 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f57a00c3190 msgr2=0x7f57a00b1d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:37.632 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:37.636 7f57a50fa700 1 --2- 172.21.2.112:0/3380108330 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f57a00c3190 0x7f57a00b1d50 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 rx=0x7f579400b5b0 tx=0x7f579400b950).stop 2021-02-10T11:19:37.632 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:37.636 7f57a50fa700 1 -- 172.21.2.112:0/3380108330 shutdown_connections 2021-02-10T11:19:37.632 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:37.636 7f57a50fa700 1 --2- 172.21.2.112:0/3380108330 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f57840186c0 0x7f578401abc0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:37.632 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:37.636 7f57a50fa700 1 --2- 172.21.2.112:0/3380108330 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f57a00b2370 0x7f57a00b2d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:19:37.632 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:37.636 7f57a50fa700 1 --2- 172.21.2.112:0/3380108330 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f57a00c3190 0x7f57a00b1d50 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:37.632 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:37.636 7f57a50fa700 1 --2- 172.21.2.112:0/3380108330 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f57a00c36a0 0x7f57a00c2b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:19:37.632 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:37.636 7f57a50fa700 1 -- 172.21.2.112:0/3380108330 >> 172.21.2.112:0/3380108330 conn(0x7f57a009f630 msgr2=0x7f57a00a0130 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:19:37.632 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:37.636 7f57a50fa700 1 -- 172.21.2.112:0/3380108330 shutdown_connections 2021-02-10T11:19:37.632 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:37.636 7f57a50fa700 1 -- 172.21.2.112:0/3380108330 wait complete. 2021-02-10T11:19:37.640 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000348594 2021-02-10T11:19:38.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:38.059 7fe6f811a700 1 Processor -- start 2021-02-10T11:19:38.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:38.059 7fe6f811a700 1 -- start start 2021-02-10T11:19:38.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:38.059 7fe6f811a700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fe6f01b0470 0x7fe6f01b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:38.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:38.059 7fe6f811a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe6f00f20e0 0x7fe6f00f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:38.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:38.059 7fe6f811a700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6f00f0b50 con 0x7fe6f00ef610 2021-02-10T11:19:38.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:38.059 7fe6f811a700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6f00f0820 con 0x7fe6f01b0470 2021-02-10T11:19:38.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:38.059 7fe6f811a700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6f00f24e0 con 0x7fe6f00f20e0 2021-02-10T11:19:38.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:38.060 7fe6f5735700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7fe6f00ef610 0x7fe6f00efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:37552/0 (socket says 172.21.2.112:37552) 2021-02-10T11:19:38.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:38.060 7fe6f5f36700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fe6f01b0470 0x7fe6f01b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:38.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:38.060 7fe6f5735700 1 -- 172.21.2.112:0/106202242 learned_addr learned my addr 172.21.2.112:0/106202242 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:19:38.073 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:38.060 7fe6f6737700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe6f00f20e0 0x7fe6f00f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:38.073 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:38.061 7fe6f6737700 1 -- 172.21.2.112:0/106202242 >> v2:172.21.2.120:3300/0 conn(0x7fe6f01b0470 msgr2=0x7fe6f01b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:19:38.073 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:38.061 7fe6f6737700 1 --2- 172.21.2.112:0/106202242 >> v2:172.21.2.120:3300/0 conn(0x7fe6f01b0470 0x7fe6f01b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:38.073 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:38.061 7fe6f6737700 1 -- 172.21.2.112:0/106202242 >> v1:172.21.2.120:6789/0 conn(0x7fe6f00ef610 legacy=0x7fe6f00efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:38.073 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:38.061 7fe6f6737700 1 -- 172.21.2.112:0/106202242 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6f00e52e0 con 0x7fe6f00f20e0 2021-02-10T11:19:38.073 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:38.061 7fe6f6737700 1 --2- 172.21.2.112:0/106202242 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe6f00f20e0 0x7fe6f00f0200 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7fe6e8006e50 tx=0x7fe6e8009e70).ready entity=mon.2 client_cookie=f61321b56bcb6d83 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:19:38.073 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:38.061 7fe6f4f34700 1 -- 172.21.2.112:0/106202242 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe6e8006a20 con 0x7fe6f00f20e0 2021-02-10T11:19:38.073 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:38.061 7fe6f4f34700 1 -- 172.21.2.112:0/106202242 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6e8006790 con 0x7fe6f00f20e0 2021-02-10T11:19:38.073 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:38.061 7fe6f4f34700 1 -- 172.21.2.112:0/106202242 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe6e800b700 con 0x7fe6f00f20e0 2021-02-10T11:19:38.073 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:38.061 7fe6f811a700 1 -- 172.21.2.112:0/106202242 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe6f00f20e0 msgr2=0x7fe6f00f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:38.073 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:38.061 7fe6f811a700 1 --2- 172.21.2.112:0/106202242 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe6f00f20e0 0x7fe6f00f0200 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7fe6e8006e50 tx=0x7fe6e8009e70).stop 2021-02-10T11:19:38.073 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:38.062 7fe6f811a700 1 -- 172.21.2.112:0/106202242 shutdown_connections 2021-02-10T11:19:38.073 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:38.062 7fe6f811a700 1 --2- 172.21.2.112:0/106202242 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe6f00f20e0 0x7fe6f00f0200 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:38.074 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:38.062 7fe6f811a700 1 --2- 172.21.2.112:0/106202242 >> v2:172.21.2.120:3300/0 conn(0x7fe6f01b0470 0x7fe6f01b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:38.074 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:38.062 7fe6f811a700 1 -- 172.21.2.112:0/106202242 >> 172.21.2.112:0/106202242 conn(0x7fe6f00ce3e0 msgr2=0x7fe6f00e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:19:38.074 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:38.062 7fe6f811a700 1 -- 172.21.2.112:0/106202242 shutdown_connections 2021-02-10T11:19:38.074 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:38.062 7fe6f811a700 1 -- 172.21.2.112:0/106202242 wait complete. 2021-02-10T11:19:38.075 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:38.077 7fe6f811a700 1 Processor -- start 2021-02-10T11:19:38.075 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:38.077 7fe6f811a700 1 -- start start 2021-02-10T11:19:38.076 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:38.077 7fe6f811a700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe6f0105d10 0x7fe6f009eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:38.076 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:38.077 7fe6f811a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe6f0106110 0x7fe6f009f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:38.076 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:38.077 7fe6f811a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe6f00b2a00 0x7fe6f00b2e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:38.076 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:38.077 7fe6f811a700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6f009e560 con 0x7fe6f0106110 2021-02-10T11:19:38.076 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:38.077 7fe6f811a700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6f009c090 con 0x7fe6f0105d10 2021-02-10T11:19:38.076 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:38.077 7fe6f811a700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6f00b3420 con 0x7fe6f00b2a00 2021-02-10T11:19:38.076 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:38.077 7fe6f5735700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe6f0106110 0x7fe6f009f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:38.076 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:38.077 7fe6f5735700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe6f0106110 0x7fe6f009f100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:44540/0 (socket says 172.21.2.112:44540) 2021-02-10T11:19:38.076 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:38.077 7fe6f5735700 1 -- 172.21.2.112:0/3186429784 learned_addr learned my addr 172.21.2.112:0/3186429784 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:19:38.076 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:38.078 7fe6f5735700 1 -- 172.21.2.112:0/3186429784 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe6f00b2a00 msgr2=0x7fe6f00b2e00 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:19:38.076 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:38.078 7fe6f5735700 1 --2- 172.21.2.112:0/3186429784 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe6f00b2a00 0x7fe6f00b2e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:19:38.076 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:38.078 7fe6f5735700 1 -- 172.21.2.112:0/3186429784 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe6f0105d10 msgr2=0x7fe6f009eae0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:19:38.076 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:38.078 7fe6f5735700 1 --2- 172.21.2.112:0/3186429784 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe6f0105d10 0x7fe6f009eae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:19:38.077 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:38.078 7fe6f5735700 1 -- 172.21.2.112:0/3186429784 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6f00b3640 con 0x7fe6f0106110 2021-02-10T11:19:38.077 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:38.078 7fe6f5735700 1 --2- 172.21.2.112:0/3186429784 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe6f0106110 0x7fe6f009f100 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7fe6ec001c60 tx=0x7fe6ec00a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:19:38.077 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:38.078 7fe6e27fc700 1 -- 172.21.2.112:0/3186429784 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe6ec00acc0 con 0x7fe6f0106110 2021-02-10T11:19:38.077 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:38.078 7fe6e27fc700 1 -- 172.21.2.112:0/3186429784 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6ec004140 con 0x7fe6f0106110 2021-02-10T11:19:38.077 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:38.078 7fe6f811a700 1 -- 172.21.2.112:0/3186429784 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6f00b3640 con 0x7fe6f0106110 2021-02-10T11:19:38.077 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:38.078 7fe6e27fc700 1 -- 172.21.2.112:0/3186429784 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe6ec004620 con 0x7fe6f0106110 2021-02-10T11:19:38.077 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:38.078 7fe6f811a700 1 -- 172.21.2.112:0/3186429784 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6f00b3640 con 0x7fe6f0106110 2021-02-10T11:19:38.077 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:38.079 7fe6e27fc700 1 -- 172.21.2.112:0/3186429784 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe6ec000ce0 con 0x7fe6f0106110 2021-02-10T11:19:38.077 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:38.079 7fe6e27fc700 1 --2- 172.21.2.112:0/3186429784 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe6d8018730 0x7fe6d801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:38.077 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:38.079 7fe6e27fc700 1 -- 172.21.2.112:0/3186429784 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fe6ec032630 con 0x7fe6f0106110 2021-02-10T11:19:38.077 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:38.079 7fe6f5f36700 1 --2- 172.21.2.112:0/3186429784 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe6d8018730 0x7fe6d801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:38.077 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:38.080 7fe6f5f36700 1 --2- 172.21.2.112:0/3186429784 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe6d8018730 0x7fe6d801ac10 crc :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:19:38.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:38.110 7fe6f811a700 1 -- 172.21.2.112:0/3186429784 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6f0105ae0 con 0x7fe6f0106110 2021-02-10T11:19:38.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:38.112 7fe6e27fc700 1 -- 172.21.2.112:0/3186429784 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe6ec0009b0 con 0x7fe6f0106110 2021-02-10T11:19:38.238 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:38.243 7fe6f811a700 1 -- 172.21.2.112:0/3186429784 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348594", "target": ["mgr", ""]}) v1 -- 0x7fe6f01a16c0 con 0x7fe6d8018730 2021-02-10T11:19:38.241 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:38.246 7fe6e27fc700 1 -- 172.21.2.112:0/3186429784 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7fe6e4000940 con 0x7fe6d8018730 2021-02-10T11:19:38.245 INFO:teuthology.orchestra.run.gibba012.stdout:/volumes/_nogroup/subvolume_0000000000348594/8643b3e2-e693-4601-8eba-a88ea9decad4 2021-02-10T11:19:38.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:38.251 7fe6f811a700 1 -- 172.21.2.112:0/3186429784 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe6d8018730 msgr2=0x7fe6d801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:38.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:38.251 7fe6f811a700 1 --2- 172.21.2.112:0/3186429784 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe6d8018730 0x7fe6d801ac10 crc :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:38.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:38.251 7fe6f811a700 1 -- 172.21.2.112:0/3186429784 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe6f0106110 msgr2=0x7fe6f009f100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:38.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:38.251 7fe6f811a700 1 --2- 172.21.2.112:0/3186429784 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe6f0106110 0x7fe6f009f100 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7fe6ec001c60 tx=0x7fe6ec00a8d0).stop 2021-02-10T11:19:38.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:38.252 7fe6f811a700 1 -- 172.21.2.112:0/3186429784 shutdown_connections 2021-02-10T11:19:38.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:38.252 7fe6f811a700 1 --2- 172.21.2.112:0/3186429784 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe6d8018730 0x7fe6d801ac10 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:38.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:38.252 7fe6f811a700 1 --2- 172.21.2.112:0/3186429784 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe6f00b2a00 0x7fe6f00b2e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:19:38.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:38.252 7fe6f811a700 1 --2- 172.21.2.112:0/3186429784 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe6f0106110 0x7fe6f009f100 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:38.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:38.252 7fe6f811a700 1 --2- 172.21.2.112:0/3186429784 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe6f0105d10 0x7fe6f009eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:19:38.248 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:38.252 7fe6f811a700 1 -- 172.21.2.112:0/3186429784 >> 172.21.2.112:0/3186429784 conn(0x7fe6f00c36f0 msgr2=0x7fe6f00bd640 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:19:38.248 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:38.252 7fe6f811a700 1 -- 172.21.2.112:0/3186429784 shutdown_connections 2021-02-10T11:19:38.248 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:38.252 7fe6f811a700 1 -- 172.21.2.112:0/3186429784 wait complete. 2021-02-10T11:19:38.255 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000348594 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000348594/8643b3e2-e693-4601-8eba-a88ea9decad4 2021-02-10T11:19:38.255 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:38.255 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:38.255 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:38.591 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:19:38.591 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:38.591 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348594/8643b3e2-e693-4601-8eba-a88ea9decad4/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:38.971 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:19:38.971 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:19:38.971 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0234174 s, 44.8 MB/s 2021-02-10T11:19:38.972 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:38.972 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:38.973 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:39.307 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:19:39.308 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:39.308 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348594/8643b3e2-e693-4601-8eba-a88ea9decad4/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:39.680 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:19:39.680 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:19:39.680 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0244643 s, 42.9 MB/s 2021-02-10T11:19:39.681 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:39.681 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:39.681 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:40.017 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:19:40.017 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:40.017 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348594/8643b3e2-e693-4601-8eba-a88ea9decad4/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:40.391 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:19:40.391 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:19:40.391 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0227543 s, 46.1 MB/s 2021-02-10T11:19:40.392 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:40.392 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:40.392 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:40.728 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:19:40.729 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:40.729 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348594/8643b3e2-e693-4601-8eba-a88ea9decad4/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:41.103 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:19:41.103 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:19:41.103 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.023565 s, 44.5 MB/s 2021-02-10T11:19:41.104 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:41.104 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:41.104 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:41.440 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:19:41.440 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:41.440 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348594/8643b3e2-e693-4601-8eba-a88ea9decad4/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:41.815 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:19:41.815 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:19:41.815 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0247074 s, 42.4 MB/s 2021-02-10T11:19:41.816 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:41.816 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:41.817 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:42.153 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:19:42.153 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:42.153 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348594/8643b3e2-e693-4601-8eba-a88ea9decad4/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:42.529 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:19:42.530 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:19:42.530 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0249875 s, 42.0 MB/s 2021-02-10T11:19:42.531 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:42.531 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:42.531 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:42.867 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:19:42.867 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:42.867 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348594/8643b3e2-e693-4601-8eba-a88ea9decad4/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:43.244 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:19:43.244 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:19:43.244 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0257179 s, 40.8 MB/s 2021-02-10T11:19:43.245 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:43.245 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:43.245 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:43.581 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:19:43.581 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:43.581 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348594/8643b3e2-e693-4601-8eba-a88ea9decad4/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:43.955 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:19:43.955 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:19:43.955 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0244642 s, 42.9 MB/s 2021-02-10T11:19:43.957 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:43.957 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:43.957 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:44.291 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:19:44.291 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:44.291 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348594/8643b3e2-e693-4601-8eba-a88ea9decad4/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:44.664 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:19:44.664 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:19:44.664 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0255769 s, 41.0 MB/s 2021-02-10T11:19:44.665 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:44.665 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:44.665 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:44.999 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:19:44.999 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:44.999 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348594/8643b3e2-e693-4601-8eba-a88ea9decad4/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:45.375 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:19:45.375 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:19:45.376 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0241759 s, 43.4 MB/s 2021-02-10T11:19:45.377 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000348595 2021-02-10T11:19:45.808 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:45.795 7fd38019a700 1 Processor -- start 2021-02-10T11:19:45.808 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:45.795 7fd38019a700 1 -- start start 2021-02-10T11:19:45.808 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:45.796 7fd38019a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd3781b0870 0x7fd3780f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:45.808 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:45.796 7fd38019a700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fd3780efe00 0x7fd3780f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:45.808 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:45.796 7fd38019a700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3781b0c70 con 0x7fd3781b0470 2021-02-10T11:19:45.808 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:45.796 7fd38019a700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3780f0b50 con 0x7fd3780efe00 2021-02-10T11:19:45.809 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:45.796 7fd38019a700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3780f0820 con 0x7fd3781b0870 2021-02-10T11:19:45.809 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:45.796 7fd37dfb6700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fd3781b0470 0x7fd3780ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:44010/0 (socket says 172.21.2.112:44010) 2021-02-10T11:19:45.809 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:45.796 7fd37dfb6700 1 -- 172.21.2.112:0/796702164 learned_addr learned my addr 172.21.2.112:0/796702164 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:19:45.809 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:45.796 7fd37cfb4700 1 -- 172.21.2.112:0/796702164 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 739457348 0 0) 0x7fd374000ce0 con 0x7fd3781b0470 2021-02-10T11:19:45.809 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:45.796 7fd37cfb4700 1 -- 172.21.2.112:0/796702164 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd364002fe0 con 0x7fd3781b0470 2021-02-10T11:19:45.809 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:45.796 7fd37e7b7700 1 --2- 172.21.2.112:0/796702164 >> v2:172.21.2.120:3300/0 conn(0x7fd3780efe00 0x7fd3780f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:45.809 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:45.796 7fd37d7b5700 1 --2- 172.21.2.112:0/796702164 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd3781b0870 0x7fd3780f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:45.809 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:45.797 7fd37cfb4700 1 -- 172.21.2.112:0/796702164 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1165295512 0 0) 0x7fd3740010e0 con 0x7fd3781b0470 2021-02-10T11:19:45.809 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:45.797 7fd37cfb4700 1 -- 172.21.2.112:0/796702164 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd3781b0870 msgr2=0x7fd3780f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:19:45.809 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:45.797 7fd37cfb4700 1 --2- 172.21.2.112:0/796702164 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd3781b0870 0x7fd3780f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:45.809 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:45.797 7fd37cfb4700 1 -- 172.21.2.112:0/796702164 >> v2:172.21.2.120:3300/0 conn(0x7fd3780efe00 msgr2=0x7fd3780f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:19:45.809 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:45.797 7fd37cfb4700 1 --2- 172.21.2.112:0/796702164 >> v2:172.21.2.120:3300/0 conn(0x7fd3780efe00 0x7fd3780f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:45.810 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:45.797 7fd37cfb4700 1 -- 172.21.2.112:0/796702164 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3780e52e0 con 0x7fd3781b0470 2021-02-10T11:19:45.810 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:45.797 7fd37cfb4700 1 -- 172.21.2.112:0/796702164 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7fd374001660 con 0x7fd3781b0470 2021-02-10T11:19:45.810 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:45.797 7fd37cfb4700 1 -- 172.21.2.112:0/796702164 >> v1:172.21.2.112:6789/0 conn(0x7fd3781b0470 legacy=0x7fd3780ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:45.810 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:45.797 7fd37cfb4700 1 --2- 172.21.2.112:0/796702164 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd364003af0 0x7fd364005f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:45.810 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:45.797 7fd37cfb4700 1 --2- 172.21.2.112:0/796702164 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd36404a790 0x7fd36404cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:45.810 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:45.797 7fd37cfb4700 1 --2- 172.21.2.112:0/796702164 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd36404d500 0x7fd36404fa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:45.810 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:45.797 7fd37cfb4700 1 -- 172.21.2.112:0/796702164 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3640502d0 con 0x7fd36404d500 2021-02-10T11:19:45.810 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:45.797 7fd37cfb4700 1 -- 172.21.2.112:0/796702164 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3640504f0 con 0x7fd364003af0 2021-02-10T11:19:45.810 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:45.797 7fd37cfb4700 1 -- 172.21.2.112:0/796702164 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd364050710 con 0x7fd36404a790 2021-02-10T11:19:45.810 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:45.797 7fd37dfb6700 1 --2- 172.21.2.112:0/796702164 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd364003af0 0x7fd364005f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:45.810 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:45.797 7fd37e7b7700 1 --2- 172.21.2.112:0/796702164 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd36404d500 0x7fd36404fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:45.810 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:45.797 7fd37d7b5700 1 --2- 172.21.2.112:0/796702164 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd36404a790 0x7fd36404cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:45.810 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:45.797 7fd37e7b7700 1 -- 172.21.2.112:0/796702164 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd36404a790 msgr2=0x7fd36404cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:19:45.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:45.797 7fd37e7b7700 1 --2- 172.21.2.112:0/796702164 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd36404a790 0x7fd36404cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:45.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:45.797 7fd37e7b7700 1 -- 172.21.2.112:0/796702164 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd364003af0 msgr2=0x7fd364005f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:19:45.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:45.797 7fd37e7b7700 1 --2- 172.21.2.112:0/796702164 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd364003af0 0x7fd364005f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:45.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:45.797 7fd37e7b7700 1 -- 172.21.2.112:0/796702164 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3640509b0 con 0x7fd36404d500 2021-02-10T11:19:45.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:45.797 7fd37d7b5700 1 --2- 172.21.2.112:0/796702164 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd36404a790 0x7fd36404cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:19:45.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:45.797 7fd37e7b7700 1 --2- 172.21.2.112:0/796702164 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd36404d500 0x7fd36404fa20 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7fd370006a40 tx=0x7fd37000c0c0).ready entity=mon.0 client_cookie=556ee1adef8aa487 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:19:45.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:45.797 7fd37cfb4700 1 -- 172.21.2.112:0/796702164 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd37000c4b0 con 0x7fd36404d500 2021-02-10T11:19:45.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:45.797 7fd37cfb4700 1 -- 172.21.2.112:0/796702164 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd37000c770 con 0x7fd36404d500 2021-02-10T11:19:45.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:45.797 7fd37cfb4700 1 -- 172.21.2.112:0/796702164 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd37000cc50 con 0x7fd36404d500 2021-02-10T11:19:45.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:45.797 7fd38019a700 1 -- 172.21.2.112:0/796702164 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd36404d500 msgr2=0x7fd36404fa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:45.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:45.797 7fd38019a700 1 --2- 172.21.2.112:0/796702164 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd36404d500 0x7fd36404fa20 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7fd370006a40 tx=0x7fd37000c0c0).stop 2021-02-10T11:19:45.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:45.798 7fd37e7b7700 1 -- 172.21.2.112:0/796702164 reap_dead start 2021-02-10T11:19:45.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:45.798 7fd38019a700 1 -- 172.21.2.112:0/796702164 shutdown_connections 2021-02-10T11:19:45.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:45.798 7fd38019a700 1 -- 172.21.2.112:0/796702164 >> 172.21.2.112:0/796702164 conn(0x7fd3780ce3e0 msgr2=0x7fd3780e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:19:45.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:45.798 7fd38019a700 1 -- 172.21.2.112:0/796702164 shutdown_connections 2021-02-10T11:19:45.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:45.798 7fd38019a700 1 -- 172.21.2.112:0/796702164 wait complete. 2021-02-10T11:19:45.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:45.813 7fd38019a700 1 Processor -- start 2021-02-10T11:19:45.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:45.813 7fd38019a700 1 -- start start 2021-02-10T11:19:45.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:45.813 7fd38019a700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd3780a6890 0x7fd3780a9900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:45.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:45.813 7fd38019a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd3780a6c90 0x7fd3780a5d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:45.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:45.813 7fd38019a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd3780a6380 0x7fd3780aa6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:45.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:45.813 7fd38019a700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3780aa120 con 0x7fd3780a6c90 2021-02-10T11:19:45.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:45.813 7fd38019a700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3780aacc0 con 0x7fd3780a6890 2021-02-10T11:19:45.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:45.813 7fd38019a700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3780aaee0 con 0x7fd3780a6380 2021-02-10T11:19:45.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:45.813 7fd37d7b5700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd3780a6c90 0x7fd3780a5d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:45.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:45.813 7fd37d7b5700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd3780a6c90 0x7fd3780a5d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:44560/0 (socket says 172.21.2.112:44560) 2021-02-10T11:19:45.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:45.813 7fd37d7b5700 1 -- 172.21.2.112:0/3890144217 learned_addr learned my addr 172.21.2.112:0/3890144217 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:19:45.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:45.814 7fd37d7b5700 1 -- 172.21.2.112:0/3890144217 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd3780a6380 msgr2=0x7fd3780aa6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:45.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:45.814 7fd37d7b5700 1 --2- 172.21.2.112:0/3890144217 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd3780a6380 0x7fd3780aa6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:19:45.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:45.814 7fd37d7b5700 1 -- 172.21.2.112:0/3890144217 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd3780a6890 msgr2=0x7fd3780a9900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:45.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:45.814 7fd37d7b5700 1 --2- 172.21.2.112:0/3890144217 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd3780a6890 0x7fd3780a9900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:19:45.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:45.814 7fd37d7b5700 1 -- 172.21.2.112:0/3890144217 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3780ab100 con 0x7fd3780a6c90 2021-02-10T11:19:45.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:45.814 7fd37d7b5700 1 --2- 172.21.2.112:0/3890144217 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd3780a6c90 0x7fd3780a5d60 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 rx=0x7fd36800b5e0 tx=0x7fd36800b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:19:45.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:45.814 7fd36e7fc700 1 -- 172.21.2.112:0/3890144217 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd3680067f0 con 0x7fd3780a6c90 2021-02-10T11:19:45.815 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:45.814 7fd36e7fc700 1 -- 172.21.2.112:0/3890144217 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd36800bd70 con 0x7fd3780a6c90 2021-02-10T11:19:45.815 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:45.814 7fd36e7fc700 1 -- 172.21.2.112:0/3890144217 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd3680022b0 con 0x7fd3780a6c90 2021-02-10T11:19:45.815 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:45.814 7fd38019a700 1 -- 172.21.2.112:0/3890144217 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3780ab100 con 0x7fd3780a6c90 2021-02-10T11:19:45.815 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:45.814 7fd38019a700 1 -- 172.21.2.112:0/3890144217 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3780ab100 con 0x7fd3780a6c90 2021-02-10T11:19:45.815 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:45.815 7fd36e7fc700 1 -- 172.21.2.112:0/3890144217 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd368002020 con 0x7fd3780a6c90 2021-02-10T11:19:45.815 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:45.815 7fd36e7fc700 1 --2- 172.21.2.112:0/3890144217 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd3600186d0 0x7fd36001abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:45.815 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:45.815 7fd36e7fc700 1 -- 172.21.2.112:0/3890144217 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fd3680327d0 con 0x7fd3780a6c90 2021-02-10T11:19:45.815 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:45.815 7fd37dfb6700 1 --2- 172.21.2.112:0/3890144217 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd3600186d0 0x7fd36001abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:45.815 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:45.816 7fd37dfb6700 1 --2- 172.21.2.112:0/3890144217 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd3600186d0 0x7fd36001abd0 crc :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:19:45.840 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:45.845 7fd38019a700 1 -- 172.21.2.112:0/3890144217 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd378105840 con 0x7fd3780a6c90 2021-02-10T11:19:45.843 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:45.848 7fd36e7fc700 1 -- 172.21.2.112:0/3890144217 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd368000c90 con 0x7fd3780a6c90 2021-02-10T11:19:45.971 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:45.976 7fd38019a700 1 -- 172.21.2.112:0/3890144217 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348595", "target": ["mgr", ""]}) v1 -- 0x7fd378105840 con 0x7fd3600186d0 2021-02-10T11:19:45.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:46.003 7fd36e7fc700 1 -- 172.21.2.112:0/3890144217 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fd374001cd0 con 0x7fd3600186d0 2021-02-10T11:19:46.005 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:46.010 7fd38019a700 1 -- 172.21.2.112:0/3890144217 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd3600186d0 msgr2=0x7fd36001abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:46.005 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:46.010 7fd38019a700 1 --2- 172.21.2.112:0/3890144217 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd3600186d0 0x7fd36001abd0 crc :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:46.005 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:46.010 7fd38019a700 1 -- 172.21.2.112:0/3890144217 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd3780a6c90 msgr2=0x7fd3780a5d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:46.005 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:46.010 7fd38019a700 1 --2- 172.21.2.112:0/3890144217 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd3780a6c90 0x7fd3780a5d60 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 rx=0x7fd36800b5e0 tx=0x7fd36800b980).stop 2021-02-10T11:19:46.005 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:46.010 7fd38019a700 1 -- 172.21.2.112:0/3890144217 shutdown_connections 2021-02-10T11:19:46.005 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:46.010 7fd38019a700 1 --2- 172.21.2.112:0/3890144217 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd3600186d0 0x7fd36001abd0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:46.006 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:46.010 7fd38019a700 1 --2- 172.21.2.112:0/3890144217 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd3780a6380 0x7fd3780aa6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:19:46.006 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:46.010 7fd38019a700 1 --2- 172.21.2.112:0/3890144217 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd3780a6c90 0x7fd3780a5d60 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:46.006 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:46.010 7fd38019a700 1 --2- 172.21.2.112:0/3890144217 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd3780a6890 0x7fd3780a9900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:19:46.006 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:46.010 7fd38019a700 1 -- 172.21.2.112:0/3890144217 >> 172.21.2.112:0/3890144217 conn(0x7fd3780a2800 msgr2=0x7fd378105db0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:19:46.006 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:46.010 7fd38019a700 1 -- 172.21.2.112:0/3890144217 shutdown_connections 2021-02-10T11:19:46.006 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:46.010 7fd38019a700 1 -- 172.21.2.112:0/3890144217 wait complete. 2021-02-10T11:19:46.013 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000348595 2021-02-10T11:19:46.442 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:46.431 7efbe1392700 1 Processor -- start 2021-02-10T11:19:46.443 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:46.431 7efbe1392700 1 -- start start 2021-02-10T11:19:46.443 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:46.431 7efbe1392700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7efbdc0f05f0 0x7efbdc0f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:46.443 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:46.431 7efbe1392700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efbdc1b0c70 con 0x7efbdc1b0870 2021-02-10T11:19:46.443 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:46.431 7efbe1392700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efbdc0f0b50 con 0x7efbdc1b0470 2021-02-10T11:19:46.443 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:46.431 7efbe1392700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efbdc0e5ea0 con 0x7efbdc0f05f0 2021-02-10T11:19:46.443 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:46.431 7efbdb87e700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7efbdc0f05f0 0x7efbdc0f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:46.443 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:46.431 7efbdb87e700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7efbdc0f05f0 0x7efbdc0f20e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:44568/0 (socket says 172.21.2.112:44568) 2021-02-10T11:19:46.443 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:46.431 7efbda87c700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7efbdc1b0870 0x7efbdc0efe00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:44026/0 (socket says 172.21.2.112:44026) 2021-02-10T11:19:46.443 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:46.431 7efbdb87e700 1 -- 172.21.2.112:0/2314164422 learned_addr learned my addr 172.21.2.112:0/2314164422 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:19:46.443 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:46.432 7efbdb87e700 1 -- 172.21.2.112:0/2314164422 >> v1:172.21.2.120:6789/0 conn(0x7efbdc1b0470 legacy=0x7efbdc0ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:46.443 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:46.432 7efbdb87e700 1 -- 172.21.2.112:0/2314164422 >> v1:172.21.2.112:6789/0 conn(0x7efbdc1b0870 legacy=0x7efbdc0efe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:46.443 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:46.432 7efbdb87e700 1 -- 172.21.2.112:0/2314164422 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efbdc0e52e0 con 0x7efbdc0f05f0 2021-02-10T11:19:46.443 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:46.432 7efbdb87e700 1 --2- 172.21.2.112:0/2314164422 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efbdc0f05f0 0x7efbdc0f20e0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 rx=0x7efbd4006f00 tx=0x7efbd4009f10).ready entity=mon.0 client_cookie=d96cfeda1feeea2d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:19:46.444 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:46.432 7efbda07b700 1 -- 172.21.2.112:0/2314164422 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efbd400b3b0 con 0x7efbdc0f05f0 2021-02-10T11:19:46.444 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:46.432 7efbda07b700 1 -- 172.21.2.112:0/2314164422 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efbd400b630 con 0x7efbdc0f05f0 2021-02-10T11:19:46.444 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:46.432 7efbda07b700 1 -- 172.21.2.112:0/2314164422 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efbd400bb10 con 0x7efbdc0f05f0 2021-02-10T11:19:46.444 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:46.432 7efbe1392700 1 -- 172.21.2.112:0/2314164422 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efbdc0f05f0 msgr2=0x7efbdc0f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:46.444 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:46.432 7efbe1392700 1 --2- 172.21.2.112:0/2314164422 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efbdc0f05f0 0x7efbdc0f20e0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 rx=0x7efbd4006f00 tx=0x7efbd4009f10).stop 2021-02-10T11:19:46.444 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:46.433 7efbe1392700 1 -- 172.21.2.112:0/2314164422 shutdown_connections 2021-02-10T11:19:46.444 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:46.433 7efbe1392700 1 --2- 172.21.2.112:0/2314164422 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efbdc0f05f0 0x7efbdc0f20e0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:46.444 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:46.433 7efbe1392700 1 -- 172.21.2.112:0/2314164422 >> 172.21.2.112:0/2314164422 conn(0x7efbdc0ce3e0 msgr2=0x7efbdc0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:19:46.444 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:46.433 7efbe1392700 1 -- 172.21.2.112:0/2314164422 shutdown_connections 2021-02-10T11:19:46.444 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:46.433 7efbe1392700 1 -- 172.21.2.112:0/2314164422 wait complete. 2021-02-10T11:19:46.446 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:46.448 7efbe1392700 1 Processor -- start 2021-02-10T11:19:46.446 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:46.448 7efbe1392700 1 -- start start 2021-02-10T11:19:46.446 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:46.448 7efbe1392700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7efbdc082170 0x7efbdc0894a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:46.446 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:46.448 7efbe1392700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efbdc1b01a0 0x7efbdc1b05a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:46.446 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:46.448 7efbe1392700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efbdc1b1460 0x7efbdc1b1860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:46.446 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:46.448 7efbe1392700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efbdc1bf2a0 con 0x7efbdc1b01a0 2021-02-10T11:19:46.446 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:46.448 7efbe1392700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efbdc1b0ee0 con 0x7efbdc082170 2021-02-10T11:19:46.446 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:46.448 7efbe1392700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efbdc1b60c0 con 0x7efbdc1b1460 2021-02-10T11:19:46.446 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:46.448 7efbda87c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efbdc1b01a0 0x7efbdc1b05a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:46.446 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:46.448 7efbda87c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efbdc1b01a0 0x7efbdc1b05a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:44574/0 (socket says 172.21.2.112:44574) 2021-02-10T11:19:46.447 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:46.448 7efbda87c700 1 -- 172.21.2.112:0/303954023 learned_addr learned my addr 172.21.2.112:0/303954023 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:19:46.447 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:46.449 7efbda87c700 1 -- 172.21.2.112:0/303954023 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efbdc1b1460 msgr2=0x7efbdc1b1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:46.447 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:46.449 7efbda87c700 1 --2- 172.21.2.112:0/303954023 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efbdc1b1460 0x7efbdc1b1860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:19:46.447 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:46.449 7efbda87c700 1 -- 172.21.2.112:0/303954023 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7efbdc082170 msgr2=0x7efbdc0894a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:46.447 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:46.449 7efbda87c700 1 --2- 172.21.2.112:0/303954023 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7efbdc082170 0x7efbdc0894a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:19:46.447 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:46.449 7efbda87c700 1 -- 172.21.2.112:0/303954023 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efbdc1b0bc0 con 0x7efbdc1b01a0 2021-02-10T11:19:46.447 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:46.449 7efbda87c700 1 --2- 172.21.2.112:0/303954023 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efbdc1b01a0 0x7efbdc1b05a0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7efbcc006f00 tx=0x7efbcc00b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:19:46.447 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:46.449 7efbd3fff700 1 -- 172.21.2.112:0/303954023 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efbcc006020 con 0x7efbdc1b01a0 2021-02-10T11:19:46.447 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:46.449 7efbd3fff700 1 -- 172.21.2.112:0/303954023 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efbcc0063a0 con 0x7efbdc1b01a0 2021-02-10T11:19:46.447 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:46.449 7efbd3fff700 1 -- 172.21.2.112:0/303954023 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efbcc004620 con 0x7efbdc1b01a0 2021-02-10T11:19:46.447 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:46.449 7efbe1392700 1 -- 172.21.2.112:0/303954023 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efbdc1b0bc0 con 0x7efbdc1b01a0 2021-02-10T11:19:46.447 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:46.449 7efbe1392700 1 -- 172.21.2.112:0/303954023 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efbdc1b0bc0 con 0x7efbdc1b01a0 2021-02-10T11:19:46.447 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:46.450 7efbd3fff700 1 -- 172.21.2.112:0/303954023 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7efbcc00a000 con 0x7efbdc1b01a0 2021-02-10T11:19:46.447 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:46.450 7efbd3fff700 1 --2- 172.21.2.112:0/303954023 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7efbc00186d0 0x7efbc001abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:46.448 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:46.450 7efbd3fff700 1 -- 172.21.2.112:0/303954023 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7efbcc032780 con 0x7efbdc1b01a0 2021-02-10T11:19:46.448 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:46.450 7efbdb07d700 1 --2- 172.21.2.112:0/303954023 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7efbc00186d0 0x7efbc001abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:46.448 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:46.451 7efbdb07d700 1 --2- 172.21.2.112:0/303954023 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7efbc00186d0 0x7efbc001abd0 crc :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:19:46.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:46.481 7efbe1392700 1 -- 172.21.2.112:0/303954023 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efbdc1bf2a0 con 0x7efbdc1b01a0 2021-02-10T11:19:46.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:46.483 7efbd3fff700 1 -- 172.21.2.112:0/303954023 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7efbcc006b90 con 0x7efbdc1b01a0 2021-02-10T11:19:46.607 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:46.612 7efbe1392700 1 -- 172.21.2.112:0/303954023 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348595", "target": ["mgr", ""]}) v1 -- 0x7efbdc1bf2a0 con 0x7efbc00186d0 2021-02-10T11:19:46.611 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:46.616 7efbd3fff700 1 -- 172.21.2.112:0/303954023 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7efbc4006a10 con 0x7efbc00186d0 2021-02-10T11:19:46.615 INFO:teuthology.orchestra.run.gibba012.stdout:/volumes/_nogroup/subvolume_0000000000348595/839f16e6-8ee1-4016-9f1e-c4b08635938e 2021-02-10T11:19:46.616 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:46.621 7efbe1392700 1 -- 172.21.2.112:0/303954023 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7efbc00186d0 msgr2=0x7efbc001abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:46.616 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:46.621 7efbe1392700 1 --2- 172.21.2.112:0/303954023 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7efbc00186d0 0x7efbc001abd0 crc :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:46.617 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:46.621 7efbe1392700 1 -- 172.21.2.112:0/303954023 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efbdc1b01a0 msgr2=0x7efbdc1b05a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:46.617 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:46.621 7efbe1392700 1 --2- 172.21.2.112:0/303954023 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efbdc1b01a0 0x7efbdc1b05a0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7efbcc006f00 tx=0x7efbcc00b940).stop 2021-02-10T11:19:46.617 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:46.621 7efbe1392700 1 -- 172.21.2.112:0/303954023 shutdown_connections 2021-02-10T11:19:46.617 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:46.621 7efbe1392700 1 --2- 172.21.2.112:0/303954023 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7efbc00186d0 0x7efbc001abd0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:46.617 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:46.621 7efbe1392700 1 --2- 172.21.2.112:0/303954023 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efbdc1b1460 0x7efbdc1b1860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:19:46.617 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:46.621 7efbe1392700 1 --2- 172.21.2.112:0/303954023 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efbdc1b01a0 0x7efbdc1b05a0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:46.617 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:46.621 7efbe1392700 1 --2- 172.21.2.112:0/303954023 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7efbdc082170 0x7efbdc0894a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:19:46.617 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:46.621 7efbe1392700 1 -- 172.21.2.112:0/303954023 >> 172.21.2.112:0/303954023 conn(0x7efbdc088d10 msgr2=0x7efbdc0afb80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:19:46.617 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:46.621 7efbe1392700 1 -- 172.21.2.112:0/303954023 shutdown_connections 2021-02-10T11:19:46.617 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:46.621 7efbe1392700 1 -- 172.21.2.112:0/303954023 wait complete. 2021-02-10T11:19:46.624 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000348595 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000348595/839f16e6-8ee1-4016-9f1e-c4b08635938e 2021-02-10T11:19:46.624 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:46.625 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:46.625 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:46.959 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:19:46.959 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:46.960 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348595/839f16e6-8ee1-4016-9f1e-c4b08635938e/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:47.339 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:19:47.339 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:19:47.339 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0224891 s, 46.6 MB/s 2021-02-10T11:19:47.340 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:47.340 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:47.340 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:47.675 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:19:47.676 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:47.676 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348595/839f16e6-8ee1-4016-9f1e-c4b08635938e/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:48.049 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:19:48.049 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:19:48.049 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0229468 s, 45.7 MB/s 2021-02-10T11:19:48.050 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:48.050 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:48.050 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:48.386 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:19:48.386 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:48.386 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348595/839f16e6-8ee1-4016-9f1e-c4b08635938e/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:48.761 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:19:48.762 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:19:48.762 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0248263 s, 42.2 MB/s 2021-02-10T11:19:48.762 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:48.763 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:48.763 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:49.099 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:19:49.099 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:49.099 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348595/839f16e6-8ee1-4016-9f1e-c4b08635938e/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:49.470 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:19:49.470 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:19:49.470 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0231589 s, 45.3 MB/s 2021-02-10T11:19:49.471 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:49.471 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:49.471 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:49.805 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:19:49.805 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:49.805 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348595/839f16e6-8ee1-4016-9f1e-c4b08635938e/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:50.181 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:19:50.182 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:19:50.182 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0243194 s, 43.1 MB/s 2021-02-10T11:19:50.183 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:50.183 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:50.183 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:50.517 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:19:50.517 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:50.518 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348595/839f16e6-8ee1-4016-9f1e-c4b08635938e/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:50.890 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:19:50.890 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:19:50.890 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0229207 s, 45.7 MB/s 2021-02-10T11:19:50.891 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:50.891 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:50.891 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:51.227 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:19:51.227 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:51.227 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348595/839f16e6-8ee1-4016-9f1e-c4b08635938e/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:51.603 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:19:51.603 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:19:51.604 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0253978 s, 41.3 MB/s 2021-02-10T11:19:51.604 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:51.605 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:51.605 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:51.939 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:19:51.939 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:51.939 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348595/839f16e6-8ee1-4016-9f1e-c4b08635938e/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:52.312 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:19:52.313 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:19:52.313 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0247325 s, 42.4 MB/s 2021-02-10T11:19:52.314 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:52.314 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:52.314 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:52.649 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:19:52.650 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:52.650 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348595/839f16e6-8ee1-4016-9f1e-c4b08635938e/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:53.026 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:19:53.026 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:19:53.026 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0248196 s, 42.2 MB/s 2021-02-10T11:19:53.027 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:53.027 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:53.027 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:53.362 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:19:53.363 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:53.363 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348595/839f16e6-8ee1-4016-9f1e-c4b08635938e/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:53.738 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:19:53.738 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:19:53.738 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0255559 s, 41.0 MB/s 2021-02-10T11:19:53.739 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000348596 2021-02-10T11:19:54.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.157 7fcc7cae6700 1 Processor -- start 2021-02-10T11:19:54.169 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.157 7fcc7cae6700 1 -- start start 2021-02-10T11:19:54.169 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.157 7fcc7cae6700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcc781b0470 0x7fcc781b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:54.169 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.157 7fcc7cae6700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc780f0b50 con 0x7fcc780ef610 2021-02-10T11:19:54.169 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.157 7fcc7cae6700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc780f24e0 con 0x7fcc780f20e0 2021-02-10T11:19:54.169 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.157 7fcc7cae6700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc780e5ea0 con 0x7fcc781b0470 2021-02-10T11:19:54.169 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.158 7fcc7605d700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fcc780ef610 0x7fcc780efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:44040/0 (socket says 172.21.2.112:44040) 2021-02-10T11:19:54.169 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.158 7fcc7605d700 1 -- 172.21.2.112:0/2418716932 learned_addr learned my addr 172.21.2.112:0/2418716932 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:19:54.169 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.158 7fcc7685e700 1 --2- 172.21.2.112:0/2418716932 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcc781b0470 0x7fcc781b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:54.169 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.158 7fcc7585c700 1 -- 172.21.2.112:0/2418716932 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 506660323 0 0) 0x7fcc6c0010b0 con 0x7fcc780ef610 2021-02-10T11:19:54.169 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.158 7fcc7585c700 1 -- 172.21.2.112:0/2418716932 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc60002fe0 con 0x7fcc780ef610 2021-02-10T11:19:54.170 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.158 7fcc7585c700 1 -- 172.21.2.112:0/2418716932 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1480483133 0 0) 0x7fcc70001400 con 0x7fcc780f20e0 2021-02-10T11:19:54.170 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.158 7fcc7585c700 1 -- 172.21.2.112:0/2418716932 --> v1:172.21.2.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc60004370 con 0x7fcc780f20e0 2021-02-10T11:19:54.170 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.158 7fcc7685e700 1 -- 172.21.2.112:0/2418716932 >> v1:172.21.2.120:6789/0 conn(0x7fcc780f20e0 legacy=0x7fcc780f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:54.170 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.158 7fcc7685e700 1 -- 172.21.2.112:0/2418716932 >> v1:172.21.2.112:6789/0 conn(0x7fcc780ef610 legacy=0x7fcc780efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:54.170 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.158 7fcc7685e700 1 -- 172.21.2.112:0/2418716932 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc780e52e0 con 0x7fcc781b0470 2021-02-10T11:19:54.170 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.158 7fcc7685e700 1 --2- 172.21.2.112:0/2418716932 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcc781b0470 0x7fcc781b0870 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7fcc68006650 tx=0x7fcc68009a80).ready entity=mon.2 client_cookie=9aa62886bac73633 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:19:54.170 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.159 7fcc7585c700 1 -- 172.21.2.112:0/2418716932 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcc680009b0 con 0x7fcc781b0470 2021-02-10T11:19:54.170 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.159 7fcc7585c700 1 -- 172.21.2.112:0/2418716932 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc6800c0c0 con 0x7fcc781b0470 2021-02-10T11:19:54.170 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.159 7fcc7585c700 1 -- 172.21.2.112:0/2418716932 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcc6800c520 con 0x7fcc781b0470 2021-02-10T11:19:54.170 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.159 7fcc7cae6700 1 -- 172.21.2.112:0/2418716932 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcc781b0470 msgr2=0x7fcc781b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:54.170 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.159 7fcc7cae6700 1 --2- 172.21.2.112:0/2418716932 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcc781b0470 0x7fcc781b0870 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7fcc68006650 tx=0x7fcc68009a80).stop 2021-02-10T11:19:54.170 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.159 7fcc7cae6700 1 -- 172.21.2.112:0/2418716932 shutdown_connections 2021-02-10T11:19:54.170 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.159 7fcc7cae6700 1 --2- 172.21.2.112:0/2418716932 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcc781b0470 0x7fcc781b0870 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:54.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.159 7fcc7cae6700 1 -- 172.21.2.112:0/2418716932 >> 172.21.2.112:0/2418716932 conn(0x7fcc780ce3e0 msgr2=0x7fcc780e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:19:54.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.159 7fcc7cae6700 1 -- 172.21.2.112:0/2418716932 shutdown_connections 2021-02-10T11:19:54.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.159 7fcc7cae6700 1 -- 172.21.2.112:0/2418716932 wait complete. 2021-02-10T11:19:54.172 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.174 7fcc7cae6700 1 Processor -- start 2021-02-10T11:19:54.172 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.174 7fcc7cae6700 1 -- start start 2021-02-10T11:19:54.172 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.174 7fcc7cae6700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcc780bb250 0x7fcc7818d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:54.172 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.174 7fcc7cae6700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcc7810a2b0 0x7fcc781a7810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:54.173 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.174 7fcc7cae6700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcc781a7e30 0x7fcc7809d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:54.173 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.174 7fcc7cae6700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc7810a730 con 0x7fcc7810a2b0 2021-02-10T11:19:54.173 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.174 7fcc7cae6700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc780986a0 con 0x7fcc780bb250 2021-02-10T11:19:54.173 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.174 7fcc7cae6700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc7818db60 con 0x7fcc781a7e30 2021-02-10T11:19:54.173 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.175 7fcc7605d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcc7810a2b0 0x7fcc781a7810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:54.173 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.175 7fcc7605d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcc7810a2b0 0x7fcc781a7810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:44582/0 (socket says 172.21.2.112:44582) 2021-02-10T11:19:54.173 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.175 7fcc7605d700 1 -- 172.21.2.112:0/2413702395 learned_addr learned my addr 172.21.2.112:0/2413702395 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:19:54.173 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.175 7fcc7685e700 1 --2- 172.21.2.112:0/2413702395 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcc780bb250 0x7fcc7818d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:54.173 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.175 7fcc7705f700 1 --2- 172.21.2.112:0/2413702395 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcc781a7e30 0x7fcc7809d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:54.173 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.175 7fcc7605d700 1 -- 172.21.2.112:0/2413702395 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcc781a7e30 msgr2=0x7fcc7809d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:54.173 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.175 7fcc7605d700 1 --2- 172.21.2.112:0/2413702395 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcc781a7e30 0x7fcc7809d960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:54.173 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.175 7fcc7605d700 1 -- 172.21.2.112:0/2413702395 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcc780bb250 msgr2=0x7fcc7818d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:54.173 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.175 7fcc7605d700 1 --2- 172.21.2.112:0/2413702395 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcc780bb250 0x7fcc7818d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:54.174 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.175 7fcc7605d700 1 -- 172.21.2.112:0/2413702395 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc7809df80 con 0x7fcc7810a2b0 2021-02-10T11:19:54.174 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.175 7fcc7605d700 1 --2- 172.21.2.112:0/2413702395 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcc7810a2b0 0x7fcc781a7810 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7fcc6c001c60 tx=0x7fcc6c00a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:19:54.174 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.175 7fcc677fe700 1 -- 172.21.2.112:0/2413702395 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcc6c00acc0 con 0x7fcc7810a2b0 2021-02-10T11:19:54.174 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.175 7fcc677fe700 1 -- 172.21.2.112:0/2413702395 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc6c004350 con 0x7fcc7810a2b0 2021-02-10T11:19:54.174 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.175 7fcc7cae6700 1 -- 172.21.2.112:0/2413702395 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc7809df80 con 0x7fcc7810a2b0 2021-02-10T11:19:54.174 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.176 7fcc677fe700 1 -- 172.21.2.112:0/2413702395 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcc6c004830 con 0x7fcc7810a2b0 2021-02-10T11:19:54.174 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.176 7fcc7cae6700 1 -- 172.21.2.112:0/2413702395 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc7809df80 con 0x7fcc7810a2b0 2021-02-10T11:19:54.174 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.176 7fcc677fe700 1 -- 172.21.2.112:0/2413702395 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fcc6c007f30 con 0x7fcc7810a2b0 2021-02-10T11:19:54.174 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.176 7fcc677fe700 1 --2- 172.21.2.112:0/2413702395 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcc5c0186c0 0x7fcc5c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:54.174 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.176 7fcc677fe700 1 -- 172.21.2.112:0/2413702395 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fcc6c032530 con 0x7fcc7810a2b0 2021-02-10T11:19:54.174 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.177 7fcc7685e700 1 --2- 172.21.2.112:0/2413702395 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcc5c0186c0 0x7fcc5c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:54.174 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.177 7fcc7685e700 1 --2- 172.21.2.112:0/2413702395 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcc5c0186c0 0x7fcc5c01abc0 crc :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:19:54.202 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.207 7fcc7cae6700 1 -- 172.21.2.112:0/2413702395 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc7809e510 con 0x7fcc7810a2b0 2021-02-10T11:19:54.205 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.210 7fcc677fe700 1 -- 172.21.2.112:0/2413702395 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fcc6c025f60 con 0x7fcc7810a2b0 2021-02-10T11:19:54.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.343 7fcc7cae6700 1 -- 172.21.2.112:0/2413702395 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348596", "target": ["mgr", ""]}) v1 -- 0x7fcc7809e510 con 0x7fcc5c0186c0 2021-02-10T11:19:54.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.366 7fcc677fe700 1 -- 172.21.2.112:0/2413702395 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fcc68006420 con 0x7fcc5c0186c0 2021-02-10T11:19:54.372 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.376 7fcc7cae6700 1 -- 172.21.2.112:0/2413702395 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcc5c0186c0 msgr2=0x7fcc5c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:54.372 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.376 7fcc7cae6700 1 --2- 172.21.2.112:0/2413702395 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcc5c0186c0 0x7fcc5c01abc0 crc :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:54.372 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.376 7fcc7cae6700 1 -- 172.21.2.112:0/2413702395 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcc7810a2b0 msgr2=0x7fcc781a7810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:54.372 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.376 7fcc7cae6700 1 --2- 172.21.2.112:0/2413702395 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcc7810a2b0 0x7fcc781a7810 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7fcc6c001c60 tx=0x7fcc6c00a8d0).stop 2021-02-10T11:19:54.372 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.376 7fcc7cae6700 1 -- 172.21.2.112:0/2413702395 shutdown_connections 2021-02-10T11:19:54.372 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.376 7fcc7cae6700 1 --2- 172.21.2.112:0/2413702395 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcc5c0186c0 0x7fcc5c01abc0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:54.372 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.376 7fcc7cae6700 1 --2- 172.21.2.112:0/2413702395 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcc781a7e30 0x7fcc7809d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:54.372 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.376 7fcc7cae6700 1 --2- 172.21.2.112:0/2413702395 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcc7810a2b0 0x7fcc781a7810 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:54.372 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.376 7fcc7cae6700 1 --2- 172.21.2.112:0/2413702395 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcc780bb250 0x7fcc7818d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:54.372 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.376 7fcc7cae6700 1 -- 172.21.2.112:0/2413702395 >> 172.21.2.112:0/2413702395 conn(0x7fcc780bbf60 msgr2=0x7fcc7809cf60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:19:54.372 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.376 7fcc7cae6700 1 -- 172.21.2.112:0/2413702395 shutdown_connections 2021-02-10T11:19:54.372 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.376 7fcc7cae6700 1 -- 172.21.2.112:0/2413702395 wait complete. 2021-02-10T11:19:54.379 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000348596 2021-02-10T11:19:54.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.799 7efcbe356700 1 Processor -- start 2021-02-10T11:19:54.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.799 7efcbe356700 1 -- start start 2021-02-10T11:19:54.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.799 7efcbe356700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efcb81b0470 0x7efcb81b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:54.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.799 7efcbe356700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7efcb80ef610 0x7efcb80efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:54.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.799 7efcbe356700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7efcb80f0030 0x7efcb80f0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:54.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.799 7efcbe356700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efcb80e5ea0 con 0x7efcb80f0030 2021-02-10T11:19:54.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.799 7efcbe356700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efcb80f0b50 con 0x7efcb80ef610 2021-02-10T11:19:54.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.799 7efcbe356700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcb80e52e0 con 0x7efcb81b0470 2021-02-10T11:19:54.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.800 7efcbc973700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7efcb80f0030 0x7efcb80f0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:54.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.800 7efcbc973700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7efcb80f0030 0x7efcb80f0430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:44590/0 (socket says 172.21.2.112:44590) 2021-02-10T11:19:54.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.800 7efcbc973700 1 -- 172.21.2.112:0/2218760631 learned_addr learned my addr 172.21.2.112:0/2218760631 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:19:54.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.800 7efcb7fff700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efcb81b0470 0x7efcb81b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:54.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.800 7efcb77fe700 1 --2- 172.21.2.112:0/2218760631 >> v2:172.21.2.120:3300/0 conn(0x7efcb80ef610 0x7efcb80efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:54.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.800 7efcbc973700 1 -- 172.21.2.112:0/2218760631 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efcb81b0470 msgr2=0x7efcb81b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:19:54.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.800 7efcbc973700 1 --2- 172.21.2.112:0/2218760631 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efcb81b0470 0x7efcb81b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:54.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.800 7efcbc973700 1 -- 172.21.2.112:0/2218760631 >> v2:172.21.2.120:3300/0 conn(0x7efcb80ef610 msgr2=0x7efcb80efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:19:54.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.800 7efcbc973700 1 --2- 172.21.2.112:0/2218760631 >> v2:172.21.2.120:3300/0 conn(0x7efcb80ef610 0x7efcb80efa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:54.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.800 7efcbc973700 1 -- 172.21.2.112:0/2218760631 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcb80f10d0 con 0x7efcb80f0030 2021-02-10T11:19:54.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.800 7efcbc973700 1 --2- 172.21.2.112:0/2218760631 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efcb80f0030 0x7efcb80f0430 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 rx=0x7efcb0006f00 tx=0x7efcb0009f10).ready entity=mon.0 client_cookie=e73934c91c8fdc6a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:19:54.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.800 7efcb6ffd700 1 -- 172.21.2.112:0/2218760631 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efcb000b3b0 con 0x7efcb80f0030 2021-02-10T11:19:54.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.800 7efcb6ffd700 1 -- 172.21.2.112:0/2218760631 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcb000b630 con 0x7efcb80f0030 2021-02-10T11:19:54.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.800 7efcb6ffd700 1 -- 172.21.2.112:0/2218760631 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efcb000bb10 con 0x7efcb80f0030 2021-02-10T11:19:54.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.800 7efcbe356700 1 -- 172.21.2.112:0/2218760631 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efcb80f0030 msgr2=0x7efcb80f0430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:54.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.800 7efcbe356700 1 --2- 172.21.2.112:0/2218760631 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efcb80f0030 0x7efcb80f0430 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 rx=0x7efcb0006f00 tx=0x7efcb0009f10).stop 2021-02-10T11:19:54.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.801 7efcbe356700 1 -- 172.21.2.112:0/2218760631 shutdown_connections 2021-02-10T11:19:54.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.801 7efcbe356700 1 --2- 172.21.2.112:0/2218760631 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efcb80f0030 0x7efcb80f0430 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:54.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.801 7efcbe356700 1 --2- 172.21.2.112:0/2218760631 >> v2:172.21.2.120:3300/0 conn(0x7efcb80ef610 0x7efcb80efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:54.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.801 7efcbe356700 1 --2- 172.21.2.112:0/2218760631 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efcb81b0470 0x7efcb81b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:54.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.801 7efcbe356700 1 -- 172.21.2.112:0/2218760631 >> 172.21.2.112:0/2218760631 conn(0x7efcb80ce3e0 msgr2=0x7efcb80e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:19:54.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.801 7efcbe356700 1 -- 172.21.2.112:0/2218760631 shutdown_connections 2021-02-10T11:19:54.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.801 7efcbe356700 1 -- 172.21.2.112:0/2218760631 wait complete. 2021-02-10T11:19:54.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.816 7efcbe356700 1 Processor -- start 2021-02-10T11:19:54.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.816 7efcbe356700 1 -- start start 2021-02-10T11:19:54.815 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.816 7efcbe356700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efcb809b930 0x7efcb80b7c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:54.815 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.816 7efcbe356700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efcb80b8410 0x7efcb80b8810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:54.815 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.816 7efcbe356700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7efcb80c1f80 0x7efcb80c2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:54.815 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.816 7efcbe356700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcb80c1a00 con 0x7efcb809b930 2021-02-10T11:19:54.815 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.816 7efcbe356700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcb80b8e30 con 0x7efcb80c1f80 2021-02-10T11:19:54.815 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.816 7efcbe356700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcb80b9050 con 0x7efcb80b8410 2021-02-10T11:19:54.815 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.816 7efcb7fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efcb809b930 0x7efcb80b7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:54.815 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.816 7efcb7fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efcb809b930 0x7efcb80b7c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:44596/0 (socket says 172.21.2.112:44596) 2021-02-10T11:19:54.815 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.816 7efcb7fff700 1 -- 172.21.2.112:0/1942091888 learned_addr learned my addr 172.21.2.112:0/1942091888 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:19:54.815 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.816 7efcb77fe700 1 --2- 172.21.2.112:0/1942091888 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efcb80b8410 0x7efcb80b8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:54.816 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.816 7efcbc973700 1 --2- 172.21.2.112:0/1942091888 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7efcb80c1f80 0x7efcb80c2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:54.816 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.816 7efcb7fff700 1 -- 172.21.2.112:0/1942091888 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efcb80b8410 msgr2=0x7efcb80b8810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:54.816 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.816 7efcb7fff700 1 --2- 172.21.2.112:0/1942091888 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efcb80b8410 0x7efcb80b8810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:54.816 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.816 7efcb7fff700 1 -- 172.21.2.112:0/1942091888 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7efcb80c1f80 msgr2=0x7efcb80c2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:54.816 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.816 7efcb7fff700 1 --2- 172.21.2.112:0/1942091888 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7efcb80c1f80 0x7efcb80c2380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:54.816 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.817 7efcb7fff700 1 -- 172.21.2.112:0/1942091888 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcb80c29a0 con 0x7efcb809b930 2021-02-10T11:19:54.816 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.817 7efcb7fff700 1 --2- 172.21.2.112:0/1942091888 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efcb809b930 0x7efcb80b7c20 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7efca800b560 tx=0x7efca800b900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:19:54.816 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.817 7efcb4ff9700 1 -- 172.21.2.112:0/1942091888 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efca800bcf0 con 0x7efcb809b930 2021-02-10T11:19:54.816 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.817 7efcb4ff9700 1 -- 172.21.2.112:0/1942091888 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efca8000c90 con 0x7efcb809b930 2021-02-10T11:19:54.816 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.817 7efcbe356700 1 -- 172.21.2.112:0/1942091888 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcb80c29a0 con 0x7efcb809b930 2021-02-10T11:19:54.816 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.817 7efcbe356700 1 -- 172.21.2.112:0/1942091888 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcb80c29a0 con 0x7efcb809b930 2021-02-10T11:19:54.816 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.817 7efcb4ff9700 1 -- 172.21.2.112:0/1942091888 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efca8002220 con 0x7efcb809b930 2021-02-10T11:19:54.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.817 7efcb4ff9700 1 -- 172.21.2.112:0/1942091888 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7efca80052e0 con 0x7efcb809b930 2021-02-10T11:19:54.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.817 7efcb4ff9700 1 --2- 172.21.2.112:0/1942091888 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7efc98018730 0x7efc9801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:54.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.817 7efcb4ff9700 1 -- 172.21.2.112:0/1942091888 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7efca80327f0 con 0x7efcb809b930 2021-02-10T11:19:54.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.818 7efcb77fe700 1 --2- 172.21.2.112:0/1942091888 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7efc98018730 0x7efc9801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:54.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.818 7efcb77fe700 1 --2- 172.21.2.112:0/1942091888 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7efc98018730 0x7efc9801abc0 crc :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:19:54.844 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.849 7efcbe356700 1 -- 172.21.2.112:0/1942091888 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcb8105b30 con 0x7efcb809b930 2021-02-10T11:19:54.846 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.851 7efcb4ff9700 1 -- 172.21.2.112:0/1942091888 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7efca8000c90 con 0x7efcb809b930 2021-02-10T11:19:54.975 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.980 7efcbe356700 1 -- 172.21.2.112:0/1942091888 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348596", "target": ["mgr", ""]}) v1 -- 0x7efcb80b8e30 con 0x7efc98018730 2021-02-10T11:19:54.977 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.982 7efcb4ff9700 1 -- 172.21.2.112:0/1942091888 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7efcac006e70 con 0x7efc98018730 2021-02-10T11:19:54.978 INFO:teuthology.orchestra.run.gibba012.stdout:/volumes/_nogroup/subvolume_0000000000348596/581e235a-43bc-4b61-b170-75f995ba781e 2021-02-10T11:19:54.979 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.984 7efcbe356700 1 -- 172.21.2.112:0/1942091888 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7efc98018730 msgr2=0x7efc9801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:54.979 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.984 7efcbe356700 1 --2- 172.21.2.112:0/1942091888 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7efc98018730 0x7efc9801abc0 crc :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:54.979 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.984 7efcbe356700 1 -- 172.21.2.112:0/1942091888 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efcb809b930 msgr2=0x7efcb80b7c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:54.979 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.984 7efcbe356700 1 --2- 172.21.2.112:0/1942091888 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efcb809b930 0x7efcb80b7c20 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7efca800b560 tx=0x7efca800b900).stop 2021-02-10T11:19:54.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.984 7efcbe356700 1 -- 172.21.2.112:0/1942091888 shutdown_connections 2021-02-10T11:19:54.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.984 7efcbe356700 1 --2- 172.21.2.112:0/1942091888 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7efc98018730 0x7efc9801abc0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:54.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.984 7efcbe356700 1 --2- 172.21.2.112:0/1942091888 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7efcb80c1f80 0x7efcb80c2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:54.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.984 7efcbe356700 1 --2- 172.21.2.112:0/1942091888 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efcb80b8410 0x7efcb80b8810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:54.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.984 7efcbe356700 1 --2- 172.21.2.112:0/1942091888 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efcb809b930 0x7efcb80b7c20 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:54.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.984 7efcbe356700 1 -- 172.21.2.112:0/1942091888 >> 172.21.2.112:0/1942091888 conn(0x7efcb8094e10 msgr2=0x7efcb80a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:19:54.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.985 7efcbe356700 1 -- 172.21.2.112:0/1942091888 shutdown_connections 2021-02-10T11:19:54.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:19:54.985 7efcbe356700 1 -- 172.21.2.112:0/1942091888 wait complete. 2021-02-10T11:19:54.988 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000348596 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000348596/581e235a-43bc-4b61-b170-75f995ba781e 2021-02-10T11:19:54.988 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:54.988 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:54.988 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:55.323 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:19:55.323 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:55.324 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348596/581e235a-43bc-4b61-b170-75f995ba781e/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:55.702 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:19:55.702 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:19:55.702 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0226138 s, 46.4 MB/s 2021-02-10T11:19:55.703 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:55.703 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:55.703 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:56.037 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:19:56.037 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:56.038 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348596/581e235a-43bc-4b61-b170-75f995ba781e/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:56.413 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:19:56.414 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:19:56.414 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0244433 s, 42.9 MB/s 2021-02-10T11:19:56.414 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:56.415 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:56.415 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:56.750 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:19:56.750 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:56.750 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348596/581e235a-43bc-4b61-b170-75f995ba781e/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:57.124 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:19:57.124 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:19:57.124 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0241575 s, 43.4 MB/s 2021-02-10T11:19:57.125 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:57.126 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:57.126 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:57.461 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:19:57.462 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:57.462 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348596/581e235a-43bc-4b61-b170-75f995ba781e/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:57.839 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:19:57.839 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:19:57.839 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0256681 s, 40.9 MB/s 2021-02-10T11:19:57.840 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:57.840 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:57.840 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:58.174 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:19:58.175 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:58.175 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348596/581e235a-43bc-4b61-b170-75f995ba781e/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:58.551 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:19:58.552 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:19:58.552 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0262677 s, 39.9 MB/s 2021-02-10T11:19:58.553 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:58.553 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:58.553 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:58.886 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:19:58.887 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:58.887 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348596/581e235a-43bc-4b61-b170-75f995ba781e/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:59.261 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:19:59.261 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:19:59.261 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0249795 s, 42.0 MB/s 2021-02-10T11:19:59.262 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:59.262 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:59.262 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:59.597 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:19:59.597 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:59.597 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348596/581e235a-43bc-4b61-b170-75f995ba781e/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:59.972 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:19:59.973 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:19:59.973 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0261132 s, 40.2 MB/s 2021-02-10T11:19:59.974 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:59.974 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:59.974 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:20:00.309 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:20:00.310 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:20:00.310 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348596/581e235a-43bc-4b61-b170-75f995ba781e/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:20:00.684 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:20:00.684 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:20:00.684 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.023621 s, 44.4 MB/s 2021-02-10T11:20:00.686 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:20:00.686 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:20:00.686 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:20:01.024 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:20:01.024 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:20:01.024 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348596/581e235a-43bc-4b61-b170-75f995ba781e/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:20:01.399 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:20:01.400 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:20:01.400 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0244444 s, 42.9 MB/s 2021-02-10T11:20:01.400 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:20:01.401 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:20:01.401 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:20:01.741 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:20:01.741 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:20:01.741 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348596/581e235a-43bc-4b61-b170-75f995ba781e/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:20:02.121 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:20:02.121 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:20:02.121 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0231103 s, 45.4 MB/s 2021-02-10T11:20:02.122 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000348597 2021-02-10T11:20:02.553 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:02.540 7f97c4917700 1 Processor -- start 2021-02-10T11:20:02.553 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:02.540 7f97c4917700 1 -- start start 2021-02-10T11:20:02.553 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:02.541 7f97c4917700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f97c01b0470 0x7f97c01b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:02.553 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:02.541 7f97c4917700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f97c00f20e0 0x7f97c00f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:02.553 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:02.541 7f97c4917700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97c00f0b50 con 0x7f97c00ef610 2021-02-10T11:20:02.553 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:02.541 7f97c4917700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97c00f0820 con 0x7f97c01b0470 2021-02-10T11:20:02.553 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:02.541 7f97c4917700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97c00f24e0 con 0x7f97c00f20e0 2021-02-10T11:20:02.553 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:02.541 7f97bde1b700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f97c00ef610 0x7f97c00efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:44064/0 (socket says 172.21.2.112:44064) 2021-02-10T11:20:02.554 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:02.541 7f97bde1b700 1 -- 172.21.2.112:0/1923528238 learned_addr learned my addr 172.21.2.112:0/1923528238 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:20:02.554 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:02.541 7f97bd61a700 1 -- 172.21.2.112:0/1923528238 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1529110835 0 0) 0x7f97b00010b0 con 0x7f97c00ef610 2021-02-10T11:20:02.554 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:02.541 7f97bd61a700 1 -- 172.21.2.112:0/1923528238 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97ac002fe0 con 0x7f97c00ef610 2021-02-10T11:20:02.554 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:02.542 7f97be61c700 1 --2- 172.21.2.112:0/1923528238 >> v2:172.21.2.120:3300/0 conn(0x7f97c01b0470 0x7f97c01b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:02.554 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:02.542 7f97bee1d700 1 --2- 172.21.2.112:0/1923528238 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f97c00f20e0 0x7f97c00f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:02.554 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:02.542 7f97bd61a700 1 -- 172.21.2.112:0/1923528238 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 868233052 0 0) 0x7f97b00010b0 con 0x7f97c00ef610 2021-02-10T11:20:02.554 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:02.542 7f97bd61a700 1 -- 172.21.2.112:0/1923528238 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f97c00f20e0 msgr2=0x7f97c00f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:20:02.554 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:02.542 7f97bd61a700 1 --2- 172.21.2.112:0/1923528238 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f97c00f20e0 0x7f97c00f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:02.554 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:02.542 7f97bd61a700 1 -- 172.21.2.112:0/1923528238 >> v2:172.21.2.120:3300/0 conn(0x7f97c01b0470 msgr2=0x7f97c01b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:20:02.554 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:02.542 7f97bd61a700 1 --2- 172.21.2.112:0/1923528238 >> v2:172.21.2.120:3300/0 conn(0x7f97c01b0470 0x7f97c01b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:02.554 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:02.542 7f97bd61a700 1 -- 172.21.2.112:0/1923528238 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97c00e52e0 con 0x7f97c00ef610 2021-02-10T11:20:02.554 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:02.542 7f97bd61a700 1 -- 172.21.2.112:0/1923528238 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f97b00015b0 con 0x7f97c00ef610 2021-02-10T11:20:02.554 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:02.542 7f97bd61a700 1 -- 172.21.2.112:0/1923528238 >> v1:172.21.2.112:6789/0 conn(0x7f97c00ef610 legacy=0x7f97c00efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:02.555 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:02.542 7f97bd61a700 1 --2- 172.21.2.112:0/1923528238 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f97ac003a40 0x7f97ac005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:02.555 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:02.542 7f97bd61a700 1 --2- 172.21.2.112:0/1923528238 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f97ac04a7a0 0x7f97ac04cca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:02.555 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:02.542 7f97bd61a700 1 --2- 172.21.2.112:0/1923528238 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f97ac04d5a0 0x7f97ac04fa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:02.555 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:02.542 7f97bd61a700 1 -- 172.21.2.112:0/1923528238 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97ac0502e0 con 0x7f97ac04a7a0 2021-02-10T11:20:02.555 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:02.542 7f97bd61a700 1 -- 172.21.2.112:0/1923528238 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97ac050500 con 0x7f97ac003a40 2021-02-10T11:20:02.555 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:02.542 7f97bd61a700 1 -- 172.21.2.112:0/1923528238 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97ac050720 con 0x7f97ac04d5a0 2021-02-10T11:20:02.555 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:02.542 7f97bde1b700 1 --2- 172.21.2.112:0/1923528238 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f97ac04a7a0 0x7f97ac04cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:02.555 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:02.542 7f97be61c700 1 --2- 172.21.2.112:0/1923528238 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f97ac003a40 0x7f97ac005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:02.555 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:02.542 7f97bee1d700 1 --2- 172.21.2.112:0/1923528238 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f97ac04d5a0 0x7f97ac04fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:02.555 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:02.542 7f97bde1b700 1 -- 172.21.2.112:0/1923528238 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f97ac04d5a0 msgr2=0x7f97ac04fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:20:02.555 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:02.542 7f97bde1b700 1 --2- 172.21.2.112:0/1923528238 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f97ac04d5a0 0x7f97ac04fa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:02.555 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:02.542 7f97bde1b700 1 -- 172.21.2.112:0/1923528238 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f97ac003a40 msgr2=0x7f97ac005ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:20:02.555 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:02.543 7f97bde1b700 1 --2- 172.21.2.112:0/1923528238 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f97ac003a40 0x7f97ac005ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:02.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:02.543 7f97bde1b700 1 -- 172.21.2.112:0/1923528238 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97ac0509c0 con 0x7f97ac04a7a0 2021-02-10T11:20:02.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:02.543 7f97bde1b700 1 --2- 172.21.2.112:0/1923528238 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f97ac04a7a0 0x7f97ac04cca0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7f97b00017d0 tx=0x7f97b0007d30).ready entity=mon.0 client_cookie=fd8d4a9818076fcd server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:02.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:02.543 7f97bd61a700 1 -- 172.21.2.112:0/1923528238 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f97b00013a0 con 0x7f97ac04a7a0 2021-02-10T11:20:02.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:02.543 7f97bd61a700 1 -- 172.21.2.112:0/1923528238 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97b000a310 con 0x7f97ac04a7a0 2021-02-10T11:20:02.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:02.543 7f97bd61a700 1 -- 172.21.2.112:0/1923528238 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f97b000a7a0 con 0x7f97ac04a7a0 2021-02-10T11:20:02.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:02.543 7f97c4917700 1 -- 172.21.2.112:0/1923528238 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f97ac04a7a0 msgr2=0x7f97ac04cca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:02.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:02.543 7f97c4917700 1 --2- 172.21.2.112:0/1923528238 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f97ac04a7a0 0x7f97ac04cca0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7f97b00017d0 tx=0x7f97b0007d30).stop 2021-02-10T11:20:02.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:02.543 7f97bee1d700 1 -- 172.21.2.112:0/1923528238 reap_dead start 2021-02-10T11:20:02.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:02.544 7f97c4917700 1 -- 172.21.2.112:0/1923528238 shutdown_connections 2021-02-10T11:20:02.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:02.544 7f97c4917700 1 -- 172.21.2.112:0/1923528238 >> 172.21.2.112:0/1923528238 conn(0x7f97c00ce3e0 msgr2=0x7f97c00e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:20:02.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:02.544 7f97c4917700 1 -- 172.21.2.112:0/1923528238 shutdown_connections 2021-02-10T11:20:02.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:02.544 7f97c4917700 1 -- 172.21.2.112:0/1923528238 wait complete. 2021-02-10T11:20:02.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:02.558 7f97c4917700 1 Processor -- start 2021-02-10T11:20:02.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:02.558 7f97c4917700 1 -- start start 2021-02-10T11:20:02.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:02.559 7f97c4917700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f97c009b930 0x7f97c00b7c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:02.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:02.559 7f97c4917700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f97c00b8410 0x7f97c00b8810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:02.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:02.559 7f97c4917700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f97c00c1f80 0x7f97c00c2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:02.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:02.559 7f97c4917700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97c00c1a00 con 0x7f97c00b8410 2021-02-10T11:20:02.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:02.559 7f97c4917700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97c00b8e30 con 0x7f97c009b930 2021-02-10T11:20:02.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:02.559 7f97c4917700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97c00b9050 con 0x7f97c00c1f80 2021-02-10T11:20:02.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:02.559 7f97bde1b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f97c00b8410 0x7f97c00b8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:02.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:02.559 7f97bde1b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f97c00b8410 0x7f97c00b8810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:44610/0 (socket says 172.21.2.112:44610) 2021-02-10T11:20:02.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:02.559 7f97bde1b700 1 -- 172.21.2.112:0/3106893550 learned_addr learned my addr 172.21.2.112:0/3106893550 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:20:02.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:02.559 7f97be61c700 1 --2- 172.21.2.112:0/3106893550 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f97c009b930 0x7f97c00b7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:02.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:02.559 7f97bde1b700 1 -- 172.21.2.112:0/3106893550 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f97c00c1f80 msgr2=0x7f97c00c2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:02.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:02.559 7f97bde1b700 1 --2- 172.21.2.112:0/3106893550 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f97c00c1f80 0x7f97c00c2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:20:02.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:02.559 7f97bde1b700 1 -- 172.21.2.112:0/3106893550 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f97c009b930 msgr2=0x7f97c00b7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:02.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:02.559 7f97bde1b700 1 --2- 172.21.2.112:0/3106893550 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f97c009b930 0x7f97c00b7c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:02.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:02.559 7f97bde1b700 1 -- 172.21.2.112:0/3106893550 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97c00c29a0 con 0x7f97c00b8410 2021-02-10T11:20:02.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:02.560 7f97be61c700 1 --2- 172.21.2.112:0/3106893550 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f97c009b930 0x7f97c00b7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:20:02.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:02.560 7f97bde1b700 1 --2- 172.21.2.112:0/3106893550 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f97c00b8410 0x7f97c00b8810 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 rx=0x7f97b0007f80 tx=0x7f97b000a880).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:02.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:02.560 7f97b6ffd700 1 -- 172.21.2.112:0/3106893550 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f97b0007b70 con 0x7f97c00b8410 2021-02-10T11:20:02.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:02.560 7f97b6ffd700 1 -- 172.21.2.112:0/3106893550 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97b000ad00 con 0x7f97c00b8410 2021-02-10T11:20:02.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:02.560 7f97b6ffd700 1 -- 172.21.2.112:0/3106893550 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f97b0004330 con 0x7f97c00b8410 2021-02-10T11:20:02.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:02.560 7f97c4917700 1 -- 172.21.2.112:0/3106893550 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97c00c29a0 con 0x7f97c00b8410 2021-02-10T11:20:02.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:02.560 7f97c4917700 1 -- 172.21.2.112:0/3106893550 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97c00c29a0 con 0x7f97c00b8410 2021-02-10T11:20:02.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:02.561 7f97b6ffd700 1 -- 172.21.2.112:0/3106893550 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f97b0004140 con 0x7f97c00b8410 2021-02-10T11:20:02.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:02.561 7f97b6ffd700 1 --2- 172.21.2.112:0/3106893550 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f97a40186c0 0x7f97a401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:02.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:02.561 7f97b6ffd700 1 -- 172.21.2.112:0/3106893550 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f97b0032520 con 0x7f97c00b8410 2021-02-10T11:20:02.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:02.561 7f97be61c700 1 --2- 172.21.2.112:0/3106893550 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f97a40186c0 0x7f97a401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:02.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:02.562 7f97be61c700 1 --2- 172.21.2.112:0/3106893550 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f97a40186c0 0x7f97a401abc0 crc :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:02.586 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:02.591 7f97c4917700 1 -- 172.21.2.112:0/3106893550 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97c0105b30 con 0x7f97c00b8410 2021-02-10T11:20:02.588 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:02.593 7f97b6ffd700 1 -- 172.21.2.112:0/3106893550 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f97b0007b70 con 0x7f97c00b8410 2021-02-10T11:20:02.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:02.720 7f97c4917700 1 -- 172.21.2.112:0/3106893550 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348597", "target": ["mgr", ""]}) v1 -- 0x7f97c0105b30 con 0x7f97a40186c0 2021-02-10T11:20:02.741 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:02.746 7f97b6ffd700 1 -- 172.21.2.112:0/3106893550 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f97a8006b10 con 0x7f97a40186c0 2021-02-10T11:20:02.748 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:02.753 7f97c4917700 1 -- 172.21.2.112:0/3106893550 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f97a40186c0 msgr2=0x7f97a401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:02.748 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:02.753 7f97c4917700 1 --2- 172.21.2.112:0/3106893550 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f97a40186c0 0x7f97a401abc0 crc :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:02.748 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:02.753 7f97c4917700 1 -- 172.21.2.112:0/3106893550 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f97c00b8410 msgr2=0x7f97c00b8810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:02.749 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:02.753 7f97c4917700 1 --2- 172.21.2.112:0/3106893550 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f97c00b8410 0x7f97c00b8810 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 rx=0x7f97b0007f80 tx=0x7f97b000a880).stop 2021-02-10T11:20:02.749 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:02.753 7f97c4917700 1 -- 172.21.2.112:0/3106893550 shutdown_connections 2021-02-10T11:20:02.749 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:02.753 7f97c4917700 1 --2- 172.21.2.112:0/3106893550 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f97a40186c0 0x7f97a401abc0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:02.749 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:02.753 7f97c4917700 1 --2- 172.21.2.112:0/3106893550 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f97c00c1f80 0x7f97c00c2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:20:02.749 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:02.753 7f97c4917700 1 --2- 172.21.2.112:0/3106893550 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f97c00b8410 0x7f97c00b8810 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:02.749 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:02.753 7f97c4917700 1 --2- 172.21.2.112:0/3106893550 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f97c009b930 0x7f97c00b7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:02.749 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:02.753 7f97c4917700 1 -- 172.21.2.112:0/3106893550 >> 172.21.2.112:0/3106893550 conn(0x7f97c0094e10 msgr2=0x7f97c00a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:20:02.749 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:02.754 7f97c4917700 1 -- 172.21.2.112:0/3106893550 shutdown_connections 2021-02-10T11:20:02.749 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:02.754 7f97c4917700 1 -- 172.21.2.112:0/3106893550 wait complete. 2021-02-10T11:20:02.756 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000348597 2021-02-10T11:20:03.186 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:03.175 7ff94a714700 1 Processor -- start 2021-02-10T11:20:03.186 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:03.175 7ff94a714700 1 -- start start 2021-02-10T11:20:03.186 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:03.176 7ff94a714700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff9441b0470 0x7ff9441b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:03.186 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:03.176 7ff94a714700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7ff9440ef610 0x7ff9440efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:03.186 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:03.176 7ff94a714700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7ff9440f0030 0x7ff9440f0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:03.187 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:03.176 7ff94a714700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9440e5ea0 con 0x7ff9440ef610 2021-02-10T11:20:03.187 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:03.176 7ff94a714700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9440f0b50 con 0x7ff9440f0030 2021-02-10T11:20:03.187 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:03.176 7ff94a714700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9440e52e0 con 0x7ff9441b0470 2021-02-10T11:20:03.187 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:03.176 7ff9437fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7ff9440ef610 0x7ff9440efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:03.187 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:03.176 7ff9437fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7ff9440ef610 0x7ff9440efa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:44622/0 (socket says 172.21.2.112:44622) 2021-02-10T11:20:03.187 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:03.176 7ff9437fe700 1 -- 172.21.2.112:0/4285159419 learned_addr learned my addr 172.21.2.112:0/4285159419 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:20:03.187 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:03.176 7ff9437fe700 1 -- 172.21.2.112:0/4285159419 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff9441b0470 msgr2=0x7ff9441b0870 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:20:03.187 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:03.176 7ff9437fe700 1 --2- 172.21.2.112:0/4285159419 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff9441b0470 0x7ff9441b0870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:20:03.187 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:03.176 7ff9437fe700 1 -- 172.21.2.112:0/4285159419 >> v2:172.21.2.120:3300/0 conn(0x7ff9440f0030 msgr2=0x7ff9440f0430 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:20:03.187 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:03.176 7ff9437fe700 1 --2- 172.21.2.112:0/4285159419 >> v2:172.21.2.120:3300/0 conn(0x7ff9440f0030 0x7ff9440f0430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:20:03.187 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:03.176 7ff9437fe700 1 -- 172.21.2.112:0/4285159419 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9440f10d0 con 0x7ff9440ef610 2021-02-10T11:20:03.187 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:03.177 7ff9437fe700 1 --2- 172.21.2.112:0/4285159419 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff9440ef610 0x7ff9440efa10 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7ff938006700 tx=0x7ff938009b80).ready entity=mon.0 client_cookie=8145fdcdaf1dd60c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:03.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:03.177 7ff942ffd700 1 -- 172.21.2.112:0/4285159419 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff93800c0c0 con 0x7ff9440ef610 2021-02-10T11:20:03.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:03.177 7ff942ffd700 1 -- 172.21.2.112:0/4285159419 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff93800c340 con 0x7ff9440ef610 2021-02-10T11:20:03.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:03.177 7ff942ffd700 1 -- 172.21.2.112:0/4285159419 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff93800c860 con 0x7ff9440ef610 2021-02-10T11:20:03.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:03.177 7ff94a714700 1 -- 172.21.2.112:0/4285159419 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff9440ef610 msgr2=0x7ff9440efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:03.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:03.177 7ff94a714700 1 --2- 172.21.2.112:0/4285159419 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff9440ef610 0x7ff9440efa10 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7ff938006700 tx=0x7ff938009b80).stop 2021-02-10T11:20:03.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:03.177 7ff94a714700 1 -- 172.21.2.112:0/4285159419 shutdown_connections 2021-02-10T11:20:03.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:03.177 7ff94a714700 1 --2- 172.21.2.112:0/4285159419 >> v2:172.21.2.120:3300/0 conn(0x7ff9440f0030 0x7ff9440f0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:20:03.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:03.177 7ff94a714700 1 --2- 172.21.2.112:0/4285159419 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff9440ef610 0x7ff9440efa10 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:03.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:03.177 7ff94a714700 1 --2- 172.21.2.112:0/4285159419 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff9441b0470 0x7ff9441b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:20:03.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:03.177 7ff94a714700 1 -- 172.21.2.112:0/4285159419 >> 172.21.2.112:0/4285159419 conn(0x7ff9440ce3e0 msgr2=0x7ff9440e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:20:03.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:03.177 7ff94a714700 1 -- 172.21.2.112:0/4285159419 shutdown_connections 2021-02-10T11:20:03.189 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:03.177 7ff94a714700 1 -- 172.21.2.112:0/4285159419 wait complete. 2021-02-10T11:20:03.190 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:03.192 7ff94a714700 1 Processor -- start 2021-02-10T11:20:03.190 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:03.192 7ff94a714700 1 -- start start 2021-02-10T11:20:03.190 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:03.192 7ff94a714700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff944190920 0x7ff94408f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:03.190 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:03.192 7ff94a714700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff944190d20 0x7ff94408fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:03.191 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:03.192 7ff94a714700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff944193270 0x7ff944193670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:03.191 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:03.192 7ff94a714700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff944090470 con 0x7ff944190d20 2021-02-10T11:20:03.191 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:03.192 7ff94a714700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff944193c90 con 0x7ff944190920 2021-02-10T11:20:03.191 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:03.192 7ff94a714700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff944193e70 con 0x7ff944193270 2021-02-10T11:20:03.191 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:03.192 7ff9437fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff944190d20 0x7ff94408fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:03.191 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:03.192 7ff9437fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff944190d20 0x7ff94408fc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:44624/0 (socket says 172.21.2.112:44624) 2021-02-10T11:20:03.191 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:03.192 7ff9437fe700 1 -- 172.21.2.112:0/919456077 learned_addr learned my addr 172.21.2.112:0/919456077 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:20:03.191 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:03.193 7ff9437fe700 1 -- 172.21.2.112:0/919456077 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff944193270 msgr2=0x7ff944193670 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:20:03.191 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:03.193 7ff9437fe700 1 --2- 172.21.2.112:0/919456077 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff944193270 0x7ff944193670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:20:03.191 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:03.193 7ff9437fe700 1 -- 172.21.2.112:0/919456077 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff944190920 msgr2=0x7ff94408f600 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:20:03.191 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:03.193 7ff9437fe700 1 --2- 172.21.2.112:0/919456077 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff944190920 0x7ff94408f600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:20:03.191 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:03.193 7ff9437fe700 1 -- 172.21.2.112:0/919456077 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff944194090 con 0x7ff944190d20 2021-02-10T11:20:03.191 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:03.193 7ff9437fe700 1 --2- 172.21.2.112:0/919456077 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff944190d20 0x7ff94408fc20 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 rx=0x7ff93800b5e0 tx=0x7ff93800b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:03.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:03.193 7ff940ff9700 1 -- 172.21.2.112:0/919456077 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff93800bd70 con 0x7ff944190d20 2021-02-10T11:20:03.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:03.193 7ff940ff9700 1 -- 172.21.2.112:0/919456077 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff938002030 con 0x7ff944190d20 2021-02-10T11:20:03.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:03.193 7ff94a714700 1 -- 172.21.2.112:0/919456077 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff944194090 con 0x7ff944190d20 2021-02-10T11:20:03.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:03.193 7ff94a714700 1 -- 172.21.2.112:0/919456077 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff944194090 con 0x7ff944190d20 2021-02-10T11:20:03.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:03.193 7ff940ff9700 1 -- 172.21.2.112:0/919456077 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff938002550 con 0x7ff944190d20 2021-02-10T11:20:03.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:03.194 7ff940ff9700 1 -- 172.21.2.112:0/919456077 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff938000c90 con 0x7ff944190d20 2021-02-10T11:20:03.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:03.194 7ff940ff9700 1 --2- 172.21.2.112:0/919456077 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff924018730 0x7ff92401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:03.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:03.194 7ff940ff9700 1 -- 172.21.2.112:0/919456077 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7ff9380327d0 con 0x7ff944190d20 2021-02-10T11:20:03.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:03.194 7ff943fff700 1 --2- 172.21.2.112:0/919456077 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff924018730 0x7ff92401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:03.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:03.195 7ff943fff700 1 --2- 172.21.2.112:0/919456077 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff924018730 0x7ff92401ac10 crc :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:03.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:03.226 7ff94a714700 1 -- 172.21.2.112:0/919456077 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff944105af0 con 0x7ff944190d20 2021-02-10T11:20:03.223 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:03.228 7ff940ff9700 1 -- 172.21.2.112:0/919456077 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff93801f510 con 0x7ff944190d20 2021-02-10T11:20:03.352 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:03.357 7ff94a714700 1 -- 172.21.2.112:0/919456077 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348597", "target": ["mgr", ""]}) v1 -- 0x7ff944193c90 con 0x7ff924018730 2021-02-10T11:20:03.355 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:03.360 7ff940ff9700 1 -- 172.21.2.112:0/919456077 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7ff934006990 con 0x7ff924018730 2021-02-10T11:20:03.359 INFO:teuthology.orchestra.run.gibba012.stdout:/volumes/_nogroup/subvolume_0000000000348597/513a3ff1-831d-44cb-90a7-41a0f75e4d40 2021-02-10T11:20:03.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:03.366 7ff94a714700 1 -- 172.21.2.112:0/919456077 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff924018730 msgr2=0x7ff92401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:03.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:03.366 7ff94a714700 1 --2- 172.21.2.112:0/919456077 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff924018730 0x7ff92401ac10 crc :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:03.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:03.366 7ff94a714700 1 -- 172.21.2.112:0/919456077 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff944190d20 msgr2=0x7ff94408fc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:03.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:03.366 7ff94a714700 1 --2- 172.21.2.112:0/919456077 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff944190d20 0x7ff94408fc20 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 rx=0x7ff93800b5e0 tx=0x7ff93800b980).stop 2021-02-10T11:20:03.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:03.366 7ff94a714700 1 -- 172.21.2.112:0/919456077 shutdown_connections 2021-02-10T11:20:03.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:03.366 7ff94a714700 1 --2- 172.21.2.112:0/919456077 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff924018730 0x7ff92401ac10 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:03.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:03.366 7ff94a714700 1 --2- 172.21.2.112:0/919456077 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff944193270 0x7ff944193670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:20:03.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:03.366 7ff94a714700 1 --2- 172.21.2.112:0/919456077 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff944190d20 0x7ff94408fc20 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:03.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:03.366 7ff94a714700 1 --2- 172.21.2.112:0/919456077 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff944190920 0x7ff94408f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:20:03.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:03.366 7ff94a714700 1 -- 172.21.2.112:0/919456077 >> 172.21.2.112:0/919456077 conn(0x7ff9440bf4e0 msgr2=0x7ff9440be3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:20:03.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:03.366 7ff94a714700 1 -- 172.21.2.112:0/919456077 shutdown_connections 2021-02-10T11:20:03.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:03.366 7ff94a714700 1 -- 172.21.2.112:0/919456077 wait complete. 2021-02-10T11:20:03.369 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000348597 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000348597/513a3ff1-831d-44cb-90a7-41a0f75e4d40 2021-02-10T11:20:03.369 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:20:03.369 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:20:03.369 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:20:03.705 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:20:03.705 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:20:03.705 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348597/513a3ff1-831d-44cb-90a7-41a0f75e4d40/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:20:04.084 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:20:04.085 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:20:04.085 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0249975 s, 41.9 MB/s 2021-02-10T11:20:04.086 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:20:04.086 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:20:04.086 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:20:04.421 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:20:04.422 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:20:04.422 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348597/513a3ff1-831d-44cb-90a7-41a0f75e4d40/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:20:04.791 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:20:04.792 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:20:04.792 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0228692 s, 45.9 MB/s 2021-02-10T11:20:04.793 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:20:04.793 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:20:04.793 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:20:05.129 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:20:05.129 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:20:05.129 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348597/513a3ff1-831d-44cb-90a7-41a0f75e4d40/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:20:05.502 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:20:05.502 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:20:05.502 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0227048 s, 46.2 MB/s 2021-02-10T11:20:05.503 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:20:05.503 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:20:05.503 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:20:05.591 DEBUG:teuthology.orchestra.run.gibba012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:20:05.686 DEBUG:teuthology.orchestra.run.gibba020:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:20:05.837 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:20:05.838 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:20:05.838 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348597/513a3ff1-831d-44cb-90a7-41a0f75e4d40/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:20:06.214 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:20:06.215 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:20:06.215 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0255481 s, 41.0 MB/s 2021-02-10T11:20:06.216 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:20:06.216 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:20:06.216 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:20:06.550 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:20:06.551 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:20:06.551 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348597/513a3ff1-831d-44cb-90a7-41a0f75e4d40/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:20:06.924 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:20:06.924 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:20:06.924 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0234195 s, 44.8 MB/s 2021-02-10T11:20:06.925 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:20:06.925 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:20:06.925 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:20:07.259 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:20:07.260 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:20:07.260 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348597/513a3ff1-831d-44cb-90a7-41a0f75e4d40/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:20:07.634 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:20:07.634 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:20:07.634 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0251127 s, 41.8 MB/s 2021-02-10T11:20:07.635 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:20:07.635 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:20:07.635 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:20:07.971 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:20:07.971 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:20:07.971 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348597/513a3ff1-831d-44cb-90a7-41a0f75e4d40/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:20:08.346 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:20:08.348 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:20:08.348 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0229969 s, 45.6 MB/s 2021-02-10T11:20:08.349 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:20:08.349 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:20:08.349 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:20:08.683 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:20:08.684 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:20:08.684 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348597/513a3ff1-831d-44cb-90a7-41a0f75e4d40/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:20:09.058 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:20:09.058 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:20:09.058 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0233002 s, 45.0 MB/s 2021-02-10T11:20:09.059 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:20:09.059 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:20:09.059 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:20:09.394 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:20:09.395 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:20:09.395 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348597/513a3ff1-831d-44cb-90a7-41a0f75e4d40/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:20:09.767 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:20:09.767 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:20:09.767 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0243562 s, 43.1 MB/s 2021-02-10T11:20:09.768 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:20:09.768 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:20:09.768 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:20:10.103 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:20:10.104 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:20:10.104 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348597/513a3ff1-831d-44cb-90a7-41a0f75e4d40/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:20:10.479 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:20:10.479 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:20:10.479 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0249493 s, 42.0 MB/s 2021-02-10T11:20:10.480 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000348598 2021-02-10T11:20:10.909 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:10.898 7fe2f4eee700 1 Processor -- start 2021-02-10T11:20:10.910 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:10.898 7fe2f4eee700 1 -- start start 2021-02-10T11:20:10.910 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:10.899 7fe2f4eee700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fe2f01b0470 0x7fe2f01b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:10.910 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:10.899 7fe2f4eee700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe2f00f20e0 0x7fe2f00f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:10.910 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:10.899 7fe2f4eee700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2f00f0b50 con 0x7fe2f00ef610 2021-02-10T11:20:10.910 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:10.899 7fe2f4eee700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2f00f0820 con 0x7fe2f01b0470 2021-02-10T11:20:10.910 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:10.899 7fe2f4eee700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2f00f24e0 con 0x7fe2f00f20e0 2021-02-10T11:20:10.910 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:10.899 7fe2ee312700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fe2f00ef610 0x7fe2f00efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:44094/0 (socket says 172.21.2.112:44094) 2021-02-10T11:20:10.910 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:10.899 7fe2ee312700 1 -- 172.21.2.112:0/2263209133 learned_addr learned my addr 172.21.2.112:0/2263209133 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:20:10.910 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:10.899 7fe2ef314700 1 --2- 172.21.2.112:0/2263209133 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe2f00f20e0 0x7fe2f00f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:10.910 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:10.899 7fe2eeb13700 1 --2- 172.21.2.112:0/2263209133 >> v2:172.21.2.120:3300/0 conn(0x7fe2f01b0470 0x7fe2f01b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:10.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:10.899 7fe2edb11700 1 -- 172.21.2.112:0/2263209133 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1580437788 0 0) 0x7fe2e00010b0 con 0x7fe2f00ef610 2021-02-10T11:20:10.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:10.899 7fe2edb11700 1 -- 172.21.2.112:0/2263209133 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2dc002fe0 con 0x7fe2f00ef610 2021-02-10T11:20:10.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:10.899 7fe2ef314700 1 -- 172.21.2.112:0/2263209133 >> v2:172.21.2.120:3300/0 conn(0x7fe2f01b0470 msgr2=0x7fe2f01b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:20:10.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:10.899 7fe2ef314700 1 --2- 172.21.2.112:0/2263209133 >> v2:172.21.2.120:3300/0 conn(0x7fe2f01b0470 0x7fe2f01b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:10.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:10.899 7fe2ef314700 1 -- 172.21.2.112:0/2263209133 >> v1:172.21.2.112:6789/0 conn(0x7fe2f00ef610 legacy=0x7fe2f00efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:10.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:10.899 7fe2ef314700 1 -- 172.21.2.112:0/2263209133 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2f00e52e0 con 0x7fe2f00f20e0 2021-02-10T11:20:10.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:10.900 7fe2ef314700 1 --2- 172.21.2.112:0/2263209133 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe2f00f20e0 0x7fe2f00f0200 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7fe2e8006e50 tx=0x7fe2e8009e70).ready entity=mon.2 client_cookie=dc335093e9147924 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:10.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:10.900 7fe2edb11700 1 -- 172.21.2.112:0/2263209133 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe2e800b510 con 0x7fe2f00f20e0 2021-02-10T11:20:10.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:10.900 7fe2edb11700 1 -- 172.21.2.112:0/2263209133 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2e8006790 con 0x7fe2f00f20e0 2021-02-10T11:20:10.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:10.900 7fe2edb11700 1 -- 172.21.2.112:0/2263209133 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe2e800b940 con 0x7fe2f00f20e0 2021-02-10T11:20:10.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:10.900 7fe2f4eee700 1 -- 172.21.2.112:0/2263209133 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe2f00f20e0 msgr2=0x7fe2f00f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:10.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:10.900 7fe2f4eee700 1 --2- 172.21.2.112:0/2263209133 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe2f00f20e0 0x7fe2f00f0200 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7fe2e8006e50 tx=0x7fe2e8009e70).stop 2021-02-10T11:20:10.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:10.901 7fe2f4eee700 1 -- 172.21.2.112:0/2263209133 shutdown_connections 2021-02-10T11:20:10.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:10.901 7fe2f4eee700 1 --2- 172.21.2.112:0/2263209133 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe2f00f20e0 0x7fe2f00f0200 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:10.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:10.901 7fe2f4eee700 1 --2- 172.21.2.112:0/2263209133 >> v2:172.21.2.120:3300/0 conn(0x7fe2f01b0470 0x7fe2f01b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:10.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:10.901 7fe2f4eee700 1 -- 172.21.2.112:0/2263209133 >> 172.21.2.112:0/2263209133 conn(0x7fe2f00ce3e0 msgr2=0x7fe2f00e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:20:10.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:10.901 7fe2f4eee700 1 -- 172.21.2.112:0/2263209133 shutdown_connections 2021-02-10T11:20:10.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:10.901 7fe2f4eee700 1 -- 172.21.2.112:0/2263209133 wait complete. 2021-02-10T11:20:10.913 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:10.915 7fe2f4eee700 1 Processor -- start 2021-02-10T11:20:10.913 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:10.915 7fe2f4eee700 1 -- start start 2021-02-10T11:20:10.913 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:10.916 7fe2f4eee700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe2f0190920 0x7fe2f00b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:10.914 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:10.916 7fe2f4eee700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe2f00b8200 0x7fe2f00b8600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:10.914 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:10.916 7fe2f4eee700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe2f00c1d80 0x7fe2f00c2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:10.914 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:10.916 7fe2f4eee700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2f0190da0 con 0x7fe2f00c1d80 2021-02-10T11:20:10.914 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:10.916 7fe2f4eee700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2f00b8c20 con 0x7fe2f00b8200 2021-02-10T11:20:10.914 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:10.916 7fe2f4eee700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2f00b8e10 con 0x7fe2f0190920 2021-02-10T11:20:10.914 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:10.916 7fe2ef314700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe2f00c1d80 0x7fe2f00c2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:10.914 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:10.916 7fe2ef314700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe2f00c1d80 0x7fe2f00c2180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:44640/0 (socket says 172.21.2.112:44640) 2021-02-10T11:20:10.914 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:10.916 7fe2ef314700 1 -- 172.21.2.112:0/3025517116 learned_addr learned my addr 172.21.2.112:0/3025517116 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:20:10.914 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:10.916 7fe2ee312700 1 --2- 172.21.2.112:0/3025517116 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe2f00b8200 0x7fe2f00b8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:10.914 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:10.916 7fe2eeb13700 1 --2- 172.21.2.112:0/3025517116 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe2f0190920 0x7fe2f00b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:10.914 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:10.916 7fe2ef314700 1 -- 172.21.2.112:0/3025517116 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe2f0190920 msgr2=0x7fe2f00b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:10.914 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:10.916 7fe2ef314700 1 --2- 172.21.2.112:0/3025517116 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe2f0190920 0x7fe2f00b7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:10.914 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:10.916 7fe2ef314700 1 -- 172.21.2.112:0/3025517116 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe2f00b8200 msgr2=0x7fe2f00b8600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:10.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:10.916 7fe2ef314700 1 --2- 172.21.2.112:0/3025517116 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe2f00b8200 0x7fe2f00b8600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:10.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:10.916 7fe2ef314700 1 -- 172.21.2.112:0/3025517116 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2f00c27d0 con 0x7fe2f00c1d80 2021-02-10T11:20:10.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:10.916 7fe2ef314700 1 --2- 172.21.2.112:0/3025517116 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe2f00c1d80 0x7fe2f00c2180 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 rx=0x7fe2e8006c50 tx=0x7fe2e800bd20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:10.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:10.917 7fe2e77fe700 1 -- 172.21.2.112:0/3025517116 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe2e80093d0 con 0x7fe2f00c1d80 2021-02-10T11:20:10.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:10.917 7fe2e77fe700 1 -- 172.21.2.112:0/3025517116 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2e800be60 con 0x7fe2f00c1d80 2021-02-10T11:20:10.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:10.917 7fe2e77fe700 1 -- 172.21.2.112:0/3025517116 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe2e800c570 con 0x7fe2f00c1d80 2021-02-10T11:20:10.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:10.917 7fe2f4eee700 1 -- 172.21.2.112:0/3025517116 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2f00c27d0 con 0x7fe2f00c1d80 2021-02-10T11:20:10.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:10.917 7fe2f4eee700 1 -- 172.21.2.112:0/3025517116 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2f00c27d0 con 0x7fe2f00c1d80 2021-02-10T11:20:10.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:10.917 7fe2e77fe700 1 -- 172.21.2.112:0/3025517116 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe2e8002ef0 con 0x7fe2f00c1d80 2021-02-10T11:20:10.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:10.917 7fe2e77fe700 1 --2- 172.21.2.112:0/3025517116 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe2d4018730 0x7fe2d401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:10.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:10.917 7fe2e77fe700 1 -- 172.21.2.112:0/3025517116 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fe2e80329f0 con 0x7fe2f00c1d80 2021-02-10T11:20:10.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:10.917 7fe2eeb13700 1 --2- 172.21.2.112:0/3025517116 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe2d4018730 0x7fe2d401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:10.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:10.918 7fe2eeb13700 1 --2- 172.21.2.112:0/3025517116 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe2d4018730 0x7fe2d401ac10 crc :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:10.943 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:10.949 7fe2f4eee700 1 -- 172.21.2.112:0/3025517116 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2f0105ae0 con 0x7fe2f00c1d80 2021-02-10T11:20:10.946 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:10.951 7fe2e77fe700 1 -- 172.21.2.112:0/3025517116 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe2e8000df0 con 0x7fe2f00c1d80 2021-02-10T11:20:11.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:11.076 7fe2f4eee700 1 -- 172.21.2.112:0/3025517116 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348598", "target": ["mgr", ""]}) v1 -- 0x7fe2f0105ae0 con 0x7fe2d4018730 2021-02-10T11:20:11.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:11.101 7fe2e77fe700 1 -- 172.21.2.112:0/3025517116 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fe2d8006a70 con 0x7fe2d4018730 2021-02-10T11:20:11.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:11.109 7fe2f4eee700 1 -- 172.21.2.112:0/3025517116 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe2d4018730 msgr2=0x7fe2d401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:11.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:11.109 7fe2f4eee700 1 --2- 172.21.2.112:0/3025517116 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe2d4018730 0x7fe2d401ac10 crc :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:11.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:11.109 7fe2f4eee700 1 -- 172.21.2.112:0/3025517116 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe2f00c1d80 msgr2=0x7fe2f00c2180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:11.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:11.110 7fe2f4eee700 1 --2- 172.21.2.112:0/3025517116 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe2f00c1d80 0x7fe2f00c2180 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 rx=0x7fe2e8006c50 tx=0x7fe2e800bd20).stop 2021-02-10T11:20:11.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:11.110 7fe2f4eee700 1 -- 172.21.2.112:0/3025517116 shutdown_connections 2021-02-10T11:20:11.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:11.110 7fe2f4eee700 1 --2- 172.21.2.112:0/3025517116 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe2d4018730 0x7fe2d401ac10 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:11.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:11.110 7fe2f4eee700 1 --2- 172.21.2.112:0/3025517116 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe2f00b8200 0x7fe2f00b8600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:11.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:11.110 7fe2f4eee700 1 --2- 172.21.2.112:0/3025517116 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe2f00c1d80 0x7fe2f00c2180 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:11.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:11.110 7fe2f4eee700 1 --2- 172.21.2.112:0/3025517116 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe2f0190920 0x7fe2f00b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:11.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:11.110 7fe2f4eee700 1 -- 172.21.2.112:0/3025517116 >> 172.21.2.112:0/3025517116 conn(0x7fe2f0094e10 msgr2=0x7fe2f00a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:20:11.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:11.110 7fe2f4eee700 1 -- 172.21.2.112:0/3025517116 shutdown_connections 2021-02-10T11:20:11.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:11.110 7fe2f4eee700 1 -- 172.21.2.112:0/3025517116 wait complete. 2021-02-10T11:20:11.113 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000348598 2021-02-10T11:20:11.544 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:11.531 7f6a00927700 1 Processor -- start 2021-02-10T11:20:11.545 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:11.531 7f6a00927700 1 -- start start 2021-02-10T11:20:11.545 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:11.532 7f6a00927700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f69fc1b0470 0x7f69fc1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:11.545 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:11.532 7f6a00927700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f69fc0ef610 0x7f69fc0efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:11.545 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:11.532 7f6a00927700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69fc0f0b50 con 0x7f69fc0f0030 2021-02-10T11:20:11.545 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:11.532 7f6a00927700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69fc0e52e0 con 0x7f69fc0ef610 2021-02-10T11:20:11.545 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:11.532 7f6a00927700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69fc0e5490 con 0x7f69fc1b0470 2021-02-10T11:20:11.545 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:11.532 7f69fae1d700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f69fc0f0030 0x7f69fc0f10d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:44106/0 (socket says 172.21.2.112:44106) 2021-02-10T11:20:11.545 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:11.532 7f69fae1d700 1 -- 172.21.2.112:0/4261216374 learned_addr learned my addr 172.21.2.112:0/4261216374 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:20:11.545 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:11.532 7f69f961a700 1 -- 172.21.2.112:0/4261216374 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1346235050 0 0) 0x7f69f4001500 con 0x7f69fc0f0030 2021-02-10T11:20:11.545 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:11.532 7f69f9e1b700 1 --2- 172.21.2.112:0/4261216374 >> v2:172.21.2.120:3300/0 conn(0x7f69fc0ef610 0x7f69fc0efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:11.545 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:11.532 7f69fa61c700 1 --2- 172.21.2.112:0/4261216374 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f69fc1b0470 0x7f69fc1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:11.546 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:11.532 7f69f961a700 1 -- 172.21.2.112:0/4261216374 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69e8002fe0 con 0x7f69fc0f0030 2021-02-10T11:20:11.546 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:11.533 7f69f961a700 1 -- 172.21.2.112:0/4261216374 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3157492086 0 0) 0x7f69f4001270 con 0x7f69fc0f0030 2021-02-10T11:20:11.546 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:11.533 7f69f961a700 1 -- 172.21.2.112:0/4261216374 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f69fc1b0470 msgr2=0x7f69fc1b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:20:11.546 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:11.533 7f69f961a700 1 --2- 172.21.2.112:0/4261216374 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f69fc1b0470 0x7f69fc1b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:11.546 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:11.533 7f69f961a700 1 -- 172.21.2.112:0/4261216374 >> v2:172.21.2.120:3300/0 conn(0x7f69fc0ef610 msgr2=0x7f69fc0efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:20:11.546 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:11.533 7f69f961a700 1 --2- 172.21.2.112:0/4261216374 >> v2:172.21.2.120:3300/0 conn(0x7f69fc0ef610 0x7f69fc0efa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:11.546 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:11.533 7f69f961a700 1 -- 172.21.2.112:0/4261216374 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69fc0f0430 con 0x7f69fc0f0030 2021-02-10T11:20:11.546 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:11.533 7f69f961a700 1 -- 172.21.2.112:0/4261216374 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f69f4001960 con 0x7f69fc0f0030 2021-02-10T11:20:11.546 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:11.533 7f69f961a700 1 -- 172.21.2.112:0/4261216374 >> v1:172.21.2.112:6789/0 conn(0x7f69fc0f0030 legacy=0x7f69fc0f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:11.546 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:11.533 7f69f961a700 1 --2- 172.21.2.112:0/4261216374 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f69e8003a60 0x7f69e8005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:11.546 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:11.533 7f69f961a700 1 --2- 172.21.2.112:0/4261216374 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f69e804a7a0 0x7f69e804cc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:11.546 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:11.533 7f69f961a700 1 --2- 172.21.2.112:0/4261216374 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f69e804d510 0x7f69e804fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:11.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:11.533 7f69f961a700 1 -- 172.21.2.112:0/4261216374 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69e80502e0 con 0x7f69e804d510 2021-02-10T11:20:11.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:11.533 7f69f961a700 1 -- 172.21.2.112:0/4261216374 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69e8050500 con 0x7f69e8003a60 2021-02-10T11:20:11.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:11.533 7f69f961a700 1 -- 172.21.2.112:0/4261216374 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69e8050720 con 0x7f69e804a7a0 2021-02-10T11:20:11.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:11.533 7f69fae1d700 1 --2- 172.21.2.112:0/4261216374 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f69e804d510 0x7f69e804fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:11.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:11.533 7f69f9e1b700 1 --2- 172.21.2.112:0/4261216374 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f69e804a7a0 0x7f69e804cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:11.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:11.533 7f69fa61c700 1 --2- 172.21.2.112:0/4261216374 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f69e8003a60 0x7f69e8005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:11.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:11.533 7f69fae1d700 1 -- 172.21.2.112:0/4261216374 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f69e804a7a0 msgr2=0x7f69e804cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:20:11.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:11.533 7f69fae1d700 1 --2- 172.21.2.112:0/4261216374 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f69e804a7a0 0x7f69e804cc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:11.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:11.533 7f69fae1d700 1 -- 172.21.2.112:0/4261216374 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f69e8003a60 msgr2=0x7f69e8005ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:20:11.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:11.533 7f69fae1d700 1 --2- 172.21.2.112:0/4261216374 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f69e8003a60 0x7f69e8005ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:11.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:11.533 7f69fae1d700 1 -- 172.21.2.112:0/4261216374 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69e80509c0 con 0x7f69e804d510 2021-02-10T11:20:11.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:11.533 7f69fae1d700 1 --2- 172.21.2.112:0/4261216374 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f69e804d510 0x7f69e804fa30 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 rx=0x7f69f4007e30 tx=0x7f69f400a310).ready entity=mon.0 client_cookie=6a599c1f18ccb34b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:11.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:11.534 7f69f961a700 1 -- 172.21.2.112:0/4261216374 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f69f4007a00 con 0x7f69e804d510 2021-02-10T11:20:11.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:11.534 7f69f961a700 1 -- 172.21.2.112:0/4261216374 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69f40070c0 con 0x7f69e804d510 2021-02-10T11:20:11.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:11.534 7f69f961a700 1 -- 172.21.2.112:0/4261216374 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f69f4007510 con 0x7f69e804d510 2021-02-10T11:20:11.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:11.534 7f6a00927700 1 -- 172.21.2.112:0/4261216374 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f69e804d510 msgr2=0x7f69e804fa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:11.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:11.534 7f6a00927700 1 --2- 172.21.2.112:0/4261216374 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f69e804d510 0x7f69e804fa30 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 rx=0x7f69f4007e30 tx=0x7f69f400a310).stop 2021-02-10T11:20:11.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:11.534 7f69fae1d700 1 -- 172.21.2.112:0/4261216374 reap_dead start 2021-02-10T11:20:11.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:11.535 7f6a00927700 1 -- 172.21.2.112:0/4261216374 shutdown_connections 2021-02-10T11:20:11.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:11.535 7f6a00927700 1 -- 172.21.2.112:0/4261216374 >> 172.21.2.112:0/4261216374 conn(0x7f69fc0ce3e0 msgr2=0x7f69fc0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:20:11.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:11.535 7f6a00927700 1 -- 172.21.2.112:0/4261216374 shutdown_connections 2021-02-10T11:20:11.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:11.535 7f6a00927700 1 -- 172.21.2.112:0/4261216374 wait complete. 2021-02-10T11:20:11.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:11.549 7f6a00927700 1 Processor -- start 2021-02-10T11:20:11.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:11.550 7f6a00927700 1 -- start start 2021-02-10T11:20:11.550 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:11.550 7f6a00927700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f69fc09b930 0x7f69fc0b7c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:11.550 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:11.550 7f6a00927700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f69fc0b8410 0x7f69fc0b8810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:11.550 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:11.550 7f6a00927700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f69fc0c1f80 0x7f69fc0c2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:11.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:11.550 7f6a00927700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69fc0c1a00 con 0x7f69fc0b8410 2021-02-10T11:20:11.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:11.550 7f6a00927700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69fc0b8e30 con 0x7f69fc0c1f80 2021-02-10T11:20:11.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:11.550 7f6a00927700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69fc0b9050 con 0x7f69fc09b930 2021-02-10T11:20:11.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:11.550 7f69fae1d700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f69fc0c1f80 0x7f69fc0c2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:11.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:11.550 7f69fae1d700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f69fc0c1f80 0x7f69fc0c2380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3300/0 says I am v2:172.21.2.112:46934/0 (socket says 172.21.2.112:46934) 2021-02-10T11:20:11.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:11.550 7f69fae1d700 1 -- 172.21.2.112:0/4280319650 learned_addr learned my addr 172.21.2.112:0/4280319650 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:20:11.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:11.550 7f69f9e1b700 1 --2- 172.21.2.112:0/4280319650 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f69fc0b8410 0x7f69fc0b8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:11.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:11.550 7f69fa61c700 1 --2- 172.21.2.112:0/4280319650 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f69fc09b930 0x7f69fc0b7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:11.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:11.551 7f69f9e1b700 1 -- 172.21.2.112:0/4280319650 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f69fc09b930 msgr2=0x7f69fc0b7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:11.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:11.551 7f69f9e1b700 1 --2- 172.21.2.112:0/4280319650 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f69fc09b930 0x7f69fc0b7c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:11.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:11.551 7f69f9e1b700 1 -- 172.21.2.112:0/4280319650 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f69fc0c1f80 msgr2=0x7f69fc0c2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:11.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:11.551 7f69f9e1b700 1 --2- 172.21.2.112:0/4280319650 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f69fc0c1f80 0x7f69fc0c2380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:11.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:11.551 7f69f9e1b700 1 -- 172.21.2.112:0/4280319650 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69fc0c29a0 con 0x7f69fc0b8410 2021-02-10T11:20:11.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:11.551 7f69fa61c700 1 --2- 172.21.2.112:0/4280319650 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f69fc09b930 0x7f69fc0b7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:20:11.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:11.551 7f69f9e1b700 1 --2- 172.21.2.112:0/4280319650 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f69fc0b8410 0x7f69fc0b8810 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7f69ec009540 tx=0x7f69ec00b8e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:11.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:11.551 7f69f2ffd700 1 -- 172.21.2.112:0/4280319650 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f69ec00bcd0 con 0x7f69fc0b8410 2021-02-10T11:20:11.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:11.551 7f69f2ffd700 1 -- 172.21.2.112:0/4280319650 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69ec006740 con 0x7f69fc0b8410 2021-02-10T11:20:11.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:11.551 7f69f2ffd700 1 -- 172.21.2.112:0/4280319650 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f69ec002030 con 0x7f69fc0b8410 2021-02-10T11:20:11.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:11.551 7f6a00927700 1 -- 172.21.2.112:0/4280319650 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69fc0c29a0 con 0x7f69fc0b8410 2021-02-10T11:20:11.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:11.551 7f6a00927700 1 -- 172.21.2.112:0/4280319650 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69fc0c2f30 con 0x7f69fc0b8410 2021-02-10T11:20:11.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:11.552 7f69f2ffd700 1 -- 172.21.2.112:0/4280319650 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f69ec000c90 con 0x7f69fc0b8410 2021-02-10T11:20:11.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:11.552 7f69f2ffd700 1 --2- 172.21.2.112:0/4280319650 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f69e00186c0 0x7f69e001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:11.553 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:11.552 7f69f2ffd700 1 -- 172.21.2.112:0/4280319650 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f69ec032730 con 0x7f69fc0b8410 2021-02-10T11:20:11.553 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:11.552 7f69fa61c700 1 --2- 172.21.2.112:0/4280319650 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f69e00186c0 0x7f69e001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:11.553 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:11.553 7f69fa61c700 1 --2- 172.21.2.112:0/4280319650 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f69e00186c0 0x7f69e001abc0 crc :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:11.576 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:11.581 7f6a00927700 1 -- 172.21.2.112:0/4280319650 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69fc105b30 con 0x7f69fc0b8410 2021-02-10T11:20:11.579 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:11.584 7f69f2ffd700 1 -- 172.21.2.112:0/4280319650 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f69ec006740 con 0x7f69fc0b8410 2021-02-10T11:20:11.708 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:11.713 7f6a00927700 1 -- 172.21.2.112:0/4280319650 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348598", "target": ["mgr", ""]}) v1 -- 0x7f69fc0c2f30 con 0x7f69e00186c0 2021-02-10T11:20:11.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:11.716 7f69f2ffd700 1 -- 172.21.2.112:0/4280319650 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f69e4006cc0 con 0x7f69e00186c0 2021-02-10T11:20:11.711 INFO:teuthology.orchestra.run.gibba012.stdout:/volumes/_nogroup/subvolume_0000000000348598/aa5dd245-4e43-43d1-ad31-1e3258f01596 2021-02-10T11:20:11.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:11.717 7f6a00927700 1 -- 172.21.2.112:0/4280319650 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f69e00186c0 msgr2=0x7f69e001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:11.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:11.717 7f6a00927700 1 --2- 172.21.2.112:0/4280319650 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f69e00186c0 0x7f69e001abc0 crc :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:11.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:11.717 7f6a00927700 1 -- 172.21.2.112:0/4280319650 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f69fc0b8410 msgr2=0x7f69fc0b8810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:11.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:11.717 7f6a00927700 1 --2- 172.21.2.112:0/4280319650 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f69fc0b8410 0x7f69fc0b8810 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7f69ec009540 tx=0x7f69ec00b8e0).stop 2021-02-10T11:20:11.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:11.717 7f6a00927700 1 -- 172.21.2.112:0/4280319650 shutdown_connections 2021-02-10T11:20:11.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:11.717 7f6a00927700 1 --2- 172.21.2.112:0/4280319650 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f69e00186c0 0x7f69e001abc0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:11.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:11.717 7f6a00927700 1 --2- 172.21.2.112:0/4280319650 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f69fc0c1f80 0x7f69fc0c2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:11.714 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:11.717 7f6a00927700 1 --2- 172.21.2.112:0/4280319650 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f69fc0b8410 0x7f69fc0b8810 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:11.714 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:11.717 7f6a00927700 1 --2- 172.21.2.112:0/4280319650 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f69fc09b930 0x7f69fc0b7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:11.714 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:11.717 7f6a00927700 1 -- 172.21.2.112:0/4280319650 >> 172.21.2.112:0/4280319650 conn(0x7f69fc094e10 msgr2=0x7f69fc0a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:20:11.718 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:11.718 7f6a00927700 1 -- 172.21.2.112:0/4280319650 shutdown_connections 2021-02-10T11:20:11.718 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:11.718 7f6a00927700 1 -- 172.21.2.112:0/4280319650 wait complete. 2021-02-10T11:20:11.721 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000348598 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000348598/aa5dd245-4e43-43d1-ad31-1e3258f01596 2021-02-10T11:20:11.721 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:20:11.721 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:20:11.722 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:20:12.082 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:20:12.083 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:20:12.083 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348598/aa5dd245-4e43-43d1-ad31-1e3258f01596/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:20:12.465 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:20:12.465 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:20:12.465 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0237206 s, 44.2 MB/s 2021-02-10T11:20:12.466 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:20:12.466 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:20:12.466 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:20:12.807 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:20:12.808 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:20:12.808 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348598/aa5dd245-4e43-43d1-ad31-1e3258f01596/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:20:13.186 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:20:13.186 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:20:13.186 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0223915 s, 46.8 MB/s 2021-02-10T11:20:13.187 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:20:13.187 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:20:13.187 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:20:13.527 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:20:13.528 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:20:13.528 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348598/aa5dd245-4e43-43d1-ad31-1e3258f01596/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:20:13.904 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:20:13.904 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:20:13.904 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0239922 s, 43.7 MB/s 2021-02-10T11:20:13.905 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:20:13.906 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:20:13.906 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:20:14.241 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:20:14.242 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:20:14.242 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348598/aa5dd245-4e43-43d1-ad31-1e3258f01596/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:20:14.614 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:20:14.614 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:20:14.614 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0240709 s, 43.6 MB/s 2021-02-10T11:20:14.615 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:20:14.616 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:20:14.616 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:20:14.951 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:20:14.952 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:20:14.952 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348598/aa5dd245-4e43-43d1-ad31-1e3258f01596/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:20:15.329 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:20:15.329 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:20:15.329 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0259589 s, 40.4 MB/s 2021-02-10T11:20:15.330 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:20:15.330 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:20:15.330 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:20:15.664 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:20:15.665 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:20:15.665 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348598/aa5dd245-4e43-43d1-ad31-1e3258f01596/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:20:16.042 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:20:16.042 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:20:16.042 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0261494 s, 40.1 MB/s 2021-02-10T11:20:16.043 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:20:16.043 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:20:16.043 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:20:16.378 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:20:16.379 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:20:16.379 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348598/aa5dd245-4e43-43d1-ad31-1e3258f01596/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:20:16.751 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:20:16.751 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:20:16.751 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0224041 s, 46.8 MB/s 2021-02-10T11:20:16.752 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:20:16.752 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:20:16.752 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:20:17.086 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:20:17.086 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:20:17.087 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348598/aa5dd245-4e43-43d1-ad31-1e3258f01596/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:20:17.464 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:20:17.464 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:20:17.464 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0254407 s, 41.2 MB/s 2021-02-10T11:20:17.465 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:20:17.465 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:20:17.465 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:20:17.800 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:20:17.801 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:20:17.801 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348598/aa5dd245-4e43-43d1-ad31-1e3258f01596/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:20:18.176 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:20:18.176 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:20:18.176 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0243664 s, 43.0 MB/s 2021-02-10T11:20:18.177 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:20:18.177 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:20:18.177 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:20:18.512 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:20:18.513 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:20:18.513 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348598/aa5dd245-4e43-43d1-ad31-1e3258f01596/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:20:18.886 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:20:18.887 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:20:18.887 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0225884 s, 46.4 MB/s 2021-02-10T11:20:18.888 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000348599 2021-02-10T11:20:19.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.306 7f1490ad3700 1 Processor -- start 2021-02-10T11:20:19.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.306 7f1490ad3700 1 -- start start 2021-02-10T11:20:19.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.307 7f1490ad3700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f148c1b0870 0x7f148c0f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:19.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.307 7f1490ad3700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f148c0efe00 0x7f148c0f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:19.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.307 7f1490ad3700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f148c1b0c70 con 0x7f148c1b0470 2021-02-10T11:20:19.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.307 7f1490ad3700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f148c0e5ea0 con 0x7f148c0efe00 2021-02-10T11:20:19.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.307 7f1490ad3700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f148c0f0b50 con 0x7f148c1b0870 2021-02-10T11:20:19.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.307 7f148b05f700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f148c0efe00 0x7f148c0f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:19.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.307 7f148a05d700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f148c1b0870 0x7f148c0f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:19.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.307 7f148a05d700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f148c1b0870 0x7f148c0f20e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3300/0 says I am v2:172.21.2.112:46942/0 (socket says 172.21.2.112:46942) 2021-02-10T11:20:19.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.307 7f148b05f700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f148c0efe00 0x7f148c0f0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:44662/0 (socket says 172.21.2.112:44662) 2021-02-10T11:20:19.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.307 7f148b05f700 1 -- 172.21.2.112:0/2815914000 learned_addr learned my addr 172.21.2.112:0/2815914000 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:20:19.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.308 7f148a05d700 1 -- 172.21.2.112:0/2815914000 >> v2:172.21.2.112:3300/0 conn(0x7f148c0efe00 msgr2=0x7f148c0f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:20:19.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.308 7f148a05d700 1 --2- 172.21.2.112:0/2815914000 >> v2:172.21.2.112:3300/0 conn(0x7f148c0efe00 0x7f148c0f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:19.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.308 7f148a05d700 1 -- 172.21.2.112:0/2815914000 >> v1:172.21.2.120:6789/0 conn(0x7f148c1b0470 legacy=0x7f148c0ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:19.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.308 7f148a05d700 1 -- 172.21.2.112:0/2815914000 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f148c0f0820 con 0x7f148c1b0870 2021-02-10T11:20:19.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.308 7f148b05f700 1 --2- 172.21.2.112:0/2815914000 >> v2:172.21.2.112:3300/0 conn(0x7f148c0efe00 0x7f148c0f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:20:19.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.308 7f148a05d700 1 --2- 172.21.2.112:0/2815914000 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f148c1b0870 0x7f148c0f20e0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f1480006650 tx=0x7f1480009a80).ready entity=mon.1 client_cookie=995acb44ca4579cb server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:19.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.308 7f148985c700 1 -- 172.21.2.112:0/2815914000 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f14800009b0 con 0x7f148c1b0870 2021-02-10T11:20:19.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.308 7f148985c700 1 -- 172.21.2.112:0/2815914000 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f148000c0c0 con 0x7f148c1b0870 2021-02-10T11:20:19.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.308 7f148985c700 1 -- 172.21.2.112:0/2815914000 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f148000c5e0 con 0x7f148c1b0870 2021-02-10T11:20:19.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.308 7f1490ad3700 1 -- 172.21.2.112:0/2815914000 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f148c1b0870 msgr2=0x7f148c0f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:19.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.308 7f1490ad3700 1 --2- 172.21.2.112:0/2815914000 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f148c1b0870 0x7f148c0f20e0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f1480006650 tx=0x7f1480009a80).stop 2021-02-10T11:20:19.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.309 7f1490ad3700 1 -- 172.21.2.112:0/2815914000 shutdown_connections 2021-02-10T11:20:19.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.309 7f1490ad3700 1 --2- 172.21.2.112:0/2815914000 >> v2:172.21.2.112:3300/0 conn(0x7f148c0efe00 0x7f148c0f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:19.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.309 7f1490ad3700 1 --2- 172.21.2.112:0/2815914000 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f148c1b0870 0x7f148c0f20e0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:19.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.309 7f1490ad3700 1 -- 172.21.2.112:0/2815914000 >> 172.21.2.112:0/2815914000 conn(0x7f148c0ce3e0 msgr2=0x7f148c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:20:19.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.309 7f1490ad3700 1 -- 172.21.2.112:0/2815914000 shutdown_connections 2021-02-10T11:20:19.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.309 7f1490ad3700 1 -- 172.21.2.112:0/2815914000 wait complete. 2021-02-10T11:20:19.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.324 7f1490ad3700 1 Processor -- start 2021-02-10T11:20:19.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.324 7f1490ad3700 1 -- start start 2021-02-10T11:20:19.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.324 7f1490ad3700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f148c08ece0 0x7f148c0a6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:19.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.324 7f1490ad3700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f148c0a6860 0x7f148c0a6c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:19.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.324 7f1490ad3700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f148c0a7280 0x7f148c0a4590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:19.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.324 7f1490ad3700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f148c0a5cc0 con 0x7f148c0a7280 2021-02-10T11:20:19.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.324 7f1490ad3700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f148c08f0e0 con 0x7f148c08ece0 2021-02-10T11:20:19.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.324 7f1490ad3700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f148c08f2c0 con 0x7f148c0a6860 2021-02-10T11:20:19.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.325 7f148b05f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f148c0a7280 0x7f148c0a4590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:19.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.325 7f148b05f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f148c0a7280 0x7f148c0a4590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:44666/0 (socket says 172.21.2.112:44666) 2021-02-10T11:20:19.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.325 7f148b05f700 1 -- 172.21.2.112:0/3623774670 learned_addr learned my addr 172.21.2.112:0/3623774670 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:20:19.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.325 7f148a85e700 1 --2- 172.21.2.112:0/3623774670 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f148c08ece0 0x7f148c0a6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:19.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.325 7f148a05d700 1 --2- 172.21.2.112:0/3623774670 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f148c0a6860 0x7f148c0a6c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:19.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.325 7f148b05f700 1 -- 172.21.2.112:0/3623774670 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f148c0a6860 msgr2=0x7f148c0a6c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:19.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.325 7f148b05f700 1 --2- 172.21.2.112:0/3623774670 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f148c0a6860 0x7f148c0a6c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:19.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.325 7f148b05f700 1 -- 172.21.2.112:0/3623774670 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f148c08ece0 msgr2=0x7f148c0a6240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:19.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.325 7f148b05f700 1 --2- 172.21.2.112:0/3623774670 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f148c08ece0 0x7f148c0a6240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:19.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.325 7f148b05f700 1 -- 172.21.2.112:0/3623774670 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f148c0a4bb0 con 0x7f148c0a7280 2021-02-10T11:20:19.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.325 7f148b05f700 1 --2- 172.21.2.112:0/3623774670 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f148c0a7280 0x7f148c0a4590 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 rx=0x7f148400b940 tx=0x7f148400bce0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:19.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.325 7f147b7fe700 1 -- 172.21.2.112:0/3623774670 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1484000f90 con 0x7f148c0a7280 2021-02-10T11:20:19.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.325 7f147b7fe700 1 -- 172.21.2.112:0/3623774670 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1484003030 con 0x7f148c0a7280 2021-02-10T11:20:19.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.325 7f147b7fe700 1 -- 172.21.2.112:0/3623774670 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1484001460 con 0x7f148c0a7280 2021-02-10T11:20:19.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.325 7f1490ad3700 1 -- 172.21.2.112:0/3623774670 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f148c0a4bb0 con 0x7f148c0a7280 2021-02-10T11:20:19.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.325 7f1490ad3700 1 -- 172.21.2.112:0/3623774670 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f148c0a4bb0 con 0x7f148c0a7280 2021-02-10T11:20:19.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.326 7f147b7fe700 1 -- 172.21.2.112:0/3623774670 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1484001270 con 0x7f148c0a7280 2021-02-10T11:20:19.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.326 7f147b7fe700 1 --2- 172.21.2.112:0/3623774670 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1470018730 0x7f147001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:19.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.326 7f147b7fe700 1 -- 172.21.2.112:0/3623774670 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f1484032b70 con 0x7f148c0a7280 2021-02-10T11:20:19.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.326 7f148a85e700 1 --2- 172.21.2.112:0/3623774670 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1470018730 0x7f147001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:19.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.327 7f148a85e700 1 --2- 172.21.2.112:0/3623774670 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1470018730 0x7f147001ac10 crc :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:19.349 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.354 7f1490ad3700 1 -- 172.21.2.112:0/3623774670 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f148c105ae0 con 0x7f148c0a7280 2021-02-10T11:20:19.351 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.357 7f147b7fe700 1 -- 172.21.2.112:0/3623774670 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f148400be20 con 0x7f148c0a7280 2021-02-10T11:20:19.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.486 7f1490ad3700 1 -- 172.21.2.112:0/3623774670 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348599", "target": ["mgr", ""]}) v1 -- 0x7f148c0f0b50 con 0x7f1470018730 2021-02-10T11:20:19.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.512 7f147b7fe700 1 -- 172.21.2.112:0/3623774670 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f147c006a90 con 0x7f1470018730 2021-02-10T11:20:19.514 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.519 7f1490ad3700 1 -- 172.21.2.112:0/3623774670 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1470018730 msgr2=0x7f147001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:19.514 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.519 7f1490ad3700 1 --2- 172.21.2.112:0/3623774670 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1470018730 0x7f147001ac10 crc :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:19.514 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.519 7f1490ad3700 1 -- 172.21.2.112:0/3623774670 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f148c0a7280 msgr2=0x7f148c0a4590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:19.514 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.519 7f1490ad3700 1 --2- 172.21.2.112:0/3623774670 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f148c0a7280 0x7f148c0a4590 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 rx=0x7f148400b940 tx=0x7f148400bce0).stop 2021-02-10T11:20:19.514 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.519 7f1490ad3700 1 -- 172.21.2.112:0/3623774670 shutdown_connections 2021-02-10T11:20:19.515 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.519 7f1490ad3700 1 --2- 172.21.2.112:0/3623774670 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1470018730 0x7f147001ac10 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:19.515 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.519 7f1490ad3700 1 --2- 172.21.2.112:0/3623774670 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f148c0a7280 0x7f148c0a4590 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:19.515 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.519 7f1490ad3700 1 --2- 172.21.2.112:0/3623774670 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f148c0a6860 0x7f148c0a6c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:19.515 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.519 7f1490ad3700 1 --2- 172.21.2.112:0/3623774670 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f148c08ece0 0x7f148c0a6240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:19.515 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.519 7f1490ad3700 1 -- 172.21.2.112:0/3623774670 >> 172.21.2.112:0/3623774670 conn(0x7f148c08ffe0 msgr2=0x7f148c0bf050 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:20:19.515 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.519 7f1490ad3700 1 -- 172.21.2.112:0/3623774670 shutdown_connections 2021-02-10T11:20:19.515 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.519 7f1490ad3700 1 -- 172.21.2.112:0/3623774670 wait complete. 2021-02-10T11:20:19.522 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000348599 2021-02-10T11:20:19.954 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.942 7f187b715700 1 Processor -- start 2021-02-10T11:20:19.954 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.943 7f187b715700 1 -- start start 2021-02-10T11:20:19.954 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.943 7f187b715700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f18740f05f0 0x7f18740f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:19.954 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.943 7f187b715700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18741b0c70 con 0x7f18741b0870 2021-02-10T11:20:19.955 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.943 7f187b715700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18740f0b50 con 0x7f18741b0470 2021-02-10T11:20:19.955 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.943 7f187b715700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18740e5ea0 con 0x7f18740f05f0 2021-02-10T11:20:19.955 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.943 7f1878d30700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f18741b0870 0x7f18740efe00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:44134/0 (socket says 172.21.2.112:44134) 2021-02-10T11:20:19.955 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.943 7f1878d30700 1 -- 172.21.2.112:0/3648273860 learned_addr learned my addr 172.21.2.112:0/3648273860 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:20:19.955 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.943 7f1879d32700 1 --2- 172.21.2.112:0/3648273860 >> v2:172.21.2.120:3300/0 conn(0x7f18740f05f0 0x7f18740f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:19.955 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.944 7f1867fff700 1 -- 172.21.2.112:0/3648273860 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3420856178 0 0) 0x7f1870000ce0 con 0x7f18741b0870 2021-02-10T11:20:19.955 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.944 7f1867fff700 1 -- 172.21.2.112:0/3648273860 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1860002fe0 con 0x7f18741b0870 2021-02-10T11:20:19.955 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.944 7f1879d32700 1 -- 172.21.2.112:0/3648273860 >> v1:172.21.2.120:6789/0 conn(0x7f18741b0470 legacy=0x7f18740ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:19.955 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.944 7f1879d32700 1 -- 172.21.2.112:0/3648273860 >> v1:172.21.2.112:6789/0 conn(0x7f18741b0870 legacy=0x7f18740efe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:19.955 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.944 7f1879d32700 1 -- 172.21.2.112:0/3648273860 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18740e52e0 con 0x7f18740f05f0 2021-02-10T11:20:19.955 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.944 7f1879d32700 1 --2- 172.21.2.112:0/3648273860 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f18740f05f0 0x7f18740f20e0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f186c006e50 tx=0x7f186c009e70).ready entity=mon.1 client_cookie=def91561e9f4c988 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:19.955 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.944 7f1867fff700 1 -- 172.21.2.112:0/3648273860 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f186c00b510 con 0x7f18740f05f0 2021-02-10T11:20:19.955 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.944 7f1867fff700 1 -- 172.21.2.112:0/3648273860 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f186c006790 con 0x7f18740f05f0 2021-02-10T11:20:19.956 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.944 7f1867fff700 1 -- 172.21.2.112:0/3648273860 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f186c00b940 con 0x7f18740f05f0 2021-02-10T11:20:19.956 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.944 7f187b715700 1 -- 172.21.2.112:0/3648273860 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f18740f05f0 msgr2=0x7f18740f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:19.956 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.944 7f187b715700 1 --2- 172.21.2.112:0/3648273860 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f18740f05f0 0x7f18740f20e0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f186c006e50 tx=0x7f186c009e70).stop 2021-02-10T11:20:19.956 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.945 7f187b715700 1 -- 172.21.2.112:0/3648273860 shutdown_connections 2021-02-10T11:20:19.956 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.945 7f187b715700 1 --2- 172.21.2.112:0/3648273860 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f18740f05f0 0x7f18740f20e0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:19.956 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.945 7f187b715700 1 -- 172.21.2.112:0/3648273860 >> 172.21.2.112:0/3648273860 conn(0x7f18740ce3e0 msgr2=0x7f18740e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:20:19.956 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.945 7f187b715700 1 -- 172.21.2.112:0/3648273860 shutdown_connections 2021-02-10T11:20:19.956 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.945 7f187b715700 1 -- 172.21.2.112:0/3648273860 wait complete. 2021-02-10T11:20:19.957 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.960 7f187b715700 1 Processor -- start 2021-02-10T11:20:19.958 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.960 7f187b715700 1 -- start start 2021-02-10T11:20:19.958 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.960 7f187b715700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f18740a0960 0x7f18740a67f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:19.958 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.960 7f187b715700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f18741924c0 0x7f18741928c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:19.958 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.960 7f187b715700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1874192ee0 0x7f18740a29f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:19.958 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.960 7f187b715700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18740a6e10 con 0x7f1874192ee0 2021-02-10T11:20:19.958 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.960 7f187b715700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1874190630 con 0x7f18741924c0 2021-02-10T11:20:19.958 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.960 7f187b715700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18741932e0 con 0x7f18740a0960 2021-02-10T11:20:19.958 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.960 7f1879d32700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1874192ee0 0x7f18740a29f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:19.958 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.960 7f1879d32700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1874192ee0 0x7f18740a29f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:44682/0 (socket says 172.21.2.112:44682) 2021-02-10T11:20:19.958 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.960 7f1879d32700 1 -- 172.21.2.112:0/2521578698 learned_addr learned my addr 172.21.2.112:0/2521578698 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:20:19.958 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.961 7f1878d30700 1 --2- 172.21.2.112:0/2521578698 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f18741924c0 0x7f18741928c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:19.958 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.961 7f1879531700 1 --2- 172.21.2.112:0/2521578698 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f18740a0960 0x7f18740a67f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:19.959 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.961 7f1879d32700 1 -- 172.21.2.112:0/2521578698 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f18740a0960 msgr2=0x7f18740a67f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:19.959 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.961 7f1879d32700 1 --2- 172.21.2.112:0/2521578698 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f18740a0960 0x7f18740a67f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:19.959 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.961 7f1879d32700 1 -- 172.21.2.112:0/2521578698 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f18741924c0 msgr2=0x7f18741928c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:19.959 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.961 7f1879d32700 1 --2- 172.21.2.112:0/2521578698 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f18741924c0 0x7f18741928c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:19.959 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.961 7f1879d32700 1 -- 172.21.2.112:0/2521578698 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1874193500 con 0x7f1874192ee0 2021-02-10T11:20:19.959 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.961 7f1879531700 1 --2- 172.21.2.112:0/2521578698 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f18740a0960 0x7f18740a67f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:20:19.959 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.961 7f1879d32700 1 --2- 172.21.2.112:0/2521578698 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1874192ee0 0x7f18740a29f0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 rx=0x7f186c006c50 tx=0x7f186c00bd20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:19.959 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.961 7f1865ffb700 1 -- 172.21.2.112:0/2521578698 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f186c0093d0 con 0x7f1874192ee0 2021-02-10T11:20:19.959 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.961 7f1865ffb700 1 -- 172.21.2.112:0/2521578698 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f186c00be60 con 0x7f1874192ee0 2021-02-10T11:20:19.959 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.961 7f187b715700 1 -- 172.21.2.112:0/2521578698 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1874193500 con 0x7f1874192ee0 2021-02-10T11:20:19.959 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.961 7f1865ffb700 1 -- 172.21.2.112:0/2521578698 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f186c00c570 con 0x7f1874192ee0 2021-02-10T11:20:19.959 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.961 7f187b715700 1 -- 172.21.2.112:0/2521578698 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1874193500 con 0x7f1874192ee0 2021-02-10T11:20:19.959 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.962 7f1865ffb700 1 -- 172.21.2.112:0/2521578698 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f186c002ef0 con 0x7f1874192ee0 2021-02-10T11:20:19.960 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.962 7f1865ffb700 1 --2- 172.21.2.112:0/2521578698 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f18540186d0 0x7f185401abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:19.960 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.962 7f1865ffb700 1 -- 172.21.2.112:0/2521578698 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f186c0329f0 con 0x7f1874192ee0 2021-02-10T11:20:19.960 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.962 7f1879531700 1 --2- 172.21.2.112:0/2521578698 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f18540186d0 0x7f185401abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:19.960 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.963 7f1879531700 1 --2- 172.21.2.112:0/2521578698 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f18540186d0 0x7f185401abd0 crc :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:19.988 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.993 7f187b715700 1 -- 172.21.2.112:0/2521578698 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18740a24c0 con 0x7f1874192ee0 2021-02-10T11:20:19.990 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:19.996 7f1865ffb700 1 -- 172.21.2.112:0/2521578698 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f186c000df0 con 0x7f1874192ee0 2021-02-10T11:20:20.119 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:20.125 7f187b715700 1 -- 172.21.2.112:0/2521578698 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348599", "target": ["mgr", ""]}) v1 -- 0x7f18740a24c0 con 0x7f18540186d0 2021-02-10T11:20:20.123 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:20.129 7f1865ffb700 1 -- 172.21.2.112:0/2521578698 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f1868000960 con 0x7f18540186d0 2021-02-10T11:20:20.126 INFO:teuthology.orchestra.run.gibba012.stdout:/volumes/_nogroup/subvolume_0000000000348599/d02b8005-90d6-4c9a-8069-5f47215bd78f 2021-02-10T11:20:20.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:20.133 7f187b715700 1 -- 172.21.2.112:0/2521578698 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f18540186d0 msgr2=0x7f185401abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:20.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:20.133 7f187b715700 1 --2- 172.21.2.112:0/2521578698 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f18540186d0 0x7f185401abd0 crc :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:20.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:20.133 7f187b715700 1 -- 172.21.2.112:0/2521578698 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1874192ee0 msgr2=0x7f18740a29f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:20.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:20.133 7f187b715700 1 --2- 172.21.2.112:0/2521578698 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1874192ee0 0x7f18740a29f0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 rx=0x7f186c006c50 tx=0x7f186c00bd20).stop 2021-02-10T11:20:20.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:20.134 7f187b715700 1 -- 172.21.2.112:0/2521578698 shutdown_connections 2021-02-10T11:20:20.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:20.134 7f187b715700 1 --2- 172.21.2.112:0/2521578698 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f18540186d0 0x7f185401abd0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:20.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:20.134 7f187b715700 1 --2- 172.21.2.112:0/2521578698 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f18741924c0 0x7f18741928c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:20.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:20.134 7f187b715700 1 --2- 172.21.2.112:0/2521578698 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1874192ee0 0x7f18740a29f0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:20.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:20.134 7f187b715700 1 --2- 172.21.2.112:0/2521578698 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f18740a0960 0x7f18740a67f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:20.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:20.134 7f187b715700 1 -- 172.21.2.112:0/2521578698 >> 172.21.2.112:0/2521578698 conn(0x7f18740ac550 msgr2=0x7f18740be1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:20:20.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:20.134 7f187b715700 1 -- 172.21.2.112:0/2521578698 shutdown_connections 2021-02-10T11:20:20.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:20.134 7f187b715700 1 -- 172.21.2.112:0/2521578698 wait complete. 2021-02-10T11:20:20.136 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000348599 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000348599/d02b8005-90d6-4c9a-8069-5f47215bd78f 2021-02-10T11:20:20.136 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:20:20.136 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:20:20.137 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:20:20.472 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:20:20.473 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:20:20.473 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348599/d02b8005-90d6-4c9a-8069-5f47215bd78f/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:20:20.854 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:20:20.854 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:20:20.854 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0236757 s, 44.3 MB/s 2021-02-10T11:20:20.855 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:20:20.855 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:20:20.855 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:20:21.189 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:20:21.189 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:20:21.190 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348599/d02b8005-90d6-4c9a-8069-5f47215bd78f/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:20:21.567 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:20:21.567 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:20:21.567 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.025475 s, 41.2 MB/s 2021-02-10T11:20:21.568 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:20:21.568 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:20:21.568 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:20:21.903 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:20:21.903 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:20:21.904 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348599/d02b8005-90d6-4c9a-8069-5f47215bd78f/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:20:22.278 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:20:22.278 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:20:22.278 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0252391 s, 41.5 MB/s 2021-02-10T11:20:22.279 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:20:22.279 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:20:22.280 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:20:22.614 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:20:22.615 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:20:22.615 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348599/d02b8005-90d6-4c9a-8069-5f47215bd78f/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:20:22.992 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:20:22.992 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:20:22.992 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0250248 s, 41.9 MB/s 2021-02-10T11:20:22.993 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:20:22.993 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:20:22.993 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:20:23.328 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:20:23.329 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:20:23.329 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348599/d02b8005-90d6-4c9a-8069-5f47215bd78f/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:20:23.703 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:20:23.703 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:20:23.703 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0251734 s, 41.7 MB/s 2021-02-10T11:20:23.704 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:20:23.704 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:20:23.704 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:20:24.040 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:20:24.041 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:20:24.041 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348599/d02b8005-90d6-4c9a-8069-5f47215bd78f/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:20:24.411 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:20:24.411 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:20:24.411 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0225933 s, 46.4 MB/s 2021-02-10T11:20:24.412 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:20:24.412 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:20:24.412 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:20:24.747 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:20:24.747 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:20:24.748 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348599/d02b8005-90d6-4c9a-8069-5f47215bd78f/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:20:25.123 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:20:25.123 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:20:25.123 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0244903 s, 42.8 MB/s 2021-02-10T11:20:25.124 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:20:25.124 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:20:25.124 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:20:25.459 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:20:25.459 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:20:25.459 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348599/d02b8005-90d6-4c9a-8069-5f47215bd78f/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:20:25.838 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:20:25.838 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:20:25.839 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0290124 s, 36.1 MB/s 2021-02-10T11:20:25.839 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:20:25.840 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:20:25.840 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:20:26.176 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:20:26.176 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:20:26.176 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348599/d02b8005-90d6-4c9a-8069-5f47215bd78f/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:20:26.550 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:20:26.550 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:20:26.550 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0231617 s, 45.3 MB/s 2021-02-10T11:20:26.551 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:20:26.552 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:20:26.552 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:20:26.886 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:20:26.886 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:20:26.886 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348599/d02b8005-90d6-4c9a-8069-5f47215bd78f/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:20:27.261 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:20:27.262 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:20:27.262 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0246112 s, 42.6 MB/s 2021-02-10T11:20:27.263 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000348600 2021-02-10T11:20:27.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:27.682 7f5c0b6f5700 1 Processor -- start 2021-02-10T11:20:27.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:27.682 7f5c0b6f5700 1 -- start start 2021-02-10T11:20:27.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:27.682 7f5c0b6f5700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f5c041b0870 0x7f5c040f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:27.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:27.682 7f5c0b6f5700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f5c040efe00 0x7f5c040f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:27.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:27.682 7f5c0b6f5700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c041b0c70 con 0x7f5c041b0470 2021-02-10T11:20:27.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:27.682 7f5c0b6f5700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c040e5ea0 con 0x7f5c040efe00 2021-02-10T11:20:27.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:27.682 7f5c0b6f5700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c040f0b50 con 0x7f5c041b0870 2021-02-10T11:20:27.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:27.683 7f5c09d12700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f5c040efe00 0x7f5c040f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:27.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:27.683 7f5c09d12700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f5c040efe00 0x7f5c040f0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:44690/0 (socket says 172.21.2.112:44690) 2021-02-10T11:20:27.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:27.683 7f5c09d12700 1 -- 172.21.2.112:0/2006043953 learned_addr learned my addr 172.21.2.112:0/2006043953 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:20:27.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:27.683 7f5c09d12700 1 -- 172.21.2.112:0/2006043953 >> v2:172.21.2.120:3300/0 conn(0x7f5c041b0870 msgr2=0x7f5c040f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:20:27.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:27.683 7f5c09d12700 1 --2- 172.21.2.112:0/2006043953 >> v2:172.21.2.120:3300/0 conn(0x7f5c041b0870 0x7f5c040f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:20:27.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:27.683 7f5c09d12700 1 -- 172.21.2.112:0/2006043953 >> v1:172.21.2.120:6789/0 conn(0x7f5c041b0470 legacy=0x7f5c040ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:20:27.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:27.683 7f5c09d12700 1 -- 172.21.2.112:0/2006043953 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c040f0820 con 0x7f5c040efe00 2021-02-10T11:20:27.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:27.683 7f5c09d12700 1 --2- 172.21.2.112:0/2006043953 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5c040efe00 0x7f5c040f0200 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 rx=0x7f5bfc006f00 tx=0x7f5bfc009f10).ready entity=mon.0 client_cookie=5999b7660f4997fd server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:27.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:27.683 7f5bf7fff700 1 -- 172.21.2.112:0/2006043953 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5bfc00b3b0 con 0x7f5c040efe00 2021-02-10T11:20:27.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:27.683 7f5bf7fff700 1 -- 172.21.2.112:0/2006043953 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bfc000d30 con 0x7f5c040efe00 2021-02-10T11:20:27.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:27.683 7f5bf7fff700 1 -- 172.21.2.112:0/2006043953 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5bfc00b8b0 con 0x7f5c040efe00 2021-02-10T11:20:27.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:27.683 7f5c0b6f5700 1 -- 172.21.2.112:0/2006043953 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5c040efe00 msgr2=0x7f5c040f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:27.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:27.683 7f5c0b6f5700 1 --2- 172.21.2.112:0/2006043953 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5c040efe00 0x7f5c040f0200 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 rx=0x7f5bfc006f00 tx=0x7f5bfc009f10).stop 2021-02-10T11:20:27.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:27.684 7f5c0b6f5700 1 -- 172.21.2.112:0/2006043953 shutdown_connections 2021-02-10T11:20:27.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:27.684 7f5c0b6f5700 1 --2- 172.21.2.112:0/2006043953 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5c040efe00 0x7f5c040f0200 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:27.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:27.684 7f5c0b6f5700 1 --2- 172.21.2.112:0/2006043953 >> v2:172.21.2.120:3300/0 conn(0x7f5c041b0870 0x7f5c040f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:20:27.695 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:27.684 7f5c0b6f5700 1 -- 172.21.2.112:0/2006043953 >> 172.21.2.112:0/2006043953 conn(0x7f5c040ce3e0 msgr2=0x7f5c040e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:20:27.695 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:27.684 7f5c0b6f5700 1 -- 172.21.2.112:0/2006043953 shutdown_connections 2021-02-10T11:20:27.695 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:27.684 7f5c0b6f5700 1 -- 172.21.2.112:0/2006043953 wait complete. 2021-02-10T11:20:27.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:27.698 7f5c0b6f5700 1 Processor -- start 2021-02-10T11:20:27.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:27.699 7f5c0b6f5700 1 -- start start 2021-02-10T11:20:27.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:27.699 7f5c0b6f5700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5c040c36a0 0x7f5c040c2b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:27.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:27.699 7f5c0b6f5700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5c040c3190 0x7f5c040b1d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:27.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:27.699 7f5c0b6f5700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5c040b2370 0x7f5c040b2d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:27.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:27.699 7f5c0b6f5700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c040b27c0 con 0x7f5c040c3190 2021-02-10T11:20:27.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:27.699 7f5c0b6f5700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c040b3360 con 0x7f5c040b2370 2021-02-10T11:20:27.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:27.699 7f5c0b6f5700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c040b3540 con 0x7f5c040c36a0 2021-02-10T11:20:27.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:27.699 7f5c08d10700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5c040c3190 0x7f5c040b1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:27.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:27.699 7f5c08d10700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5c040c3190 0x7f5c040b1d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:44696/0 (socket says 172.21.2.112:44696) 2021-02-10T11:20:27.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:27.699 7f5c08d10700 1 -- 172.21.2.112:0/510729118 learned_addr learned my addr 172.21.2.112:0/510729118 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:20:27.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:27.699 7f5c08d10700 1 -- 172.21.2.112:0/510729118 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5c040c36a0 msgr2=0x7f5c040c2b70 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:20:27.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:27.699 7f5c08d10700 1 --2- 172.21.2.112:0/510729118 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5c040c36a0 0x7f5c040c2b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:20:27.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:27.699 7f5c08d10700 1 -- 172.21.2.112:0/510729118 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5c040b2370 msgr2=0x7f5c040b2d40 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:20:27.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:27.699 7f5c08d10700 1 --2- 172.21.2.112:0/510729118 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5c040b2370 0x7f5c040b2d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:20:27.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:27.699 7f5c08d10700 1 -- 172.21.2.112:0/510729118 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c040b3760 con 0x7f5c040c3190 2021-02-10T11:20:27.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:27.700 7f5c08d10700 1 --2- 172.21.2.112:0/510729118 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5c040c3190 0x7f5c040b1d50 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 rx=0x7f5bf800b530 tx=0x7f5bf800b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:27.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:27.700 7f5bf5ffb700 1 -- 172.21.2.112:0/510729118 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5bf800bcc0 con 0x7f5c040c3190 2021-02-10T11:20:27.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:27.700 7f5bf5ffb700 1 -- 172.21.2.112:0/510729118 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bf8000c90 con 0x7f5c040c3190 2021-02-10T11:20:27.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:27.700 7f5bf5ffb700 1 -- 172.21.2.112:0/510729118 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5bf80022a0 con 0x7f5c040c3190 2021-02-10T11:20:27.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:27.700 7f5c0b6f5700 1 -- 172.21.2.112:0/510729118 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c040b3760 con 0x7f5c040c3190 2021-02-10T11:20:27.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:27.700 7f5c0b6f5700 1 -- 172.21.2.112:0/510729118 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c040b3760 con 0x7f5c040c3190 2021-02-10T11:20:27.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:27.701 7f5bf5ffb700 1 -- 172.21.2.112:0/510729118 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5bf8005310 con 0x7f5c040c3190 2021-02-10T11:20:27.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:27.701 7f5bf5ffb700 1 --2- 172.21.2.112:0/510729118 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5be40186c0 0x7f5be401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:27.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:27.701 7f5bf5ffb700 1 -- 172.21.2.112:0/510729118 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f5bf8032540 con 0x7f5c040c3190 2021-02-10T11:20:27.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:27.701 7f5c09511700 1 --2- 172.21.2.112:0/510729118 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5be40186c0 0x7f5be401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:27.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:27.702 7f5c09511700 1 --2- 172.21.2.112:0/510729118 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5be40186c0 0x7f5be401abc0 crc :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:27.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:27.732 7f5c0b6f5700 1 -- 172.21.2.112:0/510729118 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c040edd10 con 0x7f5c040c3190 2021-02-10T11:20:27.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:27.735 7f5bf5ffb700 1 -- 172.21.2.112:0/510729118 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5bf8025fe0 con 0x7f5c040c3190 2021-02-10T11:20:27.857 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:27.862 7f5c0b6f5700 1 -- 172.21.2.112:0/510729118 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348600", "target": ["mgr", ""]}) v1 -- 0x7f5c040e52e0 con 0x7f5be40186c0 2021-02-10T11:20:27.882 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:27.887 7f5bf5ffb700 1 -- 172.21.2.112:0/510729118 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f5c00006970 con 0x7f5be40186c0 2021-02-10T11:20:27.889 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:27.895 7f5c0b6f5700 1 -- 172.21.2.112:0/510729118 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5be40186c0 msgr2=0x7f5be401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:27.890 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:27.895 7f5c0b6f5700 1 --2- 172.21.2.112:0/510729118 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5be40186c0 0x7f5be401abc0 crc :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:27.890 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:27.895 7f5c0b6f5700 1 -- 172.21.2.112:0/510729118 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5c040c3190 msgr2=0x7f5c040b1d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:27.890 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:27.895 7f5c0b6f5700 1 --2- 172.21.2.112:0/510729118 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5c040c3190 0x7f5c040b1d50 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 rx=0x7f5bf800b530 tx=0x7f5bf800b8d0).stop 2021-02-10T11:20:27.890 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:27.895 7f5c0b6f5700 1 -- 172.21.2.112:0/510729118 shutdown_connections 2021-02-10T11:20:27.890 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:27.895 7f5c0b6f5700 1 --2- 172.21.2.112:0/510729118 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5be40186c0 0x7f5be401abc0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:27.890 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:27.895 7f5c0b6f5700 1 --2- 172.21.2.112:0/510729118 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5c040b2370 0x7f5c040b2d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:20:27.891 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:27.895 7f5c0b6f5700 1 --2- 172.21.2.112:0/510729118 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5c040c3190 0x7f5c040b1d50 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:27.891 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:27.895 7f5c0b6f5700 1 --2- 172.21.2.112:0/510729118 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5c040c36a0 0x7f5c040c2b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:20:27.891 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:27.895 7f5c0b6f5700 1 -- 172.21.2.112:0/510729118 >> 172.21.2.112:0/510729118 conn(0x7f5c0409f630 msgr2=0x7f5c040a0130 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:20:27.891 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:27.895 7f5c0b6f5700 1 -- 172.21.2.112:0/510729118 shutdown_connections 2021-02-10T11:20:27.891 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:27.895 7f5c0b6f5700 1 -- 172.21.2.112:0/510729118 wait complete. 2021-02-10T11:20:27.898 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000348600 2021-02-10T11:20:28.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:28.319 7f5153126700 1 Processor -- start 2021-02-10T11:20:28.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:28.319 7f5153126700 1 -- start start 2021-02-10T11:20:28.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:28.319 7f5153126700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f514c0f05f0 0x7f514c0f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:28.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:28.319 7f5153126700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f514c1b0c70 con 0x7f514c1b0470 2021-02-10T11:20:28.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:28.319 7f5153126700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f514c0f0b50 con 0x7f514c1b0870 2021-02-10T11:20:28.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:28.319 7f5153126700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f514c0e5ea0 con 0x7f514c0f05f0 2021-02-10T11:20:28.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:28.319 7f5151743700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f514c0f05f0 0x7f514c0f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:28.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:28.319 7f5151743700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f514c0f05f0 0x7f514c0f20e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:44704/0 (socket says 172.21.2.112:44704) 2021-02-10T11:20:28.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:28.319 7f5151743700 1 -- 172.21.2.112:0/4209624173 learned_addr learned my addr 172.21.2.112:0/4209624173 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:20:28.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:28.320 7f5151743700 1 -- 172.21.2.112:0/4209624173 >> v1:172.21.2.120:6789/0 conn(0x7f514c1b0870 legacy=0x7f514c0efe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:28.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:28.320 7f5151743700 1 -- 172.21.2.112:0/4209624173 >> v1:172.21.2.112:6789/0 conn(0x7f514c1b0470 legacy=0x7f514c0ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:28.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:28.320 7f5151743700 1 -- 172.21.2.112:0/4209624173 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f514c0e52e0 con 0x7f514c0f05f0 2021-02-10T11:20:28.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:28.320 7f5151743700 1 --2- 172.21.2.112:0/4209624173 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f514c0f05f0 0x7f514c0f20e0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 rx=0x7f5144006b80 tx=0x7f5144009b90).ready entity=mon.0 client_cookie=fbf7f006358289f9 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:28.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:28.320 7f514b7fe700 1 -- 172.21.2.112:0/4209624173 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f51440022e0 con 0x7f514c0f05f0 2021-02-10T11:20:28.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:28.320 7f514b7fe700 1 -- 172.21.2.112:0/4209624173 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5144002560 con 0x7f514c0f05f0 2021-02-10T11:20:28.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:28.320 7f514b7fe700 1 -- 172.21.2.112:0/4209624173 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5144002a80 con 0x7f514c0f05f0 2021-02-10T11:20:28.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:28.320 7f5153126700 1 -- 172.21.2.112:0/4209624173 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f514c0f05f0 msgr2=0x7f514c0f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:28.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:28.320 7f5153126700 1 --2- 172.21.2.112:0/4209624173 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f514c0f05f0 0x7f514c0f20e0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 rx=0x7f5144006b80 tx=0x7f5144009b90).stop 2021-02-10T11:20:28.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:28.321 7f5153126700 1 -- 172.21.2.112:0/4209624173 shutdown_connections 2021-02-10T11:20:28.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:28.321 7f5153126700 1 --2- 172.21.2.112:0/4209624173 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f514c0f05f0 0x7f514c0f20e0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:28.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:28.321 7f5153126700 1 -- 172.21.2.112:0/4209624173 >> 172.21.2.112:0/4209624173 conn(0x7f514c0ce3e0 msgr2=0x7f514c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:20:28.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:28.321 7f5153126700 1 -- 172.21.2.112:0/4209624173 shutdown_connections 2021-02-10T11:20:28.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:28.321 7f5153126700 1 -- 172.21.2.112:0/4209624173 wait complete. 2021-02-10T11:20:28.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:28.336 7f5153126700 1 Processor -- start 2021-02-10T11:20:28.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:28.336 7f5153126700 1 -- start start 2021-02-10T11:20:28.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:28.336 7f5153126700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f514c0a3b50 0x7f514c1b3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:28.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:28.336 7f5153126700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f514c1c25b0 0x7f514c1b3d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:28.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:28.336 7f5153126700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f514c1b4350 0x7f514c1b18a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:28.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:28.336 7f5153126700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f514c1b3190 con 0x7f514c0a3b50 2021-02-10T11:20:28.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:28.336 7f5153126700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f514c087750 con 0x7f514c1c25b0 2021-02-10T11:20:28.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:28.336 7f5153126700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f514c0a6bf0 con 0x7f514c1b4350 2021-02-10T11:20:28.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:28.336 7f5150f42700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f514c0a3b50 0x7f514c1b3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:28.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:28.336 7f5150f42700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f514c0a3b50 0x7f514c1b3710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:44708/0 (socket says 172.21.2.112:44708) 2021-02-10T11:20:28.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:28.336 7f5150f42700 1 -- 172.21.2.112:0/3204907668 learned_addr learned my addr 172.21.2.112:0/3204907668 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:20:28.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:28.337 7f514bfff700 1 --2- 172.21.2.112:0/3204907668 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f514c1c25b0 0x7f514c1b3d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:28.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:28.337 7f5151743700 1 --2- 172.21.2.112:0/3204907668 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f514c1b4350 0x7f514c1b18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:28.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:28.337 7f5150f42700 1 -- 172.21.2.112:0/3204907668 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f514c1b4350 msgr2=0x7f514c1b18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:28.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:28.337 7f5150f42700 1 --2- 172.21.2.112:0/3204907668 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f514c1b4350 0x7f514c1b18a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:28.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:28.337 7f5150f42700 1 -- 172.21.2.112:0/3204907668 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f514c1c25b0 msgr2=0x7f514c1b3d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:28.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:28.337 7f5150f42700 1 --2- 172.21.2.112:0/3204907668 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f514c1c25b0 0x7f514c1b3d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:28.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:28.337 7f5150f42700 1 -- 172.21.2.112:0/3204907668 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f514c1b1ec0 con 0x7f514c0a3b50 2021-02-10T11:20:28.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:28.337 7f5150f42700 1 --2- 172.21.2.112:0/3204907668 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f514c0a3b50 0x7f514c1b3710 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 rx=0x7f5134006ed0 tx=0x7f513400b8e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:28.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:28.337 7f51497fa700 1 -- 172.21.2.112:0/3204907668 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5134006020 con 0x7f514c0a3b50 2021-02-10T11:20:28.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:28.337 7f51497fa700 1 -- 172.21.2.112:0/3204907668 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5134006360 con 0x7f514c0a3b50 2021-02-10T11:20:28.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:28.337 7f51497fa700 1 -- 172.21.2.112:0/3204907668 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5134005730 con 0x7f514c0a3b50 2021-02-10T11:20:28.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:28.337 7f5153126700 1 -- 172.21.2.112:0/3204907668 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f514c1b1ec0 con 0x7f514c0a3b50 2021-02-10T11:20:28.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:28.337 7f5153126700 1 -- 172.21.2.112:0/3204907668 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f514c1b1ec0 con 0x7f514c0a3b50 2021-02-10T11:20:28.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:28.338 7f51497fa700 1 -- 172.21.2.112:0/3204907668 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f51340054b0 con 0x7f514c0a3b50 2021-02-10T11:20:28.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:28.338 7f51497fa700 1 --2- 172.21.2.112:0/3204907668 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5130018730 0x7f513001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:28.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:28.338 7f51497fa700 1 -- 172.21.2.112:0/3204907668 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f51340327b0 con 0x7f514c0a3b50 2021-02-10T11:20:28.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:28.338 7f514bfff700 1 --2- 172.21.2.112:0/3204907668 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5130018730 0x7f513001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:28.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:28.339 7f514bfff700 1 --2- 172.21.2.112:0/3204907668 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5130018730 0x7f513001abc0 crc :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:28.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:28.369 7f5153126700 1 -- 172.21.2.112:0/3204907668 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f514c07e6b0 con 0x7f514c0a3b50 2021-02-10T11:20:28.366 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:28.371 7f51497fa700 1 -- 172.21.2.112:0/3204907668 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f51340054b0 con 0x7f514c0a3b50 2021-02-10T11:20:28.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:28.499 7f5153126700 1 -- 172.21.2.112:0/3204907668 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348600", "target": ["mgr", ""]}) v1 -- 0x7f514c1b3190 con 0x7f5130018730 2021-02-10T11:20:28.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:28.502 7f51497fa700 1 -- 172.21.2.112:0/3204907668 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f513c005830 con 0x7f5130018730 2021-02-10T11:20:28.501 INFO:teuthology.orchestra.run.gibba012.stdout:/volumes/_nogroup/subvolume_0000000000348600/184b85c3-24f9-4491-9c54-42b7199d4b7b 2021-02-10T11:20:28.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:28.508 7f5153126700 1 -- 172.21.2.112:0/3204907668 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5130018730 msgr2=0x7f513001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:28.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:28.508 7f5153126700 1 --2- 172.21.2.112:0/3204907668 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5130018730 0x7f513001abc0 crc :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:28.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:28.508 7f5153126700 1 -- 172.21.2.112:0/3204907668 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f514c0a3b50 msgr2=0x7f514c1b3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:28.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:28.508 7f5153126700 1 --2- 172.21.2.112:0/3204907668 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f514c0a3b50 0x7f514c1b3710 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 rx=0x7f5134006ed0 tx=0x7f513400b8e0).stop 2021-02-10T11:20:28.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:28.508 7f5153126700 1 -- 172.21.2.112:0/3204907668 shutdown_connections 2021-02-10T11:20:28.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:28.508 7f5153126700 1 --2- 172.21.2.112:0/3204907668 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5130018730 0x7f513001abc0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:28.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:28.508 7f5153126700 1 --2- 172.21.2.112:0/3204907668 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f514c1c25b0 0x7f514c1b3d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:28.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:28.508 7f5153126700 1 --2- 172.21.2.112:0/3204907668 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f514c1b4350 0x7f514c1b18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:28.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:28.508 7f5153126700 1 --2- 172.21.2.112:0/3204907668 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f514c0a3b50 0x7f514c1b3710 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:28.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:28.508 7f5153126700 1 -- 172.21.2.112:0/3204907668 >> 172.21.2.112:0/3204907668 conn(0x7f514c0886d0 msgr2=0x7f514c081c40 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:20:28.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:28.508 7f5153126700 1 -- 172.21.2.112:0/3204907668 shutdown_connections 2021-02-10T11:20:28.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:28.508 7f5153126700 1 -- 172.21.2.112:0/3204907668 wait complete. 2021-02-10T11:20:28.511 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000348600 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000348600/184b85c3-24f9-4491-9c54-42b7199d4b7b 2021-02-10T11:20:28.511 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:20:28.511 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:20:28.511 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:20:28.846 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:20:28.847 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:20:28.847 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348600/184b85c3-24f9-4491-9c54-42b7199d4b7b/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:20:29.228 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:20:29.228 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:20:29.228 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0240256 s, 43.6 MB/s 2021-02-10T11:20:29.229 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:20:29.229 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:20:29.229 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:20:29.564 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:20:29.565 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:20:29.565 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348600/184b85c3-24f9-4491-9c54-42b7199d4b7b/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:20:29.938 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:20:29.938 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:20:29.938 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0248481 s, 42.2 MB/s 2021-02-10T11:20:29.939 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:20:29.939 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:20:29.940 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:20:30.275 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:20:30.275 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:20:30.275 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348600/184b85c3-24f9-4491-9c54-42b7199d4b7b/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:20:30.651 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:20:30.651 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:20:30.651 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.025011 s, 41.9 MB/s 2021-02-10T11:20:30.652 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:20:30.652 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:20:30.652 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:20:30.988 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:20:30.988 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:20:30.988 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348600/184b85c3-24f9-4491-9c54-42b7199d4b7b/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:20:31.364 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:20:31.364 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:20:31.364 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0258853 s, 40.5 MB/s 2021-02-10T11:20:31.365 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:20:31.365 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:20:31.366 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:20:31.699 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:20:31.699 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:20:31.699 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348600/184b85c3-24f9-4491-9c54-42b7199d4b7b/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:20:32.073 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:20:32.074 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:20:32.074 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0246293 s, 42.6 MB/s 2021-02-10T11:20:32.075 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:20:32.075 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:20:32.075 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:20:32.410 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:20:32.411 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:20:32.411 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348600/184b85c3-24f9-4491-9c54-42b7199d4b7b/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:20:32.787 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:20:33.340 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:20:33.340 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0255075 s, 41.1 MB/s 2021-02-10T11:20:33.341 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:20:33.341 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:20:33.341 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:20:33.677 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:20:33.677 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:20:33.678 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348600/184b85c3-24f9-4491-9c54-42b7199d4b7b/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:20:34.053 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:20:34.053 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:20:34.053 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0253744 s, 41.3 MB/s 2021-02-10T11:20:34.054 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:20:34.054 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:20:34.054 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:20:34.388 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:20:34.388 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:20:34.388 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348600/184b85c3-24f9-4491-9c54-42b7199d4b7b/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:20:34.760 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:20:34.760 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:20:34.760 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0238223 s, 44.0 MB/s 2021-02-10T11:20:34.761 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:20:34.761 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:20:34.761 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:20:35.096 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:20:35.096 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:20:35.097 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348600/184b85c3-24f9-4491-9c54-42b7199d4b7b/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:20:35.474 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:20:35.474 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:20:35.474 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0251324 s, 41.7 MB/s 2021-02-10T11:20:35.475 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:20:35.475 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:20:35.475 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:20:35.810 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:20:35.811 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:20:35.811 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348600/184b85c3-24f9-4491-9c54-42b7199d4b7b/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:20:36.052 DEBUG:teuthology.orchestra.run.gibba012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:20:36.055 DEBUG:teuthology.orchestra.run.gibba020:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:20:36.191 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:20:36.192 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:20:36.192 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0304891 s, 34.4 MB/s 2021-02-10T11:20:36.193 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000348601 2021-02-10T11:20:36.624 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:36.610 7f5fa8543700 1 Processor -- start 2021-02-10T11:20:36.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:36.610 7f5fa8543700 1 -- start start 2021-02-10T11:20:36.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:36.611 7f5fa8543700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f5fa01b0870 0x7f5fa00f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:36.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:36.611 7f5fa8543700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fa01b0c70 con 0x7f5fa00efe00 2021-02-10T11:20:36.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:36.611 7f5fa8543700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fa00f0b50 con 0x7f5fa01b0470 2021-02-10T11:20:36.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:36.611 7f5fa8543700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fa00e5ea0 con 0x7f5fa01b0870 2021-02-10T11:20:36.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:36.611 7f5fa6b60700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f5fa00efe00 0x7f5fa00f0200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:44180/0 (socket says 172.21.2.112:44180) 2021-02-10T11:20:36.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:36.611 7f5fa6b60700 1 -- 172.21.2.112:0/1923107155 learned_addr learned my addr 172.21.2.112:0/1923107155 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:20:36.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:36.611 7f5fa535d700 1 -- 172.21.2.112:0/1923107155 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2121884896 0 0) 0x7f5f98001500 con 0x7f5fa00efe00 2021-02-10T11:20:36.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:36.611 7f5fa535d700 1 -- 172.21.2.112:0/1923107155 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f8c002fe0 con 0x7f5fa00efe00 2021-02-10T11:20:36.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:36.612 7f5fa5b5e700 1 --2- 172.21.2.112:0/1923107155 >> v2:172.21.2.120:3300/0 conn(0x7f5fa01b0870 0x7f5fa00f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:36.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:36.612 7f5fa535d700 1 -- 172.21.2.112:0/1923107155 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 411546279 0 0) 0x7f5f98001460 con 0x7f5fa00efe00 2021-02-10T11:20:36.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:36.612 7f5fa535d700 1 -- 172.21.2.112:0/1923107155 >> v2:172.21.2.120:3300/0 conn(0x7f5fa01b0870 msgr2=0x7f5fa00f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:20:36.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:36.612 7f5fa535d700 1 --2- 172.21.2.112:0/1923107155 >> v2:172.21.2.120:3300/0 conn(0x7f5fa01b0870 0x7f5fa00f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:36.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:36.612 7f5fa535d700 1 -- 172.21.2.112:0/1923107155 >> v1:172.21.2.120:6789/0 conn(0x7f5fa01b0470 legacy=0x7f5fa00ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:20:36.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:36.612 7f5fa535d700 1 -- 172.21.2.112:0/1923107155 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fa00e52e0 con 0x7f5fa00efe00 2021-02-10T11:20:36.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:36.612 7f5fa535d700 1 -- 172.21.2.112:0/1923107155 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f5f98001980 con 0x7f5fa00efe00 2021-02-10T11:20:36.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:36.612 7f5fa535d700 1 -- 172.21.2.112:0/1923107155 >> v1:172.21.2.112:6789/0 conn(0x7f5fa00efe00 legacy=0x7f5fa00f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:36.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:36.612 7f5fa535d700 1 --2- 172.21.2.112:0/1923107155 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5f8c003aa0 0x7f5f8c005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:36.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:36.612 7f5fa535d700 1 --2- 172.21.2.112:0/1923107155 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5f8c006770 0x7f5f8c008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:36.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:36.612 7f5fa535d700 1 --2- 172.21.2.112:0/1923107155 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5f8c009590 0x7f5f8c00bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:36.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:36.612 7f5fa535d700 1 -- 172.21.2.112:0/1923107155 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f8c00c300 con 0x7f5f8c009590 2021-02-10T11:20:36.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:36.612 7f5fa535d700 1 -- 172.21.2.112:0/1923107155 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f8c00c520 con 0x7f5f8c003aa0 2021-02-10T11:20:36.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:36.612 7f5fa535d700 1 -- 172.21.2.112:0/1923107155 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f8c00c740 con 0x7f5f8c006770 2021-02-10T11:20:36.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:36.612 7f5fa6b60700 1 --2- 172.21.2.112:0/1923107155 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5f8c009590 0x7f5f8c00bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:36.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:36.612 7f5fa635f700 1 --2- 172.21.2.112:0/1923107155 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5f8c003aa0 0x7f5f8c005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:36.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:36.612 7f5fa5b5e700 1 --2- 172.21.2.112:0/1923107155 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5f8c006770 0x7f5f8c008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:36.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:36.612 7f5fa6b60700 1 -- 172.21.2.112:0/1923107155 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5f8c006770 msgr2=0x7f5f8c008c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:20:36.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:36.612 7f5fa6b60700 1 --2- 172.21.2.112:0/1923107155 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5f8c006770 0x7f5f8c008c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:36.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:36.612 7f5fa6b60700 1 -- 172.21.2.112:0/1923107155 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5f8c003aa0 msgr2=0x7f5f8c005ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:20:36.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:36.612 7f5fa6b60700 1 --2- 172.21.2.112:0/1923107155 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5f8c003aa0 0x7f5f8c005ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:36.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:36.612 7f5fa6b60700 1 -- 172.21.2.112:0/1923107155 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f8c00c9e0 con 0x7f5f8c009590 2021-02-10T11:20:36.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:36.613 7f5fa6b60700 1 -- 172.21.2.112:0/1923107155 reap_dead start 2021-02-10T11:20:36.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:36.613 7f5fa6b60700 1 --2- 172.21.2.112:0/1923107155 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5f8c009590 0x7f5f8c00bab0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 rx=0x7f5f98001ae0 tx=0x7f5f9800a310).ready entity=mon.0 client_cookie=c277efeb1673f2e5 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:36.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:36.613 7f5fa535d700 1 -- 172.21.2.112:0/1923107155 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5f980070c0 con 0x7f5f8c009590 2021-02-10T11:20:36.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:36.613 7f5fa535d700 1 -- 172.21.2.112:0/1923107155 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f98007440 con 0x7f5f8c009590 2021-02-10T11:20:36.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:36.613 7f5fa535d700 1 -- 172.21.2.112:0/1923107155 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5f98004910 con 0x7f5f8c009590 2021-02-10T11:20:36.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:36.613 7f5fa8543700 1 -- 172.21.2.112:0/1923107155 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5f8c009590 msgr2=0x7f5f8c00bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:36.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:36.613 7f5fa8543700 1 --2- 172.21.2.112:0/1923107155 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5f8c009590 0x7f5f8c00bab0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 rx=0x7f5f98001ae0 tx=0x7f5f9800a310).stop 2021-02-10T11:20:36.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:36.614 7f5fa8543700 1 -- 172.21.2.112:0/1923107155 shutdown_connections 2021-02-10T11:20:36.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:36.614 7f5fa8543700 1 --2- 172.21.2.112:0/1923107155 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5f8c009590 0x7f5f8c00bab0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:36.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:36.614 7f5fa8543700 1 -- 172.21.2.112:0/1923107155 >> 172.21.2.112:0/1923107155 conn(0x7f5fa00ce3e0 msgr2=0x7f5fa00e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:20:36.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:36.614 7f5fa8543700 1 -- 172.21.2.112:0/1923107155 shutdown_connections 2021-02-10T11:20:36.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:36.614 7f5fa8543700 1 -- 172.21.2.112:0/1923107155 wait complete. 2021-02-10T11:20:36.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:36.628 7f5fa8543700 1 Processor -- start 2021-02-10T11:20:36.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:36.628 7f5fa8543700 1 -- start start 2021-02-10T11:20:36.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:36.629 7f5fa8543700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5fa009b930 0x7f5fa00b7c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:36.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:36.629 7f5fa8543700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5fa00b8410 0x7f5fa00b8810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:36.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:36.629 7f5fa8543700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5fa00c1f80 0x7f5fa00c2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:36.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:36.629 7f5fa8543700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fa00c1a00 con 0x7f5fa00c1f80 2021-02-10T11:20:36.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:36.629 7f5fa8543700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fa00b8e30 con 0x7f5fa00b8410 2021-02-10T11:20:36.629 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:36.629 7f5fa8543700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fa00b9050 con 0x7f5fa009b930 2021-02-10T11:20:36.629 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:36.629 7f5fa6b60700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5fa00c1f80 0x7f5fa00c2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:36.629 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:36.629 7f5fa6b60700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5fa00c1f80 0x7f5fa00c2380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:44730/0 (socket says 172.21.2.112:44730) 2021-02-10T11:20:36.629 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:36.629 7f5fa6b60700 1 -- 172.21.2.112:0/338637047 learned_addr learned my addr 172.21.2.112:0/338637047 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:20:36.629 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:36.629 7f5fa5b5e700 1 --2- 172.21.2.112:0/338637047 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5fa00b8410 0x7f5fa00b8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:36.629 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:36.629 7f5fa635f700 1 --2- 172.21.2.112:0/338637047 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5fa009b930 0x7f5fa00b7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:36.629 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:36.629 7f5fa6b60700 1 -- 172.21.2.112:0/338637047 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5fa009b930 msgr2=0x7f5fa00b7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:36.629 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:36.629 7f5fa6b60700 1 --2- 172.21.2.112:0/338637047 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5fa009b930 0x7f5fa00b7c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:36.629 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:36.629 7f5fa6b60700 1 -- 172.21.2.112:0/338637047 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5fa00b8410 msgr2=0x7f5fa00b8810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:36.629 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:36.629 7f5fa6b60700 1 --2- 172.21.2.112:0/338637047 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5fa00b8410 0x7f5fa00b8810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:36.629 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:36.630 7f5fa6b60700 1 -- 172.21.2.112:0/338637047 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fa00c29a0 con 0x7f5fa00c1f80 2021-02-10T11:20:36.629 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:36.630 7f5fa5b5e700 1 --2- 172.21.2.112:0/338637047 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5fa00b8410 0x7f5fa00b8810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:20:36.629 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:36.630 7f5fa6b60700 1 --2- 172.21.2.112:0/338637047 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5fa00c1f80 0x7f5fa00c2380 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 rx=0x7f5f98001c20 tx=0x7f5f9800ad20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:36.630 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:36.630 7f5f92ffd700 1 -- 172.21.2.112:0/338637047 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5f98004720 con 0x7f5fa00c1f80 2021-02-10T11:20:36.630 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:36.630 7f5f92ffd700 1 -- 172.21.2.112:0/338637047 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f98004aa0 con 0x7f5fa00c1f80 2021-02-10T11:20:36.630 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:36.630 7f5f92ffd700 1 -- 172.21.2.112:0/338637047 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5f980070d0 con 0x7f5fa00c1f80 2021-02-10T11:20:36.630 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:36.630 7f5fa8543700 1 -- 172.21.2.112:0/338637047 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fa00c29a0 con 0x7f5fa00c1f80 2021-02-10T11:20:36.630 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:36.630 7f5fa8543700 1 -- 172.21.2.112:0/338637047 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fa00c2f30 con 0x7f5fa00c1f80 2021-02-10T11:20:36.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:36.631 7f5f92ffd700 1 -- 172.21.2.112:0/338637047 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5f98004720 con 0x7f5fa00c1f80 2021-02-10T11:20:36.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:36.631 7f5f92ffd700 1 --2- 172.21.2.112:0/338637047 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5f880186c0 0x7f5f8801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:36.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:36.631 7f5f92ffd700 1 -- 172.21.2.112:0/338637047 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f5f980328a0 con 0x7f5fa00c1f80 2021-02-10T11:20:36.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:36.631 7f5fa635f700 1 --2- 172.21.2.112:0/338637047 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5f880186c0 0x7f5f8801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:36.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:36.631 7f5fa635f700 1 --2- 172.21.2.112:0/338637047 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5f880186c0 0x7f5f8801abc0 crc :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:36.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:36.661 7f5fa8543700 1 -- 172.21.2.112:0/338637047 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fa0105b30 con 0x7f5fa00c1f80 2021-02-10T11:20:36.657 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:36.663 7f5f92ffd700 1 -- 172.21.2.112:0/338637047 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5f98004d40 con 0x7f5fa00c1f80 2021-02-10T11:20:36.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:36.789 7f5fa8543700 1 -- 172.21.2.112:0/338637047 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348601", "target": ["mgr", ""]}) v1 -- 0x7f5fa00c2f30 con 0x7f5f880186c0 2021-02-10T11:20:36.808 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:36.814 7f5f92ffd700 1 -- 172.21.2.112:0/338637047 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f5f9404bd60 con 0x7f5f880186c0 2021-02-10T11:20:36.816 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:36.822 7f5fa8543700 1 -- 172.21.2.112:0/338637047 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5f880186c0 msgr2=0x7f5f8801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:36.816 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:36.822 7f5fa8543700 1 --2- 172.21.2.112:0/338637047 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5f880186c0 0x7f5f8801abc0 crc :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:36.816 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:36.822 7f5fa8543700 1 -- 172.21.2.112:0/338637047 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5fa00c1f80 msgr2=0x7f5fa00c2380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:36.816 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:36.822 7f5fa8543700 1 --2- 172.21.2.112:0/338637047 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5fa00c1f80 0x7f5fa00c2380 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 rx=0x7f5f98001c20 tx=0x7f5f9800ad20).stop 2021-02-10T11:20:36.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:36.822 7f5fa8543700 1 -- 172.21.2.112:0/338637047 shutdown_connections 2021-02-10T11:20:36.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:36.822 7f5fa8543700 1 --2- 172.21.2.112:0/338637047 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5f880186c0 0x7f5f8801abc0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:36.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:36.822 7f5fa8543700 1 --2- 172.21.2.112:0/338637047 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5fa00b8410 0x7f5fa00b8810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:36.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:36.822 7f5fa8543700 1 --2- 172.21.2.112:0/338637047 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5fa00c1f80 0x7f5fa00c2380 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:36.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:36.822 7f5fa8543700 1 --2- 172.21.2.112:0/338637047 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5fa009b930 0x7f5fa00b7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:36.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:36.822 7f5fa8543700 1 -- 172.21.2.112:0/338637047 >> 172.21.2.112:0/338637047 conn(0x7f5fa0094e10 msgr2=0x7f5fa00a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:20:36.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:36.822 7f5fa8543700 1 -- 172.21.2.112:0/338637047 shutdown_connections 2021-02-10T11:20:36.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:36.822 7f5fa8543700 1 -- 172.21.2.112:0/338637047 wait complete. 2021-02-10T11:20:36.824 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000348601 2021-02-10T11:20:37.254 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:37.242 7f0bf8334700 1 Processor -- start 2021-02-10T11:20:37.254 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:37.242 7f0bf8334700 1 -- start start 2021-02-10T11:20:37.254 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:37.243 7f0bf8334700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f0bf01b0470 0x7f0bf01b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:37.254 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:37.243 7f0bf8334700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0bf00f20e0 0x7f0bf00f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:37.254 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:37.243 7f0bf8334700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bf00f0b50 con 0x7f0bf00ef610 2021-02-10T11:20:37.255 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:37.243 7f0bf8334700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bf00f0820 con 0x7f0bf01b0470 2021-02-10T11:20:37.255 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:37.243 7f0bf8334700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bf00f24e0 con 0x7f0bf00f20e0 2021-02-10T11:20:37.255 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:37.244 7f0bf6150700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f0bf01b0470 0x7f0bf01b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:37.255 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:37.244 7f0bf6150700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f0bf01b0470 0x7f0bf01b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3300/0 says I am v2:172.21.2.112:47014/0 (socket says 172.21.2.112:47014) 2021-02-10T11:20:37.255 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:37.244 7f0bf6150700 1 -- 172.21.2.112:0/2344755890 learned_addr learned my addr 172.21.2.112:0/2344755890 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:20:37.255 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:37.244 7f0bf594f700 1 --1- 172.21.2.112:0/2344755890 >> v1:172.21.2.120:6789/0 conn(0x7f0bf00ef610 0x7f0bf00efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:37754/0 (socket says 172.21.2.112:37754) 2021-02-10T11:20:37.255 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:37.244 7f0bf6951700 1 --2- 172.21.2.112:0/2344755890 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0bf00f20e0 0x7f0bf00f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:37.255 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:37.244 7f0bf6150700 1 -- 172.21.2.112:0/2344755890 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0bf00f20e0 msgr2=0x7f0bf00f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:20:37.255 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:37.244 7f0bf6150700 1 --2- 172.21.2.112:0/2344755890 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0bf00f20e0 0x7f0bf00f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:37.255 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:37.244 7f0bf6150700 1 -- 172.21.2.112:0/2344755890 >> v1:172.21.2.120:6789/0 conn(0x7f0bf00ef610 legacy=0x7f0bf00efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:37.255 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:37.244 7f0bf6150700 1 -- 172.21.2.112:0/2344755890 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bf00e52e0 con 0x7f0bf01b0470 2021-02-10T11:20:37.255 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:37.244 7f0bf6150700 1 --2- 172.21.2.112:0/2344755890 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0bf01b0470 0x7f0bf01b0870 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f0be0006700 tx=0x7f0be0009b00).ready entity=mon.1 client_cookie=fb1e290a85bc1041 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:37.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:37.245 7f0bf514e700 1 -- 172.21.2.112:0/2344755890 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0be000c0c0 con 0x7f0bf01b0470 2021-02-10T11:20:37.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:37.245 7f0bf514e700 1 -- 172.21.2.112:0/2344755890 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0be000c340 con 0x7f0bf01b0470 2021-02-10T11:20:37.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:37.245 7f0bf514e700 1 -- 172.21.2.112:0/2344755890 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0be000c820 con 0x7f0bf01b0470 2021-02-10T11:20:37.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:37.245 7f0bf8334700 1 -- 172.21.2.112:0/2344755890 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0bf01b0470 msgr2=0x7f0bf01b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:37.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:37.245 7f0bf8334700 1 --2- 172.21.2.112:0/2344755890 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0bf01b0470 0x7f0bf01b0870 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f0be0006700 tx=0x7f0be0009b00).stop 2021-02-10T11:20:37.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:37.245 7f0bf8334700 1 -- 172.21.2.112:0/2344755890 shutdown_connections 2021-02-10T11:20:37.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:37.245 7f0bf8334700 1 --2- 172.21.2.112:0/2344755890 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0bf00f20e0 0x7f0bf00f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:37.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:37.246 7f0bf8334700 1 --2- 172.21.2.112:0/2344755890 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0bf01b0470 0x7f0bf01b0870 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:37.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:37.246 7f0bf8334700 1 -- 172.21.2.112:0/2344755890 >> 172.21.2.112:0/2344755890 conn(0x7f0bf00ce3e0 msgr2=0x7f0bf00e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:20:37.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:37.246 7f0bf8334700 1 -- 172.21.2.112:0/2344755890 shutdown_connections 2021-02-10T11:20:37.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:37.246 7f0bf8334700 1 -- 172.21.2.112:0/2344755890 wait complete. 2021-02-10T11:20:37.258 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:37.260 7f0bf8334700 1 Processor -- start 2021-02-10T11:20:37.258 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:37.260 7f0bf8334700 1 -- start start 2021-02-10T11:20:37.258 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:37.261 7f0bf8334700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0bf0105d10 0x7f0bf009eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:37.258 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:37.261 7f0bf8334700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0bf0106110 0x7f0bf009f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:37.258 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:37.261 7f0bf8334700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0bf00b2a00 0x7f0bf00b2e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:37.258 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:37.261 7f0bf8334700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bf009e560 con 0x7f0bf0105d10 2021-02-10T11:20:37.258 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:37.261 7f0bf8334700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bf009c090 con 0x7f0bf0106110 2021-02-10T11:20:37.258 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:37.261 7f0bf8334700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bf00b3420 con 0x7f0bf00b2a00 2021-02-10T11:20:37.258 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:37.261 7f0bf6150700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0bf0105d10 0x7f0bf009eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:37.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:37.261 7f0bf6150700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0bf0105d10 0x7f0bf009eae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:44744/0 (socket says 172.21.2.112:44744) 2021-02-10T11:20:37.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:37.261 7f0bf6150700 1 -- 172.21.2.112:0/3002336891 learned_addr learned my addr 172.21.2.112:0/3002336891 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:20:37.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:37.261 7f0bf6150700 1 -- 172.21.2.112:0/3002336891 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0bf00b2a00 msgr2=0x7f0bf00b2e00 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:20:37.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:37.261 7f0bf6150700 1 --2- 172.21.2.112:0/3002336891 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0bf00b2a00 0x7f0bf00b2e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:20:37.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:37.261 7f0bf6150700 1 -- 172.21.2.112:0/3002336891 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0bf0106110 msgr2=0x7f0bf009f100 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:20:37.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:37.261 7f0bf6150700 1 --2- 172.21.2.112:0/3002336891 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0bf0106110 0x7f0bf009f100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:20:37.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:37.261 7f0bf6150700 1 -- 172.21.2.112:0/3002336891 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bf00b3640 con 0x7f0bf0105d10 2021-02-10T11:20:37.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:37.261 7f0bf6150700 1 --2- 172.21.2.112:0/3002336891 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0bf0105d10 0x7f0bf009eae0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 rx=0x7f0be0006f20 tx=0x7f0be000b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:37.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:37.261 7f0be6ffd700 1 -- 172.21.2.112:0/3002336891 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0be0000c90 con 0x7f0bf0105d10 2021-02-10T11:20:37.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:37.261 7f0be6ffd700 1 -- 172.21.2.112:0/3002336891 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0be0009100 con 0x7f0bf0105d10 2021-02-10T11:20:37.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:37.261 7f0be6ffd700 1 -- 172.21.2.112:0/3002336891 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0be0002220 con 0x7f0bf0105d10 2021-02-10T11:20:37.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:37.261 7f0bf8334700 1 -- 172.21.2.112:0/3002336891 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bf00b3640 con 0x7f0bf0105d10 2021-02-10T11:20:37.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:37.262 7f0bf8334700 1 -- 172.21.2.112:0/3002336891 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bf00b3640 con 0x7f0bf0105d10 2021-02-10T11:20:37.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:37.262 7f0be6ffd700 1 -- 172.21.2.112:0/3002336891 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0be000a000 con 0x7f0bf0105d10 2021-02-10T11:20:37.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:37.262 7f0be6ffd700 1 --2- 172.21.2.112:0/3002336891 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0bd80186c0 0x7f0bd801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:37.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:37.262 7f0be6ffd700 1 -- 172.21.2.112:0/3002336891 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f0be0032540 con 0x7f0bf0105d10 2021-02-10T11:20:37.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:37.263 7f0bf594f700 1 --2- 172.21.2.112:0/3002336891 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0bd80186c0 0x7f0bd801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:37.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:37.263 7f0bf594f700 1 --2- 172.21.2.112:0/3002336891 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0bd80186c0 0x7f0bd801abc0 crc :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:37.287 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:37.292 7f0bf8334700 1 -- 172.21.2.112:0/3002336891 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bf0105ae0 con 0x7f0bf0105d10 2021-02-10T11:20:37.289 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:37.295 7f0be6ffd700 1 -- 172.21.2.112:0/3002336891 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0be0000c90 con 0x7f0bf0105d10 2021-02-10T11:20:37.417 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:37.423 7f0bf8334700 1 -- 172.21.2.112:0/3002336891 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348601", "target": ["mgr", ""]}) v1 -- 0x7f0bf01a16c0 con 0x7f0bd80186c0 2021-02-10T11:20:37.420 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:37.426 7f0be6ffd700 1 -- 172.21.2.112:0/3002336891 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f0be8000f90 con 0x7f0bd80186c0 2021-02-10T11:20:37.425 INFO:teuthology.orchestra.run.gibba012.stdout:/volumes/_nogroup/subvolume_0000000000348601/9a032cd0-3d6b-4f78-8039-adc07e5bc7e3 2021-02-10T11:20:37.427 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:37.432 7f0bf8334700 1 -- 172.21.2.112:0/3002336891 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0bd80186c0 msgr2=0x7f0bd801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:37.427 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:37.432 7f0bf8334700 1 --2- 172.21.2.112:0/3002336891 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0bd80186c0 0x7f0bd801abc0 crc :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:37.427 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:37.432 7f0bf8334700 1 -- 172.21.2.112:0/3002336891 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0bf0105d10 msgr2=0x7f0bf009eae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:37.427 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:37.432 7f0bf8334700 1 --2- 172.21.2.112:0/3002336891 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0bf0105d10 0x7f0bf009eae0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 rx=0x7f0be0006f20 tx=0x7f0be000b990).stop 2021-02-10T11:20:37.427 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:37.432 7f0bf8334700 1 -- 172.21.2.112:0/3002336891 shutdown_connections 2021-02-10T11:20:37.427 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:37.432 7f0bf8334700 1 --2- 172.21.2.112:0/3002336891 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0bd80186c0 0x7f0bd801abc0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:37.427 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:37.432 7f0bf8334700 1 --2- 172.21.2.112:0/3002336891 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0bf0106110 0x7f0bf009f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:20:37.427 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:37.432 7f0bf8334700 1 --2- 172.21.2.112:0/3002336891 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0bf00b2a00 0x7f0bf00b2e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:20:37.427 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:37.432 7f0bf8334700 1 --2- 172.21.2.112:0/3002336891 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0bf0105d10 0x7f0bf009eae0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:37.427 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:37.432 7f0bf8334700 1 -- 172.21.2.112:0/3002336891 >> 172.21.2.112:0/3002336891 conn(0x7f0bf00c36f0 msgr2=0x7f0bf00bd640 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:20:37.427 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:37.432 7f0bf8334700 1 -- 172.21.2.112:0/3002336891 shutdown_connections 2021-02-10T11:20:37.428 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:37.433 7f0bf8334700 1 -- 172.21.2.112:0/3002336891 wait complete. 2021-02-10T11:20:37.434 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000348601 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000348601/9a032cd0-3d6b-4f78-8039-adc07e5bc7e3 2021-02-10T11:20:37.435 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:20:37.435 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:20:37.435 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:20:37.771 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:20:37.772 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:20:37.774 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348601/9a032cd0-3d6b-4f78-8039-adc07e5bc7e3/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:20:38.202 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:20:38.202 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:20:38.202 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0227381 s, 46.1 MB/s 2021-02-10T11:20:38.203 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:20:38.203 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:20:38.203 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:20:38.540 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:20:38.540 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:20:38.541 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348601/9a032cd0-3d6b-4f78-8039-adc07e5bc7e3/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:20:38.915 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:20:38.915 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:20:38.915 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0241602 s, 43.4 MB/s 2021-02-10T11:20:38.916 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:20:38.916 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:20:38.916 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:20:39.253 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:20:39.254 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:20:39.255 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348601/9a032cd0-3d6b-4f78-8039-adc07e5bc7e3/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:20:39.630 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:20:39.630 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:20:39.630 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0268017 s, 39.1 MB/s 2021-02-10T11:20:39.631 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:20:39.631 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:20:39.631 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:20:39.965 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:20:39.966 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:20:39.966 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348601/9a032cd0-3d6b-4f78-8039-adc07e5bc7e3/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:20:40.341 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:20:40.341 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:20:40.342 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0247435 s, 42.4 MB/s 2021-02-10T11:20:40.343 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:20:40.343 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:20:40.343 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:20:40.678 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:20:40.678 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:20:40.678 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348601/9a032cd0-3d6b-4f78-8039-adc07e5bc7e3/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:20:41.052 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:20:41.052 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:20:41.053 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0235804 s, 44.5 MB/s 2021-02-10T11:20:41.053 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:20:41.054 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:20:41.054 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:20:41.387 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:20:41.388 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:20:41.388 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348601/9a032cd0-3d6b-4f78-8039-adc07e5bc7e3/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:20:41.762 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:20:41.762 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:20:41.762 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.024775 s, 42.3 MB/s 2021-02-10T11:20:41.763 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:20:41.763 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:20:41.763 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:20:42.097 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:20:42.097 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:20:42.098 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348601/9a032cd0-3d6b-4f78-8039-adc07e5bc7e3/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:20:42.470 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:20:42.471 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:20:42.471 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0246315 s, 42.6 MB/s 2021-02-10T11:20:42.472 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:20:42.472 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:20:42.472 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:20:42.806 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:20:42.806 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:20:42.807 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348601/9a032cd0-3d6b-4f78-8039-adc07e5bc7e3/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:20:43.180 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:20:43.180 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:20:43.180 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0225682 s, 46.5 MB/s 2021-02-10T11:20:43.181 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:20:43.181 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:20:43.181 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:20:43.515 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:20:43.516 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:20:43.516 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348601/9a032cd0-3d6b-4f78-8039-adc07e5bc7e3/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:20:43.890 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:20:43.890 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:20:43.890 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0240769 s, 43.6 MB/s 2021-02-10T11:20:43.891 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:20:43.891 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:20:43.891 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:20:44.226 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:20:44.226 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:20:44.227 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348601/9a032cd0-3d6b-4f78-8039-adc07e5bc7e3/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:20:44.599 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:20:44.600 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:20:44.600 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0257762 s, 40.7 MB/s 2021-02-10T11:20:44.601 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000348602 2021-02-10T11:20:45.031 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.019 7f969688e700 1 Processor -- start 2021-02-10T11:20:45.031 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.019 7f969688e700 1 -- start start 2021-02-10T11:20:45.031 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.020 7f969688e700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f96901b0870 0x7f96900f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:45.031 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.020 7f969688e700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f96900efe00 0x7f96900f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:45.031 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.020 7f969688e700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96901b0c70 con 0x7f96901b0470 2021-02-10T11:20:45.031 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.020 7f969688e700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96900f0b50 con 0x7f96900efe00 2021-02-10T11:20:45.031 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.020 7f969688e700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96900f0820 con 0x7f96901b0870 2021-02-10T11:20:45.032 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.021 7f9694eab700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f96900efe00 0x7f96900f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:45.032 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.021 7f968ffff700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7f96901b0470 0x7f96900ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:37768/0 (socket says 172.21.2.112:37768) 2021-02-10T11:20:45.032 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.021 7f968f7fe700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f96901b0870 0x7f96900f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:45.032 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.021 7f968ffff700 1 -- 172.21.2.112:0/259374231 learned_addr learned my addr 172.21.2.112:0/259374231 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:20:45.032 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.021 7f968f7fe700 1 -- 172.21.2.112:0/259374231 >> v2:172.21.2.120:3300/0 conn(0x7f96900efe00 msgr2=0x7f96900f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:20:45.032 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.021 7f968f7fe700 1 --2- 172.21.2.112:0/259374231 >> v2:172.21.2.120:3300/0 conn(0x7f96900efe00 0x7f96900f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:45.032 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.021 7f968f7fe700 1 -- 172.21.2.112:0/259374231 >> v1:172.21.2.120:6789/0 conn(0x7f96901b0470 legacy=0x7f96900ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:45.032 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.021 7f968f7fe700 1 -- 172.21.2.112:0/259374231 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96900e52e0 con 0x7f96901b0870 2021-02-10T11:20:45.032 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.021 7f9694eab700 1 --2- 172.21.2.112:0/259374231 >> v2:172.21.2.120:3300/0 conn(0x7f96900efe00 0x7f96900f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:20:45.032 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.021 7f968f7fe700 1 --2- 172.21.2.112:0/259374231 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f96901b0870 0x7f96900f20e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f9684006650 tx=0x7f9684009a80).ready entity=mon.2 client_cookie=82da9971afb7c113 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:45.032 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.022 7f968effd700 1 -- 172.21.2.112:0/259374231 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f96840009b0 con 0x7f96901b0870 2021-02-10T11:20:45.032 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.022 7f968effd700 1 -- 172.21.2.112:0/259374231 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f968400c0c0 con 0x7f96901b0870 2021-02-10T11:20:45.032 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.022 7f968effd700 1 -- 172.21.2.112:0/259374231 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f968400c5e0 con 0x7f96901b0870 2021-02-10T11:20:45.033 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.022 7f969688e700 1 -- 172.21.2.112:0/259374231 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f96901b0870 msgr2=0x7f96900f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:45.033 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.022 7f969688e700 1 --2- 172.21.2.112:0/259374231 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f96901b0870 0x7f96900f20e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f9684006650 tx=0x7f9684009a80).stop 2021-02-10T11:20:45.033 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.022 7f969688e700 1 -- 172.21.2.112:0/259374231 shutdown_connections 2021-02-10T11:20:45.033 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.022 7f969688e700 1 --2- 172.21.2.112:0/259374231 >> v2:172.21.2.120:3300/0 conn(0x7f96900efe00 0x7f96900f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:45.033 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.022 7f969688e700 1 --2- 172.21.2.112:0/259374231 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f96901b0870 0x7f96900f20e0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:45.033 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.022 7f969688e700 1 -- 172.21.2.112:0/259374231 >> 172.21.2.112:0/259374231 conn(0x7f96900ce3e0 msgr2=0x7f96900e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:20:45.033 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.022 7f969688e700 1 -- 172.21.2.112:0/259374231 shutdown_connections 2021-02-10T11:20:45.033 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.022 7f969688e700 1 -- 172.21.2.112:0/259374231 wait complete. 2021-02-10T11:20:45.035 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.037 7f969688e700 1 Processor -- start 2021-02-10T11:20:45.035 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.037 7f969688e700 1 -- start start 2021-02-10T11:20:45.035 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.038 7f969688e700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f969010a1e0 0x7f96900ab870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:45.035 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.038 7f969688e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f96900abe90 0x7f96900ac290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:45.035 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.038 7f969688e700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f96900ad200 0x7f96900ad600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:45.035 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.038 7f969688e700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f969010a710 con 0x7f96900abe90 2021-02-10T11:20:45.035 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.038 7f969688e700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96900ac8b0 con 0x7f96900ad200 2021-02-10T11:20:45.035 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.038 7f969688e700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96900adc20 con 0x7f969010a1e0 2021-02-10T11:20:45.035 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.038 7f968f7fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f96900abe90 0x7f96900ac290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:45.035 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.038 7f968f7fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f96900abe90 0x7f96900ac290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:44758/0 (socket says 172.21.2.112:44758) 2021-02-10T11:20:45.035 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.038 7f968f7fe700 1 -- 172.21.2.112:0/3496615885 learned_addr learned my addr 172.21.2.112:0/3496615885 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:20:45.036 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.038 7f968ffff700 1 --2- 172.21.2.112:0/3496615885 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f969010a1e0 0x7f96900ab870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:45.036 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.038 7f9694eab700 1 --2- 172.21.2.112:0/3496615885 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f96900ad200 0x7f96900ad600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:45.036 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.038 7f968f7fe700 1 -- 172.21.2.112:0/3496615885 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f969010a1e0 msgr2=0x7f96900ab870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:45.036 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.038 7f968f7fe700 1 --2- 172.21.2.112:0/3496615885 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f969010a1e0 0x7f96900ab870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:45.036 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.038 7f968f7fe700 1 -- 172.21.2.112:0/3496615885 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f96900ad200 msgr2=0x7f96900ad600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:45.036 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.038 7f968f7fe700 1 --2- 172.21.2.112:0/3496615885 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f96900ad200 0x7f96900ad600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:45.036 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.038 7f968f7fe700 1 -- 172.21.2.112:0/3496615885 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96900ade40 con 0x7f96900abe90 2021-02-10T11:20:45.036 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.038 7f968ffff700 1 --2- 172.21.2.112:0/3496615885 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f969010a1e0 0x7f96900ab870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:20:45.036 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.038 7f968f7fe700 1 --2- 172.21.2.112:0/3496615885 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f96900abe90 0x7f96900ac290 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 rx=0x7f9684009630 tx=0x7f968400b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:45.036 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.039 7f968cff9700 1 -- 172.21.2.112:0/3496615885 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f968400bd70 con 0x7f96900abe90 2021-02-10T11:20:45.036 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.039 7f968cff9700 1 -- 172.21.2.112:0/3496615885 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9684000960 con 0x7f96900abe90 2021-02-10T11:20:45.036 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.039 7f968cff9700 1 -- 172.21.2.112:0/3496615885 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9684005560 con 0x7f96900abe90 2021-02-10T11:20:45.036 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.039 7f969688e700 1 -- 172.21.2.112:0/3496615885 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96900ade40 con 0x7f96900abe90 2021-02-10T11:20:45.036 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.039 7f969688e700 1 -- 172.21.2.112:0/3496615885 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96900ade40 con 0x7f96900abe90 2021-02-10T11:20:45.037 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.039 7f968cff9700 1 -- 172.21.2.112:0/3496615885 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f968400a000 con 0x7f96900abe90 2021-02-10T11:20:45.037 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.039 7f968cff9700 1 --2- 172.21.2.112:0/3496615885 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f96700186d0 0x7f967001abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:45.037 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.039 7f968cff9700 1 -- 172.21.2.112:0/3496615885 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f96840325f0 con 0x7f96900abe90 2021-02-10T11:20:45.037 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.039 7f968ffff700 1 --2- 172.21.2.112:0/3496615885 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f96700186d0 0x7f967001abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:45.037 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.040 7f968ffff700 1 --2- 172.21.2.112:0/3496615885 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f96700186d0 0x7f967001abd0 crc :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:45.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.070 7f969688e700 1 -- 172.21.2.112:0/3496615885 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9690105ae0 con 0x7f96900abe90 2021-02-10T11:20:45.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.072 7f968cff9700 1 -- 172.21.2.112:0/3496615885 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9684000940 con 0x7f96900abe90 2021-02-10T11:20:45.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.198 7f969688e700 1 -- 172.21.2.112:0/3496615885 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348602", "target": ["mgr", ""]}) v1 -- 0x7f96900ac8b0 con 0x7f96700186d0 2021-02-10T11:20:45.218 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.224 7f968cff9700 1 -- 172.21.2.112:0/3496615885 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f9680000e70 con 0x7f96700186d0 2021-02-10T11:20:45.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.231 7f969688e700 1 -- 172.21.2.112:0/3496615885 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f96700186d0 msgr2=0x7f967001abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:45.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.231 7f969688e700 1 --2- 172.21.2.112:0/3496615885 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f96700186d0 0x7f967001abd0 crc :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:45.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.231 7f969688e700 1 -- 172.21.2.112:0/3496615885 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f96900abe90 msgr2=0x7f96900ac290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:45.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.231 7f969688e700 1 --2- 172.21.2.112:0/3496615885 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f96900abe90 0x7f96900ac290 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 rx=0x7f9684009630 tx=0x7f968400b980).stop 2021-02-10T11:20:45.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.231 7f969688e700 1 -- 172.21.2.112:0/3496615885 shutdown_connections 2021-02-10T11:20:45.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.231 7f969688e700 1 --2- 172.21.2.112:0/3496615885 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f96700186d0 0x7f967001abd0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:45.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.231 7f969688e700 1 --2- 172.21.2.112:0/3496615885 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f96900ad200 0x7f96900ad600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:45.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.231 7f969688e700 1 --2- 172.21.2.112:0/3496615885 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f96900abe90 0x7f96900ac290 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:45.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.231 7f969688e700 1 --2- 172.21.2.112:0/3496615885 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f969010a1e0 0x7f96900ab870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:45.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.231 7f969688e700 1 -- 172.21.2.112:0/3496615885 >> 172.21.2.112:0/3496615885 conn(0x7f96900bbf60 msgr2=0x7f9690105d10 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:20:45.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.231 7f969688e700 1 -- 172.21.2.112:0/3496615885 shutdown_connections 2021-02-10T11:20:45.227 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.232 7f969688e700 1 -- 172.21.2.112:0/3496615885 wait complete. 2021-02-10T11:20:45.234 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000348602 2021-02-10T11:20:45.666 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.654 7f4e52830700 1 Processor -- start 2021-02-10T11:20:45.666 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.654 7f4e52830700 1 -- start start 2021-02-10T11:20:45.666 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.655 7f4e52830700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f4e4c0f05f0 0x7f4e4c0f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:45.666 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.655 7f4e52830700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e4c1b0c70 con 0x7f4e4c1b0870 2021-02-10T11:20:45.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.655 7f4e52830700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e4c0f0b50 con 0x7f4e4c1b0470 2021-02-10T11:20:45.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.655 7f4e52830700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e4c0e5ea0 con 0x7f4e4c0f05f0 2021-02-10T11:20:45.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.655 7f4e4b7fe700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f4e4c1b0870 0x7f4e4c0efe00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:44224/0 (socket says 172.21.2.112:44224) 2021-02-10T11:20:45.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.655 7f4e4b7fe700 1 -- 172.21.2.112:0/2437769748 learned_addr learned my addr 172.21.2.112:0/2437769748 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:20:45.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.655 7f4e4affd700 1 -- 172.21.2.112:0/2437769748 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 721918318 0 0) 0x7f4e34000ce0 con 0x7f4e4c1b0870 2021-02-10T11:20:45.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.656 7f4e4affd700 1 -- 172.21.2.112:0/2437769748 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e38002fe0 con 0x7f4e4c1b0870 2021-02-10T11:20:45.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.656 7f4e50e4d700 1 --2- 172.21.2.112:0/2437769748 >> v2:172.21.2.120:3300/0 conn(0x7f4e4c0f05f0 0x7f4e4c0f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:45.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.656 7f4e4affd700 1 -- 172.21.2.112:0/2437769748 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2302002497 0 0) 0x7f4e340010e0 con 0x7f4e4c1b0870 2021-02-10T11:20:45.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.656 7f4e4affd700 1 -- 172.21.2.112:0/2437769748 >> v2:172.21.2.120:3300/0 conn(0x7f4e4c0f05f0 msgr2=0x7f4e4c0f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:20:45.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.656 7f4e4affd700 1 --2- 172.21.2.112:0/2437769748 >> v2:172.21.2.120:3300/0 conn(0x7f4e4c0f05f0 0x7f4e4c0f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:45.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.656 7f4e4affd700 1 -- 172.21.2.112:0/2437769748 >> v1:172.21.2.120:6789/0 conn(0x7f4e4c1b0470 legacy=0x7f4e4c0ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:45.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.656 7f4e4affd700 1 -- 172.21.2.112:0/2437769748 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e4c0e52e0 con 0x7f4e4c1b0870 2021-02-10T11:20:45.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.656 7f4e4affd700 1 -- 172.21.2.112:0/2437769748 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f4e34001600 con 0x7f4e4c1b0870 2021-02-10T11:20:45.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.656 7f4e4affd700 1 -- 172.21.2.112:0/2437769748 >> v1:172.21.2.112:6789/0 conn(0x7f4e4c1b0870 legacy=0x7f4e4c0efe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:45.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.656 7f4e4affd700 1 --2- 172.21.2.112:0/2437769748 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4e38003a70 0x7f4e38005f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:45.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.656 7f4e4affd700 1 --2- 172.21.2.112:0/2437769748 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4e38006790 0x7f4e38008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:45.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.656 7f4e4affd700 1 --2- 172.21.2.112:0/2437769748 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4e38009590 0x7f4e3800bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:45.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.656 7f4e4affd700 1 -- 172.21.2.112:0/2437769748 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e3800c300 con 0x7f4e38009590 2021-02-10T11:20:45.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.656 7f4e4affd700 1 -- 172.21.2.112:0/2437769748 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e3800c520 con 0x7f4e38006790 2021-02-10T11:20:45.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.656 7f4e4affd700 1 -- 172.21.2.112:0/2437769748 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e3800c740 con 0x7f4e38003a70 2021-02-10T11:20:45.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.656 7f4e50e4d700 1 --2- 172.21.2.112:0/2437769748 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4e38009590 0x7f4e3800bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:45.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.656 7f4e4bfff700 1 --2- 172.21.2.112:0/2437769748 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4e38003a70 0x7f4e38005f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:45.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.656 7f4e4b7fe700 1 --2- 172.21.2.112:0/2437769748 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4e38006790 0x7f4e38008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:45.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.657 7f4e50e4d700 1 -- 172.21.2.112:0/2437769748 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4e38003a70 msgr2=0x7f4e38005f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:20:45.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.657 7f4e50e4d700 1 --2- 172.21.2.112:0/2437769748 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4e38003a70 0x7f4e38005f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:45.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.657 7f4e50e4d700 1 -- 172.21.2.112:0/2437769748 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4e38006790 msgr2=0x7f4e38008c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:20:45.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.657 7f4e50e4d700 1 --2- 172.21.2.112:0/2437769748 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4e38006790 0x7f4e38008c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:45.669 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.657 7f4e50e4d700 1 -- 172.21.2.112:0/2437769748 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e380509b0 con 0x7f4e38009590 2021-02-10T11:20:45.669 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.657 7f4e50e4d700 1 -- 172.21.2.112:0/2437769748 reap_dead start 2021-02-10T11:20:45.669 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.657 7f4e50e4d700 1 --2- 172.21.2.112:0/2437769748 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4e38009590 0x7f4e3800bab0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 rx=0x7f4e4400c870 tx=0x7f4e4400cc10).ready entity=mon.0 client_cookie=271982705b1fafc7 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:45.669 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.657 7f4e4affd700 1 -- 172.21.2.112:0/2437769748 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4e440067d0 con 0x7f4e38009590 2021-02-10T11:20:45.669 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.657 7f4e4affd700 1 -- 172.21.2.112:0/2437769748 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e440090d0 con 0x7f4e38009590 2021-02-10T11:20:45.669 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.657 7f4e4affd700 1 -- 172.21.2.112:0/2437769748 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4e44009520 con 0x7f4e38009590 2021-02-10T11:20:45.669 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.657 7f4e52830700 1 -- 172.21.2.112:0/2437769748 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4e38009590 msgr2=0x7f4e3800bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:45.669 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.657 7f4e52830700 1 --2- 172.21.2.112:0/2437769748 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4e38009590 0x7f4e3800bab0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 rx=0x7f4e4400c870 tx=0x7f4e4400cc10).stop 2021-02-10T11:20:45.669 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.658 7f4e52830700 1 -- 172.21.2.112:0/2437769748 shutdown_connections 2021-02-10T11:20:45.669 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.658 7f4e52830700 1 --2- 172.21.2.112:0/2437769748 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4e38009590 0x7f4e3800bab0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:45.670 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.658 7f4e52830700 1 -- 172.21.2.112:0/2437769748 >> 172.21.2.112:0/2437769748 conn(0x7f4e4c0ce3e0 msgr2=0x7f4e4c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:20:45.670 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.658 7f4e52830700 1 -- 172.21.2.112:0/2437769748 shutdown_connections 2021-02-10T11:20:45.670 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.658 7f4e52830700 1 -- 172.21.2.112:0/2437769748 wait complete. 2021-02-10T11:20:45.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.672 7f4e52830700 1 Processor -- start 2021-02-10T11:20:45.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.672 7f4e52830700 1 -- start start 2021-02-10T11:20:45.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.673 7f4e52830700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4e4c09b930 0x7f4e4c0b7c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:45.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.673 7f4e52830700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4e4c0b8410 0x7f4e4c0b8810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:45.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.673 7f4e52830700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4e4c0c1f80 0x7f4e4c0c2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:45.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.673 7f4e52830700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e4c0c1a00 con 0x7f4e4c0c1f80 2021-02-10T11:20:45.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.673 7f4e52830700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e4c0b8e30 con 0x7f4e4c0b8410 2021-02-10T11:20:45.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.673 7f4e52830700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e4c0b9050 con 0x7f4e4c09b930 2021-02-10T11:20:45.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.673 7f4e50e4d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4e4c0c1f80 0x7f4e4c0c2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:45.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.673 7f4e50e4d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4e4c0c1f80 0x7f4e4c0c2380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:44770/0 (socket says 172.21.2.112:44770) 2021-02-10T11:20:45.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.673 7f4e50e4d700 1 -- 172.21.2.112:0/2832083701 learned_addr learned my addr 172.21.2.112:0/2832083701 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:20:45.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.673 7f4e50e4d700 1 -- 172.21.2.112:0/2832083701 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4e4c09b930 msgr2=0x7f4e4c0b7c20 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:20:45.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.673 7f4e50e4d700 1 --2- 172.21.2.112:0/2832083701 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4e4c09b930 0x7f4e4c0b7c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:20:45.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.673 7f4e50e4d700 1 -- 172.21.2.112:0/2832083701 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4e4c0b8410 msgr2=0x7f4e4c0b8810 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:20:45.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.673 7f4e50e4d700 1 --2- 172.21.2.112:0/2832083701 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4e4c0b8410 0x7f4e4c0b8810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:20:45.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.673 7f4e50e4d700 1 -- 172.21.2.112:0/2832083701 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e4c0c29a0 con 0x7f4e4c0c1f80 2021-02-10T11:20:45.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.673 7f4e50e4d700 1 --2- 172.21.2.112:0/2832083701 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4e4c0c1f80 0x7f4e4c0c2380 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 rx=0x7f4e4400b650 tx=0x7f4e4400b9f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:45.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.673 7f4e48ff9700 1 -- 172.21.2.112:0/2832083701 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4e4400bde0 con 0x7f4e4c0c1f80 2021-02-10T11:20:45.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.673 7f4e48ff9700 1 -- 172.21.2.112:0/2832083701 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e44000e20 con 0x7f4e4c0c1f80 2021-02-10T11:20:45.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.673 7f4e48ff9700 1 -- 172.21.2.112:0/2832083701 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4e44002220 con 0x7f4e4c0c1f80 2021-02-10T11:20:45.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.673 7f4e52830700 1 -- 172.21.2.112:0/2832083701 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e4c0c29a0 con 0x7f4e4c0c1f80 2021-02-10T11:20:45.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.674 7f4e52830700 1 -- 172.21.2.112:0/2832083701 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e4c0c29a0 con 0x7f4e4c0c1f80 2021-02-10T11:20:45.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.674 7f4e48ff9700 1 -- 172.21.2.112:0/2832083701 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4e44000a70 con 0x7f4e4c0c1f80 2021-02-10T11:20:45.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.674 7f4e48ff9700 1 --2- 172.21.2.112:0/2832083701 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4e30018730 0x7f4e3001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:45.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.674 7f4e48ff9700 1 -- 172.21.2.112:0/2832083701 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f4e44032630 con 0x7f4e4c0c1f80 2021-02-10T11:20:45.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.675 7f4e4bfff700 1 --2- 172.21.2.112:0/2832083701 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4e30018730 0x7f4e3001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:45.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.675 7f4e4bfff700 1 --2- 172.21.2.112:0/2832083701 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4e30018730 0x7f4e3001ac10 crc :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:45.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.705 7f4e52830700 1 -- 172.21.2.112:0/2832083701 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e4c105b30 con 0x7f4e4c0c1f80 2021-02-10T11:20:45.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.708 7f4e48ff9700 1 -- 172.21.2.112:0/2832083701 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4e44000e20 con 0x7f4e4c0c1f80 2021-02-10T11:20:45.833 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.839 7f4e52830700 1 -- 172.21.2.112:0/2832083701 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348602", "target": ["mgr", ""]}) v1 -- 0x7f4e4c105b30 con 0x7f4e30018730 2021-02-10T11:20:45.836 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.842 7f4e48ff9700 1 -- 172.21.2.112:0/2832083701 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f4e3c006b30 con 0x7f4e30018730 2021-02-10T11:20:45.836 INFO:teuthology.orchestra.run.gibba012.stdout:/volumes/_nogroup/subvolume_0000000000348602/a51e264b-07b6-45fe-a758-9fffd6111ca9 2021-02-10T11:20:45.839 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.844 7f4e52830700 1 -- 172.21.2.112:0/2832083701 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4e30018730 msgr2=0x7f4e3001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:45.839 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.844 7f4e52830700 1 --2- 172.21.2.112:0/2832083701 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4e30018730 0x7f4e3001ac10 crc :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:45.839 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.844 7f4e52830700 1 -- 172.21.2.112:0/2832083701 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4e4c0c1f80 msgr2=0x7f4e4c0c2380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:45.839 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.844 7f4e52830700 1 --2- 172.21.2.112:0/2832083701 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4e4c0c1f80 0x7f4e4c0c2380 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 rx=0x7f4e4400b650 tx=0x7f4e4400b9f0).stop 2021-02-10T11:20:45.839 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.844 7f4e52830700 1 -- 172.21.2.112:0/2832083701 shutdown_connections 2021-02-10T11:20:45.839 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.844 7f4e52830700 1 --2- 172.21.2.112:0/2832083701 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4e30018730 0x7f4e3001ac10 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:45.839 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.844 7f4e52830700 1 --2- 172.21.2.112:0/2832083701 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4e4c0b8410 0x7f4e4c0b8810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:20:45.839 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.844 7f4e52830700 1 --2- 172.21.2.112:0/2832083701 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4e4c0c1f80 0x7f4e4c0c2380 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:45.839 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.844 7f4e52830700 1 --2- 172.21.2.112:0/2832083701 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4e4c09b930 0x7f4e4c0b7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:20:45.839 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.844 7f4e52830700 1 -- 172.21.2.112:0/2832083701 >> 172.21.2.112:0/2832083701 conn(0x7f4e4c094e10 msgr2=0x7f4e4c0a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:20:45.839 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.844 7f4e52830700 1 -- 172.21.2.112:0/2832083701 shutdown_connections 2021-02-10T11:20:45.839 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:45.844 7f4e52830700 1 -- 172.21.2.112:0/2832083701 wait complete. 2021-02-10T11:20:45.847 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000348602 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000348602/a51e264b-07b6-45fe-a758-9fffd6111ca9 2021-02-10T11:20:45.847 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:20:45.848 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:20:45.848 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:20:46.183 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:20:46.184 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:20:46.184 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348602/a51e264b-07b6-45fe-a758-9fffd6111ca9/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:20:46.568 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:20:46.568 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:20:46.568 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0246942 s, 42.5 MB/s 2021-02-10T11:20:46.568 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:20:46.568 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:20:46.569 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:20:46.902 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:20:46.903 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:20:46.903 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348602/a51e264b-07b6-45fe-a758-9fffd6111ca9/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:20:47.277 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:20:47.278 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:20:47.278 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0247533 s, 42.4 MB/s 2021-02-10T11:20:47.279 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:20:47.279 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:20:47.279 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:20:47.614 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:20:47.615 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:20:47.615 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348602/a51e264b-07b6-45fe-a758-9fffd6111ca9/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:20:47.991 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:20:47.991 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:20:47.991 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0242427 s, 43.3 MB/s 2021-02-10T11:20:47.992 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:20:47.992 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:20:47.992 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:20:48.328 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:20:48.328 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:20:48.328 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348602/a51e264b-07b6-45fe-a758-9fffd6111ca9/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:20:48.702 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:20:48.702 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:20:48.703 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0228744 s, 45.8 MB/s 2021-02-10T11:20:48.703 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:20:48.704 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:20:48.704 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:20:49.039 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:20:49.039 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:20:49.039 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348602/a51e264b-07b6-45fe-a758-9fffd6111ca9/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:20:49.412 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:20:49.412 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:20:49.412 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0248466 s, 42.2 MB/s 2021-02-10T11:20:49.413 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:20:49.413 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:20:49.413 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:20:49.748 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:20:49.748 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:20:49.748 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348602/a51e264b-07b6-45fe-a758-9fffd6111ca9/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:20:50.121 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:20:50.121 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:20:50.122 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.022774 s, 46.0 MB/s 2021-02-10T11:20:50.123 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:20:50.123 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:20:50.123 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:20:50.457 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:20:50.457 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:20:50.458 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348602/a51e264b-07b6-45fe-a758-9fffd6111ca9/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:20:50.832 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:20:50.832 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:20:50.832 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.024571 s, 42.7 MB/s 2021-02-10T11:20:50.833 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:20:50.833 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:20:50.833 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:20:51.169 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:20:51.169 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:20:51.170 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348602/a51e264b-07b6-45fe-a758-9fffd6111ca9/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:20:51.544 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:20:51.544 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:20:51.544 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0225757 s, 46.4 MB/s 2021-02-10T11:20:51.545 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:20:51.545 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:20:51.545 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:20:51.880 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:20:51.880 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:20:51.881 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348602/a51e264b-07b6-45fe-a758-9fffd6111ca9/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:20:52.258 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:20:52.258 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:20:52.258 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0248356 s, 42.2 MB/s 2021-02-10T11:20:52.260 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:20:52.260 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:20:52.260 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:20:52.596 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:20:52.596 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:20:52.596 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348602/a51e264b-07b6-45fe-a758-9fffd6111ca9/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:20:52.973 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:20:52.974 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:20:52.974 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0261767 s, 40.1 MB/s 2021-02-10T11:20:52.975 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000348603 2021-02-10T11:20:53.408 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:53.395 7f45a44f1700 1 Processor -- start 2021-02-10T11:20:53.408 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:53.396 7f45a44f1700 1 -- start start 2021-02-10T11:20:53.408 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:53.396 7f45a44f1700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f459c1b0470 0x7f459c1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:53.409 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:53.396 7f45a44f1700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f459c0ef610 0x7f459c0efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:53.409 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:53.396 7f45a44f1700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f459c0f0b50 con 0x7f459c0f0030 2021-02-10T11:20:53.409 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:53.396 7f45a44f1700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f459c0e52e0 con 0x7f459c0ef610 2021-02-10T11:20:53.409 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:53.396 7f45a44f1700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f459c0e5490 con 0x7f459c1b0470 2021-02-10T11:20:53.409 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:53.397 7f45a2b0e700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7f459c0f0030 0x7f459c0f10d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:37798/0 (socket says 172.21.2.112:37798) 2021-02-10T11:20:53.409 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:53.397 7f45a1b0c700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f459c0ef610 0x7f459c0efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:53.409 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:53.397 7f45a230d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f459c1b0470 0x7f459c1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:53.409 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:53.397 7f45a2b0e700 1 -- 172.21.2.112:0/3619391482 learned_addr learned my addr 172.21.2.112:0/3619391482 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:20:53.409 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:53.398 7f45a1b0c700 1 -- 172.21.2.112:0/3619391482 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f459c1b0470 msgr2=0x7f459c1b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:20:53.409 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:53.398 7f45a1b0c700 1 --2- 172.21.2.112:0/3619391482 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f459c1b0470 0x7f459c1b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:53.409 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:53.398 7f45a1b0c700 1 -- 172.21.2.112:0/3619391482 >> v1:172.21.2.120:6789/0 conn(0x7f459c0f0030 legacy=0x7f459c0f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:53.409 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:53.398 7f45a1b0c700 1 -- 172.21.2.112:0/3619391482 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f459c0f0430 con 0x7f459c0ef610 2021-02-10T11:20:53.410 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:53.398 7f45a230d700 1 --2- 172.21.2.112:0/3619391482 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f459c1b0470 0x7f459c1b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:20:53.410 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:53.398 7f45a1b0c700 1 --2- 172.21.2.112:0/3619391482 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f459c0ef610 0x7f459c0efa10 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f4598006650 tx=0x7f4598009a80).ready entity=mon.1 client_cookie=b12b557263430d85 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:53.410 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:53.399 7f45a130b700 1 -- 172.21.2.112:0/3619391482 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f45980009b0 con 0x7f459c0ef610 2021-02-10T11:20:53.410 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:53.399 7f45a130b700 1 -- 172.21.2.112:0/3619391482 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f459800c0c0 con 0x7f459c0ef610 2021-02-10T11:20:53.410 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:53.399 7f45a130b700 1 -- 172.21.2.112:0/3619391482 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f459800c5e0 con 0x7f459c0ef610 2021-02-10T11:20:53.410 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:53.399 7f45a44f1700 1 -- 172.21.2.112:0/3619391482 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f459c0ef610 msgr2=0x7f459c0efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:53.410 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:53.399 7f45a44f1700 1 --2- 172.21.2.112:0/3619391482 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f459c0ef610 0x7f459c0efa10 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f4598006650 tx=0x7f4598009a80).stop 2021-02-10T11:20:53.410 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:53.400 7f45a44f1700 1 -- 172.21.2.112:0/3619391482 shutdown_connections 2021-02-10T11:20:53.410 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:53.400 7f45a44f1700 1 --2- 172.21.2.112:0/3619391482 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f459c0ef610 0x7f459c0efa10 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:53.410 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:53.400 7f45a44f1700 1 --2- 172.21.2.112:0/3619391482 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f459c1b0470 0x7f459c1b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:53.410 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:53.400 7f45a44f1700 1 -- 172.21.2.112:0/3619391482 >> 172.21.2.112:0/3619391482 conn(0x7f459c0ce3e0 msgr2=0x7f459c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:20:53.410 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:53.400 7f45a44f1700 1 -- 172.21.2.112:0/3619391482 shutdown_connections 2021-02-10T11:20:53.410 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:53.400 7f45a44f1700 1 -- 172.21.2.112:0/3619391482 wait complete. 2021-02-10T11:20:53.412 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:53.414 7f45a44f1700 1 Processor -- start 2021-02-10T11:20:53.412 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:53.414 7f45a44f1700 1 -- start start 2021-02-10T11:20:53.412 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:53.415 7f45a44f1700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f459c08ece0 0x7f459c0a6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:53.412 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:53.415 7f45a44f1700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f459c0a6860 0x7f459c0a6c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:53.412 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:53.415 7f45a44f1700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f459c0a7280 0x7f459c0a4590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:53.412 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:53.415 7f45a44f1700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f459c0a5cc0 con 0x7f459c0a7280 2021-02-10T11:20:53.412 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:53.415 7f45a44f1700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f459c08f0e0 con 0x7f459c0a6860 2021-02-10T11:20:53.412 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:53.415 7f45a44f1700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f459c08f2c0 con 0x7f459c08ece0 2021-02-10T11:20:53.413 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:53.415 7f45a2b0e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f459c0a7280 0x7f459c0a4590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:53.413 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:53.415 7f45a2b0e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f459c0a7280 0x7f459c0a4590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:44784/0 (socket says 172.21.2.112:44784) 2021-02-10T11:20:53.413 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:53.415 7f45a2b0e700 1 -- 172.21.2.112:0/3521342413 learned_addr learned my addr 172.21.2.112:0/3521342413 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:20:53.413 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:53.415 7f45a2b0e700 1 -- 172.21.2.112:0/3521342413 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f459c08ece0 msgr2=0x7f459c0a6240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:53.413 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:53.415 7f45a2b0e700 1 --2- 172.21.2.112:0/3521342413 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f459c08ece0 0x7f459c0a6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:20:53.413 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:53.415 7f45a2b0e700 1 -- 172.21.2.112:0/3521342413 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f459c0a6860 msgr2=0x7f459c0a6c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:53.413 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:53.415 7f45a2b0e700 1 --2- 172.21.2.112:0/3521342413 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f459c0a6860 0x7f459c0a6c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:20:53.413 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:53.415 7f45a2b0e700 1 -- 172.21.2.112:0/3521342413 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f459c0a4bb0 con 0x7f459c0a7280 2021-02-10T11:20:53.413 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:53.415 7f45a2b0e700 1 --2- 172.21.2.112:0/3521342413 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f459c0a7280 0x7f459c0a4590 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 rx=0x7f4594007f90 tx=0x7f459400ac70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:53.413 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:53.416 7f458effd700 1 -- 172.21.2.112:0/3521342413 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f45940070d0 con 0x7f459c0a7280 2021-02-10T11:20:53.413 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:53.416 7f458effd700 1 -- 172.21.2.112:0/3521342413 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4594007390 con 0x7f459c0a7280 2021-02-10T11:20:53.413 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:53.416 7f458effd700 1 -- 172.21.2.112:0/3521342413 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f45940048b0 con 0x7f459c0a7280 2021-02-10T11:20:53.413 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:53.416 7f45a44f1700 1 -- 172.21.2.112:0/3521342413 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f459c0a4bb0 con 0x7f459c0a7280 2021-02-10T11:20:53.413 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:53.416 7f45a44f1700 1 -- 172.21.2.112:0/3521342413 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f459c0a4bb0 con 0x7f459c0a7280 2021-02-10T11:20:53.414 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:53.416 7f458effd700 1 -- 172.21.2.112:0/3521342413 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4594007570 con 0x7f459c0a7280 2021-02-10T11:20:53.414 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:53.416 7f458effd700 1 --2- 172.21.2.112:0/3521342413 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f45840186c0 0x7f458401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:53.414 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:53.416 7f458effd700 1 -- 172.21.2.112:0/3521342413 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f45940328d0 con 0x7f459c0a7280 2021-02-10T11:20:53.414 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:53.417 7f45a230d700 1 --2- 172.21.2.112:0/3521342413 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f45840186c0 0x7f458401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:53.414 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:53.418 7f45a230d700 1 --2- 172.21.2.112:0/3521342413 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f45840186c0 0x7f458401abc0 crc :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:53.440 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:53.446 7f45a44f1700 1 -- 172.21.2.112:0/3521342413 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f459c105ae0 con 0x7f459c0a7280 2021-02-10T11:20:53.442 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:53.448 7f458effd700 1 -- 172.21.2.112:0/3521342413 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f45940012b0 con 0x7f459c0a7280 2021-02-10T11:20:53.567 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:53.573 7f45a44f1700 1 -- 172.21.2.112:0/3521342413 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348603", "target": ["mgr", ""]}) v1 -- 0x7f459c0f0b50 con 0x7f45840186c0 2021-02-10T11:20:53.595 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:53.601 7f458effd700 1 -- 172.21.2.112:0/3521342413 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f4590006420 con 0x7f45840186c0 2021-02-10T11:20:53.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:53.607 7f45a44f1700 1 -- 172.21.2.112:0/3521342413 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f45840186c0 msgr2=0x7f458401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:53.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:53.607 7f45a44f1700 1 --2- 172.21.2.112:0/3521342413 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f45840186c0 0x7f458401abc0 crc :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:53.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:53.607 7f45a44f1700 1 -- 172.21.2.112:0/3521342413 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f459c0a7280 msgr2=0x7f459c0a4590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:53.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:53.607 7f45a44f1700 1 --2- 172.21.2.112:0/3521342413 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f459c0a7280 0x7f459c0a4590 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 rx=0x7f4594007f90 tx=0x7f459400ac70).stop 2021-02-10T11:20:53.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:53.607 7f45a44f1700 1 -- 172.21.2.112:0/3521342413 shutdown_connections 2021-02-10T11:20:53.602 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:53.607 7f45a44f1700 1 --2- 172.21.2.112:0/3521342413 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f45840186c0 0x7f458401abc0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:53.602 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:53.607 7f45a44f1700 1 --2- 172.21.2.112:0/3521342413 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f459c0a6860 0x7f459c0a6c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:20:53.602 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:53.607 7f45a44f1700 1 --2- 172.21.2.112:0/3521342413 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f459c0a7280 0x7f459c0a4590 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:53.602 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:53.607 7f45a44f1700 1 --2- 172.21.2.112:0/3521342413 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f459c08ece0 0x7f459c0a6240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:20:53.602 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:53.607 7f45a44f1700 1 -- 172.21.2.112:0/3521342413 >> 172.21.2.112:0/3521342413 conn(0x7f459c08ffe0 msgr2=0x7f459c0bf050 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:20:53.602 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:53.607 7f45a44f1700 1 -- 172.21.2.112:0/3521342413 shutdown_connections 2021-02-10T11:20:53.602 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:53.607 7f45a44f1700 1 -- 172.21.2.112:0/3521342413 wait complete. 2021-02-10T11:20:53.609 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000348603 2021-02-10T11:20:54.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:54.029 7f6bb710a700 1 Processor -- start 2021-02-10T11:20:54.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:54.029 7f6bb710a700 1 -- start start 2021-02-10T11:20:54.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:54.030 7f6bb710a700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f6bb01b0470 0x7f6bb01b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:54.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:54.030 7f6bb710a700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bb00f0b50 con 0x7f6bb00f20e0 2021-02-10T11:20:54.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:54.030 7f6bb710a700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bb00f24e0 con 0x7f6bb00ef610 2021-02-10T11:20:54.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:54.030 7f6bb710a700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bb00e5ea0 con 0x7f6bb01b0470 2021-02-10T11:20:54.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:54.030 7f6bb5727700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f6bb00f20e0 0x7f6bb00f0200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:44256/0 (socket says 172.21.2.112:44256) 2021-02-10T11:20:54.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:54.030 7f6bb5727700 1 -- 172.21.2.112:0/1547106259 learned_addr learned my addr 172.21.2.112:0/1547106259 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:20:54.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:54.030 7f6ba7fff700 1 --1- 172.21.2.112:0/1547106259 >> v1:172.21.2.120:6789/0 conn(0x7f6bb00ef610 0x7f6bb00efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:37810/0 (socket says 172.21.2.112:37810) 2021-02-10T11:20:54.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:54.030 7f6bb4f26700 1 --2- 172.21.2.112:0/1547106259 >> v2:172.21.2.120:3300/0 conn(0x7f6bb01b0470 0x7f6bb01b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:54.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:54.030 7f6ba77fe700 1 -- 172.21.2.112:0/1547106259 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 229995313 0 0) 0x7f6bac001080 con 0x7f6bb00ef610 2021-02-10T11:20:54.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:54.030 7f6ba77fe700 1 -- 172.21.2.112:0/1547106259 --> v1:172.21.2.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b98002fe0 con 0x7f6bb00ef610 2021-02-10T11:20:54.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:54.030 7f6ba77fe700 1 -- 172.21.2.112:0/1547106259 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2821543008 0 0) 0x7f6ba8001500 con 0x7f6bb00f20e0 2021-02-10T11:20:54.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:54.030 7f6ba77fe700 1 -- 172.21.2.112:0/1547106259 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b98004370 con 0x7f6bb00f20e0 2021-02-10T11:20:54.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:54.030 7f6bb4f26700 1 -- 172.21.2.112:0/1547106259 >> v1:172.21.2.120:6789/0 conn(0x7f6bb00ef610 legacy=0x7f6bb00efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:54.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:54.030 7f6bb4f26700 1 -- 172.21.2.112:0/1547106259 >> v1:172.21.2.112:6789/0 conn(0x7f6bb00f20e0 legacy=0x7f6bb00f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:54.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:54.030 7f6bb4f26700 1 -- 172.21.2.112:0/1547106259 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bb00d3330 con 0x7f6bb01b0470 2021-02-10T11:20:54.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:54.031 7f6bb4f26700 1 --2- 172.21.2.112:0/1547106259 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6bb01b0470 0x7f6bb01b0870 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f6ba0006650 tx=0x7f6ba0009a80).ready entity=mon.1 client_cookie=4c9c902d430a0930 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:54.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:54.031 7f6ba77fe700 1 -- 172.21.2.112:0/1547106259 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6ba00009b0 con 0x7f6bb01b0470 2021-02-10T11:20:54.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:54.031 7f6ba77fe700 1 -- 172.21.2.112:0/1547106259 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ba000c0c0 con 0x7f6bb01b0470 2021-02-10T11:20:54.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:54.031 7f6ba77fe700 1 -- 172.21.2.112:0/1547106259 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6ba000c5e0 con 0x7f6bb01b0470 2021-02-10T11:20:54.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:54.031 7f6bb710a700 1 -- 172.21.2.112:0/1547106259 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6bb01b0470 msgr2=0x7f6bb01b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:54.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:54.031 7f6bb710a700 1 --2- 172.21.2.112:0/1547106259 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6bb01b0470 0x7f6bb01b0870 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f6ba0006650 tx=0x7f6ba0009a80).stop 2021-02-10T11:20:54.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:54.031 7f6bb710a700 1 -- 172.21.2.112:0/1547106259 shutdown_connections 2021-02-10T11:20:54.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:54.031 7f6bb710a700 1 --2- 172.21.2.112:0/1547106259 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6bb01b0470 0x7f6bb01b0870 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:54.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:54.031 7f6bb710a700 1 -- 172.21.2.112:0/1547106259 >> 172.21.2.112:0/1547106259 conn(0x7f6bb00ce3e0 msgr2=0x7f6bb00e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:20:54.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:54.031 7f6bb710a700 1 -- 172.21.2.112:0/1547106259 shutdown_connections 2021-02-10T11:20:54.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:54.031 7f6bb710a700 1 -- 172.21.2.112:0/1547106259 wait complete. 2021-02-10T11:20:54.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:54.046 7f6bb710a700 1 Processor -- start 2021-02-10T11:20:54.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:54.046 7f6bb710a700 1 -- start start 2021-02-10T11:20:54.045 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:54.047 7f6bb710a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6bb008f480 0x7f6bb008f880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:54.045 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:54.047 7f6bb710a700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6bb008fea0 0x7f6bb0090cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:54.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:54.047 7f6bb710a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6bb00902a0 0x7f6bb0091310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:54.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:54.047 7f6bb710a700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bb0090770 con 0x7f6bb00902a0 2021-02-10T11:20:54.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:54.047 7f6bb710a700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bb010a1f0 con 0x7f6bb008fea0 2021-02-10T11:20:54.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:54.047 7f6bb710a700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bb010a3d0 con 0x7f6bb008f480 2021-02-10T11:20:54.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:54.047 7f6bb5727700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6bb00902a0 0x7f6bb0091310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:54.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:54.047 7f6bb5727700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6bb00902a0 0x7f6bb0091310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:44800/0 (socket says 172.21.2.112:44800) 2021-02-10T11:20:54.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:54.047 7f6bb5727700 1 -- 172.21.2.112:0/123992985 learned_addr learned my addr 172.21.2.112:0/123992985 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:20:54.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:54.047 7f6bb4f26700 1 --2- 172.21.2.112:0/123992985 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6bb008f480 0x7f6bb008f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:54.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:54.047 7f6ba7fff700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6bb008fea0 0x7f6bb0090cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:54.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:54.047 7f6bb5727700 1 -- 172.21.2.112:0/123992985 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6bb008f480 msgr2=0x7f6bb008f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:54.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:54.047 7f6bb5727700 1 --2- 172.21.2.112:0/123992985 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6bb008f480 0x7f6bb008f880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:54.047 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:54.047 7f6bb5727700 1 -- 172.21.2.112:0/123992985 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6bb008fea0 msgr2=0x7f6bb0090cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:54.047 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:54.047 7f6bb5727700 1 --2- 172.21.2.112:0/123992985 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6bb008fea0 0x7f6bb0090cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:54.047 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:54.047 7f6bb5727700 1 -- 172.21.2.112:0/123992985 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bb010a700 con 0x7f6bb00902a0 2021-02-10T11:20:54.047 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:54.047 7f6ba7fff700 1 --2- 172.21.2.112:0/123992985 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6bb008fea0 0x7f6bb0090cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:20:54.047 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:54.047 7f6bb5727700 1 --2- 172.21.2.112:0/123992985 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6bb00902a0 0x7f6bb0091310 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 rx=0x7f6ba8001c10 tx=0x7f6ba800ac50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:54.047 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:54.047 7f6ba57fa700 1 -- 172.21.2.112:0/123992985 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6ba8000d30 con 0x7f6bb00902a0 2021-02-10T11:20:54.047 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:54.048 7f6ba57fa700 1 -- 172.21.2.112:0/123992985 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ba8007020 con 0x7f6bb00902a0 2021-02-10T11:20:54.047 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:54.048 7f6ba57fa700 1 -- 172.21.2.112:0/123992985 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6ba80074f0 con 0x7f6bb00902a0 2021-02-10T11:20:54.047 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:54.048 7f6bb710a700 1 -- 172.21.2.112:0/123992985 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bb010a700 con 0x7f6bb00902a0 2021-02-10T11:20:54.047 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:54.048 7f6bb710a700 1 -- 172.21.2.112:0/123992985 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bb010a700 con 0x7f6bb00902a0 2021-02-10T11:20:54.047 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:54.048 7f6ba57fa700 1 -- 172.21.2.112:0/123992985 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6ba800ad90 con 0x7f6bb00902a0 2021-02-10T11:20:54.047 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:54.048 7f6ba57fa700 1 --2- 172.21.2.112:0/123992985 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6b900186c0 0x7f6b9001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:54.048 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:54.048 7f6ba57fa700 1 -- 172.21.2.112:0/123992985 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f6ba80328b0 con 0x7f6bb00902a0 2021-02-10T11:20:54.048 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:54.049 7f6bb4f26700 1 --2- 172.21.2.112:0/123992985 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6b900186c0 0x7f6b9001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:54.048 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:54.049 7f6bb4f26700 1 --2- 172.21.2.112:0/123992985 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6b900186c0 0x7f6b9001abc0 crc :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:54.073 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:54.079 7f6bb710a700 1 -- 172.21.2.112:0/123992985 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bb0105ae0 con 0x7f6bb00902a0 2021-02-10T11:20:54.075 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:54.081 7f6ba57fa700 1 -- 172.21.2.112:0/123992985 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6ba8007cb0 con 0x7f6bb00902a0 2021-02-10T11:20:54.205 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:54.211 7f6bb710a700 1 -- 172.21.2.112:0/123992985 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348603", "target": ["mgr", ""]}) v1 -- 0x7f6bb00f0b50 con 0x7f6b900186c0 2021-02-10T11:20:54.209 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:54.215 7f6ba57fa700 1 -- 172.21.2.112:0/123992985 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f6ba0006420 con 0x7f6b900186c0 2021-02-10T11:20:54.212 INFO:teuthology.orchestra.run.gibba012.stdout:/volumes/_nogroup/subvolume_0000000000348603/b751b9b6-f891-4179-8021-8fcf53ca6a60 2021-02-10T11:20:54.214 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:54.220 7f6bb710a700 1 -- 172.21.2.112:0/123992985 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6b900186c0 msgr2=0x7f6b9001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:54.214 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:54.220 7f6bb710a700 1 --2- 172.21.2.112:0/123992985 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6b900186c0 0x7f6b9001abc0 crc :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:54.214 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:54.220 7f6bb710a700 1 -- 172.21.2.112:0/123992985 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6bb00902a0 msgr2=0x7f6bb0091310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:54.214 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:54.220 7f6bb710a700 1 --2- 172.21.2.112:0/123992985 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6bb00902a0 0x7f6bb0091310 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 rx=0x7f6ba8001c10 tx=0x7f6ba800ac50).stop 2021-02-10T11:20:54.214 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:54.220 7f6bb710a700 1 -- 172.21.2.112:0/123992985 shutdown_connections 2021-02-10T11:20:54.214 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:54.220 7f6bb710a700 1 --2- 172.21.2.112:0/123992985 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6b900186c0 0x7f6b9001abc0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:54.214 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:54.220 7f6bb710a700 1 --2- 172.21.2.112:0/123992985 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6bb008fea0 0x7f6bb0090cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:54.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:54.220 7f6bb710a700 1 --2- 172.21.2.112:0/123992985 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6bb00902a0 0x7f6bb0091310 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:54.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:54.220 7f6bb710a700 1 --2- 172.21.2.112:0/123992985 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6bb008f480 0x7f6bb008f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:54.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:54.220 7f6bb710a700 1 -- 172.21.2.112:0/123992985 >> 172.21.2.112:0/123992985 conn(0x7f6bb00bbf60 msgr2=0x7f6bb0105d10 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:20:54.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:54.220 7f6bb710a700 1 -- 172.21.2.112:0/123992985 shutdown_connections 2021-02-10T11:20:54.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:20:54.220 7f6bb710a700 1 -- 172.21.2.112:0/123992985 wait complete. 2021-02-10T11:20:54.222 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000348603 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000348603/b751b9b6-f891-4179-8021-8fcf53ca6a60 2021-02-10T11:20:54.222 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:20:54.223 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:20:54.223 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:20:54.558 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:20:54.558 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:20:54.559 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348603/b751b9b6-f891-4179-8021-8fcf53ca6a60/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:20:54.940 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:20:54.940 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:20:54.940 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0253936 s, 41.3 MB/s 2021-02-10T11:20:54.941 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:20:54.941 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:20:54.941 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:20:55.277 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:20:55.277 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:20:55.278 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348603/b751b9b6-f891-4179-8021-8fcf53ca6a60/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:20:55.653 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:20:55.653 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:20:55.654 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0231788 s, 45.2 MB/s 2021-02-10T11:20:55.654 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:20:55.655 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:20:55.655 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:20:55.989 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:20:55.990 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:20:55.990 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348603/b751b9b6-f891-4179-8021-8fcf53ca6a60/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:20:56.367 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:20:56.367 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:20:56.367 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0252845 s, 41.5 MB/s 2021-02-10T11:20:56.368 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:20:56.368 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:20:56.368 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:20:56.704 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:20:56.705 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:20:56.705 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348603/b751b9b6-f891-4179-8021-8fcf53ca6a60/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:20:57.079 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:20:57.079 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:20:57.080 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0230958 s, 45.4 MB/s 2021-02-10T11:20:57.080 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:20:57.081 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:20:57.081 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:20:57.417 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:20:57.417 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:20:57.418 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348603/b751b9b6-f891-4179-8021-8fcf53ca6a60/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:20:57.792 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:20:57.792 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:20:57.792 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.025132 s, 41.7 MB/s 2021-02-10T11:20:57.793 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:20:57.793 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:20:57.793 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:20:58.126 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:20:58.127 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:20:58.127 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348603/b751b9b6-f891-4179-8021-8fcf53ca6a60/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:20:58.503 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:20:58.504 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:20:58.504 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0262124 s, 40.0 MB/s 2021-02-10T11:20:58.505 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:20:58.505 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:20:58.505 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:20:58.841 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:20:58.841 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:20:58.841 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348603/b751b9b6-f891-4179-8021-8fcf53ca6a60/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:20:59.215 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:20:59.216 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:20:59.216 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0234212 s, 44.8 MB/s 2021-02-10T11:20:59.217 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:20:59.217 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:20:59.217 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:20:59.551 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:20:59.551 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:20:59.551 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348603/b751b9b6-f891-4179-8021-8fcf53ca6a60/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:20:59.923 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:20:59.924 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:20:59.924 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0244417 s, 42.9 MB/s 2021-02-10T11:20:59.925 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:20:59.925 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:20:59.925 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:21:00.259 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:21:00.260 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:21:00.260 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348603/b751b9b6-f891-4179-8021-8fcf53ca6a60/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:21:00.636 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:21:00.636 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:21:00.636 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0248644 s, 42.2 MB/s 2021-02-10T11:21:00.637 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:21:00.637 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:21:00.637 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:21:00.972 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:21:00.972 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:21:00.972 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348603/b751b9b6-f891-4179-8021-8fcf53ca6a60/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:21:01.348 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:21:01.349 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:21:01.349 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0252216 s, 41.6 MB/s 2021-02-10T11:21:01.350 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000348604 2021-02-10T11:21:01.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:01.769 7efd39ef9700 1 Processor -- start 2021-02-10T11:21:01.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:01.769 7efd39ef9700 1 -- start start 2021-02-10T11:21:01.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:01.770 7efd39ef9700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7efd341b0870 0x7efd340f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:01.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:01.770 7efd39ef9700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7efd340efe00 0x7efd340f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:01.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:01.770 7efd39ef9700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd341b0c70 con 0x7efd341b0470 2021-02-10T11:21:01.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:01.770 7efd39ef9700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd340e5ea0 con 0x7efd341b0870 2021-02-10T11:21:01.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:01.770 7efd39ef9700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd340f0b50 con 0x7efd340efe00 2021-02-10T11:21:01.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:01.770 7efd32ffd700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7efd341b0870 0x7efd340f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:01.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:01.770 7efd32ffd700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7efd341b0870 0x7efd340f20e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:44804/0 (socket says 172.21.2.112:44804) 2021-02-10T11:21:01.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:01.770 7efd32ffd700 1 -- 172.21.2.112:0/1260065226 learned_addr learned my addr 172.21.2.112:0/1260065226 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:21:01.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:01.770 7efd33fff700 1 --2- 172.21.2.112:0/1260065226 >> v2:172.21.2.120:3300/0 conn(0x7efd340efe00 0x7efd340f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:01.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:01.770 7efd32ffd700 1 -- 172.21.2.112:0/1260065226 >> v2:172.21.2.120:3300/0 conn(0x7efd340efe00 msgr2=0x7efd340f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:21:01.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:01.770 7efd32ffd700 1 --2- 172.21.2.112:0/1260065226 >> v2:172.21.2.120:3300/0 conn(0x7efd340efe00 0x7efd340f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:01.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:01.770 7efd32ffd700 1 -- 172.21.2.112:0/1260065226 >> v1:172.21.2.120:6789/0 conn(0x7efd341b0470 legacy=0x7efd340ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:21:01.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:01.770 7efd32ffd700 1 -- 172.21.2.112:0/1260065226 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd340f0820 con 0x7efd341b0870 2021-02-10T11:21:01.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:01.770 7efd32ffd700 1 --2- 172.21.2.112:0/1260065226 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efd341b0870 0x7efd340f20e0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 rx=0x7efd28006700 tx=0x7efd28009b00).ready entity=mon.0 client_cookie=5d7c6e6b327a5411 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:01.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:01.771 7efd327fc700 1 -- 172.21.2.112:0/1260065226 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efd2800c0c0 con 0x7efd341b0870 2021-02-10T11:21:01.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:01.771 7efd327fc700 1 -- 172.21.2.112:0/1260065226 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd2800c340 con 0x7efd341b0870 2021-02-10T11:21:01.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:01.771 7efd327fc700 1 -- 172.21.2.112:0/1260065226 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efd2800c820 con 0x7efd341b0870 2021-02-10T11:21:01.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:01.771 7efd39ef9700 1 -- 172.21.2.112:0/1260065226 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efd341b0870 msgr2=0x7efd340f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:01.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:01.771 7efd39ef9700 1 --2- 172.21.2.112:0/1260065226 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efd341b0870 0x7efd340f20e0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 rx=0x7efd28006700 tx=0x7efd28009b00).stop 2021-02-10T11:21:01.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:01.771 7efd39ef9700 1 -- 172.21.2.112:0/1260065226 shutdown_connections 2021-02-10T11:21:01.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:01.771 7efd39ef9700 1 --2- 172.21.2.112:0/1260065226 >> v2:172.21.2.120:3300/0 conn(0x7efd340efe00 0x7efd340f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:01.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:01.771 7efd39ef9700 1 --2- 172.21.2.112:0/1260065226 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efd341b0870 0x7efd340f20e0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:01.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:01.771 7efd39ef9700 1 -- 172.21.2.112:0/1260065226 >> 172.21.2.112:0/1260065226 conn(0x7efd340ce3e0 msgr2=0x7efd340e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:01.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:01.771 7efd39ef9700 1 -- 172.21.2.112:0/1260065226 shutdown_connections 2021-02-10T11:21:01.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:01.771 7efd39ef9700 1 -- 172.21.2.112:0/1260065226 wait complete. 2021-02-10T11:21:01.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:01.786 7efd39ef9700 1 Processor -- start 2021-02-10T11:21:01.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:01.786 7efd39ef9700 1 -- start start 2021-02-10T11:21:01.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:01.786 7efd39ef9700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efd340a0420 0x7efd340a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:01.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:01.786 7efd39ef9700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efd34189d00 0x7efd3418a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:01.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:01.786 7efd39ef9700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7efd3418a720 0x7efd340b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:01.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:01.786 7efd39ef9700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd34189800 con 0x7efd340a0420 2021-02-10T11:21:01.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:01.786 7efd39ef9700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd3418ab20 con 0x7efd3418a720 2021-02-10T11:21:01.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:01.786 7efd39ef9700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd340a1140 con 0x7efd34189d00 2021-02-10T11:21:01.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:01.786 7efd337fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efd340a0420 0x7efd340a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:01.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:01.786 7efd337fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efd340a0420 0x7efd340a0820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:44808/0 (socket says 172.21.2.112:44808) 2021-02-10T11:21:01.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:01.786 7efd337fe700 1 -- 172.21.2.112:0/1121918774 learned_addr learned my addr 172.21.2.112:0/1121918774 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:21:01.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:01.786 7efd32ffd700 1 --2- 172.21.2.112:0/1121918774 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efd34189d00 0x7efd3418a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:01.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:01.786 7efd33fff700 1 --2- 172.21.2.112:0/1121918774 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7efd3418a720 0x7efd340b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:01.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:01.787 7efd337fe700 1 -- 172.21.2.112:0/1121918774 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efd34189d00 msgr2=0x7efd3418a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:01.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:01.787 7efd337fe700 1 --2- 172.21.2.112:0/1121918774 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efd34189d00 0x7efd3418a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:01.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:01.787 7efd337fe700 1 -- 172.21.2.112:0/1121918774 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7efd3418a720 msgr2=0x7efd340b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:01.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:01.787 7efd337fe700 1 --2- 172.21.2.112:0/1121918774 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7efd3418a720 0x7efd340b7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:01.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:01.787 7efd337fe700 1 -- 172.21.2.112:0/1121918774 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd340b8200 con 0x7efd340a0420 2021-02-10T11:21:01.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:01.787 7efd33fff700 1 --2- 172.21.2.112:0/1121918774 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7efd3418a720 0x7efd340b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:21:01.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:01.787 7efd337fe700 1 --2- 172.21.2.112:0/1121918774 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efd340a0420 0x7efd340a0820 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 rx=0x7efd2400b5b0 tx=0x7efd2400b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:01.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:01.787 7efd1bfff700 1 -- 172.21.2.112:0/1121918774 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efd2400bd40 con 0x7efd340a0420 2021-02-10T11:21:01.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:01.787 7efd1bfff700 1 -- 172.21.2.112:0/1121918774 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd24000e20 con 0x7efd340a0420 2021-02-10T11:21:01.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:01.787 7efd1bfff700 1 -- 172.21.2.112:0/1121918774 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efd240054a0 con 0x7efd340a0420 2021-02-10T11:21:01.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:01.787 7efd39ef9700 1 -- 172.21.2.112:0/1121918774 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd340b8200 con 0x7efd340a0420 2021-02-10T11:21:01.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:01.787 7efd39ef9700 1 -- 172.21.2.112:0/1121918774 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd340b8200 con 0x7efd340a0420 2021-02-10T11:21:01.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:01.787 7efd1bfff700 1 -- 172.21.2.112:0/1121918774 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7efd24006030 con 0x7efd340a0420 2021-02-10T11:21:01.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:01.787 7efd1bfff700 1 --2- 172.21.2.112:0/1121918774 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7efd14018730 0x7efd1401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:01.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:01.787 7efd1bfff700 1 -- 172.21.2.112:0/1121918774 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7efd24032660 con 0x7efd340a0420 2021-02-10T11:21:01.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:01.788 7efd32ffd700 1 --2- 172.21.2.112:0/1121918774 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7efd14018730 0x7efd1401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:01.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:01.788 7efd32ffd700 1 --2- 172.21.2.112:0/1121918774 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7efd14018730 0x7efd1401ac10 crc :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:01.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:01.820 7efd39ef9700 1 -- 172.21.2.112:0/1121918774 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd34105ae0 con 0x7efd340a0420 2021-02-10T11:21:01.816 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:01.822 7efd1bfff700 1 -- 172.21.2.112:0/1121918774 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7efd2400bd40 con 0x7efd340a0420 2021-02-10T11:21:01.944 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:01.950 7efd39ef9700 1 -- 172.21.2.112:0/1121918774 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348604", "target": ["mgr", ""]}) v1 -- 0x7efd3418ab20 con 0x7efd14018730 2021-02-10T11:21:01.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:01.976 7efd1bfff700 1 -- 172.21.2.112:0/1121918774 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7efd28009100 con 0x7efd14018730 2021-02-10T11:21:01.977 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:01.983 7efd39ef9700 1 -- 172.21.2.112:0/1121918774 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7efd14018730 msgr2=0x7efd1401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:01.977 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:01.983 7efd39ef9700 1 --2- 172.21.2.112:0/1121918774 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7efd14018730 0x7efd1401ac10 crc :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:01.977 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:01.983 7efd39ef9700 1 -- 172.21.2.112:0/1121918774 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efd340a0420 msgr2=0x7efd340a0820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:01.977 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:01.983 7efd39ef9700 1 --2- 172.21.2.112:0/1121918774 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efd340a0420 0x7efd340a0820 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 rx=0x7efd2400b5b0 tx=0x7efd2400b950).stop 2021-02-10T11:21:01.977 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:01.983 7efd39ef9700 1 -- 172.21.2.112:0/1121918774 shutdown_connections 2021-02-10T11:21:01.977 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:01.983 7efd39ef9700 1 --2- 172.21.2.112:0/1121918774 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7efd14018730 0x7efd1401ac10 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:01.977 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:01.983 7efd39ef9700 1 --2- 172.21.2.112:0/1121918774 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7efd3418a720 0x7efd340b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:01.978 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:01.983 7efd39ef9700 1 --2- 172.21.2.112:0/1121918774 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efd34189d00 0x7efd3418a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:01.978 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:01.983 7efd39ef9700 1 --2- 172.21.2.112:0/1121918774 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efd340a0420 0x7efd340a0820 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:01.978 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:01.983 7efd39ef9700 1 -- 172.21.2.112:0/1121918774 >> 172.21.2.112:0/1121918774 conn(0x7efd34094e10 msgr2=0x7efd340a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:01.978 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:01.983 7efd39ef9700 1 -- 172.21.2.112:0/1121918774 shutdown_connections 2021-02-10T11:21:01.978 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:01.983 7efd39ef9700 1 -- 172.21.2.112:0/1121918774 wait complete. 2021-02-10T11:21:01.985 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000348604 2021-02-10T11:21:02.416 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:02.405 7f8593ba3700 1 Processor -- start 2021-02-10T11:21:02.416 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:02.405 7f8593ba3700 1 -- start start 2021-02-10T11:21:02.416 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:02.406 7f8593ba3700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f858c1b0470 0x7f858c1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:02.416 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:02.406 7f8593ba3700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f858c0ef610 0x7f858c0efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:02.416 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:02.406 7f8593ba3700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f858c0f0b50 con 0x7f858c0f0030 2021-02-10T11:21:02.416 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:02.406 7f8593ba3700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f858c0e52e0 con 0x7f858c0ef610 2021-02-10T11:21:02.416 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:02.406 7f8593ba3700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f858c0e5490 con 0x7f858c1b0470 2021-02-10T11:21:02.417 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:02.406 7f85911be700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f858c0ef610 0x7f858c0efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:02.417 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:02.406 7f85911be700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f858c0ef610 0x7f858c0efa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:44816/0 (socket says 172.21.2.112:44816) 2021-02-10T11:21:02.417 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:02.406 7f85911be700 1 -- 172.21.2.112:0/2541452230 learned_addr learned my addr 172.21.2.112:0/2541452230 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:21:02.417 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:02.406 7f85911be700 1 -- 172.21.2.112:0/2541452230 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f858c1b0470 msgr2=0x7f858c1b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:21:02.417 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:02.406 7f85911be700 1 --2- 172.21.2.112:0/2541452230 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f858c1b0470 0x7f858c1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:02.417 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:02.406 7f85911be700 1 -- 172.21.2.112:0/2541452230 >> v1:172.21.2.112:6789/0 conn(0x7f858c0f0030 legacy=0x7f858c0f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:02.417 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:02.406 7f85911be700 1 -- 172.21.2.112:0/2541452230 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f858c0f0430 con 0x7f858c0ef610 2021-02-10T11:21:02.417 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:02.406 7f85911be700 1 --2- 172.21.2.112:0/2541452230 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f858c0ef610 0x7f858c0efa10 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 rx=0x7f8580006700 tx=0x7f8580009b80).ready entity=mon.0 client_cookie=fc2f0f867a902531 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:02.417 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:02.406 7f85909bd700 1 -- 172.21.2.112:0/2541452230 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f858000c0c0 con 0x7f858c0ef610 2021-02-10T11:21:02.417 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:02.407 7f85909bd700 1 -- 172.21.2.112:0/2541452230 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f858000c340 con 0x7f858c0ef610 2021-02-10T11:21:02.417 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:02.407 7f85909bd700 1 -- 172.21.2.112:0/2541452230 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f858000c820 con 0x7f858c0ef610 2021-02-10T11:21:02.417 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:02.407 7f8593ba3700 1 -- 172.21.2.112:0/2541452230 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f858c0ef610 msgr2=0x7f858c0efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:02.417 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:02.407 7f8593ba3700 1 --2- 172.21.2.112:0/2541452230 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f858c0ef610 0x7f858c0efa10 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 rx=0x7f8580006700 tx=0x7f8580009b80).stop 2021-02-10T11:21:02.418 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:02.407 7f8593ba3700 1 -- 172.21.2.112:0/2541452230 shutdown_connections 2021-02-10T11:21:02.418 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:02.407 7f8593ba3700 1 --2- 172.21.2.112:0/2541452230 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f858c0ef610 0x7f858c0efa10 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:02.418 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:02.407 7f8593ba3700 1 --2- 172.21.2.112:0/2541452230 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f858c1b0470 0x7f858c1b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:02.418 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:02.407 7f8593ba3700 1 -- 172.21.2.112:0/2541452230 >> 172.21.2.112:0/2541452230 conn(0x7f858c0ce3e0 msgr2=0x7f858c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:02.418 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:02.407 7f8593ba3700 1 -- 172.21.2.112:0/2541452230 shutdown_connections 2021-02-10T11:21:02.418 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:02.407 7f8593ba3700 1 -- 172.21.2.112:0/2541452230 wait complete. 2021-02-10T11:21:02.420 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:02.422 7f8593ba3700 1 Processor -- start 2021-02-10T11:21:02.420 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:02.422 7f8593ba3700 1 -- start start 2021-02-10T11:21:02.420 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:02.423 7f8593ba3700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f858c094fb0 0x7f858c09e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:02.420 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:02.423 7f8593ba3700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f858c0c2be0 0x7f858c0c2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:02.420 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:02.423 7f8593ba3700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f858c0c3600 0x7f858c0a9c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:02.420 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:02.423 7f8593ba3700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f858c09ebf0 con 0x7f858c0c2be0 2021-02-10T11:21:02.420 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:02.423 7f8593ba3700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f858c0aa2a0 con 0x7f858c094fb0 2021-02-10T11:21:02.420 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:02.423 7f8593ba3700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f858c0aa4c0 con 0x7f858c0c3600 2021-02-10T11:21:02.420 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:02.423 7f85911be700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f858c0c2be0 0x7f858c0c2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:02.420 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:02.423 7f85911be700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f858c0c2be0 0x7f858c0c2fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:44820/0 (socket says 172.21.2.112:44820) 2021-02-10T11:21:02.420 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:02.423 7f85911be700 1 -- 172.21.2.112:0/1852690487 learned_addr learned my addr 172.21.2.112:0/1852690487 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:21:02.421 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:02.423 7f85919bf700 1 --2- 172.21.2.112:0/1852690487 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f858c094fb0 0x7f858c09e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:02.421 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:02.423 7f85921c0700 1 --2- 172.21.2.112:0/1852690487 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f858c0c3600 0x7f858c0a9c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:02.421 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:02.423 7f85911be700 1 -- 172.21.2.112:0/1852690487 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f858c0c3600 msgr2=0x7f858c0a9c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:02.421 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:02.423 7f85911be700 1 --2- 172.21.2.112:0/1852690487 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f858c0c3600 0x7f858c0a9c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:02.421 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:02.423 7f85911be700 1 -- 172.21.2.112:0/1852690487 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f858c094fb0 msgr2=0x7f858c09e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:02.421 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:02.423 7f85911be700 1 --2- 172.21.2.112:0/1852690487 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f858c094fb0 0x7f858c09e5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:02.421 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:02.423 7f85911be700 1 -- 172.21.2.112:0/1852690487 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f858c0aa6e0 con 0x7f858c0c2be0 2021-02-10T11:21:02.421 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:02.423 7f85911be700 1 --2- 172.21.2.112:0/1852690487 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f858c0c2be0 0x7f858c0c2fe0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 rx=0x7f858000b650 tx=0x7f858000b9f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:02.421 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:02.424 7f857e7fc700 1 -- 172.21.2.112:0/1852690487 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f858000bde0 con 0x7f858c0c2be0 2021-02-10T11:21:02.421 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:02.424 7f857e7fc700 1 -- 172.21.2.112:0/1852690487 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8580000c90 con 0x7f858c0c2be0 2021-02-10T11:21:02.421 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:02.424 7f857e7fc700 1 -- 172.21.2.112:0/1852690487 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f85800022b0 con 0x7f858c0c2be0 2021-02-10T11:21:02.421 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:02.424 7f8593ba3700 1 -- 172.21.2.112:0/1852690487 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f858c0aa6e0 con 0x7f858c0c2be0 2021-02-10T11:21:02.421 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:02.424 7f8593ba3700 1 -- 172.21.2.112:0/1852690487 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f858c0aa6e0 con 0x7f858c0c2be0 2021-02-10T11:21:02.421 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:02.424 7f857e7fc700 1 -- 172.21.2.112:0/1852690487 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8580009f90 con 0x7f858c0c2be0 2021-02-10T11:21:02.422 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:02.424 7f857e7fc700 1 --2- 172.21.2.112:0/1852690487 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f85740186c0 0x7f857401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:02.422 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:02.424 7f857e7fc700 1 -- 172.21.2.112:0/1852690487 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f8580032400 con 0x7f858c0c2be0 2021-02-10T11:21:02.422 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:02.425 7f85919bf700 1 --2- 172.21.2.112:0/1852690487 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f85740186c0 0x7f857401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:02.422 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:02.426 7f85919bf700 1 --2- 172.21.2.112:0/1852690487 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f85740186c0 0x7f857401abc0 crc :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:02.450 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:02.456 7f8593ba3700 1 -- 172.21.2.112:0/1852690487 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f858c0a9750 con 0x7f858c0c2be0 2021-02-10T11:21:02.452 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:02.458 7f857e7fc700 1 -- 172.21.2.112:0/1852690487 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8580030d30 con 0x7f858c0c2be0 2021-02-10T11:21:02.581 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:02.586 7f8593ba3700 1 -- 172.21.2.112:0/1852690487 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348604", "target": ["mgr", ""]}) v1 -- 0x7f858c0a9750 con 0x7f85740186c0 2021-02-10T11:21:02.584 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:02.590 7f857e7fc700 1 -- 172.21.2.112:0/1852690487 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f8588000940 con 0x7f85740186c0 2021-02-10T11:21:02.588 INFO:teuthology.orchestra.run.gibba012.stdout:/volumes/_nogroup/subvolume_0000000000348604/ff30e54c-90d1-48aa-bbd9-e632ffc23a3e 2021-02-10T11:21:02.590 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:02.596 7f8593ba3700 1 -- 172.21.2.112:0/1852690487 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f85740186c0 msgr2=0x7f857401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:02.590 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:02.596 7f8593ba3700 1 --2- 172.21.2.112:0/1852690487 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f85740186c0 0x7f857401abc0 crc :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:02.591 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:02.596 7f8593ba3700 1 -- 172.21.2.112:0/1852690487 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f858c0c2be0 msgr2=0x7f858c0c2fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:02.591 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:02.596 7f8593ba3700 1 --2- 172.21.2.112:0/1852690487 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f858c0c2be0 0x7f858c0c2fe0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 rx=0x7f858000b650 tx=0x7f858000b9f0).stop 2021-02-10T11:21:02.591 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:02.596 7f8593ba3700 1 -- 172.21.2.112:0/1852690487 shutdown_connections 2021-02-10T11:21:02.592 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:02.596 7f8593ba3700 1 --2- 172.21.2.112:0/1852690487 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f85740186c0 0x7f857401abc0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:02.592 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:02.596 7f8593ba3700 1 --2- 172.21.2.112:0/1852690487 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f858c0c3600 0x7f858c0a9c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:02.592 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:02.596 7f8593ba3700 1 --2- 172.21.2.112:0/1852690487 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f858c0c2be0 0x7f858c0c2fe0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:02.592 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:02.596 7f8593ba3700 1 --2- 172.21.2.112:0/1852690487 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f858c094fb0 0x7f858c09e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:02.592 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:02.596 7f8593ba3700 1 -- 172.21.2.112:0/1852690487 >> 172.21.2.112:0/1852690487 conn(0x7f858c09f630 msgr2=0x7f858c0a0130 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:02.592 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:02.597 7f8593ba3700 1 -- 172.21.2.112:0/1852690487 shutdown_connections 2021-02-10T11:21:02.593 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:02.597 7f8593ba3700 1 -- 172.21.2.112:0/1852690487 wait complete. 2021-02-10T11:21:02.598 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000348604 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000348604/ff30e54c-90d1-48aa-bbd9-e632ffc23a3e 2021-02-10T11:21:02.598 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:21:02.599 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:21:02.599 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:21:02.937 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:21:02.937 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:21:02.937 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348604/ff30e54c-90d1-48aa-bbd9-e632ffc23a3e/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:21:03.319 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:21:03.319 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:21:03.319 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0250971 s, 41.8 MB/s 2021-02-10T11:21:03.320 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:21:03.320 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:21:03.321 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:21:03.656 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:21:03.656 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:21:03.656 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348604/ff30e54c-90d1-48aa-bbd9-e632ffc23a3e/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:21:04.032 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:21:04.032 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:21:04.033 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0258928 s, 40.5 MB/s 2021-02-10T11:21:04.033 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:21:04.034 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:21:04.034 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:21:04.370 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:21:04.370 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:21:04.370 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348604/ff30e54c-90d1-48aa-bbd9-e632ffc23a3e/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:21:04.752 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:21:04.753 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:21:04.753 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0271373 s, 38.6 MB/s 2021-02-10T11:21:04.754 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:21:04.755 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:21:04.756 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:21:05.095 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:21:05.095 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:21:05.095 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348604/ff30e54c-90d1-48aa-bbd9-e632ffc23a3e/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:21:05.482 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:21:05.482 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:21:05.482 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0244276 s, 42.9 MB/s 2021-02-10T11:21:05.484 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:21:05.484 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:21:05.484 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:21:05.820 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:21:05.820 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:21:05.821 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348604/ff30e54c-90d1-48aa-bbd9-e632ffc23a3e/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:21:06.199 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:21:06.199 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:21:06.199 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0258463 s, 40.6 MB/s 2021-02-10T11:21:06.200 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:21:06.200 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:21:06.200 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:21:06.437 DEBUG:teuthology.orchestra.run.gibba012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:21:06.441 DEBUG:teuthology.orchestra.run.gibba020:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:21:06.536 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:21:06.537 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:21:06.537 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348604/ff30e54c-90d1-48aa-bbd9-e632ffc23a3e/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:21:06.812 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:21:06.812 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:21:06.812 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0229168 s, 45.8 MB/s 2021-02-10T11:21:06.813 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:21:06.814 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:21:06.814 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:21:07.187 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:21:07.187 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:21:07.187 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348604/ff30e54c-90d1-48aa-bbd9-e632ffc23a3e/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:21:07.563 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:21:07.563 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:21:07.563 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0253961 s, 41.3 MB/s 2021-02-10T11:21:07.564 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:21:07.564 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:21:07.564 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:21:07.899 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:21:07.899 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:21:07.899 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348604/ff30e54c-90d1-48aa-bbd9-e632ffc23a3e/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:21:08.275 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:21:08.275 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:21:08.275 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.025612 s, 40.9 MB/s 2021-02-10T11:21:08.276 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:21:08.276 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:21:08.276 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:21:08.611 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:21:08.611 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:21:08.611 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348604/ff30e54c-90d1-48aa-bbd9-e632ffc23a3e/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:21:08.983 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:21:08.983 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:21:08.983 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0220767 s, 47.5 MB/s 2021-02-10T11:21:08.984 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:21:08.984 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:21:08.984 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:21:09.324 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:21:09.324 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:21:09.325 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348604/ff30e54c-90d1-48aa-bbd9-e632ffc23a3e/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:21:09.697 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:21:09.697 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:21:09.697 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0235062 s, 44.6 MB/s 2021-02-10T11:21:09.698 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000348605 2021-02-10T11:21:10.126 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.117 7f1c92614700 1 Processor -- start 2021-02-10T11:21:10.126 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.117 7f1c92614700 1 -- start start 2021-02-10T11:21:10.126 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.117 7f1c92614700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1c8c1b0470 0x7f1c8c1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:10.126 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.117 7f1c92614700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f1c8c0ef610 0x7f1c8c0efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:10.127 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.117 7f1c92614700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f1c8c0f0030 0x7f1c8c0f0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:10.127 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.117 7f1c92614700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c8c0e5ea0 con 0x7f1c8c0f0030 2021-02-10T11:21:10.127 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.117 7f1c92614700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c8c0f0b50 con 0x7f1c8c0ef610 2021-02-10T11:21:10.127 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.117 7f1c92614700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c8c0e52e0 con 0x7f1c8c1b0470 2021-02-10T11:21:10.127 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.117 7f1c90c31700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f1c8c0f0030 0x7f1c8c0f0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:10.127 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.117 7f1c90c31700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f1c8c0f0030 0x7f1c8c0f0430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:44828/0 (socket says 172.21.2.112:44828) 2021-02-10T11:21:10.127 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.117 7f1c90c31700 1 -- 172.21.2.112:0/613131122 learned_addr learned my addr 172.21.2.112:0/613131122 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:21:10.127 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.117 7f1c90c31700 1 -- 172.21.2.112:0/613131122 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1c8c1b0470 msgr2=0x7f1c8c1b0870 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:21:10.127 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.117 7f1c90c31700 1 --2- 172.21.2.112:0/613131122 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1c8c1b0470 0x7f1c8c1b0870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:10.127 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.117 7f1c90c31700 1 -- 172.21.2.112:0/613131122 >> v2:172.21.2.120:3300/0 conn(0x7f1c8c0ef610 msgr2=0x7f1c8c0efa10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:21:10.127 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.117 7f1c90c31700 1 --2- 172.21.2.112:0/613131122 >> v2:172.21.2.120:3300/0 conn(0x7f1c8c0ef610 0x7f1c8c0efa10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:10.127 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.117 7f1c90c31700 1 -- 172.21.2.112:0/613131122 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c8c0f10d0 con 0x7f1c8c0f0030 2021-02-10T11:21:10.127 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.118 7f1c90c31700 1 --2- 172.21.2.112:0/613131122 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1c8c0f0030 0x7f1c8c0f0430 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 rx=0x7f1c84006f00 tx=0x7f1c84009f10).ready entity=mon.0 client_cookie=2ad1e2f1c7cddd87 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:10.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.118 7f1c8affd700 1 -- 172.21.2.112:0/613131122 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1c8400b3b0 con 0x7f1c8c0f0030 2021-02-10T11:21:10.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.118 7f1c8affd700 1 -- 172.21.2.112:0/613131122 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c8400b730 con 0x7f1c8c0f0030 2021-02-10T11:21:10.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.118 7f1c8affd700 1 -- 172.21.2.112:0/613131122 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1c8400bb90 con 0x7f1c8c0f0030 2021-02-10T11:21:10.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.118 7f1c92614700 1 -- 172.21.2.112:0/613131122 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1c8c0f0030 msgr2=0x7f1c8c0f0430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:10.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.118 7f1c92614700 1 --2- 172.21.2.112:0/613131122 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1c8c0f0030 0x7f1c8c0f0430 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 rx=0x7f1c84006f00 tx=0x7f1c84009f10).stop 2021-02-10T11:21:10.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.118 7f1c92614700 1 -- 172.21.2.112:0/613131122 shutdown_connections 2021-02-10T11:21:10.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.118 7f1c92614700 1 --2- 172.21.2.112:0/613131122 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1c8c0f0030 0x7f1c8c0f0430 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:10.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.118 7f1c92614700 1 --2- 172.21.2.112:0/613131122 >> v2:172.21.2.120:3300/0 conn(0x7f1c8c0ef610 0x7f1c8c0efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:10.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.118 7f1c92614700 1 --2- 172.21.2.112:0/613131122 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1c8c1b0470 0x7f1c8c1b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:10.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.118 7f1c92614700 1 -- 172.21.2.112:0/613131122 >> 172.21.2.112:0/613131122 conn(0x7f1c8c0ce3e0 msgr2=0x7f1c8c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:10.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.118 7f1c92614700 1 -- 172.21.2.112:0/613131122 shutdown_connections 2021-02-10T11:21:10.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.118 7f1c92614700 1 -- 172.21.2.112:0/613131122 wait complete. 2021-02-10T11:21:10.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.133 7f1c92614700 1 Processor -- start 2021-02-10T11:21:10.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.133 7f1c92614700 1 -- start start 2021-02-10T11:21:10.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.133 7f1c92614700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1c8c0c0b90 0x7f1c8c0c0f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:10.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.133 7f1c92614700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1c8c0c15b0 0x7f1c8c0c19b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:10.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.133 7f1c92614700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1c8c0aa4e0 0x7f1c8c0aa8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:10.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.133 7f1c92614700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c8c199f40 con 0x7f1c8c0c15b0 2021-02-10T11:21:10.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.133 7f1c92614700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c8c094bb0 con 0x7f1c8c0c0b90 2021-02-10T11:21:10.131 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.133 7f1c92614700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c8c0aaf00 con 0x7f1c8c0aa4e0 2021-02-10T11:21:10.131 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.133 7f1c8b7fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1c8c0c15b0 0x7f1c8c0c19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:10.131 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.133 7f1c8b7fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1c8c0c15b0 0x7f1c8c0c19b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:44836/0 (socket says 172.21.2.112:44836) 2021-02-10T11:21:10.131 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.133 7f1c8b7fe700 1 -- 172.21.2.112:0/806439186 learned_addr learned my addr 172.21.2.112:0/806439186 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:21:10.131 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.134 7f1c8b7fe700 1 -- 172.21.2.112:0/806439186 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1c8c0aa4e0 msgr2=0x7f1c8c0aa8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:10.131 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.134 7f1c8b7fe700 1 --2- 172.21.2.112:0/806439186 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1c8c0aa4e0 0x7f1c8c0aa8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:10.131 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.134 7f1c8b7fe700 1 -- 172.21.2.112:0/806439186 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1c8c0c0b90 msgr2=0x7f1c8c0c0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:10.131 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.134 7f1c8b7fe700 1 --2- 172.21.2.112:0/806439186 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1c8c0c0b90 0x7f1c8c0c0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:10.131 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.134 7f1c8b7fe700 1 -- 172.21.2.112:0/806439186 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c8c0ab120 con 0x7f1c8c0c15b0 2021-02-10T11:21:10.131 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.134 7f1c8b7fe700 1 --2- 172.21.2.112:0/806439186 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1c8c0c15b0 0x7f1c8c0c19b0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 rx=0x7f1c8000b5c0 tx=0x7f1c8000b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:10.131 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.134 7f1c88ff9700 1 -- 172.21.2.112:0/806439186 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1c8000bd50 con 0x7f1c8c0c15b0 2021-02-10T11:21:10.131 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.134 7f1c88ff9700 1 -- 172.21.2.112:0/806439186 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c80006030 con 0x7f1c8c0c15b0 2021-02-10T11:21:10.132 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.134 7f1c88ff9700 1 -- 172.21.2.112:0/806439186 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1c800054d0 con 0x7f1c8c0c15b0 2021-02-10T11:21:10.132 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.134 7f1c92614700 1 -- 172.21.2.112:0/806439186 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c8c0ab120 con 0x7f1c8c0c15b0 2021-02-10T11:21:10.132 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.134 7f1c92614700 1 -- 172.21.2.112:0/806439186 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c8c0ab6b0 con 0x7f1c8c0c15b0 2021-02-10T11:21:10.132 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.135 7f1c88ff9700 1 -- 172.21.2.112:0/806439186 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1c80006210 con 0x7f1c8c0c15b0 2021-02-10T11:21:10.132 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.135 7f1c88ff9700 1 --2- 172.21.2.112:0/806439186 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1c6c01cae0 0x7f1c6c01efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:10.132 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.135 7f1c88ff9700 1 -- 172.21.2.112:0/806439186 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f1c800325e0 con 0x7f1c8c0c15b0 2021-02-10T11:21:10.132 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.136 7f1c8bfff700 1 --2- 172.21.2.112:0/806439186 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1c6c01cae0 0x7f1c6c01efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:10.132 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.137 7f1c8bfff700 1 --2- 172.21.2.112:0/806439186 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1c6c01cae0 0x7f1c6c01efe0 crc :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:10.161 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.167 7f1c92614700 1 -- 172.21.2.112:0/806439186 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c8c0ab960 con 0x7f1c8c0c15b0 2021-02-10T11:21:10.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.169 7f1c88ff9700 1 -- 172.21.2.112:0/806439186 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1c80030d90 con 0x7f1c8c0c15b0 2021-02-10T11:21:10.288 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.294 7f1c92614700 1 -- 172.21.2.112:0/806439186 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348605", "target": ["mgr", ""]}) v1 -- 0x7f1c8c0ab6b0 con 0x7f1c6c01cae0 2021-02-10T11:21:10.315 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.321 7f1c88ff9700 1 -- 172.21.2.112:0/806439186 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f1c7c006e60 con 0x7f1c6c01cae0 2021-02-10T11:21:10.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.327 7f1c92614700 1 -- 172.21.2.112:0/806439186 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1c6c01cae0 msgr2=0x7f1c6c01efe0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:10.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.327 7f1c92614700 1 --2- 172.21.2.112:0/806439186 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1c6c01cae0 0x7f1c6c01efe0 crc :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:10.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.327 7f1c92614700 1 -- 172.21.2.112:0/806439186 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1c8c0c15b0 msgr2=0x7f1c8c0c19b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:10.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.327 7f1c92614700 1 --2- 172.21.2.112:0/806439186 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1c8c0c15b0 0x7f1c8c0c19b0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 rx=0x7f1c8000b5c0 tx=0x7f1c8000b960).stop 2021-02-10T11:21:10.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.328 7f1c92614700 1 -- 172.21.2.112:0/806439186 shutdown_connections 2021-02-10T11:21:10.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.328 7f1c92614700 1 --2- 172.21.2.112:0/806439186 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1c6c01cae0 0x7f1c6c01efe0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:10.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.328 7f1c92614700 1 --2- 172.21.2.112:0/806439186 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1c8c0aa4e0 0x7f1c8c0aa8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:10.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.328 7f1c92614700 1 --2- 172.21.2.112:0/806439186 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1c8c0c15b0 0x7f1c8c0c19b0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:10.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.328 7f1c92614700 1 --2- 172.21.2.112:0/806439186 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1c8c0c0b90 0x7f1c8c0c0f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:10.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.328 7f1c92614700 1 -- 172.21.2.112:0/806439186 >> 172.21.2.112:0/806439186 conn(0x7f1c8c0c40c0 msgr2=0x7f1c8c0a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:10.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.328 7f1c92614700 1 -- 172.21.2.112:0/806439186 shutdown_connections 2021-02-10T11:21:10.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.328 7f1c92614700 1 -- 172.21.2.112:0/806439186 wait complete. 2021-02-10T11:21:10.330 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000348605 2021-02-10T11:21:10.765 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.750 7fb29a29b700 1 Processor -- start 2021-02-10T11:21:10.765 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.750 7fb29a29b700 1 -- start start 2021-02-10T11:21:10.766 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.750 7fb29a29b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb2941b0870 0x7fb2940f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:10.766 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.750 7fb29a29b700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2941b0c70 con 0x7fb2941b0470 2021-02-10T11:21:10.766 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.750 7fb29a29b700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2940f0b50 con 0x7fb2940efe00 2021-02-10T11:21:10.766 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.750 7fb29a29b700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2940e5ea0 con 0x7fb2941b0870 2021-02-10T11:21:10.766 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.751 7fb293fff700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fb2941b0470 0x7fb2940ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:44304/0 (socket says 172.21.2.112:44304) 2021-02-10T11:21:10.766 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.751 7fb293fff700 1 -- 172.21.2.112:0/4112473485 learned_addr learned my addr 172.21.2.112:0/4112473485 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:21:10.766 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.751 7fb2937fe700 1 --2- 172.21.2.112:0/4112473485 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb2941b0870 0x7fb2940f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:10.766 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.751 7fb292ffd700 1 -- 172.21.2.112:0/4112473485 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 271435755 0 0) 0x7fb284000ce0 con 0x7fb2941b0470 2021-02-10T11:21:10.766 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.751 7fb292ffd700 1 -- 172.21.2.112:0/4112473485 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb27c002fe0 con 0x7fb2941b0470 2021-02-10T11:21:10.766 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.751 7fb292ffd700 1 -- 172.21.2.112:0/4112473485 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2564356136 0 0) 0x7fb28c000d30 con 0x7fb2940efe00 2021-02-10T11:21:10.766 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.751 7fb292ffd700 1 -- 172.21.2.112:0/4112473485 --> v1:172.21.2.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb27c004370 con 0x7fb2940efe00 2021-02-10T11:21:10.766 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.751 7fb2937fe700 1 -- 172.21.2.112:0/4112473485 >> v1:172.21.2.120:6789/0 conn(0x7fb2940efe00 legacy=0x7fb2940f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:10.767 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.751 7fb2937fe700 1 -- 172.21.2.112:0/4112473485 >> v1:172.21.2.112:6789/0 conn(0x7fb2941b0470 legacy=0x7fb2940ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:10.767 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.751 7fb2937fe700 1 -- 172.21.2.112:0/4112473485 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2940e52e0 con 0x7fb2941b0870 2021-02-10T11:21:10.767 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.751 7fb2937fe700 1 --2- 172.21.2.112:0/4112473485 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb2941b0870 0x7fb2940f20e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7fb288006650 tx=0x7fb288009a80).ready entity=mon.2 client_cookie=d86b8511942fda0a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:10.767 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.752 7fb292ffd700 1 -- 172.21.2.112:0/4112473485 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb2880009b0 con 0x7fb2941b0870 2021-02-10T11:21:10.767 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.752 7fb292ffd700 1 -- 172.21.2.112:0/4112473485 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb28800c0c0 con 0x7fb2941b0870 2021-02-10T11:21:10.767 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.752 7fb292ffd700 1 -- 172.21.2.112:0/4112473485 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb28800c5e0 con 0x7fb2941b0870 2021-02-10T11:21:10.767 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.752 7fb29a29b700 1 -- 172.21.2.112:0/4112473485 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb2941b0870 msgr2=0x7fb2940f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:10.767 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.752 7fb29a29b700 1 --2- 172.21.2.112:0/4112473485 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb2941b0870 0x7fb2940f20e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7fb288006650 tx=0x7fb288009a80).stop 2021-02-10T11:21:10.767 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.752 7fb29a29b700 1 -- 172.21.2.112:0/4112473485 shutdown_connections 2021-02-10T11:21:10.767 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.752 7fb29a29b700 1 --2- 172.21.2.112:0/4112473485 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb2941b0870 0x7fb2940f20e0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:10.767 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.752 7fb29a29b700 1 -- 172.21.2.112:0/4112473485 >> 172.21.2.112:0/4112473485 conn(0x7fb2940ce3e0 msgr2=0x7fb2940e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:10.767 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.752 7fb29a29b700 1 -- 172.21.2.112:0/4112473485 shutdown_connections 2021-02-10T11:21:10.767 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.752 7fb29a29b700 1 -- 172.21.2.112:0/4112473485 wait complete. 2021-02-10T11:21:10.767 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.767 7fb29a29b700 1 Processor -- start 2021-02-10T11:21:10.768 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.767 7fb29a29b700 1 -- start start 2021-02-10T11:21:10.768 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.767 7fb29a29b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb294190920 0x7fb2940b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:10.768 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.767 7fb29a29b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb2940b8200 0x7fb2940b8600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:10.768 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.767 7fb29a29b700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb2940c1d80 0x7fb2940c2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:10.768 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.767 7fb29a29b700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb294190d20 con 0x7fb2940b8200 2021-02-10T11:21:10.768 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.767 7fb29a29b700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2940b8c20 con 0x7fb2940c1d80 2021-02-10T11:21:10.768 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.767 7fb29a29b700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2940c27a0 con 0x7fb294190920 2021-02-10T11:21:10.768 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.768 7fb2937fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb2940b8200 0x7fb2940b8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:10.768 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.768 7fb2937fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb2940b8200 0x7fb2940b8600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:44850/0 (socket says 172.21.2.112:44850) 2021-02-10T11:21:10.768 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.768 7fb2937fe700 1 -- 172.21.2.112:0/778932896 learned_addr learned my addr 172.21.2.112:0/778932896 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:21:10.768 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.768 7fb2988b8700 1 --2- 172.21.2.112:0/778932896 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb2940c1d80 0x7fb2940c2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:10.769 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.768 7fb293fff700 1 --2- 172.21.2.112:0/778932896 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb294190920 0x7fb2940b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:10.769 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.768 7fb2937fe700 1 -- 172.21.2.112:0/778932896 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb294190920 msgr2=0x7fb2940b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:10.769 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.768 7fb2937fe700 1 --2- 172.21.2.112:0/778932896 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb294190920 0x7fb2940b7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:10.770 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.768 7fb2937fe700 1 -- 172.21.2.112:0/778932896 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb2940c1d80 msgr2=0x7fb2940c2180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:10.770 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.768 7fb2937fe700 1 --2- 172.21.2.112:0/778932896 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb2940c1d80 0x7fb2940c2180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:10.770 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.768 7fb2937fe700 1 -- 172.21.2.112:0/778932896 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2940c29c0 con 0x7fb2940b8200 2021-02-10T11:21:10.770 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.768 7fb293fff700 1 --2- 172.21.2.112:0/778932896 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb294190920 0x7fb2940b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:21:10.770 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.768 7fb2988b8700 1 --2- 172.21.2.112:0/778932896 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb2940c1d80 0x7fb2940c2180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:21:10.770 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.768 7fb2937fe700 1 --2- 172.21.2.112:0/778932896 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb2940b8200 0x7fb2940b8600 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 rx=0x7fb288009630 tx=0x7fb28800b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:10.770 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.768 7fb290ff9700 1 -- 172.21.2.112:0/778932896 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb2880093a0 con 0x7fb2940b8200 2021-02-10T11:21:10.770 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.768 7fb290ff9700 1 -- 172.21.2.112:0/778932896 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb28800bd70 con 0x7fb2940b8200 2021-02-10T11:21:10.770 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.768 7fb290ff9700 1 -- 172.21.2.112:0/778932896 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb2880054d0 con 0x7fb2940b8200 2021-02-10T11:21:10.770 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.768 7fb29a29b700 1 -- 172.21.2.112:0/778932896 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2940c29c0 con 0x7fb2940b8200 2021-02-10T11:21:10.770 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.768 7fb29a29b700 1 -- 172.21.2.112:0/778932896 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2940c2f50 con 0x7fb2940b8200 2021-02-10T11:21:10.770 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.769 7fb290ff9700 1 -- 172.21.2.112:0/778932896 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb28800a000 con 0x7fb2940b8200 2021-02-10T11:21:10.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.769 7fb290ff9700 1 --2- 172.21.2.112:0/778932896 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb2740186c0 0x7fb27401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:10.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.769 7fb290ff9700 1 -- 172.21.2.112:0/778932896 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fb288032420 con 0x7fb2940b8200 2021-02-10T11:21:10.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.769 7fb293fff700 1 --2- 172.21.2.112:0/778932896 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb2740186c0 0x7fb27401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:10.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.770 7fb293fff700 1 --2- 172.21.2.112:0/778932896 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb2740186c0 0x7fb27401abc0 crc :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:10.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.800 7fb29a29b700 1 -- 172.21.2.112:0/778932896 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb294105ae0 con 0x7fb2940b8200 2021-02-10T11:21:10.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.803 7fb290ff9700 1 -- 172.21.2.112:0/778932896 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb288027540 con 0x7fb2940b8200 2021-02-10T11:21:10.926 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.932 7fb29a29b700 1 -- 172.21.2.112:0/778932896 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348605", "target": ["mgr", ""]}) v1 -- 0x7fb294190d20 con 0x7fb2740186c0 2021-02-10T11:21:10.928 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.935 7fb290ff9700 1 -- 172.21.2.112:0/778932896 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7fb284006880 con 0x7fb2740186c0 2021-02-10T11:21:10.928 INFO:teuthology.orchestra.run.gibba012.stdout:/volumes/_nogroup/subvolume_0000000000348605/2bb29e3c-8297-461e-b7c7-c676876d14a4 2021-02-10T11:21:10.930 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.936 7fb29a29b700 1 -- 172.21.2.112:0/778932896 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb2740186c0 msgr2=0x7fb27401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:10.930 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.936 7fb29a29b700 1 --2- 172.21.2.112:0/778932896 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb2740186c0 0x7fb27401abc0 crc :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:10.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.936 7fb29a29b700 1 -- 172.21.2.112:0/778932896 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb2940b8200 msgr2=0x7fb2940b8600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:10.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.936 7fb29a29b700 1 --2- 172.21.2.112:0/778932896 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb2940b8200 0x7fb2940b8600 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 rx=0x7fb288009630 tx=0x7fb28800b980).stop 2021-02-10T11:21:10.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.936 7fb29a29b700 1 -- 172.21.2.112:0/778932896 shutdown_connections 2021-02-10T11:21:10.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.936 7fb29a29b700 1 --2- 172.21.2.112:0/778932896 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb2740186c0 0x7fb27401abc0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:10.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.936 7fb29a29b700 1 --2- 172.21.2.112:0/778932896 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb2940c1d80 0x7fb2940c2180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:10.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.936 7fb29a29b700 1 --2- 172.21.2.112:0/778932896 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb2940b8200 0x7fb2940b8600 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:10.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.936 7fb29a29b700 1 --2- 172.21.2.112:0/778932896 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb294190920 0x7fb2940b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:10.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.936 7fb29a29b700 1 -- 172.21.2.112:0/778932896 >> 172.21.2.112:0/778932896 conn(0x7fb294094e10 msgr2=0x7fb2940a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:10.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.937 7fb29a29b700 1 -- 172.21.2.112:0/778932896 shutdown_connections 2021-02-10T11:21:10.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:10.937 7fb29a29b700 1 -- 172.21.2.112:0/778932896 wait complete. 2021-02-10T11:21:10.938 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000348605 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000348605/2bb29e3c-8297-461e-b7c7-c676876d14a4 2021-02-10T11:21:10.939 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:21:10.939 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:21:10.939 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:21:11.275 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:21:11.275 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:21:11.275 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348605/2bb29e3c-8297-461e-b7c7-c676876d14a4/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:21:11.659 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:21:11.659 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:21:11.660 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0267128 s, 39.3 MB/s 2021-02-10T11:21:11.660 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:21:11.661 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:21:11.661 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:21:11.995 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:21:11.996 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:21:11.996 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348605/2bb29e3c-8297-461e-b7c7-c676876d14a4/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:21:12.371 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:21:12.372 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:21:12.372 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0244774 s, 42.8 MB/s 2021-02-10T11:21:12.373 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:21:12.373 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:21:12.373 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:21:12.707 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:21:12.707 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:21:12.707 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348605/2bb29e3c-8297-461e-b7c7-c676876d14a4/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:21:13.082 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:21:13.083 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:21:13.083 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.024652 s, 42.5 MB/s 2021-02-10T11:21:13.084 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:21:13.084 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:21:13.084 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:21:13.420 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:21:13.420 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:21:13.420 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348605/2bb29e3c-8297-461e-b7c7-c676876d14a4/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:21:13.794 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:21:13.795 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:21:13.795 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0231548 s, 45.3 MB/s 2021-02-10T11:21:13.796 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:21:13.796 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:21:13.796 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:21:14.132 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:21:14.132 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:21:14.132 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348605/2bb29e3c-8297-461e-b7c7-c676876d14a4/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:21:14.506 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:21:14.506 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:21:14.506 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0251899 s, 41.6 MB/s 2021-02-10T11:21:14.507 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:21:14.507 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:21:14.507 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:21:14.842 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:21:14.843 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:21:14.843 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348605/2bb29e3c-8297-461e-b7c7-c676876d14a4/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:21:15.218 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:21:15.219 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:21:15.219 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0240344 s, 43.6 MB/s 2021-02-10T11:21:15.220 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:21:15.220 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:21:15.220 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:21:15.556 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:21:15.556 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:21:15.556 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348605/2bb29e3c-8297-461e-b7c7-c676876d14a4/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:21:15.931 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:21:15.931 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:21:15.931 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0244528 s, 42.9 MB/s 2021-02-10T11:21:15.932 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:21:15.932 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:21:15.933 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:21:16.267 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:21:16.267 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:21:16.268 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348605/2bb29e3c-8297-461e-b7c7-c676876d14a4/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:21:16.643 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:21:16.643 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:21:16.643 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0241662 s, 43.4 MB/s 2021-02-10T11:21:16.644 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:21:16.644 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:21:16.644 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:21:16.980 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:21:16.980 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:21:16.980 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348605/2bb29e3c-8297-461e-b7c7-c676876d14a4/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:21:17.357 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:21:17.357 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:21:17.357 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.025646 s, 40.9 MB/s 2021-02-10T11:21:17.358 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:21:17.358 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:21:17.358 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:21:17.692 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:21:17.692 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:21:17.693 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348605/2bb29e3c-8297-461e-b7c7-c676876d14a4/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:21:18.067 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:21:18.067 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:21:18.067 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0246211 s, 42.6 MB/s 2021-02-10T11:21:18.068 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000348606 2021-02-10T11:21:18.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:18.487 7f66f6089700 1 Processor -- start 2021-02-10T11:21:18.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:18.487 7f66f6089700 1 -- start start 2021-02-10T11:21:18.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:18.487 7f66f6089700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f66f01b0870 0x7f66f00f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:18.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:18.487 7f66f6089700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66f01b0c70 con 0x7f66f01b0470 2021-02-10T11:21:18.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:18.487 7f66f6089700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66f00f0b50 con 0x7f66f00efe00 2021-02-10T11:21:18.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:18.487 7f66f6089700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66f00e5ea0 con 0x7f66f01b0870 2021-02-10T11:21:18.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:18.487 7f66ef7fe700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f66f01b0470 0x7f66f00ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:44318/0 (socket says 172.21.2.112:44318) 2021-02-10T11:21:18.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:18.488 7f66ef7fe700 1 -- 172.21.2.112:0/1036081997 learned_addr learned my addr 172.21.2.112:0/1036081997 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:21:18.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:18.488 7f66ee7fc700 1 -- 172.21.2.112:0/1036081997 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3734969026 0 0) 0x7f66d80010b0 con 0x7f66f01b0470 2021-02-10T11:21:18.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:18.488 7f66ee7fc700 1 -- 172.21.2.112:0/1036081997 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66dc002fe0 con 0x7f66f01b0470 2021-02-10T11:21:18.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:18.488 7f66eeffd700 1 --2- 172.21.2.112:0/1036081997 >> v2:172.21.2.120:3300/0 conn(0x7f66f01b0870 0x7f66f00f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:18.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:18.488 7f66ee7fc700 1 -- 172.21.2.112:0/1036081997 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2110854588 0 0) 0x7f66d80010b0 con 0x7f66f01b0470 2021-02-10T11:21:18.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:18.488 7f66ee7fc700 1 -- 172.21.2.112:0/1036081997 >> v2:172.21.2.120:3300/0 conn(0x7f66f01b0870 msgr2=0x7f66f00f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:21:18.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:18.488 7f66ee7fc700 1 --2- 172.21.2.112:0/1036081997 >> v2:172.21.2.120:3300/0 conn(0x7f66f01b0870 0x7f66f00f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:18.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:18.488 7f66ee7fc700 1 -- 172.21.2.112:0/1036081997 >> v1:172.21.2.120:6789/0 conn(0x7f66f00efe00 legacy=0x7f66f00f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:18.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:18.488 7f66ee7fc700 1 -- 172.21.2.112:0/1036081997 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66f00e52e0 con 0x7f66f01b0470 2021-02-10T11:21:18.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:18.488 7f66ee7fc700 1 -- 172.21.2.112:0/1036081997 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f66d80015b0 con 0x7f66f01b0470 2021-02-10T11:21:18.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:18.488 7f66ee7fc700 1 -- 172.21.2.112:0/1036081997 >> v1:172.21.2.112:6789/0 conn(0x7f66f01b0470 legacy=0x7f66f00ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:18.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:18.488 7f66ee7fc700 1 --2- 172.21.2.112:0/1036081997 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f66dc003b90 0x7f66dc0060b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:18.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:18.488 7f66ee7fc700 1 --2- 172.21.2.112:0/1036081997 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f66dc0068d0 0x7f66dc008df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:18.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:18.488 7f66ee7fc700 1 --2- 172.21.2.112:0/1036081997 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f66dc0096f0 0x7f66dc00bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:18.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:18.488 7f66ee7fc700 1 -- 172.21.2.112:0/1036081997 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66dc00c3c0 con 0x7f66dc0068d0 2021-02-10T11:21:18.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:18.488 7f66ee7fc700 1 -- 172.21.2.112:0/1036081997 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66dc00c5e0 con 0x7f66dc0096f0 2021-02-10T11:21:18.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:18.488 7f66ee7fc700 1 -- 172.21.2.112:0/1036081997 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66dc00c790 con 0x7f66dc003b90 2021-02-10T11:21:18.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:18.488 7f66eeffd700 1 --2- 172.21.2.112:0/1036081997 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f66dc0068d0 0x7f66dc008df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:18.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:18.489 7f66eeffd700 1 -- 172.21.2.112:0/1036081997 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f66dc003b90 msgr2=0x7f66dc0060b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:21:18.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:18.489 7f66eeffd700 1 --2- 172.21.2.112:0/1036081997 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f66dc003b90 0x7f66dc0060b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:18.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:18.489 7f66eeffd700 1 -- 172.21.2.112:0/1036081997 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f66dc0096f0 msgr2=0x7f66dc00bbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:21:18.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:18.489 7f66effff700 1 --2- 172.21.2.112:0/1036081997 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f66dc0096f0 0x7f66dc00bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:18.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:18.489 7f66eeffd700 1 --2- 172.21.2.112:0/1036081997 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f66dc0096f0 0x7f66dc00bbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:18.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:18.489 7f66eeffd700 1 -- 172.21.2.112:0/1036081997 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66dc050a00 con 0x7f66dc0068d0 2021-02-10T11:21:18.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:18.489 7f66effff700 1 -- 172.21.2.112:0/1036081997 reap_dead start 2021-02-10T11:21:18.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:18.489 7f66eeffd700 1 --2- 172.21.2.112:0/1036081997 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f66dc0068d0 0x7f66dc008df0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 rx=0x7f66e0000b50 tx=0x7f66e0006c20).ready entity=mon.0 client_cookie=124322fc224f653a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:18.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:18.489 7f66ee7fc700 1 -- 172.21.2.112:0/1036081997 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f66e00090d0 con 0x7f66dc0068d0 2021-02-10T11:21:18.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:18.489 7f66ee7fc700 1 -- 172.21.2.112:0/1036081997 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66e0009450 con 0x7f66dc0068d0 2021-02-10T11:21:18.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:18.489 7f66ee7fc700 1 -- 172.21.2.112:0/1036081997 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f66e0009930 con 0x7f66dc0068d0 2021-02-10T11:21:18.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:18.489 7f66f6089700 1 -- 172.21.2.112:0/1036081997 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f66dc0068d0 msgr2=0x7f66dc008df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:18.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:18.489 7f66f6089700 1 --2- 172.21.2.112:0/1036081997 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f66dc0068d0 0x7f66dc008df0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 rx=0x7f66e0000b50 tx=0x7f66e0006c20).stop 2021-02-10T11:21:18.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:18.489 7f66f6089700 1 -- 172.21.2.112:0/1036081997 shutdown_connections 2021-02-10T11:21:18.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:18.489 7f66f6089700 1 --2- 172.21.2.112:0/1036081997 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f66dc0068d0 0x7f66dc008df0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:18.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:18.489 7f66f6089700 1 -- 172.21.2.112:0/1036081997 >> 172.21.2.112:0/1036081997 conn(0x7f66f00ce3e0 msgr2=0x7f66f00e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:18.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:18.489 7f66f6089700 1 -- 172.21.2.112:0/1036081997 shutdown_connections 2021-02-10T11:21:18.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:18.489 7f66f6089700 1 -- 172.21.2.112:0/1036081997 wait complete. 2021-02-10T11:21:18.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:18.504 7f66f6089700 1 Processor -- start 2021-02-10T11:21:18.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:18.504 7f66f6089700 1 -- start start 2021-02-10T11:21:18.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:18.505 7f66f6089700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f66f009b930 0x7f66f00b7c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:18.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:18.505 7f66f6089700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f66f00b8410 0x7f66f00b8810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:18.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:18.505 7f66f6089700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f66f00c1f80 0x7f66f00c2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:18.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:18.505 7f66f6089700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66f00c1a00 con 0x7f66f00c1f80 2021-02-10T11:21:18.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:18.505 7f66f6089700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66f00b8e30 con 0x7f66f009b930 2021-02-10T11:21:18.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:18.505 7f66f6089700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66f00b9050 con 0x7f66f00b8410 2021-02-10T11:21:18.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:18.505 7f66effff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f66f00c1f80 0x7f66f00c2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:18.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:18.505 7f66effff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f66f00c1f80 0x7f66f00c2380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:44870/0 (socket says 172.21.2.112:44870) 2021-02-10T11:21:18.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:18.505 7f66effff700 1 -- 172.21.2.112:0/3898610707 learned_addr learned my addr 172.21.2.112:0/3898610707 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:21:18.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:18.505 7f66ef7fe700 1 --2- 172.21.2.112:0/3898610707 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f66f009b930 0x7f66f00b7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:18.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:18.505 7f66eeffd700 1 --2- 172.21.2.112:0/3898610707 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f66f00b8410 0x7f66f00b8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:18.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:18.505 7f66effff700 1 -- 172.21.2.112:0/3898610707 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f66f00b8410 msgr2=0x7f66f00b8810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:18.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:18.505 7f66effff700 1 --2- 172.21.2.112:0/3898610707 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f66f00b8410 0x7f66f00b8810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:18.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:18.505 7f66effff700 1 -- 172.21.2.112:0/3898610707 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f66f009b930 msgr2=0x7f66f00b7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:18.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:18.505 7f66effff700 1 --2- 172.21.2.112:0/3898610707 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f66f009b930 0x7f66f00b7c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:18.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:18.505 7f66effff700 1 -- 172.21.2.112:0/3898610707 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66f00c29a0 con 0x7f66f00c1f80 2021-02-10T11:21:18.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:18.505 7f66ef7fe700 1 --2- 172.21.2.112:0/3898610707 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f66f009b930 0x7f66f00b7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:21:18.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:18.506 7f66effff700 1 --2- 172.21.2.112:0/3898610707 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f66f00c1f80 0x7f66f00c2380 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 rx=0x7f66e8009980 tx=0x7f66e800bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:18.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:18.506 7f66e7fff700 1 -- 172.21.2.112:0/3898610707 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f66e80066f0 con 0x7f66f00c1f80 2021-02-10T11:21:18.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:18.506 7f66e7fff700 1 -- 172.21.2.112:0/3898610707 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66e800c2f0 con 0x7f66f00c1f80 2021-02-10T11:21:18.505 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:18.506 7f66e7fff700 1 -- 172.21.2.112:0/3898610707 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f66e800c740 con 0x7f66f00c1f80 2021-02-10T11:21:18.505 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:18.506 7f66f6089700 1 -- 172.21.2.112:0/3898610707 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66f00c29a0 con 0x7f66f00c1f80 2021-02-10T11:21:18.505 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:18.506 7f66f6089700 1 -- 172.21.2.112:0/3898610707 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66f00c29a0 con 0x7f66f00c1f80 2021-02-10T11:21:18.505 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:18.506 7f66e7fff700 1 -- 172.21.2.112:0/3898610707 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f66e8000dd0 con 0x7f66f00c1f80 2021-02-10T11:21:18.505 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:18.506 7f66e7fff700 1 --2- 172.21.2.112:0/3898610707 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f66d40186d0 0x7f66d401abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:18.505 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:18.506 7f66e7fff700 1 -- 172.21.2.112:0/3898610707 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f66e8032a40 con 0x7f66f00c1f80 2021-02-10T11:21:18.505 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:18.507 7f66ef7fe700 1 --2- 172.21.2.112:0/3898610707 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f66d40186d0 0x7f66d401abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:18.505 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:18.507 7f66ef7fe700 1 --2- 172.21.2.112:0/3898610707 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f66d40186d0 0x7f66d401abd0 crc :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:18.531 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:18.537 7f66f6089700 1 -- 172.21.2.112:0/3898610707 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66f0105b30 con 0x7f66f00c1f80 2021-02-10T11:21:18.533 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:18.540 7f66e7fff700 1 -- 172.21.2.112:0/3898610707 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f66e8002ef0 con 0x7f66f00c1f80 2021-02-10T11:21:18.662 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:18.668 7f66f6089700 1 -- 172.21.2.112:0/3898610707 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348606", "target": ["mgr", ""]}) v1 -- 0x7f66f0105b30 con 0x7f66d40186d0 2021-02-10T11:21:18.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:18.694 7f66e7fff700 1 -- 172.21.2.112:0/3898610707 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f66d8000ce0 con 0x7f66d40186d0 2021-02-10T11:21:18.695 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:18.701 7f66f6089700 1 -- 172.21.2.112:0/3898610707 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f66d40186d0 msgr2=0x7f66d401abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:18.695 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:18.701 7f66f6089700 1 --2- 172.21.2.112:0/3898610707 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f66d40186d0 0x7f66d401abd0 crc :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:18.695 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:18.701 7f66f6089700 1 -- 172.21.2.112:0/3898610707 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f66f00c1f80 msgr2=0x7f66f00c2380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:18.695 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:18.701 7f66f6089700 1 --2- 172.21.2.112:0/3898610707 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f66f00c1f80 0x7f66f00c2380 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 rx=0x7f66e8009980 tx=0x7f66e800bcd0).stop 2021-02-10T11:21:18.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:18.701 7f66f6089700 1 -- 172.21.2.112:0/3898610707 shutdown_connections 2021-02-10T11:21:18.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:18.701 7f66f6089700 1 --2- 172.21.2.112:0/3898610707 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f66d40186d0 0x7f66d401abd0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:18.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:18.701 7f66f6089700 1 --2- 172.21.2.112:0/3898610707 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f66f00c1f80 0x7f66f00c2380 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:18.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:18.701 7f66f6089700 1 --2- 172.21.2.112:0/3898610707 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f66f00b8410 0x7f66f00b8810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:18.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:18.701 7f66f6089700 1 --2- 172.21.2.112:0/3898610707 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f66f009b930 0x7f66f00b7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:18.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:18.701 7f66f6089700 1 -- 172.21.2.112:0/3898610707 >> 172.21.2.112:0/3898610707 conn(0x7f66f0094e10 msgr2=0x7f66f00a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:18.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:18.702 7f66f6089700 1 -- 172.21.2.112:0/3898610707 shutdown_connections 2021-02-10T11:21:18.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:18.702 7f66f6089700 1 -- 172.21.2.112:0/3898610707 wait complete. 2021-02-10T11:21:18.703 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000348606 2021-02-10T11:21:19.135 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:19.123 7f98f8209700 1 Processor -- start 2021-02-10T11:21:19.135 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:19.123 7f98f8209700 1 -- start start 2021-02-10T11:21:19.136 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:19.124 7f98f8209700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f98f01b0470 0x7f98f01b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:19.136 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:19.124 7f98f8209700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f98f00ef610 0x7f98f00efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:19.136 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:19.124 7f98f8209700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98f00f0b50 con 0x7f98f00f0030 2021-02-10T11:21:19.136 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:19.124 7f98f8209700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98f00e52e0 con 0x7f98f00ef610 2021-02-10T11:21:19.136 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:19.124 7f98f8209700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98f00e5490 con 0x7f98f01b0470 2021-02-10T11:21:19.136 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:19.124 7f98f6826700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f98f00f0030 0x7f98f00f10d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:44338/0 (socket says 172.21.2.112:44338) 2021-02-10T11:21:19.136 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:19.124 7f98f6826700 1 -- 172.21.2.112:0/2322231830 learned_addr learned my addr 172.21.2.112:0/2322231830 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:21:19.136 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:19.124 7f98f5023700 1 -- 172.21.2.112:0/2322231830 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3684709887 0 0) 0x7f98e8001500 con 0x7f98f00f0030 2021-02-10T11:21:19.136 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:19.124 7f98f5023700 1 -- 172.21.2.112:0/2322231830 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98dc002fe0 con 0x7f98f00f0030 2021-02-10T11:21:19.136 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:19.125 7f98f5023700 1 -- 172.21.2.112:0/2322231830 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 423950447 0 0) 0x7f98e8001460 con 0x7f98f00f0030 2021-02-10T11:21:19.136 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:19.125 7f98f5824700 1 --2- 172.21.2.112:0/2322231830 >> v2:172.21.2.120:3300/0 conn(0x7f98f00ef610 0x7f98f00efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:19.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:19.125 7f98f6025700 1 --2- 172.21.2.112:0/2322231830 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f98f01b0470 0x7f98f01b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:19.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:19.125 7f98f5023700 1 -- 172.21.2.112:0/2322231830 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f98f01b0470 msgr2=0x7f98f01b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:21:19.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:19.125 7f98f5023700 1 --2- 172.21.2.112:0/2322231830 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f98f01b0470 0x7f98f01b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:19.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:19.125 7f98f5023700 1 -- 172.21.2.112:0/2322231830 >> v2:172.21.2.120:3300/0 conn(0x7f98f00ef610 msgr2=0x7f98f00efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:21:19.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:19.125 7f98f5023700 1 --2- 172.21.2.112:0/2322231830 >> v2:172.21.2.120:3300/0 conn(0x7f98f00ef610 0x7f98f00efa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:19.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:19.125 7f98f5023700 1 -- 172.21.2.112:0/2322231830 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98f00f0430 con 0x7f98f00f0030 2021-02-10T11:21:19.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:19.125 7f98f5023700 1 -- 172.21.2.112:0/2322231830 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f98e8001980 con 0x7f98f00f0030 2021-02-10T11:21:19.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:19.125 7f98f5023700 1 -- 172.21.2.112:0/2322231830 >> v1:172.21.2.112:6789/0 conn(0x7f98f00f0030 legacy=0x7f98f00f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:19.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:19.125 7f98f5023700 1 --2- 172.21.2.112:0/2322231830 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f98dc003b90 0x7f98dc006020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:19.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:19.125 7f98f5023700 1 --2- 172.21.2.112:0/2322231830 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f98dc0068c0 0x7f98dc008db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:19.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:19.125 7f98f5023700 1 --2- 172.21.2.112:0/2322231830 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f98dc009650 0x7f98dc00bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:19.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:19.125 7f98f5023700 1 -- 172.21.2.112:0/2322231830 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98dc00c420 con 0x7f98dc003b90 2021-02-10T11:21:19.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:19.125 7f98f5023700 1 -- 172.21.2.112:0/2322231830 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98dc00c640 con 0x7f98dc009650 2021-02-10T11:21:19.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:19.125 7f98f5023700 1 -- 172.21.2.112:0/2322231830 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98dc00c860 con 0x7f98dc0068c0 2021-02-10T11:21:19.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:19.125 7f98f6025700 1 --2- 172.21.2.112:0/2322231830 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f98f01b0470 0x7f98f01b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:21:19.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:19.125 7f98f5824700 1 --2- 172.21.2.112:0/2322231830 >> v2:172.21.2.120:3300/0 conn(0x7f98f00ef610 0x7f98f00efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:21:19.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:19.125 7f98f6025700 1 --2- 172.21.2.112:0/2322231830 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f98dc003b90 0x7f98dc006020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:19.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:19.125 7f98f6025700 1 -- 172.21.2.112:0/2322231830 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f98dc0068c0 msgr2=0x7f98dc008db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:21:19.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:19.125 7f98f6025700 1 --2- 172.21.2.112:0/2322231830 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f98dc0068c0 0x7f98dc008db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:19.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:19.125 7f98f6025700 1 -- 172.21.2.112:0/2322231830 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f98dc009650 msgr2=0x7f98dc00bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:21:19.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:19.125 7f98f6025700 1 --2- 172.21.2.112:0/2322231830 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f98dc009650 0x7f98dc00bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:19.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:19.125 7f98f6025700 1 -- 172.21.2.112:0/2322231830 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98dc00cb00 con 0x7f98dc003b90 2021-02-10T11:21:19.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:19.125 7f98f6025700 1 --2- 172.21.2.112:0/2322231830 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f98dc003b90 0x7f98dc006020 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 rx=0x7f98ec002e00 tx=0x7f98ec04eab0).ready entity=mon.0 client_cookie=d1817e6cd323b9e6 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:19.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:19.126 7f98f5023700 1 -- 172.21.2.112:0/2322231830 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f98ec000c90 con 0x7f98dc003b90 2021-02-10T11:21:19.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:19.126 7f98f5023700 1 -- 172.21.2.112:0/2322231830 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98ec0510c0 con 0x7f98dc003b90 2021-02-10T11:21:19.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:19.126 7f98f5023700 1 -- 172.21.2.112:0/2322231830 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f98ec051600 con 0x7f98dc003b90 2021-02-10T11:21:19.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:19.126 7f98f8209700 1 -- 172.21.2.112:0/2322231830 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f98dc003b90 msgr2=0x7f98dc006020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:19.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:19.126 7f98f8209700 1 --2- 172.21.2.112:0/2322231830 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f98dc003b90 0x7f98dc006020 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 rx=0x7f98ec002e00 tx=0x7f98ec04eab0).stop 2021-02-10T11:21:19.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:19.126 7f98f6826700 1 -- 172.21.2.112:0/2322231830 reap_dead start 2021-02-10T11:21:19.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:19.127 7f98f8209700 1 -- 172.21.2.112:0/2322231830 shutdown_connections 2021-02-10T11:21:19.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:19.127 7f98f8209700 1 -- 172.21.2.112:0/2322231830 >> 172.21.2.112:0/2322231830 conn(0x7f98f00ce3e0 msgr2=0x7f98f00e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:19.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:19.127 7f98f8209700 1 -- 172.21.2.112:0/2322231830 shutdown_connections 2021-02-10T11:21:19.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:19.127 7f98f8209700 1 -- 172.21.2.112:0/2322231830 wait complete. 2021-02-10T11:21:19.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:19.141 7f98f8209700 1 Processor -- start 2021-02-10T11:21:19.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:19.141 7f98f8209700 1 -- start start 2021-02-10T11:21:19.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:19.142 7f98f8209700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f98f009b930 0x7f98f00b7c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:19.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:19.142 7f98f8209700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f98f00b8410 0x7f98f00b8810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:19.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:19.142 7f98f8209700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f98f00c1f80 0x7f98f00c2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:19.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:19.142 7f98f8209700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98f00c1a00 con 0x7f98f00b8410 2021-02-10T11:21:19.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:19.142 7f98f8209700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98f00b8e30 con 0x7f98f009b930 2021-02-10T11:21:19.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:19.142 7f98f8209700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98f00b9050 con 0x7f98f00c1f80 2021-02-10T11:21:19.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:19.142 7f98f5824700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f98f00b8410 0x7f98f00b8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:19.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:19.142 7f98f5824700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f98f00b8410 0x7f98f00b8810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:44886/0 (socket says 172.21.2.112:44886) 2021-02-10T11:21:19.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:19.142 7f98f5824700 1 -- 172.21.2.112:0/1256449087 learned_addr learned my addr 172.21.2.112:0/1256449087 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:21:19.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:19.142 7f98f5824700 1 -- 172.21.2.112:0/1256449087 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f98f00c1f80 msgr2=0x7f98f00c2380 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:21:19.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:19.142 7f98f5824700 1 --2- 172.21.2.112:0/1256449087 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f98f00c1f80 0x7f98f00c2380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:19.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:19.142 7f98f5824700 1 -- 172.21.2.112:0/1256449087 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f98f009b930 msgr2=0x7f98f00b7c20 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:21:19.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:19.142 7f98f5824700 1 --2- 172.21.2.112:0/1256449087 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f98f009b930 0x7f98f00b7c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:19.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:19.142 7f98f5824700 1 -- 172.21.2.112:0/1256449087 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98f00c29a0 con 0x7f98f00b8410 2021-02-10T11:21:19.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:19.142 7f98f5824700 1 --2- 172.21.2.112:0/1256449087 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f98f00b8410 0x7f98f00b8810 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 rx=0x7f98e400b5e0 tx=0x7f98e400b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:19.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:19.143 7f98e2ffd700 1 -- 172.21.2.112:0/1256449087 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f98e40065d0 con 0x7f98f00b8410 2021-02-10T11:21:19.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:19.143 7f98e2ffd700 1 -- 172.21.2.112:0/1256449087 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98e40052e0 con 0x7f98f00b8410 2021-02-10T11:21:19.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:19.143 7f98e2ffd700 1 -- 172.21.2.112:0/1256449087 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f98e4005800 con 0x7f98f00b8410 2021-02-10T11:21:19.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:19.143 7f98f8209700 1 -- 172.21.2.112:0/1256449087 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98f00c29a0 con 0x7f98f00b8410 2021-02-10T11:21:19.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:19.143 7f98f8209700 1 -- 172.21.2.112:0/1256449087 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98f00c2f30 con 0x7f98f00b8410 2021-02-10T11:21:19.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:19.143 7f98e2ffd700 1 -- 172.21.2.112:0/1256449087 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f98e40065d0 con 0x7f98f00b8410 2021-02-10T11:21:19.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:19.143 7f98e2ffd700 1 --2- 172.21.2.112:0/1256449087 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f98d80186c0 0x7f98d801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:19.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:19.143 7f98e2ffd700 1 -- 172.21.2.112:0/1256449087 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f98e40325e0 con 0x7f98f00b8410 2021-02-10T11:21:19.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:19.144 7f98f6025700 1 --2- 172.21.2.112:0/1256449087 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f98d80186c0 0x7f98d801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:19.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:19.144 7f98f6025700 1 --2- 172.21.2.112:0/1256449087 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f98d80186c0 0x7f98d801abc0 crc :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:19.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:19.174 7f98f8209700 1 -- 172.21.2.112:0/1256449087 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98f0105b30 con 0x7f98f00b8410 2021-02-10T11:21:19.170 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:19.177 7f98e2ffd700 1 -- 172.21.2.112:0/1256449087 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f98e4028150 con 0x7f98f00b8410 2021-02-10T11:21:19.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:19.305 7f98f8209700 1 -- 172.21.2.112:0/1256449087 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348606", "target": ["mgr", ""]}) v1 -- 0x7f98f00c2f30 con 0x7f98d80186c0 2021-02-10T11:21:19.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:19.308 7f98e2ffd700 1 -- 172.21.2.112:0/1256449087 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f98ec000940 con 0x7f98d80186c0 2021-02-10T11:21:19.302 INFO:teuthology.orchestra.run.gibba012.stdout:/volumes/_nogroup/subvolume_0000000000348606/a1d573bf-f632-4d78-86e7-3c7b73bbdf41 2021-02-10T11:21:19.303 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:19.309 7f98f8209700 1 -- 172.21.2.112:0/1256449087 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f98d80186c0 msgr2=0x7f98d801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:19.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:19.309 7f98f8209700 1 --2- 172.21.2.112:0/1256449087 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f98d80186c0 0x7f98d801abc0 crc :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:19.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:19.309 7f98f8209700 1 -- 172.21.2.112:0/1256449087 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f98f00b8410 msgr2=0x7f98f00b8810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:19.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:19.309 7f98f8209700 1 --2- 172.21.2.112:0/1256449087 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f98f00b8410 0x7f98f00b8810 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 rx=0x7f98e400b5e0 tx=0x7f98e400b980).stop 2021-02-10T11:21:19.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:19.310 7f98f8209700 1 -- 172.21.2.112:0/1256449087 shutdown_connections 2021-02-10T11:21:19.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:19.310 7f98f8209700 1 --2- 172.21.2.112:0/1256449087 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f98d80186c0 0x7f98d801abc0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:19.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:19.310 7f98f8209700 1 --2- 172.21.2.112:0/1256449087 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f98f00c1f80 0x7f98f00c2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:19.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:19.310 7f98f8209700 1 --2- 172.21.2.112:0/1256449087 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f98f00b8410 0x7f98f00b8810 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:19.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:19.310 7f98f8209700 1 --2- 172.21.2.112:0/1256449087 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f98f009b930 0x7f98f00b7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:19.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:19.310 7f98f8209700 1 -- 172.21.2.112:0/1256449087 >> 172.21.2.112:0/1256449087 conn(0x7f98f0094e10 msgr2=0x7f98f00a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:19.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:19.310 7f98f8209700 1 -- 172.21.2.112:0/1256449087 shutdown_connections 2021-02-10T11:21:19.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:19.310 7f98f8209700 1 -- 172.21.2.112:0/1256449087 wait complete. 2021-02-10T11:21:19.312 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000348606 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000348606/a1d573bf-f632-4d78-86e7-3c7b73bbdf41 2021-02-10T11:21:19.312 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:21:19.312 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:21:19.312 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:21:19.647 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:21:19.648 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:21:19.648 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348606/a1d573bf-f632-4d78-86e7-3c7b73bbdf41/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:21:20.028 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:21:20.028 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:21:20.028 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0261075 s, 40.2 MB/s 2021-02-10T11:21:20.029 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:21:20.029 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:21:20.029 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:21:20.364 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:21:20.365 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:21:20.365 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348606/a1d573bf-f632-4d78-86e7-3c7b73bbdf41/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:21:20.741 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:21:20.741 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:21:20.741 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0248522 s, 42.2 MB/s 2021-02-10T11:21:20.742 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:21:20.742 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:21:20.742 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:21:21.077 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:21:21.077 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:21:21.077 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348606/a1d573bf-f632-4d78-86e7-3c7b73bbdf41/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:21:21.454 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:21:21.454 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:21:21.454 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0260984 s, 40.2 MB/s 2021-02-10T11:21:21.455 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:21:21.455 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:21:21.455 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:21:21.789 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:21:21.789 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:21:21.790 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348606/a1d573bf-f632-4d78-86e7-3c7b73bbdf41/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:21:22.168 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:21:22.168 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:21:22.168 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0258763 s, 40.5 MB/s 2021-02-10T11:21:22.169 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:21:22.169 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:21:22.169 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:21:22.503 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:21:22.504 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:21:22.504 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348606/a1d573bf-f632-4d78-86e7-3c7b73bbdf41/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:21:22.875 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:21:22.875 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:21:22.876 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0223521 s, 46.9 MB/s 2021-02-10T11:21:22.877 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:21:22.877 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:21:22.877 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:21:23.212 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:21:23.213 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:21:23.213 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348606/a1d573bf-f632-4d78-86e7-3c7b73bbdf41/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:21:23.589 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:21:23.590 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:21:23.590 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0252372 s, 41.5 MB/s 2021-02-10T11:21:23.590 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:21:23.591 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:21:23.591 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:21:23.925 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:21:23.925 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:21:23.926 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348606/a1d573bf-f632-4d78-86e7-3c7b73bbdf41/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:21:24.300 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:21:24.300 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:21:24.301 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0232357 s, 45.1 MB/s 2021-02-10T11:21:24.302 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:21:24.302 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:21:24.303 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:21:24.636 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:21:24.637 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:21:24.637 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348606/a1d573bf-f632-4d78-86e7-3c7b73bbdf41/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:21:25.009 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:21:25.009 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:21:25.009 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0253315 s, 41.4 MB/s 2021-02-10T11:21:25.010 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:21:25.011 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:21:25.011 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:21:25.349 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:21:25.350 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:21:25.350 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348606/a1d573bf-f632-4d78-86e7-3c7b73bbdf41/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:21:25.727 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:21:25.727 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:21:25.727 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0252186 s, 41.6 MB/s 2021-02-10T11:21:25.728 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:21:25.728 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:21:25.728 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:21:26.063 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:21:26.063 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:21:26.063 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348606/a1d573bf-f632-4d78-86e7-3c7b73bbdf41/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:21:26.436 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:21:26.437 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:21:26.437 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0234585 s, 44.7 MB/s 2021-02-10T11:21:26.438 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000348607 2021-02-10T11:21:26.868 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:26.858 7fe2bdd7d700 1 Processor -- start 2021-02-10T11:21:26.868 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:26.858 7fe2bdd7d700 1 -- start start 2021-02-10T11:21:26.868 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:26.858 7fe2bdd7d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fe2b80f05f0 0x7fe2b80f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:26.868 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:26.858 7fe2bdd7d700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2b81b0c70 con 0x7fe2b81b0870 2021-02-10T11:21:26.868 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:26.858 7fe2bdd7d700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2b80f0b50 con 0x7fe2b81b0470 2021-02-10T11:21:26.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:26.858 7fe2bdd7d700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2b80e5ea0 con 0x7fe2b80f05f0 2021-02-10T11:21:26.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:26.858 7fe2b7fff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fe2b80f05f0 0x7fe2b80f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:26.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:26.859 7fe2b7fff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fe2b80f05f0 0x7fe2b80f20e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:44898/0 (socket says 172.21.2.112:44898) 2021-02-10T11:21:26.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:26.859 7fe2b7fff700 1 -- 172.21.2.112:0/268960900 learned_addr learned my addr 172.21.2.112:0/268960900 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:21:26.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:26.859 7fe2b6ffd700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fe2b81b0870 0x7fe2b80efe00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:44356/0 (socket says 172.21.2.112:44356) 2021-02-10T11:21:26.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:26.859 7fe2b7fff700 1 -- 172.21.2.112:0/268960900 >> v1:172.21.2.120:6789/0 conn(0x7fe2b81b0470 legacy=0x7fe2b80ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:26.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:26.859 7fe2b7fff700 1 -- 172.21.2.112:0/268960900 >> v1:172.21.2.112:6789/0 conn(0x7fe2b81b0870 legacy=0x7fe2b80efe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:26.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:26.859 7fe2b7fff700 1 -- 172.21.2.112:0/268960900 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2b80e52e0 con 0x7fe2b80f05f0 2021-02-10T11:21:26.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:26.859 7fe2b7fff700 1 --2- 172.21.2.112:0/268960900 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe2b80f05f0 0x7fe2b80f20e0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 rx=0x7fe2b0006f00 tx=0x7fe2b0009f10).ready entity=mon.0 client_cookie=45320d459d43b42a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:26.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:26.859 7fe2b67fc700 1 -- 172.21.2.112:0/268960900 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe2b000b3b0 con 0x7fe2b80f05f0 2021-02-10T11:21:26.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:26.859 7fe2b67fc700 1 -- 172.21.2.112:0/268960900 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2b000b630 con 0x7fe2b80f05f0 2021-02-10T11:21:26.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:26.860 7fe2b67fc700 1 -- 172.21.2.112:0/268960900 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe2b000bb50 con 0x7fe2b80f05f0 2021-02-10T11:21:26.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:26.860 7fe2bdd7d700 1 -- 172.21.2.112:0/268960900 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe2b80f05f0 msgr2=0x7fe2b80f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:26.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:26.860 7fe2bdd7d700 1 --2- 172.21.2.112:0/268960900 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe2b80f05f0 0x7fe2b80f20e0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 rx=0x7fe2b0006f00 tx=0x7fe2b0009f10).stop 2021-02-10T11:21:26.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:26.860 7fe2bdd7d700 1 -- 172.21.2.112:0/268960900 shutdown_connections 2021-02-10T11:21:26.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:26.860 7fe2bdd7d700 1 --2- 172.21.2.112:0/268960900 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe2b80f05f0 0x7fe2b80f20e0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:26.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:26.860 7fe2bdd7d700 1 -- 172.21.2.112:0/268960900 >> 172.21.2.112:0/268960900 conn(0x7fe2b80ce3e0 msgr2=0x7fe2b80e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:26.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:26.860 7fe2bdd7d700 1 -- 172.21.2.112:0/268960900 shutdown_connections 2021-02-10T11:21:26.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:26.860 7fe2bdd7d700 1 -- 172.21.2.112:0/268960900 wait complete. 2021-02-10T11:21:26.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:26.875 7fe2bdd7d700 1 Processor -- start 2021-02-10T11:21:26.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:26.875 7fe2bdd7d700 1 -- start start 2021-02-10T11:21:26.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:26.875 7fe2bdd7d700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe2b809c730 0x7fe2b8082d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:26.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:26.875 7fe2bdd7d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe2b8083370 0x7fe2b80a0c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:26.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:26.875 7fe2bdd7d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe2b809cb30 0x7fe2b80a1220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:26.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:26.875 7fe2bdd7d700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2b80a1840 con 0x7fe2b8083370 2021-02-10T11:21:26.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:26.875 7fe2bdd7d700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2b80a1a30 con 0x7fe2b809c730 2021-02-10T11:21:26.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:26.875 7fe2bdd7d700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2b80a1c50 con 0x7fe2b809cb30 2021-02-10T11:21:26.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:26.876 7fe2b6ffd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe2b8083370 0x7fe2b80a0c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:26.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:26.876 7fe2b6ffd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe2b8083370 0x7fe2b80a0c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:44900/0 (socket says 172.21.2.112:44900) 2021-02-10T11:21:26.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:26.876 7fe2b6ffd700 1 -- 172.21.2.112:0/4227824471 learned_addr learned my addr 172.21.2.112:0/4227824471 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:21:26.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:26.876 7fe2b7fff700 1 --2- 172.21.2.112:0/4227824471 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe2b809cb30 0x7fe2b80a1220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:26.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:26.876 7fe2b77fe700 1 --2- 172.21.2.112:0/4227824471 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe2b809c730 0x7fe2b8082d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:26.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:26.876 7fe2b6ffd700 1 -- 172.21.2.112:0/4227824471 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe2b809cb30 msgr2=0x7fe2b80a1220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:26.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:26.876 7fe2b6ffd700 1 --2- 172.21.2.112:0/4227824471 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe2b809cb30 0x7fe2b80a1220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:26.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:26.876 7fe2b6ffd700 1 -- 172.21.2.112:0/4227824471 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe2b809c730 msgr2=0x7fe2b8082d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:26.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:26.876 7fe2b6ffd700 1 --2- 172.21.2.112:0/4227824471 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe2b809c730 0x7fe2b8082d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:26.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:26.876 7fe2b6ffd700 1 -- 172.21.2.112:0/4227824471 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2b80818a0 con 0x7fe2b8083370 2021-02-10T11:21:26.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:26.876 7fe2b6ffd700 1 --2- 172.21.2.112:0/4227824471 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe2b8083370 0x7fe2b80a0c00 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 rx=0x7fe2ac0014d0 tx=0x7fe2ac00a8a0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:26.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:26.877 7fe29ffff700 1 -- 172.21.2.112:0/4227824471 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe2ac000ef0 con 0x7fe2b8083370 2021-02-10T11:21:26.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:26.877 7fe29ffff700 1 -- 172.21.2.112:0/4227824471 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2ac00ac90 con 0x7fe2b8083370 2021-02-10T11:21:26.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:26.877 7fe29ffff700 1 -- 172.21.2.112:0/4227824471 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe2ac004590 con 0x7fe2b8083370 2021-02-10T11:21:26.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:26.877 7fe2bdd7d700 1 -- 172.21.2.112:0/4227824471 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2b80818a0 con 0x7fe2b8083370 2021-02-10T11:21:26.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:26.877 7fe2bdd7d700 1 -- 172.21.2.112:0/4227824471 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2b8081e30 con 0x7fe2b8083370 2021-02-10T11:21:26.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:26.877 7fe29ffff700 1 -- 172.21.2.112:0/4227824471 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe2ac0045b0 con 0x7fe2b8083370 2021-02-10T11:21:26.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:26.877 7fe29ffff700 1 --2- 172.21.2.112:0/4227824471 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe2980186c0 0x7fe29801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:26.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:26.877 7fe29ffff700 1 -- 172.21.2.112:0/4227824471 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fe2ac032590 con 0x7fe2b8083370 2021-02-10T11:21:26.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:26.878 7fe2b77fe700 1 --2- 172.21.2.112:0/4227824471 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe2980186c0 0x7fe29801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:26.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:26.878 7fe2b77fe700 1 --2- 172.21.2.112:0/4227824471 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe2980186c0 0x7fe29801abc0 crc :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:26.902 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:26.908 7fe2bdd7d700 1 -- 172.21.2.112:0/4227824471 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2b80a2020 con 0x7fe2b8083370 2021-02-10T11:21:26.904 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:26.911 7fe29ffff700 1 -- 172.21.2.112:0/4227824471 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe2ac0170f0 con 0x7fe2b8083370 2021-02-10T11:21:27.031 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:27.037 7fe2bdd7d700 1 -- 172.21.2.112:0/4227824471 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348607", "target": ["mgr", ""]}) v1 -- 0x7fe2b80a2020 con 0x7fe2980186c0 2021-02-10T11:21:27.058 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:27.065 7fe29ffff700 1 -- 172.21.2.112:0/4227824471 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fe2a8006550 con 0x7fe2980186c0 2021-02-10T11:21:27.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:27.070 7fe2bdd7d700 1 -- 172.21.2.112:0/4227824471 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe2980186c0 msgr2=0x7fe29801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:27.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:27.070 7fe2bdd7d700 1 --2- 172.21.2.112:0/4227824471 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe2980186c0 0x7fe29801abc0 crc :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:27.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:27.070 7fe2bdd7d700 1 -- 172.21.2.112:0/4227824471 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe2b8083370 msgr2=0x7fe2b80a0c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:27.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:27.070 7fe2bdd7d700 1 --2- 172.21.2.112:0/4227824471 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe2b8083370 0x7fe2b80a0c00 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 rx=0x7fe2ac0014d0 tx=0x7fe2ac00a8a0).stop 2021-02-10T11:21:27.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:27.071 7fe2bdd7d700 1 -- 172.21.2.112:0/4227824471 shutdown_connections 2021-02-10T11:21:27.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:27.071 7fe2bdd7d700 1 --2- 172.21.2.112:0/4227824471 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe2980186c0 0x7fe29801abc0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:27.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:27.071 7fe2bdd7d700 1 --2- 172.21.2.112:0/4227824471 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe2b809cb30 0x7fe2b80a1220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:27.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:27.071 7fe2bdd7d700 1 --2- 172.21.2.112:0/4227824471 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe2b8083370 0x7fe2b80a0c00 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:27.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:27.071 7fe2bdd7d700 1 --2- 172.21.2.112:0/4227824471 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe2b809c730 0x7fe2b8082d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:27.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:27.071 7fe2bdd7d700 1 -- 172.21.2.112:0/4227824471 >> 172.21.2.112:0/4227824471 conn(0x7fe2b80a6930 msgr2=0x7fe2b80c1230 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:27.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:27.071 7fe2bdd7d700 1 -- 172.21.2.112:0/4227824471 shutdown_connections 2021-02-10T11:21:27.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:27.071 7fe2bdd7d700 1 -- 172.21.2.112:0/4227824471 wait complete. 2021-02-10T11:21:27.072 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000348607 2021-02-10T11:21:27.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:27.493 7f4cd96cb700 1 Processor -- start 2021-02-10T11:21:27.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:27.493 7f4cd96cb700 1 -- start start 2021-02-10T11:21:27.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:27.493 7f4cd96cb700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f4cd41b0870 0x7f4cd40f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:27.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:27.493 7f4cd96cb700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4cd40efe00 0x7f4cd40f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:27.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:27.493 7f4cd96cb700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cd41b0c70 con 0x7f4cd41b0470 2021-02-10T11:21:27.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:27.493 7f4cd96cb700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4cd40f0b50 con 0x7f4cd41b0870 2021-02-10T11:21:27.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:27.493 7f4cd96cb700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cd40f0820 con 0x7f4cd40efe00 2021-02-10T11:21:27.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:27.493 7f4cd2abe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f4cd41b0870 0x7f4cd40f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:27.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:27.493 7f4cd2abe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f4cd41b0870 0x7f4cd40f20e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:44908/0 (socket says 172.21.2.112:44908) 2021-02-10T11:21:27.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:27.493 7f4cd2abe700 1 -- 172.21.2.112:0/600909346 learned_addr learned my addr 172.21.2.112:0/600909346 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:21:27.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:27.494 7f4cd2abe700 1 -- 172.21.2.112:0/600909346 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4cd40efe00 msgr2=0x7f4cd40f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:21:27.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:27.494 7f4cd2abe700 1 --2- 172.21.2.112:0/600909346 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4cd40efe00 0x7f4cd40f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:27.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:27.494 7f4cd2abe700 1 -- 172.21.2.112:0/600909346 >> v1:172.21.2.120:6789/0 conn(0x7f4cd41b0470 legacy=0x7f4cd40ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:21:27.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:27.494 7f4cd2abe700 1 -- 172.21.2.112:0/600909346 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cd40e52e0 con 0x7f4cd41b0870 2021-02-10T11:21:27.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:27.494 7f4cd2abe700 1 --2- 172.21.2.112:0/600909346 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4cd41b0870 0x7f4cd40f20e0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 rx=0x7f4cbc006700 tx=0x7f4cbc009b80).ready entity=mon.0 client_cookie=a9f4ef1df72c3432 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:27.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:27.494 7f4cd22bd700 1 -- 172.21.2.112:0/600909346 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4cbc00c0c0 con 0x7f4cd41b0870 2021-02-10T11:21:27.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:27.494 7f4cd22bd700 1 -- 172.21.2.112:0/600909346 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cbc00c340 con 0x7f4cd41b0870 2021-02-10T11:21:27.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:27.494 7f4cd22bd700 1 -- 172.21.2.112:0/600909346 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4cbc00c860 con 0x7f4cd41b0870 2021-02-10T11:21:27.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:27.494 7f4cd96cb700 1 -- 172.21.2.112:0/600909346 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4cd41b0870 msgr2=0x7f4cd40f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:27.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:27.494 7f4cd96cb700 1 --2- 172.21.2.112:0/600909346 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4cd41b0870 0x7f4cd40f20e0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 rx=0x7f4cbc006700 tx=0x7f4cbc009b80).stop 2021-02-10T11:21:27.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:27.495 7f4cd96cb700 1 -- 172.21.2.112:0/600909346 shutdown_connections 2021-02-10T11:21:27.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:27.495 7f4cd96cb700 1 --2- 172.21.2.112:0/600909346 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4cd41b0870 0x7f4cd40f20e0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:27.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:27.495 7f4cd96cb700 1 --2- 172.21.2.112:0/600909346 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4cd40efe00 0x7f4cd40f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:27.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:27.495 7f4cd96cb700 1 -- 172.21.2.112:0/600909346 >> 172.21.2.112:0/600909346 conn(0x7f4cd40ce3e0 msgr2=0x7f4cd40e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:27.505 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:27.495 7f4cd96cb700 1 -- 172.21.2.112:0/600909346 shutdown_connections 2021-02-10T11:21:27.505 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:27.495 7f4cd96cb700 1 -- 172.21.2.112:0/600909346 wait complete. 2021-02-10T11:21:27.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:27.509 7f4cd96cb700 1 Processor -- start 2021-02-10T11:21:27.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:27.509 7f4cd96cb700 1 -- start start 2021-02-10T11:21:27.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:27.510 7f4cd96cb700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4cd4094fb0 0x7f4cd409e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:27.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:27.510 7f4cd96cb700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4cd40c2be0 0x7f4cd40c2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:27.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:27.510 7f4cd96cb700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4cd40c3600 0x7f4cd40a9c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:27.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:27.510 7f4cd96cb700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cd409ebf0 con 0x7f4cd4094fb0 2021-02-10T11:21:27.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:27.510 7f4cd96cb700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cd40aa2a0 con 0x7f4cd40c2be0 2021-02-10T11:21:27.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:27.510 7f4cd96cb700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cd40aa4c0 con 0x7f4cd40c3600 2021-02-10T11:21:27.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:27.510 7f4cd32bf700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4cd4094fb0 0x7f4cd409e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:27.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:27.510 7f4cd32bf700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4cd4094fb0 0x7f4cd409e5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:44914/0 (socket says 172.21.2.112:44914) 2021-02-10T11:21:27.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:27.510 7f4cd32bf700 1 -- 172.21.2.112:0/3253917131 learned_addr learned my addr 172.21.2.112:0/3253917131 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:21:27.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:27.510 7f4cd32bf700 1 -- 172.21.2.112:0/3253917131 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4cd40c3600 msgr2=0x7f4cd40a9c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:27.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:27.510 7f4cd32bf700 1 --2- 172.21.2.112:0/3253917131 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4cd40c3600 0x7f4cd40a9c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:27.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:27.510 7f4cd32bf700 1 -- 172.21.2.112:0/3253917131 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4cd40c2be0 msgr2=0x7f4cd40c2fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:27.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:27.510 7f4cd32bf700 1 --2- 172.21.2.112:0/3253917131 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4cd40c2be0 0x7f4cd40c2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:27.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:27.510 7f4cd32bf700 1 -- 172.21.2.112:0/3253917131 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cd40aa6e0 con 0x7f4cd4094fb0 2021-02-10T11:21:27.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:27.510 7f4cd32bf700 1 --2- 172.21.2.112:0/3253917131 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4cd4094fb0 0x7f4cd409e5d0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 rx=0x7f4cc400b5b0 tx=0x7f4cc400b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:27.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:27.510 7f4ccbfff700 1 -- 172.21.2.112:0/3253917131 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4cc400bd40 con 0x7f4cd4094fb0 2021-02-10T11:21:27.508 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:27.511 7f4ccbfff700 1 -- 172.21.2.112:0/3253917131 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cc4000e20 con 0x7f4cd4094fb0 2021-02-10T11:21:27.508 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:27.511 7f4ccbfff700 1 -- 172.21.2.112:0/3253917131 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4cc40054a0 con 0x7f4cd4094fb0 2021-02-10T11:21:27.508 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:27.511 7f4cd96cb700 1 -- 172.21.2.112:0/3253917131 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cd40aa6e0 con 0x7f4cd4094fb0 2021-02-10T11:21:27.508 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:27.511 7f4cd96cb700 1 -- 172.21.2.112:0/3253917131 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cd40aac70 con 0x7f4cd4094fb0 2021-02-10T11:21:27.508 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:27.511 7f4ccbfff700 1 -- 172.21.2.112:0/3253917131 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4cc4006030 con 0x7f4cd4094fb0 2021-02-10T11:21:27.508 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:27.511 7f4ccbfff700 1 --2- 172.21.2.112:0/3253917131 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4cb80186c0 0x7f4cb801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:27.508 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:27.511 7f4ccbfff700 1 -- 172.21.2.112:0/3253917131 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f4cc40323d0 con 0x7f4cd4094fb0 2021-02-10T11:21:27.508 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:27.511 7f4cd2abe700 1 --2- 172.21.2.112:0/3253917131 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4cb80186c0 0x7f4cb801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:27.509 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:27.512 7f4cd2abe700 1 --2- 172.21.2.112:0/3253917131 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4cb80186c0 0x7f4cb801abc0 crc :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:27.537 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:27.543 7f4cd96cb700 1 -- 172.21.2.112:0/3253917131 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cd40a9750 con 0x7f4cd4094fb0 2021-02-10T11:21:27.540 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:27.546 7f4ccbfff700 1 -- 172.21.2.112:0/3253917131 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4cc402c770 con 0x7f4cd4094fb0 2021-02-10T11:21:27.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:27.673 7f4cd96cb700 1 -- 172.21.2.112:0/3253917131 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348607", "target": ["mgr", ""]}) v1 -- 0x7f4cd40a9750 con 0x7f4cb80186c0 2021-02-10T11:21:27.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:27.678 7f4ccbfff700 1 -- 172.21.2.112:0/3253917131 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f4cbc000970 con 0x7f4cb80186c0 2021-02-10T11:21:27.675 INFO:teuthology.orchestra.run.gibba012.stdout:/volumes/_nogroup/subvolume_0000000000348607/79b17e1d-a5ae-45d3-82a4-cd67cd115d4c 2021-02-10T11:21:27.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:27.683 7f4cd96cb700 1 -- 172.21.2.112:0/3253917131 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4cb80186c0 msgr2=0x7f4cb801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:27.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:27.683 7f4cd96cb700 1 --2- 172.21.2.112:0/3253917131 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4cb80186c0 0x7f4cb801abc0 crc :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:27.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:27.683 7f4cd96cb700 1 -- 172.21.2.112:0/3253917131 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4cd4094fb0 msgr2=0x7f4cd409e5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:27.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:27.683 7f4cd96cb700 1 --2- 172.21.2.112:0/3253917131 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4cd4094fb0 0x7f4cd409e5d0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 rx=0x7f4cc400b5b0 tx=0x7f4cc400b950).stop 2021-02-10T11:21:27.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:27.683 7f4cd96cb700 1 -- 172.21.2.112:0/3253917131 shutdown_connections 2021-02-10T11:21:27.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:27.683 7f4cd96cb700 1 --2- 172.21.2.112:0/3253917131 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4cb80186c0 0x7f4cb801abc0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:27.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:27.683 7f4cd96cb700 1 --2- 172.21.2.112:0/3253917131 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4cd40c2be0 0x7f4cd40c2fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:27.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:27.683 7f4cd96cb700 1 --2- 172.21.2.112:0/3253917131 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4cd40c3600 0x7f4cd40a9c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:27.678 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:27.683 7f4cd96cb700 1 --2- 172.21.2.112:0/3253917131 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4cd4094fb0 0x7f4cd409e5d0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:27.678 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:27.683 7f4cd96cb700 1 -- 172.21.2.112:0/3253917131 >> 172.21.2.112:0/3253917131 conn(0x7f4cd409f630 msgr2=0x7f4cd40a0130 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:27.678 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:27.683 7f4cd96cb700 1 -- 172.21.2.112:0/3253917131 shutdown_connections 2021-02-10T11:21:27.678 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:27.683 7f4cd96cb700 1 -- 172.21.2.112:0/3253917131 wait complete. 2021-02-10T11:21:27.685 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000348607 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000348607/79b17e1d-a5ae-45d3-82a4-cd67cd115d4c 2021-02-10T11:21:27.685 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:21:27.685 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:21:27.685 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:21:28.021 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:21:28.021 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:21:28.021 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348607/79b17e1d-a5ae-45d3-82a4-cd67cd115d4c/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:21:28.402 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:21:28.403 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:21:28.403 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0237098 s, 44.2 MB/s 2021-02-10T11:21:28.404 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:21:28.404 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:21:28.404 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:21:28.739 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:21:28.740 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:21:28.740 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348607/79b17e1d-a5ae-45d3-82a4-cd67cd115d4c/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:21:29.113 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:21:29.113 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:21:29.113 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.022901 s, 45.8 MB/s 2021-02-10T11:21:29.114 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:21:29.114 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:21:29.114 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:21:29.449 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:21:29.450 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:21:29.450 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348607/79b17e1d-a5ae-45d3-82a4-cd67cd115d4c/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:21:29.849 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:21:29.849 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:21:29.849 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.025138 s, 41.7 MB/s 2021-02-10T11:21:29.850 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:21:29.850 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:21:29.850 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:21:30.188 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:21:30.189 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:21:30.189 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348607/79b17e1d-a5ae-45d3-82a4-cd67cd115d4c/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:21:30.567 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:21:30.567 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:21:30.567 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0245287 s, 42.7 MB/s 2021-02-10T11:21:30.568 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:21:30.568 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:21:30.568 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:21:30.906 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:21:30.906 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:21:30.907 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348607/79b17e1d-a5ae-45d3-82a4-cd67cd115d4c/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:21:31.284 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:21:31.284 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:21:31.284 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0234701 s, 44.7 MB/s 2021-02-10T11:21:31.285 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:21:31.285 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:21:31.285 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:21:31.620 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:21:31.621 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:21:31.621 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348607/79b17e1d-a5ae-45d3-82a4-cd67cd115d4c/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:21:31.994 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:21:31.994 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:21:31.994 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0233721 s, 44.9 MB/s 2021-02-10T11:21:31.996 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:21:31.996 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:21:31.996 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:21:32.331 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:21:32.332 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:21:32.332 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348607/79b17e1d-a5ae-45d3-82a4-cd67cd115d4c/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:21:32.707 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:21:32.707 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:21:32.707 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0240913 s, 43.5 MB/s 2021-02-10T11:21:32.708 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:21:32.708 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:21:32.708 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:21:33.043 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:21:33.044 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:21:33.044 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348607/79b17e1d-a5ae-45d3-82a4-cd67cd115d4c/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:21:33.419 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:21:33.420 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:21:33.420 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0247505 s, 42.4 MB/s 2021-02-10T11:21:33.420 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:21:33.421 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:21:33.421 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:21:33.755 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:21:33.755 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:21:33.756 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348607/79b17e1d-a5ae-45d3-82a4-cd67cd115d4c/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:21:34.130 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:21:34.130 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:21:34.130 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.023771 s, 44.1 MB/s 2021-02-10T11:21:34.131 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:21:34.132 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:21:34.132 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:21:34.466 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:21:34.466 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:21:34.466 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348607/79b17e1d-a5ae-45d3-82a4-cd67cd115d4c/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:21:34.837 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:21:34.837 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:21:34.837 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0239964 s, 43.7 MB/s 2021-02-10T11:21:34.838 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000348608 2021-02-10T11:21:35.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.259 7ff9c66b6700 1 Processor -- start 2021-02-10T11:21:35.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.259 7ff9c66b6700 1 -- start start 2021-02-10T11:21:35.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.260 7ff9c66b6700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7ff9c01b0870 0x7ff9c00f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:35.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.260 7ff9c66b6700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff9c00efe00 0x7ff9c00f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:35.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.260 7ff9c66b6700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9c01b0c70 con 0x7ff9c01b0470 2021-02-10T11:21:35.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.260 7ff9c66b6700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9c00f0b50 con 0x7ff9c01b0870 2021-02-10T11:21:35.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.260 7ff9c66b6700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9c00f0820 con 0x7ff9c00efe00 2021-02-10T11:21:35.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.260 7ff9bf7fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7ff9c01b0870 0x7ff9c00f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:35.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.260 7ff9bffff700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7ff9c01b0470 0x7ff9c00ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:44384/0 (socket says 172.21.2.112:44384) 2021-02-10T11:21:35.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.260 7ff9bffff700 1 -- 172.21.2.112:0/1269549524 learned_addr learned my addr 172.21.2.112:0/1269549524 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:21:35.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.261 7ff9bf7fe700 1 -- 172.21.2.112:0/1269549524 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff9c00efe00 msgr2=0x7ff9c00f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:21:35.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.261 7ff9bf7fe700 1 --2- 172.21.2.112:0/1269549524 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff9c00efe00 0x7ff9c00f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:35.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.261 7ff9bf7fe700 1 -- 172.21.2.112:0/1269549524 >> v1:172.21.2.112:6789/0 conn(0x7ff9c01b0470 legacy=0x7ff9c00ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:35.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.261 7ff9beffd700 1 -- 172.21.2.112:0/1269549524 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 977290221 0 0) 0x7ff9b0001180 con 0x7ff9c01b0470 2021-02-10T11:21:35.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.261 7ff9bf7fe700 1 -- 172.21.2.112:0/1269549524 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9c00e52e0 con 0x7ff9c01b0870 2021-02-10T11:21:35.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.261 7ff9bf7fe700 1 --2- 172.21.2.112:0/1269549524 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff9c01b0870 0x7ff9c00f20e0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 rx=0x7ff9b4006650 tx=0x7ff9b4009a80).ready entity=mon.0 client_cookie=3f91c3af774623bc server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:35.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.261 7ff9beffd700 1 -- 172.21.2.112:0/1269549524 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff9b40009b0 con 0x7ff9c01b0870 2021-02-10T11:21:35.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.261 7ff9beffd700 1 -- 172.21.2.112:0/1269549524 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9b400c0c0 con 0x7ff9c01b0870 2021-02-10T11:21:35.272 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.261 7ff9beffd700 1 -- 172.21.2.112:0/1269549524 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff9b400c520 con 0x7ff9c01b0870 2021-02-10T11:21:35.272 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.261 7ff9c66b6700 1 -- 172.21.2.112:0/1269549524 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff9c01b0870 msgr2=0x7ff9c00f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:35.272 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.261 7ff9c66b6700 1 --2- 172.21.2.112:0/1269549524 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff9c01b0870 0x7ff9c00f20e0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 rx=0x7ff9b4006650 tx=0x7ff9b4009a80).stop 2021-02-10T11:21:35.272 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.262 7ff9c66b6700 1 -- 172.21.2.112:0/1269549524 shutdown_connections 2021-02-10T11:21:35.272 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.262 7ff9c66b6700 1 --2- 172.21.2.112:0/1269549524 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff9c00efe00 0x7ff9c00f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:35.272 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.262 7ff9c66b6700 1 --2- 172.21.2.112:0/1269549524 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff9c01b0870 0x7ff9c00f20e0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:35.272 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.262 7ff9c66b6700 1 -- 172.21.2.112:0/1269549524 >> 172.21.2.112:0/1269549524 conn(0x7ff9c00ce3e0 msgr2=0x7ff9c00e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:35.272 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.262 7ff9c66b6700 1 -- 172.21.2.112:0/1269549524 shutdown_connections 2021-02-10T11:21:35.272 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.262 7ff9c66b6700 1 -- 172.21.2.112:0/1269549524 wait complete. 2021-02-10T11:21:35.273 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.276 7ff9c66b6700 1 Processor -- start 2021-02-10T11:21:35.273 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.277 7ff9c66b6700 1 -- start start 2021-02-10T11:21:35.274 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.277 7ff9c66b6700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff9c00b20b0 0x7ff9c00ac970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:35.274 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.277 7ff9c66b6700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff9c00b24b0 0x7ff9c0191dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:35.274 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.277 7ff9c66b6700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff9c00acf90 0x7ff9c01923e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:35.274 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.277 7ff9c66b6700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9c00b1b30 con 0x7ff9c00acf90 2021-02-10T11:21:35.274 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.277 7ff9c66b6700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9c01a04e0 con 0x7ff9c00b20b0 2021-02-10T11:21:35.274 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.277 7ff9c66b6700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9c01a06c0 con 0x7ff9c00b24b0 2021-02-10T11:21:35.274 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.277 7ff9c4cd3700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff9c00acf90 0x7ff9c01923e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:35.274 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.277 7ff9c4cd3700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff9c00acf90 0x7ff9c01923e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:44926/0 (socket says 172.21.2.112:44926) 2021-02-10T11:21:35.274 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.277 7ff9c4cd3700 1 -- 172.21.2.112:0/3989498705 learned_addr learned my addr 172.21.2.112:0/3989498705 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:21:35.274 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.278 7ff9c4cd3700 1 -- 172.21.2.112:0/3989498705 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff9c00b24b0 msgr2=0x7ff9c0191dc0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:21:35.274 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.278 7ff9c4cd3700 1 --2- 172.21.2.112:0/3989498705 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff9c00b24b0 0x7ff9c0191dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:35.274 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.278 7ff9c4cd3700 1 -- 172.21.2.112:0/3989498705 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff9c00b20b0 msgr2=0x7ff9c00ac970 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:21:35.275 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.278 7ff9c4cd3700 1 --2- 172.21.2.112:0/3989498705 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff9c00b20b0 0x7ff9c00ac970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:35.275 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.278 7ff9c4cd3700 1 -- 172.21.2.112:0/3989498705 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9c0192be0 con 0x7ff9c00acf90 2021-02-10T11:21:35.275 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.278 7ff9c4cd3700 1 --2- 172.21.2.112:0/3989498705 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff9c00acf90 0x7ff9c01923e0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 rx=0x7ff9b8009980 tx=0x7ff9b800bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:35.275 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.278 7ff9bcff9700 1 -- 172.21.2.112:0/3989498705 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff9b8005af0 con 0x7ff9c00acf90 2021-02-10T11:21:35.275 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.278 7ff9bcff9700 1 -- 172.21.2.112:0/3989498705 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9b8005d70 con 0x7ff9c00acf90 2021-02-10T11:21:35.275 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.278 7ff9c66b6700 1 -- 172.21.2.112:0/3989498705 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9c0192be0 con 0x7ff9c00acf90 2021-02-10T11:21:35.275 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.278 7ff9c66b6700 1 -- 172.21.2.112:0/3989498705 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9c0193170 con 0x7ff9c00acf90 2021-02-10T11:21:35.275 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.278 7ff9bcff9700 1 -- 172.21.2.112:0/3989498705 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff9b80062b0 con 0x7ff9c00acf90 2021-02-10T11:21:35.275 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.279 7ff9bcff9700 1 -- 172.21.2.112:0/3989498705 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff9b8006030 con 0x7ff9c00acf90 2021-02-10T11:21:35.275 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.279 7ff9bcff9700 1 --2- 172.21.2.112:0/3989498705 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff9a00186c0 0x7ff9a001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:35.275 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.279 7ff9bcff9700 1 -- 172.21.2.112:0/3989498705 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7ff9b8032b50 con 0x7ff9c00acf90 2021-02-10T11:21:35.275 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.279 7ff9bffff700 1 --2- 172.21.2.112:0/3989498705 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff9a00186c0 0x7ff9a001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:35.276 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.280 7ff9bffff700 1 --2- 172.21.2.112:0/3989498705 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff9a00186c0 0x7ff9a001abc0 crc :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:35.303 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.310 7ff9c66b6700 1 -- 172.21.2.112:0/3989498705 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9c0105af0 con 0x7ff9c00acf90 2021-02-10T11:21:35.306 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.312 7ff9bcff9700 1 -- 172.21.2.112:0/3989498705 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff9b8006570 con 0x7ff9c00acf90 2021-02-10T11:21:35.432 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.438 7ff9c66b6700 1 -- 172.21.2.112:0/3989498705 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348608", "target": ["mgr", ""]}) v1 -- 0x7ff9c00b1b30 con 0x7ff9a00186c0 2021-02-10T11:21:35.456 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.463 7ff9bcff9700 1 -- 172.21.2.112:0/3989498705 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7ff9b0001180 con 0x7ff9a00186c0 2021-02-10T11:21:35.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.471 7ff9c66b6700 1 -- 172.21.2.112:0/3989498705 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff9a00186c0 msgr2=0x7ff9a001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:35.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.471 7ff9c66b6700 1 --2- 172.21.2.112:0/3989498705 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff9a00186c0 0x7ff9a001abc0 crc :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:35.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.471 7ff9c66b6700 1 -- 172.21.2.112:0/3989498705 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff9c00acf90 msgr2=0x7ff9c01923e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:35.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.471 7ff9c66b6700 1 --2- 172.21.2.112:0/3989498705 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff9c00acf90 0x7ff9c01923e0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 rx=0x7ff9b8009980 tx=0x7ff9b800bcd0).stop 2021-02-10T11:21:35.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.471 7ff9c66b6700 1 -- 172.21.2.112:0/3989498705 shutdown_connections 2021-02-10T11:21:35.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.471 7ff9c66b6700 1 --2- 172.21.2.112:0/3989498705 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff9a00186c0 0x7ff9a001abc0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:35.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.471 7ff9c66b6700 1 --2- 172.21.2.112:0/3989498705 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff9c00acf90 0x7ff9c01923e0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:35.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.471 7ff9c66b6700 1 --2- 172.21.2.112:0/3989498705 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff9c00b24b0 0x7ff9c0191dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:35.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.471 7ff9c66b6700 1 --2- 172.21.2.112:0/3989498705 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff9c00b20b0 0x7ff9c00ac970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:35.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.471 7ff9c66b6700 1 -- 172.21.2.112:0/3989498705 >> 172.21.2.112:0/3989498705 conn(0x7ff9c00bf4e0 msgr2=0x7ff9c00be3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:35.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.471 7ff9c66b6700 1 -- 172.21.2.112:0/3989498705 shutdown_connections 2021-02-10T11:21:35.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.471 7ff9c66b6700 1 -- 172.21.2.112:0/3989498705 wait complete. 2021-02-10T11:21:35.473 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000348608 2021-02-10T11:21:35.902 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.892 7fa5c5ef8700 1 Processor -- start 2021-02-10T11:21:35.902 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.892 7fa5c5ef8700 1 -- start start 2021-02-10T11:21:35.902 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.893 7fa5c5ef8700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fa5c01b0470 0x7fa5c01b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:35.903 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.893 7fa5c5ef8700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fa5c00f20e0 0x7fa5c00f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:35.903 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.893 7fa5c5ef8700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5c00f0b50 con 0x7fa5c00ef610 2021-02-10T11:21:35.903 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.893 7fa5c5ef8700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5c00f0820 con 0x7fa5c00f20e0 2021-02-10T11:21:35.903 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.893 7fa5c5ef8700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5c00f24e0 con 0x7fa5c01b0470 2021-02-10T11:21:35.903 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.893 7fa5bffff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fa5c00f20e0 0x7fa5c00f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:35.903 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.893 7fa5bffff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fa5c00f20e0 0x7fa5c00f0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:44934/0 (socket says 172.21.2.112:44934) 2021-02-10T11:21:35.903 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.893 7fa5bffff700 1 -- 172.21.2.112:0/2601996682 learned_addr learned my addr 172.21.2.112:0/2601996682 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:21:35.903 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.893 7fa5bf7fe700 1 --2- 172.21.2.112:0/2601996682 >> v2:172.21.2.120:3300/0 conn(0x7fa5c01b0470 0x7fa5c01b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:35.903 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.893 7fa5bffff700 1 -- 172.21.2.112:0/2601996682 >> v2:172.21.2.120:3300/0 conn(0x7fa5c01b0470 msgr2=0x7fa5c01b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:21:35.903 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.893 7fa5bffff700 1 --2- 172.21.2.112:0/2601996682 >> v2:172.21.2.120:3300/0 conn(0x7fa5c01b0470 0x7fa5c01b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:35.903 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.893 7fa5bffff700 1 -- 172.21.2.112:0/2601996682 >> v1:172.21.2.112:6789/0 conn(0x7fa5c00ef610 legacy=0x7fa5c00efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:35.903 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.893 7fa5bffff700 1 -- 172.21.2.112:0/2601996682 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5c00e52e0 con 0x7fa5c00f20e0 2021-02-10T11:21:35.904 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.894 7fa5bffff700 1 --2- 172.21.2.112:0/2601996682 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa5c00f20e0 0x7fa5c00f0200 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 rx=0x7fa5b8006f00 tx=0x7fa5b8009f10).ready entity=mon.0 client_cookie=7587d08b78fb6ba0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:35.904 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.894 7fa5be7fc700 1 -- 172.21.2.112:0/2601996682 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa5b800b3b0 con 0x7fa5c00f20e0 2021-02-10T11:21:35.904 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.894 7fa5be7fc700 1 -- 172.21.2.112:0/2601996682 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5b800b630 con 0x7fa5c00f20e0 2021-02-10T11:21:35.904 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.894 7fa5be7fc700 1 -- 172.21.2.112:0/2601996682 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa5b800bb50 con 0x7fa5c00f20e0 2021-02-10T11:21:35.904 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.894 7fa5c5ef8700 1 -- 172.21.2.112:0/2601996682 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa5c00f20e0 msgr2=0x7fa5c00f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:35.904 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.894 7fa5c5ef8700 1 --2- 172.21.2.112:0/2601996682 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa5c00f20e0 0x7fa5c00f0200 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 rx=0x7fa5b8006f00 tx=0x7fa5b8009f10).stop 2021-02-10T11:21:35.904 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.894 7fa5c5ef8700 1 -- 172.21.2.112:0/2601996682 shutdown_connections 2021-02-10T11:21:35.904 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.894 7fa5c5ef8700 1 --2- 172.21.2.112:0/2601996682 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa5c00f20e0 0x7fa5c00f0200 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:35.904 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.894 7fa5c5ef8700 1 --2- 172.21.2.112:0/2601996682 >> v2:172.21.2.120:3300/0 conn(0x7fa5c01b0470 0x7fa5c01b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:35.904 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.894 7fa5c5ef8700 1 -- 172.21.2.112:0/2601996682 >> 172.21.2.112:0/2601996682 conn(0x7fa5c00ce3e0 msgr2=0x7fa5c00e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:35.904 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.894 7fa5c5ef8700 1 -- 172.21.2.112:0/2601996682 shutdown_connections 2021-02-10T11:21:35.904 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.894 7fa5c5ef8700 1 -- 172.21.2.112:0/2601996682 wait complete. 2021-02-10T11:21:35.906 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.909 7fa5c5ef8700 1 Processor -- start 2021-02-10T11:21:35.906 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.909 7fa5c5ef8700 1 -- start start 2021-02-10T11:21:35.906 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.910 7fa5c5ef8700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa5c0190920 0x7fa5c00b81a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:35.906 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.910 7fa5c5ef8700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa5c00b87c0 0x7fa5c00b70c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:35.906 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.910 7fa5c5ef8700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa5c00b8bc0 0x7fa5c00b76e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:35.906 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.910 7fa5c5ef8700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5c0190d20 con 0x7fa5c0190920 2021-02-10T11:21:35.906 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.910 7fa5c5ef8700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5c00b5ad0 con 0x7fa5c00b8bc0 2021-02-10T11:21:35.906 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.910 7fa5c5ef8700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5c00b7d00 con 0x7fa5c00b87c0 2021-02-10T11:21:35.907 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.910 7fa5bf7fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa5c0190920 0x7fa5c00b81a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:35.907 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.910 7fa5bf7fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa5c0190920 0x7fa5c00b81a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:44940/0 (socket says 172.21.2.112:44940) 2021-02-10T11:21:35.907 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.910 7fa5bf7fe700 1 -- 172.21.2.112:0/1496719219 learned_addr learned my addr 172.21.2.112:0/1496719219 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:21:35.907 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.910 7fa5bf7fe700 1 -- 172.21.2.112:0/1496719219 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa5c00b87c0 msgr2=0x7fa5c00b70c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:35.907 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.910 7fa5bf7fe700 1 --2- 172.21.2.112:0/1496719219 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa5c00b87c0 0x7fa5c00b70c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:35.907 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.910 7fa5bf7fe700 1 -- 172.21.2.112:0/1496719219 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa5c00b8bc0 msgr2=0x7fa5c00b76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:35.907 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.910 7fa5bf7fe700 1 --2- 172.21.2.112:0/1496719219 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa5c00b8bc0 0x7fa5c00b76e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:35.907 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.910 7fa5bf7fe700 1 -- 172.21.2.112:0/1496719219 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5c00a4590 con 0x7fa5c0190920 2021-02-10T11:21:35.907 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.910 7fa5bf7fe700 1 --2- 172.21.2.112:0/1496719219 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa5c0190920 0x7fa5c00b81a0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 rx=0x7fa5a8006f60 tx=0x7fa5a800b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:35.907 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.910 7fa5b7fff700 1 -- 172.21.2.112:0/1496719219 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa5a80009b0 con 0x7fa5c0190920 2021-02-10T11:21:35.907 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.910 7fa5b7fff700 1 -- 172.21.2.112:0/1496719219 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5a800bdf0 con 0x7fa5c0190920 2021-02-10T11:21:35.907 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.910 7fa5c5ef8700 1 -- 172.21.2.112:0/1496719219 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5c00a4590 con 0x7fa5c0190920 2021-02-10T11:21:35.907 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.910 7fa5c5ef8700 1 -- 172.21.2.112:0/1496719219 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5c00a4590 con 0x7fa5c0190920 2021-02-10T11:21:35.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.910 7fa5b7fff700 1 -- 172.21.2.112:0/1496719219 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa5a8002030 con 0x7fa5c0190920 2021-02-10T11:21:35.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.911 7fa5b7fff700 1 -- 172.21.2.112:0/1496719219 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa5a8000cf0 con 0x7fa5c0190920 2021-02-10T11:21:35.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.911 7fa5b7fff700 1 --2- 172.21.2.112:0/1496719219 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa5a4018730 0x7fa5a401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:35.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.911 7fa5b7fff700 1 -- 172.21.2.112:0/1496719219 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fa5a8032820 con 0x7fa5c0190920 2021-02-10T11:21:35.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.912 7fa5beffd700 1 --2- 172.21.2.112:0/1496719219 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa5a4018730 0x7fa5a401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:35.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.912 7fa5beffd700 1 --2- 172.21.2.112:0/1496719219 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa5a4018730 0x7fa5a401ac10 crc :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:35.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.943 7fa5c5ef8700 1 -- 172.21.2.112:0/1496719219 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5c0105ae0 con 0x7fa5c0190920 2021-02-10T11:21:35.939 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:35.946 7fa5b7fff700 1 -- 172.21.2.112:0/1496719219 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa5a8006b00 con 0x7fa5c0190920 2021-02-10T11:21:36.069 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:36.075 7fa5c5ef8700 1 -- 172.21.2.112:0/1496719219 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348608", "target": ["mgr", ""]}) v1 -- 0x7fa5c00f0b50 con 0x7fa5a4018730 2021-02-10T11:21:36.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:36.079 7fa5b7fff700 1 -- 172.21.2.112:0/1496719219 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7fa5b0006e40 con 0x7fa5a4018730 2021-02-10T11:21:36.076 INFO:teuthology.orchestra.run.gibba012.stdout:/volumes/_nogroup/subvolume_0000000000348608/1b4f3a4e-6cda-450c-973f-d6144bf80085 2021-02-10T11:21:36.079 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:36.084 7fa5c5ef8700 1 -- 172.21.2.112:0/1496719219 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa5a4018730 msgr2=0x7fa5a401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:36.079 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:36.084 7fa5c5ef8700 1 --2- 172.21.2.112:0/1496719219 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa5a4018730 0x7fa5a401ac10 crc :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:36.079 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:36.084 7fa5c5ef8700 1 -- 172.21.2.112:0/1496719219 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa5c0190920 msgr2=0x7fa5c00b81a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:36.079 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:36.084 7fa5c5ef8700 1 --2- 172.21.2.112:0/1496719219 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa5c0190920 0x7fa5c00b81a0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 rx=0x7fa5a8006f60 tx=0x7fa5a800b970).stop 2021-02-10T11:21:36.079 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:36.085 7fa5c5ef8700 1 -- 172.21.2.112:0/1496719219 shutdown_connections 2021-02-10T11:21:36.080 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:36.085 7fa5c5ef8700 1 --2- 172.21.2.112:0/1496719219 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa5a4018730 0x7fa5a401ac10 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:36.080 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:36.085 7fa5c5ef8700 1 --2- 172.21.2.112:0/1496719219 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa5c00b8bc0 0x7fa5c00b76e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:36.080 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:36.085 7fa5c5ef8700 1 --2- 172.21.2.112:0/1496719219 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa5c00b87c0 0x7fa5c00b70c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:36.080 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:36.085 7fa5c5ef8700 1 --2- 172.21.2.112:0/1496719219 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa5c0190920 0x7fa5c00b81a0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:36.080 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:36.085 7fa5c5ef8700 1 -- 172.21.2.112:0/1496719219 >> 172.21.2.112:0/1496719219 conn(0x7fa5c008ffe0 msgr2=0x7fa5c00bf050 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:36.080 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:36.085 7fa5c5ef8700 1 -- 172.21.2.112:0/1496719219 shutdown_connections 2021-02-10T11:21:36.080 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:36.085 7fa5c5ef8700 1 -- 172.21.2.112:0/1496719219 wait complete. 2021-02-10T11:21:36.086 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000348608 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000348608/1b4f3a4e-6cda-450c-973f-d6144bf80085 2021-02-10T11:21:36.086 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:21:36.086 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:21:36.086 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:21:36.422 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:21:36.423 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:21:36.423 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348608/1b4f3a4e-6cda-450c-973f-d6144bf80085/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:21:36.804 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:21:36.805 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:21:36.805 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0252978 s, 41.4 MB/s 2021-02-10T11:21:36.806 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:21:36.806 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:21:36.806 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:21:36.807 DEBUG:teuthology.orchestra.run.gibba012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:21:36.809 DEBUG:teuthology.orchestra.run.gibba020:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:21:37.143 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:21:37.143 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:21:37.143 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348608/1b4f3a4e-6cda-450c-973f-d6144bf80085/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:21:37.520 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:21:37.520 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:21:37.520 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0253598 s, 41.3 MB/s 2021-02-10T11:21:37.521 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:21:37.521 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:21:37.522 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:21:37.856 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:21:37.857 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:21:37.857 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348608/1b4f3a4e-6cda-450c-973f-d6144bf80085/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:21:38.233 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:21:38.233 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:21:38.234 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0230177 s, 45.6 MB/s 2021-02-10T11:21:38.234 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:21:38.235 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:21:38.235 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:21:38.569 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:21:38.569 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:21:38.570 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348608/1b4f3a4e-6cda-450c-973f-d6144bf80085/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:21:38.943 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:21:38.943 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:21:38.943 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0226534 s, 46.3 MB/s 2021-02-10T11:21:38.944 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:21:38.945 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:21:38.945 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:21:39.280 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:21:39.281 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:21:39.281 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348608/1b4f3a4e-6cda-450c-973f-d6144bf80085/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:21:39.654 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:21:39.654 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:21:39.655 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0236038 s, 44.4 MB/s 2021-02-10T11:21:39.655 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:21:39.656 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:21:39.656 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:21:39.991 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:21:39.992 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:21:39.992 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348608/1b4f3a4e-6cda-450c-973f-d6144bf80085/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:21:40.367 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:21:40.368 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:21:40.368 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0244322 s, 42.9 MB/s 2021-02-10T11:21:40.369 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:21:40.369 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:21:40.370 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:21:40.705 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:21:40.705 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:21:40.705 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348608/1b4f3a4e-6cda-450c-973f-d6144bf80085/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:21:41.080 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:21:41.080 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:21:41.080 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0234729 s, 44.7 MB/s 2021-02-10T11:21:41.081 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:21:41.081 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:21:41.081 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:21:41.417 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:21:41.418 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:21:41.418 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348608/1b4f3a4e-6cda-450c-973f-d6144bf80085/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:21:41.795 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:21:41.795 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:21:41.795 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0256596 s, 40.9 MB/s 2021-02-10T11:21:41.796 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:21:41.797 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:21:41.797 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:21:42.133 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:21:42.133 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:21:42.133 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348608/1b4f3a4e-6cda-450c-973f-d6144bf80085/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:21:42.507 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:21:42.508 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:21:42.509 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0247314 s, 42.4 MB/s 2021-02-10T11:21:42.510 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:21:42.510 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:21:42.510 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:21:42.845 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:21:42.845 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:21:42.845 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348608/1b4f3a4e-6cda-450c-973f-d6144bf80085/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:21:43.219 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:21:43.220 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:21:43.220 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0237377 s, 44.2 MB/s 2021-02-10T11:21:43.221 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000348609 2021-02-10T11:21:43.652 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:43.642 7fc126e9e700 1 Processor -- start 2021-02-10T11:21:43.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:43.642 7fc126e9e700 1 -- start start 2021-02-10T11:21:43.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:43.643 7fc126e9e700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fc1201b0470 0x7fc1201b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:43.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:43.643 7fc126e9e700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc1200f20e0 0x7fc1200f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:43.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:43.643 7fc126e9e700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1200f0b50 con 0x7fc1200ef610 2021-02-10T11:21:43.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:43.643 7fc126e9e700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1200f0820 con 0x7fc1201b0470 2021-02-10T11:21:43.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:43.643 7fc126e9e700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1200f24e0 con 0x7fc1200f20e0 2021-02-10T11:21:43.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:43.643 7fc124cba700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fc1201b0470 0x7fc1201b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:43.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:43.643 7fc1254bb700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc1200f20e0 0x7fc1200f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:43.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:43.643 7fc11ffff700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fc1200ef610 0x7fc1200efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:44408/0 (socket says 172.21.2.112:44408) 2021-02-10T11:21:43.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:43.643 7fc11ffff700 1 -- 172.21.2.112:0/4035689220 learned_addr learned my addr 172.21.2.112:0/4035689220 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:21:43.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:43.644 7fc124cba700 1 -- 172.21.2.112:0/4035689220 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc1200f20e0 msgr2=0x7fc1200f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:21:43.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:43.644 7fc124cba700 1 --2- 172.21.2.112:0/4035689220 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc1200f20e0 0x7fc1200f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:43.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:43.644 7fc124cba700 1 -- 172.21.2.112:0/4035689220 >> v1:172.21.2.112:6789/0 conn(0x7fc1200ef610 legacy=0x7fc1200efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:43.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:43.644 7fc124cba700 1 -- 172.21.2.112:0/4035689220 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1200e52e0 con 0x7fc1201b0470 2021-02-10T11:21:43.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:43.644 7fc1254bb700 1 --2- 172.21.2.112:0/4035689220 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc1200f20e0 0x7fc1200f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:21:43.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:43.644 7fc124cba700 1 --2- 172.21.2.112:0/4035689220 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc1201b0470 0x7fc1201b0870 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 rx=0x7fc108006650 tx=0x7fc108009a80).ready entity=mon.0 client_cookie=4192b22685927c7d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:43.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:43.644 7fc11f7fe700 1 -- 172.21.2.112:0/4035689220 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc10800c0c0 con 0x7fc1201b0470 2021-02-10T11:21:43.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:43.644 7fc11f7fe700 1 -- 172.21.2.112:0/4035689220 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc10800c340 con 0x7fc1201b0470 2021-02-10T11:21:43.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:43.644 7fc11f7fe700 1 -- 172.21.2.112:0/4035689220 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc10800c860 con 0x7fc1201b0470 2021-02-10T11:21:43.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:43.644 7fc126e9e700 1 -- 172.21.2.112:0/4035689220 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc1201b0470 msgr2=0x7fc1201b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:43.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:43.644 7fc126e9e700 1 --2- 172.21.2.112:0/4035689220 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc1201b0470 0x7fc1201b0870 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 rx=0x7fc108006650 tx=0x7fc108009a80).stop 2021-02-10T11:21:43.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:43.645 7fc126e9e700 1 -- 172.21.2.112:0/4035689220 shutdown_connections 2021-02-10T11:21:43.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:43.645 7fc126e9e700 1 --2- 172.21.2.112:0/4035689220 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc1200f20e0 0x7fc1200f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:43.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:43.645 7fc126e9e700 1 --2- 172.21.2.112:0/4035689220 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc1201b0470 0x7fc1201b0870 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:43.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:43.645 7fc126e9e700 1 -- 172.21.2.112:0/4035689220 >> 172.21.2.112:0/4035689220 conn(0x7fc1200ce3e0 msgr2=0x7fc1200e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:43.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:43.645 7fc126e9e700 1 -- 172.21.2.112:0/4035689220 shutdown_connections 2021-02-10T11:21:43.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:43.645 7fc126e9e700 1 -- 172.21.2.112:0/4035689220 wait complete. 2021-02-10T11:21:43.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:43.659 7fc126e9e700 1 Processor -- start 2021-02-10T11:21:43.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:43.659 7fc126e9e700 1 -- start start 2021-02-10T11:21:43.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:43.660 7fc126e9e700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc1200a0420 0x7fc1200a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:43.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:43.660 7fc126e9e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc120189d00 0x7fc12018a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:43.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:43.660 7fc126e9e700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc12018a720 0x7fc1200b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:43.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:43.660 7fc126e9e700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc120189780 con 0x7fc120189d00 2021-02-10T11:21:43.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:43.660 7fc126e9e700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc12018ab20 con 0x7fc12018a720 2021-02-10T11:21:43.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:43.660 7fc126e9e700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1200a1140 con 0x7fc1200a0420 2021-02-10T11:21:43.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:43.660 7fc11ffff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc120189d00 0x7fc12018a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:43.657 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:43.660 7fc11ffff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc120189d00 0x7fc12018a100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:44956/0 (socket says 172.21.2.112:44956) 2021-02-10T11:21:43.657 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:43.660 7fc11ffff700 1 -- 172.21.2.112:0/718765032 learned_addr learned my addr 172.21.2.112:0/718765032 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:21:43.657 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:43.661 7fc11ffff700 1 -- 172.21.2.112:0/718765032 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc1200a0420 msgr2=0x7fc1200a0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:43.657 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:43.661 7fc11ffff700 1 --2- 172.21.2.112:0/718765032 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc1200a0420 0x7fc1200a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:43.657 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:43.661 7fc11ffff700 1 -- 172.21.2.112:0/718765032 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc12018a720 msgr2=0x7fc1200b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:43.657 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:43.661 7fc11ffff700 1 --2- 172.21.2.112:0/718765032 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc12018a720 0x7fc1200b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:43.657 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:43.661 7fc11ffff700 1 -- 172.21.2.112:0/718765032 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1200b8200 con 0x7fc120189d00 2021-02-10T11:21:43.657 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:43.661 7fc11ffff700 1 --2- 172.21.2.112:0/718765032 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc120189d00 0x7fc12018a100 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 rx=0x7fc11000a940 tx=0x7fc11000ace0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:43.657 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:43.661 7fc11d7fa700 1 -- 172.21.2.112:0/718765032 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc1100070d0 con 0x7fc120189d00 2021-02-10T11:21:43.657 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:43.661 7fc11d7fa700 1 -- 172.21.2.112:0/718765032 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc110001010 con 0x7fc120189d00 2021-02-10T11:21:43.657 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:43.661 7fc11d7fa700 1 -- 172.21.2.112:0/718765032 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc110007550 con 0x7fc120189d00 2021-02-10T11:21:43.657 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:43.661 7fc126e9e700 1 -- 172.21.2.112:0/718765032 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1200b8200 con 0x7fc120189d00 2021-02-10T11:21:43.657 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:43.661 7fc126e9e700 1 -- 172.21.2.112:0/718765032 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1200b8790 con 0x7fc120189d00 2021-02-10T11:21:43.657 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:43.662 7fc11d7fa700 1 -- 172.21.2.112:0/718765032 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc11000ae20 con 0x7fc120189d00 2021-02-10T11:21:43.658 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:43.662 7fc11d7fa700 1 --2- 172.21.2.112:0/718765032 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc1040186c0 0x7fc10401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:43.658 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:43.662 7fc11d7fa700 1 -- 172.21.2.112:0/718765032 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fc1100329a0 con 0x7fc120189d00 2021-02-10T11:21:43.658 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:43.662 7fc124cba700 1 --2- 172.21.2.112:0/718765032 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc1040186c0 0x7fc10401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:43.658 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:43.663 7fc124cba700 1 --2- 172.21.2.112:0/718765032 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc1040186c0 0x7fc10401abc0 crc :-1 s=READY pgs=257 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:43.686 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:43.693 7fc126e9e700 1 -- 172.21.2.112:0/718765032 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc120105ae0 con 0x7fc120189d00 2021-02-10T11:21:43.689 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:43.695 7fc11d7fa700 1 -- 172.21.2.112:0/718765032 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc110001250 con 0x7fc120189d00 2021-02-10T11:21:43.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:43.820 7fc126e9e700 1 -- 172.21.2.112:0/718765032 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348609", "target": ["mgr", ""]}) v1 -- 0x7fc1200b9020 con 0x7fc1040186c0 2021-02-10T11:21:43.840 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:43.846 7fc11d7fa700 1 -- 172.21.2.112:0/718765032 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fc108000940 con 0x7fc1040186c0 2021-02-10T11:21:43.848 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:43.853 7fc126e9e700 1 -- 172.21.2.112:0/718765032 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc1040186c0 msgr2=0x7fc10401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:43.848 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:43.853 7fc126e9e700 1 --2- 172.21.2.112:0/718765032 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc1040186c0 0x7fc10401abc0 crc :-1 s=READY pgs=257 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:43.849 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:43.853 7fc126e9e700 1 -- 172.21.2.112:0/718765032 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc120189d00 msgr2=0x7fc12018a100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:43.849 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:43.853 7fc126e9e700 1 --2- 172.21.2.112:0/718765032 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc120189d00 0x7fc12018a100 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 rx=0x7fc11000a940 tx=0x7fc11000ace0).stop 2021-02-10T11:21:43.849 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:43.854 7fc126e9e700 1 -- 172.21.2.112:0/718765032 shutdown_connections 2021-02-10T11:21:43.849 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:43.854 7fc126e9e700 1 --2- 172.21.2.112:0/718765032 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc1040186c0 0x7fc10401abc0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:43.849 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:43.854 7fc126e9e700 1 --2- 172.21.2.112:0/718765032 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc12018a720 0x7fc1200b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:43.849 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:43.854 7fc126e9e700 1 --2- 172.21.2.112:0/718765032 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc120189d00 0x7fc12018a100 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:43.850 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:43.854 7fc126e9e700 1 --2- 172.21.2.112:0/718765032 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc1200a0420 0x7fc1200a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:43.850 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:43.854 7fc126e9e700 1 -- 172.21.2.112:0/718765032 >> 172.21.2.112:0/718765032 conn(0x7fc120094e10 msgr2=0x7fc1200a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:43.850 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:43.854 7fc126e9e700 1 -- 172.21.2.112:0/718765032 shutdown_connections 2021-02-10T11:21:43.850 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:43.854 7fc126e9e700 1 -- 172.21.2.112:0/718765032 wait complete. 2021-02-10T11:21:43.855 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000348609 2021-02-10T11:21:44.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:44.275 7fe3a4e3b700 1 Processor -- start 2021-02-10T11:21:44.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:44.275 7fe3a4e3b700 1 -- start start 2021-02-10T11:21:44.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:44.276 7fe3a4e3b700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fe3a01b0870 0x7fe3a00f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:44.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:44.276 7fe3a4e3b700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fe3a00efe00 0x7fe3a00f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:44.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:44.276 7fe3a4e3b700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3a01b0c70 con 0x7fe3a01b0470 2021-02-10T11:21:44.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:44.276 7fe3a4e3b700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3a00e5ea0 con 0x7fe3a01b0870 2021-02-10T11:21:44.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:44.276 7fe3a4e3b700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3a00f0b50 con 0x7fe3a00efe00 2021-02-10T11:21:44.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:44.276 7fe39e312700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fe3a01b0870 0x7fe3a00f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:44.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:44.276 7fe39e312700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fe3a01b0870 0x7fe3a00f20e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:44962/0 (socket says 172.21.2.112:44962) 2021-02-10T11:21:44.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:44.276 7fe39eb13700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fe3a01b0470 0x7fe3a00ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:44422/0 (socket says 172.21.2.112:44422) 2021-02-10T11:21:44.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:44.276 7fe39eb13700 1 -- 172.21.2.112:0/512179008 learned_addr learned my addr 172.21.2.112:0/512179008 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:21:44.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:44.276 7fe39db11700 1 -- 172.21.2.112:0/512179008 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2919298065 0 0) 0x7fe3900010b0 con 0x7fe3a01b0470 2021-02-10T11:21:44.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:44.276 7fe39db11700 1 -- 172.21.2.112:0/512179008 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe38c002fe0 con 0x7fe3a01b0470 2021-02-10T11:21:44.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:44.276 7fe39e312700 1 -- 172.21.2.112:0/512179008 >> v2:172.21.2.120:3300/0 conn(0x7fe3a00efe00 msgr2=0x7fe3a00f0200 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:21:44.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:44.276 7fe39e312700 1 --2- 172.21.2.112:0/512179008 >> v2:172.21.2.120:3300/0 conn(0x7fe3a00efe00 0x7fe3a00f0200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:44.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:44.276 7fe39e312700 1 -- 172.21.2.112:0/512179008 >> v1:172.21.2.112:6789/0 conn(0x7fe3a01b0470 legacy=0x7fe3a00ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:44.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:44.276 7fe39e312700 1 -- 172.21.2.112:0/512179008 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3a00f0820 con 0x7fe3a01b0870 2021-02-10T11:21:44.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:44.276 7fe39e312700 1 --2- 172.21.2.112:0/512179008 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe3a01b0870 0x7fe3a00f20e0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 rx=0x7fe388006650 tx=0x7fe388009a80).ready entity=mon.0 client_cookie=8434fb3253dd859b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:44.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:44.276 7fe39db11700 1 -- 172.21.2.112:0/512179008 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe38800c0c0 con 0x7fe3a01b0870 2021-02-10T11:21:44.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:44.276 7fe39db11700 1 -- 172.21.2.112:0/512179008 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe38800c340 con 0x7fe3a01b0870 2021-02-10T11:21:44.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:44.276 7fe39db11700 1 -- 172.21.2.112:0/512179008 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe38800c820 con 0x7fe3a01b0870 2021-02-10T11:21:44.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:44.277 7fe3a4e3b700 1 -- 172.21.2.112:0/512179008 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe3a01b0870 msgr2=0x7fe3a00f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:44.287 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:44.277 7fe3a4e3b700 1 --2- 172.21.2.112:0/512179008 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe3a01b0870 0x7fe3a00f20e0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 rx=0x7fe388006650 tx=0x7fe388009a80).stop 2021-02-10T11:21:44.287 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:44.277 7fe3a4e3b700 1 -- 172.21.2.112:0/512179008 shutdown_connections 2021-02-10T11:21:44.287 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:44.277 7fe3a4e3b700 1 --2- 172.21.2.112:0/512179008 >> v2:172.21.2.120:3300/0 conn(0x7fe3a00efe00 0x7fe3a00f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:44.287 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:44.277 7fe3a4e3b700 1 --2- 172.21.2.112:0/512179008 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe3a01b0870 0x7fe3a00f20e0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:44.287 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:44.277 7fe3a4e3b700 1 -- 172.21.2.112:0/512179008 >> 172.21.2.112:0/512179008 conn(0x7fe3a00ce3e0 msgr2=0x7fe3a00e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:44.287 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:44.277 7fe3a4e3b700 1 -- 172.21.2.112:0/512179008 shutdown_connections 2021-02-10T11:21:44.287 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:44.277 7fe3a4e3b700 1 -- 172.21.2.112:0/512179008 wait complete. 2021-02-10T11:21:44.288 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:44.291 7fe3a4e3b700 1 Processor -- start 2021-02-10T11:21:44.288 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:44.292 7fe3a4e3b700 1 -- start start 2021-02-10T11:21:44.289 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:44.292 7fe3a4e3b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe3a0105d10 0x7fe3a009eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:44.289 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:44.292 7fe3a4e3b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe3a0106110 0x7fe3a009f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:44.289 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:44.292 7fe3a4e3b700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe3a00b2a00 0x7fe3a00b2e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:44.289 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:44.292 7fe3a4e3b700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3a009e560 con 0x7fe3a0106110 2021-02-10T11:21:44.289 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:44.292 7fe3a4e3b700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3a009c090 con 0x7fe3a00b2a00 2021-02-10T11:21:44.289 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:44.292 7fe3a4e3b700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3a00b3420 con 0x7fe3a0105d10 2021-02-10T11:21:44.289 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:44.292 7fe39e312700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe3a0106110 0x7fe3a009f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:44.289 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:44.292 7fe39e312700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe3a0106110 0x7fe3a009f100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:44968/0 (socket says 172.21.2.112:44968) 2021-02-10T11:21:44.289 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:44.292 7fe39e312700 1 -- 172.21.2.112:0/1729409987 learned_addr learned my addr 172.21.2.112:0/1729409987 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:21:44.289 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:44.292 7fe39e312700 1 -- 172.21.2.112:0/1729409987 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe3a0105d10 msgr2=0x7fe3a009eae0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:21:44.289 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:44.292 7fe39e312700 1 --2- 172.21.2.112:0/1729409987 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe3a0105d10 0x7fe3a009eae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:44.289 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:44.292 7fe39e312700 1 -- 172.21.2.112:0/1729409987 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe3a00b2a00 msgr2=0x7fe3a00b2e00 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:21:44.290 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:44.292 7fe39e312700 1 --2- 172.21.2.112:0/1729409987 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe3a00b2a00 0x7fe3a00b2e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:44.290 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:44.292 7fe39e312700 1 -- 172.21.2.112:0/1729409987 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3a00b3640 con 0x7fe3a0106110 2021-02-10T11:21:44.290 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:44.293 7fe39e312700 1 --2- 172.21.2.112:0/1729409987 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe3a0106110 0x7fe3a009f100 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 rx=0x7fe388009630 tx=0x7fe38800b900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:44.290 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:44.293 7fe3977fe700 1 -- 172.21.2.112:0/1729409987 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe3880093a0 con 0x7fe3a0106110 2021-02-10T11:21:44.290 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:44.293 7fe3977fe700 1 -- 172.21.2.112:0/1729409987 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe38800bcf0 con 0x7fe3a0106110 2021-02-10T11:21:44.290 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:44.293 7fe3977fe700 1 -- 172.21.2.112:0/1729409987 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe388005560 con 0x7fe3a0106110 2021-02-10T11:21:44.290 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:44.293 7fe3a4e3b700 1 -- 172.21.2.112:0/1729409987 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3a00b3640 con 0x7fe3a0106110 2021-02-10T11:21:44.290 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:44.293 7fe3a4e3b700 1 -- 172.21.2.112:0/1729409987 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3a00b3bd0 con 0x7fe3a0106110 2021-02-10T11:21:44.290 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:44.294 7fe3977fe700 1 -- 172.21.2.112:0/1729409987 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe388008240 con 0x7fe3a0106110 2021-02-10T11:21:44.290 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:44.294 7fe3977fe700 1 --2- 172.21.2.112:0/1729409987 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe3840186c0 0x7fe38401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:44.290 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:44.294 7fe3977fe700 1 -- 172.21.2.112:0/1729409987 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fe388032510 con 0x7fe3a0106110 2021-02-10T11:21:44.290 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:44.295 7fe39eb13700 1 --2- 172.21.2.112:0/1729409987 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe3840186c0 0x7fe38401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:44.291 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:44.296 7fe39eb13700 1 --2- 172.21.2.112:0/1729409987 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe3840186c0 0x7fe38401abc0 crc :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:44.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:44.326 7fe3a4e3b700 1 -- 172.21.2.112:0/1729409987 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3a0105ae0 con 0x7fe3a0106110 2021-02-10T11:21:44.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:44.328 7fe3977fe700 1 -- 172.21.2.112:0/1729409987 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe38801c990 con 0x7fe3a0106110 2021-02-10T11:21:44.452 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:44.458 7fe3a4e3b700 1 -- 172.21.2.112:0/1729409987 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348609", "target": ["mgr", ""]}) v1 -- 0x7fe3a01a16c0 con 0x7fe3840186c0 2021-02-10T11:21:44.455 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:44.462 7fe3977fe700 1 -- 172.21.2.112:0/1729409987 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7fe390001520 con 0x7fe3840186c0 2021-02-10T11:21:44.459 INFO:teuthology.orchestra.run.gibba012.stdout:/volumes/_nogroup/subvolume_0000000000348609/90a98ec0-fd93-4af2-ba35-36ce213c6132 2021-02-10T11:21:44.461 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:44.467 7fe3a4e3b700 1 -- 172.21.2.112:0/1729409987 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe3840186c0 msgr2=0x7fe38401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:44.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:44.467 7fe3a4e3b700 1 --2- 172.21.2.112:0/1729409987 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe3840186c0 0x7fe38401abc0 crc :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:44.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:44.468 7fe3a4e3b700 1 -- 172.21.2.112:0/1729409987 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe3a0106110 msgr2=0x7fe3a009f100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:44.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:44.468 7fe3a4e3b700 1 --2- 172.21.2.112:0/1729409987 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe3a0106110 0x7fe3a009f100 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 rx=0x7fe388009630 tx=0x7fe38800b900).stop 2021-02-10T11:21:44.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:44.468 7fe3a4e3b700 1 -- 172.21.2.112:0/1729409987 shutdown_connections 2021-02-10T11:21:44.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:44.468 7fe3a4e3b700 1 --2- 172.21.2.112:0/1729409987 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe3840186c0 0x7fe38401abc0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:44.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:44.468 7fe3a4e3b700 1 --2- 172.21.2.112:0/1729409987 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe3a00b2a00 0x7fe3a00b2e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:44.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:44.468 7fe3a4e3b700 1 --2- 172.21.2.112:0/1729409987 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe3a0106110 0x7fe3a009f100 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:44.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:44.468 7fe3a4e3b700 1 --2- 172.21.2.112:0/1729409987 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe3a0105d10 0x7fe3a009eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:44.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:44.468 7fe3a4e3b700 1 -- 172.21.2.112:0/1729409987 >> 172.21.2.112:0/1729409987 conn(0x7fe3a00c36f0 msgr2=0x7fe3a00bd640 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:44.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:44.468 7fe3a4e3b700 1 -- 172.21.2.112:0/1729409987 shutdown_connections 2021-02-10T11:21:44.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:44.468 7fe3a4e3b700 1 -- 172.21.2.112:0/1729409987 wait complete. 2021-02-10T11:21:44.469 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000348609 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000348609/90a98ec0-fd93-4af2-ba35-36ce213c6132 2021-02-10T11:21:44.470 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:21:44.470 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:21:44.470 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:21:44.805 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:21:44.806 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:21:44.806 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348609/90a98ec0-fd93-4af2-ba35-36ce213c6132/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:21:45.185 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:21:45.186 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:21:45.186 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.024683 s, 42.5 MB/s 2021-02-10T11:21:45.187 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:21:45.187 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:21:45.187 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:21:45.522 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:21:45.523 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:21:45.523 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348609/90a98ec0-fd93-4af2-ba35-36ce213c6132/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:21:45.899 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:21:45.899 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:21:45.899 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0244864 s, 42.8 MB/s 2021-02-10T11:21:45.900 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:21:45.900 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:21:45.900 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:21:46.235 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:21:46.235 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:21:46.236 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348609/90a98ec0-fd93-4af2-ba35-36ce213c6132/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:21:46.610 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:21:46.610 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:21:46.610 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0245568 s, 42.7 MB/s 2021-02-10T11:21:46.611 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:21:46.611 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:21:46.611 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:21:46.946 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:21:46.946 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:21:46.946 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348609/90a98ec0-fd93-4af2-ba35-36ce213c6132/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:21:47.322 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:21:47.322 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:21:47.323 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0249897 s, 42.0 MB/s 2021-02-10T11:21:47.323 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:21:47.324 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:21:47.324 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:21:47.659 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:21:47.659 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:21:47.660 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348609/90a98ec0-fd93-4af2-ba35-36ce213c6132/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:21:48.034 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:21:48.035 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:21:48.035 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0247839 s, 42.3 MB/s 2021-02-10T11:21:48.036 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:21:48.036 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:21:48.036 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:21:48.370 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:21:48.370 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:21:48.371 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348609/90a98ec0-fd93-4af2-ba35-36ce213c6132/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:21:48.745 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:21:48.745 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:21:48.745 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0236671 s, 44.3 MB/s 2021-02-10T11:21:48.746 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:21:48.746 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:21:48.746 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:21:49.080 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:21:49.081 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:21:49.081 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348609/90a98ec0-fd93-4af2-ba35-36ce213c6132/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:21:49.450 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:21:49.451 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:21:49.451 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0223096 s, 47.0 MB/s 2021-02-10T11:21:49.452 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:21:49.452 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:21:49.452 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:21:49.786 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:21:49.787 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:21:49.787 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348609/90a98ec0-fd93-4af2-ba35-36ce213c6132/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:21:50.161 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:21:50.161 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:21:50.161 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0235459 s, 44.5 MB/s 2021-02-10T11:21:50.163 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:21:50.163 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:21:50.163 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:21:50.499 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:21:50.499 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:21:50.499 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348609/90a98ec0-fd93-4af2-ba35-36ce213c6132/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:21:50.871 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:21:50.871 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:21:50.871 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0214867 s, 48.8 MB/s 2021-02-10T11:21:50.872 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:21:50.873 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:21:50.873 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:21:51.209 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:21:51.209 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:21:51.210 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348609/90a98ec0-fd93-4af2-ba35-36ce213c6132/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:21:51.584 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:21:51.904 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:21:51.905 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0249755 s, 42.0 MB/s 2021-02-10T11:21:51.906 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000348610 2021-02-10T11:21:52.339 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.327 7fe96138e700 1 Processor -- start 2021-02-10T11:21:52.339 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.327 7fe96138e700 1 -- start start 2021-02-10T11:21:52.339 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.327 7fe96138e700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fe95c1b0470 0x7fe95c1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:52.339 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.327 7fe96138e700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe95c0f20e0 0x7fe95c0f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:52.339 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.327 7fe96138e700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe95c0f0b50 con 0x7fe95c0ef610 2021-02-10T11:21:52.339 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.327 7fe96138e700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe95c0f0820 con 0x7fe95c1b0470 2021-02-10T11:21:52.339 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.327 7fe96138e700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe95c0f24e0 con 0x7fe95c0f20e0 2021-02-10T11:21:52.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.329 7fe95b07d700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fe95c1b0470 0x7fe95c1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:52.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.329 7fe95b87e700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe95c0f20e0 0x7fe95c0f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:52.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.329 7fe95a87c700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7fe95c0ef610 0x7fe95c0efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:37994/0 (socket says 172.21.2.112:37994) 2021-02-10T11:21:52.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.329 7fe95a87c700 1 -- 172.21.2.112:0/4109112821 learned_addr learned my addr 172.21.2.112:0/4109112821 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:21:52.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.329 7fe95b87e700 1 -- 172.21.2.112:0/4109112821 >> v2:172.21.2.120:3300/0 conn(0x7fe95c1b0470 msgr2=0x7fe95c1b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:21:52.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.329 7fe95b87e700 1 --2- 172.21.2.112:0/4109112821 >> v2:172.21.2.120:3300/0 conn(0x7fe95c1b0470 0x7fe95c1b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:52.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.329 7fe95b87e700 1 -- 172.21.2.112:0/4109112821 >> v1:172.21.2.120:6789/0 conn(0x7fe95c0ef610 legacy=0x7fe95c0efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:52.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.329 7fe95b87e700 1 -- 172.21.2.112:0/4109112821 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe95c0e52e0 con 0x7fe95c0f20e0 2021-02-10T11:21:52.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.329 7fe95a07b700 1 -- 172.21.2.112:0/4109112821 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 724201940 0 0) 0x7fe9440009b0 con 0x7fe95c0ef610 2021-02-10T11:21:52.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.329 7fe95b07d700 1 --2- 172.21.2.112:0/4109112821 >> v2:172.21.2.120:3300/0 conn(0x7fe95c1b0470 0x7fe95c1b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:21:52.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.329 7fe95b87e700 1 --2- 172.21.2.112:0/4109112821 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe95c0f20e0 0x7fe95c0f0200 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7fe954006e50 tx=0x7fe954009e70).ready entity=mon.2 client_cookie=616743ccff8d2c6a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:52.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.330 7fe95a07b700 1 -- 172.21.2.112:0/4109112821 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe954000f90 con 0x7fe95c0f20e0 2021-02-10T11:21:52.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.330 7fe95a07b700 1 -- 172.21.2.112:0/4109112821 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9540066f0 con 0x7fe95c0f20e0 2021-02-10T11:21:52.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.330 7fe95a07b700 1 -- 172.21.2.112:0/4109112821 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe95400b700 con 0x7fe95c0f20e0 2021-02-10T11:21:52.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.330 7fe96138e700 1 -- 172.21.2.112:0/4109112821 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe95c0f20e0 msgr2=0x7fe95c0f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:52.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.330 7fe96138e700 1 --2- 172.21.2.112:0/4109112821 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe95c0f20e0 0x7fe95c0f0200 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7fe954006e50 tx=0x7fe954009e70).stop 2021-02-10T11:21:52.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.331 7fe96138e700 1 -- 172.21.2.112:0/4109112821 shutdown_connections 2021-02-10T11:21:52.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.331 7fe96138e700 1 --2- 172.21.2.112:0/4109112821 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe95c0f20e0 0x7fe95c0f0200 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:52.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.331 7fe96138e700 1 --2- 172.21.2.112:0/4109112821 >> v2:172.21.2.120:3300/0 conn(0x7fe95c1b0470 0x7fe95c1b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:52.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.331 7fe96138e700 1 -- 172.21.2.112:0/4109112821 >> 172.21.2.112:0/4109112821 conn(0x7fe95c0ce3e0 msgr2=0x7fe95c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:52.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.331 7fe96138e700 1 -- 172.21.2.112:0/4109112821 shutdown_connections 2021-02-10T11:21:52.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.331 7fe96138e700 1 -- 172.21.2.112:0/4109112821 wait complete. 2021-02-10T11:21:52.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.345 7fe96138e700 1 Processor -- start 2021-02-10T11:21:52.342 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.345 7fe96138e700 1 -- start start 2021-02-10T11:21:52.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.346 7fe96138e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe95c190920 0x7fe95c0b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:52.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.346 7fe96138e700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe95c0b8200 0x7fe95c0b8600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:52.344 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.346 7fe96138e700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe95c0c1d80 0x7fe95c0c2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:52.344 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.346 7fe96138e700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe95c190da0 con 0x7fe95c190920 2021-02-10T11:21:52.344 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.346 7fe96138e700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe95c0b8c20 con 0x7fe95c0b8200 2021-02-10T11:21:52.344 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.346 7fe96138e700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe95c0b8e10 con 0x7fe95c0c1d80 2021-02-10T11:21:52.344 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.346 7fe95b07d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe95c190920 0x7fe95c0b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:52.344 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.346 7fe95b07d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe95c190920 0x7fe95c0b7be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:44978/0 (socket says 172.21.2.112:44978) 2021-02-10T11:21:52.344 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.346 7fe95b07d700 1 -- 172.21.2.112:0/1000488070 learned_addr learned my addr 172.21.2.112:0/1000488070 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:21:52.344 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.346 7fe95b07d700 1 -- 172.21.2.112:0/1000488070 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe95c0c1d80 msgr2=0x7fe95c0c2180 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:21:52.344 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.346 7fe95b07d700 1 --2- 172.21.2.112:0/1000488070 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe95c0c1d80 0x7fe95c0c2180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:52.344 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.346 7fe95b07d700 1 -- 172.21.2.112:0/1000488070 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe95c0b8200 msgr2=0x7fe95c0b8600 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:21:52.344 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.346 7fe95b07d700 1 --2- 172.21.2.112:0/1000488070 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe95c0b8200 0x7fe95c0b8600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:52.344 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.346 7fe95b07d700 1 -- 172.21.2.112:0/1000488070 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe95c0c27d0 con 0x7fe95c190920 2021-02-10T11:21:52.345 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.346 7fe95b07d700 1 --2- 172.21.2.112:0/1000488070 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe95c190920 0x7fe95c0b7be0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 rx=0x7fe94c006f40 tx=0x7fe94c00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:52.345 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.347 7fe953fff700 1 -- 172.21.2.112:0/1000488070 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe94c00bd80 con 0x7fe95c190920 2021-02-10T11:21:52.345 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.347 7fe953fff700 1 -- 172.21.2.112:0/1000488070 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe94c000c90 con 0x7fe95c190920 2021-02-10T11:21:52.345 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.347 7fe96138e700 1 -- 172.21.2.112:0/1000488070 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe95c0c27d0 con 0x7fe95c190920 2021-02-10T11:21:52.345 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.347 7fe96138e700 1 -- 172.21.2.112:0/1000488070 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe95c0c2d60 con 0x7fe95c190920 2021-02-10T11:21:52.345 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.347 7fe953fff700 1 -- 172.21.2.112:0/1000488070 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe94c002220 con 0x7fe95c190920 2021-02-10T11:21:52.345 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.347 7fe953fff700 1 -- 172.21.2.112:0/1000488070 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe94c00a000 con 0x7fe95c190920 2021-02-10T11:21:52.345 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.347 7fe953fff700 1 --2- 172.21.2.112:0/1000488070 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe9400186c0 0x7fe94001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:52.345 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.347 7fe953fff700 1 -- 172.21.2.112:0/1000488070 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fe94c032560 con 0x7fe95c190920 2021-02-10T11:21:52.345 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.348 7fe95a87c700 1 --2- 172.21.2.112:0/1000488070 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe9400186c0 0x7fe94001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:52.345 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.348 7fe95a87c700 1 --2- 172.21.2.112:0/1000488070 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe9400186c0 0x7fe94001abc0 crc :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:52.371 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.377 7fe96138e700 1 -- 172.21.2.112:0/1000488070 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe934002fd0 con 0x7fe95c190920 2021-02-10T11:21:52.373 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.380 7fe953fff700 1 -- 172.21.2.112:0/1000488070 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe94c024f00 con 0x7fe95c190920 2021-02-10T11:21:52.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.508 7fe96138e700 1 -- 172.21.2.112:0/1000488070 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348610", "target": ["mgr", ""]}) v1 -- 0x7fe934002f80 con 0x7fe9400186c0 2021-02-10T11:21:52.528 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.535 7fe953fff700 1 -- 172.21.2.112:0/1000488070 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fe944000e20 con 0x7fe9400186c0 2021-02-10T11:21:52.534 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.541 7fe96138e700 1 -- 172.21.2.112:0/1000488070 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe9400186c0 msgr2=0x7fe94001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:52.534 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.541 7fe96138e700 1 --2- 172.21.2.112:0/1000488070 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe9400186c0 0x7fe94001abc0 crc :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:52.534 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.541 7fe96138e700 1 -- 172.21.2.112:0/1000488070 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe95c190920 msgr2=0x7fe95c0b7be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:52.534 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.541 7fe96138e700 1 --2- 172.21.2.112:0/1000488070 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe95c190920 0x7fe95c0b7be0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 rx=0x7fe94c006f40 tx=0x7fe94c00b990).stop 2021-02-10T11:21:52.535 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.541 7fe96138e700 1 -- 172.21.2.112:0/1000488070 shutdown_connections 2021-02-10T11:21:52.535 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.541 7fe96138e700 1 --2- 172.21.2.112:0/1000488070 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe9400186c0 0x7fe94001abc0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:52.535 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.541 7fe96138e700 1 --2- 172.21.2.112:0/1000488070 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe95c0b8200 0x7fe95c0b8600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:52.535 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.541 7fe96138e700 1 --2- 172.21.2.112:0/1000488070 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe95c0c1d80 0x7fe95c0c2180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:52.535 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.541 7fe96138e700 1 --2- 172.21.2.112:0/1000488070 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe95c190920 0x7fe95c0b7be0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:52.535 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.541 7fe96138e700 1 -- 172.21.2.112:0/1000488070 >> 172.21.2.112:0/1000488070 conn(0x7fe95c094e10 msgr2=0x7fe95c0a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:52.535 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.541 7fe96138e700 1 -- 172.21.2.112:0/1000488070 shutdown_connections 2021-02-10T11:21:52.535 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.541 7fe96138e700 1 -- 172.21.2.112:0/1000488070 wait complete. 2021-02-10T11:21:52.542 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000348610 2021-02-10T11:21:52.974 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.963 7f4e4fc0f700 1 Processor -- start 2021-02-10T11:21:52.975 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.963 7f4e4fc0f700 1 -- start start 2021-02-10T11:21:52.975 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.963 7f4e4fc0f700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4e481b0470 0x7f4e481b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:52.975 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.963 7f4e4fc0f700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f4e480f20e0 0x7f4e480f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:52.975 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.963 7f4e4fc0f700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e480f0b50 con 0x7f4e480ef610 2021-02-10T11:21:52.975 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.963 7f4e4fc0f700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e480f0820 con 0x7f4e480f20e0 2021-02-10T11:21:52.975 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.963 7f4e4fc0f700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e480f24e0 con 0x7f4e481b0470 2021-02-10T11:21:52.975 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.964 7f4e4e22c700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f4e480f20e0 0x7f4e480f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:52.975 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.964 7f4e4da2b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4e481b0470 0x7f4e481b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:52.975 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.964 7f4e4e22c700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f4e480f20e0 0x7f4e480f0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3300/0 says I am v2:172.21.2.112:47266/0 (socket says 172.21.2.112:47266) 2021-02-10T11:21:52.975 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.964 7f4e4da2b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4e481b0470 0x7f4e481b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3301/0 says I am v2:172.21.2.112:44668/0 (socket says 172.21.2.112:44668) 2021-02-10T11:21:52.975 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.964 7f4e4e22c700 1 -- 172.21.2.112:0/3366857117 learned_addr learned my addr 172.21.2.112:0/3366857117 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:21:52.975 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.965 7f4e4da2b700 1 -- 172.21.2.112:0/3366857117 >> v2:172.21.2.120:3300/0 conn(0x7f4e480f20e0 msgr2=0x7f4e480f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:21:52.976 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.965 7f4e4da2b700 1 --2- 172.21.2.112:0/3366857117 >> v2:172.21.2.120:3300/0 conn(0x7f4e480f20e0 0x7f4e480f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:52.976 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.965 7f4e4da2b700 1 -- 172.21.2.112:0/3366857117 >> v1:172.21.2.120:6789/0 conn(0x7f4e480ef610 legacy=0x7f4e480efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:52.976 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.965 7f4e4da2b700 1 -- 172.21.2.112:0/3366857117 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e480e52e0 con 0x7f4e481b0470 2021-02-10T11:21:52.976 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.965 7f4e4e22c700 1 --2- 172.21.2.112:0/3366857117 >> v2:172.21.2.120:3300/0 conn(0x7f4e480f20e0 0x7f4e480f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:21:52.976 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.965 7f4e4da2b700 1 --2- 172.21.2.112:0/3366857117 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4e481b0470 0x7f4e481b0870 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f4e44006650 tx=0x7f4e44009a80).ready entity=mon.2 client_cookie=8ee9d076754543e4 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:52.976 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.966 7f4e4ca29700 1 -- 172.21.2.112:0/3366857117 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4e440009b0 con 0x7f4e481b0470 2021-02-10T11:21:52.976 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.966 7f4e4ca29700 1 -- 172.21.2.112:0/3366857117 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e4400c0c0 con 0x7f4e481b0470 2021-02-10T11:21:52.976 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.966 7f4e4ca29700 1 -- 172.21.2.112:0/3366857117 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4e4400c5e0 con 0x7f4e481b0470 2021-02-10T11:21:52.976 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.966 7f4e4fc0f700 1 -- 172.21.2.112:0/3366857117 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4e481b0470 msgr2=0x7f4e481b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:52.976 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.966 7f4e4fc0f700 1 --2- 172.21.2.112:0/3366857117 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4e481b0470 0x7f4e481b0870 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f4e44006650 tx=0x7f4e44009a80).stop 2021-02-10T11:21:52.976 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.967 7f4e4fc0f700 1 -- 172.21.2.112:0/3366857117 shutdown_connections 2021-02-10T11:21:52.976 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.967 7f4e4fc0f700 1 --2- 172.21.2.112:0/3366857117 >> v2:172.21.2.120:3300/0 conn(0x7f4e480f20e0 0x7f4e480f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:52.976 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.967 7f4e4fc0f700 1 --2- 172.21.2.112:0/3366857117 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4e481b0470 0x7f4e481b0870 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:52.977 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.967 7f4e4fc0f700 1 -- 172.21.2.112:0/3366857117 >> 172.21.2.112:0/3366857117 conn(0x7f4e480ce3e0 msgr2=0x7f4e480e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:52.977 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.967 7f4e4fc0f700 1 -- 172.21.2.112:0/3366857117 shutdown_connections 2021-02-10T11:21:52.977 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.967 7f4e4fc0f700 1 -- 172.21.2.112:0/3366857117 wait complete. 2021-02-10T11:21:52.978 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.981 7f4e4fc0f700 1 Processor -- start 2021-02-10T11:21:52.978 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.982 7f4e4fc0f700 1 -- start start 2021-02-10T11:21:52.978 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.982 7f4e4fc0f700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4e480b7c20 0x7f4e480b8410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:52.979 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.982 7f4e4fc0f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4e480b8a30 0x7f4e480c1f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:52.979 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.982 7f4e4fc0f700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4e480c25a0 0x7f4e480c29a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:52.979 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.982 7f4e4fc0f700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e480c1a80 con 0x7f4e480b8a30 2021-02-10T11:21:52.979 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.982 7f4e4fc0f700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e480b8020 con 0x7f4e480c25a0 2021-02-10T11:21:52.979 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.982 7f4e4fc0f700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e4809b9c0 con 0x7f4e480b7c20 2021-02-10T11:21:52.979 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.982 7f4e4d22a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4e480b8a30 0x7f4e480c1f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:52.979 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.982 7f4e4d22a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4e480b8a30 0x7f4e480c1f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:44990/0 (socket says 172.21.2.112:44990) 2021-02-10T11:21:52.979 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.982 7f4e4d22a700 1 -- 172.21.2.112:0/2134774930 learned_addr learned my addr 172.21.2.112:0/2134774930 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:21:52.979 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.982 7f4e4d22a700 1 -- 172.21.2.112:0/2134774930 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4e480b7c20 msgr2=0x7f4e480b8410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:52.979 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.982 7f4e4d22a700 1 --2- 172.21.2.112:0/2134774930 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4e480b7c20 0x7f4e480b8410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:52.979 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.982 7f4e4d22a700 1 -- 172.21.2.112:0/2134774930 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4e480c25a0 msgr2=0x7f4e480c29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:52.979 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.982 7f4e4d22a700 1 --2- 172.21.2.112:0/2134774930 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4e480c25a0 0x7f4e480c29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:52.979 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.982 7f4e4d22a700 1 -- 172.21.2.112:0/2134774930 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e480b8e30 con 0x7f4e480b8a30 2021-02-10T11:21:52.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.983 7f4e4d22a700 1 --2- 172.21.2.112:0/2134774930 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4e480b8a30 0x7f4e480c1f80 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 rx=0x7f4e38006ed0 tx=0x7f4e3800b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:52.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.983 7f4e3e7fc700 1 -- 172.21.2.112:0/2134774930 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4e38006240 con 0x7f4e480b8a30 2021-02-10T11:21:52.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.983 7f4e3e7fc700 1 -- 172.21.2.112:0/2134774930 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e380054b0 con 0x7f4e480b8a30 2021-02-10T11:21:52.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.983 7f4e3e7fc700 1 -- 172.21.2.112:0/2134774930 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4e380059d0 con 0x7f4e480b8a30 2021-02-10T11:21:52.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.983 7f4e4fc0f700 1 -- 172.21.2.112:0/2134774930 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e480b8e30 con 0x7f4e480b8a30 2021-02-10T11:21:52.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.983 7f4e4fc0f700 1 -- 172.21.2.112:0/2134774930 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e480b8e30 con 0x7f4e480b8a30 2021-02-10T11:21:52.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.983 7f4e3e7fc700 1 -- 172.21.2.112:0/2134774930 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4e38006030 con 0x7f4e480b8a30 2021-02-10T11:21:52.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.984 7f4e3e7fc700 1 --2- 172.21.2.112:0/2134774930 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4e300186c0 0x7f4e3001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:52.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.984 7f4e3e7fc700 1 -- 172.21.2.112:0/2134774930 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f4e380325c0 con 0x7f4e480b8a30 2021-02-10T11:21:52.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.984 7f4e4da2b700 1 --2- 172.21.2.112:0/2134774930 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4e300186c0 0x7f4e3001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:52.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:52.984 7f4e4da2b700 1 --2- 172.21.2.112:0/2134774930 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4e300186c0 0x7f4e3001abc0 crc :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:53.007 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:53.014 7f4e4fc0f700 1 -- 172.21.2.112:0/2134774930 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e48105b30 con 0x7f4e480b8a30 2021-02-10T11:21:53.009 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:53.016 7f4e3e7fc700 1 -- 172.21.2.112:0/2134774930 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4e38000de0 con 0x7f4e480b8a30 2021-02-10T11:21:53.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:53.144 7f4e4fc0f700 1 -- 172.21.2.112:0/2134774930 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348610", "target": ["mgr", ""]}) v1 -- 0x7f4e480f0b50 con 0x7f4e300186c0 2021-02-10T11:21:53.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:53.148 7f4e3e7fc700 1 -- 172.21.2.112:0/2134774930 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f4e44006060 con 0x7f4e300186c0 2021-02-10T11:21:53.141 INFO:teuthology.orchestra.run.gibba012.stdout:/volumes/_nogroup/subvolume_0000000000348610/66c99d94-f362-4a14-97c3-d0a76e6540db 2021-02-10T11:21:53.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:53.149 7f4e4fc0f700 1 -- 172.21.2.112:0/2134774930 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4e300186c0 msgr2=0x7f4e3001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:53.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:53.149 7f4e4fc0f700 1 --2- 172.21.2.112:0/2134774930 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4e300186c0 0x7f4e3001abc0 crc :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:53.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:53.149 7f4e4fc0f700 1 -- 172.21.2.112:0/2134774930 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4e480b8a30 msgr2=0x7f4e480c1f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:53.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:53.149 7f4e4fc0f700 1 --2- 172.21.2.112:0/2134774930 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4e480b8a30 0x7f4e480c1f80 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 rx=0x7f4e38006ed0 tx=0x7f4e3800b960).stop 2021-02-10T11:21:53.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:53.150 7f4e4fc0f700 1 -- 172.21.2.112:0/2134774930 shutdown_connections 2021-02-10T11:21:53.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:53.150 7f4e4fc0f700 1 --2- 172.21.2.112:0/2134774930 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4e300186c0 0x7f4e3001abc0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:53.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:53.150 7f4e4fc0f700 1 --2- 172.21.2.112:0/2134774930 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4e480c25a0 0x7f4e480c29a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:53.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:53.150 7f4e4fc0f700 1 --2- 172.21.2.112:0/2134774930 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4e480b8a30 0x7f4e480c1f80 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:53.145 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:53.150 7f4e4fc0f700 1 --2- 172.21.2.112:0/2134774930 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4e480b7c20 0x7f4e480b8410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:53.145 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:53.150 7f4e4fc0f700 1 -- 172.21.2.112:0/2134774930 >> 172.21.2.112:0/2134774930 conn(0x7f4e48094e10 msgr2=0x7f4e480a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:53.145 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:53.150 7f4e4fc0f700 1 -- 172.21.2.112:0/2134774930 shutdown_connections 2021-02-10T11:21:53.145 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:21:53.150 7f4e4fc0f700 1 -- 172.21.2.112:0/2134774930 wait complete. 2021-02-10T11:21:53.151 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000348610 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000348610/66c99d94-f362-4a14-97c3-d0a76e6540db 2021-02-10T11:21:53.151 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:21:53.151 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:21:53.151 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:21:53.488 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:21:53.488 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:21:53.488 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348610/66c99d94-f362-4a14-97c3-d0a76e6540db/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:21:53.870 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:21:53.870 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:21:53.870 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0244091 s, 43.0 MB/s 2021-02-10T11:21:53.871 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:21:53.872 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:21:53.872 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:21:54.207 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:21:54.207 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:21:54.207 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348610/66c99d94-f362-4a14-97c3-d0a76e6540db/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:21:54.578 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:21:54.578 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:21:54.578 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.023017 s, 45.6 MB/s 2021-02-10T11:21:54.579 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:21:54.580 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:21:54.580 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:21:54.916 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:21:54.916 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:21:54.916 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348610/66c99d94-f362-4a14-97c3-d0a76e6540db/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:21:55.291 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:21:55.291 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:21:55.292 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0220993 s, 47.4 MB/s 2021-02-10T11:21:55.293 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:21:55.293 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:21:55.293 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:21:55.630 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:21:55.630 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:21:55.631 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348610/66c99d94-f362-4a14-97c3-d0a76e6540db/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:21:56.006 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:21:56.007 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:21:56.007 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0254927 s, 41.1 MB/s 2021-02-10T11:21:56.008 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:21:56.008 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:21:56.008 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:21:56.342 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:21:56.343 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:21:56.343 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348610/66c99d94-f362-4a14-97c3-d0a76e6540db/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:21:56.718 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:21:56.718 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:21:56.718 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.024442 s, 42.9 MB/s 2021-02-10T11:21:56.719 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:21:56.719 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:21:56.720 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:21:57.053 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:21:57.053 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:21:57.054 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348610/66c99d94-f362-4a14-97c3-d0a76e6540db/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:21:57.431 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:21:57.432 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:21:57.432 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0258382 s, 40.6 MB/s 2021-02-10T11:21:57.433 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:21:57.433 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:21:57.433 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:21:57.768 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:21:57.768 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:21:57.769 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348610/66c99d94-f362-4a14-97c3-d0a76e6540db/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:21:58.145 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:21:58.145 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:21:58.145 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0259773 s, 40.4 MB/s 2021-02-10T11:21:58.146 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:21:58.146 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:21:58.146 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:21:58.484 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:21:58.484 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:21:58.485 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348610/66c99d94-f362-4a14-97c3-d0a76e6540db/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:21:58.865 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:21:58.865 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:21:58.865 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0260847 s, 40.2 MB/s 2021-02-10T11:21:58.866 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:21:58.866 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:21:58.866 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:21:59.202 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:21:59.202 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:21:59.202 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348610/66c99d94-f362-4a14-97c3-d0a76e6540db/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:21:59.576 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:21:59.577 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:21:59.577 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0245915 s, 42.6 MB/s 2021-02-10T11:21:59.577 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:21:59.578 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:21:59.578 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:21:59.922 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:21:59.922 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:21:59.922 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348610/66c99d94-f362-4a14-97c3-d0a76e6540db/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:22:00.302 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:22:00.303 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:22:00.303 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0256969 s, 40.8 MB/s 2021-02-10T11:22:00.304 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000348611 2021-02-10T11:22:00.736 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:00.725 7f765856e700 1 Processor -- start 2021-02-10T11:22:00.737 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:00.725 7f765856e700 1 -- start start 2021-02-10T11:22:00.737 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:00.725 7f765856e700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f76501b0870 0x7f76500f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:00.737 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:00.725 7f765856e700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76501b0c70 con 0x7f76501b0470 2021-02-10T11:22:00.737 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:00.725 7f765856e700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76500f0b50 con 0x7f76500efe00 2021-02-10T11:22:00.737 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:00.725 7f765856e700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76500e5ea0 con 0x7f76501b0870 2021-02-10T11:22:00.737 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:00.726 7f765638a700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f76501b0470 0x7f76500ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:44460/0 (socket says 172.21.2.112:44460) 2021-02-10T11:22:00.737 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:00.726 7f765638a700 1 -- 172.21.2.112:0/639330033 learned_addr learned my addr 172.21.2.112:0/639330033 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:22:00.737 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:00.726 7f7655388700 1 -- 172.21.2.112:0/639330033 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 803040688 0 0) 0x7f76440010b0 con 0x7f76501b0470 2021-02-10T11:22:00.737 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:00.726 7f7655388700 1 -- 172.21.2.112:0/639330033 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f763c002fe0 con 0x7f76501b0470 2021-02-10T11:22:00.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:00.726 7f7655b89700 1 --2- 172.21.2.112:0/639330033 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f76501b0870 0x7f76500f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:00.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:00.726 7f7655388700 1 -- 172.21.2.112:0/639330033 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3135075415 0 0) 0x7f76440010b0 con 0x7f76501b0470 2021-02-10T11:22:00.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:00.726 7f7655388700 1 -- 172.21.2.112:0/639330033 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f76501b0870 msgr2=0x7f76500f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:00.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:00.726 7f7655388700 1 --2- 172.21.2.112:0/639330033 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f76501b0870 0x7f76500f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:00.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:00.726 7f7655388700 1 -- 172.21.2.112:0/639330033 >> v1:172.21.2.120:6789/0 conn(0x7f76500efe00 legacy=0x7f76500f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:00.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:00.726 7f7655388700 1 -- 172.21.2.112:0/639330033 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76500e52e0 con 0x7f76501b0470 2021-02-10T11:22:00.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:00.726 7f7655388700 1 -- 172.21.2.112:0/639330033 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f76440015b0 con 0x7f76501b0470 2021-02-10T11:22:00.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:00.726 7f7655388700 1 -- 172.21.2.112:0/639330033 >> v1:172.21.2.112:6789/0 conn(0x7f76501b0470 legacy=0x7f76500ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:00.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:00.726 7f7655388700 1 --2- 172.21.2.112:0/639330033 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f763c003aa0 0x7f763c005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:00.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:00.726 7f7655388700 1 --2- 172.21.2.112:0/639330033 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f763c006790 0x7f763c008c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:00.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:00.726 7f7655388700 1 --2- 172.21.2.112:0/639330033 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f763c009500 0x7f763c00ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:00.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:00.726 7f7655388700 1 -- 172.21.2.112:0/639330033 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f763c00c2d0 con 0x7f763c003aa0 2021-02-10T11:22:00.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:00.726 7f7655388700 1 -- 172.21.2.112:0/639330033 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f763c00c4f0 con 0x7f763c009500 2021-02-10T11:22:00.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:00.726 7f7655388700 1 -- 172.21.2.112:0/639330033 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f763c00c710 con 0x7f763c006790 2021-02-10T11:22:00.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:00.727 7f765638a700 1 --2- 172.21.2.112:0/639330033 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f763c003aa0 0x7f763c005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:00.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:00.727 7f7655b89700 1 --2- 172.21.2.112:0/639330033 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f763c006790 0x7f763c008c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:00.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:00.727 7f7656b8b700 1 --2- 172.21.2.112:0/639330033 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f763c009500 0x7f763c00ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:00.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:00.727 7f765638a700 1 -- 172.21.2.112:0/639330033 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f763c006790 msgr2=0x7f763c008c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:00.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:00.727 7f765638a700 1 --2- 172.21.2.112:0/639330033 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f763c006790 0x7f763c008c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:00.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:00.727 7f765638a700 1 -- 172.21.2.112:0/639330033 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f763c009500 msgr2=0x7f763c00ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:00.740 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:00.727 7f765638a700 1 --2- 172.21.2.112:0/639330033 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f763c009500 0x7f763c00ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:00.740 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:00.727 7f765638a700 1 -- 172.21.2.112:0/639330033 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f763c050980 con 0x7f763c003aa0 2021-02-10T11:22:00.740 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:00.727 7f765638a700 1 --2- 172.21.2.112:0/639330033 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f763c003aa0 0x7f763c005ef0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 rx=0x7f7644001360 tx=0x7f7644007d60).ready entity=mon.0 client_cookie=ce717fa8412542cf server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:00.740 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:00.727 7f7655388700 1 -- 172.21.2.112:0/639330033 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f764400a520 con 0x7f763c003aa0 2021-02-10T11:22:00.740 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:00.727 7f7655388700 1 -- 172.21.2.112:0/639330033 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f764400a7a0 con 0x7f763c003aa0 2021-02-10T11:22:00.740 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:00.727 7f7655388700 1 -- 172.21.2.112:0/639330033 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f764400acc0 con 0x7f763c003aa0 2021-02-10T11:22:00.740 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:00.727 7f765856e700 1 -- 172.21.2.112:0/639330033 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f763c003aa0 msgr2=0x7f763c005ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:00.740 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:00.727 7f765856e700 1 --2- 172.21.2.112:0/639330033 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f763c003aa0 0x7f763c005ef0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 rx=0x7f7644001360 tx=0x7f7644007d60).stop 2021-02-10T11:22:00.741 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:00.727 7f7656b8b700 1 -- 172.21.2.112:0/639330033 reap_dead start 2021-02-10T11:22:00.742 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:00.728 7f765856e700 1 -- 172.21.2.112:0/639330033 shutdown_connections 2021-02-10T11:22:00.742 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:00.728 7f765856e700 1 -- 172.21.2.112:0/639330033 >> 172.21.2.112:0/639330033 conn(0x7f76500ce3e0 msgr2=0x7f76500e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:00.742 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:00.728 7f765856e700 1 -- 172.21.2.112:0/639330033 shutdown_connections 2021-02-10T11:22:00.742 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:00.728 7f765856e700 1 -- 172.21.2.112:0/639330033 wait complete. 2021-02-10T11:22:00.742 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:00.743 7f765856e700 1 Processor -- start 2021-02-10T11:22:00.742 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:00.743 7f765856e700 1 -- start start 2021-02-10T11:22:00.744 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:00.743 7f765856e700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f765009b930 0x7f76500b7c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:00.744 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:00.743 7f765856e700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f76500b8410 0x7f76500b8810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:00.744 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:00.743 7f765856e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f76500c1f80 0x7f76500c2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:00.744 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:00.743 7f765856e700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76500c1a00 con 0x7f76500c1f80 2021-02-10T11:22:00.744 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:00.743 7f765856e700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76500b8e30 con 0x7f76500b8410 2021-02-10T11:22:00.744 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:00.743 7f765856e700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76500b9050 con 0x7f765009b930 2021-02-10T11:22:00.745 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:00.744 7f7656b8b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f76500c1f80 0x7f76500c2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:00.745 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:00.744 7f7656b8b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f76500c1f80 0x7f76500c2380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:45012/0 (socket says 172.21.2.112:45012) 2021-02-10T11:22:00.745 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:00.744 7f7656b8b700 1 -- 172.21.2.112:0/2758848558 learned_addr learned my addr 172.21.2.112:0/2758848558 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:22:00.745 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:00.744 7f7656b8b700 1 -- 172.21.2.112:0/2758848558 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f765009b930 msgr2=0x7f76500b7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:00.745 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:00.744 7f7656b8b700 1 --2- 172.21.2.112:0/2758848558 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f765009b930 0x7f76500b7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:00.745 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:00.744 7f7656b8b700 1 -- 172.21.2.112:0/2758848558 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f76500b8410 msgr2=0x7f76500b8810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:00.745 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:00.744 7f7656b8b700 1 --2- 172.21.2.112:0/2758848558 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f76500b8410 0x7f76500b8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:00.745 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:00.744 7f7656b8b700 1 -- 172.21.2.112:0/2758848558 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76500c29a0 con 0x7f76500c1f80 2021-02-10T11:22:00.745 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:00.744 7f7656b8b700 1 --2- 172.21.2.112:0/2758848558 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f76500c1f80 0x7f76500c2380 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 rx=0x7f7648006900 tx=0x7f764800bd20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:00.745 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:00.744 7f7642ffd700 1 -- 172.21.2.112:0/2758848558 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7648006ba0 con 0x7f76500c1f80 2021-02-10T11:22:00.746 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:00.744 7f7642ffd700 1 -- 172.21.2.112:0/2758848558 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7648000db0 con 0x7f76500c1f80 2021-02-10T11:22:00.746 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:00.744 7f7642ffd700 1 -- 172.21.2.112:0/2758848558 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7648006430 con 0x7f76500c1f80 2021-02-10T11:22:00.746 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:00.744 7f765856e700 1 -- 172.21.2.112:0/2758848558 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76500c29a0 con 0x7f76500c1f80 2021-02-10T11:22:00.746 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:00.744 7f765856e700 1 -- 172.21.2.112:0/2758848558 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76500c2f30 con 0x7f76500c1f80 2021-02-10T11:22:00.746 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:00.745 7f7642ffd700 1 -- 172.21.2.112:0/2758848558 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7648006030 con 0x7f76500c1f80 2021-02-10T11:22:00.746 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:00.745 7f7642ffd700 1 --2- 172.21.2.112:0/2758848558 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f76380186c0 0x7f763801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:00.746 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:00.745 7f7642ffd700 1 -- 172.21.2.112:0/2758848558 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f7648032b40 con 0x7f76500c1f80 2021-02-10T11:22:00.746 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:00.745 7f765638a700 1 --2- 172.21.2.112:0/2758848558 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f76380186c0 0x7f763801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:00.746 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:00.746 7f765638a700 1 --2- 172.21.2.112:0/2758848558 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f76380186c0 0x7f763801abc0 crc :-1 s=READY pgs=261 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:00.768 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:00.775 7f765856e700 1 -- 172.21.2.112:0/2758848558 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7650105b30 con 0x7f76500c1f80 2021-02-10T11:22:00.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:00.778 7f7642ffd700 1 -- 172.21.2.112:0/2758848558 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7648000d90 con 0x7f76500c1f80 2021-02-10T11:22:00.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:00.905 7f765856e700 1 -- 172.21.2.112:0/2758848558 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348611", "target": ["mgr", ""]}) v1 -- 0x7f76500f0b50 con 0x7f76380186c0 2021-02-10T11:22:00.925 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:00.932 7f7642ffd700 1 -- 172.21.2.112:0/2758848558 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f7644001590 con 0x7f76380186c0 2021-02-10T11:22:00.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:00.939 7f765856e700 1 -- 172.21.2.112:0/2758848558 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f76380186c0 msgr2=0x7f763801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:00.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:00.939 7f765856e700 1 --2- 172.21.2.112:0/2758848558 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f76380186c0 0x7f763801abc0 crc :-1 s=READY pgs=261 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:00.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:00.939 7f765856e700 1 -- 172.21.2.112:0/2758848558 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f76500c1f80 msgr2=0x7f76500c2380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:00.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:00.939 7f765856e700 1 --2- 172.21.2.112:0/2758848558 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f76500c1f80 0x7f76500c2380 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 rx=0x7f7648006900 tx=0x7f764800bd20).stop 2021-02-10T11:22:00.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:00.939 7f765856e700 1 -- 172.21.2.112:0/2758848558 shutdown_connections 2021-02-10T11:22:00.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:00.939 7f765856e700 1 --2- 172.21.2.112:0/2758848558 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f76380186c0 0x7f763801abc0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:00.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:00.939 7f765856e700 1 --2- 172.21.2.112:0/2758848558 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f76500b8410 0x7f76500b8810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:00.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:00.939 7f765856e700 1 --2- 172.21.2.112:0/2758848558 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f76500c1f80 0x7f76500c2380 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:00.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:00.939 7f765856e700 1 --2- 172.21.2.112:0/2758848558 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f765009b930 0x7f76500b7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:00.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:00.939 7f765856e700 1 -- 172.21.2.112:0/2758848558 >> 172.21.2.112:0/2758848558 conn(0x7f7650094e10 msgr2=0x7f76500a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:00.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:00.939 7f765856e700 1 -- 172.21.2.112:0/2758848558 shutdown_connections 2021-02-10T11:22:00.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:00.939 7f765856e700 1 -- 172.21.2.112:0/2758848558 wait complete. 2021-02-10T11:22:00.940 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000348611 2021-02-10T11:22:01.371 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:01.361 7f0fee8df700 1 Processor -- start 2021-02-10T11:22:01.372 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:01.361 7f0fee8df700 1 -- start start 2021-02-10T11:22:01.372 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:01.362 7f0fee8df700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0fe81b0470 0x7f0fe81b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:01.372 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:01.362 7f0fee8df700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f0fe80ef610 0x7f0fe80efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:01.372 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:01.362 7f0fee8df700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f0fe80f0030 0x7f0fe80f0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:01.372 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:01.362 7f0fee8df700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fe80e5ea0 con 0x7f0fe80ef610 2021-02-10T11:22:01.372 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:01.362 7f0fee8df700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fe80f0b50 con 0x7f0fe80f0030 2021-02-10T11:22:01.372 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:01.362 7f0fee8df700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fe80e52e0 con 0x7f0fe81b0470 2021-02-10T11:22:01.372 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:01.362 7f0fe77fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f0fe80ef610 0x7f0fe80efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:01.372 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:01.362 7f0fe77fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f0fe80ef610 0x7f0fe80efa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:45018/0 (socket says 172.21.2.112:45018) 2021-02-10T11:22:01.372 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:01.362 7f0fe77fe700 1 -- 172.21.2.112:0/4273531957 learned_addr learned my addr 172.21.2.112:0/4273531957 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:22:01.372 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:01.362 7f0fe77fe700 1 -- 172.21.2.112:0/4273531957 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0fe81b0470 msgr2=0x7f0fe81b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:01.372 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:01.362 7f0fe77fe700 1 --2- 172.21.2.112:0/4273531957 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0fe81b0470 0x7f0fe81b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:01.372 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:01.363 7f0fe77fe700 1 -- 172.21.2.112:0/4273531957 >> v2:172.21.2.120:3300/0 conn(0x7f0fe80f0030 msgr2=0x7f0fe80f0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:01.373 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:01.363 7f0fe77fe700 1 --2- 172.21.2.112:0/4273531957 >> v2:172.21.2.120:3300/0 conn(0x7f0fe80f0030 0x7f0fe80f0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:01.373 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:01.363 7f0fe77fe700 1 -- 172.21.2.112:0/4273531957 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fe80f10d0 con 0x7f0fe80ef610 2021-02-10T11:22:01.373 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:01.363 7f0fe77fe700 1 --2- 172.21.2.112:0/4273531957 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0fe80ef610 0x7f0fe80efa10 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 rx=0x7f0fdc006700 tx=0x7f0fdc009b00).ready entity=mon.0 client_cookie=3178b9f5940f1782 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:01.373 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:01.363 7f0fe6ffd700 1 -- 172.21.2.112:0/4273531957 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0fdc00c0c0 con 0x7f0fe80ef610 2021-02-10T11:22:01.373 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:01.363 7f0fe6ffd700 1 -- 172.21.2.112:0/4273531957 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fdc00c400 con 0x7f0fe80ef610 2021-02-10T11:22:01.373 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:01.363 7f0fe6ffd700 1 -- 172.21.2.112:0/4273531957 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0fdc00c920 con 0x7f0fe80ef610 2021-02-10T11:22:01.373 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:01.363 7f0fee8df700 1 -- 172.21.2.112:0/4273531957 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0fe80ef610 msgr2=0x7f0fe80efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:01.373 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:01.363 7f0fee8df700 1 --2- 172.21.2.112:0/4273531957 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0fe80ef610 0x7f0fe80efa10 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 rx=0x7f0fdc006700 tx=0x7f0fdc009b00).stop 2021-02-10T11:22:01.373 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:01.364 7f0fee8df700 1 -- 172.21.2.112:0/4273531957 shutdown_connections 2021-02-10T11:22:01.373 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:01.364 7f0fee8df700 1 --2- 172.21.2.112:0/4273531957 >> v2:172.21.2.120:3300/0 conn(0x7f0fe80f0030 0x7f0fe80f0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:01.373 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:01.364 7f0fee8df700 1 --2- 172.21.2.112:0/4273531957 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0fe80ef610 0x7f0fe80efa10 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:01.373 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:01.364 7f0fee8df700 1 --2- 172.21.2.112:0/4273531957 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0fe81b0470 0x7f0fe81b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:01.373 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:01.364 7f0fee8df700 1 -- 172.21.2.112:0/4273531957 >> 172.21.2.112:0/4273531957 conn(0x7f0fe80ce3e0 msgr2=0x7f0fe80e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:01.374 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:01.364 7f0fee8df700 1 -- 172.21.2.112:0/4273531957 shutdown_connections 2021-02-10T11:22:01.374 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:01.364 7f0fee8df700 1 -- 172.21.2.112:0/4273531957 wait complete. 2021-02-10T11:22:01.375 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:01.378 7f0fee8df700 1 Processor -- start 2021-02-10T11:22:01.375 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:01.379 7f0fee8df700 1 -- start start 2021-02-10T11:22:01.375 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:01.379 7f0fee8df700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0fe80b20b0 0x7f0fe80ac970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:01.375 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:01.379 7f0fee8df700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0fe80b24b0 0x7f0fe8191dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:01.375 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:01.379 7f0fee8df700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0fe80acf90 0x7f0fe81923e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:01.375 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:01.379 7f0fee8df700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fe80b1b30 con 0x7f0fe80b20b0 2021-02-10T11:22:01.376 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:01.379 7f0fee8df700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fe81a04e0 con 0x7f0fe80b24b0 2021-02-10T11:22:01.376 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:01.379 7f0fee8df700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fe81a06c0 con 0x7f0fe80acf90 2021-02-10T11:22:01.376 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:01.379 7f0fe7fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0fe80b20b0 0x7f0fe80ac970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:01.376 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:01.379 7f0fe7fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0fe80b20b0 0x7f0fe80ac970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:45024/0 (socket says 172.21.2.112:45024) 2021-02-10T11:22:01.376 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:01.379 7f0fe7fff700 1 -- 172.21.2.112:0/2647571247 learned_addr learned my addr 172.21.2.112:0/2647571247 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:22:01.376 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:01.379 7f0fe7fff700 1 -- 172.21.2.112:0/2647571247 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0fe80acf90 msgr2=0x7f0fe81923e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:01.376 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:01.379 7f0fe7fff700 1 --2- 172.21.2.112:0/2647571247 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0fe80acf90 0x7f0fe81923e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:01.376 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:01.380 7f0fe7fff700 1 -- 172.21.2.112:0/2647571247 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0fe80b24b0 msgr2=0x7f0fe8191dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:01.376 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:01.380 7f0fe7fff700 1 --2- 172.21.2.112:0/2647571247 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0fe80b24b0 0x7f0fe8191dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:01.376 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:01.380 7f0fe7fff700 1 -- 172.21.2.112:0/2647571247 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fe8192be0 con 0x7f0fe80b20b0 2021-02-10T11:22:01.376 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:01.380 7f0fe7fff700 1 --2- 172.21.2.112:0/2647571247 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0fe80b20b0 0x7f0fe80ac970 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 rx=0x7f0fd8006f00 tx=0x7f0fd800b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:01.376 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:01.380 7f0fe4ff9700 1 -- 172.21.2.112:0/2647571247 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0fd8006350 con 0x7f0fe80b20b0 2021-02-10T11:22:01.377 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:01.380 7f0fe4ff9700 1 -- 172.21.2.112:0/2647571247 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fd800be00 con 0x7f0fe80b20b0 2021-02-10T11:22:01.377 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:01.380 7f0fe4ff9700 1 -- 172.21.2.112:0/2647571247 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0fd8002020 con 0x7f0fe80b20b0 2021-02-10T11:22:01.377 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:01.380 7f0fee8df700 1 -- 172.21.2.112:0/2647571247 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fe8192be0 con 0x7f0fe80b20b0 2021-02-10T11:22:01.377 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:01.380 7f0fee8df700 1 -- 172.21.2.112:0/2647571247 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fe8193170 con 0x7f0fe80b20b0 2021-02-10T11:22:01.377 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:01.380 7f0fe4ff9700 1 -- 172.21.2.112:0/2647571247 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0fd8005310 con 0x7f0fe80b20b0 2021-02-10T11:22:01.377 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:01.381 7f0fe4ff9700 1 --2- 172.21.2.112:0/2647571247 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0fc80186c0 0x7f0fc801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:01.377 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:01.381 7f0fe4ff9700 1 -- 172.21.2.112:0/2647571247 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f0fd8032570 con 0x7f0fe80b20b0 2021-02-10T11:22:01.377 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:01.381 7f0fe77fe700 1 --2- 172.21.2.112:0/2647571247 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0fc80186c0 0x7f0fc801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:01.377 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:01.382 7f0fe77fe700 1 --2- 172.21.2.112:0/2647571247 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0fc80186c0 0x7f0fc801abc0 crc :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:01.405 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:01.412 7f0fee8df700 1 -- 172.21.2.112:0/2647571247 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fe8105af0 con 0x7f0fe80b20b0 2021-02-10T11:22:01.408 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:01.415 7f0fe4ff9700 1 -- 172.21.2.112:0/2647571247 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0fd8000940 con 0x7f0fe80b20b0 2021-02-10T11:22:01.538 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:01.545 7f0fee8df700 1 -- 172.21.2.112:0/2647571247 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348611", "target": ["mgr", ""]}) v1 -- 0x7f0fe80b1b30 con 0x7f0fc80186c0 2021-02-10T11:22:01.542 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:01.549 7f0fe4ff9700 1 -- 172.21.2.112:0/2647571247 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f0fdc0091c0 con 0x7f0fc80186c0 2021-02-10T11:22:01.545 INFO:teuthology.orchestra.run.gibba012.stdout:/volumes/_nogroup/subvolume_0000000000348611/38aa7c27-5916-420b-af64-87c50cbebe94 2021-02-10T11:22:01.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:01.553 7f0fee8df700 1 -- 172.21.2.112:0/2647571247 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0fc80186c0 msgr2=0x7f0fc801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:01.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:01.553 7f0fee8df700 1 --2- 172.21.2.112:0/2647571247 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0fc80186c0 0x7f0fc801abc0 crc :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:01.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:01.553 7f0fee8df700 1 -- 172.21.2.112:0/2647571247 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0fe80b20b0 msgr2=0x7f0fe80ac970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:01.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:01.553 7f0fee8df700 1 --2- 172.21.2.112:0/2647571247 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0fe80b20b0 0x7f0fe80ac970 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 rx=0x7f0fd8006f00 tx=0x7f0fd800b940).stop 2021-02-10T11:22:01.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:01.553 7f0fee8df700 1 -- 172.21.2.112:0/2647571247 shutdown_connections 2021-02-10T11:22:01.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:01.553 7f0fee8df700 1 --2- 172.21.2.112:0/2647571247 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0fc80186c0 0x7f0fc801abc0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:01.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:01.553 7f0fee8df700 1 --2- 172.21.2.112:0/2647571247 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0fe80b24b0 0x7f0fe8191dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:01.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:01.553 7f0fee8df700 1 --2- 172.21.2.112:0/2647571247 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0fe80acf90 0x7f0fe81923e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:01.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:01.553 7f0fee8df700 1 --2- 172.21.2.112:0/2647571247 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0fe80b20b0 0x7f0fe80ac970 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:01.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:01.553 7f0fee8df700 1 -- 172.21.2.112:0/2647571247 >> 172.21.2.112:0/2647571247 conn(0x7f0fe80bf4e0 msgr2=0x7f0fe80be3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:01.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:01.554 7f0fee8df700 1 -- 172.21.2.112:0/2647571247 shutdown_connections 2021-02-10T11:22:01.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:01.554 7f0fee8df700 1 -- 172.21.2.112:0/2647571247 wait complete. 2021-02-10T11:22:01.555 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000348611 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000348611/38aa7c27-5916-420b-af64-87c50cbebe94 2021-02-10T11:22:01.555 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:22:01.555 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:22:01.555 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:22:01.889 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:22:01.890 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:22:01.890 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348611/38aa7c27-5916-420b-af64-87c50cbebe94/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:22:02.268 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:22:02.268 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:22:02.268 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0221508 s, 47.3 MB/s 2021-02-10T11:22:02.269 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:22:02.270 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:22:02.270 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:22:02.605 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:22:02.606 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:22:02.606 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348611/38aa7c27-5916-420b-af64-87c50cbebe94/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:22:02.982 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:22:02.983 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:22:02.983 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0243649 s, 43.0 MB/s 2021-02-10T11:22:02.983 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:22:02.984 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:22:02.984 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:22:03.320 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:22:03.320 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:22:03.320 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348611/38aa7c27-5916-420b-af64-87c50cbebe94/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:22:03.696 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:22:03.697 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:22:03.697 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0238188 s, 44.0 MB/s 2021-02-10T11:22:03.698 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:22:03.698 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:22:03.698 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:22:04.034 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:22:04.034 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:22:04.034 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348611/38aa7c27-5916-420b-af64-87c50cbebe94/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:22:04.403 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:22:04.403 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:22:04.403 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0233282 s, 44.9 MB/s 2021-02-10T11:22:04.404 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:22:04.404 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:22:04.405 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:22:04.739 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:22:04.739 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:22:04.739 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348611/38aa7c27-5916-420b-af64-87c50cbebe94/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:22:05.115 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:22:05.115 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:22:05.115 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0242408 s, 43.3 MB/s 2021-02-10T11:22:05.116 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:22:05.116 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:22:05.116 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:22:05.453 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:22:05.453 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:22:05.454 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348611/38aa7c27-5916-420b-af64-87c50cbebe94/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:22:05.830 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:22:05.830 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:22:05.830 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0252431 s, 41.5 MB/s 2021-02-10T11:22:05.831 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:22:05.831 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:22:05.831 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:22:06.166 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:22:06.166 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:22:06.167 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348611/38aa7c27-5916-420b-af64-87c50cbebe94/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:22:06.541 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:22:06.541 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:22:06.542 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0243663 s, 43.0 MB/s 2021-02-10T11:22:06.542 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:22:06.543 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:22:06.543 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:22:06.879 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:22:06.879 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:22:06.879 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348611/38aa7c27-5916-420b-af64-87c50cbebe94/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:22:07.195 DEBUG:teuthology.orchestra.run.gibba012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:22:07.198 DEBUG:teuthology.orchestra.run.gibba020:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:22:07.254 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:22:07.255 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:22:07.255 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0248388 s, 42.2 MB/s 2021-02-10T11:22:07.256 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:22:07.256 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:22:07.256 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:22:07.533 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:22:07.533 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:22:07.534 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348611/38aa7c27-5916-420b-af64-87c50cbebe94/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:22:07.907 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:22:07.907 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:22:07.907 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0230045 s, 45.6 MB/s 2021-02-10T11:22:07.909 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:22:07.909 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:22:07.909 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:22:08.243 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:22:08.243 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:22:08.243 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348611/38aa7c27-5916-420b-af64-87c50cbebe94/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:22:08.619 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:22:08.620 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:22:08.620 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0242741 s, 43.2 MB/s 2021-02-10T11:22:08.621 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000348612 2021-02-10T11:22:09.053 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.042 7fc8a852b700 1 Processor -- start 2021-02-10T11:22:09.053 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.042 7fc8a852b700 1 -- start start 2021-02-10T11:22:09.053 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.042 7fc8a852b700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fc8a00f05f0 0x7fc8a00f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:09.053 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.042 7fc8a852b700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8a01b0c70 con 0x7fc8a01b0870 2021-02-10T11:22:09.053 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.042 7fc8a852b700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8a00f0b50 con 0x7fc8a01b0470 2021-02-10T11:22:09.053 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.042 7fc8a852b700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8a00e5ea0 con 0x7fc8a00f05f0 2021-02-10T11:22:09.053 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.043 7fc8a5b46700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fc8a01b0870 0x7fc8a00efe00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:44492/0 (socket says 172.21.2.112:44492) 2021-02-10T11:22:09.053 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.043 7fc8a5b46700 1 -- 172.21.2.112:0/1524671672 learned_addr learned my addr 172.21.2.112:0/1524671672 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:22:09.053 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.043 7fc8a6b48700 1 --2- 172.21.2.112:0/1524671672 >> v2:172.21.2.120:3300/0 conn(0x7fc8a00f05f0 0x7fc8a00f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:09.053 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.043 7fc8a5345700 1 -- 172.21.2.112:0/1524671672 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3479800529 0 0) 0x7fc89c001180 con 0x7fc8a01b0870 2021-02-10T11:22:09.054 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.043 7fc8a5345700 1 -- 172.21.2.112:0/1524671672 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc88c002fe0 con 0x7fc8a01b0870 2021-02-10T11:22:09.054 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.043 7fc8a5345700 1 -- 172.21.2.112:0/1524671672 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3756805037 0 0) 0x7fc89c0010e0 con 0x7fc8a01b0870 2021-02-10T11:22:09.054 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.043 7fc8a5345700 1 -- 172.21.2.112:0/1524671672 >> v2:172.21.2.120:3300/0 conn(0x7fc8a00f05f0 msgr2=0x7fc8a00f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:09.054 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.043 7fc8a5345700 1 --2- 172.21.2.112:0/1524671672 >> v2:172.21.2.120:3300/0 conn(0x7fc8a00f05f0 0x7fc8a00f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:09.054 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.043 7fc8a5345700 1 -- 172.21.2.112:0/1524671672 >> v1:172.21.2.120:6789/0 conn(0x7fc8a01b0470 legacy=0x7fc8a00ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:09.054 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.043 7fc8a5345700 1 -- 172.21.2.112:0/1524671672 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8a00e52e0 con 0x7fc8a01b0870 2021-02-10T11:22:09.054 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.043 7fc8a5345700 1 -- 172.21.2.112:0/1524671672 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7fc89c001600 con 0x7fc8a01b0870 2021-02-10T11:22:09.054 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.043 7fc8a5345700 1 -- 172.21.2.112:0/1524671672 >> v1:172.21.2.112:6789/0 conn(0x7fc8a01b0870 legacy=0x7fc8a00efe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:09.054 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.043 7fc8a5345700 1 --2- 172.21.2.112:0/1524671672 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc88c003ae0 0x7fc88c005f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:09.054 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.043 7fc8a5345700 1 --2- 172.21.2.112:0/1524671672 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc88c006830 0x7fc88c008d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:09.054 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.043 7fc8a5345700 1 --2- 172.21.2.112:0/1524671672 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc88c009630 0x7fc88c00bb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:09.054 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.043 7fc8a5345700 1 -- 172.21.2.112:0/1524671672 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc88c00c3a0 con 0x7fc88c003ae0 2021-02-10T11:22:09.054 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.043 7fc8a5345700 1 -- 172.21.2.112:0/1524671672 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc88c00c5c0 con 0x7fc88c006830 2021-02-10T11:22:09.055 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.043 7fc8a5345700 1 -- 172.21.2.112:0/1524671672 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc88c00c7e0 con 0x7fc88c009630 2021-02-10T11:22:09.055 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.043 7fc8a6347700 1 --2- 172.21.2.112:0/1524671672 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc88c003ae0 0x7fc88c005f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:09.055 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.044 7fc8a5b46700 1 --2- 172.21.2.112:0/1524671672 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc88c006830 0x7fc88c008d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:09.055 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.044 7fc8a6b48700 1 --2- 172.21.2.112:0/1524671672 >> v2:172.21.2.120:3300/0 conn(0x7fc8a00f05f0 0x7fc8a00f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:22:09.055 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.044 7fc8a6347700 1 -- 172.21.2.112:0/1524671672 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc88c009630 msgr2=0x7fc88c00bb50 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:22:09.055 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.044 7fc8a6347700 1 --2- 172.21.2.112:0/1524671672 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc88c009630 0x7fc88c00bb50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:09.055 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.044 7fc8a6347700 1 -- 172.21.2.112:0/1524671672 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc88c006830 msgr2=0x7fc88c008d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:09.055 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.044 7fc8a6347700 1 --2- 172.21.2.112:0/1524671672 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc88c006830 0x7fc88c008d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:09.055 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.044 7fc8a6347700 1 -- 172.21.2.112:0/1524671672 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc88c050a50 con 0x7fc88c003ae0 2021-02-10T11:22:09.055 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.044 7fc8a6b48700 1 -- 172.21.2.112:0/1524671672 reap_dead start 2021-02-10T11:22:09.055 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.044 7fc8a6347700 1 --2- 172.21.2.112:0/1524671672 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc88c003ae0 0x7fc88c005f90 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 rx=0x7fc894006ab0 tx=0x7fc894009b40).ready entity=mon.0 client_cookie=72461673e5c196fe server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:09.055 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.044 7fc8a5345700 1 -- 172.21.2.112:0/1524671672 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc89400b2e0 con 0x7fc88c003ae0 2021-02-10T11:22:09.055 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.044 7fc8a5345700 1 -- 172.21.2.112:0/1524671672 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc89400b520 con 0x7fc88c003ae0 2021-02-10T11:22:09.056 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.044 7fc8a5345700 1 -- 172.21.2.112:0/1524671672 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc89400b970 con 0x7fc88c003ae0 2021-02-10T11:22:09.056 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.044 7fc8a852b700 1 -- 172.21.2.112:0/1524671672 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc88c003ae0 msgr2=0x7fc88c005f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:09.056 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.044 7fc8a852b700 1 --2- 172.21.2.112:0/1524671672 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc88c003ae0 0x7fc88c005f90 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 rx=0x7fc894006ab0 tx=0x7fc894009b40).stop 2021-02-10T11:22:09.056 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.045 7fc8a852b700 1 -- 172.21.2.112:0/1524671672 shutdown_connections 2021-02-10T11:22:09.056 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.045 7fc8a852b700 1 --2- 172.21.2.112:0/1524671672 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc88c003ae0 0x7fc88c005f90 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:09.056 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.045 7fc8a852b700 1 -- 172.21.2.112:0/1524671672 >> 172.21.2.112:0/1524671672 conn(0x7fc8a00ce3e0 msgr2=0x7fc8a00e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:09.056 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.045 7fc8a852b700 1 -- 172.21.2.112:0/1524671672 shutdown_connections 2021-02-10T11:22:09.056 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.045 7fc8a852b700 1 -- 172.21.2.112:0/1524671672 wait complete. 2021-02-10T11:22:09.056 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.060 7fc8a852b700 1 Processor -- start 2021-02-10T11:22:09.056 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.060 7fc8a852b700 1 -- start start 2021-02-10T11:22:09.058 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.060 7fc8a852b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc8a009b930 0x7fc8a00b7c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:09.058 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.060 7fc8a852b700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc8a00b8410 0x7fc8a00b8810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:09.058 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.060 7fc8a852b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc8a00c1f80 0x7fc8a00c2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:09.058 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.060 7fc8a852b700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8a00c1a00 con 0x7fc8a00c1f80 2021-02-10T11:22:09.058 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.060 7fc8a852b700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8a00b8e30 con 0x7fc8a00b8410 2021-02-10T11:22:09.058 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.060 7fc8a852b700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8a00b9050 con 0x7fc8a009b930 2021-02-10T11:22:09.058 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.060 7fc8a6b48700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc8a00c1f80 0x7fc8a00c2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:09.058 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.060 7fc8a6b48700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc8a00c1f80 0x7fc8a00c2380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:45044/0 (socket says 172.21.2.112:45044) 2021-02-10T11:22:09.058 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.060 7fc8a6b48700 1 -- 172.21.2.112:0/3546658320 learned_addr learned my addr 172.21.2.112:0/3546658320 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:22:09.058 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.061 7fc8a6b48700 1 -- 172.21.2.112:0/3546658320 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc8a009b930 msgr2=0x7fc8a00b7c20 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:22:09.058 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.061 7fc8a6b48700 1 --2- 172.21.2.112:0/3546658320 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc8a009b930 0x7fc8a00b7c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:09.058 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.061 7fc8a6b48700 1 -- 172.21.2.112:0/3546658320 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc8a00b8410 msgr2=0x7fc8a00b8810 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:22:09.059 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.061 7fc8a6b48700 1 --2- 172.21.2.112:0/3546658320 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc8a00b8410 0x7fc8a00b8810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:09.059 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.061 7fc8a6b48700 1 -- 172.21.2.112:0/3546658320 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8a00c29a0 con 0x7fc8a00c1f80 2021-02-10T11:22:09.059 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.061 7fc8a6b48700 1 --2- 172.21.2.112:0/3546658320 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc8a00c1f80 0x7fc8a00c2380 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 rx=0x7fc898005fd0 tx=0x7fc89800acf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:09.059 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.061 7fc892ffd700 1 -- 172.21.2.112:0/3546658320 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc89800c300 con 0x7fc8a00c1f80 2021-02-10T11:22:09.059 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.061 7fc892ffd700 1 -- 172.21.2.112:0/3546658320 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc898003030 con 0x7fc8a00c1f80 2021-02-10T11:22:09.059 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.061 7fc8a852b700 1 -- 172.21.2.112:0/3546658320 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8a00c29a0 con 0x7fc8a00c1f80 2021-02-10T11:22:09.059 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.061 7fc8a852b700 1 -- 172.21.2.112:0/3546658320 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8a00c29a0 con 0x7fc8a00c1f80 2021-02-10T11:22:09.059 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.061 7fc892ffd700 1 -- 172.21.2.112:0/3546658320 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc89800c870 con 0x7fc8a00c1f80 2021-02-10T11:22:09.059 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.062 7fc892ffd700 1 -- 172.21.2.112:0/3546658320 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc898000e70 con 0x7fc8a00c1f80 2021-02-10T11:22:09.059 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.062 7fc892ffd700 1 --2- 172.21.2.112:0/3546658320 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc888018730 0x7fc88801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:09.059 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.062 7fc892ffd700 1 -- 172.21.2.112:0/3546658320 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fc898032b60 con 0x7fc8a00c1f80 2021-02-10T11:22:09.059 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.062 7fc8a6347700 1 --2- 172.21.2.112:0/3546658320 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc888018730 0x7fc88801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:09.059 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.063 7fc8a6347700 1 --2- 172.21.2.112:0/3546658320 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc888018730 0x7fc88801ac10 crc :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:09.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.092 7fc8a852b700 1 -- 172.21.2.112:0/3546658320 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8a0105b30 con 0x7fc8a00c1f80 2021-02-10T11:22:09.088 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.095 7fc892ffd700 1 -- 172.21.2.112:0/3546658320 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc898001ee0 con 0x7fc8a00c1f80 2021-02-10T11:22:09.214 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.221 7fc8a852b700 1 -- 172.21.2.112:0/3546658320 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348612", "target": ["mgr", ""]}) v1 -- 0x7fc8a0105b30 con 0x7fc888018730 2021-02-10T11:22:09.240 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.248 7fc892ffd700 1 -- 172.21.2.112:0/3546658320 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fc894006c80 con 0x7fc888018730 2021-02-10T11:22:09.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.254 7fc8a852b700 1 -- 172.21.2.112:0/3546658320 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc888018730 msgr2=0x7fc88801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:09.248 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.254 7fc8a852b700 1 --2- 172.21.2.112:0/3546658320 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc888018730 0x7fc88801ac10 crc :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:09.248 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.254 7fc8a852b700 1 -- 172.21.2.112:0/3546658320 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc8a00c1f80 msgr2=0x7fc8a00c2380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:09.248 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.254 7fc8a852b700 1 --2- 172.21.2.112:0/3546658320 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc8a00c1f80 0x7fc8a00c2380 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 rx=0x7fc898005fd0 tx=0x7fc89800acf0).stop 2021-02-10T11:22:09.248 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.254 7fc8a852b700 1 -- 172.21.2.112:0/3546658320 shutdown_connections 2021-02-10T11:22:09.248 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.254 7fc8a852b700 1 --2- 172.21.2.112:0/3546658320 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc888018730 0x7fc88801ac10 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:09.248 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.254 7fc8a852b700 1 --2- 172.21.2.112:0/3546658320 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc8a00b8410 0x7fc8a00b8810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:09.248 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.254 7fc8a852b700 1 --2- 172.21.2.112:0/3546658320 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc8a00c1f80 0x7fc8a00c2380 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:09.248 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.254 7fc8a852b700 1 --2- 172.21.2.112:0/3546658320 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc8a009b930 0x7fc8a00b7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:09.248 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.254 7fc8a852b700 1 -- 172.21.2.112:0/3546658320 >> 172.21.2.112:0/3546658320 conn(0x7fc8a0094e10 msgr2=0x7fc8a00a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:09.248 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.254 7fc8a852b700 1 -- 172.21.2.112:0/3546658320 shutdown_connections 2021-02-10T11:22:09.248 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.254 7fc8a852b700 1 -- 172.21.2.112:0/3546658320 wait complete. 2021-02-10T11:22:09.255 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000348612 2021-02-10T11:22:09.686 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.676 7f4be54ac700 1 Processor -- start 2021-02-10T11:22:09.686 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.676 7f4be54ac700 1 -- start start 2021-02-10T11:22:09.686 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.676 7f4be54ac700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f4be00f05f0 0x7f4be00f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:09.686 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.676 7f4be54ac700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4be01b0c70 con 0x7f4be01b0870 2021-02-10T11:22:09.686 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.676 7f4be54ac700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4be00f0b50 con 0x7f4be01b0470 2021-02-10T11:22:09.686 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.676 7f4be54ac700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4be00e5ea0 con 0x7f4be00f05f0 2021-02-10T11:22:09.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.677 7f4bdeabe700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f4be01b0870 0x7f4be00efe00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:44510/0 (socket says 172.21.2.112:44510) 2021-02-10T11:22:09.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.677 7f4bdeabe700 1 -- 172.21.2.112:0/613212718 learned_addr learned my addr 172.21.2.112:0/613212718 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:22:09.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.677 7f4bdfac0700 1 --2- 172.21.2.112:0/613212718 >> v2:172.21.2.120:3300/0 conn(0x7f4be00f05f0 0x7f4be00f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:09.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.677 7f4bde2bd700 1 -- 172.21.2.112:0/613212718 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 399476733 0 0) 0x7f4bd00010b0 con 0x7f4be01b0870 2021-02-10T11:22:09.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.677 7f4bde2bd700 1 -- 172.21.2.112:0/613212718 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4bcc002fe0 con 0x7f4be01b0870 2021-02-10T11:22:09.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.677 7f4bdfac0700 1 -- 172.21.2.112:0/613212718 >> v1:172.21.2.120:6789/0 conn(0x7f4be01b0470 legacy=0x7f4be00ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:09.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.677 7f4bdfac0700 1 -- 172.21.2.112:0/613212718 >> v1:172.21.2.112:6789/0 conn(0x7f4be01b0870 legacy=0x7f4be00efe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:09.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.677 7f4bdfac0700 1 -- 172.21.2.112:0/613212718 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4be00e52e0 con 0x7f4be00f05f0 2021-02-10T11:22:09.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.677 7f4bdfac0700 1 --2- 172.21.2.112:0/613212718 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4be00f05f0 0x7f4be00f20e0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f4bd8006e50 tx=0x7f4bd8009e70).ready entity=mon.1 client_cookie=c95770c9f201dac server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:09.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.678 7f4bde2bd700 1 -- 172.21.2.112:0/613212718 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4bd800b510 con 0x7f4be00f05f0 2021-02-10T11:22:09.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.678 7f4bde2bd700 1 -- 172.21.2.112:0/613212718 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bd8006790 con 0x7f4be00f05f0 2021-02-10T11:22:09.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.678 7f4bde2bd700 1 -- 172.21.2.112:0/613212718 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4bd800b940 con 0x7f4be00f05f0 2021-02-10T11:22:09.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.678 7f4be54ac700 1 -- 172.21.2.112:0/613212718 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4be00f05f0 msgr2=0x7f4be00f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:09.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.678 7f4be54ac700 1 --2- 172.21.2.112:0/613212718 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4be00f05f0 0x7f4be00f20e0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f4bd8006e50 tx=0x7f4bd8009e70).stop 2021-02-10T11:22:09.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.679 7f4be54ac700 1 -- 172.21.2.112:0/613212718 shutdown_connections 2021-02-10T11:22:09.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.679 7f4be54ac700 1 --2- 172.21.2.112:0/613212718 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4be00f05f0 0x7f4be00f20e0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:09.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.679 7f4be54ac700 1 -- 172.21.2.112:0/613212718 >> 172.21.2.112:0/613212718 conn(0x7f4be00ce3e0 msgr2=0x7f4be00e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:09.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.679 7f4be54ac700 1 -- 172.21.2.112:0/613212718 shutdown_connections 2021-02-10T11:22:09.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.679 7f4be54ac700 1 -- 172.21.2.112:0/613212718 wait complete. 2021-02-10T11:22:09.690 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.693 7f4be54ac700 1 Processor -- start 2021-02-10T11:22:09.690 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.693 7f4be54ac700 1 -- start start 2021-02-10T11:22:09.690 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.694 7f4be54ac700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4be00c36a0 0x7f4be00c2b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:09.690 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.694 7f4be54ac700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4be00c3190 0x7f4be00b1d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:09.690 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.694 7f4be54ac700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4be00b2370 0x7f4be00b2d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:09.690 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.694 7f4be54ac700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4be00b27c0 con 0x7f4be00b2370 2021-02-10T11:22:09.690 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.694 7f4be54ac700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4be00b3360 con 0x7f4be00c3190 2021-02-10T11:22:09.690 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.694 7f4be54ac700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4be00b3540 con 0x7f4be00c36a0 2021-02-10T11:22:09.690 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.694 7f4bdfac0700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4be00b2370 0x7f4be00b2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:09.690 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.694 7f4bdfac0700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4be00b2370 0x7f4be00b2d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:45056/0 (socket says 172.21.2.112:45056) 2021-02-10T11:22:09.690 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.694 7f4bdfac0700 1 -- 172.21.2.112:0/1150219870 learned_addr learned my addr 172.21.2.112:0/1150219870 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:22:09.690 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.694 7f4bdeabe700 1 --2- 172.21.2.112:0/1150219870 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4be00c3190 0x7f4be00b1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:09.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.694 7f4bdf2bf700 1 --2- 172.21.2.112:0/1150219870 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4be00c36a0 0x7f4be00c2b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:09.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.694 7f4bdfac0700 1 -- 172.21.2.112:0/1150219870 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4be00c36a0 msgr2=0x7f4be00c2b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:09.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.694 7f4bdfac0700 1 --2- 172.21.2.112:0/1150219870 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4be00c36a0 0x7f4be00c2b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:09.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.694 7f4bdfac0700 1 -- 172.21.2.112:0/1150219870 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4be00c3190 msgr2=0x7f4be00b1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:09.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.694 7f4bdfac0700 1 --2- 172.21.2.112:0/1150219870 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4be00c3190 0x7f4be00b1d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:09.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.694 7f4bdfac0700 1 -- 172.21.2.112:0/1150219870 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4be00b3760 con 0x7f4be00b2370 2021-02-10T11:22:09.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.695 7f4bdeabe700 1 --2- 172.21.2.112:0/1150219870 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4be00c3190 0x7f4be00b1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:22:09.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.695 7f4bdfac0700 1 --2- 172.21.2.112:0/1150219870 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4be00b2370 0x7f4be00b2d40 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 rx=0x7f4bd80091d0 tx=0x7f4bd800bd60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:09.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.695 7f4bd7fff700 1 -- 172.21.2.112:0/1150219870 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4bd8006ca0 con 0x7f4be00b2370 2021-02-10T11:22:09.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.695 7f4bd7fff700 1 -- 172.21.2.112:0/1150219870 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bd800c2f0 con 0x7f4be00b2370 2021-02-10T11:22:09.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.695 7f4bd7fff700 1 -- 172.21.2.112:0/1150219870 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4bd800c7d0 con 0x7f4be00b2370 2021-02-10T11:22:09.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.695 7f4be54ac700 1 -- 172.21.2.112:0/1150219870 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4be00b3760 con 0x7f4be00b2370 2021-02-10T11:22:09.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.695 7f4be54ac700 1 -- 172.21.2.112:0/1150219870 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4be00b3cf0 con 0x7f4be00b2370 2021-02-10T11:22:09.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.695 7f4bd7fff700 1 -- 172.21.2.112:0/1150219870 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4bd80011a0 con 0x7f4be00b2370 2021-02-10T11:22:09.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.696 7f4bd7fff700 1 --2- 172.21.2.112:0/1150219870 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4bc40186c0 0x7f4bc401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:09.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.696 7f4bd7fff700 1 -- 172.21.2.112:0/1150219870 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f4bd8032900 con 0x7f4be00b2370 2021-02-10T11:22:09.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.696 7f4bdf2bf700 1 --2- 172.21.2.112:0/1150219870 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4bc40186c0 0x7f4bc401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:09.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.696 7f4bdf2bf700 1 --2- 172.21.2.112:0/1150219870 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4bc40186c0 0x7f4bc401abc0 crc :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:09.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.727 7f4be54ac700 1 -- 172.21.2.112:0/1150219870 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4be00edd10 con 0x7f4be00b2370 2021-02-10T11:22:09.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.729 7f4bd7fff700 1 -- 172.21.2.112:0/1150219870 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4bd80093d0 con 0x7f4be00b2370 2021-02-10T11:22:09.852 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.859 7f4be54ac700 1 -- 172.21.2.112:0/1150219870 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348612", "target": ["mgr", ""]}) v1 -- 0x7f4be00b3cf0 con 0x7f4bc40186c0 2021-02-10T11:22:09.855 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.862 7f4bd7fff700 1 -- 172.21.2.112:0/1150219870 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f4bc8006b60 con 0x7f4bc40186c0 2021-02-10T11:22:09.855 INFO:teuthology.orchestra.run.gibba012.stdout:/volumes/_nogroup/subvolume_0000000000348612/39a4a267-c8b9-4f25-87e9-2efbebba643d 2021-02-10T11:22:09.857 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.864 7f4be54ac700 1 -- 172.21.2.112:0/1150219870 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4bc40186c0 msgr2=0x7f4bc401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:09.857 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.864 7f4be54ac700 1 --2- 172.21.2.112:0/1150219870 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4bc40186c0 0x7f4bc401abc0 crc :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:09.857 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.864 7f4be54ac700 1 -- 172.21.2.112:0/1150219870 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4be00b2370 msgr2=0x7f4be00b2d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:09.857 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.864 7f4be54ac700 1 --2- 172.21.2.112:0/1150219870 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4be00b2370 0x7f4be00b2d40 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 rx=0x7f4bd80091d0 tx=0x7f4bd800bd60).stop 2021-02-10T11:22:09.857 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.864 7f4be54ac700 1 -- 172.21.2.112:0/1150219870 shutdown_connections 2021-02-10T11:22:09.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.864 7f4be54ac700 1 --2- 172.21.2.112:0/1150219870 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4bc40186c0 0x7f4bc401abc0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:09.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.864 7f4be54ac700 1 --2- 172.21.2.112:0/1150219870 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4be00c3190 0x7f4be00b1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:09.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.864 7f4be54ac700 1 --2- 172.21.2.112:0/1150219870 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4be00b2370 0x7f4be00b2d40 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:09.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.864 7f4be54ac700 1 --2- 172.21.2.112:0/1150219870 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4be00c36a0 0x7f4be00c2b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:09.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.864 7f4be54ac700 1 -- 172.21.2.112:0/1150219870 >> 172.21.2.112:0/1150219870 conn(0x7f4be009f630 msgr2=0x7f4be00a0130 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:09.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.864 7f4be54ac700 1 -- 172.21.2.112:0/1150219870 shutdown_connections 2021-02-10T11:22:09.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:09.864 7f4be54ac700 1 -- 172.21.2.112:0/1150219870 wait complete. 2021-02-10T11:22:09.865 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000348612 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000348612/39a4a267-c8b9-4f25-87e9-2efbebba643d 2021-02-10T11:22:09.865 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:22:09.865 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:22:09.865 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:22:10.202 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:22:10.202 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:22:10.203 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348612/39a4a267-c8b9-4f25-87e9-2efbebba643d/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:22:10.581 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:22:10.581 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:22:10.581 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0246175 s, 42.6 MB/s 2021-02-10T11:22:10.582 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:22:10.583 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:22:10.583 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:22:10.919 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:22:10.919 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:22:10.919 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348612/39a4a267-c8b9-4f25-87e9-2efbebba643d/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:22:11.294 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:22:11.295 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:22:11.295 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0248827 s, 42.1 MB/s 2021-02-10T11:22:11.295 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:22:11.296 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:22:11.296 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:22:11.631 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:22:11.632 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:22:11.632 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348612/39a4a267-c8b9-4f25-87e9-2efbebba643d/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:22:12.008 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:22:12.008 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:22:12.008 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0246725 s, 42.5 MB/s 2021-02-10T11:22:12.009 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:22:12.009 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:22:12.009 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:22:12.345 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:22:12.345 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:22:12.345 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348612/39a4a267-c8b9-4f25-87e9-2efbebba643d/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:22:12.720 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:22:12.720 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:22:12.720 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0240422 s, 43.6 MB/s 2021-02-10T11:22:12.721 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:22:12.721 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:22:12.721 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:22:13.056 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:22:13.056 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:22:13.056 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348612/39a4a267-c8b9-4f25-87e9-2efbebba643d/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:22:13.431 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:22:13.431 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:22:13.431 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0237264 s, 44.2 MB/s 2021-02-10T11:22:13.432 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:22:13.432 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:22:13.432 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:22:13.767 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:22:13.767 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:22:13.767 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348612/39a4a267-c8b9-4f25-87e9-2efbebba643d/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:22:14.144 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:22:14.144 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:22:14.145 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.025237 s, 41.5 MB/s 2021-02-10T11:22:14.145 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:22:14.146 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:22:14.146 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:22:14.480 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:22:14.481 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:22:14.481 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348612/39a4a267-c8b9-4f25-87e9-2efbebba643d/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:22:14.853 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:22:14.853 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:22:14.853 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0236961 s, 44.3 MB/s 2021-02-10T11:22:14.854 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:22:14.854 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:22:14.854 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:22:15.189 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:22:15.190 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:22:15.190 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348612/39a4a267-c8b9-4f25-87e9-2efbebba643d/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:22:15.566 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:22:15.566 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:22:15.566 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0244184 s, 42.9 MB/s 2021-02-10T11:22:15.567 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:22:15.567 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:22:15.567 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:22:15.902 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:22:15.902 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:22:15.902 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348612/39a4a267-c8b9-4f25-87e9-2efbebba643d/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:22:16.279 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:22:16.279 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:22:16.279 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0245464 s, 42.7 MB/s 2021-02-10T11:22:16.280 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:22:16.280 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:22:16.280 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:22:16.616 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:22:16.616 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:22:16.616 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348612/39a4a267-c8b9-4f25-87e9-2efbebba643d/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:22:16.991 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:22:16.991 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:22:16.991 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0231563 s, 45.3 MB/s 2021-02-10T11:22:16.992 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000348613 2021-02-10T11:22:17.423 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:17.414 7fa5bc34b700 1 Processor -- start 2021-02-10T11:22:17.423 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:17.414 7fa5bc34b700 1 -- start start 2021-02-10T11:22:17.424 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:17.414 7fa5bc34b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa5b41b0870 0x7fa5b40f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:17.424 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:17.414 7fa5bc34b700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fa5b40efe00 0x7fa5b40f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:17.424 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:17.414 7fa5bc34b700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5b41b0c70 con 0x7fa5b41b0470 2021-02-10T11:22:17.424 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:17.414 7fa5bc34b700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5b40f0b50 con 0x7fa5b40efe00 2021-02-10T11:22:17.424 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:17.414 7fa5bc34b700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5b40f0820 con 0x7fa5b41b0870 2021-02-10T11:22:17.424 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:17.414 7fa5ba167700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fa5b41b0470 0x7fa5b40ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:44522/0 (socket says 172.21.2.112:44522) 2021-02-10T11:22:17.424 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:17.414 7fa5ba167700 1 -- 172.21.2.112:0/3836603836 learned_addr learned my addr 172.21.2.112:0/3836603836 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:22:17.424 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:17.415 7fa5ba968700 1 --2- 172.21.2.112:0/3836603836 >> v2:172.21.2.120:3300/0 conn(0x7fa5b40efe00 0x7fa5b40f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:17.424 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:17.415 7fa5b9966700 1 --2- 172.21.2.112:0/3836603836 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa5b41b0870 0x7fa5b40f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:17.424 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:17.415 7fa5b9165700 1 -- 172.21.2.112:0/3836603836 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1160269816 0 0) 0x7fa5b00010b0 con 0x7fa5b41b0470 2021-02-10T11:22:17.424 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:17.415 7fa5b9165700 1 -- 172.21.2.112:0/3836603836 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5a0002fe0 con 0x7fa5b41b0470 2021-02-10T11:22:17.424 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:17.415 7fa5ba968700 1 -- 172.21.2.112:0/3836603836 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa5b41b0870 msgr2=0x7fa5b40f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:17.424 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:17.415 7fa5ba968700 1 --2- 172.21.2.112:0/3836603836 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa5b41b0870 0x7fa5b40f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:17.425 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:17.415 7fa5ba968700 1 -- 172.21.2.112:0/3836603836 >> v1:172.21.2.112:6789/0 conn(0x7fa5b41b0470 legacy=0x7fa5b40ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:17.425 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:17.415 7fa5ba968700 1 -- 172.21.2.112:0/3836603836 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5b40e52e0 con 0x7fa5b40efe00 2021-02-10T11:22:17.425 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:17.415 7fa5ba968700 1 --2- 172.21.2.112:0/3836603836 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa5b40efe00 0x7fa5b40f0200 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7fa5ac006e50 tx=0x7fa5ac009e70).ready entity=mon.1 client_cookie=6ab9067487cb30e4 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:17.425 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:17.415 7fa5b9165700 1 -- 172.21.2.112:0/3836603836 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa5ac00b510 con 0x7fa5b40efe00 2021-02-10T11:22:17.425 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:17.415 7fa5b9165700 1 -- 172.21.2.112:0/3836603836 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5ac006790 con 0x7fa5b40efe00 2021-02-10T11:22:17.425 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:17.415 7fa5b9165700 1 -- 172.21.2.112:0/3836603836 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa5ac00b940 con 0x7fa5b40efe00 2021-02-10T11:22:17.425 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:17.415 7fa5bc34b700 1 -- 172.21.2.112:0/3836603836 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa5b40efe00 msgr2=0x7fa5b40f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:17.425 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:17.415 7fa5bc34b700 1 --2- 172.21.2.112:0/3836603836 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa5b40efe00 0x7fa5b40f0200 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7fa5ac006e50 tx=0x7fa5ac009e70).stop 2021-02-10T11:22:17.425 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:17.416 7fa5bc34b700 1 -- 172.21.2.112:0/3836603836 shutdown_connections 2021-02-10T11:22:17.425 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:17.416 7fa5bc34b700 1 --2- 172.21.2.112:0/3836603836 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa5b40efe00 0x7fa5b40f0200 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:17.425 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:17.416 7fa5bc34b700 1 --2- 172.21.2.112:0/3836603836 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa5b41b0870 0x7fa5b40f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:17.425 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:17.416 7fa5bc34b700 1 -- 172.21.2.112:0/3836603836 >> 172.21.2.112:0/3836603836 conn(0x7fa5b40ce3e0 msgr2=0x7fa5b40e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:17.425 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:17.416 7fa5bc34b700 1 -- 172.21.2.112:0/3836603836 shutdown_connections 2021-02-10T11:22:17.426 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:17.416 7fa5bc34b700 1 -- 172.21.2.112:0/3836603836 wait complete. 2021-02-10T11:22:17.427 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:17.431 7fa5bc34b700 1 Processor -- start 2021-02-10T11:22:17.427 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:17.431 7fa5bc34b700 1 -- start start 2021-02-10T11:22:17.427 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:17.431 7fa5bc34b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa5b409f140 0x7fa5b40c1940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:17.427 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:17.431 7fa5bc34b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa5b40c1f60 0x7fa5b409e300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:17.427 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:17.431 7fa5bc34b700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa5b409e920 0x7fa5b40c2960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:17.428 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:17.431 7fa5bc34b700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5b40c23e0 con 0x7fa5b409f140 2021-02-10T11:22:17.428 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:17.431 7fa5bc34b700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5b409ed20 con 0x7fa5b409e920 2021-02-10T11:22:17.428 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:17.431 7fa5bc34b700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5b4105d10 con 0x7fa5b40c1f60 2021-02-10T11:22:17.428 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:17.431 7fa5ba968700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa5b409e920 0x7fa5b40c2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:17.428 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:17.431 7fa5ba167700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa5b409f140 0x7fa5b40c1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:17.428 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:17.431 7fa5ba167700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa5b409f140 0x7fa5b40c1940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:45066/0 (socket says 172.21.2.112:45066) 2021-02-10T11:22:17.428 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:17.431 7fa5ba167700 1 -- 172.21.2.112:0/1220235757 learned_addr learned my addr 172.21.2.112:0/1220235757 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:22:17.428 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:17.432 7fa5b9966700 1 --2- 172.21.2.112:0/1220235757 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa5b40c1f60 0x7fa5b409e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:17.428 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:17.432 7fa5ba167700 1 -- 172.21.2.112:0/1220235757 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa5b40c1f60 msgr2=0x7fa5b409e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:17.428 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:17.432 7fa5ba167700 1 --2- 172.21.2.112:0/1220235757 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa5b40c1f60 0x7fa5b409e300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:17.428 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:17.432 7fa5ba167700 1 -- 172.21.2.112:0/1220235757 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa5b409e920 msgr2=0x7fa5b40c2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:17.428 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:17.432 7fa5ba167700 1 --2- 172.21.2.112:0/1220235757 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa5b409e920 0x7fa5b40c2960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:17.428 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:17.432 7fa5ba167700 1 -- 172.21.2.112:0/1220235757 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5b41a4910 con 0x7fa5b409f140 2021-02-10T11:22:17.429 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:17.432 7fa5b9966700 1 --2- 172.21.2.112:0/1220235757 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa5b40c1f60 0x7fa5b409e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:22:17.429 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:17.432 7fa5ba167700 1 --2- 172.21.2.112:0/1220235757 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa5b409f140 0x7fa5b40c1940 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 rx=0x7fa5b0001c60 tx=0x7fa5b000a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:17.429 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:17.432 7fa5a6ffd700 1 -- 172.21.2.112:0/1220235757 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa5b000acc0 con 0x7fa5b409f140 2021-02-10T11:22:17.430 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:17.432 7fa5a6ffd700 1 -- 172.21.2.112:0/1220235757 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5b0004350 con 0x7fa5b409f140 2021-02-10T11:22:17.430 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:17.432 7fa5bc34b700 1 -- 172.21.2.112:0/1220235757 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5b41a4910 con 0x7fa5b409f140 2021-02-10T11:22:17.430 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:17.432 7fa5bc34b700 1 -- 172.21.2.112:0/1220235757 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5b41a4910 con 0x7fa5b409f140 2021-02-10T11:22:17.430 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:17.432 7fa5a6ffd700 1 -- 172.21.2.112:0/1220235757 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa5b00047b0 con 0x7fa5b409f140 2021-02-10T11:22:17.430 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:17.433 7fa5a6ffd700 1 -- 172.21.2.112:0/1220235757 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa5b0004530 con 0x7fa5b409f140 2021-02-10T11:22:17.430 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:17.433 7fa5a6ffd700 1 --2- 172.21.2.112:0/1220235757 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa59c018730 0x7fa59c01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:17.430 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:17.433 7fa5a6ffd700 1 -- 172.21.2.112:0/1220235757 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fa5b00327a0 con 0x7fa5b409f140 2021-02-10T11:22:17.430 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:17.433 7fa5b9966700 1 --2- 172.21.2.112:0/1220235757 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa59c018730 0x7fa59c01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:17.430 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:17.433 7fa5b9966700 1 --2- 172.21.2.112:0/1220235757 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa59c018730 0x7fa59c01ac10 crc :-1 s=READY pgs=265 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:17.457 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:17.464 7fa5bc34b700 1 -- 172.21.2.112:0/1220235757 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5b40f0b50 con 0x7fa5b409f140 2021-02-10T11:22:17.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:17.466 7fa5a6ffd700 1 -- 172.21.2.112:0/1220235757 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa5b0004140 con 0x7fa5b409f140 2021-02-10T11:22:17.586 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:17.593 7fa5bc34b700 1 -- 172.21.2.112:0/1220235757 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348613", "target": ["mgr", ""]}) v1 -- 0x7fa5b40f0b50 con 0x7fa59c018730 2021-02-10T11:22:17.613 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:17.620 7fa5a6ffd700 1 -- 172.21.2.112:0/1220235757 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fa5a80009b0 con 0x7fa59c018730 2021-02-10T11:22:17.619 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:17.626 7fa5bc34b700 1 -- 172.21.2.112:0/1220235757 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa59c018730 msgr2=0x7fa59c01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:17.620 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:17.626 7fa5bc34b700 1 --2- 172.21.2.112:0/1220235757 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa59c018730 0x7fa59c01ac10 crc :-1 s=READY pgs=265 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:17.620 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:17.626 7fa5bc34b700 1 -- 172.21.2.112:0/1220235757 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa5b409f140 msgr2=0x7fa5b40c1940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:17.620 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:17.626 7fa5bc34b700 1 --2- 172.21.2.112:0/1220235757 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa5b409f140 0x7fa5b40c1940 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 rx=0x7fa5b0001c60 tx=0x7fa5b000a8d0).stop 2021-02-10T11:22:17.620 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:17.626 7fa5bc34b700 1 -- 172.21.2.112:0/1220235757 shutdown_connections 2021-02-10T11:22:17.620 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:17.626 7fa5bc34b700 1 --2- 172.21.2.112:0/1220235757 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa59c018730 0x7fa59c01ac10 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:17.620 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:17.626 7fa5bc34b700 1 --2- 172.21.2.112:0/1220235757 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa5b409e920 0x7fa5b40c2960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:17.620 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:17.626 7fa5bc34b700 1 --2- 172.21.2.112:0/1220235757 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa5b40c1f60 0x7fa5b409e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:17.620 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:17.626 7fa5bc34b700 1 --2- 172.21.2.112:0/1220235757 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa5b409f140 0x7fa5b40c1940 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:17.620 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:17.626 7fa5bc34b700 1 -- 172.21.2.112:0/1220235757 >> 172.21.2.112:0/1220235757 conn(0x7fa5b409fc10 msgr2=0x7fa5b40b3250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:17.620 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:17.626 7fa5bc34b700 1 -- 172.21.2.112:0/1220235757 shutdown_connections 2021-02-10T11:22:17.620 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:17.626 7fa5bc34b700 1 -- 172.21.2.112:0/1220235757 wait complete. 2021-02-10T11:22:17.628 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000348613 2021-02-10T11:22:18.059 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:18.049 7feabae9a700 1 Processor -- start 2021-02-10T11:22:18.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:18.049 7feabae9a700 1 -- start start 2021-02-10T11:22:18.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:18.050 7feabae9a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7feab41b0470 0x7feab41b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:18.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:18.050 7feabae9a700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feab40f0b50 con 0x7feab40ef610 2021-02-10T11:22:18.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:18.050 7feabae9a700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feab40f24e0 con 0x7feab40f20e0 2021-02-10T11:22:18.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:18.050 7feabae9a700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feab40e5ea0 con 0x7feab41b0470 2021-02-10T11:22:18.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:18.050 7feab3fff700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7feab40ef610 0x7feab40efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:44536/0 (socket says 172.21.2.112:44536) 2021-02-10T11:22:18.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:18.050 7feab3fff700 1 -- 172.21.2.112:0/2649337673 learned_addr learned my addr 172.21.2.112:0/2649337673 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:22:18.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:18.050 7feab8cb6700 1 --2- 172.21.2.112:0/2649337673 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7feab41b0470 0x7feab41b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:18.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:18.050 7feab37fe700 1 -- 172.21.2.112:0/2649337673 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1386493496 0 0) 0x7feaa4001430 con 0x7feab40ef610 2021-02-10T11:22:18.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:18.051 7feab37fe700 1 -- 172.21.2.112:0/2649337673 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feaa0002fe0 con 0x7feab40ef610 2021-02-10T11:22:18.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:18.051 7feab37fe700 1 -- 172.21.2.112:0/2649337673 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1695018545 0 0) 0x7feaac001080 con 0x7feab40f20e0 2021-02-10T11:22:18.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:18.051 7feab37fe700 1 -- 172.21.2.112:0/2649337673 --> v1:172.21.2.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feaa0003210 con 0x7feab40f20e0 2021-02-10T11:22:18.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:18.051 7feab8cb6700 1 -- 172.21.2.112:0/2649337673 >> v1:172.21.2.120:6789/0 conn(0x7feab40f20e0 legacy=0x7feab40f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:18.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:18.051 7feab8cb6700 1 -- 172.21.2.112:0/2649337673 >> v1:172.21.2.112:6789/0 conn(0x7feab40ef610 legacy=0x7feab40efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:18.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:18.051 7feab37fe700 1 -- 172.21.2.112:0/2649337673 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 318300861 0 0) 0x7feaa4001390 con 0x7feab40ef610 2021-02-10T11:22:18.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:18.051 7feab8cb6700 1 -- 172.21.2.112:0/2649337673 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feab40e52e0 con 0x7feab41b0470 2021-02-10T11:22:18.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:18.051 7feab8cb6700 1 --2- 172.21.2.112:0/2649337673 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7feab41b0470 0x7feab41b0870 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7fea9c006650 tx=0x7fea9c009a80).ready entity=mon.2 client_cookie=ff3d6ae83579b8a8 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:18.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:18.051 7feab37fe700 1 -- 172.21.2.112:0/2649337673 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fea9c0009b0 con 0x7feab41b0470 2021-02-10T11:22:18.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:18.051 7feab37fe700 1 -- 172.21.2.112:0/2649337673 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea9c00c0c0 con 0x7feab41b0470 2021-02-10T11:22:18.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:18.051 7feab37fe700 1 -- 172.21.2.112:0/2649337673 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fea9c00c520 con 0x7feab41b0470 2021-02-10T11:22:18.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:18.052 7feabae9a700 1 -- 172.21.2.112:0/2649337673 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7feab41b0470 msgr2=0x7feab41b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:18.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:18.052 7feabae9a700 1 --2- 172.21.2.112:0/2649337673 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7feab41b0470 0x7feab41b0870 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7fea9c006650 tx=0x7fea9c009a80).stop 2021-02-10T11:22:18.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:18.052 7feabae9a700 1 -- 172.21.2.112:0/2649337673 shutdown_connections 2021-02-10T11:22:18.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:18.052 7feabae9a700 1 --2- 172.21.2.112:0/2649337673 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7feab41b0470 0x7feab41b0870 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:18.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:18.052 7feabae9a700 1 -- 172.21.2.112:0/2649337673 >> 172.21.2.112:0/2649337673 conn(0x7feab40ce3e0 msgr2=0x7feab40e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:18.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:18.052 7feabae9a700 1 -- 172.21.2.112:0/2649337673 shutdown_connections 2021-02-10T11:22:18.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:18.052 7feabae9a700 1 -- 172.21.2.112:0/2649337673 wait complete. 2021-02-10T11:22:18.063 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:18.067 7feabae9a700 1 Processor -- start 2021-02-10T11:22:18.063 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:18.067 7feabae9a700 1 -- start start 2021-02-10T11:22:18.063 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:18.067 7feabae9a700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7feab4190920 0x7feab40b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:18.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:18.067 7feabae9a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7feab40b8200 0x7feab40b8600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:18.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:18.067 7feabae9a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7feab40c1d80 0x7feab40c2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:18.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:18.067 7feabae9a700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feab4190d20 con 0x7feab40c1d80 2021-02-10T11:22:18.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:18.067 7feabae9a700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feab40b8c20 con 0x7feab4190920 2021-02-10T11:22:18.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:18.067 7feabae9a700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feab40c27a0 con 0x7feab40b8200 2021-02-10T11:22:18.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:18.067 7feab94b7700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7feab40c1d80 0x7feab40c2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:18.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:18.067 7feab94b7700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7feab40c1d80 0x7feab40c2180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:45078/0 (socket says 172.21.2.112:45078) 2021-02-10T11:22:18.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:18.067 7feab94b7700 1 -- 172.21.2.112:0/3219923977 learned_addr learned my addr 172.21.2.112:0/3219923977 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:22:18.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:18.068 7feab8cb6700 1 --2- 172.21.2.112:0/3219923977 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7feab4190920 0x7feab40b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:18.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:18.068 7feab3fff700 1 --2- 172.21.2.112:0/3219923977 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7feab40b8200 0x7feab40b8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:18.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:18.068 7feab94b7700 1 -- 172.21.2.112:0/3219923977 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7feab40b8200 msgr2=0x7feab40b8600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:18.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:18.068 7feab94b7700 1 --2- 172.21.2.112:0/3219923977 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7feab40b8200 0x7feab40b8600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:18.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:18.068 7feab94b7700 1 -- 172.21.2.112:0/3219923977 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7feab4190920 msgr2=0x7feab40b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:18.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:18.068 7feab94b7700 1 --2- 172.21.2.112:0/3219923977 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7feab4190920 0x7feab40b7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:18.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:18.068 7feab94b7700 1 -- 172.21.2.112:0/3219923977 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feab40c29c0 con 0x7feab40c1d80 2021-02-10T11:22:18.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:18.068 7feab8cb6700 1 --2- 172.21.2.112:0/3219923977 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7feab4190920 0x7feab40b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:22:18.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:18.068 7feab94b7700 1 --2- 172.21.2.112:0/3219923977 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7feab40c1d80 0x7feab40c2180 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 rx=0x7feaac006f00 tx=0x7feaac00b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:18.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:18.069 7feab17fa700 1 -- 172.21.2.112:0/3219923977 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feaac00bd30 con 0x7feab40c1d80 2021-02-10T11:22:18.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:18.069 7feab17fa700 1 -- 172.21.2.112:0/3219923977 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feaac000e20 con 0x7feab40c1d80 2021-02-10T11:22:18.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:18.069 7feab17fa700 1 -- 172.21.2.112:0/3219923977 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feaac0054f0 con 0x7feab40c1d80 2021-02-10T11:22:18.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:18.069 7feabae9a700 1 -- 172.21.2.112:0/3219923977 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feab40c29c0 con 0x7feab40c1d80 2021-02-10T11:22:18.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:18.069 7feabae9a700 1 -- 172.21.2.112:0/3219923977 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feab40c2f50 con 0x7feab40c1d80 2021-02-10T11:22:18.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:18.069 7feab17fa700 1 -- 172.21.2.112:0/3219923977 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7feaac006240 con 0x7feab40c1d80 2021-02-10T11:22:18.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:18.069 7feab17fa700 1 --2- 172.21.2.112:0/3219923977 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fea98018730 0x7fea9801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:18.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:18.069 7feab17fa700 1 -- 172.21.2.112:0/3219923977 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7feaac032730 con 0x7feab40c1d80 2021-02-10T11:22:18.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:18.070 7feab8cb6700 1 --2- 172.21.2.112:0/3219923977 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fea98018730 0x7fea9801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:18.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:18.070 7feab8cb6700 1 --2- 172.21.2.112:0/3219923977 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fea98018730 0x7fea9801ac10 crc :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:18.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:18.099 7feabae9a700 1 -- 172.21.2.112:0/3219923977 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feab4105ae0 con 0x7feab40c1d80 2021-02-10T11:22:18.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:18.102 7feab17fa700 1 -- 172.21.2.112:0/3219923977 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7feaac006030 con 0x7feab40c1d80 2021-02-10T11:22:18.224 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:18.231 7feabae9a700 1 -- 172.21.2.112:0/3219923977 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348613", "target": ["mgr", ""]}) v1 -- 0x7feab40f0b50 con 0x7fea98018730 2021-02-10T11:22:18.227 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:18.235 7feab17fa700 1 -- 172.21.2.112:0/3219923977 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7fea9c0090e0 con 0x7fea98018730 2021-02-10T11:22:18.232 INFO:teuthology.orchestra.run.gibba012.stdout:/volumes/_nogroup/subvolume_0000000000348613/ea952de4-3df3-4e55-adba-42d82f544242 2021-02-10T11:22:18.233 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:18.241 7feabae9a700 1 -- 172.21.2.112:0/3219923977 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fea98018730 msgr2=0x7fea9801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:18.234 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:18.241 7feabae9a700 1 --2- 172.21.2.112:0/3219923977 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fea98018730 0x7fea9801ac10 crc :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:18.234 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:18.241 7feabae9a700 1 -- 172.21.2.112:0/3219923977 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7feab40c1d80 msgr2=0x7feab40c2180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:18.234 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:18.241 7feabae9a700 1 --2- 172.21.2.112:0/3219923977 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7feab40c1d80 0x7feab40c2180 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 rx=0x7feaac006f00 tx=0x7feaac00b940).stop 2021-02-10T11:22:18.234 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:18.241 7feabae9a700 1 -- 172.21.2.112:0/3219923977 shutdown_connections 2021-02-10T11:22:18.234 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:18.241 7feabae9a700 1 --2- 172.21.2.112:0/3219923977 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fea98018730 0x7fea9801ac10 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:18.234 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:18.241 7feabae9a700 1 --2- 172.21.2.112:0/3219923977 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7feab40c1d80 0x7feab40c2180 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:18.234 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:18.241 7feabae9a700 1 --2- 172.21.2.112:0/3219923977 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7feab40b8200 0x7feab40b8600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:18.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:18.241 7feabae9a700 1 --2- 172.21.2.112:0/3219923977 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7feab4190920 0x7feab40b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:18.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:18.241 7feabae9a700 1 -- 172.21.2.112:0/3219923977 >> 172.21.2.112:0/3219923977 conn(0x7feab4094e10 msgr2=0x7feab40a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:18.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:18.241 7feabae9a700 1 -- 172.21.2.112:0/3219923977 shutdown_connections 2021-02-10T11:22:18.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:18.241 7feabae9a700 1 -- 172.21.2.112:0/3219923977 wait complete. 2021-02-10T11:22:18.242 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000348613 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000348613/ea952de4-3df3-4e55-adba-42d82f544242 2021-02-10T11:22:18.242 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:22:18.242 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:22:18.242 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:22:18.578 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:22:18.579 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:22:18.579 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348613/ea952de4-3df3-4e55-adba-42d82f544242/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:22:18.961 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:22:18.962 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:22:18.962 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0249746 s, 42.0 MB/s 2021-02-10T11:22:18.963 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:22:18.963 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:22:18.963 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:22:19.299 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:22:19.299 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:22:19.299 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348613/ea952de4-3df3-4e55-adba-42d82f544242/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:22:19.671 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:22:19.671 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:22:19.671 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0237105 s, 44.2 MB/s 2021-02-10T11:22:19.672 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:22:19.672 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:22:19.672 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:22:20.008 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:22:20.008 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:22:20.008 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348613/ea952de4-3df3-4e55-adba-42d82f544242/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:22:20.384 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:22:20.406 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:22:20.406 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0245389 s, 42.7 MB/s 2021-02-10T11:22:20.407 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:22:20.407 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:22:20.407 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:22:20.743 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:22:20.743 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:22:20.743 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348613/ea952de4-3df3-4e55-adba-42d82f544242/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:22:21.118 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:22:21.118 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:22:21.118 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0236165 s, 44.4 MB/s 2021-02-10T11:22:21.119 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:22:21.119 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:22:21.119 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:22:21.454 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:22:21.454 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:22:21.455 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348613/ea952de4-3df3-4e55-adba-42d82f544242/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:22:21.831 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:22:21.831 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:22:21.831 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0247893 s, 42.3 MB/s 2021-02-10T11:22:21.832 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:22:21.832 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:22:21.832 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:22:22.168 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:22:22.168 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:22:22.168 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348613/ea952de4-3df3-4e55-adba-42d82f544242/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:22:22.545 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:22:22.545 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:22:22.545 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0261599 s, 40.1 MB/s 2021-02-10T11:22:22.547 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:22:22.547 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:22:22.547 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:22:22.882 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:22:22.882 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:22:22.882 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348613/ea952de4-3df3-4e55-adba-42d82f544242/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:22:23.256 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:22:23.256 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:22:23.256 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0222831 s, 47.1 MB/s 2021-02-10T11:22:23.257 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:22:23.257 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:22:23.257 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:22:23.593 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:22:23.593 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:22:23.593 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348613/ea952de4-3df3-4e55-adba-42d82f544242/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:22:23.968 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:22:23.968 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:22:23.968 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0234381 s, 44.7 MB/s 2021-02-10T11:22:23.969 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:22:23.969 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:22:23.970 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:22:24.306 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:22:24.306 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:22:24.307 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348613/ea952de4-3df3-4e55-adba-42d82f544242/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:22:24.693 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:22:24.693 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:22:24.693 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0239582 s, 43.8 MB/s 2021-02-10T11:22:24.694 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:22:24.694 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:22:24.694 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:22:25.032 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:22:25.032 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:22:25.032 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348613/ea952de4-3df3-4e55-adba-42d82f544242/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:22:25.410 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:22:25.411 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:22:25.411 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.024347 s, 43.1 MB/s 2021-02-10T11:22:25.412 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000348614 2021-02-10T11:22:25.848 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:25.838 7fa4bc19d700 1 Processor -- start 2021-02-10T11:22:25.849 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:25.839 7fa4bc19d700 1 -- start start 2021-02-10T11:22:25.849 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:25.839 7fa4bc19d700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fa4b41b0470 0x7fa4b41b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:25.851 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:25.839 7fa4bc19d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa4b40f20e0 0x7fa4b40f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:25.851 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:25.839 7fa4bc19d700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4b40f0b50 con 0x7fa4b40ef610 2021-02-10T11:22:25.851 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:25.839 7fa4bc19d700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4b40f0820 con 0x7fa4b41b0470 2021-02-10T11:22:25.851 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:25.839 7fa4bc19d700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4b40f24e0 con 0x7fa4b40f20e0 2021-02-10T11:22:25.852 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:25.839 7fa4b97b8700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fa4b40ef610 0x7fa4b40efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:44548/0 (socket says 172.21.2.112:44548) 2021-02-10T11:22:25.852 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:25.839 7fa4b97b8700 1 -- 172.21.2.112:0/2175269557 learned_addr learned my addr 172.21.2.112:0/2175269557 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:22:25.852 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:25.840 7fa4b9fb9700 1 --2- 172.21.2.112:0/2175269557 >> v2:172.21.2.120:3300/0 conn(0x7fa4b41b0470 0x7fa4b41b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:25.852 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:25.840 7fa4ba7ba700 1 --2- 172.21.2.112:0/2175269557 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa4b40f20e0 0x7fa4b40f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:25.852 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:25.840 7fa4b8fb7700 1 -- 172.21.2.112:0/2175269557 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4104557531 0 0) 0x7fa4a80010b0 con 0x7fa4b40ef610 2021-02-10T11:22:25.852 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:25.840 7fa4b8fb7700 1 -- 172.21.2.112:0/2175269557 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4a0002fe0 con 0x7fa4b40ef610 2021-02-10T11:22:25.853 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:25.840 7fa4ba7ba700 1 -- 172.21.2.112:0/2175269557 >> v2:172.21.2.120:3300/0 conn(0x7fa4b41b0470 msgr2=0x7fa4b41b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:25.853 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:25.840 7fa4ba7ba700 1 --2- 172.21.2.112:0/2175269557 >> v2:172.21.2.120:3300/0 conn(0x7fa4b41b0470 0x7fa4b41b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:25.853 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:25.840 7fa4ba7ba700 1 -- 172.21.2.112:0/2175269557 >> v1:172.21.2.112:6789/0 conn(0x7fa4b40ef610 legacy=0x7fa4b40efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:25.853 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:25.840 7fa4ba7ba700 1 -- 172.21.2.112:0/2175269557 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4b40e52e0 con 0x7fa4b40f20e0 2021-02-10T11:22:25.853 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:25.840 7fa4b9fb9700 1 --2- 172.21.2.112:0/2175269557 >> v2:172.21.2.120:3300/0 conn(0x7fa4b41b0470 0x7fa4b41b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:22:25.853 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:25.840 7fa4ba7ba700 1 --2- 172.21.2.112:0/2175269557 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa4b40f20e0 0x7fa4b40f0200 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7fa4ac006e50 tx=0x7fa4ac009e70).ready entity=mon.2 client_cookie=f46324bbe0789f29 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:25.854 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:25.840 7fa4b8fb7700 1 -- 172.21.2.112:0/2175269557 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa4ac006a20 con 0x7fa4b40f20e0 2021-02-10T11:22:25.854 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:25.840 7fa4b8fb7700 1 -- 172.21.2.112:0/2175269557 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4ac006790 con 0x7fa4b40f20e0 2021-02-10T11:22:25.854 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:25.841 7fa4bc19d700 1 -- 172.21.2.112:0/2175269557 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa4b40f20e0 msgr2=0x7fa4b40f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:25.854 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:25.841 7fa4bc19d700 1 --2- 172.21.2.112:0/2175269557 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa4b40f20e0 0x7fa4b40f0200 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7fa4ac006e50 tx=0x7fa4ac009e70).stop 2021-02-10T11:22:25.854 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:25.840 7fa4b8fb7700 1 -- 172.21.2.112:0/2175269557 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa4ac00b700 con 0x7fa4b40f20e0 2021-02-10T11:22:25.854 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:25.841 7fa4bc19d700 1 -- 172.21.2.112:0/2175269557 shutdown_connections 2021-02-10T11:22:25.854 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:25.841 7fa4bc19d700 1 --2- 172.21.2.112:0/2175269557 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa4b40f20e0 0x7fa4b40f0200 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:25.854 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:25.841 7fa4bc19d700 1 --2- 172.21.2.112:0/2175269557 >> v2:172.21.2.120:3300/0 conn(0x7fa4b41b0470 0x7fa4b41b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:25.855 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:25.841 7fa4bc19d700 1 -- 172.21.2.112:0/2175269557 >> 172.21.2.112:0/2175269557 conn(0x7fa4b40ce3e0 msgr2=0x7fa4b40e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:25.855 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:25.841 7fa4bc19d700 1 -- 172.21.2.112:0/2175269557 shutdown_connections 2021-02-10T11:22:25.855 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:25.841 7fa4bc19d700 1 -- 172.21.2.112:0/2175269557 wait complete. 2021-02-10T11:22:25.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:25.855 7fa4bc19d700 1 Processor -- start 2021-02-10T11:22:25.860 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:25.856 7fa4bc19d700 1 -- start start 2021-02-10T11:22:25.860 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:25.856 7fa4bc19d700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa4b4190920 0x7fa4b40b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:25.860 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:25.856 7fa4bc19d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa4b40b8200 0x7fa4b40b8600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:25.861 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:25.856 7fa4bc19d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa4b40c1d80 0x7fa4b40c2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:25.861 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:25.856 7fa4bc19d700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4b4190d20 con 0x7fa4b40c1d80 2021-02-10T11:22:25.861 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:25.856 7fa4bc19d700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4b40b8c20 con 0x7fa4b4190920 2021-02-10T11:22:25.861 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:25.856 7fa4bc19d700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4b40c27a0 con 0x7fa4b40b8200 2021-02-10T11:22:25.862 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:25.856 7fa4ba7ba700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa4b40c1d80 0x7fa4b40c2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:25.862 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:25.856 7fa4ba7ba700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa4b40c1d80 0x7fa4b40c2180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:45096/0 (socket says 172.21.2.112:45096) 2021-02-10T11:22:25.862 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:25.856 7fa4ba7ba700 1 -- 172.21.2.112:0/2076233485 learned_addr learned my addr 172.21.2.112:0/2076233485 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:22:25.862 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:25.857 7fa4ba7ba700 1 -- 172.21.2.112:0/2076233485 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa4b40b8200 msgr2=0x7fa4b40b8600 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:22:25.862 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:25.857 7fa4ba7ba700 1 --2- 172.21.2.112:0/2076233485 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa4b40b8200 0x7fa4b40b8600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:25.862 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:25.857 7fa4ba7ba700 1 -- 172.21.2.112:0/2076233485 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa4b4190920 msgr2=0x7fa4b40b7be0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:22:25.862 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:25.857 7fa4ba7ba700 1 --2- 172.21.2.112:0/2076233485 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa4b4190920 0x7fa4b40b7be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:25.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:25.857 7fa4ba7ba700 1 -- 172.21.2.112:0/2076233485 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4b40c29c0 con 0x7fa4b40c1d80 2021-02-10T11:22:25.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:25.857 7fa4ba7ba700 1 --2- 172.21.2.112:0/2076233485 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa4b40c1d80 0x7fa4b40c2180 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 rx=0x7fa4ac0099d0 tx=0x7fa4ac00bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:25.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:25.857 7fa4a67fc700 1 -- 172.21.2.112:0/2076233485 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa4ac0082e0 con 0x7fa4b40c1d80 2021-02-10T11:22:25.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:25.857 7fa4a67fc700 1 -- 172.21.2.112:0/2076233485 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4ac008560 con 0x7fa4b40c1d80 2021-02-10T11:22:25.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:25.857 7fa4a67fc700 1 -- 172.21.2.112:0/2076233485 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa4ac008a80 con 0x7fa4b40c1d80 2021-02-10T11:22:25.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:25.857 7fa4bc19d700 1 -- 172.21.2.112:0/2076233485 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4b40c29c0 con 0x7fa4b40c1d80 2021-02-10T11:22:25.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:25.857 7fa4bc19d700 1 -- 172.21.2.112:0/2076233485 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4b40c29c0 con 0x7fa4b40c1d80 2021-02-10T11:22:25.865 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:25.858 7fa4a67fc700 1 -- 172.21.2.112:0/2076233485 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa4ac008740 con 0x7fa4b40c1d80 2021-02-10T11:22:25.865 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:25.858 7fa4a67fc700 1 --2- 172.21.2.112:0/2076233485 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa49c0186c0 0x7fa49c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:25.865 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:25.858 7fa4a67fc700 1 -- 172.21.2.112:0/2076233485 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fa4ac032970 con 0x7fa4b40c1d80 2021-02-10T11:22:25.866 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:25.858 7fa4b9fb9700 1 --2- 172.21.2.112:0/2076233485 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa49c0186c0 0x7fa49c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:25.866 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:25.859 7fa4b9fb9700 1 --2- 172.21.2.112:0/2076233485 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa49c0186c0 0x7fa49c01abc0 crc :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:25.882 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:25.889 7fa4bc19d700 1 -- 172.21.2.112:0/2076233485 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4b4105ae0 con 0x7fa4b40c1d80 2021-02-10T11:22:25.884 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:25.892 7fa4a67fc700 1 -- 172.21.2.112:0/2076233485 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa4ac0080b0 con 0x7fa4b40c1d80 2021-02-10T11:22:26.010 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:26.017 7fa4bc19d700 1 -- 172.21.2.112:0/2076233485 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348614", "target": ["mgr", ""]}) v1 -- 0x7fa4b40f0b50 con 0x7fa49c0186c0 2021-02-10T11:22:26.036 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:26.044 7fa4a67fc700 1 -- 172.21.2.112:0/2076233485 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fa4b00009b0 con 0x7fa49c0186c0 2021-02-10T11:22:26.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:26.051 7fa4bc19d700 1 -- 172.21.2.112:0/2076233485 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa49c0186c0 msgr2=0x7fa49c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:26.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:26.051 7fa4bc19d700 1 --2- 172.21.2.112:0/2076233485 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa49c0186c0 0x7fa49c01abc0 crc :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:26.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:26.051 7fa4bc19d700 1 -- 172.21.2.112:0/2076233485 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa4b40c1d80 msgr2=0x7fa4b40c2180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:26.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:26.051 7fa4bc19d700 1 --2- 172.21.2.112:0/2076233485 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa4b40c1d80 0x7fa4b40c2180 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 rx=0x7fa4ac0099d0 tx=0x7fa4ac00bd40).stop 2021-02-10T11:22:26.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:26.051 7fa4bc19d700 1 -- 172.21.2.112:0/2076233485 shutdown_connections 2021-02-10T11:22:26.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:26.051 7fa4bc19d700 1 --2- 172.21.2.112:0/2076233485 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa49c0186c0 0x7fa49c01abc0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:26.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:26.051 7fa4bc19d700 1 --2- 172.21.2.112:0/2076233485 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa4b40c1d80 0x7fa4b40c2180 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:26.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:26.051 7fa4bc19d700 1 --2- 172.21.2.112:0/2076233485 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa4b40b8200 0x7fa4b40b8600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:26.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:26.051 7fa4bc19d700 1 --2- 172.21.2.112:0/2076233485 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa4b4190920 0x7fa4b40b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:26.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:26.051 7fa4bc19d700 1 -- 172.21.2.112:0/2076233485 >> 172.21.2.112:0/2076233485 conn(0x7fa4b4094e10 msgr2=0x7fa4b40a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:26.045 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:26.051 7fa4bc19d700 1 -- 172.21.2.112:0/2076233485 shutdown_connections 2021-02-10T11:22:26.045 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:26.051 7fa4bc19d700 1 -- 172.21.2.112:0/2076233485 wait complete. 2021-02-10T11:22:26.052 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000348614 2021-02-10T11:22:26.484 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:26.475 7f659295a700 1 Processor -- start 2021-02-10T11:22:26.485 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:26.475 7f659295a700 1 -- start start 2021-02-10T11:22:26.485 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:26.475 7f659295a700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f658c1b0470 0x7f658c1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:26.485 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:26.475 7f659295a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f658c0f20e0 0x7f658c0f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:26.485 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:26.475 7f659295a700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f658c0f0b50 con 0x7f658c0ef610 2021-02-10T11:22:26.485 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:26.475 7f659295a700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f658c0f0820 con 0x7f658c1b0470 2021-02-10T11:22:26.485 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:26.475 7f659295a700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f658c0f24e0 con 0x7f658c0f20e0 2021-02-10T11:22:26.485 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:26.476 7f6590f77700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f658c0f20e0 0x7f658c0f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:26.485 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:26.476 7f658bfff700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f658c1b0470 0x7f658c1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:26.485 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:26.476 7f658b7fe700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7f658c0ef610 0x7f658c0efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:38120/0 (socket says 172.21.2.112:38120) 2021-02-10T11:22:26.485 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:26.476 7f6590f77700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f658c0f20e0 0x7f658c0f0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3301/0 says I am v2:172.21.2.112:44782/0 (socket says 172.21.2.112:44782) 2021-02-10T11:22:26.485 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:26.476 7f658b7fe700 1 -- 172.21.2.112:0/2659213691 learned_addr learned my addr 172.21.2.112:0/2659213691 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:22:26.485 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:26.476 7f658bfff700 1 -- 172.21.2.112:0/2659213691 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f658c0f20e0 msgr2=0x7f658c0f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:26.486 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:26.476 7f658bfff700 1 --2- 172.21.2.112:0/2659213691 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f658c0f20e0 0x7f658c0f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:26.486 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:26.476 7f658bfff700 1 -- 172.21.2.112:0/2659213691 >> v1:172.21.2.120:6789/0 conn(0x7f658c0ef610 legacy=0x7f658c0efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:26.486 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:26.476 7f658bfff700 1 -- 172.21.2.112:0/2659213691 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f658c0e52e0 con 0x7f658c1b0470 2021-02-10T11:22:26.486 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:26.476 7f658bfff700 1 --2- 172.21.2.112:0/2659213691 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f658c1b0470 0x7f658c1b0870 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7f657c006650 tx=0x7f657c009a80).ready entity=mon.1 client_cookie=fa99b1761640c2f2 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:26.486 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:26.476 7f658affd700 1 -- 172.21.2.112:0/2659213691 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f657c0009b0 con 0x7f658c1b0470 2021-02-10T11:22:26.486 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:26.477 7f658affd700 1 -- 172.21.2.112:0/2659213691 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f657c006740 con 0x7f658c1b0470 2021-02-10T11:22:26.486 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:26.477 7f658affd700 1 -- 172.21.2.112:0/2659213691 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f657c00c0c0 con 0x7f658c1b0470 2021-02-10T11:22:26.486 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:26.477 7f659295a700 1 -- 172.21.2.112:0/2659213691 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f658c1b0470 msgr2=0x7f658c1b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:26.486 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:26.477 7f659295a700 1 --2- 172.21.2.112:0/2659213691 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f658c1b0470 0x7f658c1b0870 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7f657c006650 tx=0x7f657c009a80).stop 2021-02-10T11:22:26.486 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:26.477 7f659295a700 1 -- 172.21.2.112:0/2659213691 shutdown_connections 2021-02-10T11:22:26.486 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:26.477 7f659295a700 1 --2- 172.21.2.112:0/2659213691 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f658c0f20e0 0x7f658c0f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:26.487 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:26.477 7f659295a700 1 --2- 172.21.2.112:0/2659213691 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f658c1b0470 0x7f658c1b0870 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:26.487 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:26.477 7f659295a700 1 -- 172.21.2.112:0/2659213691 >> 172.21.2.112:0/2659213691 conn(0x7f658c0ce3e0 msgr2=0x7f658c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:26.487 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:26.477 7f659295a700 1 -- 172.21.2.112:0/2659213691 shutdown_connections 2021-02-10T11:22:26.487 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:26.477 7f659295a700 1 -- 172.21.2.112:0/2659213691 wait complete. 2021-02-10T11:22:26.488 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:26.492 7f659295a700 1 Processor -- start 2021-02-10T11:22:26.489 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:26.492 7f659295a700 1 -- start start 2021-02-10T11:22:26.489 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:26.492 7f659295a700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f658c0a0420 0x7f658c0a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:26.489 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:26.492 7f659295a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f658c189d00 0x7f658c18a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:26.489 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:26.492 7f659295a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f658c18a720 0x7f658c0b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:26.489 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:26.492 7f659295a700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f658c189800 con 0x7f658c189d00 2021-02-10T11:22:26.489 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:26.492 7f659295a700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f658c18ab20 con 0x7f658c0a0420 2021-02-10T11:22:26.489 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:26.492 7f659295a700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f658c0a1140 con 0x7f658c18a720 2021-02-10T11:22:26.489 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:26.493 7f658b7fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f658c189d00 0x7f658c18a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:26.489 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:26.493 7f658bfff700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f658c0a0420 0x7f658c0a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:26.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:26.493 7f658bfff700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f658c0a0420 0x7f658c0a0820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3300/0 says I am v2:172.21.2.112:47382/0 (socket says 172.21.2.112:47382) 2021-02-10T11:22:26.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:26.493 7f658bfff700 1 -- 172.21.2.112:0/2109361939 learned_addr learned my addr 172.21.2.112:0/2109361939 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:22:26.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:26.493 7f6590f77700 1 --2- 172.21.2.112:0/2109361939 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f658c18a720 0x7f658c0b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:26.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:26.493 7f658bfff700 1 -- 172.21.2.112:0/2109361939 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f658c18a720 msgr2=0x7f658c0b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:26.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:26.493 7f658bfff700 1 --2- 172.21.2.112:0/2109361939 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f658c18a720 0x7f658c0b7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:26.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:26.493 7f658bfff700 1 -- 172.21.2.112:0/2109361939 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f658c189d00 msgr2=0x7f658c18a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:26.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:26.493 7f658bfff700 1 --2- 172.21.2.112:0/2109361939 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f658c189d00 0x7f658c18a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:26.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:26.493 7f658bfff700 1 -- 172.21.2.112:0/2109361939 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f658c0b8200 con 0x7f658c0a0420 2021-02-10T11:22:26.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:26.493 7f658b7fe700 1 --2- 172.21.2.112:0/2109361939 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f658c189d00 0x7f658c18a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:22:26.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:26.493 7f658bfff700 1 --2- 172.21.2.112:0/2109361939 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f658c0a0420 0x7f658c0a0820 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7f657c009630 tx=0x7f657c00b970).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:26.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:26.494 7f6588ff9700 1 -- 172.21.2.112:0/2109361939 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f657c002030 con 0x7f658c0a0420 2021-02-10T11:22:26.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:26.494 7f6588ff9700 1 -- 172.21.2.112:0/2109361939 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f657c002370 con 0x7f658c0a0420 2021-02-10T11:22:26.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:26.494 7f6588ff9700 1 -- 172.21.2.112:0/2109361939 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f657c002890 con 0x7f658c0a0420 2021-02-10T11:22:26.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:26.494 7f659295a700 1 -- 172.21.2.112:0/2109361939 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f658c0b8200 con 0x7f658c0a0420 2021-02-10T11:22:26.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:26.494 7f659295a700 1 -- 172.21.2.112:0/2109361939 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f658c0b8790 con 0x7f658c0a0420 2021-02-10T11:22:26.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:26.494 7f6588ff9700 1 -- 172.21.2.112:0/2109361939 <== mon.1 v2:172.21.2.120:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f657c002020 con 0x7f658c0a0420 2021-02-10T11:22:26.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:26.494 7f6588ff9700 1 --2- 172.21.2.112:0/2109361939 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f656c0186c0 0x7f656c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:26.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:26.494 7f6588ff9700 1 -- 172.21.2.112:0/2109361939 <== mon.1 v2:172.21.2.120:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f657c032550 con 0x7f658c0a0420 2021-02-10T11:22:26.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:26.495 7f658b7fe700 1 --2- 172.21.2.112:0/2109361939 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f656c0186c0 0x7f656c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:26.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:26.495 7f658b7fe700 1 --2- 172.21.2.112:0/2109361939 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f656c0186c0 0x7f656c01abc0 crc :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:26.518 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:26.525 7f659295a700 1 -- 172.21.2.112:0/2109361939 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f658c105ae0 con 0x7f658c0a0420 2021-02-10T11:22:26.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:26.528 7f6588ff9700 1 -- 172.21.2.112:0/2109361939 <== mon.1 v2:172.21.2.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f657c0093a0 con 0x7f658c0a0420 2021-02-10T11:22:26.652 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:26.659 7f659295a700 1 -- 172.21.2.112:0/2109361939 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348614", "target": ["mgr", ""]}) v1 -- 0x7f658c0b8790 con 0x7f656c0186c0 2021-02-10T11:22:26.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:26.663 7f6588ff9700 1 -- 172.21.2.112:0/2109361939 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f6580000dd0 con 0x7f656c0186c0 2021-02-10T11:22:26.655 INFO:teuthology.orchestra.run.gibba012.stdout:/volumes/_nogroup/subvolume_0000000000348614/68b87d41-98e4-4f8d-b452-703fac8cde51 2021-02-10T11:22:26.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:26.664 7f659295a700 1 -- 172.21.2.112:0/2109361939 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f656c0186c0 msgr2=0x7f656c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:26.657 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:26.664 7f659295a700 1 --2- 172.21.2.112:0/2109361939 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f656c0186c0 0x7f656c01abc0 crc :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:26.657 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:26.664 7f659295a700 1 -- 172.21.2.112:0/2109361939 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f658c0a0420 msgr2=0x7f658c0a0820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:26.657 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:26.664 7f659295a700 1 --2- 172.21.2.112:0/2109361939 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f658c0a0420 0x7f658c0a0820 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7f657c009630 tx=0x7f657c00b970).stop 2021-02-10T11:22:26.657 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:26.664 7f659295a700 1 -- 172.21.2.112:0/2109361939 shutdown_connections 2021-02-10T11:22:26.657 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:26.664 7f659295a700 1 --2- 172.21.2.112:0/2109361939 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f656c0186c0 0x7f656c01abc0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:26.657 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:26.664 7f659295a700 1 --2- 172.21.2.112:0/2109361939 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f658c18a720 0x7f658c0b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:26.657 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:26.664 7f659295a700 1 --2- 172.21.2.112:0/2109361939 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f658c189d00 0x7f658c18a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:26.657 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:26.664 7f659295a700 1 --2- 172.21.2.112:0/2109361939 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f658c0a0420 0x7f658c0a0820 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:26.657 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:26.664 7f659295a700 1 -- 172.21.2.112:0/2109361939 >> 172.21.2.112:0/2109361939 conn(0x7f658c094e10 msgr2=0x7f658c0a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:26.658 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:26.664 7f659295a700 1 -- 172.21.2.112:0/2109361939 shutdown_connections 2021-02-10T11:22:26.658 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:26.664 7f659295a700 1 -- 172.21.2.112:0/2109361939 wait complete. 2021-02-10T11:22:26.665 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000348614 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000348614/68b87d41-98e4-4f8d-b452-703fac8cde51 2021-02-10T11:22:26.665 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:22:26.665 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:22:26.665 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:22:27.001 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:22:27.001 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:22:27.001 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348614/68b87d41-98e4-4f8d-b452-703fac8cde51/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:22:27.381 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:22:27.381 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:22:27.381 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0223863 s, 46.8 MB/s 2021-02-10T11:22:27.382 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:22:27.382 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:22:27.382 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:22:27.719 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:22:27.719 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:22:27.719 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348614/68b87d41-98e4-4f8d-b452-703fac8cde51/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:22:28.094 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:22:28.094 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:22:28.094 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0236963 s, 44.3 MB/s 2021-02-10T11:22:28.095 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:22:28.095 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:22:28.095 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:22:28.430 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:22:28.431 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:22:28.431 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348614/68b87d41-98e4-4f8d-b452-703fac8cde51/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:22:28.808 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:22:28.808 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:22:28.809 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0252537 s, 41.5 MB/s 2021-02-10T11:22:28.809 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:22:28.810 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:22:28.810 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:22:29.145 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:22:29.145 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:22:29.145 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348614/68b87d41-98e4-4f8d-b452-703fac8cde51/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:22:29.518 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:22:29.519 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:22:29.519 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.025057 s, 41.8 MB/s 2021-02-10T11:22:29.520 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:22:29.520 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:22:29.520 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:22:29.856 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:22:29.856 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:22:29.856 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348614/68b87d41-98e4-4f8d-b452-703fac8cde51/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:22:30.231 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:22:30.231 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:22:30.231 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0243064 s, 43.1 MB/s 2021-02-10T11:22:30.232 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:22:30.232 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:22:30.232 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:22:30.568 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:22:30.568 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:22:30.568 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348614/68b87d41-98e4-4f8d-b452-703fac8cde51/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:22:30.943 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:22:30.943 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:22:30.943 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0237402 s, 44.2 MB/s 2021-02-10T11:22:30.944 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:22:30.944 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:22:30.944 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:22:31.279 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:22:31.279 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:22:31.280 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348614/68b87d41-98e4-4f8d-b452-703fac8cde51/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:22:31.655 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:22:31.655 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:22:31.656 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0245111 s, 42.8 MB/s 2021-02-10T11:22:31.657 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:22:31.658 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:22:31.658 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:22:31.995 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:22:31.996 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:22:31.996 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348614/68b87d41-98e4-4f8d-b452-703fac8cde51/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:22:32.369 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:22:32.369 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:22:32.369 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0221365 s, 47.4 MB/s 2021-02-10T11:22:32.370 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:22:32.370 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:22:32.371 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:22:32.706 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:22:32.706 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:22:32.706 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348614/68b87d41-98e4-4f8d-b452-703fac8cde51/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:22:33.081 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:22:33.082 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:22:33.082 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.024164 s, 43.4 MB/s 2021-02-10T11:22:33.083 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:22:33.083 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:22:33.083 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:22:33.418 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:22:33.418 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:22:33.419 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348614/68b87d41-98e4-4f8d-b452-703fac8cde51/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:22:33.794 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:22:33.794 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:22:33.794 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0241468 s, 43.4 MB/s 2021-02-10T11:22:33.795 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000348615 2021-02-10T11:22:34.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.215 7f172cb13700 1 Processor -- start 2021-02-10T11:22:34.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.215 7f172cb13700 1 -- start start 2021-02-10T11:22:34.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.216 7f172cb13700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f17281b0870 0x7f17280f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:34.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.216 7f172cb13700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17281b0c70 con 0x7f17281b0470 2021-02-10T11:22:34.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.216 7f172cb13700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17280f0b50 con 0x7f17280efe00 2021-02-10T11:22:34.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.216 7f172cb13700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17280e5ea0 con 0x7f17281b0870 2021-02-10T11:22:34.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.216 7f172605d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f17281b0870 0x7f17280f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:34.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.216 7f172685e700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f17281b0470 0x7f17280ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:44572/0 (socket says 172.21.2.112:44572) 2021-02-10T11:22:34.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.216 7f172685e700 1 -- 172.21.2.112:0/449781752 learned_addr learned my addr 172.21.2.112:0/449781752 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:22:34.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.217 7f172605d700 1 -- 172.21.2.112:0/449781752 >> v1:172.21.2.120:6789/0 conn(0x7f17280efe00 legacy=0x7f17280f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:34.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.217 7f172605d700 1 -- 172.21.2.112:0/449781752 >> v1:172.21.2.112:6789/0 conn(0x7f17281b0470 legacy=0x7f17280ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:34.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.217 7f172605d700 1 -- 172.21.2.112:0/449781752 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17280e52e0 con 0x7f17281b0870 2021-02-10T11:22:34.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.217 7f172605d700 1 --2- 172.21.2.112:0/449781752 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f17281b0870 0x7f17280f20e0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 rx=0x7f17200069d0 tx=0x7f1720009e40).ready entity=mon.0 client_cookie=c5154daadeea4f2 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:34.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.217 7f172585c700 1 -- 172.21.2.112:0/449781752 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f172000b510 con 0x7f17281b0870 2021-02-10T11:22:34.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.217 7f172585c700 1 -- 172.21.2.112:0/449781752 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f172000b790 con 0x7f17281b0870 2021-02-10T11:22:34.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.217 7f172585c700 1 -- 172.21.2.112:0/449781752 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f172000bcb0 con 0x7f17281b0870 2021-02-10T11:22:34.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.217 7f172cb13700 1 -- 172.21.2.112:0/449781752 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f17281b0870 msgr2=0x7f17280f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:34.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.217 7f172cb13700 1 --2- 172.21.2.112:0/449781752 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f17281b0870 0x7f17280f20e0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 rx=0x7f17200069d0 tx=0x7f1720009e40).stop 2021-02-10T11:22:34.227 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.218 7f172cb13700 1 -- 172.21.2.112:0/449781752 shutdown_connections 2021-02-10T11:22:34.227 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.218 7f172cb13700 1 --2- 172.21.2.112:0/449781752 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f17281b0870 0x7f17280f20e0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:34.227 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.218 7f172cb13700 1 -- 172.21.2.112:0/449781752 >> 172.21.2.112:0/449781752 conn(0x7f17280ce3e0 msgr2=0x7f17280e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:34.227 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.218 7f172cb13700 1 -- 172.21.2.112:0/449781752 shutdown_connections 2021-02-10T11:22:34.227 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.218 7f172cb13700 1 -- 172.21.2.112:0/449781752 wait complete. 2021-02-10T11:22:34.228 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.233 7f172cb13700 1 Processor -- start 2021-02-10T11:22:34.228 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.233 7f172cb13700 1 -- start start 2021-02-10T11:22:34.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.233 7f172cb13700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f17280a3b50 0x7f17281b3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:34.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.233 7f172cb13700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f17281c25b0 0x7f17281b3d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:34.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.233 7f172cb13700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f17281b4350 0x7f17281b18a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:34.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.233 7f172cb13700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17281b3190 con 0x7f17281b4350 2021-02-10T11:22:34.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.233 7f172cb13700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1728087750 con 0x7f17281c25b0 2021-02-10T11:22:34.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.233 7f172cb13700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17280a6bf0 con 0x7f17280a3b50 2021-02-10T11:22:34.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.233 7f172705f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f17281b4350 0x7f17281b18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:34.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.233 7f172605d700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f17281c25b0 0x7f17281b3d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:34.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.233 7f172705f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f17281b4350 0x7f17281b18a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:45120/0 (socket says 172.21.2.112:45120) 2021-02-10T11:22:34.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.233 7f172705f700 1 -- 172.21.2.112:0/3992987974 learned_addr learned my addr 172.21.2.112:0/3992987974 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:22:34.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.233 7f172685e700 1 --2- 172.21.2.112:0/3992987974 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f17280a3b50 0x7f17281b3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:34.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.234 7f172705f700 1 -- 172.21.2.112:0/3992987974 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f17280a3b50 msgr2=0x7f17281b3710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:34.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.234 7f172705f700 1 --2- 172.21.2.112:0/3992987974 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f17280a3b50 0x7f17281b3710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:34.230 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.234 7f172705f700 1 -- 172.21.2.112:0/3992987974 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f17281c25b0 msgr2=0x7f17281b3d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:34.230 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.234 7f172705f700 1 --2- 172.21.2.112:0/3992987974 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f17281c25b0 0x7f17281b3d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:34.230 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.234 7f172705f700 1 -- 172.21.2.112:0/3992987974 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17281b1ec0 con 0x7f17281b4350 2021-02-10T11:22:34.230 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.234 7f172705f700 1 --2- 172.21.2.112:0/3992987974 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f17281b4350 0x7f17281b18a0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 rx=0x7f1718006ed0 tx=0x7f171800b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:34.230 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.234 7f17177fe700 1 -- 172.21.2.112:0/3992987974 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f171800bd50 con 0x7f17281b4350 2021-02-10T11:22:34.230 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.234 7f17177fe700 1 -- 172.21.2.112:0/3992987974 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1718006020 con 0x7f17281b4350 2021-02-10T11:22:34.230 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.234 7f17177fe700 1 -- 172.21.2.112:0/3992987974 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f17180054b0 con 0x7f17281b4350 2021-02-10T11:22:34.230 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.234 7f172cb13700 1 -- 172.21.2.112:0/3992987974 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17281b1ec0 con 0x7f17281b4350 2021-02-10T11:22:34.230 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.234 7f172cb13700 1 -- 172.21.2.112:0/3992987974 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17281b1ec0 con 0x7f17281b4350 2021-02-10T11:22:34.230 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.235 7f17177fe700 1 -- 172.21.2.112:0/3992987974 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1718006200 con 0x7f17281b4350 2021-02-10T11:22:34.230 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.235 7f17177fe700 1 --2- 172.21.2.112:0/3992987974 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f170c0186c0 0x7f170c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:34.230 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.235 7f17177fe700 1 -- 172.21.2.112:0/3992987974 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f17180325a0 con 0x7f17281b4350 2021-02-10T11:22:34.230 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.235 7f172685e700 1 --2- 172.21.2.112:0/3992987974 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f170c0186c0 0x7f170c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:34.231 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.235 7f172685e700 1 --2- 172.21.2.112:0/3992987974 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f170c0186c0 0x7f170c01abc0 crc :-1 s=READY pgs=269 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:34.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.266 7f172cb13700 1 -- 172.21.2.112:0/3992987974 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f172807e6b0 con 0x7f17281b4350 2021-02-10T11:22:34.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.268 7f17177fe700 1 -- 172.21.2.112:0/3992987974 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1718030dd0 con 0x7f17281b4350 2021-02-10T11:22:34.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.399 7f172cb13700 1 -- 172.21.2.112:0/3992987974 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348615", "target": ["mgr", ""]}) v1 -- 0x7f17281b3190 con 0x7f170c0186c0 2021-02-10T11:22:34.416 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.423 7f17177fe700 1 -- 172.21.2.112:0/3992987974 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f171c001050 con 0x7f170c0186c0 2021-02-10T11:22:34.425 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.432 7f172cb13700 1 -- 172.21.2.112:0/3992987974 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f170c0186c0 msgr2=0x7f170c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:34.425 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.432 7f172cb13700 1 --2- 172.21.2.112:0/3992987974 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f170c0186c0 0x7f170c01abc0 crc :-1 s=READY pgs=269 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:34.425 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.432 7f172cb13700 1 -- 172.21.2.112:0/3992987974 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f17281b4350 msgr2=0x7f17281b18a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:34.426 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.432 7f172cb13700 1 --2- 172.21.2.112:0/3992987974 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f17281b4350 0x7f17281b18a0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 rx=0x7f1718006ed0 tx=0x7f171800b960).stop 2021-02-10T11:22:34.426 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.432 7f172cb13700 1 -- 172.21.2.112:0/3992987974 shutdown_connections 2021-02-10T11:22:34.426 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.432 7f172cb13700 1 --2- 172.21.2.112:0/3992987974 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f170c0186c0 0x7f170c01abc0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:34.426 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.432 7f172cb13700 1 --2- 172.21.2.112:0/3992987974 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f17281c25b0 0x7f17281b3d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:34.426 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.432 7f172cb13700 1 --2- 172.21.2.112:0/3992987974 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f17281b4350 0x7f17281b18a0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:34.426 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.432 7f172cb13700 1 --2- 172.21.2.112:0/3992987974 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f17280a3b50 0x7f17281b3710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:34.427 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.432 7f172cb13700 1 -- 172.21.2.112:0/3992987974 >> 172.21.2.112:0/3992987974 conn(0x7f17280886d0 msgr2=0x7f1728081c40 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:34.427 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.432 7f172cb13700 1 -- 172.21.2.112:0/3992987974 shutdown_connections 2021-02-10T11:22:34.427 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.432 7f172cb13700 1 -- 172.21.2.112:0/3992987974 wait complete. 2021-02-10T11:22:34.433 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000348615 2021-02-10T11:22:34.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.854 7fa0b4cf1700 1 Processor -- start 2021-02-10T11:22:34.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.854 7fa0b4cf1700 1 -- start start 2021-02-10T11:22:34.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.855 7fa0b4cf1700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fa0b01b0870 0x7fa0b00f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:34.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.855 7fa0b4cf1700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa0b00efe00 0x7fa0b00f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:34.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.855 7fa0b4cf1700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0b01b0c70 con 0x7fa0b01b0470 2021-02-10T11:22:34.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.855 7fa0b4cf1700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0b00f0b50 con 0x7fa0b01b0870 2021-02-10T11:22:34.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.855 7fa0b4cf1700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0b00f0820 con 0x7fa0b00efe00 2021-02-10T11:22:34.865 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.855 7fa0ae2e7700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fa0b01b0870 0x7fa0b00f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:34.865 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.855 7fa0aeae8700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fa0b01b0470 0x7fa0b00ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:44586/0 (socket says 172.21.2.112:44586) 2021-02-10T11:22:34.865 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.855 7fa0aeae8700 1 -- 172.21.2.112:0/2183264189 learned_addr learned my addr 172.21.2.112:0/2183264189 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:22:34.865 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.855 7fa0af2e9700 1 --2- 172.21.2.112:0/2183264189 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa0b00efe00 0x7fa0b00f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:34.865 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.855 7fa0ae2e7700 1 -- 172.21.2.112:0/2183264189 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa0b00efe00 msgr2=0x7fa0b00f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:34.865 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.855 7fa0ae2e7700 1 --2- 172.21.2.112:0/2183264189 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa0b00efe00 0x7fa0b00f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:34.865 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.855 7fa0ae2e7700 1 -- 172.21.2.112:0/2183264189 >> v1:172.21.2.112:6789/0 conn(0x7fa0b01b0470 legacy=0x7fa0b00ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:34.865 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.855 7fa0adae6700 1 -- 172.21.2.112:0/2183264189 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3332992000 0 0) 0x7fa0a0001180 con 0x7fa0b01b0470 2021-02-10T11:22:34.865 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.855 7fa0ae2e7700 1 -- 172.21.2.112:0/2183264189 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0b00e52e0 con 0x7fa0b01b0870 2021-02-10T11:22:34.865 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.855 7fa0af2e9700 1 --2- 172.21.2.112:0/2183264189 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa0b00efe00 0x7fa0b00f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:22:34.865 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.855 7fa0ae2e7700 1 --2- 172.21.2.112:0/2183264189 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa0b01b0870 0x7fa0b00f20e0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 rx=0x7fa0a4006650 tx=0x7fa0a4009a80).ready entity=mon.0 client_cookie=54fb7da541ff601c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:34.865 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.856 7fa0adae6700 1 -- 172.21.2.112:0/2183264189 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa0a40009b0 con 0x7fa0b01b0870 2021-02-10T11:22:34.865 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.856 7fa0adae6700 1 -- 172.21.2.112:0/2183264189 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0a400c0c0 con 0x7fa0b01b0870 2021-02-10T11:22:34.866 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.856 7fa0adae6700 1 -- 172.21.2.112:0/2183264189 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa0a400c5e0 con 0x7fa0b01b0870 2021-02-10T11:22:34.866 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.856 7fa0b4cf1700 1 -- 172.21.2.112:0/2183264189 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa0b01b0870 msgr2=0x7fa0b00f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:34.866 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.856 7fa0b4cf1700 1 --2- 172.21.2.112:0/2183264189 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa0b01b0870 0x7fa0b00f20e0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 rx=0x7fa0a4006650 tx=0x7fa0a4009a80).stop 2021-02-10T11:22:34.866 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.857 7fa0b4cf1700 1 -- 172.21.2.112:0/2183264189 shutdown_connections 2021-02-10T11:22:34.866 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.857 7fa0b4cf1700 1 --2- 172.21.2.112:0/2183264189 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa0b00efe00 0x7fa0b00f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:34.866 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.857 7fa0b4cf1700 1 --2- 172.21.2.112:0/2183264189 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa0b01b0870 0x7fa0b00f20e0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:34.866 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.857 7fa0b4cf1700 1 -- 172.21.2.112:0/2183264189 >> 172.21.2.112:0/2183264189 conn(0x7fa0b00ce3e0 msgr2=0x7fa0b00e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:34.866 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.857 7fa0b4cf1700 1 -- 172.21.2.112:0/2183264189 shutdown_connections 2021-02-10T11:22:34.866 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.857 7fa0b4cf1700 1 -- 172.21.2.112:0/2183264189 wait complete. 2021-02-10T11:22:34.868 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.872 7fa0b4cf1700 1 Processor -- start 2021-02-10T11:22:34.868 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.872 7fa0b4cf1700 1 -- start start 2021-02-10T11:22:34.868 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.872 7fa0b4cf1700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa0b00a0420 0x7fa0b00a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:34.868 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.872 7fa0b4cf1700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa0b0189d00 0x7fa0b018a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:34.868 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.872 7fa0b4cf1700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa0b018a720 0x7fa0b00b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:34.868 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.872 7fa0b4cf1700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0b0189800 con 0x7fa0b0189d00 2021-02-10T11:22:34.868 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.872 7fa0b4cf1700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0b018ab20 con 0x7fa0b018a720 2021-02-10T11:22:34.868 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.872 7fa0b4cf1700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0b00a1140 con 0x7fa0b00a0420 2021-02-10T11:22:34.868 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.872 7fa0aeae8700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa0b00a0420 0x7fa0b00a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:34.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.872 7fa0aeae8700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa0b00a0420 0x7fa0b00a0820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3301/0 says I am v2:172.21.2.112:44814/0 (socket says 172.21.2.112:44814) 2021-02-10T11:22:34.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.872 7fa0aeae8700 1 -- 172.21.2.112:0/880832722 learned_addr learned my addr 172.21.2.112:0/880832722 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:22:34.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.872 7fa0ae2e7700 1 --2- 172.21.2.112:0/880832722 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa0b0189d00 0x7fa0b018a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:34.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.872 7fa0af2e9700 1 --2- 172.21.2.112:0/880832722 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa0b018a720 0x7fa0b00b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:34.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.873 7fa0ae2e7700 1 -- 172.21.2.112:0/880832722 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa0b00a0420 msgr2=0x7fa0b00a0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:34.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.873 7fa0ae2e7700 1 --2- 172.21.2.112:0/880832722 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa0b00a0420 0x7fa0b00a0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:34.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.873 7fa0ae2e7700 1 -- 172.21.2.112:0/880832722 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa0b018a720 msgr2=0x7fa0b00b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:34.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.873 7fa0ae2e7700 1 --2- 172.21.2.112:0/880832722 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa0b018a720 0x7fa0b00b7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:34.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.873 7fa0ae2e7700 1 -- 172.21.2.112:0/880832722 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0b00b8200 con 0x7fa0b0189d00 2021-02-10T11:22:34.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.873 7fa0ae2e7700 1 --2- 172.21.2.112:0/880832722 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa0b0189d00 0x7fa0b018a100 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 rx=0x7fa0a400b4b0 tx=0x7fa0a400b850).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:34.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.873 7fa09f7fe700 1 -- 172.21.2.112:0/880832722 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa0a4000c90 con 0x7fa0b0189d00 2021-02-10T11:22:34.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.873 7fa09f7fe700 1 -- 172.21.2.112:0/880832722 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0a4002030 con 0x7fa0b0189d00 2021-02-10T11:22:34.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.873 7fa0b4cf1700 1 -- 172.21.2.112:0/880832722 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0b00b8200 con 0x7fa0b0189d00 2021-02-10T11:22:34.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.873 7fa0b4cf1700 1 -- 172.21.2.112:0/880832722 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0b00b8200 con 0x7fa0b0189d00 2021-02-10T11:22:34.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.873 7fa09f7fe700 1 -- 172.21.2.112:0/880832722 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa0a4002510 con 0x7fa0b0189d00 2021-02-10T11:22:34.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.874 7fa09f7fe700 1 -- 172.21.2.112:0/880832722 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa0a4000940 con 0x7fa0b0189d00 2021-02-10T11:22:34.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.874 7fa09f7fe700 1 --2- 172.21.2.112:0/880832722 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa094018730 0x7fa09401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:34.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.874 7fa09f7fe700 1 -- 172.21.2.112:0/880832722 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fa0a4032790 con 0x7fa0b0189d00 2021-02-10T11:22:34.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.874 7fa0aeae8700 1 --2- 172.21.2.112:0/880832722 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa094018730 0x7fa09401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:34.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.874 7fa0aeae8700 1 --2- 172.21.2.112:0/880832722 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa094018730 0x7fa09401ac10 crc :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:34.897 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.905 7fa0b4cf1700 1 -- 172.21.2.112:0/880832722 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0b0105ae0 con 0x7fa0b0189d00 2021-02-10T11:22:34.900 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:34.907 7fa09f7fe700 1 -- 172.21.2.112:0/880832722 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa0a4000c90 con 0x7fa0b0189d00 2021-02-10T11:22:35.027 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:35.034 7fa0b4cf1700 1 -- 172.21.2.112:0/880832722 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348615", "target": ["mgr", ""]}) v1 -- 0x7fa0b018ab20 con 0x7fa094018730 2021-02-10T11:22:35.031 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:35.039 7fa09f7fe700 1 -- 172.21.2.112:0/880832722 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7fa0a0001180 con 0x7fa094018730 2021-02-10T11:22:35.034 INFO:teuthology.orchestra.run.gibba012.stdout:/volumes/_nogroup/subvolume_0000000000348615/9c0e4c55-fba3-4f7a-a57f-f2960ec3109d 2021-02-10T11:22:35.036 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:35.043 7fa0b4cf1700 1 -- 172.21.2.112:0/880832722 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa094018730 msgr2=0x7fa09401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:35.036 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:35.043 7fa0b4cf1700 1 --2- 172.21.2.112:0/880832722 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa094018730 0x7fa09401ac10 crc :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:35.036 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:35.043 7fa0b4cf1700 1 -- 172.21.2.112:0/880832722 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa0b0189d00 msgr2=0x7fa0b018a100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:35.037 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:35.043 7fa0b4cf1700 1 --2- 172.21.2.112:0/880832722 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa0b0189d00 0x7fa0b018a100 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 rx=0x7fa0a400b4b0 tx=0x7fa0a400b850).stop 2021-02-10T11:22:35.037 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:35.044 7fa0b4cf1700 1 -- 172.21.2.112:0/880832722 shutdown_connections 2021-02-10T11:22:35.037 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:35.044 7fa0b4cf1700 1 --2- 172.21.2.112:0/880832722 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa094018730 0x7fa09401ac10 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:35.037 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:35.044 7fa0b4cf1700 1 --2- 172.21.2.112:0/880832722 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa0b018a720 0x7fa0b00b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:35.037 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:35.044 7fa0b4cf1700 1 --2- 172.21.2.112:0/880832722 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa0b0189d00 0x7fa0b018a100 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:35.037 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:35.044 7fa0b4cf1700 1 --2- 172.21.2.112:0/880832722 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa0b00a0420 0x7fa0b00a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:35.037 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:35.044 7fa0b4cf1700 1 -- 172.21.2.112:0/880832722 >> 172.21.2.112:0/880832722 conn(0x7fa0b0094e10 msgr2=0x7fa0b00a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:35.037 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:35.044 7fa0b4cf1700 1 -- 172.21.2.112:0/880832722 shutdown_connections 2021-02-10T11:22:35.037 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:35.044 7fa0b4cf1700 1 -- 172.21.2.112:0/880832722 wait complete. 2021-02-10T11:22:35.044 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000348615 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000348615/9c0e4c55-fba3-4f7a-a57f-f2960ec3109d 2021-02-10T11:22:35.044 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:22:35.044 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:22:35.044 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:22:35.380 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:22:35.380 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:22:35.381 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348615/9c0e4c55-fba3-4f7a-a57f-f2960ec3109d/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:22:35.762 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:22:35.763 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:22:35.763 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0253549 s, 41.4 MB/s 2021-02-10T11:22:35.763 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:22:35.764 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:22:35.764 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:22:36.099 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:22:36.099 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:22:36.099 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348615/9c0e4c55-fba3-4f7a-a57f-f2960ec3109d/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:22:36.477 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:22:36.477 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:22:36.477 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0259774 s, 40.4 MB/s 2021-02-10T11:22:36.478 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:22:36.478 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:22:36.478 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:22:36.812 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:22:36.812 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:22:36.813 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348615/9c0e4c55-fba3-4f7a-a57f-f2960ec3109d/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:22:37.187 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:22:37.187 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:22:37.187 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0240294 s, 43.6 MB/s 2021-02-10T11:22:37.188 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:22:37.188 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:22:37.188 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:22:37.524 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:22:37.525 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:22:37.525 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348615/9c0e4c55-fba3-4f7a-a57f-f2960ec3109d/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:22:37.598 DEBUG:teuthology.orchestra.run.gibba012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:22:37.601 DEBUG:teuthology.orchestra.run.gibba020:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:22:37.902 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:22:37.902 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:22:37.902 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0233755 s, 44.9 MB/s 2021-02-10T11:22:37.903 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:22:37.904 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:22:37.904 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:22:38.239 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:22:38.240 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:22:38.240 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348615/9c0e4c55-fba3-4f7a-a57f-f2960ec3109d/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:22:38.614 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:22:38.614 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:22:38.614 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0234457 s, 44.7 MB/s 2021-02-10T11:22:38.615 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:22:38.615 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:22:38.615 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:22:38.950 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:22:38.951 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:22:38.951 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348615/9c0e4c55-fba3-4f7a-a57f-f2960ec3109d/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:22:39.325 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:22:39.325 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:22:39.325 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0225379 s, 46.5 MB/s 2021-02-10T11:22:39.326 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:22:39.326 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:22:39.326 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:22:39.662 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:22:39.663 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:22:39.663 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348615/9c0e4c55-fba3-4f7a-a57f-f2960ec3109d/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:22:40.039 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:22:40.039 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:22:40.039 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0241455 s, 43.4 MB/s 2021-02-10T11:22:40.040 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:22:40.040 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:22:40.040 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:22:40.376 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:22:40.377 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:22:40.377 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348615/9c0e4c55-fba3-4f7a-a57f-f2960ec3109d/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:22:40.753 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:22:40.753 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:22:40.753 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0257221 s, 40.8 MB/s 2021-02-10T11:22:40.754 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:22:40.755 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:22:40.755 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:22:41.090 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:22:41.090 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:22:41.091 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348615/9c0e4c55-fba3-4f7a-a57f-f2960ec3109d/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:22:41.466 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:22:41.466 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:22:41.466 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0243973 s, 43.0 MB/s 2021-02-10T11:22:41.467 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:22:41.467 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:22:41.467 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:22:41.802 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:22:41.802 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:22:41.802 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348615/9c0e4c55-fba3-4f7a-a57f-f2960ec3109d/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:22:42.173 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:22:42.173 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:22:42.173 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0233633 s, 44.9 MB/s 2021-02-10T11:22:42.174 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000348616 2021-02-10T11:22:42.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:42.595 7fbf79b91700 1 Processor -- start 2021-02-10T11:22:42.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:42.595 7fbf79b91700 1 -- start start 2021-02-10T11:22:42.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:42.595 7fbf79b91700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fbf741b0470 0x7fbf741b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:42.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:42.595 7fbf79b91700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbf740ef610 0x7fbf740efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:42.606 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:42.595 7fbf79b91700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf740f0b50 con 0x7fbf740f0030 2021-02-10T11:22:42.606 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:42.595 7fbf79b91700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf740e52e0 con 0x7fbf741b0470 2021-02-10T11:22:42.606 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:42.595 7fbf79b91700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf740e54d0 con 0x7fbf740ef610 2021-02-10T11:22:42.606 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:42.596 7fbf737fe700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fbf741b0470 0x7fbf741b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:42.606 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:42.596 7fbf72ffd700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbf740ef610 0x7fbf740efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:42.606 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:42.596 7fbf73fff700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7fbf740f0030 0x7fbf740f10d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:38158/0 (socket says 172.21.2.112:38158) 2021-02-10T11:22:42.606 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:42.596 7fbf73fff700 1 -- 172.21.2.112:0/1803520753 learned_addr learned my addr 172.21.2.112:0/1803520753 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:22:42.606 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:42.597 7fbf727fc700 1 -- 172.21.2.112:0/1803520753 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 282107118 0 0) 0x7fbf6c001510 con 0x7fbf740f0030 2021-02-10T11:22:42.606 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:42.597 7fbf727fc700 1 -- 172.21.2.112:0/1803520753 --> v1:172.21.2.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf60002fe0 con 0x7fbf740f0030 2021-02-10T11:22:42.606 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:42.597 7fbf737fe700 1 -- 172.21.2.112:0/1803520753 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbf740ef610 msgr2=0x7fbf740efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:42.606 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:42.597 7fbf737fe700 1 --2- 172.21.2.112:0/1803520753 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbf740ef610 0x7fbf740efa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:42.607 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:42.597 7fbf737fe700 1 -- 172.21.2.112:0/1803520753 >> v1:172.21.2.120:6789/0 conn(0x7fbf740f0030 legacy=0x7fbf740f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:42.607 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:42.597 7fbf737fe700 1 -- 172.21.2.112:0/1803520753 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf740f0430 con 0x7fbf741b0470 2021-02-10T11:22:42.607 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:42.597 7fbf72ffd700 1 --2- 172.21.2.112:0/1803520753 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbf740ef610 0x7fbf740efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:22:42.607 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:42.597 7fbf737fe700 1 --2- 172.21.2.112:0/1803520753 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fbf741b0470 0x7fbf741b0870 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7fbf5c006650 tx=0x7fbf5c009a80).ready entity=mon.1 client_cookie=8dd4379517cbad18 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:42.607 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:42.598 7fbf727fc700 1 -- 172.21.2.112:0/1803520753 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbf5c00c0c0 con 0x7fbf741b0470 2021-02-10T11:22:42.607 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:42.598 7fbf727fc700 1 -- 172.21.2.112:0/1803520753 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf5c00c340 con 0x7fbf741b0470 2021-02-10T11:22:42.607 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:42.598 7fbf727fc700 1 -- 172.21.2.112:0/1803520753 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbf5c00c860 con 0x7fbf741b0470 2021-02-10T11:22:42.607 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:42.598 7fbf79b91700 1 -- 172.21.2.112:0/1803520753 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fbf741b0470 msgr2=0x7fbf741b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:42.607 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:42.598 7fbf79b91700 1 --2- 172.21.2.112:0/1803520753 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fbf741b0470 0x7fbf741b0870 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7fbf5c006650 tx=0x7fbf5c009a80).stop 2021-02-10T11:22:42.607 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:42.598 7fbf79b91700 1 -- 172.21.2.112:0/1803520753 shutdown_connections 2021-02-10T11:22:42.607 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:42.598 7fbf79b91700 1 --2- 172.21.2.112:0/1803520753 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbf740ef610 0x7fbf740efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:42.607 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:42.598 7fbf79b91700 1 --2- 172.21.2.112:0/1803520753 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fbf741b0470 0x7fbf741b0870 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:42.608 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:42.598 7fbf79b91700 1 -- 172.21.2.112:0/1803520753 >> 172.21.2.112:0/1803520753 conn(0x7fbf740ce3e0 msgr2=0x7fbf740e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:42.608 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:42.598 7fbf79b91700 1 -- 172.21.2.112:0/1803520753 shutdown_connections 2021-02-10T11:22:42.608 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:42.598 7fbf79b91700 1 -- 172.21.2.112:0/1803520753 wait complete. 2021-02-10T11:22:42.609 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:42.613 7fbf79b91700 1 Processor -- start 2021-02-10T11:22:42.610 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:42.613 7fbf79b91700 1 -- start start 2021-02-10T11:22:42.610 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:42.613 7fbf79b91700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbf74105d10 0x7fbf7409f880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:42.610 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:42.613 7fbf79b91700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fbf740961f0 0x7fbf740965f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:42.610 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:42.613 7fbf79b91700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbf74106890 0x7fbf74106c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:42.610 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:42.613 7fbf79b91700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf74106110 con 0x7fbf74106890 2021-02-10T11:22:42.610 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:42.613 7fbf79b91700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf74106380 con 0x7fbf740961f0 2021-02-10T11:22:42.610 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:42.613 7fbf79b91700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf740a6620 con 0x7fbf74105d10 2021-02-10T11:22:42.610 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:42.614 7fbf73fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbf74106890 0x7fbf74106c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:42.610 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:42.614 7fbf73fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbf74106890 0x7fbf74106c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:45144/0 (socket says 172.21.2.112:45144) 2021-02-10T11:22:42.610 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:42.614 7fbf73fff700 1 -- 172.21.2.112:0/2458403410 learned_addr learned my addr 172.21.2.112:0/2458403410 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:22:42.610 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:42.614 7fbf72ffd700 1 --2- 172.21.2.112:0/2458403410 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fbf740961f0 0x7fbf740965f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:42.610 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:42.614 7fbf737fe700 1 --2- 172.21.2.112:0/2458403410 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbf74105d10 0x7fbf7409f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:42.611 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:42.614 7fbf73fff700 1 -- 172.21.2.112:0/2458403410 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbf74105d10 msgr2=0x7fbf7409f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:42.611 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:42.614 7fbf73fff700 1 --2- 172.21.2.112:0/2458403410 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbf74105d10 0x7fbf7409f880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:42.611 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:42.614 7fbf73fff700 1 -- 172.21.2.112:0/2458403410 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fbf740961f0 msgr2=0x7fbf740965f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:42.611 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:42.614 7fbf73fff700 1 --2- 172.21.2.112:0/2458403410 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fbf740961f0 0x7fbf740965f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:42.611 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:42.614 7fbf73fff700 1 -- 172.21.2.112:0/2458403410 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf741072b0 con 0x7fbf74106890 2021-02-10T11:22:42.611 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:42.614 7fbf73fff700 1 --2- 172.21.2.112:0/2458403410 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbf74106890 0x7fbf74106c90 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 rx=0x7fbf6c007f90 tx=0x7fbf6c00aca0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:42.611 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:42.614 7fbf6bfff700 1 -- 172.21.2.112:0/2458403410 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbf6c0070d0 con 0x7fbf74106890 2021-02-10T11:22:42.611 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:42.614 7fbf6bfff700 1 -- 172.21.2.112:0/2458403410 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf6c007450 con 0x7fbf74106890 2021-02-10T11:22:42.611 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:42.614 7fbf79b91700 1 -- 172.21.2.112:0/2458403410 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf741072b0 con 0x7fbf74106890 2021-02-10T11:22:42.611 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:42.614 7fbf6bfff700 1 -- 172.21.2.112:0/2458403410 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbf6c004920 con 0x7fbf74106890 2021-02-10T11:22:42.611 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:42.614 7fbf79b91700 1 -- 172.21.2.112:0/2458403410 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf74107840 con 0x7fbf74106890 2021-02-10T11:22:42.611 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:42.615 7fbf6bfff700 1 -- 172.21.2.112:0/2458403410 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fbf6c0046a0 con 0x7fbf74106890 2021-02-10T11:22:42.611 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:42.615 7fbf6bfff700 1 --2- 172.21.2.112:0/2458403410 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fbf58018730 0x7fbf5801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:42.612 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:42.615 7fbf6bfff700 1 -- 172.21.2.112:0/2458403410 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fbf6c0329b0 con 0x7fbf74106890 2021-02-10T11:22:42.612 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:42.615 7fbf737fe700 1 --2- 172.21.2.112:0/2458403410 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fbf58018730 0x7fbf5801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:42.612 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:42.616 7fbf737fe700 1 --2- 172.21.2.112:0/2458403410 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fbf58018730 0x7fbf5801abc0 crc :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:42.638 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:42.645 7fbf79b91700 1 -- 172.21.2.112:0/2458403410 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf74105ae0 con 0x7fbf74106890 2021-02-10T11:22:42.640 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:42.648 7fbf6bfff700 1 -- 172.21.2.112:0/2458403410 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fbf6c0011f0 con 0x7fbf74106890 2021-02-10T11:22:42.766 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:42.773 7fbf79b91700 1 -- 172.21.2.112:0/2458403410 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348616", "target": ["mgr", ""]}) v1 -- 0x7fbf74107840 con 0x7fbf58018730 2021-02-10T11:22:42.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:42.799 7fbf6bfff700 1 -- 172.21.2.112:0/2458403410 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fbf5c006420 con 0x7fbf58018730 2021-02-10T11:22:42.800 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:42.807 7fbf79b91700 1 -- 172.21.2.112:0/2458403410 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fbf58018730 msgr2=0x7fbf5801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:42.800 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:42.807 7fbf79b91700 1 --2- 172.21.2.112:0/2458403410 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fbf58018730 0x7fbf5801abc0 crc :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:42.800 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:42.807 7fbf79b91700 1 -- 172.21.2.112:0/2458403410 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbf74106890 msgr2=0x7fbf74106c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:42.800 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:42.807 7fbf79b91700 1 --2- 172.21.2.112:0/2458403410 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbf74106890 0x7fbf74106c90 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 rx=0x7fbf6c007f90 tx=0x7fbf6c00aca0).stop 2021-02-10T11:22:42.800 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:42.807 7fbf79b91700 1 -- 172.21.2.112:0/2458403410 shutdown_connections 2021-02-10T11:22:42.801 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:42.807 7fbf79b91700 1 --2- 172.21.2.112:0/2458403410 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fbf58018730 0x7fbf5801abc0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:42.801 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:42.807 7fbf79b91700 1 --2- 172.21.2.112:0/2458403410 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fbf740961f0 0x7fbf740965f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:42.801 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:42.807 7fbf79b91700 1 --2- 172.21.2.112:0/2458403410 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbf74106890 0x7fbf74106c90 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:42.801 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:42.807 7fbf79b91700 1 --2- 172.21.2.112:0/2458403410 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbf74105d10 0x7fbf7409f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:42.801 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:42.807 7fbf79b91700 1 -- 172.21.2.112:0/2458403410 >> 172.21.2.112:0/2458403410 conn(0x7fbf74095a70 msgr2=0x7fbf740bf050 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:42.801 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:42.807 7fbf79b91700 1 -- 172.21.2.112:0/2458403410 shutdown_connections 2021-02-10T11:22:42.801 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:42.807 7fbf79b91700 1 -- 172.21.2.112:0/2458403410 wait complete. 2021-02-10T11:22:42.808 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000348616 2021-02-10T11:22:43.239 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:43.230 7f2055c8b700 1 Processor -- start 2021-02-10T11:22:43.239 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:43.230 7f2055c8b700 1 -- start start 2021-02-10T11:22:43.239 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:43.230 7f2055c8b700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f20501b0870 0x7f20500f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:43.239 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:43.230 7f2055c8b700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20501b0c70 con 0x7f20500efe00 2021-02-10T11:22:43.239 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:43.230 7f2055c8b700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20500f0b50 con 0x7f20501b0470 2021-02-10T11:22:43.239 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:43.230 7f2055c8b700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20500e5ea0 con 0x7f20501b0870 2021-02-10T11:22:43.239 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:43.230 7f204effd700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f20501b0870 0x7f20500f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:43.239 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:43.230 7f204effd700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f20501b0870 0x7f20500f20e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:45150/0 (socket says 172.21.2.112:45150) 2021-02-10T11:22:43.239 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:43.230 7f204effd700 1 -- 172.21.2.112:0/3130290688 learned_addr learned my addr 172.21.2.112:0/3130290688 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:22:43.239 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:43.231 7f204effd700 1 -- 172.21.2.112:0/3130290688 >> v1:172.21.2.120:6789/0 conn(0x7f20501b0470 legacy=0x7f20500ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:43.240 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:43.231 7f204effd700 1 -- 172.21.2.112:0/3130290688 >> v1:172.21.2.112:6789/0 conn(0x7f20500efe00 legacy=0x7f20500f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:43.240 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:43.231 7f204effd700 1 -- 172.21.2.112:0/3130290688 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20500e52e0 con 0x7f20501b0870 2021-02-10T11:22:43.240 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:43.231 7f204effd700 1 --2- 172.21.2.112:0/3130290688 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f20501b0870 0x7f20500f20e0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 rx=0x7f2038006700 tx=0x7f2038009b80).ready entity=mon.0 client_cookie=5af60246ae48616d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:43.240 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:43.231 7f204e7fc700 1 -- 172.21.2.112:0/3130290688 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f203800c0c0 con 0x7f20501b0870 2021-02-10T11:22:43.240 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:43.231 7f204e7fc700 1 -- 172.21.2.112:0/3130290688 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f203800c340 con 0x7f20501b0870 2021-02-10T11:22:43.240 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:43.231 7f204e7fc700 1 -- 172.21.2.112:0/3130290688 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f203800c860 con 0x7f20501b0870 2021-02-10T11:22:43.240 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:43.231 7f2055c8b700 1 -- 172.21.2.112:0/3130290688 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f20501b0870 msgr2=0x7f20500f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:43.240 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:43.231 7f2055c8b700 1 --2- 172.21.2.112:0/3130290688 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f20501b0870 0x7f20500f20e0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 rx=0x7f2038006700 tx=0x7f2038009b80).stop 2021-02-10T11:22:43.240 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:43.232 7f2055c8b700 1 -- 172.21.2.112:0/3130290688 shutdown_connections 2021-02-10T11:22:43.240 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:43.232 7f2055c8b700 1 --2- 172.21.2.112:0/3130290688 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f20501b0870 0x7f20500f20e0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:43.240 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:43.232 7f2055c8b700 1 -- 172.21.2.112:0/3130290688 >> 172.21.2.112:0/3130290688 conn(0x7f20500ce3e0 msgr2=0x7f20500e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:43.240 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:43.232 7f2055c8b700 1 -- 172.21.2.112:0/3130290688 shutdown_connections 2021-02-10T11:22:43.240 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:43.232 7f2055c8b700 1 -- 172.21.2.112:0/3130290688 wait complete. 2021-02-10T11:22:43.242 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:43.246 7f2055c8b700 1 Processor -- start 2021-02-10T11:22:43.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:43.247 7f2055c8b700 1 -- start start 2021-02-10T11:22:43.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:43.247 7f2055c8b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2050087510 0x7f20500aedc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:43.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:43.247 7f2055c8b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f20501c68f0 0x7f20501b1f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:43.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:43.247 7f2055c8b700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f205008c4d0 0x7f20501b18a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:43.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:43.247 7f2055c8b700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20500a2d50 con 0x7f2050087510 2021-02-10T11:22:43.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:43.247 7f2055c8b700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20500a2f70 con 0x7f205008c4d0 2021-02-10T11:22:43.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:43.247 7f2055c8b700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20500a3190 con 0x7f20501c68f0 2021-02-10T11:22:43.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:43.247 7f204f7fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2050087510 0x7f20500aedc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:43.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:43.247 7f204ffff700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f205008c4d0 0x7f20501b18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:43.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:43.247 7f204ffff700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f205008c4d0 0x7f20501b18a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3300/0 says I am v2:172.21.2.112:47436/0 (socket says 172.21.2.112:47436) 2021-02-10T11:22:43.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:43.247 7f204ffff700 1 -- 172.21.2.112:0/1199785557 learned_addr learned my addr 172.21.2.112:0/1199785557 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:22:43.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:43.247 7f204f7fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2050087510 0x7f20500aedc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:45158/0 (socket says 172.21.2.112:45158) 2021-02-10T11:22:43.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:43.247 7f204effd700 1 --2- 172.21.2.112:0/1199785557 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f20501c68f0 0x7f20501b1f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:43.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:43.248 7f204f7fe700 1 -- 172.21.2.112:0/1199785557 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f20501c68f0 msgr2=0x7f20501b1f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:43.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:43.248 7f204f7fe700 1 --2- 172.21.2.112:0/1199785557 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f20501c68f0 0x7f20501b1f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:43.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:43.248 7f204f7fe700 1 -- 172.21.2.112:0/1199785557 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f205008c4d0 msgr2=0x7f20501b18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:43.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:43.248 7f204f7fe700 1 --2- 172.21.2.112:0/1199785557 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f205008c4d0 0x7f20501b18a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:43.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:43.248 7f204f7fe700 1 -- 172.21.2.112:0/1199785557 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20500a33b0 con 0x7f2050087510 2021-02-10T11:22:43.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:43.248 7f204f7fe700 1 --2- 172.21.2.112:0/1199785557 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2050087510 0x7f20500aedc0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 rx=0x7f204000b580 tx=0x7f204000b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:43.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:43.248 7f2047fff700 1 -- 172.21.2.112:0/1199785557 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2040006240 con 0x7f2050087510 2021-02-10T11:22:43.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:43.248 7f2047fff700 1 -- 172.21.2.112:0/1199785557 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2040005540 con 0x7f2050087510 2021-02-10T11:22:43.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:43.248 7f2047fff700 1 -- 172.21.2.112:0/1199785557 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2040005a60 con 0x7f2050087510 2021-02-10T11:22:43.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:43.248 7f2055c8b700 1 -- 172.21.2.112:0/1199785557 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20500a33b0 con 0x7f2050087510 2021-02-10T11:22:43.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:43.248 7f2055c8b700 1 -- 172.21.2.112:0/1199785557 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20500a33b0 con 0x7f2050087510 2021-02-10T11:22:43.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:43.248 7f2047fff700 1 -- 172.21.2.112:0/1199785557 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2040006030 con 0x7f2050087510 2021-02-10T11:22:43.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:43.249 7f2047fff700 1 --2- 172.21.2.112:0/1199785557 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f20340186c0 0x7f203401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:43.245 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:43.249 7f2047fff700 1 -- 172.21.2.112:0/1199785557 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f2040032720 con 0x7f2050087510 2021-02-10T11:22:43.245 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:43.249 7f204effd700 1 --2- 172.21.2.112:0/1199785557 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f20340186c0 0x7f203401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:43.245 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:43.249 7f204effd700 1 --2- 172.21.2.112:0/1199785557 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f20340186c0 0x7f203401abc0 crc :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:43.273 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:43.280 7f2055c8b700 1 -- 172.21.2.112:0/1199785557 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f205007e6b0 con 0x7f2050087510 2021-02-10T11:22:43.275 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:43.283 7f2047fff700 1 -- 172.21.2.112:0/1199785557 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2040000c60 con 0x7f2050087510 2021-02-10T11:22:43.404 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:43.412 7f2055c8b700 1 -- 172.21.2.112:0/1199785557 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348616", "target": ["mgr", ""]}) v1 -- 0x7f205007e6b0 con 0x7f20340186c0 2021-02-10T11:22:43.408 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:43.416 7f2047fff700 1 -- 172.21.2.112:0/1199785557 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f2038006430 con 0x7f20340186c0 2021-02-10T11:22:43.411 INFO:teuthology.orchestra.run.gibba012.stdout:/volumes/_nogroup/subvolume_0000000000348616/22fdd624-d2cb-487f-9dc1-b9294e7626fb 2021-02-10T11:22:43.413 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:43.420 7f2055c8b700 1 -- 172.21.2.112:0/1199785557 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f20340186c0 msgr2=0x7f203401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:43.413 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:43.420 7f2055c8b700 1 --2- 172.21.2.112:0/1199785557 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f20340186c0 0x7f203401abc0 crc :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:43.413 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:43.420 7f2055c8b700 1 -- 172.21.2.112:0/1199785557 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2050087510 msgr2=0x7f20500aedc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:43.413 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:43.420 7f2055c8b700 1 --2- 172.21.2.112:0/1199785557 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2050087510 0x7f20500aedc0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 rx=0x7f204000b580 tx=0x7f204000b920).stop 2021-02-10T11:22:43.413 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:43.421 7f2055c8b700 1 -- 172.21.2.112:0/1199785557 shutdown_connections 2021-02-10T11:22:43.414 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:43.421 7f2055c8b700 1 --2- 172.21.2.112:0/1199785557 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f20340186c0 0x7f203401abc0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:43.414 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:43.421 7f2055c8b700 1 --2- 172.21.2.112:0/1199785557 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f205008c4d0 0x7f20501b18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:43.414 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:43.421 7f2055c8b700 1 --2- 172.21.2.112:0/1199785557 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f20501c68f0 0x7f20501b1f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:43.414 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:43.421 7f2055c8b700 1 --2- 172.21.2.112:0/1199785557 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2050087510 0x7f20500aedc0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:43.414 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:43.421 7f2055c8b700 1 -- 172.21.2.112:0/1199785557 >> 172.21.2.112:0/1199785557 conn(0x7f20500886d0 msgr2=0x7f2050081c40 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:43.414 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:43.421 7f2055c8b700 1 -- 172.21.2.112:0/1199785557 shutdown_connections 2021-02-10T11:22:43.414 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:43.421 7f2055c8b700 1 -- 172.21.2.112:0/1199785557 wait complete. 2021-02-10T11:22:43.421 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000348616 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000348616/22fdd624-d2cb-487f-9dc1-b9294e7626fb 2021-02-10T11:22:43.422 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:22:43.422 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:22:43.422 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:22:43.759 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:22:43.759 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:22:43.760 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348616/22fdd624-d2cb-487f-9dc1-b9294e7626fb/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:22:44.141 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:22:44.141 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:22:44.141 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0241142 s, 43.5 MB/s 2021-02-10T11:22:44.142 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:22:44.142 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:22:44.142 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:22:44.478 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:22:44.478 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:22:44.478 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348616/22fdd624-d2cb-487f-9dc1-b9294e7626fb/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:22:44.849 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:22:44.849 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:22:44.849 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0244571 s, 42.9 MB/s 2021-02-10T11:22:44.850 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:22:44.851 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:22:44.851 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:22:45.186 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:22:45.187 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:22:45.187 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348616/22fdd624-d2cb-487f-9dc1-b9294e7626fb/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:22:45.562 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:22:45.562 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:22:45.562 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0234964 s, 44.6 MB/s 2021-02-10T11:22:45.563 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:22:45.563 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:22:45.563 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:22:45.899 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:22:45.900 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:22:45.900 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348616/22fdd624-d2cb-487f-9dc1-b9294e7626fb/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:22:46.276 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:22:46.277 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:22:46.277 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0256166 s, 40.9 MB/s 2021-02-10T11:22:46.278 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:22:46.278 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:22:46.278 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:22:46.612 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:22:46.612 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:22:46.612 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348616/22fdd624-d2cb-487f-9dc1-b9294e7626fb/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:22:46.989 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:22:46.989 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:22:46.989 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0243251 s, 43.1 MB/s 2021-02-10T11:22:46.990 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:22:46.990 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:22:46.990 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:22:47.325 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:22:47.326 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:22:47.326 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348616/22fdd624-d2cb-487f-9dc1-b9294e7626fb/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:22:47.702 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:22:47.702 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:22:47.702 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0244022 s, 43.0 MB/s 2021-02-10T11:22:47.703 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:22:47.704 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:22:47.704 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:22:48.040 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:22:48.041 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:22:48.041 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348616/22fdd624-d2cb-487f-9dc1-b9294e7626fb/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:22:48.417 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:22:48.417 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:22:48.417 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0247146 s, 42.4 MB/s 2021-02-10T11:22:48.418 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:22:48.418 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:22:48.418 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:22:48.753 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:22:48.753 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:22:48.754 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348616/22fdd624-d2cb-487f-9dc1-b9294e7626fb/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:22:49.127 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:22:49.127 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:22:49.127 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0233186 s, 45.0 MB/s 2021-02-10T11:22:49.128 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:22:49.128 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:22:49.128 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:22:49.464 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:22:49.464 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:22:49.465 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348616/22fdd624-d2cb-487f-9dc1-b9294e7626fb/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:22:49.836 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:22:49.837 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:22:49.837 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0233723 s, 44.9 MB/s 2021-02-10T11:22:49.838 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:22:49.838 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:22:49.838 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:22:50.173 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:22:50.174 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:22:50.174 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348616/22fdd624-d2cb-487f-9dc1-b9294e7626fb/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:22:50.549 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:22:50.550 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:22:50.550 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0248137 s, 42.3 MB/s 2021-02-10T11:22:50.551 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000348617 2021-02-10T11:22:50.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:50.971 7ff9d09be700 1 Processor -- start 2021-02-10T11:22:50.981 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:50.971 7ff9d09be700 1 -- start start 2021-02-10T11:22:50.981 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:50.972 7ff9d09be700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7ff9cc1b0470 0x7ff9cc1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:50.981 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:50.972 7ff9d09be700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7ff9cc0ef610 0x7ff9cc0efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:50.981 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:50.972 7ff9d09be700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9cc0f0b50 con 0x7ff9cc0f0030 2021-02-10T11:22:50.981 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:50.972 7ff9d09be700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9cc0e52e0 con 0x7ff9cc1b0470 2021-02-10T11:22:50.981 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:50.972 7ff9d09be700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9cc0e5490 con 0x7ff9cc0ef610 2021-02-10T11:22:50.981 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:50.972 7ff9ca61c700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7ff9cc1b0470 0x7ff9cc1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:50.981 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:50.972 7ff9ca61c700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7ff9cc1b0470 0x7ff9cc1b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:45164/0 (socket says 172.21.2.112:45164) 2021-02-10T11:22:50.981 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:50.972 7ff9cae1d700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7ff9cc0f0030 0x7ff9cc0f10d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:44626/0 (socket says 172.21.2.112:44626) 2021-02-10T11:22:50.981 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:50.972 7ff9ca61c700 1 -- 172.21.2.112:0/3935960546 learned_addr learned my addr 172.21.2.112:0/3935960546 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:22:50.981 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:50.972 7ff9c9e1b700 1 --2- 172.21.2.112:0/3935960546 >> v2:172.21.2.120:3300/0 conn(0x7ff9cc0ef610 0x7ff9cc0efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:50.982 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:50.972 7ff9ca61c700 1 -- 172.21.2.112:0/3935960546 >> v2:172.21.2.120:3300/0 conn(0x7ff9cc0ef610 msgr2=0x7ff9cc0efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:50.982 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:50.973 7ff9ca61c700 1 --2- 172.21.2.112:0/3935960546 >> v2:172.21.2.120:3300/0 conn(0x7ff9cc0ef610 0x7ff9cc0efa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:50.982 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:50.973 7ff9ca61c700 1 -- 172.21.2.112:0/3935960546 >> v1:172.21.2.112:6789/0 conn(0x7ff9cc0f0030 legacy=0x7ff9cc0f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:50.982 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:50.973 7ff9ca61c700 1 -- 172.21.2.112:0/3935960546 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9cc0f0430 con 0x7ff9cc1b0470 2021-02-10T11:22:50.982 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:50.973 7ff9ca61c700 1 --2- 172.21.2.112:0/3935960546 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff9cc1b0470 0x7ff9cc1b0870 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 rx=0x7ff9bc006700 tx=0x7ff9bc009b80).ready entity=mon.0 client_cookie=37d22af72ebdce9a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:50.982 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:50.973 7ff9c961a700 1 -- 172.21.2.112:0/3935960546 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff9bc00c0c0 con 0x7ff9cc1b0470 2021-02-10T11:22:50.982 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:50.973 7ff9c961a700 1 -- 172.21.2.112:0/3935960546 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9bc00c340 con 0x7ff9cc1b0470 2021-02-10T11:22:50.982 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:50.973 7ff9c961a700 1 -- 172.21.2.112:0/3935960546 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff9bc00c860 con 0x7ff9cc1b0470 2021-02-10T11:22:50.982 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:50.973 7ff9d09be700 1 -- 172.21.2.112:0/3935960546 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff9cc1b0470 msgr2=0x7ff9cc1b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:50.982 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:50.973 7ff9d09be700 1 --2- 172.21.2.112:0/3935960546 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff9cc1b0470 0x7ff9cc1b0870 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 rx=0x7ff9bc006700 tx=0x7ff9bc009b80).stop 2021-02-10T11:22:50.982 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:50.974 7ff9d09be700 1 -- 172.21.2.112:0/3935960546 shutdown_connections 2021-02-10T11:22:50.982 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:50.974 7ff9d09be700 1 --2- 172.21.2.112:0/3935960546 >> v2:172.21.2.120:3300/0 conn(0x7ff9cc0ef610 0x7ff9cc0efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:50.982 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:50.974 7ff9d09be700 1 --2- 172.21.2.112:0/3935960546 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff9cc1b0470 0x7ff9cc1b0870 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:50.983 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:50.974 7ff9d09be700 1 -- 172.21.2.112:0/3935960546 >> 172.21.2.112:0/3935960546 conn(0x7ff9cc0ce3e0 msgr2=0x7ff9cc0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:50.983 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:50.974 7ff9d09be700 1 -- 172.21.2.112:0/3935960546 shutdown_connections 2021-02-10T11:22:50.983 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:50.974 7ff9d09be700 1 -- 172.21.2.112:0/3935960546 wait complete. 2021-02-10T11:22:50.984 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:50.988 7ff9d09be700 1 Processor -- start 2021-02-10T11:22:50.984 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:50.988 7ff9d09be700 1 -- start start 2021-02-10T11:22:50.984 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:50.989 7ff9d09be700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff9cc09f140 0x7ff9cc0c1940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:50.984 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:50.989 7ff9d09be700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff9cc0c1f60 0x7ff9cc09e300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:50.985 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:50.989 7ff9d09be700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff9cc09e920 0x7ff9cc0c2960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:50.985 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:50.989 7ff9d09be700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9cc0c23e0 con 0x7ff9cc09e920 2021-02-10T11:22:50.985 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:50.989 7ff9d09be700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9cc09ed20 con 0x7ff9cc0c1f60 2021-02-10T11:22:50.985 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:50.989 7ff9d09be700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9cc105d10 con 0x7ff9cc09f140 2021-02-10T11:22:50.985 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:50.989 7ff9cae1d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff9cc09e920 0x7ff9cc0c2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:50.985 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:50.989 7ff9cae1d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff9cc09e920 0x7ff9cc0c2960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:45170/0 (socket says 172.21.2.112:45170) 2021-02-10T11:22:50.985 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:50.989 7ff9cae1d700 1 -- 172.21.2.112:0/468219399 learned_addr learned my addr 172.21.2.112:0/468219399 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:22:50.985 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:50.989 7ff9c9e1b700 1 --2- 172.21.2.112:0/468219399 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff9cc0c1f60 0x7ff9cc09e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:50.985 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:50.989 7ff9ca61c700 1 --2- 172.21.2.112:0/468219399 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff9cc09f140 0x7ff9cc0c1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:50.985 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:50.990 7ff9cae1d700 1 -- 172.21.2.112:0/468219399 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff9cc09f140 msgr2=0x7ff9cc0c1940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:50.985 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:50.990 7ff9cae1d700 1 --2- 172.21.2.112:0/468219399 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff9cc09f140 0x7ff9cc0c1940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:50.985 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:50.990 7ff9cae1d700 1 -- 172.21.2.112:0/468219399 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff9cc0c1f60 msgr2=0x7ff9cc09e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:50.985 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:50.990 7ff9cae1d700 1 --2- 172.21.2.112:0/468219399 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff9cc0c1f60 0x7ff9cc09e300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:50.986 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:50.990 7ff9cae1d700 1 -- 172.21.2.112:0/468219399 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9cc1a4910 con 0x7ff9cc09e920 2021-02-10T11:22:50.986 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:50.990 7ff9cae1d700 1 --2- 172.21.2.112:0/468219399 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff9cc09e920 0x7ff9cc0c2960 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 rx=0x7ff9c4007f90 tx=0x7ff9c400ac50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:50.986 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:50.990 7ff9baffd700 1 -- 172.21.2.112:0/468219399 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff9c4007020 con 0x7ff9cc09e920 2021-02-10T11:22:50.986 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:50.990 7ff9baffd700 1 -- 172.21.2.112:0/468219399 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9c40072a0 con 0x7ff9cc09e920 2021-02-10T11:22:50.986 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:50.990 7ff9baffd700 1 -- 172.21.2.112:0/468219399 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff9c4004720 con 0x7ff9cc09e920 2021-02-10T11:22:50.986 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:50.990 7ff9d09be700 1 -- 172.21.2.112:0/468219399 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9cc1a4910 con 0x7ff9cc09e920 2021-02-10T11:22:50.986 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:50.990 7ff9d09be700 1 -- 172.21.2.112:0/468219399 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9cc0c3270 con 0x7ff9cc09e920 2021-02-10T11:22:50.986 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:50.991 7ff9baffd700 1 -- 172.21.2.112:0/468219399 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff9c4007480 con 0x7ff9cc09e920 2021-02-10T11:22:50.986 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:50.991 7ff9baffd700 1 --2- 172.21.2.112:0/468219399 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff9b00186c0 0x7ff9b001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:50.986 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:50.991 7ff9baffd700 1 -- 172.21.2.112:0/468219399 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7ff9c40328d0 con 0x7ff9cc09e920 2021-02-10T11:22:50.986 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:50.991 7ff9ca61c700 1 --2- 172.21.2.112:0/468219399 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff9b00186c0 0x7ff9b001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:50.986 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:50.991 7ff9ca61c700 1 --2- 172.21.2.112:0/468219399 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff9b00186c0 0x7ff9b001abc0 crc :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:51.014 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:51.022 7ff9d09be700 1 -- 172.21.2.112:0/468219399 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9cc0f0b50 con 0x7ff9cc09e920 2021-02-10T11:22:51.017 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:51.025 7ff9baffd700 1 -- 172.21.2.112:0/468219399 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff9c4001270 con 0x7ff9cc09e920 2021-02-10T11:22:51.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:51.151 7ff9d09be700 1 -- 172.21.2.112:0/468219399 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348617", "target": ["mgr", ""]}) v1 -- 0x7ff9cc0c3270 con 0x7ff9b00186c0 2021-02-10T11:22:51.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:51.176 7ff9baffd700 1 -- 172.21.2.112:0/468219399 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7ff9bc006430 con 0x7ff9b00186c0 2021-02-10T11:22:51.177 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:51.185 7ff9d09be700 1 -- 172.21.2.112:0/468219399 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff9b00186c0 msgr2=0x7ff9b001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:51.177 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:51.185 7ff9d09be700 1 --2- 172.21.2.112:0/468219399 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff9b00186c0 0x7ff9b001abc0 crc :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:51.177 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:51.185 7ff9d09be700 1 -- 172.21.2.112:0/468219399 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff9cc09e920 msgr2=0x7ff9cc0c2960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:51.178 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:51.185 7ff9d09be700 1 --2- 172.21.2.112:0/468219399 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff9cc09e920 0x7ff9cc0c2960 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 rx=0x7ff9c4007f90 tx=0x7ff9c400ac50).stop 2021-02-10T11:22:51.178 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:51.185 7ff9d09be700 1 -- 172.21.2.112:0/468219399 shutdown_connections 2021-02-10T11:22:51.178 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:51.185 7ff9d09be700 1 --2- 172.21.2.112:0/468219399 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff9b00186c0 0x7ff9b001abc0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:51.178 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:51.185 7ff9d09be700 1 --2- 172.21.2.112:0/468219399 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff9cc0c1f60 0x7ff9cc09e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:51.178 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:51.185 7ff9d09be700 1 --2- 172.21.2.112:0/468219399 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff9cc09e920 0x7ff9cc0c2960 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:51.178 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:51.185 7ff9d09be700 1 --2- 172.21.2.112:0/468219399 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff9cc09f140 0x7ff9cc0c1940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:51.178 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:51.185 7ff9d09be700 1 -- 172.21.2.112:0/468219399 >> 172.21.2.112:0/468219399 conn(0x7ff9cc09fc10 msgr2=0x7ff9cc0b3250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:51.178 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:51.185 7ff9d09be700 1 -- 172.21.2.112:0/468219399 shutdown_connections 2021-02-10T11:22:51.178 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:51.185 7ff9d09be700 1 -- 172.21.2.112:0/468219399 wait complete. 2021-02-10T11:22:51.186 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000348617 2021-02-10T11:22:51.618 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:51.608 7f33c52e7700 1 Processor -- start 2021-02-10T11:22:51.619 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:51.608 7f33c52e7700 1 -- start start 2021-02-10T11:22:51.619 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:51.609 7f33c52e7700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f33c01b0870 0x7f33c00f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:51.619 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:51.609 7f33c52e7700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33c01b0c70 con 0x7f33c00efe00 2021-02-10T11:22:51.619 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:51.609 7f33c52e7700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33c00f0b50 con 0x7f33c01b0470 2021-02-10T11:22:51.619 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:51.609 7f33c52e7700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33c00e5ea0 con 0x7f33c01b0870 2021-02-10T11:22:51.619 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:51.609 7f33bf87e700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f33c00efe00 0x7f33c00f0200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:44638/0 (socket says 172.21.2.112:44638) 2021-02-10T11:22:51.619 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:51.609 7f33bf87e700 1 -- 172.21.2.112:0/2876442162 learned_addr learned my addr 172.21.2.112:0/2876442162 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:22:51.619 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:51.609 7f33be07b700 1 -- 172.21.2.112:0/2876442162 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1484797652 0 0) 0x7f33b8000f90 con 0x7f33c00efe00 2021-02-10T11:22:51.619 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:51.609 7f33be07b700 1 -- 172.21.2.112:0/2876442162 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33ac002fe0 con 0x7f33c00efe00 2021-02-10T11:22:51.619 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:51.609 7f33be87c700 1 --2- 172.21.2.112:0/2876442162 >> v2:172.21.2.120:3300/0 conn(0x7f33c01b0870 0x7f33c00f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:51.620 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:51.610 7f33be07b700 1 -- 172.21.2.112:0/2876442162 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1588521707 0 0) 0x7f33b8001460 con 0x7f33c00efe00 2021-02-10T11:22:51.620 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:51.610 7f33be07b700 1 -- 172.21.2.112:0/2876442162 >> v2:172.21.2.120:3300/0 conn(0x7f33c01b0870 msgr2=0x7f33c00f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:51.620 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:51.610 7f33be07b700 1 --2- 172.21.2.112:0/2876442162 >> v2:172.21.2.120:3300/0 conn(0x7f33c01b0870 0x7f33c00f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:51.620 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:51.610 7f33be07b700 1 -- 172.21.2.112:0/2876442162 >> v1:172.21.2.120:6789/0 conn(0x7f33c01b0470 legacy=0x7f33c00ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:51.620 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:51.610 7f33be07b700 1 -- 172.21.2.112:0/2876442162 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33c00e52e0 con 0x7f33c00efe00 2021-02-10T11:22:51.620 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:51.610 7f33be07b700 1 -- 172.21.2.112:0/2876442162 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f33b8001980 con 0x7f33c00efe00 2021-02-10T11:22:51.620 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:51.610 7f33be07b700 1 -- 172.21.2.112:0/2876442162 >> v1:172.21.2.112:6789/0 conn(0x7f33c00efe00 legacy=0x7f33c00f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:51.620 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:51.610 7f33be07b700 1 --2- 172.21.2.112:0/2876442162 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f33ac003a70 0x7f33ac005ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:51.620 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:51.610 7f33be07b700 1 --2- 172.21.2.112:0/2876442162 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f33ac006780 0x7f33ac008ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:51.620 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:51.610 7f33be07b700 1 --2- 172.21.2.112:0/2876442162 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f33ac0095a0 0x7f33ac00bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:51.620 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:51.610 7f33be07b700 1 -- 172.21.2.112:0/2876442162 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33ac00c310 con 0x7f33ac003a70 2021-02-10T11:22:51.620 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:51.610 7f33be07b700 1 -- 172.21.2.112:0/2876442162 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33ac00c530 con 0x7f33ac0095a0 2021-02-10T11:22:51.620 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:51.610 7f33be07b700 1 -- 172.21.2.112:0/2876442162 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33ac00c750 con 0x7f33ac006780 2021-02-10T11:22:51.621 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:51.610 7f33bf07d700 1 --2- 172.21.2.112:0/2876442162 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f33ac003a70 0x7f33ac005ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:51.621 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:51.610 7f33bf87e700 1 --2- 172.21.2.112:0/2876442162 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f33ac0095a0 0x7f33ac00bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:51.621 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:51.610 7f33be87c700 1 --2- 172.21.2.112:0/2876442162 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f33ac006780 0x7f33ac008ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:51.621 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:51.610 7f33bf07d700 1 -- 172.21.2.112:0/2876442162 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f33ac006780 msgr2=0x7f33ac008ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:51.621 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:51.610 7f33bf07d700 1 --2- 172.21.2.112:0/2876442162 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f33ac006780 0x7f33ac008ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:51.621 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:51.610 7f33bf07d700 1 -- 172.21.2.112:0/2876442162 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f33ac0095a0 msgr2=0x7f33ac00bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:51.621 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:51.610 7f33bf07d700 1 --2- 172.21.2.112:0/2876442162 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f33ac0095a0 0x7f33ac00bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:51.621 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:51.610 7f33bf07d700 1 -- 172.21.2.112:0/2876442162 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33ac050990 con 0x7f33ac003a70 2021-02-10T11:22:51.621 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:51.610 7f33bf87e700 1 -- 172.21.2.112:0/2876442162 reap_dead start 2021-02-10T11:22:51.621 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:51.611 7f33bf07d700 1 --2- 172.21.2.112:0/2876442162 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f33ac003a70 0x7f33ac005ee0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 rx=0x7f33b0006ac0 tx=0x7f33b0009af0).ready entity=mon.0 client_cookie=89d0531d70c1f4b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:51.621 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:51.611 7f33be07b700 1 -- 172.21.2.112:0/2876442162 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f33b000c340 con 0x7f33ac003a70 2021-02-10T11:22:51.621 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:51.611 7f33be07b700 1 -- 172.21.2.112:0/2876442162 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33b000c6c0 con 0x7f33ac003a70 2021-02-10T11:22:51.621 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:51.611 7f33be07b700 1 -- 172.21.2.112:0/2876442162 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f33b000c0c0 con 0x7f33ac003a70 2021-02-10T11:22:51.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:51.611 7f33c52e7700 1 -- 172.21.2.112:0/2876442162 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f33ac003a70 msgr2=0x7f33ac005ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:51.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:51.611 7f33c52e7700 1 --2- 172.21.2.112:0/2876442162 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f33ac003a70 0x7f33ac005ee0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 rx=0x7f33b0006ac0 tx=0x7f33b0009af0).stop 2021-02-10T11:22:51.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:51.611 7f33c52e7700 1 -- 172.21.2.112:0/2876442162 shutdown_connections 2021-02-10T11:22:51.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:51.611 7f33c52e7700 1 --2- 172.21.2.112:0/2876442162 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f33ac003a70 0x7f33ac005ee0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:51.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:51.611 7f33c52e7700 1 -- 172.21.2.112:0/2876442162 >> 172.21.2.112:0/2876442162 conn(0x7f33c00ce3e0 msgr2=0x7f33c00e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:51.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:51.611 7f33c52e7700 1 -- 172.21.2.112:0/2876442162 shutdown_connections 2021-02-10T11:22:51.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:51.611 7f33c52e7700 1 -- 172.21.2.112:0/2876442162 wait complete. 2021-02-10T11:22:51.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:51.626 7f33c52e7700 1 Processor -- start 2021-02-10T11:22:51.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:51.626 7f33c52e7700 1 -- start start 2021-02-10T11:22:51.624 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:51.626 7f33c52e7700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f33c00b7db0 0x7f33c00b8260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:51.624 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:51.626 7f33c52e7700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f33c00b8880 0x7f33c01a7450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:51.624 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:51.626 7f33c52e7700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f33c00b8c80 0x7f33c01a7a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:51.624 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:51.626 7f33c52e7700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33c00b79b0 con 0x7f33c00b7db0 2021-02-10T11:22:51.624 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:51.626 7f33c52e7700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33c01a8090 con 0x7f33c00b8880 2021-02-10T11:22:51.624 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:51.626 7f33c52e7700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33c01a82b0 con 0x7f33c00b8c80 2021-02-10T11:22:51.624 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:51.627 7f33bf07d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f33c00b7db0 0x7f33c00b8260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:51.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:51.627 7f33bf07d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f33c00b7db0 0x7f33c00b8260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:45184/0 (socket says 172.21.2.112:45184) 2021-02-10T11:22:51.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:51.627 7f33bf07d700 1 -- 172.21.2.112:0/2275808752 learned_addr learned my addr 172.21.2.112:0/2275808752 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:22:51.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:51.627 7f33be87c700 1 --2- 172.21.2.112:0/2275808752 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f33c00b8880 0x7f33c01a7450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:51.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:51.627 7f33bf07d700 1 -- 172.21.2.112:0/2275808752 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f33c00b8c80 msgr2=0x7f33c01a7a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:51.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:51.627 7f33bf87e700 1 --2- 172.21.2.112:0/2275808752 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f33c00b8c80 0x7f33c01a7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:51.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:51.627 7f33bf07d700 1 --2- 172.21.2.112:0/2275808752 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f33c00b8c80 0x7f33c01a7a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:51.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:51.627 7f33bf07d700 1 -- 172.21.2.112:0/2275808752 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f33c00b8880 msgr2=0x7f33c01a7450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:51.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:51.627 7f33bf07d700 1 --2- 172.21.2.112:0/2275808752 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f33c00b8880 0x7f33c01a7450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:51.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:51.627 7f33bf07d700 1 -- 172.21.2.112:0/2275808752 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33c0190c10 con 0x7f33c00b7db0 2021-02-10T11:22:51.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:51.627 7f33be87c700 1 --2- 172.21.2.112:0/2275808752 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f33c00b8880 0x7f33c01a7450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:22:51.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:51.627 7f33bf87e700 1 --2- 172.21.2.112:0/2275808752 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f33c00b8c80 0x7f33c01a7a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:22:51.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:51.627 7f33bf07d700 1 --2- 172.21.2.112:0/2275808752 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f33c00b7db0 0x7f33c00b8260 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 rx=0x7f33b000be80 tx=0x7f33b000d8e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:51.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:51.627 7f33b7fff700 1 -- 172.21.2.112:0/2275808752 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f33b000b030 con 0x7f33c00b7db0 2021-02-10T11:22:51.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:51.628 7f33b7fff700 1 -- 172.21.2.112:0/2275808752 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33b000b330 con 0x7f33c00b7db0 2021-02-10T11:22:51.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:51.628 7f33b7fff700 1 -- 172.21.2.112:0/2275808752 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f33b00095f0 con 0x7f33c00b7db0 2021-02-10T11:22:51.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:51.628 7f33c52e7700 1 -- 172.21.2.112:0/2275808752 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33c0190c10 con 0x7f33c00b7db0 2021-02-10T11:22:51.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:51.628 7f33c52e7700 1 -- 172.21.2.112:0/2275808752 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33c00c1eb0 con 0x7f33c00b7db0 2021-02-10T11:22:51.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:51.628 7f33b7fff700 1 -- 172.21.2.112:0/2275808752 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f33b00095d0 con 0x7f33c00b7db0 2021-02-10T11:22:51.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:51.628 7f33b7fff700 1 --2- 172.21.2.112:0/2275808752 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f33a40186c0 0x7f33a401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:51.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:51.628 7f33b7fff700 1 -- 172.21.2.112:0/2275808752 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f33b0032550 con 0x7f33c00b7db0 2021-02-10T11:22:51.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:51.629 7f33be87c700 1 --2- 172.21.2.112:0/2275808752 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f33a40186c0 0x7f33a401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:51.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:51.629 7f33be87c700 1 --2- 172.21.2.112:0/2275808752 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f33a40186c0 0x7f33a401abc0 crc :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:51.651 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:51.659 7f33c52e7700 1 -- 172.21.2.112:0/2275808752 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33c0105b30 con 0x7f33c00b7db0 2021-02-10T11:22:51.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:51.661 7f33b7fff700 1 -- 172.21.2.112:0/2275808752 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f33b0009280 con 0x7f33c00b7db0 2021-02-10T11:22:51.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:51.788 7f33c52e7700 1 -- 172.21.2.112:0/2275808752 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348617", "target": ["mgr", ""]}) v1 -- 0x7f33c00c1eb0 con 0x7f33a40186c0 2021-02-10T11:22:51.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:51.792 7f33b7fff700 1 -- 172.21.2.112:0/2275808752 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f33a8006b10 con 0x7f33a40186c0 2021-02-10T11:22:51.788 INFO:teuthology.orchestra.run.gibba012.stdout:/volumes/_nogroup/subvolume_0000000000348617/0ffda9d2-c035-4153-8c34-e77a7ce3183a 2021-02-10T11:22:51.790 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:51.798 7f33c52e7700 1 -- 172.21.2.112:0/2275808752 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f33a40186c0 msgr2=0x7f33a401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:51.790 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:51.798 7f33c52e7700 1 --2- 172.21.2.112:0/2275808752 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f33a40186c0 0x7f33a401abc0 crc :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:51.790 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:51.798 7f33c52e7700 1 -- 172.21.2.112:0/2275808752 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f33c00b7db0 msgr2=0x7f33c00b8260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:51.790 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:51.798 7f33c52e7700 1 --2- 172.21.2.112:0/2275808752 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f33c00b7db0 0x7f33c00b8260 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 rx=0x7f33b000be80 tx=0x7f33b000d8e0).stop 2021-02-10T11:22:51.791 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:51.798 7f33c52e7700 1 -- 172.21.2.112:0/2275808752 shutdown_connections 2021-02-10T11:22:51.791 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:51.798 7f33c52e7700 1 --2- 172.21.2.112:0/2275808752 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f33a40186c0 0x7f33a401abc0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:51.791 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:51.798 7f33c52e7700 1 --2- 172.21.2.112:0/2275808752 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f33c00b8880 0x7f33c01a7450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:51.791 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:51.798 7f33c52e7700 1 --2- 172.21.2.112:0/2275808752 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f33c00b8c80 0x7f33c01a7a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:51.791 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:51.798 7f33c52e7700 1 --2- 172.21.2.112:0/2275808752 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f33c00b7db0 0x7f33c00b8260 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:51.791 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:51.798 7f33c52e7700 1 -- 172.21.2.112:0/2275808752 >> 172.21.2.112:0/2275808752 conn(0x7f33c0094e10 msgr2=0x7f33c00a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:51.791 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:51.798 7f33c52e7700 1 -- 172.21.2.112:0/2275808752 shutdown_connections 2021-02-10T11:22:51.791 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:51.798 7f33c52e7700 1 -- 172.21.2.112:0/2275808752 wait complete. 2021-02-10T11:22:51.798 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000348617 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000348617/0ffda9d2-c035-4153-8c34-e77a7ce3183a 2021-02-10T11:22:51.799 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:22:51.799 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:22:51.799 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:22:52.134 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:22:52.134 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:22:52.135 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348617/0ffda9d2-c035-4153-8c34-e77a7ce3183a/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:22:52.515 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:22:52.516 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:22:52.516 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0227313 s, 46.1 MB/s 2021-02-10T11:22:52.517 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:22:52.517 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:22:52.517 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:22:52.853 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:22:52.853 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:22:52.854 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348617/0ffda9d2-c035-4153-8c34-e77a7ce3183a/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:22:53.230 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:22:53.230 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:22:53.230 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0247484 s, 42.4 MB/s 2021-02-10T11:22:53.231 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:22:53.231 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:22:53.232 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:22:53.567 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:22:53.567 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:22:53.567 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348617/0ffda9d2-c035-4153-8c34-e77a7ce3183a/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:22:53.944 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:22:53.945 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:22:53.945 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0255476 s, 41.0 MB/s 2021-02-10T11:22:53.946 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:22:53.946 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:22:53.946 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:22:54.281 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:22:54.282 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:22:54.282 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348617/0ffda9d2-c035-4153-8c34-e77a7ce3183a/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:22:54.659 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:22:54.659 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:22:54.659 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0249906 s, 42.0 MB/s 2021-02-10T11:22:54.660 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:22:54.660 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:22:54.660 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:22:54.996 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:22:54.996 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:22:54.997 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348617/0ffda9d2-c035-4153-8c34-e77a7ce3183a/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:22:55.372 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:22:55.372 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:22:55.372 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0229527 s, 45.7 MB/s 2021-02-10T11:22:55.374 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:22:55.374 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:22:55.374 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:22:55.709 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:22:55.709 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:22:55.710 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348617/0ffda9d2-c035-4153-8c34-e77a7ce3183a/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:22:56.085 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:22:56.085 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:22:56.085 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0241682 s, 43.4 MB/s 2021-02-10T11:22:56.086 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:22:56.086 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:22:56.086 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:22:56.421 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:22:56.421 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:22:56.422 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348617/0ffda9d2-c035-4153-8c34-e77a7ce3183a/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:22:56.795 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:22:56.795 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:22:56.795 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0228688 s, 45.9 MB/s 2021-02-10T11:22:56.796 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:22:56.796 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:22:56.796 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:22:57.132 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:22:57.133 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:22:57.133 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348617/0ffda9d2-c035-4153-8c34-e77a7ce3183a/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:22:57.531 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:22:57.531 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:22:57.531 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0259374 s, 40.4 MB/s 2021-02-10T11:22:57.532 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:22:57.532 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:22:57.532 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:22:57.879 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:22:57.880 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:22:57.880 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348617/0ffda9d2-c035-4153-8c34-e77a7ce3183a/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:22:58.258 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:22:58.258 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:22:58.258 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0252112 s, 41.6 MB/s 2021-02-10T11:22:58.260 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:22:58.260 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:22:58.260 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:22:58.597 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:22:58.598 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:22:58.598 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348617/0ffda9d2-c035-4153-8c34-e77a7ce3183a/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:22:58.976 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:22:58.976 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:22:58.976 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0268322 s, 39.1 MB/s 2021-02-10T11:22:58.977 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000348618 2021-02-10T11:22:59.409 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:59.399 7f81e37de700 1 Processor -- start 2021-02-10T11:22:59.409 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:59.399 7f81e37de700 1 -- start start 2021-02-10T11:22:59.409 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:59.400 7f81e37de700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f81dc1b0470 0x7f81dc1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:59.409 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:59.400 7f81e37de700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f81dc0f20e0 0x7f81dc0f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:59.409 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:59.400 7f81e37de700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81dc0f0b50 con 0x7f81dc0ef610 2021-02-10T11:22:59.409 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:59.400 7f81e37de700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81dc0f0820 con 0x7f81dc1b0470 2021-02-10T11:22:59.409 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:59.400 7f81e37de700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81dc0f24e0 con 0x7f81dc0f20e0 2021-02-10T11:22:59.409 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:59.400 7f81e0df9700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f81dc0ef610 0x7f81dc0efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:44654/0 (socket says 172.21.2.112:44654) 2021-02-10T11:22:59.410 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:59.400 7f81e0df9700 1 -- 172.21.2.112:0/4066744188 learned_addr learned my addr 172.21.2.112:0/4066744188 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:22:59.410 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:59.400 7f81e15fa700 1 --2- 172.21.2.112:0/4066744188 >> v2:172.21.2.120:3300/0 conn(0x7f81dc1b0470 0x7f81dc1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:59.410 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:59.400 7f81e1dfb700 1 --2- 172.21.2.112:0/4066744188 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f81dc0f20e0 0x7f81dc0f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:59.410 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:59.400 7f81e15fa700 1 -- 172.21.2.112:0/4066744188 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f81dc0f20e0 msgr2=0x7f81dc0f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:59.410 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:59.400 7f81e15fa700 1 --2- 172.21.2.112:0/4066744188 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f81dc0f20e0 0x7f81dc0f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:59.410 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:59.400 7f81cffff700 1 -- 172.21.2.112:0/4066744188 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3061331330 0 0) 0x7f81d80009b0 con 0x7f81dc0ef610 2021-02-10T11:22:59.410 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:59.400 7f81e15fa700 1 -- 172.21.2.112:0/4066744188 >> v1:172.21.2.112:6789/0 conn(0x7f81dc0ef610 legacy=0x7f81dc0efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:59.410 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:59.400 7f81e15fa700 1 -- 172.21.2.112:0/4066744188 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81dc0e52e0 con 0x7f81dc1b0470 2021-02-10T11:22:59.410 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:59.401 7f81e15fa700 1 --2- 172.21.2.112:0/4066744188 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f81dc1b0470 0x7f81dc1b0870 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7f81d0006650 tx=0x7f81d0009a80).ready entity=mon.1 client_cookie=2b8858215a834e23 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:59.410 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:59.401 7f81cffff700 1 -- 172.21.2.112:0/4066744188 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f81d000c0c0 con 0x7f81dc1b0470 2021-02-10T11:22:59.410 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:59.401 7f81cffff700 1 -- 172.21.2.112:0/4066744188 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81d000c340 con 0x7f81dc1b0470 2021-02-10T11:22:59.410 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:59.401 7f81cffff700 1 -- 172.21.2.112:0/4066744188 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f81d000c860 con 0x7f81dc1b0470 2021-02-10T11:22:59.410 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:59.401 7f81e37de700 1 -- 172.21.2.112:0/4066744188 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f81dc1b0470 msgr2=0x7f81dc1b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:59.411 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:59.401 7f81e37de700 1 --2- 172.21.2.112:0/4066744188 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f81dc1b0470 0x7f81dc1b0870 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7f81d0006650 tx=0x7f81d0009a80).stop 2021-02-10T11:22:59.411 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:59.402 7f81e37de700 1 -- 172.21.2.112:0/4066744188 shutdown_connections 2021-02-10T11:22:59.411 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:59.402 7f81e37de700 1 --2- 172.21.2.112:0/4066744188 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f81dc0f20e0 0x7f81dc0f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:59.411 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:59.402 7f81e37de700 1 --2- 172.21.2.112:0/4066744188 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f81dc1b0470 0x7f81dc1b0870 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:59.411 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:59.402 7f81e37de700 1 -- 172.21.2.112:0/4066744188 >> 172.21.2.112:0/4066744188 conn(0x7f81dc0ce3e0 msgr2=0x7f81dc0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:59.411 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:59.402 7f81e37de700 1 -- 172.21.2.112:0/4066744188 shutdown_connections 2021-02-10T11:22:59.411 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:59.402 7f81e37de700 1 -- 172.21.2.112:0/4066744188 wait complete. 2021-02-10T11:22:59.413 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:59.417 7f81e37de700 1 Processor -- start 2021-02-10T11:22:59.413 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:59.417 7f81e37de700 1 -- start start 2021-02-10T11:22:59.413 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:59.417 7f81e37de700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f81dc105d10 0x7f81dc09eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:59.414 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:59.417 7f81e37de700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f81dc106110 0x7f81dc09f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:59.414 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:59.417 7f81e37de700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f81dc0b2a00 0x7f81dc0b2e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:59.414 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:59.417 7f81e37de700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81dc09e560 con 0x7f81dc106110 2021-02-10T11:22:59.414 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:59.417 7f81e37de700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81dc09c090 con 0x7f81dc105d10 2021-02-10T11:22:59.414 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:59.417 7f81e37de700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81dc0b3420 con 0x7f81dc0b2a00 2021-02-10T11:22:59.414 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:59.417 7f81e0df9700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f81dc106110 0x7f81dc09f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:59.414 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:59.417 7f81e0df9700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f81dc106110 0x7f81dc09f100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:45196/0 (socket says 172.21.2.112:45196) 2021-02-10T11:22:59.414 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:59.417 7f81e0df9700 1 -- 172.21.2.112:0/2939547826 learned_addr learned my addr 172.21.2.112:0/2939547826 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:22:59.415 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:59.417 7f81e1dfb700 1 --2- 172.21.2.112:0/2939547826 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f81dc0b2a00 0x7f81dc0b2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:59.415 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:59.417 7f81e15fa700 1 --2- 172.21.2.112:0/2939547826 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f81dc105d10 0x7f81dc09eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:59.415 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:59.418 7f81e0df9700 1 -- 172.21.2.112:0/2939547826 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f81dc0b2a00 msgr2=0x7f81dc0b2e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:59.415 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:59.418 7f81e0df9700 1 --2- 172.21.2.112:0/2939547826 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f81dc0b2a00 0x7f81dc0b2e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:59.415 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:59.418 7f81e0df9700 1 -- 172.21.2.112:0/2939547826 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f81dc105d10 msgr2=0x7f81dc09eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:59.415 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:59.418 7f81e0df9700 1 --2- 172.21.2.112:0/2939547826 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f81dc105d10 0x7f81dc09eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:59.415 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:59.418 7f81e0df9700 1 -- 172.21.2.112:0/2939547826 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81dc0b3640 con 0x7f81dc106110 2021-02-10T11:22:59.415 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:59.418 7f81e1dfb700 1 --2- 172.21.2.112:0/2939547826 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f81dc0b2a00 0x7f81dc0b2e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:22:59.415 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:59.418 7f81e0df9700 1 --2- 172.21.2.112:0/2939547826 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f81dc106110 0x7f81dc09f100 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 rx=0x7f81d8001c60 tx=0x7f81d800a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:59.415 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:59.418 7f81cdffb700 1 -- 172.21.2.112:0/2939547826 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f81d800acc0 con 0x7f81dc106110 2021-02-10T11:22:59.415 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:59.418 7f81cdffb700 1 -- 172.21.2.112:0/2939547826 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81d8004140 con 0x7f81dc106110 2021-02-10T11:22:59.416 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:59.418 7f81cdffb700 1 -- 172.21.2.112:0/2939547826 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f81d8004590 con 0x7f81dc106110 2021-02-10T11:22:59.416 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:59.418 7f81e37de700 1 -- 172.21.2.112:0/2939547826 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81dc0b3640 con 0x7f81dc106110 2021-02-10T11:22:59.416 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:59.418 7f81e37de700 1 -- 172.21.2.112:0/2939547826 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81dc0b3bd0 con 0x7f81dc106110 2021-02-10T11:22:59.416 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:59.419 7f81cdffb700 1 -- 172.21.2.112:0/2939547826 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f81d8004320 con 0x7f81dc106110 2021-02-10T11:22:59.416 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:59.419 7f81cdffb700 1 --2- 172.21.2.112:0/2939547826 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f81bc0186d0 0x7f81bc01abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:59.416 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:59.419 7f81cdffb700 1 -- 172.21.2.112:0/2939547826 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f81d80325e0 con 0x7f81dc106110 2021-02-10T11:22:59.416 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:59.419 7f81e15fa700 1 --2- 172.21.2.112:0/2939547826 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f81bc0186d0 0x7f81bc01abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:59.416 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:59.419 7f81e15fa700 1 --2- 172.21.2.112:0/2939547826 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f81bc0186d0 0x7f81bc01abd0 crc :-1 s=READY pgs=275 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:59.443 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:59.449 7f81e37de700 1 -- 172.21.2.112:0/2939547826 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81dc105ae0 con 0x7f81dc106110 2021-02-10T11:22:59.444 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:59.452 7f81cdffb700 1 -- 172.21.2.112:0/2939547826 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f81d8000dd0 con 0x7f81dc106110 2021-02-10T11:22:59.570 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:59.577 7f81e37de700 1 -- 172.21.2.112:0/2939547826 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348618", "target": ["mgr", ""]}) v1 -- 0x7f81dc09c090 con 0x7f81bc0186d0 2021-02-10T11:22:59.594 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:59.602 7f81cdffb700 1 -- 172.21.2.112:0/2939547826 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f81d0006420 con 0x7f81bc0186d0 2021-02-10T11:22:59.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:59.611 7f81e37de700 1 -- 172.21.2.112:0/2939547826 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f81bc0186d0 msgr2=0x7f81bc01abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:59.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:59.611 7f81e37de700 1 --2- 172.21.2.112:0/2939547826 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f81bc0186d0 0x7f81bc01abd0 crc :-1 s=READY pgs=275 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:59.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:59.611 7f81e37de700 1 -- 172.21.2.112:0/2939547826 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f81dc106110 msgr2=0x7f81dc09f100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:59.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:59.611 7f81e37de700 1 --2- 172.21.2.112:0/2939547826 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f81dc106110 0x7f81dc09f100 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 rx=0x7f81d8001c60 tx=0x7f81d800a8d0).stop 2021-02-10T11:22:59.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:59.611 7f81e37de700 1 -- 172.21.2.112:0/2939547826 shutdown_connections 2021-02-10T11:22:59.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:59.611 7f81e37de700 1 --2- 172.21.2.112:0/2939547826 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f81bc0186d0 0x7f81bc01abd0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:59.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:59.611 7f81e37de700 1 --2- 172.21.2.112:0/2939547826 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f81dc0b2a00 0x7f81dc0b2e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:59.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:59.611 7f81e37de700 1 --2- 172.21.2.112:0/2939547826 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f81dc106110 0x7f81dc09f100 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:59.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:59.611 7f81e37de700 1 --2- 172.21.2.112:0/2939547826 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f81dc105d10 0x7f81dc09eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:59.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:59.611 7f81e37de700 1 -- 172.21.2.112:0/2939547826 >> 172.21.2.112:0/2939547826 conn(0x7f81dc0c36f0 msgr2=0x7f81dc0bd640 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:59.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:59.611 7f81e37de700 1 -- 172.21.2.112:0/2939547826 shutdown_connections 2021-02-10T11:22:59.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:22:59.611 7f81e37de700 1 -- 172.21.2.112:0/2939547826 wait complete. 2021-02-10T11:22:59.612 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000348618 2021-02-10T11:23:00.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:00.034 7fd9eef20700 1 Processor -- start 2021-02-10T11:23:00.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:00.034 7fd9eef20700 1 -- start start 2021-02-10T11:23:00.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:00.034 7fd9eef20700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fd9e81b0870 0x7fd9e80f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:00.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:00.035 7fd9eef20700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd9e80efe00 0x7fd9e80f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:00.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:00.035 7fd9eef20700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9e81b0c70 con 0x7fd9e81b0470 2021-02-10T11:23:00.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:00.035 7fd9eef20700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9e80f0b50 con 0x7fd9e81b0870 2021-02-10T11:23:00.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:00.035 7fd9eef20700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9e80f0820 con 0x7fd9e80efe00 2021-02-10T11:23:00.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:00.035 7fd9ecd3c700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fd9e81b0470 0x7fd9e80ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:44666/0 (socket says 172.21.2.112:44666) 2021-02-10T11:23:00.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:00.035 7fd9ecd3c700 1 -- 172.21.2.112:0/1646963709 learned_addr learned my addr 172.21.2.112:0/1646963709 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:23:00.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:00.035 7fd9e7fff700 1 --2- 172.21.2.112:0/1646963709 >> v2:172.21.2.120:3300/0 conn(0x7fd9e81b0870 0x7fd9e80f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:00.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:00.035 7fd9ed53d700 1 --2- 172.21.2.112:0/1646963709 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd9e80efe00 0x7fd9e80f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:00.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:00.035 7fd9e77fe700 1 -- 172.21.2.112:0/1646963709 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1818820974 0 0) 0x7fd9d0000ce0 con 0x7fd9e81b0470 2021-02-10T11:23:00.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:00.035 7fd9e77fe700 1 -- 172.21.2.112:0/1646963709 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9d4002fe0 con 0x7fd9e81b0470 2021-02-10T11:23:00.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:00.035 7fd9e7fff700 1 -- 172.21.2.112:0/1646963709 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd9e80efe00 msgr2=0x7fd9e80f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:00.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:00.035 7fd9e7fff700 1 --2- 172.21.2.112:0/1646963709 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd9e80efe00 0x7fd9e80f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:00.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:00.035 7fd9e7fff700 1 -- 172.21.2.112:0/1646963709 >> v1:172.21.2.112:6789/0 conn(0x7fd9e81b0470 legacy=0x7fd9e80ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:00.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:00.035 7fd9e7fff700 1 -- 172.21.2.112:0/1646963709 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9e80e52e0 con 0x7fd9e81b0870 2021-02-10T11:23:00.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:00.035 7fd9e7fff700 1 --2- 172.21.2.112:0/1646963709 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd9e81b0870 0x7fd9e80f20e0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7fd9d80069d0 tx=0x7fd9d8009dc0).ready entity=mon.1 client_cookie=83283cf380d50e5 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:00.045 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:00.036 7fd9e77fe700 1 -- 172.21.2.112:0/1646963709 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd9d800b510 con 0x7fd9e81b0870 2021-02-10T11:23:00.045 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:00.036 7fd9e77fe700 1 -- 172.21.2.112:0/1646963709 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9d8006ac0 con 0x7fd9e81b0870 2021-02-10T11:23:00.045 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:00.036 7fd9e77fe700 1 -- 172.21.2.112:0/1646963709 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd9d800ba80 con 0x7fd9e81b0870 2021-02-10T11:23:00.045 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:00.036 7fd9eef20700 1 -- 172.21.2.112:0/1646963709 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd9e81b0870 msgr2=0x7fd9e80f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:00.045 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:00.036 7fd9eef20700 1 --2- 172.21.2.112:0/1646963709 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd9e81b0870 0x7fd9e80f20e0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7fd9d80069d0 tx=0x7fd9d8009dc0).stop 2021-02-10T11:23:00.045 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:00.036 7fd9eef20700 1 -- 172.21.2.112:0/1646963709 shutdown_connections 2021-02-10T11:23:00.045 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:00.036 7fd9eef20700 1 --2- 172.21.2.112:0/1646963709 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd9e80efe00 0x7fd9e80f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:00.045 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:00.036 7fd9eef20700 1 --2- 172.21.2.112:0/1646963709 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd9e81b0870 0x7fd9e80f20e0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:00.045 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:00.036 7fd9eef20700 1 -- 172.21.2.112:0/1646963709 >> 172.21.2.112:0/1646963709 conn(0x7fd9e80ce3e0 msgr2=0x7fd9e80e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:00.045 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:00.037 7fd9eef20700 1 -- 172.21.2.112:0/1646963709 shutdown_connections 2021-02-10T11:23:00.045 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:00.037 7fd9eef20700 1 -- 172.21.2.112:0/1646963709 wait complete. 2021-02-10T11:23:00.047 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:00.051 7fd9eef20700 1 Processor -- start 2021-02-10T11:23:00.047 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:00.051 7fd9eef20700 1 -- start start 2021-02-10T11:23:00.047 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:00.051 7fd9eef20700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd9e81a42e0 0x7fd9e809ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:00.047 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:00.051 7fd9eef20700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd9e81a46e0 0x7fd9e809f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:00.047 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:00.051 7fd9eef20700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd9e808ad30 0x7fd9e808b130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:00.047 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:00.051 7fd9eef20700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9e808a7b0 con 0x7fd9e81a46e0 2021-02-10T11:23:00.047 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:00.051 7fd9eef20700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9e80a65f0 con 0x7fd9e808ad30 2021-02-10T11:23:00.048 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:00.052 7fd9eef20700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9e808a430 con 0x7fd9e81a42e0 2021-02-10T11:23:00.048 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:00.052 7fd9e7fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd9e81a46e0 0x7fd9e809f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:00.048 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:00.052 7fd9e7fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd9e81a46e0 0x7fd9e809f310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:45210/0 (socket says 172.21.2.112:45210) 2021-02-10T11:23:00.048 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:00.052 7fd9e7fff700 1 -- 172.21.2.112:0/1077156697 learned_addr learned my addr 172.21.2.112:0/1077156697 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:23:00.048 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:00.052 7fd9e7fff700 1 -- 172.21.2.112:0/1077156697 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd9e81a42e0 msgr2=0x7fd9e809ecf0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:23:00.048 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:00.052 7fd9e7fff700 1 --2- 172.21.2.112:0/1077156697 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd9e81a42e0 0x7fd9e809ecf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:00.048 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:00.052 7fd9e7fff700 1 -- 172.21.2.112:0/1077156697 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd9e808ad30 msgr2=0x7fd9e808b130 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:23:00.048 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:00.052 7fd9e7fff700 1 --2- 172.21.2.112:0/1077156697 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd9e808ad30 0x7fd9e808b130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:00.048 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:00.052 7fd9e7fff700 1 -- 172.21.2.112:0/1077156697 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9e809fa50 con 0x7fd9e81a46e0 2021-02-10T11:23:00.048 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:00.052 7fd9e7fff700 1 --2- 172.21.2.112:0/1077156697 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd9e81a46e0 0x7fd9e809f310 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 rx=0x7fd9d80099e0 tx=0x7fd9d800bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:00.048 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:00.052 7fd9e57fa700 1 -- 172.21.2.112:0/1077156697 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd9d8006030 con 0x7fd9e81a46e0 2021-02-10T11:23:00.048 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:00.052 7fd9e57fa700 1 -- 172.21.2.112:0/1077156697 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9d8006270 con 0x7fd9e81a46e0 2021-02-10T11:23:00.049 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:00.052 7fd9e57fa700 1 -- 172.21.2.112:0/1077156697 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd9d8005660 con 0x7fd9e81a46e0 2021-02-10T11:23:00.049 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:00.052 7fd9eef20700 1 -- 172.21.2.112:0/1077156697 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9e809fa50 con 0x7fd9e81a46e0 2021-02-10T11:23:00.049 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:00.052 7fd9eef20700 1 -- 172.21.2.112:0/1077156697 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9e809fa50 con 0x7fd9e81a46e0 2021-02-10T11:23:00.049 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:00.053 7fd9e57fa700 1 -- 172.21.2.112:0/1077156697 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd9d8006c00 con 0x7fd9e81a46e0 2021-02-10T11:23:00.049 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:00.053 7fd9e57fa700 1 --2- 172.21.2.112:0/1077156697 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd9cc0186c0 0x7fd9cc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:00.049 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:00.053 7fd9e57fa700 1 -- 172.21.2.112:0/1077156697 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fd9d80328e0 con 0x7fd9e81a46e0 2021-02-10T11:23:00.049 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:00.053 7fd9ecd3c700 1 --2- 172.21.2.112:0/1077156697 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd9cc0186c0 0x7fd9cc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:00.049 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:00.054 7fd9ecd3c700 1 --2- 172.21.2.112:0/1077156697 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd9cc0186c0 0x7fd9cc01abc0 crc :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:00.076 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:00.084 7fd9eef20700 1 -- 172.21.2.112:0/1077156697 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9e8105ae0 con 0x7fd9e81a46e0 2021-02-10T11:23:00.079 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:00.087 7fd9e57fa700 1 -- 172.21.2.112:0/1077156697 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd9d802cf90 con 0x7fd9e81a46e0 2021-02-10T11:23:00.207 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:00.214 7fd9eef20700 1 -- 172.21.2.112:0/1077156697 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348618", "target": ["mgr", ""]}) v1 -- 0x7fd9e80a65f0 con 0x7fd9cc0186c0 2021-02-10T11:23:00.210 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:00.218 7fd9e57fa700 1 -- 172.21.2.112:0/1077156697 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7fd9d00076f0 con 0x7fd9cc0186c0 2021-02-10T11:23:00.214 INFO:teuthology.orchestra.run.gibba012.stdout:/volumes/_nogroup/subvolume_0000000000348618/d5421c2b-7064-4824-975f-8d2b761bc2d9 2021-02-10T11:23:00.216 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:00.224 7fd9eef20700 1 -- 172.21.2.112:0/1077156697 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd9cc0186c0 msgr2=0x7fd9cc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:00.216 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:00.224 7fd9eef20700 1 --2- 172.21.2.112:0/1077156697 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd9cc0186c0 0x7fd9cc01abc0 crc :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:00.216 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:00.224 7fd9eef20700 1 -- 172.21.2.112:0/1077156697 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd9e81a46e0 msgr2=0x7fd9e809f310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:00.216 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:00.224 7fd9eef20700 1 --2- 172.21.2.112:0/1077156697 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd9e81a46e0 0x7fd9e809f310 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 rx=0x7fd9d80099e0 tx=0x7fd9d800bcd0).stop 2021-02-10T11:23:00.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:00.224 7fd9eef20700 1 -- 172.21.2.112:0/1077156697 shutdown_connections 2021-02-10T11:23:00.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:00.224 7fd9eef20700 1 --2- 172.21.2.112:0/1077156697 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd9cc0186c0 0x7fd9cc01abc0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:00.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:00.224 7fd9eef20700 1 --2- 172.21.2.112:0/1077156697 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd9e808ad30 0x7fd9e808b130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:00.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:00.224 7fd9eef20700 1 --2- 172.21.2.112:0/1077156697 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd9e81a46e0 0x7fd9e809f310 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:00.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:00.224 7fd9eef20700 1 --2- 172.21.2.112:0/1077156697 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd9e81a42e0 0x7fd9e809ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:00.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:00.224 7fd9eef20700 1 -- 172.21.2.112:0/1077156697 >> 172.21.2.112:0/1077156697 conn(0x7fd9e8095a70 msgr2=0x7fd9e80bf050 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:00.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:00.224 7fd9eef20700 1 -- 172.21.2.112:0/1077156697 shutdown_connections 2021-02-10T11:23:00.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:00.224 7fd9eef20700 1 -- 172.21.2.112:0/1077156697 wait complete. 2021-02-10T11:23:00.224 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000348618 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000348618/d5421c2b-7064-4824-975f-8d2b761bc2d9 2021-02-10T11:23:00.224 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:00.225 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:23:00.225 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:23:00.561 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:23:00.561 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:23:00.562 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348618/d5421c2b-7064-4824-975f-8d2b761bc2d9/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:23:00.943 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:23:00.943 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:23:00.943 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0249405 s, 42.0 MB/s 2021-02-10T11:23:00.944 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:00.944 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:23:00.944 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:23:01.280 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:23:01.280 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:23:01.281 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348618/d5421c2b-7064-4824-975f-8d2b761bc2d9/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:23:01.655 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:23:01.656 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:23:01.656 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0247433 s, 42.4 MB/s 2021-02-10T11:23:01.657 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:01.657 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:23:01.657 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:23:01.992 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:23:01.992 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:23:01.993 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348618/d5421c2b-7064-4824-975f-8d2b761bc2d9/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:23:02.367 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:23:02.367 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:23:02.367 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0231632 s, 45.3 MB/s 2021-02-10T11:23:02.368 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:02.368 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:23:02.368 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:23:02.703 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:23:02.704 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:23:02.704 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348618/d5421c2b-7064-4824-975f-8d2b761bc2d9/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:23:03.078 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:23:03.079 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:23:03.079 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0239293 s, 43.8 MB/s 2021-02-10T11:23:03.080 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:03.080 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:23:03.080 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:23:03.415 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:23:03.415 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:23:03.416 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348618/d5421c2b-7064-4824-975f-8d2b761bc2d9/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:23:03.790 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:23:03.790 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:23:03.790 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0229694 s, 45.7 MB/s 2021-02-10T11:23:03.791 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:03.791 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:23:03.792 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:23:04.126 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:23:04.126 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:23:04.127 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348618/d5421c2b-7064-4824-975f-8d2b761bc2d9/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:23:04.497 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:23:04.497 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:23:04.497 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0223183 s, 47.0 MB/s 2021-02-10T11:23:04.498 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:04.498 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:23:04.498 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:23:04.834 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:23:04.834 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:23:04.834 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348618/d5421c2b-7064-4824-975f-8d2b761bc2d9/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:23:05.208 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:23:05.209 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:23:05.209 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0225173 s, 46.6 MB/s 2021-02-10T11:23:05.210 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:05.210 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:23:05.210 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:23:05.544 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:23:05.544 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:23:05.545 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348618/d5421c2b-7064-4824-975f-8d2b761bc2d9/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:23:05.922 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:23:05.922 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:23:05.923 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0254912 s, 41.1 MB/s 2021-02-10T11:23:05.923 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:05.924 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:23:05.924 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:23:06.259 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:23:06.259 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:23:06.260 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348618/d5421c2b-7064-4824-975f-8d2b761bc2d9/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:23:06.637 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:23:06.637 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:23:06.637 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0251163 s, 41.7 MB/s 2021-02-10T11:23:06.638 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:06.638 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:23:06.638 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:23:06.973 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:23:06.974 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:23:06.974 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348618/d5421c2b-7064-4824-975f-8d2b761bc2d9/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:23:07.351 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:23:07.351 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:23:07.351 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0247995 s, 42.3 MB/s 2021-02-10T11:23:07.352 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000348619 2021-02-10T11:23:07.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:07.774 7fa215973700 1 Processor -- start 2021-02-10T11:23:07.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:07.774 7fa215973700 1 -- start start 2021-02-10T11:23:07.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:07.774 7fa215973700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa2100f05f0 0x7fa2100f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:07.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:07.774 7fa215973700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2101b0c70 con 0x7fa2101b0470 2021-02-10T11:23:07.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:07.774 7fa215973700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2100f0b50 con 0x7fa2101b0870 2021-02-10T11:23:07.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:07.774 7fa215973700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2100e5ea0 con 0x7fa2100f05f0 2021-02-10T11:23:07.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:07.774 7fa20f574700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fa2101b0470 0x7fa2100ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:44680/0 (socket says 172.21.2.112:44680) 2021-02-10T11:23:07.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:07.774 7fa20f574700 1 -- 172.21.2.112:0/727755955 learned_addr learned my addr 172.21.2.112:0/727755955 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:23:07.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:07.774 7fa20fd75700 1 --2- 172.21.2.112:0/727755955 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa2100f05f0 0x7fa2100f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:07.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:07.775 7fa20e572700 1 -- 172.21.2.112:0/727755955 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4046245457 0 0) 0x7fa1f80010b0 con 0x7fa2101b0470 2021-02-10T11:23:07.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:07.775 7fa20e572700 1 -- 172.21.2.112:0/727755955 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1fc002fe0 con 0x7fa2101b0470 2021-02-10T11:23:07.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:07.775 7fa20e572700 1 -- 172.21.2.112:0/727755955 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3315165146 0 0) 0x7fa1f80010b0 con 0x7fa2101b0470 2021-02-10T11:23:07.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:07.775 7fa20e572700 1 -- 172.21.2.112:0/727755955 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa2100f05f0 msgr2=0x7fa2100f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:07.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:07.775 7fa20e572700 1 --2- 172.21.2.112:0/727755955 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa2100f05f0 0x7fa2100f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:07.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:07.775 7fa20e572700 1 -- 172.21.2.112:0/727755955 >> v1:172.21.2.120:6789/0 conn(0x7fa2101b0870 legacy=0x7fa2100efe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:07.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:07.775 7fa20e572700 1 -- 172.21.2.112:0/727755955 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2100e52e0 con 0x7fa2101b0470 2021-02-10T11:23:07.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:07.775 7fa20e572700 1 -- 172.21.2.112:0/727755955 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7fa1f80015b0 con 0x7fa2101b0470 2021-02-10T11:23:07.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:07.775 7fa20e572700 1 -- 172.21.2.112:0/727755955 >> v1:172.21.2.112:6789/0 conn(0x7fa2101b0470 legacy=0x7fa2100ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:07.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:07.775 7fa20e572700 1 --2- 172.21.2.112:0/727755955 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa1fc003b20 0x7fa1fc005f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:07.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:07.775 7fa20e572700 1 --2- 172.21.2.112:0/727755955 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa1fc006830 0x7fa1fc008d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:07.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:07.775 7fa20e572700 1 --2- 172.21.2.112:0/727755955 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa1fc009630 0x7fa1fc00bb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:07.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:07.775 7fa20e572700 1 -- 172.21.2.112:0/727755955 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1fc00c370 con 0x7fa1fc006830 2021-02-10T11:23:07.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:07.775 7fa20e572700 1 -- 172.21.2.112:0/727755955 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1fc00c590 con 0x7fa1fc003b20 2021-02-10T11:23:07.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:07.775 7fa20e572700 1 -- 172.21.2.112:0/727755955 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1fc00c7b0 con 0x7fa1fc009630 2021-02-10T11:23:07.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:07.775 7fa20ed73700 1 --2- 172.21.2.112:0/727755955 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa1fc006830 0x7fa1fc008d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:07.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:07.775 7fa20fd75700 1 --2- 172.21.2.112:0/727755955 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa2100f05f0 0x7fa2100f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:23:07.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:07.775 7fa20f574700 1 --2- 172.21.2.112:0/727755955 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa1fc003b20 0x7fa1fc005f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:07.786 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:07.775 7fa20fd75700 1 --2- 172.21.2.112:0/727755955 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa1fc009630 0x7fa1fc00bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:07.786 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:07.775 7fa20ed73700 1 -- 172.21.2.112:0/727755955 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa1fc009630 msgr2=0x7fa1fc00bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:07.786 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:07.775 7fa20ed73700 1 --2- 172.21.2.112:0/727755955 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa1fc009630 0x7fa1fc00bb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:07.786 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:07.775 7fa20ed73700 1 -- 172.21.2.112:0/727755955 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa1fc003b20 msgr2=0x7fa1fc005f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:07.786 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:07.775 7fa20ed73700 1 --2- 172.21.2.112:0/727755955 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa1fc003b20 0x7fa1fc005f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:07.786 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:07.775 7fa20ed73700 1 -- 172.21.2.112:0/727755955 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1fc050a20 con 0x7fa1fc006830 2021-02-10T11:23:07.786 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:07.775 7fa20ed73700 1 --2- 172.21.2.112:0/727755955 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa1fc006830 0x7fa1fc008d30 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 rx=0x7fa200000b80 tx=0x7fa200009a10).ready entity=mon.0 client_cookie=db8289db404a5c0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:07.786 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:07.776 7fa20e572700 1 -- 172.21.2.112:0/727755955 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa200007280 con 0x7fa1fc006830 2021-02-10T11:23:07.786 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:07.776 7fa20e572700 1 -- 172.21.2.112:0/727755955 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa200007580 con 0x7fa1fc006830 2021-02-10T11:23:07.786 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:07.776 7fa20e572700 1 -- 172.21.2.112:0/727755955 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa2000079d0 con 0x7fa1fc006830 2021-02-10T11:23:07.786 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:07.776 7fa215973700 1 -- 172.21.2.112:0/727755955 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa1fc006830 msgr2=0x7fa1fc008d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:07.787 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:07.776 7fa215973700 1 --2- 172.21.2.112:0/727755955 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa1fc006830 0x7fa1fc008d30 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 rx=0x7fa200000b80 tx=0x7fa200009a10).stop 2021-02-10T11:23:07.787 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:07.776 7fa20fd75700 1 -- 172.21.2.112:0/727755955 reap_dead start 2021-02-10T11:23:07.787 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:07.776 7fa215973700 1 -- 172.21.2.112:0/727755955 shutdown_connections 2021-02-10T11:23:07.787 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:07.776 7fa215973700 1 -- 172.21.2.112:0/727755955 >> 172.21.2.112:0/727755955 conn(0x7fa2100ce3e0 msgr2=0x7fa2100e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:07.787 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:07.776 7fa215973700 1 -- 172.21.2.112:0/727755955 shutdown_connections 2021-02-10T11:23:07.787 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:07.776 7fa215973700 1 -- 172.21.2.112:0/727755955 wait complete. 2021-02-10T11:23:07.787 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:07.791 7fa215973700 1 Processor -- start 2021-02-10T11:23:07.787 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:07.791 7fa215973700 1 -- start start 2021-02-10T11:23:07.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:07.791 7fa215973700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa21009b930 0x7fa2100b7c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:07.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:07.791 7fa215973700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa2100b8410 0x7fa2100b8810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:07.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:07.791 7fa215973700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa2100c1f80 0x7fa2100c2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:07.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:07.791 7fa215973700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2100c1a00 con 0x7fa2100c1f80 2021-02-10T11:23:07.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:07.791 7fa215973700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2100b8e30 con 0x7fa2100b8410 2021-02-10T11:23:07.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:07.791 7fa215973700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2100b9050 con 0x7fa21009b930 2021-02-10T11:23:07.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:07.792 7fa20fd75700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa2100c1f80 0x7fa2100c2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:07.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:07.792 7fa20fd75700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa2100c1f80 0x7fa2100c2380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:45234/0 (socket says 172.21.2.112:45234) 2021-02-10T11:23:07.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:07.792 7fa20fd75700 1 -- 172.21.2.112:0/598406192 learned_addr learned my addr 172.21.2.112:0/598406192 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:23:07.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:07.792 7fa20f574700 1 --2- 172.21.2.112:0/598406192 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa21009b930 0x7fa2100b7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:07.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:07.792 7fa20ed73700 1 --2- 172.21.2.112:0/598406192 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa2100b8410 0x7fa2100b8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:07.790 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:07.792 7fa20fd75700 1 -- 172.21.2.112:0/598406192 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa21009b930 msgr2=0x7fa2100b7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:07.790 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:07.792 7fa20fd75700 1 --2- 172.21.2.112:0/598406192 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa21009b930 0x7fa2100b7c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:07.790 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:07.792 7fa20fd75700 1 -- 172.21.2.112:0/598406192 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa2100b8410 msgr2=0x7fa2100b8810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:07.790 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:07.792 7fa20fd75700 1 --2- 172.21.2.112:0/598406192 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa2100b8410 0x7fa2100b8810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:07.790 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:07.792 7fa20fd75700 1 -- 172.21.2.112:0/598406192 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2100c29a0 con 0x7fa2100c1f80 2021-02-10T11:23:07.790 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:07.792 7fa20fd75700 1 --2- 172.21.2.112:0/598406192 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa2100c1f80 0x7fa2100c2380 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 rx=0x7fa2080099c0 tx=0x7fa20800bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:07.790 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:07.792 7fa207fff700 1 -- 172.21.2.112:0/598406192 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa2080011f0 con 0x7fa2100c1f80 2021-02-10T11:23:07.790 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:07.792 7fa207fff700 1 -- 172.21.2.112:0/598406192 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa208003030 con 0x7fa2100c1f80 2021-02-10T11:23:07.790 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:07.792 7fa207fff700 1 -- 172.21.2.112:0/598406192 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa2080017b0 con 0x7fa2100c1f80 2021-02-10T11:23:07.790 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:07.792 7fa215973700 1 -- 172.21.2.112:0/598406192 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2100c29a0 con 0x7fa2100c1f80 2021-02-10T11:23:07.790 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:07.792 7fa215973700 1 -- 172.21.2.112:0/598406192 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2100c29a0 con 0x7fa2100c1f80 2021-02-10T11:23:07.790 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:07.793 7fa207fff700 1 -- 172.21.2.112:0/598406192 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa208006a90 con 0x7fa2100c1f80 2021-02-10T11:23:07.790 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:07.793 7fa207fff700 1 --2- 172.21.2.112:0/598406192 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa1f4018730 0x7fa1f401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:07.791 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:07.793 7fa207fff700 1 -- 172.21.2.112:0/598406192 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fa2080329c0 con 0x7fa2100c1f80 2021-02-10T11:23:07.791 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:07.793 7fa20f574700 1 --2- 172.21.2.112:0/598406192 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa1f4018730 0x7fa1f401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:07.791 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:07.793 7fa20f574700 1 --2- 172.21.2.112:0/598406192 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa1f4018730 0x7fa1f401ac10 crc :-1 s=READY pgs=277 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:07.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:07.824 7fa215973700 1 -- 172.21.2.112:0/598406192 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa210105b30 con 0x7fa2100c1f80 2021-02-10T11:23:07.819 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:07.827 7fa207fff700 1 -- 172.21.2.112:0/598406192 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa20800be30 con 0x7fa2100c1f80 2021-02-10T11:23:07.948 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:07.955 7fa215973700 1 -- 172.21.2.112:0/598406192 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348619", "target": ["mgr", ""]}) v1 -- 0x7fa210105b30 con 0x7fa1f4018730 2021-02-10T11:23:07.962 DEBUG:teuthology.orchestra.run.gibba012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:23:07.965 DEBUG:teuthology.orchestra.run.gibba020:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:23:07.972 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:07.980 7fa207fff700 1 -- 172.21.2.112:0/598406192 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fa1f8000d40 con 0x7fa1f4018730 2021-02-10T11:23:07.981 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:07.989 7fa215973700 1 -- 172.21.2.112:0/598406192 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa1f4018730 msgr2=0x7fa1f401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:07.981 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:07.989 7fa215973700 1 --2- 172.21.2.112:0/598406192 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa1f4018730 0x7fa1f401ac10 crc :-1 s=READY pgs=277 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:07.982 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:07.989 7fa215973700 1 -- 172.21.2.112:0/598406192 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa2100c1f80 msgr2=0x7fa2100c2380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:07.982 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:07.989 7fa215973700 1 --2- 172.21.2.112:0/598406192 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa2100c1f80 0x7fa2100c2380 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 rx=0x7fa2080099c0 tx=0x7fa20800bcf0).stop 2021-02-10T11:23:07.982 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:07.989 7fa215973700 1 -- 172.21.2.112:0/598406192 shutdown_connections 2021-02-10T11:23:07.982 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:07.989 7fa215973700 1 --2- 172.21.2.112:0/598406192 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa1f4018730 0x7fa1f401ac10 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:07.982 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:07.989 7fa215973700 1 --2- 172.21.2.112:0/598406192 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa2100b8410 0x7fa2100b8810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:07.982 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:07.989 7fa215973700 1 --2- 172.21.2.112:0/598406192 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa2100c1f80 0x7fa2100c2380 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:07.982 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:07.989 7fa215973700 1 --2- 172.21.2.112:0/598406192 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa21009b930 0x7fa2100b7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:07.982 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:07.989 7fa215973700 1 -- 172.21.2.112:0/598406192 >> 172.21.2.112:0/598406192 conn(0x7fa210094e10 msgr2=0x7fa2100a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:07.982 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:07.989 7fa215973700 1 -- 172.21.2.112:0/598406192 shutdown_connections 2021-02-10T11:23:07.982 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:07.989 7fa215973700 1 -- 172.21.2.112:0/598406192 wait complete. 2021-02-10T11:23:07.989 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000348619 2021-02-10T11:23:08.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:08.387 7f2f99050700 1 Processor -- start 2021-02-10T11:23:08.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:08.387 7f2f99050700 1 -- start start 2021-02-10T11:23:08.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:08.387 7f2f99050700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2f941b0470 0x7f2f941b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:08.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:08.387 7f2f99050700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f940f0b50 con 0x7f2f940f20e0 2021-02-10T11:23:08.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:08.387 7f2f99050700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f940f24e0 con 0x7f2f940ef610 2021-02-10T11:23:08.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:08.387 7f2f99050700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f940e5ea0 con 0x7f2f941b0470 2021-02-10T11:23:08.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:08.388 7f2f9359e700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f2f940f20e0 0x7f2f940f0200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:44700/0 (socket says 172.21.2.112:44700) 2021-02-10T11:23:08.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:08.388 7f2f9359e700 1 -- 172.21.2.112:0/877088842 learned_addr learned my addr 172.21.2.112:0/877088842 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:23:08.398 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:08.388 7f2f9259c700 1 -- 172.21.2.112:0/877088842 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3435620704 0 0) 0x7f2f8c001500 con 0x7f2f940f20e0 2021-02-10T11:23:08.398 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:08.388 7f2f9259c700 1 -- 172.21.2.112:0/877088842 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f80002fe0 con 0x7f2f940f20e0 2021-02-10T11:23:08.398 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:08.388 7f2f9259c700 1 -- 172.21.2.112:0/877088842 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1181754499 0 0) 0x7f2f8c001460 con 0x7f2f940f20e0 2021-02-10T11:23:08.398 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:08.388 7f2f9259c700 1 -- 172.21.2.112:0/877088842 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2f941b0470 msgr2=0x7f2f941b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:08.398 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:08.388 7f2f9259c700 1 --2- 172.21.2.112:0/877088842 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2f941b0470 0x7f2f941b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:08.398 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:08.388 7f2f9259c700 1 -- 172.21.2.112:0/877088842 >> v1:172.21.2.120:6789/0 conn(0x7f2f940ef610 legacy=0x7f2f940efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:08.398 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:08.388 7f2f9259c700 1 -- 172.21.2.112:0/877088842 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f940e52e0 con 0x7f2f940f20e0 2021-02-10T11:23:08.398 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:08.388 7f2f9259c700 1 -- 172.21.2.112:0/877088842 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f2f8c001980 con 0x7f2f940f20e0 2021-02-10T11:23:08.398 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:08.388 7f2f9259c700 1 -- 172.21.2.112:0/877088842 >> v1:172.21.2.112:6789/0 conn(0x7f2f940f20e0 legacy=0x7f2f940f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:08.398 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:08.388 7f2f9259c700 1 --2- 172.21.2.112:0/877088842 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2f80003a70 0x7f2f80005ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:08.398 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:08.388 7f2f9259c700 1 --2- 172.21.2.112:0/877088842 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2f80006760 0x7f2f80008c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:08.398 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:08.388 7f2f9259c700 1 --2- 172.21.2.112:0/877088842 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2f80009500 0x7f2f8000ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:08.398 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:08.388 7f2f9259c700 1 -- 172.21.2.112:0/877088842 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f8000c2d0 con 0x7f2f80006760 2021-02-10T11:23:08.399 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:08.388 7f2f9259c700 1 -- 172.21.2.112:0/877088842 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f8000c4f0 con 0x7f2f80009500 2021-02-10T11:23:08.399 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:08.388 7f2f9259c700 1 -- 172.21.2.112:0/877088842 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f8000c710 con 0x7f2f80003a70 2021-02-10T11:23:08.399 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:08.388 7f2f9359e700 1 --2- 172.21.2.112:0/877088842 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2f80009500 0x7f2f8000ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:08.399 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:08.388 7f2f8a59c700 1 --2- 172.21.2.112:0/877088842 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2f80006760 0x7f2f80008c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:08.399 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:08.388 7f2f92d9d700 1 --2- 172.21.2.112:0/877088842 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2f80003a70 0x7f2f80005ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:08.399 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:08.389 7f2f9359e700 1 -- 172.21.2.112:0/877088842 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2f80003a70 msgr2=0x7f2f80005ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:08.399 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:08.389 7f2f9359e700 1 --2- 172.21.2.112:0/877088842 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2f80003a70 0x7f2f80005ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:08.399 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:08.389 7f2f9359e700 1 -- 172.21.2.112:0/877088842 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2f80006760 msgr2=0x7f2f80008c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:08.399 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:08.389 7f2f9359e700 1 --2- 172.21.2.112:0/877088842 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2f80006760 0x7f2f80008c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:08.399 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:08.389 7f2f9359e700 1 -- 172.21.2.112:0/877088842 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f8000c9b0 con 0x7f2f80009500 2021-02-10T11:23:08.399 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:08.389 7f2f9359e700 1 --2- 172.21.2.112:0/877088842 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2f80009500 0x7f2f8000ba20 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f2f8c0078b0 tx=0x7f2f8c04e320).ready entity=mon.1 client_cookie=d1c2db6d589fddce server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:08.399 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:08.389 7f2f9259c700 1 -- 172.21.2.112:0/877088842 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2f8c007d40 con 0x7f2f80009500 2021-02-10T11:23:08.399 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:08.389 7f2f9259c700 1 -- 172.21.2.112:0/877088842 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f8c0070d0 con 0x7f2f80009500 2021-02-10T11:23:08.400 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:08.389 7f2f9259c700 1 -- 172.21.2.112:0/877088842 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2f8c0075f0 con 0x7f2f80009500 2021-02-10T11:23:08.400 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:08.389 7f2f99050700 1 -- 172.21.2.112:0/877088842 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2f80009500 msgr2=0x7f2f8000ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:08.400 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:08.389 7f2f99050700 1 --2- 172.21.2.112:0/877088842 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2f80009500 0x7f2f8000ba20 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f2f8c0078b0 tx=0x7f2f8c04e320).stop 2021-02-10T11:23:08.400 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:08.389 7f2f9359e700 1 -- 172.21.2.112:0/877088842 reap_dead start 2021-02-10T11:23:08.400 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:08.390 7f2f99050700 1 -- 172.21.2.112:0/877088842 shutdown_connections 2021-02-10T11:23:08.400 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:08.390 7f2f99050700 1 -- 172.21.2.112:0/877088842 >> 172.21.2.112:0/877088842 conn(0x7f2f940ce3e0 msgr2=0x7f2f940e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:08.400 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:08.390 7f2f99050700 1 -- 172.21.2.112:0/877088842 shutdown_connections 2021-02-10T11:23:08.400 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:08.390 7f2f99050700 1 -- 172.21.2.112:0/877088842 wait complete. 2021-02-10T11:23:08.400 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:08.404 7f2f99050700 1 Processor -- start 2021-02-10T11:23:08.400 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:08.404 7f2f99050700 1 -- start start 2021-02-10T11:23:08.402 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:08.405 7f2f99050700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2f940b7db0 0x7f2f940b8260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:08.402 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:08.405 7f2f99050700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2f940b8880 0x7f2f941a7450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:08.402 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:08.405 7f2f99050700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2f940b8c80 0x7f2f941a7a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:08.402 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:08.405 7f2f99050700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f940b79b0 con 0x7f2f940b8c80 2021-02-10T11:23:08.402 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:08.405 7f2f99050700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f941a8090 con 0x7f2f940b8880 2021-02-10T11:23:08.402 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:08.405 7f2f99050700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f941a82b0 con 0x7f2f940b7db0 2021-02-10T11:23:08.402 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:08.405 7f2f9359e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2f940b8c80 0x7f2f941a7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:08.402 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:08.405 7f2f9359e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2f940b8c80 0x7f2f941a7a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:45250/0 (socket says 172.21.2.112:45250) 2021-02-10T11:23:08.402 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:08.405 7f2f9359e700 1 -- 172.21.2.112:0/3262986857 learned_addr learned my addr 172.21.2.112:0/3262986857 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:23:08.403 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:08.405 7f2f8a59c700 1 --2- 172.21.2.112:0/3262986857 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2f940b8880 0x7f2f941a7450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:08.403 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:08.405 7f2f92d9d700 1 --2- 172.21.2.112:0/3262986857 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2f940b7db0 0x7f2f940b8260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:08.403 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:08.406 7f2f9359e700 1 -- 172.21.2.112:0/3262986857 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2f940b7db0 msgr2=0x7f2f940b8260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:08.403 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:08.406 7f2f9359e700 1 --2- 172.21.2.112:0/3262986857 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2f940b7db0 0x7f2f940b8260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:08.403 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:08.406 7f2f9359e700 1 -- 172.21.2.112:0/3262986857 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2f940b8880 msgr2=0x7f2f941a7450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:08.403 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:08.406 7f2f9359e700 1 --2- 172.21.2.112:0/3262986857 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2f940b8880 0x7f2f941a7450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:08.403 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:08.406 7f2f9359e700 1 -- 172.21.2.112:0/3262986857 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f94190c10 con 0x7f2f940b8c80 2021-02-10T11:23:08.403 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:08.406 7f2f8a59c700 1 --2- 172.21.2.112:0/3262986857 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2f940b8880 0x7f2f941a7450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:23:08.403 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:08.406 7f2f9359e700 1 --2- 172.21.2.112:0/3262986857 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2f940b8c80 0x7f2f941a7a70 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 rx=0x7f2f8c001be0 tx=0x7f2f8c04fca0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:08.403 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:08.406 7f2f8bfff700 1 -- 172.21.2.112:0/3262986857 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2f8c007d20 con 0x7f2f940b8c80 2021-02-10T11:23:08.403 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:08.406 7f2f8bfff700 1 -- 172.21.2.112:0/3262986857 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f8c001950 con 0x7f2f940b8c80 2021-02-10T11:23:08.403 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:08.406 7f2f8bfff700 1 -- 172.21.2.112:0/3262986857 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2f8c0072f0 con 0x7f2f940b8c80 2021-02-10T11:23:08.403 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:08.406 7f2f99050700 1 -- 172.21.2.112:0/3262986857 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f94190c10 con 0x7f2f940b8c80 2021-02-10T11:23:08.404 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:08.406 7f2f99050700 1 -- 172.21.2.112:0/3262986857 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f940c1eb0 con 0x7f2f940b8c80 2021-02-10T11:23:08.404 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:08.406 7f2f8bfff700 1 -- 172.21.2.112:0/3262986857 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2f8c04fde0 con 0x7f2f940b8c80 2021-02-10T11:23:08.404 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:08.407 7f2f8bfff700 1 --2- 172.21.2.112:0/3262986857 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2f780186c0 0x7f2f7801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:08.404 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:08.407 7f2f8bfff700 1 -- 172.21.2.112:0/3262986857 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f2f8c076af0 con 0x7f2f940b8c80 2021-02-10T11:23:08.404 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:08.407 7f2f92d9d700 1 --2- 172.21.2.112:0/3262986857 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2f780186c0 0x7f2f7801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:08.404 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:08.407 7f2f92d9d700 1 --2- 172.21.2.112:0/3262986857 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2f780186c0 0x7f2f7801abc0 crc :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:08.430 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:08.438 7f2f99050700 1 -- 172.21.2.112:0/3262986857 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f94105b30 con 0x7f2f940b8c80 2021-02-10T11:23:08.433 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:08.440 7f2f8bfff700 1 -- 172.21.2.112:0/3262986857 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2f8c001950 con 0x7f2f940b8c80 2021-02-10T11:23:08.563 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:08.570 7f2f99050700 1 -- 172.21.2.112:0/3262986857 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348619", "target": ["mgr", ""]}) v1 -- 0x7f2f940c1eb0 con 0x7f2f780186c0 2021-02-10T11:23:08.566 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:08.574 7f2f8bfff700 1 -- 172.21.2.112:0/3262986857 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f2f84000940 con 0x7f2f780186c0 2021-02-10T11:23:08.570 INFO:teuthology.orchestra.run.gibba012.stdout:/volumes/_nogroup/subvolume_0000000000348619/035bbbba-21c9-40ac-87c9-db7ea19f57f2 2021-02-10T11:23:08.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:08.579 7f2f99050700 1 -- 172.21.2.112:0/3262986857 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2f780186c0 msgr2=0x7f2f7801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:08.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:08.579 7f2f99050700 1 --2- 172.21.2.112:0/3262986857 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2f780186c0 0x7f2f7801abc0 crc :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:08.572 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:08.579 7f2f99050700 1 -- 172.21.2.112:0/3262986857 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2f940b8c80 msgr2=0x7f2f941a7a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:08.572 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:08.579 7f2f99050700 1 --2- 172.21.2.112:0/3262986857 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2f940b8c80 0x7f2f941a7a70 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 rx=0x7f2f8c001be0 tx=0x7f2f8c04fca0).stop 2021-02-10T11:23:08.572 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:08.579 7f2f99050700 1 -- 172.21.2.112:0/3262986857 shutdown_connections 2021-02-10T11:23:08.572 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:08.579 7f2f99050700 1 --2- 172.21.2.112:0/3262986857 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2f780186c0 0x7f2f7801abc0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:08.572 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:08.579 7f2f99050700 1 --2- 172.21.2.112:0/3262986857 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2f940b8880 0x7f2f941a7450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:08.572 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:08.579 7f2f99050700 1 --2- 172.21.2.112:0/3262986857 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2f940b8c80 0x7f2f941a7a70 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:08.572 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:08.579 7f2f99050700 1 --2- 172.21.2.112:0/3262986857 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2f940b7db0 0x7f2f940b8260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:08.572 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:08.579 7f2f99050700 1 -- 172.21.2.112:0/3262986857 >> 172.21.2.112:0/3262986857 conn(0x7f2f94094e10 msgr2=0x7f2f940a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:08.573 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:08.579 7f2f99050700 1 -- 172.21.2.112:0/3262986857 shutdown_connections 2021-02-10T11:23:08.573 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:08.580 7f2f99050700 1 -- 172.21.2.112:0/3262986857 wait complete. 2021-02-10T11:23:08.580 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000348619 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000348619/035bbbba-21c9-40ac-87c9-db7ea19f57f2 2021-02-10T11:23:08.580 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:08.580 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:23:08.580 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:23:08.917 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:23:08.917 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:23:08.918 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348619/035bbbba-21c9-40ac-87c9-db7ea19f57f2/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:23:09.300 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:23:09.300 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:23:09.300 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0255158 s, 41.1 MB/s 2021-02-10T11:23:09.301 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:09.301 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:23:09.301 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:23:09.637 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:23:09.637 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:23:09.638 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348619/035bbbba-21c9-40ac-87c9-db7ea19f57f2/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:23:10.009 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:23:10.010 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:23:10.010 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0240346 s, 43.6 MB/s 2021-02-10T11:23:10.011 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:10.011 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:23:10.011 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:23:10.346 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:23:10.346 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:23:10.347 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348619/035bbbba-21c9-40ac-87c9-db7ea19f57f2/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:23:10.720 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:23:10.721 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:23:10.721 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0232741 s, 45.1 MB/s 2021-02-10T11:23:10.722 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:10.722 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:23:10.722 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:23:11.056 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:23:11.056 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:23:11.057 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348619/035bbbba-21c9-40ac-87c9-db7ea19f57f2/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:23:11.433 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:23:11.433 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:23:11.433 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0241888 s, 43.3 MB/s 2021-02-10T11:23:11.434 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:11.434 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:23:11.434 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:23:11.769 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:23:11.769 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:23:11.769 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348619/035bbbba-21c9-40ac-87c9-db7ea19f57f2/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:23:12.145 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:23:12.145 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:23:12.145 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0249545 s, 42.0 MB/s 2021-02-10T11:23:12.146 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:12.146 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:23:12.146 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:23:12.481 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:23:12.482 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:23:12.482 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348619/035bbbba-21c9-40ac-87c9-db7ea19f57f2/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:23:12.857 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:23:12.858 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:23:12.858 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0234964 s, 44.6 MB/s 2021-02-10T11:23:12.859 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:12.859 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:23:12.859 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:23:13.193 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:23:13.193 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:23:13.194 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348619/035bbbba-21c9-40ac-87c9-db7ea19f57f2/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:23:13.567 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:23:13.568 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:23:13.568 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0231766 s, 45.2 MB/s 2021-02-10T11:23:13.569 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:13.569 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:23:13.569 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:23:13.905 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:23:13.905 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:23:13.906 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348619/035bbbba-21c9-40ac-87c9-db7ea19f57f2/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:23:14.282 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:23:14.282 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:23:14.282 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0248181 s, 42.3 MB/s 2021-02-10T11:23:14.283 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:14.283 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:23:14.284 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:23:14.621 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:23:14.621 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:23:14.622 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348619/035bbbba-21c9-40ac-87c9-db7ea19f57f2/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:23:14.993 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:23:14.994 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:23:14.994 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0244593 s, 42.9 MB/s 2021-02-10T11:23:14.995 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:14.995 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:23:14.995 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:23:15.332 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:23:15.332 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:23:15.332 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348619/035bbbba-21c9-40ac-87c9-db7ea19f57f2/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:23:15.708 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:23:15.709 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:23:15.709 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0238207 s, 44.0 MB/s 2021-02-10T11:23:15.710 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000348620 2021-02-10T11:23:16.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.131 7f73eb897700 1 Processor -- start 2021-02-10T11:23:16.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.132 7f73eb897700 1 -- start start 2021-02-10T11:23:16.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.132 7f73eb897700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f73e41b0470 0x7f73e41b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:16.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.132 7f73eb897700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f73e40ef610 0x7f73e40efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:16.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.132 7f73eb897700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73e40f0b50 con 0x7f73e40f0030 2021-02-10T11:23:16.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.132 7f73eb897700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73e40e52e0 con 0x7f73e40ef610 2021-02-10T11:23:16.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.132 7f73eb897700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73e40e5490 con 0x7f73e41b0470 2021-02-10T11:23:16.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.132 7f73e8eb2700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f73e40ef610 0x7f73e40efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:16.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.133 7f73e8eb2700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f73e40ef610 0x7f73e40efa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:45258/0 (socket says 172.21.2.112:45258) 2021-02-10T11:23:16.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.133 7f73e8eb2700 1 -- 172.21.2.112:0/291264325 learned_addr learned my addr 172.21.2.112:0/291264325 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:23:16.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.133 7f73e9eb4700 1 --1- 172.21.2.112:0/291264325 >> v1:172.21.2.120:6789/0 conn(0x7f73e40f0030 0x7f73e40f10d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:38278/0 (socket says 172.21.2.112:38278) 2021-02-10T11:23:16.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.133 7f73e96b3700 1 --2- 172.21.2.112:0/291264325 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f73e41b0470 0x7f73e41b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:16.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.133 7f73e8eb2700 1 -- 172.21.2.112:0/291264325 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f73e41b0470 msgr2=0x7f73e41b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:16.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.133 7f73e8eb2700 1 --2- 172.21.2.112:0/291264325 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f73e41b0470 0x7f73e41b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:16.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.133 7f73e8eb2700 1 -- 172.21.2.112:0/291264325 >> v1:172.21.2.120:6789/0 conn(0x7f73e40f0030 legacy=0x7f73e40f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:16.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.133 7f73e8eb2700 1 -- 172.21.2.112:0/291264325 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73e40f0430 con 0x7f73e40ef610 2021-02-10T11:23:16.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.133 7f73e8eb2700 1 --2- 172.21.2.112:0/291264325 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f73e40ef610 0x7f73e40efa10 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 rx=0x7f73d4006700 tx=0x7f73d4009b80).ready entity=mon.0 client_cookie=164df306d5e52d8e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:16.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.134 7f73dbfff700 1 -- 172.21.2.112:0/291264325 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f73d400c0c0 con 0x7f73e40ef610 2021-02-10T11:23:16.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.134 7f73dbfff700 1 -- 172.21.2.112:0/291264325 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73d400c0c0 con 0x7f73e40ef610 2021-02-10T11:23:16.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.134 7f73dbfff700 1 -- 172.21.2.112:0/291264325 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f73d400c440 con 0x7f73e40ef610 2021-02-10T11:23:16.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.134 7f73eb897700 1 -- 172.21.2.112:0/291264325 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f73e40ef610 msgr2=0x7f73e40efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:16.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.134 7f73eb897700 1 --2- 172.21.2.112:0/291264325 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f73e40ef610 0x7f73e40efa10 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 rx=0x7f73d4006700 tx=0x7f73d4009b80).stop 2021-02-10T11:23:16.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.135 7f73eb897700 1 -- 172.21.2.112:0/291264325 shutdown_connections 2021-02-10T11:23:16.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.135 7f73eb897700 1 --2- 172.21.2.112:0/291264325 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f73e40ef610 0x7f73e40efa10 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:16.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.135 7f73eb897700 1 --2- 172.21.2.112:0/291264325 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f73e41b0470 0x7f73e41b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:16.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.135 7f73eb897700 1 -- 172.21.2.112:0/291264325 >> 172.21.2.112:0/291264325 conn(0x7f73e40ce3e0 msgr2=0x7f73e40e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:16.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.135 7f73eb897700 1 -- 172.21.2.112:0/291264325 shutdown_connections 2021-02-10T11:23:16.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.135 7f73eb897700 1 -- 172.21.2.112:0/291264325 wait complete. 2021-02-10T11:23:16.145 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.149 7f73eb897700 1 Processor -- start 2021-02-10T11:23:16.145 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.150 7f73eb897700 1 -- start start 2021-02-10T11:23:16.146 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.150 7f73eb897700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f73e4105d10 0x7f73e409eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:16.146 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.150 7f73eb897700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f73e4106110 0x7f73e409f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:16.146 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.150 7f73eb897700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f73e40b2a00 0x7f73e40b2e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:16.146 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.150 7f73eb897700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73e409e560 con 0x7f73e4105d10 2021-02-10T11:23:16.146 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.150 7f73eb897700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73e409c090 con 0x7f73e40b2a00 2021-02-10T11:23:16.146 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.150 7f73eb897700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73e40b3420 con 0x7f73e4106110 2021-02-10T11:23:16.146 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.150 7f73e96b3700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f73e4105d10 0x7f73e409eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:16.146 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.150 7f73e8eb2700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f73e4106110 0x7f73e409f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:16.146 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.150 7f73e9eb4700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f73e40b2a00 0x7f73e40b2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:16.146 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.150 7f73e96b3700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f73e4105d10 0x7f73e409eae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:45264/0 (socket says 172.21.2.112:45264) 2021-02-10T11:23:16.147 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.150 7f73e96b3700 1 -- 172.21.2.112:0/719613129 learned_addr learned my addr 172.21.2.112:0/719613129 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:23:16.147 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.151 7f73e96b3700 1 -- 172.21.2.112:0/719613129 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f73e4106110 msgr2=0x7f73e409f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:16.147 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.151 7f73e96b3700 1 --2- 172.21.2.112:0/719613129 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f73e4106110 0x7f73e409f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:16.147 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.151 7f73e96b3700 1 -- 172.21.2.112:0/719613129 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f73e40b2a00 msgr2=0x7f73e40b2e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:16.147 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.151 7f73e96b3700 1 --2- 172.21.2.112:0/719613129 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f73e40b2a00 0x7f73e40b2e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:16.147 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.151 7f73e96b3700 1 -- 172.21.2.112:0/719613129 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73e40b3640 con 0x7f73e4105d10 2021-02-10T11:23:16.147 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.151 7f73e9eb4700 1 --2- 172.21.2.112:0/719613129 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f73e40b2a00 0x7f73e40b2e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:23:16.147 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.151 7f73e96b3700 1 --2- 172.21.2.112:0/719613129 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f73e4105d10 0x7f73e409eae0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 rx=0x7f73e0006f60 tx=0x7f73e000b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:16.147 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.151 7f73d9ffb700 1 -- 172.21.2.112:0/719613129 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f73e000bd60 con 0x7f73e4105d10 2021-02-10T11:23:16.147 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.151 7f73d9ffb700 1 -- 172.21.2.112:0/719613129 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73e0000c90 con 0x7f73e4105d10 2021-02-10T11:23:16.147 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.151 7f73d9ffb700 1 -- 172.21.2.112:0/719613129 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f73e0002220 con 0x7f73e4105d10 2021-02-10T11:23:16.147 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.151 7f73eb897700 1 -- 172.21.2.112:0/719613129 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73e40b3640 con 0x7f73e4105d10 2021-02-10T11:23:16.147 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.151 7f73eb897700 1 -- 172.21.2.112:0/719613129 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73e40b3640 con 0x7f73e4105d10 2021-02-10T11:23:16.148 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.152 7f73d9ffb700 1 -- 172.21.2.112:0/719613129 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f73e00052e0 con 0x7f73e4105d10 2021-02-10T11:23:16.148 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.152 7f73d9ffb700 1 --2- 172.21.2.112:0/719613129 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f73c40186c0 0x7f73c401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:16.148 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.152 7f73d9ffb700 1 -- 172.21.2.112:0/719613129 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f73e00325a0 con 0x7f73e4105d10 2021-02-10T11:23:16.148 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.152 7f73e8eb2700 1 --2- 172.21.2.112:0/719613129 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f73c40186c0 0x7f73c401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:16.148 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.153 7f73e8eb2700 1 --2- 172.21.2.112:0/719613129 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f73c40186c0 0x7f73c401abc0 crc :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:16.173 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.181 7f73eb897700 1 -- 172.21.2.112:0/719613129 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73e40b3ec0 con 0x7f73e4105d10 2021-02-10T11:23:16.176 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.184 7f73d9ffb700 1 -- 172.21.2.112:0/719613129 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f73e0030d50 con 0x7f73e4105d10 2021-02-10T11:23:16.303 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.311 7f73eb897700 1 -- 172.21.2.112:0/719613129 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348620", "target": ["mgr", ""]}) v1 -- 0x7f73e41a16c0 con 0x7f73c40186c0 2021-02-10T11:23:16.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.336 7f73d9ffb700 1 -- 172.21.2.112:0/719613129 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f73d4006430 con 0x7f73c40186c0 2021-02-10T11:23:16.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.344 7f73eb897700 1 -- 172.21.2.112:0/719613129 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f73c40186c0 msgr2=0x7f73c401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:16.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.344 7f73eb897700 1 --2- 172.21.2.112:0/719613129 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f73c40186c0 0x7f73c401abc0 crc :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:16.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.344 7f73eb897700 1 -- 172.21.2.112:0/719613129 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f73e4105d10 msgr2=0x7f73e409eae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:16.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.344 7f73eb897700 1 --2- 172.21.2.112:0/719613129 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f73e4105d10 0x7f73e409eae0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 rx=0x7f73e0006f60 tx=0x7f73e000b970).stop 2021-02-10T11:23:16.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.345 7f73eb897700 1 -- 172.21.2.112:0/719613129 shutdown_connections 2021-02-10T11:23:16.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.345 7f73eb897700 1 --2- 172.21.2.112:0/719613129 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f73c40186c0 0x7f73c401abc0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:16.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.345 7f73eb897700 1 --2- 172.21.2.112:0/719613129 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f73e40b2a00 0x7f73e40b2e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:16.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.345 7f73eb897700 1 --2- 172.21.2.112:0/719613129 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f73e4106110 0x7f73e409f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:16.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.345 7f73eb897700 1 --2- 172.21.2.112:0/719613129 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f73e4105d10 0x7f73e409eae0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:16.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.345 7f73eb897700 1 -- 172.21.2.112:0/719613129 >> 172.21.2.112:0/719613129 conn(0x7f73e40c36f0 msgr2=0x7f73e40bd640 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:16.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.345 7f73eb897700 1 -- 172.21.2.112:0/719613129 shutdown_connections 2021-02-10T11:23:16.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.345 7f73eb897700 1 -- 172.21.2.112:0/719613129 wait complete. 2021-02-10T11:23:16.345 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000348620 2021-02-10T11:23:16.777 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.768 7f691b19e700 1 Processor -- start 2021-02-10T11:23:16.777 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.768 7f691b19e700 1 -- start start 2021-02-10T11:23:16.777 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.768 7f691b19e700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f69140f05f0 0x7f69140f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:16.777 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.768 7f691b19e700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69141b0c70 con 0x7f69141b0470 2021-02-10T11:23:16.777 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.768 7f691b19e700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69140f0b50 con 0x7f69141b0870 2021-02-10T11:23:16.778 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.768 7f691b19e700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69140e5ea0 con 0x7f69140f05f0 2021-02-10T11:23:16.778 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.768 7f6918fba700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f69141b0470 0x7f69140ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:44732/0 (socket says 172.21.2.112:44732) 2021-02-10T11:23:16.778 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.768 7f6918fba700 1 -- 172.21.2.112:0/3991051839 learned_addr learned my addr 172.21.2.112:0/3991051839 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:23:16.778 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.769 7f69197bb700 1 --2- 172.21.2.112:0/3991051839 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f69140f05f0 0x7f69140f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:16.778 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.769 7f690b7fe700 1 -- 172.21.2.112:0/3991051839 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 721371357 0 0) 0x7f6904000ce0 con 0x7f69141b0470 2021-02-10T11:23:16.778 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.769 7f690b7fe700 1 -- 172.21.2.112:0/3991051839 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68fc002fe0 con 0x7f69141b0470 2021-02-10T11:23:16.778 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.769 7f690b7fe700 1 -- 172.21.2.112:0/3991051839 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1920576272 0 0) 0x7f6910000c60 con 0x7f69141b0870 2021-02-10T11:23:16.778 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.769 7f690b7fe700 1 -- 172.21.2.112:0/3991051839 --> v1:172.21.2.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68fc004370 con 0x7f69141b0870 2021-02-10T11:23:16.778 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.769 7f69197bb700 1 -- 172.21.2.112:0/3991051839 >> v1:172.21.2.120:6789/0 conn(0x7f69141b0870 legacy=0x7f69140efe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:16.778 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.769 7f69197bb700 1 -- 172.21.2.112:0/3991051839 >> v1:172.21.2.112:6789/0 conn(0x7f69141b0470 legacy=0x7f69140ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:16.778 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.769 7f69197bb700 1 -- 172.21.2.112:0/3991051839 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69140e52e0 con 0x7f69140f05f0 2021-02-10T11:23:16.778 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.769 7f69197bb700 1 --2- 172.21.2.112:0/3991051839 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f69140f05f0 0x7f69140f20e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f690c006e50 tx=0x7f690c009e70).ready entity=mon.2 client_cookie=b7df08aec7ecd962 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:16.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.770 7f690b7fe700 1 -- 172.21.2.112:0/3991051839 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f690c006a20 con 0x7f69140f05f0 2021-02-10T11:23:16.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.770 7f690b7fe700 1 -- 172.21.2.112:0/3991051839 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f690c006790 con 0x7f69140f05f0 2021-02-10T11:23:16.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.770 7f690b7fe700 1 -- 172.21.2.112:0/3991051839 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f690c00b700 con 0x7f69140f05f0 2021-02-10T11:23:16.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.770 7f691b19e700 1 -- 172.21.2.112:0/3991051839 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f69140f05f0 msgr2=0x7f69140f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:16.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.770 7f691b19e700 1 --2- 172.21.2.112:0/3991051839 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f69140f05f0 0x7f69140f20e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f690c006e50 tx=0x7f690c009e70).stop 2021-02-10T11:23:16.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.771 7f691b19e700 1 -- 172.21.2.112:0/3991051839 shutdown_connections 2021-02-10T11:23:16.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.771 7f691b19e700 1 --2- 172.21.2.112:0/3991051839 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f69140f05f0 0x7f69140f20e0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:16.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.771 7f691b19e700 1 -- 172.21.2.112:0/3991051839 >> 172.21.2.112:0/3991051839 conn(0x7f69140ce3e0 msgr2=0x7f69140e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:16.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.771 7f691b19e700 1 -- 172.21.2.112:0/3991051839 shutdown_connections 2021-02-10T11:23:16.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.771 7f691b19e700 1 -- 172.21.2.112:0/3991051839 wait complete. 2021-02-10T11:23:16.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.785 7f691b19e700 1 Processor -- start 2021-02-10T11:23:16.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.785 7f691b19e700 1 -- start start 2021-02-10T11:23:16.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.786 7f691b19e700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6914105d10 0x7f691409eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:16.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.786 7f691b19e700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6914106110 0x7f691409f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:16.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.786 7f691b19e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f69140b2a00 0x7f69140b2e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:16.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.786 7f691b19e700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f691409e560 con 0x7f69140b2a00 2021-02-10T11:23:16.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.786 7f691b19e700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f691409c090 con 0x7f6914106110 2021-02-10T11:23:16.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.786 7f691b19e700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69140b3420 con 0x7f6914105d10 2021-02-10T11:23:16.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.786 7f690bfff700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6914106110 0x7f691409f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:16.782 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.786 7f69197bb700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f69140b2a00 0x7f69140b2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:16.782 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.786 7f690bfff700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6914106110 0x7f691409f100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3300/0 says I am v2:172.21.2.112:47556/0 (socket says 172.21.2.112:47556) 2021-02-10T11:23:16.782 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.786 7f690bfff700 1 -- 172.21.2.112:0/3564462732 learned_addr learned my addr 172.21.2.112:0/3564462732 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:23:16.782 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.786 7f6918fba700 1 --2- 172.21.2.112:0/3564462732 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6914105d10 0x7f691409eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:16.782 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.786 7f69197bb700 1 -- 172.21.2.112:0/3564462732 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6914105d10 msgr2=0x7f691409eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:16.782 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.786 7f69197bb700 1 --2- 172.21.2.112:0/3564462732 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6914105d10 0x7f691409eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:16.782 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.786 7f69197bb700 1 -- 172.21.2.112:0/3564462732 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6914106110 msgr2=0x7f691409f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:16.782 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.786 7f69197bb700 1 --2- 172.21.2.112:0/3564462732 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6914106110 0x7f691409f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:16.782 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.786 7f69197bb700 1 -- 172.21.2.112:0/3564462732 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69140b3640 con 0x7f69140b2a00 2021-02-10T11:23:16.782 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.786 7f69197bb700 1 --2- 172.21.2.112:0/3564462732 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f69140b2a00 0x7f69140b2e00 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 rx=0x7f690c00b860 tx=0x7f690c00bc00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:16.782 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.787 7f69097fa700 1 -- 172.21.2.112:0/3564462732 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f690c001130 con 0x7f69140b2a00 2021-02-10T11:23:16.782 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.787 7f69097fa700 1 -- 172.21.2.112:0/3564462732 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f690c001430 con 0x7f69140b2a00 2021-02-10T11:23:16.782 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.787 7f69097fa700 1 -- 172.21.2.112:0/3564462732 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f690c001890 con 0x7f69140b2a00 2021-02-10T11:23:16.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.787 7f691b19e700 1 -- 172.21.2.112:0/3564462732 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69140b3640 con 0x7f69140b2a00 2021-02-10T11:23:16.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.787 7f691b19e700 1 -- 172.21.2.112:0/3564462732 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69140b3640 con 0x7f69140b2a00 2021-02-10T11:23:16.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.787 7f69097fa700 1 -- 172.21.2.112:0/3564462732 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f690c001130 con 0x7f69140b2a00 2021-02-10T11:23:16.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.787 7f69097fa700 1 --2- 172.21.2.112:0/3564462732 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f68f4018730 0x7f68f401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:16.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.787 7f69097fa700 1 -- 172.21.2.112:0/3564462732 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f690c0328f0 con 0x7f69140b2a00 2021-02-10T11:23:16.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.788 7f6918fba700 1 --2- 172.21.2.112:0/3564462732 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f68f4018730 0x7f68f401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:16.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.788 7f6918fba700 1 --2- 172.21.2.112:0/3564462732 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f68f4018730 0x7f68f401ac10 crc :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:16.810 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.818 7f691b19e700 1 -- 172.21.2.112:0/3564462732 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69140b3ec0 con 0x7f69140b2a00 2021-02-10T11:23:16.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.820 7f69097fa700 1 -- 172.21.2.112:0/3564462732 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f690c000df0 con 0x7f69140b2a00 2021-02-10T11:23:16.943 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.951 7f691b19e700 1 -- 172.21.2.112:0/3564462732 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348620", "target": ["mgr", ""]}) v1 -- 0x7f69141a16c0 con 0x7f68f4018730 2021-02-10T11:23:16.946 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.955 7f69097fa700 1 -- 172.21.2.112:0/3564462732 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f6904006a10 con 0x7f68f4018730 2021-02-10T11:23:16.951 INFO:teuthology.orchestra.run.gibba012.stdout:/volumes/_nogroup/subvolume_0000000000348620/52b0dd1b-7bce-4ce5-b76a-97e4102eccb8 2021-02-10T11:23:16.952 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.960 7f691b19e700 1 -- 172.21.2.112:0/3564462732 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f68f4018730 msgr2=0x7f68f401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:16.952 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.960 7f691b19e700 1 --2- 172.21.2.112:0/3564462732 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f68f4018730 0x7f68f401ac10 crc :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:16.952 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.960 7f691b19e700 1 -- 172.21.2.112:0/3564462732 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f69140b2a00 msgr2=0x7f69140b2e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:16.952 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.960 7f691b19e700 1 --2- 172.21.2.112:0/3564462732 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f69140b2a00 0x7f69140b2e00 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 rx=0x7f690c00b860 tx=0x7f690c00bc00).stop 2021-02-10T11:23:16.952 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.960 7f691b19e700 1 -- 172.21.2.112:0/3564462732 shutdown_connections 2021-02-10T11:23:16.953 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.960 7f691b19e700 1 --2- 172.21.2.112:0/3564462732 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f68f4018730 0x7f68f401ac10 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:16.953 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.960 7f691b19e700 1 --2- 172.21.2.112:0/3564462732 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6914106110 0x7f691409f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:16.953 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.960 7f691b19e700 1 --2- 172.21.2.112:0/3564462732 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f69140b2a00 0x7f69140b2e00 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:16.953 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.960 7f691b19e700 1 --2- 172.21.2.112:0/3564462732 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6914105d10 0x7f691409eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:16.953 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.960 7f691b19e700 1 -- 172.21.2.112:0/3564462732 >> 172.21.2.112:0/3564462732 conn(0x7f69140c36f0 msgr2=0x7f69140bd640 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:16.953 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.960 7f691b19e700 1 -- 172.21.2.112:0/3564462732 shutdown_connections 2021-02-10T11:23:16.953 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:16.960 7f691b19e700 1 -- 172.21.2.112:0/3564462732 wait complete. 2021-02-10T11:23:16.960 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000348620 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000348620/52b0dd1b-7bce-4ce5-b76a-97e4102eccb8 2021-02-10T11:23:16.960 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:16.960 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:23:16.960 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:23:17.296 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:23:17.296 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:23:17.296 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348620/52b0dd1b-7bce-4ce5-b76a-97e4102eccb8/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:23:17.678 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:23:17.678 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:23:17.679 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0249484 s, 42.0 MB/s 2021-02-10T11:23:17.679 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:17.680 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:23:17.680 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:23:18.015 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:23:18.015 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:23:18.015 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348620/52b0dd1b-7bce-4ce5-b76a-97e4102eccb8/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:23:18.392 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:23:18.392 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:23:18.392 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0248816 s, 42.1 MB/s 2021-02-10T11:23:18.393 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:18.393 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:23:18.394 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:23:18.729 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:23:18.729 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:23:18.729 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348620/52b0dd1b-7bce-4ce5-b76a-97e4102eccb8/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:23:19.103 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:23:19.103 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:23:19.104 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0239688 s, 43.7 MB/s 2021-02-10T11:23:19.105 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:19.105 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:23:19.105 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:23:19.440 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:23:19.440 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:23:19.440 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348620/52b0dd1b-7bce-4ce5-b76a-97e4102eccb8/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:23:19.813 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:23:19.814 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:23:19.814 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0261623 s, 40.1 MB/s 2021-02-10T11:23:19.815 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:19.815 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:23:19.815 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:23:20.151 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:23:20.151 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:23:20.151 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348620/52b0dd1b-7bce-4ce5-b76a-97e4102eccb8/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:23:20.526 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:23:20.527 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:23:20.527 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0241507 s, 43.4 MB/s 2021-02-10T11:23:20.528 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:20.528 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:23:20.528 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:23:20.863 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:23:20.863 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:23:20.863 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348620/52b0dd1b-7bce-4ce5-b76a-97e4102eccb8/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:23:21.241 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:23:21.241 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:23:21.241 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0266111 s, 39.4 MB/s 2021-02-10T11:23:21.242 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:21.243 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:23:21.243 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:23:21.578 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:23:21.578 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:23:21.578 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348620/52b0dd1b-7bce-4ce5-b76a-97e4102eccb8/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:23:21.951 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:23:21.951 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:23:21.951 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.022557 s, 46.5 MB/s 2021-02-10T11:23:21.952 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:21.952 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:23:21.952 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:23:22.287 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:23:22.287 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:23:22.288 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348620/52b0dd1b-7bce-4ce5-b76a-97e4102eccb8/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:23:22.663 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:23:22.663 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:23:22.663 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0248347 s, 42.2 MB/s 2021-02-10T11:23:22.664 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:22.664 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:23:22.664 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:23:23.000 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:23:23.000 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:23:23.000 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348620/52b0dd1b-7bce-4ce5-b76a-97e4102eccb8/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:23:23.376 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:23:23.519 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:23:23.519 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0245451 s, 42.7 MB/s 2021-02-10T11:23:23.520 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:23.521 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:23:23.521 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:23:23.856 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:23:23.856 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:23:23.856 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348620/52b0dd1b-7bce-4ce5-b76a-97e4102eccb8/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:23:24.231 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:23:24.231 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:23:24.231 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0244886 s, 42.8 MB/s 2021-02-10T11:23:24.232 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000348621 2021-02-10T11:23:24.663 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:24.654 7fc5ceb56700 1 Processor -- start 2021-02-10T11:23:24.663 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:24.654 7fc5ceb56700 1 -- start start 2021-02-10T11:23:24.663 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:24.655 7fc5ceb56700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fc5c81b0470 0x7fc5c81b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:24.663 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:24.655 7fc5ceb56700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc5c80f20e0 0x7fc5c80f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:24.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:24.655 7fc5ceb56700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5c80f0b50 con 0x7fc5c80ef610 2021-02-10T11:23:24.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:24.655 7fc5ceb56700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5c80f0820 con 0x7fc5c81b0470 2021-02-10T11:23:24.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:24.655 7fc5ceb56700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5c80f24e0 con 0x7fc5c80f20e0 2021-02-10T11:23:24.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:24.655 7fc5c7fff700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fc5c80ef610 0x7fc5c80efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:44746/0 (socket says 172.21.2.112:44746) 2021-02-10T11:23:24.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:24.655 7fc5c7fff700 1 -- 172.21.2.112:0/3610460874 learned_addr learned my addr 172.21.2.112:0/3610460874 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:23:24.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:24.655 7fc5cc972700 1 --2- 172.21.2.112:0/3610460874 >> v2:172.21.2.120:3300/0 conn(0x7fc5c81b0470 0x7fc5c81b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:24.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:24.655 7fc5cd173700 1 --2- 172.21.2.112:0/3610460874 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc5c80f20e0 0x7fc5c80f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:24.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:24.656 7fc5c77fe700 1 -- 172.21.2.112:0/3610460874 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3585844339 0 0) 0x7fc5b80009b0 con 0x7fc5c80ef610 2021-02-10T11:23:24.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:24.656 7fc5c77fe700 1 -- 172.21.2.112:0/3610460874 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5b4002fe0 con 0x7fc5c80ef610 2021-02-10T11:23:24.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:24.656 7fc5cd173700 1 -- 172.21.2.112:0/3610460874 >> v2:172.21.2.120:3300/0 conn(0x7fc5c81b0470 msgr2=0x7fc5c81b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:24.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:24.656 7fc5cd173700 1 --2- 172.21.2.112:0/3610460874 >> v2:172.21.2.120:3300/0 conn(0x7fc5c81b0470 0x7fc5c81b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:24.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:24.656 7fc5cd173700 1 -- 172.21.2.112:0/3610460874 >> v1:172.21.2.112:6789/0 conn(0x7fc5c80ef610 legacy=0x7fc5c80efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:24.665 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:24.656 7fc5cd173700 1 -- 172.21.2.112:0/3610460874 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5c80e52e0 con 0x7fc5c80f20e0 2021-02-10T11:23:24.665 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:24.656 7fc5cc972700 1 --2- 172.21.2.112:0/3610460874 >> v2:172.21.2.120:3300/0 conn(0x7fc5c81b0470 0x7fc5c81b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:23:24.665 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:24.656 7fc5cd173700 1 --2- 172.21.2.112:0/3610460874 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc5c80f20e0 0x7fc5c80f0200 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7fc5c0006e50 tx=0x7fc5c0009e70).ready entity=mon.2 client_cookie=5bb0ee9e0b41386 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:24.665 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:24.656 7fc5c77fe700 1 -- 172.21.2.112:0/3610460874 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc5c000b510 con 0x7fc5c80f20e0 2021-02-10T11:23:24.665 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:24.656 7fc5c77fe700 1 -- 172.21.2.112:0/3610460874 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5c0006790 con 0x7fc5c80f20e0 2021-02-10T11:23:24.665 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:24.656 7fc5ceb56700 1 -- 172.21.2.112:0/3610460874 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc5c80f20e0 msgr2=0x7fc5c80f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:24.665 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:24.656 7fc5ceb56700 1 --2- 172.21.2.112:0/3610460874 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc5c80f20e0 0x7fc5c80f0200 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7fc5c0006e50 tx=0x7fc5c0009e70).stop 2021-02-10T11:23:24.665 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:24.656 7fc5c77fe700 1 -- 172.21.2.112:0/3610460874 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc5c000b940 con 0x7fc5c80f20e0 2021-02-10T11:23:24.665 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:24.657 7fc5ceb56700 1 -- 172.21.2.112:0/3610460874 shutdown_connections 2021-02-10T11:23:24.665 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:24.657 7fc5ceb56700 1 --2- 172.21.2.112:0/3610460874 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc5c80f20e0 0x7fc5c80f0200 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:24.665 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:24.657 7fc5ceb56700 1 --2- 172.21.2.112:0/3610460874 >> v2:172.21.2.120:3300/0 conn(0x7fc5c81b0470 0x7fc5c81b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:24.665 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:24.657 7fc5ceb56700 1 -- 172.21.2.112:0/3610460874 >> 172.21.2.112:0/3610460874 conn(0x7fc5c80ce3e0 msgr2=0x7fc5c80e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:24.665 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:24.657 7fc5ceb56700 1 -- 172.21.2.112:0/3610460874 shutdown_connections 2021-02-10T11:23:24.665 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:24.657 7fc5ceb56700 1 -- 172.21.2.112:0/3610460874 wait complete. 2021-02-10T11:23:24.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:24.672 7fc5ceb56700 1 Processor -- start 2021-02-10T11:23:24.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:24.672 7fc5ceb56700 1 -- start start 2021-02-10T11:23:24.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:24.672 7fc5ceb56700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc5c80a0420 0x7fc5c80a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:24.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:24.672 7fc5ceb56700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc5c8189d00 0x7fc5c818a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:24.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:24.672 7fc5ceb56700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc5c818a720 0x7fc5c80b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:24.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:24.672 7fc5ceb56700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5c8189780 con 0x7fc5c818a720 2021-02-10T11:23:24.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:24.672 7fc5ceb56700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5c818ab20 con 0x7fc5c80a0420 2021-02-10T11:23:24.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:24.672 7fc5ceb56700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5c80a1140 con 0x7fc5c8189d00 2021-02-10T11:23:24.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:24.672 7fc5cd173700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc5c818a720 0x7fc5c80b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:24.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:24.672 7fc5cd173700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc5c818a720 0x7fc5c80b7be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:45292/0 (socket says 172.21.2.112:45292) 2021-02-10T11:23:24.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:24.672 7fc5c7fff700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc5c8189d00 0x7fc5c818a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:24.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:24.672 7fc5cd173700 1 -- 172.21.2.112:0/3564120028 learned_addr learned my addr 172.21.2.112:0/3564120028 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:23:24.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:24.673 7fc5cc972700 1 --2- 172.21.2.112:0/3564120028 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc5c80a0420 0x7fc5c80a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:24.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:24.673 7fc5c7fff700 1 -- 172.21.2.112:0/3564120028 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc5c80a0420 msgr2=0x7fc5c80a0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:24.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:24.673 7fc5c7fff700 1 --2- 172.21.2.112:0/3564120028 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc5c80a0420 0x7fc5c80a0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:24.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:24.673 7fc5c7fff700 1 -- 172.21.2.112:0/3564120028 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc5c818a720 msgr2=0x7fc5c80b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:24.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:24.673 7fc5c7fff700 1 --2- 172.21.2.112:0/3564120028 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc5c818a720 0x7fc5c80b7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:24.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:24.673 7fc5c7fff700 1 -- 172.21.2.112:0/3564120028 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5c80b8200 con 0x7fc5c8189d00 2021-02-10T11:23:24.669 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:24.673 7fc5c7fff700 1 --2- 172.21.2.112:0/3564120028 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc5c8189d00 0x7fc5c818a100 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7fc5b8007f80 tx=0x7fc5b800a800).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:24.669 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:24.673 7fc5c57fa700 1 -- 172.21.2.112:0/3564120028 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc5b80009b0 con 0x7fc5c8189d00 2021-02-10T11:23:24.669 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:24.673 7fc5c57fa700 1 -- 172.21.2.112:0/3564120028 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5b8000f10 con 0x7fc5c8189d00 2021-02-10T11:23:24.669 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:24.673 7fc5c57fa700 1 -- 172.21.2.112:0/3564120028 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc5b800ade0 con 0x7fc5c8189d00 2021-02-10T11:23:24.669 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:24.673 7fc5ceb56700 1 -- 172.21.2.112:0/3564120028 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5c80b8200 con 0x7fc5c8189d00 2021-02-10T11:23:24.669 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:24.673 7fc5ceb56700 1 -- 172.21.2.112:0/3564120028 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5c80b8790 con 0x7fc5c8189d00 2021-02-10T11:23:24.669 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:24.674 7fc5c57fa700 1 -- 172.21.2.112:0/3564120028 <== mon.2 v2:172.21.2.120:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc5b8004140 con 0x7fc5c8189d00 2021-02-10T11:23:24.669 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:24.674 7fc5c57fa700 1 --2- 172.21.2.112:0/3564120028 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc5ac0186c0 0x7fc5ac01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:24.669 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:24.674 7fc5c57fa700 1 -- 172.21.2.112:0/3564120028 <== mon.2 v2:172.21.2.120:3301/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fc5b8032340 con 0x7fc5c8189d00 2021-02-10T11:23:24.669 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:24.674 7fc5cc972700 1 --2- 172.21.2.112:0/3564120028 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc5ac0186c0 0x7fc5ac01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:24.670 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:24.675 7fc5cc972700 1 --2- 172.21.2.112:0/3564120028 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc5ac0186c0 0x7fc5ac01abc0 crc :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:24.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:24.705 7fc5ceb56700 1 -- 172.21.2.112:0/3564120028 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5c80a1360 con 0x7fc5c8189d00 2021-02-10T11:23:24.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:24.708 7fc5c57fa700 1 -- 172.21.2.112:0/3564120028 <== mon.2 v2:172.21.2.120:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc5b80009b0 con 0x7fc5c8189d00 2021-02-10T11:23:24.826 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:24.834 7fc5ceb56700 1 -- 172.21.2.112:0/3564120028 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348621", "target": ["mgr", ""]}) v1 -- 0x7fc5c80b9020 con 0x7fc5ac0186c0 2021-02-10T11:23:24.848 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:24.856 7fc5c57fa700 1 -- 172.21.2.112:0/3564120028 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fc5b0006a30 con 0x7fc5ac0186c0 2021-02-10T11:23:24.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:24.867 7fc5ceb56700 1 -- 172.21.2.112:0/3564120028 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc5ac0186c0 msgr2=0x7fc5ac01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:24.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:24.867 7fc5ceb56700 1 --2- 172.21.2.112:0/3564120028 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc5ac0186c0 0x7fc5ac01abc0 crc :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:24.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:24.867 7fc5ceb56700 1 -- 172.21.2.112:0/3564120028 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc5c8189d00 msgr2=0x7fc5c818a100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:24.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:24.867 7fc5ceb56700 1 --2- 172.21.2.112:0/3564120028 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc5c8189d00 0x7fc5c818a100 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7fc5b8007f80 tx=0x7fc5b800a800).stop 2021-02-10T11:23:24.860 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:24.867 7fc5ceb56700 1 -- 172.21.2.112:0/3564120028 shutdown_connections 2021-02-10T11:23:24.860 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:24.867 7fc5ceb56700 1 --2- 172.21.2.112:0/3564120028 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc5ac0186c0 0x7fc5ac01abc0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:24.860 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:24.867 7fc5ceb56700 1 --2- 172.21.2.112:0/3564120028 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc5c818a720 0x7fc5c80b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:24.860 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:24.867 7fc5ceb56700 1 --2- 172.21.2.112:0/3564120028 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc5c8189d00 0x7fc5c818a100 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:24.860 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:24.867 7fc5ceb56700 1 --2- 172.21.2.112:0/3564120028 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc5c80a0420 0x7fc5c80a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:24.860 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:24.867 7fc5ceb56700 1 -- 172.21.2.112:0/3564120028 >> 172.21.2.112:0/3564120028 conn(0x7fc5c8094e10 msgr2=0x7fc5c80a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:24.860 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:24.868 7fc5ceb56700 1 -- 172.21.2.112:0/3564120028 shutdown_connections 2021-02-10T11:23:24.860 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:24.868 7fc5ceb56700 1 -- 172.21.2.112:0/3564120028 wait complete. 2021-02-10T11:23:24.867 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000348621 2021-02-10T11:23:25.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:25.291 7fd702171700 1 Processor -- start 2021-02-10T11:23:25.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:25.291 7fd702171700 1 -- start start 2021-02-10T11:23:25.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:25.291 7fd702171700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fd6fc1b0470 0x7fd6fc1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:25.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:25.291 7fd702171700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fd6fc0ef610 0x7fd6fc0efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:25.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:25.291 7fd702171700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6fc0f0b50 con 0x7fd6fc0f0030 2021-02-10T11:23:25.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:25.291 7fd702171700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6fc0e52e0 con 0x7fd6fc0ef610 2021-02-10T11:23:25.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:25.291 7fd702171700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6fc0e5490 con 0x7fd6fc1b0470 2021-02-10T11:23:25.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:25.292 7fd6faffd700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fd6fc0ef610 0x7fd6fc0efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:25.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:25.292 7fd6faffd700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fd6fc0ef610 0x7fd6fc0efa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:45296/0 (socket says 172.21.2.112:45296) 2021-02-10T11:23:25.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:25.292 7fd6faffd700 1 -- 172.21.2.112:0/171730494 learned_addr learned my addr 172.21.2.112:0/171730494 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:23:25.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:25.292 7fd6fbfff700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fd6fc0f0030 0x7fd6fc0f10d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:44758/0 (socket says 172.21.2.112:44758) 2021-02-10T11:23:25.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:25.292 7fd6faffd700 1 -- 172.21.2.112:0/171730494 >> v2:172.21.2.120:3300/0 conn(0x7fd6fc1b0470 msgr2=0x7fd6fc1b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:25.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:25.292 7fd6faffd700 1 --2- 172.21.2.112:0/171730494 >> v2:172.21.2.120:3300/0 conn(0x7fd6fc1b0470 0x7fd6fc1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:25.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:25.292 7fd6faffd700 1 -- 172.21.2.112:0/171730494 >> v1:172.21.2.112:6789/0 conn(0x7fd6fc0f0030 legacy=0x7fd6fc0f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:25.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:25.292 7fd6fa7fc700 1 -- 172.21.2.112:0/171730494 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3281834018 0 0) 0x7fd6f4001480 con 0x7fd6fc0f0030 2021-02-10T11:23:25.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:25.292 7fd6faffd700 1 -- 172.21.2.112:0/171730494 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6fc0f0430 con 0x7fd6fc0ef610 2021-02-10T11:23:25.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:25.292 7fd6faffd700 1 --2- 172.21.2.112:0/171730494 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd6fc0ef610 0x7fd6fc0efa10 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 rx=0x7fd6ec006700 tx=0x7fd6ec009b00).ready entity=mon.0 client_cookie=ff0dc3ee45418633 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:25.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:25.292 7fd6fa7fc700 1 -- 172.21.2.112:0/171730494 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd6ec00c0c0 con 0x7fd6fc0ef610 2021-02-10T11:23:25.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:25.292 7fd6fa7fc700 1 -- 172.21.2.112:0/171730494 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6ec00c340 con 0x7fd6fc0ef610 2021-02-10T11:23:25.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:25.292 7fd6fa7fc700 1 -- 172.21.2.112:0/171730494 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd6ec00c820 con 0x7fd6fc0ef610 2021-02-10T11:23:25.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:25.292 7fd702171700 1 -- 172.21.2.112:0/171730494 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd6fc0ef610 msgr2=0x7fd6fc0efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:25.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:25.292 7fd702171700 1 --2- 172.21.2.112:0/171730494 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd6fc0ef610 0x7fd6fc0efa10 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 rx=0x7fd6ec006700 tx=0x7fd6ec009b00).stop 2021-02-10T11:23:25.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:25.293 7fd702171700 1 -- 172.21.2.112:0/171730494 shutdown_connections 2021-02-10T11:23:25.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:25.293 7fd702171700 1 --2- 172.21.2.112:0/171730494 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd6fc0ef610 0x7fd6fc0efa10 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:25.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:25.293 7fd702171700 1 --2- 172.21.2.112:0/171730494 >> v2:172.21.2.120:3300/0 conn(0x7fd6fc1b0470 0x7fd6fc1b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:25.302 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:25.293 7fd702171700 1 -- 172.21.2.112:0/171730494 >> 172.21.2.112:0/171730494 conn(0x7fd6fc0ce3e0 msgr2=0x7fd6fc0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:25.302 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:25.293 7fd702171700 1 -- 172.21.2.112:0/171730494 shutdown_connections 2021-02-10T11:23:25.302 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:25.293 7fd702171700 1 -- 172.21.2.112:0/171730494 wait complete. 2021-02-10T11:23:25.303 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:25.308 7fd702171700 1 Processor -- start 2021-02-10T11:23:25.303 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:25.308 7fd702171700 1 -- start start 2021-02-10T11:23:25.303 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:25.308 7fd702171700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd6fc0b20b0 0x7fd6fc0ac970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:25.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:25.308 7fd702171700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd6fc0b24b0 0x7fd6fc191dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:25.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:25.308 7fd702171700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd6fc0acf90 0x7fd6fc1923e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:25.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:25.308 7fd702171700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6fc0b1b30 con 0x7fd6fc0b20b0 2021-02-10T11:23:25.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:25.308 7fd702171700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6fc1a04e0 con 0x7fd6fc0b24b0 2021-02-10T11:23:25.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:25.308 7fd702171700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6fc1a06c0 con 0x7fd6fc0acf90 2021-02-10T11:23:25.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:25.308 7fd6fb7fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd6fc0b20b0 0x7fd6fc0ac970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:25.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:25.308 7fd6fb7fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd6fc0b20b0 0x7fd6fc0ac970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:45300/0 (socket says 172.21.2.112:45300) 2021-02-10T11:23:25.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:25.308 7fd6fb7fe700 1 -- 172.21.2.112:0/3063842916 learned_addr learned my addr 172.21.2.112:0/3063842916 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:23:25.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:25.309 7fd6faffd700 1 --2- 172.21.2.112:0/3063842916 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd6fc0b24b0 0x7fd6fc191dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:25.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:25.309 7fd6fbfff700 1 --2- 172.21.2.112:0/3063842916 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd6fc0acf90 0x7fd6fc1923e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:25.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:25.309 7fd6fb7fe700 1 -- 172.21.2.112:0/3063842916 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd6fc0acf90 msgr2=0x7fd6fc1923e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:25.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:25.309 7fd6fb7fe700 1 --2- 172.21.2.112:0/3063842916 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd6fc0acf90 0x7fd6fc1923e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:25.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:25.309 7fd6fb7fe700 1 -- 172.21.2.112:0/3063842916 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd6fc0b24b0 msgr2=0x7fd6fc191dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:25.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:25.309 7fd6fb7fe700 1 --2- 172.21.2.112:0/3063842916 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd6fc0b24b0 0x7fd6fc191dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:25.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:25.309 7fd6fb7fe700 1 -- 172.21.2.112:0/3063842916 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6fc192be0 con 0x7fd6fc0b20b0 2021-02-10T11:23:25.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:25.309 7fd6fb7fe700 1 --2- 172.21.2.112:0/3063842916 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd6fc0b20b0 0x7fd6fc0ac970 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 rx=0x7fd6e4006f00 tx=0x7fd6e400b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:25.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:25.309 7fd6f3fff700 1 -- 172.21.2.112:0/3063842916 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd6e40009b0 con 0x7fd6fc0b20b0 2021-02-10T11:23:25.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:25.309 7fd6f3fff700 1 -- 172.21.2.112:0/3063842916 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6e4006350 con 0x7fd6fc0b20b0 2021-02-10T11:23:25.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:25.309 7fd6f3fff700 1 -- 172.21.2.112:0/3063842916 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd6e4000c90 con 0x7fd6fc0b20b0 2021-02-10T11:23:25.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:25.309 7fd702171700 1 -- 172.21.2.112:0/3063842916 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6fc192be0 con 0x7fd6fc0b20b0 2021-02-10T11:23:25.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:25.309 7fd702171700 1 -- 172.21.2.112:0/3063842916 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6fc193170 con 0x7fd6fc0b20b0 2021-02-10T11:23:25.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:25.310 7fd6f3fff700 1 -- 172.21.2.112:0/3063842916 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd6e4000c90 con 0x7fd6fc0b20b0 2021-02-10T11:23:25.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:25.310 7fd6f3fff700 1 --2- 172.21.2.112:0/3063842916 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd6e00186d0 0x7fd6e001abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:25.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:25.310 7fd6f3fff700 1 -- 172.21.2.112:0/3063842916 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fd6e4032690 con 0x7fd6fc0b20b0 2021-02-10T11:23:25.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:25.310 7fd6faffd700 1 --2- 172.21.2.112:0/3063842916 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd6e00186d0 0x7fd6e001abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:25.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:25.311 7fd6faffd700 1 --2- 172.21.2.112:0/3063842916 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd6e00186d0 0x7fd6e001abd0 crc :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:25.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:25.341 7fd702171700 1 -- 172.21.2.112:0/3063842916 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6fc105af0 con 0x7fd6fc0b20b0 2021-02-10T11:23:25.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:25.344 7fd6f3fff700 1 -- 172.21.2.112:0/3063842916 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd6e4000960 con 0x7fd6fc0b20b0 2021-02-10T11:23:25.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:25.473 7fd702171700 1 -- 172.21.2.112:0/3063842916 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348621", "target": ["mgr", ""]}) v1 -- 0x7fd6fc1a06c0 con 0x7fd6e00186d0 2021-02-10T11:23:25.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:25.476 7fd6f3fff700 1 -- 172.21.2.112:0/3063842916 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7fd6ec0094c0 con 0x7fd6e00186d0 2021-02-10T11:23:25.472 INFO:teuthology.orchestra.run.gibba012.stdout:/volumes/_nogroup/subvolume_0000000000348621/7dae7022-71e1-4376-8d11-3801901f193e 2021-02-10T11:23:25.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:25.482 7fd702171700 1 -- 172.21.2.112:0/3063842916 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd6e00186d0 msgr2=0x7fd6e001abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:25.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:25.482 7fd702171700 1 --2- 172.21.2.112:0/3063842916 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd6e00186d0 0x7fd6e001abd0 crc :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:25.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:25.482 7fd702171700 1 -- 172.21.2.112:0/3063842916 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd6fc0b20b0 msgr2=0x7fd6fc0ac970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:25.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:25.482 7fd702171700 1 --2- 172.21.2.112:0/3063842916 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd6fc0b20b0 0x7fd6fc0ac970 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 rx=0x7fd6e4006f00 tx=0x7fd6e400b940).stop 2021-02-10T11:23:25.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:25.482 7fd702171700 1 -- 172.21.2.112:0/3063842916 shutdown_connections 2021-02-10T11:23:25.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:25.482 7fd702171700 1 --2- 172.21.2.112:0/3063842916 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd6e00186d0 0x7fd6e001abd0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:25.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:25.482 7fd702171700 1 --2- 172.21.2.112:0/3063842916 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd6fc0b24b0 0x7fd6fc191dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:25.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:25.482 7fd702171700 1 --2- 172.21.2.112:0/3063842916 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd6fc0acf90 0x7fd6fc1923e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:25.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:25.482 7fd702171700 1 --2- 172.21.2.112:0/3063842916 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd6fc0b20b0 0x7fd6fc0ac970 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:25.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:25.482 7fd702171700 1 -- 172.21.2.112:0/3063842916 >> 172.21.2.112:0/3063842916 conn(0x7fd6fc0bf4e0 msgr2=0x7fd6fc0be3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:25.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:25.483 7fd702171700 1 -- 172.21.2.112:0/3063842916 shutdown_connections 2021-02-10T11:23:25.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:25.483 7fd702171700 1 -- 172.21.2.112:0/3063842916 wait complete. 2021-02-10T11:23:25.482 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000348621 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000348621/7dae7022-71e1-4376-8d11-3801901f193e 2021-02-10T11:23:25.482 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:25.483 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:23:25.483 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:23:25.820 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:23:25.820 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:23:25.820 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348621/7dae7022-71e1-4376-8d11-3801901f193e/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:23:26.199 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:23:26.199 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:23:26.199 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0228863 s, 45.8 MB/s 2021-02-10T11:23:26.200 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:26.200 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:23:26.200 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:23:26.536 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:23:26.537 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:23:26.537 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348621/7dae7022-71e1-4376-8d11-3801901f193e/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:23:26.914 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:23:26.914 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:23:26.914 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0257522 s, 40.7 MB/s 2021-02-10T11:23:26.915 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:26.915 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:23:26.915 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:23:27.251 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:23:27.251 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:23:27.251 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348621/7dae7022-71e1-4376-8d11-3801901f193e/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:23:27.627 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:23:27.627 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:23:27.628 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0239593 s, 43.8 MB/s 2021-02-10T11:23:27.628 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:27.629 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:23:27.629 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:23:27.964 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:23:27.964 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:23:27.964 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348621/7dae7022-71e1-4376-8d11-3801901f193e/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:23:28.342 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:23:28.342 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:23:28.342 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0251168 s, 41.7 MB/s 2021-02-10T11:23:28.343 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:28.343 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:23:28.343 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:23:28.681 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:23:28.681 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:23:28.681 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348621/7dae7022-71e1-4376-8d11-3801901f193e/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:23:29.058 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:23:29.058 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:23:29.058 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.023097 s, 45.4 MB/s 2021-02-10T11:23:29.060 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:29.061 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:23:29.061 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:23:29.399 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:23:29.399 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:23:29.399 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348621/7dae7022-71e1-4376-8d11-3801901f193e/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:23:29.772 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:23:29.772 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:23:29.772 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0258247 s, 40.6 MB/s 2021-02-10T11:23:29.773 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:29.773 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:23:29.773 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:23:30.108 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:23:30.109 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:23:30.109 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348621/7dae7022-71e1-4376-8d11-3801901f193e/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:23:30.484 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:23:30.484 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:23:30.485 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0242731 s, 43.2 MB/s 2021-02-10T11:23:30.485 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:30.486 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:23:30.486 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:23:30.820 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:23:30.820 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:23:30.820 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348621/7dae7022-71e1-4376-8d11-3801901f193e/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:23:31.199 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:23:31.200 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:23:31.200 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0265468 s, 39.5 MB/s 2021-02-10T11:23:31.201 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:31.201 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:23:31.201 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:23:31.537 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:23:31.537 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:23:31.537 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348621/7dae7022-71e1-4376-8d11-3801901f193e/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:23:31.914 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:23:31.914 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:23:31.914 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0265478 s, 39.5 MB/s 2021-02-10T11:23:31.915 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:31.915 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:23:31.915 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:23:32.250 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:23:32.251 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:23:32.251 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348621/7dae7022-71e1-4376-8d11-3801901f193e/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:23:32.627 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:23:32.627 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:23:32.627 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0241145 s, 43.5 MB/s 2021-02-10T11:23:32.629 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000348622 2021-02-10T11:23:33.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.052 7f16d9dca700 1 Processor -- start 2021-02-10T11:23:33.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.052 7f16d9dca700 1 -- start start 2021-02-10T11:23:33.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.052 7f16d9dca700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f16d41b0870 0x7f16d40f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:33.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.052 7f16d9dca700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f16d40efe00 0x7f16d40f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:33.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.052 7f16d9dca700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16d41b0c70 con 0x7f16d41b0470 2021-02-10T11:23:33.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.052 7f16d9dca700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16d40e5ea0 con 0x7f16d40efe00 2021-02-10T11:23:33.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.052 7f16d9dca700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16d40f0b50 con 0x7f16d41b0870 2021-02-10T11:23:33.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.053 7f16d3fff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f16d40efe00 0x7f16d40f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:33.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.053 7f16d3fff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f16d40efe00 0x7f16d40f0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:45310/0 (socket says 172.21.2.112:45310) 2021-02-10T11:23:33.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.053 7f16d3fff700 1 -- 172.21.2.112:0/3112675472 learned_addr learned my addr 172.21.2.112:0/3112675472 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:23:33.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.053 7f16d2ffd700 1 --2- 172.21.2.112:0/3112675472 >> v2:172.21.2.120:3300/0 conn(0x7f16d41b0870 0x7f16d40f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:33.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.053 7f16d3fff700 1 -- 172.21.2.112:0/3112675472 >> v2:172.21.2.120:3300/0 conn(0x7f16d41b0870 msgr2=0x7f16d40f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:33.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.053 7f16d3fff700 1 --2- 172.21.2.112:0/3112675472 >> v2:172.21.2.120:3300/0 conn(0x7f16d41b0870 0x7f16d40f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:33.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.053 7f16d3fff700 1 -- 172.21.2.112:0/3112675472 >> v1:172.21.2.120:6789/0 conn(0x7f16d41b0470 legacy=0x7f16d40ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:33.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.053 7f16d3fff700 1 -- 172.21.2.112:0/3112675472 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16d40f0820 con 0x7f16d40efe00 2021-02-10T11:23:33.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.053 7f16d2ffd700 1 --2- 172.21.2.112:0/3112675472 >> v2:172.21.2.120:3300/0 conn(0x7f16d41b0870 0x7f16d40f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:23:33.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.053 7f16d3fff700 1 --2- 172.21.2.112:0/3112675472 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f16d40efe00 0x7f16d40f0200 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 rx=0x7f16cc006f00 tx=0x7f16cc009f10).ready entity=mon.0 client_cookie=c12338a45a2da201 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:33.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.053 7f16d27fc700 1 -- 172.21.2.112:0/3112675472 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f16cc00b3b0 con 0x7f16d40efe00 2021-02-10T11:23:33.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.053 7f16d27fc700 1 -- 172.21.2.112:0/3112675472 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16cc000d30 con 0x7f16d40efe00 2021-02-10T11:23:33.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.053 7f16d27fc700 1 -- 172.21.2.112:0/3112675472 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f16cc00b820 con 0x7f16d40efe00 2021-02-10T11:23:33.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.053 7f16d9dca700 1 -- 172.21.2.112:0/3112675472 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f16d40efe00 msgr2=0x7f16d40f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:33.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.053 7f16d9dca700 1 --2- 172.21.2.112:0/3112675472 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f16d40efe00 0x7f16d40f0200 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 rx=0x7f16cc006f00 tx=0x7f16cc009f10).stop 2021-02-10T11:23:33.063 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.054 7f16d9dca700 1 -- 172.21.2.112:0/3112675472 shutdown_connections 2021-02-10T11:23:33.063 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.054 7f16d9dca700 1 --2- 172.21.2.112:0/3112675472 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f16d40efe00 0x7f16d40f0200 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:33.063 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.054 7f16d9dca700 1 --2- 172.21.2.112:0/3112675472 >> v2:172.21.2.120:3300/0 conn(0x7f16d41b0870 0x7f16d40f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:33.063 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.054 7f16d9dca700 1 -- 172.21.2.112:0/3112675472 >> 172.21.2.112:0/3112675472 conn(0x7f16d40ce3e0 msgr2=0x7f16d40e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:33.063 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.054 7f16d9dca700 1 -- 172.21.2.112:0/3112675472 shutdown_connections 2021-02-10T11:23:33.063 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.054 7f16d9dca700 1 -- 172.21.2.112:0/3112675472 wait complete. 2021-02-10T11:23:33.063 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.069 7f16d9dca700 1 Processor -- start 2021-02-10T11:23:33.063 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.069 7f16d9dca700 1 -- start start 2021-02-10T11:23:33.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.069 7f16d9dca700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f16d4106fd0 0x7f16d40bc380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:33.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.069 7f16d9dca700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f16d40bc9a0 0x7f16d41a0e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:33.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.069 7f16d9dca700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f16d41a1430 0x7f16d408e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:33.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.069 7f16d9dca700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16d41073d0 con 0x7f16d4106fd0 2021-02-10T11:23:33.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.069 7f16d9dca700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16d41a1830 con 0x7f16d40bc9a0 2021-02-10T11:23:33.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.069 7f16d9dca700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16d41a1a50 con 0x7f16d41a1430 2021-02-10T11:23:33.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.070 7f16d37fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f16d4106fd0 0x7f16d40bc380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:33.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.070 7f16d2ffd700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f16d40bc9a0 0x7f16d41a0e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:33.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.070 7f16d37fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f16d4106fd0 0x7f16d40bc380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:45312/0 (socket says 172.21.2.112:45312) 2021-02-10T11:23:33.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.070 7f16d37fe700 1 -- 172.21.2.112:0/1234926140 learned_addr learned my addr 172.21.2.112:0/1234926140 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:23:33.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.070 7f16d3fff700 1 --2- 172.21.2.112:0/1234926140 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f16d41a1430 0x7f16d408e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:33.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.070 7f16d37fe700 1 -- 172.21.2.112:0/1234926140 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f16d41a1430 msgr2=0x7f16d408e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:33.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.070 7f16d37fe700 1 --2- 172.21.2.112:0/1234926140 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f16d41a1430 0x7f16d408e890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:33.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.070 7f16d37fe700 1 -- 172.21.2.112:0/1234926140 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f16d40bc9a0 msgr2=0x7f16d41a0e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:33.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.070 7f16d37fe700 1 --2- 172.21.2.112:0/1234926140 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f16d40bc9a0 0x7f16d41a0e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:33.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.070 7f16d37fe700 1 -- 172.21.2.112:0/1234926140 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16d408eeb0 con 0x7f16d4106fd0 2021-02-10T11:23:33.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.070 7f16d2ffd700 1 --2- 172.21.2.112:0/1234926140 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f16d40bc9a0 0x7f16d41a0e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:23:33.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.070 7f16d37fe700 1 --2- 172.21.2.112:0/1234926140 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f16d4106fd0 0x7f16d40bc380 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 rx=0x7f16c400b5b0 tx=0x7f16c400b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:33.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.070 7f16bbfff700 1 -- 172.21.2.112:0/1234926140 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f16c400bd40 con 0x7f16d4106fd0 2021-02-10T11:23:33.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.071 7f16bbfff700 1 -- 172.21.2.112:0/1234926140 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16c4006240 con 0x7f16d4106fd0 2021-02-10T11:23:33.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.071 7f16bbfff700 1 -- 172.21.2.112:0/1234926140 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f16c4005690 con 0x7f16d4106fd0 2021-02-10T11:23:33.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.071 7f16d9dca700 1 -- 172.21.2.112:0/1234926140 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16d408eeb0 con 0x7f16d4106fd0 2021-02-10T11:23:33.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.071 7f16d9dca700 1 -- 172.21.2.112:0/1234926140 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16d408f440 con 0x7f16d4106fd0 2021-02-10T11:23:33.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.071 7f16bbfff700 1 -- 172.21.2.112:0/1234926140 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f16c40054a0 con 0x7f16d4106fd0 2021-02-10T11:23:33.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.071 7f16bbfff700 1 --2- 172.21.2.112:0/1234926140 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f16b40186d0 0x7f16b401abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:33.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.071 7f16bbfff700 1 -- 172.21.2.112:0/1234926140 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f16c4032650 con 0x7f16d4106fd0 2021-02-10T11:23:33.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.071 7f16d2ffd700 1 --2- 172.21.2.112:0/1234926140 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f16b40186d0 0x7f16b401abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:33.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.072 7f16d2ffd700 1 --2- 172.21.2.112:0/1234926140 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f16b40186d0 0x7f16b401abd0 crc :-1 s=READY pgs=283 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:33.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.102 7f16d9dca700 1 -- 172.21.2.112:0/1234926140 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16d41078f0 con 0x7f16d4106fd0 2021-02-10T11:23:33.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.105 7f16bbfff700 1 -- 172.21.2.112:0/1234926140 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f16c4000c60 con 0x7f16d4106fd0 2021-02-10T11:23:33.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.233 7f16d9dca700 1 -- 172.21.2.112:0/1234926140 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348622", "target": ["mgr", ""]}) v1 -- 0x7f16d41073d0 con 0x7f16b40186d0 2021-02-10T11:23:33.249 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.256 7f16bbfff700 1 -- 172.21.2.112:0/1234926140 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f16c80009b0 con 0x7f16b40186d0 2021-02-10T11:23:33.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.266 7f16d9dca700 1 -- 172.21.2.112:0/1234926140 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f16b40186d0 msgr2=0x7f16b401abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:33.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.266 7f16d9dca700 1 --2- 172.21.2.112:0/1234926140 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f16b40186d0 0x7f16b401abd0 crc :-1 s=READY pgs=283 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:33.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.266 7f16d9dca700 1 -- 172.21.2.112:0/1234926140 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f16d4106fd0 msgr2=0x7f16d40bc380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:33.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.266 7f16d9dca700 1 --2- 172.21.2.112:0/1234926140 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f16d4106fd0 0x7f16d40bc380 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 rx=0x7f16c400b5b0 tx=0x7f16c400b950).stop 2021-02-10T11:23:33.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.266 7f16d9dca700 1 -- 172.21.2.112:0/1234926140 shutdown_connections 2021-02-10T11:23:33.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.266 7f16d9dca700 1 --2- 172.21.2.112:0/1234926140 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f16b40186d0 0x7f16b401abd0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:33.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.266 7f16d9dca700 1 --2- 172.21.2.112:0/1234926140 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f16d40bc9a0 0x7f16d41a0e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:33.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.266 7f16d9dca700 1 --2- 172.21.2.112:0/1234926140 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f16d41a1430 0x7f16d408e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:33.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.266 7f16d9dca700 1 --2- 172.21.2.112:0/1234926140 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f16d4106fd0 0x7f16d40bc380 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:33.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.266 7f16d9dca700 1 -- 172.21.2.112:0/1234926140 >> 172.21.2.112:0/1234926140 conn(0x7f16d40bbf60 msgr2=0x7f16d4105d10 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:33.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.266 7f16d9dca700 1 -- 172.21.2.112:0/1234926140 shutdown_connections 2021-02-10T11:23:33.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.266 7f16d9dca700 1 -- 172.21.2.112:0/1234926140 wait complete. 2021-02-10T11:23:33.267 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000348622 2021-02-10T11:23:33.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.688 7f61ebd86700 1 Processor -- start 2021-02-10T11:23:33.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.689 7f61ebd86700 1 -- start start 2021-02-10T11:23:33.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.689 7f61ebd86700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f61e41b0470 0x7f61e41b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:33.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.689 7f61ebd86700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f61e40f20e0 0x7f61e40f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:33.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.689 7f61ebd86700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61e40f0b50 con 0x7f61e40ef610 2021-02-10T11:23:33.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.689 7f61ebd86700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61e40f0820 con 0x7f61e41b0470 2021-02-10T11:23:33.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.689 7f61ebd86700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61e40f24e0 con 0x7f61e40f20e0 2021-02-10T11:23:33.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.690 7f61e9ba2700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f61e41b0470 0x7f61e41b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:33.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.690 7f61e93a1700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7f61e40ef610 0x7f61e40efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:38340/0 (socket says 172.21.2.112:38340) 2021-02-10T11:23:33.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.690 7f61ea3a3700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f61e40f20e0 0x7f61e40f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:33.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.690 7f61e93a1700 1 -- 172.21.2.112:0/2007986838 learned_addr learned my addr 172.21.2.112:0/2007986838 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:23:33.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.690 7f61e9ba2700 1 -- 172.21.2.112:0/2007986838 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f61e40f20e0 msgr2=0x7f61e40f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:33.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.690 7f61e9ba2700 1 --2- 172.21.2.112:0/2007986838 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f61e40f20e0 0x7f61e40f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:33.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.690 7f61e9ba2700 1 -- 172.21.2.112:0/2007986838 >> v1:172.21.2.120:6789/0 conn(0x7f61e40ef610 legacy=0x7f61e40efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:33.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.690 7f61e9ba2700 1 -- 172.21.2.112:0/2007986838 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61e40e52e0 con 0x7f61e41b0470 2021-02-10T11:23:33.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.691 7f61e9ba2700 1 --2- 172.21.2.112:0/2007986838 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f61e41b0470 0x7f61e41b0870 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7f61d8006650 tx=0x7f61d8009a80).ready entity=mon.1 client_cookie=d1b569ec73a80b76 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:33.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.691 7f61e8ba0700 1 -- 172.21.2.112:0/2007986838 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f61d800c0c0 con 0x7f61e41b0470 2021-02-10T11:23:33.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.691 7f61e8ba0700 1 -- 172.21.2.112:0/2007986838 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61d800c340 con 0x7f61e41b0470 2021-02-10T11:23:33.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.691 7f61e8ba0700 1 -- 172.21.2.112:0/2007986838 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f61d800c860 con 0x7f61e41b0470 2021-02-10T11:23:33.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.691 7f61ebd86700 1 -- 172.21.2.112:0/2007986838 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f61e41b0470 msgr2=0x7f61e41b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:33.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.691 7f61ebd86700 1 --2- 172.21.2.112:0/2007986838 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f61e41b0470 0x7f61e41b0870 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7f61d8006650 tx=0x7f61d8009a80).stop 2021-02-10T11:23:33.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.692 7f61ebd86700 1 -- 172.21.2.112:0/2007986838 shutdown_connections 2021-02-10T11:23:33.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.692 7f61ebd86700 1 --2- 172.21.2.112:0/2007986838 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f61e40f20e0 0x7f61e40f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:33.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.692 7f61ebd86700 1 --2- 172.21.2.112:0/2007986838 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f61e41b0470 0x7f61e41b0870 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:33.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.692 7f61ebd86700 1 -- 172.21.2.112:0/2007986838 >> 172.21.2.112:0/2007986838 conn(0x7f61e40ce3e0 msgr2=0x7f61e40e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:33.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.692 7f61ebd86700 1 -- 172.21.2.112:0/2007986838 shutdown_connections 2021-02-10T11:23:33.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.692 7f61ebd86700 1 -- 172.21.2.112:0/2007986838 wait complete. 2021-02-10T11:23:33.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.707 7f61ebd86700 1 Processor -- start 2021-02-10T11:23:33.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.707 7f61ebd86700 1 -- start start 2021-02-10T11:23:33.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.707 7f61ebd86700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f61e4190920 0x7f61e40b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:33.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.707 7f61ebd86700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f61e40b8200 0x7f61e40b8600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:33.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.707 7f61ebd86700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f61e40c1d80 0x7f61e40c2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:33.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.707 7f61ebd86700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61e4190d20 con 0x7f61e4190920 2021-02-10T11:23:33.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.707 7f61ebd86700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61e40b8c20 con 0x7f61e40c1d80 2021-02-10T11:23:33.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.707 7f61ebd86700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61e40c27a0 con 0x7f61e40b8200 2021-02-10T11:23:33.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.707 7f61e9ba2700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f61e4190920 0x7f61e40b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:33.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.707 7f61e9ba2700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f61e4190920 0x7f61e40b7be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:45324/0 (socket says 172.21.2.112:45324) 2021-02-10T11:23:33.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.707 7f61e9ba2700 1 -- 172.21.2.112:0/240306605 learned_addr learned my addr 172.21.2.112:0/240306605 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:23:33.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.708 7f61e9ba2700 1 -- 172.21.2.112:0/240306605 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f61e40b8200 msgr2=0x7f61e40b8600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:33.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.708 7f61e9ba2700 1 --2- 172.21.2.112:0/240306605 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f61e40b8200 0x7f61e40b8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:33.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.708 7f61e9ba2700 1 -- 172.21.2.112:0/240306605 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f61e40c1d80 msgr2=0x7f61e40c2180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:33.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.708 7f61e9ba2700 1 --2- 172.21.2.112:0/240306605 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f61e40c1d80 0x7f61e40c2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:33.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.708 7f61e9ba2700 1 -- 172.21.2.112:0/240306605 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61e40c29c0 con 0x7f61e4190920 2021-02-10T11:23:33.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.708 7f61e9ba2700 1 --2- 172.21.2.112:0/240306605 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f61e4190920 0x7f61e40b7be0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 rx=0x7f61d8009630 tx=0x7f61d800b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:33.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.708 7f61d67fc700 1 -- 172.21.2.112:0/240306605 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f61d80093a0 con 0x7f61e4190920 2021-02-10T11:23:33.704 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.708 7f61d67fc700 1 -- 172.21.2.112:0/240306605 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61d800bd70 con 0x7f61e4190920 2021-02-10T11:23:33.704 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.708 7f61ebd86700 1 -- 172.21.2.112:0/240306605 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61e40c29c0 con 0x7f61e4190920 2021-02-10T11:23:33.704 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.708 7f61d67fc700 1 -- 172.21.2.112:0/240306605 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f61d80054d0 con 0x7f61e4190920 2021-02-10T11:23:33.704 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.709 7f61ebd86700 1 -- 172.21.2.112:0/240306605 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61e40c29c0 con 0x7f61e4190920 2021-02-10T11:23:33.704 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.709 7f61d67fc700 1 -- 172.21.2.112:0/240306605 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f61d800a000 con 0x7f61e4190920 2021-02-10T11:23:33.704 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.709 7f61d67fc700 1 --2- 172.21.2.112:0/240306605 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f61cc018730 0x7f61cc01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:33.704 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.709 7f61d67fc700 1 -- 172.21.2.112:0/240306605 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f61d8032a10 con 0x7f61e4190920 2021-02-10T11:23:33.704 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.709 7f61e93a1700 1 --2- 172.21.2.112:0/240306605 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f61cc018730 0x7f61cc01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:33.704 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.710 7f61e93a1700 1 --2- 172.21.2.112:0/240306605 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f61cc018730 0x7f61cc01ac10 crc :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:33.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.739 7f61ebd86700 1 -- 172.21.2.112:0/240306605 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61e4105ae0 con 0x7f61e4190920 2021-02-10T11:23:33.733 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.741 7f61d67fc700 1 -- 172.21.2.112:0/240306605 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f61d800a000 con 0x7f61e4190920 2021-02-10T11:23:33.862 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.870 7f61ebd86700 1 -- 172.21.2.112:0/240306605 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348622", "target": ["mgr", ""]}) v1 -- 0x7f61e40c32c0 con 0x7f61cc018730 2021-02-10T11:23:33.865 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.874 7f61d67fc700 1 -- 172.21.2.112:0/240306605 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f61e0001530 con 0x7f61cc018730 2021-02-10T11:23:33.869 INFO:teuthology.orchestra.run.gibba012.stdout:/volumes/_nogroup/subvolume_0000000000348622/440d021a-393a-4104-a079-e05a97ce025b 2021-02-10T11:23:33.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.879 7f61ebd86700 1 -- 172.21.2.112:0/240306605 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f61cc018730 msgr2=0x7f61cc01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:33.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.879 7f61ebd86700 1 --2- 172.21.2.112:0/240306605 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f61cc018730 0x7f61cc01ac10 crc :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:33.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.879 7f61ebd86700 1 -- 172.21.2.112:0/240306605 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f61e4190920 msgr2=0x7f61e40b7be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:33.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.879 7f61ebd86700 1 --2- 172.21.2.112:0/240306605 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f61e4190920 0x7f61e40b7be0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 rx=0x7f61d8009630 tx=0x7f61d800b980).stop 2021-02-10T11:23:33.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.879 7f61ebd86700 1 -- 172.21.2.112:0/240306605 shutdown_connections 2021-02-10T11:23:33.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.879 7f61ebd86700 1 --2- 172.21.2.112:0/240306605 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f61cc018730 0x7f61cc01ac10 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:33.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.879 7f61ebd86700 1 --2- 172.21.2.112:0/240306605 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f61e40c1d80 0x7f61e40c2180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:33.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.879 7f61ebd86700 1 --2- 172.21.2.112:0/240306605 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f61e40b8200 0x7f61e40b8600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:33.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.879 7f61ebd86700 1 --2- 172.21.2.112:0/240306605 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f61e4190920 0x7f61e40b7be0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:33.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.879 7f61ebd86700 1 -- 172.21.2.112:0/240306605 >> 172.21.2.112:0/240306605 conn(0x7f61e4094e10 msgr2=0x7f61e40a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:33.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.880 7f61ebd86700 1 -- 172.21.2.112:0/240306605 shutdown_connections 2021-02-10T11:23:33.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:33.880 7f61ebd86700 1 -- 172.21.2.112:0/240306605 wait complete. 2021-02-10T11:23:33.880 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000348622 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000348622/440d021a-393a-4104-a079-e05a97ce025b 2021-02-10T11:23:33.880 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:33.880 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:23:33.880 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:23:34.223 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:23:34.223 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:23:34.223 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348622/440d021a-393a-4104-a079-e05a97ce025b/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:23:34.601 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:23:34.601 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:23:34.601 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0243953 s, 43.0 MB/s 2021-02-10T11:23:34.602 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:34.602 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:23:34.603 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:23:34.939 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:23:34.939 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:23:34.939 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348622/440d021a-393a-4104-a079-e05a97ce025b/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:23:35.314 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:23:35.314 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:23:35.314 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0248153 s, 42.3 MB/s 2021-02-10T11:23:35.315 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:35.316 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:23:35.316 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:23:35.651 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:23:35.651 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:23:35.651 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348622/440d021a-393a-4104-a079-e05a97ce025b/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:23:36.025 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:23:36.025 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:23:36.026 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0232086 s, 45.2 MB/s 2021-02-10T11:23:36.027 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:36.027 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:23:36.027 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:23:36.363 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:23:36.363 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:23:36.363 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348622/440d021a-393a-4104-a079-e05a97ce025b/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:23:36.737 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:23:36.737 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:23:36.738 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0224594 s, 46.7 MB/s 2021-02-10T11:23:36.739 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:36.739 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:23:36.740 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:23:37.075 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:23:37.075 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:23:37.075 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348622/440d021a-393a-4104-a079-e05a97ce025b/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:23:37.451 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:23:37.451 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:23:37.451 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0242779 s, 43.2 MB/s 2021-02-10T11:23:37.452 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:37.452 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:23:37.452 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:23:37.787 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:23:37.787 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:23:37.787 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348622/440d021a-393a-4104-a079-e05a97ce025b/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:23:38.162 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:23:38.162 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:23:38.162 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.022986 s, 45.6 MB/s 2021-02-10T11:23:38.163 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:38.164 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:23:38.164 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:23:38.461 DEBUG:teuthology.orchestra.run.gibba012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:23:38.465 DEBUG:teuthology.orchestra.run.gibba020:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:23:38.497 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:23:38.498 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:23:38.498 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348622/440d021a-393a-4104-a079-e05a97ce025b/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:23:38.839 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:23:38.839 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:23:38.840 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0262944 s, 39.9 MB/s 2021-02-10T11:23:38.841 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:38.841 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:23:38.841 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:23:39.178 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:23:39.179 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:23:39.179 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348622/440d021a-393a-4104-a079-e05a97ce025b/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:23:39.551 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:23:39.551 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:23:39.551 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0259015 s, 40.5 MB/s 2021-02-10T11:23:39.552 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:39.552 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:23:39.553 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:23:39.889 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:23:39.889 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:23:39.889 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348622/440d021a-393a-4104-a079-e05a97ce025b/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:23:40.266 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:23:40.266 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:23:40.266 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0254834 s, 41.1 MB/s 2021-02-10T11:23:40.267 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:40.267 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:23:40.267 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:23:40.608 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:23:40.608 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:23:40.608 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348622/440d021a-393a-4104-a079-e05a97ce025b/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:23:40.985 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:23:40.987 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:23:40.987 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0253599 s, 41.3 MB/s 2021-02-10T11:23:40.988 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000348623 2021-02-10T11:23:41.419 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:41.410 7f7e3170a700 1 Processor -- start 2021-02-10T11:23:41.419 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:41.410 7f7e3170a700 1 -- start start 2021-02-10T11:23:41.419 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:41.411 7f7e3170a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7e2c0f05f0 0x7f7e2c0f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:41.419 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:41.411 7f7e3170a700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e2c1b0c70 con 0x7f7e2c1b0870 2021-02-10T11:23:41.420 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:41.411 7f7e3170a700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e2c0f0b50 con 0x7f7e2c1b0470 2021-02-10T11:23:41.420 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:41.411 7f7e3170a700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e2c0e5ea0 con 0x7f7e2c0f05f0 2021-02-10T11:23:41.420 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:41.411 7f7e2aabe700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f7e2c1b0870 0x7f7e2c0efe00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:44794/0 (socket says 172.21.2.112:44794) 2021-02-10T11:23:41.420 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:41.411 7f7e2aabe700 1 -- 172.21.2.112:0/603798259 learned_addr learned my addr 172.21.2.112:0/603798259 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:23:41.420 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:41.411 7f7e2bac0700 1 --2- 172.21.2.112:0/603798259 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7e2c0f05f0 0x7f7e2c0f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:41.420 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:41.411 7f7e2a2bd700 1 -- 172.21.2.112:0/603798259 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 183713706 0 0) 0x7f7e1c0010b0 con 0x7f7e2c1b0870 2021-02-10T11:23:41.420 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:41.411 7f7e2a2bd700 1 -- 172.21.2.112:0/603798259 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e18002fe0 con 0x7f7e2c1b0870 2021-02-10T11:23:41.420 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:41.412 7f7e2bac0700 1 -- 172.21.2.112:0/603798259 >> v1:172.21.2.120:6789/0 conn(0x7f7e2c1b0470 legacy=0x7f7e2c0ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:41.420 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:41.412 7f7e2bac0700 1 -- 172.21.2.112:0/603798259 >> v1:172.21.2.112:6789/0 conn(0x7f7e2c1b0870 legacy=0x7f7e2c0efe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:41.420 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:41.412 7f7e2bac0700 1 -- 172.21.2.112:0/603798259 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e2c0e52e0 con 0x7f7e2c0f05f0 2021-02-10T11:23:41.420 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:41.412 7f7e2bac0700 1 --2- 172.21.2.112:0/603798259 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7e2c0f05f0 0x7f7e2c0f20e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f7e24006e50 tx=0x7f7e24009df0).ready entity=mon.2 client_cookie=451d5dac816da6db server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:41.420 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:41.412 7f7e2a2bd700 1 -- 172.21.2.112:0/603798259 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7e2400b510 con 0x7f7e2c0f05f0 2021-02-10T11:23:41.421 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:41.412 7f7e2a2bd700 1 -- 172.21.2.112:0/603798259 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e24003030 con 0x7f7e2c0f05f0 2021-02-10T11:23:41.421 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:41.412 7f7e2a2bd700 1 -- 172.21.2.112:0/603798259 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7e2400b7d0 con 0x7f7e2c0f05f0 2021-02-10T11:23:41.421 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:41.412 7f7e3170a700 1 -- 172.21.2.112:0/603798259 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7e2c0f05f0 msgr2=0x7f7e2c0f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:41.421 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:41.412 7f7e3170a700 1 --2- 172.21.2.112:0/603798259 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7e2c0f05f0 0x7f7e2c0f20e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f7e24006e50 tx=0x7f7e24009df0).stop 2021-02-10T11:23:41.421 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:41.413 7f7e3170a700 1 -- 172.21.2.112:0/603798259 shutdown_connections 2021-02-10T11:23:41.421 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:41.413 7f7e3170a700 1 --2- 172.21.2.112:0/603798259 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7e2c0f05f0 0x7f7e2c0f20e0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:41.421 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:41.413 7f7e3170a700 1 -- 172.21.2.112:0/603798259 >> 172.21.2.112:0/603798259 conn(0x7f7e2c0ce3e0 msgr2=0x7f7e2c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:41.421 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:41.413 7f7e3170a700 1 -- 172.21.2.112:0/603798259 shutdown_connections 2021-02-10T11:23:41.421 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:41.413 7f7e3170a700 1 -- 172.21.2.112:0/603798259 wait complete. 2021-02-10T11:23:41.423 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:41.428 7f7e3170a700 1 Processor -- start 2021-02-10T11:23:41.423 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:41.428 7f7e3170a700 1 -- start start 2021-02-10T11:23:41.423 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:41.428 7f7e3170a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7e2c094fb0 0x7f7e2c09e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:41.423 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:41.428 7f7e3170a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7e2c0c2be0 0x7f7e2c0c2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:41.423 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:41.428 7f7e3170a700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7e2c0c3600 0x7f7e2c0a9c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:41.423 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:41.428 7f7e3170a700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e2c09ebf0 con 0x7f7e2c0c2be0 2021-02-10T11:23:41.423 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:41.428 7f7e3170a700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e2c0aa2a0 con 0x7f7e2c0c3600 2021-02-10T11:23:41.423 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:41.428 7f7e3170a700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e2c0aa4c0 con 0x7f7e2c094fb0 2021-02-10T11:23:41.423 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:41.428 7f7e2aabe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7e2c0c2be0 0x7f7e2c0c2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:41.423 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:41.428 7f7e2aabe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7e2c0c2be0 0x7f7e2c0c2fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:45340/0 (socket says 172.21.2.112:45340) 2021-02-10T11:23:41.423 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:41.428 7f7e2aabe700 1 -- 172.21.2.112:0/2742099969 learned_addr learned my addr 172.21.2.112:0/2742099969 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:23:41.423 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:41.429 7f7e2b2bf700 1 --2- 172.21.2.112:0/2742099969 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7e2c094fb0 0x7f7e2c09e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:41.424 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:41.429 7f7e2bac0700 1 --2- 172.21.2.112:0/2742099969 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7e2c0c3600 0x7f7e2c0a9c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:41.424 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:41.429 7f7e2aabe700 1 -- 172.21.2.112:0/2742099969 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7e2c094fb0 msgr2=0x7f7e2c09e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:41.424 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:41.429 7f7e2aabe700 1 --2- 172.21.2.112:0/2742099969 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7e2c094fb0 0x7f7e2c09e5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:41.424 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:41.429 7f7e2aabe700 1 -- 172.21.2.112:0/2742099969 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7e2c0c3600 msgr2=0x7f7e2c0a9c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:41.424 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:41.429 7f7e2aabe700 1 --2- 172.21.2.112:0/2742099969 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7e2c0c3600 0x7f7e2c0a9c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:41.424 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:41.429 7f7e2aabe700 1 -- 172.21.2.112:0/2742099969 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e2c0aa6e0 con 0x7f7e2c0c2be0 2021-02-10T11:23:41.424 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:41.429 7f7e2aabe700 1 --2- 172.21.2.112:0/2742099969 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7e2c0c2be0 0x7f7e2c0c2fe0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 rx=0x7f7e1c007650 tx=0x7f7e1c00a8e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:41.424 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:41.429 7f7e23fff700 1 -- 172.21.2.112:0/2742099969 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7e1c00acd0 con 0x7f7e2c0c2be0 2021-02-10T11:23:41.424 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:41.429 7f7e23fff700 1 -- 172.21.2.112:0/2742099969 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e1c004350 con 0x7f7e2c0c2be0 2021-02-10T11:23:41.424 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:41.429 7f7e23fff700 1 -- 172.21.2.112:0/2742099969 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7e1c004870 con 0x7f7e2c0c2be0 2021-02-10T11:23:41.424 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:41.429 7f7e3170a700 1 -- 172.21.2.112:0/2742099969 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e2c0aa6e0 con 0x7f7e2c0c2be0 2021-02-10T11:23:41.424 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:41.429 7f7e3170a700 1 -- 172.21.2.112:0/2742099969 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e2c0aac70 con 0x7f7e2c0c2be0 2021-02-10T11:23:41.424 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:41.430 7f7e23fff700 1 -- 172.21.2.112:0/2742099969 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7e1c004530 con 0x7f7e2c0c2be0 2021-02-10T11:23:41.424 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:41.430 7f7e23fff700 1 --2- 172.21.2.112:0/2742099969 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7e100186c0 0x7f7e1001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:41.425 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:41.430 7f7e23fff700 1 -- 172.21.2.112:0/2742099969 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f7e1c032590 con 0x7f7e2c0c2be0 2021-02-10T11:23:41.425 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:41.430 7f7e2b2bf700 1 --2- 172.21.2.112:0/2742099969 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7e100186c0 0x7f7e1001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:41.425 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:41.431 7f7e2b2bf700 1 --2- 172.21.2.112:0/2742099969 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7e100186c0 0x7f7e1001abc0 crc :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:41.453 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:41.461 7f7e3170a700 1 -- 172.21.2.112:0/2742099969 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e2c0a9750 con 0x7f7e2c0c2be0 2021-02-10T11:23:41.455 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:41.464 7f7e23fff700 1 -- 172.21.2.112:0/2742099969 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7e1c0019e0 con 0x7f7e2c0c2be0 2021-02-10T11:23:41.582 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:41.591 7f7e3170a700 1 -- 172.21.2.112:0/2742099969 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348623", "target": ["mgr", ""]}) v1 -- 0x7f7e2c0aac70 con 0x7f7e100186c0 2021-02-10T11:23:41.609 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:41.617 7f7e23fff700 1 -- 172.21.2.112:0/2742099969 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f7e140009f0 con 0x7f7e100186c0 2021-02-10T11:23:41.616 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:41.623 7f7e3170a700 1 -- 172.21.2.112:0/2742099969 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7e100186c0 msgr2=0x7f7e1001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:41.616 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:41.623 7f7e3170a700 1 --2- 172.21.2.112:0/2742099969 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7e100186c0 0x7f7e1001abc0 crc :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:41.616 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:41.623 7f7e3170a700 1 -- 172.21.2.112:0/2742099969 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7e2c0c2be0 msgr2=0x7f7e2c0c2fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:41.616 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:41.623 7f7e3170a700 1 --2- 172.21.2.112:0/2742099969 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7e2c0c2be0 0x7f7e2c0c2fe0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 rx=0x7f7e1c007650 tx=0x7f7e1c00a8e0).stop 2021-02-10T11:23:41.616 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:41.623 7f7e3170a700 1 -- 172.21.2.112:0/2742099969 shutdown_connections 2021-02-10T11:23:41.616 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:41.623 7f7e3170a700 1 --2- 172.21.2.112:0/2742099969 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7e100186c0 0x7f7e1001abc0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:41.616 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:41.623 7f7e3170a700 1 --2- 172.21.2.112:0/2742099969 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7e2c0c3600 0x7f7e2c0a9c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:41.616 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:41.623 7f7e3170a700 1 --2- 172.21.2.112:0/2742099969 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7e2c0c2be0 0x7f7e2c0c2fe0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:41.616 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:41.623 7f7e3170a700 1 --2- 172.21.2.112:0/2742099969 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7e2c094fb0 0x7f7e2c09e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:41.616 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:41.623 7f7e3170a700 1 -- 172.21.2.112:0/2742099969 >> 172.21.2.112:0/2742099969 conn(0x7f7e2c09f630 msgr2=0x7f7e2c0a0130 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:41.617 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:41.624 7f7e3170a700 1 -- 172.21.2.112:0/2742099969 shutdown_connections 2021-02-10T11:23:41.617 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:41.624 7f7e3170a700 1 -- 172.21.2.112:0/2742099969 wait complete. 2021-02-10T11:23:41.624 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000348623 2021-02-10T11:23:42.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:42.049 7f54068b2700 1 Processor -- start 2021-02-10T11:23:42.058 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:42.049 7f54068b2700 1 -- start start 2021-02-10T11:23:42.058 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:42.050 7f54068b2700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f54001b0870 0x7f54000f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:42.058 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:42.050 7f54068b2700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f54000efe00 0x7f54000f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:42.058 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:42.050 7f54068b2700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54001b0c70 con 0x7f54001b0470 2021-02-10T11:23:42.058 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:42.050 7f54068b2700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54000f0b50 con 0x7f54000efe00 2021-02-10T11:23:42.058 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:42.050 7f54068b2700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54000f0820 con 0x7f54001b0870 2021-02-10T11:23:42.058 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:42.050 7f53fffff700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f54001b0470 0x7f54000ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:44808/0 (socket says 172.21.2.112:44808) 2021-02-10T11:23:42.058 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:42.050 7f53fffff700 1 -- 172.21.2.112:0/56310540 learned_addr learned my addr 172.21.2.112:0/56310540 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:23:42.058 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:42.050 7f5404ecf700 1 --2- 172.21.2.112:0/56310540 >> v2:172.21.2.120:3300/0 conn(0x7f54000efe00 0x7f54000f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:42.058 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:42.050 7f53ff7fe700 1 --2- 172.21.2.112:0/56310540 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f54001b0870 0x7f54000f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:42.058 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:42.050 7f53feffd700 1 -- 172.21.2.112:0/56310540 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3843043460 0 0) 0x7f53f0001180 con 0x7f54001b0470 2021-02-10T11:23:42.059 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:42.050 7f53feffd700 1 -- 172.21.2.112:0/56310540 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53e8002fe0 con 0x7f54001b0470 2021-02-10T11:23:42.059 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:42.050 7f5404ecf700 1 -- 172.21.2.112:0/56310540 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f54001b0870 msgr2=0x7f54000f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:42.059 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:42.050 7f5404ecf700 1 --2- 172.21.2.112:0/56310540 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f54001b0870 0x7f54000f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:42.059 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:42.050 7f5404ecf700 1 -- 172.21.2.112:0/56310540 >> v1:172.21.2.112:6789/0 conn(0x7f54001b0470 legacy=0x7f54000ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:42.059 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:42.050 7f5404ecf700 1 -- 172.21.2.112:0/56310540 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54000e52e0 con 0x7f54000efe00 2021-02-10T11:23:42.059 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:42.051 7f5404ecf700 1 --2- 172.21.2.112:0/56310540 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f54000efe00 0x7f54000f0200 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7f53f8006e50 tx=0x7f53f8009e70).ready entity=mon.1 client_cookie=8d23f0b0b09040d6 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:42.059 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:42.051 7f53feffd700 1 -- 172.21.2.112:0/56310540 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f53f800b510 con 0x7f54000efe00 2021-02-10T11:23:42.059 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:42.051 7f53feffd700 1 -- 172.21.2.112:0/56310540 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53f8006790 con 0x7f54000efe00 2021-02-10T11:23:42.059 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:42.051 7f53feffd700 1 -- 172.21.2.112:0/56310540 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f53f800b9d0 con 0x7f54000efe00 2021-02-10T11:23:42.059 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:42.051 7f54068b2700 1 -- 172.21.2.112:0/56310540 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f54000efe00 msgr2=0x7f54000f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:42.059 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:42.051 7f54068b2700 1 --2- 172.21.2.112:0/56310540 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f54000efe00 0x7f54000f0200 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7f53f8006e50 tx=0x7f53f8009e70).stop 2021-02-10T11:23:42.059 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:42.052 7f54068b2700 1 -- 172.21.2.112:0/56310540 shutdown_connections 2021-02-10T11:23:42.059 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:42.052 7f54068b2700 1 --2- 172.21.2.112:0/56310540 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f54000efe00 0x7f54000f0200 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:42.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:42.052 7f54068b2700 1 --2- 172.21.2.112:0/56310540 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f54001b0870 0x7f54000f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:42.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:42.052 7f54068b2700 1 -- 172.21.2.112:0/56310540 >> 172.21.2.112:0/56310540 conn(0x7f54000ce3e0 msgr2=0x7f54000e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:42.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:42.052 7f54068b2700 1 -- 172.21.2.112:0/56310540 shutdown_connections 2021-02-10T11:23:42.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:42.052 7f54068b2700 1 -- 172.21.2.112:0/56310540 wait complete. 2021-02-10T11:23:42.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:42.066 7f54068b2700 1 Processor -- start 2021-02-10T11:23:42.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:42.066 7f54068b2700 1 -- start start 2021-02-10T11:23:42.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:42.067 7f54068b2700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5400190920 0x7f54000b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:42.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:42.067 7f54068b2700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f54000b8200 0x7f54000b8600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:42.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:42.067 7f54068b2700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f54000c1d80 0x7f54000c2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:42.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:42.067 7f54068b2700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5400190da0 con 0x7f54000c1d80 2021-02-10T11:23:42.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:42.067 7f54068b2700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54000b8c20 con 0x7f54000b8200 2021-02-10T11:23:42.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:42.067 7f54068b2700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54000b8e10 con 0x7f5400190920 2021-02-10T11:23:42.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:42.067 7f5404ecf700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f54000c1d80 0x7f54000c2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:42.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:42.067 7f5404ecf700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f54000c1d80 0x7f54000c2180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:45352/0 (socket says 172.21.2.112:45352) 2021-02-10T11:23:42.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:42.067 7f5404ecf700 1 -- 172.21.2.112:0/4217810327 learned_addr learned my addr 172.21.2.112:0/4217810327 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:23:42.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:42.067 7f53fffff700 1 --2- 172.21.2.112:0/4217810327 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5400190920 0x7f54000b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:42.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:42.067 7f53ff7fe700 1 --2- 172.21.2.112:0/4217810327 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f54000b8200 0x7f54000b8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:42.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:42.067 7f5404ecf700 1 -- 172.21.2.112:0/4217810327 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5400190920 msgr2=0x7f54000b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:42.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:42.067 7f5404ecf700 1 --2- 172.21.2.112:0/4217810327 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5400190920 0x7f54000b7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:42.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:42.067 7f5404ecf700 1 -- 172.21.2.112:0/4217810327 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f54000b8200 msgr2=0x7f54000b8600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:42.063 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:42.067 7f5404ecf700 1 --2- 172.21.2.112:0/4217810327 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f54000b8200 0x7f54000b8600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:42.063 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:42.067 7f5404ecf700 1 -- 172.21.2.112:0/4217810327 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54000c27d0 con 0x7f54000c1d80 2021-02-10T11:23:42.063 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:42.067 7f5404ecf700 1 --2- 172.21.2.112:0/4217810327 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f54000c1d80 0x7f54000c2180 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 rx=0x7f53f80099b0 tx=0x7f53f800bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:42.063 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:42.067 7f53fcff9700 1 -- 172.21.2.112:0/4217810327 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f53f8002030 con 0x7f54000c1d80 2021-02-10T11:23:42.063 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:42.068 7f53fcff9700 1 -- 172.21.2.112:0/4217810327 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53f8002370 con 0x7f54000c1d80 2021-02-10T11:23:42.063 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:42.068 7f53fcff9700 1 -- 172.21.2.112:0/4217810327 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f53f80027c0 con 0x7f54000c1d80 2021-02-10T11:23:42.063 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:42.068 7f54068b2700 1 -- 172.21.2.112:0/4217810327 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54000c27d0 con 0x7f54000c1d80 2021-02-10T11:23:42.063 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:42.068 7f54068b2700 1 -- 172.21.2.112:0/4217810327 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54000c27d0 con 0x7f54000c1d80 2021-02-10T11:23:42.063 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:42.068 7f53fcff9700 1 -- 172.21.2.112:0/4217810327 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f53f8002550 con 0x7f54000c1d80 2021-02-10T11:23:42.063 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:42.068 7f53fcff9700 1 --2- 172.21.2.112:0/4217810327 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f53e0018730 0x7f53e001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:42.063 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:42.068 7f53fcff9700 1 -- 172.21.2.112:0/4217810327 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f53f80329e0 con 0x7f54000c1d80 2021-02-10T11:23:42.063 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:42.068 7f53fffff700 1 --2- 172.21.2.112:0/4217810327 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f53e0018730 0x7f53e001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:42.063 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:42.069 7f53fffff700 1 --2- 172.21.2.112:0/4217810327 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f53e0018730 0x7f53e001ac10 crc :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:42.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:42.100 7f54068b2700 1 -- 172.21.2.112:0/4217810327 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5400105ae0 con 0x7f54000c1d80 2021-02-10T11:23:42.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:42.102 7f53fcff9700 1 -- 172.21.2.112:0/4217810327 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f53f800be10 con 0x7f54000c1d80 2021-02-10T11:23:42.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:42.230 7f54068b2700 1 -- 172.21.2.112:0/4217810327 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348623", "target": ["mgr", ""]}) v1 -- 0x7f54000f0b50 con 0x7f53e0018730 2021-02-10T11:23:42.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:42.234 7f53fcff9700 1 -- 172.21.2.112:0/4217810327 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f53f0007770 con 0x7f53e0018730 2021-02-10T11:23:42.229 INFO:teuthology.orchestra.run.gibba012.stdout:/volumes/_nogroup/subvolume_0000000000348623/2f45612c-cdee-4ed0-8668-004c82433c05 2021-02-10T11:23:42.231 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:42.239 7f54068b2700 1 -- 172.21.2.112:0/4217810327 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f53e0018730 msgr2=0x7f53e001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:42.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:42.239 7f54068b2700 1 --2- 172.21.2.112:0/4217810327 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f53e0018730 0x7f53e001ac10 crc :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:42.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:42.239 7f54068b2700 1 -- 172.21.2.112:0/4217810327 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f54000c1d80 msgr2=0x7f54000c2180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:42.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:42.239 7f54068b2700 1 --2- 172.21.2.112:0/4217810327 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f54000c1d80 0x7f54000c2180 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 rx=0x7f53f80099b0 tx=0x7f53f800bcd0).stop 2021-02-10T11:23:42.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:42.239 7f54068b2700 1 -- 172.21.2.112:0/4217810327 shutdown_connections 2021-02-10T11:23:42.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:42.239 7f54068b2700 1 --2- 172.21.2.112:0/4217810327 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f53e0018730 0x7f53e001ac10 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:42.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:42.239 7f54068b2700 1 --2- 172.21.2.112:0/4217810327 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f54000b8200 0x7f54000b8600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:42.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:42.239 7f54068b2700 1 --2- 172.21.2.112:0/4217810327 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f54000c1d80 0x7f54000c2180 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:42.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:42.239 7f54068b2700 1 --2- 172.21.2.112:0/4217810327 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5400190920 0x7f54000b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:42.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:42.240 7f54068b2700 1 -- 172.21.2.112:0/4217810327 >> 172.21.2.112:0/4217810327 conn(0x7f5400094e10 msgr2=0x7f54000a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:42.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:42.240 7f54068b2700 1 -- 172.21.2.112:0/4217810327 shutdown_connections 2021-02-10T11:23:42.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:42.240 7f54068b2700 1 -- 172.21.2.112:0/4217810327 wait complete. 2021-02-10T11:23:42.240 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000348623 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000348623/2f45612c-cdee-4ed0-8668-004c82433c05 2021-02-10T11:23:42.240 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:42.240 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:23:42.240 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:23:42.576 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:23:42.576 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:23:42.576 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348623/2f45612c-cdee-4ed0-8668-004c82433c05/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:23:42.955 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:23:42.955 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:23:42.955 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0232935 s, 45.0 MB/s 2021-02-10T11:23:42.956 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:42.956 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:23:42.956 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:23:43.292 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:23:43.292 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:23:43.293 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348623/2f45612c-cdee-4ed0-8668-004c82433c05/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:23:43.668 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:23:43.668 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:23:43.668 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0242914 s, 43.2 MB/s 2021-02-10T11:23:43.669 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:43.669 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:23:43.669 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:23:44.005 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:23:44.006 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:23:44.006 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348623/2f45612c-cdee-4ed0-8668-004c82433c05/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:23:44.385 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:23:44.385 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:23:44.385 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.023287 s, 45.0 MB/s 2021-02-10T11:23:44.386 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:44.386 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:23:44.386 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:23:44.723 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:23:44.724 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:23:44.724 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348623/2f45612c-cdee-4ed0-8668-004c82433c05/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:23:45.102 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:23:45.102 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:23:45.102 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0255164 s, 41.1 MB/s 2021-02-10T11:23:45.104 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:45.104 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:23:45.104 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:23:45.440 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:23:45.440 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:23:45.440 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348623/2f45612c-cdee-4ed0-8668-004c82433c05/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:23:45.815 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:23:45.815 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:23:45.815 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0246618 s, 42.5 MB/s 2021-02-10T11:23:45.816 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:45.816 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:23:45.816 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:23:46.153 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:23:46.153 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:23:46.153 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348623/2f45612c-cdee-4ed0-8668-004c82433c05/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:23:46.530 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:23:46.531 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:23:46.531 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0262658 s, 39.9 MB/s 2021-02-10T11:23:46.532 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:46.532 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:23:46.532 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:23:46.867 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:23:46.867 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:23:46.867 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348623/2f45612c-cdee-4ed0-8668-004c82433c05/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:23:47.243 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:23:47.243 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:23:47.244 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0241653 s, 43.4 MB/s 2021-02-10T11:23:47.244 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:47.245 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:23:47.245 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:23:47.580 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:23:47.581 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:23:47.581 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348623/2f45612c-cdee-4ed0-8668-004c82433c05/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:23:47.955 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:23:47.956 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:23:47.956 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.023422 s, 44.8 MB/s 2021-02-10T11:23:47.957 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:47.957 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:23:47.957 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:23:48.293 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:23:48.294 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:23:48.294 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348623/2f45612c-cdee-4ed0-8668-004c82433c05/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:23:48.670 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:23:48.670 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:23:48.670 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0252651 s, 41.5 MB/s 2021-02-10T11:23:48.671 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:48.671 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:23:48.671 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:23:49.007 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:23:49.007 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:23:49.007 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348623/2f45612c-cdee-4ed0-8668-004c82433c05/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:23:49.382 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:23:49.382 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:23:49.382 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0256225 s, 40.9 MB/s 2021-02-10T11:23:49.383 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000348624 2021-02-10T11:23:49.816 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:49.806 7f7788f31700 1 Processor -- start 2021-02-10T11:23:49.816 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:49.806 7f7788f31700 1 -- start start 2021-02-10T11:23:49.816 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:49.806 7f7788f31700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f77841b0470 0x7f77841b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:49.816 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:49.806 7f7788f31700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f77840ef610 0x7f77840efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:49.816 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:49.806 7f7788f31700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77840f0b50 con 0x7f77840f0030 2021-02-10T11:23:49.816 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:49.806 7f7788f31700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77840e52e0 con 0x7f77841b0470 2021-02-10T11:23:49.816 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:49.806 7f7788f31700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77840e54d0 con 0x7f77840ef610 2021-02-10T11:23:49.816 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:49.806 7f7782b13700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f77841b0470 0x7f77841b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:49.816 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:49.806 7f7782b13700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f77841b0470 0x7f77841b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:45360/0 (socket says 172.21.2.112:45360) 2021-02-10T11:23:49.816 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:49.806 7f7782b13700 1 -- 172.21.2.112:0/3204084959 learned_addr learned my addr 172.21.2.112:0/3204084959 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:23:49.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:49.807 7f7782b13700 1 -- 172.21.2.112:0/3204084959 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f77840ef610 msgr2=0x7f77840efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:49.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:49.807 7f7782b13700 1 --2- 172.21.2.112:0/3204084959 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f77840ef610 0x7f77840efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:49.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:49.807 7f7782b13700 1 -- 172.21.2.112:0/3204084959 >> v1:172.21.2.120:6789/0 conn(0x7f77840f0030 legacy=0x7f77840f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:49.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:49.807 7f7782b13700 1 -- 172.21.2.112:0/3204084959 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77840f0430 con 0x7f77841b0470 2021-02-10T11:23:49.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:49.807 7f7782b13700 1 --2- 172.21.2.112:0/3204084959 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f77841b0470 0x7f77841b0870 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 rx=0x7f776c006700 tx=0x7f776c009b80).ready entity=mon.0 client_cookie=5201f67fb5b5da12 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:49.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:49.807 7f7781b11700 1 -- 172.21.2.112:0/3204084959 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f776c006340 con 0x7f77841b0470 2021-02-10T11:23:49.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:49.807 7f7781b11700 1 -- 172.21.2.112:0/3204084959 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f776c00c0c0 con 0x7f77841b0470 2021-02-10T11:23:49.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:49.807 7f7781b11700 1 -- 172.21.2.112:0/3204084959 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f776c00c5e0 con 0x7f77841b0470 2021-02-10T11:23:49.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:49.807 7f7788f31700 1 -- 172.21.2.112:0/3204084959 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f77841b0470 msgr2=0x7f77841b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:49.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:49.807 7f7788f31700 1 --2- 172.21.2.112:0/3204084959 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f77841b0470 0x7f77841b0870 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 rx=0x7f776c006700 tx=0x7f776c009b80).stop 2021-02-10T11:23:49.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:49.808 7f7788f31700 1 -- 172.21.2.112:0/3204084959 shutdown_connections 2021-02-10T11:23:49.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:49.808 7f7788f31700 1 --2- 172.21.2.112:0/3204084959 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f77840ef610 0x7f77840efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:49.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:49.808 7f7788f31700 1 --2- 172.21.2.112:0/3204084959 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f77841b0470 0x7f77841b0870 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:49.818 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:49.808 7f7788f31700 1 -- 172.21.2.112:0/3204084959 >> 172.21.2.112:0/3204084959 conn(0x7f77840ce3e0 msgr2=0x7f77840e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:49.818 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:49.808 7f7788f31700 1 -- 172.21.2.112:0/3204084959 shutdown_connections 2021-02-10T11:23:49.818 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:49.808 7f7788f31700 1 -- 172.21.2.112:0/3204084959 wait complete. 2021-02-10T11:23:49.818 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:49.823 7f7788f31700 1 Processor -- start 2021-02-10T11:23:49.818 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:49.823 7f7788f31700 1 -- start start 2021-02-10T11:23:49.818 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:49.823 7f7788f31700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f77840a6c90 0x7f77841924c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:49.818 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:49.823 7f7788f31700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7784192ae0 0x7f7784192ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:49.818 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:49.823 7f7788f31700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f77840919e0 0x7f7784091de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:49.818 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:49.823 7f7788f31700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f778419d2e0 con 0x7f7784192ae0 2021-02-10T11:23:49.818 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:49.823 7f7788f31700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77840914d0 con 0x7f77840a6c90 2021-02-10T11:23:49.818 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:49.823 7f7788f31700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77840a6820 con 0x7f77840919e0 2021-02-10T11:23:49.818 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:49.823 7f7782312700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7784192ae0 0x7f7784192ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:49.818 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:49.823 7f7782312700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7784192ae0 0x7f7784192ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:45368/0 (socket says 172.21.2.112:45368) 2021-02-10T11:23:49.819 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:49.823 7f7782312700 1 -- 172.21.2.112:0/588934906 learned_addr learned my addr 172.21.2.112:0/588934906 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:23:49.819 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:49.823 7f7782b13700 1 --2- 172.21.2.112:0/588934906 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f77840a6c90 0x7f77841924c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:49.819 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:49.823 7f7783314700 1 --2- 172.21.2.112:0/588934906 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f77840919e0 0x7f7784091de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:49.819 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:49.823 7f7782312700 1 -- 172.21.2.112:0/588934906 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f77840919e0 msgr2=0x7f7784091de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:49.819 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:49.823 7f7782312700 1 --2- 172.21.2.112:0/588934906 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f77840919e0 0x7f7784091de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:49.819 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:49.823 7f7782312700 1 -- 172.21.2.112:0/588934906 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f77840a6c90 msgr2=0x7f77841924c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:49.819 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:49.823 7f7782312700 1 --2- 172.21.2.112:0/588934906 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f77840a6c90 0x7f77841924c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:49.819 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:49.823 7f7782312700 1 -- 172.21.2.112:0/588934906 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7784092400 con 0x7f7784192ae0 2021-02-10T11:23:49.819 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:49.824 7f7782312700 1 --2- 172.21.2.112:0/588934906 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7784192ae0 0x7f7784192ee0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 rx=0x7f777400b530 tx=0x7f777400b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:49.819 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:49.824 7f777b7fe700 1 -- 172.21.2.112:0/588934906 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f777400bcc0 con 0x7f7784192ae0 2021-02-10T11:23:49.819 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:49.824 7f777b7fe700 1 -- 172.21.2.112:0/588934906 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7774000c90 con 0x7f7784192ae0 2021-02-10T11:23:49.819 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:49.824 7f7788f31700 1 -- 172.21.2.112:0/588934906 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7784092400 con 0x7f7784192ae0 2021-02-10T11:23:49.819 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:49.824 7f7788f31700 1 -- 172.21.2.112:0/588934906 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7784092990 con 0x7f7784192ae0 2021-02-10T11:23:49.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:49.824 7f777b7fe700 1 -- 172.21.2.112:0/588934906 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f77740022a0 con 0x7f7784192ae0 2021-02-10T11:23:49.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:49.825 7f777b7fe700 1 -- 172.21.2.112:0/588934906 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7774005310 con 0x7f7784192ae0 2021-02-10T11:23:49.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:49.825 7f777b7fe700 1 --2- 172.21.2.112:0/588934906 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7768018700 0x7f776801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:49.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:49.825 7f777b7fe700 1 -- 172.21.2.112:0/588934906 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f77740327d0 con 0x7f7784192ae0 2021-02-10T11:23:49.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:49.825 7f7782b13700 1 --2- 172.21.2.112:0/588934906 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7768018700 0x7f776801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:49.821 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:49.826 7f7782b13700 1 --2- 172.21.2.112:0/588934906 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7768018700 0x7f776801abc0 crc :-1 s=READY pgs=287 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:49.848 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:49.856 7f7788f31700 1 -- 172.21.2.112:0/588934906 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77840e52e0 con 0x7f7784192ae0 2021-02-10T11:23:49.850 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:49.859 7f777b7fe700 1 -- 172.21.2.112:0/588934906 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7774000940 con 0x7f7784192ae0 2021-02-10T11:23:49.979 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:49.987 7f7788f31700 1 -- 172.21.2.112:0/588934906 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348624", "target": ["mgr", ""]}) v1 -- 0x7f7784092990 con 0x7f7768018700 2021-02-10T11:23:50.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:50.010 7f777b7fe700 1 -- 172.21.2.112:0/588934906 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f776c0009b0 con 0x7f7768018700 2021-02-10T11:23:50.012 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:50.020 7f7788f31700 1 -- 172.21.2.112:0/588934906 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7768018700 msgr2=0x7f776801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:50.012 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:50.020 7f7788f31700 1 --2- 172.21.2.112:0/588934906 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7768018700 0x7f776801abc0 crc :-1 s=READY pgs=287 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:50.012 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:50.020 7f7788f31700 1 -- 172.21.2.112:0/588934906 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7784192ae0 msgr2=0x7f7784192ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:50.012 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:50.020 7f7788f31700 1 --2- 172.21.2.112:0/588934906 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7784192ae0 0x7f7784192ee0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 rx=0x7f777400b530 tx=0x7f777400b8d0).stop 2021-02-10T11:23:50.012 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:50.020 7f7788f31700 1 -- 172.21.2.112:0/588934906 shutdown_connections 2021-02-10T11:23:50.012 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:50.020 7f7788f31700 1 --2- 172.21.2.112:0/588934906 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7768018700 0x7f776801abc0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:50.012 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:50.020 7f7788f31700 1 --2- 172.21.2.112:0/588934906 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f77840919e0 0x7f7784091de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:50.012 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:50.020 7f7788f31700 1 --2- 172.21.2.112:0/588934906 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7784192ae0 0x7f7784192ee0 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:50.012 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:50.020 7f7788f31700 1 --2- 172.21.2.112:0/588934906 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f77840a6c90 0x7f77841924c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:50.012 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:50.020 7f7788f31700 1 -- 172.21.2.112:0/588934906 >> 172.21.2.112:0/588934906 conn(0x7f77840ac550 msgr2=0x7f77840be1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:50.013 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:50.020 7f7788f31700 1 -- 172.21.2.112:0/588934906 shutdown_connections 2021-02-10T11:23:50.013 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:50.020 7f7788f31700 1 -- 172.21.2.112:0/588934906 wait complete. 2021-02-10T11:23:50.024 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000348624 2021-02-10T11:23:50.456 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:50.447 7f5173ed3700 1 Processor -- start 2021-02-10T11:23:50.457 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:50.448 7f5173ed3700 1 -- start start 2021-02-10T11:23:50.457 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:50.448 7f5173ed3700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f516c1b0470 0x7f516c1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:50.457 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:50.448 7f5173ed3700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f516c0ef610 0x7f516c0efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:50.457 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:50.448 7f5173ed3700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f516c0f0b50 con 0x7f516c0f0030 2021-02-10T11:23:50.457 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:50.448 7f5173ed3700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f516c0e52e0 con 0x7f516c1b0470 2021-02-10T11:23:50.457 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:50.448 7f5173ed3700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f516c0e54d0 con 0x7f516c0ef610 2021-02-10T11:23:50.457 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:50.449 7f5171cef700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f516c1b0470 0x7f516c1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:50.457 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:50.449 7f51714ee700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f516c0ef610 0x7f516c0efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:50.457 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:50.449 7f5171cef700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f516c1b0470 0x7f516c1b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3300/0 says I am v2:172.21.2.112:47654/0 (socket says 172.21.2.112:47654) 2021-02-10T11:23:50.457 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:50.449 7f51714ee700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f516c0ef610 0x7f516c0efa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3301/0 says I am v2:172.21.2.112:45056/0 (socket says 172.21.2.112:45056) 2021-02-10T11:23:50.458 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:50.449 7f5171cef700 1 -- 172.21.2.112:0/440717990 learned_addr learned my addr 172.21.2.112:0/440717990 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:23:50.458 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:50.449 7f5171cef700 1 -- 172.21.2.112:0/440717990 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f516c0ef610 msgr2=0x7f516c0efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:50.458 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:50.449 7f5171cef700 1 --2- 172.21.2.112:0/440717990 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f516c0ef610 0x7f516c0efa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:50.458 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:50.449 7f5171cef700 1 -- 172.21.2.112:0/440717990 >> v1:172.21.2.120:6789/0 conn(0x7f516c0f0030 legacy=0x7f516c0f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:50.458 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:50.449 7f5171cef700 1 -- 172.21.2.112:0/440717990 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f516c0f0430 con 0x7f516c1b0470 2021-02-10T11:23:50.458 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:50.450 7f5171cef700 1 --2- 172.21.2.112:0/440717990 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f516c1b0470 0x7f516c1b0870 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7f5168006700 tx=0x7f5168009b80).ready entity=mon.1 client_cookie=4048f8deec73145a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:50.458 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:50.450 7f5170ced700 1 -- 172.21.2.112:0/440717990 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f516800c0c0 con 0x7f516c1b0470 2021-02-10T11:23:50.458 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:50.450 7f5170ced700 1 -- 172.21.2.112:0/440717990 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f516800c340 con 0x7f516c1b0470 2021-02-10T11:23:50.458 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:50.450 7f5170ced700 1 -- 172.21.2.112:0/440717990 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f516800c860 con 0x7f516c1b0470 2021-02-10T11:23:50.458 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:50.450 7f5173ed3700 1 -- 172.21.2.112:0/440717990 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f516c1b0470 msgr2=0x7f516c1b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:50.458 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:50.450 7f5173ed3700 1 --2- 172.21.2.112:0/440717990 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f516c1b0470 0x7f516c1b0870 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7f5168006700 tx=0x7f5168009b80).stop 2021-02-10T11:23:50.458 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:50.451 7f5173ed3700 1 -- 172.21.2.112:0/440717990 shutdown_connections 2021-02-10T11:23:50.458 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:50.451 7f5173ed3700 1 --2- 172.21.2.112:0/440717990 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f516c0ef610 0x7f516c0efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:50.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:50.451 7f5173ed3700 1 --2- 172.21.2.112:0/440717990 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f516c1b0470 0x7f516c1b0870 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:50.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:50.451 7f5173ed3700 1 -- 172.21.2.112:0/440717990 >> 172.21.2.112:0/440717990 conn(0x7f516c0ce3e0 msgr2=0x7f516c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:50.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:50.451 7f5173ed3700 1 -- 172.21.2.112:0/440717990 shutdown_connections 2021-02-10T11:23:50.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:50.451 7f5173ed3700 1 -- 172.21.2.112:0/440717990 wait complete. 2021-02-10T11:23:50.460 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:50.465 7f5173ed3700 1 Processor -- start 2021-02-10T11:23:50.460 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:50.465 7f5173ed3700 1 -- start start 2021-02-10T11:23:50.460 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:50.466 7f5173ed3700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f516c0bf930 0x7f516c0bfd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:50.461 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:50.466 7f5173ed3700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f516c0906c0 0x7f516c090ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:50.461 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:50.466 7f5173ed3700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f516c0c0960 0x7f516c0c0d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:50.461 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:50.466 7f5173ed3700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f516c0915c0 con 0x7f516c0906c0 2021-02-10T11:23:50.461 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:50.466 7f5173ed3700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f516c19c130 con 0x7f516c0bf930 2021-02-10T11:23:50.461 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:50.466 7f5173ed3700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f516c0910e0 con 0x7f516c0c0960 2021-02-10T11:23:50.461 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:50.466 7f51714ee700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f516c0906c0 0x7f516c090ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:50.461 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:50.466 7f51714ee700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f516c0906c0 0x7f516c090ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:45378/0 (socket says 172.21.2.112:45378) 2021-02-10T11:23:50.461 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:50.466 7f51714ee700 1 -- 172.21.2.112:0/638802299 learned_addr learned my addr 172.21.2.112:0/638802299 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:23:50.461 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:50.467 7f51714ee700 1 -- 172.21.2.112:0/638802299 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f516c0c0960 msgr2=0x7f516c0c0d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:50.461 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:50.467 7f51714ee700 1 --2- 172.21.2.112:0/638802299 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f516c0c0960 0x7f516c0c0d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:50.461 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:50.467 7f51714ee700 1 -- 172.21.2.112:0/638802299 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f516c0bf930 msgr2=0x7f516c0bfd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:50.461 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:50.467 7f5171cef700 1 --2- 172.21.2.112:0/638802299 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f516c0bf930 0x7f516c0bfd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:50.461 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:50.467 7f51714ee700 1 --2- 172.21.2.112:0/638802299 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f516c0bf930 0x7f516c0bfd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:50.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:50.467 7f51714ee700 1 -- 172.21.2.112:0/638802299 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f516c0c1380 con 0x7f516c0906c0 2021-02-10T11:23:50.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:50.467 7f5171cef700 1 --2- 172.21.2.112:0/638802299 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f516c0bf930 0x7f516c0bfd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:23:50.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:50.467 7f51714ee700 1 --2- 172.21.2.112:0/638802299 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f516c0906c0 0x7f516c090ac0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 rx=0x7f515c00b560 tx=0x7f515c00b900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:50.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:50.467 7f51627fc700 1 -- 172.21.2.112:0/638802299 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f515c00bcf0 con 0x7f516c0906c0 2021-02-10T11:23:50.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:50.467 7f51627fc700 1 -- 172.21.2.112:0/638802299 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f515c000c90 con 0x7f516c0906c0 2021-02-10T11:23:50.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:50.467 7f51627fc700 1 -- 172.21.2.112:0/638802299 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f515c002220 con 0x7f516c0906c0 2021-02-10T11:23:50.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:50.467 7f5173ed3700 1 -- 172.21.2.112:0/638802299 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f516c0c1380 con 0x7f516c0906c0 2021-02-10T11:23:50.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:50.467 7f5173ed3700 1 -- 172.21.2.112:0/638802299 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f516c0c1380 con 0x7f516c0906c0 2021-02-10T11:23:50.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:50.468 7f51627fc700 1 -- 172.21.2.112:0/638802299 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f515c002020 con 0x7f516c0906c0 2021-02-10T11:23:50.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:50.468 7f51627fc700 1 --2- 172.21.2.112:0/638802299 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f51540186c0 0x7f515401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:50.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:50.468 7f51627fc700 1 -- 172.21.2.112:0/638802299 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f515c032570 con 0x7f516c0906c0 2021-02-10T11:23:50.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:50.468 7f5171cef700 1 --2- 172.21.2.112:0/638802299 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f51540186c0 0x7f515401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:50.463 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:50.469 7f5171cef700 1 --2- 172.21.2.112:0/638802299 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f51540186c0 0x7f515401abc0 crc :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:50.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:50.498 7f5173ed3700 1 -- 172.21.2.112:0/638802299 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f516c105ae0 con 0x7f516c0906c0 2021-02-10T11:23:50.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:50.500 7f51627fc700 1 -- 172.21.2.112:0/638802299 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f515c02cf20 con 0x7f516c0906c0 2021-02-10T11:23:50.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:50.630 7f5173ed3700 1 -- 172.21.2.112:0/638802299 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348624", "target": ["mgr", ""]}) v1 -- 0x7f516c105ae0 con 0x7f51540186c0 2021-02-10T11:23:50.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:50.634 7f51627fc700 1 -- 172.21.2.112:0/638802299 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f51680009b0 con 0x7f51540186c0 2021-02-10T11:23:50.629 INFO:teuthology.orchestra.run.gibba012.stdout:/volumes/_nogroup/subvolume_0000000000348624/42a9d009-5e88-4f4d-a472-d8a82e87dcd5 2021-02-10T11:23:50.630 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:50.639 7f5173ed3700 1 -- 172.21.2.112:0/638802299 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f51540186c0 msgr2=0x7f515401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:50.630 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:50.639 7f5173ed3700 1 --2- 172.21.2.112:0/638802299 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f51540186c0 0x7f515401abc0 crc :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:50.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:50.639 7f5173ed3700 1 -- 172.21.2.112:0/638802299 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f516c0906c0 msgr2=0x7f516c090ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:50.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:50.639 7f5173ed3700 1 --2- 172.21.2.112:0/638802299 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f516c0906c0 0x7f516c090ac0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 rx=0x7f515c00b560 tx=0x7f515c00b900).stop 2021-02-10T11:23:50.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:50.639 7f5173ed3700 1 -- 172.21.2.112:0/638802299 shutdown_connections 2021-02-10T11:23:50.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:50.639 7f5173ed3700 1 --2- 172.21.2.112:0/638802299 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f51540186c0 0x7f515401abc0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:50.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:50.639 7f5173ed3700 1 --2- 172.21.2.112:0/638802299 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f516c0c0960 0x7f516c0c0d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:50.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:50.639 7f5173ed3700 1 --2- 172.21.2.112:0/638802299 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f516c0906c0 0x7f516c090ac0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:50.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:50.639 7f5173ed3700 1 --2- 172.21.2.112:0/638802299 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f516c0bf930 0x7f516c0bfd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:50.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:50.639 7f5173ed3700 1 -- 172.21.2.112:0/638802299 >> 172.21.2.112:0/638802299 conn(0x7f516c094e10 msgr2=0x7f516c0a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:50.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:50.639 7f5173ed3700 1 -- 172.21.2.112:0/638802299 shutdown_connections 2021-02-10T11:23:50.632 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:50.639 7f5173ed3700 1 -- 172.21.2.112:0/638802299 wait complete. 2021-02-10T11:23:50.638 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000348624 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000348624/42a9d009-5e88-4f4d-a472-d8a82e87dcd5 2021-02-10T11:23:50.638 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:50.639 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:23:50.639 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:23:50.976 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:23:50.976 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:23:50.976 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348624/42a9d009-5e88-4f4d-a472-d8a82e87dcd5/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:23:51.356 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:23:51.356 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:23:51.356 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0236389 s, 44.4 MB/s 2021-02-10T11:23:51.357 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:51.357 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:23:51.357 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:23:51.692 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:23:51.692 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:23:51.693 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348624/42a9d009-5e88-4f4d-a472-d8a82e87dcd5/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:23:52.069 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:23:52.069 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:23:52.069 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0244949 s, 42.8 MB/s 2021-02-10T11:23:52.070 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:52.070 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:23:52.070 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:23:52.406 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:23:52.406 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:23:52.406 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348624/42a9d009-5e88-4f4d-a472-d8a82e87dcd5/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:23:52.783 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:23:52.783 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:23:52.783 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0252449 s, 41.5 MB/s 2021-02-10T11:23:52.784 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:52.784 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:23:52.784 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:23:53.120 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:23:53.121 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:23:53.121 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348624/42a9d009-5e88-4f4d-a472-d8a82e87dcd5/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:23:53.494 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:23:53.495 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:23:53.495 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0227522 s, 46.1 MB/s 2021-02-10T11:23:53.496 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:53.496 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:23:53.496 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:23:53.831 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:23:53.832 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:23:53.832 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348624/42a9d009-5e88-4f4d-a472-d8a82e87dcd5/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:23:54.209 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:23:54.210 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:23:54.210 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.025877 s, 40.5 MB/s 2021-02-10T11:23:54.211 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:54.211 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:23:54.211 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:23:54.547 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:23:54.649 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:23:54.649 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348624/42a9d009-5e88-4f4d-a472-d8a82e87dcd5/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:23:55.023 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:23:55.024 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:23:55.024 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0245102 s, 42.8 MB/s 2021-02-10T11:23:55.025 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:55.025 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:23:55.025 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:23:55.359 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:23:55.359 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:23:55.359 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348624/42a9d009-5e88-4f4d-a472-d8a82e87dcd5/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:23:55.736 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:23:55.737 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:23:55.737 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0241908 s, 43.3 MB/s 2021-02-10T11:23:55.738 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:55.738 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:23:55.738 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:23:56.073 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:23:56.074 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:23:56.074 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348624/42a9d009-5e88-4f4d-a472-d8a82e87dcd5/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:23:56.449 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:23:56.449 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:23:56.449 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0232441 s, 45.1 MB/s 2021-02-10T11:23:56.450 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:56.451 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:23:56.451 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:23:56.787 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:23:56.788 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:23:56.788 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348624/42a9d009-5e88-4f4d-a472-d8a82e87dcd5/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:23:57.164 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:23:57.164 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:23:57.164 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0242802 s, 43.2 MB/s 2021-02-10T11:23:57.165 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:57.165 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:23:57.165 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:23:57.501 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:23:57.502 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:23:57.502 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348624/42a9d009-5e88-4f4d-a472-d8a82e87dcd5/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:23:57.877 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:23:57.877 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:23:57.877 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0236607 s, 44.3 MB/s 2021-02-10T11:23:57.878 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000348625 2021-02-10T11:23:58.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.301 7fed464c1700 1 Processor -- start 2021-02-10T11:23:58.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.301 7fed464c1700 1 -- start start 2021-02-10T11:23:58.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.302 7fed464c1700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fed401b0470 0x7fed401b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:58.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.302 7fed464c1700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fed400f20e0 0x7fed400f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:58.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.302 7fed464c1700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed400f0b50 con 0x7fed400ef610 2021-02-10T11:23:58.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.302 7fed464c1700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed400f0820 con 0x7fed401b0470 2021-02-10T11:23:58.311 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.302 7fed464c1700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed400f24e0 con 0x7fed400f20e0 2021-02-10T11:23:58.311 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.302 7fed3f7fe700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fed400ef610 0x7fed400efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:44850/0 (socket says 172.21.2.112:44850) 2021-02-10T11:23:58.311 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.302 7fed3f7fe700 1 -- 172.21.2.112:0/3494255534 learned_addr learned my addr 172.21.2.112:0/3494255534 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:23:58.311 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.302 7fed3effd700 1 -- 172.21.2.112:0/3494255534 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1600867199 0 0) 0x7fed340009b0 con 0x7fed400ef610 2021-02-10T11:23:58.311 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.302 7fed3effd700 1 -- 172.21.2.112:0/3494255534 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed28002fe0 con 0x7fed400ef610 2021-02-10T11:23:58.311 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.303 7fed3effd700 1 -- 172.21.2.112:0/3494255534 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4060509648 0 0) 0x7fed34000e20 con 0x7fed400ef610 2021-02-10T11:23:58.311 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.303 7fed3effd700 1 -- 172.21.2.112:0/3494255534 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fed400f20e0 msgr2=0x7fed400f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:58.311 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.303 7fed3effd700 1 --2- 172.21.2.112:0/3494255534 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fed400f20e0 0x7fed400f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:58.311 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.303 7fed3effd700 1 -- 172.21.2.112:0/3494255534 >> v2:172.21.2.120:3300/0 conn(0x7fed401b0470 msgr2=0x7fed401b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:58.311 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.303 7fed3effd700 1 --2- 172.21.2.112:0/3494255534 >> v2:172.21.2.120:3300/0 conn(0x7fed401b0470 0x7fed401b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:58.311 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.303 7fed3effd700 1 -- 172.21.2.112:0/3494255534 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed400e52e0 con 0x7fed400ef610 2021-02-10T11:23:58.311 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.303 7fed3effd700 1 -- 172.21.2.112:0/3494255534 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7fed34001530 con 0x7fed400ef610 2021-02-10T11:23:58.311 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.303 7fed3effd700 1 -- 172.21.2.112:0/3494255534 >> v1:172.21.2.112:6789/0 conn(0x7fed400ef610 legacy=0x7fed400efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:58.311 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.303 7fed3effd700 1 --2- 172.21.2.112:0/3494255534 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fed28003a60 0x7fed28005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:58.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.303 7fed3effd700 1 --2- 172.21.2.112:0/3494255534 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fed28006790 0x7fed28008c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:58.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.303 7fed3effd700 1 --2- 172.21.2.112:0/3494255534 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fed28009500 0x7fed2800ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:58.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.303 7fed3effd700 1 -- 172.21.2.112:0/3494255534 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed2800c2d0 con 0x7fed28009500 2021-02-10T11:23:58.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.303 7fed3effd700 1 -- 172.21.2.112:0/3494255534 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed2800c4f0 con 0x7fed28003a60 2021-02-10T11:23:58.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.303 7fed3effd700 1 -- 172.21.2.112:0/3494255534 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed2800c710 con 0x7fed28006790 2021-02-10T11:23:58.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.303 7fed44ade700 1 --2- 172.21.2.112:0/3494255534 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fed28009500 0x7fed2800ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:58.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.303 7fed3ffff700 1 --2- 172.21.2.112:0/3494255534 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fed28003a60 0x7fed28005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:58.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.303 7fed3f7fe700 1 --2- 172.21.2.112:0/3494255534 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fed28006790 0x7fed28008c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:58.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.303 7fed44ade700 1 -- 172.21.2.112:0/3494255534 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fed28006790 msgr2=0x7fed28008c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:58.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.303 7fed44ade700 1 --2- 172.21.2.112:0/3494255534 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fed28006790 0x7fed28008c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:58.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.303 7fed44ade700 1 -- 172.21.2.112:0/3494255534 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fed28003a60 msgr2=0x7fed28005ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:58.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.303 7fed44ade700 1 --2- 172.21.2.112:0/3494255534 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fed28003a60 0x7fed28005ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:58.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.303 7fed44ade700 1 -- 172.21.2.112:0/3494255534 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed2800c9b0 con 0x7fed28009500 2021-02-10T11:23:58.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.303 7fed44ade700 1 --2- 172.21.2.112:0/3494255534 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fed28009500 0x7fed2800ba20 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 rx=0x7fed38006e40 tx=0x7fed3804ddc0).ready entity=mon.0 client_cookie=a5f23390686890ae server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:58.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.303 7fed3effd700 1 -- 172.21.2.112:0/3494255534 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fed3804f300 con 0x7fed28009500 2021-02-10T11:23:58.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.303 7fed3effd700 1 -- 172.21.2.112:0/3494255534 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed3804f600 con 0x7fed28009500 2021-02-10T11:23:58.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.303 7fed3effd700 1 -- 172.21.2.112:0/3494255534 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fed3804fb00 con 0x7fed28009500 2021-02-10T11:23:58.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.303 7fed464c1700 1 -- 172.21.2.112:0/3494255534 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fed28009500 msgr2=0x7fed2800ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:58.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.303 7fed464c1700 1 --2- 172.21.2.112:0/3494255534 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fed28009500 0x7fed2800ba20 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 rx=0x7fed38006e40 tx=0x7fed3804ddc0).stop 2021-02-10T11:23:58.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.303 7fed44ade700 1 -- 172.21.2.112:0/3494255534 reap_dead start 2021-02-10T11:23:58.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.304 7fed464c1700 1 -- 172.21.2.112:0/3494255534 shutdown_connections 2021-02-10T11:23:58.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.304 7fed464c1700 1 -- 172.21.2.112:0/3494255534 >> 172.21.2.112:0/3494255534 conn(0x7fed400ce3e0 msgr2=0x7fed400e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:58.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.304 7fed464c1700 1 -- 172.21.2.112:0/3494255534 shutdown_connections 2021-02-10T11:23:58.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.304 7fed464c1700 1 -- 172.21.2.112:0/3494255534 wait complete. 2021-02-10T11:23:58.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.318 7fed464c1700 1 Processor -- start 2021-02-10T11:23:58.314 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.318 7fed464c1700 1 -- start start 2021-02-10T11:23:58.315 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.319 7fed464c1700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fed4009e560 0x7fed400a0130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:58.315 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.319 7fed464c1700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fed4009e960 0x7fed400a0750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:58.315 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.319 7fed464c1700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fed400a0d70 0x7fed400a1af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:58.316 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.319 7fed464c1700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed400c38f0 con 0x7fed4009e560 2021-02-10T11:23:58.316 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.319 7fed464c1700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed400a1170 con 0x7fed400a0d70 2021-02-10T11:23:58.316 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.319 7fed464c1700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed400a1390 con 0x7fed4009e960 2021-02-10T11:23:58.316 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.319 7fed3ffff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fed4009e560 0x7fed400a0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:58.316 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.319 7fed3ffff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fed4009e560 0x7fed400a0130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:45396/0 (socket says 172.21.2.112:45396) 2021-02-10T11:23:58.316 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.319 7fed3ffff700 1 -- 172.21.2.112:0/1240014175 learned_addr learned my addr 172.21.2.112:0/1240014175 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:23:58.316 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.319 7fed44ade700 1 --2- 172.21.2.112:0/1240014175 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fed400a0d70 0x7fed400a1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:58.316 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.319 7fed3f7fe700 1 --2- 172.21.2.112:0/1240014175 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fed4009e960 0x7fed400a0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:58.316 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.320 7fed3ffff700 1 -- 172.21.2.112:0/1240014175 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fed4009e960 msgr2=0x7fed400a0750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:58.316 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.320 7fed3ffff700 1 --2- 172.21.2.112:0/1240014175 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fed4009e960 0x7fed400a0750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:58.316 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.320 7fed3ffff700 1 -- 172.21.2.112:0/1240014175 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fed400a0d70 msgr2=0x7fed400a1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:58.316 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.320 7fed3ffff700 1 --2- 172.21.2.112:0/1240014175 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fed400a0d70 0x7fed400a1af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:58.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.320 7fed3ffff700 1 -- 172.21.2.112:0/1240014175 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed400a2110 con 0x7fed4009e560 2021-02-10T11:23:58.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.320 7fed3ffff700 1 --2- 172.21.2.112:0/1240014175 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fed4009e560 0x7fed400a0130 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 rx=0x7fed3000b590 tx=0x7fed3000b930).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:58.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.320 7fed3cff9700 1 -- 172.21.2.112:0/1240014175 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fed3000bd20 con 0x7fed4009e560 2021-02-10T11:23:58.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.320 7fed3cff9700 1 -- 172.21.2.112:0/1240014175 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed30002030 con 0x7fed4009e560 2021-02-10T11:23:58.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.320 7fed3cff9700 1 -- 172.21.2.112:0/1240014175 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fed30002550 con 0x7fed4009e560 2021-02-10T11:23:58.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.320 7fed464c1700 1 -- 172.21.2.112:0/1240014175 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed400a2110 con 0x7fed4009e560 2021-02-10T11:23:58.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.320 7fed464c1700 1 -- 172.21.2.112:0/1240014175 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed400a26a0 con 0x7fed4009e560 2021-02-10T11:23:58.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.321 7fed3cff9700 1 -- 172.21.2.112:0/1240014175 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fed30000c90 con 0x7fed4009e560 2021-02-10T11:23:58.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.321 7fed3cff9700 1 --2- 172.21.2.112:0/1240014175 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fed200186c0 0x7fed2001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:58.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.321 7fed3cff9700 1 -- 172.21.2.112:0/1240014175 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fed30032580 con 0x7fed4009e560 2021-02-10T11:23:58.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.321 7fed3f7fe700 1 --2- 172.21.2.112:0/1240014175 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fed200186c0 0x7fed2001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:58.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.322 7fed3f7fe700 1 --2- 172.21.2.112:0/1240014175 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fed200186c0 0x7fed2001abc0 crc :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:58.344 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.351 7fed464c1700 1 -- 172.21.2.112:0/1240014175 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed40105b30 con 0x7fed4009e560 2021-02-10T11:23:58.346 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.354 7fed3cff9700 1 -- 172.21.2.112:0/1240014175 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fed30022830 con 0x7fed4009e560 2021-02-10T11:23:58.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.480 7fed464c1700 1 -- 172.21.2.112:0/1240014175 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348625", "target": ["mgr", ""]}) v1 -- 0x7fed400a26a0 con 0x7fed200186c0 2021-02-10T11:23:58.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.504 7fed3cff9700 1 -- 172.21.2.112:0/1240014175 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fed340011c0 con 0x7fed200186c0 2021-02-10T11:23:58.505 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.513 7fed464c1700 1 -- 172.21.2.112:0/1240014175 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fed200186c0 msgr2=0x7fed2001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:58.505 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.513 7fed464c1700 1 --2- 172.21.2.112:0/1240014175 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fed200186c0 0x7fed2001abc0 crc :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:58.505 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.513 7fed464c1700 1 -- 172.21.2.112:0/1240014175 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fed4009e560 msgr2=0x7fed400a0130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:58.505 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.513 7fed464c1700 1 --2- 172.21.2.112:0/1240014175 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fed4009e560 0x7fed400a0130 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 rx=0x7fed3000b590 tx=0x7fed3000b930).stop 2021-02-10T11:23:58.505 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.513 7fed464c1700 1 -- 172.21.2.112:0/1240014175 shutdown_connections 2021-02-10T11:23:58.505 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.513 7fed464c1700 1 --2- 172.21.2.112:0/1240014175 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fed200186c0 0x7fed2001abc0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:58.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.513 7fed464c1700 1 --2- 172.21.2.112:0/1240014175 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fed400a0d70 0x7fed400a1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:58.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.513 7fed464c1700 1 --2- 172.21.2.112:0/1240014175 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fed4009e960 0x7fed400a0750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:58.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.513 7fed464c1700 1 --2- 172.21.2.112:0/1240014175 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fed4009e560 0x7fed400a0130 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:58.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.513 7fed464c1700 1 -- 172.21.2.112:0/1240014175 >> 172.21.2.112:0/1240014175 conn(0x7fed4009fc10 msgr2=0x7fed40094ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:58.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.513 7fed464c1700 1 -- 172.21.2.112:0/1240014175 shutdown_connections 2021-02-10T11:23:58.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.513 7fed464c1700 1 -- 172.21.2.112:0/1240014175 wait complete. 2021-02-10T11:23:58.513 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000348625 2021-02-10T11:23:58.943 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.935 7fec42f1e700 1 Processor -- start 2021-02-10T11:23:58.944 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.936 7fec42f1e700 1 -- start start 2021-02-10T11:23:58.944 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.936 7fec42f1e700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fec3c1b0470 0x7fec3c1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:58.944 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.936 7fec42f1e700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fec3c0ef610 0x7fec3c0efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:58.944 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.936 7fec42f1e700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec3c0f0b50 con 0x7fec3c0f0030 2021-02-10T11:23:58.944 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.936 7fec42f1e700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec3c0e52e0 con 0x7fec3c0ef610 2021-02-10T11:23:58.944 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.936 7fec42f1e700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec3c0e5490 con 0x7fec3c1b0470 2021-02-10T11:23:58.944 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.936 7fec3bfff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fec3c0ef610 0x7fec3c0efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:58.944 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.936 7fec3bfff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fec3c0ef610 0x7fec3c0efa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:45404/0 (socket says 172.21.2.112:45404) 2021-02-10T11:23:58.944 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.936 7fec3bfff700 1 -- 172.21.2.112:0/2675708725 learned_addr learned my addr 172.21.2.112:0/2675708725 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:23:58.944 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.936 7fec40d3a700 1 --2- 172.21.2.112:0/2675708725 >> v2:172.21.2.120:3300/0 conn(0x7fec3c1b0470 0x7fec3c1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:58.944 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.937 7fec3bfff700 1 -- 172.21.2.112:0/2675708725 >> v2:172.21.2.120:3300/0 conn(0x7fec3c1b0470 msgr2=0x7fec3c1b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:58.945 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.937 7fec3bfff700 1 --2- 172.21.2.112:0/2675708725 >> v2:172.21.2.120:3300/0 conn(0x7fec3c1b0470 0x7fec3c1b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:58.945 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.937 7fec3bfff700 1 -- 172.21.2.112:0/2675708725 >> v1:172.21.2.120:6789/0 conn(0x7fec3c0f0030 legacy=0x7fec3c0f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:58.945 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.937 7fec3bfff700 1 -- 172.21.2.112:0/2675708725 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec3c0f0430 con 0x7fec3c0ef610 2021-02-10T11:23:58.945 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.937 7fec3bfff700 1 --2- 172.21.2.112:0/2675708725 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fec3c0ef610 0x7fec3c0efa10 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 rx=0x7fec2c006700 tx=0x7fec2c009b00).ready entity=mon.0 client_cookie=910614c20863a661 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:58.945 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.937 7fec3b7fe700 1 -- 172.21.2.112:0/2675708725 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fec2c00c0c0 con 0x7fec3c0ef610 2021-02-10T11:23:58.945 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.937 7fec3b7fe700 1 -- 172.21.2.112:0/2675708725 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec2c00c400 con 0x7fec3c0ef610 2021-02-10T11:23:58.945 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.937 7fec3b7fe700 1 -- 172.21.2.112:0/2675708725 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fec2c00c920 con 0x7fec3c0ef610 2021-02-10T11:23:58.945 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.937 7fec42f1e700 1 -- 172.21.2.112:0/2675708725 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fec3c0ef610 msgr2=0x7fec3c0efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:58.945 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.937 7fec42f1e700 1 --2- 172.21.2.112:0/2675708725 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fec3c0ef610 0x7fec3c0efa10 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 rx=0x7fec2c006700 tx=0x7fec2c009b00).stop 2021-02-10T11:23:58.945 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.938 7fec42f1e700 1 -- 172.21.2.112:0/2675708725 shutdown_connections 2021-02-10T11:23:58.945 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.938 7fec42f1e700 1 --2- 172.21.2.112:0/2675708725 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fec3c0ef610 0x7fec3c0efa10 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:58.945 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.938 7fec42f1e700 1 --2- 172.21.2.112:0/2675708725 >> v2:172.21.2.120:3300/0 conn(0x7fec3c1b0470 0x7fec3c1b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:58.945 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.938 7fec42f1e700 1 -- 172.21.2.112:0/2675708725 >> 172.21.2.112:0/2675708725 conn(0x7fec3c0ce3e0 msgr2=0x7fec3c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:58.946 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.938 7fec42f1e700 1 -- 172.21.2.112:0/2675708725 shutdown_connections 2021-02-10T11:23:58.946 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.938 7fec42f1e700 1 -- 172.21.2.112:0/2675708725 wait complete. 2021-02-10T11:23:58.947 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.952 7fec42f1e700 1 Processor -- start 2021-02-10T11:23:58.947 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.952 7fec42f1e700 1 -- start start 2021-02-10T11:23:58.947 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.953 7fec42f1e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fec3c09f140 0x7fec3c0c1940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:58.947 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.953 7fec42f1e700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fec3c0c1f60 0x7fec3c09e300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:58.947 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.953 7fec42f1e700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fec3c09e920 0x7fec3c0c2960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:58.947 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.953 7fec42f1e700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec3c0c23e0 con 0x7fec3c09f140 2021-02-10T11:23:58.947 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.953 7fec42f1e700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec3c09ed20 con 0x7fec3c0c1f60 2021-02-10T11:23:58.948 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.953 7fec42f1e700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec3c105d10 con 0x7fec3c09e920 2021-02-10T11:23:58.948 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.953 7fec40d3a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fec3c09f140 0x7fec3c0c1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:58.948 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.953 7fec40d3a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fec3c09f140 0x7fec3c0c1940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:45410/0 (socket says 172.21.2.112:45410) 2021-02-10T11:23:58.948 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.953 7fec40d3a700 1 -- 172.21.2.112:0/1450006834 learned_addr learned my addr 172.21.2.112:0/1450006834 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:23:58.948 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.953 7fec40d3a700 1 -- 172.21.2.112:0/1450006834 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fec3c09e920 msgr2=0x7fec3c0c2960 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:23:58.948 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.953 7fec40d3a700 1 --2- 172.21.2.112:0/1450006834 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fec3c09e920 0x7fec3c0c2960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:58.948 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.953 7fec40d3a700 1 -- 172.21.2.112:0/1450006834 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fec3c0c1f60 msgr2=0x7fec3c09e300 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:23:58.948 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.953 7fec40d3a700 1 --2- 172.21.2.112:0/1450006834 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fec3c0c1f60 0x7fec3c09e300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:58.948 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.953 7fec40d3a700 1 -- 172.21.2.112:0/1450006834 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec3c1a4910 con 0x7fec3c09f140 2021-02-10T11:23:58.948 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.954 7fec40d3a700 1 --2- 172.21.2.112:0/1450006834 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fec3c09f140 0x7fec3c0c1940 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 rx=0x7fec240067e0 tx=0x7fec2400b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:58.948 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.954 7fec397fa700 1 -- 172.21.2.112:0/1450006834 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fec2400bd80 con 0x7fec3c09f140 2021-02-10T11:23:58.948 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.954 7fec397fa700 1 -- 172.21.2.112:0/1450006834 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec24006900 con 0x7fec3c09f140 2021-02-10T11:23:58.948 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.954 7fec397fa700 1 -- 172.21.2.112:0/1450006834 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fec24002080 con 0x7fec3c09f140 2021-02-10T11:23:58.948 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.954 7fec42f1e700 1 -- 172.21.2.112:0/1450006834 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec3c1a4910 con 0x7fec3c09f140 2021-02-10T11:23:58.949 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.954 7fec42f1e700 1 -- 172.21.2.112:0/1450006834 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec3c0c3270 con 0x7fec3c09f140 2021-02-10T11:23:58.949 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.954 7fec397fa700 1 -- 172.21.2.112:0/1450006834 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fec24002020 con 0x7fec3c09f140 2021-02-10T11:23:58.949 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.954 7fec397fa700 1 --2- 172.21.2.112:0/1450006834 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fec200186c0 0x7fec2001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:58.949 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.955 7fec397fa700 1 -- 172.21.2.112:0/1450006834 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fec24032800 con 0x7fec3c09f140 2021-02-10T11:23:58.949 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.955 7fec3bfff700 1 --2- 172.21.2.112:0/1450006834 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fec200186c0 0x7fec2001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:58.949 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.955 7fec3bfff700 1 --2- 172.21.2.112:0/1450006834 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fec200186c0 0x7fec2001abc0 crc :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:58.977 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.986 7fec42f1e700 1 -- 172.21.2.112:0/1450006834 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec3c0f0b50 con 0x7fec3c09f140 2021-02-10T11:23:58.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:58.988 7fec397fa700 1 -- 172.21.2.112:0/1450006834 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fec24002320 con 0x7fec3c09f140 2021-02-10T11:23:59.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:59.116 7fec42f1e700 1 -- 172.21.2.112:0/1450006834 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348625", "target": ["mgr", ""]}) v1 -- 0x7fec3c0c3270 con 0x7fec200186c0 2021-02-10T11:23:59.110 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:59.119 7fec397fa700 1 -- 172.21.2.112:0/1450006834 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7fec2c009440 con 0x7fec200186c0 2021-02-10T11:23:59.111 INFO:teuthology.orchestra.run.gibba012.stdout:/volumes/_nogroup/subvolume_0000000000348625/142b8f11-51bf-4d6a-8c6b-d023a0d85eaf 2021-02-10T11:23:59.112 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:59.121 7fec42f1e700 1 -- 172.21.2.112:0/1450006834 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fec200186c0 msgr2=0x7fec2001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:59.112 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:59.121 7fec42f1e700 1 --2- 172.21.2.112:0/1450006834 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fec200186c0 0x7fec2001abc0 crc :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:59.112 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:59.121 7fec42f1e700 1 -- 172.21.2.112:0/1450006834 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fec3c09f140 msgr2=0x7fec3c0c1940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:59.112 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:59.121 7fec42f1e700 1 --2- 172.21.2.112:0/1450006834 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fec3c09f140 0x7fec3c0c1940 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 rx=0x7fec240067e0 tx=0x7fec2400b990).stop 2021-02-10T11:23:59.113 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:59.121 7fec42f1e700 1 -- 172.21.2.112:0/1450006834 shutdown_connections 2021-02-10T11:23:59.113 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:59.121 7fec42f1e700 1 --2- 172.21.2.112:0/1450006834 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fec200186c0 0x7fec2001abc0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:59.113 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:59.121 7fec42f1e700 1 --2- 172.21.2.112:0/1450006834 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fec3c0c1f60 0x7fec3c09e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:59.113 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:59.121 7fec42f1e700 1 --2- 172.21.2.112:0/1450006834 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fec3c09e920 0x7fec3c0c2960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:59.113 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:59.121 7fec42f1e700 1 --2- 172.21.2.112:0/1450006834 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fec3c09f140 0x7fec3c0c1940 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:59.113 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:59.121 7fec42f1e700 1 -- 172.21.2.112:0/1450006834 >> 172.21.2.112:0/1450006834 conn(0x7fec3c09fc10 msgr2=0x7fec3c0b3250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:59.113 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:59.121 7fec42f1e700 1 -- 172.21.2.112:0/1450006834 shutdown_connections 2021-02-10T11:23:59.113 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:23:59.121 7fec42f1e700 1 -- 172.21.2.112:0/1450006834 wait complete. 2021-02-10T11:23:59.120 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000348625 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000348625/142b8f11-51bf-4d6a-8c6b-d023a0d85eaf 2021-02-10T11:23:59.121 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:59.121 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:23:59.121 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:23:59.456 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:23:59.456 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:23:59.457 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348625/142b8f11-51bf-4d6a-8c6b-d023a0d85eaf/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:23:59.835 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:23:59.836 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:23:59.836 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0247284 s, 42.4 MB/s 2021-02-10T11:23:59.837 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:59.837 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:23:59.837 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:00.173 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:24:00.173 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:24:00.173 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348625/142b8f11-51bf-4d6a-8c6b-d023a0d85eaf/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:24:00.547 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:24:00.547 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:24:00.547 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0238866 s, 43.9 MB/s 2021-02-10T11:24:00.548 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:00.549 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:00.549 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:00.883 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:24:00.883 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:24:00.883 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348625/142b8f11-51bf-4d6a-8c6b-d023a0d85eaf/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:24:01.260 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:24:01.260 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:24:01.260 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0250852 s, 41.8 MB/s 2021-02-10T11:24:01.261 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:01.261 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:01.261 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:01.597 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:24:01.597 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:24:01.598 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348625/142b8f11-51bf-4d6a-8c6b-d023a0d85eaf/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:24:01.973 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:24:01.974 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:24:01.974 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0248574 s, 42.2 MB/s 2021-02-10T11:24:01.975 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:01.975 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:01.975 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:02.310 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:24:02.311 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:24:02.311 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348625/142b8f11-51bf-4d6a-8c6b-d023a0d85eaf/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:24:02.687 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:24:02.687 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:24:02.687 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0245125 s, 42.8 MB/s 2021-02-10T11:24:02.688 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:02.688 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:02.688 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:03.024 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:24:03.024 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:24:03.025 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348625/142b8f11-51bf-4d6a-8c6b-d023a0d85eaf/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:24:03.399 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:24:03.399 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:24:03.399 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0242809 s, 43.2 MB/s 2021-02-10T11:24:03.400 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:03.400 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:03.400 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:03.736 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:24:03.736 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:24:03.737 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348625/142b8f11-51bf-4d6a-8c6b-d023a0d85eaf/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:24:04.114 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:24:04.114 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:24:04.114 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0272136 s, 38.5 MB/s 2021-02-10T11:24:04.115 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:04.115 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:04.115 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:04.450 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:24:04.450 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:24:04.451 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348625/142b8f11-51bf-4d6a-8c6b-d023a0d85eaf/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:24:04.826 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:24:04.826 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:24:04.826 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0270339 s, 38.8 MB/s 2021-02-10T11:24:04.827 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:04.827 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:04.828 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:05.175 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:24:05.175 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:24:05.176 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348625/142b8f11-51bf-4d6a-8c6b-d023a0d85eaf/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:24:05.554 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:24:05.554 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:24:05.554 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0255201 s, 41.1 MB/s 2021-02-10T11:24:05.555 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:05.555 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:05.555 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:05.890 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:24:05.890 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:24:05.890 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348625/142b8f11-51bf-4d6a-8c6b-d023a0d85eaf/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:24:06.267 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:24:06.267 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:24:06.267 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0248304 s, 42.2 MB/s 2021-02-10T11:24:06.268 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000348626 2021-02-10T11:24:06.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:06.690 7fc06e764700 1 Processor -- start 2021-02-10T11:24:06.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:06.690 7fc06e764700 1 -- start start 2021-02-10T11:24:06.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:06.690 7fc06e764700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc0681b0470 0x7fc0681b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:06.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:06.690 7fc06e764700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0680f0b50 con 0x7fc0680f20e0 2021-02-10T11:24:06.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:06.690 7fc06e764700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0680f24e0 con 0x7fc0680ef610 2021-02-10T11:24:06.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:06.690 7fc06e764700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0680e5ea0 con 0x7fc0681b0470 2021-02-10T11:24:06.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:06.691 7fc06cd81700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fc0680f20e0 0x7fc0680f0200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:44878/0 (socket says 172.21.2.112:44878) 2021-02-10T11:24:06.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:06.691 7fc06cd81700 1 -- 172.21.2.112:0/3680706283 learned_addr learned my addr 172.21.2.112:0/3680706283 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:24:06.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:06.691 7fc067fff700 1 --2- 172.21.2.112:0/3680706283 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc0681b0470 0x7fc0681b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:06.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:06.691 7fc066ffd700 1 -- 172.21.2.112:0/3680706283 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2157458478 0 0) 0x7fc060001500 con 0x7fc0680f20e0 2021-02-10T11:24:06.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:06.691 7fc066ffd700 1 -- 172.21.2.112:0/3680706283 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc050002fe0 con 0x7fc0680f20e0 2021-02-10T11:24:06.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:06.691 7fc066ffd700 1 -- 172.21.2.112:0/3680706283 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 579818657 0 0) 0x7fc05c000ff0 con 0x7fc0680ef610 2021-02-10T11:24:06.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:06.691 7fc066ffd700 1 -- 172.21.2.112:0/3680706283 --> v1:172.21.2.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc050003210 con 0x7fc0680ef610 2021-02-10T11:24:06.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:06.691 7fc067fff700 1 -- 172.21.2.112:0/3680706283 >> v1:172.21.2.120:6789/0 conn(0x7fc0680ef610 legacy=0x7fc0680efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:06.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:06.691 7fc067fff700 1 -- 172.21.2.112:0/3680706283 >> v1:172.21.2.112:6789/0 conn(0x7fc0680f20e0 legacy=0x7fc0680f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:06.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:06.691 7fc067fff700 1 -- 172.21.2.112:0/3680706283 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0680e52e0 con 0x7fc0681b0470 2021-02-10T11:24:06.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:06.692 7fc067fff700 1 --2- 172.21.2.112:0/3680706283 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc0681b0470 0x7fc0681b0870 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7fc058006650 tx=0x7fc058009a80).ready entity=mon.2 client_cookie=65b4504dab388f78 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:06.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:06.692 7fc066ffd700 1 -- 172.21.2.112:0/3680706283 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc0580009b0 con 0x7fc0681b0470 2021-02-10T11:24:06.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:06.692 7fc066ffd700 1 -- 172.21.2.112:0/3680706283 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc058006740 con 0x7fc0681b0470 2021-02-10T11:24:06.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:06.692 7fc066ffd700 1 -- 172.21.2.112:0/3680706283 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc05800c0c0 con 0x7fc0681b0470 2021-02-10T11:24:06.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:06.692 7fc06e764700 1 -- 172.21.2.112:0/3680706283 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc0681b0470 msgr2=0x7fc0681b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:06.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:06.692 7fc06e764700 1 --2- 172.21.2.112:0/3680706283 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc0681b0470 0x7fc0681b0870 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7fc058006650 tx=0x7fc058009a80).stop 2021-02-10T11:24:06.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:06.693 7fc06e764700 1 -- 172.21.2.112:0/3680706283 shutdown_connections 2021-02-10T11:24:06.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:06.693 7fc06e764700 1 --2- 172.21.2.112:0/3680706283 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc0681b0470 0x7fc0681b0870 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:06.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:06.693 7fc06e764700 1 -- 172.21.2.112:0/3680706283 >> 172.21.2.112:0/3680706283 conn(0x7fc0680ce3e0 msgr2=0x7fc0680e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:06.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:06.693 7fc06e764700 1 -- 172.21.2.112:0/3680706283 shutdown_connections 2021-02-10T11:24:06.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:06.693 7fc06e764700 1 -- 172.21.2.112:0/3680706283 wait complete. 2021-02-10T11:24:06.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:06.707 7fc06e764700 1 Processor -- start 2021-02-10T11:24:06.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:06.707 7fc06e764700 1 -- start start 2021-02-10T11:24:06.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:06.708 7fc06e764700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc0681a0470 0x7fc06808a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:06.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:06.708 7fc06e764700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc06808aa50 0x7fc068099180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:06.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:06.708 7fc06e764700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc0680997a0 0x7fc068099ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:06.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:06.708 7fc06e764700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc068095610 con 0x7fc06808aa50 2021-02-10T11:24:06.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:06.708 7fc06e764700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc068095800 con 0x7fc0680997a0 2021-02-10T11:24:06.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:06.708 7fc06e764700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc06809a1c0 con 0x7fc0681a0470 2021-02-10T11:24:06.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:06.708 7fc0677fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc06808aa50 0x7fc068099180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:06.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:06.708 7fc0677fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc06808aa50 0x7fc068099180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:45424/0 (socket says 172.21.2.112:45424) 2021-02-10T11:24:06.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:06.708 7fc0677fe700 1 -- 172.21.2.112:0/1298545103 learned_addr learned my addr 172.21.2.112:0/1298545103 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:24:06.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:06.708 7fc0677fe700 1 -- 172.21.2.112:0/1298545103 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc0681a0470 msgr2=0x7fc06808a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:06.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:06.708 7fc0677fe700 1 --2- 172.21.2.112:0/1298545103 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc0681a0470 0x7fc06808a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:06.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:06.708 7fc0677fe700 1 -- 172.21.2.112:0/1298545103 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc0680997a0 msgr2=0x7fc068099ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:06.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:06.708 7fc0677fe700 1 --2- 172.21.2.112:0/1298545103 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc0680997a0 0x7fc068099ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:06.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:06.708 7fc0677fe700 1 -- 172.21.2.112:0/1298545103 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc06809a3b0 con 0x7fc06808aa50 2021-02-10T11:24:06.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:06.708 7fc0677fe700 1 --2- 172.21.2.112:0/1298545103 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc06808aa50 0x7fc068099180 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 rx=0x7fc05c006ed0 tx=0x7fc05c00b8e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:06.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:06.709 7fc064ff9700 1 -- 172.21.2.112:0/1298545103 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc05c00bcd0 con 0x7fc06808aa50 2021-02-10T11:24:06.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:06.709 7fc06e764700 1 -- 172.21.2.112:0/1298545103 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc06809a3b0 con 0x7fc06808aa50 2021-02-10T11:24:06.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:06.709 7fc06e764700 1 -- 172.21.2.112:0/1298545103 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc06808b850 con 0x7fc06808aa50 2021-02-10T11:24:06.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:06.709 7fc064ff9700 1 -- 172.21.2.112:0/1298545103 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc05c00bcd0 con 0x7fc06808aa50 2021-02-10T11:24:06.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:06.709 7fc064ff9700 1 -- 172.21.2.112:0/1298545103 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc05c006240 con 0x7fc06808aa50 2021-02-10T11:24:06.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:06.709 7fc064ff9700 1 -- 172.21.2.112:0/1298545103 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc05c0056c0 con 0x7fc06808aa50 2021-02-10T11:24:06.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:06.709 7fc064ff9700 1 --2- 172.21.2.112:0/1298545103 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc0480186c0 0x7fc04801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:06.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:06.709 7fc064ff9700 1 -- 172.21.2.112:0/1298545103 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fc05c032410 con 0x7fc06808aa50 2021-02-10T11:24:06.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:06.710 7fc067fff700 1 --2- 172.21.2.112:0/1298545103 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc0480186c0 0x7fc04801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:06.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:06.710 7fc067fff700 1 --2- 172.21.2.112:0/1298545103 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc0480186c0 0x7fc04801abc0 crc :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:06.732 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:06.740 7fc06e764700 1 -- 172.21.2.112:0/1298545103 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc068105ae0 con 0x7fc06808aa50 2021-02-10T11:24:06.734 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:06.743 7fc064ff9700 1 -- 172.21.2.112:0/1298545103 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc05c025fc0 con 0x7fc06808aa50 2021-02-10T11:24:06.862 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:06.871 7fc06e764700 1 -- 172.21.2.112:0/1298545103 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348626", "target": ["mgr", ""]}) v1 -- 0x7fc06808b850 con 0x7fc0480186c0 2021-02-10T11:24:06.887 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:06.896 7fc064ff9700 1 -- 172.21.2.112:0/1298545103 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fc058000940 con 0x7fc0480186c0 2021-02-10T11:24:06.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:06.904 7fc06e764700 1 -- 172.21.2.112:0/1298545103 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc0480186c0 msgr2=0x7fc04801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:06.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:06.904 7fc06e764700 1 --2- 172.21.2.112:0/1298545103 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc0480186c0 0x7fc04801abc0 crc :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:06.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:06.904 7fc06e764700 1 -- 172.21.2.112:0/1298545103 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc06808aa50 msgr2=0x7fc068099180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:06.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:06.904 7fc06e764700 1 --2- 172.21.2.112:0/1298545103 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc06808aa50 0x7fc068099180 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 rx=0x7fc05c006ed0 tx=0x7fc05c00b8e0).stop 2021-02-10T11:24:06.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:06.904 7fc06e764700 1 -- 172.21.2.112:0/1298545103 shutdown_connections 2021-02-10T11:24:06.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:06.904 7fc06e764700 1 --2- 172.21.2.112:0/1298545103 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc0480186c0 0x7fc04801abc0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:06.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:06.904 7fc06e764700 1 --2- 172.21.2.112:0/1298545103 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc0680997a0 0x7fc068099ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:06.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:06.904 7fc06e764700 1 --2- 172.21.2.112:0/1298545103 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc06808aa50 0x7fc068099180 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:06.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:06.904 7fc06e764700 1 --2- 172.21.2.112:0/1298545103 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc0681a0470 0x7fc06808a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:06.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:06.904 7fc06e764700 1 -- 172.21.2.112:0/1298545103 >> 172.21.2.112:0/1298545103 conn(0x7fc0680bbf60 msgr2=0x7fc068105d10 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:06.897 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:06.904 7fc06e764700 1 -- 172.21.2.112:0/1298545103 shutdown_connections 2021-02-10T11:24:06.897 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:06.904 7fc06e764700 1 -- 172.21.2.112:0/1298545103 wait complete. 2021-02-10T11:24:06.904 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000348626 2021-02-10T11:24:07.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:07.326 7f9c62fd6700 1 Processor -- start 2021-02-10T11:24:07.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:07.327 7f9c62fd6700 1 -- start start 2021-02-10T11:24:07.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:07.327 7f9c62fd6700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9c5c1b0470 0x7f9c5c1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:07.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:07.327 7f9c62fd6700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c5c0f0b50 con 0x7f9c5c0ef610 2021-02-10T11:24:07.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:07.327 7f9c62fd6700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c5c0f24e0 con 0x7f9c5c0f20e0 2021-02-10T11:24:07.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:07.327 7f9c62fd6700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c5c0e5ea0 con 0x7f9c5c1b0470 2021-02-10T11:24:07.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:07.327 7f9c5bfff700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f9c5c0ef610 0x7f9c5c0efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:44890/0 (socket says 172.21.2.112:44890) 2021-02-10T11:24:07.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:07.327 7f9c5bfff700 1 -- 172.21.2.112:0/289164506 learned_addr learned my addr 172.21.2.112:0/289164506 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:24:07.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:07.327 7f9c5b7fe700 1 -- 172.21.2.112:0/289164506 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 833817154 0 0) 0x7f9c4c000ce0 con 0x7f9c5c0ef610 2021-02-10T11:24:07.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:07.327 7f9c5b7fe700 1 -- 172.21.2.112:0/289164506 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c48002fe0 con 0x7f9c5c0ef610 2021-02-10T11:24:07.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:07.328 7f9c5b7fe700 1 -- 172.21.2.112:0/289164506 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1269258671 0 0) 0x7f9c4c0010e0 con 0x7f9c5c0ef610 2021-02-10T11:24:07.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:07.328 7f9c5b7fe700 1 -- 172.21.2.112:0/289164506 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9c5c1b0470 msgr2=0x7f9c5c1b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:07.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:07.328 7f9c5b7fe700 1 --2- 172.21.2.112:0/289164506 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9c5c1b0470 0x7f9c5c1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:07.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:07.328 7f9c5b7fe700 1 -- 172.21.2.112:0/289164506 >> v1:172.21.2.120:6789/0 conn(0x7f9c5c0f20e0 legacy=0x7f9c5c0f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:07.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:07.328 7f9c5b7fe700 1 -- 172.21.2.112:0/289164506 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c5c0e52e0 con 0x7f9c5c0ef610 2021-02-10T11:24:07.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:07.328 7f9c5b7fe700 1 -- 172.21.2.112:0/289164506 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f9c4c001600 con 0x7f9c5c0ef610 2021-02-10T11:24:07.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:07.328 7f9c5b7fe700 1 -- 172.21.2.112:0/289164506 >> v1:172.21.2.112:6789/0 conn(0x7f9c5c0ef610 legacy=0x7f9c5c0efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:07.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:07.328 7f9c5b7fe700 1 --2- 172.21.2.112:0/289164506 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9c48003b20 0x7f9c48005f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:07.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:07.328 7f9c5b7fe700 1 --2- 172.21.2.112:0/289164506 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9c480067b0 0x7f9c48008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:07.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:07.328 7f9c5b7fe700 1 --2- 172.21.2.112:0/289164506 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9c48009590 0x7f9c4800ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:07.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:07.328 7f9c5b7fe700 1 -- 172.21.2.112:0/289164506 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c4800c2d0 con 0x7f9c48003b20 2021-02-10T11:24:07.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:07.328 7f9c5b7fe700 1 -- 172.21.2.112:0/289164506 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c4800c4f0 con 0x7f9c480067b0 2021-02-10T11:24:07.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:07.328 7f9c5b7fe700 1 -- 172.21.2.112:0/289164506 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c4800c710 con 0x7f9c48009590 2021-02-10T11:24:07.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:07.328 7f9c60df2700 1 --2- 172.21.2.112:0/289164506 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9c48003b20 0x7f9c48005f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:07.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:07.328 7f9c5bfff700 1 --2- 172.21.2.112:0/289164506 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9c480067b0 0x7f9c48008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:07.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:07.328 7f9c615f3700 1 --2- 172.21.2.112:0/289164506 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9c48009590 0x7f9c4800ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:07.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:07.328 7f9c60df2700 1 -- 172.21.2.112:0/289164506 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9c48009590 msgr2=0x7f9c4800ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:07.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:07.328 7f9c60df2700 1 --2- 172.21.2.112:0/289164506 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9c48009590 0x7f9c4800ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:07.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:07.328 7f9c60df2700 1 -- 172.21.2.112:0/289164506 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9c480067b0 msgr2=0x7f9c48008c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:07.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:07.328 7f9c60df2700 1 --2- 172.21.2.112:0/289164506 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9c480067b0 0x7f9c48008c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:07.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:07.328 7f9c60df2700 1 -- 172.21.2.112:0/289164506 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c4800c9b0 con 0x7f9c48003b20 2021-02-10T11:24:07.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:07.328 7f9c60df2700 1 --2- 172.21.2.112:0/289164506 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9c48003b20 0x7f9c48005f30 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 rx=0x7f9c44006ab0 tx=0x7f9c44009b40).ready entity=mon.0 client_cookie=e9ca8f7e3bfeb65e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:07.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:07.329 7f9c5b7fe700 1 -- 172.21.2.112:0/289164506 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9c44002020 con 0x7f9c48003b20 2021-02-10T11:24:07.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:07.329 7f9c5b7fe700 1 -- 172.21.2.112:0/289164506 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c44002360 con 0x7f9c48003b20 2021-02-10T11:24:07.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:07.329 7f9c5b7fe700 1 -- 172.21.2.112:0/289164506 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9c44002880 con 0x7f9c48003b20 2021-02-10T11:24:07.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:07.329 7f9c62fd6700 1 -- 172.21.2.112:0/289164506 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9c48003b20 msgr2=0x7f9c48005f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:07.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:07.329 7f9c62fd6700 1 --2- 172.21.2.112:0/289164506 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9c48003b20 0x7f9c48005f30 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 rx=0x7f9c44006ab0 tx=0x7f9c44009b40).stop 2021-02-10T11:24:07.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:07.329 7f9c615f3700 1 -- 172.21.2.112:0/289164506 reap_dead start 2021-02-10T11:24:07.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:07.329 7f9c62fd6700 1 -- 172.21.2.112:0/289164506 shutdown_connections 2021-02-10T11:24:07.339 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:07.329 7f9c62fd6700 1 -- 172.21.2.112:0/289164506 >> 172.21.2.112:0/289164506 conn(0x7f9c5c0ce3e0 msgr2=0x7f9c5c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:07.339 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:07.330 7f9c62fd6700 1 -- 172.21.2.112:0/289164506 shutdown_connections 2021-02-10T11:24:07.339 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:07.330 7f9c62fd6700 1 -- 172.21.2.112:0/289164506 wait complete. 2021-02-10T11:24:07.339 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:07.344 7f9c62fd6700 1 Processor -- start 2021-02-10T11:24:07.339 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:07.344 7f9c62fd6700 1 -- start start 2021-02-10T11:24:07.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:07.344 7f9c62fd6700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9c5c0b7db0 0x7f9c5c0b8260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:07.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:07.344 7f9c62fd6700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9c5c0b8880 0x7f9c5c1a7450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:07.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:07.345 7f9c62fd6700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9c5c0b8c80 0x7f9c5c1a7a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:07.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:07.345 7f9c62fd6700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c5c0b79b0 con 0x7f9c5c0b8c80 2021-02-10T11:24:07.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:07.345 7f9c62fd6700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c5c1a8090 con 0x7f9c5c0b7db0 2021-02-10T11:24:07.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:07.345 7f9c62fd6700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c5c1a82b0 con 0x7f9c5c0b8880 2021-02-10T11:24:07.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:07.345 7f9c615f3700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9c5c0b8c80 0x7f9c5c1a7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:07.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:07.345 7f9c615f3700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9c5c0b8c80 0x7f9c5c1a7a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:45440/0 (socket says 172.21.2.112:45440) 2021-02-10T11:24:07.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:07.345 7f9c615f3700 1 -- 172.21.2.112:0/4226016537 learned_addr learned my addr 172.21.2.112:0/4226016537 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:24:07.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:07.345 7f9c615f3700 1 -- 172.21.2.112:0/4226016537 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9c5c0b8880 msgr2=0x7f9c5c1a7450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:07.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:07.345 7f9c615f3700 1 --2- 172.21.2.112:0/4226016537 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9c5c0b8880 0x7f9c5c1a7450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:07.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:07.345 7f9c615f3700 1 -- 172.21.2.112:0/4226016537 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9c5c0b7db0 msgr2=0x7f9c5c0b8260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:07.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:07.345 7f9c615f3700 1 --2- 172.21.2.112:0/4226016537 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9c5c0b7db0 0x7f9c5c0b8260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:07.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:07.345 7f9c615f3700 1 -- 172.21.2.112:0/4226016537 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c5c190c10 con 0x7f9c5c0b8c80 2021-02-10T11:24:07.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:07.345 7f9c615f3700 1 --2- 172.21.2.112:0/4226016537 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9c5c0b8c80 0x7f9c5c1a7a70 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 rx=0x7f9c54009980 tx=0x7f9c5400bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:07.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:07.345 7f9c597fa700 1 -- 172.21.2.112:0/4226016537 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9c54006ba0 con 0x7f9c5c0b8c80 2021-02-10T11:24:07.342 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:07.345 7f9c597fa700 1 -- 172.21.2.112:0/4226016537 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c5400c2f0 con 0x7f9c5c0b8c80 2021-02-10T11:24:07.342 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:07.345 7f9c62fd6700 1 -- 172.21.2.112:0/4226016537 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c5c190c10 con 0x7f9c5c0b8c80 2021-02-10T11:24:07.342 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:07.345 7f9c597fa700 1 -- 172.21.2.112:0/4226016537 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9c5400c750 con 0x7f9c5c0b8c80 2021-02-10T11:24:07.342 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:07.345 7f9c62fd6700 1 -- 172.21.2.112:0/4226016537 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c5c190c10 con 0x7f9c5c0b8c80 2021-02-10T11:24:07.342 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:07.346 7f9c597fa700 1 -- 172.21.2.112:0/4226016537 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9c540011f0 con 0x7f9c5c0b8c80 2021-02-10T11:24:07.342 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:07.346 7f9c597fa700 1 --2- 172.21.2.112:0/4226016537 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9c40018730 0x7f9c4001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:07.342 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:07.346 7f9c597fa700 1 -- 172.21.2.112:0/4226016537 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f9c540329a0 con 0x7f9c5c0b8c80 2021-02-10T11:24:07.342 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:07.346 7f9c60df2700 1 --2- 172.21.2.112:0/4226016537 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9c40018730 0x7f9c4001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:07.342 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:07.347 7f9c60df2700 1 --2- 172.21.2.112:0/4226016537 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9c40018730 0x7f9c4001ac10 crc :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:07.368 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:07.377 7f9c62fd6700 1 -- 172.21.2.112:0/4226016537 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c5c105b30 con 0x7f9c5c0b8c80 2021-02-10T11:24:07.370 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:07.379 7f9c597fa700 1 -- 172.21.2.112:0/4226016537 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9c5400be10 con 0x7f9c5c0b8c80 2021-02-10T11:24:07.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:07.506 7f9c62fd6700 1 -- 172.21.2.112:0/4226016537 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348626", "target": ["mgr", ""]}) v1 -- 0x7f9c5c0f0b50 con 0x7f9c40018730 2021-02-10T11:24:07.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:07.510 7f9c597fa700 1 -- 172.21.2.112:0/4226016537 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f9c44000970 con 0x7f9c40018730 2021-02-10T11:24:07.505 INFO:teuthology.orchestra.run.gibba012.stdout:/volumes/_nogroup/subvolume_0000000000348626/e41a339f-a505-4b63-87ff-8c2f3e323dec 2021-02-10T11:24:07.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:07.516 7f9c62fd6700 1 -- 172.21.2.112:0/4226016537 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9c40018730 msgr2=0x7f9c4001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:07.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:07.516 7f9c62fd6700 1 --2- 172.21.2.112:0/4226016537 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9c40018730 0x7f9c4001ac10 crc :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:07.508 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:07.516 7f9c62fd6700 1 -- 172.21.2.112:0/4226016537 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9c5c0b8c80 msgr2=0x7f9c5c1a7a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:07.508 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:07.516 7f9c62fd6700 1 --2- 172.21.2.112:0/4226016537 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9c5c0b8c80 0x7f9c5c1a7a70 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 rx=0x7f9c54009980 tx=0x7f9c5400bcd0).stop 2021-02-10T11:24:07.508 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:07.516 7f9c62fd6700 1 -- 172.21.2.112:0/4226016537 shutdown_connections 2021-02-10T11:24:07.508 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:07.516 7f9c62fd6700 1 --2- 172.21.2.112:0/4226016537 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9c40018730 0x7f9c4001ac10 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:07.508 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:07.516 7f9c62fd6700 1 --2- 172.21.2.112:0/4226016537 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9c5c0b8c80 0x7f9c5c1a7a70 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:07.508 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:07.516 7f9c62fd6700 1 --2- 172.21.2.112:0/4226016537 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9c5c0b8880 0x7f9c5c1a7450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:07.508 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:07.516 7f9c62fd6700 1 --2- 172.21.2.112:0/4226016537 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9c5c0b7db0 0x7f9c5c0b8260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:07.509 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:07.516 7f9c62fd6700 1 -- 172.21.2.112:0/4226016537 >> 172.21.2.112:0/4226016537 conn(0x7f9c5c094e10 msgr2=0x7f9c5c0a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:07.509 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:07.516 7f9c62fd6700 1 -- 172.21.2.112:0/4226016537 shutdown_connections 2021-02-10T11:24:07.509 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:07.517 7f9c62fd6700 1 -- 172.21.2.112:0/4226016537 wait complete. 2021-02-10T11:24:07.515 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000348626 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000348626/e41a339f-a505-4b63-87ff-8c2f3e323dec 2021-02-10T11:24:07.516 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:07.516 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:07.516 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:07.853 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:24:07.853 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:24:07.853 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348626/e41a339f-a505-4b63-87ff-8c2f3e323dec/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:24:08.233 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:24:08.233 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:24:08.233 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0238601 s, 43.9 MB/s 2021-02-10T11:24:08.234 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:08.234 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:08.235 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:08.569 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:24:08.569 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:24:08.569 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348626/e41a339f-a505-4b63-87ff-8c2f3e323dec/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:24:08.869 DEBUG:teuthology.orchestra.run.gibba012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:24:08.873 DEBUG:teuthology.orchestra.run.gibba020:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:24:08.944 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:24:08.944 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:24:08.944 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0227005 s, 46.2 MB/s 2021-02-10T11:24:08.945 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:08.945 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:08.945 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:09.206 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:24:09.207 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:24:09.207 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348626/e41a339f-a505-4b63-87ff-8c2f3e323dec/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:24:09.581 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:24:09.581 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:24:09.581 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0254598 s, 41.2 MB/s 2021-02-10T11:24:09.582 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:09.582 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:09.582 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:09.919 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:24:09.919 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:24:09.920 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348626/e41a339f-a505-4b63-87ff-8c2f3e323dec/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:24:10.298 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:24:10.298 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:24:10.298 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0255981 s, 41.0 MB/s 2021-02-10T11:24:10.299 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:10.299 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:10.299 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:10.635 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:24:10.635 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:24:10.635 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348626/e41a339f-a505-4b63-87ff-8c2f3e323dec/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:24:11.011 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:24:11.012 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:24:11.012 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0241387 s, 43.4 MB/s 2021-02-10T11:24:11.013 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:11.013 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:11.013 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:11.349 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:24:11.349 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:24:11.350 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348626/e41a339f-a505-4b63-87ff-8c2f3e323dec/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:24:11.727 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:24:11.727 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:24:11.727 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0244082 s, 43.0 MB/s 2021-02-10T11:24:11.727 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:11.728 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:11.728 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:12.063 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:24:12.063 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:24:12.064 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348626/e41a339f-a505-4b63-87ff-8c2f3e323dec/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:24:12.440 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:24:12.441 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:24:12.441 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0252628 s, 41.5 MB/s 2021-02-10T11:24:12.441 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:12.442 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:12.442 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:12.777 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:24:12.777 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:24:12.777 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348626/e41a339f-a505-4b63-87ff-8c2f3e323dec/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:24:13.155 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:24:13.155 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:24:13.155 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0256822 s, 40.8 MB/s 2021-02-10T11:24:13.156 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:13.157 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:13.157 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:13.493 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:24:13.494 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:24:13.494 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348626/e41a339f-a505-4b63-87ff-8c2f3e323dec/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:24:13.870 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:24:13.871 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:24:13.871 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0247703 s, 42.3 MB/s 2021-02-10T11:24:13.872 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:13.872 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:13.872 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:14.208 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:24:14.208 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:24:14.209 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348626/e41a339f-a505-4b63-87ff-8c2f3e323dec/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:24:14.579 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:24:14.579 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:24:14.579 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0239211 s, 43.8 MB/s 2021-02-10T11:24:14.580 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000348627 2021-02-10T11:24:15.011 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.003 7f1032421700 1 Processor -- start 2021-02-10T11:24:15.011 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.003 7f1032421700 1 -- start start 2021-02-10T11:24:15.011 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.004 7f1032421700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f102c1b0470 0x7f102c1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:15.011 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.004 7f1032421700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f102c0f20e0 0x7f102c0f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:15.011 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.004 7f1032421700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f102c0f0b50 con 0x7f102c0ef610 2021-02-10T11:24:15.011 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.004 7f1032421700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f102c0f0820 con 0x7f102c1b0470 2021-02-10T11:24:15.011 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.004 7f1032421700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f102c0f24e0 con 0x7f102c0f20e0 2021-02-10T11:24:15.012 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.004 7f102bfff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f102c1b0470 0x7f102c1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:15.012 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.004 7f102bfff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f102c1b0470 0x7f102c1b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:45446/0 (socket says 172.21.2.112:45446) 2021-02-10T11:24:15.012 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.004 7f102bfff700 1 -- 172.21.2.112:0/298075457 learned_addr learned my addr 172.21.2.112:0/298075457 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:24:15.012 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.004 7f1030a3e700 1 --2- 172.21.2.112:0/298075457 >> v2:172.21.2.120:3300/0 conn(0x7f102c0f20e0 0x7f102c0f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:15.012 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.004 7f102bfff700 1 -- 172.21.2.112:0/298075457 >> v2:172.21.2.120:3300/0 conn(0x7f102c0f20e0 msgr2=0x7f102c0f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:15.012 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.004 7f102bfff700 1 --2- 172.21.2.112:0/298075457 >> v2:172.21.2.120:3300/0 conn(0x7f102c0f20e0 0x7f102c0f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:15.012 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.004 7f102bfff700 1 -- 172.21.2.112:0/298075457 >> v1:172.21.2.112:6789/0 conn(0x7f102c0ef610 legacy=0x7f102c0efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:15.012 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.004 7f102bfff700 1 -- 172.21.2.112:0/298075457 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f102c0e52e0 con 0x7f102c1b0470 2021-02-10T11:24:15.012 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.005 7f102bfff700 1 --2- 172.21.2.112:0/298075457 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f102c1b0470 0x7f102c1b0870 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 rx=0x7f101c006700 tx=0x7f101c009b80).ready entity=mon.0 client_cookie=72cd66a04ae81a7 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:15.012 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.005 7f102affd700 1 -- 172.21.2.112:0/298075457 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f101c00c0c0 con 0x7f102c1b0470 2021-02-10T11:24:15.012 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.005 7f102affd700 1 -- 172.21.2.112:0/298075457 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f101c00c340 con 0x7f102c1b0470 2021-02-10T11:24:15.012 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.005 7f102affd700 1 -- 172.21.2.112:0/298075457 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f101c00c860 con 0x7f102c1b0470 2021-02-10T11:24:15.012 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.005 7f1032421700 1 -- 172.21.2.112:0/298075457 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f102c1b0470 msgr2=0x7f102c1b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:15.013 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.005 7f1032421700 1 --2- 172.21.2.112:0/298075457 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f102c1b0470 0x7f102c1b0870 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 rx=0x7f101c006700 tx=0x7f101c009b80).stop 2021-02-10T11:24:15.013 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.005 7f1032421700 1 -- 172.21.2.112:0/298075457 shutdown_connections 2021-02-10T11:24:15.013 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.005 7f1032421700 1 --2- 172.21.2.112:0/298075457 >> v2:172.21.2.120:3300/0 conn(0x7f102c0f20e0 0x7f102c0f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:15.013 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.005 7f1032421700 1 --2- 172.21.2.112:0/298075457 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f102c1b0470 0x7f102c1b0870 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:15.013 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.005 7f1032421700 1 -- 172.21.2.112:0/298075457 >> 172.21.2.112:0/298075457 conn(0x7f102c0ce3e0 msgr2=0x7f102c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:15.013 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.005 7f1032421700 1 -- 172.21.2.112:0/298075457 shutdown_connections 2021-02-10T11:24:15.013 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.005 7f1032421700 1 -- 172.21.2.112:0/298075457 wait complete. 2021-02-10T11:24:15.015 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.020 7f1032421700 1 Processor -- start 2021-02-10T11:24:15.015 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.020 7f1032421700 1 -- start start 2021-02-10T11:24:15.015 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.020 7f1032421700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f102c190920 0x7f102c0b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:15.015 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.020 7f1032421700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f102c0b8200 0x7f102c0b8600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:15.015 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.020 7f1032421700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f102c0c1d80 0x7f102c0c2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:15.015 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.020 7f1032421700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f102c190da0 con 0x7f102c0b8200 2021-02-10T11:24:15.015 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.020 7f1032421700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f102c0b8c20 con 0x7f102c0c1d80 2021-02-10T11:24:15.015 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.020 7f1032421700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f102c0b8e10 con 0x7f102c190920 2021-02-10T11:24:15.015 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.021 7f102b7fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f102c0b8200 0x7f102c0b8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:15.015 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.021 7f102b7fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f102c0b8200 0x7f102c0b8600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:45454/0 (socket says 172.21.2.112:45454) 2021-02-10T11:24:15.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.021 7f102b7fe700 1 -- 172.21.2.112:0/4084616086 learned_addr learned my addr 172.21.2.112:0/4084616086 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:24:15.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.021 7f102bfff700 1 --2- 172.21.2.112:0/4084616086 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f102c190920 0x7f102c0b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:15.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.021 7f1030a3e700 1 --2- 172.21.2.112:0/4084616086 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f102c0c1d80 0x7f102c0c2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:15.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.021 7f102b7fe700 1 -- 172.21.2.112:0/4084616086 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f102c190920 msgr2=0x7f102c0b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:15.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.021 7f102b7fe700 1 --2- 172.21.2.112:0/4084616086 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f102c190920 0x7f102c0b7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:15.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.021 7f102b7fe700 1 -- 172.21.2.112:0/4084616086 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f102c0c1d80 msgr2=0x7f102c0c2180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:15.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.021 7f102b7fe700 1 --2- 172.21.2.112:0/4084616086 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f102c0c1d80 0x7f102c0c2180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:15.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.021 7f102b7fe700 1 -- 172.21.2.112:0/4084616086 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f102c0c27d0 con 0x7f102c0b8200 2021-02-10T11:24:15.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.021 7f102bfff700 1 --2- 172.21.2.112:0/4084616086 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f102c190920 0x7f102c0b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:24:15.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.021 7f1030a3e700 1 --2- 172.21.2.112:0/4084616086 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f102c0c1d80 0x7f102c0c2180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:24:15.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.021 7f102b7fe700 1 --2- 172.21.2.112:0/4084616086 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f102c0b8200 0x7f102c0b8600 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 rx=0x7f1020006ed0 tx=0x7f102000b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:15.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.021 7f1028ff9700 1 -- 172.21.2.112:0/4084616086 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1020006020 con 0x7f102c0b8200 2021-02-10T11:24:15.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.022 7f1028ff9700 1 -- 172.21.2.112:0/4084616086 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10200009b0 con 0x7f102c0b8200 2021-02-10T11:24:15.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.022 7f1028ff9700 1 -- 172.21.2.112:0/4084616086 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f10200054b0 con 0x7f102c0b8200 2021-02-10T11:24:15.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.022 7f1032421700 1 -- 172.21.2.112:0/4084616086 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f102c0c27d0 con 0x7f102c0b8200 2021-02-10T11:24:15.017 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.022 7f1032421700 1 -- 172.21.2.112:0/4084616086 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f102c0c2d60 con 0x7f102c0b8200 2021-02-10T11:24:15.017 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.022 7f1028ff9700 1 -- 172.21.2.112:0/4084616086 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1020006020 con 0x7f102c0b8200 2021-02-10T11:24:15.017 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.022 7f1028ff9700 1 --2- 172.21.2.112:0/4084616086 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f100c0186c0 0x7f100c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:15.017 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.022 7f1028ff9700 1 -- 172.21.2.112:0/4084616086 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f1020032470 con 0x7f102c0b8200 2021-02-10T11:24:15.017 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.022 7f102bfff700 1 --2- 172.21.2.112:0/4084616086 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f100c0186c0 0x7f100c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:15.017 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.023 7f102bfff700 1 --2- 172.21.2.112:0/4084616086 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f100c0186c0 0x7f100c01abc0 crc :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:15.045 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.054 7f1032421700 1 -- 172.21.2.112:0/4084616086 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f102c105ae0 con 0x7f102c0b8200 2021-02-10T11:24:15.047 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.056 7f1028ff9700 1 -- 172.21.2.112:0/4084616086 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f102002cfa0 con 0x7f102c0b8200 2021-02-10T11:24:15.174 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.182 7f1032421700 1 -- 172.21.2.112:0/4084616086 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348627", "target": ["mgr", ""]}) v1 -- 0x7f102c0c2d60 con 0x7f100c0186c0 2021-02-10T11:24:15.200 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.210 7f1028ff9700 1 -- 172.21.2.112:0/4084616086 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f101c0090b0 con 0x7f100c0186c0 2021-02-10T11:24:15.206 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.215 7f1032421700 1 -- 172.21.2.112:0/4084616086 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f100c0186c0 msgr2=0x7f100c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:15.207 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.215 7f1032421700 1 --2- 172.21.2.112:0/4084616086 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f100c0186c0 0x7f100c01abc0 crc :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:15.207 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.215 7f1032421700 1 -- 172.21.2.112:0/4084616086 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f102c0b8200 msgr2=0x7f102c0b8600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:15.207 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.215 7f1032421700 1 --2- 172.21.2.112:0/4084616086 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f102c0b8200 0x7f102c0b8600 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 rx=0x7f1020006ed0 tx=0x7f102000b960).stop 2021-02-10T11:24:15.207 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.215 7f1032421700 1 -- 172.21.2.112:0/4084616086 shutdown_connections 2021-02-10T11:24:15.207 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.215 7f1032421700 1 --2- 172.21.2.112:0/4084616086 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f100c0186c0 0x7f100c01abc0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:15.207 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.215 7f1032421700 1 --2- 172.21.2.112:0/4084616086 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f102c0c1d80 0x7f102c0c2180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:15.207 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.215 7f1032421700 1 --2- 172.21.2.112:0/4084616086 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f102c0b8200 0x7f102c0b8600 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:15.208 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.215 7f1032421700 1 --2- 172.21.2.112:0/4084616086 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f102c190920 0x7f102c0b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:15.208 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.215 7f1032421700 1 -- 172.21.2.112:0/4084616086 >> 172.21.2.112:0/4084616086 conn(0x7f102c094e10 msgr2=0x7f102c0a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:15.208 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.216 7f1032421700 1 -- 172.21.2.112:0/4084616086 shutdown_connections 2021-02-10T11:24:15.208 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.216 7f1032421700 1 -- 172.21.2.112:0/4084616086 wait complete. 2021-02-10T11:24:15.215 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000348627 2021-02-10T11:24:15.650 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.641 7ff06faf0700 1 Processor -- start 2021-02-10T11:24:15.651 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.642 7ff06faf0700 1 -- start start 2021-02-10T11:24:15.651 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.642 7ff06faf0700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff0681b0470 0x7ff0681b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:15.652 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.642 7ff06faf0700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7ff0680ef610 0x7ff0680efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:15.652 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.642 7ff06faf0700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0680f0b50 con 0x7ff0680f0030 2021-02-10T11:24:15.652 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.642 7ff06faf0700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0680e52e0 con 0x7ff0680ef610 2021-02-10T11:24:15.652 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.642 7ff06faf0700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0680e5490 con 0x7ff0681b0470 2021-02-10T11:24:15.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.642 7ff06e10d700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7ff0680f0030 0x7ff0680f10d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:38480/0 (socket says 172.21.2.112:38480) 2021-02-10T11:24:15.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.642 7ff06e10d700 1 -- 172.21.2.112:0/3422334203 learned_addr learned my addr 172.21.2.112:0/3422334203 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:24:15.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.642 7ff06d90c700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff0681b0470 0x7ff0681b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:15.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.642 7ff06d10b700 1 --2- 172.21.2.112:0/3422334203 >> v2:172.21.2.120:3300/0 conn(0x7ff0680ef610 0x7ff0680efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:15.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.643 7ff06c90a700 1 -- 172.21.2.112:0/3422334203 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 76883296 0 0) 0x7ff060001440 con 0x7ff0680f0030 2021-02-10T11:24:15.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.643 7ff06c90a700 1 -- 172.21.2.112:0/3422334203 --> v1:172.21.2.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff054002fe0 con 0x7ff0680f0030 2021-02-10T11:24:15.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.643 7ff06d90c700 1 -- 172.21.2.112:0/3422334203 >> v2:172.21.2.120:3300/0 conn(0x7ff0680ef610 msgr2=0x7ff0680efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:15.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.643 7ff06d90c700 1 --2- 172.21.2.112:0/3422334203 >> v2:172.21.2.120:3300/0 conn(0x7ff0680ef610 0x7ff0680efa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:15.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.643 7ff06d90c700 1 -- 172.21.2.112:0/3422334203 >> v1:172.21.2.120:6789/0 conn(0x7ff0680f0030 legacy=0x7ff0680f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:15.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.643 7ff06d90c700 1 -- 172.21.2.112:0/3422334203 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0680f0430 con 0x7ff0681b0470 2021-02-10T11:24:15.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.643 7ff06d10b700 1 --2- 172.21.2.112:0/3422334203 >> v2:172.21.2.120:3300/0 conn(0x7ff0680ef610 0x7ff0680efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:24:15.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.643 7ff06d90c700 1 --2- 172.21.2.112:0/3422334203 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff0681b0470 0x7ff0681b0870 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7ff05c006650 tx=0x7ff05c009a80).ready entity=mon.2 client_cookie=78e9c8cc1556333d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:15.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.643 7ff06c90a700 1 -- 172.21.2.112:0/3422334203 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff05c0009b0 con 0x7ff0681b0470 2021-02-10T11:24:15.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.643 7ff06c90a700 1 -- 172.21.2.112:0/3422334203 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff05c00c0c0 con 0x7ff0681b0470 2021-02-10T11:24:15.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.643 7ff06c90a700 1 -- 172.21.2.112:0/3422334203 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff05c00c520 con 0x7ff0681b0470 2021-02-10T11:24:15.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.643 7ff06faf0700 1 -- 172.21.2.112:0/3422334203 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff0681b0470 msgr2=0x7ff0681b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:15.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.643 7ff06faf0700 1 --2- 172.21.2.112:0/3422334203 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff0681b0470 0x7ff0681b0870 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7ff05c006650 tx=0x7ff05c009a80).stop 2021-02-10T11:24:15.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.644 7ff06faf0700 1 -- 172.21.2.112:0/3422334203 shutdown_connections 2021-02-10T11:24:15.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.644 7ff06faf0700 1 --2- 172.21.2.112:0/3422334203 >> v2:172.21.2.120:3300/0 conn(0x7ff0680ef610 0x7ff0680efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:15.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.644 7ff06faf0700 1 --2- 172.21.2.112:0/3422334203 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff0681b0470 0x7ff0681b0870 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:15.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.644 7ff06faf0700 1 -- 172.21.2.112:0/3422334203 >> 172.21.2.112:0/3422334203 conn(0x7ff0680ce3e0 msgr2=0x7ff0680e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:15.657 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.644 7ff06faf0700 1 -- 172.21.2.112:0/3422334203 shutdown_connections 2021-02-10T11:24:15.657 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.644 7ff06faf0700 1 -- 172.21.2.112:0/3422334203 wait complete. 2021-02-10T11:24:15.660 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.659 7ff06faf0700 1 Processor -- start 2021-02-10T11:24:15.661 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.659 7ff06faf0700 1 -- start start 2021-02-10T11:24:15.661 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.659 7ff06faf0700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff0681a42e0 0x7ff06809ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:15.661 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.659 7ff06faf0700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff0681a46e0 0x7ff06809f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:15.661 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.659 7ff06faf0700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff06808ad30 0x7ff06808b130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:15.662 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.659 7ff06faf0700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff06808a7b0 con 0x7ff0681a42e0 2021-02-10T11:24:15.662 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.659 7ff06faf0700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0680a65f0 con 0x7ff0681a46e0 2021-02-10T11:24:15.662 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.659 7ff06faf0700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff06808a430 con 0x7ff06808ad30 2021-02-10T11:24:15.662 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.659 7ff06d10b700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff0681a46e0 0x7ff06809f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:15.662 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.659 7ff06e10d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff06808ad30 0x7ff06808b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:15.662 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.659 7ff06d90c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff0681a42e0 0x7ff06809ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:15.663 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.659 7ff06d10b700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff0681a46e0 0x7ff06809f310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3300/0 says I am v2:172.21.2.112:47744/0 (socket says 172.21.2.112:47744) 2021-02-10T11:24:15.663 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.659 7ff06d10b700 1 -- 172.21.2.112:0/3876720602 learned_addr learned my addr 172.21.2.112:0/3876720602 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:24:15.663 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.660 7ff06d10b700 1 -- 172.21.2.112:0/3876720602 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff06808ad30 msgr2=0x7ff06808b130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:15.663 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.660 7ff06d10b700 1 --2- 172.21.2.112:0/3876720602 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff06808ad30 0x7ff06808b130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:15.663 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.660 7ff06d10b700 1 -- 172.21.2.112:0/3876720602 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff0681a42e0 msgr2=0x7ff06809ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:15.663 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.660 7ff06d10b700 1 --2- 172.21.2.112:0/3876720602 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff0681a42e0 0x7ff06809ecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:15.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.660 7ff06d10b700 1 -- 172.21.2.112:0/3876720602 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff06809fa50 con 0x7ff0681a46e0 2021-02-10T11:24:15.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.660 7ff06d90c700 1 --2- 172.21.2.112:0/3876720602 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff0681a42e0 0x7ff06809ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:24:15.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.660 7ff06d10b700 1 --2- 172.21.2.112:0/3876720602 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff0681a46e0 0x7ff06809f310 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7ff064006f40 tx=0x7ff06400b990).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:15.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.661 7ff05a7fc700 1 -- 172.21.2.112:0/3876720602 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff06400bd80 con 0x7ff0681a46e0 2021-02-10T11:24:15.665 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.661 7ff05a7fc700 1 -- 172.21.2.112:0/3876720602 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff064000c90 con 0x7ff0681a46e0 2021-02-10T11:24:15.665 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.661 7ff05a7fc700 1 -- 172.21.2.112:0/3876720602 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff064002220 con 0x7ff0681a46e0 2021-02-10T11:24:15.666 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.661 7ff06faf0700 1 -- 172.21.2.112:0/3876720602 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff06809fa50 con 0x7ff0681a46e0 2021-02-10T11:24:15.666 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.661 7ff06faf0700 1 -- 172.21.2.112:0/3876720602 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff06809fa50 con 0x7ff0681a46e0 2021-02-10T11:24:15.666 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.662 7ff05a7fc700 1 -- 172.21.2.112:0/3876720602 <== mon.1 v2:172.21.2.120:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff064000c90 con 0x7ff0681a46e0 2021-02-10T11:24:15.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.662 7ff05a7fc700 1 --2- 172.21.2.112:0/3876720602 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff0500186c0 0x7ff05001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:15.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.662 7ff05a7fc700 1 -- 172.21.2.112:0/3876720602 <== mon.1 v2:172.21.2.120:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7ff064032560 con 0x7ff0681a46e0 2021-02-10T11:24:15.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.662 7ff06d90c700 1 --2- 172.21.2.112:0/3876720602 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff0500186c0 0x7ff05001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:15.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.663 7ff06d90c700 1 --2- 172.21.2.112:0/3876720602 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff0500186c0 0x7ff05001abc0 crc :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:15.684 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.692 7ff06faf0700 1 -- 172.21.2.112:0/3876720602 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff068105ae0 con 0x7ff0681a46e0 2021-02-10T11:24:15.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.695 7ff05a7fc700 1 -- 172.21.2.112:0/3876720602 <== mon.1 v2:172.21.2.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff064030d10 con 0x7ff0681a46e0 2021-02-10T11:24:15.818 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.826 7ff06faf0700 1 -- 172.21.2.112:0/3876720602 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348627", "target": ["mgr", ""]}) v1 -- 0x7ff0680f0b50 con 0x7ff0500186c0 2021-02-10T11:24:15.821 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.830 7ff05a7fc700 1 -- 172.21.2.112:0/3876720602 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7ff05c009270 con 0x7ff0500186c0 2021-02-10T11:24:15.825 INFO:teuthology.orchestra.run.gibba012.stdout:/volumes/_nogroup/subvolume_0000000000348627/0fafc15d-737e-40ca-805d-afbd52434edc 2021-02-10T11:24:15.826 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.835 7ff06faf0700 1 -- 172.21.2.112:0/3876720602 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff0500186c0 msgr2=0x7ff05001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:15.826 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.835 7ff06faf0700 1 --2- 172.21.2.112:0/3876720602 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff0500186c0 0x7ff05001abc0 crc :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:15.827 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.835 7ff06faf0700 1 -- 172.21.2.112:0/3876720602 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff0681a46e0 msgr2=0x7ff06809f310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:15.827 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.835 7ff06faf0700 1 --2- 172.21.2.112:0/3876720602 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff0681a46e0 0x7ff06809f310 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7ff064006f40 tx=0x7ff06400b990).stop 2021-02-10T11:24:15.827 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.835 7ff06faf0700 1 -- 172.21.2.112:0/3876720602 shutdown_connections 2021-02-10T11:24:15.827 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.835 7ff06faf0700 1 --2- 172.21.2.112:0/3876720602 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff0500186c0 0x7ff05001abc0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:15.827 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.835 7ff06faf0700 1 --2- 172.21.2.112:0/3876720602 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff0681a46e0 0x7ff06809f310 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:15.827 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.835 7ff06faf0700 1 --2- 172.21.2.112:0/3876720602 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff06808ad30 0x7ff06808b130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:15.827 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.835 7ff06faf0700 1 --2- 172.21.2.112:0/3876720602 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff0681a42e0 0x7ff06809ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:15.827 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.835 7ff06faf0700 1 -- 172.21.2.112:0/3876720602 >> 172.21.2.112:0/3876720602 conn(0x7ff068095a70 msgr2=0x7ff0680bf050 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:15.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.835 7ff06faf0700 1 -- 172.21.2.112:0/3876720602 shutdown_connections 2021-02-10T11:24:15.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:15.835 7ff06faf0700 1 -- 172.21.2.112:0/3876720602 wait complete. 2021-02-10T11:24:15.835 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000348627 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000348627/0fafc15d-737e-40ca-805d-afbd52434edc 2021-02-10T11:24:15.835 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:15.835 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:15.835 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:16.174 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:24:16.174 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:24:16.174 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348627/0fafc15d-737e-40ca-805d-afbd52434edc/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:24:16.555 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:24:16.555 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:24:16.555 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0227157 s, 46.2 MB/s 2021-02-10T11:24:16.556 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:16.556 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:16.556 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:16.891 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:24:16.891 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:24:16.891 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348627/0fafc15d-737e-40ca-805d-afbd52434edc/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:24:17.268 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:24:17.268 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:24:17.268 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.025558 s, 41.0 MB/s 2021-02-10T11:24:17.269 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:17.269 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:17.270 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:17.605 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:24:17.605 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:24:17.605 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348627/0fafc15d-737e-40ca-805d-afbd52434edc/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:24:17.982 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:24:17.983 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:24:17.983 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0252428 s, 41.5 MB/s 2021-02-10T11:24:17.984 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:17.984 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:17.984 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:18.320 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:24:18.320 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:24:18.321 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348627/0fafc15d-737e-40ca-805d-afbd52434edc/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:24:18.697 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:24:18.698 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:24:18.698 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0257611 s, 40.7 MB/s 2021-02-10T11:24:18.699 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:18.699 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:18.699 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:19.035 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:24:19.035 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:24:19.035 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348627/0fafc15d-737e-40ca-805d-afbd52434edc/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:24:19.405 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:24:19.405 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:24:19.405 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.022203 s, 47.2 MB/s 2021-02-10T11:24:19.406 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:19.406 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:19.406 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:19.741 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:24:19.741 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:24:19.741 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348627/0fafc15d-737e-40ca-805d-afbd52434edc/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:24:20.117 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:24:20.117 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:24:20.117 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0252187 s, 41.6 MB/s 2021-02-10T11:24:20.118 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:20.118 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:20.118 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:20.454 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:24:20.455 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:24:20.455 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348627/0fafc15d-737e-40ca-805d-afbd52434edc/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:24:20.830 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:24:20.831 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:24:20.831 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0237892 s, 44.1 MB/s 2021-02-10T11:24:20.832 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:20.832 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:20.832 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:21.168 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:24:21.168 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:24:21.168 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348627/0fafc15d-737e-40ca-805d-afbd52434edc/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:24:21.543 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:24:21.543 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:24:21.543 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0235519 s, 44.5 MB/s 2021-02-10T11:24:21.544 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:21.544 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:21.545 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:21.879 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:24:21.879 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:24:21.880 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348627/0fafc15d-737e-40ca-805d-afbd52434edc/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:24:22.255 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:24:22.255 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:24:22.255 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0236903 s, 44.3 MB/s 2021-02-10T11:24:22.256 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:22.256 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:22.257 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:22.593 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:24:22.594 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:24:22.594 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348627/0fafc15d-737e-40ca-805d-afbd52434edc/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:24:22.969 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:24:22.970 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:24:22.970 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0229992 s, 45.6 MB/s 2021-02-10T11:24:22.971 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000348628 2021-02-10T11:24:23.402 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:23.394 7fb64f0ed700 1 Processor -- start 2021-02-10T11:24:23.402 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:23.394 7fb64f0ed700 1 -- start start 2021-02-10T11:24:23.402 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:23.395 7fb64f0ed700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fb6481b0470 0x7fb6481b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:23.402 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:23.395 7fb64f0ed700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb6480f20e0 0x7fb6480f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:23.403 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:23.395 7fb64f0ed700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6480f0b50 con 0x7fb6480ef610 2021-02-10T11:24:23.403 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:23.395 7fb64f0ed700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6480f0820 con 0x7fb6481b0470 2021-02-10T11:24:23.403 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:23.395 7fb64f0ed700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6480f24e0 con 0x7fb6480f20e0 2021-02-10T11:24:23.403 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:23.395 7fb64cf09700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fb6481b0470 0x7fb6481b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:23.403 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:23.395 7fb647fff700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fb6480ef610 0x7fb6480efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:44934/0 (socket says 172.21.2.112:44934) 2021-02-10T11:24:23.403 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:23.395 7fb647fff700 1 -- 172.21.2.112:0/255546086 learned_addr learned my addr 172.21.2.112:0/255546086 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:24:23.403 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:23.395 7fb64d70a700 1 --2- 172.21.2.112:0/255546086 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb6480f20e0 0x7fb6480f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:23.403 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:23.395 7fb64cf09700 1 -- 172.21.2.112:0/255546086 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb6480f20e0 msgr2=0x7fb6480f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:23.403 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:23.395 7fb64cf09700 1 --2- 172.21.2.112:0/255546086 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb6480f20e0 0x7fb6480f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:23.403 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:23.395 7fb64cf09700 1 -- 172.21.2.112:0/255546086 >> v1:172.21.2.112:6789/0 conn(0x7fb6480ef610 legacy=0x7fb6480efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:23.403 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:23.395 7fb64cf09700 1 -- 172.21.2.112:0/255546086 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6480e52e0 con 0x7fb6481b0470 2021-02-10T11:24:23.403 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:23.396 7fb64cf09700 1 --2- 172.21.2.112:0/255546086 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb6481b0470 0x7fb6481b0870 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 rx=0x7fb630006650 tx=0x7fb630009a80).ready entity=mon.0 client_cookie=d104bae95706eae3 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:23.404 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:23.396 7fb6477fe700 1 -- 172.21.2.112:0/255546086 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb6300009b0 con 0x7fb6481b0470 2021-02-10T11:24:23.404 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:23.396 7fb6477fe700 1 -- 172.21.2.112:0/255546086 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb63000c0c0 con 0x7fb6481b0470 2021-02-10T11:24:23.404 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:23.396 7fb6477fe700 1 -- 172.21.2.112:0/255546086 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb63000c520 con 0x7fb6481b0470 2021-02-10T11:24:23.404 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:23.396 7fb64f0ed700 1 -- 172.21.2.112:0/255546086 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb6481b0470 msgr2=0x7fb6481b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:23.404 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:23.396 7fb64f0ed700 1 --2- 172.21.2.112:0/255546086 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb6481b0470 0x7fb6481b0870 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 rx=0x7fb630006650 tx=0x7fb630009a80).stop 2021-02-10T11:24:23.404 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:23.397 7fb64f0ed700 1 -- 172.21.2.112:0/255546086 shutdown_connections 2021-02-10T11:24:23.404 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:23.397 7fb64f0ed700 1 --2- 172.21.2.112:0/255546086 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb6480f20e0 0x7fb6480f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:23.404 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:23.397 7fb64f0ed700 1 --2- 172.21.2.112:0/255546086 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb6481b0470 0x7fb6481b0870 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:23.404 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:23.397 7fb64f0ed700 1 -- 172.21.2.112:0/255546086 >> 172.21.2.112:0/255546086 conn(0x7fb6480ce3e0 msgr2=0x7fb6480e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:23.404 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:23.397 7fb64f0ed700 1 -- 172.21.2.112:0/255546086 shutdown_connections 2021-02-10T11:24:23.404 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:23.397 7fb64f0ed700 1 -- 172.21.2.112:0/255546086 wait complete. 2021-02-10T11:24:23.406 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:23.411 7fb64f0ed700 1 Processor -- start 2021-02-10T11:24:23.406 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:23.411 7fb64f0ed700 1 -- start start 2021-02-10T11:24:23.406 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:23.412 7fb64f0ed700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb64808d480 0x7fb648190740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:23.406 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:23.412 7fb64f0ed700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb64810a1d0 0x7fb6480a4840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:23.406 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:23.412 7fb64f0ed700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb64810a5d0 0x7fb6480a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:23.406 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:23.412 7fb64f0ed700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6480a5480 con 0x7fb64808d480 2021-02-10T11:24:23.406 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:23.412 7fb64f0ed700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb64808cf00 con 0x7fb64810a1d0 2021-02-10T11:24:23.407 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:23.412 7fb64f0ed700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb64808d880 con 0x7fb64810a5d0 2021-02-10T11:24:23.407 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:23.412 7fb64cf09700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb64808d480 0x7fb648190740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:23.407 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:23.412 7fb64cf09700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb64808d480 0x7fb648190740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:45480/0 (socket says 172.21.2.112:45480) 2021-02-10T11:24:23.407 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:23.412 7fb64cf09700 1 -- 172.21.2.112:0/3966964446 learned_addr learned my addr 172.21.2.112:0/3966964446 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:24:23.407 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:23.412 7fb64cf09700 1 -- 172.21.2.112:0/3966964446 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb64810a5d0 msgr2=0x7fb6480a4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:23.407 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:23.412 7fb64cf09700 1 --2- 172.21.2.112:0/3966964446 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb64810a5d0 0x7fb6480a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:23.407 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:23.412 7fb64cf09700 1 -- 172.21.2.112:0/3966964446 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb64810a1d0 msgr2=0x7fb6480a4840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:23.407 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:23.412 7fb64cf09700 1 --2- 172.21.2.112:0/3966964446 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb64810a1d0 0x7fb6480a4840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:23.407 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:23.412 7fb64cf09700 1 -- 172.21.2.112:0/3966964446 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6481069f0 con 0x7fb64808d480 2021-02-10T11:24:23.407 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:23.412 7fb64cf09700 1 --2- 172.21.2.112:0/3966964446 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb64808d480 0x7fb648190740 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 rx=0x7fb630009630 tx=0x7fb63000b900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:23.407 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:23.412 7fb6457fa700 1 -- 172.21.2.112:0/3966964446 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb6300093a0 con 0x7fb64808d480 2021-02-10T11:24:23.407 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:23.412 7fb6457fa700 1 -- 172.21.2.112:0/3966964446 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb63000bcf0 con 0x7fb64808d480 2021-02-10T11:24:23.407 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:23.412 7fb6457fa700 1 -- 172.21.2.112:0/3966964446 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb630005560 con 0x7fb64808d480 2021-02-10T11:24:23.408 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:23.413 7fb64f0ed700 1 -- 172.21.2.112:0/3966964446 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6481069f0 con 0x7fb64808d480 2021-02-10T11:24:23.408 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:23.413 7fb64f0ed700 1 -- 172.21.2.112:0/3966964446 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6481069f0 con 0x7fb64808d480 2021-02-10T11:24:23.408 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:23.413 7fb6457fa700 1 -- 172.21.2.112:0/3966964446 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb630005740 con 0x7fb64808d480 2021-02-10T11:24:23.408 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:23.413 7fb6457fa700 1 --2- 172.21.2.112:0/3966964446 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb62c0186c0 0x7fb62c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:23.408 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:23.413 7fb6457fa700 1 -- 172.21.2.112:0/3966964446 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fb630032560 con 0x7fb64808d480 2021-02-10T11:24:23.408 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:23.414 7fb647fff700 1 --2- 172.21.2.112:0/3966964446 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb62c0186c0 0x7fb62c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:23.408 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:23.414 7fb647fff700 1 --2- 172.21.2.112:0/3966964446 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb62c0186c0 0x7fb62c01abc0 crc :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:23.436 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:23.444 7fb64f0ed700 1 -- 172.21.2.112:0/3966964446 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb648105ae0 con 0x7fb64808d480 2021-02-10T11:24:23.438 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:23.447 7fb6457fa700 1 -- 172.21.2.112:0/3966964446 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb63002ef10 con 0x7fb64808d480 2021-02-10T11:24:23.565 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:23.573 7fb64f0ed700 1 -- 172.21.2.112:0/3966964446 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348628", "target": ["mgr", ""]}) v1 -- 0x7fb6480f0b50 con 0x7fb62c0186c0 2021-02-10T11:24:23.592 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:23.600 7fb6457fa700 1 -- 172.21.2.112:0/3966964446 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fb638001210 con 0x7fb62c0186c0 2021-02-10T11:24:23.598 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:23.607 7fb64f0ed700 1 -- 172.21.2.112:0/3966964446 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb62c0186c0 msgr2=0x7fb62c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:23.598 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:23.607 7fb64f0ed700 1 --2- 172.21.2.112:0/3966964446 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb62c0186c0 0x7fb62c01abc0 crc :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:23.598 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:23.607 7fb64f0ed700 1 -- 172.21.2.112:0/3966964446 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb64808d480 msgr2=0x7fb648190740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:23.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:23.607 7fb64f0ed700 1 --2- 172.21.2.112:0/3966964446 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb64808d480 0x7fb648190740 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 rx=0x7fb630009630 tx=0x7fb63000b900).stop 2021-02-10T11:24:23.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:23.607 7fb64f0ed700 1 -- 172.21.2.112:0/3966964446 shutdown_connections 2021-02-10T11:24:23.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:23.607 7fb64f0ed700 1 --2- 172.21.2.112:0/3966964446 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb62c0186c0 0x7fb62c01abc0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:23.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:23.607 7fb64f0ed700 1 --2- 172.21.2.112:0/3966964446 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb64810a1d0 0x7fb6480a4840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:23.600 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:23.607 7fb64f0ed700 1 --2- 172.21.2.112:0/3966964446 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb64810a5d0 0x7fb6480a4e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:23.600 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:23.607 7fb64f0ed700 1 --2- 172.21.2.112:0/3966964446 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb64808d480 0x7fb648190740 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:23.600 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:23.607 7fb64f0ed700 1 -- 172.21.2.112:0/3966964446 >> 172.21.2.112:0/3966964446 conn(0x7fb6480bbf60 msgr2=0x7fb648105d10 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:23.600 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:23.608 7fb64f0ed700 1 -- 172.21.2.112:0/3966964446 shutdown_connections 2021-02-10T11:24:23.600 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:23.608 7fb64f0ed700 1 -- 172.21.2.112:0/3966964446 wait complete. 2021-02-10T11:24:23.607 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000348628 2021-02-10T11:24:24.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:24.032 7faa07558700 1 Processor -- start 2021-02-10T11:24:24.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:24.033 7faa07558700 1 -- start start 2021-02-10T11:24:24.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:24.033 7faa07558700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7faa000f05f0 0x7faa000f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:24.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:24.033 7faa07558700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa001b0c70 con 0x7faa001b0470 2021-02-10T11:24:24.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:24.033 7faa07558700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa000f0b50 con 0x7faa001b0870 2021-02-10T11:24:24.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:24.033 7faa07558700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa000e5ea0 con 0x7faa000f05f0 2021-02-10T11:24:24.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:24.033 7faa05374700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7faa001b0470 0x7faa000ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:44948/0 (socket says 172.21.2.112:44948) 2021-02-10T11:24:24.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:24.033 7faa05374700 1 -- 172.21.2.112:0/3493430394 learned_addr learned my addr 172.21.2.112:0/3493430394 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:24:24.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:24.033 7faa05b75700 1 --2- 172.21.2.112:0/3493430394 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7faa000f05f0 0x7faa000f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:24.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:24.034 7fa9f3fff700 1 -- 172.21.2.112:0/3493430394 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2075402792 0 0) 0x7fa9fc0010b0 con 0x7faa001b0470 2021-02-10T11:24:24.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:24.034 7fa9f3fff700 1 -- 172.21.2.112:0/3493430394 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9ec002fe0 con 0x7faa001b0470 2021-02-10T11:24:24.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:24.034 7fa9f3fff700 1 -- 172.21.2.112:0/3493430394 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3488236711 0 0) 0x7fa9f4000ff0 con 0x7faa001b0870 2021-02-10T11:24:24.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:24.034 7fa9f3fff700 1 -- 172.21.2.112:0/3493430394 --> v1:172.21.2.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9ec004370 con 0x7faa001b0870 2021-02-10T11:24:24.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:24.034 7faa05b75700 1 -- 172.21.2.112:0/3493430394 >> v1:172.21.2.120:6789/0 conn(0x7faa001b0870 legacy=0x7faa000efe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:24.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:24.034 7faa05b75700 1 -- 172.21.2.112:0/3493430394 >> v1:172.21.2.112:6789/0 conn(0x7faa001b0470 legacy=0x7faa000ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:24.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:24.034 7faa05b75700 1 -- 172.21.2.112:0/3493430394 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa000e52e0 con 0x7faa000f05f0 2021-02-10T11:24:24.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:24.034 7faa05b75700 1 --2- 172.21.2.112:0/3493430394 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7faa000f05f0 0x7faa000f20e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7fa9f8006e50 tx=0x7fa9f8009e70).ready entity=mon.2 client_cookie=106db4a46326b78e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:24.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:24.034 7fa9f3fff700 1 -- 172.21.2.112:0/3493430394 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa9f800b510 con 0x7faa000f05f0 2021-02-10T11:24:24.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:24.034 7fa9f3fff700 1 -- 172.21.2.112:0/3493430394 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9f8006790 con 0x7faa000f05f0 2021-02-10T11:24:24.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:24.034 7fa9f3fff700 1 -- 172.21.2.112:0/3493430394 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa9f800b940 con 0x7faa000f05f0 2021-02-10T11:24:24.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:24.034 7faa07558700 1 -- 172.21.2.112:0/3493430394 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7faa000f05f0 msgr2=0x7faa000f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:24.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:24.034 7faa07558700 1 --2- 172.21.2.112:0/3493430394 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7faa000f05f0 0x7faa000f20e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7fa9f8006e50 tx=0x7fa9f8009e70).stop 2021-02-10T11:24:24.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:24.035 7faa07558700 1 -- 172.21.2.112:0/3493430394 shutdown_connections 2021-02-10T11:24:24.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:24.035 7faa07558700 1 --2- 172.21.2.112:0/3493430394 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7faa000f05f0 0x7faa000f20e0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:24.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:24.035 7faa07558700 1 -- 172.21.2.112:0/3493430394 >> 172.21.2.112:0/3493430394 conn(0x7faa000ce3e0 msgr2=0x7faa000e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:24.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:24.035 7faa07558700 1 -- 172.21.2.112:0/3493430394 shutdown_connections 2021-02-10T11:24:24.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:24.035 7faa07558700 1 -- 172.21.2.112:0/3493430394 wait complete. 2021-02-10T11:24:24.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:24.050 7faa07558700 1 Processor -- start 2021-02-10T11:24:24.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:24.050 7faa07558700 1 -- start start 2021-02-10T11:24:24.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:24.050 7faa07558700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faa000a0420 0x7faa000a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:24.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:24.050 7faa07558700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7faa00189d00 0x7faa0018a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:24.045 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:24.050 7faa07558700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7faa0018a720 0x7faa000b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:24.045 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:24.050 7faa07558700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa00189780 con 0x7faa000a0420 2021-02-10T11:24:24.045 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:24.050 7faa07558700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa0018ab20 con 0x7faa0018a720 2021-02-10T11:24:24.045 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:24.050 7faa07558700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa000a1140 con 0x7faa00189d00 2021-02-10T11:24:24.045 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:24.050 7faa05374700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faa000a0420 0x7faa000a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:24.045 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:24.050 7faa05374700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faa000a0420 0x7faa000a0820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:45494/0 (socket says 172.21.2.112:45494) 2021-02-10T11:24:24.045 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:24.050 7faa05374700 1 -- 172.21.2.112:0/2950030931 learned_addr learned my addr 172.21.2.112:0/2950030931 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:24:24.045 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:24.051 7faa04b73700 1 --2- 172.21.2.112:0/2950030931 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7faa00189d00 0x7faa0018a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:24.045 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:24.051 7faa05b75700 1 --2- 172.21.2.112:0/2950030931 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7faa0018a720 0x7faa000b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:24.045 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:24.051 7faa05374700 1 -- 172.21.2.112:0/2950030931 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7faa00189d00 msgr2=0x7faa0018a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:24.045 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:24.051 7faa05374700 1 --2- 172.21.2.112:0/2950030931 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7faa00189d00 0x7faa0018a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:24.045 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:24.051 7faa05374700 1 -- 172.21.2.112:0/2950030931 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7faa0018a720 msgr2=0x7faa000b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:24.045 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:24.051 7faa05374700 1 --2- 172.21.2.112:0/2950030931 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7faa0018a720 0x7faa000b7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:24.045 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:24.051 7faa05374700 1 -- 172.21.2.112:0/2950030931 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa000b8200 con 0x7faa000a0420 2021-02-10T11:24:24.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:24.051 7faa05374700 1 --2- 172.21.2.112:0/2950030931 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faa000a0420 0x7faa000a0820 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 rx=0x7fa9fc001c60 tx=0x7fa9fc00a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:24.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:24.051 7fa9f1ffb700 1 -- 172.21.2.112:0/2950030931 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa9fc000e00 con 0x7faa000a0420 2021-02-10T11:24:24.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:24.051 7fa9f1ffb700 1 -- 172.21.2.112:0/2950030931 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9fc00acc0 con 0x7faa000a0420 2021-02-10T11:24:24.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:24.051 7fa9f1ffb700 1 -- 172.21.2.112:0/2950030931 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa9fc004540 con 0x7faa000a0420 2021-02-10T11:24:24.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:24.051 7faa07558700 1 -- 172.21.2.112:0/2950030931 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa000b8200 con 0x7faa000a0420 2021-02-10T11:24:24.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:24.051 7faa07558700 1 -- 172.21.2.112:0/2950030931 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa000b8790 con 0x7faa000a0420 2021-02-10T11:24:24.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:24.052 7fa9f1ffb700 1 -- 172.21.2.112:0/2950030931 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa9fc004140 con 0x7faa000a0420 2021-02-10T11:24:24.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:24.052 7fa9f1ffb700 1 --2- 172.21.2.112:0/2950030931 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa9e00186c0 0x7fa9e001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:24.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:24.052 7fa9f1ffb700 1 -- 172.21.2.112:0/2950030931 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fa9fc0323a0 con 0x7faa000a0420 2021-02-10T11:24:24.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:24.052 7faa04b73700 1 --2- 172.21.2.112:0/2950030931 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa9e00186c0 0x7fa9e001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:24.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:24.052 7faa04b73700 1 --2- 172.21.2.112:0/2950030931 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa9e00186c0 0x7fa9e001abc0 crc :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:24.074 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:24.083 7faa07558700 1 -- 172.21.2.112:0/2950030931 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa00105ae0 con 0x7faa000a0420 2021-02-10T11:24:24.076 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:24.085 7fa9f1ffb700 1 -- 172.21.2.112:0/2950030931 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa9fc0009b0 con 0x7faa000a0420 2021-02-10T11:24:24.206 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:24.215 7faa07558700 1 -- 172.21.2.112:0/2950030931 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348628", "target": ["mgr", ""]}) v1 -- 0x7faa000b8790 con 0x7fa9e00186c0 2021-02-10T11:24:24.209 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:24.218 7fa9f1ffb700 1 -- 172.21.2.112:0/2950030931 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7fa9f40069e0 con 0x7fa9e00186c0 2021-02-10T11:24:24.209 INFO:teuthology.orchestra.run.gibba012.stdout:/volumes/_nogroup/subvolume_0000000000348628/a23fd001-6145-4007-9088-940dcc6d2d97 2021-02-10T11:24:24.210 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:24.219 7faa07558700 1 -- 172.21.2.112:0/2950030931 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa9e00186c0 msgr2=0x7fa9e001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:24.210 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:24.219 7faa07558700 1 --2- 172.21.2.112:0/2950030931 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa9e00186c0 0x7fa9e001abc0 crc :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:24.210 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:24.219 7faa07558700 1 -- 172.21.2.112:0/2950030931 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faa000a0420 msgr2=0x7faa000a0820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:24.210 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:24.219 7faa07558700 1 --2- 172.21.2.112:0/2950030931 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faa000a0420 0x7faa000a0820 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 rx=0x7fa9fc001c60 tx=0x7fa9fc00a8d0).stop 2021-02-10T11:24:24.211 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:24.219 7faa07558700 1 -- 172.21.2.112:0/2950030931 shutdown_connections 2021-02-10T11:24:24.211 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:24.219 7faa07558700 1 --2- 172.21.2.112:0/2950030931 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa9e00186c0 0x7fa9e001abc0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:24.211 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:24.219 7faa07558700 1 --2- 172.21.2.112:0/2950030931 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7faa0018a720 0x7faa000b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:24.211 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:24.219 7faa07558700 1 --2- 172.21.2.112:0/2950030931 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7faa00189d00 0x7faa0018a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:24.211 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:24.219 7faa07558700 1 --2- 172.21.2.112:0/2950030931 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faa000a0420 0x7faa000a0820 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:24.211 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:24.219 7faa07558700 1 -- 172.21.2.112:0/2950030931 >> 172.21.2.112:0/2950030931 conn(0x7faa00094e10 msgr2=0x7faa000a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:24.211 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:24.219 7faa07558700 1 -- 172.21.2.112:0/2950030931 shutdown_connections 2021-02-10T11:24:24.211 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:24.219 7faa07558700 1 -- 172.21.2.112:0/2950030931 wait complete. 2021-02-10T11:24:24.218 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000348628 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000348628/a23fd001-6145-4007-9088-940dcc6d2d97 2021-02-10T11:24:24.219 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:24.219 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:24.219 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:24.554 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:24:24.554 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:24:24.555 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348628/a23fd001-6145-4007-9088-940dcc6d2d97/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:24:24.933 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:24:24.933 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:24:24.933 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0240639 s, 43.6 MB/s 2021-02-10T11:24:24.934 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:24.934 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:24.934 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:25.270 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:24:25.271 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:24:25.271 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348628/a23fd001-6145-4007-9088-940dcc6d2d97/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:24:25.647 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:24:25.647 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:24:25.647 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0252635 s, 41.5 MB/s 2021-02-10T11:24:25.648 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:25.648 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:25.648 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:25.983 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:24:25.983 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:24:25.983 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348628/a23fd001-6145-4007-9088-940dcc6d2d97/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:24:26.361 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:24:26.361 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:24:26.361 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0266074 s, 39.4 MB/s 2021-02-10T11:24:26.362 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:26.362 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:26.363 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:26.698 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:24:26.698 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:24:26.698 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348628/a23fd001-6145-4007-9088-940dcc6d2d97/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:24:27.073 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:24:27.073 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:24:27.073 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0239059 s, 43.9 MB/s 2021-02-10T11:24:27.074 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:27.074 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:27.074 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:27.410 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:24:27.410 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:24:27.411 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348628/a23fd001-6145-4007-9088-940dcc6d2d97/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:24:27.787 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:24:27.787 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:24:27.787 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.023357 s, 44.9 MB/s 2021-02-10T11:24:27.788 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:27.788 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:27.788 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:28.122 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:24:28.122 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:24:28.123 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348628/a23fd001-6145-4007-9088-940dcc6d2d97/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:24:28.497 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:24:28.497 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:24:28.497 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0234915 s, 44.6 MB/s 2021-02-10T11:24:28.498 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:28.499 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:28.499 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:28.834 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:24:28.834 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:24:28.834 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348628/a23fd001-6145-4007-9088-940dcc6d2d97/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:24:29.208 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:24:29.208 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:24:29.209 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0234342 s, 44.7 MB/s 2021-02-10T11:24:29.209 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:29.210 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:29.210 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:29.544 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:24:29.544 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:24:29.544 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348628/a23fd001-6145-4007-9088-940dcc6d2d97/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:24:29.918 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:24:29.918 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:24:29.919 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0253061 s, 41.4 MB/s 2021-02-10T11:24:29.919 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:29.920 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:29.920 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:30.256 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:24:30.256 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:24:30.256 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348628/a23fd001-6145-4007-9088-940dcc6d2d97/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:24:30.634 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:24:30.634 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:24:30.634 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0256268 s, 40.9 MB/s 2021-02-10T11:24:30.635 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:30.636 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:30.636 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:30.971 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:24:30.971 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:24:30.971 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348628/a23fd001-6145-4007-9088-940dcc6d2d97/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:24:31.347 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:24:31.348 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:24:31.348 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0254029 s, 41.3 MB/s 2021-02-10T11:24:31.349 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000348629 2021-02-10T11:24:31.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:31.772 7f1c6b129700 1 Processor -- start 2021-02-10T11:24:31.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:31.772 7f1c6b129700 1 -- start start 2021-02-10T11:24:31.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:31.772 7f1c6b129700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f1c641b0470 0x7f1c641b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:31.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:31.772 7f1c6b129700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1c640ef610 0x7f1c640efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:31.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:31.772 7f1c6b129700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c640f0b50 con 0x7f1c640f0030 2021-02-10T11:24:31.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:31.772 7f1c6b129700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c640e52e0 con 0x7f1c641b0470 2021-02-10T11:24:31.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:31.772 7f1c6b129700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c640e54d0 con 0x7f1c640ef610 2021-02-10T11:24:31.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:31.773 7f1c68f45700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f1c641b0470 0x7f1c641b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:31.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:31.773 7f1c68f45700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f1c641b0470 0x7f1c641b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:45500/0 (socket says 172.21.2.112:45500) 2021-02-10T11:24:31.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:31.773 7f1c68f45700 1 -- 172.21.2.112:0/1938384377 learned_addr learned my addr 172.21.2.112:0/1938384377 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:24:31.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:31.773 7f1c68f45700 1 -- 172.21.2.112:0/1938384377 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1c640ef610 msgr2=0x7f1c640efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:31.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:31.773 7f1c68f45700 1 --2- 172.21.2.112:0/1938384377 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1c640ef610 0x7f1c640efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:31.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:31.773 7f1c68f45700 1 -- 172.21.2.112:0/1938384377 >> v1:172.21.2.112:6789/0 conn(0x7f1c640f0030 legacy=0x7f1c640f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:31.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:31.773 7f1c68f45700 1 -- 172.21.2.112:0/1938384377 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c640f0430 con 0x7f1c641b0470 2021-02-10T11:24:31.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:31.773 7f1c68f45700 1 --2- 172.21.2.112:0/1938384377 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1c641b0470 0x7f1c641b0870 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 rx=0x7f1c4c006700 tx=0x7f1c4c009b80).ready entity=mon.0 client_cookie=7c05bd34497c5623 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:31.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:31.773 7f1c637fe700 1 -- 172.21.2.112:0/1938384377 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1c4c00c0c0 con 0x7f1c641b0470 2021-02-10T11:24:31.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:31.774 7f1c637fe700 1 -- 172.21.2.112:0/1938384377 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c4c00c440 con 0x7f1c641b0470 2021-02-10T11:24:31.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:31.774 7f1c637fe700 1 -- 172.21.2.112:0/1938384377 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1c4c00c920 con 0x7f1c641b0470 2021-02-10T11:24:31.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:31.774 7f1c6b129700 1 -- 172.21.2.112:0/1938384377 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1c641b0470 msgr2=0x7f1c641b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:31.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:31.774 7f1c6b129700 1 --2- 172.21.2.112:0/1938384377 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1c641b0470 0x7f1c641b0870 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 rx=0x7f1c4c006700 tx=0x7f1c4c009b80).stop 2021-02-10T11:24:31.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:31.774 7f1c6b129700 1 -- 172.21.2.112:0/1938384377 shutdown_connections 2021-02-10T11:24:31.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:31.774 7f1c6b129700 1 --2- 172.21.2.112:0/1938384377 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1c640ef610 0x7f1c640efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:31.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:31.774 7f1c6b129700 1 --2- 172.21.2.112:0/1938384377 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1c641b0470 0x7f1c641b0870 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:31.782 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:31.774 7f1c6b129700 1 -- 172.21.2.112:0/1938384377 >> 172.21.2.112:0/1938384377 conn(0x7f1c640ce3e0 msgr2=0x7f1c640e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:31.782 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:31.774 7f1c6b129700 1 -- 172.21.2.112:0/1938384377 shutdown_connections 2021-02-10T11:24:31.782 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:31.774 7f1c6b129700 1 -- 172.21.2.112:0/1938384377 wait complete. 2021-02-10T11:24:31.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:31.789 7f1c6b129700 1 Processor -- start 2021-02-10T11:24:31.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:31.789 7f1c6b129700 1 -- start start 2021-02-10T11:24:31.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:31.789 7f1c6b129700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1c640b7ad0 0x7f1c640f0770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:31.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:31.789 7f1c6b129700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1c64092c40 0x7f1c64093040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:31.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:31.789 7f1c6b129700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1c64093660 0x7f1c640eef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:31.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:31.789 7f1c6b129700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c640c0460 con 0x7f1c64093660 2021-02-10T11:24:31.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:31.789 7f1c6b129700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c640b7550 con 0x7f1c64092c40 2021-02-10T11:24:31.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:31.789 7f1c6b129700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c640b7ed0 con 0x7f1c640b7ad0 2021-02-10T11:24:31.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:31.790 7f1c69746700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1c64093660 0x7f1c640eef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:31.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:31.790 7f1c69746700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1c64093660 0x7f1c640eef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:45508/0 (socket says 172.21.2.112:45508) 2021-02-10T11:24:31.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:31.790 7f1c69746700 1 -- 172.21.2.112:0/450312531 learned_addr learned my addr 172.21.2.112:0/450312531 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:24:31.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:31.790 7f1c68f45700 1 --2- 172.21.2.112:0/450312531 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1c640b7ad0 0x7f1c640f0770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:31.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:31.790 7f1c63fff700 1 --2- 172.21.2.112:0/450312531 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1c64092c40 0x7f1c64093040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:31.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:31.790 7f1c69746700 1 -- 172.21.2.112:0/450312531 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1c640b7ad0 msgr2=0x7f1c640f0770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:31.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:31.790 7f1c69746700 1 --2- 172.21.2.112:0/450312531 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1c640b7ad0 0x7f1c640f0770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:31.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:31.790 7f1c69746700 1 -- 172.21.2.112:0/450312531 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1c64092c40 msgr2=0x7f1c64093040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:31.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:31.790 7f1c69746700 1 --2- 172.21.2.112:0/450312531 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1c64092c40 0x7f1c64093040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:31.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:31.790 7f1c69746700 1 -- 172.21.2.112:0/450312531 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c640ef570 con 0x7f1c64093660 2021-02-10T11:24:31.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:31.790 7f1c69746700 1 --2- 172.21.2.112:0/450312531 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1c64093660 0x7f1c640eef50 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 rx=0x7f1c5c006f00 tx=0x7f1c5c00b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:31.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:31.790 7f1c617fa700 1 -- 172.21.2.112:0/450312531 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1c5c006030 con 0x7f1c64093660 2021-02-10T11:24:31.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:31.791 7f1c617fa700 1 -- 172.21.2.112:0/450312531 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c5c0062f0 con 0x7f1c64093660 2021-02-10T11:24:31.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:31.791 7f1c6b129700 1 -- 172.21.2.112:0/450312531 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c640ef570 con 0x7f1c64093660 2021-02-10T11:24:31.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:31.791 7f1c617fa700 1 -- 172.21.2.112:0/450312531 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1c5c005770 con 0x7f1c64093660 2021-02-10T11:24:31.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:31.791 7f1c6b129700 1 -- 172.21.2.112:0/450312531 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c640efb00 con 0x7f1c64093660 2021-02-10T11:24:31.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:31.791 7f1c617fa700 1 -- 172.21.2.112:0/450312531 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1c5c0054f0 con 0x7f1c64093660 2021-02-10T11:24:31.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:31.791 7f1c617fa700 1 --2- 172.21.2.112:0/450312531 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1c480186c0 0x7f1c4801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:31.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:31.791 7f1c617fa700 1 -- 172.21.2.112:0/450312531 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f1c5c032620 con 0x7f1c64093660 2021-02-10T11:24:31.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:31.791 7f1c68f45700 1 --2- 172.21.2.112:0/450312531 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1c480186c0 0x7f1c4801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:31.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:31.792 7f1c68f45700 1 --2- 172.21.2.112:0/450312531 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1c480186c0 0x7f1c4801abc0 crc :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:31.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:31.822 7f1c6b129700 1 -- 172.21.2.112:0/450312531 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c640a7140 con 0x7f1c64093660 2021-02-10T11:24:31.815 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:31.824 7f1c617fa700 1 -- 172.21.2.112:0/450312531 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1c5c0062d0 con 0x7f1c64093660 2021-02-10T11:24:31.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:31.951 7f1c6b129700 1 -- 172.21.2.112:0/450312531 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348629", "target": ["mgr", ""]}) v1 -- 0x7f1c640b7550 con 0x7f1c480186c0 2021-02-10T11:24:31.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:31.976 7f1c617fa700 1 -- 172.21.2.112:0/450312531 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f1c4c006430 con 0x7f1c480186c0 2021-02-10T11:24:31.975 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:31.984 7f1c6b129700 1 -- 172.21.2.112:0/450312531 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1c480186c0 msgr2=0x7f1c4801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:31.975 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:31.984 7f1c6b129700 1 --2- 172.21.2.112:0/450312531 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1c480186c0 0x7f1c4801abc0 crc :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:31.976 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:31.984 7f1c6b129700 1 -- 172.21.2.112:0/450312531 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1c64093660 msgr2=0x7f1c640eef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:31.976 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:31.984 7f1c6b129700 1 --2- 172.21.2.112:0/450312531 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1c64093660 0x7f1c640eef50 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 rx=0x7f1c5c006f00 tx=0x7f1c5c00b940).stop 2021-02-10T11:24:31.976 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:31.985 7f1c6b129700 1 -- 172.21.2.112:0/450312531 shutdown_connections 2021-02-10T11:24:31.976 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:31.985 7f1c6b129700 1 --2- 172.21.2.112:0/450312531 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1c480186c0 0x7f1c4801abc0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:31.976 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:31.985 7f1c6b129700 1 --2- 172.21.2.112:0/450312531 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1c64092c40 0x7f1c64093040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:31.976 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:31.985 7f1c6b129700 1 --2- 172.21.2.112:0/450312531 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1c64093660 0x7f1c640eef50 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:31.976 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:31.985 7f1c6b129700 1 --2- 172.21.2.112:0/450312531 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1c640b7ad0 0x7f1c640f0770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:31.977 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:31.985 7f1c6b129700 1 -- 172.21.2.112:0/450312531 >> 172.21.2.112:0/450312531 conn(0x7f1c640a4a60 msgr2=0x7f1c640b8250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:31.977 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:31.985 7f1c6b129700 1 -- 172.21.2.112:0/450312531 shutdown_connections 2021-02-10T11:24:31.977 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:31.985 7f1c6b129700 1 -- 172.21.2.112:0/450312531 wait complete. 2021-02-10T11:24:31.984 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000348629 2021-02-10T11:24:32.419 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:32.408 7f1230665700 1 Processor -- start 2021-02-10T11:24:32.420 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:32.408 7f1230665700 1 -- start start 2021-02-10T11:24:32.420 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:32.408 7f1230665700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f12281b0870 0x7f12280f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:32.420 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:32.408 7f1230665700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f12280efe00 0x7f12280f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:32.420 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:32.408 7f1230665700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12281b0c70 con 0x7f12281b0470 2021-02-10T11:24:32.420 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:32.408 7f1230665700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12280f0b50 con 0x7f12281b0870 2021-02-10T11:24:32.420 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:32.408 7f1230665700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12280f0820 con 0x7f12280efe00 2021-02-10T11:24:32.421 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:32.408 7f122dc80700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f12281b0870 0x7f12280f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:32.421 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:32.408 7f122e481700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f12281b0470 0x7f12280ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:44974/0 (socket says 172.21.2.112:44974) 2021-02-10T11:24:32.421 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:32.408 7f122e481700 1 -- 172.21.2.112:0/3245432529 learned_addr learned my addr 172.21.2.112:0/3245432529 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:24:32.421 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:32.408 7f122dc80700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f12281b0870 0x7f12280f20e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:45514/0 (socket says 172.21.2.112:45514) 2021-02-10T11:24:32.421 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:32.409 7f122ec82700 1 --2- 172.21.2.112:0/3245432529 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f12280efe00 0x7f12280f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:32.421 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:32.409 7f122dc80700 1 -- 172.21.2.112:0/3245432529 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f12280efe00 msgr2=0x7f12280f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:32.422 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:32.409 7f122dc80700 1 --2- 172.21.2.112:0/3245432529 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f12280efe00 0x7f12280f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:32.422 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:32.409 7f122dc80700 1 -- 172.21.2.112:0/3245432529 >> v1:172.21.2.112:6789/0 conn(0x7f12281b0470 legacy=0x7f12280ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:32.422 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:32.409 7f122dc80700 1 -- 172.21.2.112:0/3245432529 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12280e52e0 con 0x7f12281b0870 2021-02-10T11:24:32.422 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:32.409 7f122dc80700 1 --2- 172.21.2.112:0/3245432529 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f12281b0870 0x7f12280f20e0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 rx=0x7f1218006650 tx=0x7f1218009a80).ready entity=mon.0 client_cookie=e4a80d39175bba28 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:32.423 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:32.409 7f122d47f700 1 -- 172.21.2.112:0/3245432529 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f12180009b0 con 0x7f12281b0870 2021-02-10T11:24:32.423 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:32.409 7f122d47f700 1 -- 172.21.2.112:0/3245432529 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f121800c0c0 con 0x7f12281b0870 2021-02-10T11:24:32.423 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:32.409 7f122d47f700 1 -- 172.21.2.112:0/3245432529 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f121800c510 con 0x7f12281b0870 2021-02-10T11:24:32.423 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:32.409 7f1230665700 1 -- 172.21.2.112:0/3245432529 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f12281b0870 msgr2=0x7f12280f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:32.423 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:32.409 7f1230665700 1 --2- 172.21.2.112:0/3245432529 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f12281b0870 0x7f12280f20e0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 rx=0x7f1218006650 tx=0x7f1218009a80).stop 2021-02-10T11:24:32.423 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:32.410 7f1230665700 1 -- 172.21.2.112:0/3245432529 shutdown_connections 2021-02-10T11:24:32.424 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:32.410 7f1230665700 1 --2- 172.21.2.112:0/3245432529 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f12280efe00 0x7f12280f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:32.424 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:32.410 7f1230665700 1 --2- 172.21.2.112:0/3245432529 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f12281b0870 0x7f12280f20e0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:32.424 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:32.410 7f1230665700 1 -- 172.21.2.112:0/3245432529 >> 172.21.2.112:0/3245432529 conn(0x7f12280ce3e0 msgr2=0x7f12280e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:32.424 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:32.410 7f1230665700 1 -- 172.21.2.112:0/3245432529 shutdown_connections 2021-02-10T11:24:32.424 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:32.410 7f1230665700 1 -- 172.21.2.112:0/3245432529 wait complete. 2021-02-10T11:24:32.425 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:32.425 7f1230665700 1 Processor -- start 2021-02-10T11:24:32.425 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:32.425 7f1230665700 1 -- start start 2021-02-10T11:24:32.425 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:32.425 7f1230665700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f12280a0420 0x7f12280a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:32.425 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:32.425 7f1230665700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1228189d00 0x7f122818a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:32.425 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:32.425 7f1230665700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f122818a720 0x7f12280b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:32.425 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:32.425 7f1230665700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1228189800 con 0x7f122818a720 2021-02-10T11:24:32.426 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:32.425 7f1230665700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f122818ab20 con 0x7f1228189d00 2021-02-10T11:24:32.426 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:32.425 7f1230665700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12280a1140 con 0x7f12280a0420 2021-02-10T11:24:32.426 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:32.425 7f122ec82700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f122818a720 0x7f12280b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:32.426 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:32.425 7f122ec82700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f122818a720 0x7f12280b7be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:45518/0 (socket says 172.21.2.112:45518) 2021-02-10T11:24:32.426 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:32.425 7f122ec82700 1 -- 172.21.2.112:0/1466140529 learned_addr learned my addr 172.21.2.112:0/1466140529 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:24:32.426 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:32.425 7f122e481700 1 --2- 172.21.2.112:0/1466140529 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f12280a0420 0x7f12280a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:32.427 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:32.425 7f122dc80700 1 --2- 172.21.2.112:0/1466140529 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1228189d00 0x7f122818a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:32.427 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:32.426 7f122ec82700 1 -- 172.21.2.112:0/1466140529 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f12280a0420 msgr2=0x7f12280a0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:32.427 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:32.426 7f122ec82700 1 --2- 172.21.2.112:0/1466140529 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f12280a0420 0x7f12280a0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:32.427 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:32.426 7f122ec82700 1 -- 172.21.2.112:0/1466140529 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1228189d00 msgr2=0x7f122818a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:32.427 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:32.426 7f122ec82700 1 --2- 172.21.2.112:0/1466140529 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1228189d00 0x7f122818a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:32.428 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:32.426 7f122ec82700 1 -- 172.21.2.112:0/1466140529 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12280b8200 con 0x7f122818a720 2021-02-10T11:24:32.428 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:32.426 7f122ec82700 1 --2- 172.21.2.112:0/1466140529 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f122818a720 0x7f12280b7be0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 rx=0x7f122000b970 tx=0x7f122000bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:32.428 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:32.426 7f121effd700 1 -- 172.21.2.112:0/1466140529 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1220000f90 con 0x7f122818a720 2021-02-10T11:24:32.428 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:32.426 7f121effd700 1 -- 172.21.2.112:0/1466140529 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1220006aa0 con 0x7f122818a720 2021-02-10T11:24:32.428 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:32.426 7f1230665700 1 -- 172.21.2.112:0/1466140529 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12280b8200 con 0x7f122818a720 2021-02-10T11:24:32.428 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:32.426 7f1230665700 1 -- 172.21.2.112:0/1466140529 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12280b8200 con 0x7f122818a720 2021-02-10T11:24:32.429 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:32.426 7f121effd700 1 -- 172.21.2.112:0/1466140529 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f12200015f0 con 0x7f122818a720 2021-02-10T11:24:32.429 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:32.427 7f121effd700 1 -- 172.21.2.112:0/1466140529 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1220000e70 con 0x7f122818a720 2021-02-10T11:24:32.429 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:32.427 7f121effd700 1 --2- 172.21.2.112:0/1466140529 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1210018730 0x7f121001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:32.429 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:32.427 7f121effd700 1 -- 172.21.2.112:0/1466140529 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f1220032b40 con 0x7f122818a720 2021-02-10T11:24:32.429 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:32.427 7f122e481700 1 --2- 172.21.2.112:0/1466140529 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1210018730 0x7f121001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:32.429 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:32.428 7f122e481700 1 --2- 172.21.2.112:0/1466140529 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1210018730 0x7f121001ac10 crc :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:32.449 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:32.458 7f1230665700 1 -- 172.21.2.112:0/1466140529 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f122818ab20 con 0x7f122818a720 2021-02-10T11:24:32.451 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:32.460 7f121effd700 1 -- 172.21.2.112:0/1466140529 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1220006aa0 con 0x7f122818a720 2021-02-10T11:24:32.582 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:32.591 7f1230665700 1 -- 172.21.2.112:0/1466140529 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348629", "target": ["mgr", ""]}) v1 -- 0x7f122818ab20 con 0x7f1210018730 2021-02-10T11:24:32.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:32.595 7f121effd700 1 -- 172.21.2.112:0/1466140529 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f1224007740 con 0x7f1210018730 2021-02-10T11:24:32.589 INFO:teuthology.orchestra.run.gibba012.stdout:/volumes/_nogroup/subvolume_0000000000348629/c5327a44-6a59-47f6-af08-f928b8cf47d4 2021-02-10T11:24:32.591 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:32.600 7f1230665700 1 -- 172.21.2.112:0/1466140529 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1210018730 msgr2=0x7f121001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:32.591 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:32.600 7f1230665700 1 --2- 172.21.2.112:0/1466140529 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1210018730 0x7f121001ac10 crc :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:32.591 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:32.600 7f1230665700 1 -- 172.21.2.112:0/1466140529 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f122818a720 msgr2=0x7f12280b7be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:32.591 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:32.600 7f1230665700 1 --2- 172.21.2.112:0/1466140529 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f122818a720 0x7f12280b7be0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 rx=0x7f122000b970 tx=0x7f122000bd10).stop 2021-02-10T11:24:32.591 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:32.600 7f1230665700 1 -- 172.21.2.112:0/1466140529 shutdown_connections 2021-02-10T11:24:32.591 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:32.600 7f1230665700 1 --2- 172.21.2.112:0/1466140529 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1210018730 0x7f121001ac10 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:32.591 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:32.600 7f1230665700 1 --2- 172.21.2.112:0/1466140529 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1228189d00 0x7f122818a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:32.592 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:32.600 7f1230665700 1 --2- 172.21.2.112:0/1466140529 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f122818a720 0x7f12280b7be0 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:32.592 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:32.600 7f1230665700 1 --2- 172.21.2.112:0/1466140529 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f12280a0420 0x7f12280a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:32.592 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:32.600 7f1230665700 1 -- 172.21.2.112:0/1466140529 >> 172.21.2.112:0/1466140529 conn(0x7f1228094e10 msgr2=0x7f12280a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:32.592 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:32.600 7f1230665700 1 -- 172.21.2.112:0/1466140529 shutdown_connections 2021-02-10T11:24:32.592 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:32.600 7f1230665700 1 -- 172.21.2.112:0/1466140529 wait complete. 2021-02-10T11:24:32.599 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000348629 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000348629/c5327a44-6a59-47f6-af08-f928b8cf47d4 2021-02-10T11:24:32.599 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:32.599 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:32.599 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:32.935 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:24:32.935 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:24:32.935 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348629/c5327a44-6a59-47f6-af08-f928b8cf47d4/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:24:33.315 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:24:33.315 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:24:33.315 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0231626 s, 45.3 MB/s 2021-02-10T11:24:33.316 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:33.316 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:33.316 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:33.652 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:24:33.652 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:24:33.652 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348629/c5327a44-6a59-47f6-af08-f928b8cf47d4/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:24:34.028 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:24:34.028 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:24:34.028 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.02504 s, 41.9 MB/s 2021-02-10T11:24:34.029 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:34.029 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:34.029 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:34.365 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:24:34.366 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:24:34.366 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348629/c5327a44-6a59-47f6-af08-f928b8cf47d4/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:24:34.740 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:24:34.740 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:24:34.740 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0252139 s, 41.6 MB/s 2021-02-10T11:24:34.741 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:34.741 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:34.741 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:35.076 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:24:35.076 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:24:35.077 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348629/c5327a44-6a59-47f6-af08-f928b8cf47d4/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:24:35.450 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:24:35.450 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:24:35.450 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0225223 s, 46.6 MB/s 2021-02-10T11:24:35.451 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:35.451 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:35.452 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:35.785 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:24:35.786 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:24:35.786 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348629/c5327a44-6a59-47f6-af08-f928b8cf47d4/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:24:36.164 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:24:36.164 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:24:36.164 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0249745 s, 42.0 MB/s 2021-02-10T11:24:36.165 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:36.165 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:36.165 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:36.502 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:24:36.502 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:24:36.503 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348629/c5327a44-6a59-47f6-af08-f928b8cf47d4/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:24:36.877 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:24:36.878 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:24:36.878 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0230834 s, 45.4 MB/s 2021-02-10T11:24:36.879 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:36.879 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:36.879 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:37.214 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:24:37.214 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:24:37.214 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348629/c5327a44-6a59-47f6-af08-f928b8cf47d4/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:24:37.590 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:24:37.590 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:24:37.590 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0233799 s, 44.8 MB/s 2021-02-10T11:24:37.591 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:37.591 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:37.591 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:37.927 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:24:37.927 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:24:37.927 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348629/c5327a44-6a59-47f6-af08-f928b8cf47d4/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:24:38.305 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:24:38.305 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:24:38.305 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0250933 s, 41.8 MB/s 2021-02-10T11:24:38.306 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:38.306 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:38.306 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:38.642 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:24:38.642 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:24:38.642 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348629/c5327a44-6a59-47f6-af08-f928b8cf47d4/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:24:39.017 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:24:39.017 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:24:39.017 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0231461 s, 45.3 MB/s 2021-02-10T11:24:39.018 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:39.018 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:39.018 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:39.283 DEBUG:teuthology.orchestra.run.gibba012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:24:39.286 DEBUG:teuthology.orchestra.run.gibba020:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:24:39.353 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:24:39.354 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:24:39.354 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348629/c5327a44-6a59-47f6-af08-f928b8cf47d4/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:24:39.657 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:24:39.657 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:24:39.657 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0239004 s, 43.9 MB/s 2021-02-10T11:24:39.658 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000348630 2021-02-10T11:24:40.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.081 7f9da4c4d700 1 Processor -- start 2021-02-10T11:24:40.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.081 7f9da4c4d700 1 -- start start 2021-02-10T11:24:40.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.081 7f9da4c4d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9da01b0470 0x7f9da01b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:40.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.081 7f9da4c4d700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f9da00ef610 0x7f9da00efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:40.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.081 7f9da4c4d700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9da00f0b50 con 0x7f9da00f0030 2021-02-10T11:24:40.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.081 7f9da4c4d700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9da00e52e0 con 0x7f9da00ef610 2021-02-10T11:24:40.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.081 7f9da4c4d700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9da00e5490 con 0x7f9da01b0470 2021-02-10T11:24:40.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.082 7f9d9f05f700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f9da00f0030 0x7f9da00f10d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:44992/0 (socket says 172.21.2.112:44992) 2021-02-10T11:24:40.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.082 7f9d9f05f700 1 -- 172.21.2.112:0/867132621 learned_addr learned my addr 172.21.2.112:0/867132621 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:24:40.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.082 7f9d9e05d700 1 --2- 172.21.2.112:0/867132621 >> v2:172.21.2.120:3300/0 conn(0x7f9da00ef610 0x7f9da00efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:40.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.082 7f9d9e85e700 1 --2- 172.21.2.112:0/867132621 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9da01b0470 0x7f9da01b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:40.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.082 7f9d9d85c700 1 -- 172.21.2.112:0/867132621 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2978331717 0 0) 0x7f9d98001500 con 0x7f9da00f0030 2021-02-10T11:24:40.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.082 7f9d9d85c700 1 -- 172.21.2.112:0/867132621 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d8c002fe0 con 0x7f9da00f0030 2021-02-10T11:24:40.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.082 7f9d9e05d700 1 -- 172.21.2.112:0/867132621 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9da01b0470 msgr2=0x7f9da01b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:40.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.082 7f9d9e05d700 1 --2- 172.21.2.112:0/867132621 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9da01b0470 0x7f9da01b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:40.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.082 7f9d9e05d700 1 -- 172.21.2.112:0/867132621 >> v1:172.21.2.112:6789/0 conn(0x7f9da00f0030 legacy=0x7f9da00f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:40.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.082 7f9d9e05d700 1 -- 172.21.2.112:0/867132621 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9da00f0430 con 0x7f9da00ef610 2021-02-10T11:24:40.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.082 7f9d9e05d700 1 --2- 172.21.2.112:0/867132621 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9da00ef610 0x7f9da00efa10 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7f9d90006650 tx=0x7f9d90009a80).ready entity=mon.1 client_cookie=3bda8bdb22be80a5 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:40.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.083 7f9d9d85c700 1 -- 172.21.2.112:0/867132621 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9d900009b0 con 0x7f9da00ef610 2021-02-10T11:24:40.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.083 7f9d9d85c700 1 -- 172.21.2.112:0/867132621 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d9000c0c0 con 0x7f9da00ef610 2021-02-10T11:24:40.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.083 7f9d9d85c700 1 -- 172.21.2.112:0/867132621 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9d9000c5e0 con 0x7f9da00ef610 2021-02-10T11:24:40.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.083 7f9da4c4d700 1 -- 172.21.2.112:0/867132621 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9da00ef610 msgr2=0x7f9da00efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:40.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.083 7f9da4c4d700 1 --2- 172.21.2.112:0/867132621 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9da00ef610 0x7f9da00efa10 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7f9d90006650 tx=0x7f9d90009a80).stop 2021-02-10T11:24:40.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.083 7f9da4c4d700 1 -- 172.21.2.112:0/867132621 shutdown_connections 2021-02-10T11:24:40.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.083 7f9da4c4d700 1 --2- 172.21.2.112:0/867132621 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9da00ef610 0x7f9da00efa10 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:40.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.083 7f9da4c4d700 1 --2- 172.21.2.112:0/867132621 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9da01b0470 0x7f9da01b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:40.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.083 7f9da4c4d700 1 -- 172.21.2.112:0/867132621 >> 172.21.2.112:0/867132621 conn(0x7f9da00ce3e0 msgr2=0x7f9da00e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:40.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.084 7f9da4c4d700 1 -- 172.21.2.112:0/867132621 shutdown_connections 2021-02-10T11:24:40.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.084 7f9da4c4d700 1 -- 172.21.2.112:0/867132621 wait complete. 2021-02-10T11:24:40.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.098 7f9da4c4d700 1 Processor -- start 2021-02-10T11:24:40.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.098 7f9da4c4d700 1 -- start start 2021-02-10T11:24:40.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.098 7f9da4c4d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9da00bb090 0x7f9da010a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:40.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.098 7f9da4c4d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9da00a66b0 0x7f9da00a0a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:40.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.098 7f9da4c4d700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9da00a10a0 0x7f9da01069f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:40.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.099 7f9da4c4d700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9da00bb490 con 0x7f9da00a66b0 2021-02-10T11:24:40.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.099 7f9da4c4d700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9da00a6170 con 0x7f9da00a10a0 2021-02-10T11:24:40.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.099 7f9da4c4d700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9da008d8c0 con 0x7f9da00bb090 2021-02-10T11:24:40.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.099 7f9d9e05d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9da00a66b0 0x7f9da00a0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:40.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.099 7f9d9e05d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9da00a66b0 0x7f9da00a0a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:45532/0 (socket says 172.21.2.112:45532) 2021-02-10T11:24:40.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.099 7f9d9e05d700 1 -- 172.21.2.112:0/1928556549 learned_addr learned my addr 172.21.2.112:0/1928556549 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:24:40.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.099 7f9d9f05f700 1 --2- 172.21.2.112:0/1928556549 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9da00a10a0 0x7f9da01069f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:40.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.099 7f9d9e85e700 1 --2- 172.21.2.112:0/1928556549 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9da00bb090 0x7f9da010a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:40.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.099 7f9d9e05d700 1 -- 172.21.2.112:0/1928556549 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9da00bb090 msgr2=0x7f9da010a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:40.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.099 7f9d9e05d700 1 --2- 172.21.2.112:0/1928556549 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9da00bb090 0x7f9da010a1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:40.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.099 7f9d9e05d700 1 -- 172.21.2.112:0/1928556549 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9da00a10a0 msgr2=0x7f9da01069f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:40.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.099 7f9d9e05d700 1 --2- 172.21.2.112:0/1928556549 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9da00a10a0 0x7f9da01069f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:40.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.099 7f9d9e05d700 1 -- 172.21.2.112:0/1928556549 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9da00a14a0 con 0x7f9da00a66b0 2021-02-10T11:24:40.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.099 7f9d9e05d700 1 --2- 172.21.2.112:0/1928556549 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9da00a66b0 0x7f9da00a0a80 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 rx=0x7f9d90009630 tx=0x7f9d9000b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:40.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.100 7f9d977fe700 1 -- 172.21.2.112:0/1928556549 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9d9000bd70 con 0x7f9da00a66b0 2021-02-10T11:24:40.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.100 7f9d977fe700 1 -- 172.21.2.112:0/1928556549 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d90000960 con 0x7f9da00a66b0 2021-02-10T11:24:40.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.100 7f9da4c4d700 1 -- 172.21.2.112:0/1928556549 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9da00a14a0 con 0x7f9da00a66b0 2021-02-10T11:24:40.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.100 7f9d977fe700 1 -- 172.21.2.112:0/1928556549 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9d90005560 con 0x7f9da00a66b0 2021-02-10T11:24:40.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.100 7f9da4c4d700 1 -- 172.21.2.112:0/1928556549 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9da00a14a0 con 0x7f9da00a66b0 2021-02-10T11:24:40.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.100 7f9d977fe700 1 -- 172.21.2.112:0/1928556549 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9d90000c90 con 0x7f9da00a66b0 2021-02-10T11:24:40.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.100 7f9d977fe700 1 --2- 172.21.2.112:0/1928556549 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9d840186c0 0x7f9d8401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:40.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.100 7f9d977fe700 1 -- 172.21.2.112:0/1928556549 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f9d900323f0 con 0x7f9da00a66b0 2021-02-10T11:24:40.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.101 7f9d9e85e700 1 --2- 172.21.2.112:0/1928556549 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9d840186c0 0x7f9d8401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:40.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.101 7f9d9e85e700 1 --2- 172.21.2.112:0/1928556549 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9d840186c0 0x7f9d8401abc0 crc :-1 s=READY pgs=299 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:40.122 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.131 7f9da4c4d700 1 -- 172.21.2.112:0/1928556549 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9da008d8c0 con 0x7f9da00a66b0 2021-02-10T11:24:40.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.134 7f9d977fe700 1 -- 172.21.2.112:0/1928556549 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9d9002ded0 con 0x7f9da00a66b0 2021-02-10T11:24:40.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.260 7f9da4c4d700 1 -- 172.21.2.112:0/1928556549 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348630", "target": ["mgr", ""]}) v1 -- 0x7f9da00a1740 con 0x7f9d840186c0 2021-02-10T11:24:40.277 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.286 7f9d977fe700 1 -- 172.21.2.112:0/1928556549 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f9d88006a70 con 0x7f9d840186c0 2021-02-10T11:24:40.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.294 7f9da4c4d700 1 -- 172.21.2.112:0/1928556549 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9d840186c0 msgr2=0x7f9d8401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:40.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.294 7f9da4c4d700 1 --2- 172.21.2.112:0/1928556549 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9d840186c0 0x7f9d8401abc0 crc :-1 s=READY pgs=299 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:40.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.294 7f9da4c4d700 1 -- 172.21.2.112:0/1928556549 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9da00a66b0 msgr2=0x7f9da00a0a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:40.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.294 7f9da4c4d700 1 --2- 172.21.2.112:0/1928556549 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9da00a66b0 0x7f9da00a0a80 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 rx=0x7f9d90009630 tx=0x7f9d9000b980).stop 2021-02-10T11:24:40.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.294 7f9da4c4d700 1 -- 172.21.2.112:0/1928556549 shutdown_connections 2021-02-10T11:24:40.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.294 7f9da4c4d700 1 --2- 172.21.2.112:0/1928556549 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9d840186c0 0x7f9d8401abc0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:40.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.294 7f9da4c4d700 1 --2- 172.21.2.112:0/1928556549 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9da00a10a0 0x7f9da01069f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:40.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.294 7f9da4c4d700 1 --2- 172.21.2.112:0/1928556549 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9da00a66b0 0x7f9da00a0a80 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:40.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.294 7f9da4c4d700 1 --2- 172.21.2.112:0/1928556549 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9da00bb090 0x7f9da010a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:40.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.294 7f9da4c4d700 1 -- 172.21.2.112:0/1928556549 >> 172.21.2.112:0/1928556549 conn(0x7f9da00bbf60 msgr2=0x7f9da0105d10 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:40.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.294 7f9da4c4d700 1 -- 172.21.2.112:0/1928556549 shutdown_connections 2021-02-10T11:24:40.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.294 7f9da4c4d700 1 -- 172.21.2.112:0/1928556549 wait complete. 2021-02-10T11:24:40.293 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000348630 2021-02-10T11:24:40.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.718 7f7392380700 1 Processor -- start 2021-02-10T11:24:40.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.718 7f7392380700 1 -- start start 2021-02-10T11:24:40.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.718 7f7392380700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f738c1b0470 0x7f738c1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:40.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.718 7f7392380700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f738c0f20e0 0x7f738c0f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:40.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.718 7f7392380700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f738c0f0b50 con 0x7f738c0ef610 2021-02-10T11:24:40.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.718 7f7392380700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f738c0f0820 con 0x7f738c1b0470 2021-02-10T11:24:40.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.718 7f7392380700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f738c0f24e0 con 0x7f738c0f20e0 2021-02-10T11:24:40.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.718 7f738bfff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f738c1b0470 0x7f738c1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:40.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.718 7f738b7fe700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f738c0ef610 0x7f738c0efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:45002/0 (socket says 172.21.2.112:45002) 2021-02-10T11:24:40.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.718 7f738b7fe700 1 -- 172.21.2.112:0/1493255949 learned_addr learned my addr 172.21.2.112:0/1493255949 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:24:40.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.718 7f739099d700 1 --2- 172.21.2.112:0/1493255949 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f738c0f20e0 0x7f738c0f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:40.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.719 7f739099d700 1 -- 172.21.2.112:0/1493255949 >> v2:172.21.2.112:3300/0 conn(0x7f738c1b0470 msgr2=0x7f738c1b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:40.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.719 7f739099d700 1 --2- 172.21.2.112:0/1493255949 >> v2:172.21.2.112:3300/0 conn(0x7f738c1b0470 0x7f738c1b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:40.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.719 7f739099d700 1 -- 172.21.2.112:0/1493255949 >> v1:172.21.2.112:6789/0 conn(0x7f738c0ef610 legacy=0x7f738c0efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:40.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.719 7f739099d700 1 -- 172.21.2.112:0/1493255949 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f738c0e52e0 con 0x7f738c0f20e0 2021-02-10T11:24:40.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.719 7f739099d700 1 --2- 172.21.2.112:0/1493255949 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f738c0f20e0 0x7f738c0f0200 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f7384006e50 tx=0x7f7384009e70).ready entity=mon.2 client_cookie=1e4374524406380b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:40.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.719 7f738affd700 1 -- 172.21.2.112:0/1493255949 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7384006a20 con 0x7f738c0f20e0 2021-02-10T11:24:40.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.719 7f738affd700 1 -- 172.21.2.112:0/1493255949 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7384006790 con 0x7f738c0f20e0 2021-02-10T11:24:40.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.719 7f738affd700 1 -- 172.21.2.112:0/1493255949 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f738400b700 con 0x7f738c0f20e0 2021-02-10T11:24:40.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.719 7f7392380700 1 -- 172.21.2.112:0/1493255949 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f738c0f20e0 msgr2=0x7f738c0f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:40.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.719 7f7392380700 1 --2- 172.21.2.112:0/1493255949 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f738c0f20e0 0x7f738c0f0200 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f7384006e50 tx=0x7f7384009e70).stop 2021-02-10T11:24:40.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.720 7f7392380700 1 -- 172.21.2.112:0/1493255949 shutdown_connections 2021-02-10T11:24:40.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.720 7f7392380700 1 --2- 172.21.2.112:0/1493255949 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f738c0f20e0 0x7f738c0f0200 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:40.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.720 7f7392380700 1 --2- 172.21.2.112:0/1493255949 >> v2:172.21.2.112:3300/0 conn(0x7f738c1b0470 0x7f738c1b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:40.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.720 7f7392380700 1 -- 172.21.2.112:0/1493255949 >> 172.21.2.112:0/1493255949 conn(0x7f738c0ce3e0 msgr2=0x7f738c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:40.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.720 7f7392380700 1 -- 172.21.2.112:0/1493255949 shutdown_connections 2021-02-10T11:24:40.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.720 7f7392380700 1 -- 172.21.2.112:0/1493255949 wait complete. 2021-02-10T11:24:40.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.734 7f7392380700 1 Processor -- start 2021-02-10T11:24:40.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.735 7f7392380700 1 -- start start 2021-02-10T11:24:40.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.735 7f7392380700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f738c190920 0x7f738c0b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:40.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.735 7f7392380700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f738c0b8200 0x7f738c0b8600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:40.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.735 7f7392380700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f738c0c1d80 0x7f738c0c2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:40.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.735 7f7392380700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f738c190da0 con 0x7f738c190920 2021-02-10T11:24:40.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.735 7f7392380700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f738c0b8c20 con 0x7f738c0c1d80 2021-02-10T11:24:40.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.735 7f7392380700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f738c0b8e10 con 0x7f738c0b8200 2021-02-10T11:24:40.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.735 7f738bfff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f738c190920 0x7f738c0b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:40.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.735 7f738bfff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f738c190920 0x7f738c0b7be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:45546/0 (socket says 172.21.2.112:45546) 2021-02-10T11:24:40.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.735 7f738bfff700 1 -- 172.21.2.112:0/1901211504 learned_addr learned my addr 172.21.2.112:0/1901211504 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:24:40.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.736 7f738bfff700 1 -- 172.21.2.112:0/1901211504 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f738c0b8200 msgr2=0x7f738c0b8600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:40.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.736 7f738bfff700 1 --2- 172.21.2.112:0/1901211504 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f738c0b8200 0x7f738c0b8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:40.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.736 7f738bfff700 1 -- 172.21.2.112:0/1901211504 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f738c0c1d80 msgr2=0x7f738c0c2180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:40.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.736 7f738bfff700 1 --2- 172.21.2.112:0/1901211504 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f738c0c1d80 0x7f738c0c2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:40.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.736 7f738bfff700 1 -- 172.21.2.112:0/1901211504 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f738c0c27d0 con 0x7f738c190920 2021-02-10T11:24:40.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.736 7f738bfff700 1 --2- 172.21.2.112:0/1901211504 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f738c190920 0x7f738c0b7be0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 rx=0x7f737c006f60 tx=0x7f737c00b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:40.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.736 7f7388ff9700 1 -- 172.21.2.112:0/1901211504 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f737c0009b0 con 0x7f738c190920 2021-02-10T11:24:40.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.736 7f7388ff9700 1 -- 172.21.2.112:0/1901211504 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f737c00bdf0 con 0x7f738c190920 2021-02-10T11:24:40.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.736 7f7392380700 1 -- 172.21.2.112:0/1901211504 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f738c0c27d0 con 0x7f738c190920 2021-02-10T11:24:40.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.736 7f7388ff9700 1 -- 172.21.2.112:0/1901211504 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f737c002030 con 0x7f738c190920 2021-02-10T11:24:40.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.736 7f7392380700 1 -- 172.21.2.112:0/1901211504 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f738c0c2d60 con 0x7f738c190920 2021-02-10T11:24:40.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.737 7f7388ff9700 1 -- 172.21.2.112:0/1901211504 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f737c006b00 con 0x7f738c190920 2021-02-10T11:24:40.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.737 7f7388ff9700 1 --2- 172.21.2.112:0/1901211504 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f736c0186c0 0x7f736c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:40.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.737 7f7388ff9700 1 -- 172.21.2.112:0/1901211504 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f737c0323c0 con 0x7f738c190920 2021-02-10T11:24:40.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.737 7f738b7fe700 1 --2- 172.21.2.112:0/1901211504 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f736c0186c0 0x7f736c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:40.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.738 7f738b7fe700 1 --2- 172.21.2.112:0/1901211504 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f736c0186c0 0x7f736c01abc0 crc :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:40.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.768 7f7392380700 1 -- 172.21.2.112:0/1901211504 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f738c105ae0 con 0x7f738c190920 2021-02-10T11:24:40.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.771 7f7388ff9700 1 -- 172.21.2.112:0/1901211504 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f737c0009b0 con 0x7f738c190920 2021-02-10T11:24:40.890 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.899 7f7392380700 1 -- 172.21.2.112:0/1901211504 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348630", "target": ["mgr", ""]}) v1 -- 0x7f738c0c2d60 con 0x7f736c0186c0 2021-02-10T11:24:40.893 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.903 7f7388ff9700 1 -- 172.21.2.112:0/1901211504 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f7380001530 con 0x7f736c0186c0 2021-02-10T11:24:40.897 INFO:teuthology.orchestra.run.gibba012.stdout:/volumes/_nogroup/subvolume_0000000000348630/064411bc-8f98-40aa-be67-19fd2ad6651c 2021-02-10T11:24:40.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.908 7f7392380700 1 -- 172.21.2.112:0/1901211504 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f736c0186c0 msgr2=0x7f736c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:40.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.908 7f7392380700 1 --2- 172.21.2.112:0/1901211504 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f736c0186c0 0x7f736c01abc0 crc :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:40.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.908 7f7392380700 1 -- 172.21.2.112:0/1901211504 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f738c190920 msgr2=0x7f738c0b7be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:40.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.908 7f7392380700 1 --2- 172.21.2.112:0/1901211504 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f738c190920 0x7f738c0b7be0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 rx=0x7f737c006f60 tx=0x7f737c00b970).stop 2021-02-10T11:24:40.900 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.908 7f7392380700 1 -- 172.21.2.112:0/1901211504 shutdown_connections 2021-02-10T11:24:40.900 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.908 7f7392380700 1 --2- 172.21.2.112:0/1901211504 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f736c0186c0 0x7f736c01abc0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:40.900 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.908 7f7392380700 1 --2- 172.21.2.112:0/1901211504 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f738c0c1d80 0x7f738c0c2180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:40.900 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.908 7f7392380700 1 --2- 172.21.2.112:0/1901211504 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f738c0b8200 0x7f738c0b8600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:40.900 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.908 7f7392380700 1 --2- 172.21.2.112:0/1901211504 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f738c190920 0x7f738c0b7be0 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:40.900 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.908 7f7392380700 1 -- 172.21.2.112:0/1901211504 >> 172.21.2.112:0/1901211504 conn(0x7f738c094e10 msgr2=0x7f738c0a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:40.900 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.909 7f7392380700 1 -- 172.21.2.112:0/1901211504 shutdown_connections 2021-02-10T11:24:40.901 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:40.909 7f7392380700 1 -- 172.21.2.112:0/1901211504 wait complete. 2021-02-10T11:24:40.908 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000348630 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000348630/064411bc-8f98-40aa-be67-19fd2ad6651c 2021-02-10T11:24:40.908 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:40.908 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:40.908 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:41.244 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:24:41.244 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:24:41.244 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348630/064411bc-8f98-40aa-be67-19fd2ad6651c/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:24:41.626 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:24:41.627 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:24:41.627 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0246721 s, 42.5 MB/s 2021-02-10T11:24:41.627 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:41.628 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:41.628 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:41.963 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:24:41.964 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:24:41.964 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348630/064411bc-8f98-40aa-be67-19fd2ad6651c/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:24:42.340 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:24:42.340 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:24:42.340 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0233996 s, 44.8 MB/s 2021-02-10T11:24:42.341 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:42.341 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:42.341 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:42.678 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:24:42.678 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:24:42.678 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348630/064411bc-8f98-40aa-be67-19fd2ad6651c/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:24:43.055 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:24:43.055 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:24:43.055 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0255853 s, 41.0 MB/s 2021-02-10T11:24:43.056 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:43.056 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:43.057 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:43.393 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:24:43.393 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:24:43.393 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348630/064411bc-8f98-40aa-be67-19fd2ad6651c/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:24:43.769 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:24:43.769 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:24:43.770 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0250221 s, 41.9 MB/s 2021-02-10T11:24:43.770 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:43.771 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:43.771 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:44.106 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:24:44.106 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:24:44.106 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348630/064411bc-8f98-40aa-be67-19fd2ad6651c/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:24:44.478 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:24:44.479 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:24:44.479 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0259628 s, 40.4 MB/s 2021-02-10T11:24:44.480 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:44.480 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:44.480 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:44.816 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:24:44.816 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:24:44.816 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348630/064411bc-8f98-40aa-be67-19fd2ad6651c/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:24:45.193 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:24:45.193 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:24:45.194 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0249961 s, 41.9 MB/s 2021-02-10T11:24:45.194 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:45.195 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:45.195 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:45.530 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:24:45.530 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:24:45.530 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348630/064411bc-8f98-40aa-be67-19fd2ad6651c/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:24:45.906 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:24:45.906 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:24:45.906 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.024722 s, 42.4 MB/s 2021-02-10T11:24:45.907 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:45.908 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:45.908 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:46.242 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:24:46.242 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:24:46.242 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348630/064411bc-8f98-40aa-be67-19fd2ad6651c/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:24:46.625 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:24:46.625 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:24:46.626 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0237532 s, 44.1 MB/s 2021-02-10T11:24:46.627 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:46.627 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:46.627 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:46.976 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:24:46.976 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:24:46.976 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348630/064411bc-8f98-40aa-be67-19fd2ad6651c/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:24:47.352 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:24:47.352 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:24:47.353 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.022422 s, 46.8 MB/s 2021-02-10T11:24:47.353 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:47.354 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:47.354 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:47.689 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:24:47.689 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:24:47.689 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348630/064411bc-8f98-40aa-be67-19fd2ad6651c/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:24:48.065 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:24:48.065 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:24:48.066 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0254695 s, 41.2 MB/s 2021-02-10T11:24:48.066 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000348631 2021-02-10T11:24:48.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:48.490 7fc47ede3700 1 Processor -- start 2021-02-10T11:24:48.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:48.490 7fc47ede3700 1 -- start start 2021-02-10T11:24:48.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:48.491 7fc47ede3700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fc4781b0470 0x7fc4781b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:48.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:48.491 7fc47ede3700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4780f0b50 con 0x7fc4780f20e0 2021-02-10T11:24:48.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:48.491 7fc47ede3700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4780f24e0 con 0x7fc4780ef610 2021-02-10T11:24:48.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:48.491 7fc47ede3700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4780e5ea0 con 0x7fc4781b0470 2021-02-10T11:24:48.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:48.491 7fc47d400700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fc4780f20e0 0x7fc4780f0200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:45018/0 (socket says 172.21.2.112:45018) 2021-02-10T11:24:48.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:48.491 7fc47d400700 1 -- 172.21.2.112:0/2197613477 learned_addr learned my addr 172.21.2.112:0/2197613477 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:24:48.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:48.491 7fc47cbff700 1 --2- 172.21.2.112:0/2197613477 >> v2:172.21.2.120:3300/0 conn(0x7fc4781b0470 0x7fc4781b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:48.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:48.491 7fc4777fe700 1 -- 172.21.2.112:0/2197613477 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4279692408 0 0) 0x7fc468000fe0 con 0x7fc4780ef610 2021-02-10T11:24:48.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:48.491 7fc4777fe700 1 -- 172.21.2.112:0/2197613477 --> v1:172.21.2.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc464002fe0 con 0x7fc4780ef610 2021-02-10T11:24:48.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:48.491 7fc4777fe700 1 -- 172.21.2.112:0/2197613477 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3262112142 0 0) 0x7fc470001180 con 0x7fc4780f20e0 2021-02-10T11:24:48.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:48.491 7fc4777fe700 1 -- 172.21.2.112:0/2197613477 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc464004370 con 0x7fc4780f20e0 2021-02-10T11:24:48.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:48.491 7fc47cbff700 1 -- 172.21.2.112:0/2197613477 >> v1:172.21.2.120:6789/0 conn(0x7fc4780ef610 legacy=0x7fc4780efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:48.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:48.491 7fc47cbff700 1 -- 172.21.2.112:0/2197613477 >> v1:172.21.2.112:6789/0 conn(0x7fc4780f20e0 legacy=0x7fc4780f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:48.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:48.491 7fc47cbff700 1 -- 172.21.2.112:0/2197613477 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4780e52e0 con 0x7fc4781b0470 2021-02-10T11:24:48.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:48.492 7fc47cbff700 1 --2- 172.21.2.112:0/2197613477 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc4781b0470 0x7fc4781b0870 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7fc460006650 tx=0x7fc460009a80).ready entity=mon.1 client_cookie=c260d417bc8a78c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:48.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:48.492 7fc4777fe700 1 -- 172.21.2.112:0/2197613477 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc46000c0c0 con 0x7fc4781b0470 2021-02-10T11:24:48.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:48.492 7fc4777fe700 1 -- 172.21.2.112:0/2197613477 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc46000c340 con 0x7fc4781b0470 2021-02-10T11:24:48.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:48.492 7fc4777fe700 1 -- 172.21.2.112:0/2197613477 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc46000c860 con 0x7fc4781b0470 2021-02-10T11:24:48.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:48.492 7fc47ede3700 1 -- 172.21.2.112:0/2197613477 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc4781b0470 msgr2=0x7fc4781b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:48.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:48.492 7fc47ede3700 1 --2- 172.21.2.112:0/2197613477 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc4781b0470 0x7fc4781b0870 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7fc460006650 tx=0x7fc460009a80).stop 2021-02-10T11:24:48.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:48.492 7fc47ede3700 1 -- 172.21.2.112:0/2197613477 shutdown_connections 2021-02-10T11:24:48.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:48.492 7fc47ede3700 1 --2- 172.21.2.112:0/2197613477 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc4781b0470 0x7fc4781b0870 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:48.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:48.492 7fc47ede3700 1 -- 172.21.2.112:0/2197613477 >> 172.21.2.112:0/2197613477 conn(0x7fc4780ce3e0 msgr2=0x7fc4780e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:48.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:48.493 7fc47ede3700 1 -- 172.21.2.112:0/2197613477 shutdown_connections 2021-02-10T11:24:48.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:48.493 7fc47ede3700 1 -- 172.21.2.112:0/2197613477 wait complete. 2021-02-10T11:24:48.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:48.507 7fc47ede3700 1 Processor -- start 2021-02-10T11:24:48.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:48.507 7fc47ede3700 1 -- start start 2021-02-10T11:24:48.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:48.508 7fc47ede3700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc4780a0420 0x7fc4780a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:48.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:48.508 7fc47ede3700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc478189d00 0x7fc47818a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:48.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:48.508 7fc47ede3700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc47818a720 0x7fc4780b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:48.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:48.508 7fc47ede3700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc478189780 con 0x7fc47818a720 2021-02-10T11:24:48.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:48.508 7fc47ede3700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc47818ab20 con 0x7fc4780a0420 2021-02-10T11:24:48.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:48.508 7fc47ede3700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4780a1140 con 0x7fc478189d00 2021-02-10T11:24:48.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:48.508 7fc47d400700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc47818a720 0x7fc4780b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:48.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:48.508 7fc47d400700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc47818a720 0x7fc4780b7be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:45562/0 (socket says 172.21.2.112:45562) 2021-02-10T11:24:48.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:48.508 7fc47d400700 1 -- 172.21.2.112:0/3849748893 learned_addr learned my addr 172.21.2.112:0/3849748893 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:24:48.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:48.508 7fc47cbff700 1 --2- 172.21.2.112:0/3849748893 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc4780a0420 0x7fc4780a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:48.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:48.508 7fc477fff700 1 --2- 172.21.2.112:0/3849748893 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc478189d00 0x7fc47818a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:48.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:48.508 7fc47d400700 1 -- 172.21.2.112:0/3849748893 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc478189d00 msgr2=0x7fc47818a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:48.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:48.508 7fc47d400700 1 --2- 172.21.2.112:0/3849748893 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc478189d00 0x7fc47818a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:48.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:48.508 7fc47d400700 1 -- 172.21.2.112:0/3849748893 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc4780a0420 msgr2=0x7fc4780a0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:48.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:48.508 7fc47d400700 1 --2- 172.21.2.112:0/3849748893 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc4780a0420 0x7fc4780a0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:48.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:48.508 7fc47d400700 1 -- 172.21.2.112:0/3849748893 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4780b8200 con 0x7fc47818a720 2021-02-10T11:24:48.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:48.508 7fc47cbff700 1 --2- 172.21.2.112:0/3849748893 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc4780a0420 0x7fc4780a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:24:48.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:48.509 7fc47d400700 1 --2- 172.21.2.112:0/3849748893 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc47818a720 0x7fc4780b7be0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 rx=0x7fc47000a5d0 tx=0x7fc47000a970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:48.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:48.509 7fc4757fa700 1 -- 172.21.2.112:0/3849748893 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc47000ad60 con 0x7fc47818a720 2021-02-10T11:24:48.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:48.509 7fc4757fa700 1 -- 172.21.2.112:0/3849748893 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4700043a0 con 0x7fc47818a720 2021-02-10T11:24:48.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:48.509 7fc4757fa700 1 -- 172.21.2.112:0/3849748893 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc470004800 con 0x7fc47818a720 2021-02-10T11:24:48.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:48.509 7fc47ede3700 1 -- 172.21.2.112:0/3849748893 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4780b8200 con 0x7fc47818a720 2021-02-10T11:24:48.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:48.509 7fc47ede3700 1 -- 172.21.2.112:0/3849748893 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4780b8790 con 0x7fc47818a720 2021-02-10T11:24:48.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:48.509 7fc4757fa700 1 -- 172.21.2.112:0/3849748893 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc470007070 con 0x7fc47818a720 2021-02-10T11:24:48.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:48.510 7fc4757fa700 1 --2- 172.21.2.112:0/3849748893 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc45c0186c0 0x7fc45c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:48.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:48.510 7fc4757fa700 1 -- 172.21.2.112:0/3849748893 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fc4700325b0 con 0x7fc47818a720 2021-02-10T11:24:48.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:48.510 7fc47cbff700 1 --2- 172.21.2.112:0/3849748893 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc45c0186c0 0x7fc45c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:48.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:48.510 7fc47cbff700 1 --2- 172.21.2.112:0/3849748893 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc45c0186c0 0x7fc45c01abc0 crc :-1 s=READY pgs=301 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:48.531 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:48.540 7fc47ede3700 1 -- 172.21.2.112:0/3849748893 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc478105ae0 con 0x7fc47818a720 2021-02-10T11:24:48.533 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:48.542 7fc4757fa700 1 -- 172.21.2.112:0/3849748893 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc470000bc0 con 0x7fc47818a720 2021-02-10T11:24:48.660 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:48.669 7fc47ede3700 1 -- 172.21.2.112:0/3849748893 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348631", "target": ["mgr", ""]}) v1 -- 0x7fc4780b9020 con 0x7fc45c0186c0 2021-02-10T11:24:48.685 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:48.695 7fc4757fa700 1 -- 172.21.2.112:0/3849748893 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fc460006c70 con 0x7fc45c0186c0 2021-02-10T11:24:48.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:48.703 7fc47ede3700 1 -- 172.21.2.112:0/3849748893 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc45c0186c0 msgr2=0x7fc45c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:48.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:48.703 7fc47ede3700 1 --2- 172.21.2.112:0/3849748893 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc45c0186c0 0x7fc45c01abc0 crc :-1 s=READY pgs=301 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:48.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:48.703 7fc47ede3700 1 -- 172.21.2.112:0/3849748893 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc47818a720 msgr2=0x7fc4780b7be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:48.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:48.703 7fc47ede3700 1 --2- 172.21.2.112:0/3849748893 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc47818a720 0x7fc4780b7be0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 rx=0x7fc47000a5d0 tx=0x7fc47000a970).stop 2021-02-10T11:24:48.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:48.703 7fc47ede3700 1 -- 172.21.2.112:0/3849748893 shutdown_connections 2021-02-10T11:24:48.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:48.703 7fc47ede3700 1 --2- 172.21.2.112:0/3849748893 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc45c0186c0 0x7fc45c01abc0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:48.695 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:48.703 7fc47ede3700 1 --2- 172.21.2.112:0/3849748893 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc47818a720 0x7fc4780b7be0 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:48.695 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:48.703 7fc47ede3700 1 --2- 172.21.2.112:0/3849748893 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc478189d00 0x7fc47818a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:48.695 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:48.703 7fc47ede3700 1 --2- 172.21.2.112:0/3849748893 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc4780a0420 0x7fc4780a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:48.695 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:48.703 7fc47ede3700 1 -- 172.21.2.112:0/3849748893 >> 172.21.2.112:0/3849748893 conn(0x7fc478094e10 msgr2=0x7fc4780a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:48.695 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:48.703 7fc47ede3700 1 -- 172.21.2.112:0/3849748893 shutdown_connections 2021-02-10T11:24:48.695 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:48.703 7fc47ede3700 1 -- 172.21.2.112:0/3849748893 wait complete. 2021-02-10T11:24:48.702 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000348631 2021-02-10T11:24:49.136 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:49.127 7fcf9e80b700 1 Processor -- start 2021-02-10T11:24:49.136 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:49.127 7fcf9e80b700 1 -- start start 2021-02-10T11:24:49.136 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:49.128 7fcf9e80b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcf981b0870 0x7fcf980f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:49.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:49.128 7fcf9e80b700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fcf980efe00 0x7fcf980f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:49.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:49.128 7fcf9e80b700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf981b0c70 con 0x7fcf981b0470 2021-02-10T11:24:49.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:49.128 7fcf9e80b700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf980f0b50 con 0x7fcf980efe00 2021-02-10T11:24:49.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:49.128 7fcf9e80b700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf980f0820 con 0x7fcf981b0870 2021-02-10T11:24:49.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:49.128 7fcf97fff700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fcf981b0470 0x7fcf980ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:45028/0 (socket says 172.21.2.112:45028) 2021-02-10T11:24:49.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:49.128 7fcf97fff700 1 -- 172.21.2.112:0/4271270653 learned_addr learned my addr 172.21.2.112:0/4271270653 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:24:49.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:49.128 7fcf977fe700 1 --2- 172.21.2.112:0/4271270653 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcf981b0870 0x7fcf980f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:49.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:49.128 7fcf9ce28700 1 --2- 172.21.2.112:0/4271270653 >> v2:172.21.2.120:3300/0 conn(0x7fcf980efe00 0x7fcf980f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:49.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:49.128 7fcf96ffd700 1 -- 172.21.2.112:0/4271270653 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2484993714 0 0) 0x7fcf88001180 con 0x7fcf981b0470 2021-02-10T11:24:49.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:49.128 7fcf96ffd700 1 -- 172.21.2.112:0/4271270653 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf80002fe0 con 0x7fcf981b0470 2021-02-10T11:24:49.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:49.129 7fcf96ffd700 1 -- 172.21.2.112:0/4271270653 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3210537919 0 0) 0x7fcf880010e0 con 0x7fcf981b0470 2021-02-10T11:24:49.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:49.129 7fcf96ffd700 1 -- 172.21.2.112:0/4271270653 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcf981b0870 msgr2=0x7fcf980f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:49.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:49.129 7fcf96ffd700 1 --2- 172.21.2.112:0/4271270653 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcf981b0870 0x7fcf980f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:49.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:49.129 7fcf96ffd700 1 -- 172.21.2.112:0/4271270653 >> v2:172.21.2.120:3300/0 conn(0x7fcf980efe00 msgr2=0x7fcf980f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:49.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:49.129 7fcf96ffd700 1 --2- 172.21.2.112:0/4271270653 >> v2:172.21.2.120:3300/0 conn(0x7fcf980efe00 0x7fcf980f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:49.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:49.129 7fcf96ffd700 1 -- 172.21.2.112:0/4271270653 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf980e52e0 con 0x7fcf981b0470 2021-02-10T11:24:49.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:49.129 7fcf96ffd700 1 -- 172.21.2.112:0/4271270653 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7fcf88001600 con 0x7fcf981b0470 2021-02-10T11:24:49.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:49.129 7fcf96ffd700 1 -- 172.21.2.112:0/4271270653 >> v1:172.21.2.112:6789/0 conn(0x7fcf981b0470 legacy=0x7fcf980ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:49.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:49.129 7fcf96ffd700 1 --2- 172.21.2.112:0/4271270653 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcf80003a40 0x7fcf80005f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:49.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:49.129 7fcf96ffd700 1 --2- 172.21.2.112:0/4271270653 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcf8004a7d0 0x7fcf8004cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:49.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:49.129 7fcf96ffd700 1 --2- 172.21.2.112:0/4271270653 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcf8004d500 0x7fcf8004fa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:49.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:49.129 7fcf96ffd700 1 -- 172.21.2.112:0/4271270653 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf800502d0 con 0x7fcf80003a40 2021-02-10T11:24:49.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:49.129 7fcf96ffd700 1 -- 172.21.2.112:0/4271270653 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf800504f0 con 0x7fcf8004d500 2021-02-10T11:24:49.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:49.129 7fcf96ffd700 1 -- 172.21.2.112:0/4271270653 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf80050710 con 0x7fcf8004a7d0 2021-02-10T11:24:49.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:49.129 7fcf97fff700 1 --2- 172.21.2.112:0/4271270653 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcf80003a40 0x7fcf80005f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:49.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:49.129 7fcf977fe700 1 --2- 172.21.2.112:0/4271270653 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcf8004a7d0 0x7fcf8004cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:49.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:49.129 7fcf9ce28700 1 --2- 172.21.2.112:0/4271270653 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcf8004d500 0x7fcf8004fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:49.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:49.129 7fcf97fff700 1 -- 172.21.2.112:0/4271270653 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcf8004a7d0 msgr2=0x7fcf8004cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:49.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:49.129 7fcf97fff700 1 --2- 172.21.2.112:0/4271270653 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcf8004a7d0 0x7fcf8004cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:49.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:49.129 7fcf97fff700 1 -- 172.21.2.112:0/4271270653 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcf8004d500 msgr2=0x7fcf8004fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:49.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:49.129 7fcf97fff700 1 --2- 172.21.2.112:0/4271270653 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcf8004d500 0x7fcf8004fa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:49.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:49.129 7fcf97fff700 1 -- 172.21.2.112:0/4271270653 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf800509b0 con 0x7fcf80003a40 2021-02-10T11:24:49.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:49.129 7fcf97fff700 1 --2- 172.21.2.112:0/4271270653 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcf80003a40 0x7fcf80005f60 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 rx=0x7fcf88007a90 tx=0x7fcf88001c00).ready entity=mon.0 client_cookie=f03d85656d0f4867 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:49.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:49.129 7fcf96ffd700 1 -- 172.21.2.112:0/4271270653 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcf880070c0 con 0x7fcf80003a40 2021-02-10T11:24:49.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:49.129 7fcf96ffd700 1 -- 172.21.2.112:0/4271270653 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf88007340 con 0x7fcf80003a40 2021-02-10T11:24:49.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:49.129 7fcf96ffd700 1 -- 172.21.2.112:0/4271270653 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcf8800a520 con 0x7fcf80003a40 2021-02-10T11:24:49.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:49.129 7fcf9e80b700 1 -- 172.21.2.112:0/4271270653 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcf80003a40 msgr2=0x7fcf80005f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:49.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:49.129 7fcf9e80b700 1 --2- 172.21.2.112:0/4271270653 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcf80003a40 0x7fcf80005f60 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 rx=0x7fcf88007a90 tx=0x7fcf88001c00).stop 2021-02-10T11:24:49.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:49.129 7fcf9ce28700 1 -- 172.21.2.112:0/4271270653 reap_dead start 2021-02-10T11:24:49.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:49.130 7fcf9e80b700 1 -- 172.21.2.112:0/4271270653 shutdown_connections 2021-02-10T11:24:49.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:49.130 7fcf9e80b700 1 -- 172.21.2.112:0/4271270653 >> 172.21.2.112:0/4271270653 conn(0x7fcf980ce3e0 msgr2=0x7fcf980e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:49.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:49.130 7fcf9e80b700 1 -- 172.21.2.112:0/4271270653 shutdown_connections 2021-02-10T11:24:49.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:49.130 7fcf9e80b700 1 -- 172.21.2.112:0/4271270653 wait complete. 2021-02-10T11:24:49.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:49.145 7fcf9e80b700 1 Processor -- start 2021-02-10T11:24:49.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:49.145 7fcf9e80b700 1 -- start start 2021-02-10T11:24:49.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:49.145 7fcf9e80b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcf9809e560 0x7fcf980a0130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:49.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:49.145 7fcf9e80b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcf9809e960 0x7fcf980a0750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:49.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:49.145 7fcf9e80b700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcf980a0d70 0x7fcf980a1af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:49.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:49.145 7fcf9e80b700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf980c38f0 con 0x7fcf9809e960 2021-02-10T11:24:49.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:49.145 7fcf9e80b700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf980a1170 con 0x7fcf980a0d70 2021-02-10T11:24:49.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:49.145 7fcf9e80b700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf980a1390 con 0x7fcf9809e560 2021-02-10T11:24:49.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:49.145 7fcf977fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcf9809e960 0x7fcf980a0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:49.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:49.145 7fcf977fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcf9809e960 0x7fcf980a0750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:45578/0 (socket says 172.21.2.112:45578) 2021-02-10T11:24:49.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:49.145 7fcf977fe700 1 -- 172.21.2.112:0/382343254 learned_addr learned my addr 172.21.2.112:0/382343254 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:24:49.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:49.146 7fcf9ce28700 1 --2- 172.21.2.112:0/382343254 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcf980a0d70 0x7fcf980a1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:49.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:49.146 7fcf977fe700 1 -- 172.21.2.112:0/382343254 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcf9809e560 msgr2=0x7fcf980a0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:49.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:49.146 7fcf977fe700 1 --2- 172.21.2.112:0/382343254 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcf9809e560 0x7fcf980a0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:49.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:49.146 7fcf977fe700 1 -- 172.21.2.112:0/382343254 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcf980a0d70 msgr2=0x7fcf980a1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:49.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:49.146 7fcf977fe700 1 --2- 172.21.2.112:0/382343254 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcf980a0d70 0x7fcf980a1af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:49.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:49.146 7fcf977fe700 1 -- 172.21.2.112:0/382343254 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf980a2110 con 0x7fcf9809e960 2021-02-10T11:24:49.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:49.146 7fcf977fe700 1 --2- 172.21.2.112:0/382343254 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcf9809e960 0x7fcf980a0750 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 rx=0x7fcf8c006f60 tx=0x7fcf8c00b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:49.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:49.146 7fcf94ff9700 1 -- 172.21.2.112:0/382343254 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcf8c00bd60 con 0x7fcf9809e960 2021-02-10T11:24:49.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:49.146 7fcf94ff9700 1 -- 172.21.2.112:0/382343254 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf8c000c90 con 0x7fcf9809e960 2021-02-10T11:24:49.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:49.146 7fcf94ff9700 1 -- 172.21.2.112:0/382343254 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcf8c002300 con 0x7fcf9809e960 2021-02-10T11:24:49.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:49.146 7fcf9e80b700 1 -- 172.21.2.112:0/382343254 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf980a2110 con 0x7fcf9809e960 2021-02-10T11:24:49.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:49.146 7fcf9e80b700 1 -- 172.21.2.112:0/382343254 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf980a26a0 con 0x7fcf9809e960 2021-02-10T11:24:49.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:49.147 7fcf94ff9700 1 -- 172.21.2.112:0/382343254 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fcf8c006b00 con 0x7fcf9809e960 2021-02-10T11:24:49.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:49.147 7fcf94ff9700 1 --2- 172.21.2.112:0/382343254 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcf780186c0 0x7fcf7801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:49.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:49.147 7fcf94ff9700 1 -- 172.21.2.112:0/382343254 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fcf8c032590 con 0x7fcf9809e960 2021-02-10T11:24:49.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:49.147 7fcf97fff700 1 --2- 172.21.2.112:0/382343254 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcf780186c0 0x7fcf7801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:49.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:49.148 7fcf97fff700 1 --2- 172.21.2.112:0/382343254 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcf780186c0 0x7fcf7801abc0 crc :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:49.169 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:49.178 7fcf9e80b700 1 -- 172.21.2.112:0/382343254 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf98105b30 con 0x7fcf9809e960 2021-02-10T11:24:49.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:49.180 7fcf94ff9700 1 -- 172.21.2.112:0/382343254 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fcf8c02cf10 con 0x7fcf9809e960 2021-02-10T11:24:49.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:49.308 7fcf9e80b700 1 -- 172.21.2.112:0/382343254 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348631", "target": ["mgr", ""]}) v1 -- 0x7fcf980a26a0 con 0x7fcf780186c0 2021-02-10T11:24:49.303 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:49.312 7fcf94ff9700 1 -- 172.21.2.112:0/382343254 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7fcf88001a30 con 0x7fcf780186c0 2021-02-10T11:24:49.306 INFO:teuthology.orchestra.run.gibba012.stdout:/volumes/_nogroup/subvolume_0000000000348631/9da0393a-c5da-4f55-8268-fc91bee9f809 2021-02-10T11:24:49.308 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:49.317 7fcf9e80b700 1 -- 172.21.2.112:0/382343254 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcf780186c0 msgr2=0x7fcf7801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:49.308 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:49.317 7fcf9e80b700 1 --2- 172.21.2.112:0/382343254 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcf780186c0 0x7fcf7801abc0 crc :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:49.308 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:49.317 7fcf9e80b700 1 -- 172.21.2.112:0/382343254 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcf9809e960 msgr2=0x7fcf980a0750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:49.308 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:49.317 7fcf9e80b700 1 --2- 172.21.2.112:0/382343254 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcf9809e960 0x7fcf980a0750 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 rx=0x7fcf8c006f60 tx=0x7fcf8c00b970).stop 2021-02-10T11:24:49.308 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:49.317 7fcf9e80b700 1 -- 172.21.2.112:0/382343254 shutdown_connections 2021-02-10T11:24:49.308 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:49.317 7fcf9e80b700 1 --2- 172.21.2.112:0/382343254 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcf780186c0 0x7fcf7801abc0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:49.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:49.317 7fcf9e80b700 1 --2- 172.21.2.112:0/382343254 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcf980a0d70 0x7fcf980a1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:49.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:49.317 7fcf9e80b700 1 --2- 172.21.2.112:0/382343254 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcf9809e960 0x7fcf980a0750 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:49.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:49.317 7fcf9e80b700 1 --2- 172.21.2.112:0/382343254 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcf9809e560 0x7fcf980a0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:49.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:49.317 7fcf9e80b700 1 -- 172.21.2.112:0/382343254 >> 172.21.2.112:0/382343254 conn(0x7fcf9809fc10 msgr2=0x7fcf98094ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:49.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:49.317 7fcf9e80b700 1 -- 172.21.2.112:0/382343254 shutdown_connections 2021-02-10T11:24:49.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:49.317 7fcf9e80b700 1 -- 172.21.2.112:0/382343254 wait complete. 2021-02-10T11:24:49.316 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000348631 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000348631/9da0393a-c5da-4f55-8268-fc91bee9f809 2021-02-10T11:24:49.316 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:49.317 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:49.317 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:49.653 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:24:49.653 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:24:49.653 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348631/9da0393a-c5da-4f55-8268-fc91bee9f809/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:24:50.031 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:24:50.031 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:24:50.032 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.023546 s, 44.5 MB/s 2021-02-10T11:24:50.032 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:50.033 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:50.033 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:50.370 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:24:50.370 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:24:50.370 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348631/9da0393a-c5da-4f55-8268-fc91bee9f809/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:24:50.747 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:24:50.747 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:24:50.747 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0249529 s, 42.0 MB/s 2021-02-10T11:24:50.748 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:50.748 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:50.748 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:51.083 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:24:51.084 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:24:51.084 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348631/9da0393a-c5da-4f55-8268-fc91bee9f809/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:24:51.462 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:24:51.462 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:24:51.462 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0261045 s, 40.2 MB/s 2021-02-10T11:24:51.463 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:51.463 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:51.463 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:51.797 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:24:51.797 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:24:51.798 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348631/9da0393a-c5da-4f55-8268-fc91bee9f809/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:24:52.174 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:24:52.174 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:24:52.174 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0246304 s, 42.6 MB/s 2021-02-10T11:24:52.175 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:52.175 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:52.175 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:52.511 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:24:52.511 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:24:52.511 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348631/9da0393a-c5da-4f55-8268-fc91bee9f809/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:24:52.886 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:24:52.886 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:24:52.886 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0244899 s, 42.8 MB/s 2021-02-10T11:24:52.887 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:52.888 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:52.888 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:53.224 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:24:53.224 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:24:53.224 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348631/9da0393a-c5da-4f55-8268-fc91bee9f809/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:24:53.599 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:24:53.599 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:24:53.599 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0237797 s, 44.1 MB/s 2021-02-10T11:24:53.600 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:53.600 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:53.600 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:53.936 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:24:53.936 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:24:53.936 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348631/9da0393a-c5da-4f55-8268-fc91bee9f809/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:24:54.314 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:24:54.314 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:24:54.314 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0261092 s, 40.2 MB/s 2021-02-10T11:24:54.315 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:54.316 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:54.316 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:54.652 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:24:54.652 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:24:54.652 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348631/9da0393a-c5da-4f55-8268-fc91bee9f809/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:24:55.023 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:24:55.024 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:24:55.024 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0236348 s, 44.4 MB/s 2021-02-10T11:24:55.025 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:55.025 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:55.025 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:55.361 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:24:55.362 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:24:55.362 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348631/9da0393a-c5da-4f55-8268-fc91bee9f809/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:24:55.740 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:24:55.740 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:24:55.740 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0246036 s, 42.6 MB/s 2021-02-10T11:24:55.741 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:55.741 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:55.741 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:56.077 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:24:56.078 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:24:56.078 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348631/9da0393a-c5da-4f55-8268-fc91bee9f809/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:24:56.452 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:24:56.452 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:24:56.452 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0229464 s, 45.7 MB/s 2021-02-10T11:24:56.453 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000348632 2021-02-10T11:24:56.884 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:56.877 7f7072395700 1 Processor -- start 2021-02-10T11:24:56.884 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:56.877 7f7072395700 1 -- start start 2021-02-10T11:24:56.884 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:56.878 7f7072395700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f706c1b0470 0x7f706c1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:56.884 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:56.878 7f7072395700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f706c0ef610 0x7f706c0efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:56.884 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:56.878 7f7072395700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f706c0f0030 0x7f706c0f0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:56.885 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:56.878 7f7072395700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f706c0e5ea0 con 0x7f706c0f0030 2021-02-10T11:24:56.885 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:56.878 7f7072395700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f706c0f0b50 con 0x7f706c1b0470 2021-02-10T11:24:56.885 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:56.878 7f7072395700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f706c0e52e0 con 0x7f706c0ef610 2021-02-10T11:24:56.885 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:56.878 7f70709b2700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f706c0f0030 0x7f706c0f0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:56.885 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:56.878 7f70709b2700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f706c0f0030 0x7f706c0f0430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:45590/0 (socket says 172.21.2.112:45590) 2021-02-10T11:24:56.885 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:56.878 7f70709b2700 1 -- 172.21.2.112:0/4202323536 learned_addr learned my addr 172.21.2.112:0/4202323536 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:24:56.885 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:56.878 7f70709b2700 1 -- 172.21.2.112:0/4202323536 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f706c0ef610 msgr2=0x7f706c0efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:56.885 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:56.878 7f70709b2700 1 --2- 172.21.2.112:0/4202323536 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f706c0ef610 0x7f706c0efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:56.885 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:56.878 7f70709b2700 1 -- 172.21.2.112:0/4202323536 >> v2:172.21.2.120:3300/0 conn(0x7f706c1b0470 msgr2=0x7f706c1b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:56.885 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:56.878 7f70709b2700 1 --2- 172.21.2.112:0/4202323536 >> v2:172.21.2.120:3300/0 conn(0x7f706c1b0470 0x7f706c1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:56.885 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:56.878 7f70709b2700 1 -- 172.21.2.112:0/4202323536 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f706c0f10d0 con 0x7f706c0f0030 2021-02-10T11:24:56.885 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:56.878 7f70709b2700 1 --2- 172.21.2.112:0/4202323536 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f706c0f0030 0x7f706c0f0430 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 rx=0x7f7064006f00 tx=0x7f7064009f10).ready entity=mon.0 client_cookie=208556f122091952 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:56.886 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:56.878 7f706affd700 1 -- 172.21.2.112:0/4202323536 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f706400b3b0 con 0x7f706c0f0030 2021-02-10T11:24:56.886 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:56.878 7f706affd700 1 -- 172.21.2.112:0/4202323536 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f706400b630 con 0x7f706c0f0030 2021-02-10T11:24:56.886 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:56.878 7f706affd700 1 -- 172.21.2.112:0/4202323536 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f706400bb10 con 0x7f706c0f0030 2021-02-10T11:24:56.886 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:56.878 7f7072395700 1 -- 172.21.2.112:0/4202323536 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f706c0f0030 msgr2=0x7f706c0f0430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:56.886 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:56.878 7f7072395700 1 --2- 172.21.2.112:0/4202323536 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f706c0f0030 0x7f706c0f0430 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 rx=0x7f7064006f00 tx=0x7f7064009f10).stop 2021-02-10T11:24:56.886 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:56.879 7f7072395700 1 -- 172.21.2.112:0/4202323536 shutdown_connections 2021-02-10T11:24:56.886 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:56.879 7f7072395700 1 --2- 172.21.2.112:0/4202323536 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f706c0f0030 0x7f706c0f0430 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:56.886 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:56.879 7f7072395700 1 --2- 172.21.2.112:0/4202323536 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f706c0ef610 0x7f706c0efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:56.886 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:56.879 7f7072395700 1 --2- 172.21.2.112:0/4202323536 >> v2:172.21.2.120:3300/0 conn(0x7f706c1b0470 0x7f706c1b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:56.886 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:56.879 7f7072395700 1 -- 172.21.2.112:0/4202323536 >> 172.21.2.112:0/4202323536 conn(0x7f706c0ce3e0 msgr2=0x7f706c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:56.886 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:56.879 7f7072395700 1 -- 172.21.2.112:0/4202323536 shutdown_connections 2021-02-10T11:24:56.886 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:56.879 7f7072395700 1 -- 172.21.2.112:0/4202323536 wait complete. 2021-02-10T11:24:56.888 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:56.894 7f7072395700 1 Processor -- start 2021-02-10T11:24:56.888 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:56.894 7f7072395700 1 -- start start 2021-02-10T11:24:56.888 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:56.894 7f7072395700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f706c0b20b0 0x7f706c0ac970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:56.888 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:56.894 7f7072395700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f706c0b24b0 0x7f706c191dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:56.888 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:56.894 7f7072395700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f706c0acf90 0x7f706c1923e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:56.888 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:56.894 7f7072395700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f706c0b1b30 con 0x7f706c0acf90 2021-02-10T11:24:56.888 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:56.894 7f7072395700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f706c1a04e0 con 0x7f706c0b24b0 2021-02-10T11:24:56.888 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:56.894 7f7072395700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f706c1a06c0 con 0x7f706c0b20b0 2021-02-10T11:24:56.889 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:56.894 7f70709b2700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f706c0acf90 0x7f706c1923e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:56.889 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:56.894 7f70709b2700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f706c0acf90 0x7f706c1923e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:45592/0 (socket says 172.21.2.112:45592) 2021-02-10T11:24:56.889 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:56.894 7f70709b2700 1 -- 172.21.2.112:0/2405687093 learned_addr learned my addr 172.21.2.112:0/2405687093 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:24:56.889 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:56.895 7f70709b2700 1 -- 172.21.2.112:0/2405687093 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f706c0b20b0 msgr2=0x7f706c0ac970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:56.889 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:56.895 7f70709b2700 1 --2- 172.21.2.112:0/2405687093 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f706c0b20b0 0x7f706c0ac970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:56.889 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:56.895 7f70709b2700 1 -- 172.21.2.112:0/2405687093 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f706c0b24b0 msgr2=0x7f706c191dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:56.889 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:56.895 7f70709b2700 1 --2- 172.21.2.112:0/2405687093 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f706c0b24b0 0x7f706c191dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:56.889 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:56.895 7f70709b2700 1 -- 172.21.2.112:0/2405687093 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f706c192be0 con 0x7f706c0acf90 2021-02-10T11:24:56.890 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:56.895 7f70709b2700 1 --2- 172.21.2.112:0/2405687093 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f706c0acf90 0x7f706c1923e0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 rx=0x7f706400b8c0 tx=0x7f706400bc60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:56.890 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:56.895 7f7068ff9700 1 -- 172.21.2.112:0/2405687093 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7064000f90 con 0x7f706c0acf90 2021-02-10T11:24:56.890 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:56.895 7f7068ff9700 1 -- 172.21.2.112:0/2405687093 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f706400bda0 con 0x7f706c0acf90 2021-02-10T11:24:56.890 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:56.895 7f7072395700 1 -- 172.21.2.112:0/2405687093 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f706c192be0 con 0x7f706c0acf90 2021-02-10T11:24:56.890 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:56.895 7f7072395700 1 -- 172.21.2.112:0/2405687093 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f706c193170 con 0x7f706c0acf90 2021-02-10T11:24:56.890 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:56.895 7f7068ff9700 1 -- 172.21.2.112:0/2405687093 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f70640014f0 con 0x7f706c0acf90 2021-02-10T11:24:56.890 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:56.896 7f7068ff9700 1 -- 172.21.2.112:0/2405687093 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7064001270 con 0x7f706c0acf90 2021-02-10T11:24:56.890 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:56.896 7f7068ff9700 1 --2- 172.21.2.112:0/2405687093 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f704c0186c0 0x7f704c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:56.891 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:56.896 7f7068ff9700 1 -- 172.21.2.112:0/2405687093 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f70640328e0 con 0x7f706c0acf90 2021-02-10T11:24:56.891 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:56.896 7f706bfff700 1 --2- 172.21.2.112:0/2405687093 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f704c0186c0 0x7f704c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:56.891 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:56.897 7f706bfff700 1 --2- 172.21.2.112:0/2405687093 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f704c0186c0 0x7f704c01abc0 crc :-1 s=READY pgs=303 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:56.918 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:56.927 7f7072395700 1 -- 172.21.2.112:0/2405687093 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f706c105af0 con 0x7f706c0acf90 2021-02-10T11:24:56.921 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:56.930 7f7068ff9700 1 -- 172.21.2.112:0/2405687093 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f706402b010 con 0x7f706c0acf90 2021-02-10T11:24:57.047 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:57.057 7f7072395700 1 -- 172.21.2.112:0/2405687093 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348632", "target": ["mgr", ""]}) v1 -- 0x7f706c1a06c0 con 0x7f704c0186c0 2021-02-10T11:24:57.073 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:57.082 7f7068ff9700 1 -- 172.21.2.112:0/2405687093 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f705c0009b0 con 0x7f704c0186c0 2021-02-10T11:24:57.081 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:57.090 7f7072395700 1 -- 172.21.2.112:0/2405687093 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f704c0186c0 msgr2=0x7f704c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:57.081 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:57.090 7f7072395700 1 --2- 172.21.2.112:0/2405687093 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f704c0186c0 0x7f704c01abc0 crc :-1 s=READY pgs=303 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:57.081 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:57.090 7f7072395700 1 -- 172.21.2.112:0/2405687093 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f706c0acf90 msgr2=0x7f706c1923e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:57.081 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:57.090 7f7072395700 1 --2- 172.21.2.112:0/2405687093 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f706c0acf90 0x7f706c1923e0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 rx=0x7f706400b8c0 tx=0x7f706400bc60).stop 2021-02-10T11:24:57.082 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:57.090 7f7072395700 1 -- 172.21.2.112:0/2405687093 shutdown_connections 2021-02-10T11:24:57.082 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:57.090 7f7072395700 1 --2- 172.21.2.112:0/2405687093 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f704c0186c0 0x7f704c01abc0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:57.082 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:57.090 7f7072395700 1 --2- 172.21.2.112:0/2405687093 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f706c0b24b0 0x7f706c191dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:57.082 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:57.090 7f7072395700 1 --2- 172.21.2.112:0/2405687093 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f706c0acf90 0x7f706c1923e0 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:57.082 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:57.090 7f7072395700 1 --2- 172.21.2.112:0/2405687093 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f706c0b20b0 0x7f706c0ac970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:57.082 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:57.090 7f7072395700 1 -- 172.21.2.112:0/2405687093 >> 172.21.2.112:0/2405687093 conn(0x7f706c0bf4e0 msgr2=0x7f706c0be3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:57.082 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:57.091 7f7072395700 1 -- 172.21.2.112:0/2405687093 shutdown_connections 2021-02-10T11:24:57.082 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:57.091 7f7072395700 1 -- 172.21.2.112:0/2405687093 wait complete. 2021-02-10T11:24:57.089 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000348632 2021-02-10T11:24:57.524 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:57.515 7fd161eb8700 1 Processor -- start 2021-02-10T11:24:57.524 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:57.515 7fd161eb8700 1 -- start start 2021-02-10T11:24:57.524 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:57.515 7fd161eb8700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fd15c0f05f0 0x7fd15c0f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:57.524 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:57.515 7fd161eb8700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd15c1b0c70 con 0x7fd15c1b0870 2021-02-10T11:24:57.524 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:57.515 7fd161eb8700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd15c0f0b50 con 0x7fd15c1b0470 2021-02-10T11:24:57.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:57.515 7fd161eb8700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd15c0e5ea0 con 0x7fd15c0f05f0 2021-02-10T11:24:57.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:57.516 7fd15affd700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fd15c1b0870 0x7fd15c0efe00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:45062/0 (socket says 172.21.2.112:45062) 2021-02-10T11:24:57.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:57.516 7fd15affd700 1 -- 172.21.2.112:0/1577765642 learned_addr learned my addr 172.21.2.112:0/1577765642 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:24:57.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:57.516 7fd15a7fc700 1 -- 172.21.2.112:0/1577765642 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1716316351 0 0) 0x7fd14c000ce0 con 0x7fd15c1b0870 2021-02-10T11:24:57.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:57.516 7fd15a7fc700 1 -- 172.21.2.112:0/1577765642 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd148002fe0 con 0x7fd15c1b0870 2021-02-10T11:24:57.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:57.516 7fd15bfff700 1 --2- 172.21.2.112:0/1577765642 >> v2:172.21.2.120:3300/0 conn(0x7fd15c0f05f0 0x7fd15c0f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:57.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:57.516 7fd15a7fc700 1 -- 172.21.2.112:0/1577765642 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 69818967 0 0) 0x7fd14c001180 con 0x7fd15c1b0870 2021-02-10T11:24:57.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:57.516 7fd15a7fc700 1 -- 172.21.2.112:0/1577765642 >> v2:172.21.2.120:3300/0 conn(0x7fd15c0f05f0 msgr2=0x7fd15c0f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:57.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:57.516 7fd15a7fc700 1 --2- 172.21.2.112:0/1577765642 >> v2:172.21.2.120:3300/0 conn(0x7fd15c0f05f0 0x7fd15c0f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:57.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:57.516 7fd15a7fc700 1 -- 172.21.2.112:0/1577765642 >> v1:172.21.2.120:6789/0 conn(0x7fd15c1b0470 legacy=0x7fd15c0ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:57.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:57.516 7fd15a7fc700 1 -- 172.21.2.112:0/1577765642 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd15c0e52e0 con 0x7fd15c1b0870 2021-02-10T11:24:57.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:57.517 7fd15a7fc700 1 -- 172.21.2.112:0/1577765642 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7fd14c001680 con 0x7fd15c1b0870 2021-02-10T11:24:57.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:57.517 7fd15a7fc700 1 -- 172.21.2.112:0/1577765642 >> v1:172.21.2.112:6789/0 conn(0x7fd15c1b0870 legacy=0x7fd15c0efe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:57.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:57.517 7fd15a7fc700 1 --2- 172.21.2.112:0/1577765642 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd148003a60 0x7fd148005f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:57.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:57.517 7fd15a7fc700 1 --2- 172.21.2.112:0/1577765642 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd148006780 0x7fd148008ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:57.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:57.517 7fd15a7fc700 1 --2- 172.21.2.112:0/1577765642 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd1480095a0 0x7fd14800bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:57.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:57.517 7fd15a7fc700 1 -- 172.21.2.112:0/1577765642 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd14800c2f0 con 0x7fd148006780 2021-02-10T11:24:57.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:57.517 7fd15a7fc700 1 -- 172.21.2.112:0/1577765642 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd14800c510 con 0x7fd1480095a0 2021-02-10T11:24:57.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:57.517 7fd15a7fc700 1 -- 172.21.2.112:0/1577765642 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd14800c730 con 0x7fd148003a60 2021-02-10T11:24:57.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:57.517 7fd15b7fe700 1 --2- 172.21.2.112:0/1577765642 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd148003a60 0x7fd148005f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:57.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:57.517 7fd15affd700 1 --2- 172.21.2.112:0/1577765642 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd148006780 0x7fd148008ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:57.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:57.517 7fd15bfff700 1 --2- 172.21.2.112:0/1577765642 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd1480095a0 0x7fd14800bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:57.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:57.517 7fd15b7fe700 1 -- 172.21.2.112:0/1577765642 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd1480095a0 msgr2=0x7fd14800bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:57.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:57.517 7fd15b7fe700 1 --2- 172.21.2.112:0/1577765642 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd1480095a0 0x7fd14800bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:57.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:57.517 7fd15b7fe700 1 -- 172.21.2.112:0/1577765642 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd148006780 msgr2=0x7fd148008ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:57.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:57.517 7fd15b7fe700 1 --2- 172.21.2.112:0/1577765642 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd148006780 0x7fd148008ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:57.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:57.517 7fd15b7fe700 1 -- 172.21.2.112:0/1577765642 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1480509a0 con 0x7fd148003a60 2021-02-10T11:24:57.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:57.517 7fd15affd700 1 --2- 172.21.2.112:0/1577765642 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd148006780 0x7fd148008ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:24:57.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:57.517 7fd15bfff700 1 --2- 172.21.2.112:0/1577765642 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd1480095a0 0x7fd14800bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:24:57.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:57.517 7fd15bfff700 1 -- 172.21.2.112:0/1577765642 reap_dead start 2021-02-10T11:24:57.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:57.517 7fd15b7fe700 1 --2- 172.21.2.112:0/1577765642 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd148003a60 0x7fd148005f60 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7fd144006650 tx=0x7fd144009a80).ready entity=mon.2 client_cookie=842c9108831b9914 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:57.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:57.518 7fd15a7fc700 1 -- 172.21.2.112:0/1577765642 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd144006340 con 0x7fd148003a60 2021-02-10T11:24:57.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:57.518 7fd15a7fc700 1 -- 172.21.2.112:0/1577765642 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd14400c0c0 con 0x7fd148003a60 2021-02-10T11:24:57.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:57.518 7fd15a7fc700 1 -- 172.21.2.112:0/1577765642 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd14400c5e0 con 0x7fd148003a60 2021-02-10T11:24:57.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:57.518 7fd161eb8700 1 -- 172.21.2.112:0/1577765642 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd148003a60 msgr2=0x7fd148005f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:57.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:57.518 7fd161eb8700 1 --2- 172.21.2.112:0/1577765642 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd148003a60 0x7fd148005f60 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7fd144006650 tx=0x7fd144009a80).stop 2021-02-10T11:24:57.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:57.519 7fd161eb8700 1 -- 172.21.2.112:0/1577765642 shutdown_connections 2021-02-10T11:24:57.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:57.519 7fd161eb8700 1 --2- 172.21.2.112:0/1577765642 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd148003a60 0x7fd148005f60 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:57.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:57.519 7fd161eb8700 1 -- 172.21.2.112:0/1577765642 >> 172.21.2.112:0/1577765642 conn(0x7fd15c0ce3e0 msgr2=0x7fd15c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:57.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:57.519 7fd161eb8700 1 -- 172.21.2.112:0/1577765642 shutdown_connections 2021-02-10T11:24:57.528 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:57.519 7fd161eb8700 1 -- 172.21.2.112:0/1577765642 wait complete. 2021-02-10T11:24:57.528 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:57.533 7fd161eb8700 1 Processor -- start 2021-02-10T11:24:57.528 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:57.533 7fd161eb8700 1 -- start start 2021-02-10T11:24:57.529 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:57.534 7fd161eb8700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd15c0924b0 0x7fd15c1a3590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:57.529 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:57.534 7fd161eb8700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd15c1a3bb0 0x7fd15c0bf920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:57.529 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:57.534 7fd161eb8700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd15c0bff40 0x7fd15c0c0340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:57.529 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:57.534 7fd161eb8700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd15c09a0a0 con 0x7fd15c0924b0 2021-02-10T11:24:57.529 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:57.534 7fd161eb8700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd15c0c0960 con 0x7fd15c0bff40 2021-02-10T11:24:57.529 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:57.534 7fd161eb8700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd15c098420 con 0x7fd15c1a3bb0 2021-02-10T11:24:57.529 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:57.534 7fd15b7fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd15c0924b0 0x7fd15c1a3590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:57.530 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:57.534 7fd15affd700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd15c1a3bb0 0x7fd15c0bf920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:57.530 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:57.534 7fd15b7fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd15c0924b0 0x7fd15c1a3590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:45612/0 (socket says 172.21.2.112:45612) 2021-02-10T11:24:57.530 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:57.534 7fd15b7fe700 1 -- 172.21.2.112:0/1633059964 learned_addr learned my addr 172.21.2.112:0/1633059964 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:24:57.530 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:57.534 7fd15bfff700 1 --2- 172.21.2.112:0/1633059964 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd15c0bff40 0x7fd15c0c0340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:57.530 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:57.535 7fd15affd700 1 -- 172.21.2.112:0/1633059964 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd15c0bff40 msgr2=0x7fd15c0c0340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:57.530 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:57.535 7fd15affd700 1 --2- 172.21.2.112:0/1633059964 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd15c0bff40 0x7fd15c0c0340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:57.530 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:57.535 7fd15affd700 1 -- 172.21.2.112:0/1633059964 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd15c0924b0 msgr2=0x7fd15c1a3590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:57.530 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:57.535 7fd15affd700 1 --2- 172.21.2.112:0/1633059964 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd15c0924b0 0x7fd15c1a3590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:57.530 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:57.535 7fd15affd700 1 -- 172.21.2.112:0/1633059964 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd15c098640 con 0x7fd15c1a3bb0 2021-02-10T11:24:57.530 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:57.535 7fd15affd700 1 --2- 172.21.2.112:0/1633059964 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd15c1a3bb0 0x7fd15c0bf920 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7fd14c001060 tx=0x7fd14c00a8a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:57.530 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:57.535 7fd153fff700 1 -- 172.21.2.112:0/1633059964 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd14c0070d0 con 0x7fd15c1a3bb0 2021-02-10T11:24:57.530 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:57.535 7fd153fff700 1 -- 172.21.2.112:0/1633059964 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd14c001c40 con 0x7fd15c1a3bb0 2021-02-10T11:24:57.530 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:57.535 7fd153fff700 1 -- 172.21.2.112:0/1633059964 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd14c0045b0 con 0x7fd15c1a3bb0 2021-02-10T11:24:57.531 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:57.535 7fd161eb8700 1 -- 172.21.2.112:0/1633059964 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd15c098640 con 0x7fd15c1a3bb0 2021-02-10T11:24:57.531 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:57.535 7fd161eb8700 1 -- 172.21.2.112:0/1633059964 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd15c098640 con 0x7fd15c1a3bb0 2021-02-10T11:24:57.531 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:57.536 7fd153fff700 1 -- 172.21.2.112:0/1633059964 <== mon.2 v2:172.21.2.120:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd14c0043a0 con 0x7fd15c1a3bb0 2021-02-10T11:24:57.531 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:57.536 7fd153fff700 1 --2- 172.21.2.112:0/1633059964 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd1400186c0 0x7fd14001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:57.531 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:57.536 7fd153fff700 1 -- 172.21.2.112:0/1633059964 <== mon.2 v2:172.21.2.120:3301/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fd14c032460 con 0x7fd15c1a3bb0 2021-02-10T11:24:57.531 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:57.536 7fd15b7fe700 1 --2- 172.21.2.112:0/1633059964 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd1400186c0 0x7fd14001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:57.531 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:57.537 7fd15b7fe700 1 --2- 172.21.2.112:0/1633059964 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd1400186c0 0x7fd14001abc0 crc :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:57.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:57.565 7fd161eb8700 1 -- 172.21.2.112:0/1633059964 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd15c105b40 con 0x7fd15c1a3bb0 2021-02-10T11:24:57.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:57.568 7fd153fff700 1 -- 172.21.2.112:0/1633059964 <== mon.2 v2:172.21.2.120:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd14c02dfc0 con 0x7fd15c1a3bb0 2021-02-10T11:24:57.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:57.698 7fd161eb8700 1 -- 172.21.2.112:0/1633059964 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348632", "target": ["mgr", ""]}) v1 -- 0x7fd15c09a060 con 0x7fd1400186c0 2021-02-10T11:24:57.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:57.702 7fd153fff700 1 -- 172.21.2.112:0/1633059964 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7fd1440067a0 con 0x7fd1400186c0 2021-02-10T11:24:57.696 INFO:teuthology.orchestra.run.gibba012.stdout:/volumes/_nogroup/subvolume_0000000000348632/2d750519-10ea-4758-89f8-109f496b1b8e 2021-02-10T11:24:57.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:57.707 7fd161eb8700 1 -- 172.21.2.112:0/1633059964 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd1400186c0 msgr2=0x7fd14001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:57.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:57.707 7fd161eb8700 1 --2- 172.21.2.112:0/1633059964 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd1400186c0 0x7fd14001abc0 crc :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:57.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:57.707 7fd161eb8700 1 -- 172.21.2.112:0/1633059964 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd15c1a3bb0 msgr2=0x7fd15c0bf920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:57.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:57.707 7fd161eb8700 1 --2- 172.21.2.112:0/1633059964 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd15c1a3bb0 0x7fd15c0bf920 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7fd14c001060 tx=0x7fd14c00a8a0).stop 2021-02-10T11:24:57.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:57.707 7fd161eb8700 1 -- 172.21.2.112:0/1633059964 shutdown_connections 2021-02-10T11:24:57.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:57.707 7fd161eb8700 1 --2- 172.21.2.112:0/1633059964 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd1400186c0 0x7fd14001abc0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:57.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:57.707 7fd161eb8700 1 --2- 172.21.2.112:0/1633059964 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd15c0bff40 0x7fd15c0c0340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:57.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:57.707 7fd161eb8700 1 --2- 172.21.2.112:0/1633059964 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd15c1a3bb0 0x7fd15c0bf920 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:57.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:57.707 7fd161eb8700 1 --2- 172.21.2.112:0/1633059964 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd15c0924b0 0x7fd15c1a3590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:57.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:57.707 7fd161eb8700 1 -- 172.21.2.112:0/1633059964 >> 172.21.2.112:0/1633059964 conn(0x7fd15c0b50c0 msgr2=0x7fd15c092d90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:57.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:57.708 7fd161eb8700 1 -- 172.21.2.112:0/1633059964 shutdown_connections 2021-02-10T11:24:57.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:24:57.708 7fd161eb8700 1 -- 172.21.2.112:0/1633059964 wait complete. 2021-02-10T11:24:57.706 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000348632 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000348632/2d750519-10ea-4758-89f8-109f496b1b8e 2021-02-10T11:24:57.706 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:57.706 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:57.706 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:58.041 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:24:58.041 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:24:58.042 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348632/2d750519-10ea-4758-89f8-109f496b1b8e/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:24:58.424 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:24:58.424 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:24:58.424 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0238948 s, 43.9 MB/s 2021-02-10T11:24:58.425 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:58.426 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:58.426 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:58.761 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:24:58.762 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:24:58.762 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348632/2d750519-10ea-4758-89f8-109f496b1b8e/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:24:59.137 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:24:59.137 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:24:59.138 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0256183 s, 40.9 MB/s 2021-02-10T11:24:59.138 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:59.139 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:59.139 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:59.474 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:24:59.474 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:24:59.474 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348632/2d750519-10ea-4758-89f8-109f496b1b8e/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:24:59.847 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:24:59.847 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:24:59.847 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0250031 s, 41.9 MB/s 2021-02-10T11:24:59.848 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:59.848 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:59.849 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:00.183 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:25:00.184 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:00.184 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348632/2d750519-10ea-4758-89f8-109f496b1b8e/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:00.560 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:25:00.560 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:25:00.560 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0244406 s, 42.9 MB/s 2021-02-10T11:25:00.561 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:00.561 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:00.561 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:00.896 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:25:00.896 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:00.897 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348632/2d750519-10ea-4758-89f8-109f496b1b8e/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:01.272 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:25:01.272 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:25:01.272 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0247811 s, 42.3 MB/s 2021-02-10T11:25:01.273 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:01.273 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:01.273 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:01.610 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:25:01.610 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:01.610 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348632/2d750519-10ea-4758-89f8-109f496b1b8e/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:01.984 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:25:01.985 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:25:01.985 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.022848 s, 45.9 MB/s 2021-02-10T11:25:01.986 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:01.986 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:01.986 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:02.321 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:25:02.322 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:02.322 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348632/2d750519-10ea-4758-89f8-109f496b1b8e/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:02.698 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:25:02.699 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:25:02.699 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0249099 s, 42.1 MB/s 2021-02-10T11:25:02.699 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:02.700 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:02.700 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:03.034 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:25:03.035 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:03.035 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348632/2d750519-10ea-4758-89f8-109f496b1b8e/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:03.410 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:25:03.410 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:25:03.410 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.022819 s, 46.0 MB/s 2021-02-10T11:25:03.411 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:03.412 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:03.412 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:03.746 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:25:03.747 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:03.747 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348632/2d750519-10ea-4758-89f8-109f496b1b8e/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:04.122 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:25:04.122 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:25:04.122 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0242499 s, 43.2 MB/s 2021-02-10T11:25:04.123 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:04.123 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:04.123 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:04.460 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:25:04.460 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:04.461 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348632/2d750519-10ea-4758-89f8-109f496b1b8e/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:04.832 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:25:04.832 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:25:04.833 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0242517 s, 43.2 MB/s 2021-02-10T11:25:04.833 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000348633 2021-02-10T11:25:05.267 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.258 7fef52edd700 1 Processor -- start 2021-02-10T11:25:05.267 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.258 7fef52edd700 1 -- start start 2021-02-10T11:25:05.267 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.259 7fef52edd700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fef4c1b0870 0x7fef4c0f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:05.267 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.259 7fef52edd700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fef4c0efe00 0x7fef4c0f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:05.267 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.259 7fef52edd700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef4c1b0c70 con 0x7fef4c1b0470 2021-02-10T11:25:05.268 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.259 7fef52edd700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef4c0f0b50 con 0x7fef4c1b0870 2021-02-10T11:25:05.268 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.259 7fef52edd700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef4c0f0820 con 0x7fef4c0efe00 2021-02-10T11:25:05.268 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.259 7fef50cf9700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fef4c1b0470 0x7fef4c0ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:45082/0 (socket says 172.21.2.112:45082) 2021-02-10T11:25:05.268 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.259 7fef50cf9700 1 -- 172.21.2.112:0/1218377303 learned_addr learned my addr 172.21.2.112:0/1218377303 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:25:05.268 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.259 7fef4b7fe700 1 -- 172.21.2.112:0/1218377303 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 803021200 0 0) 0x7fef34001180 con 0x7fef4c1b0470 2021-02-10T11:25:05.268 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.260 7fef4b7fe700 1 -- 172.21.2.112:0/1218377303 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef38002fe0 con 0x7fef4c1b0470 2021-02-10T11:25:05.268 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.260 7fef4bfff700 1 --2- 172.21.2.112:0/1218377303 >> v2:172.21.2.120:3300/0 conn(0x7fef4c1b0870 0x7fef4c0f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:05.268 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.260 7fef514fa700 1 --2- 172.21.2.112:0/1218377303 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fef4c0efe00 0x7fef4c0f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:05.268 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.260 7fef4b7fe700 1 -- 172.21.2.112:0/1218377303 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1058954204 0 0) 0x7fef34001180 con 0x7fef4c1b0470 2021-02-10T11:25:05.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.260 7fef4b7fe700 1 -- 172.21.2.112:0/1218377303 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fef4c0efe00 msgr2=0x7fef4c0f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:05.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.260 7fef4b7fe700 1 --2- 172.21.2.112:0/1218377303 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fef4c0efe00 0x7fef4c0f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:05.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.260 7fef4b7fe700 1 -- 172.21.2.112:0/1218377303 >> v2:172.21.2.120:3300/0 conn(0x7fef4c1b0870 msgr2=0x7fef4c0f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:05.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.260 7fef4b7fe700 1 --2- 172.21.2.112:0/1218377303 >> v2:172.21.2.120:3300/0 conn(0x7fef4c1b0870 0x7fef4c0f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:05.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.260 7fef4b7fe700 1 -- 172.21.2.112:0/1218377303 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef4c0e52e0 con 0x7fef4c1b0470 2021-02-10T11:25:05.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.260 7fef4b7fe700 1 -- 172.21.2.112:0/1218377303 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7fef34001680 con 0x7fef4c1b0470 2021-02-10T11:25:05.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.260 7fef4b7fe700 1 -- 172.21.2.112:0/1218377303 >> v1:172.21.2.112:6789/0 conn(0x7fef4c1b0470 legacy=0x7fef4c0ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:05.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.260 7fef4b7fe700 1 --2- 172.21.2.112:0/1218377303 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fef38003af0 0x7fef38005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:05.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.260 7fef4b7fe700 1 --2- 172.21.2.112:0/1218377303 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fef3804a850 0x7fef3804cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:05.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.260 7fef4b7fe700 1 --2- 172.21.2.112:0/1218377303 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fef3804d650 0x7fef3804fb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:05.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.260 7fef4b7fe700 1 -- 172.21.2.112:0/1218377303 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef38050390 con 0x7fef38003af0 2021-02-10T11:25:05.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.260 7fef4b7fe700 1 -- 172.21.2.112:0/1218377303 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef380505b0 con 0x7fef3804a850 2021-02-10T11:25:05.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.260 7fef4b7fe700 1 -- 172.21.2.112:0/1218377303 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef380507d0 con 0x7fef3804d650 2021-02-10T11:25:05.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.260 7fef50cf9700 1 --2- 172.21.2.112:0/1218377303 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fef38003af0 0x7fef38005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:05.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.260 7fef4bfff700 1 --2- 172.21.2.112:0/1218377303 >> v2:172.21.2.120:3300/0 conn(0x7fef4c1b0870 0x7fef4c0f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:25:05.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.261 7fef514fa700 1 --2- 172.21.2.112:0/1218377303 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fef3804d650 0x7fef3804fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:05.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.261 7fef50cf9700 1 -- 172.21.2.112:0/1218377303 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fef3804d650 msgr2=0x7fef3804fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:05.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.261 7fef50cf9700 1 --2- 172.21.2.112:0/1218377303 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fef3804d650 0x7fef3804fb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:05.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.261 7fef50cf9700 1 -- 172.21.2.112:0/1218377303 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fef3804a850 msgr2=0x7fef3804cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:05.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.261 7fef4bfff700 1 --2- 172.21.2.112:0/1218377303 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fef3804a850 0x7fef3804cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:05.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.261 7fef50cf9700 1 --2- 172.21.2.112:0/1218377303 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fef3804a850 0x7fef3804cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:05.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.261 7fef50cf9700 1 -- 172.21.2.112:0/1218377303 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef38050a70 con 0x7fef38003af0 2021-02-10T11:25:05.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.261 7fef514fa700 1 --2- 172.21.2.112:0/1218377303 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fef3804d650 0x7fef3804fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:25:05.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.261 7fef4bfff700 1 --2- 172.21.2.112:0/1218377303 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fef3804a850 0x7fef3804cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:25:05.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.261 7fef50cf9700 1 --2- 172.21.2.112:0/1218377303 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fef38003af0 0x7fef38005fa0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 rx=0x7fef34000b60 tx=0x7fef34007da0).ready entity=mon.0 client_cookie=53c6afa0e34f8412 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:05.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.261 7fef4b7fe700 1 -- 172.21.2.112:0/1218377303 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fef340070c0 con 0x7fef38003af0 2021-02-10T11:25:05.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.261 7fef4b7fe700 1 -- 172.21.2.112:0/1218377303 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef340043a0 con 0x7fef38003af0 2021-02-10T11:25:05.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.261 7fef4b7fe700 1 -- 172.21.2.112:0/1218377303 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fef340048c0 con 0x7fef38003af0 2021-02-10T11:25:05.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.261 7fef52edd700 1 -- 172.21.2.112:0/1218377303 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fef38003af0 msgr2=0x7fef38005fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:05.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.261 7fef52edd700 1 --2- 172.21.2.112:0/1218377303 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fef38003af0 0x7fef38005fa0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 rx=0x7fef34000b60 tx=0x7fef34007da0).stop 2021-02-10T11:25:05.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.261 7fef514fa700 1 -- 172.21.2.112:0/1218377303 reap_dead start 2021-02-10T11:25:05.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.262 7fef52edd700 1 -- 172.21.2.112:0/1218377303 shutdown_connections 2021-02-10T11:25:05.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.262 7fef52edd700 1 -- 172.21.2.112:0/1218377303 >> 172.21.2.112:0/1218377303 conn(0x7fef4c0ce3e0 msgr2=0x7fef4c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:05.272 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.262 7fef52edd700 1 -- 172.21.2.112:0/1218377303 shutdown_connections 2021-02-10T11:25:05.272 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.262 7fef52edd700 1 -- 172.21.2.112:0/1218377303 wait complete. 2021-02-10T11:25:05.272 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.276 7fef52edd700 1 Processor -- start 2021-02-10T11:25:05.272 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.276 7fef52edd700 1 -- start start 2021-02-10T11:25:05.273 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.277 7fef52edd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fef4c0b7db0 0x7fef4c0b8260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:05.273 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.277 7fef52edd700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fef4c0b8880 0x7fef4c1a7450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:05.273 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.277 7fef52edd700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fef4c0b8c80 0x7fef4c1a7a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:05.273 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.277 7fef52edd700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef4c0b79b0 con 0x7fef4c0b7db0 2021-02-10T11:25:05.274 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.277 7fef52edd700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef4c1a8090 con 0x7fef4c0b8880 2021-02-10T11:25:05.274 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.277 7fef52edd700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef4c1a82b0 con 0x7fef4c0b8c80 2021-02-10T11:25:05.274 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.277 7fef50cf9700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fef4c0b7db0 0x7fef4c0b8260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:05.274 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.277 7fef50cf9700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fef4c0b7db0 0x7fef4c0b8260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:45630/0 (socket says 172.21.2.112:45630) 2021-02-10T11:25:05.274 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.277 7fef50cf9700 1 -- 172.21.2.112:0/3887813014 learned_addr learned my addr 172.21.2.112:0/3887813014 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:25:05.274 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.277 7fef50cf9700 1 -- 172.21.2.112:0/3887813014 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fef4c0b8c80 msgr2=0x7fef4c1a7a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:05.274 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.277 7fef50cf9700 1 --2- 172.21.2.112:0/3887813014 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fef4c0b8c80 0x7fef4c1a7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:05.274 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.277 7fef50cf9700 1 -- 172.21.2.112:0/3887813014 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fef4c0b8880 msgr2=0x7fef4c1a7450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:05.274 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.277 7fef50cf9700 1 --2- 172.21.2.112:0/3887813014 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fef4c0b8880 0x7fef4c1a7450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:05.274 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.277 7fef50cf9700 1 -- 172.21.2.112:0/3887813014 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef4c190c10 con 0x7fef4c0b7db0 2021-02-10T11:25:05.274 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.277 7fef50cf9700 1 --2- 172.21.2.112:0/3887813014 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fef4c0b7db0 0x7fef4c0b8260 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 rx=0x7fef3400a590 tx=0x7fef3400a930).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:05.275 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.278 7fef497fa700 1 -- 172.21.2.112:0/3887813014 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fef340013b0 con 0x7fef4c0b7db0 2021-02-10T11:25:05.275 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.278 7fef497fa700 1 -- 172.21.2.112:0/3887813014 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef34007280 con 0x7fef4c0b7db0 2021-02-10T11:25:05.275 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.278 7fef52edd700 1 -- 172.21.2.112:0/3887813014 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef4c190c10 con 0x7fef4c0b7db0 2021-02-10T11:25:05.275 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.278 7fef497fa700 1 -- 172.21.2.112:0/3887813014 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fef34004620 con 0x7fef4c0b7db0 2021-02-10T11:25:05.275 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.278 7fef52edd700 1 -- 172.21.2.112:0/3887813014 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef4c0c1eb0 con 0x7fef4c0b7db0 2021-02-10T11:25:05.275 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.278 7fef497fa700 1 -- 172.21.2.112:0/3887813014 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fef34001390 con 0x7fef4c0b7db0 2021-02-10T11:25:05.275 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.278 7fef497fa700 1 --2- 172.21.2.112:0/3887813014 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fef300186c0 0x7fef3001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:05.275 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.278 7fef497fa700 1 -- 172.21.2.112:0/3887813014 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fef340323e0 con 0x7fef4c0b7db0 2021-02-10T11:25:05.275 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.279 7fef4bfff700 1 --2- 172.21.2.112:0/3887813014 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fef300186c0 0x7fef3001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:05.275 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.279 7fef4bfff700 1 --2- 172.21.2.112:0/3887813014 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fef300186c0 0x7fef3001abc0 crc :-1 s=READY pgs=305 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:05.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.309 7fef52edd700 1 -- 172.21.2.112:0/3887813014 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef4c105b30 con 0x7fef4c0b7db0 2021-02-10T11:25:05.302 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.311 7fef497fa700 1 -- 172.21.2.112:0/3887813014 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fef34030d10 con 0x7fef4c0b7db0 2021-02-10T11:25:05.427 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.436 7fef52edd700 1 -- 172.21.2.112:0/3887813014 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348633", "target": ["mgr", ""]}) v1 -- 0x7fef4c0c1eb0 con 0x7fef300186c0 2021-02-10T11:25:05.452 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.462 7fef497fa700 1 -- 172.21.2.112:0/3887813014 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fef3c006760 con 0x7fef300186c0 2021-02-10T11:25:05.460 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.470 7fef52edd700 1 -- 172.21.2.112:0/3887813014 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fef300186c0 msgr2=0x7fef3001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:05.460 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.470 7fef52edd700 1 --2- 172.21.2.112:0/3887813014 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fef300186c0 0x7fef3001abc0 crc :-1 s=READY pgs=305 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:05.460 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.470 7fef52edd700 1 -- 172.21.2.112:0/3887813014 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fef4c0b7db0 msgr2=0x7fef4c0b8260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:05.460 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.470 7fef52edd700 1 --2- 172.21.2.112:0/3887813014 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fef4c0b7db0 0x7fef4c0b8260 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 rx=0x7fef3400a590 tx=0x7fef3400a930).stop 2021-02-10T11:25:05.461 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.470 7fef52edd700 1 -- 172.21.2.112:0/3887813014 shutdown_connections 2021-02-10T11:25:05.461 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.470 7fef52edd700 1 --2- 172.21.2.112:0/3887813014 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fef300186c0 0x7fef3001abc0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:05.461 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.470 7fef52edd700 1 --2- 172.21.2.112:0/3887813014 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fef4c0b8880 0x7fef4c1a7450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:05.461 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.470 7fef52edd700 1 --2- 172.21.2.112:0/3887813014 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fef4c0b8c80 0x7fef4c1a7a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:05.461 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.470 7fef52edd700 1 --2- 172.21.2.112:0/3887813014 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fef4c0b7db0 0x7fef4c0b8260 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:05.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.470 7fef52edd700 1 -- 172.21.2.112:0/3887813014 >> 172.21.2.112:0/3887813014 conn(0x7fef4c094e10 msgr2=0x7fef4c0a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:05.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.470 7fef52edd700 1 -- 172.21.2.112:0/3887813014 shutdown_connections 2021-02-10T11:25:05.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.470 7fef52edd700 1 -- 172.21.2.112:0/3887813014 wait complete. 2021-02-10T11:25:05.469 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000348633 2021-02-10T11:25:05.901 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.894 7fbbf5fba700 1 Processor -- start 2021-02-10T11:25:05.901 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.894 7fbbf5fba700 1 -- start start 2021-02-10T11:25:05.901 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.895 7fbbf5fba700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fbbf01b0470 0x7fbbf01b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:05.901 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.895 7fbbf5fba700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fbbf00f20e0 0x7fbbf00f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:05.901 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.895 7fbbf5fba700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbf00f0b50 con 0x7fbbf00ef610 2021-02-10T11:25:05.901 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.895 7fbbf5fba700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbbf00f0820 con 0x7fbbf00f20e0 2021-02-10T11:25:05.901 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.895 7fbbf5fba700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbbf00f24e0 con 0x7fbbf01b0470 2021-02-10T11:25:05.902 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.895 7fbbeffff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fbbf00f20e0 0x7fbbf00f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:05.902 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.895 7fbbeffff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fbbf00f20e0 0x7fbbf00f0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:45638/0 (socket says 172.21.2.112:45638) 2021-02-10T11:25:05.902 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.895 7fbbeffff700 1 -- 172.21.2.112:0/2259665326 learned_addr learned my addr 172.21.2.112:0/2259665326 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:25:05.902 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.895 7fbbeffff700 1 -- 172.21.2.112:0/2259665326 >> v2:172.21.2.120:3300/0 conn(0x7fbbf01b0470 msgr2=0x7fbbf01b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:05.902 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.895 7fbbeffff700 1 --2- 172.21.2.112:0/2259665326 >> v2:172.21.2.120:3300/0 conn(0x7fbbf01b0470 0x7fbbf01b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:05.902 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.895 7fbbeffff700 1 -- 172.21.2.112:0/2259665326 >> v1:172.21.2.120:6789/0 conn(0x7fbbf00ef610 legacy=0x7fbbf00efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:05.902 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.895 7fbbeffff700 1 -- 172.21.2.112:0/2259665326 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbf00e52e0 con 0x7fbbf00f20e0 2021-02-10T11:25:05.902 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.895 7fbbeffff700 1 --2- 172.21.2.112:0/2259665326 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbbf00f20e0 0x7fbbf00f0200 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 rx=0x7fbbe8006f00 tx=0x7fbbe8009f10).ready entity=mon.0 client_cookie=8ab6e213ae90315 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:05.902 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.896 7fbbee7fc700 1 -- 172.21.2.112:0/2259665326 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbbe8003030 con 0x7fbbf00f20e0 2021-02-10T11:25:05.902 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.896 7fbbee7fc700 1 -- 172.21.2.112:0/2259665326 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbe800b3b0 con 0x7fbbf00f20e0 2021-02-10T11:25:05.902 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.896 7fbbee7fc700 1 -- 172.21.2.112:0/2259665326 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbbe800b890 con 0x7fbbf00f20e0 2021-02-10T11:25:05.902 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.896 7fbbf5fba700 1 -- 172.21.2.112:0/2259665326 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbbf00f20e0 msgr2=0x7fbbf00f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:05.903 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.896 7fbbf5fba700 1 --2- 172.21.2.112:0/2259665326 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbbf00f20e0 0x7fbbf00f0200 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 rx=0x7fbbe8006f00 tx=0x7fbbe8009f10).stop 2021-02-10T11:25:05.903 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.896 7fbbf5fba700 1 -- 172.21.2.112:0/2259665326 shutdown_connections 2021-02-10T11:25:05.903 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.896 7fbbf5fba700 1 --2- 172.21.2.112:0/2259665326 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbbf00f20e0 0x7fbbf00f0200 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:05.903 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.896 7fbbf5fba700 1 --2- 172.21.2.112:0/2259665326 >> v2:172.21.2.120:3300/0 conn(0x7fbbf01b0470 0x7fbbf01b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:05.903 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.896 7fbbf5fba700 1 -- 172.21.2.112:0/2259665326 >> 172.21.2.112:0/2259665326 conn(0x7fbbf00ce3e0 msgr2=0x7fbbf00e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:05.903 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.896 7fbbf5fba700 1 -- 172.21.2.112:0/2259665326 shutdown_connections 2021-02-10T11:25:05.903 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.896 7fbbf5fba700 1 -- 172.21.2.112:0/2259665326 wait complete. 2021-02-10T11:25:05.905 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.911 7fbbf5fba700 1 Processor -- start 2021-02-10T11:25:05.905 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.911 7fbbf5fba700 1 -- start start 2021-02-10T11:25:05.905 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.911 7fbbf5fba700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fbbf00afa70 0x7fbbf00afe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:05.905 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.911 7fbbf5fba700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbbf00b0490 0x7fbbf00a78f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:05.905 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.911 7fbbf5fba700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbbf00a7f10 0x7fbbf00a8310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:05.905 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.911 7fbbf5fba700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbf00a8a40 con 0x7fbbf00a7f10 2021-02-10T11:25:05.905 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.911 7fbbf5fba700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbf00a8fc0 con 0x7fbbf00afa70 2021-02-10T11:25:05.905 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.911 7fbbf5fba700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbf00a91a0 con 0x7fbbf00b0490 2021-02-10T11:25:05.906 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.911 7fbbeffff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbbf00a7f10 0x7fbbf00a8310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:05.906 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.911 7fbbeffff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbbf00a7f10 0x7fbbf00a8310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:45646/0 (socket says 172.21.2.112:45646) 2021-02-10T11:25:05.906 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.911 7fbbeffff700 1 -- 172.21.2.112:0/3088852064 learned_addr learned my addr 172.21.2.112:0/3088852064 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:25:05.906 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.911 7fbbef7fe700 1 --2- 172.21.2.112:0/3088852064 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fbbf00afa70 0x7fbbf00afe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:05.906 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.912 7fbbeeffd700 1 --2- 172.21.2.112:0/3088852064 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbbf00b0490 0x7fbbf00a78f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:05.906 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.912 7fbbeffff700 1 -- 172.21.2.112:0/3088852064 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbbf00b0490 msgr2=0x7fbbf00a78f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:05.906 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.912 7fbbeffff700 1 --2- 172.21.2.112:0/3088852064 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbbf00b0490 0x7fbbf00a78f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:05.906 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.912 7fbbeffff700 1 -- 172.21.2.112:0/3088852064 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fbbf00afa70 msgr2=0x7fbbf00afe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:05.906 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.912 7fbbeffff700 1 --2- 172.21.2.112:0/3088852064 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fbbf00afa70 0x7fbbf00afe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:05.906 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.912 7fbbeffff700 1 -- 172.21.2.112:0/3088852064 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbf00a93c0 con 0x7fbbf00a7f10 2021-02-10T11:25:05.906 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.912 7fbbeeffd700 1 --2- 172.21.2.112:0/3088852064 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbbf00b0490 0x7fbbf00a78f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:25:05.906 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.912 7fbbeffff700 1 --2- 172.21.2.112:0/3088852064 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbbf00a7f10 0x7fbbf00a8310 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 rx=0x7fbbe800b940 tx=0x7fbbe800bce0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:05.907 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.912 7fbbe7fff700 1 -- 172.21.2.112:0/3088852064 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbbe8009450 con 0x7fbbf00a7f10 2021-02-10T11:25:05.907 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.912 7fbbe7fff700 1 -- 172.21.2.112:0/3088852064 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbe8003030 con 0x7fbbf00a7f10 2021-02-10T11:25:05.907 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.912 7fbbf5fba700 1 -- 172.21.2.112:0/3088852064 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbf00a93c0 con 0x7fbbf00a7f10 2021-02-10T11:25:05.907 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.912 7fbbf5fba700 1 -- 172.21.2.112:0/3088852064 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbf00a9950 con 0x7fbbf00a7f10 2021-02-10T11:25:05.907 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.912 7fbbe7fff700 1 -- 172.21.2.112:0/3088852064 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbbe800c4e0 con 0x7fbbf00a7f10 2021-02-10T11:25:05.907 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.913 7fbbe7fff700 1 -- 172.21.2.112:0/3088852064 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fbbe8005820 con 0x7fbbf00a7f10 2021-02-10T11:25:05.907 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.913 7fbbe7fff700 1 --2- 172.21.2.112:0/3088852064 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fbbd40186c0 0x7fbbd401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:05.907 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.913 7fbbe7fff700 1 -- 172.21.2.112:0/3088852064 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fbbe80328c0 con 0x7fbbf00a7f10 2021-02-10T11:25:05.907 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.913 7fbbef7fe700 1 --2- 172.21.2.112:0/3088852064 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fbbd40186c0 0x7fbbd401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:05.907 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.914 7fbbef7fe700 1 --2- 172.21.2.112:0/3088852064 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fbbd40186c0 0x7fbbd401abc0 crc :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:05.935 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.944 7fbbf5fba700 1 -- 172.21.2.112:0/3088852064 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbf00e52e0 con 0x7fbbf00a7f10 2021-02-10T11:25:05.938 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:05.947 7fbbe7fff700 1 -- 172.21.2.112:0/3088852064 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fbbe8009450 con 0x7fbbf00a7f10 2021-02-10T11:25:06.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:06.075 7fbbf5fba700 1 -- 172.21.2.112:0/3088852064 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348633", "target": ["mgr", ""]}) v1 -- 0x7fbbf00a8a40 con 0x7fbbd40186c0 2021-02-10T11:25:06.069 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:06.080 7fbbe7fff700 1 -- 172.21.2.112:0/3088852064 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7fbbe0006550 con 0x7fbbd40186c0 2021-02-10T11:25:06.074 INFO:teuthology.orchestra.run.gibba012.stdout:/volumes/_nogroup/subvolume_0000000000348633/18449c58-f58a-4b11-a13f-d2d2707ed376 2021-02-10T11:25:06.076 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:06.085 7fbbf5fba700 1 -- 172.21.2.112:0/3088852064 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fbbd40186c0 msgr2=0x7fbbd401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:06.076 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:06.085 7fbbf5fba700 1 --2- 172.21.2.112:0/3088852064 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fbbd40186c0 0x7fbbd401abc0 crc :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:06.076 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:06.085 7fbbf5fba700 1 -- 172.21.2.112:0/3088852064 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbbf00a7f10 msgr2=0x7fbbf00a8310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:06.076 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:06.085 7fbbf5fba700 1 --2- 172.21.2.112:0/3088852064 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbbf00a7f10 0x7fbbf00a8310 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 rx=0x7fbbe800b940 tx=0x7fbbe800bce0).stop 2021-02-10T11:25:06.076 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:06.085 7fbbf5fba700 1 -- 172.21.2.112:0/3088852064 shutdown_connections 2021-02-10T11:25:06.076 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:06.085 7fbbf5fba700 1 --2- 172.21.2.112:0/3088852064 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fbbd40186c0 0x7fbbd401abc0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:06.077 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:06.085 7fbbf5fba700 1 --2- 172.21.2.112:0/3088852064 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbbf00a7f10 0x7fbbf00a8310 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:06.077 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:06.085 7fbbf5fba700 1 --2- 172.21.2.112:0/3088852064 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbbf00b0490 0x7fbbf00a78f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:06.077 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:06.085 7fbbf5fba700 1 --2- 172.21.2.112:0/3088852064 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fbbf00afa70 0x7fbbf00afe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:06.077 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:06.085 7fbbf5fba700 1 -- 172.21.2.112:0/3088852064 >> 172.21.2.112:0/3088852064 conn(0x7fbbf00ac550 msgr2=0x7fbbf00acb70 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:06.078 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:06.086 7fbbf5fba700 1 -- 172.21.2.112:0/3088852064 shutdown_connections 2021-02-10T11:25:06.078 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:06.086 7fbbf5fba700 1 -- 172.21.2.112:0/3088852064 wait complete. 2021-02-10T11:25:06.084 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000348633 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000348633/18449c58-f58a-4b11-a13f-d2d2707ed376 2021-02-10T11:25:06.085 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:06.085 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:06.085 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:06.422 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:25:06.422 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:06.423 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348633/18449c58-f58a-4b11-a13f-d2d2707ed376/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:06.805 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:25:06.805 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:25:06.806 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0244109 s, 43.0 MB/s 2021-02-10T11:25:06.807 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:06.807 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:06.807 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:07.142 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:25:07.143 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:07.143 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348633/18449c58-f58a-4b11-a13f-d2d2707ed376/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:07.519 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:25:07.520 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:25:07.520 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0239875 s, 43.7 MB/s 2021-02-10T11:25:07.521 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:07.521 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:07.521 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:07.858 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:25:07.858 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:07.859 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348633/18449c58-f58a-4b11-a13f-d2d2707ed376/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:08.236 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:25:08.236 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:25:08.236 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0254126 s, 41.3 MB/s 2021-02-10T11:25:08.238 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:08.238 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:08.238 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:08.574 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:25:08.575 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:08.575 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348633/18449c58-f58a-4b11-a13f-d2d2707ed376/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:08.953 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:25:08.953 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:25:08.953 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.025675 s, 40.8 MB/s 2021-02-10T11:25:08.954 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:08.954 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:08.954 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:09.289 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:25:09.289 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:09.290 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348633/18449c58-f58a-4b11-a13f-d2d2707ed376/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:09.665 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:25:09.665 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:25:09.666 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0257939 s, 40.7 MB/s 2021-02-10T11:25:09.666 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:09.667 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:09.667 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:09.703 DEBUG:teuthology.orchestra.run.gibba012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:25:09.711 DEBUG:teuthology.orchestra.run.gibba020:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:25:10.001 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:25:10.001 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:10.002 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348633/18449c58-f58a-4b11-a13f-d2d2707ed376/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:10.376 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:25:10.376 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:25:10.376 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0227647 s, 46.1 MB/s 2021-02-10T11:25:10.377 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:10.377 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:10.377 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:10.712 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:25:10.712 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:10.713 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348633/18449c58-f58a-4b11-a13f-d2d2707ed376/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:11.086 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:25:11.086 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:25:11.086 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0215469 s, 48.7 MB/s 2021-02-10T11:25:11.087 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:11.087 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:11.088 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:11.423 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:25:11.424 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:11.424 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348633/18449c58-f58a-4b11-a13f-d2d2707ed376/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:11.800 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:25:11.801 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:25:11.801 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0246674 s, 42.5 MB/s 2021-02-10T11:25:11.802 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:11.802 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:11.802 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:12.136 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:25:12.136 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:12.136 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348633/18449c58-f58a-4b11-a13f-d2d2707ed376/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:12.510 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:25:12.510 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:25:12.510 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0228563 s, 45.9 MB/s 2021-02-10T11:25:12.511 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:12.511 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:12.511 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:12.846 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:25:12.847 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:12.847 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348633/18449c58-f58a-4b11-a13f-d2d2707ed376/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:13.226 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:25:13.226 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:25:13.226 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0259494 s, 40.4 MB/s 2021-02-10T11:25:13.227 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000348634 2021-02-10T11:25:13.658 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:13.651 7fb400478700 1 Processor -- start 2021-02-10T11:25:13.658 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:13.651 7fb400478700 1 -- start start 2021-02-10T11:25:13.658 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:13.651 7fb400478700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fb3f81b0870 0x7fb3f80f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:13.658 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:13.651 7fb400478700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb3f80efe00 0x7fb3f80f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:13.658 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:13.651 7fb400478700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3f81b0c70 con 0x7fb3f81b0470 2021-02-10T11:25:13.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:13.651 7fb400478700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3f80f0b50 con 0x7fb3f81b0870 2021-02-10T11:25:13.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:13.651 7fb400478700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3f80d2fb0 con 0x7fb3f80efe00 2021-02-10T11:25:13.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:13.652 7fb3fe294700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7fb3f81b0470 0x7fb3f80ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:38670/0 (socket says 172.21.2.112:38670) 2021-02-10T11:25:13.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:13.652 7fb3fe294700 1 -- 172.21.2.112:0/3007606058 learned_addr learned my addr 172.21.2.112:0/3007606058 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:25:13.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:13.652 7fb3fda93700 1 --2- 172.21.2.112:0/3007606058 >> v2:172.21.2.120:3300/0 conn(0x7fb3f81b0870 0x7fb3f80f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:13.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:13.652 7fb3fea95700 1 --2- 172.21.2.112:0/3007606058 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb3f80efe00 0x7fb3f80f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:13.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:13.652 7fb3fea95700 1 -- 172.21.2.112:0/3007606058 >> v2:172.21.2.120:3300/0 conn(0x7fb3f81b0870 msgr2=0x7fb3f80f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:13.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:13.652 7fb3fd292700 1 -- 172.21.2.112:0/3007606058 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2133939327 0 0) 0x7fb3f0000fe0 con 0x7fb3f81b0470 2021-02-10T11:25:13.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:13.652 7fb3fea95700 1 --2- 172.21.2.112:0/3007606058 >> v2:172.21.2.120:3300/0 conn(0x7fb3f81b0870 0x7fb3f80f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:13.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:13.652 7fb3fea95700 1 -- 172.21.2.112:0/3007606058 >> v1:172.21.2.120:6789/0 conn(0x7fb3f81b0470 legacy=0x7fb3f80ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:13.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:13.652 7fb3fea95700 1 -- 172.21.2.112:0/3007606058 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3f80f0820 con 0x7fb3f80efe00 2021-02-10T11:25:13.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:13.652 7fb3fea95700 1 --2- 172.21.2.112:0/3007606058 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb3f80efe00 0x7fb3f80f0200 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7fb3f4006ad0 tx=0x7fb3f4009ad0).ready entity=mon.2 client_cookie=2624a1e6c5706a8d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:13.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:13.652 7fb3fd292700 1 -- 172.21.2.112:0/3007606058 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb3f40022e0 con 0x7fb3f80efe00 2021-02-10T11:25:13.660 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:13.652 7fb3fd292700 1 -- 172.21.2.112:0/3007606058 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3f4006410 con 0x7fb3f80efe00 2021-02-10T11:25:13.660 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:13.652 7fb3fd292700 1 -- 172.21.2.112:0/3007606058 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb3f4002620 con 0x7fb3f80efe00 2021-02-10T11:25:13.660 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:13.653 7fb400478700 1 -- 172.21.2.112:0/3007606058 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb3f80efe00 msgr2=0x7fb3f80f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:13.660 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:13.653 7fb400478700 1 --2- 172.21.2.112:0/3007606058 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb3f80efe00 0x7fb3f80f0200 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7fb3f4006ad0 tx=0x7fb3f4009ad0).stop 2021-02-10T11:25:13.660 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:13.653 7fb400478700 1 -- 172.21.2.112:0/3007606058 shutdown_connections 2021-02-10T11:25:13.660 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:13.653 7fb400478700 1 --2- 172.21.2.112:0/3007606058 >> v2:172.21.2.120:3300/0 conn(0x7fb3f81b0870 0x7fb3f80f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:13.660 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:13.653 7fb400478700 1 --2- 172.21.2.112:0/3007606058 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb3f80efe00 0x7fb3f80f0200 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:13.660 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:13.653 7fb400478700 1 -- 172.21.2.112:0/3007606058 >> 172.21.2.112:0/3007606058 conn(0x7fb3f80ce3e0 msgr2=0x7fb3f80e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:13.660 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:13.653 7fb400478700 1 -- 172.21.2.112:0/3007606058 shutdown_connections 2021-02-10T11:25:13.660 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:13.653 7fb400478700 1 -- 172.21.2.112:0/3007606058 wait complete. 2021-02-10T11:25:13.662 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:13.668 7fb400478700 1 Processor -- start 2021-02-10T11:25:13.662 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:13.668 7fb400478700 1 -- start start 2021-02-10T11:25:13.662 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:13.668 7fb400478700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb3f8190920 0x7fb3f80b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:13.663 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:13.668 7fb400478700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb3f80b8200 0x7fb3f80b8600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:13.663 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:13.668 7fb400478700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb3f80c1d80 0x7fb3f80c2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:13.663 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:13.668 7fb400478700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3f8190d20 con 0x7fb3f80c1d80 2021-02-10T11:25:13.663 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:13.668 7fb400478700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3f80b8c20 con 0x7fb3f8190920 2021-02-10T11:25:13.663 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:13.668 7fb400478700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3f80c27a0 con 0x7fb3f80b8200 2021-02-10T11:25:13.663 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:13.669 7fb3fda93700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb3f80b8200 0x7fb3f80b8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:13.663 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:13.669 7fb3fda93700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb3f80b8200 0x7fb3f80b8600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3301/0 says I am v2:172.21.2.112:45338/0 (socket says 172.21.2.112:45338) 2021-02-10T11:25:13.663 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:13.669 7fb3fda93700 1 -- 172.21.2.112:0/4251186872 learned_addr learned my addr 172.21.2.112:0/4251186872 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:25:13.663 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:13.669 7fb3fea95700 1 --2- 172.21.2.112:0/4251186872 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb3f80c1d80 0x7fb3f80c2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:13.663 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:13.669 7fb3fe294700 1 --2- 172.21.2.112:0/4251186872 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb3f8190920 0x7fb3f80b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:13.663 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:13.669 7fb3fda93700 1 -- 172.21.2.112:0/4251186872 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb3f8190920 msgr2=0x7fb3f80b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:13.663 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:13.669 7fb3fda93700 1 --2- 172.21.2.112:0/4251186872 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb3f8190920 0x7fb3f80b7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:13.663 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:13.669 7fb3fda93700 1 -- 172.21.2.112:0/4251186872 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb3f80c1d80 msgr2=0x7fb3f80c2180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:13.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:13.669 7fb3fda93700 1 --2- 172.21.2.112:0/4251186872 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb3f80c1d80 0x7fb3f80c2180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:13.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:13.669 7fb3fda93700 1 -- 172.21.2.112:0/4251186872 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3f80c29c0 con 0x7fb3f80b8200 2021-02-10T11:25:13.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:13.669 7fb3fda93700 1 --2- 172.21.2.112:0/4251186872 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb3f80b8200 0x7fb3f80b8600 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7fb3e8006f60 tx=0x7fb3e800b970).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:13.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:13.669 7fb3eeffd700 1 -- 172.21.2.112:0/4251186872 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb3e800bd60 con 0x7fb3f80b8200 2021-02-10T11:25:13.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:13.669 7fb3eeffd700 1 -- 172.21.2.112:0/4251186872 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3e8000c90 con 0x7fb3f80b8200 2021-02-10T11:25:13.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:13.669 7fb400478700 1 -- 172.21.2.112:0/4251186872 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3f80c29c0 con 0x7fb3f80b8200 2021-02-10T11:25:13.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:13.669 7fb400478700 1 -- 172.21.2.112:0/4251186872 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3f80c2f50 con 0x7fb3f80b8200 2021-02-10T11:25:13.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:13.669 7fb3eeffd700 1 -- 172.21.2.112:0/4251186872 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb3e8002220 con 0x7fb3f80b8200 2021-02-10T11:25:13.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:13.670 7fb3eeffd700 1 -- 172.21.2.112:0/4251186872 <== mon.2 v2:172.21.2.120:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb3e8006b00 con 0x7fb3f80b8200 2021-02-10T11:25:13.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:13.670 7fb3eeffd700 1 --2- 172.21.2.112:0/4251186872 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb3e0018730 0x7fb3e001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:13.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:13.670 7fb3eeffd700 1 -- 172.21.2.112:0/4251186872 <== mon.2 v2:172.21.2.120:3301/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fb3e8032790 con 0x7fb3f80b8200 2021-02-10T11:25:13.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:13.670 7fb3fe294700 1 --2- 172.21.2.112:0/4251186872 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb3e0018730 0x7fb3e001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:13.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:13.671 7fb3fe294700 1 --2- 172.21.2.112:0/4251186872 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb3e0018730 0x7fb3e001ac10 crc :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:13.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:13.701 7fb400478700 1 -- 172.21.2.112:0/4251186872 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3f8105ae0 con 0x7fb3f80b8200 2021-02-10T11:25:13.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:13.703 7fb3eeffd700 1 -- 172.21.2.112:0/4251186872 <== mon.2 v2:172.21.2.120:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb3e8000960 con 0x7fb3f80b8200 2021-02-10T11:25:13.819 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:13.829 7fb400478700 1 -- 172.21.2.112:0/4251186872 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348634", "target": ["mgr", ""]}) v1 -- 0x7fb3f80c2f50 con 0x7fb3e0018730 2021-02-10T11:25:13.843 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:13.854 7fb3eeffd700 1 -- 172.21.2.112:0/4251186872 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fb3f0000fe0 con 0x7fb3e0018730 2021-02-10T11:25:13.852 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:13.861 7fb400478700 1 -- 172.21.2.112:0/4251186872 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb3e0018730 msgr2=0x7fb3e001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:13.852 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:13.861 7fb400478700 1 --2- 172.21.2.112:0/4251186872 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb3e0018730 0x7fb3e001ac10 crc :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:13.852 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:13.861 7fb400478700 1 -- 172.21.2.112:0/4251186872 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb3f80b8200 msgr2=0x7fb3f80b8600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:13.852 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:13.861 7fb400478700 1 --2- 172.21.2.112:0/4251186872 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb3f80b8200 0x7fb3f80b8600 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7fb3e8006f60 tx=0x7fb3e800b970).stop 2021-02-10T11:25:13.852 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:13.862 7fb400478700 1 -- 172.21.2.112:0/4251186872 shutdown_connections 2021-02-10T11:25:13.853 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:13.862 7fb400478700 1 --2- 172.21.2.112:0/4251186872 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb3e0018730 0x7fb3e001ac10 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:13.853 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:13.862 7fb400478700 1 --2- 172.21.2.112:0/4251186872 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb3f80c1d80 0x7fb3f80c2180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:13.853 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:13.862 7fb400478700 1 --2- 172.21.2.112:0/4251186872 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb3f80b8200 0x7fb3f80b8600 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:13.853 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:13.862 7fb400478700 1 --2- 172.21.2.112:0/4251186872 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb3f8190920 0x7fb3f80b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:13.853 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:13.862 7fb400478700 1 -- 172.21.2.112:0/4251186872 >> 172.21.2.112:0/4251186872 conn(0x7fb3f8094e10 msgr2=0x7fb3f80a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:13.853 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:13.862 7fb400478700 1 -- 172.21.2.112:0/4251186872 shutdown_connections 2021-02-10T11:25:13.853 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:13.862 7fb400478700 1 -- 172.21.2.112:0/4251186872 wait complete. 2021-02-10T11:25:13.860 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000348634 2021-02-10T11:25:14.289 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:14.283 7faee0ebc700 1 Processor -- start 2021-02-10T11:25:14.290 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:14.283 7faee0ebc700 1 -- start start 2021-02-10T11:25:14.290 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:14.284 7faee0ebc700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7faedc1b0470 0x7faedc1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:14.290 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:14.284 7faee0ebc700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7faedc0ef610 0x7faedc0efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:14.290 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:14.284 7faee0ebc700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7faedc0f0030 0x7faedc0f0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:14.290 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:14.284 7faee0ebc700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faedc0e5ea0 con 0x7faedc1b0470 2021-02-10T11:25:14.290 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:14.284 7faee0ebc700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faedc0f0b50 con 0x7faedc0ef610 2021-02-10T11:25:14.290 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:14.284 7faee0ebc700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faedc0e52e0 con 0x7faedc0f0030 2021-02-10T11:25:14.290 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:14.284 7faedab13700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7faedc1b0470 0x7faedc1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:14.290 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:14.284 7faedab13700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7faedc1b0470 0x7faedc1b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:45664/0 (socket says 172.21.2.112:45664) 2021-02-10T11:25:14.291 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:14.284 7faedab13700 1 -- 172.21.2.112:0/10656559 learned_addr learned my addr 172.21.2.112:0/10656559 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:25:14.291 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:14.284 7faeda312700 1 --2- 172.21.2.112:0/10656559 >> v2:172.21.2.120:3300/0 conn(0x7faedc0ef610 0x7faedc0efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:14.291 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:14.284 7faedab13700 1 -- 172.21.2.112:0/10656559 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7faedc0f0030 msgr2=0x7faedc0f0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:14.291 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:14.284 7faedab13700 1 --2- 172.21.2.112:0/10656559 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7faedc0f0030 0x7faedc0f0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:14.291 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:14.284 7faedab13700 1 -- 172.21.2.112:0/10656559 >> v2:172.21.2.120:3300/0 conn(0x7faedc0ef610 msgr2=0x7faedc0efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:14.291 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:14.284 7faedab13700 1 --2- 172.21.2.112:0/10656559 >> v2:172.21.2.120:3300/0 conn(0x7faedc0ef610 0x7faedc0efa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:14.291 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:14.284 7faedab13700 1 -- 172.21.2.112:0/10656559 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faedc0f10d0 con 0x7faedc1b0470 2021-02-10T11:25:14.291 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:14.284 7faedab13700 1 --2- 172.21.2.112:0/10656559 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faedc1b0470 0x7faedc1b0870 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 rx=0x7faecc006700 tx=0x7faecc009b00).ready entity=mon.0 client_cookie=4b2935c4d06e6bfe server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:14.291 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:14.284 7faed9b11700 1 -- 172.21.2.112:0/10656559 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faecc00c0c0 con 0x7faedc1b0470 2021-02-10T11:25:14.291 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:14.284 7faed9b11700 1 -- 172.21.2.112:0/10656559 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faecc00c400 con 0x7faedc1b0470 2021-02-10T11:25:14.291 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:14.284 7faed9b11700 1 -- 172.21.2.112:0/10656559 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faecc00c8e0 con 0x7faedc1b0470 2021-02-10T11:25:14.291 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:14.284 7faee0ebc700 1 -- 172.21.2.112:0/10656559 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faedc1b0470 msgr2=0x7faedc1b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:14.291 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:14.284 7faee0ebc700 1 --2- 172.21.2.112:0/10656559 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faedc1b0470 0x7faedc1b0870 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 rx=0x7faecc006700 tx=0x7faecc009b00).stop 2021-02-10T11:25:14.292 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:14.285 7faee0ebc700 1 -- 172.21.2.112:0/10656559 shutdown_connections 2021-02-10T11:25:14.292 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:14.285 7faee0ebc700 1 --2- 172.21.2.112:0/10656559 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7faedc0f0030 0x7faedc0f0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:14.292 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:14.285 7faee0ebc700 1 --2- 172.21.2.112:0/10656559 >> v2:172.21.2.120:3300/0 conn(0x7faedc0ef610 0x7faedc0efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:14.292 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:14.285 7faee0ebc700 1 --2- 172.21.2.112:0/10656559 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faedc1b0470 0x7faedc1b0870 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:14.292 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:14.285 7faee0ebc700 1 -- 172.21.2.112:0/10656559 >> 172.21.2.112:0/10656559 conn(0x7faedc0ce3e0 msgr2=0x7faedc0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:14.292 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:14.285 7faee0ebc700 1 -- 172.21.2.112:0/10656559 shutdown_connections 2021-02-10T11:25:14.292 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:14.285 7faee0ebc700 1 -- 172.21.2.112:0/10656559 wait complete. 2021-02-10T11:25:14.293 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:14.299 7faee0ebc700 1 Processor -- start 2021-02-10T11:25:14.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:14.299 7faee0ebc700 1 -- start start 2021-02-10T11:25:14.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:14.300 7faee0ebc700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faedc0b3940 0x7faedc0b30b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:14.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:14.300 7faee0ebc700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7faedc19bb00 0x7faedc0b3e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:14.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:14.300 7faee0ebc700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7faedc19bf00 0x7faedc0b4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:14.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:14.300 7faee0ebc700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faedc0b7710 con 0x7faedc0b3940 2021-02-10T11:25:14.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:14.300 7faee0ebc700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faedc0b4a50 con 0x7faedc19bf00 2021-02-10T11:25:14.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:14.300 7faee0ebc700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faedc0b7c90 con 0x7faedc19bb00 2021-02-10T11:25:14.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:14.300 7faedab13700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faedc0b3940 0x7faedc0b30b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:14.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:14.300 7faedab13700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faedc0b3940 0x7faedc0b30b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:45674/0 (socket says 172.21.2.112:45674) 2021-02-10T11:25:14.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:14.300 7faedab13700 1 -- 172.21.2.112:0/1463032108 learned_addr learned my addr 172.21.2.112:0/1463032108 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:25:14.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:14.300 7faedab13700 1 -- 172.21.2.112:0/1463032108 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7faedc19bb00 msgr2=0x7faedc0b3e10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:25:14.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:14.300 7faedab13700 1 --2- 172.21.2.112:0/1463032108 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7faedc19bb00 0x7faedc0b3e10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:14.295 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:14.300 7faedab13700 1 -- 172.21.2.112:0/1463032108 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7faedc19bf00 msgr2=0x7faedc0b4430 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:25:14.295 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:14.300 7faedab13700 1 --2- 172.21.2.112:0/1463032108 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7faedc19bf00 0x7faedc0b4430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:14.295 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:14.300 7faedab13700 1 -- 172.21.2.112:0/1463032108 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faedc0b7eb0 con 0x7faedc0b3940 2021-02-10T11:25:14.295 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:14.300 7faedab13700 1 --2- 172.21.2.112:0/1463032108 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faedc0b3940 0x7faedc0b30b0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 rx=0x7faecc00b660 tx=0x7faecc00ba00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:14.295 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:14.300 7faecb7fe700 1 -- 172.21.2.112:0/1463032108 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faecc009440 con 0x7faedc0b3940 2021-02-10T11:25:14.295 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:14.301 7faecb7fe700 1 -- 172.21.2.112:0/1463032108 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faecc006390 con 0x7faedc0b3940 2021-02-10T11:25:14.295 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:14.301 7faecb7fe700 1 -- 172.21.2.112:0/1463032108 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faecc002030 con 0x7faedc0b3940 2021-02-10T11:25:14.295 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:14.301 7faee0ebc700 1 -- 172.21.2.112:0/1463032108 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faedc0b7eb0 con 0x7faedc0b3940 2021-02-10T11:25:14.295 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:14.301 7faee0ebc700 1 -- 172.21.2.112:0/1463032108 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faedc0b7eb0 con 0x7faedc0b3940 2021-02-10T11:25:14.295 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:14.301 7faecb7fe700 1 -- 172.21.2.112:0/1463032108 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7faecc0093e0 con 0x7faedc0b3940 2021-02-10T11:25:14.295 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:14.301 7faecb7fe700 1 --2- 172.21.2.112:0/1463032108 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7faec0018730 0x7faec001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:14.295 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:14.301 7faecb7fe700 1 -- 172.21.2.112:0/1463032108 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7faecc032650 con 0x7faedc0b3940 2021-02-10T11:25:14.295 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:14.302 7faeda312700 1 --2- 172.21.2.112:0/1463032108 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7faec0018730 0x7faec001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:14.296 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:14.302 7faeda312700 1 --2- 172.21.2.112:0/1463032108 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7faec0018730 0x7faec001ac10 crc :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:14.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:14.333 7faee0ebc700 1 -- 172.21.2.112:0/1463032108 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faedc105b30 con 0x7faedc0b3940 2021-02-10T11:25:14.326 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:14.336 7faecb7fe700 1 -- 172.21.2.112:0/1463032108 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7faecc0090c0 con 0x7faedc0b3940 2021-02-10T11:25:14.457 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:14.466 7faee0ebc700 1 -- 172.21.2.112:0/1463032108 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348634", "target": ["mgr", ""]}) v1 -- 0x7faedc0b7710 con 0x7faec0018730 2021-02-10T11:25:14.460 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:14.470 7faecb7fe700 1 -- 172.21.2.112:0/1463032108 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7faed00009c0 con 0x7faec0018730 2021-02-10T11:25:14.460 INFO:teuthology.orchestra.run.gibba012.stdout:/volumes/_nogroup/subvolume_0000000000348634/8ff0e139-deeb-4653-9c6d-078d897ba8b6 2021-02-10T11:25:14.461 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:14.471 7faee0ebc700 1 -- 172.21.2.112:0/1463032108 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7faec0018730 msgr2=0x7faec001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:14.461 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:14.471 7faee0ebc700 1 --2- 172.21.2.112:0/1463032108 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7faec0018730 0x7faec001ac10 crc :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:14.461 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:14.471 7faee0ebc700 1 -- 172.21.2.112:0/1463032108 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faedc0b3940 msgr2=0x7faedc0b30b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:14.461 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:14.471 7faee0ebc700 1 --2- 172.21.2.112:0/1463032108 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faedc0b3940 0x7faedc0b30b0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 rx=0x7faecc00b660 tx=0x7faecc00ba00).stop 2021-02-10T11:25:14.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:14.471 7faee0ebc700 1 -- 172.21.2.112:0/1463032108 shutdown_connections 2021-02-10T11:25:14.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:14.471 7faee0ebc700 1 --2- 172.21.2.112:0/1463032108 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7faec0018730 0x7faec001ac10 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:14.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:14.471 7faee0ebc700 1 --2- 172.21.2.112:0/1463032108 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7faedc19bf00 0x7faedc0b4430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:14.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:14.471 7faee0ebc700 1 --2- 172.21.2.112:0/1463032108 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7faedc19bb00 0x7faedc0b3e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:14.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:14.471 7faee0ebc700 1 --2- 172.21.2.112:0/1463032108 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faedc0b3940 0x7faedc0b30b0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:14.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:14.471 7faee0ebc700 1 -- 172.21.2.112:0/1463032108 >> 172.21.2.112:0/1463032108 conn(0x7faedc094e10 msgr2=0x7faedc0a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:14.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:14.471 7faee0ebc700 1 -- 172.21.2.112:0/1463032108 shutdown_connections 2021-02-10T11:25:14.463 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:14.471 7faee0ebc700 1 -- 172.21.2.112:0/1463032108 wait complete. 2021-02-10T11:25:14.470 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000348634 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000348634/8ff0e139-deeb-4653-9c6d-078d897ba8b6 2021-02-10T11:25:14.470 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:14.470 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:14.470 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:14.806 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:25:14.807 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:14.807 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348634/8ff0e139-deeb-4653-9c6d-078d897ba8b6/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:15.186 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:25:15.186 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:25:15.186 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0238684 s, 43.9 MB/s 2021-02-10T11:25:15.187 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:15.187 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:15.188 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:15.523 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:25:15.523 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:15.523 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348634/8ff0e139-deeb-4653-9c6d-078d897ba8b6/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:15.898 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:25:15.899 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:25:15.899 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0245058 s, 42.8 MB/s 2021-02-10T11:25:15.900 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:15.900 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:15.900 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:16.236 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:25:16.236 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:16.237 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348634/8ff0e139-deeb-4653-9c6d-078d897ba8b6/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:16.614 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:25:16.614 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:25:16.614 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0264471 s, 39.6 MB/s 2021-02-10T11:25:16.615 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:16.615 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:16.615 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:16.950 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:25:16.950 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:16.950 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348634/8ff0e139-deeb-4653-9c6d-078d897ba8b6/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:17.326 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:25:17.326 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:25:17.326 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0232369 s, 45.1 MB/s 2021-02-10T11:25:17.327 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:17.327 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:17.327 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:17.667 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:25:17.667 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:17.668 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348634/8ff0e139-deeb-4653-9c6d-078d897ba8b6/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:18.042 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:25:18.042 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:25:18.043 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0231401 s, 45.3 MB/s 2021-02-10T11:25:18.043 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:18.043 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:18.043 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:18.379 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:25:18.379 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:18.379 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348634/8ff0e139-deeb-4653-9c6d-078d897ba8b6/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:18.755 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:25:18.755 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:25:18.755 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0269893 s, 38.9 MB/s 2021-02-10T11:25:18.756 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:18.756 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:18.757 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:19.092 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:25:19.092 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:19.093 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348634/8ff0e139-deeb-4653-9c6d-078d897ba8b6/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:19.462 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:25:19.462 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:25:19.462 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0222196 s, 47.2 MB/s 2021-02-10T11:25:19.463 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:19.463 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:19.463 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:19.799 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:25:19.799 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:19.800 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348634/8ff0e139-deeb-4653-9c6d-078d897ba8b6/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:20.175 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:25:20.175 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:25:20.175 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0229592 s, 45.7 MB/s 2021-02-10T11:25:20.176 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:20.176 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:20.176 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:20.511 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:25:20.511 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:20.511 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348634/8ff0e139-deeb-4653-9c6d-078d897ba8b6/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:20.888 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:25:20.888 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:25:20.888 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0241807 s, 43.4 MB/s 2021-02-10T11:25:20.889 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:20.889 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:20.889 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:21.224 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:25:21.224 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:21.225 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348634/8ff0e139-deeb-4653-9c6d-078d897ba8b6/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:21.601 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:25:21.601 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:25:21.601 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0247758 s, 42.3 MB/s 2021-02-10T11:25:21.602 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000348635 2021-02-10T11:25:22.034 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.027 7febe43a4700 1 Processor -- start 2021-02-10T11:25:22.034 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.027 7febe43a4700 1 -- start start 2021-02-10T11:25:22.034 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.028 7febe43a4700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7febdc1b0470 0x7febdc1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:22.034 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.028 7febe43a4700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7febdc0f20e0 0x7febdc0f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:22.034 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.028 7febe43a4700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febdc0f0b50 con 0x7febdc0ef610 2021-02-10T11:25:22.034 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.028 7febe43a4700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febdc0f0820 con 0x7febdc0f20e0 2021-02-10T11:25:22.034 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.028 7febe43a4700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febdc0f24e0 con 0x7febdc1b0470 2021-02-10T11:25:22.034 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.028 7febe29c1700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7febdc0f20e0 0x7febdc0f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:22.034 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.028 7febe29c1700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7febdc0f20e0 0x7febdc0f0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:45678/0 (socket says 172.21.2.112:45678) 2021-02-10T11:25:22.034 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.028 7febe29c1700 1 -- 172.21.2.112:0/4148313707 learned_addr learned my addr 172.21.2.112:0/4148313707 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:25:22.035 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.028 7febe29c1700 1 -- 172.21.2.112:0/4148313707 >> v2:172.21.2.120:3300/0 conn(0x7febdc1b0470 msgr2=0x7febdc1b0870 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:25:22.035 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.028 7febe29c1700 1 --2- 172.21.2.112:0/4148313707 >> v2:172.21.2.120:3300/0 conn(0x7febdc1b0470 0x7febdc1b0870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:22.035 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.028 7febe29c1700 1 -- 172.21.2.112:0/4148313707 >> v1:172.21.2.120:6789/0 conn(0x7febdc0ef610 legacy=0x7febdc0efa10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:25:22.035 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.028 7febe29c1700 1 -- 172.21.2.112:0/4148313707 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febdc0e52e0 con 0x7febdc0f20e0 2021-02-10T11:25:22.035 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.028 7febe29c1700 1 --2- 172.21.2.112:0/4148313707 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7febdc0f20e0 0x7febdc0f0200 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 rx=0x7febd4006f00 tx=0x7febd4009f10).ready entity=mon.0 client_cookie=d9c6c52a2a5d2da9 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:22.035 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.028 7febe11be700 1 -- 172.21.2.112:0/4148313707 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7febd400b3b0 con 0x7febdc0f20e0 2021-02-10T11:25:22.035 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.028 7febe11be700 1 -- 172.21.2.112:0/4148313707 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febd400b730 con 0x7febdc0f20e0 2021-02-10T11:25:22.035 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.028 7febe11be700 1 -- 172.21.2.112:0/4148313707 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7febd400bc50 con 0x7febdc0f20e0 2021-02-10T11:25:22.035 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.028 7febe43a4700 1 -- 172.21.2.112:0/4148313707 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7febdc0f20e0 msgr2=0x7febdc0f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:22.035 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.028 7febe43a4700 1 --2- 172.21.2.112:0/4148313707 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7febdc0f20e0 0x7febdc0f0200 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 rx=0x7febd4006f00 tx=0x7febd4009f10).stop 2021-02-10T11:25:22.035 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.029 7febe43a4700 1 -- 172.21.2.112:0/4148313707 shutdown_connections 2021-02-10T11:25:22.035 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.029 7febe43a4700 1 --2- 172.21.2.112:0/4148313707 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7febdc0f20e0 0x7febdc0f0200 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:22.035 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.029 7febe43a4700 1 --2- 172.21.2.112:0/4148313707 >> v2:172.21.2.120:3300/0 conn(0x7febdc1b0470 0x7febdc1b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:22.035 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.029 7febe43a4700 1 -- 172.21.2.112:0/4148313707 >> 172.21.2.112:0/4148313707 conn(0x7febdc0ce3e0 msgr2=0x7febdc0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:22.036 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.029 7febe43a4700 1 -- 172.21.2.112:0/4148313707 shutdown_connections 2021-02-10T11:25:22.036 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.029 7febe43a4700 1 -- 172.21.2.112:0/4148313707 wait complete. 2021-02-10T11:25:22.037 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.043 7febe43a4700 1 Processor -- start 2021-02-10T11:25:22.037 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.044 7febe43a4700 1 -- start start 2021-02-10T11:25:22.037 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.044 7febe43a4700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7febdc0c36a0 0x7febdc0c2b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:22.037 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.044 7febe43a4700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7febdc0c3190 0x7febdc0b1d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:22.037 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.044 7febe43a4700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7febdc0b2370 0x7febdc0b2d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:22.037 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.044 7febe43a4700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febdc0b27c0 con 0x7febdc0c36a0 2021-02-10T11:25:22.037 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.044 7febe43a4700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febdc0b3360 con 0x7febdc0c3190 2021-02-10T11:25:22.037 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.044 7febe43a4700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febdc0b3540 con 0x7febdc0b2370 2021-02-10T11:25:22.037 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.044 7febe21c0700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7febdc0c36a0 0x7febdc0c2b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:22.038 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.044 7febe21c0700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7febdc0c36a0 0x7febdc0c2b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:45682/0 (socket says 172.21.2.112:45682) 2021-02-10T11:25:22.038 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.044 7febe21c0700 1 -- 172.21.2.112:0/1864132040 learned_addr learned my addr 172.21.2.112:0/1864132040 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:25:22.038 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.044 7febe21c0700 1 -- 172.21.2.112:0/1864132040 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7febdc0b2370 msgr2=0x7febdc0b2d40 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:25:22.038 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.044 7febe21c0700 1 --2- 172.21.2.112:0/1864132040 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7febdc0b2370 0x7febdc0b2d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:22.038 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.044 7febe21c0700 1 -- 172.21.2.112:0/1864132040 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7febdc0c3190 msgr2=0x7febdc0b1d50 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:25:22.038 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.044 7febe21c0700 1 --2- 172.21.2.112:0/1864132040 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7febdc0c3190 0x7febdc0b1d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:22.038 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.044 7febe21c0700 1 -- 172.21.2.112:0/1864132040 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febdc0b3760 con 0x7febdc0c36a0 2021-02-10T11:25:22.038 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.045 7febe21c0700 1 --2- 172.21.2.112:0/1864132040 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7febdc0c36a0 0x7febdc0c2b70 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 rx=0x7febd800b5c0 tx=0x7febd800b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:22.038 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.045 7febceffd700 1 -- 172.21.2.112:0/1864132040 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7febd800bd50 con 0x7febdc0c36a0 2021-02-10T11:25:22.038 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.045 7febceffd700 1 -- 172.21.2.112:0/1864132040 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febd8006030 con 0x7febdc0c36a0 2021-02-10T11:25:22.038 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.045 7febceffd700 1 -- 172.21.2.112:0/1864132040 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7febd80054d0 con 0x7febdc0c36a0 2021-02-10T11:25:22.038 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.045 7febe43a4700 1 -- 172.21.2.112:0/1864132040 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febdc0b3760 con 0x7febdc0c36a0 2021-02-10T11:25:22.038 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.045 7febe43a4700 1 -- 172.21.2.112:0/1864132040 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febdc0b3cf0 con 0x7febdc0c36a0 2021-02-10T11:25:22.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.045 7febceffd700 1 -- 172.21.2.112:0/1864132040 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7febd8006210 con 0x7febdc0c36a0 2021-02-10T11:25:22.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.045 7febceffd700 1 --2- 172.21.2.112:0/1864132040 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7febc40186c0 0x7febc401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:22.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.046 7febceffd700 1 -- 172.21.2.112:0/1864132040 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7febd80325b0 con 0x7febdc0c36a0 2021-02-10T11:25:22.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.046 7febe19bf700 1 --2- 172.21.2.112:0/1864132040 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7febc40186c0 0x7febc401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:22.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.047 7febe19bf700 1 --2- 172.21.2.112:0/1864132040 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7febc40186c0 0x7febc401abc0 crc :-1 s=READY pgs=309 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:22.068 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.078 7febe43a4700 1 -- 172.21.2.112:0/1864132040 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febdc0edd10 con 0x7febdc0c36a0 2021-02-10T11:25:22.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.080 7febceffd700 1 -- 172.21.2.112:0/1864132040 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7febd8000940 con 0x7febdc0c36a0 2021-02-10T11:25:22.197 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.207 7febe43a4700 1 -- 172.21.2.112:0/1864132040 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348635", "target": ["mgr", ""]}) v1 -- 0x7febdc0b3cf0 con 0x7febc40186c0 2021-02-10T11:25:22.223 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.233 7febceffd700 1 -- 172.21.2.112:0/1864132040 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7febd00009b0 con 0x7febc40186c0 2021-02-10T11:25:22.231 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.240 7febe43a4700 1 -- 172.21.2.112:0/1864132040 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7febc40186c0 msgr2=0x7febc401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:22.231 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.240 7febe43a4700 1 --2- 172.21.2.112:0/1864132040 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7febc40186c0 0x7febc401abc0 crc :-1 s=READY pgs=309 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:22.231 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.240 7febe43a4700 1 -- 172.21.2.112:0/1864132040 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7febdc0c36a0 msgr2=0x7febdc0c2b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:22.231 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.240 7febe43a4700 1 --2- 172.21.2.112:0/1864132040 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7febdc0c36a0 0x7febdc0c2b70 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 rx=0x7febd800b5c0 tx=0x7febd800b960).stop 2021-02-10T11:25:22.231 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.240 7febe43a4700 1 -- 172.21.2.112:0/1864132040 shutdown_connections 2021-02-10T11:25:22.231 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.240 7febe43a4700 1 --2- 172.21.2.112:0/1864132040 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7febc40186c0 0x7febc401abc0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:22.231 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.240 7febe43a4700 1 --2- 172.21.2.112:0/1864132040 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7febdc0c3190 0x7febdc0b1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:22.231 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.240 7febe43a4700 1 --2- 172.21.2.112:0/1864132040 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7febdc0b2370 0x7febdc0b2d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:22.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.240 7febe43a4700 1 --2- 172.21.2.112:0/1864132040 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7febdc0c36a0 0x7febdc0c2b70 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:22.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.240 7febe43a4700 1 -- 172.21.2.112:0/1864132040 >> 172.21.2.112:0/1864132040 conn(0x7febdc09f630 msgr2=0x7febdc0a0130 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:22.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.240 7febe43a4700 1 -- 172.21.2.112:0/1864132040 shutdown_connections 2021-02-10T11:25:22.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.240 7febe43a4700 1 -- 172.21.2.112:0/1864132040 wait complete. 2021-02-10T11:25:22.239 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000348635 2021-02-10T11:25:22.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.664 7f1dd65c0700 1 Processor -- start 2021-02-10T11:25:22.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.664 7f1dd65c0700 1 -- start start 2021-02-10T11:25:22.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.664 7f1dd65c0700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1dd01b0870 0x7f1dd00f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:22.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.664 7f1dd65c0700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f1dd00efe00 0x7f1dd00f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:22.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.664 7f1dd65c0700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dd01b0c70 con 0x7f1dd01b0470 2021-02-10T11:25:22.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.664 7f1dd65c0700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1dd00f0b50 con 0x7f1dd00efe00 2021-02-10T11:25:22.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.664 7f1dd65c0700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1dd00f0820 con 0x7f1dd01b0870 2021-02-10T11:25:22.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.665 7f1dd4bdd700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f1dd00efe00 0x7f1dd00f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:22.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.665 7f1dd4bdd700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f1dd00efe00 0x7f1dd00f0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:45694/0 (socket says 172.21.2.112:45694) 2021-02-10T11:25:22.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.665 7f1dd4bdd700 1 -- 172.21.2.112:0/557266524 learned_addr learned my addr 172.21.2.112:0/557266524 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:25:22.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.665 7f1dcf7fe700 1 --2- 172.21.2.112:0/557266524 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1dd01b0870 0x7f1dd00f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:22.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.665 7f1dd4bdd700 1 -- 172.21.2.112:0/557266524 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1dd01b0870 msgr2=0x7f1dd00f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:22.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.665 7f1dd4bdd700 1 --2- 172.21.2.112:0/557266524 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1dd01b0870 0x7f1dd00f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:22.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.665 7f1dd4bdd700 1 -- 172.21.2.112:0/557266524 >> v1:172.21.2.120:6789/0 conn(0x7f1dd01b0470 legacy=0x7f1dd00ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:22.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.665 7f1dd4bdd700 1 -- 172.21.2.112:0/557266524 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dd00e52e0 con 0x7f1dd00efe00 2021-02-10T11:25:22.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.665 7f1dd4bdd700 1 --2- 172.21.2.112:0/557266524 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1dd00efe00 0x7f1dd00f0200 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 rx=0x7f1dc8006f00 tx=0x7f1dc8009f10).ready entity=mon.0 client_cookie=d955cde2bb2c1c0a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:22.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.665 7f1dceffd700 1 -- 172.21.2.112:0/557266524 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1dc800b3b0 con 0x7f1dd00efe00 2021-02-10T11:25:22.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.665 7f1dceffd700 1 -- 172.21.2.112:0/557266524 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1dc800b730 con 0x7f1dd00efe00 2021-02-10T11:25:22.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.665 7f1dceffd700 1 -- 172.21.2.112:0/557266524 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1dc800bc10 con 0x7f1dd00efe00 2021-02-10T11:25:22.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.665 7f1dd65c0700 1 -- 172.21.2.112:0/557266524 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1dd00efe00 msgr2=0x7f1dd00f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:22.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.665 7f1dd65c0700 1 --2- 172.21.2.112:0/557266524 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1dd00efe00 0x7f1dd00f0200 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 rx=0x7f1dc8006f00 tx=0x7f1dc8009f10).stop 2021-02-10T11:25:22.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.666 7f1dd65c0700 1 -- 172.21.2.112:0/557266524 shutdown_connections 2021-02-10T11:25:22.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.666 7f1dd65c0700 1 --2- 172.21.2.112:0/557266524 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1dd00efe00 0x7f1dd00f0200 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:22.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.666 7f1dd65c0700 1 --2- 172.21.2.112:0/557266524 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1dd01b0870 0x7f1dd00f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:22.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.666 7f1dd65c0700 1 -- 172.21.2.112:0/557266524 >> 172.21.2.112:0/557266524 conn(0x7f1dd00ce3e0 msgr2=0x7f1dd00e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:22.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.666 7f1dd65c0700 1 -- 172.21.2.112:0/557266524 shutdown_connections 2021-02-10T11:25:22.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.666 7f1dd65c0700 1 -- 172.21.2.112:0/557266524 wait complete. 2021-02-10T11:25:22.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.681 7f1dd65c0700 1 Processor -- start 2021-02-10T11:25:22.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.681 7f1dd65c0700 1 -- start start 2021-02-10T11:25:22.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.681 7f1dd65c0700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1dd00a0420 0x7f1dd00a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:22.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.681 7f1dd65c0700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1dd0189d00 0x7f1dd018a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:22.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.681 7f1dd65c0700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1dd018a720 0x7f1dd00b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:22.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.681 7f1dd65c0700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dd0189780 con 0x7f1dd018a720 2021-02-10T11:25:22.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.681 7f1dd65c0700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dd018ab20 con 0x7f1dd0189d00 2021-02-10T11:25:22.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.681 7f1dd65c0700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1dd00a1140 con 0x7f1dd00a0420 2021-02-10T11:25:22.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.681 7f1dd4bdd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1dd018a720 0x7f1dd00b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:22.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.681 7f1dd4bdd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1dd018a720 0x7f1dd00b7be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:45698/0 (socket says 172.21.2.112:45698) 2021-02-10T11:25:22.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.681 7f1dd4bdd700 1 -- 172.21.2.112:0/1942002909 learned_addr learned my addr 172.21.2.112:0/1942002909 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:25:22.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.682 7f1dd4bdd700 1 -- 172.21.2.112:0/1942002909 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1dd00a0420 msgr2=0x7f1dd00a0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:22.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.682 7f1dd4bdd700 1 --2- 172.21.2.112:0/1942002909 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1dd00a0420 0x7f1dd00a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:22.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.682 7f1dd4bdd700 1 -- 172.21.2.112:0/1942002909 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1dd0189d00 msgr2=0x7f1dd018a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:22.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.682 7f1dd4bdd700 1 --2- 172.21.2.112:0/1942002909 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1dd0189d00 0x7f1dd018a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:22.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.682 7f1dd4bdd700 1 -- 172.21.2.112:0/1942002909 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dd00b8200 con 0x7f1dd018a720 2021-02-10T11:25:22.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.682 7f1dd4bdd700 1 --2- 172.21.2.112:0/1942002909 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1dd018a720 0x7f1dd00b7be0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 rx=0x7f1dc8009390 tx=0x7f1dc800bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:22.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.682 7f1dccff9700 1 -- 172.21.2.112:0/1942002909 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1dc8003030 con 0x7f1dd018a720 2021-02-10T11:25:22.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.682 7f1dccff9700 1 -- 172.21.2.112:0/1942002909 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1dc8006c80 con 0x7f1dd018a720 2021-02-10T11:25:22.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.682 7f1dccff9700 1 -- 172.21.2.112:0/1942002909 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1dc8001480 con 0x7f1dd018a720 2021-02-10T11:25:22.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.682 7f1dd65c0700 1 -- 172.21.2.112:0/1942002909 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1dd00b8200 con 0x7f1dd018a720 2021-02-10T11:25:22.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.682 7f1dd65c0700 1 -- 172.21.2.112:0/1942002909 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1dd00b8790 con 0x7f1dd018a720 2021-02-10T11:25:22.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.683 7f1dccff9700 1 -- 172.21.2.112:0/1942002909 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1dc8001270 con 0x7f1dd018a720 2021-02-10T11:25:22.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.683 7f1dccff9700 1 --2- 172.21.2.112:0/1942002909 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1db00186c0 0x7f1db001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:22.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.683 7f1dccff9700 1 -- 172.21.2.112:0/1942002909 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f1dc8032950 con 0x7f1dd018a720 2021-02-10T11:25:22.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.683 7f1dcffff700 1 --2- 172.21.2.112:0/1942002909 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1db00186c0 0x7f1db001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:22.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.683 7f1dcffff700 1 --2- 172.21.2.112:0/1942002909 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1db00186c0 0x7f1db001abc0 crc :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:22.704 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.714 7f1dd65c0700 1 -- 172.21.2.112:0/1942002909 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1dd0105ae0 con 0x7f1dd018a720 2021-02-10T11:25:22.707 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.716 7f1dccff9700 1 -- 172.21.2.112:0/1942002909 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1dc802dfc0 con 0x7f1dd018a720 2021-02-10T11:25:22.834 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.844 7f1dd65c0700 1 -- 172.21.2.112:0/1942002909 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348635", "target": ["mgr", ""]}) v1 -- 0x7f1dd00b9020 con 0x7f1db00186c0 2021-02-10T11:25:22.838 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.848 7f1dccff9700 1 -- 172.21.2.112:0/1942002909 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f1dc0006a10 con 0x7f1db00186c0 2021-02-10T11:25:22.841 INFO:teuthology.orchestra.run.gibba012.stdout:/volumes/_nogroup/subvolume_0000000000348635/2619a057-58fe-4fca-8b50-72c64e53935d 2021-02-10T11:25:22.843 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.852 7f1dd65c0700 1 -- 172.21.2.112:0/1942002909 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1db00186c0 msgr2=0x7f1db001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:22.843 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.852 7f1dd65c0700 1 --2- 172.21.2.112:0/1942002909 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1db00186c0 0x7f1db001abc0 crc :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:22.843 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.852 7f1dd65c0700 1 -- 172.21.2.112:0/1942002909 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1dd018a720 msgr2=0x7f1dd00b7be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:22.843 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.852 7f1dd65c0700 1 --2- 172.21.2.112:0/1942002909 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1dd018a720 0x7f1dd00b7be0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 rx=0x7f1dc8009390 tx=0x7f1dc800bd40).stop 2021-02-10T11:25:22.843 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.852 7f1dd65c0700 1 -- 172.21.2.112:0/1942002909 shutdown_connections 2021-02-10T11:25:22.843 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.852 7f1dd65c0700 1 --2- 172.21.2.112:0/1942002909 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1db00186c0 0x7f1db001abc0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:22.844 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.852 7f1dd65c0700 1 --2- 172.21.2.112:0/1942002909 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1dd0189d00 0x7f1dd018a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:22.844 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.852 7f1dd65c0700 1 --2- 172.21.2.112:0/1942002909 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1dd018a720 0x7f1dd00b7be0 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:22.844 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.852 7f1dd65c0700 1 --2- 172.21.2.112:0/1942002909 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1dd00a0420 0x7f1dd00a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:22.844 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.852 7f1dd65c0700 1 -- 172.21.2.112:0/1942002909 >> 172.21.2.112:0/1942002909 conn(0x7f1dd0094e10 msgr2=0x7f1dd00a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:22.844 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.853 7f1dd65c0700 1 -- 172.21.2.112:0/1942002909 shutdown_connections 2021-02-10T11:25:22.844 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:22.853 7f1dd65c0700 1 -- 172.21.2.112:0/1942002909 wait complete. 2021-02-10T11:25:22.851 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000348635 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000348635/2619a057-58fe-4fca-8b50-72c64e53935d 2021-02-10T11:25:22.851 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:22.852 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:22.852 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:23.187 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:25:23.187 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:23.188 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348635/2619a057-58fe-4fca-8b50-72c64e53935d/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:23.569 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:25:23.569 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:25:23.569 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.024198 s, 43.3 MB/s 2021-02-10T11:25:23.570 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:23.570 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:23.571 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:23.905 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:25:23.906 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:23.906 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348635/2619a057-58fe-4fca-8b50-72c64e53935d/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:24.281 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:25:24.281 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:25:24.281 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0221886 s, 47.3 MB/s 2021-02-10T11:25:24.282 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:24.282 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:24.282 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:24.617 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:25:24.618 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:24.618 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348635/2619a057-58fe-4fca-8b50-72c64e53935d/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:24.992 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:25:24.992 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:25:24.992 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0253604 s, 41.3 MB/s 2021-02-10T11:25:24.993 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:24.993 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:24.993 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:25.328 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:25:25.329 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:25.329 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348635/2619a057-58fe-4fca-8b50-72c64e53935d/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:25.705 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:25:25.705 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:25:25.705 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0238645 s, 43.9 MB/s 2021-02-10T11:25:25.706 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:25.706 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:25.706 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:26.040 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:25:26.041 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:26.041 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348635/2619a057-58fe-4fca-8b50-72c64e53935d/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:26.418 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:25:26.418 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:25:26.418 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0243909 s, 43.0 MB/s 2021-02-10T11:25:26.419 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:26.419 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:26.419 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:26.755 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:25:26.755 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:26.755 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348635/2619a057-58fe-4fca-8b50-72c64e53935d/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:27.130 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:25:27.130 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:25:27.131 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0226716 s, 46.3 MB/s 2021-02-10T11:25:27.131 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:27.132 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:27.132 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:27.467 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:25:27.467 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:27.467 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348635/2619a057-58fe-4fca-8b50-72c64e53935d/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:27.842 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:25:28.017 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:25:28.017 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0234235 s, 44.8 MB/s 2021-02-10T11:25:28.018 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:28.019 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:28.019 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:28.354 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:25:28.354 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:28.354 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348635/2619a057-58fe-4fca-8b50-72c64e53935d/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:28.730 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:25:28.730 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:25:28.730 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0237931 s, 44.1 MB/s 2021-02-10T11:25:28.731 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:28.731 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:28.731 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:29.067 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:25:29.067 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:29.068 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348635/2619a057-58fe-4fca-8b50-72c64e53935d/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:29.440 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:25:29.440 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:25:29.440 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0259022 s, 40.5 MB/s 2021-02-10T11:25:29.441 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:29.441 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:29.441 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:29.776 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:25:29.776 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:29.776 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348635/2619a057-58fe-4fca-8b50-72c64e53935d/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:30.150 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:25:30.151 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:25:30.151 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0249694 s, 42.0 MB/s 2021-02-10T11:25:30.152 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000348636 2021-02-10T11:25:30.584 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:30.576 7f3c84e6e700 1 Processor -- start 2021-02-10T11:25:30.584 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:30.576 7f3c84e6e700 1 -- start start 2021-02-10T11:25:30.584 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:30.577 7f3c84e6e700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3c801b0870 0x7f3c800f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:30.584 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:30.577 7f3c84e6e700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f3c800efe00 0x7f3c800f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:30.584 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:30.577 7f3c84e6e700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c801b0c70 con 0x7f3c801b0470 2021-02-10T11:25:30.584 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:30.577 7f3c84e6e700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c800f0b50 con 0x7f3c800efe00 2021-02-10T11:25:30.584 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:30.577 7f3c84e6e700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c800f0820 con 0x7f3c801b0870 2021-02-10T11:25:30.584 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:30.577 7f3c7eb13700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f3c801b0470 0x7f3c800ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:45166/0 (socket says 172.21.2.112:45166) 2021-02-10T11:25:30.584 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:30.577 7f3c7eb13700 1 -- 172.21.2.112:0/3647791844 learned_addr learned my addr 172.21.2.112:0/3647791844 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:25:30.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:30.577 7f3c7f314700 1 --2- 172.21.2.112:0/3647791844 >> v2:172.21.2.112:3300/0 conn(0x7f3c800efe00 0x7f3c800f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:30.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:30.577 7f3c7db11700 1 -- 172.21.2.112:0/3647791844 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2732654776 0 0) 0x7f3c78001500 con 0x7f3c801b0470 2021-02-10T11:25:30.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:30.577 7f3c7db11700 1 -- 172.21.2.112:0/3647791844 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c68002fe0 con 0x7f3c801b0470 2021-02-10T11:25:30.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:30.578 7f3c7f314700 1 -- 172.21.2.112:0/3647791844 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3c801b0870 msgr2=0x7f3c800f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:30.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:30.578 7f3c7f314700 1 --2- 172.21.2.112:0/3647791844 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3c801b0870 0x7f3c800f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:30.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:30.578 7f3c7f314700 1 -- 172.21.2.112:0/3647791844 >> v1:172.21.2.112:6789/0 conn(0x7f3c801b0470 legacy=0x7f3c800ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:30.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:30.578 7f3c7f314700 1 -- 172.21.2.112:0/3647791844 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c800e52e0 con 0x7f3c800efe00 2021-02-10T11:25:30.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:30.578 7f3c7f314700 1 --2- 172.21.2.112:0/3647791844 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3c800efe00 0x7f3c800f0200 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 rx=0x7f3c70006ab0 tx=0x7f3c70009b40).ready entity=mon.0 client_cookie=3762a7bd7e5b3163 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:30.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:30.578 7f3c7db11700 1 -- 172.21.2.112:0/3647791844 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3c700022e0 con 0x7f3c800efe00 2021-02-10T11:25:30.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:30.578 7f3c7db11700 1 -- 172.21.2.112:0/3647791844 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c70002560 con 0x7f3c800efe00 2021-02-10T11:25:30.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:30.578 7f3c7db11700 1 -- 172.21.2.112:0/3647791844 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3c70002930 con 0x7f3c800efe00 2021-02-10T11:25:30.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:30.578 7f3c84e6e700 1 -- 172.21.2.112:0/3647791844 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3c800efe00 msgr2=0x7f3c800f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:30.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:30.578 7f3c84e6e700 1 --2- 172.21.2.112:0/3647791844 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3c800efe00 0x7f3c800f0200 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 rx=0x7f3c70006ab0 tx=0x7f3c70009b40).stop 2021-02-10T11:25:30.586 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:30.579 7f3c84e6e700 1 -- 172.21.2.112:0/3647791844 shutdown_connections 2021-02-10T11:25:30.586 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:30.579 7f3c84e6e700 1 --2- 172.21.2.112:0/3647791844 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3c800efe00 0x7f3c800f0200 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:30.586 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:30.579 7f3c84e6e700 1 --2- 172.21.2.112:0/3647791844 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3c801b0870 0x7f3c800f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:30.586 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:30.579 7f3c84e6e700 1 -- 172.21.2.112:0/3647791844 >> 172.21.2.112:0/3647791844 conn(0x7f3c800ce3e0 msgr2=0x7f3c800e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:30.586 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:30.579 7f3c84e6e700 1 -- 172.21.2.112:0/3647791844 shutdown_connections 2021-02-10T11:25:30.586 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:30.579 7f3c84e6e700 1 -- 172.21.2.112:0/3647791844 wait complete. 2021-02-10T11:25:30.587 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:30.594 7f3c84e6e700 1 Processor -- start 2021-02-10T11:25:30.588 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:30.594 7f3c84e6e700 1 -- start start 2021-02-10T11:25:30.588 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:30.594 7f3c84e6e700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3c8009f140 0x7f3c800c1940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:30.588 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:30.594 7f3c84e6e700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3c800c1f60 0x7f3c8009e300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:30.588 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:30.594 7f3c84e6e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3c8009e920 0x7f3c800c2960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:30.588 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:30.594 7f3c84e6e700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c800c23e0 con 0x7f3c8009e920 2021-02-10T11:25:30.588 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:30.594 7f3c84e6e700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c8009ed20 con 0x7f3c800c1f60 2021-02-10T11:25:30.588 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:30.594 7f3c84e6e700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c80105d10 con 0x7f3c8009f140 2021-02-10T11:25:30.588 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:30.594 7f3c7f314700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3c8009e920 0x7f3c800c2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:30.588 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:30.594 7f3c7f314700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3c8009e920 0x7f3c800c2960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:45710/0 (socket says 172.21.2.112:45710) 2021-02-10T11:25:30.588 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:30.594 7f3c7f314700 1 -- 172.21.2.112:0/1646641613 learned_addr learned my addr 172.21.2.112:0/1646641613 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:25:30.588 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:30.595 7f3c7f314700 1 -- 172.21.2.112:0/1646641613 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3c8009f140 msgr2=0x7f3c800c1940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:30.588 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:30.595 7f3c7f314700 1 --2- 172.21.2.112:0/1646641613 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3c8009f140 0x7f3c800c1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:30.588 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:30.595 7f3c7f314700 1 -- 172.21.2.112:0/1646641613 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3c800c1f60 msgr2=0x7f3c8009e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:30.589 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:30.595 7f3c7f314700 1 --2- 172.21.2.112:0/1646641613 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3c800c1f60 0x7f3c8009e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:30.589 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:30.595 7f3c7f314700 1 -- 172.21.2.112:0/1646641613 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c801a4910 con 0x7f3c8009e920 2021-02-10T11:25:30.589 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:30.595 7f3c7f314700 1 --2- 172.21.2.112:0/1646641613 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3c8009e920 0x7f3c800c2960 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 rx=0x7f3c7000b5e0 tx=0x7f3c7000b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:30.589 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:30.595 7f3c6f7fe700 1 -- 172.21.2.112:0/1646641613 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3c70009070 con 0x7f3c8009e920 2021-02-10T11:25:30.589 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:30.595 7f3c6f7fe700 1 -- 172.21.2.112:0/1646641613 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c70002280 con 0x7f3c8009e920 2021-02-10T11:25:30.589 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:30.595 7f3c84e6e700 1 -- 172.21.2.112:0/1646641613 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c801a4910 con 0x7f3c8009e920 2021-02-10T11:25:30.589 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:30.595 7f3c84e6e700 1 -- 172.21.2.112:0/1646641613 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c800c3270 con 0x7f3c8009e920 2021-02-10T11:25:30.589 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:30.595 7f3c6f7fe700 1 -- 172.21.2.112:0/1646641613 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3c70002760 con 0x7f3c8009e920 2021-02-10T11:25:30.589 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:30.596 7f3c6f7fe700 1 -- 172.21.2.112:0/1646641613 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3c70002460 con 0x7f3c8009e920 2021-02-10T11:25:30.589 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:30.596 7f3c6f7fe700 1 --2- 172.21.2.112:0/1646641613 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3c64018730 0x7f3c6401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:30.589 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:30.596 7f3c6f7fe700 1 -- 172.21.2.112:0/1646641613 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f3c70032810 con 0x7f3c8009e920 2021-02-10T11:25:30.589 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:30.597 7f3c7eb13700 1 --2- 172.21.2.112:0/1646641613 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3c64018730 0x7f3c6401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:30.589 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:30.597 7f3c7eb13700 1 --2- 172.21.2.112:0/1646641613 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3c64018730 0x7f3c6401ac10 crc :-1 s=READY pgs=311 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:30.617 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:30.627 7f3c84e6e700 1 -- 172.21.2.112:0/1646641613 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c800f0b50 con 0x7f3c8009e920 2021-02-10T11:25:30.620 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:30.630 7f3c6f7fe700 1 -- 172.21.2.112:0/1646641613 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3c700093e0 con 0x7f3c8009e920 2021-02-10T11:25:30.747 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:30.757 7f3c84e6e700 1 -- 172.21.2.112:0/1646641613 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348636", "target": ["mgr", ""]}) v1 -- 0x7f3c800c3270 con 0x7f3c64018730 2021-02-10T11:25:30.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:30.780 7f3c6f7fe700 1 -- 172.21.2.112:0/1646641613 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f3c78004720 con 0x7f3c64018730 2021-02-10T11:25:30.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:30.789 7f3c84e6e700 1 -- 172.21.2.112:0/1646641613 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3c64018730 msgr2=0x7f3c6401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:30.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:30.789 7f3c84e6e700 1 --2- 172.21.2.112:0/1646641613 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3c64018730 0x7f3c6401ac10 crc :-1 s=READY pgs=311 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:30.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:30.789 7f3c84e6e700 1 -- 172.21.2.112:0/1646641613 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3c8009e920 msgr2=0x7f3c800c2960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:30.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:30.789 7f3c84e6e700 1 --2- 172.21.2.112:0/1646641613 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3c8009e920 0x7f3c800c2960 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 rx=0x7f3c7000b5e0 tx=0x7f3c7000b980).stop 2021-02-10T11:25:30.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:30.790 7f3c84e6e700 1 -- 172.21.2.112:0/1646641613 shutdown_connections 2021-02-10T11:25:30.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:30.790 7f3c84e6e700 1 --2- 172.21.2.112:0/1646641613 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3c64018730 0x7f3c6401ac10 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:30.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:30.790 7f3c84e6e700 1 --2- 172.21.2.112:0/1646641613 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3c800c1f60 0x7f3c8009e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:30.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:30.790 7f3c84e6e700 1 --2- 172.21.2.112:0/1646641613 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3c8009e920 0x7f3c800c2960 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:30.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:30.790 7f3c84e6e700 1 --2- 172.21.2.112:0/1646641613 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3c8009f140 0x7f3c800c1940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:30.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:30.790 7f3c84e6e700 1 -- 172.21.2.112:0/1646641613 >> 172.21.2.112:0/1646641613 conn(0x7f3c8009fc10 msgr2=0x7f3c800b3250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:30.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:30.790 7f3c84e6e700 1 -- 172.21.2.112:0/1646641613 shutdown_connections 2021-02-10T11:25:30.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:30.790 7f3c84e6e700 1 -- 172.21.2.112:0/1646641613 wait complete. 2021-02-10T11:25:30.788 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000348636 2021-02-10T11:25:31.220 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:31.213 7fa09ab70700 1 Processor -- start 2021-02-10T11:25:31.220 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:31.213 7fa09ab70700 1 -- start start 2021-02-10T11:25:31.220 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:31.213 7fa09ab70700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fa0941b0470 0x7fa0941b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:31.220 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:31.213 7fa09ab70700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa0940ef610 0x7fa0940efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:31.220 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:31.213 7fa09ab70700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0940f0b50 con 0x7fa0940f0030 2021-02-10T11:25:31.220 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:31.213 7fa09ab70700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0940e52e0 con 0x7fa0941b0470 2021-02-10T11:25:31.220 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:31.213 7fa09ab70700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0940e54d0 con 0x7fa0940ef610 2021-02-10T11:25:31.220 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:31.214 7fa09898c700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fa0941b0470 0x7fa0941b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:31.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:31.214 7fa09918d700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fa0940f0030 0x7fa0940f10d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:45180/0 (socket says 172.21.2.112:45180) 2021-02-10T11:25:31.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:31.214 7fa09918d700 1 -- 172.21.2.112:0/1785634469 learned_addr learned my addr 172.21.2.112:0/1785634469 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:25:31.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:31.214 7fa093fff700 1 --2- 172.21.2.112:0/1785634469 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa0940ef610 0x7fa0940efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:31.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:31.214 7fa093fff700 1 -- 172.21.2.112:0/1785634469 >> v2:172.21.2.112:3300/0 conn(0x7fa0941b0470 msgr2=0x7fa0941b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:31.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:31.214 7fa093fff700 1 --2- 172.21.2.112:0/1785634469 >> v2:172.21.2.112:3300/0 conn(0x7fa0941b0470 0x7fa0941b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:31.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:31.214 7fa093fff700 1 -- 172.21.2.112:0/1785634469 >> v1:172.21.2.112:6789/0 conn(0x7fa0940f0030 legacy=0x7fa0940f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:31.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:31.214 7fa093fff700 1 -- 172.21.2.112:0/1785634469 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0940f0430 con 0x7fa0940ef610 2021-02-10T11:25:31.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:31.214 7fa09898c700 1 --2- 172.21.2.112:0/1785634469 >> v2:172.21.2.112:3300/0 conn(0x7fa0941b0470 0x7fa0941b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:25:31.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:31.214 7fa093fff700 1 --2- 172.21.2.112:0/1785634469 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa0940ef610 0x7fa0940efa10 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7fa084006650 tx=0x7fa084009a80).ready entity=mon.2 client_cookie=42fce90bc85d8b22 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:31.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:31.215 7fa0937fe700 1 -- 172.21.2.112:0/1785634469 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa0840009b0 con 0x7fa0940ef610 2021-02-10T11:25:31.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:31.215 7fa0937fe700 1 -- 172.21.2.112:0/1785634469 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa08400c0c0 con 0x7fa0940ef610 2021-02-10T11:25:31.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:31.215 7fa09ab70700 1 -- 172.21.2.112:0/1785634469 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa0940ef610 msgr2=0x7fa0940efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:31.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:31.215 7fa09ab70700 1 --2- 172.21.2.112:0/1785634469 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa0940ef610 0x7fa0940efa10 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7fa084006650 tx=0x7fa084009a80).stop 2021-02-10T11:25:31.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:31.215 7fa09ab70700 1 -- 172.21.2.112:0/1785634469 shutdown_connections 2021-02-10T11:25:31.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:31.215 7fa09ab70700 1 --2- 172.21.2.112:0/1785634469 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa0940ef610 0x7fa0940efa10 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:31.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:31.215 7fa09ab70700 1 --2- 172.21.2.112:0/1785634469 >> v2:172.21.2.112:3300/0 conn(0x7fa0941b0470 0x7fa0941b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:31.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:31.215 7fa09ab70700 1 -- 172.21.2.112:0/1785634469 >> 172.21.2.112:0/1785634469 conn(0x7fa0940ce3e0 msgr2=0x7fa0940e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:31.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:31.215 7fa09ab70700 1 -- 172.21.2.112:0/1785634469 shutdown_connections 2021-02-10T11:25:31.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:31.215 7fa09ab70700 1 -- 172.21.2.112:0/1785634469 wait complete. 2021-02-10T11:25:31.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:31.230 7fa09ab70700 1 Processor -- start 2021-02-10T11:25:31.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:31.230 7fa09ab70700 1 -- start start 2021-02-10T11:25:31.224 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:31.230 7fa09ab70700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa0940a65f0 0x7fa0940a5d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:31.224 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:31.230 7fa09ab70700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa0940a78f0 0x7fa0940a7cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:31.224 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:31.230 7fa09ab70700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa0940a8a20 0x7fa0940a8e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:31.224 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:31.230 7fa09ab70700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0940a84a0 con 0x7fa0940a78f0 2021-02-10T11:25:31.224 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:31.230 7fa09ab70700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0940a6db0 con 0x7fa0940a65f0 2021-02-10T11:25:31.224 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:31.230 7fa09ab70700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0940a69f0 con 0x7fa0940a8a20 2021-02-10T11:25:31.224 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:31.230 7fa093fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa0940a78f0 0x7fa0940a7cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:31.224 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:31.230 7fa093fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa0940a78f0 0x7fa0940a7cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:45724/0 (socket says 172.21.2.112:45724) 2021-02-10T11:25:31.224 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:31.230 7fa093fff700 1 -- 172.21.2.112:0/3756815086 learned_addr learned my addr 172.21.2.112:0/3756815086 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:25:31.224 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:31.230 7fa09918d700 1 --2- 172.21.2.112:0/3756815086 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa0940a8a20 0x7fa0940a8e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:31.224 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:31.231 7fa09898c700 1 --2- 172.21.2.112:0/3756815086 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa0940a65f0 0x7fa0940a5d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:31.224 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:31.231 7fa093fff700 1 -- 172.21.2.112:0/3756815086 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa0940a8a20 msgr2=0x7fa0940a8e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:31.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:31.231 7fa093fff700 1 --2- 172.21.2.112:0/3756815086 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa0940a8a20 0x7fa0940a8e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:31.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:31.231 7fa093fff700 1 -- 172.21.2.112:0/3756815086 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa0940a65f0 msgr2=0x7fa0940a5d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:31.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:31.231 7fa093fff700 1 --2- 172.21.2.112:0/3756815086 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa0940a65f0 0x7fa0940a5d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:31.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:31.231 7fa093fff700 1 -- 172.21.2.112:0/3756815086 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0940a9440 con 0x7fa0940a78f0 2021-02-10T11:25:31.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:31.231 7fa093fff700 1 --2- 172.21.2.112:0/3756815086 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa0940a78f0 0x7fa0940a7cf0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 rx=0x7fa084009650 tx=0x7fa08400b8e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:31.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:31.231 7fa0917fa700 1 -- 172.21.2.112:0/3756815086 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa0840093a0 con 0x7fa0940a78f0 2021-02-10T11:25:31.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:31.231 7fa0917fa700 1 -- 172.21.2.112:0/3756815086 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa084006100 con 0x7fa0940a78f0 2021-02-10T11:25:31.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:31.231 7fa0917fa700 1 -- 172.21.2.112:0/3756815086 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa0840052e0 con 0x7fa0940a78f0 2021-02-10T11:25:31.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:31.231 7fa09ab70700 1 -- 172.21.2.112:0/3756815086 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0940a9440 con 0x7fa0940a78f0 2021-02-10T11:25:31.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:31.231 7fa09ab70700 1 -- 172.21.2.112:0/3756815086 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0940a99d0 con 0x7fa0940a78f0 2021-02-10T11:25:31.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:31.232 7fa0917fa700 1 -- 172.21.2.112:0/3756815086 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa0840052e0 con 0x7fa0940a78f0 2021-02-10T11:25:31.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:31.232 7fa0917fa700 1 --2- 172.21.2.112:0/3756815086 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa0780186c0 0x7fa07801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:31.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:31.232 7fa0917fa700 1 -- 172.21.2.112:0/3756815086 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fa0840327e0 con 0x7fa0940a78f0 2021-02-10T11:25:31.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:31.232 7fa09898c700 1 --2- 172.21.2.112:0/3756815086 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa0780186c0 0x7fa07801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:31.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:31.233 7fa09898c700 1 --2- 172.21.2.112:0/3756815086 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa0780186c0 0x7fa07801abc0 crc :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:31.253 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:31.263 7fa09ab70700 1 -- 172.21.2.112:0/3756815086 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa094105ae0 con 0x7fa0940a78f0 2021-02-10T11:25:31.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:31.266 7fa0917fa700 1 -- 172.21.2.112:0/3756815086 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa0840093a0 con 0x7fa0940a78f0 2021-02-10T11:25:31.383 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:31.393 7fa09ab70700 1 -- 172.21.2.112:0/3756815086 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348636", "target": ["mgr", ""]}) v1 -- 0x7fa0940a9440 con 0x7fa0780186c0 2021-02-10T11:25:31.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:31.396 7fa0917fa700 1 -- 172.21.2.112:0/3756815086 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7fa07c006a30 con 0x7fa0780186c0 2021-02-10T11:25:31.391 INFO:teuthology.orchestra.run.gibba012.stdout:/volumes/_nogroup/subvolume_0000000000348636/e1d6d37c-ee3a-41cb-a328-5ab19ff0e2f7 2021-02-10T11:25:31.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:31.402 7fa09ab70700 1 -- 172.21.2.112:0/3756815086 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa0780186c0 msgr2=0x7fa07801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:31.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:31.402 7fa09ab70700 1 --2- 172.21.2.112:0/3756815086 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa0780186c0 0x7fa07801abc0 crc :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:31.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:31.402 7fa09ab70700 1 -- 172.21.2.112:0/3756815086 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa0940a78f0 msgr2=0x7fa0940a7cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:31.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:31.402 7fa09ab70700 1 --2- 172.21.2.112:0/3756815086 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa0940a78f0 0x7fa0940a7cf0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 rx=0x7fa084009650 tx=0x7fa08400b8e0).stop 2021-02-10T11:25:31.393 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:31.402 7fa09ab70700 1 -- 172.21.2.112:0/3756815086 shutdown_connections 2021-02-10T11:25:31.393 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:31.402 7fa09ab70700 1 --2- 172.21.2.112:0/3756815086 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa0780186c0 0x7fa07801abc0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:31.393 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:31.402 7fa09ab70700 1 --2- 172.21.2.112:0/3756815086 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa0940a8a20 0x7fa0940a8e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:31.393 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:31.402 7fa09ab70700 1 --2- 172.21.2.112:0/3756815086 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa0940a78f0 0x7fa0940a7cf0 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:31.393 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:31.402 7fa09ab70700 1 --2- 172.21.2.112:0/3756815086 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa0940a65f0 0x7fa0940a5d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:31.393 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:31.402 7fa09ab70700 1 -- 172.21.2.112:0/3756815086 >> 172.21.2.112:0/3756815086 conn(0x7fa0940a2800 msgr2=0x7fa094105da0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:31.393 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:31.402 7fa09ab70700 1 -- 172.21.2.112:0/3756815086 shutdown_connections 2021-02-10T11:25:31.393 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:31.402 7fa09ab70700 1 -- 172.21.2.112:0/3756815086 wait complete. 2021-02-10T11:25:31.400 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000348636 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000348636/e1d6d37c-ee3a-41cb-a328-5ab19ff0e2f7 2021-02-10T11:25:31.401 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:31.401 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:31.401 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:31.736 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:25:31.736 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:31.736 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348636/e1d6d37c-ee3a-41cb-a328-5ab19ff0e2f7/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:32.117 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:25:32.118 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:25:32.118 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0248001 s, 42.3 MB/s 2021-02-10T11:25:32.119 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:32.119 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:32.119 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:32.455 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:25:32.455 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:32.455 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348636/e1d6d37c-ee3a-41cb-a328-5ab19ff0e2f7/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:32.831 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:25:32.831 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:25:32.831 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0238525 s, 44.0 MB/s 2021-02-10T11:25:32.832 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:32.832 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:32.832 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:33.168 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:25:33.168 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:33.169 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348636/e1d6d37c-ee3a-41cb-a328-5ab19ff0e2f7/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:33.544 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:25:33.544 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:25:33.544 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0239212 s, 43.8 MB/s 2021-02-10T11:25:33.545 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:33.545 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:33.545 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:33.880 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:25:33.880 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:33.880 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348636/e1d6d37c-ee3a-41cb-a328-5ab19ff0e2f7/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:34.256 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:25:34.257 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:25:34.257 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0252382 s, 41.5 MB/s 2021-02-10T11:25:34.258 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:34.258 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:34.258 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:34.593 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:25:34.593 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:34.594 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348636/e1d6d37c-ee3a-41cb-a328-5ab19ff0e2f7/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:34.966 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:25:34.967 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:25:34.967 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0249526 s, 42.0 MB/s 2021-02-10T11:25:34.968 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:34.968 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:34.968 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:35.303 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:25:35.303 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:35.303 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348636/e1d6d37c-ee3a-41cb-a328-5ab19ff0e2f7/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:35.677 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:25:35.678 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:25:35.678 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0229462 s, 45.7 MB/s 2021-02-10T11:25:35.679 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:35.679 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:35.679 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:36.015 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:25:36.015 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:36.015 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348636/e1d6d37c-ee3a-41cb-a328-5ab19ff0e2f7/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:36.390 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:25:36.391 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:25:36.391 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0247767 s, 42.3 MB/s 2021-02-10T11:25:36.392 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:36.392 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:36.392 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:36.728 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:25:36.728 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:36.729 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348636/e1d6d37c-ee3a-41cb-a328-5ab19ff0e2f7/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:37.104 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:25:37.104 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:25:37.104 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0255656 s, 41.0 MB/s 2021-02-10T11:25:37.105 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:37.106 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:37.106 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:37.441 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:25:37.442 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:37.442 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348636/e1d6d37c-ee3a-41cb-a328-5ab19ff0e2f7/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:37.818 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:25:37.818 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:25:37.818 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0243765 s, 43.0 MB/s 2021-02-10T11:25:37.819 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:37.819 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:37.819 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:38.153 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:25:38.154 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:38.154 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348636/e1d6d37c-ee3a-41cb-a328-5ab19ff0e2f7/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:38.529 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:25:38.529 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:25:38.529 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0242301 s, 43.3 MB/s 2021-02-10T11:25:38.530 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000348637 2021-02-10T11:25:38.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:38.957 7fdd6024f700 1 Processor -- start 2021-02-10T11:25:38.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:38.957 7fdd6024f700 1 -- start start 2021-02-10T11:25:38.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:38.958 7fdd6024f700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdd581b0470 0x7fdd581b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:38.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:38.958 7fdd6024f700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fdd580ef610 0x7fdd580efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:38.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:38.958 7fdd6024f700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fdd580f0030 0x7fdd580f0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:38.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:38.958 7fdd6024f700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd580e5ea0 con 0x7fdd580ef610 2021-02-10T11:25:38.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:38.958 7fdd6024f700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd580f0b50 con 0x7fdd580f0030 2021-02-10T11:25:38.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:38.958 7fdd6024f700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd580e52e0 con 0x7fdd581b0470 2021-02-10T11:25:38.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:38.958 7fdd5d86a700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fdd580ef610 0x7fdd580efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:38.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:38.958 7fdd5d86a700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fdd580ef610 0x7fdd580efa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:45732/0 (socket says 172.21.2.112:45732) 2021-02-10T11:25:38.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:38.958 7fdd5d86a700 1 -- 172.21.2.112:0/4024150249 learned_addr learned my addr 172.21.2.112:0/4024150249 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:25:38.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:38.958 7fdd5d86a700 1 -- 172.21.2.112:0/4024150249 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdd581b0470 msgr2=0x7fdd581b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:38.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:38.958 7fdd5d86a700 1 --2- 172.21.2.112:0/4024150249 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdd581b0470 0x7fdd581b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:38.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:38.958 7fdd5d86a700 1 -- 172.21.2.112:0/4024150249 >> v2:172.21.2.120:3300/0 conn(0x7fdd580f0030 msgr2=0x7fdd580f0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:38.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:38.958 7fdd5d86a700 1 --2- 172.21.2.112:0/4024150249 >> v2:172.21.2.120:3300/0 conn(0x7fdd580f0030 0x7fdd580f0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:38.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:38.958 7fdd5d86a700 1 -- 172.21.2.112:0/4024150249 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd580f10d0 con 0x7fdd580ef610 2021-02-10T11:25:38.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:38.958 7fdd5d86a700 1 --2- 172.21.2.112:0/4024150249 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdd580ef610 0x7fdd580efa10 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 rx=0x7fdd50006700 tx=0x7fdd50009b80).ready entity=mon.0 client_cookie=38a99b7e2b996b7d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:38.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:38.959 7fdd5d069700 1 -- 172.21.2.112:0/4024150249 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdd5000c0c0 con 0x7fdd580ef610 2021-02-10T11:25:38.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:38.959 7fdd5d069700 1 -- 172.21.2.112:0/4024150249 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd5000c440 con 0x7fdd580ef610 2021-02-10T11:25:38.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:38.959 7fdd5d069700 1 -- 172.21.2.112:0/4024150249 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdd5000c960 con 0x7fdd580ef610 2021-02-10T11:25:38.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:38.959 7fdd6024f700 1 -- 172.21.2.112:0/4024150249 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdd580ef610 msgr2=0x7fdd580efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:38.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:38.959 7fdd6024f700 1 --2- 172.21.2.112:0/4024150249 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdd580ef610 0x7fdd580efa10 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 rx=0x7fdd50006700 tx=0x7fdd50009b80).stop 2021-02-10T11:25:38.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:38.960 7fdd6024f700 1 -- 172.21.2.112:0/4024150249 shutdown_connections 2021-02-10T11:25:38.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:38.960 7fdd6024f700 1 --2- 172.21.2.112:0/4024150249 >> v2:172.21.2.120:3300/0 conn(0x7fdd580f0030 0x7fdd580f0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:38.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:38.960 7fdd6024f700 1 --2- 172.21.2.112:0/4024150249 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdd580ef610 0x7fdd580efa10 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:38.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:38.960 7fdd6024f700 1 --2- 172.21.2.112:0/4024150249 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdd581b0470 0x7fdd581b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:38.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:38.960 7fdd6024f700 1 -- 172.21.2.112:0/4024150249 >> 172.21.2.112:0/4024150249 conn(0x7fdd580ce3e0 msgr2=0x7fdd580e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:38.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:38.960 7fdd6024f700 1 -- 172.21.2.112:0/4024150249 shutdown_connections 2021-02-10T11:25:38.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:38.960 7fdd6024f700 1 -- 172.21.2.112:0/4024150249 wait complete. 2021-02-10T11:25:38.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:38.974 7fdd6024f700 1 Processor -- start 2021-02-10T11:25:38.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:38.974 7fdd6024f700 1 -- start start 2021-02-10T11:25:38.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:38.975 7fdd6024f700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fdd58193230 0x7fdd580a6a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:38.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:38.975 7fdd6024f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdd5808f5b0 0x7fdd5808f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:38.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:38.975 7fdd6024f700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdd5808e6a0 0x7fdd5808eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:38.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:38.975 7fdd6024f700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd5808ffd0 con 0x7fdd5808f5b0 2021-02-10T11:25:38.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:38.975 7fdd6024f700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd58090240 con 0x7fdd58193230 2021-02-10T11:25:38.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:38.975 7fdd6024f700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd5808f0c0 con 0x7fdd5808e6a0 2021-02-10T11:25:38.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:38.975 7fdd5d86a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdd5808f5b0 0x7fdd5808f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:38.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:38.975 7fdd5d86a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdd5808f5b0 0x7fdd5808f9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:45738/0 (socket says 172.21.2.112:45738) 2021-02-10T11:25:38.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:38.975 7fdd5d86a700 1 -- 172.21.2.112:0/2411169641 learned_addr learned my addr 172.21.2.112:0/2411169641 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:25:38.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:38.975 7fdd5d86a700 1 -- 172.21.2.112:0/2411169641 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdd5808e6a0 msgr2=0x7fdd5808eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:38.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:38.975 7fdd5d86a700 1 --2- 172.21.2.112:0/2411169641 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdd5808e6a0 0x7fdd5808eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:38.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:38.975 7fdd5d86a700 1 -- 172.21.2.112:0/2411169641 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fdd58193230 msgr2=0x7fdd580a6a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:38.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:38.975 7fdd5d86a700 1 --2- 172.21.2.112:0/2411169641 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fdd58193230 0x7fdd580a6a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:38.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:38.975 7fdd5d86a700 1 -- 172.21.2.112:0/2411169641 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd5808f2a0 con 0x7fdd5808f5b0 2021-02-10T11:25:38.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:38.975 7fdd5d86a700 1 --2- 172.21.2.112:0/2411169641 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdd5808f5b0 0x7fdd5808f9b0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 rx=0x7fdd5000b650 tx=0x7fdd5000b9f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:38.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:38.976 7fdd56ffd700 1 -- 172.21.2.112:0/2411169641 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdd5000bde0 con 0x7fdd5808f5b0 2021-02-10T11:25:38.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:38.976 7fdd56ffd700 1 -- 172.21.2.112:0/2411169641 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd500091e0 con 0x7fdd5808f5b0 2021-02-10T11:25:38.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:38.976 7fdd6024f700 1 -- 172.21.2.112:0/2411169641 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd5808f2a0 con 0x7fdd5808f5b0 2021-02-10T11:25:38.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:38.976 7fdd6024f700 1 -- 172.21.2.112:0/2411169641 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd581924f0 con 0x7fdd5808f5b0 2021-02-10T11:25:38.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:38.976 7fdd56ffd700 1 -- 172.21.2.112:0/2411169641 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdd500022b0 con 0x7fdd5808f5b0 2021-02-10T11:25:38.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:38.976 7fdd56ffd700 1 -- 172.21.2.112:0/2411169641 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fdd50000c90 con 0x7fdd5808f5b0 2021-02-10T11:25:38.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:38.976 7fdd56ffd700 1 --2- 172.21.2.112:0/2411169641 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fdd38018730 0x7fdd3801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:38.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:38.976 7fdd56ffd700 1 -- 172.21.2.112:0/2411169641 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fdd50032790 con 0x7fdd5808f5b0 2021-02-10T11:25:38.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:38.977 7fdd5e06b700 1 --2- 172.21.2.112:0/2411169641 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fdd38018730 0x7fdd3801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:38.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:38.977 7fdd5e06b700 1 --2- 172.21.2.112:0/2411169641 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fdd38018730 0x7fdd3801ac10 crc :-1 s=READY pgs=313 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:38.997 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:39.007 7fdd6024f700 1 -- 172.21.2.112:0/2411169641 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd580f0b50 con 0x7fdd5808f5b0 2021-02-10T11:25:39.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:39.010 7fdd56ffd700 1 -- 172.21.2.112:0/2411169641 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fdd50000990 con 0x7fdd5808f5b0 2021-02-10T11:25:39.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:39.139 7fdd6024f700 1 -- 172.21.2.112:0/2411169641 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348637", "target": ["mgr", ""]}) v1 -- 0x7fdd5808ffd0 con 0x7fdd38018730 2021-02-10T11:25:39.154 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:39.164 7fdd56ffd700 1 -- 172.21.2.112:0/2411169641 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fdd4c002080 con 0x7fdd38018730 2021-02-10T11:25:39.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:39.172 7fdd6024f700 1 -- 172.21.2.112:0/2411169641 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fdd38018730 msgr2=0x7fdd3801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:39.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:39.172 7fdd6024f700 1 --2- 172.21.2.112:0/2411169641 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fdd38018730 0x7fdd3801ac10 crc :-1 s=READY pgs=313 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:39.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:39.172 7fdd6024f700 1 -- 172.21.2.112:0/2411169641 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdd5808f5b0 msgr2=0x7fdd5808f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:39.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:39.172 7fdd6024f700 1 --2- 172.21.2.112:0/2411169641 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdd5808f5b0 0x7fdd5808f9b0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 rx=0x7fdd5000b650 tx=0x7fdd5000b9f0).stop 2021-02-10T11:25:39.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:39.172 7fdd6024f700 1 -- 172.21.2.112:0/2411169641 shutdown_connections 2021-02-10T11:25:39.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:39.172 7fdd6024f700 1 --2- 172.21.2.112:0/2411169641 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fdd38018730 0x7fdd3801ac10 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:39.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:39.172 7fdd6024f700 1 --2- 172.21.2.112:0/2411169641 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdd5808e6a0 0x7fdd5808eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:39.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:39.172 7fdd6024f700 1 --2- 172.21.2.112:0/2411169641 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdd5808f5b0 0x7fdd5808f9b0 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:39.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:39.172 7fdd6024f700 1 --2- 172.21.2.112:0/2411169641 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fdd58193230 0x7fdd580a6a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:39.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:39.172 7fdd6024f700 1 -- 172.21.2.112:0/2411169641 >> 172.21.2.112:0/2411169641 conn(0x7fdd580bf4e0 msgr2=0x7fdd580be160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:39.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:39.172 7fdd6024f700 1 -- 172.21.2.112:0/2411169641 shutdown_connections 2021-02-10T11:25:39.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:39.172 7fdd6024f700 1 -- 172.21.2.112:0/2411169641 wait complete. 2021-02-10T11:25:39.170 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000348637 2021-02-10T11:25:39.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:39.595 7ff557dc3700 1 Processor -- start 2021-02-10T11:25:39.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:39.596 7ff557dc3700 1 -- start start 2021-02-10T11:25:39.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:39.596 7ff557dc3700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff5500f05f0 0x7ff5500f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:39.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:39.596 7ff557dc3700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5501b0c70 con 0x7ff5501b0470 2021-02-10T11:25:39.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:39.596 7ff557dc3700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5500f0b50 con 0x7ff5501b0870 2021-02-10T11:25:39.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:39.596 7ff557dc3700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5500e5ea0 con 0x7ff5500f05f0 2021-02-10T11:25:39.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:39.596 7ff555bdf700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7ff5501b0470 0x7ff5500ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:45208/0 (socket says 172.21.2.112:45208) 2021-02-10T11:25:39.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:39.596 7ff555bdf700 1 -- 172.21.2.112:0/575828839 learned_addr learned my addr 172.21.2.112:0/575828839 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:25:39.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:39.596 7ff5563e0700 1 --2- 172.21.2.112:0/575828839 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff5500f05f0 0x7ff5500f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:39.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:39.597 7ff554bdd700 1 -- 172.21.2.112:0/575828839 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4259772972 0 0) 0x7ff5440010b0 con 0x7ff5501b0470 2021-02-10T11:25:39.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:39.597 7ff554bdd700 1 -- 172.21.2.112:0/575828839 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff53c002fe0 con 0x7ff5501b0470 2021-02-10T11:25:39.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:39.597 7ff554bdd700 1 -- 172.21.2.112:0/575828839 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 775498931 0 0) 0x7ff5440010b0 con 0x7ff5501b0470 2021-02-10T11:25:39.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:39.597 7ff554bdd700 1 -- 172.21.2.112:0/575828839 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff5500f05f0 msgr2=0x7ff5500f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:39.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:39.597 7ff554bdd700 1 --2- 172.21.2.112:0/575828839 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff5500f05f0 0x7ff5500f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:39.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:39.597 7ff554bdd700 1 -- 172.21.2.112:0/575828839 >> v1:172.21.2.120:6789/0 conn(0x7ff5501b0870 legacy=0x7ff5500efe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:39.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:39.597 7ff554bdd700 1 -- 172.21.2.112:0/575828839 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5500e52e0 con 0x7ff5501b0470 2021-02-10T11:25:39.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:39.597 7ff554bdd700 1 -- 172.21.2.112:0/575828839 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7ff5440015b0 con 0x7ff5501b0470 2021-02-10T11:25:39.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:39.597 7ff5563e0700 1 --2- 172.21.2.112:0/575828839 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff5500f05f0 0x7ff5500f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:25:39.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:39.597 7ff554bdd700 1 -- 172.21.2.112:0/575828839 >> v1:172.21.2.112:6789/0 conn(0x7ff5501b0470 legacy=0x7ff5500ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:39.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:39.597 7ff554bdd700 1 --2- 172.21.2.112:0/575828839 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff53c003a60 0x7ff53c005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:39.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:39.597 7ff554bdd700 1 --2- 172.21.2.112:0/575828839 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff53c006790 0x7ff53c008cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:39.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:39.597 7ff554bdd700 1 --2- 172.21.2.112:0/575828839 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff53c009590 0x7ff53c00ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:39.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:39.597 7ff554bdd700 1 -- 172.21.2.112:0/575828839 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff53c00c2d0 con 0x7ff53c003a60 2021-02-10T11:25:39.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:39.597 7ff554bdd700 1 -- 172.21.2.112:0/575828839 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff53c00c4f0 con 0x7ff53c006790 2021-02-10T11:25:39.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:39.597 7ff554bdd700 1 -- 172.21.2.112:0/575828839 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff53c00c710 con 0x7ff53c009590 2021-02-10T11:25:39.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:39.597 7ff555bdf700 1 --2- 172.21.2.112:0/575828839 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff53c003a60 0x7ff53c005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:39.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:39.597 7ff5553de700 1 --2- 172.21.2.112:0/575828839 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff53c006790 0x7ff53c008cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:39.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:39.597 7ff5563e0700 1 --2- 172.21.2.112:0/575828839 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff53c009590 0x7ff53c00ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:39.606 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:39.597 7ff555bdf700 1 -- 172.21.2.112:0/575828839 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff53c009590 msgr2=0x7ff53c00ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:39.606 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:39.597 7ff555bdf700 1 --2- 172.21.2.112:0/575828839 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff53c009590 0x7ff53c00ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:39.606 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:39.598 7ff555bdf700 1 -- 172.21.2.112:0/575828839 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff53c006790 msgr2=0x7ff53c008cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:39.606 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:39.598 7ff555bdf700 1 --2- 172.21.2.112:0/575828839 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff53c006790 0x7ff53c008cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:39.606 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:39.598 7ff555bdf700 1 -- 172.21.2.112:0/575828839 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff53c00c9b0 con 0x7ff53c003a60 2021-02-10T11:25:39.606 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:39.598 7ff5553de700 1 --2- 172.21.2.112:0/575828839 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff53c006790 0x7ff53c008cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:25:39.606 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:39.598 7ff555bdf700 1 --2- 172.21.2.112:0/575828839 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff53c003a60 0x7ff53c005ef0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 rx=0x7ff54404b940 tx=0x7ff54404bce0).ready entity=mon.0 client_cookie=d5592406c9376a5b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:39.606 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:39.598 7ff554bdd700 1 -- 172.21.2.112:0/575828839 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff54404e310 con 0x7ff53c003a60 2021-02-10T11:25:39.606 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:39.598 7ff554bdd700 1 -- 172.21.2.112:0/575828839 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff54404be20 con 0x7ff53c003a60 2021-02-10T11:25:39.606 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:39.598 7ff554bdd700 1 -- 172.21.2.112:0/575828839 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff54404e8d0 con 0x7ff53c003a60 2021-02-10T11:25:39.606 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:39.598 7ff557dc3700 1 -- 172.21.2.112:0/575828839 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff53c003a60 msgr2=0x7ff53c005ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:39.606 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:39.598 7ff557dc3700 1 --2- 172.21.2.112:0/575828839 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff53c003a60 0x7ff53c005ef0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 rx=0x7ff54404b940 tx=0x7ff54404bce0).stop 2021-02-10T11:25:39.607 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:39.598 7ff5563e0700 1 -- 172.21.2.112:0/575828839 reap_dead start 2021-02-10T11:25:39.607 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:39.599 7ff557dc3700 1 -- 172.21.2.112:0/575828839 shutdown_connections 2021-02-10T11:25:39.607 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:39.599 7ff557dc3700 1 -- 172.21.2.112:0/575828839 >> 172.21.2.112:0/575828839 conn(0x7ff5500ce3e0 msgr2=0x7ff5500e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:39.607 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:39.599 7ff557dc3700 1 -- 172.21.2.112:0/575828839 shutdown_connections 2021-02-10T11:25:39.607 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:39.599 7ff557dc3700 1 -- 172.21.2.112:0/575828839 wait complete. 2021-02-10T11:25:39.607 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:39.613 7ff557dc3700 1 Processor -- start 2021-02-10T11:25:39.607 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:39.613 7ff557dc3700 1 -- start start 2021-02-10T11:25:39.608 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:39.614 7ff557dc3700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff5500a86f0 0x7ff5501a7440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:39.609 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:39.614 7ff557dc3700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff5500ad9d0 0x7ff5501a7a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:39.609 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:39.614 7ff557dc3700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff5500965f0 0x7ff5500969f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:39.609 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:39.614 7ff557dc3700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5501a8100 con 0x7ff5500a86f0 2021-02-10T11:25:39.609 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:39.614 7ff557dc3700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff550097010 con 0x7ff5500ad9d0 2021-02-10T11:25:39.609 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:39.614 7ff557dc3700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff550097230 con 0x7ff5500965f0 2021-02-10T11:25:39.609 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:39.614 7ff555bdf700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff5500a86f0 0x7ff5501a7440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:39.609 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:39.614 7ff555bdf700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff5500a86f0 0x7ff5501a7440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:45760/0 (socket says 172.21.2.112:45760) 2021-02-10T11:25:39.609 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:39.614 7ff555bdf700 1 -- 172.21.2.112:0/4054781954 learned_addr learned my addr 172.21.2.112:0/4054781954 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:25:39.609 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:39.614 7ff5563e0700 1 --2- 172.21.2.112:0/4054781954 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff5500965f0 0x7ff5500969f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:39.609 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:39.614 7ff5553de700 1 --2- 172.21.2.112:0/4054781954 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff5500ad9d0 0x7ff5501a7a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:39.609 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:39.614 7ff5553de700 1 -- 172.21.2.112:0/4054781954 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff5500965f0 msgr2=0x7ff5500969f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:39.609 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:39.614 7ff5553de700 1 --2- 172.21.2.112:0/4054781954 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff5500965f0 0x7ff5500969f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:39.610 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:39.614 7ff5553de700 1 -- 172.21.2.112:0/4054781954 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff5500a86f0 msgr2=0x7ff5501a7440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:39.610 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:39.614 7ff5553de700 1 --2- 172.21.2.112:0/4054781954 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff5500a86f0 0x7ff5501a7440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:39.610 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:39.614 7ff5553de700 1 -- 172.21.2.112:0/4054781954 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff550097450 con 0x7ff5500ad9d0 2021-02-10T11:25:39.610 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:39.614 7ff555bdf700 1 --2- 172.21.2.112:0/4054781954 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff5500a86f0 0x7ff5501a7440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:25:39.610 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:39.615 7ff5553de700 1 --2- 172.21.2.112:0/4054781954 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff5500ad9d0 0x7ff5501a7a60 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7ff54c0068f0 tx=0x7ff54c00b910).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:39.610 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:39.615 7ff5427fc700 1 -- 172.21.2.112:0/4054781954 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff54c00bd00 con 0x7ff5500ad9d0 2021-02-10T11:25:39.610 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:39.615 7ff5427fc700 1 -- 172.21.2.112:0/4054781954 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff54c006950 con 0x7ff5500ad9d0 2021-02-10T11:25:39.610 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:39.615 7ff5427fc700 1 -- 172.21.2.112:0/4054781954 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff54c002220 con 0x7ff5500ad9d0 2021-02-10T11:25:39.610 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:39.615 7ff557dc3700 1 -- 172.21.2.112:0/4054781954 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff550097450 con 0x7ff5500ad9d0 2021-02-10T11:25:39.610 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:39.615 7ff557dc3700 1 -- 172.21.2.112:0/4054781954 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff550097450 con 0x7ff5500ad9d0 2021-02-10T11:25:39.610 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:39.616 7ff5427fc700 1 -- 172.21.2.112:0/4054781954 <== mon.1 v2:172.21.2.120:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff54c000d20 con 0x7ff5500ad9d0 2021-02-10T11:25:39.610 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:39.616 7ff5427fc700 1 --2- 172.21.2.112:0/4054781954 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff5380186c0 0x7ff53801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:39.610 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:39.616 7ff5427fc700 1 -- 172.21.2.112:0/4054781954 <== mon.1 v2:172.21.2.120:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7ff54c032450 con 0x7ff5500ad9d0 2021-02-10T11:25:39.611 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:39.616 7ff555bdf700 1 --2- 172.21.2.112:0/4054781954 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff5380186c0 0x7ff53801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:39.611 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:39.616 7ff555bdf700 1 --2- 172.21.2.112:0/4054781954 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff5380186c0 0x7ff53801abc0 crc :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:39.636 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:39.646 7ff557dc3700 1 -- 172.21.2.112:0/4054781954 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff550105b30 con 0x7ff5500ad9d0 2021-02-10T11:25:39.639 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:39.649 7ff5427fc700 1 -- 172.21.2.112:0/4054781954 <== mon.1 v2:172.21.2.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff54c000ac0 con 0x7ff5500ad9d0 2021-02-10T11:25:39.770 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:39.780 7ff557dc3700 1 -- 172.21.2.112:0/4054781954 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348637", "target": ["mgr", ""]}) v1 -- 0x7ff550105b30 con 0x7ff5380186c0 2021-02-10T11:25:39.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:39.784 7ff5427fc700 1 -- 172.21.2.112:0/4054781954 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7ff54404b260 con 0x7ff5380186c0 2021-02-10T11:25:39.777 INFO:teuthology.orchestra.run.gibba012.stdout:/volumes/_nogroup/subvolume_0000000000348637/7a22793d-6d85-4414-8fa0-56f52518aea9 2021-02-10T11:25:39.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:39.789 7ff557dc3700 1 -- 172.21.2.112:0/4054781954 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff5380186c0 msgr2=0x7ff53801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:39.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:39.789 7ff557dc3700 1 --2- 172.21.2.112:0/4054781954 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff5380186c0 0x7ff53801abc0 crc :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:39.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:39.789 7ff557dc3700 1 -- 172.21.2.112:0/4054781954 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff5500ad9d0 msgr2=0x7ff5501a7a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:39.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:39.789 7ff557dc3700 1 --2- 172.21.2.112:0/4054781954 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff5500ad9d0 0x7ff5501a7a60 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7ff54c0068f0 tx=0x7ff54c00b910).stop 2021-02-10T11:25:39.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:39.790 7ff557dc3700 1 -- 172.21.2.112:0/4054781954 shutdown_connections 2021-02-10T11:25:39.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:39.790 7ff557dc3700 1 --2- 172.21.2.112:0/4054781954 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff5380186c0 0x7ff53801abc0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:39.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:39.790 7ff557dc3700 1 --2- 172.21.2.112:0/4054781954 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff5500ad9d0 0x7ff5501a7a60 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:39.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:39.790 7ff557dc3700 1 --2- 172.21.2.112:0/4054781954 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff5500965f0 0x7ff5500969f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:39.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:39.790 7ff557dc3700 1 --2- 172.21.2.112:0/4054781954 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff5500a86f0 0x7ff5501a7440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:39.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:39.790 7ff557dc3700 1 -- 172.21.2.112:0/4054781954 >> 172.21.2.112:0/4054781954 conn(0x7ff55009e580 msgr2=0x7ff550190630 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:39.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:39.790 7ff557dc3700 1 -- 172.21.2.112:0/4054781954 shutdown_connections 2021-02-10T11:25:39.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:39.790 7ff557dc3700 1 -- 172.21.2.112:0/4054781954 wait complete. 2021-02-10T11:25:39.788 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000348637 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000348637/7a22793d-6d85-4414-8fa0-56f52518aea9 2021-02-10T11:25:39.788 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:39.788 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:39.788 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:40.075 DEBUG:teuthology.orchestra.run.gibba012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:25:40.078 DEBUG:teuthology.orchestra.run.gibba020:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:25:40.125 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:25:40.125 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:40.125 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348637/7a22793d-6d85-4414-8fa0-56f52518aea9/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:40.462 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:25:40.462 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:25:40.462 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0311564 s, 33.7 MB/s 2021-02-10T11:25:40.463 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:40.463 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:40.463 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:40.838 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:25:40.838 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:40.838 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348637/7a22793d-6d85-4414-8fa0-56f52518aea9/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:41.213 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:25:41.213 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:25:41.213 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0240236 s, 43.6 MB/s 2021-02-10T11:25:41.214 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:41.214 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:41.214 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:41.550 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:25:41.550 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:41.550 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348637/7a22793d-6d85-4414-8fa0-56f52518aea9/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:41.929 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:25:41.929 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:25:41.930 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0267399 s, 39.2 MB/s 2021-02-10T11:25:41.930 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:41.931 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:41.931 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:42.267 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:25:42.267 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:42.267 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348637/7a22793d-6d85-4414-8fa0-56f52518aea9/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:42.642 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:25:42.642 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:25:42.642 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0233663 s, 44.9 MB/s 2021-02-10T11:25:42.643 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:42.643 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:42.643 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:42.979 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:25:42.980 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:42.980 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348637/7a22793d-6d85-4414-8fa0-56f52518aea9/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:43.354 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:25:43.354 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:25:43.354 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0235326 s, 44.6 MB/s 2021-02-10T11:25:43.355 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:43.355 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:43.355 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:43.693 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:25:43.693 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:43.693 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348637/7a22793d-6d85-4414-8fa0-56f52518aea9/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:44.069 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:25:44.069 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:25:44.069 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0241826 s, 43.4 MB/s 2021-02-10T11:25:44.070 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:44.070 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:44.070 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:44.405 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:25:44.405 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:44.405 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348637/7a22793d-6d85-4414-8fa0-56f52518aea9/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:44.778 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:25:44.778 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:25:44.778 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0246956 s, 42.5 MB/s 2021-02-10T11:25:44.779 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:44.779 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:44.779 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:45.115 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:25:45.115 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:45.115 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348637/7a22793d-6d85-4414-8fa0-56f52518aea9/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:45.490 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:25:45.491 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:25:45.491 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0229525 s, 45.7 MB/s 2021-02-10T11:25:45.492 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:45.492 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:45.492 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:45.828 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:25:45.828 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:45.828 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348637/7a22793d-6d85-4414-8fa0-56f52518aea9/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:46.204 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:25:46.204 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:25:46.204 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0237558 s, 44.1 MB/s 2021-02-10T11:25:46.205 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:46.205 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:46.205 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:46.540 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:25:46.540 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:46.541 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348637/7a22793d-6d85-4414-8fa0-56f52518aea9/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:46.918 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:25:46.918 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:25:46.918 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0250867 s, 41.8 MB/s 2021-02-10T11:25:46.919 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000348638 2021-02-10T11:25:47.352 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:47.346 7f542959b700 1 Processor -- start 2021-02-10T11:25:47.352 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:47.346 7f542959b700 1 -- start start 2021-02-10T11:25:47.352 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:47.346 7f542959b700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f54241b0470 0x7f54241b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:47.352 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:47.346 7f542959b700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f54240f20e0 0x7f54240f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:47.353 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:47.346 7f542959b700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54240f0b50 con 0x7f54240ef610 2021-02-10T11:25:47.353 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:47.346 7f542959b700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54240f0820 con 0x7f54240f20e0 2021-02-10T11:25:47.353 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:47.346 7f542959b700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54240f24e0 con 0x7f54241b0470 2021-02-10T11:25:47.353 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:47.346 7f5423ac0700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f54240f20e0 0x7f54240f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:47.353 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:47.346 7f5423ac0700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f54240f20e0 0x7f54240f0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:45766/0 (socket says 172.21.2.112:45766) 2021-02-10T11:25:47.353 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:47.346 7f5422abe700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f54240ef610 0x7f54240efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:45228/0 (socket says 172.21.2.112:45228) 2021-02-10T11:25:47.353 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:47.346 7f5423ac0700 1 -- 172.21.2.112:0/3086792378 learned_addr learned my addr 172.21.2.112:0/3086792378 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:25:47.353 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:47.347 7f54232bf700 1 --2- 172.21.2.112:0/3086792378 >> v2:172.21.2.120:3300/0 conn(0x7f54241b0470 0x7f54241b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:47.353 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:47.347 7f5423ac0700 1 -- 172.21.2.112:0/3086792378 >> v2:172.21.2.120:3300/0 conn(0x7f54241b0470 msgr2=0x7f54241b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:47.353 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:47.347 7f5423ac0700 1 --2- 172.21.2.112:0/3086792378 >> v2:172.21.2.120:3300/0 conn(0x7f54241b0470 0x7f54241b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:47.353 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:47.347 7f5423ac0700 1 -- 172.21.2.112:0/3086792378 >> v1:172.21.2.112:6789/0 conn(0x7f54240ef610 legacy=0x7f54240efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:47.353 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:47.347 7f5423ac0700 1 -- 172.21.2.112:0/3086792378 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54240e52e0 con 0x7f54240f20e0 2021-02-10T11:25:47.353 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:47.347 7f54232bf700 1 --2- 172.21.2.112:0/3086792378 >> v2:172.21.2.120:3300/0 conn(0x7f54241b0470 0x7f54241b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:25:47.354 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:47.347 7f5423ac0700 1 --2- 172.21.2.112:0/3086792378 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f54240f20e0 0x7f54240f0200 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 rx=0x7f541c006b80 tx=0x7f541c009b90).ready entity=mon.0 client_cookie=95e4a4cbd46e6c44 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:47.354 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:47.347 7f54222bd700 1 -- 172.21.2.112:0/3086792378 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f541c0022e0 con 0x7f54240f20e0 2021-02-10T11:25:47.354 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:47.347 7f54222bd700 1 -- 172.21.2.112:0/3086792378 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f541c002620 con 0x7f54240f20e0 2021-02-10T11:25:47.354 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:47.347 7f54222bd700 1 -- 172.21.2.112:0/3086792378 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f541c002b00 con 0x7f54240f20e0 2021-02-10T11:25:47.354 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:47.347 7f542959b700 1 -- 172.21.2.112:0/3086792378 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f54240f20e0 msgr2=0x7f54240f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:47.354 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:47.347 7f542959b700 1 --2- 172.21.2.112:0/3086792378 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f54240f20e0 0x7f54240f0200 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 rx=0x7f541c006b80 tx=0x7f541c009b90).stop 2021-02-10T11:25:47.354 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:47.348 7f542959b700 1 -- 172.21.2.112:0/3086792378 shutdown_connections 2021-02-10T11:25:47.354 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:47.348 7f542959b700 1 --2- 172.21.2.112:0/3086792378 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f54240f20e0 0x7f54240f0200 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:47.355 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:47.348 7f542959b700 1 --2- 172.21.2.112:0/3086792378 >> v2:172.21.2.120:3300/0 conn(0x7f54241b0470 0x7f54241b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:47.355 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:47.348 7f542959b700 1 -- 172.21.2.112:0/3086792378 >> 172.21.2.112:0/3086792378 conn(0x7f54240ce3e0 msgr2=0x7f54240e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:47.355 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:47.348 7f542959b700 1 -- 172.21.2.112:0/3086792378 shutdown_connections 2021-02-10T11:25:47.355 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:47.348 7f542959b700 1 -- 172.21.2.112:0/3086792378 wait complete. 2021-02-10T11:25:47.358 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:47.362 7f542959b700 1 Processor -- start 2021-02-10T11:25:47.358 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:47.363 7f542959b700 1 -- start start 2021-02-10T11:25:47.358 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:47.363 7f542959b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5424105d10 0x7f542409ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:47.358 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:47.363 7f542959b700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5424106110 0x7f54240b2a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:47.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:47.363 7f542959b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f542409e0a0 0x7f54240b3040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:47.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:47.363 7f542959b700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f542409e640 con 0x7f542409e0a0 2021-02-10T11:25:47.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:47.363 7f542959b700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54241a1620 con 0x7f5424106110 2021-02-10T11:25:47.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:47.363 7f542959b700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54240b3a70 con 0x7f5424105d10 2021-02-10T11:25:47.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:47.363 7f5423ac0700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f542409e0a0 0x7f54240b3040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:47.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:47.363 7f5423ac0700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f542409e0a0 0x7f54240b3040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:45772/0 (socket says 172.21.2.112:45772) 2021-02-10T11:25:47.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:47.363 7f5423ac0700 1 -- 172.21.2.112:0/1657448442 learned_addr learned my addr 172.21.2.112:0/1657448442 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:25:47.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:47.363 7f5423ac0700 1 -- 172.21.2.112:0/1657448442 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5424105d10 msgr2=0x7f542409ebc0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:25:47.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:47.363 7f5423ac0700 1 --2- 172.21.2.112:0/1657448442 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5424105d10 0x7f542409ebc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:47.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:47.363 7f5423ac0700 1 -- 172.21.2.112:0/1657448442 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5424106110 msgr2=0x7f54240b2a20 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:25:47.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:47.363 7f5423ac0700 1 --2- 172.21.2.112:0/1657448442 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5424106110 0x7f54240b2a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:47.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:47.363 7f5423ac0700 1 -- 172.21.2.112:0/1657448442 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54240b3660 con 0x7f542409e0a0 2021-02-10T11:25:47.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:47.363 7f5423ac0700 1 --2- 172.21.2.112:0/1657448442 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f542409e0a0 0x7f54240b3040 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 rx=0x7f541c009640 tx=0x7f541c00b8e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:47.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:47.364 7f541bfff700 1 -- 172.21.2.112:0/1657448442 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f541c009100 con 0x7f542409e0a0 2021-02-10T11:25:47.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:47.364 7f541bfff700 1 -- 172.21.2.112:0/1657448442 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f541c00bda0 con 0x7f542409e0a0 2021-02-10T11:25:47.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:47.364 7f542959b700 1 -- 172.21.2.112:0/1657448442 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54240b3660 con 0x7f542409e0a0 2021-02-10T11:25:47.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:47.364 7f541bfff700 1 -- 172.21.2.112:0/1657448442 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f541c006470 con 0x7f542409e0a0 2021-02-10T11:25:47.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:47.364 7f542959b700 1 -- 172.21.2.112:0/1657448442 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54240b3660 con 0x7f542409e0a0 2021-02-10T11:25:47.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:47.364 7f541bfff700 1 -- 172.21.2.112:0/1657448442 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f541c006240 con 0x7f542409e0a0 2021-02-10T11:25:47.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:47.364 7f541bfff700 1 --2- 172.21.2.112:0/1657448442 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f54080186c0 0x7f540801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:47.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:47.364 7f541bfff700 1 -- 172.21.2.112:0/1657448442 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f541c0325b0 con 0x7f542409e0a0 2021-02-10T11:25:47.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:47.365 7f54232bf700 1 --2- 172.21.2.112:0/1657448442 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f54080186c0 0x7f540801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:47.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:47.365 7f54232bf700 1 --2- 172.21.2.112:0/1657448442 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f54080186c0 0x7f540801abc0 crc :-1 s=READY pgs=315 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:47.386 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:47.396 7f542959b700 1 -- 172.21.2.112:0/1657448442 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54241a1620 con 0x7f542409e0a0 2021-02-10T11:25:47.389 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:47.399 7f541bfff700 1 -- 172.21.2.112:0/1657448442 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f541c02ef60 con 0x7f542409e0a0 2021-02-10T11:25:47.516 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:47.526 7f542959b700 1 -- 172.21.2.112:0/1657448442 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348638", "target": ["mgr", ""]}) v1 -- 0x7f542409e640 con 0x7f54080186c0 2021-02-10T11:25:47.542 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:47.552 7f541bfff700 1 -- 172.21.2.112:0/1657448442 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f540c000d30 con 0x7f54080186c0 2021-02-10T11:25:47.550 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:47.559 7f542959b700 1 -- 172.21.2.112:0/1657448442 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f54080186c0 msgr2=0x7f540801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:47.550 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:47.559 7f542959b700 1 --2- 172.21.2.112:0/1657448442 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f54080186c0 0x7f540801abc0 crc :-1 s=READY pgs=315 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:47.550 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:47.559 7f542959b700 1 -- 172.21.2.112:0/1657448442 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f542409e0a0 msgr2=0x7f54240b3040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:47.550 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:47.559 7f542959b700 1 --2- 172.21.2.112:0/1657448442 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f542409e0a0 0x7f54240b3040 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 rx=0x7f541c009640 tx=0x7f541c00b8e0).stop 2021-02-10T11:25:47.550 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:47.560 7f542959b700 1 -- 172.21.2.112:0/1657448442 shutdown_connections 2021-02-10T11:25:47.550 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:47.560 7f542959b700 1 --2- 172.21.2.112:0/1657448442 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f54080186c0 0x7f540801abc0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:47.550 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:47.560 7f542959b700 1 --2- 172.21.2.112:0/1657448442 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5424106110 0x7f54240b2a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:47.550 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:47.560 7f542959b700 1 --2- 172.21.2.112:0/1657448442 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f542409e0a0 0x7f54240b3040 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:47.550 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:47.560 7f542959b700 1 --2- 172.21.2.112:0/1657448442 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5424105d10 0x7f542409ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:47.550 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:47.560 7f542959b700 1 -- 172.21.2.112:0/1657448442 >> 172.21.2.112:0/1657448442 conn(0x7f54240c3c70 msgr2=0x7f54240bd640 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:47.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:47.560 7f542959b700 1 -- 172.21.2.112:0/1657448442 shutdown_connections 2021-02-10T11:25:47.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:47.560 7f542959b700 1 -- 172.21.2.112:0/1657448442 wait complete. 2021-02-10T11:25:47.558 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000348638 2021-02-10T11:25:47.992 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:47.985 7f83750a1700 1 Processor -- start 2021-02-10T11:25:47.992 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:47.985 7f83750a1700 1 -- start start 2021-02-10T11:25:47.992 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:47.985 7f83750a1700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f83700f05f0 0x7f83700f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:47.992 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:47.985 7f83750a1700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83701b0c70 con 0x7f83701b0470 2021-02-10T11:25:47.992 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:47.985 7f83750a1700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83700f0b50 con 0x7f83701b0870 2021-02-10T11:25:47.992 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:47.985 7f83750a1700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83700e5ea0 con 0x7f83700f05f0 2021-02-10T11:25:47.993 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:47.985 7f836ed9d700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f83701b0470 0x7f83700ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:45240/0 (socket says 172.21.2.112:45240) 2021-02-10T11:25:47.993 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:47.985 7f836ed9d700 1 -- 172.21.2.112:0/1498612135 learned_addr learned my addr 172.21.2.112:0/1498612135 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:25:47.993 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:47.986 7f836dd9b700 1 -- 172.21.2.112:0/1498612135 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2042369972 0 0) 0x7f8358001180 con 0x7f83701b0470 2021-02-10T11:25:47.993 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:47.986 7f836dd9b700 1 -- 172.21.2.112:0/1498612135 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f835c002fe0 con 0x7f83701b0470 2021-02-10T11:25:47.993 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:47.986 7f836f59e700 1 --2- 172.21.2.112:0/1498612135 >> v2:172.21.2.120:3300/0 conn(0x7f83700f05f0 0x7f83700f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:47.993 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:47.986 7f836dd9b700 1 -- 172.21.2.112:0/1498612135 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3401803737 0 0) 0x7f8358001180 con 0x7f83701b0470 2021-02-10T11:25:47.993 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:47.986 7f836dd9b700 1 -- 172.21.2.112:0/1498612135 >> v2:172.21.2.120:3300/0 conn(0x7f83700f05f0 msgr2=0x7f83700f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:47.993 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:47.986 7f836dd9b700 1 --2- 172.21.2.112:0/1498612135 >> v2:172.21.2.120:3300/0 conn(0x7f83700f05f0 0x7f83700f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:47.993 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:47.986 7f836dd9b700 1 -- 172.21.2.112:0/1498612135 >> v1:172.21.2.120:6789/0 conn(0x7f83701b0870 legacy=0x7f83700efe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:47.993 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:47.986 7f836dd9b700 1 -- 172.21.2.112:0/1498612135 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83700e52e0 con 0x7f83701b0470 2021-02-10T11:25:47.993 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:47.986 7f836dd9b700 1 -- 172.21.2.112:0/1498612135 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f8358001680 con 0x7f83701b0470 2021-02-10T11:25:47.993 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:47.986 7f836dd9b700 1 -- 172.21.2.112:0/1498612135 >> v1:172.21.2.112:6789/0 conn(0x7f83701b0470 legacy=0x7f83700ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:47.993 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:47.986 7f836dd9b700 1 --2- 172.21.2.112:0/1498612135 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f835c003b10 0x7f835c005f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:47.994 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:47.986 7f836dd9b700 1 --2- 172.21.2.112:0/1498612135 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f835c0067a0 0x7f835c008c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:47.994 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:47.986 7f836dd9b700 1 --2- 172.21.2.112:0/1498612135 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f835c009580 0x7f835c00baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:47.994 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:47.986 7f836dd9b700 1 -- 172.21.2.112:0/1498612135 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f835c00c2f0 con 0x7f835c003b10 2021-02-10T11:25:47.994 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:47.986 7f836dd9b700 1 -- 172.21.2.112:0/1498612135 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f835c00c510 con 0x7f835c009580 2021-02-10T11:25:47.994 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:47.986 7f836dd9b700 1 -- 172.21.2.112:0/1498612135 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f835c00c730 con 0x7f835c0067a0 2021-02-10T11:25:47.994 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:47.986 7f836ed9d700 1 --2- 172.21.2.112:0/1498612135 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f835c003b10 0x7f835c005f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:47.994 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:47.986 7f836ed9d700 1 -- 172.21.2.112:0/1498612135 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f835c0067a0 msgr2=0x7f835c008c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:47.994 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:47.986 7f836ed9d700 1 --2- 172.21.2.112:0/1498612135 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f835c0067a0 0x7f835c008c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:47.994 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:47.986 7f836ed9d700 1 -- 172.21.2.112:0/1498612135 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f835c009580 msgr2=0x7f835c00baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:47.994 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:47.986 7f836ed9d700 1 --2- 172.21.2.112:0/1498612135 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f835c009580 0x7f835c00baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:47.994 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:47.986 7f836ed9d700 1 -- 172.21.2.112:0/1498612135 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f835c0509a0 con 0x7f835c003b10 2021-02-10T11:25:47.994 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:47.986 7f836f59e700 1 -- 172.21.2.112:0/1498612135 reap_dead start 2021-02-10T11:25:47.995 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:47.986 7f836ed9d700 1 --2- 172.21.2.112:0/1498612135 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f835c003b10 0x7f835c005f20 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 rx=0x7f83580018c0 tx=0x7f8358007cf0).ready entity=mon.0 client_cookie=d3e2570b11fb3402 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:47.995 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:47.987 7f836dd9b700 1 -- 172.21.2.112:0/1498612135 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f83580014e0 con 0x7f835c003b10 2021-02-10T11:25:47.995 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:47.987 7f836dd9b700 1 -- 172.21.2.112:0/1498612135 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8358007150 con 0x7f835c003b10 2021-02-10T11:25:47.995 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:47.987 7f836dd9b700 1 -- 172.21.2.112:0/1498612135 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8358004590 con 0x7f835c003b10 2021-02-10T11:25:47.995 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:47.987 7f83750a1700 1 -- 172.21.2.112:0/1498612135 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f835c003b10 msgr2=0x7f835c005f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:47.995 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:47.987 7f83750a1700 1 --2- 172.21.2.112:0/1498612135 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f835c003b10 0x7f835c005f20 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 rx=0x7f83580018c0 tx=0x7f8358007cf0).stop 2021-02-10T11:25:47.995 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:47.987 7f83750a1700 1 -- 172.21.2.112:0/1498612135 shutdown_connections 2021-02-10T11:25:47.995 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:47.987 7f83750a1700 1 --2- 172.21.2.112:0/1498612135 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f835c003b10 0x7f835c005f20 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:47.995 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:47.987 7f83750a1700 1 -- 172.21.2.112:0/1498612135 >> 172.21.2.112:0/1498612135 conn(0x7f83700ce3e0 msgr2=0x7f83700e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:47.995 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:47.987 7f83750a1700 1 -- 172.21.2.112:0/1498612135 shutdown_connections 2021-02-10T11:25:47.995 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:47.987 7f83750a1700 1 -- 172.21.2.112:0/1498612135 wait complete. 2021-02-10T11:25:47.995 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:48.002 7f83750a1700 1 Processor -- start 2021-02-10T11:25:47.995 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:48.002 7f83750a1700 1 -- start start 2021-02-10T11:25:47.997 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:48.002 7f83750a1700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f83700b7db0 0x7f83700b8260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:47.997 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:48.002 7f83750a1700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f83700b8880 0x7f83701a7450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:47.997 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:48.002 7f83750a1700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f83700b8c80 0x7f83701a7a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:47.997 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:48.002 7f83750a1700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83700b79b0 con 0x7f83700b8880 2021-02-10T11:25:47.997 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:48.002 7f83750a1700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83701a8090 con 0x7f83700b7db0 2021-02-10T11:25:47.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:48.002 7f83750a1700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83701a82b0 con 0x7f83700b8c80 2021-02-10T11:25:47.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:48.003 7f836e59c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f83700b8880 0x7f83701a7450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:47.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:48.003 7f836e59c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f83700b8880 0x7f83701a7450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:45792/0 (socket says 172.21.2.112:45792) 2021-02-10T11:25:47.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:48.003 7f836e59c700 1 -- 172.21.2.112:0/717334754 learned_addr learned my addr 172.21.2.112:0/717334754 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:25:47.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:48.003 7f836e59c700 1 -- 172.21.2.112:0/717334754 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f83700b8c80 msgr2=0x7f83701a7a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:47.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:48.003 7f836e59c700 1 --2- 172.21.2.112:0/717334754 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f83700b8c80 0x7f83701a7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:47.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:48.003 7f836e59c700 1 -- 172.21.2.112:0/717334754 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f83700b7db0 msgr2=0x7f83700b8260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:47.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:48.003 7f836e59c700 1 --2- 172.21.2.112:0/717334754 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f83700b7db0 0x7f83700b8260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:47.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:48.003 7f836e59c700 1 -- 172.21.2.112:0/717334754 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8370190c10 con 0x7f83700b8880 2021-02-10T11:25:47.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:48.003 7f836e59c700 1 --2- 172.21.2.112:0/717334754 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f83700b8880 0x7f83701a7450 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 rx=0x7f8360006ed0 tx=0x7f836000b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:47.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:48.003 7f83677fe700 1 -- 172.21.2.112:0/717334754 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8360006240 con 0x7f83700b8880 2021-02-10T11:25:47.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:48.003 7f83677fe700 1 -- 172.21.2.112:0/717334754 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83600054b0 con 0x7f83700b8880 2021-02-10T11:25:47.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:48.003 7f83750a1700 1 -- 172.21.2.112:0/717334754 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8370190c10 con 0x7f83700b8880 2021-02-10T11:25:47.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:48.003 7f83677fe700 1 -- 172.21.2.112:0/717334754 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8360005990 con 0x7f83700b8880 2021-02-10T11:25:47.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:48.003 7f83750a1700 1 -- 172.21.2.112:0/717334754 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83700c1eb0 con 0x7f83700b8880 2021-02-10T11:25:47.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:48.004 7f83677fe700 1 -- 172.21.2.112:0/717334754 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8360006030 con 0x7f83700b8880 2021-02-10T11:25:47.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:48.004 7f83677fe700 1 --2- 172.21.2.112:0/717334754 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f83540186c0 0x7f835401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:47.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:48.004 7f83677fe700 1 -- 172.21.2.112:0/717334754 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f83600325d0 con 0x7f83700b8880 2021-02-10T11:25:47.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:48.005 7f836ed9d700 1 --2- 172.21.2.112:0/717334754 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f83540186c0 0x7f835401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:47.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:48.005 7f836ed9d700 1 --2- 172.21.2.112:0/717334754 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f83540186c0 0x7f835401abc0 crc :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:48.025 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:48.035 7f83750a1700 1 -- 172.21.2.112:0/717334754 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8370105b30 con 0x7f83700b8880 2021-02-10T11:25:48.027 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:48.037 7f83677fe700 1 -- 172.21.2.112:0/717334754 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8360030d80 con 0x7f83700b8880 2021-02-10T11:25:48.158 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:48.167 7f83750a1700 1 -- 172.21.2.112:0/717334754 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348638", "target": ["mgr", ""]}) v1 -- 0x7f83700c1eb0 con 0x7f83540186c0 2021-02-10T11:25:48.161 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:48.172 7f83677fe700 1 -- 172.21.2.112:0/717334754 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f8358001640 con 0x7f83540186c0 2021-02-10T11:25:48.165 INFO:teuthology.orchestra.run.gibba012.stdout:/volumes/_nogroup/subvolume_0000000000348638/610050ee-5511-4d2f-9ef9-0030582439c5 2021-02-10T11:25:48.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:48.177 7f83750a1700 1 -- 172.21.2.112:0/717334754 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f83540186c0 msgr2=0x7f835401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:48.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:48.177 7f83750a1700 1 --2- 172.21.2.112:0/717334754 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f83540186c0 0x7f835401abc0 crc :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:48.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:48.177 7f83750a1700 1 -- 172.21.2.112:0/717334754 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f83700b8880 msgr2=0x7f83701a7450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:48.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:48.177 7f83750a1700 1 --2- 172.21.2.112:0/717334754 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f83700b8880 0x7f83701a7450 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 rx=0x7f8360006ed0 tx=0x7f836000b960).stop 2021-02-10T11:25:48.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:48.177 7f83750a1700 1 -- 172.21.2.112:0/717334754 shutdown_connections 2021-02-10T11:25:48.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:48.177 7f83750a1700 1 --2- 172.21.2.112:0/717334754 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f83540186c0 0x7f835401abc0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:48.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:48.177 7f83750a1700 1 --2- 172.21.2.112:0/717334754 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f83700b8c80 0x7f83701a7a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:48.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:48.177 7f83750a1700 1 --2- 172.21.2.112:0/717334754 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f83700b8880 0x7f83701a7450 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:48.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:48.177 7f83750a1700 1 --2- 172.21.2.112:0/717334754 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f83700b7db0 0x7f83700b8260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:48.169 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:48.177 7f83750a1700 1 -- 172.21.2.112:0/717334754 >> 172.21.2.112:0/717334754 conn(0x7f8370094e10 msgr2=0x7f83700a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:48.169 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:48.177 7f83750a1700 1 -- 172.21.2.112:0/717334754 shutdown_connections 2021-02-10T11:25:48.169 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:48.177 7f83750a1700 1 -- 172.21.2.112:0/717334754 wait complete. 2021-02-10T11:25:48.175 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000348638 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000348638/610050ee-5511-4d2f-9ef9-0030582439c5 2021-02-10T11:25:48.175 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:48.175 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:48.175 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:48.511 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:25:48.511 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:48.512 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348638/610050ee-5511-4d2f-9ef9-0030582439c5/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:48.890 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:25:48.890 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:25:48.891 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0224232 s, 46.8 MB/s 2021-02-10T11:25:48.891 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:48.892 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:48.892 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:49.227 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:25:49.228 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:49.228 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348638/610050ee-5511-4d2f-9ef9-0030582439c5/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:49.600 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:25:49.600 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:25:49.600 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0247321 s, 42.4 MB/s 2021-02-10T11:25:49.601 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:49.602 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:49.602 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:49.936 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:25:49.937 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:49.937 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348638/610050ee-5511-4d2f-9ef9-0030582439c5/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:50.314 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:25:50.314 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:25:50.314 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0250475 s, 41.9 MB/s 2021-02-10T11:25:50.315 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:50.315 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:50.315 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:50.652 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:25:50.652 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:50.652 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348638/610050ee-5511-4d2f-9ef9-0030582439c5/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:51.028 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:25:51.028 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:25:51.028 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0247641 s, 42.3 MB/s 2021-02-10T11:25:51.029 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:51.029 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:51.029 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:51.366 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:25:51.366 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:51.366 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348638/610050ee-5511-4d2f-9ef9-0030582439c5/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:51.741 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:25:51.741 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:25:51.741 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0233374 s, 44.9 MB/s 2021-02-10T11:25:51.742 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:51.742 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:51.742 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:52.077 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:25:52.077 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:52.077 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348638/610050ee-5511-4d2f-9ef9-0030582439c5/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:52.455 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:25:52.456 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:25:52.456 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0249212 s, 42.1 MB/s 2021-02-10T11:25:52.457 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:52.457 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:52.457 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:52.794 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:25:52.794 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:52.794 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348638/610050ee-5511-4d2f-9ef9-0030582439c5/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:53.172 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:25:53.173 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:25:53.173 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.025099 s, 41.8 MB/s 2021-02-10T11:25:53.174 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:53.174 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:53.174 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:53.508 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:25:53.509 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:53.509 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348638/610050ee-5511-4d2f-9ef9-0030582439c5/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:53.896 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:25:53.896 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:25:53.896 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0231531 s, 45.3 MB/s 2021-02-10T11:25:53.897 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:53.897 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:53.897 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:54.236 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:25:54.236 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:54.236 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348638/610050ee-5511-4d2f-9ef9-0030582439c5/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:54.606 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:25:54.607 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:25:54.607 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0229707 s, 45.6 MB/s 2021-02-10T11:25:54.608 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:54.608 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:54.608 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:54.944 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:25:54.944 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:54.944 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348638/610050ee-5511-4d2f-9ef9-0030582439c5/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:55.323 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:25:55.323 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:25:55.324 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0260386 s, 40.3 MB/s 2021-02-10T11:25:55.324 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000348639 2021-02-10T11:25:55.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:55.750 7f998f15e700 1 Processor -- start 2021-02-10T11:25:55.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:55.750 7f998f15e700 1 -- start start 2021-02-10T11:25:55.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:55.750 7f998f15e700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f99881b0470 0x7f99881b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:55.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:55.750 7f998f15e700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f99880f20e0 0x7f99880f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:55.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:55.750 7f998f15e700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99880f0b50 con 0x7f99880ef610 2021-02-10T11:25:55.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:55.750 7f998f15e700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99880f0820 con 0x7f99881b0470 2021-02-10T11:25:55.757 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:55.750 7f998f15e700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99880f24e0 con 0x7f99880f20e0 2021-02-10T11:25:55.757 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:55.751 7f998cf7a700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f99881b0470 0x7f99881b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:55.757 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:55.751 7f9987fff700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f99880ef610 0x7f99880efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:45260/0 (socket says 172.21.2.112:45260) 2021-02-10T11:25:55.757 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:55.751 7f998cf7a700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f99881b0470 0x7f99881b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:45798/0 (socket says 172.21.2.112:45798) 2021-02-10T11:25:55.757 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:55.751 7f9987fff700 1 -- 172.21.2.112:0/493635625 learned_addr learned my addr 172.21.2.112:0/493635625 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:25:55.757 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:55.751 7f998d77b700 1 --2- 172.21.2.112:0/493635625 >> v2:172.21.2.120:3300/0 conn(0x7f99880f20e0 0x7f99880f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:55.757 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:55.751 7f998cf7a700 1 -- 172.21.2.112:0/493635625 >> v2:172.21.2.120:3300/0 conn(0x7f99880f20e0 msgr2=0x7f99880f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:55.757 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:55.751 7f998cf7a700 1 --2- 172.21.2.112:0/493635625 >> v2:172.21.2.120:3300/0 conn(0x7f99880f20e0 0x7f99880f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:55.757 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:55.751 7f998cf7a700 1 -- 172.21.2.112:0/493635625 >> v1:172.21.2.112:6789/0 conn(0x7f99880ef610 legacy=0x7f99880efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:55.757 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:55.751 7f998cf7a700 1 -- 172.21.2.112:0/493635625 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99880e52e0 con 0x7f99881b0470 2021-02-10T11:25:55.757 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:55.751 7f998d77b700 1 --2- 172.21.2.112:0/493635625 >> v2:172.21.2.120:3300/0 conn(0x7f99880f20e0 0x7f99880f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:25:55.757 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:55.751 7f998cf7a700 1 --2- 172.21.2.112:0/493635625 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f99881b0470 0x7f99881b0870 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 rx=0x7f9970006650 tx=0x7f9970009a80).ready entity=mon.0 client_cookie=95cc646164754de2 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:55.757 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:55.751 7f99877fe700 1 -- 172.21.2.112:0/493635625 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f99700009b0 con 0x7f99881b0470 2021-02-10T11:25:55.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:55.751 7f99877fe700 1 -- 172.21.2.112:0/493635625 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f997000c0c0 con 0x7f99881b0470 2021-02-10T11:25:55.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:55.751 7f99877fe700 1 -- 172.21.2.112:0/493635625 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f997000c510 con 0x7f99881b0470 2021-02-10T11:25:55.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:55.752 7f998f15e700 1 -- 172.21.2.112:0/493635625 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f99881b0470 msgr2=0x7f99881b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:55.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:55.752 7f998f15e700 1 --2- 172.21.2.112:0/493635625 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f99881b0470 0x7f99881b0870 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 rx=0x7f9970006650 tx=0x7f9970009a80).stop 2021-02-10T11:25:55.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:55.752 7f998f15e700 1 -- 172.21.2.112:0/493635625 shutdown_connections 2021-02-10T11:25:55.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:55.752 7f998f15e700 1 --2- 172.21.2.112:0/493635625 >> v2:172.21.2.120:3300/0 conn(0x7f99880f20e0 0x7f99880f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:55.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:55.752 7f998f15e700 1 --2- 172.21.2.112:0/493635625 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f99881b0470 0x7f99881b0870 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:55.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:55.752 7f998f15e700 1 -- 172.21.2.112:0/493635625 >> 172.21.2.112:0/493635625 conn(0x7f99880ce3e0 msgr2=0x7f99880e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:55.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:55.752 7f998f15e700 1 -- 172.21.2.112:0/493635625 shutdown_connections 2021-02-10T11:25:55.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:55.752 7f998f15e700 1 -- 172.21.2.112:0/493635625 wait complete. 2021-02-10T11:25:55.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:55.767 7f998f15e700 1 Processor -- start 2021-02-10T11:25:55.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:55.767 7f998f15e700 1 -- start start 2021-02-10T11:25:55.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:55.767 7f998f15e700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f99880bf930 0x7f99880bfd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:55.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:55.767 7f998f15e700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f99880906c0 0x7f9988090ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:55.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:55.767 7f998f15e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f99880c0960 0x7f99880c0d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:55.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:55.767 7f998f15e700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99880915c0 con 0x7f99880c0960 2021-02-10T11:25:55.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:55.767 7f998f15e700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f998819c130 con 0x7f99880906c0 2021-02-10T11:25:55.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:55.767 7f998f15e700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99880910e0 con 0x7f99880bf930 2021-02-10T11:25:55.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:55.767 7f998d77b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f99880c0960 0x7f99880c0d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:55.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:55.768 7f998d77b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f99880c0960 0x7f99880c0d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:45804/0 (socket says 172.21.2.112:45804) 2021-02-10T11:25:55.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:55.768 7f998d77b700 1 -- 172.21.2.112:0/3644376366 learned_addr learned my addr 172.21.2.112:0/3644376366 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:25:55.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:55.768 7f998cf7a700 1 --2- 172.21.2.112:0/3644376366 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f99880bf930 0x7f99880bfd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:55.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:55.768 7f9987fff700 1 --2- 172.21.2.112:0/3644376366 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f99880906c0 0x7f9988090ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:55.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:55.768 7f998d77b700 1 -- 172.21.2.112:0/3644376366 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f99880bf930 msgr2=0x7f99880bfd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:55.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:55.768 7f998d77b700 1 --2- 172.21.2.112:0/3644376366 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f99880bf930 0x7f99880bfd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:55.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:55.768 7f998d77b700 1 -- 172.21.2.112:0/3644376366 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f99880906c0 msgr2=0x7f9988090ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:55.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:55.768 7f998d77b700 1 --2- 172.21.2.112:0/3644376366 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f99880906c0 0x7f9988090ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:55.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:55.768 7f998d77b700 1 -- 172.21.2.112:0/3644376366 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99880c1380 con 0x7f99880c0960 2021-02-10T11:25:55.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:55.768 7f998cf7a700 1 --2- 172.21.2.112:0/3644376366 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f99880bf930 0x7f99880bfd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:25:55.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:55.768 7f998d77b700 1 --2- 172.21.2.112:0/3644376366 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f99880c0960 0x7f99880c0d60 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 rx=0x7f998000b970 tx=0x7f998000bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:55.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:55.768 7f99857fa700 1 -- 172.21.2.112:0/3644376366 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9980000f90 con 0x7f99880c0960 2021-02-10T11:25:55.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:55.768 7f99857fa700 1 -- 172.21.2.112:0/3644376366 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9980003030 con 0x7f99880c0960 2021-02-10T11:25:55.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:55.768 7f99857fa700 1 -- 172.21.2.112:0/3644376366 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9980001400 con 0x7f99880c0960 2021-02-10T11:25:55.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:55.768 7f998f15e700 1 -- 172.21.2.112:0/3644376366 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99880c1380 con 0x7f99880c0960 2021-02-10T11:25:55.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:55.768 7f998f15e700 1 -- 172.21.2.112:0/3644376366 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99880c1380 con 0x7f99880c0960 2021-02-10T11:25:55.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:55.769 7f99857fa700 1 -- 172.21.2.112:0/3644376366 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9980000df0 con 0x7f99880c0960 2021-02-10T11:25:55.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:55.769 7f99857fa700 1 --2- 172.21.2.112:0/3644376366 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f996c0186c0 0x7f996c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:55.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:55.769 7f99857fa700 1 -- 172.21.2.112:0/3644376366 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f9980032740 con 0x7f99880c0960 2021-02-10T11:25:55.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:55.769 7f998cf7a700 1 --2- 172.21.2.112:0/3644376366 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f996c0186c0 0x7f996c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:55.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:55.770 7f998cf7a700 1 --2- 172.21.2.112:0/3644376366 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f996c0186c0 0x7f996c01abc0 crc :-1 s=READY pgs=317 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:55.790 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:55.800 7f998f15e700 1 -- 172.21.2.112:0/3644376366 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9988105ae0 con 0x7f99880c0960 2021-02-10T11:25:55.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:55.803 7f99857fa700 1 -- 172.21.2.112:0/3644376366 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f998002df30 con 0x7f99880c0960 2021-02-10T11:25:55.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:55.930 7f998f15e700 1 -- 172.21.2.112:0/3644376366 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348639", "target": ["mgr", ""]}) v1 -- 0x7f99880f0b50 con 0x7f996c0186c0 2021-02-10T11:25:55.946 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:55.956 7f99857fa700 1 -- 172.21.2.112:0/3644376366 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f9970000c90 con 0x7f996c0186c0 2021-02-10T11:25:55.954 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:55.964 7f998f15e700 1 -- 172.21.2.112:0/3644376366 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f996c0186c0 msgr2=0x7f996c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:55.954 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:55.964 7f998f15e700 1 --2- 172.21.2.112:0/3644376366 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f996c0186c0 0x7f996c01abc0 crc :-1 s=READY pgs=317 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:55.954 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:55.964 7f998f15e700 1 -- 172.21.2.112:0/3644376366 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f99880c0960 msgr2=0x7f99880c0d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:55.954 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:55.964 7f998f15e700 1 --2- 172.21.2.112:0/3644376366 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f99880c0960 0x7f99880c0d60 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 rx=0x7f998000b970 tx=0x7f998000bd10).stop 2021-02-10T11:25:55.955 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:55.964 7f998f15e700 1 -- 172.21.2.112:0/3644376366 shutdown_connections 2021-02-10T11:25:55.955 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:55.964 7f998f15e700 1 --2- 172.21.2.112:0/3644376366 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f996c0186c0 0x7f996c01abc0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:55.955 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:55.964 7f998f15e700 1 --2- 172.21.2.112:0/3644376366 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f99880906c0 0x7f9988090ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:55.955 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:55.964 7f998f15e700 1 --2- 172.21.2.112:0/3644376366 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f99880c0960 0x7f99880c0d60 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:55.955 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:55.964 7f998f15e700 1 --2- 172.21.2.112:0/3644376366 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f99880bf930 0x7f99880bfd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:55.955 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:55.964 7f998f15e700 1 -- 172.21.2.112:0/3644376366 >> 172.21.2.112:0/3644376366 conn(0x7f9988094e10 msgr2=0x7f99880a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:55.955 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:55.965 7f998f15e700 1 -- 172.21.2.112:0/3644376366 shutdown_connections 2021-02-10T11:25:55.955 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:55.965 7f998f15e700 1 -- 172.21.2.112:0/3644376366 wait complete. 2021-02-10T11:25:55.963 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000348639 2021-02-10T11:25:56.393 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:56.387 7f69054fc700 1 Processor -- start 2021-02-10T11:25:56.393 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:56.387 7f69054fc700 1 -- start start 2021-02-10T11:25:56.393 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:56.387 7f69054fc700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f69001b0470 0x7f69001b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:56.393 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:56.387 7f69054fc700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f69000f20e0 0x7f69000f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:56.393 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:56.387 7f69054fc700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69000f0b50 con 0x7f69000ef610 2021-02-10T11:25:56.393 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:56.387 7f69054fc700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69000f0820 con 0x7f69000f20e0 2021-02-10T11:25:56.393 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:56.387 7f69054fc700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69000f24e0 con 0x7f69001b0470 2021-02-10T11:25:56.393 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:56.387 7f68ffac0700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f69000f20e0 0x7f69000f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:56.393 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:56.388 7f68ffac0700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f69000f20e0 0x7f69000f0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:45812/0 (socket says 172.21.2.112:45812) 2021-02-10T11:25:56.393 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:56.388 7f68ffac0700 1 -- 172.21.2.112:0/1574337109 learned_addr learned my addr 172.21.2.112:0/1574337109 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:25:56.394 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:56.388 7f68ffac0700 1 -- 172.21.2.112:0/1574337109 >> v2:172.21.2.120:3300/0 conn(0x7f69001b0470 msgr2=0x7f69001b0870 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:25:56.394 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:56.388 7f68ffac0700 1 --2- 172.21.2.112:0/1574337109 >> v2:172.21.2.120:3300/0 conn(0x7f69001b0470 0x7f69001b0870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:56.394 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:56.388 7f68ffac0700 1 -- 172.21.2.112:0/1574337109 >> v1:172.21.2.120:6789/0 conn(0x7f69000ef610 legacy=0x7f69000efa10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:25:56.394 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:56.388 7f68ffac0700 1 -- 172.21.2.112:0/1574337109 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69000e52e0 con 0x7f69000f20e0 2021-02-10T11:25:56.394 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:56.388 7f68ffac0700 1 --2- 172.21.2.112:0/1574337109 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f69000f20e0 0x7f69000f0200 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 rx=0x7f68f8006f00 tx=0x7f68f8009f10).ready entity=mon.0 client_cookie=cf73ae42299f8a5c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:56.394 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:56.388 7f68fe2bd700 1 -- 172.21.2.112:0/1574337109 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f68f800b3b0 con 0x7f69000f20e0 2021-02-10T11:25:56.394 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:56.388 7f68fe2bd700 1 -- 172.21.2.112:0/1574337109 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68f800b630 con 0x7f69000f20e0 2021-02-10T11:25:56.394 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:56.388 7f68fe2bd700 1 -- 172.21.2.112:0/1574337109 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f68f800bb50 con 0x7f69000f20e0 2021-02-10T11:25:56.394 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:56.388 7f69054fc700 1 -- 172.21.2.112:0/1574337109 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f69000f20e0 msgr2=0x7f69000f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:56.394 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:56.388 7f69054fc700 1 --2- 172.21.2.112:0/1574337109 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f69000f20e0 0x7f69000f0200 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 rx=0x7f68f8006f00 tx=0x7f68f8009f10).stop 2021-02-10T11:25:56.394 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:56.389 7f69054fc700 1 -- 172.21.2.112:0/1574337109 shutdown_connections 2021-02-10T11:25:56.394 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:56.389 7f69054fc700 1 --2- 172.21.2.112:0/1574337109 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f69000f20e0 0x7f69000f0200 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:56.394 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:56.389 7f69054fc700 1 --2- 172.21.2.112:0/1574337109 >> v2:172.21.2.120:3300/0 conn(0x7f69001b0470 0x7f69001b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:56.395 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:56.389 7f69054fc700 1 -- 172.21.2.112:0/1574337109 >> 172.21.2.112:0/1574337109 conn(0x7f69000ce3e0 msgr2=0x7f69000e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:56.395 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:56.389 7f69054fc700 1 -- 172.21.2.112:0/1574337109 shutdown_connections 2021-02-10T11:25:56.395 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:56.389 7f69054fc700 1 -- 172.21.2.112:0/1574337109 wait complete. 2021-02-10T11:25:56.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:56.403 7f69054fc700 1 Processor -- start 2021-02-10T11:25:56.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:56.403 7f69054fc700 1 -- start start 2021-02-10T11:25:56.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:56.404 7f69054fc700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f69000c36a0 0x7f69000c2b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:56.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:56.404 7f69054fc700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f69000c3190 0x7f69000b1d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:56.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:56.404 7f69054fc700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f69000b2370 0x7f69000b2d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:56.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:56.404 7f69054fc700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69000b27c0 con 0x7f69000c36a0 2021-02-10T11:25:56.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:56.404 7f69054fc700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69000b3360 con 0x7f69000c3190 2021-02-10T11:25:56.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:56.404 7f69054fc700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69000b3540 con 0x7f69000b2370 2021-02-10T11:25:56.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:56.404 7f68ff2bf700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f69000c36a0 0x7f69000c2b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:56.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:56.404 7f68ff2bf700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f69000c36a0 0x7f69000c2b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:45818/0 (socket says 172.21.2.112:45818) 2021-02-10T11:25:56.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:56.404 7f68ff2bf700 1 -- 172.21.2.112:0/333852787 learned_addr learned my addr 172.21.2.112:0/333852787 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:25:56.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:56.404 7f68ff2bf700 1 -- 172.21.2.112:0/333852787 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f69000b2370 msgr2=0x7f69000b2d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:56.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:56.404 7f68ff2bf700 1 --2- 172.21.2.112:0/333852787 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f69000b2370 0x7f69000b2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:56.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:56.404 7f68ff2bf700 1 -- 172.21.2.112:0/333852787 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f69000c3190 msgr2=0x7f69000b1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:56.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:56.404 7f68ff2bf700 1 --2- 172.21.2.112:0/333852787 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f69000c3190 0x7f69000b1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:56.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:56.404 7f68ff2bf700 1 -- 172.21.2.112:0/333852787 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69000b3760 con 0x7f69000c36a0 2021-02-10T11:25:56.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:56.405 7f68ff2bf700 1 --2- 172.21.2.112:0/333852787 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f69000c36a0 0x7f69000c2b70 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 rx=0x7f68f000b5c0 tx=0x7f68f000b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:56.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:56.405 7f68f7fff700 1 -- 172.21.2.112:0/333852787 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f68f00009b0 con 0x7f69000c36a0 2021-02-10T11:25:56.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:56.405 7f68f7fff700 1 -- 172.21.2.112:0/333852787 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68f000bd50 con 0x7f69000c36a0 2021-02-10T11:25:56.398 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:56.405 7f68f7fff700 1 -- 172.21.2.112:0/333852787 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f68f00062d0 con 0x7f69000c36a0 2021-02-10T11:25:56.398 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:56.405 7f69054fc700 1 -- 172.21.2.112:0/333852787 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69000b3760 con 0x7f69000c36a0 2021-02-10T11:25:56.398 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:56.405 7f69054fc700 1 -- 172.21.2.112:0/333852787 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69000b3760 con 0x7f69000c36a0 2021-02-10T11:25:56.398 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:56.405 7f68f7fff700 1 -- 172.21.2.112:0/333852787 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f68f0006030 con 0x7f69000c36a0 2021-02-10T11:25:56.398 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:56.405 7f68f7fff700 1 --2- 172.21.2.112:0/333852787 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f68e40186c0 0x7f68e401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:56.398 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:56.405 7f68f7fff700 1 -- 172.21.2.112:0/333852787 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f68f0032590 con 0x7f69000c36a0 2021-02-10T11:25:56.398 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:56.406 7f68feabe700 1 --2- 172.21.2.112:0/333852787 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f68e40186c0 0x7f68e401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:56.398 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:56.406 7f68feabe700 1 --2- 172.21.2.112:0/333852787 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f68e40186c0 0x7f68e401abc0 crc :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:56.428 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:56.438 7f69054fc700 1 -- 172.21.2.112:0/333852787 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69000edd10 con 0x7f69000c36a0 2021-02-10T11:25:56.430 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:56.440 7f68f7fff700 1 -- 172.21.2.112:0/333852787 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f68f0000c40 con 0x7f69000c36a0 2021-02-10T11:25:56.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:56.570 7f69054fc700 1 -- 172.21.2.112:0/333852787 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348639", "target": ["mgr", ""]}) v1 -- 0x7f69000edd10 con 0x7f68e40186c0 2021-02-10T11:25:56.564 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:56.574 7f68f7fff700 1 -- 172.21.2.112:0/333852787 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f68e8006a10 con 0x7f68e40186c0 2021-02-10T11:25:56.567 INFO:teuthology.orchestra.run.gibba012.stdout:/volumes/_nogroup/subvolume_0000000000348639/4fd3bc10-7ced-4c6e-8d1b-75d2a6b760d7 2021-02-10T11:25:56.569 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:56.579 7f69054fc700 1 -- 172.21.2.112:0/333852787 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f68e40186c0 msgr2=0x7f68e401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:56.569 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:56.580 7f69054fc700 1 --2- 172.21.2.112:0/333852787 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f68e40186c0 0x7f68e401abc0 crc :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:56.569 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:56.580 7f69054fc700 1 -- 172.21.2.112:0/333852787 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f69000c36a0 msgr2=0x7f69000c2b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:56.570 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:56.580 7f69054fc700 1 --2- 172.21.2.112:0/333852787 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f69000c36a0 0x7f69000c2b70 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 rx=0x7f68f000b5c0 tx=0x7f68f000b960).stop 2021-02-10T11:25:56.570 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:56.580 7f69054fc700 1 -- 172.21.2.112:0/333852787 shutdown_connections 2021-02-10T11:25:56.570 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:56.580 7f69054fc700 1 --2- 172.21.2.112:0/333852787 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f68e40186c0 0x7f68e401abc0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:56.570 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:56.580 7f69054fc700 1 --2- 172.21.2.112:0/333852787 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f69000c3190 0x7f69000b1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:56.570 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:56.580 7f69054fc700 1 --2- 172.21.2.112:0/333852787 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f69000b2370 0x7f69000b2d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:56.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:56.580 7f69054fc700 1 --2- 172.21.2.112:0/333852787 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f69000c36a0 0x7f69000c2b70 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:56.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:56.580 7f69054fc700 1 -- 172.21.2.112:0/333852787 >> 172.21.2.112:0/333852787 conn(0x7f690009f630 msgr2=0x7f69000a0130 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:56.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:56.580 7f69054fc700 1 -- 172.21.2.112:0/333852787 shutdown_connections 2021-02-10T11:25:56.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:25:56.580 7f69054fc700 1 -- 172.21.2.112:0/333852787 wait complete. 2021-02-10T11:25:56.577 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000348639 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000348639/4fd3bc10-7ced-4c6e-8d1b-75d2a6b760d7 2021-02-10T11:25:56.578 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:56.578 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:56.578 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:56.913 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:25:56.913 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:56.913 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348639/4fd3bc10-7ced-4c6e-8d1b-75d2a6b760d7/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:57.292 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:25:57.292 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:25:57.293 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.023069 s, 45.5 MB/s 2021-02-10T11:25:57.293 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:57.294 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:57.294 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:57.628 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:25:57.629 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:57.629 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348639/4fd3bc10-7ced-4c6e-8d1b-75d2a6b760d7/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:58.004 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:25:58.004 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:25:58.004 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0222995 s, 47.0 MB/s 2021-02-10T11:25:58.005 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:58.005 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:58.005 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:58.340 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:25:58.341 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:58.341 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348639/4fd3bc10-7ced-4c6e-8d1b-75d2a6b760d7/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:58.715 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:25:58.870 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:25:58.870 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0226581 s, 46.3 MB/s 2021-02-10T11:25:58.872 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:58.872 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:58.872 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:59.207 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:25:59.208 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:59.208 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348639/4fd3bc10-7ced-4c6e-8d1b-75d2a6b760d7/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:59.584 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:25:59.584 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:25:59.584 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.026914 s, 39.0 MB/s 2021-02-10T11:25:59.585 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:59.585 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:59.585 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:59.923 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:25:59.923 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:59.923 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348639/4fd3bc10-7ced-4c6e-8d1b-75d2a6b760d7/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:00.296 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:26:00.296 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:26:00.296 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0231262 s, 45.3 MB/s 2021-02-10T11:26:00.297 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:00.297 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:00.298 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:00.633 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:26:00.633 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:00.634 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348639/4fd3bc10-7ced-4c6e-8d1b-75d2a6b760d7/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:01.009 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:26:01.010 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:26:01.010 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0244182 s, 42.9 MB/s 2021-02-10T11:26:01.011 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:01.011 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:01.011 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:01.347 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:26:01.348 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:01.348 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348639/4fd3bc10-7ced-4c6e-8d1b-75d2a6b760d7/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:01.724 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:26:01.724 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:26:01.724 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.024719 s, 42.4 MB/s 2021-02-10T11:26:01.725 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:01.725 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:01.725 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:02.061 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:26:02.062 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:02.062 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348639/4fd3bc10-7ced-4c6e-8d1b-75d2a6b760d7/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:02.440 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:26:02.440 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:26:02.440 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0265108 s, 39.6 MB/s 2021-02-10T11:26:02.441 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:02.441 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:02.441 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:02.777 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:26:02.777 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:02.777 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348639/4fd3bc10-7ced-4c6e-8d1b-75d2a6b760d7/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:03.156 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:26:03.156 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:26:03.156 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0257726 s, 40.7 MB/s 2021-02-10T11:26:03.157 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:03.157 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:03.157 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:03.493 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:26:03.494 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:03.494 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348639/4fd3bc10-7ced-4c6e-8d1b-75d2a6b760d7/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:03.870 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:26:03.870 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:26:03.870 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0249982 s, 41.9 MB/s 2021-02-10T11:26:03.872 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000348640 2021-02-10T11:26:04.302 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.296 7fd07b056700 1 Processor -- start 2021-02-10T11:26:04.302 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.297 7fd07b056700 1 -- start start 2021-02-10T11:26:04.302 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.297 7fd07b056700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fd0741b0470 0x7fd0741b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:04.303 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.297 7fd07b056700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fd0740ef610 0x7fd0740efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:04.303 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.297 7fd07b056700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd0740f0030 0x7fd0740f0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:04.303 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.297 7fd07b056700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0740e5ea0 con 0x7fd0741b0470 2021-02-10T11:26:04.303 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.297 7fd07b056700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0740f0b50 con 0x7fd0740ef610 2021-02-10T11:26:04.303 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.297 7fd07b056700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0740e52e0 con 0x7fd0740f0030 2021-02-10T11:26:04.303 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.297 7fd078e72700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fd0741b0470 0x7fd0741b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:04.303 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.297 7fd078e72700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fd0741b0470 0x7fd0741b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:45828/0 (socket says 172.21.2.112:45828) 2021-02-10T11:26:04.303 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.297 7fd078e72700 1 -- 172.21.2.112:0/1497867999 learned_addr learned my addr 172.21.2.112:0/1497867999 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:26:04.303 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.297 7fd078e72700 1 -- 172.21.2.112:0/1497867999 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd0740f0030 msgr2=0x7fd0740f0430 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:26:04.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.297 7fd078e72700 1 --2- 172.21.2.112:0/1497867999 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd0740f0030 0x7fd0740f0430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:04.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.297 7fd078e72700 1 -- 172.21.2.112:0/1497867999 >> v2:172.21.2.120:3300/0 conn(0x7fd0740ef610 msgr2=0x7fd0740efa10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:26:04.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.297 7fd078e72700 1 --2- 172.21.2.112:0/1497867999 >> v2:172.21.2.120:3300/0 conn(0x7fd0740ef610 0x7fd0740efa10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:04.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.297 7fd078e72700 1 -- 172.21.2.112:0/1497867999 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0740f10d0 con 0x7fd0741b0470 2021-02-10T11:26:04.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.298 7fd078e72700 1 --2- 172.21.2.112:0/1497867999 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd0741b0470 0x7fd0741b0870 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 rx=0x7fd05c006700 tx=0x7fd05c009b00).ready entity=mon.0 client_cookie=10762382b6525dc2 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:04.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.298 7fd0737fe700 1 -- 172.21.2.112:0/1497867999 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd05c00c0c0 con 0x7fd0741b0470 2021-02-10T11:26:04.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.298 7fd0737fe700 1 -- 172.21.2.112:0/1497867999 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd05c00c400 con 0x7fd0741b0470 2021-02-10T11:26:04.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.298 7fd0737fe700 1 -- 172.21.2.112:0/1497867999 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd05c00c860 con 0x7fd0741b0470 2021-02-10T11:26:04.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.298 7fd07b056700 1 -- 172.21.2.112:0/1497867999 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd0741b0470 msgr2=0x7fd0741b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:04.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.298 7fd07b056700 1 --2- 172.21.2.112:0/1497867999 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd0741b0470 0x7fd0741b0870 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 rx=0x7fd05c006700 tx=0x7fd05c009b00).stop 2021-02-10T11:26:04.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.298 7fd07b056700 1 -- 172.21.2.112:0/1497867999 shutdown_connections 2021-02-10T11:26:04.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.298 7fd07b056700 1 --2- 172.21.2.112:0/1497867999 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd0740f0030 0x7fd0740f0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:04.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.298 7fd07b056700 1 --2- 172.21.2.112:0/1497867999 >> v2:172.21.2.120:3300/0 conn(0x7fd0740ef610 0x7fd0740efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:04.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.298 7fd07b056700 1 --2- 172.21.2.112:0/1497867999 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd0741b0470 0x7fd0741b0870 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:04.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.298 7fd07b056700 1 -- 172.21.2.112:0/1497867999 >> 172.21.2.112:0/1497867999 conn(0x7fd0740ce3e0 msgr2=0x7fd0740e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:04.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.298 7fd07b056700 1 -- 172.21.2.112:0/1497867999 shutdown_connections 2021-02-10T11:26:04.306 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.298 7fd07b056700 1 -- 172.21.2.112:0/1497867999 wait complete. 2021-02-10T11:26:04.308 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.313 7fd07b056700 1 Processor -- start 2021-02-10T11:26:04.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.313 7fd07b056700 1 -- start start 2021-02-10T11:26:04.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.313 7fd07b056700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd0740b20b0 0x7fd0740ac970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:04.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.313 7fd07b056700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd0740b24b0 0x7fd074191dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:04.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.313 7fd07b056700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd0740acf90 0x7fd0741923e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:04.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.313 7fd07b056700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0740b1b30 con 0x7fd0740b24b0 2021-02-10T11:26:04.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.313 7fd07b056700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0741a04e0 con 0x7fd0740b20b0 2021-02-10T11:26:04.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.313 7fd07b056700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0741a06c0 con 0x7fd0740acf90 2021-02-10T11:26:04.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.313 7fd073fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd0740b24b0 0x7fd074191dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:04.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.313 7fd073fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd0740b24b0 0x7fd074191dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:45830/0 (socket says 172.21.2.112:45830) 2021-02-10T11:26:04.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.313 7fd073fff700 1 -- 172.21.2.112:0/358963757 learned_addr learned my addr 172.21.2.112:0/358963757 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:26:04.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.314 7fd073fff700 1 -- 172.21.2.112:0/358963757 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd0740acf90 msgr2=0x7fd0741923e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:04.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.314 7fd073fff700 1 --2- 172.21.2.112:0/358963757 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd0740acf90 0x7fd0741923e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:04.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.314 7fd073fff700 1 -- 172.21.2.112:0/358963757 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd0740b20b0 msgr2=0x7fd0740ac970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:04.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.314 7fd073fff700 1 --2- 172.21.2.112:0/358963757 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd0740b20b0 0x7fd0740ac970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:04.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.314 7fd073fff700 1 -- 172.21.2.112:0/358963757 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd074192be0 con 0x7fd0740b24b0 2021-02-10T11:26:04.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.314 7fd073fff700 1 --2- 172.21.2.112:0/358963757 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd0740b24b0 0x7fd074191dc0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 rx=0x7fd06400b5c0 tx=0x7fd06400b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:04.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.314 7fd0717fa700 1 -- 172.21.2.112:0/358963757 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd064000c40 con 0x7fd0740b24b0 2021-02-10T11:26:04.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.314 7fd0717fa700 1 -- 172.21.2.112:0/358963757 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd06400bd50 con 0x7fd0740b24b0 2021-02-10T11:26:04.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.314 7fd07b056700 1 -- 172.21.2.112:0/358963757 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd074192be0 con 0x7fd0740b24b0 2021-02-10T11:26:04.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.314 7fd0717fa700 1 -- 172.21.2.112:0/358963757 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd0640054d0 con 0x7fd0740b24b0 2021-02-10T11:26:04.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.314 7fd07b056700 1 -- 172.21.2.112:0/358963757 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd074192be0 con 0x7fd0740b24b0 2021-02-10T11:26:04.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.315 7fd0717fa700 1 -- 172.21.2.112:0/358963757 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd064006030 con 0x7fd0740b24b0 2021-02-10T11:26:04.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.315 7fd0717fa700 1 --2- 172.21.2.112:0/358963757 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd0580186d0 0x7fd05801abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:04.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.315 7fd0717fa700 1 -- 172.21.2.112:0/358963757 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fd064032660 con 0x7fd0740b24b0 2021-02-10T11:26:04.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.315 7fd078e72700 1 --2- 172.21.2.112:0/358963757 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd0580186d0 0x7fd05801abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:04.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.315 7fd078e72700 1 --2- 172.21.2.112:0/358963757 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd0580186d0 0x7fd05801abd0 crc :-1 s=READY pgs=319 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:04.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.347 7fd07b056700 1 -- 172.21.2.112:0/358963757 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd074105af0 con 0x7fd0740b24b0 2021-02-10T11:26:04.339 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.349 7fd0717fa700 1 -- 172.21.2.112:0/358963757 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd0640009b0 con 0x7fd0740b24b0 2021-02-10T11:26:04.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.477 7fd07b056700 1 -- 172.21.2.112:0/358963757 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348640", "target": ["mgr", ""]}) v1 -- 0x7fd0741a04e0 con 0x7fd0580186d0 2021-02-10T11:26:04.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.504 7fd0717fa700 1 -- 172.21.2.112:0/358963757 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fd05c0091c0 con 0x7fd0580186d0 2021-02-10T11:26:04.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.511 7fd07b056700 1 -- 172.21.2.112:0/358963757 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd0580186d0 msgr2=0x7fd05801abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:04.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.511 7fd07b056700 1 --2- 172.21.2.112:0/358963757 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd0580186d0 0x7fd05801abd0 crc :-1 s=READY pgs=319 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:04.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.511 7fd07b056700 1 -- 172.21.2.112:0/358963757 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd0740b24b0 msgr2=0x7fd074191dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:04.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.511 7fd07b056700 1 --2- 172.21.2.112:0/358963757 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd0740b24b0 0x7fd074191dc0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 rx=0x7fd06400b5c0 tx=0x7fd06400b960).stop 2021-02-10T11:26:04.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.511 7fd07b056700 1 -- 172.21.2.112:0/358963757 shutdown_connections 2021-02-10T11:26:04.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.511 7fd07b056700 1 --2- 172.21.2.112:0/358963757 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd0580186d0 0x7fd05801abd0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:04.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.511 7fd07b056700 1 --2- 172.21.2.112:0/358963757 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd0740acf90 0x7fd0741923e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:04.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.511 7fd07b056700 1 --2- 172.21.2.112:0/358963757 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd0740b24b0 0x7fd074191dc0 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:04.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.511 7fd07b056700 1 --2- 172.21.2.112:0/358963757 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd0740b20b0 0x7fd0740ac970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:04.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.511 7fd07b056700 1 -- 172.21.2.112:0/358963757 >> 172.21.2.112:0/358963757 conn(0x7fd0740bf4e0 msgr2=0x7fd0740be3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:04.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.511 7fd07b056700 1 -- 172.21.2.112:0/358963757 shutdown_connections 2021-02-10T11:26:04.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.511 7fd07b056700 1 -- 172.21.2.112:0/358963757 wait complete. 2021-02-10T11:26:04.509 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000348640 2021-02-10T11:26:04.941 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.934 7f83b8152700 1 Processor -- start 2021-02-10T11:26:04.941 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.934 7f83b8152700 1 -- start start 2021-02-10T11:26:04.941 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.935 7f83b8152700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f83b01b0870 0x7f83b00f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:04.941 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.935 7f83b8152700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f83b00efe00 0x7f83b00f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:04.941 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.935 7f83b8152700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83b01b0c70 con 0x7f83b01b0470 2021-02-10T11:26:04.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.935 7f83b8152700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83b00e5ea0 con 0x7f83b01b0870 2021-02-10T11:26:04.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.935 7f83b8152700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83b00f0b50 con 0x7f83b00efe00 2021-02-10T11:26:04.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.935 7f83b576d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f83b01b0870 0x7f83b00f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:04.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.935 7f83b576d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f83b01b0870 0x7f83b00f20e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:45838/0 (socket says 172.21.2.112:45838) 2021-02-10T11:26:04.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.935 7f83b576d700 1 -- 172.21.2.112:0/2115034826 learned_addr learned my addr 172.21.2.112:0/2115034826 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:26:04.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.935 7f83b676f700 1 --2- 172.21.2.112:0/2115034826 >> v2:172.21.2.120:3300/0 conn(0x7f83b00efe00 0x7f83b00f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:04.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.935 7f83b576d700 1 -- 172.21.2.112:0/2115034826 >> v2:172.21.2.120:3300/0 conn(0x7f83b00efe00 msgr2=0x7f83b00f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:26:04.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.935 7f83b576d700 1 --2- 172.21.2.112:0/2115034826 >> v2:172.21.2.120:3300/0 conn(0x7f83b00efe00 0x7f83b00f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:04.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.935 7f83b576d700 1 -- 172.21.2.112:0/2115034826 >> v1:172.21.2.112:6789/0 conn(0x7f83b01b0470 legacy=0x7f83b00ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:04.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.935 7f83b576d700 1 -- 172.21.2.112:0/2115034826 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83b00f0820 con 0x7f83b01b0870 2021-02-10T11:26:04.943 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.935 7f83b676f700 1 --2- 172.21.2.112:0/2115034826 >> v2:172.21.2.120:3300/0 conn(0x7f83b00efe00 0x7f83b00f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:26:04.943 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.936 7f83b576d700 1 --2- 172.21.2.112:0/2115034826 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f83b01b0870 0x7f83b00f20e0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 rx=0x7f83a4006700 tx=0x7f83a4009b80).ready entity=mon.0 client_cookie=8494e50f044336e0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:04.943 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.936 7f83b4f6c700 1 -- 172.21.2.112:0/2115034826 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f83a400c0c0 con 0x7f83b01b0870 2021-02-10T11:26:04.943 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.936 7f83b4f6c700 1 -- 172.21.2.112:0/2115034826 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83a400c340 con 0x7f83b01b0870 2021-02-10T11:26:04.943 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.936 7f83b4f6c700 1 -- 172.21.2.112:0/2115034826 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f83a400c860 con 0x7f83b01b0870 2021-02-10T11:26:04.943 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.936 7f83b8152700 1 -- 172.21.2.112:0/2115034826 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f83b01b0870 msgr2=0x7f83b00f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:04.943 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.936 7f83b8152700 1 --2- 172.21.2.112:0/2115034826 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f83b01b0870 0x7f83b00f20e0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 rx=0x7f83a4006700 tx=0x7f83a4009b80).stop 2021-02-10T11:26:04.943 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.937 7f83b8152700 1 -- 172.21.2.112:0/2115034826 shutdown_connections 2021-02-10T11:26:04.943 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.937 7f83b8152700 1 --2- 172.21.2.112:0/2115034826 >> v2:172.21.2.120:3300/0 conn(0x7f83b00efe00 0x7f83b00f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:04.943 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.937 7f83b8152700 1 --2- 172.21.2.112:0/2115034826 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f83b01b0870 0x7f83b00f20e0 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:04.943 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.937 7f83b8152700 1 -- 172.21.2.112:0/2115034826 >> 172.21.2.112:0/2115034826 conn(0x7f83b00ce3e0 msgr2=0x7f83b00e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:04.943 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.937 7f83b8152700 1 -- 172.21.2.112:0/2115034826 shutdown_connections 2021-02-10T11:26:04.944 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.937 7f83b8152700 1 -- 172.21.2.112:0/2115034826 wait complete. 2021-02-10T11:26:04.945 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.951 7f83b8152700 1 Processor -- start 2021-02-10T11:26:04.945 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.951 7f83b8152700 1 -- start start 2021-02-10T11:26:04.946 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.952 7f83b8152700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f83b009f170 0x7f83b00bb310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:04.946 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.952 7f83b8152700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f83b009e640 0x7f83b009ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:04.946 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.952 7f83b8152700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f83b00bc1d0 0x7f83b00bc5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:04.946 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.952 7f83b8152700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83b00bbcd0 con 0x7f83b009f170 2021-02-10T11:26:04.946 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.952 7f83b8152700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83b00bf930 con 0x7f83b009e640 2021-02-10T11:26:04.946 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.952 7f83b8152700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83b00bb930 con 0x7f83b00bc1d0 2021-02-10T11:26:04.946 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.952 7f83b5f6e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f83b009f170 0x7f83b00bb310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:04.946 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.952 7f83b5f6e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f83b009f170 0x7f83b00bb310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:45846/0 (socket says 172.21.2.112:45846) 2021-02-10T11:26:04.946 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.952 7f83b5f6e700 1 -- 172.21.2.112:0/1855874246 learned_addr learned my addr 172.21.2.112:0/1855874246 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:26:04.946 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.953 7f83b576d700 1 --2- 172.21.2.112:0/1855874246 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f83b009e640 0x7f83b009ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:04.946 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.953 7f83b676f700 1 --2- 172.21.2.112:0/1855874246 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f83b00bc1d0 0x7f83b00bc5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:04.947 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.953 7f83b5f6e700 1 -- 172.21.2.112:0/1855874246 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f83b00bc1d0 msgr2=0x7f83b00bc5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:04.947 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.953 7f83b5f6e700 1 --2- 172.21.2.112:0/1855874246 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f83b00bc1d0 0x7f83b00bc5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:04.947 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.953 7f83b5f6e700 1 -- 172.21.2.112:0/1855874246 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f83b009e640 msgr2=0x7f83b009ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:04.947 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.953 7f83b5f6e700 1 --2- 172.21.2.112:0/1855874246 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f83b009e640 0x7f83b009ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:04.947 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.953 7f83b5f6e700 1 -- 172.21.2.112:0/1855874246 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83b00bcbf0 con 0x7f83b009f170 2021-02-10T11:26:04.947 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.953 7f83b576d700 1 --2- 172.21.2.112:0/1855874246 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f83b009e640 0x7f83b009ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:26:04.947 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.953 7f83b676f700 1 --2- 172.21.2.112:0/1855874246 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f83b00bc1d0 0x7f83b00bc5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:26:04.947 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.953 7f83b5f6e700 1 --2- 172.21.2.112:0/1855874246 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f83b009f170 0x7f83b00bb310 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 rx=0x7f83ac006f00 tx=0x7f83ac00b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:04.947 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.953 7f83a27fc700 1 -- 172.21.2.112:0/1855874246 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f83ac007070 con 0x7f83b009f170 2021-02-10T11:26:04.947 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.953 7f83a27fc700 1 -- 172.21.2.112:0/1855874246 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83ac007270 con 0x7f83b009f170 2021-02-10T11:26:04.947 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.953 7f83a27fc700 1 -- 172.21.2.112:0/1855874246 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f83ac007750 con 0x7f83b009f170 2021-02-10T11:26:04.948 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.953 7f83b8152700 1 -- 172.21.2.112:0/1855874246 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83b00bcbf0 con 0x7f83b009f170 2021-02-10T11:26:04.948 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.953 7f83b8152700 1 -- 172.21.2.112:0/1855874246 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83b00bcbf0 con 0x7f83b009f170 2021-02-10T11:26:04.948 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.954 7f83a27fc700 1 -- 172.21.2.112:0/1855874246 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f83ac002020 con 0x7f83b009f170 2021-02-10T11:26:04.948 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.954 7f83a27fc700 1 --2- 172.21.2.112:0/1855874246 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f83980186d0 0x7f839801abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:04.948 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.954 7f83a27fc700 1 -- 172.21.2.112:0/1855874246 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f83ac0327d0 con 0x7f83b009f170 2021-02-10T11:26:04.948 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.954 7f83b576d700 1 --2- 172.21.2.112:0/1855874246 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f83980186d0 0x7f839801abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:04.948 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.954 7f83b576d700 1 --2- 172.21.2.112:0/1855874246 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f83980186d0 0x7f839801abd0 crc :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:04.975 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.985 7f83b8152700 1 -- 172.21.2.112:0/1855874246 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83b0105ae0 con 0x7f83b009f170 2021-02-10T11:26:04.977 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:04.987 7f83a27fc700 1 -- 172.21.2.112:0/1855874246 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f83ac006b90 con 0x7f83b009f170 2021-02-10T11:26:05.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:05.117 7f83b8152700 1 -- 172.21.2.112:0/1855874246 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348640", "target": ["mgr", ""]}) v1 -- 0x7f83b00f0b50 con 0x7f83980186d0 2021-02-10T11:26:05.110 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:05.121 7f83a27fc700 1 -- 172.21.2.112:0/1855874246 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f83a40009b0 con 0x7f83980186d0 2021-02-10T11:26:05.114 INFO:teuthology.orchestra.run.gibba012.stdout:/volumes/_nogroup/subvolume_0000000000348640/1f032912-1649-45a6-8d44-39d17f2e65c6 2021-02-10T11:26:05.115 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:05.125 7f83b8152700 1 -- 172.21.2.112:0/1855874246 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f83980186d0 msgr2=0x7f839801abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:05.115 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:05.125 7f83b8152700 1 --2- 172.21.2.112:0/1855874246 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f83980186d0 0x7f839801abd0 crc :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:05.115 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:05.125 7f83b8152700 1 -- 172.21.2.112:0/1855874246 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f83b009f170 msgr2=0x7f83b00bb310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:05.115 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:05.125 7f83b8152700 1 --2- 172.21.2.112:0/1855874246 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f83b009f170 0x7f83b00bb310 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 rx=0x7f83ac006f00 tx=0x7f83ac00b940).stop 2021-02-10T11:26:05.116 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:05.126 7f83b8152700 1 -- 172.21.2.112:0/1855874246 shutdown_connections 2021-02-10T11:26:05.116 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:05.126 7f83b8152700 1 --2- 172.21.2.112:0/1855874246 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f83980186d0 0x7f839801abd0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:05.116 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:05.126 7f83b8152700 1 --2- 172.21.2.112:0/1855874246 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f83b009e640 0x7f83b009ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:05.116 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:05.126 7f83b8152700 1 --2- 172.21.2.112:0/1855874246 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f83b00bc1d0 0x7f83b00bc5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:05.116 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:05.126 7f83b8152700 1 --2- 172.21.2.112:0/1855874246 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f83b009f170 0x7f83b00bb310 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:05.116 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:05.126 7f83b8152700 1 -- 172.21.2.112:0/1855874246 >> 172.21.2.112:0/1855874246 conn(0x7f83b009fc10 msgr2=0x7f83b00b3250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:05.116 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:05.126 7f83b8152700 1 -- 172.21.2.112:0/1855874246 shutdown_connections 2021-02-10T11:26:05.117 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:05.126 7f83b8152700 1 -- 172.21.2.112:0/1855874246 wait complete. 2021-02-10T11:26:05.127 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000348640 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000348640/1f032912-1649-45a6-8d44-39d17f2e65c6 2021-02-10T11:26:05.127 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:05.128 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:05.128 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:05.465 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:26:05.465 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:05.465 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348640/1f032912-1649-45a6-8d44-39d17f2e65c6/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:05.848 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:26:05.848 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:26:05.848 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0254412 s, 41.2 MB/s 2021-02-10T11:26:05.849 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:05.849 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:05.849 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:06.184 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:26:06.185 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:06.185 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348640/1f032912-1649-45a6-8d44-39d17f2e65c6/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:06.563 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:26:06.563 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:26:06.563 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0265514 s, 39.5 MB/s 2021-02-10T11:26:06.564 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:06.564 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:06.564 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:06.900 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:26:06.901 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:06.901 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348640/1f032912-1649-45a6-8d44-39d17f2e65c6/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:07.278 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:26:07.278 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:26:07.278 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0245211 s, 42.8 MB/s 2021-02-10T11:26:07.279 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:07.280 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:07.280 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:07.614 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:26:07.614 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:07.614 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348640/1f032912-1649-45a6-8d44-39d17f2e65c6/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:07.992 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:26:07.992 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:26:07.992 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0248891 s, 42.1 MB/s 2021-02-10T11:26:07.993 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:07.994 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:07.994 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:08.329 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:26:08.329 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:08.329 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348640/1f032912-1649-45a6-8d44-39d17f2e65c6/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:08.707 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:26:08.707 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:26:08.707 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0244922 s, 42.8 MB/s 2021-02-10T11:26:08.708 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:08.708 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:08.708 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:09.042 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:26:09.043 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:09.043 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348640/1f032912-1649-45a6-8d44-39d17f2e65c6/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:09.414 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:26:09.415 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:26:09.415 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0245024 s, 42.8 MB/s 2021-02-10T11:26:09.416 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:09.416 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:09.416 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:09.751 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:26:09.751 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:09.752 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348640/1f032912-1649-45a6-8d44-39d17f2e65c6/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:10.127 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:26:10.127 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:26:10.128 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0237873 s, 44.1 MB/s 2021-02-10T11:26:10.128 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:10.129 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:10.129 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:10.445 DEBUG:teuthology.orchestra.run.gibba012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:26:10.448 DEBUG:teuthology.orchestra.run.gibba020:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:26:10.465 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:26:10.465 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:10.466 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348640/1f032912-1649-45a6-8d44-39d17f2e65c6/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:10.821 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:26:10.821 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:26:10.821 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0244079 s, 43.0 MB/s 2021-02-10T11:26:10.822 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:10.822 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:10.822 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:11.158 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:26:11.158 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:11.159 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348640/1f032912-1649-45a6-8d44-39d17f2e65c6/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:11.535 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:26:11.535 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:26:11.535 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0244625 s, 42.9 MB/s 2021-02-10T11:26:11.536 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:11.536 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:11.536 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:11.872 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:26:11.873 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:11.873 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348640/1f032912-1649-45a6-8d44-39d17f2e65c6/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:12.247 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:26:12.247 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:26:12.247 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0232017 s, 45.2 MB/s 2021-02-10T11:26:12.248 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000348641 2021-02-10T11:26:12.679 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:12.673 7f17eb512700 1 Processor -- start 2021-02-10T11:26:12.679 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:12.673 7f17eb512700 1 -- start start 2021-02-10T11:26:12.679 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:12.673 7f17eb512700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f17e41b0470 0x7f17e41b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:12.679 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:12.673 7f17eb512700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f17e40f20e0 0x7f17e40f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:12.679 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:12.673 7f17eb512700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17e40f0b50 con 0x7f17e40ef610 2021-02-10T11:26:12.679 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:12.673 7f17eb512700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17e40f0820 con 0x7f17e40f20e0 2021-02-10T11:26:12.680 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:12.673 7f17eb512700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17e40f24e0 con 0x7f17e41b0470 2021-02-10T11:26:12.680 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:12.673 7f17e9b2f700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f17e40f20e0 0x7f17e40f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:12.680 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:12.674 7f17e9b2f700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f17e40f20e0 0x7f17e40f0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:45852/0 (socket says 172.21.2.112:45852) 2021-02-10T11:26:12.680 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:12.674 7f17e9b2f700 1 -- 172.21.2.112:0/2182250405 learned_addr learned my addr 172.21.2.112:0/2182250405 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:26:12.680 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:12.674 7f17e8b2d700 1 --1- 172.21.2.112:0/2182250405 >> v1:172.21.2.112:6789/0 conn(0x7f17e40ef610 0x7f17e40efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:45314/0 (socket says 172.21.2.112:45314) 2021-02-10T11:26:12.680 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:12.674 7f17e9b2f700 1 -- 172.21.2.112:0/2182250405 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f17e41b0470 msgr2=0x7f17e41b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:26:12.680 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:12.674 7f17e9b2f700 1 --2- 172.21.2.112:0/2182250405 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f17e41b0470 0x7f17e41b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:12.680 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:12.674 7f17e9b2f700 1 -- 172.21.2.112:0/2182250405 >> v1:172.21.2.112:6789/0 conn(0x7f17e40ef610 legacy=0x7f17e40efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:12.680 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:12.674 7f17dbfff700 1 -- 172.21.2.112:0/2182250405 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2036864708 0 0) 0x7f17d40010b0 con 0x7f17e40ef610 2021-02-10T11:26:12.680 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:12.674 7f17e9b2f700 1 -- 172.21.2.112:0/2182250405 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17e40e52e0 con 0x7f17e40f20e0 2021-02-10T11:26:12.680 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:12.674 7f17e9b2f700 1 --2- 172.21.2.112:0/2182250405 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f17e40f20e0 0x7f17e40f0200 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 rx=0x7f17dc006f00 tx=0x7f17dc009e90).ready entity=mon.0 client_cookie=6a31d794b3bb0399 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:12.680 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:12.674 7f17dbfff700 1 -- 172.21.2.112:0/2182250405 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f17dc00b300 con 0x7f17e40f20e0 2021-02-10T11:26:12.680 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:12.674 7f17dbfff700 1 -- 172.21.2.112:0/2182250405 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17dc00b580 con 0x7f17e40f20e0 2021-02-10T11:26:12.680 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:12.674 7f17dbfff700 1 -- 172.21.2.112:0/2182250405 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f17dc00baa0 con 0x7f17e40f20e0 2021-02-10T11:26:12.681 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:12.674 7f17eb512700 1 -- 172.21.2.112:0/2182250405 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f17e40f20e0 msgr2=0x7f17e40f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:12.681 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:12.674 7f17eb512700 1 --2- 172.21.2.112:0/2182250405 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f17e40f20e0 0x7f17e40f0200 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 rx=0x7f17dc006f00 tx=0x7f17dc009e90).stop 2021-02-10T11:26:12.681 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:12.675 7f17eb512700 1 -- 172.21.2.112:0/2182250405 shutdown_connections 2021-02-10T11:26:12.681 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:12.675 7f17eb512700 1 --2- 172.21.2.112:0/2182250405 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f17e40f20e0 0x7f17e40f0200 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:12.681 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:12.675 7f17eb512700 1 --2- 172.21.2.112:0/2182250405 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f17e41b0470 0x7f17e41b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:12.681 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:12.675 7f17eb512700 1 -- 172.21.2.112:0/2182250405 >> 172.21.2.112:0/2182250405 conn(0x7f17e40ce3e0 msgr2=0x7f17e40e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:12.681 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:12.675 7f17eb512700 1 -- 172.21.2.112:0/2182250405 shutdown_connections 2021-02-10T11:26:12.681 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:12.675 7f17eb512700 1 -- 172.21.2.112:0/2182250405 wait complete. 2021-02-10T11:26:12.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:12.690 7f17eb512700 1 Processor -- start 2021-02-10T11:26:12.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:12.690 7f17eb512700 1 -- start start 2021-02-10T11:26:12.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:12.690 7f17eb512700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f17e40b4c60 0x7f17e4106fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:12.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:12.690 7f17eb512700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f17e4190630 0x7f17e4099f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:12.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:12.690 7f17eb512700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f17e4190a30 0x7f17e409b130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:12.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:12.690 7f17eb512700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17e4107920 con 0x7f17e4190a30 2021-02-10T11:26:12.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:12.690 7f17eb512700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17e409abb0 con 0x7f17e4190630 2021-02-10T11:26:12.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:12.690 7f17eb512700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17e40b5060 con 0x7f17e40b4c60 2021-02-10T11:26:12.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:12.690 7f17e9b2f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f17e4190a30 0x7f17e409b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:12.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:12.690 7f17e9b2f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f17e4190a30 0x7f17e409b130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:45860/0 (socket says 172.21.2.112:45860) 2021-02-10T11:26:12.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:12.690 7f17e9b2f700 1 -- 172.21.2.112:0/1818134425 learned_addr learned my addr 172.21.2.112:0/1818134425 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:26:12.684 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:12.691 7f17e9b2f700 1 -- 172.21.2.112:0/1818134425 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f17e40b4c60 msgr2=0x7f17e4106fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:12.684 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:12.691 7f17e9b2f700 1 --2- 172.21.2.112:0/1818134425 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f17e40b4c60 0x7f17e4106fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:12.684 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:12.691 7f17e9b2f700 1 -- 172.21.2.112:0/1818134425 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f17e4190630 msgr2=0x7f17e4099f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:12.684 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:12.691 7f17e9b2f700 1 --2- 172.21.2.112:0/1818134425 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f17e4190630 0x7f17e4099f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:12.684 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:12.691 7f17e9b2f700 1 -- 172.21.2.112:0/1818134425 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17e409a5b0 con 0x7f17e4190a30 2021-02-10T11:26:12.684 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:12.691 7f17e9b2f700 1 --2- 172.21.2.112:0/1818134425 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f17e4190a30 0x7f17e409b130 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 rx=0x7f17dc00b930 tx=0x7f17dc00bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:12.684 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:12.691 7f17d9ffb700 1 -- 172.21.2.112:0/1818134425 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f17dc009450 con 0x7f17e4190a30 2021-02-10T11:26:12.684 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:12.691 7f17d9ffb700 1 -- 172.21.2.112:0/1818134425 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17dc00be10 con 0x7f17e4190a30 2021-02-10T11:26:12.684 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:12.691 7f17d9ffb700 1 -- 172.21.2.112:0/1818134425 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f17dc005a10 con 0x7f17e4190a30 2021-02-10T11:26:12.684 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:12.691 7f17eb512700 1 -- 172.21.2.112:0/1818134425 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17e409a5b0 con 0x7f17e4190a30 2021-02-10T11:26:12.684 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:12.691 7f17eb512700 1 -- 172.21.2.112:0/1818134425 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17e409b750 con 0x7f17e4190a30 2021-02-10T11:26:12.684 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:12.692 7f17d9ffb700 1 -- 172.21.2.112:0/1818134425 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f17dc005820 con 0x7f17e4190a30 2021-02-10T11:26:12.684 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:12.692 7f17d9ffb700 1 --2- 172.21.2.112:0/1818134425 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f17c40186c0 0x7f17c401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:12.684 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:12.692 7f17d9ffb700 1 -- 172.21.2.112:0/1818134425 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f17dc0328d0 con 0x7f17e4190a30 2021-02-10T11:26:12.685 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:12.692 7f17e932e700 1 --2- 172.21.2.112:0/1818134425 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f17c40186c0 0x7f17c401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:12.685 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:12.693 7f17e932e700 1 --2- 172.21.2.112:0/1818134425 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f17c40186c0 0x7f17c401abc0 crc :-1 s=READY pgs=321 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:12.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:12.723 7f17eb512700 1 -- 172.21.2.112:0/1818134425 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17e409bcb0 con 0x7f17e4190a30 2021-02-10T11:26:12.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:12.726 7f17d9ffb700 1 -- 172.21.2.112:0/1818134425 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f17dc009450 con 0x7f17e4190a30 2021-02-10T11:26:12.844 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:12.855 7f17eb512700 1 -- 172.21.2.112:0/1818134425 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348641", "target": ["mgr", ""]}) v1 -- 0x7f17e409b750 con 0x7f17c40186c0 2021-02-10T11:26:12.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:12.880 7f17d9ffb700 1 -- 172.21.2.112:0/1818134425 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f17e00009b0 con 0x7f17c40186c0 2021-02-10T11:26:12.878 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:12.888 7f17eb512700 1 -- 172.21.2.112:0/1818134425 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f17c40186c0 msgr2=0x7f17c401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:12.878 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:12.888 7f17eb512700 1 --2- 172.21.2.112:0/1818134425 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f17c40186c0 0x7f17c401abc0 crc :-1 s=READY pgs=321 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:12.878 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:12.888 7f17eb512700 1 -- 172.21.2.112:0/1818134425 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f17e4190a30 msgr2=0x7f17e409b130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:12.878 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:12.888 7f17eb512700 1 --2- 172.21.2.112:0/1818134425 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f17e4190a30 0x7f17e409b130 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 rx=0x7f17dc00b930 tx=0x7f17dc00bcd0).stop 2021-02-10T11:26:12.878 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:12.888 7f17eb512700 1 -- 172.21.2.112:0/1818134425 shutdown_connections 2021-02-10T11:26:12.878 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:12.888 7f17eb512700 1 --2- 172.21.2.112:0/1818134425 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f17c40186c0 0x7f17c401abc0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:12.879 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:12.888 7f17eb512700 1 --2- 172.21.2.112:0/1818134425 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f17e4190630 0x7f17e4099f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:12.879 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:12.888 7f17eb512700 1 --2- 172.21.2.112:0/1818134425 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f17e4190a30 0x7f17e409b130 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:12.879 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:12.888 7f17eb512700 1 --2- 172.21.2.112:0/1818134425 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f17e40b4c60 0x7f17e4106fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:12.879 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:12.888 7f17eb512700 1 -- 172.21.2.112:0/1818134425 >> 172.21.2.112:0/1818134425 conn(0x7f17e40c40c0 msgr2=0x7f17e40a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:12.879 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:12.888 7f17eb512700 1 -- 172.21.2.112:0/1818134425 shutdown_connections 2021-02-10T11:26:12.879 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:12.888 7f17eb512700 1 -- 172.21.2.112:0/1818134425 wait complete. 2021-02-10T11:26:12.886 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000348641 2021-02-10T11:26:13.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:13.310 7f80479c0700 1 Processor -- start 2021-02-10T11:26:13.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:13.310 7f80479c0700 1 -- start start 2021-02-10T11:26:13.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:13.310 7f80479c0700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f80401b0470 0x7f80401b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:13.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:13.310 7f80479c0700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f80400ef610 0x7f80400efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:13.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:13.310 7f80479c0700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80400f0b50 con 0x7f80400f0030 2021-02-10T11:26:13.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:13.310 7f80479c0700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80400e52e0 con 0x7f80400ef610 2021-02-10T11:26:13.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:13.310 7f80479c0700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80400e5490 con 0x7f80401b0470 2021-02-10T11:26:13.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:13.312 7f8044fdb700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f80400ef610 0x7f80400efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:13.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:13.312 7f80457dc700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f80401b0470 0x7f80401b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:13.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:13.312 7f8045fdd700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7f80400f0030 0x7f80400f10d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:38884/0 (socket says 172.21.2.112:38884) 2021-02-10T11:26:13.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:13.312 7f8045fdd700 1 -- 172.21.2.112:0/687570245 learned_addr learned my addr 172.21.2.112:0/687570245 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:26:13.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:13.312 7f80457dc700 1 -- 172.21.2.112:0/687570245 >> v2:172.21.2.120:3300/0 conn(0x7f80400ef610 msgr2=0x7f80400efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:26:13.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:13.312 7f80457dc700 1 --2- 172.21.2.112:0/687570245 >> v2:172.21.2.120:3300/0 conn(0x7f80400ef610 0x7f80400efa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:13.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:13.312 7f80457dc700 1 -- 172.21.2.112:0/687570245 >> v1:172.21.2.120:6789/0 conn(0x7f80400f0030 legacy=0x7f80400f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:13.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:13.313 7f80457dc700 1 -- 172.21.2.112:0/687570245 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80400f0430 con 0x7f80401b0470 2021-02-10T11:26:13.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:13.312 7f8033fff700 1 -- 172.21.2.112:0/687570245 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2926308673 0 0) 0x7f8038001440 con 0x7f80400f0030 2021-02-10T11:26:13.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:13.313 7f8044fdb700 1 --2- 172.21.2.112:0/687570245 >> v2:172.21.2.120:3300/0 conn(0x7f80400ef610 0x7f80400efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:26:13.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:13.313 7f80457dc700 1 --2- 172.21.2.112:0/687570245 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f80401b0470 0x7f80401b0870 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f8034006650 tx=0x7f8034009a80).ready entity=mon.2 client_cookie=64b8e7febc2bfaa0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:13.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:13.313 7f8033fff700 1 -- 172.21.2.112:0/687570245 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f803400c0c0 con 0x7f80401b0470 2021-02-10T11:26:13.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:13.313 7f8033fff700 1 -- 172.21.2.112:0/687570245 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f803400c340 con 0x7f80401b0470 2021-02-10T11:26:13.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:13.313 7f8033fff700 1 -- 172.21.2.112:0/687570245 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f803400c860 con 0x7f80401b0470 2021-02-10T11:26:13.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:13.313 7f80479c0700 1 -- 172.21.2.112:0/687570245 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f80401b0470 msgr2=0x7f80401b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:13.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:13.313 7f80479c0700 1 --2- 172.21.2.112:0/687570245 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f80401b0470 0x7f80401b0870 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f8034006650 tx=0x7f8034009a80).stop 2021-02-10T11:26:13.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:13.314 7f80479c0700 1 -- 172.21.2.112:0/687570245 shutdown_connections 2021-02-10T11:26:13.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:13.314 7f80479c0700 1 --2- 172.21.2.112:0/687570245 >> v2:172.21.2.120:3300/0 conn(0x7f80400ef610 0x7f80400efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:13.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:13.314 7f80479c0700 1 --2- 172.21.2.112:0/687570245 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f80401b0470 0x7f80401b0870 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:13.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:13.314 7f80479c0700 1 -- 172.21.2.112:0/687570245 >> 172.21.2.112:0/687570245 conn(0x7f80400ce3e0 msgr2=0x7f80400e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:13.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:13.314 7f80479c0700 1 -- 172.21.2.112:0/687570245 shutdown_connections 2021-02-10T11:26:13.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:13.314 7f80479c0700 1 -- 172.21.2.112:0/687570245 wait complete. 2021-02-10T11:26:13.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:13.329 7f80479c0700 1 Processor -- start 2021-02-10T11:26:13.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:13.329 7f80479c0700 1 -- start start 2021-02-10T11:26:13.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:13.329 7f80479c0700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f80400a0420 0x7f80400a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:13.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:13.329 7f80479c0700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8040189d00 0x7f804018a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:13.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:13.329 7f80479c0700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f804018a720 0x7f80400b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:13.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:13.329 7f80479c0700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8040189800 con 0x7f80400a0420 2021-02-10T11:26:13.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:13.329 7f80479c0700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f804018ab20 con 0x7f8040189d00 2021-02-10T11:26:13.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:13.329 7f80479c0700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80400a1140 con 0x7f804018a720 2021-02-10T11:26:13.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:13.329 7f80457dc700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f80400a0420 0x7f80400a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:13.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:13.329 7f80457dc700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f80400a0420 0x7f80400a0820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:45872/0 (socket says 172.21.2.112:45872) 2021-02-10T11:26:13.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:13.329 7f80457dc700 1 -- 172.21.2.112:0/131645296 learned_addr learned my addr 172.21.2.112:0/131645296 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:26:13.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:13.330 7f8044fdb700 1 --2- 172.21.2.112:0/131645296 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8040189d00 0x7f804018a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:13.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:13.330 7f80457dc700 1 -- 172.21.2.112:0/131645296 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f804018a720 msgr2=0x7f80400b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:13.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:13.330 7f80457dc700 1 --2- 172.21.2.112:0/131645296 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f804018a720 0x7f80400b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:13.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:13.330 7f80457dc700 1 -- 172.21.2.112:0/131645296 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8040189d00 msgr2=0x7f804018a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:13.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:13.330 7f80457dc700 1 --2- 172.21.2.112:0/131645296 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8040189d00 0x7f804018a100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:13.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:13.330 7f80457dc700 1 -- 172.21.2.112:0/131645296 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80400b8200 con 0x7f80400a0420 2021-02-10T11:26:13.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:13.330 7f80457dc700 1 --2- 172.21.2.112:0/131645296 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f80400a0420 0x7f80400a0820 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 rx=0x7f8034009630 tx=0x7f803400b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:13.326 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:13.330 7f8031ffb700 1 -- 172.21.2.112:0/131645296 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f80340093a0 con 0x7f80400a0420 2021-02-10T11:26:13.326 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:13.330 7f8031ffb700 1 -- 172.21.2.112:0/131645296 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f803400bd70 con 0x7f80400a0420 2021-02-10T11:26:13.326 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:13.330 7f80479c0700 1 -- 172.21.2.112:0/131645296 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80400b8200 con 0x7f80400a0420 2021-02-10T11:26:13.326 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:13.330 7f80479c0700 1 -- 172.21.2.112:0/131645296 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80400b8200 con 0x7f80400a0420 2021-02-10T11:26:13.326 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:13.330 7f8031ffb700 1 -- 172.21.2.112:0/131645296 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f80340054d0 con 0x7f80400a0420 2021-02-10T11:26:13.326 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:13.331 7f8031ffb700 1 -- 172.21.2.112:0/131645296 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f80340052e0 con 0x7f80400a0420 2021-02-10T11:26:13.326 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:13.331 7f8031ffb700 1 --2- 172.21.2.112:0/131645296 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8020018700 0x7f802001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:13.326 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:13.331 7f8031ffb700 1 -- 172.21.2.112:0/131645296 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f80340327a0 con 0x7f80400a0420 2021-02-10T11:26:13.326 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:13.331 7f8044fdb700 1 --2- 172.21.2.112:0/131645296 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8020018700 0x7f802001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:13.326 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:13.332 7f8044fdb700 1 --2- 172.21.2.112:0/131645296 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8020018700 0x7f802001abc0 crc :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:13.351 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:13.361 7f80479c0700 1 -- 172.21.2.112:0/131645296 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8040105ae0 con 0x7f80400a0420 2021-02-10T11:26:13.353 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:13.364 7f8031ffb700 1 -- 172.21.2.112:0/131645296 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f803401f4e0 con 0x7f80400a0420 2021-02-10T11:26:13.481 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:13.492 7f80479c0700 1 -- 172.21.2.112:0/131645296 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348641", "target": ["mgr", ""]}) v1 -- 0x7f804018ab20 con 0x7f8020018700 2021-02-10T11:26:13.485 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:13.496 7f8031ffb700 1 -- 172.21.2.112:0/131645296 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f803c000940 con 0x7f8020018700 2021-02-10T11:26:13.489 INFO:teuthology.orchestra.run.gibba012.stdout:/volumes/_nogroup/subvolume_0000000000348641/153bf083-b9cc-445e-820b-6dec1dbecfc8 2021-02-10T11:26:13.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:13.501 7f80479c0700 1 -- 172.21.2.112:0/131645296 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8020018700 msgr2=0x7f802001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:13.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:13.501 7f80479c0700 1 --2- 172.21.2.112:0/131645296 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8020018700 0x7f802001abc0 crc :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:13.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:13.501 7f80479c0700 1 -- 172.21.2.112:0/131645296 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f80400a0420 msgr2=0x7f80400a0820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:13.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:13.501 7f80479c0700 1 --2- 172.21.2.112:0/131645296 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f80400a0420 0x7f80400a0820 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 rx=0x7f8034009630 tx=0x7f803400b980).stop 2021-02-10T11:26:13.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:13.502 7f80479c0700 1 -- 172.21.2.112:0/131645296 shutdown_connections 2021-02-10T11:26:13.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:13.502 7f80479c0700 1 --2- 172.21.2.112:0/131645296 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8020018700 0x7f802001abc0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:13.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:13.502 7f80479c0700 1 --2- 172.21.2.112:0/131645296 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8040189d00 0x7f804018a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:13.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:13.502 7f80479c0700 1 --2- 172.21.2.112:0/131645296 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f804018a720 0x7f80400b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:13.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:13.502 7f80479c0700 1 --2- 172.21.2.112:0/131645296 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f80400a0420 0x7f80400a0820 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:13.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:13.502 7f80479c0700 1 -- 172.21.2.112:0/131645296 >> 172.21.2.112:0/131645296 conn(0x7f8040094e10 msgr2=0x7f80400a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:13.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:13.502 7f80479c0700 1 -- 172.21.2.112:0/131645296 shutdown_connections 2021-02-10T11:26:13.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:13.502 7f80479c0700 1 -- 172.21.2.112:0/131645296 wait complete. 2021-02-10T11:26:13.499 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000348641 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000348641/153bf083-b9cc-445e-820b-6dec1dbecfc8 2021-02-10T11:26:13.499 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:13.500 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:13.500 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:13.835 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:26:13.835 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:13.836 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348641/153bf083-b9cc-445e-820b-6dec1dbecfc8/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:14.217 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:26:14.217 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:26:14.217 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0240034 s, 43.7 MB/s 2021-02-10T11:26:14.218 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:14.218 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:14.218 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:14.552 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:26:14.552 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:14.552 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348641/153bf083-b9cc-445e-820b-6dec1dbecfc8/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:14.923 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:26:14.923 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:26:14.923 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0230183 s, 45.6 MB/s 2021-02-10T11:26:14.924 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:14.924 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:14.924 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:15.260 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:26:15.261 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:15.261 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348641/153bf083-b9cc-445e-820b-6dec1dbecfc8/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:15.637 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:26:15.637 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:26:15.637 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.024024 s, 43.6 MB/s 2021-02-10T11:26:15.638 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:15.638 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:15.638 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:15.973 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:26:15.973 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:15.973 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348641/153bf083-b9cc-445e-820b-6dec1dbecfc8/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:16.352 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:26:16.352 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:26:16.352 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0260596 s, 40.2 MB/s 2021-02-10T11:26:16.353 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:16.353 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:16.353 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:16.690 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:26:16.691 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:16.691 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348641/153bf083-b9cc-445e-820b-6dec1dbecfc8/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:17.066 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:26:17.066 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:26:17.067 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.024103 s, 43.5 MB/s 2021-02-10T11:26:17.067 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:17.068 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:17.068 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:17.402 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:26:17.402 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:17.402 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348641/153bf083-b9cc-445e-820b-6dec1dbecfc8/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:17.777 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:26:17.778 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:26:17.778 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0248561 s, 42.2 MB/s 2021-02-10T11:26:17.779 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:17.779 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:17.779 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:18.114 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:26:18.114 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:18.115 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348641/153bf083-b9cc-445e-820b-6dec1dbecfc8/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:18.488 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:26:18.489 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:26:18.489 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0242912 s, 43.2 MB/s 2021-02-10T11:26:18.490 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:18.490 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:18.490 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:18.826 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:26:18.827 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:18.827 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348641/153bf083-b9cc-445e-820b-6dec1dbecfc8/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:19.202 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:26:19.202 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:26:19.202 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.024277 s, 43.2 MB/s 2021-02-10T11:26:19.203 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:19.203 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:19.204 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:19.538 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:26:19.538 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:19.539 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348641/153bf083-b9cc-445e-820b-6dec1dbecfc8/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:19.913 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:26:19.913 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:26:19.913 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0259462 s, 40.4 MB/s 2021-02-10T11:26:19.914 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:19.914 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:19.914 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:20.250 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:26:20.251 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:20.251 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348641/153bf083-b9cc-445e-820b-6dec1dbecfc8/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:20.627 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:26:20.628 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:26:20.628 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0255475 s, 41.0 MB/s 2021-02-10T11:26:20.629 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000348642 2021-02-10T11:26:21.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.054 7f8db9330700 1 Processor -- start 2021-02-10T11:26:21.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.054 7f8db9330700 1 -- start start 2021-02-10T11:26:21.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.054 7f8db9330700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f8db40f05f0 0x7f8db40f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:21.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.054 7f8db9330700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8db41b0c70 con 0x7f8db41b0470 2021-02-10T11:26:21.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.054 7f8db9330700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8db40f0b50 con 0x7f8db41b0870 2021-02-10T11:26:21.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.054 7f8db9330700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8db40e5ea0 con 0x7f8db40f05f0 2021-02-10T11:26:21.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.054 7f8db307d700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f8db41b0470 0x7f8db40ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:45338/0 (socket says 172.21.2.112:45338) 2021-02-10T11:26:21.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.054 7f8db307d700 1 -- 172.21.2.112:0/3621678004 learned_addr learned my addr 172.21.2.112:0/3621678004 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:26:21.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.054 7f8db387e700 1 --2- 172.21.2.112:0/3621678004 >> v2:172.21.2.112:3300/0 conn(0x7f8db40f05f0 0x7f8db40f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:21.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.055 7f8db387e700 1 -- 172.21.2.112:0/3621678004 >> v1:172.21.2.120:6789/0 conn(0x7f8db41b0870 legacy=0x7f8db40efe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:26:21.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.055 7f8db387e700 1 -- 172.21.2.112:0/3621678004 >> v1:172.21.2.112:6789/0 conn(0x7f8db41b0470 legacy=0x7f8db40ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:21.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.055 7f8db207b700 1 -- 172.21.2.112:0/3621678004 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2608972812 0 0) 0x7f8d9c000ce0 con 0x7f8db41b0470 2021-02-10T11:26:21.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.055 7f8db387e700 1 -- 172.21.2.112:0/3621678004 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8db40e52e0 con 0x7f8db40f05f0 2021-02-10T11:26:21.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.055 7f8db387e700 1 --2- 172.21.2.112:0/3621678004 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8db40f05f0 0x7f8db40f20e0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 rx=0x7f8dac006e30 tx=0x7f8dac009de0).ready entity=mon.0 client_cookie=fc261e39e4803f8b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:21.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.055 7f8db207b700 1 -- 172.21.2.112:0/3621678004 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8dac00b300 con 0x7f8db40f05f0 2021-02-10T11:26:21.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.055 7f8db207b700 1 -- 172.21.2.112:0/3621678004 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8dac00b680 con 0x7f8db40f05f0 2021-02-10T11:26:21.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.055 7f8db207b700 1 -- 172.21.2.112:0/3621678004 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8dac00bba0 con 0x7f8db40f05f0 2021-02-10T11:26:21.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.055 7f8db9330700 1 -- 172.21.2.112:0/3621678004 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8db40f05f0 msgr2=0x7f8db40f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:21.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.055 7f8db9330700 1 --2- 172.21.2.112:0/3621678004 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8db40f05f0 0x7f8db40f20e0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 rx=0x7f8dac006e30 tx=0x7f8dac009de0).stop 2021-02-10T11:26:21.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.056 7f8db9330700 1 -- 172.21.2.112:0/3621678004 shutdown_connections 2021-02-10T11:26:21.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.056 7f8db9330700 1 --2- 172.21.2.112:0/3621678004 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8db40f05f0 0x7f8db40f20e0 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:21.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.056 7f8db9330700 1 -- 172.21.2.112:0/3621678004 >> 172.21.2.112:0/3621678004 conn(0x7f8db40ce3e0 msgr2=0x7f8db40e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:21.063 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.056 7f8db9330700 1 -- 172.21.2.112:0/3621678004 shutdown_connections 2021-02-10T11:26:21.063 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.056 7f8db9330700 1 -- 172.21.2.112:0/3621678004 wait complete. 2021-02-10T11:26:21.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.071 7f8db9330700 1 Processor -- start 2021-02-10T11:26:21.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.071 7f8db9330700 1 -- start start 2021-02-10T11:26:21.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.071 7f8db9330700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8db419c090 0x7f8db409d090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:21.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.071 7f8db9330700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8db40be650 0x7f8db40bd830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:21.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.071 7f8db9330700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8db40bde50 0x7f8db409de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:21.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.071 7f8db9330700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8db409d6b0 con 0x7f8db40bde50 2021-02-10T11:26:21.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.071 7f8db9330700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8db409d900 con 0x7f8db40be650 2021-02-10T11:26:21.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.071 7f8db9330700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8db40be250 con 0x7f8db419c090 2021-02-10T11:26:21.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.072 7f8db387e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8db40bde50 0x7f8db409de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:21.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.072 7f8db387e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8db40bde50 0x7f8db409de80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:45886/0 (socket says 172.21.2.112:45886) 2021-02-10T11:26:21.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.072 7f8db387e700 1 -- 172.21.2.112:0/68335016 learned_addr learned my addr 172.21.2.112:0/68335016 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:26:21.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.072 7f8db387e700 1 -- 172.21.2.112:0/68335016 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8db419c090 msgr2=0x7f8db409d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:21.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.072 7f8db387e700 1 --2- 172.21.2.112:0/68335016 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8db419c090 0x7f8db409d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:21.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.072 7f8db387e700 1 -- 172.21.2.112:0/68335016 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8db40be650 msgr2=0x7f8db40bd830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:21.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.072 7f8db287c700 1 --2- 172.21.2.112:0/68335016 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8db40be650 0x7f8db40bd830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:21.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.072 7f8db387e700 1 --2- 172.21.2.112:0/68335016 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8db40be650 0x7f8db40bd830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:21.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.072 7f8db387e700 1 -- 172.21.2.112:0/68335016 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8db409e4a0 con 0x7f8db40bde50 2021-02-10T11:26:21.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.072 7f8db287c700 1 --2- 172.21.2.112:0/68335016 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8db40be650 0x7f8db40bd830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:26:21.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.072 7f8db387e700 1 --2- 172.21.2.112:0/68335016 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8db40bde50 0x7f8db409de80 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 rx=0x7f8dac00b910 tx=0x7f8dac00bcb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:21.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.073 7f8dabfff700 1 -- 172.21.2.112:0/68335016 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8dac00bdf0 con 0x7f8db40bde50 2021-02-10T11:26:21.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.073 7f8dabfff700 1 -- 172.21.2.112:0/68335016 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8dac001270 con 0x7f8db40bde50 2021-02-10T11:26:21.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.073 7f8dabfff700 1 -- 172.21.2.112:0/68335016 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8dac001700 con 0x7f8db40bde50 2021-02-10T11:26:21.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.073 7f8db9330700 1 -- 172.21.2.112:0/68335016 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8db409e4a0 con 0x7f8db40bde50 2021-02-10T11:26:21.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.073 7f8db9330700 1 -- 172.21.2.112:0/68335016 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8db409e4a0 con 0x7f8db40bde50 2021-02-10T11:26:21.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.073 7f8dabfff700 1 -- 172.21.2.112:0/68335016 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8dac0016c0 con 0x7f8db40bde50 2021-02-10T11:26:21.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.073 7f8dabfff700 1 --2- 172.21.2.112:0/68335016 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8d980186c0 0x7f8d9801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:21.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.073 7f8dabfff700 1 -- 172.21.2.112:0/68335016 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f8dac032b40 con 0x7f8db40bde50 2021-02-10T11:26:21.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.074 7f8db307d700 1 --2- 172.21.2.112:0/68335016 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8d980186c0 0x7f8d9801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:21.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.075 7f8db307d700 1 --2- 172.21.2.112:0/68335016 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8d980186c0 0x7f8d9801abc0 crc :-1 s=READY pgs=323 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:21.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.104 7f8db9330700 1 -- 172.21.2.112:0/68335016 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8db4097e00 con 0x7f8db40bde50 2021-02-10T11:26:21.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.107 7f8dabfff700 1 -- 172.21.2.112:0/68335016 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8dac006dc0 con 0x7f8db40bde50 2021-02-10T11:26:21.224 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.234 7f8db9330700 1 -- 172.21.2.112:0/68335016 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348642", "target": ["mgr", ""]}) v1 -- 0x7f8db4097e00 con 0x7f8d980186c0 2021-02-10T11:26:21.250 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.261 7f8dabfff700 1 -- 172.21.2.112:0/68335016 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f8d9c006d80 con 0x7f8d980186c0 2021-02-10T11:26:21.258 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.268 7f8db9330700 1 -- 172.21.2.112:0/68335016 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8d980186c0 msgr2=0x7f8d9801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:21.258 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.268 7f8db9330700 1 --2- 172.21.2.112:0/68335016 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8d980186c0 0x7f8d9801abc0 crc :-1 s=READY pgs=323 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:21.258 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.268 7f8db9330700 1 -- 172.21.2.112:0/68335016 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8db40bde50 msgr2=0x7f8db409de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:21.258 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.268 7f8db9330700 1 --2- 172.21.2.112:0/68335016 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8db40bde50 0x7f8db409de80 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 rx=0x7f8dac00b910 tx=0x7f8dac00bcb0).stop 2021-02-10T11:26:21.258 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.268 7f8db9330700 1 -- 172.21.2.112:0/68335016 shutdown_connections 2021-02-10T11:26:21.258 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.268 7f8db9330700 1 --2- 172.21.2.112:0/68335016 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8d980186c0 0x7f8d9801abc0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:21.258 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.268 7f8db9330700 1 --2- 172.21.2.112:0/68335016 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8db40be650 0x7f8db40bd830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:21.258 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.268 7f8db9330700 1 --2- 172.21.2.112:0/68335016 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8db40bde50 0x7f8db409de80 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:21.258 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.268 7f8db9330700 1 --2- 172.21.2.112:0/68335016 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8db419c090 0x7f8db409d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:21.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.268 7f8db9330700 1 -- 172.21.2.112:0/68335016 >> 172.21.2.112:0/68335016 conn(0x7f8db40bf4f0 msgr2=0x7f8db418f460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:21.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.268 7f8db9330700 1 -- 172.21.2.112:0/68335016 shutdown_connections 2021-02-10T11:26:21.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.268 7f8db9330700 1 -- 172.21.2.112:0/68335016 wait complete. 2021-02-10T11:26:21.265 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000348642 2021-02-10T11:26:21.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.692 7f78e5cd6700 1 Processor -- start 2021-02-10T11:26:21.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.692 7f78e5cd6700 1 -- start start 2021-02-10T11:26:21.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.693 7f78e5cd6700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f78e01b0470 0x7f78e01b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:21.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.693 7f78e5cd6700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78e00f0b50 con 0x7f78e00f20e0 2021-02-10T11:26:21.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.693 7f78e5cd6700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78e00f24e0 con 0x7f78e00ef610 2021-02-10T11:26:21.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.693 7f78e5cd6700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78e00e5ea0 con 0x7f78e01b0470 2021-02-10T11:26:21.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.693 7f78df7fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f78e01b0470 0x7f78e01b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:21.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.693 7f78dffff700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f78e00f20e0 0x7f78e00f0200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:45354/0 (socket says 172.21.2.112:45354) 2021-02-10T11:26:21.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.693 7f78dffff700 1 -- 172.21.2.112:0/1596374690 learned_addr learned my addr 172.21.2.112:0/1596374690 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:26:21.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.693 7f78de7fc700 1 -- 172.21.2.112:0/1596374690 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3655569274 0 0) 0x7f78d8000f90 con 0x7f78e00f20e0 2021-02-10T11:26:21.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.693 7f78de7fc700 1 -- 172.21.2.112:0/1596374690 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78cc002fe0 con 0x7f78e00f20e0 2021-02-10T11:26:21.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.693 7f78df7fe700 1 -- 172.21.2.112:0/1596374690 >> v1:172.21.2.120:6789/0 conn(0x7f78e00ef610 legacy=0x7f78e00efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:21.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.693 7f78df7fe700 1 -- 172.21.2.112:0/1596374690 >> v1:172.21.2.112:6789/0 conn(0x7f78e00f20e0 legacy=0x7f78e00f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:21.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.693 7f78df7fe700 1 -- 172.21.2.112:0/1596374690 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78e00e52e0 con 0x7f78e01b0470 2021-02-10T11:26:21.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.694 7f78df7fe700 1 --2- 172.21.2.112:0/1596374690 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f78e01b0470 0x7f78e01b0870 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 rx=0x7f78c8006650 tx=0x7f78c8009a00).ready entity=mon.0 client_cookie=3b3276e56f6e8edc server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:21.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.694 7f78de7fc700 1 -- 172.21.2.112:0/1596374690 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f78c80009b0 con 0x7f78e01b0470 2021-02-10T11:26:21.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.694 7f78de7fc700 1 -- 172.21.2.112:0/1596374690 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78c800c0c0 con 0x7f78e01b0470 2021-02-10T11:26:21.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.694 7f78de7fc700 1 -- 172.21.2.112:0/1596374690 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f78c800c5a0 con 0x7f78e01b0470 2021-02-10T11:26:21.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.694 7f78e5cd6700 1 -- 172.21.2.112:0/1596374690 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f78e01b0470 msgr2=0x7f78e01b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:21.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.694 7f78e5cd6700 1 --2- 172.21.2.112:0/1596374690 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f78e01b0470 0x7f78e01b0870 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 rx=0x7f78c8006650 tx=0x7f78c8009a00).stop 2021-02-10T11:26:21.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.694 7f78e5cd6700 1 -- 172.21.2.112:0/1596374690 shutdown_connections 2021-02-10T11:26:21.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.694 7f78e5cd6700 1 --2- 172.21.2.112:0/1596374690 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f78e01b0470 0x7f78e01b0870 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:21.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.694 7f78e5cd6700 1 -- 172.21.2.112:0/1596374690 >> 172.21.2.112:0/1596374690 conn(0x7f78e00ce3e0 msgr2=0x7f78e00e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:21.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.694 7f78e5cd6700 1 -- 172.21.2.112:0/1596374690 shutdown_connections 2021-02-10T11:26:21.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.694 7f78e5cd6700 1 -- 172.21.2.112:0/1596374690 wait complete. 2021-02-10T11:26:21.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.709 7f78e5cd6700 1 Processor -- start 2021-02-10T11:26:21.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.709 7f78e5cd6700 1 -- start start 2021-02-10T11:26:21.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.709 7f78e5cd6700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f78e00c36a0 0x7f78e00c2b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:21.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.709 7f78e5cd6700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f78e00c3190 0x7f78e00b1d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:21.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.709 7f78e5cd6700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f78e00b2370 0x7f78e00b2d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:21.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.709 7f78e5cd6700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78e00b27c0 con 0x7f78e00c3190 2021-02-10T11:26:21.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.709 7f78e5cd6700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78e00b3360 con 0x7f78e00b2370 2021-02-10T11:26:21.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.709 7f78e5cd6700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78e00b3540 con 0x7f78e00c36a0 2021-02-10T11:26:21.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.710 7f78deffd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f78e00c3190 0x7f78e00b1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:21.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.710 7f78deffd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f78e00c3190 0x7f78e00b1d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:45894/0 (socket says 172.21.2.112:45894) 2021-02-10T11:26:21.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.710 7f78deffd700 1 -- 172.21.2.112:0/1009823368 learned_addr learned my addr 172.21.2.112:0/1009823368 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:26:21.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.710 7f78dffff700 1 --2- 172.21.2.112:0/1009823368 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f78e00b2370 0x7f78e00b2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:21.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.710 7f78df7fe700 1 --2- 172.21.2.112:0/1009823368 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f78e00c36a0 0x7f78e00c2b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:21.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.710 7f78deffd700 1 -- 172.21.2.112:0/1009823368 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f78e00c36a0 msgr2=0x7f78e00c2b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:21.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.710 7f78deffd700 1 --2- 172.21.2.112:0/1009823368 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f78e00c36a0 0x7f78e00c2b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:21.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.710 7f78deffd700 1 -- 172.21.2.112:0/1009823368 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f78e00b2370 msgr2=0x7f78e00b2d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:21.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.710 7f78deffd700 1 --2- 172.21.2.112:0/1009823368 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f78e00b2370 0x7f78e00b2d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:21.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.710 7f78deffd700 1 -- 172.21.2.112:0/1009823368 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78e00b3760 con 0x7f78e00c3190 2021-02-10T11:26:21.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.710 7f78deffd700 1 --2- 172.21.2.112:0/1009823368 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f78e00c3190 0x7f78e00b1d50 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 rx=0x7f78d000b5b0 tx=0x7f78d000b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:21.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.711 7f78d7fff700 1 -- 172.21.2.112:0/1009823368 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f78d0006240 con 0x7f78e00c3190 2021-02-10T11:26:21.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.711 7f78d7fff700 1 -- 172.21.2.112:0/1009823368 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78d00054a0 con 0x7f78e00c3190 2021-02-10T11:26:21.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.711 7f78d7fff700 1 -- 172.21.2.112:0/1009823368 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f78d0005900 con 0x7f78e00c3190 2021-02-10T11:26:21.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.711 7f78e5cd6700 1 -- 172.21.2.112:0/1009823368 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78e00b3760 con 0x7f78e00c3190 2021-02-10T11:26:21.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.711 7f78e5cd6700 1 -- 172.21.2.112:0/1009823368 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78e00b3cf0 con 0x7f78e00c3190 2021-02-10T11:26:21.704 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.711 7f78d7fff700 1 -- 172.21.2.112:0/1009823368 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f78d0006030 con 0x7f78e00c3190 2021-02-10T11:26:21.704 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.711 7f78d7fff700 1 --2- 172.21.2.112:0/1009823368 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f78c40186c0 0x7f78c401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:21.704 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.711 7f78d7fff700 1 -- 172.21.2.112:0/1009823368 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f78d0032440 con 0x7f78e00c3190 2021-02-10T11:26:21.704 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.712 7f78df7fe700 1 --2- 172.21.2.112:0/1009823368 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f78c40186c0 0x7f78c401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:21.704 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.712 7f78df7fe700 1 --2- 172.21.2.112:0/1009823368 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f78c40186c0 0x7f78c401abc0 crc :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:21.732 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.742 7f78e5cd6700 1 -- 172.21.2.112:0/1009823368 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78e00edd10 con 0x7f78e00c3190 2021-02-10T11:26:21.734 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.745 7f78d7fff700 1 -- 172.21.2.112:0/1009823368 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f78d0030d70 con 0x7f78e00c3190 2021-02-10T11:26:21.865 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.875 7f78e5cd6700 1 -- 172.21.2.112:0/1009823368 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348642", "target": ["mgr", ""]}) v1 -- 0x7f78e00b27c0 con 0x7f78c40186c0 2021-02-10T11:26:21.867 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.879 7f78d7fff700 1 -- 172.21.2.112:0/1009823368 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f78c8006450 con 0x7f78c40186c0 2021-02-10T11:26:21.868 INFO:teuthology.orchestra.run.gibba012.stdout:/volumes/_nogroup/subvolume_0000000000348642/32902b46-10d5-44de-8af1-a2735d819876 2021-02-10T11:26:21.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.879 7f78e5cd6700 1 -- 172.21.2.112:0/1009823368 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f78c40186c0 msgr2=0x7f78c401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:21.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.879 7f78e5cd6700 1 --2- 172.21.2.112:0/1009823368 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f78c40186c0 0x7f78c401abc0 crc :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:21.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.879 7f78e5cd6700 1 -- 172.21.2.112:0/1009823368 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f78e00c3190 msgr2=0x7f78e00b1d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:21.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.879 7f78e5cd6700 1 --2- 172.21.2.112:0/1009823368 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f78e00c3190 0x7f78e00b1d50 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 rx=0x7f78d000b5b0 tx=0x7f78d000b950).stop 2021-02-10T11:26:21.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.880 7f78e5cd6700 1 -- 172.21.2.112:0/1009823368 shutdown_connections 2021-02-10T11:26:21.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.880 7f78e5cd6700 1 --2- 172.21.2.112:0/1009823368 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f78c40186c0 0x7f78c401abc0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:21.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.880 7f78e5cd6700 1 --2- 172.21.2.112:0/1009823368 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f78e00b2370 0x7f78e00b2d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:21.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.880 7f78e5cd6700 1 --2- 172.21.2.112:0/1009823368 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f78e00c3190 0x7f78e00b1d50 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:21.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.880 7f78e5cd6700 1 --2- 172.21.2.112:0/1009823368 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f78e00c36a0 0x7f78e00c2b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:21.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.880 7f78e5cd6700 1 -- 172.21.2.112:0/1009823368 >> 172.21.2.112:0/1009823368 conn(0x7f78e009f630 msgr2=0x7f78e00a0130 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:21.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.880 7f78e5cd6700 1 -- 172.21.2.112:0/1009823368 shutdown_connections 2021-02-10T11:26:21.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:21.880 7f78e5cd6700 1 -- 172.21.2.112:0/1009823368 wait complete. 2021-02-10T11:26:21.877 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000348642 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000348642/32902b46-10d5-44de-8af1-a2735d819876 2021-02-10T11:26:21.878 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:21.878 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:21.878 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:22.213 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:26:22.213 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:22.214 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348642/32902b46-10d5-44de-8af1-a2735d819876/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:22.594 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:26:22.595 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:26:22.595 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0232644 s, 45.1 MB/s 2021-02-10T11:26:22.596 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:22.596 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:22.596 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:22.931 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:26:22.931 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:22.932 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348642/32902b46-10d5-44de-8af1-a2735d819876/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:23.307 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:26:23.307 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:26:23.307 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0233109 s, 45.0 MB/s 2021-02-10T11:26:23.308 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:23.308 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:23.308 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:23.643 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:26:23.643 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:23.644 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348642/32902b46-10d5-44de-8af1-a2735d819876/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:24.022 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:26:24.022 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:26:24.022 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0252925 s, 41.5 MB/s 2021-02-10T11:26:24.023 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:24.023 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:24.023 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:24.358 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:26:24.358 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:24.359 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348642/32902b46-10d5-44de-8af1-a2735d819876/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:24.747 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:26:24.748 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:26:24.748 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0241885 s, 43.4 MB/s 2021-02-10T11:26:24.749 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:24.749 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:24.749 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:25.090 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:26:25.090 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:25.091 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348642/32902b46-10d5-44de-8af1-a2735d819876/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:25.466 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:26:25.467 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:26:25.467 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0233957 s, 44.8 MB/s 2021-02-10T11:26:25.468 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:25.468 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:25.468 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:25.803 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:26:25.803 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:25.804 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348642/32902b46-10d5-44de-8af1-a2735d819876/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:26.179 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:26:26.179 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:26:26.179 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.023744 s, 44.2 MB/s 2021-02-10T11:26:26.180 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:26.180 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:26.180 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:26.516 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:26:26.516 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:26.517 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348642/32902b46-10d5-44de-8af1-a2735d819876/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:26.893 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:26:26.894 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:26:26.894 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0254815 s, 41.2 MB/s 2021-02-10T11:26:26.895 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:26.895 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:26.895 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:27.230 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:26:27.230 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:27.231 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348642/32902b46-10d5-44de-8af1-a2735d819876/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:27.608 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:26:27.608 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:26:27.608 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0254028 s, 41.3 MB/s 2021-02-10T11:26:27.609 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:27.609 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:27.609 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:27.944 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:26:27.944 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:27.944 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348642/32902b46-10d5-44de-8af1-a2735d819876/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:28.321 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:26:28.321 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:26:28.321 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.024487 s, 42.8 MB/s 2021-02-10T11:26:28.322 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:28.322 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:28.323 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:28.657 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:26:28.657 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:28.657 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348642/32902b46-10d5-44de-8af1-a2735d819876/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:29.034 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:26:29.034 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:26:29.034 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0251466 s, 41.7 MB/s 2021-02-10T11:26:29.035 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000348643 2021-02-10T11:26:29.466 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:29.461 7fbf3d6d6700 1 Processor -- start 2021-02-10T11:26:29.466 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:29.461 7fbf3d6d6700 1 -- start start 2021-02-10T11:26:29.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:29.461 7fbf3d6d6700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fbf381b0470 0x7fbf381b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:29.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:29.461 7fbf3d6d6700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf380f0b50 con 0x7fbf380f20e0 2021-02-10T11:26:29.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:29.461 7fbf3d6d6700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf380f24e0 con 0x7fbf380ef610 2021-02-10T11:26:29.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:29.461 7fbf3d6d6700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf380e5ea0 con 0x7fbf381b0470 2021-02-10T11:26:29.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:29.461 7fbf372bf700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fbf381b0470 0x7fbf381b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:29.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:29.461 7fbf372bf700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fbf381b0470 0x7fbf381b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:45900/0 (socket says 172.21.2.112:45900) 2021-02-10T11:26:29.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:29.461 7fbf372bf700 1 -- 172.21.2.112:0/2877102418 learned_addr learned my addr 172.21.2.112:0/2877102418 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:26:29.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:29.461 7fbf37ac0700 1 --1- 172.21.2.112:0/2877102418 >> v1:172.21.2.112:6789/0 conn(0x7fbf380f20e0 0x7fbf380f0200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:45362/0 (socket says 172.21.2.112:45362) 2021-02-10T11:26:29.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:29.462 7fbf372bf700 1 -- 172.21.2.112:0/2877102418 >> v1:172.21.2.120:6789/0 conn(0x7fbf380ef610 legacy=0x7fbf380efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:29.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:29.462 7fbf372bf700 1 -- 172.21.2.112:0/2877102418 >> v1:172.21.2.112:6789/0 conn(0x7fbf380f20e0 legacy=0x7fbf380f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:29.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:29.462 7fbf372bf700 1 -- 172.21.2.112:0/2877102418 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf380e52e0 con 0x7fbf381b0470 2021-02-10T11:26:29.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:29.462 7fbf372bf700 1 --2- 172.21.2.112:0/2877102418 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbf381b0470 0x7fbf381b0870 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 rx=0x7fbf28006700 tx=0x7fbf28009b80).ready entity=mon.0 client_cookie=3ce1367a57fb30a8 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:29.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:29.462 7fbf362bd700 1 -- 172.21.2.112:0/2877102418 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbf2800c0c0 con 0x7fbf381b0470 2021-02-10T11:26:29.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:29.462 7fbf362bd700 1 -- 172.21.2.112:0/2877102418 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf2800c340 con 0x7fbf381b0470 2021-02-10T11:26:29.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:29.462 7fbf362bd700 1 -- 172.21.2.112:0/2877102418 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbf2800c860 con 0x7fbf381b0470 2021-02-10T11:26:29.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:29.462 7fbf3d6d6700 1 -- 172.21.2.112:0/2877102418 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbf381b0470 msgr2=0x7fbf381b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:29.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:29.462 7fbf3d6d6700 1 --2- 172.21.2.112:0/2877102418 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbf381b0470 0x7fbf381b0870 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 rx=0x7fbf28006700 tx=0x7fbf28009b80).stop 2021-02-10T11:26:29.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:29.463 7fbf3d6d6700 1 -- 172.21.2.112:0/2877102418 shutdown_connections 2021-02-10T11:26:29.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:29.463 7fbf3d6d6700 1 --2- 172.21.2.112:0/2877102418 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbf381b0470 0x7fbf381b0870 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:29.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:29.463 7fbf3d6d6700 1 -- 172.21.2.112:0/2877102418 >> 172.21.2.112:0/2877102418 conn(0x7fbf380ce3e0 msgr2=0x7fbf380e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:29.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:29.463 7fbf3d6d6700 1 -- 172.21.2.112:0/2877102418 shutdown_connections 2021-02-10T11:26:29.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:29.463 7fbf3d6d6700 1 -- 172.21.2.112:0/2877102418 wait complete. 2021-02-10T11:26:29.470 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:29.477 7fbf3d6d6700 1 Processor -- start 2021-02-10T11:26:29.470 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:29.477 7fbf3d6d6700 1 -- start start 2021-02-10T11:26:29.470 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:29.478 7fbf3d6d6700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fbf381c6980 0x7fbf380ae870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:29.470 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:29.478 7fbf3d6d6700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbf38088590 0x7fbf380aee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:29.470 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:29.478 7fbf3d6d6700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbf380a7b20 0x7fbf380ac850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:29.470 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:29.478 7fbf3d6d6700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf380ae2f0 con 0x7fbf380a7b20 2021-02-10T11:26:29.470 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:29.478 7fbf3d6d6700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf380af4b0 con 0x7fbf381c6980 2021-02-10T11:26:29.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:29.478 7fbf3d6d6700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf380af690 con 0x7fbf38088590 2021-02-10T11:26:29.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:29.478 7fbf37ac0700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbf380a7b20 0x7fbf380ac850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:29.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:29.478 7fbf37ac0700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbf380a7b20 0x7fbf380ac850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:45904/0 (socket says 172.21.2.112:45904) 2021-02-10T11:26:29.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:29.478 7fbf37ac0700 1 -- 172.21.2.112:0/1065488169 learned_addr learned my addr 172.21.2.112:0/1065488169 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:26:29.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:29.478 7fbf372bf700 1 --2- 172.21.2.112:0/1065488169 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fbf381c6980 0x7fbf380ae870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:29.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:29.478 7fbf36abe700 1 --2- 172.21.2.112:0/1065488169 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbf38088590 0x7fbf380aee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:29.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:29.478 7fbf37ac0700 1 -- 172.21.2.112:0/1065488169 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbf38088590 msgr2=0x7fbf380aee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:29.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:29.478 7fbf37ac0700 1 --2- 172.21.2.112:0/1065488169 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbf38088590 0x7fbf380aee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:29.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:29.478 7fbf37ac0700 1 -- 172.21.2.112:0/1065488169 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fbf381c6980 msgr2=0x7fbf380ae870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:29.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:29.478 7fbf37ac0700 1 --2- 172.21.2.112:0/1065488169 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fbf381c6980 0x7fbf380ae870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:29.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:29.478 7fbf37ac0700 1 -- 172.21.2.112:0/1065488169 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf380ace70 con 0x7fbf380a7b20 2021-02-10T11:26:29.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:29.478 7fbf37ac0700 1 --2- 172.21.2.112:0/1065488169 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbf380a7b20 0x7fbf380ac850 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 rx=0x7fbf30007f90 tx=0x7fbf3000ac50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:29.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:29.478 7fbf2ffff700 1 -- 172.21.2.112:0/1065488169 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbf30001270 con 0x7fbf380a7b20 2021-02-10T11:26:29.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:29.478 7fbf2ffff700 1 -- 172.21.2.112:0/1065488169 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf30007020 con 0x7fbf380a7b20 2021-02-10T11:26:29.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:29.478 7fbf2ffff700 1 -- 172.21.2.112:0/1065488169 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbf300073f0 con 0x7fbf380a7b20 2021-02-10T11:26:29.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:29.478 7fbf3d6d6700 1 -- 172.21.2.112:0/1065488169 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf380ace70 con 0x7fbf380a7b20 2021-02-10T11:26:29.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:29.479 7fbf3d6d6700 1 -- 172.21.2.112:0/1065488169 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf380ace70 con 0x7fbf380a7b20 2021-02-10T11:26:29.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:29.479 7fbf2ffff700 1 -- 172.21.2.112:0/1065488169 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fbf30007200 con 0x7fbf380a7b20 2021-02-10T11:26:29.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:29.479 7fbf2ffff700 1 --2- 172.21.2.112:0/1065488169 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fbf1c0186d0 0x7fbf1c01abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:29.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:29.479 7fbf2ffff700 1 -- 172.21.2.112:0/1065488169 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fbf30032b10 con 0x7fbf380a7b20 2021-02-10T11:26:29.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:29.479 7fbf372bf700 1 --2- 172.21.2.112:0/1065488169 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fbf1c0186d0 0x7fbf1c01abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:29.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:29.480 7fbf372bf700 1 --2- 172.21.2.112:0/1065488169 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fbf1c0186d0 0x7fbf1c01abd0 crc :-1 s=READY pgs=325 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:29.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:29.511 7fbf3d6d6700 1 -- 172.21.2.112:0/1065488169 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf3807e6b0 con 0x7fbf380a7b20 2021-02-10T11:26:29.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:29.513 7fbf2ffff700 1 -- 172.21.2.112:0/1065488169 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fbf30001270 con 0x7fbf380a7b20 2021-02-10T11:26:29.630 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:29.640 7fbf3d6d6700 1 -- 172.21.2.112:0/1065488169 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348643", "target": ["mgr", ""]}) v1 -- 0x7fbf3807e6b0 con 0x7fbf1c0186d0 2021-02-10T11:26:29.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:29.663 7fbf2ffff700 1 -- 172.21.2.112:0/1065488169 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fbf28009100 con 0x7fbf1c0186d0 2021-02-10T11:26:29.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:29.673 7fbf3d6d6700 1 -- 172.21.2.112:0/1065488169 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fbf1c0186d0 msgr2=0x7fbf1c01abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:29.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:29.673 7fbf3d6d6700 1 --2- 172.21.2.112:0/1065488169 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fbf1c0186d0 0x7fbf1c01abd0 crc :-1 s=READY pgs=325 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:29.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:29.673 7fbf3d6d6700 1 -- 172.21.2.112:0/1065488169 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbf380a7b20 msgr2=0x7fbf380ac850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:29.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:29.673 7fbf3d6d6700 1 --2- 172.21.2.112:0/1065488169 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbf380a7b20 0x7fbf380ac850 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 rx=0x7fbf30007f90 tx=0x7fbf3000ac50).stop 2021-02-10T11:26:29.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:29.673 7fbf3d6d6700 1 -- 172.21.2.112:0/1065488169 shutdown_connections 2021-02-10T11:26:29.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:29.673 7fbf3d6d6700 1 --2- 172.21.2.112:0/1065488169 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fbf1c0186d0 0x7fbf1c01abd0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:29.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:29.673 7fbf3d6d6700 1 --2- 172.21.2.112:0/1065488169 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbf380a7b20 0x7fbf380ac850 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:29.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:29.673 7fbf3d6d6700 1 --2- 172.21.2.112:0/1065488169 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbf38088590 0x7fbf380aee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:29.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:29.673 7fbf3d6d6700 1 --2- 172.21.2.112:0/1065488169 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fbf381c6980 0x7fbf380ae870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:29.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:29.673 7fbf3d6d6700 1 -- 172.21.2.112:0/1065488169 >> 172.21.2.112:0/1065488169 conn(0x7fbf380ac310 msgr2=0x7fbf380a49d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:29.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:29.674 7fbf3d6d6700 1 -- 172.21.2.112:0/1065488169 shutdown_connections 2021-02-10T11:26:29.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:29.674 7fbf3d6d6700 1 -- 172.21.2.112:0/1065488169 wait complete. 2021-02-10T11:26:29.897 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000348643 2021-02-10T11:26:30.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:30.324 7fd1f244f700 1 Processor -- start 2021-02-10T11:26:30.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:30.324 7fd1f244f700 1 -- start start 2021-02-10T11:26:30.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:30.324 7fd1f244f700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fd1ec1b0470 0x7fd1ec1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:30.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:30.324 7fd1f244f700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fd1ec0ef610 0x7fd1ec0efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:30.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:30.324 7fd1f244f700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1ec0f0b50 con 0x7fd1ec0f0030 2021-02-10T11:26:30.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:30.324 7fd1f244f700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1ec0e52e0 con 0x7fd1ec0ef610 2021-02-10T11:26:30.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:30.324 7fd1f244f700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1ec0e5490 con 0x7fd1ec1b0470 2021-02-10T11:26:30.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:30.324 7fd1eb7fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fd1ec0ef610 0x7fd1ec0efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:30.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:30.324 7fd1eb7fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fd1ec0ef610 0x7fd1ec0efa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:45910/0 (socket says 172.21.2.112:45910) 2021-02-10T11:26:30.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:30.324 7fd1eb7fe700 1 -- 172.21.2.112:0/2860433352 learned_addr learned my addr 172.21.2.112:0/2860433352 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:26:30.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:30.324 7fd1ebfff700 1 --2- 172.21.2.112:0/2860433352 >> v2:172.21.2.120:3300/0 conn(0x7fd1ec1b0470 0x7fd1ec1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:30.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:30.325 7fd1eb7fe700 1 -- 172.21.2.112:0/2860433352 >> v2:172.21.2.120:3300/0 conn(0x7fd1ec1b0470 msgr2=0x7fd1ec1b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:26:30.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:30.325 7fd1eb7fe700 1 --2- 172.21.2.112:0/2860433352 >> v2:172.21.2.120:3300/0 conn(0x7fd1ec1b0470 0x7fd1ec1b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:30.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:30.325 7fd1eb7fe700 1 -- 172.21.2.112:0/2860433352 >> v1:172.21.2.120:6789/0 conn(0x7fd1ec0f0030 legacy=0x7fd1ec0f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:30.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:30.325 7fd1eb7fe700 1 -- 172.21.2.112:0/2860433352 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1ec0f0430 con 0x7fd1ec0ef610 2021-02-10T11:26:30.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:30.325 7fd1eb7fe700 1 --2- 172.21.2.112:0/2860433352 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd1ec0ef610 0x7fd1ec0efa10 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 rx=0x7fd1e0006700 tx=0x7fd1e0009b80).ready entity=mon.0 client_cookie=c0293ad9a50ba4f8 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:30.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:30.325 7fd1eaffd700 1 -- 172.21.2.112:0/2860433352 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd1e000c0c0 con 0x7fd1ec0ef610 2021-02-10T11:26:30.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:30.325 7fd1eaffd700 1 -- 172.21.2.112:0/2860433352 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1e000c340 con 0x7fd1ec0ef610 2021-02-10T11:26:30.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:30.325 7fd1eaffd700 1 -- 172.21.2.112:0/2860433352 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd1e000c860 con 0x7fd1ec0ef610 2021-02-10T11:26:30.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:30.325 7fd1f244f700 1 -- 172.21.2.112:0/2860433352 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd1ec0ef610 msgr2=0x7fd1ec0efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:30.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:30.325 7fd1f244f700 1 --2- 172.21.2.112:0/2860433352 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd1ec0ef610 0x7fd1ec0efa10 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 rx=0x7fd1e0006700 tx=0x7fd1e0009b80).stop 2021-02-10T11:26:30.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:30.326 7fd1f244f700 1 -- 172.21.2.112:0/2860433352 shutdown_connections 2021-02-10T11:26:30.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:30.326 7fd1f244f700 1 --2- 172.21.2.112:0/2860433352 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd1ec0ef610 0x7fd1ec0efa10 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:30.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:30.326 7fd1f244f700 1 --2- 172.21.2.112:0/2860433352 >> v2:172.21.2.120:3300/0 conn(0x7fd1ec1b0470 0x7fd1ec1b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:30.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:30.326 7fd1f244f700 1 -- 172.21.2.112:0/2860433352 >> 172.21.2.112:0/2860433352 conn(0x7fd1ec0ce3e0 msgr2=0x7fd1ec0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:30.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:30.326 7fd1f244f700 1 -- 172.21.2.112:0/2860433352 shutdown_connections 2021-02-10T11:26:30.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:30.326 7fd1f244f700 1 -- 172.21.2.112:0/2860433352 wait complete. 2021-02-10T11:26:30.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:30.341 7fd1f244f700 1 Processor -- start 2021-02-10T11:26:30.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:30.341 7fd1f244f700 1 -- start start 2021-02-10T11:26:30.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:30.341 7fd1f244f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd1ec0a0420 0x7fd1ec0a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:30.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:30.341 7fd1f244f700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd1ec189d00 0x7fd1ec18a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:30.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:30.341 7fd1f244f700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd1ec18a720 0x7fd1ec0b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:30.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:30.341 7fd1f244f700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1ec189800 con 0x7fd1ec0a0420 2021-02-10T11:26:30.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:30.341 7fd1f244f700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1ec18ab20 con 0x7fd1ec18a720 2021-02-10T11:26:30.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:30.341 7fd1f244f700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1ec0a1140 con 0x7fd1ec189d00 2021-02-10T11:26:30.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:30.341 7fd1ebfff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd1ec0a0420 0x7fd1ec0a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:30.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:30.341 7fd1ebfff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd1ec0a0420 0x7fd1ec0a0820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:45916/0 (socket says 172.21.2.112:45916) 2021-02-10T11:26:30.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:30.341 7fd1ebfff700 1 -- 172.21.2.112:0/1181261153 learned_addr learned my addr 172.21.2.112:0/1181261153 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:26:30.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:30.341 7fd1f0a6c700 1 --2- 172.21.2.112:0/1181261153 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd1ec18a720 0x7fd1ec0b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:30.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:30.341 7fd1eb7fe700 1 --2- 172.21.2.112:0/1181261153 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd1ec189d00 0x7fd1ec18a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:30.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:30.341 7fd1ebfff700 1 -- 172.21.2.112:0/1181261153 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd1ec189d00 msgr2=0x7fd1ec18a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:30.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:30.341 7fd1ebfff700 1 --2- 172.21.2.112:0/1181261153 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd1ec189d00 0x7fd1ec18a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:30.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:30.342 7fd1ebfff700 1 -- 172.21.2.112:0/1181261153 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd1ec18a720 msgr2=0x7fd1ec0b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:30.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:30.342 7fd1ebfff700 1 --2- 172.21.2.112:0/1181261153 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd1ec18a720 0x7fd1ec0b7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:30.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:30.342 7fd1ebfff700 1 -- 172.21.2.112:0/1181261153 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1ec0b8200 con 0x7fd1ec0a0420 2021-02-10T11:26:30.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:30.342 7fd1ebfff700 1 --2- 172.21.2.112:0/1181261153 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd1ec0a0420 0x7fd1ec0a0820 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 rx=0x7fd1dc0099e0 tx=0x7fd1dc00bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:30.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:30.342 7fd1e8ff9700 1 -- 172.21.2.112:0/1181261153 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd1dc00c2f0 con 0x7fd1ec0a0420 2021-02-10T11:26:30.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:30.342 7fd1e8ff9700 1 -- 172.21.2.112:0/1181261153 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1dc00c670 con 0x7fd1ec0a0420 2021-02-10T11:26:30.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:30.342 7fd1e8ff9700 1 -- 172.21.2.112:0/1181261153 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd1dc00cad0 con 0x7fd1ec0a0420 2021-02-10T11:26:30.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:30.342 7fd1f244f700 1 -- 172.21.2.112:0/1181261153 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1ec0b8200 con 0x7fd1ec0a0420 2021-02-10T11:26:30.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:30.342 7fd1f244f700 1 -- 172.21.2.112:0/1181261153 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1ec0b8790 con 0x7fd1ec0a0420 2021-02-10T11:26:30.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:30.342 7fd1e8ff9700 1 -- 172.21.2.112:0/1181261153 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd1dc00c2f0 con 0x7fd1ec0a0420 2021-02-10T11:26:30.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:30.342 7fd1e8ff9700 1 --2- 172.21.2.112:0/1181261153 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd1cc0186c0 0x7fd1cc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:30.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:30.343 7fd1e8ff9700 1 -- 172.21.2.112:0/1181261153 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fd1dc0329c0 con 0x7fd1ec0a0420 2021-02-10T11:26:30.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:30.343 7fd1eb7fe700 1 --2- 172.21.2.112:0/1181261153 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd1cc0186c0 0x7fd1cc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:30.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:30.343 7fd1eb7fe700 1 --2- 172.21.2.112:0/1181261153 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd1cc0186c0 0x7fd1cc01abc0 crc :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:30.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:30.374 7fd1f244f700 1 -- 172.21.2.112:0/1181261153 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1ec105ae0 con 0x7fd1ec0a0420 2021-02-10T11:26:30.366 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:30.377 7fd1e8ff9700 1 -- 172.21.2.112:0/1181261153 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd1dc000ce0 con 0x7fd1ec0a0420 2021-02-10T11:26:30.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:30.507 7fd1f244f700 1 -- 172.21.2.112:0/1181261153 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348643", "target": ["mgr", ""]}) v1 -- 0x7fd1ec0b9020 con 0x7fd1cc0186c0 2021-02-10T11:26:30.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:30.511 7fd1e8ff9700 1 -- 172.21.2.112:0/1181261153 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7fd1e0000940 con 0x7fd1cc0186c0 2021-02-10T11:26:30.504 INFO:teuthology.orchestra.run.gibba012.stdout:/volumes/_nogroup/subvolume_0000000000348643/3d159c38-6ca4-4031-8e8c-2144c065006a 2021-02-10T11:26:30.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:30.516 7fd1f244f700 1 -- 172.21.2.112:0/1181261153 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd1cc0186c0 msgr2=0x7fd1cc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:30.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:30.516 7fd1f244f700 1 --2- 172.21.2.112:0/1181261153 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd1cc0186c0 0x7fd1cc01abc0 crc :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:30.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:30.516 7fd1f244f700 1 -- 172.21.2.112:0/1181261153 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd1ec0a0420 msgr2=0x7fd1ec0a0820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:30.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:30.516 7fd1f244f700 1 --2- 172.21.2.112:0/1181261153 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd1ec0a0420 0x7fd1ec0a0820 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 rx=0x7fd1dc0099e0 tx=0x7fd1dc00bd40).stop 2021-02-10T11:26:30.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:30.516 7fd1f244f700 1 -- 172.21.2.112:0/1181261153 shutdown_connections 2021-02-10T11:26:30.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:30.516 7fd1f244f700 1 --2- 172.21.2.112:0/1181261153 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd1cc0186c0 0x7fd1cc01abc0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:30.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:30.516 7fd1f244f700 1 --2- 172.21.2.112:0/1181261153 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd1ec18a720 0x7fd1ec0b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:30.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:30.516 7fd1f244f700 1 --2- 172.21.2.112:0/1181261153 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd1ec189d00 0x7fd1ec18a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:30.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:30.516 7fd1f244f700 1 --2- 172.21.2.112:0/1181261153 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd1ec0a0420 0x7fd1ec0a0820 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:30.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:30.516 7fd1f244f700 1 -- 172.21.2.112:0/1181261153 >> 172.21.2.112:0/1181261153 conn(0x7fd1ec094e10 msgr2=0x7fd1ec0a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:30.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:30.517 7fd1f244f700 1 -- 172.21.2.112:0/1181261153 shutdown_connections 2021-02-10T11:26:30.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:30.517 7fd1f244f700 1 -- 172.21.2.112:0/1181261153 wait complete. 2021-02-10T11:26:30.514 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000348643 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000348643/3d159c38-6ca4-4031-8e8c-2144c065006a 2021-02-10T11:26:30.514 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:30.514 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:30.514 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:30.848 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:26:30.849 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:30.849 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348643/3d159c38-6ca4-4031-8e8c-2144c065006a/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:31.232 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:26:31.232 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:26:31.232 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0246374 s, 42.6 MB/s 2021-02-10T11:26:31.233 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:31.233 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:31.234 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:31.568 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:26:31.568 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:31.569 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348643/3d159c38-6ca4-4031-8e8c-2144c065006a/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:31.944 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:26:31.944 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:26:31.944 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0245496 s, 42.7 MB/s 2021-02-10T11:26:31.945 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:31.946 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:31.946 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:32.282 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:26:32.282 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:32.283 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348643/3d159c38-6ca4-4031-8e8c-2144c065006a/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:32.657 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:26:32.657 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:26:32.657 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0241187 s, 43.5 MB/s 2021-02-10T11:26:32.658 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:32.658 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:32.659 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:32.994 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:26:32.994 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:32.995 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348643/3d159c38-6ca4-4031-8e8c-2144c065006a/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:33.375 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:26:33.376 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:26:33.376 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0282545 s, 37.1 MB/s 2021-02-10T11:26:33.377 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:33.377 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:33.377 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:33.712 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:26:33.712 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:33.713 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348643/3d159c38-6ca4-4031-8e8c-2144c065006a/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:34.087 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:26:34.087 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:26:34.087 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0227739 s, 46.0 MB/s 2021-02-10T11:26:34.088 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:34.088 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:34.088 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:34.425 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:26:34.425 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:34.426 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348643/3d159c38-6ca4-4031-8e8c-2144c065006a/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:34.801 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:26:34.801 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:26:34.801 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0261157 s, 40.2 MB/s 2021-02-10T11:26:34.802 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:34.802 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:34.802 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:35.137 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:26:35.137 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:35.137 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348643/3d159c38-6ca4-4031-8e8c-2144c065006a/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:35.514 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:26:35.514 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:26:35.514 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0252507 s, 41.5 MB/s 2021-02-10T11:26:35.515 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:35.515 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:35.515 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:35.851 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:26:35.851 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:35.851 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348643/3d159c38-6ca4-4031-8e8c-2144c065006a/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:36.227 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:26:36.227 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:26:36.227 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0251583 s, 41.7 MB/s 2021-02-10T11:26:36.228 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:36.228 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:36.228 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:36.564 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:26:36.564 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:36.565 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348643/3d159c38-6ca4-4031-8e8c-2144c065006a/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:36.939 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:26:36.940 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:26:36.940 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0231302 s, 45.3 MB/s 2021-02-10T11:26:36.941 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:36.941 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:36.941 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:37.277 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:26:37.277 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:37.278 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348643/3d159c38-6ca4-4031-8e8c-2144c065006a/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:37.655 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:26:37.655 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:26:37.655 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0260742 s, 40.2 MB/s 2021-02-10T11:26:37.656 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000348644 2021-02-10T11:26:38.087 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.082 7f46aac8c700 1 Processor -- start 2021-02-10T11:26:38.087 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.082 7f46aac8c700 1 -- start start 2021-02-10T11:26:38.088 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.083 7f46aac8c700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f46a41b0870 0x7f46a40f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:38.088 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.083 7f46aac8c700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f46a40efe00 0x7f46a40f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:38.088 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.083 7f46aac8c700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46a41b0c70 con 0x7f46a41b0470 2021-02-10T11:26:38.088 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.083 7f46aac8c700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46a40e5ea0 con 0x7f46a41b0870 2021-02-10T11:26:38.088 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.083 7f46aac8c700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46a40f0b50 con 0x7f46a40efe00 2021-02-10T11:26:38.088 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.083 7f46a3fff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f46a41b0870 0x7f46a40f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:38.088 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.083 7f46a3fff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f46a41b0870 0x7f46a40f20e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:45924/0 (socket says 172.21.2.112:45924) 2021-02-10T11:26:38.088 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.083 7f46a8aa8700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f46a41b0470 0x7f46a40ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:45386/0 (socket says 172.21.2.112:45386) 2021-02-10T11:26:38.088 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.083 7f46a3fff700 1 -- 172.21.2.112:0/12436589 learned_addr learned my addr 172.21.2.112:0/12436589 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:26:38.088 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.083 7f46a3fff700 1 -- 172.21.2.112:0/12436589 >> v2:172.21.2.120:3300/0 conn(0x7f46a40efe00 msgr2=0x7f46a40f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:26:38.088 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.083 7f46a3fff700 1 --2- 172.21.2.112:0/12436589 >> v2:172.21.2.120:3300/0 conn(0x7f46a40efe00 0x7f46a40f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:38.088 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.083 7f46a3fff700 1 -- 172.21.2.112:0/12436589 >> v1:172.21.2.112:6789/0 conn(0x7f46a41b0470 legacy=0x7f46a40ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:38.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.083 7f46a3fff700 1 -- 172.21.2.112:0/12436589 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46a40f0820 con 0x7f46a41b0870 2021-02-10T11:26:38.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.083 7f46a3fff700 1 --2- 172.21.2.112:0/12436589 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f46a41b0870 0x7f46a40f20e0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 rx=0x7f4694006a80 tx=0x7f4694009f00).ready entity=mon.0 client_cookie=1e854b3888064a70 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:38.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.083 7f46a37fe700 1 -- 172.21.2.112:0/12436589 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f469400b5c0 con 0x7f46a41b0870 2021-02-10T11:26:38.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.084 7f46a37fe700 1 -- 172.21.2.112:0/12436589 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f469400b8c0 con 0x7f46a41b0870 2021-02-10T11:26:38.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.084 7f46a37fe700 1 -- 172.21.2.112:0/12436589 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f469400bde0 con 0x7f46a41b0870 2021-02-10T11:26:38.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.084 7f46aac8c700 1 -- 172.21.2.112:0/12436589 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f46a41b0870 msgr2=0x7f46a40f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:38.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.084 7f46aac8c700 1 --2- 172.21.2.112:0/12436589 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f46a41b0870 0x7f46a40f20e0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 rx=0x7f4694006a80 tx=0x7f4694009f00).stop 2021-02-10T11:26:38.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.084 7f46aac8c700 1 -- 172.21.2.112:0/12436589 shutdown_connections 2021-02-10T11:26:38.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.084 7f46aac8c700 1 --2- 172.21.2.112:0/12436589 >> v2:172.21.2.120:3300/0 conn(0x7f46a40efe00 0x7f46a40f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:38.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.084 7f46aac8c700 1 --2- 172.21.2.112:0/12436589 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f46a41b0870 0x7f46a40f20e0 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:38.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.084 7f46aac8c700 1 -- 172.21.2.112:0/12436589 >> 172.21.2.112:0/12436589 conn(0x7f46a40ce3e0 msgr2=0x7f46a40e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:38.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.084 7f46aac8c700 1 -- 172.21.2.112:0/12436589 shutdown_connections 2021-02-10T11:26:38.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.084 7f46aac8c700 1 -- 172.21.2.112:0/12436589 wait complete. 2021-02-10T11:26:38.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.098 7f46aac8c700 1 Processor -- start 2021-02-10T11:26:38.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.098 7f46aac8c700 1 -- start start 2021-02-10T11:26:38.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.099 7f46aac8c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f46a4190920 0x7f46a408f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:38.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.099 7f46aac8c700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f46a4190d20 0x7f46a408fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:38.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.099 7f46aac8c700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f46a4193270 0x7f46a4193670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:38.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.099 7f46aac8c700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46a4090470 con 0x7f46a4190920 2021-02-10T11:26:38.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.099 7f46aac8c700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46a4193c90 con 0x7f46a4193270 2021-02-10T11:26:38.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.099 7f46aac8c700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46a4193e70 con 0x7f46a4190d20 2021-02-10T11:26:38.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.099 7f46a8aa8700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f46a4190920 0x7f46a408f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:38.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.099 7f46a8aa8700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f46a4190920 0x7f46a408f600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:45932/0 (socket says 172.21.2.112:45932) 2021-02-10T11:26:38.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.099 7f46a8aa8700 1 -- 172.21.2.112:0/2705566762 learned_addr learned my addr 172.21.2.112:0/2705566762 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:26:38.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.099 7f46a8aa8700 1 -- 172.21.2.112:0/2705566762 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f46a4190d20 msgr2=0x7f46a408fc20 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:26:38.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.099 7f46a8aa8700 1 --2- 172.21.2.112:0/2705566762 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f46a4190d20 0x7f46a408fc20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:38.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.099 7f46a8aa8700 1 -- 172.21.2.112:0/2705566762 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f46a4193270 msgr2=0x7f46a4193670 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:26:38.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.099 7f46a8aa8700 1 --2- 172.21.2.112:0/2705566762 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f46a4193270 0x7f46a4193670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:38.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.099 7f46a8aa8700 1 -- 172.21.2.112:0/2705566762 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46a4194090 con 0x7f46a4190920 2021-02-10T11:26:38.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.100 7f46a8aa8700 1 --2- 172.21.2.112:0/2705566762 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f46a4190920 0x7f46a408f600 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 rx=0x7f468c001c60 tx=0x7f468c00a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:38.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.100 7f46a17fa700 1 -- 172.21.2.112:0/2705566762 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f468c00acc0 con 0x7f46a4190920 2021-02-10T11:26:38.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.100 7f46a17fa700 1 -- 172.21.2.112:0/2705566762 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f468c004140 con 0x7f46a4190920 2021-02-10T11:26:38.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.100 7f46a17fa700 1 -- 172.21.2.112:0/2705566762 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f468c004660 con 0x7f46a4190920 2021-02-10T11:26:38.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.100 7f46aac8c700 1 -- 172.21.2.112:0/2705566762 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46a4194090 con 0x7f46a4190920 2021-02-10T11:26:38.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.100 7f46aac8c700 1 -- 172.21.2.112:0/2705566762 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46a41920b0 con 0x7f46a4190920 2021-02-10T11:26:38.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.100 7f46a17fa700 1 -- 172.21.2.112:0/2705566762 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f468c004320 con 0x7f46a4190920 2021-02-10T11:26:38.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.100 7f46a17fa700 1 --2- 172.21.2.112:0/2705566762 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f46880186c0 0x7f468801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:38.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.100 7f46a17fa700 1 -- 172.21.2.112:0/2705566762 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f468c032590 con 0x7f46a4190920 2021-02-10T11:26:38.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.101 7f46a3fff700 1 --2- 172.21.2.112:0/2705566762 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f46880186c0 0x7f468801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:38.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.102 7f46a3fff700 1 --2- 172.21.2.112:0/2705566762 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f46880186c0 0x7f468801abc0 crc :-1 s=READY pgs=327 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:38.121 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.132 7f46aac8c700 1 -- 172.21.2.112:0/2705566762 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46a4105af0 con 0x7f46a4190920 2021-02-10T11:26:38.124 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.135 7f46a17fa700 1 -- 172.21.2.112:0/2705566762 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f468c025fc0 con 0x7f46a4190920 2021-02-10T11:26:38.250 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.260 7f46aac8c700 1 -- 172.21.2.112:0/2705566762 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348644", "target": ["mgr", ""]}) v1 -- 0x7f46a4090470 con 0x7f46880186c0 2021-02-10T11:26:38.275 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.287 7f46a17fa700 1 -- 172.21.2.112:0/2705566762 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f46940067c0 con 0x7f46880186c0 2021-02-10T11:26:38.283 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.294 7f46aac8c700 1 -- 172.21.2.112:0/2705566762 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f46880186c0 msgr2=0x7f468801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:38.283 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.294 7f46aac8c700 1 --2- 172.21.2.112:0/2705566762 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f46880186c0 0x7f468801abc0 crc :-1 s=READY pgs=327 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:38.283 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.294 7f46aac8c700 1 -- 172.21.2.112:0/2705566762 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f46a4190920 msgr2=0x7f46a408f600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:38.283 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.294 7f46aac8c700 1 --2- 172.21.2.112:0/2705566762 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f46a4190920 0x7f46a408f600 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 rx=0x7f468c001c60 tx=0x7f468c00a8d0).stop 2021-02-10T11:26:38.284 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.294 7f46aac8c700 1 -- 172.21.2.112:0/2705566762 shutdown_connections 2021-02-10T11:26:38.284 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.294 7f46aac8c700 1 --2- 172.21.2.112:0/2705566762 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f46880186c0 0x7f468801abc0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:38.284 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.294 7f46aac8c700 1 --2- 172.21.2.112:0/2705566762 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f46a4193270 0x7f46a4193670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:38.284 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.294 7f46aac8c700 1 --2- 172.21.2.112:0/2705566762 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f46a4190d20 0x7f46a408fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:38.284 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.294 7f46aac8c700 1 --2- 172.21.2.112:0/2705566762 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f46a4190920 0x7f46a408f600 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:38.284 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.294 7f46aac8c700 1 -- 172.21.2.112:0/2705566762 >> 172.21.2.112:0/2705566762 conn(0x7f46a40bf4e0 msgr2=0x7f46a40be3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:38.284 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.294 7f46aac8c700 1 -- 172.21.2.112:0/2705566762 shutdown_connections 2021-02-10T11:26:38.284 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.294 7f46aac8c700 1 -- 172.21.2.112:0/2705566762 wait complete. 2021-02-10T11:26:38.291 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000348644 2021-02-10T11:26:38.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.718 7fc3749ad700 1 Processor -- start 2021-02-10T11:26:38.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.718 7fc3749ad700 1 -- start start 2021-02-10T11:26:38.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.719 7fc3749ad700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc3701b0870 0x7fc3700f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:38.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.719 7fc3749ad700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fc3700efe00 0x7fc3700f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:38.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.719 7fc3749ad700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3701b0c70 con 0x7fc3701b0470 2021-02-10T11:26:38.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.719 7fc3749ad700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3700f0b50 con 0x7fc3700efe00 2021-02-10T11:26:38.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.719 7fc3749ad700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3700f0820 con 0x7fc3701b0870 2021-02-10T11:26:38.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.719 7fc36e61c700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fc3701b0470 0x7fc3700ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:45398/0 (socket says 172.21.2.112:45398) 2021-02-10T11:26:38.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.719 7fc36e61c700 1 -- 172.21.2.112:0/2318117958 learned_addr learned my addr 172.21.2.112:0/2318117958 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:26:38.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.719 7fc36d61a700 1 -- 172.21.2.112:0/2318117958 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1793781972 0 0) 0x7fc3600009b0 con 0x7fc3701b0470 2021-02-10T11:26:38.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.719 7fc36d61a700 1 -- 172.21.2.112:0/2318117958 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc35c002fe0 con 0x7fc3701b0470 2021-02-10T11:26:38.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.720 7fc36de1b700 1 --2- 172.21.2.112:0/2318117958 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc3701b0870 0x7fc3700f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:38.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.720 7fc36ee1d700 1 --2- 172.21.2.112:0/2318117958 >> v2:172.21.2.120:3300/0 conn(0x7fc3700efe00 0x7fc3700f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:38.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.720 7fc36d61a700 1 -- 172.21.2.112:0/2318117958 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2625615027 0 0) 0x7fc360001010 con 0x7fc3701b0470 2021-02-10T11:26:38.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.720 7fc36d61a700 1 -- 172.21.2.112:0/2318117958 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc3701b0870 msgr2=0x7fc3700f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:26:38.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.720 7fc36d61a700 1 --2- 172.21.2.112:0/2318117958 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc3701b0870 0x7fc3700f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:38.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.720 7fc36d61a700 1 -- 172.21.2.112:0/2318117958 >> v2:172.21.2.120:3300/0 conn(0x7fc3700efe00 msgr2=0x7fc3700f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:26:38.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.720 7fc36d61a700 1 --2- 172.21.2.112:0/2318117958 >> v2:172.21.2.120:3300/0 conn(0x7fc3700efe00 0x7fc3700f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:38.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.720 7fc36d61a700 1 -- 172.21.2.112:0/2318117958 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3700e52e0 con 0x7fc3701b0470 2021-02-10T11:26:38.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.720 7fc36d61a700 1 -- 172.21.2.112:0/2318117958 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7fc360001590 con 0x7fc3701b0470 2021-02-10T11:26:38.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.720 7fc36d61a700 1 -- 172.21.2.112:0/2318117958 >> v1:172.21.2.112:6789/0 conn(0x7fc3701b0470 legacy=0x7fc3700ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:38.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.720 7fc36d61a700 1 --2- 172.21.2.112:0/2318117958 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc35c003a60 0x7fc35c005eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:38.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.720 7fc36d61a700 1 --2- 172.21.2.112:0/2318117958 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc35c006750 0x7fc35c008c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:38.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.720 7fc36d61a700 1 --2- 172.21.2.112:0/2318117958 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc35c0094f0 0x7fc35c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:38.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.720 7fc36d61a700 1 -- 172.21.2.112:0/2318117958 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc35c0502d0 con 0x7fc35c006750 2021-02-10T11:26:38.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.720 7fc36d61a700 1 -- 172.21.2.112:0/2318117958 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc35c0504b0 con 0x7fc35c0094f0 2021-02-10T11:26:38.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.720 7fc36d61a700 1 -- 172.21.2.112:0/2318117958 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc35c0506d0 con 0x7fc35c003a60 2021-02-10T11:26:38.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.720 7fc36de1b700 1 --2- 172.21.2.112:0/2318117958 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc35c006750 0x7fc35c008c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:38.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.720 7fc36e61c700 1 --2- 172.21.2.112:0/2318117958 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc35c003a60 0x7fc35c005eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:38.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.720 7fc36ee1d700 1 --2- 172.21.2.112:0/2318117958 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc35c0094f0 0x7fc35c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:38.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.720 7fc36de1b700 1 -- 172.21.2.112:0/2318117958 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc35c003a60 msgr2=0x7fc35c005eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:26:38.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.720 7fc36de1b700 1 --2- 172.21.2.112:0/2318117958 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc35c003a60 0x7fc35c005eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:38.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.720 7fc36de1b700 1 -- 172.21.2.112:0/2318117958 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc35c0094f0 msgr2=0x7fc35c00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:26:38.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.720 7fc36de1b700 1 --2- 172.21.2.112:0/2318117958 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc35c0094f0 0x7fc35c00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:38.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.720 7fc36de1b700 1 -- 172.21.2.112:0/2318117958 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc35c050970 con 0x7fc35c006750 2021-02-10T11:26:38.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.720 7fc36ee1d700 1 --2- 172.21.2.112:0/2318117958 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc35c0094f0 0x7fc35c00ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:26:38.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.720 7fc36de1b700 1 --2- 172.21.2.112:0/2318117958 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc35c006750 0x7fc35c008c50 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 rx=0x7fc358006ad0 tx=0x7fc358009ac0).ready entity=mon.0 client_cookie=50304c2264ed0253 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:38.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.720 7fc36d61a700 1 -- 172.21.2.112:0/2318117958 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc3580066d0 con 0x7fc35c006750 2021-02-10T11:26:38.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.720 7fc36d61a700 1 -- 172.21.2.112:0/2318117958 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc35800c0c0 con 0x7fc35c006750 2021-02-10T11:26:38.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.720 7fc36d61a700 1 -- 172.21.2.112:0/2318117958 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc35800c510 con 0x7fc35c006750 2021-02-10T11:26:38.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.721 7fc3749ad700 1 -- 172.21.2.112:0/2318117958 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc35c006750 msgr2=0x7fc35c008c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:38.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.721 7fc3749ad700 1 --2- 172.21.2.112:0/2318117958 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc35c006750 0x7fc35c008c50 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 rx=0x7fc358006ad0 tx=0x7fc358009ac0).stop 2021-02-10T11:26:38.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.721 7fc36ee1d700 1 -- 172.21.2.112:0/2318117958 reap_dead start 2021-02-10T11:26:38.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.721 7fc3749ad700 1 -- 172.21.2.112:0/2318117958 shutdown_connections 2021-02-10T11:26:38.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.721 7fc3749ad700 1 -- 172.21.2.112:0/2318117958 >> 172.21.2.112:0/2318117958 conn(0x7fc3700ce3e0 msgr2=0x7fc3700e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:38.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.721 7fc3749ad700 1 -- 172.21.2.112:0/2318117958 shutdown_connections 2021-02-10T11:26:38.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.721 7fc3749ad700 1 -- 172.21.2.112:0/2318117958 wait complete. 2021-02-10T11:26:38.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.736 7fc3749ad700 1 Processor -- start 2021-02-10T11:26:38.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.736 7fc3749ad700 1 -- start start 2021-02-10T11:26:38.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.736 7fc3749ad700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc37009b930 0x7fc3700b7c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:38.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.736 7fc3749ad700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc3700b8410 0x7fc3700b8810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:38.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.736 7fc3749ad700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc3700c1f80 0x7fc3700c2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:38.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.736 7fc3749ad700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3700c1a00 con 0x7fc37009b930 2021-02-10T11:26:38.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.736 7fc3749ad700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3700b8e30 con 0x7fc3700b8410 2021-02-10T11:26:38.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.736 7fc3749ad700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3700b9050 con 0x7fc3700c1f80 2021-02-10T11:26:38.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.736 7fc36e61c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc37009b930 0x7fc3700b7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:38.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.736 7fc36e61c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc37009b930 0x7fc3700b7c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:45948/0 (socket says 172.21.2.112:45948) 2021-02-10T11:26:38.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.736 7fc36e61c700 1 -- 172.21.2.112:0/2687458217 learned_addr learned my addr 172.21.2.112:0/2687458217 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:26:38.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.737 7fc36e61c700 1 -- 172.21.2.112:0/2687458217 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc3700c1f80 msgr2=0x7fc3700c2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:38.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.737 7fc36e61c700 1 --2- 172.21.2.112:0/2687458217 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc3700c1f80 0x7fc3700c2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:38.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.737 7fc36e61c700 1 -- 172.21.2.112:0/2687458217 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc3700b8410 msgr2=0x7fc3700b8810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:38.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.737 7fc36e61c700 1 --2- 172.21.2.112:0/2687458217 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc3700b8410 0x7fc3700b8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:38.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.737 7fc36e61c700 1 -- 172.21.2.112:0/2687458217 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3700c29a0 con 0x7fc37009b930 2021-02-10T11:26:38.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.737 7fc36e61c700 1 --2- 172.21.2.112:0/2687458217 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc37009b930 0x7fc3700b7c20 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 rx=0x7fc360001980 tx=0x7fc36000a900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:38.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.737 7fc366ffd700 1 -- 172.21.2.112:0/2687458217 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc360000fd0 con 0x7fc37009b930 2021-02-10T11:26:38.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.737 7fc366ffd700 1 -- 172.21.2.112:0/2687458217 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc36000acf0 con 0x7fc37009b930 2021-02-10T11:26:38.732 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.737 7fc366ffd700 1 -- 172.21.2.112:0/2687458217 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc3600045d0 con 0x7fc37009b930 2021-02-10T11:26:38.732 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.737 7fc3749ad700 1 -- 172.21.2.112:0/2687458217 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3700c29a0 con 0x7fc37009b930 2021-02-10T11:26:38.732 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.737 7fc3749ad700 1 -- 172.21.2.112:0/2687458217 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3700c29a0 con 0x7fc37009b930 2021-02-10T11:26:38.732 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.738 7fc366ffd700 1 -- 172.21.2.112:0/2687458217 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc360004350 con 0x7fc37009b930 2021-02-10T11:26:38.732 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.738 7fc366ffd700 1 --2- 172.21.2.112:0/2687458217 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc3540186c0 0x7fc35401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:38.733 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.738 7fc366ffd700 1 -- 172.21.2.112:0/2687458217 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fc360032560 con 0x7fc37009b930 2021-02-10T11:26:38.733 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.738 7fc36de1b700 1 --2- 172.21.2.112:0/2687458217 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc3540186c0 0x7fc35401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:38.733 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.739 7fc36de1b700 1 --2- 172.21.2.112:0/2687458217 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc3540186c0 0x7fc35401abc0 crc :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:38.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.769 7fc3749ad700 1 -- 172.21.2.112:0/2687458217 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc370105b30 con 0x7fc37009b930 2021-02-10T11:26:38.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.771 7fc366ffd700 1 -- 172.21.2.112:0/2687458217 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc360026000 con 0x7fc37009b930 2021-02-10T11:26:38.890 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.900 7fc3749ad700 1 -- 172.21.2.112:0/2687458217 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348644", "target": ["mgr", ""]}) v1 -- 0x7fc3700b8e30 con 0x7fc3540186c0 2021-02-10T11:26:38.892 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.904 7fc366ffd700 1 -- 172.21.2.112:0/2687458217 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7fc3580009b0 con 0x7fc3540186c0 2021-02-10T11:26:38.893 INFO:teuthology.orchestra.run.gibba012.stdout:/volumes/_nogroup/subvolume_0000000000348644/fefca694-1ffc-4af4-9fe5-c9d548344c93 2021-02-10T11:26:38.894 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.905 7fc3749ad700 1 -- 172.21.2.112:0/2687458217 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc3540186c0 msgr2=0x7fc35401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:38.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.905 7fc3749ad700 1 --2- 172.21.2.112:0/2687458217 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc3540186c0 0x7fc35401abc0 crc :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:38.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.905 7fc3749ad700 1 -- 172.21.2.112:0/2687458217 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc37009b930 msgr2=0x7fc3700b7c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:38.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.905 7fc3749ad700 1 --2- 172.21.2.112:0/2687458217 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc37009b930 0x7fc3700b7c20 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 rx=0x7fc360001980 tx=0x7fc36000a900).stop 2021-02-10T11:26:38.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.905 7fc3749ad700 1 -- 172.21.2.112:0/2687458217 shutdown_connections 2021-02-10T11:26:38.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.905 7fc3749ad700 1 --2- 172.21.2.112:0/2687458217 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc3540186c0 0x7fc35401abc0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:38.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.905 7fc3749ad700 1 --2- 172.21.2.112:0/2687458217 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc3700b8410 0x7fc3700b8810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:38.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.905 7fc3749ad700 1 --2- 172.21.2.112:0/2687458217 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc3700c1f80 0x7fc3700c2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:38.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.905 7fc3749ad700 1 --2- 172.21.2.112:0/2687458217 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc37009b930 0x7fc3700b7c20 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:38.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.905 7fc3749ad700 1 -- 172.21.2.112:0/2687458217 >> 172.21.2.112:0/2687458217 conn(0x7fc370094e10 msgr2=0x7fc3700a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:38.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.905 7fc3749ad700 1 -- 172.21.2.112:0/2687458217 shutdown_connections 2021-02-10T11:26:38.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:38.905 7fc3749ad700 1 -- 172.21.2.112:0/2687458217 wait complete. 2021-02-10T11:26:38.903 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000348644 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000348644/fefca694-1ffc-4af4-9fe5-c9d548344c93 2021-02-10T11:26:38.903 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:38.903 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:38.903 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:39.239 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:26:39.239 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:39.239 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348644/fefca694-1ffc-4af4-9fe5-c9d548344c93/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:39.618 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:26:39.619 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:26:39.619 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0248996 s, 42.1 MB/s 2021-02-10T11:26:39.620 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:39.620 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:39.620 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:39.955 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:26:39.956 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:39.956 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348644/fefca694-1ffc-4af4-9fe5-c9d548344c93/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:40.332 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:26:40.332 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:26:40.332 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0248228 s, 42.2 MB/s 2021-02-10T11:26:40.333 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:40.333 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:40.334 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:40.669 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:26:40.669 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:40.669 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348644/fefca694-1ffc-4af4-9fe5-c9d548344c93/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:40.866 DEBUG:teuthology.orchestra.run.gibba012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:26:40.869 DEBUG:teuthology.orchestra.run.gibba020:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:26:41.043 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:26:41.043 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:26:41.043 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0218235 s, 48.0 MB/s 2021-02-10T11:26:41.044 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:41.044 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:41.044 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:41.379 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:26:41.379 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:41.380 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348644/fefca694-1ffc-4af4-9fe5-c9d548344c93/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:41.755 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:26:41.755 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:26:41.755 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0222818 s, 47.1 MB/s 2021-02-10T11:26:41.756 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:41.756 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:41.756 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:42.092 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:26:42.093 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:42.093 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348644/fefca694-1ffc-4af4-9fe5-c9d548344c93/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:42.468 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:26:42.468 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:26:42.468 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0246503 s, 42.5 MB/s 2021-02-10T11:26:42.469 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:42.469 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:42.470 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:42.805 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:26:42.805 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:42.806 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348644/fefca694-1ffc-4af4-9fe5-c9d548344c93/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:43.181 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:26:43.182 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:26:43.182 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0232586 s, 45.1 MB/s 2021-02-10T11:26:43.183 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:43.183 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:43.183 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:43.519 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:26:43.519 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:43.519 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348644/fefca694-1ffc-4af4-9fe5-c9d548344c93/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:43.892 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:26:43.892 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:26:43.892 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0219488 s, 47.8 MB/s 2021-02-10T11:26:43.893 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:43.893 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:43.893 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:44.230 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:26:44.230 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:44.231 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348644/fefca694-1ffc-4af4-9fe5-c9d548344c93/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:44.602 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:26:44.602 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:26:44.602 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0237625 s, 44.1 MB/s 2021-02-10T11:26:44.603 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:44.603 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:44.603 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:44.938 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:26:44.938 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:44.938 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348644/fefca694-1ffc-4af4-9fe5-c9d548344c93/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:45.315 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:26:45.315 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:26:45.315 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.025206 s, 41.6 MB/s 2021-02-10T11:26:45.316 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:45.316 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:45.316 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:45.652 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:26:45.652 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:45.653 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348644/fefca694-1ffc-4af4-9fe5-c9d548344c93/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:46.028 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:26:46.028 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:26:46.028 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.024222 s, 43.3 MB/s 2021-02-10T11:26:46.029 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000348645 2021-02-10T11:26:46.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:46.456 7f52ee2b5700 1 Processor -- start 2021-02-10T11:26:46.463 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:46.456 7f52ee2b5700 1 -- start start 2021-02-10T11:26:46.463 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:46.457 7f52ee2b5700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f52e81b0470 0x7f52e81b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:46.463 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:46.457 7f52ee2b5700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52e80f0b50 con 0x7f52e80f20e0 2021-02-10T11:26:46.463 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:46.457 7f52ee2b5700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52e80f24e0 con 0x7f52e80ef610 2021-02-10T11:26:46.463 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:46.457 7f52ee2b5700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52e80e5ea0 con 0x7f52e81b0470 2021-02-10T11:26:46.463 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:46.457 7f52ec8d2700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f52e80f20e0 0x7f52e80f0200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:45418/0 (socket says 172.21.2.112:45418) 2021-02-10T11:26:46.463 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:46.457 7f52ec8d2700 1 -- 172.21.2.112:0/3279861811 learned_addr learned my addr 172.21.2.112:0/3279861811 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:26:46.463 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:46.457 7f52e7fff700 1 --2- 172.21.2.112:0/3279861811 >> v2:172.21.2.120:3300/0 conn(0x7f52e81b0470 0x7f52e81b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:46.463 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:46.457 7f52e6ffd700 1 -- 172.21.2.112:0/3279861811 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1099592219 0 0) 0x7f52e0001500 con 0x7f52e80f20e0 2021-02-10T11:26:46.464 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:46.457 7f52e6ffd700 1 -- 172.21.2.112:0/3279861811 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52d0002fe0 con 0x7f52e80f20e0 2021-02-10T11:26:46.464 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:46.457 7f52e7fff700 1 -- 172.21.2.112:0/3279861811 >> v1:172.21.2.120:6789/0 conn(0x7f52e80ef610 legacy=0x7f52e80efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:46.464 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:46.457 7f52e7fff700 1 -- 172.21.2.112:0/3279861811 >> v1:172.21.2.112:6789/0 conn(0x7f52e80f20e0 legacy=0x7f52e80f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:46.464 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:46.457 7f52e7fff700 1 -- 172.21.2.112:0/3279861811 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52e80e52e0 con 0x7f52e81b0470 2021-02-10T11:26:46.464 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:46.458 7f52e7fff700 1 --2- 172.21.2.112:0/3279861811 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f52e81b0470 0x7f52e81b0870 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7f52d8006650 tx=0x7f52d8009a80).ready entity=mon.1 client_cookie=9a305b7650ca7d75 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:46.464 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:46.458 7f52e6ffd700 1 -- 172.21.2.112:0/3279861811 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f52d800c0c0 con 0x7f52e81b0470 2021-02-10T11:26:46.464 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:46.458 7f52e6ffd700 1 -- 172.21.2.112:0/3279861811 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52d800c340 con 0x7f52e81b0470 2021-02-10T11:26:46.464 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:46.458 7f52e6ffd700 1 -- 172.21.2.112:0/3279861811 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f52d800c860 con 0x7f52e81b0470 2021-02-10T11:26:46.464 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:46.458 7f52ee2b5700 1 -- 172.21.2.112:0/3279861811 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f52e81b0470 msgr2=0x7f52e81b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:46.464 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:46.458 7f52ee2b5700 1 --2- 172.21.2.112:0/3279861811 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f52e81b0470 0x7f52e81b0870 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7f52d8006650 tx=0x7f52d8009a80).stop 2021-02-10T11:26:46.464 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:46.459 7f52ee2b5700 1 -- 172.21.2.112:0/3279861811 shutdown_connections 2021-02-10T11:26:46.464 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:46.459 7f52ee2b5700 1 --2- 172.21.2.112:0/3279861811 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f52e81b0470 0x7f52e81b0870 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:46.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:46.459 7f52ee2b5700 1 -- 172.21.2.112:0/3279861811 >> 172.21.2.112:0/3279861811 conn(0x7f52e80ce3e0 msgr2=0x7f52e80e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:46.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:46.459 7f52ee2b5700 1 -- 172.21.2.112:0/3279861811 shutdown_connections 2021-02-10T11:26:46.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:46.459 7f52ee2b5700 1 -- 172.21.2.112:0/3279861811 wait complete. 2021-02-10T11:26:46.466 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:46.474 7f52ee2b5700 1 Processor -- start 2021-02-10T11:26:46.466 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:46.474 7f52ee2b5700 1 -- start start 2021-02-10T11:26:46.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:46.474 7f52ee2b5700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f52e8094fb0 0x7f52e809e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:46.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:46.474 7f52ee2b5700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f52e80c2be0 0x7f52e80c2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:46.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:46.474 7f52ee2b5700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f52e80c3600 0x7f52e80a9c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:46.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:46.474 7f52ee2b5700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52e809ebf0 con 0x7f52e8094fb0 2021-02-10T11:26:46.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:46.474 7f52ee2b5700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52e80aa2a0 con 0x7f52e80c2be0 2021-02-10T11:26:46.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:46.474 7f52ee2b5700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52e80aa4c0 con 0x7f52e80c3600 2021-02-10T11:26:46.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:46.474 7f52e7fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f52e8094fb0 0x7f52e809e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:46.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:46.474 7f52e7fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f52e8094fb0 0x7f52e809e5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:45960/0 (socket says 172.21.2.112:45960) 2021-02-10T11:26:46.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:46.474 7f52e7fff700 1 -- 172.21.2.112:0/1163598083 learned_addr learned my addr 172.21.2.112:0/1163598083 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:26:46.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:46.475 7f52e77fe700 1 --2- 172.21.2.112:0/1163598083 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f52e80c2be0 0x7f52e80c2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:46.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:46.475 7f52ec8d2700 1 --2- 172.21.2.112:0/1163598083 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f52e80c3600 0x7f52e80a9c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:46.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:46.475 7f52e7fff700 1 -- 172.21.2.112:0/1163598083 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f52e80c3600 msgr2=0x7f52e80a9c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:46.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:46.475 7f52e7fff700 1 --2- 172.21.2.112:0/1163598083 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f52e80c3600 0x7f52e80a9c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:46.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:46.475 7f52e7fff700 1 -- 172.21.2.112:0/1163598083 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f52e80c2be0 msgr2=0x7f52e80c2fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:46.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:46.475 7f52e7fff700 1 --2- 172.21.2.112:0/1163598083 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f52e80c2be0 0x7f52e80c2fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:46.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:46.475 7f52e7fff700 1 -- 172.21.2.112:0/1163598083 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52e80aa6e0 con 0x7f52e8094fb0 2021-02-10T11:26:46.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:46.475 7f52e77fe700 1 --2- 172.21.2.112:0/1163598083 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f52e80c2be0 0x7f52e80c2fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:26:46.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:46.475 7f52e7fff700 1 --2- 172.21.2.112:0/1163598083 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f52e8094fb0 0x7f52e809e5d0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 rx=0x7f52d80096a0 tx=0x7f52d800b9b0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:46.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:46.475 7f52e4ff9700 1 -- 172.21.2.112:0/1163598083 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f52d80093a0 con 0x7f52e8094fb0 2021-02-10T11:26:46.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:46.475 7f52e4ff9700 1 -- 172.21.2.112:0/1163598083 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52d80067f0 con 0x7f52e8094fb0 2021-02-10T11:26:46.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:46.475 7f52e4ff9700 1 -- 172.21.2.112:0/1163598083 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f52d8002220 con 0x7f52e8094fb0 2021-02-10T11:26:46.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:46.475 7f52ee2b5700 1 -- 172.21.2.112:0/1163598083 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52e80aa6e0 con 0x7f52e8094fb0 2021-02-10T11:26:46.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:46.475 7f52ee2b5700 1 -- 172.21.2.112:0/1163598083 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52e80aac70 con 0x7f52e8094fb0 2021-02-10T11:26:46.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:46.476 7f52e4ff9700 1 -- 172.21.2.112:0/1163598083 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f52d80052e0 con 0x7f52e8094fb0 2021-02-10T11:26:46.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:46.476 7f52e4ff9700 1 --2- 172.21.2.112:0/1163598083 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f52c80186c0 0x7f52c801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:46.469 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:46.476 7f52e4ff9700 1 -- 172.21.2.112:0/1163598083 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f52d80323c0 con 0x7f52e8094fb0 2021-02-10T11:26:46.469 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:46.476 7f52e77fe700 1 --2- 172.21.2.112:0/1163598083 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f52c80186c0 0x7f52c801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:46.469 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:46.477 7f52e77fe700 1 --2- 172.21.2.112:0/1163598083 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f52c80186c0 0x7f52c801abc0 crc :-1 s=READY pgs=329 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:46.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:46.506 7f52ee2b5700 1 -- 172.21.2.112:0/1163598083 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52e80a9750 con 0x7f52e8094fb0 2021-02-10T11:26:46.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:46.509 7f52e4ff9700 1 -- 172.21.2.112:0/1163598083 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f52d80093a0 con 0x7f52e8094fb0 2021-02-10T11:26:46.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:46.636 7f52ee2b5700 1 -- 172.21.2.112:0/1163598083 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348645", "target": ["mgr", ""]}) v1 -- 0x7f52e80aa2a0 con 0x7f52c80186c0 2021-02-10T11:26:46.651 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:46.662 7f52e4ff9700 1 -- 172.21.2.112:0/1163598083 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f52dc006be0 con 0x7f52c80186c0 2021-02-10T11:26:46.658 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:46.669 7f52ee2b5700 1 -- 172.21.2.112:0/1163598083 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f52c80186c0 msgr2=0x7f52c801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:46.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:46.669 7f52ee2b5700 1 --2- 172.21.2.112:0/1163598083 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f52c80186c0 0x7f52c801abc0 crc :-1 s=READY pgs=329 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:46.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:46.669 7f52ee2b5700 1 -- 172.21.2.112:0/1163598083 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f52e8094fb0 msgr2=0x7f52e809e5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:46.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:46.669 7f52ee2b5700 1 --2- 172.21.2.112:0/1163598083 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f52e8094fb0 0x7f52e809e5d0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 rx=0x7f52d80096a0 tx=0x7f52d800b9b0).stop 2021-02-10T11:26:46.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:46.669 7f52ee2b5700 1 -- 172.21.2.112:0/1163598083 shutdown_connections 2021-02-10T11:26:46.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:46.669 7f52ee2b5700 1 --2- 172.21.2.112:0/1163598083 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f52c80186c0 0x7f52c801abc0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:46.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:46.669 7f52ee2b5700 1 --2- 172.21.2.112:0/1163598083 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f52e80c2be0 0x7f52e80c2fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:46.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:46.669 7f52ee2b5700 1 --2- 172.21.2.112:0/1163598083 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f52e80c3600 0x7f52e80a9c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:46.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:46.669 7f52ee2b5700 1 --2- 172.21.2.112:0/1163598083 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f52e8094fb0 0x7f52e809e5d0 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:46.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:46.669 7f52ee2b5700 1 -- 172.21.2.112:0/1163598083 >> 172.21.2.112:0/1163598083 conn(0x7f52e809f630 msgr2=0x7f52e80a0130 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:46.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:46.670 7f52ee2b5700 1 -- 172.21.2.112:0/1163598083 shutdown_connections 2021-02-10T11:26:46.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:46.670 7f52ee2b5700 1 -- 172.21.2.112:0/1163598083 wait complete. 2021-02-10T11:26:46.667 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000348645 2021-02-10T11:26:47.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:47.093 7fd42920a700 1 Processor -- start 2021-02-10T11:26:47.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:47.093 7fd42920a700 1 -- start start 2021-02-10T11:26:47.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:47.093 7fd42920a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd4241b0470 0x7fd4241b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:47.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:47.093 7fd42920a700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fd4240f20e0 0x7fd4240f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:47.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:47.094 7fd42920a700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4240f0b50 con 0x7fd4240ef610 2021-02-10T11:26:47.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:47.094 7fd42920a700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4240f0820 con 0x7fd4240f20e0 2021-02-10T11:26:47.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:47.094 7fd42920a700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4240f24e0 con 0x7fd4241b0470 2021-02-10T11:26:47.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:47.095 7fd422d9d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd4241b0470 0x7fd4241b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:47.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:47.095 7fd42359e700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fd4240f20e0 0x7fd4240f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:47.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:47.095 7fd422d9d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd4241b0470 0x7fd4241b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3301/0 says I am v2:172.21.2.112:45650/0 (socket says 172.21.2.112:45650) 2021-02-10T11:26:47.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:47.095 7fd422d9d700 1 -- 172.21.2.112:0/1285353908 learned_addr learned my addr 172.21.2.112:0/1285353908 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:26:47.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:47.095 7fd42359e700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fd4240f20e0 0x7fd4240f0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3300/0 says I am v2:172.21.2.112:48252/0 (socket says 172.21.2.112:48252) 2021-02-10T11:26:47.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:47.095 7fd42259c700 1 --1- 172.21.2.112:0/1285353908 >> v1:172.21.2.120:6789/0 conn(0x7fd4240ef610 0x7fd4240efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:38990/0 (socket says 172.21.2.112:38990) 2021-02-10T11:26:47.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:47.095 7fd422d9d700 1 -- 172.21.2.112:0/1285353908 >> v2:172.21.2.120:3300/0 conn(0x7fd4240f20e0 msgr2=0x7fd4240f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:26:47.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:47.095 7fd422d9d700 1 --2- 172.21.2.112:0/1285353908 >> v2:172.21.2.120:3300/0 conn(0x7fd4240f20e0 0x7fd4240f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:47.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:47.095 7fd422d9d700 1 -- 172.21.2.112:0/1285353908 >> v1:172.21.2.120:6789/0 conn(0x7fd4240ef610 legacy=0x7fd4240efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:47.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:47.095 7fd422d9d700 1 -- 172.21.2.112:0/1285353908 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4240e52e0 con 0x7fd4241b0470 2021-02-10T11:26:47.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:47.095 7fd422d9d700 1 --2- 172.21.2.112:0/1285353908 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd4241b0470 0x7fd4241b0870 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7fd414006700 tx=0x7fd414009b80).ready entity=mon.2 client_cookie=d92db069a87eabe7 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:47.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:47.096 7fd421d9b700 1 -- 172.21.2.112:0/1285353908 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd41400c0c0 con 0x7fd4241b0470 2021-02-10T11:26:47.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:47.096 7fd421d9b700 1 -- 172.21.2.112:0/1285353908 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd41400c340 con 0x7fd4241b0470 2021-02-10T11:26:47.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:47.096 7fd421d9b700 1 -- 172.21.2.112:0/1285353908 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd41400c860 con 0x7fd4241b0470 2021-02-10T11:26:47.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:47.096 7fd42920a700 1 -- 172.21.2.112:0/1285353908 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd4241b0470 msgr2=0x7fd4241b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:47.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:47.096 7fd42920a700 1 --2- 172.21.2.112:0/1285353908 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd4241b0470 0x7fd4241b0870 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7fd414006700 tx=0x7fd414009b80).stop 2021-02-10T11:26:47.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:47.096 7fd42920a700 1 -- 172.21.2.112:0/1285353908 shutdown_connections 2021-02-10T11:26:47.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:47.096 7fd42920a700 1 --2- 172.21.2.112:0/1285353908 >> v2:172.21.2.120:3300/0 conn(0x7fd4240f20e0 0x7fd4240f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:47.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:47.096 7fd42920a700 1 --2- 172.21.2.112:0/1285353908 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd4241b0470 0x7fd4241b0870 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:47.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:47.096 7fd42920a700 1 -- 172.21.2.112:0/1285353908 >> 172.21.2.112:0/1285353908 conn(0x7fd4240ce3e0 msgr2=0x7fd4240e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:47.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:47.096 7fd42920a700 1 -- 172.21.2.112:0/1285353908 shutdown_connections 2021-02-10T11:26:47.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:47.096 7fd42920a700 1 -- 172.21.2.112:0/1285353908 wait complete. 2021-02-10T11:26:47.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:47.111 7fd42920a700 1 Processor -- start 2021-02-10T11:26:47.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:47.111 7fd42920a700 1 -- start start 2021-02-10T11:26:47.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:47.111 7fd42920a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd4240b7db0 0x7fd4240b8260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:47.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:47.111 7fd42920a700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd4240b8880 0x7fd4241a7450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:47.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:47.111 7fd42920a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd4240b8c80 0x7fd4241a7a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:47.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:47.111 7fd42920a700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4240b79b0 con 0x7fd4240b7db0 2021-02-10T11:26:47.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:47.111 7fd42920a700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4241a8090 con 0x7fd4240b8880 2021-02-10T11:26:47.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:47.111 7fd42920a700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4241a82b0 con 0x7fd4240b8c80 2021-02-10T11:26:47.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:47.111 7fd422d9d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd4240b7db0 0x7fd4240b8260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:47.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:47.111 7fd422d9d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd4240b7db0 0x7fd4240b8260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:45978/0 (socket says 172.21.2.112:45978) 2021-02-10T11:26:47.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:47.111 7fd422d9d700 1 -- 172.21.2.112:0/718168230 learned_addr learned my addr 172.21.2.112:0/718168230 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:26:47.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:47.112 7fd422d9d700 1 -- 172.21.2.112:0/718168230 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd4240b8c80 msgr2=0x7fd4241a7a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:47.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:47.112 7fd422d9d700 1 --2- 172.21.2.112:0/718168230 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd4240b8c80 0x7fd4241a7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:47.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:47.112 7fd422d9d700 1 -- 172.21.2.112:0/718168230 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd4240b8880 msgr2=0x7fd4241a7450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:47.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:47.112 7fd422d9d700 1 --2- 172.21.2.112:0/718168230 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd4240b8880 0x7fd4241a7450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:47.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:47.112 7fd422d9d700 1 -- 172.21.2.112:0/718168230 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd424190c10 con 0x7fd4240b7db0 2021-02-10T11:26:47.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:47.112 7fd422d9d700 1 --2- 172.21.2.112:0/718168230 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd4240b7db0 0x7fd4240b8260 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 rx=0x7fd41400b650 tx=0x7fd41400b9f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:47.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:47.112 7fd41b7fe700 1 -- 172.21.2.112:0/718168230 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd41400bde0 con 0x7fd4240b7db0 2021-02-10T11:26:47.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:47.112 7fd41b7fe700 1 -- 172.21.2.112:0/718168230 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd414006480 con 0x7fd4240b7db0 2021-02-10T11:26:47.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:47.112 7fd41b7fe700 1 -- 172.21.2.112:0/718168230 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd414002220 con 0x7fd4240b7db0 2021-02-10T11:26:47.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:47.112 7fd42920a700 1 -- 172.21.2.112:0/718168230 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd424190c10 con 0x7fd4240b7db0 2021-02-10T11:26:47.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:47.112 7fd42920a700 1 -- 172.21.2.112:0/718168230 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4240c1eb0 con 0x7fd4240b7db0 2021-02-10T11:26:47.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:47.113 7fd41b7fe700 1 -- 172.21.2.112:0/718168230 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd414000c90 con 0x7fd4240b7db0 2021-02-10T11:26:47.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:47.113 7fd41b7fe700 1 --2- 172.21.2.112:0/718168230 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd4080186c0 0x7fd40801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:47.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:47.113 7fd41b7fe700 1 -- 172.21.2.112:0/718168230 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fd414032560 con 0x7fd4240b7db0 2021-02-10T11:26:47.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:47.114 7fd42259c700 1 --2- 172.21.2.112:0/718168230 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd4080186c0 0x7fd40801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:47.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:47.114 7fd42259c700 1 --2- 172.21.2.112:0/718168230 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd4080186c0 0x7fd40801abc0 crc :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:47.132 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:47.143 7fd42920a700 1 -- 172.21.2.112:0/718168230 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd424105b30 con 0x7fd4240b7db0 2021-02-10T11:26:47.135 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:47.146 7fd41b7fe700 1 -- 172.21.2.112:0/718168230 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd4140090b0 con 0x7fd4240b7db0 2021-02-10T11:26:47.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:47.274 7fd42920a700 1 -- 172.21.2.112:0/718168230 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348645", "target": ["mgr", ""]}) v1 -- 0x7fd4240c1eb0 con 0x7fd4080186c0 2021-02-10T11:26:47.266 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:47.278 7fd41b7fe700 1 -- 172.21.2.112:0/718168230 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7fd40c001530 con 0x7fd4080186c0 2021-02-10T11:26:47.271 INFO:teuthology.orchestra.run.gibba012.stdout:/volumes/_nogroup/subvolume_0000000000348645/1ceae693-29bd-43da-91d4-766472d41f46 2021-02-10T11:26:47.272 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:47.283 7fd42920a700 1 -- 172.21.2.112:0/718168230 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd4080186c0 msgr2=0x7fd40801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:47.273 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:47.283 7fd42920a700 1 --2- 172.21.2.112:0/718168230 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd4080186c0 0x7fd40801abc0 crc :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:47.273 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:47.284 7fd42920a700 1 -- 172.21.2.112:0/718168230 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd4240b7db0 msgr2=0x7fd4240b8260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:47.273 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:47.284 7fd42920a700 1 --2- 172.21.2.112:0/718168230 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd4240b7db0 0x7fd4240b8260 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 rx=0x7fd41400b650 tx=0x7fd41400b9f0).stop 2021-02-10T11:26:47.273 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:47.284 7fd42920a700 1 -- 172.21.2.112:0/718168230 shutdown_connections 2021-02-10T11:26:47.273 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:47.284 7fd42920a700 1 --2- 172.21.2.112:0/718168230 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd4080186c0 0x7fd40801abc0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:47.274 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:47.284 7fd42920a700 1 --2- 172.21.2.112:0/718168230 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd4240b8880 0x7fd4241a7450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:47.274 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:47.284 7fd42920a700 1 --2- 172.21.2.112:0/718168230 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd4240b8c80 0x7fd4241a7a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:47.274 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:47.284 7fd42920a700 1 --2- 172.21.2.112:0/718168230 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd4240b7db0 0x7fd4240b8260 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:47.274 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:47.284 7fd42920a700 1 -- 172.21.2.112:0/718168230 >> 172.21.2.112:0/718168230 conn(0x7fd424094e10 msgr2=0x7fd4240a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:47.274 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:47.284 7fd42920a700 1 -- 172.21.2.112:0/718168230 shutdown_connections 2021-02-10T11:26:47.274 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:47.284 7fd42920a700 1 -- 172.21.2.112:0/718168230 wait complete. 2021-02-10T11:26:47.281 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000348645 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000348645/1ceae693-29bd-43da-91d4-766472d41f46 2021-02-10T11:26:47.281 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:47.281 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:47.281 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:47.618 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:26:47.618 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:47.619 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348645/1ceae693-29bd-43da-91d4-766472d41f46/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:47.999 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:26:48.000 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:26:48.000 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.024289 s, 43.2 MB/s 2021-02-10T11:26:48.001 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:48.001 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:48.001 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:48.336 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:26:48.337 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:48.337 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348645/1ceae693-29bd-43da-91d4-766472d41f46/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:48.715 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:26:48.715 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:26:48.715 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0254784 s, 41.2 MB/s 2021-02-10T11:26:48.716 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:48.716 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:48.716 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:49.053 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:26:49.053 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:49.053 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348645/1ceae693-29bd-43da-91d4-766472d41f46/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:49.425 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:26:49.425 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:26:49.425 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0242395 s, 43.3 MB/s 2021-02-10T11:26:49.426 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:49.426 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:49.427 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:49.762 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:26:49.762 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:49.763 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348645/1ceae693-29bd-43da-91d4-766472d41f46/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:50.138 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:26:50.138 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:26:50.138 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0232764 s, 45.0 MB/s 2021-02-10T11:26:50.139 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:50.139 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:50.139 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:50.475 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:26:50.475 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:50.475 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348645/1ceae693-29bd-43da-91d4-766472d41f46/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:50.852 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:26:50.853 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:26:50.853 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.025097 s, 41.8 MB/s 2021-02-10T11:26:50.854 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:50.854 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:50.854 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:51.190 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:26:51.190 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:51.191 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348645/1ceae693-29bd-43da-91d4-766472d41f46/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:51.567 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:26:51.567 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:26:51.567 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0223549 s, 46.9 MB/s 2021-02-10T11:26:51.568 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:51.568 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:51.568 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:51.904 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:26:51.904 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:51.904 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348645/1ceae693-29bd-43da-91d4-766472d41f46/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:52.281 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:26:52.281 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:26:52.281 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0238361 s, 44.0 MB/s 2021-02-10T11:26:52.282 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:52.282 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:52.282 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:52.618 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:26:52.618 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:52.618 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348645/1ceae693-29bd-43da-91d4-766472d41f46/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:52.995 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:26:52.995 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:26:52.995 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.024608 s, 42.6 MB/s 2021-02-10T11:26:52.996 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:52.996 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:52.996 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:53.332 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:26:53.332 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:53.332 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348645/1ceae693-29bd-43da-91d4-766472d41f46/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:53.709 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:26:53.709 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:26:53.709 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0243778 s, 43.0 MB/s 2021-02-10T11:26:53.710 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:53.710 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:53.710 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:54.045 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:26:54.046 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:54.046 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348645/1ceae693-29bd-43da-91d4-766472d41f46/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:54.417 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:26:54.418 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:26:54.418 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0243209 s, 43.1 MB/s 2021-02-10T11:26:54.419 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000348646 2021-02-10T11:26:54.850 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:54.845 7f401b08f700 1 Processor -- start 2021-02-10T11:26:54.850 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:54.845 7f401b08f700 1 -- start start 2021-02-10T11:26:54.851 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:54.845 7f401b08f700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f40141b0470 0x7f40141b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:54.851 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:54.845 7f401b08f700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f40140f20e0 0x7f40140f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:54.851 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:54.845 7f401b08f700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40140f0b50 con 0x7f40140ef610 2021-02-10T11:26:54.851 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:54.846 7f401b08f700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40140f0820 con 0x7f40141b0470 2021-02-10T11:26:54.851 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:54.846 7f401b08f700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40140f24e0 con 0x7f40140f20e0 2021-02-10T11:26:54.851 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:54.846 7f4018eab700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f40141b0470 0x7f40141b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:54.851 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:54.846 7f4018eab700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f40141b0470 0x7f40141b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:45982/0 (socket says 172.21.2.112:45982) 2021-02-10T11:26:54.851 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:54.846 7f4018eab700 1 -- 172.21.2.112:0/1434635677 learned_addr learned my addr 172.21.2.112:0/1434635677 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:26:54.851 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:54.846 7f40196ac700 1 --2- 172.21.2.112:0/1434635677 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f40140f20e0 0x7f40140f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:54.851 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:54.846 7f4018eab700 1 -- 172.21.2.112:0/1434635677 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f40140f20e0 msgr2=0x7f40140f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:26:54.851 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:54.846 7f4018eab700 1 --2- 172.21.2.112:0/1434635677 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f40140f20e0 0x7f40140f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:54.852 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:54.846 7f4018eab700 1 -- 172.21.2.112:0/1434635677 >> v1:172.21.2.120:6789/0 conn(0x7f40140ef610 legacy=0x7f40140efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:54.852 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:54.846 7f4018eab700 1 -- 172.21.2.112:0/1434635677 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40140e52e0 con 0x7f40141b0470 2021-02-10T11:26:54.852 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:54.846 7f4018eab700 1 --2- 172.21.2.112:0/1434635677 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f40141b0470 0x7f40141b0870 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 rx=0x7f3ffc006700 tx=0x7f3ffc009b80).ready entity=mon.0 client_cookie=74fa60bcefe26dde server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:54.852 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:54.846 7f40137fe700 1 -- 172.21.2.112:0/1434635677 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3ffc00c0c0 con 0x7f40141b0470 2021-02-10T11:26:54.852 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:54.847 7f40137fe700 1 -- 172.21.2.112:0/1434635677 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ffc00c440 con 0x7f40141b0470 2021-02-10T11:26:54.852 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:54.847 7f40137fe700 1 -- 172.21.2.112:0/1434635677 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3ffc00c920 con 0x7f40141b0470 2021-02-10T11:26:54.852 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:54.847 7f401b08f700 1 -- 172.21.2.112:0/1434635677 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f40141b0470 msgr2=0x7f40141b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:54.852 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:54.847 7f401b08f700 1 --2- 172.21.2.112:0/1434635677 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f40141b0470 0x7f40141b0870 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 rx=0x7f3ffc006700 tx=0x7f3ffc009b80).stop 2021-02-10T11:26:54.852 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:54.847 7f401b08f700 1 -- 172.21.2.112:0/1434635677 shutdown_connections 2021-02-10T11:26:54.852 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:54.847 7f401b08f700 1 --2- 172.21.2.112:0/1434635677 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f40140f20e0 0x7f40140f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:54.852 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:54.847 7f401b08f700 1 --2- 172.21.2.112:0/1434635677 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f40141b0470 0x7f40141b0870 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:54.852 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:54.847 7f401b08f700 1 -- 172.21.2.112:0/1434635677 >> 172.21.2.112:0/1434635677 conn(0x7f40140ce3e0 msgr2=0x7f40140e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:54.853 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:54.847 7f401b08f700 1 -- 172.21.2.112:0/1434635677 shutdown_connections 2021-02-10T11:26:54.853 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:54.847 7f401b08f700 1 -- 172.21.2.112:0/1434635677 wait complete. 2021-02-10T11:26:54.853 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:54.860 7f401b08f700 1 Processor -- start 2021-02-10T11:26:54.853 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:54.861 7f401b08f700 1 -- start start 2021-02-10T11:26:54.854 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:54.861 7f401b08f700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f40140a0420 0x7f40140a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:54.854 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:54.861 7f401b08f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4014189d00 0x7f401418a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:54.854 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:54.861 7f401b08f700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f401418a720 0x7f40140b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:54.855 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:54.861 7f401b08f700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4014189800 con 0x7f4014189d00 2021-02-10T11:26:54.855 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:54.861 7f401b08f700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f401418ab20 con 0x7f40140a0420 2021-02-10T11:26:54.855 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:54.861 7f401b08f700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40140a1140 con 0x7f401418a720 2021-02-10T11:26:54.855 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:54.861 7f4013fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4014189d00 0x7f401418a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:54.855 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:54.861 7f4013fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4014189d00 0x7f401418a100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:45988/0 (socket says 172.21.2.112:45988) 2021-02-10T11:26:54.855 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:54.861 7f4013fff700 1 -- 172.21.2.112:0/1145300235 learned_addr learned my addr 172.21.2.112:0/1145300235 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:26:54.855 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:54.861 7f4018eab700 1 --2- 172.21.2.112:0/1145300235 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f40140a0420 0x7f40140a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:54.855 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:54.862 7f40196ac700 1 --2- 172.21.2.112:0/1145300235 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f401418a720 0x7f40140b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:54.855 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:54.862 7f4013fff700 1 -- 172.21.2.112:0/1145300235 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f401418a720 msgr2=0x7f40140b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:54.855 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:54.862 7f4013fff700 1 --2- 172.21.2.112:0/1145300235 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f401418a720 0x7f40140b7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:54.855 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:54.862 7f4013fff700 1 -- 172.21.2.112:0/1145300235 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f40140a0420 msgr2=0x7f40140a0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:54.855 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:54.862 7f4013fff700 1 --2- 172.21.2.112:0/1145300235 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f40140a0420 0x7f40140a0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:54.855 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:54.862 7f4013fff700 1 -- 172.21.2.112:0/1145300235 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40140b8200 con 0x7f4014189d00 2021-02-10T11:26:54.856 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:54.862 7f4013fff700 1 --2- 172.21.2.112:0/1145300235 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4014189d00 0x7f401418a100 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 rx=0x7f4004009940 tx=0x7f400400bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:54.856 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:54.862 7f40117fa700 1 -- 172.21.2.112:0/1145300235 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4004005ab0 con 0x7f4014189d00 2021-02-10T11:26:54.856 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:54.862 7f40117fa700 1 -- 172.21.2.112:0/1145300235 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4004005d30 con 0x7f4014189d00 2021-02-10T11:26:54.856 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:54.862 7f401b08f700 1 -- 172.21.2.112:0/1145300235 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40140b8200 con 0x7f4014189d00 2021-02-10T11:26:54.856 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:54.862 7f401b08f700 1 -- 172.21.2.112:0/1145300235 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40140b8200 con 0x7f4014189d00 2021-02-10T11:26:54.856 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:54.862 7f40117fa700 1 -- 172.21.2.112:0/1145300235 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f40040062b0 con 0x7f4014189d00 2021-02-10T11:26:54.856 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:54.863 7f40117fa700 1 -- 172.21.2.112:0/1145300235 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4004006030 con 0x7f4014189d00 2021-02-10T11:26:54.856 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:54.863 7f40117fa700 1 --2- 172.21.2.112:0/1145300235 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3ff80186c0 0x7f3ff801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:54.856 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:54.863 7f40117fa700 1 -- 172.21.2.112:0/1145300235 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f40040328b0 con 0x7f4014189d00 2021-02-10T11:26:54.856 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:54.864 7f4018eab700 1 --2- 172.21.2.112:0/1145300235 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3ff80186c0 0x7f3ff801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:54.857 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:54.865 7f4018eab700 1 --2- 172.21.2.112:0/1145300235 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3ff80186c0 0x7f3ff801abc0 crc :-1 s=READY pgs=331 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:54.885 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:54.895 7f401b08f700 1 -- 172.21.2.112:0/1145300235 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f401418ab20 con 0x7f4014189d00 2021-02-10T11:26:54.887 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:54.897 7f40117fa700 1 -- 172.21.2.112:0/1145300235 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f400400be30 con 0x7f4014189d00 2021-02-10T11:26:55.013 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:55.023 7f401b08f700 1 -- 172.21.2.112:0/1145300235 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348646", "target": ["mgr", ""]}) v1 -- 0x7f401418ab20 con 0x7f3ff80186c0 2021-02-10T11:26:55.038 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:55.049 7f40117fa700 1 -- 172.21.2.112:0/1145300235 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f3ffc006430 con 0x7f3ff80186c0 2021-02-10T11:26:55.045 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:55.055 7f401b08f700 1 -- 172.21.2.112:0/1145300235 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3ff80186c0 msgr2=0x7f3ff801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:55.045 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:55.055 7f401b08f700 1 --2- 172.21.2.112:0/1145300235 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3ff80186c0 0x7f3ff801abc0 crc :-1 s=READY pgs=331 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:55.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:55.055 7f401b08f700 1 -- 172.21.2.112:0/1145300235 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4014189d00 msgr2=0x7f401418a100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:55.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:55.055 7f401b08f700 1 --2- 172.21.2.112:0/1145300235 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4014189d00 0x7f401418a100 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 rx=0x7f4004009940 tx=0x7f400400bcf0).stop 2021-02-10T11:26:55.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:55.056 7f401b08f700 1 -- 172.21.2.112:0/1145300235 shutdown_connections 2021-02-10T11:26:55.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:55.056 7f401b08f700 1 --2- 172.21.2.112:0/1145300235 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3ff80186c0 0x7f3ff801abc0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:55.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:55.056 7f401b08f700 1 --2- 172.21.2.112:0/1145300235 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f401418a720 0x7f40140b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:55.047 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:55.056 7f401b08f700 1 --2- 172.21.2.112:0/1145300235 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4014189d00 0x7f401418a100 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:55.047 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:55.056 7f401b08f700 1 --2- 172.21.2.112:0/1145300235 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f40140a0420 0x7f40140a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:55.047 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:55.056 7f401b08f700 1 -- 172.21.2.112:0/1145300235 >> 172.21.2.112:0/1145300235 conn(0x7f4014094e10 msgr2=0x7f40140a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:55.047 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:55.056 7f401b08f700 1 -- 172.21.2.112:0/1145300235 shutdown_connections 2021-02-10T11:26:55.047 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:55.056 7f401b08f700 1 -- 172.21.2.112:0/1145300235 wait complete. 2021-02-10T11:26:55.054 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000348646 2021-02-10T11:26:55.488 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:55.480 7f7d3308b700 1 Processor -- start 2021-02-10T11:26:55.488 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:55.480 7f7d3308b700 1 -- start start 2021-02-10T11:26:55.488 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:55.481 7f7d3308b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7d2c1b0870 0x7f7d2c0f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:55.488 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:55.481 7f7d3308b700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f7d2c0efe00 0x7f7d2c0f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:55.488 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:55.481 7f7d3308b700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d2c1b0c70 con 0x7f7d2c1b0470 2021-02-10T11:26:55.488 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:55.481 7f7d3308b700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d2c0f0b50 con 0x7f7d2c0efe00 2021-02-10T11:26:55.488 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:55.481 7f7d3308b700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d2c0f0820 con 0x7f7d2c1b0870 2021-02-10T11:26:55.489 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:55.481 7f7d30ea7700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f7d2c1b0470 0x7f7d2c0ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:45458/0 (socket says 172.21.2.112:45458) 2021-02-10T11:26:55.489 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:55.481 7f7d30ea7700 1 -- 172.21.2.112:0/4289555237 learned_addr learned my addr 172.21.2.112:0/4289555237 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:26:55.489 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:55.481 7f7d2b7fe700 1 -- 172.21.2.112:0/4289555237 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2699355706 0 0) 0x7f7d140009b0 con 0x7f7d2c1b0470 2021-02-10T11:26:55.489 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:55.482 7f7d2b7fe700 1 -- 172.21.2.112:0/4289555237 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d18002fe0 con 0x7f7d2c1b0470 2021-02-10T11:26:55.489 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:55.482 7f7d316a8700 1 --2- 172.21.2.112:0/4289555237 >> v2:172.21.2.120:3300/0 conn(0x7f7d2c0efe00 0x7f7d2c0f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:55.489 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:55.482 7f7d2bfff700 1 --2- 172.21.2.112:0/4289555237 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7d2c1b0870 0x7f7d2c0f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:55.489 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:55.482 7f7d2b7fe700 1 -- 172.21.2.112:0/4289555237 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 539269089 0 0) 0x7f7d14001010 con 0x7f7d2c1b0470 2021-02-10T11:26:55.489 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:55.482 7f7d2b7fe700 1 -- 172.21.2.112:0/4289555237 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7d2c1b0870 msgr2=0x7f7d2c0f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:26:55.489 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:55.482 7f7d2b7fe700 1 --2- 172.21.2.112:0/4289555237 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7d2c1b0870 0x7f7d2c0f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:55.489 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:55.482 7f7d2b7fe700 1 -- 172.21.2.112:0/4289555237 >> v2:172.21.2.120:3300/0 conn(0x7f7d2c0efe00 msgr2=0x7f7d2c0f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:26:55.489 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:55.482 7f7d2b7fe700 1 --2- 172.21.2.112:0/4289555237 >> v2:172.21.2.120:3300/0 conn(0x7f7d2c0efe00 0x7f7d2c0f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:55.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:55.482 7f7d2b7fe700 1 -- 172.21.2.112:0/4289555237 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d2c0e52e0 con 0x7f7d2c1b0470 2021-02-10T11:26:55.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:55.482 7f7d2b7fe700 1 -- 172.21.2.112:0/4289555237 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f7d14001590 con 0x7f7d2c1b0470 2021-02-10T11:26:55.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:55.482 7f7d2b7fe700 1 -- 172.21.2.112:0/4289555237 >> v1:172.21.2.112:6789/0 conn(0x7f7d2c1b0470 legacy=0x7f7d2c0ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:55.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:55.482 7f7d2b7fe700 1 --2- 172.21.2.112:0/4289555237 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7d18003b10 0x7f7d18005f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:55.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:55.482 7f7d2b7fe700 1 --2- 172.21.2.112:0/4289555237 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7d18006820 0x7f7d18008d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:55.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:55.482 7f7d2b7fe700 1 --2- 172.21.2.112:0/4289555237 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7d18009620 0x7f7d1800bb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:55.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:55.482 7f7d2b7fe700 1 -- 172.21.2.112:0/4289555237 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d18050370 con 0x7f7d18003b10 2021-02-10T11:26:55.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:55.482 7f7d2b7fe700 1 -- 172.21.2.112:0/4289555237 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d18050590 con 0x7f7d18006820 2021-02-10T11:26:55.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:55.482 7f7d2b7fe700 1 -- 172.21.2.112:0/4289555237 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d180507b0 con 0x7f7d18009620 2021-02-10T11:26:55.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:55.482 7f7d2bfff700 1 --2- 172.21.2.112:0/4289555237 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7d2c1b0870 0x7f7d2c0f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:26:55.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:55.482 7f7d30ea7700 1 --2- 172.21.2.112:0/4289555237 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7d18003b10 0x7f7d18005f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:55.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:55.482 7f7d316a8700 1 --2- 172.21.2.112:0/4289555237 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7d18009620 0x7f7d1800bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:55.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:55.482 7f7d30ea7700 1 -- 172.21.2.112:0/4289555237 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7d18009620 msgr2=0x7f7d1800bb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:26:55.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:55.482 7f7d30ea7700 1 --2- 172.21.2.112:0/4289555237 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7d18009620 0x7f7d1800bb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:55.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:55.482 7f7d30ea7700 1 -- 172.21.2.112:0/4289555237 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7d18006820 msgr2=0x7f7d18008d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:26:55.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:55.482 7f7d2bfff700 1 --2- 172.21.2.112:0/4289555237 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7d18006820 0x7f7d18008d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:55.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:55.482 7f7d30ea7700 1 --2- 172.21.2.112:0/4289555237 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7d18006820 0x7f7d18008d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:55.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:55.482 7f7d30ea7700 1 -- 172.21.2.112:0/4289555237 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d18050a50 con 0x7f7d18003b10 2021-02-10T11:26:55.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:55.483 7f7d2bfff700 1 --2- 172.21.2.112:0/4289555237 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7d18006820 0x7f7d18008d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:26:55.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:55.483 7f7d316a8700 1 --2- 172.21.2.112:0/4289555237 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7d18009620 0x7f7d1800bb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:26:55.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:55.483 7f7d30ea7700 1 --2- 172.21.2.112:0/4289555237 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7d18003b10 0x7f7d18005f80 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 rx=0x7f7d14007560 tx=0x7f7d14007d10).ready entity=mon.0 client_cookie=c395c610ada90b61 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:55.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:55.483 7f7d2b7fe700 1 -- 172.21.2.112:0/4289555237 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7d140070c0 con 0x7f7d18003b10 2021-02-10T11:26:55.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:55.483 7f7d2b7fe700 1 -- 172.21.2.112:0/4289555237 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d140009b0 con 0x7f7d18003b10 2021-02-10T11:26:55.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:55.483 7f7d2b7fe700 1 -- 172.21.2.112:0/4289555237 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7d1400a710 con 0x7f7d18003b10 2021-02-10T11:26:55.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:55.483 7f7d3308b700 1 -- 172.21.2.112:0/4289555237 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7d18003b10 msgr2=0x7f7d18005f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:55.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:55.483 7f7d3308b700 1 --2- 172.21.2.112:0/4289555237 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7d18003b10 0x7f7d18005f80 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 rx=0x7f7d14007560 tx=0x7f7d14007d10).stop 2021-02-10T11:26:55.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:55.483 7f7d316a8700 1 -- 172.21.2.112:0/4289555237 reap_dead start 2021-02-10T11:26:55.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:55.484 7f7d3308b700 1 -- 172.21.2.112:0/4289555237 shutdown_connections 2021-02-10T11:26:55.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:55.484 7f7d3308b700 1 -- 172.21.2.112:0/4289555237 >> 172.21.2.112:0/4289555237 conn(0x7f7d2c0ce3e0 msgr2=0x7f7d2c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:55.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:55.484 7f7d3308b700 1 -- 172.21.2.112:0/4289555237 shutdown_connections 2021-02-10T11:26:55.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:55.484 7f7d3308b700 1 -- 172.21.2.112:0/4289555237 wait complete. 2021-02-10T11:26:55.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:55.498 7f7d3308b700 1 Processor -- start 2021-02-10T11:26:55.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:55.498 7f7d3308b700 1 -- start start 2021-02-10T11:26:55.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:55.499 7f7d3308b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7d2c0a56a0 0x7f7d2c0a5f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:55.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:55.499 7f7d3308b700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7d2c0a6550 0x7f7d2c1a7430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:55.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:55.499 7f7d3308b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7d2c0a6950 0x7f7d2c1a7a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:55.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:55.499 7f7d3308b700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d2c1085d0 con 0x7f7d2c0a6950 2021-02-10T11:26:55.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:55.499 7f7d3308b700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d2c0a5aa0 con 0x7f7d2c0a6550 2021-02-10T11:26:55.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:55.499 7f7d3308b700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d2c1a8070 con 0x7f7d2c0a56a0 2021-02-10T11:26:55.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:55.499 7f7d316a8700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7d2c0a6950 0x7f7d2c1a7a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:55.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:55.499 7f7d316a8700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7d2c0a6950 0x7f7d2c1a7a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:46006/0 (socket says 172.21.2.112:46006) 2021-02-10T11:26:55.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:55.499 7f7d316a8700 1 -- 172.21.2.112:0/1417255780 learned_addr learned my addr 172.21.2.112:0/1417255780 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:26:55.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:55.499 7f7d2bfff700 1 --2- 172.21.2.112:0/1417255780 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7d2c0a6550 0x7f7d2c1a7430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:55.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:55.499 7f7d316a8700 1 -- 172.21.2.112:0/1417255780 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7d2c0a56a0 msgr2=0x7f7d2c0a5f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:55.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:55.499 7f7d30ea7700 1 --2- 172.21.2.112:0/1417255780 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7d2c0a56a0 0x7f7d2c0a5f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:55.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:55.499 7f7d316a8700 1 --2- 172.21.2.112:0/1417255780 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7d2c0a56a0 0x7f7d2c0a5f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:55.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:55.499 7f7d316a8700 1 -- 172.21.2.112:0/1417255780 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7d2c0a6550 msgr2=0x7f7d2c1a7430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:55.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:55.499 7f7d316a8700 1 --2- 172.21.2.112:0/1417255780 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7d2c0a6550 0x7f7d2c1a7430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:55.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:55.499 7f7d316a8700 1 -- 172.21.2.112:0/1417255780 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d2c108ed0 con 0x7f7d2c0a6950 2021-02-10T11:26:55.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:55.499 7f7d2bfff700 1 --2- 172.21.2.112:0/1417255780 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7d2c0a6550 0x7f7d2c1a7430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:26:55.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:55.499 7f7d30ea7700 1 --2- 172.21.2.112:0/1417255780 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7d2c0a56a0 0x7f7d2c0a5f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:26:55.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:55.499 7f7d316a8700 1 --2- 172.21.2.112:0/1417255780 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7d2c0a6950 0x7f7d2c1a7a50 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 rx=0x7f7d24006f40 tx=0x7f7d2400b9b0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:55.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:55.500 7f7d297fa700 1 -- 172.21.2.112:0/1417255780 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7d2400bda0 con 0x7f7d2c0a6950 2021-02-10T11:26:55.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:55.500 7f7d297fa700 1 -- 172.21.2.112:0/1417255780 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d24000a70 con 0x7f7d2c0a6950 2021-02-10T11:26:55.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:55.500 7f7d297fa700 1 -- 172.21.2.112:0/1417255780 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7d24002220 con 0x7f7d2c0a6950 2021-02-10T11:26:55.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:55.500 7f7d3308b700 1 -- 172.21.2.112:0/1417255780 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d2c108ed0 con 0x7f7d2c0a6950 2021-02-10T11:26:55.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:55.500 7f7d3308b700 1 -- 172.21.2.112:0/1417255780 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d2c109460 con 0x7f7d2c0a6950 2021-02-10T11:26:55.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:55.500 7f7d297fa700 1 -- 172.21.2.112:0/1417255780 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7d24005330 con 0x7f7d2c0a6950 2021-02-10T11:26:55.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:55.501 7f7d297fa700 1 --2- 172.21.2.112:0/1417255780 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7d100186c0 0x7f7d1001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:55.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:55.501 7f7d297fa700 1 -- 172.21.2.112:0/1417255780 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f7d24032400 con 0x7f7d2c0a6950 2021-02-10T11:26:55.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:55.501 7f7d30ea7700 1 --2- 172.21.2.112:0/1417255780 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7d100186c0 0x7f7d1001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:55.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:55.501 7f7d30ea7700 1 --2- 172.21.2.112:0/1417255780 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7d100186c0 0x7f7d1001abc0 crc :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:55.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:55.530 7f7d3308b700 1 -- 172.21.2.112:0/1417255780 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d2c105b30 con 0x7f7d2c0a6950 2021-02-10T11:26:55.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:55.533 7f7d297fa700 1 -- 172.21.2.112:0/1417255780 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7d24006830 con 0x7f7d2c0a6950 2021-02-10T11:26:55.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:55.664 7f7d3308b700 1 -- 172.21.2.112:0/1417255780 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348646", "target": ["mgr", ""]}) v1 -- 0x7f7d2c109460 con 0x7f7d100186c0 2021-02-10T11:26:55.657 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:55.668 7f7d297fa700 1 -- 172.21.2.112:0/1417255780 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f7d14001960 con 0x7f7d100186c0 2021-02-10T11:26:55.661 INFO:teuthology.orchestra.run.gibba012.stdout:/volumes/_nogroup/subvolume_0000000000348646/8e901153-fcdd-4db5-a9a4-4715398cc55c 2021-02-10T11:26:55.663 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:55.673 7f7d3308b700 1 -- 172.21.2.112:0/1417255780 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7d100186c0 msgr2=0x7f7d1001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:55.663 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:55.673 7f7d3308b700 1 --2- 172.21.2.112:0/1417255780 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7d100186c0 0x7f7d1001abc0 crc :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:55.663 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:55.673 7f7d3308b700 1 -- 172.21.2.112:0/1417255780 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7d2c0a6950 msgr2=0x7f7d2c1a7a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:55.663 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:55.673 7f7d3308b700 1 --2- 172.21.2.112:0/1417255780 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7d2c0a6950 0x7f7d2c1a7a50 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 rx=0x7f7d24006f40 tx=0x7f7d2400b9b0).stop 2021-02-10T11:26:55.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:55.673 7f7d3308b700 1 -- 172.21.2.112:0/1417255780 shutdown_connections 2021-02-10T11:26:55.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:55.673 7f7d3308b700 1 --2- 172.21.2.112:0/1417255780 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7d100186c0 0x7f7d1001abc0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:55.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:55.673 7f7d3308b700 1 --2- 172.21.2.112:0/1417255780 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7d2c0a6550 0x7f7d2c1a7430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:55.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:55.673 7f7d3308b700 1 --2- 172.21.2.112:0/1417255780 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7d2c0a6950 0x7f7d2c1a7a50 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:55.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:55.673 7f7d3308b700 1 --2- 172.21.2.112:0/1417255780 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7d2c0a56a0 0x7f7d2c0a5f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:55.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:55.673 7f7d3308b700 1 -- 172.21.2.112:0/1417255780 >> 172.21.2.112:0/1417255780 conn(0x7f7d2c09c070 msgr2=0x7f7d2c0a6fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:55.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:55.674 7f7d3308b700 1 -- 172.21.2.112:0/1417255780 shutdown_connections 2021-02-10T11:26:55.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:26:55.674 7f7d3308b700 1 -- 172.21.2.112:0/1417255780 wait complete. 2021-02-10T11:26:55.671 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000348646 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000348646/8e901153-fcdd-4db5-a9a4-4715398cc55c 2021-02-10T11:26:55.671 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:55.671 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:55.672 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:56.009 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:26:56.009 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:56.010 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348646/8e901153-fcdd-4db5-a9a4-4715398cc55c/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:56.393 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:26:56.393 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:26:56.393 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0263755 s, 39.8 MB/s 2021-02-10T11:26:56.394 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:56.395 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:56.395 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:56.730 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:26:56.731 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:56.731 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348646/8e901153-fcdd-4db5-a9a4-4715398cc55c/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:57.108 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:26:57.108 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:26:57.108 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.025791 s, 40.7 MB/s 2021-02-10T11:26:57.109 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:57.110 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:57.110 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:57.446 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:26:57.446 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:57.446 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348646/8e901153-fcdd-4db5-a9a4-4715398cc55c/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:57.823 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:26:57.824 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:26:57.824 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0260825 s, 40.2 MB/s 2021-02-10T11:26:57.825 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:57.825 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:57.825 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:58.159 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:26:58.159 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:58.160 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348646/8e901153-fcdd-4db5-a9a4-4715398cc55c/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:58.537 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:26:58.537 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:26:58.537 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0255475 s, 41.0 MB/s 2021-02-10T11:26:58.538 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:58.538 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:58.538 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:58.874 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:26:58.874 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:58.874 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348646/8e901153-fcdd-4db5-a9a4-4715398cc55c/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:59.252 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:26:59.252 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:26:59.252 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0256604 s, 40.9 MB/s 2021-02-10T11:26:59.254 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:59.254 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:59.254 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:59.590 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:26:59.590 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:59.590 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348646/8e901153-fcdd-4db5-a9a4-4715398cc55c/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:59.965 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:26:59.965 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:26:59.965 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0247855 s, 42.3 MB/s 2021-02-10T11:26:59.966 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:59.966 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:59.966 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:00.302 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:27:00.302 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:00.303 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348646/8e901153-fcdd-4db5-a9a4-4715398cc55c/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:00.676 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:27:00.922 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:27:00.922 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0227195 s, 46.2 MB/s 2021-02-10T11:27:00.923 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:00.924 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:00.924 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:01.260 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:27:01.260 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:01.260 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348646/8e901153-fcdd-4db5-a9a4-4715398cc55c/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:01.638 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:27:01.638 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:27:01.638 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0259228 s, 40.4 MB/s 2021-02-10T11:27:01.639 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:01.639 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:01.640 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:01.976 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:27:01.976 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:01.976 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348646/8e901153-fcdd-4db5-a9a4-4715398cc55c/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:02.352 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:27:02.352 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:27:02.352 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0237847 s, 44.1 MB/s 2021-02-10T11:27:02.353 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:02.353 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:02.353 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:02.688 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:27:02.689 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:02.689 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348646/8e901153-fcdd-4db5-a9a4-4715398cc55c/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:03.064 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:27:03.064 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:27:03.064 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0239982 s, 43.7 MB/s 2021-02-10T11:27:03.065 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000348647 2021-02-10T11:27:03.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:03.491 7fa747f56700 1 Processor -- start 2021-02-10T11:27:03.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:03.491 7fa747f56700 1 -- start start 2021-02-10T11:27:03.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:03.492 7fa747f56700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fa7401b0470 0x7fa7401b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:03.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:03.492 7fa747f56700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fa7400ef610 0x7fa7400efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:03.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:03.492 7fa747f56700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7400f0b50 con 0x7fa7400f0030 2021-02-10T11:27:03.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:03.492 7fa747f56700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7400e52e0 con 0x7fa7400ef610 2021-02-10T11:27:03.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:03.492 7fa747f56700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7400e5490 con 0x7fa7401b0470 2021-02-10T11:27:03.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:03.492 7fa745571700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fa7400ef610 0x7fa7400efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:03.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:03.492 7fa745571700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fa7400ef610 0x7fa7400efa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:46014/0 (socket says 172.21.2.112:46014) 2021-02-10T11:27:03.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:03.492 7fa745571700 1 -- 172.21.2.112:0/2508496175 learned_addr learned my addr 172.21.2.112:0/2508496175 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:27:03.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:03.492 7fa745571700 1 -- 172.21.2.112:0/2508496175 >> v2:172.21.2.120:3300/0 conn(0x7fa7401b0470 msgr2=0x7fa7401b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:27:03.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:03.492 7fa745571700 1 --2- 172.21.2.112:0/2508496175 >> v2:172.21.2.120:3300/0 conn(0x7fa7401b0470 0x7fa7401b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:03.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:03.492 7fa745571700 1 -- 172.21.2.112:0/2508496175 >> v1:172.21.2.112:6789/0 conn(0x7fa7400f0030 legacy=0x7fa7400f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:03.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:03.492 7fa745571700 1 -- 172.21.2.112:0/2508496175 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7400f0430 con 0x7fa7400ef610 2021-02-10T11:27:03.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:03.492 7fa744d70700 1 -- 172.21.2.112:0/2508496175 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3749156849 0 0) 0x7fa738000f90 con 0x7fa7400f0030 2021-02-10T11:27:03.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:03.493 7fa745571700 1 --2- 172.21.2.112:0/2508496175 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa7400ef610 0x7fa7400efa10 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 rx=0x7fa73c006700 tx=0x7fa73c009b00).ready entity=mon.0 client_cookie=892f7054ee344fd5 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:03.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:03.493 7fa744d70700 1 -- 172.21.2.112:0/2508496175 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa73c00c0c0 con 0x7fa7400ef610 2021-02-10T11:27:03.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:03.493 7fa744d70700 1 -- 172.21.2.112:0/2508496175 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa73c00c400 con 0x7fa7400ef610 2021-02-10T11:27:03.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:03.493 7fa744d70700 1 -- 172.21.2.112:0/2508496175 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa73c00c8e0 con 0x7fa7400ef610 2021-02-10T11:27:03.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:03.493 7fa747f56700 1 -- 172.21.2.112:0/2508496175 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa7400ef610 msgr2=0x7fa7400efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:03.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:03.493 7fa747f56700 1 --2- 172.21.2.112:0/2508496175 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa7400ef610 0x7fa7400efa10 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 rx=0x7fa73c006700 tx=0x7fa73c009b00).stop 2021-02-10T11:27:03.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:03.493 7fa747f56700 1 -- 172.21.2.112:0/2508496175 shutdown_connections 2021-02-10T11:27:03.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:03.493 7fa747f56700 1 --2- 172.21.2.112:0/2508496175 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa7400ef610 0x7fa7400efa10 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:03.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:03.493 7fa747f56700 1 --2- 172.21.2.112:0/2508496175 >> v2:172.21.2.120:3300/0 conn(0x7fa7401b0470 0x7fa7401b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:03.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:03.493 7fa747f56700 1 -- 172.21.2.112:0/2508496175 >> 172.21.2.112:0/2508496175 conn(0x7fa7400ce3e0 msgr2=0x7fa7400e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:03.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:03.494 7fa747f56700 1 -- 172.21.2.112:0/2508496175 shutdown_connections 2021-02-10T11:27:03.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:03.494 7fa747f56700 1 -- 172.21.2.112:0/2508496175 wait complete. 2021-02-10T11:27:03.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:03.508 7fa747f56700 1 Processor -- start 2021-02-10T11:27:03.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:03.508 7fa747f56700 1 -- start start 2021-02-10T11:27:03.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:03.508 7fa747f56700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa7400b20b0 0x7fa7400ac970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:03.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:03.509 7fa747f56700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa7400b24b0 0x7fa740191dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:03.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:03.509 7fa747f56700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa7400acf90 0x7fa7401923e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:03.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:03.509 7fa747f56700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7400b1b30 con 0x7fa7400b20b0 2021-02-10T11:27:03.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:03.509 7fa747f56700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7401a04e0 con 0x7fa7400b24b0 2021-02-10T11:27:03.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:03.509 7fa747f56700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7401a06c0 con 0x7fa7400acf90 2021-02-10T11:27:03.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:03.509 7fa745d72700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa7400b20b0 0x7fa7400ac970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:03.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:03.509 7fa745d72700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa7400b20b0 0x7fa7400ac970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:46018/0 (socket says 172.21.2.112:46018) 2021-02-10T11:27:03.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:03.509 7fa745d72700 1 -- 172.21.2.112:0/2200468113 learned_addr learned my addr 172.21.2.112:0/2200468113 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:27:03.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:03.509 7fa745d72700 1 -- 172.21.2.112:0/2200468113 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa7400acf90 msgr2=0x7fa7401923e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:03.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:03.509 7fa745d72700 1 --2- 172.21.2.112:0/2200468113 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa7400acf90 0x7fa7401923e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:03.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:03.509 7fa745d72700 1 -- 172.21.2.112:0/2200468113 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa7400b24b0 msgr2=0x7fa740191dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:03.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:03.509 7fa745d72700 1 --2- 172.21.2.112:0/2200468113 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa7400b24b0 0x7fa740191dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:03.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:03.509 7fa745d72700 1 -- 172.21.2.112:0/2200468113 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa740192be0 con 0x7fa7400b20b0 2021-02-10T11:27:03.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:03.509 7fa745d72700 1 --2- 172.21.2.112:0/2200468113 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa7400b20b0 0x7fa7400ac970 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 rx=0x7fa734006f00 tx=0x7fa73400b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:03.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:03.509 7fa7327fc700 1 -- 172.21.2.112:0/2200468113 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa73400bd30 con 0x7fa7400b20b0 2021-02-10T11:27:03.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:03.509 7fa7327fc700 1 -- 172.21.2.112:0/2200468113 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa734000c90 con 0x7fa7400b20b0 2021-02-10T11:27:03.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:03.509 7fa7327fc700 1 -- 172.21.2.112:0/2200468113 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa7340022a0 con 0x7fa7400b20b0 2021-02-10T11:27:03.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:03.509 7fa747f56700 1 -- 172.21.2.112:0/2200468113 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa740192be0 con 0x7fa7400b20b0 2021-02-10T11:27:03.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:03.510 7fa747f56700 1 -- 172.21.2.112:0/2200468113 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa740192be0 con 0x7fa7400b20b0 2021-02-10T11:27:03.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:03.510 7fa7327fc700 1 -- 172.21.2.112:0/2200468113 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa734005310 con 0x7fa7400b20b0 2021-02-10T11:27:03.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:03.510 7fa7327fc700 1 --2- 172.21.2.112:0/2200468113 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa728018730 0x7fa72801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:03.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:03.510 7fa7327fc700 1 -- 172.21.2.112:0/2200468113 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fa734005310 con 0x7fa7400b20b0 2021-02-10T11:27:03.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:03.511 7fa745571700 1 --2- 172.21.2.112:0/2200468113 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa728018730 0x7fa72801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:03.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:03.511 7fa745571700 1 --2- 172.21.2.112:0/2200468113 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa728018730 0x7fa72801ac10 crc :-1 s=READY pgs=333 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:03.532 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:03.542 7fa747f56700 1 -- 172.21.2.112:0/2200468113 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa740105af0 con 0x7fa7400b20b0 2021-02-10T11:27:03.534 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:03.544 7fa7327fc700 1 -- 172.21.2.112:0/2200468113 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa734000940 con 0x7fa7400b20b0 2021-02-10T11:27:03.660 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:03.670 7fa747f56700 1 -- 172.21.2.112:0/2200468113 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348647", "target": ["mgr", ""]}) v1 -- 0x7fa7401a06c0 con 0x7fa728018730 2021-02-10T11:27:03.686 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:03.697 7fa7327fc700 1 -- 172.21.2.112:0/2200468113 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fa73c0091a0 con 0x7fa728018730 2021-02-10T11:27:03.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:03.703 7fa747f56700 1 -- 172.21.2.112:0/2200468113 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa728018730 msgr2=0x7fa72801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:03.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:03.703 7fa747f56700 1 --2- 172.21.2.112:0/2200468113 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa728018730 0x7fa72801ac10 crc :-1 s=READY pgs=333 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:03.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:03.703 7fa747f56700 1 -- 172.21.2.112:0/2200468113 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa7400b20b0 msgr2=0x7fa7400ac970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:03.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:03.703 7fa747f56700 1 --2- 172.21.2.112:0/2200468113 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa7400b20b0 0x7fa7400ac970 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 rx=0x7fa734006f00 tx=0x7fa73400b940).stop 2021-02-10T11:27:03.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:03.703 7fa747f56700 1 -- 172.21.2.112:0/2200468113 shutdown_connections 2021-02-10T11:27:03.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:03.704 7fa747f56700 1 --2- 172.21.2.112:0/2200468113 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa728018730 0x7fa72801ac10 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:03.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:03.704 7fa747f56700 1 --2- 172.21.2.112:0/2200468113 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa7400b24b0 0x7fa740191dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:03.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:03.704 7fa747f56700 1 --2- 172.21.2.112:0/2200468113 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa7400acf90 0x7fa7401923e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:03.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:03.704 7fa747f56700 1 --2- 172.21.2.112:0/2200468113 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa7400b20b0 0x7fa7400ac970 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:03.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:03.704 7fa747f56700 1 -- 172.21.2.112:0/2200468113 >> 172.21.2.112:0/2200468113 conn(0x7fa7400bf4e0 msgr2=0x7fa7400be3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:03.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:03.704 7fa747f56700 1 -- 172.21.2.112:0/2200468113 shutdown_connections 2021-02-10T11:27:03.695 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:03.704 7fa747f56700 1 -- 172.21.2.112:0/2200468113 wait complete. 2021-02-10T11:27:03.702 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000348647 2021-02-10T11:27:04.132 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:04.126 7f66b3230700 1 Processor -- start 2021-02-10T11:27:04.133 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:04.127 7f66b3230700 1 -- start start 2021-02-10T11:27:04.133 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:04.127 7f66b3230700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f66ac1b0470 0x7f66ac1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:04.133 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:04.127 7f66b3230700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f66ac0ef610 0x7f66ac0efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:04.133 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:04.127 7f66b3230700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66ac0f0b50 con 0x7f66ac0f0030 2021-02-10T11:27:04.133 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:04.127 7f66b3230700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66ac0e52e0 con 0x7f66ac1b0470 2021-02-10T11:27:04.133 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:04.127 7f66b3230700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66ac0e5490 con 0x7f66ac0ef610 2021-02-10T11:27:04.133 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:04.127 7f66b104c700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f66ac1b0470 0x7f66ac1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:04.133 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:04.127 7f66b104c700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f66ac1b0470 0x7f66ac1b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:46026/0 (socket says 172.21.2.112:46026) 2021-02-10T11:27:04.133 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:04.127 7f66b104c700 1 -- 172.21.2.112:0/4095259214 learned_addr learned my addr 172.21.2.112:0/4095259214 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:27:04.133 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:04.128 7f66b084b700 1 --2- 172.21.2.112:0/4095259214 >> v2:172.21.2.120:3300/0 conn(0x7f66ac0ef610 0x7f66ac0efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:04.134 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:04.128 7f66b104c700 1 -- 172.21.2.112:0/4095259214 >> v2:172.21.2.120:3300/0 conn(0x7f66ac0ef610 msgr2=0x7f66ac0efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:27:04.134 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:04.128 7f66b104c700 1 --2- 172.21.2.112:0/4095259214 >> v2:172.21.2.120:3300/0 conn(0x7f66ac0ef610 0x7f66ac0efa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:04.134 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:04.128 7f66b104c700 1 -- 172.21.2.112:0/4095259214 >> v1:172.21.2.120:6789/0 conn(0x7f66ac0f0030 legacy=0x7f66ac0f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:27:04.134 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:04.128 7f66b104c700 1 -- 172.21.2.112:0/4095259214 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66ac0f0430 con 0x7f66ac1b0470 2021-02-10T11:27:04.134 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:04.128 7f66b084b700 1 --2- 172.21.2.112:0/4095259214 >> v2:172.21.2.120:3300/0 conn(0x7f66ac0ef610 0x7f66ac0efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:27:04.134 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:04.128 7f66b104c700 1 --2- 172.21.2.112:0/4095259214 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f66ac1b0470 0x7f66ac1b0870 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 rx=0x7f66a0006700 tx=0x7f66a0009b80).ready entity=mon.0 client_cookie=9ab1dc333b1a6b9c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:04.134 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:04.128 7f669ffff700 1 -- 172.21.2.112:0/4095259214 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f66a000c0c0 con 0x7f66ac1b0470 2021-02-10T11:27:04.134 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:04.128 7f669ffff700 1 -- 172.21.2.112:0/4095259214 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66a000c440 con 0x7f66ac1b0470 2021-02-10T11:27:04.134 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:04.128 7f669ffff700 1 -- 172.21.2.112:0/4095259214 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f66a000c960 con 0x7f66ac1b0470 2021-02-10T11:27:04.134 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:04.128 7f66b3230700 1 -- 172.21.2.112:0/4095259214 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f66ac1b0470 msgr2=0x7f66ac1b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:04.134 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:04.128 7f66b3230700 1 --2- 172.21.2.112:0/4095259214 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f66ac1b0470 0x7f66ac1b0870 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 rx=0x7f66a0006700 tx=0x7f66a0009b80).stop 2021-02-10T11:27:04.134 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:04.129 7f66b3230700 1 -- 172.21.2.112:0/4095259214 shutdown_connections 2021-02-10T11:27:04.135 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:04.129 7f66b3230700 1 --2- 172.21.2.112:0/4095259214 >> v2:172.21.2.120:3300/0 conn(0x7f66ac0ef610 0x7f66ac0efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:04.135 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:04.129 7f66b3230700 1 --2- 172.21.2.112:0/4095259214 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f66ac1b0470 0x7f66ac1b0870 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:04.135 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:04.129 7f66b3230700 1 -- 172.21.2.112:0/4095259214 >> 172.21.2.112:0/4095259214 conn(0x7f66ac0ce3e0 msgr2=0x7f66ac0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:04.135 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:04.129 7f66b3230700 1 -- 172.21.2.112:0/4095259214 shutdown_connections 2021-02-10T11:27:04.135 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:04.129 7f66b3230700 1 -- 172.21.2.112:0/4095259214 wait complete. 2021-02-10T11:27:04.136 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:04.143 7f66b3230700 1 Processor -- start 2021-02-10T11:27:04.136 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:04.143 7f66b3230700 1 -- start start 2021-02-10T11:27:04.136 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:04.144 7f66b3230700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f66ac0b20b0 0x7f66ac0ac970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:04.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:04.144 7f66b3230700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f66ac0b24b0 0x7f66ac191dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:04.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:04.144 7f66b3230700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f66ac0acf90 0x7f66ac1923e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:04.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:04.144 7f66b3230700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66ac0b1b30 con 0x7f66ac0b20b0 2021-02-10T11:27:04.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:04.144 7f66b3230700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66ac1a04e0 con 0x7f66ac0b24b0 2021-02-10T11:27:04.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:04.144 7f66b3230700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66ac1a06c0 con 0x7f66ac0acf90 2021-02-10T11:27:04.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:04.144 7f66b104c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f66ac0b20b0 0x7f66ac0ac970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:04.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:04.144 7f66b104c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f66ac0b20b0 0x7f66ac0ac970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:46030/0 (socket says 172.21.2.112:46030) 2021-02-10T11:27:04.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:04.144 7f66b104c700 1 -- 172.21.2.112:0/1288130628 learned_addr learned my addr 172.21.2.112:0/1288130628 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:27:04.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:04.144 7f66b104c700 1 -- 172.21.2.112:0/1288130628 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f66ac0acf90 msgr2=0x7f66ac1923e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:04.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:04.144 7f66b104c700 1 --2- 172.21.2.112:0/1288130628 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f66ac0acf90 0x7f66ac1923e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:04.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:04.144 7f66b104c700 1 -- 172.21.2.112:0/1288130628 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f66ac0b24b0 msgr2=0x7f66ac191dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:04.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:04.144 7f66b104c700 1 --2- 172.21.2.112:0/1288130628 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f66ac0b24b0 0x7f66ac191dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:04.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:04.144 7f66b104c700 1 -- 172.21.2.112:0/1288130628 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66ac192be0 con 0x7f66ac0b20b0 2021-02-10T11:27:04.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:04.144 7f66b104c700 1 --2- 172.21.2.112:0/1288130628 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f66ac0b20b0 0x7f66ac0ac970 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 rx=0x7f66a000b580 tx=0x7f66a000b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:04.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:04.144 7f669dffb700 1 -- 172.21.2.112:0/1288130628 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f66a000bd10 con 0x7f66ac0b20b0 2021-02-10T11:27:04.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:04.144 7f669dffb700 1 -- 172.21.2.112:0/1288130628 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66a0002030 con 0x7f66ac0b20b0 2021-02-10T11:27:04.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:04.144 7f66b3230700 1 -- 172.21.2.112:0/1288130628 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66ac192be0 con 0x7f66ac0b20b0 2021-02-10T11:27:04.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:04.144 7f669dffb700 1 -- 172.21.2.112:0/1288130628 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f66a0002400 con 0x7f66ac0b20b0 2021-02-10T11:27:04.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:04.144 7f66b3230700 1 -- 172.21.2.112:0/1288130628 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66ac192be0 con 0x7f66ac0b20b0 2021-02-10T11:27:04.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:04.145 7f669dffb700 1 -- 172.21.2.112:0/1288130628 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f66a0000c90 con 0x7f66ac0b20b0 2021-02-10T11:27:04.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:04.145 7f669dffb700 1 --2- 172.21.2.112:0/1288130628 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f668c018730 0x7f668c01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:04.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:04.145 7f669dffb700 1 -- 172.21.2.112:0/1288130628 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f66a0032650 con 0x7f66ac0b20b0 2021-02-10T11:27:04.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:04.145 7f66b084b700 1 --2- 172.21.2.112:0/1288130628 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f668c018730 0x7f668c01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:04.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:04.146 7f66b084b700 1 --2- 172.21.2.112:0/1288130628 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f668c018730 0x7f668c01ac10 crc :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:04.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:04.177 7f66b3230700 1 -- 172.21.2.112:0/1288130628 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66ac105af0 con 0x7f66ac0b20b0 2021-02-10T11:27:04.169 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:04.180 7f669dffb700 1 -- 172.21.2.112:0/1288130628 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f66a001f510 con 0x7f66ac0b20b0 2021-02-10T11:27:04.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:04.309 7f66b3230700 1 -- 172.21.2.112:0/1288130628 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348647", "target": ["mgr", ""]}) v1 -- 0x7f66ac0f0b50 con 0x7f668c018730 2021-02-10T11:27:04.303 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:04.314 7f669dffb700 1 -- 172.21.2.112:0/1288130628 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f66a80077a0 con 0x7f668c018730 2021-02-10T11:27:04.306 INFO:teuthology.orchestra.run.gibba012.stdout:/volumes/_nogroup/subvolume_0000000000348647/b1b24b1d-1858-4c3e-82c8-caf15b740db4 2021-02-10T11:27:04.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:04.318 7f66b3230700 1 -- 172.21.2.112:0/1288130628 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f668c018730 msgr2=0x7f668c01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:04.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:04.318 7f66b3230700 1 --2- 172.21.2.112:0/1288130628 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f668c018730 0x7f668c01ac10 crc :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:04.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:04.318 7f66b3230700 1 -- 172.21.2.112:0/1288130628 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f66ac0b20b0 msgr2=0x7f66ac0ac970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:04.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:04.318 7f66b3230700 1 --2- 172.21.2.112:0/1288130628 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f66ac0b20b0 0x7f66ac0ac970 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 rx=0x7f66a000b580 tx=0x7f66a000b920).stop 2021-02-10T11:27:04.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:04.319 7f66b3230700 1 -- 172.21.2.112:0/1288130628 shutdown_connections 2021-02-10T11:27:04.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:04.319 7f66b3230700 1 --2- 172.21.2.112:0/1288130628 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f668c018730 0x7f668c01ac10 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:04.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:04.319 7f66b3230700 1 --2- 172.21.2.112:0/1288130628 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f66ac0b24b0 0x7f66ac191dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:04.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:04.319 7f66b3230700 1 --2- 172.21.2.112:0/1288130628 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f66ac0acf90 0x7f66ac1923e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:04.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:04.319 7f66b3230700 1 --2- 172.21.2.112:0/1288130628 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f66ac0b20b0 0x7f66ac0ac970 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:04.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:04.319 7f66b3230700 1 -- 172.21.2.112:0/1288130628 >> 172.21.2.112:0/1288130628 conn(0x7f66ac0bf4e0 msgr2=0x7f66ac0be3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:04.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:04.319 7f66b3230700 1 -- 172.21.2.112:0/1288130628 shutdown_connections 2021-02-10T11:27:04.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:04.319 7f66b3230700 1 -- 172.21.2.112:0/1288130628 wait complete. 2021-02-10T11:27:04.317 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000348647 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000348647/b1b24b1d-1858-4c3e-82c8-caf15b740db4 2021-02-10T11:27:04.317 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:04.317 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:04.317 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:04.653 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:27:04.653 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:04.653 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348647/b1b24b1d-1858-4c3e-82c8-caf15b740db4/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:05.033 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:27:05.034 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:27:05.034 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0235571 s, 44.5 MB/s 2021-02-10T11:27:05.035 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:05.035 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:05.035 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:05.371 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:27:05.371 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:05.371 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348647/b1b24b1d-1858-4c3e-82c8-caf15b740db4/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:05.746 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:27:05.746 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:27:05.747 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0244158 s, 42.9 MB/s 2021-02-10T11:27:05.747 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:05.748 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:05.748 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:06.083 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:27:06.084 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:06.084 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348647/b1b24b1d-1858-4c3e-82c8-caf15b740db4/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:06.459 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:27:06.459 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:27:06.460 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0250648 s, 41.8 MB/s 2021-02-10T11:27:06.460 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:06.461 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:06.461 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:06.796 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:27:06.797 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:06.797 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348647/b1b24b1d-1858-4c3e-82c8-caf15b740db4/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:07.173 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:27:07.173 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:27:07.173 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0243662 s, 43.0 MB/s 2021-02-10T11:27:07.174 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:07.174 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:07.174 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:07.510 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:27:07.510 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:07.510 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348647/b1b24b1d-1858-4c3e-82c8-caf15b740db4/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:07.886 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:27:07.886 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:27:07.887 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.024021 s, 43.7 MB/s 2021-02-10T11:27:07.887 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:07.888 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:07.888 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:08.222 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:27:08.223 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:08.223 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348647/b1b24b1d-1858-4c3e-82c8-caf15b740db4/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:08.598 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:27:08.599 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:27:08.599 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0246798 s, 42.5 MB/s 2021-02-10T11:27:08.600 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:08.600 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:08.600 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:08.936 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:27:08.937 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:08.937 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348647/b1b24b1d-1858-4c3e-82c8-caf15b740db4/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:09.314 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:27:09.314 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:27:09.314 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0253055 s, 41.4 MB/s 2021-02-10T11:27:09.315 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:09.315 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:09.315 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:09.651 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:27:09.651 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:09.651 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348647/b1b24b1d-1858-4c3e-82c8-caf15b740db4/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:10.025 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:27:10.025 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:27:10.025 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0239823 s, 43.7 MB/s 2021-02-10T11:27:10.026 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:10.026 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:10.026 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:10.362 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:27:10.362 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:10.362 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348647/b1b24b1d-1858-4c3e-82c8-caf15b740db4/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:10.740 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:27:10.740 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:27:10.740 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0250776 s, 41.8 MB/s 2021-02-10T11:27:10.741 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:10.742 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:10.742 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:11.077 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:27:11.078 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:11.078 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348647/b1b24b1d-1858-4c3e-82c8-caf15b740db4/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:11.237 DEBUG:teuthology.orchestra.run.gibba012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:27:11.240 DEBUG:teuthology.orchestra.run.gibba020:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:27:11.453 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:27:11.453 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:27:11.453 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0228625 s, 45.9 MB/s 2021-02-10T11:27:11.455 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000348648 2021-02-10T11:27:11.884 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:11.879 7f61bcf42700 1 Processor -- start 2021-02-10T11:27:11.884 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:11.879 7f61bcf42700 1 -- start start 2021-02-10T11:27:11.884 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:11.879 7f61bcf42700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f61b81b0870 0x7f61b80f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:11.885 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:11.879 7f61bcf42700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f61b80efe00 0x7f61b80f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:11.885 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:11.879 7f61bcf42700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61b81b0c70 con 0x7f61b81b0470 2021-02-10T11:27:11.885 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:11.879 7f61bcf42700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61b80f0b50 con 0x7f61b80efe00 2021-02-10T11:27:11.885 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:11.879 7f61bcf42700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61b80f0820 con 0x7f61b81b0870 2021-02-10T11:27:11.885 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:11.879 7f61b7314700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f61b80efe00 0x7f61b80f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:11.885 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:11.879 7f61b7314700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f61b80efe00 0x7f61b80f0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:46040/0 (socket says 172.21.2.112:46040) 2021-02-10T11:27:11.885 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:11.879 7f61b7314700 1 -- 172.21.2.112:0/2395764046 learned_addr learned my addr 172.21.2.112:0/2395764046 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:27:11.885 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:11.880 7f61b7314700 1 -- 172.21.2.112:0/2395764046 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f61b81b0870 msgr2=0x7f61b80f20e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:27:11.885 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:11.880 7f61b7314700 1 --2- 172.21.2.112:0/2395764046 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f61b81b0870 0x7f61b80f20e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:11.885 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:11.880 7f61b7314700 1 -- 172.21.2.112:0/2395764046 >> v1:172.21.2.120:6789/0 conn(0x7f61b81b0470 legacy=0x7f61b80ef610 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:27:11.885 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:11.880 7f61b7314700 1 -- 172.21.2.112:0/2395764046 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61b80e52e0 con 0x7f61b80efe00 2021-02-10T11:27:11.885 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:11.880 7f61b7314700 1 --2- 172.21.2.112:0/2395764046 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f61b80efe00 0x7f61b80f0200 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 rx=0x7f61b0006f00 tx=0x7f61b0009f10).ready entity=mon.0 client_cookie=326bd156bf4abf2b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:11.886 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:11.880 7f61b5b11700 1 -- 172.21.2.112:0/2395764046 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f61b000b3b0 con 0x7f61b80efe00 2021-02-10T11:27:11.886 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:11.880 7f61b5b11700 1 -- 172.21.2.112:0/2395764046 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61b000b630 con 0x7f61b80efe00 2021-02-10T11:27:11.886 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:11.880 7f61b5b11700 1 -- 172.21.2.112:0/2395764046 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f61b000bb50 con 0x7f61b80efe00 2021-02-10T11:27:11.886 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:11.880 7f61bcf42700 1 -- 172.21.2.112:0/2395764046 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f61b80efe00 msgr2=0x7f61b80f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:11.886 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:11.880 7f61bcf42700 1 --2- 172.21.2.112:0/2395764046 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f61b80efe00 0x7f61b80f0200 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 rx=0x7f61b0006f00 tx=0x7f61b0009f10).stop 2021-02-10T11:27:11.886 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:11.880 7f61bcf42700 1 -- 172.21.2.112:0/2395764046 shutdown_connections 2021-02-10T11:27:11.886 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:11.880 7f61bcf42700 1 --2- 172.21.2.112:0/2395764046 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f61b80efe00 0x7f61b80f0200 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:11.886 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:11.880 7f61bcf42700 1 --2- 172.21.2.112:0/2395764046 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f61b81b0870 0x7f61b80f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:11.886 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:11.880 7f61bcf42700 1 -- 172.21.2.112:0/2395764046 >> 172.21.2.112:0/2395764046 conn(0x7f61b80ce3e0 msgr2=0x7f61b80e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:11.886 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:11.881 7f61bcf42700 1 -- 172.21.2.112:0/2395764046 shutdown_connections 2021-02-10T11:27:11.886 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:11.881 7f61bcf42700 1 -- 172.21.2.112:0/2395764046 wait complete. 2021-02-10T11:27:11.888 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:11.895 7f61bcf42700 1 Processor -- start 2021-02-10T11:27:11.888 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:11.895 7f61bcf42700 1 -- start start 2021-02-10T11:27:11.888 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:11.895 7f61bcf42700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f61b80c36a0 0x7f61b80c2b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:11.888 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:11.895 7f61bcf42700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f61b80c3190 0x7f61b80b1d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:11.888 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:11.895 7f61bcf42700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f61b80b2370 0x7f61b80b2d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:11.888 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:11.895 7f61bcf42700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61b80b27c0 con 0x7f61b80c36a0 2021-02-10T11:27:11.888 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:11.895 7f61bcf42700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61b80b3360 con 0x7f61b80b2370 2021-02-10T11:27:11.888 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:11.895 7f61bcf42700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61b80b3540 con 0x7f61b80c3190 2021-02-10T11:27:11.889 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:11.895 7f61b6b13700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f61b80c36a0 0x7f61b80c2b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:11.889 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:11.895 7f61b6b13700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f61b80c36a0 0x7f61b80c2b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:46046/0 (socket says 172.21.2.112:46046) 2021-02-10T11:27:11.889 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:11.895 7f61b6b13700 1 -- 172.21.2.112:0/3943346550 learned_addr learned my addr 172.21.2.112:0/3943346550 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:27:11.889 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:11.896 7f61b6b13700 1 -- 172.21.2.112:0/3943346550 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f61b80c3190 msgr2=0x7f61b80b1d50 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:27:11.889 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:11.896 7f61b6b13700 1 --2- 172.21.2.112:0/3943346550 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f61b80c3190 0x7f61b80b1d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:11.889 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:11.896 7f61b6b13700 1 -- 172.21.2.112:0/3943346550 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f61b80b2370 msgr2=0x7f61b80b2d40 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:27:11.889 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:11.896 7f61b6b13700 1 --2- 172.21.2.112:0/3943346550 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f61b80b2370 0x7f61b80b2d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:11.889 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:11.896 7f61b6b13700 1 -- 172.21.2.112:0/3943346550 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61b80b3760 con 0x7f61b80c36a0 2021-02-10T11:27:11.889 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:11.896 7f61b6b13700 1 --2- 172.21.2.112:0/3943346550 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f61b80c36a0 0x7f61b80c2b70 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 rx=0x7f61a000b5b0 tx=0x7f61a000b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:11.889 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:11.896 7f61af7fe700 1 -- 172.21.2.112:0/3943346550 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f61a000bd40 con 0x7f61b80c36a0 2021-02-10T11:27:11.889 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:11.896 7f61af7fe700 1 -- 172.21.2.112:0/3943346550 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61a00009b0 con 0x7f61b80c36a0 2021-02-10T11:27:11.889 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:11.896 7f61bcf42700 1 -- 172.21.2.112:0/3943346550 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61b80b3760 con 0x7f61b80c36a0 2021-02-10T11:27:11.889 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:11.896 7f61bcf42700 1 -- 172.21.2.112:0/3943346550 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61b80b3760 con 0x7f61b80c36a0 2021-02-10T11:27:11.889 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:11.896 7f61af7fe700 1 -- 172.21.2.112:0/3943346550 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f61a00062a0 con 0x7f61b80c36a0 2021-02-10T11:27:11.890 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:11.897 7f61af7fe700 1 -- 172.21.2.112:0/3943346550 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f61a0006020 con 0x7f61b80c36a0 2021-02-10T11:27:11.890 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:11.897 7f61af7fe700 1 --2- 172.21.2.112:0/3943346550 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f619c018730 0x7f619c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:11.890 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:11.897 7f61af7fe700 1 -- 172.21.2.112:0/3943346550 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f61a00327c0 con 0x7f61b80c36a0 2021-02-10T11:27:11.890 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:11.897 7f61b6312700 1 --2- 172.21.2.112:0/3943346550 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f619c018730 0x7f619c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:11.890 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:11.898 7f61b6312700 1 --2- 172.21.2.112:0/3943346550 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f619c018730 0x7f619c01abc0 crc :-1 s=READY pgs=335 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:11.919 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:11.929 7f61bcf42700 1 -- 172.21.2.112:0/3943346550 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61b80edd10 con 0x7f61b80c36a0 2021-02-10T11:27:11.921 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:11.932 7f61af7fe700 1 -- 172.21.2.112:0/3943346550 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f61a001f500 con 0x7f61b80c36a0 2021-02-10T11:27:12.049 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:12.059 7f61bcf42700 1 -- 172.21.2.112:0/3943346550 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348648", "target": ["mgr", ""]}) v1 -- 0x7f61b80edd10 con 0x7f619c018730 2021-02-10T11:27:12.073 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:12.084 7f61af7fe700 1 -- 172.21.2.112:0/3943346550 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f61a80009b0 con 0x7f619c018730 2021-02-10T11:27:12.082 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:12.092 7f61bcf42700 1 -- 172.21.2.112:0/3943346550 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f619c018730 msgr2=0x7f619c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:12.082 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:12.092 7f61bcf42700 1 --2- 172.21.2.112:0/3943346550 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f619c018730 0x7f619c01abc0 crc :-1 s=READY pgs=335 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:12.083 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:12.092 7f61bcf42700 1 -- 172.21.2.112:0/3943346550 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f61b80c36a0 msgr2=0x7f61b80c2b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:12.083 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:12.092 7f61bcf42700 1 --2- 172.21.2.112:0/3943346550 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f61b80c36a0 0x7f61b80c2b70 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 rx=0x7f61a000b5b0 tx=0x7f61a000b950).stop 2021-02-10T11:27:12.083 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:12.092 7f61bcf42700 1 -- 172.21.2.112:0/3943346550 shutdown_connections 2021-02-10T11:27:12.083 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:12.092 7f61bcf42700 1 --2- 172.21.2.112:0/3943346550 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f619c018730 0x7f619c01abc0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:12.083 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:12.092 7f61bcf42700 1 --2- 172.21.2.112:0/3943346550 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f61b80b2370 0x7f61b80b2d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:12.083 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:12.092 7f61bcf42700 1 --2- 172.21.2.112:0/3943346550 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f61b80c3190 0x7f61b80b1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:12.083 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:12.092 7f61bcf42700 1 --2- 172.21.2.112:0/3943346550 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f61b80c36a0 0x7f61b80c2b70 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:12.083 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:12.092 7f61bcf42700 1 -- 172.21.2.112:0/3943346550 >> 172.21.2.112:0/3943346550 conn(0x7f61b809f630 msgr2=0x7f61b80a0130 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:12.083 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:12.093 7f61bcf42700 1 -- 172.21.2.112:0/3943346550 shutdown_connections 2021-02-10T11:27:12.083 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:12.093 7f61bcf42700 1 -- 172.21.2.112:0/3943346550 wait complete. 2021-02-10T11:27:12.090 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000348648 2021-02-10T11:27:12.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:12.515 7f1360831700 1 Processor -- start 2021-02-10T11:27:12.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:12.515 7f1360831700 1 -- start start 2021-02-10T11:27:12.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:12.515 7f1360831700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f135c1b0470 0x7f135c1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:12.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:12.515 7f1360831700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f135c0ef610 0x7f135c0efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:12.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:12.516 7f1360831700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f135c0f0030 0x7f135c0f0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:12.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:12.516 7f1360831700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f135c0e5ea0 con 0x7f135c0f0030 2021-02-10T11:27:12.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:12.516 7f1360831700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f135c0f0b50 con 0x7f135c1b0470 2021-02-10T11:27:12.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:12.516 7f1360831700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f135c0e52e0 con 0x7f135c0ef610 2021-02-10T11:27:12.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:12.516 7f135ae1d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f135c0f0030 0x7f135c0f0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:12.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:12.516 7f135ae1d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f135c0f0030 0x7f135c0f0430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:46050/0 (socket says 172.21.2.112:46050) 2021-02-10T11:27:12.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:12.516 7f135ae1d700 1 -- 172.21.2.112:0/3303675376 learned_addr learned my addr 172.21.2.112:0/3303675376 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:27:12.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:12.516 7f135ae1d700 1 -- 172.21.2.112:0/3303675376 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f135c0ef610 msgr2=0x7f135c0efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:27:12.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:12.516 7f135ae1d700 1 --2- 172.21.2.112:0/3303675376 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f135c0ef610 0x7f135c0efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:12.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:12.516 7f135ae1d700 1 -- 172.21.2.112:0/3303675376 >> v2:172.21.2.120:3300/0 conn(0x7f135c1b0470 msgr2=0x7f135c1b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:27:12.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:12.516 7f135ae1d700 1 --2- 172.21.2.112:0/3303675376 >> v2:172.21.2.120:3300/0 conn(0x7f135c1b0470 0x7f135c1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:12.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:12.516 7f135ae1d700 1 -- 172.21.2.112:0/3303675376 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f135c0f10d0 con 0x7f135c0f0030 2021-02-10T11:27:12.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:12.516 7f135ae1d700 1 --2- 172.21.2.112:0/3303675376 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f135c0f0030 0x7f135c0f0430 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 rx=0x7f1354006f00 tx=0x7f1354009f10).ready entity=mon.0 client_cookie=e4ae451b47c08782 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:12.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:12.516 7f135961a700 1 -- 172.21.2.112:0/3303675376 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f135400b3b0 con 0x7f135c0f0030 2021-02-10T11:27:12.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:12.516 7f135961a700 1 -- 172.21.2.112:0/3303675376 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f135400b670 con 0x7f135c0f0030 2021-02-10T11:27:12.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:12.516 7f135961a700 1 -- 172.21.2.112:0/3303675376 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f135400bb90 con 0x7f135c0f0030 2021-02-10T11:27:12.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:12.516 7f1360831700 1 -- 172.21.2.112:0/3303675376 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f135c0f0030 msgr2=0x7f135c0f0430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:12.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:12.516 7f1360831700 1 --2- 172.21.2.112:0/3303675376 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f135c0f0030 0x7f135c0f0430 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 rx=0x7f1354006f00 tx=0x7f1354009f10).stop 2021-02-10T11:27:12.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:12.517 7f1360831700 1 -- 172.21.2.112:0/3303675376 shutdown_connections 2021-02-10T11:27:12.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:12.517 7f1360831700 1 --2- 172.21.2.112:0/3303675376 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f135c0f0030 0x7f135c0f0430 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:12.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:12.517 7f1360831700 1 --2- 172.21.2.112:0/3303675376 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f135c0ef610 0x7f135c0efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:12.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:12.517 7f1360831700 1 --2- 172.21.2.112:0/3303675376 >> v2:172.21.2.120:3300/0 conn(0x7f135c1b0470 0x7f135c1b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:12.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:12.517 7f1360831700 1 -- 172.21.2.112:0/3303675376 >> 172.21.2.112:0/3303675376 conn(0x7f135c0ce3e0 msgr2=0x7f135c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:12.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:12.517 7f1360831700 1 -- 172.21.2.112:0/3303675376 shutdown_connections 2021-02-10T11:27:12.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:12.517 7f1360831700 1 -- 172.21.2.112:0/3303675376 wait complete. 2021-02-10T11:27:12.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:12.532 7f1360831700 1 Processor -- start 2021-02-10T11:27:12.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:12.532 7f1360831700 1 -- start start 2021-02-10T11:27:12.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:12.532 7f1360831700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f135c0b20b0 0x7f135c0ac970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:12.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:12.532 7f1360831700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f135c0b24b0 0x7f135c191dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:12.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:12.532 7f1360831700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f135c0acf90 0x7f135c1923e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:12.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:12.532 7f1360831700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f135c0b1b30 con 0x7f135c0acf90 2021-02-10T11:27:12.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:12.532 7f1360831700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f135c1a04e0 con 0x7f135c0b24b0 2021-02-10T11:27:12.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:12.532 7f1360831700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f135c1a06c0 con 0x7f135c0b20b0 2021-02-10T11:27:12.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:12.532 7f135ae1d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f135c0acf90 0x7f135c1923e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:12.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:12.532 7f135ae1d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f135c0acf90 0x7f135c1923e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:46058/0 (socket says 172.21.2.112:46058) 2021-02-10T11:27:12.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:12.532 7f135ae1d700 1 -- 172.21.2.112:0/760690937 learned_addr learned my addr 172.21.2.112:0/760690937 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:27:12.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:12.532 7f135ae1d700 1 -- 172.21.2.112:0/760690937 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f135c0b20b0 msgr2=0x7f135c0ac970 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:27:12.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:12.532 7f135ae1d700 1 --2- 172.21.2.112:0/760690937 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f135c0b20b0 0x7f135c0ac970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:12.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:12.532 7f135ae1d700 1 -- 172.21.2.112:0/760690937 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f135c0b24b0 msgr2=0x7f135c191dc0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:27:12.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:12.532 7f135ae1d700 1 --2- 172.21.2.112:0/760690937 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f135c0b24b0 0x7f135c191dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:12.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:12.532 7f135ae1d700 1 -- 172.21.2.112:0/760690937 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f135c192be0 con 0x7f135c0acf90 2021-02-10T11:27:12.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:12.533 7f135ae1d700 1 --2- 172.21.2.112:0/760690937 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f135c0acf90 0x7f135c1923e0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 rx=0x7f135400b8e0 tx=0x7f135400bc80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:12.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:12.533 7f1352ffd700 1 -- 172.21.2.112:0/760690937 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1354003030 con 0x7f135c0acf90 2021-02-10T11:27:12.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:12.533 7f1352ffd700 1 -- 172.21.2.112:0/760690937 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f135400bdc0 con 0x7f135c0acf90 2021-02-10T11:27:12.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:12.533 7f1352ffd700 1 -- 172.21.2.112:0/760690937 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f135400c4e0 con 0x7f135c0acf90 2021-02-10T11:27:12.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:12.533 7f1360831700 1 -- 172.21.2.112:0/760690937 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f135c192be0 con 0x7f135c0acf90 2021-02-10T11:27:12.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:12.533 7f1360831700 1 -- 172.21.2.112:0/760690937 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f135c192be0 con 0x7f135c0acf90 2021-02-10T11:27:12.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:12.533 7f1352ffd700 1 -- 172.21.2.112:0/760690937 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1354001270 con 0x7f135c0acf90 2021-02-10T11:27:12.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:12.533 7f1352ffd700 1 --2- 172.21.2.112:0/760690937 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1340018730 0x7f134001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:12.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:12.533 7f1352ffd700 1 -- 172.21.2.112:0/760690937 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f1354032b30 con 0x7f135c0acf90 2021-02-10T11:27:12.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:12.534 7f135a61c700 1 --2- 172.21.2.112:0/760690937 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1340018730 0x7f134001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:12.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:12.534 7f135a61c700 1 --2- 172.21.2.112:0/760690937 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1340018730 0x7f134001abc0 crc :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:12.555 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:12.565 7f1360831700 1 -- 172.21.2.112:0/760690937 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f135c105af0 con 0x7f135c0acf90 2021-02-10T11:27:12.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:12.568 7f1352ffd700 1 -- 172.21.2.112:0/760690937 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1354006c80 con 0x7f135c0acf90 2021-02-10T11:27:12.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:12.697 7f1360831700 1 -- 172.21.2.112:0/760690937 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348648", "target": ["mgr", ""]}) v1 -- 0x7f135c1a04e0 con 0x7f1340018730 2021-02-10T11:27:12.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:12.702 7f1352ffd700 1 -- 172.21.2.112:0/760690937 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f1344006a10 con 0x7f1340018730 2021-02-10T11:27:12.695 INFO:teuthology.orchestra.run.gibba012.stdout:/volumes/_nogroup/subvolume_0000000000348648/62da7199-f7b9-41e9-8cb1-6e6b46414fc0 2021-02-10T11:27:12.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:12.707 7f1360831700 1 -- 172.21.2.112:0/760690937 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1340018730 msgr2=0x7f134001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:12.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:12.707 7f1360831700 1 --2- 172.21.2.112:0/760690937 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1340018730 0x7f134001abc0 crc :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:12.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:12.707 7f1360831700 1 -- 172.21.2.112:0/760690937 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f135c0acf90 msgr2=0x7f135c1923e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:12.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:12.707 7f1360831700 1 --2- 172.21.2.112:0/760690937 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f135c0acf90 0x7f135c1923e0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 rx=0x7f135400b8e0 tx=0x7f135400bc80).stop 2021-02-10T11:27:12.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:12.707 7f1360831700 1 -- 172.21.2.112:0/760690937 shutdown_connections 2021-02-10T11:27:12.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:12.707 7f1360831700 1 --2- 172.21.2.112:0/760690937 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1340018730 0x7f134001abc0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:12.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:12.707 7f1360831700 1 --2- 172.21.2.112:0/760690937 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f135c0b24b0 0x7f135c191dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:12.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:12.707 7f1360831700 1 --2- 172.21.2.112:0/760690937 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f135c0acf90 0x7f135c1923e0 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:12.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:12.707 7f1360831700 1 --2- 172.21.2.112:0/760690937 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f135c0b20b0 0x7f135c0ac970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:12.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:12.707 7f1360831700 1 -- 172.21.2.112:0/760690937 >> 172.21.2.112:0/760690937 conn(0x7f135c0bf4e0 msgr2=0x7f135c0be3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:12.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:12.708 7f1360831700 1 -- 172.21.2.112:0/760690937 shutdown_connections 2021-02-10T11:27:12.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:12.708 7f1360831700 1 -- 172.21.2.112:0/760690937 wait complete. 2021-02-10T11:27:12.705 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000348648 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000348648/62da7199-f7b9-41e9-8cb1-6e6b46414fc0 2021-02-10T11:27:12.705 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:12.705 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:12.706 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:13.041 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:27:13.042 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:13.042 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348648/62da7199-f7b9-41e9-8cb1-6e6b46414fc0/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:13.425 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:27:13.425 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:27:13.425 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0259038 s, 40.5 MB/s 2021-02-10T11:27:13.426 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:13.426 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:13.426 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:13.763 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:27:13.763 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:13.763 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348648/62da7199-f7b9-41e9-8cb1-6e6b46414fc0/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:14.139 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:27:14.139 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:27:14.139 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0251233 s, 41.7 MB/s 2021-02-10T11:27:14.140 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:14.140 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:14.141 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:14.475 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:27:14.476 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:14.476 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348648/62da7199-f7b9-41e9-8cb1-6e6b46414fc0/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:14.848 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:27:14.848 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:27:14.848 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0244817 s, 42.8 MB/s 2021-02-10T11:27:14.849 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:14.849 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:14.849 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:15.185 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:27:15.186 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:15.186 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348648/62da7199-f7b9-41e9-8cb1-6e6b46414fc0/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:15.563 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:27:15.563 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:27:15.563 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0256265 s, 40.9 MB/s 2021-02-10T11:27:15.564 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:15.564 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:15.564 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:15.901 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:27:15.901 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:15.901 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348648/62da7199-f7b9-41e9-8cb1-6e6b46414fc0/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:16.279 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:27:16.279 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:27:16.279 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0231143 s, 45.4 MB/s 2021-02-10T11:27:16.280 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:16.280 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:16.280 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:16.622 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:27:16.623 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:16.623 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348648/62da7199-f7b9-41e9-8cb1-6e6b46414fc0/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:16.997 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:27:16.998 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:27:16.998 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0248794 s, 42.1 MB/s 2021-02-10T11:27:16.999 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:16.999 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:16.999 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:17.333 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:27:17.334 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:17.334 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348648/62da7199-f7b9-41e9-8cb1-6e6b46414fc0/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:17.711 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:27:17.711 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:27:17.711 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0246471 s, 42.5 MB/s 2021-02-10T11:27:17.712 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:17.712 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:17.713 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:18.049 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:27:18.049 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:18.049 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348648/62da7199-f7b9-41e9-8cb1-6e6b46414fc0/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:18.422 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:27:18.422 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:27:18.422 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0231721 s, 45.3 MB/s 2021-02-10T11:27:18.423 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:18.423 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:18.423 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:18.759 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:27:18.759 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:18.760 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348648/62da7199-f7b9-41e9-8cb1-6e6b46414fc0/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:19.135 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:27:19.136 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:27:19.136 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0240551 s, 43.6 MB/s 2021-02-10T11:27:19.136 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:19.137 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:19.137 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:19.471 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:27:19.472 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:19.472 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348648/62da7199-f7b9-41e9-8cb1-6e6b46414fc0/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:19.847 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:27:19.847 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:27:19.847 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0252555 s, 41.5 MB/s 2021-02-10T11:27:19.848 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000348649 2021-02-10T11:27:20.280 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.273 7f4b8168a700 1 Processor -- start 2021-02-10T11:27:20.280 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.273 7f4b8168a700 1 -- start start 2021-02-10T11:27:20.281 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.273 7f4b8168a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4b7c0f05f0 0x7f4b7c0f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:20.281 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.273 7f4b8168a700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b7c1b0c70 con 0x7f4b7c1b0870 2021-02-10T11:27:20.281 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.273 7f4b8168a700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b7c0f0b50 con 0x7f4b7c1b0470 2021-02-10T11:27:20.281 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.273 7f4b8168a700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b7c0e5ea0 con 0x7f4b7c0f05f0 2021-02-10T11:27:20.281 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.274 7f4b7aabe700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f4b7c1b0870 0x7f4b7c0efe00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:45526/0 (socket says 172.21.2.112:45526) 2021-02-10T11:27:20.281 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.274 7f4b7aabe700 1 -- 172.21.2.112:0/1424999824 learned_addr learned my addr 172.21.2.112:0/1424999824 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:27:20.281 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.274 7f4b7a2bd700 1 -- 172.21.2.112:0/1424999824 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2661950422 0 0) 0x7f4b64001180 con 0x7f4b7c1b0870 2021-02-10T11:27:20.281 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.274 7f4b7a2bd700 1 -- 172.21.2.112:0/1424999824 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b68002fe0 con 0x7f4b7c1b0870 2021-02-10T11:27:20.281 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.274 7f4b7bac0700 1 --2- 172.21.2.112:0/1424999824 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4b7c0f05f0 0x7f4b7c0f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:20.281 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.274 7f4b7a2bd700 1 -- 172.21.2.112:0/1424999824 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3631566303 0 0) 0x7f4b64001180 con 0x7f4b7c1b0870 2021-02-10T11:27:20.281 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.274 7f4b7a2bd700 1 -- 172.21.2.112:0/1424999824 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4b7c0f05f0 msgr2=0x7f4b7c0f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:27:20.281 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.274 7f4b7a2bd700 1 --2- 172.21.2.112:0/1424999824 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4b7c0f05f0 0x7f4b7c0f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:20.281 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.274 7f4b7a2bd700 1 -- 172.21.2.112:0/1424999824 >> v1:172.21.2.120:6789/0 conn(0x7f4b7c1b0470 legacy=0x7f4b7c0ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:20.282 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.274 7f4b7a2bd700 1 -- 172.21.2.112:0/1424999824 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b7c0e52e0 con 0x7f4b7c1b0870 2021-02-10T11:27:20.282 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.274 7f4b7a2bd700 1 -- 172.21.2.112:0/1424999824 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f4b64001680 con 0x7f4b7c1b0870 2021-02-10T11:27:20.282 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.274 7f4b7a2bd700 1 -- 172.21.2.112:0/1424999824 >> v1:172.21.2.112:6789/0 conn(0x7f4b7c1b0870 legacy=0x7f4b7c0efe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:20.282 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.274 7f4b7a2bd700 1 --2- 172.21.2.112:0/1424999824 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4b68003aa0 0x7f4b68005ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:20.282 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.274 7f4b7a2bd700 1 --2- 172.21.2.112:0/1424999824 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4b68006770 0x7f4b68008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:20.282 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.274 7f4b7a2bd700 1 --2- 172.21.2.112:0/1424999824 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4b68009590 0x7f4b6800ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:20.282 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.274 7f4b7a2bd700 1 -- 172.21.2.112:0/1424999824 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b6800c2d0 con 0x7f4b68006770 2021-02-10T11:27:20.282 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.274 7f4b7a2bd700 1 -- 172.21.2.112:0/1424999824 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b6800c4f0 con 0x7f4b68003aa0 2021-02-10T11:27:20.282 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.274 7f4b7a2bd700 1 -- 172.21.2.112:0/1424999824 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b68050720 con 0x7f4b68009590 2021-02-10T11:27:20.282 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.274 7f4b7aabe700 1 --2- 172.21.2.112:0/1424999824 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4b68006770 0x7f4b68008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:20.282 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.275 7f4b7b2bf700 1 --2- 172.21.2.112:0/1424999824 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4b68003aa0 0x7f4b68005ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:20.282 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.275 7f4b7bac0700 1 --2- 172.21.2.112:0/1424999824 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4b68009590 0x7f4b6800ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:20.283 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.275 7f4b7aabe700 1 -- 172.21.2.112:0/1424999824 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4b68009590 msgr2=0x7f4b6800ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:27:20.283 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.275 7f4b7aabe700 1 --2- 172.21.2.112:0/1424999824 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4b68009590 0x7f4b6800ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:20.283 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.275 7f4b7aabe700 1 -- 172.21.2.112:0/1424999824 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4b68003aa0 msgr2=0x7f4b68005ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:27:20.283 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.275 7f4b7aabe700 1 --2- 172.21.2.112:0/1424999824 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4b68003aa0 0x7f4b68005ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:20.283 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.275 7f4b7aabe700 1 -- 172.21.2.112:0/1424999824 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b68050980 con 0x7f4b68006770 2021-02-10T11:27:20.283 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.275 7f4b7bac0700 1 --2- 172.21.2.112:0/1424999824 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4b68009590 0x7f4b6800ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:27:20.283 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.275 7f4b7aabe700 1 --2- 172.21.2.112:0/1424999824 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4b68006770 0x7f4b68008c90 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 rx=0x7f4b64000eb0 tx=0x7f4b64007d00).ready entity=mon.0 client_cookie=ae5ba5094bf45149 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:20.283 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.275 7f4b7a2bd700 1 -- 172.21.2.112:0/1424999824 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4b640070c0 con 0x7f4b68006770 2021-02-10T11:27:20.283 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.275 7f4b7a2bd700 1 -- 172.21.2.112:0/1424999824 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b64001180 con 0x7f4b68006770 2021-02-10T11:27:20.283 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.275 7f4b7a2bd700 1 -- 172.21.2.112:0/1424999824 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4b6400a7a0 con 0x7f4b68006770 2021-02-10T11:27:20.283 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.275 7f4b8168a700 1 -- 172.21.2.112:0/1424999824 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4b68006770 msgr2=0x7f4b68008c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:20.283 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.275 7f4b8168a700 1 --2- 172.21.2.112:0/1424999824 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4b68006770 0x7f4b68008c90 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 rx=0x7f4b64000eb0 tx=0x7f4b64007d00).stop 2021-02-10T11:27:20.283 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.275 7f4b7bac0700 1 -- 172.21.2.112:0/1424999824 reap_dead start 2021-02-10T11:27:20.284 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.276 7f4b8168a700 1 -- 172.21.2.112:0/1424999824 shutdown_connections 2021-02-10T11:27:20.284 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.276 7f4b8168a700 1 -- 172.21.2.112:0/1424999824 >> 172.21.2.112:0/1424999824 conn(0x7f4b7c0ce3e0 msgr2=0x7f4b7c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:20.284 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.276 7f4b8168a700 1 -- 172.21.2.112:0/1424999824 shutdown_connections 2021-02-10T11:27:20.284 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.276 7f4b8168a700 1 -- 172.21.2.112:0/1424999824 wait complete. 2021-02-10T11:27:20.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.291 7f4b8168a700 1 Processor -- start 2021-02-10T11:27:20.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.291 7f4b8168a700 1 -- start start 2021-02-10T11:27:20.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.291 7f4b8168a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4b7c09b930 0x7f4b7c0b7c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:20.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.291 7f4b8168a700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4b7c0b8410 0x7f4b7c0b8810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:20.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.291 7f4b8168a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4b7c0c1f80 0x7f4b7c0c2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:20.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.291 7f4b8168a700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b7c0c1a00 con 0x7f4b7c09b930 2021-02-10T11:27:20.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.291 7f4b8168a700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b7c0b8e30 con 0x7f4b7c0b8410 2021-02-10T11:27:20.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.291 7f4b8168a700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b7c0b9050 con 0x7f4b7c0c1f80 2021-02-10T11:27:20.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.292 7f4b7b2bf700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4b7c09b930 0x7f4b7c0b7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:20.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.292 7f4b7b2bf700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4b7c09b930 0x7f4b7c0b7c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:46076/0 (socket says 172.21.2.112:46076) 2021-02-10T11:27:20.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.292 7f4b7b2bf700 1 -- 172.21.2.112:0/3452624814 learned_addr learned my addr 172.21.2.112:0/3452624814 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:27:20.287 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.292 7f4b7aabe700 1 --2- 172.21.2.112:0/3452624814 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4b7c0b8410 0x7f4b7c0b8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:20.287 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.292 7f4b7bac0700 1 --2- 172.21.2.112:0/3452624814 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4b7c0c1f80 0x7f4b7c0c2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:20.287 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.292 7f4b7aabe700 1 -- 172.21.2.112:0/3452624814 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4b7c0c1f80 msgr2=0x7f4b7c0c2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:20.287 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.292 7f4b7aabe700 1 --2- 172.21.2.112:0/3452624814 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4b7c0c1f80 0x7f4b7c0c2380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:20.287 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.292 7f4b7aabe700 1 -- 172.21.2.112:0/3452624814 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4b7c09b930 msgr2=0x7f4b7c0b7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:20.287 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.292 7f4b7aabe700 1 --2- 172.21.2.112:0/3452624814 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4b7c09b930 0x7f4b7c0b7c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:20.287 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.292 7f4b7aabe700 1 -- 172.21.2.112:0/3452624814 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b7c0c29a0 con 0x7f4b7c0b8410 2021-02-10T11:27:20.287 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.292 7f4b7b2bf700 1 --2- 172.21.2.112:0/3452624814 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4b7c09b930 0x7f4b7c0b7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:27:20.287 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.292 7f4b7bac0700 1 --2- 172.21.2.112:0/3452624814 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4b7c0c1f80 0x7f4b7c0c2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:27:20.287 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.292 7f4b7aabe700 1 --2- 172.21.2.112:0/3452624814 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4b7c0b8410 0x7f4b7c0b8810 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7f4b6400a4a0 tx=0x7f4b6400a840).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:20.287 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.292 7f4b73fff700 1 -- 172.21.2.112:0/3452624814 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4b6400ac30 con 0x7f4b7c0b8410 2021-02-10T11:27:20.287 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.292 7f4b73fff700 1 -- 172.21.2.112:0/3452624814 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b64001bb0 con 0x7f4b7c0b8410 2021-02-10T11:27:20.287 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.292 7f4b8168a700 1 -- 172.21.2.112:0/3452624814 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b7c0c29a0 con 0x7f4b7c0b8410 2021-02-10T11:27:20.287 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.292 7f4b73fff700 1 -- 172.21.2.112:0/3452624814 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4b64007220 con 0x7f4b7c0b8410 2021-02-10T11:27:20.288 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.293 7f4b8168a700 1 -- 172.21.2.112:0/3452624814 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b7c0c29a0 con 0x7f4b7c0b8410 2021-02-10T11:27:20.288 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.293 7f4b73fff700 1 -- 172.21.2.112:0/3452624814 <== mon.1 v2:172.21.2.120:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4b640043a0 con 0x7f4b7c0b8410 2021-02-10T11:27:20.288 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.293 7f4b73fff700 1 --2- 172.21.2.112:0/3452624814 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4b600186c0 0x7f4b6001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:20.288 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.293 7f4b73fff700 1 -- 172.21.2.112:0/3452624814 <== mon.1 v2:172.21.2.120:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f4b640322e0 con 0x7f4b7c0b8410 2021-02-10T11:27:20.288 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.293 7f4b7b2bf700 1 --2- 172.21.2.112:0/3452624814 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4b600186c0 0x7f4b6001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:20.288 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.294 7f4b7b2bf700 1 --2- 172.21.2.112:0/3452624814 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4b600186c0 0x7f4b6001abc0 crc :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:20.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.323 7f4b8168a700 1 -- 172.21.2.112:0/3452624814 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b7c105b30 con 0x7f4b7c0b8410 2021-02-10T11:27:20.316 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.326 7f4b73fff700 1 -- 172.21.2.112:0/3452624814 <== mon.1 v2:172.21.2.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4b64025f30 con 0x7f4b7c0b8410 2021-02-10T11:27:20.442 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.452 7f4b8168a700 1 -- 172.21.2.112:0/3452624814 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348649", "target": ["mgr", ""]}) v1 -- 0x7f4b7c105b30 con 0x7f4b600186c0 2021-02-10T11:27:20.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.476 7f4b73fff700 1 -- 172.21.2.112:0/3452624814 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f4b6c000960 con 0x7f4b600186c0 2021-02-10T11:27:20.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.485 7f4b8168a700 1 -- 172.21.2.112:0/3452624814 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4b600186c0 msgr2=0x7f4b6001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:20.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.485 7f4b8168a700 1 --2- 172.21.2.112:0/3452624814 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4b600186c0 0x7f4b6001abc0 crc :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:20.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.485 7f4b8168a700 1 -- 172.21.2.112:0/3452624814 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4b7c0b8410 msgr2=0x7f4b7c0b8810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:20.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.485 7f4b8168a700 1 --2- 172.21.2.112:0/3452624814 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4b7c0b8410 0x7f4b7c0b8810 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7f4b6400a4a0 tx=0x7f4b6400a840).stop 2021-02-10T11:27:20.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.485 7f4b8168a700 1 -- 172.21.2.112:0/3452624814 shutdown_connections 2021-02-10T11:27:20.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.485 7f4b8168a700 1 --2- 172.21.2.112:0/3452624814 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4b600186c0 0x7f4b6001abc0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:20.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.485 7f4b8168a700 1 --2- 172.21.2.112:0/3452624814 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4b7c0b8410 0x7f4b7c0b8810 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:20.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.485 7f4b8168a700 1 --2- 172.21.2.112:0/3452624814 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4b7c0c1f80 0x7f4b7c0c2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:20.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.485 7f4b8168a700 1 --2- 172.21.2.112:0/3452624814 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4b7c09b930 0x7f4b7c0b7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:20.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.485 7f4b8168a700 1 -- 172.21.2.112:0/3452624814 >> 172.21.2.112:0/3452624814 conn(0x7f4b7c094e10 msgr2=0x7f4b7c0a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:20.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.486 7f4b8168a700 1 -- 172.21.2.112:0/3452624814 shutdown_connections 2021-02-10T11:27:20.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.486 7f4b8168a700 1 -- 172.21.2.112:0/3452624814 wait complete. 2021-02-10T11:27:20.483 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000348649 2021-02-10T11:27:20.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.909 7f521da7f700 1 Processor -- start 2021-02-10T11:27:20.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.909 7f521da7f700 1 -- start start 2021-02-10T11:27:20.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.909 7f521da7f700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f52181b0470 0x7f52181b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:20.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.909 7f521da7f700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f52180f20e0 0x7f52180f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:20.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.909 7f521da7f700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52180f0b50 con 0x7f52180ef610 2021-02-10T11:27:20.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.909 7f521da7f700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52180f0820 con 0x7f52180f20e0 2021-02-10T11:27:20.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.909 7f521da7f700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52180f24e0 con 0x7f52181b0470 2021-02-10T11:27:20.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.909 7f5217fff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f52180f20e0 0x7f52180f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:20.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.910 7f5217fff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f52180f20e0 0x7f52180f0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:46082/0 (socket says 172.21.2.112:46082) 2021-02-10T11:27:20.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.910 7f5217fff700 1 -- 172.21.2.112:0/3396491135 learned_addr learned my addr 172.21.2.112:0/3396491135 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:27:20.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.910 7f5216ffd700 1 --1- 172.21.2.112:0/3396491135 >> v1:172.21.2.112:6789/0 conn(0x7f52180ef610 0x7f52180efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:45544/0 (socket says 172.21.2.112:45544) 2021-02-10T11:27:20.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.910 7f5217fff700 1 -- 172.21.2.112:0/3396491135 >> v2:172.21.2.120:3300/0 conn(0x7f52181b0470 msgr2=0x7f52181b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:27:20.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.910 7f5217fff700 1 --2- 172.21.2.112:0/3396491135 >> v2:172.21.2.120:3300/0 conn(0x7f52181b0470 0x7f52181b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:20.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.910 7f5217fff700 1 -- 172.21.2.112:0/3396491135 >> v1:172.21.2.112:6789/0 conn(0x7f52180ef610 legacy=0x7f52180efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:20.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.910 7f5217fff700 1 -- 172.21.2.112:0/3396491135 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52180e52e0 con 0x7f52180f20e0 2021-02-10T11:27:20.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.910 7f5217fff700 1 --2- 172.21.2.112:0/3396491135 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f52180f20e0 0x7f52180f0200 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 rx=0x7f5210006f00 tx=0x7f5210009f10).ready entity=mon.0 client_cookie=cc76b59ad9423206 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:20.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.910 7f52167fc700 1 -- 172.21.2.112:0/3396491135 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f521000b3b0 con 0x7f52180f20e0 2021-02-10T11:27:20.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.910 7f52167fc700 1 -- 172.21.2.112:0/3396491135 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5210000d30 con 0x7f52180f20e0 2021-02-10T11:27:20.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.910 7f52167fc700 1 -- 172.21.2.112:0/3396491135 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f521000b820 con 0x7f52180f20e0 2021-02-10T11:27:20.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.910 7f521da7f700 1 -- 172.21.2.112:0/3396491135 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f52180f20e0 msgr2=0x7f52180f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:20.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.910 7f521da7f700 1 --2- 172.21.2.112:0/3396491135 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f52180f20e0 0x7f52180f0200 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 rx=0x7f5210006f00 tx=0x7f5210009f10).stop 2021-02-10T11:27:20.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.911 7f521da7f700 1 -- 172.21.2.112:0/3396491135 shutdown_connections 2021-02-10T11:27:20.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.911 7f521da7f700 1 --2- 172.21.2.112:0/3396491135 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f52180f20e0 0x7f52180f0200 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:20.917 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.911 7f521da7f700 1 --2- 172.21.2.112:0/3396491135 >> v2:172.21.2.120:3300/0 conn(0x7f52181b0470 0x7f52181b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:20.917 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.911 7f521da7f700 1 -- 172.21.2.112:0/3396491135 >> 172.21.2.112:0/3396491135 conn(0x7f52180ce3e0 msgr2=0x7f52180e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:20.917 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.911 7f521da7f700 1 -- 172.21.2.112:0/3396491135 shutdown_connections 2021-02-10T11:27:20.917 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.911 7f521da7f700 1 -- 172.21.2.112:0/3396491135 wait complete. 2021-02-10T11:27:20.917 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.926 7f521da7f700 1 Processor -- start 2021-02-10T11:27:20.918 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.926 7f521da7f700 1 -- start start 2021-02-10T11:27:20.918 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.926 7f521da7f700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f52180b20b0 0x7f52180ac970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:20.918 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.926 7f521da7f700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f52180b24b0 0x7f5218191dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:20.918 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.926 7f521da7f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f52180acf90 0x7f52181923e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:20.918 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.926 7f521da7f700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52180b1b30 con 0x7f52180acf90 2021-02-10T11:27:20.918 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.926 7f521da7f700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52181a04e0 con 0x7f52180b24b0 2021-02-10T11:27:20.918 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.926 7f521da7f700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52181a06c0 con 0x7f52180b20b0 2021-02-10T11:27:20.919 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.926 7f5217fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f52180acf90 0x7f52181923e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:20.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.926 7f5217fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f52180acf90 0x7f52181923e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:46092/0 (socket says 172.21.2.112:46092) 2021-02-10T11:27:20.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.926 7f5217fff700 1 -- 172.21.2.112:0/798494148 learned_addr learned my addr 172.21.2.112:0/798494148 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:27:20.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.927 7f5216ffd700 1 --2- 172.21.2.112:0/798494148 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f52180b24b0 0x7f5218191dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:20.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.927 7f5217fff700 1 -- 172.21.2.112:0/798494148 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f52180b20b0 msgr2=0x7f52180ac970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:20.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.927 7f5217fff700 1 --2- 172.21.2.112:0/798494148 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f52180b20b0 0x7f52180ac970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:20.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.927 7f5217fff700 1 -- 172.21.2.112:0/798494148 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f52180b24b0 msgr2=0x7f5218191dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:20.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.927 7f5217fff700 1 --2- 172.21.2.112:0/798494148 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f52180b24b0 0x7f5218191dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:20.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.927 7f5217fff700 1 -- 172.21.2.112:0/798494148 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5218192be0 con 0x7f52180acf90 2021-02-10T11:27:20.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.927 7f5216ffd700 1 --2- 172.21.2.112:0/798494148 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f52180b24b0 0x7f5218191dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:27:20.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.927 7f5217fff700 1 --2- 172.21.2.112:0/798494148 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f52180acf90 0x7f52181923e0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 rx=0x7f5210009530 tx=0x7f5210002c70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:20.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.927 7f520ffff700 1 -- 172.21.2.112:0/798494148 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5210006ba0 con 0x7f52180acf90 2021-02-10T11:27:20.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.927 7f520ffff700 1 -- 172.21.2.112:0/798494148 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52100058b0 con 0x7f52180acf90 2021-02-10T11:27:20.921 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.927 7f521da7f700 1 -- 172.21.2.112:0/798494148 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5218192be0 con 0x7f52180acf90 2021-02-10T11:27:20.921 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.927 7f520ffff700 1 -- 172.21.2.112:0/798494148 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5210005d40 con 0x7f52180acf90 2021-02-10T11:27:20.921 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.927 7f521da7f700 1 -- 172.21.2.112:0/798494148 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5218193170 con 0x7f52180acf90 2021-02-10T11:27:20.921 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.928 7f520ffff700 1 -- 172.21.2.112:0/798494148 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5210006ba0 con 0x7f52180acf90 2021-02-10T11:27:20.921 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.928 7f520ffff700 1 --2- 172.21.2.112:0/798494148 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f51fc0186c0 0x7f51fc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:20.921 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.928 7f520ffff700 1 -- 172.21.2.112:0/798494148 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f52100327c0 con 0x7f52180acf90 2021-02-10T11:27:20.921 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.928 7f52177fe700 1 --2- 172.21.2.112:0/798494148 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f51fc0186c0 0x7f51fc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:20.921 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.929 7f52177fe700 1 --2- 172.21.2.112:0/798494148 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f51fc0186c0 0x7f51fc01abc0 crc :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:20.949 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.959 7f521da7f700 1 -- 172.21.2.112:0/798494148 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5218105af0 con 0x7f52180acf90 2021-02-10T11:27:20.951 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:20.962 7f520ffff700 1 -- 172.21.2.112:0/798494148 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f521001f8c0 con 0x7f52180acf90 2021-02-10T11:27:21.079 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:21.089 7f521da7f700 1 -- 172.21.2.112:0/798494148 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348649", "target": ["mgr", ""]}) v1 -- 0x7f52180b1b30 con 0x7f51fc0186c0 2021-02-10T11:27:21.082 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:21.094 7f520ffff700 1 -- 172.21.2.112:0/798494148 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f5200006a10 con 0x7f51fc0186c0 2021-02-10T11:27:21.086 INFO:teuthology.orchestra.run.gibba012.stdout:/volumes/_nogroup/subvolume_0000000000348649/e301abf4-ce97-4c41-b227-26eb0f7fad22 2021-02-10T11:27:21.088 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:21.099 7f521da7f700 1 -- 172.21.2.112:0/798494148 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f51fc0186c0 msgr2=0x7f51fc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:21.088 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:21.099 7f521da7f700 1 --2- 172.21.2.112:0/798494148 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f51fc0186c0 0x7f51fc01abc0 crc :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:21.088 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:21.099 7f521da7f700 1 -- 172.21.2.112:0/798494148 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f52180acf90 msgr2=0x7f52181923e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:21.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:21.099 7f521da7f700 1 --2- 172.21.2.112:0/798494148 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f52180acf90 0x7f52181923e0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 rx=0x7f5210009530 tx=0x7f5210002c70).stop 2021-02-10T11:27:21.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:21.099 7f521da7f700 1 -- 172.21.2.112:0/798494148 shutdown_connections 2021-02-10T11:27:21.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:21.099 7f521da7f700 1 --2- 172.21.2.112:0/798494148 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f51fc0186c0 0x7f51fc01abc0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:21.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:21.099 7f521da7f700 1 --2- 172.21.2.112:0/798494148 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f52180b24b0 0x7f5218191dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:21.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:21.099 7f521da7f700 1 --2- 172.21.2.112:0/798494148 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f52180acf90 0x7f52181923e0 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:21.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:21.099 7f521da7f700 1 --2- 172.21.2.112:0/798494148 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f52180b20b0 0x7f52180ac970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:21.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:21.099 7f521da7f700 1 -- 172.21.2.112:0/798494148 >> 172.21.2.112:0/798494148 conn(0x7f52180bf4e0 msgr2=0x7f52180be3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:21.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:21.099 7f521da7f700 1 -- 172.21.2.112:0/798494148 shutdown_connections 2021-02-10T11:27:21.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:21.099 7f521da7f700 1 -- 172.21.2.112:0/798494148 wait complete. 2021-02-10T11:27:21.097 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000348649 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000348649/e301abf4-ce97-4c41-b227-26eb0f7fad22 2021-02-10T11:27:21.097 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:21.097 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:21.097 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:21.433 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:27:21.434 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:21.434 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348649/e301abf4-ce97-4c41-b227-26eb0f7fad22/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:21.815 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:27:21.815 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:27:21.815 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0229247 s, 45.7 MB/s 2021-02-10T11:27:21.816 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:21.817 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:21.817 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:22.152 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:27:22.153 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:22.153 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348649/e301abf4-ce97-4c41-b227-26eb0f7fad22/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:22.529 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:27:22.529 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:27:22.529 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0241702 s, 43.4 MB/s 2021-02-10T11:27:22.530 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:22.531 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:22.531 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:22.865 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:27:22.866 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:22.866 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348649/e301abf4-ce97-4c41-b227-26eb0f7fad22/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:23.241 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:27:23.241 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:27:23.241 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0236285 s, 44.4 MB/s 2021-02-10T11:27:23.242 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:23.242 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:23.242 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:23.577 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:27:23.577 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:23.578 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348649/e301abf4-ce97-4c41-b227-26eb0f7fad22/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:23.954 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:27:23.954 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:27:23.954 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0250365 s, 41.9 MB/s 2021-02-10T11:27:23.955 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:23.955 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:23.955 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:24.291 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:27:24.291 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:24.291 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348649/e301abf4-ce97-4c41-b227-26eb0f7fad22/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:24.668 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:27:24.668 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:27:24.668 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0256531 s, 40.9 MB/s 2021-02-10T11:27:24.669 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:24.669 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:24.669 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:25.005 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:27:25.006 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:25.006 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348649/e301abf4-ce97-4c41-b227-26eb0f7fad22/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:25.380 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:27:25.381 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:27:25.381 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0231337 s, 45.3 MB/s 2021-02-10T11:27:25.381 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:25.382 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:25.382 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:25.717 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:27:25.718 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:25.718 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348649/e301abf4-ce97-4c41-b227-26eb0f7fad22/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:26.095 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:27:26.095 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:27:26.095 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0251636 s, 41.7 MB/s 2021-02-10T11:27:26.096 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:26.096 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:26.096 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:26.433 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:27:26.434 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:26.434 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348649/e301abf4-ce97-4c41-b227-26eb0f7fad22/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:26.812 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:27:26.812 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:27:26.812 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0229566 s, 45.7 MB/s 2021-02-10T11:27:26.813 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:26.813 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:26.813 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:27.151 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:27:27.151 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:27.152 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348649/e301abf4-ce97-4c41-b227-26eb0f7fad22/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:27.530 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:27:27.530 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:27:27.530 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0257709 s, 40.7 MB/s 2021-02-10T11:27:27.531 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:27.531 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:27.531 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:27.866 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:27:27.867 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:27.867 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348649/e301abf4-ce97-4c41-b227-26eb0f7fad22/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:28.242 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:27:28.243 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:27:28.243 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0245777 s, 42.7 MB/s 2021-02-10T11:27:28.243 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000348650 2021-02-10T11:27:28.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:28.671 7f32fd0e8700 1 Processor -- start 2021-02-10T11:27:28.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:28.671 7f32fd0e8700 1 -- start start 2021-02-10T11:27:28.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:28.671 7f32fd0e8700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f32f81b0470 0x7f32f81b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:28.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:28.672 7f32fd0e8700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f32f80f20e0 0x7f32f80f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:28.678 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:28.672 7f32fd0e8700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32f80f0b50 con 0x7f32f80ef610 2021-02-10T11:27:28.678 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:28.672 7f32fd0e8700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32f80f0820 con 0x7f32f80f20e0 2021-02-10T11:27:28.678 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:28.672 7f32fd0e8700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32f80f24e0 con 0x7f32f81b0470 2021-02-10T11:27:28.678 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:28.672 7f32f659c700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f32f80ef610 0x7f32f80efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:45560/0 (socket says 172.21.2.112:45560) 2021-02-10T11:27:28.678 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:28.672 7f32f659c700 1 -- 172.21.2.112:0/3680675715 learned_addr learned my addr 172.21.2.112:0/3680675715 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:27:28.678 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:28.672 7f32f6d9d700 1 --2- 172.21.2.112:0/3680675715 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f32f81b0470 0x7f32f81b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:28.678 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:28.672 7f32f759e700 1 --2- 172.21.2.112:0/3680675715 >> v2:172.21.2.120:3300/0 conn(0x7f32f80f20e0 0x7f32f80f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:28.678 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:28.672 7f32f6d9d700 1 -- 172.21.2.112:0/3680675715 >> v2:172.21.2.120:3300/0 conn(0x7f32f80f20e0 msgr2=0x7f32f80f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:27:28.678 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:28.672 7f32f6d9d700 1 --2- 172.21.2.112:0/3680675715 >> v2:172.21.2.120:3300/0 conn(0x7f32f80f20e0 0x7f32f80f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:28.678 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:28.672 7f32f6d9d700 1 -- 172.21.2.112:0/3680675715 >> v1:172.21.2.112:6789/0 conn(0x7f32f80ef610 legacy=0x7f32f80efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:28.678 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:28.672 7f32f6d9d700 1 -- 172.21.2.112:0/3680675715 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32f80e52e0 con 0x7f32f81b0470 2021-02-10T11:27:28.678 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:28.672 7f32f6d9d700 1 --2- 172.21.2.112:0/3680675715 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f32f81b0470 0x7f32f81b0870 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f32e8006650 tx=0x7f32e8009a80).ready entity=mon.2 client_cookie=bbe232c60f60cd25 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:28.678 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:28.673 7f32f5d9b700 1 -- 172.21.2.112:0/3680675715 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f32e800c0c0 con 0x7f32f81b0470 2021-02-10T11:27:28.679 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:28.673 7f32f5d9b700 1 -- 172.21.2.112:0/3680675715 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32e800c340 con 0x7f32f81b0470 2021-02-10T11:27:28.679 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:28.673 7f32f5d9b700 1 -- 172.21.2.112:0/3680675715 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f32e800c7a0 con 0x7f32f81b0470 2021-02-10T11:27:28.679 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:28.673 7f32fd0e8700 1 -- 172.21.2.112:0/3680675715 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f32f81b0470 msgr2=0x7f32f81b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:28.679 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:28.673 7f32fd0e8700 1 --2- 172.21.2.112:0/3680675715 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f32f81b0470 0x7f32f81b0870 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f32e8006650 tx=0x7f32e8009a80).stop 2021-02-10T11:27:28.679 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:28.673 7f32fd0e8700 1 -- 172.21.2.112:0/3680675715 shutdown_connections 2021-02-10T11:27:28.679 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:28.673 7f32fd0e8700 1 --2- 172.21.2.112:0/3680675715 >> v2:172.21.2.120:3300/0 conn(0x7f32f80f20e0 0x7f32f80f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:28.679 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:28.673 7f32fd0e8700 1 --2- 172.21.2.112:0/3680675715 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f32f81b0470 0x7f32f81b0870 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:28.679 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:28.673 7f32fd0e8700 1 -- 172.21.2.112:0/3680675715 >> 172.21.2.112:0/3680675715 conn(0x7f32f80ce3e0 msgr2=0x7f32f80e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:28.679 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:28.674 7f32fd0e8700 1 -- 172.21.2.112:0/3680675715 shutdown_connections 2021-02-10T11:27:28.679 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:28.674 7f32fd0e8700 1 -- 172.21.2.112:0/3680675715 wait complete. 2021-02-10T11:27:28.681 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:28.688 7f32fd0e8700 1 Processor -- start 2021-02-10T11:27:28.681 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:28.688 7f32fd0e8700 1 -- start start 2021-02-10T11:27:28.681 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:28.688 7f32fd0e8700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f32f80b7c20 0x7f32f80b8410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:28.681 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:28.688 7f32fd0e8700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f32f80b8a30 0x7f32f80c1f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:28.681 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:28.689 7f32fd0e8700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f32f80c25a0 0x7f32f80c29a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:28.681 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:28.689 7f32fd0e8700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32f80c1a80 con 0x7f32f80b8a30 2021-02-10T11:27:28.681 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:28.689 7f32fd0e8700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32f80b8020 con 0x7f32f80c25a0 2021-02-10T11:27:28.681 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:28.689 7f32fd0e8700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32f809b9c0 con 0x7f32f80b7c20 2021-02-10T11:27:28.681 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:28.689 7f32f659c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f32f80b8a30 0x7f32f80c1f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:28.681 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:28.689 7f32f659c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f32f80b8a30 0x7f32f80c1f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:46104/0 (socket says 172.21.2.112:46104) 2021-02-10T11:27:28.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:28.689 7f32f659c700 1 -- 172.21.2.112:0/1037592629 learned_addr learned my addr 172.21.2.112:0/1037592629 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:27:28.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:28.689 7f32f6d9d700 1 --2- 172.21.2.112:0/1037592629 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f32f80b7c20 0x7f32f80b8410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:28.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:28.689 7f32f759e700 1 --2- 172.21.2.112:0/1037592629 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f32f80c25a0 0x7f32f80c29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:28.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:28.689 7f32f659c700 1 -- 172.21.2.112:0/1037592629 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f32f80b7c20 msgr2=0x7f32f80b8410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:28.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:28.689 7f32f659c700 1 --2- 172.21.2.112:0/1037592629 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f32f80b7c20 0x7f32f80b8410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:28.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:28.689 7f32f659c700 1 -- 172.21.2.112:0/1037592629 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f32f80c25a0 msgr2=0x7f32f80c29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:28.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:28.689 7f32f659c700 1 --2- 172.21.2.112:0/1037592629 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f32f80c25a0 0x7f32f80c29a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:28.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:28.689 7f32f659c700 1 -- 172.21.2.112:0/1037592629 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32f80b8e30 con 0x7f32f80b8a30 2021-02-10T11:27:28.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:28.689 7f32f659c700 1 --2- 172.21.2.112:0/1037592629 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f32f80b8a30 0x7f32f80c1f80 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 rx=0x7f32ec001c10 tx=0x7f32ec00a870).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:28.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:28.689 7f32e77fe700 1 -- 172.21.2.112:0/1037592629 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f32ec0012c0 con 0x7f32f80b8a30 2021-02-10T11:27:28.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:28.690 7f32e77fe700 1 -- 172.21.2.112:0/1037592629 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32ec00ac60 con 0x7f32f80b8a30 2021-02-10T11:27:28.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:28.690 7f32fd0e8700 1 -- 172.21.2.112:0/1037592629 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32f80b8e30 con 0x7f32f80b8a30 2021-02-10T11:27:28.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:28.690 7f32e77fe700 1 -- 172.21.2.112:0/1037592629 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f32ec0045d0 con 0x7f32f80b8a30 2021-02-10T11:27:28.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:28.690 7f32fd0e8700 1 -- 172.21.2.112:0/1037592629 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32f80c32b0 con 0x7f32f80b8a30 2021-02-10T11:27:28.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:28.690 7f32e77fe700 1 -- 172.21.2.112:0/1037592629 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f32ec007fd0 con 0x7f32f80b8a30 2021-02-10T11:27:28.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:28.690 7f32e77fe700 1 --2- 172.21.2.112:0/1037592629 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f32dc0186c0 0x7f32dc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:28.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:28.690 7f32e77fe700 1 -- 172.21.2.112:0/1037592629 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f32ec032410 con 0x7f32f80b8a30 2021-02-10T11:27:28.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:28.691 7f32f6d9d700 1 --2- 172.21.2.112:0/1037592629 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f32dc0186c0 0x7f32dc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:28.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:28.691 7f32f6d9d700 1 --2- 172.21.2.112:0/1037592629 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f32dc0186c0 0x7f32dc01abc0 crc :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:28.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:28.721 7f32fd0e8700 1 -- 172.21.2.112:0/1037592629 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32f8105b30 con 0x7f32f80b8a30 2021-02-10T11:27:28.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:28.724 7f32e77fe700 1 -- 172.21.2.112:0/1037592629 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f32ec000dd0 con 0x7f32f80b8a30 2021-02-10T11:27:28.839 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:28.850 7f32fd0e8700 1 -- 172.21.2.112:0/1037592629 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348650", "target": ["mgr", ""]}) v1 -- 0x7f32f80b8020 con 0x7f32dc0186c0 2021-02-10T11:27:28.865 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:28.876 7f32e77fe700 1 -- 172.21.2.112:0/1037592629 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f32e8009180 con 0x7f32dc0186c0 2021-02-10T11:27:28.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:28.883 7f32fd0e8700 1 -- 172.21.2.112:0/1037592629 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f32dc0186c0 msgr2=0x7f32dc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:28.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:28.883 7f32fd0e8700 1 --2- 172.21.2.112:0/1037592629 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f32dc0186c0 0x7f32dc01abc0 crc :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:28.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:28.883 7f32fd0e8700 1 -- 172.21.2.112:0/1037592629 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f32f80b8a30 msgr2=0x7f32f80c1f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:28.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:28.883 7f32fd0e8700 1 --2- 172.21.2.112:0/1037592629 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f32f80b8a30 0x7f32f80c1f80 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 rx=0x7f32ec001c10 tx=0x7f32ec00a870).stop 2021-02-10T11:27:28.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:28.883 7f32fd0e8700 1 -- 172.21.2.112:0/1037592629 shutdown_connections 2021-02-10T11:27:28.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:28.883 7f32fd0e8700 1 --2- 172.21.2.112:0/1037592629 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f32dc0186c0 0x7f32dc01abc0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:28.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:28.883 7f32fd0e8700 1 --2- 172.21.2.112:0/1037592629 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f32f80c25a0 0x7f32f80c29a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:28.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:28.883 7f32fd0e8700 1 --2- 172.21.2.112:0/1037592629 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f32f80b8a30 0x7f32f80c1f80 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:28.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:28.883 7f32fd0e8700 1 --2- 172.21.2.112:0/1037592629 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f32f80b7c20 0x7f32f80b8410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:28.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:28.883 7f32fd0e8700 1 -- 172.21.2.112:0/1037592629 >> 172.21.2.112:0/1037592629 conn(0x7f32f8094e10 msgr2=0x7f32f80a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:28.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:28.883 7f32fd0e8700 1 -- 172.21.2.112:0/1037592629 shutdown_connections 2021-02-10T11:27:28.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:28.883 7f32fd0e8700 1 -- 172.21.2.112:0/1037592629 wait complete. 2021-02-10T11:27:28.881 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000348650 2021-02-10T11:27:29.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:29.307 7f5f7e560700 1 Processor -- start 2021-02-10T11:27:29.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:29.307 7f5f7e560700 1 -- start start 2021-02-10T11:27:29.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:29.307 7f5f7e560700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5f781b0870 0x7f5f780f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:29.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:29.307 7f5f7e560700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f5f780efe00 0x7f5f780f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:29.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:29.307 7f5f7e560700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f781b0c70 con 0x7f5f781b0470 2021-02-10T11:27:29.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:29.307 7f5f7e560700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f780f0b50 con 0x7f5f780efe00 2021-02-10T11:27:29.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:29.307 7f5f7e560700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f780f0820 con 0x7f5f781b0870 2021-02-10T11:27:29.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:29.307 7f5f77fff700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f5f781b0470 0x7f5f780ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:45572/0 (socket says 172.21.2.112:45572) 2021-02-10T11:27:29.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:29.307 7f5f77fff700 1 -- 172.21.2.112:0/2110217055 learned_addr learned my addr 172.21.2.112:0/2110217055 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:27:29.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:29.308 7f5f7cb7d700 1 --2- 172.21.2.112:0/2110217055 >> v2:172.21.2.120:3300/0 conn(0x7f5f780efe00 0x7f5f780f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:29.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:29.308 7f5f777fe700 1 --2- 172.21.2.112:0/2110217055 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5f781b0870 0x7f5f780f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:29.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:29.308 7f5f76ffd700 1 -- 172.21.2.112:0/2110217055 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1521496297 0 0) 0x7f5f68000ce0 con 0x7f5f781b0470 2021-02-10T11:27:29.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:29.308 7f5f76ffd700 1 -- 172.21.2.112:0/2110217055 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f60002fe0 con 0x7f5f781b0470 2021-02-10T11:27:29.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:29.308 7f5f76ffd700 1 -- 172.21.2.112:0/2110217055 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1889162687 0 0) 0x7f5f680010e0 con 0x7f5f781b0470 2021-02-10T11:27:29.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:29.308 7f5f76ffd700 1 -- 172.21.2.112:0/2110217055 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5f781b0870 msgr2=0x7f5f780f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:27:29.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:29.308 7f5f76ffd700 1 --2- 172.21.2.112:0/2110217055 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5f781b0870 0x7f5f780f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:29.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:29.308 7f5f76ffd700 1 -- 172.21.2.112:0/2110217055 >> v2:172.21.2.120:3300/0 conn(0x7f5f780efe00 msgr2=0x7f5f780f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:27:29.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:29.308 7f5f76ffd700 1 --2- 172.21.2.112:0/2110217055 >> v2:172.21.2.120:3300/0 conn(0x7f5f780efe00 0x7f5f780f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:29.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:29.308 7f5f76ffd700 1 -- 172.21.2.112:0/2110217055 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f780e52e0 con 0x7f5f781b0470 2021-02-10T11:27:29.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:29.308 7f5f76ffd700 1 -- 172.21.2.112:0/2110217055 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f5f68001600 con 0x7f5f781b0470 2021-02-10T11:27:29.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:29.308 7f5f76ffd700 1 -- 172.21.2.112:0/2110217055 >> v1:172.21.2.112:6789/0 conn(0x7f5f781b0470 legacy=0x7f5f780ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:29.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:29.308 7f5f76ffd700 1 --2- 172.21.2.112:0/2110217055 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5f60003a60 0x7f5f60005ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:29.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:29.308 7f5f76ffd700 1 --2- 172.21.2.112:0/2110217055 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5f6004a780 0x7f5f6004cc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:29.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:29.308 7f5f76ffd700 1 --2- 172.21.2.112:0/2110217055 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5f6004d510 0x7f5f6004fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:29.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:29.308 7f5f76ffd700 1 -- 172.21.2.112:0/2110217055 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f600502e0 con 0x7f5f60003a60 2021-02-10T11:27:29.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:29.308 7f5f76ffd700 1 -- 172.21.2.112:0/2110217055 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f60050500 con 0x7f5f6004d510 2021-02-10T11:27:29.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:29.308 7f5f76ffd700 1 -- 172.21.2.112:0/2110217055 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f60050720 con 0x7f5f6004a780 2021-02-10T11:27:29.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:29.308 7f5f77fff700 1 --2- 172.21.2.112:0/2110217055 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5f60003a60 0x7f5f60005ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:29.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:29.308 7f5f77fff700 1 -- 172.21.2.112:0/2110217055 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5f6004a780 msgr2=0x7f5f6004cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:27:29.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:29.308 7f5f77fff700 1 --2- 172.21.2.112:0/2110217055 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5f6004a780 0x7f5f6004cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:29.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:29.309 7f5f77fff700 1 -- 172.21.2.112:0/2110217055 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5f6004d510 msgr2=0x7f5f6004fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:27:29.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:29.309 7f5f77fff700 1 --2- 172.21.2.112:0/2110217055 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5f6004d510 0x7f5f6004fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:29.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:29.309 7f5f77fff700 1 -- 172.21.2.112:0/2110217055 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f600509c0 con 0x7f5f60003a60 2021-02-10T11:27:29.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:29.309 7f5f77fff700 1 --2- 172.21.2.112:0/2110217055 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5f60003a60 0x7f5f60005ed0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 rx=0x7f5f68007970 tx=0x7f5f68007d10).ready entity=mon.0 client_cookie=4160e63a7bd3c612 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:29.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:29.309 7f5f76ffd700 1 -- 172.21.2.112:0/2110217055 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5f680010e0 con 0x7f5f60003a60 2021-02-10T11:27:29.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:29.309 7f5f76ffd700 1 -- 172.21.2.112:0/2110217055 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f6800a520 con 0x7f5f60003a60 2021-02-10T11:27:29.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:29.309 7f5f76ffd700 1 -- 172.21.2.112:0/2110217055 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5f680070c0 con 0x7f5f60003a60 2021-02-10T11:27:29.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:29.309 7f5f7e560700 1 -- 172.21.2.112:0/2110217055 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5f60003a60 msgr2=0x7f5f60005ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:29.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:29.309 7f5f7e560700 1 --2- 172.21.2.112:0/2110217055 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5f60003a60 0x7f5f60005ed0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 rx=0x7f5f68007970 tx=0x7f5f68007d10).stop 2021-02-10T11:27:29.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:29.309 7f5f7cb7d700 1 -- 172.21.2.112:0/2110217055 reap_dead start 2021-02-10T11:27:29.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:29.309 7f5f7e560700 1 -- 172.21.2.112:0/2110217055 shutdown_connections 2021-02-10T11:27:29.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:29.309 7f5f7e560700 1 -- 172.21.2.112:0/2110217055 >> 172.21.2.112:0/2110217055 conn(0x7f5f780ce3e0 msgr2=0x7f5f780e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:29.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:29.309 7f5f7e560700 1 -- 172.21.2.112:0/2110217055 shutdown_connections 2021-02-10T11:27:29.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:29.309 7f5f7e560700 1 -- 172.21.2.112:0/2110217055 wait complete. 2021-02-10T11:27:29.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:29.325 7f5f7e560700 1 Processor -- start 2021-02-10T11:27:29.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:29.325 7f5f7e560700 1 -- start start 2021-02-10T11:27:29.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:29.325 7f5f7e560700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5f7809b930 0x7f5f780b7c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:29.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:29.325 7f5f7e560700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5f780b8410 0x7f5f780b8810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:29.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:29.325 7f5f7e560700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5f780c1f80 0x7f5f780c2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:29.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:29.325 7f5f7e560700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f780c1a00 con 0x7f5f780b8410 2021-02-10T11:27:29.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:29.325 7f5f7e560700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f780b8e30 con 0x7f5f780c1f80 2021-02-10T11:27:29.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:29.325 7f5f7e560700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f780b9050 con 0x7f5f7809b930 2021-02-10T11:27:29.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:29.326 7f5f777fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5f780b8410 0x7f5f780b8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:29.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:29.326 7f5f777fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5f780b8410 0x7f5f780b8810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:46124/0 (socket says 172.21.2.112:46124) 2021-02-10T11:27:29.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:29.326 7f5f777fe700 1 -- 172.21.2.112:0/3173794414 learned_addr learned my addr 172.21.2.112:0/3173794414 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:27:29.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:29.326 7f5f77fff700 1 --2- 172.21.2.112:0/3173794414 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5f7809b930 0x7f5f780b7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:29.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:29.326 7f5f7cb7d700 1 --2- 172.21.2.112:0/3173794414 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5f780c1f80 0x7f5f780c2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:29.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:29.326 7f5f777fe700 1 -- 172.21.2.112:0/3173794414 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5f7809b930 msgr2=0x7f5f780b7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:29.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:29.326 7f5f777fe700 1 --2- 172.21.2.112:0/3173794414 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5f7809b930 0x7f5f780b7c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:29.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:29.326 7f5f777fe700 1 -- 172.21.2.112:0/3173794414 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5f780c1f80 msgr2=0x7f5f780c2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:29.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:29.326 7f5f777fe700 1 --2- 172.21.2.112:0/3173794414 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5f780c1f80 0x7f5f780c2380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:29.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:29.326 7f5f777fe700 1 -- 172.21.2.112:0/3173794414 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f780c29a0 con 0x7f5f780b8410 2021-02-10T11:27:29.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:29.326 7f5f777fe700 1 --2- 172.21.2.112:0/3173794414 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5f780b8410 0x7f5f780b8810 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 rx=0x7f5f6c00b590 tx=0x7f5f6c00b930).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:29.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:29.326 7f5f74ff9700 1 -- 172.21.2.112:0/3173794414 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5f6c006bb0 con 0x7f5f780b8410 2021-02-10T11:27:29.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:29.326 7f5f74ff9700 1 -- 172.21.2.112:0/3173794414 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f6c00bda0 con 0x7f5f780b8410 2021-02-10T11:27:29.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:29.326 7f5f74ff9700 1 -- 172.21.2.112:0/3173794414 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5f6c006220 con 0x7f5f780b8410 2021-02-10T11:27:29.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:29.326 7f5f7e560700 1 -- 172.21.2.112:0/3173794414 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f780c29a0 con 0x7f5f780b8410 2021-02-10T11:27:29.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:29.326 7f5f7e560700 1 -- 172.21.2.112:0/3173794414 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f780c2f30 con 0x7f5f780b8410 2021-02-10T11:27:29.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:29.327 7f5f74ff9700 1 -- 172.21.2.112:0/3173794414 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5f6c006030 con 0x7f5f780b8410 2021-02-10T11:27:29.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:29.327 7f5f74ff9700 1 --2- 172.21.2.112:0/3173794414 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5f58018730 0x7f5f5801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:29.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:29.327 7f5f74ff9700 1 -- 172.21.2.112:0/3173794414 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f5f6c032640 con 0x7f5f780b8410 2021-02-10T11:27:29.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:29.327 7f5f77fff700 1 --2- 172.21.2.112:0/3173794414 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5f58018730 0x7f5f5801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:29.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:29.328 7f5f77fff700 1 --2- 172.21.2.112:0/3173794414 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5f58018730 0x7f5f5801ac10 crc :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:29.347 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:29.357 7f5f7e560700 1 -- 172.21.2.112:0/3173794414 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f78105b30 con 0x7f5f780b8410 2021-02-10T11:27:29.349 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:29.360 7f5f74ff9700 1 -- 172.21.2.112:0/3173794414 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5f6c006bb0 con 0x7f5f780b8410 2021-02-10T11:27:29.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:29.489 7f5f7e560700 1 -- 172.21.2.112:0/3173794414 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348650", "target": ["mgr", ""]}) v1 -- 0x7f5f780c2f30 con 0x7f5f58018730 2021-02-10T11:27:29.483 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:29.494 7f5f74ff9700 1 -- 172.21.2.112:0/3173794414 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f5f68000f90 con 0x7f5f58018730 2021-02-10T11:27:29.486 INFO:teuthology.orchestra.run.gibba012.stdout:/volumes/_nogroup/subvolume_0000000000348650/8d6765ac-4236-4b96-9305-8135614d4afd 2021-02-10T11:27:29.488 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:29.498 7f5f7e560700 1 -- 172.21.2.112:0/3173794414 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5f58018730 msgr2=0x7f5f5801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:29.488 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:29.498 7f5f7e560700 1 --2- 172.21.2.112:0/3173794414 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5f58018730 0x7f5f5801ac10 crc :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:29.488 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:29.498 7f5f7e560700 1 -- 172.21.2.112:0/3173794414 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5f780b8410 msgr2=0x7f5f780b8810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:29.488 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:29.498 7f5f7e560700 1 --2- 172.21.2.112:0/3173794414 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5f780b8410 0x7f5f780b8810 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 rx=0x7f5f6c00b590 tx=0x7f5f6c00b930).stop 2021-02-10T11:27:29.488 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:29.499 7f5f7e560700 1 -- 172.21.2.112:0/3173794414 shutdown_connections 2021-02-10T11:27:29.488 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:29.499 7f5f7e560700 1 --2- 172.21.2.112:0/3173794414 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5f58018730 0x7f5f5801ac10 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:29.488 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:29.499 7f5f7e560700 1 --2- 172.21.2.112:0/3173794414 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5f780c1f80 0x7f5f780c2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:29.488 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:29.499 7f5f7e560700 1 --2- 172.21.2.112:0/3173794414 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5f780b8410 0x7f5f780b8810 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:29.488 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:29.499 7f5f7e560700 1 --2- 172.21.2.112:0/3173794414 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5f7809b930 0x7f5f780b7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:29.489 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:29.499 7f5f7e560700 1 -- 172.21.2.112:0/3173794414 >> 172.21.2.112:0/3173794414 conn(0x7f5f78094e10 msgr2=0x7f5f780a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:29.489 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:29.499 7f5f7e560700 1 -- 172.21.2.112:0/3173794414 shutdown_connections 2021-02-10T11:27:29.489 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:29.499 7f5f7e560700 1 -- 172.21.2.112:0/3173794414 wait complete. 2021-02-10T11:27:29.496 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000348650 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000348650/8d6765ac-4236-4b96-9305-8135614d4afd 2021-02-10T11:27:29.496 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:29.496 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:29.496 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:29.832 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:27:29.833 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:29.833 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348650/8d6765ac-4236-4b96-9305-8135614d4afd/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:30.214 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:27:30.214 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:27:30.214 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0255434 s, 41.1 MB/s 2021-02-10T11:27:30.215 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:30.215 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:30.216 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:30.551 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:27:30.552 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:30.552 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348650/8d6765ac-4236-4b96-9305-8135614d4afd/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:30.929 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:27:30.930 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:27:30.930 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0266476 s, 39.3 MB/s 2021-02-10T11:27:30.931 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:30.931 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:30.931 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:31.266 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:27:31.267 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:31.267 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348650/8d6765ac-4236-4b96-9305-8135614d4afd/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:31.645 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:27:31.645 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:27:31.645 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0249604 s, 42.0 MB/s 2021-02-10T11:27:31.646 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:31.646 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:31.646 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:31.982 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:27:32.052 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:32.052 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348650/8d6765ac-4236-4b96-9305-8135614d4afd/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:32.428 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:27:32.428 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:27:32.428 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0230262 s, 45.5 MB/s 2021-02-10T11:27:32.430 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:32.430 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:32.430 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:32.766 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:27:32.767 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:32.767 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348650/8d6765ac-4236-4b96-9305-8135614d4afd/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:33.143 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:27:33.143 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:27:33.143 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0231272 s, 45.3 MB/s 2021-02-10T11:27:33.144 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:33.144 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:33.144 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:33.480 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:27:33.480 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:33.481 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348650/8d6765ac-4236-4b96-9305-8135614d4afd/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:33.857 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:27:33.857 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:27:33.857 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0232386 s, 45.1 MB/s 2021-02-10T11:27:33.858 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:33.858 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:33.858 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:34.194 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:27:34.195 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:34.195 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348650/8d6765ac-4236-4b96-9305-8135614d4afd/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:34.569 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:27:34.569 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:27:34.569 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0243209 s, 43.1 MB/s 2021-02-10T11:27:34.570 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:34.570 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:34.570 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:34.906 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:27:34.906 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:34.907 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348650/8d6765ac-4236-4b96-9305-8135614d4afd/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:35.282 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:27:35.282 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:27:35.282 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0229366 s, 45.7 MB/s 2021-02-10T11:27:35.283 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:35.283 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:35.283 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:35.618 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:27:35.618 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:35.619 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348650/8d6765ac-4236-4b96-9305-8135614d4afd/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:35.996 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:27:35.996 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:27:35.996 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.024019 s, 43.7 MB/s 2021-02-10T11:27:35.998 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:35.998 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:35.998 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:36.334 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:27:36.335 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:36.335 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348650/8d6765ac-4236-4b96-9305-8135614d4afd/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:36.711 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:27:36.711 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:27:36.711 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0246502 s, 42.5 MB/s 2021-02-10T11:27:36.712 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000348651 2021-02-10T11:27:37.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.138 7f0b4f18f700 1 Processor -- start 2021-02-10T11:27:37.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.138 7f0b4f18f700 1 -- start start 2021-02-10T11:27:37.145 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.138 7f0b4f18f700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0b481b0870 0x7f0b480f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:37.145 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.138 7f0b4f18f700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f0b480efe00 0x7f0b480f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:37.145 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.138 7f0b4f18f700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b481b0c70 con 0x7f0b481b0470 2021-02-10T11:27:37.145 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.138 7f0b4f18f700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b480f0b50 con 0x7f0b480efe00 2021-02-10T11:27:37.145 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.138 7f0b4f18f700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b480f0820 con 0x7f0b481b0870 2021-02-10T11:27:37.145 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.139 7f0b47fff700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0b481b0870 0x7f0b480f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:37.145 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.139 7f0b4d7ac700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f0b480efe00 0x7f0b480f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:37.145 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.139 7f0b4cfab700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7f0b481b0470 0x7f0b480ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:39150/0 (socket says 172.21.2.112:39150) 2021-02-10T11:27:37.145 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.139 7f0b4cfab700 1 -- 172.21.2.112:0/3165461291 learned_addr learned my addr 172.21.2.112:0/3165461291 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:27:37.145 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.139 7f0b47fff700 1 -- 172.21.2.112:0/3165461291 >> v2:172.21.2.120:3300/0 conn(0x7f0b480efe00 msgr2=0x7f0b480f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:27:37.145 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.139 7f0b47fff700 1 --2- 172.21.2.112:0/3165461291 >> v2:172.21.2.120:3300/0 conn(0x7f0b480efe00 0x7f0b480f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:37.145 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.139 7f0b47fff700 1 -- 172.21.2.112:0/3165461291 >> v1:172.21.2.120:6789/0 conn(0x7f0b481b0470 legacy=0x7f0b480ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:37.145 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.139 7f0b47fff700 1 -- 172.21.2.112:0/3165461291 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b480e52e0 con 0x7f0b481b0870 2021-02-10T11:27:37.146 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.139 7f0b4d7ac700 1 --2- 172.21.2.112:0/3165461291 >> v2:172.21.2.120:3300/0 conn(0x7f0b480efe00 0x7f0b480f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:27:37.146 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.140 7f0b47fff700 1 --2- 172.21.2.112:0/3165461291 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0b481b0870 0x7f0b480f20e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f0b380069d0 tx=0x7f0b38009e40).ready entity=mon.2 client_cookie=5a5100b58ec87834 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:37.146 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.140 7f0b477fe700 1 -- 172.21.2.112:0/3165461291 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0b380066c0 con 0x7f0b481b0870 2021-02-10T11:27:37.146 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.140 7f0b477fe700 1 -- 172.21.2.112:0/3165461291 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b3800b510 con 0x7f0b481b0870 2021-02-10T11:27:37.149 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.140 7f0b477fe700 1 -- 172.21.2.112:0/3165461291 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0b3800b9f0 con 0x7f0b481b0870 2021-02-10T11:27:37.149 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.140 7f0b4f18f700 1 -- 172.21.2.112:0/3165461291 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0b481b0870 msgr2=0x7f0b480f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:37.149 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.140 7f0b4f18f700 1 --2- 172.21.2.112:0/3165461291 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0b481b0870 0x7f0b480f20e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f0b380069d0 tx=0x7f0b38009e40).stop 2021-02-10T11:27:37.149 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.141 7f0b4f18f700 1 -- 172.21.2.112:0/3165461291 shutdown_connections 2021-02-10T11:27:37.149 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.141 7f0b4f18f700 1 --2- 172.21.2.112:0/3165461291 >> v2:172.21.2.120:3300/0 conn(0x7f0b480efe00 0x7f0b480f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:37.149 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.141 7f0b4f18f700 1 --2- 172.21.2.112:0/3165461291 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0b481b0870 0x7f0b480f20e0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:37.149 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.141 7f0b4f18f700 1 -- 172.21.2.112:0/3165461291 >> 172.21.2.112:0/3165461291 conn(0x7f0b480ce3e0 msgr2=0x7f0b480e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:37.149 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.141 7f0b4f18f700 1 -- 172.21.2.112:0/3165461291 shutdown_connections 2021-02-10T11:27:37.150 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.141 7f0b4f18f700 1 -- 172.21.2.112:0/3165461291 wait complete. 2021-02-10T11:27:37.150 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.155 7f0b4f18f700 1 Processor -- start 2021-02-10T11:27:37.150 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.155 7f0b4f18f700 1 -- start start 2021-02-10T11:27:37.150 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.156 7f0b4f18f700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0b480a0420 0x7f0b480a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:37.150 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.156 7f0b4f18f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0b48189d00 0x7f0b4818a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:37.150 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.156 7f0b4f18f700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0b4818a720 0x7f0b480b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:37.150 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.156 7f0b4f18f700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b48189780 con 0x7f0b48189d00 2021-02-10T11:27:37.150 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.156 7f0b4f18f700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b4818ab20 con 0x7f0b4818a720 2021-02-10T11:27:37.150 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.156 7f0b4f18f700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b480a1140 con 0x7f0b480a0420 2021-02-10T11:27:37.150 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.156 7f0b47fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0b48189d00 0x7f0b4818a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:37.150 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.156 7f0b47fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0b48189d00 0x7f0b4818a100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:46138/0 (socket says 172.21.2.112:46138) 2021-02-10T11:27:37.150 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.156 7f0b47fff700 1 -- 172.21.2.112:0/3419385996 learned_addr learned my addr 172.21.2.112:0/3419385996 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:27:37.150 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.156 7f0b47fff700 1 -- 172.21.2.112:0/3419385996 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0b480a0420 msgr2=0x7f0b480a0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:37.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.156 7f0b47fff700 1 --2- 172.21.2.112:0/3419385996 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0b480a0420 0x7f0b480a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:37.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.156 7f0b47fff700 1 -- 172.21.2.112:0/3419385996 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0b4818a720 msgr2=0x7f0b480b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:37.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.156 7f0b47fff700 1 --2- 172.21.2.112:0/3419385996 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0b4818a720 0x7f0b480b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:37.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.156 7f0b47fff700 1 -- 172.21.2.112:0/3419385996 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b480b8200 con 0x7f0b48189d00 2021-02-10T11:27:37.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.157 7f0b47fff700 1 --2- 172.21.2.112:0/3419385996 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0b48189d00 0x7f0b4818a100 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 rx=0x7f0b380069d0 tx=0x7f0b3800bd20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:37.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.157 7f0b457fa700 1 -- 172.21.2.112:0/3419385996 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0b38006240 con 0x7f0b48189d00 2021-02-10T11:27:37.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.157 7f0b457fa700 1 -- 172.21.2.112:0/3419385996 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b3800be60 con 0x7f0b48189d00 2021-02-10T11:27:37.152 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.157 7f0b4f18f700 1 -- 172.21.2.112:0/3419385996 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b480b8200 con 0x7f0b48189d00 2021-02-10T11:27:37.152 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.157 7f0b4f18f700 1 -- 172.21.2.112:0/3419385996 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b480b8790 con 0x7f0b48189d00 2021-02-10T11:27:37.152 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.157 7f0b457fa700 1 -- 172.21.2.112:0/3419385996 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0b38005660 con 0x7f0b48189d00 2021-02-10T11:27:37.152 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.158 7f0b457fa700 1 -- 172.21.2.112:0/3419385996 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0b3800a000 con 0x7f0b48189d00 2021-02-10T11:27:37.152 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.158 7f0b457fa700 1 --2- 172.21.2.112:0/3419385996 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0b2c0186d0 0x7f0b2c01abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:37.152 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.158 7f0b457fa700 1 -- 172.21.2.112:0/3419385996 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f0b38032b00 con 0x7f0b48189d00 2021-02-10T11:27:37.152 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.158 7f0b4cfab700 1 --2- 172.21.2.112:0/3419385996 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0b2c0186d0 0x7f0b2c01abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:37.153 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.159 7f0b4cfab700 1 --2- 172.21.2.112:0/3419385996 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0b2c0186d0 0x7f0b2c01abd0 crc :-1 s=READY pgs=341 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:37.178 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.188 7f0b4f18f700 1 -- 172.21.2.112:0/3419385996 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b480a1360 con 0x7f0b48189d00 2021-02-10T11:27:37.180 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.191 7f0b457fa700 1 -- 172.21.2.112:0/3419385996 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0b38000d30 con 0x7f0b48189d00 2021-02-10T11:27:37.307 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.317 7f0b4f18f700 1 -- 172.21.2.112:0/3419385996 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348651", "target": ["mgr", ""]}) v1 -- 0x7f0b480b9020 con 0x7f0b2c0186d0 2021-02-10T11:27:37.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.344 7f0b457fa700 1 -- 172.21.2.112:0/3419385996 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f0b300077b0 con 0x7f0b2c0186d0 2021-02-10T11:27:37.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.350 7f0b4f18f700 1 -- 172.21.2.112:0/3419385996 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0b2c0186d0 msgr2=0x7f0b2c01abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:37.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.350 7f0b4f18f700 1 --2- 172.21.2.112:0/3419385996 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0b2c0186d0 0x7f0b2c01abd0 crc :-1 s=READY pgs=341 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:37.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.350 7f0b4f18f700 1 -- 172.21.2.112:0/3419385996 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0b48189d00 msgr2=0x7f0b4818a100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:37.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.350 7f0b4f18f700 1 --2- 172.21.2.112:0/3419385996 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0b48189d00 0x7f0b4818a100 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 rx=0x7f0b380069d0 tx=0x7f0b3800bd20).stop 2021-02-10T11:27:37.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.351 7f0b4f18f700 1 -- 172.21.2.112:0/3419385996 shutdown_connections 2021-02-10T11:27:37.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.351 7f0b4f18f700 1 --2- 172.21.2.112:0/3419385996 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0b2c0186d0 0x7f0b2c01abd0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:37.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.351 7f0b4f18f700 1 --2- 172.21.2.112:0/3419385996 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0b4818a720 0x7f0b480b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:37.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.351 7f0b4f18f700 1 --2- 172.21.2.112:0/3419385996 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0b48189d00 0x7f0b4818a100 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:37.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.351 7f0b4f18f700 1 --2- 172.21.2.112:0/3419385996 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0b480a0420 0x7f0b480a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:37.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.351 7f0b4f18f700 1 -- 172.21.2.112:0/3419385996 >> 172.21.2.112:0/3419385996 conn(0x7f0b48094e10 msgr2=0x7f0b480a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:37.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.351 7f0b4f18f700 1 -- 172.21.2.112:0/3419385996 shutdown_connections 2021-02-10T11:27:37.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.351 7f0b4f18f700 1 -- 172.21.2.112:0/3419385996 wait complete. 2021-02-10T11:27:37.348 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000348651 2021-02-10T11:27:37.791 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.785 7f057e5f5700 1 Processor -- start 2021-02-10T11:27:37.791 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.785 7f057e5f5700 1 -- start start 2021-02-10T11:27:37.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.785 7f057e5f5700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f05781b0870 0x7f05780f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:37.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.785 7f057e5f5700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05781b0c70 con 0x7f05780efe00 2021-02-10T11:27:37.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.785 7f057e5f5700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05780f0b50 con 0x7f05781b0470 2021-02-10T11:27:37.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.785 7f057e5f5700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05780e5ea0 con 0x7f05781b0870 2021-02-10T11:27:37.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.785 7f05777fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f05781b0870 0x7f05780f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:37.793 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.785 7f057cc12700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f05780efe00 0x7f05780f0200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:45608/0 (socket says 172.21.2.112:45608) 2021-02-10T11:27:37.793 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.785 7f057cc12700 1 -- 172.21.2.112:0/1353697088 learned_addr learned my addr 172.21.2.112:0/1353697088 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:27:37.793 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.786 7f05777fe700 1 -- 172.21.2.112:0/1353697088 >> v1:172.21.2.120:6789/0 conn(0x7f05781b0470 legacy=0x7f05780ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:37.793 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.786 7f05777fe700 1 -- 172.21.2.112:0/1353697088 >> v1:172.21.2.112:6789/0 conn(0x7f05780efe00 legacy=0x7f05780f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:37.793 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.786 7f0576ffd700 1 -- 172.21.2.112:0/1353697088 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2455040469 0 0) 0x7f0570001500 con 0x7f05780efe00 2021-02-10T11:27:37.793 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.786 7f05777fe700 1 -- 172.21.2.112:0/1353697088 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05780d2fb0 con 0x7f05781b0870 2021-02-10T11:27:37.793 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.786 7f05777fe700 1 --2- 172.21.2.112:0/1353697088 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f05781b0870 0x7f05780f20e0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 rx=0x7f056c006650 tx=0x7f056c009a00).ready entity=mon.0 client_cookie=2c9f2d2f5a230f4d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:37.793 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.786 7f0576ffd700 1 -- 172.21.2.112:0/1353697088 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f056c009df0 con 0x7f05781b0870 2021-02-10T11:27:37.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.786 7f0576ffd700 1 -- 172.21.2.112:0/1353697088 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f056c0009b0 con 0x7f05781b0870 2021-02-10T11:27:37.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.786 7f0576ffd700 1 -- 172.21.2.112:0/1353697088 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f056c00c2b0 con 0x7f05781b0870 2021-02-10T11:27:37.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.786 7f057e5f5700 1 -- 172.21.2.112:0/1353697088 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f05781b0870 msgr2=0x7f05780f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:37.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.786 7f057e5f5700 1 --2- 172.21.2.112:0/1353697088 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f05781b0870 0x7f05780f20e0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 rx=0x7f056c006650 tx=0x7f056c009a00).stop 2021-02-10T11:27:37.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.787 7f057e5f5700 1 -- 172.21.2.112:0/1353697088 shutdown_connections 2021-02-10T11:27:37.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.787 7f057e5f5700 1 --2- 172.21.2.112:0/1353697088 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f05781b0870 0x7f05780f20e0 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:37.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.787 7f057e5f5700 1 -- 172.21.2.112:0/1353697088 >> 172.21.2.112:0/1353697088 conn(0x7f05780ce3e0 msgr2=0x7f05780e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:37.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.787 7f057e5f5700 1 -- 172.21.2.112:0/1353697088 shutdown_connections 2021-02-10T11:27:37.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.787 7f057e5f5700 1 -- 172.21.2.112:0/1353697088 wait complete. 2021-02-10T11:27:37.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.802 7f057e5f5700 1 Processor -- start 2021-02-10T11:27:37.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.802 7f057e5f5700 1 -- start start 2021-02-10T11:27:37.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.802 7f057e5f5700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f057809c860 0x7f0578089430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:37.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.802 7f057e5f5700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f05780a8900 0x7f05781c2140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:37.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.802 7f057e5f5700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f05781c2760 0x7f05780a97f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:37.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.802 7f057e5f5700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05780a9270 con 0x7f05780a8900 2021-02-10T11:27:37.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.802 7f057e5f5700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f057809cc60 con 0x7f057809c860 2021-02-10T11:27:37.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.802 7f057e5f5700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05780ab8e0 con 0x7f05781c2760 2021-02-10T11:27:37.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.802 7f05777fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f05780a8900 0x7f05781c2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:37.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.802 7f05777fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f05780a8900 0x7f05781c2140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:46152/0 (socket says 172.21.2.112:46152) 2021-02-10T11:27:37.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.802 7f05777fe700 1 -- 172.21.2.112:0/1438931536 learned_addr learned my addr 172.21.2.112:0/1438931536 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:27:37.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.802 7f057cc12700 1 --2- 172.21.2.112:0/1438931536 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f05781c2760 0x7f05780a97f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:37.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.802 7f0577fff700 1 --2- 172.21.2.112:0/1438931536 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f057809c860 0x7f0578089430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:37.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.803 7f05777fe700 1 -- 172.21.2.112:0/1438931536 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f05781c2760 msgr2=0x7f05780a97f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:37.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.803 7f05777fe700 1 --2- 172.21.2.112:0/1438931536 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f05781c2760 0x7f05780a97f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:37.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.803 7f05777fe700 1 -- 172.21.2.112:0/1438931536 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f057809c860 msgr2=0x7f0578089430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:37.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.803 7f05777fe700 1 --2- 172.21.2.112:0/1438931536 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f057809c860 0x7f0578089430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:37.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.803 7f05777fe700 1 -- 172.21.2.112:0/1438931536 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05780a8d00 con 0x7f05780a8900 2021-02-10T11:27:37.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.803 7f0577fff700 1 --2- 172.21.2.112:0/1438931536 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f057809c860 0x7f0578089430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:27:37.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.803 7f05777fe700 1 --2- 172.21.2.112:0/1438931536 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f05780a8900 0x7f05781c2140 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 rx=0x7f056c0096a0 tx=0x7f056c00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:37.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.803 7f0574ff9700 1 -- 172.21.2.112:0/1438931536 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f056c000c90 con 0x7f05780a8900 2021-02-10T11:27:37.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.803 7f0574ff9700 1 -- 172.21.2.112:0/1438931536 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f056c002030 con 0x7f05780a8900 2021-02-10T11:27:37.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.803 7f0574ff9700 1 -- 172.21.2.112:0/1438931536 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f056c002490 con 0x7f05780a8900 2021-02-10T11:27:37.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.803 7f057e5f5700 1 -- 172.21.2.112:0/1438931536 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05780a8d00 con 0x7f05780a8900 2021-02-10T11:27:37.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.803 7f057e5f5700 1 -- 172.21.2.112:0/1438931536 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05780aa100 con 0x7f05780a8900 2021-02-10T11:27:37.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.804 7f0574ff9700 1 -- 172.21.2.112:0/1438931536 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f056c002210 con 0x7f05780a8900 2021-02-10T11:27:37.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.804 7f0574ff9700 1 --2- 172.21.2.112:0/1438931536 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f05580186c0 0x7f055801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:37.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.804 7f0574ff9700 1 -- 172.21.2.112:0/1438931536 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f056c032400 con 0x7f05780a8900 2021-02-10T11:27:37.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.804 7f0577fff700 1 --2- 172.21.2.112:0/1438931536 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f05580186c0 0x7f055801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:37.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.804 7f0577fff700 1 --2- 172.21.2.112:0/1438931536 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f05580186c0 0x7f055801abc0 crc :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:37.824 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.835 7f057e5f5700 1 -- 172.21.2.112:0/1438931536 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f057807e710 con 0x7f05780a8900 2021-02-10T11:27:37.826 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.837 7f0574ff9700 1 -- 172.21.2.112:0/1438931536 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f056c029050 con 0x7f05780a8900 2021-02-10T11:27:37.955 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.966 7f057e5f5700 1 -- 172.21.2.112:0/1438931536 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348651", "target": ["mgr", ""]}) v1 -- 0x7f05780aa100 con 0x7f05580186c0 2021-02-10T11:27:37.959 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.970 7f0574ff9700 1 -- 172.21.2.112:0/1438931536 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f0568000960 con 0x7f05580186c0 2021-02-10T11:27:37.963 INFO:teuthology.orchestra.run.gibba012.stdout:/volumes/_nogroup/subvolume_0000000000348651/a9ccdeb9-1699-45b0-a275-36c0dda308cd 2021-02-10T11:27:37.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.975 7f057e5f5700 1 -- 172.21.2.112:0/1438931536 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f05580186c0 msgr2=0x7f055801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:37.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.975 7f057e5f5700 1 --2- 172.21.2.112:0/1438931536 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f05580186c0 0x7f055801abc0 crc :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:37.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.975 7f057e5f5700 1 -- 172.21.2.112:0/1438931536 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f05780a8900 msgr2=0x7f05781c2140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:37.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.975 7f057e5f5700 1 --2- 172.21.2.112:0/1438931536 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f05780a8900 0x7f05781c2140 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 rx=0x7f056c0096a0 tx=0x7f056c00b990).stop 2021-02-10T11:27:37.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.975 7f057e5f5700 1 -- 172.21.2.112:0/1438931536 shutdown_connections 2021-02-10T11:27:37.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.975 7f057e5f5700 1 --2- 172.21.2.112:0/1438931536 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f05580186c0 0x7f055801abc0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:37.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.975 7f057e5f5700 1 --2- 172.21.2.112:0/1438931536 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f05781c2760 0x7f05780a97f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:37.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.975 7f057e5f5700 1 --2- 172.21.2.112:0/1438931536 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f05780a8900 0x7f05781c2140 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:37.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.975 7f057e5f5700 1 --2- 172.21.2.112:0/1438931536 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f057809c860 0x7f0578089430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:37.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.976 7f057e5f5700 1 -- 172.21.2.112:0/1438931536 >> 172.21.2.112:0/1438931536 conn(0x7f05780b8b80 msgr2=0x7f057809aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:37.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.976 7f057e5f5700 1 -- 172.21.2.112:0/1438931536 shutdown_connections 2021-02-10T11:27:37.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:37.976 7f057e5f5700 1 -- 172.21.2.112:0/1438931536 wait complete. 2021-02-10T11:27:37.974 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000348651 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000348651/a9ccdeb9-1699-45b0-a275-36c0dda308cd 2021-02-10T11:27:37.974 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:37.974 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:37.974 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:38.311 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:27:38.311 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:38.312 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348651/a9ccdeb9-1699-45b0-a275-36c0dda308cd/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:38.692 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:27:38.693 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:27:38.693 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.023926 s, 43.8 MB/s 2021-02-10T11:27:38.694 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:38.694 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:38.694 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:39.029 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:27:39.030 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:39.030 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348651/a9ccdeb9-1699-45b0-a275-36c0dda308cd/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:39.403 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:27:39.403 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:27:39.403 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.025099 s, 41.8 MB/s 2021-02-10T11:27:39.404 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:39.404 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:39.404 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:39.740 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:27:39.740 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:39.741 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348651/a9ccdeb9-1699-45b0-a275-36c0dda308cd/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:40.117 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:27:40.117 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:27:40.117 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0230085 s, 45.6 MB/s 2021-02-10T11:27:40.118 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:40.118 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:40.118 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:40.454 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:27:40.454 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:40.455 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348651/a9ccdeb9-1699-45b0-a275-36c0dda308cd/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:40.832 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:27:40.832 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:27:40.832 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0250122 s, 41.9 MB/s 2021-02-10T11:27:40.833 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:40.833 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:40.833 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:41.169 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:27:41.169 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:41.170 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348651/a9ccdeb9-1699-45b0-a275-36c0dda308cd/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:41.546 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:27:41.546 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:27:41.547 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0235404 s, 44.5 MB/s 2021-02-10T11:27:41.548 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:41.548 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:41.548 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:41.682 DEBUG:teuthology.orchestra.run.gibba012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:27:41.685 DEBUG:teuthology.orchestra.run.gibba020:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:27:41.883 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:27:41.883 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:41.884 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348651/a9ccdeb9-1699-45b0-a275-36c0dda308cd/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:42.259 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:27:42.259 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:27:42.260 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0253182 s, 41.4 MB/s 2021-02-10T11:27:42.260 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:42.261 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:42.261 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:42.597 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:27:42.597 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:42.597 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348651/a9ccdeb9-1699-45b0-a275-36c0dda308cd/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:42.975 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:27:42.975 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:27:42.975 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0248479 s, 42.2 MB/s 2021-02-10T11:27:42.976 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:42.976 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:42.976 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:43.312 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:27:43.312 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:43.312 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348651/a9ccdeb9-1699-45b0-a275-36c0dda308cd/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:43.687 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:27:43.687 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:27:43.687 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0226394 s, 46.3 MB/s 2021-02-10T11:27:43.688 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:43.688 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:43.688 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:44.024 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:27:44.024 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:44.024 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348651/a9ccdeb9-1699-45b0-a275-36c0dda308cd/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:44.395 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:27:44.396 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:27:44.396 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0241188 s, 43.5 MB/s 2021-02-10T11:27:44.397 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:44.397 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:44.397 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:44.732 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:27:44.732 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:44.732 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348651/a9ccdeb9-1699-45b0-a275-36c0dda308cd/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:45.108 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:27:45.109 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:27:45.109 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0243901 s, 43.0 MB/s 2021-02-10T11:27:45.110 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000348652 2021-02-10T11:27:45.543 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:45.537 7f73974fc700 1 Processor -- start 2021-02-10T11:27:45.543 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:45.537 7f73974fc700 1 -- start start 2021-02-10T11:27:45.543 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:45.537 7f73974fc700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f73900f05f0 0x7f73900f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:45.543 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:45.537 7f73974fc700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73901b0c70 con 0x7f73901b0870 2021-02-10T11:27:45.543 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:45.537 7f73974fc700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73900f0b50 con 0x7f73901b0470 2021-02-10T11:27:45.543 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:45.537 7f73974fc700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73900e5ea0 con 0x7f73900f05f0 2021-02-10T11:27:45.543 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:45.537 7f7395b19700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f73900f05f0 0x7f73900f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:45.543 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:45.537 7f7395318700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7f73901b0470 0x7f73900ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:39176/0 (socket says 172.21.2.112:39176) 2021-02-10T11:27:45.543 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:45.538 7f7394b17700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f73901b0870 0x7f73900efe00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:45618/0 (socket says 172.21.2.112:45618) 2021-02-10T11:27:45.544 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:45.538 7f7395b19700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f73900f05f0 0x7f73900f20e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:46160/0 (socket says 172.21.2.112:46160) 2021-02-10T11:27:45.544 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:45.538 7f7395318700 1 -- 172.21.2.112:0/2595556115 learned_addr learned my addr 172.21.2.112:0/2595556115 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:27:45.544 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:45.538 7f7383fff700 1 -- 172.21.2.112:0/2595556115 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1452692201 0 0) 0x7f7384001440 con 0x7f73901b0470 2021-02-10T11:27:45.544 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:45.538 7f7383fff700 1 -- 172.21.2.112:0/2595556115 --> v1:172.21.2.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f737c002fe0 con 0x7f73901b0470 2021-02-10T11:27:45.544 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:45.538 7f7395b19700 1 -- 172.21.2.112:0/2595556115 >> v1:172.21.2.120:6789/0 conn(0x7f73901b0470 legacy=0x7f73900ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:45.544 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:45.538 7f7395b19700 1 -- 172.21.2.112:0/2595556115 >> v1:172.21.2.112:6789/0 conn(0x7f73901b0870 legacy=0x7f73900efe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:45.544 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:45.538 7f7395b19700 1 -- 172.21.2.112:0/2595556115 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73900e52e0 con 0x7f73900f05f0 2021-02-10T11:27:45.544 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:45.538 7f7395b19700 1 --2- 172.21.2.112:0/2595556115 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f73900f05f0 0x7f73900f20e0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 rx=0x7f7388006ab0 tx=0x7f7388009b40).ready entity=mon.0 client_cookie=d3481c65f963d0b2 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:45.544 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:45.538 7f7383fff700 1 -- 172.21.2.112:0/2595556115 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f73880022e0 con 0x7f73900f05f0 2021-02-10T11:27:45.544 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:45.538 7f7383fff700 1 -- 172.21.2.112:0/2595556115 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7388002660 con 0x7f73900f05f0 2021-02-10T11:27:45.544 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:45.538 7f7383fff700 1 -- 172.21.2.112:0/2595556115 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7388002ac0 con 0x7f73900f05f0 2021-02-10T11:27:45.544 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:45.538 7f73974fc700 1 -- 172.21.2.112:0/2595556115 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f73900f05f0 msgr2=0x7f73900f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:45.544 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:45.538 7f73974fc700 1 --2- 172.21.2.112:0/2595556115 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f73900f05f0 0x7f73900f20e0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 rx=0x7f7388006ab0 tx=0x7f7388009b40).stop 2021-02-10T11:27:45.545 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:45.539 7f73974fc700 1 -- 172.21.2.112:0/2595556115 shutdown_connections 2021-02-10T11:27:45.545 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:45.539 7f73974fc700 1 --2- 172.21.2.112:0/2595556115 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f73900f05f0 0x7f73900f20e0 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:45.545 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:45.539 7f73974fc700 1 -- 172.21.2.112:0/2595556115 >> 172.21.2.112:0/2595556115 conn(0x7f73900ce3e0 msgr2=0x7f73900e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:45.545 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:45.539 7f73974fc700 1 -- 172.21.2.112:0/2595556115 shutdown_connections 2021-02-10T11:27:45.545 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:45.539 7f73974fc700 1 -- 172.21.2.112:0/2595556115 wait complete. 2021-02-10T11:27:45.546 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:45.554 7f73974fc700 1 Processor -- start 2021-02-10T11:27:45.546 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:45.554 7f73974fc700 1 -- start start 2021-02-10T11:27:45.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:45.555 7f73974fc700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f73900b20b0 0x7f73900ac970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:45.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:45.555 7f73974fc700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f73900b24b0 0x7f7390191dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:45.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:45.555 7f73974fc700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f73900acf90 0x7f73901923e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:45.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:45.555 7f73974fc700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73900b1b30 con 0x7f73900acf90 2021-02-10T11:27:45.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:45.555 7f73974fc700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73901a04e0 con 0x7f73900b24b0 2021-02-10T11:27:45.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:45.555 7f73974fc700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73901a06c0 con 0x7f73900b20b0 2021-02-10T11:27:45.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:45.555 7f7395b19700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f73900acf90 0x7f73901923e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:45.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:45.555 7f7395b19700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f73900acf90 0x7f73901923e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:46164/0 (socket says 172.21.2.112:46164) 2021-02-10T11:27:45.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:45.555 7f7395b19700 1 -- 172.21.2.112:0/34857492 learned_addr learned my addr 172.21.2.112:0/34857492 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:27:45.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:45.555 7f7394b17700 1 --2- 172.21.2.112:0/34857492 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f73900b24b0 0x7f7390191dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:45.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:45.555 7f7395318700 1 --2- 172.21.2.112:0/34857492 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f73900b20b0 0x7f73900ac970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:45.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:45.555 7f7395b19700 1 -- 172.21.2.112:0/34857492 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f73900b20b0 msgr2=0x7f73900ac970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:45.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:45.555 7f7395b19700 1 --2- 172.21.2.112:0/34857492 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f73900b20b0 0x7f73900ac970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:45.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:45.555 7f7395b19700 1 -- 172.21.2.112:0/34857492 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f73900b24b0 msgr2=0x7f7390191dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:45.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:45.555 7f7395b19700 1 --2- 172.21.2.112:0/34857492 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f73900b24b0 0x7f7390191dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:45.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:45.555 7f7395b19700 1 -- 172.21.2.112:0/34857492 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7390192be0 con 0x7f73900acf90 2021-02-10T11:27:45.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:45.555 7f7395b19700 1 --2- 172.21.2.112:0/34857492 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f73900acf90 0x7f73901923e0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 rx=0x7f738800b5e0 tx=0x7f738800b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:45.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:45.556 7f7381ffb700 1 -- 172.21.2.112:0/34857492 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7388002280 con 0x7f73900acf90 2021-02-10T11:27:45.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:45.556 7f7381ffb700 1 -- 172.21.2.112:0/34857492 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7388009070 con 0x7f73900acf90 2021-02-10T11:27:45.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:45.556 7f7381ffb700 1 -- 172.21.2.112:0/34857492 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f73880027c0 con 0x7f73900acf90 2021-02-10T11:27:45.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:45.556 7f73974fc700 1 -- 172.21.2.112:0/34857492 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7390192be0 con 0x7f73900acf90 2021-02-10T11:27:45.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:45.556 7f73974fc700 1 -- 172.21.2.112:0/34857492 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7390193170 con 0x7f73900acf90 2021-02-10T11:27:45.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:45.556 7f7381ffb700 1 -- 172.21.2.112:0/34857492 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7388006a20 con 0x7f73900acf90 2021-02-10T11:27:45.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:45.556 7f7381ffb700 1 --2- 172.21.2.112:0/34857492 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f73700186d0 0x7f737001abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:45.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:45.556 7f7381ffb700 1 -- 172.21.2.112:0/34857492 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f73880327c0 con 0x7f73900acf90 2021-02-10T11:27:45.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:45.557 7f7395318700 1 --2- 172.21.2.112:0/34857492 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f73700186d0 0x7f737001abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:45.549 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:45.557 7f7395318700 1 --2- 172.21.2.112:0/34857492 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f73700186d0 0x7f737001abd0 crc :-1 s=READY pgs=343 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:45.577 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:45.588 7f73974fc700 1 -- 172.21.2.112:0/34857492 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7390105af0 con 0x7f73900acf90 2021-02-10T11:27:45.579 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:45.590 7f7381ffb700 1 -- 172.21.2.112:0/34857492 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7388008f60 con 0x7f73900acf90 2021-02-10T11:27:45.708 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:45.719 7f73974fc700 1 -- 172.21.2.112:0/34857492 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348652", "target": ["mgr", ""]}) v1 -- 0x7f73901a04e0 con 0x7f73700186d0 2021-02-10T11:27:45.733 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:45.745 7f7381ffb700 1 -- 172.21.2.112:0/34857492 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f738400c050 con 0x7f73700186d0 2021-02-10T11:27:45.741 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:45.752 7f73974fc700 1 -- 172.21.2.112:0/34857492 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f73700186d0 msgr2=0x7f737001abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:45.741 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:45.752 7f73974fc700 1 --2- 172.21.2.112:0/34857492 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f73700186d0 0x7f737001abd0 crc :-1 s=READY pgs=343 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:45.742 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:45.752 7f73974fc700 1 -- 172.21.2.112:0/34857492 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f73900acf90 msgr2=0x7f73901923e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:45.742 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:45.752 7f73974fc700 1 --2- 172.21.2.112:0/34857492 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f73900acf90 0x7f73901923e0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 rx=0x7f738800b5e0 tx=0x7f738800b980).stop 2021-02-10T11:27:45.742 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:45.753 7f73974fc700 1 -- 172.21.2.112:0/34857492 shutdown_connections 2021-02-10T11:27:45.742 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:45.753 7f73974fc700 1 --2- 172.21.2.112:0/34857492 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f73700186d0 0x7f737001abd0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:45.742 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:45.753 7f73974fc700 1 --2- 172.21.2.112:0/34857492 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f73900b24b0 0x7f7390191dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:45.743 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:45.753 7f73974fc700 1 --2- 172.21.2.112:0/34857492 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f73900acf90 0x7f73901923e0 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:45.743 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:45.753 7f73974fc700 1 --2- 172.21.2.112:0/34857492 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f73900b20b0 0x7f73900ac970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:45.743 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:45.753 7f73974fc700 1 -- 172.21.2.112:0/34857492 >> 172.21.2.112:0/34857492 conn(0x7f73900bf4e0 msgr2=0x7f73900be3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:45.743 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:45.753 7f73974fc700 1 -- 172.21.2.112:0/34857492 shutdown_connections 2021-02-10T11:27:45.743 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:45.753 7f73974fc700 1 -- 172.21.2.112:0/34857492 wait complete. 2021-02-10T11:27:45.750 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000348652 2021-02-10T11:27:46.181 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:46.175 7f1cc92be700 1 Processor -- start 2021-02-10T11:27:46.182 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:46.175 7f1cc92be700 1 -- start start 2021-02-10T11:27:46.182 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:46.175 7f1cc92be700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1cc41b0470 0x7f1cc41b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:46.182 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:46.175 7f1cc92be700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f1cc40ef610 0x7f1cc40efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:46.182 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:46.175 7f1cc92be700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cc40f0b50 con 0x7f1cc40f0030 2021-02-10T11:27:46.182 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:46.175 7f1cc92be700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cc40e52e0 con 0x7f1cc40ef610 2021-02-10T11:27:46.182 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:46.176 7f1cc92be700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cc40e5490 con 0x7f1cc41b0470 2021-02-10T11:27:46.182 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:46.177 7f1cc287c700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f1cc40ef610 0x7f1cc40efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:46.182 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:46.177 7f1cc307d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1cc41b0470 0x7f1cc41b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:46.182 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:46.177 7f1cc387e700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7f1cc40f0030 0x7f1cc40f10d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:39190/0 (socket says 172.21.2.112:39190) 2021-02-10T11:27:46.182 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:46.177 7f1cc387e700 1 -- 172.21.2.112:0/807322001 learned_addr learned my addr 172.21.2.112:0/807322001 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:27:46.182 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:46.177 7f1cc307d700 1 -- 172.21.2.112:0/807322001 >> v2:172.21.2.120:3300/0 conn(0x7f1cc40ef610 msgr2=0x7f1cc40efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:27:46.183 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:46.177 7f1cc307d700 1 --2- 172.21.2.112:0/807322001 >> v2:172.21.2.120:3300/0 conn(0x7f1cc40ef610 0x7f1cc40efa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:46.183 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:46.177 7f1cc307d700 1 -- 172.21.2.112:0/807322001 >> v1:172.21.2.120:6789/0 conn(0x7f1cc40f0030 legacy=0x7f1cc40f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:46.183 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:46.177 7f1cc207b700 1 -- 172.21.2.112:0/807322001 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3839320107 0 0) 0x7f1cbc001510 con 0x7f1cc40f0030 2021-02-10T11:27:46.183 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:46.177 7f1cc307d700 1 -- 172.21.2.112:0/807322001 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cc40f0430 con 0x7f1cc41b0470 2021-02-10T11:27:46.183 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:46.177 7f1cc287c700 1 --2- 172.21.2.112:0/807322001 >> v2:172.21.2.120:3300/0 conn(0x7f1cc40ef610 0x7f1cc40efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:27:46.183 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:46.177 7f1cc307d700 1 --2- 172.21.2.112:0/807322001 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1cc41b0470 0x7f1cc41b0870 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f1cb4006650 tx=0x7f1cb4009a80).ready entity=mon.2 client_cookie=ab03e1d0ebaff606 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:46.183 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:46.177 7f1cc207b700 1 -- 172.21.2.112:0/807322001 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1cb400c0c0 con 0x7f1cc41b0470 2021-02-10T11:27:46.183 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:46.177 7f1cc207b700 1 -- 172.21.2.112:0/807322001 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cb400c340 con 0x7f1cc41b0470 2021-02-10T11:27:46.183 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:46.177 7f1cc207b700 1 -- 172.21.2.112:0/807322001 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1cb400c860 con 0x7f1cc41b0470 2021-02-10T11:27:46.183 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:46.178 7f1cc92be700 1 -- 172.21.2.112:0/807322001 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1cc41b0470 msgr2=0x7f1cc41b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:46.183 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:46.178 7f1cc92be700 1 --2- 172.21.2.112:0/807322001 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1cc41b0470 0x7f1cc41b0870 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f1cb4006650 tx=0x7f1cb4009a80).stop 2021-02-10T11:27:46.183 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:46.178 7f1cc92be700 1 -- 172.21.2.112:0/807322001 shutdown_connections 2021-02-10T11:27:46.183 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:46.178 7f1cc92be700 1 --2- 172.21.2.112:0/807322001 >> v2:172.21.2.120:3300/0 conn(0x7f1cc40ef610 0x7f1cc40efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:46.183 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:46.178 7f1cc92be700 1 --2- 172.21.2.112:0/807322001 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1cc41b0470 0x7f1cc41b0870 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:46.184 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:46.178 7f1cc92be700 1 -- 172.21.2.112:0/807322001 >> 172.21.2.112:0/807322001 conn(0x7f1cc40ce3e0 msgr2=0x7f1cc40e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:46.184 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:46.178 7f1cc92be700 1 -- 172.21.2.112:0/807322001 shutdown_connections 2021-02-10T11:27:46.184 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:46.178 7f1cc92be700 1 -- 172.21.2.112:0/807322001 wait complete. 2021-02-10T11:27:46.185 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:46.193 7f1cc92be700 1 Processor -- start 2021-02-10T11:27:46.186 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:46.193 7f1cc92be700 1 -- start start 2021-02-10T11:27:46.186 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:46.193 7f1cc92be700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1cc40a0420 0x7f1cc40a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:46.186 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:46.193 7f1cc92be700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1cc4189d00 0x7f1cc418a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:46.186 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:46.193 7f1cc92be700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1cc418a720 0x7f1cc40b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:46.186 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:46.193 7f1cc92be700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cc4189800 con 0x7f1cc4189d00 2021-02-10T11:27:46.186 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:46.193 7f1cc92be700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cc418ab20 con 0x7f1cc418a720 2021-02-10T11:27:46.186 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:46.193 7f1cc92be700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cc40a1140 con 0x7f1cc40a0420 2021-02-10T11:27:46.186 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:46.193 7f1cc287c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1cc4189d00 0x7f1cc418a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:46.186 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:46.193 7f1cc287c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1cc4189d00 0x7f1cc418a100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:46174/0 (socket says 172.21.2.112:46174) 2021-02-10T11:27:46.186 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:46.193 7f1cc287c700 1 -- 172.21.2.112:0/3244844206 learned_addr learned my addr 172.21.2.112:0/3244844206 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:27:46.186 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:46.194 7f1cc287c700 1 -- 172.21.2.112:0/3244844206 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1cc40a0420 msgr2=0x7f1cc40a0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:46.187 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:46.194 7f1cc287c700 1 --2- 172.21.2.112:0/3244844206 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1cc40a0420 0x7f1cc40a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:46.187 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:46.194 7f1cc287c700 1 -- 172.21.2.112:0/3244844206 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1cc418a720 msgr2=0x7f1cc40b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:46.187 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:46.194 7f1cc287c700 1 --2- 172.21.2.112:0/3244844206 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1cc418a720 0x7f1cc40b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:46.187 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:46.194 7f1cc287c700 1 -- 172.21.2.112:0/3244844206 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cc40b8200 con 0x7f1cc4189d00 2021-02-10T11:27:46.187 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:46.194 7f1cc287c700 1 --2- 172.21.2.112:0/3244844206 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1cc4189d00 0x7f1cc418a100 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 rx=0x7f1cac006f40 tx=0x7f1cac00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:46.187 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:46.194 7f1cbbfff700 1 -- 172.21.2.112:0/3244844206 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1cac00bd80 con 0x7f1cc4189d00 2021-02-10T11:27:46.187 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:46.194 7f1cbbfff700 1 -- 172.21.2.112:0/3244844206 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cac000c90 con 0x7f1cc4189d00 2021-02-10T11:27:46.187 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:46.194 7f1cbbfff700 1 -- 172.21.2.112:0/3244844206 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1cac002220 con 0x7f1cc4189d00 2021-02-10T11:27:46.187 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:46.194 7f1cc92be700 1 -- 172.21.2.112:0/3244844206 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cc40b8200 con 0x7f1cc4189d00 2021-02-10T11:27:46.187 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:46.194 7f1cc92be700 1 -- 172.21.2.112:0/3244844206 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cc40b8200 con 0x7f1cc4189d00 2021-02-10T11:27:46.187 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:46.195 7f1cbbfff700 1 -- 172.21.2.112:0/3244844206 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1cac00a000 con 0x7f1cc4189d00 2021-02-10T11:27:46.187 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:46.195 7f1cbbfff700 1 --2- 172.21.2.112:0/3244844206 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1ca80186c0 0x7f1ca801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:46.187 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:46.195 7f1cbbfff700 1 -- 172.21.2.112:0/3244844206 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f1cac0323f0 con 0x7f1cc4189d00 2021-02-10T11:27:46.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:46.195 7f1cc307d700 1 --2- 172.21.2.112:0/3244844206 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1ca80186c0 0x7f1ca801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:46.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:46.196 7f1cc307d700 1 --2- 172.21.2.112:0/3244844206 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1ca80186c0 0x7f1ca801abc0 crc :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:46.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:46.226 7f1cc92be700 1 -- 172.21.2.112:0/3244844206 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cc4105ae0 con 0x7f1cc4189d00 2021-02-10T11:27:46.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:46.228 7f1cbbfff700 1 -- 172.21.2.112:0/3244844206 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1cac02cef0 con 0x7f1cc4189d00 2021-02-10T11:27:46.348 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:46.358 7f1cc92be700 1 -- 172.21.2.112:0/3244844206 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348652", "target": ["mgr", ""]}) v1 -- 0x7f1cc418ab20 con 0x7f1ca80186c0 2021-02-10T11:27:46.351 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:46.362 7f1cbbfff700 1 -- 172.21.2.112:0/3244844206 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f1cb4006060 con 0x7f1ca80186c0 2021-02-10T11:27:46.357 INFO:teuthology.orchestra.run.gibba012.stdout:/volumes/_nogroup/subvolume_0000000000348652/153043db-13f2-4d2b-b7d6-9780075b1168 2021-02-10T11:27:46.357 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:46.367 7f1cc92be700 1 -- 172.21.2.112:0/3244844206 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1ca80186c0 msgr2=0x7f1ca801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:46.357 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:46.367 7f1cc92be700 1 --2- 172.21.2.112:0/3244844206 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1ca80186c0 0x7f1ca801abc0 crc :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:46.357 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:46.368 7f1cc92be700 1 -- 172.21.2.112:0/3244844206 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1cc4189d00 msgr2=0x7f1cc418a100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:46.357 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:46.368 7f1cc92be700 1 --2- 172.21.2.112:0/3244844206 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1cc4189d00 0x7f1cc418a100 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 rx=0x7f1cac006f40 tx=0x7f1cac00b990).stop 2021-02-10T11:27:46.358 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:46.368 7f1cc92be700 1 -- 172.21.2.112:0/3244844206 shutdown_connections 2021-02-10T11:27:46.358 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:46.368 7f1cc92be700 1 --2- 172.21.2.112:0/3244844206 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1ca80186c0 0x7f1ca801abc0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:46.358 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:46.368 7f1cc92be700 1 --2- 172.21.2.112:0/3244844206 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1cc418a720 0x7f1cc40b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:46.358 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:46.368 7f1cc92be700 1 --2- 172.21.2.112:0/3244844206 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1cc4189d00 0x7f1cc418a100 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:46.358 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:46.368 7f1cc92be700 1 --2- 172.21.2.112:0/3244844206 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1cc40a0420 0x7f1cc40a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:46.358 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:46.368 7f1cc92be700 1 -- 172.21.2.112:0/3244844206 >> 172.21.2.112:0/3244844206 conn(0x7f1cc4094e10 msgr2=0x7f1cc40a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:46.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:46.368 7f1cc92be700 1 -- 172.21.2.112:0/3244844206 shutdown_connections 2021-02-10T11:27:46.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:46.368 7f1cc92be700 1 -- 172.21.2.112:0/3244844206 wait complete. 2021-02-10T11:27:46.365 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000348652 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000348652/153043db-13f2-4d2b-b7d6-9780075b1168 2021-02-10T11:27:46.366 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:46.366 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:46.366 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:46.702 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:27:46.703 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:46.703 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348652/153043db-13f2-4d2b-b7d6-9780075b1168/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:47.086 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:27:47.086 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:27:47.086 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0250978 s, 41.8 MB/s 2021-02-10T11:27:47.087 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:47.088 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:47.088 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:47.422 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:27:47.423 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:47.423 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348652/153043db-13f2-4d2b-b7d6-9780075b1168/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:47.802 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:27:47.802 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:27:47.802 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0263751 s, 39.8 MB/s 2021-02-10T11:27:47.803 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:47.803 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:47.803 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:48.139 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:27:48.139 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:48.139 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348652/153043db-13f2-4d2b-b7d6-9780075b1168/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:48.517 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:27:48.517 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:27:48.518 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0251962 s, 41.6 MB/s 2021-02-10T11:27:48.519 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:48.519 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:48.519 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:48.855 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:27:48.855 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:48.855 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348652/153043db-13f2-4d2b-b7d6-9780075b1168/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:49.230 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:27:49.230 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:27:49.230 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0236986 s, 44.2 MB/s 2021-02-10T11:27:49.231 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:49.232 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:49.232 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:49.567 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:27:49.567 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:49.567 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348652/153043db-13f2-4d2b-b7d6-9780075b1168/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:49.940 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:27:49.940 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:27:49.940 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0237998 s, 44.1 MB/s 2021-02-10T11:27:49.941 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:49.941 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:49.941 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:50.277 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:27:50.277 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:50.278 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348652/153043db-13f2-4d2b-b7d6-9780075b1168/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:50.655 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:27:50.656 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:27:50.656 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0245231 s, 42.8 MB/s 2021-02-10T11:27:50.656 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:50.657 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:50.657 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:50.993 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:27:50.993 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:50.994 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348652/153043db-13f2-4d2b-b7d6-9780075b1168/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:51.369 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:27:51.369 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:27:51.369 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0225843 s, 46.4 MB/s 2021-02-10T11:27:51.370 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:51.370 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:51.371 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:51.707 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:27:51.707 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:51.707 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348652/153043db-13f2-4d2b-b7d6-9780075b1168/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:52.083 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:27:52.083 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:27:52.083 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0249238 s, 42.1 MB/s 2021-02-10T11:27:52.084 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:52.084 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:52.084 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:52.418 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:27:52.419 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:52.419 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348652/153043db-13f2-4d2b-b7d6-9780075b1168/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:52.793 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:27:52.794 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:27:52.794 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0233663 s, 44.9 MB/s 2021-02-10T11:27:52.795 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:52.795 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:52.795 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:53.131 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:27:53.131 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:53.131 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348652/153043db-13f2-4d2b-b7d6-9780075b1168/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:53.509 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:27:53.509 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:27:53.509 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0255833 s, 41.0 MB/s 2021-02-10T11:27:53.510 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000348653 2021-02-10T11:27:53.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:53.936 7fac3f658700 1 Processor -- start 2021-02-10T11:27:53.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:53.937 7fac3f658700 1 -- start start 2021-02-10T11:27:53.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:53.937 7fac3f658700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fac381b0870 0x7fac380f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:53.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:53.937 7fac3f658700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fac380efe00 0x7fac380f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:53.943 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:53.937 7fac3f658700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac381b0c70 con 0x7fac381b0470 2021-02-10T11:27:53.943 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:53.937 7fac3f658700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac380e5ea0 con 0x7fac380efe00 2021-02-10T11:27:53.943 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:53.937 7fac3f658700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac380f0b50 con 0x7fac381b0870 2021-02-10T11:27:53.943 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:53.937 7fac3d474700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fac381b0470 0x7fac380ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:45644/0 (socket says 172.21.2.112:45644) 2021-02-10T11:27:53.943 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:53.937 7fac3d474700 1 -- 172.21.2.112:0/231377202 learned_addr learned my addr 172.21.2.112:0/231377202 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:27:53.943 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:53.937 7fac3dc75700 1 --2- 172.21.2.112:0/231377202 >> v2:172.21.2.112:3300/0 conn(0x7fac380efe00 0x7fac380f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:53.943 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:53.937 7fac3cc73700 1 --2- 172.21.2.112:0/231377202 >> v2:172.21.2.120:3300/0 conn(0x7fac381b0870 0x7fac380f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:53.943 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:53.937 7fac2bfff700 1 -- 172.21.2.112:0/231377202 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2423159409 0 0) 0x7fac340010b0 con 0x7fac381b0470 2021-02-10T11:27:53.943 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:53.937 7fac2bfff700 1 -- 172.21.2.112:0/231377202 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac24002fe0 con 0x7fac381b0470 2021-02-10T11:27:53.943 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:53.938 7fac3dc75700 1 -- 172.21.2.112:0/231377202 >> v2:172.21.2.120:3300/0 conn(0x7fac381b0870 msgr2=0x7fac380f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:27:53.943 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:53.938 7fac3dc75700 1 --2- 172.21.2.112:0/231377202 >> v2:172.21.2.120:3300/0 conn(0x7fac381b0870 0x7fac380f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:53.943 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:53.938 7fac3dc75700 1 -- 172.21.2.112:0/231377202 >> v1:172.21.2.112:6789/0 conn(0x7fac381b0470 legacy=0x7fac380ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:53.943 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:53.938 7fac3dc75700 1 -- 172.21.2.112:0/231377202 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac380f0820 con 0x7fac380efe00 2021-02-10T11:27:53.944 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:53.938 7fac3dc75700 1 --2- 172.21.2.112:0/231377202 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fac380efe00 0x7fac380f0200 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 rx=0x7fac30006e30 tx=0x7fac30009e60).ready entity=mon.0 client_cookie=e8e864443e7f88b2 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:53.944 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:53.938 7fac2bfff700 1 -- 172.21.2.112:0/231377202 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fac30006a50 con 0x7fac380efe00 2021-02-10T11:27:53.944 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:53.938 7fac2bfff700 1 -- 172.21.2.112:0/231377202 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac3000b300 con 0x7fac380efe00 2021-02-10T11:27:53.944 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:53.938 7fac3f658700 1 -- 172.21.2.112:0/231377202 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fac380efe00 msgr2=0x7fac380f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:53.944 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:53.938 7fac3f658700 1 --2- 172.21.2.112:0/231377202 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fac380efe00 0x7fac380f0200 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 rx=0x7fac30006e30 tx=0x7fac30009e60).stop 2021-02-10T11:27:53.944 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:53.938 7fac2bfff700 1 -- 172.21.2.112:0/231377202 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fac3000b820 con 0x7fac380efe00 2021-02-10T11:27:53.944 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:53.939 7fac3f658700 1 -- 172.21.2.112:0/231377202 shutdown_connections 2021-02-10T11:27:53.944 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:53.939 7fac3f658700 1 --2- 172.21.2.112:0/231377202 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fac380efe00 0x7fac380f0200 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:53.944 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:53.939 7fac3f658700 1 --2- 172.21.2.112:0/231377202 >> v2:172.21.2.120:3300/0 conn(0x7fac381b0870 0x7fac380f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:53.944 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:53.939 7fac3f658700 1 -- 172.21.2.112:0/231377202 >> 172.21.2.112:0/231377202 conn(0x7fac380ce3e0 msgr2=0x7fac380e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:53.944 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:53.939 7fac3f658700 1 -- 172.21.2.112:0/231377202 shutdown_connections 2021-02-10T11:27:53.944 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:53.939 7fac3f658700 1 -- 172.21.2.112:0/231377202 wait complete. 2021-02-10T11:27:53.946 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:53.953 7fac3f658700 1 Processor -- start 2021-02-10T11:27:53.946 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:53.953 7fac3f658700 1 -- start start 2021-02-10T11:27:53.946 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:53.954 7fac3f658700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fac380c28d0 0x7fac380c2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:53.946 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:53.954 7fac3f658700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fac3808e690 0x7fac3808ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:53.946 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:53.954 7fac3f658700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fac381069f0 0x7fac38106df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:53.946 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:53.954 7fac3f658700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac3808f340 con 0x7fac3808e690 2021-02-10T11:27:53.946 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:53.954 7fac3f658700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac3808f0b0 con 0x7fac380c28d0 2021-02-10T11:27:53.946 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:53.954 7fac3f658700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac380c33b0 con 0x7fac381069f0 2021-02-10T11:27:53.946 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:53.954 7fac3cc73700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fac3808e690 0x7fac3808ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:53.946 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:53.954 7fac3cc73700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fac3808e690 0x7fac3808ea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:46192/0 (socket says 172.21.2.112:46192) 2021-02-10T11:27:53.947 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:53.954 7fac3cc73700 1 -- 172.21.2.112:0/999325917 learned_addr learned my addr 172.21.2.112:0/999325917 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:27:53.947 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:53.954 7fac3dc75700 1 --2- 172.21.2.112:0/999325917 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fac381069f0 0x7fac38106df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:53.947 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:53.954 7fac3d474700 1 --2- 172.21.2.112:0/999325917 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fac380c28d0 0x7fac380c2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:53.947 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:53.954 7fac3cc73700 1 -- 172.21.2.112:0/999325917 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fac381069f0 msgr2=0x7fac38106df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:53.947 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:53.954 7fac3cc73700 1 --2- 172.21.2.112:0/999325917 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fac381069f0 0x7fac38106df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:53.947 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:53.954 7fac3cc73700 1 -- 172.21.2.112:0/999325917 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fac380c28d0 msgr2=0x7fac380c2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:53.947 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:53.954 7fac3cc73700 1 --2- 172.21.2.112:0/999325917 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fac380c28d0 0x7fac380c2cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:53.947 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:53.954 7fac3cc73700 1 -- 172.21.2.112:0/999325917 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac38107410 con 0x7fac3808e690 2021-02-10T11:27:53.947 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:53.955 7fac3cc73700 1 --2- 172.21.2.112:0/999325917 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fac3808e690 0x7fac3808ea90 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 rx=0x7fac2c006f60 tx=0x7fac2c00b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:53.947 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:53.955 7fac29ffb700 1 -- 172.21.2.112:0/999325917 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fac2c00bd60 con 0x7fac3808e690 2021-02-10T11:27:53.947 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:53.955 7fac29ffb700 1 -- 172.21.2.112:0/999325917 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac2c000c90 con 0x7fac3808e690 2021-02-10T11:27:53.947 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:53.955 7fac29ffb700 1 -- 172.21.2.112:0/999325917 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fac2c0022b0 con 0x7fac3808e690 2021-02-10T11:27:53.947 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:53.955 7fac3f658700 1 -- 172.21.2.112:0/999325917 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac38107410 con 0x7fac3808e690 2021-02-10T11:27:53.948 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:53.955 7fac3f658700 1 -- 172.21.2.112:0/999325917 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac380bc380 con 0x7fac3808e690 2021-02-10T11:27:53.948 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:53.955 7fac29ffb700 1 -- 172.21.2.112:0/999325917 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fac2c006b00 con 0x7fac3808e690 2021-02-10T11:27:53.948 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:53.955 7fac29ffb700 1 --2- 172.21.2.112:0/999325917 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fac180186c0 0x7fac1801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:53.948 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:53.955 7fac29ffb700 1 -- 172.21.2.112:0/999325917 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fac2c0327b0 con 0x7fac3808e690 2021-02-10T11:27:53.948 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:53.956 7fac3d474700 1 --2- 172.21.2.112:0/999325917 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fac180186c0 0x7fac1801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:53.948 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:53.956 7fac3d474700 1 --2- 172.21.2.112:0/999325917 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fac180186c0 0x7fac1801abc0 crc :-1 s=READY pgs=345 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:53.976 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:53.987 7fac3f658700 1 -- 172.21.2.112:0/999325917 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac38105ae0 con 0x7fac3808e690 2021-02-10T11:27:53.978 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:53.989 7fac29ffb700 1 -- 172.21.2.112:0/999325917 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fac2c000940 con 0x7fac3808e690 2021-02-10T11:27:54.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:54.117 7fac3f658700 1 -- 172.21.2.112:0/999325917 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348653", "target": ["mgr", ""]}) v1 -- 0x7fac380bc380 con 0x7fac180186c0 2021-02-10T11:27:54.132 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:54.143 7fac29ffb700 1 -- 172.21.2.112:0/999325917 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fac34000ec0 con 0x7fac180186c0 2021-02-10T11:27:54.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:54.150 7fac3f658700 1 -- 172.21.2.112:0/999325917 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fac180186c0 msgr2=0x7fac1801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:54.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:54.150 7fac3f658700 1 --2- 172.21.2.112:0/999325917 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fac180186c0 0x7fac1801abc0 crc :-1 s=READY pgs=345 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:54.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:54.150 7fac3f658700 1 -- 172.21.2.112:0/999325917 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fac3808e690 msgr2=0x7fac3808ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:54.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:54.150 7fac3f658700 1 --2- 172.21.2.112:0/999325917 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fac3808e690 0x7fac3808ea90 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 rx=0x7fac2c006f60 tx=0x7fac2c00b970).stop 2021-02-10T11:27:54.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:54.150 7fac3f658700 1 -- 172.21.2.112:0/999325917 shutdown_connections 2021-02-10T11:27:54.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:54.150 7fac3f658700 1 --2- 172.21.2.112:0/999325917 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fac180186c0 0x7fac1801abc0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:54.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:54.150 7fac3f658700 1 --2- 172.21.2.112:0/999325917 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fac381069f0 0x7fac38106df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:54.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:54.150 7fac3f658700 1 --2- 172.21.2.112:0/999325917 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fac3808e690 0x7fac3808ea90 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:54.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:54.150 7fac3f658700 1 --2- 172.21.2.112:0/999325917 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fac380c28d0 0x7fac380c2cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:54.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:54.150 7fac3f658700 1 -- 172.21.2.112:0/999325917 >> 172.21.2.112:0/999325917 conn(0x7fac380bbf60 msgr2=0x7fac38105d10 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:54.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:54.151 7fac3f658700 1 -- 172.21.2.112:0/999325917 shutdown_connections 2021-02-10T11:27:54.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:54.151 7fac3f658700 1 -- 172.21.2.112:0/999325917 wait complete. 2021-02-10T11:27:54.147 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000348653 2021-02-10T11:27:54.581 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:54.574 7f634bb01700 1 Processor -- start 2021-02-10T11:27:54.581 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:54.574 7f634bb01700 1 -- start start 2021-02-10T11:27:54.581 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:54.575 7f634bb01700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f63441b0470 0x7f63441b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:54.581 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:54.575 7f634bb01700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f63440ef610 0x7f63440efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:54.581 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:54.575 7f634bb01700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63440f0b50 con 0x7f63440f0030 2021-02-10T11:27:54.581 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:54.575 7f634bb01700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63440e52e0 con 0x7f63441b0470 2021-02-10T11:27:54.581 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:54.575 7f634bb01700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63440e54d0 con 0x7f63440ef610 2021-02-10T11:27:54.581 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:54.575 7f634991d700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f63441b0470 0x7f63441b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:54.582 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:54.575 7f634911c700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f63440ef610 0x7f63440efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:54.582 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:54.575 7f634a11e700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7f63440f0030 0x7f63440f10d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:39216/0 (socket says 172.21.2.112:39216) 2021-02-10T11:27:54.582 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:54.575 7f634a11e700 1 -- 172.21.2.112:0/1884836635 learned_addr learned my addr 172.21.2.112:0/1884836635 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:27:54.582 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:54.576 7f634911c700 1 -- 172.21.2.112:0/1884836635 >> v2:172.21.2.120:3300/0 conn(0x7f63441b0470 msgr2=0x7f63441b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:27:54.582 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:54.576 7f634911c700 1 --2- 172.21.2.112:0/1884836635 >> v2:172.21.2.120:3300/0 conn(0x7f63441b0470 0x7f63441b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:54.582 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:54.576 7f634911c700 1 -- 172.21.2.112:0/1884836635 >> v1:172.21.2.120:6789/0 conn(0x7f63440f0030 legacy=0x7f63440f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:54.582 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:54.576 7f634911c700 1 -- 172.21.2.112:0/1884836635 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63440f0430 con 0x7f63440ef610 2021-02-10T11:27:54.582 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:54.576 7f634991d700 1 --2- 172.21.2.112:0/1884836635 >> v2:172.21.2.120:3300/0 conn(0x7f63441b0470 0x7f63441b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:27:54.582 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:54.576 7f634911c700 1 --2- 172.21.2.112:0/1884836635 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f63440ef610 0x7f63440efa10 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f6338006650 tx=0x7f6338009a80).ready entity=mon.2 client_cookie=e79d8fda142be2bb server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:54.582 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:54.576 7f634891b700 1 -- 172.21.2.112:0/1884836635 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f633800c0c0 con 0x7f63440ef610 2021-02-10T11:27:54.582 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:54.576 7f634891b700 1 -- 172.21.2.112:0/1884836635 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f633800c340 con 0x7f63440ef610 2021-02-10T11:27:54.582 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:54.576 7f634891b700 1 -- 172.21.2.112:0/1884836635 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f633800c860 con 0x7f63440ef610 2021-02-10T11:27:54.583 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:54.576 7f634bb01700 1 -- 172.21.2.112:0/1884836635 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f63440ef610 msgr2=0x7f63440efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:54.583 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:54.576 7f634bb01700 1 --2- 172.21.2.112:0/1884836635 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f63440ef610 0x7f63440efa10 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f6338006650 tx=0x7f6338009a80).stop 2021-02-10T11:27:54.583 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:54.577 7f634bb01700 1 -- 172.21.2.112:0/1884836635 shutdown_connections 2021-02-10T11:27:54.583 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:54.577 7f634bb01700 1 --2- 172.21.2.112:0/1884836635 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f63440ef610 0x7f63440efa10 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:54.583 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:54.577 7f634bb01700 1 --2- 172.21.2.112:0/1884836635 >> v2:172.21.2.120:3300/0 conn(0x7f63441b0470 0x7f63441b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:54.583 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:54.577 7f634bb01700 1 -- 172.21.2.112:0/1884836635 >> 172.21.2.112:0/1884836635 conn(0x7f63440ce3e0 msgr2=0x7f63440e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:54.583 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:54.577 7f634bb01700 1 -- 172.21.2.112:0/1884836635 shutdown_connections 2021-02-10T11:27:54.583 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:54.577 7f634bb01700 1 -- 172.21.2.112:0/1884836635 wait complete. 2021-02-10T11:27:54.583 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:54.592 7f634bb01700 1 Processor -- start 2021-02-10T11:27:54.583 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:54.592 7f634bb01700 1 -- start start 2021-02-10T11:27:54.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:54.592 7f634bb01700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f63440a0420 0x7f63440a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:54.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:54.592 7f634bb01700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6344189d00 0x7f634418a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:54.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:54.592 7f634bb01700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f634418a720 0x7f63440b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:54.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:54.592 7f634bb01700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6344189800 con 0x7f634418a720 2021-02-10T11:27:54.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:54.592 7f634bb01700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f634418ab20 con 0x7f63440a0420 2021-02-10T11:27:54.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:54.592 7f634bb01700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63440a1140 con 0x7f6344189d00 2021-02-10T11:27:54.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:54.592 7f634a11e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f634418a720 0x7f63440b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:54.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:54.592 7f634a11e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f634418a720 0x7f63440b7be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:46204/0 (socket says 172.21.2.112:46204) 2021-02-10T11:27:54.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:54.592 7f634a11e700 1 -- 172.21.2.112:0/2451909931 learned_addr learned my addr 172.21.2.112:0/2451909931 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:27:54.586 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:54.592 7f634911c700 1 --2- 172.21.2.112:0/2451909931 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6344189d00 0x7f634418a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:54.586 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:54.592 7f634991d700 1 --2- 172.21.2.112:0/2451909931 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f63440a0420 0x7f63440a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:54.586 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:54.593 7f634a11e700 1 -- 172.21.2.112:0/2451909931 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6344189d00 msgr2=0x7f634418a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:54.586 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:54.593 7f634a11e700 1 --2- 172.21.2.112:0/2451909931 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6344189d00 0x7f634418a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:54.586 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:54.593 7f634a11e700 1 -- 172.21.2.112:0/2451909931 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f63440a0420 msgr2=0x7f63440a0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:54.586 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:54.593 7f634a11e700 1 --2- 172.21.2.112:0/2451909931 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f63440a0420 0x7f63440a0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:54.586 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:54.593 7f634a11e700 1 -- 172.21.2.112:0/2451909931 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63440b8200 con 0x7f634418a720 2021-02-10T11:27:54.586 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:54.593 7f634a11e700 1 --2- 172.21.2.112:0/2451909931 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f634418a720 0x7f63440b7be0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 rx=0x7f633c00a900 tx=0x7f633c00aca0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:54.586 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:54.593 7f63367fc700 1 -- 172.21.2.112:0/2451909931 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f633c0070d0 con 0x7f634418a720 2021-02-10T11:27:54.586 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:54.593 7f63367fc700 1 -- 172.21.2.112:0/2451909931 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f633c0073d0 con 0x7f634418a720 2021-02-10T11:27:54.587 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:54.593 7f63367fc700 1 -- 172.21.2.112:0/2451909931 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f633c004920 con 0x7f634418a720 2021-02-10T11:27:54.587 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:54.593 7f634bb01700 1 -- 172.21.2.112:0/2451909931 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63440b8200 con 0x7f634418a720 2021-02-10T11:27:54.587 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:54.593 7f634bb01700 1 -- 172.21.2.112:0/2451909931 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63440b8200 con 0x7f634418a720 2021-02-10T11:27:54.587 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:54.593 7f63367fc700 1 -- 172.21.2.112:0/2451909931 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f633c0046a0 con 0x7f634418a720 2021-02-10T11:27:54.587 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:54.594 7f63367fc700 1 --2- 172.21.2.112:0/2451909931 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f632c018730 0x7f632c01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:54.587 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:54.594 7f63367fc700 1 -- 172.21.2.112:0/2451909931 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f633c032b50 con 0x7f634418a720 2021-02-10T11:27:54.587 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:54.594 7f634991d700 1 --2- 172.21.2.112:0/2451909931 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f632c018730 0x7f632c01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:54.587 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:54.594 7f634991d700 1 --2- 172.21.2.112:0/2451909931 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f632c018730 0x7f632c01ac10 crc :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:54.614 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:54.625 7f634bb01700 1 -- 172.21.2.112:0/2451909931 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6344105ae0 con 0x7f634418a720 2021-02-10T11:27:54.616 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:54.627 7f63367fc700 1 -- 172.21.2.112:0/2451909931 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f633c000f40 con 0x7f634418a720 2021-02-10T11:27:54.745 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:54.755 7f634bb01700 1 -- 172.21.2.112:0/2451909931 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348653", "target": ["mgr", ""]}) v1 -- 0x7f6344105ae0 con 0x7f632c018730 2021-02-10T11:27:54.748 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:54.759 7f63367fc700 1 -- 172.21.2.112:0/2451909931 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f63400068f0 con 0x7f632c018730 2021-02-10T11:27:54.752 INFO:teuthology.orchestra.run.gibba012.stdout:/volumes/_nogroup/subvolume_0000000000348653/d14b8cd6-f934-4383-a888-bb916910ab1d 2021-02-10T11:27:54.754 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:54.765 7f634bb01700 1 -- 172.21.2.112:0/2451909931 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f632c018730 msgr2=0x7f632c01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:54.754 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:54.765 7f634bb01700 1 --2- 172.21.2.112:0/2451909931 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f632c018730 0x7f632c01ac10 crc :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:54.754 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:54.765 7f634bb01700 1 -- 172.21.2.112:0/2451909931 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f634418a720 msgr2=0x7f63440b7be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:54.754 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:54.765 7f634bb01700 1 --2- 172.21.2.112:0/2451909931 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f634418a720 0x7f63440b7be0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 rx=0x7f633c00a900 tx=0x7f633c00aca0).stop 2021-02-10T11:27:54.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:54.765 7f634bb01700 1 -- 172.21.2.112:0/2451909931 shutdown_connections 2021-02-10T11:27:54.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:54.765 7f634bb01700 1 --2- 172.21.2.112:0/2451909931 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f632c018730 0x7f632c01ac10 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:54.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:54.765 7f634bb01700 1 --2- 172.21.2.112:0/2451909931 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f634418a720 0x7f63440b7be0 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:54.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:54.765 7f634bb01700 1 --2- 172.21.2.112:0/2451909931 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6344189d00 0x7f634418a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:54.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:54.765 7f634bb01700 1 --2- 172.21.2.112:0/2451909931 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f63440a0420 0x7f63440a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:54.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:54.765 7f634bb01700 1 -- 172.21.2.112:0/2451909931 >> 172.21.2.112:0/2451909931 conn(0x7f6344094e10 msgr2=0x7f63440a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:54.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:54.765 7f634bb01700 1 -- 172.21.2.112:0/2451909931 shutdown_connections 2021-02-10T11:27:54.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:27:54.765 7f634bb01700 1 -- 172.21.2.112:0/2451909931 wait complete. 2021-02-10T11:27:54.762 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000348653 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000348653/d14b8cd6-f934-4383-a888-bb916910ab1d 2021-02-10T11:27:54.762 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:54.762 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:54.762 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:55.099 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:27:55.099 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:55.100 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348653/d14b8cd6-f934-4383-a888-bb916910ab1d/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:55.478 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:27:55.478 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:27:55.478 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0225796 s, 46.4 MB/s 2021-02-10T11:27:55.479 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:55.479 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:55.479 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:55.815 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:27:55.815 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:55.816 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348653/d14b8cd6-f934-4383-a888-bb916910ab1d/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:56.192 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:27:56.192 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:27:56.192 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0256467 s, 40.9 MB/s 2021-02-10T11:27:56.193 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:56.193 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:56.193 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:56.529 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:27:56.529 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:56.529 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348653/d14b8cd6-f934-4383-a888-bb916910ab1d/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:56.905 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:27:56.905 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:27:56.905 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0238682 s, 43.9 MB/s 2021-02-10T11:27:56.906 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:56.906 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:56.906 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:57.242 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:27:57.242 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:57.242 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348653/d14b8cd6-f934-4383-a888-bb916910ab1d/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:57.620 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:27:57.620 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:27:57.620 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0260051 s, 40.3 MB/s 2021-02-10T11:27:57.621 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:57.621 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:57.622 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:57.957 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:27:57.958 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:57.958 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348653/d14b8cd6-f934-4383-a888-bb916910ab1d/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:58.335 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:27:58.335 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:27:58.336 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0226249 s, 46.3 MB/s 2021-02-10T11:27:58.336 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:58.337 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:58.337 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:58.673 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:27:58.673 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:58.673 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348653/d14b8cd6-f934-4383-a888-bb916910ab1d/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:59.050 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:27:59.050 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:27:59.050 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0247057 s, 42.4 MB/s 2021-02-10T11:27:59.051 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:59.051 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:59.051 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:59.386 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:27:59.386 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:59.387 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348653/d14b8cd6-f934-4383-a888-bb916910ab1d/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:59.758 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:27:59.759 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:27:59.759 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0226956 s, 46.2 MB/s 2021-02-10T11:27:59.760 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:59.760 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:59.760 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:00.095 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:28:00.095 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:00.096 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348653/d14b8cd6-f934-4383-a888-bb916910ab1d/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:00.471 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:28:00.471 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:28:00.471 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0230123 s, 45.6 MB/s 2021-02-10T11:28:00.472 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:00.472 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:00.472 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:00.808 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:28:00.808 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:00.808 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348653/d14b8cd6-f934-4383-a888-bb916910ab1d/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:01.186 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:28:01.186 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:28:01.186 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0241671 s, 43.4 MB/s 2021-02-10T11:28:01.187 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:01.187 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:01.187 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:01.523 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:28:01.523 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:01.523 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348653/d14b8cd6-f934-4383-a888-bb916910ab1d/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:01.899 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:28:01.899 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:28:01.899 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.024647 s, 42.5 MB/s 2021-02-10T11:28:01.900 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000348654 2021-02-10T11:28:02.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.326 7f4c1dd57700 1 Processor -- start 2021-02-10T11:28:02.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.326 7f4c1dd57700 1 -- start start 2021-02-10T11:28:02.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.327 7f4c1dd57700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f4c181b0470 0x7f4c181b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:02.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.327 7f4c1dd57700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4c180f20e0 0x7f4c180f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:02.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.327 7f4c1dd57700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c180f0b50 con 0x7f4c180ef610 2021-02-10T11:28:02.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.327 7f4c1dd57700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c180f0820 con 0x7f4c181b0470 2021-02-10T11:28:02.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.327 7f4c1dd57700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c180f24e0 con 0x7f4c180f20e0 2021-02-10T11:28:02.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.327 7f4c177fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f4c181b0470 0x7f4c181b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:02.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.327 7f4c177fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f4c181b0470 0x7f4c181b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:46208/0 (socket says 172.21.2.112:46208) 2021-02-10T11:28:02.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.327 7f4c177fe700 1 -- 172.21.2.112:0/1880811763 learned_addr learned my addr 172.21.2.112:0/1880811763 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:28:02.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.327 7f4c177fe700 1 -- 172.21.2.112:0/1880811763 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4c180f20e0 msgr2=0x7f4c180f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:28:02.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.327 7f4c177fe700 1 --2- 172.21.2.112:0/1880811763 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4c180f20e0 0x7f4c180f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:02.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.327 7f4c177fe700 1 -- 172.21.2.112:0/1880811763 >> v1:172.21.2.120:6789/0 conn(0x7f4c180ef610 legacy=0x7f4c180efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:28:02.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.327 7f4c177fe700 1 -- 172.21.2.112:0/1880811763 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c180e52e0 con 0x7f4c181b0470 2021-02-10T11:28:02.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.327 7f4c177fe700 1 --2- 172.21.2.112:0/1880811763 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4c181b0470 0x7f4c181b0870 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 rx=0x7f4c00006700 tx=0x7f4c00009b80).ready entity=mon.0 client_cookie=52f5e7dfd0be0272 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:02.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.328 7f4c167fc700 1 -- 172.21.2.112:0/1880811763 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4c0000c0c0 con 0x7f4c181b0470 2021-02-10T11:28:02.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.328 7f4c167fc700 1 -- 172.21.2.112:0/1880811763 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c0000c400 con 0x7f4c181b0470 2021-02-10T11:28:02.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.328 7f4c167fc700 1 -- 172.21.2.112:0/1880811763 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4c0000c920 con 0x7f4c181b0470 2021-02-10T11:28:02.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.328 7f4c1dd57700 1 -- 172.21.2.112:0/1880811763 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4c181b0470 msgr2=0x7f4c181b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:02.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.328 7f4c1dd57700 1 --2- 172.21.2.112:0/1880811763 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4c181b0470 0x7f4c181b0870 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 rx=0x7f4c00006700 tx=0x7f4c00009b80).stop 2021-02-10T11:28:02.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.328 7f4c1dd57700 1 -- 172.21.2.112:0/1880811763 shutdown_connections 2021-02-10T11:28:02.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.328 7f4c1dd57700 1 --2- 172.21.2.112:0/1880811763 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4c180f20e0 0x7f4c180f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:02.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.328 7f4c1dd57700 1 --2- 172.21.2.112:0/1880811763 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4c181b0470 0x7f4c181b0870 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:02.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.328 7f4c1dd57700 1 -- 172.21.2.112:0/1880811763 >> 172.21.2.112:0/1880811763 conn(0x7f4c180ce3e0 msgr2=0x7f4c180e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:28:02.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.328 7f4c1dd57700 1 -- 172.21.2.112:0/1880811763 shutdown_connections 2021-02-10T11:28:02.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.329 7f4c1dd57700 1 -- 172.21.2.112:0/1880811763 wait complete. 2021-02-10T11:28:02.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.343 7f4c1dd57700 1 Processor -- start 2021-02-10T11:28:02.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.343 7f4c1dd57700 1 -- start start 2021-02-10T11:28:02.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.344 7f4c1dd57700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4c180b2840 0x7f4c181a4300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:02.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.344 7f4c1dd57700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4c180b30d0 0x7f4c180b34d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:02.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.344 7f4c1dd57700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4c180b3af0 0x7f4c180f0580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:02.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.344 7f4c1dd57700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c180b2c40 con 0x7f4c180b2840 2021-02-10T11:28:02.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.344 7f4c1dd57700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c181a4920 con 0x7f4c180b30d0 2021-02-10T11:28:02.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.344 7f4c1dd57700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c180f0ba0 con 0x7f4c180b3af0 2021-02-10T11:28:02.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.344 7f4c177fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4c180b2840 0x7f4c181a4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:02.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.344 7f4c177fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4c180b2840 0x7f4c181a4300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:46210/0 (socket says 172.21.2.112:46210) 2021-02-10T11:28:02.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.344 7f4c177fe700 1 -- 172.21.2.112:0/4038528159 learned_addr learned my addr 172.21.2.112:0/4038528159 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:28:02.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.344 7f4c177fe700 1 -- 172.21.2.112:0/4038528159 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4c180b3af0 msgr2=0x7f4c180f0580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:28:02.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.344 7f4c177fe700 1 --2- 172.21.2.112:0/4038528159 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4c180b3af0 0x7f4c180f0580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:02.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.344 7f4c177fe700 1 -- 172.21.2.112:0/4038528159 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4c180b30d0 msgr2=0x7f4c180b34d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:28:02.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.344 7f4c177fe700 1 --2- 172.21.2.112:0/4038528159 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4c180b30d0 0x7f4c180b34d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:02.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.344 7f4c177fe700 1 -- 172.21.2.112:0/4038528159 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c180f0dc0 con 0x7f4c180b2840 2021-02-10T11:28:02.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.344 7f4c177fe700 1 --2- 172.21.2.112:0/4038528159 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4c180b2840 0x7f4c181a4300 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 rx=0x7f4c0000b5e0 tx=0x7f4c0000b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:02.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.344 7f4c0ffff700 1 -- 172.21.2.112:0/4038528159 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4c0000bd70 con 0x7f4c180b2840 2021-02-10T11:28:02.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.344 7f4c0ffff700 1 -- 172.21.2.112:0/4038528159 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c00002030 con 0x7f4c180b2840 2021-02-10T11:28:02.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.344 7f4c0ffff700 1 -- 172.21.2.112:0/4038528159 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4c00002550 con 0x7f4c180b2840 2021-02-10T11:28:02.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.344 7f4c1dd57700 1 -- 172.21.2.112:0/4038528159 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c180f0dc0 con 0x7f4c180b2840 2021-02-10T11:28:02.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.344 7f4c1dd57700 1 -- 172.21.2.112:0/4038528159 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c180f1350 con 0x7f4c180b2840 2021-02-10T11:28:02.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.345 7f4c0ffff700 1 -- 172.21.2.112:0/4038528159 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4c00000c90 con 0x7f4c180b2840 2021-02-10T11:28:02.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.345 7f4c0ffff700 1 --2- 172.21.2.112:0/4038528159 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4bfc0186c0 0x7f4bfc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:02.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.345 7f4c0ffff700 1 -- 172.21.2.112:0/4038528159 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f4c000327e0 con 0x7f4c180b2840 2021-02-10T11:28:02.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.346 7f4c16ffd700 1 --2- 172.21.2.112:0/4038528159 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4bfc0186c0 0x7f4bfc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:02.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.346 7f4c16ffd700 1 --2- 172.21.2.112:0/4038528159 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4bfc0186c0 0x7f4bfc01abc0 crc :-1 s=READY pgs=347 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:02.366 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.377 7f4c1dd57700 1 -- 172.21.2.112:0/4038528159 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c180f18f0 con 0x7f4c180b2840 2021-02-10T11:28:02.368 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.379 7f4c0ffff700 1 -- 172.21.2.112:0/4038528159 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4c00000940 con 0x7f4c180b2840 2021-02-10T11:28:02.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.505 7f4c1dd57700 1 -- 172.21.2.112:0/4038528159 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348654", "target": ["mgr", ""]}) v1 -- 0x7f4c180f0050 con 0x7f4bfc0186c0 2021-02-10T11:28:02.519 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.531 7f4c0ffff700 1 -- 172.21.2.112:0/4038528159 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f4c08006970 con 0x7f4bfc0186c0 2021-02-10T11:28:02.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.538 7f4c1dd57700 1 -- 172.21.2.112:0/4038528159 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4bfc0186c0 msgr2=0x7f4bfc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:02.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.538 7f4c1dd57700 1 --2- 172.21.2.112:0/4038528159 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4bfc0186c0 0x7f4bfc01abc0 crc :-1 s=READY pgs=347 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:02.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.538 7f4c1dd57700 1 -- 172.21.2.112:0/4038528159 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4c180b2840 msgr2=0x7f4c181a4300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:02.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.538 7f4c1dd57700 1 --2- 172.21.2.112:0/4038528159 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4c180b2840 0x7f4c181a4300 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 rx=0x7f4c0000b5e0 tx=0x7f4c0000b980).stop 2021-02-10T11:28:02.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.538 7f4c1dd57700 1 -- 172.21.2.112:0/4038528159 shutdown_connections 2021-02-10T11:28:02.528 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.538 7f4c1dd57700 1 --2- 172.21.2.112:0/4038528159 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4bfc0186c0 0x7f4bfc01abc0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:02.528 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.538 7f4c1dd57700 1 --2- 172.21.2.112:0/4038528159 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4c180b30d0 0x7f4c180b34d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:02.528 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.538 7f4c1dd57700 1 --2- 172.21.2.112:0/4038528159 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4c180b3af0 0x7f4c180f0580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:02.528 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.538 7f4c1dd57700 1 --2- 172.21.2.112:0/4038528159 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4c180b2840 0x7f4c181a4300 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:02.528 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.538 7f4c1dd57700 1 -- 172.21.2.112:0/4038528159 >> 172.21.2.112:0/4038528159 conn(0x7f4c180c36f0 msgr2=0x7f4c18094c30 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:28:02.528 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.539 7f4c1dd57700 1 -- 172.21.2.112:0/4038528159 shutdown_connections 2021-02-10T11:28:02.528 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.539 7f4c1dd57700 1 -- 172.21.2.112:0/4038528159 wait complete. 2021-02-10T11:28:02.536 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000348654 2021-02-10T11:28:02.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.962 7fb2b979f700 1 Processor -- start 2021-02-10T11:28:02.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.962 7fb2b979f700 1 -- start start 2021-02-10T11:28:02.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.963 7fb2b979f700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fb2b40f05f0 0x7fb2b40f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:02.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.963 7fb2b979f700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2b41b0c70 con 0x7fb2b41b0470 2021-02-10T11:28:02.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.963 7fb2b979f700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2b40f0b50 con 0x7fb2b41b0870 2021-02-10T11:28:02.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.963 7fb2b979f700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2b40e5ea0 con 0x7fb2b40f05f0 2021-02-10T11:28:02.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.963 7fb2b3574700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fb2b41b0470 0x7fb2b40ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:45680/0 (socket says 172.21.2.112:45680) 2021-02-10T11:28:02.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.963 7fb2b3574700 1 -- 172.21.2.112:0/2646612740 learned_addr learned my addr 172.21.2.112:0/2646612740 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:28:02.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.963 7fb2b3d75700 1 --2- 172.21.2.112:0/2646612740 >> v2:172.21.2.120:3300/0 conn(0x7fb2b40f05f0 0x7fb2b40f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:02.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.963 7fb2b2572700 1 -- 172.21.2.112:0/2646612740 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3599544639 0 0) 0x7fb29c0009b0 con 0x7fb2b41b0470 2021-02-10T11:28:02.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.963 7fb2b2572700 1 -- 172.21.2.112:0/2646612740 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2a0002fe0 con 0x7fb2b41b0470 2021-02-10T11:28:02.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.964 7fb2b3d75700 1 -- 172.21.2.112:0/2646612740 >> v1:172.21.2.120:6789/0 conn(0x7fb2b41b0870 legacy=0x7fb2b40efe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:02.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.964 7fb2b3d75700 1 -- 172.21.2.112:0/2646612740 >> v1:172.21.2.112:6789/0 conn(0x7fb2b41b0470 legacy=0x7fb2b40ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:02.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.964 7fb2b3d75700 1 -- 172.21.2.112:0/2646612740 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2b40e52e0 con 0x7fb2b40f05f0 2021-02-10T11:28:02.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.964 7fb2b3d75700 1 --2- 172.21.2.112:0/2646612740 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb2b40f05f0 0x7fb2b40f20e0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0x7fb2ac006e50 tx=0x7fb2ac009e70).ready entity=mon.1 client_cookie=5977ecafbac9dd3f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:02.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.964 7fb2b2572700 1 -- 172.21.2.112:0/2646612740 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb2ac00b510 con 0x7fb2b40f05f0 2021-02-10T11:28:02.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.964 7fb2b2572700 1 -- 172.21.2.112:0/2646612740 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2ac006790 con 0x7fb2b40f05f0 2021-02-10T11:28:02.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.964 7fb2b2572700 1 -- 172.21.2.112:0/2646612740 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb2ac00b940 con 0x7fb2b40f05f0 2021-02-10T11:28:02.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.964 7fb2b979f700 1 -- 172.21.2.112:0/2646612740 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb2b40f05f0 msgr2=0x7fb2b40f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:02.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.964 7fb2b979f700 1 --2- 172.21.2.112:0/2646612740 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb2b40f05f0 0x7fb2b40f20e0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0x7fb2ac006e50 tx=0x7fb2ac009e70).stop 2021-02-10T11:28:02.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.965 7fb2b979f700 1 -- 172.21.2.112:0/2646612740 shutdown_connections 2021-02-10T11:28:02.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.965 7fb2b979f700 1 --2- 172.21.2.112:0/2646612740 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb2b40f05f0 0x7fb2b40f20e0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:02.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.965 7fb2b979f700 1 -- 172.21.2.112:0/2646612740 >> 172.21.2.112:0/2646612740 conn(0x7fb2b40ce3e0 msgr2=0x7fb2b40e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:28:02.971 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.965 7fb2b979f700 1 -- 172.21.2.112:0/2646612740 shutdown_connections 2021-02-10T11:28:02.971 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.965 7fb2b979f700 1 -- 172.21.2.112:0/2646612740 wait complete. 2021-02-10T11:28:02.971 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.980 7fb2b979f700 1 Processor -- start 2021-02-10T11:28:02.978 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.980 7fb2b979f700 1 -- start start 2021-02-10T11:28:02.978 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.980 7fb2b979f700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb2b4094fb0 0x7fb2b409e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:02.978 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.980 7fb2b979f700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb2b40c2be0 0x7fb2b40c2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:02.978 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.980 7fb2b979f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb2b40c3600 0x7fb2b40a9c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:02.979 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.980 7fb2b979f700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2b409ebf0 con 0x7fb2b40c3600 2021-02-10T11:28:02.979 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.980 7fb2b979f700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2b40aa2a0 con 0x7fb2b4094fb0 2021-02-10T11:28:02.979 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.980 7fb2b979f700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2b40aa4c0 con 0x7fb2b40c2be0 2021-02-10T11:28:02.979 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.980 7fb2b3d75700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb2b40c3600 0x7fb2b40a9c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:02.979 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.980 7fb2b3d75700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb2b40c3600 0x7fb2b40a9c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:46224/0 (socket says 172.21.2.112:46224) 2021-02-10T11:28:02.979 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.980 7fb2b3d75700 1 -- 172.21.2.112:0/2424964480 learned_addr learned my addr 172.21.2.112:0/2424964480 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:28:02.979 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.981 7fb2b3574700 1 --2- 172.21.2.112:0/2424964480 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb2b4094fb0 0x7fb2b409e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:02.979 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.981 7fb2b2d73700 1 --2- 172.21.2.112:0/2424964480 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb2b40c2be0 0x7fb2b40c2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:02.979 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.981 7fb2b3d75700 1 -- 172.21.2.112:0/2424964480 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb2b40c2be0 msgr2=0x7fb2b40c2fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:02.979 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.981 7fb2b3d75700 1 --2- 172.21.2.112:0/2424964480 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb2b40c2be0 0x7fb2b40c2fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:02.979 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.981 7fb2b3d75700 1 -- 172.21.2.112:0/2424964480 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb2b4094fb0 msgr2=0x7fb2b409e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:02.979 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.981 7fb2b3d75700 1 --2- 172.21.2.112:0/2424964480 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb2b4094fb0 0x7fb2b409e5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:02.979 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.981 7fb2b3d75700 1 -- 172.21.2.112:0/2424964480 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2b40aa6e0 con 0x7fb2b40c3600 2021-02-10T11:28:02.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.981 7fb2b3574700 1 --2- 172.21.2.112:0/2424964480 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb2b4094fb0 0x7fb2b409e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:28:02.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.981 7fb2b3d75700 1 --2- 172.21.2.112:0/2424964480 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb2b40c3600 0x7fb2b40a9c80 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 rx=0x7fb2ac006c50 tx=0x7fb2ac00bd20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:02.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.981 7fb2abfff700 1 -- 172.21.2.112:0/2424964480 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb2ac0093d0 con 0x7fb2b40c3600 2021-02-10T11:28:02.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.981 7fb2abfff700 1 -- 172.21.2.112:0/2424964480 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2ac00be60 con 0x7fb2b40c3600 2021-02-10T11:28:02.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.981 7fb2b979f700 1 -- 172.21.2.112:0/2424964480 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2b40aa6e0 con 0x7fb2b40c3600 2021-02-10T11:28:02.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.981 7fb2b979f700 1 -- 172.21.2.112:0/2424964480 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2b40aac70 con 0x7fb2b40c3600 2021-02-10T11:28:02.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.981 7fb2abfff700 1 -- 172.21.2.112:0/2424964480 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb2ac00c4e0 con 0x7fb2b40c3600 2021-02-10T11:28:02.989 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.982 7fb2abfff700 1 -- 172.21.2.112:0/2424964480 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb2ac0093d0 con 0x7fb2b40c3600 2021-02-10T11:28:02.989 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.982 7fb2abfff700 1 --2- 172.21.2.112:0/2424964480 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb2980186c0 0x7fb29801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:02.989 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.982 7fb2abfff700 1 -- 172.21.2.112:0/2424964480 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fb2ac0328f0 con 0x7fb2b40c3600 2021-02-10T11:28:02.989 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.982 7fb2b3574700 1 --2- 172.21.2.112:0/2424964480 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb2980186c0 0x7fb29801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:02.989 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:02.983 7fb2b3574700 1 --2- 172.21.2.112:0/2424964480 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb2980186c0 0x7fb29801abc0 crc :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:03.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:03.013 7fb2b979f700 1 -- 172.21.2.112:0/2424964480 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2b40a9750 con 0x7fb2b40c3600 2021-02-10T11:28:03.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:03.015 7fb2abfff700 1 -- 172.21.2.112:0/2424964480 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb2ac002ef0 con 0x7fb2b40c3600 2021-02-10T11:28:03.133 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:03.144 7fb2b979f700 1 -- 172.21.2.112:0/2424964480 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348654", "target": ["mgr", ""]}) v1 -- 0x7fb2b40aa2a0 con 0x7fb2980186c0 2021-02-10T11:28:03.136 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:03.148 7fb2abfff700 1 -- 172.21.2.112:0/2424964480 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7fb29c000960 con 0x7fb2980186c0 2021-02-10T11:28:03.137 INFO:teuthology.orchestra.run.gibba012.stdout:/volumes/_nogroup/subvolume_0000000000348654/c833ef87-ac80-4ba2-a8e7-c7c4f55b7bf9 2021-02-10T11:28:03.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:03.149 7fb2b979f700 1 -- 172.21.2.112:0/2424964480 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb2980186c0 msgr2=0x7fb29801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:03.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:03.149 7fb2b979f700 1 --2- 172.21.2.112:0/2424964480 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb2980186c0 0x7fb29801abc0 crc :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:03.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:03.149 7fb2b979f700 1 -- 172.21.2.112:0/2424964480 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb2b40c3600 msgr2=0x7fb2b40a9c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:03.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:03.149 7fb2b979f700 1 --2- 172.21.2.112:0/2424964480 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb2b40c3600 0x7fb2b40a9c80 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 rx=0x7fb2ac006c50 tx=0x7fb2ac00bd20).stop 2021-02-10T11:28:03.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:03.149 7fb2b979f700 1 -- 172.21.2.112:0/2424964480 shutdown_connections 2021-02-10T11:28:03.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:03.149 7fb2b979f700 1 --2- 172.21.2.112:0/2424964480 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb2980186c0 0x7fb29801abc0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:03.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:03.149 7fb2b979f700 1 --2- 172.21.2.112:0/2424964480 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb2b40c3600 0x7fb2b40a9c80 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:03.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:03.149 7fb2b979f700 1 --2- 172.21.2.112:0/2424964480 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb2b40c2be0 0x7fb2b40c2fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:03.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:03.149 7fb2b979f700 1 --2- 172.21.2.112:0/2424964480 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb2b4094fb0 0x7fb2b409e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:03.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:03.149 7fb2b979f700 1 -- 172.21.2.112:0/2424964480 >> 172.21.2.112:0/2424964480 conn(0x7fb2b409f630 msgr2=0x7fb2b40a0130 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:28:03.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:03.149 7fb2b979f700 1 -- 172.21.2.112:0/2424964480 shutdown_connections 2021-02-10T11:28:03.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:03.149 7fb2b979f700 1 -- 172.21.2.112:0/2424964480 wait complete. 2021-02-10T11:28:03.147 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000348654 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000348654/c833ef87-ac80-4ba2-a8e7-c7c4f55b7bf9 2021-02-10T11:28:03.147 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:03.147 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:03.147 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:03.491 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:28:03.492 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:03.492 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348654/c833ef87-ac80-4ba2-a8e7-c7c4f55b7bf9/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:03.877 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:28:03.878 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:28:03.878 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0227538 s, 46.1 MB/s 2021-02-10T11:28:03.878 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:03.879 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:03.879 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:04.214 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:28:04.214 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:04.214 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348654/c833ef87-ac80-4ba2-a8e7-c7c4f55b7bf9/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:04.589 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:28:04.589 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:28:04.589 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0259806 s, 40.4 MB/s 2021-02-10T11:28:04.590 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:04.590 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:04.590 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:04.926 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:28:04.926 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:04.926 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348654/c833ef87-ac80-4ba2-a8e7-c7c4f55b7bf9/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:05.304 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:28:05.304 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:28:05.304 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.023713 s, 44.2 MB/s 2021-02-10T11:28:05.305 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:05.305 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:05.305 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:05.642 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:28:05.643 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:05.643 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348654/c833ef87-ac80-4ba2-a8e7-c7c4f55b7bf9/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:06.020 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:28:06.021 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:28:06.021 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0248434 s, 42.2 MB/s 2021-02-10T11:28:06.022 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:06.022 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:06.022 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:06.357 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:28:06.357 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:06.358 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348654/c833ef87-ac80-4ba2-a8e7-c7c4f55b7bf9/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:06.733 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:28:06.734 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:28:06.734 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0242426 s, 43.3 MB/s 2021-02-10T11:28:06.735 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:06.736 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:06.736 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:07.071 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:28:07.071 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:07.072 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348654/c833ef87-ac80-4ba2-a8e7-c7c4f55b7bf9/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:07.446 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:28:07.447 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:28:07.447 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0255354 s, 41.1 MB/s 2021-02-10T11:28:07.448 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:07.448 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:07.448 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:07.783 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:28:07.783 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:07.783 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348654/c833ef87-ac80-4ba2-a8e7-c7c4f55b7bf9/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:08.160 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:28:08.160 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:28:08.160 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0245236 s, 42.8 MB/s 2021-02-10T11:28:08.161 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:08.161 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:08.161 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:08.499 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:28:08.499 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:08.499 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348654/c833ef87-ac80-4ba2-a8e7-c7c4f55b7bf9/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:08.886 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:28:08.886 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:28:08.886 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0244492 s, 42.9 MB/s 2021-02-10T11:28:08.887 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:08.887 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:08.887 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:09.223 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:28:09.224 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:09.224 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348654/c833ef87-ac80-4ba2-a8e7-c7c4f55b7bf9/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:09.596 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:28:09.596 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:28:09.597 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0243584 s, 43.0 MB/s 2021-02-10T11:28:09.597 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:09.598 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:09.598 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:09.934 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:28:09.934 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:09.934 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348654/c833ef87-ac80-4ba2-a8e7-c7c4f55b7bf9/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:10.312 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:28:10.313 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:28:10.313 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0252143 s, 41.6 MB/s 2021-02-10T11:28:10.314 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000348655 2021-02-10T11:28:10.745 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:10.738 7ff0954e3700 1 Processor -- start 2021-02-10T11:28:10.745 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:10.738 7ff0954e3700 1 -- start start 2021-02-10T11:28:10.745 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:10.739 7ff0954e3700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7ff0901b0470 0x7ff0901b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:10.745 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:10.739 7ff0954e3700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0900f0b50 con 0x7ff0900f20e0 2021-02-10T11:28:10.745 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:10.739 7ff0954e3700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0900f24e0 con 0x7ff0900ef610 2021-02-10T11:28:10.745 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:10.739 7ff0954e3700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0900e5ea0 con 0x7ff0901b0470 2021-02-10T11:28:10.746 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:10.739 7ff08fac0700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7ff0900f20e0 0x7ff0900f0200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:45692/0 (socket says 172.21.2.112:45692) 2021-02-10T11:28:10.746 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:10.739 7ff08fac0700 1 -- 172.21.2.112:0/620369643 learned_addr learned my addr 172.21.2.112:0/620369643 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:28:10.746 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:10.739 7ff08f2bf700 1 --2- 172.21.2.112:0/620369643 >> v2:172.21.2.120:3300/0 conn(0x7ff0901b0470 0x7ff0901b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:10.746 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:10.739 7ff08e2bd700 1 -- 172.21.2.112:0/620369643 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2710991848 0 0) 0x7ff088001500 con 0x7ff0900f20e0 2021-02-10T11:28:10.746 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:10.739 7ff08e2bd700 1 -- 172.21.2.112:0/620369643 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff07c002fe0 con 0x7ff0900f20e0 2021-02-10T11:28:10.746 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:10.739 7ff08e2bd700 1 -- 172.21.2.112:0/620369643 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3160780252 0 0) 0x7ff080000ff0 con 0x7ff0900ef610 2021-02-10T11:28:10.746 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:10.739 7ff08e2bd700 1 -- 172.21.2.112:0/620369643 --> v1:172.21.2.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff07c003210 con 0x7ff0900ef610 2021-02-10T11:28:10.746 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:10.740 7ff08e2bd700 1 -- 172.21.2.112:0/620369643 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3694837382 0 0) 0x7ff088001460 con 0x7ff0900f20e0 2021-02-10T11:28:10.746 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:10.740 7ff08e2bd700 1 -- 172.21.2.112:0/620369643 >> v2:172.21.2.120:3300/0 conn(0x7ff0901b0470 msgr2=0x7ff0901b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:28:10.746 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:10.740 7ff08e2bd700 1 --2- 172.21.2.112:0/620369643 >> v2:172.21.2.120:3300/0 conn(0x7ff0901b0470 0x7ff0901b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:10.746 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:10.740 7ff08e2bd700 1 -- 172.21.2.112:0/620369643 >> v1:172.21.2.120:6789/0 conn(0x7ff0900ef610 legacy=0x7ff0900efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:10.746 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:10.740 7ff08e2bd700 1 -- 172.21.2.112:0/620369643 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0900e52e0 con 0x7ff0900f20e0 2021-02-10T11:28:10.747 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:10.740 7ff08e2bd700 1 -- 172.21.2.112:0/620369643 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7ff088001980 con 0x7ff0900f20e0 2021-02-10T11:28:10.747 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:10.740 7ff08e2bd700 1 -- 172.21.2.112:0/620369643 >> v1:172.21.2.112:6789/0 conn(0x7ff0900f20e0 legacy=0x7ff0900f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:10.747 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:10.740 7ff08e2bd700 1 --2- 172.21.2.112:0/620369643 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff07c0038e0 0x7ff07c003ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:10.747 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:10.740 7ff08e2bd700 1 --2- 172.21.2.112:0/620369643 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff07c0067e0 0x7ff07c008c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:10.747 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:10.740 7ff08e2bd700 1 --2- 172.21.2.112:0/620369643 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff07c009580 0x7ff07c00baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:10.747 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:10.740 7ff08e2bd700 1 -- 172.21.2.112:0/620369643 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff07c00c2f0 con 0x7ff07c0067e0 2021-02-10T11:28:10.747 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:10.740 7ff08e2bd700 1 -- 172.21.2.112:0/620369643 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff07c0504e0 con 0x7ff07c0038e0 2021-02-10T11:28:10.747 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:10.740 7ff08e2bd700 1 -- 172.21.2.112:0/620369643 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff07c050700 con 0x7ff07c009580 2021-02-10T11:28:10.747 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:10.740 7ff08f2bf700 1 --2- 172.21.2.112:0/620369643 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff07c0038e0 0x7ff07c003ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:10.747 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:10.740 7ff08eabe700 1 --2- 172.21.2.112:0/620369643 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff07c0067e0 0x7ff07c008c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:10.747 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:10.740 7ff08fac0700 1 --2- 172.21.2.112:0/620369643 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff07c009580 0x7ff07c00baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:10.747 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:10.740 7ff08f2bf700 1 -- 172.21.2.112:0/620369643 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff07c009580 msgr2=0x7ff07c00baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:28:10.748 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:10.740 7ff08f2bf700 1 --2- 172.21.2.112:0/620369643 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff07c009580 0x7ff07c00baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:10.748 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:10.740 7ff08f2bf700 1 -- 172.21.2.112:0/620369643 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff07c0067e0 msgr2=0x7ff07c008c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:28:10.748 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:10.740 7ff08f2bf700 1 --2- 172.21.2.112:0/620369643 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff07c0067e0 0x7ff07c008c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:10.748 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:10.740 7ff08f2bf700 1 -- 172.21.2.112:0/620369643 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff07c0509a0 con 0x7ff07c0038e0 2021-02-10T11:28:10.748 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:10.740 7ff08eabe700 1 --2- 172.21.2.112:0/620369643 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff07c0067e0 0x7ff07c008c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:28:10.748 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:10.740 7ff08fac0700 1 -- 172.21.2.112:0/620369643 reap_dead start 2021-02-10T11:28:10.748 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:10.740 7ff08f2bf700 1 --2- 172.21.2.112:0/620369643 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff07c0038e0 0x7ff07c003ce0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0x7ff07800c8f0 tx=0x7ff07800cc90).ready entity=mon.1 client_cookie=8a3ffc283489b4ea server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:10.748 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:10.741 7ff08e2bd700 1 -- 172.21.2.112:0/620369643 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff0780009b0 con 0x7ff07c0038e0 2021-02-10T11:28:10.748 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:10.741 7ff08e2bd700 1 -- 172.21.2.112:0/620369643 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff078009310 con 0x7ff07c0038e0 2021-02-10T11:28:10.748 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:10.741 7ff08e2bd700 1 -- 172.21.2.112:0/620369643 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff078009830 con 0x7ff07c0038e0 2021-02-10T11:28:10.748 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:10.741 7ff0954e3700 1 -- 172.21.2.112:0/620369643 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff07c0038e0 msgr2=0x7ff07c003ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:10.748 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:10.741 7ff0954e3700 1 --2- 172.21.2.112:0/620369643 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff07c0038e0 0x7ff07c003ce0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0x7ff07800c8f0 tx=0x7ff07800cc90).stop 2021-02-10T11:28:10.749 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:10.741 7ff0954e3700 1 -- 172.21.2.112:0/620369643 shutdown_connections 2021-02-10T11:28:10.749 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:10.741 7ff0954e3700 1 --2- 172.21.2.112:0/620369643 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff07c0038e0 0x7ff07c003ce0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:10.749 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:10.741 7ff0954e3700 1 -- 172.21.2.112:0/620369643 >> 172.21.2.112:0/620369643 conn(0x7ff0900ce3e0 msgr2=0x7ff0900e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:28:10.749 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:10.741 7ff0954e3700 1 -- 172.21.2.112:0/620369643 shutdown_connections 2021-02-10T11:28:10.749 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:10.741 7ff0954e3700 1 -- 172.21.2.112:0/620369643 wait complete. 2021-02-10T11:28:10.749 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:10.756 7ff0954e3700 1 Processor -- start 2021-02-10T11:28:10.749 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:10.756 7ff0954e3700 1 -- start start 2021-02-10T11:28:10.751 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:10.756 7ff0954e3700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff090092c40 0x7ff0900b3160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:10.751 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:10.756 7ff0954e3700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff090093040 0x7ff090106a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:10.751 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:10.756 7ff0954e3700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff090107060 0x7ff090107460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:10.751 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:10.756 7ff0954e3700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0900c2120 con 0x7ff090107060 2021-02-10T11:28:10.751 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:10.756 7ff0954e3700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0900b3c00 con 0x7ff090093040 2021-02-10T11:28:10.751 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:10.756 7ff0954e3700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0900b3780 con 0x7ff090092c40 2021-02-10T11:28:10.751 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:10.757 7ff08fac0700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff090107060 0x7ff090107460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:10.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:10.757 7ff08fac0700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff090107060 0x7ff090107460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:46244/0 (socket says 172.21.2.112:46244) 2021-02-10T11:28:10.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:10.757 7ff08fac0700 1 -- 172.21.2.112:0/479806730 learned_addr learned my addr 172.21.2.112:0/479806730 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:28:10.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:10.757 7ff08eabe700 1 --2- 172.21.2.112:0/479806730 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff090093040 0x7ff090106a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:10.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:10.757 7ff08f2bf700 1 --2- 172.21.2.112:0/479806730 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff090092c40 0x7ff0900b3160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:10.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:10.757 7ff08fac0700 1 -- 172.21.2.112:0/479806730 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff090092c40 msgr2=0x7ff0900b3160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:10.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:10.757 7ff08fac0700 1 --2- 172.21.2.112:0/479806730 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff090092c40 0x7ff0900b3160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:10.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:10.757 7ff08fac0700 1 -- 172.21.2.112:0/479806730 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff090093040 msgr2=0x7ff090106a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:10.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:10.757 7ff08fac0700 1 --2- 172.21.2.112:0/479806730 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff090093040 0x7ff090106a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:10.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:10.757 7ff08fac0700 1 -- 172.21.2.112:0/479806730 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0900c0ba0 con 0x7ff090107060 2021-02-10T11:28:10.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:10.757 7ff08fac0700 1 --2- 172.21.2.112:0/479806730 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff090107060 0x7ff090107460 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 rx=0x7ff088001c30 tx=0x7ff08800ac90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:10.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:10.758 7ff087fff700 1 -- 172.21.2.112:0/479806730 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff088004930 con 0x7ff090107060 2021-02-10T11:28:10.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:10.758 7ff087fff700 1 -- 172.21.2.112:0/479806730 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff088004bf0 con 0x7ff090107060 2021-02-10T11:28:10.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:10.758 7ff087fff700 1 -- 172.21.2.112:0/479806730 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff0880072a0 con 0x7ff090107060 2021-02-10T11:28:10.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:10.758 7ff0954e3700 1 -- 172.21.2.112:0/479806730 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0900c0ba0 con 0x7ff090107060 2021-02-10T11:28:10.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:10.758 7ff0954e3700 1 -- 172.21.2.112:0/479806730 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0900c1130 con 0x7ff090107060 2021-02-10T11:28:10.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:10.758 7ff087fff700 1 -- 172.21.2.112:0/479806730 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff088004720 con 0x7ff090107060 2021-02-10T11:28:10.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:10.758 7ff087fff700 1 --2- 172.21.2.112:0/479806730 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff0740186c0 0x7ff07401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:10.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:10.758 7ff087fff700 1 -- 172.21.2.112:0/479806730 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7ff0880328c0 con 0x7ff090107060 2021-02-10T11:28:10.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:10.759 7ff08f2bf700 1 --2- 172.21.2.112:0/479806730 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff0740186c0 0x7ff07401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:10.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:10.759 7ff08f2bf700 1 --2- 172.21.2.112:0/479806730 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff0740186c0 0x7ff07401abc0 crc :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:10.777 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:10.789 7ff0954e3700 1 -- 172.21.2.112:0/479806730 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff090105b30 con 0x7ff090107060 2021-02-10T11:28:10.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:10.791 7ff087fff700 1 -- 172.21.2.112:0/479806730 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff08802ebb0 con 0x7ff090107060 2021-02-10T11:28:10.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:10.920 7ff0954e3700 1 -- 172.21.2.112:0/479806730 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348655", "target": ["mgr", ""]}) v1 -- 0x7ff0900f0b50 con 0x7ff0740186c0 2021-02-10T11:28:10.935 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:10.947 7ff087fff700 1 -- 172.21.2.112:0/479806730 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7ff0780069d0 con 0x7ff0740186c0 2021-02-10T11:28:10.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:10.953 7ff0954e3700 1 -- 172.21.2.112:0/479806730 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff0740186c0 msgr2=0x7ff07401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:10.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:10.953 7ff0954e3700 1 --2- 172.21.2.112:0/479806730 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff0740186c0 0x7ff07401abc0 crc :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:10.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:10.953 7ff0954e3700 1 -- 172.21.2.112:0/479806730 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff090107060 msgr2=0x7ff090107460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:10.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:10.953 7ff0954e3700 1 --2- 172.21.2.112:0/479806730 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff090107060 0x7ff090107460 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 rx=0x7ff088001c30 tx=0x7ff08800ac90).stop 2021-02-10T11:28:10.943 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:10.953 7ff0954e3700 1 -- 172.21.2.112:0/479806730 shutdown_connections 2021-02-10T11:28:10.943 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:10.953 7ff0954e3700 1 --2- 172.21.2.112:0/479806730 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff0740186c0 0x7ff07401abc0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:10.943 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:10.953 7ff0954e3700 1 --2- 172.21.2.112:0/479806730 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff090093040 0x7ff090106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:10.943 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:10.953 7ff0954e3700 1 --2- 172.21.2.112:0/479806730 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff090107060 0x7ff090107460 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:10.943 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:10.953 7ff0954e3700 1 --2- 172.21.2.112:0/479806730 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff090092c40 0x7ff0900b3160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:10.943 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:10.953 7ff0954e3700 1 -- 172.21.2.112:0/479806730 >> 172.21.2.112:0/479806730 conn(0x7ff0900c40e0 msgr2=0x7ff0900bf620 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:28:10.943 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:10.954 7ff0954e3700 1 -- 172.21.2.112:0/479806730 shutdown_connections 2021-02-10T11:28:10.943 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:10.954 7ff0954e3700 1 -- 172.21.2.112:0/479806730 wait complete. 2021-02-10T11:28:10.950 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000348655 2021-02-10T11:28:11.383 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:11.378 7ff390d79700 1 Processor -- start 2021-02-10T11:28:11.383 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:11.378 7ff390d79700 1 -- start start 2021-02-10T11:28:11.383 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:11.378 7ff390d79700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7ff38c1b0470 0x7ff38c1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:11.383 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:11.378 7ff390d79700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7ff38c0ef610 0x7ff38c0efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:11.383 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:11.378 7ff390d79700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff38c0f0b50 con 0x7ff38c0f0030 2021-02-10T11:28:11.383 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:11.378 7ff390d79700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff38c0e52e0 con 0x7ff38c1b0470 2021-02-10T11:28:11.383 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:11.378 7ff390d79700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff38c0e5490 con 0x7ff38c0ef610 2021-02-10T11:28:11.383 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:11.378 7ff38ab13700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7ff38c1b0470 0x7ff38c1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:11.383 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:11.379 7ff38ab13700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7ff38c1b0470 0x7ff38c1b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:46248/0 (socket says 172.21.2.112:46248) 2021-02-10T11:28:11.383 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:11.379 7ff38ab13700 1 -- 172.21.2.112:0/1635672132 learned_addr learned my addr 172.21.2.112:0/1635672132 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:28:11.384 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:11.379 7ff382312700 1 --2- 172.21.2.112:0/1635672132 >> v2:172.21.2.120:3300/0 conn(0x7ff38c0ef610 0x7ff38c0efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:11.384 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:11.379 7ff38ab13700 1 -- 172.21.2.112:0/1635672132 >> v2:172.21.2.120:3300/0 conn(0x7ff38c0ef610 msgr2=0x7ff38c0efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:28:11.384 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:11.379 7ff38ab13700 1 --2- 172.21.2.112:0/1635672132 >> v2:172.21.2.120:3300/0 conn(0x7ff38c0ef610 0x7ff38c0efa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:11.384 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:11.379 7ff38ab13700 1 -- 172.21.2.112:0/1635672132 >> v1:172.21.2.120:6789/0 conn(0x7ff38c0f0030 legacy=0x7ff38c0f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:28:11.384 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:11.379 7ff38ab13700 1 -- 172.21.2.112:0/1635672132 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff38c0f0430 con 0x7ff38c1b0470 2021-02-10T11:28:11.384 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:11.379 7ff38ab13700 1 --2- 172.21.2.112:0/1635672132 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff38c1b0470 0x7ff38c1b0870 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 rx=0x7ff37c006700 tx=0x7ff37c009b80).ready entity=mon.0 client_cookie=6bae3ee34a8e0d50 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:11.384 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:11.379 7ff38a312700 1 -- 172.21.2.112:0/1635672132 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff37c00c0c0 con 0x7ff38c1b0470 2021-02-10T11:28:11.384 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:11.379 7ff38a312700 1 -- 172.21.2.112:0/1635672132 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff37c00c440 con 0x7ff38c1b0470 2021-02-10T11:28:11.384 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:11.379 7ff38a312700 1 -- 172.21.2.112:0/1635672132 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff37c00c8a0 con 0x7ff38c1b0470 2021-02-10T11:28:11.384 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:11.379 7ff390d79700 1 -- 172.21.2.112:0/1635672132 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff38c1b0470 msgr2=0x7ff38c1b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:11.384 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:11.379 7ff390d79700 1 --2- 172.21.2.112:0/1635672132 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff38c1b0470 0x7ff38c1b0870 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 rx=0x7ff37c006700 tx=0x7ff37c009b80).stop 2021-02-10T11:28:11.384 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:11.380 7ff390d79700 1 -- 172.21.2.112:0/1635672132 shutdown_connections 2021-02-10T11:28:11.384 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:11.380 7ff390d79700 1 --2- 172.21.2.112:0/1635672132 >> v2:172.21.2.120:3300/0 conn(0x7ff38c0ef610 0x7ff38c0efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:11.385 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:11.380 7ff390d79700 1 --2- 172.21.2.112:0/1635672132 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff38c1b0470 0x7ff38c1b0870 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:11.385 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:11.380 7ff390d79700 1 -- 172.21.2.112:0/1635672132 >> 172.21.2.112:0/1635672132 conn(0x7ff38c0ce3e0 msgr2=0x7ff38c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:28:11.385 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:11.380 7ff390d79700 1 -- 172.21.2.112:0/1635672132 shutdown_connections 2021-02-10T11:28:11.385 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:11.380 7ff390d79700 1 -- 172.21.2.112:0/1635672132 wait complete. 2021-02-10T11:28:11.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:11.394 7ff390d79700 1 Processor -- start 2021-02-10T11:28:11.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:11.394 7ff390d79700 1 -- start start 2021-02-10T11:28:11.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:11.395 7ff390d79700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff38c0a0280 0x7ff38c0aeff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:11.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:11.395 7ff390d79700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff38c1a4450 0x7ff38c1a4850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:11.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:11.395 7ff390d79700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff38c08bf50 0x7ff38c08c350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:11.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:11.395 7ff390d79700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff38c0a0700 con 0x7ff38c1a4450 2021-02-10T11:28:11.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:11.395 7ff390d79700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff38c0af610 con 0x7ff38c08bf50 2021-02-10T11:28:11.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:11.395 7ff390d79700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff38c08c970 con 0x7ff38c0a0280 2021-02-10T11:28:11.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:11.395 7ff382312700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff38c1a4450 0x7ff38c1a4850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:11.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:11.395 7ff382312700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff38c1a4450 0x7ff38c1a4850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:46254/0 (socket says 172.21.2.112:46254) 2021-02-10T11:28:11.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:11.395 7ff382312700 1 -- 172.21.2.112:0/3756472236 learned_addr learned my addr 172.21.2.112:0/3756472236 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:28:11.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:11.395 7ff38ab13700 1 --2- 172.21.2.112:0/3756472236 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff38c0a0280 0x7ff38c0aeff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:11.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:11.395 7ff38b314700 1 --2- 172.21.2.112:0/3756472236 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff38c08bf50 0x7ff38c08c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:11.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:11.395 7ff382312700 1 -- 172.21.2.112:0/3756472236 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff38c0a0280 msgr2=0x7ff38c0aeff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:11.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:11.395 7ff382312700 1 --2- 172.21.2.112:0/3756472236 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff38c0a0280 0x7ff38c0aeff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:11.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:11.395 7ff382312700 1 -- 172.21.2.112:0/3756472236 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff38c08bf50 msgr2=0x7ff38c08c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:11.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:11.395 7ff382312700 1 --2- 172.21.2.112:0/3756472236 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff38c08bf50 0x7ff38c08c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:11.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:11.395 7ff382312700 1 -- 172.21.2.112:0/3756472236 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff38c08cb90 con 0x7ff38c1a4450 2021-02-10T11:28:11.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:11.395 7ff38b314700 1 --2- 172.21.2.112:0/3756472236 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff38c08bf50 0x7ff38c08c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:28:11.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:11.395 7ff382312700 1 --2- 172.21.2.112:0/3756472236 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff38c1a4450 0x7ff38c1a4850 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 rx=0x7ff374006f60 tx=0x7ff37400b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:11.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:11.396 7ff383fff700 1 -- 172.21.2.112:0/3756472236 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff3740009b0 con 0x7ff38c1a4450 2021-02-10T11:28:11.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:11.396 7ff383fff700 1 -- 172.21.2.112:0/3756472236 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff37400bd60 con 0x7ff38c1a4450 2021-02-10T11:28:11.389 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:11.396 7ff383fff700 1 -- 172.21.2.112:0/3756472236 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff374002030 con 0x7ff38c1a4450 2021-02-10T11:28:11.389 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:11.396 7ff390d79700 1 -- 172.21.2.112:0/3756472236 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff38c08cb90 con 0x7ff38c1a4450 2021-02-10T11:28:11.389 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:11.396 7ff390d79700 1 -- 172.21.2.112:0/3756472236 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff38c08cb90 con 0x7ff38c1a4450 2021-02-10T11:28:11.389 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:11.396 7ff383fff700 1 -- 172.21.2.112:0/3756472236 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff374000c90 con 0x7ff38c1a4450 2021-02-10T11:28:11.389 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:11.396 7ff383fff700 1 --2- 172.21.2.112:0/3756472236 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff3700186c0 0x7ff37001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:11.389 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:11.396 7ff383fff700 1 -- 172.21.2.112:0/3756472236 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7ff374032580 con 0x7ff38c1a4450 2021-02-10T11:28:11.389 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:11.397 7ff38ab13700 1 --2- 172.21.2.112:0/3756472236 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff3700186c0 0x7ff37001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:11.389 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:11.397 7ff38ab13700 1 --2- 172.21.2.112:0/3756472236 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff3700186c0 0x7ff37001abc0 crc :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:11.417 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:11.428 7ff390d79700 1 -- 172.21.2.112:0/3756472236 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff38c105ae0 con 0x7ff38c1a4450 2021-02-10T11:28:11.420 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:11.431 7ff383fff700 1 -- 172.21.2.112:0/3756472236 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff374006b00 con 0x7ff38c1a4450 2021-02-10T11:28:11.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:11.560 7ff390d79700 1 -- 172.21.2.112:0/3756472236 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348655", "target": ["mgr", ""]}) v1 -- 0x7ff38c0f0b50 con 0x7ff3700186c0 2021-02-10T11:28:11.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:11.565 7ff383fff700 1 -- 172.21.2.112:0/3756472236 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7ff37c006430 con 0x7ff3700186c0 2021-02-10T11:28:11.556 INFO:teuthology.orchestra.run.gibba012.stdout:/volumes/_nogroup/subvolume_0000000000348655/43e93143-cdd6-4e63-a619-52619f736935 2021-02-10T11:28:11.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:11.569 7ff390d79700 1 -- 172.21.2.112:0/3756472236 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff3700186c0 msgr2=0x7ff37001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:11.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:11.569 7ff390d79700 1 --2- 172.21.2.112:0/3756472236 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff3700186c0 0x7ff37001abc0 crc :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:11.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:11.569 7ff390d79700 1 -- 172.21.2.112:0/3756472236 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff38c1a4450 msgr2=0x7ff38c1a4850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:11.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:11.569 7ff390d79700 1 --2- 172.21.2.112:0/3756472236 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff38c1a4450 0x7ff38c1a4850 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 rx=0x7ff374006f60 tx=0x7ff37400b970).stop 2021-02-10T11:28:11.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:11.569 7ff390d79700 1 -- 172.21.2.112:0/3756472236 shutdown_connections 2021-02-10T11:28:11.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:11.569 7ff390d79700 1 --2- 172.21.2.112:0/3756472236 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff3700186c0 0x7ff37001abc0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:11.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:11.569 7ff390d79700 1 --2- 172.21.2.112:0/3756472236 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff38c08bf50 0x7ff38c08c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:11.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:11.569 7ff390d79700 1 --2- 172.21.2.112:0/3756472236 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff38c1a4450 0x7ff38c1a4850 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:11.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:11.569 7ff390d79700 1 --2- 172.21.2.112:0/3756472236 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff38c0a0280 0x7ff38c0aeff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:11.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:11.569 7ff390d79700 1 -- 172.21.2.112:0/3756472236 >> 172.21.2.112:0/3756472236 conn(0x7ff38c0bbf60 msgr2=0x7ff38c105d10 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:28:11.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:11.570 7ff390d79700 1 -- 172.21.2.112:0/3756472236 shutdown_connections 2021-02-10T11:28:11.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:11.570 7ff390d79700 1 -- 172.21.2.112:0/3756472236 wait complete. 2021-02-10T11:28:11.566 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000348655 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000348655/43e93143-cdd6-4e63-a619-52619f736935 2021-02-10T11:28:11.566 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:11.567 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:11.567 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:11.904 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:28:11.904 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:11.905 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348655/43e93143-cdd6-4e63-a619-52619f736935/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:12.054 DEBUG:teuthology.orchestra.run.gibba012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:28:12.057 DEBUG:teuthology.orchestra.run.gibba020:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:28:12.286 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:28:12.286 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:28:12.286 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0227266 s, 46.1 MB/s 2021-02-10T11:28:12.287 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:12.287 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:12.287 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:12.623 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:28:12.623 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:12.623 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348655/43e93143-cdd6-4e63-a619-52619f736935/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:13.001 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:28:13.001 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:28:13.001 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0255048 s, 41.1 MB/s 2021-02-10T11:28:13.002 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:13.002 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:13.002 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:13.337 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:28:13.338 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:13.338 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348655/43e93143-cdd6-4e63-a619-52619f736935/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:13.714 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:28:13.715 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:28:13.715 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.02415 s, 43.4 MB/s 2021-02-10T11:28:13.715 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:13.716 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:13.716 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:14.052 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:28:14.052 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:14.053 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348655/43e93143-cdd6-4e63-a619-52619f736935/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:14.424 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:28:14.425 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:28:14.425 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0236567 s, 44.3 MB/s 2021-02-10T11:28:14.426 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:14.426 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:14.426 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:14.762 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:28:14.762 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:14.763 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348655/43e93143-cdd6-4e63-a619-52619f736935/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:15.139 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:28:15.139 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:28:15.139 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0243052 s, 43.1 MB/s 2021-02-10T11:28:15.140 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:15.140 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:15.140 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:15.476 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:28:15.477 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:15.477 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348655/43e93143-cdd6-4e63-a619-52619f736935/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:15.852 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:28:15.852 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:28:15.852 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0232227 s, 45.2 MB/s 2021-02-10T11:28:15.853 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:15.853 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:15.854 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:16.189 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:28:16.189 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:16.189 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348655/43e93143-cdd6-4e63-a619-52619f736935/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:16.565 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:28:16.565 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:28:16.565 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.022899 s, 45.8 MB/s 2021-02-10T11:28:16.566 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:16.566 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:16.566 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:16.901 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:28:16.902 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:16.902 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348655/43e93143-cdd6-4e63-a619-52619f736935/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:17.278 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:28:17.279 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:28:17.279 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0249276 s, 42.1 MB/s 2021-02-10T11:28:17.280 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:17.280 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:17.280 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:17.615 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:28:17.616 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:17.616 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348655/43e93143-cdd6-4e63-a619-52619f736935/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:17.991 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:28:17.991 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:28:17.991 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0234669 s, 44.7 MB/s 2021-02-10T11:28:17.992 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:17.992 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:17.992 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:18.329 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:28:18.329 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:18.329 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348655/43e93143-cdd6-4e63-a619-52619f736935/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:18.706 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:28:18.706 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:28:18.706 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0250533 s, 41.9 MB/s 2021-02-10T11:28:18.707 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000348656 2021-02-10T11:28:19.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.137 7f4601f77700 1 Processor -- start 2021-02-10T11:28:19.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.137 7f4601f77700 1 -- start start 2021-02-10T11:28:19.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.138 7f4601f77700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f45fc1b0470 0x7f45fc1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:19.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.138 7f4601f77700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45fc0f0b50 con 0x7f45fc0ef610 2021-02-10T11:28:19.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.138 7f4601f77700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45fc0f24e0 con 0x7f45fc0f20e0 2021-02-10T11:28:19.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.138 7f4601f77700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45fc0e5ea0 con 0x7f45fc1b0470 2021-02-10T11:28:19.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.138 7f45faffd700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f45fc0ef610 0x7f45fc0efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:45724/0 (socket says 172.21.2.112:45724) 2021-02-10T11:28:19.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.138 7f45faffd700 1 -- 172.21.2.112:0/2576408206 learned_addr learned my addr 172.21.2.112:0/2576408206 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:28:19.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.138 7f45fb7fe700 1 --2- 172.21.2.112:0/2576408206 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f45fc1b0470 0x7f45fc1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:19.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.138 7f45fa7fc700 1 -- 172.21.2.112:0/2576408206 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3514931569 0 0) 0x7f45e40010b0 con 0x7f45fc0ef610 2021-02-10T11:28:19.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.138 7f45fa7fc700 1 -- 172.21.2.112:0/2576408206 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45e8002fe0 con 0x7f45fc0ef610 2021-02-10T11:28:19.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.138 7f45fa7fc700 1 -- 172.21.2.112:0/2576408206 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2446802674 0 0) 0x7f45f4001400 con 0x7f45fc0f20e0 2021-02-10T11:28:19.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.138 7f45fa7fc700 1 -- 172.21.2.112:0/2576408206 --> v1:172.21.2.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45e8003210 con 0x7f45fc0f20e0 2021-02-10T11:28:19.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.138 7f45fa7fc700 1 -- 172.21.2.112:0/2576408206 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1142253737 0 0) 0x7f45e40010b0 con 0x7f45fc0ef610 2021-02-10T11:28:19.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.138 7f45fa7fc700 1 -- 172.21.2.112:0/2576408206 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f45fc1b0470 msgr2=0x7f45fc1b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:28:19.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.138 7f45fa7fc700 1 --2- 172.21.2.112:0/2576408206 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f45fc1b0470 0x7f45fc1b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:19.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.138 7f45fa7fc700 1 -- 172.21.2.112:0/2576408206 >> v1:172.21.2.120:6789/0 conn(0x7f45fc0f20e0 legacy=0x7f45fc0f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:19.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.138 7f45fa7fc700 1 -- 172.21.2.112:0/2576408206 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45fc0e52e0 con 0x7f45fc0ef610 2021-02-10T11:28:19.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.138 7f45fa7fc700 1 -- 172.21.2.112:0/2576408206 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f45e40015b0 con 0x7f45fc0ef610 2021-02-10T11:28:19.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.138 7f45fa7fc700 1 -- 172.21.2.112:0/2576408206 >> v1:172.21.2.112:6789/0 conn(0x7f45fc0ef610 legacy=0x7f45fc0efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:19.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.138 7f45fa7fc700 1 --2- 172.21.2.112:0/2576408206 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f45e8003bb0 0x7f45e8005fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:19.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.138 7f45fa7fc700 1 --2- 172.21.2.112:0/2576408206 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f45e8006790 0x7f45e8008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:19.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.138 7f45fa7fc700 1 --2- 172.21.2.112:0/2576408206 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f45e8009590 0x7f45e800ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:19.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.138 7f45fa7fc700 1 -- 172.21.2.112:0/2576408206 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45e80502e0 con 0x7f45e8003bb0 2021-02-10T11:28:19.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.138 7f45fb7fe700 1 --2- 172.21.2.112:0/2576408206 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f45e8003bb0 0x7f45e8005fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:19.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.138 7f45fa7fc700 1 -- 172.21.2.112:0/2576408206 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45e80504c0 con 0x7f45e8006790 2021-02-10T11:28:19.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.139 7f45fa7fc700 1 -- 172.21.2.112:0/2576408206 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45e80506e0 con 0x7f45e8009590 2021-02-10T11:28:19.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.139 7f45faffd700 1 --2- 172.21.2.112:0/2576408206 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f45e8006790 0x7f45e8008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:19.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.139 7f45fbfff700 1 --2- 172.21.2.112:0/2576408206 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f45e8009590 0x7f45e800ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:19.145 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.139 7f45fb7fe700 1 -- 172.21.2.112:0/2576408206 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f45e8009590 msgr2=0x7f45e800ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:28:19.145 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.139 7f45fb7fe700 1 --2- 172.21.2.112:0/2576408206 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f45e8009590 0x7f45e800ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:19.145 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.139 7f45fb7fe700 1 -- 172.21.2.112:0/2576408206 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f45e8006790 msgr2=0x7f45e8008c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:28:19.145 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.139 7f45fb7fe700 1 --2- 172.21.2.112:0/2576408206 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f45e8006790 0x7f45e8008c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:19.145 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.139 7f45fb7fe700 1 -- 172.21.2.112:0/2576408206 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45e8050980 con 0x7f45e8003bb0 2021-02-10T11:28:19.145 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.139 7f45fb7fe700 1 --2- 172.21.2.112:0/2576408206 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f45e8003bb0 0x7f45e8005fc0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 rx=0x7f45ec006ac0 tx=0x7f45ec009ab0).ready entity=mon.0 client_cookie=17ee8635f9a0c7cf server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:19.145 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.139 7f45fa7fc700 1 -- 172.21.2.112:0/2576408206 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f45ec00c0c0 con 0x7f45e8003bb0 2021-02-10T11:28:19.145 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.139 7f45fa7fc700 1 -- 172.21.2.112:0/2576408206 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45ec00c3c0 con 0x7f45e8003bb0 2021-02-10T11:28:19.145 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.139 7f45fa7fc700 1 -- 172.21.2.112:0/2576408206 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f45ec00c8a0 con 0x7f45e8003bb0 2021-02-10T11:28:19.145 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.139 7f4601f77700 1 -- 172.21.2.112:0/2576408206 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f45e8003bb0 msgr2=0x7f45e8005fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:19.145 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.139 7f4601f77700 1 --2- 172.21.2.112:0/2576408206 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f45e8003bb0 0x7f45e8005fc0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 rx=0x7f45ec006ac0 tx=0x7f45ec009ab0).stop 2021-02-10T11:28:19.145 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.139 7f45fbfff700 1 -- 172.21.2.112:0/2576408206 reap_dead start 2021-02-10T11:28:19.145 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.140 7f4601f77700 1 -- 172.21.2.112:0/2576408206 shutdown_connections 2021-02-10T11:28:19.146 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.140 7f4601f77700 1 -- 172.21.2.112:0/2576408206 >> 172.21.2.112:0/2576408206 conn(0x7f45fc0ce3e0 msgr2=0x7f45fc0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:28:19.146 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.140 7f4601f77700 1 -- 172.21.2.112:0/2576408206 shutdown_connections 2021-02-10T11:28:19.146 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.140 7f4601f77700 1 -- 172.21.2.112:0/2576408206 wait complete. 2021-02-10T11:28:19.146 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.154 7f4601f77700 1 Processor -- start 2021-02-10T11:28:19.146 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.154 7f4601f77700 1 -- start start 2021-02-10T11:28:19.147 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.155 7f4601f77700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f45fc0b7db0 0x7f45fc0b8260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:19.147 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.155 7f4601f77700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f45fc0b8880 0x7f45fc1a7450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:19.148 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.155 7f4601f77700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f45fc0b8c80 0x7f45fc1a7a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:19.148 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.155 7f4601f77700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45fc0b79b0 con 0x7f45fc0b8880 2021-02-10T11:28:19.148 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.155 7f4601f77700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45fc1a8090 con 0x7f45fc0b7db0 2021-02-10T11:28:19.148 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.155 7f4601f77700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45fc1a82b0 con 0x7f45fc0b8c80 2021-02-10T11:28:19.148 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.155 7f45faffd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f45fc0b8880 0x7f45fc1a7450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:19.148 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.155 7f45faffd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f45fc0b8880 0x7f45fc1a7450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:46272/0 (socket says 172.21.2.112:46272) 2021-02-10T11:28:19.148 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.155 7f45faffd700 1 -- 172.21.2.112:0/3213115232 learned_addr learned my addr 172.21.2.112:0/3213115232 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:28:19.148 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.155 7f45fb7fe700 1 --2- 172.21.2.112:0/3213115232 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f45fc0b7db0 0x7f45fc0b8260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:19.148 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.155 7f45fbfff700 1 --2- 172.21.2.112:0/3213115232 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f45fc0b8c80 0x7f45fc1a7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:19.148 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.155 7f45faffd700 1 -- 172.21.2.112:0/3213115232 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f45fc0b8c80 msgr2=0x7f45fc1a7a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:19.148 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.155 7f45faffd700 1 --2- 172.21.2.112:0/3213115232 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f45fc0b8c80 0x7f45fc1a7a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:19.148 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.155 7f45faffd700 1 -- 172.21.2.112:0/3213115232 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f45fc0b7db0 msgr2=0x7f45fc0b8260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:19.148 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.155 7f45faffd700 1 --2- 172.21.2.112:0/3213115232 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f45fc0b7db0 0x7f45fc0b8260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:19.149 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.155 7f45faffd700 1 -- 172.21.2.112:0/3213115232 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45fc190c10 con 0x7f45fc0b8880 2021-02-10T11:28:19.149 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.155 7f45fbfff700 1 --2- 172.21.2.112:0/3213115232 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f45fc0b8c80 0x7f45fc1a7a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:28:19.149 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.155 7f45fb7fe700 1 --2- 172.21.2.112:0/3213115232 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f45fc0b7db0 0x7f45fc0b8260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:28:19.149 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.156 7f45faffd700 1 --2- 172.21.2.112:0/3213115232 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f45fc0b8880 0x7f45fc1a7450 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 rx=0x7f45e4001c60 tx=0x7f45e400a8f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:19.149 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.156 7f45f3fff700 1 -- 172.21.2.112:0/3213115232 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f45e400ace0 con 0x7f45fc0b8880 2021-02-10T11:28:19.149 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.156 7f45f3fff700 1 -- 172.21.2.112:0/3213115232 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45e4004350 con 0x7f45fc0b8880 2021-02-10T11:28:19.149 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.156 7f45f3fff700 1 -- 172.21.2.112:0/3213115232 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f45e40047a0 con 0x7f45fc0b8880 2021-02-10T11:28:19.149 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.156 7f4601f77700 1 -- 172.21.2.112:0/3213115232 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45fc190c10 con 0x7f45fc0b8880 2021-02-10T11:28:19.149 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.156 7f4601f77700 1 -- 172.21.2.112:0/3213115232 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45fc190c10 con 0x7f45fc0b8880 2021-02-10T11:28:19.149 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.156 7f45f3fff700 1 -- 172.21.2.112:0/3213115232 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f45e4004530 con 0x7f45fc0b8880 2021-02-10T11:28:19.149 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.156 7f45f3fff700 1 --2- 172.21.2.112:0/3213115232 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f45e0018730 0x7f45e001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:19.149 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.156 7f45f3fff700 1 -- 172.21.2.112:0/3213115232 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f45e40327a0 con 0x7f45fc0b8880 2021-02-10T11:28:19.149 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.157 7f45fb7fe700 1 --2- 172.21.2.112:0/3213115232 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f45e0018730 0x7f45e001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:19.150 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.157 7f45fb7fe700 1 --2- 172.21.2.112:0/3213115232 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f45e0018730 0x7f45e001abc0 crc :-1 s=READY pgs=351 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:19.176 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.188 7f4601f77700 1 -- 172.21.2.112:0/3213115232 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45fc105b30 con 0x7f45fc0b8880 2021-02-10T11:28:19.178 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.190 7f45f3fff700 1 -- 172.21.2.112:0/3213115232 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f45e4004140 con 0x7f45fc0b8880 2021-02-10T11:28:19.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.316 7f4601f77700 1 -- 172.21.2.112:0/3213115232 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348656", "target": ["mgr", ""]}) v1 -- 0x7f45fc1a8090 con 0x7f45e0018730 2021-02-10T11:28:19.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.342 7f45f3fff700 1 -- 172.21.2.112:0/3213115232 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f45ec0092f0 con 0x7f45e0018730 2021-02-10T11:28:19.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.349 7f4601f77700 1 -- 172.21.2.112:0/3213115232 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f45e0018730 msgr2=0x7f45e001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:19.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.349 7f4601f77700 1 --2- 172.21.2.112:0/3213115232 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f45e0018730 0x7f45e001abc0 crc :-1 s=READY pgs=351 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:19.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.349 7f4601f77700 1 -- 172.21.2.112:0/3213115232 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f45fc0b8880 msgr2=0x7f45fc1a7450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:19.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.349 7f4601f77700 1 --2- 172.21.2.112:0/3213115232 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f45fc0b8880 0x7f45fc1a7450 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 rx=0x7f45e4001c60 tx=0x7f45e400a8f0).stop 2021-02-10T11:28:19.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.349 7f4601f77700 1 -- 172.21.2.112:0/3213115232 shutdown_connections 2021-02-10T11:28:19.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.349 7f4601f77700 1 --2- 172.21.2.112:0/3213115232 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f45e0018730 0x7f45e001abc0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:19.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.349 7f4601f77700 1 --2- 172.21.2.112:0/3213115232 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f45fc0b8c80 0x7f45fc1a7a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:19.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.349 7f4601f77700 1 --2- 172.21.2.112:0/3213115232 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f45fc0b8880 0x7f45fc1a7450 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:19.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.349 7f4601f77700 1 --2- 172.21.2.112:0/3213115232 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f45fc0b7db0 0x7f45fc0b8260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:19.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.349 7f4601f77700 1 -- 172.21.2.112:0/3213115232 >> 172.21.2.112:0/3213115232 conn(0x7f45fc094e10 msgr2=0x7f45fc0a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:28:19.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.349 7f4601f77700 1 -- 172.21.2.112:0/3213115232 shutdown_connections 2021-02-10T11:28:19.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.349 7f4601f77700 1 -- 172.21.2.112:0/3213115232 wait complete. 2021-02-10T11:28:19.346 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000348656 2021-02-10T11:28:19.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.775 7f19af30f700 1 Processor -- start 2021-02-10T11:28:19.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.775 7f19af30f700 1 -- start start 2021-02-10T11:28:19.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.776 7f19af30f700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f19a81b0470 0x7f19a81b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:19.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.776 7f19af30f700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f19a80ef610 0x7f19a80efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:19.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.776 7f19af30f700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f19a80f0030 0x7f19a80f0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:19.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.776 7f19af30f700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19a80e5ea0 con 0x7f19a81b0470 2021-02-10T11:28:19.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.776 7f19af30f700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19a80f0b50 con 0x7f19a80f0030 2021-02-10T11:28:19.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.776 7f19af30f700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19a80e52e0 con 0x7f19a80ef610 2021-02-10T11:28:19.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.776 7f19ad12b700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f19a81b0470 0x7f19a81b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:19.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.776 7f19ad12b700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f19a81b0470 0x7f19a81b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:46280/0 (socket says 172.21.2.112:46280) 2021-02-10T11:28:19.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.776 7f19ad12b700 1 -- 172.21.2.112:0/4293459966 learned_addr learned my addr 172.21.2.112:0/4293459966 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:28:19.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.776 7f19ad12b700 1 -- 172.21.2.112:0/4293459966 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f19a80ef610 msgr2=0x7f19a80efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:28:19.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.776 7f19ad12b700 1 --2- 172.21.2.112:0/4293459966 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f19a80ef610 0x7f19a80efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:19.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.776 7f19ad12b700 1 -- 172.21.2.112:0/4293459966 >> v2:172.21.2.120:3300/0 conn(0x7f19a80f0030 msgr2=0x7f19a80f0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:28:19.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.776 7f19ad12b700 1 --2- 172.21.2.112:0/4293459966 >> v2:172.21.2.120:3300/0 conn(0x7f19a80f0030 0x7f19a80f0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:19.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.776 7f19ad12b700 1 -- 172.21.2.112:0/4293459966 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19a80f10d0 con 0x7f19a81b0470 2021-02-10T11:28:19.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.777 7f19ad12b700 1 --2- 172.21.2.112:0/4293459966 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f19a81b0470 0x7f19a81b0870 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 rx=0x7f19a4006700 tx=0x7f19a4009b80).ready entity=mon.0 client_cookie=cfe8455c74a5c7fd server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:19.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.777 7f199bfff700 1 -- 172.21.2.112:0/4293459966 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f19a400c0c0 con 0x7f19a81b0470 2021-02-10T11:28:19.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.777 7f199bfff700 1 -- 172.21.2.112:0/4293459966 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19a400c440 con 0x7f19a81b0470 2021-02-10T11:28:19.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.777 7f199bfff700 1 -- 172.21.2.112:0/4293459966 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f19a400c960 con 0x7f19a81b0470 2021-02-10T11:28:19.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.777 7f19af30f700 1 -- 172.21.2.112:0/4293459966 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f19a81b0470 msgr2=0x7f19a81b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:19.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.777 7f19af30f700 1 --2- 172.21.2.112:0/4293459966 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f19a81b0470 0x7f19a81b0870 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 rx=0x7f19a4006700 tx=0x7f19a4009b80).stop 2021-02-10T11:28:19.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.777 7f19af30f700 1 -- 172.21.2.112:0/4293459966 shutdown_connections 2021-02-10T11:28:19.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.777 7f19af30f700 1 --2- 172.21.2.112:0/4293459966 >> v2:172.21.2.120:3300/0 conn(0x7f19a80f0030 0x7f19a80f0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:19.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.777 7f19af30f700 1 --2- 172.21.2.112:0/4293459966 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f19a80ef610 0x7f19a80efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:19.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.777 7f19af30f700 1 --2- 172.21.2.112:0/4293459966 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f19a81b0470 0x7f19a81b0870 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:19.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.777 7f19af30f700 1 -- 172.21.2.112:0/4293459966 >> 172.21.2.112:0/4293459966 conn(0x7f19a80ce3e0 msgr2=0x7f19a80e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:28:19.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.777 7f19af30f700 1 -- 172.21.2.112:0/4293459966 shutdown_connections 2021-02-10T11:28:19.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.777 7f19af30f700 1 -- 172.21.2.112:0/4293459966 wait complete. 2021-02-10T11:28:19.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.792 7f19af30f700 1 Processor -- start 2021-02-10T11:28:19.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.792 7f19af30f700 1 -- start start 2021-02-10T11:28:19.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.792 7f19af30f700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f19a8190920 0x7f19a808f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:19.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.792 7f19af30f700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f19a8190d20 0x7f19a808fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:19.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.792 7f19af30f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f19a8193270 0x7f19a8193670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:19.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.792 7f19af30f700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19a8090470 con 0x7f19a8193270 2021-02-10T11:28:19.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.792 7f19af30f700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19a8193c90 con 0x7f19a8190d20 2021-02-10T11:28:19.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.792 7f19af30f700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19a8193e70 con 0x7f19a8190920 2021-02-10T11:28:19.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.792 7f19ad92c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f19a8193270 0x7f19a8193670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:19.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.792 7f19ad92c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f19a8193270 0x7f19a8193670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:46288/0 (socket says 172.21.2.112:46288) 2021-02-10T11:28:19.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.792 7f19ad92c700 1 -- 172.21.2.112:0/1399448852 learned_addr learned my addr 172.21.2.112:0/1399448852 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:28:19.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.793 7f19ad92c700 1 -- 172.21.2.112:0/1399448852 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f19a8190920 msgr2=0x7f19a808f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:19.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.793 7f19ad92c700 1 --2- 172.21.2.112:0/1399448852 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f19a8190920 0x7f19a808f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:19.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.793 7f19ad92c700 1 -- 172.21.2.112:0/1399448852 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f19a8190d20 msgr2=0x7f19a808fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:19.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.793 7f19ad92c700 1 --2- 172.21.2.112:0/1399448852 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f19a8190d20 0x7f19a808fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:19.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.793 7f19ad92c700 1 -- 172.21.2.112:0/1399448852 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19a8194090 con 0x7f19a8193270 2021-02-10T11:28:19.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.793 7f19ad92c700 1 --2- 172.21.2.112:0/1399448852 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f19a8193270 0x7f19a8193670 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 rx=0x7f19a0009980 tx=0x7f19a000bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:19.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.793 7f1999ffb700 1 -- 172.21.2.112:0/1399448852 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f19a0003020 con 0x7f19a8193270 2021-02-10T11:28:19.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.793 7f1999ffb700 1 -- 172.21.2.112:0/1399448852 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19a0005a80 con 0x7f19a8193270 2021-02-10T11:28:19.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.793 7f19af30f700 1 -- 172.21.2.112:0/1399448852 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19a8194090 con 0x7f19a8193270 2021-02-10T11:28:19.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.793 7f19af30f700 1 -- 172.21.2.112:0/1399448852 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19a81920b0 con 0x7f19a8193270 2021-02-10T11:28:19.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.793 7f1999ffb700 1 -- 172.21.2.112:0/1399448852 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f19a0006030 con 0x7f19a8193270 2021-02-10T11:28:19.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.794 7f1999ffb700 1 -- 172.21.2.112:0/1399448852 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f19a0006030 con 0x7f19a8193270 2021-02-10T11:28:19.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.794 7f1999ffb700 1 --2- 172.21.2.112:0/1399448852 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f19880186c0 0x7f198801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:19.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.794 7f1999ffb700 1 -- 172.21.2.112:0/1399448852 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f19a00328d0 con 0x7f19a8193270 2021-02-10T11:28:19.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.794 7f19ad12b700 1 --2- 172.21.2.112:0/1399448852 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f19880186c0 0x7f198801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:19.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.794 7f19ad12b700 1 --2- 172.21.2.112:0/1399448852 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f19880186c0 0x7f198801abc0 crc :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:19.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.826 7f19af30f700 1 -- 172.21.2.112:0/1399448852 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19a8105af0 con 0x7f19a8193270 2021-02-10T11:28:19.816 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.828 7f1999ffb700 1 -- 172.21.2.112:0/1399448852 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f19a0002f00 con 0x7f19a8193270 2021-02-10T11:28:19.946 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.958 7f19af30f700 1 -- 172.21.2.112:0/1399448852 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348656", "target": ["mgr", ""]}) v1 -- 0x7f19a8090470 con 0x7f19880186c0 2021-02-10T11:28:19.949 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.961 7f1999ffb700 1 -- 172.21.2.112:0/1399448852 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f19a4000970 con 0x7f19880186c0 2021-02-10T11:28:19.949 INFO:teuthology.orchestra.run.gibba012.stdout:/volumes/_nogroup/subvolume_0000000000348656/06d8cd07-54d0-4360-81bb-099bab794606 2021-02-10T11:28:19.951 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.963 7f19af30f700 1 -- 172.21.2.112:0/1399448852 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f19880186c0 msgr2=0x7f198801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:19.951 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.963 7f19af30f700 1 --2- 172.21.2.112:0/1399448852 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f19880186c0 0x7f198801abc0 crc :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:19.951 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.963 7f19af30f700 1 -- 172.21.2.112:0/1399448852 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f19a8193270 msgr2=0x7f19a8193670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:19.951 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.963 7f19af30f700 1 --2- 172.21.2.112:0/1399448852 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f19a8193270 0x7f19a8193670 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 rx=0x7f19a0009980 tx=0x7f19a000bcd0).stop 2021-02-10T11:28:19.952 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.963 7f19af30f700 1 -- 172.21.2.112:0/1399448852 shutdown_connections 2021-02-10T11:28:19.952 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.963 7f19af30f700 1 --2- 172.21.2.112:0/1399448852 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f19880186c0 0x7f198801abc0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:19.952 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.963 7f19af30f700 1 --2- 172.21.2.112:0/1399448852 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f19a8190d20 0x7f19a808fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:19.952 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.963 7f19af30f700 1 --2- 172.21.2.112:0/1399448852 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f19a8193270 0x7f19a8193670 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:19.952 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.963 7f19af30f700 1 --2- 172.21.2.112:0/1399448852 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f19a8190920 0x7f19a808f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:19.952 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.963 7f19af30f700 1 -- 172.21.2.112:0/1399448852 >> 172.21.2.112:0/1399448852 conn(0x7f19a80bf4e0 msgr2=0x7f19a80be3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:28:19.952 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.963 7f19af30f700 1 -- 172.21.2.112:0/1399448852 shutdown_connections 2021-02-10T11:28:19.952 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:19.963 7f19af30f700 1 -- 172.21.2.112:0/1399448852 wait complete. 2021-02-10T11:28:19.959 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000348656 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000348656/06d8cd07-54d0-4360-81bb-099bab794606 2021-02-10T11:28:19.959 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:19.959 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:19.959 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:20.294 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:28:20.295 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:20.295 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348656/06d8cd07-54d0-4360-81bb-099bab794606/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:20.678 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:28:20.678 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:28:20.678 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.025777 s, 40.7 MB/s 2021-02-10T11:28:20.679 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:20.679 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:20.679 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:21.014 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:28:21.015 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:21.015 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348656/06d8cd07-54d0-4360-81bb-099bab794606/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:21.390 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:28:21.391 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:28:21.391 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0245398 s, 42.7 MB/s 2021-02-10T11:28:21.391 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:21.392 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:21.392 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:21.727 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:28:21.728 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:21.728 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348656/06d8cd07-54d0-4360-81bb-099bab794606/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:22.105 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:28:22.105 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:28:22.105 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0250417 s, 41.9 MB/s 2021-02-10T11:28:22.107 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:22.107 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:22.107 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:22.443 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:28:22.444 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:22.444 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348656/06d8cd07-54d0-4360-81bb-099bab794606/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:22.820 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:28:22.821 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:28:22.821 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.024523 s, 42.8 MB/s 2021-02-10T11:28:22.822 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:22.822 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:22.822 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:23.159 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:28:23.159 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:23.160 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348656/06d8cd07-54d0-4360-81bb-099bab794606/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:23.535 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:28:23.535 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:28:23.535 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0227193 s, 46.2 MB/s 2021-02-10T11:28:23.536 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:23.537 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:23.537 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:23.898 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:28:23.898 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:23.899 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348656/06d8cd07-54d0-4360-81bb-099bab794606/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:24.284 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:28:24.284 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:28:24.284 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0223981 s, 46.8 MB/s 2021-02-10T11:28:24.285 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:24.285 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:24.285 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:24.624 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:28:24.624 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:24.625 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348656/06d8cd07-54d0-4360-81bb-099bab794606/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:25.001 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:28:25.001 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:28:25.002 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0273129 s, 38.4 MB/s 2021-02-10T11:28:25.002 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:25.003 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:25.003 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:25.338 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:28:25.338 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:25.338 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348656/06d8cd07-54d0-4360-81bb-099bab794606/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:25.715 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:28:25.716 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:28:25.716 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0248426 s, 42.2 MB/s 2021-02-10T11:28:25.717 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:25.717 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:25.717 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:26.052 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:28:26.052 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:26.053 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348656/06d8cd07-54d0-4360-81bb-099bab794606/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:26.427 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:28:26.428 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:28:26.428 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0239764 s, 43.7 MB/s 2021-02-10T11:28:26.428 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:26.429 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:26.429 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:26.765 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:28:26.765 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:26.765 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348656/06d8cd07-54d0-4360-81bb-099bab794606/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:27.139 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:28:27.140 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:28:27.140 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0224687 s, 46.7 MB/s 2021-02-10T11:28:27.141 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000348657 2021-02-10T11:28:27.574 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:27.570 7f42663f5700 1 Processor -- start 2021-02-10T11:28:27.574 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:27.570 7f42663f5700 1 -- start start 2021-02-10T11:28:27.575 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:27.570 7f42663f5700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f42601b0470 0x7f42601b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:27.575 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:27.570 7f42663f5700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f42600ef610 0x7f42600efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:27.575 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:27.570 7f42663f5700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f42600f0030 0x7f42600f0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:27.575 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:27.570 7f42663f5700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42600e5ea0 con 0x7f42601b0470 2021-02-10T11:28:27.575 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:27.570 7f42663f5700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42600f0b50 con 0x7f42600ef610 2021-02-10T11:28:27.575 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:27.570 7f42663f5700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42600e52e0 con 0x7f42600f0030 2021-02-10T11:28:27.575 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:27.571 7f425ffff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f42601b0470 0x7f42601b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:27.575 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:27.571 7f425ffff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f42601b0470 0x7f42601b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:46292/0 (socket says 172.21.2.112:46292) 2021-02-10T11:28:27.575 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:27.571 7f425ffff700 1 -- 172.21.2.112:0/4022115145 learned_addr learned my addr 172.21.2.112:0/4022115145 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:28:27.575 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:27.571 7f425ffff700 1 -- 172.21.2.112:0/4022115145 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f42600f0030 msgr2=0x7f42600f0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:28:27.575 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:27.571 7f425ffff700 1 --2- 172.21.2.112:0/4022115145 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f42600f0030 0x7f42600f0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:27.575 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:27.571 7f425ffff700 1 -- 172.21.2.112:0/4022115145 >> v2:172.21.2.120:3300/0 conn(0x7f42600ef610 msgr2=0x7f42600efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:28:27.575 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:27.571 7f425ffff700 1 --2- 172.21.2.112:0/4022115145 >> v2:172.21.2.120:3300/0 conn(0x7f42600ef610 0x7f42600efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:27.576 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:27.571 7f425ffff700 1 -- 172.21.2.112:0/4022115145 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42600f10d0 con 0x7f42601b0470 2021-02-10T11:28:27.576 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:27.571 7f425ffff700 1 --2- 172.21.2.112:0/4022115145 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f42601b0470 0x7f42601b0870 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 rx=0x7f4250006700 tx=0x7f4250009b80).ready entity=mon.0 client_cookie=857ef77a8f598d63 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:27.576 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:27.571 7f425effd700 1 -- 172.21.2.112:0/4022115145 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f425000c0c0 con 0x7f42601b0470 2021-02-10T11:28:27.576 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:27.571 7f425effd700 1 -- 172.21.2.112:0/4022115145 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f425000c340 con 0x7f42601b0470 2021-02-10T11:28:27.576 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:27.571 7f425effd700 1 -- 172.21.2.112:0/4022115145 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f425000c820 con 0x7f42601b0470 2021-02-10T11:28:27.576 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:27.571 7f42663f5700 1 -- 172.21.2.112:0/4022115145 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f42601b0470 msgr2=0x7f42601b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:27.576 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:27.571 7f42663f5700 1 --2- 172.21.2.112:0/4022115145 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f42601b0470 0x7f42601b0870 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 rx=0x7f4250006700 tx=0x7f4250009b80).stop 2021-02-10T11:28:27.576 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:27.572 7f42663f5700 1 -- 172.21.2.112:0/4022115145 shutdown_connections 2021-02-10T11:28:27.576 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:27.572 7f42663f5700 1 --2- 172.21.2.112:0/4022115145 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f42600f0030 0x7f42600f0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:27.576 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:27.572 7f42663f5700 1 --2- 172.21.2.112:0/4022115145 >> v2:172.21.2.120:3300/0 conn(0x7f42600ef610 0x7f42600efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:27.576 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:27.572 7f42663f5700 1 --2- 172.21.2.112:0/4022115145 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f42601b0470 0x7f42601b0870 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:27.576 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:27.572 7f42663f5700 1 -- 172.21.2.112:0/4022115145 >> 172.21.2.112:0/4022115145 conn(0x7f42600ce3e0 msgr2=0x7f42600e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:28:27.576 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:27.572 7f42663f5700 1 -- 172.21.2.112:0/4022115145 shutdown_connections 2021-02-10T11:28:27.577 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:27.572 7f42663f5700 1 -- 172.21.2.112:0/4022115145 wait complete. 2021-02-10T11:28:27.578 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:27.587 7f42663f5700 1 Processor -- start 2021-02-10T11:28:27.578 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:27.587 7f42663f5700 1 -- start start 2021-02-10T11:28:27.578 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:27.587 7f42663f5700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f42600b20b0 0x7f42600ac970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:27.578 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:27.587 7f42663f5700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f42600b24b0 0x7f4260191dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:27.578 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:27.587 7f42663f5700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f42600acf90 0x7f42601923e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:27.578 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:27.587 7f42663f5700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42600b1b30 con 0x7f42600b20b0 2021-02-10T11:28:27.578 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:27.587 7f42663f5700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42601a04e0 con 0x7f42600b24b0 2021-02-10T11:28:27.578 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:27.587 7f42663f5700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42601a06c0 con 0x7f42600acf90 2021-02-10T11:28:27.578 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:27.588 7f425ffff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f42600b20b0 0x7f42600ac970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:27.579 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:27.588 7f425ffff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f42600b20b0 0x7f42600ac970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:46294/0 (socket says 172.21.2.112:46294) 2021-02-10T11:28:27.579 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:27.588 7f425ffff700 1 -- 172.21.2.112:0/4243393159 learned_addr learned my addr 172.21.2.112:0/4243393159 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:28:27.579 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:27.588 7f425ffff700 1 -- 172.21.2.112:0/4243393159 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f42600acf90 msgr2=0x7f42601923e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:27.579 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:27.588 7f425ffff700 1 --2- 172.21.2.112:0/4243393159 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f42600acf90 0x7f42601923e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:27.579 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:27.588 7f425ffff700 1 -- 172.21.2.112:0/4243393159 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f42600b24b0 msgr2=0x7f4260191dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:27.579 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:27.588 7f425ffff700 1 --2- 172.21.2.112:0/4243393159 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f42600b24b0 0x7f4260191dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:27.579 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:27.588 7f425ffff700 1 -- 172.21.2.112:0/4243393159 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4260192be0 con 0x7f42600b20b0 2021-02-10T11:28:27.579 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:27.588 7f425ffff700 1 --2- 172.21.2.112:0/4243393159 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f42600b20b0 0x7f42600ac970 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 rx=0x7f425000b5e0 tx=0x7f425000b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:27.579 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:27.588 7f425cff9700 1 -- 172.21.2.112:0/4243393159 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f425000bd70 con 0x7f42600b20b0 2021-02-10T11:28:27.579 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:27.588 7f425cff9700 1 -- 172.21.2.112:0/4243393159 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4250002030 con 0x7f42600b20b0 2021-02-10T11:28:27.579 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:27.588 7f42663f5700 1 -- 172.21.2.112:0/4243393159 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4260192be0 con 0x7f42600b20b0 2021-02-10T11:28:27.579 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:27.588 7f42663f5700 1 -- 172.21.2.112:0/4243393159 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4260193170 con 0x7f42600b20b0 2021-02-10T11:28:27.579 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:27.588 7f425cff9700 1 -- 172.21.2.112:0/4243393159 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4250002550 con 0x7f42600b20b0 2021-02-10T11:28:27.580 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:27.589 7f425cff9700 1 -- 172.21.2.112:0/4243393159 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4250000c90 con 0x7f42600b20b0 2021-02-10T11:28:27.580 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:27.589 7f425cff9700 1 --2- 172.21.2.112:0/4243393159 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f42400186c0 0x7f424001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:27.580 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:27.589 7f425cff9700 1 -- 172.21.2.112:0/4243393159 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f4250032810 con 0x7f42600b20b0 2021-02-10T11:28:27.580 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:27.590 7f425f7fe700 1 --2- 172.21.2.112:0/4243393159 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f42400186c0 0x7f424001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:27.580 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:27.590 7f425f7fe700 1 --2- 172.21.2.112:0/4243393159 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f42400186c0 0x7f424001abc0 crc :-1 s=READY pgs=353 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:27.609 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:27.621 7f42663f5700 1 -- 172.21.2.112:0/4243393159 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4260105af0 con 0x7f42600b20b0 2021-02-10T11:28:27.612 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:27.623 7f425cff9700 1 -- 172.21.2.112:0/4243393159 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4250000940 con 0x7f42600b20b0 2021-02-10T11:28:27.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:27.750 7f42663f5700 1 -- 172.21.2.112:0/4243393159 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348657", "target": ["mgr", ""]}) v1 -- 0x7f42600b1b30 con 0x7f42400186c0 2021-02-10T11:28:27.765 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:27.776 7f425cff9700 1 -- 172.21.2.112:0/4243393159 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f42540009b0 con 0x7f42400186c0 2021-02-10T11:28:27.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:27.783 7f42663f5700 1 -- 172.21.2.112:0/4243393159 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f42400186c0 msgr2=0x7f424001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:27.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:27.783 7f42663f5700 1 --2- 172.21.2.112:0/4243393159 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f42400186c0 0x7f424001abc0 crc :-1 s=READY pgs=353 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:27.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:27.783 7f42663f5700 1 -- 172.21.2.112:0/4243393159 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f42600b20b0 msgr2=0x7f42600ac970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:27.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:27.783 7f42663f5700 1 --2- 172.21.2.112:0/4243393159 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f42600b20b0 0x7f42600ac970 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 rx=0x7f425000b5e0 tx=0x7f425000b980).stop 2021-02-10T11:28:27.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:27.783 7f42663f5700 1 -- 172.21.2.112:0/4243393159 shutdown_connections 2021-02-10T11:28:27.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:27.783 7f42663f5700 1 --2- 172.21.2.112:0/4243393159 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f42400186c0 0x7f424001abc0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:27.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:27.783 7f42663f5700 1 --2- 172.21.2.112:0/4243393159 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f42600b24b0 0x7f4260191dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:27.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:27.783 7f42663f5700 1 --2- 172.21.2.112:0/4243393159 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f42600acf90 0x7f42601923e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:27.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:27.783 7f42663f5700 1 --2- 172.21.2.112:0/4243393159 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f42600b20b0 0x7f42600ac970 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:27.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:27.783 7f42663f5700 1 -- 172.21.2.112:0/4243393159 >> 172.21.2.112:0/4243393159 conn(0x7f42600bf4e0 msgr2=0x7f42600be3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:28:27.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:27.784 7f42663f5700 1 -- 172.21.2.112:0/4243393159 shutdown_connections 2021-02-10T11:28:27.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:27.784 7f42663f5700 1 -- 172.21.2.112:0/4243393159 wait complete. 2021-02-10T11:28:27.780 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000348657 2021-02-10T11:28:28.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:28.208 7f72e74ab700 1 Processor -- start 2021-02-10T11:28:28.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:28.208 7f72e74ab700 1 -- start start 2021-02-10T11:28:28.213 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:28.208 7f72e74ab700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f72e01b0470 0x7f72e01b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:28.213 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:28.208 7f72e74ab700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f72e00f20e0 0x7f72e00f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:28.213 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:28.208 7f72e74ab700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72e00f0b50 con 0x7f72e00ef610 2021-02-10T11:28:28.213 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:28.208 7f72e74ab700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72e00f0820 con 0x7f72e01b0470 2021-02-10T11:28:28.213 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:28.208 7f72e74ab700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72e00f24e0 con 0x7f72e00f20e0 2021-02-10T11:28:28.213 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:28.209 7f72e52c7700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f72e01b0470 0x7f72e01b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:28.213 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:28.209 7f72e52c7700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f72e01b0470 0x7f72e01b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:46302/0 (socket says 172.21.2.112:46302) 2021-02-10T11:28:28.213 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:28.209 7f72e52c7700 1 -- 172.21.2.112:0/2977605874 learned_addr learned my addr 172.21.2.112:0/2977605874 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:28:28.213 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:28.209 7f72e52c7700 1 -- 172.21.2.112:0/2977605874 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f72e00f20e0 msgr2=0x7f72e00f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:28:28.213 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:28.209 7f72e52c7700 1 --2- 172.21.2.112:0/2977605874 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f72e00f20e0 0x7f72e00f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:28.213 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:28.209 7f72e52c7700 1 -- 172.21.2.112:0/2977605874 >> v1:172.21.2.120:6789/0 conn(0x7f72e00ef610 legacy=0x7f72e00efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:28:28.213 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:28.209 7f72e52c7700 1 -- 172.21.2.112:0/2977605874 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72e00e52e0 con 0x7f72e01b0470 2021-02-10T11:28:28.214 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:28.209 7f72e52c7700 1 --2- 172.21.2.112:0/2977605874 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f72e01b0470 0x7f72e01b0870 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 rx=0x7f72dc006700 tx=0x7f72dc009b80).ready entity=mon.0 client_cookie=48e5956e0dd5cd2b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:28.214 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:28.209 7f72d3fff700 1 -- 172.21.2.112:0/2977605874 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f72dc00c0c0 con 0x7f72e01b0470 2021-02-10T11:28:28.214 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:28.209 7f72d3fff700 1 -- 172.21.2.112:0/2977605874 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72dc00c340 con 0x7f72e01b0470 2021-02-10T11:28:28.214 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:28.209 7f72d3fff700 1 -- 172.21.2.112:0/2977605874 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f72dc00c860 con 0x7f72e01b0470 2021-02-10T11:28:28.214 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:28.209 7f72e74ab700 1 -- 172.21.2.112:0/2977605874 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f72e01b0470 msgr2=0x7f72e01b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:28.214 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:28.209 7f72e74ab700 1 --2- 172.21.2.112:0/2977605874 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f72e01b0470 0x7f72e01b0870 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 rx=0x7f72dc006700 tx=0x7f72dc009b80).stop 2021-02-10T11:28:28.214 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:28.210 7f72e74ab700 1 -- 172.21.2.112:0/2977605874 shutdown_connections 2021-02-10T11:28:28.214 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:28.210 7f72e74ab700 1 --2- 172.21.2.112:0/2977605874 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f72e00f20e0 0x7f72e00f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:28.214 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:28.210 7f72e74ab700 1 --2- 172.21.2.112:0/2977605874 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f72e01b0470 0x7f72e01b0870 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:28.214 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:28.210 7f72e74ab700 1 -- 172.21.2.112:0/2977605874 >> 172.21.2.112:0/2977605874 conn(0x7f72e00ce3e0 msgr2=0x7f72e00e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:28:28.214 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:28.210 7f72e74ab700 1 -- 172.21.2.112:0/2977605874 shutdown_connections 2021-02-10T11:28:28.214 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:28.210 7f72e74ab700 1 -- 172.21.2.112:0/2977605874 wait complete. 2021-02-10T11:28:28.216 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:28.225 7f72e74ab700 1 Processor -- start 2021-02-10T11:28:28.216 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:28.225 7f72e74ab700 1 -- start start 2021-02-10T11:28:28.216 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:28.225 7f72e74ab700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f72e00c36a0 0x7f72e00c2b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:28.216 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:28.225 7f72e74ab700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f72e00c3190 0x7f72e00b1d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:28.216 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:28.225 7f72e74ab700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f72e00b2370 0x7f72e00b2d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:28.216 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:28.225 7f72e74ab700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72e00b27c0 con 0x7f72e00b2370 2021-02-10T11:28:28.216 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:28.225 7f72e74ab700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72e00b3360 con 0x7f72e00c36a0 2021-02-10T11:28:28.216 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:28.225 7f72e74ab700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72e00b3540 con 0x7f72e00c3190 2021-02-10T11:28:28.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:28.226 7f72e5ac8700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f72e00b2370 0x7f72e00b2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:28.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:28.226 7f72e5ac8700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f72e00b2370 0x7f72e00b2d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:46310/0 (socket says 172.21.2.112:46310) 2021-02-10T11:28:28.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:28.226 7f72e5ac8700 1 -- 172.21.2.112:0/986798122 learned_addr learned my addr 172.21.2.112:0/986798122 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:28:28.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:28.226 7f72e5ac8700 1 -- 172.21.2.112:0/986798122 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f72e00c3190 msgr2=0x7f72e00b1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:28.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:28.226 7f72e5ac8700 1 --2- 172.21.2.112:0/986798122 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f72e00c3190 0x7f72e00b1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:28.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:28.226 7f72e5ac8700 1 -- 172.21.2.112:0/986798122 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f72e00c36a0 msgr2=0x7f72e00c2b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:28.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:28.226 7f72e5ac8700 1 --2- 172.21.2.112:0/986798122 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f72e00c36a0 0x7f72e00c2b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:28.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:28.226 7f72e5ac8700 1 -- 172.21.2.112:0/986798122 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72e00b3760 con 0x7f72e00b2370 2021-02-10T11:28:28.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:28.226 7f72e5ac8700 1 --2- 172.21.2.112:0/986798122 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f72e00b2370 0x7f72e00b2d40 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 rx=0x7f72d8009980 tx=0x7f72d800bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:28.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:28.226 7f72d1ffb700 1 -- 172.21.2.112:0/986798122 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f72d8003020 con 0x7f72e00b2370 2021-02-10T11:28:28.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:28.226 7f72d1ffb700 1 -- 172.21.2.112:0/986798122 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72d8005a80 con 0x7f72e00b2370 2021-02-10T11:28:28.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:28.226 7f72e74ab700 1 -- 172.21.2.112:0/986798122 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72e00b3760 con 0x7f72e00b2370 2021-02-10T11:28:28.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:28.226 7f72e74ab700 1 -- 172.21.2.112:0/986798122 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72e00b3cf0 con 0x7f72e00b2370 2021-02-10T11:28:28.218 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:28.227 7f72d1ffb700 1 -- 172.21.2.112:0/986798122 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f72d8006030 con 0x7f72e00b2370 2021-02-10T11:28:28.218 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:28.227 7f72d1ffb700 1 -- 172.21.2.112:0/986798122 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f72d8006500 con 0x7f72e00b2370 2021-02-10T11:28:28.218 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:28.227 7f72d1ffb700 1 --2- 172.21.2.112:0/986798122 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f72c00186c0 0x7f72c001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:28.218 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:28.227 7f72d1ffb700 1 -- 172.21.2.112:0/986798122 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f72d8032960 con 0x7f72e00b2370 2021-02-10T11:28:28.218 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:28.228 7f72e52c7700 1 --2- 172.21.2.112:0/986798122 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f72c00186c0 0x7f72c001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:28.218 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:28.228 7f72e52c7700 1 --2- 172.21.2.112:0/986798122 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f72c00186c0 0x7f72c001abc0 crc :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:28.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:28.258 7f72e74ab700 1 -- 172.21.2.112:0/986798122 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72e00edd10 con 0x7f72e00b2370 2021-02-10T11:28:28.248 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:28.261 7f72d1ffb700 1 -- 172.21.2.112:0/986798122 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f72d80061d0 con 0x7f72e00b2370 2021-02-10T11:28:28.378 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:28.390 7f72e74ab700 1 -- 172.21.2.112:0/986798122 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348657", "target": ["mgr", ""]}) v1 -- 0x7f72e00b27c0 con 0x7f72c00186c0 2021-02-10T11:28:28.381 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:28.393 7f72d1ffb700 1 -- 172.21.2.112:0/986798122 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f72dc000970 con 0x7f72c00186c0 2021-02-10T11:28:28.385 INFO:teuthology.orchestra.run.gibba012.stdout:/volumes/_nogroup/subvolume_0000000000348657/8c35f54b-ac01-4b88-a68d-e7e5f93ed038 2021-02-10T11:28:28.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:28.399 7f72e74ab700 1 -- 172.21.2.112:0/986798122 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f72c00186c0 msgr2=0x7f72c001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:28.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:28.399 7f72e74ab700 1 --2- 172.21.2.112:0/986798122 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f72c00186c0 0x7f72c001abc0 crc :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:28.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:28.399 7f72e74ab700 1 -- 172.21.2.112:0/986798122 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f72e00b2370 msgr2=0x7f72e00b2d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:28.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:28.399 7f72e74ab700 1 --2- 172.21.2.112:0/986798122 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f72e00b2370 0x7f72e00b2d40 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 rx=0x7f72d8009980 tx=0x7f72d800bcd0).stop 2021-02-10T11:28:28.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:28.400 7f72e74ab700 1 -- 172.21.2.112:0/986798122 shutdown_connections 2021-02-10T11:28:28.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:28.400 7f72e74ab700 1 --2- 172.21.2.112:0/986798122 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f72c00186c0 0x7f72c001abc0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:28.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:28.400 7f72e74ab700 1 --2- 172.21.2.112:0/986798122 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f72e00b2370 0x7f72e00b2d40 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:28.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:28.400 7f72e74ab700 1 --2- 172.21.2.112:0/986798122 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f72e00c3190 0x7f72e00b1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:28.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:28.400 7f72e74ab700 1 --2- 172.21.2.112:0/986798122 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f72e00c36a0 0x7f72e00c2b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:28.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:28.400 7f72e74ab700 1 -- 172.21.2.112:0/986798122 >> 172.21.2.112:0/986798122 conn(0x7f72e009f630 msgr2=0x7f72e00a0130 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:28:28.389 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:28.400 7f72e74ab700 1 -- 172.21.2.112:0/986798122 shutdown_connections 2021-02-10T11:28:28.389 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:28.400 7f72e74ab700 1 -- 172.21.2.112:0/986798122 wait complete. 2021-02-10T11:28:28.396 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000348657 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000348657/8c35f54b-ac01-4b88-a68d-e7e5f93ed038 2021-02-10T11:28:28.396 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:28.396 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:28.397 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:28.733 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:28:28.733 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:28.733 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348657/8c35f54b-ac01-4b88-a68d-e7e5f93ed038/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:29.117 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:28:29.117 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:28:29.117 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.026101 s, 40.2 MB/s 2021-02-10T11:28:29.118 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:29.118 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:29.118 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:29.455 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:28:29.456 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:29.456 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348657/8c35f54b-ac01-4b88-a68d-e7e5f93ed038/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:29.827 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:28:29.828 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:28:29.828 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0228777 s, 45.8 MB/s 2021-02-10T11:28:29.829 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:29.829 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:29.829 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:30.165 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:28:30.166 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:30.166 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348657/8c35f54b-ac01-4b88-a68d-e7e5f93ed038/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:30.541 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:28:30.542 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:28:30.542 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0228676 s, 45.9 MB/s 2021-02-10T11:28:30.543 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:30.543 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:30.543 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:30.878 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:28:30.879 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:30.879 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348657/8c35f54b-ac01-4b88-a68d-e7e5f93ed038/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:31.252 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:28:31.253 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:28:31.253 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0227829 s, 46.0 MB/s 2021-02-10T11:28:31.253 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:31.254 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:31.254 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:31.591 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:28:31.592 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:31.592 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348657/8c35f54b-ac01-4b88-a68d-e7e5f93ed038/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:31.968 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:28:31.969 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:28:31.969 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0246109 s, 42.6 MB/s 2021-02-10T11:28:31.970 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:31.970 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:31.970 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:32.305 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:28:32.305 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:32.305 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348657/8c35f54b-ac01-4b88-a68d-e7e5f93ed038/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:32.681 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:28:32.681 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:28:32.681 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0243816 s, 43.0 MB/s 2021-02-10T11:28:32.682 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:32.682 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:32.682 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:33.018 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:28:33.019 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:33.019 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348657/8c35f54b-ac01-4b88-a68d-e7e5f93ed038/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:33.394 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:28:33.394 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:28:33.394 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0244896 s, 42.8 MB/s 2021-02-10T11:28:33.395 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:33.395 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:33.395 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:33.731 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:28:33.732 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:33.732 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348657/8c35f54b-ac01-4b88-a68d-e7e5f93ed038/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:34.105 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:28:34.519 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:28:34.519 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.022882 s, 45.8 MB/s 2021-02-10T11:28:34.521 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:34.521 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:34.521 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:34.869 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:28:34.870 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:34.870 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348657/8c35f54b-ac01-4b88-a68d-e7e5f93ed038/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:35.244 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:28:35.244 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:28:35.245 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0250341 s, 41.9 MB/s 2021-02-10T11:28:35.245 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:35.246 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:35.246 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:35.583 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:28:35.583 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:35.583 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348657/8c35f54b-ac01-4b88-a68d-e7e5f93ed038/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:35.958 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:28:35.959 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:28:35.959 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0230819 s, 45.4 MB/s 2021-02-10T11:28:35.960 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000348658 2021-02-10T11:28:36.391 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:36.387 7f509abdc700 1 Processor -- start 2021-02-10T11:28:36.391 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:36.387 7f509abdc700 1 -- start start 2021-02-10T11:28:36.391 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:36.388 7f509abdc700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f50941b0470 0x7f50941b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:36.391 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:36.388 7f509abdc700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f50940ef610 0x7f50940efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:36.391 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:36.388 7f509abdc700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f50940f0030 0x7f50940f0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:36.391 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:36.388 7f509abdc700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50940e5ea0 con 0x7f50940f0030 2021-02-10T11:28:36.391 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:36.388 7f509abdc700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50940f0b50 con 0x7f50940ef610 2021-02-10T11:28:36.391 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:36.388 7f509abdc700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50940e52e0 con 0x7f50941b0470 2021-02-10T11:28:36.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:36.388 7f50991f9700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f50940f0030 0x7f50940f0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:36.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:36.388 7f50991f9700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f50940f0030 0x7f50940f0430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:46316/0 (socket says 172.21.2.112:46316) 2021-02-10T11:28:36.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:36.388 7f50991f9700 1 -- 172.21.2.112:0/725996133 learned_addr learned my addr 172.21.2.112:0/725996133 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:28:36.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:36.388 7f50991f9700 1 -- 172.21.2.112:0/725996133 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f50941b0470 msgr2=0x7f50941b0870 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:28:36.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:36.388 7f50991f9700 1 --2- 172.21.2.112:0/725996133 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f50941b0470 0x7f50941b0870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:36.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:36.388 7f50991f9700 1 -- 172.21.2.112:0/725996133 >> v2:172.21.2.120:3300/0 conn(0x7f50940ef610 msgr2=0x7f50940efa10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:28:36.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:36.388 7f50991f9700 1 --2- 172.21.2.112:0/725996133 >> v2:172.21.2.120:3300/0 conn(0x7f50940ef610 0x7f50940efa10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:36.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:36.388 7f50991f9700 1 -- 172.21.2.112:0/725996133 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50940f10d0 con 0x7f50940f0030 2021-02-10T11:28:36.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:36.388 7f50991f9700 1 --2- 172.21.2.112:0/725996133 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f50940f0030 0x7f50940f0430 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 rx=0x7f508c006b80 tx=0x7f508c009b90).ready entity=mon.0 client_cookie=9e191e378dbbd1e7 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:36.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:36.388 7f50937fe700 1 -- 172.21.2.112:0/725996133 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f508c0022e0 con 0x7f50940f0030 2021-02-10T11:28:36.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:36.388 7f50937fe700 1 -- 172.21.2.112:0/725996133 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f508c002620 con 0x7f50940f0030 2021-02-10T11:28:36.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:36.388 7f50937fe700 1 -- 172.21.2.112:0/725996133 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f508c002a80 con 0x7f50940f0030 2021-02-10T11:28:36.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:36.388 7f509abdc700 1 -- 172.21.2.112:0/725996133 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f50940f0030 msgr2=0x7f50940f0430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:36.393 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:36.388 7f509abdc700 1 --2- 172.21.2.112:0/725996133 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f50940f0030 0x7f50940f0430 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 rx=0x7f508c006b80 tx=0x7f508c009b90).stop 2021-02-10T11:28:36.393 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:36.389 7f509abdc700 1 -- 172.21.2.112:0/725996133 shutdown_connections 2021-02-10T11:28:36.393 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:36.389 7f509abdc700 1 --2- 172.21.2.112:0/725996133 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f50940f0030 0x7f50940f0430 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:36.393 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:36.389 7f509abdc700 1 --2- 172.21.2.112:0/725996133 >> v2:172.21.2.120:3300/0 conn(0x7f50940ef610 0x7f50940efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:36.393 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:36.389 7f509abdc700 1 --2- 172.21.2.112:0/725996133 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f50941b0470 0x7f50941b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:36.393 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:36.389 7f509abdc700 1 -- 172.21.2.112:0/725996133 >> 172.21.2.112:0/725996133 conn(0x7f50940ce3e0 msgr2=0x7f50940e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:28:36.393 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:36.389 7f509abdc700 1 -- 172.21.2.112:0/725996133 shutdown_connections 2021-02-10T11:28:36.393 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:36.389 7f509abdc700 1 -- 172.21.2.112:0/725996133 wait complete. 2021-02-10T11:28:36.394 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:36.404 7f509abdc700 1 Processor -- start 2021-02-10T11:28:36.395 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:36.404 7f509abdc700 1 -- start start 2021-02-10T11:28:36.395 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:36.404 7f509abdc700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5094190920 0x7f509408f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:36.395 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:36.404 7f509abdc700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5094190d20 0x7f509408fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:36.395 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:36.404 7f509abdc700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5094193270 0x7f5094193670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:36.395 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:36.404 7f509abdc700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5094090470 con 0x7f5094193270 2021-02-10T11:28:36.395 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:36.404 7f509abdc700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5094193c90 con 0x7f5094190920 2021-02-10T11:28:36.395 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:36.404 7f509abdc700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5094193e70 con 0x7f5094190d20 2021-02-10T11:28:36.395 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:36.404 7f50991f9700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5094193270 0x7f5094193670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:36.395 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:36.404 7f50991f9700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5094193270 0x7f5094193670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:46324/0 (socket says 172.21.2.112:46324) 2021-02-10T11:28:36.395 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:36.404 7f50991f9700 1 -- 172.21.2.112:0/3473684832 learned_addr learned my addr 172.21.2.112:0/3473684832 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:28:36.395 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:36.405 7f50991f9700 1 -- 172.21.2.112:0/3473684832 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5094190d20 msgr2=0x7f509408fc20 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:28:36.395 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:36.405 7f50991f9700 1 --2- 172.21.2.112:0/3473684832 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5094190d20 0x7f509408fc20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:36.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:36.405 7f50991f9700 1 -- 172.21.2.112:0/3473684832 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5094190920 msgr2=0x7f509408f600 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:28:36.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:36.405 7f50991f9700 1 --2- 172.21.2.112:0/3473684832 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5094190920 0x7f509408f600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:36.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:36.405 7f50991f9700 1 -- 172.21.2.112:0/3473684832 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5094194090 con 0x7f5094193270 2021-02-10T11:28:36.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:36.405 7f50991f9700 1 --2- 172.21.2.112:0/3473684832 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5094193270 0x7f5094193670 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 rx=0x7f508c00b630 tx=0x7f508c00b9d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:36.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:36.405 7f50917fa700 1 -- 172.21.2.112:0/3473684832 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f508c000b30 con 0x7f5094193270 2021-02-10T11:28:36.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:36.405 7f50917fa700 1 -- 172.21.2.112:0/3473684832 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f508c006530 con 0x7f5094193270 2021-02-10T11:28:36.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:36.405 7f50917fa700 1 -- 172.21.2.112:0/3473684832 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f508c002280 con 0x7f5094193270 2021-02-10T11:28:36.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:36.405 7f509abdc700 1 -- 172.21.2.112:0/3473684832 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5094194090 con 0x7f5094193270 2021-02-10T11:28:36.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:36.405 7f509abdc700 1 -- 172.21.2.112:0/3473684832 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50941920b0 con 0x7f5094193270 2021-02-10T11:28:36.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:36.406 7f50917fa700 1 -- 172.21.2.112:0/3473684832 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f508c00a020 con 0x7f5094193270 2021-02-10T11:28:36.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:36.406 7f50917fa700 1 --2- 172.21.2.112:0/3473684832 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f50780186c0 0x7f507801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:36.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:36.406 7f50917fa700 1 -- 172.21.2.112:0/3473684832 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f508c032570 con 0x7f5094193270 2021-02-10T11:28:36.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:36.406 7f50989f8700 1 --2- 172.21.2.112:0/3473684832 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f50780186c0 0x7f507801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:36.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:36.407 7f50989f8700 1 --2- 172.21.2.112:0/3473684832 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f50780186c0 0x7f507801abc0 crc :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:36.425 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:36.438 7f509abdc700 1 -- 172.21.2.112:0/3473684832 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5094105af0 con 0x7f5094193270 2021-02-10T11:28:36.428 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:36.440 7f50917fa700 1 -- 172.21.2.112:0/3473684832 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f508c009480 con 0x7f5094193270 2021-02-10T11:28:36.555 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:36.567 7f509abdc700 1 -- 172.21.2.112:0/3473684832 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348658", "target": ["mgr", ""]}) v1 -- 0x7f50941920b0 con 0x7f50780186c0 2021-02-10T11:28:36.581 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:36.593 7f50917fa700 1 -- 172.21.2.112:0/3473684832 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f507c0009b0 con 0x7f50780186c0 2021-02-10T11:28:36.589 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:36.600 7f509abdc700 1 -- 172.21.2.112:0/3473684832 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f50780186c0 msgr2=0x7f507801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:36.589 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:36.600 7f509abdc700 1 --2- 172.21.2.112:0/3473684832 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f50780186c0 0x7f507801abc0 crc :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:36.589 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:36.600 7f509abdc700 1 -- 172.21.2.112:0/3473684832 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5094193270 msgr2=0x7f5094193670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:36.589 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:36.600 7f509abdc700 1 --2- 172.21.2.112:0/3473684832 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5094193270 0x7f5094193670 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 rx=0x7f508c00b630 tx=0x7f508c00b9d0).stop 2021-02-10T11:28:36.589 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:36.601 7f509abdc700 1 -- 172.21.2.112:0/3473684832 shutdown_connections 2021-02-10T11:28:36.589 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:36.601 7f509abdc700 1 --2- 172.21.2.112:0/3473684832 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f50780186c0 0x7f507801abc0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:36.589 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:36.601 7f509abdc700 1 --2- 172.21.2.112:0/3473684832 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5094193270 0x7f5094193670 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:36.589 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:36.601 7f509abdc700 1 --2- 172.21.2.112:0/3473684832 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5094190d20 0x7f509408fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:36.589 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:36.601 7f509abdc700 1 --2- 172.21.2.112:0/3473684832 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5094190920 0x7f509408f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:36.589 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:36.601 7f509abdc700 1 -- 172.21.2.112:0/3473684832 >> 172.21.2.112:0/3473684832 conn(0x7f50940bf4e0 msgr2=0x7f50940be3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:28:36.590 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:36.601 7f509abdc700 1 -- 172.21.2.112:0/3473684832 shutdown_connections 2021-02-10T11:28:36.590 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:36.601 7f509abdc700 1 -- 172.21.2.112:0/3473684832 wait complete. 2021-02-10T11:28:36.596 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000348658 2021-02-10T11:28:37.029 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:37.024 7f785c723700 1 Processor -- start 2021-02-10T11:28:37.029 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:37.024 7f785c723700 1 -- start start 2021-02-10T11:28:37.029 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:37.024 7f785c723700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f78541b0470 0x7f78541b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:37.029 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:37.024 7f785c723700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f78540f20e0 0x7f78540f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:37.029 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:37.024 7f785c723700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78540f0b50 con 0x7f78540ef610 2021-02-10T11:28:37.029 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:37.024 7f785c723700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78540f0820 con 0x7f78540f20e0 2021-02-10T11:28:37.029 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:37.024 7f785c723700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78540f24e0 con 0x7f78541b0470 2021-02-10T11:28:37.030 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:37.025 7f785a53f700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f78541b0470 0x7f78541b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:37.030 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:37.025 7f785ad40700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f78540f20e0 0x7f78540f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:37.030 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:37.025 7f785a53f700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f78541b0470 0x7f78541b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3300/0 says I am v2:172.21.2.112:48608/0 (socket says 172.21.2.112:48608) 2021-02-10T11:28:37.030 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:37.025 7f785a53f700 1 -- 172.21.2.112:0/1812714071 learned_addr learned my addr 172.21.2.112:0/1812714071 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:28:37.030 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:37.025 7f785a53f700 1 -- 172.21.2.112:0/1812714071 >> v2:172.21.2.112:3300/0 conn(0x7f78540f20e0 msgr2=0x7f78540f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:28:37.030 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:37.025 7f785a53f700 1 --2- 172.21.2.112:0/1812714071 >> v2:172.21.2.112:3300/0 conn(0x7f78540f20e0 0x7f78540f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:37.030 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:37.025 7f785a53f700 1 -- 172.21.2.112:0/1812714071 >> v1:172.21.2.112:6789/0 conn(0x7f78540ef610 legacy=0x7f78540efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:37.030 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:37.025 7f785953d700 1 -- 172.21.2.112:0/1812714071 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3925170911 0 0) 0x7f7844000ce0 con 0x7f78540ef610 2021-02-10T11:28:37.030 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:37.025 7f785a53f700 1 -- 172.21.2.112:0/1812714071 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78540e52e0 con 0x7f78541b0470 2021-02-10T11:28:37.030 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:37.025 7f785ad40700 1 --2- 172.21.2.112:0/1812714071 >> v2:172.21.2.112:3300/0 conn(0x7f78540f20e0 0x7f78540f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:28:37.030 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:37.025 7f785a53f700 1 --2- 172.21.2.112:0/1812714071 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f78541b0470 0x7f78541b0870 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7f7850006700 tx=0x7f7850009b80).ready entity=mon.1 client_cookie=4fe6e01fbd6e4029 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:37.031 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:37.025 7f785953d700 1 -- 172.21.2.112:0/1812714071 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f785000c0c0 con 0x7f78541b0470 2021-02-10T11:28:37.031 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:37.025 7f785953d700 1 -- 172.21.2.112:0/1812714071 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f785000c440 con 0x7f78541b0470 2021-02-10T11:28:37.031 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:37.025 7f785953d700 1 -- 172.21.2.112:0/1812714071 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f785000c960 con 0x7f78541b0470 2021-02-10T11:28:37.031 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:37.025 7f785c723700 1 -- 172.21.2.112:0/1812714071 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f78541b0470 msgr2=0x7f78541b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:37.031 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:37.025 7f785c723700 1 --2- 172.21.2.112:0/1812714071 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f78541b0470 0x7f78541b0870 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7f7850006700 tx=0x7f7850009b80).stop 2021-02-10T11:28:37.031 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:37.026 7f785c723700 1 -- 172.21.2.112:0/1812714071 shutdown_connections 2021-02-10T11:28:37.031 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:37.026 7f785c723700 1 --2- 172.21.2.112:0/1812714071 >> v2:172.21.2.112:3300/0 conn(0x7f78540f20e0 0x7f78540f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:37.031 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:37.026 7f785c723700 1 --2- 172.21.2.112:0/1812714071 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f78541b0470 0x7f78541b0870 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:37.031 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:37.026 7f785c723700 1 -- 172.21.2.112:0/1812714071 >> 172.21.2.112:0/1812714071 conn(0x7f78540ce3e0 msgr2=0x7f78540e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:28:37.031 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:37.026 7f785c723700 1 -- 172.21.2.112:0/1812714071 shutdown_connections 2021-02-10T11:28:37.031 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:37.026 7f785c723700 1 -- 172.21.2.112:0/1812714071 wait complete. 2021-02-10T11:28:37.031 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:37.041 7f785c723700 1 Processor -- start 2021-02-10T11:28:37.031 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:37.041 7f785c723700 1 -- start start 2021-02-10T11:28:37.033 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:37.042 7f785c723700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f78540a6a70 0x7f78540a5f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:37.034 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:37.042 7f785c723700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f78540a6e70 0x7f78540ba3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:37.034 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:37.042 7f785c723700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f78540b9810 0x7f78540ba9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:37.034 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:37.042 7f785c723700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78540b9c10 con 0x7f78540a6a70 2021-02-10T11:28:37.034 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:37.042 7f785c723700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78540b9e20 con 0x7f78540b9810 2021-02-10T11:28:37.034 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:37.042 7f785c723700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78540a0a80 con 0x7f78540a6e70 2021-02-10T11:28:37.034 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:37.042 7f785a53f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f78540a6a70 0x7f78540a5f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:37.034 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:37.042 7f785ad40700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f78540b9810 0x7f78540ba9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:37.034 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:37.042 7f785ad40700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f78540b9810 0x7f78540ba9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3300/0 says I am v2:172.21.2.112:48612/0 (socket says 172.21.2.112:48612) 2021-02-10T11:28:37.034 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:37.042 7f785ad40700 1 -- 172.21.2.112:0/186725480 learned_addr learned my addr 172.21.2.112:0/186725480 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:28:37.034 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:37.042 7f785a53f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f78540a6a70 0x7f78540a5f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:46334/0 (socket says 172.21.2.112:46334) 2021-02-10T11:28:37.034 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:37.042 7f7859d3e700 1 --2- 172.21.2.112:0/186725480 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f78540a6e70 0x7f78540ba3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:37.035 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:37.042 7f785ad40700 1 -- 172.21.2.112:0/186725480 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f78540a6e70 msgr2=0x7f78540ba3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:37.035 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:37.042 7f785ad40700 1 --2- 172.21.2.112:0/186725480 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f78540a6e70 0x7f78540ba3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:37.035 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:37.042 7f785ad40700 1 -- 172.21.2.112:0/186725480 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f78540a6a70 msgr2=0x7f78540a5f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:37.035 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:37.042 7f785ad40700 1 --2- 172.21.2.112:0/186725480 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f78540a6a70 0x7f78540a5f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:37.035 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:37.043 7f785ad40700 1 -- 172.21.2.112:0/186725480 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78540a6540 con 0x7f78540b9810 2021-02-10T11:28:37.035 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:37.043 7f785a53f700 1 --2- 172.21.2.112:0/186725480 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f78540a6a70 0x7f78540a5f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:28:37.035 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:37.043 7f785ad40700 1 --2- 172.21.2.112:0/186725480 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f78540b9810 0x7f78540ba9c0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0x7f784c00b970 tx=0x7f784c00bd10).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:37.035 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:37.043 7f784affd700 1 -- 172.21.2.112:0/186725480 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f784c000f90 con 0x7f78540b9810 2021-02-10T11:28:37.035 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:37.043 7f784affd700 1 -- 172.21.2.112:0/186725480 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f784c001270 con 0x7f78540b9810 2021-02-10T11:28:37.035 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:37.043 7f785c723700 1 -- 172.21.2.112:0/186725480 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78540a6540 con 0x7f78540b9810 2021-02-10T11:28:37.035 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:37.043 7f784affd700 1 -- 172.21.2.112:0/186725480 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f784c000f90 con 0x7f78540b9810 2021-02-10T11:28:37.035 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:37.043 7f785c723700 1 -- 172.21.2.112:0/186725480 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78540a6540 con 0x7f78540b9810 2021-02-10T11:28:37.035 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:37.043 7f784affd700 1 -- 172.21.2.112:0/186725480 <== mon.1 v2:172.21.2.120:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f784c009f30 con 0x7f78540b9810 2021-02-10T11:28:37.036 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:37.043 7f784affd700 1 --2- 172.21.2.112:0/186725480 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f783c018730 0x7f783c01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:37.036 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:37.043 7f784affd700 1 -- 172.21.2.112:0/186725480 <== mon.1 v2:172.21.2.120:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f784c0329b0 con 0x7f78540b9810 2021-02-10T11:28:37.036 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:37.044 7f785a53f700 1 --2- 172.21.2.112:0/186725480 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f783c018730 0x7f783c01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:37.036 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:37.044 7f785a53f700 1 --2- 172.21.2.112:0/186725480 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f783c018730 0x7f783c01ac10 crc :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:37.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:37.074 7f785c723700 1 -- 172.21.2.112:0/186725480 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7854105ae0 con 0x7f78540b9810 2021-02-10T11:28:37.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:37.076 7f784affd700 1 -- 172.21.2.112:0/186725480 <== mon.1 v2:172.21.2.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f784c000f90 con 0x7f78540b9810 2021-02-10T11:28:37.194 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:37.206 7f785c723700 1 -- 172.21.2.112:0/186725480 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348658", "target": ["mgr", ""]}) v1 -- 0x7f78540b9c10 con 0x7f783c018730 2021-02-10T11:28:37.197 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:37.209 7f784affd700 1 -- 172.21.2.112:0/186725480 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f7850006e80 con 0x7f783c018730 2021-02-10T11:28:37.201 INFO:teuthology.orchestra.run.gibba012.stdout:/volumes/_nogroup/subvolume_0000000000348658/dfe74796-0b18-444c-9bb4-84306fa4bdce 2021-02-10T11:28:37.202 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:37.214 7f785c723700 1 -- 172.21.2.112:0/186725480 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f783c018730 msgr2=0x7f783c01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:37.202 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:37.214 7f785c723700 1 --2- 172.21.2.112:0/186725480 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f783c018730 0x7f783c01ac10 crc :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:37.202 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:37.214 7f785c723700 1 -- 172.21.2.112:0/186725480 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f78540b9810 msgr2=0x7f78540ba9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:37.202 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:37.214 7f785c723700 1 --2- 172.21.2.112:0/186725480 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f78540b9810 0x7f78540ba9c0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0x7f784c00b970 tx=0x7f784c00bd10).stop 2021-02-10T11:28:37.203 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:37.215 7f785c723700 1 -- 172.21.2.112:0/186725480 shutdown_connections 2021-02-10T11:28:37.203 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:37.215 7f785c723700 1 --2- 172.21.2.112:0/186725480 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f783c018730 0x7f783c01ac10 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:37.203 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:37.215 7f785c723700 1 --2- 172.21.2.112:0/186725480 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f78540b9810 0x7f78540ba9c0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:37.203 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:37.215 7f785c723700 1 --2- 172.21.2.112:0/186725480 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f78540a6e70 0x7f78540ba3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:37.203 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:37.215 7f785c723700 1 --2- 172.21.2.112:0/186725480 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f78540a6a70 0x7f78540a5f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:37.203 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:37.215 7f785c723700 1 -- 172.21.2.112:0/186725480 >> 172.21.2.112:0/186725480 conn(0x7f78540bbf60 msgr2=0x7f7854105d10 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:28:37.203 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:37.215 7f785c723700 1 -- 172.21.2.112:0/186725480 shutdown_connections 2021-02-10T11:28:37.203 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:37.215 7f785c723700 1 -- 172.21.2.112:0/186725480 wait complete. 2021-02-10T11:28:37.210 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000348658 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000348658/dfe74796-0b18-444c-9bb4-84306fa4bdce 2021-02-10T11:28:37.210 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:37.211 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:37.211 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:37.546 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:28:37.546 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:37.547 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348658/dfe74796-0b18-444c-9bb4-84306fa4bdce/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:37.928 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:28:37.928 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:28:37.928 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0233955 s, 44.8 MB/s 2021-02-10T11:28:37.930 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:37.930 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:37.930 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:38.267 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:28:38.267 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:38.267 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348658/dfe74796-0b18-444c-9bb4-84306fa4bdce/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:38.645 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:28:38.645 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:28:38.645 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0256734 s, 40.8 MB/s 2021-02-10T11:28:38.646 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:38.646 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:38.647 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:38.984 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:28:38.984 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:38.984 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348658/dfe74796-0b18-444c-9bb4-84306fa4bdce/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:39.358 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:28:39.359 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:28:39.359 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.026169 s, 40.1 MB/s 2021-02-10T11:28:39.360 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:39.360 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:39.360 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:39.695 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:28:39.696 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:39.696 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348658/dfe74796-0b18-444c-9bb4-84306fa4bdce/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:40.074 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:28:40.074 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:28:40.074 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0260248 s, 40.3 MB/s 2021-02-10T11:28:40.075 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:40.075 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:40.075 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:40.412 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:28:40.412 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:40.412 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348658/dfe74796-0b18-444c-9bb4-84306fa4bdce/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:40.788 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:28:40.788 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:28:40.788 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0237644 s, 44.1 MB/s 2021-02-10T11:28:40.789 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:40.789 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:40.789 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:41.126 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:28:41.127 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:41.127 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348658/dfe74796-0b18-444c-9bb4-84306fa4bdce/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:41.504 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:28:41.504 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:28:41.504 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0237243 s, 44.2 MB/s 2021-02-10T11:28:41.505 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:41.505 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:41.505 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:41.840 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:28:41.841 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:41.841 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348658/dfe74796-0b18-444c-9bb4-84306fa4bdce/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:42.220 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:28:42.220 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:28:42.220 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.025456 s, 41.2 MB/s 2021-02-10T11:28:42.221 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:42.221 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:42.221 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:42.423 DEBUG:teuthology.orchestra.run.gibba012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:28:42.427 DEBUG:teuthology.orchestra.run.gibba020:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:28:42.556 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:28:42.556 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:42.556 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348658/dfe74796-0b18-444c-9bb4-84306fa4bdce/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:42.931 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:28:42.931 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:28:42.931 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.023691 s, 44.3 MB/s 2021-02-10T11:28:42.932 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:42.932 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:42.932 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:43.268 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:28:43.269 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:43.269 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348658/dfe74796-0b18-444c-9bb4-84306fa4bdce/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:43.646 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:28:43.647 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:28:43.647 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0244369 s, 42.9 MB/s 2021-02-10T11:28:43.648 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:43.648 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:43.648 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:43.983 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:28:43.984 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:43.984 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348658/dfe74796-0b18-444c-9bb4-84306fa4bdce/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:44.358 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:28:44.359 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:28:44.359 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0235759 s, 44.5 MB/s 2021-02-10T11:28:44.360 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000348659 2021-02-10T11:28:44.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:44.786 7fb902d34700 1 Processor -- start 2021-02-10T11:28:44.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:44.786 7fb902d34700 1 -- start start 2021-02-10T11:28:44.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:44.787 7fb902d34700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb8fc1b0470 0x7fb8fc1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:44.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:44.787 7fb902d34700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fb8fc0f20e0 0x7fb8fc0f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:44.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:44.787 7fb902d34700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8fc0f0b50 con 0x7fb8fc0ef610 2021-02-10T11:28:44.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:44.787 7fb902d34700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8fc0f0820 con 0x7fb8fc0f20e0 2021-02-10T11:28:44.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:44.787 7fb902d34700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8fc0f24e0 con 0x7fb8fc1b0470 2021-02-10T11:28:44.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:44.787 7fb8fbfff700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fb8fc0ef610 0x7fb8fc0efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:45806/0 (socket says 172.21.2.112:45806) 2021-02-10T11:28:44.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:44.787 7fb8fbfff700 1 -- 172.21.2.112:0/814570213 learned_addr learned my addr 172.21.2.112:0/814570213 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:28:44.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:44.787 7fb8fb7fe700 1 -- 172.21.2.112:0/814570213 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 945733273 0 0) 0x7fb8ec0009b0 con 0x7fb8fc0ef610 2021-02-10T11:28:44.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:44.787 7fb8fb7fe700 1 -- 172.21.2.112:0/814570213 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8e8002fe0 con 0x7fb8fc0ef610 2021-02-10T11:28:44.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:44.787 7fb8fb7fe700 1 -- 172.21.2.112:0/814570213 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3863267446 0 0) 0x7fb8ec001010 con 0x7fb8fc0ef610 2021-02-10T11:28:44.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:44.787 7fb8fb7fe700 1 -- 172.21.2.112:0/814570213 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb8fc1b0470 msgr2=0x7fb8fc1b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:28:44.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:44.787 7fb8fb7fe700 1 --2- 172.21.2.112:0/814570213 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb8fc1b0470 0x7fb8fc1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:44.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:44.787 7fb8fb7fe700 1 -- 172.21.2.112:0/814570213 >> v2:172.21.2.120:3300/0 conn(0x7fb8fc0f20e0 msgr2=0x7fb8fc0f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:28:44.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:44.787 7fb8fb7fe700 1 --2- 172.21.2.112:0/814570213 >> v2:172.21.2.120:3300/0 conn(0x7fb8fc0f20e0 0x7fb8fc0f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:44.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:44.787 7fb8fb7fe700 1 -- 172.21.2.112:0/814570213 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8fc0e52e0 con 0x7fb8fc0ef610 2021-02-10T11:28:44.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:44.787 7fb8fb7fe700 1 -- 172.21.2.112:0/814570213 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7fb8ec001590 con 0x7fb8fc0ef610 2021-02-10T11:28:44.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:44.787 7fb8fb7fe700 1 -- 172.21.2.112:0/814570213 >> v1:172.21.2.112:6789/0 conn(0x7fb8fc0ef610 legacy=0x7fb8fc0efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:44.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:44.787 7fb8fb7fe700 1 --2- 172.21.2.112:0/814570213 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb8e8003af0 0x7fb8e8005f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:44.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:44.787 7fb8fb7fe700 1 --2- 172.21.2.112:0/814570213 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb8e8006780 0x7fb8e8008c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:44.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:44.787 7fb8fb7fe700 1 --2- 172.21.2.112:0/814570213 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb8e8009520 0x7fb8e800ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:44.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:44.787 7fb8fb7fe700 1 -- 172.21.2.112:0/814570213 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8e800c2d0 con 0x7fb8e8006780 2021-02-10T11:28:44.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:44.787 7fb8fb7fe700 1 -- 172.21.2.112:0/814570213 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8e800c4f0 con 0x7fb8e8009520 2021-02-10T11:28:44.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:44.787 7fb8fb7fe700 1 -- 172.21.2.112:0/814570213 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8e800c710 con 0x7fb8e8003af0 2021-02-10T11:28:44.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:44.787 7fb8fbfff700 1 --2- 172.21.2.112:0/814570213 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb8e8006780 0x7fb8e8008c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:44.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:44.788 7fb901351700 1 --2- 172.21.2.112:0/814570213 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb8e8009520 0x7fb8e800ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:44.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:44.788 7fb900b50700 1 --2- 172.21.2.112:0/814570213 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb8e8003af0 0x7fb8e8005f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:44.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:44.788 7fb8fbfff700 1 -- 172.21.2.112:0/814570213 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb8e8003af0 msgr2=0x7fb8e8005f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:28:44.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:44.788 7fb8fbfff700 1 --2- 172.21.2.112:0/814570213 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb8e8003af0 0x7fb8e8005f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:44.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:44.788 7fb8fbfff700 1 -- 172.21.2.112:0/814570213 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb8e8009520 msgr2=0x7fb8e800ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:28:44.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:44.788 7fb8fbfff700 1 --2- 172.21.2.112:0/814570213 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb8e8009520 0x7fb8e800ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:44.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:44.788 7fb8fbfff700 1 -- 172.21.2.112:0/814570213 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8e800c9b0 con 0x7fb8e8006780 2021-02-10T11:28:44.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:44.788 7fb8fbfff700 1 --2- 172.21.2.112:0/814570213 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb8e8006780 0x7fb8e8008c80 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 rx=0x7fb8ec001c60 tx=0x7fb8ec007d40).ready entity=mon.0 client_cookie=3eca9dc8554ff92a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:44.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:44.788 7fb8fb7fe700 1 -- 172.21.2.112:0/814570213 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb8ec0013f0 con 0x7fb8e8006780 2021-02-10T11:28:44.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:44.788 7fb8fb7fe700 1 -- 172.21.2.112:0/814570213 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8ec001010 con 0x7fb8e8006780 2021-02-10T11:28:44.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:44.788 7fb8fb7fe700 1 -- 172.21.2.112:0/814570213 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb8ec04e510 con 0x7fb8e8006780 2021-02-10T11:28:44.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:44.788 7fb902d34700 1 -- 172.21.2.112:0/814570213 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb8e8006780 msgr2=0x7fb8e8008c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:44.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:44.788 7fb902d34700 1 --2- 172.21.2.112:0/814570213 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb8e8006780 0x7fb8e8008c80 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 rx=0x7fb8ec001c60 tx=0x7fb8ec007d40).stop 2021-02-10T11:28:44.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:44.788 7fb901351700 1 -- 172.21.2.112:0/814570213 reap_dead start 2021-02-10T11:28:44.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:44.789 7fb902d34700 1 -- 172.21.2.112:0/814570213 shutdown_connections 2021-02-10T11:28:44.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:44.789 7fb902d34700 1 -- 172.21.2.112:0/814570213 >> 172.21.2.112:0/814570213 conn(0x7fb8fc0ce3e0 msgr2=0x7fb8fc0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:28:44.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:44.789 7fb902d34700 1 -- 172.21.2.112:0/814570213 shutdown_connections 2021-02-10T11:28:44.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:44.789 7fb902d34700 1 -- 172.21.2.112:0/814570213 wait complete. 2021-02-10T11:28:44.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:44.804 7fb902d34700 1 Processor -- start 2021-02-10T11:28:44.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:44.804 7fb902d34700 1 -- start start 2021-02-10T11:28:44.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:44.805 7fb902d34700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb8fc0b7db0 0x7fb8fc0b8260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:44.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:44.805 7fb902d34700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb8fc0b8880 0x7fb8fc1a7450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:44.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:44.805 7fb902d34700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb8fc0b8c80 0x7fb8fc1a7a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:44.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:44.805 7fb902d34700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8fc0b79b0 con 0x7fb8fc0b8c80 2021-02-10T11:28:44.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:44.805 7fb902d34700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8fc1a8090 con 0x7fb8fc0b8880 2021-02-10T11:28:44.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:44.805 7fb902d34700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8fc1a82b0 con 0x7fb8fc0b7db0 2021-02-10T11:28:44.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:44.805 7fb901351700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb8fc0b8c80 0x7fb8fc1a7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:44.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:44.805 7fb900b50700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb8fc0b7db0 0x7fb8fc0b8260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:44.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:44.805 7fb8fbfff700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb8fc0b8880 0x7fb8fc1a7450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:44.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:44.805 7fb901351700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb8fc0b8c80 0x7fb8fc1a7a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:46356/0 (socket says 172.21.2.112:46356) 2021-02-10T11:28:44.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:44.805 7fb901351700 1 -- 172.21.2.112:0/1478088569 learned_addr learned my addr 172.21.2.112:0/1478088569 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:28:44.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:44.805 7fb8fbfff700 1 -- 172.21.2.112:0/1478088569 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb8fc0b7db0 msgr2=0x7fb8fc0b8260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:44.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:44.805 7fb8fbfff700 1 --2- 172.21.2.112:0/1478088569 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb8fc0b7db0 0x7fb8fc0b8260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:44.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:44.805 7fb8fbfff700 1 -- 172.21.2.112:0/1478088569 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb8fc0b8c80 msgr2=0x7fb8fc1a7a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:44.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:44.805 7fb8fbfff700 1 --2- 172.21.2.112:0/1478088569 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb8fc0b8c80 0x7fb8fc1a7a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:44.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:44.805 7fb8fbfff700 1 -- 172.21.2.112:0/1478088569 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8fc190c10 con 0x7fb8fc0b8880 2021-02-10T11:28:44.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:44.805 7fb900b50700 1 --2- 172.21.2.112:0/1478088569 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb8fc0b7db0 0x7fb8fc0b8260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:28:44.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:44.805 7fb8fbfff700 1 --2- 172.21.2.112:0/1478088569 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb8fc0b8880 0x7fb8fc1a7450 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7fb8ec000e30 tx=0x7fb8ec04f870).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:44.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:44.806 7fb8f97fa700 1 -- 172.21.2.112:0/1478088569 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb8ec007630 con 0x7fb8fc0b8880 2021-02-10T11:28:44.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:44.806 7fb8f97fa700 1 -- 172.21.2.112:0/1478088569 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8ec04fc60 con 0x7fb8fc0b8880 2021-02-10T11:28:44.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:44.806 7fb8f97fa700 1 -- 172.21.2.112:0/1478088569 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb8ec0502f0 con 0x7fb8fc0b8880 2021-02-10T11:28:44.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:44.806 7fb902d34700 1 -- 172.21.2.112:0/1478088569 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8fc190c10 con 0x7fb8fc0b8880 2021-02-10T11:28:44.800 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:44.806 7fb902d34700 1 -- 172.21.2.112:0/1478088569 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8fc0c1eb0 con 0x7fb8fc0b8880 2021-02-10T11:28:44.801 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:44.806 7fb8f97fa700 1 -- 172.21.2.112:0/1478088569 <== mon.1 v2:172.21.2.120:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb8ec04c050 con 0x7fb8fc0b8880 2021-02-10T11:28:44.801 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:44.806 7fb8f97fa700 1 --2- 172.21.2.112:0/1478088569 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb8e00186c0 0x7fb8e001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:44.801 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:44.806 7fb8f97fa700 1 -- 172.21.2.112:0/1478088569 <== mon.1 v2:172.21.2.120:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fb8ec076480 con 0x7fb8fc0b8880 2021-02-10T11:28:44.801 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:44.807 7fb900b50700 1 --2- 172.21.2.112:0/1478088569 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb8e00186c0 0x7fb8e001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:44.802 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:44.807 7fb900b50700 1 --2- 172.21.2.112:0/1478088569 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb8e00186c0 0x7fb8e001abc0 crc :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:44.824 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:44.836 7fb902d34700 1 -- 172.21.2.112:0/1478088569 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8fc105b30 con 0x7fb8fc0b8880 2021-02-10T11:28:44.827 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:44.839 7fb8f97fa700 1 -- 172.21.2.112:0/1478088569 <== mon.1 v2:172.21.2.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb8ec007610 con 0x7fb8fc0b8880 2021-02-10T11:28:44.956 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:44.969 7fb902d34700 1 -- 172.21.2.112:0/1478088569 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348659", "target": ["mgr", ""]}) v1 -- 0x7fb8fc0c1eb0 con 0x7fb8e00186c0 2021-02-10T11:28:44.982 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:44.994 7fb8f97fa700 1 -- 172.21.2.112:0/1478088569 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fb8e40009b0 con 0x7fb8e00186c0 2021-02-10T11:28:44.990 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:45.002 7fb902d34700 1 -- 172.21.2.112:0/1478088569 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb8e00186c0 msgr2=0x7fb8e001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:44.990 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:45.002 7fb902d34700 1 --2- 172.21.2.112:0/1478088569 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb8e00186c0 0x7fb8e001abc0 crc :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:44.990 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:45.002 7fb902d34700 1 -- 172.21.2.112:0/1478088569 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb8fc0b8880 msgr2=0x7fb8fc1a7450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:44.990 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:45.002 7fb902d34700 1 --2- 172.21.2.112:0/1478088569 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb8fc0b8880 0x7fb8fc1a7450 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7fb8ec000e30 tx=0x7fb8ec04f870).stop 2021-02-10T11:28:44.990 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:45.002 7fb902d34700 1 -- 172.21.2.112:0/1478088569 shutdown_connections 2021-02-10T11:28:44.991 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:45.002 7fb902d34700 1 --2- 172.21.2.112:0/1478088569 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb8e00186c0 0x7fb8e001abc0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:44.991 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:45.002 7fb902d34700 1 --2- 172.21.2.112:0/1478088569 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb8fc0b8880 0x7fb8fc1a7450 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:44.991 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:45.002 7fb902d34700 1 --2- 172.21.2.112:0/1478088569 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb8fc0b8c80 0x7fb8fc1a7a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:44.991 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:45.002 7fb902d34700 1 --2- 172.21.2.112:0/1478088569 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb8fc0b7db0 0x7fb8fc0b8260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:44.991 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:45.002 7fb902d34700 1 -- 172.21.2.112:0/1478088569 >> 172.21.2.112:0/1478088569 conn(0x7fb8fc094e10 msgr2=0x7fb8fc0a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:28:44.991 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:45.002 7fb902d34700 1 -- 172.21.2.112:0/1478088569 shutdown_connections 2021-02-10T11:28:44.991 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:45.002 7fb902d34700 1 -- 172.21.2.112:0/1478088569 wait complete. 2021-02-10T11:28:44.998 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000348659 2021-02-10T11:28:45.430 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:45.426 7f25dcfc6700 1 Processor -- start 2021-02-10T11:28:45.430 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:45.426 7f25dcfc6700 1 -- start start 2021-02-10T11:28:45.430 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:45.426 7f25dcfc6700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f25d81b0470 0x7f25d81b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:45.430 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:45.426 7f25dcfc6700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f25d80ef610 0x7f25d80efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:45.430 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:45.426 7f25dcfc6700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f25d80f0030 0x7f25d80f0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:45.430 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:45.426 7f25dcfc6700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25d80e5ea0 con 0x7f25d81b0470 2021-02-10T11:28:45.430 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:45.426 7f25dcfc6700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25d80f0b50 con 0x7f25d80ef610 2021-02-10T11:28:45.431 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:45.426 7f25dcfc6700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25d80e52e0 con 0x7f25d80f0030 2021-02-10T11:28:45.431 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:45.427 7f25d6d9d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f25d81b0470 0x7f25d81b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:45.431 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:45.427 7f25d6d9d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f25d81b0470 0x7f25d81b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:46360/0 (socket says 172.21.2.112:46360) 2021-02-10T11:28:45.431 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:45.427 7f25d6d9d700 1 -- 172.21.2.112:0/148834285 learned_addr learned my addr 172.21.2.112:0/148834285 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:28:45.431 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:45.427 7f25d6d9d700 1 -- 172.21.2.112:0/148834285 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f25d80f0030 msgr2=0x7f25d80f0430 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:28:45.431 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:45.427 7f25d6d9d700 1 --2- 172.21.2.112:0/148834285 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f25d80f0030 0x7f25d80f0430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:45.431 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:45.427 7f25d6d9d700 1 -- 172.21.2.112:0/148834285 >> v2:172.21.2.120:3300/0 conn(0x7f25d80ef610 msgr2=0x7f25d80efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:28:45.431 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:45.427 7f25d6d9d700 1 --2- 172.21.2.112:0/148834285 >> v2:172.21.2.120:3300/0 conn(0x7f25d80ef610 0x7f25d80efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:45.431 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:45.427 7f25d6d9d700 1 -- 172.21.2.112:0/148834285 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25d80f10d0 con 0x7f25d81b0470 2021-02-10T11:28:45.431 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:45.427 7f25d6d9d700 1 --2- 172.21.2.112:0/148834285 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f25d81b0470 0x7f25d81b0870 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 rx=0x7f25c8006700 tx=0x7f25c8009b00).ready entity=mon.0 client_cookie=c7140705cb6534c8 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:45.431 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:45.427 7f25d5d9b700 1 -- 172.21.2.112:0/148834285 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f25c800c0c0 con 0x7f25d81b0470 2021-02-10T11:28:45.432 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:45.427 7f25d5d9b700 1 -- 172.21.2.112:0/148834285 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25c800c340 con 0x7f25d81b0470 2021-02-10T11:28:45.432 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:45.427 7f25d5d9b700 1 -- 172.21.2.112:0/148834285 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f25c800c820 con 0x7f25d81b0470 2021-02-10T11:28:45.432 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:45.427 7f25dcfc6700 1 -- 172.21.2.112:0/148834285 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f25d81b0470 msgr2=0x7f25d81b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:45.432 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:45.427 7f25dcfc6700 1 --2- 172.21.2.112:0/148834285 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f25d81b0470 0x7f25d81b0870 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 rx=0x7f25c8006700 tx=0x7f25c8009b00).stop 2021-02-10T11:28:45.432 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:45.428 7f25dcfc6700 1 -- 172.21.2.112:0/148834285 shutdown_connections 2021-02-10T11:28:45.432 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:45.428 7f25dcfc6700 1 --2- 172.21.2.112:0/148834285 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f25d80f0030 0x7f25d80f0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:45.432 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:45.428 7f25dcfc6700 1 --2- 172.21.2.112:0/148834285 >> v2:172.21.2.120:3300/0 conn(0x7f25d80ef610 0x7f25d80efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:45.432 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:45.428 7f25dcfc6700 1 --2- 172.21.2.112:0/148834285 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f25d81b0470 0x7f25d81b0870 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:45.432 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:45.428 7f25dcfc6700 1 -- 172.21.2.112:0/148834285 >> 172.21.2.112:0/148834285 conn(0x7f25d80ce3e0 msgr2=0x7f25d80e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:28:45.432 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:45.428 7f25dcfc6700 1 -- 172.21.2.112:0/148834285 shutdown_connections 2021-02-10T11:28:45.432 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:45.428 7f25dcfc6700 1 -- 172.21.2.112:0/148834285 wait complete. 2021-02-10T11:28:45.434 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:45.442 7f25dcfc6700 1 Processor -- start 2021-02-10T11:28:45.434 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:45.442 7f25dcfc6700 1 -- start start 2021-02-10T11:28:45.434 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:45.443 7f25dcfc6700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f25d8193230 0x7f25d80a6a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:45.434 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:45.443 7f25dcfc6700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f25d808f5b0 0x7f25d808f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:45.434 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:45.443 7f25dcfc6700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f25d808e6a0 0x7f25d808eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:45.434 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:45.443 7f25dcfc6700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25d808ffd0 con 0x7f25d808f5b0 2021-02-10T11:28:45.434 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:45.443 7f25dcfc6700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25d8090240 con 0x7f25d808e6a0 2021-02-10T11:28:45.434 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:45.443 7f25dcfc6700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25d808f0c0 con 0x7f25d8193230 2021-02-10T11:28:45.435 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:45.443 7f25d659c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f25d808f5b0 0x7f25d808f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:45.435 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:45.443 7f25d659c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f25d808f5b0 0x7f25d808f9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:46368/0 (socket says 172.21.2.112:46368) 2021-02-10T11:28:45.435 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:45.443 7f25d659c700 1 -- 172.21.2.112:0/2920323837 learned_addr learned my addr 172.21.2.112:0/2920323837 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:28:45.435 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:45.443 7f25d659c700 1 -- 172.21.2.112:0/2920323837 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f25d8193230 msgr2=0x7f25d80a6a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:45.435 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:45.443 7f25d659c700 1 --2- 172.21.2.112:0/2920323837 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f25d8193230 0x7f25d80a6a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:45.435 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:45.443 7f25d659c700 1 -- 172.21.2.112:0/2920323837 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f25d808e6a0 msgr2=0x7f25d808eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:45.435 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:45.443 7f25d659c700 1 --2- 172.21.2.112:0/2920323837 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f25d808e6a0 0x7f25d808eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:45.435 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:45.443 7f25d659c700 1 -- 172.21.2.112:0/2920323837 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25d808f2a0 con 0x7f25d808f5b0 2021-02-10T11:28:45.435 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:45.443 7f25d659c700 1 --2- 172.21.2.112:0/2920323837 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f25d808f5b0 0x7f25d808f9b0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 rx=0x7f25c0006f00 tx=0x7f25c000b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:45.435 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:45.444 7f25cf7fe700 1 -- 172.21.2.112:0/2920323837 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f25c0006350 con 0x7f25d808f5b0 2021-02-10T11:28:45.435 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:45.444 7f25cf7fe700 1 -- 172.21.2.112:0/2920323837 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25c000bdc0 con 0x7f25d808f5b0 2021-02-10T11:28:45.435 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:45.444 7f25cf7fe700 1 -- 172.21.2.112:0/2920323837 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f25c0002020 con 0x7f25d808f5b0 2021-02-10T11:28:45.435 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:45.444 7f25dcfc6700 1 -- 172.21.2.112:0/2920323837 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25d808f2a0 con 0x7f25d808f5b0 2021-02-10T11:28:45.436 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:45.444 7f25dcfc6700 1 -- 172.21.2.112:0/2920323837 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25d81924f0 con 0x7f25d808f5b0 2021-02-10T11:28:45.436 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:45.444 7f25cf7fe700 1 -- 172.21.2.112:0/2920323837 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f25c0005310 con 0x7f25d808f5b0 2021-02-10T11:28:45.436 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:45.444 7f25cf7fe700 1 --2- 172.21.2.112:0/2920323837 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f25bc0186c0 0x7f25bc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:45.436 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:45.444 7f25cf7fe700 1 -- 172.21.2.112:0/2920323837 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f25c0032510 con 0x7f25d808f5b0 2021-02-10T11:28:45.436 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:45.445 7f25d6d9d700 1 --2- 172.21.2.112:0/2920323837 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f25bc0186c0 0x7f25bc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:45.436 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:45.445 7f25d6d9d700 1 --2- 172.21.2.112:0/2920323837 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f25bc0186c0 0x7f25bc01abc0 crc :-1 s=READY pgs=363 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:45.464 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:45.476 7f25dcfc6700 1 -- 172.21.2.112:0/2920323837 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25d80f0b50 con 0x7f25d808f5b0 2021-02-10T11:28:45.466 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:45.479 7f25cf7fe700 1 -- 172.21.2.112:0/2920323837 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f25c0021690 con 0x7f25d808f5b0 2021-02-10T11:28:45.595 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:45.607 7f25dcfc6700 1 -- 172.21.2.112:0/2920323837 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348659", "target": ["mgr", ""]}) v1 -- 0x7f25d808f2a0 con 0x7f25bc0186c0 2021-02-10T11:28:45.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:45.611 7f25cf7fe700 1 -- 172.21.2.112:0/2920323837 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f25c8009550 con 0x7f25bc0186c0 2021-02-10T11:28:45.602 INFO:teuthology.orchestra.run.gibba012.stdout:/volumes/_nogroup/subvolume_0000000000348659/00bcdbbc-2b27-4265-9375-39bc137d7cd1 2021-02-10T11:28:45.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:45.616 7f25dcfc6700 1 -- 172.21.2.112:0/2920323837 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f25bc0186c0 msgr2=0x7f25bc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:45.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:45.616 7f25dcfc6700 1 --2- 172.21.2.112:0/2920323837 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f25bc0186c0 0x7f25bc01abc0 crc :-1 s=READY pgs=363 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:45.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:45.616 7f25dcfc6700 1 -- 172.21.2.112:0/2920323837 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f25d808f5b0 msgr2=0x7f25d808f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:45.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:45.616 7f25dcfc6700 1 --2- 172.21.2.112:0/2920323837 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f25d808f5b0 0x7f25d808f9b0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 rx=0x7f25c0006f00 tx=0x7f25c000b940).stop 2021-02-10T11:28:45.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:45.617 7f25dcfc6700 1 -- 172.21.2.112:0/2920323837 shutdown_connections 2021-02-10T11:28:45.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:45.617 7f25dcfc6700 1 --2- 172.21.2.112:0/2920323837 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f25bc0186c0 0x7f25bc01abc0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:45.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:45.617 7f25dcfc6700 1 --2- 172.21.2.112:0/2920323837 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f25d808e6a0 0x7f25d808eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:45.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:45.617 7f25dcfc6700 1 --2- 172.21.2.112:0/2920323837 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f25d808f5b0 0x7f25d808f9b0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:45.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:45.617 7f25dcfc6700 1 --2- 172.21.2.112:0/2920323837 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f25d8193230 0x7f25d80a6a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:45.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:45.617 7f25dcfc6700 1 -- 172.21.2.112:0/2920323837 >> 172.21.2.112:0/2920323837 conn(0x7f25d80bf4e0 msgr2=0x7f25d80be160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:28:45.606 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:45.617 7f25dcfc6700 1 -- 172.21.2.112:0/2920323837 shutdown_connections 2021-02-10T11:28:45.606 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:45.617 7f25dcfc6700 1 -- 172.21.2.112:0/2920323837 wait complete. 2021-02-10T11:28:45.613 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000348659 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000348659/00bcdbbc-2b27-4265-9375-39bc137d7cd1 2021-02-10T11:28:45.613 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:45.613 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:45.613 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:45.948 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:28:45.949 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:45.949 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348659/00bcdbbc-2b27-4265-9375-39bc137d7cd1/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:46.333 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:28:46.333 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:28:46.333 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0256973 s, 40.8 MB/s 2021-02-10T11:28:46.334 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:46.335 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:46.335 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:46.672 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:28:46.672 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:46.672 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348659/00bcdbbc-2b27-4265-9375-39bc137d7cd1/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:47.049 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:28:47.049 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:28:47.049 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0250279 s, 41.9 MB/s 2021-02-10T11:28:47.050 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:47.050 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:47.051 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:47.387 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:28:47.387 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:47.387 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348659/00bcdbbc-2b27-4265-9375-39bc137d7cd1/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:47.765 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:28:47.766 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:28:47.766 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0248233 s, 42.2 MB/s 2021-02-10T11:28:47.767 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:47.767 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:47.768 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:48.102 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:28:48.103 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:48.103 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348659/00bcdbbc-2b27-4265-9375-39bc137d7cd1/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:48.479 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:28:48.479 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:28:48.479 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.024188 s, 43.4 MB/s 2021-02-10T11:28:48.480 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:48.480 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:48.481 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:48.816 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:28:48.817 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:48.817 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348659/00bcdbbc-2b27-4265-9375-39bc137d7cd1/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:49.193 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:28:49.193 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:28:49.193 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0232006 s, 45.2 MB/s 2021-02-10T11:28:49.194 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:49.194 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:49.194 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:49.529 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:28:49.530 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:49.530 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348659/00bcdbbc-2b27-4265-9375-39bc137d7cd1/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:49.904 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:28:49.904 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:28:49.905 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0254005 s, 41.3 MB/s 2021-02-10T11:28:49.905 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:49.906 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:49.906 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:50.241 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:28:50.241 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:50.241 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348659/00bcdbbc-2b27-4265-9375-39bc137d7cd1/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:50.617 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:28:50.618 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:28:50.618 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0251782 s, 41.6 MB/s 2021-02-10T11:28:50.619 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:50.619 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:50.619 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:50.954 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:28:50.955 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:50.955 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348659/00bcdbbc-2b27-4265-9375-39bc137d7cd1/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:51.332 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:28:51.332 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:28:51.332 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0247094 s, 42.4 MB/s 2021-02-10T11:28:51.333 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:51.333 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:51.333 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:51.669 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:28:51.670 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:51.670 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348659/00bcdbbc-2b27-4265-9375-39bc137d7cd1/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:52.046 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:28:52.046 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:28:52.046 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0254111 s, 41.3 MB/s 2021-02-10T11:28:52.047 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:52.047 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:52.047 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:52.383 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:28:52.383 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:52.384 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348659/00bcdbbc-2b27-4265-9375-39bc137d7cd1/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:52.761 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:28:52.762 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:28:52.762 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0265253 s, 39.5 MB/s 2021-02-10T11:28:52.763 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000348660 2021-02-10T11:28:53.198 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.194 7fada94e6700 1 Processor -- start 2021-02-10T11:28:53.198 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.194 7fada94e6700 1 -- start start 2021-02-10T11:28:53.198 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.195 7fada94e6700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fada41b0870 0x7fada40f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:53.199 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.195 7fada94e6700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fada40efe00 0x7fada40f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:53.199 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.195 7fada94e6700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fada41b0c70 con 0x7fada41b0470 2021-02-10T11:28:53.199 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.195 7fada94e6700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fada40e5ea0 con 0x7fada40efe00 2021-02-10T11:28:53.199 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.195 7fada94e6700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fada40f0b50 con 0x7fada41b0870 2021-02-10T11:28:53.199 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.195 7fada3ac0700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fada40efe00 0x7fada40f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:53.199 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.195 7fada3ac0700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fada40efe00 0x7fada40f0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:46376/0 (socket says 172.21.2.112:46376) 2021-02-10T11:28:53.199 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.195 7fada3ac0700 1 -- 172.21.2.112:0/1439269247 learned_addr learned my addr 172.21.2.112:0/1439269247 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:28:53.199 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.195 7fada3ac0700 1 -- 172.21.2.112:0/1439269247 >> v2:172.21.2.120:3300/0 conn(0x7fada41b0870 msgr2=0x7fada40f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:28:53.199 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.195 7fada3ac0700 1 --2- 172.21.2.112:0/1439269247 >> v2:172.21.2.120:3300/0 conn(0x7fada41b0870 0x7fada40f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:53.199 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.195 7fada3ac0700 1 -- 172.21.2.112:0/1439269247 >> v1:172.21.2.112:6789/0 conn(0x7fada41b0470 legacy=0x7fada40ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:53.199 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.195 7fada3ac0700 1 -- 172.21.2.112:0/1439269247 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fada40f0820 con 0x7fada40efe00 2021-02-10T11:28:53.200 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.195 7fada3ac0700 1 --2- 172.21.2.112:0/1439269247 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fada40efe00 0x7fada40f0200 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 rx=0x7fad9c006f00 tx=0x7fad9c009e90).ready entity=mon.0 client_cookie=d24ffe353a38442f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:53.200 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.195 7fada22bd700 1 -- 172.21.2.112:0/1439269247 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fad9c00b300 con 0x7fada40efe00 2021-02-10T11:28:53.200 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.196 7fada22bd700 1 -- 172.21.2.112:0/1439269247 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad9c00b640 con 0x7fada40efe00 2021-02-10T11:28:53.200 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.196 7fada22bd700 1 -- 172.21.2.112:0/1439269247 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fad9c00bb20 con 0x7fada40efe00 2021-02-10T11:28:53.200 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.196 7fada94e6700 1 -- 172.21.2.112:0/1439269247 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fada40efe00 msgr2=0x7fada40f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:53.200 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.196 7fada94e6700 1 --2- 172.21.2.112:0/1439269247 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fada40efe00 0x7fada40f0200 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 rx=0x7fad9c006f00 tx=0x7fad9c009e90).stop 2021-02-10T11:28:53.200 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.196 7fada94e6700 1 -- 172.21.2.112:0/1439269247 shutdown_connections 2021-02-10T11:28:53.200 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.196 7fada94e6700 1 --2- 172.21.2.112:0/1439269247 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fada40efe00 0x7fada40f0200 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:53.200 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.196 7fada94e6700 1 --2- 172.21.2.112:0/1439269247 >> v2:172.21.2.120:3300/0 conn(0x7fada41b0870 0x7fada40f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:53.200 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.196 7fada94e6700 1 -- 172.21.2.112:0/1439269247 >> 172.21.2.112:0/1439269247 conn(0x7fada40ce3e0 msgr2=0x7fada40e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:28:53.200 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.196 7fada94e6700 1 -- 172.21.2.112:0/1439269247 shutdown_connections 2021-02-10T11:28:53.200 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.196 7fada94e6700 1 -- 172.21.2.112:0/1439269247 wait complete. 2021-02-10T11:28:53.202 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.211 7fada94e6700 1 Processor -- start 2021-02-10T11:28:53.202 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.211 7fada94e6700 1 -- start start 2021-02-10T11:28:53.202 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.211 7fada94e6700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fada40b2840 0x7fada41a4300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:53.202 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.211 7fada94e6700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fada40b30d0 0x7fada40b34d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:53.202 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.211 7fada94e6700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fada40b3af0 0x7fada40f0580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:53.203 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.211 7fada94e6700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fada40b2c40 con 0x7fada40b3af0 2021-02-10T11:28:53.203 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.211 7fada94e6700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fada41a4920 con 0x7fada40b2840 2021-02-10T11:28:53.203 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.211 7fada94e6700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fada40f0ba0 con 0x7fada40b30d0 2021-02-10T11:28:53.203 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.212 7fada3ac0700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fada40b3af0 0x7fada40f0580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:53.203 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.212 7fada3ac0700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fada40b3af0 0x7fada40f0580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:46382/0 (socket says 172.21.2.112:46382) 2021-02-10T11:28:53.203 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.212 7fada3ac0700 1 -- 172.21.2.112:0/1655190269 learned_addr learned my addr 172.21.2.112:0/1655190269 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:28:53.203 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.212 7fada32bf700 1 --2- 172.21.2.112:0/1655190269 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fada40b2840 0x7fada41a4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:53.203 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.212 7fada2abe700 1 --2- 172.21.2.112:0/1655190269 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fada40b30d0 0x7fada40b34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:53.203 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.212 7fada3ac0700 1 -- 172.21.2.112:0/1655190269 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fada40b30d0 msgr2=0x7fada40b34d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:53.203 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.212 7fada3ac0700 1 --2- 172.21.2.112:0/1655190269 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fada40b30d0 0x7fada40b34d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:53.203 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.212 7fada3ac0700 1 -- 172.21.2.112:0/1655190269 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fada40b2840 msgr2=0x7fada41a4300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:53.203 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.212 7fada3ac0700 1 --2- 172.21.2.112:0/1655190269 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fada40b2840 0x7fada41a4300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:53.203 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.212 7fada3ac0700 1 -- 172.21.2.112:0/1655190269 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fada40f0dc0 con 0x7fada40b3af0 2021-02-10T11:28:53.204 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.212 7fada2abe700 1 --2- 172.21.2.112:0/1655190269 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fada40b30d0 0x7fada40b34d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:28:53.204 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.212 7fada3ac0700 1 --2- 172.21.2.112:0/1655190269 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fada40b3af0 0x7fada40f0580 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 rx=0x7fad9c009fc0 tx=0x7fad9c00bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:53.204 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.212 7fad9bfff700 1 -- 172.21.2.112:0/1655190269 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fad9c00be10 con 0x7fada40b3af0 2021-02-10T11:28:53.204 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.212 7fad9bfff700 1 -- 172.21.2.112:0/1655190269 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad9c005820 con 0x7fada40b3af0 2021-02-10T11:28:53.204 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.212 7fad9bfff700 1 -- 172.21.2.112:0/1655190269 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fad9c005d00 con 0x7fada40b3af0 2021-02-10T11:28:53.204 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.212 7fada94e6700 1 -- 172.21.2.112:0/1655190269 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fada40f0dc0 con 0x7fada40b3af0 2021-02-10T11:28:53.204 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.212 7fada94e6700 1 -- 172.21.2.112:0/1655190269 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fada40f1350 con 0x7fada40b3af0 2021-02-10T11:28:53.204 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.213 7fad9bfff700 1 -- 172.21.2.112:0/1655190269 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fad9c005a00 con 0x7fada40b3af0 2021-02-10T11:28:53.204 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.213 7fad9bfff700 1 --2- 172.21.2.112:0/1655190269 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fad880186c0 0x7fad8801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:53.204 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.213 7fad9bfff700 1 -- 172.21.2.112:0/1655190269 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fad9c0328e0 con 0x7fada40b3af0 2021-02-10T11:28:53.205 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.213 7fada32bf700 1 --2- 172.21.2.112:0/1655190269 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fad880186c0 0x7fad8801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:53.205 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.214 7fada32bf700 1 --2- 172.21.2.112:0/1655190269 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fad880186c0 0x7fad8801abc0 crc :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:53.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.244 7fada94e6700 1 -- 172.21.2.112:0/1655190269 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fada40f18f0 con 0x7fada40b3af0 2021-02-10T11:28:53.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.247 7fad9bfff700 1 -- 172.21.2.112:0/1655190269 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fad9c000f90 con 0x7fada40b3af0 2021-02-10T11:28:53.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.374 7fada94e6700 1 -- 172.21.2.112:0/1655190269 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348660", "target": ["mgr", ""]}) v1 -- 0x7fada40f1350 con 0x7fad880186c0 2021-02-10T11:28:53.386 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.399 7fad9bfff700 1 -- 172.21.2.112:0/1655190269 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fad94006b60 con 0x7fad880186c0 2021-02-10T11:28:53.395 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.407 7fada94e6700 1 -- 172.21.2.112:0/1655190269 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fad880186c0 msgr2=0x7fad8801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:53.395 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.407 7fada94e6700 1 --2- 172.21.2.112:0/1655190269 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fad880186c0 0x7fad8801abc0 crc :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:53.395 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.407 7fada94e6700 1 -- 172.21.2.112:0/1655190269 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fada40b3af0 msgr2=0x7fada40f0580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:53.395 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.407 7fada94e6700 1 --2- 172.21.2.112:0/1655190269 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fada40b3af0 0x7fada40f0580 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 rx=0x7fad9c009fc0 tx=0x7fad9c00bcd0).stop 2021-02-10T11:28:53.395 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.407 7fada94e6700 1 -- 172.21.2.112:0/1655190269 shutdown_connections 2021-02-10T11:28:53.395 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.407 7fada94e6700 1 --2- 172.21.2.112:0/1655190269 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fad880186c0 0x7fad8801abc0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:53.395 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.407 7fada94e6700 1 --2- 172.21.2.112:0/1655190269 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fada40b3af0 0x7fada40f0580 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:53.395 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.407 7fada94e6700 1 --2- 172.21.2.112:0/1655190269 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fada40b30d0 0x7fada40b34d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:53.395 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.407 7fada94e6700 1 --2- 172.21.2.112:0/1655190269 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fada40b2840 0x7fada41a4300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:53.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.407 7fada94e6700 1 -- 172.21.2.112:0/1655190269 >> 172.21.2.112:0/1655190269 conn(0x7fada40c36f0 msgr2=0x7fada4094c30 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:28:53.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.407 7fada94e6700 1 -- 172.21.2.112:0/1655190269 shutdown_connections 2021-02-10T11:28:53.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.407 7fada94e6700 1 -- 172.21.2.112:0/1655190269 wait complete. 2021-02-10T11:28:53.403 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000348660 2021-02-10T11:28:53.839 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.833 7fa9c8453700 1 Processor -- start 2021-02-10T11:28:53.840 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.834 7fa9c8453700 1 -- start start 2021-02-10T11:28:53.840 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.834 7fa9c8453700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa9c01b0470 0x7fa9c01b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:53.840 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.834 7fa9c8453700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fa9c00ef610 0x7fa9c00efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:53.840 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.834 7fa9c8453700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9c00f0b50 con 0x7fa9c00f0030 2021-02-10T11:28:53.840 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.834 7fa9c8453700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9c00e52e0 con 0x7fa9c00ef610 2021-02-10T11:28:53.840 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.834 7fa9c8453700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9c00e5490 con 0x7fa9c01b0470 2021-02-10T11:28:53.840 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.835 7fa9c6a70700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7fa9c00f0030 0x7fa9c00f10d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:39412/0 (socket says 172.21.2.112:39412) 2021-02-10T11:28:53.840 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.835 7fa9c6a70700 1 -- 172.21.2.112:0/4224638893 learned_addr learned my addr 172.21.2.112:0/4224638893 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:28:53.840 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.835 7fa9c5a6e700 1 --2- 172.21.2.112:0/4224638893 >> v2:172.21.2.120:3300/0 conn(0x7fa9c00ef610 0x7fa9c00efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:53.841 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.835 7fa9c626f700 1 --2- 172.21.2.112:0/4224638893 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa9c01b0470 0x7fa9c01b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:53.841 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.835 7fa9c526d700 1 -- 172.21.2.112:0/4224638893 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3757096061 0 0) 0x7fa9b8001510 con 0x7fa9c00f0030 2021-02-10T11:28:53.841 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.835 7fa9c526d700 1 -- 172.21.2.112:0/4224638893 --> v1:172.21.2.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9ac002fe0 con 0x7fa9c00f0030 2021-02-10T11:28:53.841 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.835 7fa9c5a6e700 1 -- 172.21.2.112:0/4224638893 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa9c01b0470 msgr2=0x7fa9c01b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:28:53.841 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.835 7fa9c5a6e700 1 --2- 172.21.2.112:0/4224638893 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa9c01b0470 0x7fa9c01b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:53.841 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.835 7fa9c5a6e700 1 -- 172.21.2.112:0/4224638893 >> v1:172.21.2.120:6789/0 conn(0x7fa9c00f0030 legacy=0x7fa9c00f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:53.841 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.835 7fa9c5a6e700 1 -- 172.21.2.112:0/4224638893 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9c00f0430 con 0x7fa9c00ef610 2021-02-10T11:28:53.841 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.835 7fa9c626f700 1 --2- 172.21.2.112:0/4224638893 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa9c01b0470 0x7fa9c01b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:28:53.841 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.835 7fa9c5a6e700 1 --2- 172.21.2.112:0/4224638893 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa9c00ef610 0x7fa9c00efa10 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7fa9b0006650 tx=0x7fa9b0009a80).ready entity=mon.1 client_cookie=ee4e6b7703ae879 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:53.841 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.836 7fa9c526d700 1 -- 172.21.2.112:0/4224638893 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa9b00009b0 con 0x7fa9c00ef610 2021-02-10T11:28:53.841 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.836 7fa9c526d700 1 -- 172.21.2.112:0/4224638893 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9b000c0c0 con 0x7fa9c00ef610 2021-02-10T11:28:53.842 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.836 7fa9c526d700 1 -- 172.21.2.112:0/4224638893 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa9b000c5e0 con 0x7fa9c00ef610 2021-02-10T11:28:53.842 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.836 7fa9c8453700 1 -- 172.21.2.112:0/4224638893 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa9c00ef610 msgr2=0x7fa9c00efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:53.842 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.836 7fa9c8453700 1 --2- 172.21.2.112:0/4224638893 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa9c00ef610 0x7fa9c00efa10 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7fa9b0006650 tx=0x7fa9b0009a80).stop 2021-02-10T11:28:53.842 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.837 7fa9c8453700 1 -- 172.21.2.112:0/4224638893 shutdown_connections 2021-02-10T11:28:53.842 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.837 7fa9c8453700 1 --2- 172.21.2.112:0/4224638893 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa9c00ef610 0x7fa9c00efa10 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:53.842 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.837 7fa9c8453700 1 --2- 172.21.2.112:0/4224638893 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa9c01b0470 0x7fa9c01b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:53.842 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.837 7fa9c8453700 1 -- 172.21.2.112:0/4224638893 >> 172.21.2.112:0/4224638893 conn(0x7fa9c00ce3e0 msgr2=0x7fa9c00e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:28:53.842 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.837 7fa9c8453700 1 -- 172.21.2.112:0/4224638893 shutdown_connections 2021-02-10T11:28:53.842 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.837 7fa9c8453700 1 -- 172.21.2.112:0/4224638893 wait complete. 2021-02-10T11:28:53.842 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.852 7fa9c8453700 1 Processor -- start 2021-02-10T11:28:53.842 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.852 7fa9c8453700 1 -- start start 2021-02-10T11:28:53.844 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.852 7fa9c8453700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa9c00a0420 0x7fa9c00a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:53.844 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.852 7fa9c8453700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa9c0189d00 0x7fa9c018a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:53.845 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.852 7fa9c8453700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa9c018a720 0x7fa9c00b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:53.845 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.852 7fa9c8453700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9c0189800 con 0x7fa9c018a720 2021-02-10T11:28:53.845 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.852 7fa9c8453700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9c018ab20 con 0x7fa9c00a0420 2021-02-10T11:28:53.845 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.852 7fa9c8453700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9c00a1140 con 0x7fa9c0189d00 2021-02-10T11:28:53.845 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.853 7fa9c6a70700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa9c018a720 0x7fa9c00b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:53.845 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.853 7fa9c626f700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa9c00a0420 0x7fa9c00a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:53.845 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.853 7fa9c6a70700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa9c018a720 0x7fa9c00b7be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:46398/0 (socket says 172.21.2.112:46398) 2021-02-10T11:28:53.845 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.853 7fa9c6a70700 1 -- 172.21.2.112:0/4038369236 learned_addr learned my addr 172.21.2.112:0/4038369236 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:28:53.846 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.853 7fa9c5a6e700 1 --2- 172.21.2.112:0/4038369236 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa9c0189d00 0x7fa9c018a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:53.846 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.853 7fa9c6a70700 1 -- 172.21.2.112:0/4038369236 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa9c0189d00 msgr2=0x7fa9c018a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:53.846 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.853 7fa9c6a70700 1 --2- 172.21.2.112:0/4038369236 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa9c0189d00 0x7fa9c018a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:53.846 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.853 7fa9c6a70700 1 -- 172.21.2.112:0/4038369236 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa9c00a0420 msgr2=0x7fa9c00a0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:53.846 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.853 7fa9c6a70700 1 --2- 172.21.2.112:0/4038369236 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa9c00a0420 0x7fa9c00a0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:53.846 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.853 7fa9c6a70700 1 -- 172.21.2.112:0/4038369236 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9c00b8200 con 0x7fa9c018a720 2021-02-10T11:28:53.846 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.853 7fa9c626f700 1 --2- 172.21.2.112:0/4038369236 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa9c00a0420 0x7fa9c00a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:28:53.846 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.853 7fa9c5a6e700 1 --2- 172.21.2.112:0/4038369236 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa9c0189d00 0x7fa9c018a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:28:53.846 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.853 7fa9c6a70700 1 --2- 172.21.2.112:0/4038369236 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa9c018a720 0x7fa9c00b7be0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 rx=0x7fa9b8001ba0 tx=0x7fa9b800ac80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:53.846 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.853 7fa9b6ffd700 1 -- 172.21.2.112:0/4038369236 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa9b80070d0 con 0x7fa9c018a720 2021-02-10T11:28:53.846 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.853 7fa9b6ffd700 1 -- 172.21.2.112:0/4038369236 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9b8000f40 con 0x7fa9c018a720 2021-02-10T11:28:53.847 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.853 7fa9b6ffd700 1 -- 172.21.2.112:0/4038369236 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa9b80048b0 con 0x7fa9c018a720 2021-02-10T11:28:53.847 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.853 7fa9c8453700 1 -- 172.21.2.112:0/4038369236 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9c00b8200 con 0x7fa9c018a720 2021-02-10T11:28:53.847 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.853 7fa9c8453700 1 -- 172.21.2.112:0/4038369236 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9c00b8200 con 0x7fa9c018a720 2021-02-10T11:28:53.847 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.854 7fa9b6ffd700 1 -- 172.21.2.112:0/4038369236 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa9b8007240 con 0x7fa9c018a720 2021-02-10T11:28:53.847 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.854 7fa9b6ffd700 1 --2- 172.21.2.112:0/4038369236 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa9a80186c0 0x7fa9a801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:53.847 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.854 7fa9b6ffd700 1 -- 172.21.2.112:0/4038369236 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fa9b80329c0 con 0x7fa9c018a720 2021-02-10T11:28:53.847 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.855 7fa9c626f700 1 --2- 172.21.2.112:0/4038369236 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa9a80186c0 0x7fa9a801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:53.847 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.855 7fa9c626f700 1 --2- 172.21.2.112:0/4038369236 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa9a80186c0 0x7fa9a801abc0 crc :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:53.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.884 7fa9c8453700 1 -- 172.21.2.112:0/4038369236 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9c018ab20 con 0x7fa9c018a720 2021-02-10T11:28:53.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:53.886 7fa9b6ffd700 1 -- 172.21.2.112:0/4038369236 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa9b8006f20 con 0x7fa9c018a720 2021-02-10T11:28:54.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:54.016 7fa9c8453700 1 -- 172.21.2.112:0/4038369236 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348660", "target": ["mgr", ""]}) v1 -- 0x7fa9c018ab20 con 0x7fa9a80186c0 2021-02-10T11:28:54.007 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:54.019 7fa9b6ffd700 1 -- 172.21.2.112:0/4038369236 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7fa9bc000940 con 0x7fa9a80186c0 2021-02-10T11:28:54.010 INFO:teuthology.orchestra.run.gibba012.stdout:/volumes/_nogroup/subvolume_0000000000348660/8d8fa1cf-b9af-46a1-84c1-064725ae113c 2021-02-10T11:28:54.012 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:54.024 7fa9c8453700 1 -- 172.21.2.112:0/4038369236 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa9a80186c0 msgr2=0x7fa9a801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:54.012 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:54.024 7fa9c8453700 1 --2- 172.21.2.112:0/4038369236 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa9a80186c0 0x7fa9a801abc0 crc :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:54.012 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:54.024 7fa9c8453700 1 -- 172.21.2.112:0/4038369236 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa9c018a720 msgr2=0x7fa9c00b7be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:54.012 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:54.024 7fa9c8453700 1 --2- 172.21.2.112:0/4038369236 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa9c018a720 0x7fa9c00b7be0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 rx=0x7fa9b8001ba0 tx=0x7fa9b800ac80).stop 2021-02-10T11:28:54.013 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:54.024 7fa9c8453700 1 -- 172.21.2.112:0/4038369236 shutdown_connections 2021-02-10T11:28:54.013 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:54.024 7fa9c8453700 1 --2- 172.21.2.112:0/4038369236 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa9a80186c0 0x7fa9a801abc0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:54.013 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:54.024 7fa9c8453700 1 --2- 172.21.2.112:0/4038369236 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa9c018a720 0x7fa9c00b7be0 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:54.013 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:54.024 7fa9c8453700 1 --2- 172.21.2.112:0/4038369236 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa9c0189d00 0x7fa9c018a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:54.013 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:54.024 7fa9c8453700 1 --2- 172.21.2.112:0/4038369236 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa9c00a0420 0x7fa9c00a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:54.013 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:54.024 7fa9c8453700 1 -- 172.21.2.112:0/4038369236 >> 172.21.2.112:0/4038369236 conn(0x7fa9c0094e10 msgr2=0x7fa9c00a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:28:54.013 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:54.025 7fa9c8453700 1 -- 172.21.2.112:0/4038369236 shutdown_connections 2021-02-10T11:28:54.013 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:28:54.025 7fa9c8453700 1 -- 172.21.2.112:0/4038369236 wait complete. 2021-02-10T11:28:54.020 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000348660 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000348660/8d8fa1cf-b9af-46a1-84c1-064725ae113c 2021-02-10T11:28:54.021 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:54.021 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:54.021 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:54.355 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:28:54.355 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:54.356 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348660/8d8fa1cf-b9af-46a1-84c1-064725ae113c/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:54.736 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:28:54.736 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:28:54.736 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.025521 s, 41.1 MB/s 2021-02-10T11:28:54.737 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:54.737 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:54.737 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:55.074 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:28:55.074 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:55.074 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348660/8d8fa1cf-b9af-46a1-84c1-064725ae113c/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:55.485 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:28:55.485 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:28:55.485 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0238869 s, 43.9 MB/s 2021-02-10T11:28:55.486 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:55.486 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:55.487 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:55.822 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:28:55.822 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:55.823 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348660/8d8fa1cf-b9af-46a1-84c1-064725ae113c/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:56.201 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:28:56.201 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:28:56.201 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0259808 s, 40.4 MB/s 2021-02-10T11:28:56.202 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:56.202 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:56.202 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:56.539 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:28:56.539 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:56.539 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348660/8d8fa1cf-b9af-46a1-84c1-064725ae113c/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:56.915 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:28:56.915 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:28:56.915 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0235605 s, 44.5 MB/s 2021-02-10T11:28:56.916 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:56.916 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:56.916 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:57.252 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:28:57.252 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:57.252 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348660/8d8fa1cf-b9af-46a1-84c1-064725ae113c/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:57.629 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:28:57.629 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:28:57.629 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0245948 s, 42.6 MB/s 2021-02-10T11:28:57.630 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:57.630 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:57.630 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:57.965 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:28:57.966 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:57.966 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348660/8d8fa1cf-b9af-46a1-84c1-064725ae113c/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:58.342 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:28:58.343 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:28:58.343 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0230912 s, 45.4 MB/s 2021-02-10T11:28:58.344 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:58.344 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:58.344 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:58.678 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:28:58.678 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:58.678 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348660/8d8fa1cf-b9af-46a1-84c1-064725ae113c/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:59.055 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:28:59.055 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:28:59.056 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0259116 s, 40.5 MB/s 2021-02-10T11:28:59.056 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:59.057 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:59.057 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:59.392 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:28:59.393 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:59.393 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348660/8d8fa1cf-b9af-46a1-84c1-064725ae113c/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:59.765 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:28:59.766 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:28:59.766 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0250669 s, 41.8 MB/s 2021-02-10T11:28:59.767 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:59.767 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:59.767 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:00.102 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:29:00.103 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:00.103 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348660/8d8fa1cf-b9af-46a1-84c1-064725ae113c/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:00.479 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:29:00.479 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:29:00.479 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0246777 s, 42.5 MB/s 2021-02-10T11:29:00.480 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:00.480 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:00.481 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:00.816 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:29:00.817 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:00.817 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348660/8d8fa1cf-b9af-46a1-84c1-064725ae113c/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:01.195 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:29:01.195 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:29:01.195 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0247187 s, 42.4 MB/s 2021-02-10T11:29:01.196 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000348661 2021-02-10T11:29:01.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:01.622 7fb04262a700 1 Processor -- start 2021-02-10T11:29:01.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:01.622 7fb04262a700 1 -- start start 2021-02-10T11:29:01.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:01.623 7fb04262a700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fb03c1b0870 0x7fb03c0f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:01.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:01.623 7fb04262a700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fb03c0efe00 0x7fb03c0f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:01.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:01.623 7fb04262a700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb03c1b0c70 con 0x7fb03c1b0470 2021-02-10T11:29:01.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:01.623 7fb04262a700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb03c0e5ea0 con 0x7fb03c1b0870 2021-02-10T11:29:01.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:01.623 7fb04262a700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb03c0f0b50 con 0x7fb03c0efe00 2021-02-10T11:29:01.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:01.623 7fb03b7fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fb03c1b0870 0x7fb03c0f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:01.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:01.623 7fb03b7fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fb03c1b0870 0x7fb03c0f20e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:46402/0 (socket says 172.21.2.112:46402) 2021-02-10T11:29:01.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:01.623 7fb03b7fe700 1 -- 172.21.2.112:0/1237595221 learned_addr learned my addr 172.21.2.112:0/1237595221 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:29:01.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:01.623 7fb03b7fe700 1 -- 172.21.2.112:0/1237595221 >> v2:172.21.2.120:3300/0 conn(0x7fb03c0efe00 msgr2=0x7fb03c0f0200 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:29:01.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:01.623 7fb03b7fe700 1 --2- 172.21.2.112:0/1237595221 >> v2:172.21.2.120:3300/0 conn(0x7fb03c0efe00 0x7fb03c0f0200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:01.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:01.623 7fb03b7fe700 1 -- 172.21.2.112:0/1237595221 >> v1:172.21.2.120:6789/0 conn(0x7fb03c1b0470 legacy=0x7fb03c0ef610 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:29:01.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:01.623 7fb03b7fe700 1 -- 172.21.2.112:0/1237595221 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb03c0f0820 con 0x7fb03c1b0870 2021-02-10T11:29:01.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:01.623 7fb03b7fe700 1 --2- 172.21.2.112:0/1237595221 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb03c1b0870 0x7fb03c0f20e0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 rx=0x7fb030006700 tx=0x7fb030009b80).ready entity=mon.0 client_cookie=75a5429712da4591 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:01.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:01.624 7fb03affd700 1 -- 172.21.2.112:0/1237595221 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb03000c0c0 con 0x7fb03c1b0870 2021-02-10T11:29:01.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:01.624 7fb03affd700 1 -- 172.21.2.112:0/1237595221 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb03000c440 con 0x7fb03c1b0870 2021-02-10T11:29:01.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:01.624 7fb03affd700 1 -- 172.21.2.112:0/1237595221 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb03000c960 con 0x7fb03c1b0870 2021-02-10T11:29:01.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:01.624 7fb04262a700 1 -- 172.21.2.112:0/1237595221 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb03c1b0870 msgr2=0x7fb03c0f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:01.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:01.624 7fb04262a700 1 --2- 172.21.2.112:0/1237595221 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb03c1b0870 0x7fb03c0f20e0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 rx=0x7fb030006700 tx=0x7fb030009b80).stop 2021-02-10T11:29:01.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:01.624 7fb04262a700 1 -- 172.21.2.112:0/1237595221 shutdown_connections 2021-02-10T11:29:01.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:01.624 7fb04262a700 1 --2- 172.21.2.112:0/1237595221 >> v2:172.21.2.120:3300/0 conn(0x7fb03c0efe00 0x7fb03c0f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:01.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:01.624 7fb04262a700 1 --2- 172.21.2.112:0/1237595221 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb03c1b0870 0x7fb03c0f20e0 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:01.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:01.624 7fb04262a700 1 -- 172.21.2.112:0/1237595221 >> 172.21.2.112:0/1237595221 conn(0x7fb03c0ce3e0 msgr2=0x7fb03c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:29:01.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:01.624 7fb04262a700 1 -- 172.21.2.112:0/1237595221 shutdown_connections 2021-02-10T11:29:01.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:01.624 7fb04262a700 1 -- 172.21.2.112:0/1237595221 wait complete. 2021-02-10T11:29:01.629 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:01.639 7fb04262a700 1 Processor -- start 2021-02-10T11:29:01.630 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:01.639 7fb04262a700 1 -- start start 2021-02-10T11:29:01.630 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:01.639 7fb04262a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb03c0b2840 0x7fb03c1a4300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:01.630 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:01.639 7fb04262a700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb03c0b30d0 0x7fb03c0b34d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:01.630 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:01.639 7fb04262a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb03c0b3af0 0x7fb03c0f0580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:01.630 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:01.639 7fb04262a700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb03c0b2c40 con 0x7fb03c0b3af0 2021-02-10T11:29:01.630 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:01.639 7fb04262a700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb03c1a4920 con 0x7fb03c0b30d0 2021-02-10T11:29:01.630 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:01.639 7fb04262a700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb03c0f0ba0 con 0x7fb03c0b2840 2021-02-10T11:29:01.630 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:01.639 7fb040c47700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb03c0b3af0 0x7fb03c0f0580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:01.630 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:01.639 7fb040c47700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb03c0b3af0 0x7fb03c0f0580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:46412/0 (socket says 172.21.2.112:46412) 2021-02-10T11:29:01.630 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:01.639 7fb040c47700 1 -- 172.21.2.112:0/857017376 learned_addr learned my addr 172.21.2.112:0/857017376 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:29:01.630 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:01.640 7fb040c47700 1 -- 172.21.2.112:0/857017376 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb03c0b2840 msgr2=0x7fb03c1a4300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:01.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:01.640 7fb040c47700 1 --2- 172.21.2.112:0/857017376 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb03c0b2840 0x7fb03c1a4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:01.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:01.640 7fb040c47700 1 -- 172.21.2.112:0/857017376 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb03c0b30d0 msgr2=0x7fb03c0b34d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:29:01.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:01.640 7fb040c47700 1 --2- 172.21.2.112:0/857017376 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb03c0b30d0 0x7fb03c0b34d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:01.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:01.640 7fb040c47700 1 -- 172.21.2.112:0/857017376 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb03c0f0dc0 con 0x7fb03c0b3af0 2021-02-10T11:29:01.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:01.640 7fb040c47700 1 --2- 172.21.2.112:0/857017376 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb03c0b3af0 0x7fb03c0f0580 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 rx=0x7fb034009960 tx=0x7fb03400bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:01.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:01.640 7fb038ff9700 1 -- 172.21.2.112:0/857017376 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb034000d10 con 0x7fb03c0b3af0 2021-02-10T11:29:01.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:01.640 7fb038ff9700 1 -- 172.21.2.112:0/857017376 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb034003020 con 0x7fb03c0b3af0 2021-02-10T11:29:01.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:01.640 7fb038ff9700 1 -- 172.21.2.112:0/857017376 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb034005ad0 con 0x7fb03c0b3af0 2021-02-10T11:29:01.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:01.640 7fb04262a700 1 -- 172.21.2.112:0/857017376 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb03c0f0dc0 con 0x7fb03c0b3af0 2021-02-10T11:29:01.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:01.640 7fb04262a700 1 -- 172.21.2.112:0/857017376 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb03c0f1350 con 0x7fb03c0b3af0 2021-02-10T11:29:01.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:01.641 7fb038ff9700 1 -- 172.21.2.112:0/857017376 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb034005850 con 0x7fb03c0b3af0 2021-02-10T11:29:01.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:01.641 7fb038ff9700 1 --2- 172.21.2.112:0/857017376 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb01c0186c0 0x7fb01c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:01.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:01.641 7fb038ff9700 1 -- 172.21.2.112:0/857017376 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fb034032980 con 0x7fb03c0b3af0 2021-02-10T11:29:01.632 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:01.641 7fb03bfff700 1 --2- 172.21.2.112:0/857017376 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb01c0186c0 0x7fb01c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:01.632 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:01.642 7fb03bfff700 1 --2- 172.21.2.112:0/857017376 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb01c0186c0 0x7fb01c01abc0 crc :-1 s=READY pgs=373 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:01.660 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:01.673 7fb04262a700 1 -- 172.21.2.112:0/857017376 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb03c0f0050 con 0x7fb03c0b3af0 2021-02-10T11:29:01.663 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:01.675 7fb038ff9700 1 -- 172.21.2.112:0/857017376 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb034005a60 con 0x7fb03c0b3af0 2021-02-10T11:29:01.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:01.801 7fb04262a700 1 -- 172.21.2.112:0/857017376 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348661", "target": ["mgr", ""]}) v1 -- 0x7fb03c0f0050 con 0x7fb01c0186c0 2021-02-10T11:29:01.815 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:01.828 7fb038ff9700 1 -- 172.21.2.112:0/857017376 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fb02c0009b0 con 0x7fb01c0186c0 2021-02-10T11:29:01.822 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:01.834 7fb04262a700 1 -- 172.21.2.112:0/857017376 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb01c0186c0 msgr2=0x7fb01c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:01.822 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:01.834 7fb04262a700 1 --2- 172.21.2.112:0/857017376 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb01c0186c0 0x7fb01c01abc0 crc :-1 s=READY pgs=373 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:01.822 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:01.834 7fb04262a700 1 -- 172.21.2.112:0/857017376 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb03c0b3af0 msgr2=0x7fb03c0f0580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:01.823 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:01.834 7fb04262a700 1 --2- 172.21.2.112:0/857017376 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb03c0b3af0 0x7fb03c0f0580 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 rx=0x7fb034009960 tx=0x7fb03400bcf0).stop 2021-02-10T11:29:01.823 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:01.835 7fb04262a700 1 -- 172.21.2.112:0/857017376 shutdown_connections 2021-02-10T11:29:01.823 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:01.835 7fb04262a700 1 --2- 172.21.2.112:0/857017376 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb01c0186c0 0x7fb01c01abc0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:01.823 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:01.835 7fb04262a700 1 --2- 172.21.2.112:0/857017376 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb03c0b30d0 0x7fb03c0b34d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:01.823 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:01.835 7fb04262a700 1 --2- 172.21.2.112:0/857017376 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb03c0b3af0 0x7fb03c0f0580 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:01.823 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:01.835 7fb04262a700 1 --2- 172.21.2.112:0/857017376 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb03c0b2840 0x7fb03c1a4300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:01.823 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:01.835 7fb04262a700 1 -- 172.21.2.112:0/857017376 >> 172.21.2.112:0/857017376 conn(0x7fb03c0c36f0 msgr2=0x7fb03c094c30 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:29:01.824 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:01.835 7fb04262a700 1 -- 172.21.2.112:0/857017376 shutdown_connections 2021-02-10T11:29:01.824 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:01.835 7fb04262a700 1 -- 172.21.2.112:0/857017376 wait complete. 2021-02-10T11:29:01.831 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000348661 2021-02-10T11:29:02.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:02.259 7f72efdbf700 1 Processor -- start 2021-02-10T11:29:02.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:02.259 7f72efdbf700 1 -- start start 2021-02-10T11:29:02.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:02.259 7f72efdbf700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f72e81b0470 0x7f72e81b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:02.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:02.259 7f72efdbf700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f72e80ef610 0x7f72e80efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:02.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:02.259 7f72efdbf700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f72e80f0030 0x7f72e80f0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:02.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:02.259 7f72efdbf700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72e80e5ea0 con 0x7f72e80f0030 2021-02-10T11:29:02.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:02.259 7f72efdbf700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72e80f0b50 con 0x7f72e81b0470 2021-02-10T11:29:02.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:02.259 7f72efdbf700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72e80e52e0 con 0x7f72e80ef610 2021-02-10T11:29:02.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:02.260 7f72ee3dc700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f72e80f0030 0x7f72e80f0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:02.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:02.260 7f72ee3dc700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f72e80f0030 0x7f72e80f0430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:46416/0 (socket says 172.21.2.112:46416) 2021-02-10T11:29:02.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:02.260 7f72ee3dc700 1 -- 172.21.2.112:0/3415581606 learned_addr learned my addr 172.21.2.112:0/3415581606 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:29:02.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:02.260 7f72ee3dc700 1 -- 172.21.2.112:0/3415581606 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f72e80ef610 msgr2=0x7f72e80efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:29:02.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:02.260 7f72ee3dc700 1 --2- 172.21.2.112:0/3415581606 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f72e80ef610 0x7f72e80efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:02.265 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:02.260 7f72ee3dc700 1 -- 172.21.2.112:0/3415581606 >> v2:172.21.2.120:3300/0 conn(0x7f72e81b0470 msgr2=0x7f72e81b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:29:02.265 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:02.260 7f72edbdb700 1 --2- 172.21.2.112:0/3415581606 >> v2:172.21.2.120:3300/0 conn(0x7f72e81b0470 0x7f72e81b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:02.265 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:02.260 7f72ee3dc700 1 --2- 172.21.2.112:0/3415581606 >> v2:172.21.2.120:3300/0 conn(0x7f72e81b0470 0x7f72e81b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:02.265 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:02.260 7f72ee3dc700 1 -- 172.21.2.112:0/3415581606 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72e80f10d0 con 0x7f72e80f0030 2021-02-10T11:29:02.265 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:02.260 7f72edbdb700 1 --2- 172.21.2.112:0/3415581606 >> v2:172.21.2.120:3300/0 conn(0x7f72e81b0470 0x7f72e81b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:29:02.265 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:02.260 7f72ee3dc700 1 --2- 172.21.2.112:0/3415581606 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f72e80f0030 0x7f72e80f0430 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 rx=0x7f72e0006f00 tx=0x7f72e0009e90).ready entity=mon.0 client_cookie=bbacab061b709114 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:02.265 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:02.261 7f72ecbd9700 1 -- 172.21.2.112:0/3415581606 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f72e000b300 con 0x7f72e80f0030 2021-02-10T11:29:02.265 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:02.261 7f72ecbd9700 1 -- 172.21.2.112:0/3415581606 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72e000b580 con 0x7f72e80f0030 2021-02-10T11:29:02.265 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:02.261 7f72ecbd9700 1 -- 172.21.2.112:0/3415581606 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f72e000baa0 con 0x7f72e80f0030 2021-02-10T11:29:02.265 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:02.261 7f72efdbf700 1 -- 172.21.2.112:0/3415581606 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f72e80f0030 msgr2=0x7f72e80f0430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:02.265 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:02.261 7f72efdbf700 1 --2- 172.21.2.112:0/3415581606 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f72e80f0030 0x7f72e80f0430 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 rx=0x7f72e0006f00 tx=0x7f72e0009e90).stop 2021-02-10T11:29:02.265 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:02.261 7f72efdbf700 1 -- 172.21.2.112:0/3415581606 shutdown_connections 2021-02-10T11:29:02.265 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:02.261 7f72efdbf700 1 --2- 172.21.2.112:0/3415581606 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f72e80f0030 0x7f72e80f0430 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:02.265 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:02.261 7f72efdbf700 1 --2- 172.21.2.112:0/3415581606 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f72e80ef610 0x7f72e80efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:02.266 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:02.261 7f72efdbf700 1 --2- 172.21.2.112:0/3415581606 >> v2:172.21.2.120:3300/0 conn(0x7f72e81b0470 0x7f72e81b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:02.266 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:02.261 7f72efdbf700 1 -- 172.21.2.112:0/3415581606 >> 172.21.2.112:0/3415581606 conn(0x7f72e80ce3e0 msgr2=0x7f72e80e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:29:02.266 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:02.262 7f72efdbf700 1 -- 172.21.2.112:0/3415581606 shutdown_connections 2021-02-10T11:29:02.266 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:02.262 7f72efdbf700 1 -- 172.21.2.112:0/3415581606 wait complete. 2021-02-10T11:29:02.267 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:02.276 7f72efdbf700 1 Processor -- start 2021-02-10T11:29:02.267 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:02.276 7f72efdbf700 1 -- start start 2021-02-10T11:29:02.267 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:02.277 7f72efdbf700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f72e809b930 0x7f72e80b7c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:02.268 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:02.277 7f72efdbf700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f72e80b8410 0x7f72e80b8810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:02.268 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:02.277 7f72efdbf700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f72e80c1f80 0x7f72e80c2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:02.268 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:02.277 7f72efdbf700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72e80c1a00 con 0x7f72e809b930 2021-02-10T11:29:02.268 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:02.277 7f72efdbf700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72e80b8e30 con 0x7f72e80b8410 2021-02-10T11:29:02.268 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:02.277 7f72efdbf700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72e80b9050 con 0x7f72e80c1f80 2021-02-10T11:29:02.268 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:02.277 7f72edbdb700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f72e809b930 0x7f72e80b7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:02.268 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:02.277 7f72edbdb700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f72e809b930 0x7f72e80b7c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:46420/0 (socket says 172.21.2.112:46420) 2021-02-10T11:29:02.268 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:02.277 7f72edbdb700 1 -- 172.21.2.112:0/2805868906 learned_addr learned my addr 172.21.2.112:0/2805868906 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:29:02.268 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:02.277 7f72edbdb700 1 -- 172.21.2.112:0/2805868906 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f72e80c1f80 msgr2=0x7f72e80c2380 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:29:02.268 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:02.277 7f72edbdb700 1 --2- 172.21.2.112:0/2805868906 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f72e80c1f80 0x7f72e80c2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:02.268 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:02.277 7f72edbdb700 1 -- 172.21.2.112:0/2805868906 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f72e80b8410 msgr2=0x7f72e80b8810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:02.268 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:02.277 7f72edbdb700 1 --2- 172.21.2.112:0/2805868906 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f72e80b8410 0x7f72e80b8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:02.268 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:02.277 7f72edbdb700 1 -- 172.21.2.112:0/2805868906 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72e80c29a0 con 0x7f72e809b930 2021-02-10T11:29:02.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:02.277 7f72edbdb700 1 --2- 172.21.2.112:0/2805868906 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f72e809b930 0x7f72e80b7c20 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 rx=0x7f72e4006f20 tx=0x7f72e400b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:02.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:02.277 7f72de7fc700 1 -- 172.21.2.112:0/2805868906 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f72e400bd80 con 0x7f72e809b930 2021-02-10T11:29:02.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:02.277 7f72de7fc700 1 -- 172.21.2.112:0/2805868906 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72e4000c90 con 0x7f72e809b930 2021-02-10T11:29:02.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:02.277 7f72efdbf700 1 -- 172.21.2.112:0/2805868906 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72e80c29a0 con 0x7f72e809b930 2021-02-10T11:29:02.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:02.278 7f72de7fc700 1 -- 172.21.2.112:0/2805868906 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f72e40022b0 con 0x7f72e809b930 2021-02-10T11:29:02.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:02.278 7f72efdbf700 1 -- 172.21.2.112:0/2805868906 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72e80c29a0 con 0x7f72e809b930 2021-02-10T11:29:02.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:02.278 7f72de7fc700 1 -- 172.21.2.112:0/2805868906 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f72e400a000 con 0x7f72e809b930 2021-02-10T11:29:02.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:02.278 7f72de7fc700 1 --2- 172.21.2.112:0/2805868906 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f72d0018730 0x7f72d001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:02.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:02.278 7f72de7fc700 1 -- 172.21.2.112:0/2805868906 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f72e4032620 con 0x7f72e809b930 2021-02-10T11:29:02.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:02.278 7f72ed3da700 1 --2- 172.21.2.112:0/2805868906 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f72d0018730 0x7f72d001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:02.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:02.279 7f72ed3da700 1 --2- 172.21.2.112:0/2805868906 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f72d0018730 0x7f72d001ac10 crc :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:02.297 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:02.310 7f72efdbf700 1 -- 172.21.2.112:0/2805868906 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72e8105b30 con 0x7f72e809b930 2021-02-10T11:29:02.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:02.312 7f72de7fc700 1 -- 172.21.2.112:0/2805868906 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f72e4006bb0 con 0x7f72e809b930 2021-02-10T11:29:02.429 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:02.441 7f72efdbf700 1 -- 172.21.2.112:0/2805868906 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348661", "target": ["mgr", ""]}) v1 -- 0x7f72e80b8e30 con 0x7f72d0018730 2021-02-10T11:29:02.433 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:02.445 7f72de7fc700 1 -- 172.21.2.112:0/2805868906 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f72d8006a10 con 0x7f72d0018730 2021-02-10T11:29:02.437 INFO:teuthology.orchestra.run.gibba012.stdout:/volumes/_nogroup/subvolume_0000000000348661/d22265a8-7435-43b3-86c9-151f6b042589 2021-02-10T11:29:02.439 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:02.451 7f72efdbf700 1 -- 172.21.2.112:0/2805868906 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f72d0018730 msgr2=0x7f72d001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:02.439 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:02.451 7f72efdbf700 1 --2- 172.21.2.112:0/2805868906 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f72d0018730 0x7f72d001ac10 crc :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:02.439 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:02.451 7f72efdbf700 1 -- 172.21.2.112:0/2805868906 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f72e809b930 msgr2=0x7f72e80b7c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:02.439 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:02.451 7f72efdbf700 1 --2- 172.21.2.112:0/2805868906 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f72e809b930 0x7f72e80b7c20 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 rx=0x7f72e4006f20 tx=0x7f72e400b990).stop 2021-02-10T11:29:02.439 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:02.451 7f72efdbf700 1 -- 172.21.2.112:0/2805868906 shutdown_connections 2021-02-10T11:29:02.439 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:02.451 7f72efdbf700 1 --2- 172.21.2.112:0/2805868906 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f72d0018730 0x7f72d001ac10 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:02.439 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:02.451 7f72efdbf700 1 --2- 172.21.2.112:0/2805868906 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f72e80b8410 0x7f72e80b8810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:02.439 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:02.451 7f72efdbf700 1 --2- 172.21.2.112:0/2805868906 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f72e80c1f80 0x7f72e80c2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:02.440 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:02.451 7f72efdbf700 1 --2- 172.21.2.112:0/2805868906 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f72e809b930 0x7f72e80b7c20 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:02.440 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:02.451 7f72efdbf700 1 -- 172.21.2.112:0/2805868906 >> 172.21.2.112:0/2805868906 conn(0x7f72e8094e10 msgr2=0x7f72e80a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:29:02.440 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:02.451 7f72efdbf700 1 -- 172.21.2.112:0/2805868906 shutdown_connections 2021-02-10T11:29:02.440 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:02.451 7f72efdbf700 1 -- 172.21.2.112:0/2805868906 wait complete. 2021-02-10T11:29:02.447 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000348661 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000348661/d22265a8-7435-43b3-86c9-151f6b042589 2021-02-10T11:29:02.447 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:02.447 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:02.447 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:02.783 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:29:02.784 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:02.784 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348661/d22265a8-7435-43b3-86c9-151f6b042589/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:03.164 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:29:03.164 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:29:03.164 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0226019 s, 46.4 MB/s 2021-02-10T11:29:03.165 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:03.166 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:03.166 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:03.501 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:29:03.502 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:03.502 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348661/d22265a8-7435-43b3-86c9-151f6b042589/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:03.878 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:29:03.878 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:29:03.879 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0249552 s, 42.0 MB/s 2021-02-10T11:29:03.879 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:03.880 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:03.880 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:04.215 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:29:04.216 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:04.216 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348661/d22265a8-7435-43b3-86c9-151f6b042589/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:04.588 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:29:04.589 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:29:04.589 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0239024 s, 43.9 MB/s 2021-02-10T11:29:04.590 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:04.591 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:04.591 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:04.926 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:29:04.927 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:04.927 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348661/d22265a8-7435-43b3-86c9-151f6b042589/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:05.305 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:29:05.483 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:29:05.483 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0237733 s, 44.1 MB/s 2021-02-10T11:29:05.484 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:05.484 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:05.485 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:05.842 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:29:05.842 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:05.843 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348661/d22265a8-7435-43b3-86c9-151f6b042589/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:06.222 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:29:06.222 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:29:06.222 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0268793 s, 39.0 MB/s 2021-02-10T11:29:06.223 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:06.223 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:06.223 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:06.559 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:29:06.560 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:06.560 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348661/d22265a8-7435-43b3-86c9-151f6b042589/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:06.938 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:29:06.939 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:29:06.939 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0254404 s, 41.2 MB/s 2021-02-10T11:29:06.940 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:06.940 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:06.940 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:07.276 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:29:07.277 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:07.277 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348661/d22265a8-7435-43b3-86c9-151f6b042589/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:07.652 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:29:07.652 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:29:07.652 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0244207 s, 42.9 MB/s 2021-02-10T11:29:07.653 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:07.653 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:07.653 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:07.991 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:29:07.991 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:07.991 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348661/d22265a8-7435-43b3-86c9-151f6b042589/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:08.367 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:29:08.368 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:29:08.368 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0236637 s, 44.3 MB/s 2021-02-10T11:29:08.369 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:08.369 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:08.369 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:08.704 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:29:08.705 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:08.705 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348661/d22265a8-7435-43b3-86c9-151f6b042589/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:09.081 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:29:09.082 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:29:09.082 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0253882 s, 41.3 MB/s 2021-02-10T11:29:09.083 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:09.083 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:09.083 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:09.418 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:29:09.419 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:09.419 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348661/d22265a8-7435-43b3-86c9-151f6b042589/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:09.793 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:29:09.793 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:29:09.793 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0251162 s, 41.7 MB/s 2021-02-10T11:29:09.794 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000348662 2021-02-10T11:29:10.224 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.220 7f742793a700 1 Processor -- start 2021-02-10T11:29:10.224 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.220 7f742793a700 1 -- start start 2021-02-10T11:29:10.224 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.220 7f742793a700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f74201b0470 0x7f74201b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:10.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.220 7f742793a700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74200f0b50 con 0x7f74200f20e0 2021-02-10T11:29:10.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.221 7f742793a700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74200f24e0 con 0x7f74200ef610 2021-02-10T11:29:10.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.221 7f742793a700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74200e5ea0 con 0x7f74201b0470 2021-02-10T11:29:10.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.221 7f7425756700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f74201b0470 0x7f74201b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:10.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.221 7f7425756700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f74201b0470 0x7f74201b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:46426/0 (socket says 172.21.2.112:46426) 2021-02-10T11:29:10.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.221 7f7425f57700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f74200f20e0 0x7f74200f0200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:45888/0 (socket says 172.21.2.112:45888) 2021-02-10T11:29:10.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.221 7f7425756700 1 -- 172.21.2.112:0/1592002908 learned_addr learned my addr 172.21.2.112:0/1592002908 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:29:10.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.221 7f7413fff700 1 -- 172.21.2.112:0/1592002908 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2114532148 0 0) 0x7f7418001100 con 0x7f74200f20e0 2021-02-10T11:29:10.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.221 7f7425756700 1 -- 172.21.2.112:0/1592002908 >> v1:172.21.2.120:6789/0 conn(0x7f74200ef610 legacy=0x7f74200efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:29:10.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.221 7f7425756700 1 -- 172.21.2.112:0/1592002908 >> v1:172.21.2.112:6789/0 conn(0x7f74200f20e0 legacy=0x7f74200f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:10.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.221 7f7425756700 1 -- 172.21.2.112:0/1592002908 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74200e52e0 con 0x7f74201b0470 2021-02-10T11:29:10.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.221 7f7425756700 1 --2- 172.21.2.112:0/1592002908 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f74201b0470 0x7f74201b0870 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 rx=0x7f7414006a80 tx=0x7f7414009f00).ready entity=mon.0 client_cookie=470fbeb999275d9c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:10.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.222 7f7413fff700 1 -- 172.21.2.112:0/1592002908 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f741400b5c0 con 0x7f74201b0470 2021-02-10T11:29:10.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.222 7f7413fff700 1 -- 172.21.2.112:0/1592002908 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f741400b8c0 con 0x7f74201b0470 2021-02-10T11:29:10.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.222 7f7413fff700 1 -- 172.21.2.112:0/1592002908 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f741400bde0 con 0x7f74201b0470 2021-02-10T11:29:10.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.222 7f742793a700 1 -- 172.21.2.112:0/1592002908 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f74201b0470 msgr2=0x7f74201b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:10.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.222 7f742793a700 1 --2- 172.21.2.112:0/1592002908 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f74201b0470 0x7f74201b0870 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 rx=0x7f7414006a80 tx=0x7f7414009f00).stop 2021-02-10T11:29:10.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.222 7f742793a700 1 -- 172.21.2.112:0/1592002908 shutdown_connections 2021-02-10T11:29:10.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.222 7f742793a700 1 --2- 172.21.2.112:0/1592002908 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f74201b0470 0x7f74201b0870 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:10.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.222 7f742793a700 1 -- 172.21.2.112:0/1592002908 >> 172.21.2.112:0/1592002908 conn(0x7f74200ce3e0 msgr2=0x7f74200e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:29:10.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.223 7f742793a700 1 -- 172.21.2.112:0/1592002908 shutdown_connections 2021-02-10T11:29:10.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.223 7f742793a700 1 -- 172.21.2.112:0/1592002908 wait complete. 2021-02-10T11:29:10.228 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.237 7f742793a700 1 Processor -- start 2021-02-10T11:29:10.228 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.237 7f742793a700 1 -- start start 2021-02-10T11:29:10.228 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.238 7f742793a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f74200bdbf0 0x7f74201a7450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:10.228 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.238 7f742793a700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f74200bdff0 0x7f74201a7a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:10.228 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.238 7f742793a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f74200abcc0 0x7f74200ac0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:10.228 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.238 7f742793a700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74200be590 con 0x7f74200abcc0 2021-02-10T11:29:10.228 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.238 7f742793a700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74201a8090 con 0x7f74200bdff0 2021-02-10T11:29:10.228 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.238 7f742793a700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74201a82b0 con 0x7f74200bdbf0 2021-02-10T11:29:10.228 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.238 7f7425f57700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f74200abcc0 0x7f74200ac0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:10.228 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.238 7f7425f57700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f74200abcc0 0x7f74200ac0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:46432/0 (socket says 172.21.2.112:46432) 2021-02-10T11:29:10.228 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.238 7f7425f57700 1 -- 172.21.2.112:0/3701315192 learned_addr learned my addr 172.21.2.112:0/3701315192 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:29:10.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.238 7f7425f57700 1 -- 172.21.2.112:0/3701315192 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f74200bdbf0 msgr2=0x7f74201a7450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:10.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.238 7f7425f57700 1 --2- 172.21.2.112:0/3701315192 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f74200bdbf0 0x7f74201a7450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:10.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.238 7f7425f57700 1 -- 172.21.2.112:0/3701315192 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f74200bdff0 msgr2=0x7f74201a7a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:10.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.238 7f7425f57700 1 --2- 172.21.2.112:0/3701315192 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f74200bdff0 0x7f74201a7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:10.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.238 7f7425f57700 1 -- 172.21.2.112:0/3701315192 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74200ac750 con 0x7f74200abcc0 2021-02-10T11:29:10.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.238 7f7425f57700 1 --2- 172.21.2.112:0/3701315192 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f74200abcc0 0x7f74200ac0c0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 rx=0x7f741800a520 tx=0x7f741800a8c0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:10.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.239 7f7411ffb700 1 -- 172.21.2.112:0/3701315192 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f741800acb0 con 0x7f74200abcc0 2021-02-10T11:29:10.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.239 7f7411ffb700 1 -- 172.21.2.112:0/3701315192 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74180043a0 con 0x7f74200abcc0 2021-02-10T11:29:10.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.239 7f742793a700 1 -- 172.21.2.112:0/3701315192 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74200ac750 con 0x7f74200abcc0 2021-02-10T11:29:10.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.239 7f7411ffb700 1 -- 172.21.2.112:0/3701315192 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7418004830 con 0x7f74200abcc0 2021-02-10T11:29:10.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.239 7f742793a700 1 -- 172.21.2.112:0/3701315192 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74200ad2d0 con 0x7f74200abcc0 2021-02-10T11:29:10.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.239 7f7411ffb700 1 -- 172.21.2.112:0/3701315192 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7418000af0 con 0x7f74200abcc0 2021-02-10T11:29:10.230 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.239 7f7411ffb700 1 --2- 172.21.2.112:0/3701315192 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f74000186c0 0x7f740001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:10.230 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.240 7f7411ffb700 1 -- 172.21.2.112:0/3701315192 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f7418032520 con 0x7f74200abcc0 2021-02-10T11:29:10.230 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.240 7f7425756700 1 --2- 172.21.2.112:0/3701315192 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f74000186c0 0x7f740001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:10.230 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.240 7f7425756700 1 --2- 172.21.2.112:0/3701315192 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f74000186c0 0x7f740001abc0 crc :-1 s=READY pgs=375 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:10.258 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.271 7f742793a700 1 -- 172.21.2.112:0/3701315192 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74200edcc0 con 0x7f74200abcc0 2021-02-10T11:29:10.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.273 7f7411ffb700 1 -- 172.21.2.112:0/3701315192 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7418000ef0 con 0x7f74200abcc0 2021-02-10T11:29:10.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.400 7f742793a700 1 -- 172.21.2.112:0/3701315192 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348662", "target": ["mgr", ""]}) v1 -- 0x7f74200ad290 con 0x7f74000186c0 2021-02-10T11:29:10.417 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.429 7f7411ffb700 1 -- 172.21.2.112:0/3701315192 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f7414006710 con 0x7f74000186c0 2021-02-10T11:29:10.421 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.434 7f742793a700 1 -- 172.21.2.112:0/3701315192 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f74000186c0 msgr2=0x7f740001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:10.421 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.434 7f742793a700 1 --2- 172.21.2.112:0/3701315192 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f74000186c0 0x7f740001abc0 crc :-1 s=READY pgs=375 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:10.421 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.434 7f742793a700 1 -- 172.21.2.112:0/3701315192 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f74200abcc0 msgr2=0x7f74200ac0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:10.421 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.434 7f742793a700 1 --2- 172.21.2.112:0/3701315192 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f74200abcc0 0x7f74200ac0c0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 rx=0x7f741800a520 tx=0x7f741800a8c0).stop 2021-02-10T11:29:10.422 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.434 7f742793a700 1 -- 172.21.2.112:0/3701315192 shutdown_connections 2021-02-10T11:29:10.422 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.434 7f742793a700 1 --2- 172.21.2.112:0/3701315192 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f74000186c0 0x7f740001abc0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:10.422 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.434 7f742793a700 1 --2- 172.21.2.112:0/3701315192 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f74200bdff0 0x7f74201a7a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:10.422 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.434 7f742793a700 1 --2- 172.21.2.112:0/3701315192 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f74200abcc0 0x7f74200ac0c0 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:10.422 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.434 7f742793a700 1 --2- 172.21.2.112:0/3701315192 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f74200bdbf0 0x7f74201a7450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:10.422 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.434 7f742793a700 1 -- 172.21.2.112:0/3701315192 >> 172.21.2.112:0/3701315192 conn(0x7f7420095a70 msgr2=0x7f74200bf040 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:29:10.423 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.434 7f742793a700 1 -- 172.21.2.112:0/3701315192 shutdown_connections 2021-02-10T11:29:10.423 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.434 7f742793a700 1 -- 172.21.2.112:0/3701315192 wait complete. 2021-02-10T11:29:10.430 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000348662 2021-02-10T11:29:10.862 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.858 7f95f3b32700 1 Processor -- start 2021-02-10T11:29:10.862 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.858 7f95f3b32700 1 -- start start 2021-02-10T11:29:10.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.858 7f95f3b32700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f95ec1b0470 0x7f95ec1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:10.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.858 7f95f3b32700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f95ec0ef610 0x7f95ec0efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:10.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.858 7f95f3b32700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f95ec0f0030 0x7f95ec0f0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:10.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.858 7f95f3b32700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95ec0e5ea0 con 0x7f95ec0ef610 2021-02-10T11:29:10.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.858 7f95f3b32700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95ec0f0b50 con 0x7f95ec1b0470 2021-02-10T11:29:10.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.858 7f95f3b32700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95ec0e52e0 con 0x7f95ec0f0030 2021-02-10T11:29:10.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.859 7f95f114d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f95ec0ef610 0x7f95ec0efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:10.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.859 7f95f114d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f95ec0ef610 0x7f95ec0efa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:46438/0 (socket says 172.21.2.112:46438) 2021-02-10T11:29:10.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.859 7f95f114d700 1 -- 172.21.2.112:0/3580648683 learned_addr learned my addr 172.21.2.112:0/3580648683 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:29:10.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.859 7f95f214f700 1 --2- 172.21.2.112:0/3580648683 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f95ec0f0030 0x7f95ec0f0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:10.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.859 7f95f194e700 1 --2- 172.21.2.112:0/3580648683 >> v2:172.21.2.120:3300/0 conn(0x7f95ec1b0470 0x7f95ec1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:10.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.859 7f95f114d700 1 -- 172.21.2.112:0/3580648683 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f95ec0f0030 msgr2=0x7f95ec0f0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:29:10.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.859 7f95f114d700 1 --2- 172.21.2.112:0/3580648683 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f95ec0f0030 0x7f95ec0f0430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:10.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.859 7f95f114d700 1 -- 172.21.2.112:0/3580648683 >> v2:172.21.2.120:3300/0 conn(0x7f95ec1b0470 msgr2=0x7f95ec1b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:29:10.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.859 7f95f114d700 1 --2- 172.21.2.112:0/3580648683 >> v2:172.21.2.120:3300/0 conn(0x7f95ec1b0470 0x7f95ec1b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:10.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.859 7f95f114d700 1 -- 172.21.2.112:0/3580648683 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95ec0f10d0 con 0x7f95ec0ef610 2021-02-10T11:29:10.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.859 7f95f194e700 1 --2- 172.21.2.112:0/3580648683 >> v2:172.21.2.120:3300/0 conn(0x7f95ec1b0470 0x7f95ec1b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:29:10.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.859 7f95f114d700 1 --2- 172.21.2.112:0/3580648683 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f95ec0ef610 0x7f95ec0efa10 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 rx=0x7f95dc006700 tx=0x7f95dc009b80).ready entity=mon.0 client_cookie=3268d6abb828c8ba server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:10.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.859 7f95f094c700 1 -- 172.21.2.112:0/3580648683 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f95dc006340 con 0x7f95ec0ef610 2021-02-10T11:29:10.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.860 7f95f094c700 1 -- 172.21.2.112:0/3580648683 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95dc00c0c0 con 0x7f95ec0ef610 2021-02-10T11:29:10.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.860 7f95f094c700 1 -- 172.21.2.112:0/3580648683 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f95dc00c620 con 0x7f95ec0ef610 2021-02-10T11:29:10.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.860 7f95f3b32700 1 -- 172.21.2.112:0/3580648683 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f95ec0ef610 msgr2=0x7f95ec0efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:10.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.860 7f95f3b32700 1 --2- 172.21.2.112:0/3580648683 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f95ec0ef610 0x7f95ec0efa10 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 rx=0x7f95dc006700 tx=0x7f95dc009b80).stop 2021-02-10T11:29:10.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.860 7f95f3b32700 1 -- 172.21.2.112:0/3580648683 shutdown_connections 2021-02-10T11:29:10.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.860 7f95f3b32700 1 --2- 172.21.2.112:0/3580648683 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f95ec0f0030 0x7f95ec0f0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:10.865 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.860 7f95f3b32700 1 --2- 172.21.2.112:0/3580648683 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f95ec0ef610 0x7f95ec0efa10 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:10.865 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.860 7f95f3b32700 1 --2- 172.21.2.112:0/3580648683 >> v2:172.21.2.120:3300/0 conn(0x7f95ec1b0470 0x7f95ec1b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:10.865 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.860 7f95f3b32700 1 -- 172.21.2.112:0/3580648683 >> 172.21.2.112:0/3580648683 conn(0x7f95ec0ce3e0 msgr2=0x7f95ec0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:29:10.865 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.861 7f95f3b32700 1 -- 172.21.2.112:0/3580648683 shutdown_connections 2021-02-10T11:29:10.865 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.861 7f95f3b32700 1 -- 172.21.2.112:0/3580648683 wait complete. 2021-02-10T11:29:10.866 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.875 7f95f3b32700 1 Processor -- start 2021-02-10T11:29:10.866 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.875 7f95f3b32700 1 -- start start 2021-02-10T11:29:10.867 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.875 7f95f3b32700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f95ec09b930 0x7f95ec0b7c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:10.867 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.875 7f95f3b32700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f95ec0b8410 0x7f95ec0b8810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:10.867 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.875 7f95f3b32700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f95ec0c1f80 0x7f95ec0c2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:10.867 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.875 7f95f3b32700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95ec0c1a00 con 0x7f95ec0b8410 2021-02-10T11:29:10.867 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.875 7f95f3b32700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95ec0b8e30 con 0x7f95ec09b930 2021-02-10T11:29:10.867 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.875 7f95f3b32700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95ec0b9050 con 0x7f95ec0c1f80 2021-02-10T11:29:10.867 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.876 7f95f114d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f95ec0b8410 0x7f95ec0b8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:10.867 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.876 7f95f114d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f95ec0b8410 0x7f95ec0b8810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:46442/0 (socket says 172.21.2.112:46442) 2021-02-10T11:29:10.867 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.876 7f95f114d700 1 -- 172.21.2.112:0/3213063190 learned_addr learned my addr 172.21.2.112:0/3213063190 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:29:10.867 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.876 7f95f114d700 1 -- 172.21.2.112:0/3213063190 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f95ec0c1f80 msgr2=0x7f95ec0c2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:10.867 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.876 7f95f114d700 1 --2- 172.21.2.112:0/3213063190 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f95ec0c1f80 0x7f95ec0c2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:10.867 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.876 7f95f114d700 1 -- 172.21.2.112:0/3213063190 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f95ec09b930 msgr2=0x7f95ec0b7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:10.867 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.876 7f95f114d700 1 --2- 172.21.2.112:0/3213063190 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f95ec09b930 0x7f95ec0b7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:10.868 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.876 7f95f114d700 1 -- 172.21.2.112:0/3213063190 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95ec0c29a0 con 0x7f95ec0b8410 2021-02-10T11:29:10.868 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.876 7f95f114d700 1 --2- 172.21.2.112:0/3213063190 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f95ec0b8410 0x7f95ec0b8810 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 rx=0x7f95dc00b5e0 tx=0x7f95dc00b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:10.868 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.876 7f95e27fc700 1 -- 172.21.2.112:0/3213063190 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f95dc002030 con 0x7f95ec0b8410 2021-02-10T11:29:10.868 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.876 7f95e27fc700 1 -- 172.21.2.112:0/3213063190 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95dc0022f0 con 0x7f95ec0b8410 2021-02-10T11:29:10.868 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.876 7f95e27fc700 1 -- 172.21.2.112:0/3213063190 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f95dc0027d0 con 0x7f95ec0b8410 2021-02-10T11:29:10.868 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.876 7f95f3b32700 1 -- 172.21.2.112:0/3213063190 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95ec0c29a0 con 0x7f95ec0b8410 2021-02-10T11:29:10.868 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.876 7f95f3b32700 1 -- 172.21.2.112:0/3213063190 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95ec0c29a0 con 0x7f95ec0b8410 2021-02-10T11:29:10.868 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.877 7f95e27fc700 1 -- 172.21.2.112:0/3213063190 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f95dc000c90 con 0x7f95ec0b8410 2021-02-10T11:29:10.868 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.877 7f95e27fc700 1 --2- 172.21.2.112:0/3213063190 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f95d40186c0 0x7f95d401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:10.868 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.877 7f95e27fc700 1 -- 172.21.2.112:0/3213063190 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f95dc032460 con 0x7f95ec0b8410 2021-02-10T11:29:10.868 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.877 7f95f194e700 1 --2- 172.21.2.112:0/3213063190 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f95d40186c0 0x7f95d401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:10.868 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.878 7f95f194e700 1 --2- 172.21.2.112:0/3213063190 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f95d40186c0 0x7f95d401abc0 crc :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:10.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.908 7f95f3b32700 1 -- 172.21.2.112:0/3213063190 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95ec105b30 con 0x7f95ec0b8410 2021-02-10T11:29:10.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:10.911 7f95e27fc700 1 -- 172.21.2.112:0/3213063190 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f95dc002020 con 0x7f95ec0b8410 2021-02-10T11:29:11.027 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:11.039 7f95f3b32700 1 -- 172.21.2.112:0/3213063190 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348662", "target": ["mgr", ""]}) v1 -- 0x7f95ec105b30 con 0x7f95d40186c0 2021-02-10T11:29:11.030 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:11.043 7f95e27fc700 1 -- 172.21.2.112:0/3213063190 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f95e8006420 con 0x7f95d40186c0 2021-02-10T11:29:11.034 INFO:teuthology.orchestra.run.gibba012.stdout:/volumes/_nogroup/subvolume_0000000000348662/12c2d745-8ca5-4731-8af7-e6d1da4bbd58 2021-02-10T11:29:11.036 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:11.048 7f95f3b32700 1 -- 172.21.2.112:0/3213063190 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f95d40186c0 msgr2=0x7f95d401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:11.036 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:11.048 7f95f3b32700 1 --2- 172.21.2.112:0/3213063190 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f95d40186c0 0x7f95d401abc0 crc :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:11.036 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:11.048 7f95f3b32700 1 -- 172.21.2.112:0/3213063190 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f95ec0b8410 msgr2=0x7f95ec0b8810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:11.036 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:11.048 7f95f3b32700 1 --2- 172.21.2.112:0/3213063190 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f95ec0b8410 0x7f95ec0b8810 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 rx=0x7f95dc00b5e0 tx=0x7f95dc00b980).stop 2021-02-10T11:29:11.036 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:11.048 7f95f3b32700 1 -- 172.21.2.112:0/3213063190 shutdown_connections 2021-02-10T11:29:11.036 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:11.048 7f95f3b32700 1 --2- 172.21.2.112:0/3213063190 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f95d40186c0 0x7f95d401abc0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:11.036 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:11.048 7f95f3b32700 1 --2- 172.21.2.112:0/3213063190 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f95ec0c1f80 0x7f95ec0c2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:11.036 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:11.048 7f95f3b32700 1 --2- 172.21.2.112:0/3213063190 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f95ec0b8410 0x7f95ec0b8810 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:11.036 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:11.048 7f95f3b32700 1 --2- 172.21.2.112:0/3213063190 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f95ec09b930 0x7f95ec0b7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:11.036 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:11.048 7f95f3b32700 1 -- 172.21.2.112:0/3213063190 >> 172.21.2.112:0/3213063190 conn(0x7f95ec094e10 msgr2=0x7f95ec0a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:29:11.037 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:11.048 7f95f3b32700 1 -- 172.21.2.112:0/3213063190 shutdown_connections 2021-02-10T11:29:11.037 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:11.048 7f95f3b32700 1 -- 172.21.2.112:0/3213063190 wait complete. 2021-02-10T11:29:11.044 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000348662 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000348662/12c2d745-8ca5-4731-8af7-e6d1da4bbd58 2021-02-10T11:29:11.044 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:11.044 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:11.044 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:11.381 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:29:11.382 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:11.382 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348662/12c2d745-8ca5-4731-8af7-e6d1da4bbd58/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:11.769 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:29:11.770 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:29:11.770 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0296904 s, 35.3 MB/s 2021-02-10T11:29:11.771 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:11.771 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:11.771 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:12.106 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:29:12.107 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:12.107 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348662/12c2d745-8ca5-4731-8af7-e6d1da4bbd58/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:12.482 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:29:12.483 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:29:12.483 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0240093 s, 43.7 MB/s 2021-02-10T11:29:12.484 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:12.484 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:12.484 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:12.795 DEBUG:teuthology.orchestra.run.gibba012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:29:12.798 DEBUG:teuthology.orchestra.run.gibba020:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:29:12.820 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:29:12.821 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:12.821 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348662/12c2d745-8ca5-4731-8af7-e6d1da4bbd58/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:13.172 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:29:13.173 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:29:13.173 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0241078 s, 43.5 MB/s 2021-02-10T11:29:13.174 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:13.174 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:13.174 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:13.510 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:29:13.511 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:13.511 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348662/12c2d745-8ca5-4731-8af7-e6d1da4bbd58/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:13.886 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:29:13.887 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:29:13.887 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0237658 s, 44.1 MB/s 2021-02-10T11:29:13.888 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:13.888 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:13.888 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:14.222 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:29:14.223 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:14.223 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348662/12c2d745-8ca5-4731-8af7-e6d1da4bbd58/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:14.597 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:29:14.597 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:29:14.598 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0249992 s, 41.9 MB/s 2021-02-10T11:29:14.598 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:14.599 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:14.599 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:14.936 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:29:14.936 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:14.936 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348662/12c2d745-8ca5-4731-8af7-e6d1da4bbd58/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:15.312 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:29:15.313 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:29:15.313 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0239072 s, 43.9 MB/s 2021-02-10T11:29:15.313 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:15.314 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:15.314 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:15.650 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:29:15.651 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:15.651 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348662/12c2d745-8ca5-4731-8af7-e6d1da4bbd58/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:16.026 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:29:16.026 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:29:16.027 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0227183 s, 46.2 MB/s 2021-02-10T11:29:16.027 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:16.028 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:16.028 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:16.364 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:29:16.364 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:16.364 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348662/12c2d745-8ca5-4731-8af7-e6d1da4bbd58/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:16.739 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:29:16.739 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:29:16.739 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.023115 s, 45.4 MB/s 2021-02-10T11:29:16.740 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:16.740 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:16.740 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:17.076 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:29:17.076 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:17.077 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348662/12c2d745-8ca5-4731-8af7-e6d1da4bbd58/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:17.454 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:29:17.454 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:29:17.454 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0241826 s, 43.4 MB/s 2021-02-10T11:29:17.455 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:17.455 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:17.455 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:17.790 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:29:17.791 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:17.791 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348662/12c2d745-8ca5-4731-8af7-e6d1da4bbd58/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:18.166 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:29:18.166 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:29:18.166 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0244083 s, 43.0 MB/s 2021-02-10T11:29:18.167 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000348663 2021-02-10T11:29:18.600 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:18.596 7fb589205700 1 Processor -- start 2021-02-10T11:29:18.600 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:18.596 7fb589205700 1 -- start start 2021-02-10T11:29:18.600 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:18.597 7fb589205700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fb5840f05f0 0x7fb5840f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:18.600 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:18.597 7fb589205700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5841b0c70 con 0x7fb5841b0870 2021-02-10T11:29:18.600 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:18.597 7fb589205700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5840f0b50 con 0x7fb5841b0470 2021-02-10T11:29:18.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:18.597 7fb589205700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5840e5ea0 con 0x7fb5840f05f0 2021-02-10T11:29:18.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:18.597 7fb58259c700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fb5841b0870 0x7fb5840efe00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:45914/0 (socket says 172.21.2.112:45914) 2021-02-10T11:29:18.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:18.597 7fb58259c700 1 -- 172.21.2.112:0/3619206918 learned_addr learned my addr 172.21.2.112:0/3619206918 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:29:18.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:18.597 7fb58359e700 1 --2- 172.21.2.112:0/3619206918 >> v2:172.21.2.112:3300/0 conn(0x7fb5840f05f0 0x7fb5840f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:18.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:18.597 7fb58359e700 1 -- 172.21.2.112:0/3619206918 >> v1:172.21.2.120:6789/0 conn(0x7fb5841b0470 legacy=0x7fb5840ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:29:18.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:18.597 7fb58359e700 1 -- 172.21.2.112:0/3619206918 >> v1:172.21.2.112:6789/0 conn(0x7fb5841b0870 legacy=0x7fb5840efe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:18.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:18.597 7fb58359e700 1 -- 172.21.2.112:0/3619206918 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5840e52e0 con 0x7fb5840f05f0 2021-02-10T11:29:18.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:18.597 7fb58359e700 1 --2- 172.21.2.112:0/3619206918 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb5840f05f0 0x7fb5840f20e0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 rx=0x7fb57c006e30 tx=0x7fb57c009e60).ready entity=mon.0 client_cookie=c49b67545aceb13 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:18.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:18.598 7fb581d9b700 1 -- 172.21.2.112:0/3619206918 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb57c006a50 con 0x7fb5840f05f0 2021-02-10T11:29:18.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:18.598 7fb581d9b700 1 -- 172.21.2.112:0/3619206918 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb57c00b300 con 0x7fb5840f05f0 2021-02-10T11:29:18.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:18.598 7fb581d9b700 1 -- 172.21.2.112:0/3619206918 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb57c00b760 con 0x7fb5840f05f0 2021-02-10T11:29:18.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:18.598 7fb589205700 1 -- 172.21.2.112:0/3619206918 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb5840f05f0 msgr2=0x7fb5840f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:18.602 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:18.598 7fb589205700 1 --2- 172.21.2.112:0/3619206918 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb5840f05f0 0x7fb5840f20e0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 rx=0x7fb57c006e30 tx=0x7fb57c009e60).stop 2021-02-10T11:29:18.602 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:18.598 7fb589205700 1 -- 172.21.2.112:0/3619206918 shutdown_connections 2021-02-10T11:29:18.602 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:18.598 7fb589205700 1 --2- 172.21.2.112:0/3619206918 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb5840f05f0 0x7fb5840f20e0 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:18.602 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:18.598 7fb589205700 1 -- 172.21.2.112:0/3619206918 >> 172.21.2.112:0/3619206918 conn(0x7fb5840ce3e0 msgr2=0x7fb5840e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:29:18.602 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:18.599 7fb589205700 1 -- 172.21.2.112:0/3619206918 shutdown_connections 2021-02-10T11:29:18.602 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:18.599 7fb589205700 1 -- 172.21.2.112:0/3619206918 wait complete. 2021-02-10T11:29:18.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:18.613 7fb589205700 1 Processor -- start 2021-02-10T11:29:18.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:18.613 7fb589205700 1 -- start start 2021-02-10T11:29:18.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:18.613 7fb589205700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb5840a3b50 0x7fb5841b3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:18.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:18.613 7fb589205700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb5841c25b0 0x7fb5841b3d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:18.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:18.613 7fb589205700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb5841b4350 0x7fb5841b18a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:18.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:18.613 7fb589205700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5841b3190 con 0x7fb5840a3b50 2021-02-10T11:29:18.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:18.613 7fb589205700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb584087750 con 0x7fb5841c25b0 2021-02-10T11:29:18.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:18.613 7fb589205700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5840a6bf0 con 0x7fb5841b4350 2021-02-10T11:29:18.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:18.613 7fb582d9d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb5840a3b50 0x7fb5841b3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:18.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:18.613 7fb582d9d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb5840a3b50 0x7fb5841b3710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:46456/0 (socket says 172.21.2.112:46456) 2021-02-10T11:29:18.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:18.613 7fb582d9d700 1 -- 172.21.2.112:0/3428458938 learned_addr learned my addr 172.21.2.112:0/3428458938 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:29:18.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:18.614 7fb582d9d700 1 -- 172.21.2.112:0/3428458938 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb5841b4350 msgr2=0x7fb5841b18a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:29:18.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:18.614 7fb582d9d700 1 --2- 172.21.2.112:0/3428458938 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb5841b4350 0x7fb5841b18a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:18.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:18.614 7fb582d9d700 1 -- 172.21.2.112:0/3428458938 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb5841c25b0 msgr2=0x7fb5841b3d30 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:29:18.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:18.614 7fb582d9d700 1 --2- 172.21.2.112:0/3428458938 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb5841c25b0 0x7fb5841b3d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:18.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:18.614 7fb582d9d700 1 -- 172.21.2.112:0/3428458938 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5841b1ec0 con 0x7fb5840a3b50 2021-02-10T11:29:18.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:18.614 7fb582d9d700 1 --2- 172.21.2.112:0/3428458938 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb5840a3b50 0x7fb5841b3710 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 rx=0x7fb57400b5b0 tx=0x7fb57400b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:18.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:18.614 7fb57b7fe700 1 -- 172.21.2.112:0/3428458938 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb57400bd40 con 0x7fb5840a3b50 2021-02-10T11:29:18.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:18.614 7fb57b7fe700 1 -- 172.21.2.112:0/3428458938 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5740009b0 con 0x7fb5840a3b50 2021-02-10T11:29:18.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:18.614 7fb589205700 1 -- 172.21.2.112:0/3428458938 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5841b1ec0 con 0x7fb5840a3b50 2021-02-10T11:29:18.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:18.614 7fb589205700 1 -- 172.21.2.112:0/3428458938 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5841b2450 con 0x7fb5840a3b50 2021-02-10T11:29:18.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:18.614 7fb57b7fe700 1 -- 172.21.2.112:0/3428458938 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb5740062a0 con 0x7fb5840a3b50 2021-02-10T11:29:18.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:18.615 7fb57b7fe700 1 -- 172.21.2.112:0/3428458938 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb574006020 con 0x7fb5840a3b50 2021-02-10T11:29:18.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:18.615 7fb57b7fe700 1 --2- 172.21.2.112:0/3428458938 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb568018730 0x7fb56801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:18.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:18.615 7fb57b7fe700 1 -- 172.21.2.112:0/3428458938 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fb5740327f0 con 0x7fb5840a3b50 2021-02-10T11:29:18.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:18.615 7fb58259c700 1 --2- 172.21.2.112:0/3428458938 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb568018730 0x7fb56801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:18.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:18.616 7fb58259c700 1 --2- 172.21.2.112:0/3428458938 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb568018730 0x7fb56801ac10 crc :-1 s=READY pgs=377 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:18.634 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:18.646 7fb589205700 1 -- 172.21.2.112:0/3428458938 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb58407e6b0 con 0x7fb5840a3b50 2021-02-10T11:29:18.636 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:18.649 7fb57b7fe700 1 -- 172.21.2.112:0/3428458938 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb574006230 con 0x7fb5840a3b50 2021-02-10T11:29:18.763 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:18.775 7fb589205700 1 -- 172.21.2.112:0/3428458938 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348663", "target": ["mgr", ""]}) v1 -- 0x7fb5840ce3e0 con 0x7fb568018730 2021-02-10T11:29:18.790 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:18.803 7fb57b7fe700 1 -- 172.21.2.112:0/3428458938 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fb56c0014d0 con 0x7fb568018730 2021-02-10T11:29:18.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:18.809 7fb589205700 1 -- 172.21.2.112:0/3428458938 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb568018730 msgr2=0x7fb56801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:18.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:18.809 7fb589205700 1 --2- 172.21.2.112:0/3428458938 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb568018730 0x7fb56801ac10 crc :-1 s=READY pgs=377 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:18.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:18.809 7fb589205700 1 -- 172.21.2.112:0/3428458938 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb5840a3b50 msgr2=0x7fb5841b3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:18.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:18.809 7fb589205700 1 --2- 172.21.2.112:0/3428458938 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb5840a3b50 0x7fb5841b3710 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 rx=0x7fb57400b5b0 tx=0x7fb57400b950).stop 2021-02-10T11:29:18.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:18.809 7fb589205700 1 -- 172.21.2.112:0/3428458938 shutdown_connections 2021-02-10T11:29:18.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:18.809 7fb589205700 1 --2- 172.21.2.112:0/3428458938 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb568018730 0x7fb56801ac10 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:18.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:18.809 7fb589205700 1 --2- 172.21.2.112:0/3428458938 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb5841c25b0 0x7fb5841b3d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:18.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:18.809 7fb589205700 1 --2- 172.21.2.112:0/3428458938 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb5841b4350 0x7fb5841b18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:18.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:18.809 7fb589205700 1 --2- 172.21.2.112:0/3428458938 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb5840a3b50 0x7fb5841b3710 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:18.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:18.809 7fb589205700 1 -- 172.21.2.112:0/3428458938 >> 172.21.2.112:0/3428458938 conn(0x7fb5840886d0 msgr2=0x7fb584081c40 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:29:18.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:18.809 7fb589205700 1 -- 172.21.2.112:0/3428458938 shutdown_connections 2021-02-10T11:29:18.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:18.809 7fb589205700 1 -- 172.21.2.112:0/3428458938 wait complete. 2021-02-10T11:29:18.805 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000348663 2021-02-10T11:29:19.240 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:19.235 7f1c516d4700 1 Processor -- start 2021-02-10T11:29:19.241 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:19.235 7f1c516d4700 1 -- start start 2021-02-10T11:29:19.241 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:19.235 7f1c516d4700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1c4c1b0470 0x7f1c4c1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:19.241 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:19.235 7f1c516d4700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c4c0f0b50 con 0x7f1c4c0ef610 2021-02-10T11:29:19.241 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:19.235 7f1c516d4700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c4c0f24e0 con 0x7f1c4c0f20e0 2021-02-10T11:29:19.241 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:19.235 7f1c516d4700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c4c0e5ea0 con 0x7f1c4c1b0470 2021-02-10T11:29:19.241 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:19.236 7f1c4aabe700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f1c4c0ef610 0x7f1c4c0efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:45926/0 (socket says 172.21.2.112:45926) 2021-02-10T11:29:19.241 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:19.236 7f1c4aabe700 1 -- 172.21.2.112:0/269216311 learned_addr learned my addr 172.21.2.112:0/269216311 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:29:19.241 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:19.236 7f1c4b2bf700 1 --2- 172.21.2.112:0/269216311 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1c4c1b0470 0x7f1c4c1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:19.241 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:19.236 7f1c4a2bd700 1 -- 172.21.2.112:0/269216311 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1784507098 0 0) 0x7f1c3c0009b0 con 0x7f1c4c0ef610 2021-02-10T11:29:19.241 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:19.236 7f1c4a2bd700 1 -- 172.21.2.112:0/269216311 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c38002fe0 con 0x7f1c4c0ef610 2021-02-10T11:29:19.241 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:19.236 7f1c4a2bd700 1 -- 172.21.2.112:0/269216311 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 279043646 0 0) 0x7f1c3c001010 con 0x7f1c4c0ef610 2021-02-10T11:29:19.241 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:19.236 7f1c4a2bd700 1 -- 172.21.2.112:0/269216311 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1c4c1b0470 msgr2=0x7f1c4c1b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:29:19.242 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:19.236 7f1c4a2bd700 1 --2- 172.21.2.112:0/269216311 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1c4c1b0470 0x7f1c4c1b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:19.242 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:19.236 7f1c4a2bd700 1 -- 172.21.2.112:0/269216311 >> v1:172.21.2.120:6789/0 conn(0x7f1c4c0f20e0 legacy=0x7f1c4c0f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:19.242 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:19.236 7f1c4a2bd700 1 -- 172.21.2.112:0/269216311 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c4c0e52e0 con 0x7f1c4c0ef610 2021-02-10T11:29:19.242 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:19.236 7f1c4a2bd700 1 -- 172.21.2.112:0/269216311 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f1c3c001590 con 0x7f1c4c0ef610 2021-02-10T11:29:19.242 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:19.236 7f1c4a2bd700 1 -- 172.21.2.112:0/269216311 >> v1:172.21.2.112:6789/0 conn(0x7f1c4c0ef610 legacy=0x7f1c4c0efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:19.242 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:19.236 7f1c4a2bd700 1 --2- 172.21.2.112:0/269216311 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1c38003af0 0x7f1c38005f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:19.242 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:19.236 7f1c4a2bd700 1 --2- 172.21.2.112:0/269216311 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1c38006780 0x7f1c38008c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:19.242 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:19.236 7f1c4a2bd700 1 --2- 172.21.2.112:0/269216311 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1c380094f0 0x7f1c3800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:19.242 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:19.236 7f1c4a2bd700 1 -- 172.21.2.112:0/269216311 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c3800c2c0 con 0x7f1c38003af0 2021-02-10T11:29:19.242 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:19.236 7f1c4a2bd700 1 -- 172.21.2.112:0/269216311 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c3800c4e0 con 0x7f1c380094f0 2021-02-10T11:29:19.242 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:19.236 7f1c4a2bd700 1 -- 172.21.2.112:0/269216311 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c3800c700 con 0x7f1c38006780 2021-02-10T11:29:19.242 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:19.236 7f1c4b2bf700 1 --2- 172.21.2.112:0/269216311 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1c38003af0 0x7f1c38005f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:19.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:19.236 7f1c4bac0700 1 --2- 172.21.2.112:0/269216311 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1c380094f0 0x7f1c3800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:19.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:19.237 7f1c4aabe700 1 --2- 172.21.2.112:0/269216311 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1c38006780 0x7f1c38008c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:19.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:19.237 7f1c4b2bf700 1 -- 172.21.2.112:0/269216311 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1c38006780 msgr2=0x7f1c38008c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:29:19.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:19.237 7f1c4b2bf700 1 --2- 172.21.2.112:0/269216311 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1c38006780 0x7f1c38008c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:19.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:19.237 7f1c4b2bf700 1 -- 172.21.2.112:0/269216311 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1c380094f0 msgr2=0x7f1c3800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:29:19.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:19.237 7f1c4b2bf700 1 --2- 172.21.2.112:0/269216311 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1c380094f0 0x7f1c3800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:19.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:19.237 7f1c4b2bf700 1 -- 172.21.2.112:0/269216311 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c38050970 con 0x7f1c38003af0 2021-02-10T11:29:19.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:19.237 7f1c4aabe700 1 --2- 172.21.2.112:0/269216311 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1c38006780 0x7f1c38008c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:29:19.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:19.237 7f1c4bac0700 1 --2- 172.21.2.112:0/269216311 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1c380094f0 0x7f1c3800ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:29:19.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:19.237 7f1c4b2bf700 1 --2- 172.21.2.112:0/269216311 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1c38003af0 0x7f1c38005f00 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 rx=0x7f1c34009310 tx=0x7f1c34009a90).ready entity=mon.0 client_cookie=c3875411c5269a9c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:19.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:19.237 7f1c4a2bd700 1 -- 172.21.2.112:0/269216311 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1c34006340 con 0x7f1c38003af0 2021-02-10T11:29:19.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:19.237 7f1c4a2bd700 1 -- 172.21.2.112:0/269216311 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c3400c0c0 con 0x7f1c38003af0 2021-02-10T11:29:19.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:19.237 7f1c4a2bd700 1 -- 172.21.2.112:0/269216311 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1c3400c550 con 0x7f1c38003af0 2021-02-10T11:29:19.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:19.237 7f1c516d4700 1 -- 172.21.2.112:0/269216311 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1c38003af0 msgr2=0x7f1c38005f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:19.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:19.237 7f1c516d4700 1 --2- 172.21.2.112:0/269216311 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1c38003af0 0x7f1c38005f00 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 rx=0x7f1c34009310 tx=0x7f1c34009a90).stop 2021-02-10T11:29:19.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:19.237 7f1c4bac0700 1 -- 172.21.2.112:0/269216311 reap_dead start 2021-02-10T11:29:19.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:19.238 7f1c516d4700 1 -- 172.21.2.112:0/269216311 shutdown_connections 2021-02-10T11:29:19.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:19.238 7f1c516d4700 1 -- 172.21.2.112:0/269216311 >> 172.21.2.112:0/269216311 conn(0x7f1c4c0ce3e0 msgr2=0x7f1c4c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:29:19.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:19.238 7f1c516d4700 1 -- 172.21.2.112:0/269216311 shutdown_connections 2021-02-10T11:29:19.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:19.238 7f1c516d4700 1 -- 172.21.2.112:0/269216311 wait complete. 2021-02-10T11:29:19.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:19.252 7f1c516d4700 1 Processor -- start 2021-02-10T11:29:19.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:19.252 7f1c516d4700 1 -- start start 2021-02-10T11:29:19.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:19.253 7f1c516d4700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1c4c0924b0 0x7f1c4c1a3590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:19.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:19.253 7f1c516d4700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1c4c1a3bb0 0x7f1c4c0bf920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:19.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:19.253 7f1c516d4700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1c4c0bff40 0x7f1c4c0c0340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:19.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:19.253 7f1c516d4700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c4c09a0a0 con 0x7f1c4c0bff40 2021-02-10T11:29:19.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:19.253 7f1c516d4700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c4c0c0960 con 0x7f1c4c1a3bb0 2021-02-10T11:29:19.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:19.253 7f1c516d4700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c4c098420 con 0x7f1c4c0924b0 2021-02-10T11:29:19.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:19.253 7f1c4bac0700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1c4c0bff40 0x7f1c4c0c0340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:19.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:19.253 7f1c4bac0700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1c4c0bff40 0x7f1c4c0c0340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:46478/0 (socket says 172.21.2.112:46478) 2021-02-10T11:29:19.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:19.253 7f1c4bac0700 1 -- 172.21.2.112:0/1316118367 learned_addr learned my addr 172.21.2.112:0/1316118367 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:29:19.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:19.253 7f1c4aabe700 1 --2- 172.21.2.112:0/1316118367 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1c4c1a3bb0 0x7f1c4c0bf920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:19.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:19.253 7f1c4b2bf700 1 --2- 172.21.2.112:0/1316118367 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1c4c0924b0 0x7f1c4c1a3590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:19.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:19.253 7f1c4bac0700 1 -- 172.21.2.112:0/1316118367 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1c4c0924b0 msgr2=0x7f1c4c1a3590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:19.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:19.253 7f1c4bac0700 1 --2- 172.21.2.112:0/1316118367 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1c4c0924b0 0x7f1c4c1a3590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:19.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:19.253 7f1c4bac0700 1 -- 172.21.2.112:0/1316118367 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1c4c1a3bb0 msgr2=0x7f1c4c0bf920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:19.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:19.253 7f1c4bac0700 1 --2- 172.21.2.112:0/1316118367 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1c4c1a3bb0 0x7f1c4c0bf920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:19.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:19.253 7f1c4bac0700 1 -- 172.21.2.112:0/1316118367 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c4c098640 con 0x7f1c4c0bff40 2021-02-10T11:29:19.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:19.254 7f1c4bac0700 1 --2- 172.21.2.112:0/1316118367 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1c4c0bff40 0x7f1c4c0c0340 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 rx=0x7f1c44009920 tx=0x7f1c4400bd20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:19.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:19.254 7f1c43fff700 1 -- 172.21.2.112:0/1316118367 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1c4400c2f0 con 0x7f1c4c0bff40 2021-02-10T11:29:19.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:19.254 7f1c43fff700 1 -- 172.21.2.112:0/1316118367 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c4400be60 con 0x7f1c4c0bff40 2021-02-10T11:29:19.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:19.254 7f1c43fff700 1 -- 172.21.2.112:0/1316118367 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1c4400c760 con 0x7f1c4c0bff40 2021-02-10T11:29:19.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:19.254 7f1c516d4700 1 -- 172.21.2.112:0/1316118367 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c4c098640 con 0x7f1c4c0bff40 2021-02-10T11:29:19.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:19.254 7f1c516d4700 1 -- 172.21.2.112:0/1316118367 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c4c098640 con 0x7f1c4c0bff40 2021-02-10T11:29:19.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:19.254 7f1c43fff700 1 -- 172.21.2.112:0/1316118367 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1c440056a0 con 0x7f1c4c0bff40 2021-02-10T11:29:19.248 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:19.254 7f1c43fff700 1 --2- 172.21.2.112:0/1316118367 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1c300186c0 0x7f1c3001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:19.248 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:19.254 7f1c43fff700 1 -- 172.21.2.112:0/1316118367 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f1c44032910 con 0x7f1c4c0bff40 2021-02-10T11:29:19.248 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:19.255 7f1c4b2bf700 1 --2- 172.21.2.112:0/1316118367 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1c300186c0 0x7f1c3001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:19.248 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:19.255 7f1c4b2bf700 1 --2- 172.21.2.112:0/1316118367 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1c300186c0 0x7f1c3001abc0 crc :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:19.273 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:19.285 7f1c516d4700 1 -- 172.21.2.112:0/1316118367 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c4c105b40 con 0x7f1c4c0bff40 2021-02-10T11:29:19.275 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:19.288 7f1c43fff700 1 -- 172.21.2.112:0/1316118367 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1c44025f20 con 0x7f1c4c0bff40 2021-02-10T11:29:19.407 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:19.419 7f1c516d4700 1 -- 172.21.2.112:0/1316118367 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348663", "target": ["mgr", ""]}) v1 -- 0x7f1c4c0c0960 con 0x7f1c300186c0 2021-02-10T11:29:19.410 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:19.423 7f1c43fff700 1 -- 172.21.2.112:0/1316118367 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f1c34009370 con 0x7f1c300186c0 2021-02-10T11:29:19.414 INFO:teuthology.orchestra.run.gibba012.stdout:/volumes/_nogroup/subvolume_0000000000348663/d1de0f0b-e845-43bb-bd68-7c943127b724 2021-02-10T11:29:19.415 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:19.428 7f1c516d4700 1 -- 172.21.2.112:0/1316118367 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1c300186c0 msgr2=0x7f1c3001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:19.416 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:19.428 7f1c516d4700 1 --2- 172.21.2.112:0/1316118367 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1c300186c0 0x7f1c3001abc0 crc :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:19.416 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:19.428 7f1c516d4700 1 -- 172.21.2.112:0/1316118367 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1c4c0bff40 msgr2=0x7f1c4c0c0340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:19.416 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:19.428 7f1c516d4700 1 --2- 172.21.2.112:0/1316118367 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1c4c0bff40 0x7f1c4c0c0340 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 rx=0x7f1c44009920 tx=0x7f1c4400bd20).stop 2021-02-10T11:29:19.416 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:19.428 7f1c516d4700 1 -- 172.21.2.112:0/1316118367 shutdown_connections 2021-02-10T11:29:19.416 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:19.428 7f1c516d4700 1 --2- 172.21.2.112:0/1316118367 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1c300186c0 0x7f1c3001abc0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:19.416 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:19.428 7f1c516d4700 1 --2- 172.21.2.112:0/1316118367 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1c4c1a3bb0 0x7f1c4c0bf920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:19.416 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:19.428 7f1c516d4700 1 --2- 172.21.2.112:0/1316118367 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1c4c0bff40 0x7f1c4c0c0340 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:19.416 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:19.428 7f1c516d4700 1 --2- 172.21.2.112:0/1316118367 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1c4c0924b0 0x7f1c4c1a3590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:19.416 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:19.428 7f1c516d4700 1 -- 172.21.2.112:0/1316118367 >> 172.21.2.112:0/1316118367 conn(0x7f1c4c0b50c0 msgr2=0x7f1c4c092d90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:29:19.416 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:19.428 7f1c516d4700 1 -- 172.21.2.112:0/1316118367 shutdown_connections 2021-02-10T11:29:19.416 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:19.428 7f1c516d4700 1 -- 172.21.2.112:0/1316118367 wait complete. 2021-02-10T11:29:19.424 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000348663 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000348663/d1de0f0b-e845-43bb-bd68-7c943127b724 2021-02-10T11:29:19.424 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:19.424 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:19.424 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:19.761 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:29:19.761 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:19.762 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348663/d1de0f0b-e845-43bb-bd68-7c943127b724/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:20.142 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:29:20.142 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:29:20.142 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0251791 s, 41.6 MB/s 2021-02-10T11:29:20.143 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:20.143 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:20.143 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:20.479 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:29:20.480 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:20.480 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348663/d1de0f0b-e845-43bb-bd68-7c943127b724/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:20.857 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:29:20.858 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:29:20.858 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0256479 s, 40.9 MB/s 2021-02-10T11:29:20.859 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:20.859 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:20.859 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:21.193 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:29:21.194 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:21.194 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348663/d1de0f0b-e845-43bb-bd68-7c943127b724/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:21.568 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:29:21.598 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:29:21.598 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0235361 s, 44.6 MB/s 2021-02-10T11:29:21.599 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:21.599 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:21.599 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:21.981 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:29:21.982 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:21.982 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348663/d1de0f0b-e845-43bb-bd68-7c943127b724/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:22.373 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:29:22.374 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:29:22.374 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0245752 s, 42.7 MB/s 2021-02-10T11:29:22.374 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:22.375 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:22.375 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:22.718 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:29:22.718 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:22.718 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348663/d1de0f0b-e845-43bb-bd68-7c943127b724/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:23.101 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:29:23.101 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:29:23.102 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0234964 s, 44.6 MB/s 2021-02-10T11:29:23.103 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:23.103 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:23.103 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:23.447 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:29:23.447 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:23.447 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348663/d1de0f0b-e845-43bb-bd68-7c943127b724/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:23.824 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:29:23.824 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:29:23.824 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0237932 s, 44.1 MB/s 2021-02-10T11:29:23.825 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:23.825 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:23.825 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:24.160 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:29:24.160 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:24.161 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348663/d1de0f0b-e845-43bb-bd68-7c943127b724/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:24.534 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:29:24.534 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:29:24.534 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0242577 s, 43.2 MB/s 2021-02-10T11:29:24.535 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:24.535 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:24.535 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:24.870 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:29:24.871 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:24.871 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348663/d1de0f0b-e845-43bb-bd68-7c943127b724/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:25.250 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:29:25.250 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:29:25.250 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0266469 s, 39.4 MB/s 2021-02-10T11:29:25.251 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:25.251 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:25.251 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:25.587 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:29:25.587 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:25.588 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348663/d1de0f0b-e845-43bb-bd68-7c943127b724/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:25.963 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:29:25.963 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:29:25.964 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0240848 s, 43.5 MB/s 2021-02-10T11:29:25.964 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:25.965 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:25.965 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:26.300 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:29:26.301 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:26.301 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348663/d1de0f0b-e845-43bb-bd68-7c943127b724/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:26.678 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:29:26.678 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:29:26.678 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0249221 s, 42.1 MB/s 2021-02-10T11:29:26.679 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000348664 2021-02-10T11:29:27.111 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.106 7fcc30289700 1 Processor -- start 2021-02-10T11:29:27.112 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.106 7fcc30289700 1 -- start start 2021-02-10T11:29:27.112 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.106 7fcc30289700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcc281b0470 0x7fcc281b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:27.112 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.106 7fcc30289700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc280f0b50 con 0x7fcc280ef610 2021-02-10T11:29:27.112 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.106 7fcc30289700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc280f24e0 con 0x7fcc280f20e0 2021-02-10T11:29:27.112 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.106 7fcc30289700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc280e5ea0 con 0x7fcc281b0470 2021-02-10T11:29:27.112 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.106 7fcc2d8a4700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fcc280ef610 0x7fcc280efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:45944/0 (socket says 172.21.2.112:45944) 2021-02-10T11:29:27.112 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.106 7fcc2d8a4700 1 -- 172.21.2.112:0/1026794553 learned_addr learned my addr 172.21.2.112:0/1026794553 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:29:27.112 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.107 7fcc2e0a5700 1 --2- 172.21.2.112:0/1026794553 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcc281b0470 0x7fcc281b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:27.113 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.107 7fcc2d0a3700 1 -- 172.21.2.112:0/1026794553 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2873688991 0 0) 0x7fcc1c0010b0 con 0x7fcc280ef610 2021-02-10T11:29:27.113 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.107 7fcc2d0a3700 1 -- 172.21.2.112:0/1026794553 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc14002fe0 con 0x7fcc280ef610 2021-02-10T11:29:27.113 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.107 7fcc2d0a3700 1 -- 172.21.2.112:0/1026794553 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1742691999 0 0) 0x7fcc20001400 con 0x7fcc280f20e0 2021-02-10T11:29:27.113 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.107 7fcc2d0a3700 1 -- 172.21.2.112:0/1026794553 --> v1:172.21.2.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc14004370 con 0x7fcc280f20e0 2021-02-10T11:29:27.113 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.107 7fcc2e0a5700 1 -- 172.21.2.112:0/1026794553 >> v1:172.21.2.120:6789/0 conn(0x7fcc280f20e0 legacy=0x7fcc280f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:27.113 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.107 7fcc2e0a5700 1 -- 172.21.2.112:0/1026794553 >> v1:172.21.2.112:6789/0 conn(0x7fcc280ef610 legacy=0x7fcc280efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:27.113 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.107 7fcc2e0a5700 1 -- 172.21.2.112:0/1026794553 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc280e52e0 con 0x7fcc281b0470 2021-02-10T11:29:27.113 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.107 7fcc2e0a5700 1 --2- 172.21.2.112:0/1026794553 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcc281b0470 0x7fcc281b0870 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7fcc24006650 tx=0x7fcc24009a80).ready entity=mon.2 client_cookie=3f1393db4052d3f3 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:27.113 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.107 7fcc2d0a3700 1 -- 172.21.2.112:0/1026794553 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcc2400c0c0 con 0x7fcc281b0470 2021-02-10T11:29:27.113 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.107 7fcc2d0a3700 1 -- 172.21.2.112:0/1026794553 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc2400c340 con 0x7fcc281b0470 2021-02-10T11:29:27.113 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.107 7fcc2d0a3700 1 -- 172.21.2.112:0/1026794553 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcc2400c860 con 0x7fcc281b0470 2021-02-10T11:29:27.113 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.107 7fcc30289700 1 -- 172.21.2.112:0/1026794553 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcc281b0470 msgr2=0x7fcc281b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:27.114 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.107 7fcc30289700 1 --2- 172.21.2.112:0/1026794553 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcc281b0470 0x7fcc281b0870 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7fcc24006650 tx=0x7fcc24009a80).stop 2021-02-10T11:29:27.114 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.108 7fcc30289700 1 -- 172.21.2.112:0/1026794553 shutdown_connections 2021-02-10T11:29:27.114 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.108 7fcc30289700 1 --2- 172.21.2.112:0/1026794553 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcc281b0470 0x7fcc281b0870 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:27.114 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.108 7fcc30289700 1 -- 172.21.2.112:0/1026794553 >> 172.21.2.112:0/1026794553 conn(0x7fcc280ce3e0 msgr2=0x7fcc280e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:29:27.114 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.108 7fcc30289700 1 -- 172.21.2.112:0/1026794553 shutdown_connections 2021-02-10T11:29:27.114 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.108 7fcc30289700 1 -- 172.21.2.112:0/1026794553 wait complete. 2021-02-10T11:29:27.115 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.123 7fcc30289700 1 Processor -- start 2021-02-10T11:29:27.115 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.123 7fcc30289700 1 -- start start 2021-02-10T11:29:27.116 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.124 7fcc30289700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcc280a01f0 0x7fcc2809f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:27.116 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.124 7fcc30289700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcc280add60 0x7fcc2809e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:27.116 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.124 7fcc30289700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcc2809edf0 0x7fcc2808c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:27.116 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.124 7fcc30289700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc2809fe20 con 0x7fcc280add60 2021-02-10T11:29:27.116 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.124 7fcc30289700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc280ae160 con 0x7fcc2809edf0 2021-02-10T11:29:27.116 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.124 7fcc30289700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc280ae380 con 0x7fcc280a01f0 2021-02-10T11:29:27.116 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.124 7fcc2e8a6700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcc2809edf0 0x7fcc2808c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:27.116 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.124 7fcc2d8a4700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcc280add60 0x7fcc2809e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:27.116 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.124 7fcc2d8a4700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcc280add60 0x7fcc2809e7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:46488/0 (socket says 172.21.2.112:46488) 2021-02-10T11:29:27.116 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.124 7fcc2d8a4700 1 -- 172.21.2.112:0/2630239985 learned_addr learned my addr 172.21.2.112:0/2630239985 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:29:27.116 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.124 7fcc2e8a6700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcc2809edf0 0x7fcc2808c680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3300/0 says I am v2:172.21.2.112:48766/0 (socket says 172.21.2.112:48766) 2021-02-10T11:29:27.116 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.124 7fcc2e0a5700 1 --2- 172.21.2.112:0/2630239985 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcc280a01f0 0x7fcc2809f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:27.116 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.124 7fcc2d8a4700 1 -- 172.21.2.112:0/2630239985 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcc280a01f0 msgr2=0x7fcc2809f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:27.117 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.124 7fcc2d8a4700 1 --2- 172.21.2.112:0/2630239985 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcc280a01f0 0x7fcc2809f800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:27.117 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.124 7fcc2d8a4700 1 -- 172.21.2.112:0/2630239985 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcc2809edf0 msgr2=0x7fcc2808c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:27.117 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.124 7fcc2d8a4700 1 --2- 172.21.2.112:0/2630239985 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcc2809edf0 0x7fcc2808c680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:27.117 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.124 7fcc2d8a4700 1 -- 172.21.2.112:0/2630239985 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc2808cca0 con 0x7fcc280add60 2021-02-10T11:29:27.117 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.125 7fcc2d8a4700 1 --2- 172.21.2.112:0/2630239985 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcc280add60 0x7fcc2809e7d0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 rx=0x7fcc1c001c60 tx=0x7fcc1c00a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:27.117 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.125 7fcc1affd700 1 -- 172.21.2.112:0/2630239985 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcc1c00acc0 con 0x7fcc280add60 2021-02-10T11:29:27.117 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.125 7fcc1affd700 1 -- 172.21.2.112:0/2630239985 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc1c004350 con 0x7fcc280add60 2021-02-10T11:29:27.117 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.125 7fcc1affd700 1 -- 172.21.2.112:0/2630239985 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcc1c004830 con 0x7fcc280add60 2021-02-10T11:29:27.117 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.125 7fcc30289700 1 -- 172.21.2.112:0/2630239985 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc2808cca0 con 0x7fcc280add60 2021-02-10T11:29:27.117 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.125 7fcc30289700 1 -- 172.21.2.112:0/2630239985 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc281069f0 con 0x7fcc280add60 2021-02-10T11:29:27.117 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.125 7fcc1affd700 1 -- 172.21.2.112:0/2630239985 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fcc1c004530 con 0x7fcc280add60 2021-02-10T11:29:27.117 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.126 7fcc1affd700 1 --2- 172.21.2.112:0/2630239985 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcc100186d0 0x7fcc1001abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:27.117 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.126 7fcc1affd700 1 -- 172.21.2.112:0/2630239985 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fcc1c0327b0 con 0x7fcc280add60 2021-02-10T11:29:27.118 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.126 7fcc2e0a5700 1 --2- 172.21.2.112:0/2630239985 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcc100186d0 0x7fcc1001abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:27.118 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.126 7fcc2e0a5700 1 --2- 172.21.2.112:0/2630239985 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcc100186d0 0x7fcc1001abd0 crc :-1 s=READY pgs=379 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:27.145 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.156 7fcc30289700 1 -- 172.21.2.112:0/2630239985 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc28105ae0 con 0x7fcc280add60 2021-02-10T11:29:27.147 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.159 7fcc1affd700 1 -- 172.21.2.112:0/2630239985 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fcc1c004140 con 0x7fcc280add60 2021-02-10T11:29:27.273 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.284 7fcc30289700 1 -- 172.21.2.112:0/2630239985 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348664", "target": ["mgr", ""]}) v1 -- 0x7fcc280f0b50 con 0x7fcc100186d0 2021-02-10T11:29:27.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.311 7fcc1affd700 1 -- 172.21.2.112:0/2630239985 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fcc24009180 con 0x7fcc100186d0 2021-02-10T11:29:27.306 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.318 7fcc30289700 1 -- 172.21.2.112:0/2630239985 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcc100186d0 msgr2=0x7fcc1001abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:27.306 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.318 7fcc30289700 1 --2- 172.21.2.112:0/2630239985 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcc100186d0 0x7fcc1001abd0 crc :-1 s=READY pgs=379 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:27.306 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.318 7fcc30289700 1 -- 172.21.2.112:0/2630239985 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcc280add60 msgr2=0x7fcc2809e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:27.306 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.318 7fcc30289700 1 --2- 172.21.2.112:0/2630239985 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcc280add60 0x7fcc2809e7d0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 rx=0x7fcc1c001c60 tx=0x7fcc1c00a8d0).stop 2021-02-10T11:29:27.307 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.318 7fcc30289700 1 -- 172.21.2.112:0/2630239985 shutdown_connections 2021-02-10T11:29:27.307 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.318 7fcc30289700 1 --2- 172.21.2.112:0/2630239985 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcc100186d0 0x7fcc1001abd0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:27.307 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.318 7fcc30289700 1 --2- 172.21.2.112:0/2630239985 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcc2809edf0 0x7fcc2808c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:27.307 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.318 7fcc30289700 1 --2- 172.21.2.112:0/2630239985 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcc280add60 0x7fcc2809e7d0 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:27.307 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.318 7fcc30289700 1 --2- 172.21.2.112:0/2630239985 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcc280a01f0 0x7fcc2809f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:27.307 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.318 7fcc30289700 1 -- 172.21.2.112:0/2630239985 >> 172.21.2.112:0/2630239985 conn(0x7fcc280bbf60 msgr2=0x7fcc28105d10 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:29:27.307 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.318 7fcc30289700 1 -- 172.21.2.112:0/2630239985 shutdown_connections 2021-02-10T11:29:27.307 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.318 7fcc30289700 1 -- 172.21.2.112:0/2630239985 wait complete. 2021-02-10T11:29:27.315 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000348664 2021-02-10T11:29:27.748 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.743 7f9c4eb4c700 1 Processor -- start 2021-02-10T11:29:27.749 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.743 7f9c4eb4c700 1 -- start start 2021-02-10T11:29:27.749 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.744 7f9c4eb4c700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f9c481b0470 0x7f9c481b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:27.749 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.744 7f9c4eb4c700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c480f0b50 con 0x7f9c480ef610 2021-02-10T11:29:27.749 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.744 7f9c4eb4c700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c480f24e0 con 0x7f9c480f20e0 2021-02-10T11:29:27.749 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.744 7f9c4eb4c700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c480e5ea0 con 0x7f9c481b0470 2021-02-10T11:29:27.749 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.744 7f9c47fff700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f9c480ef610 0x7f9c480efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:45956/0 (socket says 172.21.2.112:45956) 2021-02-10T11:29:27.749 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.744 7f9c47fff700 1 -- 172.21.2.112:0/4156195219 learned_addr learned my addr 172.21.2.112:0/4156195219 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:29:27.749 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.744 7f9c4c968700 1 --2- 172.21.2.112:0/4156195219 >> v2:172.21.2.120:3300/0 conn(0x7f9c481b0470 0x7f9c481b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:27.749 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.744 7f9c477fe700 1 -- 172.21.2.112:0/4156195219 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3240335721 0 0) 0x7f9c38001060 con 0x7f9c480ef610 2021-02-10T11:29:27.749 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.744 7f9c477fe700 1 -- 172.21.2.112:0/4156195219 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c34002fe0 con 0x7f9c480ef610 2021-02-10T11:29:27.749 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.744 7f9c477fe700 1 -- 172.21.2.112:0/4156195219 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4068336359 0 0) 0x7f9c400009b0 con 0x7f9c480f20e0 2021-02-10T11:29:27.750 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.744 7f9c477fe700 1 -- 172.21.2.112:0/4156195219 --> v1:172.21.2.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c34003210 con 0x7f9c480f20e0 2021-02-10T11:29:27.750 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.745 7f9c4c968700 1 -- 172.21.2.112:0/4156195219 >> v1:172.21.2.120:6789/0 conn(0x7f9c480f20e0 legacy=0x7f9c480f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:27.750 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.745 7f9c4c968700 1 -- 172.21.2.112:0/4156195219 >> v1:172.21.2.112:6789/0 conn(0x7f9c480ef610 legacy=0x7f9c480efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:27.750 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.745 7f9c4c968700 1 -- 172.21.2.112:0/4156195219 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c480e52e0 con 0x7f9c481b0470 2021-02-10T11:29:27.750 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.745 7f9c4c968700 1 --2- 172.21.2.112:0/4156195219 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9c481b0470 0x7f9c481b0870 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7f9c30006650 tx=0x7f9c30009a80).ready entity=mon.1 client_cookie=3df686370ff1f567 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:27.750 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.745 7f9c477fe700 1 -- 172.21.2.112:0/4156195219 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9c3000c0c0 con 0x7f9c481b0470 2021-02-10T11:29:27.750 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.745 7f9c477fe700 1 -- 172.21.2.112:0/4156195219 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c3000c340 con 0x7f9c481b0470 2021-02-10T11:29:27.750 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.745 7f9c477fe700 1 -- 172.21.2.112:0/4156195219 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9c3000c860 con 0x7f9c481b0470 2021-02-10T11:29:27.750 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.745 7f9c4eb4c700 1 -- 172.21.2.112:0/4156195219 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9c481b0470 msgr2=0x7f9c481b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:27.750 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.745 7f9c4eb4c700 1 --2- 172.21.2.112:0/4156195219 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9c481b0470 0x7f9c481b0870 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7f9c30006650 tx=0x7f9c30009a80).stop 2021-02-10T11:29:27.750 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.746 7f9c4eb4c700 1 -- 172.21.2.112:0/4156195219 shutdown_connections 2021-02-10T11:29:27.750 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.746 7f9c4eb4c700 1 --2- 172.21.2.112:0/4156195219 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9c481b0470 0x7f9c481b0870 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:27.750 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.746 7f9c4eb4c700 1 -- 172.21.2.112:0/4156195219 >> 172.21.2.112:0/4156195219 conn(0x7f9c480ce3e0 msgr2=0x7f9c480e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:29:27.751 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.746 7f9c4eb4c700 1 -- 172.21.2.112:0/4156195219 shutdown_connections 2021-02-10T11:29:27.751 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.746 7f9c4eb4c700 1 -- 172.21.2.112:0/4156195219 wait complete. 2021-02-10T11:29:27.751 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.760 7f9c4eb4c700 1 Processor -- start 2021-02-10T11:29:27.751 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.760 7f9c4eb4c700 1 -- start start 2021-02-10T11:29:27.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.761 7f9c4eb4c700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9c48105d10 0x7f9c4809eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:27.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.761 7f9c4eb4c700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9c48106110 0x7f9c4809f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:27.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.761 7f9c4eb4c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9c480b2a00 0x7f9c480b2e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:27.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.761 7f9c4eb4c700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c4809e560 con 0x7f9c480b2a00 2021-02-10T11:29:27.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.761 7f9c4eb4c700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c4809c090 con 0x7f9c48105d10 2021-02-10T11:29:27.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.761 7f9c4eb4c700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c480b3420 con 0x7f9c48106110 2021-02-10T11:29:27.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.761 7f9c4d169700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9c480b2a00 0x7f9c480b2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:27.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.761 7f9c4d169700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9c480b2a00 0x7f9c480b2e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:46504/0 (socket says 172.21.2.112:46504) 2021-02-10T11:29:27.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.761 7f9c4d169700 1 -- 172.21.2.112:0/1102849518 learned_addr learned my addr 172.21.2.112:0/1102849518 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:29:27.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.761 7f9c4c968700 1 --2- 172.21.2.112:0/1102849518 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9c48105d10 0x7f9c4809eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:27.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.761 7f9c4d169700 1 -- 172.21.2.112:0/1102849518 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9c48106110 msgr2=0x7f9c4809f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:27.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.761 7f9c4d169700 1 --2- 172.21.2.112:0/1102849518 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9c48106110 0x7f9c4809f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:27.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.761 7f9c4d169700 1 -- 172.21.2.112:0/1102849518 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9c48105d10 msgr2=0x7f9c4809eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:27.754 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.761 7f9c4d169700 1 --2- 172.21.2.112:0/1102849518 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9c48105d10 0x7f9c4809eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:27.754 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.761 7f9c4d169700 1 -- 172.21.2.112:0/1102849518 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c480b3640 con 0x7f9c480b2a00 2021-02-10T11:29:27.754 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.761 7f9c4d169700 1 --2- 172.21.2.112:0/1102849518 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9c480b2a00 0x7f9c480b2e00 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 rx=0x7f9c40006f10 tx=0x7f9c4000b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:27.754 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.762 7f9c457fa700 1 -- 172.21.2.112:0/1102849518 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9c4000bd80 con 0x7f9c480b2a00 2021-02-10T11:29:27.754 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.762 7f9c457fa700 1 -- 172.21.2.112:0/1102849518 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c40000e20 con 0x7f9c480b2a00 2021-02-10T11:29:27.754 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.762 7f9c457fa700 1 -- 172.21.2.112:0/1102849518 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9c400054f0 con 0x7f9c480b2a00 2021-02-10T11:29:27.754 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.762 7f9c4eb4c700 1 -- 172.21.2.112:0/1102849518 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c480b3640 con 0x7f9c480b2a00 2021-02-10T11:29:27.754 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.762 7f9c4eb4c700 1 -- 172.21.2.112:0/1102849518 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c480b3bd0 con 0x7f9c480b2a00 2021-02-10T11:29:27.754 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.762 7f9c457fa700 1 -- 172.21.2.112:0/1102849518 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9c40006030 con 0x7f9c480b2a00 2021-02-10T11:29:27.754 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.762 7f9c457fa700 1 --2- 172.21.2.112:0/1102849518 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9c2c0186c0 0x7f9c2c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:27.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.762 7f9c457fa700 1 -- 172.21.2.112:0/1102849518 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f9c400325e0 con 0x7f9c480b2a00 2021-02-10T11:29:27.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.763 7f9c4c968700 1 --2- 172.21.2.112:0/1102849518 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9c2c0186c0 0x7f9c2c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:27.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.763 7f9c4c968700 1 --2- 172.21.2.112:0/1102849518 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9c2c0186c0 0x7f9c2c01abc0 crc :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:27.782 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.794 7f9c4eb4c700 1 -- 172.21.2.112:0/1102849518 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c48105ae0 con 0x7f9c480b2a00 2021-02-10T11:29:27.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.796 7f9c457fa700 1 -- 172.21.2.112:0/1102849518 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9c40000940 con 0x7f9c480b2a00 2021-02-10T11:29:27.913 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.924 7f9c4eb4c700 1 -- 172.21.2.112:0/1102849518 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348664", "target": ["mgr", ""]}) v1 -- 0x7f9c480b3bd0 con 0x7f9c2c0186c0 2021-02-10T11:29:27.917 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.930 7f9c457fa700 1 -- 172.21.2.112:0/1102849518 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f9c30000c90 con 0x7f9c2c0186c0 2021-02-10T11:29:27.922 INFO:teuthology.orchestra.run.gibba012.stdout:/volumes/_nogroup/subvolume_0000000000348664/ac8f0713-d4b5-4f55-b30b-4eb4bc68bc97 2021-02-10T11:29:27.923 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.934 7f9c4eb4c700 1 -- 172.21.2.112:0/1102849518 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9c2c0186c0 msgr2=0x7f9c2c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:27.923 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.934 7f9c4eb4c700 1 --2- 172.21.2.112:0/1102849518 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9c2c0186c0 0x7f9c2c01abc0 crc :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:27.923 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.934 7f9c4eb4c700 1 -- 172.21.2.112:0/1102849518 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9c480b2a00 msgr2=0x7f9c480b2e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:27.923 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.934 7f9c4eb4c700 1 --2- 172.21.2.112:0/1102849518 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9c480b2a00 0x7f9c480b2e00 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 rx=0x7f9c40006f10 tx=0x7f9c4000b990).stop 2021-02-10T11:29:27.923 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.934 7f9c4eb4c700 1 -- 172.21.2.112:0/1102849518 shutdown_connections 2021-02-10T11:29:27.923 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.934 7f9c4eb4c700 1 --2- 172.21.2.112:0/1102849518 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9c2c0186c0 0x7f9c2c01abc0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:27.923 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.934 7f9c4eb4c700 1 --2- 172.21.2.112:0/1102849518 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9c480b2a00 0x7f9c480b2e00 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:27.923 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.934 7f9c4eb4c700 1 --2- 172.21.2.112:0/1102849518 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9c48106110 0x7f9c4809f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:27.924 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.934 7f9c4eb4c700 1 --2- 172.21.2.112:0/1102849518 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9c48105d10 0x7f9c4809eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:27.924 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.934 7f9c4eb4c700 1 -- 172.21.2.112:0/1102849518 >> 172.21.2.112:0/1102849518 conn(0x7f9c480c36f0 msgr2=0x7f9c480bd640 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:29:27.924 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.934 7f9c4eb4c700 1 -- 172.21.2.112:0/1102849518 shutdown_connections 2021-02-10T11:29:27.924 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:27.935 7f9c4eb4c700 1 -- 172.21.2.112:0/1102849518 wait complete. 2021-02-10T11:29:27.932 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000348664 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000348664/ac8f0713-d4b5-4f55-b30b-4eb4bc68bc97 2021-02-10T11:29:27.932 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:27.933 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:27.933 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:28.268 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:29:28.269 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:28.269 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348664/ac8f0713-d4b5-4f55-b30b-4eb4bc68bc97/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:28.651 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:29:28.651 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:29:28.651 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0254798 s, 41.2 MB/s 2021-02-10T11:29:28.652 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:28.652 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:28.652 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:28.989 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:29:28.990 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:28.990 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348664/ac8f0713-d4b5-4f55-b30b-4eb4bc68bc97/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:29.363 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:29:29.363 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:29:29.363 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0239301 s, 43.8 MB/s 2021-02-10T11:29:29.364 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:29.364 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:29.364 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:29.699 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:29:29.699 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:29.700 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348664/ac8f0713-d4b5-4f55-b30b-4eb4bc68bc97/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:30.076 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:29:30.076 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:29:30.076 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.024636 s, 42.6 MB/s 2021-02-10T11:29:30.077 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:30.077 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:30.077 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:30.413 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:29:30.414 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:30.414 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348664/ac8f0713-d4b5-4f55-b30b-4eb4bc68bc97/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:30.789 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:29:30.789 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:29:30.789 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0232295 s, 45.1 MB/s 2021-02-10T11:29:30.790 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:30.790 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:30.790 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:31.126 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:29:31.127 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:31.127 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348664/ac8f0713-d4b5-4f55-b30b-4eb4bc68bc97/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:31.503 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:29:31.503 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:29:31.503 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0228394 s, 45.9 MB/s 2021-02-10T11:29:31.504 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:31.505 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:31.505 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:31.841 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:29:31.841 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:31.842 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348664/ac8f0713-d4b5-4f55-b30b-4eb4bc68bc97/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:32.217 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:29:32.217 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:29:32.217 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0233711 s, 44.9 MB/s 2021-02-10T11:29:32.218 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:32.218 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:32.218 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:32.553 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:29:32.553 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:32.554 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348664/ac8f0713-d4b5-4f55-b30b-4eb4bc68bc97/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:32.930 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:29:32.930 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:29:32.930 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0234369 s, 44.7 MB/s 2021-02-10T11:29:32.931 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:32.931 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:32.931 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:33.267 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:29:33.268 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:33.268 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348664/ac8f0713-d4b5-4f55-b30b-4eb4bc68bc97/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:33.645 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:29:33.645 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:29:33.645 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0248022 s, 42.3 MB/s 2021-02-10T11:29:33.646 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:33.646 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:33.646 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:33.983 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:29:33.984 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:33.984 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348664/ac8f0713-d4b5-4f55-b30b-4eb4bc68bc97/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:34.358 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:29:34.358 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:29:34.358 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0239721 s, 43.7 MB/s 2021-02-10T11:29:34.359 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:34.359 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:34.359 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:34.693 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:29:34.693 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:34.694 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348664/ac8f0713-d4b5-4f55-b30b-4eb4bc68bc97/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:35.069 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:29:35.069 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:29:35.069 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0234753 s, 44.7 MB/s 2021-02-10T11:29:35.070 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000348665 2021-02-10T11:29:35.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:35.493 7fadc3bb1700 1 Processor -- start 2021-02-10T11:29:35.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:35.493 7fadc3bb1700 1 -- start start 2021-02-10T11:29:35.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:35.494 7fadc3bb1700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fadbc1b0870 0x7fadbc0f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:35.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:35.494 7fadc3bb1700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fadbc0efe00 0x7fadbc0f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:35.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:35.494 7fadc3bb1700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadbc1b0c70 con 0x7fadbc1b0470 2021-02-10T11:29:35.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:35.494 7fadc3bb1700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fadbc0e5ea0 con 0x7fadbc0efe00 2021-02-10T11:29:35.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:35.494 7fadc3bb1700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fadbc0f0b50 con 0x7fadbc1b0870 2021-02-10T11:29:35.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:35.494 7fadc21ce700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fadbc0efe00 0x7fadbc0f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:35.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:35.494 7fadc21ce700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fadbc0efe00 0x7fadbc0f0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:46512/0 (socket says 172.21.2.112:46512) 2021-02-10T11:29:35.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:35.494 7fadc21ce700 1 -- 172.21.2.112:0/2314125876 learned_addr learned my addr 172.21.2.112:0/2314125876 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:29:35.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:35.494 7fadc21ce700 1 -- 172.21.2.112:0/2314125876 >> v2:172.21.2.120:3300/0 conn(0x7fadbc1b0870 msgr2=0x7fadbc0f20e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:29:35.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:35.494 7fadc21ce700 1 --2- 172.21.2.112:0/2314125876 >> v2:172.21.2.120:3300/0 conn(0x7fadbc1b0870 0x7fadbc0f20e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:35.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:35.494 7fadc21ce700 1 -- 172.21.2.112:0/2314125876 >> v1:172.21.2.120:6789/0 conn(0x7fadbc1b0470 legacy=0x7fadbc0ef610 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:29:35.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:35.494 7fadc21ce700 1 -- 172.21.2.112:0/2314125876 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadbc0f0820 con 0x7fadbc0efe00 2021-02-10T11:29:35.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:35.494 7fadc21ce700 1 --2- 172.21.2.112:0/2314125876 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fadbc0efe00 0x7fadbc0f0200 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 rx=0x7fadb4006f00 tx=0x7fadb4009f10).ready entity=mon.0 client_cookie=e0e1df7b96aa8502 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:35.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:35.495 7fadc09cb700 1 -- 172.21.2.112:0/2314125876 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fadb400b3b0 con 0x7fadbc0efe00 2021-02-10T11:29:35.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:35.495 7fadc09cb700 1 -- 172.21.2.112:0/2314125876 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadb400b630 con 0x7fadbc0efe00 2021-02-10T11:29:35.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:35.495 7fadc09cb700 1 -- 172.21.2.112:0/2314125876 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fadb400ba90 con 0x7fadbc0efe00 2021-02-10T11:29:35.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:35.495 7fadc3bb1700 1 -- 172.21.2.112:0/2314125876 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fadbc0efe00 msgr2=0x7fadbc0f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:35.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:35.495 7fadc3bb1700 1 --2- 172.21.2.112:0/2314125876 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fadbc0efe00 0x7fadbc0f0200 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 rx=0x7fadb4006f00 tx=0x7fadb4009f10).stop 2021-02-10T11:29:35.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:35.495 7fadc3bb1700 1 -- 172.21.2.112:0/2314125876 shutdown_connections 2021-02-10T11:29:35.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:35.495 7fadc3bb1700 1 --2- 172.21.2.112:0/2314125876 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fadbc0efe00 0x7fadbc0f0200 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:35.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:35.495 7fadc3bb1700 1 --2- 172.21.2.112:0/2314125876 >> v2:172.21.2.120:3300/0 conn(0x7fadbc1b0870 0x7fadbc0f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:35.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:35.495 7fadc3bb1700 1 -- 172.21.2.112:0/2314125876 >> 172.21.2.112:0/2314125876 conn(0x7fadbc0ce3e0 msgr2=0x7fadbc0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:29:35.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:35.495 7fadc3bb1700 1 -- 172.21.2.112:0/2314125876 shutdown_connections 2021-02-10T11:29:35.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:35.495 7fadc3bb1700 1 -- 172.21.2.112:0/2314125876 wait complete. 2021-02-10T11:29:35.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:35.510 7fadc3bb1700 1 Processor -- start 2021-02-10T11:29:35.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:35.510 7fadc3bb1700 1 -- start start 2021-02-10T11:29:35.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:35.510 7fadc3bb1700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fadbc1a0420 0x7fadbc1a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:35.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:35.510 7fadc3bb1700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fadbc0c1a40 0x7fadbc0c1e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:35.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:35.510 7fadc3bb1700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fadbc0c2460 0x7fadbc0c2860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:35.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:35.510 7fadc3bb1700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadbc0c14c0 con 0x7fadbc1a0420 2021-02-10T11:29:35.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:35.510 7fadc3bb1700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadbc0a6ee0 con 0x7fadbc0c1a40 2021-02-10T11:29:35.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:35.510 7fadc3bb1700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadbc0bfc80 con 0x7fadbc0c2460 2021-02-10T11:29:35.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:35.510 7fadc19cd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fadbc1a0420 0x7fadbc1a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:35.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:35.510 7fadc19cd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fadbc1a0420 0x7fadbc1a0820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:46514/0 (socket says 172.21.2.112:46514) 2021-02-10T11:29:35.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:35.510 7fadc19cd700 1 -- 172.21.2.112:0/1339241391 learned_addr learned my addr 172.21.2.112:0/1339241391 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:29:35.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:35.510 7fadc19cd700 1 -- 172.21.2.112:0/1339241391 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fadbc0c2460 msgr2=0x7fadbc0c2860 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:29:35.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:35.510 7fadc19cd700 1 --2- 172.21.2.112:0/1339241391 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fadbc0c2460 0x7fadbc0c2860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:35.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:35.510 7fadc19cd700 1 -- 172.21.2.112:0/1339241391 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fadbc0c1a40 msgr2=0x7fadbc0c1e40 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:29:35.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:35.510 7fadc19cd700 1 --2- 172.21.2.112:0/1339241391 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fadbc0c1a40 0x7fadbc0c1e40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:35.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:35.510 7fadc19cd700 1 -- 172.21.2.112:0/1339241391 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadbc0bfea0 con 0x7fadbc1a0420 2021-02-10T11:29:35.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:35.511 7fadc19cd700 1 --2- 172.21.2.112:0/1339241391 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fadbc1a0420 0x7fadbc1a0820 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 rx=0x7fadb000b5b0 tx=0x7fadb000b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:35.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:35.511 7fadae7fc700 1 -- 172.21.2.112:0/1339241391 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fadb000bd40 con 0x7fadbc1a0420 2021-02-10T11:29:35.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:35.511 7fadae7fc700 1 -- 172.21.2.112:0/1339241391 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadb0006240 con 0x7fadbc1a0420 2021-02-10T11:29:35.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:35.511 7fadae7fc700 1 -- 172.21.2.112:0/1339241391 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fadb0005720 con 0x7fadbc1a0420 2021-02-10T11:29:35.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:35.511 7fadc3bb1700 1 -- 172.21.2.112:0/1339241391 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadbc0bfea0 con 0x7fadbc1a0420 2021-02-10T11:29:35.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:35.511 7fadc3bb1700 1 -- 172.21.2.112:0/1339241391 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadbc0bfea0 con 0x7fadbc1a0420 2021-02-10T11:29:35.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:35.511 7fadae7fc700 1 -- 172.21.2.112:0/1339241391 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fadb00054a0 con 0x7fadbc1a0420 2021-02-10T11:29:35.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:35.513 7fadae7fc700 1 --2- 172.21.2.112:0/1339241391 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fada401cae0 0x7fada401efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:35.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:35.513 7fadae7fc700 1 -- 172.21.2.112:0/1339241391 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fadb0032400 con 0x7fadbc1a0420 2021-02-10T11:29:35.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:35.513 7fadc11cc700 1 --2- 172.21.2.112:0/1339241391 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fada401cae0 0x7fada401efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:35.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:35.513 7fadc11cc700 1 --2- 172.21.2.112:0/1339241391 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fada401cae0 0x7fada401efe0 crc :-1 s=READY pgs=381 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:35.532 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:35.544 7fadc3bb1700 1 -- 172.21.2.112:0/1339241391 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadbc0eda20 con 0x7fadbc1a0420 2021-02-10T11:29:35.535 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:35.546 7fadae7fc700 1 -- 172.21.2.112:0/1339241391 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fadb0030d30 con 0x7fadbc1a0420 2021-02-10T11:29:35.663 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:35.675 7fadc3bb1700 1 -- 172.21.2.112:0/1339241391 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348665", "target": ["mgr", ""]}) v1 -- 0x7fadbc0eda20 con 0x7fada401cae0 2021-02-10T11:29:35.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:35.700 7fadae7fc700 1 -- 172.21.2.112:0/1339241391 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fadb80009b0 con 0x7fada401cae0 2021-02-10T11:29:35.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:35.707 7fadc3bb1700 1 -- 172.21.2.112:0/1339241391 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fada401cae0 msgr2=0x7fada401efe0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:35.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:35.707 7fadc3bb1700 1 --2- 172.21.2.112:0/1339241391 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fada401cae0 0x7fada401efe0 crc :-1 s=READY pgs=381 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:35.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:35.707 7fadc3bb1700 1 -- 172.21.2.112:0/1339241391 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fadbc1a0420 msgr2=0x7fadbc1a0820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:35.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:35.707 7fadc3bb1700 1 --2- 172.21.2.112:0/1339241391 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fadbc1a0420 0x7fadbc1a0820 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 rx=0x7fadb000b5b0 tx=0x7fadb000b950).stop 2021-02-10T11:29:35.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:35.707 7fadc3bb1700 1 -- 172.21.2.112:0/1339241391 shutdown_connections 2021-02-10T11:29:35.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:35.707 7fadc3bb1700 1 --2- 172.21.2.112:0/1339241391 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fada401cae0 0x7fada401efe0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:35.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:35.707 7fadc3bb1700 1 --2- 172.21.2.112:0/1339241391 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fadbc0c1a40 0x7fadbc0c1e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:35.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:35.707 7fadc3bb1700 1 --2- 172.21.2.112:0/1339241391 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fadbc0c2460 0x7fadbc0c2860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:35.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:35.707 7fadc3bb1700 1 --2- 172.21.2.112:0/1339241391 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fadbc1a0420 0x7fadbc1a0820 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:35.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:35.707 7fadc3bb1700 1 -- 172.21.2.112:0/1339241391 >> 172.21.2.112:0/1339241391 conn(0x7fadbc09dfd0 msgr2=0x7fadbc09abf0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:29:35.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:35.707 7fadc3bb1700 1 -- 172.21.2.112:0/1339241391 shutdown_connections 2021-02-10T11:29:35.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:35.707 7fadc3bb1700 1 -- 172.21.2.112:0/1339241391 wait complete. 2021-02-10T11:29:35.704 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000348665 2021-02-10T11:29:36.135 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:36.130 7fd0b7d2f700 1 Processor -- start 2021-02-10T11:29:36.136 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:36.130 7fd0b7d2f700 1 -- start start 2021-02-10T11:29:36.136 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:36.131 7fd0b7d2f700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd0b01b0470 0x7fd0b01b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:36.136 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:36.131 7fd0b7d2f700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fd0b00ef610 0x7fd0b00efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:36.136 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:36.131 7fd0b7d2f700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fd0b00f0030 0x7fd0b00f0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:36.136 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:36.131 7fd0b7d2f700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0b00e5ea0 con 0x7fd0b00ef610 2021-02-10T11:29:36.136 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:36.131 7fd0b7d2f700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0b00f0b50 con 0x7fd0b00f0030 2021-02-10T11:29:36.136 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:36.131 7fd0b7d2f700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0b00e52e0 con 0x7fd0b01b0470 2021-02-10T11:29:36.136 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:36.131 7fd0b534a700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fd0b00ef610 0x7fd0b00efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:36.136 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:36.131 7fd0b534a700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fd0b00ef610 0x7fd0b00efa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:46520/0 (socket says 172.21.2.112:46520) 2021-02-10T11:29:36.136 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:36.131 7fd0b534a700 1 -- 172.21.2.112:0/788590732 learned_addr learned my addr 172.21.2.112:0/788590732 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:29:36.136 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:36.131 7fd0b534a700 1 -- 172.21.2.112:0/788590732 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd0b01b0470 msgr2=0x7fd0b01b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:29:36.136 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:36.131 7fd0b534a700 1 --2- 172.21.2.112:0/788590732 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd0b01b0470 0x7fd0b01b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:36.136 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:36.131 7fd0b534a700 1 -- 172.21.2.112:0/788590732 >> v2:172.21.2.120:3300/0 conn(0x7fd0b00f0030 msgr2=0x7fd0b00f0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:29:36.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:36.131 7fd0b534a700 1 --2- 172.21.2.112:0/788590732 >> v2:172.21.2.120:3300/0 conn(0x7fd0b00f0030 0x7fd0b00f0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:36.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:36.131 7fd0b534a700 1 -- 172.21.2.112:0/788590732 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0b00f10d0 con 0x7fd0b00ef610 2021-02-10T11:29:36.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:36.132 7fd0b534a700 1 --2- 172.21.2.112:0/788590732 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd0b00ef610 0x7fd0b00efa10 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 rx=0x7fd0a4006700 tx=0x7fd0a4009b00).ready entity=mon.0 client_cookie=11cbab03dd62cf10 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:36.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:36.132 7fd0b4b49700 1 -- 172.21.2.112:0/788590732 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd0a400c0c0 con 0x7fd0b00ef610 2021-02-10T11:29:36.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:36.132 7fd0b4b49700 1 -- 172.21.2.112:0/788590732 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0a400c340 con 0x7fd0b00ef610 2021-02-10T11:29:36.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:36.132 7fd0b4b49700 1 -- 172.21.2.112:0/788590732 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd0a400c7a0 con 0x7fd0b00ef610 2021-02-10T11:29:36.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:36.132 7fd0b7d2f700 1 -- 172.21.2.112:0/788590732 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd0b00ef610 msgr2=0x7fd0b00efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:36.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:36.132 7fd0b7d2f700 1 --2- 172.21.2.112:0/788590732 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd0b00ef610 0x7fd0b00efa10 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 rx=0x7fd0a4006700 tx=0x7fd0a4009b00).stop 2021-02-10T11:29:36.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:36.133 7fd0b7d2f700 1 -- 172.21.2.112:0/788590732 shutdown_connections 2021-02-10T11:29:36.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:36.133 7fd0b7d2f700 1 --2- 172.21.2.112:0/788590732 >> v2:172.21.2.120:3300/0 conn(0x7fd0b00f0030 0x7fd0b00f0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:36.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:36.133 7fd0b7d2f700 1 --2- 172.21.2.112:0/788590732 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd0b00ef610 0x7fd0b00efa10 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:36.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:36.133 7fd0b7d2f700 1 --2- 172.21.2.112:0/788590732 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd0b01b0470 0x7fd0b01b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:36.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:36.133 7fd0b7d2f700 1 -- 172.21.2.112:0/788590732 >> 172.21.2.112:0/788590732 conn(0x7fd0b00ce3e0 msgr2=0x7fd0b00e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:29:36.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:36.133 7fd0b7d2f700 1 -- 172.21.2.112:0/788590732 shutdown_connections 2021-02-10T11:29:36.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:36.133 7fd0b7d2f700 1 -- 172.21.2.112:0/788590732 wait complete. 2021-02-10T11:29:36.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:36.148 7fd0b7d2f700 1 Processor -- start 2021-02-10T11:29:36.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:36.148 7fd0b7d2f700 1 -- start start 2021-02-10T11:29:36.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:36.148 7fd0b7d2f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd0b0193230 0x7fd0b00a6a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:36.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:36.148 7fd0b7d2f700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd0b008f5b0 0x7fd0b008f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:36.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:36.148 7fd0b7d2f700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd0b008e6a0 0x7fd0b008eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:36.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:36.148 7fd0b7d2f700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0b008ffd0 con 0x7fd0b0193230 2021-02-10T11:29:36.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:36.148 7fd0b7d2f700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0b0090240 con 0x7fd0b008e6a0 2021-02-10T11:29:36.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:36.148 7fd0b7d2f700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0b008f0c0 con 0x7fd0b008f5b0 2021-02-10T11:29:36.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:36.148 7fd0b5b4b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd0b0193230 0x7fd0b00a6a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:36.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:36.149 7fd0b5b4b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd0b0193230 0x7fd0b00a6a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:46522/0 (socket says 172.21.2.112:46522) 2021-02-10T11:29:36.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:36.149 7fd0b5b4b700 1 -- 172.21.2.112:0/596439948 learned_addr learned my addr 172.21.2.112:0/596439948 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:29:36.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:36.149 7fd0b534a700 1 --2- 172.21.2.112:0/596439948 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd0b008f5b0 0x7fd0b008f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:36.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:36.149 7fd0b634c700 1 --2- 172.21.2.112:0/596439948 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd0b008e6a0 0x7fd0b008eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:36.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:36.149 7fd0b5b4b700 1 -- 172.21.2.112:0/596439948 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd0b008f5b0 msgr2=0x7fd0b008f9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:36.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:36.149 7fd0b5b4b700 1 --2- 172.21.2.112:0/596439948 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd0b008f5b0 0x7fd0b008f9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:36.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:36.149 7fd0b5b4b700 1 -- 172.21.2.112:0/596439948 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd0b008e6a0 msgr2=0x7fd0b008eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:36.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:36.149 7fd0b5b4b700 1 --2- 172.21.2.112:0/596439948 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd0b008e6a0 0x7fd0b008eaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:36.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:36.149 7fd0b5b4b700 1 -- 172.21.2.112:0/596439948 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0b008f2a0 con 0x7fd0b0193230 2021-02-10T11:29:36.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:36.149 7fd0b534a700 1 --2- 172.21.2.112:0/596439948 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd0b008f5b0 0x7fd0b008f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:29:36.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:36.149 7fd0b5b4b700 1 --2- 172.21.2.112:0/596439948 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd0b0193230 0x7fd0b00a6a80 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 rx=0x7fd0ac006f00 tx=0x7fd0ac00b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:36.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:36.149 7fd0a27fc700 1 -- 172.21.2.112:0/596439948 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd0ac00bd30 con 0x7fd0b0193230 2021-02-10T11:29:36.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:36.149 7fd0a27fc700 1 -- 172.21.2.112:0/596439948 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0ac000c90 con 0x7fd0b0193230 2021-02-10T11:29:36.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:36.149 7fd0b7d2f700 1 -- 172.21.2.112:0/596439948 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0b008f2a0 con 0x7fd0b0193230 2021-02-10T11:29:36.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:36.149 7fd0a27fc700 1 -- 172.21.2.112:0/596439948 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd0ac0022a0 con 0x7fd0b0193230 2021-02-10T11:29:36.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:36.150 7fd0b7d2f700 1 -- 172.21.2.112:0/596439948 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0b008f2a0 con 0x7fd0b0193230 2021-02-10T11:29:36.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:36.150 7fd0a27fc700 1 -- 172.21.2.112:0/596439948 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd0ac005310 con 0x7fd0b0193230 2021-02-10T11:29:36.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:36.150 7fd0a27fc700 1 --2- 172.21.2.112:0/596439948 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd098018730 0x7fd09801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:36.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:36.150 7fd0a27fc700 1 -- 172.21.2.112:0/596439948 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fd0ac032770 con 0x7fd0b0193230 2021-02-10T11:29:36.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:36.151 7fd0b534a700 1 --2- 172.21.2.112:0/596439948 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd098018730 0x7fd09801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:36.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:36.151 7fd0b534a700 1 --2- 172.21.2.112:0/596439948 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd098018730 0x7fd09801ac10 crc :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:36.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:36.180 7fd0b7d2f700 1 -- 172.21.2.112:0/596439948 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0b00f0b50 con 0x7fd0b0193230 2021-02-10T11:29:36.170 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:36.183 7fd0a27fc700 1 -- 172.21.2.112:0/596439948 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd0ac006b90 con 0x7fd0b0193230 2021-02-10T11:29:36.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:36.314 7fd0b7d2f700 1 -- 172.21.2.112:0/596439948 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348665", "target": ["mgr", ""]}) v1 -- 0x7fd0b00f0b50 con 0x7fd098018730 2021-02-10T11:29:36.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:36.317 7fd0a27fc700 1 -- 172.21.2.112:0/596439948 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7fd0a4009100 con 0x7fd098018730 2021-02-10T11:29:36.309 INFO:teuthology.orchestra.run.gibba012.stdout:/volumes/_nogroup/subvolume_0000000000348665/f1550576-40f7-45c8-a208-eb0da99c89bd 2021-02-10T11:29:36.311 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:36.323 7fd0b7d2f700 1 -- 172.21.2.112:0/596439948 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd098018730 msgr2=0x7fd09801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:36.311 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:36.323 7fd0b7d2f700 1 --2- 172.21.2.112:0/596439948 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd098018730 0x7fd09801ac10 crc :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:36.311 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:36.323 7fd0b7d2f700 1 -- 172.21.2.112:0/596439948 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd0b0193230 msgr2=0x7fd0b00a6a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:36.311 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:36.323 7fd0b7d2f700 1 --2- 172.21.2.112:0/596439948 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd0b0193230 0x7fd0b00a6a80 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 rx=0x7fd0ac006f00 tx=0x7fd0ac00b940).stop 2021-02-10T11:29:36.311 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:36.323 7fd0b7d2f700 1 -- 172.21.2.112:0/596439948 shutdown_connections 2021-02-10T11:29:36.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:36.323 7fd0b7d2f700 1 --2- 172.21.2.112:0/596439948 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd098018730 0x7fd09801ac10 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:36.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:36.323 7fd0b7d2f700 1 --2- 172.21.2.112:0/596439948 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd0b008e6a0 0x7fd0b008eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:36.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:36.323 7fd0b7d2f700 1 --2- 172.21.2.112:0/596439948 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd0b008f5b0 0x7fd0b008f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:36.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:36.323 7fd0b7d2f700 1 --2- 172.21.2.112:0/596439948 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd0b0193230 0x7fd0b00a6a80 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:36.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:36.323 7fd0b7d2f700 1 -- 172.21.2.112:0/596439948 >> 172.21.2.112:0/596439948 conn(0x7fd0b00bf4e0 msgr2=0x7fd0b00be160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:29:36.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:36.323 7fd0b7d2f700 1 -- 172.21.2.112:0/596439948 shutdown_connections 2021-02-10T11:29:36.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:36.323 7fd0b7d2f700 1 -- 172.21.2.112:0/596439948 wait complete. 2021-02-10T11:29:36.319 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000348665 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000348665/f1550576-40f7-45c8-a208-eb0da99c89bd 2021-02-10T11:29:36.319 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:36.319 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:36.319 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:36.653 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:29:36.654 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:36.654 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348665/f1550576-40f7-45c8-a208-eb0da99c89bd/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:37.037 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:29:37.037 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:29:37.037 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0257278 s, 40.8 MB/s 2021-02-10T11:29:37.038 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:37.038 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:37.038 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:37.372 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:29:37.373 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:37.373 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348665/f1550576-40f7-45c8-a208-eb0da99c89bd/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:37.747 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:29:37.747 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:29:37.747 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.024938 s, 42.0 MB/s 2021-02-10T11:29:37.748 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:37.748 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:37.748 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:38.083 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:29:38.084 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:38.084 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348665/f1550576-40f7-45c8-a208-eb0da99c89bd/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:38.460 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:29:38.460 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:29:38.460 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.024907 s, 42.1 MB/s 2021-02-10T11:29:38.461 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:38.461 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:38.461 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:38.794 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:29:38.795 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:38.795 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348665/f1550576-40f7-45c8-a208-eb0da99c89bd/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:39.171 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:29:39.171 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:29:39.172 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0256825 s, 40.8 MB/s 2021-02-10T11:29:39.173 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:39.173 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:39.173 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:39.508 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:29:39.509 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:39.509 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348665/f1550576-40f7-45c8-a208-eb0da99c89bd/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:39.880 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:29:39.881 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:29:39.881 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0244465 s, 42.9 MB/s 2021-02-10T11:29:39.882 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:39.882 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:39.882 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:40.216 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:29:40.217 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:40.217 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348665/f1550576-40f7-45c8-a208-eb0da99c89bd/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:40.592 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:29:40.592 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:29:40.592 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0249306 s, 42.1 MB/s 2021-02-10T11:29:40.593 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:40.593 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:40.593 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:40.928 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:29:40.929 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:40.929 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348665/f1550576-40f7-45c8-a208-eb0da99c89bd/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:41.306 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:29:41.306 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:29:41.306 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0253048 s, 41.4 MB/s 2021-02-10T11:29:41.307 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:41.307 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:41.307 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:41.641 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:29:41.642 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:41.642 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348665/f1550576-40f7-45c8-a208-eb0da99c89bd/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:42.017 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:29:42.017 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:29:42.017 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0258958 s, 40.5 MB/s 2021-02-10T11:29:42.018 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:42.018 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:42.019 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:42.351 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:29:42.352 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:42.352 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348665/f1550576-40f7-45c8-a208-eb0da99c89bd/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:42.725 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:29:42.725 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:29:42.725 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0231123 s, 45.4 MB/s 2021-02-10T11:29:42.726 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:42.727 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:42.727 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:43.061 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:29:43.061 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:43.062 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348665/f1550576-40f7-45c8-a208-eb0da99c89bd/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:43.233 DEBUG:teuthology.orchestra.run.gibba012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:29:43.236 DEBUG:teuthology.orchestra.run.gibba020:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:29:43.435 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:29:43.435 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:29:43.435 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0244491 s, 42.9 MB/s 2021-02-10T11:29:43.437 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000348666 2021-02-10T11:29:43.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:43.860 7f6668da4700 1 Processor -- start 2021-02-10T11:29:43.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:43.860 7f6668da4700 1 -- start start 2021-02-10T11:29:43.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:43.860 7f6668da4700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f66641b0470 0x7f66641b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:43.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:43.860 7f6668da4700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66640f0b50 con 0x7f66640f20e0 2021-02-10T11:29:43.865 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:43.860 7f6668da4700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66640f24e0 con 0x7f66640ef610 2021-02-10T11:29:43.865 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:43.860 7f6668da4700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66640e5ea0 con 0x7f66641b0470 2021-02-10T11:29:43.865 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:43.861 7f6663314700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f66640f20e0 0x7f66640f0200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:45996/0 (socket says 172.21.2.112:45996) 2021-02-10T11:29:43.865 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:43.861 7f6663314700 1 -- 172.21.2.112:0/3231254243 learned_addr learned my addr 172.21.2.112:0/3231254243 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:29:43.865 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:43.861 7f6662b13700 1 --2- 172.21.2.112:0/3231254243 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f66641b0470 0x7f66641b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:43.865 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:43.861 7f6661b11700 1 -- 172.21.2.112:0/3231254243 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2357437352 0 0) 0x7f665c001180 con 0x7f66640f20e0 2021-02-10T11:29:43.865 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:43.861 7f6661b11700 1 -- 172.21.2.112:0/3231254243 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6650002fe0 con 0x7f66640f20e0 2021-02-10T11:29:43.865 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:43.861 7f6661b11700 1 -- 172.21.2.112:0/3231254243 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 726325804 0 0) 0x7f6654000ff0 con 0x7f66640ef610 2021-02-10T11:29:43.865 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:43.861 7f6661b11700 1 -- 172.21.2.112:0/3231254243 --> v1:172.21.2.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6650003020 con 0x7f66640ef610 2021-02-10T11:29:43.866 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:43.861 7f6662b13700 1 -- 172.21.2.112:0/3231254243 >> v1:172.21.2.120:6789/0 conn(0x7f66640ef610 legacy=0x7f66640efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:43.866 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:43.861 7f6662b13700 1 -- 172.21.2.112:0/3231254243 >> v1:172.21.2.112:6789/0 conn(0x7f66640f20e0 legacy=0x7f66640f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:43.866 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:43.861 7f6662b13700 1 -- 172.21.2.112:0/3231254243 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66640e52e0 con 0x7f66641b0470 2021-02-10T11:29:43.866 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:43.861 7f6662b13700 1 --2- 172.21.2.112:0/3231254243 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f66641b0470 0x7f66641b0870 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f664c0069d0 tx=0x7f664c009e40).ready entity=mon.2 client_cookie=f4042fa22d997743 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:43.866 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:43.862 7f6661b11700 1 -- 172.21.2.112:0/3231254243 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f664c00b510 con 0x7f66641b0470 2021-02-10T11:29:43.866 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:43.862 7f6661b11700 1 -- 172.21.2.112:0/3231254243 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f664c00b890 con 0x7f66641b0470 2021-02-10T11:29:43.866 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:43.862 7f6661b11700 1 -- 172.21.2.112:0/3231254243 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f664c00bd70 con 0x7f66641b0470 2021-02-10T11:29:43.866 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:43.862 7f6668da4700 1 -- 172.21.2.112:0/3231254243 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f66641b0470 msgr2=0x7f66641b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:43.866 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:43.862 7f6668da4700 1 --2- 172.21.2.112:0/3231254243 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f66641b0470 0x7f66641b0870 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f664c0069d0 tx=0x7f664c009e40).stop 2021-02-10T11:29:43.866 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:43.862 7f6668da4700 1 -- 172.21.2.112:0/3231254243 shutdown_connections 2021-02-10T11:29:43.866 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:43.862 7f6668da4700 1 --2- 172.21.2.112:0/3231254243 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f66641b0470 0x7f66641b0870 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:43.866 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:43.862 7f6668da4700 1 -- 172.21.2.112:0/3231254243 >> 172.21.2.112:0/3231254243 conn(0x7f66640ce3e0 msgr2=0x7f66640e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:29:43.866 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:43.862 7f6668da4700 1 -- 172.21.2.112:0/3231254243 shutdown_connections 2021-02-10T11:29:43.867 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:43.862 7f6668da4700 1 -- 172.21.2.112:0/3231254243 wait complete. 2021-02-10T11:29:43.867 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:43.877 7f6668da4700 1 Processor -- start 2021-02-10T11:29:43.867 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:43.877 7f6668da4700 1 -- start start 2021-02-10T11:29:43.867 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:43.877 7f6668da4700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f66640a0420 0x7f66640a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:43.867 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:43.877 7f6668da4700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6664189d00 0x7f666418a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:43.868 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:43.877 7f6668da4700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f666418a720 0x7f66640b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:43.868 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:43.877 7f6668da4700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6664189780 con 0x7f6664189d00 2021-02-10T11:29:43.868 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:43.877 7f6668da4700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f666418ab20 con 0x7f66640a0420 2021-02-10T11:29:43.868 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:43.877 7f6668da4700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66640a1140 con 0x7f666418a720 2021-02-10T11:29:43.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:43.877 7f6662b13700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f66640a0420 0x7f66640a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:43.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:43.877 7f6662312700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6664189d00 0x7f666418a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:43.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:43.877 7f6662312700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6664189d00 0x7f666418a100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:46536/0 (socket says 172.21.2.112:46536) 2021-02-10T11:29:43.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:43.877 7f6662312700 1 -- 172.21.2.112:0/2333983879 learned_addr learned my addr 172.21.2.112:0/2333983879 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:29:43.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:43.877 7f6663314700 1 --2- 172.21.2.112:0/2333983879 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f666418a720 0x7f66640b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:43.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:43.878 7f6662b13700 1 -- 172.21.2.112:0/2333983879 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f666418a720 msgr2=0x7f66640b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:43.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:43.878 7f6662b13700 1 --2- 172.21.2.112:0/2333983879 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f666418a720 0x7f66640b7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:43.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:43.878 7f6662b13700 1 -- 172.21.2.112:0/2333983879 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6664189d00 msgr2=0x7f666418a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:43.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:43.878 7f6662b13700 1 --2- 172.21.2.112:0/2333983879 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6664189d00 0x7f666418a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:43.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:43.878 7f6662b13700 1 -- 172.21.2.112:0/2333983879 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66640b8200 con 0x7f66640a0420 2021-02-10T11:29:43.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:43.878 7f6663314700 1 --2- 172.21.2.112:0/2333983879 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f666418a720 0x7f66640b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:29:43.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:43.878 7f6662312700 1 --2- 172.21.2.112:0/2333983879 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6664189d00 0x7f666418a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:29:43.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:43.878 7f6662b13700 1 --2- 172.21.2.112:0/2333983879 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f66640a0420 0x7f66640a0820 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7f664c00b8a0 tx=0x7f664c00bc40).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:43.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:43.878 7f665b7fe700 1 -- 172.21.2.112:0/2333983879 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f664c006030 con 0x7f66640a0420 2021-02-10T11:29:43.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:43.878 7f665b7fe700 1 -- 172.21.2.112:0/2333983879 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f664c0063b0 con 0x7f66640a0420 2021-02-10T11:29:43.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:43.878 7f665b7fe700 1 -- 172.21.2.112:0/2333983879 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f664c0058e0 con 0x7f66640a0420 2021-02-10T11:29:43.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:43.878 7f6668da4700 1 -- 172.21.2.112:0/2333983879 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66640b8200 con 0x7f66640a0420 2021-02-10T11:29:43.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:43.878 7f6668da4700 1 -- 172.21.2.112:0/2333983879 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66640b8200 con 0x7f66640a0420 2021-02-10T11:29:43.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:43.879 7f665b7fe700 1 -- 172.21.2.112:0/2333983879 <== mon.1 v2:172.21.2.120:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f664c006030 con 0x7f66640a0420 2021-02-10T11:29:43.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:43.879 7f665b7fe700 1 --2- 172.21.2.112:0/2333983879 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f66480186c0 0x7f664801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:43.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:43.879 7f665b7fe700 1 -- 172.21.2.112:0/2333983879 <== mon.1 v2:172.21.2.120:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f664c032a40 con 0x7f66640a0420 2021-02-10T11:29:43.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:43.879 7f6662312700 1 --2- 172.21.2.112:0/2333983879 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f66480186c0 0x7f664801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:43.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:43.880 7f6662312700 1 --2- 172.21.2.112:0/2333983879 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f66480186c0 0x7f664801abc0 crc :-1 s=READY pgs=383 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:43.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:43.910 7f6668da4700 1 -- 172.21.2.112:0/2333983879 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6664105ae0 con 0x7f66640a0420 2021-02-10T11:29:43.900 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:43.913 7f665b7fe700 1 -- 172.21.2.112:0/2333983879 <== mon.1 v2:172.21.2.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f664c00bd80 con 0x7f66640a0420 2021-02-10T11:29:44.026 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:44.038 7f6668da4700 1 -- 172.21.2.112:0/2333983879 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348666", "target": ["mgr", ""]}) v1 -- 0x7f6664189780 con 0x7f66480186c0 2021-02-10T11:29:44.052 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:44.065 7f665b7fe700 1 -- 172.21.2.112:0/2333983879 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f66540009b0 con 0x7f66480186c0 2021-02-10T11:29:44.059 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:44.072 7f6668da4700 1 -- 172.21.2.112:0/2333983879 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f66480186c0 msgr2=0x7f664801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:44.059 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:44.072 7f6668da4700 1 --2- 172.21.2.112:0/2333983879 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f66480186c0 0x7f664801abc0 crc :-1 s=READY pgs=383 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:44.059 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:44.072 7f6668da4700 1 -- 172.21.2.112:0/2333983879 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f66640a0420 msgr2=0x7f66640a0820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:44.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:44.072 7f6668da4700 1 --2- 172.21.2.112:0/2333983879 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f66640a0420 0x7f66640a0820 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7f664c00b8a0 tx=0x7f664c00bc40).stop 2021-02-10T11:29:44.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:44.072 7f6668da4700 1 -- 172.21.2.112:0/2333983879 shutdown_connections 2021-02-10T11:29:44.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:44.072 7f6668da4700 1 --2- 172.21.2.112:0/2333983879 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f66480186c0 0x7f664801abc0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:44.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:44.072 7f6668da4700 1 --2- 172.21.2.112:0/2333983879 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f666418a720 0x7f66640b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:44.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:44.072 7f6668da4700 1 --2- 172.21.2.112:0/2333983879 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6664189d00 0x7f666418a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:44.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:44.072 7f6668da4700 1 --2- 172.21.2.112:0/2333983879 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f66640a0420 0x7f66640a0820 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:44.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:44.072 7f6668da4700 1 -- 172.21.2.112:0/2333983879 >> 172.21.2.112:0/2333983879 conn(0x7f6664094e10 msgr2=0x7f66640a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:29:44.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:44.072 7f6668da4700 1 -- 172.21.2.112:0/2333983879 shutdown_connections 2021-02-10T11:29:44.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:44.072 7f6668da4700 1 -- 172.21.2.112:0/2333983879 wait complete. 2021-02-10T11:29:44.068 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000348666 2021-02-10T11:29:44.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:44.492 7fa534beb700 1 Processor -- start 2021-02-10T11:29:44.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:44.493 7fa534beb700 1 -- start start 2021-02-10T11:29:44.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:44.493 7fa534beb700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fa5301b0470 0x7fa5301b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:44.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:44.493 7fa534beb700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fa5300ef610 0x7fa5300efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:44.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:44.493 7fa534beb700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5300f0b50 con 0x7fa5300f0030 2021-02-10T11:29:44.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:44.493 7fa534beb700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5300e52e0 con 0x7fa5300ef610 2021-02-10T11:29:44.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:44.493 7fa534beb700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5300e5490 con 0x7fa5301b0470 2021-02-10T11:29:44.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:44.493 7fa52f05f700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fa5300f0030 0x7fa5300f10d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:46010/0 (socket says 172.21.2.112:46010) 2021-02-10T11:29:44.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:44.493 7fa52f05f700 1 -- 172.21.2.112:0/4034542659 learned_addr learned my addr 172.21.2.112:0/4034542659 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:29:44.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:44.493 7fa52e05d700 1 --2- 172.21.2.112:0/4034542659 >> v2:172.21.2.112:3300/0 conn(0x7fa5300ef610 0x7fa5300efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:44.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:44.494 7fa52e05d700 1 -- 172.21.2.112:0/4034542659 >> v2:172.21.2.120:3300/0 conn(0x7fa5301b0470 msgr2=0x7fa5301b0870 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:29:44.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:44.494 7fa52e05d700 1 --2- 172.21.2.112:0/4034542659 >> v2:172.21.2.120:3300/0 conn(0x7fa5301b0470 0x7fa5301b0870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:44.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:44.494 7fa52e05d700 1 -- 172.21.2.112:0/4034542659 >> v1:172.21.2.112:6789/0 conn(0x7fa5300f0030 legacy=0x7fa5300f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:44.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:44.494 7fa52e05d700 1 -- 172.21.2.112:0/4034542659 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5300f0430 con 0x7fa5300ef610 2021-02-10T11:29:44.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:44.494 7fa52e05d700 1 --2- 172.21.2.112:0/4034542659 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa5300ef610 0x7fa5300efa10 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 rx=0x7fa518006650 tx=0x7fa518009a80).ready entity=mon.0 client_cookie=4d22736bed06f6f0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:44.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:44.494 7fa52d85c700 1 -- 172.21.2.112:0/4034542659 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa51800c0c0 con 0x7fa5300ef610 2021-02-10T11:29:44.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:44.494 7fa52d85c700 1 -- 172.21.2.112:0/4034542659 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa51800c340 con 0x7fa5300ef610 2021-02-10T11:29:44.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:44.494 7fa52d85c700 1 -- 172.21.2.112:0/4034542659 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa51800c860 con 0x7fa5300ef610 2021-02-10T11:29:44.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:44.494 7fa534beb700 1 -- 172.21.2.112:0/4034542659 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa5300ef610 msgr2=0x7fa5300efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:44.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:44.494 7fa534beb700 1 --2- 172.21.2.112:0/4034542659 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa5300ef610 0x7fa5300efa10 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 rx=0x7fa518006650 tx=0x7fa518009a80).stop 2021-02-10T11:29:44.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:44.495 7fa534beb700 1 -- 172.21.2.112:0/4034542659 shutdown_connections 2021-02-10T11:29:44.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:44.495 7fa534beb700 1 --2- 172.21.2.112:0/4034542659 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa5300ef610 0x7fa5300efa10 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:44.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:44.495 7fa534beb700 1 --2- 172.21.2.112:0/4034542659 >> v2:172.21.2.120:3300/0 conn(0x7fa5301b0470 0x7fa5301b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:44.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:44.495 7fa534beb700 1 -- 172.21.2.112:0/4034542659 >> 172.21.2.112:0/4034542659 conn(0x7fa5300ce3e0 msgr2=0x7fa5300e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:29:44.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:44.495 7fa534beb700 1 -- 172.21.2.112:0/4034542659 shutdown_connections 2021-02-10T11:29:44.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:44.495 7fa534beb700 1 -- 172.21.2.112:0/4034542659 wait complete. 2021-02-10T11:29:44.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:44.509 7fa534beb700 1 Processor -- start 2021-02-10T11:29:44.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:44.509 7fa534beb700 1 -- start start 2021-02-10T11:29:44.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:44.509 7fa534beb700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa5300a9690 0x7fa5300a9a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:44.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:44.509 7fa534beb700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa5300aa0b0 0x7fa5300aa4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:44.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:44.509 7fa534beb700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa5300f1800 0x7fa5300f1c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:44.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:44.509 7fa534beb700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa530092350 con 0x7fa5300aa0b0 2021-02-10T11:29:44.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:44.509 7fa534beb700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5300a6540 con 0x7fa5300a9690 2021-02-10T11:29:44.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:44.509 7fa534beb700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5300a6720 con 0x7fa5300f1800 2021-02-10T11:29:44.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:44.510 7fa52e05d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa5300aa0b0 0x7fa5300aa4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:44.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:44.510 7fa52e05d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa5300aa0b0 0x7fa5300aa4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:46552/0 (socket says 172.21.2.112:46552) 2021-02-10T11:29:44.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:44.510 7fa52e05d700 1 -- 172.21.2.112:0/3690361520 learned_addr learned my addr 172.21.2.112:0/3690361520 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:29:44.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:44.510 7fa52e05d700 1 -- 172.21.2.112:0/3690361520 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa5300f1800 msgr2=0x7fa5300f1c00 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:29:44.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:44.510 7fa52e05d700 1 --2- 172.21.2.112:0/3690361520 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa5300f1800 0x7fa5300f1c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:44.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:44.510 7fa52e05d700 1 -- 172.21.2.112:0/3690361520 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa5300a9690 msgr2=0x7fa5300a9a90 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:29:44.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:44.510 7fa52e05d700 1 --2- 172.21.2.112:0/3690361520 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa5300a9690 0x7fa5300a9a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:44.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:44.510 7fa52e05d700 1 -- 172.21.2.112:0/3690361520 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5300a69e0 con 0x7fa5300aa0b0 2021-02-10T11:29:44.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:44.510 7fa52e05d700 1 --2- 172.21.2.112:0/3690361520 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa5300aa0b0 0x7fa5300aa4b0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 rx=0x7fa518009630 tx=0x7fa51800b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:44.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:44.510 7fa5277fe700 1 -- 172.21.2.112:0/3690361520 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa5180093a0 con 0x7fa5300aa0b0 2021-02-10T11:29:44.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:44.510 7fa5277fe700 1 -- 172.21.2.112:0/3690361520 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa51800bd70 con 0x7fa5300aa0b0 2021-02-10T11:29:44.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:44.510 7fa5277fe700 1 -- 172.21.2.112:0/3690361520 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa518005560 con 0x7fa5300aa0b0 2021-02-10T11:29:44.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:44.510 7fa534beb700 1 -- 172.21.2.112:0/3690361520 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5300a69e0 con 0x7fa5300aa0b0 2021-02-10T11:29:44.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:44.510 7fa534beb700 1 -- 172.21.2.112:0/3690361520 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5300a69e0 con 0x7fa5300aa0b0 2021-02-10T11:29:44.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:44.511 7fa5277fe700 1 -- 172.21.2.112:0/3690361520 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa518000c90 con 0x7fa5300aa0b0 2021-02-10T11:29:44.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:44.511 7fa5277fe700 1 --2- 172.21.2.112:0/3690361520 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa5140186d0 0x7fa51401abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:44.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:44.511 7fa5277fe700 1 -- 172.21.2.112:0/3690361520 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fa518032680 con 0x7fa5300aa0b0 2021-02-10T11:29:44.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:44.512 7fa52e85e700 1 --2- 172.21.2.112:0/3690361520 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa5140186d0 0x7fa51401abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:44.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:44.512 7fa52e85e700 1 --2- 172.21.2.112:0/3690361520 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa5140186d0 0x7fa51401abd0 crc :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:44.530 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:44.543 7fa534beb700 1 -- 172.21.2.112:0/3690361520 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5300edd10 con 0x7fa5300aa0b0 2021-02-10T11:29:44.533 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:44.545 7fa5277fe700 1 -- 172.21.2.112:0/3690361520 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa51800a000 con 0x7fa5300aa0b0 2021-02-10T11:29:44.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:44.676 7fa534beb700 1 -- 172.21.2.112:0/3690361520 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348666", "target": ["mgr", ""]}) v1 -- 0x7fa5300e52e0 con 0x7fa5140186d0 2021-02-10T11:29:44.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:44.680 7fa5277fe700 1 -- 172.21.2.112:0/3690361520 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7fa520006990 con 0x7fa5140186d0 2021-02-10T11:29:44.671 INFO:teuthology.orchestra.run.gibba012.stdout:/volumes/_nogroup/subvolume_0000000000348666/0fbc845e-1094-4c35-bbd0-edfec3f384bc 2021-02-10T11:29:44.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:44.686 7fa534beb700 1 -- 172.21.2.112:0/3690361520 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa5140186d0 msgr2=0x7fa51401abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:44.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:44.686 7fa534beb700 1 --2- 172.21.2.112:0/3690361520 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa5140186d0 0x7fa51401abd0 crc :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:44.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:44.686 7fa534beb700 1 -- 172.21.2.112:0/3690361520 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa5300aa0b0 msgr2=0x7fa5300aa4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:44.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:44.686 7fa534beb700 1 --2- 172.21.2.112:0/3690361520 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa5300aa0b0 0x7fa5300aa4b0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 rx=0x7fa518009630 tx=0x7fa51800b980).stop 2021-02-10T11:29:44.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:44.686 7fa534beb700 1 -- 172.21.2.112:0/3690361520 shutdown_connections 2021-02-10T11:29:44.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:44.686 7fa534beb700 1 --2- 172.21.2.112:0/3690361520 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa5140186d0 0x7fa51401abd0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:44.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:44.686 7fa534beb700 1 --2- 172.21.2.112:0/3690361520 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa5300f1800 0x7fa5300f1c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:44.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:44.686 7fa534beb700 1 --2- 172.21.2.112:0/3690361520 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa5300aa0b0 0x7fa5300aa4b0 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:44.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:44.686 7fa534beb700 1 --2- 172.21.2.112:0/3690361520 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa5300a9690 0x7fa5300a9a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:44.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:44.686 7fa534beb700 1 -- 172.21.2.112:0/3690361520 >> 172.21.2.112:0/3690361520 conn(0x7fa5300c40e0 msgr2=0x7fa530094eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:29:44.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:44.686 7fa534beb700 1 -- 172.21.2.112:0/3690361520 shutdown_connections 2021-02-10T11:29:44.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:44.686 7fa534beb700 1 -- 172.21.2.112:0/3690361520 wait complete. 2021-02-10T11:29:44.681 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000348666 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000348666/0fbc845e-1094-4c35-bbd0-edfec3f384bc 2021-02-10T11:29:44.682 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:44.682 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:44.682 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:45.017 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:29:45.018 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:45.018 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348666/0fbc845e-1094-4c35-bbd0-edfec3f384bc/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:45.398 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:29:45.398 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:29:45.398 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0257597 s, 40.7 MB/s 2021-02-10T11:29:45.399 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:45.399 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:45.399 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:45.735 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:29:45.735 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:45.735 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348666/0fbc845e-1094-4c35-bbd0-edfec3f384bc/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:46.109 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:29:46.109 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:29:46.109 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.025018 s, 41.9 MB/s 2021-02-10T11:29:46.110 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:46.110 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:46.111 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:46.444 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:29:46.445 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:46.445 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348666/0fbc845e-1094-4c35-bbd0-edfec3f384bc/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:46.820 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:29:46.820 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:29:46.820 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0245144 s, 42.8 MB/s 2021-02-10T11:29:46.821 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:46.821 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:46.821 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:47.155 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:29:47.499 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:47.499 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348666/0fbc845e-1094-4c35-bbd0-edfec3f384bc/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:47.875 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:29:47.875 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:29:47.875 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0239937 s, 43.7 MB/s 2021-02-10T11:29:47.876 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:47.876 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:47.876 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:48.209 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:29:48.210 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:48.210 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348666/0fbc845e-1094-4c35-bbd0-edfec3f384bc/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:48.584 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:29:48.584 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:29:48.584 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0248827 s, 42.1 MB/s 2021-02-10T11:29:48.585 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:48.585 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:48.585 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:48.919 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:29:48.920 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:48.920 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348666/0fbc845e-1094-4c35-bbd0-edfec3f384bc/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:49.294 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:29:49.294 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:29:49.294 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0237094 s, 44.2 MB/s 2021-02-10T11:29:49.295 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:49.295 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:49.295 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:49.629 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:29:49.629 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:49.630 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348666/0fbc845e-1094-4c35-bbd0-edfec3f384bc/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:50.001 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:29:50.001 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:29:50.001 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0243154 s, 43.1 MB/s 2021-02-10T11:29:50.002 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:50.002 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:50.002 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:50.337 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:29:50.338 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:50.338 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348666/0fbc845e-1094-4c35-bbd0-edfec3f384bc/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:50.710 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:29:50.710 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:29:50.710 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0225965 s, 46.4 MB/s 2021-02-10T11:29:50.711 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:50.712 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:50.712 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:51.047 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:29:51.047 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:51.047 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348666/0fbc845e-1094-4c35-bbd0-edfec3f384bc/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:51.425 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:29:51.425 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:29:51.425 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.026211 s, 40.0 MB/s 2021-02-10T11:29:51.426 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:51.426 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:51.426 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:51.761 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:29:51.761 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:51.762 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348666/0fbc845e-1094-4c35-bbd0-edfec3f384bc/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:52.134 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:29:52.135 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:29:52.135 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0225544 s, 46.5 MB/s 2021-02-10T11:29:52.137 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000348667 2021-02-10T11:29:52.565 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:52.561 7fefd49b2700 1 Processor -- start 2021-02-10T11:29:52.565 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:52.561 7fefd49b2700 1 -- start start 2021-02-10T11:29:52.565 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:52.561 7fefd49b2700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fefd01b0470 0x7fefd01b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:52.565 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:52.561 7fefd49b2700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fefd00ef610 0x7fefd00efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:52.565 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:52.562 7fefd49b2700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefd00f0b50 con 0x7fefd00f0030 2021-02-10T11:29:52.565 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:52.562 7fefd49b2700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fefd00e52e0 con 0x7fefd01b0470 2021-02-10T11:29:52.566 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:52.562 7fefd49b2700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fefd00e5490 con 0x7fefd00ef610 2021-02-10T11:29:52.566 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:52.562 7fefce61c700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fefd01b0470 0x7fefd01b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:52.566 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:52.562 7fefce61c700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fefd01b0470 0x7fefd01b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:46558/0 (socket says 172.21.2.112:46558) 2021-02-10T11:29:52.566 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:52.562 7fefce61c700 1 -- 172.21.2.112:0/4049734008 learned_addr learned my addr 172.21.2.112:0/4049734008 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:29:52.566 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:52.562 7fefcee1d700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fefd00f0030 0x7fefd00f10d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:46020/0 (socket says 172.21.2.112:46020) 2021-02-10T11:29:52.566 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:52.562 7fefce61c700 1 -- 172.21.2.112:0/4049734008 >> v2:172.21.2.120:3300/0 conn(0x7fefd00ef610 msgr2=0x7fefd00efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:29:52.566 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:52.562 7fefce61c700 1 --2- 172.21.2.112:0/4049734008 >> v2:172.21.2.120:3300/0 conn(0x7fefd00ef610 0x7fefd00efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:52.566 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:52.562 7fefce61c700 1 -- 172.21.2.112:0/4049734008 >> v1:172.21.2.112:6789/0 conn(0x7fefd00f0030 legacy=0x7fefd00f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:52.566 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:52.562 7fefce61c700 1 -- 172.21.2.112:0/4049734008 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefd00f0430 con 0x7fefd01b0470 2021-02-10T11:29:52.566 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:52.562 7fefce61c700 1 --2- 172.21.2.112:0/4049734008 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fefd01b0470 0x7fefd01b0870 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 rx=0x7fefc0006700 tx=0x7fefc0009b80).ready entity=mon.0 client_cookie=d307d0d58023072b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:52.566 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:52.563 7fefcd61a700 1 -- 172.21.2.112:0/4049734008 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fefc000c0c0 con 0x7fefd01b0470 2021-02-10T11:29:52.566 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:52.563 7fefcd61a700 1 -- 172.21.2.112:0/4049734008 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefc000c440 con 0x7fefd01b0470 2021-02-10T11:29:52.566 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:52.563 7fefcd61a700 1 -- 172.21.2.112:0/4049734008 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fefc000c920 con 0x7fefd01b0470 2021-02-10T11:29:52.567 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:52.563 7fefd49b2700 1 -- 172.21.2.112:0/4049734008 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fefd01b0470 msgr2=0x7fefd01b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:52.567 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:52.563 7fefd49b2700 1 --2- 172.21.2.112:0/4049734008 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fefd01b0470 0x7fefd01b0870 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 rx=0x7fefc0006700 tx=0x7fefc0009b80).stop 2021-02-10T11:29:52.567 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:52.563 7fefd49b2700 1 -- 172.21.2.112:0/4049734008 shutdown_connections 2021-02-10T11:29:52.567 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:52.563 7fefd49b2700 1 --2- 172.21.2.112:0/4049734008 >> v2:172.21.2.120:3300/0 conn(0x7fefd00ef610 0x7fefd00efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:52.567 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:52.563 7fefd49b2700 1 --2- 172.21.2.112:0/4049734008 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fefd01b0470 0x7fefd01b0870 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:52.567 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:52.563 7fefd49b2700 1 -- 172.21.2.112:0/4049734008 >> 172.21.2.112:0/4049734008 conn(0x7fefd00ce3e0 msgr2=0x7fefd00e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:29:52.567 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:52.563 7fefd49b2700 1 -- 172.21.2.112:0/4049734008 shutdown_connections 2021-02-10T11:29:52.567 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:52.564 7fefd49b2700 1 -- 172.21.2.112:0/4049734008 wait complete. 2021-02-10T11:29:52.567 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:52.578 7fefd49b2700 1 Processor -- start 2021-02-10T11:29:52.575 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:52.578 7fefd49b2700 1 -- start start 2021-02-10T11:29:52.575 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:52.578 7fefd49b2700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fefd00b20b0 0x7fefd00ac970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:52.575 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:52.578 7fefd49b2700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fefd00b24b0 0x7fefd0191dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:52.575 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:52.578 7fefd49b2700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fefd00acf90 0x7fefd01923e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:52.575 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:52.578 7fefd49b2700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefd00b1b30 con 0x7fefd00b24b0 2021-02-10T11:29:52.575 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:52.578 7fefd49b2700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefd01a04e0 con 0x7fefd00b20b0 2021-02-10T11:29:52.575 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:52.578 7fefd49b2700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefd01a06c0 con 0x7fefd00acf90 2021-02-10T11:29:52.576 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:52.578 7fefcde1b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fefd00b24b0 0x7fefd0191dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:52.576 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:52.578 7fefcde1b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fefd00b24b0 0x7fefd0191dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:46562/0 (socket says 172.21.2.112:46562) 2021-02-10T11:29:52.576 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:52.578 7fefcde1b700 1 -- 172.21.2.112:0/1877288750 learned_addr learned my addr 172.21.2.112:0/1877288750 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:29:52.576 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:52.579 7fefcde1b700 1 -- 172.21.2.112:0/1877288750 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fefd00acf90 msgr2=0x7fefd01923e0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:29:52.576 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:52.579 7fefcde1b700 1 --2- 172.21.2.112:0/1877288750 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fefd00acf90 0x7fefd01923e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:52.577 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:52.579 7fefcde1b700 1 -- 172.21.2.112:0/1877288750 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fefd00b20b0 msgr2=0x7fefd00ac970 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:29:52.577 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:52.579 7fefcde1b700 1 --2- 172.21.2.112:0/1877288750 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fefd00b20b0 0x7fefd00ac970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:52.577 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:52.579 7fefcde1b700 1 -- 172.21.2.112:0/1877288750 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefd0192be0 con 0x7fefd00b24b0 2021-02-10T11:29:52.577 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:52.579 7fefcde1b700 1 --2- 172.21.2.112:0/1877288750 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fefd00b24b0 0x7fefd0191dc0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 rx=0x7fefc4006f00 tx=0x7fefc400b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:52.577 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:52.579 7fefbeffd700 1 -- 172.21.2.112:0/1877288750 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fefc400bd30 con 0x7fefd00b24b0 2021-02-10T11:29:52.577 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:52.579 7fefbeffd700 1 -- 172.21.2.112:0/1877288750 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefc4000c90 con 0x7fefd00b24b0 2021-02-10T11:29:52.577 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:52.579 7fefd49b2700 1 -- 172.21.2.112:0/1877288750 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefd0192be0 con 0x7fefd00b24b0 2021-02-10T11:29:52.587 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:52.579 7fefd49b2700 1 -- 172.21.2.112:0/1877288750 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefd0193170 con 0x7fefd00b24b0 2021-02-10T11:29:52.587 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:52.579 7fefbeffd700 1 -- 172.21.2.112:0/1877288750 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fefc40022a0 con 0x7fefd00b24b0 2021-02-10T11:29:52.587 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:52.580 7fefbeffd700 1 -- 172.21.2.112:0/1877288750 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fefc4005310 con 0x7fefd00b24b0 2021-02-10T11:29:52.587 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:52.580 7fefbeffd700 1 --2- 172.21.2.112:0/1877288750 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fefb4018730 0x7fefb401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:52.587 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:52.580 7fefbeffd700 1 -- 172.21.2.112:0/1877288750 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fefc4032770 con 0x7fefd00b24b0 2021-02-10T11:29:52.587 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:52.580 7fefce61c700 1 --2- 172.21.2.112:0/1877288750 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fefb4018730 0x7fefb401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:52.587 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:52.581 7fefce61c700 1 --2- 172.21.2.112:0/1877288750 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fefb4018730 0x7fefb401ac10 crc :-1 s=READY pgs=385 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:52.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:52.611 7fefd49b2700 1 -- 172.21.2.112:0/1877288750 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefd0105af0 con 0x7fefd00b24b0 2021-02-10T11:29:52.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:52.614 7fefbeffd700 1 -- 172.21.2.112:0/1877288750 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fefc4000940 con 0x7fefd00b24b0 2021-02-10T11:29:52.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:52.740 7fefd49b2700 1 -- 172.21.2.112:0/1877288750 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348667", "target": ["mgr", ""]}) v1 -- 0x7fefd01a06c0 con 0x7fefb4018730 2021-02-10T11:29:52.754 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:52.766 7fefbeffd700 1 -- 172.21.2.112:0/1877288750 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fefc0006500 con 0x7fefb4018730 2021-02-10T11:29:52.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:52.773 7fefd49b2700 1 -- 172.21.2.112:0/1877288750 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fefb4018730 msgr2=0x7fefb401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:52.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:52.773 7fefd49b2700 1 --2- 172.21.2.112:0/1877288750 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fefb4018730 0x7fefb401ac10 crc :-1 s=READY pgs=385 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:52.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:52.773 7fefd49b2700 1 -- 172.21.2.112:0/1877288750 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fefd00b24b0 msgr2=0x7fefd0191dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:52.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:52.773 7fefd49b2700 1 --2- 172.21.2.112:0/1877288750 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fefd00b24b0 0x7fefd0191dc0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 rx=0x7fefc4006f00 tx=0x7fefc400b940).stop 2021-02-10T11:29:52.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:52.774 7fefd49b2700 1 -- 172.21.2.112:0/1877288750 shutdown_connections 2021-02-10T11:29:52.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:52.774 7fefd49b2700 1 --2- 172.21.2.112:0/1877288750 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fefb4018730 0x7fefb401ac10 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:52.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:52.774 7fefd49b2700 1 --2- 172.21.2.112:0/1877288750 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fefd00acf90 0x7fefd01923e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:52.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:52.774 7fefd49b2700 1 --2- 172.21.2.112:0/1877288750 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fefd00b24b0 0x7fefd0191dc0 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:52.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:52.774 7fefd49b2700 1 --2- 172.21.2.112:0/1877288750 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fefd00b20b0 0x7fefd00ac970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:52.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:52.774 7fefd49b2700 1 -- 172.21.2.112:0/1877288750 >> 172.21.2.112:0/1877288750 conn(0x7fefd00bf4e0 msgr2=0x7fefd00be3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:29:52.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:52.774 7fefd49b2700 1 -- 172.21.2.112:0/1877288750 shutdown_connections 2021-02-10T11:29:52.766 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:52.774 7fefd49b2700 1 -- 172.21.2.112:0/1877288750 wait complete. 2021-02-10T11:29:52.769 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000348667 2021-02-10T11:29:53.207 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:53.203 7f7fe8e8c700 1 Processor -- start 2021-02-10T11:29:53.207 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:53.203 7f7fe8e8c700 1 -- start start 2021-02-10T11:29:53.208 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:53.203 7f7fe8e8c700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f7fe41b0470 0x7f7fe41b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:53.208 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:53.203 7f7fe8e8c700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f7fe40ef610 0x7f7fe40efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:53.208 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:53.203 7f7fe8e8c700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7fe40f0030 0x7f7fe40f0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:53.208 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:53.203 7f7fe8e8c700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fe40e5ea0 con 0x7f7fe40ef610 2021-02-10T11:29:53.208 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:53.203 7f7fe8e8c700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fe40f0b50 con 0x7f7fe41b0470 2021-02-10T11:29:53.208 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:53.203 7f7fe8e8c700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fe40e52e0 con 0x7f7fe40f0030 2021-02-10T11:29:53.209 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:53.204 7f7fe2312700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f7fe40ef610 0x7f7fe40efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:53.209 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:53.204 7f7fe2312700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f7fe40ef610 0x7f7fe40efa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:46572/0 (socket says 172.21.2.112:46572) 2021-02-10T11:29:53.209 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:53.204 7f7fe2312700 1 -- 172.21.2.112:0/3375814209 learned_addr learned my addr 172.21.2.112:0/3375814209 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:29:53.209 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:53.204 7f7fe2312700 1 -- 172.21.2.112:0/3375814209 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7fe40f0030 msgr2=0x7f7fe40f0430 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:29:53.209 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:53.204 7f7fe2312700 1 --2- 172.21.2.112:0/3375814209 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7fe40f0030 0x7f7fe40f0430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:53.209 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:53.204 7f7fe2312700 1 -- 172.21.2.112:0/3375814209 >> v2:172.21.2.120:3300/0 conn(0x7f7fe41b0470 msgr2=0x7f7fe41b0870 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:29:53.210 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:53.204 7f7fe2312700 1 --2- 172.21.2.112:0/3375814209 >> v2:172.21.2.120:3300/0 conn(0x7f7fe41b0470 0x7f7fe41b0870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:53.210 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:53.204 7f7fe2312700 1 -- 172.21.2.112:0/3375814209 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fe40f10d0 con 0x7f7fe40ef610 2021-02-10T11:29:53.210 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:53.204 7f7fe2312700 1 --2- 172.21.2.112:0/3375814209 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7fe40ef610 0x7f7fe40efa10 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 rx=0x7f7fcc006700 tx=0x7f7fcc009b80).ready entity=mon.0 client_cookie=72093865ae87fb0e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:53.210 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:53.204 7f7fe1b11700 1 -- 172.21.2.112:0/3375814209 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7fcc00c0c0 con 0x7f7fe40ef610 2021-02-10T11:29:53.210 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:53.204 7f7fe1b11700 1 -- 172.21.2.112:0/3375814209 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fcc00c340 con 0x7f7fe40ef610 2021-02-10T11:29:53.211 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:53.204 7f7fe1b11700 1 -- 172.21.2.112:0/3375814209 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7fcc00c860 con 0x7f7fe40ef610 2021-02-10T11:29:53.211 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:53.204 7f7fe8e8c700 1 -- 172.21.2.112:0/3375814209 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7fe40ef610 msgr2=0x7f7fe40efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:53.211 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:53.204 7f7fe8e8c700 1 --2- 172.21.2.112:0/3375814209 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7fe40ef610 0x7f7fe40efa10 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 rx=0x7f7fcc006700 tx=0x7f7fcc009b80).stop 2021-02-10T11:29:53.211 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:53.205 7f7fe8e8c700 1 -- 172.21.2.112:0/3375814209 shutdown_connections 2021-02-10T11:29:53.211 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:53.205 7f7fe8e8c700 1 --2- 172.21.2.112:0/3375814209 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7fe40f0030 0x7f7fe40f0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:53.211 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:53.205 7f7fe8e8c700 1 --2- 172.21.2.112:0/3375814209 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7fe40ef610 0x7f7fe40efa10 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:53.211 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:53.205 7f7fe8e8c700 1 --2- 172.21.2.112:0/3375814209 >> v2:172.21.2.120:3300/0 conn(0x7f7fe41b0470 0x7f7fe41b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:53.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:53.205 7f7fe8e8c700 1 -- 172.21.2.112:0/3375814209 >> 172.21.2.112:0/3375814209 conn(0x7f7fe40ce3e0 msgr2=0x7f7fe40e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:29:53.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:53.205 7f7fe8e8c700 1 -- 172.21.2.112:0/3375814209 shutdown_connections 2021-02-10T11:29:53.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:53.205 7f7fe8e8c700 1 -- 172.21.2.112:0/3375814209 wait complete. 2021-02-10T11:29:53.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:53.220 7f7fe8e8c700 1 Processor -- start 2021-02-10T11:29:53.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:53.220 7f7fe8e8c700 1 -- start start 2021-02-10T11:29:53.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:53.220 7f7fe8e8c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7fe40b20b0 0x7f7fe40ac970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:53.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:53.220 7f7fe8e8c700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7fe40b24b0 0x7f7fe4191dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:53.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:53.220 7f7fe8e8c700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7fe40acf90 0x7f7fe41923e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:53.216 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:53.220 7f7fe8e8c700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fe40b1b30 con 0x7f7fe40b20b0 2021-02-10T11:29:53.216 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:53.220 7f7fe8e8c700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fe41a04e0 con 0x7f7fe40acf90 2021-02-10T11:29:53.216 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:53.220 7f7fe8e8c700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fe41a06c0 con 0x7f7fe40b24b0 2021-02-10T11:29:53.216 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:53.220 7f7fe2b13700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7fe40b20b0 0x7f7fe40ac970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:53.216 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:53.220 7f7fe2b13700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7fe40b20b0 0x7f7fe40ac970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:46576/0 (socket says 172.21.2.112:46576) 2021-02-10T11:29:53.216 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:53.220 7f7fe2b13700 1 -- 172.21.2.112:0/3710715183 learned_addr learned my addr 172.21.2.112:0/3710715183 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:29:53.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:53.220 7f7fe2b13700 1 -- 172.21.2.112:0/3710715183 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7fe40b24b0 msgr2=0x7f7fe4191dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:53.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:53.220 7f7fe2b13700 1 --2- 172.21.2.112:0/3710715183 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7fe40b24b0 0x7f7fe4191dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:53.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:53.220 7f7fe2b13700 1 -- 172.21.2.112:0/3710715183 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7fe40acf90 msgr2=0x7f7fe41923e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:53.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:53.220 7f7fe2b13700 1 --2- 172.21.2.112:0/3710715183 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7fe40acf90 0x7f7fe41923e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:53.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:53.221 7f7fe2b13700 1 -- 172.21.2.112:0/3710715183 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fe4192be0 con 0x7f7fe40b20b0 2021-02-10T11:29:53.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:53.221 7f7fe2b13700 1 --2- 172.21.2.112:0/3710715183 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7fe40b20b0 0x7f7fe40ac970 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 rx=0x7f7fd400b5c0 tx=0x7f7fd400b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:53.218 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:53.221 7f7fdb7fe700 1 -- 172.21.2.112:0/3710715183 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7fd400bd50 con 0x7f7fe40b20b0 2021-02-10T11:29:53.218 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:53.221 7f7fdb7fe700 1 -- 172.21.2.112:0/3710715183 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fd4006030 con 0x7f7fe40b20b0 2021-02-10T11:29:53.218 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:53.221 7f7fdb7fe700 1 -- 172.21.2.112:0/3710715183 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7fd40054d0 con 0x7f7fe40b20b0 2021-02-10T11:29:53.218 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:53.221 7f7fe8e8c700 1 -- 172.21.2.112:0/3710715183 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fe4192be0 con 0x7f7fe40b20b0 2021-02-10T11:29:53.218 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:53.221 7f7fe8e8c700 1 -- 172.21.2.112:0/3710715183 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fe4192be0 con 0x7f7fe40b20b0 2021-02-10T11:29:53.219 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:53.222 7f7fdb7fe700 1 -- 172.21.2.112:0/3710715183 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7fd4006210 con 0x7f7fe40b20b0 2021-02-10T11:29:53.219 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:53.222 7f7fdb7fe700 1 --2- 172.21.2.112:0/3710715183 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7fc80186c0 0x7f7fc801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:53.219 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:53.222 7f7fdb7fe700 1 -- 172.21.2.112:0/3710715183 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f7fd4032580 con 0x7f7fe40b20b0 2021-02-10T11:29:53.219 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:53.222 7f7fe2312700 1 --2- 172.21.2.112:0/3710715183 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7fc80186c0 0x7f7fc801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:53.219 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:53.223 7f7fe2312700 1 --2- 172.21.2.112:0/3710715183 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7fc80186c0 0x7f7fc801abc0 crc :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:53.241 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:53.253 7f7fe8e8c700 1 -- 172.21.2.112:0/3710715183 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fe4105af0 con 0x7f7fe40b20b0 2021-02-10T11:29:53.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:53.256 7f7fdb7fe700 1 -- 172.21.2.112:0/3710715183 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7fd402cf00 con 0x7f7fe40b20b0 2021-02-10T11:29:53.372 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:53.384 7f7fe8e8c700 1 -- 172.21.2.112:0/3710715183 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348667", "target": ["mgr", ""]}) v1 -- 0x7f7fe40f0b50 con 0x7f7fc80186c0 2021-02-10T11:29:53.375 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:53.388 7f7fdb7fe700 1 -- 172.21.2.112:0/3710715183 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f7fcc009550 con 0x7f7fc80186c0 2021-02-10T11:29:53.379 INFO:teuthology.orchestra.run.gibba012.stdout:/volumes/_nogroup/subvolume_0000000000348667/1a461d19-11f5-4c98-a32e-b8a9c31fe06a 2021-02-10T11:29:53.381 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:53.393 7f7fe8e8c700 1 -- 172.21.2.112:0/3710715183 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7fc80186c0 msgr2=0x7f7fc801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:53.381 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:53.393 7f7fe8e8c700 1 --2- 172.21.2.112:0/3710715183 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7fc80186c0 0x7f7fc801abc0 crc :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:53.381 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:53.393 7f7fe8e8c700 1 -- 172.21.2.112:0/3710715183 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7fe40b20b0 msgr2=0x7f7fe40ac970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:53.381 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:53.393 7f7fe8e8c700 1 --2- 172.21.2.112:0/3710715183 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7fe40b20b0 0x7f7fe40ac970 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 rx=0x7f7fd400b5c0 tx=0x7f7fd400b960).stop 2021-02-10T11:29:53.382 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:53.393 7f7fe8e8c700 1 -- 172.21.2.112:0/3710715183 shutdown_connections 2021-02-10T11:29:53.382 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:53.393 7f7fe8e8c700 1 --2- 172.21.2.112:0/3710715183 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7fc80186c0 0x7f7fc801abc0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:53.382 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:53.393 7f7fe8e8c700 1 --2- 172.21.2.112:0/3710715183 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7fe40acf90 0x7f7fe41923e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:53.382 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:53.393 7f7fe8e8c700 1 --2- 172.21.2.112:0/3710715183 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7fe40b24b0 0x7f7fe4191dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:53.382 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:53.393 7f7fe8e8c700 1 --2- 172.21.2.112:0/3710715183 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7fe40b20b0 0x7f7fe40ac970 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:53.382 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:53.393 7f7fe8e8c700 1 -- 172.21.2.112:0/3710715183 >> 172.21.2.112:0/3710715183 conn(0x7f7fe40bf4e0 msgr2=0x7f7fe40be3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:29:53.383 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:53.393 7f7fe8e8c700 1 -- 172.21.2.112:0/3710715183 shutdown_connections 2021-02-10T11:29:53.383 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:29:53.393 7f7fe8e8c700 1 -- 172.21.2.112:0/3710715183 wait complete. 2021-02-10T11:29:53.388 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000348667 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000348667/1a461d19-11f5-4c98-a32e-b8a9c31fe06a 2021-02-10T11:29:53.389 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:53.389 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:53.389 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:53.725 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:29:53.726 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:53.726 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348667/1a461d19-11f5-4c98-a32e-b8a9c31fe06a/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:54.108 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:29:54.108 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:29:54.108 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.024454 s, 42.9 MB/s 2021-02-10T11:29:54.109 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:54.109 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:54.109 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:54.450 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:29:54.451 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:54.451 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348667/1a461d19-11f5-4c98-a32e-b8a9c31fe06a/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:54.824 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:29:54.824 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:29:54.824 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0240862 s, 43.5 MB/s 2021-02-10T11:29:54.825 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:54.825 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:54.825 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:55.160 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:29:55.161 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:55.161 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348667/1a461d19-11f5-4c98-a32e-b8a9c31fe06a/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:55.537 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:29:55.537 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:29:55.537 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.02553 s, 41.1 MB/s 2021-02-10T11:29:55.538 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:55.539 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:55.539 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:55.873 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:29:55.874 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:55.874 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348667/1a461d19-11f5-4c98-a32e-b8a9c31fe06a/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:56.249 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:29:56.249 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:29:56.249 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0253385 s, 41.4 MB/s 2021-02-10T11:29:56.250 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:56.251 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:56.251 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:56.584 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:29:56.585 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:56.585 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348667/1a461d19-11f5-4c98-a32e-b8a9c31fe06a/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:56.960 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:29:56.960 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:29:56.960 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.023867 s, 43.9 MB/s 2021-02-10T11:29:56.961 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:56.961 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:56.961 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:57.294 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:29:57.295 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:57.295 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348667/1a461d19-11f5-4c98-a32e-b8a9c31fe06a/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:57.670 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:29:57.670 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:29:57.670 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0250479 s, 41.9 MB/s 2021-02-10T11:29:57.671 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:57.672 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:57.672 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:58.004 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:29:58.005 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:58.005 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348667/1a461d19-11f5-4c98-a32e-b8a9c31fe06a/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:58.377 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:29:58.377 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:29:58.377 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0235101 s, 44.6 MB/s 2021-02-10T11:29:58.378 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:58.378 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:58.378 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:58.711 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:29:58.711 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:58.712 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348667/1a461d19-11f5-4c98-a32e-b8a9c31fe06a/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:59.088 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:29:59.088 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:29:59.088 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0243873 s, 43.0 MB/s 2021-02-10T11:29:59.089 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:59.090 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:59.090 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:59.423 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:29:59.423 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:59.424 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348667/1a461d19-11f5-4c98-a32e-b8a9c31fe06a/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:59.794 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:29:59.794 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:29:59.794 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.023627 s, 44.4 MB/s 2021-02-10T11:29:59.795 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:59.795 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:59.796 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:30:00.129 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:30:00.129 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:30:00.130 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000348667/1a461d19-11f5-4c98-a32e-b8a9c31fe06a/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:30:00.506 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:30:00.506 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:30:00.506 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.026336 s, 39.8 MB/s 2021-02-10T11:30:00.508 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:30:00.508 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:30:00.508 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:30:00.842 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:30:00.842 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:30:00.842 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba012.front.sepia.ceph.com... 2021-02-10T11:30:00.843 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:30:00.843 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:30:01.247 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba012.stderr:ceph-fuse[45043]: fuse finished with error 0 and tester_r 0 2021-02-10T11:30:01.249 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:30:01.249 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:30:01.249 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:30:01.584 INFO:teuthology.orchestra.run.gibba012.stdout:ext2/ext3 2021-02-10T11:30:01.584 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:30:01.584 INFO:teuthology.orchestra.run:waiting for 900 2021-02-10T11:30:07.687 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:30:07.687 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:30:08.026 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000348568 2021-02-10T11:30:08.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:08.454 7fca0e1af700 1 Processor -- start 2021-02-10T11:30:08.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:08.454 7fca0e1af700 1 -- start start 2021-02-10T11:30:08.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:08.455 7fca0e1af700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fca081b0870 0x7fca080f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:08.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:08.455 7fca0e1af700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fca080efe00 0x7fca080f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:08.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:08.455 7fca0e1af700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca081b0c70 con 0x7fca081b0470 2021-02-10T11:30:08.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:08.455 7fca0e1af700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca080f0b50 con 0x7fca081b0870 2021-02-10T11:30:08.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:08.455 7fca0e1af700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca080f0820 con 0x7fca080efe00 2021-02-10T11:30:08.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:08.455 7fca06ffd700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fca081b0870 0x7fca080f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:08.460 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:08.455 7fca077fe700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fca081b0470 0x7fca080ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:46046/0 (socket says 172.21.2.112:46046) 2021-02-10T11:30:08.460 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:08.455 7fca077fe700 1 -- 172.21.2.112:0/3359384691 learned_addr learned my addr 172.21.2.112:0/3359384691 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:30:08.460 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:08.456 7fca06ffd700 1 -- 172.21.2.112:0/3359384691 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fca080efe00 msgr2=0x7fca080f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:08.460 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:08.456 7fca06ffd700 1 --2- 172.21.2.112:0/3359384691 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fca080efe00 0x7fca080f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:08.460 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:08.456 7fca06ffd700 1 -- 172.21.2.112:0/3359384691 >> v1:172.21.2.112:6789/0 conn(0x7fca081b0470 legacy=0x7fca080ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:08.460 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:08.456 7fca06ffd700 1 -- 172.21.2.112:0/3359384691 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca080e52e0 con 0x7fca081b0870 2021-02-10T11:30:08.460 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:08.456 7fca06ffd700 1 --2- 172.21.2.112:0/3359384691 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fca081b0870 0x7fca080f20e0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 rx=0x7fc9f8006650 tx=0x7fc9f8009a00).ready entity=mon.0 client_cookie=1cee8a506dbb8c19 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:08.460 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:08.456 7fca067fc700 1 -- 172.21.2.112:0/3359384691 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc9f8009df0 con 0x7fca081b0870 2021-02-10T11:30:08.460 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:08.456 7fca067fc700 1 -- 172.21.2.112:0/3359384691 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9f80009b0 con 0x7fca081b0870 2021-02-10T11:30:08.460 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:08.456 7fca067fc700 1 -- 172.21.2.112:0/3359384691 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc9f800c2b0 con 0x7fca081b0870 2021-02-10T11:30:08.460 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:08.456 7fca0e1af700 1 -- 172.21.2.112:0/3359384691 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fca081b0870 msgr2=0x7fca080f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:08.460 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:08.456 7fca0e1af700 1 --2- 172.21.2.112:0/3359384691 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fca081b0870 0x7fca080f20e0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 rx=0x7fc9f8006650 tx=0x7fc9f8009a00).stop 2021-02-10T11:30:08.460 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:08.457 7fca0e1af700 1 -- 172.21.2.112:0/3359384691 shutdown_connections 2021-02-10T11:30:08.460 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:08.457 7fca0e1af700 1 --2- 172.21.2.112:0/3359384691 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fca080efe00 0x7fca080f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:08.461 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:08.457 7fca0e1af700 1 --2- 172.21.2.112:0/3359384691 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fca081b0870 0x7fca080f20e0 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:08.461 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:08.457 7fca0e1af700 1 -- 172.21.2.112:0/3359384691 >> 172.21.2.112:0/3359384691 conn(0x7fca080ce3e0 msgr2=0x7fca080e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:08.461 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:08.457 7fca0e1af700 1 -- 172.21.2.112:0/3359384691 shutdown_connections 2021-02-10T11:30:08.461 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:08.457 7fca0e1af700 1 -- 172.21.2.112:0/3359384691 wait complete. 2021-02-10T11:30:08.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:08.472 7fca0e1af700 1 Processor -- start 2021-02-10T11:30:08.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:08.472 7fca0e1af700 1 -- start start 2021-02-10T11:30:08.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:08.472 7fca0e1af700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fca080b2840 0x7fca081a4300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:08.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:08.472 7fca0e1af700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fca080b30d0 0x7fca080b34d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:08.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:08.472 7fca0e1af700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fca080b3af0 0x7fca080f0580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:08.463 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:08.472 7fca0e1af700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca080b2c40 con 0x7fca080b3af0 2021-02-10T11:30:08.463 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:08.472 7fca0e1af700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca081a4920 con 0x7fca080b2840 2021-02-10T11:30:08.463 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:08.472 7fca0e1af700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca080f0ba0 con 0x7fca080b30d0 2021-02-10T11:30:08.463 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:08.473 7fca07fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fca080b3af0 0x7fca080f0580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:08.463 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:08.473 7fca07fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fca080b3af0 0x7fca080f0580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:46594/0 (socket says 172.21.2.112:46594) 2021-02-10T11:30:08.463 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:08.473 7fca07fff700 1 -- 172.21.2.112:0/567072959 learned_addr learned my addr 172.21.2.112:0/567072959 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:30:08.463 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:08.473 7fca07fff700 1 -- 172.21.2.112:0/567072959 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fca080b30d0 msgr2=0x7fca080b34d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:08.463 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:08.473 7fca07fff700 1 --2- 172.21.2.112:0/567072959 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fca080b30d0 0x7fca080b34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:08.463 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:08.473 7fca07fff700 1 -- 172.21.2.112:0/567072959 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fca080b2840 msgr2=0x7fca081a4300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:08.463 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:08.473 7fca07fff700 1 --2- 172.21.2.112:0/567072959 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fca080b2840 0x7fca081a4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:08.463 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:08.473 7fca07fff700 1 -- 172.21.2.112:0/567072959 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca080f0dc0 con 0x7fca080b3af0 2021-02-10T11:30:08.463 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:08.473 7fca07fff700 1 --2- 172.21.2.112:0/567072959 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fca080b3af0 0x7fca080f0580 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 rx=0x7fca00009980 tx=0x7fca0000bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:08.463 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:08.473 7fc9fffff700 1 -- 172.21.2.112:0/567072959 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fca00005af0 con 0x7fca080b3af0 2021-02-10T11:30:08.464 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:08.474 7fc9fffff700 1 -- 172.21.2.112:0/567072959 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca00005df0 con 0x7fca080b3af0 2021-02-10T11:30:08.464 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:08.474 7fc9fffff700 1 -- 172.21.2.112:0/567072959 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fca000062b0 con 0x7fca080b3af0 2021-02-10T11:30:08.464 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:08.474 7fca0e1af700 1 -- 172.21.2.112:0/567072959 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca080f0dc0 con 0x7fca080b3af0 2021-02-10T11:30:08.464 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:08.474 7fca0e1af700 1 -- 172.21.2.112:0/567072959 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca080f0dc0 con 0x7fca080b3af0 2021-02-10T11:30:08.464 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:08.474 7fc9fffff700 1 -- 172.21.2.112:0/567072959 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fca00005af0 con 0x7fca080b3af0 2021-02-10T11:30:08.464 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:08.474 7fc9fffff700 1 --2- 172.21.2.112:0/567072959 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc9ec0186c0 0x7fc9ec01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:08.464 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:08.474 7fc9fffff700 1 -- 172.21.2.112:0/567072959 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fca000328e0 con 0x7fca080b3af0 2021-02-10T11:30:08.464 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:08.475 7fca077fe700 1 --2- 172.21.2.112:0/567072959 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc9ec0186c0 0x7fc9ec01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:08.464 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:08.475 7fca077fe700 1 --2- 172.21.2.112:0/567072959 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc9ec0186c0 0x7fc9ec01abc0 crc :-1 s=READY pgs=387 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:08.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:08.505 7fca0e1af700 1 -- 172.21.2.112:0/567072959 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca080f0050 con 0x7fca080b3af0 2021-02-10T11:30:08.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:08.507 7fc9fffff700 1 -- 172.21.2.112:0/567072959 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fca00008ef0 con 0x7fca080b3af0 2021-02-10T11:30:08.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:08.635 7fca0e1af700 1 -- 172.21.2.112:0/567072959 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348568", "target": ["mgr", ""]}) v1 -- 0x7fca080f0050 con 0x7fc9ec0186c0 2021-02-10T11:30:08.636 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:08.649 7fc9fffff700 1 -- 172.21.2.112:0/567072959 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fc9f00077d0 con 0x7fc9ec0186c0 2021-02-10T11:30:08.640 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:08.652 7fca0e1af700 1 -- 172.21.2.112:0/567072959 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc9ec0186c0 msgr2=0x7fc9ec01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:08.640 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:08.652 7fca0e1af700 1 --2- 172.21.2.112:0/567072959 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc9ec0186c0 0x7fc9ec01abc0 crc :-1 s=READY pgs=387 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:08.640 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:08.652 7fca0e1af700 1 -- 172.21.2.112:0/567072959 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fca080b3af0 msgr2=0x7fca080f0580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:08.640 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:08.652 7fca0e1af700 1 --2- 172.21.2.112:0/567072959 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fca080b3af0 0x7fca080f0580 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 rx=0x7fca00009980 tx=0x7fca0000bcd0).stop 2021-02-10T11:30:08.640 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:08.652 7fca0e1af700 1 -- 172.21.2.112:0/567072959 shutdown_connections 2021-02-10T11:30:08.640 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:08.652 7fca0e1af700 1 --2- 172.21.2.112:0/567072959 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc9ec0186c0 0x7fc9ec01abc0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:08.640 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:08.652 7fca0e1af700 1 --2- 172.21.2.112:0/567072959 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fca080b3af0 0x7fca080f0580 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:08.640 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:08.652 7fca0e1af700 1 --2- 172.21.2.112:0/567072959 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fca080b30d0 0x7fca080b34d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:08.640 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:08.652 7fca0e1af700 1 --2- 172.21.2.112:0/567072959 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fca080b2840 0x7fca081a4300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:08.641 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:08.652 7fca0e1af700 1 -- 172.21.2.112:0/567072959 >> 172.21.2.112:0/567072959 conn(0x7fca080c36f0 msgr2=0x7fca08094c30 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:08.641 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:08.653 7fca0e1af700 1 -- 172.21.2.112:0/567072959 shutdown_connections 2021-02-10T11:30:08.641 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:08.653 7fca0e1af700 1 -- 172.21.2.112:0/567072959 wait complete. 2021-02-10T11:30:08.649 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000348569 2021-02-10T11:30:09.081 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.077 7f111001f700 1 Processor -- start 2021-02-10T11:30:09.081 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.077 7f111001f700 1 -- start start 2021-02-10T11:30:09.082 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.077 7f111001f700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f11080f05f0 0x7f11080f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:09.082 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.077 7f111001f700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11081b0c70 con 0x7f11081b0470 2021-02-10T11:30:09.082 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.077 7f111001f700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11080f0b50 con 0x7f11081b0870 2021-02-10T11:30:09.082 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.077 7f111001f700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11080e5ea0 con 0x7f11080f05f0 2021-02-10T11:30:09.082 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.078 7f110de3b700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f11081b0470 0x7f11080ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:46060/0 (socket says 172.21.2.112:46060) 2021-02-10T11:30:09.082 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.078 7f110de3b700 1 -- 172.21.2.112:0/2434094859 learned_addr learned my addr 172.21.2.112:0/2434094859 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:30:09.082 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.078 7f110ce39700 1 -- 172.21.2.112:0/2434094859 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1168570511 0 0) 0x7f10fc0010b0 con 0x7f11081b0470 2021-02-10T11:30:09.082 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.078 7f110ce39700 1 -- 172.21.2.112:0/2434094859 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10f4002fe0 con 0x7f11081b0470 2021-02-10T11:30:09.082 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.078 7f110e63c700 1 --2- 172.21.2.112:0/2434094859 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f11080f05f0 0x7f11080f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:09.082 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.078 7f110ce39700 1 -- 172.21.2.112:0/2434094859 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1352936505 0 0) 0x7f10fc0010b0 con 0x7f11081b0470 2021-02-10T11:30:09.083 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.078 7f110e63c700 1 -- 172.21.2.112:0/2434094859 >> v1:172.21.2.120:6789/0 conn(0x7f11081b0870 legacy=0x7f11080efe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:09.083 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.078 7f110e63c700 1 -- 172.21.2.112:0/2434094859 >> v1:172.21.2.112:6789/0 conn(0x7f11081b0470 legacy=0x7f11080ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:09.083 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.078 7f110e63c700 1 -- 172.21.2.112:0/2434094859 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11080e52e0 con 0x7f11080f05f0 2021-02-10T11:30:09.083 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.078 7f110e63c700 1 --2- 172.21.2.112:0/2434094859 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f11080f05f0 0x7f11080f20e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f1100006e50 tx=0x7f1100009df0).ready entity=mon.2 client_cookie=bd6e9b6fdb860de0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:09.083 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.079 7f110ce39700 1 -- 172.21.2.112:0/2434094859 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f110000b510 con 0x7f11080f05f0 2021-02-10T11:30:09.083 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.079 7f110ce39700 1 -- 172.21.2.112:0/2434094859 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1100003030 con 0x7f11080f05f0 2021-02-10T11:30:09.083 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.079 7f110ce39700 1 -- 172.21.2.112:0/2434094859 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f110000b7d0 con 0x7f11080f05f0 2021-02-10T11:30:09.083 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.079 7f111001f700 1 -- 172.21.2.112:0/2434094859 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f11080f05f0 msgr2=0x7f11080f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:09.083 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.079 7f111001f700 1 --2- 172.21.2.112:0/2434094859 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f11080f05f0 0x7f11080f20e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f1100006e50 tx=0x7f1100009df0).stop 2021-02-10T11:30:09.083 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.079 7f111001f700 1 -- 172.21.2.112:0/2434094859 shutdown_connections 2021-02-10T11:30:09.083 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.079 7f111001f700 1 --2- 172.21.2.112:0/2434094859 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f11080f05f0 0x7f11080f20e0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:09.084 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.079 7f111001f700 1 -- 172.21.2.112:0/2434094859 >> 172.21.2.112:0/2434094859 conn(0x7f11080ce3e0 msgr2=0x7f11080e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:09.084 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.079 7f111001f700 1 -- 172.21.2.112:0/2434094859 shutdown_connections 2021-02-10T11:30:09.084 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.079 7f111001f700 1 -- 172.21.2.112:0/2434094859 wait complete. 2021-02-10T11:30:09.085 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.094 7f111001f700 1 Processor -- start 2021-02-10T11:30:09.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.094 7f111001f700 1 -- start start 2021-02-10T11:30:09.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.095 7f111001f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f11081a05e0 0x7f110810a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:09.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.095 7f111001f700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f11080ab6a0 0x7f1108109290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:09.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.095 7f111001f700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f11081098b0 0x7f11080a0ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:09.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.095 7f111001f700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11080a0920 con 0x7f11081a05e0 2021-02-10T11:30:09.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.095 7f111001f700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1108109cb0 con 0x7f11081098b0 2021-02-10T11:30:09.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.095 7f111001f700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f110810a6f0 con 0x7f11080ab6a0 2021-02-10T11:30:09.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.095 7f110de3b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f11081a05e0 0x7f110810a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:09.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.095 7f110de3b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f11081a05e0 0x7f110810a0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:46604/0 (socket says 172.21.2.112:46604) 2021-02-10T11:30:09.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.095 7f110de3b700 1 -- 172.21.2.112:0/4125644669 learned_addr learned my addr 172.21.2.112:0/4125644669 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:30:09.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.095 7f110e63c700 1 --2- 172.21.2.112:0/4125644669 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f11081098b0 0x7f11080a0ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:09.087 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.095 7f110d63a700 1 --2- 172.21.2.112:0/4125644669 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f11080ab6a0 0x7f1108109290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:09.087 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.095 7f110de3b700 1 -- 172.21.2.112:0/4125644669 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f11080ab6a0 msgr2=0x7f1108109290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:09.087 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.095 7f110de3b700 1 --2- 172.21.2.112:0/4125644669 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f11080ab6a0 0x7f1108109290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:09.087 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.095 7f110de3b700 1 -- 172.21.2.112:0/4125644669 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f11081098b0 msgr2=0x7f11080a0ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:09.087 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.095 7f110de3b700 1 --2- 172.21.2.112:0/4125644669 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f11081098b0 0x7f11080a0ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:09.087 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.095 7f110de3b700 1 -- 172.21.2.112:0/4125644669 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11080a14c0 con 0x7f11081a05e0 2021-02-10T11:30:09.087 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.095 7f110de3b700 1 --2- 172.21.2.112:0/4125644669 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f11081a05e0 0x7f110810a0d0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 rx=0x7f10fc001c60 tx=0x7f10fc00a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:09.087 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.095 7f10fa7fc700 1 -- 172.21.2.112:0/4125644669 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f10fc00acc0 con 0x7f11081a05e0 2021-02-10T11:30:09.087 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.095 7f10fa7fc700 1 -- 172.21.2.112:0/4125644669 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10fc004140 con 0x7f11081a05e0 2021-02-10T11:30:09.087 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.096 7f111001f700 1 -- 172.21.2.112:0/4125644669 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11080a14c0 con 0x7f11081a05e0 2021-02-10T11:30:09.087 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.096 7f10fa7fc700 1 -- 172.21.2.112:0/4125644669 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f10fc004620 con 0x7f11081a05e0 2021-02-10T11:30:09.087 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.096 7f111001f700 1 -- 172.21.2.112:0/4125644669 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11080a14c0 con 0x7f11081a05e0 2021-02-10T11:30:09.088 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.096 7f10fa7fc700 1 -- 172.21.2.112:0/4125644669 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f10fc004320 con 0x7f11081a05e0 2021-02-10T11:30:09.088 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.096 7f10fa7fc700 1 --2- 172.21.2.112:0/4125644669 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f10f0018730 0x7f10f001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:09.088 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.096 7f10fa7fc700 1 -- 172.21.2.112:0/4125644669 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f10fc000940 con 0x7f11081a05e0 2021-02-10T11:30:09.088 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.096 7f110d63a700 1 --2- 172.21.2.112:0/4125644669 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f10f0018730 0x7f10f001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:09.088 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.097 7f110d63a700 1 --2- 172.21.2.112:0/4125644669 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f10f0018730 0x7f10f001ac10 crc :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:09.115 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.127 7f111001f700 1 -- 172.21.2.112:0/4125644669 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1108105af0 con 0x7f11081a05e0 2021-02-10T11:30:09.117 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.130 7f10fa7fc700 1 -- 172.21.2.112:0/4125644669 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f10fc000940 con 0x7f11081a05e0 2021-02-10T11:30:09.245 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.258 7f111001f700 1 -- 172.21.2.112:0/4125644669 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348569", "target": ["mgr", ""]}) v1 -- 0x7f1108105af0 con 0x7f10f0018730 2021-02-10T11:30:09.257 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.269 7f10fa7fc700 1 -- 172.21.2.112:0/4125644669 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f1104000de0 con 0x7f10f0018730 2021-02-10T11:30:09.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.276 7f111001f700 1 -- 172.21.2.112:0/4125644669 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f10f0018730 msgr2=0x7f10f001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:09.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.276 7f111001f700 1 --2- 172.21.2.112:0/4125644669 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f10f0018730 0x7f10f001ac10 crc :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:09.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.276 7f111001f700 1 -- 172.21.2.112:0/4125644669 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f11081a05e0 msgr2=0x7f110810a0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:09.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.276 7f111001f700 1 --2- 172.21.2.112:0/4125644669 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f11081a05e0 0x7f110810a0d0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 rx=0x7f10fc001c60 tx=0x7f10fc00a8d0).stop 2021-02-10T11:30:09.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.276 7f111001f700 1 -- 172.21.2.112:0/4125644669 shutdown_connections 2021-02-10T11:30:09.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.276 7f111001f700 1 --2- 172.21.2.112:0/4125644669 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f10f0018730 0x7f10f001ac10 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:09.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.276 7f111001f700 1 --2- 172.21.2.112:0/4125644669 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f11081098b0 0x7f11080a0ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:09.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.276 7f111001f700 1 --2- 172.21.2.112:0/4125644669 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f11080ab6a0 0x7f1108109290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:09.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.276 7f111001f700 1 --2- 172.21.2.112:0/4125644669 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f11081a05e0 0x7f110810a0d0 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:09.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.276 7f111001f700 1 -- 172.21.2.112:0/4125644669 >> 172.21.2.112:0/4125644669 conn(0x7f11080c40c0 msgr2=0x7f11080a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:09.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.276 7f111001f700 1 -- 172.21.2.112:0/4125644669 shutdown_connections 2021-02-10T11:30:09.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.276 7f111001f700 1 -- 172.21.2.112:0/4125644669 wait complete. 2021-02-10T11:30:09.272 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000348570 2021-02-10T11:30:09.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.697 7eff99076700 1 Processor -- start 2021-02-10T11:30:09.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.697 7eff99076700 1 -- start start 2021-02-10T11:30:09.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.697 7eff99076700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7eff941b0470 0x7eff941b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:09.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.697 7eff99076700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7eff940ef610 0x7eff940efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:09.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.697 7eff99076700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff940f0b50 con 0x7eff940f0030 2021-02-10T11:30:09.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.697 7eff99076700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff940e52e0 con 0x7eff940ef610 2021-02-10T11:30:09.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.697 7eff99076700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff940e5490 con 0x7eff941b0470 2021-02-10T11:30:09.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.698 7eff9359e700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7eff940f0030 0x7eff940f10d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:46078/0 (socket says 172.21.2.112:46078) 2021-02-10T11:30:09.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.698 7eff9359e700 1 -- 172.21.2.112:0/3319048944 learned_addr learned my addr 172.21.2.112:0/3319048944 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:30:09.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.698 7eff92d9d700 1 --2- 172.21.2.112:0/3319048944 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7eff941b0470 0x7eff941b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:09.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.698 7eff9259c700 1 --2- 172.21.2.112:0/3319048944 >> v2:172.21.2.120:3300/0 conn(0x7eff940ef610 0x7eff940efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:09.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.698 7eff91d9b700 1 -- 172.21.2.112:0/3319048944 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 278024138 0 0) 0x7eff8c000f90 con 0x7eff940f0030 2021-02-10T11:30:09.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.698 7eff91d9b700 1 -- 172.21.2.112:0/3319048944 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff80002fe0 con 0x7eff940f0030 2021-02-10T11:30:09.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.698 7eff9259c700 1 -- 172.21.2.112:0/3319048944 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7eff941b0470 msgr2=0x7eff941b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:09.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.698 7eff9259c700 1 --2- 172.21.2.112:0/3319048944 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7eff941b0470 0x7eff941b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:09.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.698 7eff9259c700 1 -- 172.21.2.112:0/3319048944 >> v1:172.21.2.112:6789/0 conn(0x7eff940f0030 legacy=0x7eff940f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:09.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.698 7eff9259c700 1 -- 172.21.2.112:0/3319048944 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff940f0430 con 0x7eff940ef610 2021-02-10T11:30:09.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.698 7eff92d9d700 1 --2- 172.21.2.112:0/3319048944 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7eff941b0470 0x7eff941b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:30:09.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.698 7eff9259c700 1 --2- 172.21.2.112:0/3319048944 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7eff940ef610 0x7eff940efa10 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7eff7c006650 tx=0x7eff7c009a80).ready entity=mon.1 client_cookie=acd8911348d966fa server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:09.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.698 7eff91d9b700 1 -- 172.21.2.112:0/3319048944 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7eff7c00c0c0 con 0x7eff940ef610 2021-02-10T11:30:09.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.699 7eff91d9b700 1 -- 172.21.2.112:0/3319048944 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff7c00c0c0 con 0x7eff940ef610 2021-02-10T11:30:09.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.699 7eff91d9b700 1 -- 172.21.2.112:0/3319048944 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7eff7c00c340 con 0x7eff940ef610 2021-02-10T11:30:09.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.699 7eff99076700 1 -- 172.21.2.112:0/3319048944 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7eff940ef610 msgr2=0x7eff940efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:09.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.699 7eff99076700 1 --2- 172.21.2.112:0/3319048944 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7eff940ef610 0x7eff940efa10 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7eff7c006650 tx=0x7eff7c009a80).stop 2021-02-10T11:30:09.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.699 7eff99076700 1 -- 172.21.2.112:0/3319048944 shutdown_connections 2021-02-10T11:30:09.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.699 7eff99076700 1 --2- 172.21.2.112:0/3319048944 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7eff940ef610 0x7eff940efa10 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:09.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.699 7eff99076700 1 --2- 172.21.2.112:0/3319048944 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7eff941b0470 0x7eff941b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:09.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.699 7eff99076700 1 -- 172.21.2.112:0/3319048944 >> 172.21.2.112:0/3319048944 conn(0x7eff940ce3e0 msgr2=0x7eff940e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:09.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.699 7eff99076700 1 -- 172.21.2.112:0/3319048944 shutdown_connections 2021-02-10T11:30:09.704 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.699 7eff99076700 1 -- 172.21.2.112:0/3319048944 wait complete. 2021-02-10T11:30:09.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.714 7eff99076700 1 Processor -- start 2021-02-10T11:30:09.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.714 7eff99076700 1 -- start start 2021-02-10T11:30:09.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.714 7eff99076700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7eff940a0420 0x7eff940a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:09.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.714 7eff99076700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7eff94189d00 0x7eff9418a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:09.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.714 7eff99076700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7eff9418a720 0x7eff940b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:09.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.714 7eff99076700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff94189800 con 0x7eff9418a720 2021-02-10T11:30:09.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.714 7eff99076700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff9418ab20 con 0x7eff940a0420 2021-02-10T11:30:09.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.714 7eff99076700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff940a1140 con 0x7eff94189d00 2021-02-10T11:30:09.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.715 7eff9359e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7eff9418a720 0x7eff940b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:09.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.715 7eff9359e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7eff9418a720 0x7eff940b7be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:46618/0 (socket says 172.21.2.112:46618) 2021-02-10T11:30:09.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.715 7eff9359e700 1 -- 172.21.2.112:0/2434667252 learned_addr learned my addr 172.21.2.112:0/2434667252 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:30:09.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.715 7eff9359e700 1 -- 172.21.2.112:0/2434667252 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7eff94189d00 msgr2=0x7eff9418a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:09.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.715 7eff9359e700 1 --2- 172.21.2.112:0/2434667252 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7eff94189d00 0x7eff9418a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:09.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.715 7eff9359e700 1 -- 172.21.2.112:0/2434667252 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7eff940a0420 msgr2=0x7eff940a0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:09.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.715 7eff9359e700 1 --2- 172.21.2.112:0/2434667252 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7eff940a0420 0x7eff940a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:09.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.715 7eff9359e700 1 -- 172.21.2.112:0/2434667252 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff940b8200 con 0x7eff9418a720 2021-02-10T11:30:09.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.715 7eff9359e700 1 --2- 172.21.2.112:0/2434667252 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7eff9418a720 0x7eff940b7be0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 rx=0x7eff8c009980 tx=0x7eff8c00bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:09.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.715 7eff8b7fe700 1 -- 172.21.2.112:0/2434667252 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7eff8c00be10 con 0x7eff9418a720 2021-02-10T11:30:09.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.715 7eff8b7fe700 1 -- 172.21.2.112:0/2434667252 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff8c0060c0 con 0x7eff9418a720 2021-02-10T11:30:09.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.715 7eff99076700 1 -- 172.21.2.112:0/2434667252 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff940b8200 con 0x7eff9418a720 2021-02-10T11:30:09.707 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.716 7eff99076700 1 -- 172.21.2.112:0/2434667252 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff940b8200 con 0x7eff9418a720 2021-02-10T11:30:09.707 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.716 7eff8b7fe700 1 -- 172.21.2.112:0/2434667252 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7eff8c006550 con 0x7eff9418a720 2021-02-10T11:30:09.707 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.716 7eff8b7fe700 1 -- 172.21.2.112:0/2434667252 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7eff8c0062a0 con 0x7eff9418a720 2021-02-10T11:30:09.707 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.716 7eff8b7fe700 1 --2- 172.21.2.112:0/2434667252 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7eff78018730 0x7eff7801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:09.707 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.716 7eff8b7fe700 1 -- 172.21.2.112:0/2434667252 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7eff8c032b40 con 0x7eff9418a720 2021-02-10T11:30:09.707 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.716 7eff92d9d700 1 --2- 172.21.2.112:0/2434667252 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7eff78018730 0x7eff7801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:09.707 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.717 7eff92d9d700 1 --2- 172.21.2.112:0/2434667252 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7eff78018730 0x7eff7801ac10 crc :-1 s=READY pgs=389 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:09.735 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.747 7eff99076700 1 -- 172.21.2.112:0/2434667252 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff9418ab20 con 0x7eff9418a720 2021-02-10T11:30:09.737 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.750 7eff8b7fe700 1 -- 172.21.2.112:0/2434667252 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7eff8c0062d0 con 0x7eff9418a720 2021-02-10T11:30:09.866 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.878 7eff99076700 1 -- 172.21.2.112:0/2434667252 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348570", "target": ["mgr", ""]}) v1 -- 0x7eff9418ab20 con 0x7eff78018730 2021-02-10T11:30:09.876 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.889 7eff8b7fe700 1 -- 172.21.2.112:0/2434667252 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7eff84006420 con 0x7eff78018730 2021-02-10T11:30:09.884 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.896 7eff99076700 1 -- 172.21.2.112:0/2434667252 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7eff78018730 msgr2=0x7eff7801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:09.884 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.896 7eff99076700 1 --2- 172.21.2.112:0/2434667252 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7eff78018730 0x7eff7801ac10 crc :-1 s=READY pgs=389 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:09.884 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.896 7eff99076700 1 -- 172.21.2.112:0/2434667252 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7eff9418a720 msgr2=0x7eff940b7be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:09.884 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.896 7eff99076700 1 --2- 172.21.2.112:0/2434667252 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7eff9418a720 0x7eff940b7be0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 rx=0x7eff8c009980 tx=0x7eff8c00bcd0).stop 2021-02-10T11:30:09.884 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.896 7eff99076700 1 -- 172.21.2.112:0/2434667252 shutdown_connections 2021-02-10T11:30:09.884 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.896 7eff99076700 1 --2- 172.21.2.112:0/2434667252 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7eff78018730 0x7eff7801ac10 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:09.884 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.896 7eff99076700 1 --2- 172.21.2.112:0/2434667252 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7eff9418a720 0x7eff940b7be0 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:09.885 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.896 7eff99076700 1 --2- 172.21.2.112:0/2434667252 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7eff94189d00 0x7eff9418a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:09.885 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.896 7eff99076700 1 --2- 172.21.2.112:0/2434667252 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7eff940a0420 0x7eff940a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:09.885 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.896 7eff99076700 1 -- 172.21.2.112:0/2434667252 >> 172.21.2.112:0/2434667252 conn(0x7eff94094e10 msgr2=0x7eff940a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:09.885 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.896 7eff99076700 1 -- 172.21.2.112:0/2434667252 shutdown_connections 2021-02-10T11:30:09.885 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:09.897 7eff99076700 1 -- 172.21.2.112:0/2434667252 wait complete. 2021-02-10T11:30:09.892 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000348571 2021-02-10T11:30:10.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.318 7fb4d3fd4700 1 Processor -- start 2021-02-10T11:30:10.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.318 7fb4d3fd4700 1 -- start start 2021-02-10T11:30:10.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.319 7fb4d3fd4700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fb4cc1b0470 0x7fb4cc1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:10.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.319 7fb4d3fd4700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fb4cc0f20e0 0x7fb4cc0f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:10.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.319 7fb4d3fd4700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4cc0f0b50 con 0x7fb4cc0ef610 2021-02-10T11:30:10.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.319 7fb4d3fd4700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4cc0f0820 con 0x7fb4cc0f20e0 2021-02-10T11:30:10.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.319 7fb4d3fd4700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4cc0f24e0 con 0x7fb4cc1b0470 2021-02-10T11:30:10.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.319 7fb4d15ef700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fb4cc0ef610 0x7fb4cc0efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:46088/0 (socket says 172.21.2.112:46088) 2021-02-10T11:30:10.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.319 7fb4d15ef700 1 -- 172.21.2.112:0/3197392830 learned_addr learned my addr 172.21.2.112:0/3197392830 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:30:10.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.319 7fb4d25f1700 1 --2- 172.21.2.112:0/3197392830 >> v2:172.21.2.112:3300/0 conn(0x7fb4cc0f20e0 0x7fb4cc0f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:10.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.319 7fb4d1df0700 1 --2- 172.21.2.112:0/3197392830 >> v2:172.21.2.120:3300/0 conn(0x7fb4cc1b0470 0x7fb4cc1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:10.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.319 7fb4d1df0700 1 -- 172.21.2.112:0/3197392830 >> v2:172.21.2.112:3300/0 conn(0x7fb4cc0f20e0 msgr2=0x7fb4cc0f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:10.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.319 7fb4d1df0700 1 --2- 172.21.2.112:0/3197392830 >> v2:172.21.2.112:3300/0 conn(0x7fb4cc0f20e0 0x7fb4cc0f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:10.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.319 7fb4d1df0700 1 -- 172.21.2.112:0/3197392830 >> v1:172.21.2.112:6789/0 conn(0x7fb4cc0ef610 legacy=0x7fb4cc0efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:10.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.319 7fb4d1df0700 1 -- 172.21.2.112:0/3197392830 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4cc0e52e0 con 0x7fb4cc1b0470 2021-02-10T11:30:10.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.319 7fb4d0dee700 1 -- 172.21.2.112:0/3197392830 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1553988248 0 0) 0x7fb4c00009b0 con 0x7fb4cc0ef610 2021-02-10T11:30:10.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.320 7fb4d1df0700 1 --2- 172.21.2.112:0/3197392830 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb4cc1b0470 0x7fb4cc1b0870 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 rx=0x7fb4c8006650 tx=0x7fb4c8009a80).ready entity=mon.1 client_cookie=5a387bd17bb16233 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:10.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.320 7fb4d0dee700 1 -- 172.21.2.112:0/3197392830 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb4c800c0c0 con 0x7fb4cc1b0470 2021-02-10T11:30:10.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.320 7fb4d0dee700 1 -- 172.21.2.112:0/3197392830 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4c800c340 con 0x7fb4cc1b0470 2021-02-10T11:30:10.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.320 7fb4d0dee700 1 -- 172.21.2.112:0/3197392830 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb4c800c860 con 0x7fb4cc1b0470 2021-02-10T11:30:10.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.320 7fb4d3fd4700 1 -- 172.21.2.112:0/3197392830 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb4cc1b0470 msgr2=0x7fb4cc1b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:10.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.320 7fb4d3fd4700 1 --2- 172.21.2.112:0/3197392830 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb4cc1b0470 0x7fb4cc1b0870 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 rx=0x7fb4c8006650 tx=0x7fb4c8009a80).stop 2021-02-10T11:30:10.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.321 7fb4d3fd4700 1 -- 172.21.2.112:0/3197392830 shutdown_connections 2021-02-10T11:30:10.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.321 7fb4d3fd4700 1 --2- 172.21.2.112:0/3197392830 >> v2:172.21.2.112:3300/0 conn(0x7fb4cc0f20e0 0x7fb4cc0f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:10.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.321 7fb4d3fd4700 1 --2- 172.21.2.112:0/3197392830 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb4cc1b0470 0x7fb4cc1b0870 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:10.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.321 7fb4d3fd4700 1 -- 172.21.2.112:0/3197392830 >> 172.21.2.112:0/3197392830 conn(0x7fb4cc0ce3e0 msgr2=0x7fb4cc0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:10.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.321 7fb4d3fd4700 1 -- 172.21.2.112:0/3197392830 shutdown_connections 2021-02-10T11:30:10.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.321 7fb4d3fd4700 1 -- 172.21.2.112:0/3197392830 wait complete. 2021-02-10T11:30:10.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.336 7fb4d3fd4700 1 Processor -- start 2021-02-10T11:30:10.326 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.336 7fb4d3fd4700 1 -- start start 2021-02-10T11:30:10.327 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.336 7fb4d3fd4700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb4cc0a0420 0x7fb4cc0a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:10.327 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.336 7fb4d3fd4700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb4cc189d00 0x7fb4cc18a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:10.327 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.336 7fb4d3fd4700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb4cc18a720 0x7fb4cc0b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:10.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.336 7fb4d3fd4700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4cc189780 con 0x7fb4cc18a720 2021-02-10T11:30:10.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.336 7fb4d3fd4700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4cc18ab20 con 0x7fb4cc189d00 2021-02-10T11:30:10.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.336 7fb4d3fd4700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4cc0a1140 con 0x7fb4cc0a0420 2021-02-10T11:30:10.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.336 7fb4d25f1700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb4cc18a720 0x7fb4cc0b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:10.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.336 7fb4d25f1700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb4cc18a720 0x7fb4cc0b7be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:46634/0 (socket says 172.21.2.112:46634) 2021-02-10T11:30:10.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.336 7fb4d25f1700 1 -- 172.21.2.112:0/921217318 learned_addr learned my addr 172.21.2.112:0/921217318 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:30:10.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.336 7fb4d15ef700 1 --2- 172.21.2.112:0/921217318 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb4cc189d00 0x7fb4cc18a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:10.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.336 7fb4d1df0700 1 --2- 172.21.2.112:0/921217318 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb4cc0a0420 0x7fb4cc0a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:10.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.337 7fb4d25f1700 1 -- 172.21.2.112:0/921217318 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb4cc0a0420 msgr2=0x7fb4cc0a0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:10.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.337 7fb4d25f1700 1 --2- 172.21.2.112:0/921217318 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb4cc0a0420 0x7fb4cc0a0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:10.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.337 7fb4d25f1700 1 -- 172.21.2.112:0/921217318 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb4cc189d00 msgr2=0x7fb4cc18a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:10.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.337 7fb4d25f1700 1 --2- 172.21.2.112:0/921217318 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb4cc189d00 0x7fb4cc18a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:10.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.337 7fb4d25f1700 1 -- 172.21.2.112:0/921217318 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4cc0b8200 con 0x7fb4cc18a720 2021-02-10T11:30:10.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.337 7fb4d25f1700 1 --2- 172.21.2.112:0/921217318 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb4cc18a720 0x7fb4cc0b7be0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 rx=0x7fb4c400b970 tx=0x7fb4c400bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:10.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.337 7fb4be7fc700 1 -- 172.21.2.112:0/921217318 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb4c4000f90 con 0x7fb4cc18a720 2021-02-10T11:30:10.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.337 7fb4be7fc700 1 -- 172.21.2.112:0/921217318 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4c4001270 con 0x7fb4cc18a720 2021-02-10T11:30:10.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.337 7fb4be7fc700 1 -- 172.21.2.112:0/921217318 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb4c4001790 con 0x7fb4cc18a720 2021-02-10T11:30:10.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.337 7fb4d3fd4700 1 -- 172.21.2.112:0/921217318 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4cc0b8200 con 0x7fb4cc18a720 2021-02-10T11:30:10.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.337 7fb4d3fd4700 1 -- 172.21.2.112:0/921217318 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4cc0b8200 con 0x7fb4cc18a720 2021-02-10T11:30:10.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.338 7fb4be7fc700 1 -- 172.21.2.112:0/921217318 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb4c4000e30 con 0x7fb4cc18a720 2021-02-10T11:30:10.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.338 7fb4be7fc700 1 --2- 172.21.2.112:0/921217318 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb4b40186c0 0x7fb4b401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:10.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.338 7fb4be7fc700 1 -- 172.21.2.112:0/921217318 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fb4c4032930 con 0x7fb4cc18a720 2021-02-10T11:30:10.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.338 7fb4d1df0700 1 --2- 172.21.2.112:0/921217318 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb4b40186c0 0x7fb4b401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:10.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.339 7fb4d1df0700 1 --2- 172.21.2.112:0/921217318 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb4b40186c0 0x7fb4b401abc0 crc :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:10.356 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.368 7fb4d3fd4700 1 -- 172.21.2.112:0/921217318 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4cc105ae0 con 0x7fb4cc18a720 2021-02-10T11:30:10.358 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.371 7fb4be7fc700 1 -- 172.21.2.112:0/921217318 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb4c4005820 con 0x7fb4cc18a720 2021-02-10T11:30:10.484 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.496 7fb4d3fd4700 1 -- 172.21.2.112:0/921217318 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348571", "target": ["mgr", ""]}) v1 -- 0x7fb4cc105ae0 con 0x7fb4b40186c0 2021-02-10T11:30:10.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.508 7fb4be7fc700 1 -- 172.21.2.112:0/921217318 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fb4c8009180 con 0x7fb4b40186c0 2021-02-10T11:30:10.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.514 7fb4d3fd4700 1 -- 172.21.2.112:0/921217318 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb4b40186c0 msgr2=0x7fb4b401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:10.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.514 7fb4d3fd4700 1 --2- 172.21.2.112:0/921217318 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb4b40186c0 0x7fb4b401abc0 crc :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:10.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.514 7fb4d3fd4700 1 -- 172.21.2.112:0/921217318 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb4cc18a720 msgr2=0x7fb4cc0b7be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:10.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.514 7fb4d3fd4700 1 --2- 172.21.2.112:0/921217318 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb4cc18a720 0x7fb4cc0b7be0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 rx=0x7fb4c400b970 tx=0x7fb4c400bd10).stop 2021-02-10T11:30:10.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.514 7fb4d3fd4700 1 -- 172.21.2.112:0/921217318 shutdown_connections 2021-02-10T11:30:10.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.514 7fb4d3fd4700 1 --2- 172.21.2.112:0/921217318 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb4b40186c0 0x7fb4b401abc0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:10.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.514 7fb4d3fd4700 1 --2- 172.21.2.112:0/921217318 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb4cc189d00 0x7fb4cc18a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:10.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.514 7fb4d3fd4700 1 --2- 172.21.2.112:0/921217318 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb4cc18a720 0x7fb4cc0b7be0 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:10.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.514 7fb4d3fd4700 1 --2- 172.21.2.112:0/921217318 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb4cc0a0420 0x7fb4cc0a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:10.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.514 7fb4d3fd4700 1 -- 172.21.2.112:0/921217318 >> 172.21.2.112:0/921217318 conn(0x7fb4cc094e10 msgr2=0x7fb4cc0a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:10.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.515 7fb4d3fd4700 1 -- 172.21.2.112:0/921217318 shutdown_connections 2021-02-10T11:30:10.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.515 7fb4d3fd4700 1 -- 172.21.2.112:0/921217318 wait complete. 2021-02-10T11:30:10.511 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000348572 2021-02-10T11:30:10.940 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.936 7f79270de700 1 Processor -- start 2021-02-10T11:30:10.940 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.936 7f79270de700 1 -- start start 2021-02-10T11:30:10.940 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.936 7f79270de700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f79201b0470 0x7f79201b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:10.941 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.936 7f79270de700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f79200ef610 0x7f79200efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:10.941 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.936 7f79270de700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79200f0b50 con 0x7f79200f0030 2021-02-10T11:30:10.941 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.936 7f79270de700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79200e52e0 con 0x7f79200ef610 2021-02-10T11:30:10.941 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.936 7f79270de700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79200e5490 con 0x7f79201b0470 2021-02-10T11:30:10.941 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.936 7f79256fb700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f79200f0030 0x7f79200f10d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:46104/0 (socket says 172.21.2.112:46104) 2021-02-10T11:30:10.941 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.936 7f79256fb700 1 -- 172.21.2.112:0/1912749293 learned_addr learned my addr 172.21.2.112:0/1912749293 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:30:10.941 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.937 7f791ffff700 1 --2- 172.21.2.112:0/1912749293 >> v2:172.21.2.120:3300/0 conn(0x7f79200ef610 0x7f79200efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:10.941 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.937 7f7924efa700 1 --2- 172.21.2.112:0/1912749293 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f79201b0470 0x7f79201b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:10.941 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.937 7f791f7fe700 1 -- 172.21.2.112:0/1912749293 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2557189629 0 0) 0x7f7918001500 con 0x7f79200f0030 2021-02-10T11:30:10.941 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.937 7f791f7fe700 1 -- 172.21.2.112:0/1912749293 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f790c002fe0 con 0x7f79200f0030 2021-02-10T11:30:10.941 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.937 7f791ffff700 1 -- 172.21.2.112:0/1912749293 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f79201b0470 msgr2=0x7f79201b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:10.941 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.937 7f791ffff700 1 --2- 172.21.2.112:0/1912749293 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f79201b0470 0x7f79201b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:10.941 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.937 7f791ffff700 1 -- 172.21.2.112:0/1912749293 >> v1:172.21.2.112:6789/0 conn(0x7f79200f0030 legacy=0x7f79200f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:10.941 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.937 7f791ffff700 1 -- 172.21.2.112:0/1912749293 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79200f0430 con 0x7f79200ef610 2021-02-10T11:30:10.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.937 7f791ffff700 1 --2- 172.21.2.112:0/1912749293 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f79200ef610 0x7f79200efa10 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7f7910006650 tx=0x7f7910009a80).ready entity=mon.1 client_cookie=bcbd6cdb7d7898bd server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:10.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.937 7f791f7fe700 1 -- 172.21.2.112:0/1912749293 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f79100009b0 con 0x7f79200ef610 2021-02-10T11:30:10.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.937 7f791f7fe700 1 -- 172.21.2.112:0/1912749293 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f791000c0c0 con 0x7f79200ef610 2021-02-10T11:30:10.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.937 7f791f7fe700 1 -- 172.21.2.112:0/1912749293 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f79100009b0 con 0x7f79200ef610 2021-02-10T11:30:10.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.937 7f79270de700 1 -- 172.21.2.112:0/1912749293 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f79200ef610 msgr2=0x7f79200efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:10.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.937 7f79270de700 1 --2- 172.21.2.112:0/1912749293 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f79200ef610 0x7f79200efa10 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7f7910006650 tx=0x7f7910009a80).stop 2021-02-10T11:30:10.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.938 7f79270de700 1 -- 172.21.2.112:0/1912749293 shutdown_connections 2021-02-10T11:30:10.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.938 7f79270de700 1 --2- 172.21.2.112:0/1912749293 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f79200ef610 0x7f79200efa10 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:10.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.938 7f79270de700 1 --2- 172.21.2.112:0/1912749293 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f79201b0470 0x7f79201b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:10.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.938 7f79270de700 1 -- 172.21.2.112:0/1912749293 >> 172.21.2.112:0/1912749293 conn(0x7f79200ce3e0 msgr2=0x7f79200e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:10.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.938 7f79270de700 1 -- 172.21.2.112:0/1912749293 shutdown_connections 2021-02-10T11:30:10.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.938 7f79270de700 1 -- 172.21.2.112:0/1912749293 wait complete. 2021-02-10T11:30:10.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.953 7f79270de700 1 Processor -- start 2021-02-10T11:30:10.943 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.953 7f79270de700 1 -- start start 2021-02-10T11:30:10.944 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.953 7f79270de700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f79200a0420 0x7f79200a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:10.944 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.953 7f79270de700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7920189d00 0x7f792018a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:10.945 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.953 7f79270de700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f792018a720 0x7f79200b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:10.945 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.953 7f79270de700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7920189780 con 0x7f79200a0420 2021-02-10T11:30:10.945 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.953 7f79270de700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f792018ab20 con 0x7f7920189d00 2021-02-10T11:30:10.945 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.953 7f79270de700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79200a1140 con 0x7f792018a720 2021-02-10T11:30:10.945 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.954 7f7924efa700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f79200a0420 0x7f79200a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:10.945 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.954 7f791ffff700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7920189d00 0x7f792018a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:10.945 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.954 7f791ffff700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7920189d00 0x7f792018a100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3300/0 says I am v2:172.21.2.112:48926/0 (socket says 172.21.2.112:48926) 2021-02-10T11:30:10.945 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.954 7f7924efa700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f79200a0420 0x7f79200a0820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:46644/0 (socket says 172.21.2.112:46644) 2021-02-10T11:30:10.945 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.954 7f791ffff700 1 -- 172.21.2.112:0/3229661806 learned_addr learned my addr 172.21.2.112:0/3229661806 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:30:10.945 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.954 7f79256fb700 1 --2- 172.21.2.112:0/3229661806 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f792018a720 0x7f79200b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:10.945 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.954 7f79256fb700 1 -- 172.21.2.112:0/3229661806 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7920189d00 msgr2=0x7f792018a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:10.945 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.954 7f79256fb700 1 --2- 172.21.2.112:0/3229661806 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7920189d00 0x7f792018a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:10.945 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.954 7f79256fb700 1 -- 172.21.2.112:0/3229661806 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f79200a0420 msgr2=0x7f79200a0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:10.945 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.954 7f79256fb700 1 --2- 172.21.2.112:0/3229661806 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f79200a0420 0x7f79200a0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:10.946 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.954 7f79256fb700 1 -- 172.21.2.112:0/3229661806 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79200b8200 con 0x7f792018a720 2021-02-10T11:30:10.946 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.954 7f791ffff700 1 --2- 172.21.2.112:0/3229661806 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7920189d00 0x7f792018a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:30:10.946 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.954 7f7924efa700 1 --2- 172.21.2.112:0/3229661806 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f79200a0420 0x7f79200a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:30:10.946 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.954 7f79256fb700 1 --2- 172.21.2.112:0/3229661806 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f792018a720 0x7f79200b7be0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f7918001b90 tx=0x7f791800abe0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:10.946 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.954 7f791d7fa700 1 -- 172.21.2.112:0/3229661806 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7918001270 con 0x7f792018a720 2021-02-10T11:30:10.946 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.954 7f791d7fa700 1 -- 172.21.2.112:0/3229661806 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7918007280 con 0x7f792018a720 2021-02-10T11:30:10.946 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.954 7f791d7fa700 1 -- 172.21.2.112:0/3229661806 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7918004720 con 0x7f792018a720 2021-02-10T11:30:10.946 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.954 7f79270de700 1 -- 172.21.2.112:0/3229661806 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79200b8200 con 0x7f792018a720 2021-02-10T11:30:10.946 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.955 7f79270de700 1 -- 172.21.2.112:0/3229661806 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79200b8790 con 0x7f792018a720 2021-02-10T11:30:10.946 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.955 7f791d7fa700 1 -- 172.21.2.112:0/3229661806 <== mon.2 v2:172.21.2.120:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7918007460 con 0x7f792018a720 2021-02-10T11:30:10.946 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.955 7f791d7fa700 1 --2- 172.21.2.112:0/3229661806 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f79040186c0 0x7f790401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:10.946 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.955 7f791d7fa700 1 -- 172.21.2.112:0/3229661806 <== mon.2 v2:172.21.2.120:3301/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f7918032720 con 0x7f792018a720 2021-02-10T11:30:10.946 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.955 7f7924efa700 1 --2- 172.21.2.112:0/3229661806 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f79040186c0 0x7f790401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:10.947 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.956 7f7924efa700 1 --2- 172.21.2.112:0/3229661806 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f79040186c0 0x7f790401abc0 crc :-1 s=READY pgs=391 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:10.973 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.986 7f79270de700 1 -- 172.21.2.112:0/3229661806 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7920105ae0 con 0x7f792018a720 2021-02-10T11:30:10.976 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:10.989 7f791d7fa700 1 -- 172.21.2.112:0/3229661806 <== mon.2 v2:172.21.2.120:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7918026dd0 con 0x7f792018a720 2021-02-10T11:30:11.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:11.120 7f79270de700 1 -- 172.21.2.112:0/3229661806 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348572", "target": ["mgr", ""]}) v1 -- 0x7f79200b8790 con 0x7f79040186c0 2021-02-10T11:30:11.117 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:11.129 7f791d7fa700 1 -- 172.21.2.112:0/3229661806 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f79080009b0 con 0x7f79040186c0 2021-02-10T11:30:11.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:11.136 7f79270de700 1 -- 172.21.2.112:0/3229661806 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f79040186c0 msgr2=0x7f790401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:11.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:11.136 7f79270de700 1 --2- 172.21.2.112:0/3229661806 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f79040186c0 0x7f790401abc0 crc :-1 s=READY pgs=391 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:11.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:11.136 7f79270de700 1 -- 172.21.2.112:0/3229661806 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f792018a720 msgr2=0x7f79200b7be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:11.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:11.136 7f79270de700 1 --2- 172.21.2.112:0/3229661806 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f792018a720 0x7f79200b7be0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f7918001b90 tx=0x7f791800abe0).stop 2021-02-10T11:30:11.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:11.137 7f79270de700 1 -- 172.21.2.112:0/3229661806 shutdown_connections 2021-02-10T11:30:11.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:11.137 7f79270de700 1 --2- 172.21.2.112:0/3229661806 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f79040186c0 0x7f790401abc0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:11.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:11.137 7f79270de700 1 --2- 172.21.2.112:0/3229661806 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7920189d00 0x7f792018a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:11.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:11.137 7f79270de700 1 --2- 172.21.2.112:0/3229661806 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f792018a720 0x7f79200b7be0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:11.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:11.137 7f79270de700 1 --2- 172.21.2.112:0/3229661806 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f79200a0420 0x7f79200a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:11.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:11.137 7f79270de700 1 -- 172.21.2.112:0/3229661806 >> 172.21.2.112:0/3229661806 conn(0x7f7920094e10 msgr2=0x7f79200a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:11.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:11.137 7f79270de700 1 -- 172.21.2.112:0/3229661806 shutdown_connections 2021-02-10T11:30:11.126 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:11.137 7f79270de700 1 -- 172.21.2.112:0/3229661806 wait complete. 2021-02-10T11:30:11.133 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000348573 2021-02-10T11:30:11.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:11.557 7ff6e9955700 1 Processor -- start 2021-02-10T11:30:11.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:11.557 7ff6e9955700 1 -- start start 2021-02-10T11:30:11.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:11.557 7ff6e9955700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7ff6e41b0870 0x7ff6e40f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:11.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:11.557 7ff6e9955700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7ff6e40efe00 0x7ff6e40f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:11.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:11.558 7ff6e9955700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6e41b0c70 con 0x7ff6e41b0470 2021-02-10T11:30:11.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:11.558 7ff6e9955700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6e40e5ea0 con 0x7ff6e40efe00 2021-02-10T11:30:11.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:11.558 7ff6e9955700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6e40f0b50 con 0x7ff6e41b0870 2021-02-10T11:30:11.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:11.558 7ff6e3d75700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7ff6e40efe00 0x7ff6e40f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:11.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:11.558 7ff6e3574700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7ff6e41b0470 0x7ff6e40ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:39672/0 (socket says 172.21.2.112:39672) 2021-02-10T11:30:11.563 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:11.558 7ff6e3574700 1 -- 172.21.2.112:0/4182662217 learned_addr learned my addr 172.21.2.112:0/4182662217 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:30:11.563 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:11.558 7ff6e2d73700 1 --2- 172.21.2.112:0/4182662217 >> v2:172.21.2.120:3300/0 conn(0x7ff6e41b0870 0x7ff6e40f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:11.563 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:11.558 7ff6e3d75700 1 -- 172.21.2.112:0/4182662217 >> v2:172.21.2.120:3300/0 conn(0x7ff6e41b0870 msgr2=0x7ff6e40f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:11.563 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:11.558 7ff6e3d75700 1 --2- 172.21.2.112:0/4182662217 >> v2:172.21.2.120:3300/0 conn(0x7ff6e41b0870 0x7ff6e40f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:11.563 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:11.558 7ff6e3d75700 1 -- 172.21.2.112:0/4182662217 >> v1:172.21.2.120:6789/0 conn(0x7ff6e41b0470 legacy=0x7ff6e40ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:11.563 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:11.558 7ff6e3d75700 1 -- 172.21.2.112:0/4182662217 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6e40f0820 con 0x7ff6e40efe00 2021-02-10T11:30:11.563 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:11.559 7ff6e3d75700 1 --2- 172.21.2.112:0/4182662217 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff6e40efe00 0x7ff6e40f0200 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 rx=0x7ff6d4006ab0 tx=0x7ff6d4009b40).ready entity=mon.0 client_cookie=caed8b5036f795c4 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:11.563 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:11.559 7ff6e2572700 1 -- 172.21.2.112:0/4182662217 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff6d40022e0 con 0x7ff6e40efe00 2021-02-10T11:30:11.563 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:11.559 7ff6e2572700 1 -- 172.21.2.112:0/4182662217 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6d4002560 con 0x7ff6e40efe00 2021-02-10T11:30:11.563 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:11.559 7ff6e2572700 1 -- 172.21.2.112:0/4182662217 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff6d4002930 con 0x7ff6e40efe00 2021-02-10T11:30:11.563 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:11.559 7ff6e9955700 1 -- 172.21.2.112:0/4182662217 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff6e40efe00 msgr2=0x7ff6e40f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:11.563 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:11.559 7ff6e9955700 1 --2- 172.21.2.112:0/4182662217 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff6e40efe00 0x7ff6e40f0200 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 rx=0x7ff6d4006ab0 tx=0x7ff6d4009b40).stop 2021-02-10T11:30:11.563 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:11.560 7ff6e9955700 1 -- 172.21.2.112:0/4182662217 shutdown_connections 2021-02-10T11:30:11.564 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:11.560 7ff6e9955700 1 --2- 172.21.2.112:0/4182662217 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff6e40efe00 0x7ff6e40f0200 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:11.564 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:11.560 7ff6e9955700 1 --2- 172.21.2.112:0/4182662217 >> v2:172.21.2.120:3300/0 conn(0x7ff6e41b0870 0x7ff6e40f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:11.564 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:11.560 7ff6e9955700 1 -- 172.21.2.112:0/4182662217 >> 172.21.2.112:0/4182662217 conn(0x7ff6e40ce3e0 msgr2=0x7ff6e40e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:11.564 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:11.560 7ff6e9955700 1 -- 172.21.2.112:0/4182662217 shutdown_connections 2021-02-10T11:30:11.564 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:11.560 7ff6e9955700 1 -- 172.21.2.112:0/4182662217 wait complete. 2021-02-10T11:30:11.566 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:11.574 7ff6e9955700 1 Processor -- start 2021-02-10T11:30:11.566 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:11.575 7ff6e9955700 1 -- start start 2021-02-10T11:30:11.566 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:11.575 7ff6e9955700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff6e4105d10 0x7ff6e409eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:11.566 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:11.575 7ff6e9955700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff6e4106110 0x7ff6e409f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:11.566 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:11.575 7ff6e9955700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff6e40b2a00 0x7ff6e40b2e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:11.566 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:11.575 7ff6e9955700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6e409e560 con 0x7ff6e4106110 2021-02-10T11:30:11.566 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:11.575 7ff6e9955700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6e409c090 con 0x7ff6e40b2a00 2021-02-10T11:30:11.566 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:11.575 7ff6e9955700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6e40b3420 con 0x7ff6e4105d10 2021-02-10T11:30:11.566 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:11.575 7ff6e2d73700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff6e4106110 0x7ff6e409f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:11.566 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:11.575 7ff6e2d73700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff6e4106110 0x7ff6e409f100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:46662/0 (socket says 172.21.2.112:46662) 2021-02-10T11:30:11.566 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:11.575 7ff6e2d73700 1 -- 172.21.2.112:0/2334004847 learned_addr learned my addr 172.21.2.112:0/2334004847 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:30:11.567 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:11.575 7ff6e3574700 1 --2- 172.21.2.112:0/2334004847 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff6e4105d10 0x7ff6e409eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:11.567 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:11.575 7ff6e3d75700 1 --2- 172.21.2.112:0/2334004847 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff6e40b2a00 0x7ff6e40b2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:11.567 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:11.576 7ff6e2d73700 1 -- 172.21.2.112:0/2334004847 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff6e4105d10 msgr2=0x7ff6e409eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:11.567 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:11.576 7ff6e2d73700 1 --2- 172.21.2.112:0/2334004847 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff6e4105d10 0x7ff6e409eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:11.567 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:11.576 7ff6e2d73700 1 -- 172.21.2.112:0/2334004847 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff6e40b2a00 msgr2=0x7ff6e40b2e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:11.567 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:11.576 7ff6e2d73700 1 --2- 172.21.2.112:0/2334004847 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff6e40b2a00 0x7ff6e40b2e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:11.567 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:11.576 7ff6e2d73700 1 -- 172.21.2.112:0/2334004847 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6e40b3640 con 0x7ff6e4106110 2021-02-10T11:30:11.567 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:11.576 7ff6e2d73700 1 --2- 172.21.2.112:0/2334004847 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff6e4106110 0x7ff6e409f100 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 rx=0x7ff6d8006f60 tx=0x7ff6d800b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:11.567 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:11.576 7ff6cbfff700 1 -- 172.21.2.112:0/2334004847 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff6d800bd60 con 0x7ff6e4106110 2021-02-10T11:30:11.567 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:11.576 7ff6cbfff700 1 -- 172.21.2.112:0/2334004847 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6d8000c90 con 0x7ff6e4106110 2021-02-10T11:30:11.567 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:11.576 7ff6cbfff700 1 -- 172.21.2.112:0/2334004847 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff6d8002220 con 0x7ff6e4106110 2021-02-10T11:30:11.567 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:11.576 7ff6e9955700 1 -- 172.21.2.112:0/2334004847 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6e40b3640 con 0x7ff6e4106110 2021-02-10T11:30:11.567 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:11.576 7ff6e9955700 1 -- 172.21.2.112:0/2334004847 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6e40b3bd0 con 0x7ff6e4106110 2021-02-10T11:30:11.567 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:11.577 7ff6cbfff700 1 -- 172.21.2.112:0/2334004847 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff6d80052e0 con 0x7ff6e4106110 2021-02-10T11:30:11.568 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:11.577 7ff6cbfff700 1 --2- 172.21.2.112:0/2334004847 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff6c40186c0 0x7ff6c401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:11.568 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:11.577 7ff6cbfff700 1 -- 172.21.2.112:0/2334004847 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7ff6d8032590 con 0x7ff6e4106110 2021-02-10T11:30:11.568 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:11.577 7ff6e3574700 1 --2- 172.21.2.112:0/2334004847 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff6c40186c0 0x7ff6c401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:11.568 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:11.578 7ff6e3574700 1 --2- 172.21.2.112:0/2334004847 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff6c40186c0 0x7ff6c401abc0 crc :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:11.595 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:11.607 7ff6e9955700 1 -- 172.21.2.112:0/2334004847 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6e4105ae0 con 0x7ff6e4106110 2021-02-10T11:30:11.597 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:11.610 7ff6cbfff700 1 -- 172.21.2.112:0/2334004847 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff6d8025fc0 con 0x7ff6e4106110 2021-02-10T11:30:11.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:11.736 7ff6e9955700 1 -- 172.21.2.112:0/2334004847 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348573", "target": ["mgr", ""]}) v1 -- 0x7ff6e41a16c0 con 0x7ff6c40186c0 2021-02-10T11:30:11.734 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:11.746 7ff6cbfff700 1 -- 172.21.2.112:0/2334004847 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7ff6dc00c050 con 0x7ff6c40186c0 2021-02-10T11:30:11.741 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:11.752 7ff6e9955700 1 -- 172.21.2.112:0/2334004847 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff6c40186c0 msgr2=0x7ff6c401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:11.741 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:11.752 7ff6e9955700 1 --2- 172.21.2.112:0/2334004847 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff6c40186c0 0x7ff6c401abc0 crc :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:11.741 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:11.752 7ff6e9955700 1 -- 172.21.2.112:0/2334004847 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff6e4106110 msgr2=0x7ff6e409f100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:11.741 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:11.752 7ff6e9955700 1 --2- 172.21.2.112:0/2334004847 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff6e4106110 0x7ff6e409f100 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 rx=0x7ff6d8006f60 tx=0x7ff6d800b970).stop 2021-02-10T11:30:11.741 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:11.753 7ff6e9955700 1 -- 172.21.2.112:0/2334004847 shutdown_connections 2021-02-10T11:30:11.741 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:11.753 7ff6e9955700 1 --2- 172.21.2.112:0/2334004847 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff6c40186c0 0x7ff6c401abc0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:11.741 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:11.753 7ff6e9955700 1 --2- 172.21.2.112:0/2334004847 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff6e40b2a00 0x7ff6e40b2e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:11.742 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:11.753 7ff6e9955700 1 --2- 172.21.2.112:0/2334004847 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff6e4106110 0x7ff6e409f100 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:11.742 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:11.753 7ff6e9955700 1 --2- 172.21.2.112:0/2334004847 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff6e4105d10 0x7ff6e409eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:11.742 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:11.753 7ff6e9955700 1 -- 172.21.2.112:0/2334004847 >> 172.21.2.112:0/2334004847 conn(0x7ff6e40c36f0 msgr2=0x7ff6e40bd640 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:11.742 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:11.753 7ff6e9955700 1 -- 172.21.2.112:0/2334004847 shutdown_connections 2021-02-10T11:30:11.742 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:11.753 7ff6e9955700 1 -- 172.21.2.112:0/2334004847 wait complete. 2021-02-10T11:30:11.749 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000348574 2021-02-10T11:30:12.179 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.175 7fc6f6425700 1 Processor -- start 2021-02-10T11:30:12.180 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.175 7fc6f6425700 1 -- start start 2021-02-10T11:30:12.180 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.175 7fc6f6425700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc6f01b0870 0x7fc6f00f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:12.180 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.175 7fc6f6425700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fc6f00efe00 0x7fc6f00f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:12.180 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.175 7fc6f6425700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6f01b0c70 con 0x7fc6f01b0470 2021-02-10T11:30:12.180 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.175 7fc6f6425700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6f00f0b50 con 0x7fc6f00efe00 2021-02-10T11:30:12.180 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.175 7fc6f6425700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6f00f0820 con 0x7fc6f01b0870 2021-02-10T11:30:12.180 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.175 7fc6f4a42700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fc6f00efe00 0x7fc6f00f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:12.180 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.175 7fc6effff700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fc6f01b0470 0x7fc6f00ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:46128/0 (socket says 172.21.2.112:46128) 2021-02-10T11:30:12.180 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.175 7fc6effff700 1 -- 172.21.2.112:0/3592393476 learned_addr learned my addr 172.21.2.112:0/3592393476 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:30:12.180 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.176 7fc6ef7fe700 1 --2- 172.21.2.112:0/3592393476 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc6f01b0870 0x7fc6f00f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:12.180 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.176 7fc6f4a42700 1 -- 172.21.2.112:0/3592393476 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc6f01b0870 msgr2=0x7fc6f00f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:12.181 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.176 7fc6f4a42700 1 --2- 172.21.2.112:0/3592393476 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc6f01b0870 0x7fc6f00f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:12.181 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.176 7fc6f4a42700 1 -- 172.21.2.112:0/3592393476 >> v1:172.21.2.112:6789/0 conn(0x7fc6f01b0470 legacy=0x7fc6f00ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:12.181 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.176 7fc6f4a42700 1 -- 172.21.2.112:0/3592393476 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6f00e52e0 con 0x7fc6f00efe00 2021-02-10T11:30:12.181 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.176 7fc6eeffd700 1 -- 172.21.2.112:0/3592393476 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2934659346 0 0) 0x7fc6e0000ce0 con 0x7fc6f01b0470 2021-02-10T11:30:12.181 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.176 7fc6f4a42700 1 --2- 172.21.2.112:0/3592393476 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc6f00efe00 0x7fc6f00f0200 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 rx=0x7fc6e8006e30 tx=0x7fc6e8009de0).ready entity=mon.0 client_cookie=a668859f9699f67d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:12.181 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.176 7fc6eeffd700 1 -- 172.21.2.112:0/3592393476 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc6e800b300 con 0x7fc6f00efe00 2021-02-10T11:30:12.181 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.176 7fc6eeffd700 1 -- 172.21.2.112:0/3592393476 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6e800b680 con 0x7fc6f00efe00 2021-02-10T11:30:12.181 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.176 7fc6eeffd700 1 -- 172.21.2.112:0/3592393476 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc6e800bba0 con 0x7fc6f00efe00 2021-02-10T11:30:12.181 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.176 7fc6f6425700 1 -- 172.21.2.112:0/3592393476 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc6f00efe00 msgr2=0x7fc6f00f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:12.181 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.176 7fc6f6425700 1 --2- 172.21.2.112:0/3592393476 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc6f00efe00 0x7fc6f00f0200 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 rx=0x7fc6e8006e30 tx=0x7fc6e8009de0).stop 2021-02-10T11:30:12.181 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.177 7fc6f6425700 1 -- 172.21.2.112:0/3592393476 shutdown_connections 2021-02-10T11:30:12.181 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.177 7fc6f6425700 1 --2- 172.21.2.112:0/3592393476 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc6f00efe00 0x7fc6f00f0200 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:12.181 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.177 7fc6f6425700 1 --2- 172.21.2.112:0/3592393476 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc6f01b0870 0x7fc6f00f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:12.182 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.177 7fc6f6425700 1 -- 172.21.2.112:0/3592393476 >> 172.21.2.112:0/3592393476 conn(0x7fc6f00ce3e0 msgr2=0x7fc6f00e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:12.182 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.177 7fc6f6425700 1 -- 172.21.2.112:0/3592393476 shutdown_connections 2021-02-10T11:30:12.182 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.177 7fc6f6425700 1 -- 172.21.2.112:0/3592393476 wait complete. 2021-02-10T11:30:12.182 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.192 7fc6f6425700 1 Processor -- start 2021-02-10T11:30:12.182 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.192 7fc6f6425700 1 -- start start 2021-02-10T11:30:12.184 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.192 7fc6f6425700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc6f00a0420 0x7fc6f00a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:12.184 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.192 7fc6f6425700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc6f0189d00 0x7fc6f018a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:12.184 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.192 7fc6f6425700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc6f018a720 0x7fc6f00b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:12.184 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.192 7fc6f6425700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6f0189780 con 0x7fc6f00a0420 2021-02-10T11:30:12.184 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.192 7fc6f6425700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6f018ab20 con 0x7fc6f018a720 2021-02-10T11:30:12.184 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.192 7fc6f6425700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6f00a1140 con 0x7fc6f0189d00 2021-02-10T11:30:12.184 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.193 7fc6effff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc6f00a0420 0x7fc6f00a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:12.184 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.193 7fc6f4a42700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc6f018a720 0x7fc6f00b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:12.184 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.193 7fc6effff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc6f00a0420 0x7fc6f00a0820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:46672/0 (socket says 172.21.2.112:46672) 2021-02-10T11:30:12.184 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.193 7fc6effff700 1 -- 172.21.2.112:0/3902088383 learned_addr learned my addr 172.21.2.112:0/3902088383 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:30:12.185 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.193 7fc6ef7fe700 1 --2- 172.21.2.112:0/3902088383 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc6f0189d00 0x7fc6f018a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:12.185 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.193 7fc6effff700 1 -- 172.21.2.112:0/3902088383 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc6f0189d00 msgr2=0x7fc6f018a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:12.185 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.193 7fc6effff700 1 --2- 172.21.2.112:0/3902088383 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc6f0189d00 0x7fc6f018a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:12.185 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.193 7fc6effff700 1 -- 172.21.2.112:0/3902088383 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc6f018a720 msgr2=0x7fc6f00b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:12.185 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.193 7fc6effff700 1 --2- 172.21.2.112:0/3902088383 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc6f018a720 0x7fc6f00b7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:12.185 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.193 7fc6effff700 1 -- 172.21.2.112:0/3902088383 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6f00b8200 con 0x7fc6f00a0420 2021-02-10T11:30:12.185 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.193 7fc6effff700 1 --2- 172.21.2.112:0/3902088383 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc6f00a0420 0x7fc6f00a0820 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 rx=0x7fc6e0001430 tx=0x7fc6e000a930).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:12.185 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.193 7fc6ecff9700 1 -- 172.21.2.112:0/3902088383 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc6e0000990 con 0x7fc6f00a0420 2021-02-10T11:30:12.185 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.193 7fc6ecff9700 1 -- 172.21.2.112:0/3902088383 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6e0007510 con 0x7fc6f00a0420 2021-02-10T11:30:12.185 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.193 7fc6ecff9700 1 -- 172.21.2.112:0/3902088383 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc6e00043a0 con 0x7fc6f00a0420 2021-02-10T11:30:12.185 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.193 7fc6f6425700 1 -- 172.21.2.112:0/3902088383 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6f00b8200 con 0x7fc6f00a0420 2021-02-10T11:30:12.185 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.193 7fc6f6425700 1 -- 172.21.2.112:0/3902088383 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6f00b8790 con 0x7fc6f00a0420 2021-02-10T11:30:12.185 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.194 7fc6ecff9700 1 -- 172.21.2.112:0/3902088383 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc6e0000ce0 con 0x7fc6f00a0420 2021-02-10T11:30:12.185 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.194 7fc6ecff9700 1 --2- 172.21.2.112:0/3902088383 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc6d00186d0 0x7fc6d001abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:12.186 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.194 7fc6ecff9700 1 -- 172.21.2.112:0/3902088383 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fc6e0032660 con 0x7fc6f00a0420 2021-02-10T11:30:12.186 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.194 7fc6ef7fe700 1 --2- 172.21.2.112:0/3902088383 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc6d00186d0 0x7fc6d001abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:12.186 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.195 7fc6ef7fe700 1 --2- 172.21.2.112:0/3902088383 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc6d00186d0 0x7fc6d001abd0 crc :-1 s=READY pgs=393 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:12.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.224 7fc6f6425700 1 -- 172.21.2.112:0/3902088383 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6f00a1360 con 0x7fc6f00a0420 2021-02-10T11:30:12.214 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.227 7fc6ecff9700 1 -- 172.21.2.112:0/3902088383 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc6e00071c0 con 0x7fc6f00a0420 2021-02-10T11:30:12.342 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.354 7fc6f6425700 1 -- 172.21.2.112:0/3902088383 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348574", "target": ["mgr", ""]}) v1 -- 0x7fc6f00b9020 con 0x7fc6d00186d0 2021-02-10T11:30:12.355 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.368 7fc6ecff9700 1 -- 172.21.2.112:0/3902088383 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fc6e4006a70 con 0x7fc6d00186d0 2021-02-10T11:30:12.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.372 7fc6f6425700 1 -- 172.21.2.112:0/3902088383 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc6d00186d0 msgr2=0x7fc6d001abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:12.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.372 7fc6f6425700 1 --2- 172.21.2.112:0/3902088383 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc6d00186d0 0x7fc6d001abd0 crc :-1 s=READY pgs=393 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:12.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.372 7fc6f6425700 1 -- 172.21.2.112:0/3902088383 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc6f00a0420 msgr2=0x7fc6f00a0820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:12.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.372 7fc6f6425700 1 --2- 172.21.2.112:0/3902088383 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc6f00a0420 0x7fc6f00a0820 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 rx=0x7fc6e0001430 tx=0x7fc6e000a930).stop 2021-02-10T11:30:12.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.372 7fc6f6425700 1 -- 172.21.2.112:0/3902088383 shutdown_connections 2021-02-10T11:30:12.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.372 7fc6f6425700 1 --2- 172.21.2.112:0/3902088383 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc6d00186d0 0x7fc6d001abd0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:12.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.372 7fc6f6425700 1 --2- 172.21.2.112:0/3902088383 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc6f018a720 0x7fc6f00b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:12.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.372 7fc6f6425700 1 --2- 172.21.2.112:0/3902088383 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc6f0189d00 0x7fc6f018a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:12.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.372 7fc6f6425700 1 --2- 172.21.2.112:0/3902088383 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc6f00a0420 0x7fc6f00a0820 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:12.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.372 7fc6f6425700 1 -- 172.21.2.112:0/3902088383 >> 172.21.2.112:0/3902088383 conn(0x7fc6f0094e10 msgr2=0x7fc6f00a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:12.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.372 7fc6f6425700 1 -- 172.21.2.112:0/3902088383 shutdown_connections 2021-02-10T11:30:12.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.372 7fc6f6425700 1 -- 172.21.2.112:0/3902088383 wait complete. 2021-02-10T11:30:12.369 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000348575 2021-02-10T11:30:12.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.794 7ff30f106700 1 Processor -- start 2021-02-10T11:30:12.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.794 7ff30f106700 1 -- start start 2021-02-10T11:30:12.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.794 7ff30f106700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff3081b0470 0x7ff3081b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:12.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.795 7ff30f106700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7ff3080f20e0 0x7ff3080f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:12.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.795 7ff30f106700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3080f0b50 con 0x7ff3080ef610 2021-02-10T11:30:12.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.795 7ff30f106700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3080f0820 con 0x7ff3080f20e0 2021-02-10T11:30:12.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.795 7ff30f106700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3080f24e0 con 0x7ff3081b0470 2021-02-10T11:30:12.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.795 7ff30d723700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7ff3080f20e0 0x7ff3080f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:12.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.795 7ff30d723700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7ff3080f20e0 0x7ff3080f0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3300/0 says I am v2:172.21.2.112:48964/0 (socket says 172.21.2.112:48964) 2021-02-10T11:30:12.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.795 7ff30d723700 1 -- 172.21.2.112:0/3302073819 learned_addr learned my addr 172.21.2.112:0/3302073819 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:30:12.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.795 7ff30cf22700 1 --2- 172.21.2.112:0/3302073819 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff3081b0470 0x7ff3081b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:12.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.795 7ff30d723700 1 -- 172.21.2.112:0/3302073819 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff3081b0470 msgr2=0x7ff3081b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:12.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.795 7ff30d723700 1 --2- 172.21.2.112:0/3302073819 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff3081b0470 0x7ff3081b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:12.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.795 7ff30d723700 1 -- 172.21.2.112:0/3302073819 >> v1:172.21.2.120:6789/0 conn(0x7ff3080ef610 legacy=0x7ff3080efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:12.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.795 7ff30d723700 1 -- 172.21.2.112:0/3302073819 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3080e52e0 con 0x7ff3080f20e0 2021-02-10T11:30:12.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.795 7ff30d723700 1 --2- 172.21.2.112:0/3302073819 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff3080f20e0 0x7ff3080f0200 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 rx=0x7ff300006aa0 tx=0x7ff300009f20).ready entity=mon.1 client_cookie=70d4cf448395b348 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:12.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.796 7ff3077fe700 1 -- 172.21.2.112:0/3302073819 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff30000b5c0 con 0x7ff3080f20e0 2021-02-10T11:30:12.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.796 7ff3077fe700 1 -- 172.21.2.112:0/3302073819 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff300003030 con 0x7ff3080f20e0 2021-02-10T11:30:12.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.796 7ff3077fe700 1 -- 172.21.2.112:0/3302073819 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff30000bac0 con 0x7ff3080f20e0 2021-02-10T11:30:12.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.796 7ff30f106700 1 -- 172.21.2.112:0/3302073819 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff3080f20e0 msgr2=0x7ff3080f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:12.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.796 7ff30f106700 1 --2- 172.21.2.112:0/3302073819 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff3080f20e0 0x7ff3080f0200 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 rx=0x7ff300006aa0 tx=0x7ff300009f20).stop 2021-02-10T11:30:12.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.796 7ff30f106700 1 -- 172.21.2.112:0/3302073819 shutdown_connections 2021-02-10T11:30:12.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.796 7ff30f106700 1 --2- 172.21.2.112:0/3302073819 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff3080f20e0 0x7ff3080f0200 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:12.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.796 7ff30f106700 1 --2- 172.21.2.112:0/3302073819 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff3081b0470 0x7ff3081b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:12.800 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.796 7ff30f106700 1 -- 172.21.2.112:0/3302073819 >> 172.21.2.112:0/3302073819 conn(0x7ff3080ce3e0 msgr2=0x7ff3080e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:12.800 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.796 7ff30f106700 1 -- 172.21.2.112:0/3302073819 shutdown_connections 2021-02-10T11:30:12.800 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.796 7ff30f106700 1 -- 172.21.2.112:0/3302073819 wait complete. 2021-02-10T11:30:12.802 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.810 7ff30f106700 1 Processor -- start 2021-02-10T11:30:12.802 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.811 7ff30f106700 1 -- start start 2021-02-10T11:30:12.802 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.811 7ff30f106700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff30809b930 0x7ff3080b7c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:12.802 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.811 7ff30f106700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff3080b8410 0x7ff3080b8810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:12.802 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.811 7ff30f106700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff3080c1f80 0x7ff3080c2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:12.802 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.811 7ff30f106700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3080c1a00 con 0x7ff3080c1f80 2021-02-10T11:30:12.802 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.811 7ff30f106700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3080b8e30 con 0x7ff3080b8410 2021-02-10T11:30:12.802 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.811 7ff30f106700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3080b9050 con 0x7ff30809b930 2021-02-10T11:30:12.802 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.811 7ff307fff700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff3080b8410 0x7ff3080b8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:12.803 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.811 7ff307fff700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff3080b8410 0x7ff3080b8810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3300/0 says I am v2:172.21.2.112:48968/0 (socket says 172.21.2.112:48968) 2021-02-10T11:30:12.803 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.811 7ff307fff700 1 -- 172.21.2.112:0/3448043313 learned_addr learned my addr 172.21.2.112:0/3448043313 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:30:12.803 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.811 7ff30d723700 1 --2- 172.21.2.112:0/3448043313 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff3080c1f80 0x7ff3080c2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:12.803 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.811 7ff30cf22700 1 --2- 172.21.2.112:0/3448043313 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff30809b930 0x7ff3080b7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:12.803 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.811 7ff307fff700 1 -- 172.21.2.112:0/3448043313 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff30809b930 msgr2=0x7ff3080b7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:12.803 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.811 7ff307fff700 1 --2- 172.21.2.112:0/3448043313 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff30809b930 0x7ff3080b7c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:12.803 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.811 7ff307fff700 1 -- 172.21.2.112:0/3448043313 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff3080c1f80 msgr2=0x7ff3080c2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:12.803 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.812 7ff307fff700 1 --2- 172.21.2.112:0/3448043313 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff3080c1f80 0x7ff3080c2380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:12.803 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.812 7ff307fff700 1 -- 172.21.2.112:0/3448043313 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3080c29a0 con 0x7ff3080b8410 2021-02-10T11:30:12.803 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.812 7ff30cf22700 1 --2- 172.21.2.112:0/3448043313 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff30809b930 0x7ff3080b7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:30:12.803 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.812 7ff307fff700 1 --2- 172.21.2.112:0/3448043313 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff3080b8410 0x7ff3080b8810 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7ff2f8006f00 tx=0x7ff2f800b940).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:12.803 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.812 7ff3057fa700 1 -- 172.21.2.112:0/3448043313 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff2f8000960 con 0x7ff3080b8410 2021-02-10T11:30:12.803 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.812 7ff3057fa700 1 -- 172.21.2.112:0/3448043313 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2f8006340 con 0x7ff3080b8410 2021-02-10T11:30:12.804 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.812 7ff30f106700 1 -- 172.21.2.112:0/3448043313 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3080c29a0 con 0x7ff3080b8410 2021-02-10T11:30:12.804 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.812 7ff3057fa700 1 -- 172.21.2.112:0/3448043313 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff2f8000960 con 0x7ff3080b8410 2021-02-10T11:30:12.804 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.812 7ff30f106700 1 -- 172.21.2.112:0/3448043313 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3080c29a0 con 0x7ff3080b8410 2021-02-10T11:30:12.804 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.812 7ff3057fa700 1 -- 172.21.2.112:0/3448043313 <== mon.1 v2:172.21.2.120:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff2f80052e0 con 0x7ff3080b8410 2021-02-10T11:30:12.804 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.812 7ff3057fa700 1 --2- 172.21.2.112:0/3448043313 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff2ec0186d0 0x7ff2ec01abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:12.804 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.812 7ff3057fa700 1 -- 172.21.2.112:0/3448043313 <== mon.1 v2:172.21.2.120:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7ff2f8032600 con 0x7ff3080b8410 2021-02-10T11:30:12.804 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.813 7ff30cf22700 1 --2- 172.21.2.112:0/3448043313 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff2ec0186d0 0x7ff2ec01abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:12.804 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.813 7ff30cf22700 1 --2- 172.21.2.112:0/3448043313 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff2ec0186d0 0x7ff2ec01abd0 crc :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:12.831 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.844 7ff30f106700 1 -- 172.21.2.112:0/3448043313 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff308105b30 con 0x7ff3080b8410 2021-02-10T11:30:12.835 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.848 7ff3057fa700 1 -- 172.21.2.112:0/3448043313 <== mon.1 v2:172.21.2.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff2f8000960 con 0x7ff3080b8410 2021-02-10T11:30:12.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.977 7ff30f106700 1 -- 172.21.2.112:0/3448043313 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348575", "target": ["mgr", ""]}) v1 -- 0x7ff308105b30 con 0x7ff2ec0186d0 2021-02-10T11:30:12.977 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.989 7ff3057fa700 1 -- 172.21.2.112:0/3448043313 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7ff2f00009b0 con 0x7ff2ec0186d0 2021-02-10T11:30:12.983 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.995 7ff30f106700 1 -- 172.21.2.112:0/3448043313 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff2ec0186d0 msgr2=0x7ff2ec01abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:12.983 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.995 7ff30f106700 1 --2- 172.21.2.112:0/3448043313 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff2ec0186d0 0x7ff2ec01abd0 crc :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:12.983 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.995 7ff30f106700 1 -- 172.21.2.112:0/3448043313 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff3080b8410 msgr2=0x7ff3080b8810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:12.983 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.995 7ff30f106700 1 --2- 172.21.2.112:0/3448043313 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff3080b8410 0x7ff3080b8810 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7ff2f8006f00 tx=0x7ff2f800b940).stop 2021-02-10T11:30:12.983 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.995 7ff30f106700 1 -- 172.21.2.112:0/3448043313 shutdown_connections 2021-02-10T11:30:12.983 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.995 7ff30f106700 1 --2- 172.21.2.112:0/3448043313 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff2ec0186d0 0x7ff2ec01abd0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:12.983 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.995 7ff30f106700 1 --2- 172.21.2.112:0/3448043313 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff3080b8410 0x7ff3080b8810 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:12.983 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.995 7ff30f106700 1 --2- 172.21.2.112:0/3448043313 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff3080c1f80 0x7ff3080c2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:12.983 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.995 7ff30f106700 1 --2- 172.21.2.112:0/3448043313 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff30809b930 0x7ff3080b7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:12.983 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.995 7ff30f106700 1 -- 172.21.2.112:0/3448043313 >> 172.21.2.112:0/3448043313 conn(0x7ff308094e10 msgr2=0x7ff3080a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:12.983 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.995 7ff30f106700 1 -- 172.21.2.112:0/3448043313 shutdown_connections 2021-02-10T11:30:12.983 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:12.995 7ff30f106700 1 -- 172.21.2.112:0/3448043313 wait complete. 2021-02-10T11:30:12.991 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000348576 2021-02-10T11:30:13.421 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:13.417 7f9373129700 1 Processor -- start 2021-02-10T11:30:13.421 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:13.417 7f9373129700 1 -- start start 2021-02-10T11:30:13.421 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:13.417 7f9373129700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f936c1b0470 0x7f936c1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:13.421 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:13.417 7f9373129700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f936c0ef610 0x7f936c0efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:13.421 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:13.417 7f9373129700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f936c0f0030 0x7f936c0f0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:13.422 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:13.417 7f9373129700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f936c0e5ea0 con 0x7f936c0f0030 2021-02-10T11:30:13.422 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:13.417 7f9373129700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f936c0f0b50 con 0x7f936c0ef610 2021-02-10T11:30:13.422 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:13.417 7f9373129700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f936c0e52e0 con 0x7f936c1b0470 2021-02-10T11:30:13.422 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:13.418 7f9371746700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f936c0f0030 0x7f936c0f0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:13.422 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:13.418 7f9371746700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f936c0f0030 0x7f936c0f0430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:46694/0 (socket says 172.21.2.112:46694) 2021-02-10T11:30:13.422 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:13.418 7f9370f45700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f936c1b0470 0x7f936c1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:13.422 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:13.418 7f9371746700 1 -- 172.21.2.112:0/1752203126 learned_addr learned my addr 172.21.2.112:0/1752203126 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:30:13.422 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:13.418 7f936bfff700 1 --2- 172.21.2.112:0/1752203126 >> v2:172.21.2.120:3300/0 conn(0x7f936c0ef610 0x7f936c0efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:13.422 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:13.418 7f9370f45700 1 -- 172.21.2.112:0/1752203126 >> v2:172.21.2.120:3300/0 conn(0x7f936c0ef610 msgr2=0x7f936c0efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:13.422 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:13.418 7f9370f45700 1 --2- 172.21.2.112:0/1752203126 >> v2:172.21.2.120:3300/0 conn(0x7f936c0ef610 0x7f936c0efa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:13.422 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:13.418 7f9370f45700 1 -- 172.21.2.112:0/1752203126 >> v2:172.21.2.112:3300/0 conn(0x7f936c0f0030 msgr2=0x7f936c0f0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:13.422 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:13.418 7f9370f45700 1 --2- 172.21.2.112:0/1752203126 >> v2:172.21.2.112:3300/0 conn(0x7f936c0f0030 0x7f936c0f0430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:13.422 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:13.418 7f9370f45700 1 -- 172.21.2.112:0/1752203126 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f936c0f10d0 con 0x7f936c1b0470 2021-02-10T11:30:13.423 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:13.418 7f9371746700 1 --2- 172.21.2.112:0/1752203126 >> v2:172.21.2.112:3300/0 conn(0x7f936c0f0030 0x7f936c0f0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:30:13.423 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:13.418 7f9370f45700 1 --2- 172.21.2.112:0/1752203126 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f936c1b0470 0x7f936c1b0870 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f93640069d0 tx=0x7f9364009dc0).ready entity=mon.2 client_cookie=93b0f8b6c06fecf0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:13.423 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:13.418 7f936b7fe700 1 -- 172.21.2.112:0/1752203126 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f93640066c0 con 0x7f936c1b0470 2021-02-10T11:30:13.423 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:13.419 7f936b7fe700 1 -- 172.21.2.112:0/1752203126 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f936400b510 con 0x7f936c1b0470 2021-02-10T11:30:13.423 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:13.419 7f936b7fe700 1 -- 172.21.2.112:0/1752203126 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f936400b970 con 0x7f936c1b0470 2021-02-10T11:30:13.423 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:13.419 7f9373129700 1 -- 172.21.2.112:0/1752203126 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f936c1b0470 msgr2=0x7f936c1b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:13.423 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:13.419 7f9373129700 1 --2- 172.21.2.112:0/1752203126 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f936c1b0470 0x7f936c1b0870 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f93640069d0 tx=0x7f9364009dc0).stop 2021-02-10T11:30:13.423 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:13.419 7f9373129700 1 -- 172.21.2.112:0/1752203126 shutdown_connections 2021-02-10T11:30:13.423 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:13.419 7f9373129700 1 --2- 172.21.2.112:0/1752203126 >> v2:172.21.2.112:3300/0 conn(0x7f936c0f0030 0x7f936c0f0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:13.423 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:13.419 7f9373129700 1 --2- 172.21.2.112:0/1752203126 >> v2:172.21.2.120:3300/0 conn(0x7f936c0ef610 0x7f936c0efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:13.423 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:13.419 7f9373129700 1 --2- 172.21.2.112:0/1752203126 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f936c1b0470 0x7f936c1b0870 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:13.423 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:13.419 7f9373129700 1 -- 172.21.2.112:0/1752203126 >> 172.21.2.112:0/1752203126 conn(0x7f936c0ce3e0 msgr2=0x7f936c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:13.423 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:13.419 7f9373129700 1 -- 172.21.2.112:0/1752203126 shutdown_connections 2021-02-10T11:30:13.424 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:13.419 7f9373129700 1 -- 172.21.2.112:0/1752203126 wait complete. 2021-02-10T11:30:13.425 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:13.434 7f9373129700 1 Processor -- start 2021-02-10T11:30:13.425 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:13.434 7f9373129700 1 -- start start 2021-02-10T11:30:13.425 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:13.434 7f9373129700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f936c09e560 0x7f936c0a0130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:13.425 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:13.434 7f9373129700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f936c09e960 0x7f936c0a0750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:13.425 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:13.434 7f9373129700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f936c0a0d70 0x7f936c0a1af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:13.425 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:13.434 7f9373129700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f936c0c38f0 con 0x7f936c0a0d70 2021-02-10T11:30:13.426 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:13.434 7f9373129700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f936c0a1170 con 0x7f936c09e560 2021-02-10T11:30:13.426 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:13.434 7f9373129700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f936c0a1390 con 0x7f936c09e960 2021-02-10T11:30:13.426 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:13.435 7f9370f45700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f936c09e560 0x7f936c0a0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:13.426 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:13.435 7f9371746700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f936c0a0d70 0x7f936c0a1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:13.426 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:13.435 7f9370f45700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f936c09e560 0x7f936c0a0130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3300/0 says I am v2:172.21.2.112:48976/0 (socket says 172.21.2.112:48976) 2021-02-10T11:30:13.426 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:13.435 7f9370f45700 1 -- 172.21.2.112:0/4281867426 learned_addr learned my addr 172.21.2.112:0/4281867426 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:30:13.426 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:13.435 7f9371746700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f936c0a0d70 0x7f936c0a1af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:46696/0 (socket says 172.21.2.112:46696) 2021-02-10T11:30:13.426 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:13.435 7f936bfff700 1 --2- 172.21.2.112:0/4281867426 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f936c09e960 0x7f936c0a0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:13.426 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:13.435 7f9370f45700 1 -- 172.21.2.112:0/4281867426 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f936c09e960 msgr2=0x7f936c0a0750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:13.426 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:13.435 7f9370f45700 1 --2- 172.21.2.112:0/4281867426 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f936c09e960 0x7f936c0a0750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:13.427 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:13.435 7f9370f45700 1 -- 172.21.2.112:0/4281867426 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f936c0a0d70 msgr2=0x7f936c0a1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:13.427 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:13.435 7f9370f45700 1 --2- 172.21.2.112:0/4281867426 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f936c0a0d70 0x7f936c0a1af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:13.427 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:13.435 7f9370f45700 1 -- 172.21.2.112:0/4281867426 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f936c0a2110 con 0x7f936c09e560 2021-02-10T11:30:13.427 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:13.435 7f9370f45700 1 --2- 172.21.2.112:0/4281867426 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f936c09e560 0x7f936c0a0130 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0x7f9364009390 tx=0x7f936400bd10).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:13.427 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:13.435 7f93697fa700 1 -- 172.21.2.112:0/4281867426 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9364000d30 con 0x7f936c09e560 2021-02-10T11:30:13.427 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:13.436 7f93697fa700 1 -- 172.21.2.112:0/4281867426 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9364006020 con 0x7f936c09e560 2021-02-10T11:30:13.427 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:13.436 7f9373129700 1 -- 172.21.2.112:0/4281867426 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f936c0a2110 con 0x7f936c09e560 2021-02-10T11:30:13.427 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:13.436 7f93697fa700 1 -- 172.21.2.112:0/4281867426 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9364006470 con 0x7f936c09e560 2021-02-10T11:30:13.427 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:13.436 7f9373129700 1 -- 172.21.2.112:0/4281867426 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f936c0a2110 con 0x7f936c09e560 2021-02-10T11:30:13.427 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:13.436 7f93697fa700 1 -- 172.21.2.112:0/4281867426 <== mon.1 v2:172.21.2.120:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9364005660 con 0x7f936c09e560 2021-02-10T11:30:13.427 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:13.436 7f93697fa700 1 --2- 172.21.2.112:0/4281867426 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9350018730 0x7f935001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:13.427 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:13.436 7f93697fa700 1 -- 172.21.2.112:0/4281867426 <== mon.1 v2:172.21.2.120:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f93640329f0 con 0x7f936c09e560 2021-02-10T11:30:13.428 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:13.437 7f936bfff700 1 --2- 172.21.2.112:0/4281867426 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9350018730 0x7f935001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:13.428 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:13.438 7f936bfff700 1 --2- 172.21.2.112:0/4281867426 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9350018730 0x7f935001abc0 crc :-1 s=READY pgs=395 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:13.454 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:13.467 7f9373129700 1 -- 172.21.2.112:0/4281867426 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f936c105b30 con 0x7f936c09e560 2021-02-10T11:30:13.457 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:13.470 7f93697fa700 1 -- 172.21.2.112:0/4281867426 <== mon.1 v2:172.21.2.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9364006260 con 0x7f936c09e560 2021-02-10T11:30:13.583 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:13.595 7f9373129700 1 -- 172.21.2.112:0/4281867426 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348576", "target": ["mgr", ""]}) v1 -- 0x7f936c0f0b50 con 0x7f9350018730 2021-02-10T11:30:13.593 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:13.606 7f93697fa700 1 -- 172.21.2.112:0/4281867426 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f935c006a70 con 0x7f9350018730 2021-02-10T11:30:13.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:13.613 7f9373129700 1 -- 172.21.2.112:0/4281867426 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9350018730 msgr2=0x7f935001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:13.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:13.613 7f9373129700 1 --2- 172.21.2.112:0/4281867426 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9350018730 0x7f935001abc0 crc :-1 s=READY pgs=395 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:13.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:13.613 7f9373129700 1 -- 172.21.2.112:0/4281867426 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f936c09e560 msgr2=0x7f936c0a0130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:13.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:13.613 7f9373129700 1 --2- 172.21.2.112:0/4281867426 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f936c09e560 0x7f936c0a0130 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0x7f9364009390 tx=0x7f936400bd10).stop 2021-02-10T11:30:13.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:13.614 7f9373129700 1 -- 172.21.2.112:0/4281867426 shutdown_connections 2021-02-10T11:30:13.602 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:13.614 7f9373129700 1 --2- 172.21.2.112:0/4281867426 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9350018730 0x7f935001abc0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:13.602 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:13.614 7f9373129700 1 --2- 172.21.2.112:0/4281867426 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f936c0a0d70 0x7f936c0a1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:13.602 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:13.614 7f9373129700 1 --2- 172.21.2.112:0/4281867426 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f936c09e960 0x7f936c0a0750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:13.602 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:13.614 7f9373129700 1 --2- 172.21.2.112:0/4281867426 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f936c09e560 0x7f936c0a0130 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:13.602 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:13.614 7f9373129700 1 -- 172.21.2.112:0/4281867426 >> 172.21.2.112:0/4281867426 conn(0x7f936c09fc10 msgr2=0x7f936c094ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:13.602 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:13.614 7f9373129700 1 -- 172.21.2.112:0/4281867426 shutdown_connections 2021-02-10T11:30:13.602 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:13.614 7f9373129700 1 -- 172.21.2.112:0/4281867426 wait complete. 2021-02-10T11:30:13.605 DEBUG:teuthology.orchestra.run.gibba012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:30:13.607 DEBUG:teuthology.orchestra.run.gibba020:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:30:13.610 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000348577 2021-02-10T11:30:14.038 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.033 7f8f9677d700 1 Processor -- start 2021-02-10T11:30:14.038 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.034 7f8f9677d700 1 -- start start 2021-02-10T11:30:14.038 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.034 7f8f9677d700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f8f901b0470 0x7f8f901b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:14.038 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.034 7f8f9677d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f8f900f20e0 0x7f8f900f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:14.038 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.034 7f8f9677d700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f900f0b50 con 0x7f8f900ef610 2021-02-10T11:30:14.038 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.034 7f8f9677d700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f900f0820 con 0x7f8f900f20e0 2021-02-10T11:30:14.038 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.034 7f8f9677d700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f900f24e0 con 0x7f8f901b0470 2021-02-10T11:30:14.038 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.034 7f8f94d9a700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f8f900f20e0 0x7f8f900f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:14.038 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.034 7f8f94d9a700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f8f900f20e0 0x7f8f900f0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:46706/0 (socket says 172.21.2.112:46706) 2021-02-10T11:30:14.038 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.034 7f8f8f7fe700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7f8f900ef610 0x7f8f900efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:39724/0 (socket says 172.21.2.112:39724) 2021-02-10T11:30:14.038 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.034 7f8f94d9a700 1 -- 172.21.2.112:0/1829349816 learned_addr learned my addr 172.21.2.112:0/1829349816 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:30:14.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.034 7f8f8ffff700 1 --2- 172.21.2.112:0/1829349816 >> v2:172.21.2.120:3300/0 conn(0x7f8f901b0470 0x7f8f901b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:14.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.034 7f8f94d9a700 1 -- 172.21.2.112:0/1829349816 >> v2:172.21.2.120:3300/0 conn(0x7f8f901b0470 msgr2=0x7f8f901b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:14.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.034 7f8f94d9a700 1 --2- 172.21.2.112:0/1829349816 >> v2:172.21.2.120:3300/0 conn(0x7f8f901b0470 0x7f8f901b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:14.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.034 7f8f94d9a700 1 -- 172.21.2.112:0/1829349816 >> v1:172.21.2.120:6789/0 conn(0x7f8f900ef610 legacy=0x7f8f900efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:14.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.034 7f8f94d9a700 1 -- 172.21.2.112:0/1829349816 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f900e52e0 con 0x7f8f900f20e0 2021-02-10T11:30:14.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.035 7f8f94d9a700 1 --2- 172.21.2.112:0/1829349816 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8f900f20e0 0x7f8f900f0200 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 rx=0x7f8f80006b80 tx=0x7f8f80009b90).ready entity=mon.0 client_cookie=19c486cd83ca7bfc server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:14.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.035 7f8f8effd700 1 -- 172.21.2.112:0/1829349816 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8f800022e0 con 0x7f8f900f20e0 2021-02-10T11:30:14.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.035 7f8f8effd700 1 -- 172.21.2.112:0/1829349816 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f80002620 con 0x7f8f900f20e0 2021-02-10T11:30:14.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.035 7f8f8effd700 1 -- 172.21.2.112:0/1829349816 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8f80002b00 con 0x7f8f900f20e0 2021-02-10T11:30:14.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.035 7f8f9677d700 1 -- 172.21.2.112:0/1829349816 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8f900f20e0 msgr2=0x7f8f900f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:14.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.035 7f8f9677d700 1 --2- 172.21.2.112:0/1829349816 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8f900f20e0 0x7f8f900f0200 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 rx=0x7f8f80006b80 tx=0x7f8f80009b90).stop 2021-02-10T11:30:14.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.036 7f8f9677d700 1 -- 172.21.2.112:0/1829349816 shutdown_connections 2021-02-10T11:30:14.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.036 7f8f9677d700 1 --2- 172.21.2.112:0/1829349816 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8f900f20e0 0x7f8f900f0200 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:14.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.036 7f8f9677d700 1 --2- 172.21.2.112:0/1829349816 >> v2:172.21.2.120:3300/0 conn(0x7f8f901b0470 0x7f8f901b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:14.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.036 7f8f9677d700 1 -- 172.21.2.112:0/1829349816 >> 172.21.2.112:0/1829349816 conn(0x7f8f900ce3e0 msgr2=0x7f8f900e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:14.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.036 7f8f9677d700 1 -- 172.21.2.112:0/1829349816 shutdown_connections 2021-02-10T11:30:14.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.036 7f8f9677d700 1 -- 172.21.2.112:0/1829349816 wait complete. 2021-02-10T11:30:14.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.051 7f8f9677d700 1 Processor -- start 2021-02-10T11:30:14.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.051 7f8f9677d700 1 -- start start 2021-02-10T11:30:14.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.051 7f8f9677d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8f900b20b0 0x7f8f900ac970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:14.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.051 7f8f9677d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8f900b24b0 0x7f8f90191dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:14.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.051 7f8f9677d700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8f900acf90 0x7f8f901923e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:14.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.051 7f8f9677d700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f900b1b30 con 0x7f8f900b20b0 2021-02-10T11:30:14.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.051 7f8f9677d700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f901a04e0 con 0x7f8f900acf90 2021-02-10T11:30:14.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.051 7f8f9677d700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f901a06c0 con 0x7f8f900b24b0 2021-02-10T11:30:14.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.051 7f8f8ffff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8f900b20b0 0x7f8f900ac970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:14.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.051 7f8f8ffff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8f900b20b0 0x7f8f900ac970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:46708/0 (socket says 172.21.2.112:46708) 2021-02-10T11:30:14.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.051 7f8f8ffff700 1 -- 172.21.2.112:0/3848344211 learned_addr learned my addr 172.21.2.112:0/3848344211 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:30:14.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.051 7f8f94d9a700 1 --2- 172.21.2.112:0/3848344211 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8f900acf90 0x7f8f901923e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:14.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.052 7f8f8f7fe700 1 --2- 172.21.2.112:0/3848344211 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8f900b24b0 0x7f8f90191dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:14.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.052 7f8f8ffff700 1 -- 172.21.2.112:0/3848344211 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8f900b24b0 msgr2=0x7f8f90191dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:14.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.052 7f8f8ffff700 1 --2- 172.21.2.112:0/3848344211 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8f900b24b0 0x7f8f90191dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:14.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.052 7f8f8ffff700 1 -- 172.21.2.112:0/3848344211 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8f900acf90 msgr2=0x7f8f901923e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:14.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.052 7f8f8ffff700 1 --2- 172.21.2.112:0/3848344211 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8f900acf90 0x7f8f901923e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:14.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.052 7f8f8ffff700 1 -- 172.21.2.112:0/3848344211 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f90192be0 con 0x7f8f900b20b0 2021-02-10T11:30:14.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.052 7f8f8ffff700 1 --2- 172.21.2.112:0/3848344211 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8f900b20b0 0x7f8f900ac970 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 rx=0x7f8f88009980 tx=0x7f8f8800bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:14.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.052 7f8f8cff9700 1 -- 172.21.2.112:0/3848344211 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8f88006030 con 0x7f8f900b20b0 2021-02-10T11:30:14.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.052 7f8f8cff9700 1 -- 172.21.2.112:0/3848344211 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f880062f0 con 0x7f8f900b20b0 2021-02-10T11:30:14.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.052 7f8f9677d700 1 -- 172.21.2.112:0/3848344211 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f90192be0 con 0x7f8f900b20b0 2021-02-10T11:30:14.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.052 7f8f9677d700 1 -- 172.21.2.112:0/3848344211 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f90192be0 con 0x7f8f900b20b0 2021-02-10T11:30:14.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.052 7f8f8cff9700 1 -- 172.21.2.112:0/3848344211 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8f88005660 con 0x7f8f900b20b0 2021-02-10T11:30:14.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.053 7f8f8cff9700 1 -- 172.21.2.112:0/3848344211 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8f880064d0 con 0x7f8f900b20b0 2021-02-10T11:30:14.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.053 7f8f8cff9700 1 --2- 172.21.2.112:0/3848344211 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8f70018730 0x7f8f7001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:14.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.053 7f8f8cff9700 1 -- 172.21.2.112:0/3848344211 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f8f880329f0 con 0x7f8f900b20b0 2021-02-10T11:30:14.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.053 7f8f8f7fe700 1 --2- 172.21.2.112:0/3848344211 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8f70018730 0x7f8f7001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:14.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.053 7f8f8f7fe700 1 --2- 172.21.2.112:0/3848344211 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8f70018730 0x7f8f7001ac10 crc :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:14.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.083 7f8f9677d700 1 -- 172.21.2.112:0/3848344211 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f90105af0 con 0x7f8f900b20b0 2021-02-10T11:30:14.073 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.086 7f8f8cff9700 1 -- 172.21.2.112:0/3848344211 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8f8800be10 con 0x7f8f900b20b0 2021-02-10T11:30:14.202 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.215 7f8f9677d700 1 -- 172.21.2.112:0/3848344211 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348577", "target": ["mgr", ""]}) v1 -- 0x7f8f900f0b50 con 0x7f8f70018730 2021-02-10T11:30:14.213 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.226 7f8f8cff9700 1 -- 172.21.2.112:0/3848344211 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f8f84007cc0 con 0x7f8f70018730 2021-02-10T11:30:14.219 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.232 7f8f9677d700 1 -- 172.21.2.112:0/3848344211 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8f70018730 msgr2=0x7f8f7001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:14.220 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.232 7f8f9677d700 1 --2- 172.21.2.112:0/3848344211 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8f70018730 0x7f8f7001ac10 crc :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:14.220 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.232 7f8f9677d700 1 -- 172.21.2.112:0/3848344211 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8f900b20b0 msgr2=0x7f8f900ac970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:14.220 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.232 7f8f9677d700 1 --2- 172.21.2.112:0/3848344211 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8f900b20b0 0x7f8f900ac970 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 rx=0x7f8f88009980 tx=0x7f8f8800bcd0).stop 2021-02-10T11:30:14.220 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.232 7f8f9677d700 1 -- 172.21.2.112:0/3848344211 shutdown_connections 2021-02-10T11:30:14.220 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.232 7f8f9677d700 1 --2- 172.21.2.112:0/3848344211 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8f70018730 0x7f8f7001ac10 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:14.220 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.232 7f8f9677d700 1 --2- 172.21.2.112:0/3848344211 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8f900acf90 0x7f8f901923e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:14.220 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.232 7f8f9677d700 1 --2- 172.21.2.112:0/3848344211 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8f900b24b0 0x7f8f90191dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:14.220 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.232 7f8f9677d700 1 --2- 172.21.2.112:0/3848344211 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8f900b20b0 0x7f8f900ac970 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:14.220 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.232 7f8f9677d700 1 -- 172.21.2.112:0/3848344211 >> 172.21.2.112:0/3848344211 conn(0x7f8f900bf4e0 msgr2=0x7f8f900be3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:14.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.232 7f8f9677d700 1 -- 172.21.2.112:0/3848344211 shutdown_connections 2021-02-10T11:30:14.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.232 7f8f9677d700 1 -- 172.21.2.112:0/3848344211 wait complete. 2021-02-10T11:30:14.229 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000348578 2021-02-10T11:30:14.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.692 7f00df7d5700 1 Processor -- start 2021-02-10T11:30:14.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.692 7f00df7d5700 1 -- start start 2021-02-10T11:30:14.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.693 7f00df7d5700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f00d81b0470 0x7f00d81b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:14.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.693 7f00df7d5700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f00d80f20e0 0x7f00d80f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:14.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.693 7f00df7d5700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00d80f0b50 con 0x7f00d80ef610 2021-02-10T11:30:14.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.693 7f00df7d5700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00d80f0820 con 0x7f00d80f20e0 2021-02-10T11:30:14.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.693 7f00df7d5700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00d80f24e0 con 0x7f00d81b0470 2021-02-10T11:30:14.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.693 7f00dcdf0700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f00d80ef610 0x7f00d80efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:46178/0 (socket says 172.21.2.112:46178) 2021-02-10T11:30:14.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.693 7f00dcdf0700 1 -- 172.21.2.112:0/2784494557 learned_addr learned my addr 172.21.2.112:0/2784494557 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:30:14.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.693 7f00dddf2700 1 --2- 172.21.2.112:0/2784494557 >> v2:172.21.2.120:3300/0 conn(0x7f00d80f20e0 0x7f00d80f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:14.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.693 7f00dd5f1700 1 --2- 172.21.2.112:0/2784494557 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f00d81b0470 0x7f00d81b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:14.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.693 7f00cbfff700 1 -- 172.21.2.112:0/2784494557 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1174061544 0 0) 0x7f00cc0010b0 con 0x7f00d80ef610 2021-02-10T11:30:14.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.694 7f00cbfff700 1 -- 172.21.2.112:0/2784494557 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00c4002fe0 con 0x7f00d80ef610 2021-02-10T11:30:14.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.694 7f00dd5f1700 1 -- 172.21.2.112:0/2784494557 >> v2:172.21.2.120:3300/0 conn(0x7f00d80f20e0 msgr2=0x7f00d80f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:14.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.694 7f00dd5f1700 1 --2- 172.21.2.112:0/2784494557 >> v2:172.21.2.120:3300/0 conn(0x7f00d80f20e0 0x7f00d80f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:14.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.694 7f00dd5f1700 1 -- 172.21.2.112:0/2784494557 >> v1:172.21.2.112:6789/0 conn(0x7f00d80ef610 legacy=0x7f00d80efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:14.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.694 7f00dd5f1700 1 -- 172.21.2.112:0/2784494557 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00d80e52e0 con 0x7f00d81b0470 2021-02-10T11:30:14.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.694 7f00dd5f1700 1 --2- 172.21.2.112:0/2784494557 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f00d81b0470 0x7f00d81b0870 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f00d4006650 tx=0x7f00d4009a80).ready entity=mon.2 client_cookie=4f28cf3b43d8b17f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:14.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.694 7f00cbfff700 1 -- 172.21.2.112:0/2784494557 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f00d400c0c0 con 0x7f00d81b0470 2021-02-10T11:30:14.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.694 7f00cbfff700 1 -- 172.21.2.112:0/2784494557 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00d400c440 con 0x7f00d81b0470 2021-02-10T11:30:14.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.694 7f00cbfff700 1 -- 172.21.2.112:0/2784494557 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f00d400c8a0 con 0x7f00d81b0470 2021-02-10T11:30:14.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.694 7f00df7d5700 1 -- 172.21.2.112:0/2784494557 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f00d81b0470 msgr2=0x7f00d81b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:14.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.694 7f00df7d5700 1 --2- 172.21.2.112:0/2784494557 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f00d81b0470 0x7f00d81b0870 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f00d4006650 tx=0x7f00d4009a80).stop 2021-02-10T11:30:14.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.695 7f00df7d5700 1 -- 172.21.2.112:0/2784494557 shutdown_connections 2021-02-10T11:30:14.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.695 7f00df7d5700 1 --2- 172.21.2.112:0/2784494557 >> v2:172.21.2.120:3300/0 conn(0x7f00d80f20e0 0x7f00d80f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:14.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.695 7f00df7d5700 1 --2- 172.21.2.112:0/2784494557 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f00d81b0470 0x7f00d81b0870 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:14.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.695 7f00df7d5700 1 -- 172.21.2.112:0/2784494557 >> 172.21.2.112:0/2784494557 conn(0x7f00d80ce3e0 msgr2=0x7f00d80e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:14.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.695 7f00df7d5700 1 -- 172.21.2.112:0/2784494557 shutdown_connections 2021-02-10T11:30:14.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.695 7f00df7d5700 1 -- 172.21.2.112:0/2784494557 wait complete. 2021-02-10T11:30:14.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.710 7f00df7d5700 1 Processor -- start 2021-02-10T11:30:14.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.710 7f00df7d5700 1 -- start start 2021-02-10T11:30:14.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.710 7f00df7d5700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f00d80b7db0 0x7f00d80b8260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:14.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.710 7f00df7d5700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f00d80b8880 0x7f00d81a7450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:14.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.710 7f00df7d5700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f00d80b8c80 0x7f00d81a7a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:14.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.710 7f00df7d5700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00d80b79b0 con 0x7f00d80b8c80 2021-02-10T11:30:14.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.710 7f00df7d5700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00d81a8090 con 0x7f00d80b7db0 2021-02-10T11:30:14.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.710 7f00df7d5700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00d81a82b0 con 0x7f00d80b8880 2021-02-10T11:30:14.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.710 7f00dddf2700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f00d80b8c80 0x7f00d81a7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:14.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.710 7f00dddf2700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f00d80b8c80 0x7f00d81a7a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:46720/0 (socket says 172.21.2.112:46720) 2021-02-10T11:30:14.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.710 7f00dddf2700 1 -- 172.21.2.112:0/818385983 learned_addr learned my addr 172.21.2.112:0/818385983 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:30:14.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.711 7f00dddf2700 1 -- 172.21.2.112:0/818385983 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f00d80b8880 msgr2=0x7f00d81a7450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:14.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.711 7f00dddf2700 1 --2- 172.21.2.112:0/818385983 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f00d80b8880 0x7f00d81a7450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:14.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.711 7f00dddf2700 1 -- 172.21.2.112:0/818385983 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f00d80b7db0 msgr2=0x7f00d80b8260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:14.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.711 7f00dd5f1700 1 --2- 172.21.2.112:0/818385983 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f00d80b7db0 0x7f00d80b8260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:14.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.711 7f00dddf2700 1 --2- 172.21.2.112:0/818385983 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f00d80b7db0 0x7f00d80b8260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:14.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.711 7f00dddf2700 1 -- 172.21.2.112:0/818385983 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00d8190c10 con 0x7f00d80b8c80 2021-02-10T11:30:14.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.711 7f00dd5f1700 1 --2- 172.21.2.112:0/818385983 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f00d80b7db0 0x7f00d80b8260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:30:14.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.711 7f00dddf2700 1 --2- 172.21.2.112:0/818385983 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f00d80b8c80 0x7f00d81a7a70 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 rx=0x7f00d000b910 tx=0x7f00d000bcb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:14.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.711 7f00c9ffb700 1 -- 172.21.2.112:0/818385983 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f00d0006aa0 con 0x7f00d80b8c80 2021-02-10T11:30:14.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.711 7f00c9ffb700 1 -- 172.21.2.112:0/818385983 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00d00011f0 con 0x7f00d80b8c80 2021-02-10T11:30:14.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.711 7f00df7d5700 1 -- 172.21.2.112:0/818385983 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00d8190c10 con 0x7f00d80b8c80 2021-02-10T11:30:14.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.711 7f00df7d5700 1 -- 172.21.2.112:0/818385983 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00d80c1eb0 con 0x7f00d80b8c80 2021-02-10T11:30:14.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.711 7f00c9ffb700 1 -- 172.21.2.112:0/818385983 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f00d0001710 con 0x7f00d80b8c80 2021-02-10T11:30:14.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.712 7f00c9ffb700 1 -- 172.21.2.112:0/818385983 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f00d00056b0 con 0x7f00d80b8c80 2021-02-10T11:30:14.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.712 7f00c9ffb700 1 --2- 172.21.2.112:0/818385983 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f00b8018700 0x7f00b801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:14.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.712 7f00c9ffb700 1 -- 172.21.2.112:0/818385983 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f00d00329b0 con 0x7f00d80b8c80 2021-02-10T11:30:14.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.712 7f00dd5f1700 1 --2- 172.21.2.112:0/818385983 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f00b8018700 0x7f00b801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:14.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.713 7f00dd5f1700 1 --2- 172.21.2.112:0/818385983 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f00b8018700 0x7f00b801abc0 crc :-1 s=READY pgs=397 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:14.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.742 7f00df7d5700 1 -- 172.21.2.112:0/818385983 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00d8105b30 con 0x7f00d80b8c80 2021-02-10T11:30:14.732 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.745 7f00c9ffb700 1 -- 172.21.2.112:0/818385983 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f00d0000f40 con 0x7f00d80b8c80 2021-02-10T11:30:14.860 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.872 7f00df7d5700 1 -- 172.21.2.112:0/818385983 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348578", "target": ["mgr", ""]}) v1 -- 0x7f00d80c1eb0 con 0x7f00b8018700 2021-02-10T11:30:14.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.882 7f00c9ffb700 1 -- 172.21.2.112:0/818385983 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f00d40052e0 con 0x7f00b8018700 2021-02-10T11:30:14.876 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.889 7f00df7d5700 1 -- 172.21.2.112:0/818385983 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f00b8018700 msgr2=0x7f00b801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:14.876 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.889 7f00df7d5700 1 --2- 172.21.2.112:0/818385983 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f00b8018700 0x7f00b801abc0 crc :-1 s=READY pgs=397 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:14.876 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.889 7f00df7d5700 1 -- 172.21.2.112:0/818385983 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f00d80b8c80 msgr2=0x7f00d81a7a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:14.876 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.889 7f00df7d5700 1 --2- 172.21.2.112:0/818385983 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f00d80b8c80 0x7f00d81a7a70 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 rx=0x7f00d000b910 tx=0x7f00d000bcb0).stop 2021-02-10T11:30:14.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.889 7f00df7d5700 1 -- 172.21.2.112:0/818385983 shutdown_connections 2021-02-10T11:30:14.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.889 7f00df7d5700 1 --2- 172.21.2.112:0/818385983 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f00b8018700 0x7f00b801abc0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:14.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.889 7f00df7d5700 1 --2- 172.21.2.112:0/818385983 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f00d80b8c80 0x7f00d81a7a70 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:14.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.889 7f00df7d5700 1 --2- 172.21.2.112:0/818385983 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f00d80b8880 0x7f00d81a7450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:14.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.889 7f00df7d5700 1 --2- 172.21.2.112:0/818385983 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f00d80b7db0 0x7f00d80b8260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:14.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.889 7f00df7d5700 1 -- 172.21.2.112:0/818385983 >> 172.21.2.112:0/818385983 conn(0x7f00d8094e10 msgr2=0x7f00d80a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:14.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.889 7f00df7d5700 1 -- 172.21.2.112:0/818385983 shutdown_connections 2021-02-10T11:30:14.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:14.889 7f00df7d5700 1 -- 172.21.2.112:0/818385983 wait complete. 2021-02-10T11:30:14.885 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000348579 2021-02-10T11:30:15.315 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.311 7fd8b813a700 1 Processor -- start 2021-02-10T11:30:15.315 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.311 7fd8b813a700 1 -- start start 2021-02-10T11:30:15.315 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.311 7fd8b813a700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fd8b01b0870 0x7fd8b00f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:15.315 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.311 7fd8b813a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd8b00efe00 0x7fd8b00f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:15.316 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.311 7fd8b813a700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8b01b0c70 con 0x7fd8b01b0470 2021-02-10T11:30:15.316 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.311 7fd8b813a700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8b00f0b50 con 0x7fd8b01b0870 2021-02-10T11:30:15.316 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.311 7fd8b813a700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b00f0820 con 0x7fd8b00efe00 2021-02-10T11:30:15.316 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.311 7fd8b5755700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fd8b01b0870 0x7fd8b00f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:15.316 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.312 7fd8b5755700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fd8b01b0870 0x7fd8b00f20e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:46726/0 (socket says 172.21.2.112:46726) 2021-02-10T11:30:15.316 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.312 7fd8b5755700 1 -- 172.21.2.112:0/954489146 learned_addr learned my addr 172.21.2.112:0/954489146 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:30:15.316 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.312 7fd8b5f56700 1 --1- 172.21.2.112:0/954489146 >> v1:172.21.2.120:6789/0 conn(0x7fd8b01b0470 0x7fd8b00ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:39746/0 (socket says 172.21.2.112:39746) 2021-02-10T11:30:15.316 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.312 7fd8b6757700 1 --2- 172.21.2.112:0/954489146 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd8b00efe00 0x7fd8b00f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:15.316 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.312 7fd8b5755700 1 -- 172.21.2.112:0/954489146 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd8b00efe00 msgr2=0x7fd8b00f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:15.316 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.312 7fd8b5755700 1 --2- 172.21.2.112:0/954489146 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd8b00efe00 0x7fd8b00f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:15.316 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.312 7fd8b5755700 1 -- 172.21.2.112:0/954489146 >> v1:172.21.2.120:6789/0 conn(0x7fd8b01b0470 legacy=0x7fd8b00ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:15.316 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.312 7fd8b5755700 1 -- 172.21.2.112:0/954489146 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8b00e52e0 con 0x7fd8b01b0870 2021-02-10T11:30:15.316 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.312 7fd8b5755700 1 --2- 172.21.2.112:0/954489146 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd8b01b0870 0x7fd8b00f20e0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 rx=0x7fd8ac006700 tx=0x7fd8ac009b80).ready entity=mon.0 client_cookie=ef342e4158c4aaef server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:15.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.312 7fd8b4f54700 1 -- 172.21.2.112:0/954489146 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd8ac00c0c0 con 0x7fd8b01b0870 2021-02-10T11:30:15.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.312 7fd8b4f54700 1 -- 172.21.2.112:0/954489146 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8ac00c440 con 0x7fd8b01b0870 2021-02-10T11:30:15.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.312 7fd8b4f54700 1 -- 172.21.2.112:0/954489146 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd8ac00c8a0 con 0x7fd8b01b0870 2021-02-10T11:30:15.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.312 7fd8b813a700 1 -- 172.21.2.112:0/954489146 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd8b01b0870 msgr2=0x7fd8b00f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:15.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.312 7fd8b813a700 1 --2- 172.21.2.112:0/954489146 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd8b01b0870 0x7fd8b00f20e0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 rx=0x7fd8ac006700 tx=0x7fd8ac009b80).stop 2021-02-10T11:30:15.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.313 7fd8b813a700 1 -- 172.21.2.112:0/954489146 shutdown_connections 2021-02-10T11:30:15.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.313 7fd8b813a700 1 --2- 172.21.2.112:0/954489146 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd8b01b0870 0x7fd8b00f20e0 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:15.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.313 7fd8b813a700 1 --2- 172.21.2.112:0/954489146 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd8b00efe00 0x7fd8b00f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:15.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.313 7fd8b813a700 1 -- 172.21.2.112:0/954489146 >> 172.21.2.112:0/954489146 conn(0x7fd8b00ce3e0 msgr2=0x7fd8b00e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:15.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.313 7fd8b813a700 1 -- 172.21.2.112:0/954489146 shutdown_connections 2021-02-10T11:30:15.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.313 7fd8b813a700 1 -- 172.21.2.112:0/954489146 wait complete. 2021-02-10T11:30:15.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.328 7fd8b813a700 1 Processor -- start 2021-02-10T11:30:15.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.328 7fd8b813a700 1 -- start start 2021-02-10T11:30:15.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.328 7fd8b813a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd8b0190920 0x7fd8b00b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:15.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.328 7fd8b813a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd8b00b8200 0x7fd8b00b8600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:15.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.328 7fd8b813a700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd8b00c1d80 0x7fd8b00c2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:15.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.328 7fd8b813a700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b0190d20 con 0x7fd8b00b8200 2021-02-10T11:30:15.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.328 7fd8b813a700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b00b8c20 con 0x7fd8b00c1d80 2021-02-10T11:30:15.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.328 7fd8b813a700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b00c27a0 con 0x7fd8b0190920 2021-02-10T11:30:15.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.329 7fd8b6757700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd8b00c1d80 0x7fd8b00c2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:15.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.329 7fd8b5755700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd8b00b8200 0x7fd8b00b8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:15.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.329 7fd8b6757700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd8b00c1d80 0x7fd8b00c2180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3300/0 says I am v2:172.21.2.112:49012/0 (socket says 172.21.2.112:49012) 2021-02-10T11:30:15.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.329 7fd8b6757700 1 -- 172.21.2.112:0/2445852644 learned_addr learned my addr 172.21.2.112:0/2445852644 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:30:15.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.329 7fd8b5755700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd8b00b8200 0x7fd8b00b8600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:46732/0 (socket says 172.21.2.112:46732) 2021-02-10T11:30:15.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.329 7fd8b5f56700 1 --2- 172.21.2.112:0/2445852644 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd8b0190920 0x7fd8b00b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:15.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.329 7fd8b6757700 1 -- 172.21.2.112:0/2445852644 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd8b0190920 msgr2=0x7fd8b00b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:15.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.329 7fd8b6757700 1 --2- 172.21.2.112:0/2445852644 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd8b0190920 0x7fd8b00b7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:15.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.329 7fd8b6757700 1 -- 172.21.2.112:0/2445852644 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd8b00b8200 msgr2=0x7fd8b00b8600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:15.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.329 7fd8b6757700 1 --2- 172.21.2.112:0/2445852644 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd8b00b8200 0x7fd8b00b8600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:15.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.329 7fd8b6757700 1 -- 172.21.2.112:0/2445852644 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8b00c29c0 con 0x7fd8b00c1d80 2021-02-10T11:30:15.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.329 7fd8b5755700 1 --2- 172.21.2.112:0/2445852644 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd8b00b8200 0x7fd8b00b8600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:30:15.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.329 7fd8b6757700 1 --2- 172.21.2.112:0/2445852644 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd8b00c1d80 0x7fd8b00c2180 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7fd8a800b940 tx=0x7fd8a800bce0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:15.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.329 7fd8a27fc700 1 -- 172.21.2.112:0/2445852644 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd8a80066f0 con 0x7fd8b00c1d80 2021-02-10T11:30:15.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.329 7fd8a27fc700 1 -- 172.21.2.112:0/2445852644 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8a8003030 con 0x7fd8b00c1d80 2021-02-10T11:30:15.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.329 7fd8a27fc700 1 -- 172.21.2.112:0/2445852644 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd8a80013e0 con 0x7fd8b00c1d80 2021-02-10T11:30:15.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.329 7fd8b813a700 1 -- 172.21.2.112:0/2445852644 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8b00c29c0 con 0x7fd8b00c1d80 2021-02-10T11:30:15.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.330 7fd8b813a700 1 -- 172.21.2.112:0/2445852644 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8b00c29c0 con 0x7fd8b00c1d80 2021-02-10T11:30:15.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.330 7fd8a27fc700 1 -- 172.21.2.112:0/2445852644 <== mon.1 v2:172.21.2.120:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd8a8006a90 con 0x7fd8b00c1d80 2021-02-10T11:30:15.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.330 7fd8a27fc700 1 --2- 172.21.2.112:0/2445852644 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd8980186c0 0x7fd89801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:15.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.330 7fd8a27fc700 1 -- 172.21.2.112:0/2445852644 <== mon.1 v2:172.21.2.120:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fd8a8032920 con 0x7fd8b00c1d80 2021-02-10T11:30:15.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.330 7fd8b5f56700 1 --2- 172.21.2.112:0/2445852644 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd8980186c0 0x7fd89801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:15.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.331 7fd8b5f56700 1 --2- 172.21.2.112:0/2445852644 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd8980186c0 0x7fd89801abc0 crc :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:15.348 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.361 7fd8b813a700 1 -- 172.21.2.112:0/2445852644 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8b0105ae0 con 0x7fd8b00c1d80 2021-02-10T11:30:15.351 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.364 7fd8a27fc700 1 -- 172.21.2.112:0/2445852644 <== mon.1 v2:172.21.2.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd8a800be20 con 0x7fd8b00c1d80 2021-02-10T11:30:15.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.491 7fd8b813a700 1 -- 172.21.2.112:0/2445852644 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348579", "target": ["mgr", ""]}) v1 -- 0x7fd8b00f0b50 con 0x7fd8980186c0 2021-02-10T11:30:15.489 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.501 7fd8a27fc700 1 -- 172.21.2.112:0/2445852644 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fd8a4007da0 con 0x7fd8980186c0 2021-02-10T11:30:15.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.508 7fd8b813a700 1 -- 172.21.2.112:0/2445852644 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd8980186c0 msgr2=0x7fd89801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:15.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.508 7fd8b813a700 1 --2- 172.21.2.112:0/2445852644 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd8980186c0 0x7fd89801abc0 crc :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:15.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.508 7fd8b813a700 1 -- 172.21.2.112:0/2445852644 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd8b00c1d80 msgr2=0x7fd8b00c2180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:15.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.508 7fd8b813a700 1 --2- 172.21.2.112:0/2445852644 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd8b00c1d80 0x7fd8b00c2180 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7fd8a800b940 tx=0x7fd8a800bce0).stop 2021-02-10T11:30:15.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.508 7fd8b813a700 1 -- 172.21.2.112:0/2445852644 shutdown_connections 2021-02-10T11:30:15.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.508 7fd8b813a700 1 --2- 172.21.2.112:0/2445852644 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd8980186c0 0x7fd89801abc0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:15.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.508 7fd8b813a700 1 --2- 172.21.2.112:0/2445852644 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd8b00c1d80 0x7fd8b00c2180 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:15.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.509 7fd8b813a700 1 --2- 172.21.2.112:0/2445852644 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd8b00b8200 0x7fd8b00b8600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:15.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.509 7fd8b813a700 1 --2- 172.21.2.112:0/2445852644 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd8b0190920 0x7fd8b00b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:15.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.509 7fd8b813a700 1 -- 172.21.2.112:0/2445852644 >> 172.21.2.112:0/2445852644 conn(0x7fd8b0094e10 msgr2=0x7fd8b00a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:15.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.509 7fd8b813a700 1 -- 172.21.2.112:0/2445852644 shutdown_connections 2021-02-10T11:30:15.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.509 7fd8b813a700 1 -- 172.21.2.112:0/2445852644 wait complete. 2021-02-10T11:30:15.505 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000348580 2021-02-10T11:30:15.935 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.930 7fc1982d4700 1 Processor -- start 2021-02-10T11:30:15.935 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.930 7fc1982d4700 1 -- start start 2021-02-10T11:30:15.935 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.931 7fc1982d4700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fc1901b0470 0x7fc1901b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:15.935 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.931 7fc1982d4700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1900f0b50 con 0x7fc1900f20e0 2021-02-10T11:30:15.935 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.931 7fc1982d4700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1900f24e0 con 0x7fc1900ef610 2021-02-10T11:30:15.935 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.931 7fc1982d4700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1900e5ea0 con 0x7fc1901b0470 2021-02-10T11:30:15.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.931 7fc1960f0700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fc1901b0470 0x7fc1901b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:15.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.931 7fc1968f1700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fc1900f20e0 0x7fc1900f0200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:46200/0 (socket says 172.21.2.112:46200) 2021-02-10T11:30:15.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.931 7fc1968f1700 1 -- 172.21.2.112:0/629585094 learned_addr learned my addr 172.21.2.112:0/629585094 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:30:15.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.931 7fc1960f0700 1 -- 172.21.2.112:0/629585094 >> v1:172.21.2.120:6789/0 conn(0x7fc1900ef610 legacy=0x7fc1900efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:15.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.931 7fc1960f0700 1 -- 172.21.2.112:0/629585094 >> v1:172.21.2.112:6789/0 conn(0x7fc1900f20e0 legacy=0x7fc1900f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:15.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.931 7fc1950ee700 1 -- 172.21.2.112:0/629585094 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2151694940 0 0) 0x7fc184000f30 con 0x7fc1900ef610 2021-02-10T11:30:15.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.931 7fc1960f0700 1 -- 172.21.2.112:0/629585094 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1900e52e0 con 0x7fc1901b0470 2021-02-10T11:30:15.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.932 7fc1960f0700 1 --2- 172.21.2.112:0/629585094 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc1901b0470 0x7fc1901b0870 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 rx=0x7fc18c006650 tx=0x7fc18c009a00).ready entity=mon.0 client_cookie=589d463b1867df8c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:15.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.932 7fc1950ee700 1 -- 172.21.2.112:0/629585094 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc18c0009b0 con 0x7fc1901b0470 2021-02-10T11:30:15.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.932 7fc1950ee700 1 -- 172.21.2.112:0/629585094 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc18c00c0c0 con 0x7fc1901b0470 2021-02-10T11:30:15.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.932 7fc1950ee700 1 -- 172.21.2.112:0/629585094 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc18c00c5a0 con 0x7fc1901b0470 2021-02-10T11:30:15.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.932 7fc1982d4700 1 -- 172.21.2.112:0/629585094 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc1901b0470 msgr2=0x7fc1901b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:15.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.932 7fc1982d4700 1 --2- 172.21.2.112:0/629585094 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc1901b0470 0x7fc1901b0870 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 rx=0x7fc18c006650 tx=0x7fc18c009a00).stop 2021-02-10T11:30:15.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.933 7fc1982d4700 1 -- 172.21.2.112:0/629585094 shutdown_connections 2021-02-10T11:30:15.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.933 7fc1982d4700 1 --2- 172.21.2.112:0/629585094 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc1901b0470 0x7fc1901b0870 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:15.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.933 7fc1982d4700 1 -- 172.21.2.112:0/629585094 >> 172.21.2.112:0/629585094 conn(0x7fc1900ce3e0 msgr2=0x7fc1900e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:15.938 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.933 7fc1982d4700 1 -- 172.21.2.112:0/629585094 shutdown_connections 2021-02-10T11:30:15.938 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.933 7fc1982d4700 1 -- 172.21.2.112:0/629585094 wait complete. 2021-02-10T11:30:15.940 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.947 7fc1982d4700 1 Processor -- start 2021-02-10T11:30:15.940 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.948 7fc1982d4700 1 -- start start 2021-02-10T11:30:15.940 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.948 7fc1982d4700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc1900a1dd0 0x7fc190190920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:15.941 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.948 7fc1982d4700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc19008fd30 0x7fc1900effc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:15.941 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.948 7fc1982d4700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc1900f0e90 0x7fc1900f1290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:15.941 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.948 7fc1982d4700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1900f06f0 con 0x7fc19008fd30 2021-02-10T11:30:15.941 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.948 7fc1982d4700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc190090160 con 0x7fc1900a1dd0 2021-02-10T11:30:15.941 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.948 7fc1982d4700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1900f18b0 con 0x7fc1900f0e90 2021-02-10T11:30:15.941 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.948 7fc1958ef700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc19008fd30 0x7fc1900effc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:15.941 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.948 7fc1958ef700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc19008fd30 0x7fc1900effc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:46746/0 (socket says 172.21.2.112:46746) 2021-02-10T11:30:15.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.948 7fc1958ef700 1 -- 172.21.2.112:0/2721781652 learned_addr learned my addr 172.21.2.112:0/2721781652 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:30:15.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.948 7fc1960f0700 1 --2- 172.21.2.112:0/2721781652 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc1900a1dd0 0x7fc190190920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:15.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.948 7fc1968f1700 1 --2- 172.21.2.112:0/2721781652 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc1900f0e90 0x7fc1900f1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:15.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.949 7fc1958ef700 1 -- 172.21.2.112:0/2721781652 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc1900f0e90 msgr2=0x7fc1900f1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:15.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.949 7fc1958ef700 1 --2- 172.21.2.112:0/2721781652 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc1900f0e90 0x7fc1900f1290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:15.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.949 7fc1958ef700 1 -- 172.21.2.112:0/2721781652 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc1900a1dd0 msgr2=0x7fc190190920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:15.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.949 7fc1958ef700 1 --2- 172.21.2.112:0/2721781652 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc1900a1dd0 0x7fc190190920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:15.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.949 7fc1958ef700 1 -- 172.21.2.112:0/2721781652 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1900f1ad0 con 0x7fc19008fd30 2021-02-10T11:30:15.943 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.949 7fc1958ef700 1 --2- 172.21.2.112:0/2721781652 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc19008fd30 0x7fc1900effc0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 rx=0x7fc18400a550 tx=0x7fc18400a8f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:15.943 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.949 7fc182ffd700 1 -- 172.21.2.112:0/2721781652 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc18400b2f0 con 0x7fc19008fd30 2021-02-10T11:30:15.943 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.949 7fc182ffd700 1 -- 172.21.2.112:0/2721781652 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc18400b670 con 0x7fc19008fd30 2021-02-10T11:30:15.943 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.949 7fc1982d4700 1 -- 172.21.2.112:0/2721781652 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1900f1ad0 con 0x7fc19008fd30 2021-02-10T11:30:15.943 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.949 7fc182ffd700 1 -- 172.21.2.112:0/2721781652 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc18400bad0 con 0x7fc19008fd30 2021-02-10T11:30:15.943 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.949 7fc1982d4700 1 -- 172.21.2.112:0/2721781652 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1900f2060 con 0x7fc19008fd30 2021-02-10T11:30:15.943 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.950 7fc182ffd700 1 -- 172.21.2.112:0/2721781652 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc18400b2f0 con 0x7fc19008fd30 2021-02-10T11:30:15.943 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.950 7fc182ffd700 1 --2- 172.21.2.112:0/2721781652 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc178018730 0x7fc17801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:15.943 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.950 7fc182ffd700 1 -- 172.21.2.112:0/2721781652 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fc1840327b0 con 0x7fc19008fd30 2021-02-10T11:30:15.943 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.950 7fc1960f0700 1 --2- 172.21.2.112:0/2721781652 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc178018730 0x7fc17801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:15.943 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.951 7fc1960f0700 1 --2- 172.21.2.112:0/2721781652 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc178018730 0x7fc17801ac10 crc :-1 s=READY pgs=399 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:15.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.980 7fc1982d4700 1 -- 172.21.2.112:0/2721781652 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1900e52e0 con 0x7fc19008fd30 2021-02-10T11:30:15.971 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:15.983 7fc182ffd700 1 -- 172.21.2.112:0/2721781652 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc18400b610 con 0x7fc19008fd30 2021-02-10T11:30:16.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:16.114 7fc1982d4700 1 -- 172.21.2.112:0/2721781652 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348580", "target": ["mgr", ""]}) v1 -- 0x7fc190090160 con 0x7fc178018730 2021-02-10T11:30:16.111 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:16.124 7fc182ffd700 1 -- 172.21.2.112:0/2721781652 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fc18c0009b0 con 0x7fc178018730 2021-02-10T11:30:16.119 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:16.131 7fc1982d4700 1 -- 172.21.2.112:0/2721781652 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc178018730 msgr2=0x7fc17801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:16.119 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:16.131 7fc1982d4700 1 --2- 172.21.2.112:0/2721781652 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc178018730 0x7fc17801ac10 crc :-1 s=READY pgs=399 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:16.119 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:16.131 7fc1982d4700 1 -- 172.21.2.112:0/2721781652 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc19008fd30 msgr2=0x7fc1900effc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:16.119 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:16.131 7fc1982d4700 1 --2- 172.21.2.112:0/2721781652 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc19008fd30 0x7fc1900effc0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 rx=0x7fc18400a550 tx=0x7fc18400a8f0).stop 2021-02-10T11:30:16.119 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:16.131 7fc1982d4700 1 -- 172.21.2.112:0/2721781652 shutdown_connections 2021-02-10T11:30:16.119 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:16.131 7fc1982d4700 1 --2- 172.21.2.112:0/2721781652 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc178018730 0x7fc17801ac10 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:16.119 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:16.131 7fc1982d4700 1 --2- 172.21.2.112:0/2721781652 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc1900f0e90 0x7fc1900f1290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:16.119 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:16.131 7fc1982d4700 1 --2- 172.21.2.112:0/2721781652 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc19008fd30 0x7fc1900effc0 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:16.119 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:16.131 7fc1982d4700 1 --2- 172.21.2.112:0/2721781652 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc1900a1dd0 0x7fc190190920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:16.119 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:16.131 7fc1982d4700 1 -- 172.21.2.112:0/2721781652 >> 172.21.2.112:0/2721781652 conn(0x7fc1900958b0 msgr2=0x7fc1900bf040 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:16.119 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:16.131 7fc1982d4700 1 -- 172.21.2.112:0/2721781652 shutdown_connections 2021-02-10T11:30:16.120 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:16.131 7fc1982d4700 1 -- 172.21.2.112:0/2721781652 wait complete. 2021-02-10T11:30:16.127 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000348581 2021-02-10T11:30:16.555 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:16.551 7f1ef3bfe700 1 Processor -- start 2021-02-10T11:30:16.555 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:16.551 7f1ef3bfe700 1 -- start start 2021-02-10T11:30:16.555 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:16.552 7f1ef3bfe700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f1eec1b0470 0x7f1eec1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:16.555 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:16.552 7f1ef3bfe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f1eec0ef610 0x7f1eec0efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:16.555 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:16.552 7f1ef3bfe700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1eec0f0b50 con 0x7f1eec0f0030 2021-02-10T11:30:16.555 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:16.552 7f1ef3bfe700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1eec0e52e0 con 0x7f1eec0ef610 2021-02-10T11:30:16.555 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:16.552 7f1ef3bfe700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1eec0e5490 con 0x7f1eec1b0470 2021-02-10T11:30:16.555 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:16.552 7f1ef221b700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f1eec0f0030 0x7f1eec0f10d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:46214/0 (socket says 172.21.2.112:46214) 2021-02-10T11:30:16.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:16.552 7f1ef221b700 1 -- 172.21.2.112:0/2747712985 learned_addr learned my addr 172.21.2.112:0/2747712985 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:30:16.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:16.552 7f1ef1219700 1 --2- 172.21.2.112:0/2747712985 >> v2:172.21.2.112:3300/0 conn(0x7f1eec0ef610 0x7f1eec0efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:16.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:16.552 7f1ef1a1a700 1 --2- 172.21.2.112:0/2747712985 >> v2:172.21.2.120:3300/0 conn(0x7f1eec1b0470 0x7f1eec1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:16.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:16.552 7f1ef1219700 1 -- 172.21.2.112:0/2747712985 >> v2:172.21.2.120:3300/0 conn(0x7f1eec1b0470 msgr2=0x7f1eec1b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:16.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:16.552 7f1ef1219700 1 --2- 172.21.2.112:0/2747712985 >> v2:172.21.2.120:3300/0 conn(0x7f1eec1b0470 0x7f1eec1b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:16.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:16.552 7f1ef1219700 1 -- 172.21.2.112:0/2747712985 >> v1:172.21.2.112:6789/0 conn(0x7f1eec0f0030 legacy=0x7f1eec0f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:16.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:16.552 7f1ef1219700 1 -- 172.21.2.112:0/2747712985 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1eec0f0430 con 0x7f1eec0ef610 2021-02-10T11:30:16.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:16.553 7f1ef1219700 1 --2- 172.21.2.112:0/2747712985 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1eec0ef610 0x7f1eec0efa10 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 rx=0x7f1edc006650 tx=0x7f1edc009a80).ready entity=mon.0 client_cookie=e3e7ea940e1746b1 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:16.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:16.553 7f1ef0a18700 1 -- 172.21.2.112:0/2747712985 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1edc00c0c0 con 0x7f1eec0ef610 2021-02-10T11:30:16.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:16.553 7f1ef0a18700 1 -- 172.21.2.112:0/2747712985 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1edc00c340 con 0x7f1eec0ef610 2021-02-10T11:30:16.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:16.553 7f1ef0a18700 1 -- 172.21.2.112:0/2747712985 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1edc00c860 con 0x7f1eec0ef610 2021-02-10T11:30:16.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:16.553 7f1ef3bfe700 1 -- 172.21.2.112:0/2747712985 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1eec0ef610 msgr2=0x7f1eec0efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:16.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:16.553 7f1ef3bfe700 1 --2- 172.21.2.112:0/2747712985 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1eec0ef610 0x7f1eec0efa10 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 rx=0x7f1edc006650 tx=0x7f1edc009a80).stop 2021-02-10T11:30:16.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:16.553 7f1ef3bfe700 1 -- 172.21.2.112:0/2747712985 shutdown_connections 2021-02-10T11:30:16.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:16.553 7f1ef3bfe700 1 --2- 172.21.2.112:0/2747712985 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1eec0ef610 0x7f1eec0efa10 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:16.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:16.553 7f1ef3bfe700 1 --2- 172.21.2.112:0/2747712985 >> v2:172.21.2.120:3300/0 conn(0x7f1eec1b0470 0x7f1eec1b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:16.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:16.553 7f1ef3bfe700 1 -- 172.21.2.112:0/2747712985 >> 172.21.2.112:0/2747712985 conn(0x7f1eec0ce3e0 msgr2=0x7f1eec0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:16.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:16.553 7f1ef3bfe700 1 -- 172.21.2.112:0/2747712985 shutdown_connections 2021-02-10T11:30:16.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:16.553 7f1ef3bfe700 1 -- 172.21.2.112:0/2747712985 wait complete. 2021-02-10T11:30:16.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:16.568 7f1ef3bfe700 1 Processor -- start 2021-02-10T11:30:16.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:16.568 7f1ef3bfe700 1 -- start start 2021-02-10T11:30:16.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:16.568 7f1ef3bfe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1eec0c0790 0x7f1eec09c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:16.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:16.568 7f1ef3bfe700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1eec09c7d0 0x7f1eec09a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:16.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:16.568 7f1ef3bfe700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1eec0c0b90 0x7f1eec09adb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:16.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:16.568 7f1ef3bfe700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1eec09bc30 con 0x7f1eec0c0790 2021-02-10T11:30:16.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:16.568 7f1ef3bfe700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1eec10a730 con 0x7f1eec0c0b90 2021-02-10T11:30:16.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:16.568 7f1ef3bfe700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1eec10a270 con 0x7f1eec09c7d0 2021-02-10T11:30:16.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:16.568 7f1ef1a1a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1eec0c0790 0x7f1eec09c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:16.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:16.569 7f1ef1a1a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1eec0c0790 0x7f1eec09c1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:46758/0 (socket says 172.21.2.112:46758) 2021-02-10T11:30:16.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:16.569 7f1ef1a1a700 1 -- 172.21.2.112:0/40250194 learned_addr learned my addr 172.21.2.112:0/40250194 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:30:16.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:16.569 7f1ef221b700 1 --2- 172.21.2.112:0/40250194 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1eec0c0b90 0x7f1eec09adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:16.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:16.569 7f1ef1219700 1 --2- 172.21.2.112:0/40250194 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1eec09c7d0 0x7f1eec09a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:16.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:16.569 7f1ef1a1a700 1 -- 172.21.2.112:0/40250194 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1eec09c7d0 msgr2=0x7f1eec09a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:16.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:16.569 7f1ef1a1a700 1 --2- 172.21.2.112:0/40250194 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1eec09c7d0 0x7f1eec09a790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:16.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:16.569 7f1ef1a1a700 1 -- 172.21.2.112:0/40250194 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1eec0c0b90 msgr2=0x7f1eec09adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:16.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:16.569 7f1ef1a1a700 1 --2- 172.21.2.112:0/40250194 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1eec0c0b90 0x7f1eec09adb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:16.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:16.569 7f1ef1a1a700 1 -- 172.21.2.112:0/40250194 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1eec09b3d0 con 0x7f1eec0c0790 2021-02-10T11:30:16.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:16.569 7f1ef221b700 1 --2- 172.21.2.112:0/40250194 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1eec0c0b90 0x7f1eec09adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:30:16.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:16.569 7f1ef1a1a700 1 --2- 172.21.2.112:0/40250194 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1eec0c0790 0x7f1eec09c1b0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 rx=0x7f1ee8006f60 tx=0x7f1ee800b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:16.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:16.569 7f1ee27fc700 1 -- 172.21.2.112:0/40250194 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1ee80009b0 con 0x7f1eec0c0790 2021-02-10T11:30:16.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:16.569 7f1ee27fc700 1 -- 172.21.2.112:0/40250194 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ee800bdf0 con 0x7f1eec0c0790 2021-02-10T11:30:16.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:16.569 7f1ee27fc700 1 -- 172.21.2.112:0/40250194 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1ee8002030 con 0x7f1eec0c0790 2021-02-10T11:30:16.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:16.569 7f1ef3bfe700 1 -- 172.21.2.112:0/40250194 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1eec09b3d0 con 0x7f1eec0c0790 2021-02-10T11:30:16.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:16.570 7f1ef3bfe700 1 -- 172.21.2.112:0/40250194 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1eec09b3d0 con 0x7f1eec0c0790 2021-02-10T11:30:16.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:16.570 7f1ee27fc700 1 -- 172.21.2.112:0/40250194 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1ee8006b00 con 0x7f1eec0c0790 2021-02-10T11:30:16.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:16.570 7f1ee27fc700 1 --2- 172.21.2.112:0/40250194 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1ed40186c0 0x7f1ed401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:16.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:16.570 7f1ee27fc700 1 -- 172.21.2.112:0/40250194 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f1ee80325c0 con 0x7f1eec0c0790 2021-02-10T11:30:16.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:16.571 7f1ef1219700 1 --2- 172.21.2.112:0/40250194 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1ed40186c0 0x7f1ed401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:16.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:16.571 7f1ef1219700 1 --2- 172.21.2.112:0/40250194 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1ed40186c0 0x7f1ed401abc0 crc :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:16.589 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:16.602 7f1ef3bfe700 1 -- 172.21.2.112:0/40250194 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1eec09b9e0 con 0x7f1eec0c0790 2021-02-10T11:30:16.592 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:16.604 7f1ee27fc700 1 -- 172.21.2.112:0/40250194 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1ee8006340 con 0x7f1eec0c0790 2021-02-10T11:30:16.718 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:16.730 7f1ef3bfe700 1 -- 172.21.2.112:0/40250194 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348581", "target": ["mgr", ""]}) v1 -- 0x7f1eec09b9e0 con 0x7f1ed40186c0 2021-02-10T11:30:16.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:16.741 7f1ee27fc700 1 -- 172.21.2.112:0/40250194 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f1edc000c90 con 0x7f1ed40186c0 2021-02-10T11:30:16.735 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:16.747 7f1ef3bfe700 1 -- 172.21.2.112:0/40250194 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1ed40186c0 msgr2=0x7f1ed401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:16.735 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:16.747 7f1ef3bfe700 1 --2- 172.21.2.112:0/40250194 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1ed40186c0 0x7f1ed401abc0 crc :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:16.735 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:16.747 7f1ef3bfe700 1 -- 172.21.2.112:0/40250194 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1eec0c0790 msgr2=0x7f1eec09c1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:16.736 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:16.747 7f1ef3bfe700 1 --2- 172.21.2.112:0/40250194 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1eec0c0790 0x7f1eec09c1b0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 rx=0x7f1ee8006f60 tx=0x7f1ee800b970).stop 2021-02-10T11:30:16.736 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:16.748 7f1ef3bfe700 1 -- 172.21.2.112:0/40250194 shutdown_connections 2021-02-10T11:30:16.736 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:16.748 7f1ef3bfe700 1 --2- 172.21.2.112:0/40250194 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1ed40186c0 0x7f1ed401abc0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:16.736 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:16.748 7f1ef3bfe700 1 --2- 172.21.2.112:0/40250194 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1eec0c0b90 0x7f1eec09adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:16.736 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:16.748 7f1ef3bfe700 1 --2- 172.21.2.112:0/40250194 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1eec09c7d0 0x7f1eec09a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:16.736 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:16.748 7f1ef3bfe700 1 --2- 172.21.2.112:0/40250194 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1eec0c0790 0x7f1eec09c1b0 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:16.736 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:16.748 7f1ef3bfe700 1 -- 172.21.2.112:0/40250194 >> 172.21.2.112:0/40250194 conn(0x7f1eec0bbf60 msgr2=0x7f1eec105d10 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:16.736 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:16.748 7f1ef3bfe700 1 -- 172.21.2.112:0/40250194 shutdown_connections 2021-02-10T11:30:16.736 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:16.748 7f1ef3bfe700 1 -- 172.21.2.112:0/40250194 wait complete. 2021-02-10T11:30:16.744 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000348582 2021-02-10T11:30:17.174 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.170 7f7fb9eb8700 1 Processor -- start 2021-02-10T11:30:17.174 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.170 7f7fb9eb8700 1 -- start start 2021-02-10T11:30:17.174 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.171 7f7fb9eb8700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7fb41b0470 0x7f7fb41b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:17.175 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.171 7f7fb9eb8700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fb40f0b50 con 0x7f7fb40ef610 2021-02-10T11:30:17.175 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.171 7f7fb9eb8700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fb40f24e0 con 0x7f7fb40f20e0 2021-02-10T11:30:17.175 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.171 7f7fb9eb8700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fb40e5ea0 con 0x7f7fb41b0470 2021-02-10T11:30:17.175 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.171 7f7faaffd700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f7fb40ef610 0x7f7fb40efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:46228/0 (socket says 172.21.2.112:46228) 2021-02-10T11:30:17.175 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.171 7f7faaffd700 1 -- 172.21.2.112:0/3141219713 learned_addr learned my addr 172.21.2.112:0/3141219713 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:30:17.175 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.171 7f7fb37fe700 1 --2- 172.21.2.112:0/3141219713 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7fb41b0470 0x7f7fb41b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:17.175 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.171 7f7fb2ffd700 1 -- 172.21.2.112:0/3141219713 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3787614932 0 0) 0x7f7f9c0009b0 con 0x7f7fb40ef610 2021-02-10T11:30:17.175 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.171 7f7fb2ffd700 1 -- 172.21.2.112:0/3141219713 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fa0002fe0 con 0x7f7fb40ef610 2021-02-10T11:30:17.175 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.171 7f7fb37fe700 1 -- 172.21.2.112:0/3141219713 >> v1:172.21.2.120:6789/0 conn(0x7f7fb40f20e0 legacy=0x7f7fb40f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:17.175 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.171 7f7fb2ffd700 1 -- 172.21.2.112:0/3141219713 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1650287253 0 0) 0x7f7fac000d30 con 0x7f7fb40f20e0 2021-02-10T11:30:17.175 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.171 7f7fb37fe700 1 -- 172.21.2.112:0/3141219713 >> v1:172.21.2.112:6789/0 conn(0x7f7fb40ef610 legacy=0x7f7fb40efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:17.175 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.171 7f7fb37fe700 1 -- 172.21.2.112:0/3141219713 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fb40e52e0 con 0x7f7fb41b0470 2021-02-10T11:30:17.176 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.172 7f7fb37fe700 1 --2- 172.21.2.112:0/3141219713 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7fb41b0470 0x7f7fb41b0870 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f7fa4006650 tx=0x7f7fa4009a80).ready entity=mon.2 client_cookie=bdf3c4ec263ec1e6 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:17.176 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.172 7f7fb2ffd700 1 -- 172.21.2.112:0/3141219713 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7fa40009b0 con 0x7f7fb41b0470 2021-02-10T11:30:17.176 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.172 7f7fb2ffd700 1 -- 172.21.2.112:0/3141219713 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fa400c0c0 con 0x7f7fb41b0470 2021-02-10T11:30:17.176 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.172 7f7fb2ffd700 1 -- 172.21.2.112:0/3141219713 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7fa400c560 con 0x7f7fb41b0470 2021-02-10T11:30:17.176 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.172 7f7fb9eb8700 1 -- 172.21.2.112:0/3141219713 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7fb41b0470 msgr2=0x7f7fb41b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:17.176 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.172 7f7fb9eb8700 1 --2- 172.21.2.112:0/3141219713 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7fb41b0470 0x7f7fb41b0870 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f7fa4006650 tx=0x7f7fa4009a80).stop 2021-02-10T11:30:17.176 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.172 7f7fb9eb8700 1 -- 172.21.2.112:0/3141219713 shutdown_connections 2021-02-10T11:30:17.176 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.172 7f7fb9eb8700 1 --2- 172.21.2.112:0/3141219713 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7fb41b0470 0x7f7fb41b0870 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:17.176 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.172 7f7fb9eb8700 1 -- 172.21.2.112:0/3141219713 >> 172.21.2.112:0/3141219713 conn(0x7f7fb40ce3e0 msgr2=0x7f7fb40e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:17.176 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.172 7f7fb9eb8700 1 -- 172.21.2.112:0/3141219713 shutdown_connections 2021-02-10T11:30:17.176 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.172 7f7fb9eb8700 1 -- 172.21.2.112:0/3141219713 wait complete. 2021-02-10T11:30:17.178 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.187 7f7fb9eb8700 1 Processor -- start 2021-02-10T11:30:17.178 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.187 7f7fb9eb8700 1 -- start start 2021-02-10T11:30:17.178 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.188 7f7fb9eb8700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7fb40b4c60 0x7f7fb4106fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:17.179 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.188 7f7fb9eb8700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7fb4190630 0x7f7fb4099f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:17.179 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.188 7f7fb9eb8700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7fb4190a30 0x7f7fb409b130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:17.179 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.188 7f7fb9eb8700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fb4107920 con 0x7f7fb4190a30 2021-02-10T11:30:17.179 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.188 7f7fb9eb8700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fb409abb0 con 0x7f7fb4190630 2021-02-10T11:30:17.179 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.188 7f7fb9eb8700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fb40b5060 con 0x7f7fb40b4c60 2021-02-10T11:30:17.179 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.188 7f7faaffd700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7fb4190630 0x7f7fb4099f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:17.179 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.188 7f7fb3fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7fb4190a30 0x7f7fb409b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:17.179 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.188 7f7faaffd700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7fb4190630 0x7f7fb4099f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3300/0 says I am v2:172.21.2.112:49048/0 (socket says 172.21.2.112:49048) 2021-02-10T11:30:17.179 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.188 7f7faaffd700 1 -- 172.21.2.112:0/4016409467 learned_addr learned my addr 172.21.2.112:0/4016409467 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:30:17.179 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.188 7f7fb37fe700 1 --2- 172.21.2.112:0/4016409467 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7fb40b4c60 0x7f7fb4106fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:17.179 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.188 7f7fb37fe700 1 -- 172.21.2.112:0/4016409467 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7fb4190630 msgr2=0x7f7fb4099f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:17.180 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.188 7f7fb37fe700 1 --2- 172.21.2.112:0/4016409467 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7fb4190630 0x7f7fb4099f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:17.180 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.188 7f7fb37fe700 1 -- 172.21.2.112:0/4016409467 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7fb4190a30 msgr2=0x7f7fb409b130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:17.180 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.188 7f7fb37fe700 1 --2- 172.21.2.112:0/4016409467 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7fb4190a30 0x7f7fb409b130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:17.180 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.188 7f7fb37fe700 1 -- 172.21.2.112:0/4016409467 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fb409a5b0 con 0x7f7fb40b4c60 2021-02-10T11:30:17.180 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.188 7f7fb3fff700 1 --2- 172.21.2.112:0/4016409467 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7fb4190a30 0x7f7fb409b130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:30:17.180 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.188 7f7fb37fe700 1 --2- 172.21.2.112:0/4016409467 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7fb40b4c60 0x7f7fb4106fe0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f7fa4006620 tx=0x7f7fa400b860).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:17.180 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.189 7f7fb0ff9700 1 -- 172.21.2.112:0/4016409467 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7fa4000c90 con 0x7f7fb40b4c60 2021-02-10T11:30:17.180 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.189 7f7fb0ff9700 1 -- 172.21.2.112:0/4016409467 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fa400bc50 con 0x7f7fb40b4c60 2021-02-10T11:30:17.180 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.189 7f7fb0ff9700 1 -- 172.21.2.112:0/4016409467 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7fa40022b0 con 0x7f7fb40b4c60 2021-02-10T11:30:17.181 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.189 7f7fb9eb8700 1 -- 172.21.2.112:0/4016409467 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fb409a5b0 con 0x7f7fb40b4c60 2021-02-10T11:30:17.181 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.189 7f7fb9eb8700 1 -- 172.21.2.112:0/4016409467 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fb409a5b0 con 0x7f7fb40b4c60 2021-02-10T11:30:17.181 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.189 7f7fb0ff9700 1 -- 172.21.2.112:0/4016409467 <== mon.2 v2:172.21.2.120:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7fa40052e0 con 0x7f7fb40b4c60 2021-02-10T11:30:17.181 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.189 7f7fb0ff9700 1 --2- 172.21.2.112:0/4016409467 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7f980186c0 0x7f7f9801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:17.181 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.189 7f7fb0ff9700 1 -- 172.21.2.112:0/4016409467 <== mon.2 v2:172.21.2.120:3301/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f7fa4032710 con 0x7f7fb40b4c60 2021-02-10T11:30:17.181 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.189 7f7faaffd700 1 --2- 172.21.2.112:0/4016409467 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7f980186c0 0x7f7f9801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:17.182 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.190 7f7faaffd700 1 --2- 172.21.2.112:0/4016409467 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7f980186c0 0x7f7f9801abc0 crc :-1 s=READY pgs=401 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:17.207 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.220 7f7fb9eb8700 1 -- 172.21.2.112:0/4016409467 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fb4105af0 con 0x7f7fb40b4c60 2021-02-10T11:30:17.210 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.223 7f7fb0ff9700 1 -- 172.21.2.112:0/4016409467 <== mon.2 v2:172.21.2.120:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7fa40009a0 con 0x7f7fb40b4c60 2021-02-10T11:30:17.339 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.351 7f7fb9eb8700 1 -- 172.21.2.112:0/4016409467 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348582", "target": ["mgr", ""]}) v1 -- 0x7f7fb409abb0 con 0x7f7f980186c0 2021-02-10T11:30:17.348 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.361 7f7fb0ff9700 1 -- 172.21.2.112:0/4016409467 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f7f9c001600 con 0x7f7f980186c0 2021-02-10T11:30:17.356 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.367 7f7fb9eb8700 1 -- 172.21.2.112:0/4016409467 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7f980186c0 msgr2=0x7f7f9801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:17.356 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.367 7f7fb9eb8700 1 --2- 172.21.2.112:0/4016409467 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7f980186c0 0x7f7f9801abc0 crc :-1 s=READY pgs=401 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:17.356 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.367 7f7fb9eb8700 1 -- 172.21.2.112:0/4016409467 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7fb40b4c60 msgr2=0x7f7fb4106fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:17.356 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.367 7f7fb9eb8700 1 --2- 172.21.2.112:0/4016409467 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7fb40b4c60 0x7f7fb4106fe0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f7fa4006620 tx=0x7f7fa400b860).stop 2021-02-10T11:30:17.357 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.367 7f7fb9eb8700 1 -- 172.21.2.112:0/4016409467 shutdown_connections 2021-02-10T11:30:17.357 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.367 7f7fb9eb8700 1 --2- 172.21.2.112:0/4016409467 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7f980186c0 0x7f7f9801abc0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:17.357 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.367 7f7fb9eb8700 1 --2- 172.21.2.112:0/4016409467 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7fb4190630 0x7f7fb4099f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:17.357 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.367 7f7fb9eb8700 1 --2- 172.21.2.112:0/4016409467 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7fb4190a30 0x7f7fb409b130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:17.357 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.367 7f7fb9eb8700 1 --2- 172.21.2.112:0/4016409467 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7fb40b4c60 0x7f7fb4106fe0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:17.357 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.367 7f7fb9eb8700 1 -- 172.21.2.112:0/4016409467 >> 172.21.2.112:0/4016409467 conn(0x7f7fb40c40c0 msgr2=0x7f7fb40a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:17.358 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.368 7f7fb9eb8700 1 -- 172.21.2.112:0/4016409467 shutdown_connections 2021-02-10T11:30:17.358 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.368 7f7fb9eb8700 1 -- 172.21.2.112:0/4016409467 wait complete. 2021-02-10T11:30:17.365 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000348583 2021-02-10T11:30:17.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.790 7f4573584700 1 Processor -- start 2021-02-10T11:30:17.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.790 7f4573584700 1 -- start start 2021-02-10T11:30:17.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.791 7f4573584700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f456c1b0870 0x7f456c0f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:17.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.791 7f4573584700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f456c1b0c70 con 0x7f456c0efe00 2021-02-10T11:30:17.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.791 7f4573584700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f456c0f0b50 con 0x7f456c1b0470 2021-02-10T11:30:17.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.791 7f4573584700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f456c0e5ea0 con 0x7f456c1b0870 2021-02-10T11:30:17.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.791 7f4571ba1700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f456c0efe00 0x7f456c0f0200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:46238/0 (socket says 172.21.2.112:46238) 2021-02-10T11:30:17.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.791 7f4571ba1700 1 -- 172.21.2.112:0/4256798365 learned_addr learned my addr 172.21.2.112:0/4256798365 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:30:17.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.791 7f4570b9f700 1 --2- 172.21.2.112:0/4256798365 >> v2:172.21.2.112:3300/0 conn(0x7f456c1b0870 0x7f456c0f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:17.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.791 7f455ffff700 1 -- 172.21.2.112:0/4256798365 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2615980418 0 0) 0x7f4564001500 con 0x7f456c0efe00 2021-02-10T11:30:17.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.791 7f4570b9f700 1 -- 172.21.2.112:0/4256798365 >> v1:172.21.2.120:6789/0 conn(0x7f456c1b0470 legacy=0x7f456c0ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:17.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.791 7f4570b9f700 1 -- 172.21.2.112:0/4256798365 >> v1:172.21.2.112:6789/0 conn(0x7f456c0efe00 legacy=0x7f456c0f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:17.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.791 7f4570b9f700 1 -- 172.21.2.112:0/4256798365 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f456c0e52e0 con 0x7f456c1b0870 2021-02-10T11:30:17.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.791 7f4570b9f700 1 --2- 172.21.2.112:0/4256798365 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f456c1b0870 0x7f456c0f20e0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 rx=0x7f4568006650 tx=0x7f4568009a80).ready entity=mon.0 client_cookie=76734a183c3fb654 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:17.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.792 7f455ffff700 1 -- 172.21.2.112:0/4256798365 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f456800c0c0 con 0x7f456c1b0870 2021-02-10T11:30:17.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.792 7f455ffff700 1 -- 172.21.2.112:0/4256798365 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f456800c340 con 0x7f456c1b0870 2021-02-10T11:30:17.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.792 7f455ffff700 1 -- 172.21.2.112:0/4256798365 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f456800c860 con 0x7f456c1b0870 2021-02-10T11:30:17.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.792 7f4573584700 1 -- 172.21.2.112:0/4256798365 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f456c1b0870 msgr2=0x7f456c0f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:17.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.792 7f4573584700 1 --2- 172.21.2.112:0/4256798365 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f456c1b0870 0x7f456c0f20e0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 rx=0x7f4568006650 tx=0x7f4568009a80).stop 2021-02-10T11:30:17.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.792 7f4573584700 1 -- 172.21.2.112:0/4256798365 shutdown_connections 2021-02-10T11:30:17.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.792 7f4573584700 1 --2- 172.21.2.112:0/4256798365 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f456c1b0870 0x7f456c0f20e0 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:17.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.792 7f4573584700 1 -- 172.21.2.112:0/4256798365 >> 172.21.2.112:0/4256798365 conn(0x7f456c0ce3e0 msgr2=0x7f456c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:17.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.793 7f4573584700 1 -- 172.21.2.112:0/4256798365 shutdown_connections 2021-02-10T11:30:17.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.793 7f4573584700 1 -- 172.21.2.112:0/4256798365 wait complete. 2021-02-10T11:30:17.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.807 7f4573584700 1 Processor -- start 2021-02-10T11:30:17.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.807 7f4573584700 1 -- start start 2021-02-10T11:30:17.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.808 7f4573584700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f456c18ca80 0x7f456c18d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:17.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.808 7f4573584700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f456c18d6c0 0x7f456c18dac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:17.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.808 7f4573584700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f456c18e6b0 0x7f456c18eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:17.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.808 7f4573584700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f456c19d290 con 0x7f456c18ca80 2021-02-10T11:30:17.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.808 7f4573584700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f456c18e130 con 0x7f456c18e6b0 2021-02-10T11:30:17.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.808 7f4573584700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f456c18c5e0 con 0x7f456c18d6c0 2021-02-10T11:30:17.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.808 7f45713a0700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f456c18ca80 0x7f456c18d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:17.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.808 7f45713a0700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f456c18ca80 0x7f456c18d0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:46780/0 (socket says 172.21.2.112:46780) 2021-02-10T11:30:17.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.808 7f45713a0700 1 -- 172.21.2.112:0/3899569170 learned_addr learned my addr 172.21.2.112:0/3899569170 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:30:17.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.808 7f4570b9f700 1 --2- 172.21.2.112:0/3899569170 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f456c18d6c0 0x7f456c18dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:17.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.808 7f4571ba1700 1 --2- 172.21.2.112:0/3899569170 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f456c18e6b0 0x7f456c18eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:17.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.808 7f45713a0700 1 -- 172.21.2.112:0/3899569170 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f456c18d6c0 msgr2=0x7f456c18dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:17.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.808 7f45713a0700 1 --2- 172.21.2.112:0/3899569170 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f456c18d6c0 0x7f456c18dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:17.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.808 7f45713a0700 1 -- 172.21.2.112:0/3899569170 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f456c18e6b0 msgr2=0x7f456c18eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:17.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.808 7f45713a0700 1 --2- 172.21.2.112:0/3899569170 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f456c18e6b0 0x7f456c18eab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:17.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.808 7f45713a0700 1 -- 172.21.2.112:0/3899569170 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f456c18f0d0 con 0x7f456c18ca80 2021-02-10T11:30:17.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.809 7f45713a0700 1 --2- 172.21.2.112:0/3899569170 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f456c18ca80 0x7f456c18d0a0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 rx=0x7f4560006f30 tx=0x7f456000b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:17.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.809 7f455dffb700 1 -- 172.21.2.112:0/3899569170 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f456000bd50 con 0x7f456c18ca80 2021-02-10T11:30:17.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.809 7f455dffb700 1 -- 172.21.2.112:0/3899569170 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45600060c0 con 0x7f456c18ca80 2021-02-10T11:30:17.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.809 7f455dffb700 1 -- 172.21.2.112:0/3899569170 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4560005540 con 0x7f456c18ca80 2021-02-10T11:30:17.800 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.809 7f4573584700 1 -- 172.21.2.112:0/3899569170 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f456c18f0d0 con 0x7f456c18ca80 2021-02-10T11:30:17.800 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.809 7f4573584700 1 -- 172.21.2.112:0/3899569170 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f456c18f660 con 0x7f456c18ca80 2021-02-10T11:30:17.800 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.809 7f455dffb700 1 -- 172.21.2.112:0/3899569170 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4560005540 con 0x7f456c18ca80 2021-02-10T11:30:17.800 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.810 7f455dffb700 1 --2- 172.21.2.112:0/3899569170 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f454c0186c0 0x7f454c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:17.800 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.810 7f455dffb700 1 -- 172.21.2.112:0/3899569170 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f45600325d0 con 0x7f456c18ca80 2021-02-10T11:30:17.800 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.810 7f4570b9f700 1 --2- 172.21.2.112:0/3899569170 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f454c0186c0 0x7f454c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:17.800 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.810 7f4570b9f700 1 --2- 172.21.2.112:0/3899569170 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f454c0186c0 0x7f454c01abc0 crc :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:17.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.840 7f4573584700 1 -- 172.21.2.112:0/3899569170 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f456c0edce0 con 0x7f456c18ca80 2021-02-10T11:30:17.830 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.843 7f455dffb700 1 -- 172.21.2.112:0/3899569170 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4560006360 con 0x7f456c18ca80 2021-02-10T11:30:17.958 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.971 7f4573584700 1 -- 172.21.2.112:0/3899569170 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348583", "target": ["mgr", ""]}) v1 -- 0x7f456c18f660 con 0x7f454c0186c0 2021-02-10T11:30:17.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.981 7f455dffb700 1 -- 172.21.2.112:0/3899569170 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f4568000c90 con 0x7f454c0186c0 2021-02-10T11:30:17.975 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.987 7f4573584700 1 -- 172.21.2.112:0/3899569170 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f454c0186c0 msgr2=0x7f454c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:17.975 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.987 7f4573584700 1 --2- 172.21.2.112:0/3899569170 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f454c0186c0 0x7f454c01abc0 crc :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:17.975 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.987 7f4573584700 1 -- 172.21.2.112:0/3899569170 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f456c18ca80 msgr2=0x7f456c18d0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:17.975 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.987 7f4573584700 1 --2- 172.21.2.112:0/3899569170 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f456c18ca80 0x7f456c18d0a0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 rx=0x7f4560006f30 tx=0x7f456000b960).stop 2021-02-10T11:30:17.976 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.988 7f4573584700 1 -- 172.21.2.112:0/3899569170 shutdown_connections 2021-02-10T11:30:17.976 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.988 7f4573584700 1 --2- 172.21.2.112:0/3899569170 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f454c0186c0 0x7f454c01abc0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:17.976 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.988 7f4573584700 1 --2- 172.21.2.112:0/3899569170 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f456c18e6b0 0x7f456c18eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:17.976 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.988 7f4573584700 1 --2- 172.21.2.112:0/3899569170 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f456c18d6c0 0x7f456c18dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:17.976 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.988 7f4573584700 1 --2- 172.21.2.112:0/3899569170 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f456c18ca80 0x7f456c18d0a0 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:17.976 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.988 7f4573584700 1 -- 172.21.2.112:0/3899569170 >> 172.21.2.112:0/3899569170 conn(0x7f456c0bf4f0 msgr2=0x7f456c0be2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:17.976 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.988 7f4573584700 1 -- 172.21.2.112:0/3899569170 shutdown_connections 2021-02-10T11:30:17.977 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:17.988 7f4573584700 1 -- 172.21.2.112:0/3899569170 wait complete. 2021-02-10T11:30:17.984 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000348584 2021-02-10T11:30:18.413 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:18.409 7f8feb29f700 1 Processor -- start 2021-02-10T11:30:18.413 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:18.409 7f8feb29f700 1 -- start start 2021-02-10T11:30:18.413 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:18.410 7f8feb29f700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f8fe40f05f0 0x7f8fe40f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:18.413 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:18.410 7f8feb29f700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fe41b0c70 con 0x7f8fe41b0870 2021-02-10T11:30:18.414 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:18.410 7f8feb29f700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fe40f0b50 con 0x7f8fe41b0470 2021-02-10T11:30:18.414 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:18.410 7f8feb29f700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fe40e5ea0 con 0x7f8fe40f05f0 2021-02-10T11:30:18.414 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:18.410 7f8fe88ba700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f8fe41b0870 0x7f8fe40efe00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:46252/0 (socket says 172.21.2.112:46252) 2021-02-10T11:30:18.414 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:18.410 7f8fe88ba700 1 -- 172.21.2.112:0/1656437681 learned_addr learned my addr 172.21.2.112:0/1656437681 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:30:18.414 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:18.410 7f8fe98bc700 1 --2- 172.21.2.112:0/1656437681 >> v2:172.21.2.120:3300/0 conn(0x7f8fe40f05f0 0x7f8fe40f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:18.414 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:18.410 7f8fdbfff700 1 -- 172.21.2.112:0/1656437681 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1715588413 0 0) 0x7f8fd40010b0 con 0x7f8fe41b0870 2021-02-10T11:30:18.414 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:18.410 7f8fdbfff700 1 -- 172.21.2.112:0/1656437681 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fd0002fe0 con 0x7f8fe41b0870 2021-02-10T11:30:18.414 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:18.410 7f8fe98bc700 1 -- 172.21.2.112:0/1656437681 >> v1:172.21.2.120:6789/0 conn(0x7f8fe41b0470 legacy=0x7f8fe40ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:18.414 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:18.410 7f8fe98bc700 1 -- 172.21.2.112:0/1656437681 >> v1:172.21.2.112:6789/0 conn(0x7f8fe41b0870 legacy=0x7f8fe40efe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:18.414 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:18.410 7f8fe98bc700 1 -- 172.21.2.112:0/1656437681 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fe40e52e0 con 0x7f8fe40f05f0 2021-02-10T11:30:18.414 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:18.410 7f8fe98bc700 1 --2- 172.21.2.112:0/1656437681 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8fe40f05f0 0x7f8fe40f20e0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0x7f8fe0006ad0 tx=0x7f8fe0009ad0).ready entity=mon.1 client_cookie=660a3494b7dd64b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:18.414 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:18.411 7f8fdbfff700 1 -- 172.21.2.112:0/1656437681 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8fe00022e0 con 0x7f8fe40f05f0 2021-02-10T11:30:18.415 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:18.411 7f8fdbfff700 1 -- 172.21.2.112:0/1656437681 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fe0006410 con 0x7f8fe40f05f0 2021-02-10T11:30:18.415 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:18.411 7f8fdbfff700 1 -- 172.21.2.112:0/1656437681 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8fe0002620 con 0x7f8fe40f05f0 2021-02-10T11:30:18.415 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:18.411 7f8feb29f700 1 -- 172.21.2.112:0/1656437681 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8fe40f05f0 msgr2=0x7f8fe40f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:18.415 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:18.411 7f8feb29f700 1 --2- 172.21.2.112:0/1656437681 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8fe40f05f0 0x7f8fe40f20e0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0x7f8fe0006ad0 tx=0x7f8fe0009ad0).stop 2021-02-10T11:30:18.415 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:18.411 7f8feb29f700 1 -- 172.21.2.112:0/1656437681 shutdown_connections 2021-02-10T11:30:18.415 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:18.411 7f8feb29f700 1 --2- 172.21.2.112:0/1656437681 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8fe40f05f0 0x7f8fe40f20e0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:18.415 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:18.411 7f8feb29f700 1 -- 172.21.2.112:0/1656437681 >> 172.21.2.112:0/1656437681 conn(0x7f8fe40ce3e0 msgr2=0x7f8fe40e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:18.415 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:18.412 7f8feb29f700 1 -- 172.21.2.112:0/1656437681 shutdown_connections 2021-02-10T11:30:18.415 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:18.412 7f8feb29f700 1 -- 172.21.2.112:0/1656437681 wait complete. 2021-02-10T11:30:18.417 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:18.426 7f8feb29f700 1 Processor -- start 2021-02-10T11:30:18.417 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:18.426 7f8feb29f700 1 -- start start 2021-02-10T11:30:18.417 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:18.427 7f8feb29f700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8fe4094fb0 0x7f8fe409e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:18.417 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:18.427 7f8feb29f700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8fe40c2be0 0x7f8fe40c2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:18.417 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:18.427 7f8feb29f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8fe40c3600 0x7f8fe40a9c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:18.417 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:18.427 7f8feb29f700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fe409ebf0 con 0x7f8fe40c3600 2021-02-10T11:30:18.417 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:18.427 7f8feb29f700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fe40aa2a0 con 0x7f8fe40c2be0 2021-02-10T11:30:18.417 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:18.427 7f8feb29f700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fe40aa4c0 con 0x7f8fe4094fb0 2021-02-10T11:30:18.418 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:18.427 7f8fe98bc700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8fe40c3600 0x7f8fe40a9c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:18.418 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:18.427 7f8fe98bc700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8fe40c3600 0x7f8fe40a9c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:46798/0 (socket says 172.21.2.112:46798) 2021-02-10T11:30:18.418 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:18.427 7f8fe98bc700 1 -- 172.21.2.112:0/1814032163 learned_addr learned my addr 172.21.2.112:0/1814032163 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:30:18.418 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:18.427 7f8fe88ba700 1 --2- 172.21.2.112:0/1814032163 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8fe40c2be0 0x7f8fe40c2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:18.418 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:18.427 7f8fe90bb700 1 --2- 172.21.2.112:0/1814032163 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8fe4094fb0 0x7f8fe409e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:18.418 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:18.427 7f8fe98bc700 1 -- 172.21.2.112:0/1814032163 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8fe4094fb0 msgr2=0x7f8fe409e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:18.418 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:18.427 7f8fe98bc700 1 --2- 172.21.2.112:0/1814032163 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8fe4094fb0 0x7f8fe409e5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:18.418 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:18.427 7f8fe98bc700 1 -- 172.21.2.112:0/1814032163 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8fe40c2be0 msgr2=0x7f8fe40c2fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:18.418 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:18.427 7f8fe98bc700 1 --2- 172.21.2.112:0/1814032163 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8fe40c2be0 0x7f8fe40c2fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:18.418 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:18.427 7f8fe98bc700 1 -- 172.21.2.112:0/1814032163 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fe40aa6e0 con 0x7f8fe40c3600 2021-02-10T11:30:18.418 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:18.427 7f8fe90bb700 1 --2- 172.21.2.112:0/1814032163 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8fe4094fb0 0x7f8fe409e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:30:18.418 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:18.427 7f8fe98bc700 1 --2- 172.21.2.112:0/1814032163 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8fe40c3600 0x7f8fe40a9c80 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 rx=0x7f8fe000b610 tx=0x7f8fe000b9b0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:18.418 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:18.427 7f8fd9ffb700 1 -- 172.21.2.112:0/1814032163 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8fe0000da0 con 0x7f8fe40c3600 2021-02-10T11:30:18.419 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:18.427 7f8fd9ffb700 1 -- 172.21.2.112:0/1814032163 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fe0000a70 con 0x7f8fe40c3600 2021-02-10T11:30:18.419 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:18.427 7f8feb29f700 1 -- 172.21.2.112:0/1814032163 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fe40aa6e0 con 0x7f8fe40c3600 2021-02-10T11:30:18.419 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:18.427 7f8fd9ffb700 1 -- 172.21.2.112:0/1814032163 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8fe0002220 con 0x7f8fe40c3600 2021-02-10T11:30:18.419 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:18.427 7f8feb29f700 1 -- 172.21.2.112:0/1814032163 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fe40aac70 con 0x7f8fe40c3600 2021-02-10T11:30:18.419 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:18.428 7f8fd9ffb700 1 -- 172.21.2.112:0/1814032163 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8fe0005330 con 0x7f8fe40c3600 2021-02-10T11:30:18.419 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:18.428 7f8fd9ffb700 1 --2- 172.21.2.112:0/1814032163 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8fc4018730 0x7f8fc401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:18.419 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:18.428 7f8fd9ffb700 1 -- 172.21.2.112:0/1814032163 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f8fe0032820 con 0x7f8fe40c3600 2021-02-10T11:30:18.419 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:18.428 7f8fe90bb700 1 --2- 172.21.2.112:0/1814032163 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8fc4018730 0x7f8fc401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:18.419 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:18.429 7f8fe90bb700 1 --2- 172.21.2.112:0/1814032163 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8fc4018730 0x7f8fc401ac10 crc :-1 s=READY pgs=403 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:18.446 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:18.459 7f8feb29f700 1 -- 172.21.2.112:0/1814032163 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fe40a9750 con 0x7f8fe40c3600 2021-02-10T11:30:18.449 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:18.462 7f8fd9ffb700 1 -- 172.21.2.112:0/1814032163 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8fe00093e0 con 0x7f8fe40c3600 2021-02-10T11:30:18.577 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:18.589 7f8feb29f700 1 -- 172.21.2.112:0/1814032163 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348584", "target": ["mgr", ""]}) v1 -- 0x7f8fe40aac70 con 0x7f8fc4018730 2021-02-10T11:30:18.590 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:18.603 7f8fd9ffb700 1 -- 172.21.2.112:0/1814032163 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f8fdc002090 con 0x7f8fc4018730 2021-02-10T11:30:18.595 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:18.607 7f8feb29f700 1 -- 172.21.2.112:0/1814032163 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8fc4018730 msgr2=0x7f8fc401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:18.595 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:18.607 7f8feb29f700 1 --2- 172.21.2.112:0/1814032163 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8fc4018730 0x7f8fc401ac10 crc :-1 s=READY pgs=403 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:18.595 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:18.607 7f8feb29f700 1 -- 172.21.2.112:0/1814032163 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8fe40c3600 msgr2=0x7f8fe40a9c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:18.595 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:18.607 7f8feb29f700 1 --2- 172.21.2.112:0/1814032163 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8fe40c3600 0x7f8fe40a9c80 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 rx=0x7f8fe000b610 tx=0x7f8fe000b9b0).stop 2021-02-10T11:30:18.595 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:18.607 7f8feb29f700 1 -- 172.21.2.112:0/1814032163 shutdown_connections 2021-02-10T11:30:18.595 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:18.607 7f8feb29f700 1 --2- 172.21.2.112:0/1814032163 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8fc4018730 0x7f8fc401ac10 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:18.595 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:18.607 7f8feb29f700 1 --2- 172.21.2.112:0/1814032163 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8fe40c2be0 0x7f8fe40c2fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:18.596 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:18.607 7f8feb29f700 1 --2- 172.21.2.112:0/1814032163 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8fe40c3600 0x7f8fe40a9c80 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:18.596 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:18.607 7f8feb29f700 1 --2- 172.21.2.112:0/1814032163 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8fe4094fb0 0x7f8fe409e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:18.596 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:18.607 7f8feb29f700 1 -- 172.21.2.112:0/1814032163 >> 172.21.2.112:0/1814032163 conn(0x7f8fe409f630 msgr2=0x7f8fe40a0130 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:18.596 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:18.607 7f8feb29f700 1 -- 172.21.2.112:0/1814032163 shutdown_connections 2021-02-10T11:30:18.596 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:18.607 7f8feb29f700 1 -- 172.21.2.112:0/1814032163 wait complete. 2021-02-10T11:30:18.603 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000348585 2021-02-10T11:30:19.037 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.031 7f0d0dc87700 1 Processor -- start 2021-02-10T11:30:19.037 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.031 7f0d0dc87700 1 -- start start 2021-02-10T11:30:19.037 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.032 7f0d0dc87700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0d081b0470 0x7f0d081b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:19.037 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.032 7f0d0dc87700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f0d080ef610 0x7f0d080efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:19.037 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.032 7f0d0dc87700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f0d080f0030 0x7f0d080f0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:19.037 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.032 7f0d0dc87700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d080e5ea0 con 0x7f0d080ef610 2021-02-10T11:30:19.037 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.032 7f0d0dc87700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d080f0b50 con 0x7f0d080f0030 2021-02-10T11:30:19.037 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.032 7f0d0dc87700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d080e52e0 con 0x7f0d081b0470 2021-02-10T11:30:19.038 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.032 7f0d077fe700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0d081b0470 0x7f0d081b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:19.038 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.032 7f0d06ffd700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f0d080ef610 0x7f0d080efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:19.038 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.032 7f0d07fff700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f0d080f0030 0x7f0d080f0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:19.038 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.032 7f0d077fe700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0d081b0470 0x7f0d081b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3301/0 says I am v2:172.21.2.112:46484/0 (socket says 172.21.2.112:46484) 2021-02-10T11:30:19.038 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.032 7f0d06ffd700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f0d080ef610 0x7f0d080efa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:46802/0 (socket says 172.21.2.112:46802) 2021-02-10T11:30:19.038 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.032 7f0d07fff700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f0d080f0030 0x7f0d080f0430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3300/0 says I am v2:172.21.2.112:49082/0 (socket says 172.21.2.112:49082) 2021-02-10T11:30:19.038 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.032 7f0d077fe700 1 -- 172.21.2.112:0/1787001961 learned_addr learned my addr 172.21.2.112:0/1787001961 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:30:19.038 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.032 7f0d06ffd700 1 -- 172.21.2.112:0/1787001961 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0d081b0470 msgr2=0x7f0d081b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:19.038 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.032 7f0d06ffd700 1 --2- 172.21.2.112:0/1787001961 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0d081b0470 0x7f0d081b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:19.038 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.032 7f0d06ffd700 1 -- 172.21.2.112:0/1787001961 >> v2:172.21.2.120:3300/0 conn(0x7f0d080f0030 msgr2=0x7f0d080f0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:19.038 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.032 7f0d06ffd700 1 --2- 172.21.2.112:0/1787001961 >> v2:172.21.2.120:3300/0 conn(0x7f0d080f0030 0x7f0d080f0430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:19.038 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.032 7f0d06ffd700 1 -- 172.21.2.112:0/1787001961 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d080f10d0 con 0x7f0d080ef610 2021-02-10T11:30:19.038 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.032 7f0d077fe700 1 --2- 172.21.2.112:0/1787001961 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0d081b0470 0x7f0d081b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:30:19.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.032 7f0d07fff700 1 --2- 172.21.2.112:0/1787001961 >> v2:172.21.2.120:3300/0 conn(0x7f0d080f0030 0x7f0d080f0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:30:19.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.032 7f0d06ffd700 1 --2- 172.21.2.112:0/1787001961 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0d080ef610 0x7f0d080efa10 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 rx=0x7f0cfc006650 tx=0x7f0cfc009a80).ready entity=mon.0 client_cookie=700dceccf4d26971 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:19.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.033 7f0d067fc700 1 -- 172.21.2.112:0/1787001961 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0cfc0009b0 con 0x7f0d080ef610 2021-02-10T11:30:19.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.033 7f0d067fc700 1 -- 172.21.2.112:0/1787001961 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cfc0054f0 con 0x7f0d080ef610 2021-02-10T11:30:19.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.033 7f0d067fc700 1 -- 172.21.2.112:0/1787001961 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0cfc005940 con 0x7f0d080ef610 2021-02-10T11:30:19.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.033 7f0d0dc87700 1 -- 172.21.2.112:0/1787001961 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0d080ef610 msgr2=0x7f0d080efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:19.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.033 7f0d0dc87700 1 --2- 172.21.2.112:0/1787001961 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0d080ef610 0x7f0d080efa10 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 rx=0x7f0cfc006650 tx=0x7f0cfc009a80).stop 2021-02-10T11:30:19.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.033 7f0d0dc87700 1 -- 172.21.2.112:0/1787001961 shutdown_connections 2021-02-10T11:30:19.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.033 7f0d0dc87700 1 --2- 172.21.2.112:0/1787001961 >> v2:172.21.2.120:3300/0 conn(0x7f0d080f0030 0x7f0d080f0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:19.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.033 7f0d0dc87700 1 --2- 172.21.2.112:0/1787001961 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0d080ef610 0x7f0d080efa10 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:19.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.033 7f0d0dc87700 1 --2- 172.21.2.112:0/1787001961 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0d081b0470 0x7f0d081b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:19.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.033 7f0d0dc87700 1 -- 172.21.2.112:0/1787001961 >> 172.21.2.112:0/1787001961 conn(0x7f0d080ce3e0 msgr2=0x7f0d080e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:19.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.033 7f0d0dc87700 1 -- 172.21.2.112:0/1787001961 shutdown_connections 2021-02-10T11:30:19.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.033 7f0d0dc87700 1 -- 172.21.2.112:0/1787001961 wait complete. 2021-02-10T11:30:19.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.048 7f0d0dc87700 1 Processor -- start 2021-02-10T11:30:19.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.048 7f0d0dc87700 1 -- start start 2021-02-10T11:30:19.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.049 7f0d0dc87700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0d0809b930 0x7f0d080b7c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:19.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.049 7f0d0dc87700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0d080b8410 0x7f0d080b8810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:19.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.049 7f0d0dc87700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0d080c1f80 0x7f0d080c2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:19.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.049 7f0d0dc87700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d080c1a00 con 0x7f0d080c1f80 2021-02-10T11:30:19.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.049 7f0d0dc87700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d080b8e30 con 0x7f0d080b8410 2021-02-10T11:30:19.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.049 7f0d0dc87700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d080b9050 con 0x7f0d0809b930 2021-02-10T11:30:19.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.049 7f0d07fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0d080c1f80 0x7f0d080c2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:19.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.049 7f0d07fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0d080c1f80 0x7f0d080c2380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:46812/0 (socket says 172.21.2.112:46812) 2021-02-10T11:30:19.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.049 7f0d07fff700 1 -- 172.21.2.112:0/462404658 learned_addr learned my addr 172.21.2.112:0/462404658 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:30:19.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.049 7f0d06ffd700 1 --2- 172.21.2.112:0/462404658 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0d080b8410 0x7f0d080b8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:19.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.049 7f0d077fe700 1 --2- 172.21.2.112:0/462404658 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0d0809b930 0x7f0d080b7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:19.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.049 7f0d06ffd700 1 -- 172.21.2.112:0/462404658 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0d0809b930 msgr2=0x7f0d080b7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:19.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.049 7f0d06ffd700 1 --2- 172.21.2.112:0/462404658 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0d0809b930 0x7f0d080b7c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:19.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.049 7f0d06ffd700 1 -- 172.21.2.112:0/462404658 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0d080c1f80 msgr2=0x7f0d080c2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:19.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.049 7f0d06ffd700 1 --2- 172.21.2.112:0/462404658 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0d080c1f80 0x7f0d080c2380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:19.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.049 7f0d06ffd700 1 -- 172.21.2.112:0/462404658 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d080c29a0 con 0x7f0d080b8410 2021-02-10T11:30:19.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.049 7f0d06ffd700 1 --2- 172.21.2.112:0/462404658 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0d080b8410 0x7f0d080b8810 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7f0cfc009240 tx=0x7f0cfc00b8e0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:19.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.050 7f0ceffff700 1 -- 172.21.2.112:0/462404658 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0cfc009280 con 0x7f0d080b8410 2021-02-10T11:30:19.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.050 7f0ceffff700 1 -- 172.21.2.112:0/462404658 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cfc000c90 con 0x7f0d080b8410 2021-02-10T11:30:19.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.050 7f0d0dc87700 1 -- 172.21.2.112:0/462404658 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d080c29a0 con 0x7f0d080b8410 2021-02-10T11:30:19.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.050 7f0d0dc87700 1 -- 172.21.2.112:0/462404658 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d080c29a0 con 0x7f0d080b8410 2021-02-10T11:30:19.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.050 7f0ceffff700 1 -- 172.21.2.112:0/462404658 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0cfc0022b0 con 0x7f0d080b8410 2021-02-10T11:30:19.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.050 7f0ceffff700 1 -- 172.21.2.112:0/462404658 <== mon.1 v2:172.21.2.120:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0cfc009280 con 0x7f0d080b8410 2021-02-10T11:30:19.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.051 7f0ceffff700 1 --2- 172.21.2.112:0/462404658 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0ce8018730 0x7f0ce801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:19.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.051 7f0ceffff700 1 -- 172.21.2.112:0/462404658 <== mon.1 v2:172.21.2.120:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f0cfc0765c0 con 0x7f0d080b8410 2021-02-10T11:30:19.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.051 7f0d077fe700 1 --2- 172.21.2.112:0/462404658 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0ce8018730 0x7f0ce801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:19.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.051 7f0d077fe700 1 --2- 172.21.2.112:0/462404658 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0ce8018730 0x7f0ce801abc0 crc :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:19.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.079 7f0d0dc87700 1 -- 172.21.2.112:0/462404658 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d08105b30 con 0x7f0d080b8410 2021-02-10T11:30:19.069 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.082 7f0ceffff700 1 -- 172.21.2.112:0/462404658 <== mon.1 v2:172.21.2.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0cfc00bcd0 con 0x7f0d080b8410 2021-02-10T11:30:19.203 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.216 7f0d0dc87700 1 -- 172.21.2.112:0/462404658 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348585", "target": ["mgr", ""]}) v1 -- 0x7f0d08105b30 con 0x7f0ce8018730 2021-02-10T11:30:19.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.224 7f0ceffff700 1 -- 172.21.2.112:0/462404658 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f0cf80009b0 con 0x7f0ce8018730 2021-02-10T11:30:19.220 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.232 7f0d0dc87700 1 -- 172.21.2.112:0/462404658 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0ce8018730 msgr2=0x7f0ce801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:19.220 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.232 7f0d0dc87700 1 --2- 172.21.2.112:0/462404658 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0ce8018730 0x7f0ce801abc0 crc :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:19.220 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.232 7f0d0dc87700 1 -- 172.21.2.112:0/462404658 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0d080b8410 msgr2=0x7f0d080b8810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:19.220 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.232 7f0d0dc87700 1 --2- 172.21.2.112:0/462404658 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0d080b8410 0x7f0d080b8810 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7f0cfc009240 tx=0x7f0cfc00b8e0).stop 2021-02-10T11:30:19.220 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.232 7f0d0dc87700 1 -- 172.21.2.112:0/462404658 shutdown_connections 2021-02-10T11:30:19.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.232 7f0d0dc87700 1 --2- 172.21.2.112:0/462404658 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0ce8018730 0x7f0ce801abc0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:19.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.232 7f0d0dc87700 1 --2- 172.21.2.112:0/462404658 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0d080b8410 0x7f0d080b8810 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:19.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.232 7f0d0dc87700 1 --2- 172.21.2.112:0/462404658 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0d080c1f80 0x7f0d080c2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:19.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.232 7f0d0dc87700 1 --2- 172.21.2.112:0/462404658 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0d0809b930 0x7f0d080b7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:19.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.232 7f0d0dc87700 1 -- 172.21.2.112:0/462404658 >> 172.21.2.112:0/462404658 conn(0x7f0d08094e10 msgr2=0x7f0d080a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:19.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.233 7f0d0dc87700 1 -- 172.21.2.112:0/462404658 shutdown_connections 2021-02-10T11:30:19.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.233 7f0d0dc87700 1 -- 172.21.2.112:0/462404658 wait complete. 2021-02-10T11:30:19.229 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000348586 2021-02-10T11:30:19.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.654 7f11c8a9f700 1 Processor -- start 2021-02-10T11:30:19.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.654 7f11c8a9f700 1 -- start start 2021-02-10T11:30:19.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.655 7f11c8a9f700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f11c41b0870 0x7f11c40f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:19.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.655 7f11c8a9f700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f11c40efe00 0x7f11c40f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:19.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.655 7f11c8a9f700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11c41b0c70 con 0x7f11c41b0470 2021-02-10T11:30:19.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.655 7f11c8a9f700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11c40e5ea0 con 0x7f11c41b0870 2021-02-10T11:30:19.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.655 7f11c8a9f700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11c40f0b50 con 0x7f11c40efe00 2021-02-10T11:30:19.660 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.655 7f11c205d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f11c41b0870 0x7f11c40f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:19.660 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.655 7f11c285e700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f11c41b0470 0x7f11c40ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:46278/0 (socket says 172.21.2.112:46278) 2021-02-10T11:30:19.660 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.655 7f11c285e700 1 -- 172.21.2.112:0/825749219 learned_addr learned my addr 172.21.2.112:0/825749219 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:30:19.660 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.655 7f11c305f700 1 --2- 172.21.2.112:0/825749219 >> v2:172.21.2.120:3300/0 conn(0x7f11c40efe00 0x7f11c40f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:19.660 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.656 7f11c205d700 1 -- 172.21.2.112:0/825749219 >> v2:172.21.2.120:3300/0 conn(0x7f11c40efe00 msgr2=0x7f11c40f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:19.660 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.656 7f11c205d700 1 --2- 172.21.2.112:0/825749219 >> v2:172.21.2.120:3300/0 conn(0x7f11c40efe00 0x7f11c40f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:19.660 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.656 7f11c205d700 1 -- 172.21.2.112:0/825749219 >> v1:172.21.2.112:6789/0 conn(0x7f11c41b0470 legacy=0x7f11c40ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:19.660 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.656 7f11c205d700 1 -- 172.21.2.112:0/825749219 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11c40f0820 con 0x7f11c41b0870 2021-02-10T11:30:19.660 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.656 7f11c205d700 1 --2- 172.21.2.112:0/825749219 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f11c41b0870 0x7f11c40f20e0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 rx=0x7f11ac006650 tx=0x7f11ac009a80).ready entity=mon.0 client_cookie=b705b9b8d5b42367 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:19.660 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.656 7f11c185c700 1 -- 172.21.2.112:0/825749219 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f11ac0009b0 con 0x7f11c41b0870 2021-02-10T11:30:19.660 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.656 7f11c185c700 1 -- 172.21.2.112:0/825749219 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11ac00c0c0 con 0x7f11c41b0870 2021-02-10T11:30:19.661 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.656 7f11c185c700 1 -- 172.21.2.112:0/825749219 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f11ac00c520 con 0x7f11c41b0870 2021-02-10T11:30:19.661 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.656 7f11c8a9f700 1 -- 172.21.2.112:0/825749219 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f11c41b0870 msgr2=0x7f11c40f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:19.661 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.656 7f11c8a9f700 1 --2- 172.21.2.112:0/825749219 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f11c41b0870 0x7f11c40f20e0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 rx=0x7f11ac006650 tx=0x7f11ac009a80).stop 2021-02-10T11:30:19.661 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.657 7f11c8a9f700 1 -- 172.21.2.112:0/825749219 shutdown_connections 2021-02-10T11:30:19.661 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.657 7f11c8a9f700 1 --2- 172.21.2.112:0/825749219 >> v2:172.21.2.120:3300/0 conn(0x7f11c40efe00 0x7f11c40f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:19.661 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.657 7f11c8a9f700 1 --2- 172.21.2.112:0/825749219 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f11c41b0870 0x7f11c40f20e0 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:19.661 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.657 7f11c8a9f700 1 -- 172.21.2.112:0/825749219 >> 172.21.2.112:0/825749219 conn(0x7f11c40ce3e0 msgr2=0x7f11c40e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:19.661 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.657 7f11c8a9f700 1 -- 172.21.2.112:0/825749219 shutdown_connections 2021-02-10T11:30:19.661 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.657 7f11c8a9f700 1 -- 172.21.2.112:0/825749219 wait complete. 2021-02-10T11:30:19.663 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.672 7f11c8a9f700 1 Processor -- start 2021-02-10T11:30:19.663 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.672 7f11c8a9f700 1 -- start start 2021-02-10T11:30:19.663 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.672 7f11c8a9f700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f11c4190920 0x7f11c40b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:19.663 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.672 7f11c8a9f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f11c40b8200 0x7f11c40b8600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:19.663 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.672 7f11c8a9f700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f11c40c1d80 0x7f11c40c2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:19.663 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.672 7f11c8a9f700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11c4190da0 con 0x7f11c40b8200 2021-02-10T11:30:19.663 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.672 7f11c8a9f700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11c40b8c20 con 0x7f11c4190920 2021-02-10T11:30:19.663 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.672 7f11c8a9f700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11c40b8e10 con 0x7f11c40c1d80 2021-02-10T11:30:19.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.673 7f11c205d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f11c40b8200 0x7f11c40b8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:19.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.673 7f11c205d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f11c40b8200 0x7f11c40b8600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:46824/0 (socket says 172.21.2.112:46824) 2021-02-10T11:30:19.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.673 7f11c205d700 1 -- 172.21.2.112:0/303139058 learned_addr learned my addr 172.21.2.112:0/303139058 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:30:19.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.673 7f11c285e700 1 --2- 172.21.2.112:0/303139058 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f11c4190920 0x7f11c40b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:19.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.673 7f11c305f700 1 --2- 172.21.2.112:0/303139058 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f11c40c1d80 0x7f11c40c2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:19.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.673 7f11c205d700 1 -- 172.21.2.112:0/303139058 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f11c40c1d80 msgr2=0x7f11c40c2180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:19.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.673 7f11c205d700 1 --2- 172.21.2.112:0/303139058 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f11c40c1d80 0x7f11c40c2180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:19.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.673 7f11c205d700 1 -- 172.21.2.112:0/303139058 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f11c4190920 msgr2=0x7f11c40b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:19.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.673 7f11c205d700 1 --2- 172.21.2.112:0/303139058 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f11c4190920 0x7f11c40b7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:19.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.673 7f11c205d700 1 -- 172.21.2.112:0/303139058 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11c40c27d0 con 0x7f11c40b8200 2021-02-10T11:30:19.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.673 7f11c205d700 1 --2- 172.21.2.112:0/303139058 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f11c40b8200 0x7f11c40b8600 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 rx=0x7f11ac009630 tx=0x7f11ac00b900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:19.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.673 7f11bb7fe700 1 -- 172.21.2.112:0/303139058 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f11ac0093a0 con 0x7f11c40b8200 2021-02-10T11:30:19.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.673 7f11bb7fe700 1 -- 172.21.2.112:0/303139058 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11ac00bcf0 con 0x7f11c40b8200 2021-02-10T11:30:19.665 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.673 7f11c8a9f700 1 -- 172.21.2.112:0/303139058 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11c40c27d0 con 0x7f11c40b8200 2021-02-10T11:30:19.665 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.673 7f11c8a9f700 1 -- 172.21.2.112:0/303139058 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11c40c2d60 con 0x7f11c40b8200 2021-02-10T11:30:19.665 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.673 7f11bb7fe700 1 -- 172.21.2.112:0/303139058 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f11ac005560 con 0x7f11c40b8200 2021-02-10T11:30:19.665 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.674 7f11bb7fe700 1 -- 172.21.2.112:0/303139058 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f11ac00a000 con 0x7f11c40b8200 2021-02-10T11:30:19.665 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.674 7f11bb7fe700 1 --2- 172.21.2.112:0/303139058 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f11a8018730 0x7f11a801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:19.665 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.674 7f11bb7fe700 1 -- 172.21.2.112:0/303139058 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f11ac0327b0 con 0x7f11c40b8200 2021-02-10T11:30:19.665 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.674 7f11c285e700 1 --2- 172.21.2.112:0/303139058 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f11a8018730 0x7f11a801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:19.665 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.675 7f11c285e700 1 --2- 172.21.2.112:0/303139058 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f11a8018730 0x7f11a801ac10 crc :-1 s=READY pgs=405 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:19.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.705 7f11c8a9f700 1 -- 172.21.2.112:0/303139058 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11c4105ae0 con 0x7f11c40b8200 2021-02-10T11:30:19.695 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.707 7f11bb7fe700 1 -- 172.21.2.112:0/303139058 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f11ac009340 con 0x7f11c40b8200 2021-02-10T11:30:19.823 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.835 7f11c8a9f700 1 -- 172.21.2.112:0/303139058 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348586", "target": ["mgr", ""]}) v1 -- 0x7f11c40c2d60 con 0x7f11a8018730 2021-02-10T11:30:19.833 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.846 7f11bb7fe700 1 -- 172.21.2.112:0/303139058 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f11bc000ce0 con 0x7f11a8018730 2021-02-10T11:30:19.841 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.853 7f11c8a9f700 1 -- 172.21.2.112:0/303139058 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f11a8018730 msgr2=0x7f11a801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:19.841 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.853 7f11c8a9f700 1 --2- 172.21.2.112:0/303139058 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f11a8018730 0x7f11a801ac10 crc :-1 s=READY pgs=405 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:19.841 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.853 7f11c8a9f700 1 -- 172.21.2.112:0/303139058 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f11c40b8200 msgr2=0x7f11c40b8600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:19.841 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.853 7f11c8a9f700 1 --2- 172.21.2.112:0/303139058 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f11c40b8200 0x7f11c40b8600 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 rx=0x7f11ac009630 tx=0x7f11ac00b900).stop 2021-02-10T11:30:19.841 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.853 7f11c8a9f700 1 -- 172.21.2.112:0/303139058 shutdown_connections 2021-02-10T11:30:19.841 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.853 7f11c8a9f700 1 --2- 172.21.2.112:0/303139058 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f11a8018730 0x7f11a801ac10 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:19.842 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.853 7f11c8a9f700 1 --2- 172.21.2.112:0/303139058 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f11c40c1d80 0x7f11c40c2180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:19.842 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.853 7f11c8a9f700 1 --2- 172.21.2.112:0/303139058 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f11c40b8200 0x7f11c40b8600 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:19.842 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.853 7f11c8a9f700 1 --2- 172.21.2.112:0/303139058 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f11c4190920 0x7f11c40b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:19.842 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.853 7f11c8a9f700 1 -- 172.21.2.112:0/303139058 >> 172.21.2.112:0/303139058 conn(0x7f11c4094e10 msgr2=0x7f11c40a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:19.842 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.853 7f11c8a9f700 1 -- 172.21.2.112:0/303139058 shutdown_connections 2021-02-10T11:30:19.842 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:19.853 7f11c8a9f700 1 -- 172.21.2.112:0/303139058 wait complete. 2021-02-10T11:30:19.850 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000348587 2021-02-10T11:30:20.280 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.275 7f2328fd9700 1 Processor -- start 2021-02-10T11:30:20.280 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.275 7f2328fd9700 1 -- start start 2021-02-10T11:30:20.280 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.276 7f2328fd9700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f23241b0870 0x7f23240f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:20.280 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.276 7f2328fd9700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f23240efe00 0x7f23240f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:20.280 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.276 7f2328fd9700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23241b0c70 con 0x7f23241b0470 2021-02-10T11:30:20.280 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.276 7f2328fd9700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23240f0b50 con 0x7f23240efe00 2021-02-10T11:30:20.280 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.276 7f2328fd9700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23240f0820 con 0x7f23241b0870 2021-02-10T11:30:20.280 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.276 7f2322d9d700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f23241b0470 0x7f23240ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:46290/0 (socket says 172.21.2.112:46290) 2021-02-10T11:30:20.281 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.276 7f2322d9d700 1 -- 172.21.2.112:0/2279617980 learned_addr learned my addr 172.21.2.112:0/2279617980 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:30:20.281 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.276 7f232359e700 1 --2- 172.21.2.112:0/2279617980 >> v2:172.21.2.112:3300/0 conn(0x7f23240efe00 0x7f23240f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:20.281 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.276 7f232259c700 1 --2- 172.21.2.112:0/2279617980 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f23241b0870 0x7f23240f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:20.281 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.277 7f2321d9b700 1 -- 172.21.2.112:0/2279617980 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3324905057 0 0) 0x7f231c001060 con 0x7f23241b0470 2021-02-10T11:30:20.281 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.277 7f2321d9b700 1 -- 172.21.2.112:0/2279617980 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f230c002fe0 con 0x7f23241b0470 2021-02-10T11:30:20.281 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.277 7f232259c700 1 -- 172.21.2.112:0/2279617980 >> v2:172.21.2.112:3300/0 conn(0x7f23240efe00 msgr2=0x7f23240f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:20.281 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.277 7f232259c700 1 --2- 172.21.2.112:0/2279617980 >> v2:172.21.2.112:3300/0 conn(0x7f23240efe00 0x7f23240f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:20.281 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.277 7f232259c700 1 -- 172.21.2.112:0/2279617980 >> v1:172.21.2.112:6789/0 conn(0x7f23241b0470 legacy=0x7f23240ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:20.281 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.277 7f232259c700 1 -- 172.21.2.112:0/2279617980 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23240e52e0 con 0x7f23241b0870 2021-02-10T11:30:20.281 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.277 7f232359e700 1 --2- 172.21.2.112:0/2279617980 >> v2:172.21.2.112:3300/0 conn(0x7f23240efe00 0x7f23240f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:30:20.281 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.277 7f232259c700 1 --2- 172.21.2.112:0/2279617980 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f23241b0870 0x7f23240f20e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f2318006650 tx=0x7f2318009a80).ready entity=mon.2 client_cookie=2f1ba7cec4de2275 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:20.281 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.277 7f2321d9b700 1 -- 172.21.2.112:0/2279617980 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f23180009b0 con 0x7f23241b0870 2021-02-10T11:30:20.282 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.277 7f2321d9b700 1 -- 172.21.2.112:0/2279617980 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f231800c0c0 con 0x7f23241b0870 2021-02-10T11:30:20.282 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.277 7f2321d9b700 1 -- 172.21.2.112:0/2279617980 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f231800c5e0 con 0x7f23241b0870 2021-02-10T11:30:20.282 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.277 7f2328fd9700 1 -- 172.21.2.112:0/2279617980 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f23241b0870 msgr2=0x7f23240f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:20.282 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.277 7f2328fd9700 1 --2- 172.21.2.112:0/2279617980 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f23241b0870 0x7f23240f20e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f2318006650 tx=0x7f2318009a80).stop 2021-02-10T11:30:20.282 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.278 7f2328fd9700 1 -- 172.21.2.112:0/2279617980 shutdown_connections 2021-02-10T11:30:20.282 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.278 7f2328fd9700 1 --2- 172.21.2.112:0/2279617980 >> v2:172.21.2.112:3300/0 conn(0x7f23240efe00 0x7f23240f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:20.282 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.278 7f2328fd9700 1 --2- 172.21.2.112:0/2279617980 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f23241b0870 0x7f23240f20e0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:20.282 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.278 7f2328fd9700 1 -- 172.21.2.112:0/2279617980 >> 172.21.2.112:0/2279617980 conn(0x7f23240ce3e0 msgr2=0x7f23240e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:20.282 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.278 7f2328fd9700 1 -- 172.21.2.112:0/2279617980 shutdown_connections 2021-02-10T11:30:20.282 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.278 7f2328fd9700 1 -- 172.21.2.112:0/2279617980 wait complete. 2021-02-10T11:30:20.284 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.293 7f2328fd9700 1 Processor -- start 2021-02-10T11:30:20.284 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.293 7f2328fd9700 1 -- start start 2021-02-10T11:30:20.284 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.293 7f2328fd9700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f23240a0420 0x7f23240a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:20.284 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.293 7f2328fd9700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2324189d00 0x7f232418a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:20.284 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.293 7f2328fd9700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f232418a720 0x7f23240b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:20.284 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.293 7f2328fd9700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2324189800 con 0x7f232418a720 2021-02-10T11:30:20.284 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.293 7f2328fd9700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f232418ab20 con 0x7f23240a0420 2021-02-10T11:30:20.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.293 7f2328fd9700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23240a1140 con 0x7f2324189d00 2021-02-10T11:30:20.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.293 7f232359e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f232418a720 0x7f23240b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:20.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.293 7f232259c700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2324189d00 0x7f232418a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:20.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.294 7f232359e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f232418a720 0x7f23240b7be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:46838/0 (socket says 172.21.2.112:46838) 2021-02-10T11:30:20.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.294 7f232359e700 1 -- 172.21.2.112:0/1558770073 learned_addr learned my addr 172.21.2.112:0/1558770073 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:30:20.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.294 7f2322d9d700 1 --2- 172.21.2.112:0/1558770073 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f23240a0420 0x7f23240a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:20.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.294 7f232359e700 1 -- 172.21.2.112:0/1558770073 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2324189d00 msgr2=0x7f232418a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:20.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.294 7f232359e700 1 --2- 172.21.2.112:0/1558770073 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2324189d00 0x7f232418a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:20.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.294 7f232359e700 1 -- 172.21.2.112:0/1558770073 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f23240a0420 msgr2=0x7f23240a0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:20.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.294 7f232359e700 1 --2- 172.21.2.112:0/1558770073 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f23240a0420 0x7f23240a0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:20.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.294 7f232359e700 1 -- 172.21.2.112:0/1558770073 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23240b8200 con 0x7f232418a720 2021-02-10T11:30:20.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.294 7f232259c700 1 --2- 172.21.2.112:0/1558770073 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2324189d00 0x7f232418a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:30:20.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.294 7f232359e700 1 --2- 172.21.2.112:0/1558770073 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f232418a720 0x7f23240b7be0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 rx=0x7f231400b5e0 tx=0x7f231400b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:20.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.294 7f23137fe700 1 -- 172.21.2.112:0/1558770073 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f231400bd70 con 0x7f232418a720 2021-02-10T11:30:20.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.294 7f23137fe700 1 -- 172.21.2.112:0/1558770073 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23140022e0 con 0x7f232418a720 2021-02-10T11:30:20.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.294 7f2328fd9700 1 -- 172.21.2.112:0/1558770073 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23240b8200 con 0x7f232418a720 2021-02-10T11:30:20.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.294 7f23137fe700 1 -- 172.21.2.112:0/1558770073 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2314002800 con 0x7f232418a720 2021-02-10T11:30:20.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.294 7f2328fd9700 1 -- 172.21.2.112:0/1558770073 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23240b8200 con 0x7f232418a720 2021-02-10T11:30:20.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.295 7f23137fe700 1 -- 172.21.2.112:0/1558770073 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f23140024c0 con 0x7f232418a720 2021-02-10T11:30:20.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.295 7f23137fe700 1 --2- 172.21.2.112:0/1558770073 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f23080186c0 0x7f230801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:20.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.295 7f23137fe700 1 -- 172.21.2.112:0/1558770073 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f23140325a0 con 0x7f232418a720 2021-02-10T11:30:20.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.295 7f2322d9d700 1 --2- 172.21.2.112:0/1558770073 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f23080186c0 0x7f230801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:20.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.296 7f2322d9d700 1 --2- 172.21.2.112:0/1558770073 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f23080186c0 0x7f230801abc0 crc :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:20.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.325 7f2328fd9700 1 -- 172.21.2.112:0/1558770073 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f232418ab20 con 0x7f232418a720 2021-02-10T11:30:20.315 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.328 7f23137fe700 1 -- 172.21.2.112:0/1558770073 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f231401d8c0 con 0x7f232418a720 2021-02-10T11:30:20.442 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.454 7f2328fd9700 1 -- 172.21.2.112:0/1558770073 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348587", "target": ["mgr", ""]}) v1 -- 0x7f232418ab20 con 0x7f23080186c0 2021-02-10T11:30:20.453 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.465 7f23137fe700 1 -- 172.21.2.112:0/1558770073 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f231c0069a0 con 0x7f23080186c0 2021-02-10T11:30:20.460 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.472 7f2328fd9700 1 -- 172.21.2.112:0/1558770073 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f23080186c0 msgr2=0x7f230801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:20.460 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.472 7f2328fd9700 1 --2- 172.21.2.112:0/1558770073 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f23080186c0 0x7f230801abc0 crc :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:20.460 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.472 7f2328fd9700 1 -- 172.21.2.112:0/1558770073 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f232418a720 msgr2=0x7f23240b7be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:20.460 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.472 7f2328fd9700 1 --2- 172.21.2.112:0/1558770073 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f232418a720 0x7f23240b7be0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 rx=0x7f231400b5e0 tx=0x7f231400b980).stop 2021-02-10T11:30:20.460 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.472 7f2328fd9700 1 -- 172.21.2.112:0/1558770073 shutdown_connections 2021-02-10T11:30:20.461 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.473 7f2328fd9700 1 --2- 172.21.2.112:0/1558770073 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f23080186c0 0x7f230801abc0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:20.461 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.473 7f2328fd9700 1 --2- 172.21.2.112:0/1558770073 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f232418a720 0x7f23240b7be0 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:20.461 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.473 7f2328fd9700 1 --2- 172.21.2.112:0/1558770073 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2324189d00 0x7f232418a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:20.461 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.473 7f2328fd9700 1 --2- 172.21.2.112:0/1558770073 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f23240a0420 0x7f23240a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:20.461 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.473 7f2328fd9700 1 -- 172.21.2.112:0/1558770073 >> 172.21.2.112:0/1558770073 conn(0x7f2324094e10 msgr2=0x7f23240a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:20.461 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.473 7f2328fd9700 1 -- 172.21.2.112:0/1558770073 shutdown_connections 2021-02-10T11:30:20.461 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.473 7f2328fd9700 1 -- 172.21.2.112:0/1558770073 wait complete. 2021-02-10T11:30:20.469 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000348588 2021-02-10T11:30:20.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.893 7f5e3a248700 1 Processor -- start 2021-02-10T11:30:20.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.893 7f5e3a248700 1 -- start start 2021-02-10T11:30:20.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.894 7f5e3a248700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5e341b0470 0x7f5e341b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:20.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.894 7f5e3a248700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f5e340ef610 0x7f5e340efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:20.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.894 7f5e3a248700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e340f0b50 con 0x7f5e340f0030 2021-02-10T11:30:20.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.894 7f5e3a248700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e340e52e0 con 0x7f5e340ef610 2021-02-10T11:30:20.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.894 7f5e3a248700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e340e5490 con 0x7f5e341b0470 2021-02-10T11:30:20.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.894 7f5e38865700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f5e340f0030 0x7f5e340f10d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:46308/0 (socket says 172.21.2.112:46308) 2021-02-10T11:30:20.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.894 7f5e38865700 1 -- 172.21.2.112:0/1409939699 learned_addr learned my addr 172.21.2.112:0/1409939699 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:30:20.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.894 7f5e337fe700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f5e340ef610 0x7f5e340efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:20.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.894 7f5e33fff700 1 --2- 172.21.2.112:0/1409939699 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5e341b0470 0x7f5e341b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:20.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.894 7f5e32ffd700 1 -- 172.21.2.112:0/1409939699 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2105309858 0 0) 0x7f5e24000c10 con 0x7f5e340f0030 2021-02-10T11:30:20.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.894 7f5e32ffd700 1 -- 172.21.2.112:0/1409939699 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e1c002fe0 con 0x7f5e340f0030 2021-02-10T11:30:20.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.894 7f5e337fe700 1 -- 172.21.2.112:0/1409939699 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5e341b0470 msgr2=0x7f5e341b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:20.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.894 7f5e337fe700 1 --2- 172.21.2.112:0/1409939699 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5e341b0470 0x7f5e341b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:20.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.894 7f5e337fe700 1 -- 172.21.2.112:0/1409939699 >> v1:172.21.2.112:6789/0 conn(0x7f5e340f0030 legacy=0x7f5e340f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:20.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.894 7f5e337fe700 1 -- 172.21.2.112:0/1409939699 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e340f0430 con 0x7f5e340ef610 2021-02-10T11:30:20.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.894 7f5e33fff700 1 --2- 172.21.2.112:0/1409939699 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5e341b0470 0x7f5e341b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:30:20.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.895 7f5e337fe700 1 --2- 172.21.2.112:0/1409939699 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5e340ef610 0x7f5e340efa10 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0x7f5e28006650 tx=0x7f5e28009a80).ready entity=mon.1 client_cookie=933a497062702e95 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:20.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.895 7f5e32ffd700 1 -- 172.21.2.112:0/1409939699 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5e280009b0 con 0x7f5e340ef610 2021-02-10T11:30:20.900 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.895 7f5e32ffd700 1 -- 172.21.2.112:0/1409939699 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e2800c0c0 con 0x7f5e340ef610 2021-02-10T11:30:20.900 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.895 7f5e3a248700 1 -- 172.21.2.112:0/1409939699 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5e340ef610 msgr2=0x7f5e340efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:20.900 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.895 7f5e3a248700 1 --2- 172.21.2.112:0/1409939699 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5e340ef610 0x7f5e340efa10 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0x7f5e28006650 tx=0x7f5e28009a80).stop 2021-02-10T11:30:20.900 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.895 7f5e32ffd700 1 -- 172.21.2.112:0/1409939699 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5e2800c520 con 0x7f5e340ef610 2021-02-10T11:30:20.900 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.895 7f5e3a248700 1 -- 172.21.2.112:0/1409939699 shutdown_connections 2021-02-10T11:30:20.900 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.895 7f5e3a248700 1 --2- 172.21.2.112:0/1409939699 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5e340ef610 0x7f5e340efa10 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:20.900 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.895 7f5e3a248700 1 --2- 172.21.2.112:0/1409939699 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5e341b0470 0x7f5e341b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:20.901 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.895 7f5e3a248700 1 -- 172.21.2.112:0/1409939699 >> 172.21.2.112:0/1409939699 conn(0x7f5e340ce3e0 msgr2=0x7f5e340e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:20.901 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.896 7f5e3a248700 1 -- 172.21.2.112:0/1409939699 shutdown_connections 2021-02-10T11:30:20.901 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.896 7f5e3a248700 1 -- 172.21.2.112:0/1409939699 wait complete. 2021-02-10T11:30:20.901 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.910 7f5e3a248700 1 Processor -- start 2021-02-10T11:30:20.901 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.910 7f5e3a248700 1 -- start start 2021-02-10T11:30:20.902 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.911 7f5e3a248700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5e340c28d0 0x7f5e340c2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:20.902 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.911 7f5e3a248700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5e3408e690 0x7f5e3408ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:20.903 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.911 7f5e3a248700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5e340bc380 0x7f5e340bc780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:20.903 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.911 7f5e3a248700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e3408f340 con 0x7f5e340bc380 2021-02-10T11:30:20.903 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.911 7f5e3a248700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e3408f0b0 con 0x7f5e3408e690 2021-02-10T11:30:20.903 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.911 7f5e3a248700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e340c33b0 con 0x7f5e340c28d0 2021-02-10T11:30:20.903 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.911 7f5e38865700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5e340bc380 0x7f5e340bc780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:20.903 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.911 7f5e38865700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5e340bc380 0x7f5e340bc780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:46850/0 (socket says 172.21.2.112:46850) 2021-02-10T11:30:20.903 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.911 7f5e38865700 1 -- 172.21.2.112:0/1681174428 learned_addr learned my addr 172.21.2.112:0/1681174428 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:30:20.903 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.911 7f5e38865700 1 -- 172.21.2.112:0/1681174428 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5e340c28d0 msgr2=0x7f5e340c2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:20.903 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.911 7f5e38865700 1 --2- 172.21.2.112:0/1681174428 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5e340c28d0 0x7f5e340c2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:20.903 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.911 7f5e38865700 1 -- 172.21.2.112:0/1681174428 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5e3408e690 msgr2=0x7f5e3408ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:20.903 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.911 7f5e38865700 1 --2- 172.21.2.112:0/1681174428 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5e3408e690 0x7f5e3408ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:20.904 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.911 7f5e38865700 1 -- 172.21.2.112:0/1681174428 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e340bcda0 con 0x7f5e340bc380 2021-02-10T11:30:20.904 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.911 7f5e38865700 1 --2- 172.21.2.112:0/1681174428 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5e340bc380 0x7f5e340bc780 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 rx=0x7f5e24006f00 tx=0x7f5e2400b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:20.904 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.911 7f5e30ff9700 1 -- 172.21.2.112:0/1681174428 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5e2400bd30 con 0x7f5e340bc380 2021-02-10T11:30:20.904 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.912 7f5e30ff9700 1 -- 172.21.2.112:0/1681174428 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e24006020 con 0x7f5e340bc380 2021-02-10T11:30:20.904 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.912 7f5e30ff9700 1 -- 172.21.2.112:0/1681174428 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5e240043a0 con 0x7f5e340bc380 2021-02-10T11:30:20.904 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.912 7f5e3a248700 1 -- 172.21.2.112:0/1681174428 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e340bcda0 con 0x7f5e340bc380 2021-02-10T11:30:20.904 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.912 7f5e3a248700 1 -- 172.21.2.112:0/1681174428 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e340bcda0 con 0x7f5e340bc380 2021-02-10T11:30:20.904 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.912 7f5e30ff9700 1 -- 172.21.2.112:0/1681174428 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5e24000ac0 con 0x7f5e340bc380 2021-02-10T11:30:20.904 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.912 7f5e30ff9700 1 --2- 172.21.2.112:0/1681174428 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5e14018730 0x7f5e1401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:20.904 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.912 7f5e30ff9700 1 -- 172.21.2.112:0/1681174428 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f5e24032660 con 0x7f5e340bc380 2021-02-10T11:30:20.904 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.912 7f5e33fff700 1 --2- 172.21.2.112:0/1681174428 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5e14018730 0x7f5e1401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:20.905 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.913 7f5e33fff700 1 --2- 172.21.2.112:0/1681174428 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5e14018730 0x7f5e1401abc0 crc :-1 s=READY pgs=407 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:20.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.943 7f5e3a248700 1 -- 172.21.2.112:0/1681174428 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e34105ae0 con 0x7f5e340bc380 2021-02-10T11:30:20.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:20.946 7f5e30ff9700 1 -- 172.21.2.112:0/1681174428 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5e24006b90 con 0x7f5e340bc380 2021-02-10T11:30:21.063 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:21.076 7f5e3a248700 1 -- 172.21.2.112:0/1681174428 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348588", "target": ["mgr", ""]}) v1 -- 0x7f5e3408f0b0 con 0x7f5e14018730 2021-02-10T11:30:21.074 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:21.087 7f5e30ff9700 1 -- 172.21.2.112:0/1681174428 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f5e2c000d30 con 0x7f5e14018730 2021-02-10T11:30:21.081 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:21.093 7f5e3a248700 1 -- 172.21.2.112:0/1681174428 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5e14018730 msgr2=0x7f5e1401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:21.081 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:21.093 7f5e3a248700 1 --2- 172.21.2.112:0/1681174428 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5e14018730 0x7f5e1401abc0 crc :-1 s=READY pgs=407 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:21.081 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:21.093 7f5e3a248700 1 -- 172.21.2.112:0/1681174428 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5e340bc380 msgr2=0x7f5e340bc780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:21.081 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:21.093 7f5e3a248700 1 --2- 172.21.2.112:0/1681174428 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5e340bc380 0x7f5e340bc780 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 rx=0x7f5e24006f00 tx=0x7f5e2400b940).stop 2021-02-10T11:30:21.082 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:21.093 7f5e3a248700 1 -- 172.21.2.112:0/1681174428 shutdown_connections 2021-02-10T11:30:21.082 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:21.093 7f5e3a248700 1 --2- 172.21.2.112:0/1681174428 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5e14018730 0x7f5e1401abc0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:21.082 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:21.093 7f5e3a248700 1 --2- 172.21.2.112:0/1681174428 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5e3408e690 0x7f5e3408ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:21.082 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:21.093 7f5e3a248700 1 --2- 172.21.2.112:0/1681174428 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5e340bc380 0x7f5e340bc780 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:21.082 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:21.093 7f5e3a248700 1 --2- 172.21.2.112:0/1681174428 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5e340c28d0 0x7f5e340c2cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:21.082 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:21.093 7f5e3a248700 1 -- 172.21.2.112:0/1681174428 >> 172.21.2.112:0/1681174428 conn(0x7f5e340bbf60 msgr2=0x7f5e34105d10 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:21.082 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:21.094 7f5e3a248700 1 -- 172.21.2.112:0/1681174428 shutdown_connections 2021-02-10T11:30:21.082 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:21.094 7f5e3a248700 1 -- 172.21.2.112:0/1681174428 wait complete. 2021-02-10T11:30:21.090 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000348589 2021-02-10T11:30:21.518 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:21.513 7f9d60af9700 1 Processor -- start 2021-02-10T11:30:21.518 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:21.513 7f9d60af9700 1 -- start start 2021-02-10T11:30:21.518 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:21.513 7f9d60af9700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9d5c0f05f0 0x7f9d5c0f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:21.518 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:21.513 7f9d60af9700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d5c1b0c70 con 0x7f9d5c1b0470 2021-02-10T11:30:21.518 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:21.513 7f9d60af9700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d5c0f0b50 con 0x7f9d5c1b0870 2021-02-10T11:30:21.518 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:21.513 7f9d60af9700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d5c0e5ea0 con 0x7f9d5c0f05f0 2021-02-10T11:30:21.518 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:21.514 7f9d5a85e700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f9d5c1b0470 0x7f9d5c0ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:46318/0 (socket says 172.21.2.112:46318) 2021-02-10T11:30:21.518 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:21.514 7f9d5a85e700 1 -- 172.21.2.112:0/100456747 learned_addr learned my addr 172.21.2.112:0/100456747 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:30:21.518 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:21.514 7f9d5b05f700 1 --2- 172.21.2.112:0/100456747 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9d5c0f05f0 0x7f9d5c0f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:21.519 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:21.514 7f9d5985c700 1 -- 172.21.2.112:0/100456747 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 787945219 0 0) 0x7f9d54001060 con 0x7f9d5c1b0470 2021-02-10T11:30:21.519 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:21.514 7f9d5985c700 1 -- 172.21.2.112:0/100456747 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d44002fe0 con 0x7f9d5c1b0470 2021-02-10T11:30:21.519 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:21.514 7f9d5985c700 1 -- 172.21.2.112:0/100456747 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 331067254 0 0) 0x7f9d50000ff0 con 0x7f9d5c1b0870 2021-02-10T11:30:21.519 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:21.514 7f9d5985c700 1 -- 172.21.2.112:0/100456747 --> v1:172.21.2.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d44003210 con 0x7f9d5c1b0870 2021-02-10T11:30:21.519 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:21.514 7f9d5b05f700 1 -- 172.21.2.112:0/100456747 >> v1:172.21.2.120:6789/0 conn(0x7f9d5c1b0870 legacy=0x7f9d5c0efe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:21.519 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:21.514 7f9d5b05f700 1 -- 172.21.2.112:0/100456747 >> v1:172.21.2.112:6789/0 conn(0x7f9d5c1b0470 legacy=0x7f9d5c0ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:21.519 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:21.514 7f9d5b05f700 1 -- 172.21.2.112:0/100456747 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d5c0e52e0 con 0x7f9d5c0f05f0 2021-02-10T11:30:21.519 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:21.514 7f9d5b05f700 1 --2- 172.21.2.112:0/100456747 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9d5c0f05f0 0x7f9d5c0f20e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f9d4c006ad0 tx=0x7f9d4c009a50).ready entity=mon.2 client_cookie=813f91a4e3e9972b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:21.519 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:21.515 7f9d5985c700 1 -- 172.21.2.112:0/100456747 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9d4c0066a0 con 0x7f9d5c0f05f0 2021-02-10T11:30:21.519 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:21.515 7f9d5985c700 1 -- 172.21.2.112:0/100456747 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d4c006410 con 0x7f9d5c0f05f0 2021-02-10T11:30:21.519 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:21.515 7f9d5985c700 1 -- 172.21.2.112:0/100456747 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9d4c0024d0 con 0x7f9d5c0f05f0 2021-02-10T11:30:21.519 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:21.515 7f9d60af9700 1 -- 172.21.2.112:0/100456747 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9d5c0f05f0 msgr2=0x7f9d5c0f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:21.520 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:21.515 7f9d60af9700 1 --2- 172.21.2.112:0/100456747 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9d5c0f05f0 0x7f9d5c0f20e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f9d4c006ad0 tx=0x7f9d4c009a50).stop 2021-02-10T11:30:21.520 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:21.516 7f9d60af9700 1 -- 172.21.2.112:0/100456747 shutdown_connections 2021-02-10T11:30:21.520 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:21.516 7f9d60af9700 1 --2- 172.21.2.112:0/100456747 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9d5c0f05f0 0x7f9d5c0f20e0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:21.520 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:21.516 7f9d60af9700 1 -- 172.21.2.112:0/100456747 >> 172.21.2.112:0/100456747 conn(0x7f9d5c0ce3e0 msgr2=0x7f9d5c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:21.520 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:21.516 7f9d60af9700 1 -- 172.21.2.112:0/100456747 shutdown_connections 2021-02-10T11:30:21.520 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:21.516 7f9d60af9700 1 -- 172.21.2.112:0/100456747 wait complete. 2021-02-10T11:30:21.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:21.531 7f9d60af9700 1 Processor -- start 2021-02-10T11:30:21.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:21.531 7f9d60af9700 1 -- start start 2021-02-10T11:30:21.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:21.531 7f9d60af9700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9d5c105d10 0x7f9d5c09eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:21.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:21.531 7f9d60af9700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9d5c106110 0x7f9d5c09f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:21.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:21.531 7f9d60af9700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9d5c0b2a00 0x7f9d5c0b2e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:21.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:21.531 7f9d60af9700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d5c09e560 con 0x7f9d5c0b2a00 2021-02-10T11:30:21.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:21.531 7f9d60af9700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d5c09c090 con 0x7f9d5c106110 2021-02-10T11:30:21.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:21.531 7f9d60af9700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d5c0b3420 con 0x7f9d5c105d10 2021-02-10T11:30:21.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:21.531 7f9d5b05f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9d5c0b2a00 0x7f9d5c0b2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:21.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:21.531 7f9d5a85e700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9d5c105d10 0x7f9d5c09eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:21.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:21.531 7f9d5b05f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9d5c0b2a00 0x7f9d5c0b2e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:46864/0 (socket says 172.21.2.112:46864) 2021-02-10T11:30:21.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:21.531 7f9d5b05f700 1 -- 172.21.2.112:0/2686353547 learned_addr learned my addr 172.21.2.112:0/2686353547 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:30:21.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:21.531 7f9d5a05d700 1 --2- 172.21.2.112:0/2686353547 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9d5c106110 0x7f9d5c09f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:21.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:21.532 7f9d5b05f700 1 -- 172.21.2.112:0/2686353547 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9d5c105d10 msgr2=0x7f9d5c09eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:21.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:21.532 7f9d5b05f700 1 --2- 172.21.2.112:0/2686353547 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9d5c105d10 0x7f9d5c09eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:21.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:21.532 7f9d5b05f700 1 -- 172.21.2.112:0/2686353547 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9d5c106110 msgr2=0x7f9d5c09f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:21.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:21.532 7f9d5b05f700 1 --2- 172.21.2.112:0/2686353547 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9d5c106110 0x7f9d5c09f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:21.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:21.532 7f9d5b05f700 1 -- 172.21.2.112:0/2686353547 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d5c0b3640 con 0x7f9d5c0b2a00 2021-02-10T11:30:21.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:21.532 7f9d5b05f700 1 --2- 172.21.2.112:0/2686353547 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9d5c0b2a00 0x7f9d5c0b2e00 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 rx=0x7f9d4c0095e0 tx=0x7f9d4c00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:21.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:21.532 7f9d4b7fe700 1 -- 172.21.2.112:0/2686353547 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9d4c002030 con 0x7f9d5c0b2a00 2021-02-10T11:30:21.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:21.532 7f9d4b7fe700 1 -- 172.21.2.112:0/2686353547 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d4c002330 con 0x7f9d5c0b2a00 2021-02-10T11:30:21.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:21.532 7f9d4b7fe700 1 -- 172.21.2.112:0/2686353547 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9d4c002790 con 0x7f9d5c0b2a00 2021-02-10T11:30:21.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:21.532 7f9d60af9700 1 -- 172.21.2.112:0/2686353547 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d5c0b3640 con 0x7f9d5c0b2a00 2021-02-10T11:30:21.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:21.532 7f9d60af9700 1 -- 172.21.2.112:0/2686353547 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d5c0b3bd0 con 0x7f9d5c0b2a00 2021-02-10T11:30:21.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:21.533 7f9d4b7fe700 1 -- 172.21.2.112:0/2686353547 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9d4c000cd0 con 0x7f9d5c0b2a00 2021-02-10T11:30:21.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:21.533 7f9d4b7fe700 1 --2- 172.21.2.112:0/2686353547 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9d400186c0 0x7f9d4001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:21.524 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:21.533 7f9d4b7fe700 1 -- 172.21.2.112:0/2686353547 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f9d4c0325c0 con 0x7f9d5c0b2a00 2021-02-10T11:30:21.524 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:21.533 7f9d5a85e700 1 --2- 172.21.2.112:0/2686353547 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9d400186c0 0x7f9d4001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:21.524 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:21.534 7f9d5a85e700 1 --2- 172.21.2.112:0/2686353547 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9d400186c0 0x7f9d4001abc0 crc :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:21.550 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:21.563 7f9d60af9700 1 -- 172.21.2.112:0/2686353547 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d5c105ae0 con 0x7f9d5c0b2a00 2021-02-10T11:30:21.553 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:21.566 7f9d4b7fe700 1 -- 172.21.2.112:0/2686353547 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9d4c006ca0 con 0x7f9d5c0b2a00 2021-02-10T11:30:21.679 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:21.691 7f9d60af9700 1 -- 172.21.2.112:0/2686353547 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348589", "target": ["mgr", ""]}) v1 -- 0x7f9d5c1a16c0 con 0x7f9d400186c0 2021-02-10T11:30:21.689 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:21.702 7f9d4b7fe700 1 -- 172.21.2.112:0/2686353547 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f9d54001060 con 0x7f9d400186c0 2021-02-10T11:30:21.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:21.709 7f9d60af9700 1 -- 172.21.2.112:0/2686353547 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9d400186c0 msgr2=0x7f9d4001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:21.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:21.709 7f9d60af9700 1 --2- 172.21.2.112:0/2686353547 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9d400186c0 0x7f9d4001abc0 crc :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:21.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:21.709 7f9d60af9700 1 -- 172.21.2.112:0/2686353547 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9d5c0b2a00 msgr2=0x7f9d5c0b2e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:21.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:21.709 7f9d60af9700 1 --2- 172.21.2.112:0/2686353547 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9d5c0b2a00 0x7f9d5c0b2e00 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 rx=0x7f9d4c0095e0 tx=0x7f9d4c00b990).stop 2021-02-10T11:30:21.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:21.709 7f9d60af9700 1 -- 172.21.2.112:0/2686353547 shutdown_connections 2021-02-10T11:30:21.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:21.709 7f9d60af9700 1 --2- 172.21.2.112:0/2686353547 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9d400186c0 0x7f9d4001abc0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:21.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:21.709 7f9d60af9700 1 --2- 172.21.2.112:0/2686353547 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9d5c106110 0x7f9d5c09f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:21.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:21.709 7f9d60af9700 1 --2- 172.21.2.112:0/2686353547 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9d5c0b2a00 0x7f9d5c0b2e00 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:21.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:21.709 7f9d60af9700 1 --2- 172.21.2.112:0/2686353547 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9d5c105d10 0x7f9d5c09eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:21.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:21.709 7f9d60af9700 1 -- 172.21.2.112:0/2686353547 >> 172.21.2.112:0/2686353547 conn(0x7f9d5c0c36f0 msgr2=0x7f9d5c0bd640 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:21.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:21.709 7f9d60af9700 1 -- 172.21.2.112:0/2686353547 shutdown_connections 2021-02-10T11:30:21.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:21.709 7f9d60af9700 1 -- 172.21.2.112:0/2686353547 wait complete. 2021-02-10T11:30:21.705 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000348590 2021-02-10T11:30:22.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.133 7f653bbad700 1 Processor -- start 2021-02-10T11:30:22.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.133 7f653bbad700 1 -- start start 2021-02-10T11:30:22.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.134 7f653bbad700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f65341b0870 0x7f65340f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:22.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.134 7f653bbad700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65341b0c70 con 0x7f65341b0470 2021-02-10T11:30:22.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.134 7f653bbad700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65340f0b50 con 0x7f65340efe00 2021-02-10T11:30:22.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.134 7f653bbad700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65340e5ea0 con 0x7f65341b0870 2021-02-10T11:30:22.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.134 7f65391c8700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f65341b0870 0x7f65340f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:22.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.134 7f65399c9700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f65341b0470 0x7f65340ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:46332/0 (socket says 172.21.2.112:46332) 2021-02-10T11:30:22.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.134 7f65399c9700 1 -- 172.21.2.112:0/2799248214 learned_addr learned my addr 172.21.2.112:0/2799248214 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:30:22.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.134 7f65389c7700 1 -- 172.21.2.112:0/2799248214 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3240441835 0 0) 0x7f652c000f90 con 0x7f65340efe00 2021-02-10T11:30:22.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.134 7f65391c8700 1 -- 172.21.2.112:0/2799248214 >> v1:172.21.2.120:6789/0 conn(0x7f65340efe00 legacy=0x7f65340f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:22.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.134 7f65391c8700 1 -- 172.21.2.112:0/2799248214 >> v1:172.21.2.112:6789/0 conn(0x7f65341b0470 legacy=0x7f65340ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:22.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.134 7f65391c8700 1 -- 172.21.2.112:0/2799248214 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65340e52e0 con 0x7f65341b0870 2021-02-10T11:30:22.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.134 7f65391c8700 1 --2- 172.21.2.112:0/2799248214 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f65341b0870 0x7f65340f20e0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 rx=0x7f6530006650 tx=0x7f6530009a80).ready entity=mon.0 client_cookie=ad761e6495b4f5ab server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:22.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.134 7f65389c7700 1 -- 172.21.2.112:0/2799248214 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f65300009b0 con 0x7f65341b0870 2021-02-10T11:30:22.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.134 7f65389c7700 1 -- 172.21.2.112:0/2799248214 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f653000c0c0 con 0x7f65341b0870 2021-02-10T11:30:22.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.134 7f65389c7700 1 -- 172.21.2.112:0/2799248214 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f653000c520 con 0x7f65341b0870 2021-02-10T11:30:22.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.134 7f653bbad700 1 -- 172.21.2.112:0/2799248214 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f65341b0870 msgr2=0x7f65340f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:22.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.134 7f653bbad700 1 --2- 172.21.2.112:0/2799248214 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f65341b0870 0x7f65340f20e0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 rx=0x7f6530006650 tx=0x7f6530009a80).stop 2021-02-10T11:30:22.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.135 7f653bbad700 1 -- 172.21.2.112:0/2799248214 shutdown_connections 2021-02-10T11:30:22.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.135 7f653bbad700 1 --2- 172.21.2.112:0/2799248214 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f65341b0870 0x7f65340f20e0 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:22.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.135 7f653bbad700 1 -- 172.21.2.112:0/2799248214 >> 172.21.2.112:0/2799248214 conn(0x7f65340ce3e0 msgr2=0x7f65340e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:22.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.135 7f653bbad700 1 -- 172.21.2.112:0/2799248214 shutdown_connections 2021-02-10T11:30:22.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.135 7f653bbad700 1 -- 172.21.2.112:0/2799248214 wait complete. 2021-02-10T11:30:22.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.150 7f653bbad700 1 Processor -- start 2021-02-10T11:30:22.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.150 7f653bbad700 1 -- start start 2021-02-10T11:30:22.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.150 7f653bbad700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f65340a1dd0 0x7f6534190920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:22.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.150 7f653bbad700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f653408fd30 0x7f65340effc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:22.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.150 7f653bbad700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f65340f0e90 0x7f65340f1290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:22.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.150 7f653bbad700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65340f06f0 con 0x7f653408fd30 2021-02-10T11:30:22.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.150 7f653bbad700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6534090160 con 0x7f65340f0e90 2021-02-10T11:30:22.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.150 7f653bbad700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65340f18b0 con 0x7f65340a1dd0 2021-02-10T11:30:22.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.151 7f653a1ca700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f65340f0e90 0x7f65340f1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:22.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.151 7f65391c8700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f653408fd30 0x7f65340effc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:22.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.151 7f653a1ca700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f65340f0e90 0x7f65340f1290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3300/0 says I am v2:172.21.2.112:49156/0 (socket says 172.21.2.112:49156) 2021-02-10T11:30:22.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.151 7f653a1ca700 1 -- 172.21.2.112:0/2859766988 learned_addr learned my addr 172.21.2.112:0/2859766988 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:30:22.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.151 7f65399c9700 1 --2- 172.21.2.112:0/2859766988 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f65340a1dd0 0x7f6534190920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:22.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.151 7f65391c8700 1 -- 172.21.2.112:0/2859766988 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f65340a1dd0 msgr2=0x7f6534190920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:22.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.151 7f65391c8700 1 --2- 172.21.2.112:0/2859766988 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f65340a1dd0 0x7f6534190920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:22.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.151 7f65391c8700 1 -- 172.21.2.112:0/2859766988 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f65340f0e90 msgr2=0x7f65340f1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:22.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.151 7f65391c8700 1 --2- 172.21.2.112:0/2859766988 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f65340f0e90 0x7f65340f1290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:22.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.151 7f65391c8700 1 -- 172.21.2.112:0/2859766988 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65340f1ad0 con 0x7f653408fd30 2021-02-10T11:30:22.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.151 7f653a1ca700 1 --2- 172.21.2.112:0/2859766988 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f65340f0e90 0x7f65340f1290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:30:22.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.151 7f65391c8700 1 --2- 172.21.2.112:0/2859766988 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f653408fd30 0x7f65340effc0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 rx=0x7f65300066b0 tx=0x7f653000b8e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:22.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.151 7f65267fc700 1 -- 172.21.2.112:0/2859766988 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f65300067f0 con 0x7f653408fd30 2021-02-10T11:30:22.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.151 7f65267fc700 1 -- 172.21.2.112:0/2859766988 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6530000d60 con 0x7f653408fd30 2021-02-10T11:30:22.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.151 7f653bbad700 1 -- 172.21.2.112:0/2859766988 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65340f1ad0 con 0x7f653408fd30 2021-02-10T11:30:22.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.151 7f65267fc700 1 -- 172.21.2.112:0/2859766988 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6530002220 con 0x7f653408fd30 2021-02-10T11:30:22.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.151 7f653bbad700 1 -- 172.21.2.112:0/2859766988 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65340f1ad0 con 0x7f653408fd30 2021-02-10T11:30:22.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.152 7f65267fc700 1 -- 172.21.2.112:0/2859766988 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6530002020 con 0x7f653408fd30 2021-02-10T11:30:22.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.152 7f65267fc700 1 --2- 172.21.2.112:0/2859766988 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f651c018730 0x7f651c01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:22.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.152 7f65267fc700 1 -- 172.21.2.112:0/2859766988 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f6530032560 con 0x7f653408fd30 2021-02-10T11:30:22.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.152 7f65399c9700 1 --2- 172.21.2.112:0/2859766988 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f651c018730 0x7f651c01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:22.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.152 7f65399c9700 1 --2- 172.21.2.112:0/2859766988 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f651c018730 0x7f651c01ac10 crc :-1 s=READY pgs=409 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:22.170 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.182 7f653bbad700 1 -- 172.21.2.112:0/2859766988 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65340e52e0 con 0x7f653408fd30 2021-02-10T11:30:22.172 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.185 7f65267fc700 1 -- 172.21.2.112:0/2859766988 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f65300009a0 con 0x7f653408fd30 2021-02-10T11:30:22.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.311 7f653bbad700 1 -- 172.21.2.112:0/2859766988 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348590", "target": ["mgr", ""]}) v1 -- 0x7f65340e52e0 con 0x7f651c018730 2021-02-10T11:30:22.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.323 7f65267fc700 1 -- 172.21.2.112:0/2859766988 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f6528007450 con 0x7f651c018730 2021-02-10T11:30:22.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.329 7f653bbad700 1 -- 172.21.2.112:0/2859766988 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f651c018730 msgr2=0x7f651c01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:22.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.329 7f653bbad700 1 --2- 172.21.2.112:0/2859766988 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f651c018730 0x7f651c01ac10 crc :-1 s=READY pgs=409 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:22.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.329 7f653bbad700 1 -- 172.21.2.112:0/2859766988 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f653408fd30 msgr2=0x7f65340effc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:22.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.329 7f653bbad700 1 --2- 172.21.2.112:0/2859766988 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f653408fd30 0x7f65340effc0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 rx=0x7f65300066b0 tx=0x7f653000b8e0).stop 2021-02-10T11:30:22.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.329 7f653bbad700 1 -- 172.21.2.112:0/2859766988 shutdown_connections 2021-02-10T11:30:22.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.329 7f653bbad700 1 --2- 172.21.2.112:0/2859766988 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f651c018730 0x7f651c01ac10 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:22.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.329 7f653bbad700 1 --2- 172.21.2.112:0/2859766988 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f65340f0e90 0x7f65340f1290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:22.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.329 7f653bbad700 1 --2- 172.21.2.112:0/2859766988 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f653408fd30 0x7f65340effc0 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:22.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.329 7f653bbad700 1 --2- 172.21.2.112:0/2859766988 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f65340a1dd0 0x7f6534190920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:22.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.329 7f653bbad700 1 -- 172.21.2.112:0/2859766988 >> 172.21.2.112:0/2859766988 conn(0x7f65340958b0 msgr2=0x7f65340bf040 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:22.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.329 7f653bbad700 1 -- 172.21.2.112:0/2859766988 shutdown_connections 2021-02-10T11:30:22.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.329 7f653bbad700 1 -- 172.21.2.112:0/2859766988 wait complete. 2021-02-10T11:30:22.326 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000348591 2021-02-10T11:30:22.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.751 7f365ba00700 1 Processor -- start 2021-02-10T11:30:22.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.751 7f365ba00700 1 -- start start 2021-02-10T11:30:22.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.752 7f365ba00700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f36541b0470 0x7f36541b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:22.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.752 7f365ba00700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36540f0b50 con 0x7f36540ef610 2021-02-10T11:30:22.757 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.752 7f365ba00700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36540f24e0 con 0x7f36540f20e0 2021-02-10T11:30:22.757 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.752 7f365ba00700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36540e5ea0 con 0x7f36541b0470 2021-02-10T11:30:22.757 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.752 7f365901b700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f36540ef610 0x7f36540efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:46346/0 (socket says 172.21.2.112:46346) 2021-02-10T11:30:22.757 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.752 7f365901b700 1 -- 172.21.2.112:0/825563513 learned_addr learned my addr 172.21.2.112:0/825563513 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:30:22.757 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.752 7f365981c700 1 --2- 172.21.2.112:0/825563513 >> v2:172.21.2.120:3300/0 conn(0x7f36541b0470 0x7f36541b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:22.757 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.753 7f365881a700 1 -- 172.21.2.112:0/825563513 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3187811823 0 0) 0x7f3644000ce0 con 0x7f36540ef610 2021-02-10T11:30:22.757 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.753 7f365881a700 1 -- 172.21.2.112:0/825563513 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3640002fe0 con 0x7f36540ef610 2021-02-10T11:30:22.757 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.753 7f365881a700 1 -- 172.21.2.112:0/825563513 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 408875775 0 0) 0x7f364c000d30 con 0x7f36540f20e0 2021-02-10T11:30:22.757 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.753 7f365881a700 1 -- 172.21.2.112:0/825563513 --> v1:172.21.2.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3640004370 con 0x7f36540f20e0 2021-02-10T11:30:22.757 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.753 7f365981c700 1 -- 172.21.2.112:0/825563513 >> v1:172.21.2.120:6789/0 conn(0x7f36540f20e0 legacy=0x7f36540f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:22.757 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.753 7f365981c700 1 -- 172.21.2.112:0/825563513 >> v1:172.21.2.112:6789/0 conn(0x7f36540ef610 legacy=0x7f36540efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:22.757 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.753 7f365981c700 1 -- 172.21.2.112:0/825563513 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36540e52e0 con 0x7f36541b0470 2021-02-10T11:30:22.757 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.753 7f365981c700 1 --2- 172.21.2.112:0/825563513 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f36541b0470 0x7f36541b0870 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7f3650006650 tx=0x7f3650009a80).ready entity=mon.1 client_cookie=7b597c457277435c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:22.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.753 7f365881a700 1 -- 172.21.2.112:0/825563513 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f36500009b0 con 0x7f36541b0470 2021-02-10T11:30:22.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.753 7f365881a700 1 -- 172.21.2.112:0/825563513 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f365000c0c0 con 0x7f36541b0470 2021-02-10T11:30:22.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.753 7f365881a700 1 -- 172.21.2.112:0/825563513 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f365000c5e0 con 0x7f36541b0470 2021-02-10T11:30:22.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.753 7f365ba00700 1 -- 172.21.2.112:0/825563513 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f36541b0470 msgr2=0x7f36541b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:22.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.753 7f365ba00700 1 --2- 172.21.2.112:0/825563513 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f36541b0470 0x7f36541b0870 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7f3650006650 tx=0x7f3650009a80).stop 2021-02-10T11:30:22.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.754 7f365ba00700 1 -- 172.21.2.112:0/825563513 shutdown_connections 2021-02-10T11:30:22.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.754 7f365ba00700 1 --2- 172.21.2.112:0/825563513 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f36541b0470 0x7f36541b0870 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:22.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.754 7f365ba00700 1 -- 172.21.2.112:0/825563513 >> 172.21.2.112:0/825563513 conn(0x7f36540ce3e0 msgr2=0x7f36540e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:22.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.754 7f365ba00700 1 -- 172.21.2.112:0/825563513 shutdown_connections 2021-02-10T11:30:22.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.754 7f365ba00700 1 -- 172.21.2.112:0/825563513 wait complete. 2021-02-10T11:30:22.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.769 7f365ba00700 1 Processor -- start 2021-02-10T11:30:22.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.769 7f365ba00700 1 -- start start 2021-02-10T11:30:22.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.769 7f365ba00700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f36540a0420 0x7f36540a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:22.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.769 7f365ba00700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3654189d00 0x7f365418a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:22.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.769 7f365ba00700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f365418a720 0x7f36540b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:22.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.769 7f365ba00700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3654189800 con 0x7f365418a720 2021-02-10T11:30:22.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.769 7f365ba00700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f365418ab20 con 0x7f36540a0420 2021-02-10T11:30:22.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.769 7f365ba00700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36540a1140 con 0x7f3654189d00 2021-02-10T11:30:22.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.770 7f365a01d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f365418a720 0x7f36540b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:22.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.770 7f365981c700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f36540a0420 0x7f36540a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:22.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.770 7f365a01d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f365418a720 0x7f36540b7be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:46888/0 (socket says 172.21.2.112:46888) 2021-02-10T11:30:22.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.770 7f365a01d700 1 -- 172.21.2.112:0/1549662559 learned_addr learned my addr 172.21.2.112:0/1549662559 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:30:22.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.770 7f365901b700 1 --2- 172.21.2.112:0/1549662559 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3654189d00 0x7f365418a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:22.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.770 7f365981c700 1 -- 172.21.2.112:0/1549662559 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3654189d00 msgr2=0x7f365418a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:22.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.770 7f365981c700 1 --2- 172.21.2.112:0/1549662559 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3654189d00 0x7f365418a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:22.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.770 7f365981c700 1 -- 172.21.2.112:0/1549662559 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f365418a720 msgr2=0x7f36540b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:22.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.770 7f365981c700 1 --2- 172.21.2.112:0/1549662559 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f365418a720 0x7f36540b7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:22.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.770 7f365981c700 1 -- 172.21.2.112:0/1549662559 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36540b8200 con 0x7f36540a0420 2021-02-10T11:30:22.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.770 7f365981c700 1 --2- 172.21.2.112:0/1549662559 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f36540a0420 0x7f36540a0820 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7f365000b530 tx=0x7f365000b8d0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:22.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.770 7f364a7fc700 1 -- 172.21.2.112:0/1549662559 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3650000a10 con 0x7f36540a0420 2021-02-10T11:30:22.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.770 7f364a7fc700 1 -- 172.21.2.112:0/1549662559 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3650009280 con 0x7f36540a0420 2021-02-10T11:30:22.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.770 7f365ba00700 1 -- 172.21.2.112:0/1549662559 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36540b8200 con 0x7f36540a0420 2021-02-10T11:30:22.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.771 7f364a7fc700 1 -- 172.21.2.112:0/1549662559 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3650002030 con 0x7f36540a0420 2021-02-10T11:30:22.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.771 7f365ba00700 1 -- 172.21.2.112:0/1549662559 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36540b8200 con 0x7f36540a0420 2021-02-10T11:30:22.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.771 7f364a7fc700 1 -- 172.21.2.112:0/1549662559 <== mon.1 v2:172.21.2.120:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3650002500 con 0x7f36540a0420 2021-02-10T11:30:22.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.771 7f364a7fc700 1 --2- 172.21.2.112:0/1549662559 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f363c018730 0x7f363c01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:22.763 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.771 7f365901b700 1 --2- 172.21.2.112:0/1549662559 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f363c018730 0x7f363c01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:22.763 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.771 7f364a7fc700 1 -- 172.21.2.112:0/1549662559 <== mon.1 v2:172.21.2.120:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f36500022f0 con 0x7f36540a0420 2021-02-10T11:30:22.763 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.772 7f365901b700 1 --2- 172.21.2.112:0/1549662559 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f363c018730 0x7f363c01ac10 crc :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:22.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.801 7f365ba00700 1 -- 172.21.2.112:0/1549662559 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3654105ae0 con 0x7f36540a0420 2021-02-10T11:30:22.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.804 7f364a7fc700 1 -- 172.21.2.112:0/1549662559 <== mon.1 v2:172.21.2.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f36500022f0 con 0x7f36540a0420 2021-02-10T11:30:22.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.928 7f365ba00700 1 -- 172.21.2.112:0/1549662559 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348591", "target": ["mgr", ""]}) v1 -- 0x7f36540f0b50 con 0x7f363c018730 2021-02-10T11:30:22.926 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.939 7f364a7fc700 1 -- 172.21.2.112:0/1549662559 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f3644006b40 con 0x7f363c018730 2021-02-10T11:30:22.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.945 7f365ba00700 1 -- 172.21.2.112:0/1549662559 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f363c018730 msgr2=0x7f363c01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:22.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.945 7f365ba00700 1 --2- 172.21.2.112:0/1549662559 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f363c018730 0x7f363c01ac10 crc :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:22.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.945 7f365ba00700 1 -- 172.21.2.112:0/1549662559 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f36540a0420 msgr2=0x7f36540a0820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:22.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.945 7f365ba00700 1 --2- 172.21.2.112:0/1549662559 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f36540a0420 0x7f36540a0820 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7f365000b530 tx=0x7f365000b8d0).stop 2021-02-10T11:30:22.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.945 7f365ba00700 1 -- 172.21.2.112:0/1549662559 shutdown_connections 2021-02-10T11:30:22.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.945 7f365ba00700 1 --2- 172.21.2.112:0/1549662559 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f363c018730 0x7f363c01ac10 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:22.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.945 7f365ba00700 1 --2- 172.21.2.112:0/1549662559 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f365418a720 0x7f36540b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:22.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.945 7f365ba00700 1 --2- 172.21.2.112:0/1549662559 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3654189d00 0x7f365418a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:22.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.945 7f365ba00700 1 --2- 172.21.2.112:0/1549662559 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f36540a0420 0x7f36540a0820 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:22.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.945 7f365ba00700 1 -- 172.21.2.112:0/1549662559 >> 172.21.2.112:0/1549662559 conn(0x7f3654094e10 msgr2=0x7f36540a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:22.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.945 7f365ba00700 1 -- 172.21.2.112:0/1549662559 shutdown_connections 2021-02-10T11:30:22.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:22.945 7f365ba00700 1 -- 172.21.2.112:0/1549662559 wait complete. 2021-02-10T11:30:22.942 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000348592 2021-02-10T11:30:23.372 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:23.367 7f4dc129e700 1 Processor -- start 2021-02-10T11:30:23.372 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:23.367 7f4dc129e700 1 -- start start 2021-02-10T11:30:23.372 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:23.368 7f4dc129e700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f4dbc1b0470 0x7f4dbc1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:23.372 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:23.368 7f4dc129e700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4dbc0f20e0 0x7f4dbc0f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:23.372 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:23.368 7f4dc129e700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dbc0f0b50 con 0x7f4dbc0ef610 2021-02-10T11:30:23.373 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:23.368 7f4dc129e700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4dbc0f0820 con 0x7f4dbc1b0470 2021-02-10T11:30:23.373 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:23.368 7f4dc129e700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dbc0f24e0 con 0x7f4dbc0f20e0 2021-02-10T11:30:23.373 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:23.368 7f4dbb87e700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4dbc0f20e0 0x7f4dbc0f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:23.373 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:23.368 7f4dbb07d700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f4dbc1b0470 0x7f4dbc1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:23.373 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:23.368 7f4dba87c700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7f4dbc0ef610 0x7f4dbc0efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:39916/0 (socket says 172.21.2.112:39916) 2021-02-10T11:30:23.373 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:23.368 7f4dba87c700 1 -- 172.21.2.112:0/4194408957 learned_addr learned my addr 172.21.2.112:0/4194408957 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:30:23.373 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:23.369 7f4dbb07d700 1 -- 172.21.2.112:0/4194408957 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4dbc0f20e0 msgr2=0x7f4dbc0f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:23.373 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:23.369 7f4dbb07d700 1 --2- 172.21.2.112:0/4194408957 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4dbc0f20e0 0x7f4dbc0f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:23.373 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:23.369 7f4dbb07d700 1 -- 172.21.2.112:0/4194408957 >> v1:172.21.2.120:6789/0 conn(0x7f4dbc0ef610 legacy=0x7f4dbc0efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:23.373 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:23.369 7f4dbb07d700 1 -- 172.21.2.112:0/4194408957 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dbc0e52e0 con 0x7f4dbc1b0470 2021-02-10T11:30:23.373 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:23.369 7f4dbb87e700 1 --2- 172.21.2.112:0/4194408957 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4dbc0f20e0 0x7f4dbc0f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:30:23.373 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:23.369 7f4dbb07d700 1 --2- 172.21.2.112:0/4194408957 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4dbc1b0470 0x7f4dbc1b0870 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7f4dac006650 tx=0x7f4dac009a80).ready entity=mon.1 client_cookie=2c922644abd4ce51 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:23.373 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:23.369 7f4dba07b700 1 -- 172.21.2.112:0/4194408957 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4dac00c0c0 con 0x7f4dbc1b0470 2021-02-10T11:30:23.374 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:23.369 7f4dba07b700 1 -- 172.21.2.112:0/4194408957 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dac00c340 con 0x7f4dbc1b0470 2021-02-10T11:30:23.374 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:23.369 7f4dba07b700 1 -- 172.21.2.112:0/4194408957 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4dac00c860 con 0x7f4dbc1b0470 2021-02-10T11:30:23.374 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:23.369 7f4dc129e700 1 -- 172.21.2.112:0/4194408957 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4dbc1b0470 msgr2=0x7f4dbc1b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:23.374 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:23.369 7f4dc129e700 1 --2- 172.21.2.112:0/4194408957 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4dbc1b0470 0x7f4dbc1b0870 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7f4dac006650 tx=0x7f4dac009a80).stop 2021-02-10T11:30:23.374 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:23.370 7f4dc129e700 1 -- 172.21.2.112:0/4194408957 shutdown_connections 2021-02-10T11:30:23.374 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:23.370 7f4dc129e700 1 --2- 172.21.2.112:0/4194408957 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4dbc0f20e0 0x7f4dbc0f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:23.374 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:23.370 7f4dc129e700 1 --2- 172.21.2.112:0/4194408957 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4dbc1b0470 0x7f4dbc1b0870 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:23.375 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:23.370 7f4dc129e700 1 -- 172.21.2.112:0/4194408957 >> 172.21.2.112:0/4194408957 conn(0x7f4dbc0ce3e0 msgr2=0x7f4dbc0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:23.377 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:23.370 7f4dc129e700 1 -- 172.21.2.112:0/4194408957 shutdown_connections 2021-02-10T11:30:23.378 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:23.370 7f4dc129e700 1 -- 172.21.2.112:0/4194408957 wait complete. 2021-02-10T11:30:23.378 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:23.385 7f4dc129e700 1 Processor -- start 2021-02-10T11:30:23.378 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:23.385 7f4dc129e700 1 -- start start 2021-02-10T11:30:23.378 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:23.385 7f4dc129e700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4dbc08f480 0x7f4dbc08f880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:23.378 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:23.385 7f4dc129e700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4dbc08fea0 0x7f4dbc090cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:23.378 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:23.385 7f4dc129e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4dbc0902a0 0x7f4dbc091310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:23.378 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:23.385 7f4dc129e700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dbc090770 con 0x7f4dbc0902a0 2021-02-10T11:30:23.378 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:23.385 7f4dc129e700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dbc10a1f0 con 0x7f4dbc08fea0 2021-02-10T11:30:23.378 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:23.385 7f4dc129e700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dbc10a3d0 con 0x7f4dbc08f480 2021-02-10T11:30:23.378 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:23.386 7f4dbb87e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4dbc0902a0 0x7f4dbc091310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:23.378 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:23.386 7f4dbb87e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4dbc0902a0 0x7f4dbc091310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:46900/0 (socket says 172.21.2.112:46900) 2021-02-10T11:30:23.379 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:23.386 7f4dbb87e700 1 -- 172.21.2.112:0/750109563 learned_addr learned my addr 172.21.2.112:0/750109563 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:30:23.379 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:23.386 7f4dba87c700 1 --2- 172.21.2.112:0/750109563 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4dbc08fea0 0x7f4dbc090cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:23.379 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:23.386 7f4dbb07d700 1 --2- 172.21.2.112:0/750109563 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4dbc08f480 0x7f4dbc08f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:23.379 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:23.386 7f4dbb87e700 1 -- 172.21.2.112:0/750109563 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4dbc08f480 msgr2=0x7f4dbc08f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:23.379 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:23.386 7f4dbb87e700 1 --2- 172.21.2.112:0/750109563 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4dbc08f480 0x7f4dbc08f880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:23.379 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:23.386 7f4dbb87e700 1 -- 172.21.2.112:0/750109563 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4dbc08fea0 msgr2=0x7f4dbc090cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:23.379 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:23.386 7f4dbb87e700 1 --2- 172.21.2.112:0/750109563 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4dbc08fea0 0x7f4dbc090cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:23.379 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:23.386 7f4dbb87e700 1 -- 172.21.2.112:0/750109563 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dbc10a6b0 con 0x7f4dbc0902a0 2021-02-10T11:30:23.379 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:23.386 7f4dbb87e700 1 --2- 172.21.2.112:0/750109563 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4dbc0902a0 0x7f4dbc091310 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 rx=0x7f4db400b970 tx=0x7f4db400bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:23.379 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:23.387 7f4da3fff700 1 -- 172.21.2.112:0/750109563 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4db4000f90 con 0x7f4dbc0902a0 2021-02-10T11:30:23.379 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:23.387 7f4da3fff700 1 -- 172.21.2.112:0/750109563 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4db4006aa0 con 0x7f4dbc0902a0 2021-02-10T11:30:23.379 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:23.387 7f4da3fff700 1 -- 172.21.2.112:0/750109563 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4db4001680 con 0x7f4dbc0902a0 2021-02-10T11:30:23.379 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:23.387 7f4dc129e700 1 -- 172.21.2.112:0/750109563 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4dbc10a6b0 con 0x7f4dbc0902a0 2021-02-10T11:30:23.380 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:23.387 7f4dc129e700 1 -- 172.21.2.112:0/750109563 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4dbc091c20 con 0x7f4dbc0902a0 2021-02-10T11:30:23.380 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:23.387 7f4da3fff700 1 -- 172.21.2.112:0/750109563 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4db40011f0 con 0x7f4dbc0902a0 2021-02-10T11:30:23.380 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:23.387 7f4da3fff700 1 --2- 172.21.2.112:0/750109563 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4d9c0186c0 0x7f4d9c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:23.380 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:23.387 7f4da3fff700 1 -- 172.21.2.112:0/750109563 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f4db40328f0 con 0x7f4dbc0902a0 2021-02-10T11:30:23.380 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:23.388 7f4dbb07d700 1 --2- 172.21.2.112:0/750109563 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4d9c0186c0 0x7f4d9c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:23.380 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:23.388 7f4dbb07d700 1 --2- 172.21.2.112:0/750109563 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4d9c0186c0 0x7f4d9c01abc0 crc :-1 s=READY pgs=411 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:23.406 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:23.418 7f4dc129e700 1 -- 172.21.2.112:0/750109563 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4dbc105ae0 con 0x7f4dbc0902a0 2021-02-10T11:30:23.408 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:23.421 7f4da3fff700 1 -- 172.21.2.112:0/750109563 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4db4000e30 con 0x7f4dbc0902a0 2021-02-10T11:30:23.538 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:23.550 7f4dc129e700 1 -- 172.21.2.112:0/750109563 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348592", "target": ["mgr", ""]}) v1 -- 0x7f4dbc090770 con 0x7f4d9c0186c0 2021-02-10T11:30:23.549 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:23.562 7f4da3fff700 1 -- 172.21.2.112:0/750109563 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f4dac006420 con 0x7f4d9c0186c0 2021-02-10T11:30:23.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:23.568 7f4dc129e700 1 -- 172.21.2.112:0/750109563 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4d9c0186c0 msgr2=0x7f4d9c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:23.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:23.568 7f4dc129e700 1 --2- 172.21.2.112:0/750109563 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4d9c0186c0 0x7f4d9c01abc0 crc :-1 s=READY pgs=411 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:23.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:23.568 7f4dc129e700 1 -- 172.21.2.112:0/750109563 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4dbc0902a0 msgr2=0x7f4dbc091310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:23.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:23.568 7f4dc129e700 1 --2- 172.21.2.112:0/750109563 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4dbc0902a0 0x7f4dbc091310 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 rx=0x7f4db400b970 tx=0x7f4db400bd10).stop 2021-02-10T11:30:23.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:23.568 7f4dc129e700 1 -- 172.21.2.112:0/750109563 shutdown_connections 2021-02-10T11:30:23.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:23.568 7f4dc129e700 1 --2- 172.21.2.112:0/750109563 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4d9c0186c0 0x7f4d9c01abc0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:23.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:23.568 7f4dc129e700 1 --2- 172.21.2.112:0/750109563 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4dbc08fea0 0x7f4dbc090cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:23.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:23.568 7f4dc129e700 1 --2- 172.21.2.112:0/750109563 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4dbc0902a0 0x7f4dbc091310 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:23.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:23.568 7f4dc129e700 1 --2- 172.21.2.112:0/750109563 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4dbc08f480 0x7f4dbc08f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:23.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:23.568 7f4dc129e700 1 -- 172.21.2.112:0/750109563 >> 172.21.2.112:0/750109563 conn(0x7f4dbc0bbf60 msgr2=0x7f4dbc105d10 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:23.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:23.568 7f4dc129e700 1 -- 172.21.2.112:0/750109563 shutdown_connections 2021-02-10T11:30:23.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:23.568 7f4dc129e700 1 -- 172.21.2.112:0/750109563 wait complete. 2021-02-10T11:30:23.564 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000348593 2021-02-10T11:30:24.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:23.998 7ff1f1a6d700 1 Processor -- start 2021-02-10T11:30:24.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:23.998 7ff1f1a6d700 1 -- start start 2021-02-10T11:30:24.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:23.998 7ff1f1a6d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7ff1ec1b0470 0x7ff1ec1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:24.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:23.998 7ff1f1a6d700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7ff1ec0ef610 0x7ff1ec0efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:24.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:23.998 7ff1f1a6d700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1ec0f0b50 con 0x7ff1ec0f0030 2021-02-10T11:30:24.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:23.998 7ff1f1a6d700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1ec0e52e0 con 0x7ff1ec1b0470 2021-02-10T11:30:24.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:23.998 7ff1f1a6d700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1ec0e5490 con 0x7ff1ec0ef610 2021-02-10T11:30:24.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:23.998 7ff1eb7fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7ff1ec1b0470 0x7ff1ec1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:24.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:23.998 7ff1eb7fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7ff1ec1b0470 0x7ff1ec1b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:46906/0 (socket says 172.21.2.112:46906) 2021-02-10T11:30:24.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:23.998 7ff1eb7fe700 1 -- 172.21.2.112:0/1420721059 learned_addr learned my addr 172.21.2.112:0/1420721059 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:30:24.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:23.998 7ff1eaffd700 1 --2- 172.21.2.112:0/1420721059 >> v2:172.21.2.120:3300/0 conn(0x7ff1ec0ef610 0x7ff1ec0efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:24.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:23.999 7ff1eb7fe700 1 -- 172.21.2.112:0/1420721059 >> v2:172.21.2.120:3300/0 conn(0x7ff1ec0ef610 msgr2=0x7ff1ec0efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:24.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:23.999 7ff1eb7fe700 1 --2- 172.21.2.112:0/1420721059 >> v2:172.21.2.120:3300/0 conn(0x7ff1ec0ef610 0x7ff1ec0efa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:24.005 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:23.999 7ff1eb7fe700 1 -- 172.21.2.112:0/1420721059 >> v1:172.21.2.120:6789/0 conn(0x7ff1ec0f0030 legacy=0x7ff1ec0f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:24.005 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:23.999 7ff1eb7fe700 1 -- 172.21.2.112:0/1420721059 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1ec0f0430 con 0x7ff1ec1b0470 2021-02-10T11:30:24.005 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:23.999 7ff1eaffd700 1 --2- 172.21.2.112:0/1420721059 >> v2:172.21.2.120:3300/0 conn(0x7ff1ec0ef610 0x7ff1ec0efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:30:24.005 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:23.999 7ff1eb7fe700 1 --2- 172.21.2.112:0/1420721059 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff1ec1b0470 0x7ff1ec1b0870 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 rx=0x7ff1d4006700 tx=0x7ff1d4009b80).ready entity=mon.0 client_cookie=22a1765a223cead6 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:24.005 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:23.999 7ff1ea7fc700 1 -- 172.21.2.112:0/1420721059 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff1d400c0c0 con 0x7ff1ec1b0470 2021-02-10T11:30:24.005 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:23.999 7ff1ea7fc700 1 -- 172.21.2.112:0/1420721059 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1d400c440 con 0x7ff1ec1b0470 2021-02-10T11:30:24.005 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:23.999 7ff1ea7fc700 1 -- 172.21.2.112:0/1420721059 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff1d400c920 con 0x7ff1ec1b0470 2021-02-10T11:30:24.005 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:23.999 7ff1f1a6d700 1 -- 172.21.2.112:0/1420721059 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff1ec1b0470 msgr2=0x7ff1ec1b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:24.006 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:23.999 7ff1f1a6d700 1 --2- 172.21.2.112:0/1420721059 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff1ec1b0470 0x7ff1ec1b0870 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 rx=0x7ff1d4006700 tx=0x7ff1d4009b80).stop 2021-02-10T11:30:24.006 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.000 7ff1f1a6d700 1 -- 172.21.2.112:0/1420721059 shutdown_connections 2021-02-10T11:30:24.006 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.000 7ff1f1a6d700 1 --2- 172.21.2.112:0/1420721059 >> v2:172.21.2.120:3300/0 conn(0x7ff1ec0ef610 0x7ff1ec0efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:24.006 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.000 7ff1f1a6d700 1 --2- 172.21.2.112:0/1420721059 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff1ec1b0470 0x7ff1ec1b0870 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:24.006 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.000 7ff1f1a6d700 1 -- 172.21.2.112:0/1420721059 >> 172.21.2.112:0/1420721059 conn(0x7ff1ec0ce3e0 msgr2=0x7ff1ec0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:24.006 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.000 7ff1f1a6d700 1 -- 172.21.2.112:0/1420721059 shutdown_connections 2021-02-10T11:30:24.006 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.000 7ff1f1a6d700 1 -- 172.21.2.112:0/1420721059 wait complete. 2021-02-10T11:30:24.006 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.015 7ff1f1a6d700 1 Processor -- start 2021-02-10T11:30:24.006 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.015 7ff1f1a6d700 1 -- start start 2021-02-10T11:30:24.009 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.016 7ff1f1a6d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff1ec106fd0 0x7ff1ec0bc380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:24.009 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.016 7ff1f1a6d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff1ec0bc9a0 0x7ff1ec1a0e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:24.010 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.016 7ff1f1a6d700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff1ec1a1430 0x7ff1ec08e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:24.010 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.016 7ff1f1a6d700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1ec1073d0 con 0x7ff1ec106fd0 2021-02-10T11:30:24.010 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.016 7ff1f1a6d700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1ec1a1830 con 0x7ff1ec1a1430 2021-02-10T11:30:24.010 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.016 7ff1f1a6d700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1ec1a1a50 con 0x7ff1ec0bc9a0 2021-02-10T11:30:24.010 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.016 7ff1eb7fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff1ec106fd0 0x7ff1ec0bc380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:24.011 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.016 7ff1eb7fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff1ec106fd0 0x7ff1ec0bc380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:46912/0 (socket says 172.21.2.112:46912) 2021-02-10T11:30:24.011 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.016 7ff1eb7fe700 1 -- 172.21.2.112:0/2054045145 learned_addr learned my addr 172.21.2.112:0/2054045145 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:30:24.011 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.016 7ff1eaffd700 1 --2- 172.21.2.112:0/2054045145 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff1ec0bc9a0 0x7ff1ec1a0e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:24.011 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.016 7ff1ebfff700 1 --2- 172.21.2.112:0/2054045145 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff1ec1a1430 0x7ff1ec08e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:24.011 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.016 7ff1eb7fe700 1 -- 172.21.2.112:0/2054045145 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff1ec0bc9a0 msgr2=0x7ff1ec1a0e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:24.011 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.016 7ff1eb7fe700 1 --2- 172.21.2.112:0/2054045145 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff1ec0bc9a0 0x7ff1ec1a0e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:24.011 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.016 7ff1eb7fe700 1 -- 172.21.2.112:0/2054045145 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff1ec1a1430 msgr2=0x7ff1ec08e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:24.011 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.016 7ff1eb7fe700 1 --2- 172.21.2.112:0/2054045145 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff1ec1a1430 0x7ff1ec08e890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:24.011 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.016 7ff1eb7fe700 1 -- 172.21.2.112:0/2054045145 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1ec08eeb0 con 0x7ff1ec106fd0 2021-02-10T11:30:24.011 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.016 7ff1eb7fe700 1 --2- 172.21.2.112:0/2054045145 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff1ec106fd0 0x7ff1ec0bc380 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 rx=0x7ff1d400b580 tx=0x7ff1d400b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:24.012 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.016 7ff1e3fff700 1 -- 172.21.2.112:0/2054045145 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff1d400bd10 con 0x7ff1ec106fd0 2021-02-10T11:30:24.012 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.016 7ff1e3fff700 1 -- 172.21.2.112:0/2054045145 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1d4000c90 con 0x7ff1ec106fd0 2021-02-10T11:30:24.012 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.016 7ff1e3fff700 1 -- 172.21.2.112:0/2054045145 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff1d40022b0 con 0x7ff1ec106fd0 2021-02-10T11:30:24.012 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.017 7ff1f1a6d700 1 -- 172.21.2.112:0/2054045145 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1ec08eeb0 con 0x7ff1ec106fd0 2021-02-10T11:30:24.013 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.017 7ff1f1a6d700 1 -- 172.21.2.112:0/2054045145 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1ec08eeb0 con 0x7ff1ec106fd0 2021-02-10T11:30:24.013 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.018 7ff1e3fff700 1 -- 172.21.2.112:0/2054045145 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff1d4000c90 con 0x7ff1ec106fd0 2021-02-10T11:30:24.013 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.019 7ff1e3fff700 1 --2- 172.21.2.112:0/2054045145 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff1d00186c0 0x7ff1d001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:24.013 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.019 7ff1e3fff700 1 -- 172.21.2.112:0/2054045145 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7ff1d40323e0 con 0x7ff1ec106fd0 2021-02-10T11:30:24.013 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.019 7ff1eaffd700 1 --2- 172.21.2.112:0/2054045145 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff1d00186c0 0x7ff1d001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:24.013 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.019 7ff1eaffd700 1 --2- 172.21.2.112:0/2054045145 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff1d00186c0 0x7ff1d001abc0 crc :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:24.037 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.048 7ff1f1a6d700 1 -- 172.21.2.112:0/2054045145 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1ec1078f0 con 0x7ff1ec106fd0 2021-02-10T11:30:24.038 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.051 7ff1e3fff700 1 -- 172.21.2.112:0/2054045145 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff1d40091e0 con 0x7ff1ec106fd0 2021-02-10T11:30:24.169 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.181 7ff1f1a6d700 1 -- 172.21.2.112:0/2054045145 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348593", "target": ["mgr", ""]}) v1 -- 0x7ff1ec1a1830 con 0x7ff1d00186c0 2021-02-10T11:30:24.181 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.193 7ff1e3fff700 1 -- 172.21.2.112:0/2054045145 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7ff1dc006a30 con 0x7ff1d00186c0 2021-02-10T11:30:24.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.199 7ff1f1a6d700 1 -- 172.21.2.112:0/2054045145 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff1d00186c0 msgr2=0x7ff1d001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:24.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.199 7ff1f1a6d700 1 --2- 172.21.2.112:0/2054045145 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff1d00186c0 0x7ff1d001abc0 crc :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:24.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.199 7ff1f1a6d700 1 -- 172.21.2.112:0/2054045145 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff1ec106fd0 msgr2=0x7ff1ec0bc380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:24.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.199 7ff1f1a6d700 1 --2- 172.21.2.112:0/2054045145 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff1ec106fd0 0x7ff1ec0bc380 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 rx=0x7ff1d400b580 tx=0x7ff1d400b920).stop 2021-02-10T11:30:24.189 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.199 7ff1f1a6d700 1 -- 172.21.2.112:0/2054045145 shutdown_connections 2021-02-10T11:30:24.189 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.199 7ff1f1a6d700 1 --2- 172.21.2.112:0/2054045145 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff1d00186c0 0x7ff1d001abc0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:24.190 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.199 7ff1f1a6d700 1 --2- 172.21.2.112:0/2054045145 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff1ec1a1430 0x7ff1ec08e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:24.190 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.199 7ff1f1a6d700 1 --2- 172.21.2.112:0/2054045145 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff1ec0bc9a0 0x7ff1ec1a0e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:24.191 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.199 7ff1f1a6d700 1 --2- 172.21.2.112:0/2054045145 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff1ec106fd0 0x7ff1ec0bc380 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:24.191 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.199 7ff1f1a6d700 1 -- 172.21.2.112:0/2054045145 >> 172.21.2.112:0/2054045145 conn(0x7ff1ec0bbf60 msgr2=0x7ff1ec105d10 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:24.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.199 7ff1f1a6d700 1 -- 172.21.2.112:0/2054045145 shutdown_connections 2021-02-10T11:30:24.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.199 7ff1f1a6d700 1 -- 172.21.2.112:0/2054045145 wait complete. 2021-02-10T11:30:24.195 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000348594 2021-02-10T11:30:24.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.622 7f23b3aba700 1 Processor -- start 2021-02-10T11:30:24.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.622 7f23b3aba700 1 -- start start 2021-02-10T11:30:24.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.623 7f23b3aba700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f23ac1b0470 0x7f23ac1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:24.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.623 7f23b3aba700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f23ac0ef610 0x7f23ac0efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:24.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.623 7f23b3aba700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f23ac0f0030 0x7f23ac0f0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:24.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.623 7f23b3aba700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23ac0e5ea0 con 0x7f23ac0f0030 2021-02-10T11:30:24.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.623 7f23b3aba700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23ac0f0b50 con 0x7f23ac0ef610 2021-02-10T11:30:24.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.623 7f23b3aba700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23ac0e52e0 con 0x7f23ac1b0470 2021-02-10T11:30:24.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.623 7f23b20d7700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f23ac0f0030 0x7f23ac0f0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:24.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.623 7f23b20d7700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f23ac0f0030 0x7f23ac0f0430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:46920/0 (socket says 172.21.2.112:46920) 2021-02-10T11:30:24.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.623 7f23b10d5700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f23ac0ef610 0x7f23ac0efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:24.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.623 7f23b20d7700 1 -- 172.21.2.112:0/1396269411 learned_addr learned my addr 172.21.2.112:0/1396269411 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:30:24.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.623 7f23b18d6700 1 --2- 172.21.2.112:0/1396269411 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f23ac1b0470 0x7f23ac1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:24.629 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.623 7f23b20d7700 1 -- 172.21.2.112:0/1396269411 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f23ac1b0470 msgr2=0x7f23ac1b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:24.629 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.623 7f23b20d7700 1 --2- 172.21.2.112:0/1396269411 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f23ac1b0470 0x7f23ac1b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:24.629 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.623 7f23b20d7700 1 -- 172.21.2.112:0/1396269411 >> v2:172.21.2.120:3300/0 conn(0x7f23ac0ef610 msgr2=0x7f23ac0efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:24.629 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.623 7f23b20d7700 1 --2- 172.21.2.112:0/1396269411 >> v2:172.21.2.120:3300/0 conn(0x7f23ac0ef610 0x7f23ac0efa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:24.629 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.624 7f23b20d7700 1 -- 172.21.2.112:0/1396269411 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23ac0f10d0 con 0x7f23ac0f0030 2021-02-10T11:30:24.629 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.624 7f23b20d7700 1 --2- 172.21.2.112:0/1396269411 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f23ac0f0030 0x7f23ac0f0430 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 rx=0x7f23a4006f00 tx=0x7f23a4009f10).ready entity=mon.0 client_cookie=f2ca46d730bc4a53 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:24.629 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.624 7f23b08d4700 1 -- 172.21.2.112:0/1396269411 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f23a400b3b0 con 0x7f23ac0f0030 2021-02-10T11:30:24.629 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.624 7f23b08d4700 1 -- 172.21.2.112:0/1396269411 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23a400b730 con 0x7f23ac0f0030 2021-02-10T11:30:24.629 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.624 7f23b08d4700 1 -- 172.21.2.112:0/1396269411 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f23a400bc50 con 0x7f23ac0f0030 2021-02-10T11:30:24.629 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.624 7f23b3aba700 1 -- 172.21.2.112:0/1396269411 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f23ac0f0030 msgr2=0x7f23ac0f0430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:24.629 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.624 7f23b3aba700 1 --2- 172.21.2.112:0/1396269411 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f23ac0f0030 0x7f23ac0f0430 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 rx=0x7f23a4006f00 tx=0x7f23a4009f10).stop 2021-02-10T11:30:24.630 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.624 7f23b3aba700 1 -- 172.21.2.112:0/1396269411 shutdown_connections 2021-02-10T11:30:24.630 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.625 7f23b3aba700 1 --2- 172.21.2.112:0/1396269411 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f23ac0f0030 0x7f23ac0f0430 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:24.630 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.625 7f23b3aba700 1 --2- 172.21.2.112:0/1396269411 >> v2:172.21.2.120:3300/0 conn(0x7f23ac0ef610 0x7f23ac0efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:24.630 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.625 7f23b3aba700 1 --2- 172.21.2.112:0/1396269411 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f23ac1b0470 0x7f23ac1b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:24.630 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.625 7f23b3aba700 1 -- 172.21.2.112:0/1396269411 >> 172.21.2.112:0/1396269411 conn(0x7f23ac0ce3e0 msgr2=0x7f23ac0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:24.630 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.625 7f23b3aba700 1 -- 172.21.2.112:0/1396269411 shutdown_connections 2021-02-10T11:30:24.630 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.625 7f23b3aba700 1 -- 172.21.2.112:0/1396269411 wait complete. 2021-02-10T11:30:24.630 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.639 7f23b3aba700 1 Processor -- start 2021-02-10T11:30:24.630 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.639 7f23b3aba700 1 -- start start 2021-02-10T11:30:24.632 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.639 7f23b3aba700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f23ac09b930 0x7f23ac0b7c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:24.632 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.639 7f23b3aba700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f23ac0b8410 0x7f23ac0b8810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:24.632 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.639 7f23b3aba700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f23ac0c1f80 0x7f23ac0c2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:24.632 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.639 7f23b3aba700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23ac0c1a00 con 0x7f23ac0c1f80 2021-02-10T11:30:24.632 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.639 7f23b3aba700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23ac0b8e30 con 0x7f23ac09b930 2021-02-10T11:30:24.632 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.639 7f23b3aba700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23ac0b9050 con 0x7f23ac0b8410 2021-02-10T11:30:24.632 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.640 7f23b20d7700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f23ac0c1f80 0x7f23ac0c2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:24.633 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.640 7f23b20d7700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f23ac0c1f80 0x7f23ac0c2380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:46926/0 (socket says 172.21.2.112:46926) 2021-02-10T11:30:24.633 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.640 7f23b20d7700 1 -- 172.21.2.112:0/1317848553 learned_addr learned my addr 172.21.2.112:0/1317848553 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:30:24.633 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.640 7f23b18d6700 1 --2- 172.21.2.112:0/1317848553 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f23ac09b930 0x7f23ac0b7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:24.633 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.640 7f23b10d5700 1 --2- 172.21.2.112:0/1317848553 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f23ac0b8410 0x7f23ac0b8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:24.633 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.640 7f23b20d7700 1 -- 172.21.2.112:0/1317848553 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f23ac0b8410 msgr2=0x7f23ac0b8810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:24.633 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.640 7f23b20d7700 1 --2- 172.21.2.112:0/1317848553 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f23ac0b8410 0x7f23ac0b8810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:24.633 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.640 7f23b20d7700 1 -- 172.21.2.112:0/1317848553 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f23ac09b930 msgr2=0x7f23ac0b7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:24.633 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.640 7f23b20d7700 1 --2- 172.21.2.112:0/1317848553 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f23ac09b930 0x7f23ac0b7c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:24.633 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.640 7f23b20d7700 1 -- 172.21.2.112:0/1317848553 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23ac0c29a0 con 0x7f23ac0c1f80 2021-02-10T11:30:24.633 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.640 7f23b20d7700 1 --2- 172.21.2.112:0/1317848553 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f23ac0c1f80 0x7f23ac0c2380 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 rx=0x7f23a400b9b0 tx=0x7f23a400bd50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:24.633 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.641 7f239e7fc700 1 -- 172.21.2.112:0/1317848553 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f23a4009450 con 0x7f23ac0c1f80 2021-02-10T11:30:24.634 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.641 7f239e7fc700 1 -- 172.21.2.112:0/1317848553 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23a400c2f0 con 0x7f23ac0c1f80 2021-02-10T11:30:24.634 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.641 7f239e7fc700 1 -- 172.21.2.112:0/1317848553 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f23a400c780 con 0x7f23ac0c1f80 2021-02-10T11:30:24.634 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.641 7f23b3aba700 1 -- 172.21.2.112:0/1317848553 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23ac0c29a0 con 0x7f23ac0c1f80 2021-02-10T11:30:24.634 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.641 7f23b3aba700 1 -- 172.21.2.112:0/1317848553 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23ac0c29a0 con 0x7f23ac0c1f80 2021-02-10T11:30:24.634 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.641 7f239e7fc700 1 -- 172.21.2.112:0/1317848553 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f23a4009450 con 0x7f23ac0c1f80 2021-02-10T11:30:24.634 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.641 7f239e7fc700 1 --2- 172.21.2.112:0/1317848553 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f23940186c0 0x7f239401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:24.634 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.641 7f239e7fc700 1 -- 172.21.2.112:0/1317848553 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f23a40328f0 con 0x7f23ac0c1f80 2021-02-10T11:30:24.634 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.642 7f23b18d6700 1 --2- 172.21.2.112:0/1317848553 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f23940186c0 0x7f239401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:24.634 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.642 7f23b18d6700 1 --2- 172.21.2.112:0/1317848553 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f23940186c0 0x7f239401abc0 crc :-1 s=READY pgs=413 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:24.660 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.673 7f23b3aba700 1 -- 172.21.2.112:0/1317848553 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23ac105b30 con 0x7f23ac0c1f80 2021-02-10T11:30:24.663 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.675 7f239e7fc700 1 -- 172.21.2.112:0/1317848553 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f23a4000f90 con 0x7f23ac0c1f80 2021-02-10T11:30:24.788 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.800 7f23b3aba700 1 -- 172.21.2.112:0/1317848553 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348594", "target": ["mgr", ""]}) v1 -- 0x7f23ac105b30 con 0x7f23940186c0 2021-02-10T11:30:24.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.810 7f239e7fc700 1 -- 172.21.2.112:0/1317848553 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f23a00009c0 con 0x7f23940186c0 2021-02-10T11:30:24.806 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.818 7f23b3aba700 1 -- 172.21.2.112:0/1317848553 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f23940186c0 msgr2=0x7f239401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:24.806 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.818 7f23b3aba700 1 --2- 172.21.2.112:0/1317848553 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f23940186c0 0x7f239401abc0 crc :-1 s=READY pgs=413 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:24.806 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.818 7f23b3aba700 1 -- 172.21.2.112:0/1317848553 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f23ac0c1f80 msgr2=0x7f23ac0c2380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:24.806 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.818 7f23b3aba700 1 --2- 172.21.2.112:0/1317848553 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f23ac0c1f80 0x7f23ac0c2380 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 rx=0x7f23a400b9b0 tx=0x7f23a400bd50).stop 2021-02-10T11:30:24.806 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.818 7f23b3aba700 1 -- 172.21.2.112:0/1317848553 shutdown_connections 2021-02-10T11:30:24.807 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.818 7f23b3aba700 1 --2- 172.21.2.112:0/1317848553 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f23940186c0 0x7f239401abc0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:24.807 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.818 7f23b3aba700 1 --2- 172.21.2.112:0/1317848553 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f23ac0c1f80 0x7f23ac0c2380 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:24.807 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.818 7f23b3aba700 1 --2- 172.21.2.112:0/1317848553 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f23ac0b8410 0x7f23ac0b8810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:24.807 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.818 7f23b3aba700 1 --2- 172.21.2.112:0/1317848553 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f23ac09b930 0x7f23ac0b7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:24.807 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.818 7f23b3aba700 1 -- 172.21.2.112:0/1317848553 >> 172.21.2.112:0/1317848553 conn(0x7f23ac094e10 msgr2=0x7f23ac0a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:24.807 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.818 7f23b3aba700 1 -- 172.21.2.112:0/1317848553 shutdown_connections 2021-02-10T11:30:24.807 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:24.818 7f23b3aba700 1 -- 172.21.2.112:0/1317848553 wait complete. 2021-02-10T11:30:24.814 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000348595 2021-02-10T11:30:25.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.238 7f09d5cd0700 1 Processor -- start 2021-02-10T11:30:25.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.238 7f09d5cd0700 1 -- start start 2021-02-10T11:30:25.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.239 7f09d5cd0700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f09d01b0870 0x7f09d00f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:25.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.239 7f09d5cd0700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f09d00efe00 0x7f09d00f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:25.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.239 7f09d5cd0700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09d01b0c70 con 0x7f09d01b0470 2021-02-10T11:30:25.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.239 7f09d5cd0700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09d00e5ea0 con 0x7f09d01b0870 2021-02-10T11:30:25.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.239 7f09d5cd0700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09d00f0b50 con 0x7f09d00efe00 2021-02-10T11:30:25.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.239 7f09cf7fe700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f09d01b0470 0x7f09d00ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:46394/0 (socket says 172.21.2.112:46394) 2021-02-10T11:30:25.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.239 7f09cf7fe700 1 -- 172.21.2.112:0/2240526577 learned_addr learned my addr 172.21.2.112:0/2240526577 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:30:25.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.239 7f09ceffd700 1 --2- 172.21.2.112:0/2240526577 >> v2:172.21.2.112:3300/0 conn(0x7f09d01b0870 0x7f09d00f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:25.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.239 7f09cffff700 1 --2- 172.21.2.112:0/2240526577 >> v2:172.21.2.120:3300/0 conn(0x7f09d00efe00 0x7f09d00f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:25.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.240 7f09ce7fc700 1 -- 172.21.2.112:0/2240526577 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 5325376 0 0) 0x7f09c0000ce0 con 0x7f09d01b0470 2021-02-10T11:30:25.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.240 7f09ce7fc700 1 -- 172.21.2.112:0/2240526577 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09b8002fe0 con 0x7f09d01b0470 2021-02-10T11:30:25.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.240 7f09ceffd700 1 -- 172.21.2.112:0/2240526577 >> v2:172.21.2.120:3300/0 conn(0x7f09d00efe00 msgr2=0x7f09d00f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:25.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.240 7f09ceffd700 1 --2- 172.21.2.112:0/2240526577 >> v2:172.21.2.120:3300/0 conn(0x7f09d00efe00 0x7f09d00f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:25.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.240 7f09ceffd700 1 -- 172.21.2.112:0/2240526577 >> v1:172.21.2.112:6789/0 conn(0x7f09d01b0470 legacy=0x7f09d00ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:25.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.240 7f09ceffd700 1 -- 172.21.2.112:0/2240526577 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09d00f0820 con 0x7f09d01b0870 2021-02-10T11:30:25.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.240 7f09ceffd700 1 --2- 172.21.2.112:0/2240526577 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f09d01b0870 0x7f09d00f20e0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 rx=0x7f09c4006650 tx=0x7f09c4009a80).ready entity=mon.0 client_cookie=94c31393d36ac8eb server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:25.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.240 7f09ce7fc700 1 -- 172.21.2.112:0/2240526577 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f09c40009b0 con 0x7f09d01b0870 2021-02-10T11:30:25.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.240 7f09ce7fc700 1 -- 172.21.2.112:0/2240526577 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09c400c0c0 con 0x7f09d01b0870 2021-02-10T11:30:25.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.240 7f09ce7fc700 1 -- 172.21.2.112:0/2240526577 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f09c400c5e0 con 0x7f09d01b0870 2021-02-10T11:30:25.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.240 7f09d5cd0700 1 -- 172.21.2.112:0/2240526577 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f09d01b0870 msgr2=0x7f09d00f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:25.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.240 7f09d5cd0700 1 --2- 172.21.2.112:0/2240526577 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f09d01b0870 0x7f09d00f20e0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 rx=0x7f09c4006650 tx=0x7f09c4009a80).stop 2021-02-10T11:30:25.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.241 7f09d5cd0700 1 -- 172.21.2.112:0/2240526577 shutdown_connections 2021-02-10T11:30:25.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.241 7f09d5cd0700 1 --2- 172.21.2.112:0/2240526577 >> v2:172.21.2.120:3300/0 conn(0x7f09d00efe00 0x7f09d00f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:25.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.241 7f09d5cd0700 1 --2- 172.21.2.112:0/2240526577 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f09d01b0870 0x7f09d00f20e0 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:25.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.241 7f09d5cd0700 1 -- 172.21.2.112:0/2240526577 >> 172.21.2.112:0/2240526577 conn(0x7f09d00ce3e0 msgr2=0x7f09d00e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:25.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.241 7f09d5cd0700 1 -- 172.21.2.112:0/2240526577 shutdown_connections 2021-02-10T11:30:25.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.241 7f09d5cd0700 1 -- 172.21.2.112:0/2240526577 wait complete. 2021-02-10T11:30:25.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.256 7f09d5cd0700 1 Processor -- start 2021-02-10T11:30:25.248 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.256 7f09d5cd0700 1 -- start start 2021-02-10T11:30:25.250 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.256 7f09d5cd0700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f09d00a0420 0x7f09d00a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:25.250 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.256 7f09d5cd0700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f09d0189d00 0x7f09d018a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:25.250 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.256 7f09d5cd0700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f09d018a720 0x7f09d00b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:25.250 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.257 7f09d5cd0700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09d0189800 con 0x7f09d00a0420 2021-02-10T11:30:25.250 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.257 7f09d5cd0700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09d018ab20 con 0x7f09d0189d00 2021-02-10T11:30:25.250 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.257 7f09d5cd0700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09d00a1140 con 0x7f09d018a720 2021-02-10T11:30:25.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.257 7f09cf7fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f09d00a0420 0x7f09d00a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:25.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.257 7f09ceffd700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f09d0189d00 0x7f09d018a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:25.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.257 7f09cf7fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f09d00a0420 0x7f09d00a0820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:46938/0 (socket says 172.21.2.112:46938) 2021-02-10T11:30:25.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.257 7f09cf7fe700 1 -- 172.21.2.112:0/3753933304 learned_addr learned my addr 172.21.2.112:0/3753933304 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:30:25.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.257 7f09cffff700 1 --2- 172.21.2.112:0/3753933304 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f09d018a720 0x7f09d00b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:25.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.257 7f09ceffd700 1 -- 172.21.2.112:0/3753933304 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f09d018a720 msgr2=0x7f09d00b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:25.252 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.257 7f09ceffd700 1 --2- 172.21.2.112:0/3753933304 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f09d018a720 0x7f09d00b7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:25.252 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.257 7f09ceffd700 1 -- 172.21.2.112:0/3753933304 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f09d00a0420 msgr2=0x7f09d00a0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:25.252 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.257 7f09ceffd700 1 --2- 172.21.2.112:0/3753933304 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f09d00a0420 0x7f09d00a0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:25.252 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.257 7f09ceffd700 1 -- 172.21.2.112:0/3753933304 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09d00b8200 con 0x7f09d0189d00 2021-02-10T11:30:25.252 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.257 7f09cf7fe700 1 --2- 172.21.2.112:0/3753933304 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f09d00a0420 0x7f09d00a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:30:25.252 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.257 7f09ceffd700 1 --2- 172.21.2.112:0/3753933304 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f09d0189d00 0x7f09d018a100 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7f09c400b530 tx=0x7f09c400b8d0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:25.252 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.258 7f09b7fff700 1 -- 172.21.2.112:0/3753933304 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f09c4009280 con 0x7f09d0189d00 2021-02-10T11:30:25.252 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.258 7f09b7fff700 1 -- 172.21.2.112:0/3753933304 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09c400bcc0 con 0x7f09d0189d00 2021-02-10T11:30:25.252 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.258 7f09b7fff700 1 -- 172.21.2.112:0/3753933304 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f09c40022d0 con 0x7f09d0189d00 2021-02-10T11:30:25.252 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.258 7f09d5cd0700 1 -- 172.21.2.112:0/3753933304 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09d00b8200 con 0x7f09d0189d00 2021-02-10T11:30:25.253 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.258 7f09d5cd0700 1 -- 172.21.2.112:0/3753933304 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09d00b8200 con 0x7f09d0189d00 2021-02-10T11:30:25.253 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.258 7f09b7fff700 1 -- 172.21.2.112:0/3753933304 <== mon.1 v2:172.21.2.120:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f09c4009280 con 0x7f09d0189d00 2021-02-10T11:30:25.253 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.258 7f09b7fff700 1 --2- 172.21.2.112:0/3753933304 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f09b0018730 0x7f09b001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:25.253 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.258 7f09b7fff700 1 -- 172.21.2.112:0/3753933304 <== mon.1 v2:172.21.2.120:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f09c40326e0 con 0x7f09d0189d00 2021-02-10T11:30:25.253 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.259 7f09cf7fe700 1 --2- 172.21.2.112:0/3753933304 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f09b0018730 0x7f09b001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:25.253 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.259 7f09cf7fe700 1 --2- 172.21.2.112:0/3753933304 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f09b0018730 0x7f09b001abc0 crc :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:25.277 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.289 7f09d5cd0700 1 -- 172.21.2.112:0/3753933304 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09d018ab20 con 0x7f09d0189d00 2021-02-10T11:30:25.279 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.291 7f09b7fff700 1 -- 172.21.2.112:0/3753933304 <== mon.1 v2:172.21.2.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f09c4000c90 con 0x7f09d0189d00 2021-02-10T11:30:25.407 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.419 7f09d5cd0700 1 -- 172.21.2.112:0/3753933304 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348595", "target": ["mgr", ""]}) v1 -- 0x7f09d018ab20 con 0x7f09b0018730 2021-02-10T11:30:25.415 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.429 7f09b7fff700 1 -- 172.21.2.112:0/3753933304 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f09c00009b0 con 0x7f09b0018730 2021-02-10T11:30:25.424 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.437 7f09d5cd0700 1 -- 172.21.2.112:0/3753933304 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f09b0018730 msgr2=0x7f09b001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:25.424 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.437 7f09d5cd0700 1 --2- 172.21.2.112:0/3753933304 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f09b0018730 0x7f09b001abc0 crc :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:25.424 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.437 7f09d5cd0700 1 -- 172.21.2.112:0/3753933304 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f09d0189d00 msgr2=0x7f09d018a100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:25.424 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.437 7f09d5cd0700 1 --2- 172.21.2.112:0/3753933304 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f09d0189d00 0x7f09d018a100 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7f09c400b530 tx=0x7f09c400b8d0).stop 2021-02-10T11:30:25.425 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.437 7f09d5cd0700 1 -- 172.21.2.112:0/3753933304 shutdown_connections 2021-02-10T11:30:25.425 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.437 7f09d5cd0700 1 --2- 172.21.2.112:0/3753933304 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f09b0018730 0x7f09b001abc0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:25.425 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.437 7f09d5cd0700 1 --2- 172.21.2.112:0/3753933304 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f09d0189d00 0x7f09d018a100 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:25.425 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.437 7f09d5cd0700 1 --2- 172.21.2.112:0/3753933304 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f09d018a720 0x7f09d00b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:25.425 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.437 7f09d5cd0700 1 --2- 172.21.2.112:0/3753933304 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f09d00a0420 0x7f09d00a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:25.426 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.437 7f09d5cd0700 1 -- 172.21.2.112:0/3753933304 >> 172.21.2.112:0/3753933304 conn(0x7f09d0094e10 msgr2=0x7f09d00a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:25.426 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.437 7f09d5cd0700 1 -- 172.21.2.112:0/3753933304 shutdown_connections 2021-02-10T11:30:25.426 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.437 7f09d5cd0700 1 -- 172.21.2.112:0/3753933304 wait complete. 2021-02-10T11:30:25.433 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000348596 2021-02-10T11:30:25.865 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.861 7fc50d076700 1 Processor -- start 2021-02-10T11:30:25.865 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.861 7fc50d076700 1 -- start start 2021-02-10T11:30:25.866 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.861 7fc50d076700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fc5081b0470 0x7fc5081b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:25.866 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.861 7fc50d076700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc5080ef610 0x7fc5080efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:25.866 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.861 7fc50d076700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5080f0b50 con 0x7fc5080f0030 2021-02-10T11:30:25.866 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.861 7fc50d076700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5080e52e0 con 0x7fc5081b0470 2021-02-10T11:30:25.866 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.862 7fc50d076700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5080e54d0 con 0x7fc5080ef610 2021-02-10T11:30:25.866 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.862 7fc506d9d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fc5081b0470 0x7fc5081b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:25.866 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.862 7fc506d9d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fc5081b0470 0x7fc5081b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:46946/0 (socket says 172.21.2.112:46946) 2021-02-10T11:30:25.866 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.862 7fc506d9d700 1 -- 172.21.2.112:0/453418773 learned_addr learned my addr 172.21.2.112:0/453418773 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:30:25.866 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.862 7fc50659c700 1 --2- 172.21.2.112:0/453418773 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc5080ef610 0x7fc5080efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:25.866 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.862 7fc506d9d700 1 -- 172.21.2.112:0/453418773 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc5080ef610 msgr2=0x7fc5080efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:25.866 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.862 7fc506d9d700 1 --2- 172.21.2.112:0/453418773 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc5080ef610 0x7fc5080efa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:25.867 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.862 7fc506d9d700 1 -- 172.21.2.112:0/453418773 >> v1:172.21.2.120:6789/0 conn(0x7fc5080f0030 legacy=0x7fc5080f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:25.867 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.862 7fc506d9d700 1 -- 172.21.2.112:0/453418773 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5080f0430 con 0x7fc5081b0470 2021-02-10T11:30:25.867 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.862 7fc506d9d700 1 --2- 172.21.2.112:0/453418773 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc5081b0470 0x7fc5081b0870 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 rx=0x7fc4f0006700 tx=0x7fc4f0009b00).ready entity=mon.0 client_cookie=64c991cb47e0b54e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:25.867 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.863 7fc505d9b700 1 -- 172.21.2.112:0/453418773 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc4f000c0c0 con 0x7fc5081b0470 2021-02-10T11:30:25.867 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.863 7fc505d9b700 1 -- 172.21.2.112:0/453418773 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4f000c400 con 0x7fc5081b0470 2021-02-10T11:30:25.867 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.863 7fc505d9b700 1 -- 172.21.2.112:0/453418773 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc4f000c920 con 0x7fc5081b0470 2021-02-10T11:30:25.867 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.863 7fc50d076700 1 -- 172.21.2.112:0/453418773 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc5081b0470 msgr2=0x7fc5081b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:25.867 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.863 7fc50d076700 1 --2- 172.21.2.112:0/453418773 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc5081b0470 0x7fc5081b0870 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 rx=0x7fc4f0006700 tx=0x7fc4f0009b00).stop 2021-02-10T11:30:25.867 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.864 7fc50d076700 1 -- 172.21.2.112:0/453418773 shutdown_connections 2021-02-10T11:30:25.867 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.864 7fc50d076700 1 --2- 172.21.2.112:0/453418773 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc5080ef610 0x7fc5080efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:25.867 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.864 7fc50d076700 1 --2- 172.21.2.112:0/453418773 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc5081b0470 0x7fc5081b0870 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:25.867 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.864 7fc50d076700 1 -- 172.21.2.112:0/453418773 >> 172.21.2.112:0/453418773 conn(0x7fc5080ce3e0 msgr2=0x7fc5080e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:25.867 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.864 7fc50d076700 1 -- 172.21.2.112:0/453418773 shutdown_connections 2021-02-10T11:30:25.868 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.864 7fc50d076700 1 -- 172.21.2.112:0/453418773 wait complete. 2021-02-10T11:30:25.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.878 7fc50d076700 1 Processor -- start 2021-02-10T11:30:25.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.878 7fc50d076700 1 -- start start 2021-02-10T11:30:25.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.879 7fc50d076700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc5080a0420 0x7fc5080a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:25.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.879 7fc50d076700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc508189d00 0x7fc50818a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:25.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.879 7fc50d076700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc50818a720 0x7fc5080b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:25.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.879 7fc50d076700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc508189780 con 0x7fc50818a720 2021-02-10T11:30:25.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.879 7fc50d076700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc50818ab20 con 0x7fc5080a0420 2021-02-10T11:30:25.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.879 7fc50d076700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5080a1140 con 0x7fc508189d00 2021-02-10T11:30:25.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.879 7fc50759e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc50818a720 0x7fc5080b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:25.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.879 7fc50759e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc50818a720 0x7fc5080b7be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:46954/0 (socket says 172.21.2.112:46954) 2021-02-10T11:30:25.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.879 7fc50759e700 1 -- 172.21.2.112:0/4133917033 learned_addr learned my addr 172.21.2.112:0/4133917033 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:30:25.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.879 7fc506d9d700 1 --2- 172.21.2.112:0/4133917033 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc5080a0420 0x7fc5080a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:25.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.879 7fc50659c700 1 --2- 172.21.2.112:0/4133917033 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc508189d00 0x7fc50818a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:25.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.879 7fc50759e700 1 -- 172.21.2.112:0/4133917033 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc508189d00 msgr2=0x7fc50818a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:25.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.879 7fc50759e700 1 --2- 172.21.2.112:0/4133917033 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc508189d00 0x7fc50818a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:25.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.879 7fc50759e700 1 -- 172.21.2.112:0/4133917033 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc5080a0420 msgr2=0x7fc5080a0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:25.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.879 7fc50759e700 1 --2- 172.21.2.112:0/4133917033 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc5080a0420 0x7fc5080a0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:25.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.879 7fc50759e700 1 -- 172.21.2.112:0/4133917033 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5080b8200 con 0x7fc50818a720 2021-02-10T11:30:25.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.880 7fc50759e700 1 --2- 172.21.2.112:0/4133917033 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc50818a720 0x7fc5080b7be0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 rx=0x7fc500009980 tx=0x7fc50000bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:25.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.880 7fc4ff7fe700 1 -- 172.21.2.112:0/4133917033 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc5000058a0 con 0x7fc50818a720 2021-02-10T11:30:25.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.880 7fc4ff7fe700 1 -- 172.21.2.112:0/4133917033 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc500003210 con 0x7fc50818a720 2021-02-10T11:30:25.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.880 7fc4ff7fe700 1 -- 172.21.2.112:0/4133917033 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc500005de0 con 0x7fc50818a720 2021-02-10T11:30:25.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.880 7fc50d076700 1 -- 172.21.2.112:0/4133917033 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5080b8200 con 0x7fc50818a720 2021-02-10T11:30:25.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.880 7fc50d076700 1 -- 172.21.2.112:0/4133917033 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5080b8790 con 0x7fc50818a720 2021-02-10T11:30:25.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.880 7fc4ff7fe700 1 -- 172.21.2.112:0/4133917033 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc500006030 con 0x7fc50818a720 2021-02-10T11:30:25.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.880 7fc4ff7fe700 1 --2- 172.21.2.112:0/4133917033 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc4ec0186c0 0x7fc4ec01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:25.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.880 7fc4ff7fe700 1 -- 172.21.2.112:0/4133917033 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fc5000328a0 con 0x7fc50818a720 2021-02-10T11:30:25.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.881 7fc506d9d700 1 --2- 172.21.2.112:0/4133917033 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc4ec0186c0 0x7fc4ec01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:25.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.881 7fc506d9d700 1 --2- 172.21.2.112:0/4133917033 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc4ec0186c0 0x7fc4ec01abc0 crc :-1 s=READY pgs=415 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:25.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.911 7fc50d076700 1 -- 172.21.2.112:0/4133917033 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc508105ae0 con 0x7fc50818a720 2021-02-10T11:30:25.901 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:25.913 7fc4ff7fe700 1 -- 172.21.2.112:0/4133917033 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc500005a60 con 0x7fc50818a720 2021-02-10T11:30:26.029 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:26.042 7fc50d076700 1 -- 172.21.2.112:0/4133917033 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348596", "target": ["mgr", ""]}) v1 -- 0x7fc5080b9020 con 0x7fc4ec0186c0 2021-02-10T11:30:26.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:26.052 7fc4ff7fe700 1 -- 172.21.2.112:0/4133917033 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fc4f0006410 con 0x7fc4ec0186c0 2021-02-10T11:30:26.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:26.059 7fc50d076700 1 -- 172.21.2.112:0/4133917033 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc4ec0186c0 msgr2=0x7fc4ec01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:26.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:26.059 7fc50d076700 1 --2- 172.21.2.112:0/4133917033 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc4ec0186c0 0x7fc4ec01abc0 crc :-1 s=READY pgs=415 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:26.047 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:26.059 7fc50d076700 1 -- 172.21.2.112:0/4133917033 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc50818a720 msgr2=0x7fc5080b7be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:26.047 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:26.059 7fc50d076700 1 --2- 172.21.2.112:0/4133917033 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc50818a720 0x7fc5080b7be0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 rx=0x7fc500009980 tx=0x7fc50000bcd0).stop 2021-02-10T11:30:26.047 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:26.059 7fc50d076700 1 -- 172.21.2.112:0/4133917033 shutdown_connections 2021-02-10T11:30:26.047 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:26.059 7fc50d076700 1 --2- 172.21.2.112:0/4133917033 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc4ec0186c0 0x7fc4ec01abc0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:26.047 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:26.059 7fc50d076700 1 --2- 172.21.2.112:0/4133917033 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc50818a720 0x7fc5080b7be0 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:26.047 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:26.059 7fc50d076700 1 --2- 172.21.2.112:0/4133917033 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc508189d00 0x7fc50818a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:26.047 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:26.059 7fc50d076700 1 --2- 172.21.2.112:0/4133917033 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc5080a0420 0x7fc5080a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:26.047 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:26.059 7fc50d076700 1 -- 172.21.2.112:0/4133917033 >> 172.21.2.112:0/4133917033 conn(0x7fc508094e10 msgr2=0x7fc5080a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:26.047 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:26.059 7fc50d076700 1 -- 172.21.2.112:0/4133917033 shutdown_connections 2021-02-10T11:30:26.047 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:26.059 7fc50d076700 1 -- 172.21.2.112:0/4133917033 wait complete. 2021-02-10T11:30:26.055 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000348597 2021-02-10T11:30:26.484 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:26.480 7fc7015f4700 1 Processor -- start 2021-02-10T11:30:26.484 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:26.480 7fc7015f4700 1 -- start start 2021-02-10T11:30:26.484 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:26.480 7fc7015f4700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fc6fc1b0470 0x7fc6fc1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:26.485 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:26.480 7fc7015f4700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc6fc0f20e0 0x7fc6fc0f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:26.485 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:26.480 7fc7015f4700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6fc0f0b50 con 0x7fc6fc0ef610 2021-02-10T11:30:26.485 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:26.480 7fc7015f4700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6fc0f0820 con 0x7fc6fc1b0470 2021-02-10T11:30:26.485 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:26.480 7fc7015f4700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6fc0f24e0 con 0x7fc6fc0f20e0 2021-02-10T11:30:26.485 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:26.480 7fc6fb2bf700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fc6fc1b0470 0x7fc6fc1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:26.485 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:26.480 7fc6f2abe700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fc6fc0ef610 0x7fc6fc0efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:46426/0 (socket says 172.21.2.112:46426) 2021-02-10T11:30:26.486 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:26.480 7fc6f2abe700 1 -- 172.21.2.112:0/3570533432 learned_addr learned my addr 172.21.2.112:0/3570533432 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:30:26.486 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:26.480 7fc6fbac0700 1 --2- 172.21.2.112:0/3570533432 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc6fc0f20e0 0x7fc6fc0f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:26.486 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:26.481 7fc6fb2bf700 1 -- 172.21.2.112:0/3570533432 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc6fc0f20e0 msgr2=0x7fc6fc0f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:26.486 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:26.481 7fc6fb2bf700 1 --2- 172.21.2.112:0/3570533432 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc6fc0f20e0 0x7fc6fc0f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:26.486 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:26.481 7fc6fb2bf700 1 -- 172.21.2.112:0/3570533432 >> v1:172.21.2.112:6789/0 conn(0x7fc6fc0ef610 legacy=0x7fc6fc0efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:26.487 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:26.481 7fc6fb2bf700 1 -- 172.21.2.112:0/3570533432 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6fc0e52e0 con 0x7fc6fc1b0470 2021-02-10T11:30:26.487 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:26.481 7fc6fb2bf700 1 --2- 172.21.2.112:0/3570533432 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc6fc1b0470 0x7fc6fc1b0870 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 rx=0x7fc6ec006650 tx=0x7fc6ec009a80).ready entity=mon.0 client_cookie=dfdfb530b065e94b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:26.487 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:26.481 7fc6faabe700 1 -- 172.21.2.112:0/3570533432 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc6ec0009b0 con 0x7fc6fc1b0470 2021-02-10T11:30:26.487 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:26.481 7fc6faabe700 1 -- 172.21.2.112:0/3570533432 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6ec00c0c0 con 0x7fc6fc1b0470 2021-02-10T11:30:26.487 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:26.481 7fc6faabe700 1 -- 172.21.2.112:0/3570533432 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc6ec00c5e0 con 0x7fc6fc1b0470 2021-02-10T11:30:26.488 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:26.481 7fc7015f4700 1 -- 172.21.2.112:0/3570533432 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc6fc1b0470 msgr2=0x7fc6fc1b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:26.488 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:26.481 7fc7015f4700 1 --2- 172.21.2.112:0/3570533432 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc6fc1b0470 0x7fc6fc1b0870 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 rx=0x7fc6ec006650 tx=0x7fc6ec009a80).stop 2021-02-10T11:30:26.488 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:26.482 7fc7015f4700 1 -- 172.21.2.112:0/3570533432 shutdown_connections 2021-02-10T11:30:26.488 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:26.482 7fc7015f4700 1 --2- 172.21.2.112:0/3570533432 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc6fc0f20e0 0x7fc6fc0f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:26.488 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:26.482 7fc7015f4700 1 --2- 172.21.2.112:0/3570533432 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc6fc1b0470 0x7fc6fc1b0870 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:26.488 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:26.482 7fc7015f4700 1 -- 172.21.2.112:0/3570533432 >> 172.21.2.112:0/3570533432 conn(0x7fc6fc0ce3e0 msgr2=0x7fc6fc0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:26.489 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:26.482 7fc7015f4700 1 -- 172.21.2.112:0/3570533432 shutdown_connections 2021-02-10T11:30:26.489 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:26.482 7fc7015f4700 1 -- 172.21.2.112:0/3570533432 wait complete. 2021-02-10T11:30:26.489 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:26.496 7fc7015f4700 1 Processor -- start 2021-02-10T11:30:26.489 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:26.496 7fc7015f4700 1 -- start start 2021-02-10T11:30:26.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:26.497 7fc7015f4700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc6fc190920 0x7fc6fc0b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:26.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:26.497 7fc7015f4700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc6fc0b8200 0x7fc6fc0b8600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:26.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:26.497 7fc7015f4700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc6fc0c1d80 0x7fc6fc0c2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:26.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:26.497 7fc7015f4700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6fc190da0 con 0x7fc6fc190920 2021-02-10T11:30:26.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:26.497 7fc7015f4700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6fc0b8c20 con 0x7fc6fc0c1d80 2021-02-10T11:30:26.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:26.497 7fc7015f4700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6fc0b8e10 con 0x7fc6fc0b8200 2021-02-10T11:30:26.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:26.497 7fc6fb2bf700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc6fc190920 0x7fc6fc0b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:26.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:26.497 7fc6fb2bf700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc6fc190920 0x7fc6fc0b7be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:46968/0 (socket says 172.21.2.112:46968) 2021-02-10T11:30:26.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:26.497 7fc6fb2bf700 1 -- 172.21.2.112:0/2567724822 learned_addr learned my addr 172.21.2.112:0/2567724822 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:30:26.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:26.497 7fc6fbac0700 1 --2- 172.21.2.112:0/2567724822 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc6fc0c1d80 0x7fc6fc0c2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:26.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:26.497 7fc6f2abe700 1 --2- 172.21.2.112:0/2567724822 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc6fc0b8200 0x7fc6fc0b8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:26.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:26.497 7fc6fbac0700 1 -- 172.21.2.112:0/2567724822 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc6fc0b8200 msgr2=0x7fc6fc0b8600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:26.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:26.497 7fc6fbac0700 1 --2- 172.21.2.112:0/2567724822 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc6fc0b8200 0x7fc6fc0b8600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:26.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:26.497 7fc6fbac0700 1 -- 172.21.2.112:0/2567724822 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc6fc190920 msgr2=0x7fc6fc0b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:26.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:26.497 7fc6fbac0700 1 --2- 172.21.2.112:0/2567724822 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc6fc190920 0x7fc6fc0b7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:26.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:26.497 7fc6fbac0700 1 -- 172.21.2.112:0/2567724822 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6fc0c27d0 con 0x7fc6fc0c1d80 2021-02-10T11:30:26.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:26.497 7fc6fb2bf700 1 --2- 172.21.2.112:0/2567724822 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc6fc190920 0x7fc6fc0b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:30:26.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:26.498 7fc6fbac0700 1 --2- 172.21.2.112:0/2567724822 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc6fc0c1d80 0x7fc6fc0c2180 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7fc6f400b8a0 tx=0x7fc6f400bc40).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:26.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:26.498 7fc6f8aba700 1 -- 172.21.2.112:0/2567724822 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc6f4000f90 con 0x7fc6fc0c1d80 2021-02-10T11:30:26.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:26.498 7fc6f8aba700 1 -- 172.21.2.112:0/2567724822 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6f400bd80 con 0x7fc6fc0c1d80 2021-02-10T11:30:26.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:26.498 7fc7015f4700 1 -- 172.21.2.112:0/2567724822 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6fc0c27d0 con 0x7fc6fc0c1d80 2021-02-10T11:30:26.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:26.498 7fc7015f4700 1 -- 172.21.2.112:0/2567724822 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6fc0c27d0 con 0x7fc6fc0c1d80 2021-02-10T11:30:26.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:26.498 7fc6f8aba700 1 -- 172.21.2.112:0/2567724822 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc6f40013e0 con 0x7fc6fc0c1d80 2021-02-10T11:30:26.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:26.498 7fc6f8aba700 1 -- 172.21.2.112:0/2567724822 <== mon.1 v2:172.21.2.120:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc6f40011f0 con 0x7fc6fc0c1d80 2021-02-10T11:30:26.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:26.498 7fc6f8aba700 1 --2- 172.21.2.112:0/2567724822 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc6e00186c0 0x7fc6e001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:26.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:26.498 7fc6f8aba700 1 -- 172.21.2.112:0/2567724822 <== mon.1 v2:172.21.2.120:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fc6f4032850 con 0x7fc6fc0c1d80 2021-02-10T11:30:26.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:26.499 7fc6fb2bf700 1 --2- 172.21.2.112:0/2567724822 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc6e00186c0 0x7fc6e001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:26.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:26.499 7fc6fb2bf700 1 --2- 172.21.2.112:0/2567724822 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc6e00186c0 0x7fc6e001abc0 crc :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:26.517 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:26.529 7fc7015f4700 1 -- 172.21.2.112:0/2567724822 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6fc105ae0 con 0x7fc6fc0c1d80 2021-02-10T11:30:26.520 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:26.532 7fc6f8aba700 1 -- 172.21.2.112:0/2567724822 <== mon.1 v2:172.21.2.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc6f402ced0 con 0x7fc6fc0c1d80 2021-02-10T11:30:26.646 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:26.658 7fc7015f4700 1 -- 172.21.2.112:0/2567724822 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348597", "target": ["mgr", ""]}) v1 -- 0x7fc6fc0f0b50 con 0x7fc6e00186c0 2021-02-10T11:30:26.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:26.669 7fc6f8aba700 1 -- 172.21.2.112:0/2567724822 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fc6ec000960 con 0x7fc6e00186c0 2021-02-10T11:30:26.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:26.676 7fc7015f4700 1 -- 172.21.2.112:0/2567724822 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc6e00186c0 msgr2=0x7fc6e001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:26.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:26.676 7fc7015f4700 1 --2- 172.21.2.112:0/2567724822 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc6e00186c0 0x7fc6e001abc0 crc :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:26.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:26.676 7fc7015f4700 1 -- 172.21.2.112:0/2567724822 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc6fc0c1d80 msgr2=0x7fc6fc0c2180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:26.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:26.676 7fc7015f4700 1 --2- 172.21.2.112:0/2567724822 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc6fc0c1d80 0x7fc6fc0c2180 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7fc6f400b8a0 tx=0x7fc6f400bc40).stop 2021-02-10T11:30:26.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:26.676 7fc7015f4700 1 -- 172.21.2.112:0/2567724822 shutdown_connections 2021-02-10T11:30:26.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:26.676 7fc7015f4700 1 --2- 172.21.2.112:0/2567724822 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc6e00186c0 0x7fc6e001abc0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:26.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:26.676 7fc7015f4700 1 --2- 172.21.2.112:0/2567724822 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc6fc0c1d80 0x7fc6fc0c2180 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:26.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:26.676 7fc7015f4700 1 --2- 172.21.2.112:0/2567724822 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc6fc0b8200 0x7fc6fc0b8600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:26.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:26.676 7fc7015f4700 1 --2- 172.21.2.112:0/2567724822 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc6fc190920 0x7fc6fc0b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:26.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:26.676 7fc7015f4700 1 -- 172.21.2.112:0/2567724822 >> 172.21.2.112:0/2567724822 conn(0x7fc6fc094e10 msgr2=0x7fc6fc0a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:26.665 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:26.677 7fc7015f4700 1 -- 172.21.2.112:0/2567724822 shutdown_connections 2021-02-10T11:30:26.665 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:26.677 7fc7015f4700 1 -- 172.21.2.112:0/2567724822 wait complete. 2021-02-10T11:30:26.673 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000348598 2021-02-10T11:30:27.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.101 7f7d68da7700 1 Processor -- start 2021-02-10T11:30:27.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.101 7f7d68da7700 1 -- start start 2021-02-10T11:30:27.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.101 7f7d68da7700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f7d641b0870 0x7f7d640f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:27.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.101 7f7d68da7700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d641b0c70 con 0x7f7d640efe00 2021-02-10T11:30:27.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.101 7f7d68da7700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d640f0b50 con 0x7f7d641b0470 2021-02-10T11:30:27.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.101 7f7d68da7700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d640e5ea0 con 0x7f7d641b0870 2021-02-10T11:30:27.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.101 7f7d63314700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f7d640efe00 0x7f7d640f0200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:46438/0 (socket says 172.21.2.112:46438) 2021-02-10T11:30:27.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.101 7f7d63314700 1 -- 172.21.2.112:0/763026771 learned_addr learned my addr 172.21.2.112:0/763026771 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:30:27.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.101 7f7d5a312700 1 --2- 172.21.2.112:0/763026771 >> v2:172.21.2.112:3300/0 conn(0x7f7d641b0870 0x7f7d640f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:27.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.102 7f7d62312700 1 -- 172.21.2.112:0/763026771 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1336347729 0 0) 0x7f7d5c001500 con 0x7f7d640efe00 2021-02-10T11:30:27.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.102 7f7d62312700 1 -- 172.21.2.112:0/763026771 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d50002fe0 con 0x7f7d640efe00 2021-02-10T11:30:27.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.102 7f7d5a312700 1 -- 172.21.2.112:0/763026771 >> v1:172.21.2.120:6789/0 conn(0x7f7d641b0470 legacy=0x7f7d640ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:27.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.102 7f7d5a312700 1 -- 172.21.2.112:0/763026771 >> v1:172.21.2.112:6789/0 conn(0x7f7d640efe00 legacy=0x7f7d640f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:27.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.102 7f7d5a312700 1 -- 172.21.2.112:0/763026771 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d640e52e0 con 0x7f7d641b0870 2021-02-10T11:30:27.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.102 7f7d5a312700 1 --2- 172.21.2.112:0/763026771 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7d641b0870 0x7f7d640f20e0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 rx=0x7f7d4c006650 tx=0x7f7d4c009a80).ready entity=mon.0 client_cookie=2505bb7c22994f1e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:27.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.102 7f7d62312700 1 -- 172.21.2.112:0/763026771 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7d4c00c0c0 con 0x7f7d641b0870 2021-02-10T11:30:27.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.102 7f7d62312700 1 -- 172.21.2.112:0/763026771 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d4c00c340 con 0x7f7d641b0870 2021-02-10T11:30:27.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.102 7f7d62312700 1 -- 172.21.2.112:0/763026771 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7d4c00c860 con 0x7f7d641b0870 2021-02-10T11:30:27.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.102 7f7d68da7700 1 -- 172.21.2.112:0/763026771 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7d641b0870 msgr2=0x7f7d640f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:27.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.102 7f7d68da7700 1 --2- 172.21.2.112:0/763026771 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7d641b0870 0x7f7d640f20e0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 rx=0x7f7d4c006650 tx=0x7f7d4c009a80).stop 2021-02-10T11:30:27.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.103 7f7d68da7700 1 -- 172.21.2.112:0/763026771 shutdown_connections 2021-02-10T11:30:27.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.103 7f7d68da7700 1 --2- 172.21.2.112:0/763026771 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7d641b0870 0x7f7d640f20e0 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:27.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.103 7f7d68da7700 1 -- 172.21.2.112:0/763026771 >> 172.21.2.112:0/763026771 conn(0x7f7d640ce3e0 msgr2=0x7f7d640e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:27.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.103 7f7d68da7700 1 -- 172.21.2.112:0/763026771 shutdown_connections 2021-02-10T11:30:27.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.103 7f7d68da7700 1 -- 172.21.2.112:0/763026771 wait complete. 2021-02-10T11:30:27.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.118 7f7d68da7700 1 Processor -- start 2021-02-10T11:30:27.109 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.118 7f7d68da7700 1 -- start start 2021-02-10T11:30:27.109 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.118 7f7d68da7700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7d6409f0f0 0x7f7d64094b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:27.109 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.118 7f7d68da7700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7d64095160 0x7f7d640b7230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:27.109 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.118 7f7d68da7700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7d640b7850 0x7f7d640b7c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:27.109 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.118 7f7d68da7700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d640ada20 con 0x7f7d6409f0f0 2021-02-10T11:30:27.109 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.118 7f7d68da7700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d6409f4f0 con 0x7f7d64095160 2021-02-10T11:30:27.109 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.118 7f7d68da7700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d640adfa0 con 0x7f7d640b7850 2021-02-10T11:30:27.109 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.118 7f7d62b13700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7d6409f0f0 0x7f7d64094b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:27.109 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.118 7f7d62b13700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7d6409f0f0 0x7f7d64094b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:46984/0 (socket says 172.21.2.112:46984) 2021-02-10T11:30:27.109 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.118 7f7d62b13700 1 -- 172.21.2.112:0/1216369809 learned_addr learned my addr 172.21.2.112:0/1216369809 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:30:27.109 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.118 7f7d63314700 1 --2- 172.21.2.112:0/1216369809 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7d640b7850 0x7f7d640b7c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:27.109 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.118 7f7d5a312700 1 --2- 172.21.2.112:0/1216369809 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7d64095160 0x7f7d640b7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:27.109 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.119 7f7d62b13700 1 -- 172.21.2.112:0/1216369809 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7d640b7850 msgr2=0x7f7d640b7c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:27.110 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.119 7f7d62b13700 1 --2- 172.21.2.112:0/1216369809 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7d640b7850 0x7f7d640b7c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:27.110 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.119 7f7d62b13700 1 -- 172.21.2.112:0/1216369809 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7d64095160 msgr2=0x7f7d640b7230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:27.110 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.119 7f7d62b13700 1 --2- 172.21.2.112:0/1216369809 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7d64095160 0x7f7d640b7230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:27.110 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.119 7f7d62b13700 1 -- 172.21.2.112:0/1216369809 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d640a9690 con 0x7f7d6409f0f0 2021-02-10T11:30:27.110 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.119 7f7d62b13700 1 --2- 172.21.2.112:0/1216369809 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7d6409f0f0 0x7f7d64094b40 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 rx=0x7f7d5400b530 tx=0x7f7d5400b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:27.110 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.119 7f7d5bfff700 1 -- 172.21.2.112:0/1216369809 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7d5400bcc0 con 0x7f7d6409f0f0 2021-02-10T11:30:27.110 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.119 7f7d5bfff700 1 -- 172.21.2.112:0/1216369809 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d54006020 con 0x7f7d6409f0f0 2021-02-10T11:30:27.110 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.119 7f7d5bfff700 1 -- 172.21.2.112:0/1216369809 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7d540054a0 con 0x7f7d6409f0f0 2021-02-10T11:30:27.110 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.119 7f7d68da7700 1 -- 172.21.2.112:0/1216369809 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d640a9690 con 0x7f7d6409f0f0 2021-02-10T11:30:27.110 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.119 7f7d68da7700 1 -- 172.21.2.112:0/1216369809 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d640a9690 con 0x7f7d6409f0f0 2021-02-10T11:30:27.110 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.120 7f7d5bfff700 1 -- 172.21.2.112:0/1216369809 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7d54006200 con 0x7f7d6409f0f0 2021-02-10T11:30:27.110 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.120 7f7d5bfff700 1 --2- 172.21.2.112:0/1216369809 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7d480186c0 0x7f7d4801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:27.111 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.120 7f7d5bfff700 1 -- 172.21.2.112:0/1216369809 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f7d54032410 con 0x7f7d6409f0f0 2021-02-10T11:30:27.111 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.120 7f7d5a312700 1 --2- 172.21.2.112:0/1216369809 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7d480186c0 0x7f7d4801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:27.111 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.120 7f7d5a312700 1 --2- 172.21.2.112:0/1216369809 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7d480186c0 0x7f7d4801abc0 crc :-1 s=READY pgs=417 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:27.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.150 7f7d68da7700 1 -- 172.21.2.112:0/1216369809 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d640955b0 con 0x7f7d6409f0f0 2021-02-10T11:30:27.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.153 7f7d5bfff700 1 -- 172.21.2.112:0/1216369809 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7d54030d40 con 0x7f7d6409f0f0 2021-02-10T11:30:27.267 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.280 7f7d68da7700 1 -- 172.21.2.112:0/1216369809 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348598", "target": ["mgr", ""]}) v1 -- 0x7f7d640955b0 con 0x7f7d480186c0 2021-02-10T11:30:27.278 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.292 7f7d5bfff700 1 -- 172.21.2.112:0/1216369809 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f7d4c006420 con 0x7f7d480186c0 2021-02-10T11:30:27.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.297 7f7d68da7700 1 -- 172.21.2.112:0/1216369809 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7d480186c0 msgr2=0x7f7d4801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:27.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.297 7f7d68da7700 1 --2- 172.21.2.112:0/1216369809 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7d480186c0 0x7f7d4801abc0 crc :-1 s=READY pgs=417 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:27.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.297 7f7d68da7700 1 -- 172.21.2.112:0/1216369809 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7d6409f0f0 msgr2=0x7f7d64094b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:27.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.297 7f7d68da7700 1 --2- 172.21.2.112:0/1216369809 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7d6409f0f0 0x7f7d64094b40 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 rx=0x7f7d5400b530 tx=0x7f7d5400b8d0).stop 2021-02-10T11:30:27.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.298 7f7d68da7700 1 -- 172.21.2.112:0/1216369809 shutdown_connections 2021-02-10T11:30:27.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.298 7f7d68da7700 1 --2- 172.21.2.112:0/1216369809 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7d480186c0 0x7f7d4801abc0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:27.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.298 7f7d68da7700 1 --2- 172.21.2.112:0/1216369809 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7d64095160 0x7f7d640b7230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:27.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.298 7f7d68da7700 1 --2- 172.21.2.112:0/1216369809 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7d640b7850 0x7f7d640b7c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:27.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.298 7f7d68da7700 1 --2- 172.21.2.112:0/1216369809 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7d6409f0f0 0x7f7d64094b40 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:27.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.298 7f7d68da7700 1 -- 172.21.2.112:0/1216369809 >> 172.21.2.112:0/1216369809 conn(0x7f7d640c40e0 msgr2=0x7f7d640f05a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:27.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.298 7f7d68da7700 1 -- 172.21.2.112:0/1216369809 shutdown_connections 2021-02-10T11:30:27.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.298 7f7d68da7700 1 -- 172.21.2.112:0/1216369809 wait complete. 2021-02-10T11:30:27.294 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000348599 2021-02-10T11:30:27.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.719 7fe0f4b13700 1 Processor -- start 2021-02-10T11:30:27.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.719 7fe0f4b13700 1 -- start start 2021-02-10T11:30:27.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.720 7fe0f4b13700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe0f01b0470 0x7fe0f01b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:27.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.720 7fe0f4b13700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fe0f00ef610 0x7fe0f00efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:27.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.720 7fe0f4b13700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fe0f00f0030 0x7fe0f00f0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:27.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.720 7fe0f4b13700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0f00e5ea0 con 0x7fe0f00ef610 2021-02-10T11:30:27.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.720 7fe0f4b13700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0f00f0b50 con 0x7fe0f00f0030 2021-02-10T11:30:27.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.720 7fe0f4b13700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0f00e52e0 con 0x7fe0f01b0470 2021-02-10T11:30:27.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.720 7fe0ef05f700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fe0f00f0030 0x7fe0f00f0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:27.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.720 7fe0ee05d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fe0f00ef610 0x7fe0f00efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:27.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.720 7fe0ef05f700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fe0f00f0030 0x7fe0f00f0430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3300/0 says I am v2:172.21.2.112:49272/0 (socket says 172.21.2.112:49272) 2021-02-10T11:30:27.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.720 7fe0ee05d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fe0f00ef610 0x7fe0f00efa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:46990/0 (socket says 172.21.2.112:46990) 2021-02-10T11:30:27.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.720 7fe0ef05f700 1 -- 172.21.2.112:0/2167935726 learned_addr learned my addr 172.21.2.112:0/2167935726 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:30:27.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.720 7fe0ee85e700 1 --2- 172.21.2.112:0/2167935726 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe0f01b0470 0x7fe0f01b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:27.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.720 7fe0ee05d700 1 -- 172.21.2.112:0/2167935726 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe0f01b0470 msgr2=0x7fe0f01b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:27.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.720 7fe0ee05d700 1 --2- 172.21.2.112:0/2167935726 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe0f01b0470 0x7fe0f01b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:27.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.720 7fe0ee05d700 1 -- 172.21.2.112:0/2167935726 >> v2:172.21.2.120:3300/0 conn(0x7fe0f00f0030 msgr2=0x7fe0f00f0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:27.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.720 7fe0ee05d700 1 --2- 172.21.2.112:0/2167935726 >> v2:172.21.2.120:3300/0 conn(0x7fe0f00f0030 0x7fe0f00f0430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:27.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.720 7fe0ee05d700 1 -- 172.21.2.112:0/2167935726 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0f00f10d0 con 0x7fe0f00ef610 2021-02-10T11:30:27.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.720 7fe0ef05f700 1 --2- 172.21.2.112:0/2167935726 >> v2:172.21.2.120:3300/0 conn(0x7fe0f00f0030 0x7fe0f00f0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:30:27.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.720 7fe0ee85e700 1 --2- 172.21.2.112:0/2167935726 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe0f01b0470 0x7fe0f01b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:30:27.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.720 7fe0ee05d700 1 --2- 172.21.2.112:0/2167935726 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe0f00ef610 0x7fe0f00efa10 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 rx=0x7fe0e80069d0 tx=0x7fe0e8009e40).ready entity=mon.0 client_cookie=98fcb4e909a1eed2 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:27.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.721 7fe0ed85c700 1 -- 172.21.2.112:0/2167935726 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe0e80066c0 con 0x7fe0f00ef610 2021-02-10T11:30:27.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.721 7fe0ed85c700 1 -- 172.21.2.112:0/2167935726 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0e800b510 con 0x7fe0f00ef610 2021-02-10T11:30:27.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.721 7fe0ed85c700 1 -- 172.21.2.112:0/2167935726 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe0e800b970 con 0x7fe0f00ef610 2021-02-10T11:30:27.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.721 7fe0f4b13700 1 -- 172.21.2.112:0/2167935726 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe0f00ef610 msgr2=0x7fe0f00efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:27.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.721 7fe0f4b13700 1 --2- 172.21.2.112:0/2167935726 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe0f00ef610 0x7fe0f00efa10 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 rx=0x7fe0e80069d0 tx=0x7fe0e8009e40).stop 2021-02-10T11:30:27.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.721 7fe0f4b13700 1 -- 172.21.2.112:0/2167935726 shutdown_connections 2021-02-10T11:30:27.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.721 7fe0f4b13700 1 --2- 172.21.2.112:0/2167935726 >> v2:172.21.2.120:3300/0 conn(0x7fe0f00f0030 0x7fe0f00f0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:27.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.721 7fe0f4b13700 1 --2- 172.21.2.112:0/2167935726 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe0f00ef610 0x7fe0f00efa10 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:27.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.721 7fe0f4b13700 1 --2- 172.21.2.112:0/2167935726 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe0f01b0470 0x7fe0f01b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:27.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.721 7fe0f4b13700 1 -- 172.21.2.112:0/2167935726 >> 172.21.2.112:0/2167935726 conn(0x7fe0f00ce3e0 msgr2=0x7fe0f00e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:27.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.721 7fe0f4b13700 1 -- 172.21.2.112:0/2167935726 shutdown_connections 2021-02-10T11:30:27.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.721 7fe0f4b13700 1 -- 172.21.2.112:0/2167935726 wait complete. 2021-02-10T11:30:27.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.736 7fe0f4b13700 1 Processor -- start 2021-02-10T11:30:27.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.736 7fe0f4b13700 1 -- start start 2021-02-10T11:30:27.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.736 7fe0f4b13700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe0f009b930 0x7fe0f00b7c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:27.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.736 7fe0f4b13700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe0f00b8410 0x7fe0f00b8810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:27.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.736 7fe0f4b13700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe0f00c1f80 0x7fe0f00c2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:27.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.736 7fe0f4b13700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0f00c1a00 con 0x7fe0f009b930 2021-02-10T11:30:27.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.736 7fe0f4b13700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0f00b8e30 con 0x7fe0f00c1f80 2021-02-10T11:30:27.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.736 7fe0f4b13700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0f00b9050 con 0x7fe0f00b8410 2021-02-10T11:30:27.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.736 7fe0ef05f700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe0f00c1f80 0x7fe0f00c2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:27.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.736 7fe0ee85e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe0f009b930 0x7fe0f00b7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:27.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.736 7fe0ef05f700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe0f00c1f80 0x7fe0f00c2380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3300/0 says I am v2:172.21.2.112:49274/0 (socket says 172.21.2.112:49274) 2021-02-10T11:30:27.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.736 7fe0ef05f700 1 -- 172.21.2.112:0/2547559974 learned_addr learned my addr 172.21.2.112:0/2547559974 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:30:27.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.736 7fe0ee85e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe0f009b930 0x7fe0f00b7c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:46996/0 (socket says 172.21.2.112:46996) 2021-02-10T11:30:27.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.736 7fe0ee05d700 1 --2- 172.21.2.112:0/2547559974 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe0f00b8410 0x7fe0f00b8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:27.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.737 7fe0ee85e700 1 -- 172.21.2.112:0/2547559974 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe0f00b8410 msgr2=0x7fe0f00b8810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:27.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.737 7fe0ee85e700 1 --2- 172.21.2.112:0/2547559974 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe0f00b8410 0x7fe0f00b8810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:27.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.737 7fe0ee85e700 1 -- 172.21.2.112:0/2547559974 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe0f00c1f80 msgr2=0x7fe0f00c2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:27.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.737 7fe0ee85e700 1 --2- 172.21.2.112:0/2547559974 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe0f00c1f80 0x7fe0f00c2380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:27.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.737 7fe0ee85e700 1 -- 172.21.2.112:0/2547559974 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0f00c29a0 con 0x7fe0f009b930 2021-02-10T11:30:27.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.737 7fe0ee85e700 1 --2- 172.21.2.112:0/2547559974 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe0f009b930 0x7fe0f00b7c20 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 rx=0x7fe0e400b510 tx=0x7fe0e400b8b0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:27.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.737 7fe0df7fe700 1 -- 172.21.2.112:0/2547559974 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe0e400bca0 con 0x7fe0f009b930 2021-02-10T11:30:27.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.737 7fe0df7fe700 1 -- 172.21.2.112:0/2547559974 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0e4002030 con 0x7fe0f009b930 2021-02-10T11:30:27.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.737 7fe0df7fe700 1 -- 172.21.2.112:0/2547559974 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe0e4002550 con 0x7fe0f009b930 2021-02-10T11:30:27.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.737 7fe0f4b13700 1 -- 172.21.2.112:0/2547559974 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0f00c29a0 con 0x7fe0f009b930 2021-02-10T11:30:27.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.737 7fe0f4b13700 1 -- 172.21.2.112:0/2547559974 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0f00c29a0 con 0x7fe0f009b930 2021-02-10T11:30:27.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.738 7fe0df7fe700 1 -- 172.21.2.112:0/2547559974 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe0e4000c90 con 0x7fe0f009b930 2021-02-10T11:30:27.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.738 7fe0df7fe700 1 --2- 172.21.2.112:0/2547559974 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe0d40186c0 0x7fe0d401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:27.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.738 7fe0df7fe700 1 -- 172.21.2.112:0/2547559974 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fe0e4032710 con 0x7fe0f009b930 2021-02-10T11:30:27.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.738 7fe0ee05d700 1 --2- 172.21.2.112:0/2547559974 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe0d40186c0 0x7fe0d401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:27.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.739 7fe0ee05d700 1 --2- 172.21.2.112:0/2547559974 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe0d40186c0 0x7fe0d401abc0 crc :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:27.757 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.770 7fe0f4b13700 1 -- 172.21.2.112:0/2547559974 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0f0105b30 con 0x7fe0f009b930 2021-02-10T11:30:27.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.772 7fe0df7fe700 1 -- 172.21.2.112:0/2547559974 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe0e401f450 con 0x7fe0f009b930 2021-02-10T11:30:27.887 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.899 7fe0f4b13700 1 -- 172.21.2.112:0/2547559974 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348599", "target": ["mgr", ""]}) v1 -- 0x7fe0f00b8e30 con 0x7fe0d40186c0 2021-02-10T11:30:27.897 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.910 7fe0df7fe700 1 -- 172.21.2.112:0/2547559974 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fe0e8006c50 con 0x7fe0d40186c0 2021-02-10T11:30:27.904 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.917 7fe0f4b13700 1 -- 172.21.2.112:0/2547559974 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe0d40186c0 msgr2=0x7fe0d401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:27.905 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.917 7fe0f4b13700 1 --2- 172.21.2.112:0/2547559974 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe0d40186c0 0x7fe0d401abc0 crc :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:27.905 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.917 7fe0f4b13700 1 -- 172.21.2.112:0/2547559974 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe0f009b930 msgr2=0x7fe0f00b7c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:27.905 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.917 7fe0f4b13700 1 --2- 172.21.2.112:0/2547559974 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe0f009b930 0x7fe0f00b7c20 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 rx=0x7fe0e400b510 tx=0x7fe0e400b8b0).stop 2021-02-10T11:30:27.905 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.917 7fe0f4b13700 1 -- 172.21.2.112:0/2547559974 shutdown_connections 2021-02-10T11:30:27.905 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.917 7fe0f4b13700 1 --2- 172.21.2.112:0/2547559974 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe0d40186c0 0x7fe0d401abc0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:27.905 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.917 7fe0f4b13700 1 --2- 172.21.2.112:0/2547559974 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe0f00c1f80 0x7fe0f00c2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:27.905 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.917 7fe0f4b13700 1 --2- 172.21.2.112:0/2547559974 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe0f00b8410 0x7fe0f00b8810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:27.905 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.917 7fe0f4b13700 1 --2- 172.21.2.112:0/2547559974 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe0f009b930 0x7fe0f00b7c20 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:27.905 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.917 7fe0f4b13700 1 -- 172.21.2.112:0/2547559974 >> 172.21.2.112:0/2547559974 conn(0x7fe0f0094e10 msgr2=0x7fe0f00a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:27.906 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.917 7fe0f4b13700 1 -- 172.21.2.112:0/2547559974 shutdown_connections 2021-02-10T11:30:27.906 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:27.917 7fe0f4b13700 1 -- 172.21.2.112:0/2547559974 wait complete. 2021-02-10T11:30:27.913 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000348600 2021-02-10T11:30:28.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.338 7fbb78dd2700 1 Processor -- start 2021-02-10T11:30:28.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.338 7fbb78dd2700 1 -- start start 2021-02-10T11:30:28.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.339 7fbb78dd2700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fbb741b0870 0x7fbb740f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:28.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.339 7fbb78dd2700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fbb740efe00 0x7fbb740f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:28.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.339 7fbb78dd2700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb741b0c70 con 0x7fbb741b0470 2021-02-10T11:30:28.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.339 7fbb78dd2700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb740e5ea0 con 0x7fbb741b0870 2021-02-10T11:30:28.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.339 7fbb78dd2700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb740d2fb0 con 0x7fbb740efe00 2021-02-10T11:30:28.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.339 7fbb6a312700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fbb741b0870 0x7fbb740f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:28.344 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.339 7fbb6a312700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fbb741b0870 0x7fbb740f20e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:47004/0 (socket says 172.21.2.112:47004) 2021-02-10T11:30:28.344 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.339 7fbb6a312700 1 -- 172.21.2.112:0/4033816020 learned_addr learned my addr 172.21.2.112:0/4033816020 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:30:28.344 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.339 7fbb73314700 1 --2- 172.21.2.112:0/4033816020 >> v2:172.21.2.120:3300/0 conn(0x7fbb740efe00 0x7fbb740f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:28.344 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.340 7fbb6a312700 1 -- 172.21.2.112:0/4033816020 >> v2:172.21.2.120:3300/0 conn(0x7fbb740efe00 msgr2=0x7fbb740f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:28.344 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.340 7fbb6a312700 1 --2- 172.21.2.112:0/4033816020 >> v2:172.21.2.120:3300/0 conn(0x7fbb740efe00 0x7fbb740f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:28.344 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.340 7fbb6a312700 1 -- 172.21.2.112:0/4033816020 >> v1:172.21.2.120:6789/0 conn(0x7fbb741b0470 legacy=0x7fbb740ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:28.344 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.340 7fbb6a312700 1 -- 172.21.2.112:0/4033816020 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb740f0b50 con 0x7fbb741b0870 2021-02-10T11:30:28.344 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.340 7fbb73314700 1 --2- 172.21.2.112:0/4033816020 >> v2:172.21.2.120:3300/0 conn(0x7fbb740efe00 0x7fbb740f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:30:28.344 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.340 7fbb6a312700 1 --2- 172.21.2.112:0/4033816020 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbb741b0870 0x7fbb740f20e0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 rx=0x7fbb5c006700 tx=0x7fbb5c009b80).ready entity=mon.0 client_cookie=e7832be58b64e073 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:28.344 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.340 7fbb72312700 1 -- 172.21.2.112:0/4033816020 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbb5c00c0c0 con 0x7fbb741b0870 2021-02-10T11:30:28.344 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.340 7fbb72312700 1 -- 172.21.2.112:0/4033816020 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb5c00c440 con 0x7fbb741b0870 2021-02-10T11:30:28.344 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.340 7fbb72312700 1 -- 172.21.2.112:0/4033816020 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbb5c00c920 con 0x7fbb741b0870 2021-02-10T11:30:28.344 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.340 7fbb78dd2700 1 -- 172.21.2.112:0/4033816020 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbb741b0870 msgr2=0x7fbb740f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:28.345 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.340 7fbb78dd2700 1 --2- 172.21.2.112:0/4033816020 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbb741b0870 0x7fbb740f20e0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 rx=0x7fbb5c006700 tx=0x7fbb5c009b80).stop 2021-02-10T11:30:28.345 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.340 7fbb78dd2700 1 -- 172.21.2.112:0/4033816020 shutdown_connections 2021-02-10T11:30:28.345 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.340 7fbb78dd2700 1 --2- 172.21.2.112:0/4033816020 >> v2:172.21.2.120:3300/0 conn(0x7fbb740efe00 0x7fbb740f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:28.345 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.340 7fbb78dd2700 1 --2- 172.21.2.112:0/4033816020 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbb741b0870 0x7fbb740f20e0 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:28.345 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.340 7fbb78dd2700 1 -- 172.21.2.112:0/4033816020 >> 172.21.2.112:0/4033816020 conn(0x7fbb740ce3e0 msgr2=0x7fbb740e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:28.345 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.340 7fbb78dd2700 1 -- 172.21.2.112:0/4033816020 shutdown_connections 2021-02-10T11:30:28.345 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.340 7fbb78dd2700 1 -- 172.21.2.112:0/4033816020 wait complete. 2021-02-10T11:30:28.345 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.355 7fbb78dd2700 1 Processor -- start 2021-02-10T11:30:28.345 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.355 7fbb78dd2700 1 -- start start 2021-02-10T11:30:28.347 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.356 7fbb78dd2700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbb741a0470 0x7fbb7408a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:28.347 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.356 7fbb78dd2700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbb7408aa50 0x7fbb74099180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:28.348 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.356 7fbb78dd2700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fbb740997a0 0x7fbb74099ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:28.348 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.356 7fbb78dd2700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb74095610 con 0x7fbb7408aa50 2021-02-10T11:30:28.348 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.356 7fbb78dd2700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb74095800 con 0x7fbb740997a0 2021-02-10T11:30:28.348 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.356 7fbb78dd2700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb7409a1c0 con 0x7fbb741a0470 2021-02-10T11:30:28.348 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.356 7fbb6a312700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbb7408aa50 0x7fbb74099180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:28.348 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.356 7fbb72b13700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbb741a0470 0x7fbb7408a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:28.348 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.356 7fbb6a312700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbb7408aa50 0x7fbb74099180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:47010/0 (socket says 172.21.2.112:47010) 2021-02-10T11:30:28.348 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.356 7fbb6a312700 1 -- 172.21.2.112:0/2578635297 learned_addr learned my addr 172.21.2.112:0/2578635297 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:30:28.348 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.356 7fbb72b13700 1 --2- 172.21.2.112:0/2578635297 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbb741a0470 0x7fbb7408a430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3301/0 says I am v2:172.21.2.112:46690/0 (socket says 172.21.2.112:46690) 2021-02-10T11:30:28.348 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.356 7fbb73314700 1 --2- 172.21.2.112:0/2578635297 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fbb740997a0 0x7fbb74099ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:28.348 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.356 7fbb6a312700 1 -- 172.21.2.112:0/2578635297 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbb741a0470 msgr2=0x7fbb7408a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:28.348 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.356 7fbb6a312700 1 --2- 172.21.2.112:0/2578635297 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbb741a0470 0x7fbb7408a430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:28.349 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.356 7fbb6a312700 1 -- 172.21.2.112:0/2578635297 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fbb740997a0 msgr2=0x7fbb74099ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:28.349 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.356 7fbb6a312700 1 --2- 172.21.2.112:0/2578635297 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fbb740997a0 0x7fbb74099ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:28.349 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.357 7fbb6a312700 1 -- 172.21.2.112:0/2578635297 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb7409a3b0 con 0x7fbb7408aa50 2021-02-10T11:30:28.349 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.357 7fbb6a312700 1 --2- 172.21.2.112:0/2578635297 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbb7408aa50 0x7fbb74099180 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 rx=0x7fbb5c00b5e0 tx=0x7fbb5c00b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:28.349 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.357 7fbb6bfff700 1 -- 172.21.2.112:0/2578635297 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbb5c00bd70 con 0x7fbb7408aa50 2021-02-10T11:30:28.349 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.357 7fbb6bfff700 1 -- 172.21.2.112:0/2578635297 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb5c002030 con 0x7fbb7408aa50 2021-02-10T11:30:28.349 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.357 7fbb78dd2700 1 -- 172.21.2.112:0/2578635297 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb7409a3b0 con 0x7fbb7408aa50 2021-02-10T11:30:28.349 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.357 7fbb78dd2700 1 -- 172.21.2.112:0/2578635297 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb7408b850 con 0x7fbb7408aa50 2021-02-10T11:30:28.349 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.357 7fbb6bfff700 1 -- 172.21.2.112:0/2578635297 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbb5c002550 con 0x7fbb7408aa50 2021-02-10T11:30:28.349 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.358 7fbb6bfff700 1 -- 172.21.2.112:0/2578635297 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fbb5c000c90 con 0x7fbb7408aa50 2021-02-10T11:30:28.349 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.358 7fbb6bfff700 1 --2- 172.21.2.112:0/2578635297 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fbb58018730 0x7fbb5801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:28.349 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.358 7fbb6bfff700 1 -- 172.21.2.112:0/2578635297 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fbb5c032810 con 0x7fbb7408aa50 2021-02-10T11:30:28.350 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.358 7fbb72b13700 1 --2- 172.21.2.112:0/2578635297 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fbb58018730 0x7fbb5801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:28.350 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.358 7fbb72b13700 1 --2- 172.21.2.112:0/2578635297 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fbb58018730 0x7fbb5801ac10 crc :-1 s=READY pgs=419 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:28.376 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.388 7fbb78dd2700 1 -- 172.21.2.112:0/2578635297 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb74105ae0 con 0x7fbb7408aa50 2021-02-10T11:30:28.378 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.391 7fbb6bfff700 1 -- 172.21.2.112:0/2578635297 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fbb5c0009b0 con 0x7fbb7408aa50 2021-02-10T11:30:28.505 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.518 7fbb78dd2700 1 -- 172.21.2.112:0/2578635297 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348600", "target": ["mgr", ""]}) v1 -- 0x7fbb7408b850 con 0x7fbb58018730 2021-02-10T11:30:28.516 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.529 7fbb6bfff700 1 -- 172.21.2.112:0/2578635297 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fbb64000ce0 con 0x7fbb58018730 2021-02-10T11:30:28.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.536 7fbb78dd2700 1 -- 172.21.2.112:0/2578635297 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fbb58018730 msgr2=0x7fbb5801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:28.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.536 7fbb78dd2700 1 --2- 172.21.2.112:0/2578635297 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fbb58018730 0x7fbb5801ac10 crc :-1 s=READY pgs=419 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:28.524 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.536 7fbb78dd2700 1 -- 172.21.2.112:0/2578635297 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbb7408aa50 msgr2=0x7fbb74099180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:28.524 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.536 7fbb78dd2700 1 --2- 172.21.2.112:0/2578635297 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbb7408aa50 0x7fbb74099180 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 rx=0x7fbb5c00b5e0 tx=0x7fbb5c00b980).stop 2021-02-10T11:30:28.524 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.536 7fbb78dd2700 1 -- 172.21.2.112:0/2578635297 shutdown_connections 2021-02-10T11:30:28.524 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.536 7fbb78dd2700 1 --2- 172.21.2.112:0/2578635297 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fbb58018730 0x7fbb5801ac10 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:28.524 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.536 7fbb78dd2700 1 --2- 172.21.2.112:0/2578635297 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fbb740997a0 0x7fbb74099ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:28.524 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.536 7fbb78dd2700 1 --2- 172.21.2.112:0/2578635297 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbb7408aa50 0x7fbb74099180 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:28.524 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.536 7fbb78dd2700 1 --2- 172.21.2.112:0/2578635297 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbb741a0470 0x7fbb7408a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:28.524 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.536 7fbb78dd2700 1 -- 172.21.2.112:0/2578635297 >> 172.21.2.112:0/2578635297 conn(0x7fbb740bbf60 msgr2=0x7fbb74105d10 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:28.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.536 7fbb78dd2700 1 -- 172.21.2.112:0/2578635297 shutdown_connections 2021-02-10T11:30:28.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.536 7fbb78dd2700 1 -- 172.21.2.112:0/2578635297 wait complete. 2021-02-10T11:30:28.532 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000348601 2021-02-10T11:30:28.962 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.958 7f639dddf700 1 Processor -- start 2021-02-10T11:30:28.962 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.958 7f639dddf700 1 -- start start 2021-02-10T11:30:28.962 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.958 7f639dddf700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f63981b0470 0x7f63981b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:28.962 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.958 7f639dddf700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f63980ef610 0x7f63980efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:28.962 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.958 7f639dddf700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f63980f0030 0x7f63980f0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:28.962 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.958 7f639dddf700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63980e5ea0 con 0x7f63981b0470 2021-02-10T11:30:28.962 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.958 7f639dddf700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63980f0b50 con 0x7f63980ef610 2021-02-10T11:30:28.962 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.958 7f639dddf700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63980e52e0 con 0x7f63980f0030 2021-02-10T11:30:28.962 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.959 7f638effd700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f63980ef610 0x7f63980efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:28.962 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.959 7f63977fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f63981b0470 0x7f63981b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:28.963 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.959 7f638effd700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f63980ef610 0x7f63980efa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3300/0 says I am v2:172.21.2.112:49296/0 (socket says 172.21.2.112:49296) 2021-02-10T11:30:28.963 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.959 7f63977fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f63981b0470 0x7f63981b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:47016/0 (socket says 172.21.2.112:47016) 2021-02-10T11:30:28.963 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.959 7f638effd700 1 -- 172.21.2.112:0/1021014238 learned_addr learned my addr 172.21.2.112:0/1021014238 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:30:28.963 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.959 7f6397fff700 1 --2- 172.21.2.112:0/1021014238 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f63980f0030 0x7f63980f0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:28.963 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.959 7f6397fff700 1 -- 172.21.2.112:0/1021014238 >> v2:172.21.2.120:3300/0 conn(0x7f63980ef610 msgr2=0x7f63980efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:28.963 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.959 7f6397fff700 1 --2- 172.21.2.112:0/1021014238 >> v2:172.21.2.120:3300/0 conn(0x7f63980ef610 0x7f63980efa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:28.963 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.959 7f6397fff700 1 -- 172.21.2.112:0/1021014238 >> v2:172.21.2.112:3300/0 conn(0x7f63981b0470 msgr2=0x7f63981b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:28.963 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.959 7f6397fff700 1 --2- 172.21.2.112:0/1021014238 >> v2:172.21.2.112:3300/0 conn(0x7f63981b0470 0x7f63981b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:28.963 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.959 7f6397fff700 1 -- 172.21.2.112:0/1021014238 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63980f10d0 con 0x7f63980f0030 2021-02-10T11:30:28.963 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.959 7f6397fff700 1 --2- 172.21.2.112:0/1021014238 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f63980f0030 0x7f63980f0430 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f6390006e50 tx=0x7f6390009e70).ready entity=mon.2 client_cookie=318db3a555e45478 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:28.963 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.959 7f6396ffd700 1 -- 172.21.2.112:0/1021014238 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f639000b510 con 0x7f63980f0030 2021-02-10T11:30:28.963 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.959 7f6396ffd700 1 -- 172.21.2.112:0/1021014238 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6390006790 con 0x7f63980f0030 2021-02-10T11:30:28.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.959 7f6396ffd700 1 -- 172.21.2.112:0/1021014238 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f639000b940 con 0x7f63980f0030 2021-02-10T11:30:28.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.959 7f639dddf700 1 -- 172.21.2.112:0/1021014238 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f63980f0030 msgr2=0x7f63980f0430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:28.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.959 7f639dddf700 1 --2- 172.21.2.112:0/1021014238 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f63980f0030 0x7f63980f0430 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f6390006e50 tx=0x7f6390009e70).stop 2021-02-10T11:30:28.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.960 7f639dddf700 1 -- 172.21.2.112:0/1021014238 shutdown_connections 2021-02-10T11:30:28.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.960 7f639dddf700 1 --2- 172.21.2.112:0/1021014238 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f63980f0030 0x7f63980f0430 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:28.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.960 7f639dddf700 1 --2- 172.21.2.112:0/1021014238 >> v2:172.21.2.120:3300/0 conn(0x7f63980ef610 0x7f63980efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:28.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.960 7f639dddf700 1 --2- 172.21.2.112:0/1021014238 >> v2:172.21.2.112:3300/0 conn(0x7f63981b0470 0x7f63981b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:28.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.960 7f639dddf700 1 -- 172.21.2.112:0/1021014238 >> 172.21.2.112:0/1021014238 conn(0x7f63980ce3e0 msgr2=0x7f63980e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:28.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.960 7f639dddf700 1 -- 172.21.2.112:0/1021014238 shutdown_connections 2021-02-10T11:30:28.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.960 7f639dddf700 1 -- 172.21.2.112:0/1021014238 wait complete. 2021-02-10T11:30:28.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.974 7f639dddf700 1 Processor -- start 2021-02-10T11:30:28.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.975 7f639dddf700 1 -- start start 2021-02-10T11:30:28.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.975 7f639dddf700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f639809ba50 0x7f639810a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:28.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.975 7f639dddf700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f63980c27f0 0x7f63981095c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:28.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.975 7f639dddf700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f63980c32d0 0x7f63980c36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:28.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.975 7f639dddf700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6398199f50 con 0x7f63980c32d0 2021-02-10T11:30:28.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.975 7f639dddf700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63980c2d50 con 0x7f639809ba50 2021-02-10T11:30:28.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.975 7f639dddf700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f639810a730 con 0x7f63980c27f0 2021-02-10T11:30:28.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.975 7f63977fe700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f639809ba50 0x7f639810a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:28.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.975 7f6397fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f63980c32d0 0x7f63980c36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:28.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.975 7f63977fe700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f639809ba50 0x7f639810a110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3300/0 says I am v2:172.21.2.112:49304/0 (socket says 172.21.2.112:49304) 2021-02-10T11:30:28.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.975 7f63977fe700 1 -- 172.21.2.112:0/3776340756 learned_addr learned my addr 172.21.2.112:0/3776340756 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:30:28.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.975 7f638effd700 1 --2- 172.21.2.112:0/3776340756 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f63980c27f0 0x7f63981095c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:28.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.976 7f63977fe700 1 -- 172.21.2.112:0/3776340756 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f63980c27f0 msgr2=0x7f63981095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:28.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.976 7f63977fe700 1 --2- 172.21.2.112:0/3776340756 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f63980c27f0 0x7f63981095c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:28.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.976 7f63977fe700 1 -- 172.21.2.112:0/3776340756 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f63980c32d0 msgr2=0x7f63980c36d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:28.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.976 7f63977fe700 1 --2- 172.21.2.112:0/3776340756 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f63980c32d0 0x7f63980c36d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:28.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.976 7f63977fe700 1 -- 172.21.2.112:0/3776340756 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f639809be50 con 0x7f639809ba50 2021-02-10T11:30:28.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.976 7f6397fff700 1 --2- 172.21.2.112:0/3776340756 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f63980c32d0 0x7f63980c36d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:30:28.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.976 7f63977fe700 1 --2- 172.21.2.112:0/3776340756 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f639809ba50 0x7f639810a110 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0x7f6388006f60 tx=0x7f638800b970).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:28.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.976 7f6394ff9700 1 -- 172.21.2.112:0/3776340756 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6388006340 con 0x7f639809ba50 2021-02-10T11:30:28.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.976 7f6394ff9700 1 -- 172.21.2.112:0/3776340756 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f638800bdf0 con 0x7f639809ba50 2021-02-10T11:30:28.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.976 7f6394ff9700 1 -- 172.21.2.112:0/3776340756 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6388002030 con 0x7f639809ba50 2021-02-10T11:30:28.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.976 7f639dddf700 1 -- 172.21.2.112:0/3776340756 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f639809be50 con 0x7f639809ba50 2021-02-10T11:30:28.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.976 7f639dddf700 1 -- 172.21.2.112:0/3776340756 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f639809be50 con 0x7f639809ba50 2021-02-10T11:30:28.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.977 7f6394ff9700 1 -- 172.21.2.112:0/3776340756 <== mon.1 v2:172.21.2.120:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f63880052e0 con 0x7f639809ba50 2021-02-10T11:30:28.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.977 7f6394ff9700 1 --2- 172.21.2.112:0/3776340756 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f637c0186c0 0x7f637c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:28.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.977 7f6394ff9700 1 -- 172.21.2.112:0/3776340756 <== mon.1 v2:172.21.2.120:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f6388032530 con 0x7f639809ba50 2021-02-10T11:30:28.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.977 7f638effd700 1 --2- 172.21.2.112:0/3776340756 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f637c0186c0 0x7f637c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:28.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:28.978 7f638effd700 1 --2- 172.21.2.112:0/3776340756 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f637c0186c0 0x7f637c01abc0 crc :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:28.995 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:29.008 7f639dddf700 1 -- 172.21.2.112:0/3776340756 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6398199f50 con 0x7f639809ba50 2021-02-10T11:30:28.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:29.011 7f6394ff9700 1 -- 172.21.2.112:0/3776340756 <== mon.1 v2:172.21.2.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6388006b50 con 0x7f639809ba50 2021-02-10T11:30:29.132 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:29.144 7f639dddf700 1 -- 172.21.2.112:0/3776340756 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348601", "target": ["mgr", ""]}) v1 -- 0x7f6398199f50 con 0x7f637c0186c0 2021-02-10T11:30:29.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:29.152 7f6394ff9700 1 -- 172.21.2.112:0/3776340756 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f6380006a80 con 0x7f637c0186c0 2021-02-10T11:30:29.150 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:29.161 7f639dddf700 1 -- 172.21.2.112:0/3776340756 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f637c0186c0 msgr2=0x7f637c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:29.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:29.161 7f639dddf700 1 --2- 172.21.2.112:0/3776340756 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f637c0186c0 0x7f637c01abc0 crc :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:29.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:29.161 7f639dddf700 1 -- 172.21.2.112:0/3776340756 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f639809ba50 msgr2=0x7f639810a110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:29.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:29.161 7f639dddf700 1 --2- 172.21.2.112:0/3776340756 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f639809ba50 0x7f639810a110 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0x7f6388006f60 tx=0x7f638800b970).stop 2021-02-10T11:30:29.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:29.162 7f639dddf700 1 -- 172.21.2.112:0/3776340756 shutdown_connections 2021-02-10T11:30:29.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:29.162 7f639dddf700 1 --2- 172.21.2.112:0/3776340756 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f637c0186c0 0x7f637c01abc0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:29.152 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:29.162 7f639dddf700 1 --2- 172.21.2.112:0/3776340756 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f63980c32d0 0x7f63980c36d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:29.152 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:29.162 7f639dddf700 1 --2- 172.21.2.112:0/3776340756 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f63980c27f0 0x7f63981095c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:29.152 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:29.162 7f639dddf700 1 --2- 172.21.2.112:0/3776340756 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f639809ba50 0x7f639810a110 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:29.152 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:29.162 7f639dddf700 1 -- 172.21.2.112:0/3776340756 >> 172.21.2.112:0/3776340756 conn(0x7f63980c40e0 msgr2=0x7f63980bf620 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:29.152 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:29.162 7f639dddf700 1 -- 172.21.2.112:0/3776340756 shutdown_connections 2021-02-10T11:30:29.152 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:29.162 7f639dddf700 1 -- 172.21.2.112:0/3776340756 wait complete. 2021-02-10T11:30:29.158 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000348602 2021-02-10T11:30:29.588 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:29.583 7f576db0f700 1 Processor -- start 2021-02-10T11:30:29.588 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:29.583 7f576db0f700 1 -- start start 2021-02-10T11:30:29.589 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:29.584 7f576db0f700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f57681b0870 0x7f57680f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:29.589 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:29.584 7f576db0f700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f57680efe00 0x7f57680f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:29.589 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:29.584 7f576db0f700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57681b0c70 con 0x7f57681b0470 2021-02-10T11:30:29.589 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:29.584 7f576db0f700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57680f0b50 con 0x7f57680efe00 2021-02-10T11:30:29.589 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:29.584 7f576db0f700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57680f0820 con 0x7f57681b0870 2021-02-10T11:30:29.589 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:29.584 7f57677fe700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f57681b0470 0x7f57680ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:46492/0 (socket says 172.21.2.112:46492) 2021-02-10T11:30:29.589 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:29.584 7f57677fe700 1 -- 172.21.2.112:0/3720455319 learned_addr learned my addr 172.21.2.112:0/3720455319 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:30:29.589 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:29.584 7f5767fff700 1 --2- 172.21.2.112:0/3720455319 >> v2:172.21.2.120:3300/0 conn(0x7f57680efe00 0x7f57680f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:29.589 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:29.584 7f5766ffd700 1 --2- 172.21.2.112:0/3720455319 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f57681b0870 0x7f57680f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:29.589 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:29.584 7f57667fc700 1 -- 172.21.2.112:0/3720455319 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 733056447 0 0) 0x7f57500010b0 con 0x7f57681b0470 2021-02-10T11:30:29.589 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:29.585 7f57667fc700 1 -- 172.21.2.112:0/3720455319 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5754002fe0 con 0x7f57681b0470 2021-02-10T11:30:29.589 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:29.585 7f5767fff700 1 -- 172.21.2.112:0/3720455319 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f57681b0870 msgr2=0x7f57680f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:29.589 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:29.585 7f5767fff700 1 --2- 172.21.2.112:0/3720455319 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f57681b0870 0x7f57680f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:29.590 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:29.585 7f5767fff700 1 -- 172.21.2.112:0/3720455319 >> v1:172.21.2.112:6789/0 conn(0x7f57681b0470 legacy=0x7f57680ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:29.590 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:29.585 7f5767fff700 1 -- 172.21.2.112:0/3720455319 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57680e52e0 con 0x7f57680efe00 2021-02-10T11:30:29.590 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:29.585 7f5767fff700 1 --2- 172.21.2.112:0/3720455319 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f57680efe00 0x7f57680f0200 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7f5760006e50 tx=0x7f5760009e70).ready entity=mon.1 client_cookie=b1cdfa124c84846c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:29.590 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:29.585 7f57667fc700 1 -- 172.21.2.112:0/3720455319 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f576000b510 con 0x7f57680efe00 2021-02-10T11:30:29.590 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:29.585 7f57667fc700 1 -- 172.21.2.112:0/3720455319 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5760006790 con 0x7f57680efe00 2021-02-10T11:30:29.590 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:29.585 7f57667fc700 1 -- 172.21.2.112:0/3720455319 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f576000b940 con 0x7f57680efe00 2021-02-10T11:30:29.590 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:29.585 7f576db0f700 1 -- 172.21.2.112:0/3720455319 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f57680efe00 msgr2=0x7f57680f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:29.590 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:29.585 7f576db0f700 1 --2- 172.21.2.112:0/3720455319 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f57680efe00 0x7f57680f0200 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7f5760006e50 tx=0x7f5760009e70).stop 2021-02-10T11:30:29.590 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:29.586 7f576db0f700 1 -- 172.21.2.112:0/3720455319 shutdown_connections 2021-02-10T11:30:29.590 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:29.586 7f576db0f700 1 --2- 172.21.2.112:0/3720455319 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f57680efe00 0x7f57680f0200 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:29.590 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:29.586 7f576db0f700 1 --2- 172.21.2.112:0/3720455319 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f57681b0870 0x7f57680f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:29.590 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:29.586 7f576db0f700 1 -- 172.21.2.112:0/3720455319 >> 172.21.2.112:0/3720455319 conn(0x7f57680ce3e0 msgr2=0x7f57680e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:29.591 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:29.586 7f576db0f700 1 -- 172.21.2.112:0/3720455319 shutdown_connections 2021-02-10T11:30:29.591 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:29.586 7f576db0f700 1 -- 172.21.2.112:0/3720455319 wait complete. 2021-02-10T11:30:29.591 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:29.601 7f576db0f700 1 Processor -- start 2021-02-10T11:30:29.591 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:29.601 7f576db0f700 1 -- start start 2021-02-10T11:30:29.592 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:29.601 7f576db0f700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f57680a0420 0x7f57680a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:29.592 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:29.601 7f576db0f700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5768189d00 0x7f576818a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:29.592 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:29.601 7f576db0f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f576818a720 0x7f57680b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:29.593 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:29.601 7f576db0f700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5768189800 con 0x7f576818a720 2021-02-10T11:30:29.593 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:29.601 7f576db0f700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f576818ab20 con 0x7f57680a0420 2021-02-10T11:30:29.593 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:29.601 7f576db0f700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57680a1140 con 0x7f5768189d00 2021-02-10T11:30:29.593 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:29.601 7f5767fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f576818a720 0x7f57680b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:29.593 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:29.601 7f5767fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f576818a720 0x7f57680b7be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:47036/0 (socket says 172.21.2.112:47036) 2021-02-10T11:30:29.593 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:29.601 7f57677fe700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f57680a0420 0x7f57680a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:29.593 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:29.601 7f5767fff700 1 -- 172.21.2.112:0/4252828415 learned_addr learned my addr 172.21.2.112:0/4252828415 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:30:29.593 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:29.602 7f5766ffd700 1 --2- 172.21.2.112:0/4252828415 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5768189d00 0x7f576818a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:29.593 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:29.602 7f57677fe700 1 -- 172.21.2.112:0/4252828415 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5768189d00 msgr2=0x7f576818a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:29.593 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:29.602 7f57677fe700 1 --2- 172.21.2.112:0/4252828415 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5768189d00 0x7f576818a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:29.593 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:29.602 7f57677fe700 1 -- 172.21.2.112:0/4252828415 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f576818a720 msgr2=0x7f57680b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:29.593 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:29.602 7f57677fe700 1 --2- 172.21.2.112:0/4252828415 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f576818a720 0x7f57680b7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:29.594 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:29.602 7f57677fe700 1 -- 172.21.2.112:0/4252828415 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57680b8200 con 0x7f57680a0420 2021-02-10T11:30:29.594 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:29.602 7f57677fe700 1 --2- 172.21.2.112:0/4252828415 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f57680a0420 0x7f57680a0820 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0x7f5750001c60 tx=0x7f575000a870).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:29.594 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:29.602 7f575ffff700 1 -- 172.21.2.112:0/4252828415 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5750000cc0 con 0x7f57680a0420 2021-02-10T11:30:29.594 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:29.602 7f575ffff700 1 -- 172.21.2.112:0/4252828415 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57500009b0 con 0x7f57680a0420 2021-02-10T11:30:29.594 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:29.602 7f576db0f700 1 -- 172.21.2.112:0/4252828415 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57680b8200 con 0x7f57680a0420 2021-02-10T11:30:29.594 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:29.602 7f575ffff700 1 -- 172.21.2.112:0/4252828415 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5750000cc0 con 0x7f57680a0420 2021-02-10T11:30:29.595 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:29.602 7f576db0f700 1 -- 172.21.2.112:0/4252828415 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57680b8200 con 0x7f57680a0420 2021-02-10T11:30:29.595 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:29.603 7f575ffff700 1 -- 172.21.2.112:0/4252828415 <== mon.1 v2:172.21.2.120:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5750004350 con 0x7f57680a0420 2021-02-10T11:30:29.595 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:29.603 7f575ffff700 1 --2- 172.21.2.112:0/4252828415 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f574c0186c0 0x7f574c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:29.595 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:29.603 7f575ffff700 1 -- 172.21.2.112:0/4252828415 <== mon.1 v2:172.21.2.120:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f5750032360 con 0x7f57680a0420 2021-02-10T11:30:29.595 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:29.603 7f5766ffd700 1 --2- 172.21.2.112:0/4252828415 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f574c0186c0 0x7f574c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:29.595 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:29.604 7f5766ffd700 1 --2- 172.21.2.112:0/4252828415 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f574c0186c0 0x7f574c01abc0 crc :-1 s=READY pgs=423 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:29.621 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:29.634 7f576db0f700 1 -- 172.21.2.112:0/4252828415 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f576818ab20 con 0x7f57680a0420 2021-02-10T11:30:29.624 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:29.636 7f575ffff700 1 -- 172.21.2.112:0/4252828415 <== mon.1 v2:172.21.2.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5750030c90 con 0x7f57680a0420 2021-02-10T11:30:29.751 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:29.763 7f576db0f700 1 -- 172.21.2.112:0/4252828415 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348602", "target": ["mgr", ""]}) v1 -- 0x7f576818ab20 con 0x7f574c0186c0 2021-02-10T11:30:29.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:29.774 7f575ffff700 1 -- 172.21.2.112:0/4252828415 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f5758006a70 con 0x7f574c0186c0 2021-02-10T11:30:29.768 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:29.781 7f576db0f700 1 -- 172.21.2.112:0/4252828415 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f574c0186c0 msgr2=0x7f574c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:29.769 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:29.781 7f576db0f700 1 --2- 172.21.2.112:0/4252828415 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f574c0186c0 0x7f574c01abc0 crc :-1 s=READY pgs=423 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:29.769 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:29.781 7f576db0f700 1 -- 172.21.2.112:0/4252828415 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f57680a0420 msgr2=0x7f57680a0820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:29.769 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:29.781 7f576db0f700 1 --2- 172.21.2.112:0/4252828415 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f57680a0420 0x7f57680a0820 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0x7f5750001c60 tx=0x7f575000a870).stop 2021-02-10T11:30:29.770 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:29.781 7f576db0f700 1 -- 172.21.2.112:0/4252828415 shutdown_connections 2021-02-10T11:30:29.770 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:29.781 7f576db0f700 1 --2- 172.21.2.112:0/4252828415 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f574c0186c0 0x7f574c01abc0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:29.770 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:29.781 7f576db0f700 1 --2- 172.21.2.112:0/4252828415 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f576818a720 0x7f57680b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:29.770 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:29.781 7f576db0f700 1 --2- 172.21.2.112:0/4252828415 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5768189d00 0x7f576818a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:29.770 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:29.781 7f576db0f700 1 --2- 172.21.2.112:0/4252828415 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f57680a0420 0x7f57680a0820 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:29.770 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:29.781 7f576db0f700 1 -- 172.21.2.112:0/4252828415 >> 172.21.2.112:0/4252828415 conn(0x7f5768094e10 msgr2=0x7f57680a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:29.770 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:29.782 7f576db0f700 1 -- 172.21.2.112:0/4252828415 shutdown_connections 2021-02-10T11:30:29.770 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:29.782 7f576db0f700 1 -- 172.21.2.112:0/4252828415 wait complete. 2021-02-10T11:30:29.778 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000348603 2021-02-10T11:30:30.207 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.203 7f34149c2700 1 Processor -- start 2021-02-10T11:30:30.208 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.203 7f34149c2700 1 -- start start 2021-02-10T11:30:30.208 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.204 7f34149c2700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f34101b0470 0x7f34101b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:30.208 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.204 7f34149c2700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f34100ef610 0x7f34100efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:30.208 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.204 7f34149c2700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34100f0b50 con 0x7f34100f0030 2021-02-10T11:30:30.208 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.204 7f34149c2700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34100e52e0 con 0x7f34101b0470 2021-02-10T11:30:30.208 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.204 7f34149c2700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34100e54d0 con 0x7f34100ef610 2021-02-10T11:30:30.208 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.204 7f340de1b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f34100ef610 0x7f34100efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:30.208 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.204 7f340ee1d700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f34100f0030 0x7f34100f10d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:46510/0 (socket says 172.21.2.112:46510) 2021-02-10T11:30:30.208 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.204 7f340ee1d700 1 -- 172.21.2.112:0/404179856 learned_addr learned my addr 172.21.2.112:0/404179856 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:30:30.208 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.204 7f340e61c700 1 --2- 172.21.2.112:0/404179856 >> v2:172.21.2.120:3300/0 conn(0x7f34101b0470 0x7f34101b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:30.208 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.205 7f340e61c700 1 -- 172.21.2.112:0/404179856 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f34100ef610 msgr2=0x7f34100efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:30.208 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.205 7f340e61c700 1 --2- 172.21.2.112:0/404179856 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f34100ef610 0x7f34100efa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:30.208 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.205 7f340e61c700 1 -- 172.21.2.112:0/404179856 >> v1:172.21.2.112:6789/0 conn(0x7f34100f0030 legacy=0x7f34100f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:30.209 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.205 7f340e61c700 1 -- 172.21.2.112:0/404179856 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34100f0430 con 0x7f34101b0470 2021-02-10T11:30:30.209 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.205 7f340e61c700 1 --2- 172.21.2.112:0/404179856 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f34101b0470 0x7f34101b0870 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7f3400006650 tx=0x7f3400009a80).ready entity=mon.1 client_cookie=c02d37e730b0b363 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:30.209 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.205 7f340d61a700 1 -- 172.21.2.112:0/404179856 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f340000c0c0 con 0x7f34101b0470 2021-02-10T11:30:30.209 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.205 7f340d61a700 1 -- 172.21.2.112:0/404179856 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f340000c3c0 con 0x7f34101b0470 2021-02-10T11:30:30.209 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.205 7f340d61a700 1 -- 172.21.2.112:0/404179856 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f340000c810 con 0x7f34101b0470 2021-02-10T11:30:30.209 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.205 7f34149c2700 1 -- 172.21.2.112:0/404179856 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f34101b0470 msgr2=0x7f34101b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:30.209 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.205 7f34149c2700 1 --2- 172.21.2.112:0/404179856 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f34101b0470 0x7f34101b0870 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7f3400006650 tx=0x7f3400009a80).stop 2021-02-10T11:30:30.209 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.206 7f34149c2700 1 -- 172.21.2.112:0/404179856 shutdown_connections 2021-02-10T11:30:30.209 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.206 7f34149c2700 1 --2- 172.21.2.112:0/404179856 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f34100ef610 0x7f34100efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:30.209 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.206 7f34149c2700 1 --2- 172.21.2.112:0/404179856 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f34101b0470 0x7f34101b0870 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:30.209 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.206 7f34149c2700 1 -- 172.21.2.112:0/404179856 >> 172.21.2.112:0/404179856 conn(0x7f34100ce3e0 msgr2=0x7f34100e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:30.209 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.206 7f34149c2700 1 -- 172.21.2.112:0/404179856 shutdown_connections 2021-02-10T11:30:30.209 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.206 7f34149c2700 1 -- 172.21.2.112:0/404179856 wait complete. 2021-02-10T11:30:30.211 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.220 7f34149c2700 1 Processor -- start 2021-02-10T11:30:30.211 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.220 7f34149c2700 1 -- start start 2021-02-10T11:30:30.211 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.221 7f34149c2700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3410190920 0x7f34100b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:30.211 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.221 7f34149c2700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f34100b8200 0x7f34100b8600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:30.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.221 7f34149c2700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f34100c1d80 0x7f34100c2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:30.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.221 7f34149c2700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3410190d20 con 0x7f3410190920 2021-02-10T11:30:30.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.221 7f34149c2700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34100b8c20 con 0x7f34100c1d80 2021-02-10T11:30:30.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.221 7f34149c2700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34100c27a0 con 0x7f34100b8200 2021-02-10T11:30:30.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.221 7f340e61c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3410190920 0x7f34100b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:30.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.221 7f340ee1d700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f34100c1d80 0x7f34100c2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:30.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.221 7f340e61c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3410190920 0x7f34100b7be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:47052/0 (socket says 172.21.2.112:47052) 2021-02-10T11:30:30.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.221 7f340e61c700 1 -- 172.21.2.112:0/3782940438 learned_addr learned my addr 172.21.2.112:0/3782940438 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:30:30.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.221 7f340ee1d700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f34100c1d80 0x7f34100c2180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3300/0 says I am v2:172.21.2.112:49330/0 (socket says 172.21.2.112:49330) 2021-02-10T11:30:30.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.221 7f340de1b700 1 --2- 172.21.2.112:0/3782940438 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f34100b8200 0x7f34100b8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:30.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.221 7f340e61c700 1 -- 172.21.2.112:0/3782940438 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f34100b8200 msgr2=0x7f34100b8600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:30.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.221 7f340e61c700 1 --2- 172.21.2.112:0/3782940438 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f34100b8200 0x7f34100b8600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:30.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.221 7f340e61c700 1 -- 172.21.2.112:0/3782940438 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f34100c1d80 msgr2=0x7f34100c2180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:30.213 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.221 7f340e61c700 1 --2- 172.21.2.112:0/3782940438 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f34100c1d80 0x7f34100c2180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:30.213 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.221 7f340e61c700 1 -- 172.21.2.112:0/3782940438 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34100c29c0 con 0x7f3410190920 2021-02-10T11:30:30.213 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.221 7f340de1b700 1 --2- 172.21.2.112:0/3782940438 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f34100b8200 0x7f34100b8600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:30:30.213 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.222 7f340e61c700 1 --2- 172.21.2.112:0/3782940438 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3410190920 0x7f34100b7be0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 rx=0x7f3400006650 tx=0x7f340000b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:30.213 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.222 7f3406ffd700 1 -- 172.21.2.112:0/3782940438 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3400002030 con 0x7f3410190920 2021-02-10T11:30:30.213 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.222 7f3406ffd700 1 -- 172.21.2.112:0/3782940438 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3400002370 con 0x7f3410190920 2021-02-10T11:30:30.213 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.222 7f3406ffd700 1 -- 172.21.2.112:0/3782940438 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f34000027d0 con 0x7f3410190920 2021-02-10T11:30:30.213 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.222 7f34149c2700 1 -- 172.21.2.112:0/3782940438 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34100c29c0 con 0x7f3410190920 2021-02-10T11:30:30.213 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.222 7f34149c2700 1 -- 172.21.2.112:0/3782940438 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34100c2f50 con 0x7f3410190920 2021-02-10T11:30:30.213 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.223 7f3406ffd700 1 -- 172.21.2.112:0/3782940438 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3400000c90 con 0x7f3410190920 2021-02-10T11:30:30.213 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.223 7f3406ffd700 1 --2- 172.21.2.112:0/3782940438 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f33f40186c0 0x7f33f401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:30.213 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.223 7f3406ffd700 1 -- 172.21.2.112:0/3782940438 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f34000325e0 con 0x7f3410190920 2021-02-10T11:30:30.214 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.223 7f340de1b700 1 --2- 172.21.2.112:0/3782940438 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f33f40186c0 0x7f33f401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:30.214 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.223 7f340de1b700 1 --2- 172.21.2.112:0/3782940438 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f33f40186c0 0x7f33f401abc0 crc :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:30.241 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.254 7f34149c2700 1 -- 172.21.2.112:0/3782940438 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3410105ae0 con 0x7f3410190920 2021-02-10T11:30:30.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.256 7f3406ffd700 1 -- 172.21.2.112:0/3782940438 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3400030d90 con 0x7f3410190920 2021-02-10T11:30:30.369 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.382 7f34149c2700 1 -- 172.21.2.112:0/3782940438 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348603", "target": ["mgr", ""]}) v1 -- 0x7f3410190d20 con 0x7f33f40186c0 2021-02-10T11:30:30.381 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.394 7f3406ffd700 1 -- 172.21.2.112:0/3782940438 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f33f80009b0 con 0x7f33f40186c0 2021-02-10T11:30:30.386 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.399 7f34149c2700 1 -- 172.21.2.112:0/3782940438 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f33f40186c0 msgr2=0x7f33f401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:30.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.399 7f34149c2700 1 --2- 172.21.2.112:0/3782940438 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f33f40186c0 0x7f33f401abc0 crc :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:30.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.399 7f34149c2700 1 -- 172.21.2.112:0/3782940438 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3410190920 msgr2=0x7f34100b7be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:30.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.399 7f34149c2700 1 --2- 172.21.2.112:0/3782940438 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3410190920 0x7f34100b7be0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 rx=0x7f3400006650 tx=0x7f340000b960).stop 2021-02-10T11:30:30.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.399 7f34149c2700 1 -- 172.21.2.112:0/3782940438 shutdown_connections 2021-02-10T11:30:30.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.399 7f34149c2700 1 --2- 172.21.2.112:0/3782940438 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f33f40186c0 0x7f33f401abc0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:30.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.399 7f34149c2700 1 --2- 172.21.2.112:0/3782940438 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f34100c1d80 0x7f34100c2180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:30.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.399 7f34149c2700 1 --2- 172.21.2.112:0/3782940438 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f34100b8200 0x7f34100b8600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:30.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.399 7f34149c2700 1 --2- 172.21.2.112:0/3782940438 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3410190920 0x7f34100b7be0 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:30.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.399 7f34149c2700 1 -- 172.21.2.112:0/3782940438 >> 172.21.2.112:0/3782940438 conn(0x7f3410094e10 msgr2=0x7f34100a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:30.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.399 7f34149c2700 1 -- 172.21.2.112:0/3782940438 shutdown_connections 2021-02-10T11:30:30.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.399 7f34149c2700 1 -- 172.21.2.112:0/3782940438 wait complete. 2021-02-10T11:30:30.395 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000348604 2021-02-10T11:30:30.825 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.820 7fadb840b700 1 Processor -- start 2021-02-10T11:30:30.825 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.820 7fadb840b700 1 -- start start 2021-02-10T11:30:30.825 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.821 7fadb840b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fadb01b0470 0x7fadb01b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:30.825 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.821 7fadb840b700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fadb00ef610 0x7fadb00efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:30.825 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.821 7fadb840b700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadb00f0b50 con 0x7fadb00f0030 2021-02-10T11:30:30.826 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.821 7fadb840b700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fadb00e52e0 con 0x7fadb00ef610 2021-02-10T11:30:30.826 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.821 7fadb840b700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadb00e5490 con 0x7fadb01b0470 2021-02-10T11:30:30.826 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.821 7fadb5a26700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fadb00ef610 0x7fadb00efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:30.826 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.821 7fadb6a28700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fadb00f0030 0x7fadb00f10d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:46520/0 (socket says 172.21.2.112:46520) 2021-02-10T11:30:30.826 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.821 7fadb6227700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fadb01b0470 0x7fadb01b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:30.826 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.821 7fadb6a28700 1 -- 172.21.2.112:0/2461764225 learned_addr learned my addr 172.21.2.112:0/2461764225 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:30:30.826 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.822 7fadb6227700 1 -- 172.21.2.112:0/2461764225 >> v2:172.21.2.112:3300/0 conn(0x7fadb00ef610 msgr2=0x7fadb00efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:30.826 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.822 7fadb6227700 1 --2- 172.21.2.112:0/2461764225 >> v2:172.21.2.112:3300/0 conn(0x7fadb00ef610 0x7fadb00efa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:30.826 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.822 7fadb6227700 1 -- 172.21.2.112:0/2461764225 >> v1:172.21.2.112:6789/0 conn(0x7fadb00f0030 legacy=0x7fadb00f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:30.826 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.822 7fadb6227700 1 -- 172.21.2.112:0/2461764225 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadb00f0430 con 0x7fadb01b0470 2021-02-10T11:30:30.826 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.822 7fadb6227700 1 --2- 172.21.2.112:0/2461764225 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fadb01b0470 0x7fadb01b0870 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7fadac006650 tx=0x7fadac009a80).ready entity=mon.2 client_cookie=88bb6e7e6fbb03dd server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:30.826 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.822 7fadb5225700 1 -- 172.21.2.112:0/2461764225 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fadac00c0c0 con 0x7fadb01b0470 2021-02-10T11:30:30.827 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.822 7fadb5225700 1 -- 172.21.2.112:0/2461764225 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadac00c340 con 0x7fadb01b0470 2021-02-10T11:30:30.827 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.822 7fadb5225700 1 -- 172.21.2.112:0/2461764225 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fadac00c860 con 0x7fadb01b0470 2021-02-10T11:30:30.827 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.822 7fadb840b700 1 -- 172.21.2.112:0/2461764225 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fadb01b0470 msgr2=0x7fadb01b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:30.827 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.822 7fadb840b700 1 --2- 172.21.2.112:0/2461764225 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fadb01b0470 0x7fadb01b0870 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7fadac006650 tx=0x7fadac009a80).stop 2021-02-10T11:30:30.827 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.823 7fadb840b700 1 -- 172.21.2.112:0/2461764225 shutdown_connections 2021-02-10T11:30:30.827 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.823 7fadb840b700 1 --2- 172.21.2.112:0/2461764225 >> v2:172.21.2.112:3300/0 conn(0x7fadb00ef610 0x7fadb00efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:30.827 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.823 7fadb840b700 1 --2- 172.21.2.112:0/2461764225 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fadb01b0470 0x7fadb01b0870 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:30.827 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.823 7fadb840b700 1 -- 172.21.2.112:0/2461764225 >> 172.21.2.112:0/2461764225 conn(0x7fadb00ce3e0 msgr2=0x7fadb00e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:30.827 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.823 7fadb840b700 1 -- 172.21.2.112:0/2461764225 shutdown_connections 2021-02-10T11:30:30.827 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.823 7fadb840b700 1 -- 172.21.2.112:0/2461764225 wait complete. 2021-02-10T11:30:30.827 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.838 7fadb840b700 1 Processor -- start 2021-02-10T11:30:30.827 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.838 7fadb840b700 1 -- start start 2021-02-10T11:30:30.829 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.838 7fadb840b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fadb0190920 0x7fadb00b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:30.829 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.838 7fadb840b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fadb00b8200 0x7fadb00b8600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:30.829 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.838 7fadb840b700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fadb00c1d80 0x7fadb00c2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:30.829 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.838 7fadb840b700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadb0190d20 con 0x7fadb0190920 2021-02-10T11:30:30.829 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.838 7fadb840b700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadb00b8c20 con 0x7fadb00c1d80 2021-02-10T11:30:30.829 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.838 7fadb840b700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadb00c27a0 con 0x7fadb00b8200 2021-02-10T11:30:30.829 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.838 7fadb6227700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fadb0190920 0x7fadb00b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:30.829 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.838 7fadb6227700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fadb0190920 0x7fadb00b7be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:47064/0 (socket says 172.21.2.112:47064) 2021-02-10T11:30:30.829 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.838 7fadb6227700 1 -- 172.21.2.112:0/188275758 learned_addr learned my addr 172.21.2.112:0/188275758 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:30:30.829 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.838 7fadb6a28700 1 --2- 172.21.2.112:0/188275758 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fadb00c1d80 0x7fadb00c2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:30.830 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.838 7fadb5a26700 1 --2- 172.21.2.112:0/188275758 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fadb00b8200 0x7fadb00b8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:30.830 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.839 7fadb5a26700 1 -- 172.21.2.112:0/188275758 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fadb00c1d80 msgr2=0x7fadb00c2180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:30.830 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.839 7fadb5a26700 1 --2- 172.21.2.112:0/188275758 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fadb00c1d80 0x7fadb00c2180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:30.830 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.839 7fadb5a26700 1 -- 172.21.2.112:0/188275758 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fadb0190920 msgr2=0x7fadb00b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:30.830 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.839 7fadb5a26700 1 --2- 172.21.2.112:0/188275758 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fadb0190920 0x7fadb00b7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:30.830 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.839 7fadb5a26700 1 -- 172.21.2.112:0/188275758 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadb00c29c0 con 0x7fadb00b8200 2021-02-10T11:30:30.830 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.839 7fadb6227700 1 --2- 172.21.2.112:0/188275758 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fadb0190920 0x7fadb00b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:30:30.830 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.839 7fadb5a26700 1 --2- 172.21.2.112:0/188275758 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fadb00b8200 0x7fadb00b8600 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7fada0009540 tx=0x7fada000b8e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:30.830 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.839 7fada6ffd700 1 -- 172.21.2.112:0/188275758 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fada00009b0 con 0x7fadb00b8200 2021-02-10T11:30:30.830 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.839 7fada6ffd700 1 -- 172.21.2.112:0/188275758 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fada0006340 con 0x7fadb00b8200 2021-02-10T11:30:30.830 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.839 7fada6ffd700 1 -- 172.21.2.112:0/188275758 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fada0002030 con 0x7fadb00b8200 2021-02-10T11:30:30.830 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.839 7fadb840b700 1 -- 172.21.2.112:0/188275758 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadb00c29c0 con 0x7fadb00b8200 2021-02-10T11:30:30.831 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.839 7fadb840b700 1 -- 172.21.2.112:0/188275758 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadb00c2f50 con 0x7fadb00b8200 2021-02-10T11:30:30.831 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.840 7fada6ffd700 1 -- 172.21.2.112:0/188275758 <== mon.2 v2:172.21.2.120:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fada0000c90 con 0x7fadb00b8200 2021-02-10T11:30:30.831 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.840 7fada6ffd700 1 --2- 172.21.2.112:0/188275758 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fad980186c0 0x7fad9801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:30.831 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.840 7fada6ffd700 1 -- 172.21.2.112:0/188275758 <== mon.2 v2:172.21.2.120:3301/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fada00324c0 con 0x7fadb00b8200 2021-02-10T11:30:30.831 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.840 7fadb6227700 1 --2- 172.21.2.112:0/188275758 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fad980186c0 0x7fad9801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:30.831 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.840 7fadb6227700 1 --2- 172.21.2.112:0/188275758 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fad980186c0 0x7fad9801abc0 crc :-1 s=READY pgs=425 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:30.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.871 7fadb840b700 1 -- 172.21.2.112:0/188275758 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadb0105ae0 con 0x7fadb00b8200 2021-02-10T11:30:30.861 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.873 7fada6ffd700 1 -- 172.21.2.112:0/188275758 <== mon.2 v2:172.21.2.120:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fada001eda0 con 0x7fadb00b8200 2021-02-10T11:30:30.987 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:30.999 7fadb840b700 1 -- 172.21.2.112:0/188275758 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348604", "target": ["mgr", ""]}) v1 -- 0x7fadb00c2f50 con 0x7fad980186c0 2021-02-10T11:30:30.997 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:31.010 7fada6ffd700 1 -- 172.21.2.112:0/188275758 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fadac000960 con 0x7fad980186c0 2021-02-10T11:30:31.005 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:31.017 7fadb840b700 1 -- 172.21.2.112:0/188275758 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fad980186c0 msgr2=0x7fad9801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:31.005 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:31.017 7fadb840b700 1 --2- 172.21.2.112:0/188275758 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fad980186c0 0x7fad9801abc0 crc :-1 s=READY pgs=425 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:31.005 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:31.017 7fadb840b700 1 -- 172.21.2.112:0/188275758 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fadb00b8200 msgr2=0x7fadb00b8600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:31.005 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:31.017 7fadb840b700 1 --2- 172.21.2.112:0/188275758 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fadb00b8200 0x7fadb00b8600 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7fada0009540 tx=0x7fada000b8e0).stop 2021-02-10T11:30:31.005 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:31.017 7fadb840b700 1 -- 172.21.2.112:0/188275758 shutdown_connections 2021-02-10T11:30:31.005 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:31.017 7fadb840b700 1 --2- 172.21.2.112:0/188275758 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fad980186c0 0x7fad9801abc0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:31.005 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:31.017 7fadb840b700 1 --2- 172.21.2.112:0/188275758 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fadb00c1d80 0x7fadb00c2180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:31.005 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:31.017 7fadb840b700 1 --2- 172.21.2.112:0/188275758 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fadb00b8200 0x7fadb00b8600 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:31.006 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:31.017 7fadb840b700 1 --2- 172.21.2.112:0/188275758 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fadb0190920 0x7fadb00b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:31.006 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:31.017 7fadb840b700 1 -- 172.21.2.112:0/188275758 >> 172.21.2.112:0/188275758 conn(0x7fadb0094e10 msgr2=0x7fadb00a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:31.006 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:31.017 7fadb840b700 1 -- 172.21.2.112:0/188275758 shutdown_connections 2021-02-10T11:30:31.006 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:31.017 7fadb840b700 1 -- 172.21.2.112:0/188275758 wait complete. 2021-02-10T11:30:31.013 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000348605 2021-02-10T11:30:31.447 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:31.438 7fad6c20d700 1 Processor -- start 2021-02-10T11:30:31.447 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:31.438 7fad6c20d700 1 -- start start 2021-02-10T11:30:31.447 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:31.439 7fad6c20d700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fad641b0470 0x7fad641b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:31.448 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:31.439 7fad6c20d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fad640ef610 0x7fad640efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:31.448 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:31.439 7fad6c20d700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad640f0b50 con 0x7fad640f0030 2021-02-10T11:30:31.448 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:31.439 7fad6c20d700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad640e52e0 con 0x7fad641b0470 2021-02-10T11:30:31.448 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:31.439 7fad6c20d700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad640e54d0 con 0x7fad640ef610 2021-02-10T11:30:31.448 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:31.439 7fad6a82a700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7fad640f0030 0x7fad640f10d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:40094/0 (socket says 172.21.2.112:40094) 2021-02-10T11:30:31.448 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:31.439 7fad6a82a700 1 -- 172.21.2.112:0/3693090280 learned_addr learned my addr 172.21.2.112:0/3693090280 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:30:31.449 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:31.439 7fad69828700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fad640ef610 0x7fad640efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:31.449 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:31.439 7fad6a029700 1 --2- 172.21.2.112:0/3693090280 >> v2:172.21.2.120:3300/0 conn(0x7fad641b0470 0x7fad641b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:31.449 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:31.439 7fad69027700 1 -- 172.21.2.112:0/3693090280 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3023004684 0 0) 0x7fad5c001510 con 0x7fad640f0030 2021-02-10T11:30:31.449 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:31.439 7fad69027700 1 -- 172.21.2.112:0/3693090280 --> v1:172.21.2.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad50002fe0 con 0x7fad640f0030 2021-02-10T11:30:31.449 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:31.440 7fad69828700 1 -- 172.21.2.112:0/3693090280 >> v2:172.21.2.120:3300/0 conn(0x7fad641b0470 msgr2=0x7fad641b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:31.450 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:31.440 7fad69828700 1 --2- 172.21.2.112:0/3693090280 >> v2:172.21.2.120:3300/0 conn(0x7fad641b0470 0x7fad641b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:31.450 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:31.440 7fad69828700 1 -- 172.21.2.112:0/3693090280 >> v1:172.21.2.120:6789/0 conn(0x7fad640f0030 legacy=0x7fad640f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:31.450 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:31.440 7fad69828700 1 -- 172.21.2.112:0/3693090280 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad640f0430 con 0x7fad640ef610 2021-02-10T11:30:31.450 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:31.440 7fad6a029700 1 --2- 172.21.2.112:0/3693090280 >> v2:172.21.2.120:3300/0 conn(0x7fad641b0470 0x7fad641b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:30:31.450 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:31.440 7fad69828700 1 --2- 172.21.2.112:0/3693090280 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fad640ef610 0x7fad640efa10 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7fad58006650 tx=0x7fad58009a80).ready entity=mon.2 client_cookie=9bd8140e1ebba318 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:31.451 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:31.440 7fad69027700 1 -- 172.21.2.112:0/3693090280 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fad580009b0 con 0x7fad640ef610 2021-02-10T11:30:31.451 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:31.440 7fad69027700 1 -- 172.21.2.112:0/3693090280 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad58006740 con 0x7fad640ef610 2021-02-10T11:30:31.451 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:31.440 7fad69027700 1 -- 172.21.2.112:0/3693090280 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fad5800c0c0 con 0x7fad640ef610 2021-02-10T11:30:31.451 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:31.440 7fad6c20d700 1 -- 172.21.2.112:0/3693090280 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fad640ef610 msgr2=0x7fad640efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:31.451 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:31.440 7fad6c20d700 1 --2- 172.21.2.112:0/3693090280 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fad640ef610 0x7fad640efa10 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7fad58006650 tx=0x7fad58009a80).stop 2021-02-10T11:30:31.451 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:31.441 7fad6c20d700 1 -- 172.21.2.112:0/3693090280 shutdown_connections 2021-02-10T11:30:31.452 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:31.441 7fad6c20d700 1 --2- 172.21.2.112:0/3693090280 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fad640ef610 0x7fad640efa10 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:31.452 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:31.441 7fad6c20d700 1 --2- 172.21.2.112:0/3693090280 >> v2:172.21.2.120:3300/0 conn(0x7fad641b0470 0x7fad641b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:31.452 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:31.441 7fad6c20d700 1 -- 172.21.2.112:0/3693090280 >> 172.21.2.112:0/3693090280 conn(0x7fad640ce3e0 msgr2=0x7fad640e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:31.452 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:31.441 7fad6c20d700 1 -- 172.21.2.112:0/3693090280 shutdown_connections 2021-02-10T11:30:31.452 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:31.441 7fad6c20d700 1 -- 172.21.2.112:0/3693090280 wait complete. 2021-02-10T11:30:31.452 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:31.456 7fad6c20d700 1 Processor -- start 2021-02-10T11:30:31.453 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:31.456 7fad6c20d700 1 -- start start 2021-02-10T11:30:31.453 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:31.456 7fad6c20d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fad64190920 0x7fad640b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:31.453 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:31.456 7fad6c20d700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fad640b8200 0x7fad640b8600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:31.453 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:31.456 7fad6c20d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fad640c1d80 0x7fad640c2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:31.453 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:31.456 7fad6c20d700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad64190da0 con 0x7fad640c1d80 2021-02-10T11:30:31.453 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:31.456 7fad6c20d700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad640b8c20 con 0x7fad640b8200 2021-02-10T11:30:31.454 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:31.456 7fad6c20d700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad640b8e10 con 0x7fad64190920 2021-02-10T11:30:31.454 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:31.457 7fad6a82a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fad640c1d80 0x7fad640c2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:31.454 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:31.457 7fad6a82a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fad640c1d80 0x7fad640c2180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:47076/0 (socket says 172.21.2.112:47076) 2021-02-10T11:30:31.454 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:31.457 7fad6a82a700 1 -- 172.21.2.112:0/2424283324 learned_addr learned my addr 172.21.2.112:0/2424283324 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:30:31.454 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:31.457 7fad69828700 1 --2- 172.21.2.112:0/2424283324 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fad640b8200 0x7fad640b8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:31.454 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:31.457 7fad6a029700 1 --2- 172.21.2.112:0/2424283324 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fad64190920 0x7fad640b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:31.455 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:31.457 7fad6a82a700 1 -- 172.21.2.112:0/2424283324 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fad64190920 msgr2=0x7fad640b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:31.455 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:31.457 7fad6a82a700 1 --2- 172.21.2.112:0/2424283324 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fad64190920 0x7fad640b7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:31.455 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:31.457 7fad6a82a700 1 -- 172.21.2.112:0/2424283324 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fad640b8200 msgr2=0x7fad640b8600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:31.455 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:31.457 7fad6a82a700 1 --2- 172.21.2.112:0/2424283324 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fad640b8200 0x7fad640b8600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:31.455 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:31.457 7fad6a82a700 1 -- 172.21.2.112:0/2424283324 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad640c27d0 con 0x7fad640c1d80 2021-02-10T11:30:31.455 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:31.457 7fad69828700 1 --2- 172.21.2.112:0/2424283324 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fad640b8200 0x7fad640b8600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:30:31.456 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:31.457 7fad6a029700 1 --2- 172.21.2.112:0/2424283324 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fad64190920 0x7fad640b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:30:31.456 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:31.457 7fad6a82a700 1 --2- 172.21.2.112:0/2424283324 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fad640c1d80 0x7fad640c2180 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 rx=0x7fad5c001ba0 tx=0x7fad5c00ac80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:31.456 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:31.458 7fad56ffd700 1 -- 172.21.2.112:0/2424283324 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fad5c0070d0 con 0x7fad640c1d80 2021-02-10T11:30:31.456 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:31.458 7fad56ffd700 1 -- 172.21.2.112:0/2424283324 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad5c000f40 con 0x7fad640c1d80 2021-02-10T11:30:31.456 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:31.458 7fad56ffd700 1 -- 172.21.2.112:0/2424283324 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fad5c0048b0 con 0x7fad640c1d80 2021-02-10T11:30:31.456 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:31.458 7fad6c20d700 1 -- 172.21.2.112:0/2424283324 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad640c27d0 con 0x7fad640c1d80 2021-02-10T11:30:31.457 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:31.458 7fad6c20d700 1 -- 172.21.2.112:0/2424283324 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad640c27d0 con 0x7fad640c1d80 2021-02-10T11:30:31.457 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:31.458 7fad56ffd700 1 -- 172.21.2.112:0/2424283324 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fad5c007240 con 0x7fad640c1d80 2021-02-10T11:30:31.457 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:31.458 7fad56ffd700 1 --2- 172.21.2.112:0/2424283324 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fad4c0186c0 0x7fad4c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:31.457 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:31.458 7fad56ffd700 1 -- 172.21.2.112:0/2424283324 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fad5c032b40 con 0x7fad640c1d80 2021-02-10T11:30:31.457 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:31.459 7fad6a029700 1 --2- 172.21.2.112:0/2424283324 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fad4c0186c0 0x7fad4c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:31.458 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:31.459 7fad6a029700 1 --2- 172.21.2.112:0/2424283324 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fad4c0186c0 0x7fad4c01abc0 crc :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:31.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:31.489 7fad6c20d700 1 -- 172.21.2.112:0/2424283324 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad64105ae0 con 0x7fad640c1d80 2021-02-10T11:30:31.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:31.491 7fad56ffd700 1 -- 172.21.2.112:0/2424283324 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fad5c000f20 con 0x7fad640c1d80 2021-02-10T11:30:31.606 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:31.618 7fad6c20d700 1 -- 172.21.2.112:0/2424283324 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348605", "target": ["mgr", ""]}) v1 -- 0x7fad640f0b50 con 0x7fad4c0186c0 2021-02-10T11:30:31.616 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:31.630 7fad56ffd700 1 -- 172.21.2.112:0/2424283324 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fad600009b0 con 0x7fad4c0186c0 2021-02-10T11:30:31.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:31.635 7fad6c20d700 1 -- 172.21.2.112:0/2424283324 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fad4c0186c0 msgr2=0x7fad4c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:31.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:31.635 7fad6c20d700 1 --2- 172.21.2.112:0/2424283324 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fad4c0186c0 0x7fad4c01abc0 crc :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:31.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:31.635 7fad6c20d700 1 -- 172.21.2.112:0/2424283324 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fad640c1d80 msgr2=0x7fad640c2180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:31.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:31.635 7fad6c20d700 1 --2- 172.21.2.112:0/2424283324 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fad640c1d80 0x7fad640c2180 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 rx=0x7fad5c001ba0 tx=0x7fad5c00ac80).stop 2021-02-10T11:30:31.624 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:31.636 7fad6c20d700 1 -- 172.21.2.112:0/2424283324 shutdown_connections 2021-02-10T11:30:31.624 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:31.636 7fad6c20d700 1 --2- 172.21.2.112:0/2424283324 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fad4c0186c0 0x7fad4c01abc0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:31.624 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:31.636 7fad6c20d700 1 --2- 172.21.2.112:0/2424283324 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fad640b8200 0x7fad640b8600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:31.624 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:31.636 7fad6c20d700 1 --2- 172.21.2.112:0/2424283324 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fad640c1d80 0x7fad640c2180 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:31.624 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:31.636 7fad6c20d700 1 --2- 172.21.2.112:0/2424283324 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fad64190920 0x7fad640b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:31.624 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:31.636 7fad6c20d700 1 -- 172.21.2.112:0/2424283324 >> 172.21.2.112:0/2424283324 conn(0x7fad64094e10 msgr2=0x7fad640a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:31.624 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:31.636 7fad6c20d700 1 -- 172.21.2.112:0/2424283324 shutdown_connections 2021-02-10T11:30:31.624 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:31.636 7fad6c20d700 1 -- 172.21.2.112:0/2424283324 wait complete. 2021-02-10T11:30:31.632 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000348606 2021-02-10T11:30:32.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.061 7f5735f3a700 1 Processor -- start 2021-02-10T11:30:32.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.061 7f5735f3a700 1 -- start start 2021-02-10T11:30:32.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.062 7f5735f3a700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f57301b0470 0x7f57301b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:32.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.062 7f5735f3a700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f57300ef610 0x7f57300efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:32.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.062 7f5735f3a700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57300f0b50 con 0x7f57300f0030 2021-02-10T11:30:32.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.062 7f5735f3a700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57300e52e0 con 0x7f57301b0470 2021-02-10T11:30:32.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.062 7f5735f3a700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57300e5490 con 0x7f57300ef610 2021-02-10T11:30:32.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.062 7f572f7fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f57301b0470 0x7f57301b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:32.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.062 7f572effd700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f57300ef610 0x7f57300efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:32.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.062 7f572effd700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f57300ef610 0x7f57300efa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3300/0 says I am v2:172.21.2.112:49364/0 (socket says 172.21.2.112:49364) 2021-02-10T11:30:32.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.062 7f572f7fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f57301b0470 0x7f57301b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:47084/0 (socket says 172.21.2.112:47084) 2021-02-10T11:30:32.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.062 7f572effd700 1 -- 172.21.2.112:0/279088374 learned_addr learned my addr 172.21.2.112:0/279088374 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:30:32.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.063 7f572effd700 1 -- 172.21.2.112:0/279088374 >> v2:172.21.2.112:3300/0 conn(0x7f57301b0470 msgr2=0x7f57301b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:32.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.063 7f572effd700 1 --2- 172.21.2.112:0/279088374 >> v2:172.21.2.112:3300/0 conn(0x7f57301b0470 0x7f57301b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:32.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.063 7f572effd700 1 -- 172.21.2.112:0/279088374 >> v1:172.21.2.120:6789/0 conn(0x7f57300f0030 legacy=0x7f57300f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:32.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.063 7f572effd700 1 -- 172.21.2.112:0/279088374 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57300f0430 con 0x7f57300ef610 2021-02-10T11:30:32.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.063 7f572effd700 1 --2- 172.21.2.112:0/279088374 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f57300ef610 0x7f57300efa10 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 rx=0x7f571c006700 tx=0x7f571c009b00).ready entity=mon.1 client_cookie=cd6df6db271ecf83 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:32.068 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.063 7f572e7fc700 1 -- 172.21.2.112:0/279088374 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f571c00c0c0 con 0x7f57300ef610 2021-02-10T11:30:32.068 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.063 7f572e7fc700 1 -- 172.21.2.112:0/279088374 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f571c00c400 con 0x7f57300ef610 2021-02-10T11:30:32.068 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.063 7f572e7fc700 1 -- 172.21.2.112:0/279088374 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f571c00c920 con 0x7f57300ef610 2021-02-10T11:30:32.068 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.063 7f5735f3a700 1 -- 172.21.2.112:0/279088374 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f57300ef610 msgr2=0x7f57300efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:32.068 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.063 7f5735f3a700 1 --2- 172.21.2.112:0/279088374 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f57300ef610 0x7f57300efa10 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 rx=0x7f571c006700 tx=0x7f571c009b00).stop 2021-02-10T11:30:32.068 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.064 7f5735f3a700 1 -- 172.21.2.112:0/279088374 shutdown_connections 2021-02-10T11:30:32.068 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.064 7f5735f3a700 1 --2- 172.21.2.112:0/279088374 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f57300ef610 0x7f57300efa10 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:32.068 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.064 7f5735f3a700 1 --2- 172.21.2.112:0/279088374 >> v2:172.21.2.112:3300/0 conn(0x7f57301b0470 0x7f57301b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:32.068 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.064 7f5735f3a700 1 -- 172.21.2.112:0/279088374 >> 172.21.2.112:0/279088374 conn(0x7f57300ce3e0 msgr2=0x7f57300e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:32.068 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.064 7f5735f3a700 1 -- 172.21.2.112:0/279088374 shutdown_connections 2021-02-10T11:30:32.068 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.064 7f5735f3a700 1 -- 172.21.2.112:0/279088374 wait complete. 2021-02-10T11:30:32.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.079 7f5735f3a700 1 Processor -- start 2021-02-10T11:30:32.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.079 7f5735f3a700 1 -- start start 2021-02-10T11:30:32.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.080 7f5735f3a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f57300a0420 0x7f57300a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:32.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.080 7f5735f3a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5730189d00 0x7f573018a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:32.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.080 7f5735f3a700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f573018a720 0x7f57300b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:32.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.080 7f5735f3a700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5730189800 con 0x7f5730189d00 2021-02-10T11:30:32.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.080 7f5735f3a700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f573018ab20 con 0x7f573018a720 2021-02-10T11:30:32.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.080 7f5735f3a700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57300a1140 con 0x7f57300a0420 2021-02-10T11:30:32.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.080 7f572effd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5730189d00 0x7f573018a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:32.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.080 7f572effd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5730189d00 0x7f573018a100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:47090/0 (socket says 172.21.2.112:47090) 2021-02-10T11:30:32.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.080 7f572effd700 1 -- 172.21.2.112:0/2054519164 learned_addr learned my addr 172.21.2.112:0/2054519164 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:30:32.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.080 7f572ffff700 1 --2- 172.21.2.112:0/2054519164 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f573018a720 0x7f57300b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:32.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.080 7f572f7fe700 1 --2- 172.21.2.112:0/2054519164 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f57300a0420 0x7f57300a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:32.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.080 7f572effd700 1 -- 172.21.2.112:0/2054519164 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f57300a0420 msgr2=0x7f57300a0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:32.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.080 7f572effd700 1 --2- 172.21.2.112:0/2054519164 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f57300a0420 0x7f57300a0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:32.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.080 7f572effd700 1 -- 172.21.2.112:0/2054519164 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f573018a720 msgr2=0x7f57300b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:32.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.080 7f572effd700 1 --2- 172.21.2.112:0/2054519164 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f573018a720 0x7f57300b7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:32.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.080 7f572effd700 1 -- 172.21.2.112:0/2054519164 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57300b8200 con 0x7f5730189d00 2021-02-10T11:30:32.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.080 7f572effd700 1 --2- 172.21.2.112:0/2054519164 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5730189d00 0x7f573018a100 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 rx=0x7f571c00b660 tx=0x7f571c00ba00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:32.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.080 7f5727fff700 1 -- 172.21.2.112:0/2054519164 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f571c009440 con 0x7f5730189d00 2021-02-10T11:30:32.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.080 7f5727fff700 1 -- 172.21.2.112:0/2054519164 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f571c000c90 con 0x7f5730189d00 2021-02-10T11:30:32.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.080 7f5735f3a700 1 -- 172.21.2.112:0/2054519164 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57300b8200 con 0x7f5730189d00 2021-02-10T11:30:32.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.081 7f5727fff700 1 -- 172.21.2.112:0/2054519164 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f571c002030 con 0x7f5730189d00 2021-02-10T11:30:32.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.081 7f5735f3a700 1 -- 172.21.2.112:0/2054519164 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57300b8790 con 0x7f5730189d00 2021-02-10T11:30:32.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.081 7f5727fff700 1 -- 172.21.2.112:0/2054519164 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f571c0093e0 con 0x7f5730189d00 2021-02-10T11:30:32.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.081 7f5727fff700 1 --2- 172.21.2.112:0/2054519164 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f570c0186c0 0x7f570c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:32.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.081 7f5727fff700 1 -- 172.21.2.112:0/2054519164 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f571c0323f0 con 0x7f5730189d00 2021-02-10T11:30:32.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.081 7f572f7fe700 1 --2- 172.21.2.112:0/2054519164 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f570c0186c0 0x7f570c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:32.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.082 7f572f7fe700 1 --2- 172.21.2.112:0/2054519164 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f570c0186c0 0x7f570c01abc0 crc :-1 s=READY pgs=427 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:32.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.111 7f5735f3a700 1 -- 172.21.2.112:0/2054519164 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5730105ae0 con 0x7f5730189d00 2021-02-10T11:30:32.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.114 7f5727fff700 1 -- 172.21.2.112:0/2054519164 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f571c030d20 con 0x7f5730189d00 2021-02-10T11:30:32.231 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.243 7f5735f3a700 1 -- 172.21.2.112:0/2054519164 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348606", "target": ["mgr", ""]}) v1 -- 0x7f57300b9020 con 0x7f570c0186c0 2021-02-10T11:30:32.242 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.255 7f5727fff700 1 -- 172.21.2.112:0/2054519164 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f5728006ce0 con 0x7f570c0186c0 2021-02-10T11:30:32.248 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.261 7f5735f3a700 1 -- 172.21.2.112:0/2054519164 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f570c0186c0 msgr2=0x7f570c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:32.248 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.261 7f5735f3a700 1 --2- 172.21.2.112:0/2054519164 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f570c0186c0 0x7f570c01abc0 crc :-1 s=READY pgs=427 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:32.248 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.261 7f5735f3a700 1 -- 172.21.2.112:0/2054519164 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5730189d00 msgr2=0x7f573018a100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:32.248 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.261 7f5735f3a700 1 --2- 172.21.2.112:0/2054519164 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5730189d00 0x7f573018a100 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 rx=0x7f571c00b660 tx=0x7f571c00ba00).stop 2021-02-10T11:30:32.248 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.261 7f5735f3a700 1 -- 172.21.2.112:0/2054519164 shutdown_connections 2021-02-10T11:30:32.249 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.261 7f5735f3a700 1 --2- 172.21.2.112:0/2054519164 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f570c0186c0 0x7f570c01abc0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:32.249 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.261 7f5735f3a700 1 --2- 172.21.2.112:0/2054519164 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f573018a720 0x7f57300b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:32.249 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.261 7f5735f3a700 1 --2- 172.21.2.112:0/2054519164 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5730189d00 0x7f573018a100 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:32.249 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.261 7f5735f3a700 1 --2- 172.21.2.112:0/2054519164 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f57300a0420 0x7f57300a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:32.249 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.261 7f5735f3a700 1 -- 172.21.2.112:0/2054519164 >> 172.21.2.112:0/2054519164 conn(0x7f5730094e10 msgr2=0x7f57300a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:32.249 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.261 7f5735f3a700 1 -- 172.21.2.112:0/2054519164 shutdown_connections 2021-02-10T11:30:32.249 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.261 7f5735f3a700 1 -- 172.21.2.112:0/2054519164 wait complete. 2021-02-10T11:30:32.257 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000348607 2021-02-10T11:30:32.690 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.684 7f4e736d9700 1 Processor -- start 2021-02-10T11:30:32.690 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.684 7f4e736d9700 1 -- start start 2021-02-10T11:30:32.690 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.684 7f4e736d9700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4e6c1b0870 0x7f4e6c0f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:32.690 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.684 7f4e736d9700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f4e6c0efe00 0x7f4e6c0f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:32.690 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.684 7f4e736d9700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e6c1b0c70 con 0x7f4e6c1b0470 2021-02-10T11:30:32.690 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.684 7f4e736d9700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e6c0f0b50 con 0x7f4e6c0efe00 2021-02-10T11:30:32.690 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.684 7f4e736d9700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e6c0f0820 con 0x7f4e6c1b0870 2021-02-10T11:30:32.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.685 7f4e714f5700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f4e6c1b0470 0x7f4e6c0ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:46558/0 (socket says 172.21.2.112:46558) 2021-02-10T11:30:32.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.685 7f4e714f5700 1 -- 172.21.2.112:0/4278817628 learned_addr learned my addr 172.21.2.112:0/4278817628 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:30:32.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.685 7f4e71cf6700 1 --2- 172.21.2.112:0/4278817628 >> v2:172.21.2.120:3300/0 conn(0x7f4e6c0efe00 0x7f4e6c0f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:32.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.685 7f4e70cf4700 1 --2- 172.21.2.112:0/4278817628 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4e6c1b0870 0x7f4e6c0f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:32.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.685 7f4e63fff700 1 -- 172.21.2.112:0/4278817628 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1131856228 0 0) 0x7f4e64001060 con 0x7f4e6c1b0470 2021-02-10T11:30:32.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.685 7f4e63fff700 1 -- 172.21.2.112:0/4278817628 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e58002fe0 con 0x7f4e6c1b0470 2021-02-10T11:30:32.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.685 7f4e63fff700 1 -- 172.21.2.112:0/4278817628 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1618449217 0 0) 0x7f4e64001460 con 0x7f4e6c1b0470 2021-02-10T11:30:32.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.685 7f4e71cf6700 1 -- 172.21.2.112:0/4278817628 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4e6c1b0870 msgr2=0x7f4e6c0f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:32.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.685 7f4e71cf6700 1 --2- 172.21.2.112:0/4278817628 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4e6c1b0870 0x7f4e6c0f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:32.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.685 7f4e71cf6700 1 -- 172.21.2.112:0/4278817628 >> v1:172.21.2.112:6789/0 conn(0x7f4e6c1b0470 legacy=0x7f4e6c0ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:32.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.685 7f4e71cf6700 1 -- 172.21.2.112:0/4278817628 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e6c0e52e0 con 0x7f4e6c0efe00 2021-02-10T11:30:32.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.685 7f4e71cf6700 1 --2- 172.21.2.112:0/4278817628 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4e6c0efe00 0x7f4e6c0f0200 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7f4e68006ad0 tx=0x7f4e68009a50).ready entity=mon.1 client_cookie=6064eafd816d50e6 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:32.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.686 7f4e63fff700 1 -- 172.21.2.112:0/4278817628 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4e680066a0 con 0x7f4e6c0efe00 2021-02-10T11:30:32.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.686 7f4e63fff700 1 -- 172.21.2.112:0/4278817628 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e68006410 con 0x7f4e6c0efe00 2021-02-10T11:30:32.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.686 7f4e63fff700 1 -- 172.21.2.112:0/4278817628 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4e680024d0 con 0x7f4e6c0efe00 2021-02-10T11:30:32.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.686 7f4e736d9700 1 -- 172.21.2.112:0/4278817628 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4e6c0efe00 msgr2=0x7f4e6c0f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:32.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.686 7f4e736d9700 1 --2- 172.21.2.112:0/4278817628 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4e6c0efe00 0x7f4e6c0f0200 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7f4e68006ad0 tx=0x7f4e68009a50).stop 2021-02-10T11:30:32.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.686 7f4e736d9700 1 -- 172.21.2.112:0/4278817628 shutdown_connections 2021-02-10T11:30:32.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.686 7f4e736d9700 1 --2- 172.21.2.112:0/4278817628 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4e6c0efe00 0x7f4e6c0f0200 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:32.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.686 7f4e736d9700 1 --2- 172.21.2.112:0/4278817628 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4e6c1b0870 0x7f4e6c0f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:32.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.686 7f4e736d9700 1 -- 172.21.2.112:0/4278817628 >> 172.21.2.112:0/4278817628 conn(0x7f4e6c0ce3e0 msgr2=0x7f4e6c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:32.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.687 7f4e736d9700 1 -- 172.21.2.112:0/4278817628 shutdown_connections 2021-02-10T11:30:32.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.687 7f4e736d9700 1 -- 172.21.2.112:0/4278817628 wait complete. 2021-02-10T11:30:32.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.701 7f4e736d9700 1 Processor -- start 2021-02-10T11:30:32.695 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.701 7f4e736d9700 1 -- start start 2021-02-10T11:30:32.695 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.702 7f4e736d9700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4e6c190920 0x7f4e6c0b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:32.695 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.702 7f4e736d9700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4e6c0b8200 0x7f4e6c0b8600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:32.695 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.702 7f4e736d9700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4e6c0c1d80 0x7f4e6c0c2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:32.695 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.702 7f4e736d9700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e6c190da0 con 0x7f4e6c0b8200 2021-02-10T11:30:32.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.702 7f4e736d9700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e6c0b8c20 con 0x7f4e6c0c1d80 2021-02-10T11:30:32.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.702 7f4e736d9700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e6c0b8e10 con 0x7f4e6c190920 2021-02-10T11:30:32.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.702 7f4e70cf4700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4e6c0b8200 0x7f4e6c0b8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:32.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.702 7f4e70cf4700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4e6c0b8200 0x7f4e6c0b8600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:47102/0 (socket says 172.21.2.112:47102) 2021-02-10T11:30:32.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.702 7f4e70cf4700 1 -- 172.21.2.112:0/2998545471 learned_addr learned my addr 172.21.2.112:0/2998545471 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:30:32.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.702 7f4e70cf4700 1 -- 172.21.2.112:0/2998545471 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4e6c190920 msgr2=0x7f4e6c0b7be0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:30:32.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.702 7f4e70cf4700 1 --2- 172.21.2.112:0/2998545471 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4e6c190920 0x7f4e6c0b7be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:32.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.702 7f4e70cf4700 1 -- 172.21.2.112:0/2998545471 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4e6c0c1d80 msgr2=0x7f4e6c0c2180 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:30:32.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.702 7f4e70cf4700 1 --2- 172.21.2.112:0/2998545471 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4e6c0c1d80 0x7f4e6c0c2180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:32.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.702 7f4e70cf4700 1 -- 172.21.2.112:0/2998545471 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e6c0c27d0 con 0x7f4e6c0b8200 2021-02-10T11:30:32.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.702 7f4e70cf4700 1 --2- 172.21.2.112:0/2998545471 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4e6c0b8200 0x7f4e6c0b8600 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 rx=0x7f4e5c006f60 tx=0x7f4e5c00b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:32.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.702 7f4e61ffb700 1 -- 172.21.2.112:0/2998545471 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4e5c00bd60 con 0x7f4e6c0b8200 2021-02-10T11:30:32.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.702 7f4e61ffb700 1 -- 172.21.2.112:0/2998545471 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e5c000c90 con 0x7f4e6c0b8200 2021-02-10T11:30:32.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.702 7f4e61ffb700 1 -- 172.21.2.112:0/2998545471 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4e5c0022b0 con 0x7f4e6c0b8200 2021-02-10T11:30:32.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.702 7f4e736d9700 1 -- 172.21.2.112:0/2998545471 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e6c0c27d0 con 0x7f4e6c0b8200 2021-02-10T11:30:32.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.702 7f4e736d9700 1 -- 172.21.2.112:0/2998545471 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e6c0c2d60 con 0x7f4e6c0b8200 2021-02-10T11:30:32.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.703 7f4e61ffb700 1 -- 172.21.2.112:0/2998545471 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4e5c006b00 con 0x7f4e6c0b8200 2021-02-10T11:30:32.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.703 7f4e61ffb700 1 --2- 172.21.2.112:0/2998545471 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4e4c0186c0 0x7f4e4c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:32.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.703 7f4e61ffb700 1 -- 172.21.2.112:0/2998545471 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f4e5c0327d0 con 0x7f4e6c0b8200 2021-02-10T11:30:32.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.704 7f4e714f5700 1 --2- 172.21.2.112:0/2998545471 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4e4c0186c0 0x7f4e4c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:32.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.704 7f4e714f5700 1 --2- 172.21.2.112:0/2998545471 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4e4c0186c0 0x7f4e4c01abc0 crc :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:32.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.734 7f4e736d9700 1 -- 172.21.2.112:0/2998545471 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e6c105ae0 con 0x7f4e6c0b8200 2021-02-10T11:30:32.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.737 7f4e61ffb700 1 -- 172.21.2.112:0/2998545471 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4e5c000940 con 0x7f4e6c0b8200 2021-02-10T11:30:32.852 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.864 7f4e736d9700 1 -- 172.21.2.112:0/2998545471 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348607", "target": ["mgr", ""]}) v1 -- 0x7f4e6c0c2d60 con 0x7f4e4c0186c0 2021-02-10T11:30:32.862 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.875 7f4e61ffb700 1 -- 172.21.2.112:0/2998545471 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f4e64001cc0 con 0x7f4e4c0186c0 2021-02-10T11:30:32.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.881 7f4e736d9700 1 -- 172.21.2.112:0/2998545471 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4e4c0186c0 msgr2=0x7f4e4c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:32.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.881 7f4e736d9700 1 --2- 172.21.2.112:0/2998545471 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4e4c0186c0 0x7f4e4c01abc0 crc :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:32.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.881 7f4e736d9700 1 -- 172.21.2.112:0/2998545471 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4e6c0b8200 msgr2=0x7f4e6c0b8600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:32.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.881 7f4e736d9700 1 --2- 172.21.2.112:0/2998545471 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4e6c0b8200 0x7f4e6c0b8600 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 rx=0x7f4e5c006f60 tx=0x7f4e5c00b970).stop 2021-02-10T11:30:32.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.881 7f4e736d9700 1 -- 172.21.2.112:0/2998545471 shutdown_connections 2021-02-10T11:30:32.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.881 7f4e736d9700 1 --2- 172.21.2.112:0/2998545471 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4e4c0186c0 0x7f4e4c01abc0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:32.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.881 7f4e736d9700 1 --2- 172.21.2.112:0/2998545471 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4e6c0c1d80 0x7f4e6c0c2180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:32.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.881 7f4e736d9700 1 --2- 172.21.2.112:0/2998545471 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4e6c0b8200 0x7f4e6c0b8600 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:32.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.881 7f4e736d9700 1 --2- 172.21.2.112:0/2998545471 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4e6c190920 0x7f4e6c0b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:32.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.881 7f4e736d9700 1 -- 172.21.2.112:0/2998545471 >> 172.21.2.112:0/2998545471 conn(0x7f4e6c094e10 msgr2=0x7f4e6c0a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:32.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.882 7f4e736d9700 1 -- 172.21.2.112:0/2998545471 shutdown_connections 2021-02-10T11:30:32.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:32.882 7f4e736d9700 1 -- 172.21.2.112:0/2998545471 wait complete. 2021-02-10T11:30:32.878 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000348608 2021-02-10T11:30:33.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.306 7feec5832700 1 Processor -- start 2021-02-10T11:30:33.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.306 7feec5832700 1 -- start start 2021-02-10T11:30:33.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.306 7feec5832700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7feec01b0470 0x7feec01b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:33.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.306 7feec5832700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7feec00ef610 0x7feec00efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:33.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.306 7feec5832700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feec00f0b50 con 0x7feec00f0030 2021-02-10T11:30:33.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.306 7feec5832700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feec00e52e0 con 0x7feec01b0470 2021-02-10T11:30:33.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.306 7feec5832700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feec00e54d0 con 0x7feec00ef610 2021-02-10T11:30:33.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.307 7feebf574700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7feec01b0470 0x7feec01b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:33.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.307 7feebf574700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7feec01b0470 0x7feec01b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:47110/0 (socket says 172.21.2.112:47110) 2021-02-10T11:30:33.311 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.307 7feebed73700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7feec00ef610 0x7feec00efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:33.311 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.307 7feebf574700 1 -- 172.21.2.112:0/2482501972 learned_addr learned my addr 172.21.2.112:0/2482501972 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:30:33.311 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.307 7feebe572700 1 -- 172.21.2.112:0/2482501972 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3726194583 0 0) 0x7feeb8001400 con 0x7feec00f0030 2021-02-10T11:30:33.311 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.307 7feebf574700 1 -- 172.21.2.112:0/2482501972 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7feec00ef610 msgr2=0x7feec00efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:33.311 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.307 7feebf574700 1 --2- 172.21.2.112:0/2482501972 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7feec00ef610 0x7feec00efa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:33.311 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.307 7feebf574700 1 -- 172.21.2.112:0/2482501972 >> v1:172.21.2.120:6789/0 conn(0x7feec00f0030 legacy=0x7feec00f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:33.311 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.307 7feebf574700 1 -- 172.21.2.112:0/2482501972 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feec00f0430 con 0x7feec01b0470 2021-02-10T11:30:33.311 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.307 7feebf574700 1 --2- 172.21.2.112:0/2482501972 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7feec01b0470 0x7feec01b0870 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 rx=0x7feeb0006700 tx=0x7feeb0009b80).ready entity=mon.0 client_cookie=f54b334b63004412 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:33.311 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.307 7feebe572700 1 -- 172.21.2.112:0/2482501972 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feeb000c0c0 con 0x7feec01b0470 2021-02-10T11:30:33.311 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.307 7feebe572700 1 -- 172.21.2.112:0/2482501972 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feeb000c440 con 0x7feec01b0470 2021-02-10T11:30:33.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.307 7feebe572700 1 -- 172.21.2.112:0/2482501972 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feeb000c8a0 con 0x7feec01b0470 2021-02-10T11:30:33.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.307 7feec5832700 1 -- 172.21.2.112:0/2482501972 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7feec01b0470 msgr2=0x7feec01b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:33.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.307 7feec5832700 1 --2- 172.21.2.112:0/2482501972 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7feec01b0470 0x7feec01b0870 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 rx=0x7feeb0006700 tx=0x7feeb0009b80).stop 2021-02-10T11:30:33.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.308 7feec5832700 1 -- 172.21.2.112:0/2482501972 shutdown_connections 2021-02-10T11:30:33.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.308 7feec5832700 1 --2- 172.21.2.112:0/2482501972 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7feec00ef610 0x7feec00efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:33.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.308 7feec5832700 1 --2- 172.21.2.112:0/2482501972 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7feec01b0470 0x7feec01b0870 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:33.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.308 7feec5832700 1 -- 172.21.2.112:0/2482501972 >> 172.21.2.112:0/2482501972 conn(0x7feec00ce3e0 msgr2=0x7feec00e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:33.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.308 7feec5832700 1 -- 172.21.2.112:0/2482501972 shutdown_connections 2021-02-10T11:30:33.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.308 7feec5832700 1 -- 172.21.2.112:0/2482501972 wait complete. 2021-02-10T11:30:33.314 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.322 7feec5832700 1 Processor -- start 2021-02-10T11:30:33.314 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.323 7feec5832700 1 -- start start 2021-02-10T11:30:33.314 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.323 7feec5832700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7feec0106fd0 0x7feec01a7500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:33.314 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.323 7feec5832700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7feec01a7b20 0x7feec00c25e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:33.314 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.323 7feec5832700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7feec01a7f20 0x7feec00c2c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:33.314 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.323 7feec5832700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feec008c920 con 0x7feec01a7f20 2021-02-10T11:30:33.314 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.323 7feec5832700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feec008c680 con 0x7feec0106fd0 2021-02-10T11:30:33.314 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.323 7feec5832700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feec0107810 con 0x7feec01a7b20 2021-02-10T11:30:33.314 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.323 7feebfd75700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7feec01a7f20 0x7feec00c2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:33.315 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.323 7feebfd75700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7feec01a7f20 0x7feec00c2c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:47118/0 (socket says 172.21.2.112:47118) 2021-02-10T11:30:33.315 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.323 7feebfd75700 1 -- 172.21.2.112:0/2439647498 learned_addr learned my addr 172.21.2.112:0/2439647498 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:30:33.315 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.324 7feebfd75700 1 -- 172.21.2.112:0/2439647498 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7feec01a7b20 msgr2=0x7feec00c25e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:33.315 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.324 7feebf574700 1 --2- 172.21.2.112:0/2439647498 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7feec0106fd0 0x7feec01a7500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:33.315 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.324 7feebfd75700 1 --2- 172.21.2.112:0/2439647498 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7feec01a7b20 0x7feec00c25e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:33.315 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.324 7feebfd75700 1 -- 172.21.2.112:0/2439647498 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7feec0106fd0 msgr2=0x7feec01a7500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:33.315 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.324 7feebfd75700 1 --2- 172.21.2.112:0/2439647498 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7feec0106fd0 0x7feec01a7500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:33.315 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.324 7feebfd75700 1 -- 172.21.2.112:0/2439647498 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feec01073d0 con 0x7feec01a7f20 2021-02-10T11:30:33.315 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.324 7feebf574700 1 --2- 172.21.2.112:0/2439647498 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7feec0106fd0 0x7feec01a7500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:30:33.315 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.324 7feebfd75700 1 --2- 172.21.2.112:0/2439647498 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7feec01a7f20 0x7feec00c2c00 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 rx=0x7feeb8009940 tx=0x7feeb800bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:33.315 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.324 7feea7fff700 1 -- 172.21.2.112:0/2439647498 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feeb8000f40 con 0x7feec01a7f20 2021-02-10T11:30:33.315 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.324 7feea7fff700 1 -- 172.21.2.112:0/2439647498 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feeb800c2f0 con 0x7feec01a7f20 2021-02-10T11:30:33.315 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.324 7feea7fff700 1 -- 172.21.2.112:0/2439647498 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feeb800c750 con 0x7feec01a7f20 2021-02-10T11:30:33.316 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.324 7feec5832700 1 -- 172.21.2.112:0/2439647498 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feec01073d0 con 0x7feec01a7f20 2021-02-10T11:30:33.316 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.324 7feec5832700 1 -- 172.21.2.112:0/2439647498 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feec00c3510 con 0x7feec01a7f20 2021-02-10T11:30:33.316 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.325 7feea7fff700 1 -- 172.21.2.112:0/2439647498 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7feeb8001230 con 0x7feec01a7f20 2021-02-10T11:30:33.316 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.325 7feea7fff700 1 --2- 172.21.2.112:0/2439647498 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7feea0018730 0x7feea001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:33.316 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.325 7feea7fff700 1 -- 172.21.2.112:0/2439647498 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7feeb8032b80 con 0x7feec01a7f20 2021-02-10T11:30:33.316 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.325 7feebf574700 1 --2- 172.21.2.112:0/2439647498 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7feea0018730 0x7feea001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:33.316 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.326 7feebf574700 1 --2- 172.21.2.112:0/2439647498 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7feea0018730 0x7feea001abc0 crc :-1 s=READY pgs=429 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:33.344 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.356 7feec5832700 1 -- 172.21.2.112:0/2439647498 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feec0105ae0 con 0x7feec01a7f20 2021-02-10T11:30:33.346 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.359 7feea7fff700 1 -- 172.21.2.112:0/2439647498 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7feeb8000f40 con 0x7feec01a7f20 2021-02-10T11:30:33.473 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.485 7feec5832700 1 -- 172.21.2.112:0/2439647498 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348608", "target": ["mgr", ""]}) v1 -- 0x7feec00c3510 con 0x7feea0018730 2021-02-10T11:30:33.484 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.497 7feea7fff700 1 -- 172.21.2.112:0/2439647498 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7feeb00009b0 con 0x7feea0018730 2021-02-10T11:30:33.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.503 7feec5832700 1 -- 172.21.2.112:0/2439647498 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7feea0018730 msgr2=0x7feea001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:33.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.503 7feec5832700 1 --2- 172.21.2.112:0/2439647498 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7feea0018730 0x7feea001abc0 crc :-1 s=READY pgs=429 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:33.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.503 7feec5832700 1 -- 172.21.2.112:0/2439647498 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7feec01a7f20 msgr2=0x7feec00c2c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:33.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.503 7feec5832700 1 --2- 172.21.2.112:0/2439647498 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7feec01a7f20 0x7feec00c2c00 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 rx=0x7feeb8009940 tx=0x7feeb800bcf0).stop 2021-02-10T11:30:33.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.503 7feec5832700 1 -- 172.21.2.112:0/2439647498 shutdown_connections 2021-02-10T11:30:33.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.503 7feec5832700 1 --2- 172.21.2.112:0/2439647498 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7feea0018730 0x7feea001abc0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:33.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.503 7feec5832700 1 --2- 172.21.2.112:0/2439647498 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7feec01a7f20 0x7feec00c2c00 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:33.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.503 7feec5832700 1 --2- 172.21.2.112:0/2439647498 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7feec01a7b20 0x7feec00c25e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:33.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.503 7feec5832700 1 --2- 172.21.2.112:0/2439647498 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7feec0106fd0 0x7feec01a7500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:33.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.503 7feec5832700 1 -- 172.21.2.112:0/2439647498 >> 172.21.2.112:0/2439647498 conn(0x7feec00bbf60 msgr2=0x7feec0105d10 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:33.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.504 7feec5832700 1 -- 172.21.2.112:0/2439647498 shutdown_connections 2021-02-10T11:30:33.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.504 7feec5832700 1 -- 172.21.2.112:0/2439647498 wait complete. 2021-02-10T11:30:33.500 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000348609 2021-02-10T11:30:33.930 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.926 7fbce586c700 1 Processor -- start 2021-02-10T11:30:33.930 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.926 7fbce586c700 1 -- start start 2021-02-10T11:30:33.930 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.926 7fbce586c700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fbce01b0470 0x7fbce01b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:33.930 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.926 7fbce586c700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbce00f0b50 con 0x7fbce00f20e0 2021-02-10T11:30:33.930 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.926 7fbce586c700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbce00f24e0 con 0x7fbce00ef610 2021-02-10T11:30:33.930 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.926 7fbce586c700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbce00e5ea0 con 0x7fbce01b0470 2021-02-10T11:30:33.930 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.927 7fbcdfd75700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fbce00f20e0 0x7fbce00f0200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:46586/0 (socket says 172.21.2.112:46586) 2021-02-10T11:30:33.930 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.927 7fbcdfd75700 1 -- 172.21.2.112:0/2610442889 learned_addr learned my addr 172.21.2.112:0/2610442889 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:30:33.930 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.927 7fbcdf574700 1 --2- 172.21.2.112:0/2610442889 >> v2:172.21.2.112:3300/0 conn(0x7fbce01b0470 0x7fbce01b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:33.930 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.927 7fbcde572700 1 -- 172.21.2.112:0/2610442889 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1245949382 0 0) 0x7fbcd8001500 con 0x7fbce00f20e0 2021-02-10T11:30:33.930 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.927 7fbcde572700 1 -- 172.21.2.112:0/2610442889 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbccc002fe0 con 0x7fbce00f20e0 2021-02-10T11:30:33.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.927 7fbcde572700 1 -- 172.21.2.112:0/2610442889 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1805305455 0 0) 0x7fbcc8000ff0 con 0x7fbce00ef610 2021-02-10T11:30:33.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.927 7fbcdf574700 1 -- 172.21.2.112:0/2610442889 >> v1:172.21.2.120:6789/0 conn(0x7fbce00ef610 legacy=0x7fbce00efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:33.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.927 7fbcdf574700 1 -- 172.21.2.112:0/2610442889 >> v1:172.21.2.112:6789/0 conn(0x7fbce00f20e0 legacy=0x7fbce00f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:33.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.927 7fbcdf574700 1 -- 172.21.2.112:0/2610442889 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbce00e52e0 con 0x7fbce01b0470 2021-02-10T11:30:33.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.927 7fbcdf574700 1 --2- 172.21.2.112:0/2610442889 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbce01b0470 0x7fbce01b0870 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 rx=0x7fbcd0006650 tx=0x7fbcd0009a80).ready entity=mon.0 client_cookie=14fb612d3364b6d4 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:33.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.928 7fbcde572700 1 -- 172.21.2.112:0/2610442889 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbcd00009b0 con 0x7fbce01b0470 2021-02-10T11:30:33.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.928 7fbcde572700 1 -- 172.21.2.112:0/2610442889 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcd000c0c0 con 0x7fbce01b0470 2021-02-10T11:30:33.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.928 7fbcde572700 1 -- 172.21.2.112:0/2610442889 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbcd000c5e0 con 0x7fbce01b0470 2021-02-10T11:30:33.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.928 7fbce586c700 1 -- 172.21.2.112:0/2610442889 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbce01b0470 msgr2=0x7fbce01b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:33.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.928 7fbce586c700 1 --2- 172.21.2.112:0/2610442889 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbce01b0470 0x7fbce01b0870 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 rx=0x7fbcd0006650 tx=0x7fbcd0009a80).stop 2021-02-10T11:30:33.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.928 7fbce586c700 1 -- 172.21.2.112:0/2610442889 shutdown_connections 2021-02-10T11:30:33.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.928 7fbce586c700 1 --2- 172.21.2.112:0/2610442889 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbce01b0470 0x7fbce01b0870 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:33.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.928 7fbce586c700 1 -- 172.21.2.112:0/2610442889 >> 172.21.2.112:0/2610442889 conn(0x7fbce00ce3e0 msgr2=0x7fbce00e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:33.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.928 7fbce586c700 1 -- 172.21.2.112:0/2610442889 shutdown_connections 2021-02-10T11:30:33.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.928 7fbce586c700 1 -- 172.21.2.112:0/2610442889 wait complete. 2021-02-10T11:30:33.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.943 7fbce586c700 1 Processor -- start 2021-02-10T11:30:33.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.943 7fbce586c700 1 -- start start 2021-02-10T11:30:33.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.943 7fbce586c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbce0193230 0x7fbce00a6a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:33.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.943 7fbce586c700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbce008f5b0 0x7fbce008f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:33.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.943 7fbce586c700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fbce008e6a0 0x7fbce008eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:33.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.943 7fbce586c700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbce008ffd0 con 0x7fbce0193230 2021-02-10T11:30:33.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.943 7fbce586c700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbce0090240 con 0x7fbce008e6a0 2021-02-10T11:30:33.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.943 7fbce586c700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbce008f0c0 con 0x7fbce008f5b0 2021-02-10T11:30:33.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.943 7fbcdfd75700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fbce008e6a0 0x7fbce008eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:33.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.943 7fbcded73700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbce008f5b0 0x7fbce008f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:33.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.943 7fbcdfd75700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fbce008e6a0 0x7fbce008eaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3300/0 says I am v2:172.21.2.112:49412/0 (socket says 172.21.2.112:49412) 2021-02-10T11:30:33.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.943 7fbcded73700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbce008f5b0 0x7fbce008f9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3301/0 says I am v2:172.21.2.112:46812/0 (socket says 172.21.2.112:46812) 2021-02-10T11:30:33.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.943 7fbcdfd75700 1 -- 172.21.2.112:0/1987154902 learned_addr learned my addr 172.21.2.112:0/1987154902 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:30:33.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.943 7fbcdf574700 1 --2- 172.21.2.112:0/1987154902 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbce0193230 0x7fbce00a6a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:33.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.944 7fbcdf574700 1 -- 172.21.2.112:0/1987154902 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbce008f5b0 msgr2=0x7fbce008f9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:33.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.944 7fbcdf574700 1 --2- 172.21.2.112:0/1987154902 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbce008f5b0 0x7fbce008f9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:33.935 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.944 7fbcdf574700 1 -- 172.21.2.112:0/1987154902 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fbce008e6a0 msgr2=0x7fbce008eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:33.935 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.944 7fbcdf574700 1 --2- 172.21.2.112:0/1987154902 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fbce008e6a0 0x7fbce008eaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:33.935 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.944 7fbcdf574700 1 -- 172.21.2.112:0/1987154902 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbce008f2a0 con 0x7fbce0193230 2021-02-10T11:30:33.935 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.944 7fbcdf574700 1 --2- 172.21.2.112:0/1987154902 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbce0193230 0x7fbce00a6a80 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 rx=0x7fbcd000b530 tx=0x7fbcd000b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:33.935 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.944 7fbcd7fff700 1 -- 172.21.2.112:0/1987154902 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbcd00093a0 con 0x7fbce0193230 2021-02-10T11:30:33.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.944 7fbcd7fff700 1 -- 172.21.2.112:0/1987154902 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcd000bcc0 con 0x7fbce0193230 2021-02-10T11:30:33.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.944 7fbcd7fff700 1 -- 172.21.2.112:0/1987154902 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbcd000a480 con 0x7fbce0193230 2021-02-10T11:30:33.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.944 7fbce586c700 1 -- 172.21.2.112:0/1987154902 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbce008f2a0 con 0x7fbce0193230 2021-02-10T11:30:33.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.944 7fbce586c700 1 -- 172.21.2.112:0/1987154902 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbce01924f0 con 0x7fbce0193230 2021-02-10T11:30:33.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.945 7fbcd7fff700 1 -- 172.21.2.112:0/1987154902 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fbcd0000940 con 0x7fbce0193230 2021-02-10T11:30:33.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.945 7fbcd7fff700 1 --2- 172.21.2.112:0/1987154902 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fbcc40186c0 0x7fbcc401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:33.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.945 7fbcd7fff700 1 -- 172.21.2.112:0/1987154902 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fbcd00324c0 con 0x7fbce0193230 2021-02-10T11:30:33.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.945 7fbcded73700 1 --2- 172.21.2.112:0/1987154902 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fbcc40186c0 0x7fbcc401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:33.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.946 7fbcded73700 1 --2- 172.21.2.112:0/1987154902 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fbcc40186c0 0x7fbcc401abc0 crc :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:33.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.976 7fbce586c700 1 -- 172.21.2.112:0/1987154902 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbce00f0b50 con 0x7fbce0193230 2021-02-10T11:30:33.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:33.979 7fbcd7fff700 1 -- 172.21.2.112:0/1987154902 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fbcd0020c50 con 0x7fbce0193230 2021-02-10T11:30:34.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:34.113 7fbce586c700 1 -- 172.21.2.112:0/1987154902 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348609", "target": ["mgr", ""]}) v1 -- 0x7fbce008f2a0 con 0x7fbcc40186c0 2021-02-10T11:30:34.109 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:34.123 7fbcd7fff700 1 -- 172.21.2.112:0/1987154902 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fbcc80078a0 con 0x7fbcc40186c0 2021-02-10T11:30:34.118 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:34.130 7fbce586c700 1 -- 172.21.2.112:0/1987154902 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fbcc40186c0 msgr2=0x7fbcc401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:34.118 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:34.130 7fbce586c700 1 --2- 172.21.2.112:0/1987154902 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fbcc40186c0 0x7fbcc401abc0 crc :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:34.118 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:34.130 7fbce586c700 1 -- 172.21.2.112:0/1987154902 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbce0193230 msgr2=0x7fbce00a6a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:34.118 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:34.130 7fbce586c700 1 --2- 172.21.2.112:0/1987154902 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbce0193230 0x7fbce00a6a80 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 rx=0x7fbcd000b530 tx=0x7fbcd000b8d0).stop 2021-02-10T11:30:34.118 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:34.130 7fbce586c700 1 -- 172.21.2.112:0/1987154902 shutdown_connections 2021-02-10T11:30:34.119 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:34.130 7fbce586c700 1 --2- 172.21.2.112:0/1987154902 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fbcc40186c0 0x7fbcc401abc0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:34.119 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:34.130 7fbce586c700 1 --2- 172.21.2.112:0/1987154902 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fbce008e6a0 0x7fbce008eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:34.119 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:34.130 7fbce586c700 1 --2- 172.21.2.112:0/1987154902 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbce008f5b0 0x7fbce008f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:34.119 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:34.130 7fbce586c700 1 --2- 172.21.2.112:0/1987154902 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbce0193230 0x7fbce00a6a80 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:34.119 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:34.130 7fbce586c700 1 -- 172.21.2.112:0/1987154902 >> 172.21.2.112:0/1987154902 conn(0x7fbce00bf4e0 msgr2=0x7fbce00be160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:34.119 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:34.130 7fbce586c700 1 -- 172.21.2.112:0/1987154902 shutdown_connections 2021-02-10T11:30:34.119 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:34.130 7fbce586c700 1 -- 172.21.2.112:0/1987154902 wait complete. 2021-02-10T11:30:34.126 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000348610 2021-02-10T11:30:34.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:34.550 7f4de8e97700 1 Processor -- start 2021-02-10T11:30:34.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:34.550 7f4de8e97700 1 -- start start 2021-02-10T11:30:34.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:34.551 7f4de8e97700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f4de41b0470 0x7f4de41b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:34.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:34.551 7f4de8e97700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4de40f0b50 con 0x7f4de40f20e0 2021-02-10T11:30:34.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:34.551 7f4de8e97700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4de40f24e0 con 0x7f4de40ef610 2021-02-10T11:30:34.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:34.551 7f4de8e97700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4de40e5ea0 con 0x7f4de41b0470 2021-02-10T11:30:34.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:34.551 7f4de3314700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f4de40f20e0 0x7f4de40f0200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:46600/0 (socket says 172.21.2.112:46600) 2021-02-10T11:30:34.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:34.551 7f4de3314700 1 -- 172.21.2.112:0/3831575441 learned_addr learned my addr 172.21.2.112:0/3831575441 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:30:34.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:34.551 7f4de2b13700 1 --2- 172.21.2.112:0/3831575441 >> v2:172.21.2.112:3300/0 conn(0x7f4de41b0470 0x7f4de41b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:34.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:34.551 7f4de2b13700 1 -- 172.21.2.112:0/3831575441 >> v1:172.21.2.120:6789/0 conn(0x7f4de40ef610 legacy=0x7f4de40efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:34.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:34.551 7f4de2b13700 1 -- 172.21.2.112:0/3831575441 >> v1:172.21.2.112:6789/0 conn(0x7f4de40f20e0 legacy=0x7f4de40f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:34.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:34.551 7f4de1b11700 1 -- 172.21.2.112:0/3831575441 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3683243805 0 0) 0x7f4dd0000f30 con 0x7f4de40ef610 2021-02-10T11:30:34.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:34.551 7f4de2b13700 1 -- 172.21.2.112:0/3831575441 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4de40e52e0 con 0x7f4de41b0470 2021-02-10T11:30:34.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:34.552 7f4de2b13700 1 --2- 172.21.2.112:0/3831575441 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4de41b0470 0x7f4de41b0870 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 rx=0x7f4dd4006650 tx=0x7f4dd4009a00).ready entity=mon.0 client_cookie=a752647d873ffd99 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:34.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:34.552 7f4de1b11700 1 -- 172.21.2.112:0/3831575441 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4dd40009b0 con 0x7f4de41b0470 2021-02-10T11:30:34.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:34.552 7f4de1b11700 1 -- 172.21.2.112:0/3831575441 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dd400c0c0 con 0x7f4de41b0470 2021-02-10T11:30:34.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:34.552 7f4de1b11700 1 -- 172.21.2.112:0/3831575441 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4dd400c5a0 con 0x7f4de41b0470 2021-02-10T11:30:34.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:34.552 7f4de8e97700 1 -- 172.21.2.112:0/3831575441 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4de41b0470 msgr2=0x7f4de41b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:34.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:34.552 7f4de8e97700 1 --2- 172.21.2.112:0/3831575441 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4de41b0470 0x7f4de41b0870 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 rx=0x7f4dd4006650 tx=0x7f4dd4009a00).stop 2021-02-10T11:30:34.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:34.553 7f4de8e97700 1 -- 172.21.2.112:0/3831575441 shutdown_connections 2021-02-10T11:30:34.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:34.553 7f4de8e97700 1 --2- 172.21.2.112:0/3831575441 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4de41b0470 0x7f4de41b0870 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:34.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:34.553 7f4de8e97700 1 -- 172.21.2.112:0/3831575441 >> 172.21.2.112:0/3831575441 conn(0x7f4de40ce3e0 msgr2=0x7f4de40e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:34.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:34.553 7f4de8e97700 1 -- 172.21.2.112:0/3831575441 shutdown_connections 2021-02-10T11:30:34.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:34.553 7f4de8e97700 1 -- 172.21.2.112:0/3831575441 wait complete. 2021-02-10T11:30:34.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:34.568 7f4de8e97700 1 Processor -- start 2021-02-10T11:30:34.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:34.568 7f4de8e97700 1 -- start start 2021-02-10T11:30:34.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:34.568 7f4de8e97700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4de41a0570 0x7f4de40a4590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:34.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:34.568 7f4de8e97700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4de40f1fd0 0x7f4de40f23d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:34.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:34.568 7f4de8e97700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4de40a5450 0x7f4de40a5850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:34.563 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:34.568 7f4de8e97700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4de40a4bb0 con 0x7f4de40f1fd0 2021-02-10T11:30:34.563 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:34.568 7f4de8e97700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4de40b35e0 con 0x7f4de40a5450 2021-02-10T11:30:34.563 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:34.568 7f4de8e97700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4de40a1f00 con 0x7f4de41a0570 2021-02-10T11:30:34.563 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:34.568 7f4de3314700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4de40a5450 0x7f4de40a5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:34.563 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:34.569 7f4de3314700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4de40a5450 0x7f4de40a5850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3300/0 says I am v2:172.21.2.112:49424/0 (socket says 172.21.2.112:49424) 2021-02-10T11:30:34.563 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:34.569 7f4de2312700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4de40f1fd0 0x7f4de40f23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:34.563 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:34.569 7f4de3314700 1 -- 172.21.2.112:0/3364116784 learned_addr learned my addr 172.21.2.112:0/3364116784 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:30:34.563 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:34.569 7f4de2b13700 1 --2- 172.21.2.112:0/3364116784 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4de41a0570 0x7f4de40a4590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:34.563 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:34.569 7f4de3314700 1 -- 172.21.2.112:0/3364116784 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4de41a0570 msgr2=0x7f4de40a4590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:34.563 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:34.569 7f4de3314700 1 --2- 172.21.2.112:0/3364116784 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4de41a0570 0x7f4de40a4590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:34.563 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:34.569 7f4de3314700 1 -- 172.21.2.112:0/3364116784 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4de40f1fd0 msgr2=0x7f4de40f23d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:34.563 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:34.569 7f4de3314700 1 --2- 172.21.2.112:0/3364116784 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4de40f1fd0 0x7f4de40f23d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:34.563 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:34.569 7f4de3314700 1 -- 172.21.2.112:0/3364116784 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4de40a5e70 con 0x7f4de40a5450 2021-02-10T11:30:34.564 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:34.569 7f4de2b13700 1 --2- 172.21.2.112:0/3364116784 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4de41a0570 0x7f4de40a4590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:30:34.564 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:34.569 7f4de2312700 1 --2- 172.21.2.112:0/3364116784 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4de40f1fd0 0x7f4de40f23d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:30:34.564 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:34.569 7f4de3314700 1 --2- 172.21.2.112:0/3364116784 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4de40a5450 0x7f4de40a5850 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7f4ddc001bf0 tx=0x7f4ddc00ac50).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:34.564 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:34.569 7f4ddb7fe700 1 -- 172.21.2.112:0/3364116784 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4ddc0070d0 con 0x7f4de40a5450 2021-02-10T11:30:34.564 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:34.569 7f4ddb7fe700 1 -- 172.21.2.112:0/3364116784 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ddc007390 con 0x7f4de40a5450 2021-02-10T11:30:34.564 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:34.569 7f4de8e97700 1 -- 172.21.2.112:0/3364116784 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4de40a5e70 con 0x7f4de40a5450 2021-02-10T11:30:34.564 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:34.569 7f4ddb7fe700 1 -- 172.21.2.112:0/3364116784 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4ddc004720 con 0x7f4de40a5450 2021-02-10T11:30:34.564 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:34.569 7f4de8e97700 1 -- 172.21.2.112:0/3364116784 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4de40a5e70 con 0x7f4de40a5450 2021-02-10T11:30:34.564 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:34.570 7f4ddb7fe700 1 -- 172.21.2.112:0/3364116784 <== mon.1 v2:172.21.2.120:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4ddc007570 con 0x7f4de40a5450 2021-02-10T11:30:34.564 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:34.570 7f4ddb7fe700 1 --2- 172.21.2.112:0/3364116784 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4dc0018730 0x7f4dc001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:34.564 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:34.570 7f4ddb7fe700 1 -- 172.21.2.112:0/3364116784 <== mon.1 v2:172.21.2.120:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f4ddc0329a0 con 0x7f4de40a5450 2021-02-10T11:30:34.564 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:34.570 7f4de2b13700 1 --2- 172.21.2.112:0/3364116784 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4dc0018730 0x7f4dc001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:34.564 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:34.571 7f4de2b13700 1 --2- 172.21.2.112:0/3364116784 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4dc0018730 0x7f4dc001ac10 crc :-1 s=READY pgs=431 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:34.588 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:34.600 7f4de8e97700 1 -- 172.21.2.112:0/3364116784 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4de40a4bb0 con 0x7f4de40a5450 2021-02-10T11:30:34.592 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:34.605 7f4ddb7fe700 1 -- 172.21.2.112:0/3364116784 <== mon.1 v2:172.21.2.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4ddc000cc0 con 0x7f4de40a5450 2021-02-10T11:30:34.721 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:34.733 7f4de8e97700 1 -- 172.21.2.112:0/3364116784 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348610", "target": ["mgr", ""]}) v1 -- 0x7f4de40a4bb0 con 0x7f4dc0018730 2021-02-10T11:30:34.732 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:34.745 7f4ddb7fe700 1 -- 172.21.2.112:0/3364116784 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f4dd40009b0 con 0x7f4dc0018730 2021-02-10T11:30:34.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:34.750 7f4de8e97700 1 -- 172.21.2.112:0/3364116784 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4dc0018730 msgr2=0x7f4dc001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:34.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:34.750 7f4de8e97700 1 --2- 172.21.2.112:0/3364116784 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4dc0018730 0x7f4dc001ac10 crc :-1 s=READY pgs=431 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:34.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:34.750 7f4de8e97700 1 -- 172.21.2.112:0/3364116784 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4de40a5450 msgr2=0x7f4de40a5850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:34.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:34.750 7f4de8e97700 1 --2- 172.21.2.112:0/3364116784 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4de40a5450 0x7f4de40a5850 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7f4ddc001bf0 tx=0x7f4ddc00ac50).stop 2021-02-10T11:30:34.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:34.750 7f4de8e97700 1 -- 172.21.2.112:0/3364116784 shutdown_connections 2021-02-10T11:30:34.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:34.750 7f4de8e97700 1 --2- 172.21.2.112:0/3364116784 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4dc0018730 0x7f4dc001ac10 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:34.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:34.750 7f4de8e97700 1 --2- 172.21.2.112:0/3364116784 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4de40a5450 0x7f4de40a5850 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:34.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:34.750 7f4de8e97700 1 --2- 172.21.2.112:0/3364116784 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4de40f1fd0 0x7f4de40f23d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:34.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:34.750 7f4de8e97700 1 --2- 172.21.2.112:0/3364116784 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4de41a0570 0x7f4de40a4590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:34.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:34.750 7f4de8e97700 1 -- 172.21.2.112:0/3364116784 >> 172.21.2.112:0/3364116784 conn(0x7f4de40958b0 msgr2=0x7f4de40bf040 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:34.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:34.750 7f4de8e97700 1 -- 172.21.2.112:0/3364116784 shutdown_connections 2021-02-10T11:30:34.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:34.750 7f4de8e97700 1 -- 172.21.2.112:0/3364116784 wait complete. 2021-02-10T11:30:34.747 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000348611 2021-02-10T11:30:35.177 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.173 7fd2da2d6700 1 Processor -- start 2021-02-10T11:30:35.177 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.173 7fd2da2d6700 1 -- start start 2021-02-10T11:30:35.178 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.174 7fd2da2d6700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fd2d41b0870 0x7fd2d40f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:35.178 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.174 7fd2da2d6700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd2d40efe00 0x7fd2d40f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:35.178 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.174 7fd2da2d6700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2d41b0c70 con 0x7fd2d41b0470 2021-02-10T11:30:35.178 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.174 7fd2da2d6700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2d40f0b50 con 0x7fd2d41b0870 2021-02-10T11:30:35.178 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.174 7fd2da2d6700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2d40f0820 con 0x7fd2d40efe00 2021-02-10T11:30:35.178 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.174 7fd2d3fff700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7fd2d41b0470 0x7fd2d40ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:40172/0 (socket says 172.21.2.112:40172) 2021-02-10T11:30:35.178 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.174 7fd2d3fff700 1 -- 172.21.2.112:0/2339504087 learned_addr learned my addr 172.21.2.112:0/2339504087 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:30:35.178 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.174 7fd2d37fe700 1 --2- 172.21.2.112:0/2339504087 >> v2:172.21.2.112:3300/0 conn(0x7fd2d41b0870 0x7fd2d40f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:35.178 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.174 7fd2d88f3700 1 --2- 172.21.2.112:0/2339504087 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd2d40efe00 0x7fd2d40f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:35.179 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.174 7fd2d2ffd700 1 -- 172.21.2.112:0/2339504087 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4207749603 0 0) 0x7fd2c4000c60 con 0x7fd2d41b0470 2021-02-10T11:30:35.179 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.174 7fd2d88f3700 1 -- 172.21.2.112:0/2339504087 >> v2:172.21.2.112:3300/0 conn(0x7fd2d41b0870 msgr2=0x7fd2d40f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:35.179 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.174 7fd2d88f3700 1 --2- 172.21.2.112:0/2339504087 >> v2:172.21.2.112:3300/0 conn(0x7fd2d41b0870 0x7fd2d40f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:35.179 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.174 7fd2d88f3700 1 -- 172.21.2.112:0/2339504087 >> v1:172.21.2.120:6789/0 conn(0x7fd2d41b0470 legacy=0x7fd2d40ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:35.179 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.174 7fd2d88f3700 1 -- 172.21.2.112:0/2339504087 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2d40e52e0 con 0x7fd2d40efe00 2021-02-10T11:30:35.179 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.175 7fd2d88f3700 1 --2- 172.21.2.112:0/2339504087 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd2d40efe00 0x7fd2d40f0200 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7fd2cc006e50 tx=0x7fd2cc009e70).ready entity=mon.2 client_cookie=f5398c8c559253ca server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:35.179 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.175 7fd2d2ffd700 1 -- 172.21.2.112:0/2339504087 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd2cc006a20 con 0x7fd2d40efe00 2021-02-10T11:30:35.179 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.175 7fd2d2ffd700 1 -- 172.21.2.112:0/2339504087 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2cc006790 con 0x7fd2d40efe00 2021-02-10T11:30:35.179 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.175 7fd2d2ffd700 1 -- 172.21.2.112:0/2339504087 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd2cc00b700 con 0x7fd2d40efe00 2021-02-10T11:30:35.179 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.175 7fd2da2d6700 1 -- 172.21.2.112:0/2339504087 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd2d40efe00 msgr2=0x7fd2d40f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:35.179 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.175 7fd2da2d6700 1 --2- 172.21.2.112:0/2339504087 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd2d40efe00 0x7fd2d40f0200 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7fd2cc006e50 tx=0x7fd2cc009e70).stop 2021-02-10T11:30:35.180 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.175 7fd2da2d6700 1 -- 172.21.2.112:0/2339504087 shutdown_connections 2021-02-10T11:30:35.180 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.175 7fd2da2d6700 1 --2- 172.21.2.112:0/2339504087 >> v2:172.21.2.112:3300/0 conn(0x7fd2d41b0870 0x7fd2d40f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:35.180 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.175 7fd2da2d6700 1 --2- 172.21.2.112:0/2339504087 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd2d40efe00 0x7fd2d40f0200 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:35.180 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.175 7fd2da2d6700 1 -- 172.21.2.112:0/2339504087 >> 172.21.2.112:0/2339504087 conn(0x7fd2d40ce3e0 msgr2=0x7fd2d40e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:35.180 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.175 7fd2da2d6700 1 -- 172.21.2.112:0/2339504087 shutdown_connections 2021-02-10T11:30:35.180 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.175 7fd2da2d6700 1 -- 172.21.2.112:0/2339504087 wait complete. 2021-02-10T11:30:35.182 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.190 7fd2da2d6700 1 Processor -- start 2021-02-10T11:30:35.182 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.190 7fd2da2d6700 1 -- start start 2021-02-10T11:30:35.182 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.191 7fd2da2d6700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd2d4105d10 0x7fd2d409eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:35.182 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.191 7fd2da2d6700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd2d4106110 0x7fd2d409f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:35.182 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.191 7fd2da2d6700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd2d40b2a00 0x7fd2d40b2e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:35.182 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.191 7fd2da2d6700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2d409e560 con 0x7fd2d4105d10 2021-02-10T11:30:35.183 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.191 7fd2da2d6700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2d409c090 con 0x7fd2d40b2a00 2021-02-10T11:30:35.183 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.191 7fd2da2d6700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2d40b3420 con 0x7fd2d4106110 2021-02-10T11:30:35.183 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.191 7fd2d3fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd2d4105d10 0x7fd2d409eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:35.183 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.191 7fd2d3fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd2d4105d10 0x7fd2d409eae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:47158/0 (socket says 172.21.2.112:47158) 2021-02-10T11:30:35.183 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.191 7fd2d3fff700 1 -- 172.21.2.112:0/2458857604 learned_addr learned my addr 172.21.2.112:0/2458857604 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:30:35.183 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.191 7fd2d37fe700 1 --2- 172.21.2.112:0/2458857604 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd2d4106110 0x7fd2d409f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:35.183 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.191 7fd2d88f3700 1 --2- 172.21.2.112:0/2458857604 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd2d40b2a00 0x7fd2d40b2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:35.183 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.191 7fd2d3fff700 1 -- 172.21.2.112:0/2458857604 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd2d4106110 msgr2=0x7fd2d409f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:35.183 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.191 7fd2d3fff700 1 --2- 172.21.2.112:0/2458857604 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd2d4106110 0x7fd2d409f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:35.183 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.191 7fd2d3fff700 1 -- 172.21.2.112:0/2458857604 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd2d40b2a00 msgr2=0x7fd2d40b2e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:35.184 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.191 7fd2d3fff700 1 --2- 172.21.2.112:0/2458857604 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd2d40b2a00 0x7fd2d40b2e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:35.184 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.191 7fd2d3fff700 1 -- 172.21.2.112:0/2458857604 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2d40b3640 con 0x7fd2d4105d10 2021-02-10T11:30:35.184 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.191 7fd2d37fe700 1 --2- 172.21.2.112:0/2458857604 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd2d4106110 0x7fd2d409f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:30:35.184 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.191 7fd2d3fff700 1 --2- 172.21.2.112:0/2458857604 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd2d4105d10 0x7fd2d409eae0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 rx=0x7fd2c400a5d0 tx=0x7fd2c400a970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:35.184 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.192 7fd2d0ff9700 1 -- 172.21.2.112:0/2458857604 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd2c4000c10 con 0x7fd2d4105d10 2021-02-10T11:30:35.184 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.192 7fd2d0ff9700 1 -- 172.21.2.112:0/2458857604 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2c40070d0 con 0x7fd2d4105d10 2021-02-10T11:30:35.184 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.192 7fd2da2d6700 1 -- 172.21.2.112:0/2458857604 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2d40b3640 con 0x7fd2d4105d10 2021-02-10T11:30:35.184 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.192 7fd2d0ff9700 1 -- 172.21.2.112:0/2458857604 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd2c4004530 con 0x7fd2d4105d10 2021-02-10T11:30:35.184 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.192 7fd2da2d6700 1 -- 172.21.2.112:0/2458857604 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2d40b3bd0 con 0x7fd2d4105d10 2021-02-10T11:30:35.184 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.192 7fd2d0ff9700 1 -- 172.21.2.112:0/2458857604 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd2c4004320 con 0x7fd2d4105d10 2021-02-10T11:30:35.185 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.192 7fd2d0ff9700 1 --2- 172.21.2.112:0/2458857604 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd2b40186c0 0x7fd2b401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:35.185 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.192 7fd2d0ff9700 1 -- 172.21.2.112:0/2458857604 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fd2c4032640 con 0x7fd2d4105d10 2021-02-10T11:30:35.185 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.192 7fd2d37fe700 1 --2- 172.21.2.112:0/2458857604 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd2b40186c0 0x7fd2b401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:35.185 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.193 7fd2d37fe700 1 --2- 172.21.2.112:0/2458857604 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd2b40186c0 0x7fd2b401abc0 crc :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:35.210 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.223 7fd2da2d6700 1 -- 172.21.2.112:0/2458857604 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2d4105ae0 con 0x7fd2d4105d10 2021-02-10T11:30:35.213 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.226 7fd2d0ff9700 1 -- 172.21.2.112:0/2458857604 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd2c40070d0 con 0x7fd2d4105d10 2021-02-10T11:30:35.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.353 7fd2da2d6700 1 -- 172.21.2.112:0/2458857604 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348611", "target": ["mgr", ""]}) v1 -- 0x7fd2d41a16c0 con 0x7fd2b40186c0 2021-02-10T11:30:35.350 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.362 7fd2d0ff9700 1 -- 172.21.2.112:0/2458857604 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fd2c80009b0 con 0x7fd2b40186c0 2021-02-10T11:30:35.358 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.370 7fd2da2d6700 1 -- 172.21.2.112:0/2458857604 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd2b40186c0 msgr2=0x7fd2b401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:35.358 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.370 7fd2da2d6700 1 --2- 172.21.2.112:0/2458857604 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd2b40186c0 0x7fd2b401abc0 crc :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:35.358 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.370 7fd2da2d6700 1 -- 172.21.2.112:0/2458857604 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd2d4105d10 msgr2=0x7fd2d409eae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:35.358 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.370 7fd2da2d6700 1 --2- 172.21.2.112:0/2458857604 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd2d4105d10 0x7fd2d409eae0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 rx=0x7fd2c400a5d0 tx=0x7fd2c400a970).stop 2021-02-10T11:30:35.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.371 7fd2da2d6700 1 -- 172.21.2.112:0/2458857604 shutdown_connections 2021-02-10T11:30:35.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.371 7fd2da2d6700 1 --2- 172.21.2.112:0/2458857604 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd2b40186c0 0x7fd2b401abc0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:35.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.371 7fd2da2d6700 1 --2- 172.21.2.112:0/2458857604 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd2d40b2a00 0x7fd2d40b2e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:35.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.371 7fd2da2d6700 1 --2- 172.21.2.112:0/2458857604 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd2d4106110 0x7fd2d409f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:35.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.371 7fd2da2d6700 1 --2- 172.21.2.112:0/2458857604 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd2d4105d10 0x7fd2d409eae0 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:35.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.371 7fd2da2d6700 1 -- 172.21.2.112:0/2458857604 >> 172.21.2.112:0/2458857604 conn(0x7fd2d40c36f0 msgr2=0x7fd2d40bd640 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:35.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.371 7fd2da2d6700 1 -- 172.21.2.112:0/2458857604 shutdown_connections 2021-02-10T11:30:35.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.371 7fd2da2d6700 1 -- 172.21.2.112:0/2458857604 wait complete. 2021-02-10T11:30:35.367 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000348612 2021-02-10T11:30:35.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.792 7f275723e700 1 Processor -- start 2021-02-10T11:30:35.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.792 7f275723e700 1 -- start start 2021-02-10T11:30:35.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.792 7f275723e700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f27501b0870 0x7f27500f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:35.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.792 7f275723e700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f27500efe00 0x7f27500f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:35.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.792 7f275723e700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27501b0c70 con 0x7f27501b0470 2021-02-10T11:30:35.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.792 7f275723e700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27500f0b50 con 0x7f27501b0870 2021-02-10T11:30:35.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.792 7f275723e700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27500f0820 con 0x7f27500efe00 2021-02-10T11:30:35.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.793 7f2754859700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f27501b0870 0x7f27500f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:35.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.793 7f275505a700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f27501b0470 0x7f27500ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:46628/0 (socket says 172.21.2.112:46628) 2021-02-10T11:30:35.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.793 7f275505a700 1 -- 172.21.2.112:0/2195313354 learned_addr learned my addr 172.21.2.112:0/2195313354 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:30:35.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.793 7f275585b700 1 --2- 172.21.2.112:0/2195313354 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f27500efe00 0x7f27500f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:35.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.793 7f275585b700 1 -- 172.21.2.112:0/2195313354 >> v2:172.21.2.112:3300/0 conn(0x7f27501b0870 msgr2=0x7f27500f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:35.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.793 7f275585b700 1 --2- 172.21.2.112:0/2195313354 >> v2:172.21.2.112:3300/0 conn(0x7f27501b0870 0x7f27500f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:35.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.793 7f275585b700 1 -- 172.21.2.112:0/2195313354 >> v1:172.21.2.112:6789/0 conn(0x7f27501b0470 legacy=0x7f27500ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:35.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.793 7f275585b700 1 -- 172.21.2.112:0/2195313354 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27500e52e0 con 0x7f27500efe00 2021-02-10T11:30:35.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.793 7f2743fff700 1 -- 172.21.2.112:0/2195313354 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 138309987 0 0) 0x7f2744000ce0 con 0x7f27501b0470 2021-02-10T11:30:35.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.793 7f275585b700 1 --2- 172.21.2.112:0/2195313354 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f27500efe00 0x7f27500f0200 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f2748006e50 tx=0x7f2748009e70).ready entity=mon.2 client_cookie=2f6cb4f156049641 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:35.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.794 7f2743fff700 1 -- 172.21.2.112:0/2195313354 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f27480066f0 con 0x7f27500efe00 2021-02-10T11:30:35.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.794 7f2743fff700 1 -- 172.21.2.112:0/2195313354 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2748003030 con 0x7f27500efe00 2021-02-10T11:30:35.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.794 7f2743fff700 1 -- 172.21.2.112:0/2195313354 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f274800b790 con 0x7f27500efe00 2021-02-10T11:30:35.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.794 7f275723e700 1 -- 172.21.2.112:0/2195313354 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f27500efe00 msgr2=0x7f27500f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:35.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.794 7f275723e700 1 --2- 172.21.2.112:0/2195313354 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f27500efe00 0x7f27500f0200 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f2748006e50 tx=0x7f2748009e70).stop 2021-02-10T11:30:35.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.794 7f275723e700 1 -- 172.21.2.112:0/2195313354 shutdown_connections 2021-02-10T11:30:35.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.794 7f275723e700 1 --2- 172.21.2.112:0/2195313354 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f27500efe00 0x7f27500f0200 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:35.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.794 7f275723e700 1 --2- 172.21.2.112:0/2195313354 >> v2:172.21.2.112:3300/0 conn(0x7f27501b0870 0x7f27500f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:35.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.794 7f275723e700 1 -- 172.21.2.112:0/2195313354 >> 172.21.2.112:0/2195313354 conn(0x7f27500ce3e0 msgr2=0x7f27500e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:35.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.795 7f275723e700 1 -- 172.21.2.112:0/2195313354 shutdown_connections 2021-02-10T11:30:35.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.795 7f275723e700 1 -- 172.21.2.112:0/2195313354 wait complete. 2021-02-10T11:30:35.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.809 7f275723e700 1 Processor -- start 2021-02-10T11:30:35.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.809 7f275723e700 1 -- start start 2021-02-10T11:30:35.802 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.810 7f275723e700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2750190920 0x7f27500b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:35.803 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.810 7f275723e700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f27500b8200 0x7f27500b8600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:35.803 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.810 7f275723e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f27500c1d80 0x7f27500c2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:35.803 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.810 7f275723e700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2750190da0 con 0x7f27500c1d80 2021-02-10T11:30:35.803 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.810 7f275723e700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27500b8c20 con 0x7f2750190920 2021-02-10T11:30:35.803 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.810 7f275723e700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27500b8e10 con 0x7f27500b8200 2021-02-10T11:30:35.803 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.810 7f275585b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f27500c1d80 0x7f27500c2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:35.803 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.810 7f275585b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f27500c1d80 0x7f27500c2180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:47176/0 (socket says 172.21.2.112:47176) 2021-02-10T11:30:35.803 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.810 7f275585b700 1 -- 172.21.2.112:0/1887620486 learned_addr learned my addr 172.21.2.112:0/1887620486 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:30:35.803 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.810 7f275505a700 1 --2- 172.21.2.112:0/1887620486 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2750190920 0x7f27500b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:35.803 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.810 7f2754859700 1 --2- 172.21.2.112:0/1887620486 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f27500b8200 0x7f27500b8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:35.803 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.810 7f275585b700 1 -- 172.21.2.112:0/1887620486 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f27500b8200 msgr2=0x7f27500b8600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:35.803 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.810 7f275585b700 1 --2- 172.21.2.112:0/1887620486 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f27500b8200 0x7f27500b8600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:35.803 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.810 7f275585b700 1 -- 172.21.2.112:0/1887620486 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2750190920 msgr2=0x7f27500b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:35.804 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.810 7f275585b700 1 --2- 172.21.2.112:0/1887620486 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2750190920 0x7f27500b7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:35.804 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.810 7f275585b700 1 -- 172.21.2.112:0/1887620486 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27500c27d0 con 0x7f27500c1d80 2021-02-10T11:30:35.804 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.811 7f275585b700 1 --2- 172.21.2.112:0/1887620486 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f27500c1d80 0x7f27500c2180 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 rx=0x7f2748006d00 tx=0x7f274800bce0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:35.804 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.811 7f2741ffb700 1 -- 172.21.2.112:0/1887620486 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2748009370 con 0x7f27500c1d80 2021-02-10T11:30:35.804 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.811 7f2741ffb700 1 -- 172.21.2.112:0/1887620486 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f274800c2f0 con 0x7f27500c1d80 2021-02-10T11:30:35.804 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.811 7f2741ffb700 1 -- 172.21.2.112:0/1887620486 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f274800c7d0 con 0x7f27500c1d80 2021-02-10T11:30:35.804 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.811 7f275723e700 1 -- 172.21.2.112:0/1887620486 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27500c27d0 con 0x7f27500c1d80 2021-02-10T11:30:35.804 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.811 7f275723e700 1 -- 172.21.2.112:0/1887620486 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27500c2d60 con 0x7f27500c1d80 2021-02-10T11:30:35.804 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.811 7f2741ffb700 1 -- 172.21.2.112:0/1887620486 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2748009310 con 0x7f27500c1d80 2021-02-10T11:30:35.804 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.812 7f2741ffb700 1 --2- 172.21.2.112:0/1887620486 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f27300186c0 0x7f273001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:35.804 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.812 7f2741ffb700 1 -- 172.21.2.112:0/1887620486 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f27480328d0 con 0x7f27500c1d80 2021-02-10T11:30:35.804 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.812 7f275505a700 1 --2- 172.21.2.112:0/1887620486 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f27300186c0 0x7f273001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:35.804 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.812 7f275505a700 1 --2- 172.21.2.112:0/1887620486 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f27300186c0 0x7f273001abc0 crc :-1 s=READY pgs=433 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:35.830 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.842 7f275723e700 1 -- 172.21.2.112:0/1887620486 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2750105ae0 con 0x7f27500c1d80 2021-02-10T11:30:35.832 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.845 7f2741ffb700 1 -- 172.21.2.112:0/1887620486 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f274802cfa0 con 0x7f27500c1d80 2021-02-10T11:30:35.961 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.973 7f275723e700 1 -- 172.21.2.112:0/1887620486 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348612", "target": ["mgr", ""]}) v1 -- 0x7f27500c2d60 con 0x7f27300186c0 2021-02-10T11:30:35.971 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.983 7f2741ffb700 1 -- 172.21.2.112:0/1887620486 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f2744000ef0 con 0x7f27300186c0 2021-02-10T11:30:35.978 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.990 7f275723e700 1 -- 172.21.2.112:0/1887620486 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f27300186c0 msgr2=0x7f273001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:35.978 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.990 7f275723e700 1 --2- 172.21.2.112:0/1887620486 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f27300186c0 0x7f273001abc0 crc :-1 s=READY pgs=433 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:35.978 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.990 7f275723e700 1 -- 172.21.2.112:0/1887620486 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f27500c1d80 msgr2=0x7f27500c2180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:35.978 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.990 7f275723e700 1 --2- 172.21.2.112:0/1887620486 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f27500c1d80 0x7f27500c2180 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 rx=0x7f2748006d00 tx=0x7f274800bce0).stop 2021-02-10T11:30:35.978 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.990 7f275723e700 1 -- 172.21.2.112:0/1887620486 shutdown_connections 2021-02-10T11:30:35.978 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.990 7f275723e700 1 --2- 172.21.2.112:0/1887620486 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f27300186c0 0x7f273001abc0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:35.979 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.990 7f275723e700 1 --2- 172.21.2.112:0/1887620486 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f27500c1d80 0x7f27500c2180 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:35.979 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.990 7f275723e700 1 --2- 172.21.2.112:0/1887620486 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f27500b8200 0x7f27500b8600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:35.979 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.990 7f275723e700 1 --2- 172.21.2.112:0/1887620486 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2750190920 0x7f27500b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:35.979 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.990 7f275723e700 1 -- 172.21.2.112:0/1887620486 >> 172.21.2.112:0/1887620486 conn(0x7f2750094e10 msgr2=0x7f27500a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:35.979 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.991 7f275723e700 1 -- 172.21.2.112:0/1887620486 shutdown_connections 2021-02-10T11:30:35.979 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:35.991 7f275723e700 1 -- 172.21.2.112:0/1887620486 wait complete. 2021-02-10T11:30:35.987 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000348613 2021-02-10T11:30:36.418 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:36.414 7f5b7efe7700 1 Processor -- start 2021-02-10T11:30:36.418 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:36.414 7f5b7efe7700 1 -- start start 2021-02-10T11:30:36.419 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:36.415 7f5b7efe7700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5b781b0470 0x7f5b781b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:36.419 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:36.415 7f5b7efe7700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f5b780f20e0 0x7f5b780f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:36.419 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:36.415 7f5b7efe7700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b780f0b50 con 0x7f5b780ef610 2021-02-10T11:30:36.419 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:36.415 7f5b7efe7700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b780f0820 con 0x7f5b780f20e0 2021-02-10T11:30:36.419 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:36.415 7f5b7efe7700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b780f24e0 con 0x7f5b781b0470 2021-02-10T11:30:36.419 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:36.415 7f5b77fff700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f5b780ef610 0x7f5b780efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:46642/0 (socket says 172.21.2.112:46642) 2021-02-10T11:30:36.419 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:36.415 7f5b77fff700 1 -- 172.21.2.112:0/2033338607 learned_addr learned my addr 172.21.2.112:0/2033338607 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:30:36.419 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:36.415 7f5b7d604700 1 --2- 172.21.2.112:0/2033338607 >> v2:172.21.2.112:3300/0 conn(0x7f5b780f20e0 0x7f5b780f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:36.419 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:36.415 7f5b7ce03700 1 --2- 172.21.2.112:0/2033338607 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5b781b0470 0x7f5b781b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:36.419 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:36.415 7f5b777fe700 1 -- 172.21.2.112:0/2033338607 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 901201792 0 0) 0x7f5b6c001180 con 0x7f5b780ef610 2021-02-10T11:30:36.419 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:36.415 7f5b777fe700 1 -- 172.21.2.112:0/2033338607 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b60002fe0 con 0x7f5b780ef610 2021-02-10T11:30:36.419 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:36.415 7f5b7d604700 1 -- 172.21.2.112:0/2033338607 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5b781b0470 msgr2=0x7f5b781b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:36.419 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:36.415 7f5b7d604700 1 --2- 172.21.2.112:0/2033338607 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5b781b0470 0x7f5b781b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:36.420 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:36.415 7f5b7d604700 1 -- 172.21.2.112:0/2033338607 >> v1:172.21.2.112:6789/0 conn(0x7f5b780ef610 legacy=0x7f5b780efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:36.420 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:36.415 7f5b7d604700 1 -- 172.21.2.112:0/2033338607 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b780e52e0 con 0x7f5b780f20e0 2021-02-10T11:30:36.420 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:36.415 7f5b7ce03700 1 --2- 172.21.2.112:0/2033338607 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5b781b0470 0x7f5b781b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:30:36.420 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:36.416 7f5b7d604700 1 --2- 172.21.2.112:0/2033338607 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5b780f20e0 0x7f5b780f0200 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 rx=0x7f5b70006c30 tx=0x7f5b70009e10).ready entity=mon.0 client_cookie=a3c8f9a62fdad12d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:36.420 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:36.416 7f5b777fe700 1 -- 172.21.2.112:0/2033338607 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5b7000b300 con 0x7f5b780f20e0 2021-02-10T11:30:36.420 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:36.416 7f5b777fe700 1 -- 172.21.2.112:0/2033338607 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b7000b640 con 0x7f5b780f20e0 2021-02-10T11:30:36.420 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:36.416 7f5b777fe700 1 -- 172.21.2.112:0/2033338607 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5b7000bad0 con 0x7f5b780f20e0 2021-02-10T11:30:36.420 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:36.416 7f5b7efe7700 1 -- 172.21.2.112:0/2033338607 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5b780f20e0 msgr2=0x7f5b780f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:36.420 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:36.416 7f5b7efe7700 1 --2- 172.21.2.112:0/2033338607 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5b780f20e0 0x7f5b780f0200 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 rx=0x7f5b70006c30 tx=0x7f5b70009e10).stop 2021-02-10T11:30:36.420 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:36.417 7f5b7efe7700 1 -- 172.21.2.112:0/2033338607 shutdown_connections 2021-02-10T11:30:36.420 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:36.417 7f5b7efe7700 1 --2- 172.21.2.112:0/2033338607 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5b780f20e0 0x7f5b780f0200 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:36.420 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:36.417 7f5b7efe7700 1 --2- 172.21.2.112:0/2033338607 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5b781b0470 0x7f5b781b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:36.420 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:36.417 7f5b7efe7700 1 -- 172.21.2.112:0/2033338607 >> 172.21.2.112:0/2033338607 conn(0x7f5b780ce3e0 msgr2=0x7f5b780e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:36.421 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:36.417 7f5b7efe7700 1 -- 172.21.2.112:0/2033338607 shutdown_connections 2021-02-10T11:30:36.421 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:36.417 7f5b7efe7700 1 -- 172.21.2.112:0/2033338607 wait complete. 2021-02-10T11:30:36.422 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:36.431 7f5b7efe7700 1 Processor -- start 2021-02-10T11:30:36.422 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:36.431 7f5b7efe7700 1 -- start start 2021-02-10T11:30:36.422 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:36.432 7f5b7efe7700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5b780b2eb0 0x7f5b780b2330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:36.422 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:36.432 7f5b7efe7700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5b780b2950 0x7f5b780b3380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:36.423 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:36.432 7f5b7efe7700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5b780b39a0 0x7f5b7809d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:36.423 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:36.432 7f5b7efe7700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b7809cdc0 con 0x7f5b780b2eb0 2021-02-10T11:30:36.423 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:36.432 7f5b7efe7700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b7809d960 con 0x7f5b780b39a0 2021-02-10T11:30:36.423 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:36.432 7f5b7efe7700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b7809db80 con 0x7f5b780b2950 2021-02-10T11:30:36.423 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:36.432 7f5b7ce03700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5b780b2eb0 0x7f5b780b2330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:36.423 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:36.432 7f5b7ce03700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5b780b2eb0 0x7f5b780b2330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:47184/0 (socket says 172.21.2.112:47184) 2021-02-10T11:30:36.423 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:36.432 7f5b7ce03700 1 -- 172.21.2.112:0/4261891309 learned_addr learned my addr 172.21.2.112:0/4261891309 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:30:36.423 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:36.432 7f5b77fff700 1 --2- 172.21.2.112:0/4261891309 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5b780b2950 0x7f5b780b3380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:36.423 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:36.432 7f5b7d604700 1 --2- 172.21.2.112:0/4261891309 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5b780b39a0 0x7f5b7809d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:36.423 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:36.432 7f5b7ce03700 1 -- 172.21.2.112:0/4261891309 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5b780b2950 msgr2=0x7f5b780b3380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:36.423 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:36.432 7f5b7ce03700 1 --2- 172.21.2.112:0/4261891309 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5b780b2950 0x7f5b780b3380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:36.423 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:36.432 7f5b7ce03700 1 -- 172.21.2.112:0/4261891309 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5b780b39a0 msgr2=0x7f5b7809d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:36.423 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:36.432 7f5b7ce03700 1 --2- 172.21.2.112:0/4261891309 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5b780b39a0 0x7f5b7809d340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:36.424 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:36.432 7f5b7ce03700 1 -- 172.21.2.112:0/4261891309 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b7809dda0 con 0x7f5b780b2eb0 2021-02-10T11:30:36.424 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:36.432 7f5b7ce03700 1 --2- 172.21.2.112:0/4261891309 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5b780b2eb0 0x7f5b780b2330 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 rx=0x7f5b68006f20 tx=0x7f5b6800b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:36.424 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:36.433 7f5b757fa700 1 -- 172.21.2.112:0/4261891309 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5b6800bd80 con 0x7f5b780b2eb0 2021-02-10T11:30:36.424 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:36.433 7f5b757fa700 1 -- 172.21.2.112:0/4261891309 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b68000c90 con 0x7f5b780b2eb0 2021-02-10T11:30:36.424 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:36.433 7f5b757fa700 1 -- 172.21.2.112:0/4261891309 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5b68002220 con 0x7f5b780b2eb0 2021-02-10T11:30:36.424 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:36.433 7f5b7efe7700 1 -- 172.21.2.112:0/4261891309 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b7809dda0 con 0x7f5b780b2eb0 2021-02-10T11:30:36.424 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:36.433 7f5b7efe7700 1 -- 172.21.2.112:0/4261891309 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b7809dda0 con 0x7f5b780b2eb0 2021-02-10T11:30:36.424 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:36.433 7f5b757fa700 1 -- 172.21.2.112:0/4261891309 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5b6800a000 con 0x7f5b780b2eb0 2021-02-10T11:30:36.424 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:36.433 7f5b757fa700 1 --2- 172.21.2.112:0/4261891309 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5b58018730 0x7f5b5801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:36.424 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:36.433 7f5b757fa700 1 -- 172.21.2.112:0/4261891309 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f5b68032640 con 0x7f5b780b2eb0 2021-02-10T11:30:36.424 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:36.433 7f5b77fff700 1 --2- 172.21.2.112:0/4261891309 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5b58018730 0x7f5b5801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:36.424 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:36.434 7f5b77fff700 1 --2- 172.21.2.112:0/4261891309 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5b58018730 0x7f5b5801ac10 crc :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:36.451 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:36.464 7f5b7efe7700 1 -- 172.21.2.112:0/4261891309 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b78105b30 con 0x7f5b780b2eb0 2021-02-10T11:30:36.454 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:36.466 7f5b757fa700 1 -- 172.21.2.112:0/4261891309 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5b68006bb0 con 0x7f5b780b2eb0 2021-02-10T11:30:36.583 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:36.596 7f5b7efe7700 1 -- 172.21.2.112:0/4261891309 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348613", "target": ["mgr", ""]}) v1 -- 0x7f5b780f0b50 con 0x7f5b58018730 2021-02-10T11:30:36.594 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:36.607 7f5b757fa700 1 -- 172.21.2.112:0/4261891309 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f5b6c007cc0 con 0x7f5b58018730 2021-02-10T11:30:36.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:36.613 7f5b7efe7700 1 -- 172.21.2.112:0/4261891309 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5b58018730 msgr2=0x7f5b5801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:36.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:36.613 7f5b7efe7700 1 --2- 172.21.2.112:0/4261891309 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5b58018730 0x7f5b5801ac10 crc :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:36.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:36.613 7f5b7efe7700 1 -- 172.21.2.112:0/4261891309 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5b780b2eb0 msgr2=0x7f5b780b2330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:36.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:36.613 7f5b7efe7700 1 --2- 172.21.2.112:0/4261891309 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5b780b2eb0 0x7f5b780b2330 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 rx=0x7f5b68006f20 tx=0x7f5b6800b990).stop 2021-02-10T11:30:36.602 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:36.614 7f5b7efe7700 1 -- 172.21.2.112:0/4261891309 shutdown_connections 2021-02-10T11:30:36.602 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:36.614 7f5b7efe7700 1 --2- 172.21.2.112:0/4261891309 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5b58018730 0x7f5b5801ac10 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:36.602 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:36.614 7f5b7efe7700 1 --2- 172.21.2.112:0/4261891309 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5b780b39a0 0x7f5b7809d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:36.602 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:36.614 7f5b7efe7700 1 --2- 172.21.2.112:0/4261891309 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5b780b2950 0x7f5b780b3380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:36.602 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:36.614 7f5b7efe7700 1 --2- 172.21.2.112:0/4261891309 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5b780b2eb0 0x7f5b780b2330 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:36.602 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:36.614 7f5b7efe7700 1 -- 172.21.2.112:0/4261891309 >> 172.21.2.112:0/4261891309 conn(0x7f5b7809fc90 msgr2=0x7f5b78094ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:36.602 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:36.614 7f5b7efe7700 1 -- 172.21.2.112:0/4261891309 shutdown_connections 2021-02-10T11:30:36.602 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:36.614 7f5b7efe7700 1 -- 172.21.2.112:0/4261891309 wait complete. 2021-02-10T11:30:36.610 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000348614 2021-02-10T11:30:37.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.036 7f78d4db4700 1 Processor -- start 2021-02-10T11:30:37.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.036 7f78d4db4700 1 -- start start 2021-02-10T11:30:37.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.037 7f78d4db4700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f78d01b0470 0x7f78d01b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:37.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.037 7f78d4db4700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f78d00f20e0 0x7f78d00f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:37.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.037 7f78d4db4700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78d00f0b50 con 0x7f78d00ef610 2021-02-10T11:30:37.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.037 7f78d4db4700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78d00f0820 con 0x7f78d00f20e0 2021-02-10T11:30:37.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.037 7f78d4db4700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78d00f24e0 con 0x7f78d01b0470 2021-02-10T11:30:37.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.037 7f78ceb13700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f78d01b0470 0x7f78d01b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:37.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.037 7f78cf314700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f78d00f20e0 0x7f78d00f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:37.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.037 7f78c6312700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7f78d00ef610 0x7f78d00efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:40214/0 (socket says 172.21.2.112:40214) 2021-02-10T11:30:37.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.037 7f78cf314700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f78d00f20e0 0x7f78d00f0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:47196/0 (socket says 172.21.2.112:47196) 2021-02-10T11:30:37.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.037 7f78ceb13700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f78d01b0470 0x7f78d01b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3300/0 says I am v2:172.21.2.112:49472/0 (socket says 172.21.2.112:49472) 2021-02-10T11:30:37.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.037 7f78ceb13700 1 -- 172.21.2.112:0/3976972351 learned_addr learned my addr 172.21.2.112:0/3976972351 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:30:37.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.037 7f78cf314700 1 -- 172.21.2.112:0/3976972351 >> v2:172.21.2.120:3300/0 conn(0x7f78d01b0470 msgr2=0x7f78d01b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:37.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.037 7f78cf314700 1 --2- 172.21.2.112:0/3976972351 >> v2:172.21.2.120:3300/0 conn(0x7f78d01b0470 0x7f78d01b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:37.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.037 7f78cf314700 1 -- 172.21.2.112:0/3976972351 >> v1:172.21.2.120:6789/0 conn(0x7f78d00ef610 legacy=0x7f78d00efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:37.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.037 7f78cf314700 1 -- 172.21.2.112:0/3976972351 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78d00e52e0 con 0x7f78d00f20e0 2021-02-10T11:30:37.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.037 7f78cf314700 1 --2- 172.21.2.112:0/3976972351 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f78d00f20e0 0x7f78d00f0200 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 rx=0x7f78c8006e30 tx=0x7f78c8009e60).ready entity=mon.0 client_cookie=54e394e9fefe2750 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:37.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.038 7f78ce312700 1 -- 172.21.2.112:0/3976972351 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f78c8006a50 con 0x7f78d00f20e0 2021-02-10T11:30:37.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.038 7f78ce312700 1 -- 172.21.2.112:0/3976972351 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78c800b300 con 0x7f78d00f20e0 2021-02-10T11:30:37.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.038 7f78ce312700 1 -- 172.21.2.112:0/3976972351 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f78c800b750 con 0x7f78d00f20e0 2021-02-10T11:30:37.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.038 7f78d4db4700 1 -- 172.21.2.112:0/3976972351 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f78d00f20e0 msgr2=0x7f78d00f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:37.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.038 7f78d4db4700 1 --2- 172.21.2.112:0/3976972351 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f78d00f20e0 0x7f78d00f0200 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 rx=0x7f78c8006e30 tx=0x7f78c8009e60).stop 2021-02-10T11:30:37.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.038 7f78d4db4700 1 -- 172.21.2.112:0/3976972351 shutdown_connections 2021-02-10T11:30:37.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.038 7f78d4db4700 1 --2- 172.21.2.112:0/3976972351 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f78d00f20e0 0x7f78d00f0200 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:37.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.038 7f78d4db4700 1 --2- 172.21.2.112:0/3976972351 >> v2:172.21.2.120:3300/0 conn(0x7f78d01b0470 0x7f78d01b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:37.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.038 7f78d4db4700 1 -- 172.21.2.112:0/3976972351 >> 172.21.2.112:0/3976972351 conn(0x7f78d00ce3e0 msgr2=0x7f78d00e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:37.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.038 7f78d4db4700 1 -- 172.21.2.112:0/3976972351 shutdown_connections 2021-02-10T11:30:37.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.038 7f78d4db4700 1 -- 172.21.2.112:0/3976972351 wait complete. 2021-02-10T11:30:37.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.053 7f78d4db4700 1 Processor -- start 2021-02-10T11:30:37.045 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.053 7f78d4db4700 1 -- start start 2021-02-10T11:30:37.045 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.054 7f78d4db4700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f78d00a0420 0x7f78d00a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:37.045 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.054 7f78d4db4700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f78d0189d00 0x7f78d018a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:37.045 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.054 7f78d4db4700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f78d018a720 0x7f78d00b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:37.045 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.054 7f78d4db4700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78d0189780 con 0x7f78d00a0420 2021-02-10T11:30:37.045 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.054 7f78d4db4700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78d018ab20 con 0x7f78d018a720 2021-02-10T11:30:37.045 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.054 7f78d4db4700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78d00a1140 con 0x7f78d0189d00 2021-02-10T11:30:37.045 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.054 7f78ceb13700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f78d00a0420 0x7f78d00a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:37.045 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.054 7f78ceb13700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f78d00a0420 0x7f78d00a0820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:47200/0 (socket says 172.21.2.112:47200) 2021-02-10T11:30:37.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.054 7f78ceb13700 1 -- 172.21.2.112:0/4246604078 learned_addr learned my addr 172.21.2.112:0/4246604078 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:30:37.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.054 7f78c6312700 1 --2- 172.21.2.112:0/4246604078 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f78d0189d00 0x7f78d018a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:37.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.054 7f78cf314700 1 --2- 172.21.2.112:0/4246604078 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f78d018a720 0x7f78d00b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:37.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.055 7f78ceb13700 1 -- 172.21.2.112:0/4246604078 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f78d0189d00 msgr2=0x7f78d018a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:37.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.055 7f78ceb13700 1 --2- 172.21.2.112:0/4246604078 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f78d0189d00 0x7f78d018a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:37.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.055 7f78ceb13700 1 -- 172.21.2.112:0/4246604078 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f78d018a720 msgr2=0x7f78d00b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:37.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.055 7f78ceb13700 1 --2- 172.21.2.112:0/4246604078 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f78d018a720 0x7f78d00b7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:37.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.055 7f78ceb13700 1 -- 172.21.2.112:0/4246604078 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78d00b8200 con 0x7f78d00a0420 2021-02-10T11:30:37.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.055 7f78ceb13700 1 --2- 172.21.2.112:0/4246604078 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f78d00a0420 0x7f78d00a0820 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 rx=0x7f78c000b5e0 tx=0x7f78c000b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:37.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.055 7f78c7fff700 1 -- 172.21.2.112:0/4246604078 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f78c000bd70 con 0x7f78d00a0420 2021-02-10T11:30:37.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.055 7f78c7fff700 1 -- 172.21.2.112:0/4246604078 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78c0002030 con 0x7f78d00a0420 2021-02-10T11:30:37.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.055 7f78c7fff700 1 -- 172.21.2.112:0/4246604078 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f78c0002400 con 0x7f78d00a0420 2021-02-10T11:30:37.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.055 7f78d4db4700 1 -- 172.21.2.112:0/4246604078 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78d00b8200 con 0x7f78d00a0420 2021-02-10T11:30:37.047 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.055 7f78d4db4700 1 -- 172.21.2.112:0/4246604078 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78d00b8790 con 0x7f78d00a0420 2021-02-10T11:30:37.047 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.056 7f78c7fff700 1 -- 172.21.2.112:0/4246604078 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f78c0000c90 con 0x7f78d00a0420 2021-02-10T11:30:37.047 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.056 7f78c7fff700 1 --2- 172.21.2.112:0/4246604078 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f78b40186c0 0x7f78b401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:37.047 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.056 7f78c7fff700 1 -- 172.21.2.112:0/4246604078 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f78c0032800 con 0x7f78d00a0420 2021-02-10T11:30:37.047 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.056 7f78c6312700 1 --2- 172.21.2.112:0/4246604078 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f78b40186c0 0x7f78b401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:37.047 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.056 7f78c6312700 1 --2- 172.21.2.112:0/4246604078 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f78b40186c0 0x7f78b401abc0 crc :-1 s=READY pgs=435 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:37.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.085 7f78d4db4700 1 -- 172.21.2.112:0/4246604078 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78d0105ae0 con 0x7f78d00a0420 2021-02-10T11:30:37.075 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.088 7f78c7fff700 1 -- 172.21.2.112:0/4246604078 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f78c001f540 con 0x7f78d00a0420 2021-02-10T11:30:37.203 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.216 7f78d4db4700 1 -- 172.21.2.112:0/4246604078 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348614", "target": ["mgr", ""]}) v1 -- 0x7f78d0189780 con 0x7f78b40186c0 2021-02-10T11:30:37.213 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.226 7f78c7fff700 1 -- 172.21.2.112:0/4246604078 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f78b8001a00 con 0x7f78b40186c0 2021-02-10T11:30:37.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.233 7f78d4db4700 1 -- 172.21.2.112:0/4246604078 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f78b40186c0 msgr2=0x7f78b401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:37.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.233 7f78d4db4700 1 --2- 172.21.2.112:0/4246604078 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f78b40186c0 0x7f78b401abc0 crc :-1 s=READY pgs=435 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:37.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.233 7f78d4db4700 1 -- 172.21.2.112:0/4246604078 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f78d00a0420 msgr2=0x7f78d00a0820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:37.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.233 7f78d4db4700 1 --2- 172.21.2.112:0/4246604078 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f78d00a0420 0x7f78d00a0820 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 rx=0x7f78c000b5e0 tx=0x7f78c000b980).stop 2021-02-10T11:30:37.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.234 7f78d4db4700 1 -- 172.21.2.112:0/4246604078 shutdown_connections 2021-02-10T11:30:37.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.234 7f78d4db4700 1 --2- 172.21.2.112:0/4246604078 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f78b40186c0 0x7f78b401abc0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:37.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.234 7f78d4db4700 1 --2- 172.21.2.112:0/4246604078 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f78d018a720 0x7f78d00b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:37.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.234 7f78d4db4700 1 --2- 172.21.2.112:0/4246604078 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f78d0189d00 0x7f78d018a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:37.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.234 7f78d4db4700 1 --2- 172.21.2.112:0/4246604078 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f78d00a0420 0x7f78d00a0820 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:37.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.234 7f78d4db4700 1 -- 172.21.2.112:0/4246604078 >> 172.21.2.112:0/4246604078 conn(0x7f78d0094e10 msgr2=0x7f78d00a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:37.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.234 7f78d4db4700 1 -- 172.21.2.112:0/4246604078 shutdown_connections 2021-02-10T11:30:37.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.234 7f78d4db4700 1 -- 172.21.2.112:0/4246604078 wait complete. 2021-02-10T11:30:37.230 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000348615 2021-02-10T11:30:37.661 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.656 7fec5b17f700 1 Processor -- start 2021-02-10T11:30:37.661 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.657 7fec5b17f700 1 -- start start 2021-02-10T11:30:37.661 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.657 7fec5b17f700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fec541b0870 0x7fec540f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:37.661 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.657 7fec5b17f700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fec540efe00 0x7fec540f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:37.661 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.657 7fec5b17f700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec541b0c70 con 0x7fec541b0470 2021-02-10T11:30:37.661 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.657 7fec5b17f700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec540f0b50 con 0x7fec541b0870 2021-02-10T11:30:37.661 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.657 7fec5b17f700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec540f0820 con 0x7fec540efe00 2021-02-10T11:30:37.661 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.657 7fec58f9b700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fec541b0470 0x7fec540ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:46668/0 (socket says 172.21.2.112:46668) 2021-02-10T11:30:37.661 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.657 7fec58f9b700 1 -- 172.21.2.112:0/3486031933 learned_addr learned my addr 172.21.2.112:0/3486031933 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:30:37.662 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.657 7fec53fff700 1 --2- 172.21.2.112:0/3486031933 >> v2:172.21.2.120:3300/0 conn(0x7fec541b0870 0x7fec540f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:37.662 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.657 7fec5979c700 1 --2- 172.21.2.112:0/3486031933 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fec540efe00 0x7fec540f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:37.662 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.658 7fec537fe700 1 -- 172.21.2.112:0/3486031933 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2513511620 0 0) 0x7fec3c001180 con 0x7fec541b0470 2021-02-10T11:30:37.662 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.658 7fec53fff700 1 -- 172.21.2.112:0/3486031933 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fec540efe00 msgr2=0x7fec540f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:37.662 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.658 7fec53fff700 1 --2- 172.21.2.112:0/3486031933 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fec540efe00 0x7fec540f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:37.662 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.658 7fec53fff700 1 -- 172.21.2.112:0/3486031933 >> v1:172.21.2.112:6789/0 conn(0x7fec541b0470 legacy=0x7fec540ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:37.662 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.658 7fec53fff700 1 -- 172.21.2.112:0/3486031933 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec540e52e0 con 0x7fec541b0870 2021-02-10T11:30:37.662 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.658 7fec53fff700 1 --2- 172.21.2.112:0/3486031933 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fec541b0870 0x7fec540f20e0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 rx=0x7fec44006650 tx=0x7fec44009a80).ready entity=mon.1 client_cookie=77210b1258396d4a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:37.662 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.658 7fec537fe700 1 -- 172.21.2.112:0/3486031933 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fec440009b0 con 0x7fec541b0870 2021-02-10T11:30:37.662 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.658 7fec537fe700 1 -- 172.21.2.112:0/3486031933 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec4400c0c0 con 0x7fec541b0870 2021-02-10T11:30:37.662 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.658 7fec537fe700 1 -- 172.21.2.112:0/3486031933 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fec4400c5e0 con 0x7fec541b0870 2021-02-10T11:30:37.662 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.658 7fec5b17f700 1 -- 172.21.2.112:0/3486031933 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fec541b0870 msgr2=0x7fec540f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:37.662 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.658 7fec5b17f700 1 --2- 172.21.2.112:0/3486031933 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fec541b0870 0x7fec540f20e0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 rx=0x7fec44006650 tx=0x7fec44009a80).stop 2021-02-10T11:30:37.663 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.659 7fec5b17f700 1 -- 172.21.2.112:0/3486031933 shutdown_connections 2021-02-10T11:30:37.663 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.659 7fec5b17f700 1 --2- 172.21.2.112:0/3486031933 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fec540efe00 0x7fec540f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:37.663 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.659 7fec5b17f700 1 --2- 172.21.2.112:0/3486031933 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fec541b0870 0x7fec540f20e0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:37.663 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.659 7fec5b17f700 1 -- 172.21.2.112:0/3486031933 >> 172.21.2.112:0/3486031933 conn(0x7fec540ce3e0 msgr2=0x7fec540e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:37.663 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.659 7fec5b17f700 1 -- 172.21.2.112:0/3486031933 shutdown_connections 2021-02-10T11:30:37.663 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.659 7fec5b17f700 1 -- 172.21.2.112:0/3486031933 wait complete. 2021-02-10T11:30:37.665 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.674 7fec5b17f700 1 Processor -- start 2021-02-10T11:30:37.665 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.674 7fec5b17f700 1 -- start start 2021-02-10T11:30:37.665 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.674 7fec5b17f700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fec540bf930 0x7fec540bfd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:37.665 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.674 7fec5b17f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fec540906c0 0x7fec54090ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:37.665 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.674 7fec5b17f700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fec540c0960 0x7fec540c0d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:37.665 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.674 7fec5b17f700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec540915c0 con 0x7fec540906c0 2021-02-10T11:30:37.665 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.674 7fec5b17f700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec5419c130 con 0x7fec540bf930 2021-02-10T11:30:37.665 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.674 7fec5b17f700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec540910e0 con 0x7fec540c0960 2021-02-10T11:30:37.665 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.675 7fec53fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fec540906c0 0x7fec54090ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:37.665 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.675 7fec53fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fec540906c0 0x7fec54090ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:47214/0 (socket says 172.21.2.112:47214) 2021-02-10T11:30:37.665 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.675 7fec53fff700 1 -- 172.21.2.112:0/4030209354 learned_addr learned my addr 172.21.2.112:0/4030209354 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:30:37.665 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.675 7fec58f9b700 1 --2- 172.21.2.112:0/4030209354 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fec540bf930 0x7fec540bfd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:37.666 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.675 7fec5979c700 1 --2- 172.21.2.112:0/4030209354 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fec540c0960 0x7fec540c0d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:37.666 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.675 7fec58f9b700 1 -- 172.21.2.112:0/4030209354 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fec540c0960 msgr2=0x7fec540c0d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:37.666 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.675 7fec58f9b700 1 --2- 172.21.2.112:0/4030209354 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fec540c0960 0x7fec540c0d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:37.666 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.675 7fec58f9b700 1 -- 172.21.2.112:0/4030209354 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fec540906c0 msgr2=0x7fec54090ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:37.666 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.675 7fec58f9b700 1 --2- 172.21.2.112:0/4030209354 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fec540906c0 0x7fec54090ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:37.666 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.675 7fec58f9b700 1 -- 172.21.2.112:0/4030209354 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec540c1380 con 0x7fec540bf930 2021-02-10T11:30:37.666 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.675 7fec53fff700 1 --2- 172.21.2.112:0/4030209354 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fec540906c0 0x7fec54090ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:30:37.666 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.675 7fec58f9b700 1 --2- 172.21.2.112:0/4030209354 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fec540bf930 0x7fec540bfd30 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7fec3c001e30 tx=0x7fec3c00a800).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:37.666 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.675 7fec517fa700 1 -- 172.21.2.112:0/4030209354 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fec3c0015d0 con 0x7fec540bf930 2021-02-10T11:30:37.666 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.675 7fec517fa700 1 -- 172.21.2.112:0/4030209354 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec3c0070d0 con 0x7fec540bf930 2021-02-10T11:30:37.666 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.675 7fec5b17f700 1 -- 172.21.2.112:0/4030209354 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec540c1380 con 0x7fec540bf930 2021-02-10T11:30:37.666 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.675 7fec5b17f700 1 -- 172.21.2.112:0/4030209354 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec540c1910 con 0x7fec540bf930 2021-02-10T11:30:37.666 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.675 7fec517fa700 1 -- 172.21.2.112:0/4030209354 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fec3c00ade0 con 0x7fec540bf930 2021-02-10T11:30:37.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.676 7fec517fa700 1 -- 172.21.2.112:0/4030209354 <== mon.1 v2:172.21.2.120:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fec3c009030 con 0x7fec540bf930 2021-02-10T11:30:37.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.676 7fec517fa700 1 --2- 172.21.2.112:0/4030209354 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fec380186c0 0x7fec3801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:37.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.676 7fec517fa700 1 -- 172.21.2.112:0/4030209354 <== mon.1 v2:172.21.2.120:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fec3c032500 con 0x7fec540bf930 2021-02-10T11:30:37.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.676 7fec53fff700 1 --2- 172.21.2.112:0/4030209354 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fec380186c0 0x7fec3801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:37.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.677 7fec53fff700 1 --2- 172.21.2.112:0/4030209354 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fec380186c0 0x7fec3801abc0 crc :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:37.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.706 7fec5b17f700 1 -- 172.21.2.112:0/4030209354 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec54105ae0 con 0x7fec540bf930 2021-02-10T11:30:37.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.711 7fec517fa700 1 -- 172.21.2.112:0/4030209354 <== mon.1 v2:172.21.2.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fec3c001180 con 0x7fec540bf930 2021-02-10T11:30:37.829 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.841 7fec5b17f700 1 -- 172.21.2.112:0/4030209354 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348615", "target": ["mgr", ""]}) v1 -- 0x7fec540c1910 con 0x7fec380186c0 2021-02-10T11:30:37.838 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.851 7fec517fa700 1 -- 172.21.2.112:0/4030209354 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fec44000960 con 0x7fec380186c0 2021-02-10T11:30:37.845 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.857 7fec5b17f700 1 -- 172.21.2.112:0/4030209354 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fec380186c0 msgr2=0x7fec3801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:37.845 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.857 7fec5b17f700 1 --2- 172.21.2.112:0/4030209354 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fec380186c0 0x7fec3801abc0 crc :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:37.845 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.857 7fec5b17f700 1 -- 172.21.2.112:0/4030209354 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fec540bf930 msgr2=0x7fec540bfd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:37.845 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.857 7fec5b17f700 1 --2- 172.21.2.112:0/4030209354 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fec540bf930 0x7fec540bfd30 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7fec3c001e30 tx=0x7fec3c00a800).stop 2021-02-10T11:30:37.846 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.858 7fec5b17f700 1 -- 172.21.2.112:0/4030209354 shutdown_connections 2021-02-10T11:30:37.846 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.858 7fec5b17f700 1 --2- 172.21.2.112:0/4030209354 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fec380186c0 0x7fec3801abc0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:37.846 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.858 7fec5b17f700 1 --2- 172.21.2.112:0/4030209354 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fec540c0960 0x7fec540c0d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:37.846 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.858 7fec5b17f700 1 --2- 172.21.2.112:0/4030209354 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fec540906c0 0x7fec54090ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:37.846 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.858 7fec5b17f700 1 --2- 172.21.2.112:0/4030209354 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fec540bf930 0x7fec540bfd30 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:37.846 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.858 7fec5b17f700 1 -- 172.21.2.112:0/4030209354 >> 172.21.2.112:0/4030209354 conn(0x7fec54094e10 msgr2=0x7fec540a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:37.846 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.858 7fec5b17f700 1 -- 172.21.2.112:0/4030209354 shutdown_connections 2021-02-10T11:30:37.846 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:37.858 7fec5b17f700 1 -- 172.21.2.112:0/4030209354 wait complete. 2021-02-10T11:30:37.855 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000348616 2021-02-10T11:30:38.284 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.280 7f293b06f700 1 Processor -- start 2021-02-10T11:30:38.284 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.280 7f293b06f700 1 -- start start 2021-02-10T11:30:38.284 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.281 7f293b06f700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f29341b0470 0x7f29341b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:38.284 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.281 7f293b06f700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f29340f20e0 0x7f29340f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:38.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.281 7f293b06f700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29340f0b50 con 0x7f29340ef610 2021-02-10T11:30:38.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.281 7f293b06f700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29340f0820 con 0x7f29341b0470 2021-02-10T11:30:38.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.281 7f293b06f700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29340f24e0 con 0x7f29340f20e0 2021-02-10T11:30:38.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.281 7f2938e8b700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f29341b0470 0x7f29341b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:38.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.281 7f2933fff700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f29340ef610 0x7f29340efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:46680/0 (socket says 172.21.2.112:46680) 2021-02-10T11:30:38.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.281 7f2933fff700 1 -- 172.21.2.112:0/2787653374 learned_addr learned my addr 172.21.2.112:0/2787653374 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:30:38.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.281 7f293968c700 1 --2- 172.21.2.112:0/2787653374 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f29340f20e0 0x7f29340f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:38.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.281 7f293968c700 1 -- 172.21.2.112:0/2787653374 >> v2:172.21.2.120:3300/0 conn(0x7f29341b0470 msgr2=0x7f29341b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:38.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.281 7f293968c700 1 --2- 172.21.2.112:0/2787653374 >> v2:172.21.2.120:3300/0 conn(0x7f29341b0470 0x7f29341b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:38.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.281 7f29337fe700 1 -- 172.21.2.112:0/2787653374 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 692673837 0 0) 0x7f2924001060 con 0x7f29340ef610 2021-02-10T11:30:38.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.281 7f293968c700 1 -- 172.21.2.112:0/2787653374 >> v1:172.21.2.112:6789/0 conn(0x7f29340ef610 legacy=0x7f29340efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:38.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.281 7f293968c700 1 -- 172.21.2.112:0/2787653374 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29340e52e0 con 0x7f29340f20e0 2021-02-10T11:30:38.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.281 7f2938e8b700 1 --2- 172.21.2.112:0/2787653374 >> v2:172.21.2.120:3300/0 conn(0x7f29341b0470 0x7f29341b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:30:38.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.282 7f293968c700 1 --2- 172.21.2.112:0/2787653374 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f29340f20e0 0x7f29340f0200 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f292c006ad0 tx=0x7f292c009ad0).ready entity=mon.2 client_cookie=4d9fceb0b6f766bd server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:38.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.282 7f29337fe700 1 -- 172.21.2.112:0/2787653374 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f292c0022e0 con 0x7f29340f20e0 2021-02-10T11:30:38.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.282 7f29337fe700 1 -- 172.21.2.112:0/2787653374 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f292c006410 con 0x7f29340f20e0 2021-02-10T11:30:38.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.282 7f29337fe700 1 -- 172.21.2.112:0/2787653374 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f292c002620 con 0x7f29340f20e0 2021-02-10T11:30:38.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.282 7f293b06f700 1 -- 172.21.2.112:0/2787653374 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f29340f20e0 msgr2=0x7f29340f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:38.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.282 7f293b06f700 1 --2- 172.21.2.112:0/2787653374 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f29340f20e0 0x7f29340f0200 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f292c006ad0 tx=0x7f292c009ad0).stop 2021-02-10T11:30:38.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.282 7f293b06f700 1 -- 172.21.2.112:0/2787653374 shutdown_connections 2021-02-10T11:30:38.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.282 7f293b06f700 1 --2- 172.21.2.112:0/2787653374 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f29340f20e0 0x7f29340f0200 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:38.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.282 7f293b06f700 1 --2- 172.21.2.112:0/2787653374 >> v2:172.21.2.120:3300/0 conn(0x7f29341b0470 0x7f29341b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:38.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.282 7f293b06f700 1 -- 172.21.2.112:0/2787653374 >> 172.21.2.112:0/2787653374 conn(0x7f29340ce3e0 msgr2=0x7f29340e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:38.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.282 7f293b06f700 1 -- 172.21.2.112:0/2787653374 shutdown_connections 2021-02-10T11:30:38.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.282 7f293b06f700 1 -- 172.21.2.112:0/2787653374 wait complete. 2021-02-10T11:30:38.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.297 7f293b06f700 1 Processor -- start 2021-02-10T11:30:38.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.297 7f293b06f700 1 -- start start 2021-02-10T11:30:38.288 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.297 7f293b06f700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f293409d830 0x7f293409dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:38.289 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.297 7f293b06f700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f29340a1b50 0x7f29340a1f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:38.289 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.297 7f293b06f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f29340a2570 0x7f29340bc520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:38.289 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.297 7f293b06f700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29340a2b20 con 0x7f29340a2570 2021-02-10T11:30:38.289 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.297 7f293b06f700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29340bcb40 con 0x7f293409d830 2021-02-10T11:30:38.289 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.297 7f293b06f700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29340bcd60 con 0x7f29340a1b50 2021-02-10T11:30:38.289 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.297 7f293968c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f29340a2570 0x7f29340bc520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:38.289 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.298 7f293968c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f29340a2570 0x7f29340bc520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:47226/0 (socket says 172.21.2.112:47226) 2021-02-10T11:30:38.289 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.298 7f293968c700 1 -- 172.21.2.112:0/1588399837 learned_addr learned my addr 172.21.2.112:0/1588399837 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:30:38.289 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.298 7f2933fff700 1 --2- 172.21.2.112:0/1588399837 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f29340a1b50 0x7f29340a1f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:38.289 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.298 7f2938e8b700 1 --2- 172.21.2.112:0/1588399837 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f293409d830 0x7f293409dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:38.289 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.298 7f2933fff700 1 -- 172.21.2.112:0/1588399837 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f293409d830 msgr2=0x7f293409dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:38.289 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.298 7f2933fff700 1 --2- 172.21.2.112:0/1588399837 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f293409d830 0x7f293409dc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:38.290 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.298 7f2933fff700 1 -- 172.21.2.112:0/1588399837 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f29340a2570 msgr2=0x7f29340bc520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:38.290 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.298 7f2933fff700 1 --2- 172.21.2.112:0/1588399837 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f29340a2570 0x7f29340bc520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:38.290 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.298 7f2933fff700 1 -- 172.21.2.112:0/1588399837 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29340bcf80 con 0x7f29340a1b50 2021-02-10T11:30:38.290 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.298 7f293968c700 1 --2- 172.21.2.112:0/1588399837 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f29340a2570 0x7f29340bc520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:30:38.290 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.298 7f2933fff700 1 --2- 172.21.2.112:0/1588399837 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f29340a1b50 0x7f29340a1f50 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f292400b890 tx=0x7f292400bc30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:38.290 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.298 7f29317fa700 1 -- 172.21.2.112:0/1588399837 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2924006340 con 0x7f29340a1b50 2021-02-10T11:30:38.290 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.298 7f29317fa700 1 -- 172.21.2.112:0/1588399837 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29240066c0 con 0x7f29340a1b50 2021-02-10T11:30:38.290 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.298 7f293b06f700 1 -- 172.21.2.112:0/1588399837 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29340bcf80 con 0x7f29340a1b50 2021-02-10T11:30:38.290 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.298 7f29317fa700 1 -- 172.21.2.112:0/1588399837 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2924006340 con 0x7f29340a1b50 2021-02-10T11:30:38.290 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.298 7f293b06f700 1 -- 172.21.2.112:0/1588399837 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29340bd510 con 0x7f29340a1b50 2021-02-10T11:30:38.290 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.299 7f29317fa700 1 -- 172.21.2.112:0/1588399837 <== mon.2 v2:172.21.2.120:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f292400a000 con 0x7f29340a1b50 2021-02-10T11:30:38.290 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.299 7f29317fa700 1 --2- 172.21.2.112:0/1588399837 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f29180186c0 0x7f291801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:38.291 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.299 7f29317fa700 1 -- 172.21.2.112:0/1588399837 <== mon.2 v2:172.21.2.120:3301/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f2924032690 con 0x7f29340a1b50 2021-02-10T11:30:38.291 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.299 7f2938e8b700 1 --2- 172.21.2.112:0/1588399837 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f29180186c0 0x7f291801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:38.291 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.300 7f2938e8b700 1 --2- 172.21.2.112:0/1588399837 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f29180186c0 0x7f291801abc0 crc :-1 s=READY pgs=437 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:38.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.330 7f293b06f700 1 -- 172.21.2.112:0/1588399837 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2934105ae0 con 0x7f29340a1b50 2021-02-10T11:30:38.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.333 7f29317fa700 1 -- 172.21.2.112:0/1588399837 <== mon.2 v2:172.21.2.120:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f292400bd70 con 0x7f29340a1b50 2021-02-10T11:30:38.451 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.463 7f293b06f700 1 -- 172.21.2.112:0/1588399837 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348616", "target": ["mgr", ""]}) v1 -- 0x7f29340bd510 con 0x7f29180186c0 2021-02-10T11:30:38.461 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.474 7f29317fa700 1 -- 172.21.2.112:0/1588399837 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f291c006a30 con 0x7f29180186c0 2021-02-10T11:30:38.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.481 7f293b06f700 1 -- 172.21.2.112:0/1588399837 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f29180186c0 msgr2=0x7f291801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:38.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.481 7f293b06f700 1 --2- 172.21.2.112:0/1588399837 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f29180186c0 0x7f291801abc0 crc :-1 s=READY pgs=437 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:38.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.481 7f293b06f700 1 -- 172.21.2.112:0/1588399837 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f29340a1b50 msgr2=0x7f29340a1f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:38.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.481 7f293b06f700 1 --2- 172.21.2.112:0/1588399837 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f29340a1b50 0x7f29340a1f50 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f292400b890 tx=0x7f292400bc30).stop 2021-02-10T11:30:38.469 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.481 7f293b06f700 1 -- 172.21.2.112:0/1588399837 shutdown_connections 2021-02-10T11:30:38.469 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.481 7f293b06f700 1 --2- 172.21.2.112:0/1588399837 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f29180186c0 0x7f291801abc0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:38.469 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.481 7f293b06f700 1 --2- 172.21.2.112:0/1588399837 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f29340a2570 0x7f29340bc520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:38.469 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.481 7f293b06f700 1 --2- 172.21.2.112:0/1588399837 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f29340a1b50 0x7f29340a1f50 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:38.469 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.481 7f293b06f700 1 --2- 172.21.2.112:0/1588399837 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f293409d830 0x7f293409dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:38.469 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.481 7f293b06f700 1 -- 172.21.2.112:0/1588399837 >> 172.21.2.112:0/1588399837 conn(0x7f29340c2c20 msgr2=0x7f29340c3990 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:38.469 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.481 7f293b06f700 1 -- 172.21.2.112:0/1588399837 shutdown_connections 2021-02-10T11:30:38.469 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.481 7f293b06f700 1 -- 172.21.2.112:0/1588399837 wait complete. 2021-02-10T11:30:38.477 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000348617 2021-02-10T11:30:38.906 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.902 7fa06ef32700 1 Processor -- start 2021-02-10T11:30:38.907 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.902 7fa06ef32700 1 -- start start 2021-02-10T11:30:38.907 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.903 7fa06ef32700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fa0680f05f0 0x7fa0680f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:38.907 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.903 7fa06ef32700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0681b0c70 con 0x7fa0681b0470 2021-02-10T11:30:38.907 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.903 7fa06ef32700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0680f0b50 con 0x7fa0681b0870 2021-02-10T11:30:38.907 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.903 7fa06ef32700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0680e5ea0 con 0x7fa0680f05f0 2021-02-10T11:30:38.907 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.903 7fa06cd4e700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fa0681b0470 0x7fa0680ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:46694/0 (socket says 172.21.2.112:46694) 2021-02-10T11:30:38.907 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.903 7fa06cd4e700 1 -- 172.21.2.112:0/731735681 learned_addr learned my addr 172.21.2.112:0/731735681 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:30:38.907 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.903 7fa06d54f700 1 --2- 172.21.2.112:0/731735681 >> v2:172.21.2.120:3300/0 conn(0x7fa0680f05f0 0x7fa0680f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:38.907 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.904 7fa06d54f700 1 -- 172.21.2.112:0/731735681 >> v1:172.21.2.120:6789/0 conn(0x7fa0681b0870 legacy=0x7fa0680efe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:38.907 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.904 7fa06d54f700 1 -- 172.21.2.112:0/731735681 >> v1:172.21.2.112:6789/0 conn(0x7fa0681b0470 legacy=0x7fa0680ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:38.907 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.904 7fa06d54f700 1 -- 172.21.2.112:0/731735681 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0680e52e0 con 0x7fa0680f05f0 2021-02-10T11:30:38.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.904 7fa0677fe700 1 -- 172.21.2.112:0/731735681 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1398957954 0 0) 0x7fa058000ce0 con 0x7fa0681b0470 2021-02-10T11:30:38.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.904 7fa06d54f700 1 --2- 172.21.2.112:0/731735681 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa0680f05f0 0x7fa0680f20e0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7fa060006e50 tx=0x7fa060009e70).ready entity=mon.1 client_cookie=d8b839fceaf3bcc server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:38.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.904 7fa0677fe700 1 -- 172.21.2.112:0/731735681 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa0600066f0 con 0x7fa0680f05f0 2021-02-10T11:30:38.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.904 7fa0677fe700 1 -- 172.21.2.112:0/731735681 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa060003030 con 0x7fa0680f05f0 2021-02-10T11:30:38.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.904 7fa0677fe700 1 -- 172.21.2.112:0/731735681 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa06000b790 con 0x7fa0680f05f0 2021-02-10T11:30:38.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.904 7fa06ef32700 1 -- 172.21.2.112:0/731735681 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa0680f05f0 msgr2=0x7fa0680f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:38.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.904 7fa06ef32700 1 --2- 172.21.2.112:0/731735681 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa0680f05f0 0x7fa0680f20e0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7fa060006e50 tx=0x7fa060009e70).stop 2021-02-10T11:30:38.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.905 7fa06ef32700 1 -- 172.21.2.112:0/731735681 shutdown_connections 2021-02-10T11:30:38.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.905 7fa06ef32700 1 --2- 172.21.2.112:0/731735681 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa0680f05f0 0x7fa0680f20e0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:38.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.905 7fa06ef32700 1 -- 172.21.2.112:0/731735681 >> 172.21.2.112:0/731735681 conn(0x7fa0680ce3e0 msgr2=0x7fa0680e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:38.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.905 7fa06ef32700 1 -- 172.21.2.112:0/731735681 shutdown_connections 2021-02-10T11:30:38.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.905 7fa06ef32700 1 -- 172.21.2.112:0/731735681 wait complete. 2021-02-10T11:30:38.910 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.919 7fa06ef32700 1 Processor -- start 2021-02-10T11:30:38.910 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.919 7fa06ef32700 1 -- start start 2021-02-10T11:30:38.910 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.920 7fa06ef32700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa0681a0570 0x7fa0680a4590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:38.910 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.920 7fa06ef32700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa0680f1fd0 0x7fa0680f23d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:38.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.920 7fa06ef32700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa0680a5450 0x7fa0680a5850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:38.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.920 7fa06ef32700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0680a4bb0 con 0x7fa0680a5450 2021-02-10T11:30:38.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.920 7fa06ef32700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0680b35e0 con 0x7fa0681a0570 2021-02-10T11:30:38.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.920 7fa06ef32700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0680a1f00 con 0x7fa0680f1fd0 2021-02-10T11:30:38.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.920 7fa06d54f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa0680a5450 0x7fa0680a5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:38.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.920 7fa06d54f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa0680a5450 0x7fa0680a5850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:47242/0 (socket says 172.21.2.112:47242) 2021-02-10T11:30:38.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.920 7fa06d54f700 1 -- 172.21.2.112:0/3204896898 learned_addr learned my addr 172.21.2.112:0/3204896898 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:30:38.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.920 7fa06cd4e700 1 --2- 172.21.2.112:0/3204896898 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa0681a0570 0x7fa0680a4590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:38.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.920 7fa067fff700 1 --2- 172.21.2.112:0/3204896898 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa0680f1fd0 0x7fa0680f23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:38.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.921 7fa06d54f700 1 -- 172.21.2.112:0/3204896898 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa0680f1fd0 msgr2=0x7fa0680f23d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:38.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.921 7fa06d54f700 1 --2- 172.21.2.112:0/3204896898 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa0680f1fd0 0x7fa0680f23d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:38.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.921 7fa06d54f700 1 -- 172.21.2.112:0/3204896898 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa0681a0570 msgr2=0x7fa0680a4590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:38.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.921 7fa06d54f700 1 --2- 172.21.2.112:0/3204896898 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa0681a0570 0x7fa0680a4590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:38.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.921 7fa06d54f700 1 -- 172.21.2.112:0/3204896898 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0680a5e70 con 0x7fa0680a5450 2021-02-10T11:30:38.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.921 7fa06cd4e700 1 --2- 172.21.2.112:0/3204896898 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa0681a0570 0x7fa0680a4590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:30:38.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.921 7fa06d54f700 1 --2- 172.21.2.112:0/3204896898 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa0680a5450 0x7fa0680a5850 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 rx=0x7fa060006d00 tx=0x7fa06000bce0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:38.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.921 7fa0657fa700 1 -- 172.21.2.112:0/3204896898 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa060009310 con 0x7fa0680a5450 2021-02-10T11:30:38.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.921 7fa0657fa700 1 -- 172.21.2.112:0/3204896898 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa060009510 con 0x7fa0680a5450 2021-02-10T11:30:38.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.921 7fa0657fa700 1 -- 172.21.2.112:0/3204896898 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa06000c4e0 con 0x7fa0680a5450 2021-02-10T11:30:38.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.921 7fa06ef32700 1 -- 172.21.2.112:0/3204896898 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0680a5e70 con 0x7fa0680a5450 2021-02-10T11:30:38.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.921 7fa06ef32700 1 -- 172.21.2.112:0/3204896898 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0680a6400 con 0x7fa0680a5450 2021-02-10T11:30:38.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.922 7fa0657fa700 1 -- 172.21.2.112:0/3204896898 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa060002020 con 0x7fa0680a5450 2021-02-10T11:30:38.913 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.922 7fa0657fa700 1 --2- 172.21.2.112:0/3204896898 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa0480186c0 0x7fa04801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:38.913 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.922 7fa0657fa700 1 -- 172.21.2.112:0/3204896898 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fa060032900 con 0x7fa0680a5450 2021-02-10T11:30:38.913 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.922 7fa06cd4e700 1 --2- 172.21.2.112:0/3204896898 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa0480186c0 0x7fa04801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:38.913 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.923 7fa06cd4e700 1 --2- 172.21.2.112:0/3204896898 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa0480186c0 0x7fa04801abc0 crc :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:38.940 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.952 7fa06ef32700 1 -- 172.21.2.112:0/3204896898 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0680e52e0 con 0x7fa0680a5450 2021-02-10T11:30:38.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:38.955 7fa0657fa700 1 -- 172.21.2.112:0/3204896898 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa06000be20 con 0x7fa0680a5450 2021-02-10T11:30:39.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:39.084 7fa06ef32700 1 -- 172.21.2.112:0/3204896898 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348617", "target": ["mgr", ""]}) v1 -- 0x7fa0680a6400 con 0x7fa0480186c0 2021-02-10T11:30:39.080 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:39.094 7fa0657fa700 1 -- 172.21.2.112:0/3204896898 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fa058000ef0 con 0x7fa0480186c0 2021-02-10T11:30:39.088 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:39.101 7fa06ef32700 1 -- 172.21.2.112:0/3204896898 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa0480186c0 msgr2=0x7fa04801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:39.088 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:39.101 7fa06ef32700 1 --2- 172.21.2.112:0/3204896898 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa0480186c0 0x7fa04801abc0 crc :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:39.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:39.101 7fa06ef32700 1 -- 172.21.2.112:0/3204896898 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa0680a5450 msgr2=0x7fa0680a5850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:39.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:39.101 7fa06ef32700 1 --2- 172.21.2.112:0/3204896898 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa0680a5450 0x7fa0680a5850 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 rx=0x7fa060006d00 tx=0x7fa06000bce0).stop 2021-02-10T11:30:39.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:39.101 7fa06ef32700 1 -- 172.21.2.112:0/3204896898 shutdown_connections 2021-02-10T11:30:39.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:39.101 7fa06ef32700 1 --2- 172.21.2.112:0/3204896898 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa0480186c0 0x7fa04801abc0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:39.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:39.101 7fa06ef32700 1 --2- 172.21.2.112:0/3204896898 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa0680a5450 0x7fa0680a5850 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:39.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:39.101 7fa06ef32700 1 --2- 172.21.2.112:0/3204896898 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa0680f1fd0 0x7fa0680f23d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:39.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:39.101 7fa06ef32700 1 --2- 172.21.2.112:0/3204896898 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa0681a0570 0x7fa0680a4590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:39.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:39.101 7fa06ef32700 1 -- 172.21.2.112:0/3204896898 >> 172.21.2.112:0/3204896898 conn(0x7fa0680958b0 msgr2=0x7fa0680bf040 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:39.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:39.101 7fa06ef32700 1 -- 172.21.2.112:0/3204896898 shutdown_connections 2021-02-10T11:30:39.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:39.101 7fa06ef32700 1 -- 172.21.2.112:0/3204896898 wait complete. 2021-02-10T11:30:39.097 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000348618 2021-02-10T11:30:39.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:39.520 7f5d2bee9700 1 Processor -- start 2021-02-10T11:30:39.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:39.520 7f5d2bee9700 1 -- start start 2021-02-10T11:30:39.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:39.521 7f5d2bee9700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f5d240f05f0 0x7f5d240f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:39.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:39.521 7f5d2bee9700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d241b0c70 con 0x7f5d241b0470 2021-02-10T11:30:39.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:39.521 7f5d2bee9700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d240f0b50 con 0x7f5d241b0870 2021-02-10T11:30:39.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:39.521 7f5d2bee9700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d240e5ea0 con 0x7f5d240f05f0 2021-02-10T11:30:39.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:39.521 7f5d2a506700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f5d240f05f0 0x7f5d240f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:39.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:39.521 7f5d29d05700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f5d241b0470 0x7f5d240ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:46708/0 (socket says 172.21.2.112:46708) 2021-02-10T11:30:39.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:39.521 7f5d29d05700 1 -- 172.21.2.112:0/512048310 learned_addr learned my addr 172.21.2.112:0/512048310 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:30:39.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:39.521 7f5d29504700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7f5d241b0870 0x7f5d240efe00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:40266/0 (socket says 172.21.2.112:40266) 2021-02-10T11:30:39.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:39.522 7f5d28d03700 1 -- 172.21.2.112:0/512048310 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3632301181 0 0) 0x7f5d140010c0 con 0x7f5d241b0870 2021-02-10T11:30:39.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:39.522 7f5d2a506700 1 -- 172.21.2.112:0/512048310 >> v1:172.21.2.120:6789/0 conn(0x7f5d241b0870 legacy=0x7f5d240efe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:39.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:39.522 7f5d2a506700 1 -- 172.21.2.112:0/512048310 >> v1:172.21.2.112:6789/0 conn(0x7f5d241b0470 legacy=0x7f5d240ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:39.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:39.522 7f5d2a506700 1 -- 172.21.2.112:0/512048310 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d240e52e0 con 0x7f5d240f05f0 2021-02-10T11:30:39.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:39.522 7f5d2a506700 1 --2- 172.21.2.112:0/512048310 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5d240f05f0 0x7f5d240f20e0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 rx=0x7f5d1c006e30 tx=0x7f5d1c009e60).ready entity=mon.0 client_cookie=b5d057912b0b8c3f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:39.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:39.522 7f5d28d03700 1 -- 172.21.2.112:0/512048310 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5d1c003030 con 0x7f5d240f05f0 2021-02-10T11:30:39.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:39.522 7f5d28d03700 1 -- 172.21.2.112:0/512048310 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d1c00b300 con 0x7f5d240f05f0 2021-02-10T11:30:39.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:39.522 7f5d28d03700 1 -- 172.21.2.112:0/512048310 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5d1c00b820 con 0x7f5d240f05f0 2021-02-10T11:30:39.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:39.522 7f5d2bee9700 1 -- 172.21.2.112:0/512048310 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5d240f05f0 msgr2=0x7f5d240f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:39.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:39.522 7f5d2bee9700 1 --2- 172.21.2.112:0/512048310 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5d240f05f0 0x7f5d240f20e0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 rx=0x7f5d1c006e30 tx=0x7f5d1c009e60).stop 2021-02-10T11:30:39.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:39.523 7f5d2bee9700 1 -- 172.21.2.112:0/512048310 shutdown_connections 2021-02-10T11:30:39.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:39.523 7f5d2bee9700 1 --2- 172.21.2.112:0/512048310 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5d240f05f0 0x7f5d240f20e0 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:39.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:39.523 7f5d2bee9700 1 -- 172.21.2.112:0/512048310 >> 172.21.2.112:0/512048310 conn(0x7f5d240ce3e0 msgr2=0x7f5d240e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:39.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:39.523 7f5d2bee9700 1 -- 172.21.2.112:0/512048310 shutdown_connections 2021-02-10T11:30:39.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:39.523 7f5d2bee9700 1 -- 172.21.2.112:0/512048310 wait complete. 2021-02-10T11:30:39.529 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:39.538 7f5d2bee9700 1 Processor -- start 2021-02-10T11:30:39.529 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:39.538 7f5d2bee9700 1 -- start start 2021-02-10T11:30:39.529 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:39.538 7f5d2bee9700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5d240a1dd0 0x7f5d24190920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:39.529 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:39.538 7f5d2bee9700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5d2408fd30 0x7f5d240effc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:39.529 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:39.538 7f5d2bee9700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5d240f0e90 0x7f5d240f1290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:39.529 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:39.538 7f5d2bee9700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d240f06f0 con 0x7f5d240f0e90 2021-02-10T11:30:39.529 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:39.538 7f5d2bee9700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d24090160 con 0x7f5d2408fd30 2021-02-10T11:30:39.529 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:39.538 7f5d2bee9700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d240f18b0 con 0x7f5d240a1dd0 2021-02-10T11:30:39.529 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:39.539 7f5d2a506700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5d240f0e90 0x7f5d240f1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:39.529 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:39.539 7f5d29504700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5d2408fd30 0x7f5d240effc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:39.529 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:39.539 7f5d2a506700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5d240f0e90 0x7f5d240f1290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:47252/0 (socket says 172.21.2.112:47252) 2021-02-10T11:30:39.529 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:39.539 7f5d2a506700 1 -- 172.21.2.112:0/16112114 learned_addr learned my addr 172.21.2.112:0/16112114 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:30:39.530 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:39.539 7f5d29d05700 1 --2- 172.21.2.112:0/16112114 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5d240a1dd0 0x7f5d24190920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:39.530 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:39.539 7f5d2a506700 1 -- 172.21.2.112:0/16112114 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5d240a1dd0 msgr2=0x7f5d24190920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:39.530 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:39.539 7f5d2a506700 1 --2- 172.21.2.112:0/16112114 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5d240a1dd0 0x7f5d24190920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:39.530 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:39.539 7f5d2a506700 1 -- 172.21.2.112:0/16112114 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5d2408fd30 msgr2=0x7f5d240effc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:39.530 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:39.539 7f5d2a506700 1 --2- 172.21.2.112:0/16112114 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5d2408fd30 0x7f5d240effc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:39.530 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:39.539 7f5d2a506700 1 -- 172.21.2.112:0/16112114 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d240f1ad0 con 0x7f5d240f0e90 2021-02-10T11:30:39.530 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:39.539 7f5d2a506700 1 --2- 172.21.2.112:0/16112114 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5d240f0e90 0x7f5d240f1290 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 rx=0x7f5d1c009f80 tx=0x7f5d1c00bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:39.530 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:39.540 7f5d1a7fc700 1 -- 172.21.2.112:0/16112114 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5d1c003030 con 0x7f5d240f0e90 2021-02-10T11:30:39.530 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:39.540 7f5d1a7fc700 1 -- 172.21.2.112:0/16112114 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d1c009710 con 0x7f5d240f0e90 2021-02-10T11:30:39.530 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:39.540 7f5d1a7fc700 1 -- 172.21.2.112:0/16112114 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5d1c001700 con 0x7f5d240f0e90 2021-02-10T11:30:39.530 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:39.540 7f5d2bee9700 1 -- 172.21.2.112:0/16112114 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d240f1ad0 con 0x7f5d240f0e90 2021-02-10T11:30:39.530 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:39.540 7f5d2bee9700 1 -- 172.21.2.112:0/16112114 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d240f1ad0 con 0x7f5d240f0e90 2021-02-10T11:30:39.530 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:39.540 7f5d1a7fc700 1 -- 172.21.2.112:0/16112114 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5d1c001270 con 0x7f5d240f0e90 2021-02-10T11:30:39.531 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:39.540 7f5d1a7fc700 1 --2- 172.21.2.112:0/16112114 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5d0c0186c0 0x7f5d0c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:39.531 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:39.540 7f5d1a7fc700 1 -- 172.21.2.112:0/16112114 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f5d1c032900 con 0x7f5d240f0e90 2021-02-10T11:30:39.531 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:39.541 7f5d29d05700 1 --2- 172.21.2.112:0/16112114 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5d0c0186c0 0x7f5d0c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:39.531 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:39.541 7f5d29d05700 1 --2- 172.21.2.112:0/16112114 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5d0c0186c0 0x7f5d0c01abc0 crc :-1 s=READY pgs=439 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:39.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:39.571 7f5d2bee9700 1 -- 172.21.2.112:0/16112114 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d240e52e0 con 0x7f5d240f0e90 2021-02-10T11:30:39.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:39.573 7f5d1a7fc700 1 -- 172.21.2.112:0/16112114 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5d1c01f640 con 0x7f5d240f0e90 2021-02-10T11:30:39.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:39.700 7f5d2bee9700 1 -- 172.21.2.112:0/16112114 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348618", "target": ["mgr", ""]}) v1 -- 0x7f5d240e52e0 con 0x7f5d0c0186c0 2021-02-10T11:30:39.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:39.712 7f5d1a7fc700 1 -- 172.21.2.112:0/16112114 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f5d20007cc0 con 0x7f5d0c0186c0 2021-02-10T11:30:39.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:39.718 7f5d2bee9700 1 -- 172.21.2.112:0/16112114 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5d0c0186c0 msgr2=0x7f5d0c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:39.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:39.718 7f5d2bee9700 1 --2- 172.21.2.112:0/16112114 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5d0c0186c0 0x7f5d0c01abc0 crc :-1 s=READY pgs=439 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:39.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:39.718 7f5d2bee9700 1 -- 172.21.2.112:0/16112114 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5d240f0e90 msgr2=0x7f5d240f1290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:39.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:39.718 7f5d2bee9700 1 --2- 172.21.2.112:0/16112114 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5d240f0e90 0x7f5d240f1290 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 rx=0x7f5d1c009f80 tx=0x7f5d1c00bd10).stop 2021-02-10T11:30:39.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:39.718 7f5d2bee9700 1 -- 172.21.2.112:0/16112114 shutdown_connections 2021-02-10T11:30:39.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:39.718 7f5d2bee9700 1 --2- 172.21.2.112:0/16112114 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5d0c0186c0 0x7f5d0c01abc0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:39.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:39.718 7f5d2bee9700 1 --2- 172.21.2.112:0/16112114 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5d2408fd30 0x7f5d240effc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:39.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:39.718 7f5d2bee9700 1 --2- 172.21.2.112:0/16112114 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5d240f0e90 0x7f5d240f1290 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:39.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:39.718 7f5d2bee9700 1 --2- 172.21.2.112:0/16112114 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5d240a1dd0 0x7f5d24190920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:39.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:39.718 7f5d2bee9700 1 -- 172.21.2.112:0/16112114 >> 172.21.2.112:0/16112114 conn(0x7f5d240958b0 msgr2=0x7f5d240bf040 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:39.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:39.718 7f5d2bee9700 1 -- 172.21.2.112:0/16112114 shutdown_connections 2021-02-10T11:30:39.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:39.718 7f5d2bee9700 1 -- 172.21.2.112:0/16112114 wait complete. 2021-02-10T11:30:39.714 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000348619 2021-02-10T11:30:40.146 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.142 7fd27bcea700 1 Processor -- start 2021-02-10T11:30:40.146 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.142 7fd27bcea700 1 -- start start 2021-02-10T11:30:40.146 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.143 7fd27bcea700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fd2740f05f0 0x7fd2740f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:40.146 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.143 7fd27bcea700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2741b0c70 con 0x7fd2741b0870 2021-02-10T11:30:40.146 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.143 7fd27bcea700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2740f0b50 con 0x7fd2741b0470 2021-02-10T11:30:40.147 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.143 7fd27bcea700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2740e5ea0 con 0x7fd2740f05f0 2021-02-10T11:30:40.147 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.143 7fd27a307700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fd2740f05f0 0x7fd2740f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:40.147 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.143 7fd279305700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fd2741b0870 0x7fd2740efe00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:46726/0 (socket says 172.21.2.112:46726) 2021-02-10T11:30:40.147 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.143 7fd279305700 1 -- 172.21.2.112:0/3496681320 learned_addr learned my addr 172.21.2.112:0/3496681320 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:30:40.147 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.143 7fd27a307700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fd2740f05f0 0x7fd2740f20e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:47262/0 (socket says 172.21.2.112:47262) 2021-02-10T11:30:40.147 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.143 7fd27a307700 1 -- 172.21.2.112:0/3496681320 >> v1:172.21.2.120:6789/0 conn(0x7fd2741b0470 legacy=0x7fd2740ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:40.147 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.143 7fd27a307700 1 -- 172.21.2.112:0/3496681320 >> v1:172.21.2.112:6789/0 conn(0x7fd2741b0870 legacy=0x7fd2740efe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:40.147 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.143 7fd27a307700 1 -- 172.21.2.112:0/3496681320 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2740d2fb0 con 0x7fd2740f05f0 2021-02-10T11:30:40.147 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.143 7fd27a307700 1 --2- 172.21.2.112:0/3496681320 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd2740f05f0 0x7fd2740f20e0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 rx=0x7fd26c006ab0 tx=0x7fd26c009b40).ready entity=mon.0 client_cookie=7988e8f9b0cd0b2d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:40.147 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.143 7fd278b04700 1 -- 172.21.2.112:0/3496681320 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd26c0022e0 con 0x7fd2740f05f0 2021-02-10T11:30:40.147 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.143 7fd278b04700 1 -- 172.21.2.112:0/3496681320 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd26c002560 con 0x7fd2740f05f0 2021-02-10T11:30:40.147 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.143 7fd278b04700 1 -- 172.21.2.112:0/3496681320 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd26c002a80 con 0x7fd2740f05f0 2021-02-10T11:30:40.147 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.143 7fd27bcea700 1 -- 172.21.2.112:0/3496681320 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd2740f05f0 msgr2=0x7fd2740f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:40.148 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.143 7fd27bcea700 1 --2- 172.21.2.112:0/3496681320 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd2740f05f0 0x7fd2740f20e0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 rx=0x7fd26c006ab0 tx=0x7fd26c009b40).stop 2021-02-10T11:30:40.148 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.144 7fd27bcea700 1 -- 172.21.2.112:0/3496681320 shutdown_connections 2021-02-10T11:30:40.148 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.144 7fd27bcea700 1 --2- 172.21.2.112:0/3496681320 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd2740f05f0 0x7fd2740f20e0 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:40.148 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.144 7fd27bcea700 1 -- 172.21.2.112:0/3496681320 >> 172.21.2.112:0/3496681320 conn(0x7fd2740ce3e0 msgr2=0x7fd2740e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:40.148 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.144 7fd27bcea700 1 -- 172.21.2.112:0/3496681320 shutdown_connections 2021-02-10T11:30:40.148 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.144 7fd27bcea700 1 -- 172.21.2.112:0/3496681320 wait complete. 2021-02-10T11:30:40.149 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.159 7fd27bcea700 1 Processor -- start 2021-02-10T11:30:40.149 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.159 7fd27bcea700 1 -- start start 2021-02-10T11:30:40.149 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.159 7fd27bcea700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd274087ad0 0x7fd274087ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:40.149 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.159 7fd27bcea700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd2741b01a0 0x7fd2741b05a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:40.150 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.159 7fd27bcea700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd2741b0bc0 0x7fd2741b0fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:40.150 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.159 7fd27bcea700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2741b16e0 con 0x7fd274087ad0 2021-02-10T11:30:40.150 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.160 7fd27bcea700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2741b1c60 con 0x7fd2741b01a0 2021-02-10T11:30:40.150 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.160 7fd27bcea700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2741b1e40 con 0x7fd2741b0bc0 2021-02-10T11:30:40.150 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.160 7fd279b06700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd274087ad0 0x7fd274087ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:40.150 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.160 7fd27a307700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd2741b0bc0 0x7fd2741b0fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:40.150 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.160 7fd279b06700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd274087ad0 0x7fd274087ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:47268/0 (socket says 172.21.2.112:47268) 2021-02-10T11:30:40.150 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.160 7fd27a307700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd2741b0bc0 0x7fd2741b0fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3301/0 says I am v2:172.21.2.112:46948/0 (socket says 172.21.2.112:46948) 2021-02-10T11:30:40.150 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.160 7fd279b06700 1 -- 172.21.2.112:0/2314717330 learned_addr learned my addr 172.21.2.112:0/2314717330 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:30:40.150 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.160 7fd279305700 1 --2- 172.21.2.112:0/2314717330 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd2741b01a0 0x7fd2741b05a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:40.150 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.160 7fd279b06700 1 -- 172.21.2.112:0/2314717330 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd2741b0bc0 msgr2=0x7fd2741b0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:40.150 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.160 7fd279b06700 1 --2- 172.21.2.112:0/2314717330 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd2741b0bc0 0x7fd2741b0fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:40.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.160 7fd279b06700 1 -- 172.21.2.112:0/2314717330 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd2741b01a0 msgr2=0x7fd2741b05a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:40.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.160 7fd279b06700 1 --2- 172.21.2.112:0/2314717330 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd2741b01a0 0x7fd2741b05a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:40.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.160 7fd279b06700 1 -- 172.21.2.112:0/2314717330 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2741b2060 con 0x7fd274087ad0 2021-02-10T11:30:40.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.160 7fd279b06700 1 --2- 172.21.2.112:0/2314717330 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd274087ad0 0x7fd274087ed0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 rx=0x7fd27000b8e0 tx=0x7fd27000bc80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:40.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.160 7fd26a7fc700 1 -- 172.21.2.112:0/2314717330 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd270000ce0 con 0x7fd274087ad0 2021-02-10T11:30:40.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.160 7fd26a7fc700 1 -- 172.21.2.112:0/2314717330 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd27000c2f0 con 0x7fd274087ad0 2021-02-10T11:30:40.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.160 7fd26a7fc700 1 -- 172.21.2.112:0/2314717330 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd27000c810 con 0x7fd274087ad0 2021-02-10T11:30:40.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.160 7fd27bcea700 1 -- 172.21.2.112:0/2314717330 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2741b2060 con 0x7fd274087ad0 2021-02-10T11:30:40.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.160 7fd27bcea700 1 -- 172.21.2.112:0/2314717330 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2741b2060 con 0x7fd274087ad0 2021-02-10T11:30:40.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.161 7fd26a7fc700 1 -- 172.21.2.112:0/2314717330 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd270001230 con 0x7fd274087ad0 2021-02-10T11:30:40.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.161 7fd26a7fc700 1 --2- 172.21.2.112:0/2314717330 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd25c0186d0 0x7fd25c01abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:40.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.161 7fd26a7fc700 1 -- 172.21.2.112:0/2314717330 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fd270032a10 con 0x7fd274087ad0 2021-02-10T11:30:40.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.161 7fd279305700 1 --2- 172.21.2.112:0/2314717330 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd25c0186d0 0x7fd25c01abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:40.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.161 7fd279305700 1 --2- 172.21.2.112:0/2314717330 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd25c0186d0 0x7fd25c01abd0 crc :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:40.179 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.192 7fd27bcea700 1 -- 172.21.2.112:0/2314717330 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd27407e700 con 0x7fd274087ad0 2021-02-10T11:30:40.181 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.194 7fd26a7fc700 1 -- 172.21.2.112:0/2314717330 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd27000bdc0 con 0x7fd274087ad0 2021-02-10T11:30:40.307 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.320 7fd27bcea700 1 -- 172.21.2.112:0/2314717330 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348619", "target": ["mgr", ""]}) v1 -- 0x7fd2740e52e0 con 0x7fd25c0186d0 2021-02-10T11:30:40.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.331 7fd26a7fc700 1 -- 172.21.2.112:0/2314717330 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fd264001a00 con 0x7fd25c0186d0 2021-02-10T11:30:40.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.338 7fd27bcea700 1 -- 172.21.2.112:0/2314717330 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd25c0186d0 msgr2=0x7fd25c01abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:40.326 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.338 7fd27bcea700 1 --2- 172.21.2.112:0/2314717330 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd25c0186d0 0x7fd25c01abd0 crc :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:40.326 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.338 7fd27bcea700 1 -- 172.21.2.112:0/2314717330 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd274087ad0 msgr2=0x7fd274087ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:40.326 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.338 7fd27bcea700 1 --2- 172.21.2.112:0/2314717330 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd274087ad0 0x7fd274087ed0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 rx=0x7fd27000b8e0 tx=0x7fd27000bc80).stop 2021-02-10T11:30:40.326 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.338 7fd27bcea700 1 -- 172.21.2.112:0/2314717330 shutdown_connections 2021-02-10T11:30:40.326 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.338 7fd27bcea700 1 --2- 172.21.2.112:0/2314717330 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd25c0186d0 0x7fd25c01abd0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:40.326 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.338 7fd27bcea700 1 --2- 172.21.2.112:0/2314717330 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd2741b01a0 0x7fd2741b05a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:40.326 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.338 7fd27bcea700 1 --2- 172.21.2.112:0/2314717330 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd2741b0bc0 0x7fd2741b0fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:40.326 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.338 7fd27bcea700 1 --2- 172.21.2.112:0/2314717330 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd274087ad0 0x7fd274087ed0 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:40.326 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.338 7fd27bcea700 1 -- 172.21.2.112:0/2314717330 >> 172.21.2.112:0/2314717330 conn(0x7fd2740b8850 msgr2=0x7fd2741b58f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:40.326 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.338 7fd27bcea700 1 -- 172.21.2.112:0/2314717330 shutdown_connections 2021-02-10T11:30:40.327 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.338 7fd27bcea700 1 -- 172.21.2.112:0/2314717330 wait complete. 2021-02-10T11:30:40.334 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000348620 2021-02-10T11:30:40.766 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.762 7f0a0ec29700 1 Processor -- start 2021-02-10T11:30:40.766 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.762 7f0a0ec29700 1 -- start start 2021-02-10T11:30:40.766 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.762 7f0a0ec29700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0a081b0870 0x7f0a080f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:40.767 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.762 7f0a0ec29700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f0a080efe00 0x7f0a080f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:40.767 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.762 7f0a0ec29700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a081b0c70 con 0x7f0a081b0470 2021-02-10T11:30:40.767 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.762 7f0a0ec29700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a080f0b50 con 0x7f0a080efe00 2021-02-10T11:30:40.767 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.762 7f0a0ec29700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a080f0820 con 0x7f0a081b0870 2021-02-10T11:30:40.767 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.763 7f0a0d246700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f0a080efe00 0x7f0a080f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:40.767 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.763 7f0a0ca45700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f0a081b0470 0x7f0a080ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:46736/0 (socket says 172.21.2.112:46736) 2021-02-10T11:30:40.767 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.763 7f0a0ca45700 1 -- 172.21.2.112:0/2535764388 learned_addr learned my addr 172.21.2.112:0/2535764388 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:30:40.767 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.763 7f0a0d246700 1 --2- 172.21.2.112:0/2535764388 >> v2:172.21.2.112:3300/0 conn(0x7f0a080efe00 0x7f0a080f0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:47278/0 (socket says 172.21.2.112:47278) 2021-02-10T11:30:40.767 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.763 7f0a07fff700 1 --2- 172.21.2.112:0/2535764388 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0a081b0870 0x7f0a080f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:40.767 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.763 7f0a0d246700 1 -- 172.21.2.112:0/2535764388 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0a081b0870 msgr2=0x7f0a080f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:40.767 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.763 7f0a0d246700 1 --2- 172.21.2.112:0/2535764388 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0a081b0870 0x7f0a080f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:40.767 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.763 7f0a0d246700 1 -- 172.21.2.112:0/2535764388 >> v1:172.21.2.112:6789/0 conn(0x7f0a081b0470 legacy=0x7f0a080ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:40.767 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.763 7f0a077fe700 1 -- 172.21.2.112:0/2535764388 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3484520791 0 0) 0x7f09f0000ce0 con 0x7f0a081b0470 2021-02-10T11:30:40.768 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.763 7f0a0d246700 1 -- 172.21.2.112:0/2535764388 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a080e52e0 con 0x7f0a080efe00 2021-02-10T11:30:40.768 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.763 7f0a0d246700 1 --2- 172.21.2.112:0/2535764388 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0a080efe00 0x7f0a080f0200 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 rx=0x7f0a00006e30 tx=0x7f0a00009de0).ready entity=mon.0 client_cookie=79bca8a03ad4a503 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:40.768 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.763 7f0a077fe700 1 -- 172.21.2.112:0/2535764388 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0a00003030 con 0x7f0a080efe00 2021-02-10T11:30:40.768 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.763 7f0a077fe700 1 -- 172.21.2.112:0/2535764388 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a0000b300 con 0x7f0a080efe00 2021-02-10T11:30:40.768 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.763 7f0a077fe700 1 -- 172.21.2.112:0/2535764388 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0a0000b750 con 0x7f0a080efe00 2021-02-10T11:30:40.768 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.763 7f0a0ec29700 1 -- 172.21.2.112:0/2535764388 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0a080efe00 msgr2=0x7f0a080f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:40.768 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.763 7f0a0ec29700 1 --2- 172.21.2.112:0/2535764388 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0a080efe00 0x7f0a080f0200 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 rx=0x7f0a00006e30 tx=0x7f0a00009de0).stop 2021-02-10T11:30:40.768 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.764 7f0a0ec29700 1 -- 172.21.2.112:0/2535764388 shutdown_connections 2021-02-10T11:30:40.768 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.764 7f0a0ec29700 1 --2- 172.21.2.112:0/2535764388 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0a080efe00 0x7f0a080f0200 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:40.768 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.764 7f0a0ec29700 1 --2- 172.21.2.112:0/2535764388 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0a081b0870 0x7f0a080f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:40.768 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.764 7f0a0ec29700 1 -- 172.21.2.112:0/2535764388 >> 172.21.2.112:0/2535764388 conn(0x7f0a080ce3e0 msgr2=0x7f0a080e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:40.768 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.764 7f0a0ec29700 1 -- 172.21.2.112:0/2535764388 shutdown_connections 2021-02-10T11:30:40.768 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.764 7f0a0ec29700 1 -- 172.21.2.112:0/2535764388 wait complete. 2021-02-10T11:30:40.769 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.779 7f0a0ec29700 1 Processor -- start 2021-02-10T11:30:40.769 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.779 7f0a0ec29700 1 -- start start 2021-02-10T11:30:40.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.779 7f0a0ec29700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0a080a0420 0x7f0a080a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:40.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.779 7f0a0ec29700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0a08189d00 0x7f0a0818a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:40.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.779 7f0a0ec29700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0a0818a720 0x7f0a080b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:40.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.779 7f0a0ec29700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a08189780 con 0x7f0a08189d00 2021-02-10T11:30:40.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.779 7f0a0ec29700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a0818ab20 con 0x7f0a080a0420 2021-02-10T11:30:40.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.779 7f0a0ec29700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a080a1140 con 0x7f0a0818a720 2021-02-10T11:30:40.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.779 7f0a07fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0a08189d00 0x7f0a0818a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:40.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.780 7f0a07fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0a08189d00 0x7f0a0818a100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:47280/0 (socket says 172.21.2.112:47280) 2021-02-10T11:30:40.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.780 7f0a07fff700 1 -- 172.21.2.112:0/2797499984 learned_addr learned my addr 172.21.2.112:0/2797499984 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:30:40.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.780 7f0a0d246700 1 --2- 172.21.2.112:0/2797499984 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0a0818a720 0x7f0a080b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:40.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.780 7f0a0ca45700 1 --2- 172.21.2.112:0/2797499984 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0a080a0420 0x7f0a080a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:40.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.780 7f0a0d246700 1 -- 172.21.2.112:0/2797499984 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0a080a0420 msgr2=0x7f0a080a0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:40.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.780 7f0a0d246700 1 --2- 172.21.2.112:0/2797499984 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0a080a0420 0x7f0a080a0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:40.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.780 7f0a0d246700 1 -- 172.21.2.112:0/2797499984 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0a08189d00 msgr2=0x7f0a0818a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:40.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.780 7f0a0d246700 1 --2- 172.21.2.112:0/2797499984 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0a08189d00 0x7f0a0818a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:40.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.780 7f0a0d246700 1 -- 172.21.2.112:0/2797499984 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a080b8200 con 0x7f0a0818a720 2021-02-10T11:30:40.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.780 7f0a07fff700 1 --2- 172.21.2.112:0/2797499984 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0a08189d00 0x7f0a0818a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:30:40.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.780 7f0a0d246700 1 --2- 172.21.2.112:0/2797499984 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0a0818a720 0x7f0a080b7be0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f0a0000b830 tx=0x7f0a0000bbd0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:40.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.780 7f0a057fa700 1 -- 172.21.2.112:0/2797499984 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0a00003030 con 0x7f0a0818a720 2021-02-10T11:30:40.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.780 7f0a057fa700 1 -- 172.21.2.112:0/2797499984 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a00001270 con 0x7f0a0818a720 2021-02-10T11:30:40.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.780 7f0a057fa700 1 -- 172.21.2.112:0/2797499984 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0a000016c0 con 0x7f0a0818a720 2021-02-10T11:30:40.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.780 7f0a0ec29700 1 -- 172.21.2.112:0/2797499984 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a080b8200 con 0x7f0a0818a720 2021-02-10T11:30:40.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.780 7f0a0ec29700 1 -- 172.21.2.112:0/2797499984 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a080b8200 con 0x7f0a0818a720 2021-02-10T11:30:40.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.781 7f0a057fa700 1 -- 172.21.2.112:0/2797499984 <== mon.2 v2:172.21.2.120:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0a000094a0 con 0x7f0a0818a720 2021-02-10T11:30:40.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.781 7f0a057fa700 1 --2- 172.21.2.112:0/2797499984 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f09ec0186d0 0x7f09ec01abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:40.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.781 7f0a057fa700 1 -- 172.21.2.112:0/2797499984 <== mon.2 v2:172.21.2.120:3301/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f0a00032900 con 0x7f0a0818a720 2021-02-10T11:30:40.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.781 7f0a0ca45700 1 --2- 172.21.2.112:0/2797499984 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f09ec0186d0 0x7f09ec01abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:40.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.781 7f0a0ca45700 1 --2- 172.21.2.112:0/2797499984 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f09ec0186d0 0x7f09ec01abd0 crc :-1 s=READY pgs=441 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:40.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.812 7f0a0ec29700 1 -- 172.21.2.112:0/2797499984 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a08105ae0 con 0x7f0a0818a720 2021-02-10T11:30:40.802 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.814 7f0a057fa700 1 -- 172.21.2.112:0/2797499984 <== mon.2 v2:172.21.2.120:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0a0001f7c0 con 0x7f0a0818a720 2021-02-10T11:30:40.929 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.941 7f0a0ec29700 1 -- 172.21.2.112:0/2797499984 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348620", "target": ["mgr", ""]}) v1 -- 0x7f0a08189780 con 0x7f09ec0186d0 2021-02-10T11:30:40.940 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.953 7f0a057fa700 1 -- 172.21.2.112:0/2797499984 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f09f0007cc0 con 0x7f09ec0186d0 2021-02-10T11:30:40.946 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.959 7f0a0ec29700 1 -- 172.21.2.112:0/2797499984 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f09ec0186d0 msgr2=0x7f09ec01abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:40.946 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.959 7f0a0ec29700 1 --2- 172.21.2.112:0/2797499984 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f09ec0186d0 0x7f09ec01abd0 crc :-1 s=READY pgs=441 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:40.946 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.959 7f0a0ec29700 1 -- 172.21.2.112:0/2797499984 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0a0818a720 msgr2=0x7f0a080b7be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:40.947 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.959 7f0a0ec29700 1 --2- 172.21.2.112:0/2797499984 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0a0818a720 0x7f0a080b7be0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f0a0000b830 tx=0x7f0a0000bbd0).stop 2021-02-10T11:30:40.947 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.959 7f0a0ec29700 1 -- 172.21.2.112:0/2797499984 shutdown_connections 2021-02-10T11:30:40.947 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.959 7f0a0ec29700 1 --2- 172.21.2.112:0/2797499984 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f09ec0186d0 0x7f09ec01abd0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:40.947 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.959 7f0a0ec29700 1 --2- 172.21.2.112:0/2797499984 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0a0818a720 0x7f0a080b7be0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:40.947 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.959 7f0a0ec29700 1 --2- 172.21.2.112:0/2797499984 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0a08189d00 0x7f0a0818a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:40.947 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.959 7f0a0ec29700 1 --2- 172.21.2.112:0/2797499984 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0a080a0420 0x7f0a080a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:40.947 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.959 7f0a0ec29700 1 -- 172.21.2.112:0/2797499984 >> 172.21.2.112:0/2797499984 conn(0x7f0a08094e10 msgr2=0x7f0a080a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:40.948 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.959 7f0a0ec29700 1 -- 172.21.2.112:0/2797499984 shutdown_connections 2021-02-10T11:30:40.948 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:40.959 7f0a0ec29700 1 -- 172.21.2.112:0/2797499984 wait complete. 2021-02-10T11:30:40.955 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000348621 2021-02-10T11:30:41.384 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:41.380 7fc2ca16a700 1 Processor -- start 2021-02-10T11:30:41.384 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:41.380 7fc2ca16a700 1 -- start start 2021-02-10T11:30:41.384 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:41.381 7fc2ca16a700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fc2c40f05f0 0x7fc2c40f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:41.385 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:41.381 7fc2ca16a700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2c41b0c70 con 0x7fc2c41b0470 2021-02-10T11:30:41.385 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:41.381 7fc2ca16a700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2c40f0b50 con 0x7fc2c41b0870 2021-02-10T11:30:41.385 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:41.381 7fc2ca16a700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2c40e5ea0 con 0x7fc2c40f05f0 2021-02-10T11:30:41.385 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:41.381 7fc2c37fe700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fc2c41b0470 0x7fc2c40ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:46750/0 (socket says 172.21.2.112:46750) 2021-02-10T11:30:41.385 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:41.381 7fc2c37fe700 1 -- 172.21.2.112:0/645791945 learned_addr learned my addr 172.21.2.112:0/645791945 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:30:41.385 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:41.381 7fc2c3fff700 1 --2- 172.21.2.112:0/645791945 >> v2:172.21.2.120:3300/0 conn(0x7fc2c40f05f0 0x7fc2c40f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:41.385 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:41.381 7fc2c3fff700 1 -- 172.21.2.112:0/645791945 >> v1:172.21.2.120:6789/0 conn(0x7fc2c41b0870 legacy=0x7fc2c40efe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:41.385 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:41.381 7fc2c3fff700 1 -- 172.21.2.112:0/645791945 >> v1:172.21.2.112:6789/0 conn(0x7fc2c41b0470 legacy=0x7fc2c40ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:41.385 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:41.381 7fc2c3fff700 1 -- 172.21.2.112:0/645791945 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2c40e52e0 con 0x7fc2c40f05f0 2021-02-10T11:30:41.385 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:41.382 7fc2c3fff700 1 --2- 172.21.2.112:0/645791945 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc2c40f05f0 0x7fc2c40f20e0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7fc2bc006e50 tx=0x7fc2bc009df0).ready entity=mon.1 client_cookie=e395a288db709089 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:41.386 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:41.382 7fc2c27fc700 1 -- 172.21.2.112:0/645791945 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc2bc00b510 con 0x7fc2c40f05f0 2021-02-10T11:30:41.386 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:41.382 7fc2c27fc700 1 -- 172.21.2.112:0/645791945 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2bc006790 con 0x7fc2c40f05f0 2021-02-10T11:30:41.386 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:41.382 7fc2c27fc700 1 -- 172.21.2.112:0/645791945 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc2bc00b750 con 0x7fc2c40f05f0 2021-02-10T11:30:41.386 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:41.382 7fc2ca16a700 1 -- 172.21.2.112:0/645791945 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc2c40f05f0 msgr2=0x7fc2c40f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:41.386 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:41.382 7fc2ca16a700 1 --2- 172.21.2.112:0/645791945 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc2c40f05f0 0x7fc2c40f20e0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7fc2bc006e50 tx=0x7fc2bc009df0).stop 2021-02-10T11:30:41.386 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:41.382 7fc2ca16a700 1 -- 172.21.2.112:0/645791945 shutdown_connections 2021-02-10T11:30:41.386 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:41.382 7fc2ca16a700 1 --2- 172.21.2.112:0/645791945 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc2c40f05f0 0x7fc2c40f20e0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:41.386 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:41.383 7fc2ca16a700 1 -- 172.21.2.112:0/645791945 >> 172.21.2.112:0/645791945 conn(0x7fc2c40ce3e0 msgr2=0x7fc2c40e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:41.386 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:41.383 7fc2ca16a700 1 -- 172.21.2.112:0/645791945 shutdown_connections 2021-02-10T11:30:41.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:41.383 7fc2ca16a700 1 -- 172.21.2.112:0/645791945 wait complete. 2021-02-10T11:30:41.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:41.397 7fc2ca16a700 1 Processor -- start 2021-02-10T11:30:41.389 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:41.397 7fc2ca16a700 1 -- start start 2021-02-10T11:30:41.389 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:41.398 7fc2ca16a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc2c41c6910 0x7fc2c408e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:41.389 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:41.398 7fc2ca16a700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc2c408ee60 0x7fc2c40b7db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:41.389 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:41.398 7fc2ca16a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc2c40b83d0 0x7fc2c40b2270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:41.389 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:41.398 7fc2ca16a700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2c40b87d0 con 0x7fc2c41c6910 2021-02-10T11:30:41.389 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:41.398 7fc2ca16a700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2c40b89f0 con 0x7fc2c408ee60 2021-02-10T11:30:41.389 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:41.398 7fc2ca16a700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2c40b2890 con 0x7fc2c40b83d0 2021-02-10T11:30:41.389 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:41.398 7fc2c37fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc2c41c6910 0x7fc2c408e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:41.389 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:41.398 7fc2c37fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc2c41c6910 0x7fc2c408e840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:47294/0 (socket says 172.21.2.112:47294) 2021-02-10T11:30:41.389 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:41.398 7fc2c37fe700 1 -- 172.21.2.112:0/2494241137 learned_addr learned my addr 172.21.2.112:0/2494241137 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:30:41.389 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:41.398 7fc2c3fff700 1 --2- 172.21.2.112:0/2494241137 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc2c40b83d0 0x7fc2c40b2270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:41.390 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:41.398 7fc2c2ffd700 1 --2- 172.21.2.112:0/2494241137 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc2c408ee60 0x7fc2c40b7db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:41.390 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:41.398 7fc2c3fff700 1 -- 172.21.2.112:0/2494241137 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc2c408ee60 msgr2=0x7fc2c40b7db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:41.390 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:41.398 7fc2c3fff700 1 --2- 172.21.2.112:0/2494241137 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc2c408ee60 0x7fc2c40b7db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:41.390 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:41.398 7fc2c3fff700 1 -- 172.21.2.112:0/2494241137 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc2c41c6910 msgr2=0x7fc2c408e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:41.390 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:41.398 7fc2c3fff700 1 --2- 172.21.2.112:0/2494241137 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc2c41c6910 0x7fc2c408e840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:41.390 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:41.398 7fc2c3fff700 1 -- 172.21.2.112:0/2494241137 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2c40b2ab0 con 0x7fc2c40b83d0 2021-02-10T11:30:41.390 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:41.398 7fc2c37fe700 1 --2- 172.21.2.112:0/2494241137 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc2c41c6910 0x7fc2c408e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:30:41.390 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:41.399 7fc2c3fff700 1 --2- 172.21.2.112:0/2494241137 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc2c40b83d0 0x7fc2c40b2270 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7fc2bc006fc0 tx=0x7fc2bc00bcf0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:41.390 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:41.399 7fc2bbfff700 1 -- 172.21.2.112:0/2494241137 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc2bc0093d0 con 0x7fc2c40b83d0 2021-02-10T11:30:41.390 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:41.399 7fc2bbfff700 1 -- 172.21.2.112:0/2494241137 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2bc0091d0 con 0x7fc2c40b83d0 2021-02-10T11:30:41.391 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:41.399 7fc2bbfff700 1 -- 172.21.2.112:0/2494241137 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc2bc0012e0 con 0x7fc2c40b83d0 2021-02-10T11:30:41.391 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:41.399 7fc2ca16a700 1 -- 172.21.2.112:0/2494241137 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2c40b2ab0 con 0x7fc2c40b83d0 2021-02-10T11:30:41.391 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:41.399 7fc2ca16a700 1 -- 172.21.2.112:0/2494241137 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2c40b3040 con 0x7fc2c40b83d0 2021-02-10T11:30:41.391 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:41.399 7fc2bbfff700 1 -- 172.21.2.112:0/2494241137 <== mon.2 v2:172.21.2.120:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc2bc0056b0 con 0x7fc2c40b83d0 2021-02-10T11:30:41.391 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:41.399 7fc2bbfff700 1 --2- 172.21.2.112:0/2494241137 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc2a80186c0 0x7fc2a801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:41.391 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:41.399 7fc2bbfff700 1 -- 172.21.2.112:0/2494241137 <== mon.2 v2:172.21.2.120:3301/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fc2bc0327f0 con 0x7fc2c40b83d0 2021-02-10T11:30:41.391 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:41.400 7fc2c37fe700 1 --2- 172.21.2.112:0/2494241137 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc2a80186c0 0x7fc2a801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:41.391 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:41.400 7fc2c37fe700 1 --2- 172.21.2.112:0/2494241137 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc2a80186c0 0x7fc2a801abc0 crc :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:41.418 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:41.431 7fc2ca16a700 1 -- 172.21.2.112:0/2494241137 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2c407ed60 con 0x7fc2c40b83d0 2021-02-10T11:30:41.422 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:41.435 7fc2bbfff700 1 -- 172.21.2.112:0/2494241137 <== mon.2 v2:172.21.2.120:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc2bc0093d0 con 0x7fc2c40b83d0 2021-02-10T11:30:41.554 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:41.566 7fc2ca16a700 1 -- 172.21.2.112:0/2494241137 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348621", "target": ["mgr", ""]}) v1 -- 0x7fc2c40b3040 con 0x7fc2a80186c0 2021-02-10T11:30:41.563 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:41.576 7fc2bbfff700 1 -- 172.21.2.112:0/2494241137 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fc2ac001610 con 0x7fc2a80186c0 2021-02-10T11:30:41.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:41.583 7fc2ca16a700 1 -- 172.21.2.112:0/2494241137 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc2a80186c0 msgr2=0x7fc2a801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:41.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:41.583 7fc2ca16a700 1 --2- 172.21.2.112:0/2494241137 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc2a80186c0 0x7fc2a801abc0 crc :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:41.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:41.583 7fc2ca16a700 1 -- 172.21.2.112:0/2494241137 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc2c40b83d0 msgr2=0x7fc2c40b2270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:41.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:41.583 7fc2ca16a700 1 --2- 172.21.2.112:0/2494241137 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc2c40b83d0 0x7fc2c40b2270 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7fc2bc006fc0 tx=0x7fc2bc00bcf0).stop 2021-02-10T11:30:41.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:41.583 7fc2ca16a700 1 -- 172.21.2.112:0/2494241137 shutdown_connections 2021-02-10T11:30:41.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:41.583 7fc2ca16a700 1 --2- 172.21.2.112:0/2494241137 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc2a80186c0 0x7fc2a801abc0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:41.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:41.583 7fc2ca16a700 1 --2- 172.21.2.112:0/2494241137 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc2c408ee60 0x7fc2c40b7db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:41.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:41.583 7fc2ca16a700 1 --2- 172.21.2.112:0/2494241137 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc2c40b83d0 0x7fc2c40b2270 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:41.572 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:41.583 7fc2ca16a700 1 --2- 172.21.2.112:0/2494241137 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc2c41c6910 0x7fc2c408e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:41.572 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:41.583 7fc2ca16a700 1 -- 172.21.2.112:0/2494241137 >> 172.21.2.112:0/2494241137 conn(0x7fc2c40bb880 msgr2=0x7fc2c40b3360 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:41.572 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:41.584 7fc2ca16a700 1 -- 172.21.2.112:0/2494241137 shutdown_connections 2021-02-10T11:30:41.572 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:41.584 7fc2ca16a700 1 -- 172.21.2.112:0/2494241137 wait complete. 2021-02-10T11:30:41.579 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000348622 2021-02-10T11:30:42.011 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.006 7f736b848700 1 Processor -- start 2021-02-10T11:30:42.011 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.006 7f736b848700 1 -- start start 2021-02-10T11:30:42.011 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.006 7f736b848700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f73640f05f0 0x7f73640f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:42.011 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.006 7f736b848700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73641b0c70 con 0x7f73641b0470 2021-02-10T11:30:42.011 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.006 7f736b848700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73640f0b50 con 0x7f73641b0870 2021-02-10T11:30:42.012 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.006 7f736b848700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73640e5ea0 con 0x7f73640f05f0 2021-02-10T11:30:42.012 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.006 7f7369664700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f73641b0470 0x7f73640ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:46764/0 (socket says 172.21.2.112:46764) 2021-02-10T11:30:42.012 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.007 7f7369664700 1 -- 172.21.2.112:0/43073816 learned_addr learned my addr 172.21.2.112:0/43073816 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:30:42.012 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.007 7f7369e65700 1 --2- 172.21.2.112:0/43073816 >> v2:172.21.2.120:3300/0 conn(0x7f73640f05f0 0x7f73640f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:42.012 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.007 7f7357fff700 1 -- 172.21.2.112:0/43073816 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1321931957 0 0) 0x7f7358000c60 con 0x7f73641b0870 2021-02-10T11:30:42.012 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.007 7f7357fff700 1 -- 172.21.2.112:0/43073816 --> v1:172.21.2.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7350002fe0 con 0x7f73641b0870 2021-02-10T11:30:42.012 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.007 7f7357fff700 1 -- 172.21.2.112:0/43073816 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2523507538 0 0) 0x7f7360001180 con 0x7f73641b0470 2021-02-10T11:30:42.012 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.007 7f7357fff700 1 -- 172.21.2.112:0/43073816 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7350003090 con 0x7f73641b0470 2021-02-10T11:30:42.012 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.007 7f7369e65700 1 -- 172.21.2.112:0/43073816 >> v1:172.21.2.120:6789/0 conn(0x7f73641b0870 legacy=0x7f73640efe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:42.012 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.007 7f7369e65700 1 -- 172.21.2.112:0/43073816 >> v1:172.21.2.112:6789/0 conn(0x7f73641b0470 legacy=0x7f73640ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:42.012 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.007 7f7369e65700 1 -- 172.21.2.112:0/43073816 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73640e52e0 con 0x7f73640f05f0 2021-02-10T11:30:42.012 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.007 7f7369e65700 1 --2- 172.21.2.112:0/43073816 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f73640f05f0 0x7f73640f20e0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7f735c006e50 tx=0x7f735c009df0).ready entity=mon.1 client_cookie=fb1efe437315ddbc server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:42.013 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.008 7f7357fff700 1 -- 172.21.2.112:0/43073816 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f735c00b510 con 0x7f73640f05f0 2021-02-10T11:30:42.013 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.008 7f7357fff700 1 -- 172.21.2.112:0/43073816 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f735c006790 con 0x7f73640f05f0 2021-02-10T11:30:42.013 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.008 7f7357fff700 1 -- 172.21.2.112:0/43073816 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f735c00b750 con 0x7f73640f05f0 2021-02-10T11:30:42.013 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.008 7f736b848700 1 -- 172.21.2.112:0/43073816 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f73640f05f0 msgr2=0x7f73640f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:42.013 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.008 7f736b848700 1 --2- 172.21.2.112:0/43073816 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f73640f05f0 0x7f73640f20e0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7f735c006e50 tx=0x7f735c009df0).stop 2021-02-10T11:30:42.013 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.008 7f736b848700 1 -- 172.21.2.112:0/43073816 shutdown_connections 2021-02-10T11:30:42.013 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.008 7f736b848700 1 --2- 172.21.2.112:0/43073816 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f73640f05f0 0x7f73640f20e0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:42.013 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.008 7f736b848700 1 -- 172.21.2.112:0/43073816 >> 172.21.2.112:0/43073816 conn(0x7f73640ce3e0 msgr2=0x7f73640e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:42.013 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.008 7f736b848700 1 -- 172.21.2.112:0/43073816 shutdown_connections 2021-02-10T11:30:42.013 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.008 7f736b848700 1 -- 172.21.2.112:0/43073816 wait complete. 2021-02-10T11:30:42.015 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.024 7f736b848700 1 Processor -- start 2021-02-10T11:30:42.015 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.024 7f736b848700 1 -- start start 2021-02-10T11:30:42.015 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.024 7f736b848700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f73640a0420 0x7f73640a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:42.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.024 7f736b848700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7364189d00 0x7f736418a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:42.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.024 7f736b848700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f736418a720 0x7f73640b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:42.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.024 7f736b848700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7364189780 con 0x7f736418a720 2021-02-10T11:30:42.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.024 7f736b848700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f736418ab20 con 0x7f73640a0420 2021-02-10T11:30:42.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.024 7f736b848700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73640a1140 con 0x7f7364189d00 2021-02-10T11:30:42.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.024 7f7369e65700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f736418a720 0x7f73640b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:42.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.024 7f7369e65700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f736418a720 0x7f73640b7be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:47310/0 (socket says 172.21.2.112:47310) 2021-02-10T11:30:42.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.024 7f7369e65700 1 -- 172.21.2.112:0/1910370573 learned_addr learned my addr 172.21.2.112:0/1910370573 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:30:42.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.024 7f7369664700 1 --2- 172.21.2.112:0/1910370573 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f73640a0420 0x7f73640a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:42.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.024 7f7368e63700 1 --2- 172.21.2.112:0/1910370573 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7364189d00 0x7f736418a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:42.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.025 7f7369e65700 1 -- 172.21.2.112:0/1910370573 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7364189d00 msgr2=0x7f736418a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:42.017 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.025 7f7369e65700 1 --2- 172.21.2.112:0/1910370573 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7364189d00 0x7f736418a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:42.017 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.025 7f7369e65700 1 -- 172.21.2.112:0/1910370573 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f73640a0420 msgr2=0x7f73640a0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:42.017 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.025 7f7369e65700 1 --2- 172.21.2.112:0/1910370573 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f73640a0420 0x7f73640a0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:42.017 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.025 7f7369e65700 1 -- 172.21.2.112:0/1910370573 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73640b8200 con 0x7f736418a720 2021-02-10T11:30:42.017 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.025 7f7369e65700 1 --2- 172.21.2.112:0/1910370573 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f736418a720 0x7f73640b7be0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 rx=0x7f735c006fa0 tx=0x7f735c00bd90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:42.017 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.025 7f7355ffb700 1 -- 172.21.2.112:0/1910370573 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f735c0093d0 con 0x7f736418a720 2021-02-10T11:30:42.017 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.025 7f7355ffb700 1 -- 172.21.2.112:0/1910370573 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f735c0082e0 con 0x7f736418a720 2021-02-10T11:30:42.017 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.025 7f7355ffb700 1 -- 172.21.2.112:0/1910370573 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f735c008800 con 0x7f736418a720 2021-02-10T11:30:42.017 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.025 7f736b848700 1 -- 172.21.2.112:0/1910370573 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73640b8200 con 0x7f736418a720 2021-02-10T11:30:42.018 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.025 7f736b848700 1 -- 172.21.2.112:0/1910370573 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73640b8200 con 0x7f736418a720 2021-02-10T11:30:42.018 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.025 7f7355ffb700 1 -- 172.21.2.112:0/1910370573 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f735c0084c0 con 0x7f736418a720 2021-02-10T11:30:42.018 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.025 7f7355ffb700 1 --2- 172.21.2.112:0/1910370573 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7344018730 0x7f734401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:42.018 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.025 7f7355ffb700 1 -- 172.21.2.112:0/1910370573 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f735c0329c0 con 0x7f736418a720 2021-02-10T11:30:42.018 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.026 7f7369664700 1 --2- 172.21.2.112:0/1910370573 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7344018730 0x7f734401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:42.018 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.026 7f7369664700 1 --2- 172.21.2.112:0/1910370573 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7344018730 0x7f734401ac10 crc :-1 s=READY pgs=443 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:42.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.055 7f736b848700 1 -- 172.21.2.112:0/1910370573 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7364105ae0 con 0x7f736418a720 2021-02-10T11:30:42.045 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.058 7f7355ffb700 1 -- 172.21.2.112:0/1910370573 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f735c0080d0 con 0x7f736418a720 2021-02-10T11:30:42.173 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.186 7f736b848700 1 -- 172.21.2.112:0/1910370573 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348622", "target": ["mgr", ""]}) v1 -- 0x7f73640f0b50 con 0x7f7344018730 2021-02-10T11:30:42.184 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.197 7f7355ffb700 1 -- 172.21.2.112:0/1910370573 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f7360000ce0 con 0x7f7344018730 2021-02-10T11:30:42.191 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.204 7f736b848700 1 -- 172.21.2.112:0/1910370573 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7344018730 msgr2=0x7f734401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:42.191 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.204 7f736b848700 1 --2- 172.21.2.112:0/1910370573 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7344018730 0x7f734401ac10 crc :-1 s=READY pgs=443 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:42.191 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.204 7f736b848700 1 -- 172.21.2.112:0/1910370573 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f736418a720 msgr2=0x7f73640b7be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:42.191 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.204 7f736b848700 1 --2- 172.21.2.112:0/1910370573 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f736418a720 0x7f73640b7be0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 rx=0x7f735c006fa0 tx=0x7f735c00bd90).stop 2021-02-10T11:30:42.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.204 7f736b848700 1 -- 172.21.2.112:0/1910370573 shutdown_connections 2021-02-10T11:30:42.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.204 7f736b848700 1 --2- 172.21.2.112:0/1910370573 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7344018730 0x7f734401ac10 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:42.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.204 7f736b848700 1 --2- 172.21.2.112:0/1910370573 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f736418a720 0x7f73640b7be0 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:42.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.204 7f736b848700 1 --2- 172.21.2.112:0/1910370573 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7364189d00 0x7f736418a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:42.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.204 7f736b848700 1 --2- 172.21.2.112:0/1910370573 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f73640a0420 0x7f73640a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:42.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.204 7f736b848700 1 -- 172.21.2.112:0/1910370573 >> 172.21.2.112:0/1910370573 conn(0x7f7364094e10 msgr2=0x7f73640a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:42.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.204 7f736b848700 1 -- 172.21.2.112:0/1910370573 shutdown_connections 2021-02-10T11:30:42.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.204 7f736b848700 1 -- 172.21.2.112:0/1910370573 wait complete. 2021-02-10T11:30:42.200 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000348623 2021-02-10T11:30:42.629 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.625 7f24fed9b700 1 Processor -- start 2021-02-10T11:30:42.630 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.625 7f24fed9b700 1 -- start start 2021-02-10T11:30:42.630 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.626 7f24fed9b700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f24f81b0470 0x7f24f81b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:42.630 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.626 7f24fed9b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f24f80f20e0 0x7f24f80f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:42.630 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.626 7f24fed9b700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24f80f0b50 con 0x7f24f80ef610 2021-02-10T11:30:42.630 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.626 7f24fed9b700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24f80f0820 con 0x7f24f81b0470 2021-02-10T11:30:42.630 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.626 7f24fed9b700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24f80f24e0 con 0x7f24f80f20e0 2021-02-10T11:30:42.630 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.626 7f24effff700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f24f80ef610 0x7f24f80efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:46782/0 (socket says 172.21.2.112:46782) 2021-02-10T11:30:42.630 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.626 7f24effff700 1 -- 172.21.2.112:0/3457500470 learned_addr learned my addr 172.21.2.112:0/3457500470 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:30:42.630 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.626 7f24fcbb7700 1 --2- 172.21.2.112:0/3457500470 >> v2:172.21.2.120:3300/0 conn(0x7f24f81b0470 0x7f24f81b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:42.630 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.626 7f24fd3b8700 1 --2- 172.21.2.112:0/3457500470 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f24f80f20e0 0x7f24f80f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:42.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.626 7f24ef7fe700 1 -- 172.21.2.112:0/3457500470 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 761157092 0 0) 0x7f24f40009b0 con 0x7f24f80ef610 2021-02-10T11:30:42.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.626 7f24ef7fe700 1 -- 172.21.2.112:0/3457500470 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24e0002fe0 con 0x7f24f80ef610 2021-02-10T11:30:42.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.626 7f24fd3b8700 1 -- 172.21.2.112:0/3457500470 >> v2:172.21.2.120:3300/0 conn(0x7f24f81b0470 msgr2=0x7f24f81b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:42.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.626 7f24fd3b8700 1 --2- 172.21.2.112:0/3457500470 >> v2:172.21.2.120:3300/0 conn(0x7f24f81b0470 0x7f24f81b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:42.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.626 7f24fd3b8700 1 -- 172.21.2.112:0/3457500470 >> v1:172.21.2.112:6789/0 conn(0x7f24f80ef610 legacy=0x7f24f80efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:42.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.626 7f24fd3b8700 1 -- 172.21.2.112:0/3457500470 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24f80e52e0 con 0x7f24f80f20e0 2021-02-10T11:30:42.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.626 7f24fcbb7700 1 --2- 172.21.2.112:0/3457500470 >> v2:172.21.2.120:3300/0 conn(0x7f24f81b0470 0x7f24f81b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:30:42.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.627 7f24fd3b8700 1 --2- 172.21.2.112:0/3457500470 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f24f80f20e0 0x7f24f80f0200 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f24f0006e50 tx=0x7f24f0009e70).ready entity=mon.2 client_cookie=fa63af11636a6288 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:42.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.627 7f24ef7fe700 1 -- 172.21.2.112:0/3457500470 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f24f0006a20 con 0x7f24f80f20e0 2021-02-10T11:30:42.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.627 7f24ef7fe700 1 -- 172.21.2.112:0/3457500470 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24f0006790 con 0x7f24f80f20e0 2021-02-10T11:30:42.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.627 7f24ef7fe700 1 -- 172.21.2.112:0/3457500470 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f24f0000f90 con 0x7f24f80f20e0 2021-02-10T11:30:42.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.627 7f24fed9b700 1 -- 172.21.2.112:0/3457500470 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f24f80f20e0 msgr2=0x7f24f80f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:42.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.627 7f24fed9b700 1 --2- 172.21.2.112:0/3457500470 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f24f80f20e0 0x7f24f80f0200 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f24f0006e50 tx=0x7f24f0009e70).stop 2021-02-10T11:30:42.632 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.627 7f24fed9b700 1 -- 172.21.2.112:0/3457500470 shutdown_connections 2021-02-10T11:30:42.632 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.627 7f24fed9b700 1 --2- 172.21.2.112:0/3457500470 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f24f80f20e0 0x7f24f80f0200 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:42.632 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.627 7f24fed9b700 1 --2- 172.21.2.112:0/3457500470 >> v2:172.21.2.120:3300/0 conn(0x7f24f81b0470 0x7f24f81b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:42.632 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.627 7f24fed9b700 1 -- 172.21.2.112:0/3457500470 >> 172.21.2.112:0/3457500470 conn(0x7f24f80ce3e0 msgr2=0x7f24f80e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:42.632 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.628 7f24fed9b700 1 -- 172.21.2.112:0/3457500470 shutdown_connections 2021-02-10T11:30:42.632 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.628 7f24fed9b700 1 -- 172.21.2.112:0/3457500470 wait complete. 2021-02-10T11:30:42.632 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.642 7f24fed9b700 1 Processor -- start 2021-02-10T11:30:42.632 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.642 7f24fed9b700 1 -- start start 2021-02-10T11:30:42.635 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.643 7f24fed9b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f24f80a6a70 0x7f24f80a5f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:42.635 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.643 7f24fed9b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f24f80a6e70 0x7f24f80ba3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:42.635 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.643 7f24fed9b700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f24f80b9810 0x7f24f80ba9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:42.636 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.643 7f24fed9b700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24f80b9c10 con 0x7f24f80a6e70 2021-02-10T11:30:42.636 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.643 7f24fed9b700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24f80b9e20 con 0x7f24f80b9810 2021-02-10T11:30:42.636 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.643 7f24fed9b700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24f80a0a80 con 0x7f24f80a6a70 2021-02-10T11:30:42.636 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.643 7f24effff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f24f80a6e70 0x7f24f80ba3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:42.636 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.643 7f24fcbb7700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f24f80a6a70 0x7f24f80a5f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:42.637 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.643 7f24effff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f24f80a6e70 0x7f24f80ba3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:47324/0 (socket says 172.21.2.112:47324) 2021-02-10T11:30:42.637 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.643 7f24effff700 1 -- 172.21.2.112:0/3342697776 learned_addr learned my addr 172.21.2.112:0/3342697776 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:30:42.637 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.643 7f24fd3b8700 1 --2- 172.21.2.112:0/3342697776 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f24f80b9810 0x7f24f80ba9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:42.637 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.643 7f24fcbb7700 1 -- 172.21.2.112:0/3342697776 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f24f80b9810 msgr2=0x7f24f80ba9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:42.637 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.643 7f24fcbb7700 1 --2- 172.21.2.112:0/3342697776 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f24f80b9810 0x7f24f80ba9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:42.637 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.643 7f24fcbb7700 1 -- 172.21.2.112:0/3342697776 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f24f80a6e70 msgr2=0x7f24f80ba3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:42.638 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.643 7f24fcbb7700 1 --2- 172.21.2.112:0/3342697776 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f24f80a6e70 0x7f24f80ba3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:42.638 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.643 7f24fcbb7700 1 -- 172.21.2.112:0/3342697776 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24f80a6540 con 0x7f24f80a6a70 2021-02-10T11:30:42.638 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.644 7f24fcbb7700 1 --2- 172.21.2.112:0/3342697776 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f24f80a6a70 0x7f24f80a5f20 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f24e8009540 tx=0x7f24e800b8e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:42.638 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.644 7f24ed7fa700 1 -- 172.21.2.112:0/3342697776 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f24e800bcd0 con 0x7f24f80a6a70 2021-02-10T11:30:42.638 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.644 7f24ed7fa700 1 -- 172.21.2.112:0/3342697776 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24e8006740 con 0x7f24f80a6a70 2021-02-10T11:30:42.639 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.644 7f24ed7fa700 1 -- 172.21.2.112:0/3342697776 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f24e8002220 con 0x7f24f80a6a70 2021-02-10T11:30:42.639 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.644 7f24fed9b700 1 -- 172.21.2.112:0/3342697776 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24f80a6540 con 0x7f24f80a6a70 2021-02-10T11:30:42.639 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.644 7f24fed9b700 1 -- 172.21.2.112:0/3342697776 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24f80a6540 con 0x7f24f80a6a70 2021-02-10T11:30:42.639 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.644 7f24ed7fa700 1 -- 172.21.2.112:0/3342697776 <== mon.2 v2:172.21.2.120:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f24e8009fe0 con 0x7f24f80a6a70 2021-02-10T11:30:42.639 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.644 7f24ed7fa700 1 --2- 172.21.2.112:0/3342697776 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f24d8018730 0x7f24d801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:42.639 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.644 7f24ed7fa700 1 -- 172.21.2.112:0/3342697776 <== mon.2 v2:172.21.2.120:3301/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f24e8032570 con 0x7f24f80a6a70 2021-02-10T11:30:42.639 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.645 7f24effff700 1 --2- 172.21.2.112:0/3342697776 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f24d8018730 0x7f24d801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:42.640 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.645 7f24effff700 1 --2- 172.21.2.112:0/3342697776 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f24d8018730 0x7f24d801ac10 crc :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:42.662 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.675 7f24fed9b700 1 -- 172.21.2.112:0/3342697776 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24f8105ae0 con 0x7f24f80a6a70 2021-02-10T11:30:42.665 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.678 7f24ed7fa700 1 -- 172.21.2.112:0/3342697776 <== mon.2 v2:172.21.2.120:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f24e8006740 con 0x7f24f80a6a70 2021-02-10T11:30:42.791 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.804 7f24fed9b700 1 -- 172.21.2.112:0/3342697776 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348623", "target": ["mgr", ""]}) v1 -- 0x7f24f80b9c10 con 0x7f24d8018730 2021-02-10T11:30:42.803 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.816 7f24ed7fa700 1 -- 172.21.2.112:0/3342697776 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f24f4000e20 con 0x7f24d8018730 2021-02-10T11:30:42.808 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.820 7f24fed9b700 1 -- 172.21.2.112:0/3342697776 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f24d8018730 msgr2=0x7f24d801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:42.808 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.820 7f24fed9b700 1 --2- 172.21.2.112:0/3342697776 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f24d8018730 0x7f24d801ac10 crc :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:42.808 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.820 7f24fed9b700 1 -- 172.21.2.112:0/3342697776 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f24f80a6a70 msgr2=0x7f24f80a5f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:42.808 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.820 7f24fed9b700 1 --2- 172.21.2.112:0/3342697776 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f24f80a6a70 0x7f24f80a5f20 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f24e8009540 tx=0x7f24e800b8e0).stop 2021-02-10T11:30:42.809 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.821 7f24fed9b700 1 -- 172.21.2.112:0/3342697776 shutdown_connections 2021-02-10T11:30:42.809 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.821 7f24fed9b700 1 --2- 172.21.2.112:0/3342697776 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f24d8018730 0x7f24d801ac10 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:42.809 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.821 7f24fed9b700 1 --2- 172.21.2.112:0/3342697776 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f24f80b9810 0x7f24f80ba9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:42.809 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.821 7f24fed9b700 1 --2- 172.21.2.112:0/3342697776 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f24f80a6e70 0x7f24f80ba3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:42.809 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.821 7f24fed9b700 1 --2- 172.21.2.112:0/3342697776 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f24f80a6a70 0x7f24f80a5f20 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:42.809 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.821 7f24fed9b700 1 -- 172.21.2.112:0/3342697776 >> 172.21.2.112:0/3342697776 conn(0x7f24f80bbf60 msgr2=0x7f24f8105d10 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:42.809 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.821 7f24fed9b700 1 -- 172.21.2.112:0/3342697776 shutdown_connections 2021-02-10T11:30:42.809 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:42.821 7f24fed9b700 1 -- 172.21.2.112:0/3342697776 wait complete. 2021-02-10T11:30:42.818 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000348624 2021-02-10T11:30:43.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.246 7f046be59700 1 Processor -- start 2021-02-10T11:30:43.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.246 7f046be59700 1 -- start start 2021-02-10T11:30:43.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.247 7f046be59700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f04641b0470 0x7f04641b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:43.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.247 7f046be59700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04640f0b50 con 0x7f04640f20e0 2021-02-10T11:30:43.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.247 7f046be59700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04640f24e0 con 0x7f04640ef610 2021-02-10T11:30:43.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.247 7f046be59700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04640e5ea0 con 0x7f04641b0470 2021-02-10T11:30:43.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.247 7f046a476700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f04640f20e0 0x7f04640f0200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:46792/0 (socket says 172.21.2.112:46792) 2021-02-10T11:30:43.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.247 7f046a476700 1 -- 172.21.2.112:0/3071992664 learned_addr learned my addr 172.21.2.112:0/3071992664 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:30:43.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.247 7f0469c75700 1 --2- 172.21.2.112:0/3071992664 >> v2:172.21.2.112:3300/0 conn(0x7f04641b0470 0x7f04641b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:43.252 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.248 7f0468c73700 1 -- 172.21.2.112:0/3071992664 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1811441637 0 0) 0x7f0458000f30 con 0x7f04640ef610 2021-02-10T11:30:43.252 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.248 7f0468c73700 1 -- 172.21.2.112:0/3071992664 --> v1:172.21.2.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0450002fe0 con 0x7f04640ef610 2021-02-10T11:30:43.252 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.248 7f0468c73700 1 -- 172.21.2.112:0/3071992664 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1261062437 0 0) 0x7f045c000f90 con 0x7f04640f20e0 2021-02-10T11:30:43.252 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.248 7f0469c75700 1 -- 172.21.2.112:0/3071992664 >> v1:172.21.2.120:6789/0 conn(0x7f04640ef610 legacy=0x7f04640efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:43.252 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.248 7f0469c75700 1 -- 172.21.2.112:0/3071992664 >> v1:172.21.2.112:6789/0 conn(0x7f04640f20e0 legacy=0x7f04640f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:43.252 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.248 7f0469c75700 1 -- 172.21.2.112:0/3071992664 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04640e52e0 con 0x7f04641b0470 2021-02-10T11:30:43.252 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.248 7f0469c75700 1 --2- 172.21.2.112:0/3071992664 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f04641b0470 0x7f04641b0870 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 rx=0x7f0460006650 tx=0x7f0460009a80).ready entity=mon.0 client_cookie=26b9d42c494a83bd server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:43.252 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.248 7f0468c73700 1 -- 172.21.2.112:0/3071992664 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f04600009b0 con 0x7f04641b0470 2021-02-10T11:30:43.252 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.248 7f0468c73700 1 -- 172.21.2.112:0/3071992664 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f046000c0c0 con 0x7f04641b0470 2021-02-10T11:30:43.252 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.248 7f0468c73700 1 -- 172.21.2.112:0/3071992664 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f046000c520 con 0x7f04641b0470 2021-02-10T11:30:43.252 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.248 7f046be59700 1 -- 172.21.2.112:0/3071992664 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f04641b0470 msgr2=0x7f04641b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:43.252 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.248 7f046be59700 1 --2- 172.21.2.112:0/3071992664 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f04641b0470 0x7f04641b0870 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 rx=0x7f0460006650 tx=0x7f0460009a80).stop 2021-02-10T11:30:43.253 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.249 7f046be59700 1 -- 172.21.2.112:0/3071992664 shutdown_connections 2021-02-10T11:30:43.253 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.249 7f046be59700 1 --2- 172.21.2.112:0/3071992664 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f04641b0470 0x7f04641b0870 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:43.253 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.249 7f046be59700 1 -- 172.21.2.112:0/3071992664 >> 172.21.2.112:0/3071992664 conn(0x7f04640ce3e0 msgr2=0x7f04640e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:43.253 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.249 7f046be59700 1 -- 172.21.2.112:0/3071992664 shutdown_connections 2021-02-10T11:30:43.253 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.249 7f046be59700 1 -- 172.21.2.112:0/3071992664 wait complete. 2021-02-10T11:30:43.255 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.264 7f046be59700 1 Processor -- start 2021-02-10T11:30:43.255 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.264 7f046be59700 1 -- start start 2021-02-10T11:30:43.255 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.264 7f046be59700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f04640bb980 0x7f046419bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:43.255 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.264 7f046be59700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f046419c140 0x7f04640acf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:43.255 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.264 7f046be59700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f04640ad570 0x7f04640a78f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:43.255 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.264 7f046be59700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04640c3e00 con 0x7f04640bb980 2021-02-10T11:30:43.255 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.264 7f046be59700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04640ad970 con 0x7f04640ad570 2021-02-10T11:30:43.255 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.264 7f046be59700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04640adcb0 con 0x7f046419c140 2021-02-10T11:30:43.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.265 7f0469c75700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f04640bb980 0x7f046419bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:43.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.265 7f0469c75700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f04640bb980 0x7f046419bb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:47336/0 (socket says 172.21.2.112:47336) 2021-02-10T11:30:43.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.265 7f0469c75700 1 -- 172.21.2.112:0/1758061637 learned_addr learned my addr 172.21.2.112:0/1758061637 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:30:43.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.265 7f046a476700 1 --2- 172.21.2.112:0/1758061637 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f04640ad570 0x7f04640a78f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:43.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.265 7f0469474700 1 --2- 172.21.2.112:0/1758061637 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f046419c140 0x7f04640acf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:43.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.265 7f046a476700 1 -- 172.21.2.112:0/1758061637 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f046419c140 msgr2=0x7f04640acf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:43.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.265 7f046a476700 1 --2- 172.21.2.112:0/1758061637 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f046419c140 0x7f04640acf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:43.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.265 7f046a476700 1 -- 172.21.2.112:0/1758061637 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f04640bb980 msgr2=0x7f046419bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:43.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.265 7f046a476700 1 --2- 172.21.2.112:0/1758061637 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f04640bb980 0x7f046419bb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:43.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.265 7f046a476700 1 -- 172.21.2.112:0/1758061637 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04640a7f10 con 0x7f04640ad570 2021-02-10T11:30:43.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.265 7f046a476700 1 --2- 172.21.2.112:0/1758061637 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f04640ad570 0x7f04640a78f0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7f045c002240 tx=0x7f045c00ac50).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:43.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.266 7f04567fc700 1 -- 172.21.2.112:0/1758061637 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f045c004930 con 0x7f04640ad570 2021-02-10T11:30:43.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.266 7f04567fc700 1 -- 172.21.2.112:0/1758061637 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f045c004c70 con 0x7f04640ad570 2021-02-10T11:30:43.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.266 7f04567fc700 1 -- 172.21.2.112:0/1758061637 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f045c007350 con 0x7f04640ad570 2021-02-10T11:30:43.257 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.266 7f046be59700 1 -- 172.21.2.112:0/1758061637 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04640a7f10 con 0x7f04640ad570 2021-02-10T11:30:43.257 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.266 7f046be59700 1 -- 172.21.2.112:0/1758061637 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04640a7f10 con 0x7f04640ad570 2021-02-10T11:30:43.257 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.266 7f04567fc700 1 -- 172.21.2.112:0/1758061637 <== mon.1 v2:172.21.2.120:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f045c004720 con 0x7f04640ad570 2021-02-10T11:30:43.257 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.266 7f04567fc700 1 --2- 172.21.2.112:0/1758061637 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f044c018730 0x7f044c01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:43.257 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.266 7f04567fc700 1 -- 172.21.2.112:0/1758061637 <== mon.1 v2:172.21.2.120:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f045c032a50 con 0x7f04640ad570 2021-02-10T11:30:43.257 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.267 7f0469c75700 1 --2- 172.21.2.112:0/1758061637 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f044c018730 0x7f044c01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:43.257 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.267 7f0469c75700 1 --2- 172.21.2.112:0/1758061637 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f044c018730 0x7f044c01ac10 crc :-1 s=READY pgs=445 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:43.283 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.296 7f046be59700 1 -- 172.21.2.112:0/1758061637 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0464105af0 con 0x7f04640ad570 2021-02-10T11:30:43.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.299 7f04567fc700 1 -- 172.21.2.112:0/1758061637 <== mon.1 v2:172.21.2.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f045c001270 con 0x7f04640ad570 2021-02-10T11:30:43.415 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.427 7f046be59700 1 -- 172.21.2.112:0/1758061637 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348624", "target": ["mgr", ""]}) v1 -- 0x7f0464105af0 con 0x7f044c018730 2021-02-10T11:30:43.426 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.439 7f04567fc700 1 -- 172.21.2.112:0/1758061637 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f0460006530 con 0x7f044c018730 2021-02-10T11:30:43.433 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.444 7f046be59700 1 -- 172.21.2.112:0/1758061637 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f044c018730 msgr2=0x7f044c01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:43.433 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.444 7f046be59700 1 --2- 172.21.2.112:0/1758061637 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f044c018730 0x7f044c01ac10 crc :-1 s=READY pgs=445 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:43.433 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.444 7f046be59700 1 -- 172.21.2.112:0/1758061637 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f04640ad570 msgr2=0x7f04640a78f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:43.433 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.444 7f046be59700 1 --2- 172.21.2.112:0/1758061637 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f04640ad570 0x7f04640a78f0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7f045c002240 tx=0x7f045c00ac50).stop 2021-02-10T11:30:43.433 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.445 7f046be59700 1 -- 172.21.2.112:0/1758061637 shutdown_connections 2021-02-10T11:30:43.433 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.445 7f046be59700 1 --2- 172.21.2.112:0/1758061637 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f044c018730 0x7f044c01ac10 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:43.433 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.445 7f046be59700 1 --2- 172.21.2.112:0/1758061637 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f04640ad570 0x7f04640a78f0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:43.433 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.445 7f046be59700 1 --2- 172.21.2.112:0/1758061637 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f046419c140 0x7f04640acf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:43.433 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.445 7f046be59700 1 --2- 172.21.2.112:0/1758061637 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f04640bb980 0x7f046419bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:43.433 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.445 7f046be59700 1 -- 172.21.2.112:0/1758061637 >> 172.21.2.112:0/1758061637 conn(0x7f04640bf4e0 msgr2=0x7f04640bfdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:43.433 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.445 7f046be59700 1 -- 172.21.2.112:0/1758061637 shutdown_connections 2021-02-10T11:30:43.433 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.445 7f046be59700 1 -- 172.21.2.112:0/1758061637 wait complete. 2021-02-10T11:30:43.441 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000348625 2021-02-10T11:30:43.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.869 7fea143d5700 1 Processor -- start 2021-02-10T11:30:43.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.869 7fea143d5700 1 -- start start 2021-02-10T11:30:43.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.869 7fea143d5700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fea0c1b0470 0x7fea0c1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:43.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.869 7fea143d5700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fea0c0f20e0 0x7fea0c0f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:43.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.869 7fea143d5700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea0c0f0b50 con 0x7fea0c0ef610 2021-02-10T11:30:43.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.869 7fea143d5700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea0c0f0820 con 0x7fea0c0f20e0 2021-02-10T11:30:43.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.869 7fea143d5700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea0c0f24e0 con 0x7fea0c1b0470 2021-02-10T11:30:43.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.869 7fea121f1700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fea0c1b0470 0x7fea0c1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:43.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.869 7fea129f2700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fea0c0f20e0 0x7fea0c0f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:43.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.869 7fea119f0700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fea0c0ef610 0x7fea0c0efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:46806/0 (socket says 172.21.2.112:46806) 2021-02-10T11:30:43.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.869 7fea129f2700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fea0c0f20e0 0x7fea0c0f0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:47346/0 (socket says 172.21.2.112:47346) 2021-02-10T11:30:43.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.869 7fea119f0700 1 -- 172.21.2.112:0/2186121877 learned_addr learned my addr 172.21.2.112:0/2186121877 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:30:43.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.869 7fea121f1700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fea0c1b0470 0x7fea0c1b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3300/0 says I am v2:172.21.2.112:49622/0 (socket says 172.21.2.112:49622) 2021-02-10T11:30:43.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.870 7fea129f2700 1 -- 172.21.2.112:0/2186121877 >> v2:172.21.2.120:3300/0 conn(0x7fea0c1b0470 msgr2=0x7fea0c1b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:43.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.870 7fea129f2700 1 --2- 172.21.2.112:0/2186121877 >> v2:172.21.2.120:3300/0 conn(0x7fea0c1b0470 0x7fea0c1b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:43.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.870 7fea129f2700 1 -- 172.21.2.112:0/2186121877 >> v1:172.21.2.112:6789/0 conn(0x7fea0c0ef610 legacy=0x7fea0c0efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:43.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.870 7fea129f2700 1 -- 172.21.2.112:0/2186121877 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea0c0e52e0 con 0x7fea0c0f20e0 2021-02-10T11:30:43.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.870 7fea129f2700 1 --2- 172.21.2.112:0/2186121877 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fea0c0f20e0 0x7fea0c0f0200 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 rx=0x7fea08006ab0 tx=0x7fea08009b40).ready entity=mon.0 client_cookie=6c593c64fbb4db94 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:43.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.870 7fea111ef700 1 -- 172.21.2.112:0/2186121877 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fea080022e0 con 0x7fea0c0f20e0 2021-02-10T11:30:43.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.870 7fea111ef700 1 -- 172.21.2.112:0/2186121877 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea08002660 con 0x7fea0c0f20e0 2021-02-10T11:30:43.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.870 7fea111ef700 1 -- 172.21.2.112:0/2186121877 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fea08002ac0 con 0x7fea0c0f20e0 2021-02-10T11:30:43.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.870 7fea143d5700 1 -- 172.21.2.112:0/2186121877 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fea0c0f20e0 msgr2=0x7fea0c0f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:43.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.870 7fea143d5700 1 --2- 172.21.2.112:0/2186121877 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fea0c0f20e0 0x7fea0c0f0200 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 rx=0x7fea08006ab0 tx=0x7fea08009b40).stop 2021-02-10T11:30:43.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.871 7fea143d5700 1 -- 172.21.2.112:0/2186121877 shutdown_connections 2021-02-10T11:30:43.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.871 7fea143d5700 1 --2- 172.21.2.112:0/2186121877 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fea0c0f20e0 0x7fea0c0f0200 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:43.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.871 7fea143d5700 1 --2- 172.21.2.112:0/2186121877 >> v2:172.21.2.120:3300/0 conn(0x7fea0c1b0470 0x7fea0c1b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:43.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.871 7fea143d5700 1 -- 172.21.2.112:0/2186121877 >> 172.21.2.112:0/2186121877 conn(0x7fea0c0ce3e0 msgr2=0x7fea0c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:43.875 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.871 7fea143d5700 1 -- 172.21.2.112:0/2186121877 shutdown_connections 2021-02-10T11:30:43.875 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.871 7fea143d5700 1 -- 172.21.2.112:0/2186121877 wait complete. 2021-02-10T11:30:43.876 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.885 7fea143d5700 1 Processor -- start 2021-02-10T11:30:43.876 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.885 7fea143d5700 1 -- start start 2021-02-10T11:30:43.876 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.886 7fea143d5700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fea0c0a0420 0x7fea0c0a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:43.876 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.886 7fea143d5700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fea0c189d00 0x7fea0c18a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:43.876 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.886 7fea143d5700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fea0c18a720 0x7fea0c0b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:43.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.886 7fea143d5700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea0c189800 con 0x7fea0c189d00 2021-02-10T11:30:43.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.886 7fea143d5700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea0c18ab20 con 0x7fea0c0a0420 2021-02-10T11:30:43.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.886 7fea143d5700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea0c0a1140 con 0x7fea0c18a720 2021-02-10T11:30:43.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.886 7fea121f1700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fea0c0a0420 0x7fea0c0a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:43.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.886 7fea119f0700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fea0c189d00 0x7fea0c18a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:43.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.886 7fea121f1700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fea0c0a0420 0x7fea0c0a0820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3300/0 says I am v2:172.21.2.112:49628/0 (socket says 172.21.2.112:49628) 2021-02-10T11:30:43.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.886 7fea121f1700 1 -- 172.21.2.112:0/1968248780 learned_addr learned my addr 172.21.2.112:0/1968248780 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:30:43.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.886 7fea129f2700 1 --2- 172.21.2.112:0/1968248780 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fea0c18a720 0x7fea0c0b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:43.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.886 7fea121f1700 1 -- 172.21.2.112:0/1968248780 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fea0c18a720 msgr2=0x7fea0c0b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:43.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.886 7fea121f1700 1 --2- 172.21.2.112:0/1968248780 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fea0c18a720 0x7fea0c0b7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:43.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.886 7fea121f1700 1 -- 172.21.2.112:0/1968248780 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fea0c189d00 msgr2=0x7fea0c18a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:43.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.886 7fea121f1700 1 --2- 172.21.2.112:0/1968248780 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fea0c189d00 0x7fea0c18a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:43.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.886 7fea121f1700 1 -- 172.21.2.112:0/1968248780 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea0c0b8200 con 0x7fea0c0a0420 2021-02-10T11:30:43.878 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.887 7fea121f1700 1 --2- 172.21.2.112:0/1968248780 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fea0c0a0420 0x7fea0c0a0820 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 rx=0x7fea04009fc0 tx=0x7fea0400bcd0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:43.878 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.887 7fea02ffd700 1 -- 172.21.2.112:0/1968248780 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fea04006030 con 0x7fea0c0a0420 2021-02-10T11:30:43.878 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.887 7fea02ffd700 1 -- 172.21.2.112:0/1968248780 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea04006330 con 0x7fea0c0a0420 2021-02-10T11:30:43.878 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.887 7fea02ffd700 1 -- 172.21.2.112:0/1968248780 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fea040058e0 con 0x7fea0c0a0420 2021-02-10T11:30:43.878 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.887 7fea143d5700 1 -- 172.21.2.112:0/1968248780 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea0c0b8200 con 0x7fea0c0a0420 2021-02-10T11:30:43.878 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.887 7fea143d5700 1 -- 172.21.2.112:0/1968248780 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea0c0b8790 con 0x7fea0c0a0420 2021-02-10T11:30:43.878 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.887 7fea02ffd700 1 -- 172.21.2.112:0/1968248780 <== mon.1 v2:172.21.2.120:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fea04006030 con 0x7fea0c0a0420 2021-02-10T11:30:43.878 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.887 7fea02ffd700 1 --2- 172.21.2.112:0/1968248780 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe9f40186c0 0x7fe9f401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:43.878 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.887 7fea02ffd700 1 -- 172.21.2.112:0/1968248780 <== mon.1 v2:172.21.2.120:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fea040327b0 con 0x7fea0c0a0420 2021-02-10T11:30:43.878 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.888 7fea119f0700 1 --2- 172.21.2.112:0/1968248780 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe9f40186c0 0x7fe9f401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:43.878 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.888 7fea119f0700 1 --2- 172.21.2.112:0/1968248780 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe9f40186c0 0x7fe9f401abc0 crc :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:43.906 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.918 7fea143d5700 1 -- 172.21.2.112:0/1968248780 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea0c18ab20 con 0x7fea0c0a0420 2021-02-10T11:30:43.909 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:43.922 7fea02ffd700 1 -- 172.21.2.112:0/1968248780 <== mon.1 v2:172.21.2.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fea0400be10 con 0x7fea0c0a0420 2021-02-10T11:30:44.012 DEBUG:teuthology.orchestra.run.gibba012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:30:44.014 DEBUG:teuthology.orchestra.run.gibba020:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:30:44.047 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:44.058 7fea143d5700 1 -- 172.21.2.112:0/1968248780 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348625", "target": ["mgr", ""]}) v1 -- 0x7fea0c18ab20 con 0x7fe9f40186c0 2021-02-10T11:30:44.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:44.070 7fea02ffd700 1 -- 172.21.2.112:0/1968248780 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fe9fc001a10 con 0x7fe9f40186c0 2021-02-10T11:30:44.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:44.076 7fea143d5700 1 -- 172.21.2.112:0/1968248780 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe9f40186c0 msgr2=0x7fe9f401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:44.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:44.076 7fea143d5700 1 --2- 172.21.2.112:0/1968248780 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe9f40186c0 0x7fe9f401abc0 crc :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:44.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:44.076 7fea143d5700 1 -- 172.21.2.112:0/1968248780 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fea0c0a0420 msgr2=0x7fea0c0a0820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:44.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:44.076 7fea143d5700 1 --2- 172.21.2.112:0/1968248780 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fea0c0a0420 0x7fea0c0a0820 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 rx=0x7fea04009fc0 tx=0x7fea0400bcd0).stop 2021-02-10T11:30:44.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:44.076 7fea143d5700 1 -- 172.21.2.112:0/1968248780 shutdown_connections 2021-02-10T11:30:44.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:44.076 7fea143d5700 1 --2- 172.21.2.112:0/1968248780 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe9f40186c0 0x7fe9f401abc0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:44.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:44.076 7fea143d5700 1 --2- 172.21.2.112:0/1968248780 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fea0c18a720 0x7fea0c0b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:44.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:44.076 7fea143d5700 1 --2- 172.21.2.112:0/1968248780 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fea0c189d00 0x7fea0c18a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:44.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:44.076 7fea143d5700 1 --2- 172.21.2.112:0/1968248780 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fea0c0a0420 0x7fea0c0a0820 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:44.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:44.076 7fea143d5700 1 -- 172.21.2.112:0/1968248780 >> 172.21.2.112:0/1968248780 conn(0x7fea0c094e10 msgr2=0x7fea0c0a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:44.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:44.076 7fea143d5700 1 -- 172.21.2.112:0/1968248780 shutdown_connections 2021-02-10T11:30:44.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:44.076 7fea143d5700 1 -- 172.21.2.112:0/1968248780 wait complete. 2021-02-10T11:30:44.073 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000348626 2021-02-10T11:30:44.443 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:44.437 7f49c611a700 1 Processor -- start 2021-02-10T11:30:44.443 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:44.437 7f49c611a700 1 -- start start 2021-02-10T11:30:44.443 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:44.437 7f49c611a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f49c00f05f0 0x7f49c00f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:44.443 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:44.437 7f49c611a700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49c01b0c70 con 0x7f49c01b0470 2021-02-10T11:30:44.443 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:44.437 7f49c611a700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49c00f0b50 con 0x7f49c01b0870 2021-02-10T11:30:44.443 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:44.437 7f49c611a700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49c00e5ea0 con 0x7f49c00f05f0 2021-02-10T11:30:44.443 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:44.437 7f49bf7fe700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f49c01b0470 0x7f49c00ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:46818/0 (socket says 172.21.2.112:46818) 2021-02-10T11:30:44.443 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:44.437 7f49bf7fe700 1 -- 172.21.2.112:0/1744768766 learned_addr learned my addr 172.21.2.112:0/1744768766 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:30:44.443 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:44.438 7f49be7fc700 1 -- 172.21.2.112:0/1744768766 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1178295929 0 0) 0x7f49a80010b0 con 0x7f49c01b0470 2021-02-10T11:30:44.444 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:44.438 7f49be7fc700 1 -- 172.21.2.112:0/1744768766 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49ac002fe0 con 0x7f49c01b0470 2021-02-10T11:30:44.444 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:44.438 7f49bffff700 1 --2- 172.21.2.112:0/1744768766 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f49c00f05f0 0x7f49c00f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:44.444 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:44.438 7f49be7fc700 1 -- 172.21.2.112:0/1744768766 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2697250750 0 0) 0x7f49a8001010 con 0x7f49c01b0470 2021-02-10T11:30:44.444 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:44.438 7f49be7fc700 1 -- 172.21.2.112:0/1744768766 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f49c00f05f0 msgr2=0x7f49c00f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:44.444 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:44.438 7f49be7fc700 1 --2- 172.21.2.112:0/1744768766 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f49c00f05f0 0x7f49c00f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:44.444 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:44.438 7f49be7fc700 1 -- 172.21.2.112:0/1744768766 >> v1:172.21.2.120:6789/0 conn(0x7f49c01b0870 legacy=0x7f49c00efe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:44.444 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:44.438 7f49be7fc700 1 -- 172.21.2.112:0/1744768766 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49c00e52e0 con 0x7f49c01b0470 2021-02-10T11:30:44.444 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:44.438 7f49be7fc700 1 -- 172.21.2.112:0/1744768766 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f49a8001530 con 0x7f49c01b0470 2021-02-10T11:30:44.444 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:44.438 7f49be7fc700 1 -- 172.21.2.112:0/1744768766 >> v1:172.21.2.112:6789/0 conn(0x7f49c01b0470 legacy=0x7f49c00ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:44.444 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:44.438 7f49be7fc700 1 --2- 172.21.2.112:0/1744768766 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f49ac003a70 0x7f49ac005ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:44.444 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:44.438 7f49be7fc700 1 --2- 172.21.2.112:0/1744768766 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f49ac006740 0x7f49ac008c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:44.444 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:44.438 7f49be7fc700 1 --2- 172.21.2.112:0/1744768766 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f49ac009500 0x7f49ac00ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:44.444 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:44.438 7f49be7fc700 1 -- 172.21.2.112:0/1744768766 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49ac00c2d0 con 0x7f49ac009500 2021-02-10T11:30:44.445 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:44.438 7f49be7fc700 1 -- 172.21.2.112:0/1744768766 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49ac00c4f0 con 0x7f49ac006740 2021-02-10T11:30:44.445 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:44.438 7f49be7fc700 1 -- 172.21.2.112:0/1744768766 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49ac00c710 con 0x7f49ac003a70 2021-02-10T11:30:44.445 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:44.438 7f49beffd700 1 --2- 172.21.2.112:0/1744768766 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f49ac006740 0x7f49ac008c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:44.445 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:44.438 7f49bffff700 1 --2- 172.21.2.112:0/1744768766 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f49ac009500 0x7f49ac00ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:44.445 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:44.438 7f49bf7fe700 1 --2- 172.21.2.112:0/1744768766 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f49ac003a70 0x7f49ac005ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:44.445 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:44.438 7f49beffd700 1 -- 172.21.2.112:0/1744768766 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f49ac003a70 msgr2=0x7f49ac005ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:44.445 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:44.438 7f49beffd700 1 --2- 172.21.2.112:0/1744768766 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f49ac003a70 0x7f49ac005ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:44.445 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:44.438 7f49beffd700 1 -- 172.21.2.112:0/1744768766 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f49ac009500 msgr2=0x7f49ac00ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:44.445 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:44.438 7f49beffd700 1 --2- 172.21.2.112:0/1744768766 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f49ac009500 0x7f49ac00ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:44.445 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:44.438 7f49beffd700 1 -- 172.21.2.112:0/1744768766 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49ac050950 con 0x7f49ac006740 2021-02-10T11:30:44.445 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:44.438 7f49bffff700 1 --2- 172.21.2.112:0/1744768766 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f49ac009500 0x7f49ac00ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:30:44.445 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:44.439 7f49beffd700 1 --2- 172.21.2.112:0/1744768766 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f49ac006740 0x7f49ac008c60 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 rx=0x7f49b0006650 tx=0x7f49b0009a80).ready entity=mon.1 client_cookie=3901aa6bd4b144fe server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:44.446 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:44.439 7f49be7fc700 1 -- 172.21.2.112:0/1744768766 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f49b000c0c0 con 0x7f49ac006740 2021-02-10T11:30:44.446 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:44.439 7f49be7fc700 1 -- 172.21.2.112:0/1744768766 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49b000c340 con 0x7f49ac006740 2021-02-10T11:30:44.446 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:44.439 7f49be7fc700 1 -- 172.21.2.112:0/1744768766 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f49b000c790 con 0x7f49ac006740 2021-02-10T11:30:44.446 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:44.439 7f49c611a700 1 -- 172.21.2.112:0/1744768766 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f49ac006740 msgr2=0x7f49ac008c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:44.446 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:44.439 7f49c611a700 1 --2- 172.21.2.112:0/1744768766 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f49ac006740 0x7f49ac008c60 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 rx=0x7f49b0006650 tx=0x7f49b0009a80).stop 2021-02-10T11:30:44.446 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:44.439 7f49bffff700 1 -- 172.21.2.112:0/1744768766 reap_dead start 2021-02-10T11:30:44.446 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:44.439 7f49c611a700 1 -- 172.21.2.112:0/1744768766 shutdown_connections 2021-02-10T11:30:44.446 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:44.439 7f49c611a700 1 -- 172.21.2.112:0/1744768766 >> 172.21.2.112:0/1744768766 conn(0x7f49c00ce3e0 msgr2=0x7f49c00e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:44.446 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:44.439 7f49c611a700 1 -- 172.21.2.112:0/1744768766 shutdown_connections 2021-02-10T11:30:44.446 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:44.439 7f49c611a700 1 -- 172.21.2.112:0/1744768766 wait complete. 2021-02-10T11:30:44.446 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:44.454 7f49c611a700 1 Processor -- start 2021-02-10T11:30:44.446 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:44.454 7f49c611a700 1 -- start start 2021-02-10T11:30:44.448 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:44.455 7f49c611a700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f49c010a540 0x7f49c00a86f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:44.448 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:44.455 7f49c611a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f49c0190a70 0x7f49c00a8d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:44.448 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:44.455 7f49c611a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f49c00a0ac0 0x7f49c00a99b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:44.448 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:44.455 7f49c611a700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49c00a9430 con 0x7f49c0190a70 2021-02-10T11:30:44.448 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:44.455 7f49c611a700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49c0190630 con 0x7f49c010a540 2021-02-10T11:30:44.448 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:44.455 7f49c611a700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49c00a9fd0 con 0x7f49c00a0ac0 2021-02-10T11:30:44.448 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:44.455 7f49beffd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f49c0190a70 0x7f49c00a8d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:44.448 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:44.455 7f49beffd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f49c0190a70 0x7f49c00a8d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:47370/0 (socket says 172.21.2.112:47370) 2021-02-10T11:30:44.448 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:44.455 7f49beffd700 1 -- 172.21.2.112:0/2218795816 learned_addr learned my addr 172.21.2.112:0/2218795816 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:30:44.448 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:44.455 7f49bf7fe700 1 --2- 172.21.2.112:0/2218795816 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f49c010a540 0x7f49c00a86f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:44.449 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:44.455 7f49bffff700 1 --2- 172.21.2.112:0/2218795816 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f49c00a0ac0 0x7f49c00a99b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:44.449 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:44.455 7f49beffd700 1 -- 172.21.2.112:0/2218795816 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f49c00a0ac0 msgr2=0x7f49c00a99b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:44.449 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:44.455 7f49beffd700 1 --2- 172.21.2.112:0/2218795816 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f49c00a0ac0 0x7f49c00a99b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:44.449 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:44.455 7f49beffd700 1 -- 172.21.2.112:0/2218795816 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f49c010a540 msgr2=0x7f49c00a86f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:44.449 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:44.455 7f49beffd700 1 --2- 172.21.2.112:0/2218795816 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f49c010a540 0x7f49c00a86f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:44.449 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:44.455 7f49beffd700 1 -- 172.21.2.112:0/2218795816 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49c00aa1f0 con 0x7f49c0190a70 2021-02-10T11:30:44.449 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:44.455 7f49beffd700 1 --2- 172.21.2.112:0/2218795816 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f49c0190a70 0x7f49c00a8d10 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 rx=0x7f49b000aee0 tx=0x7f49b00025d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:44.449 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:44.455 7f49b7fff700 1 -- 172.21.2.112:0/2218795816 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f49b0009030 con 0x7f49c0190a70 2021-02-10T11:30:44.449 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:44.455 7f49b7fff700 1 -- 172.21.2.112:0/2218795816 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49b000b070 con 0x7f49c0190a70 2021-02-10T11:30:44.449 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:44.455 7f49b7fff700 1 -- 172.21.2.112:0/2218795816 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f49b000b5d0 con 0x7f49c0190a70 2021-02-10T11:30:44.449 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:44.455 7f49c611a700 1 -- 172.21.2.112:0/2218795816 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49c00aa1f0 con 0x7f49c0190a70 2021-02-10T11:30:44.449 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:44.455 7f49c611a700 1 -- 172.21.2.112:0/2218795816 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49c00aa1f0 con 0x7f49c0190a70 2021-02-10T11:30:44.449 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:44.456 7f49b7fff700 1 -- 172.21.2.112:0/2218795816 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f49b00029c0 con 0x7f49c0190a70 2021-02-10T11:30:44.450 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:44.456 7f49b7fff700 1 --2- 172.21.2.112:0/2218795816 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f49a4018730 0x7f49a401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:44.450 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:44.456 7f49b7fff700 1 -- 172.21.2.112:0/2218795816 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f49b0032600 con 0x7f49c0190a70 2021-02-10T11:30:44.450 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:44.456 7f49bf7fe700 1 --2- 172.21.2.112:0/2218795816 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f49a4018730 0x7f49a401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:44.450 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:44.456 7f49bf7fe700 1 --2- 172.21.2.112:0/2218795816 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f49a4018730 0x7f49a401ac10 crc :-1 s=READY pgs=447 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:44.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:44.486 7f49c611a700 1 -- 172.21.2.112:0/2218795816 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49c0105b30 con 0x7f49c0190a70 2021-02-10T11:30:44.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:44.489 7f49b7fff700 1 -- 172.21.2.112:0/2218795816 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f49b0009030 con 0x7f49c0190a70 2021-02-10T11:30:44.608 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:44.620 7f49c611a700 1 -- 172.21.2.112:0/2218795816 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348626", "target": ["mgr", ""]}) v1 -- 0x7f49c00a9430 con 0x7f49a4018730 2021-02-10T11:30:44.619 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:44.631 7f49b7fff700 1 -- 172.21.2.112:0/2218795816 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f49a80015d0 con 0x7f49a4018730 2021-02-10T11:30:44.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:44.636 7f49c611a700 1 -- 172.21.2.112:0/2218795816 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f49a4018730 msgr2=0x7f49a401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:44.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:44.636 7f49c611a700 1 --2- 172.21.2.112:0/2218795816 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f49a4018730 0x7f49a401ac10 crc :-1 s=READY pgs=447 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:44.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:44.636 7f49c611a700 1 -- 172.21.2.112:0/2218795816 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f49c0190a70 msgr2=0x7f49c00a8d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:44.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:44.636 7f49c611a700 1 --2- 172.21.2.112:0/2218795816 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f49c0190a70 0x7f49c00a8d10 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 rx=0x7f49b000aee0 tx=0x7f49b00025d0).stop 2021-02-10T11:30:44.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:44.637 7f49c611a700 1 -- 172.21.2.112:0/2218795816 shutdown_connections 2021-02-10T11:30:44.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:44.637 7f49c611a700 1 --2- 172.21.2.112:0/2218795816 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f49a4018730 0x7f49a401ac10 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:44.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:44.637 7f49c611a700 1 --2- 172.21.2.112:0/2218795816 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f49c00a0ac0 0x7f49c00a99b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:44.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:44.637 7f49c611a700 1 --2- 172.21.2.112:0/2218795816 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f49c0190a70 0x7f49c00a8d10 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:44.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:44.637 7f49c611a700 1 --2- 172.21.2.112:0/2218795816 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f49c010a540 0x7f49c00a86f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:44.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:44.637 7f49c611a700 1 -- 172.21.2.112:0/2218795816 >> 172.21.2.112:0/2218795816 conn(0x7f49c00c40e0 msgr2=0x7f49c0094a30 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:44.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:44.637 7f49c611a700 1 -- 172.21.2.112:0/2218795816 shutdown_connections 2021-02-10T11:30:44.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:44.637 7f49c611a700 1 -- 172.21.2.112:0/2218795816 wait complete. 2021-02-10T11:30:44.634 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000348627 2021-02-10T11:30:45.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.061 7f7040565700 1 Processor -- start 2021-02-10T11:30:45.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.061 7f7040565700 1 -- start start 2021-02-10T11:30:45.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.062 7f7040565700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f70381b0470 0x7f70381b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:45.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.062 7f7040565700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70380f0b50 con 0x7f70380f20e0 2021-02-10T11:30:45.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.062 7f7040565700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70380f24e0 con 0x7f70380ef610 2021-02-10T11:30:45.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.062 7f7040565700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70380e5ea0 con 0x7f70381b0470 2021-02-10T11:30:45.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.062 7f703eb82700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f70380f20e0 0x7f70380f0200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:46838/0 (socket says 172.21.2.112:46838) 2021-02-10T11:30:45.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.062 7f703eb82700 1 -- 172.21.2.112:0/1000436533 learned_addr learned my addr 172.21.2.112:0/1000436533 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:30:45.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.062 7f703e381700 1 --2- 172.21.2.112:0/1000436533 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f70381b0470 0x7f70381b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:45.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.062 7f703d37f700 1 -- 172.21.2.112:0/1000436533 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 676802206 0 0) 0x7f7030001500 con 0x7f70380f20e0 2021-02-10T11:30:45.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.062 7f703d37f700 1 -- 172.21.2.112:0/1000436533 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7024002fe0 con 0x7f70380f20e0 2021-02-10T11:30:45.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.062 7f703d37f700 1 -- 172.21.2.112:0/1000436533 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 697170406 0 0) 0x7f702c000ff0 con 0x7f70380ef610 2021-02-10T11:30:45.068 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.062 7f703e381700 1 -- 172.21.2.112:0/1000436533 >> v1:172.21.2.120:6789/0 conn(0x7f70380ef610 legacy=0x7f70380efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:45.068 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.062 7f703e381700 1 -- 172.21.2.112:0/1000436533 >> v1:172.21.2.112:6789/0 conn(0x7f70380f20e0 legacy=0x7f70380f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:45.068 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.062 7f703e381700 1 -- 172.21.2.112:0/1000436533 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70380e52e0 con 0x7f70381b0470 2021-02-10T11:30:45.068 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.063 7f703e381700 1 --2- 172.21.2.112:0/1000436533 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f70381b0470 0x7f70381b0870 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f7034006650 tx=0x7f7034009a80).ready entity=mon.2 client_cookie=2dab03f4b675f451 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:45.068 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.063 7f703d37f700 1 -- 172.21.2.112:0/1000436533 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f70340009b0 con 0x7f70381b0470 2021-02-10T11:30:45.068 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.063 7f703d37f700 1 -- 172.21.2.112:0/1000436533 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f703400c0c0 con 0x7f70381b0470 2021-02-10T11:30:45.068 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.063 7f703d37f700 1 -- 172.21.2.112:0/1000436533 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f703400c5e0 con 0x7f70381b0470 2021-02-10T11:30:45.068 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.063 7f7040565700 1 -- 172.21.2.112:0/1000436533 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f70381b0470 msgr2=0x7f70381b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:45.068 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.063 7f7040565700 1 --2- 172.21.2.112:0/1000436533 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f70381b0470 0x7f70381b0870 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f7034006650 tx=0x7f7034009a80).stop 2021-02-10T11:30:45.068 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.064 7f7040565700 1 -- 172.21.2.112:0/1000436533 shutdown_connections 2021-02-10T11:30:45.068 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.064 7f7040565700 1 --2- 172.21.2.112:0/1000436533 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f70381b0470 0x7f70381b0870 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:45.068 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.064 7f7040565700 1 -- 172.21.2.112:0/1000436533 >> 172.21.2.112:0/1000436533 conn(0x7f70380ce3e0 msgr2=0x7f70380e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:45.068 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.064 7f7040565700 1 -- 172.21.2.112:0/1000436533 shutdown_connections 2021-02-10T11:30:45.068 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.064 7f7040565700 1 -- 172.21.2.112:0/1000436533 wait complete. 2021-02-10T11:30:45.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.078 7f7040565700 1 Processor -- start 2021-02-10T11:30:45.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.079 7f7040565700 1 -- start start 2021-02-10T11:30:45.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.079 7f7040565700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f70380b20b0 0x7f70380ac970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:45.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.079 7f7040565700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f70380b24b0 0x7f7038191dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:45.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.079 7f7040565700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f70380acf90 0x7f70381923e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:45.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.079 7f7040565700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70380b1b30 con 0x7f70380b24b0 2021-02-10T11:30:45.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.079 7f7040565700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70381a04e0 con 0x7f70380b20b0 2021-02-10T11:30:45.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.079 7f7040565700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70381a06c0 con 0x7f70380acf90 2021-02-10T11:30:45.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.079 7f703db80700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f70380b24b0 0x7f7038191dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:45.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.079 7f703db80700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f70380b24b0 0x7f7038191dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:47382/0 (socket says 172.21.2.112:47382) 2021-02-10T11:30:45.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.079 7f703db80700 1 -- 172.21.2.112:0/2833637024 learned_addr learned my addr 172.21.2.112:0/2833637024 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:30:45.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.079 7f703eb82700 1 --2- 172.21.2.112:0/2833637024 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f70380acf90 0x7f70381923e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:45.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.079 7f703e381700 1 --2- 172.21.2.112:0/2833637024 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f70380b20b0 0x7f70380ac970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:45.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.079 7f703db80700 1 -- 172.21.2.112:0/2833637024 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f70380acf90 msgr2=0x7f70381923e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:45.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.079 7f703db80700 1 --2- 172.21.2.112:0/2833637024 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f70380acf90 0x7f70381923e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:45.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.079 7f703db80700 1 -- 172.21.2.112:0/2833637024 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f70380b20b0 msgr2=0x7f70380ac970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:45.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.079 7f703db80700 1 --2- 172.21.2.112:0/2833637024 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f70380b20b0 0x7f70380ac970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:45.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.079 7f703db80700 1 -- 172.21.2.112:0/2833637024 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7038192be0 con 0x7f70380b24b0 2021-02-10T11:30:45.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.079 7f703eb82700 1 --2- 172.21.2.112:0/2833637024 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f70380acf90 0x7f70381923e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:30:45.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.080 7f703db80700 1 --2- 172.21.2.112:0/2833637024 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f70380b24b0 0x7f7038191dc0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 rx=0x7f702c006ed0 tx=0x7f702c00b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:45.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.080 7f702affd700 1 -- 172.21.2.112:0/2833637024 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f702c00bd50 con 0x7f70380b24b0 2021-02-10T11:30:45.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.080 7f702affd700 1 -- 172.21.2.112:0/2833637024 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f702c006240 con 0x7f70380b24b0 2021-02-10T11:30:45.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.080 7f702affd700 1 -- 172.21.2.112:0/2833637024 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f702c005730 con 0x7f70380b24b0 2021-02-10T11:30:45.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.080 7f7040565700 1 -- 172.21.2.112:0/2833637024 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7038192be0 con 0x7f70380b24b0 2021-02-10T11:30:45.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.080 7f7040565700 1 -- 172.21.2.112:0/2833637024 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7038193170 con 0x7f70380b24b0 2021-02-10T11:30:45.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.081 7f702affd700 1 -- 172.21.2.112:0/2833637024 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f702c0054b0 con 0x7f70380b24b0 2021-02-10T11:30:45.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.081 7f702affd700 1 --2- 172.21.2.112:0/2833637024 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f70200186c0 0x7f702001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:45.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.081 7f702affd700 1 -- 172.21.2.112:0/2833637024 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f702c032590 con 0x7f70380b24b0 2021-02-10T11:30:45.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.081 7f703e381700 1 --2- 172.21.2.112:0/2833637024 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f70200186c0 0x7f702001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:45.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.081 7f703e381700 1 --2- 172.21.2.112:0/2833637024 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f70200186c0 0x7f702001abc0 crc :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:45.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.111 7f7040565700 1 -- 172.21.2.112:0/2833637024 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7038105af0 con 0x7f70380b24b0 2021-02-10T11:30:45.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.114 7f702affd700 1 -- 172.21.2.112:0/2833637024 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f702c0223c0 con 0x7f70380b24b0 2021-02-10T11:30:45.233 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.245 7f7040565700 1 -- 172.21.2.112:0/2833637024 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348627", "target": ["mgr", ""]}) v1 -- 0x7f70380f0b50 con 0x7f70200186c0 2021-02-10T11:30:45.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.256 7f702affd700 1 -- 172.21.2.112:0/2833637024 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f7034006420 con 0x7f70200186c0 2021-02-10T11:30:45.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.262 7f7040565700 1 -- 172.21.2.112:0/2833637024 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f70200186c0 msgr2=0x7f702001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:45.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.262 7f7040565700 1 --2- 172.21.2.112:0/2833637024 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f70200186c0 0x7f702001abc0 crc :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:45.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.262 7f7040565700 1 -- 172.21.2.112:0/2833637024 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f70380b24b0 msgr2=0x7f7038191dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:45.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.262 7f7040565700 1 --2- 172.21.2.112:0/2833637024 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f70380b24b0 0x7f7038191dc0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 rx=0x7f702c006ed0 tx=0x7f702c00b960).stop 2021-02-10T11:30:45.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.263 7f7040565700 1 -- 172.21.2.112:0/2833637024 shutdown_connections 2021-02-10T11:30:45.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.263 7f7040565700 1 --2- 172.21.2.112:0/2833637024 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f70200186c0 0x7f702001abc0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:45.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.263 7f7040565700 1 --2- 172.21.2.112:0/2833637024 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f70380acf90 0x7f70381923e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:45.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.263 7f7040565700 1 --2- 172.21.2.112:0/2833637024 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f70380b24b0 0x7f7038191dc0 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:45.252 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.263 7f7040565700 1 --2- 172.21.2.112:0/2833637024 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f70380b20b0 0x7f70380ac970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:45.252 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.263 7f7040565700 1 -- 172.21.2.112:0/2833637024 >> 172.21.2.112:0/2833637024 conn(0x7f70380bf4e0 msgr2=0x7f70380be3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:45.252 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.263 7f7040565700 1 -- 172.21.2.112:0/2833637024 shutdown_connections 2021-02-10T11:30:45.252 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.263 7f7040565700 1 -- 172.21.2.112:0/2833637024 wait complete. 2021-02-10T11:30:45.259 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000348628 2021-02-10T11:30:45.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.686 7fe6dc2ae700 1 Processor -- start 2021-02-10T11:30:45.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.686 7fe6dc2ae700 1 -- start start 2021-02-10T11:30:45.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.687 7fe6dc2ae700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe6d41b0870 0x7fe6d40f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:45.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.687 7fe6dc2ae700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fe6d40efe00 0x7fe6d40f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:45.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.687 7fe6dc2ae700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6d41b0c70 con 0x7fe6d41b0470 2021-02-10T11:30:45.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.687 7fe6dc2ae700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6d40f0b50 con 0x7fe6d40efe00 2021-02-10T11:30:45.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.687 7fe6dc2ae700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6d40f0820 con 0x7fe6d41b0870 2021-02-10T11:30:45.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.687 7fe6da8cb700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fe6d40efe00 0x7fe6d40f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:45.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.687 7fe6da0ca700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7fe6d41b0470 0x7fe6d40ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:40410/0 (socket says 172.21.2.112:40410) 2021-02-10T11:30:45.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.687 7fe6da0ca700 1 -- 172.21.2.112:0/651499103 learned_addr learned my addr 172.21.2.112:0/651499103 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:30:45.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.687 7fe6d98c9700 1 --2- 172.21.2.112:0/651499103 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe6d41b0870 0x7fe6d40f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:45.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.687 7fe6da8cb700 1 -- 172.21.2.112:0/651499103 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe6d41b0870 msgr2=0x7fe6d40f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:45.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.687 7fe6da8cb700 1 --2- 172.21.2.112:0/651499103 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe6d41b0870 0x7fe6d40f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:45.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.687 7fe6da8cb700 1 -- 172.21.2.112:0/651499103 >> v1:172.21.2.120:6789/0 conn(0x7fe6d41b0470 legacy=0x7fe6d40ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:45.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.687 7fe6da8cb700 1 -- 172.21.2.112:0/651499103 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6d40e52e0 con 0x7fe6d40efe00 2021-02-10T11:30:45.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.687 7fe6da8cb700 1 --2- 172.21.2.112:0/651499103 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe6d40efe00 0x7fe6d40f0200 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 rx=0x7fe6cc006e30 tx=0x7fe6cc009e60).ready entity=mon.0 client_cookie=ddee347c0bba55a0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:45.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.688 7fe6d90c8700 1 -- 172.21.2.112:0/651499103 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe6cc006a50 con 0x7fe6d40efe00 2021-02-10T11:30:45.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.688 7fe6d90c8700 1 -- 172.21.2.112:0/651499103 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6cc00b300 con 0x7fe6d40efe00 2021-02-10T11:30:45.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.688 7fe6d90c8700 1 -- 172.21.2.112:0/651499103 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe6cc00b820 con 0x7fe6d40efe00 2021-02-10T11:30:45.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.688 7fe6dc2ae700 1 -- 172.21.2.112:0/651499103 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe6d40efe00 msgr2=0x7fe6d40f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:45.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.688 7fe6dc2ae700 1 --2- 172.21.2.112:0/651499103 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe6d40efe00 0x7fe6d40f0200 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 rx=0x7fe6cc006e30 tx=0x7fe6cc009e60).stop 2021-02-10T11:30:45.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.688 7fe6dc2ae700 1 -- 172.21.2.112:0/651499103 shutdown_connections 2021-02-10T11:30:45.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.688 7fe6dc2ae700 1 --2- 172.21.2.112:0/651499103 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe6d40efe00 0x7fe6d40f0200 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:45.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.688 7fe6dc2ae700 1 --2- 172.21.2.112:0/651499103 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe6d41b0870 0x7fe6d40f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:45.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.688 7fe6dc2ae700 1 -- 172.21.2.112:0/651499103 >> 172.21.2.112:0/651499103 conn(0x7fe6d40ce3e0 msgr2=0x7fe6d40e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:45.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.689 7fe6dc2ae700 1 -- 172.21.2.112:0/651499103 shutdown_connections 2021-02-10T11:30:45.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.689 7fe6dc2ae700 1 -- 172.21.2.112:0/651499103 wait complete. 2021-02-10T11:30:45.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.703 7fe6dc2ae700 1 Processor -- start 2021-02-10T11:30:45.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.703 7fe6dc2ae700 1 -- start start 2021-02-10T11:30:45.695 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.704 7fe6dc2ae700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe6d4105d10 0x7fe6d409eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:45.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.704 7fe6dc2ae700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe6d4106110 0x7fe6d409f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:45.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.704 7fe6dc2ae700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe6d40b2a00 0x7fe6d40b2e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:45.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.704 7fe6dc2ae700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6d409e560 con 0x7fe6d4106110 2021-02-10T11:30:45.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.704 7fe6dc2ae700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6d409c090 con 0x7fe6d40b2a00 2021-02-10T11:30:45.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.704 7fe6dc2ae700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6d40b3420 con 0x7fe6d4105d10 2021-02-10T11:30:45.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.704 7fe6d98c9700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe6d4106110 0x7fe6d409f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:45.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.704 7fe6d98c9700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe6d4106110 0x7fe6d409f100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:47398/0 (socket says 172.21.2.112:47398) 2021-02-10T11:30:45.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.704 7fe6d98c9700 1 -- 172.21.2.112:0/163288040 learned_addr learned my addr 172.21.2.112:0/163288040 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:30:45.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.704 7fe6da8cb700 1 --2- 172.21.2.112:0/163288040 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe6d40b2a00 0x7fe6d40b2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:45.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.704 7fe6da0ca700 1 --2- 172.21.2.112:0/163288040 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe6d4105d10 0x7fe6d409eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:45.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.704 7fe6da8cb700 1 -- 172.21.2.112:0/163288040 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe6d4105d10 msgr2=0x7fe6d409eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:45.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.704 7fe6da8cb700 1 --2- 172.21.2.112:0/163288040 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe6d4105d10 0x7fe6d409eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:45.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.704 7fe6da8cb700 1 -- 172.21.2.112:0/163288040 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe6d4106110 msgr2=0x7fe6d409f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:45.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.704 7fe6da8cb700 1 --2- 172.21.2.112:0/163288040 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe6d4106110 0x7fe6d409f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:45.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.704 7fe6da8cb700 1 -- 172.21.2.112:0/163288040 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6d40b3640 con 0x7fe6d40b2a00 2021-02-10T11:30:45.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.704 7fe6d98c9700 1 --2- 172.21.2.112:0/163288040 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe6d4106110 0x7fe6d409f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:30:45.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.705 7fe6da8cb700 1 --2- 172.21.2.112:0/163288040 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe6d40b2a00 0x7fe6d40b2e00 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7fe6cc00b8f0 tx=0x7fe6cc00bc90).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:45.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.705 7fe6c6ffd700 1 -- 172.21.2.112:0/163288040 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe6cc00bdd0 con 0x7fe6d40b2a00 2021-02-10T11:30:45.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.705 7fe6c6ffd700 1 -- 172.21.2.112:0/163288040 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6cc005820 con 0x7fe6d40b2a00 2021-02-10T11:30:45.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.705 7fe6c6ffd700 1 -- 172.21.2.112:0/163288040 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe6cc005cc0 con 0x7fe6d40b2a00 2021-02-10T11:30:45.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.705 7fe6dc2ae700 1 -- 172.21.2.112:0/163288040 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6d40b3640 con 0x7fe6d40b2a00 2021-02-10T11:30:45.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.705 7fe6dc2ae700 1 -- 172.21.2.112:0/163288040 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6d40b3640 con 0x7fe6d40b2a00 2021-02-10T11:30:45.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.705 7fe6c6ffd700 1 -- 172.21.2.112:0/163288040 <== mon.1 v2:172.21.2.120:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe6cc005a00 con 0x7fe6d40b2a00 2021-02-10T11:30:45.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.705 7fe6c6ffd700 1 --2- 172.21.2.112:0/163288040 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe6bc0186c0 0x7fe6bc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:45.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.705 7fe6c6ffd700 1 -- 172.21.2.112:0/163288040 <== mon.1 v2:172.21.2.120:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fe6cc032740 con 0x7fe6d40b2a00 2021-02-10T11:30:45.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.706 7fe6da0ca700 1 --2- 172.21.2.112:0/163288040 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe6bc0186c0 0x7fe6bc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:45.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.706 7fe6da0ca700 1 --2- 172.21.2.112:0/163288040 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe6bc0186c0 0x7fe6bc01abc0 crc :-1 s=READY pgs=449 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:45.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.736 7fe6dc2ae700 1 -- 172.21.2.112:0/163288040 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6d40b3ec0 con 0x7fe6d40b2a00 2021-02-10T11:30:45.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.740 7fe6c6ffd700 1 -- 172.21.2.112:0/163288040 <== mon.1 v2:172.21.2.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe6cc00bdd0 con 0x7fe6d40b2a00 2021-02-10T11:30:45.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.870 7fe6dc2ae700 1 -- 172.21.2.112:0/163288040 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348628", "target": ["mgr", ""]}) v1 -- 0x7fe6d41a16c0 con 0x7fe6bc0186c0 2021-02-10T11:30:45.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.881 7fe6c6ffd700 1 -- 172.21.2.112:0/163288040 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fe6d00010b0 con 0x7fe6bc0186c0 2021-02-10T11:30:45.876 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.888 7fe6dc2ae700 1 -- 172.21.2.112:0/163288040 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe6bc0186c0 msgr2=0x7fe6bc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:45.876 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.888 7fe6dc2ae700 1 --2- 172.21.2.112:0/163288040 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe6bc0186c0 0x7fe6bc01abc0 crc :-1 s=READY pgs=449 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:45.876 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.888 7fe6dc2ae700 1 -- 172.21.2.112:0/163288040 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe6d40b2a00 msgr2=0x7fe6d40b2e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:45.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.888 7fe6dc2ae700 1 --2- 172.21.2.112:0/163288040 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe6d40b2a00 0x7fe6d40b2e00 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7fe6cc00b8f0 tx=0x7fe6cc00bc90).stop 2021-02-10T11:30:45.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.888 7fe6dc2ae700 1 -- 172.21.2.112:0/163288040 shutdown_connections 2021-02-10T11:30:45.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.888 7fe6dc2ae700 1 --2- 172.21.2.112:0/163288040 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe6bc0186c0 0x7fe6bc01abc0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:45.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.888 7fe6dc2ae700 1 --2- 172.21.2.112:0/163288040 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe6d40b2a00 0x7fe6d40b2e00 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:45.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.888 7fe6dc2ae700 1 --2- 172.21.2.112:0/163288040 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe6d4106110 0x7fe6d409f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:45.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.888 7fe6dc2ae700 1 --2- 172.21.2.112:0/163288040 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe6d4105d10 0x7fe6d409eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:45.878 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.888 7fe6dc2ae700 1 -- 172.21.2.112:0/163288040 >> 172.21.2.112:0/163288040 conn(0x7fe6d40c36f0 msgr2=0x7fe6d40bd640 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:45.878 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.888 7fe6dc2ae700 1 -- 172.21.2.112:0/163288040 shutdown_connections 2021-02-10T11:30:45.878 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:45.888 7fe6dc2ae700 1 -- 172.21.2.112:0/163288040 wait complete. 2021-02-10T11:30:45.885 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000348629 2021-02-10T11:30:46.314 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.309 7ff8368f8700 1 Processor -- start 2021-02-10T11:30:46.314 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.310 7ff8368f8700 1 -- start start 2021-02-10T11:30:46.315 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.310 7ff8368f8700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7ff8301b0470 0x7ff8301b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:46.315 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.310 7ff8368f8700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff8300f20e0 0x7ff8300f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:46.315 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.310 7ff8368f8700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8300f0b50 con 0x7ff8300ef610 2021-02-10T11:30:46.315 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.310 7ff8368f8700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8300f0820 con 0x7ff8301b0470 2021-02-10T11:30:46.315 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.310 7ff8368f8700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8300f24e0 con 0x7ff8300f20e0 2021-02-10T11:30:46.315 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.310 7ff82ffff700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7ff8301b0470 0x7ff8301b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:46.315 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.310 7ff82ffff700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7ff8301b0470 0x7ff8301b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3300/0 says I am v2:172.21.2.112:49682/0 (socket says 172.21.2.112:49682) 2021-02-10T11:30:46.315 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.310 7ff82ffff700 1 -- 172.21.2.112:0/2264083018 learned_addr learned my addr 172.21.2.112:0/2264083018 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:30:46.315 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.310 7ff834f15700 1 --2- 172.21.2.112:0/2264083018 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff8300f20e0 0x7ff8300f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:46.316 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.311 7ff82ffff700 1 -- 172.21.2.112:0/2264083018 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff8300f20e0 msgr2=0x7ff8300f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:46.316 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.311 7ff82ffff700 1 --2- 172.21.2.112:0/2264083018 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff8300f20e0 0x7ff8300f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:46.316 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.311 7ff82ffff700 1 -- 172.21.2.112:0/2264083018 >> v1:172.21.2.120:6789/0 conn(0x7ff8300ef610 legacy=0x7ff8300efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:46.316 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.311 7ff82ffff700 1 -- 172.21.2.112:0/2264083018 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8300e52e0 con 0x7ff8301b0470 2021-02-10T11:30:46.316 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.311 7ff82ffff700 1 --2- 172.21.2.112:0/2264083018 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff8301b0470 0x7ff8301b0870 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7ff820006700 tx=0x7ff820009b80).ready entity=mon.1 client_cookie=3b69231a795b7566 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:46.316 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.311 7ff82effd700 1 -- 172.21.2.112:0/2264083018 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff82000c0c0 con 0x7ff8301b0470 2021-02-10T11:30:46.316 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.311 7ff82effd700 1 -- 172.21.2.112:0/2264083018 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff82000c400 con 0x7ff8301b0470 2021-02-10T11:30:46.316 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.311 7ff8368f8700 1 -- 172.21.2.112:0/2264083018 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff8301b0470 msgr2=0x7ff8301b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:46.316 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.311 7ff8368f8700 1 --2- 172.21.2.112:0/2264083018 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff8301b0470 0x7ff8301b0870 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7ff820006700 tx=0x7ff820009b80).stop 2021-02-10T11:30:46.316 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.312 7ff8368f8700 1 -- 172.21.2.112:0/2264083018 shutdown_connections 2021-02-10T11:30:46.316 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.312 7ff8368f8700 1 --2- 172.21.2.112:0/2264083018 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff8300f20e0 0x7ff8300f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:46.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.312 7ff8368f8700 1 --2- 172.21.2.112:0/2264083018 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff8301b0470 0x7ff8301b0870 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:46.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.312 7ff8368f8700 1 -- 172.21.2.112:0/2264083018 >> 172.21.2.112:0/2264083018 conn(0x7ff8300ce3e0 msgr2=0x7ff8300e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:46.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.312 7ff8368f8700 1 -- 172.21.2.112:0/2264083018 shutdown_connections 2021-02-10T11:30:46.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.312 7ff8368f8700 1 -- 172.21.2.112:0/2264083018 wait complete. 2021-02-10T11:30:46.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.326 7ff8368f8700 1 Processor -- start 2021-02-10T11:30:46.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.326 7ff8368f8700 1 -- start start 2021-02-10T11:30:46.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.327 7ff8368f8700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff8300b20b0 0x7ff8300ac970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:46.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.327 7ff8368f8700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff8300b24b0 0x7ff830191dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:46.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.327 7ff8368f8700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff8300acf90 0x7ff8301923e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:46.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.327 7ff8368f8700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8300b1b30 con 0x7ff8300b20b0 2021-02-10T11:30:46.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.327 7ff8368f8700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8301a04e0 con 0x7ff8300b24b0 2021-02-10T11:30:46.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.327 7ff8368f8700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8301a06c0 con 0x7ff8300acf90 2021-02-10T11:30:46.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.327 7ff82ffff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff8300b20b0 0x7ff8300ac970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:46.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.327 7ff82ffff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff8300b20b0 0x7ff8300ac970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:47408/0 (socket says 172.21.2.112:47408) 2021-02-10T11:30:46.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.327 7ff82ffff700 1 -- 172.21.2.112:0/3382422418 learned_addr learned my addr 172.21.2.112:0/3382422418 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:30:46.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.327 7ff82f7fe700 1 --2- 172.21.2.112:0/3382422418 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff8300b24b0 0x7ff830191dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:46.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.327 7ff834f15700 1 --2- 172.21.2.112:0/3382422418 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff8300acf90 0x7ff8301923e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:46.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.328 7ff82ffff700 1 -- 172.21.2.112:0/3382422418 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff8300acf90 msgr2=0x7ff8301923e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:46.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.328 7ff82ffff700 1 --2- 172.21.2.112:0/3382422418 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff8300acf90 0x7ff8301923e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:46.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.328 7ff82ffff700 1 -- 172.21.2.112:0/3382422418 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff8300b24b0 msgr2=0x7ff830191dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:46.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.328 7ff82ffff700 1 --2- 172.21.2.112:0/3382422418 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff8300b24b0 0x7ff830191dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:46.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.328 7ff82ffff700 1 -- 172.21.2.112:0/3382422418 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff830192be0 con 0x7ff8300b20b0 2021-02-10T11:30:46.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.328 7ff82ffff700 1 --2- 172.21.2.112:0/3382422418 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff8300b20b0 0x7ff8300ac970 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 rx=0x7ff8200066a0 tx=0x7ff82000b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:46.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.328 7ff82cff9700 1 -- 172.21.2.112:0/3382422418 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff82000bd80 con 0x7ff8300b20b0 2021-02-10T11:30:46.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.328 7ff82cff9700 1 -- 172.21.2.112:0/3382422418 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8200052e0 con 0x7ff8300b20b0 2021-02-10T11:30:46.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.328 7ff82cff9700 1 -- 172.21.2.112:0/3382422418 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff820005770 con 0x7ff8300b20b0 2021-02-10T11:30:46.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.328 7ff8368f8700 1 -- 172.21.2.112:0/3382422418 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff830192be0 con 0x7ff8300b20b0 2021-02-10T11:30:46.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.328 7ff8368f8700 1 -- 172.21.2.112:0/3382422418 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff830192be0 con 0x7ff8300b20b0 2021-02-10T11:30:46.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.329 7ff82cff9700 1 -- 172.21.2.112:0/3382422418 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff820009440 con 0x7ff8300b20b0 2021-02-10T11:30:46.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.329 7ff82cff9700 1 --2- 172.21.2.112:0/3382422418 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff8100186c0 0x7ff81001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:46.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.329 7ff82cff9700 1 -- 172.21.2.112:0/3382422418 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7ff820032600 con 0x7ff8300b20b0 2021-02-10T11:30:46.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.329 7ff82f7fe700 1 --2- 172.21.2.112:0/3382422418 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff8100186c0 0x7ff81001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:46.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.330 7ff82f7fe700 1 --2- 172.21.2.112:0/3382422418 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff8100186c0 0x7ff81001abc0 crc :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:46.349 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.360 7ff8368f8700 1 -- 172.21.2.112:0/3382422418 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff830105af0 con 0x7ff8300b20b0 2021-02-10T11:30:46.350 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.362 7ff82cff9700 1 -- 172.21.2.112:0/3382422418 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff820006030 con 0x7ff8300b20b0 2021-02-10T11:30:46.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.491 7ff8368f8700 1 -- 172.21.2.112:0/3382422418 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348629", "target": ["mgr", ""]}) v1 -- 0x7ff8300f0b50 con 0x7ff8100186c0 2021-02-10T11:30:46.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.503 7ff82cff9700 1 -- 172.21.2.112:0/3382422418 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7ff8240054b0 con 0x7ff8100186c0 2021-02-10T11:30:46.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.509 7ff8368f8700 1 -- 172.21.2.112:0/3382422418 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff8100186c0 msgr2=0x7ff81001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:46.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.509 7ff8368f8700 1 --2- 172.21.2.112:0/3382422418 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff8100186c0 0x7ff81001abc0 crc :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:46.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.509 7ff8368f8700 1 -- 172.21.2.112:0/3382422418 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff8300b20b0 msgr2=0x7ff8300ac970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:46.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.509 7ff8368f8700 1 --2- 172.21.2.112:0/3382422418 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff8300b20b0 0x7ff8300ac970 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 rx=0x7ff8200066a0 tx=0x7ff82000b990).stop 2021-02-10T11:30:46.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.509 7ff8368f8700 1 -- 172.21.2.112:0/3382422418 shutdown_connections 2021-02-10T11:30:46.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.509 7ff8368f8700 1 --2- 172.21.2.112:0/3382422418 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff8100186c0 0x7ff81001abc0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:46.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.509 7ff8368f8700 1 --2- 172.21.2.112:0/3382422418 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff8300b24b0 0x7ff830191dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:46.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.509 7ff8368f8700 1 --2- 172.21.2.112:0/3382422418 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff8300acf90 0x7ff8301923e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:46.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.509 7ff8368f8700 1 --2- 172.21.2.112:0/3382422418 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff8300b20b0 0x7ff8300ac970 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:46.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.509 7ff8368f8700 1 -- 172.21.2.112:0/3382422418 >> 172.21.2.112:0/3382422418 conn(0x7ff8300bf4e0 msgr2=0x7ff8300be3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:46.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.509 7ff8368f8700 1 -- 172.21.2.112:0/3382422418 shutdown_connections 2021-02-10T11:30:46.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.509 7ff8368f8700 1 -- 172.21.2.112:0/3382422418 wait complete. 2021-02-10T11:30:46.506 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000348630 2021-02-10T11:30:46.935 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.931 7f5a205b7700 1 Processor -- start 2021-02-10T11:30:46.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.931 7f5a205b7700 1 -- start start 2021-02-10T11:30:46.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.931 7f5a205b7700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f5a181b0870 0x7f5a180f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:46.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.931 7f5a205b7700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f5a180efe00 0x7f5a180f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:46.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.931 7f5a205b7700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a181b0c70 con 0x7f5a181b0470 2021-02-10T11:30:46.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.931 7f5a205b7700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a180e5ea0 con 0x7f5a180efe00 2021-02-10T11:30:46.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.931 7f5a205b7700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a180f0b50 con 0x7f5a181b0870 2021-02-10T11:30:46.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.931 7f5a1ebd4700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f5a180efe00 0x7f5a180f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:46.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.932 7f5a1ebd4700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f5a180efe00 0x7f5a180f0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:47418/0 (socket says 172.21.2.112:47418) 2021-02-10T11:30:46.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.932 7f5a1ebd4700 1 -- 172.21.2.112:0/2543180110 learned_addr learned my addr 172.21.2.112:0/2543180110 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:30:46.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.932 7f5a1dbd2700 1 --2- 172.21.2.112:0/2543180110 >> v2:172.21.2.120:3300/0 conn(0x7f5a181b0870 0x7f5a180f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:46.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.932 7f5a1ebd4700 1 -- 172.21.2.112:0/2543180110 >> v2:172.21.2.120:3300/0 conn(0x7f5a181b0870 msgr2=0x7f5a180f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:46.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.932 7f5a1ebd4700 1 --2- 172.21.2.112:0/2543180110 >> v2:172.21.2.120:3300/0 conn(0x7f5a181b0870 0x7f5a180f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:46.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.932 7f5a1ebd4700 1 -- 172.21.2.112:0/2543180110 >> v1:172.21.2.120:6789/0 conn(0x7f5a181b0470 legacy=0x7f5a180ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:46.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.932 7f5a1ebd4700 1 -- 172.21.2.112:0/2543180110 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a180d2fb0 con 0x7f5a180efe00 2021-02-10T11:30:46.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.932 7f5a1ebd4700 1 --2- 172.21.2.112:0/2543180110 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5a180efe00 0x7f5a180f0200 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 rx=0x7f5a10006b80 tx=0x7f5a10009b90).ready entity=mon.0 client_cookie=7007b59b542acc9c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:46.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.932 7f5a1d3d1700 1 -- 172.21.2.112:0/2543180110 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5a100022e0 con 0x7f5a180efe00 2021-02-10T11:30:46.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.932 7f5a1d3d1700 1 -- 172.21.2.112:0/2543180110 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a10002620 con 0x7f5a180efe00 2021-02-10T11:30:46.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.932 7f5a1d3d1700 1 -- 172.21.2.112:0/2543180110 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5a10002a80 con 0x7f5a180efe00 2021-02-10T11:30:46.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.932 7f5a205b7700 1 -- 172.21.2.112:0/2543180110 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5a180efe00 msgr2=0x7f5a180f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:46.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.932 7f5a205b7700 1 --2- 172.21.2.112:0/2543180110 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5a180efe00 0x7f5a180f0200 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 rx=0x7f5a10006b80 tx=0x7f5a10009b90).stop 2021-02-10T11:30:46.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.933 7f5a205b7700 1 -- 172.21.2.112:0/2543180110 shutdown_connections 2021-02-10T11:30:46.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.933 7f5a205b7700 1 --2- 172.21.2.112:0/2543180110 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5a180efe00 0x7f5a180f0200 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:46.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.933 7f5a205b7700 1 --2- 172.21.2.112:0/2543180110 >> v2:172.21.2.120:3300/0 conn(0x7f5a181b0870 0x7f5a180f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:46.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.933 7f5a205b7700 1 -- 172.21.2.112:0/2543180110 >> 172.21.2.112:0/2543180110 conn(0x7f5a180ce3e0 msgr2=0x7f5a180e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:46.938 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.933 7f5a205b7700 1 -- 172.21.2.112:0/2543180110 shutdown_connections 2021-02-10T11:30:46.938 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.933 7f5a205b7700 1 -- 172.21.2.112:0/2543180110 wait complete. 2021-02-10T11:30:46.940 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.947 7f5a205b7700 1 Processor -- start 2021-02-10T11:30:46.940 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.947 7f5a205b7700 1 -- start start 2021-02-10T11:30:46.940 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.948 7f5a205b7700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5a18190920 0x7f5a180b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:46.940 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.948 7f5a205b7700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5a180b8200 0x7f5a180b8600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:46.940 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.948 7f5a205b7700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5a180c1d80 0x7f5a180c2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:46.940 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.948 7f5a205b7700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a18190d20 con 0x7f5a180b8200 2021-02-10T11:30:46.940 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.948 7f5a205b7700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a180b8c20 con 0x7f5a18190920 2021-02-10T11:30:46.940 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.948 7f5a205b7700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a180c27a0 con 0x7f5a180c1d80 2021-02-10T11:30:46.940 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.948 7f5a1dbd2700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5a180b8200 0x7f5a180b8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:46.940 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.948 7f5a1dbd2700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5a180b8200 0x7f5a180b8600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:47424/0 (socket says 172.21.2.112:47424) 2021-02-10T11:30:46.941 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.948 7f5a1dbd2700 1 -- 172.21.2.112:0/1799753518 learned_addr learned my addr 172.21.2.112:0/1799753518 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:30:46.941 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.948 7f5a1e3d3700 1 --2- 172.21.2.112:0/1799753518 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5a18190920 0x7f5a180b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:46.941 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.948 7f5a1ebd4700 1 --2- 172.21.2.112:0/1799753518 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5a180c1d80 0x7f5a180c2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:46.941 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.948 7f5a1dbd2700 1 -- 172.21.2.112:0/1799753518 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5a180c1d80 msgr2=0x7f5a180c2180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:46.941 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.948 7f5a1dbd2700 1 --2- 172.21.2.112:0/1799753518 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5a180c1d80 0x7f5a180c2180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:46.941 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.948 7f5a1dbd2700 1 -- 172.21.2.112:0/1799753518 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5a18190920 msgr2=0x7f5a180b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:46.941 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.948 7f5a1dbd2700 1 --2- 172.21.2.112:0/1799753518 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5a18190920 0x7f5a180b7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:46.941 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.949 7f5a1dbd2700 1 -- 172.21.2.112:0/1799753518 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a180c29c0 con 0x7f5a180b8200 2021-02-10T11:30:46.941 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.949 7f5a1dbd2700 1 --2- 172.21.2.112:0/1799753518 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5a180b8200 0x7f5a180b8600 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 rx=0x7f5a08006f00 tx=0x7f5a0800b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:46.941 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.949 7f5a0effd700 1 -- 172.21.2.112:0/1799753518 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5a0800bd30 con 0x7f5a180b8200 2021-02-10T11:30:46.941 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.949 7f5a0effd700 1 -- 172.21.2.112:0/1799753518 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a08002030 con 0x7f5a180b8200 2021-02-10T11:30:46.941 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.949 7f5a205b7700 1 -- 172.21.2.112:0/1799753518 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a180c29c0 con 0x7f5a180b8200 2021-02-10T11:30:46.941 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.949 7f5a205b7700 1 -- 172.21.2.112:0/1799753518 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a180c2f50 con 0x7f5a180b8200 2021-02-10T11:30:46.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.949 7f5a0effd700 1 -- 172.21.2.112:0/1799753518 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5a08002550 con 0x7f5a180b8200 2021-02-10T11:30:46.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.950 7f5a0effd700 1 -- 172.21.2.112:0/1799753518 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5a08000c90 con 0x7f5a180b8200 2021-02-10T11:30:46.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.950 7f5a0effd700 1 --2- 172.21.2.112:0/1799753518 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5a00018730 0x7f5a0001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:46.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.950 7f5a0effd700 1 -- 172.21.2.112:0/1799753518 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f5a080327d0 con 0x7f5a180b8200 2021-02-10T11:30:46.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.950 7f5a1e3d3700 1 --2- 172.21.2.112:0/1799753518 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5a00018730 0x7f5a0001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:46.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.950 7f5a1e3d3700 1 --2- 172.21.2.112:0/1799753518 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5a00018730 0x7f5a0001ac10 crc :-1 s=READY pgs=451 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:46.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.981 7f5a205b7700 1 -- 172.21.2.112:0/1799753518 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a18105ae0 con 0x7f5a180b8200 2021-02-10T11:30:46.972 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:46.983 7f5a0effd700 1 -- 172.21.2.112:0/1799753518 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5a080009b0 con 0x7f5a180b8200 2021-02-10T11:30:47.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:47.117 7f5a205b7700 1 -- 172.21.2.112:0/1799753518 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348630", "target": ["mgr", ""]}) v1 -- 0x7f5a180c2f50 con 0x7f5a00018730 2021-02-10T11:30:47.115 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:47.127 7f5a0effd700 1 -- 172.21.2.112:0/1799753518 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f5a14000ce0 con 0x7f5a00018730 2021-02-10T11:30:47.123 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:47.134 7f5a205b7700 1 -- 172.21.2.112:0/1799753518 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5a00018730 msgr2=0x7f5a0001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:47.123 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:47.134 7f5a205b7700 1 --2- 172.21.2.112:0/1799753518 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5a00018730 0x7f5a0001ac10 crc :-1 s=READY pgs=451 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:47.123 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:47.134 7f5a205b7700 1 -- 172.21.2.112:0/1799753518 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5a180b8200 msgr2=0x7f5a180b8600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:47.123 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:47.134 7f5a205b7700 1 --2- 172.21.2.112:0/1799753518 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5a180b8200 0x7f5a180b8600 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 rx=0x7f5a08006f00 tx=0x7f5a0800b940).stop 2021-02-10T11:30:47.123 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:47.135 7f5a205b7700 1 -- 172.21.2.112:0/1799753518 shutdown_connections 2021-02-10T11:30:47.124 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:47.135 7f5a205b7700 1 --2- 172.21.2.112:0/1799753518 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5a00018730 0x7f5a0001ac10 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:47.124 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:47.135 7f5a205b7700 1 --2- 172.21.2.112:0/1799753518 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5a180c1d80 0x7f5a180c2180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:47.124 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:47.135 7f5a205b7700 1 --2- 172.21.2.112:0/1799753518 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5a180b8200 0x7f5a180b8600 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:47.124 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:47.135 7f5a205b7700 1 --2- 172.21.2.112:0/1799753518 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5a18190920 0x7f5a180b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:47.124 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:47.135 7f5a205b7700 1 -- 172.21.2.112:0/1799753518 >> 172.21.2.112:0/1799753518 conn(0x7f5a18094e10 msgr2=0x7f5a180a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:47.124 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:47.135 7f5a205b7700 1 -- 172.21.2.112:0/1799753518 shutdown_connections 2021-02-10T11:30:47.124 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:47.135 7f5a205b7700 1 -- 172.21.2.112:0/1799753518 wait complete. 2021-02-10T11:30:47.131 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000348631 2021-02-10T11:30:47.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:47.556 7f1a21e36700 1 Processor -- start 2021-02-10T11:30:47.879 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:47.556 7f1a21e36700 1 -- start start 2021-02-10T11:30:47.879 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:47.557 7f1a21e36700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f1a1c1b0470 0x7f1a1c1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:47.879 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:47.557 7f1a21e36700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f1a1c0ef610 0x7f1a1c0efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:47.879 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:47.557 7f1a21e36700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1a1c0f0030 0x7f1a1c0f0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:47.880 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:47.557 7f1a21e36700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a1c0e5ea0 con 0x7f1a1c1b0470 2021-02-10T11:30:47.880 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:47.557 7f1a21e36700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a1c0f0b50 con 0x7f1a1c0ef610 2021-02-10T11:30:47.880 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:47.557 7f1a21e36700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a1c0e52e0 con 0x7f1a1c0f0030 2021-02-10T11:30:47.880 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:47.557 7f1a1b7fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f1a1c1b0470 0x7f1a1c1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:47.880 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:47.557 7f1a1bfff700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1a1c0f0030 0x7f1a1c0f0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:47.880 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:47.557 7f1a1affd700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f1a1c0ef610 0x7f1a1c0efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:47.880 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:47.557 7f1a1b7fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f1a1c1b0470 0x7f1a1c1b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:47430/0 (socket says 172.21.2.112:47430) 2021-02-10T11:30:47.880 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:47.557 7f1a1b7fe700 1 -- 172.21.2.112:0/1861747713 learned_addr learned my addr 172.21.2.112:0/1861747713 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:30:47.880 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:47.558 7f1a1affd700 1 -- 172.21.2.112:0/1861747713 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1a1c0f0030 msgr2=0x7f1a1c0f0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:47.880 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:47.558 7f1a1affd700 1 --2- 172.21.2.112:0/1861747713 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1a1c0f0030 0x7f1a1c0f0430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:47.880 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:47.558 7f1a1affd700 1 -- 172.21.2.112:0/1861747713 >> v2:172.21.2.112:3300/0 conn(0x7f1a1c1b0470 msgr2=0x7f1a1c1b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:47.880 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:47.558 7f1a1affd700 1 --2- 172.21.2.112:0/1861747713 >> v2:172.21.2.112:3300/0 conn(0x7f1a1c1b0470 0x7f1a1c1b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:47.880 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:47.558 7f1a1affd700 1 -- 172.21.2.112:0/1861747713 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a1c0f10d0 con 0x7f1a1c0ef610 2021-02-10T11:30:47.881 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:47.558 7f1a1bfff700 1 --2- 172.21.2.112:0/1861747713 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1a1c0f0030 0x7f1a1c0f0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:30:47.881 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:47.558 7f1a1affd700 1 --2- 172.21.2.112:0/1861747713 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1a1c0ef610 0x7f1a1c0efa10 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7f1a10006650 tx=0x7f1a10009a80).ready entity=mon.1 client_cookie=ed1ad15e42e76484 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:47.881 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:47.558 7f1a1a7fc700 1 -- 172.21.2.112:0/1861747713 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1a100009b0 con 0x7f1a1c0ef610 2021-02-10T11:30:47.881 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:47.558 7f1a1a7fc700 1 -- 172.21.2.112:0/1861747713 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a1000c0c0 con 0x7f1a1c0ef610 2021-02-10T11:30:47.881 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:47.558 7f1a1a7fc700 1 -- 172.21.2.112:0/1861747713 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1a1000c510 con 0x7f1a1c0ef610 2021-02-10T11:30:47.881 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:47.558 7f1a21e36700 1 -- 172.21.2.112:0/1861747713 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1a1c0ef610 msgr2=0x7f1a1c0efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:47.881 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:47.558 7f1a21e36700 1 --2- 172.21.2.112:0/1861747713 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1a1c0ef610 0x7f1a1c0efa10 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7f1a10006650 tx=0x7f1a10009a80).stop 2021-02-10T11:30:47.881 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:47.559 7f1a21e36700 1 -- 172.21.2.112:0/1861747713 shutdown_connections 2021-02-10T11:30:47.881 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:47.559 7f1a21e36700 1 --2- 172.21.2.112:0/1861747713 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1a1c0f0030 0x7f1a1c0f0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:47.881 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:47.559 7f1a21e36700 1 --2- 172.21.2.112:0/1861747713 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1a1c0ef610 0x7f1a1c0efa10 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:47.881 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:47.559 7f1a21e36700 1 --2- 172.21.2.112:0/1861747713 >> v2:172.21.2.112:3300/0 conn(0x7f1a1c1b0470 0x7f1a1c1b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:47.881 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:47.559 7f1a21e36700 1 -- 172.21.2.112:0/1861747713 >> 172.21.2.112:0/1861747713 conn(0x7f1a1c0ce3e0 msgr2=0x7f1a1c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:47.881 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:47.559 7f1a21e36700 1 -- 172.21.2.112:0/1861747713 shutdown_connections 2021-02-10T11:30:47.881 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:47.559 7f1a21e36700 1 -- 172.21.2.112:0/1861747713 wait complete. 2021-02-10T11:30:47.884 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:47.574 7f1a21e36700 1 Processor -- start 2021-02-10T11:30:47.884 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:47.574 7f1a21e36700 1 -- start start 2021-02-10T11:30:47.884 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:47.574 7f1a21e36700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1a1c0a2a90 0x7f1a1c0c0810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:47.884 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:47.574 7f1a21e36700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1a1c0c0e30 0x7f1a1c0c1230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:47.885 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:47.574 7f1a21e36700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1a1c0a9b90 0x7f1a1c0a9f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:47.885 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:47.574 7f1a21e36700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a1c0a2e90 con 0x7f1a1c0a9b90 2021-02-10T11:30:47.885 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:47.574 7f1a21e36700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a1c0c1850 con 0x7f1a1c0c0e30 2021-02-10T11:30:47.885 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:47.574 7f1a21e36700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a1c0c1a70 con 0x7f1a1c0a2a90 2021-02-10T11:30:47.885 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:47.574 7f1a1bfff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1a1c0a9b90 0x7f1a1c0a9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:47.885 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:47.574 7f1a1bfff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1a1c0a9b90 0x7f1a1c0a9f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:47440/0 (socket says 172.21.2.112:47440) 2021-02-10T11:30:47.885 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:47.574 7f1a1bfff700 1 -- 172.21.2.112:0/4281221944 learned_addr learned my addr 172.21.2.112:0/4281221944 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:30:47.885 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:47.575 7f1a1affd700 1 --2- 172.21.2.112:0/4281221944 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1a1c0c0e30 0x7f1a1c0c1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:47.885 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:47.575 7f1a1b7fe700 1 --2- 172.21.2.112:0/4281221944 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1a1c0a2a90 0x7f1a1c0c0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:47.885 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:47.575 7f1a1bfff700 1 -- 172.21.2.112:0/4281221944 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1a1c0a2a90 msgr2=0x7f1a1c0c0810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:47.885 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:47.575 7f1a1bfff700 1 --2- 172.21.2.112:0/4281221944 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1a1c0a2a90 0x7f1a1c0c0810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:47.885 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:47.575 7f1a1bfff700 1 -- 172.21.2.112:0/4281221944 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1a1c0c0e30 msgr2=0x7f1a1c0c1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:47.885 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:47.575 7f1a1bfff700 1 --2- 172.21.2.112:0/4281221944 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1a1c0c0e30 0x7f1a1c0c1230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:47.886 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:47.575 7f1a1bfff700 1 -- 172.21.2.112:0/4281221944 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a1c0aa5b0 con 0x7f1a1c0a9b90 2021-02-10T11:30:47.886 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:47.575 7f1a1b7fe700 1 --2- 172.21.2.112:0/4281221944 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1a1c0a2a90 0x7f1a1c0c0810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:30:47.886 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:47.575 7f1a1bfff700 1 --2- 172.21.2.112:0/4281221944 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1a1c0a9b90 0x7f1a1c0a9f90 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 rx=0x7f1a0c00b5e0 tx=0x7f1a0c00b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:47.886 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:47.575 7f1a03fff700 1 -- 172.21.2.112:0/4281221944 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1a0c006370 con 0x7f1a1c0a9b90 2021-02-10T11:30:47.886 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:47.575 7f1a03fff700 1 -- 172.21.2.112:0/4281221944 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a0c00bd70 con 0x7f1a1c0a9b90 2021-02-10T11:30:47.886 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:47.575 7f1a03fff700 1 -- 172.21.2.112:0/4281221944 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1a0c002560 con 0x7f1a1c0a9b90 2021-02-10T11:30:47.886 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:47.575 7f1a21e36700 1 -- 172.21.2.112:0/4281221944 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a1c0aa5b0 con 0x7f1a1c0a9b90 2021-02-10T11:30:47.886 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:47.575 7f1a21e36700 1 -- 172.21.2.112:0/4281221944 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a1c0aab40 con 0x7f1a1c0a9b90 2021-02-10T11:30:47.886 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:47.576 7f1a03fff700 1 -- 172.21.2.112:0/4281221944 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1a0c005330 con 0x7f1a1c0a9b90 2021-02-10T11:30:47.886 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:47.576 7f1a03fff700 1 --2- 172.21.2.112:0/4281221944 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f19fc0186c0 0x7f19fc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:47.886 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:47.576 7f1a03fff700 1 -- 172.21.2.112:0/4281221944 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f1a0c0325a0 con 0x7f1a1c0a9b90 2021-02-10T11:30:47.886 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:47.576 7f1a1b7fe700 1 --2- 172.21.2.112:0/4281221944 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f19fc0186c0 0x7f19fc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:47.887 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:47.577 7f1a1b7fe700 1 --2- 172.21.2.112:0/4281221944 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f19fc0186c0 0x7f19fc01abc0 crc :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:47.887 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:47.607 7f1a21e36700 1 -- 172.21.2.112:0/4281221944 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a1c0ab0a0 con 0x7f1a1c0a9b90 2021-02-10T11:30:47.887 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:47.609 7f1a03fff700 1 -- 172.21.2.112:0/4281221944 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1a0c000c10 con 0x7f1a1c0a9b90 2021-02-10T11:30:47.887 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:47.735 7f1a21e36700 1 -- 172.21.2.112:0/4281221944 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348631", "target": ["mgr", ""]}) v1 -- 0x7f1a1c0aab40 con 0x7f19fc0186c0 2021-02-10T11:30:47.887 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:47.748 7f1a03fff700 1 -- 172.21.2.112:0/4281221944 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f1a14000d30 con 0x7f19fc0186c0 2021-02-10T11:30:47.887 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:47.753 7f1a21e36700 1 -- 172.21.2.112:0/4281221944 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f19fc0186c0 msgr2=0x7f19fc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:47.887 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:47.753 7f1a21e36700 1 --2- 172.21.2.112:0/4281221944 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f19fc0186c0 0x7f19fc01abc0 crc :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:47.887 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:47.753 7f1a21e36700 1 -- 172.21.2.112:0/4281221944 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1a1c0a9b90 msgr2=0x7f1a1c0a9f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:47.887 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:47.753 7f1a21e36700 1 --2- 172.21.2.112:0/4281221944 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1a1c0a9b90 0x7f1a1c0a9f90 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 rx=0x7f1a0c00b5e0 tx=0x7f1a0c00b980).stop 2021-02-10T11:30:47.887 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:47.753 7f1a21e36700 1 -- 172.21.2.112:0/4281221944 shutdown_connections 2021-02-10T11:30:47.887 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:47.753 7f1a21e36700 1 --2- 172.21.2.112:0/4281221944 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f19fc0186c0 0x7f19fc01abc0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:47.887 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:47.753 7f1a21e36700 1 --2- 172.21.2.112:0/4281221944 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1a1c0c0e30 0x7f1a1c0c1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:47.887 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:47.753 7f1a21e36700 1 --2- 172.21.2.112:0/4281221944 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1a1c0a9b90 0x7f1a1c0a9f90 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:47.887 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:47.753 7f1a21e36700 1 --2- 172.21.2.112:0/4281221944 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1a1c0a2a90 0x7f1a1c0c0810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:47.888 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:47.753 7f1a21e36700 1 -- 172.21.2.112:0/4281221944 >> 172.21.2.112:0/4281221944 conn(0x7f1a1c0c40d0 msgr2=0x7f1a1c0a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:47.888 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:47.753 7f1a21e36700 1 -- 172.21.2.112:0/4281221944 shutdown_connections 2021-02-10T11:30:47.888 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:47.754 7f1a21e36700 1 -- 172.21.2.112:0/4281221944 wait complete. 2021-02-10T11:30:47.888 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000348632 2021-02-10T11:30:48.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.312 7f82231b6700 1 Processor -- start 2021-02-10T11:30:48.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.312 7f82231b6700 1 -- start start 2021-02-10T11:30:48.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.312 7f82231b6700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f821c1b0470 0x7f821c1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:48.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.312 7f82231b6700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f821c0f0b50 con 0x7f821c0ef610 2021-02-10T11:30:48.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.312 7f82231b6700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f821c0f24e0 con 0x7f821c0f20e0 2021-02-10T11:30:48.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.312 7f82231b6700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f821c0e5ea0 con 0x7f821c1b0470 2021-02-10T11:30:48.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.312 7f821bfff700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f821c0ef610 0x7f821c0efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:46908/0 (socket says 172.21.2.112:46908) 2021-02-10T11:30:48.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.312 7f821bfff700 1 -- 172.21.2.112:0/2660689351 learned_addr learned my addr 172.21.2.112:0/2660689351 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:30:48.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.313 7f8220fd2700 1 --2- 172.21.2.112:0/2660689351 >> v2:172.21.2.120:3300/0 conn(0x7f821c1b0470 0x7f821c1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:48.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.313 7f821b7fe700 1 -- 172.21.2.112:0/2660689351 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2657892001 0 0) 0x7f82140010b0 con 0x7f821c0ef610 2021-02-10T11:30:48.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.313 7f821b7fe700 1 -- 172.21.2.112:0/2660689351 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8208002fe0 con 0x7f821c0ef610 2021-02-10T11:30:48.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.313 7f8220fd2700 1 -- 172.21.2.112:0/2660689351 >> v1:172.21.2.120:6789/0 conn(0x7f821c0f20e0 legacy=0x7f821c0f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:48.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.313 7f8220fd2700 1 -- 172.21.2.112:0/2660689351 >> v1:172.21.2.112:6789/0 conn(0x7f821c0ef610 legacy=0x7f821c0efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:48.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.313 7f8220fd2700 1 -- 172.21.2.112:0/2660689351 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f821c0e52e0 con 0x7f821c1b0470 2021-02-10T11:30:48.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.313 7f8220fd2700 1 --2- 172.21.2.112:0/2660689351 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f821c1b0470 0x7f821c1b0870 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7f8204006650 tx=0x7f8204009a80).ready entity=mon.1 client_cookie=dbcb0c8bf020b669 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:48.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.314 7f821b7fe700 1 -- 172.21.2.112:0/2660689351 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f82040009b0 con 0x7f821c1b0470 2021-02-10T11:30:48.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.314 7f821b7fe700 1 -- 172.21.2.112:0/2660689351 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f820400c0c0 con 0x7f821c1b0470 2021-02-10T11:30:48.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.314 7f821b7fe700 1 -- 172.21.2.112:0/2660689351 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f820400c5e0 con 0x7f821c1b0470 2021-02-10T11:30:48.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.314 7f82231b6700 1 -- 172.21.2.112:0/2660689351 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f821c1b0470 msgr2=0x7f821c1b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:48.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.314 7f82231b6700 1 --2- 172.21.2.112:0/2660689351 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f821c1b0470 0x7f821c1b0870 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7f8204006650 tx=0x7f8204009a80).stop 2021-02-10T11:30:48.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.314 7f82231b6700 1 -- 172.21.2.112:0/2660689351 shutdown_connections 2021-02-10T11:30:48.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.314 7f82231b6700 1 --2- 172.21.2.112:0/2660689351 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f821c1b0470 0x7f821c1b0870 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:48.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.314 7f82231b6700 1 -- 172.21.2.112:0/2660689351 >> 172.21.2.112:0/2660689351 conn(0x7f821c0ce3e0 msgr2=0x7f821c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:48.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.314 7f82231b6700 1 -- 172.21.2.112:0/2660689351 shutdown_connections 2021-02-10T11:30:48.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.314 7f82231b6700 1 -- 172.21.2.112:0/2660689351 wait complete. 2021-02-10T11:30:48.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.329 7f82231b6700 1 Processor -- start 2021-02-10T11:30:48.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.329 7f82231b6700 1 -- start start 2021-02-10T11:30:48.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.329 7f82231b6700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f821c0c36a0 0x7f821c0c2b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:48.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.329 7f82231b6700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f821c0c3190 0x7f821c0b1d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:48.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.329 7f82231b6700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f821c0b2370 0x7f821c0b2d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:48.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.329 7f82231b6700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f821c0b27c0 con 0x7f821c0c36a0 2021-02-10T11:30:48.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.329 7f82231b6700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f821c0b3360 con 0x7f821c0b2370 2021-02-10T11:30:48.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.329 7f82231b6700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f821c0b3540 con 0x7f821c0c3190 2021-02-10T11:30:48.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.330 7f8220fd2700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f821c0c36a0 0x7f821c0c2b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:48.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.330 7f8220fd2700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f821c0c36a0 0x7f821c0c2b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:47454/0 (socket says 172.21.2.112:47454) 2021-02-10T11:30:48.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.330 7f8220fd2700 1 -- 172.21.2.112:0/24890000 learned_addr learned my addr 172.21.2.112:0/24890000 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:30:48.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.330 7f821bfff700 1 --2- 172.21.2.112:0/24890000 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f821c0c3190 0x7f821c0b1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:48.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.330 7f82217d3700 1 --2- 172.21.2.112:0/24890000 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f821c0b2370 0x7f821c0b2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:48.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.330 7f8220fd2700 1 -- 172.21.2.112:0/24890000 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f821c0c3190 msgr2=0x7f821c0b1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:48.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.330 7f8220fd2700 1 --2- 172.21.2.112:0/24890000 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f821c0c3190 0x7f821c0b1d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:48.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.330 7f8220fd2700 1 -- 172.21.2.112:0/24890000 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f821c0b2370 msgr2=0x7f821c0b2d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:48.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.330 7f8220fd2700 1 --2- 172.21.2.112:0/24890000 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f821c0b2370 0x7f821c0b2d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:48.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.330 7f8220fd2700 1 -- 172.21.2.112:0/24890000 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f821c0b3760 con 0x7f821c0c36a0 2021-02-10T11:30:48.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.330 7f8220fd2700 1 --2- 172.21.2.112:0/24890000 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f821c0c36a0 0x7f821c0c2b70 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 rx=0x7f82040096a0 tx=0x7f820400b9b0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:48.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.330 7f82197fa700 1 -- 172.21.2.112:0/24890000 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f82040093a0 con 0x7f821c0c36a0 2021-02-10T11:30:48.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.330 7f82197fa700 1 -- 172.21.2.112:0/24890000 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82040067f0 con 0x7f821c0c36a0 2021-02-10T11:30:48.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.330 7f82197fa700 1 -- 172.21.2.112:0/24890000 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f82040022b0 con 0x7f821c0c36a0 2021-02-10T11:30:48.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.330 7f82231b6700 1 -- 172.21.2.112:0/24890000 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f821c0b3760 con 0x7f821c0c36a0 2021-02-10T11:30:48.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.330 7f82231b6700 1 -- 172.21.2.112:0/24890000 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f821c0b3cf0 con 0x7f821c0c36a0 2021-02-10T11:30:48.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.331 7f82197fa700 1 -- 172.21.2.112:0/24890000 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8204000c90 con 0x7f821c0c36a0 2021-02-10T11:30:48.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.331 7f82197fa700 1 --2- 172.21.2.112:0/24890000 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f82000186c0 0x7f820001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:48.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.331 7f82197fa700 1 -- 172.21.2.112:0/24890000 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f82040323e0 con 0x7f821c0c36a0 2021-02-10T11:30:48.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.331 7f821bfff700 1 --2- 172.21.2.112:0/24890000 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f82000186c0 0x7f820001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:48.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.332 7f821bfff700 1 --2- 172.21.2.112:0/24890000 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f82000186c0 0x7f820001abc0 crc :-1 s=READY pgs=453 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:48.350 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.362 7f82231b6700 1 -- 172.21.2.112:0/24890000 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f821c0edd10 con 0x7f821c0c36a0 2021-02-10T11:30:48.353 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.364 7f82197fa700 1 -- 172.21.2.112:0/24890000 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f82040093a0 con 0x7f821c0c36a0 2021-02-10T11:30:48.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.492 7f82231b6700 1 -- 172.21.2.112:0/24890000 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348632", "target": ["mgr", ""]}) v1 -- 0x7f821c0b3cf0 con 0x7f82000186c0 2021-02-10T11:30:48.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.504 7f82197fa700 1 -- 172.21.2.112:0/24890000 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f82140009b0 con 0x7f82000186c0 2021-02-10T11:30:48.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.509 7f82231b6700 1 -- 172.21.2.112:0/24890000 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f82000186c0 msgr2=0x7f820001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:48.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.509 7f82231b6700 1 --2- 172.21.2.112:0/24890000 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f82000186c0 0x7f820001abc0 crc :-1 s=READY pgs=453 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:48.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.509 7f82231b6700 1 -- 172.21.2.112:0/24890000 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f821c0c36a0 msgr2=0x7f821c0c2b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:48.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.509 7f82231b6700 1 --2- 172.21.2.112:0/24890000 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f821c0c36a0 0x7f821c0c2b70 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 rx=0x7f82040096a0 tx=0x7f820400b9b0).stop 2021-02-10T11:30:48.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.510 7f82231b6700 1 -- 172.21.2.112:0/24890000 shutdown_connections 2021-02-10T11:30:48.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.510 7f82231b6700 1 --2- 172.21.2.112:0/24890000 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f82000186c0 0x7f820001abc0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:48.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.510 7f82231b6700 1 --2- 172.21.2.112:0/24890000 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f821c0b2370 0x7f821c0b2d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:48.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.510 7f82231b6700 1 --2- 172.21.2.112:0/24890000 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f821c0c3190 0x7f821c0b1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:48.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.510 7f82231b6700 1 --2- 172.21.2.112:0/24890000 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f821c0c36a0 0x7f821c0c2b70 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:48.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.510 7f82231b6700 1 -- 172.21.2.112:0/24890000 >> 172.21.2.112:0/24890000 conn(0x7f821c09f630 msgr2=0x7f821c0a0130 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:48.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.510 7f82231b6700 1 -- 172.21.2.112:0/24890000 shutdown_connections 2021-02-10T11:30:48.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.510 7f82231b6700 1 -- 172.21.2.112:0/24890000 wait complete. 2021-02-10T11:30:48.507 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000348633 2021-02-10T11:30:48.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.932 7fa2c284e700 1 Processor -- start 2021-02-10T11:30:48.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.932 7fa2c284e700 1 -- start start 2021-02-10T11:30:48.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.933 7fa2c284e700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa2bc0f05f0 0x7fa2bc0f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:48.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.933 7fa2c284e700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2bc1b0c70 con 0x7fa2bc1b0870 2021-02-10T11:30:48.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.933 7fa2c284e700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2bc0f0b50 con 0x7fa2bc1b0470 2021-02-10T11:30:48.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.933 7fa2c284e700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2bc0e5ea0 con 0x7fa2bc0f05f0 2021-02-10T11:30:48.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.933 7fa2bb7fe700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fa2bc1b0870 0x7fa2bc0efe00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:46920/0 (socket says 172.21.2.112:46920) 2021-02-10T11:30:48.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.933 7fa2bb7fe700 1 -- 172.21.2.112:0/2927601395 learned_addr learned my addr 172.21.2.112:0/2927601395 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:30:48.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.933 7fa2c0e6b700 1 --2- 172.21.2.112:0/2927601395 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa2bc0f05f0 0x7fa2bc0f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:48.938 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.933 7fa2baffd700 1 -- 172.21.2.112:0/2927601395 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 971932523 0 0) 0x7fa2b00010b0 con 0x7fa2bc1b0870 2021-02-10T11:30:48.938 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.933 7fa2baffd700 1 -- 172.21.2.112:0/2927601395 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2a4002fe0 con 0x7fa2bc1b0870 2021-02-10T11:30:48.938 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.933 7fa2baffd700 1 -- 172.21.2.112:0/2927601395 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2482845081 0 0) 0x7fa2ac000ff0 con 0x7fa2bc1b0470 2021-02-10T11:30:48.938 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.933 7fa2baffd700 1 -- 172.21.2.112:0/2927601395 --> v1:172.21.2.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2a4003210 con 0x7fa2bc1b0470 2021-02-10T11:30:48.938 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.933 7fa2c0e6b700 1 -- 172.21.2.112:0/2927601395 >> v1:172.21.2.120:6789/0 conn(0x7fa2bc1b0470 legacy=0x7fa2bc0ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:48.938 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.933 7fa2c0e6b700 1 -- 172.21.2.112:0/2927601395 >> v1:172.21.2.112:6789/0 conn(0x7fa2bc1b0870 legacy=0x7fa2bc0efe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:48.938 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.933 7fa2c0e6b700 1 -- 172.21.2.112:0/2927601395 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2bc0e52e0 con 0x7fa2bc0f05f0 2021-02-10T11:30:48.938 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.934 7fa2c0e6b700 1 --2- 172.21.2.112:0/2927601395 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa2bc0f05f0 0x7fa2bc0f20e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7fa2b4006e50 tx=0x7fa2b4009e70).ready entity=mon.2 client_cookie=f92f085018dd28dc server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:48.938 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.934 7fa2baffd700 1 -- 172.21.2.112:0/2927601395 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa2b4006a20 con 0x7fa2bc0f05f0 2021-02-10T11:30:48.938 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.934 7fa2baffd700 1 -- 172.21.2.112:0/2927601395 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2b4006790 con 0x7fa2bc0f05f0 2021-02-10T11:30:48.938 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.934 7fa2baffd700 1 -- 172.21.2.112:0/2927601395 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa2b400b700 con 0x7fa2bc0f05f0 2021-02-10T11:30:48.938 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.934 7fa2c284e700 1 -- 172.21.2.112:0/2927601395 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa2bc0f05f0 msgr2=0x7fa2bc0f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:48.938 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.934 7fa2c284e700 1 --2- 172.21.2.112:0/2927601395 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa2bc0f05f0 0x7fa2bc0f20e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7fa2b4006e50 tx=0x7fa2b4009e70).stop 2021-02-10T11:30:48.939 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.934 7fa2c284e700 1 -- 172.21.2.112:0/2927601395 shutdown_connections 2021-02-10T11:30:48.939 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.935 7fa2c284e700 1 --2- 172.21.2.112:0/2927601395 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa2bc0f05f0 0x7fa2bc0f20e0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:48.939 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.935 7fa2c284e700 1 -- 172.21.2.112:0/2927601395 >> 172.21.2.112:0/2927601395 conn(0x7fa2bc0ce3e0 msgr2=0x7fa2bc0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:48.939 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.935 7fa2c284e700 1 -- 172.21.2.112:0/2927601395 shutdown_connections 2021-02-10T11:30:48.939 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.935 7fa2c284e700 1 -- 172.21.2.112:0/2927601395 wait complete. 2021-02-10T11:30:48.940 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.949 7fa2c284e700 1 Processor -- start 2021-02-10T11:30:48.941 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.949 7fa2c284e700 1 -- start start 2021-02-10T11:30:48.941 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.949 7fa2c284e700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa2bc0b20b0 0x7fa2bc0ac970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:48.941 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.949 7fa2c284e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa2bc0b24b0 0x7fa2bc191dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:48.941 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.949 7fa2c284e700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa2bc0acf90 0x7fa2bc1923e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:48.941 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.949 7fa2c284e700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2bc0b1b30 con 0x7fa2bc0b24b0 2021-02-10T11:30:48.941 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.949 7fa2c284e700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2bc1a04e0 con 0x7fa2bc0acf90 2021-02-10T11:30:48.941 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.949 7fa2c284e700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2bc1a06c0 con 0x7fa2bc0b20b0 2021-02-10T11:30:48.941 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.950 7fa2bbfff700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa2bc0b20b0 0x7fa2bc0ac970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:48.941 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.950 7fa2c0e6b700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa2bc0acf90 0x7fa2bc1923e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:48.941 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.950 7fa2bb7fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa2bc0b24b0 0x7fa2bc191dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:48.941 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.950 7fa2bbfff700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa2bc0b20b0 0x7fa2bc0ac970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3301/0 says I am v2:172.21.2.112:47148/0 (socket says 172.21.2.112:47148) 2021-02-10T11:30:48.941 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.950 7fa2bbfff700 1 -- 172.21.2.112:0/2278024398 learned_addr learned my addr 172.21.2.112:0/2278024398 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:30:48.941 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.950 7fa2bbfff700 1 -- 172.21.2.112:0/2278024398 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa2bc0acf90 msgr2=0x7fa2bc1923e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:48.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.950 7fa2bbfff700 1 --2- 172.21.2.112:0/2278024398 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa2bc0acf90 0x7fa2bc1923e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:48.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.950 7fa2bbfff700 1 -- 172.21.2.112:0/2278024398 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa2bc0b24b0 msgr2=0x7fa2bc191dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:48.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.950 7fa2bbfff700 1 --2- 172.21.2.112:0/2278024398 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa2bc0b24b0 0x7fa2bc191dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:48.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.950 7fa2bbfff700 1 -- 172.21.2.112:0/2278024398 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2bc192be0 con 0x7fa2bc0b20b0 2021-02-10T11:30:48.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.950 7fa2bb7fe700 1 --2- 172.21.2.112:0/2278024398 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa2bc0b24b0 0x7fa2bc191dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:30:48.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.950 7fa2bbfff700 1 --2- 172.21.2.112:0/2278024398 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa2bc0b20b0 0x7fa2bc0ac970 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7fa2ac006f00 tx=0x7fa2ac00b940).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:48.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.950 7fa2b8ff9700 1 -- 172.21.2.112:0/2278024398 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa2ac00bd30 con 0x7fa2bc0b20b0 2021-02-10T11:30:48.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.950 7fa2b8ff9700 1 -- 172.21.2.112:0/2278024398 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2ac000ce0 con 0x7fa2bc0b20b0 2021-02-10T11:30:48.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.950 7fa2c284e700 1 -- 172.21.2.112:0/2278024398 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2bc192be0 con 0x7fa2bc0b20b0 2021-02-10T11:30:48.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.951 7fa2b8ff9700 1 -- 172.21.2.112:0/2278024398 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa2ac0022b0 con 0x7fa2bc0b20b0 2021-02-10T11:30:48.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.951 7fa2c284e700 1 -- 172.21.2.112:0/2278024398 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2bc192be0 con 0x7fa2bc0b20b0 2021-02-10T11:30:48.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.951 7fa2b8ff9700 1 -- 172.21.2.112:0/2278024398 <== mon.2 v2:172.21.2.120:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa2ac002020 con 0x7fa2bc0b20b0 2021-02-10T11:30:48.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.951 7fa2b8ff9700 1 --2- 172.21.2.112:0/2278024398 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa29c0186c0 0x7fa29c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:48.943 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.951 7fa2b8ff9700 1 -- 172.21.2.112:0/2278024398 <== mon.2 v2:172.21.2.120:3301/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fa2ac032650 con 0x7fa2bc0b20b0 2021-02-10T11:30:48.943 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.951 7fa2bb7fe700 1 --2- 172.21.2.112:0/2278024398 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa29c0186c0 0x7fa29c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:48.943 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.952 7fa2bb7fe700 1 --2- 172.21.2.112:0/2278024398 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa29c0186c0 0x7fa29c01abc0 crc :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:48.971 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.982 7fa2c284e700 1 -- 172.21.2.112:0/2278024398 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2bc105af0 con 0x7fa2bc0b20b0 2021-02-10T11:30:48.973 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:48.985 7fa2b8ff9700 1 -- 172.21.2.112:0/2278024398 <== mon.2 v2:172.21.2.120:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa2ac006b90 con 0x7fa2bc0b20b0 2021-02-10T11:30:49.109 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:49.121 7fa2c284e700 1 -- 172.21.2.112:0/2278024398 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348633", "target": ["mgr", ""]}) v1 -- 0x7fa2bc0b1b30 con 0x7fa29c0186c0 2021-02-10T11:30:49.120 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:49.131 7fa2b8ff9700 1 -- 172.21.2.112:0/2278024398 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fa2b00009b0 con 0x7fa29c0186c0 2021-02-10T11:30:49.126 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:49.137 7fa2c284e700 1 -- 172.21.2.112:0/2278024398 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa29c0186c0 msgr2=0x7fa29c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:49.126 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:49.137 7fa2c284e700 1 --2- 172.21.2.112:0/2278024398 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa29c0186c0 0x7fa29c01abc0 crc :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:49.126 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:49.137 7fa2c284e700 1 -- 172.21.2.112:0/2278024398 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa2bc0b20b0 msgr2=0x7fa2bc0ac970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:49.126 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:49.137 7fa2c284e700 1 --2- 172.21.2.112:0/2278024398 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa2bc0b20b0 0x7fa2bc0ac970 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7fa2ac006f00 tx=0x7fa2ac00b940).stop 2021-02-10T11:30:49.126 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:49.138 7fa2c284e700 1 -- 172.21.2.112:0/2278024398 shutdown_connections 2021-02-10T11:30:49.127 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:49.138 7fa2c284e700 1 --2- 172.21.2.112:0/2278024398 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa29c0186c0 0x7fa29c01abc0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:49.127 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:49.138 7fa2c284e700 1 --2- 172.21.2.112:0/2278024398 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa2bc0acf90 0x7fa2bc1923e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:49.127 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:49.138 7fa2c284e700 1 --2- 172.21.2.112:0/2278024398 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa2bc0b24b0 0x7fa2bc191dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:49.127 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:49.138 7fa2c284e700 1 --2- 172.21.2.112:0/2278024398 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa2bc0b20b0 0x7fa2bc0ac970 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:49.127 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:49.138 7fa2c284e700 1 -- 172.21.2.112:0/2278024398 >> 172.21.2.112:0/2278024398 conn(0x7fa2bc0bf4e0 msgr2=0x7fa2bc0be3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:49.127 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:49.138 7fa2c284e700 1 -- 172.21.2.112:0/2278024398 shutdown_connections 2021-02-10T11:30:49.127 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:49.138 7fa2c284e700 1 -- 172.21.2.112:0/2278024398 wait complete. 2021-02-10T11:30:49.135 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000348634 2021-02-10T11:30:49.563 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:49.558 7fad862b0700 1 Processor -- start 2021-02-10T11:30:49.564 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:49.558 7fad862b0700 1 -- start start 2021-02-10T11:30:49.564 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:49.559 7fad862b0700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fad801b0470 0x7fad801b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:49.564 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:49.559 7fad862b0700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fad800ef610 0x7fad800efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:49.564 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:49.559 7fad862b0700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad800f0b50 con 0x7fad800f0030 2021-02-10T11:30:49.564 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:49.559 7fad862b0700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad800e52e0 con 0x7fad800ef610 2021-02-10T11:30:49.564 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:49.559 7fad862b0700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad800e5490 con 0x7fad801b0470 2021-02-10T11:30:49.564 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:49.559 7fad7f7fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fad800ef610 0x7fad800efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:49.564 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:49.559 7fad848cd700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fad800f0030 0x7fad800f10d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:46934/0 (socket says 172.21.2.112:46934) 2021-02-10T11:30:49.564 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:49.559 7fad848cd700 1 -- 172.21.2.112:0/2268176033 learned_addr learned my addr 172.21.2.112:0/2268176033 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:30:49.565 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:49.559 7fad7ffff700 1 --2- 172.21.2.112:0/2268176033 >> v2:172.21.2.120:3300/0 conn(0x7fad801b0470 0x7fad801b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:49.565 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:49.560 7fad7f7fe700 1 -- 172.21.2.112:0/2268176033 >> v2:172.21.2.120:3300/0 conn(0x7fad801b0470 msgr2=0x7fad801b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:49.565 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:49.560 7fad7f7fe700 1 --2- 172.21.2.112:0/2268176033 >> v2:172.21.2.120:3300/0 conn(0x7fad801b0470 0x7fad801b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:49.565 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:49.560 7fad7f7fe700 1 -- 172.21.2.112:0/2268176033 >> v1:172.21.2.112:6789/0 conn(0x7fad800f0030 legacy=0x7fad800f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:49.565 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:49.560 7fad7f7fe700 1 -- 172.21.2.112:0/2268176033 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad800f0430 con 0x7fad800ef610 2021-02-10T11:30:49.565 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:49.560 7fad7f7fe700 1 --2- 172.21.2.112:0/2268176033 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fad800ef610 0x7fad800efa10 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 rx=0x7fad74006650 tx=0x7fad74009a80).ready entity=mon.0 client_cookie=8c127d5e7381f40a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:49.565 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:49.560 7fad7effd700 1 -- 172.21.2.112:0/2268176033 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fad740009b0 con 0x7fad800ef610 2021-02-10T11:30:49.565 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:49.560 7fad7effd700 1 -- 172.21.2.112:0/2268176033 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad7400c0c0 con 0x7fad800ef610 2021-02-10T11:30:49.565 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:49.560 7fad7effd700 1 -- 172.21.2.112:0/2268176033 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fad7400c520 con 0x7fad800ef610 2021-02-10T11:30:49.570 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:49.560 7fad862b0700 1 -- 172.21.2.112:0/2268176033 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fad800ef610 msgr2=0x7fad800efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:49.570 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:49.560 7fad862b0700 1 --2- 172.21.2.112:0/2268176033 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fad800ef610 0x7fad800efa10 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 rx=0x7fad74006650 tx=0x7fad74009a80).stop 2021-02-10T11:30:49.570 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:49.561 7fad862b0700 1 -- 172.21.2.112:0/2268176033 shutdown_connections 2021-02-10T11:30:49.570 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:49.561 7fad862b0700 1 --2- 172.21.2.112:0/2268176033 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fad800ef610 0x7fad800efa10 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:49.570 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:49.561 7fad862b0700 1 --2- 172.21.2.112:0/2268176033 >> v2:172.21.2.120:3300/0 conn(0x7fad801b0470 0x7fad801b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:49.570 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:49.561 7fad862b0700 1 -- 172.21.2.112:0/2268176033 >> 172.21.2.112:0/2268176033 conn(0x7fad800ce3e0 msgr2=0x7fad800e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:49.570 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:49.561 7fad862b0700 1 -- 172.21.2.112:0/2268176033 shutdown_connections 2021-02-10T11:30:49.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:49.561 7fad862b0700 1 -- 172.21.2.112:0/2268176033 wait complete. 2021-02-10T11:30:49.572 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:49.576 7fad862b0700 1 Processor -- start 2021-02-10T11:30:49.573 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:49.576 7fad862b0700 1 -- start start 2021-02-10T11:30:49.573 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:49.576 7fad862b0700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fad800a0420 0x7fad800a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:49.573 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:49.576 7fad862b0700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fad80189d00 0x7fad8018a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:49.573 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:49.576 7fad862b0700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fad8018a720 0x7fad800b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:49.573 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:49.576 7fad862b0700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad80189780 con 0x7fad800a0420 2021-02-10T11:30:49.573 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:49.576 7fad862b0700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad8018ab20 con 0x7fad80189d00 2021-02-10T11:30:49.573 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:49.576 7fad862b0700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad800a1140 con 0x7fad8018a720 2021-02-10T11:30:49.573 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:49.576 7fad7ffff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fad800a0420 0x7fad800a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:49.573 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:49.576 7fad7ffff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fad800a0420 0x7fad800a0820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:47478/0 (socket says 172.21.2.112:47478) 2021-02-10T11:30:49.573 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:49.576 7fad7ffff700 1 -- 172.21.2.112:0/1004166495 learned_addr learned my addr 172.21.2.112:0/1004166495 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:30:49.573 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:49.576 7fad848cd700 1 --2- 172.21.2.112:0/1004166495 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fad8018a720 0x7fad800b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:49.573 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:49.577 7fad7f7fe700 1 --2- 172.21.2.112:0/1004166495 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fad80189d00 0x7fad8018a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:49.574 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:49.577 7fad7ffff700 1 -- 172.21.2.112:0/1004166495 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fad8018a720 msgr2=0x7fad800b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:49.574 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:49.577 7fad7ffff700 1 --2- 172.21.2.112:0/1004166495 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fad8018a720 0x7fad800b7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:49.574 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:49.577 7fad7ffff700 1 -- 172.21.2.112:0/1004166495 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fad80189d00 msgr2=0x7fad8018a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:49.574 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:49.577 7fad7ffff700 1 --2- 172.21.2.112:0/1004166495 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fad80189d00 0x7fad8018a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:49.574 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:49.577 7fad7ffff700 1 -- 172.21.2.112:0/1004166495 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad800b8200 con 0x7fad800a0420 2021-02-10T11:30:49.574 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:49.577 7fad848cd700 1 --2- 172.21.2.112:0/1004166495 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fad8018a720 0x7fad800b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:30:49.574 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:49.577 7fad7f7fe700 1 --2- 172.21.2.112:0/1004166495 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fad80189d00 0x7fad8018a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:30:49.574 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:49.577 7fad7ffff700 1 --2- 172.21.2.112:0/1004166495 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fad800a0420 0x7fad800a0820 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 rx=0x7fad70006f60 tx=0x7fad7000b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:49.574 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:49.577 7fad7cff9700 1 -- 172.21.2.112:0/1004166495 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fad70006bb0 con 0x7fad800a0420 2021-02-10T11:30:49.574 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:49.577 7fad7cff9700 1 -- 172.21.2.112:0/1004166495 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad70006340 con 0x7fad800a0420 2021-02-10T11:30:49.574 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:49.577 7fad7cff9700 1 -- 172.21.2.112:0/1004166495 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fad70002030 con 0x7fad800a0420 2021-02-10T11:30:49.574 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:49.577 7fad862b0700 1 -- 172.21.2.112:0/1004166495 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad800b8200 con 0x7fad800a0420 2021-02-10T11:30:49.574 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:49.577 7fad862b0700 1 -- 172.21.2.112:0/1004166495 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad800b8790 con 0x7fad800a0420 2021-02-10T11:30:49.575 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:49.578 7fad7cff9700 1 -- 172.21.2.112:0/1004166495 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fad70006b50 con 0x7fad800a0420 2021-02-10T11:30:49.575 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:49.578 7fad7cff9700 1 --2- 172.21.2.112:0/1004166495 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fad600186c0 0x7fad6001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:49.575 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:49.578 7fad7cff9700 1 -- 172.21.2.112:0/1004166495 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fad70032580 con 0x7fad800a0420 2021-02-10T11:30:49.575 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:49.578 7fad7f7fe700 1 --2- 172.21.2.112:0/1004166495 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fad600186c0 0x7fad6001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:49.575 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:49.578 7fad7f7fe700 1 --2- 172.21.2.112:0/1004166495 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fad600186c0 0x7fad6001abc0 crc :-1 s=READY pgs=455 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:49.597 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:49.609 7fad862b0700 1 -- 172.21.2.112:0/1004166495 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad800a1360 con 0x7fad800a0420 2021-02-10T11:30:49.600 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:49.611 7fad7cff9700 1 -- 172.21.2.112:0/1004166495 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fad70000f00 con 0x7fad800a0420 2021-02-10T11:30:49.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:49.740 7fad862b0700 1 -- 172.21.2.112:0/1004166495 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348634", "target": ["mgr", ""]}) v1 -- 0x7fad800b9020 con 0x7fad600186c0 2021-02-10T11:30:49.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:49.751 7fad7cff9700 1 -- 172.21.2.112:0/1004166495 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fad74006390 con 0x7fad600186c0 2021-02-10T11:30:49.746 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:49.757 7fad862b0700 1 -- 172.21.2.112:0/1004166495 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fad600186c0 msgr2=0x7fad6001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:49.746 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:49.757 7fad862b0700 1 --2- 172.21.2.112:0/1004166495 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fad600186c0 0x7fad6001abc0 crc :-1 s=READY pgs=455 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:49.746 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:49.757 7fad862b0700 1 -- 172.21.2.112:0/1004166495 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fad800a0420 msgr2=0x7fad800a0820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:49.746 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:49.757 7fad862b0700 1 --2- 172.21.2.112:0/1004166495 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fad800a0420 0x7fad800a0820 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 rx=0x7fad70006f60 tx=0x7fad7000b970).stop 2021-02-10T11:30:49.747 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:49.758 7fad862b0700 1 -- 172.21.2.112:0/1004166495 shutdown_connections 2021-02-10T11:30:49.747 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:49.758 7fad862b0700 1 --2- 172.21.2.112:0/1004166495 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fad600186c0 0x7fad6001abc0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:49.747 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:49.758 7fad862b0700 1 --2- 172.21.2.112:0/1004166495 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fad80189d00 0x7fad8018a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:49.747 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:49.758 7fad862b0700 1 --2- 172.21.2.112:0/1004166495 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fad8018a720 0x7fad800b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:49.747 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:49.758 7fad862b0700 1 --2- 172.21.2.112:0/1004166495 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fad800a0420 0x7fad800a0820 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:49.747 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:49.758 7fad862b0700 1 -- 172.21.2.112:0/1004166495 >> 172.21.2.112:0/1004166495 conn(0x7fad80094e10 msgr2=0x7fad800a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:49.747 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:49.758 7fad862b0700 1 -- 172.21.2.112:0/1004166495 shutdown_connections 2021-02-10T11:30:49.747 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:49.758 7fad862b0700 1 -- 172.21.2.112:0/1004166495 wait complete. 2021-02-10T11:30:49.755 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000348635 2021-02-10T11:30:50.187 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.181 7f06ab698700 1 Processor -- start 2021-02-10T11:30:50.187 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.182 7f06ab698700 1 -- start start 2021-02-10T11:30:50.187 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.182 7f06ab698700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f06a41b0470 0x7f06a41b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:50.187 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.182 7f06ab698700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f06a40ef610 0x7f06a40efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:50.187 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.182 7f06ab698700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f06a40f0030 0x7f06a40f0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:50.187 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.182 7f06ab698700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06a40e5ea0 con 0x7f06a40f0030 2021-02-10T11:30:50.187 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.182 7f06ab698700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06a40f0b50 con 0x7f06a40ef610 2021-02-10T11:30:50.187 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.182 7f06ab698700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06a40e52e0 con 0x7f06a41b0470 2021-02-10T11:30:50.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.182 7f06a9cb5700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f06a40f0030 0x7f06a40f0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:50.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.182 7f06a94b4700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f06a41b0470 0x7f06a41b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:50.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.182 7f06a9cb5700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f06a40f0030 0x7f06a40f0430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:47486/0 (socket says 172.21.2.112:47486) 2021-02-10T11:30:50.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.182 7f06a9cb5700 1 -- 172.21.2.112:0/626623087 learned_addr learned my addr 172.21.2.112:0/626623087 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:30:50.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.182 7f06a8cb3700 1 --2- 172.21.2.112:0/626623087 >> v2:172.21.2.120:3300/0 conn(0x7f06a40ef610 0x7f06a40efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:50.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.183 7f06a94b4700 1 -- 172.21.2.112:0/626623087 >> v2:172.21.2.120:3300/0 conn(0x7f06a40ef610 msgr2=0x7f06a40efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:50.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.183 7f06a94b4700 1 --2- 172.21.2.112:0/626623087 >> v2:172.21.2.120:3300/0 conn(0x7f06a40ef610 0x7f06a40efa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:50.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.183 7f06a94b4700 1 -- 172.21.2.112:0/626623087 >> v2:172.21.2.112:3300/0 conn(0x7f06a40f0030 msgr2=0x7f06a40f0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:50.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.183 7f06a94b4700 1 --2- 172.21.2.112:0/626623087 >> v2:172.21.2.112:3300/0 conn(0x7f06a40f0030 0x7f06a40f0430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:50.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.183 7f06a94b4700 1 -- 172.21.2.112:0/626623087 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06a40f10d0 con 0x7f06a41b0470 2021-02-10T11:30:50.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.183 7f06a9cb5700 1 --2- 172.21.2.112:0/626623087 >> v2:172.21.2.112:3300/0 conn(0x7f06a40f0030 0x7f06a40f0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:30:50.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.183 7f06a94b4700 1 --2- 172.21.2.112:0/626623087 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f06a41b0470 0x7f06a41b0870 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f0698006650 tx=0x7f0698009a80).ready entity=mon.2 client_cookie=75b0463738e47382 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:50.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.183 7f0697fff700 1 -- 172.21.2.112:0/626623087 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f06980009b0 con 0x7f06a41b0470 2021-02-10T11:30:50.189 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.183 7f0697fff700 1 -- 172.21.2.112:0/626623087 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f069800c0c0 con 0x7f06a41b0470 2021-02-10T11:30:50.189 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.183 7f0697fff700 1 -- 172.21.2.112:0/626623087 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f069800c550 con 0x7f06a41b0470 2021-02-10T11:30:50.189 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.183 7f06ab698700 1 -- 172.21.2.112:0/626623087 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f06a41b0470 msgr2=0x7f06a41b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:50.189 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.183 7f06ab698700 1 --2- 172.21.2.112:0/626623087 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f06a41b0470 0x7f06a41b0870 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f0698006650 tx=0x7f0698009a80).stop 2021-02-10T11:30:50.189 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.184 7f06ab698700 1 -- 172.21.2.112:0/626623087 shutdown_connections 2021-02-10T11:30:50.189 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.184 7f06ab698700 1 --2- 172.21.2.112:0/626623087 >> v2:172.21.2.112:3300/0 conn(0x7f06a40f0030 0x7f06a40f0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:50.189 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.184 7f06ab698700 1 --2- 172.21.2.112:0/626623087 >> v2:172.21.2.120:3300/0 conn(0x7f06a40ef610 0x7f06a40efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:50.189 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.184 7f06ab698700 1 --2- 172.21.2.112:0/626623087 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f06a41b0470 0x7f06a41b0870 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:50.189 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.184 7f06ab698700 1 -- 172.21.2.112:0/626623087 >> 172.21.2.112:0/626623087 conn(0x7f06a40ce3e0 msgr2=0x7f06a40e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:50.189 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.184 7f06ab698700 1 -- 172.21.2.112:0/626623087 shutdown_connections 2021-02-10T11:30:50.189 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.184 7f06ab698700 1 -- 172.21.2.112:0/626623087 wait complete. 2021-02-10T11:30:50.191 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.199 7f06ab698700 1 Processor -- start 2021-02-10T11:30:50.191 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.199 7f06ab698700 1 -- start start 2021-02-10T11:30:50.191 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.199 7f06ab698700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f06a40a0910 0x7f06a4105d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:50.191 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.199 7f06ab698700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f06a41a7a10 0x7f06a41a7e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:50.191 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.199 7f06ab698700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f06a4093770 0x7f06a4093b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:50.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.199 7f06ab698700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06a4106390 con 0x7f06a40a0910 2021-02-10T11:30:50.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.199 7f06ab698700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06a4094190 con 0x7f06a4093770 2021-02-10T11:30:50.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.199 7f06ab698700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06a40943b0 con 0x7f06a41a7a10 2021-02-10T11:30:50.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.199 7f06a94b4700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f06a40a0910 0x7f06a4105d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:50.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.199 7f06a94b4700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f06a40a0910 0x7f06a4105d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:47492/0 (socket says 172.21.2.112:47492) 2021-02-10T11:30:50.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.199 7f06a94b4700 1 -- 172.21.2.112:0/3703576389 learned_addr learned my addr 172.21.2.112:0/3703576389 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:30:50.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.200 7f06a94b4700 1 -- 172.21.2.112:0/3703576389 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f06a41a7a10 msgr2=0x7f06a41a7e10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:30:50.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.200 7f06a94b4700 1 --2- 172.21.2.112:0/3703576389 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f06a41a7a10 0x7f06a41a7e10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:50.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.200 7f06a94b4700 1 -- 172.21.2.112:0/3703576389 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f06a4093770 msgr2=0x7f06a4093b70 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:30:50.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.200 7f06a94b4700 1 --2- 172.21.2.112:0/3703576389 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f06a4093770 0x7f06a4093b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:50.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.200 7f06a94b4700 1 -- 172.21.2.112:0/3703576389 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06a4106c30 con 0x7f06a40a0910 2021-02-10T11:30:50.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.200 7f06a94b4700 1 --2- 172.21.2.112:0/3703576389 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f06a40a0910 0x7f06a4105d70 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 rx=0x7f069800b530 tx=0x7f069800b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:50.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.200 7f0695ffb700 1 -- 172.21.2.112:0/3703576389 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0698000940 con 0x7f06a40a0910 2021-02-10T11:30:50.193 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.200 7f0695ffb700 1 -- 172.21.2.112:0/3703576389 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06980093a0 con 0x7f06a40a0910 2021-02-10T11:30:50.193 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.200 7f0695ffb700 1 -- 172.21.2.112:0/3703576389 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0698002080 con 0x7f06a40a0910 2021-02-10T11:30:50.193 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.200 7f06ab698700 1 -- 172.21.2.112:0/3703576389 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06a4106c30 con 0x7f06a40a0910 2021-02-10T11:30:50.193 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.200 7f06ab698700 1 -- 172.21.2.112:0/3703576389 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06a41071c0 con 0x7f06a40a0910 2021-02-10T11:30:50.193 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.201 7f0695ffb700 1 -- 172.21.2.112:0/3703576389 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0698000c90 con 0x7f06a40a0910 2021-02-10T11:30:50.193 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.201 7f0695ffb700 1 --2- 172.21.2.112:0/3703576389 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f06840186c0 0x7f068401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:50.193 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.201 7f0695ffb700 1 -- 172.21.2.112:0/3703576389 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f06980325c0 con 0x7f06a40a0910 2021-02-10T11:30:50.193 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.201 7f06a8cb3700 1 --2- 172.21.2.112:0/3703576389 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f06840186c0 0x7f068401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:50.193 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.202 7f06a8cb3700 1 --2- 172.21.2.112:0/3703576389 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f06840186c0 0x7f068401abc0 crc :-1 s=READY pgs=456 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:50.220 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.232 7f06ab698700 1 -- 172.21.2.112:0/3703576389 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06a4105b40 con 0x7f06a40a0910 2021-02-10T11:30:50.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.234 7f0695ffb700 1 -- 172.21.2.112:0/3703576389 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f069802ef70 con 0x7f06a40a0910 2021-02-10T11:30:50.351 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.363 7f06ab698700 1 -- 172.21.2.112:0/3703576389 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348635", "target": ["mgr", ""]}) v1 -- 0x7f06a41071c0 con 0x7f06840186c0 2021-02-10T11:30:50.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.373 7f0695ffb700 1 -- 172.21.2.112:0/3703576389 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f06a00009b0 con 0x7f06840186c0 2021-02-10T11:30:50.368 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.379 7f06ab698700 1 -- 172.21.2.112:0/3703576389 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f06840186c0 msgr2=0x7f068401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:50.368 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.379 7f06ab698700 1 --2- 172.21.2.112:0/3703576389 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f06840186c0 0x7f068401abc0 crc :-1 s=READY pgs=456 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:50.368 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.379 7f06ab698700 1 -- 172.21.2.112:0/3703576389 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f06a40a0910 msgr2=0x7f06a4105d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:50.368 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.379 7f06ab698700 1 --2- 172.21.2.112:0/3703576389 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f06a40a0910 0x7f06a4105d70 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 rx=0x7f069800b530 tx=0x7f069800b8d0).stop 2021-02-10T11:30:50.368 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.380 7f06ab698700 1 -- 172.21.2.112:0/3703576389 shutdown_connections 2021-02-10T11:30:50.368 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.380 7f06ab698700 1 --2- 172.21.2.112:0/3703576389 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f06840186c0 0x7f068401abc0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:50.369 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.380 7f06ab698700 1 --2- 172.21.2.112:0/3703576389 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f06a4093770 0x7f06a4093b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:50.369 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.380 7f06ab698700 1 --2- 172.21.2.112:0/3703576389 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f06a41a7a10 0x7f06a41a7e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:50.369 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.380 7f06ab698700 1 --2- 172.21.2.112:0/3703576389 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f06a40a0910 0x7f06a4105d70 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:50.369 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.380 7f06ab698700 1 -- 172.21.2.112:0/3703576389 >> 172.21.2.112:0/3703576389 conn(0x7f06a4095ef0 msgr2=0x7f06a4094e40 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:50.369 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.380 7f06ab698700 1 -- 172.21.2.112:0/3703576389 shutdown_connections 2021-02-10T11:30:50.369 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.380 7f06ab698700 1 -- 172.21.2.112:0/3703576389 wait complete. 2021-02-10T11:30:50.377 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000348636 2021-02-10T11:30:50.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.801 7ff904762700 1 Processor -- start 2021-02-10T11:30:50.815 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.801 7ff904762700 1 -- start start 2021-02-10T11:30:50.815 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.802 7ff904762700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7ff8fc1b0470 0x7ff8fc1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:50.815 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.802 7ff904762700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff8fc0f20e0 0x7ff8fc0f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:50.815 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.802 7ff904762700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8fc0f0b50 con 0x7ff8fc0ef610 2021-02-10T11:30:50.815 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.802 7ff904762700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8fc0f0820 con 0x7ff8fc1b0470 2021-02-10T11:30:50.815 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.802 7ff904762700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8fc0f24e0 con 0x7ff8fc0f20e0 2021-02-10T11:30:50.815 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.802 7ff901d7d700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7ff8fc0ef610 0x7ff8fc0efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:46960/0 (socket says 172.21.2.112:46960) 2021-02-10T11:30:50.815 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.802 7ff901d7d700 1 -- 172.21.2.112:0/2740550241 learned_addr learned my addr 172.21.2.112:0/2740550241 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:30:50.815 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.802 7ff90257e700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7ff8fc1b0470 0x7ff8fc1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:50.816 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.802 7ff902d7f700 1 --2- 172.21.2.112:0/2740550241 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff8fc0f20e0 0x7ff8fc0f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:50.816 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.802 7ff902d7f700 1 -- 172.21.2.112:0/2740550241 >> v2:172.21.2.120:3300/0 conn(0x7ff8fc1b0470 msgr2=0x7ff8fc1b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:50.816 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.802 7ff902d7f700 1 --2- 172.21.2.112:0/2740550241 >> v2:172.21.2.120:3300/0 conn(0x7ff8fc1b0470 0x7ff8fc1b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:50.816 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.802 7ff902d7f700 1 -- 172.21.2.112:0/2740550241 >> v1:172.21.2.112:6789/0 conn(0x7ff8fc0ef610 legacy=0x7ff8fc0efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:50.816 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.802 7ff90157c700 1 -- 172.21.2.112:0/2740550241 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4004468610 0 0) 0x7ff8ec0010b0 con 0x7ff8fc0ef610 2021-02-10T11:30:50.816 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.802 7ff902d7f700 1 -- 172.21.2.112:0/2740550241 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8fc0e52e0 con 0x7ff8fc0f20e0 2021-02-10T11:30:50.816 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.802 7ff90257e700 1 --2- 172.21.2.112:0/2740550241 >> v2:172.21.2.120:3300/0 conn(0x7ff8fc1b0470 0x7ff8fc1b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:30:50.816 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.803 7ff902d7f700 1 --2- 172.21.2.112:0/2740550241 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff8fc0f20e0 0x7ff8fc0f0200 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7ff8f8006ad0 tx=0x7ff8f8009ad0).ready entity=mon.2 client_cookie=a1bb3d301071f2f6 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:50.816 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.803 7ff90157c700 1 -- 172.21.2.112:0/2740550241 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff8f80066a0 con 0x7ff8fc0f20e0 2021-02-10T11:30:50.816 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.803 7ff90157c700 1 -- 172.21.2.112:0/2740550241 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8f8006410 con 0x7ff8fc0f20e0 2021-02-10T11:30:50.816 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.803 7ff90157c700 1 -- 172.21.2.112:0/2740550241 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff8f8000c10 con 0x7ff8fc0f20e0 2021-02-10T11:30:50.816 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.803 7ff904762700 1 -- 172.21.2.112:0/2740550241 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff8fc0f20e0 msgr2=0x7ff8fc0f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:50.816 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.803 7ff904762700 1 --2- 172.21.2.112:0/2740550241 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff8fc0f20e0 0x7ff8fc0f0200 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7ff8f8006ad0 tx=0x7ff8f8009ad0).stop 2021-02-10T11:30:50.816 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.803 7ff904762700 1 -- 172.21.2.112:0/2740550241 shutdown_connections 2021-02-10T11:30:50.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.803 7ff904762700 1 --2- 172.21.2.112:0/2740550241 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff8fc0f20e0 0x7ff8fc0f0200 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:50.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.803 7ff904762700 1 --2- 172.21.2.112:0/2740550241 >> v2:172.21.2.120:3300/0 conn(0x7ff8fc1b0470 0x7ff8fc1b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:50.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.803 7ff904762700 1 -- 172.21.2.112:0/2740550241 >> 172.21.2.112:0/2740550241 conn(0x7ff8fc0ce3e0 msgr2=0x7ff8fc0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:50.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.804 7ff904762700 1 -- 172.21.2.112:0/2740550241 shutdown_connections 2021-02-10T11:30:50.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.804 7ff904762700 1 -- 172.21.2.112:0/2740550241 wait complete. 2021-02-10T11:30:50.819 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.826 7ff904762700 1 Processor -- start 2021-02-10T11:30:50.819 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.827 7ff904762700 1 -- start start 2021-02-10T11:30:50.819 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.827 7ff904762700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff8fc0a0420 0x7ff8fc0a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:50.819 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.827 7ff904762700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff8fc189d00 0x7ff8fc18a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:50.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.827 7ff904762700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff8fc18a720 0x7ff8fc0b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:50.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.827 7ff904762700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8fc189800 con 0x7ff8fc189d00 2021-02-10T11:30:50.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.827 7ff904762700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8fc18ab20 con 0x7ff8fc0a0420 2021-02-10T11:30:50.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.827 7ff904762700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8fc0a1140 con 0x7ff8fc18a720 2021-02-10T11:30:50.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.827 7ff90257e700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff8fc0a0420 0x7ff8fc0a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:50.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.827 7ff90257e700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff8fc0a0420 0x7ff8fc0a0820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3300/0 says I am v2:172.21.2.112:49782/0 (socket says 172.21.2.112:49782) 2021-02-10T11:30:50.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.827 7ff90257e700 1 -- 172.21.2.112:0/1922676507 learned_addr learned my addr 172.21.2.112:0/1922676507 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:30:50.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.827 7ff901d7d700 1 --2- 172.21.2.112:0/1922676507 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff8fc189d00 0x7ff8fc18a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:50.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.827 7ff902d7f700 1 --2- 172.21.2.112:0/1922676507 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff8fc18a720 0x7ff8fc0b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:50.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.828 7ff90257e700 1 -- 172.21.2.112:0/1922676507 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff8fc18a720 msgr2=0x7ff8fc0b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:50.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.828 7ff90257e700 1 --2- 172.21.2.112:0/1922676507 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff8fc18a720 0x7ff8fc0b7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:50.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.828 7ff90257e700 1 -- 172.21.2.112:0/1922676507 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff8fc189d00 msgr2=0x7ff8fc18a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:50.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.828 7ff90257e700 1 --2- 172.21.2.112:0/1922676507 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff8fc189d00 0x7ff8fc18a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:50.821 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.828 7ff90257e700 1 -- 172.21.2.112:0/1922676507 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8fc0b8200 con 0x7ff8fc0a0420 2021-02-10T11:30:50.821 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.828 7ff90257e700 1 --2- 172.21.2.112:0/1922676507 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff8fc0a0420 0x7ff8fc0a0820 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7ff8f400b970 tx=0x7ff8f400bd10).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:50.821 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.828 7ff8f2ffd700 1 -- 172.21.2.112:0/1922676507 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff8f4000d30 con 0x7ff8fc0a0420 2021-02-10T11:30:50.821 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.828 7ff8f2ffd700 1 -- 172.21.2.112:0/1922676507 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8f4006020 con 0x7ff8fc0a0420 2021-02-10T11:30:50.821 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.828 7ff904762700 1 -- 172.21.2.112:0/1922676507 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8fc0b8200 con 0x7ff8fc0a0420 2021-02-10T11:30:50.821 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.828 7ff8f2ffd700 1 -- 172.21.2.112:0/1922676507 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff8f40064b0 con 0x7ff8fc0a0420 2021-02-10T11:30:50.821 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.828 7ff904762700 1 -- 172.21.2.112:0/1922676507 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8fc0b8200 con 0x7ff8fc0a0420 2021-02-10T11:30:50.821 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.829 7ff8f2ffd700 1 -- 172.21.2.112:0/1922676507 <== mon.1 v2:172.21.2.120:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff8f4006410 con 0x7ff8fc0a0420 2021-02-10T11:30:50.821 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.829 7ff8f2ffd700 1 --2- 172.21.2.112:0/1922676507 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff8e40186c0 0x7ff8e401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:50.821 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.829 7ff8f2ffd700 1 -- 172.21.2.112:0/1922676507 <== mon.1 v2:172.21.2.120:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7ff8f4032760 con 0x7ff8fc0a0420 2021-02-10T11:30:50.821 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.829 7ff901d7d700 1 --2- 172.21.2.112:0/1922676507 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff8e40186c0 0x7ff8e401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:50.821 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.830 7ff901d7d700 1 --2- 172.21.2.112:0/1922676507 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff8e40186c0 0x7ff8e401abc0 crc :-1 s=READY pgs=457 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:50.840 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.852 7ff904762700 1 -- 172.21.2.112:0/1922676507 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8fc18ab20 con 0x7ff8fc0a0420 2021-02-10T11:30:50.842 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.854 7ff8f2ffd700 1 -- 172.21.2.112:0/1922676507 <== mon.1 v2:172.21.2.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff8f40061e0 con 0x7ff8fc0a0420 2021-02-10T11:30:50.972 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.983 7ff904762700 1 -- 172.21.2.112:0/1922676507 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348636", "target": ["mgr", ""]}) v1 -- 0x7ff8fc18ab20 con 0x7ff8e40186c0 2021-02-10T11:30:50.982 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:50.995 7ff8f2ffd700 1 -- 172.21.2.112:0/1922676507 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7ff8ec0009b0 con 0x7ff8e40186c0 2021-02-10T11:30:50.989 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:51.000 7ff904762700 1 -- 172.21.2.112:0/1922676507 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff8e40186c0 msgr2=0x7ff8e401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:50.989 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:51.000 7ff904762700 1 --2- 172.21.2.112:0/1922676507 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff8e40186c0 0x7ff8e401abc0 crc :-1 s=READY pgs=457 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:50.989 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:51.000 7ff904762700 1 -- 172.21.2.112:0/1922676507 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff8fc0a0420 msgr2=0x7ff8fc0a0820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:50.989 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:51.000 7ff904762700 1 --2- 172.21.2.112:0/1922676507 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff8fc0a0420 0x7ff8fc0a0820 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7ff8f400b970 tx=0x7ff8f400bd10).stop 2021-02-10T11:30:50.989 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:51.000 7ff904762700 1 -- 172.21.2.112:0/1922676507 shutdown_connections 2021-02-10T11:30:50.989 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:51.000 7ff904762700 1 --2- 172.21.2.112:0/1922676507 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff8e40186c0 0x7ff8e401abc0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:50.989 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:51.000 7ff904762700 1 --2- 172.21.2.112:0/1922676507 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff8fc18a720 0x7ff8fc0b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:50.989 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:51.000 7ff904762700 1 --2- 172.21.2.112:0/1922676507 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff8fc189d00 0x7ff8fc18a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:50.989 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:51.000 7ff904762700 1 --2- 172.21.2.112:0/1922676507 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff8fc0a0420 0x7ff8fc0a0820 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:50.990 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:51.000 7ff904762700 1 -- 172.21.2.112:0/1922676507 >> 172.21.2.112:0/1922676507 conn(0x7ff8fc094e10 msgr2=0x7ff8fc0a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:50.990 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:51.001 7ff904762700 1 -- 172.21.2.112:0/1922676507 shutdown_connections 2021-02-10T11:30:50.990 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:51.001 7ff904762700 1 -- 172.21.2.112:0/1922676507 wait complete. 2021-02-10T11:30:50.997 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000348637 2021-02-10T11:30:51.428 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:51.422 7fa975bd4700 1 Processor -- start 2021-02-10T11:30:51.428 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:51.422 7fa975bd4700 1 -- start start 2021-02-10T11:30:51.428 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:51.423 7fa975bd4700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fa9701b0470 0x7fa9701b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:51.428 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:51.423 7fa975bd4700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa9700f20e0 0x7fa9700f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:51.429 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:51.423 7fa975bd4700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9700f0b50 con 0x7fa9700ef610 2021-02-10T11:30:51.429 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:51.423 7fa975bd4700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9700f0820 con 0x7fa9701b0470 2021-02-10T11:30:51.429 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:51.423 7fa975bd4700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9700f24e0 con 0x7fa9700f20e0 2021-02-10T11:30:51.429 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:51.423 7fa96f7fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fa9701b0470 0x7fa9701b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:51.429 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:51.423 7fa96effd700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7fa9700ef610 0x7fa9700efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:40530/0 (socket says 172.21.2.112:40530) 2021-02-10T11:30:51.429 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:51.423 7fa96effd700 1 -- 172.21.2.112:0/251852323 learned_addr learned my addr 172.21.2.112:0/251852323 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:30:51.429 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:51.423 7fa96ffff700 1 --2- 172.21.2.112:0/251852323 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa9700f20e0 0x7fa9700f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:51.429 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:51.424 7fa96f7fe700 1 -- 172.21.2.112:0/251852323 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa9700f20e0 msgr2=0x7fa9700f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:51.429 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:51.424 7fa96f7fe700 1 --2- 172.21.2.112:0/251852323 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa9700f20e0 0x7fa9700f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:51.429 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:51.424 7fa96f7fe700 1 -- 172.21.2.112:0/251852323 >> v1:172.21.2.120:6789/0 conn(0x7fa9700ef610 legacy=0x7fa9700efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:51.429 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:51.424 7fa96f7fe700 1 -- 172.21.2.112:0/251852323 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9700e52e0 con 0x7fa9701b0470 2021-02-10T11:30:51.429 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:51.424 7fa96f7fe700 1 --2- 172.21.2.112:0/251852323 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa9701b0470 0x7fa9701b0870 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 rx=0x7fa9680069d0 tx=0x7fa968009dc0).ready entity=mon.0 client_cookie=5bf93ecba12fefe9 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:51.430 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:51.424 7fa96e7fc700 1 -- 172.21.2.112:0/251852323 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa96800b510 con 0x7fa9701b0470 2021-02-10T11:30:51.430 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:51.424 7fa96e7fc700 1 -- 172.21.2.112:0/251852323 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa96800b790 con 0x7fa9701b0470 2021-02-10T11:30:51.430 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:51.424 7fa96e7fc700 1 -- 172.21.2.112:0/251852323 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa96800bcb0 con 0x7fa9701b0470 2021-02-10T11:30:51.430 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:51.424 7fa975bd4700 1 -- 172.21.2.112:0/251852323 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa9701b0470 msgr2=0x7fa9701b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:51.430 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:51.424 7fa975bd4700 1 --2- 172.21.2.112:0/251852323 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa9701b0470 0x7fa9701b0870 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 rx=0x7fa9680069d0 tx=0x7fa968009dc0).stop 2021-02-10T11:30:51.430 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:51.425 7fa975bd4700 1 -- 172.21.2.112:0/251852323 shutdown_connections 2021-02-10T11:30:51.430 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:51.425 7fa975bd4700 1 --2- 172.21.2.112:0/251852323 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa9700f20e0 0x7fa9700f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:51.430 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:51.425 7fa975bd4700 1 --2- 172.21.2.112:0/251852323 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa9701b0470 0x7fa9701b0870 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:51.430 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:51.425 7fa975bd4700 1 -- 172.21.2.112:0/251852323 >> 172.21.2.112:0/251852323 conn(0x7fa9700ce3e0 msgr2=0x7fa9700e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:51.430 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:51.425 7fa975bd4700 1 -- 172.21.2.112:0/251852323 shutdown_connections 2021-02-10T11:30:51.430 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:51.425 7fa975bd4700 1 -- 172.21.2.112:0/251852323 wait complete. 2021-02-10T11:30:51.430 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:51.440 7fa975bd4700 1 Processor -- start 2021-02-10T11:30:51.430 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:51.440 7fa975bd4700 1 -- start start 2021-02-10T11:30:51.432 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:51.440 7fa975bd4700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa97009e400 0x7fa9701a7a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:51.432 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:51.440 7fa975bd4700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa9701a8050 0x7fa9700b2fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:51.432 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:51.440 7fa975bd4700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa9700b35c0 0x7fa9700b39c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:51.432 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:51.440 7fa975bd4700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9700bfe70 con 0x7fa9701a8050 2021-02-10T11:30:51.432 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:51.440 7fa975bd4700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa97009e980 con 0x7fa97009e400 2021-02-10T11:30:51.432 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:51.440 7fa975bd4700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9700bf9e0 con 0x7fa9700b35c0 2021-02-10T11:30:51.433 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:51.441 7fa96effd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa9701a8050 0x7fa9700b2fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:51.433 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:51.441 7fa96effd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa9701a8050 0x7fa9700b2fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:47514/0 (socket says 172.21.2.112:47514) 2021-02-10T11:30:51.433 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:51.441 7fa96ffff700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa9700b35c0 0x7fa9700b39c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:51.433 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:51.441 7fa96effd700 1 -- 172.21.2.112:0/1496175790 learned_addr learned my addr 172.21.2.112:0/1496175790 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:30:51.433 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:51.441 7fa96f7fe700 1 --2- 172.21.2.112:0/1496175790 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa97009e400 0x7fa9701a7a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:51.433 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:51.441 7fa96ffff700 1 -- 172.21.2.112:0/1496175790 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa97009e400 msgr2=0x7fa9701a7a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:51.434 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:51.441 7fa96ffff700 1 --2- 172.21.2.112:0/1496175790 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa97009e400 0x7fa9701a7a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:51.434 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:51.441 7fa96ffff700 1 -- 172.21.2.112:0/1496175790 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa9701a8050 msgr2=0x7fa9700b2fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:51.434 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:51.441 7fa96ffff700 1 --2- 172.21.2.112:0/1496175790 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa9701a8050 0x7fa9700b2fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:51.434 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:51.441 7fa96ffff700 1 -- 172.21.2.112:0/1496175790 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9700c0370 con 0x7fa9700b35c0 2021-02-10T11:30:51.434 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:51.441 7fa96ffff700 1 --2- 172.21.2.112:0/1496175790 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa9700b35c0 0x7fa9700b39c0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7fa958006ed0 tx=0x7fa95800b910).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:51.434 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:51.441 7fa967fff700 1 -- 172.21.2.112:0/1496175790 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa958006b30 con 0x7fa9700b35c0 2021-02-10T11:30:51.434 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:51.441 7fa967fff700 1 -- 172.21.2.112:0/1496175790 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa95800bd00 con 0x7fa9700b35c0 2021-02-10T11:30:51.435 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:51.441 7fa975bd4700 1 -- 172.21.2.112:0/1496175790 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9700c0370 con 0x7fa9700b35c0 2021-02-10T11:30:51.435 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:51.441 7fa967fff700 1 -- 172.21.2.112:0/1496175790 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa958000c10 con 0x7fa9700b35c0 2021-02-10T11:30:51.435 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:51.441 7fa975bd4700 1 -- 172.21.2.112:0/1496175790 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9700c0370 con 0x7fa9700b35c0 2021-02-10T11:30:51.435 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:51.442 7fa967fff700 1 -- 172.21.2.112:0/1496175790 <== mon.2 v2:172.21.2.120:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa958005330 con 0x7fa9700b35c0 2021-02-10T11:30:51.435 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:51.442 7fa967fff700 1 --2- 172.21.2.112:0/1496175790 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa954018730 0x7fa95401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:51.435 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:51.442 7fa967fff700 1 -- 172.21.2.112:0/1496175790 <== mon.2 v2:172.21.2.120:3301/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fa958032560 con 0x7fa9700b35c0 2021-02-10T11:30:51.435 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:51.442 7fa96f7fe700 1 --2- 172.21.2.112:0/1496175790 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa954018730 0x7fa95401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:51.435 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:51.443 7fa96f7fe700 1 --2- 172.21.2.112:0/1496175790 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa954018730 0x7fa95401ac10 crc :-1 s=READY pgs=458 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:51.461 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:51.473 7fa975bd4700 1 -- 172.21.2.112:0/1496175790 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa970105ae0 con 0x7fa9700b35c0 2021-02-10T11:30:51.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:51.478 7fa967fff700 1 -- 172.21.2.112:0/1496175790 <== mon.2 v2:172.21.2.120:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa958000c10 con 0x7fa9700b35c0 2021-02-10T11:30:51.596 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:51.607 7fa975bd4700 1 -- 172.21.2.112:0/1496175790 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348637", "target": ["mgr", ""]}) v1 -- 0x7fa970105ae0 con 0x7fa954018730 2021-02-10T11:30:51.606 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:51.619 7fa967fff700 1 -- 172.21.2.112:0/1496175790 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fa968000d30 con 0x7fa954018730 2021-02-10T11:30:51.615 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:51.625 7fa975bd4700 1 -- 172.21.2.112:0/1496175790 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa954018730 msgr2=0x7fa95401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:51.615 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:51.625 7fa975bd4700 1 --2- 172.21.2.112:0/1496175790 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa954018730 0x7fa95401ac10 crc :-1 s=READY pgs=458 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:51.615 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:51.625 7fa975bd4700 1 -- 172.21.2.112:0/1496175790 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa9700b35c0 msgr2=0x7fa9700b39c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:51.615 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:51.625 7fa975bd4700 1 --2- 172.21.2.112:0/1496175790 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa9700b35c0 0x7fa9700b39c0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7fa958006ed0 tx=0x7fa95800b910).stop 2021-02-10T11:30:51.615 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:51.625 7fa975bd4700 1 -- 172.21.2.112:0/1496175790 shutdown_connections 2021-02-10T11:30:51.616 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:51.625 7fa975bd4700 1 --2- 172.21.2.112:0/1496175790 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa954018730 0x7fa95401ac10 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:51.616 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:51.625 7fa975bd4700 1 --2- 172.21.2.112:0/1496175790 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa9700b35c0 0x7fa9700b39c0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:51.616 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:51.625 7fa975bd4700 1 --2- 172.21.2.112:0/1496175790 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa9701a8050 0x7fa9700b2fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:51.616 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:51.625 7fa975bd4700 1 --2- 172.21.2.112:0/1496175790 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa97009e400 0x7fa9701a7a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:51.616 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:51.625 7fa975bd4700 1 -- 172.21.2.112:0/1496175790 >> 172.21.2.112:0/1496175790 conn(0x7fa9700a6b10 msgr2=0x7fa9700bf0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:51.617 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:51.626 7fa975bd4700 1 -- 172.21.2.112:0/1496175790 shutdown_connections 2021-02-10T11:30:51.617 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:51.626 7fa975bd4700 1 -- 172.21.2.112:0/1496175790 wait complete. 2021-02-10T11:30:51.623 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000348638 2021-02-10T11:30:52.054 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.047 7fea9bcc4700 1 Processor -- start 2021-02-10T11:30:52.056 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.047 7fea9bcc4700 1 -- start start 2021-02-10T11:30:52.056 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.048 7fea9bcc4700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fea941b0470 0x7fea941b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:52.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.048 7fea9bcc4700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fea940ef610 0x7fea940efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:52.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.048 7fea9bcc4700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea940f0b50 con 0x7fea940f0030 2021-02-10T11:30:52.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.048 7fea9bcc4700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea940e52e0 con 0x7fea941b0470 2021-02-10T11:30:52.058 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.048 7fea9bcc4700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea940e54d0 con 0x7fea940ef610 2021-02-10T11:30:52.058 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.048 7fea99ae0700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fea941b0470 0x7fea941b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:52.058 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.048 7fea9a2e1700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fea940f0030 0x7fea940f10d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:46984/0 (socket says 172.21.2.112:46984) 2021-02-10T11:30:52.058 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.048 7fea992df700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fea940ef610 0x7fea940efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:52.058 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.048 7fea9a2e1700 1 -- 172.21.2.112:0/1378989514 learned_addr learned my addr 172.21.2.112:0/1378989514 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:30:52.059 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.048 7fea98ade700 1 -- 172.21.2.112:0/1378989514 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2958884717 0 0) 0x7fea8c001500 con 0x7fea940f0030 2021-02-10T11:30:52.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.048 7fea992df700 1 -- 172.21.2.112:0/1378989514 >> v2:172.21.2.112:3300/0 conn(0x7fea941b0470 msgr2=0x7fea941b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:52.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.048 7fea992df700 1 --2- 172.21.2.112:0/1378989514 >> v2:172.21.2.112:3300/0 conn(0x7fea941b0470 0x7fea941b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:52.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.048 7fea992df700 1 -- 172.21.2.112:0/1378989514 >> v1:172.21.2.112:6789/0 conn(0x7fea940f0030 legacy=0x7fea940f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:52.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.048 7fea992df700 1 -- 172.21.2.112:0/1378989514 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea940f0430 con 0x7fea940ef610 2021-02-10T11:30:52.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.048 7fea992df700 1 --2- 172.21.2.112:0/1378989514 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fea940ef610 0x7fea940efa10 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7fea84006650 tx=0x7fea84009a80).ready entity=mon.2 client_cookie=b8250bba2fb89d7d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:52.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.049 7fea98ade700 1 -- 172.21.2.112:0/1378989514 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fea8400c0c0 con 0x7fea940ef610 2021-02-10T11:30:52.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.049 7fea98ade700 1 -- 172.21.2.112:0/1378989514 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea8400c340 con 0x7fea940ef610 2021-02-10T11:30:52.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.049 7fea98ade700 1 -- 172.21.2.112:0/1378989514 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fea8400c820 con 0x7fea940ef610 2021-02-10T11:30:52.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.049 7fea9bcc4700 1 -- 172.21.2.112:0/1378989514 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fea940ef610 msgr2=0x7fea940efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:52.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.049 7fea9bcc4700 1 --2- 172.21.2.112:0/1378989514 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fea940ef610 0x7fea940efa10 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7fea84006650 tx=0x7fea84009a80).stop 2021-02-10T11:30:52.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.049 7fea9bcc4700 1 -- 172.21.2.112:0/1378989514 shutdown_connections 2021-02-10T11:30:52.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.049 7fea9bcc4700 1 --2- 172.21.2.112:0/1378989514 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fea940ef610 0x7fea940efa10 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:52.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.049 7fea9bcc4700 1 --2- 172.21.2.112:0/1378989514 >> v2:172.21.2.112:3300/0 conn(0x7fea941b0470 0x7fea941b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:52.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.049 7fea9bcc4700 1 -- 172.21.2.112:0/1378989514 >> 172.21.2.112:0/1378989514 conn(0x7fea940ce3e0 msgr2=0x7fea940e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:52.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.049 7fea9bcc4700 1 -- 172.21.2.112:0/1378989514 shutdown_connections 2021-02-10T11:30:52.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.049 7fea9bcc4700 1 -- 172.21.2.112:0/1378989514 wait complete. 2021-02-10T11:30:52.063 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.064 7fea9bcc4700 1 Processor -- start 2021-02-10T11:30:52.063 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.064 7fea9bcc4700 1 -- start start 2021-02-10T11:30:52.063 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.065 7fea9bcc4700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fea940a0420 0x7fea940a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:52.063 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.065 7fea9bcc4700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fea94189d00 0x7fea9418a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:52.063 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.065 7fea9bcc4700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fea9418a720 0x7fea940b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:52.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.065 7fea9bcc4700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea94189780 con 0x7fea94189d00 2021-02-10T11:30:52.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.065 7fea9bcc4700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea9418ab20 con 0x7fea9418a720 2021-02-10T11:30:52.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.065 7fea9bcc4700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea940a1140 con 0x7fea940a0420 2021-02-10T11:30:52.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.065 7fea992df700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fea94189d00 0x7fea9418a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:52.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.065 7fea992df700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fea94189d00 0x7fea9418a100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:47526/0 (socket says 172.21.2.112:47526) 2021-02-10T11:30:52.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.065 7fea992df700 1 -- 172.21.2.112:0/3689127249 learned_addr learned my addr 172.21.2.112:0/3689127249 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:30:52.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.065 7fea9a2e1700 1 --2- 172.21.2.112:0/3689127249 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fea9418a720 0x7fea940b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:52.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.065 7fea99ae0700 1 --2- 172.21.2.112:0/3689127249 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fea940a0420 0x7fea940a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:52.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.065 7fea992df700 1 -- 172.21.2.112:0/3689127249 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fea940a0420 msgr2=0x7fea940a0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:52.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.065 7fea992df700 1 --2- 172.21.2.112:0/3689127249 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fea940a0420 0x7fea940a0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:52.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.066 7fea992df700 1 -- 172.21.2.112:0/3689127249 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fea9418a720 msgr2=0x7fea940b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:52.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.066 7fea992df700 1 --2- 172.21.2.112:0/3689127249 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fea9418a720 0x7fea940b7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:52.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.066 7fea992df700 1 -- 172.21.2.112:0/3689127249 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea940b8200 con 0x7fea94189d00 2021-02-10T11:30:52.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.066 7fea992df700 1 --2- 172.21.2.112:0/3689127249 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fea94189d00 0x7fea9418a100 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 rx=0x7fea84009650 tx=0x7fea8400b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:52.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.066 7fea8a7fc700 1 -- 172.21.2.112:0/3689127249 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fea8400bd50 con 0x7fea94189d00 2021-02-10T11:30:52.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.066 7fea8a7fc700 1 -- 172.21.2.112:0/3689127249 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea84006030 con 0x7fea94189d00 2021-02-10T11:30:52.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.066 7fea8a7fc700 1 -- 172.21.2.112:0/3689127249 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fea840054d0 con 0x7fea94189d00 2021-02-10T11:30:52.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.066 7fea9bcc4700 1 -- 172.21.2.112:0/3689127249 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea940b8200 con 0x7fea94189d00 2021-02-10T11:30:52.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.066 7fea9bcc4700 1 -- 172.21.2.112:0/3689127249 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea940b8200 con 0x7fea94189d00 2021-02-10T11:30:52.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.067 7fea8a7fc700 1 -- 172.21.2.112:0/3689127249 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fea84009300 con 0x7fea94189d00 2021-02-10T11:30:52.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.067 7fea8a7fc700 1 --2- 172.21.2.112:0/3689127249 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fea7c0186c0 0x7fea7c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:52.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.067 7fea8a7fc700 1 -- 172.21.2.112:0/3689127249 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fea84032580 con 0x7fea94189d00 2021-02-10T11:30:52.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.067 7fea99ae0700 1 --2- 172.21.2.112:0/3689127249 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fea7c0186c0 0x7fea7c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:52.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.067 7fea99ae0700 1 --2- 172.21.2.112:0/3689127249 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fea7c0186c0 0x7fea7c01abc0 crc :-1 s=READY pgs=459 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:52.085 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.097 7fea9bcc4700 1 -- 172.21.2.112:0/3689127249 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea94105ae0 con 0x7fea94189d00 2021-02-10T11:30:52.088 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.099 7fea8a7fc700 1 -- 172.21.2.112:0/3689127249 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fea84030db0 con 0x7fea94189d00 2021-02-10T11:30:52.218 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.230 7fea9bcc4700 1 -- 172.21.2.112:0/3689127249 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348638", "target": ["mgr", ""]}) v1 -- 0x7fea94105ae0 con 0x7fea7c0186c0 2021-02-10T11:30:52.228 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.241 7fea8a7fc700 1 -- 172.21.2.112:0/3689127249 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fea90006a70 con 0x7fea7c0186c0 2021-02-10T11:30:52.236 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.248 7fea9bcc4700 1 -- 172.21.2.112:0/3689127249 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fea7c0186c0 msgr2=0x7fea7c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:52.236 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.248 7fea9bcc4700 1 --2- 172.21.2.112:0/3689127249 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fea7c0186c0 0x7fea7c01abc0 crc :-1 s=READY pgs=459 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:52.236 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.248 7fea9bcc4700 1 -- 172.21.2.112:0/3689127249 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fea94189d00 msgr2=0x7fea9418a100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:52.236 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.248 7fea9bcc4700 1 --2- 172.21.2.112:0/3689127249 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fea94189d00 0x7fea9418a100 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 rx=0x7fea84009650 tx=0x7fea8400b960).stop 2021-02-10T11:30:52.236 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.248 7fea9bcc4700 1 -- 172.21.2.112:0/3689127249 shutdown_connections 2021-02-10T11:30:52.236 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.248 7fea9bcc4700 1 --2- 172.21.2.112:0/3689127249 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fea7c0186c0 0x7fea7c01abc0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:52.236 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.248 7fea9bcc4700 1 --2- 172.21.2.112:0/3689127249 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fea9418a720 0x7fea940b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:52.237 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.248 7fea9bcc4700 1 --2- 172.21.2.112:0/3689127249 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fea94189d00 0x7fea9418a100 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:52.237 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.248 7fea9bcc4700 1 --2- 172.21.2.112:0/3689127249 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fea940a0420 0x7fea940a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:52.237 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.248 7fea9bcc4700 1 -- 172.21.2.112:0/3689127249 >> 172.21.2.112:0/3689127249 conn(0x7fea94094e10 msgr2=0x7fea940a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:52.237 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.248 7fea9bcc4700 1 -- 172.21.2.112:0/3689127249 shutdown_connections 2021-02-10T11:30:52.237 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.248 7fea9bcc4700 1 -- 172.21.2.112:0/3689127249 wait complete. 2021-02-10T11:30:52.244 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000348639 2021-02-10T11:30:52.685 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.679 7fe5c24ac700 1 Processor -- start 2021-02-10T11:30:52.685 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.679 7fe5c24ac700 1 -- start start 2021-02-10T11:30:52.685 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.680 7fe5c24ac700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe5bc1b0870 0x7fe5bc0f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:52.685 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.680 7fe5c24ac700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5bc1b0c70 con 0x7fe5bc0efe00 2021-02-10T11:30:52.686 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.680 7fe5c24ac700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5bc0f0b50 con 0x7fe5bc1b0470 2021-02-10T11:30:52.686 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.680 7fe5c24ac700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5bc0e5ea0 con 0x7fe5bc1b0870 2021-02-10T11:30:52.686 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.680 7fe5c0ac9700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fe5bc0efe00 0x7fe5bc0f0200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:46998/0 (socket says 172.21.2.112:46998) 2021-02-10T11:30:52.686 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.680 7fe5c0ac9700 1 -- 172.21.2.112:0/3773708298 learned_addr learned my addr 172.21.2.112:0/3773708298 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:30:52.686 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.680 7fe5bb7fe700 1 --2- 172.21.2.112:0/3773708298 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe5bc1b0870 0x7fe5bc0f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:52.686 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.681 7fe5bb7fe700 1 -- 172.21.2.112:0/3773708298 >> v1:172.21.2.120:6789/0 conn(0x7fe5bc1b0470 legacy=0x7fe5bc0ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:52.686 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.681 7fe5bb7fe700 1 -- 172.21.2.112:0/3773708298 >> v1:172.21.2.112:6789/0 conn(0x7fe5bc0efe00 legacy=0x7fe5bc0f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:52.686 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.681 7fe5bb7fe700 1 -- 172.21.2.112:0/3773708298 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5bc0e52e0 con 0x7fe5bc1b0870 2021-02-10T11:30:52.686 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.681 7fe5baffd700 1 -- 172.21.2.112:0/3773708298 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 71825311 0 0) 0x7fe5b4001500 con 0x7fe5bc0efe00 2021-02-10T11:30:52.686 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.681 7fe5bb7fe700 1 --2- 172.21.2.112:0/3773708298 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe5bc1b0870 0x7fe5bc0f20e0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7fe5b0006650 tx=0x7fe5b0009a80).ready entity=mon.2 client_cookie=78c69fa50d1d58d3 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:52.686 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.681 7fe5baffd700 1 -- 172.21.2.112:0/3773708298 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe5b000c0c0 con 0x7fe5bc1b0870 2021-02-10T11:30:52.686 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.681 7fe5baffd700 1 -- 172.21.2.112:0/3773708298 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5b000c340 con 0x7fe5bc1b0870 2021-02-10T11:30:52.686 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.681 7fe5baffd700 1 -- 172.21.2.112:0/3773708298 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe5b000c860 con 0x7fe5bc1b0870 2021-02-10T11:30:52.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.681 7fe5c24ac700 1 -- 172.21.2.112:0/3773708298 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe5bc1b0870 msgr2=0x7fe5bc0f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:52.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.681 7fe5c24ac700 1 --2- 172.21.2.112:0/3773708298 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe5bc1b0870 0x7fe5bc0f20e0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7fe5b0006650 tx=0x7fe5b0009a80).stop 2021-02-10T11:30:52.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.682 7fe5c24ac700 1 -- 172.21.2.112:0/3773708298 shutdown_connections 2021-02-10T11:30:52.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.682 7fe5c24ac700 1 --2- 172.21.2.112:0/3773708298 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe5bc1b0870 0x7fe5bc0f20e0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:52.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.682 7fe5c24ac700 1 -- 172.21.2.112:0/3773708298 >> 172.21.2.112:0/3773708298 conn(0x7fe5bc0ce3e0 msgr2=0x7fe5bc0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:52.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.682 7fe5c24ac700 1 -- 172.21.2.112:0/3773708298 shutdown_connections 2021-02-10T11:30:52.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.682 7fe5c24ac700 1 -- 172.21.2.112:0/3773708298 wait complete. 2021-02-10T11:30:52.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.697 7fe5c24ac700 1 Processor -- start 2021-02-10T11:30:52.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.697 7fe5c24ac700 1 -- start start 2021-02-10T11:30:52.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.697 7fe5c24ac700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe5bc1a0570 0x7fe5bc0a4590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:52.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.697 7fe5c24ac700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe5bc0f1fd0 0x7fe5bc0f23d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:52.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.697 7fe5c24ac700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe5bc0a5450 0x7fe5bc0a5850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:52.689 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.697 7fe5c24ac700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5bc0a4bb0 con 0x7fe5bc1a0570 2021-02-10T11:30:52.689 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.697 7fe5c24ac700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5bc0b35e0 con 0x7fe5bc0f1fd0 2021-02-10T11:30:52.689 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.697 7fe5c24ac700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5bc0a1f00 con 0x7fe5bc0a5450 2021-02-10T11:30:52.689 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.697 7fe5bbfff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe5bc1a0570 0x7fe5bc0a4590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:52.689 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.697 7fe5c0ac9700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe5bc0a5450 0x7fe5bc0a5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:52.689 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.698 7fe5bbfff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe5bc1a0570 0x7fe5bc0a4590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:47544/0 (socket says 172.21.2.112:47544) 2021-02-10T11:30:52.689 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.698 7fe5c0ac9700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe5bc0a5450 0x7fe5bc0a5850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3301/0 says I am v2:172.21.2.112:47224/0 (socket says 172.21.2.112:47224) 2021-02-10T11:30:52.689 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.698 7fe5bbfff700 1 -- 172.21.2.112:0/2916152187 learned_addr learned my addr 172.21.2.112:0/2916152187 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:30:52.689 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.698 7fe5bb7fe700 1 --2- 172.21.2.112:0/2916152187 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe5bc0f1fd0 0x7fe5bc0f23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:52.689 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.698 7fe5bb7fe700 1 -- 172.21.2.112:0/2916152187 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe5bc0a5450 msgr2=0x7fe5bc0a5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:52.689 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.698 7fe5bb7fe700 1 --2- 172.21.2.112:0/2916152187 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe5bc0a5450 0x7fe5bc0a5850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:52.689 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.698 7fe5bb7fe700 1 -- 172.21.2.112:0/2916152187 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe5bc1a0570 msgr2=0x7fe5bc0a4590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:52.689 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.698 7fe5bb7fe700 1 --2- 172.21.2.112:0/2916152187 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe5bc1a0570 0x7fe5bc0a4590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:52.690 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.698 7fe5bb7fe700 1 -- 172.21.2.112:0/2916152187 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5bc0a5e70 con 0x7fe5bc0f1fd0 2021-02-10T11:30:52.690 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.698 7fe5bbfff700 1 --2- 172.21.2.112:0/2916152187 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe5bc1a0570 0x7fe5bc0a4590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:30:52.690 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.698 7fe5c0ac9700 1 --2- 172.21.2.112:0/2916152187 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe5bc0a5450 0x7fe5bc0a5850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:30:52.690 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.698 7fe5bb7fe700 1 --2- 172.21.2.112:0/2916152187 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe5bc0f1fd0 0x7fe5bc0f23d0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0x7fe5b000b530 tx=0x7fe5b000b8d0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:52.690 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.698 7fe5b8ff9700 1 -- 172.21.2.112:0/2916152187 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe5b00093a0 con 0x7fe5bc0f1fd0 2021-02-10T11:30:52.690 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.698 7fe5b8ff9700 1 -- 172.21.2.112:0/2916152187 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5b000bcc0 con 0x7fe5bc0f1fd0 2021-02-10T11:30:52.690 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.698 7fe5b8ff9700 1 -- 172.21.2.112:0/2916152187 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe5b00022b0 con 0x7fe5bc0f1fd0 2021-02-10T11:30:52.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.698 7fe5c24ac700 1 -- 172.21.2.112:0/2916152187 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5bc0a5e70 con 0x7fe5bc0f1fd0 2021-02-10T11:30:52.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.698 7fe5c24ac700 1 -- 172.21.2.112:0/2916152187 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5bc0a5e70 con 0x7fe5bc0f1fd0 2021-02-10T11:30:52.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.699 7fe5b8ff9700 1 -- 172.21.2.112:0/2916152187 <== mon.1 v2:172.21.2.120:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe5b0002020 con 0x7fe5bc0f1fd0 2021-02-10T11:30:52.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.699 7fe5b8ff9700 1 --2- 172.21.2.112:0/2916152187 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe59c0186c0 0x7fe59c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:52.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.699 7fe5b8ff9700 1 -- 172.21.2.112:0/2916152187 <== mon.1 v2:172.21.2.120:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fe5b00324a0 con 0x7fe5bc0f1fd0 2021-02-10T11:30:52.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.699 7fe5bbfff700 1 --2- 172.21.2.112:0/2916152187 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe59c0186c0 0x7fe59c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:52.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.700 7fe5bbfff700 1 --2- 172.21.2.112:0/2916152187 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe59c0186c0 0x7fe59c01abc0 crc :-1 s=READY pgs=460 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:52.718 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.729 7fe5c24ac700 1 -- 172.21.2.112:0/2916152187 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5bc0e52e0 con 0x7fe5bc0f1fd0 2021-02-10T11:30:52.721 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.733 7fe5b8ff9700 1 -- 172.21.2.112:0/2916152187 <== mon.1 v2:172.21.2.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe5b002d8b0 con 0x7fe5bc0f1fd0 2021-02-10T11:30:52.854 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.866 7fe5c24ac700 1 -- 172.21.2.112:0/2916152187 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348639", "target": ["mgr", ""]}) v1 -- 0x7fe5bc0a4bb0 con 0x7fe59c0186c0 2021-02-10T11:30:52.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.876 7fe5b8ff9700 1 -- 172.21.2.112:0/2916152187 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fe5ac000d50 con 0x7fe59c0186c0 2021-02-10T11:30:52.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.883 7fe5c24ac700 1 -- 172.21.2.112:0/2916152187 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe59c0186c0 msgr2=0x7fe59c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:52.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.883 7fe5c24ac700 1 --2- 172.21.2.112:0/2916152187 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe59c0186c0 0x7fe59c01abc0 crc :-1 s=READY pgs=460 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:52.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.883 7fe5c24ac700 1 -- 172.21.2.112:0/2916152187 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe5bc0f1fd0 msgr2=0x7fe5bc0f23d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:52.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.883 7fe5c24ac700 1 --2- 172.21.2.112:0/2916152187 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe5bc0f1fd0 0x7fe5bc0f23d0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0x7fe5b000b530 tx=0x7fe5b000b8d0).stop 2021-02-10T11:30:52.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.883 7fe5c24ac700 1 -- 172.21.2.112:0/2916152187 shutdown_connections 2021-02-10T11:30:52.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.883 7fe5c24ac700 1 --2- 172.21.2.112:0/2916152187 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe59c0186c0 0x7fe59c01abc0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:52.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.883 7fe5c24ac700 1 --2- 172.21.2.112:0/2916152187 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe5bc0f1fd0 0x7fe5bc0f23d0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:52.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.883 7fe5c24ac700 1 --2- 172.21.2.112:0/2916152187 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe5bc0a5450 0x7fe5bc0a5850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:52.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.883 7fe5c24ac700 1 --2- 172.21.2.112:0/2916152187 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe5bc1a0570 0x7fe5bc0a4590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:52.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.883 7fe5c24ac700 1 -- 172.21.2.112:0/2916152187 >> 172.21.2.112:0/2916152187 conn(0x7fe5bc0958b0 msgr2=0x7fe5bc0bf040 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:52.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.883 7fe5c24ac700 1 -- 172.21.2.112:0/2916152187 shutdown_connections 2021-02-10T11:30:52.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:52.883 7fe5c24ac700 1 -- 172.21.2.112:0/2916152187 wait complete. 2021-02-10T11:30:52.880 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000348640 2021-02-10T11:30:53.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.318 7f5ba7d06700 1 Processor -- start 2021-02-10T11:30:53.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.318 7f5ba7d06700 1 -- start start 2021-02-10T11:30:53.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.319 7f5ba7d06700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f5ba01b0470 0x7f5ba01b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:53.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.319 7f5ba7d06700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ba00f0b50 con 0x7f5ba00f20e0 2021-02-10T11:30:53.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.319 7f5ba7d06700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ba00f24e0 con 0x7f5ba00ef610 2021-02-10T11:30:53.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.319 7f5ba7d06700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ba00e5ea0 con 0x7f5ba01b0470 2021-02-10T11:30:53.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.319 7f5ba6323700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f5ba00f20e0 0x7f5ba00f0200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:47010/0 (socket says 172.21.2.112:47010) 2021-02-10T11:30:53.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.319 7f5ba6323700 1 -- 172.21.2.112:0/2917630660 learned_addr learned my addr 172.21.2.112:0/2917630660 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:30:53.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.319 7f5ba5b22700 1 --2- 172.21.2.112:0/2917630660 >> v2:172.21.2.120:3300/0 conn(0x7f5ba01b0470 0x7f5ba01b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:53.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.319 7f5ba4b20700 1 -- 172.21.2.112:0/2917630660 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4082179655 0 0) 0x7f5b98000f90 con 0x7f5ba00f20e0 2021-02-10T11:30:53.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.319 7f5ba4b20700 1 -- 172.21.2.112:0/2917630660 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b8c002fe0 con 0x7f5ba00f20e0 2021-02-10T11:30:53.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.319 7f5ba4b20700 1 -- 172.21.2.112:0/2917630660 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 671622834 0 0) 0x7f5b90000ff0 con 0x7f5ba00ef610 2021-02-10T11:30:53.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.319 7f5ba4b20700 1 -- 172.21.2.112:0/2917630660 --> v1:172.21.2.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b8c004370 con 0x7f5ba00ef610 2021-02-10T11:30:53.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.319 7f5ba5b22700 1 -- 172.21.2.112:0/2917630660 >> v1:172.21.2.120:6789/0 conn(0x7f5ba00ef610 legacy=0x7f5ba00efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:53.326 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.319 7f5ba5b22700 1 -- 172.21.2.112:0/2917630660 >> v1:172.21.2.112:6789/0 conn(0x7f5ba00f20e0 legacy=0x7f5ba00f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:53.326 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.319 7f5ba5b22700 1 -- 172.21.2.112:0/2917630660 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ba00e52e0 con 0x7f5ba01b0470 2021-02-10T11:30:53.326 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.321 7f5ba5b22700 1 --2- 172.21.2.112:0/2917630660 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5ba01b0470 0x7f5ba01b0870 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0x7f5b9c006650 tx=0x7f5b9c009a00).ready entity=mon.1 client_cookie=17df3bf9a2389e68 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:53.326 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.321 7f5ba4b20700 1 -- 172.21.2.112:0/2917630660 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5b9c009df0 con 0x7f5ba01b0470 2021-02-10T11:30:53.326 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.321 7f5ba4b20700 1 -- 172.21.2.112:0/2917630660 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b9c0009b0 con 0x7f5ba01b0470 2021-02-10T11:30:53.326 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.321 7f5ba4b20700 1 -- 172.21.2.112:0/2917630660 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5b9c00c2b0 con 0x7f5ba01b0470 2021-02-10T11:30:53.326 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.321 7f5ba7d06700 1 -- 172.21.2.112:0/2917630660 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5ba01b0470 msgr2=0x7f5ba01b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:53.326 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.321 7f5ba7d06700 1 --2- 172.21.2.112:0/2917630660 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5ba01b0470 0x7f5ba01b0870 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0x7f5b9c006650 tx=0x7f5b9c009a00).stop 2021-02-10T11:30:53.326 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.321 7f5ba7d06700 1 -- 172.21.2.112:0/2917630660 shutdown_connections 2021-02-10T11:30:53.326 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.321 7f5ba7d06700 1 --2- 172.21.2.112:0/2917630660 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5ba01b0470 0x7f5ba01b0870 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:53.326 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.321 7f5ba7d06700 1 -- 172.21.2.112:0/2917630660 >> 172.21.2.112:0/2917630660 conn(0x7f5ba00ce3e0 msgr2=0x7f5ba00e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:53.326 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.321 7f5ba7d06700 1 -- 172.21.2.112:0/2917630660 shutdown_connections 2021-02-10T11:30:53.327 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.321 7f5ba7d06700 1 -- 172.21.2.112:0/2917630660 wait complete. 2021-02-10T11:30:53.327 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.335 7f5ba7d06700 1 Processor -- start 2021-02-10T11:30:53.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.335 7f5ba7d06700 1 -- start start 2021-02-10T11:30:53.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.336 7f5ba7d06700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5ba00a0420 0x7f5ba00a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:53.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.336 7f5ba7d06700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5ba0189d00 0x7f5ba018a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:53.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.336 7f5ba7d06700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5ba018a720 0x7f5ba00b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:53.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.336 7f5ba7d06700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ba0189780 con 0x7f5ba018a720 2021-02-10T11:30:53.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.336 7f5ba7d06700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ba018ab20 con 0x7f5ba00a0420 2021-02-10T11:30:53.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.336 7f5ba7d06700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ba00a1140 con 0x7f5ba0189d00 2021-02-10T11:30:53.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.336 7f5ba6323700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5ba018a720 0x7f5ba00b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:53.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.336 7f5ba6323700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5ba018a720 0x7f5ba00b7be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:47558/0 (socket says 172.21.2.112:47558) 2021-02-10T11:30:53.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.336 7f5ba6323700 1 -- 172.21.2.112:0/680608293 learned_addr learned my addr 172.21.2.112:0/680608293 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:30:53.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.336 7f5ba6323700 1 -- 172.21.2.112:0/680608293 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5ba0189d00 msgr2=0x7f5ba018a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:53.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.336 7f5ba6323700 1 --2- 172.21.2.112:0/680608293 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5ba0189d00 0x7f5ba018a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:53.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.336 7f5ba6323700 1 -- 172.21.2.112:0/680608293 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5ba00a0420 msgr2=0x7f5ba00a0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:53.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.336 7f5ba6323700 1 --2- 172.21.2.112:0/680608293 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5ba00a0420 0x7f5ba00a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:53.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.336 7f5ba6323700 1 -- 172.21.2.112:0/680608293 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ba00b8200 con 0x7f5ba018a720 2021-02-10T11:30:53.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.336 7f5ba6323700 1 --2- 172.21.2.112:0/680608293 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5ba018a720 0x7f5ba00b7be0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 rx=0x7f5b98009980 tx=0x7f5b9800bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:53.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.337 7f5b967fc700 1 -- 172.21.2.112:0/680608293 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5b980060c0 con 0x7f5ba018a720 2021-02-10T11:30:53.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.337 7f5b967fc700 1 -- 172.21.2.112:0/680608293 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b98006300 con 0x7f5ba018a720 2021-02-10T11:30:53.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.337 7f5b967fc700 1 -- 172.21.2.112:0/680608293 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5b98004720 con 0x7f5ba018a720 2021-02-10T11:30:53.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.337 7f5ba7d06700 1 -- 172.21.2.112:0/680608293 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ba00b8200 con 0x7f5ba018a720 2021-02-10T11:30:53.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.337 7f5ba7d06700 1 -- 172.21.2.112:0/680608293 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ba00b8200 con 0x7f5ba018a720 2021-02-10T11:30:53.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.337 7f5b967fc700 1 -- 172.21.2.112:0/680608293 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5b98000e40 con 0x7f5ba018a720 2021-02-10T11:30:53.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.338 7f5b967fc700 1 --2- 172.21.2.112:0/680608293 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5b880186c0 0x7f5b8801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:53.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.338 7f5b967fc700 1 -- 172.21.2.112:0/680608293 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f5b98032b70 con 0x7f5ba018a720 2021-02-10T11:30:53.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.338 7f5ba5b22700 1 --2- 172.21.2.112:0/680608293 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5b880186c0 0x7f5b8801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:53.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.339 7f5ba5b22700 1 --2- 172.21.2.112:0/680608293 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5b880186c0 0x7f5b8801abc0 crc :-1 s=READY pgs=461 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:53.357 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.368 7f5ba7d06700 1 -- 172.21.2.112:0/680608293 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ba0105ae0 con 0x7f5ba018a720 2021-02-10T11:30:53.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.371 7f5b967fc700 1 -- 172.21.2.112:0/680608293 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5b98006f10 con 0x7f5ba018a720 2021-02-10T11:30:53.488 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.498 7f5ba7d06700 1 -- 172.21.2.112:0/680608293 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348640", "target": ["mgr", ""]}) v1 -- 0x7f5ba0105ae0 con 0x7f5b880186c0 2021-02-10T11:30:53.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.511 7f5b967fc700 1 -- 172.21.2.112:0/680608293 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f5b9c006390 con 0x7f5b880186c0 2021-02-10T11:30:53.508 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.516 7f5ba7d06700 1 -- 172.21.2.112:0/680608293 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5b880186c0 msgr2=0x7f5b8801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:53.509 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.516 7f5ba7d06700 1 --2- 172.21.2.112:0/680608293 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5b880186c0 0x7f5b8801abc0 crc :-1 s=READY pgs=461 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:53.509 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.516 7f5ba7d06700 1 -- 172.21.2.112:0/680608293 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5ba018a720 msgr2=0x7f5ba00b7be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:53.509 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.516 7f5ba7d06700 1 --2- 172.21.2.112:0/680608293 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5ba018a720 0x7f5ba00b7be0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 rx=0x7f5b98009980 tx=0x7f5b9800bcd0).stop 2021-02-10T11:30:53.510 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.516 7f5ba7d06700 1 -- 172.21.2.112:0/680608293 shutdown_connections 2021-02-10T11:30:53.510 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.516 7f5ba7d06700 1 --2- 172.21.2.112:0/680608293 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5b880186c0 0x7f5b8801abc0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:53.510 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.516 7f5ba7d06700 1 --2- 172.21.2.112:0/680608293 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5ba018a720 0x7f5ba00b7be0 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:53.511 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.516 7f5ba7d06700 1 --2- 172.21.2.112:0/680608293 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5ba0189d00 0x7f5ba018a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:53.511 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.516 7f5ba7d06700 1 --2- 172.21.2.112:0/680608293 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5ba00a0420 0x7f5ba00a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:53.512 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.516 7f5ba7d06700 1 -- 172.21.2.112:0/680608293 >> 172.21.2.112:0/680608293 conn(0x7f5ba0094e10 msgr2=0x7f5ba00a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:53.512 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.516 7f5ba7d06700 1 -- 172.21.2.112:0/680608293 shutdown_connections 2021-02-10T11:30:53.512 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.516 7f5ba7d06700 1 -- 172.21.2.112:0/680608293 wait complete. 2021-02-10T11:30:53.516 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000348641 2021-02-10T11:30:53.948 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.944 7f2c567da700 1 Processor -- start 2021-02-10T11:30:53.949 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.944 7f2c567da700 1 -- start start 2021-02-10T11:30:53.949 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.944 7f2c567da700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f2c500f05f0 0x7f2c500f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:53.949 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.944 7f2c567da700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c501b0c70 con 0x7f2c501b0470 2021-02-10T11:30:53.949 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.944 7f2c567da700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c500f0b50 con 0x7f2c501b0870 2021-02-10T11:30:53.949 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.944 7f2c567da700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c500e5ea0 con 0x7f2c500f05f0 2021-02-10T11:30:53.949 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.944 7f2c54df7700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f2c500f05f0 0x7f2c500f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:53.949 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.944 7f2c4ffff700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f2c501b0470 0x7f2c500ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:47024/0 (socket says 172.21.2.112:47024) 2021-02-10T11:30:53.949 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.944 7f2c4ffff700 1 -- 172.21.2.112:0/3594036351 learned_addr learned my addr 172.21.2.112:0/3594036351 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:30:53.949 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.944 7f2c47fff700 1 --1- 172.21.2.112:0/3594036351 >> v1:172.21.2.120:6789/0 conn(0x7f2c501b0870 0x7f2c500efe00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:40582/0 (socket says 172.21.2.112:40582) 2021-02-10T11:30:53.949 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.945 7f2c54df7700 1 -- 172.21.2.112:0/3594036351 >> v1:172.21.2.120:6789/0 conn(0x7f2c501b0870 legacy=0x7f2c500efe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:53.949 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.945 7f2c54df7700 1 -- 172.21.2.112:0/3594036351 >> v1:172.21.2.112:6789/0 conn(0x7f2c501b0470 legacy=0x7f2c500ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:53.950 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.945 7f2c54df7700 1 -- 172.21.2.112:0/3594036351 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c500e52e0 con 0x7f2c500f05f0 2021-02-10T11:30:53.950 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.945 7f2c54df7700 1 --2- 172.21.2.112:0/3594036351 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2c500f05f0 0x7f2c500f20e0 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 rx=0x7f2c48006e30 tx=0x7f2c48009e60).ready entity=mon.0 client_cookie=35deddfe1e05dffa server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:53.950 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.945 7f2c4f7fe700 1 -- 172.21.2.112:0/3594036351 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2c4800b300 con 0x7f2c500f05f0 2021-02-10T11:30:53.950 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.945 7f2c4f7fe700 1 -- 172.21.2.112:0/3594036351 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c48000d30 con 0x7f2c500f05f0 2021-02-10T11:30:53.950 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.945 7f2c4f7fe700 1 -- 172.21.2.112:0/3594036351 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2c4800b7f0 con 0x7f2c500f05f0 2021-02-10T11:30:53.950 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.945 7f2c567da700 1 -- 172.21.2.112:0/3594036351 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2c500f05f0 msgr2=0x7f2c500f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:53.950 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.945 7f2c567da700 1 --2- 172.21.2.112:0/3594036351 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2c500f05f0 0x7f2c500f20e0 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 rx=0x7f2c48006e30 tx=0x7f2c48009e60).stop 2021-02-10T11:30:53.950 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.946 7f2c567da700 1 -- 172.21.2.112:0/3594036351 shutdown_connections 2021-02-10T11:30:53.950 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.946 7f2c567da700 1 --2- 172.21.2.112:0/3594036351 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2c500f05f0 0x7f2c500f20e0 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:53.950 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.946 7f2c567da700 1 -- 172.21.2.112:0/3594036351 >> 172.21.2.112:0/3594036351 conn(0x7f2c500ce3e0 msgr2=0x7f2c500e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:53.950 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.946 7f2c567da700 1 -- 172.21.2.112:0/3594036351 shutdown_connections 2021-02-10T11:30:53.950 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.946 7f2c567da700 1 -- 172.21.2.112:0/3594036351 wait complete. 2021-02-10T11:30:53.952 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.961 7f2c567da700 1 Processor -- start 2021-02-10T11:30:53.952 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.961 7f2c567da700 1 -- start start 2021-02-10T11:30:53.952 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.961 7f2c567da700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2c50087510 0x7f2c500aedc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:53.952 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.961 7f2c567da700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2c501c68f0 0x7f2c501b1f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:53.952 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.961 7f2c567da700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2c5008c4d0 0x7f2c501b18a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:53.952 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.961 7f2c567da700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c500a2d50 con 0x7f2c501c68f0 2021-02-10T11:30:53.953 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.961 7f2c567da700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c500a2f70 con 0x7f2c50087510 2021-02-10T11:30:53.953 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.961 7f2c567da700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c500a3190 con 0x7f2c5008c4d0 2021-02-10T11:30:53.953 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.961 7f2c47fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2c501c68f0 0x7f2c501b1f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:53.953 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.961 7f2c4ffff700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2c50087510 0x7f2c500aedc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:53.953 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.961 7f2c47fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2c501c68f0 0x7f2c501b1f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:47568/0 (socket says 172.21.2.112:47568) 2021-02-10T11:30:53.953 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.961 7f2c47fff700 1 -- 172.21.2.112:0/3674775229 learned_addr learned my addr 172.21.2.112:0/3674775229 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:30:53.953 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.961 7f2c54df7700 1 --2- 172.21.2.112:0/3674775229 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2c5008c4d0 0x7f2c501b18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:53.953 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.962 7f2c47fff700 1 -- 172.21.2.112:0/3674775229 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2c5008c4d0 msgr2=0x7f2c501b18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:53.953 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.962 7f2c47fff700 1 --2- 172.21.2.112:0/3674775229 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2c5008c4d0 0x7f2c501b18a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:53.953 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.962 7f2c47fff700 1 -- 172.21.2.112:0/3674775229 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2c50087510 msgr2=0x7f2c500aedc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:53.953 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.962 7f2c47fff700 1 --2- 172.21.2.112:0/3674775229 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2c50087510 0x7f2c500aedc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:53.953 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.962 7f2c47fff700 1 -- 172.21.2.112:0/3674775229 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c500a33b0 con 0x7f2c501c68f0 2021-02-10T11:30:53.954 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.962 7f2c47fff700 1 --2- 172.21.2.112:0/3674775229 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2c501c68f0 0x7f2c501b1f60 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 rx=0x7f2c4000a590 tx=0x7f2c4000a930).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:53.954 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.962 7f2c4d7fa700 1 -- 172.21.2.112:0/3674775229 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2c4000ad20 con 0x7f2c501c68f0 2021-02-10T11:30:53.954 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.962 7f2c4d7fa700 1 -- 172.21.2.112:0/3674775229 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c400070d0 con 0x7f2c501c68f0 2021-02-10T11:30:53.954 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.962 7f2c567da700 1 -- 172.21.2.112:0/3674775229 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c500a33b0 con 0x7f2c501c68f0 2021-02-10T11:30:53.954 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.962 7f2c4d7fa700 1 -- 172.21.2.112:0/3674775229 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2c40000e50 con 0x7f2c501c68f0 2021-02-10T11:30:53.954 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.962 7f2c567da700 1 -- 172.21.2.112:0/3674775229 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c500afca0 con 0x7f2c501c68f0 2021-02-10T11:30:53.954 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.963 7f2c4d7fa700 1 -- 172.21.2.112:0/3674775229 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2c40000c60 con 0x7f2c501c68f0 2021-02-10T11:30:53.954 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.963 7f2c4d7fa700 1 --2- 172.21.2.112:0/3674775229 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2c340186c0 0x7f2c3401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:53.954 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.963 7f2c4d7fa700 1 -- 172.21.2.112:0/3674775229 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f2c40032530 con 0x7f2c501c68f0 2021-02-10T11:30:53.954 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.965 7f2c4ffff700 1 --2- 172.21.2.112:0/3674775229 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2c340186c0 0x7f2c3401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:53.954 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.965 7f2c4ffff700 1 --2- 172.21.2.112:0/3674775229 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2c340186c0 0x7f2c3401abc0 crc :-1 s=READY pgs=462 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:53.982 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.994 7f2c567da700 1 -- 172.21.2.112:0/3674775229 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c5007e6b0 con 0x7f2c501c68f0 2021-02-10T11:30:53.985 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:53.997 7f2c4d7fa700 1 -- 172.21.2.112:0/3674775229 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2c40030ce0 con 0x7f2c501c68f0 2021-02-10T11:30:54.119 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:54.132 7f2c567da700 1 -- 172.21.2.112:0/3674775229 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348641", "target": ["mgr", ""]}) v1 -- 0x7f2c500afca0 con 0x7f2c340186c0 2021-02-10T11:30:54.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:54.141 7f2c4d7fa700 1 -- 172.21.2.112:0/3674775229 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f2c38007cc0 con 0x7f2c340186c0 2021-02-10T11:30:54.136 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:54.149 7f2c567da700 1 -- 172.21.2.112:0/3674775229 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2c340186c0 msgr2=0x7f2c3401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:54.136 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:54.149 7f2c567da700 1 --2- 172.21.2.112:0/3674775229 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2c340186c0 0x7f2c3401abc0 crc :-1 s=READY pgs=462 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:54.136 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:54.149 7f2c567da700 1 -- 172.21.2.112:0/3674775229 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2c501c68f0 msgr2=0x7f2c501b1f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:54.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:54.149 7f2c567da700 1 --2- 172.21.2.112:0/3674775229 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2c501c68f0 0x7f2c501b1f60 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 rx=0x7f2c4000a590 tx=0x7f2c4000a930).stop 2021-02-10T11:30:54.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:54.149 7f2c567da700 1 -- 172.21.2.112:0/3674775229 shutdown_connections 2021-02-10T11:30:54.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:54.149 7f2c567da700 1 --2- 172.21.2.112:0/3674775229 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2c340186c0 0x7f2c3401abc0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:54.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:54.149 7f2c567da700 1 --2- 172.21.2.112:0/3674775229 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2c5008c4d0 0x7f2c501b18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:54.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:54.149 7f2c567da700 1 --2- 172.21.2.112:0/3674775229 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2c501c68f0 0x7f2c501b1f60 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:54.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:54.149 7f2c567da700 1 --2- 172.21.2.112:0/3674775229 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2c50087510 0x7f2c500aedc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:54.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:54.149 7f2c567da700 1 -- 172.21.2.112:0/3674775229 >> 172.21.2.112:0/3674775229 conn(0x7f2c500886d0 msgr2=0x7f2c50081c40 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:54.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:54.149 7f2c567da700 1 -- 172.21.2.112:0/3674775229 shutdown_connections 2021-02-10T11:30:54.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:54.149 7f2c567da700 1 -- 172.21.2.112:0/3674775229 wait complete. 2021-02-10T11:30:54.145 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000348642 2021-02-10T11:30:54.576 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:54.571 7f3feaa1c700 1 Processor -- start 2021-02-10T11:30:54.576 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:54.571 7f3feaa1c700 1 -- start start 2021-02-10T11:30:54.576 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:54.572 7f3feaa1c700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3fe41b0870 0x7f3fe40f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:54.577 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:54.572 7f3feaa1c700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f3fe40efe00 0x7f3fe40f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:54.577 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:54.572 7f3feaa1c700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fe41b0c70 con 0x7f3fe41b0470 2021-02-10T11:30:54.577 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:54.572 7f3feaa1c700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fe40f0b50 con 0x7f3fe40efe00 2021-02-10T11:30:54.577 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:54.572 7f3feaa1c700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fe40f0820 con 0x7f3fe41b0870 2021-02-10T11:30:54.577 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:54.572 7f3fe9039700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f3fe40efe00 0x7f3fe40f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:54.577 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:54.572 7f3fe3fff700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3fe41b0870 0x7f3fe40f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:54.577 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:54.572 7f3fe8838700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f3fe41b0470 0x7f3fe40ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:47036/0 (socket says 172.21.2.112:47036) 2021-02-10T11:30:54.577 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:54.572 7f3fe8838700 1 -- 172.21.2.112:0/2196387266 learned_addr learned my addr 172.21.2.112:0/2196387266 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:30:54.577 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:54.573 7f3fe37fe700 1 -- 172.21.2.112:0/2196387266 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 411085224 0 0) 0x7f3fcc000ce0 con 0x7f3fe41b0470 2021-02-10T11:30:54.577 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:54.573 7f3fe37fe700 1 -- 172.21.2.112:0/2196387266 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3fd0002fe0 con 0x7f3fe41b0470 2021-02-10T11:30:54.577 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:54.573 7f3fe9039700 1 -- 172.21.2.112:0/2196387266 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3fe41b0870 msgr2=0x7f3fe40f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:54.577 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:54.573 7f3fe9039700 1 --2- 172.21.2.112:0/2196387266 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3fe41b0870 0x7f3fe40f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:54.577 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:54.573 7f3fe9039700 1 -- 172.21.2.112:0/2196387266 >> v1:172.21.2.112:6789/0 conn(0x7f3fe41b0470 legacy=0x7f3fe40ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:54.578 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:54.573 7f3fe9039700 1 -- 172.21.2.112:0/2196387266 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fe40e52e0 con 0x7f3fe40efe00 2021-02-10T11:30:54.578 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:54.573 7f3fe9039700 1 --2- 172.21.2.112:0/2196387266 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3fe40efe00 0x7f3fe40f0200 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 rx=0x7f3fdc006e30 tx=0x7f3fdc009e60).ready entity=mon.0 client_cookie=986c82a6963b5f85 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:54.578 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:54.573 7f3fe37fe700 1 -- 172.21.2.112:0/2196387266 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3fdc005820 con 0x7f3fe40efe00 2021-02-10T11:30:54.578 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:54.573 7f3fe37fe700 1 -- 172.21.2.112:0/2196387266 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fdc005a60 con 0x7f3fe40efe00 2021-02-10T11:30:54.578 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:54.573 7f3fe37fe700 1 -- 172.21.2.112:0/2196387266 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3fdc006030 con 0x7f3fe40efe00 2021-02-10T11:30:54.578 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:54.573 7f3feaa1c700 1 -- 172.21.2.112:0/2196387266 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3fe40efe00 msgr2=0x7f3fe40f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:54.578 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:54.573 7f3feaa1c700 1 --2- 172.21.2.112:0/2196387266 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3fe40efe00 0x7f3fe40f0200 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 rx=0x7f3fdc006e30 tx=0x7f3fdc009e60).stop 2021-02-10T11:30:54.578 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:54.574 7f3feaa1c700 1 -- 172.21.2.112:0/2196387266 shutdown_connections 2021-02-10T11:30:54.578 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:54.574 7f3feaa1c700 1 --2- 172.21.2.112:0/2196387266 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3fe40efe00 0x7f3fe40f0200 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:54.578 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:54.574 7f3feaa1c700 1 --2- 172.21.2.112:0/2196387266 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3fe41b0870 0x7f3fe40f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:54.578 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:54.574 7f3feaa1c700 1 -- 172.21.2.112:0/2196387266 >> 172.21.2.112:0/2196387266 conn(0x7f3fe40ce3e0 msgr2=0x7f3fe40e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:54.578 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:54.574 7f3feaa1c700 1 -- 172.21.2.112:0/2196387266 shutdown_connections 2021-02-10T11:30:54.578 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:54.574 7f3feaa1c700 1 -- 172.21.2.112:0/2196387266 wait complete. 2021-02-10T11:30:54.579 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:54.589 7f3feaa1c700 1 Processor -- start 2021-02-10T11:30:54.579 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:54.589 7f3feaa1c700 1 -- start start 2021-02-10T11:30:54.580 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:54.589 7f3feaa1c700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3fe4190920 0x7f3fe40b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:54.580 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:54.589 7f3feaa1c700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3fe40b8200 0x7f3fe40b8600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:54.580 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:54.589 7f3feaa1c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3fe40c1d80 0x7f3fe40c2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:54.580 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:54.589 7f3feaa1c700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fe4190da0 con 0x7f3fe40c1d80 2021-02-10T11:30:54.580 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:54.589 7f3feaa1c700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fe40b8c20 con 0x7f3fe4190920 2021-02-10T11:30:54.580 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:54.589 7f3feaa1c700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fe40b8e10 con 0x7f3fe40b8200 2021-02-10T11:30:54.580 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:54.589 7f3fe9039700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3fe40c1d80 0x7f3fe40c2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:54.580 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:54.589 7f3fe9039700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3fe40c1d80 0x7f3fe40c2180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:47580/0 (socket says 172.21.2.112:47580) 2021-02-10T11:30:54.580 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:54.590 7f3fe9039700 1 -- 172.21.2.112:0/3273156327 learned_addr learned my addr 172.21.2.112:0/3273156327 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:30:54.580 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:54.590 7f3fe9039700 1 -- 172.21.2.112:0/3273156327 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3fe40b8200 msgr2=0x7f3fe40b8600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:54.581 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:54.590 7f3fe9039700 1 --2- 172.21.2.112:0/3273156327 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3fe40b8200 0x7f3fe40b8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:54.581 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:54.590 7f3fe9039700 1 -- 172.21.2.112:0/3273156327 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3fe4190920 msgr2=0x7f3fe40b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:54.581 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:54.590 7f3fe9039700 1 --2- 172.21.2.112:0/3273156327 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3fe4190920 0x7f3fe40b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:54.581 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:54.590 7f3fe9039700 1 -- 172.21.2.112:0/3273156327 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fe40c27d0 con 0x7f3fe40c1d80 2021-02-10T11:30:54.581 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:54.590 7f3fe9039700 1 --2- 172.21.2.112:0/3273156327 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3fe40c1d80 0x7f3fe40c2180 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 rx=0x7f3fdc00b970 tx=0x7f3fdc00bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:54.581 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:54.590 7f3fe17fa700 1 -- 172.21.2.112:0/3273156327 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3fdc0093d0 con 0x7f3fe40c1d80 2021-02-10T11:30:54.581 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:54.590 7f3fe17fa700 1 -- 172.21.2.112:0/3273156327 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fdc005820 con 0x7f3fe40c1d80 2021-02-10T11:30:54.581 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:54.590 7f3fe17fa700 1 -- 172.21.2.112:0/3273156327 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3fdc005d40 con 0x7f3fe40c1d80 2021-02-10T11:30:54.581 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:54.590 7f3feaa1c700 1 -- 172.21.2.112:0/3273156327 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fe40c27d0 con 0x7f3fe40c1d80 2021-02-10T11:30:54.581 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:54.590 7f3feaa1c700 1 -- 172.21.2.112:0/3273156327 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fe40c2d60 con 0x7f3fe40c1d80 2021-02-10T11:30:54.581 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:54.591 7f3fe17fa700 1 -- 172.21.2.112:0/3273156327 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3fdc005a00 con 0x7f3fe40c1d80 2021-02-10T11:30:54.581 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:54.591 7f3fe17fa700 1 --2- 172.21.2.112:0/3273156327 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3fc80186c0 0x7f3fc801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:54.581 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:54.591 7f3fe17fa700 1 -- 172.21.2.112:0/3273156327 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f3fdc0328c0 con 0x7f3fe40c1d80 2021-02-10T11:30:54.581 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:54.591 7f3fe8838700 1 --2- 172.21.2.112:0/3273156327 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3fc80186c0 0x7f3fc801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:54.582 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:54.592 7f3fe8838700 1 --2- 172.21.2.112:0/3273156327 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3fc80186c0 0x7f3fc801abc0 crc :-1 s=READY pgs=463 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:54.609 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:54.621 7f3feaa1c700 1 -- 172.21.2.112:0/3273156327 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fe4105ae0 con 0x7f3fe40c1d80 2021-02-10T11:30:54.611 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:54.624 7f3fe17fa700 1 -- 172.21.2.112:0/3273156327 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3fdc0093d0 con 0x7f3fe40c1d80 2021-02-10T11:30:54.737 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:54.749 7f3feaa1c700 1 -- 172.21.2.112:0/3273156327 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348642", "target": ["mgr", ""]}) v1 -- 0x7f3fe40c2d60 con 0x7f3fc80186c0 2021-02-10T11:30:54.747 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:54.760 7f3fe17fa700 1 -- 172.21.2.112:0/3273156327 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f3fcc0043a0 con 0x7f3fc80186c0 2021-02-10T11:30:54.754 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:54.766 7f3feaa1c700 1 -- 172.21.2.112:0/3273156327 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3fc80186c0 msgr2=0x7f3fc801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:54.754 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:54.766 7f3feaa1c700 1 --2- 172.21.2.112:0/3273156327 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3fc80186c0 0x7f3fc801abc0 crc :-1 s=READY pgs=463 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:54.754 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:54.766 7f3feaa1c700 1 -- 172.21.2.112:0/3273156327 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3fe40c1d80 msgr2=0x7f3fe40c2180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:54.754 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:54.766 7f3feaa1c700 1 --2- 172.21.2.112:0/3273156327 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3fe40c1d80 0x7f3fe40c2180 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 rx=0x7f3fdc00b970 tx=0x7f3fdc00bd10).stop 2021-02-10T11:30:54.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:54.767 7f3feaa1c700 1 -- 172.21.2.112:0/3273156327 shutdown_connections 2021-02-10T11:30:54.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:54.767 7f3feaa1c700 1 --2- 172.21.2.112:0/3273156327 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3fc80186c0 0x7f3fc801abc0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:54.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:54.767 7f3feaa1c700 1 --2- 172.21.2.112:0/3273156327 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3fe40c1d80 0x7f3fe40c2180 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:54.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:54.767 7f3feaa1c700 1 --2- 172.21.2.112:0/3273156327 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3fe40b8200 0x7f3fe40b8600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:54.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:54.767 7f3feaa1c700 1 --2- 172.21.2.112:0/3273156327 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3fe4190920 0x7f3fe40b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:54.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:54.767 7f3feaa1c700 1 -- 172.21.2.112:0/3273156327 >> 172.21.2.112:0/3273156327 conn(0x7f3fe4094e10 msgr2=0x7f3fe40a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:54.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:54.767 7f3feaa1c700 1 -- 172.21.2.112:0/3273156327 shutdown_connections 2021-02-10T11:30:54.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:54.767 7f3feaa1c700 1 -- 172.21.2.112:0/3273156327 wait complete. 2021-02-10T11:30:54.763 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000348643 2021-02-10T11:30:55.195 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.190 7f69bbdba700 1 Processor -- start 2021-02-10T11:30:55.195 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.191 7f69bbdba700 1 -- start start 2021-02-10T11:30:55.195 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.191 7f69bbdba700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f69b41b0470 0x7f69b41b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:55.195 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.191 7f69bbdba700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f69b40ef610 0x7f69b40efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:55.195 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.191 7f69bbdba700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69b40f0b50 con 0x7f69b40f0030 2021-02-10T11:30:55.195 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.191 7f69bbdba700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69b40e52e0 con 0x7f69b41b0470 2021-02-10T11:30:55.195 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.191 7f69bbdba700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69b40e5490 con 0x7f69b40ef610 2021-02-10T11:30:55.196 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.191 7f69ba3d7700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7f69b40f0030 0x7f69b40f10d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:40608/0 (socket says 172.21.2.112:40608) 2021-02-10T11:30:55.196 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.191 7f69ba3d7700 1 -- 172.21.2.112:0/3459423363 learned_addr learned my addr 172.21.2.112:0/3459423363 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:30:55.196 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.191 7f69b9bd6700 1 --2- 172.21.2.112:0/3459423363 >> v2:172.21.2.112:3300/0 conn(0x7f69b41b0470 0x7f69b41b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:55.196 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.191 7f69b93d5700 1 --2- 172.21.2.112:0/3459423363 >> v2:172.21.2.120:3300/0 conn(0x7f69b40ef610 0x7f69b40efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:55.196 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.192 7f69b9bd6700 1 -- 172.21.2.112:0/3459423363 >> v2:172.21.2.120:3300/0 conn(0x7f69b40ef610 msgr2=0x7f69b40efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:55.196 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.192 7f69b9bd6700 1 --2- 172.21.2.112:0/3459423363 >> v2:172.21.2.120:3300/0 conn(0x7f69b40ef610 0x7f69b40efa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:55.196 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.192 7f69b9bd6700 1 -- 172.21.2.112:0/3459423363 >> v1:172.21.2.120:6789/0 conn(0x7f69b40f0030 legacy=0x7f69b40f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:55.196 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.192 7f69b9bd6700 1 -- 172.21.2.112:0/3459423363 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69b40f0430 con 0x7f69b41b0470 2021-02-10T11:30:55.196 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.192 7f69b9bd6700 1 --2- 172.21.2.112:0/3459423363 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f69b41b0470 0x7f69b41b0870 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 rx=0x7f69b0006650 tx=0x7f69b0009a00).ready entity=mon.0 client_cookie=c6e23d99c2eeb75d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:55.196 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.192 7f69b8bd4700 1 -- 172.21.2.112:0/3459423363 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f69b00009b0 con 0x7f69b41b0470 2021-02-10T11:30:55.196 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.192 7f69b8bd4700 1 -- 172.21.2.112:0/3459423363 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69b000c0c0 con 0x7f69b41b0470 2021-02-10T11:30:55.197 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.192 7f69b8bd4700 1 -- 172.21.2.112:0/3459423363 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f69b000c5a0 con 0x7f69b41b0470 2021-02-10T11:30:55.197 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.192 7f69bbdba700 1 -- 172.21.2.112:0/3459423363 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f69b41b0470 msgr2=0x7f69b41b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:55.197 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.192 7f69bbdba700 1 --2- 172.21.2.112:0/3459423363 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f69b41b0470 0x7f69b41b0870 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 rx=0x7f69b0006650 tx=0x7f69b0009a00).stop 2021-02-10T11:30:55.197 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.193 7f69bbdba700 1 -- 172.21.2.112:0/3459423363 shutdown_connections 2021-02-10T11:30:55.197 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.193 7f69bbdba700 1 --2- 172.21.2.112:0/3459423363 >> v2:172.21.2.120:3300/0 conn(0x7f69b40ef610 0x7f69b40efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:55.197 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.193 7f69bbdba700 1 --2- 172.21.2.112:0/3459423363 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f69b41b0470 0x7f69b41b0870 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:55.197 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.193 7f69bbdba700 1 -- 172.21.2.112:0/3459423363 >> 172.21.2.112:0/3459423363 conn(0x7f69b40ce3e0 msgr2=0x7f69b40e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:55.197 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.193 7f69bbdba700 1 -- 172.21.2.112:0/3459423363 shutdown_connections 2021-02-10T11:30:55.197 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.193 7f69bbdba700 1 -- 172.21.2.112:0/3459423363 wait complete. 2021-02-10T11:30:55.199 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.208 7f69bbdba700 1 Processor -- start 2021-02-10T11:30:55.199 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.208 7f69bbdba700 1 -- start start 2021-02-10T11:30:55.199 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.208 7f69bbdba700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f69b4190920 0x7f69b40b81a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:55.199 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.208 7f69bbdba700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f69b40b87c0 0x7f69b40b70c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:55.199 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.208 7f69bbdba700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f69b40b8bc0 0x7f69b40b76e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:55.199 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.208 7f69bbdba700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69b4190d20 con 0x7f69b40b87c0 2021-02-10T11:30:55.199 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.208 7f69bbdba700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69b40b5ad0 con 0x7f69b40b8bc0 2021-02-10T11:30:55.200 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.208 7f69bbdba700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69b40b7d00 con 0x7f69b4190920 2021-02-10T11:30:55.200 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.208 7f69ba3d7700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f69b40b8bc0 0x7f69b40b76e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:55.200 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.209 7f69ba3d7700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f69b40b8bc0 0x7f69b40b76e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3300/0 says I am v2:172.21.2.112:49872/0 (socket says 172.21.2.112:49872) 2021-02-10T11:30:55.200 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.209 7f69ba3d7700 1 -- 172.21.2.112:0/3262893389 learned_addr learned my addr 172.21.2.112:0/3262893389 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:30:55.200 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.209 7f69b93d5700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f69b40b87c0 0x7f69b40b70c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:55.200 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.209 7f69b9bd6700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f69b4190920 0x7f69b40b81a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:55.200 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.209 7f69ba3d7700 1 -- 172.21.2.112:0/3262893389 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f69b4190920 msgr2=0x7f69b40b81a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:55.200 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.209 7f69ba3d7700 1 --2- 172.21.2.112:0/3262893389 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f69b4190920 0x7f69b40b81a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:55.200 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.209 7f69ba3d7700 1 -- 172.21.2.112:0/3262893389 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f69b40b87c0 msgr2=0x7f69b40b70c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:55.200 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.209 7f69ba3d7700 1 --2- 172.21.2.112:0/3262893389 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f69b40b87c0 0x7f69b40b70c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:55.200 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.209 7f69ba3d7700 1 -- 172.21.2.112:0/3262893389 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69b40a4590 con 0x7f69b40b8bc0 2021-02-10T11:30:55.200 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.209 7f69b93d5700 1 --2- 172.21.2.112:0/3262893389 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f69b40b87c0 0x7f69b40b70c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:30:55.200 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.209 7f69ba3d7700 1 --2- 172.21.2.112:0/3262893389 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f69b40b8bc0 0x7f69b40b76e0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7f69ac00a900 tx=0x7f69ac00aca0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:55.201 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.209 7f69a67fc700 1 -- 172.21.2.112:0/3262893389 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f69ac0070d0 con 0x7f69b40b8bc0 2021-02-10T11:30:55.201 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.209 7f69a67fc700 1 -- 172.21.2.112:0/3262893389 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69ac000db0 con 0x7f69b40b8bc0 2021-02-10T11:30:55.201 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.209 7f69bbdba700 1 -- 172.21.2.112:0/3262893389 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69b40a4590 con 0x7f69b40b8bc0 2021-02-10T11:30:55.201 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.209 7f69a67fc700 1 -- 172.21.2.112:0/3262893389 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f69ac007390 con 0x7f69b40b8bc0 2021-02-10T11:30:55.201 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.210 7f69bbdba700 1 -- 172.21.2.112:0/3262893389 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69b40a4590 con 0x7f69b40b8bc0 2021-02-10T11:30:55.201 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.210 7f69a67fc700 1 -- 172.21.2.112:0/3262893389 <== mon.1 v2:172.21.2.120:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f69ac007390 con 0x7f69b40b8bc0 2021-02-10T11:30:55.201 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.210 7f69a67fc700 1 --2- 172.21.2.112:0/3262893389 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f699c018730 0x7f699c01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:55.201 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.210 7f69a67fc700 1 -- 172.21.2.112:0/3262893389 <== mon.1 v2:172.21.2.120:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f69ac0329d0 con 0x7f69b40b8bc0 2021-02-10T11:30:55.201 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.210 7f69b9bd6700 1 --2- 172.21.2.112:0/3262893389 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f699c018730 0x7f699c01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:55.201 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.211 7f69b9bd6700 1 --2- 172.21.2.112:0/3262893389 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f699c018730 0x7f699c01ac10 crc :-1 s=READY pgs=464 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:55.228 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.240 7f69bbdba700 1 -- 172.21.2.112:0/3262893389 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69b4105ae0 con 0x7f69b40b8bc0 2021-02-10T11:30:55.231 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.244 7f69a67fc700 1 -- 172.21.2.112:0/3262893389 <== mon.1 v2:172.21.2.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f69ac0070d0 con 0x7f69b40b8bc0 2021-02-10T11:30:55.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.374 7f69bbdba700 1 -- 172.21.2.112:0/3262893389 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348643", "target": ["mgr", ""]}) v1 -- 0x7f69b40f0b50 con 0x7f699c018730 2021-02-10T11:30:55.373 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.385 7f69a67fc700 1 -- 172.21.2.112:0/3262893389 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f69b0006450 con 0x7f699c018730 2021-02-10T11:30:55.379 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.391 7f69bbdba700 1 -- 172.21.2.112:0/3262893389 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f699c018730 msgr2=0x7f699c01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:55.379 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.391 7f69bbdba700 1 --2- 172.21.2.112:0/3262893389 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f699c018730 0x7f699c01ac10 crc :-1 s=READY pgs=464 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:55.379 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.391 7f69bbdba700 1 -- 172.21.2.112:0/3262893389 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f69b40b8bc0 msgr2=0x7f69b40b76e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:55.379 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.391 7f69bbdba700 1 --2- 172.21.2.112:0/3262893389 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f69b40b8bc0 0x7f69b40b76e0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7f69ac00a900 tx=0x7f69ac00aca0).stop 2021-02-10T11:30:55.379 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.391 7f69bbdba700 1 -- 172.21.2.112:0/3262893389 shutdown_connections 2021-02-10T11:30:55.379 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.391 7f69bbdba700 1 --2- 172.21.2.112:0/3262893389 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f699c018730 0x7f699c01ac10 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:55.379 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.391 7f69bbdba700 1 --2- 172.21.2.112:0/3262893389 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f69b40b8bc0 0x7f69b40b76e0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:55.379 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.391 7f69bbdba700 1 --2- 172.21.2.112:0/3262893389 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f69b40b87c0 0x7f69b40b70c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:55.379 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.391 7f69bbdba700 1 --2- 172.21.2.112:0/3262893389 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f69b4190920 0x7f69b40b81a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:55.380 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.391 7f69bbdba700 1 -- 172.21.2.112:0/3262893389 >> 172.21.2.112:0/3262893389 conn(0x7f69b408ffe0 msgr2=0x7f69b40bf050 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:55.380 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.392 7f69bbdba700 1 -- 172.21.2.112:0/3262893389 shutdown_connections 2021-02-10T11:30:55.380 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.392 7f69bbdba700 1 -- 172.21.2.112:0/3262893389 wait complete. 2021-02-10T11:30:55.387 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000348644 2021-02-10T11:30:55.816 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.812 7f2cf27b5700 1 Processor -- start 2021-02-10T11:30:55.816 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.812 7f2cf27b5700 1 -- start start 2021-02-10T11:30:55.816 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.813 7f2cf27b5700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2cec1b0470 0x7f2cec1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:55.816 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.813 7f2cf27b5700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f2cec0f20e0 0x7f2cec0f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:55.816 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.813 7f2cf27b5700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cec0f0b50 con 0x7f2cec0ef610 2021-02-10T11:30:55.816 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.813 7f2cf27b5700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cec0f0820 con 0x7f2cec0f20e0 2021-02-10T11:30:55.816 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.813 7f2cf27b5700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cec0f24e0 con 0x7f2cec1b0470 2021-02-10T11:30:55.816 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.813 7f2cf0dd2700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f2cec0f20e0 0x7f2cec0f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:55.816 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.813 7f2cf0dd2700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f2cec0f20e0 0x7f2cec0f0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:47598/0 (socket says 172.21.2.112:47598) 2021-02-10T11:30:55.816 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.813 7f2ceb7fe700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f2cec0ef610 0x7f2cec0efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:47060/0 (socket says 172.21.2.112:47060) 2021-02-10T11:30:55.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.813 7f2cf0dd2700 1 -- 172.21.2.112:0/2981810086 learned_addr learned my addr 172.21.2.112:0/2981810086 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:30:55.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.813 7f2cebfff700 1 --2- 172.21.2.112:0/2981810086 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2cec1b0470 0x7f2cec1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:55.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.813 7f2cf0dd2700 1 -- 172.21.2.112:0/2981810086 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2cec1b0470 msgr2=0x7f2cec1b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:55.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.813 7f2cf0dd2700 1 --2- 172.21.2.112:0/2981810086 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2cec1b0470 0x7f2cec1b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:55.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.813 7f2cf0dd2700 1 -- 172.21.2.112:0/2981810086 >> v1:172.21.2.112:6789/0 conn(0x7f2cec0ef610 legacy=0x7f2cec0efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:55.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.813 7f2ceaffd700 1 -- 172.21.2.112:0/2981810086 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 789427800 0 0) 0x7f2cdc0009b0 con 0x7f2cec0ef610 2021-02-10T11:30:55.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.813 7f2cf0dd2700 1 -- 172.21.2.112:0/2981810086 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cec0e52e0 con 0x7f2cec0f20e0 2021-02-10T11:30:55.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.813 7f2cebfff700 1 --2- 172.21.2.112:0/2981810086 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2cec1b0470 0x7f2cec1b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:30:55.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.813 7f2cf0dd2700 1 --2- 172.21.2.112:0/2981810086 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2cec0f20e0 0x7f2cec0f0200 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 rx=0x7f2ce4006f00 tx=0x7f2ce4009f10).ready entity=mon.0 client_cookie=9a584a6acb9cadcb server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:55.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.813 7f2ceaffd700 1 -- 172.21.2.112:0/2981810086 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2ce400b3b0 con 0x7f2cec0f20e0 2021-02-10T11:30:55.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.813 7f2ceaffd700 1 -- 172.21.2.112:0/2981810086 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ce400b6b0 con 0x7f2cec0f20e0 2021-02-10T11:30:55.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.813 7f2ceaffd700 1 -- 172.21.2.112:0/2981810086 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2ce400bb10 con 0x7f2cec0f20e0 2021-02-10T11:30:55.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.813 7f2cf27b5700 1 -- 172.21.2.112:0/2981810086 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2cec0f20e0 msgr2=0x7f2cec0f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:55.818 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.813 7f2cf27b5700 1 --2- 172.21.2.112:0/2981810086 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2cec0f20e0 0x7f2cec0f0200 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 rx=0x7f2ce4006f00 tx=0x7f2ce4009f10).stop 2021-02-10T11:30:55.818 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.814 7f2cf27b5700 1 -- 172.21.2.112:0/2981810086 shutdown_connections 2021-02-10T11:30:55.818 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.814 7f2cf27b5700 1 --2- 172.21.2.112:0/2981810086 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2cec0f20e0 0x7f2cec0f0200 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:55.818 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.814 7f2cf27b5700 1 --2- 172.21.2.112:0/2981810086 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2cec1b0470 0x7f2cec1b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:55.818 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.814 7f2cf27b5700 1 -- 172.21.2.112:0/2981810086 >> 172.21.2.112:0/2981810086 conn(0x7f2cec0ce3e0 msgr2=0x7f2cec0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:55.818 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.814 7f2cf27b5700 1 -- 172.21.2.112:0/2981810086 shutdown_connections 2021-02-10T11:30:55.818 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.814 7f2cf27b5700 1 -- 172.21.2.112:0/2981810086 wait complete. 2021-02-10T11:30:55.819 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.829 7f2cf27b5700 1 Processor -- start 2021-02-10T11:30:55.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.829 7f2cf27b5700 1 -- start start 2021-02-10T11:30:55.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.829 7f2cf27b5700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2cec09e560 0x7f2cec0a0130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:55.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.829 7f2cf27b5700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2cec09e960 0x7f2cec0a0750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:55.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.829 7f2cf27b5700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2cec0a0d70 0x7f2cec0a1af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:55.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.829 7f2cf27b5700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cec0c38f0 con 0x7f2cec09e560 2021-02-10T11:30:55.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.829 7f2cf27b5700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cec0a1170 con 0x7f2cec0a0d70 2021-02-10T11:30:55.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.829 7f2cf27b5700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cec0a1390 con 0x7f2cec09e960 2021-02-10T11:30:55.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.829 7f2cebfff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2cec09e560 0x7f2cec0a0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:55.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.829 7f2cebfff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2cec09e560 0x7f2cec0a0130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:47602/0 (socket says 172.21.2.112:47602) 2021-02-10T11:30:55.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.829 7f2cebfff700 1 -- 172.21.2.112:0/312508504 learned_addr learned my addr 172.21.2.112:0/312508504 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:30:55.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.829 7f2cf0dd2700 1 --2- 172.21.2.112:0/312508504 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2cec0a0d70 0x7f2cec0a1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:55.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.829 7f2ceb7fe700 1 --2- 172.21.2.112:0/312508504 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2cec09e960 0x7f2cec0a0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:55.821 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.830 7f2ceb7fe700 1 -- 172.21.2.112:0/312508504 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2cec0a0d70 msgr2=0x7f2cec0a1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:55.821 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.830 7f2ceb7fe700 1 --2- 172.21.2.112:0/312508504 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2cec0a0d70 0x7f2cec0a1af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:55.821 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.830 7f2ceb7fe700 1 -- 172.21.2.112:0/312508504 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2cec09e560 msgr2=0x7f2cec0a0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:55.821 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.830 7f2ceb7fe700 1 --2- 172.21.2.112:0/312508504 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2cec09e560 0x7f2cec0a0130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:55.821 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.830 7f2ceb7fe700 1 -- 172.21.2.112:0/312508504 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cec0a2110 con 0x7f2cec09e960 2021-02-10T11:30:55.821 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.830 7f2cf0dd2700 1 --2- 172.21.2.112:0/312508504 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2cec0a0d70 0x7f2cec0a1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:30:55.821 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.830 7f2cebfff700 1 --2- 172.21.2.112:0/312508504 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2cec09e560 0x7f2cec0a0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:30:55.821 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.830 7f2ceb7fe700 1 --2- 172.21.2.112:0/312508504 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2cec09e960 0x7f2cec0a0750 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f2cdc00a4c0 tx=0x7f2cdc00a860).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:55.821 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.830 7f2ce8ff9700 1 -- 172.21.2.112:0/312508504 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2cdc000e20 con 0x7f2cec09e960 2021-02-10T11:30:55.821 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.830 7f2ce8ff9700 1 -- 172.21.2.112:0/312508504 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cdc000940 con 0x7f2cec09e960 2021-02-10T11:30:55.821 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.830 7f2ce8ff9700 1 -- 172.21.2.112:0/312508504 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2cdc00ac50 con 0x7f2cec09e960 2021-02-10T11:30:55.821 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.830 7f2cf27b5700 1 -- 172.21.2.112:0/312508504 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cec0a2110 con 0x7f2cec09e960 2021-02-10T11:30:55.821 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.830 7f2cf27b5700 1 -- 172.21.2.112:0/312508504 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cec0a2110 con 0x7f2cec09e960 2021-02-10T11:30:55.822 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.831 7f2ce8ff9700 1 -- 172.21.2.112:0/312508504 <== mon.2 v2:172.21.2.120:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2cdc004140 con 0x7f2cec09e960 2021-02-10T11:30:55.822 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.831 7f2ce8ff9700 1 --2- 172.21.2.112:0/312508504 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2cd0018730 0x7f2cd001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:55.822 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.831 7f2ce8ff9700 1 -- 172.21.2.112:0/312508504 <== mon.2 v2:172.21.2.120:3301/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f2cdc0326c0 con 0x7f2cec09e960 2021-02-10T11:30:55.822 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.831 7f2cebfff700 1 --2- 172.21.2.112:0/312508504 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2cd0018730 0x7f2cd001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:55.822 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.832 7f2cebfff700 1 --2- 172.21.2.112:0/312508504 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2cd0018730 0x7f2cd001ac10 crc :-1 s=READY pgs=465 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:55.849 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.862 7f2cf27b5700 1 -- 172.21.2.112:0/312508504 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cec105b30 con 0x7f2cec09e960 2021-02-10T11:30:55.852 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.865 7f2ce8ff9700 1 -- 172.21.2.112:0/312508504 <== mon.2 v2:172.21.2.120:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2cdc000940 con 0x7f2cec09e960 2021-02-10T11:30:55.981 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:55.993 7f2cf27b5700 1 -- 172.21.2.112:0/312508504 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348644", "target": ["mgr", ""]}) v1 -- 0x7f2cec0f0b50 con 0x7f2cd0018730 2021-02-10T11:30:55.991 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:56.004 7f2ce8ff9700 1 -- 172.21.2.112:0/312508504 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f2cd40009b0 con 0x7f2cd0018730 2021-02-10T11:30:55.997 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:56.010 7f2cf27b5700 1 -- 172.21.2.112:0/312508504 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2cd0018730 msgr2=0x7f2cd001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:55.997 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:56.010 7f2cf27b5700 1 --2- 172.21.2.112:0/312508504 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2cd0018730 0x7f2cd001ac10 crc :-1 s=READY pgs=465 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:55.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:56.010 7f2cf27b5700 1 -- 172.21.2.112:0/312508504 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2cec09e960 msgr2=0x7f2cec0a0750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:55.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:56.010 7f2cf27b5700 1 --2- 172.21.2.112:0/312508504 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2cec09e960 0x7f2cec0a0750 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f2cdc00a4c0 tx=0x7f2cdc00a860).stop 2021-02-10T11:30:55.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:56.010 7f2cf27b5700 1 -- 172.21.2.112:0/312508504 shutdown_connections 2021-02-10T11:30:55.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:56.010 7f2cf27b5700 1 --2- 172.21.2.112:0/312508504 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2cd0018730 0x7f2cd001ac10 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:55.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:56.010 7f2cf27b5700 1 --2- 172.21.2.112:0/312508504 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2cec0a0d70 0x7f2cec0a1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:55.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:56.010 7f2cf27b5700 1 --2- 172.21.2.112:0/312508504 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2cec09e960 0x7f2cec0a0750 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:55.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:56.010 7f2cf27b5700 1 --2- 172.21.2.112:0/312508504 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2cec09e560 0x7f2cec0a0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:55.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:56.010 7f2cf27b5700 1 -- 172.21.2.112:0/312508504 >> 172.21.2.112:0/312508504 conn(0x7f2cec09fc10 msgr2=0x7f2cec094ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:55.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:56.011 7f2cf27b5700 1 -- 172.21.2.112:0/312508504 shutdown_connections 2021-02-10T11:30:55.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:56.011 7f2cf27b5700 1 -- 172.21.2.112:0/312508504 wait complete. 2021-02-10T11:30:56.007 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000348645 2021-02-10T11:30:56.437 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:56.432 7f2e2055b700 1 Processor -- start 2021-02-10T11:30:56.437 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:56.432 7f2e2055b700 1 -- start start 2021-02-10T11:30:56.437 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:56.433 7f2e2055b700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f2e181b0870 0x7f2e180f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:56.437 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:56.433 7f2e2055b700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e181b0c70 con 0x7f2e181b0470 2021-02-10T11:30:56.437 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:56.433 7f2e2055b700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e180f0b50 con 0x7f2e180efe00 2021-02-10T11:30:56.437 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:56.433 7f2e2055b700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e180e5ea0 con 0x7f2e181b0870 2021-02-10T11:30:56.437 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:56.433 7f2e1e377700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f2e181b0470 0x7f2e180ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:47072/0 (socket says 172.21.2.112:47072) 2021-02-10T11:30:56.437 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:56.433 7f2e1e377700 1 -- 172.21.2.112:0/586703245 learned_addr learned my addr 172.21.2.112:0/586703245 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:30:56.437 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:56.433 7f2e1db76700 1 --2- 172.21.2.112:0/586703245 >> v2:172.21.2.120:3300/0 conn(0x7f2e181b0870 0x7f2e180f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:56.437 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:56.433 7f2e1d375700 1 -- 172.21.2.112:0/586703245 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2188904632 0 0) 0x7f2e14000ce0 con 0x7f2e181b0470 2021-02-10T11:30:56.437 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:56.434 7f2e1d375700 1 -- 172.21.2.112:0/586703245 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e04002fe0 con 0x7f2e181b0470 2021-02-10T11:30:56.438 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:56.434 7f2e1d375700 1 -- 172.21.2.112:0/586703245 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2697217741 0 0) 0x7f2e10000f90 con 0x7f2e180efe00 2021-02-10T11:30:56.438 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:56.434 7f2e1d375700 1 -- 172.21.2.112:0/586703245 --> v1:172.21.2.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e04003210 con 0x7f2e180efe00 2021-02-10T11:30:56.438 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:56.434 7f2e1db76700 1 -- 172.21.2.112:0/586703245 >> v1:172.21.2.120:6789/0 conn(0x7f2e180efe00 legacy=0x7f2e180f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:56.438 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:56.434 7f2e1db76700 1 -- 172.21.2.112:0/586703245 >> v1:172.21.2.112:6789/0 conn(0x7f2e181b0470 legacy=0x7f2e180ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:56.438 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:56.434 7f2e1db76700 1 -- 172.21.2.112:0/586703245 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e180e52e0 con 0x7f2e181b0870 2021-02-10T11:30:56.438 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:56.434 7f2e1db76700 1 --2- 172.21.2.112:0/586703245 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2e181b0870 0x7f2e180f20e0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7f2e08006650 tx=0x7f2e08009a80).ready entity=mon.1 client_cookie=eb7ad6b86d663e83 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:56.438 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:56.434 7f2e1d375700 1 -- 172.21.2.112:0/586703245 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2e080009b0 con 0x7f2e181b0870 2021-02-10T11:30:56.438 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:56.434 7f2e1d375700 1 -- 172.21.2.112:0/586703245 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e0800c0c0 con 0x7f2e181b0870 2021-02-10T11:30:56.438 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:56.434 7f2e1d375700 1 -- 172.21.2.112:0/586703245 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2e0800c510 con 0x7f2e181b0870 2021-02-10T11:30:56.438 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:56.434 7f2e2055b700 1 -- 172.21.2.112:0/586703245 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2e181b0870 msgr2=0x7f2e180f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:56.438 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:56.434 7f2e2055b700 1 --2- 172.21.2.112:0/586703245 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2e181b0870 0x7f2e180f20e0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7f2e08006650 tx=0x7f2e08009a80).stop 2021-02-10T11:30:56.438 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:56.435 7f2e2055b700 1 -- 172.21.2.112:0/586703245 shutdown_connections 2021-02-10T11:30:56.438 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:56.435 7f2e2055b700 1 --2- 172.21.2.112:0/586703245 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2e181b0870 0x7f2e180f20e0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:56.439 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:56.435 7f2e2055b700 1 -- 172.21.2.112:0/586703245 >> 172.21.2.112:0/586703245 conn(0x7f2e180ce3e0 msgr2=0x7f2e180e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:56.439 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:56.435 7f2e2055b700 1 -- 172.21.2.112:0/586703245 shutdown_connections 2021-02-10T11:30:56.439 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:56.435 7f2e2055b700 1 -- 172.21.2.112:0/586703245 wait complete. 2021-02-10T11:30:56.440 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:56.450 7f2e2055b700 1 Processor -- start 2021-02-10T11:30:56.440 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:56.450 7f2e2055b700 1 -- start start 2021-02-10T11:30:56.440 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:56.450 7f2e2055b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2e180a0420 0x7f2e180a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:56.440 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:56.450 7f2e2055b700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2e18189d00 0x7f2e1818a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:56.441 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:56.450 7f2e2055b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2e1818a720 0x7f2e180b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:56.441 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:56.450 7f2e2055b700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e18189780 con 0x7f2e1818a720 2021-02-10T11:30:56.441 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:56.450 7f2e2055b700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e1818ab20 con 0x7f2e18189d00 2021-02-10T11:30:56.441 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:56.450 7f2e2055b700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e180a1140 con 0x7f2e180a0420 2021-02-10T11:30:56.441 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:56.450 7f2e1eb78700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2e1818a720 0x7f2e180b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:56.441 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:56.450 7f2e1e377700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2e180a0420 0x7f2e180a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:56.441 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:56.450 7f2e1eb78700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2e1818a720 0x7f2e180b7be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:47618/0 (socket says 172.21.2.112:47618) 2021-02-10T11:30:56.441 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:56.450 7f2e1eb78700 1 -- 172.21.2.112:0/4064374824 learned_addr learned my addr 172.21.2.112:0/4064374824 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:30:56.441 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:56.450 7f2e1db76700 1 --2- 172.21.2.112:0/4064374824 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2e18189d00 0x7f2e1818a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:56.441 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:56.451 7f2e1eb78700 1 -- 172.21.2.112:0/4064374824 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2e180a0420 msgr2=0x7f2e180a0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:56.441 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:56.451 7f2e1eb78700 1 --2- 172.21.2.112:0/4064374824 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2e180a0420 0x7f2e180a0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:56.441 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:56.451 7f2e1eb78700 1 -- 172.21.2.112:0/4064374824 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2e18189d00 msgr2=0x7f2e1818a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:56.441 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:56.451 7f2e1eb78700 1 --2- 172.21.2.112:0/4064374824 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2e18189d00 0x7f2e1818a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:56.442 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:56.451 7f2e1eb78700 1 -- 172.21.2.112:0/4064374824 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e180b8200 con 0x7f2e1818a720 2021-02-10T11:30:56.442 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:56.451 7f2e1e377700 1 --2- 172.21.2.112:0/4064374824 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2e180a0420 0x7f2e180a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:30:56.442 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:56.451 7f2e1eb78700 1 --2- 172.21.2.112:0/4064374824 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2e1818a720 0x7f2e180b7be0 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 rx=0x7f2e100099a0 tx=0x7f2e1000bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:56.442 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:56.451 7f2e0effd700 1 -- 172.21.2.112:0/4064374824 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2e100058a0 con 0x7f2e1818a720 2021-02-10T11:30:56.442 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:56.451 7f2e0effd700 1 -- 172.21.2.112:0/4064374824 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e10003210 con 0x7f2e1818a720 2021-02-10T11:30:56.442 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:56.451 7f2e2055b700 1 -- 172.21.2.112:0/4064374824 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e180b8200 con 0x7f2e1818a720 2021-02-10T11:30:56.442 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:56.451 7f2e2055b700 1 -- 172.21.2.112:0/4064374824 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e180b8790 con 0x7f2e1818a720 2021-02-10T11:30:56.442 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:56.451 7f2e0effd700 1 -- 172.21.2.112:0/4064374824 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2e10006030 con 0x7f2e1818a720 2021-02-10T11:30:56.442 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:56.452 7f2e0effd700 1 -- 172.21.2.112:0/4064374824 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2e10006030 con 0x7f2e1818a720 2021-02-10T11:30:56.442 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:56.452 7f2e0effd700 1 --2- 172.21.2.112:0/4064374824 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2e00018730 0x7f2e0001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:56.442 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:56.452 7f2e0effd700 1 -- 172.21.2.112:0/4064374824 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f2e10032b20 con 0x7f2e1818a720 2021-02-10T11:30:56.442 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:56.452 7f2e1e377700 1 --2- 172.21.2.112:0/4064374824 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2e00018730 0x7f2e0001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:56.442 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:56.453 7f2e1e377700 1 --2- 172.21.2.112:0/4064374824 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2e00018730 0x7f2e0001ac10 crc :-1 s=READY pgs=466 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:56.470 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:56.483 7f2e2055b700 1 -- 172.21.2.112:0/4064374824 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e18105ae0 con 0x7f2e1818a720 2021-02-10T11:30:56.473 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:56.485 7f2e0effd700 1 -- 172.21.2.112:0/4064374824 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2e10005d90 con 0x7f2e1818a720 2021-02-10T11:30:56.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:56.611 7f2e2055b700 1 -- 172.21.2.112:0/4064374824 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348645", "target": ["mgr", ""]}) v1 -- 0x7f2e180b9020 con 0x7f2e00018730 2021-02-10T11:30:56.611 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:56.623 7f2e0effd700 1 -- 172.21.2.112:0/4064374824 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f2e140094c0 con 0x7f2e00018730 2021-02-10T11:30:56.616 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:56.629 7f2e2055b700 1 -- 172.21.2.112:0/4064374824 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2e00018730 msgr2=0x7f2e0001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:56.616 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:56.629 7f2e2055b700 1 --2- 172.21.2.112:0/4064374824 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2e00018730 0x7f2e0001ac10 crc :-1 s=READY pgs=466 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:56.616 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:56.629 7f2e2055b700 1 -- 172.21.2.112:0/4064374824 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2e1818a720 msgr2=0x7f2e180b7be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:56.616 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:56.629 7f2e2055b700 1 --2- 172.21.2.112:0/4064374824 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2e1818a720 0x7f2e180b7be0 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 rx=0x7f2e100099a0 tx=0x7f2e1000bcf0).stop 2021-02-10T11:30:56.617 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:56.629 7f2e2055b700 1 -- 172.21.2.112:0/4064374824 shutdown_connections 2021-02-10T11:30:56.617 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:56.629 7f2e2055b700 1 --2- 172.21.2.112:0/4064374824 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2e00018730 0x7f2e0001ac10 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:56.617 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:56.629 7f2e2055b700 1 --2- 172.21.2.112:0/4064374824 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2e18189d00 0x7f2e1818a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:56.617 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:56.629 7f2e2055b700 1 --2- 172.21.2.112:0/4064374824 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2e1818a720 0x7f2e180b7be0 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:56.617 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:56.629 7f2e2055b700 1 --2- 172.21.2.112:0/4064374824 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2e180a0420 0x7f2e180a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:56.617 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:56.629 7f2e2055b700 1 -- 172.21.2.112:0/4064374824 >> 172.21.2.112:0/4064374824 conn(0x7f2e18094e10 msgr2=0x7f2e180a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:56.617 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:56.629 7f2e2055b700 1 -- 172.21.2.112:0/4064374824 shutdown_connections 2021-02-10T11:30:56.617 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:56.629 7f2e2055b700 1 -- 172.21.2.112:0/4064374824 wait complete. 2021-02-10T11:30:56.625 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000348646 2021-02-10T11:30:57.056 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.052 7faac284a700 1 Processor -- start 2021-02-10T11:30:57.056 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.052 7faac284a700 1 -- start start 2021-02-10T11:30:57.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.053 7faac284a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7faabc1b0470 0x7faabc1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:57.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.053 7faac284a700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7faabc0f20e0 0x7faabc0f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:57.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.053 7faac284a700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faabc0f0b50 con 0x7faabc0ef610 2021-02-10T11:30:57.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.053 7faac284a700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faabc0f0820 con 0x7faabc0f20e0 2021-02-10T11:30:57.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.053 7faac284a700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faabc0f24e0 con 0x7faabc1b0470 2021-02-10T11:30:57.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.053 7faabb7fe700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7faabc0ef610 0x7faabc0efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:40644/0 (socket says 172.21.2.112:40644) 2021-02-10T11:30:57.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.053 7faabb7fe700 1 -- 172.21.2.112:0/3558650894 learned_addr learned my addr 172.21.2.112:0/3558650894 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:30:57.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.053 7faabbfff700 1 --2- 172.21.2.112:0/3558650894 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7faabc1b0470 0x7faabc1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:57.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.053 7faac0e67700 1 --2- 172.21.2.112:0/3558650894 >> v2:172.21.2.120:3300/0 conn(0x7faabc0f20e0 0x7faabc0f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:57.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.053 7faabaffd700 1 -- 172.21.2.112:0/3558650894 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1601755516 0 0) 0x7faaa40009b0 con 0x7faabc0ef610 2021-02-10T11:30:57.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.053 7faabaffd700 1 -- 172.21.2.112:0/3558650894 --> v1:172.21.2.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faaa8002fe0 con 0x7faabc0ef610 2021-02-10T11:30:57.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.053 7faabbfff700 1 -- 172.21.2.112:0/3558650894 >> v2:172.21.2.120:3300/0 conn(0x7faabc0f20e0 msgr2=0x7faabc0f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:57.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.053 7faabbfff700 1 --2- 172.21.2.112:0/3558650894 >> v2:172.21.2.120:3300/0 conn(0x7faabc0f20e0 0x7faabc0f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:57.058 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.054 7faabbfff700 1 -- 172.21.2.112:0/3558650894 >> v1:172.21.2.120:6789/0 conn(0x7faabc0ef610 legacy=0x7faabc0efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:57.058 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.054 7faabbfff700 1 -- 172.21.2.112:0/3558650894 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faabc0e52e0 con 0x7faabc1b0470 2021-02-10T11:30:57.058 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.054 7faac0e67700 1 --2- 172.21.2.112:0/3558650894 >> v2:172.21.2.120:3300/0 conn(0x7faabc0f20e0 0x7faabc0f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:30:57.058 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.054 7faabbfff700 1 --2- 172.21.2.112:0/3558650894 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7faabc1b0470 0x7faabc1b0870 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7faaac0069d0 tx=0x7faaac009e40).ready entity=mon.2 client_cookie=6873fdc18c6311d2 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:57.058 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.054 7faabaffd700 1 -- 172.21.2.112:0/3558650894 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faaac0066c0 con 0x7faabc1b0470 2021-02-10T11:30:57.058 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.054 7faabaffd700 1 -- 172.21.2.112:0/3558650894 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faaac00b510 con 0x7faabc1b0470 2021-02-10T11:30:57.058 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.054 7faabaffd700 1 -- 172.21.2.112:0/3558650894 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faaac00b970 con 0x7faabc1b0470 2021-02-10T11:30:57.058 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.054 7faac284a700 1 -- 172.21.2.112:0/3558650894 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7faabc1b0470 msgr2=0x7faabc1b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:57.058 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.054 7faac284a700 1 --2- 172.21.2.112:0/3558650894 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7faabc1b0470 0x7faabc1b0870 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7faaac0069d0 tx=0x7faaac009e40).stop 2021-02-10T11:30:57.058 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.055 7faac284a700 1 -- 172.21.2.112:0/3558650894 shutdown_connections 2021-02-10T11:30:57.058 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.055 7faac284a700 1 --2- 172.21.2.112:0/3558650894 >> v2:172.21.2.120:3300/0 conn(0x7faabc0f20e0 0x7faabc0f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:57.058 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.055 7faac284a700 1 --2- 172.21.2.112:0/3558650894 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7faabc1b0470 0x7faabc1b0870 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:57.059 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.055 7faac284a700 1 -- 172.21.2.112:0/3558650894 >> 172.21.2.112:0/3558650894 conn(0x7faabc0ce3e0 msgr2=0x7faabc0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:57.059 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.055 7faac284a700 1 -- 172.21.2.112:0/3558650894 shutdown_connections 2021-02-10T11:30:57.059 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.055 7faac284a700 1 -- 172.21.2.112:0/3558650894 wait complete. 2021-02-10T11:30:57.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.069 7faac284a700 1 Processor -- start 2021-02-10T11:30:57.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.070 7faac284a700 1 -- start start 2021-02-10T11:30:57.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.070 7faac284a700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7faabc0b7c20 0x7faabc0b8410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:57.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.070 7faac284a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faabc0b8a30 0x7faabc0c1f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:57.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.070 7faac284a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7faabc0c25a0 0x7faabc0c29a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:57.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.070 7faac284a700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faabc0c1a80 con 0x7faabc0b8a30 2021-02-10T11:30:57.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.070 7faac284a700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faabc0b8020 con 0x7faabc0b7c20 2021-02-10T11:30:57.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.070 7faac284a700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faabc09b9c0 con 0x7faabc0c25a0 2021-02-10T11:30:57.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.070 7faac0e67700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7faabc0c25a0 0x7faabc0c29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:57.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.070 7faabb7fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faabc0b8a30 0x7faabc0c1f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:57.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.070 7faac0e67700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7faabc0c25a0 0x7faabc0c29a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3301/0 says I am v2:172.21.2.112:47310/0 (socket says 172.21.2.112:47310) 2021-02-10T11:30:57.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.070 7faac0e67700 1 -- 172.21.2.112:0/3783948935 learned_addr learned my addr 172.21.2.112:0/3783948935 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:30:57.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.070 7faabbfff700 1 --2- 172.21.2.112:0/3783948935 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7faabc0b7c20 0x7faabc0b8410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:57.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.071 7faac0e67700 1 -- 172.21.2.112:0/3783948935 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7faabc0b7c20 msgr2=0x7faabc0b8410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:57.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.071 7faac0e67700 1 --2- 172.21.2.112:0/3783948935 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7faabc0b7c20 0x7faabc0b8410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:57.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.071 7faac0e67700 1 -- 172.21.2.112:0/3783948935 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faabc0b8a30 msgr2=0x7faabc0c1f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:57.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.071 7faac0e67700 1 --2- 172.21.2.112:0/3783948935 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faabc0b8a30 0x7faabc0c1f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:57.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.071 7faac0e67700 1 -- 172.21.2.112:0/3783948935 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faabc0b8e30 con 0x7faabc0c25a0 2021-02-10T11:30:57.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.071 7faac0e67700 1 --2- 172.21.2.112:0/3783948935 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7faabc0c25a0 0x7faabc0c29a0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7faab400b580 tx=0x7faab400b920).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:57.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.071 7faab8ff9700 1 -- 172.21.2.112:0/3783948935 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faab400bd10 con 0x7faabc0c25a0 2021-02-10T11:30:57.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.071 7faab8ff9700 1 -- 172.21.2.112:0/3783948935 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faab4006720 con 0x7faabc0c25a0 2021-02-10T11:30:57.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.071 7faac284a700 1 -- 172.21.2.112:0/3783948935 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faabc0b8e30 con 0x7faabc0c25a0 2021-02-10T11:30:57.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.071 7faab8ff9700 1 -- 172.21.2.112:0/3783948935 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faab40024d0 con 0x7faabc0c25a0 2021-02-10T11:30:57.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.071 7faac284a700 1 -- 172.21.2.112:0/3783948935 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faabc0b8e30 con 0x7faabc0c25a0 2021-02-10T11:30:57.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.072 7faab8ff9700 1 -- 172.21.2.112:0/3783948935 <== mon.2 v2:172.21.2.120:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7faab4000af0 con 0x7faabc0c25a0 2021-02-10T11:30:57.063 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.072 7faab8ff9700 1 --2- 172.21.2.112:0/3783948935 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7faaa00186c0 0x7faaa001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:57.063 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.072 7faab8ff9700 1 -- 172.21.2.112:0/3783948935 <== mon.2 v2:172.21.2.120:3301/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7faab40327e0 con 0x7faabc0c25a0 2021-02-10T11:30:57.063 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.072 7faabbfff700 1 --2- 172.21.2.112:0/3783948935 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7faaa00186c0 0x7faaa001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:57.063 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.072 7faabbfff700 1 --2- 172.21.2.112:0/3783948935 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7faaa00186c0 0x7faaa001abc0 crc :-1 s=READY pgs=467 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:57.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.102 7faac284a700 1 -- 172.21.2.112:0/3783948935 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faabc105b30 con 0x7faabc0c25a0 2021-02-10T11:30:57.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.106 7faab8ff9700 1 -- 172.21.2.112:0/3783948935 <== mon.2 v2:172.21.2.120:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7faab4006370 con 0x7faabc0c25a0 2021-02-10T11:30:57.223 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.236 7faac284a700 1 -- 172.21.2.112:0/3783948935 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348646", "target": ["mgr", ""]}) v1 -- 0x7faabc0f0b50 con 0x7faaa00186c0 2021-02-10T11:30:57.234 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.246 7faab8ff9700 1 -- 172.21.2.112:0/3783948935 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7faaac000d30 con 0x7faaa00186c0 2021-02-10T11:30:57.241 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.253 7faac284a700 1 -- 172.21.2.112:0/3783948935 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7faaa00186c0 msgr2=0x7faaa001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:57.241 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.253 7faac284a700 1 --2- 172.21.2.112:0/3783948935 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7faaa00186c0 0x7faaa001abc0 crc :-1 s=READY pgs=467 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:57.241 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.253 7faac284a700 1 -- 172.21.2.112:0/3783948935 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7faabc0c25a0 msgr2=0x7faabc0c29a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:57.241 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.253 7faac284a700 1 --2- 172.21.2.112:0/3783948935 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7faabc0c25a0 0x7faabc0c29a0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7faab400b580 tx=0x7faab400b920).stop 2021-02-10T11:30:57.241 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.253 7faac284a700 1 -- 172.21.2.112:0/3783948935 shutdown_connections 2021-02-10T11:30:57.241 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.253 7faac284a700 1 --2- 172.21.2.112:0/3783948935 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7faaa00186c0 0x7faaa001abc0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:57.241 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.253 7faac284a700 1 --2- 172.21.2.112:0/3783948935 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7faabc0c25a0 0x7faabc0c29a0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:57.242 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.253 7faac284a700 1 --2- 172.21.2.112:0/3783948935 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faabc0b8a30 0x7faabc0c1f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:57.242 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.253 7faac284a700 1 --2- 172.21.2.112:0/3783948935 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7faabc0b7c20 0x7faabc0b8410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:57.242 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.253 7faac284a700 1 -- 172.21.2.112:0/3783948935 >> 172.21.2.112:0/3783948935 conn(0x7faabc094e10 msgr2=0x7faabc0a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:57.242 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.253 7faac284a700 1 -- 172.21.2.112:0/3783948935 shutdown_connections 2021-02-10T11:30:57.242 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.253 7faac284a700 1 -- 172.21.2.112:0/3783948935 wait complete. 2021-02-10T11:30:57.249 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000348647 2021-02-10T11:30:57.678 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.674 7effaf314700 1 Processor -- start 2021-02-10T11:30:57.678 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.674 7effaf314700 1 -- start start 2021-02-10T11:30:57.678 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.674 7effaf314700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7effa80f05f0 0x7effa80f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:57.678 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.674 7effaf314700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effa81b0c70 con 0x7effa81b0470 2021-02-10T11:30:57.678 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.674 7effaf314700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effa80f0b50 con 0x7effa81b0870 2021-02-10T11:30:57.678 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.674 7effaf314700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7effa80e5ea0 con 0x7effa80f05f0 2021-02-10T11:30:57.679 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.674 7effad130700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7effa81b0470 0x7effa80ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:47098/0 (socket says 172.21.2.112:47098) 2021-02-10T11:30:57.679 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.674 7effad130700 1 -- 172.21.2.112:0/125568503 learned_addr learned my addr 172.21.2.112:0/125568503 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:30:57.679 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.674 7effad931700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7effa80f05f0 0x7effa80f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:57.679 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.675 7eff9bfff700 1 -- 172.21.2.112:0/125568503 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1266058059 0 0) 0x7effa4000ff0 con 0x7effa81b0870 2021-02-10T11:30:57.679 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.675 7eff9bfff700 1 -- 172.21.2.112:0/125568503 --> v1:172.21.2.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff94002fe0 con 0x7effa81b0870 2021-02-10T11:30:57.679 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.675 7eff9bfff700 1 -- 172.21.2.112:0/125568503 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 273747128 0 0) 0x7eff9c0009b0 con 0x7effa81b0470 2021-02-10T11:30:57.679 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.675 7effad931700 1 -- 172.21.2.112:0/125568503 >> v1:172.21.2.120:6789/0 conn(0x7effa81b0870 legacy=0x7effa80efe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:57.679 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.675 7effad931700 1 -- 172.21.2.112:0/125568503 >> v1:172.21.2.112:6789/0 conn(0x7effa81b0470 legacy=0x7effa80ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:57.679 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.675 7effad931700 1 -- 172.21.2.112:0/125568503 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effa80e52e0 con 0x7effa80f05f0 2021-02-10T11:30:57.679 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.675 7effad931700 1 --2- 172.21.2.112:0/125568503 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7effa80f05f0 0x7effa80f20e0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 rx=0x7effa0006e30 tx=0x7effa0009e60).ready entity=mon.0 client_cookie=19b0e3e6f1c171c7 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:57.679 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.675 7eff9bfff700 1 -- 172.21.2.112:0/125568503 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7effa0006a50 con 0x7effa80f05f0 2021-02-10T11:30:57.679 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.675 7eff9bfff700 1 -- 172.21.2.112:0/125568503 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effa000b300 con 0x7effa80f05f0 2021-02-10T11:30:57.679 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.675 7eff9bfff700 1 -- 172.21.2.112:0/125568503 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7effa000b820 con 0x7effa80f05f0 2021-02-10T11:30:57.680 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.675 7effaf314700 1 -- 172.21.2.112:0/125568503 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7effa80f05f0 msgr2=0x7effa80f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:57.680 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.675 7effaf314700 1 --2- 172.21.2.112:0/125568503 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7effa80f05f0 0x7effa80f20e0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 rx=0x7effa0006e30 tx=0x7effa0009e60).stop 2021-02-10T11:30:57.680 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.676 7effaf314700 1 -- 172.21.2.112:0/125568503 shutdown_connections 2021-02-10T11:30:57.680 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.676 7effaf314700 1 --2- 172.21.2.112:0/125568503 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7effa80f05f0 0x7effa80f20e0 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:57.680 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.676 7effaf314700 1 -- 172.21.2.112:0/125568503 >> 172.21.2.112:0/125568503 conn(0x7effa80ce3e0 msgr2=0x7effa80e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:57.680 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.676 7effaf314700 1 -- 172.21.2.112:0/125568503 shutdown_connections 2021-02-10T11:30:57.680 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.676 7effaf314700 1 -- 172.21.2.112:0/125568503 wait complete. 2021-02-10T11:30:57.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.691 7effaf314700 1 Processor -- start 2021-02-10T11:30:57.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.691 7effaf314700 1 -- start start 2021-02-10T11:30:57.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.691 7effaf314700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7effa8193230 0x7effa80a6a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:57.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.691 7effaf314700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7effa808f5b0 0x7effa808f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:57.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.691 7effaf314700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7effa808e6a0 0x7effa808eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:57.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.691 7effaf314700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effa808ffd0 con 0x7effa8193230 2021-02-10T11:30:57.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.691 7effaf314700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effa8090240 con 0x7effa808f5b0 2021-02-10T11:30:57.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.691 7effaf314700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effa808f0c0 con 0x7effa808e6a0 2021-02-10T11:30:57.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.692 7effad130700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7effa8193230 0x7effa80a6a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:57.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.692 7effad130700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7effa8193230 0x7effa80a6a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:47642/0 (socket says 172.21.2.112:47642) 2021-02-10T11:30:57.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.692 7effad130700 1 -- 172.21.2.112:0/203769256 learned_addr learned my addr 172.21.2.112:0/203769256 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:30:57.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.692 7effac92f700 1 --2- 172.21.2.112:0/203769256 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7effa808f5b0 0x7effa808f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:57.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.692 7effad931700 1 --2- 172.21.2.112:0/203769256 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7effa808e6a0 0x7effa808eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:57.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.692 7effad130700 1 -- 172.21.2.112:0/203769256 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7effa808e6a0 msgr2=0x7effa808eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:57.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.692 7effad130700 1 --2- 172.21.2.112:0/203769256 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7effa808e6a0 0x7effa808eaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:57.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.692 7effad130700 1 -- 172.21.2.112:0/203769256 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7effa808f5b0 msgr2=0x7effa808f9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:57.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.692 7effad130700 1 --2- 172.21.2.112:0/203769256 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7effa808f5b0 0x7effa808f9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:57.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.692 7effad130700 1 -- 172.21.2.112:0/203769256 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effa808f2a0 con 0x7effa8193230 2021-02-10T11:30:57.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.692 7effad130700 1 --2- 172.21.2.112:0/203769256 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7effa8193230 0x7effa80a6a80 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 rx=0x7eff9c001c60 tx=0x7eff9c00a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:57.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.692 7eff99ffb700 1 -- 172.21.2.112:0/203769256 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7eff9c00acc0 con 0x7effa8193230 2021-02-10T11:30:57.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.692 7eff99ffb700 1 -- 172.21.2.112:0/203769256 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff9c004140 con 0x7effa8193230 2021-02-10T11:30:57.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.692 7effaf314700 1 -- 172.21.2.112:0/203769256 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effa808f2a0 con 0x7effa8193230 2021-02-10T11:30:57.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.692 7effaf314700 1 -- 172.21.2.112:0/203769256 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effa81924f0 con 0x7effa8193230 2021-02-10T11:30:57.684 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.692 7eff99ffb700 1 -- 172.21.2.112:0/203769256 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7eff9c004660 con 0x7effa8193230 2021-02-10T11:30:57.684 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.693 7eff99ffb700 1 -- 172.21.2.112:0/203769256 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7eff9c009030 con 0x7effa8193230 2021-02-10T11:30:57.684 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.693 7eff99ffb700 1 --2- 172.21.2.112:0/203769256 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7eff880186c0 0x7eff8801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:57.684 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.693 7eff99ffb700 1 -- 172.21.2.112:0/203769256 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7eff9c0323e0 con 0x7effa8193230 2021-02-10T11:30:57.684 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.693 7effac92f700 1 --2- 172.21.2.112:0/203769256 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7eff880186c0 0x7eff8801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:57.684 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.694 7effac92f700 1 --2- 172.21.2.112:0/203769256 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7eff880186c0 0x7eff8801abc0 crc :-1 s=READY pgs=468 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:57.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.724 7effaf314700 1 -- 172.21.2.112:0/203769256 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effa80f0b50 con 0x7effa8193230 2021-02-10T11:30:57.714 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.727 7eff99ffb700 1 -- 172.21.2.112:0/203769256 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7eff9c030d10 con 0x7effa8193230 2021-02-10T11:30:57.841 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.854 7effaf314700 1 -- 172.21.2.112:0/203769256 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348647", "target": ["mgr", ""]}) v1 -- 0x7effa808f2a0 con 0x7eff880186c0 2021-02-10T11:30:57.852 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.864 7eff99ffb700 1 -- 172.21.2.112:0/203769256 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7effa4006550 con 0x7eff880186c0 2021-02-10T11:30:57.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.871 7effaf314700 1 -- 172.21.2.112:0/203769256 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7eff880186c0 msgr2=0x7eff8801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:57.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.871 7effaf314700 1 --2- 172.21.2.112:0/203769256 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7eff880186c0 0x7eff8801abc0 crc :-1 s=READY pgs=468 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:57.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.871 7effaf314700 1 -- 172.21.2.112:0/203769256 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7effa8193230 msgr2=0x7effa80a6a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:57.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.871 7effaf314700 1 --2- 172.21.2.112:0/203769256 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7effa8193230 0x7effa80a6a80 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 rx=0x7eff9c001c60 tx=0x7eff9c00a8d0).stop 2021-02-10T11:30:57.860 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.872 7effaf314700 1 -- 172.21.2.112:0/203769256 shutdown_connections 2021-02-10T11:30:57.860 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.872 7effaf314700 1 --2- 172.21.2.112:0/203769256 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7eff880186c0 0x7eff8801abc0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:57.860 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.872 7effaf314700 1 --2- 172.21.2.112:0/203769256 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7effa808f5b0 0x7effa808f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:57.860 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.872 7effaf314700 1 --2- 172.21.2.112:0/203769256 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7effa808e6a0 0x7effa808eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:57.860 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.872 7effaf314700 1 --2- 172.21.2.112:0/203769256 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7effa8193230 0x7effa80a6a80 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:57.860 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.872 7effaf314700 1 -- 172.21.2.112:0/203769256 >> 172.21.2.112:0/203769256 conn(0x7effa80bf4e0 msgr2=0x7effa80be160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:57.860 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.872 7effaf314700 1 -- 172.21.2.112:0/203769256 shutdown_connections 2021-02-10T11:30:57.860 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:57.872 7effaf314700 1 -- 172.21.2.112:0/203769256 wait complete. 2021-02-10T11:30:57.868 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000348648 2021-02-10T11:30:58.296 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.291 7f753ecb7700 1 Processor -- start 2021-02-10T11:30:58.296 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.291 7f753ecb7700 1 -- start start 2021-02-10T11:30:58.296 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.292 7f753ecb7700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f75381b0470 0x7f75381b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:58.296 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.292 7f753ecb7700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f75380ef610 0x7f75380efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:58.296 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.292 7f753ecb7700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75380f0b50 con 0x7f75380f0030 2021-02-10T11:30:58.296 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.292 7f753ecb7700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75380e52e0 con 0x7f75381b0470 2021-02-10T11:30:58.297 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.292 7f753ecb7700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75380e5490 con 0x7f75380ef610 2021-02-10T11:30:58.297 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.292 7f753cad3700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f75381b0470 0x7f75381b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:58.297 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.292 7f7537fff700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f75380ef610 0x7f75380efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:58.297 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.292 7f753cad3700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f75381b0470 0x7f75381b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:47648/0 (socket says 172.21.2.112:47648) 2021-02-10T11:30:58.297 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.292 7f753cad3700 1 -- 172.21.2.112:0/143907267 learned_addr learned my addr 172.21.2.112:0/143907267 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:30:58.297 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.292 7f753d2d4700 1 --1- 172.21.2.112:0/143907267 >> v1:172.21.2.120:6789/0 conn(0x7f75380f0030 0x7f75380f10d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:40670/0 (socket says 172.21.2.112:40670) 2021-02-10T11:30:58.297 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.293 7f753cad3700 1 -- 172.21.2.112:0/143907267 >> v2:172.21.2.120:3300/0 conn(0x7f75380ef610 msgr2=0x7f75380efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:58.297 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.293 7f753cad3700 1 --2- 172.21.2.112:0/143907267 >> v2:172.21.2.120:3300/0 conn(0x7f75380ef610 0x7f75380efa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:58.297 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.293 7f753cad3700 1 -- 172.21.2.112:0/143907267 >> v1:172.21.2.120:6789/0 conn(0x7f75380f0030 legacy=0x7f75380f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:58.297 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.293 7f753cad3700 1 -- 172.21.2.112:0/143907267 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75380f0430 con 0x7f75381b0470 2021-02-10T11:30:58.297 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.293 7f75377fe700 1 -- 172.21.2.112:0/143907267 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1885529970 0 0) 0x7f7530000f90 con 0x7f75380f0030 2021-02-10T11:30:58.297 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.293 7f7537fff700 1 --2- 172.21.2.112:0/143907267 >> v2:172.21.2.120:3300/0 conn(0x7f75380ef610 0x7f75380efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:30:58.297 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.293 7f753cad3700 1 --2- 172.21.2.112:0/143907267 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f75381b0470 0x7f75381b0870 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 rx=0x7f7520006700 tx=0x7f7520009b80).ready entity=mon.0 client_cookie=2b66ce6d275a2253 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:58.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.293 7f75377fe700 1 -- 172.21.2.112:0/143907267 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f752000c0c0 con 0x7f75381b0470 2021-02-10T11:30:58.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.293 7f75377fe700 1 -- 172.21.2.112:0/143907267 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f752000c340 con 0x7f75381b0470 2021-02-10T11:30:58.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.293 7f75377fe700 1 -- 172.21.2.112:0/143907267 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f752000c820 con 0x7f75381b0470 2021-02-10T11:30:58.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.293 7f753ecb7700 1 -- 172.21.2.112:0/143907267 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f75381b0470 msgr2=0x7f75381b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:58.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.293 7f753ecb7700 1 --2- 172.21.2.112:0/143907267 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f75381b0470 0x7f75381b0870 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 rx=0x7f7520006700 tx=0x7f7520009b80).stop 2021-02-10T11:30:58.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.294 7f753ecb7700 1 -- 172.21.2.112:0/143907267 shutdown_connections 2021-02-10T11:30:58.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.294 7f753ecb7700 1 --2- 172.21.2.112:0/143907267 >> v2:172.21.2.120:3300/0 conn(0x7f75380ef610 0x7f75380efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:58.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.294 7f753ecb7700 1 --2- 172.21.2.112:0/143907267 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f75381b0470 0x7f75381b0870 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:58.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.294 7f753ecb7700 1 -- 172.21.2.112:0/143907267 >> 172.21.2.112:0/143907267 conn(0x7f75380ce3e0 msgr2=0x7f75380e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:58.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.294 7f753ecb7700 1 -- 172.21.2.112:0/143907267 shutdown_connections 2021-02-10T11:30:58.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.294 7f753ecb7700 1 -- 172.21.2.112:0/143907267 wait complete. 2021-02-10T11:30:58.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.309 7f753ecb7700 1 Processor -- start 2021-02-10T11:30:58.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.309 7f753ecb7700 1 -- start start 2021-02-10T11:30:58.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.309 7f753ecb7700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f75380a0420 0x7f75380a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:58.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.309 7f753ecb7700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7538189d00 0x7f753818a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:58.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.309 7f753ecb7700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f753818a720 0x7f75380b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:58.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.309 7f753ecb7700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7538189800 con 0x7f75380a0420 2021-02-10T11:30:58.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.309 7f753ecb7700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f753818ab20 con 0x7f753818a720 2021-02-10T11:30:58.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.309 7f753ecb7700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75380a1140 con 0x7f7538189d00 2021-02-10T11:30:58.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.309 7f753cad3700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f75380a0420 0x7f75380a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:58.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.309 7f753cad3700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f75380a0420 0x7f75380a0820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:47654/0 (socket says 172.21.2.112:47654) 2021-02-10T11:30:58.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.309 7f753d2d4700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f753818a720 0x7f75380b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:58.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.309 7f753cad3700 1 -- 172.21.2.112:0/2870178542 learned_addr learned my addr 172.21.2.112:0/2870178542 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:30:58.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.309 7f7537fff700 1 --2- 172.21.2.112:0/2870178542 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7538189d00 0x7f753818a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:58.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.310 7f753d2d4700 1 -- 172.21.2.112:0/2870178542 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7538189d00 msgr2=0x7f753818a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:58.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.310 7f753d2d4700 1 --2- 172.21.2.112:0/2870178542 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7538189d00 0x7f753818a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:58.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.310 7f753d2d4700 1 -- 172.21.2.112:0/2870178542 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f75380a0420 msgr2=0x7f75380a0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:58.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.310 7f753d2d4700 1 --2- 172.21.2.112:0/2870178542 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f75380a0420 0x7f75380a0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:58.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.310 7f753d2d4700 1 -- 172.21.2.112:0/2870178542 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75380b8200 con 0x7f753818a720 2021-02-10T11:30:58.302 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.310 7f753cad3700 1 --2- 172.21.2.112:0/2870178542 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f75380a0420 0x7f75380a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:30:58.302 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.310 7f753d2d4700 1 --2- 172.21.2.112:0/2870178542 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f753818a720 0x7f75380b7be0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7f7530001b10 tx=0x7f753000ab20).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:58.302 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.310 7f75357fa700 1 -- 172.21.2.112:0/2870178542 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7530000f00 con 0x7f753818a720 2021-02-10T11:30:58.302 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.310 7f75357fa700 1 -- 172.21.2.112:0/2870178542 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75300011f0 con 0x7f753818a720 2021-02-10T11:30:58.302 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.310 7f753ecb7700 1 -- 172.21.2.112:0/2870178542 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75380b8200 con 0x7f753818a720 2021-02-10T11:30:58.302 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.310 7f753ecb7700 1 -- 172.21.2.112:0/2870178542 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75380b8200 con 0x7f753818a720 2021-02-10T11:30:58.302 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.310 7f75357fa700 1 -- 172.21.2.112:0/2870178542 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7530007490 con 0x7f753818a720 2021-02-10T11:30:58.302 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.311 7f75357fa700 1 -- 172.21.2.112:0/2870178542 <== mon.1 v2:172.21.2.120:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f75300046a0 con 0x7f753818a720 2021-02-10T11:30:58.302 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.311 7f75357fa700 1 --2- 172.21.2.112:0/2870178542 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f751c018730 0x7f751c01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:58.302 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.311 7f75357fa700 1 -- 172.21.2.112:0/2870178542 <== mon.1 v2:172.21.2.120:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f7530032a30 con 0x7f753818a720 2021-02-10T11:30:58.302 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.311 7f753cad3700 1 --2- 172.21.2.112:0/2870178542 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f751c018730 0x7f751c01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:58.302 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.311 7f753cad3700 1 --2- 172.21.2.112:0/2870178542 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f751c018730 0x7f751c01ac10 crc :-1 s=READY pgs=469 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:58.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.341 7f753ecb7700 1 -- 172.21.2.112:0/2870178542 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f753818ab20 con 0x7f753818a720 2021-02-10T11:30:58.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.344 7f75357fa700 1 -- 172.21.2.112:0/2870178542 <== mon.1 v2:172.21.2.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f75300011f0 con 0x7f753818a720 2021-02-10T11:30:58.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.472 7f753ecb7700 1 -- 172.21.2.112:0/2870178542 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348648", "target": ["mgr", ""]}) v1 -- 0x7f753818ab20 con 0x7f751c018730 2021-02-10T11:30:58.470 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.482 7f75357fa700 1 -- 172.21.2.112:0/2870178542 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f7520006e80 con 0x7f751c018730 2021-02-10T11:30:58.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.488 7f753ecb7700 1 -- 172.21.2.112:0/2870178542 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f751c018730 msgr2=0x7f751c01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:58.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.488 7f753ecb7700 1 --2- 172.21.2.112:0/2870178542 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f751c018730 0x7f751c01ac10 crc :-1 s=READY pgs=469 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:58.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.489 7f753ecb7700 1 -- 172.21.2.112:0/2870178542 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f753818a720 msgr2=0x7f75380b7be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:58.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.489 7f753ecb7700 1 --2- 172.21.2.112:0/2870178542 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f753818a720 0x7f75380b7be0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7f7530001b10 tx=0x7f753000ab20).stop 2021-02-10T11:30:58.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.489 7f753ecb7700 1 -- 172.21.2.112:0/2870178542 shutdown_connections 2021-02-10T11:30:58.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.489 7f753ecb7700 1 --2- 172.21.2.112:0/2870178542 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f751c018730 0x7f751c01ac10 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:58.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.489 7f753ecb7700 1 --2- 172.21.2.112:0/2870178542 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f753818a720 0x7f75380b7be0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:58.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.489 7f753ecb7700 1 --2- 172.21.2.112:0/2870178542 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7538189d00 0x7f753818a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:58.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.489 7f753ecb7700 1 --2- 172.21.2.112:0/2870178542 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f75380a0420 0x7f75380a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:58.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.489 7f753ecb7700 1 -- 172.21.2.112:0/2870178542 >> 172.21.2.112:0/2870178542 conn(0x7f7538094e10 msgr2=0x7f75380a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:58.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.489 7f753ecb7700 1 -- 172.21.2.112:0/2870178542 shutdown_connections 2021-02-10T11:30:58.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.489 7f753ecb7700 1 -- 172.21.2.112:0/2870178542 wait complete. 2021-02-10T11:30:58.485 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000348649 2021-02-10T11:30:58.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.912 7f4899ebb700 1 Processor -- start 2021-02-10T11:30:58.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.912 7f4899ebb700 1 -- start start 2021-02-10T11:30:58.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.913 7f4899ebb700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f48941b0870 0x7f48940f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:58.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.913 7f4899ebb700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f48940efe00 0x7f48940f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:58.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.913 7f4899ebb700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48941b0c70 con 0x7f48941b0470 2021-02-10T11:30:58.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.913 7f4899ebb700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48940f0b50 con 0x7f48940efe00 2021-02-10T11:30:58.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.913 7f4899ebb700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48940f0820 con 0x7f48941b0870 2021-02-10T11:30:58.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.913 7f4893fff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f48940efe00 0x7f48940f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:58.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.913 7f48937fe700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7f48941b0470 0x7f48940ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:40682/0 (socket says 172.21.2.112:40682) 2021-02-10T11:30:58.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.913 7f48937fe700 1 -- 172.21.2.112:0/704657959 learned_addr learned my addr 172.21.2.112:0/704657959 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:30:58.917 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.913 7f4892ffd700 1 --2- 172.21.2.112:0/704657959 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f48941b0870 0x7f48940f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:58.917 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.913 7f4893fff700 1 -- 172.21.2.112:0/704657959 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f48941b0870 msgr2=0x7f48940f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:58.917 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.913 7f4893fff700 1 --2- 172.21.2.112:0/704657959 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f48941b0870 0x7f48940f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:58.917 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.913 7f4893fff700 1 -- 172.21.2.112:0/704657959 >> v1:172.21.2.120:6789/0 conn(0x7f48941b0470 legacy=0x7f48940ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:58.917 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.913 7f48927fc700 1 -- 172.21.2.112:0/704657959 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2764767460 0 0) 0x7f487c0009b0 con 0x7f48941b0470 2021-02-10T11:30:58.917 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.914 7f4893fff700 1 -- 172.21.2.112:0/704657959 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48940e52e0 con 0x7f48940efe00 2021-02-10T11:30:58.917 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.914 7f4892ffd700 1 --2- 172.21.2.112:0/704657959 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f48941b0870 0x7f48940f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:30:58.917 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.914 7f4893fff700 1 --2- 172.21.2.112:0/704657959 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f48940efe00 0x7f48940f0200 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 rx=0x7f488c006e30 tx=0x7f488c009e60).ready entity=mon.0 client_cookie=7fbd046b25e814d9 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:58.917 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.914 7f48927fc700 1 -- 172.21.2.112:0/704657959 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f488c006a50 con 0x7f48940efe00 2021-02-10T11:30:58.917 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.914 7f48927fc700 1 -- 172.21.2.112:0/704657959 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f488c00b300 con 0x7f48940efe00 2021-02-10T11:30:58.917 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.914 7f48927fc700 1 -- 172.21.2.112:0/704657959 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f488c00b820 con 0x7f48940efe00 2021-02-10T11:30:58.917 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.914 7f4899ebb700 1 -- 172.21.2.112:0/704657959 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f48940efe00 msgr2=0x7f48940f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:58.917 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.914 7f4899ebb700 1 --2- 172.21.2.112:0/704657959 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f48940efe00 0x7f48940f0200 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 rx=0x7f488c006e30 tx=0x7f488c009e60).stop 2021-02-10T11:30:58.918 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.914 7f4899ebb700 1 -- 172.21.2.112:0/704657959 shutdown_connections 2021-02-10T11:30:58.918 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.914 7f4899ebb700 1 --2- 172.21.2.112:0/704657959 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f48940efe00 0x7f48940f0200 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:58.918 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.914 7f4899ebb700 1 --2- 172.21.2.112:0/704657959 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f48941b0870 0x7f48940f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:58.918 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.914 7f4899ebb700 1 -- 172.21.2.112:0/704657959 >> 172.21.2.112:0/704657959 conn(0x7f48940ce3e0 msgr2=0x7f48940e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:58.918 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.914 7f4899ebb700 1 -- 172.21.2.112:0/704657959 shutdown_connections 2021-02-10T11:30:58.918 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.914 7f4899ebb700 1 -- 172.21.2.112:0/704657959 wait complete. 2021-02-10T11:30:58.919 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.929 7f4899ebb700 1 Processor -- start 2021-02-10T11:30:58.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.929 7f4899ebb700 1 -- start start 2021-02-10T11:30:58.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.929 7f4899ebb700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4894190920 0x7f48940b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:58.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.929 7f4899ebb700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f48940b8200 0x7f48940b8600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:58.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.929 7f4899ebb700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f48940c1d80 0x7f48940c2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:58.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.929 7f4899ebb700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4894190d20 con 0x7f48940c1d80 2021-02-10T11:30:58.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.929 7f4899ebb700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48940b8c20 con 0x7f4894190920 2021-02-10T11:30:58.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.929 7f4899ebb700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48940c27a0 con 0x7f48940b8200 2021-02-10T11:30:58.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.929 7f4893fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f48940c1d80 0x7f48940c2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:58.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.929 7f4893fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f48940c1d80 0x7f48940c2180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:47672/0 (socket says 172.21.2.112:47672) 2021-02-10T11:30:58.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.929 7f4893fff700 1 -- 172.21.2.112:0/1221950416 learned_addr learned my addr 172.21.2.112:0/1221950416 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:30:58.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.930 7f48937fe700 1 --2- 172.21.2.112:0/1221950416 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4894190920 0x7f48940b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:58.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.930 7f4893fff700 1 -- 172.21.2.112:0/1221950416 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f48940b8200 msgr2=0x7f48940b8600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:58.921 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.930 7f4893fff700 1 --2- 172.21.2.112:0/1221950416 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f48940b8200 0x7f48940b8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:58.921 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.930 7f4893fff700 1 -- 172.21.2.112:0/1221950416 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4894190920 msgr2=0x7f48940b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:58.921 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.930 7f4893fff700 1 --2- 172.21.2.112:0/1221950416 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4894190920 0x7f48940b7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:58.921 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.930 7f4893fff700 1 -- 172.21.2.112:0/1221950416 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48940c29c0 con 0x7f48940c1d80 2021-02-10T11:30:58.921 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.930 7f4893fff700 1 --2- 172.21.2.112:0/1221950416 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f48940c1d80 0x7f48940c2180 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 rx=0x7f488c001070 tx=0x7f488c00bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:58.921 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.930 7f488bfff700 1 -- 172.21.2.112:0/1221950416 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f488c001270 con 0x7f48940c1d80 2021-02-10T11:30:58.921 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.930 7f488bfff700 1 -- 172.21.2.112:0/1221950416 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f488c0014f0 con 0x7f48940c1d80 2021-02-10T11:30:58.921 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.930 7f488bfff700 1 -- 172.21.2.112:0/1221950416 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f488c0019d0 con 0x7f48940c1d80 2021-02-10T11:30:58.921 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.930 7f4899ebb700 1 -- 172.21.2.112:0/1221950416 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48940c29c0 con 0x7f48940c1d80 2021-02-10T11:30:58.921 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.930 7f4899ebb700 1 -- 172.21.2.112:0/1221950416 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48940c2f50 con 0x7f48940c1d80 2021-02-10T11:30:58.921 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.931 7f488bfff700 1 -- 172.21.2.112:0/1221950416 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f488c0016d0 con 0x7f48940c1d80 2021-02-10T11:30:58.921 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.931 7f488bfff700 1 --2- 172.21.2.112:0/1221950416 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f48780186c0 0x7f487801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:58.921 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.931 7f488bfff700 1 -- 172.21.2.112:0/1221950416 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f488c032930 con 0x7f48940c1d80 2021-02-10T11:30:58.921 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.931 7f48937fe700 1 --2- 172.21.2.112:0/1221950416 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f48780186c0 0x7f487801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:58.922 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.932 7f48937fe700 1 --2- 172.21.2.112:0/1221950416 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f48780186c0 0x7f487801abc0 crc :-1 s=READY pgs=470 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:58.950 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.963 7f4899ebb700 1 -- 172.21.2.112:0/1221950416 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4894105ae0 con 0x7f48940c1d80 2021-02-10T11:30:58.952 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:58.965 7f488bfff700 1 -- 172.21.2.112:0/1221950416 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f488c0093d0 con 0x7f48940c1d80 2021-02-10T11:30:59.084 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:59.097 7f4899ebb700 1 -- 172.21.2.112:0/1221950416 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348649", "target": ["mgr", ""]}) v1 -- 0x7f4894190d20 con 0x7f48780186c0 2021-02-10T11:30:59.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:59.107 7f488bfff700 1 -- 172.21.2.112:0/1221950416 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f487c001950 con 0x7f48780186c0 2021-02-10T11:30:59.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:59.114 7f4899ebb700 1 -- 172.21.2.112:0/1221950416 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f48780186c0 msgr2=0x7f487801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:59.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:59.114 7f4899ebb700 1 --2- 172.21.2.112:0/1221950416 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f48780186c0 0x7f487801abc0 crc :-1 s=READY pgs=470 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:59.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:59.114 7f4899ebb700 1 -- 172.21.2.112:0/1221950416 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f48940c1d80 msgr2=0x7f48940c2180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:59.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:59.114 7f4899ebb700 1 --2- 172.21.2.112:0/1221950416 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f48940c1d80 0x7f48940c2180 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 rx=0x7f488c001070 tx=0x7f488c00bd40).stop 2021-02-10T11:30:59.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:59.115 7f4899ebb700 1 -- 172.21.2.112:0/1221950416 shutdown_connections 2021-02-10T11:30:59.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:59.115 7f4899ebb700 1 --2- 172.21.2.112:0/1221950416 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f48780186c0 0x7f487801abc0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:59.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:59.115 7f4899ebb700 1 --2- 172.21.2.112:0/1221950416 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f48940c1d80 0x7f48940c2180 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:59.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:59.115 7f4899ebb700 1 --2- 172.21.2.112:0/1221950416 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f48940b8200 0x7f48940b8600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:59.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:59.115 7f4899ebb700 1 --2- 172.21.2.112:0/1221950416 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4894190920 0x7f48940b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:59.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:59.115 7f4899ebb700 1 -- 172.21.2.112:0/1221950416 >> 172.21.2.112:0/1221950416 conn(0x7f4894094e10 msgr2=0x7f48940a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:59.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:59.115 7f4899ebb700 1 -- 172.21.2.112:0/1221950416 shutdown_connections 2021-02-10T11:30:59.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:59.115 7f4899ebb700 1 -- 172.21.2.112:0/1221950416 wait complete. 2021-02-10T11:30:59.111 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000348650 2021-02-10T11:30:59.538 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:59.534 7fe209749700 1 Processor -- start 2021-02-10T11:30:59.538 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:59.534 7fe209749700 1 -- start start 2021-02-10T11:30:59.538 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:59.534 7fe209749700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe2041b0470 0x7fe2041b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:59.538 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:59.534 7fe209749700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fe2040f20e0 0x7fe2040f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:59.538 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:59.534 7fe209749700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2040f0b50 con 0x7fe2040ef610 2021-02-10T11:30:59.538 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:59.534 7fe209749700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2040f0820 con 0x7fe2040f20e0 2021-02-10T11:30:59.538 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:59.534 7fe209749700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2040f24e0 con 0x7fe2041b0470 2021-02-10T11:30:59.538 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:59.534 7fe202d48700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fe2040ef610 0x7fe2040efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:47138/0 (socket says 172.21.2.112:47138) 2021-02-10T11:30:59.539 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:59.534 7fe202d48700 1 -- 172.21.2.112:0/2535642060 learned_addr learned my addr 172.21.2.112:0/2535642060 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:30:59.539 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:59.534 7fe203d4a700 1 --2- 172.21.2.112:0/2535642060 >> v2:172.21.2.112:3300/0 conn(0x7fe2040f20e0 0x7fe2040f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:59.539 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:59.534 7fe203549700 1 --2- 172.21.2.112:0/2535642060 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe2041b0470 0x7fe2041b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:59.539 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:59.535 7fe203549700 1 -- 172.21.2.112:0/2535642060 >> v2:172.21.2.112:3300/0 conn(0x7fe2040f20e0 msgr2=0x7fe2040f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:59.539 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:59.535 7fe203549700 1 --2- 172.21.2.112:0/2535642060 >> v2:172.21.2.112:3300/0 conn(0x7fe2040f20e0 0x7fe2040f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:59.539 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:59.535 7fe203549700 1 -- 172.21.2.112:0/2535642060 >> v1:172.21.2.112:6789/0 conn(0x7fe2040ef610 legacy=0x7fe2040efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:59.539 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:59.535 7fe203549700 1 -- 172.21.2.112:0/2535642060 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2040e52e0 con 0x7fe2041b0470 2021-02-10T11:30:59.539 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:59.535 7fe202547700 1 -- 172.21.2.112:0/2535642060 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1736109470 0 0) 0x7fe1ec0010b0 con 0x7fe2040ef610 2021-02-10T11:30:59.539 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:59.535 7fe203549700 1 --2- 172.21.2.112:0/2535642060 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe2041b0470 0x7fe2041b0870 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7fe1f4006650 tx=0x7fe1f4009a80).ready entity=mon.2 client_cookie=544eef77435cbc1f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:59.539 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:59.536 7fe202547700 1 -- 172.21.2.112:0/2535642060 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe1f400c0c0 con 0x7fe2041b0470 2021-02-10T11:30:59.539 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:59.536 7fe202547700 1 -- 172.21.2.112:0/2535642060 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1f400c340 con 0x7fe2041b0470 2021-02-10T11:30:59.539 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:59.536 7fe202547700 1 -- 172.21.2.112:0/2535642060 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe1f400c860 con 0x7fe2041b0470 2021-02-10T11:30:59.539 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:59.536 7fe209749700 1 -- 172.21.2.112:0/2535642060 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe2041b0470 msgr2=0x7fe2041b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:59.540 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:59.536 7fe209749700 1 --2- 172.21.2.112:0/2535642060 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe2041b0470 0x7fe2041b0870 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7fe1f4006650 tx=0x7fe1f4009a80).stop 2021-02-10T11:30:59.540 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:59.536 7fe209749700 1 -- 172.21.2.112:0/2535642060 shutdown_connections 2021-02-10T11:30:59.540 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:59.536 7fe209749700 1 --2- 172.21.2.112:0/2535642060 >> v2:172.21.2.112:3300/0 conn(0x7fe2040f20e0 0x7fe2040f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:59.540 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:59.536 7fe209749700 1 --2- 172.21.2.112:0/2535642060 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe2041b0470 0x7fe2041b0870 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:59.540 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:59.536 7fe209749700 1 -- 172.21.2.112:0/2535642060 >> 172.21.2.112:0/2535642060 conn(0x7fe2040ce3e0 msgr2=0x7fe2040e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:59.540 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:59.536 7fe209749700 1 -- 172.21.2.112:0/2535642060 shutdown_connections 2021-02-10T11:30:59.540 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:59.536 7fe209749700 1 -- 172.21.2.112:0/2535642060 wait complete. 2021-02-10T11:30:59.542 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:59.551 7fe209749700 1 Processor -- start 2021-02-10T11:30:59.542 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:59.551 7fe209749700 1 -- start start 2021-02-10T11:30:59.542 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:59.551 7fe209749700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe20409e560 0x7fe2040a0130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:59.542 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:59.551 7fe209749700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe20409e960 0x7fe2040a0750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:59.542 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:59.551 7fe209749700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe2040a0d70 0x7fe2040a1af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:59.542 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:59.551 7fe209749700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2040c38f0 con 0x7fe20409e960 2021-02-10T11:30:59.542 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:59.551 7fe209749700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2040a1170 con 0x7fe2040a0d70 2021-02-10T11:30:59.542 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:59.551 7fe209749700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2040a1390 con 0x7fe20409e560 2021-02-10T11:30:59.542 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:59.552 7fe202d48700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe20409e960 0x7fe2040a0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:59.542 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:59.552 7fe202d48700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe20409e960 0x7fe2040a0750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:47686/0 (socket says 172.21.2.112:47686) 2021-02-10T11:30:59.542 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:59.552 7fe202d48700 1 -- 172.21.2.112:0/377309461 learned_addr learned my addr 172.21.2.112:0/377309461 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:30:59.543 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:59.552 7fe203d4a700 1 --2- 172.21.2.112:0/377309461 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe2040a0d70 0x7fe2040a1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:59.543 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:59.552 7fe203549700 1 --2- 172.21.2.112:0/377309461 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe20409e560 0x7fe2040a0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:59.543 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:59.552 7fe202d48700 1 -- 172.21.2.112:0/377309461 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe20409e560 msgr2=0x7fe2040a0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:59.543 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:59.552 7fe202d48700 1 --2- 172.21.2.112:0/377309461 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe20409e560 0x7fe2040a0130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:59.543 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:59.552 7fe202d48700 1 -- 172.21.2.112:0/377309461 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe2040a0d70 msgr2=0x7fe2040a1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:59.543 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:59.552 7fe202d48700 1 --2- 172.21.2.112:0/377309461 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe2040a0d70 0x7fe2040a1af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:59.543 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:59.552 7fe202d48700 1 -- 172.21.2.112:0/377309461 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2040a2110 con 0x7fe20409e960 2021-02-10T11:30:59.543 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:59.552 7fe202d48700 1 --2- 172.21.2.112:0/377309461 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe20409e960 0x7fe2040a0750 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 rx=0x7fe1ec001c60 tx=0x7fe1ec00a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:59.543 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:59.553 7fe1fbfff700 1 -- 172.21.2.112:0/377309461 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe1ec00acc0 con 0x7fe20409e960 2021-02-10T11:30:59.543 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:59.553 7fe1fbfff700 1 -- 172.21.2.112:0/377309461 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1ec0012a0 con 0x7fe20409e960 2021-02-10T11:30:59.543 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:59.553 7fe1fbfff700 1 -- 172.21.2.112:0/377309461 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe1ec0043c0 con 0x7fe20409e960 2021-02-10T11:30:59.543 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:59.553 7fe209749700 1 -- 172.21.2.112:0/377309461 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2040a2110 con 0x7fe20409e960 2021-02-10T11:30:59.543 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:59.553 7fe209749700 1 -- 172.21.2.112:0/377309461 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2040a26a0 con 0x7fe20409e960 2021-02-10T11:30:59.544 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:59.553 7fe1fbfff700 1 -- 172.21.2.112:0/377309461 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe1ec004350 con 0x7fe20409e960 2021-02-10T11:30:59.544 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:59.553 7fe1fbfff700 1 --2- 172.21.2.112:0/377309461 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe1e80186c0 0x7fe1e801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:59.544 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:59.553 7fe1fbfff700 1 -- 172.21.2.112:0/377309461 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fe1ec032500 con 0x7fe20409e960 2021-02-10T11:30:59.544 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:59.554 7fe203549700 1 --2- 172.21.2.112:0/377309461 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe1e80186c0 0x7fe1e801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:59.544 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:59.554 7fe203549700 1 --2- 172.21.2.112:0/377309461 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe1e80186c0 0x7fe1e801abc0 crc :-1 s=READY pgs=471 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:59.572 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:59.584 7fe209749700 1 -- 172.21.2.112:0/377309461 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe204105b30 con 0x7fe20409e960 2021-02-10T11:30:59.578 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:59.587 7fe1fbfff700 1 -- 172.21.2.112:0/377309461 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe1ec004140 con 0x7fe20409e960 2021-02-10T11:30:59.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:59.715 7fe209749700 1 -- 172.21.2.112:0/377309461 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348650", "target": ["mgr", ""]}) v1 -- 0x7fe2040a26a0 con 0x7fe1e80186c0 2021-02-10T11:30:59.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:59.725 7fe1fbfff700 1 -- 172.21.2.112:0/377309461 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fe1f4006420 con 0x7fe1e80186c0 2021-02-10T11:30:59.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:59.732 7fe209749700 1 -- 172.21.2.112:0/377309461 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe1e80186c0 msgr2=0x7fe1e801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:59.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:59.732 7fe209749700 1 --2- 172.21.2.112:0/377309461 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe1e80186c0 0x7fe1e801abc0 crc :-1 s=READY pgs=471 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:59.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:59.732 7fe209749700 1 -- 172.21.2.112:0/377309461 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe20409e960 msgr2=0x7fe2040a0750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:59.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:59.732 7fe209749700 1 --2- 172.21.2.112:0/377309461 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe20409e960 0x7fe2040a0750 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 rx=0x7fe1ec001c60 tx=0x7fe1ec00a8d0).stop 2021-02-10T11:30:59.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:59.733 7fe209749700 1 -- 172.21.2.112:0/377309461 shutdown_connections 2021-02-10T11:30:59.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:59.733 7fe209749700 1 --2- 172.21.2.112:0/377309461 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe1e80186c0 0x7fe1e801abc0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:59.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:59.733 7fe209749700 1 --2- 172.21.2.112:0/377309461 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe2040a0d70 0x7fe2040a1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:59.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:59.733 7fe209749700 1 --2- 172.21.2.112:0/377309461 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe20409e960 0x7fe2040a0750 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:59.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:59.733 7fe209749700 1 --2- 172.21.2.112:0/377309461 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe20409e560 0x7fe2040a0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:59.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:59.733 7fe209749700 1 -- 172.21.2.112:0/377309461 >> 172.21.2.112:0/377309461 conn(0x7fe20409fc10 msgr2=0x7fe204094ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:59.721 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:59.733 7fe209749700 1 -- 172.21.2.112:0/377309461 shutdown_connections 2021-02-10T11:30:59.721 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:30:59.733 7fe209749700 1 -- 172.21.2.112:0/377309461 wait complete. 2021-02-10T11:30:59.729 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000348651 2021-02-10T11:31:00.159 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.155 7f1c1692d700 1 Processor -- start 2021-02-10T11:31:00.159 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.155 7f1c1692d700 1 -- start start 2021-02-10T11:31:00.159 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.156 7f1c1692d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f1c101b0470 0x7f1c101b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:00.160 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.156 7f1c1692d700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f1c100f20e0 0x7f1c100f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:00.160 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.156 7f1c1692d700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c100f0b50 con 0x7f1c100ef610 2021-02-10T11:31:00.160 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.156 7f1c1692d700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c100f0820 con 0x7f1c101b0470 2021-02-10T11:31:00.160 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.156 7f1c1692d700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c100f24e0 con 0x7f1c100f20e0 2021-02-10T11:31:00.160 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.156 7f1c0ffff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f1c101b0470 0x7f1c101b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:00.160 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.156 7f1c0ffff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f1c101b0470 0x7f1c101b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:47690/0 (socket says 172.21.2.112:47690) 2021-02-10T11:31:00.160 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.156 7f1c0ffff700 1 -- 172.21.2.112:0/4191626968 learned_addr learned my addr 172.21.2.112:0/4191626968 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:31:00.160 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.156 7f1c14f4a700 1 --2- 172.21.2.112:0/4191626968 >> v2:172.21.2.120:3300/0 conn(0x7f1c100f20e0 0x7f1c100f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:00.160 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.156 7f1c0ffff700 1 -- 172.21.2.112:0/4191626968 >> v2:172.21.2.120:3300/0 conn(0x7f1c100f20e0 msgr2=0x7f1c100f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:00.160 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.156 7f1c0ffff700 1 --2- 172.21.2.112:0/4191626968 >> v2:172.21.2.120:3300/0 conn(0x7f1c100f20e0 0x7f1c100f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:00.160 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.156 7f1c0ffff700 1 -- 172.21.2.112:0/4191626968 >> v1:172.21.2.120:6789/0 conn(0x7f1c100ef610 legacy=0x7f1c100efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:00.160 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.156 7f1c0ffff700 1 -- 172.21.2.112:0/4191626968 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c100e52e0 con 0x7f1c101b0470 2021-02-10T11:31:00.160 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.156 7f1c14f4a700 1 --2- 172.21.2.112:0/4191626968 >> v2:172.21.2.120:3300/0 conn(0x7f1c100f20e0 0x7f1c100f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:31:00.161 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.156 7f1c0ffff700 1 --2- 172.21.2.112:0/4191626968 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1c101b0470 0x7f1c101b0870 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 rx=0x7f1c00006700 tx=0x7f1c00009b80).ready entity=mon.0 client_cookie=bf710e85311f54a7 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:00.161 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.156 7f1c0effd700 1 -- 172.21.2.112:0/4191626968 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1c0000c0c0 con 0x7f1c101b0470 2021-02-10T11:31:00.161 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.156 7f1c0effd700 1 -- 172.21.2.112:0/4191626968 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c0000c440 con 0x7f1c101b0470 2021-02-10T11:31:00.161 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.156 7f1c0effd700 1 -- 172.21.2.112:0/4191626968 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1c0000c8a0 con 0x7f1c101b0470 2021-02-10T11:31:00.161 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.156 7f1c1692d700 1 -- 172.21.2.112:0/4191626968 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1c101b0470 msgr2=0x7f1c101b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:00.161 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.156 7f1c1692d700 1 --2- 172.21.2.112:0/4191626968 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1c101b0470 0x7f1c101b0870 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 rx=0x7f1c00006700 tx=0x7f1c00009b80).stop 2021-02-10T11:31:00.161 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.157 7f1c1692d700 1 -- 172.21.2.112:0/4191626968 shutdown_connections 2021-02-10T11:31:00.161 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.157 7f1c1692d700 1 --2- 172.21.2.112:0/4191626968 >> v2:172.21.2.120:3300/0 conn(0x7f1c100f20e0 0x7f1c100f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:00.161 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.157 7f1c1692d700 1 --2- 172.21.2.112:0/4191626968 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1c101b0470 0x7f1c101b0870 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:00.161 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.157 7f1c1692d700 1 -- 172.21.2.112:0/4191626968 >> 172.21.2.112:0/4191626968 conn(0x7f1c100ce3e0 msgr2=0x7f1c100e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:00.161 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.157 7f1c1692d700 1 -- 172.21.2.112:0/4191626968 shutdown_connections 2021-02-10T11:31:00.161 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.157 7f1c1692d700 1 -- 172.21.2.112:0/4191626968 wait complete. 2021-02-10T11:31:00.161 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.172 7f1c1692d700 1 Processor -- start 2021-02-10T11:31:00.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.172 7f1c1692d700 1 -- start start 2021-02-10T11:31:00.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.173 7f1c1692d700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1c10190920 0x7f1c100b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:00.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.173 7f1c1692d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1c100b8200 0x7f1c100b8600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:00.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.173 7f1c1692d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1c100c1d80 0x7f1c100c2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:00.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.173 7f1c1692d700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c10190da0 con 0x7f1c100c1d80 2021-02-10T11:31:00.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.173 7f1c1692d700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c100b8c20 con 0x7f1c10190920 2021-02-10T11:31:00.164 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.173 7f1c1692d700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c100b8e10 con 0x7f1c100b8200 2021-02-10T11:31:00.164 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.173 7f1c14f4a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1c100c1d80 0x7f1c100c2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:00.164 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.173 7f1c14f4a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1c100c1d80 0x7f1c100c2180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:47696/0 (socket says 172.21.2.112:47696) 2021-02-10T11:31:00.164 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.173 7f1c14f4a700 1 -- 172.21.2.112:0/3456946947 learned_addr learned my addr 172.21.2.112:0/3456946947 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:31:00.164 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.173 7f1c0ffff700 1 --2- 172.21.2.112:0/3456946947 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1c10190920 0x7f1c100b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:00.164 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.173 7f1c0f7fe700 1 --2- 172.21.2.112:0/3456946947 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1c100b8200 0x7f1c100b8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:00.164 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.173 7f1c14f4a700 1 -- 172.21.2.112:0/3456946947 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1c100b8200 msgr2=0x7f1c100b8600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:00.164 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.173 7f1c14f4a700 1 --2- 172.21.2.112:0/3456946947 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1c100b8200 0x7f1c100b8600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:00.164 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.173 7f1c14f4a700 1 -- 172.21.2.112:0/3456946947 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1c10190920 msgr2=0x7f1c100b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:00.164 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.173 7f1c14f4a700 1 --2- 172.21.2.112:0/3456946947 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1c10190920 0x7f1c100b7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:00.164 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.173 7f1c14f4a700 1 -- 172.21.2.112:0/3456946947 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c100c27d0 con 0x7f1c100c1d80 2021-02-10T11:31:00.164 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.173 7f1c14f4a700 1 --2- 172.21.2.112:0/3456946947 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1c100c1d80 0x7f1c100c2180 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 rx=0x7f1c0800b970 tx=0x7f1c0800bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:00.164 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.173 7f1c0cff9700 1 -- 172.21.2.112:0/3456946947 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1c08000f90 con 0x7f1c100c1d80 2021-02-10T11:31:00.164 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.173 7f1c0cff9700 1 -- 172.21.2.112:0/3456946947 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c08006aa0 con 0x7f1c100c1d80 2021-02-10T11:31:00.165 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.173 7f1c0cff9700 1 -- 172.21.2.112:0/3456946947 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1c08001680 con 0x7f1c100c1d80 2021-02-10T11:31:00.165 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.173 7f1c1692d700 1 -- 172.21.2.112:0/3456946947 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c100c27d0 con 0x7f1c100c1d80 2021-02-10T11:31:00.165 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.173 7f1c1692d700 1 -- 172.21.2.112:0/3456946947 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c100c27d0 con 0x7f1c100c1d80 2021-02-10T11:31:00.165 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.174 7f1c0cff9700 1 -- 172.21.2.112:0/3456946947 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1c08000df0 con 0x7f1c100c1d80 2021-02-10T11:31:00.165 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.174 7f1c0cff9700 1 --2- 172.21.2.112:0/3456946947 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1bf00186d0 0x7f1bf001abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:00.165 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.174 7f1c0cff9700 1 -- 172.21.2.112:0/3456946947 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f1c08032b40 con 0x7f1c100c1d80 2021-02-10T11:31:00.165 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.174 7f1c0ffff700 1 --2- 172.21.2.112:0/3456946947 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1bf00186d0 0x7f1bf001abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:00.165 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.174 7f1c0ffff700 1 --2- 172.21.2.112:0/3456946947 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1bf00186d0 0x7f1bf001abd0 crc :-1 s=READY pgs=472 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:00.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.205 7f1c1692d700 1 -- 172.21.2.112:0/3456946947 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c10105ae0 con 0x7f1c100c1d80 2021-02-10T11:31:00.194 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.207 7f1c0cff9700 1 -- 172.21.2.112:0/3456946947 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1c08006aa0 con 0x7f1c100c1d80 2021-02-10T11:31:00.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.334 7f1c1692d700 1 -- 172.21.2.112:0/3456946947 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348651", "target": ["mgr", ""]}) v1 -- 0x7f1c100f0b50 con 0x7f1bf00186d0 2021-02-10T11:31:00.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.345 7f1c0cff9700 1 -- 172.21.2.112:0/3456946947 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f1c000009b0 con 0x7f1bf00186d0 2021-02-10T11:31:00.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.352 7f1c1692d700 1 -- 172.21.2.112:0/3456946947 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1bf00186d0 msgr2=0x7f1bf001abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:00.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.352 7f1c1692d700 1 --2- 172.21.2.112:0/3456946947 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1bf00186d0 0x7f1bf001abd0 crc :-1 s=READY pgs=472 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:00.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.352 7f1c1692d700 1 -- 172.21.2.112:0/3456946947 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1c100c1d80 msgr2=0x7f1c100c2180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:00.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.352 7f1c1692d700 1 --2- 172.21.2.112:0/3456946947 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1c100c1d80 0x7f1c100c2180 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 rx=0x7f1c0800b970 tx=0x7f1c0800bd10).stop 2021-02-10T11:31:00.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.352 7f1c1692d700 1 -- 172.21.2.112:0/3456946947 shutdown_connections 2021-02-10T11:31:00.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.352 7f1c1692d700 1 --2- 172.21.2.112:0/3456946947 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1bf00186d0 0x7f1bf001abd0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:00.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.352 7f1c1692d700 1 --2- 172.21.2.112:0/3456946947 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1c100c1d80 0x7f1c100c2180 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:00.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.352 7f1c1692d700 1 --2- 172.21.2.112:0/3456946947 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1c100b8200 0x7f1c100b8600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:00.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.352 7f1c1692d700 1 --2- 172.21.2.112:0/3456946947 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1c10190920 0x7f1c100b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:00.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.352 7f1c1692d700 1 -- 172.21.2.112:0/3456946947 >> 172.21.2.112:0/3456946947 conn(0x7f1c10094e10 msgr2=0x7f1c100a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:00.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.353 7f1c1692d700 1 -- 172.21.2.112:0/3456946947 shutdown_connections 2021-02-10T11:31:00.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.353 7f1c1692d700 1 -- 172.21.2.112:0/3456946947 wait complete. 2021-02-10T11:31:00.348 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000348652 2021-02-10T11:31:00.778 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.774 7f4dd7bfc700 1 Processor -- start 2021-02-10T11:31:00.778 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.774 7f4dd7bfc700 1 -- start start 2021-02-10T11:31:00.778 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.774 7f4dd7bfc700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f4dd01b0870 0x7f4dd00f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:00.778 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.774 7f4dd7bfc700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4dd00efe00 0x7f4dd00f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:00.778 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.774 7f4dd7bfc700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dd01b0c70 con 0x7f4dd01b0470 2021-02-10T11:31:00.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.774 7f4dd7bfc700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4dd00f0b50 con 0x7f4dd01b0870 2021-02-10T11:31:00.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.774 7f4dd7bfc700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dd00f0820 con 0x7f4dd00efe00 2021-02-10T11:31:00.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.774 7f4dd5a18700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7f4dd01b0470 0x7f4dd00ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:40724/0 (socket says 172.21.2.112:40724) 2021-02-10T11:31:00.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.775 7f4dd5a18700 1 -- 172.21.2.112:0/803288498 learned_addr learned my addr 172.21.2.112:0/803288498 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:31:00.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.775 7f4dd6219700 1 --2- 172.21.2.112:0/803288498 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4dd00efe00 0x7f4dd00f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:00.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.775 7f4dd5217700 1 --2- 172.21.2.112:0/803288498 >> v2:172.21.2.120:3300/0 conn(0x7f4dd01b0870 0x7f4dd00f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:00.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.775 7f4dd4a16700 1 -- 172.21.2.112:0/803288498 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3692326460 0 0) 0x7f4dc40010b0 con 0x7f4dd01b0470 2021-02-10T11:31:00.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.775 7f4dd6219700 1 -- 172.21.2.112:0/803288498 >> v2:172.21.2.120:3300/0 conn(0x7f4dd01b0870 msgr2=0x7f4dd00f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:00.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.775 7f4dd6219700 1 --2- 172.21.2.112:0/803288498 >> v2:172.21.2.120:3300/0 conn(0x7f4dd01b0870 0x7f4dd00f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:00.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.775 7f4dd6219700 1 -- 172.21.2.112:0/803288498 >> v1:172.21.2.120:6789/0 conn(0x7f4dd01b0470 legacy=0x7f4dd00ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:00.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.775 7f4dd6219700 1 -- 172.21.2.112:0/803288498 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dd00e52e0 con 0x7f4dd00efe00 2021-02-10T11:31:00.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.775 7f4dd6219700 1 --2- 172.21.2.112:0/803288498 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4dd00efe00 0x7f4dd00f0200 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f4dc8006e50 tx=0x7f4dc8009e70).ready entity=mon.2 client_cookie=29157388da52ec65 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:00.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.775 7f4dd4a16700 1 -- 172.21.2.112:0/803288498 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4dc8006a20 con 0x7f4dd00efe00 2021-02-10T11:31:00.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.775 7f4dd4a16700 1 -- 172.21.2.112:0/803288498 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dc8006790 con 0x7f4dd00efe00 2021-02-10T11:31:00.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.775 7f4dd4a16700 1 -- 172.21.2.112:0/803288498 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4dc800b700 con 0x7f4dd00efe00 2021-02-10T11:31:00.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.776 7f4dd7bfc700 1 -- 172.21.2.112:0/803288498 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4dd00efe00 msgr2=0x7f4dd00f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:00.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.776 7f4dd7bfc700 1 --2- 172.21.2.112:0/803288498 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4dd00efe00 0x7f4dd00f0200 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f4dc8006e50 tx=0x7f4dc8009e70).stop 2021-02-10T11:31:00.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.776 7f4dd7bfc700 1 -- 172.21.2.112:0/803288498 shutdown_connections 2021-02-10T11:31:00.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.776 7f4dd7bfc700 1 --2- 172.21.2.112:0/803288498 >> v2:172.21.2.120:3300/0 conn(0x7f4dd01b0870 0x7f4dd00f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:00.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.776 7f4dd7bfc700 1 --2- 172.21.2.112:0/803288498 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4dd00efe00 0x7f4dd00f0200 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:00.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.776 7f4dd7bfc700 1 -- 172.21.2.112:0/803288498 >> 172.21.2.112:0/803288498 conn(0x7f4dd00ce3e0 msgr2=0x7f4dd00e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:00.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.776 7f4dd7bfc700 1 -- 172.21.2.112:0/803288498 shutdown_connections 2021-02-10T11:31:00.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.776 7f4dd7bfc700 1 -- 172.21.2.112:0/803288498 wait complete. 2021-02-10T11:31:00.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.791 7f4dd7bfc700 1 Processor -- start 2021-02-10T11:31:00.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.791 7f4dd7bfc700 1 -- start start 2021-02-10T11:31:00.782 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.791 7f4dd7bfc700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4dd008ece0 0x7f4dd00a6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:00.782 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.791 7f4dd7bfc700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4dd00a6860 0x7f4dd00a6c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:00.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.791 7f4dd7bfc700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4dd00a7280 0x7f4dd00a4590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:00.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.791 7f4dd7bfc700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dd00a5cc0 con 0x7f4dd00a7280 2021-02-10T11:31:00.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.791 7f4dd7bfc700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dd008f0e0 con 0x7f4dd00a6860 2021-02-10T11:31:00.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.791 7f4dd7bfc700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dd008f2c0 con 0x7f4dd008ece0 2021-02-10T11:31:00.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.791 7f4dd6219700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4dd00a7280 0x7f4dd00a4590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:00.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.791 7f4dd6219700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4dd00a7280 0x7f4dd00a4590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:47714/0 (socket says 172.21.2.112:47714) 2021-02-10T11:31:00.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.791 7f4dd6219700 1 -- 172.21.2.112:0/3369031827 learned_addr learned my addr 172.21.2.112:0/3369031827 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:31:00.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.792 7f4dd5217700 1 --2- 172.21.2.112:0/3369031827 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4dd00a6860 0x7f4dd00a6c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:00.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.792 7f4dd5a18700 1 --2- 172.21.2.112:0/3369031827 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4dd008ece0 0x7f4dd00a6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:00.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.792 7f4dd6219700 1 -- 172.21.2.112:0/3369031827 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4dd008ece0 msgr2=0x7f4dd00a6240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:00.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.792 7f4dd6219700 1 --2- 172.21.2.112:0/3369031827 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4dd008ece0 0x7f4dd00a6240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:00.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.792 7f4dd6219700 1 -- 172.21.2.112:0/3369031827 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4dd00a6860 msgr2=0x7f4dd00a6c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:00.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.792 7f4dd6219700 1 --2- 172.21.2.112:0/3369031827 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4dd00a6860 0x7f4dd00a6c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:00.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.792 7f4dd6219700 1 -- 172.21.2.112:0/3369031827 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dd00a4bb0 con 0x7f4dd00a7280 2021-02-10T11:31:00.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.792 7f4dd5a18700 1 --2- 172.21.2.112:0/3369031827 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4dd008ece0 0x7f4dd00a6240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:31:00.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.792 7f4dd5217700 1 --2- 172.21.2.112:0/3369031827 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4dd00a6860 0x7f4dd00a6c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:31:00.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.792 7f4dd6219700 1 --2- 172.21.2.112:0/3369031827 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4dd00a7280 0x7f4dd00a4590 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 rx=0x7f4dc8006e60 tx=0x7f4dc800bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:00.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.792 7f4dc27fc700 1 -- 172.21.2.112:0/3369031827 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4dc8009130 con 0x7f4dd00a7280 2021-02-10T11:31:00.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.792 7f4dc27fc700 1 -- 172.21.2.112:0/3369031827 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dc8000e80 con 0x7f4dd00a7280 2021-02-10T11:31:00.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.792 7f4dc27fc700 1 -- 172.21.2.112:0/3369031827 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4dc80013d0 con 0x7f4dd00a7280 2021-02-10T11:31:00.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.792 7f4dd7bfc700 1 -- 172.21.2.112:0/3369031827 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4dd00a4bb0 con 0x7f4dd00a7280 2021-02-10T11:31:00.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.792 7f4dd7bfc700 1 -- 172.21.2.112:0/3369031827 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4dd00a4bb0 con 0x7f4dd00a7280 2021-02-10T11:31:00.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.793 7f4dc27fc700 1 -- 172.21.2.112:0/3369031827 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4dc8009130 con 0x7f4dd00a7280 2021-02-10T11:31:00.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.793 7f4dc27fc700 1 --2- 172.21.2.112:0/3369031827 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4db80186c0 0x7f4db801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:00.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.793 7f4dc27fc700 1 -- 172.21.2.112:0/3369031827 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f4dc8032b10 con 0x7f4dd00a7280 2021-02-10T11:31:00.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.793 7f4dd5a18700 1 --2- 172.21.2.112:0/3369031827 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4db80186c0 0x7f4db801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:00.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.794 7f4dd5a18700 1 --2- 172.21.2.112:0/3369031827 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4db80186c0 0x7f4db801abc0 crc :-1 s=READY pgs=473 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:00.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.823 7f4dd7bfc700 1 -- 172.21.2.112:0/3369031827 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4dd019d1a0 con 0x7f4dd00a7280 2021-02-10T11:31:00.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.826 7f4dc27fc700 1 -- 172.21.2.112:0/3369031827 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4dc800be30 con 0x7f4dd00a7280 2021-02-10T11:31:00.941 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.953 7f4dd7bfc700 1 -- 172.21.2.112:0/3369031827 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348652", "target": ["mgr", ""]}) v1 -- 0x7f4dd019d1a0 con 0x7f4db80186c0 2021-02-10T11:31:00.951 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.964 7f4dc27fc700 1 -- 172.21.2.112:0/3369031827 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f4dc4001600 con 0x7f4db80186c0 2021-02-10T11:31:00.958 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.970 7f4dd7bfc700 1 -- 172.21.2.112:0/3369031827 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4db80186c0 msgr2=0x7f4db801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:00.958 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.970 7f4dd7bfc700 1 --2- 172.21.2.112:0/3369031827 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4db80186c0 0x7f4db801abc0 crc :-1 s=READY pgs=473 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:00.958 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.970 7f4dd7bfc700 1 -- 172.21.2.112:0/3369031827 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4dd00a7280 msgr2=0x7f4dd00a4590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:00.958 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.970 7f4dd7bfc700 1 --2- 172.21.2.112:0/3369031827 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4dd00a7280 0x7f4dd00a4590 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 rx=0x7f4dc8006e60 tx=0x7f4dc800bcf0).stop 2021-02-10T11:31:00.958 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.970 7f4dd7bfc700 1 -- 172.21.2.112:0/3369031827 shutdown_connections 2021-02-10T11:31:00.958 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.970 7f4dd7bfc700 1 --2- 172.21.2.112:0/3369031827 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4db80186c0 0x7f4db801abc0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:00.958 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.970 7f4dd7bfc700 1 --2- 172.21.2.112:0/3369031827 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4dd00a6860 0x7f4dd00a6c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:00.958 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.970 7f4dd7bfc700 1 --2- 172.21.2.112:0/3369031827 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4dd00a7280 0x7f4dd00a4590 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:00.958 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.970 7f4dd7bfc700 1 --2- 172.21.2.112:0/3369031827 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4dd008ece0 0x7f4dd00a6240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:00.958 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.970 7f4dd7bfc700 1 -- 172.21.2.112:0/3369031827 >> 172.21.2.112:0/3369031827 conn(0x7f4dd008ffe0 msgr2=0x7f4dd00bf050 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:00.958 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.970 7f4dd7bfc700 1 -- 172.21.2.112:0/3369031827 shutdown_connections 2021-02-10T11:31:00.958 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:00.970 7f4dd7bfc700 1 -- 172.21.2.112:0/3369031827 wait complete. 2021-02-10T11:31:00.966 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000348653 2021-02-10T11:31:01.395 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:01.390 7f892852f700 1 Processor -- start 2021-02-10T11:31:01.395 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:01.391 7f892852f700 1 -- start start 2021-02-10T11:31:01.395 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:01.391 7f892852f700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f89201b0470 0x7f89201b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:01.395 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:01.391 7f892852f700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89200f0b50 con 0x7f89200ef610 2021-02-10T11:31:01.395 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:01.391 7f892852f700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89200f24e0 con 0x7f89200f20e0 2021-02-10T11:31:01.395 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:01.391 7f892852f700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89200e5ea0 con 0x7f89201b0470 2021-02-10T11:31:01.395 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:01.391 7f8926b4c700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7f89200f20e0 0x7f89200f0200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:40738/0 (socket says 172.21.2.112:40738) 2021-02-10T11:31:01.395 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:01.391 7f892634b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f89201b0470 0x7f89201b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:01.395 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:01.391 7f8926b4c700 1 -- 172.21.2.112:0/2341896746 learned_addr learned my addr 172.21.2.112:0/2341896746 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:31:01.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:01.391 7f8925b4a700 1 --1- 172.21.2.112:0/2341896746 >> v1:172.21.2.112:6789/0 conn(0x7f89200ef610 0x7f89200efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:47180/0 (socket says 172.21.2.112:47180) 2021-02-10T11:31:01.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:01.392 7f8925349700 1 -- 172.21.2.112:0/2341896746 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 360872621 0 0) 0x7f891c0010b0 con 0x7f89200ef610 2021-02-10T11:31:01.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:01.392 7f8925349700 1 -- 172.21.2.112:0/2341896746 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f890c002fe0 con 0x7f89200ef610 2021-02-10T11:31:01.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:01.392 7f892634b700 1 -- 172.21.2.112:0/2341896746 >> v1:172.21.2.120:6789/0 conn(0x7f89200f20e0 legacy=0x7f89200f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:01.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:01.392 7f892634b700 1 -- 172.21.2.112:0/2341896746 >> v1:172.21.2.112:6789/0 conn(0x7f89200ef610 legacy=0x7f89200efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:01.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:01.392 7f892634b700 1 -- 172.21.2.112:0/2341896746 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89200e52e0 con 0x7f89201b0470 2021-02-10T11:31:01.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:01.392 7f892634b700 1 --2- 172.21.2.112:0/2341896746 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f89201b0470 0x7f89201b0870 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f8914006650 tx=0x7f8914009a80).ready entity=mon.2 client_cookie=ec79cd0eaf2983a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:01.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:01.392 7f8925349700 1 -- 172.21.2.112:0/2341896746 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f89140009b0 con 0x7f89201b0470 2021-02-10T11:31:01.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:01.392 7f8925349700 1 -- 172.21.2.112:0/2341896746 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f891400c0c0 con 0x7f89201b0470 2021-02-10T11:31:01.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:01.392 7f8925349700 1 -- 172.21.2.112:0/2341896746 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f891400c5e0 con 0x7f89201b0470 2021-02-10T11:31:01.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:01.392 7f892852f700 1 -- 172.21.2.112:0/2341896746 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f89201b0470 msgr2=0x7f89201b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:01.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:01.392 7f892852f700 1 --2- 172.21.2.112:0/2341896746 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f89201b0470 0x7f89201b0870 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f8914006650 tx=0x7f8914009a80).stop 2021-02-10T11:31:01.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:01.393 7f892852f700 1 -- 172.21.2.112:0/2341896746 shutdown_connections 2021-02-10T11:31:01.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:01.393 7f892852f700 1 --2- 172.21.2.112:0/2341896746 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f89201b0470 0x7f89201b0870 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:01.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:01.393 7f892852f700 1 -- 172.21.2.112:0/2341896746 >> 172.21.2.112:0/2341896746 conn(0x7f89200ce3e0 msgr2=0x7f89200e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:01.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:01.393 7f892852f700 1 -- 172.21.2.112:0/2341896746 shutdown_connections 2021-02-10T11:31:01.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:01.393 7f892852f700 1 -- 172.21.2.112:0/2341896746 wait complete. 2021-02-10T11:31:01.399 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:01.408 7f892852f700 1 Processor -- start 2021-02-10T11:31:01.399 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:01.408 7f892852f700 1 -- start start 2021-02-10T11:31:01.399 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:01.408 7f892852f700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f89200c36a0 0x7f89200c2b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:01.399 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:01.408 7f892852f700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f89200c3190 0x7f89200b1d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:01.399 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:01.408 7f892852f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f89200b2370 0x7f89200b2d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:01.399 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:01.408 7f892852f700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89200b27c0 con 0x7f89200b2370 2021-02-10T11:31:01.399 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:01.408 7f892852f700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89200b3360 con 0x7f89200c36a0 2021-02-10T11:31:01.399 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:01.408 7f892852f700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89200b3540 con 0x7f89200c3190 2021-02-10T11:31:01.399 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:01.408 7f892634b700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f89200c36a0 0x7f89200c2b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:01.399 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:01.408 7f892634b700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f89200c36a0 0x7f89200c2b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3300/0 says I am v2:172.21.2.112:50004/0 (socket says 172.21.2.112:50004) 2021-02-10T11:31:01.400 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:01.409 7f8925b4a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f89200c3190 0x7f89200b1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:01.400 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:01.409 7f892634b700 1 -- 172.21.2.112:0/3812208590 learned_addr learned my addr 172.21.2.112:0/3812208590 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:31:01.400 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:01.409 7f8926b4c700 1 --2- 172.21.2.112:0/3812208590 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f89200b2370 0x7f89200b2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:01.400 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:01.409 7f892634b700 1 -- 172.21.2.112:0/3812208590 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f89200c3190 msgr2=0x7f89200b1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:01.400 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:01.409 7f892634b700 1 --2- 172.21.2.112:0/3812208590 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f89200c3190 0x7f89200b1d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:01.400 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:01.409 7f892634b700 1 -- 172.21.2.112:0/3812208590 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f89200b2370 msgr2=0x7f89200b2d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:01.400 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:01.409 7f892634b700 1 --2- 172.21.2.112:0/3812208590 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f89200b2370 0x7f89200b2d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:01.400 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:01.409 7f892634b700 1 -- 172.21.2.112:0/3812208590 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89200b3760 con 0x7f89200c36a0 2021-02-10T11:31:01.400 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:01.409 7f8926b4c700 1 --2- 172.21.2.112:0/3812208590 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f89200b2370 0x7f89200b2d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:31:01.400 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:01.409 7f892634b700 1 --2- 172.21.2.112:0/3812208590 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f89200c36a0 0x7f89200c2b70 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 rx=0x7f8914006fc0 tx=0x7f891400b8e0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:01.401 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:01.409 7f8912ffd700 1 -- 172.21.2.112:0/3812208590 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8914002030 con 0x7f89200c36a0 2021-02-10T11:31:01.401 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:01.409 7f8912ffd700 1 -- 172.21.2.112:0/3812208590 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89140022b0 con 0x7f89200c36a0 2021-02-10T11:31:01.401 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:01.409 7f892852f700 1 -- 172.21.2.112:0/3812208590 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89200b3760 con 0x7f89200c36a0 2021-02-10T11:31:01.401 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:01.409 7f892852f700 1 -- 172.21.2.112:0/3812208590 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89200b3760 con 0x7f89200c36a0 2021-02-10T11:31:01.401 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:01.409 7f8912ffd700 1 -- 172.21.2.112:0/3812208590 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f89140027d0 con 0x7f89200c36a0 2021-02-10T11:31:01.401 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:01.410 7f8912ffd700 1 -- 172.21.2.112:0/3812208590 <== mon.1 v2:172.21.2.120:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8914009300 con 0x7f89200c36a0 2021-02-10T11:31:01.401 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:01.410 7f8912ffd700 1 --2- 172.21.2.112:0/3812208590 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f89080186d0 0x7f890801abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:01.401 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:01.410 7f8912ffd700 1 -- 172.21.2.112:0/3812208590 <== mon.1 v2:172.21.2.120:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f8914032770 con 0x7f89200c36a0 2021-02-10T11:31:01.401 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:01.410 7f8925b4a700 1 --2- 172.21.2.112:0/3812208590 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f89080186d0 0x7f890801abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:01.401 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:01.411 7f8925b4a700 1 --2- 172.21.2.112:0/3812208590 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f89080186d0 0x7f890801abd0 crc :-1 s=READY pgs=474 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:01.428 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:01.441 7f892852f700 1 -- 172.21.2.112:0/3812208590 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89200edd10 con 0x7f89200c36a0 2021-02-10T11:31:01.431 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:01.444 7f8912ffd700 1 -- 172.21.2.112:0/3812208590 <== mon.1 v2:172.21.2.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8914002020 con 0x7f89200c36a0 2021-02-10T11:31:01.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:01.573 7f892852f700 1 -- 172.21.2.112:0/3812208590 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348653", "target": ["mgr", ""]}) v1 -- 0x7f89200edd10 con 0x7f89080186d0 2021-02-10T11:31:01.570 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:01.583 7f8912ffd700 1 -- 172.21.2.112:0/3812208590 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f891c000ec0 con 0x7f89080186d0 2021-02-10T11:31:01.579 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:01.591 7f892852f700 1 -- 172.21.2.112:0/3812208590 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f89080186d0 msgr2=0x7f890801abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:01.579 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:01.591 7f892852f700 1 --2- 172.21.2.112:0/3812208590 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f89080186d0 0x7f890801abd0 crc :-1 s=READY pgs=474 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:01.579 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:01.591 7f892852f700 1 -- 172.21.2.112:0/3812208590 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f89200c36a0 msgr2=0x7f89200c2b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:01.579 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:01.591 7f892852f700 1 --2- 172.21.2.112:0/3812208590 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f89200c36a0 0x7f89200c2b70 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 rx=0x7f8914006fc0 tx=0x7f891400b8e0).stop 2021-02-10T11:31:01.579 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:01.591 7f892852f700 1 -- 172.21.2.112:0/3812208590 shutdown_connections 2021-02-10T11:31:01.579 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:01.591 7f892852f700 1 --2- 172.21.2.112:0/3812208590 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f89080186d0 0x7f890801abd0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:01.579 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:01.591 7f892852f700 1 --2- 172.21.2.112:0/3812208590 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f89200b2370 0x7f89200b2d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:01.579 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:01.591 7f892852f700 1 --2- 172.21.2.112:0/3812208590 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f89200c3190 0x7f89200b1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:01.579 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:01.591 7f892852f700 1 --2- 172.21.2.112:0/3812208590 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f89200c36a0 0x7f89200c2b70 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:01.579 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:01.591 7f892852f700 1 -- 172.21.2.112:0/3812208590 >> 172.21.2.112:0/3812208590 conn(0x7f892009f630 msgr2=0x7f89200a0130 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:01.580 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:01.591 7f892852f700 1 -- 172.21.2.112:0/3812208590 shutdown_connections 2021-02-10T11:31:01.580 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:01.591 7f892852f700 1 -- 172.21.2.112:0/3812208590 wait complete. 2021-02-10T11:31:01.588 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000348654 2021-02-10T11:31:02.019 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.013 7f28a37cf700 1 Processor -- start 2021-02-10T11:31:02.019 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.013 7f28a37cf700 1 -- start start 2021-02-10T11:31:02.019 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.014 7f28a37cf700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f289c1b0470 0x7f289c1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:02.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.014 7f28a37cf700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f289c0ef610 0x7f289c0efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:02.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.014 7f28a37cf700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f289c0f0b50 con 0x7f289c0f0030 2021-02-10T11:31:02.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.014 7f28a37cf700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f289c0e52e0 con 0x7f289c0ef610 2021-02-10T11:31:02.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.014 7f28a37cf700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f289c0e5490 con 0x7f289c1b0470 2021-02-10T11:31:02.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.014 7f28a0dea700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f289c0ef610 0x7f289c0efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:02.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.014 7f28a0dea700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f289c0ef610 0x7f289c0efa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3300/0 says I am v2:172.21.2.112:50012/0 (socket says 172.21.2.112:50012) 2021-02-10T11:31:02.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.014 7f28a0dea700 1 -- 172.21.2.112:0/2920648136 learned_addr learned my addr 172.21.2.112:0/2920648136 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:31:02.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.014 7f28a15eb700 1 --2- 172.21.2.112:0/2920648136 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f289c1b0470 0x7f289c1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:02.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.014 7f28a1dec700 1 --1- 172.21.2.112:0/2920648136 >> v1:172.21.2.120:6789/0 conn(0x7f289c0f0030 0x7f289c0f10d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:40756/0 (socket says 172.21.2.112:40756) 2021-02-10T11:31:02.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.015 7f28a0dea700 1 -- 172.21.2.112:0/2920648136 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f289c1b0470 msgr2=0x7f289c1b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:02.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.015 7f28a0dea700 1 --2- 172.21.2.112:0/2920648136 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f289c1b0470 0x7f289c1b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:02.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.015 7f28a0dea700 1 -- 172.21.2.112:0/2920648136 >> v1:172.21.2.120:6789/0 conn(0x7f289c0f0030 legacy=0x7f289c0f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:02.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.015 7f28a0dea700 1 -- 172.21.2.112:0/2920648136 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f289c0f0430 con 0x7f289c0ef610 2021-02-10T11:31:02.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.015 7f28a15eb700 1 --2- 172.21.2.112:0/2920648136 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f289c1b0470 0x7f289c1b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:31:02.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.015 7f28a0dea700 1 --2- 172.21.2.112:0/2920648136 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f289c0ef610 0x7f289c0efa10 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 rx=0x7f2890006700 tx=0x7f2890009b80).ready entity=mon.1 client_cookie=3c475ea2720f439a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:02.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.015 7f289bfff700 1 -- 172.21.2.112:0/2920648136 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2890006340 con 0x7f289c0ef610 2021-02-10T11:31:02.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.015 7f289bfff700 1 -- 172.21.2.112:0/2920648136 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f289000c0c0 con 0x7f289c0ef610 2021-02-10T11:31:02.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.015 7f289bfff700 1 -- 172.21.2.112:0/2920648136 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f289000c510 con 0x7f289c0ef610 2021-02-10T11:31:02.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.015 7f28a37cf700 1 -- 172.21.2.112:0/2920648136 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f289c0ef610 msgr2=0x7f289c0efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:02.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.015 7f28a37cf700 1 --2- 172.21.2.112:0/2920648136 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f289c0ef610 0x7f289c0efa10 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 rx=0x7f2890006700 tx=0x7f2890009b80).stop 2021-02-10T11:31:02.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.016 7f28a37cf700 1 -- 172.21.2.112:0/2920648136 shutdown_connections 2021-02-10T11:31:02.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.016 7f28a37cf700 1 --2- 172.21.2.112:0/2920648136 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f289c0ef610 0x7f289c0efa10 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:02.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.016 7f28a37cf700 1 --2- 172.21.2.112:0/2920648136 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f289c1b0470 0x7f289c1b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:02.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.016 7f28a37cf700 1 -- 172.21.2.112:0/2920648136 >> 172.21.2.112:0/2920648136 conn(0x7f289c0ce3e0 msgr2=0x7f289c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:02.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.016 7f28a37cf700 1 -- 172.21.2.112:0/2920648136 shutdown_connections 2021-02-10T11:31:02.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.016 7f28a37cf700 1 -- 172.21.2.112:0/2920648136 wait complete. 2021-02-10T11:31:02.025 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.032 7f28a37cf700 1 Processor -- start 2021-02-10T11:31:02.025 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.032 7f28a37cf700 1 -- start start 2021-02-10T11:31:02.025 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.033 7f28a37cf700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f289c0a01f0 0x7f289c09f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:02.025 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.033 7f28a37cf700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f289c0add60 0x7f289c09e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:02.025 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.033 7f28a37cf700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f289c09edf0 0x7f289c08c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:02.025 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.033 7f28a37cf700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f289c09fe20 con 0x7f289c09edf0 2021-02-10T11:31:02.025 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.033 7f28a37cf700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f289c0ae160 con 0x7f289c0add60 2021-02-10T11:31:02.025 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.033 7f28a37cf700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f289c0ae380 con 0x7f289c0a01f0 2021-02-10T11:31:02.025 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.033 7f28a1dec700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f289c09edf0 0x7f289c08c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:02.025 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.033 7f28a1dec700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f289c09edf0 0x7f289c08c680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:47742/0 (socket says 172.21.2.112:47742) 2021-02-10T11:31:02.026 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.033 7f28a1dec700 1 -- 172.21.2.112:0/1141099730 learned_addr learned my addr 172.21.2.112:0/1141099730 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:31:02.026 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.033 7f28a0dea700 1 --2- 172.21.2.112:0/1141099730 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f289c0add60 0x7f289c09e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:02.026 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.033 7f28a15eb700 1 --2- 172.21.2.112:0/1141099730 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f289c0a01f0 0x7f289c09f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:02.026 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.033 7f28a1dec700 1 -- 172.21.2.112:0/1141099730 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f289c0a01f0 msgr2=0x7f289c09f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:02.026 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.033 7f28a1dec700 1 --2- 172.21.2.112:0/1141099730 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f289c0a01f0 0x7f289c09f800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:02.026 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.033 7f28a1dec700 1 -- 172.21.2.112:0/1141099730 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f289c0add60 msgr2=0x7f289c09e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:02.026 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.033 7f28a1dec700 1 --2- 172.21.2.112:0/1141099730 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f289c0add60 0x7f289c09e7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:02.026 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.033 7f28a1dec700 1 -- 172.21.2.112:0/1141099730 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f289c08cca0 con 0x7f289c09edf0 2021-02-10T11:31:02.026 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.033 7f28a0dea700 1 --2- 172.21.2.112:0/1141099730 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f289c0add60 0x7f289c09e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:31:02.026 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.033 7f28a1dec700 1 --2- 172.21.2.112:0/1141099730 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f289c09edf0 0x7f289c08c680 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 rx=0x7f289400a8e0 tx=0x7f289400ac80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:02.026 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.033 7f2899ffb700 1 -- 172.21.2.112:0/1141099730 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2894007020 con 0x7f289c09edf0 2021-02-10T11:31:02.026 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.033 7f2899ffb700 1 -- 172.21.2.112:0/1141099730 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28940072a0 con 0x7f289c09edf0 2021-02-10T11:31:02.026 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.033 7f28a37cf700 1 -- 172.21.2.112:0/1141099730 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f289c08cca0 con 0x7f289c09edf0 2021-02-10T11:31:02.027 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.033 7f2899ffb700 1 -- 172.21.2.112:0/1141099730 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2894004920 con 0x7f289c09edf0 2021-02-10T11:31:02.027 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.033 7f28a37cf700 1 -- 172.21.2.112:0/1141099730 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f289c08cca0 con 0x7f289c09edf0 2021-02-10T11:31:02.027 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.034 7f2899ffb700 1 -- 172.21.2.112:0/1141099730 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f289400adc0 con 0x7f289c09edf0 2021-02-10T11:31:02.027 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.034 7f2899ffb700 1 --2- 172.21.2.112:0/1141099730 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f287c018730 0x7f287c01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:02.027 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.034 7f2899ffb700 1 -- 172.21.2.112:0/1141099730 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f2894032a30 con 0x7f289c09edf0 2021-02-10T11:31:02.027 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.034 7f28a15eb700 1 --2- 172.21.2.112:0/1141099730 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f287c018730 0x7f287c01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:02.027 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.034 7f28a15eb700 1 --2- 172.21.2.112:0/1141099730 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f287c018730 0x7f287c01ac10 crc :-1 s=READY pgs=475 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:02.051 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.063 7f28a37cf700 1 -- 172.21.2.112:0/1141099730 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f289c105ae0 con 0x7f289c09edf0 2021-02-10T11:31:02.053 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.066 7f2899ffb700 1 -- 172.21.2.112:0/1141099730 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2894000f50 con 0x7f289c09edf0 2021-02-10T11:31:02.183 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.196 7f28a37cf700 1 -- 172.21.2.112:0/1141099730 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348654", "target": ["mgr", ""]}) v1 -- 0x7f289c0f0b50 con 0x7f287c018730 2021-02-10T11:31:02.194 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.207 7f2899ffb700 1 -- 172.21.2.112:0/1141099730 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f288c006a30 con 0x7f287c018730 2021-02-10T11:31:02.200 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.213 7f28a37cf700 1 -- 172.21.2.112:0/1141099730 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f287c018730 msgr2=0x7f287c01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:02.200 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.213 7f28a37cf700 1 --2- 172.21.2.112:0/1141099730 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f287c018730 0x7f287c01ac10 crc :-1 s=READY pgs=475 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:02.201 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.213 7f28a37cf700 1 -- 172.21.2.112:0/1141099730 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f289c09edf0 msgr2=0x7f289c08c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:02.201 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.213 7f28a37cf700 1 --2- 172.21.2.112:0/1141099730 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f289c09edf0 0x7f289c08c680 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 rx=0x7f289400a8e0 tx=0x7f289400ac80).stop 2021-02-10T11:31:02.201 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.213 7f28a37cf700 1 -- 172.21.2.112:0/1141099730 shutdown_connections 2021-02-10T11:31:02.201 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.213 7f28a37cf700 1 --2- 172.21.2.112:0/1141099730 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f287c018730 0x7f287c01ac10 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:02.201 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.213 7f28a37cf700 1 --2- 172.21.2.112:0/1141099730 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f289c0add60 0x7f289c09e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:02.201 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.213 7f28a37cf700 1 --2- 172.21.2.112:0/1141099730 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f289c09edf0 0x7f289c08c680 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:02.201 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.213 7f28a37cf700 1 --2- 172.21.2.112:0/1141099730 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f289c0a01f0 0x7f289c09f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:02.201 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.213 7f28a37cf700 1 -- 172.21.2.112:0/1141099730 >> 172.21.2.112:0/1141099730 conn(0x7f289c0bbf60 msgr2=0x7f289c105d10 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:02.201 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.213 7f28a37cf700 1 -- 172.21.2.112:0/1141099730 shutdown_connections 2021-02-10T11:31:02.201 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.213 7f28a37cf700 1 -- 172.21.2.112:0/1141099730 wait complete. 2021-02-10T11:31:02.209 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000348655 2021-02-10T11:31:02.639 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.635 7fcdfa7b0700 1 Processor -- start 2021-02-10T11:31:02.640 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.635 7fcdfa7b0700 1 -- start start 2021-02-10T11:31:02.640 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.636 7fcdfa7b0700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcdf41b0470 0x7fcdf41b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:02.640 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.636 7fcdfa7b0700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdf40f0b50 con 0x7fcdf40ef610 2021-02-10T11:31:02.640 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.636 7fcdfa7b0700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdf40f24e0 con 0x7fcdf40f20e0 2021-02-10T11:31:02.640 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.636 7fcdfa7b0700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdf40e5ea0 con 0x7fcdf41b0470 2021-02-10T11:31:02.640 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.636 7fcdf37fe700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fcdf40ef610 0x7fcdf40efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:47208/0 (socket says 172.21.2.112:47208) 2021-02-10T11:31:02.641 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.636 7fcdf37fe700 1 -- 172.21.2.112:0/1337298971 learned_addr learned my addr 172.21.2.112:0/1337298971 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:31:02.641 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.636 7fcdf3fff700 1 --2- 172.21.2.112:0/1337298971 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcdf41b0470 0x7fcdf41b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:02.641 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.636 7fcdf2ffd700 1 -- 172.21.2.112:0/1337298971 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2612659041 0 0) 0x7fcde80010b0 con 0x7fcdf40ef610 2021-02-10T11:31:02.641 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.636 7fcdf2ffd700 1 -- 172.21.2.112:0/1337298971 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcddc002fe0 con 0x7fcdf40ef610 2021-02-10T11:31:02.641 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.637 7fcdf3fff700 1 -- 172.21.2.112:0/1337298971 >> v1:172.21.2.120:6789/0 conn(0x7fcdf40f20e0 legacy=0x7fcdf40f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:02.641 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.637 7fcdf3fff700 1 -- 172.21.2.112:0/1337298971 >> v1:172.21.2.112:6789/0 conn(0x7fcdf40ef610 legacy=0x7fcdf40efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:02.641 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.637 7fcdf3fff700 1 -- 172.21.2.112:0/1337298971 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdf40e52e0 con 0x7fcdf41b0470 2021-02-10T11:31:02.641 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.637 7fcdf3fff700 1 --2- 172.21.2.112:0/1337298971 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcdf41b0470 0x7fcdf41b0870 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7fcde4006650 tx=0x7fcde4009a80).ready entity=mon.2 client_cookie=69230a7b527faca4 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:02.641 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.637 7fcdf2ffd700 1 -- 172.21.2.112:0/1337298971 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcde400c0c0 con 0x7fcdf41b0470 2021-02-10T11:31:02.641 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.637 7fcdf2ffd700 1 -- 172.21.2.112:0/1337298971 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcde400c340 con 0x7fcdf41b0470 2021-02-10T11:31:02.641 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.637 7fcdf2ffd700 1 -- 172.21.2.112:0/1337298971 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcde40009b0 con 0x7fcdf41b0470 2021-02-10T11:31:02.642 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.637 7fcdfa7b0700 1 -- 172.21.2.112:0/1337298971 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcdf41b0470 msgr2=0x7fcdf41b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:02.642 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.637 7fcdfa7b0700 1 --2- 172.21.2.112:0/1337298971 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcdf41b0470 0x7fcdf41b0870 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7fcde4006650 tx=0x7fcde4009a80).stop 2021-02-10T11:31:02.642 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.638 7fcdfa7b0700 1 -- 172.21.2.112:0/1337298971 shutdown_connections 2021-02-10T11:31:02.642 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.638 7fcdfa7b0700 1 --2- 172.21.2.112:0/1337298971 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcdf41b0470 0x7fcdf41b0870 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:02.642 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.638 7fcdfa7b0700 1 -- 172.21.2.112:0/1337298971 >> 172.21.2.112:0/1337298971 conn(0x7fcdf40ce3e0 msgr2=0x7fcdf40e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:02.642 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.638 7fcdfa7b0700 1 -- 172.21.2.112:0/1337298971 shutdown_connections 2021-02-10T11:31:02.642 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.638 7fcdfa7b0700 1 -- 172.21.2.112:0/1337298971 wait complete. 2021-02-10T11:31:02.642 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.652 7fcdfa7b0700 1 Processor -- start 2021-02-10T11:31:02.642 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.652 7fcdfa7b0700 1 -- start start 2021-02-10T11:31:02.645 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.653 7fcdfa7b0700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcdf41a0570 0x7fcdf40a4590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:02.645 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.653 7fcdfa7b0700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcdf40f1fd0 0x7fcdf40f23d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:02.645 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.653 7fcdfa7b0700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcdf40a5450 0x7fcdf40a5850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:02.645 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.653 7fcdfa7b0700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdf40a4bb0 con 0x7fcdf41a0570 2021-02-10T11:31:02.645 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.653 7fcdfa7b0700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdf40b35e0 con 0x7fcdf40a5450 2021-02-10T11:31:02.645 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.653 7fcdfa7b0700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdf40a1f00 con 0x7fcdf40f1fd0 2021-02-10T11:31:02.645 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.653 7fcdf3fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcdf41a0570 0x7fcdf40a4590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:02.645 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.653 7fcdf3fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcdf41a0570 0x7fcdf40a4590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:47752/0 (socket says 172.21.2.112:47752) 2021-02-10T11:31:02.645 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.653 7fcdf3fff700 1 -- 172.21.2.112:0/3263018866 learned_addr learned my addr 172.21.2.112:0/3263018866 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:31:02.645 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.653 7fcdf8dcd700 1 --2- 172.21.2.112:0/3263018866 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcdf40a5450 0x7fcdf40a5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:02.646 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.653 7fcdf37fe700 1 --2- 172.21.2.112:0/3263018866 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcdf40f1fd0 0x7fcdf40f23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:02.646 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.653 7fcdf3fff700 1 -- 172.21.2.112:0/3263018866 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcdf40f1fd0 msgr2=0x7fcdf40f23d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:02.646 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.653 7fcdf3fff700 1 --2- 172.21.2.112:0/3263018866 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcdf40f1fd0 0x7fcdf40f23d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:02.646 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.653 7fcdf3fff700 1 -- 172.21.2.112:0/3263018866 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcdf40a5450 msgr2=0x7fcdf40a5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:02.646 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.653 7fcdf3fff700 1 --2- 172.21.2.112:0/3263018866 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcdf40a5450 0x7fcdf40a5850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:02.646 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.653 7fcdf3fff700 1 -- 172.21.2.112:0/3263018866 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdf40a5e70 con 0x7fcdf41a0570 2021-02-10T11:31:02.646 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.653 7fcdf37fe700 1 --2- 172.21.2.112:0/3263018866 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcdf40f1fd0 0x7fcdf40f23d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:31:02.646 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.653 7fcdf3fff700 1 --2- 172.21.2.112:0/3263018866 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcdf41a0570 0x7fcdf40a4590 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 rx=0x7fcde400b660 tx=0x7fcde400ba00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:02.646 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.654 7fcdf0ff9700 1 -- 172.21.2.112:0/3263018866 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcde400bdf0 con 0x7fcdf41a0570 2021-02-10T11:31:02.646 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.654 7fcdf0ff9700 1 -- 172.21.2.112:0/3263018866 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcde40009b0 con 0x7fcdf41a0570 2021-02-10T11:31:02.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.654 7fcdf0ff9700 1 -- 172.21.2.112:0/3263018866 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcde4002030 con 0x7fcdf41a0570 2021-02-10T11:31:02.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.654 7fcdfa7b0700 1 -- 172.21.2.112:0/3263018866 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdf40a5e70 con 0x7fcdf41a0570 2021-02-10T11:31:02.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.654 7fcdfa7b0700 1 -- 172.21.2.112:0/3263018866 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdf40a6400 con 0x7fcdf41a0570 2021-02-10T11:31:02.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.654 7fcdf0ff9700 1 -- 172.21.2.112:0/3263018866 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fcde40052e0 con 0x7fcdf41a0570 2021-02-10T11:31:02.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.654 7fcdf0ff9700 1 --2- 172.21.2.112:0/3263018866 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcdd40186c0 0x7fcdd401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:02.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.654 7fcdf0ff9700 1 -- 172.21.2.112:0/3263018866 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fcde40323e0 con 0x7fcdf41a0570 2021-02-10T11:31:02.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.655 7fcdf37fe700 1 --2- 172.21.2.112:0/3263018866 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcdd40186c0 0x7fcdd401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:02.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.655 7fcdf37fe700 1 --2- 172.21.2.112:0/3263018866 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcdd40186c0 0x7fcdd401abc0 crc :-1 s=READY pgs=476 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:02.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.685 7fcdfa7b0700 1 -- 172.21.2.112:0/3263018866 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdf40e52e0 con 0x7fcdf41a0570 2021-02-10T11:31:02.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.688 7fcdf0ff9700 1 -- 172.21.2.112:0/3263018866 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fcde4030d10 con 0x7fcdf41a0570 2021-02-10T11:31:02.801 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.814 7fcdfa7b0700 1 -- 172.21.2.112:0/3263018866 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348655", "target": ["mgr", ""]}) v1 -- 0x7fcdf40a6400 con 0x7fcdd40186c0 2021-02-10T11:31:02.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.825 7fcdf0ff9700 1 -- 172.21.2.112:0/3263018866 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fcde80009b0 con 0x7fcdd40186c0 2021-02-10T11:31:02.819 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.832 7fcdfa7b0700 1 -- 172.21.2.112:0/3263018866 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcdd40186c0 msgr2=0x7fcdd401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:02.819 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.832 7fcdfa7b0700 1 --2- 172.21.2.112:0/3263018866 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcdd40186c0 0x7fcdd401abc0 crc :-1 s=READY pgs=476 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:02.819 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.832 7fcdfa7b0700 1 -- 172.21.2.112:0/3263018866 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcdf41a0570 msgr2=0x7fcdf40a4590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:02.819 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.832 7fcdfa7b0700 1 --2- 172.21.2.112:0/3263018866 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcdf41a0570 0x7fcdf40a4590 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 rx=0x7fcde400b660 tx=0x7fcde400ba00).stop 2021-02-10T11:31:02.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.832 7fcdfa7b0700 1 -- 172.21.2.112:0/3263018866 shutdown_connections 2021-02-10T11:31:02.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.832 7fcdfa7b0700 1 --2- 172.21.2.112:0/3263018866 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcdd40186c0 0x7fcdd401abc0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:02.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.832 7fcdfa7b0700 1 --2- 172.21.2.112:0/3263018866 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcdf40a5450 0x7fcdf40a5850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:02.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.832 7fcdfa7b0700 1 --2- 172.21.2.112:0/3263018866 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcdf40f1fd0 0x7fcdf40f23d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:02.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.832 7fcdfa7b0700 1 --2- 172.21.2.112:0/3263018866 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcdf41a0570 0x7fcdf40a4590 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:02.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.832 7fcdfa7b0700 1 -- 172.21.2.112:0/3263018866 >> 172.21.2.112:0/3263018866 conn(0x7fcdf40958b0 msgr2=0x7fcdf40bf040 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:02.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.832 7fcdfa7b0700 1 -- 172.21.2.112:0/3263018866 shutdown_connections 2021-02-10T11:31:02.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:02.832 7fcdfa7b0700 1 -- 172.21.2.112:0/3263018866 wait complete. 2021-02-10T11:31:02.828 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000348656 2021-02-10T11:31:03.257 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.253 7f2df1576700 1 Processor -- start 2021-02-10T11:31:03.258 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.253 7f2df1576700 1 -- start start 2021-02-10T11:31:03.258 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.254 7f2df1576700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f2dec1b0870 0x7f2dec0f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:03.258 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.254 7f2df1576700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f2dec0efe00 0x7f2dec0f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:03.258 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.254 7f2df1576700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dec1b0c70 con 0x7f2dec1b0470 2021-02-10T11:31:03.258 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.254 7f2df1576700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2dec0e5ea0 con 0x7f2dec1b0870 2021-02-10T11:31:03.258 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.254 7f2df1576700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2dec0f0b50 con 0x7f2dec0efe00 2021-02-10T11:31:03.258 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.254 7f2debac0700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f2dec0efe00 0x7f2dec0f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:03.258 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.254 7f2deb2bf700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f2dec1b0470 0x7f2dec0ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:47220/0 (socket says 172.21.2.112:47220) 2021-02-10T11:31:03.258 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.254 7f2deb2bf700 1 -- 172.21.2.112:0/1285376165 learned_addr learned my addr 172.21.2.112:0/1285376165 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:31:03.258 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.254 7f2deaabe700 1 --2- 172.21.2.112:0/1285376165 >> v2:172.21.2.112:3300/0 conn(0x7f2dec1b0870 0x7f2dec0f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:03.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.255 7f2debac0700 1 -- 172.21.2.112:0/1285376165 >> v2:172.21.2.112:3300/0 conn(0x7f2dec1b0870 msgr2=0x7f2dec0f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:03.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.255 7f2debac0700 1 --2- 172.21.2.112:0/1285376165 >> v2:172.21.2.112:3300/0 conn(0x7f2dec1b0870 0x7f2dec0f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:03.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.255 7f2debac0700 1 -- 172.21.2.112:0/1285376165 >> v1:172.21.2.112:6789/0 conn(0x7f2dec1b0470 legacy=0x7f2dec0ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:03.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.255 7f2debac0700 1 -- 172.21.2.112:0/1285376165 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dec0f0820 con 0x7f2dec0efe00 2021-02-10T11:31:03.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.255 7f2debac0700 1 --2- 172.21.2.112:0/1285376165 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2dec0efe00 0x7f2dec0f0200 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 rx=0x7f2de4006e50 tx=0x7f2de4009e70).ready entity=mon.1 client_cookie=1b8e6b9658397aec server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:03.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.255 7f2dea2bd700 1 -- 172.21.2.112:0/1285376165 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2de400b510 con 0x7f2dec0efe00 2021-02-10T11:31:03.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.255 7f2dea2bd700 1 -- 172.21.2.112:0/1285376165 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2de4006790 con 0x7f2dec0efe00 2021-02-10T11:31:03.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.255 7f2dea2bd700 1 -- 172.21.2.112:0/1285376165 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2de4000f90 con 0x7f2dec0efe00 2021-02-10T11:31:03.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.255 7f2df1576700 1 -- 172.21.2.112:0/1285376165 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2dec0efe00 msgr2=0x7f2dec0f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:03.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.255 7f2df1576700 1 --2- 172.21.2.112:0/1285376165 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2dec0efe00 0x7f2dec0f0200 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 rx=0x7f2de4006e50 tx=0x7f2de4009e70).stop 2021-02-10T11:31:03.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.256 7f2df1576700 1 -- 172.21.2.112:0/1285376165 shutdown_connections 2021-02-10T11:31:03.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.256 7f2df1576700 1 --2- 172.21.2.112:0/1285376165 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2dec0efe00 0x7f2dec0f0200 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:03.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.256 7f2df1576700 1 --2- 172.21.2.112:0/1285376165 >> v2:172.21.2.112:3300/0 conn(0x7f2dec1b0870 0x7f2dec0f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:03.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.256 7f2df1576700 1 -- 172.21.2.112:0/1285376165 >> 172.21.2.112:0/1285376165 conn(0x7f2dec0ce3e0 msgr2=0x7f2dec0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:03.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.256 7f2df1576700 1 -- 172.21.2.112:0/1285376165 shutdown_connections 2021-02-10T11:31:03.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.256 7f2df1576700 1 -- 172.21.2.112:0/1285376165 wait complete. 2021-02-10T11:31:03.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.271 7f2df1576700 1 Processor -- start 2021-02-10T11:31:03.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.271 7f2df1576700 1 -- start start 2021-02-10T11:31:03.262 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.271 7f2df1576700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2dec190920 0x7f2dec0b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:03.262 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.271 7f2df1576700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2dec0b8200 0x7f2dec0b8600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:03.262 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.271 7f2df1576700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2dec0c1d80 0x7f2dec0c2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:03.262 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.271 7f2df1576700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dec190da0 con 0x7f2dec0c1d80 2021-02-10T11:31:03.262 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.271 7f2df1576700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dec0b8c20 con 0x7f2dec0b8200 2021-02-10T11:31:03.262 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.271 7f2df1576700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2dec0b8e10 con 0x7f2dec190920 2021-02-10T11:31:03.262 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.271 7f2debac0700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2dec0c1d80 0x7f2dec0c2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:03.262 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.271 7f2debac0700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2dec0c1d80 0x7f2dec0c2180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:47762/0 (socket says 172.21.2.112:47762) 2021-02-10T11:31:03.262 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.271 7f2debac0700 1 -- 172.21.2.112:0/2211532837 learned_addr learned my addr 172.21.2.112:0/2211532837 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:31:03.262 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.271 7f2deb2bf700 1 --2- 172.21.2.112:0/2211532837 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2dec190920 0x7f2dec0b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:03.262 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.271 7f2deaabe700 1 --2- 172.21.2.112:0/2211532837 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2dec0b8200 0x7f2dec0b8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:03.262 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.272 7f2debac0700 1 -- 172.21.2.112:0/2211532837 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2dec190920 msgr2=0x7f2dec0b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:03.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.272 7f2debac0700 1 --2- 172.21.2.112:0/2211532837 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2dec190920 0x7f2dec0b7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:03.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.272 7f2debac0700 1 -- 172.21.2.112:0/2211532837 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2dec0b8200 msgr2=0x7f2dec0b8600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:03.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.272 7f2debac0700 1 --2- 172.21.2.112:0/2211532837 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2dec0b8200 0x7f2dec0b8600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:03.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.272 7f2debac0700 1 -- 172.21.2.112:0/2211532837 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dec0c27d0 con 0x7f2dec0c1d80 2021-02-10T11:31:03.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.272 7f2debac0700 1 --2- 172.21.2.112:0/2211532837 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2dec0c1d80 0x7f2dec0c2180 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 rx=0x7f2de4006c50 tx=0x7f2de400bd20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:03.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.272 7f2dd3fff700 1 -- 172.21.2.112:0/2211532837 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2de40093d0 con 0x7f2dec0c1d80 2021-02-10T11:31:03.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.272 7f2dd3fff700 1 -- 172.21.2.112:0/2211532837 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2de400be60 con 0x7f2dec0c1d80 2021-02-10T11:31:03.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.272 7f2df1576700 1 -- 172.21.2.112:0/2211532837 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2dec0c27d0 con 0x7f2dec0c1d80 2021-02-10T11:31:03.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.272 7f2dd3fff700 1 -- 172.21.2.112:0/2211532837 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2de400c570 con 0x7f2dec0c1d80 2021-02-10T11:31:03.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.272 7f2df1576700 1 -- 172.21.2.112:0/2211532837 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2dec0c27d0 con 0x7f2dec0c1d80 2021-02-10T11:31:03.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.273 7f2dd3fff700 1 -- 172.21.2.112:0/2211532837 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2de4002ef0 con 0x7f2dec0c1d80 2021-02-10T11:31:03.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.273 7f2dd3fff700 1 --2- 172.21.2.112:0/2211532837 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2dcc018730 0x7f2dcc01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:03.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.273 7f2dd3fff700 1 -- 172.21.2.112:0/2211532837 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f2de4032b70 con 0x7f2dec0c1d80 2021-02-10T11:31:03.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.273 7f2deb2bf700 1 --2- 172.21.2.112:0/2211532837 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2dcc018730 0x7f2dcc01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:03.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.273 7f2deb2bf700 1 --2- 172.21.2.112:0/2211532837 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2dcc018730 0x7f2dcc01ac10 crc :-1 s=READY pgs=477 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:03.291 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.303 7f2df1576700 1 -- 172.21.2.112:0/2211532837 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2dec105ae0 con 0x7f2dec0c1d80 2021-02-10T11:31:03.293 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.306 7f2dd3fff700 1 -- 172.21.2.112:0/2211532837 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2de4000df0 con 0x7f2dec0c1d80 2021-02-10T11:31:03.422 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.434 7f2df1576700 1 -- 172.21.2.112:0/2211532837 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348656", "target": ["mgr", ""]}) v1 -- 0x7f2dec0f0b50 con 0x7f2dcc018730 2021-02-10T11:31:03.432 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.445 7f2dd3fff700 1 -- 172.21.2.112:0/2211532837 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f2ddc001a00 con 0x7f2dcc018730 2021-02-10T11:31:03.438 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.451 7f2df1576700 1 -- 172.21.2.112:0/2211532837 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2dcc018730 msgr2=0x7f2dcc01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:03.438 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.451 7f2df1576700 1 --2- 172.21.2.112:0/2211532837 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2dcc018730 0x7f2dcc01ac10 crc :-1 s=READY pgs=477 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:03.438 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.451 7f2df1576700 1 -- 172.21.2.112:0/2211532837 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2dec0c1d80 msgr2=0x7f2dec0c2180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:03.438 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.451 7f2df1576700 1 --2- 172.21.2.112:0/2211532837 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2dec0c1d80 0x7f2dec0c2180 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 rx=0x7f2de4006c50 tx=0x7f2de400bd20).stop 2021-02-10T11:31:03.438 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.451 7f2df1576700 1 -- 172.21.2.112:0/2211532837 shutdown_connections 2021-02-10T11:31:03.439 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.451 7f2df1576700 1 --2- 172.21.2.112:0/2211532837 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2dcc018730 0x7f2dcc01ac10 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:03.439 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.451 7f2df1576700 1 --2- 172.21.2.112:0/2211532837 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2dec0b8200 0x7f2dec0b8600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:03.439 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.451 7f2df1576700 1 --2- 172.21.2.112:0/2211532837 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2dec0c1d80 0x7f2dec0c2180 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:03.439 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.451 7f2df1576700 1 --2- 172.21.2.112:0/2211532837 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2dec190920 0x7f2dec0b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:03.439 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.451 7f2df1576700 1 -- 172.21.2.112:0/2211532837 >> 172.21.2.112:0/2211532837 conn(0x7f2dec094e10 msgr2=0x7f2dec0a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:03.439 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.451 7f2df1576700 1 -- 172.21.2.112:0/2211532837 shutdown_connections 2021-02-10T11:31:03.439 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.451 7f2df1576700 1 -- 172.21.2.112:0/2211532837 wait complete. 2021-02-10T11:31:03.447 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000348657 2021-02-10T11:31:03.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.872 7ff7fddb9700 1 Processor -- start 2021-02-10T11:31:03.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.872 7ff7fddb9700 1 -- start start 2021-02-10T11:31:03.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.873 7ff7fddb9700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7ff7f81b0470 0x7ff7f81b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:03.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.873 7ff7fddb9700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7ff7f80ef610 0x7ff7f80efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:03.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.873 7ff7fddb9700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7f80f0b50 con 0x7ff7f80f0030 2021-02-10T11:31:03.878 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.873 7ff7fddb9700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7f80e52e0 con 0x7ff7f81b0470 2021-02-10T11:31:03.878 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.873 7ff7fddb9700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7f80e5490 con 0x7ff7f80ef610 2021-02-10T11:31:03.878 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.873 7ff7f77fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7ff7f81b0470 0x7ff7f81b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:03.878 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.873 7ff7f77fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7ff7f81b0470 0x7ff7f81b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:47770/0 (socket says 172.21.2.112:47770) 2021-02-10T11:31:03.878 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.873 7ff7f77fe700 1 -- 172.21.2.112:0/1312712006 learned_addr learned my addr 172.21.2.112:0/1312712006 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:31:03.878 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.873 7ff7f6ffd700 1 --2- 172.21.2.112:0/1312712006 >> v2:172.21.2.120:3300/0 conn(0x7ff7f80ef610 0x7ff7f80efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:03.878 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.873 7ff7f77fe700 1 -- 172.21.2.112:0/1312712006 >> v2:172.21.2.120:3300/0 conn(0x7ff7f80ef610 msgr2=0x7ff7f80efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:03.878 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.873 7ff7f77fe700 1 --2- 172.21.2.112:0/1312712006 >> v2:172.21.2.120:3300/0 conn(0x7ff7f80ef610 0x7ff7f80efa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:03.878 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.873 7ff7f77fe700 1 -- 172.21.2.112:0/1312712006 >> v1:172.21.2.120:6789/0 conn(0x7ff7f80f0030 legacy=0x7ff7f80f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:03.878 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.873 7ff7f77fe700 1 -- 172.21.2.112:0/1312712006 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7f80f0430 con 0x7ff7f81b0470 2021-02-10T11:31:03.878 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.874 7ff7f77fe700 1 --2- 172.21.2.112:0/1312712006 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff7f81b0470 0x7ff7f81b0870 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 rx=0x7ff7e0006700 tx=0x7ff7e0009b80).ready entity=mon.0 client_cookie=af7648dc3f7c64c3 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:03.879 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.874 7ff7f67fc700 1 -- 172.21.2.112:0/1312712006 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff7e000c0c0 con 0x7ff7f81b0470 2021-02-10T11:31:03.879 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.874 7ff7f67fc700 1 -- 172.21.2.112:0/1312712006 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7e000c340 con 0x7ff7f81b0470 2021-02-10T11:31:03.879 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.874 7ff7f67fc700 1 -- 172.21.2.112:0/1312712006 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff7e000c860 con 0x7ff7f81b0470 2021-02-10T11:31:03.879 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.874 7ff7fddb9700 1 -- 172.21.2.112:0/1312712006 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff7f81b0470 msgr2=0x7ff7f81b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:03.879 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.874 7ff7fddb9700 1 --2- 172.21.2.112:0/1312712006 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff7f81b0470 0x7ff7f81b0870 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 rx=0x7ff7e0006700 tx=0x7ff7e0009b80).stop 2021-02-10T11:31:03.879 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.875 7ff7fddb9700 1 -- 172.21.2.112:0/1312712006 shutdown_connections 2021-02-10T11:31:03.879 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.875 7ff7fddb9700 1 --2- 172.21.2.112:0/1312712006 >> v2:172.21.2.120:3300/0 conn(0x7ff7f80ef610 0x7ff7f80efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:03.879 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.875 7ff7fddb9700 1 --2- 172.21.2.112:0/1312712006 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff7f81b0470 0x7ff7f81b0870 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:03.879 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.875 7ff7fddb9700 1 -- 172.21.2.112:0/1312712006 >> 172.21.2.112:0/1312712006 conn(0x7ff7f80ce3e0 msgr2=0x7ff7f80e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:03.879 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.875 7ff7fddb9700 1 -- 172.21.2.112:0/1312712006 shutdown_connections 2021-02-10T11:31:03.879 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.875 7ff7fddb9700 1 -- 172.21.2.112:0/1312712006 wait complete. 2021-02-10T11:31:03.879 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.890 7ff7fddb9700 1 Processor -- start 2021-02-10T11:31:03.879 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.890 7ff7fddb9700 1 -- start start 2021-02-10T11:31:03.881 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.890 7ff7fddb9700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff7f8190920 0x7ff7f80b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:03.881 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.890 7ff7fddb9700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff7f80b8200 0x7ff7f80b8600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:03.881 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.890 7ff7fddb9700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff7f80c1d80 0x7ff7f80c2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:03.881 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.890 7ff7fddb9700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7f8190d20 con 0x7ff7f80c1d80 2021-02-10T11:31:03.881 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.890 7ff7fddb9700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7f80b8c20 con 0x7ff7f80b8200 2021-02-10T11:31:03.881 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.890 7ff7fddb9700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7f80c27a0 con 0x7ff7f8190920 2021-02-10T11:31:03.881 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.890 7ff7f6ffd700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff7f80b8200 0x7ff7f80b8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:03.881 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.890 7ff7f6ffd700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff7f80b8200 0x7ff7f80b8600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3300/0 says I am v2:172.21.2.112:50052/0 (socket says 172.21.2.112:50052) 2021-02-10T11:31:03.882 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.890 7ff7f6ffd700 1 -- 172.21.2.112:0/3622515373 learned_addr learned my addr 172.21.2.112:0/3622515373 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:31:03.882 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.890 7ff7f7fff700 1 --2- 172.21.2.112:0/3622515373 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff7f80c1d80 0x7ff7f80c2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:03.882 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.890 7ff7f77fe700 1 --2- 172.21.2.112:0/3622515373 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff7f8190920 0x7ff7f80b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:03.882 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.891 7ff7f6ffd700 1 -- 172.21.2.112:0/3622515373 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff7f8190920 msgr2=0x7ff7f80b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:03.882 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.891 7ff7f6ffd700 1 --2- 172.21.2.112:0/3622515373 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff7f8190920 0x7ff7f80b7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:03.882 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.891 7ff7f6ffd700 1 -- 172.21.2.112:0/3622515373 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff7f80c1d80 msgr2=0x7ff7f80c2180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:03.882 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.891 7ff7f6ffd700 1 --2- 172.21.2.112:0/3622515373 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff7f80c1d80 0x7ff7f80c2180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:03.882 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.891 7ff7f6ffd700 1 -- 172.21.2.112:0/3622515373 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7f80c29c0 con 0x7ff7f80b8200 2021-02-10T11:31:03.882 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.891 7ff7f6ffd700 1 --2- 172.21.2.112:0/3622515373 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff7f80b8200 0x7ff7f80b8600 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 rx=0x7ff7e8006f60 tx=0x7ff7e800b970).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:03.882 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.891 7ff7effff700 1 -- 172.21.2.112:0/3622515373 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff7e800bd60 con 0x7ff7f80b8200 2021-02-10T11:31:03.882 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.891 7ff7effff700 1 -- 172.21.2.112:0/3622515373 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7e8000c90 con 0x7ff7f80b8200 2021-02-10T11:31:03.883 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.891 7ff7fddb9700 1 -- 172.21.2.112:0/3622515373 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7f80c29c0 con 0x7ff7f80b8200 2021-02-10T11:31:03.883 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.891 7ff7effff700 1 -- 172.21.2.112:0/3622515373 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff7e80022b0 con 0x7ff7f80b8200 2021-02-10T11:31:03.883 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.891 7ff7fddb9700 1 -- 172.21.2.112:0/3622515373 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7f80c2f50 con 0x7ff7f80b8200 2021-02-10T11:31:03.883 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.892 7ff7effff700 1 -- 172.21.2.112:0/3622515373 <== mon.1 v2:172.21.2.120:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff7e8006b00 con 0x7ff7f80b8200 2021-02-10T11:31:03.883 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.892 7ff7effff700 1 --2- 172.21.2.112:0/3622515373 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff7dc0186c0 0x7ff7dc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:03.883 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.892 7ff7effff700 1 -- 172.21.2.112:0/3622515373 <== mon.1 v2:172.21.2.120:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7ff7e80323a0 con 0x7ff7f80b8200 2021-02-10T11:31:03.883 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.892 7ff7f77fe700 1 --2- 172.21.2.112:0/3622515373 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff7dc0186c0 0x7ff7dc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:03.883 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.892 7ff7f77fe700 1 --2- 172.21.2.112:0/3622515373 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff7dc0186c0 0x7ff7dc01abc0 crc :-1 s=READY pgs=478 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:03.909 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.922 7ff7fddb9700 1 -- 172.21.2.112:0/3622515373 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7f8105ae0 con 0x7ff7f80b8200 2021-02-10T11:31:03.913 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:03.925 7ff7effff700 1 -- 172.21.2.112:0/3622515373 <== mon.1 v2:172.21.2.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff7e801b7e0 con 0x7ff7f80b8200 2021-02-10T11:31:04.045 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:04.057 7ff7fddb9700 1 -- 172.21.2.112:0/3622515373 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348657", "target": ["mgr", ""]}) v1 -- 0x7ff7f8190d20 con 0x7ff7dc0186c0 2021-02-10T11:31:04.054 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:04.067 7ff7effff700 1 -- 172.21.2.112:0/3622515373 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7ff7e0006430 con 0x7ff7dc0186c0 2021-02-10T11:31:04.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:04.074 7ff7fddb9700 1 -- 172.21.2.112:0/3622515373 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff7dc0186c0 msgr2=0x7ff7dc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:04.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:04.074 7ff7fddb9700 1 --2- 172.21.2.112:0/3622515373 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff7dc0186c0 0x7ff7dc01abc0 crc :-1 s=READY pgs=478 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:04.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:04.074 7ff7fddb9700 1 -- 172.21.2.112:0/3622515373 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff7f80b8200 msgr2=0x7ff7f80b8600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:04.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:04.074 7ff7fddb9700 1 --2- 172.21.2.112:0/3622515373 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff7f80b8200 0x7ff7f80b8600 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 rx=0x7ff7e8006f60 tx=0x7ff7e800b970).stop 2021-02-10T11:31:04.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:04.074 7ff7fddb9700 1 -- 172.21.2.112:0/3622515373 shutdown_connections 2021-02-10T11:31:04.063 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:04.074 7ff7fddb9700 1 --2- 172.21.2.112:0/3622515373 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff7dc0186c0 0x7ff7dc01abc0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:04.063 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:04.074 7ff7fddb9700 1 --2- 172.21.2.112:0/3622515373 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff7f80b8200 0x7ff7f80b8600 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:04.063 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:04.074 7ff7fddb9700 1 --2- 172.21.2.112:0/3622515373 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff7f80c1d80 0x7ff7f80c2180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:04.063 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:04.074 7ff7fddb9700 1 --2- 172.21.2.112:0/3622515373 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff7f8190920 0x7ff7f80b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:04.063 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:04.074 7ff7fddb9700 1 -- 172.21.2.112:0/3622515373 >> 172.21.2.112:0/3622515373 conn(0x7ff7f8094e10 msgr2=0x7ff7f80a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:04.063 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:04.075 7ff7fddb9700 1 -- 172.21.2.112:0/3622515373 shutdown_connections 2021-02-10T11:31:04.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:04.075 7ff7fddb9700 1 -- 172.21.2.112:0/3622515373 wait complete. 2021-02-10T11:31:04.070 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000348658 2021-02-10T11:31:04.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:04.495 7fa6efad2700 1 Processor -- start 2021-02-10T11:31:04.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:04.495 7fa6efad2700 1 -- start start 2021-02-10T11:31:04.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:04.495 7fa6efad2700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fa6e81b0470 0x7fa6e81b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:04.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:04.495 7fa6efad2700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6e80f0b50 con 0x7fa6e80ef610 2021-02-10T11:31:04.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:04.495 7fa6efad2700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6e80f24e0 con 0x7fa6e80f20e0 2021-02-10T11:31:04.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:04.495 7fa6efad2700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6e80e5ea0 con 0x7fa6e81b0470 2021-02-10T11:31:04.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:04.496 7fa6ed0ed700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fa6e80ef610 0x7fa6e80efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:47242/0 (socket says 172.21.2.112:47242) 2021-02-10T11:31:04.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:04.496 7fa6ed0ed700 1 -- 172.21.2.112:0/3074813198 learned_addr learned my addr 172.21.2.112:0/3074813198 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:31:04.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:04.496 7fa6ed8ee700 1 --2- 172.21.2.112:0/3074813198 >> v2:172.21.2.120:3300/0 conn(0x7fa6e81b0470 0x7fa6e81b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:04.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:04.496 7fa6ec8ec700 1 -- 172.21.2.112:0/3074813198 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1355982203 0 0) 0x7fa6e40010b0 con 0x7fa6e80ef610 2021-02-10T11:31:04.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:04.496 7fa6ec8ec700 1 -- 172.21.2.112:0/3074813198 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6d4002fe0 con 0x7fa6e80ef610 2021-02-10T11:31:04.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:04.496 7fa6ed8ee700 1 -- 172.21.2.112:0/3074813198 >> v1:172.21.2.120:6789/0 conn(0x7fa6e80f20e0 legacy=0x7fa6e80f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:04.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:04.496 7fa6ed8ee700 1 -- 172.21.2.112:0/3074813198 >> v1:172.21.2.112:6789/0 conn(0x7fa6e80ef610 legacy=0x7fa6e80efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:04.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:04.496 7fa6ed8ee700 1 -- 172.21.2.112:0/3074813198 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6e80e52e0 con 0x7fa6e81b0470 2021-02-10T11:31:04.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:04.496 7fa6ed8ee700 1 --2- 172.21.2.112:0/3074813198 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa6e81b0470 0x7fa6e81b0870 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0x7fa6dc006650 tx=0x7fa6dc009a80).ready entity=mon.1 client_cookie=33158648c7f0e3cd server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:04.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:04.497 7fa6ec8ec700 1 -- 172.21.2.112:0/3074813198 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa6dc0009b0 con 0x7fa6e81b0470 2021-02-10T11:31:04.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:04.497 7fa6ec8ec700 1 -- 172.21.2.112:0/3074813198 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6dc00c0c0 con 0x7fa6e81b0470 2021-02-10T11:31:04.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:04.497 7fa6ec8ec700 1 -- 172.21.2.112:0/3074813198 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa6dc00c5e0 con 0x7fa6e81b0470 2021-02-10T11:31:04.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:04.497 7fa6efad2700 1 -- 172.21.2.112:0/3074813198 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa6e81b0470 msgr2=0x7fa6e81b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:04.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:04.497 7fa6efad2700 1 --2- 172.21.2.112:0/3074813198 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa6e81b0470 0x7fa6e81b0870 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0x7fa6dc006650 tx=0x7fa6dc009a80).stop 2021-02-10T11:31:04.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:04.497 7fa6efad2700 1 -- 172.21.2.112:0/3074813198 shutdown_connections 2021-02-10T11:31:04.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:04.497 7fa6efad2700 1 --2- 172.21.2.112:0/3074813198 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa6e81b0470 0x7fa6e81b0870 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:04.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:04.497 7fa6efad2700 1 -- 172.21.2.112:0/3074813198 >> 172.21.2.112:0/3074813198 conn(0x7fa6e80ce3e0 msgr2=0x7fa6e80e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:04.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:04.497 7fa6efad2700 1 -- 172.21.2.112:0/3074813198 shutdown_connections 2021-02-10T11:31:04.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:04.497 7fa6efad2700 1 -- 172.21.2.112:0/3074813198 wait complete. 2021-02-10T11:31:04.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:04.512 7fa6efad2700 1 Processor -- start 2021-02-10T11:31:04.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:04.512 7fa6efad2700 1 -- start start 2021-02-10T11:31:04.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:04.512 7fa6efad2700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa6e80c36a0 0x7fa6e80c2b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:04.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:04.512 7fa6efad2700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa6e80c3190 0x7fa6e80b1d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:04.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:04.512 7fa6efad2700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa6e80b2370 0x7fa6e80b2d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:04.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:04.512 7fa6efad2700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6e80b27c0 con 0x7fa6e80c3190 2021-02-10T11:31:04.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:04.512 7fa6efad2700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6e80b3360 con 0x7fa6e80c36a0 2021-02-10T11:31:04.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:04.512 7fa6efad2700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6e80b3540 con 0x7fa6e80b2370 2021-02-10T11:31:04.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:04.513 7fa6ed8ee700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa6e80c36a0 0x7fa6e80c2b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:04.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:04.513 7fa6ed8ee700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa6e80c36a0 0x7fa6e80c2b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3300/0 says I am v2:172.21.2.112:50066/0 (socket says 172.21.2.112:50066) 2021-02-10T11:31:04.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:04.513 7fa6ed8ee700 1 -- 172.21.2.112:0/1927397783 learned_addr learned my addr 172.21.2.112:0/1927397783 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:31:04.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:04.513 7fa6ed0ed700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa6e80c3190 0x7fa6e80b1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:04.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:04.513 7fa6ee0ef700 1 --2- 172.21.2.112:0/1927397783 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa6e80b2370 0x7fa6e80b2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:04.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:04.513 7fa6ed8ee700 1 -- 172.21.2.112:0/1927397783 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa6e80b2370 msgr2=0x7fa6e80b2d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:04.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:04.513 7fa6ed8ee700 1 --2- 172.21.2.112:0/1927397783 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa6e80b2370 0x7fa6e80b2d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:04.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:04.513 7fa6ed8ee700 1 -- 172.21.2.112:0/1927397783 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa6e80c3190 msgr2=0x7fa6e80b1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:04.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:04.513 7fa6ed8ee700 1 --2- 172.21.2.112:0/1927397783 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa6e80c3190 0x7fa6e80b1d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:04.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:04.513 7fa6ed8ee700 1 -- 172.21.2.112:0/1927397783 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6e80b3760 con 0x7fa6e80c36a0 2021-02-10T11:31:04.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:04.513 7fa6ed0ed700 1 --2- 172.21.2.112:0/1927397783 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa6e80c3190 0x7fa6e80b1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:31:04.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:04.513 7fa6ed8ee700 1 --2- 172.21.2.112:0/1927397783 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa6e80c36a0 0x7fa6e80c2b70 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7fa6dc006f70 tx=0x7fa6dc00b960).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:04.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:04.513 7fa6da7fc700 1 -- 172.21.2.112:0/1927397783 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa6dc0064a0 con 0x7fa6e80c36a0 2021-02-10T11:31:04.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:04.514 7fa6da7fc700 1 -- 172.21.2.112:0/1927397783 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6dc00bd50 con 0x7fa6e80c36a0 2021-02-10T11:31:04.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:04.514 7fa6da7fc700 1 -- 172.21.2.112:0/1927397783 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa6dc0052e0 con 0x7fa6e80c36a0 2021-02-10T11:31:04.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:04.514 7fa6efad2700 1 -- 172.21.2.112:0/1927397783 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6e80b3760 con 0x7fa6e80c36a0 2021-02-10T11:31:04.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:04.514 7fa6efad2700 1 -- 172.21.2.112:0/1927397783 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6e80b3760 con 0x7fa6e80c36a0 2021-02-10T11:31:04.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:04.514 7fa6da7fc700 1 -- 172.21.2.112:0/1927397783 <== mon.1 v2:172.21.2.120:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa6dc006030 con 0x7fa6e80c36a0 2021-02-10T11:31:04.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:04.514 7fa6da7fc700 1 --2- 172.21.2.112:0/1927397783 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa6d00186c0 0x7fa6d001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:04.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:04.514 7fa6da7fc700 1 -- 172.21.2.112:0/1927397783 <== mon.1 v2:172.21.2.120:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fa6dc032380 con 0x7fa6e80c36a0 2021-02-10T11:31:04.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:04.514 7fa6ed0ed700 1 --2- 172.21.2.112:0/1927397783 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa6d00186c0 0x7fa6d001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:04.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:04.515 7fa6ed0ed700 1 --2- 172.21.2.112:0/1927397783 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa6d00186c0 0x7fa6d001abc0 crc :-1 s=READY pgs=479 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:04.532 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:04.545 7fa6efad2700 1 -- 172.21.2.112:0/1927397783 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6e80edd10 con 0x7fa6e80c36a0 2021-02-10T11:31:04.537 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:04.550 7fa6da7fc700 1 -- 172.21.2.112:0/1927397783 <== mon.1 v2:172.21.2.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa6dc006bc0 con 0x7fa6e80c36a0 2021-02-10T11:31:04.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:04.680 7fa6efad2700 1 -- 172.21.2.112:0/1927397783 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348658", "target": ["mgr", ""]}) v1 -- 0x7fa6e80edd10 con 0x7fa6d00186c0 2021-02-10T11:31:04.679 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:04.692 7fa6da7fc700 1 -- 172.21.2.112:0/1927397783 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fa6e40009b0 con 0x7fa6d00186c0 2021-02-10T11:31:04.686 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:04.698 7fa6efad2700 1 -- 172.21.2.112:0/1927397783 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa6d00186c0 msgr2=0x7fa6d001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:04.686 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:04.698 7fa6efad2700 1 --2- 172.21.2.112:0/1927397783 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa6d00186c0 0x7fa6d001abc0 crc :-1 s=READY pgs=479 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:04.686 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:04.698 7fa6efad2700 1 -- 172.21.2.112:0/1927397783 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa6e80c36a0 msgr2=0x7fa6e80c2b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:04.686 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:04.698 7fa6efad2700 1 --2- 172.21.2.112:0/1927397783 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa6e80c36a0 0x7fa6e80c2b70 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7fa6dc006f70 tx=0x7fa6dc00b960).stop 2021-02-10T11:31:04.686 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:04.698 7fa6efad2700 1 -- 172.21.2.112:0/1927397783 shutdown_connections 2021-02-10T11:31:04.686 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:04.698 7fa6efad2700 1 --2- 172.21.2.112:0/1927397783 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa6d00186c0 0x7fa6d001abc0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:04.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:04.698 7fa6efad2700 1 --2- 172.21.2.112:0/1927397783 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa6e80b2370 0x7fa6e80b2d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:04.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:04.698 7fa6efad2700 1 --2- 172.21.2.112:0/1927397783 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa6e80c3190 0x7fa6e80b1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:04.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:04.698 7fa6efad2700 1 --2- 172.21.2.112:0/1927397783 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa6e80c36a0 0x7fa6e80c2b70 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:04.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:04.698 7fa6efad2700 1 -- 172.21.2.112:0/1927397783 >> 172.21.2.112:0/1927397783 conn(0x7fa6e809f630 msgr2=0x7fa6e80a0130 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:04.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:04.698 7fa6efad2700 1 -- 172.21.2.112:0/1927397783 shutdown_connections 2021-02-10T11:31:04.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:04.698 7fa6efad2700 1 -- 172.21.2.112:0/1927397783 wait complete. 2021-02-10T11:31:04.695 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000348659 2021-02-10T11:31:05.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.121 7f60a12df700 1 Processor -- start 2021-02-10T11:31:05.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.121 7f60a12df700 1 -- start start 2021-02-10T11:31:05.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.122 7f60a12df700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f609c1b0470 0x7f609c1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:05.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.122 7f60a12df700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f609c0f20e0 0x7f609c0f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:05.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.122 7f60a12df700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f609c0f0b50 con 0x7f609c0ef610 2021-02-10T11:31:05.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.122 7f60a12df700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f609c0f0820 con 0x7f609c1b0470 2021-02-10T11:31:05.126 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.122 7f60a12df700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f609c0f24e0 con 0x7f609c0f20e0 2021-02-10T11:31:05.126 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.122 7f609a87c700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f609c0ef610 0x7f609c0efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:47256/0 (socket says 172.21.2.112:47256) 2021-02-10T11:31:05.126 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.122 7f609a87c700 1 -- 172.21.2.112:0/3453603885 learned_addr learned my addr 172.21.2.112:0/3453603885 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:31:05.126 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.122 7f609b07d700 1 --2- 172.21.2.112:0/3453603885 >> v2:172.21.2.120:3300/0 conn(0x7f609c1b0470 0x7f609c1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:05.126 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.122 7f609b87e700 1 --2- 172.21.2.112:0/3453603885 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f609c0f20e0 0x7f609c0f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:05.126 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.122 7f609b07d700 1 -- 172.21.2.112:0/3453603885 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f609c0f20e0 msgr2=0x7f609c0f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:05.126 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.122 7f609b07d700 1 --2- 172.21.2.112:0/3453603885 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f609c0f20e0 0x7f609c0f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:05.126 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.122 7f609b07d700 1 -- 172.21.2.112:0/3453603885 >> v1:172.21.2.112:6789/0 conn(0x7f609c0ef610 legacy=0x7f609c0efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:05.126 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.122 7f609b07d700 1 -- 172.21.2.112:0/3453603885 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f609c0e52e0 con 0x7f609c1b0470 2021-02-10T11:31:05.126 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.122 7f609b07d700 1 --2- 172.21.2.112:0/3453603885 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f609c1b0470 0x7f609c1b0870 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 rx=0x7f608c006650 tx=0x7f608c009a80).ready entity=mon.1 client_cookie=1765d3c0decb5271 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:05.126 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.122 7f609a07b700 1 -- 172.21.2.112:0/3453603885 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f608c0009b0 con 0x7f609c1b0470 2021-02-10T11:31:05.126 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.122 7f609a07b700 1 -- 172.21.2.112:0/3453603885 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f608c00c0c0 con 0x7f609c1b0470 2021-02-10T11:31:05.127 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.122 7f609a07b700 1 -- 172.21.2.112:0/3453603885 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f608c00c5e0 con 0x7f609c1b0470 2021-02-10T11:31:05.127 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.123 7f60a12df700 1 -- 172.21.2.112:0/3453603885 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f609c1b0470 msgr2=0x7f609c1b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:05.127 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.123 7f60a12df700 1 --2- 172.21.2.112:0/3453603885 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f609c1b0470 0x7f609c1b0870 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 rx=0x7f608c006650 tx=0x7f608c009a80).stop 2021-02-10T11:31:05.127 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.123 7f60a12df700 1 -- 172.21.2.112:0/3453603885 shutdown_connections 2021-02-10T11:31:05.127 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.123 7f60a12df700 1 --2- 172.21.2.112:0/3453603885 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f609c0f20e0 0x7f609c0f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:05.127 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.123 7f60a12df700 1 --2- 172.21.2.112:0/3453603885 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f609c1b0470 0x7f609c1b0870 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:05.127 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.123 7f60a12df700 1 -- 172.21.2.112:0/3453603885 >> 172.21.2.112:0/3453603885 conn(0x7f609c0ce3e0 msgr2=0x7f609c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:05.127 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.123 7f60a12df700 1 -- 172.21.2.112:0/3453603885 shutdown_connections 2021-02-10T11:31:05.127 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.123 7f60a12df700 1 -- 172.21.2.112:0/3453603885 wait complete. 2021-02-10T11:31:05.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.138 7f60a12df700 1 Processor -- start 2021-02-10T11:31:05.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.138 7f60a12df700 1 -- start start 2021-02-10T11:31:05.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.139 7f60a12df700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f609c0a0420 0x7f609c0a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:05.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.139 7f60a12df700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f609c189d00 0x7f609c18a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:05.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.139 7f60a12df700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f609c18a720 0x7f609c0b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:05.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.139 7f60a12df700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f609c189800 con 0x7f609c0a0420 2021-02-10T11:31:05.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.139 7f60a12df700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f609c18ab20 con 0x7f609c189d00 2021-02-10T11:31:05.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.139 7f60a12df700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f609c0a1140 con 0x7f609c18a720 2021-02-10T11:31:05.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.139 7f609b07d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f609c0a0420 0x7f609c0a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:05.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.139 7f609b07d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f609c0a0420 0x7f609c0a0820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:47796/0 (socket says 172.21.2.112:47796) 2021-02-10T11:31:05.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.139 7f609b07d700 1 -- 172.21.2.112:0/4134686957 learned_addr learned my addr 172.21.2.112:0/4134686957 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:31:05.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.139 7f609a87c700 1 --2- 172.21.2.112:0/4134686957 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f609c189d00 0x7f609c18a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:05.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.139 7f609b87e700 1 --2- 172.21.2.112:0/4134686957 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f609c18a720 0x7f609c0b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:05.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.139 7f609b07d700 1 -- 172.21.2.112:0/4134686957 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f609c18a720 msgr2=0x7f609c0b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:05.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.139 7f609b07d700 1 --2- 172.21.2.112:0/4134686957 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f609c18a720 0x7f609c0b7be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:05.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.139 7f609b07d700 1 -- 172.21.2.112:0/4134686957 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f609c189d00 msgr2=0x7f609c18a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:05.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.139 7f609b07d700 1 --2- 172.21.2.112:0/4134686957 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f609c189d00 0x7f609c18a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:05.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.139 7f609b07d700 1 -- 172.21.2.112:0/4134686957 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f609c0b8200 con 0x7f609c0a0420 2021-02-10T11:31:05.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.139 7f609b07d700 1 --2- 172.21.2.112:0/4134686957 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f609c0a0420 0x7f609c0a0820 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 rx=0x7f608c009650 tx=0x7f608c00b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:05.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.139 7f6093fff700 1 -- 172.21.2.112:0/4134686957 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f608c0093a0 con 0x7f609c0a0420 2021-02-10T11:31:05.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.139 7f6093fff700 1 -- 172.21.2.112:0/4134686957 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f608c00bde0 con 0x7f609c0a0420 2021-02-10T11:31:05.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.139 7f6093fff700 1 -- 172.21.2.112:0/4134686957 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f608c0052e0 con 0x7f609c0a0420 2021-02-10T11:31:05.131 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.139 7f60a12df700 1 -- 172.21.2.112:0/4134686957 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f609c0b8200 con 0x7f609c0a0420 2021-02-10T11:31:05.131 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.140 7f60a12df700 1 -- 172.21.2.112:0/4134686957 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f609c0b8200 con 0x7f609c0a0420 2021-02-10T11:31:05.131 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.140 7f6093fff700 1 -- 172.21.2.112:0/4134686957 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f608c006030 con 0x7f609c0a0420 2021-02-10T11:31:05.131 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.140 7f6093fff700 1 --2- 172.21.2.112:0/4134686957 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f60800186c0 0x7f608001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:05.131 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.140 7f6093fff700 1 -- 172.21.2.112:0/4134686957 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f608c0327e0 con 0x7f609c0a0420 2021-02-10T11:31:05.131 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.140 7f609a87c700 1 --2- 172.21.2.112:0/4134686957 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f60800186c0 0x7f608001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:05.131 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.141 7f609a87c700 1 --2- 172.21.2.112:0/4134686957 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f60800186c0 0x7f608001abc0 crc :-1 s=READY pgs=480 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:05.158 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.171 7f60a12df700 1 -- 172.21.2.112:0/4134686957 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f609c105ae0 con 0x7f609c0a0420 2021-02-10T11:31:05.161 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.173 7f6093fff700 1 -- 172.21.2.112:0/4134686957 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f608c0009c0 con 0x7f609c0a0420 2021-02-10T11:31:05.290 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.303 7f60a12df700 1 -- 172.21.2.112:0/4134686957 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348659", "target": ["mgr", ""]}) v1 -- 0x7f609c18ab20 con 0x7f60800186c0 2021-02-10T11:31:05.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.313 7f6093fff700 1 -- 172.21.2.112:0/4134686957 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f6084000e20 con 0x7f60800186c0 2021-02-10T11:31:05.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.320 7f60a12df700 1 -- 172.21.2.112:0/4134686957 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f60800186c0 msgr2=0x7f608001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:05.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.320 7f60a12df700 1 --2- 172.21.2.112:0/4134686957 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f60800186c0 0x7f608001abc0 crc :-1 s=READY pgs=480 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:05.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.320 7f60a12df700 1 -- 172.21.2.112:0/4134686957 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f609c0a0420 msgr2=0x7f609c0a0820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:05.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.320 7f60a12df700 1 --2- 172.21.2.112:0/4134686957 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f609c0a0420 0x7f609c0a0820 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 rx=0x7f608c009650 tx=0x7f608c00b960).stop 2021-02-10T11:31:05.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.320 7f60a12df700 1 -- 172.21.2.112:0/4134686957 shutdown_connections 2021-02-10T11:31:05.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.320 7f60a12df700 1 --2- 172.21.2.112:0/4134686957 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f60800186c0 0x7f608001abc0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:05.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.320 7f60a12df700 1 --2- 172.21.2.112:0/4134686957 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f609c189d00 0x7f609c18a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:05.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.320 7f60a12df700 1 --2- 172.21.2.112:0/4134686957 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f609c18a720 0x7f609c0b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:05.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.320 7f60a12df700 1 --2- 172.21.2.112:0/4134686957 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f609c0a0420 0x7f609c0a0820 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:05.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.320 7f60a12df700 1 -- 172.21.2.112:0/4134686957 >> 172.21.2.112:0/4134686957 conn(0x7f609c094e10 msgr2=0x7f609c0a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:05.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.321 7f60a12df700 1 -- 172.21.2.112:0/4134686957 shutdown_connections 2021-02-10T11:31:05.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.321 7f60a12df700 1 -- 172.21.2.112:0/4134686957 wait complete. 2021-02-10T11:31:05.317 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000348660 2021-02-10T11:31:05.746 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.743 7fcd94ff5700 1 Processor -- start 2021-02-10T11:31:05.747 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.743 7fcd94ff5700 1 -- start start 2021-02-10T11:31:05.747 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.743 7fcd94ff5700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcd900f05f0 0x7fcd900f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:05.747 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.743 7fcd94ff5700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd901b0c70 con 0x7fcd901b0470 2021-02-10T11:31:05.747 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.743 7fcd94ff5700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd900f0b50 con 0x7fcd901b0870 2021-02-10T11:31:05.747 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.743 7fcd94ff5700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd900e5ea0 con 0x7fcd900f05f0 2021-02-10T11:31:05.747 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.743 7fcd8f59e700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcd900f05f0 0x7fcd900f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:05.747 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.743 7fcd8e59c700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7fcd901b0870 0x7fcd900efe00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:40824/0 (socket says 172.21.2.112:40824) 2021-02-10T11:31:05.747 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.743 7fcd8f59e700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcd900f05f0 0x7fcd900f20e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3301/0 says I am v2:172.21.2.112:47490/0 (socket says 172.21.2.112:47490) 2021-02-10T11:31:05.747 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.743 7fcd8ed9d700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fcd901b0470 0x7fcd900ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:47266/0 (socket says 172.21.2.112:47266) 2021-02-10T11:31:05.747 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.743 7fcd8e59c700 1 -- 172.21.2.112:0/240377094 learned_addr learned my addr 172.21.2.112:0/240377094 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:31:05.747 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.744 7fcd8dd9b700 1 -- 172.21.2.112:0/240377094 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1901878212 0 0) 0x7fcd78001100 con 0x7fcd901b0470 2021-02-10T11:31:05.748 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.744 7fcd8f59e700 1 -- 172.21.2.112:0/240377094 >> v1:172.21.2.120:6789/0 conn(0x7fcd901b0870 legacy=0x7fcd900efe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:05.748 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.744 7fcd8f59e700 1 -- 172.21.2.112:0/240377094 >> v1:172.21.2.112:6789/0 conn(0x7fcd901b0470 legacy=0x7fcd900ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:05.748 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.744 7fcd8f59e700 1 -- 172.21.2.112:0/240377094 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd900e52e0 con 0x7fcd900f05f0 2021-02-10T11:31:05.748 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.744 7fcd8f59e700 1 --2- 172.21.2.112:0/240377094 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcd900f05f0 0x7fcd900f20e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7fcd88006e50 tx=0x7fcd88009e70).ready entity=mon.2 client_cookie=98f074b3413bfecf server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:05.748 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.744 7fcd8dd9b700 1 -- 172.21.2.112:0/240377094 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcd8800b300 con 0x7fcd900f05f0 2021-02-10T11:31:05.748 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.744 7fcd8dd9b700 1 -- 172.21.2.112:0/240377094 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd88003030 con 0x7fcd900f05f0 2021-02-10T11:31:05.748 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.744 7fcd8dd9b700 1 -- 172.21.2.112:0/240377094 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcd88000f90 con 0x7fcd900f05f0 2021-02-10T11:31:05.748 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.744 7fcd94ff5700 1 -- 172.21.2.112:0/240377094 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcd900f05f0 msgr2=0x7fcd900f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:05.748 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.744 7fcd94ff5700 1 --2- 172.21.2.112:0/240377094 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcd900f05f0 0x7fcd900f20e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7fcd88006e50 tx=0x7fcd88009e70).stop 2021-02-10T11:31:05.748 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.745 7fcd94ff5700 1 -- 172.21.2.112:0/240377094 shutdown_connections 2021-02-10T11:31:05.748 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.745 7fcd94ff5700 1 --2- 172.21.2.112:0/240377094 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcd900f05f0 0x7fcd900f20e0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:05.748 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.745 7fcd94ff5700 1 -- 172.21.2.112:0/240377094 >> 172.21.2.112:0/240377094 conn(0x7fcd900ce3e0 msgr2=0x7fcd900e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:05.748 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.745 7fcd94ff5700 1 -- 172.21.2.112:0/240377094 shutdown_connections 2021-02-10T11:31:05.749 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.745 7fcd94ff5700 1 -- 172.21.2.112:0/240377094 wait complete. 2021-02-10T11:31:05.750 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.760 7fcd94ff5700 1 Processor -- start 2021-02-10T11:31:05.750 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.760 7fcd94ff5700 1 -- start start 2021-02-10T11:31:05.750 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.760 7fcd94ff5700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcd900b2840 0x7fcd901a4300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:05.750 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.760 7fcd94ff5700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcd900b30d0 0x7fcd900b34d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:05.750 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.760 7fcd94ff5700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcd900b3af0 0x7fcd900f0580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:05.751 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.760 7fcd94ff5700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd900b2c40 con 0x7fcd900b2840 2021-02-10T11:31:05.751 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.760 7fcd94ff5700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd901a4920 con 0x7fcd900b3af0 2021-02-10T11:31:05.751 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.760 7fcd94ff5700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd900f0ba0 con 0x7fcd900b30d0 2021-02-10T11:31:05.751 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.760 7fcd8f59e700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcd900b3af0 0x7fcd900f0580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:05.751 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.760 7fcd8f59e700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcd900b3af0 0x7fcd900f0580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3300/0 says I am v2:172.21.2.112:50094/0 (socket says 172.21.2.112:50094) 2021-02-10T11:31:05.751 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.760 7fcd8f59e700 1 -- 172.21.2.112:0/1727801934 learned_addr learned my addr 172.21.2.112:0/1727801934 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:31:05.751 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.760 7fcd8ed9d700 1 --2- 172.21.2.112:0/1727801934 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcd900b2840 0x7fcd901a4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:05.751 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.760 7fcd8e59c700 1 --2- 172.21.2.112:0/1727801934 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcd900b30d0 0x7fcd900b34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:05.751 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.761 7fcd8f59e700 1 -- 172.21.2.112:0/1727801934 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcd900b30d0 msgr2=0x7fcd900b34d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:05.751 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.761 7fcd8f59e700 1 --2- 172.21.2.112:0/1727801934 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcd900b30d0 0x7fcd900b34d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:05.751 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.761 7fcd8f59e700 1 -- 172.21.2.112:0/1727801934 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcd900b2840 msgr2=0x7fcd901a4300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:05.751 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.761 7fcd8f59e700 1 --2- 172.21.2.112:0/1727801934 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcd900b2840 0x7fcd901a4300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:05.751 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.761 7fcd8f59e700 1 -- 172.21.2.112:0/1727801934 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd900f0dc0 con 0x7fcd900b3af0 2021-02-10T11:31:05.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.761 7fcd8ed9d700 1 --2- 172.21.2.112:0/1727801934 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcd900b2840 0x7fcd901a4300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:31:05.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.761 7fcd8f59e700 1 --2- 172.21.2.112:0/1727801934 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcd900b3af0 0x7fcd900f0580 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 rx=0x7fcd880069a0 tx=0x7fcd8800bd10).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:05.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.761 7fcd877fe700 1 -- 172.21.2.112:0/1727801934 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcd88009700 con 0x7fcd900b3af0 2021-02-10T11:31:05.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.761 7fcd877fe700 1 -- 172.21.2.112:0/1727801934 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd88009aa0 con 0x7fcd900b3af0 2021-02-10T11:31:05.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.761 7fcd94ff5700 1 -- 172.21.2.112:0/1727801934 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd900f0dc0 con 0x7fcd900b3af0 2021-02-10T11:31:05.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.761 7fcd94ff5700 1 -- 172.21.2.112:0/1727801934 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd900f0dc0 con 0x7fcd900b3af0 2021-02-10T11:31:05.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.761 7fcd877fe700 1 -- 172.21.2.112:0/1727801934 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcd880074c0 con 0x7fcd900b3af0 2021-02-10T11:31:05.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.762 7fcd877fe700 1 -- 172.21.2.112:0/1727801934 <== mon.1 v2:172.21.2.120:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fcd88007240 con 0x7fcd900b3af0 2021-02-10T11:31:05.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.762 7fcd877fe700 1 --2- 172.21.2.112:0/1727801934 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcd74018730 0x7fcd7401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:05.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.762 7fcd877fe700 1 -- 172.21.2.112:0/1727801934 <== mon.1 v2:172.21.2.120:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fcd88032b10 con 0x7fcd900b3af0 2021-02-10T11:31:05.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.762 7fcd8ed9d700 1 --2- 172.21.2.112:0/1727801934 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcd74018730 0x7fcd7401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:05.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.763 7fcd8ed9d700 1 --2- 172.21.2.112:0/1727801934 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcd74018730 0x7fcd7401ac10 crc :-1 s=READY pgs=481 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:05.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.793 7fcd94ff5700 1 -- 172.21.2.112:0/1727801934 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd900f0050 con 0x7fcd900b3af0 2021-02-10T11:31:05.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.798 7fcd877fe700 1 -- 172.21.2.112:0/1727801934 <== mon.1 v2:172.21.2.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fcd88007030 con 0x7fcd900b3af0 2021-02-10T11:31:05.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.928 7fcd94ff5700 1 -- 172.21.2.112:0/1727801934 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348660", "target": ["mgr", ""]}) v1 -- 0x7fcd900f0050 con 0x7fcd74018730 2021-02-10T11:31:05.926 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.939 7fcd877fe700 1 -- 172.21.2.112:0/1727801934 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fcd78000d70 con 0x7fcd74018730 2021-02-10T11:31:05.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.945 7fcd94ff5700 1 -- 172.21.2.112:0/1727801934 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcd74018730 msgr2=0x7fcd7401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:05.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.945 7fcd94ff5700 1 --2- 172.21.2.112:0/1727801934 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcd74018730 0x7fcd7401ac10 crc :-1 s=READY pgs=481 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:05.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.945 7fcd94ff5700 1 -- 172.21.2.112:0/1727801934 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcd900b3af0 msgr2=0x7fcd900f0580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:05.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.945 7fcd94ff5700 1 --2- 172.21.2.112:0/1727801934 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcd900b3af0 0x7fcd900f0580 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 rx=0x7fcd880069a0 tx=0x7fcd8800bd10).stop 2021-02-10T11:31:05.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.945 7fcd94ff5700 1 -- 172.21.2.112:0/1727801934 shutdown_connections 2021-02-10T11:31:05.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.945 7fcd94ff5700 1 --2- 172.21.2.112:0/1727801934 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcd74018730 0x7fcd7401ac10 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:05.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.945 7fcd94ff5700 1 --2- 172.21.2.112:0/1727801934 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcd900b3af0 0x7fcd900f0580 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:05.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.945 7fcd94ff5700 1 --2- 172.21.2.112:0/1727801934 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcd900b30d0 0x7fcd900b34d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:05.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.945 7fcd94ff5700 1 --2- 172.21.2.112:0/1727801934 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcd900b2840 0x7fcd901a4300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:05.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.945 7fcd94ff5700 1 -- 172.21.2.112:0/1727801934 >> 172.21.2.112:0/1727801934 conn(0x7fcd900c36f0 msgr2=0x7fcd90094c30 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:05.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.945 7fcd94ff5700 1 -- 172.21.2.112:0/1727801934 shutdown_connections 2021-02-10T11:31:05.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:05.945 7fcd94ff5700 1 -- 172.21.2.112:0/1727801934 wait complete. 2021-02-10T11:31:05.941 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000348661 2021-02-10T11:31:06.372 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:06.367 7f2c01360700 1 Processor -- start 2021-02-10T11:31:06.372 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:06.368 7f2c01360700 1 -- start start 2021-02-10T11:31:06.372 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:06.368 7f2c01360700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2bfc1b0870 0x7f2bfc0f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:06.372 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:06.368 7f2c01360700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f2bfc0efe00 0x7f2bfc0f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:06.372 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:06.368 7f2c01360700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bfc1b0c70 con 0x7f2bfc1b0470 2021-02-10T11:31:06.372 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:06.368 7f2c01360700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bfc0f0b50 con 0x7f2bfc0efe00 2021-02-10T11:31:06.372 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:06.368 7f2c01360700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bfc0f0820 con 0x7f2bfc1b0870 2021-02-10T11:31:06.372 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:06.368 7f2bfb87e700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f2bfc0efe00 0x7f2bfc0f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:06.373 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:06.368 7f2bfa87c700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2bfc1b0870 0x7f2bfc0f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:06.373 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:06.368 7f2bfb07d700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f2bfc1b0470 0x7f2bfc0ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:47280/0 (socket says 172.21.2.112:47280) 2021-02-10T11:31:06.373 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:06.368 7f2bfb07d700 1 -- 172.21.2.112:0/2274852996 learned_addr learned my addr 172.21.2.112:0/2274852996 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:31:06.373 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:06.369 7f2bfa87c700 1 -- 172.21.2.112:0/2274852996 >> v2:172.21.2.120:3300/0 conn(0x7f2bfc0efe00 msgr2=0x7f2bfc0f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:06.373 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:06.369 7f2bfa87c700 1 --2- 172.21.2.112:0/2274852996 >> v2:172.21.2.120:3300/0 conn(0x7f2bfc0efe00 0x7f2bfc0f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:06.373 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:06.369 7f2bfa87c700 1 -- 172.21.2.112:0/2274852996 >> v1:172.21.2.112:6789/0 conn(0x7f2bfc1b0470 legacy=0x7f2bfc0ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:06.373 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:06.369 7f2bfa87c700 1 -- 172.21.2.112:0/2274852996 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bfc0e52e0 con 0x7f2bfc1b0870 2021-02-10T11:31:06.373 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:06.369 7f2bfa07b700 1 -- 172.21.2.112:0/2274852996 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 631171781 0 0) 0x7f2bec001180 con 0x7f2bfc1b0470 2021-02-10T11:31:06.373 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:06.369 7f2bfb87e700 1 --2- 172.21.2.112:0/2274852996 >> v2:172.21.2.120:3300/0 conn(0x7f2bfc0efe00 0x7f2bfc0f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:31:06.373 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:06.369 7f2bfa87c700 1 --2- 172.21.2.112:0/2274852996 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2bfc1b0870 0x7f2bfc0f20e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f2bf0006650 tx=0x7f2bf0009a80).ready entity=mon.2 client_cookie=9887362147cb322f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:06.373 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:06.370 7f2bfa07b700 1 -- 172.21.2.112:0/2274852996 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2bf00009b0 con 0x7f2bfc1b0870 2021-02-10T11:31:06.373 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:06.370 7f2bfa07b700 1 -- 172.21.2.112:0/2274852996 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bf0006740 con 0x7f2bfc1b0870 2021-02-10T11:31:06.374 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:06.370 7f2bfa07b700 1 -- 172.21.2.112:0/2274852996 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2bf000c0c0 con 0x7f2bfc1b0870 2021-02-10T11:31:06.374 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:06.370 7f2c01360700 1 -- 172.21.2.112:0/2274852996 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2bfc1b0870 msgr2=0x7f2bfc0f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:06.374 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:06.370 7f2c01360700 1 --2- 172.21.2.112:0/2274852996 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2bfc1b0870 0x7f2bfc0f20e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f2bf0006650 tx=0x7f2bf0009a80).stop 2021-02-10T11:31:06.374 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:06.370 7f2c01360700 1 -- 172.21.2.112:0/2274852996 shutdown_connections 2021-02-10T11:31:06.374 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:06.370 7f2c01360700 1 --2- 172.21.2.112:0/2274852996 >> v2:172.21.2.120:3300/0 conn(0x7f2bfc0efe00 0x7f2bfc0f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:06.374 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:06.370 7f2c01360700 1 --2- 172.21.2.112:0/2274852996 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2bfc1b0870 0x7f2bfc0f20e0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:06.374 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:06.370 7f2c01360700 1 -- 172.21.2.112:0/2274852996 >> 172.21.2.112:0/2274852996 conn(0x7f2bfc0ce3e0 msgr2=0x7f2bfc0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:06.374 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:06.370 7f2c01360700 1 -- 172.21.2.112:0/2274852996 shutdown_connections 2021-02-10T11:31:06.374 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:06.370 7f2c01360700 1 -- 172.21.2.112:0/2274852996 wait complete. 2021-02-10T11:31:06.376 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:06.385 7f2c01360700 1 Processor -- start 2021-02-10T11:31:06.376 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:06.385 7f2c01360700 1 -- start start 2021-02-10T11:31:06.376 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:06.385 7f2c01360700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2bfc1a42e0 0x7f2bfc09ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:06.376 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:06.385 7f2c01360700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2bfc1a46e0 0x7f2bfc09f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:06.376 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:06.385 7f2c01360700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2bfc08ad30 0x7f2bfc08b130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:06.376 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:06.385 7f2c01360700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bfc08a7b0 con 0x7f2bfc1a42e0 2021-02-10T11:31:06.376 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:06.385 7f2c01360700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bfc0a65f0 con 0x7f2bfc1a46e0 2021-02-10T11:31:06.376 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:06.385 7f2c01360700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bfc08a430 con 0x7f2bfc08ad30 2021-02-10T11:31:06.376 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:06.386 7f2bfb07d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2bfc1a42e0 0x7f2bfc09ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:06.376 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:06.386 7f2bfb87e700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2bfc08ad30 0x7f2bfc08b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:06.377 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:06.386 7f2bfb07d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2bfc1a42e0 0x7f2bfc09ecf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:47826/0 (socket says 172.21.2.112:47826) 2021-02-10T11:31:06.377 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:06.386 7f2bfb07d700 1 -- 172.21.2.112:0/2431342254 learned_addr learned my addr 172.21.2.112:0/2431342254 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:31:06.377 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:06.386 7f2bfa87c700 1 --2- 172.21.2.112:0/2431342254 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2bfc1a46e0 0x7f2bfc09f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:06.377 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:06.386 7f2bfb07d700 1 -- 172.21.2.112:0/2431342254 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2bfc08ad30 msgr2=0x7f2bfc08b130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:06.377 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:06.386 7f2bfb07d700 1 --2- 172.21.2.112:0/2431342254 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2bfc08ad30 0x7f2bfc08b130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:06.377 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:06.386 7f2bfb07d700 1 -- 172.21.2.112:0/2431342254 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2bfc1a46e0 msgr2=0x7f2bfc09f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:06.377 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:06.386 7f2bfb07d700 1 --2- 172.21.2.112:0/2431342254 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2bfc1a46e0 0x7f2bfc09f310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:06.377 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:06.386 7f2bfb07d700 1 -- 172.21.2.112:0/2431342254 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bfc09fa50 con 0x7f2bfc1a42e0 2021-02-10T11:31:06.377 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:06.386 7f2bfb87e700 1 --2- 172.21.2.112:0/2431342254 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2bfc08ad30 0x7f2bfc08b130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:31:06.377 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:06.386 7f2bfb07d700 1 --2- 172.21.2.112:0/2431342254 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2bfc1a42e0 0x7f2bfc09ecf0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 rx=0x7f2bec00a570 tx=0x7f2bec00a910).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:06.377 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:06.387 7f2be3fff700 1 -- 172.21.2.112:0/2431342254 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2bec00ad00 con 0x7f2bfc1a42e0 2021-02-10T11:31:06.377 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:06.387 7f2be3fff700 1 -- 172.21.2.112:0/2431342254 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bec001180 con 0x7f2bfc1a42e0 2021-02-10T11:31:06.377 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:06.387 7f2be3fff700 1 -- 172.21.2.112:0/2431342254 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2bec004590 con 0x7f2bfc1a42e0 2021-02-10T11:31:06.378 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:06.387 7f2c01360700 1 -- 172.21.2.112:0/2431342254 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bfc09fa50 con 0x7f2bfc1a42e0 2021-02-10T11:31:06.378 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:06.387 7f2c01360700 1 -- 172.21.2.112:0/2431342254 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bfc09fa50 con 0x7f2bfc1a42e0 2021-02-10T11:31:06.378 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:06.387 7f2be3fff700 1 -- 172.21.2.112:0/2431342254 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2bec0043a0 con 0x7f2bfc1a42e0 2021-02-10T11:31:06.378 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:06.387 7f2be3fff700 1 --2- 172.21.2.112:0/2431342254 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2bdc0186c0 0x7f2bdc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:06.378 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:06.387 7f2be3fff700 1 -- 172.21.2.112:0/2431342254 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f2bec032580 con 0x7f2bfc1a42e0 2021-02-10T11:31:06.378 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:06.388 7f2bfa87c700 1 --2- 172.21.2.112:0/2431342254 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2bdc0186c0 0x7f2bdc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:06.378 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:06.388 7f2bfa87c700 1 --2- 172.21.2.112:0/2431342254 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2bdc0186c0 0x7f2bdc01abc0 crc :-1 s=READY pgs=482 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:06.405 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:06.417 7f2c01360700 1 -- 172.21.2.112:0/2431342254 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bfc105ae0 con 0x7f2bfc1a42e0 2021-02-10T11:31:06.407 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:06.420 7f2be3fff700 1 -- 172.21.2.112:0/2431342254 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2bec000c90 con 0x7f2bfc1a42e0 2021-02-10T11:31:06.535 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:06.547 7f2c01360700 1 -- 172.21.2.112:0/2431342254 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348661", "target": ["mgr", ""]}) v1 -- 0x7f2bfc105ae0 con 0x7f2bdc0186c0 2021-02-10T11:31:06.545 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:06.558 7f2be3fff700 1 -- 172.21.2.112:0/2431342254 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f2bf0006420 con 0x7f2bdc0186c0 2021-02-10T11:31:06.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:06.565 7f2c01360700 1 -- 172.21.2.112:0/2431342254 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2bdc0186c0 msgr2=0x7f2bdc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:06.553 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:06.565 7f2c01360700 1 --2- 172.21.2.112:0/2431342254 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2bdc0186c0 0x7f2bdc01abc0 crc :-1 s=READY pgs=482 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:06.553 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:06.565 7f2c01360700 1 -- 172.21.2.112:0/2431342254 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2bfc1a42e0 msgr2=0x7f2bfc09ecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:06.553 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:06.565 7f2c01360700 1 --2- 172.21.2.112:0/2431342254 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2bfc1a42e0 0x7f2bfc09ecf0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 rx=0x7f2bec00a570 tx=0x7f2bec00a910).stop 2021-02-10T11:31:06.553 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:06.565 7f2c01360700 1 -- 172.21.2.112:0/2431342254 shutdown_connections 2021-02-10T11:31:06.553 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:06.565 7f2c01360700 1 --2- 172.21.2.112:0/2431342254 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2bdc0186c0 0x7f2bdc01abc0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:06.553 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:06.565 7f2c01360700 1 --2- 172.21.2.112:0/2431342254 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2bfc1a46e0 0x7f2bfc09f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:06.553 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:06.565 7f2c01360700 1 --2- 172.21.2.112:0/2431342254 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2bfc08ad30 0x7f2bfc08b130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:06.553 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:06.565 7f2c01360700 1 --2- 172.21.2.112:0/2431342254 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2bfc1a42e0 0x7f2bfc09ecf0 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:06.553 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:06.565 7f2c01360700 1 -- 172.21.2.112:0/2431342254 >> 172.21.2.112:0/2431342254 conn(0x7f2bfc095a70 msgr2=0x7f2bfc0bf050 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:06.554 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:06.566 7f2c01360700 1 -- 172.21.2.112:0/2431342254 shutdown_connections 2021-02-10T11:31:06.554 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:06.566 7f2c01360700 1 -- 172.21.2.112:0/2431342254 wait complete. 2021-02-10T11:31:06.561 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000348662 2021-02-10T11:31:06.991 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:06.987 7fb9c869d700 1 Processor -- start 2021-02-10T11:31:06.991 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:06.987 7fb9c869d700 1 -- start start 2021-02-10T11:31:06.991 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:06.988 7fb9c869d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb9c01b0470 0x7fb9c01b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:06.991 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:06.988 7fb9c869d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fb9c00f20e0 0x7fb9c00f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:06.991 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:06.988 7fb9c869d700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9c00f0b50 con 0x7fb9c00ef610 2021-02-10T11:31:06.991 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:06.988 7fb9c869d700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9c00f0820 con 0x7fb9c00f20e0 2021-02-10T11:31:06.991 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:06.988 7fb9c869d700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9c00f24e0 con 0x7fb9c01b0470 2021-02-10T11:31:06.991 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:06.988 7fb9c6cba700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fb9c00f20e0 0x7fb9c00f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:06.992 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:06.988 7fb9c6cba700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fb9c00f20e0 0x7fb9c00f0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:47836/0 (socket says 172.21.2.112:47836) 2021-02-10T11:31:06.992 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:06.988 7fb9c6cba700 1 -- 172.21.2.112:0/4256770079 learned_addr learned my addr 172.21.2.112:0/4256770079 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:31:06.992 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:06.988 7fb9c64b9700 1 --2- 172.21.2.112:0/4256770079 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb9c01b0470 0x7fb9c01b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:06.992 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:06.988 7fb9c6cba700 1 -- 172.21.2.112:0/4256770079 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb9c01b0470 msgr2=0x7fb9c01b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:06.992 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:06.988 7fb9c6cba700 1 --2- 172.21.2.112:0/4256770079 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb9c01b0470 0x7fb9c01b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:06.992 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:06.988 7fb9c6cba700 1 -- 172.21.2.112:0/4256770079 >> v1:172.21.2.120:6789/0 conn(0x7fb9c00ef610 legacy=0x7fb9c00efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:06.992 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:06.988 7fb9c6cba700 1 -- 172.21.2.112:0/4256770079 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9c00e52e0 con 0x7fb9c00f20e0 2021-02-10T11:31:06.992 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:06.989 7fb9c6cba700 1 --2- 172.21.2.112:0/4256770079 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb9c00f20e0 0x7fb9c00f0200 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 rx=0x7fb9bc006b80 tx=0x7fb9bc009b90).ready entity=mon.0 client_cookie=af6e98194dcfc503 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:06.992 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:06.989 7fb9c54b7700 1 -- 172.21.2.112:0/4256770079 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb9bc0022e0 con 0x7fb9c00f20e0 2021-02-10T11:31:06.992 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:06.989 7fb9c54b7700 1 -- 172.21.2.112:0/4256770079 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9bc002620 con 0x7fb9c00f20e0 2021-02-10T11:31:06.992 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:06.989 7fb9c54b7700 1 -- 172.21.2.112:0/4256770079 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb9bc002b00 con 0x7fb9c00f20e0 2021-02-10T11:31:06.992 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:06.989 7fb9c869d700 1 -- 172.21.2.112:0/4256770079 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb9c00f20e0 msgr2=0x7fb9c00f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:06.992 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:06.989 7fb9c869d700 1 --2- 172.21.2.112:0/4256770079 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb9c00f20e0 0x7fb9c00f0200 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 rx=0x7fb9bc006b80 tx=0x7fb9bc009b90).stop 2021-02-10T11:31:06.993 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:06.989 7fb9c869d700 1 -- 172.21.2.112:0/4256770079 shutdown_connections 2021-02-10T11:31:06.993 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:06.989 7fb9c869d700 1 --2- 172.21.2.112:0/4256770079 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb9c00f20e0 0x7fb9c00f0200 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:06.993 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:06.989 7fb9c869d700 1 --2- 172.21.2.112:0/4256770079 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb9c01b0470 0x7fb9c01b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:06.993 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:06.989 7fb9c869d700 1 -- 172.21.2.112:0/4256770079 >> 172.21.2.112:0/4256770079 conn(0x7fb9c00ce3e0 msgr2=0x7fb9c00e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:06.993 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:06.990 7fb9c869d700 1 -- 172.21.2.112:0/4256770079 shutdown_connections 2021-02-10T11:31:06.993 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:06.990 7fb9c869d700 1 -- 172.21.2.112:0/4256770079 wait complete. 2021-02-10T11:31:06.994 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.004 7fb9c869d700 1 Processor -- start 2021-02-10T11:31:06.995 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.004 7fb9c869d700 1 -- start start 2021-02-10T11:31:06.995 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.004 7fb9c869d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb9c00b7de0 0x7fb9c0190920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:06.995 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.004 7fb9c869d700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb9c00b82f0 0x7fb9c00b86f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:06.995 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.004 7fb9c869d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb9c00b8d10 0x7fb9c00bf9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:06.995 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.004 7fb9c869d700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9c00b7930 con 0x7fb9c00b8d10 2021-02-10T11:31:06.995 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.004 7fb9c869d700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9c00b7b20 con 0x7fb9c00b82f0 2021-02-10T11:31:06.995 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.004 7fb9c869d700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9c00bffd0 con 0x7fb9c00b7de0 2021-02-10T11:31:06.995 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.004 7fb9c6cba700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb9c00b8d10 0x7fb9c00bf9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:06.995 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.004 7fb9c6cba700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb9c00b8d10 0x7fb9c00bf9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:47838/0 (socket says 172.21.2.112:47838) 2021-02-10T11:31:06.995 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.004 7fb9c5cb8700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb9c00b82f0 0x7fb9c00b86f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:06.995 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.004 7fb9c6cba700 1 -- 172.21.2.112:0/1591958628 learned_addr learned my addr 172.21.2.112:0/1591958628 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:31:06.995 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.004 7fb9c64b9700 1 --2- 172.21.2.112:0/1591958628 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb9c00b7de0 0x7fb9c0190920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:06.995 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.005 7fb9c6cba700 1 -- 172.21.2.112:0/1591958628 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb9c00b7de0 msgr2=0x7fb9c0190920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:06.996 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.005 7fb9c6cba700 1 --2- 172.21.2.112:0/1591958628 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb9c00b7de0 0x7fb9c0190920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:06.996 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.005 7fb9c6cba700 1 -- 172.21.2.112:0/1591958628 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb9c00b82f0 msgr2=0x7fb9c00b86f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:06.996 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.005 7fb9c6cba700 1 --2- 172.21.2.112:0/1591958628 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb9c00b82f0 0x7fb9c00b86f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:06.996 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.005 7fb9c6cba700 1 -- 172.21.2.112:0/1591958628 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9c00c01c0 con 0x7fb9c00b8d10 2021-02-10T11:31:06.996 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.005 7fb9c6cba700 1 --2- 172.21.2.112:0/1591958628 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb9c00b8d10 0x7fb9c00bf9b0 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 rx=0x7fb9bc006f10 tx=0x7fb9bc00b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:06.996 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.005 7fb9b6ffd700 1 -- 172.21.2.112:0/1591958628 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb9bc009100 con 0x7fb9c00b8d10 2021-02-10T11:31:06.996 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.005 7fb9b6ffd700 1 -- 172.21.2.112:0/1591958628 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9bc00bd50 con 0x7fb9c00b8d10 2021-02-10T11:31:06.996 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.005 7fb9c869d700 1 -- 172.21.2.112:0/1591958628 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9c00c01c0 con 0x7fb9c00b8d10 2021-02-10T11:31:06.996 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.005 7fb9c869d700 1 -- 172.21.2.112:0/1591958628 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9c00c01c0 con 0x7fb9c00b8d10 2021-02-10T11:31:06.996 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.005 7fb9b6ffd700 1 -- 172.21.2.112:0/1591958628 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb9bc002220 con 0x7fb9c00b8d10 2021-02-10T11:31:06.996 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.006 7fb9b6ffd700 1 -- 172.21.2.112:0/1591958628 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb9bc009100 con 0x7fb9c00b8d10 2021-02-10T11:31:06.996 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.006 7fb9b6ffd700 1 --2- 172.21.2.112:0/1591958628 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb9a8018730 0x7fb9a801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:06.996 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.006 7fb9b6ffd700 1 -- 172.21.2.112:0/1591958628 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fb9bc032630 con 0x7fb9c00b8d10 2021-02-10T11:31:06.996 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.006 7fb9c64b9700 1 --2- 172.21.2.112:0/1591958628 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb9a8018730 0x7fb9a801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:06.997 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.007 7fb9c64b9700 1 --2- 172.21.2.112:0/1591958628 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb9a8018730 0x7fb9a801ac10 crc :-1 s=READY pgs=483 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:07.024 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.037 7fb9c869d700 1 -- 172.21.2.112:0/1591958628 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9c0105b30 con 0x7fb9c00b8d10 2021-02-10T11:31:07.027 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.040 7fb9b6ffd700 1 -- 172.21.2.112:0/1591958628 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb9bc000b90 con 0x7fb9c00b8d10 2021-02-10T11:31:07.158 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.171 7fb9c869d700 1 -- 172.21.2.112:0/1591958628 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348662", "target": ["mgr", ""]}) v1 -- 0x7fb9c0105b30 con 0x7fb9a8018730 2021-02-10T11:31:07.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.180 7fb9b6ffd700 1 -- 172.21.2.112:0/1591958628 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fb9b0006a70 con 0x7fb9a8018730 2021-02-10T11:31:07.176 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.188 7fb9c869d700 1 -- 172.21.2.112:0/1591958628 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb9a8018730 msgr2=0x7fb9a801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:07.176 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.188 7fb9c869d700 1 --2- 172.21.2.112:0/1591958628 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb9a8018730 0x7fb9a801ac10 crc :-1 s=READY pgs=483 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:07.176 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.188 7fb9c869d700 1 -- 172.21.2.112:0/1591958628 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb9c00b8d10 msgr2=0x7fb9c00bf9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:07.176 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.188 7fb9c869d700 1 --2- 172.21.2.112:0/1591958628 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb9c00b8d10 0x7fb9c00bf9b0 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 rx=0x7fb9bc006f10 tx=0x7fb9bc00b960).stop 2021-02-10T11:31:07.176 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.188 7fb9c869d700 1 -- 172.21.2.112:0/1591958628 shutdown_connections 2021-02-10T11:31:07.176 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.188 7fb9c869d700 1 --2- 172.21.2.112:0/1591958628 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb9a8018730 0x7fb9a801ac10 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:07.176 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.188 7fb9c869d700 1 --2- 172.21.2.112:0/1591958628 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb9c00b82f0 0x7fb9c00b86f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:07.176 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.188 7fb9c869d700 1 --2- 172.21.2.112:0/1591958628 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb9c00b8d10 0x7fb9c00bf9b0 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:07.176 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.188 7fb9c869d700 1 --2- 172.21.2.112:0/1591958628 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb9c00b7de0 0x7fb9c0190920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:07.176 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.188 7fb9c869d700 1 -- 172.21.2.112:0/1591958628 >> 172.21.2.112:0/1591958628 conn(0x7fb9c0094e10 msgr2=0x7fb9c00a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:07.177 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.189 7fb9c869d700 1 -- 172.21.2.112:0/1591958628 shutdown_connections 2021-02-10T11:31:07.177 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.189 7fb9c869d700 1 -- 172.21.2.112:0/1591958628 wait complete. 2021-02-10T11:31:07.184 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000348663 2021-02-10T11:31:07.616 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.612 7ff2a658f700 1 Processor -- start 2021-02-10T11:31:07.616 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.612 7ff2a658f700 1 -- start start 2021-02-10T11:31:07.616 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.612 7ff2a658f700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7ff2a01b0870 0x7ff2a00f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:07.616 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.612 7ff2a658f700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7ff2a00efe00 0x7ff2a00f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:07.616 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.612 7ff2a658f700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2a01b0c70 con 0x7ff2a01b0470 2021-02-10T11:31:07.616 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.612 7ff2a658f700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2a00e5ea0 con 0x7ff2a01b0870 2021-02-10T11:31:07.616 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.612 7ff2a658f700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2a00f0b50 con 0x7ff2a00efe00 2021-02-10T11:31:07.616 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.613 7ff29ffff700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7ff2a01b0470 0x7ff2a00ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:47308/0 (socket says 172.21.2.112:47308) 2021-02-10T11:31:07.616 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.613 7ff29f7fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7ff2a01b0870 0x7ff2a00f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:07.617 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.613 7ff29ffff700 1 -- 172.21.2.112:0/2966402099 learned_addr learned my addr 172.21.2.112:0/2966402099 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:31:07.617 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.613 7ff2a4bac700 1 --2- 172.21.2.112:0/2966402099 >> v2:172.21.2.120:3300/0 conn(0x7ff2a00efe00 0x7ff2a00f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:07.617 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.613 7ff29f7fe700 1 -- 172.21.2.112:0/2966402099 >> v2:172.21.2.120:3300/0 conn(0x7ff2a00efe00 msgr2=0x7ff2a00f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:07.617 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.613 7ff29f7fe700 1 --2- 172.21.2.112:0/2966402099 >> v2:172.21.2.120:3300/0 conn(0x7ff2a00efe00 0x7ff2a00f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:07.617 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.613 7ff29f7fe700 1 -- 172.21.2.112:0/2966402099 >> v1:172.21.2.112:6789/0 conn(0x7ff2a01b0470 legacy=0x7ff2a00ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:07.617 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.613 7ff29f7fe700 1 -- 172.21.2.112:0/2966402099 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2a00f0820 con 0x7ff2a01b0870 2021-02-10T11:31:07.617 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.613 7ff2a4bac700 1 --2- 172.21.2.112:0/2966402099 >> v2:172.21.2.120:3300/0 conn(0x7ff2a00efe00 0x7ff2a00f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:31:07.617 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.613 7ff29f7fe700 1 --2- 172.21.2.112:0/2966402099 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff2a01b0870 0x7ff2a00f20e0 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 rx=0x7ff294006650 tx=0x7ff294009a80).ready entity=mon.0 client_cookie=dee2bfdb75bc3d10 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:07.617 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.614 7ff29effd700 1 -- 172.21.2.112:0/2966402099 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff294006390 con 0x7ff2a01b0870 2021-02-10T11:31:07.617 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.614 7ff29effd700 1 -- 172.21.2.112:0/2966402099 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2940009b0 con 0x7ff2a01b0870 2021-02-10T11:31:07.617 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.614 7ff29effd700 1 -- 172.21.2.112:0/2966402099 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff29400c2b0 con 0x7ff2a01b0870 2021-02-10T11:31:07.617 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.614 7ff2a658f700 1 -- 172.21.2.112:0/2966402099 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff2a01b0870 msgr2=0x7ff2a00f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:07.617 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.614 7ff2a658f700 1 --2- 172.21.2.112:0/2966402099 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff2a01b0870 0x7ff2a00f20e0 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 rx=0x7ff294006650 tx=0x7ff294009a80).stop 2021-02-10T11:31:07.618 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.614 7ff2a658f700 1 -- 172.21.2.112:0/2966402099 shutdown_connections 2021-02-10T11:31:07.618 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.614 7ff2a658f700 1 --2- 172.21.2.112:0/2966402099 >> v2:172.21.2.120:3300/0 conn(0x7ff2a00efe00 0x7ff2a00f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:07.618 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.614 7ff2a658f700 1 --2- 172.21.2.112:0/2966402099 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff2a01b0870 0x7ff2a00f20e0 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:07.618 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.614 7ff2a658f700 1 -- 172.21.2.112:0/2966402099 >> 172.21.2.112:0/2966402099 conn(0x7ff2a00ce3e0 msgr2=0x7ff2a00e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:07.618 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.614 7ff2a658f700 1 -- 172.21.2.112:0/2966402099 shutdown_connections 2021-02-10T11:31:07.618 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.614 7ff2a658f700 1 -- 172.21.2.112:0/2966402099 wait complete. 2021-02-10T11:31:07.620 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.629 7ff2a658f700 1 Processor -- start 2021-02-10T11:31:07.620 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.629 7ff2a658f700 1 -- start start 2021-02-10T11:31:07.620 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.629 7ff2a658f700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff2a00a0420 0x7ff2a00a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:07.620 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.629 7ff2a658f700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff2a0189d00 0x7ff2a018a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:07.620 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.629 7ff2a658f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff2a018a720 0x7ff2a00b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:07.620 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.629 7ff2a658f700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2a0189780 con 0x7ff2a018a720 2021-02-10T11:31:07.620 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.629 7ff2a658f700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2a018ab20 con 0x7ff2a0189d00 2021-02-10T11:31:07.620 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.629 7ff2a658f700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2a00a1140 con 0x7ff2a00a0420 2021-02-10T11:31:07.620 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.630 7ff2a4bac700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff2a018a720 0x7ff2a00b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:07.620 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.630 7ff2a4bac700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff2a018a720 0x7ff2a00b7be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:47850/0 (socket says 172.21.2.112:47850) 2021-02-10T11:31:07.621 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.630 7ff29f7fe700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff2a0189d00 0x7ff2a018a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:07.621 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.630 7ff2a4bac700 1 -- 172.21.2.112:0/2665641223 learned_addr learned my addr 172.21.2.112:0/2665641223 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:31:07.621 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.630 7ff29ffff700 1 --2- 172.21.2.112:0/2665641223 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff2a00a0420 0x7ff2a00a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:07.621 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.630 7ff2a4bac700 1 -- 172.21.2.112:0/2665641223 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff2a00a0420 msgr2=0x7ff2a00a0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:07.621 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.630 7ff2a4bac700 1 --2- 172.21.2.112:0/2665641223 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff2a00a0420 0x7ff2a00a0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:07.621 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.630 7ff2a4bac700 1 -- 172.21.2.112:0/2665641223 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff2a0189d00 msgr2=0x7ff2a018a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:07.621 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.630 7ff2a4bac700 1 --2- 172.21.2.112:0/2665641223 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff2a0189d00 0x7ff2a018a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:07.621 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.630 7ff2a4bac700 1 -- 172.21.2.112:0/2665641223 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2a00b8200 con 0x7ff2a018a720 2021-02-10T11:31:07.621 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.630 7ff29f7fe700 1 --2- 172.21.2.112:0/2665641223 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff2a0189d00 0x7ff2a018a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:31:07.621 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.630 7ff2a4bac700 1 --2- 172.21.2.112:0/2665641223 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff2a018a720 0x7ff2a00b7be0 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 rx=0x7ff298009980 tx=0x7ff29800bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:07.621 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.630 7ff29cff9700 1 -- 172.21.2.112:0/2665641223 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff29800c2f0 con 0x7ff2a018a720 2021-02-10T11:31:07.621 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.631 7ff29cff9700 1 -- 172.21.2.112:0/2665641223 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff298006790 con 0x7ff2a018a720 2021-02-10T11:31:07.621 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.631 7ff2a658f700 1 -- 172.21.2.112:0/2665641223 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2a00b8200 con 0x7ff2a018a720 2021-02-10T11:31:07.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.631 7ff29cff9700 1 -- 172.21.2.112:0/2665641223 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff29800c870 con 0x7ff2a018a720 2021-02-10T11:31:07.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.631 7ff2a658f700 1 -- 172.21.2.112:0/2665641223 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2a00b8200 con 0x7ff2a018a720 2021-02-10T11:31:07.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.631 7ff29cff9700 1 -- 172.21.2.112:0/2665641223 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff2980011f0 con 0x7ff2a018a720 2021-02-10T11:31:07.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.631 7ff29cff9700 1 --2- 172.21.2.112:0/2665641223 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff280018730 0x7ff28001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:07.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.631 7ff29cff9700 1 -- 172.21.2.112:0/2665641223 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7ff298032a50 con 0x7ff2a018a720 2021-02-10T11:31:07.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.631 7ff29ffff700 1 --2- 172.21.2.112:0/2665641223 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff280018730 0x7ff28001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:07.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.632 7ff29ffff700 1 --2- 172.21.2.112:0/2665641223 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff280018730 0x7ff28001ac10 crc :-1 s=READY pgs=484 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:07.649 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.662 7ff2a658f700 1 -- 172.21.2.112:0/2665641223 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2a0105ae0 con 0x7ff2a018a720 2021-02-10T11:31:07.652 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.665 7ff29cff9700 1 -- 172.21.2.112:0/2665641223 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff29800be10 con 0x7ff2a018a720 2021-02-10T11:31:07.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.798 7ff2a658f700 1 -- 172.21.2.112:0/2665641223 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348663", "target": ["mgr", ""]}) v1 -- 0x7ff2a0189780 con 0x7ff280018730 2021-02-10T11:31:07.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.807 7ff29cff9700 1 -- 172.21.2.112:0/2665641223 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7ff290000fa0 con 0x7ff280018730 2021-02-10T11:31:07.803 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.815 7ff2a658f700 1 -- 172.21.2.112:0/2665641223 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff280018730 msgr2=0x7ff28001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:07.803 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.815 7ff2a658f700 1 --2- 172.21.2.112:0/2665641223 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff280018730 0x7ff28001ac10 crc :-1 s=READY pgs=484 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:07.803 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.815 7ff2a658f700 1 -- 172.21.2.112:0/2665641223 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff2a018a720 msgr2=0x7ff2a00b7be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:07.803 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.815 7ff2a658f700 1 --2- 172.21.2.112:0/2665641223 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff2a018a720 0x7ff2a00b7be0 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 rx=0x7ff298009980 tx=0x7ff29800bcd0).stop 2021-02-10T11:31:07.803 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.815 7ff2a658f700 1 -- 172.21.2.112:0/2665641223 shutdown_connections 2021-02-10T11:31:07.803 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.815 7ff2a658f700 1 --2- 172.21.2.112:0/2665641223 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff280018730 0x7ff28001ac10 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:07.803 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.815 7ff2a658f700 1 --2- 172.21.2.112:0/2665641223 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff2a0189d00 0x7ff2a018a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:07.804 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.815 7ff2a658f700 1 --2- 172.21.2.112:0/2665641223 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff2a018a720 0x7ff2a00b7be0 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:07.804 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.815 7ff2a658f700 1 --2- 172.21.2.112:0/2665641223 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff2a00a0420 0x7ff2a00a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:07.804 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.815 7ff2a658f700 1 -- 172.21.2.112:0/2665641223 >> 172.21.2.112:0/2665641223 conn(0x7ff2a0094e10 msgr2=0x7ff2a00a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:07.804 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.816 7ff2a658f700 1 -- 172.21.2.112:0/2665641223 shutdown_connections 2021-02-10T11:31:07.804 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:07.816 7ff2a658f700 1 -- 172.21.2.112:0/2665641223 wait complete. 2021-02-10T11:31:07.811 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000348664 2021-02-10T11:31:08.242 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.237 7fe19d6c9700 1 Processor -- start 2021-02-10T11:31:08.242 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.237 7fe19d6c9700 1 -- start start 2021-02-10T11:31:08.242 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.238 7fe19d6c9700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe1981b0470 0x7fe1981b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:08.242 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.238 7fe19d6c9700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1980f0b50 con 0x7fe1980f20e0 2021-02-10T11:31:08.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.238 7fe19d6c9700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1980f24e0 con 0x7fe1980ef610 2021-02-10T11:31:08.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.238 7fe19d6c9700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1980e5ea0 con 0x7fe1981b0470 2021-02-10T11:31:08.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.238 7fe197ac0700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fe1980f20e0 0x7fe1980f0200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:47320/0 (socket says 172.21.2.112:47320) 2021-02-10T11:31:08.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.238 7fe197ac0700 1 -- 172.21.2.112:0/1336461793 learned_addr learned my addr 172.21.2.112:0/1336461793 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:31:08.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.238 7fe1972bf700 1 --2- 172.21.2.112:0/1336461793 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe1981b0470 0x7fe1981b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:08.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.238 7fe1962bd700 1 -- 172.21.2.112:0/1336461793 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 787971226 0 0) 0x7fe190000f90 con 0x7fe1980f20e0 2021-02-10T11:31:08.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.238 7fe1962bd700 1 -- 172.21.2.112:0/1336461793 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe184002fe0 con 0x7fe1980f20e0 2021-02-10T11:31:08.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.238 7fe1962bd700 1 -- 172.21.2.112:0/1336461793 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1533301673 0 0) 0x7fe188000ff0 con 0x7fe1980ef610 2021-02-10T11:31:08.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.238 7fe1962bd700 1 -- 172.21.2.112:0/1336461793 --> v1:172.21.2.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1840042c0 con 0x7fe1980ef610 2021-02-10T11:31:08.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.239 7fe1972bf700 1 -- 172.21.2.112:0/1336461793 >> v1:172.21.2.120:6789/0 conn(0x7fe1980ef610 legacy=0x7fe1980efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:08.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.239 7fe1972bf700 1 -- 172.21.2.112:0/1336461793 >> v1:172.21.2.112:6789/0 conn(0x7fe1980f20e0 legacy=0x7fe1980f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:08.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.239 7fe1972bf700 1 -- 172.21.2.112:0/1336461793 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1980e52e0 con 0x7fe1981b0470 2021-02-10T11:31:08.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.239 7fe1972bf700 1 --2- 172.21.2.112:0/1336461793 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe1981b0470 0x7fe1981b0870 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7fe180006650 tx=0x7fe180009a80).ready entity=mon.2 client_cookie=e11c209876d2ecfa server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:08.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.239 7fe1962bd700 1 -- 172.21.2.112:0/1336461793 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe18000c0c0 con 0x7fe1981b0470 2021-02-10T11:31:08.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.239 7fe1962bd700 1 -- 172.21.2.112:0/1336461793 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe18000c340 con 0x7fe1981b0470 2021-02-10T11:31:08.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.239 7fe1962bd700 1 -- 172.21.2.112:0/1336461793 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe18000c860 con 0x7fe1981b0470 2021-02-10T11:31:08.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.239 7fe19d6c9700 1 -- 172.21.2.112:0/1336461793 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe1981b0470 msgr2=0x7fe1981b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:08.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.239 7fe19d6c9700 1 --2- 172.21.2.112:0/1336461793 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe1981b0470 0x7fe1981b0870 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7fe180006650 tx=0x7fe180009a80).stop 2021-02-10T11:31:08.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.240 7fe19d6c9700 1 -- 172.21.2.112:0/1336461793 shutdown_connections 2021-02-10T11:31:08.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.240 7fe19d6c9700 1 --2- 172.21.2.112:0/1336461793 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe1981b0470 0x7fe1981b0870 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:08.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.240 7fe19d6c9700 1 -- 172.21.2.112:0/1336461793 >> 172.21.2.112:0/1336461793 conn(0x7fe1980ce3e0 msgr2=0x7fe1980e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:08.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.240 7fe19d6c9700 1 -- 172.21.2.112:0/1336461793 shutdown_connections 2021-02-10T11:31:08.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.240 7fe19d6c9700 1 -- 172.21.2.112:0/1336461793 wait complete. 2021-02-10T11:31:08.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.255 7fe19d6c9700 1 Processor -- start 2021-02-10T11:31:08.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.255 7fe19d6c9700 1 -- start start 2021-02-10T11:31:08.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.255 7fe19d6c9700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe198190920 0x7fe1980b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:08.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.255 7fe19d6c9700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe1980b8200 0x7fe1980b8600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:08.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.255 7fe19d6c9700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe1980c1d80 0x7fe1980c2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:08.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.255 7fe19d6c9700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe198190d20 con 0x7fe1980b8200 2021-02-10T11:31:08.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.255 7fe19d6c9700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1980b8c20 con 0x7fe198190920 2021-02-10T11:31:08.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.255 7fe19d6c9700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1980c27a0 con 0x7fe1980c1d80 2021-02-10T11:31:08.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.255 7fe196abe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe1980b8200 0x7fe1980b8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:08.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.256 7fe197ac0700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe1980c1d80 0x7fe1980c2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:08.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.256 7fe196abe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe1980b8200 0x7fe1980b8600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:47866/0 (socket says 172.21.2.112:47866) 2021-02-10T11:31:08.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.256 7fe196abe700 1 -- 172.21.2.112:0/2555304020 learned_addr learned my addr 172.21.2.112:0/2555304020 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:31:08.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.256 7fe1972bf700 1 --2- 172.21.2.112:0/2555304020 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe198190920 0x7fe1980b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:08.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.256 7fe197ac0700 1 -- 172.21.2.112:0/2555304020 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe198190920 msgr2=0x7fe1980b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:08.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.256 7fe197ac0700 1 --2- 172.21.2.112:0/2555304020 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe198190920 0x7fe1980b7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:08.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.256 7fe197ac0700 1 -- 172.21.2.112:0/2555304020 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe1980b8200 msgr2=0x7fe1980b8600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:08.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.256 7fe197ac0700 1 --2- 172.21.2.112:0/2555304020 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe1980b8200 0x7fe1980b8600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:08.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.256 7fe197ac0700 1 -- 172.21.2.112:0/2555304020 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1980c29c0 con 0x7fe1980c1d80 2021-02-10T11:31:08.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.256 7fe196abe700 1 --2- 172.21.2.112:0/2555304020 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe1980b8200 0x7fe1980b8600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:31:08.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.256 7fe197ac0700 1 --2- 172.21.2.112:0/2555304020 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe1980c1d80 0x7fe1980c2180 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7fe19000b890 tx=0x7fe19000bc30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:08.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.256 7fe18ffff700 1 -- 172.21.2.112:0/2555304020 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe190000f90 con 0x7fe1980c1d80 2021-02-10T11:31:08.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.256 7fe18ffff700 1 -- 172.21.2.112:0/2555304020 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1900060c0 con 0x7fe1980c1d80 2021-02-10T11:31:08.248 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.256 7fe18ffff700 1 -- 172.21.2.112:0/2555304020 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe190000f90 con 0x7fe1980c1d80 2021-02-10T11:31:08.248 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.256 7fe19d6c9700 1 -- 172.21.2.112:0/2555304020 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1980c29c0 con 0x7fe1980c1d80 2021-02-10T11:31:08.248 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.256 7fe19d6c9700 1 -- 172.21.2.112:0/2555304020 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1980c29c0 con 0x7fe1980c1d80 2021-02-10T11:31:08.248 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.257 7fe18ffff700 1 -- 172.21.2.112:0/2555304020 <== mon.2 v2:172.21.2.120:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe19000bd70 con 0x7fe1980c1d80 2021-02-10T11:31:08.248 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.257 7fe18ffff700 1 --2- 172.21.2.112:0/2555304020 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe17c0186c0 0x7fe17c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:08.248 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.257 7fe18ffff700 1 -- 172.21.2.112:0/2555304020 <== mon.2 v2:172.21.2.120:3301/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fe1900326b0 con 0x7fe1980c1d80 2021-02-10T11:31:08.248 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.257 7fe1972bf700 1 --2- 172.21.2.112:0/2555304020 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe17c0186c0 0x7fe17c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:08.248 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.258 7fe1972bf700 1 --2- 172.21.2.112:0/2555304020 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe17c0186c0 0x7fe17c01abc0 crc :-1 s=READY pgs=485 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:08.275 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.287 7fe19d6c9700 1 -- 172.21.2.112:0/2555304020 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe198105ae0 con 0x7fe1980c1d80 2021-02-10T11:31:08.280 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.292 7fe18ffff700 1 -- 172.21.2.112:0/2555304020 <== mon.2 v2:172.21.2.120:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe190020f90 con 0x7fe1980c1d80 2021-02-10T11:31:08.409 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.421 7fe19d6c9700 1 -- 172.21.2.112:0/2555304020 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348664", "target": ["mgr", ""]}) v1 -- 0x7fe198105ae0 con 0x7fe17c0186c0 2021-02-10T11:31:08.419 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.431 7fe18ffff700 1 -- 172.21.2.112:0/2555304020 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fe180006420 con 0x7fe17c0186c0 2021-02-10T11:31:08.426 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.438 7fe19d6c9700 1 -- 172.21.2.112:0/2555304020 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe17c0186c0 msgr2=0x7fe17c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:08.426 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.438 7fe19d6c9700 1 --2- 172.21.2.112:0/2555304020 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe17c0186c0 0x7fe17c01abc0 crc :-1 s=READY pgs=485 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:08.426 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.438 7fe19d6c9700 1 -- 172.21.2.112:0/2555304020 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe1980c1d80 msgr2=0x7fe1980c2180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:08.426 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.438 7fe19d6c9700 1 --2- 172.21.2.112:0/2555304020 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe1980c1d80 0x7fe1980c2180 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7fe19000b890 tx=0x7fe19000bc30).stop 2021-02-10T11:31:08.427 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.439 7fe19d6c9700 1 -- 172.21.2.112:0/2555304020 shutdown_connections 2021-02-10T11:31:08.427 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.439 7fe19d6c9700 1 --2- 172.21.2.112:0/2555304020 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe17c0186c0 0x7fe17c01abc0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:08.427 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.439 7fe19d6c9700 1 --2- 172.21.2.112:0/2555304020 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe1980c1d80 0x7fe1980c2180 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:08.427 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.439 7fe19d6c9700 1 --2- 172.21.2.112:0/2555304020 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe1980b8200 0x7fe1980b8600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:08.427 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.439 7fe19d6c9700 1 --2- 172.21.2.112:0/2555304020 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe198190920 0x7fe1980b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:08.427 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.439 7fe19d6c9700 1 -- 172.21.2.112:0/2555304020 >> 172.21.2.112:0/2555304020 conn(0x7fe198094e10 msgr2=0x7fe1980a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:08.427 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.439 7fe19d6c9700 1 -- 172.21.2.112:0/2555304020 shutdown_connections 2021-02-10T11:31:08.427 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.439 7fe19d6c9700 1 -- 172.21.2.112:0/2555304020 wait complete. 2021-02-10T11:31:08.435 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000348665 2021-02-10T11:31:08.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.862 7f030f438700 1 Processor -- start 2021-02-10T11:31:08.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.862 7f030f438700 1 -- start start 2021-02-10T11:31:08.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.863 7f030f438700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f03080f05f0 0x7f03080f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:08.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.863 7f030f438700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03081b0c70 con 0x7f03081b0870 2021-02-10T11:31:08.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.863 7f030f438700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03080f0b50 con 0x7f03081b0470 2021-02-10T11:31:08.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.863 7f030f438700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03080e5ea0 con 0x7f03080f05f0 2021-02-10T11:31:08.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.863 7f030ca53700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f03081b0870 0x7f03080efe00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:47332/0 (socket says 172.21.2.112:47332) 2021-02-10T11:31:08.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.863 7f030ca53700 1 -- 172.21.2.112:0/1396788977 learned_addr learned my addr 172.21.2.112:0/1396788977 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:31:08.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.863 7f030da55700 1 --2- 172.21.2.112:0/1396788977 >> v2:172.21.2.120:3300/0 conn(0x7f03080f05f0 0x7f03080f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:08.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.864 7f02fbfff700 1 -- 172.21.2.112:0/1396788977 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 959902143 0 0) 0x7f03040009b0 con 0x7f03081b0870 2021-02-10T11:31:08.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.864 7f030da55700 1 -- 172.21.2.112:0/1396788977 >> v1:172.21.2.120:6789/0 conn(0x7f03081b0470 legacy=0x7f03080ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:08.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.864 7f030da55700 1 -- 172.21.2.112:0/1396788977 >> v1:172.21.2.112:6789/0 conn(0x7f03081b0870 legacy=0x7f03080efe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:08.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.864 7f030da55700 1 -- 172.21.2.112:0/1396788977 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03080e52e0 con 0x7f03080f05f0 2021-02-10T11:31:08.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.864 7f030da55700 1 --2- 172.21.2.112:0/1396788977 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f03080f05f0 0x7f03080f20e0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 rx=0x7f0300006e50 tx=0x7f0300009e70).ready entity=mon.1 client_cookie=93fb7d7cf7eba078 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:08.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.864 7f02fbfff700 1 -- 172.21.2.112:0/1396788977 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0300006a20 con 0x7f03080f05f0 2021-02-10T11:31:08.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.864 7f02fbfff700 1 -- 172.21.2.112:0/1396788977 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0300006790 con 0x7f03080f05f0 2021-02-10T11:31:08.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.864 7f02fbfff700 1 -- 172.21.2.112:0/1396788977 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0300000f90 con 0x7f03080f05f0 2021-02-10T11:31:08.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.864 7f030f438700 1 -- 172.21.2.112:0/1396788977 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f03080f05f0 msgr2=0x7f03080f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:08.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.864 7f030f438700 1 --2- 172.21.2.112:0/1396788977 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f03080f05f0 0x7f03080f20e0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 rx=0x7f0300006e50 tx=0x7f0300009e70).stop 2021-02-10T11:31:08.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.865 7f030f438700 1 -- 172.21.2.112:0/1396788977 shutdown_connections 2021-02-10T11:31:08.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.865 7f030f438700 1 --2- 172.21.2.112:0/1396788977 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f03080f05f0 0x7f03080f20e0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:08.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.865 7f030f438700 1 -- 172.21.2.112:0/1396788977 >> 172.21.2.112:0/1396788977 conn(0x7f03080ce3e0 msgr2=0x7f03080e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:08.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.865 7f030f438700 1 -- 172.21.2.112:0/1396788977 shutdown_connections 2021-02-10T11:31:08.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.865 7f030f438700 1 -- 172.21.2.112:0/1396788977 wait complete. 2021-02-10T11:31:08.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.880 7f030f438700 1 Processor -- start 2021-02-10T11:31:08.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.880 7f030f438700 1 -- start start 2021-02-10T11:31:08.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.880 7f030f438700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f03080bdbf0 0x7f03081a7450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:08.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.880 7f030f438700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f03080bdff0 0x7f03081a7a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:08.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.880 7f030f438700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f03080abcc0 0x7f03080ac0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:08.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.880 7f030f438700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03080be590 con 0x7f03080bdbf0 2021-02-10T11:31:08.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.880 7f030f438700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03081a8090 con 0x7f03080abcc0 2021-02-10T11:31:08.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.880 7f030f438700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03081a82b0 con 0x7f03080bdff0 2021-02-10T11:31:08.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.880 7f030d254700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f03080bdbf0 0x7f03081a7450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:08.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.880 7f030d254700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f03080bdbf0 0x7f03081a7450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:47874/0 (socket says 172.21.2.112:47874) 2021-02-10T11:31:08.875 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.880 7f030d254700 1 -- 172.21.2.112:0/3158785084 learned_addr learned my addr 172.21.2.112:0/3158785084 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:31:08.875 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.881 7f030d254700 1 -- 172.21.2.112:0/3158785084 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f03080bdff0 msgr2=0x7f03081a7a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:08.875 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.881 7f030d254700 1 --2- 172.21.2.112:0/3158785084 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f03080bdff0 0x7f03081a7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:08.875 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.881 7f030d254700 1 -- 172.21.2.112:0/3158785084 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f03080abcc0 msgr2=0x7f03080ac0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:08.875 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.881 7f030d254700 1 --2- 172.21.2.112:0/3158785084 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f03080abcc0 0x7f03080ac0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:08.875 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.881 7f030d254700 1 -- 172.21.2.112:0/3158785084 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03080ac750 con 0x7f03080bdbf0 2021-02-10T11:31:08.876 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.881 7f030d254700 1 --2- 172.21.2.112:0/3158785084 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f03080bdbf0 0x7f03081a7450 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 rx=0x7f02fc006ed0 tx=0x7f02fc00b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:08.876 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.881 7f02f9ffb700 1 -- 172.21.2.112:0/3158785084 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f02fc006020 con 0x7f03080bdbf0 2021-02-10T11:31:08.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.881 7f02f9ffb700 1 -- 172.21.2.112:0/3158785084 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02fc0062a0 con 0x7f03080bdbf0 2021-02-10T11:31:08.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.881 7f02f9ffb700 1 -- 172.21.2.112:0/3158785084 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f02fc005730 con 0x7f03080bdbf0 2021-02-10T11:31:08.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.881 7f030f438700 1 -- 172.21.2.112:0/3158785084 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03080ac750 con 0x7f03080bdbf0 2021-02-10T11:31:08.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.881 7f030f438700 1 -- 172.21.2.112:0/3158785084 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03080ad2d0 con 0x7f03080bdbf0 2021-02-10T11:31:08.878 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.882 7f02f9ffb700 1 -- 172.21.2.112:0/3158785084 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f02fc0054b0 con 0x7f03080bdbf0 2021-02-10T11:31:08.878 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.882 7f02f9ffb700 1 --2- 172.21.2.112:0/3158785084 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f02e80186c0 0x7f02e801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:08.878 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.882 7f02f9ffb700 1 -- 172.21.2.112:0/3158785084 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f02fc032540 con 0x7f03080bdbf0 2021-02-10T11:31:08.878 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.882 7f030ca53700 1 --2- 172.21.2.112:0/3158785084 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f02e80186c0 0x7f02e801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:08.878 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.883 7f030ca53700 1 --2- 172.21.2.112:0/3158785084 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f02e80186c0 0x7f02e801abc0 crc :-1 s=READY pgs=486 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:08.900 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.912 7f030f438700 1 -- 172.21.2.112:0/3158785084 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03080edcc0 con 0x7f03080bdbf0 2021-02-10T11:31:08.902 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:08.915 7f02f9ffb700 1 -- 172.21.2.112:0/3158785084 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f02fc026640 con 0x7f03080bdbf0 2021-02-10T11:31:09.035 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:09.047 7f030f438700 1 -- 172.21.2.112:0/3158785084 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348665", "target": ["mgr", ""]}) v1 -- 0x7f03080ad290 con 0x7f02e80186c0 2021-02-10T11:31:09.045 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:09.057 7f02f9ffb700 1 -- 172.21.2.112:0/3158785084 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f0304000940 con 0x7f02e80186c0 2021-02-10T11:31:09.053 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:09.064 7f030f438700 1 -- 172.21.2.112:0/3158785084 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f02e80186c0 msgr2=0x7f02e801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:09.053 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:09.064 7f030f438700 1 --2- 172.21.2.112:0/3158785084 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f02e80186c0 0x7f02e801abc0 crc :-1 s=READY pgs=486 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:09.053 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:09.064 7f030f438700 1 -- 172.21.2.112:0/3158785084 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f03080bdbf0 msgr2=0x7f03081a7450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:09.053 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:09.064 7f030f438700 1 --2- 172.21.2.112:0/3158785084 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f03080bdbf0 0x7f03081a7450 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 rx=0x7f02fc006ed0 tx=0x7f02fc00b960).stop 2021-02-10T11:31:09.054 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:09.064 7f030f438700 1 -- 172.21.2.112:0/3158785084 shutdown_connections 2021-02-10T11:31:09.054 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:09.064 7f030f438700 1 --2- 172.21.2.112:0/3158785084 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f02e80186c0 0x7f02e801abc0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:09.054 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:09.064 7f030f438700 1 --2- 172.21.2.112:0/3158785084 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f03080abcc0 0x7f03080ac0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:09.054 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:09.064 7f030f438700 1 --2- 172.21.2.112:0/3158785084 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f03080bdff0 0x7f03081a7a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:09.054 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:09.064 7f030f438700 1 --2- 172.21.2.112:0/3158785084 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f03080bdbf0 0x7f03081a7450 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:09.055 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:09.064 7f030f438700 1 -- 172.21.2.112:0/3158785084 >> 172.21.2.112:0/3158785084 conn(0x7f0308095a70 msgr2=0x7f03080bf040 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:09.055 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:09.064 7f030f438700 1 -- 172.21.2.112:0/3158785084 shutdown_connections 2021-02-10T11:31:09.055 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:09.065 7f030f438700 1 -- 172.21.2.112:0/3158785084 wait complete. 2021-02-10T11:31:09.061 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000348666 2021-02-10T11:31:09.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:09.487 7f11425f0700 1 Processor -- start 2021-02-10T11:31:09.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:09.487 7f11425f0700 1 -- start start 2021-02-10T11:31:09.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:09.487 7f11425f0700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f113c1b0470 0x7f113c1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:09.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:09.487 7f11425f0700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f113c0f20e0 0x7f113c0f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:09.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:09.487 7f11425f0700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f113c0f0b50 con 0x7f113c0ef610 2021-02-10T11:31:09.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:09.487 7f11425f0700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f113c0f0820 con 0x7f113c1b0470 2021-02-10T11:31:09.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:09.487 7f11425f0700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f113c0f24e0 con 0x7f113c0f20e0 2021-02-10T11:31:09.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:09.488 7f113bfff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f113c1b0470 0x7f113c1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:09.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:09.488 7f113bfff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f113c1b0470 0x7f113c1b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:47882/0 (socket says 172.21.2.112:47882) 2021-02-10T11:31:09.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:09.488 7f113bfff700 1 -- 172.21.2.112:0/789810577 learned_addr learned my addr 172.21.2.112:0/789810577 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:31:09.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:09.488 7f1140c0d700 1 --2- 172.21.2.112:0/789810577 >> v2:172.21.2.120:3300/0 conn(0x7f113c0f20e0 0x7f113c0f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:09.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:09.488 7f113bfff700 1 -- 172.21.2.112:0/789810577 >> v2:172.21.2.120:3300/0 conn(0x7f113c0f20e0 msgr2=0x7f113c0f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:09.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:09.488 7f113bfff700 1 --2- 172.21.2.112:0/789810577 >> v2:172.21.2.120:3300/0 conn(0x7f113c0f20e0 0x7f113c0f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:09.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:09.488 7f113bfff700 1 -- 172.21.2.112:0/789810577 >> v1:172.21.2.120:6789/0 conn(0x7f113c0ef610 legacy=0x7f113c0efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:09.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:09.488 7f113bfff700 1 -- 172.21.2.112:0/789810577 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f113c0e52e0 con 0x7f113c1b0470 2021-02-10T11:31:09.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:09.488 7f1140c0d700 1 --2- 172.21.2.112:0/789810577 >> v2:172.21.2.120:3300/0 conn(0x7f113c0f20e0 0x7f113c0f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:31:09.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:09.488 7f113bfff700 1 --2- 172.21.2.112:0/789810577 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f113c1b0470 0x7f113c1b0870 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 rx=0x7f112c006700 tx=0x7f112c009b80).ready entity=mon.0 client_cookie=ad754c75aa0af994 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:09.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:09.488 7f113affd700 1 -- 172.21.2.112:0/789810577 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f112c006340 con 0x7f113c1b0470 2021-02-10T11:31:09.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:09.488 7f113affd700 1 -- 172.21.2.112:0/789810577 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f112c00c0c0 con 0x7f113c1b0470 2021-02-10T11:31:09.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:09.488 7f113affd700 1 -- 172.21.2.112:0/789810577 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f112c00c5e0 con 0x7f113c1b0470 2021-02-10T11:31:09.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:09.489 7f11425f0700 1 -- 172.21.2.112:0/789810577 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f113c1b0470 msgr2=0x7f113c1b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:09.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:09.489 7f11425f0700 1 --2- 172.21.2.112:0/789810577 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f113c1b0470 0x7f113c1b0870 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 rx=0x7f112c006700 tx=0x7f112c009b80).stop 2021-02-10T11:31:09.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:09.489 7f11425f0700 1 -- 172.21.2.112:0/789810577 shutdown_connections 2021-02-10T11:31:09.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:09.489 7f11425f0700 1 --2- 172.21.2.112:0/789810577 >> v2:172.21.2.120:3300/0 conn(0x7f113c0f20e0 0x7f113c0f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:09.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:09.489 7f11425f0700 1 --2- 172.21.2.112:0/789810577 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f113c1b0470 0x7f113c1b0870 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:09.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:09.489 7f11425f0700 1 -- 172.21.2.112:0/789810577 >> 172.21.2.112:0/789810577 conn(0x7f113c0ce3e0 msgr2=0x7f113c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:09.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:09.489 7f11425f0700 1 -- 172.21.2.112:0/789810577 shutdown_connections 2021-02-10T11:31:09.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:09.489 7f11425f0700 1 -- 172.21.2.112:0/789810577 wait complete. 2021-02-10T11:31:09.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:09.504 7f11425f0700 1 Processor -- start 2021-02-10T11:31:09.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:09.504 7f11425f0700 1 -- start start 2021-02-10T11:31:09.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:09.504 7f11425f0700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f113c190920 0x7f113c0b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:09.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:09.504 7f11425f0700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f113c0b8200 0x7f113c0b8600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:09.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:09.504 7f11425f0700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f113c0c1d80 0x7f113c0c2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:09.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:09.504 7f11425f0700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f113c190da0 con 0x7f113c0c1d80 2021-02-10T11:31:09.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:09.504 7f11425f0700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f113c0b8c20 con 0x7f113c0b8200 2021-02-10T11:31:09.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:09.504 7f11425f0700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f113c0b8e10 con 0x7f113c190920 2021-02-10T11:31:09.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:09.505 7f1140c0d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f113c0c1d80 0x7f113c0c2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:09.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:09.505 7f1140c0d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f113c0c1d80 0x7f113c0c2180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:47890/0 (socket says 172.21.2.112:47890) 2021-02-10T11:31:09.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:09.505 7f1140c0d700 1 -- 172.21.2.112:0/516070684 learned_addr learned my addr 172.21.2.112:0/516070684 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:31:09.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:09.505 7f113b7fe700 1 --2- 172.21.2.112:0/516070684 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f113c0b8200 0x7f113c0b8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:09.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:09.505 7f113bfff700 1 --2- 172.21.2.112:0/516070684 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f113c190920 0x7f113c0b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:09.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:09.505 7f1140c0d700 1 -- 172.21.2.112:0/516070684 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f113c190920 msgr2=0x7f113c0b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:09.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:09.505 7f1140c0d700 1 --2- 172.21.2.112:0/516070684 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f113c190920 0x7f113c0b7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:09.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:09.505 7f1140c0d700 1 -- 172.21.2.112:0/516070684 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f113c0b8200 msgr2=0x7f113c0b8600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:09.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:09.505 7f1140c0d700 1 --2- 172.21.2.112:0/516070684 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f113c0b8200 0x7f113c0b8600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:09.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:09.505 7f1140c0d700 1 -- 172.21.2.112:0/516070684 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f113c0c27d0 con 0x7f113c0c1d80 2021-02-10T11:31:09.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:09.505 7f113bfff700 1 --2- 172.21.2.112:0/516070684 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f113c190920 0x7f113c0b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:31:09.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:09.505 7f113b7fe700 1 --2- 172.21.2.112:0/516070684 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f113c0b8200 0x7f113c0b8600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:31:09.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:09.505 7f1140c0d700 1 --2- 172.21.2.112:0/516070684 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f113c0c1d80 0x7f113c0c2180 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 rx=0x7f113400b970 tx=0x7f113400bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:09.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:09.506 7f1138ff9700 1 -- 172.21.2.112:0/516070684 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1134000f90 con 0x7f113c0c1d80 2021-02-10T11:31:09.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:09.506 7f1138ff9700 1 -- 172.21.2.112:0/516070684 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1134001400 con 0x7f113c0c1d80 2021-02-10T11:31:09.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:09.506 7f1138ff9700 1 -- 172.21.2.112:0/516070684 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1134001920 con 0x7f113c0c1d80 2021-02-10T11:31:09.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:09.506 7f11425f0700 1 -- 172.21.2.112:0/516070684 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f113c0c27d0 con 0x7f113c0c1d80 2021-02-10T11:31:09.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:09.506 7f11425f0700 1 -- 172.21.2.112:0/516070684 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f113c0c2d60 con 0x7f113c0c1d80 2021-02-10T11:31:09.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:09.506 7f1138ff9700 1 -- 172.21.2.112:0/516070684 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f11340015e0 con 0x7f113c0c1d80 2021-02-10T11:31:09.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:09.506 7f1138ff9700 1 --2- 172.21.2.112:0/516070684 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f111c0186c0 0x7f111c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:09.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:09.507 7f1138ff9700 1 -- 172.21.2.112:0/516070684 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f1134032940 con 0x7f113c0c1d80 2021-02-10T11:31:09.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:09.507 7f113bfff700 1 --2- 172.21.2.112:0/516070684 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f111c0186c0 0x7f111c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:09.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:09.508 7f113bfff700 1 --2- 172.21.2.112:0/516070684 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f111c0186c0 0x7f111c01abc0 crc :-1 s=READY pgs=487 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:09.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:09.537 7f11425f0700 1 -- 172.21.2.112:0/516070684 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f113c105ae0 con 0x7f113c0c1d80 2021-02-10T11:31:09.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:09.540 7f1138ff9700 1 -- 172.21.2.112:0/516070684 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1134000f20 con 0x7f113c0c1d80 2021-02-10T11:31:09.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:09.667 7f11425f0700 1 -- 172.21.2.112:0/516070684 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348666", "target": ["mgr", ""]}) v1 -- 0x7f113c0c2d60 con 0x7f111c0186c0 2021-02-10T11:31:09.666 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:09.678 7f1138ff9700 1 -- 172.21.2.112:0/516070684 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f112c0009b0 con 0x7f111c0186c0 2021-02-10T11:31:09.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:09.685 7f11425f0700 1 -- 172.21.2.112:0/516070684 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f111c0186c0 msgr2=0x7f111c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:09.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:09.685 7f11425f0700 1 --2- 172.21.2.112:0/516070684 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f111c0186c0 0x7f111c01abc0 crc :-1 s=READY pgs=487 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:09.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:09.685 7f11425f0700 1 -- 172.21.2.112:0/516070684 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f113c0c1d80 msgr2=0x7f113c0c2180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:09.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:09.685 7f11425f0700 1 --2- 172.21.2.112:0/516070684 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f113c0c1d80 0x7f113c0c2180 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 rx=0x7f113400b970 tx=0x7f113400bd10).stop 2021-02-10T11:31:09.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:09.685 7f11425f0700 1 -- 172.21.2.112:0/516070684 shutdown_connections 2021-02-10T11:31:09.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:09.685 7f11425f0700 1 --2- 172.21.2.112:0/516070684 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f111c0186c0 0x7f111c01abc0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:09.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:09.685 7f11425f0700 1 --2- 172.21.2.112:0/516070684 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f113c0b8200 0x7f113c0b8600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:09.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:09.685 7f11425f0700 1 --2- 172.21.2.112:0/516070684 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f113c0c1d80 0x7f113c0c2180 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:09.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:09.685 7f11425f0700 1 --2- 172.21.2.112:0/516070684 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f113c190920 0x7f113c0b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:09.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:09.685 7f11425f0700 1 -- 172.21.2.112:0/516070684 >> 172.21.2.112:0/516070684 conn(0x7f113c094e10 msgr2=0x7f113c0a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:09.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:09.685 7f11425f0700 1 -- 172.21.2.112:0/516070684 shutdown_connections 2021-02-10T11:31:09.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:09.685 7f11425f0700 1 -- 172.21.2.112:0/516070684 wait complete. 2021-02-10T11:31:09.681 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000348667 2021-02-10T11:31:10.114 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.110 7ff0bb525700 1 Processor -- start 2021-02-10T11:31:10.115 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.111 7ff0bb525700 1 -- start start 2021-02-10T11:31:10.115 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.111 7ff0bb525700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7ff0b41b0470 0x7ff0b41b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:10.115 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.111 7ff0bb525700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7ff0b40ef610 0x7ff0b40efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:10.115 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.111 7ff0bb525700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff0b40f0030 0x7ff0b40f0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:10.115 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.111 7ff0bb525700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0b40e5ea0 con 0x7ff0b40ef610 2021-02-10T11:31:10.115 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.111 7ff0bb525700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0b40f0b50 con 0x7ff0b41b0470 2021-02-10T11:31:10.115 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.111 7ff0bb525700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0b40e52e0 con 0x7ff0b40f0030 2021-02-10T11:31:10.115 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.111 7ff0b9341700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7ff0b41b0470 0x7ff0b41b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:10.115 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.111 7ff0b8b40700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7ff0b40ef610 0x7ff0b40efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:10.116 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.111 7ff0b9341700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7ff0b41b0470 0x7ff0b41b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3300/0 says I am v2:172.21.2.112:50176/0 (socket says 172.21.2.112:50176) 2021-02-10T11:31:10.116 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.111 7ff0b9341700 1 -- 172.21.2.112:0/4052058396 learned_addr learned my addr 172.21.2.112:0/4052058396 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:31:10.116 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.111 7ff0b9b42700 1 --2- 172.21.2.112:0/4052058396 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff0b40f0030 0x7ff0b40f0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:10.116 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.111 7ff0b8b40700 1 -- 172.21.2.112:0/4052058396 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff0b40f0030 msgr2=0x7ff0b40f0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:10.116 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.111 7ff0b8b40700 1 --2- 172.21.2.112:0/4052058396 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff0b40f0030 0x7ff0b40f0430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:10.116 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.111 7ff0b8b40700 1 -- 172.21.2.112:0/4052058396 >> v2:172.21.2.120:3300/0 conn(0x7ff0b41b0470 msgr2=0x7ff0b41b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:10.116 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.111 7ff0b8b40700 1 --2- 172.21.2.112:0/4052058396 >> v2:172.21.2.120:3300/0 conn(0x7ff0b41b0470 0x7ff0b41b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:10.116 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.112 7ff0b8b40700 1 -- 172.21.2.112:0/4052058396 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0b40f10d0 con 0x7ff0b40ef610 2021-02-10T11:31:10.117 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.112 7ff0b9b42700 1 --2- 172.21.2.112:0/4052058396 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff0b40f0030 0x7ff0b40f0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:31:10.117 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.112 7ff0b8b40700 1 --2- 172.21.2.112:0/4052058396 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff0b40ef610 0x7ff0b40efa10 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 rx=0x7ff0a4006650 tx=0x7ff0a4009a80).ready entity=mon.0 client_cookie=c22caf61fead2ffe server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:10.117 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.112 7ff0abfff700 1 -- 172.21.2.112:0/4052058396 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff0a40009b0 con 0x7ff0b40ef610 2021-02-10T11:31:10.117 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.112 7ff0abfff700 1 -- 172.21.2.112:0/4052058396 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0a400c0c0 con 0x7ff0b40ef610 2021-02-10T11:31:10.117 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.112 7ff0abfff700 1 -- 172.21.2.112:0/4052058396 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff0a400c510 con 0x7ff0b40ef610 2021-02-10T11:31:10.117 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.112 7ff0bb525700 1 -- 172.21.2.112:0/4052058396 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff0b40ef610 msgr2=0x7ff0b40efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:10.117 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.112 7ff0bb525700 1 --2- 172.21.2.112:0/4052058396 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff0b40ef610 0x7ff0b40efa10 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 rx=0x7ff0a4006650 tx=0x7ff0a4009a80).stop 2021-02-10T11:31:10.117 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.112 7ff0bb525700 1 -- 172.21.2.112:0/4052058396 shutdown_connections 2021-02-10T11:31:10.117 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.112 7ff0bb525700 1 --2- 172.21.2.112:0/4052058396 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff0b40f0030 0x7ff0b40f0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:10.117 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.112 7ff0bb525700 1 --2- 172.21.2.112:0/4052058396 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff0b40ef610 0x7ff0b40efa10 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:10.117 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.112 7ff0bb525700 1 --2- 172.21.2.112:0/4052058396 >> v2:172.21.2.120:3300/0 conn(0x7ff0b41b0470 0x7ff0b41b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:10.118 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.112 7ff0bb525700 1 -- 172.21.2.112:0/4052058396 >> 172.21.2.112:0/4052058396 conn(0x7ff0b40ce3e0 msgr2=0x7ff0b40e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:10.118 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.112 7ff0bb525700 1 -- 172.21.2.112:0/4052058396 shutdown_connections 2021-02-10T11:31:10.118 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.112 7ff0bb525700 1 -- 172.21.2.112:0/4052058396 wait complete. 2021-02-10T11:31:10.120 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.127 7ff0bb525700 1 Processor -- start 2021-02-10T11:31:10.120 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.127 7ff0bb525700 1 -- start start 2021-02-10T11:31:10.120 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.128 7ff0bb525700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff0b409b930 0x7ff0b40b7c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:10.120 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.128 7ff0bb525700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff0b40b8410 0x7ff0b40b8810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:10.120 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.128 7ff0bb525700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff0b40c1f80 0x7ff0b40c2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:10.120 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.128 7ff0bb525700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0b40c1a00 con 0x7ff0b40b8410 2021-02-10T11:31:10.120 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.128 7ff0bb525700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0b40b8e30 con 0x7ff0b409b930 2021-02-10T11:31:10.121 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.128 7ff0bb525700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0b40b9050 con 0x7ff0b40c1f80 2021-02-10T11:31:10.121 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.128 7ff0b8b40700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff0b40b8410 0x7ff0b40b8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:10.121 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.128 7ff0b8b40700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff0b40b8410 0x7ff0b40b8810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:47904/0 (socket says 172.21.2.112:47904) 2021-02-10T11:31:10.121 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.128 7ff0b8b40700 1 -- 172.21.2.112:0/1694781062 learned_addr learned my addr 172.21.2.112:0/1694781062 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:31:10.121 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.128 7ff0b9b42700 1 --2- 172.21.2.112:0/1694781062 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff0b40c1f80 0x7ff0b40c2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:10.121 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.128 7ff0b9341700 1 --2- 172.21.2.112:0/1694781062 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff0b409b930 0x7ff0b40b7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:10.121 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.128 7ff0b8b40700 1 -- 172.21.2.112:0/1694781062 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff0b40c1f80 msgr2=0x7ff0b40c2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:10.121 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.128 7ff0b8b40700 1 --2- 172.21.2.112:0/1694781062 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff0b40c1f80 0x7ff0b40c2380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:10.121 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.128 7ff0b8b40700 1 -- 172.21.2.112:0/1694781062 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff0b409b930 msgr2=0x7ff0b40b7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:10.121 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.128 7ff0b8b40700 1 --2- 172.21.2.112:0/1694781062 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff0b409b930 0x7ff0b40b7c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:10.121 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.128 7ff0b8b40700 1 -- 172.21.2.112:0/1694781062 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0b40c29a0 con 0x7ff0b40b8410 2021-02-10T11:31:10.122 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.128 7ff0b8b40700 1 --2- 172.21.2.112:0/1694781062 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff0b40b8410 0x7ff0b40b8810 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 rx=0x7ff0a4009630 tx=0x7ff0a400b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:10.122 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.128 7ff0a9ffb700 1 -- 172.21.2.112:0/1694781062 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff0a40093a0 con 0x7ff0b40b8410 2021-02-10T11:31:10.122 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.128 7ff0a9ffb700 1 -- 172.21.2.112:0/1694781062 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0a400bd70 con 0x7ff0b40b8410 2021-02-10T11:31:10.122 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.128 7ff0a9ffb700 1 -- 172.21.2.112:0/1694781062 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff0a40054d0 con 0x7ff0b40b8410 2021-02-10T11:31:10.122 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.128 7ff0bb525700 1 -- 172.21.2.112:0/1694781062 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0b40c29a0 con 0x7ff0b40b8410 2021-02-10T11:31:10.122 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.128 7ff0bb525700 1 -- 172.21.2.112:0/1694781062 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0b40c29a0 con 0x7ff0b40b8410 2021-02-10T11:31:10.122 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.129 7ff0a9ffb700 1 -- 172.21.2.112:0/1694781062 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff0a40054f0 con 0x7ff0b40b8410 2021-02-10T11:31:10.122 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.129 7ff0a9ffb700 1 --2- 172.21.2.112:0/1694781062 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff094018730 0x7ff09401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:10.122 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.129 7ff0a9ffb700 1 -- 172.21.2.112:0/1694781062 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7ff0a4032620 con 0x7ff0b40b8410 2021-02-10T11:31:10.122 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.129 7ff0b9341700 1 --2- 172.21.2.112:0/1694781062 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff094018730 0x7ff09401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:10.123 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.130 7ff0b9341700 1 --2- 172.21.2.112:0/1694781062 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff094018730 0x7ff09401ac10 crc :-1 s=READY pgs=488 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:10.148 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.160 7ff0bb525700 1 -- 172.21.2.112:0/1694781062 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0b4105b30 con 0x7ff0b40b8410 2021-02-10T11:31:10.150 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.163 7ff0a9ffb700 1 -- 172.21.2.112:0/1694781062 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff0a40052e0 con 0x7ff0b40b8410 2021-02-10T11:31:10.279 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.291 7ff0bb525700 1 -- 172.21.2.112:0/1694781062 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000348667", "target": ["mgr", ""]}) v1 -- 0x7ff0b4105b30 con 0x7ff094018730 2021-02-10T11:31:10.290 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.302 7ff0a9ffb700 1 -- 172.21.2.112:0/1694781062 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7ff0b0006e60 con 0x7ff094018730 2021-02-10T11:31:10.296 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.309 7ff0bb525700 1 -- 172.21.2.112:0/1694781062 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff094018730 msgr2=0x7ff09401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:10.296 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.309 7ff0bb525700 1 --2- 172.21.2.112:0/1694781062 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff094018730 0x7ff09401ac10 crc :-1 s=READY pgs=488 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:10.296 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.309 7ff0bb525700 1 -- 172.21.2.112:0/1694781062 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff0b40b8410 msgr2=0x7ff0b40b8810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:10.296 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.309 7ff0bb525700 1 --2- 172.21.2.112:0/1694781062 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff0b40b8410 0x7ff0b40b8810 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 rx=0x7ff0a4009630 tx=0x7ff0a400b980).stop 2021-02-10T11:31:10.297 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.309 7ff0bb525700 1 -- 172.21.2.112:0/1694781062 shutdown_connections 2021-02-10T11:31:10.297 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.309 7ff0bb525700 1 --2- 172.21.2.112:0/1694781062 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff094018730 0x7ff09401ac10 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:10.297 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.309 7ff0bb525700 1 --2- 172.21.2.112:0/1694781062 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff0b40c1f80 0x7ff0b40c2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:10.297 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.309 7ff0bb525700 1 --2- 172.21.2.112:0/1694781062 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff0b40b8410 0x7ff0b40b8810 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:10.297 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.309 7ff0bb525700 1 --2- 172.21.2.112:0/1694781062 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff0b409b930 0x7ff0b40b7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:10.297 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.309 7ff0bb525700 1 -- 172.21.2.112:0/1694781062 >> 172.21.2.112:0/1694781062 conn(0x7ff0b4094e10 msgr2=0x7ff0b40a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:10.297 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.309 7ff0bb525700 1 -- 172.21.2.112:0/1694781062 shutdown_connections 2021-02-10T11:31:10.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.309 7ff0bb525700 1 -- 172.21.2.112:0/1694781062 wait complete. 2021-02-10T11:31:10.305 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:31:10.733 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.728 7f597b42c700 1 Processor -- start 2021-02-10T11:31:10.733 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.728 7f597b42c700 1 -- start start 2021-02-10T11:31:10.733 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.729 7f597b42c700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f59740e5070 0x7f59741b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:10.733 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.729 7f597b42c700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f59740e5470 0x7f5974080310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:10.733 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.729 7f597b42c700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f597407ed80 con 0x7f59741b3e40 2021-02-10T11:31:10.733 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.729 7f597b42c700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59740d1790 con 0x7f59740e5070 2021-02-10T11:31:10.734 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.729 7f597b42c700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59740d19b0 con 0x7f59740e5470 2021-02-10T11:31:10.734 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.729 7f5979248700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f59740e5070 0x7f59741b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:10.734 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.729 7f5978a47700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f59741b3e40 0x7f597407d690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:47372/0 (socket says 172.21.2.112:47372) 2021-02-10T11:31:10.734 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.729 7f5978a47700 1 -- 172.21.2.112:0/868841126 learned_addr learned my addr 172.21.2.112:0/868841126 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:31:10.734 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.729 7f5979a49700 1 --2- 172.21.2.112:0/868841126 >> v2:172.21.2.120:3300/0 conn(0x7f59740e5470 0x7f5974080310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:10.734 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.730 7f5967fff700 1 -- 172.21.2.112:0/868841126 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1729765622 0 0) 0x7f59700010b0 con 0x7f59741b3e40 2021-02-10T11:31:10.734 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.730 7f5979a49700 1 -- 172.21.2.112:0/868841126 >> v2:172.21.2.112:3300/0 conn(0x7f59740e5070 msgr2=0x7f59741b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:10.734 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.730 7f5979a49700 1 --2- 172.21.2.112:0/868841126 >> v2:172.21.2.112:3300/0 conn(0x7f59740e5070 0x7f59741b3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:10.734 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.730 7f5979a49700 1 -- 172.21.2.112:0/868841126 >> v1:172.21.2.112:6789/0 conn(0x7f59741b3e40 legacy=0x7f597407d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:10.734 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.730 7f5979a49700 1 -- 172.21.2.112:0/868841126 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59740d1bd0 con 0x7f59740e5470 2021-02-10T11:31:10.734 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.730 7f5979248700 1 --2- 172.21.2.112:0/868841126 >> v2:172.21.2.112:3300/0 conn(0x7f59740e5070 0x7f59741b3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:31:10.734 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.730 7f5979a49700 1 --2- 172.21.2.112:0/868841126 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f59740e5470 0x7f5974080310 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 rx=0x7f596c006e50 tx=0x7f596c009e70).ready entity=mon.1 client_cookie=49ab528cf6c555ec server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:10.734 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.730 7f5967fff700 1 -- 172.21.2.112:0/868841126 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f596c006a20 con 0x7f59740e5470 2021-02-10T11:31:10.735 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.730 7f5967fff700 1 -- 172.21.2.112:0/868841126 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f596c006790 con 0x7f59740e5470 2021-02-10T11:31:10.735 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.730 7f5967fff700 1 -- 172.21.2.112:0/868841126 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f596c00b700 con 0x7f59740e5470 2021-02-10T11:31:10.735 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.730 7f597b42c700 1 -- 172.21.2.112:0/868841126 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f59740e5470 msgr2=0x7f5974080310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:10.735 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.730 7f597b42c700 1 --2- 172.21.2.112:0/868841126 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f59740e5470 0x7f5974080310 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 rx=0x7f596c006e50 tx=0x7f596c009e70).stop 2021-02-10T11:31:10.735 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.731 7f597b42c700 1 -- 172.21.2.112:0/868841126 shutdown_connections 2021-02-10T11:31:10.735 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.731 7f597b42c700 1 --2- 172.21.2.112:0/868841126 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f59740e5470 0x7f5974080310 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:10.735 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.731 7f597b42c700 1 --2- 172.21.2.112:0/868841126 >> v2:172.21.2.112:3300/0 conn(0x7f59740e5070 0x7f59741b3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:10.735 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.731 7f597b42c700 1 -- 172.21.2.112:0/868841126 >> 172.21.2.112:0/868841126 conn(0x7f59740ce690 msgr2=0x7f59740e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:10.735 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.731 7f597b42c700 1 -- 172.21.2.112:0/868841126 shutdown_connections 2021-02-10T11:31:10.735 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.731 7f597b42c700 1 -- 172.21.2.112:0/868841126 wait complete. 2021-02-10T11:31:10.737 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.746 7f597b42c700 1 Processor -- start 2021-02-10T11:31:10.737 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.746 7f597b42c700 1 -- start start 2021-02-10T11:31:10.737 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.747 7f597b42c700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5974191bb0 0x7f59740a2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:10.737 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.747 7f597b42c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f59740a29a0 0x7f59740a2da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:10.737 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.747 7f597b42c700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f59740a33c0 0x7f59740a03d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:10.737 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.747 7f597b42c700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59740adf90 con 0x7f59740a29a0 2021-02-10T11:31:10.737 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.747 7f597b42c700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59740a1e00 con 0x7f59740a33c0 2021-02-10T11:31:10.737 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.747 7f597b42c700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59740a09f0 con 0x7f5974191bb0 2021-02-10T11:31:10.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.747 7f5978a47700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f59740a29a0 0x7f59740a2da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:10.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.747 7f5978a47700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f59740a29a0 0x7f59740a2da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:47914/0 (socket says 172.21.2.112:47914) 2021-02-10T11:31:10.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.747 7f5978a47700 1 -- 172.21.2.112:0/3464848111 learned_addr learned my addr 172.21.2.112:0/3464848111 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:31:10.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.747 7f5979a49700 1 --2- 172.21.2.112:0/3464848111 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f59740a33c0 0x7f59740a03d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:10.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.747 7f5979248700 1 --2- 172.21.2.112:0/3464848111 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5974191bb0 0x7f59740a2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:10.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.747 7f5978a47700 1 -- 172.21.2.112:0/3464848111 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5974191bb0 msgr2=0x7f59740a2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:10.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.747 7f5978a47700 1 --2- 172.21.2.112:0/3464848111 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5974191bb0 0x7f59740a2380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:10.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.747 7f5978a47700 1 -- 172.21.2.112:0/3464848111 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f59740a33c0 msgr2=0x7f59740a03d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:10.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.747 7f5978a47700 1 --2- 172.21.2.112:0/3464848111 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f59740a33c0 0x7f59740a03d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:10.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.747 7f5978a47700 1 -- 172.21.2.112:0/3464848111 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59740a0bd0 con 0x7f59740a29a0 2021-02-10T11:31:10.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.747 7f5979248700 1 --2- 172.21.2.112:0/3464848111 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5974191bb0 0x7f59740a2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:31:10.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.747 7f5978a47700 1 --2- 172.21.2.112:0/3464848111 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f59740a29a0 0x7f59740a2da0 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 rx=0x7f5970001c10 tx=0x7f597000a8f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:10.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.747 7f5965ffb700 1 -- 172.21.2.112:0/3464848111 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f59700009b0 con 0x7f59740a29a0 2021-02-10T11:31:10.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.748 7f5965ffb700 1 -- 172.21.2.112:0/3464848111 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59700012c0 con 0x7f59740a29a0 2021-02-10T11:31:10.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.748 7f5965ffb700 1 -- 172.21.2.112:0/3464848111 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5970004350 con 0x7f59740a29a0 2021-02-10T11:31:10.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.748 7f597b42c700 1 -- 172.21.2.112:0/3464848111 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59740a0bd0 con 0x7f59740a29a0 2021-02-10T11:31:10.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.748 7f597b42c700 1 -- 172.21.2.112:0/3464848111 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59740a0bd0 con 0x7f59740a29a0 2021-02-10T11:31:10.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.748 7f5965ffb700 1 -- 172.21.2.112:0/3464848111 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5970004140 con 0x7f59740a29a0 2021-02-10T11:31:10.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.748 7f5965ffb700 1 --2- 172.21.2.112:0/3464848111 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5954018730 0x7f595401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:10.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.748 7f5965ffb700 1 -- 172.21.2.112:0/3464848111 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f5970032610 con 0x7f59740a29a0 2021-02-10T11:31:10.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.749 7f5979248700 1 --2- 172.21.2.112:0/3464848111 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5954018730 0x7f595401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:10.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.749 7f5979248700 1 --2- 172.21.2.112:0/3464848111 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5954018730 0x7f595401ac10 crc :-1 s=READY pgs=489 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:10.766 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.779 7f597b42c700 1 -- 172.21.2.112:0/3464848111 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59740a1e00 con 0x7f59740a29a0 2021-02-10T11:31:10.768 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.781 7f5965ffb700 1 -- 172.21.2.112:0/3464848111 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5970000dd0 con 0x7f59740a29a0 2021-02-10T11:31:10.894 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.906 7f597b42c700 1 -- 172.21.2.112:0/3464848111 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f59740a1e00 con 0x7f59740a29a0 2021-02-10T11:31:10.894 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.907 7f5965ffb700 1 -- 172.21.2.112:0/3464848111 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) v1 ==== 74+0+13993 (secure 0 0 0) 0x7f5970004140 con 0x7f59740a29a0 2021-02-10T11:31:10.895 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:31:10.895 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":29,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:15:34.922144","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:15:31.539720","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":5,"pool_name":"cephfs_data","create_time":"2021-02-10 11:15:32.965643","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:31:10.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.909 7f597b42c700 1 -- 172.21.2.112:0/3464848111 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5954018730 msgr2=0x7f595401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:10.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.909 7f597b42c700 1 --2- 172.21.2.112:0/3464848111 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5954018730 0x7f595401ac10 crc :-1 s=READY pgs=489 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:10.897 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.909 7f597b42c700 1 -- 172.21.2.112:0/3464848111 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f59740a29a0 msgr2=0x7f59740a2da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:10.897 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.909 7f597b42c700 1 --2- 172.21.2.112:0/3464848111 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f59740a29a0 0x7f59740a2da0 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 rx=0x7f5970001c10 tx=0x7f597000a8f0).stop 2021-02-10T11:31:10.897 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.909 7f597b42c700 1 -- 172.21.2.112:0/3464848111 shutdown_connections 2021-02-10T11:31:10.897 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.909 7f597b42c700 1 --2- 172.21.2.112:0/3464848111 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5954018730 0x7f595401ac10 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:10.897 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.909 7f597b42c700 1 --2- 172.21.2.112:0/3464848111 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f59740a33c0 0x7f59740a03d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:10.897 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.909 7f597b42c700 1 --2- 172.21.2.112:0/3464848111 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f59740a29a0 0x7f59740a2da0 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:10.897 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.909 7f597b42c700 1 --2- 172.21.2.112:0/3464848111 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5974191bb0 0x7f59740a2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:10.897 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.909 7f597b42c700 1 -- 172.21.2.112:0/3464848111 >> 172.21.2.112:0/3464848111 conn(0x7f59741a31d0 msgr2=0x7f597408b830 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:10.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.909 7f597b42c700 1 -- 172.21.2.112:0/3464848111 shutdown_connections 2021-02-10T11:31:10.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:10.909 7f597b42c700 1 -- 172.21.2.112:0/3464848111 wait complete. 2021-02-10T11:31:10.905 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:13:57.508758', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:15:31.539720', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '28', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:15:32.965643', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '29', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:31:10.905 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:31:11.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.331 7fe9e36a3700 1 Processor -- start 2021-02-10T11:31:11.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.331 7fe9e36a3700 1 -- start start 2021-02-10T11:31:11.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.331 7fe9e36a3700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fe9dc1b0870 0x7fe9dc0f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:11.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.331 7fe9e36a3700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe9dc0efe00 0x7fe9dc0f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:11.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.331 7fe9e36a3700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9dc1b0c70 con 0x7fe9dc1b0470 2021-02-10T11:31:11.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.331 7fe9e36a3700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9dc0f0b50 con 0x7fe9dc1b0870 2021-02-10T11:31:11.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.331 7fe9e36a3700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9dc0f0820 con 0x7fe9dc0efe00 2021-02-10T11:31:11.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.332 7fe9e0cbe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fe9dc1b0870 0x7fe9dc0f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:11.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.332 7fe9e0cbe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fe9dc1b0870 0x7fe9dc0f20e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:47920/0 (socket says 172.21.2.112:47920) 2021-02-10T11:31:11.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.332 7fe9e0cbe700 1 -- 172.21.2.112:0/612209637 learned_addr learned my addr 172.21.2.112:0/612209637 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:31:11.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.332 7fe9e0cbe700 1 -- 172.21.2.112:0/612209637 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe9dc0efe00 msgr2=0x7fe9dc0f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:11.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.332 7fe9e0cbe700 1 --2- 172.21.2.112:0/612209637 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe9dc0efe00 0x7fe9dc0f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:11.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.332 7fe9e0cbe700 1 -- 172.21.2.112:0/612209637 >> v1:172.21.2.120:6789/0 conn(0x7fe9dc1b0470 legacy=0x7fe9dc0ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:11.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.332 7fe9e0cbe700 1 -- 172.21.2.112:0/612209637 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9dc0e52e0 con 0x7fe9dc1b0870 2021-02-10T11:31:11.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.332 7fe9e0cbe700 1 --2- 172.21.2.112:0/612209637 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe9dc1b0870 0x7fe9dc0f20e0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 rx=0x7fe9d0006700 tx=0x7fe9d0009b80).ready entity=mon.0 client_cookie=e986bc347345065a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:11.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.332 7fe9cffff700 1 -- 172.21.2.112:0/612209637 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe9d000c0c0 con 0x7fe9dc1b0870 2021-02-10T11:31:11.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.332 7fe9cffff700 1 -- 172.21.2.112:0/612209637 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9d000c340 con 0x7fe9dc1b0870 2021-02-10T11:31:11.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.332 7fe9cffff700 1 -- 172.21.2.112:0/612209637 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe9d000c860 con 0x7fe9dc1b0870 2021-02-10T11:31:11.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.333 7fe9e36a3700 1 -- 172.21.2.112:0/612209637 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe9dc1b0870 msgr2=0x7fe9dc0f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:11.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.333 7fe9e36a3700 1 --2- 172.21.2.112:0/612209637 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe9dc1b0870 0x7fe9dc0f20e0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 rx=0x7fe9d0006700 tx=0x7fe9d0009b80).stop 2021-02-10T11:31:11.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.333 7fe9e36a3700 1 -- 172.21.2.112:0/612209637 shutdown_connections 2021-02-10T11:31:11.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.333 7fe9e36a3700 1 --2- 172.21.2.112:0/612209637 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe9dc1b0870 0x7fe9dc0f20e0 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:11.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.333 7fe9e36a3700 1 --2- 172.21.2.112:0/612209637 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe9dc0efe00 0x7fe9dc0f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:11.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.333 7fe9e36a3700 1 -- 172.21.2.112:0/612209637 >> 172.21.2.112:0/612209637 conn(0x7fe9dc0ce3e0 msgr2=0x7fe9dc0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:11.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.333 7fe9e36a3700 1 -- 172.21.2.112:0/612209637 shutdown_connections 2021-02-10T11:31:11.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.333 7fe9e36a3700 1 -- 172.21.2.112:0/612209637 wait complete. 2021-02-10T11:31:11.339 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.348 7fe9e36a3700 1 Processor -- start 2021-02-10T11:31:11.339 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.348 7fe9e36a3700 1 -- start start 2021-02-10T11:31:11.339 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.348 7fe9e36a3700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe9dc094fb0 0x7fe9dc09e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:11.339 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.348 7fe9e36a3700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe9dc0c2be0 0x7fe9dc0c2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:11.339 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.348 7fe9e36a3700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe9dc0c3600 0x7fe9dc0a9c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:11.339 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.349 7fe9e36a3700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9dc09ebf0 con 0x7fe9dc0c2be0 2021-02-10T11:31:11.339 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.349 7fe9e36a3700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9dc0aa2a0 con 0x7fe9dc094fb0 2021-02-10T11:31:11.339 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.349 7fe9e36a3700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9dc0aa4c0 con 0x7fe9dc0c3600 2021-02-10T11:31:11.339 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.349 7fe9e14bf700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe9dc094fb0 0x7fe9dc09e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:11.339 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.349 7fe9e0cbe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe9dc0c2be0 0x7fe9dc0c2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:11.339 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.349 7fe9e0cbe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe9dc0c2be0 0x7fe9dc0c2fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:47928/0 (socket says 172.21.2.112:47928) 2021-02-10T11:31:11.339 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.349 7fe9e14bf700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe9dc094fb0 0x7fe9dc09e5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3300/0 says I am v2:172.21.2.112:50206/0 (socket says 172.21.2.112:50206) 2021-02-10T11:31:11.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.349 7fe9e0cbe700 1 -- 172.21.2.112:0/3637675308 learned_addr learned my addr 172.21.2.112:0/3637675308 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:31:11.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.349 7fe9e1cc0700 1 --2- 172.21.2.112:0/3637675308 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe9dc0c3600 0x7fe9dc0a9c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:11.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.349 7fe9e0cbe700 1 -- 172.21.2.112:0/3637675308 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe9dc0c3600 msgr2=0x7fe9dc0a9c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:11.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.349 7fe9e0cbe700 1 --2- 172.21.2.112:0/3637675308 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe9dc0c3600 0x7fe9dc0a9c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:11.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.349 7fe9e0cbe700 1 -- 172.21.2.112:0/3637675308 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe9dc094fb0 msgr2=0x7fe9dc09e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:11.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.349 7fe9e0cbe700 1 --2- 172.21.2.112:0/3637675308 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe9dc094fb0 0x7fe9dc09e5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:11.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.349 7fe9e0cbe700 1 -- 172.21.2.112:0/3637675308 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9dc0aa6e0 con 0x7fe9dc0c2be0 2021-02-10T11:31:11.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.349 7fe9e0cbe700 1 --2- 172.21.2.112:0/3637675308 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe9dc0c2be0 0x7fe9dc0c2fe0 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 rx=0x7fe9d000b5e0 tx=0x7fe9d000b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:11.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.350 7fe9cdffb700 1 -- 172.21.2.112:0/3637675308 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe9d000bd70 con 0x7fe9dc0c2be0 2021-02-10T11:31:11.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.350 7fe9cdffb700 1 -- 172.21.2.112:0/3637675308 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9d0002030 con 0x7fe9dc0c2be0 2021-02-10T11:31:11.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.350 7fe9cdffb700 1 -- 172.21.2.112:0/3637675308 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe9d0002550 con 0x7fe9dc0c2be0 2021-02-10T11:31:11.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.350 7fe9e36a3700 1 -- 172.21.2.112:0/3637675308 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9dc0aa6e0 con 0x7fe9dc0c2be0 2021-02-10T11:31:11.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.350 7fe9e36a3700 1 -- 172.21.2.112:0/3637675308 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9dc0aac70 con 0x7fe9dc0c2be0 2021-02-10T11:31:11.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.350 7fe9cdffb700 1 -- 172.21.2.112:0/3637675308 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe9d0000c90 con 0x7fe9dc0c2be0 2021-02-10T11:31:11.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.350 7fe9cdffb700 1 --2- 172.21.2.112:0/3637675308 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe9bc0186c0 0x7fe9bc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:11.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.350 7fe9cdffb700 1 -- 172.21.2.112:0/3637675308 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fe9d00325f0 con 0x7fe9dc0c2be0 2021-02-10T11:31:11.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.351 7fe9e14bf700 1 --2- 172.21.2.112:0/3637675308 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe9bc0186c0 0x7fe9bc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:11.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.351 7fe9e14bf700 1 --2- 172.21.2.112:0/3637675308 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe9bc0186c0 0x7fe9bc01abc0 crc :-1 s=READY pgs=490 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:11.368 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.381 7fe9e36a3700 1 -- 172.21.2.112:0/3637675308 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9dc0a9750 con 0x7fe9dc0c2be0 2021-02-10T11:31:11.371 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.384 7fe9cdffb700 1 -- 172.21.2.112:0/3637675308 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe9d0030da0 con 0x7fe9dc0c2be0 2021-02-10T11:31:11.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.512 7fe9e36a3700 1 -- 172.21.2.112:0/3637675308 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7fe9dc0a9750 con 0x7fe9dc0c2be0 2021-02-10T11:31:11.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.512 7fe9cdffb700 1 -- 172.21.2.112:0/3637675308 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v29) v1 ==== 92+0+10 (secure 0 0 0) 0x7fe9d0000c90 con 0x7fe9dc0c2be0 2021-02-10T11:31:11.500 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:31:11.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.514 7fe9e36a3700 1 -- 172.21.2.112:0/3637675308 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe9bc0186c0 msgr2=0x7fe9bc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:11.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.514 7fe9e36a3700 1 --2- 172.21.2.112:0/3637675308 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe9bc0186c0 0x7fe9bc01abc0 crc :-1 s=READY pgs=490 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:11.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.514 7fe9e36a3700 1 -- 172.21.2.112:0/3637675308 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe9dc0c2be0 msgr2=0x7fe9dc0c2fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:11.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.514 7fe9e36a3700 1 --2- 172.21.2.112:0/3637675308 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe9dc0c2be0 0x7fe9dc0c2fe0 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 rx=0x7fe9d000b5e0 tx=0x7fe9d000b980).stop 2021-02-10T11:31:11.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.515 7fe9e36a3700 1 -- 172.21.2.112:0/3637675308 shutdown_connections 2021-02-10T11:31:11.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.515 7fe9e36a3700 1 --2- 172.21.2.112:0/3637675308 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe9bc0186c0 0x7fe9bc01abc0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:11.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.515 7fe9e36a3700 1 --2- 172.21.2.112:0/3637675308 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe9dc0c3600 0x7fe9dc0a9c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:11.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.515 7fe9e36a3700 1 --2- 172.21.2.112:0/3637675308 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe9dc0c2be0 0x7fe9dc0c2fe0 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:11.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.515 7fe9e36a3700 1 --2- 172.21.2.112:0/3637675308 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe9dc094fb0 0x7fe9dc09e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:11.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.515 7fe9e36a3700 1 -- 172.21.2.112:0/3637675308 >> 172.21.2.112:0/3637675308 conn(0x7fe9dc09f630 msgr2=0x7fe9dc0a0130 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:11.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.515 7fe9e36a3700 1 -- 172.21.2.112:0/3637675308 shutdown_connections 2021-02-10T11:31:11.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.515 7fe9e36a3700 1 -- 172.21.2.112:0/3637675308 wait complete. 2021-02-10T11:31:11.511 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:31:11.940 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.935 7ff78fa94700 1 Processor -- start 2021-02-10T11:31:11.940 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.936 7ff78fa94700 1 -- start start 2021-02-10T11:31:11.941 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.936 7ff78fa94700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7ff7881b0470 0x7ff7881b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:11.941 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.936 7ff78fa94700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7880f0b50 con 0x7ff7880f20e0 2021-02-10T11:31:11.941 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.936 7ff78fa94700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7880f24e0 con 0x7ff7880ef610 2021-02-10T11:31:11.941 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.936 7ff78fa94700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7880e5ea0 con 0x7ff7881b0470 2021-02-10T11:31:11.941 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.936 7ff78d8b0700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7ff7881b0470 0x7ff7881b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:11.941 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.936 7ff78e0b1700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7ff7880f20e0 0x7ff7880f0200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:47396/0 (socket says 172.21.2.112:47396) 2021-02-10T11:31:11.941 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.936 7ff78d0af700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7ff7880ef610 0x7ff7880efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:40954/0 (socket says 172.21.2.112:40954) 2021-02-10T11:31:11.941 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.936 7ff78e0b1700 1 -- 172.21.2.112:0/1699391187 learned_addr learned my addr 172.21.2.112:0/1699391187 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:31:11.941 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.937 7ff78c8ae700 1 -- 172.21.2.112:0/1699391187 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3510988032 0 0) 0x7ff7840009b0 con 0x7ff7880ef610 2021-02-10T11:31:11.941 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.937 7ff78c8ae700 1 -- 172.21.2.112:0/1699391187 --> v1:172.21.2.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff774002fe0 con 0x7ff7880ef610 2021-02-10T11:31:11.941 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.937 7ff78d8b0700 1 -- 172.21.2.112:0/1699391187 >> v1:172.21.2.120:6789/0 conn(0x7ff7880ef610 legacy=0x7ff7880efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:11.941 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.937 7ff78d8b0700 1 -- 172.21.2.112:0/1699391187 >> v1:172.21.2.112:6789/0 conn(0x7ff7880f20e0 legacy=0x7ff7880f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:11.941 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.937 7ff78d8b0700 1 -- 172.21.2.112:0/1699391187 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7880e52e0 con 0x7ff7881b0470 2021-02-10T11:31:11.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.937 7ff78d8b0700 1 --2- 172.21.2.112:0/1699391187 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff7881b0470 0x7ff7881b0870 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 rx=0x7ff77c006650 tx=0x7ff77c009a00).ready entity=mon.0 client_cookie=ff68c74a72489328 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:11.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.937 7ff78c8ae700 1 -- 172.21.2.112:0/1699391187 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff77c009df0 con 0x7ff7881b0470 2021-02-10T11:31:11.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.938 7ff78c8ae700 1 -- 172.21.2.112:0/1699391187 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff77c0009b0 con 0x7ff7881b0470 2021-02-10T11:31:11.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.938 7ff78c8ae700 1 -- 172.21.2.112:0/1699391187 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff77c00c2b0 con 0x7ff7881b0470 2021-02-10T11:31:11.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.938 7ff78fa94700 1 -- 172.21.2.112:0/1699391187 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff7881b0470 msgr2=0x7ff7881b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:11.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.938 7ff78fa94700 1 --2- 172.21.2.112:0/1699391187 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff7881b0470 0x7ff7881b0870 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 rx=0x7ff77c006650 tx=0x7ff77c009a00).stop 2021-02-10T11:31:11.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.938 7ff78fa94700 1 -- 172.21.2.112:0/1699391187 shutdown_connections 2021-02-10T11:31:11.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.938 7ff78fa94700 1 --2- 172.21.2.112:0/1699391187 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff7881b0470 0x7ff7881b0870 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:11.943 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.938 7ff78fa94700 1 -- 172.21.2.112:0/1699391187 >> 172.21.2.112:0/1699391187 conn(0x7ff7880ce3e0 msgr2=0x7ff7880e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:11.943 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.939 7ff78fa94700 1 -- 172.21.2.112:0/1699391187 shutdown_connections 2021-02-10T11:31:11.943 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.939 7ff78fa94700 1 -- 172.21.2.112:0/1699391187 wait complete. 2021-02-10T11:31:11.945 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.953 7ff78fa94700 1 Processor -- start 2021-02-10T11:31:11.945 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.953 7ff78fa94700 1 -- start start 2021-02-10T11:31:11.945 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.954 7ff78fa94700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff7880b20b0 0x7ff7880ac970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:11.945 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.954 7ff78fa94700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff7880b24b0 0x7ff788191dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:11.945 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.954 7ff78fa94700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff7880acf90 0x7ff7881923e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:11.945 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.954 7ff78fa94700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7880b1b30 con 0x7ff7880acf90 2021-02-10T11:31:11.945 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.954 7ff78fa94700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7881a04e0 con 0x7ff7880b20b0 2021-02-10T11:31:11.946 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.954 7ff78fa94700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7881a06c0 con 0x7ff7880b24b0 2021-02-10T11:31:11.946 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.954 7ff78e0b1700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff7880acf90 0x7ff7881923e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:11.946 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.954 7ff78e0b1700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff7880acf90 0x7ff7881923e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:47940/0 (socket says 172.21.2.112:47940) 2021-02-10T11:31:11.946 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.954 7ff78e0b1700 1 -- 172.21.2.112:0/1787740932 learned_addr learned my addr 172.21.2.112:0/1787740932 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:31:11.946 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.954 7ff78d8b0700 1 --2- 172.21.2.112:0/1787740932 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff7880b20b0 0x7ff7880ac970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:11.946 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.954 7ff78e0b1700 1 -- 172.21.2.112:0/1787740932 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff7880b24b0 msgr2=0x7ff788191dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:11.946 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.954 7ff78e0b1700 1 --2- 172.21.2.112:0/1787740932 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff7880b24b0 0x7ff788191dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:11.946 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.954 7ff78e0b1700 1 -- 172.21.2.112:0/1787740932 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff7880b20b0 msgr2=0x7ff7880ac970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:11.946 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.954 7ff78e0b1700 1 --2- 172.21.2.112:0/1787740932 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff7880b20b0 0x7ff7880ac970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:11.946 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.955 7ff78e0b1700 1 -- 172.21.2.112:0/1787740932 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff788192be0 con 0x7ff7880acf90 2021-02-10T11:31:11.946 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.955 7ff78d8b0700 1 --2- 172.21.2.112:0/1787740932 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff7880b20b0 0x7ff7880ac970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:31:11.946 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.955 7ff78e0b1700 1 --2- 172.21.2.112:0/1787740932 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff7880acf90 0x7ff7881923e0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 rx=0x7ff780001bf0 tx=0x7ff78000abd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:11.947 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.955 7ff77a7fc700 1 -- 172.21.2.112:0/1787740932 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff780007030 con 0x7ff7880acf90 2021-02-10T11:31:11.947 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.955 7ff77a7fc700 1 -- 172.21.2.112:0/1787740932 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7800072f0 con 0x7ff7880acf90 2021-02-10T11:31:11.947 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.955 7ff77a7fc700 1 -- 172.21.2.112:0/1787740932 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff780004720 con 0x7ff7880acf90 2021-02-10T11:31:11.947 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.955 7ff78fa94700 1 -- 172.21.2.112:0/1787740932 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff788192be0 con 0x7ff7880acf90 2021-02-10T11:31:11.947 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.955 7ff78fa94700 1 -- 172.21.2.112:0/1787740932 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff788192be0 con 0x7ff7880acf90 2021-02-10T11:31:11.947 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.956 7ff77a7fc700 1 -- 172.21.2.112:0/1787740932 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff7800074d0 con 0x7ff7880acf90 2021-02-10T11:31:11.947 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.956 7ff77a7fc700 1 --2- 172.21.2.112:0/1787740932 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff7700186c0 0x7ff77001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:11.947 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.956 7ff77a7fc700 1 -- 172.21.2.112:0/1787740932 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7ff780032910 con 0x7ff7880acf90 2021-02-10T11:31:11.947 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.956 7ff78d8b0700 1 --2- 172.21.2.112:0/1787740932 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff7700186c0 0x7ff77001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:11.947 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.957 7ff78d8b0700 1 --2- 172.21.2.112:0/1787740932 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff7700186c0 0x7ff77001abc0 crc :-1 s=READY pgs=491 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:11.973 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.986 7ff78fa94700 1 -- 172.21.2.112:0/1787740932 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff788105af0 con 0x7ff7880acf90 2021-02-10T11:31:11.976 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:11.988 7ff77a7fc700 1 -- 172.21.2.112:0/1787740932 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff7800076c0 con 0x7ff7880acf90 2021-02-10T11:31:12.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:12.118 7ff78fa94700 1 -- 172.21.2.112:0/1787740932 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7ff788105af0 con 0x7ff7880acf90 2021-02-10T11:31:12.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:12.118 7ff77a7fc700 1 -- 172.21.2.112:0/1787740932 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v29) v1 ==== 104+0+10 (secure 0 0 0) 0x7ff7800074b0 con 0x7ff7880acf90 2021-02-10T11:31:12.106 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:31:12.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:12.120 7ff78fa94700 1 -- 172.21.2.112:0/1787740932 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff7700186c0 msgr2=0x7ff77001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:12.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:12.120 7ff78fa94700 1 --2- 172.21.2.112:0/1787740932 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff7700186c0 0x7ff77001abc0 crc :-1 s=READY pgs=491 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:12.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:12.120 7ff78fa94700 1 -- 172.21.2.112:0/1787740932 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff7880acf90 msgr2=0x7ff7881923e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:12.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:12.120 7ff78fa94700 1 --2- 172.21.2.112:0/1787740932 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff7880acf90 0x7ff7881923e0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 rx=0x7ff780001bf0 tx=0x7ff78000abd0).stop 2021-02-10T11:31:12.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:12.120 7ff78fa94700 1 -- 172.21.2.112:0/1787740932 shutdown_connections 2021-02-10T11:31:12.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:12.120 7ff78fa94700 1 --2- 172.21.2.112:0/1787740932 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff7700186c0 0x7ff77001abc0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:12.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:12.120 7ff78fa94700 1 --2- 172.21.2.112:0/1787740932 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff7880acf90 0x7ff7881923e0 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:12.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:12.120 7ff78fa94700 1 --2- 172.21.2.112:0/1787740932 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff7880b24b0 0x7ff788191dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:12.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:12.120 7ff78fa94700 1 --2- 172.21.2.112:0/1787740932 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff7880b20b0 0x7ff7880ac970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:12.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:12.120 7ff78fa94700 1 -- 172.21.2.112:0/1787740932 >> 172.21.2.112:0/1787740932 conn(0x7ff7880bf4e0 msgr2=0x7ff7880be3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:12.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:12.120 7ff78fa94700 1 -- 172.21.2.112:0/1787740932 shutdown_connections 2021-02-10T11:31:12.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:12.120 7ff78fa94700 1 -- 172.21.2.112:0/1787740932 wait complete. 2021-02-10T11:31:12.116 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:31:12.546 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:12.541 7f73e5beb700 1 Processor -- start 2021-02-10T11:31:12.546 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:12.541 7f73e5beb700 1 -- start start 2021-02-10T11:31:12.546 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:12.542 7f73e5beb700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f73e01b0470 0x7f73e01b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:12.546 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:12.542 7f73e5beb700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f73e00ef610 0x7f73e00efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:12.546 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:12.542 7f73e5beb700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73e00f0b50 con 0x7f73e00f0030 2021-02-10T11:31:12.546 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:12.542 7f73e5beb700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73e00e52e0 con 0x7f73e00ef610 2021-02-10T11:31:12.546 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:12.542 7f73e5beb700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73e00e5490 con 0x7f73e01b0470 2021-02-10T11:31:12.546 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:12.542 7f73dffff700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f73e00f0030 0x7f73e00f10d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:47406/0 (socket says 172.21.2.112:47406) 2021-02-10T11:31:12.546 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:12.542 7f73dffff700 1 -- 172.21.2.112:0/694585447 learned_addr learned my addr 172.21.2.112:0/694585447 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:31:12.546 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:12.542 7f73deffd700 1 --2- 172.21.2.112:0/694585447 >> v2:172.21.2.120:3300/0 conn(0x7f73e00ef610 0x7f73e00efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:12.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:12.542 7f73df7fe700 1 --2- 172.21.2.112:0/694585447 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f73e01b0470 0x7f73e01b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:12.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:12.542 7f73de7fc700 1 -- 172.21.2.112:0/694585447 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2978782018 0 0) 0x7f73d8001500 con 0x7f73e00f0030 2021-02-10T11:31:12.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:12.543 7f73de7fc700 1 -- 172.21.2.112:0/694585447 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73cc002fe0 con 0x7f73e00f0030 2021-02-10T11:31:12.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:12.543 7f73de7fc700 1 -- 172.21.2.112:0/694585447 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2612737439 0 0) 0x7f73d8001460 con 0x7f73e00f0030 2021-02-10T11:31:12.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:12.543 7f73df7fe700 1 -- 172.21.2.112:0/694585447 >> v2:172.21.2.120:3300/0 conn(0x7f73e00ef610 msgr2=0x7f73e00efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:12.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:12.543 7f73df7fe700 1 --2- 172.21.2.112:0/694585447 >> v2:172.21.2.120:3300/0 conn(0x7f73e00ef610 0x7f73e00efa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:12.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:12.543 7f73df7fe700 1 -- 172.21.2.112:0/694585447 >> v1:172.21.2.112:6789/0 conn(0x7f73e00f0030 legacy=0x7f73e00f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:12.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:12.543 7f73df7fe700 1 -- 172.21.2.112:0/694585447 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73e00f0430 con 0x7f73e01b0470 2021-02-10T11:31:12.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:12.543 7f73deffd700 1 --2- 172.21.2.112:0/694585447 >> v2:172.21.2.120:3300/0 conn(0x7f73e00ef610 0x7f73e00efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:31:12.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:12.543 7f73df7fe700 1 --2- 172.21.2.112:0/694585447 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f73e01b0470 0x7f73e01b0870 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f73c8006650 tx=0x7f73c8009a80).ready entity=mon.2 client_cookie=24378ed6e627b5f3 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:12.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:12.543 7f73de7fc700 1 -- 172.21.2.112:0/694585447 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f73c800c0c0 con 0x7f73e01b0470 2021-02-10T11:31:12.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:12.543 7f73de7fc700 1 -- 172.21.2.112:0/694585447 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73c800c340 con 0x7f73e01b0470 2021-02-10T11:31:12.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:12.543 7f73de7fc700 1 -- 172.21.2.112:0/694585447 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f73c800c860 con 0x7f73e01b0470 2021-02-10T11:31:12.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:12.543 7f73e5beb700 1 -- 172.21.2.112:0/694585447 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f73e01b0470 msgr2=0x7f73e01b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:12.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:12.543 7f73e5beb700 1 --2- 172.21.2.112:0/694585447 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f73e01b0470 0x7f73e01b0870 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f73c8006650 tx=0x7f73c8009a80).stop 2021-02-10T11:31:12.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:12.544 7f73e5beb700 1 -- 172.21.2.112:0/694585447 shutdown_connections 2021-02-10T11:31:12.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:12.544 7f73e5beb700 1 --2- 172.21.2.112:0/694585447 >> v2:172.21.2.120:3300/0 conn(0x7f73e00ef610 0x7f73e00efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:12.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:12.544 7f73e5beb700 1 --2- 172.21.2.112:0/694585447 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f73e01b0470 0x7f73e01b0870 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:12.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:12.544 7f73e5beb700 1 -- 172.21.2.112:0/694585447 >> 172.21.2.112:0/694585447 conn(0x7f73e00ce3e0 msgr2=0x7f73e00e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:12.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:12.544 7f73e5beb700 1 -- 172.21.2.112:0/694585447 shutdown_connections 2021-02-10T11:31:12.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:12.544 7f73e5beb700 1 -- 172.21.2.112:0/694585447 wait complete. 2021-02-10T11:31:12.549 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:12.559 7f73e5beb700 1 Processor -- start 2021-02-10T11:31:12.549 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:12.559 7f73e5beb700 1 -- start start 2021-02-10T11:31:12.550 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:12.559 7f73e5beb700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f73e00a0420 0x7f73e00a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:12.550 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:12.559 7f73e5beb700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f73e0189d00 0x7f73e018a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:12.550 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:12.559 7f73e5beb700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f73e018a720 0x7f73e00b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:12.550 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:12.559 7f73e5beb700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73e0189800 con 0x7f73e018a720 2021-02-10T11:31:12.550 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:12.559 7f73e5beb700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73e018ab20 con 0x7f73e0189d00 2021-02-10T11:31:12.550 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:12.559 7f73e5beb700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73e00a1140 con 0x7f73e00a0420 2021-02-10T11:31:12.550 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:12.559 7f73dffff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f73e018a720 0x7f73e00b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:12.550 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:12.559 7f73dffff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f73e018a720 0x7f73e00b7be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:47952/0 (socket says 172.21.2.112:47952) 2021-02-10T11:31:12.550 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:12.559 7f73dffff700 1 -- 172.21.2.112:0/2540857393 learned_addr learned my addr 172.21.2.112:0/2540857393 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:31:12.550 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:12.560 7f73dffff700 1 -- 172.21.2.112:0/2540857393 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f73e00a0420 msgr2=0x7f73e00a0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:12.550 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:12.560 7f73dffff700 1 --2- 172.21.2.112:0/2540857393 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f73e00a0420 0x7f73e00a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:12.550 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:12.560 7f73dffff700 1 -- 172.21.2.112:0/2540857393 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f73e0189d00 msgr2=0x7f73e018a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:12.550 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:12.560 7f73dffff700 1 --2- 172.21.2.112:0/2540857393 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f73e0189d00 0x7f73e018a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:12.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:12.560 7f73dffff700 1 -- 172.21.2.112:0/2540857393 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73e00b8200 con 0x7f73e018a720 2021-02-10T11:31:12.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:12.560 7f73dffff700 1 --2- 172.21.2.112:0/2540857393 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f73e018a720 0x7f73e00b7be0 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 rx=0x7f73d800b8c0 tx=0x7f73d800bc60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:12.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:12.560 7f73d7fff700 1 -- 172.21.2.112:0/2540857393 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f73d8006030 con 0x7f73e018a720 2021-02-10T11:31:12.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:12.560 7f73d7fff700 1 -- 172.21.2.112:0/2540857393 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73d8006370 con 0x7f73e018a720 2021-02-10T11:31:12.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:12.560 7f73d7fff700 1 -- 172.21.2.112:0/2540857393 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f73d8004720 con 0x7f73e018a720 2021-02-10T11:31:12.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:12.560 7f73e5beb700 1 -- 172.21.2.112:0/2540857393 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73e00b8200 con 0x7f73e018a720 2021-02-10T11:31:12.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:12.560 7f73e5beb700 1 -- 172.21.2.112:0/2540857393 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73e00b8790 con 0x7f73e018a720 2021-02-10T11:31:12.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:12.561 7f73d7fff700 1 -- 172.21.2.112:0/2540857393 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f73d8006550 con 0x7f73e018a720 2021-02-10T11:31:12.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:12.561 7f73d7fff700 1 --2- 172.21.2.112:0/2540857393 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f73c40186c0 0x7f73c401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:12.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:12.561 7f73d7fff700 1 -- 172.21.2.112:0/2540857393 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f73d8032730 con 0x7f73e018a720 2021-02-10T11:31:12.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:12.561 7f73df7fe700 1 --2- 172.21.2.112:0/2540857393 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f73c40186c0 0x7f73c401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:12.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:12.561 7f73df7fe700 1 --2- 172.21.2.112:0/2540857393 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f73c40186c0 0x7f73c401abc0 crc :-1 s=READY pgs=492 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:12.579 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:12.592 7f73e5beb700 1 -- 172.21.2.112:0/2540857393 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73e0105ae0 con 0x7f73e018a720 2021-02-10T11:31:12.581 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:12.594 7f73d7fff700 1 -- 172.21.2.112:0/2540857393 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f73d800bda0 con 0x7f73e018a720 2021-02-10T11:31:12.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:12.722 7f73e5beb700 1 -- 172.21.2.112:0/2540857393 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f73e00f0b50 con 0x7f73e018a720 2021-02-10T11:31:12.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:12.723 7f73d7fff700 1 -- 172.21.2.112:0/2540857393 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v29) v1 ==== 100+0+10 (secure 0 0 0) 0x7f73d8006550 con 0x7f73e018a720 2021-02-10T11:31:12.710 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:31:12.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:12.724 7f73e5beb700 1 -- 172.21.2.112:0/2540857393 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f73c40186c0 msgr2=0x7f73c401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:12.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:12.725 7f73e5beb700 1 --2- 172.21.2.112:0/2540857393 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f73c40186c0 0x7f73c401abc0 crc :-1 s=READY pgs=492 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:12.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:12.725 7f73e5beb700 1 -- 172.21.2.112:0/2540857393 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f73e018a720 msgr2=0x7f73e00b7be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:12.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:12.725 7f73e5beb700 1 --2- 172.21.2.112:0/2540857393 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f73e018a720 0x7f73e00b7be0 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 rx=0x7f73d800b8c0 tx=0x7f73d800bc60).stop 2021-02-10T11:31:12.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:12.725 7f73e5beb700 1 -- 172.21.2.112:0/2540857393 shutdown_connections 2021-02-10T11:31:12.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:12.725 7f73e5beb700 1 --2- 172.21.2.112:0/2540857393 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f73c40186c0 0x7f73c401abc0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:12.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:12.725 7f73e5beb700 1 --2- 172.21.2.112:0/2540857393 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f73e0189d00 0x7f73e018a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:12.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:12.725 7f73e5beb700 1 --2- 172.21.2.112:0/2540857393 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f73e018a720 0x7f73e00b7be0 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:12.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:12.725 7f73e5beb700 1 --2- 172.21.2.112:0/2540857393 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f73e00a0420 0x7f73e00a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:12.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:12.725 7f73e5beb700 1 -- 172.21.2.112:0/2540857393 >> 172.21.2.112:0/2540857393 conn(0x7f73e0094e10 msgr2=0x7f73e00a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:12.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:12.725 7f73e5beb700 1 -- 172.21.2.112:0/2540857393 shutdown_connections 2021-02-10T11:31:12.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:12.725 7f73e5beb700 1 -- 172.21.2.112:0/2540857393 wait complete. 2021-02-10T11:31:12.721 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:31:12.721 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:31:13.146 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:13.142 7f68e2cc3700 1 Processor -- start 2021-02-10T11:31:13.147 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:13.142 7f68e2cc3700 1 -- start start 2021-02-10T11:31:13.147 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:13.143 7f68e2cc3700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f68dc0e5470 0x7f68dc07e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:13.147 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:13.143 7f68e2cc3700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68dc07ed80 con 0x7f68dc07f300 2021-02-10T11:31:13.147 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:13.143 7f68e2cc3700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68dc1b4010 con 0x7f68dc0e5070 2021-02-10T11:31:13.147 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:13.143 7f68e2cc3700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68dc0d1790 con 0x7f68dc0e5470 2021-02-10T11:31:13.147 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:13.143 7f68e12e0700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f68dc07f300 0x7f68dc07f700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:47420/0 (socket says 172.21.2.112:47420) 2021-02-10T11:31:13.147 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:13.143 7f68e12e0700 1 -- 172.21.2.112:0/1957349289 learned_addr learned my addr 172.21.2.112:0/1957349289 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:31:13.147 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:13.143 7f68dbfff700 1 --2- 172.21.2.112:0/1957349289 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f68dc0e5470 0x7f68dc07e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:13.147 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:13.143 7f68db7fe700 1 -- 172.21.2.112:0/1957349289 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2888425790 0 0) 0x7f68d4001500 con 0x7f68dc07f300 2021-02-10T11:31:13.147 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:13.143 7f68db7fe700 1 -- 172.21.2.112:0/1957349289 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68c8002fe0 con 0x7f68dc07f300 2021-02-10T11:31:13.147 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:13.144 7f68db7fe700 1 -- 172.21.2.112:0/1957349289 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 331491093 0 0) 0x7f68c4001080 con 0x7f68dc0e5070 2021-02-10T11:31:13.147 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:13.144 7f68db7fe700 1 -- 172.21.2.112:0/1957349289 --> v1:172.21.2.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68c8003210 con 0x7f68dc0e5070 2021-02-10T11:31:13.148 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:13.144 7f68dbfff700 1 -- 172.21.2.112:0/1957349289 >> v1:172.21.2.120:6789/0 conn(0x7f68dc0e5070 legacy=0x7f68dc1b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:13.148 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:13.144 7f68dbfff700 1 -- 172.21.2.112:0/1957349289 >> v1:172.21.2.112:6789/0 conn(0x7f68dc07f300 legacy=0x7f68dc07f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:13.148 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:13.144 7f68dbfff700 1 -- 172.21.2.112:0/1957349289 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68dc0d1980 con 0x7f68dc0e5470 2021-02-10T11:31:13.148 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:13.144 7f68dbfff700 1 --2- 172.21.2.112:0/1957349289 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f68dc0e5470 0x7f68dc07e6a0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f68cc006650 tx=0x7f68cc009a80).ready entity=mon.2 client_cookie=dc860b47d6888d81 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:13.148 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:13.144 7f68db7fe700 1 -- 172.21.2.112:0/1957349289 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f68cc00c0c0 con 0x7f68dc0e5470 2021-02-10T11:31:13.148 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:13.144 7f68db7fe700 1 -- 172.21.2.112:0/1957349289 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68cc00c340 con 0x7f68dc0e5470 2021-02-10T11:31:13.148 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:13.144 7f68db7fe700 1 -- 172.21.2.112:0/1957349289 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f68cc00c860 con 0x7f68dc0e5470 2021-02-10T11:31:13.148 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:13.144 7f68e2cc3700 1 -- 172.21.2.112:0/1957349289 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f68dc0e5470 msgr2=0x7f68dc07e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:13.148 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:13.144 7f68e2cc3700 1 --2- 172.21.2.112:0/1957349289 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f68dc0e5470 0x7f68dc07e6a0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f68cc006650 tx=0x7f68cc009a80).stop 2021-02-10T11:31:13.148 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:13.145 7f68e2cc3700 1 -- 172.21.2.112:0/1957349289 shutdown_connections 2021-02-10T11:31:13.148 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:13.145 7f68e2cc3700 1 --2- 172.21.2.112:0/1957349289 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f68dc0e5470 0x7f68dc07e6a0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:13.148 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:13.145 7f68e2cc3700 1 -- 172.21.2.112:0/1957349289 >> 172.21.2.112:0/1957349289 conn(0x7f68dc0ce690 msgr2=0x7f68dc0e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:13.148 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:13.145 7f68e2cc3700 1 -- 172.21.2.112:0/1957349289 shutdown_connections 2021-02-10T11:31:13.149 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:13.145 7f68e2cc3700 1 -- 172.21.2.112:0/1957349289 wait complete. 2021-02-10T11:31:13.150 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:13.160 7f68e2cc3700 1 Processor -- start 2021-02-10T11:31:13.150 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:13.160 7f68e2cc3700 1 -- start start 2021-02-10T11:31:13.150 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:13.160 7f68e2cc3700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f68dc0b1970 0x7f68dc0b1d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:13.150 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:13.160 7f68e2cc3700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f68dc08acb0 0x7f68dc08b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:13.150 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:13.160 7f68e2cc3700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f68dc08b6d0 0x7f68dc1aac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:13.150 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:13.160 7f68e2cc3700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68dc08bba0 con 0x7f68dc08acb0 2021-02-10T11:31:13.150 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:13.160 7f68e2cc3700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68dc1ab280 con 0x7f68dc08b6d0 2021-02-10T11:31:13.150 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:13.160 7f68e2cc3700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68dc1ab4a0 con 0x7f68dc0b1970 2021-02-10T11:31:13.150 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:13.160 7f68dbfff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f68dc08acb0 0x7f68dc08b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:13.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:13.160 7f68dbfff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f68dc08acb0 0x7f68dc08b0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:47960/0 (socket says 172.21.2.112:47960) 2021-02-10T11:31:13.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:13.160 7f68dbfff700 1 -- 172.21.2.112:0/1255396819 learned_addr learned my addr 172.21.2.112:0/1255396819 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:31:13.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:13.160 7f68e0adf700 1 --2- 172.21.2.112:0/1255396819 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f68dc0b1970 0x7f68dc0b1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:13.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:13.161 7f68e12e0700 1 --2- 172.21.2.112:0/1255396819 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f68dc08b6d0 0x7f68dc1aac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:13.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:13.161 7f68dbfff700 1 -- 172.21.2.112:0/1255396819 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f68dc0b1970 msgr2=0x7f68dc0b1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:13.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:13.161 7f68dbfff700 1 --2- 172.21.2.112:0/1255396819 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f68dc0b1970 0x7f68dc0b1d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:13.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:13.161 7f68dbfff700 1 -- 172.21.2.112:0/1255396819 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f68dc08b6d0 msgr2=0x7f68dc1aac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:13.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:13.161 7f68dbfff700 1 --2- 172.21.2.112:0/1255396819 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f68dc08b6d0 0x7f68dc1aac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:13.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:13.161 7f68dbfff700 1 -- 172.21.2.112:0/1255396819 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68dc1ab6c0 con 0x7f68dc08acb0 2021-02-10T11:31:13.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:13.161 7f68dbfff700 1 --2- 172.21.2.112:0/1255396819 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f68dc08acb0 0x7f68dc08b0b0 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 rx=0x7f68cc009630 tx=0x7f68cc00b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:13.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:13.161 7f68d97fa700 1 -- 172.21.2.112:0/1255396819 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f68cc0093a0 con 0x7f68dc08acb0 2021-02-10T11:31:13.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:13.161 7f68d97fa700 1 -- 172.21.2.112:0/1255396819 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68cc00bd70 con 0x7f68dc08acb0 2021-02-10T11:31:13.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:13.161 7f68d97fa700 1 -- 172.21.2.112:0/1255396819 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f68cc005560 con 0x7f68dc08acb0 2021-02-10T11:31:13.152 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:13.161 7f68e2cc3700 1 -- 172.21.2.112:0/1255396819 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68dc1ab6c0 con 0x7f68dc08acb0 2021-02-10T11:31:13.152 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:13.161 7f68e2cc3700 1 -- 172.21.2.112:0/1255396819 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68dc08c0a0 con 0x7f68dc08acb0 2021-02-10T11:31:13.152 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:13.162 7f68d97fa700 1 -- 172.21.2.112:0/1255396819 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f68cc0052e0 con 0x7f68dc08acb0 2021-02-10T11:31:13.152 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:13.162 7f68d97fa700 1 --2- 172.21.2.112:0/1255396819 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f68c00186c0 0x7f68c001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:13.152 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:13.162 7f68d97fa700 1 -- 172.21.2.112:0/1255396819 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f68cc032570 con 0x7f68dc08acb0 2021-02-10T11:31:13.152 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:13.163 7f68e0adf700 1 --2- 172.21.2.112:0/1255396819 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f68c00186c0 0x7f68c001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:13.152 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:13.163 7f68e0adf700 1 --2- 172.21.2.112:0/1255396819 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f68c00186c0 0x7f68c001abc0 crc :-1 s=READY pgs=493 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:13.180 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:13.192 7f68e2cc3700 1 -- 172.21.2.112:0/1255396819 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68dc107180 con 0x7f68dc08acb0 2021-02-10T11:31:13.182 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:13.195 7f68d97fa700 1 -- 172.21.2.112:0/1255396819 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f68cc02ded0 con 0x7f68dc08acb0 2021-02-10T11:31:13.311 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:13.323 7f68e2cc3700 1 -- 172.21.2.112:0/1255396819 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f68dc08bb20 con 0x7f68dc08acb0 2021-02-10T11:31:13.311 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:13.324 7f68d97fa700 1 -- 172.21.2.112:0/1255396819 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+2098 (secure 0 0 0) 0x7f68cc02ded0 con 0x7f68dc08acb0 2021-02-10T11:31:13.312 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 14 2021-02-10T11:31:13.312 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:31:13.312 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":14,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4627,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/2302110857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":2302110857},{"type":"v1","addr":"172.21.2.112:6833","nonce":2302110857}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":9}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:15:33.919772","modified":"2021-02-10 11:15:36.309405","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4427},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4427":{"gid":4427,"name":"b","rank":0,"incarnation":10,"state":"up:active","state_seq":4,"addr":"172.21.2.120:6835/3730022736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":3730022736},{"type":"v1","addr":"172.21.2.120:6835","nonce":3730022736}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2021-02-10T11:31:13.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:13.326 7f68e2cc3700 1 -- 172.21.2.112:0/1255396819 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f68c00186c0 msgr2=0x7f68c001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:13.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:13.326 7f68e2cc3700 1 --2- 172.21.2.112:0/1255396819 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f68c00186c0 0x7f68c001abc0 crc :-1 s=READY pgs=493 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:13.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:13.326 7f68e2cc3700 1 -- 172.21.2.112:0/1255396819 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f68dc08acb0 msgr2=0x7f68dc08b0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:13.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:13.326 7f68e2cc3700 1 --2- 172.21.2.112:0/1255396819 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f68dc08acb0 0x7f68dc08b0b0 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 rx=0x7f68cc009630 tx=0x7f68cc00b980).stop 2021-02-10T11:31:13.314 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:13.326 7f68e2cc3700 1 -- 172.21.2.112:0/1255396819 shutdown_connections 2021-02-10T11:31:13.314 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:13.326 7f68e2cc3700 1 --2- 172.21.2.112:0/1255396819 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f68c00186c0 0x7f68c001abc0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:13.314 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:13.326 7f68e2cc3700 1 --2- 172.21.2.112:0/1255396819 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f68dc08b6d0 0x7f68dc1aac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:13.314 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:13.326 7f68e2cc3700 1 --2- 172.21.2.112:0/1255396819 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f68dc08acb0 0x7f68dc08b0b0 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:13.314 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:13.326 7f68e2cc3700 1 --2- 172.21.2.112:0/1255396819 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f68dc0b1970 0x7f68dc0b1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:13.314 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:13.326 7f68e2cc3700 1 -- 172.21.2.112:0/1255396819 >> 172.21.2.112:0/1255396819 conn(0x7f68dc1ac620 msgr2=0x7f68dc0893d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:13.314 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:13.326 7f68e2cc3700 1 -- 172.21.2.112:0/1255396819 shutdown_connections 2021-02-10T11:31:13.314 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:13.326 7f68e2cc3700 1 -- 172.21.2.112:0/1255396819 wait complete. 2021-02-10T11:31:13.524 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 13, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:15:33.919772', 'modified': '2021-02-10 11:15:36.309405', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4427}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4427': {'gid': 4427, 'name': 'b', 'rank': 0, 'incarnation': 10, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.2.120:6835/3730022736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.120:6834', 'nonce': 3730022736}, {'type': 'v1', 'addr': '172.21.2.120:6835', 'nonce': 3730022736}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:31:13.524 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:31:13.524 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:31:13.922 INFO:teuthology.orchestra.run.gibba020.stdout:{ 2021-02-10T11:31:13.923 INFO:teuthology.orchestra.run.gibba020.stdout: "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", 2021-02-10T11:31:13.923 INFO:teuthology.orchestra.run.gibba020.stdout: "whoami": 0, 2021-02-10T11:31:13.923 INFO:teuthology.orchestra.run.gibba020.stdout: "id": 4427, 2021-02-10T11:31:13.923 INFO:teuthology.orchestra.run.gibba020.stdout: "want_state": "up:active", 2021-02-10T11:31:13.923 INFO:teuthology.orchestra.run.gibba020.stdout: "state": "up:active", 2021-02-10T11:31:13.923 INFO:teuthology.orchestra.run.gibba020.stdout: "rank_uptime": 940.01643579500001, 2021-02-10T11:31:13.923 INFO:teuthology.orchestra.run.gibba020.stdout: "mdsmap_epoch": 13, 2021-02-10T11:31:13.923 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch": 29, 2021-02-10T11:31:13.923 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch_barrier": 29, 2021-02-10T11:31:13.923 INFO:teuthology.orchestra.run.gibba020.stdout: "uptime": 949.65354721699998 2021-02-10T11:31:13.923 INFO:teuthology.orchestra.run.gibba020.stdout:} 2021-02-10T11:31:13.926 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", "whoami": 0, "id": 4427, "want_state": "up:active", "state": "up:active", "rank_uptime": 940.01643579500001, "mdsmap_epoch": 13, "osdmap_epoch": 29, "osdmap_epoch_barrier": 29, "uptime": 949.65354721699998 } 2021-02-10T11:31:13.927 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:31:13.927 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2021-02-10T11:31:13.927 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@gibba012.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2021-02-10T11:31:13.927 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:13.927 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:31:14.262 DEBUG:teuthology.orchestra.run.gibba012:> sudo modprobe fuse 2021-02-10T11:31:14.394 DEBUG:teuthology.orchestra.run.gibba012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:31:14.397 DEBUG:teuthology.orchestra.run.gibba020:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:31:14.606 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:31:14.606 DEBUG:teuthology.orchestra.run.gibba012:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:31:14.950 INFO:teuthology.orchestra.run.gibba012.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:31:14.950 INFO:teuthology.orchestra.run.gibba012.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:31:14.950 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:31:14.951 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:14.951 DEBUG:teuthology.orchestra.run.gibba012:> ls /sys/fs/fuse/connections 2021-02-10T11:31:15.284 INFO:teuthology.orchestra.run.gibba012.stdout:43 2021-02-10T11:31:15.285 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [43] 2021-02-10T11:31:15.285 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:31:15.288 DEBUG:teuthology.orchestra.run.gibba012:> sudo modprobe fuse 2021-02-10T11:31:15.631 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:31:15.631 DEBUG:teuthology.orchestra.run.gibba012:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:31:15.698 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba012.stderr:ceph-fuse[118603]: starting ceph client 2021-02-10T11:31:15.698 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba012.stderr:2021-02-10 11:31:15.710 7fe8dcdcaf80 -1 init, newargv = 0x55ce2e7b6100 newargc=9 2021-02-10T11:31:15.703 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba012.stderr:ceph-fuse[118603]: starting fuse 2021-02-10T11:31:15.973 INFO:teuthology.orchestra.run.gibba012.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:31:15.973 INFO:teuthology.orchestra.run.gibba012.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:31:15.974 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:31:15.974 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:15.974 DEBUG:teuthology.orchestra.run.gibba012:> ls /sys/fs/fuse/connections 2021-02-10T11:31:16.307 INFO:teuthology.orchestra.run.gibba012.stdout:42 2021-02-10T11:31:16.307 INFO:teuthology.orchestra.run.gibba012.stdout:43 2021-02-10T11:31:16.308 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42, 43] 2021-02-10T11:31:16.308 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:16.308 DEBUG:teuthology.orchestra.run.gibba012:> sudo python3 -c ' 2021-02-10T11:31:16.308 DEBUG:teuthology.orchestra.run.gibba012:> import glob 2021-02-10T11:31:16.308 DEBUG:teuthology.orchestra.run.gibba012:> import re 2021-02-10T11:31:16.308 DEBUG:teuthology.orchestra.run.gibba012:> import os 2021-02-10T11:31:16.308 DEBUG:teuthology.orchestra.run.gibba012:> import subprocess 2021-02-10T11:31:16.309 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:31:16.309 DEBUG:teuthology.orchestra.run.gibba012:> def find_socket(client_name): 2021-02-10T11:31:16.309 DEBUG:teuthology.orchestra.run.gibba012:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2021-02-10T11:31:16.309 DEBUG:teuthology.orchestra.run.gibba012:> files = glob.glob(asok_path) 2021-02-10T11:31:16.309 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:31:16.309 DEBUG:teuthology.orchestra.run.gibba012:> # Given a non-glob path, it better be there 2021-02-10T11:31:16.309 DEBUG:teuthology.orchestra.run.gibba012:> if "*" not in asok_path: 2021-02-10T11:31:16.309 DEBUG:teuthology.orchestra.run.gibba012:> assert(len(files) == 1) 2021-02-10T11:31:16.309 DEBUG:teuthology.orchestra.run.gibba012:> return files[0] 2021-02-10T11:31:16.309 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:31:16.309 DEBUG:teuthology.orchestra.run.gibba012:> for f in files: 2021-02-10T11:31:16.309 DEBUG:teuthology.orchestra.run.gibba012:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:31:16.310 DEBUG:teuthology.orchestra.run.gibba012:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:31:16.310 DEBUG:teuthology.orchestra.run.gibba012:> return f 2021-02-10T11:31:16.310 DEBUG:teuthology.orchestra.run.gibba012:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:31:16.310 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:31:16.310 DEBUG:teuthology.orchestra.run.gibba012:> print(find_socket("client.0")) 2021-02-10T11:31:16.310 DEBUG:teuthology.orchestra.run.gibba012:> ' 2021-02-10T11:31:16.672 INFO:teuthology.orchestra.run.gibba012.stdout:/var/run/ceph/ceph-client.0.118603.asok 2021-02-10T11:31:16.674 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.118603.asok 2021-02-10T11:31:16.675 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:16.675 DEBUG:teuthology.orchestra.run.gibba012:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.118603.asok status 2021-02-10T11:31:17.064 INFO:teuthology.orchestra.run.gibba012.stdout:{ 2021-02-10T11:31:17.064 INFO:teuthology.orchestra.run.gibba012.stdout: "metadata": { 2021-02-10T11:31:17.064 INFO:teuthology.orchestra.run.gibba012.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:31:17.064 INFO:teuthology.orchestra.run.gibba012.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:31:17.064 INFO:teuthology.orchestra.run.gibba012.stdout: "entity_id": "0", 2021-02-10T11:31:17.065 INFO:teuthology.orchestra.run.gibba012.stdout: "hostname": "gibba012", 2021-02-10T11:31:17.065 INFO:teuthology.orchestra.run.gibba012.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2021-02-10T11:31:17.065 INFO:teuthology.orchestra.run.gibba012.stdout: "pid": "118603", 2021-02-10T11:31:17.065 INFO:teuthology.orchestra.run.gibba012.stdout: "root": "/" 2021-02-10T11:31:17.065 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:31:17.065 INFO:teuthology.orchestra.run.gibba012.stdout: "dentry_count": 0, 2021-02-10T11:31:17.065 INFO:teuthology.orchestra.run.gibba012.stdout: "dentry_pinned_count": 0, 2021-02-10T11:31:17.065 INFO:teuthology.orchestra.run.gibba012.stdout: "id": 7077, 2021-02-10T11:31:17.065 INFO:teuthology.orchestra.run.gibba012.stdout: "inst": { 2021-02-10T11:31:17.065 INFO:teuthology.orchestra.run.gibba012.stdout: "name": { 2021-02-10T11:31:17.066 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "client", 2021-02-10T11:31:17.066 INFO:teuthology.orchestra.run.gibba012.stdout: "num": 7077 2021-02-10T11:31:17.066 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:31:17.066 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": { 2021-02-10T11:31:17.066 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:31:17.066 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:0", 2021-02-10T11:31:17.066 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 3079890847 2021-02-10T11:31:17.066 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:31:17.066 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:31:17.066 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": { 2021-02-10T11:31:17.066 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:31:17.067 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:0", 2021-02-10T11:31:17.067 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 3079890847 2021-02-10T11:31:17.067 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:31:17.067 INFO:teuthology.orchestra.run.gibba012.stdout: "inst_str": "client.7077 172.21.2.112:0/3079890847", 2021-02-10T11:31:17.067 INFO:teuthology.orchestra.run.gibba012.stdout: "addr_str": "172.21.2.112:0/3079890847", 2021-02-10T11:31:17.067 INFO:teuthology.orchestra.run.gibba012.stdout: "inode_count": 1, 2021-02-10T11:31:17.067 INFO:teuthology.orchestra.run.gibba012.stdout: "mds_epoch": 13, 2021-02-10T11:31:17.067 INFO:teuthology.orchestra.run.gibba012.stdout: "osd_epoch": 29, 2021-02-10T11:31:17.067 INFO:teuthology.orchestra.run.gibba012.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:31:17.067 INFO:teuthology.orchestra.run.gibba012.stdout: "blacklisted": false 2021-02-10T11:31:17.067 INFO:teuthology.orchestra.run.gibba012.stdout:} 2021-02-10T11:31:17.068 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo getfattr --only-values -n ceph.dir.entries /home/ubuntu/cephtest/mnt.0/./volumes/_deleting 2021-02-10T11:31:17.417 INFO:teuthology.orchestra.run.gibba012.stderr:getfattr: Removing leading '/' from absolute path names 2021-02-10T11:31:17.418 INFO:teuthology.orchestra.run.gibba012.stdout:0 2021-02-10T11:31:17.418 DEBUG:tasks.cephfs.mount:Directory ./volumes/_deleting seen empty from 0 after 0s 2021-02-10T11:31:17.418 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:31:17.419 DEBUG:teuthology.orchestra.run.gibba012:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:31:17.764 DEBUG:teuthology.orchestra.run.gibba020:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:31:18.109 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:31:18.109 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:18.109 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:31:18.109 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:31:18.445 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:31:18.445 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:31:18.445 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba012.front.sepia.ceph.com... 2021-02-10T11:31:18.446 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:31:18.446 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:31:18.801 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba012.stderr:ceph-fuse[118603]: fuse finished with error 0 and tester_r 0 2021-02-10T11:31:18.802 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:18.803 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:31:18.803 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:31:19.136 INFO:teuthology.orchestra.run.gibba012.stdout:ext2/ext3 2021-02-10T11:31:19.137 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:31:19.137 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:31:19.137 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:31:19.471 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:19.471 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:31:19.471 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:31:19.806 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:31:19.807 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:31:19.807 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba012.front.sepia.ceph.com... 2021-02-10T11:31:19.807 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:31:19.807 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:31:20.159 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba012.stderr:ceph-fuse[45683]: fuse finished with error 0 and tester_r 0 2021-02-10T11:31:20.160 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:20.161 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:31:20.161 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:31:20.494 INFO:teuthology.orchestra.run.gibba012.stdout:ext2/ext3 2021-02-10T11:31:20.495 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:31:20.495 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:31:20.495 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:31:20.828 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:20.829 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:31:20.829 DEBUG:teuthology.orchestra.run.gibba020:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:31:21.162 INFO:teuthology.orchestra.run.gibba020.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:31:21.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:31:21.163 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:31:21.163 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:31:21.163 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:31:21.498 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:21.498 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:31:21.499 DEBUG:teuthology.orchestra.run.gibba020:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:31:21.832 INFO:teuthology.orchestra.run.gibba020.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:31:21.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:31:21.832 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:31:21.833 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:31:21.833 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:31:22.167 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config rm mon mon_allow_pool_delete 2021-02-10T11:31:22.597 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:22.593 7f6be98d3700 1 Processor -- start 2021-02-10T11:31:22.597 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:22.593 7f6be98d3700 1 -- start start 2021-02-10T11:31:22.597 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:22.594 7f6be98d3700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6be40d17f0 0x7f6be41b31e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:22.598 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:22.594 7f6be98d3700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f6be40cff00 0x7f6be40ef620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:22.598 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:22.594 7f6be98d3700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6be40f0b20 con 0x7f6be40d1bf0 2021-02-10T11:31:22.598 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:22.594 7f6be98d3700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6be40e5eb0 con 0x7f6be40cff00 2021-02-10T11:31:22.598 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:22.594 7f6be98d3700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6be41b3ff0 con 0x7f6be40d17f0 2021-02-10T11:31:22.598 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:22.594 7f6be2d73700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f6be40d1bf0 0x7f6be41b3800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:47444/0 (socket says 172.21.2.112:47444) 2021-02-10T11:31:22.598 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:22.594 7f6be2d73700 1 -- 172.21.2.112:0/2751699732 learned_addr learned my addr 172.21.2.112:0/2751699732 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:31:22.598 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:22.594 7f6be3d75700 1 --2- 172.21.2.112:0/2751699732 >> v2:172.21.2.120:3300/0 conn(0x7f6be40cff00 0x7f6be40ef620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:22.598 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:22.594 7f6be3574700 1 --2- 172.21.2.112:0/2751699732 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6be40d17f0 0x7f6be41b31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:22.598 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:22.594 7f6be2572700 1 -- 172.21.2.112:0/2751699732 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2907433657 0 0) 0x7f6bd40010b0 con 0x7f6be40d1bf0 2021-02-10T11:31:22.598 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:22.594 7f6be2572700 1 -- 172.21.2.112:0/2751699732 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6bd0002fe0 con 0x7f6be40d1bf0 2021-02-10T11:31:22.598 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:22.594 7f6be3d75700 1 -- 172.21.2.112:0/2751699732 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6be40d17f0 msgr2=0x7f6be41b31e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:22.598 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:22.594 7f6be3d75700 1 --2- 172.21.2.112:0/2751699732 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6be40d17f0 0x7f6be41b31e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:22.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:22.594 7f6be3d75700 1 -- 172.21.2.112:0/2751699732 >> v1:172.21.2.112:6789/0 conn(0x7f6be40d1bf0 legacy=0x7f6be41b3800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:22.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:22.594 7f6be3d75700 1 -- 172.21.2.112:0/2751699732 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6be40f20b0 con 0x7f6be40cff00 2021-02-10T11:31:22.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:22.595 7f6be3d75700 1 --2- 172.21.2.112:0/2751699732 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6be40cff00 0x7f6be40ef620 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 rx=0x7f6bdc006e50 tx=0x7f6bdc009e70).ready entity=mon.1 client_cookie=b996e99fdf2f33cf server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:22.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:22.595 7f6be2572700 1 -- 172.21.2.112:0/2751699732 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6bdc006a20 con 0x7f6be40cff00 2021-02-10T11:31:22.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:22.595 7f6be2572700 1 -- 172.21.2.112:0/2751699732 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bdc006790 con 0x7f6be40cff00 2021-02-10T11:31:22.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:22.595 7f6be2572700 1 -- 172.21.2.112:0/2751699732 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6bdc00b700 con 0x7f6be40cff00 2021-02-10T11:31:22.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:22.595 7f6be98d3700 1 -- 172.21.2.112:0/2751699732 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6be40cff00 msgr2=0x7f6be40ef620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:22.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:22.595 7f6be98d3700 1 --2- 172.21.2.112:0/2751699732 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6be40cff00 0x7f6be40ef620 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 rx=0x7f6bdc006e50 tx=0x7f6bdc009e70).stop 2021-02-10T11:31:22.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:22.595 7f6be98d3700 1 -- 172.21.2.112:0/2751699732 shutdown_connections 2021-02-10T11:31:22.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:22.595 7f6be98d3700 1 --2- 172.21.2.112:0/2751699732 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6be40cff00 0x7f6be40ef620 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:22.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:22.595 7f6be98d3700 1 --2- 172.21.2.112:0/2751699732 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6be40d17f0 0x7f6be41b31e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:22.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:22.595 7f6be98d3700 1 -- 172.21.2.112:0/2751699732 >> 172.21.2.112:0/2751699732 conn(0x7f6be40fde40 msgr2=0x7f6be40ce390 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:22.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:22.595 7f6be98d3700 1 -- 172.21.2.112:0/2751699732 shutdown_connections 2021-02-10T11:31:22.600 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:22.595 7f6be98d3700 1 -- 172.21.2.112:0/2751699732 wait complete. 2021-02-10T11:31:22.600 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:22.610 7f6be98d3700 1 Processor -- start 2021-02-10T11:31:22.600 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:22.610 7f6be98d3700 1 -- start start 2021-02-10T11:31:22.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:22.611 7f6be98d3700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6be408d230 0x7f6be4190980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:22.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:22.611 7f6be98d3700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6be4190fa0 0x7f6be408df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:22.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:22.611 7f6be98d3700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6be41913a0 0x7f6be408e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:22.602 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:22.611 7f6be98d3700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6be408d990 con 0x7f6be4190fa0 2021-02-10T11:31:22.602 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:22.611 7f6be98d3700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6be409ec20 con 0x7f6be408d230 2021-02-10T11:31:22.602 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:22.611 7f6be98d3700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6be408d630 con 0x7f6be41913a0 2021-02-10T11:31:22.602 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:22.611 7f6be2d73700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6be4190fa0 0x7f6be408df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:22.602 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:22.611 7f6be2d73700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6be4190fa0 0x7f6be408df10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:47988/0 (socket says 172.21.2.112:47988) 2021-02-10T11:31:22.602 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:22.611 7f6be2d73700 1 -- 172.21.2.112:0/151689536 learned_addr learned my addr 172.21.2.112:0/151689536 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:31:22.602 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:22.611 7f6be3574700 1 --2- 172.21.2.112:0/151689536 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6be408d230 0x7f6be4190980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:22.602 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:22.611 7f6be3d75700 1 --2- 172.21.2.112:0/151689536 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6be41913a0 0x7f6be408e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:22.602 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:22.611 7f6be3574700 1 -- 172.21.2.112:0/151689536 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6be41913a0 msgr2=0x7f6be408e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:22.602 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:22.611 7f6be3574700 1 --2- 172.21.2.112:0/151689536 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6be41913a0 0x7f6be408e530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:22.602 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:22.611 7f6be3574700 1 -- 172.21.2.112:0/151689536 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6be4190fa0 msgr2=0x7f6be408df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:22.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:22.611 7f6be3574700 1 --2- 172.21.2.112:0/151689536 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6be4190fa0 0x7f6be408df10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:22.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:22.611 7f6be3574700 1 -- 172.21.2.112:0/151689536 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6be408eb50 con 0x7f6be408d230 2021-02-10T11:31:22.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:22.611 7f6be2d73700 1 --2- 172.21.2.112:0/151689536 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6be4190fa0 0x7f6be408df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:31:22.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:22.611 7f6be3574700 1 --2- 172.21.2.112:0/151689536 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6be408d230 0x7f6be4190980 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7f6bcc009540 tx=0x7f6bcc00b8e0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:22.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:22.611 7f6bdbfff700 1 -- 172.21.2.112:0/151689536 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6bcc0009b0 con 0x7f6be408d230 2021-02-10T11:31:22.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:22.611 7f6bdbfff700 1 -- 172.21.2.112:0/151689536 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bcc006340 con 0x7f6be408d230 2021-02-10T11:31:22.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:22.611 7f6be98d3700 1 -- 172.21.2.112:0/151689536 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6be408eb50 con 0x7f6be408d230 2021-02-10T11:31:22.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:22.611 7f6bdbfff700 1 -- 172.21.2.112:0/151689536 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6bcc0009b0 con 0x7f6be408d230 2021-02-10T11:31:22.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:22.612 7f6be98d3700 1 -- 172.21.2.112:0/151689536 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6be408eb50 con 0x7f6be408d230 2021-02-10T11:31:22.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:22.612 7f6bdbfff700 1 -- 172.21.2.112:0/151689536 <== mon.1 v2:172.21.2.120:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6bcc0052e0 con 0x7f6be408d230 2021-02-10T11:31:22.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:22.612 7f6bdbfff700 1 --2- 172.21.2.112:0/151689536 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6bc8018730 0x7f6bc801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:22.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:22.612 7f6bdbfff700 1 -- 172.21.2.112:0/151689536 <== mon.1 v2:172.21.2.120:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f6bcc032540 con 0x7f6be408d230 2021-02-10T11:31:22.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:22.612 7f6be2d73700 1 --2- 172.21.2.112:0/151689536 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6bc8018730 0x7f6bc801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:22.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:22.613 7f6be2d73700 1 --2- 172.21.2.112:0/151689536 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6bc8018730 0x7f6bc801ac10 crc :-1 s=READY pgs=494 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:22.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:22.644 7f6be98d3700 1 -- 172.21.2.112:0/151689536 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6be408d990 con 0x7f6be408d230 2021-02-10T11:31:22.634 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:22.646 7f6bdbfff700 1 -- 172.21.2.112:0/151689536 <== mon.1 v2:172.21.2.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6bcc0009b0 con 0x7f6be408d230 2021-02-10T11:31:22.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:22.796 7f6be98d3700 1 -- 172.21.2.112:0/151689536 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f6be408d990 con 0x7f6be408d230 2021-02-10T11:31:22.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:22.801 7f6bdbfff700 1 -- 172.21.2.112:0/151689536 <== mon.1 v2:172.21.2.120:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v3) v1 ==== 104+0+0 (secure 0 0 0) 0x7f6bcc01f400 con 0x7f6be408d230 2021-02-10T11:31:22.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:22.804 7f6be98d3700 1 -- 172.21.2.112:0/151689536 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6bc8018730 msgr2=0x7f6bc801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:22.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:22.804 7f6be98d3700 1 --2- 172.21.2.112:0/151689536 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6bc8018730 0x7f6bc801ac10 crc :-1 s=READY pgs=494 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:22.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:22.804 7f6be98d3700 1 -- 172.21.2.112:0/151689536 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6be408d230 msgr2=0x7f6be4190980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:22.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:22.804 7f6be98d3700 1 --2- 172.21.2.112:0/151689536 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6be408d230 0x7f6be4190980 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7f6bcc009540 tx=0x7f6bcc00b8e0).stop 2021-02-10T11:31:22.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:22.805 7f6be98d3700 1 -- 172.21.2.112:0/151689536 shutdown_connections 2021-02-10T11:31:22.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:22.805 7f6be98d3700 1 --2- 172.21.2.112:0/151689536 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6bc8018730 0x7f6bc801ac10 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:22.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:22.805 7f6be98d3700 1 --2- 172.21.2.112:0/151689536 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6be41913a0 0x7f6be408e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:22.793 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:22.805 7f6be98d3700 1 --2- 172.21.2.112:0/151689536 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6be4190fa0 0x7f6be408df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:22.793 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:22.805 7f6be98d3700 1 --2- 172.21.2.112:0/151689536 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6be408d230 0x7f6be4190980 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:22.793 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:22.805 7f6be98d3700 1 -- 172.21.2.112:0/151689536 >> 172.21.2.112:0/151689536 conn(0x7f6be409fbb0 msgr2=0x7f6be40a7580 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:22.793 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:22.805 7f6be98d3700 1 -- 172.21.2.112:0/151689536 shutdown_connections 2021-02-10T11:31:22.793 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:22.805 7f6be98d3700 1 -- 172.21.2.112:0/151689536 wait complete. 2021-02-10T11:31:22.801 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestVolumes.test_async_subvolume_rm' 2021-02-10T11:31:23.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:23.240 7f03472c8700 1 Processor -- start 2021-02-10T11:31:23.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:23.240 7f03472c8700 1 -- start start 2021-02-10T11:31:23.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:23.240 7f03472c8700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f03400e5070 0x7f03401b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:23.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:23.240 7f03472c8700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f03401b3e40 0x7f034007d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:23.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:23.240 7f03472c8700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f034007ed80 con 0x7f03400e5470 2021-02-10T11:31:23.245 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:23.241 7f03472c8700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0340080310 con 0x7f03401b3e40 2021-02-10T11:31:23.245 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:23.241 7f03472c8700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0340080530 con 0x7f03400e5070 2021-02-10T11:31:23.245 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:23.241 7f03448e3700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f03401b3e40 0x7f034007d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:23.245 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:23.241 7f03448e3700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f03401b3e40 0x7f034007d690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:47994/0 (socket says 172.21.2.112:47994) 2021-02-10T11:31:23.245 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:23.241 7f03448e3700 1 -- 172.21.2.112:0/1905765611 learned_addr learned my addr 172.21.2.112:0/1905765611 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:31:23.245 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:23.241 7f03458e5700 1 --1- 172.21.2.112:0/1905765611 >> v1:172.21.2.112:6789/0 conn(0x7f03400e5470 0x7f034007dcb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:47456/0 (socket says 172.21.2.112:47456) 2021-02-10T11:31:23.245 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:23.241 7f03448e3700 1 -- 172.21.2.112:0/1905765611 >> v2:172.21.2.120:3300/0 conn(0x7f03400e5070 msgr2=0x7f03401b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:23.245 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:23.241 7f03448e3700 1 --2- 172.21.2.112:0/1905765611 >> v2:172.21.2.120:3300/0 conn(0x7f03400e5070 0x7f03401b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:23.245 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:23.241 7f03448e3700 1 -- 172.21.2.112:0/1905765611 >> v1:172.21.2.112:6789/0 conn(0x7f03400e5470 legacy=0x7f034007dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:23.245 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:23.241 7f03448e3700 1 -- 172.21.2.112:0/1905765611 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0340080750 con 0x7f03401b3e40 2021-02-10T11:31:23.245 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:23.241 7f03448e3700 1 --2- 172.21.2.112:0/1905765611 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f03401b3e40 0x7f034007d690 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 rx=0x7f0334006700 tx=0x7f0334009b80).ready entity=mon.0 client_cookie=71710e22a4eb7a14 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:23.245 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:23.242 7f0333fff700 1 -- 172.21.2.112:0/1905765611 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f033400c0c0 con 0x7f03401b3e40 2021-02-10T11:31:23.245 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:23.242 7f0333fff700 1 -- 172.21.2.112:0/1905765611 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f033400c440 con 0x7f03401b3e40 2021-02-10T11:31:23.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:23.242 7f0333fff700 1 -- 172.21.2.112:0/1905765611 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f033400c8a0 con 0x7f03401b3e40 2021-02-10T11:31:23.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:23.242 7f03472c8700 1 -- 172.21.2.112:0/1905765611 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f03401b3e40 msgr2=0x7f034007d690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:23.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:23.242 7f03472c8700 1 --2- 172.21.2.112:0/1905765611 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f03401b3e40 0x7f034007d690 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 rx=0x7f0334006700 tx=0x7f0334009b80).stop 2021-02-10T11:31:23.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:23.242 7f03472c8700 1 -- 172.21.2.112:0/1905765611 shutdown_connections 2021-02-10T11:31:23.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:23.242 7f03472c8700 1 --2- 172.21.2.112:0/1905765611 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f03401b3e40 0x7f034007d690 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:23.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:23.242 7f03472c8700 1 --2- 172.21.2.112:0/1905765611 >> v2:172.21.2.120:3300/0 conn(0x7f03400e5070 0x7f03401b3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:23.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:23.242 7f03472c8700 1 -- 172.21.2.112:0/1905765611 >> 172.21.2.112:0/1905765611 conn(0x7f03400ce690 msgr2=0x7f03400e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:23.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:23.243 7f03472c8700 1 -- 172.21.2.112:0/1905765611 shutdown_connections 2021-02-10T11:31:23.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:23.243 7f03472c8700 1 -- 172.21.2.112:0/1905765611 wait complete. 2021-02-10T11:31:23.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:23.257 7f03472c8700 1 Processor -- start 2021-02-10T11:31:23.248 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:23.257 7f03472c8700 1 -- start start 2021-02-10T11:31:23.248 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:23.258 7f03472c8700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0340192fc0 0x7f0340192780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:23.248 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:23.258 7f03472c8700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f03400b1430 0x7f03400b1830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:23.248 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:23.258 7f03472c8700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f03400b1e50 0x7f034008a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:23.248 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:23.258 7f03472c8700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03400b2250 con 0x7f03400b1430 2021-02-10T11:31:23.248 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:23.258 7f03472c8700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f034008b5d0 con 0x7f0340192fc0 2021-02-10T11:31:23.248 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:23.258 7f03472c8700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f034008aa80 con 0x7f03400b1e50 2021-02-10T11:31:23.248 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:23.258 7f03448e3700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f03400b1430 0x7f03400b1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:23.248 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:23.258 7f03448e3700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f03400b1430 0x7f03400b1830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:47998/0 (socket says 172.21.2.112:47998) 2021-02-10T11:31:23.248 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:23.258 7f03448e3700 1 -- 172.21.2.112:0/4186293566 learned_addr learned my addr 172.21.2.112:0/4186293566 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:31:23.248 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:23.258 7f03448e3700 1 -- 172.21.2.112:0/4186293566 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f03400b1e50 msgr2=0x7f034008a460 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:31:23.248 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:23.258 7f03448e3700 1 --2- 172.21.2.112:0/4186293566 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f03400b1e50 0x7f034008a460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:23.249 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:23.258 7f03448e3700 1 -- 172.21.2.112:0/4186293566 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0340192fc0 msgr2=0x7f0340192780 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:31:23.249 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:23.258 7f03448e3700 1 --2- 172.21.2.112:0/4186293566 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0340192fc0 0x7f0340192780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:23.249 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:23.258 7f03448e3700 1 -- 172.21.2.112:0/4186293566 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f034008ac60 con 0x7f03400b1430 2021-02-10T11:31:23.249 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:23.258 7f03448e3700 1 --2- 172.21.2.112:0/4186293566 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f03400b1430 0x7f03400b1830 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 rx=0x7f033400b5e0 tx=0x7f033400b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:23.249 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:23.259 7f0331ffb700 1 -- 172.21.2.112:0/4186293566 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f033400bd70 con 0x7f03400b1430 2021-02-10T11:31:23.249 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:23.259 7f0331ffb700 1 -- 172.21.2.112:0/4186293566 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0334002030 con 0x7f03400b1430 2021-02-10T11:31:23.249 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:23.259 7f0331ffb700 1 -- 172.21.2.112:0/4186293566 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0334002550 con 0x7f03400b1430 2021-02-10T11:31:23.249 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:23.259 7f03472c8700 1 -- 172.21.2.112:0/4186293566 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f034008ac60 con 0x7f03400b1430 2021-02-10T11:31:23.249 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:23.259 7f03472c8700 1 -- 172.21.2.112:0/4186293566 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f034008b1f0 con 0x7f03400b1430 2021-02-10T11:31:23.249 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:23.259 7f0331ffb700 1 -- 172.21.2.112:0/4186293566 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0334000c90 con 0x7f03400b1430 2021-02-10T11:31:23.249 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:23.259 7f0331ffb700 1 --2- 172.21.2.112:0/4186293566 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f03200186c0 0x7f032001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:23.249 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:23.259 7f0331ffb700 1 -- 172.21.2.112:0/4186293566 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f03340325f0 con 0x7f03400b1430 2021-02-10T11:31:23.249 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:23.260 7f03450e4700 1 --2- 172.21.2.112:0/4186293566 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f03200186c0 0x7f032001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:23.249 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:23.260 7f03450e4700 1 --2- 172.21.2.112:0/4186293566 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f03200186c0 0x7f032001abc0 crc :-1 s=READY pgs=495 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:23.279 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:23.291 7f03472c8700 1 -- 172.21.2.112:0/4186293566 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0340107120 con 0x7f03400b1430 2021-02-10T11:31:23.281 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:23.293 7f0331ffb700 1 -- 172.21.2.112:0/4186293566 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f033402cf70 con 0x7f03400b1430 2021-02-10T11:31:23.411 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:23.424 7f03472c8700 1 -- 172.21.2.112:0/4186293566 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_async_subvolume_rm"]} v 0) v1 -- 0x7f034007ed80 con 0x7f03400b1430 2021-02-10T11:31:24.026 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:24.038 7f0331ffb700 1 -- 172.21.2.112:0/4186293566 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_async_subvolume_rm"]}]=0 v205) v1 ==== 140+0+0 (secure 0 0 0) 0x7f033402cf70 con 0x7f03400b1430 2021-02-10T11:31:24.031 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:24.043 7f03472c8700 1 -- 172.21.2.112:0/4186293566 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f03200186c0 msgr2=0x7f032001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:24.031 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:24.043 7f03472c8700 1 --2- 172.21.2.112:0/4186293566 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f03200186c0 0x7f032001abc0 crc :-1 s=READY pgs=495 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:24.031 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:24.043 7f03472c8700 1 -- 172.21.2.112:0/4186293566 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f03400b1430 msgr2=0x7f03400b1830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:24.031 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:24.043 7f03472c8700 1 --2- 172.21.2.112:0/4186293566 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f03400b1430 0x7f03400b1830 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 rx=0x7f033400b5e0 tx=0x7f033400b980).stop 2021-02-10T11:31:24.032 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:24.044 7f03472c8700 1 -- 172.21.2.112:0/4186293566 shutdown_connections 2021-02-10T11:31:24.032 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:24.044 7f03472c8700 1 --2- 172.21.2.112:0/4186293566 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f03200186c0 0x7f032001abc0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:24.032 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:24.044 7f03472c8700 1 --2- 172.21.2.112:0/4186293566 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f03400b1e50 0x7f034008a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:24.032 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:24.044 7f03472c8700 1 --2- 172.21.2.112:0/4186293566 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f03400b1430 0x7f03400b1830 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:24.032 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:24.044 7f03472c8700 1 --2- 172.21.2.112:0/4186293566 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0340192fc0 0x7f0340192780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:24.032 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:24.044 7f03472c8700 1 -- 172.21.2.112:0/4186293566 >> 172.21.2.112:0/4186293566 conn(0x7f03401ac620 msgr2=0x7f034019fee0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:24.032 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:24.044 7f03472c8700 1 -- 172.21.2.112:0/4186293566 shutdown_connections 2021-02-10T11:31:24.032 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:24.044 7f03472c8700 1 -- 172.21.2.112:0/4186293566 wait complete. 2021-02-10T11:31:24.040 INFO:tasks.cephfs_test_runner:test_async_subvolume_rm (tasks.cephfs.test_volumes.TestVolumes) ... ok 2021-02-10T11:31:24.040 INFO:tasks.cephfs_test_runner:Starting test: test_authorize_allow_existing_id_option (tasks.cephfs.test_volumes.TestVolumes) 2021-02-10T11:31:24.040 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestVolumes.test_authorize_allow_existing_id_option' 2021-02-10T11:31:24.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:24.467 7f7d3391d700 1 Processor -- start 2021-02-10T11:31:24.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:24.467 7f7d3391d700 1 -- start start 2021-02-10T11:31:24.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:24.467 7f7d3391d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7d2c0d17f0 0x7f7d2c1b31e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:24.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:24.467 7f7d3391d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f7d2c0cff00 0x7f7d2c0ef620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:24.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:24.467 7f7d3391d700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d2c0f0b20 con 0x7f7d2c0d1bf0 2021-02-10T11:31:24.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:24.467 7f7d3391d700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d2c0e5eb0 con 0x7f7d2c0cff00 2021-02-10T11:31:24.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:24.468 7f7d3391d700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d2c1b3ff0 con 0x7f7d2c0d17f0 2021-02-10T11:31:24.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:24.468 7f7d31f3a700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f7d2c0cff00 0x7f7d2c0ef620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:24.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:24.468 7f7d30f38700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7f7d2c0d1bf0 0x7f7d2c1b3800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:41026/0 (socket says 172.21.2.112:41026) 2021-02-10T11:31:24.473 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:24.468 7f7d30f38700 1 -- 172.21.2.112:0/3771527278 learned_addr learned my addr 172.21.2.112:0/3771527278 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:31:24.473 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:24.468 7f7d31739700 1 --2- 172.21.2.112:0/3771527278 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7d2c0d17f0 0x7f7d2c1b31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:24.473 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:24.468 7f7d1ffff700 1 -- 172.21.2.112:0/3771527278 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2398286133 0 0) 0x7f7d200010c0 con 0x7f7d2c0d1bf0 2021-02-10T11:31:24.473 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:24.468 7f7d1ffff700 1 -- 172.21.2.112:0/3771527278 --> v1:172.21.2.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d18002fe0 con 0x7f7d2c0d1bf0 2021-02-10T11:31:24.473 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:24.468 7f7d31f3a700 1 -- 172.21.2.112:0/3771527278 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7d2c0d17f0 msgr2=0x7f7d2c1b31e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:24.473 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:24.468 7f7d31f3a700 1 --2- 172.21.2.112:0/3771527278 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7d2c0d17f0 0x7f7d2c1b31e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:24.473 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:24.468 7f7d31f3a700 1 -- 172.21.2.112:0/3771527278 >> v1:172.21.2.120:6789/0 conn(0x7f7d2c0d1bf0 legacy=0x7f7d2c1b3800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:24.473 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:24.468 7f7d31f3a700 1 -- 172.21.2.112:0/3771527278 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d2c0f20b0 con 0x7f7d2c0cff00 2021-02-10T11:31:24.473 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:24.469 7f7d31f3a700 1 --2- 172.21.2.112:0/3771527278 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7d2c0cff00 0x7f7d2c0ef620 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 rx=0x7f7d24006e30 tx=0x7f7d24009e60).ready entity=mon.0 client_cookie=f05366b125ac0ad0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:24.473 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:24.469 7f7d1ffff700 1 -- 172.21.2.112:0/3771527278 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7d24006a50 con 0x7f7d2c0cff00 2021-02-10T11:31:24.473 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:24.469 7f7d1ffff700 1 -- 172.21.2.112:0/3771527278 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d2400b300 con 0x7f7d2c0cff00 2021-02-10T11:31:24.473 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:24.469 7f7d1ffff700 1 -- 172.21.2.112:0/3771527278 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7d2400b7e0 con 0x7f7d2c0cff00 2021-02-10T11:31:24.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:24.469 7f7d3391d700 1 -- 172.21.2.112:0/3771527278 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7d2c0cff00 msgr2=0x7f7d2c0ef620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:24.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:24.469 7f7d3391d700 1 --2- 172.21.2.112:0/3771527278 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7d2c0cff00 0x7f7d2c0ef620 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 rx=0x7f7d24006e30 tx=0x7f7d24009e60).stop 2021-02-10T11:31:24.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:24.470 7f7d3391d700 1 -- 172.21.2.112:0/3771527278 shutdown_connections 2021-02-10T11:31:24.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:24.470 7f7d3391d700 1 --2- 172.21.2.112:0/3771527278 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7d2c0cff00 0x7f7d2c0ef620 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:24.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:24.470 7f7d3391d700 1 --2- 172.21.2.112:0/3771527278 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7d2c0d17f0 0x7f7d2c1b31e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:24.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:24.470 7f7d3391d700 1 -- 172.21.2.112:0/3771527278 >> 172.21.2.112:0/3771527278 conn(0x7f7d2c0fde40 msgr2=0x7f7d2c0ce390 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:24.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:24.470 7f7d3391d700 1 -- 172.21.2.112:0/3771527278 shutdown_connections 2021-02-10T11:31:24.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:24.470 7f7d3391d700 1 -- 172.21.2.112:0/3771527278 wait complete. 2021-02-10T11:31:24.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:24.485 7f7d3391d700 1 Processor -- start 2021-02-10T11:31:24.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:24.485 7f7d3391d700 1 -- start start 2021-02-10T11:31:24.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:24.485 7f7d3391d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7d2c098310 0x7f7d2c097730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:24.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:24.485 7f7d3391d700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7d2c0bf830 0x7f7d2c0966c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:24.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:24.485 7f7d3391d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7d2c0bfc30 0x7f7d2c096ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:24.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:24.485 7f7d3391d700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d2c0961c0 con 0x7f7d2c098310 2021-02-10T11:31:24.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:24.485 7f7d3391d700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d2c097e40 con 0x7f7d2c0bf830 2021-02-10T11:31:24.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:24.485 7f7d3391d700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d2c0987a0 con 0x7f7d2c0bfc30 2021-02-10T11:31:24.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:24.485 7f7d31739700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7d2c098310 0x7f7d2c097730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:24.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:24.485 7f7d31739700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7d2c098310 0x7f7d2c097730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:48014/0 (socket says 172.21.2.112:48014) 2021-02-10T11:31:24.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:24.485 7f7d31739700 1 -- 172.21.2.112:0/1656374319 learned_addr learned my addr 172.21.2.112:0/1656374319 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:31:24.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:24.486 7f7d31739700 1 -- 172.21.2.112:0/1656374319 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7d2c0bfc30 msgr2=0x7f7d2c096ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:24.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:24.486 7f7d31739700 1 --2- 172.21.2.112:0/1656374319 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7d2c0bfc30 0x7f7d2c096ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:24.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:24.486 7f7d31739700 1 -- 172.21.2.112:0/1656374319 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7d2c0bf830 msgr2=0x7f7d2c0966c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:24.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:24.486 7f7d31739700 1 --2- 172.21.2.112:0/1656374319 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7d2c0bf830 0x7f7d2c0966c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:24.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:24.486 7f7d31739700 1 -- 172.21.2.112:0/1656374319 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d2c0989c0 con 0x7f7d2c098310 2021-02-10T11:31:24.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:24.486 7f7d31739700 1 --2- 172.21.2.112:0/1656374319 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7d2c098310 0x7f7d2c097730 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 rx=0x7f7d28006f60 tx=0x7f7d2800b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:24.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:24.486 7f7d1dffb700 1 -- 172.21.2.112:0/1656374319 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7d2800bd60 con 0x7f7d2c098310 2021-02-10T11:31:24.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:24.486 7f7d1dffb700 1 -- 172.21.2.112:0/1656374319 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d28000c90 con 0x7f7d2c098310 2021-02-10T11:31:24.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:24.486 7f7d3391d700 1 -- 172.21.2.112:0/1656374319 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d2c0989c0 con 0x7f7d2c098310 2021-02-10T11:31:24.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:24.486 7f7d3391d700 1 -- 172.21.2.112:0/1656374319 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d2c098f50 con 0x7f7d2c098310 2021-02-10T11:31:24.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:24.486 7f7d1dffb700 1 -- 172.21.2.112:0/1656374319 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7d28002220 con 0x7f7d2c098310 2021-02-10T11:31:24.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:24.487 7f7d1dffb700 1 -- 172.21.2.112:0/1656374319 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7d280052e0 con 0x7f7d2c098310 2021-02-10T11:31:24.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:24.487 7f7d1dffb700 1 --2- 172.21.2.112:0/1656374319 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7d0c0186c0 0x7f7d0c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:24.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:24.487 7f7d1dffb700 1 -- 172.21.2.112:0/1656374319 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f7d28032520 con 0x7f7d2c098310 2021-02-10T11:31:24.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:24.487 7f7d30f38700 1 --2- 172.21.2.112:0/1656374319 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7d0c0186c0 0x7f7d0c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:24.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:24.488 7f7d30f38700 1 --2- 172.21.2.112:0/1656374319 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7d0c0186c0 0x7f7d0c01abc0 crc :-1 s=READY pgs=496 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:24.505 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:24.517 7f7d3391d700 1 -- 172.21.2.112:0/1656374319 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d2c1036a0 con 0x7f7d2c098310 2021-02-10T11:31:24.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:24.520 7f7d1dffb700 1 -- 172.21.2.112:0/1656374319 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7d28000940 con 0x7f7d2c098310 2021-02-10T11:31:24.639 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:24.652 7f7d3391d700 1 -- 172.21.2.112:0/1656374319 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_authorize_allow_existing_id_option"]} v 0) v1 -- 0x7f7d2c0f0b20 con 0x7f7d2c098310 2021-02-10T11:31:25.029 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:25.041 7f7d1dffb700 1 -- 172.21.2.112:0/1656374319 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_authorize_allow_existing_id_option"]}]=0 v206) v1 ==== 159+0+0 (secure 0 0 0) 0x7f7d280052e0 con 0x7f7d2c098310 2021-02-10T11:31:25.058 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:25.070 7f7d3391d700 1 -- 172.21.2.112:0/1656374319 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7d0c0186c0 msgr2=0x7f7d0c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:25.058 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:25.070 7f7d3391d700 1 --2- 172.21.2.112:0/1656374319 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7d0c0186c0 0x7f7d0c01abc0 crc :-1 s=READY pgs=496 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:25.058 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:25.070 7f7d3391d700 1 -- 172.21.2.112:0/1656374319 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7d2c098310 msgr2=0x7f7d2c097730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:25.058 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:25.070 7f7d3391d700 1 --2- 172.21.2.112:0/1656374319 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7d2c098310 0x7f7d2c097730 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 rx=0x7f7d28006f60 tx=0x7f7d2800b970).stop 2021-02-10T11:31:25.058 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:25.070 7f7d3391d700 1 -- 172.21.2.112:0/1656374319 shutdown_connections 2021-02-10T11:31:25.058 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:25.070 7f7d3391d700 1 --2- 172.21.2.112:0/1656374319 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7d0c0186c0 0x7f7d0c01abc0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:25.059 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:25.070 7f7d3391d700 1 --2- 172.21.2.112:0/1656374319 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7d2c0bf830 0x7f7d2c0966c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:25.059 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:25.070 7f7d3391d700 1 --2- 172.21.2.112:0/1656374319 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7d2c0bfc30 0x7f7d2c096ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:25.059 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:25.070 7f7d3391d700 1 --2- 172.21.2.112:0/1656374319 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7d2c098310 0x7f7d2c097730 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:25.059 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:25.070 7f7d3391d700 1 -- 172.21.2.112:0/1656374319 >> 172.21.2.112:0/1656374319 conn(0x7f7d2c099980 msgr2=0x7f7d2c0c3c60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:25.059 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:25.071 7f7d3391d700 1 -- 172.21.2.112:0/1656374319 shutdown_connections 2021-02-10T11:31:25.059 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:25.071 7f7d3391d700 1 -- 172.21.2.112:0/1656374319 wait complete. 2021-02-10T11:31:25.066 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:31:25.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:25.492 7f5a90137700 1 Processor -- start 2021-02-10T11:31:25.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:25.492 7f5a90137700 1 -- start start 2021-02-10T11:31:25.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:25.492 7f5a90137700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5a881b0470 0x7f5a881b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:25.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:25.492 7f5a90137700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f5a880f20e0 0x7f5a880f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:25.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:25.492 7f5a90137700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a880f0b50 con 0x7f5a880ef610 2021-02-10T11:31:25.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:25.492 7f5a90137700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a880f0820 con 0x7f5a880f20e0 2021-02-10T11:31:25.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:25.492 7f5a90137700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a880f24e0 con 0x7f5a881b0470 2021-02-10T11:31:25.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:25.493 7f5a8e754700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f5a880f20e0 0x7f5a880f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:25.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:25.493 7f5a8df53700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5a881b0470 0x7f5a881b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:25.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:25.493 7f5a8d752700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7f5a880ef610 0x7f5a880efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:41040/0 (socket says 172.21.2.112:41040) 2021-02-10T11:31:25.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:25.493 7f5a8d752700 1 -- 172.21.2.112:0/3866737833 learned_addr learned my addr 172.21.2.112:0/3866737833 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:31:25.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:25.494 7f5a8cf51700 1 -- 172.21.2.112:0/3866737833 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3045884806 0 0) 0x7f5a840009b0 con 0x7f5a880ef610 2021-02-10T11:31:25.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:25.494 7f5a8cf51700 1 -- 172.21.2.112:0/3866737833 --> v1:172.21.2.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a74002fe0 con 0x7f5a880ef610 2021-02-10T11:31:25.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:25.494 7f5a8e754700 1 -- 172.21.2.112:0/3866737833 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5a881b0470 msgr2=0x7f5a881b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:25.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:25.494 7f5a8e754700 1 --2- 172.21.2.112:0/3866737833 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5a881b0470 0x7f5a881b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:25.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:25.494 7f5a8e754700 1 -- 172.21.2.112:0/3866737833 >> v1:172.21.2.120:6789/0 conn(0x7f5a880ef610 legacy=0x7f5a880efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:25.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:25.494 7f5a8e754700 1 -- 172.21.2.112:0/3866737833 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a880e52e0 con 0x7f5a880f20e0 2021-02-10T11:31:25.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:25.494 7f5a8df53700 1 --2- 172.21.2.112:0/3866737833 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5a881b0470 0x7f5a881b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:31:25.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:25.494 7f5a8e754700 1 --2- 172.21.2.112:0/3866737833 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5a880f20e0 0x7f5a880f0200 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 rx=0x7f5a80006e50 tx=0x7f5a80009e70).ready entity=mon.1 client_cookie=2a43622d32b816a8 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:25.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:25.495 7f5a8cf51700 1 -- 172.21.2.112:0/3866737833 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5a80006aa0 con 0x7f5a880f20e0 2021-02-10T11:31:25.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:25.495 7f5a8cf51700 1 -- 172.21.2.112:0/3866737833 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a80006790 con 0x7f5a880f20e0 2021-02-10T11:31:25.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:25.495 7f5a8cf51700 1 -- 172.21.2.112:0/3866737833 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5a8000b700 con 0x7f5a880f20e0 2021-02-10T11:31:25.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:25.495 7f5a90137700 1 -- 172.21.2.112:0/3866737833 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5a880f20e0 msgr2=0x7f5a880f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:25.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:25.495 7f5a90137700 1 --2- 172.21.2.112:0/3866737833 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5a880f20e0 0x7f5a880f0200 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 rx=0x7f5a80006e50 tx=0x7f5a80009e70).stop 2021-02-10T11:31:25.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:25.495 7f5a90137700 1 -- 172.21.2.112:0/3866737833 shutdown_connections 2021-02-10T11:31:25.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:25.495 7f5a90137700 1 --2- 172.21.2.112:0/3866737833 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5a880f20e0 0x7f5a880f0200 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:25.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:25.495 7f5a90137700 1 --2- 172.21.2.112:0/3866737833 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5a881b0470 0x7f5a881b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:25.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:25.495 7f5a90137700 1 -- 172.21.2.112:0/3866737833 >> 172.21.2.112:0/3866737833 conn(0x7f5a880ce3e0 msgr2=0x7f5a880e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:25.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:25.496 7f5a90137700 1 -- 172.21.2.112:0/3866737833 shutdown_connections 2021-02-10T11:31:25.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:25.496 7f5a90137700 1 -- 172.21.2.112:0/3866737833 wait complete. 2021-02-10T11:31:25.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:25.510 7f5a90137700 1 Processor -- start 2021-02-10T11:31:25.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:25.510 7f5a90137700 1 -- start start 2021-02-10T11:31:25.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:25.510 7f5a90137700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5a880b7db0 0x7f5a880b8260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:25.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:25.510 7f5a90137700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5a880b8880 0x7f5a881a7450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:25.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:25.510 7f5a90137700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5a880b8c80 0x7f5a881a7a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:25.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:25.510 7f5a90137700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a880b79b0 con 0x7f5a880b8880 2021-02-10T11:31:25.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:25.510 7f5a90137700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a881a8090 con 0x7f5a880b8c80 2021-02-10T11:31:25.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:25.510 7f5a90137700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a881a82b0 con 0x7f5a880b7db0 2021-02-10T11:31:25.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:25.511 7f5a8d752700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5a880b8880 0x7f5a881a7450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:25.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:25.511 7f5a8d752700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5a880b8880 0x7f5a881a7450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:48024/0 (socket says 172.21.2.112:48024) 2021-02-10T11:31:25.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:25.511 7f5a8d752700 1 -- 172.21.2.112:0/220862278 learned_addr learned my addr 172.21.2.112:0/220862278 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:31:25.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:25.511 7f5a8d752700 1 -- 172.21.2.112:0/220862278 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5a880b7db0 msgr2=0x7f5a880b8260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:25.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:25.511 7f5a8d752700 1 --2- 172.21.2.112:0/220862278 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5a880b7db0 0x7f5a880b8260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:25.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:25.511 7f5a8d752700 1 -- 172.21.2.112:0/220862278 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5a880b8c80 msgr2=0x7f5a881a7a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:25.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:25.511 7f5a8d752700 1 --2- 172.21.2.112:0/220862278 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5a880b8c80 0x7f5a881a7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:25.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:25.511 7f5a8d752700 1 -- 172.21.2.112:0/220862278 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a88190c10 con 0x7f5a880b8880 2021-02-10T11:31:25.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:25.511 7f5a8d752700 1 --2- 172.21.2.112:0/220862278 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5a880b8880 0x7f5a881a7450 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 rx=0x7f5a84001c60 tx=0x7f5a8400a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:25.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:25.511 7f5a7a7fc700 1 -- 172.21.2.112:0/220862278 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5a8400acc0 con 0x7f5a880b8880 2021-02-10T11:31:25.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:25.511 7f5a7a7fc700 1 -- 172.21.2.112:0/220862278 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a84004350 con 0x7f5a880b8880 2021-02-10T11:31:25.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:25.511 7f5a7a7fc700 1 -- 172.21.2.112:0/220862278 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5a840047b0 con 0x7f5a880b8880 2021-02-10T11:31:25.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:25.511 7f5a90137700 1 -- 172.21.2.112:0/220862278 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a88190c10 con 0x7f5a880b8880 2021-02-10T11:31:25.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:25.512 7f5a90137700 1 -- 172.21.2.112:0/220862278 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a88190c10 con 0x7f5a880b8880 2021-02-10T11:31:25.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:25.512 7f5a7a7fc700 1 -- 172.21.2.112:0/220862278 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5a84004530 con 0x7f5a880b8880 2021-02-10T11:31:25.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:25.512 7f5a7a7fc700 1 --2- 172.21.2.112:0/220862278 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5a700186c0 0x7f5a7001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:25.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:25.512 7f5a7a7fc700 1 -- 172.21.2.112:0/220862278 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f5a84032410 con 0x7f5a880b8880 2021-02-10T11:31:25.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:25.513 7f5a8df53700 1 --2- 172.21.2.112:0/220862278 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5a700186c0 0x7f5a7001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:25.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:25.513 7f5a8df53700 1 --2- 172.21.2.112:0/220862278 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5a700186c0 0x7f5a7001abc0 crc :-1 s=READY pgs=497 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:25.530 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:25.542 7f5a90137700 1 -- 172.21.2.112:0/220862278 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a88105b30 con 0x7f5a880b8880 2021-02-10T11:31:25.532 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:25.545 7f5a7a7fc700 1 -- 172.21.2.112:0/220862278 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5a84004140 con 0x7f5a880b8880 2021-02-10T11:31:25.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:25.695 7f5a90137700 1 -- 172.21.2.112:0/220862278 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f5a880f0b50 con 0x7f5a880b8880 2021-02-10T11:31:25.686 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:25.699 7f5a7a7fc700 1 -- 172.21.2.112:0/220862278 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v4)=0 v4) v1 ==== 122+0+0 (secure 0 0 0) 0x7f5a84004530 con 0x7f5a880b8880 2021-02-10T11:31:25.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:25.704 7f5a90137700 1 -- 172.21.2.112:0/220862278 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5a700186c0 msgr2=0x7f5a7001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:25.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:25.704 7f5a90137700 1 --2- 172.21.2.112:0/220862278 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5a700186c0 0x7f5a7001abc0 crc :-1 s=READY pgs=497 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:25.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:25.704 7f5a90137700 1 -- 172.21.2.112:0/220862278 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5a880b8880 msgr2=0x7f5a881a7450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:25.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:25.704 7f5a90137700 1 --2- 172.21.2.112:0/220862278 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5a880b8880 0x7f5a881a7450 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 rx=0x7f5a84001c60 tx=0x7f5a8400a8d0).stop 2021-02-10T11:31:25.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:25.704 7f5a90137700 1 -- 172.21.2.112:0/220862278 shutdown_connections 2021-02-10T11:31:25.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:25.704 7f5a90137700 1 --2- 172.21.2.112:0/220862278 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5a700186c0 0x7f5a7001abc0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:25.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:25.704 7f5a90137700 1 --2- 172.21.2.112:0/220862278 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5a880b8c80 0x7f5a881a7a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:25.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:25.704 7f5a90137700 1 --2- 172.21.2.112:0/220862278 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5a880b8880 0x7f5a881a7450 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:25.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:25.704 7f5a90137700 1 --2- 172.21.2.112:0/220862278 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5a880b7db0 0x7f5a880b8260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:25.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:25.704 7f5a90137700 1 -- 172.21.2.112:0/220862278 >> 172.21.2.112:0/220862278 conn(0x7f5a88094e10 msgr2=0x7f5a880a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:25.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:25.704 7f5a90137700 1 -- 172.21.2.112:0/220862278 shutdown_connections 2021-02-10T11:31:25.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:25.704 7f5a90137700 1 -- 172.21.2.112:0/220862278 wait complete. 2021-02-10T11:31:25.700 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:31:25.700 DEBUG:teuthology.orchestra.run.gibba012:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:31:26.044 DEBUG:teuthology.orchestra.run.gibba020:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:31:26.390 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:31:26.390 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:26.390 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:31:26.390 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:31:26.726 INFO:teuthology.orchestra.run.gibba012.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.0’: No such file or directory 2021-02-10T11:31:26.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:31:26.727 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2021-02-10T11:31:26.727 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:26.727 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:31:26.727 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:31:27.062 INFO:teuthology.orchestra.run.gibba012.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: No such file or directory 2021-02-10T11:31:27.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:31:27.062 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2021-02-10T11:31:27.062 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:27.062 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:31:27.063 DEBUG:teuthology.orchestra.run.gibba020:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:31:27.398 INFO:teuthology.orchestra.run.gibba020.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:31:27.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:31:27.399 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:31:27.399 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:27.399 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:31:27.399 DEBUG:teuthology.orchestra.run.gibba020:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:31:27.732 INFO:teuthology.orchestra.run.gibba020.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:31:27.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:31:27.733 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:31:27.733 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json-pretty 2021-02-10T11:31:28.158 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.154 7f123c21e700 1 Processor -- start 2021-02-10T11:31:28.159 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.154 7f123c21e700 1 -- start start 2021-02-10T11:31:28.159 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.155 7f123c21e700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f12340e5070 0x7f12341b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:28.159 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.155 7f123c21e700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f12341b3e40 0x7f123407d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:28.159 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.155 7f123c21e700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f123407ed80 con 0x7f12340e5470 2021-02-10T11:31:28.159 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.155 7f123c21e700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1234080310 con 0x7f12341b3e40 2021-02-10T11:31:28.159 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.155 7f123c21e700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1234080530 con 0x7f12340e5070 2021-02-10T11:31:28.159 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.155 7f1239839700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f12341b3e40 0x7f123407d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:28.159 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.155 7f1239839700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f12341b3e40 0x7f123407d690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:48030/0 (socket says 172.21.2.112:48030) 2021-02-10T11:31:28.159 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.155 7f1239839700 1 -- 172.21.2.112:0/3034427798 learned_addr learned my addr 172.21.2.112:0/3034427798 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:31:28.159 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.155 7f123a03a700 1 --2- 172.21.2.112:0/3034427798 >> v2:172.21.2.120:3300/0 conn(0x7f12340e5070 0x7f12341b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:28.159 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.155 7f1239839700 1 -- 172.21.2.112:0/3034427798 >> v2:172.21.2.120:3300/0 conn(0x7f12340e5070 msgr2=0x7f12341b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:28.160 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.155 7f1239839700 1 --2- 172.21.2.112:0/3034427798 >> v2:172.21.2.120:3300/0 conn(0x7f12340e5070 0x7f12341b3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:28.160 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.155 7f1239839700 1 -- 172.21.2.112:0/3034427798 >> v1:172.21.2.120:6789/0 conn(0x7f12340e5470 legacy=0x7f123407dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:28.160 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.156 7f1239839700 1 -- 172.21.2.112:0/3034427798 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1234080750 con 0x7f12341b3e40 2021-02-10T11:31:28.160 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.156 7f123a03a700 1 --2- 172.21.2.112:0/3034427798 >> v2:172.21.2.120:3300/0 conn(0x7f12340e5070 0x7f12341b3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:31:28.160 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.156 7f1239839700 1 --2- 172.21.2.112:0/3034427798 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f12341b3e40 0x7f123407d690 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 rx=0x7f1230006700 tx=0x7f1230009b80).ready entity=mon.0 client_cookie=2d759d46d37b7d74 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:28.160 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.156 7f1239038700 1 -- 172.21.2.112:0/3034427798 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f123000c0c0 con 0x7f12341b3e40 2021-02-10T11:31:28.160 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.156 7f1239038700 1 -- 172.21.2.112:0/3034427798 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f123000c440 con 0x7f12341b3e40 2021-02-10T11:31:28.160 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.156 7f1239038700 1 -- 172.21.2.112:0/3034427798 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f123000c960 con 0x7f12341b3e40 2021-02-10T11:31:28.160 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.156 7f123c21e700 1 -- 172.21.2.112:0/3034427798 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f12341b3e40 msgr2=0x7f123407d690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:28.160 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.156 7f123c21e700 1 --2- 172.21.2.112:0/3034427798 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f12341b3e40 0x7f123407d690 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 rx=0x7f1230006700 tx=0x7f1230009b80).stop 2021-02-10T11:31:28.160 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.157 7f123c21e700 1 -- 172.21.2.112:0/3034427798 shutdown_connections 2021-02-10T11:31:28.160 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.157 7f123c21e700 1 --2- 172.21.2.112:0/3034427798 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f12341b3e40 0x7f123407d690 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:28.160 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.157 7f123c21e700 1 --2- 172.21.2.112:0/3034427798 >> v2:172.21.2.120:3300/0 conn(0x7f12340e5070 0x7f12341b3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:28.161 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.157 7f123c21e700 1 -- 172.21.2.112:0/3034427798 >> 172.21.2.112:0/3034427798 conn(0x7f12340ce690 msgr2=0x7f12340e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:28.161 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.157 7f123c21e700 1 -- 172.21.2.112:0/3034427798 shutdown_connections 2021-02-10T11:31:28.161 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.157 7f123c21e700 1 -- 172.21.2.112:0/3034427798 wait complete. 2021-02-10T11:31:28.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.172 7f123c21e700 1 Processor -- start 2021-02-10T11:31:28.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.172 7f123c21e700 1 -- start start 2021-02-10T11:31:28.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.172 7f123c21e700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1234192960 0x7f1234192d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:28.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.172 7f123c21e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f12340ad110 0x7f12340ad510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:28.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.172 7f123c21e700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f12340adb30 0x7f123408e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:28.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.172 7f123c21e700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12341a03e0 con 0x7f12340ad110 2021-02-10T11:31:28.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.172 7f123c21e700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1234193380 con 0x7f12340adb30 2021-02-10T11:31:28.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.172 7f123c21e700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f123408ea80 con 0x7f1234192960 2021-02-10T11:31:28.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.172 7f1239839700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f12340ad110 0x7f12340ad510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:28.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.172 7f1239839700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f12340ad110 0x7f12340ad510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:48036/0 (socket says 172.21.2.112:48036) 2021-02-10T11:31:28.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.172 7f1239839700 1 -- 172.21.2.112:0/1446752561 learned_addr learned my addr 172.21.2.112:0/1446752561 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:31:28.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.173 7f1239839700 1 -- 172.21.2.112:0/1446752561 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1234192960 msgr2=0x7f1234192d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:28.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.173 7f1239839700 1 --2- 172.21.2.112:0/1446752561 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1234192960 0x7f1234192d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:28.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.173 7f1239839700 1 -- 172.21.2.112:0/1446752561 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f12340adb30 msgr2=0x7f123408e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:28.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.173 7f1239839700 1 --2- 172.21.2.112:0/1446752561 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f12340adb30 0x7f123408e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:28.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.173 7f1239839700 1 -- 172.21.2.112:0/1446752561 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f123408eca0 con 0x7f12340ad110 2021-02-10T11:31:28.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.173 7f1239839700 1 --2- 172.21.2.112:0/1446752561 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f12340ad110 0x7f12340ad510 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 rx=0x7f123000b5e0 tx=0x7f123000b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:28.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.173 7f1226ffd700 1 -- 172.21.2.112:0/1446752561 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f123000bd70 con 0x7f12340ad110 2021-02-10T11:31:28.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.173 7f1226ffd700 1 -- 172.21.2.112:0/1446752561 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1230002030 con 0x7f12340ad110 2021-02-10T11:31:28.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.173 7f123c21e700 1 -- 172.21.2.112:0/1446752561 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f123408eca0 con 0x7f12340ad110 2021-02-10T11:31:28.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.173 7f123c21e700 1 -- 172.21.2.112:0/1446752561 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f123408f230 con 0x7f12340ad110 2021-02-10T11:31:28.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.173 7f1226ffd700 1 -- 172.21.2.112:0/1446752561 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1230002550 con 0x7f12340ad110 2021-02-10T11:31:28.164 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.174 7f1226ffd700 1 -- 172.21.2.112:0/1446752561 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1230000c90 con 0x7f12340ad110 2021-02-10T11:31:28.164 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.174 7f1226ffd700 1 --2- 172.21.2.112:0/1446752561 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f121c018730 0x7f121c01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:28.164 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.174 7f1226ffd700 1 -- 172.21.2.112:0/1446752561 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f12300327e0 con 0x7f12340ad110 2021-02-10T11:31:28.164 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.174 7f123a03a700 1 --2- 172.21.2.112:0/1446752561 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f121c018730 0x7f121c01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:28.164 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.175 7f123a03a700 1 --2- 172.21.2.112:0/1446752561 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f121c018730 0x7f121c01ac10 crc :-1 s=READY pgs=498 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:28.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.205 7f123c21e700 1 -- 172.21.2.112:0/1446752561 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1234107120 con 0x7f12340ad110 2021-02-10T11:31:28.194 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.207 7f1226ffd700 1 -- 172.21.2.112:0/1446752561 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f12300009b0 con 0x7f12340ad110 2021-02-10T11:31:28.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.333 7f123c21e700 1 -- 172.21.2.112:0/1446752561 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f123407ed80 con 0x7f12340ad110 2021-02-10T11:31:28.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.334 7f1226ffd700 1 -- 172.21.2.112:0/1446752561 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v29) v1 ==== 81+0+30360 (secure 0 0 0) 0x7f1230000c90 con 0x7f12340ad110 2021-02-10T11:31:28.321 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:31:28.322 INFO:teuthology.orchestra.run.gibba012.stdout:{ 2021-02-10T11:31:28.322 INFO:teuthology.orchestra.run.gibba012.stdout: "epoch": 29, 2021-02-10T11:31:28.322 INFO:teuthology.orchestra.run.gibba012.stdout: "fsid": "f14f004e-0699-4584-ad81-3754258493de", 2021-02-10T11:31:28.322 INFO:teuthology.orchestra.run.gibba012.stdout: "created": "2021-02-10 11:13:26.501424", 2021-02-10T11:31:28.322 INFO:teuthology.orchestra.run.gibba012.stdout: "modified": "2021-02-10 11:15:34.922144", 2021-02-10T11:31:28.322 INFO:teuthology.orchestra.run.gibba012.stdout: "last_up_change": "2021-02-10 11:13:52.780316", 2021-02-10T11:31:28.322 INFO:teuthology.orchestra.run.gibba012.stdout: "last_in_change": "2021-02-10 11:13:52.780316", 2021-02-10T11:31:28.322 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-02-10T11:31:28.322 INFO:teuthology.orchestra.run.gibba012.stdout: "flags_num": 5799936, 2021-02-10T11:31:28.322 INFO:teuthology.orchestra.run.gibba012.stdout: "flags_set": [ 2021-02-10T11:31:28.322 INFO:teuthology.orchestra.run.gibba012.stdout: "pglog_hardlimit", 2021-02-10T11:31:28.322 INFO:teuthology.orchestra.run.gibba012.stdout: "purged_snapdirs", 2021-02-10T11:31:28.323 INFO:teuthology.orchestra.run.gibba012.stdout: "recovery_deletes", 2021-02-10T11:31:28.323 INFO:teuthology.orchestra.run.gibba012.stdout: "sortbitwise" 2021-02-10T11:31:28.323 INFO:teuthology.orchestra.run.gibba012.stdout: ], 2021-02-10T11:31:28.323 INFO:teuthology.orchestra.run.gibba012.stdout: "crush_version": 4, 2021-02-10T11:31:28.323 INFO:teuthology.orchestra.run.gibba012.stdout: "full_ratio": 0.94999998807907104, 2021-02-10T11:31:28.323 INFO:teuthology.orchestra.run.gibba012.stdout: "backfillfull_ratio": 0.89999997615814209, 2021-02-10T11:31:28.323 INFO:teuthology.orchestra.run.gibba012.stdout: "nearfull_ratio": 0.85000002384185791, 2021-02-10T11:31:28.323 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_snapshot": "", 2021-02-10T11:31:28.323 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_max": 5, 2021-02-10T11:31:28.323 INFO:teuthology.orchestra.run.gibba012.stdout: "max_osd": 8, 2021-02-10T11:31:28.323 INFO:teuthology.orchestra.run.gibba012.stdout: "require_min_compat_client": "jewel", 2021-02-10T11:31:28.323 INFO:teuthology.orchestra.run.gibba012.stdout: "min_compat_client": "jewel", 2021-02-10T11:31:28.323 INFO:teuthology.orchestra.run.gibba012.stdout: "require_osd_release": "nautilus", 2021-02-10T11:31:28.324 INFO:teuthology.orchestra.run.gibba012.stdout: "pools": [ 2021-02-10T11:31:28.324 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:31:28.324 INFO:teuthology.orchestra.run.gibba012.stdout: "pool": 1, 2021-02-10T11:31:28.324 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_name": "rbd", 2021-02-10T11:31:28.324 INFO:teuthology.orchestra.run.gibba012.stdout: "create_time": "2021-02-10 11:13:57.508758", 2021-02-10T11:31:28.324 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:31:28.324 INFO:teuthology.orchestra.run.gibba012.stdout: "flags_names": "hashpspool", 2021-02-10T11:31:28.324 INFO:teuthology.orchestra.run.gibba012.stdout: "type": 1, 2021-02-10T11:31:28.324 INFO:teuthology.orchestra.run.gibba012.stdout: "size": 2, 2021-02-10T11:31:28.324 INFO:teuthology.orchestra.run.gibba012.stdout: "min_size": 1, 2021-02-10T11:31:28.324 INFO:teuthology.orchestra.run.gibba012.stdout: "crush_rule": 0, 2021-02-10T11:31:28.324 INFO:teuthology.orchestra.run.gibba012.stdout: "object_hash": 2, 2021-02-10T11:31:28.324 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:31:28.325 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num": 8, 2021-02-10T11:31:28.325 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num": 8, 2021-02-10T11:31:28.325 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num_target": 8, 2021-02-10T11:31:28.325 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_target": 8, 2021-02-10T11:31:28.325 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_pending": 8, 2021-02-10T11:31:28.325 INFO:teuthology.orchestra.run.gibba012.stdout: "last_pg_merge_meta": { 2021-02-10T11:31:28.325 INFO:teuthology.orchestra.run.gibba012.stdout: "source_pgid": "0.0", 2021-02-10T11:31:28.325 INFO:teuthology.orchestra.run.gibba012.stdout: "ready_epoch": 0, 2021-02-10T11:31:28.325 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_started": 0, 2021-02-10T11:31:28.325 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_clean": 0, 2021-02-10T11:31:28.325 INFO:teuthology.orchestra.run.gibba012.stdout: "source_version": "0'0", 2021-02-10T11:31:28.325 INFO:teuthology.orchestra.run.gibba012.stdout: "target_version": "0'0" 2021-02-10T11:31:28.325 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:31:28.326 INFO:teuthology.orchestra.run.gibba012.stdout: "last_change": "16", 2021-02-10T11:31:28.326 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend": "0", 2021-02-10T11:31:28.326 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:31:28.326 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:31:28.326 INFO:teuthology.orchestra.run.gibba012.stdout: "auid": 0, 2021-02-10T11:31:28.326 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:31:28.326 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_seq": 0, 2021-02-10T11:31:28.326 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_epoch": 0, 2021-02-10T11:31:28.326 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_snaps": [], 2021-02-10T11:31:28.326 INFO:teuthology.orchestra.run.gibba012.stdout: "removed_snaps": "[]", 2021-02-10T11:31:28.326 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_bytes": 0, 2021-02-10T11:31:28.326 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_objects": 0, 2021-02-10T11:31:28.326 INFO:teuthology.orchestra.run.gibba012.stdout: "tiers": [], 2021-02-10T11:31:28.327 INFO:teuthology.orchestra.run.gibba012.stdout: "tier_of": -1, 2021-02-10T11:31:28.327 INFO:teuthology.orchestra.run.gibba012.stdout: "read_tier": -1, 2021-02-10T11:31:28.327 INFO:teuthology.orchestra.run.gibba012.stdout: "write_tier": -1, 2021-02-10T11:31:28.327 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_mode": "none", 2021-02-10T11:31:28.327 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_bytes": 0, 2021-02-10T11:31:28.327 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_objects": 0, 2021-02-10T11:31:28.327 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:31:28.327 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:31:28.327 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:31:28.327 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_flush_age": 0, 2021-02-10T11:31:28.327 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_evict_age": 0, 2021-02-10T11:31:28.327 INFO:teuthology.orchestra.run.gibba012.stdout: "erasure_code_profile": "", 2021-02-10T11:31:28.328 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_params": { 2021-02-10T11:31:28.328 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "none" 2021-02-10T11:31:28.328 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:31:28.328 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_period": 0, 2021-02-10T11:31:28.328 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_count": 0, 2021-02-10T11:31:28.328 INFO:teuthology.orchestra.run.gibba012.stdout: "use_gmt_hitset": true, 2021-02-10T11:31:28.328 INFO:teuthology.orchestra.run.gibba012.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:31:28.328 INFO:teuthology.orchestra.run.gibba012.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:31:28.328 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:31:28.328 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:31:28.328 INFO:teuthology.orchestra.run.gibba012.stdout: "grade_table": [], 2021-02-10T11:31:28.328 INFO:teuthology.orchestra.run.gibba012.stdout: "stripe_width": 0, 2021-02-10T11:31:28.328 INFO:teuthology.orchestra.run.gibba012.stdout: "expected_num_objects": 0, 2021-02-10T11:31:28.329 INFO:teuthology.orchestra.run.gibba012.stdout: "fast_read": false, 2021-02-10T11:31:28.329 INFO:teuthology.orchestra.run.gibba012.stdout: "options": {}, 2021-02-10T11:31:28.329 INFO:teuthology.orchestra.run.gibba012.stdout: "application_metadata": { 2021-02-10T11:31:28.329 INFO:teuthology.orchestra.run.gibba012.stdout: "rbd": {} 2021-02-10T11:31:28.329 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:31:28.329 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:31:28.329 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:31:28.329 INFO:teuthology.orchestra.run.gibba012.stdout: "pool": 4, 2021-02-10T11:31:28.329 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_name": "cephfs_metadata", 2021-02-10T11:31:28.329 INFO:teuthology.orchestra.run.gibba012.stdout: "create_time": "2021-02-10 11:15:31.539720", 2021-02-10T11:31:28.329 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:31:28.329 INFO:teuthology.orchestra.run.gibba012.stdout: "flags_names": "hashpspool", 2021-02-10T11:31:28.329 INFO:teuthology.orchestra.run.gibba012.stdout: "type": 1, 2021-02-10T11:31:28.330 INFO:teuthology.orchestra.run.gibba012.stdout: "size": 2, 2021-02-10T11:31:28.330 INFO:teuthology.orchestra.run.gibba012.stdout: "min_size": 1, 2021-02-10T11:31:28.330 INFO:teuthology.orchestra.run.gibba012.stdout: "crush_rule": 0, 2021-02-10T11:31:28.330 INFO:teuthology.orchestra.run.gibba012.stdout: "object_hash": 2, 2021-02-10T11:31:28.330 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:31:28.330 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num": 8, 2021-02-10T11:31:28.330 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num": 8, 2021-02-10T11:31:28.330 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num_target": 8, 2021-02-10T11:31:28.330 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_target": 8, 2021-02-10T11:31:28.330 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_pending": 8, 2021-02-10T11:31:28.330 INFO:teuthology.orchestra.run.gibba012.stdout: "last_pg_merge_meta": { 2021-02-10T11:31:28.330 INFO:teuthology.orchestra.run.gibba012.stdout: "source_pgid": "0.0", 2021-02-10T11:31:28.330 INFO:teuthology.orchestra.run.gibba012.stdout: "ready_epoch": 0, 2021-02-10T11:31:28.331 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_started": 0, 2021-02-10T11:31:28.331 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_clean": 0, 2021-02-10T11:31:28.331 INFO:teuthology.orchestra.run.gibba012.stdout: "source_version": "0'0", 2021-02-10T11:31:28.331 INFO:teuthology.orchestra.run.gibba012.stdout: "target_version": "0'0" 2021-02-10T11:31:28.331 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:31:28.331 INFO:teuthology.orchestra.run.gibba012.stdout: "last_change": "28", 2021-02-10T11:31:28.331 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend": "0", 2021-02-10T11:31:28.331 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:31:28.331 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:31:28.331 INFO:teuthology.orchestra.run.gibba012.stdout: "auid": 0, 2021-02-10T11:31:28.331 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:31:28.331 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_seq": 0, 2021-02-10T11:31:28.331 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_epoch": 0, 2021-02-10T11:31:28.332 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_snaps": [], 2021-02-10T11:31:28.332 INFO:teuthology.orchestra.run.gibba012.stdout: "removed_snaps": "[]", 2021-02-10T11:31:28.332 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_bytes": 0, 2021-02-10T11:31:28.332 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_objects": 0, 2021-02-10T11:31:28.332 INFO:teuthology.orchestra.run.gibba012.stdout: "tiers": [], 2021-02-10T11:31:28.332 INFO:teuthology.orchestra.run.gibba012.stdout: "tier_of": -1, 2021-02-10T11:31:28.332 INFO:teuthology.orchestra.run.gibba012.stdout: "read_tier": -1, 2021-02-10T11:31:28.332 INFO:teuthology.orchestra.run.gibba012.stdout: "write_tier": -1, 2021-02-10T11:31:28.332 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_mode": "none", 2021-02-10T11:31:28.332 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_bytes": 0, 2021-02-10T11:31:28.332 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_objects": 0, 2021-02-10T11:31:28.332 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:31:28.332 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:31:28.333 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:31:28.333 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_flush_age": 0, 2021-02-10T11:31:28.333 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_evict_age": 0, 2021-02-10T11:31:28.333 INFO:teuthology.orchestra.run.gibba012.stdout: "erasure_code_profile": "", 2021-02-10T11:31:28.333 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_params": { 2021-02-10T11:31:28.333 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "none" 2021-02-10T11:31:28.333 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:31:28.333 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_period": 0, 2021-02-10T11:31:28.333 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_count": 0, 2021-02-10T11:31:28.333 INFO:teuthology.orchestra.run.gibba012.stdout: "use_gmt_hitset": true, 2021-02-10T11:31:28.333 INFO:teuthology.orchestra.run.gibba012.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:31:28.333 INFO:teuthology.orchestra.run.gibba012.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:31:28.333 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:31:28.334 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:31:28.334 INFO:teuthology.orchestra.run.gibba012.stdout: "grade_table": [], 2021-02-10T11:31:28.334 INFO:teuthology.orchestra.run.gibba012.stdout: "stripe_width": 0, 2021-02-10T11:31:28.334 INFO:teuthology.orchestra.run.gibba012.stdout: "expected_num_objects": 0, 2021-02-10T11:31:28.334 INFO:teuthology.orchestra.run.gibba012.stdout: "fast_read": false, 2021-02-10T11:31:28.334 INFO:teuthology.orchestra.run.gibba012.stdout: "options": { 2021-02-10T11:31:28.334 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_autoscale_bias": 4, 2021-02-10T11:31:28.334 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_min": 16, 2021-02-10T11:31:28.334 INFO:teuthology.orchestra.run.gibba012.stdout: "recovery_priority": 5 2021-02-10T11:31:28.334 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:31:28.334 INFO:teuthology.orchestra.run.gibba012.stdout: "application_metadata": { 2021-02-10T11:31:28.334 INFO:teuthology.orchestra.run.gibba012.stdout: "cephfs": { 2021-02-10T11:31:28.334 INFO:teuthology.orchestra.run.gibba012.stdout: "metadata": "cephfs" 2021-02-10T11:31:28.335 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:31:28.335 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:31:28.335 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:31:28.335 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:31:28.335 INFO:teuthology.orchestra.run.gibba012.stdout: "pool": 5, 2021-02-10T11:31:28.335 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_name": "cephfs_data", 2021-02-10T11:31:28.335 INFO:teuthology.orchestra.run.gibba012.stdout: "create_time": "2021-02-10 11:15:32.965643", 2021-02-10T11:31:28.335 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:31:28.335 INFO:teuthology.orchestra.run.gibba012.stdout: "flags_names": "hashpspool", 2021-02-10T11:31:28.335 INFO:teuthology.orchestra.run.gibba012.stdout: "type": 1, 2021-02-10T11:31:28.335 INFO:teuthology.orchestra.run.gibba012.stdout: "size": 2, 2021-02-10T11:31:28.335 INFO:teuthology.orchestra.run.gibba012.stdout: "min_size": 1, 2021-02-10T11:31:28.335 INFO:teuthology.orchestra.run.gibba012.stdout: "crush_rule": 0, 2021-02-10T11:31:28.335 INFO:teuthology.orchestra.run.gibba012.stdout: "object_hash": 2, 2021-02-10T11:31:28.336 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:31:28.336 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num": 8, 2021-02-10T11:31:28.336 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num": 8, 2021-02-10T11:31:28.336 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num_target": 8, 2021-02-10T11:31:28.336 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_target": 8, 2021-02-10T11:31:28.336 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_pending": 8, 2021-02-10T11:31:28.336 INFO:teuthology.orchestra.run.gibba012.stdout: "last_pg_merge_meta": { 2021-02-10T11:31:28.336 INFO:teuthology.orchestra.run.gibba012.stdout: "source_pgid": "0.0", 2021-02-10T11:31:28.336 INFO:teuthology.orchestra.run.gibba012.stdout: "ready_epoch": 0, 2021-02-10T11:31:28.336 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_started": 0, 2021-02-10T11:31:28.336 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_clean": 0, 2021-02-10T11:31:28.336 INFO:teuthology.orchestra.run.gibba012.stdout: "source_version": "0'0", 2021-02-10T11:31:28.336 INFO:teuthology.orchestra.run.gibba012.stdout: "target_version": "0'0" 2021-02-10T11:31:28.337 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:31:28.337 INFO:teuthology.orchestra.run.gibba012.stdout: "last_change": "29", 2021-02-10T11:31:28.337 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend": "0", 2021-02-10T11:31:28.337 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:31:28.337 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:31:28.337 INFO:teuthology.orchestra.run.gibba012.stdout: "auid": 0, 2021-02-10T11:31:28.337 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:31:28.337 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_seq": 0, 2021-02-10T11:31:28.337 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_epoch": 0, 2021-02-10T11:31:28.337 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_snaps": [], 2021-02-10T11:31:28.337 INFO:teuthology.orchestra.run.gibba012.stdout: "removed_snaps": "[]", 2021-02-10T11:31:28.337 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_bytes": 0, 2021-02-10T11:31:28.337 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_objects": 0, 2021-02-10T11:31:28.337 INFO:teuthology.orchestra.run.gibba012.stdout: "tiers": [], 2021-02-10T11:31:28.338 INFO:teuthology.orchestra.run.gibba012.stdout: "tier_of": -1, 2021-02-10T11:31:28.338 INFO:teuthology.orchestra.run.gibba012.stdout: "read_tier": -1, 2021-02-10T11:31:28.338 INFO:teuthology.orchestra.run.gibba012.stdout: "write_tier": -1, 2021-02-10T11:31:28.338 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_mode": "none", 2021-02-10T11:31:28.338 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_bytes": 0, 2021-02-10T11:31:28.338 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_objects": 0, 2021-02-10T11:31:28.338 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:31:28.338 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:31:28.338 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:31:28.338 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_flush_age": 0, 2021-02-10T11:31:28.338 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_evict_age": 0, 2021-02-10T11:31:28.338 INFO:teuthology.orchestra.run.gibba012.stdout: "erasure_code_profile": "", 2021-02-10T11:31:28.338 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_params": { 2021-02-10T11:31:28.339 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "none" 2021-02-10T11:31:28.339 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:31:28.339 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_period": 0, 2021-02-10T11:31:28.339 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_count": 0, 2021-02-10T11:31:28.339 INFO:teuthology.orchestra.run.gibba012.stdout: "use_gmt_hitset": true, 2021-02-10T11:31:28.339 INFO:teuthology.orchestra.run.gibba012.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:31:28.339 INFO:teuthology.orchestra.run.gibba012.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:31:28.339 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:31:28.339 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:31:28.339 INFO:teuthology.orchestra.run.gibba012.stdout: "grade_table": [], 2021-02-10T11:31:28.339 INFO:teuthology.orchestra.run.gibba012.stdout: "stripe_width": 0, 2021-02-10T11:31:28.339 INFO:teuthology.orchestra.run.gibba012.stdout: "expected_num_objects": 0, 2021-02-10T11:31:28.339 INFO:teuthology.orchestra.run.gibba012.stdout: "fast_read": false, 2021-02-10T11:31:28.340 INFO:teuthology.orchestra.run.gibba012.stdout: "options": {}, 2021-02-10T11:31:28.340 INFO:teuthology.orchestra.run.gibba012.stdout: "application_metadata": { 2021-02-10T11:31:28.340 INFO:teuthology.orchestra.run.gibba012.stdout: "cephfs": { 2021-02-10T11:31:28.340 INFO:teuthology.orchestra.run.gibba012.stdout: "data": "cephfs" 2021-02-10T11:31:28.340 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:31:28.340 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:31:28.340 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:31:28.340 INFO:teuthology.orchestra.run.gibba012.stdout: ], 2021-02-10T11:31:28.340 INFO:teuthology.orchestra.run.gibba012.stdout: "osds": [ 2021-02-10T11:31:28.340 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:31:28.340 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 0, 2021-02-10T11:31:28.340 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "95444b8f-6370-4a1d-ae5f-df62651b77da", 2021-02-10T11:31:28.340 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:31:28.341 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:31:28.341 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:31:28.341 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:31:28.341 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:31:28.341 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:31:28.341 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:31:28.341 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 27, 2021-02-10T11:31:28.341 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:31:28.341 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:31:28.341 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:31:28.341 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:31:28.341 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:31:28.341 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:31:28.342 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6817", 2021-02-10T11:31:28.342 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:31:28.342 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:31:28.342 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:31:28.342 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:31:28.342 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6818", 2021-02-10T11:31:28.342 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:31:28.342 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:31:28.342 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:31:28.342 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:31:28.342 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:31:28.342 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:31:28.343 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:31:28.343 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:31:28.343 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6820", 2021-02-10T11:31:28.343 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:31:28.343 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:31:28.343 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:31:28.343 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:31:28.343 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6822", 2021-02-10T11:31:28.343 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:31:28.343 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:31:28.343 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:31:28.343 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:31:28.343 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:31:28.343 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:31:28.344 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:31:28.344 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:31:28.344 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6828", 2021-02-10T11:31:28.344 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:31:28.344 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:31:28.344 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:31:28.344 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:31:28.344 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6830", 2021-02-10T11:31:28.344 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:31:28.344 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:31:28.344 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:31:28.344 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:31:28.344 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:31:28.345 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:31:28.345 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:31:28.345 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:31:28.345 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6824", 2021-02-10T11:31:28.345 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:31:28.345 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:31:28.345 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:31:28.345 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:31:28.345 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6826", 2021-02-10T11:31:28.345 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:31:28.345 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:31:28.345 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:31:28.345 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:31:28.346 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.112:6818/36030", 2021-02-10T11:31:28.346 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.112:6822/36030", 2021-02-10T11:31:28.346 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.112:6830/36030", 2021-02-10T11:31:28.346 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.112:6826/36030", 2021-02-10T11:31:28.346 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:31:28.346 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:31:28.346 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:31:28.346 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:31:28.346 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:31:28.346 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:31:28.346 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 1, 2021-02-10T11:31:28.346 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "a23e217e-0d8e-486e-be07-712adfabd262", 2021-02-10T11:31:28.346 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:31:28.347 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:31:28.347 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:31:28.347 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:31:28.347 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:31:28.347 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:31:28.347 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:31:28.347 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 26, 2021-02-10T11:31:28.347 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:31:28.347 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:31:28.347 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:31:28.347 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:31:28.347 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:31:28.347 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:31:28.348 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6800", 2021-02-10T11:31:28.348 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:31:28.348 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:31:28.348 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:31:28.348 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:31:28.348 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6802", 2021-02-10T11:31:28.348 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:31:28.348 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:31:28.348 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:31:28.348 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:31:28.348 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:31:28.348 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:31:28.348 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:31:28.349 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:31:28.349 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6804", 2021-02-10T11:31:28.349 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:31:28.349 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:31:28.349 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:31:28.349 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:31:28.349 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6806", 2021-02-10T11:31:28.349 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:31:28.349 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:31:28.349 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:31:28.349 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:31:28.349 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:31:28.349 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:31:28.350 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:31:28.350 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:31:28.350 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6812", 2021-02-10T11:31:28.350 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:31:28.350 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:31:28.350 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:31:28.350 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:31:28.350 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6814", 2021-02-10T11:31:28.350 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:31:28.350 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:31:28.350 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:31:28.350 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:31:28.350 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:31:28.351 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:31:28.351 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:31:28.351 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:31:28.351 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6808", 2021-02-10T11:31:28.351 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:31:28.351 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:31:28.351 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:31:28.351 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:31:28.351 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6810", 2021-02-10T11:31:28.351 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:31:28.351 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:31:28.351 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:31:28.351 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:31:28.351 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.112:6802/36029", 2021-02-10T11:31:28.352 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.112:6806/36029", 2021-02-10T11:31:28.352 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.112:6814/36029", 2021-02-10T11:31:28.352 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.112:6810/36029", 2021-02-10T11:31:28.352 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:31:28.352 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:31:28.352 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:31:28.352 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:31:28.352 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:31:28.352 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:31:28.352 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 2, 2021-02-10T11:31:28.352 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "026afa8a-c0f1-4b35-9a76-53c00fe0c2fc", 2021-02-10T11:31:28.352 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:31:28.352 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:31:28.353 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:31:28.353 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:31:28.353 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:31:28.353 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:31:28.353 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:31:28.353 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 27, 2021-02-10T11:31:28.353 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:31:28.353 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:31:28.353 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:31:28.353 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:31:28.353 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:31:28.353 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:31:28.353 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6801", 2021-02-10T11:31:28.354 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:31:28.354 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:31:28.354 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:31:28.354 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:31:28.354 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6803", 2021-02-10T11:31:28.354 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:31:28.354 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:31:28.354 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:31:28.354 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:31:28.354 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:31:28.354 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:31:28.354 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:31:28.354 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:31:28.355 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6805", 2021-02-10T11:31:28.355 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:31:28.355 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:31:28.355 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:31:28.355 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:31:28.355 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6807", 2021-02-10T11:31:28.355 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:31:28.355 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:31:28.355 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:31:28.355 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:31:28.355 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:31:28.355 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:31:28.355 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:31:28.356 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:31:28.356 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6813", 2021-02-10T11:31:28.356 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:31:28.356 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:31:28.356 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:31:28.356 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:31:28.356 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6815", 2021-02-10T11:31:28.356 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:31:28.356 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:31:28.356 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:31:28.356 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:31:28.356 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:31:28.356 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:31:28.357 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:31:28.357 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:31:28.357 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6809", 2021-02-10T11:31:28.357 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:31:28.357 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:31:28.357 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:31:28.357 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:31:28.357 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6811", 2021-02-10T11:31:28.357 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:31:28.357 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:31:28.357 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:31:28.357 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:31:28.357 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.112:6803/36031", 2021-02-10T11:31:28.357 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.112:6807/36031", 2021-02-10T11:31:28.358 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.112:6815/36031", 2021-02-10T11:31:28.358 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.112:6811/36031", 2021-02-10T11:31:28.358 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:31:28.358 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:31:28.358 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:31:28.358 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:31:28.358 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:31:28.358 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:31:28.358 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 3, 2021-02-10T11:31:28.358 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "58795682-e00d-4173-bb84-f7075ca89a11", 2021-02-10T11:31:28.358 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:31:28.358 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:31:28.359 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:31:28.359 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:31:28.359 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:31:28.359 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:31:28.359 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:31:28.359 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 27, 2021-02-10T11:31:28.359 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:31:28.359 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:31:28.359 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:31:28.359 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:31:28.359 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:31:28.359 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:31:28.359 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6816", 2021-02-10T11:31:28.360 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:31:28.360 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:31:28.360 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:31:28.360 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:31:28.360 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6819", 2021-02-10T11:31:28.360 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:31:28.361 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:31:28.361 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:31:28.361 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:31:28.362 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:31:28.362 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:31:28.362 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:31:28.362 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:31:28.362 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6821", 2021-02-10T11:31:28.362 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:31:28.362 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:31:28.362 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:31:28.362 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:31:28.362 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6823", 2021-02-10T11:31:28.362 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:31:28.362 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:31:28.362 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:31:28.363 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:31:28.363 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:31:28.363 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:31:28.363 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:31:28.363 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:31:28.363 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6829", 2021-02-10T11:31:28.363 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:31:28.363 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:31:28.363 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:31:28.363 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:31:28.363 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6831", 2021-02-10T11:31:28.363 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:31:28.363 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:31:28.364 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:31:28.364 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:31:28.364 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:31:28.364 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:31:28.364 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:31:28.364 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:31:28.364 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6825", 2021-02-10T11:31:28.364 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:31:28.364 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:31:28.364 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:31:28.364 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:31:28.364 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6827", 2021-02-10T11:31:28.364 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:31:28.365 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:31:28.365 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:31:28.365 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:31:28.365 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.112:6819/36028", 2021-02-10T11:31:28.365 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.112:6823/36028", 2021-02-10T11:31:28.365 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.112:6831/36028", 2021-02-10T11:31:28.365 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.112:6827/36028", 2021-02-10T11:31:28.365 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:31:28.365 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:31:28.365 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:31:28.365 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:31:28.365 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:31:28.365 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:31:28.366 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 4, 2021-02-10T11:31:28.366 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "6763f8d6-53dc-4e45-bdf6-a6eabcf81e39", 2021-02-10T11:31:28.366 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:31:28.366 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:31:28.366 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:31:28.366 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:31:28.366 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:31:28.366 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:31:28.366 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:31:28.366 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 27, 2021-02-10T11:31:28.366 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:31:28.366 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:31:28.367 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:31:28.367 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:31:28.367 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:31:28.367 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:31:28.367 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6810", 2021-02-10T11:31:28.367 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:31:28.367 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:31:28.367 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:31:28.367 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:31:28.367 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6813", 2021-02-10T11:31:28.367 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:31:28.367 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:31:28.367 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:31:28.368 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:31:28.368 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:31:28.368 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:31:28.368 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:31:28.368 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:31:28.368 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6816", 2021-02-10T11:31:28.368 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:31:28.368 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:31:28.368 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:31:28.368 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:31:28.368 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6819", 2021-02-10T11:31:28.368 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:31:28.368 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:31:28.369 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:31:28.369 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:31:28.369 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:31:28.369 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:31:28.369 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:31:28.369 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:31:28.369 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6825", 2021-02-10T11:31:28.369 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:31:28.369 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:31:28.369 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:31:28.369 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:31:28.369 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6827", 2021-02-10T11:31:28.370 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:31:28.370 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:31:28.370 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:31:28.370 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:31:28.370 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:31:28.370 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:31:28.370 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:31:28.370 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:31:28.370 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6821", 2021-02-10T11:31:28.370 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:31:28.370 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:31:28.370 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:31:28.370 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:31:28.371 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6823", 2021-02-10T11:31:28.371 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:31:28.371 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:31:28.371 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:31:28.371 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:31:28.371 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.120:6813/36388", 2021-02-10T11:31:28.371 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.120:6819/36388", 2021-02-10T11:31:28.371 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.120:6827/36388", 2021-02-10T11:31:28.371 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.120:6823/36388", 2021-02-10T11:31:28.371 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:31:28.371 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:31:28.371 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:31:28.371 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:31:28.372 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:31:28.372 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:31:28.372 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 5, 2021-02-10T11:31:28.372 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "5d0e4dd6-0cb0-4bf8-8314-626acaa1b544", 2021-02-10T11:31:28.372 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:31:28.372 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:31:28.372 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:31:28.372 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:31:28.372 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:31:28.372 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:31:28.372 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:31:28.372 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 27, 2021-02-10T11:31:28.372 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:31:28.373 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:31:28.373 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:31:28.373 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:31:28.373 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:31:28.373 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:31:28.373 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6804", 2021-02-10T11:31:28.373 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:31:28.373 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:31:28.373 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:31:28.373 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:31:28.373 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6806", 2021-02-10T11:31:28.373 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:31:28.373 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:31:28.374 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:31:28.374 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:31:28.374 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:31:28.374 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:31:28.374 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:31:28.374 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:31:28.374 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6809", 2021-02-10T11:31:28.374 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:31:28.374 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:31:28.374 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:31:28.374 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:31:28.374 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6812", 2021-02-10T11:31:28.374 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:31:28.375 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:31:28.375 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:31:28.375 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:31:28.375 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:31:28.375 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:31:28.375 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:31:28.375 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:31:28.375 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6820", 2021-02-10T11:31:28.375 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:31:28.375 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:31:28.375 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:31:28.375 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:31:28.375 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6822", 2021-02-10T11:31:28.376 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:31:28.376 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:31:28.376 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:31:28.376 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:31:28.376 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:31:28.376 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:31:28.376 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:31:28.376 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:31:28.376 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6815", 2021-02-10T11:31:28.376 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:31:28.376 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:31:28.376 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:31:28.376 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:31:28.376 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6818", 2021-02-10T11:31:28.377 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:31:28.377 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:31:28.377 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:31:28.377 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:31:28.377 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.120:6806/36387", 2021-02-10T11:31:28.377 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.120:6812/36387", 2021-02-10T11:31:28.377 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.120:6822/36387", 2021-02-10T11:31:28.377 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.120:6818/36387", 2021-02-10T11:31:28.377 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:31:28.377 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:31:28.377 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:31:28.377 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:31:28.377 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:31:28.378 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:31:28.378 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 6, 2021-02-10T11:31:28.378 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "b010e187-4541-4b00-80dd-f948f6ab8f15", 2021-02-10T11:31:28.378 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:31:28.378 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:31:28.378 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:31:28.378 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:31:28.378 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:31:28.378 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:31:28.378 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:31:28.378 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 27, 2021-02-10T11:31:28.378 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:31:28.378 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:31:28.379 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:31:28.379 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:31:28.379 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:31:28.379 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:31:28.379 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6824", 2021-02-10T11:31:28.379 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:31:28.379 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:31:28.379 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:31:28.379 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:31:28.379 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6826", 2021-02-10T11:31:28.379 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:31:28.379 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:31:28.379 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:31:28.380 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:31:28.380 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:31:28.380 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:31:28.380 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:31:28.380 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:31:28.380 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6828", 2021-02-10T11:31:28.380 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:31:28.380 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:31:28.380 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:31:28.380 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:31:28.380 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6829", 2021-02-10T11:31:28.380 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:31:28.380 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:31:28.380 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:31:28.381 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:31:28.381 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:31:28.381 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:31:28.381 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:31:28.381 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:31:28.381 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6832", 2021-02-10T11:31:28.381 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:31:28.381 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:31:28.381 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:31:28.381 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:31:28.381 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6833", 2021-02-10T11:31:28.381 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:31:28.381 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:31:28.382 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:31:28.382 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:31:28.382 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:31:28.382 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:31:28.382 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:31:28.382 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:31:28.382 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6830", 2021-02-10T11:31:28.382 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:31:28.382 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:31:28.382 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:31:28.382 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:31:28.382 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6831", 2021-02-10T11:31:28.382 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:31:28.383 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:31:28.383 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:31:28.383 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:31:28.383 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.120:6826/36385", 2021-02-10T11:31:28.383 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.120:6829/36385", 2021-02-10T11:31:28.383 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.120:6833/36385", 2021-02-10T11:31:28.383 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.120:6831/36385", 2021-02-10T11:31:28.383 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:31:28.383 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:31:28.383 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:31:28.383 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:31:28.383 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:31:28.383 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:31:28.384 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 7, 2021-02-10T11:31:28.384 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "671e2772-33b7-497e-8e44-2178bd850bcf", 2021-02-10T11:31:28.384 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:31:28.384 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:31:28.384 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:31:28.384 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:31:28.384 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:31:28.384 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:31:28.384 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:31:28.384 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 27, 2021-02-10T11:31:28.384 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:31:28.384 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:31:28.384 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:31:28.385 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:31:28.385 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:31:28.385 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:31:28.385 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6802", 2021-02-10T11:31:28.385 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:31:28.385 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:31:28.385 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:31:28.385 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:31:28.385 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6803", 2021-02-10T11:31:28.385 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:31:28.385 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:31:28.385 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:31:28.385 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:31:28.386 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:31:28.386 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:31:28.386 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:31:28.386 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:31:28.386 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6805", 2021-02-10T11:31:28.386 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:31:28.386 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:31:28.386 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:31:28.386 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:31:28.386 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6807", 2021-02-10T11:31:28.386 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:31:28.386 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:31:28.386 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:31:28.387 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:31:28.387 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:31:28.387 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:31:28.387 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:31:28.387 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:31:28.387 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6814", 2021-02-10T11:31:28.387 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:31:28.387 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:31:28.387 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:31:28.387 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:31:28.387 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6817", 2021-02-10T11:31:28.387 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:31:28.387 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:31:28.387 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:31:28.388 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:31:28.388 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:31:28.388 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:31:28.388 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:31:28.388 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:31:28.388 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6808", 2021-02-10T11:31:28.388 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:31:28.388 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:31:28.388 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:31:28.388 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:31:28.388 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6811", 2021-02-10T11:31:28.388 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:31:28.388 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:31:28.389 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:31:28.389 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:31:28.389 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.120:6803/36386", 2021-02-10T11:31:28.389 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.120:6807/36386", 2021-02-10T11:31:28.389 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.120:6817/36386", 2021-02-10T11:31:28.389 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.120:6811/36386", 2021-02-10T11:31:28.389 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:31:28.389 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:31:28.389 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:31:28.389 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:31:28.389 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:31:28.389 INFO:teuthology.orchestra.run.gibba012.stdout: ], 2021-02-10T11:31:28.389 INFO:teuthology.orchestra.run.gibba012.stdout: "osd_xinfo": [ 2021-02-10T11:31:28.390 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:31:28.390 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 0, 2021-02-10T11:31:28.390 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:31:28.390 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:31:28.390 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:31:28.390 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:31:28.390 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:31:28.390 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:31:28.390 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:31:28.390 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 1, 2021-02-10T11:31:28.390 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:31:28.390 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:31:28.390 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:31:28.391 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:31:28.391 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:31:28.391 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:31:28.391 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:31:28.391 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 2, 2021-02-10T11:31:28.391 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:31:28.391 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:31:28.391 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:31:28.391 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:31:28.391 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:31:28.391 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:31:28.391 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:31:28.391 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 3, 2021-02-10T11:31:28.391 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:31:28.392 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:31:28.392 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:31:28.392 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:31:28.392 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:31:28.392 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:31:28.392 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:31:28.392 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 4, 2021-02-10T11:31:28.392 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:31:28.392 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:31:28.392 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:31:28.392 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:31:28.392 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:31:28.392 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:31:28.393 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:31:28.393 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 5, 2021-02-10T11:31:28.393 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:31:28.393 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:31:28.393 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:31:28.393 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:31:28.393 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:31:28.393 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:31:28.393 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:31:28.393 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 6, 2021-02-10T11:31:28.393 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:31:28.393 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:31:28.393 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:31:28.393 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:31:28.394 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:31:28.394 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:31:28.394 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:31:28.394 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 7, 2021-02-10T11:31:28.394 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:31:28.394 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:31:28.394 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:31:28.394 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:31:28.394 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:31:28.394 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:31:28.394 INFO:teuthology.orchestra.run.gibba012.stdout: ], 2021-02-10T11:31:28.394 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_upmap": [], 2021-02-10T11:31:28.394 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_upmap_items": [], 2021-02-10T11:31:28.395 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_temp": [], 2021-02-10T11:31:28.395 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_temp": [], 2021-02-10T11:31:28.395 INFO:teuthology.orchestra.run.gibba012.stdout: "blacklist": {}, 2021-02-10T11:31:28.395 INFO:teuthology.orchestra.run.gibba012.stdout: "erasure_code_profiles": { 2021-02-10T11:31:28.395 INFO:teuthology.orchestra.run.gibba012.stdout: "default": { 2021-02-10T11:31:28.395 INFO:teuthology.orchestra.run.gibba012.stdout: "crush-failure-domain": "osd", 2021-02-10T11:31:28.395 INFO:teuthology.orchestra.run.gibba012.stdout: "k": "2", 2021-02-10T11:31:28.395 INFO:teuthology.orchestra.run.gibba012.stdout: "m": "1", 2021-02-10T11:31:28.395 INFO:teuthology.orchestra.run.gibba012.stdout: "plugin": "jerasure", 2021-02-10T11:31:28.395 INFO:teuthology.orchestra.run.gibba012.stdout: "ruleset-failure-domain": "osd", 2021-02-10T11:31:28.395 INFO:teuthology.orchestra.run.gibba012.stdout: "technique": "reed_sol_van" 2021-02-10T11:31:28.395 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:31:28.395 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:31:28.396 INFO:teuthology.orchestra.run.gibba012.stdout: "removed_snaps_queue": [], 2021-02-10T11:31:28.396 INFO:teuthology.orchestra.run.gibba012.stdout: "new_removed_snaps": [], 2021-02-10T11:31:28.396 INFO:teuthology.orchestra.run.gibba012.stdout: "new_purged_snaps": [], 2021-02-10T11:31:28.396 INFO:teuthology.orchestra.run.gibba012.stdout: "crush_node_flags": {}, 2021-02-10T11:31:28.396 INFO:teuthology.orchestra.run.gibba012.stdout: "device_class_flags": {} 2021-02-10T11:31:28.396 INFO:teuthology.orchestra.run.gibba012.stdout:} 2021-02-10T11:31:28.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.335 7f123c21e700 1 -- 172.21.2.112:0/1446752561 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f121c018730 msgr2=0x7f121c01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:28.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.335 7f123c21e700 1 --2- 172.21.2.112:0/1446752561 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f121c018730 0x7f121c01ac10 crc :-1 s=READY pgs=498 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:28.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.335 7f123c21e700 1 -- 172.21.2.112:0/1446752561 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f12340ad110 msgr2=0x7f12340ad510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:28.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.335 7f123c21e700 1 --2- 172.21.2.112:0/1446752561 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f12340ad110 0x7f12340ad510 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 rx=0x7f123000b5e0 tx=0x7f123000b980).stop 2021-02-10T11:31:28.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.335 7f123c21e700 1 -- 172.21.2.112:0/1446752561 shutdown_connections 2021-02-10T11:31:28.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.335 7f123c21e700 1 --2- 172.21.2.112:0/1446752561 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f121c018730 0x7f121c01ac10 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:28.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.335 7f123c21e700 1 --2- 172.21.2.112:0/1446752561 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f12340adb30 0x7f123408e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:28.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.335 7f123c21e700 1 --2- 172.21.2.112:0/1446752561 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f12340ad110 0x7f12340ad510 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:28.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.335 7f123c21e700 1 --2- 172.21.2.112:0/1446752561 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1234192960 0x7f1234192d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:28.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.335 7f123c21e700 1 -- 172.21.2.112:0/1446752561 >> 172.21.2.112:0/1446752561 conn(0x7f12340891f0 msgr2=0x7f12341a0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:28.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.335 7f123c21e700 1 -- 172.21.2.112:0/1446752561 shutdown_connections 2021-02-10T11:31:28.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.336 7f123c21e700 1 -- 172.21.2.112:0/1446752561 wait complete. 2021-02-10T11:31:28.397 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:31:28.824 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.820 7f48948c6700 1 Processor -- start 2021-02-10T11:31:28.824 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.820 7f48948c6700 1 -- start start 2021-02-10T11:31:28.824 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.821 7f48948c6700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f48900e5070 0x7f48901b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:28.825 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.821 7f48948c6700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f48900e5470 0x7f4890080310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:28.825 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.821 7f48948c6700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f489007ed80 con 0x7f48901b3e40 2021-02-10T11:31:28.825 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.821 7f48948c6700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48900d1790 con 0x7f48900e5070 2021-02-10T11:31:28.825 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.821 7f48948c6700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48900d19b0 con 0x7f48900e5470 2021-02-10T11:31:28.825 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.821 7f488ee1d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f48900e5470 0x7f4890080310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:28.825 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.821 7f488ee1d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f48900e5470 0x7f4890080310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3301/0 says I am v2:172.21.2.112:47724/0 (socket says 172.21.2.112:47724) 2021-02-10T11:31:28.825 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.821 7f488ee1d700 1 -- 172.21.2.112:0/3272774822 learned_addr learned my addr 172.21.2.112:0/3272774822 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:31:28.825 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.821 7f488e61c700 1 --2- 172.21.2.112:0/3272774822 >> v2:172.21.2.120:3300/0 conn(0x7f48900e5070 0x7f48901b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:28.825 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.822 7f488ee1d700 1 -- 172.21.2.112:0/3272774822 >> v2:172.21.2.120:3300/0 conn(0x7f48900e5070 msgr2=0x7f48901b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:28.825 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.822 7f488ee1d700 1 --2- 172.21.2.112:0/3272774822 >> v2:172.21.2.120:3300/0 conn(0x7f48900e5070 0x7f48901b3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:28.825 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.822 7f488ee1d700 1 -- 172.21.2.112:0/3272774822 >> v1:172.21.2.120:6789/0 conn(0x7f48901b3e40 legacy=0x7f489007d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:28.825 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.822 7f488ee1d700 1 -- 172.21.2.112:0/3272774822 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48900d1bd0 con 0x7f48900e5470 2021-02-10T11:31:28.825 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.822 7f488ee1d700 1 --2- 172.21.2.112:0/3272774822 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f48900e5470 0x7f4890080310 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f4888006aa0 tx=0x7f4888009f20).ready entity=mon.2 client_cookie=6f8e180ef5645c02 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:28.826 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.822 7f488d61a700 1 -- 172.21.2.112:0/3272774822 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f488800b5c0 con 0x7f48900e5470 2021-02-10T11:31:28.826 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.822 7f488d61a700 1 -- 172.21.2.112:0/3272774822 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4888003030 con 0x7f48900e5470 2021-02-10T11:31:28.826 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.822 7f488d61a700 1 -- 172.21.2.112:0/3272774822 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f488800ba80 con 0x7f48900e5470 2021-02-10T11:31:28.826 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.822 7f48948c6700 1 -- 172.21.2.112:0/3272774822 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f48900e5470 msgr2=0x7f4890080310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:28.826 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.822 7f48948c6700 1 --2- 172.21.2.112:0/3272774822 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f48900e5470 0x7f4890080310 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f4888006aa0 tx=0x7f4888009f20).stop 2021-02-10T11:31:28.826 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.823 7f48948c6700 1 -- 172.21.2.112:0/3272774822 shutdown_connections 2021-02-10T11:31:28.826 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.823 7f48948c6700 1 --2- 172.21.2.112:0/3272774822 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f48900e5470 0x7f4890080310 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:28.826 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.823 7f48948c6700 1 --2- 172.21.2.112:0/3272774822 >> v2:172.21.2.120:3300/0 conn(0x7f48900e5070 0x7f48901b3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:28.826 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.823 7f48948c6700 1 -- 172.21.2.112:0/3272774822 >> 172.21.2.112:0/3272774822 conn(0x7f48900ce690 msgr2=0x7f48900e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:28.826 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.823 7f48948c6700 1 -- 172.21.2.112:0/3272774822 shutdown_connections 2021-02-10T11:31:28.826 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.823 7f48948c6700 1 -- 172.21.2.112:0/3272774822 wait complete. 2021-02-10T11:31:28.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.837 7f48948c6700 1 Processor -- start 2021-02-10T11:31:28.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.837 7f48948c6700 1 -- start start 2021-02-10T11:31:28.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.838 7f48948c6700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f489008d400 0x7f4890096c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:28.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.838 7f48948c6700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f48900b1450 0x7f48900b1850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:28.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.838 7f48948c6700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f48900b1e70 0x7f48901b1f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:28.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.838 7f48948c6700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4890097310 con 0x7f489008d400 2021-02-10T11:31:28.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.838 7f48948c6700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f489008d800 con 0x7f48900b1e70 2021-02-10T11:31:28.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.838 7f48948c6700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f489008d9f0 con 0x7f48900b1450 2021-02-10T11:31:28.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.838 7f488e61c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f489008d400 0x7f4890096c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:28.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.838 7f488e61c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f489008d400 0x7f4890096c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:48046/0 (socket says 172.21.2.112:48046) 2021-02-10T11:31:28.829 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.838 7f488e61c700 1 -- 172.21.2.112:0/780253648 learned_addr learned my addr 172.21.2.112:0/780253648 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:31:28.829 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.838 7f488ee1d700 1 --2- 172.21.2.112:0/780253648 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f48900b1e70 0x7f48901b1f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:28.829 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.838 7f488de1b700 1 --2- 172.21.2.112:0/780253648 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f48900b1450 0x7f48900b1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:28.829 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.838 7f488e61c700 1 -- 172.21.2.112:0/780253648 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f48900b1450 msgr2=0x7f48900b1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:28.829 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.839 7f488e61c700 1 --2- 172.21.2.112:0/780253648 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f48900b1450 0x7f48900b1850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:28.829 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.839 7f488e61c700 1 -- 172.21.2.112:0/780253648 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f48900b1e70 msgr2=0x7f48901b1f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:28.829 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.839 7f488e61c700 1 --2- 172.21.2.112:0/780253648 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f48900b1e70 0x7f48901b1f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:28.829 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.839 7f488e61c700 1 -- 172.21.2.112:0/780253648 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48901b25f0 con 0x7f489008d400 2021-02-10T11:31:28.829 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.839 7f488de1b700 1 --2- 172.21.2.112:0/780253648 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f48900b1450 0x7f48900b1850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:31:28.829 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.839 7f488e61c700 1 --2- 172.21.2.112:0/780253648 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f489008d400 0x7f4890096c70 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 rx=0x7f4878006f60 tx=0x7f487800b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:28.829 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.839 7f4886ffd700 1 -- 172.21.2.112:0/780253648 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f487800bd60 con 0x7f489008d400 2021-02-10T11:31:28.829 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.839 7f4886ffd700 1 -- 172.21.2.112:0/780253648 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4878000c90 con 0x7f489008d400 2021-02-10T11:31:28.829 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.839 7f4886ffd700 1 -- 172.21.2.112:0/780253648 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f48780022b0 con 0x7f489008d400 2021-02-10T11:31:28.830 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.839 7f48948c6700 1 -- 172.21.2.112:0/780253648 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48901b25f0 con 0x7f489008d400 2021-02-10T11:31:28.830 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.839 7f48948c6700 1 -- 172.21.2.112:0/780253648 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48901b2b80 con 0x7f489008d400 2021-02-10T11:31:28.830 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.839 7f4886ffd700 1 -- 172.21.2.112:0/780253648 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4878006b00 con 0x7f489008d400 2021-02-10T11:31:28.830 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.840 7f4886ffd700 1 --2- 172.21.2.112:0/780253648 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f48740186c0 0x7f487401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:28.830 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.840 7f4886ffd700 1 -- 172.21.2.112:0/780253648 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f48780323e0 con 0x7f489008d400 2021-02-10T11:31:28.830 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.840 7f488de1b700 1 --2- 172.21.2.112:0/780253648 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f48740186c0 0x7f487401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:28.830 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.840 7f488de1b700 1 --2- 172.21.2.112:0/780253648 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f48740186c0 0x7f487401abc0 crc :-1 s=READY pgs=499 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:28.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.870 7f48948c6700 1 -- 172.21.2.112:0/780253648 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f489007ed80 con 0x7f489008d400 2021-02-10T11:31:28.860 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.873 7f4886ffd700 1 -- 172.21.2.112:0/780253648 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f487802a490 con 0x7f489008d400 2021-02-10T11:31:28.987 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:28.999 7f48948c6700 1 -- 172.21.2.112:0/780253648 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f489007ed80 con 0x7f489008d400 2021-02-10T11:31:28.987 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:29.000 7f4886ffd700 1 -- 172.21.2.112:0/780253648 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+2098 (secure 0 0 0) 0x7f487802a490 con 0x7f489008d400 2021-02-10T11:31:28.988 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 14 2021-02-10T11:31:28.988 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:31:28.988 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":14,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4627,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/2302110857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":2302110857},{"type":"v1","addr":"172.21.2.112:6833","nonce":2302110857}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":9}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:15:33.919772","modified":"2021-02-10 11:15:36.309405","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4427},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4427":{"gid":4427,"name":"b","rank":0,"incarnation":10,"state":"up:active","state_seq":4,"addr":"172.21.2.120:6835/3730022736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":3730022736},{"type":"v1","addr":"172.21.2.120:6835","nonce":3730022736}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2021-02-10T11:31:28.989 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:29.002 7f48948c6700 1 -- 172.21.2.112:0/780253648 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f48740186c0 msgr2=0x7f487401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:28.989 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:29.002 7f48948c6700 1 --2- 172.21.2.112:0/780253648 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f48740186c0 0x7f487401abc0 crc :-1 s=READY pgs=499 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:28.989 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:29.002 7f48948c6700 1 -- 172.21.2.112:0/780253648 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f489008d400 msgr2=0x7f4890096c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:28.990 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:29.002 7f48948c6700 1 --2- 172.21.2.112:0/780253648 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f489008d400 0x7f4890096c70 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 rx=0x7f4878006f60 tx=0x7f487800b970).stop 2021-02-10T11:31:28.990 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:29.002 7f48948c6700 1 -- 172.21.2.112:0/780253648 shutdown_connections 2021-02-10T11:31:28.990 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:29.002 7f48948c6700 1 --2- 172.21.2.112:0/780253648 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f48740186c0 0x7f487401abc0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:28.990 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:29.002 7f48948c6700 1 --2- 172.21.2.112:0/780253648 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f48900b1e70 0x7f48901b1f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:28.990 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:29.002 7f48948c6700 1 --2- 172.21.2.112:0/780253648 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f48900b1450 0x7f48900b1850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:28.990 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:29.002 7f48948c6700 1 --2- 172.21.2.112:0/780253648 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f489008d400 0x7f4890096c70 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:28.990 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:29.002 7f48948c6700 1 -- 172.21.2.112:0/780253648 >> 172.21.2.112:0/780253648 conn(0x7f48901ac620 msgr2=0x7f48900896e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:28.990 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:29.002 7f48948c6700 1 -- 172.21.2.112:0/780253648 shutdown_connections 2021-02-10T11:31:28.990 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:29.002 7f48948c6700 1 -- 172.21.2.112:0/780253648 wait complete. 2021-02-10T11:31:28.998 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs fail cephfs 2021-02-10T11:31:29.424 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:29.421 7f8052a3b700 1 Processor -- start 2021-02-10T11:31:29.425 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:29.421 7f8052a3b700 1 -- start start 2021-02-10T11:31:29.425 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:29.421 7f8052a3b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f804c0e5070 0x7f804c1b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:29.425 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:29.421 7f8052a3b700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f804c1b3e40 0x7f804c080310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:29.425 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:29.421 7f8052a3b700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f804c07ed80 con 0x7f804c0e5470 2021-02-10T11:31:29.425 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:29.421 7f8052a3b700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f804c0d1790 con 0x7f804c1b3e40 2021-02-10T11:31:29.425 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:29.421 7f8052a3b700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f804c0d1980 con 0x7f804c0e5070 2021-02-10T11:31:29.425 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:29.422 7f8051058700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f804c1b3e40 0x7f804c080310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:29.425 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:29.422 7f8051058700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f804c1b3e40 0x7f804c080310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:48054/0 (socket says 172.21.2.112:48054) 2021-02-10T11:31:29.425 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:29.422 7f8051058700 1 -- 172.21.2.112:0/876376293 learned_addr learned my addr 172.21.2.112:0/876376293 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:31:29.425 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:29.422 7f8050857700 1 --2- 172.21.2.112:0/876376293 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f804c0e5070 0x7f804c1b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:29.425 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:29.422 7f8051058700 1 -- 172.21.2.112:0/876376293 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f804c0e5070 msgr2=0x7f804c1b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:29.425 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:29.422 7f8051058700 1 --2- 172.21.2.112:0/876376293 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f804c0e5070 0x7f804c1b3820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:29.426 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:29.422 7f8051058700 1 -- 172.21.2.112:0/876376293 >> v1:172.21.2.120:6789/0 conn(0x7f804c0e5470 legacy=0x7f804c07d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:29.426 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:29.422 7f8051058700 1 -- 172.21.2.112:0/876376293 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f804c0d1ba0 con 0x7f804c1b3e40 2021-02-10T11:31:29.426 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:29.422 7f8051058700 1 --2- 172.21.2.112:0/876376293 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f804c1b3e40 0x7f804c080310 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 rx=0x7f8044006f00 tx=0x7f8044009f10).ready entity=mon.0 client_cookie=23bdc8eef2af0a0c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:29.426 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:29.422 7f804b7fe700 1 -- 172.21.2.112:0/876376293 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f804400b3b0 con 0x7f804c1b3e40 2021-02-10T11:31:29.426 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:29.422 7f804b7fe700 1 -- 172.21.2.112:0/876376293 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f804400b630 con 0x7f804c1b3e40 2021-02-10T11:31:29.426 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:29.422 7f804b7fe700 1 -- 172.21.2.112:0/876376293 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f804400bb50 con 0x7f804c1b3e40 2021-02-10T11:31:29.426 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:29.422 7f8052a3b700 1 -- 172.21.2.112:0/876376293 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f804c1b3e40 msgr2=0x7f804c080310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:29.426 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:29.422 7f8052a3b700 1 --2- 172.21.2.112:0/876376293 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f804c1b3e40 0x7f804c080310 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 rx=0x7f8044006f00 tx=0x7f8044009f10).stop 2021-02-10T11:31:29.426 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:29.423 7f8052a3b700 1 -- 172.21.2.112:0/876376293 shutdown_connections 2021-02-10T11:31:29.426 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:29.423 7f8052a3b700 1 --2- 172.21.2.112:0/876376293 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f804c1b3e40 0x7f804c080310 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:29.426 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:29.423 7f8052a3b700 1 --2- 172.21.2.112:0/876376293 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f804c0e5070 0x7f804c1b3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:29.426 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:29.423 7f8052a3b700 1 -- 172.21.2.112:0/876376293 >> 172.21.2.112:0/876376293 conn(0x7f804c0ce690 msgr2=0x7f804c0e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:29.427 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:29.423 7f8052a3b700 1 -- 172.21.2.112:0/876376293 shutdown_connections 2021-02-10T11:31:29.427 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:29.423 7f8052a3b700 1 -- 172.21.2.112:0/876376293 wait complete. 2021-02-10T11:31:29.428 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:29.437 7f8052a3b700 1 Processor -- start 2021-02-10T11:31:29.428 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:29.438 7f8052a3b700 1 -- start start 2021-02-10T11:31:29.428 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:29.438 7f8052a3b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f804c0817e0 0x7f804c081be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:29.428 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:29.438 7f8052a3b700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f804c082200 0x7f804c082600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:29.428 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:29.438 7f8052a3b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f804c0be320 0x7f804c0be720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:29.428 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:29.438 7f8052a3b700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f804c19d580 con 0x7f804c0817e0 2021-02-10T11:31:29.428 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:29.438 7f8052a3b700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f804c19d770 con 0x7f804c082200 2021-02-10T11:31:29.429 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:29.438 7f8052a3b700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f804c0bed40 con 0x7f804c0be320 2021-02-10T11:31:29.429 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:29.438 7f8050857700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f804c0817e0 0x7f804c081be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:29.429 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:29.438 7f8050857700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f804c0817e0 0x7f804c081be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:48058/0 (socket says 172.21.2.112:48058) 2021-02-10T11:31:29.429 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:29.438 7f8050857700 1 -- 172.21.2.112:0/3179883243 learned_addr learned my addr 172.21.2.112:0/3179883243 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:31:29.429 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:29.439 7f8050857700 1 -- 172.21.2.112:0/3179883243 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f804c0be320 msgr2=0x7f804c0be720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:29.429 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:29.439 7f8050857700 1 --2- 172.21.2.112:0/3179883243 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f804c0be320 0x7f804c0be720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:29.429 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:29.439 7f8050857700 1 -- 172.21.2.112:0/3179883243 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f804c082200 msgr2=0x7f804c082600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:29.429 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:29.439 7f8050857700 1 --2- 172.21.2.112:0/3179883243 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f804c082200 0x7f804c082600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:29.429 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:29.439 7f8050857700 1 -- 172.21.2.112:0/3179883243 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f804c0bef60 con 0x7f804c0817e0 2021-02-10T11:31:29.429 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:29.439 7f8050857700 1 --2- 172.21.2.112:0/3179883243 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f804c0817e0 0x7f804c081be0 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 rx=0x7f8034006f10 tx=0x7f803400b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:29.429 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:29.439 7f80497fa700 1 -- 172.21.2.112:0/3179883243 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f803400bd80 con 0x7f804c0817e0 2021-02-10T11:31:29.429 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:29.439 7f80497fa700 1 -- 172.21.2.112:0/3179883243 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8034000c90 con 0x7f804c0817e0 2021-02-10T11:31:29.429 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:29.439 7f80497fa700 1 -- 172.21.2.112:0/3179883243 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f80340022b0 con 0x7f804c0817e0 2021-02-10T11:31:29.430 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:29.439 7f8052a3b700 1 -- 172.21.2.112:0/3179883243 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f804c0bef60 con 0x7f804c0817e0 2021-02-10T11:31:29.430 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:29.439 7f8052a3b700 1 -- 172.21.2.112:0/3179883243 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f804c0bf4f0 con 0x7f804c0817e0 2021-02-10T11:31:29.430 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:29.440 7f80497fa700 1 -- 172.21.2.112:0/3179883243 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f80340052e0 con 0x7f804c0817e0 2021-02-10T11:31:29.430 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:29.440 7f80497fa700 1 --2- 172.21.2.112:0/3179883243 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f80300186c0 0x7f803001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:29.430 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:29.440 7f80497fa700 1 -- 172.21.2.112:0/3179883243 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f8034032560 con 0x7f804c0817e0 2021-02-10T11:31:29.430 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:29.440 7f804bfff700 1 --2- 172.21.2.112:0/3179883243 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f80300186c0 0x7f803001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:29.430 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:29.440 7f804bfff700 1 --2- 172.21.2.112:0/3179883243 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f80300186c0 0x7f803001abc0 crc :-1 s=READY pgs=500 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:29.458 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:29.471 7f8052a3b700 1 -- 172.21.2.112:0/3179883243 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f804c0812b0 con 0x7f804c0817e0 2021-02-10T11:31:29.460 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:29.473 7f80497fa700 1 -- 172.21.2.112:0/3179883243 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8034000940 con 0x7f804c0817e0 2021-02-10T11:31:29.587 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:29.600 7f8052a3b700 1 -- 172.21.2.112:0/3179883243 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f804c0812b0 con 0x7f804c0817e0 2021-02-10T11:31:30.022 INFO:tasks.ceph.mon.a.gibba012.stderr:2021-02-10 11:31:30.035 7fc3fe6e2700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-02-10T11:31:30.025 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:30.037 7f80497fa700 1 -- 172.21.2.112:0/3179883243 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v15) v1 ==== 161+0+0 (secure 0 0 0) 0x7f803401f300 con 0x7f804c0817e0 2021-02-10T11:31:30.054 INFO:teuthology.orchestra.run.gibba012.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2021-02-10T11:31:30.056 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:30.069 7f8052a3b700 1 -- 172.21.2.112:0/3179883243 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f80300186c0 msgr2=0x7f803001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:30.056 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:30.069 7f8052a3b700 1 --2- 172.21.2.112:0/3179883243 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f80300186c0 0x7f803001abc0 crc :-1 s=READY pgs=500 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:30.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:30.069 7f8052a3b700 1 -- 172.21.2.112:0/3179883243 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f804c0817e0 msgr2=0x7f804c081be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:30.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:30.069 7f8052a3b700 1 --2- 172.21.2.112:0/3179883243 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f804c0817e0 0x7f804c081be0 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 rx=0x7f8034006f10 tx=0x7f803400b990).stop 2021-02-10T11:31:30.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:30.069 7f8052a3b700 1 -- 172.21.2.112:0/3179883243 shutdown_connections 2021-02-10T11:31:30.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:30.069 7f8052a3b700 1 --2- 172.21.2.112:0/3179883243 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f80300186c0 0x7f803001abc0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:30.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:30.069 7f8052a3b700 1 --2- 172.21.2.112:0/3179883243 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f804c082200 0x7f804c082600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:30.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:30.069 7f8052a3b700 1 --2- 172.21.2.112:0/3179883243 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f804c0be320 0x7f804c0be720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:30.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:30.069 7f8052a3b700 1 --2- 172.21.2.112:0/3179883243 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f804c0817e0 0x7f804c081be0 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:30.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:30.069 7f8052a3b700 1 -- 172.21.2.112:0/3179883243 >> 172.21.2.112:0/3179883243 conn(0x7f804c0a6c50 msgr2=0x7f804c0c04c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:30.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:30.069 7f8052a3b700 1 -- 172.21.2.112:0/3179883243 shutdown_connections 2021-02-10T11:31:30.058 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:30.070 7f8052a3b700 1 -- 172.21.2.112:0/3179883243 wait complete. 2021-02-10T11:31:30.065 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:31:30.078 INFO:tasks.ceph.mds.b.gibba020.stdout:starting mds.b at 2021-02-10T11:31:30.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:30.487 7f423e5bd700 1 Processor -- start 2021-02-10T11:31:30.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:30.487 7f423e5bd700 1 -- start start 2021-02-10T11:31:30.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:30.487 7f423e5bd700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f42380d17f0 0x7f42381b31e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:30.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:30.487 7f423e5bd700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f42380cff00 0x7f42380ef620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:30.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:30.487 7f423e5bd700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42380f0b20 con 0x7f42380d1bf0 2021-02-10T11:31:30.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:30.487 7f423e5bd700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42381b3ff0 con 0x7f42380d17f0 2021-02-10T11:31:30.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:30.487 7f423e5bd700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42380f20b0 con 0x7f42380cff00 2021-02-10T11:31:30.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:30.488 7f4237fff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f42380d17f0 0x7f42381b31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:30.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:30.488 7f4237fff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f42380d17f0 0x7f42381b31e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:48066/0 (socket says 172.21.2.112:48066) 2021-02-10T11:31:30.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:30.488 7f4237fff700 1 -- 172.21.2.112:0/4000641638 learned_addr learned my addr 172.21.2.112:0/4000641638 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:31:30.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:30.488 7f4237fff700 1 -- 172.21.2.112:0/4000641638 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f42380cff00 msgr2=0x7f42380ef620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:30.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:30.488 7f4237fff700 1 --2- 172.21.2.112:0/4000641638 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f42380cff00 0x7f42380ef620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:30.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:30.488 7f4237fff700 1 -- 172.21.2.112:0/4000641638 >> v1:172.21.2.120:6789/0 conn(0x7f42380d1bf0 legacy=0x7f42381b3800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:30.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:30.488 7f4237fff700 1 -- 172.21.2.112:0/4000641638 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42380f22d0 con 0x7f42380d17f0 2021-02-10T11:31:30.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:30.488 7f4237fff700 1 --2- 172.21.2.112:0/4000641638 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f42380d17f0 0x7f42381b31e0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 rx=0x7f4228006700 tx=0x7f4228009b80).ready entity=mon.0 client_cookie=97f8981fa3b6c0ae server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:30.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:30.488 7f4236ffd700 1 -- 172.21.2.112:0/4000641638 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f422800c0c0 con 0x7f42380d17f0 2021-02-10T11:31:30.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:30.488 7f4236ffd700 1 -- 172.21.2.112:0/4000641638 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f422800c340 con 0x7f42380d17f0 2021-02-10T11:31:30.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:30.488 7f4236ffd700 1 -- 172.21.2.112:0/4000641638 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f422800c860 con 0x7f42380d17f0 2021-02-10T11:31:30.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:30.488 7f423e5bd700 1 -- 172.21.2.112:0/4000641638 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f42380d17f0 msgr2=0x7f42381b31e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:30.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:30.488 7f423e5bd700 1 --2- 172.21.2.112:0/4000641638 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f42380d17f0 0x7f42381b31e0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 rx=0x7f4228006700 tx=0x7f4228009b80).stop 2021-02-10T11:31:30.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:30.489 7f423e5bd700 1 -- 172.21.2.112:0/4000641638 shutdown_connections 2021-02-10T11:31:30.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:30.489 7f423e5bd700 1 --2- 172.21.2.112:0/4000641638 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f42380cff00 0x7f42380ef620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:30.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:30.489 7f423e5bd700 1 --2- 172.21.2.112:0/4000641638 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f42380d17f0 0x7f42381b31e0 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:30.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:30.489 7f423e5bd700 1 -- 172.21.2.112:0/4000641638 >> 172.21.2.112:0/4000641638 conn(0x7f42380fde40 msgr2=0x7f42380ce390 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:30.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:30.489 7f423e5bd700 1 -- 172.21.2.112:0/4000641638 shutdown_connections 2021-02-10T11:31:30.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:30.489 7f423e5bd700 1 -- 172.21.2.112:0/4000641638 wait complete. 2021-02-10T11:31:30.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:30.504 7f423e5bd700 1 Processor -- start 2021-02-10T11:31:30.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:30.504 7f423e5bd700 1 -- start start 2021-02-10T11:31:30.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:30.504 7f423e5bd700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f42380b3480 0x7f42380b3880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:30.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:30.504 7f423e5bd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4238106330 0x7f4238106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:30.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:30.504 7f423e5bd700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4238106d50 0x7f4238107150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:30.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:30.504 7f423e5bd700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42381038d0 con 0x7f4238106330 2021-02-10T11:31:30.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:30.504 7f423e5bd700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f423818d9c0 con 0x7f42380b3480 2021-02-10T11:31:30.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:30.504 7f423e5bd700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f423818dbe0 con 0x7f4238106d50 2021-02-10T11:31:30.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:30.505 7f42377fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4238106330 0x7f4238106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:30.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:30.505 7f42377fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4238106330 0x7f4238106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:48074/0 (socket says 172.21.2.112:48074) 2021-02-10T11:31:30.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:30.505 7f42377fe700 1 -- 172.21.2.112:0/3963039771 learned_addr learned my addr 172.21.2.112:0/3963039771 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:31:30.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:30.505 7f4237fff700 1 --2- 172.21.2.112:0/3963039771 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f42380b3480 0x7f42380b3880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:30.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:30.505 7f423cbda700 1 --2- 172.21.2.112:0/3963039771 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4238106d50 0x7f4238107150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:30.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:30.505 7f42377fe700 1 -- 172.21.2.112:0/3963039771 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4238106d50 msgr2=0x7f4238107150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:30.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:30.505 7f42377fe700 1 --2- 172.21.2.112:0/3963039771 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4238106d50 0x7f4238107150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:30.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:30.505 7f42377fe700 1 -- 172.21.2.112:0/3963039771 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f42380b3480 msgr2=0x7f42380b3880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:30.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:30.505 7f42377fe700 1 --2- 172.21.2.112:0/3963039771 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f42380b3480 0x7f42380b3880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:30.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:30.505 7f42377fe700 1 -- 172.21.2.112:0/3963039771 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f423818de00 con 0x7f4238106330 2021-02-10T11:31:30.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:30.505 7f42377fe700 1 --2- 172.21.2.112:0/3963039771 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4238106330 0x7f4238106730 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 rx=0x7f422c00b5b0 tx=0x7f422c00b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:30.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:30.505 7f4234ff9700 1 -- 172.21.2.112:0/3963039771 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f422c00bd40 con 0x7f4238106330 2021-02-10T11:31:30.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:30.505 7f4234ff9700 1 -- 172.21.2.112:0/3963039771 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f422c000ce0 con 0x7f4238106330 2021-02-10T11:31:30.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:30.505 7f4234ff9700 1 -- 172.21.2.112:0/3963039771 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f422c006210 con 0x7f4238106330 2021-02-10T11:31:30.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:30.505 7f423e5bd700 1 -- 172.21.2.112:0/3963039771 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f423818de00 con 0x7f4238106330 2021-02-10T11:31:30.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:30.506 7f423e5bd700 1 -- 172.21.2.112:0/3963039771 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f423818e390 con 0x7f4238106330 2021-02-10T11:31:30.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:30.506 7f4234ff9700 1 -- 172.21.2.112:0/3963039771 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f422c006020 con 0x7f4238106330 2021-02-10T11:31:30.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:30.506 7f4234ff9700 1 --2- 172.21.2.112:0/3963039771 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f42180186c0 0x7f421801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:30.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:30.506 7f4234ff9700 1 -- 172.21.2.112:0/3963039771 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5659+0+0 (secure 0 0 0) 0x7f422c0325d0 con 0x7f4238106330 2021-02-10T11:31:30.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:30.507 7f4237fff700 1 --2- 172.21.2.112:0/3963039771 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f42180186c0 0x7f421801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:30.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:30.507 7f4237fff700 1 --2- 172.21.2.112:0/3963039771 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f42180186c0 0x7f421801abc0 crc :-1 s=READY pgs=501 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:30.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:30.537 7f423e5bd700 1 -- 172.21.2.112:0/3963039771 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42380f0b20 con 0x7f4238106330 2021-02-10T11:31:30.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:30.540 7f4234ff9700 1 -- 172.21.2.112:0/3963039771 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f422c030db0 con 0x7f4238106330 2021-02-10T11:31:30.657 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:30.670 7f423e5bd700 1 -- 172.21.2.112:0/3963039771 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f42380f0b20 con 0x7f4238106330 2021-02-10T11:31:30.658 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:30.670 7f4234ff9700 1 -- 172.21.2.112:0/3963039771 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+1753 (secure 0 0 0) 0x7f422c00bd40 con 0x7f4238106330 2021-02-10T11:31:30.658 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:31:30.658 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":15,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4627,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/2302110857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":2302110857},{"type":"v1","addr":"172.21.2.112:6833","nonce":2302110857}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":9}],"filesystems":[{"mdsmap":{"epoch":15,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:15:33.919772","modified":"2021-02-10 11:31:30.035625","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2021-02-10T11:31:30.659 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 15 2021-02-10T11:31:30.660 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:30.672 7f423e5bd700 1 -- 172.21.2.112:0/3963039771 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f42180186c0 msgr2=0x7f421801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:30.660 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:30.672 7f423e5bd700 1 --2- 172.21.2.112:0/3963039771 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f42180186c0 0x7f421801abc0 crc :-1 s=READY pgs=501 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:30.660 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:30.672 7f423e5bd700 1 -- 172.21.2.112:0/3963039771 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4238106330 msgr2=0x7f4238106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:30.660 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:30.672 7f423e5bd700 1 --2- 172.21.2.112:0/3963039771 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4238106330 0x7f4238106730 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 rx=0x7f422c00b5b0 tx=0x7f422c00b950).stop 2021-02-10T11:31:30.660 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:30.672 7f423e5bd700 1 -- 172.21.2.112:0/3963039771 shutdown_connections 2021-02-10T11:31:30.660 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:30.672 7f423e5bd700 1 --2- 172.21.2.112:0/3963039771 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f42180186c0 0x7f421801abc0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:30.660 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:30.672 7f423e5bd700 1 --2- 172.21.2.112:0/3963039771 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4238106d50 0x7f4238107150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:30.660 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:30.672 7f423e5bd700 1 --2- 172.21.2.112:0/3963039771 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4238106330 0x7f4238106730 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:30.660 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:30.672 7f423e5bd700 1 --2- 172.21.2.112:0/3963039771 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f42380b3480 0x7f42380b3880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:30.661 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:30.672 7f423e5bd700 1 -- 172.21.2.112:0/3963039771 >> 172.21.2.112:0/3963039771 conn(0x7f42381a3780 msgr2=0x7f42381a5db0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:30.661 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:30.673 7f423e5bd700 1 -- 172.21.2.112:0/3963039771 shutdown_connections 2021-02-10T11:31:30.661 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:30.673 7f423e5bd700 1 -- 172.21.2.112:0/3963039771 wait complete. 2021-02-10T11:31:30.669 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2021-02-10T11:31:31.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:31.099 7fb480859700 1 Processor -- start 2021-02-10T11:31:31.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:31.099 7fb480859700 1 -- start start 2021-02-10T11:31:31.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:31.100 7fb480859700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fb47c0d17f0 0x7fb47c1b31e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:31.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:31.100 7fb480859700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb47c0cff00 0x7fb47c0ef620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:31.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:31.100 7fb480859700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb47c0f0b20 con 0x7fb47c0d1bf0 2021-02-10T11:31:31.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:31.100 7fb480859700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb47c1b3ff0 con 0x7fb47c0d17f0 2021-02-10T11:31:31.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:31.100 7fb480859700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb47c0f20b0 con 0x7fb47c0cff00 2021-02-10T11:31:31.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:31.100 7fb47a61c700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fb47c0d17f0 0x7fb47c1b31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:31.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:31.100 7fb479e1b700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7fb47c0d1bf0 0x7fb47c1b3800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:41100/0 (socket says 172.21.2.112:41100) 2021-02-10T11:31:31.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:31.100 7fb479e1b700 1 -- 172.21.2.112:0/2371504807 learned_addr learned my addr 172.21.2.112:0/2371504807 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:31:31.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:31.100 7fb47ae1d700 1 --2- 172.21.2.112:0/2371504807 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb47c0cff00 0x7fb47c0ef620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:31.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:31.100 7fb47a61c700 1 -- 172.21.2.112:0/2371504807 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb47c0cff00 msgr2=0x7fb47c0ef620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:31.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:31.100 7fb47a61c700 1 --2- 172.21.2.112:0/2371504807 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb47c0cff00 0x7fb47c0ef620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:31.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:31.100 7fb47a61c700 1 -- 172.21.2.112:0/2371504807 >> v1:172.21.2.120:6789/0 conn(0x7fb47c0d1bf0 legacy=0x7fb47c1b3800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:31.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:31.100 7fb47a61c700 1 -- 172.21.2.112:0/2371504807 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb47c0f22d0 con 0x7fb47c0d17f0 2021-02-10T11:31:31.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:31.101 7fb47a61c700 1 --2- 172.21.2.112:0/2371504807 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb47c0d17f0 0x7fb47c1b31e0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7fb464006650 tx=0x7fb464009a80).ready entity=mon.1 client_cookie=5cd0ba12238ea6a7 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:31.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:31.101 7fb47961a700 1 -- 172.21.2.112:0/2371504807 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb46400c0c0 con 0x7fb47c0d17f0 2021-02-10T11:31:31.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:31.101 7fb47961a700 1 -- 172.21.2.112:0/2371504807 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb46400c340 con 0x7fb47c0d17f0 2021-02-10T11:31:31.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:31.101 7fb47961a700 1 -- 172.21.2.112:0/2371504807 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb46400c860 con 0x7fb47c0d17f0 2021-02-10T11:31:31.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:31.101 7fb480859700 1 -- 172.21.2.112:0/2371504807 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb47c0d17f0 msgr2=0x7fb47c1b31e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:31.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:31.101 7fb480859700 1 --2- 172.21.2.112:0/2371504807 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb47c0d17f0 0x7fb47c1b31e0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7fb464006650 tx=0x7fb464009a80).stop 2021-02-10T11:31:31.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:31.101 7fb480859700 1 -- 172.21.2.112:0/2371504807 shutdown_connections 2021-02-10T11:31:31.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:31.101 7fb480859700 1 --2- 172.21.2.112:0/2371504807 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb47c0cff00 0x7fb47c0ef620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:31.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:31.101 7fb480859700 1 --2- 172.21.2.112:0/2371504807 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb47c0d17f0 0x7fb47c1b31e0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:31.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:31.101 7fb480859700 1 -- 172.21.2.112:0/2371504807 >> 172.21.2.112:0/2371504807 conn(0x7fb47c0fde40 msgr2=0x7fb47c0ce390 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:31.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:31.102 7fb480859700 1 -- 172.21.2.112:0/2371504807 shutdown_connections 2021-02-10T11:31:31.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:31.102 7fb480859700 1 -- 172.21.2.112:0/2371504807 wait complete. 2021-02-10T11:31:31.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:31.116 7fb480859700 1 Processor -- start 2021-02-10T11:31:31.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:31.116 7fb480859700 1 -- start start 2021-02-10T11:31:31.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:31.117 7fb480859700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb47c0ae7e0 0x7fb47c1038c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:31.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:31.117 7fb480859700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb47c103ee0 0x7fb47c19dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:31.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:31.117 7fb480859700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb47c19e340 0x7fb47c098f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:31.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:31.117 7fb480859700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb47c19d470 con 0x7fb47c0ae7e0 2021-02-10T11:31:31.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:31.117 7fb480859700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb47c099540 con 0x7fb47c103ee0 2021-02-10T11:31:31.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:31.117 7fb480859700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb47c099760 con 0x7fb47c19e340 2021-02-10T11:31:31.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:31.117 7fb47a61c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb47c0ae7e0 0x7fb47c1038c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:31.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:31.117 7fb47a61c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb47c0ae7e0 0x7fb47c1038c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:48088/0 (socket says 172.21.2.112:48088) 2021-02-10T11:31:31.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:31.117 7fb47a61c700 1 -- 172.21.2.112:0/3153894456 learned_addr learned my addr 172.21.2.112:0/3153894456 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:31:31.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:31.117 7fb47a61c700 1 -- 172.21.2.112:0/3153894456 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb47c19e340 msgr2=0x7fb47c098f20 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:31:31.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:31.117 7fb47a61c700 1 --2- 172.21.2.112:0/3153894456 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb47c19e340 0x7fb47c098f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:31.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:31.117 7fb47a61c700 1 -- 172.21.2.112:0/3153894456 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb47c103ee0 msgr2=0x7fb47c19dd20 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:31:31.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:31.117 7fb47a61c700 1 --2- 172.21.2.112:0/3153894456 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb47c103ee0 0x7fb47c19dd20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:31.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:31.117 7fb47a61c700 1 -- 172.21.2.112:0/3153894456 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb47c099980 con 0x7fb47c0ae7e0 2021-02-10T11:31:31.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:31.117 7fb47a61c700 1 --2- 172.21.2.112:0/3153894456 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb47c0ae7e0 0x7fb47c1038c0 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 rx=0x7fb464009630 tx=0x7fb46400b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:31.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:31.117 7fb472ffd700 1 -- 172.21.2.112:0/3153894456 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb4640093a0 con 0x7fb47c0ae7e0 2021-02-10T11:31:31.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:31.117 7fb472ffd700 1 -- 172.21.2.112:0/3153894456 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb46400bd70 con 0x7fb47c0ae7e0 2021-02-10T11:31:31.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:31.117 7fb480859700 1 -- 172.21.2.112:0/3153894456 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb47c099980 con 0x7fb47c0ae7e0 2021-02-10T11:31:31.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:31.117 7fb472ffd700 1 -- 172.21.2.112:0/3153894456 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb464005560 con 0x7fb47c0ae7e0 2021-02-10T11:31:31.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:31.118 7fb480859700 1 -- 172.21.2.112:0/3153894456 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb47c18f8a0 con 0x7fb47c0ae7e0 2021-02-10T11:31:31.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:31.118 7fb472ffd700 1 -- 172.21.2.112:0/3153894456 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb46400a000 con 0x7fb47c0ae7e0 2021-02-10T11:31:31.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:31.118 7fb472ffd700 1 --2- 172.21.2.112:0/3153894456 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb4600186c0 0x7fb46001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:31.109 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:31.118 7fb472ffd700 1 -- 172.21.2.112:0/3153894456 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5659+0+0 (secure 0 0 0) 0x7fb4640325d0 con 0x7fb47c0ae7e0 2021-02-10T11:31:31.109 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:31.119 7fb479e1b700 1 --2- 172.21.2.112:0/3153894456 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb4600186c0 0x7fb46001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:31.109 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:31.119 7fb479e1b700 1 --2- 172.21.2.112:0/3153894456 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb4600186c0 0x7fb46001abc0 crc :-1 s=READY pgs=502 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:31.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:31.149 7fb480859700 1 -- 172.21.2.112:0/3153894456 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb47c103690 con 0x7fb47c0ae7e0 2021-02-10T11:31:31.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:31.152 7fb472ffd700 1 -- 172.21.2.112:0/3153894456 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb464009340 con 0x7fb47c0ae7e0 2021-02-10T11:31:31.267 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:31.280 7fb480859700 1 -- 172.21.2.112:0/3153894456 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fb47c0f0b20 con 0x7fb47c0ae7e0 2021-02-10T11:31:32.036 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:32.049 7fb472ffd700 1 -- 172.21.2.112:0/3153894456 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v16) v1 ==== 104+0+0 (secure 0 0 0) 0x7fb46400a000 con 0x7fb47c0ae7e0 2021-02-10T11:31:32.038 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:32.051 7fb480859700 1 -- 172.21.2.112:0/3153894456 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb4600186c0 msgr2=0x7fb46001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:32.038 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:32.051 7fb480859700 1 --2- 172.21.2.112:0/3153894456 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb4600186c0 0x7fb46001abc0 crc :-1 s=READY pgs=502 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:32.038 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:32.051 7fb480859700 1 -- 172.21.2.112:0/3153894456 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb47c0ae7e0 msgr2=0x7fb47c1038c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:32.038 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:32.051 7fb480859700 1 --2- 172.21.2.112:0/3153894456 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb47c0ae7e0 0x7fb47c1038c0 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 rx=0x7fb464009630 tx=0x7fb46400b980).stop 2021-02-10T11:31:32.038 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:32.051 7fb480859700 1 -- 172.21.2.112:0/3153894456 shutdown_connections 2021-02-10T11:31:32.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:32.051 7fb480859700 1 --2- 172.21.2.112:0/3153894456 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb4600186c0 0x7fb46001abc0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:32.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:32.051 7fb480859700 1 --2- 172.21.2.112:0/3153894456 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb47c103ee0 0x7fb47c19dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:32.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:32.051 7fb480859700 1 --2- 172.21.2.112:0/3153894456 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb47c19e340 0x7fb47c098f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:32.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:32.051 7fb480859700 1 --2- 172.21.2.112:0/3153894456 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb47c0ae7e0 0x7fb47c1038c0 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:32.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:32.051 7fb480859700 1 -- 172.21.2.112:0/3153894456 >> 172.21.2.112:0/3153894456 conn(0x7fb47c1b04b0 msgr2=0x7fb47c19aa40 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:32.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:32.051 7fb480859700 1 -- 172.21.2.112:0/3153894456 shutdown_connections 2021-02-10T11:31:32.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:32.051 7fb480859700 1 -- 172.21.2.112:0/3153894456 wait complete. 2021-02-10T11:31:32.047 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2021-02-10T11:31:32.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:32.473 7f8a2c961700 1 Processor -- start 2021-02-10T11:31:32.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:32.473 7f8a2c961700 1 -- start start 2021-02-10T11:31:32.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:32.473 7f8a2c961700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f8a280d1870 0x7f8a281b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:32.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:32.473 7f8a2c961700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8a281b0ae0 0x7f8a28108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:32.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:32.473 7f8a2c961700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f8a280d1c70 0x7f8a28108f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:32.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:32.473 7f8a2c961700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a280fe1c0 con 0x7f8a280d1c70 2021-02-10T11:31:32.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:32.473 7f8a2c961700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a28109a60 con 0x7f8a280d1870 2021-02-10T11:31:32.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:32.473 7f8a2c961700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a28088020 con 0x7f8a281b0ae0 2021-02-10T11:31:32.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:32.474 7f8a26e1d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f8a280d1c70 0x7f8a28108f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:32.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:32.474 7f8a26e1d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f8a280d1c70 0x7f8a28108f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:48098/0 (socket says 172.21.2.112:48098) 2021-02-10T11:31:32.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:32.474 7f8a26e1d700 1 -- 172.21.2.112:0/3153999197 learned_addr learned my addr 172.21.2.112:0/3153999197 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:31:32.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:32.474 7f8a25e1b700 1 --2- 172.21.2.112:0/3153999197 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8a281b0ae0 0x7f8a28108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:32.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:32.474 7f8a2661c700 1 --2- 172.21.2.112:0/3153999197 >> v2:172.21.2.120:3300/0 conn(0x7f8a280d1870 0x7f8a281b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:32.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:32.474 7f8a26e1d700 1 -- 172.21.2.112:0/3153999197 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8a281b0ae0 msgr2=0x7f8a28108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:32.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:32.474 7f8a26e1d700 1 --2- 172.21.2.112:0/3153999197 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8a281b0ae0 0x7f8a28108940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:32.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:32.474 7f8a26e1d700 1 -- 172.21.2.112:0/3153999197 >> v2:172.21.2.120:3300/0 conn(0x7f8a280d1870 msgr2=0x7f8a281b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:32.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:32.474 7f8a26e1d700 1 --2- 172.21.2.112:0/3153999197 >> v2:172.21.2.120:3300/0 conn(0x7f8a280d1870 0x7f8a281b04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:32.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:32.474 7f8a26e1d700 1 -- 172.21.2.112:0/3153999197 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a28109580 con 0x7f8a280d1c70 2021-02-10T11:31:32.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:32.474 7f8a26e1d700 1 --2- 172.21.2.112:0/3153999197 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8a280d1c70 0x7f8a28108f60 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 rx=0x7f8a20006f00 tx=0x7f8a20009f10).ready entity=mon.0 client_cookie=21e37d5a34e7f013 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:32.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:32.474 7f8a2561a700 1 -- 172.21.2.112:0/3153999197 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8a2000b3b0 con 0x7f8a280d1c70 2021-02-10T11:31:32.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:32.474 7f8a2561a700 1 -- 172.21.2.112:0/3153999197 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a2000b630 con 0x7f8a280d1c70 2021-02-10T11:31:32.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:32.474 7f8a2561a700 1 -- 172.21.2.112:0/3153999197 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8a2000ba90 con 0x7f8a280d1c70 2021-02-10T11:31:32.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:32.474 7f8a2c961700 1 -- 172.21.2.112:0/3153999197 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8a280d1c70 msgr2=0x7f8a28108f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:32.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:32.474 7f8a2c961700 1 --2- 172.21.2.112:0/3153999197 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8a280d1c70 0x7f8a28108f60 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 rx=0x7f8a20006f00 tx=0x7f8a20009f10).stop 2021-02-10T11:31:32.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:32.475 7f8a2c961700 1 -- 172.21.2.112:0/3153999197 shutdown_connections 2021-02-10T11:31:32.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:32.475 7f8a2c961700 1 --2- 172.21.2.112:0/3153999197 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8a280d1c70 0x7f8a28108f60 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:32.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:32.475 7f8a2c961700 1 --2- 172.21.2.112:0/3153999197 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8a281b0ae0 0x7f8a28108940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:32.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:32.475 7f8a2c961700 1 --2- 172.21.2.112:0/3153999197 >> v2:172.21.2.120:3300/0 conn(0x7f8a280d1870 0x7f8a281b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:32.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:32.475 7f8a2c961700 1 -- 172.21.2.112:0/3153999197 >> 172.21.2.112:0/3153999197 conn(0x7f8a280ce420 msgr2=0x7f8a280d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:32.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:32.475 7f8a2c961700 1 -- 172.21.2.112:0/3153999197 shutdown_connections 2021-02-10T11:31:32.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:32.475 7f8a2c961700 1 -- 172.21.2.112:0/3153999197 wait complete. 2021-02-10T11:31:32.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:32.489 7f8a2c961700 1 Processor -- start 2021-02-10T11:31:32.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:32.489 7f8a2c961700 1 -- start start 2021-02-10T11:31:32.481 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:32.490 7f8a2c961700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8a280b8070 0x7f8a280b77c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:32.481 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:32.490 7f8a2c961700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8a280ba4a0 0x7f8a280ba8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:32.481 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:32.490 7f8a2c961700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8a281a2fd0 0x7f8a281a33d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:32.481 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:32.490 7f8a2c961700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a280baec0 con 0x7f8a281a2fd0 2021-02-10T11:31:32.481 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:32.490 7f8a2c961700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a281a39f0 con 0x7f8a280b8070 2021-02-10T11:31:32.481 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:32.490 7f8a2c961700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a281a3c10 con 0x7f8a280ba4a0 2021-02-10T11:31:32.481 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:32.490 7f8a25e1b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8a280ba4a0 0x7f8a280ba8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:32.481 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:32.490 7f8a26e1d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8a281a2fd0 0x7f8a281a33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:32.481 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:32.490 7f8a25e1b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8a280ba4a0 0x7f8a280ba8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3301/0 says I am v2:172.21.2.112:47782/0 (socket says 172.21.2.112:47782) 2021-02-10T11:31:32.481 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:32.490 7f8a25e1b700 1 -- 172.21.2.112:0/2925680021 learned_addr learned my addr 172.21.2.112:0/2925680021 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:31:32.482 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:32.490 7f8a2661c700 1 --2- 172.21.2.112:0/2925680021 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8a280b8070 0x7f8a280b77c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:32.482 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:32.490 7f8a25e1b700 1 -- 172.21.2.112:0/2925680021 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8a280b8070 msgr2=0x7f8a280b77c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:32.482 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:32.490 7f8a25e1b700 1 --2- 172.21.2.112:0/2925680021 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8a280b8070 0x7f8a280b77c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:32.482 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:32.490 7f8a25e1b700 1 -- 172.21.2.112:0/2925680021 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8a281a2fd0 msgr2=0x7f8a281a33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:32.482 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:32.490 7f8a25e1b700 1 --2- 172.21.2.112:0/2925680021 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8a281a2fd0 0x7f8a281a33d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:32.482 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:32.490 7f8a25e1b700 1 -- 172.21.2.112:0/2925680021 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a280ad700 con 0x7f8a280ba4a0 2021-02-10T11:31:32.482 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:32.491 7f8a25e1b700 1 --2- 172.21.2.112:0/2925680021 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8a280ba4a0 0x7f8a280ba8a0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f8a18006f00 tx=0x7f8a1800b940).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:32.482 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:32.491 7f8a1effd700 1 -- 172.21.2.112:0/2925680021 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8a1800bd30 con 0x7f8a280ba4a0 2021-02-10T11:31:32.482 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:32.491 7f8a1effd700 1 -- 172.21.2.112:0/2925680021 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a18000c90 con 0x7f8a280ba4a0 2021-02-10T11:31:32.482 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:32.491 7f8a2c961700 1 -- 172.21.2.112:0/2925680021 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a280ad700 con 0x7f8a280ba4a0 2021-02-10T11:31:32.482 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:32.491 7f8a2c961700 1 -- 172.21.2.112:0/2925680021 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a280ad700 con 0x7f8a280ba4a0 2021-02-10T11:31:32.482 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:32.491 7f8a1effd700 1 -- 172.21.2.112:0/2925680021 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8a180022b0 con 0x7f8a280ba4a0 2021-02-10T11:31:32.482 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:32.491 7f8a1effd700 1 -- 172.21.2.112:0/2925680021 <== mon.2 v2:172.21.2.120:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8a18009fb0 con 0x7f8a280ba4a0 2021-02-10T11:31:32.483 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:32.492 7f8a1effd700 1 --2- 172.21.2.112:0/2925680021 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8a0c018730 0x7f8a0c01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:32.483 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:32.492 7f8a1effd700 1 -- 172.21.2.112:0/2925680021 <== mon.2 v2:172.21.2.120:3301/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5659+0+0 (secure 0 0 0) 0x7f8a18032800 con 0x7f8a280ba4a0 2021-02-10T11:31:32.483 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:32.492 7f8a2661c700 1 --2- 172.21.2.112:0/2925680021 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8a0c018730 0x7f8a0c01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:32.483 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:32.492 7f8a2661c700 1 --2- 172.21.2.112:0/2925680021 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8a0c018730 0x7f8a0c01ac10 crc :-1 s=READY pgs=503 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:32.510 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:32.523 7f8a2c961700 1 -- 172.21.2.112:0/2925680021 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a280828e0 con 0x7f8a280ba4a0 2021-02-10T11:31:32.514 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:32.526 7f8a1effd700 1 -- 172.21.2.112:0/2925680021 <== mon.2 v2:172.21.2.120:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8a18006b90 con 0x7f8a280ba4a0 2021-02-10T11:31:32.645 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:32.658 7f8a2c961700 1 -- 172.21.2.112:0/2925680021 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_command({"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f8a280828e0 con 0x7f8a280ba4a0 2021-02-10T11:31:33.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:33.053 7f8a1effd700 1 -- 172.21.2.112:0/2925680021 <== mon.2 v2:172.21.2.120:3301/0 7 ==== mon_command_ack([{"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v31) v1 ==== 185+0+0 (secure 0 0 0) 0x7f8a18000c90 con 0x7f8a280ba4a0 2021-02-10T11:31:33.062 INFO:teuthology.orchestra.run.gibba012.stderr:pool 'cephfs_metadata' removed 2021-02-10T11:31:33.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:33.077 7f8a2c961700 1 -- 172.21.2.112:0/2925680021 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8a0c018730 msgr2=0x7f8a0c01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:33.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:33.077 7f8a2c961700 1 --2- 172.21.2.112:0/2925680021 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8a0c018730 0x7f8a0c01ac10 crc :-1 s=READY pgs=503 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:33.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:33.077 7f8a2c961700 1 -- 172.21.2.112:0/2925680021 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8a280ba4a0 msgr2=0x7f8a280ba8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:33.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:33.077 7f8a2c961700 1 --2- 172.21.2.112:0/2925680021 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8a280ba4a0 0x7f8a280ba8a0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f8a18006f00 tx=0x7f8a1800b940).stop 2021-02-10T11:31:33.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:33.077 7f8a2c961700 1 -- 172.21.2.112:0/2925680021 shutdown_connections 2021-02-10T11:31:33.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:33.077 7f8a2c961700 1 --2- 172.21.2.112:0/2925680021 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8a0c018730 0x7f8a0c01ac10 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:33.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:33.077 7f8a2c961700 1 --2- 172.21.2.112:0/2925680021 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8a281a2fd0 0x7f8a281a33d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:33.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:33.077 7f8a2c961700 1 --2- 172.21.2.112:0/2925680021 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8a280ba4a0 0x7f8a280ba8a0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:33.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:33.077 7f8a2c961700 1 --2- 172.21.2.112:0/2925680021 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8a280b8070 0x7f8a280b77c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:33.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:33.077 7f8a2c961700 1 -- 172.21.2.112:0/2925680021 >> 172.21.2.112:0/2925680021 conn(0x7f8a280b58d0 msgr2=0x7f8a280b5010 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:33.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:33.077 7f8a2c961700 1 -- 172.21.2.112:0/2925680021 shutdown_connections 2021-02-10T11:31:33.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:33.078 7f8a2c961700 1 -- 172.21.2.112:0/2925680021 wait complete. 2021-02-10T11:31:33.073 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2021-02-10T11:31:33.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:33.499 7f7837942700 1 Processor -- start 2021-02-10T11:31:33.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:33.499 7f7837942700 1 -- start start 2021-02-10T11:31:33.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:33.499 7f7837942700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f78300d1870 0x7f78301b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:33.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:33.499 7f7837942700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f78301b0ae0 0x7f7830108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:33.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:33.499 7f7837942700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f78300d1c70 0x7f7830108f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:33.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:33.499 7f7837942700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78300fe1c0 con 0x7f78300d1c70 2021-02-10T11:31:33.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:33.499 7f7837942700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7830109a60 con 0x7f78300d1870 2021-02-10T11:31:33.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:33.499 7f7837942700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7830088020 con 0x7f78301b0ae0 2021-02-10T11:31:33.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:33.500 7f7835f5f700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f78300d1c70 0x7f7830108f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:33.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:33.500 7f7835f5f700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f78300d1c70 0x7f7830108f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:48108/0 (socket says 172.21.2.112:48108) 2021-02-10T11:31:33.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:33.500 7f7835f5f700 1 -- 172.21.2.112:0/2172240915 learned_addr learned my addr 172.21.2.112:0/2172240915 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:31:33.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:33.500 7f7835f5f700 1 -- 172.21.2.112:0/2172240915 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f78301b0ae0 msgr2=0x7f7830108940 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:31:33.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:33.500 7f7835f5f700 1 --2- 172.21.2.112:0/2172240915 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f78301b0ae0 0x7f7830108940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:33.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:33.500 7f7835f5f700 1 -- 172.21.2.112:0/2172240915 >> v2:172.21.2.120:3300/0 conn(0x7f78300d1870 msgr2=0x7f78301b04c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:31:33.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:33.500 7f7835f5f700 1 --2- 172.21.2.112:0/2172240915 >> v2:172.21.2.120:3300/0 conn(0x7f78300d1870 0x7f78301b04c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:33.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:33.500 7f7835f5f700 1 -- 172.21.2.112:0/2172240915 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7830109580 con 0x7f78300d1c70 2021-02-10T11:31:33.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:33.500 7f7835f5f700 1 --2- 172.21.2.112:0/2172240915 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f78300d1c70 0x7f7830108f60 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 rx=0x7f7828006f00 tx=0x7f7828009f10).ready entity=mon.0 client_cookie=25ab78d92268a3aa server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:33.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:33.500 7f7823fff700 1 -- 172.21.2.112:0/2172240915 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f782800b3b0 con 0x7f78300d1c70 2021-02-10T11:31:33.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:33.500 7f7823fff700 1 -- 172.21.2.112:0/2172240915 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f782800b630 con 0x7f78300d1c70 2021-02-10T11:31:33.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:33.500 7f7823fff700 1 -- 172.21.2.112:0/2172240915 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f782800bb50 con 0x7f78300d1c70 2021-02-10T11:31:33.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:33.500 7f7837942700 1 -- 172.21.2.112:0/2172240915 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f78300d1c70 msgr2=0x7f7830108f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:33.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:33.500 7f7837942700 1 --2- 172.21.2.112:0/2172240915 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f78300d1c70 0x7f7830108f60 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 rx=0x7f7828006f00 tx=0x7f7828009f10).stop 2021-02-10T11:31:33.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:33.501 7f7837942700 1 -- 172.21.2.112:0/2172240915 shutdown_connections 2021-02-10T11:31:33.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:33.501 7f7837942700 1 --2- 172.21.2.112:0/2172240915 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f78300d1c70 0x7f7830108f60 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:33.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:33.501 7f7837942700 1 --2- 172.21.2.112:0/2172240915 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f78301b0ae0 0x7f7830108940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:33.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:33.501 7f7837942700 1 --2- 172.21.2.112:0/2172240915 >> v2:172.21.2.120:3300/0 conn(0x7f78300d1870 0x7f78301b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:33.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:33.501 7f7837942700 1 -- 172.21.2.112:0/2172240915 >> 172.21.2.112:0/2172240915 conn(0x7f78300ce420 msgr2=0x7f78300d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:33.505 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:33.501 7f7837942700 1 -- 172.21.2.112:0/2172240915 shutdown_connections 2021-02-10T11:31:33.505 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:33.501 7f7837942700 1 -- 172.21.2.112:0/2172240915 wait complete. 2021-02-10T11:31:33.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:33.515 7f7837942700 1 Processor -- start 2021-02-10T11:31:33.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:33.516 7f7837942700 1 -- start start 2021-02-10T11:31:33.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:33.516 7f7837942700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f78300aaa20 0x7f78301af940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:33.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:33.516 7f7837942700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f783019bc20 0x7f783019c020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:33.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:33.516 7f7837942700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f783019cd90 0x7f783019d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:33.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:33.516 7f7837942700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f783019c810 con 0x7f783019cd90 2021-02-10T11:31:33.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:33.516 7f7837942700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78301b01f0 con 0x7f783019bc20 2021-02-10T11:31:33.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:33.516 7f7837942700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78301a2a80 con 0x7f78300aaa20 2021-02-10T11:31:33.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:33.516 7f7835f5f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f783019cd90 0x7f783019d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:33.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:33.516 7f7835f5f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f783019cd90 0x7f783019d190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:48116/0 (socket says 172.21.2.112:48116) 2021-02-10T11:31:33.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:33.516 7f7835f5f700 1 -- 172.21.2.112:0/1901240889 learned_addr learned my addr 172.21.2.112:0/1901240889 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:31:33.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:33.517 7f7835f5f700 1 -- 172.21.2.112:0/1901240889 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f78300aaa20 msgr2=0x7f78301af940 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:31:33.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:33.517 7f7835f5f700 1 --2- 172.21.2.112:0/1901240889 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f78300aaa20 0x7f78301af940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:33.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:33.517 7f7835f5f700 1 -- 172.21.2.112:0/1901240889 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f783019bc20 msgr2=0x7f783019c020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:33.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:33.517 7f7835f5f700 1 --2- 172.21.2.112:0/1901240889 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f783019bc20 0x7f783019c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:33.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:33.517 7f7835f5f700 1 -- 172.21.2.112:0/1901240889 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f783019d7b0 con 0x7f783019cd90 2021-02-10T11:31:33.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:33.517 7f7835f5f700 1 --2- 172.21.2.112:0/1901240889 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f783019cd90 0x7f783019d190 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 rx=0x7f7828009390 tx=0x7f782800bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:33.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:33.517 7f7821ffb700 1 -- 172.21.2.112:0/1901240889 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7828003030 con 0x7f783019cd90 2021-02-10T11:31:33.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:33.517 7f7821ffb700 1 -- 172.21.2.112:0/1901240889 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78280068e0 con 0x7f783019cd90 2021-02-10T11:31:33.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:33.517 7f7821ffb700 1 -- 172.21.2.112:0/1901240889 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f782800c570 con 0x7f783019cd90 2021-02-10T11:31:33.508 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:33.517 7f7837942700 1 -- 172.21.2.112:0/1901240889 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f783019d7b0 con 0x7f783019cd90 2021-02-10T11:31:33.508 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:33.517 7f7837942700 1 -- 172.21.2.112:0/1901240889 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f783019dd40 con 0x7f783019cd90 2021-02-10T11:31:33.508 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:33.518 7f7821ffb700 1 -- 172.21.2.112:0/1901240889 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f78280011d0 con 0x7f783019cd90 2021-02-10T11:31:33.508 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:33.518 7f7821ffb700 1 --2- 172.21.2.112:0/1901240889 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f78100186c0 0x7f781001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:33.508 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:33.518 7f7821ffb700 1 -- 172.21.2.112:0/1901240889 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5221+0+0 (secure 0 0 0) 0x7f7828032780 con 0x7f783019cd90 2021-02-10T11:31:33.508 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:33.518 7f783575e700 1 --2- 172.21.2.112:0/1901240889 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f78100186c0 0x7f781001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:33.508 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:33.518 7f783575e700 1 --2- 172.21.2.112:0/1901240889 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f78100186c0 0x7f781001abc0 crc :-1 s=READY pgs=504 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:33.536 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:33.549 7f7837942700 1 -- 172.21.2.112:0/1901240889 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7830082920 con 0x7f783019cd90 2021-02-10T11:31:33.539 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:33.552 7f7821ffb700 1 -- 172.21.2.112:0/1901240889 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7828009450 con 0x7f783019cd90 2021-02-10T11:31:33.666 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:33.679 7f7837942700 1 -- 172.21.2.112:0/1901240889 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f7830082920 con 0x7f783019cd90 2021-02-10T11:31:34.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:34.057 7f7821ffb700 1 -- 172.21.2.112:0/1901240889 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v32) v1 ==== 173+0+0 (secure 0 0 0) 0x7f782801f620 con 0x7f783019cd90 2021-02-10T11:31:34.082 INFO:teuthology.orchestra.run.gibba012.stderr:pool 'cephfs_data' removed 2021-02-10T11:31:34.084 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:34.097 7f7837942700 1 -- 172.21.2.112:0/1901240889 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f78100186c0 msgr2=0x7f781001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:34.084 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:34.097 7f7837942700 1 --2- 172.21.2.112:0/1901240889 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f78100186c0 0x7f781001abc0 crc :-1 s=READY pgs=504 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:34.084 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:34.097 7f7837942700 1 -- 172.21.2.112:0/1901240889 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f783019cd90 msgr2=0x7f783019d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:34.084 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:34.097 7f7837942700 1 --2- 172.21.2.112:0/1901240889 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f783019cd90 0x7f783019d190 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 rx=0x7f7828009390 tx=0x7f782800bd40).stop 2021-02-10T11:31:34.085 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:34.097 7f7837942700 1 -- 172.21.2.112:0/1901240889 shutdown_connections 2021-02-10T11:31:34.085 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:34.097 7f7837942700 1 --2- 172.21.2.112:0/1901240889 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f78100186c0 0x7f781001abc0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:34.085 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:34.097 7f7837942700 1 --2- 172.21.2.112:0/1901240889 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f783019bc20 0x7f783019c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:34.085 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:34.097 7f7837942700 1 --2- 172.21.2.112:0/1901240889 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f783019cd90 0x7f783019d190 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:34.085 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:34.097 7f7837942700 1 --2- 172.21.2.112:0/1901240889 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f78300aaa20 0x7f78301af940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:34.085 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:34.097 7f7837942700 1 -- 172.21.2.112:0/1901240889 >> 172.21.2.112:0/1901240889 conn(0x7f7830084530 msgr2=0x7f783019ad40 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:34.085 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:34.097 7f7837942700 1 -- 172.21.2.112:0/1901240889 shutdown_connections 2021-02-10T11:31:34.085 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:34.097 7f7837942700 1 -- 172.21.2.112:0/1901240889 wait complete. 2021-02-10T11:31:34.093 INFO:tasks.ceph.mds.a:Restarting daemon 2021-02-10T11:31:34.093 INFO:tasks.ceph.mds.a:Stopping old one... 2021-02-10T11:31:34.093 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2021-02-10T11:31:34.093 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:31:34.094 INFO:tasks.ceph.mds.b:Restarting daemon 2021-02-10T11:31:34.094 INFO:tasks.ceph.mds.b:Stopping old one... 2021-02-10T11:31:34.094 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2021-02-10T11:31:34.094 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:31:34.099 INFO:tasks.ceph.mds.a:Stopped 2021-02-10T11:31:34.099 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2021-02-10T11:31:34.100 INFO:tasks.ceph.mds.b:Stopped 2021-02-10T11:31:34.100 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2021-02-10T11:31:34.140 INFO:tasks.ceph.mds.a:Started 2021-02-10T11:31:34.141 DEBUG:teuthology.parallel:result is None 2021-02-10T11:31:34.141 INFO:tasks.ceph.mds.b:Started 2021-02-10T11:31:34.141 DEBUG:teuthology.parallel:result is None 2021-02-10T11:31:34.142 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd blacklist clear 2021-02-10T11:31:34.545 INFO:tasks.ceph.mds.b.gibba020.stdout:starting mds.b at 2021-02-10T11:31:34.548 INFO:tasks.ceph.mds.a.gibba012.stdout:starting mds.a at 2021-02-10T11:31:34.570 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:34.566 7f983469f700 1 Processor -- start 2021-02-10T11:31:34.570 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:34.567 7f983469f700 1 -- start start 2021-02-10T11:31:34.570 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:34.567 7f983469f700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f982c0e5070 0x7f982c1b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:34.570 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:34.567 7f983469f700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f982c07ed80 con 0x7f982c1b3e40 2021-02-10T11:31:34.570 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:34.567 7f983469f700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f982c07e670 con 0x7f982c0e5470 2021-02-10T11:31:34.570 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:34.567 7f983469f700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f982c07e890 con 0x7f982c0e5070 2021-02-10T11:31:34.570 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:34.567 7f98324bb700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f982c0e5070 0x7f982c1b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:34.570 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:34.567 7f9831cba700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f982c1b3e40 0x7f982c07d690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:47600/0 (socket says 172.21.2.112:47600) 2021-02-10T11:31:34.570 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:34.567 7f9831cba700 1 -- 172.21.2.112:0/1284263727 learned_addr learned my addr 172.21.2.112:0/1284263727 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:31:34.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:34.567 7f98324bb700 1 -- 172.21.2.112:0/1284263727 >> v1:172.21.2.120:6789/0 conn(0x7f982c0e5470 legacy=0x7f982c07de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:34.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:34.567 7f98324bb700 1 -- 172.21.2.112:0/1284263727 >> v1:172.21.2.112:6789/0 conn(0x7f982c1b3e40 legacy=0x7f982c07d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:34.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:34.567 7f98324bb700 1 -- 172.21.2.112:0/1284263727 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f982c07eab0 con 0x7f982c0e5070 2021-02-10T11:31:34.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:34.568 7f98324bb700 1 --2- 172.21.2.112:0/1284263727 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f982c0e5070 0x7f982c1b3820 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 rx=0x7f9828006650 tx=0x7f9828009a80).ready entity=mon.0 client_cookie=65357a069ee84970 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:34.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:34.568 7f98314b9700 1 -- 172.21.2.112:0/1284263727 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f98280009b0 con 0x7f982c0e5070 2021-02-10T11:31:34.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:34.568 7f98314b9700 1 -- 172.21.2.112:0/1284263727 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f982800c0c0 con 0x7f982c0e5070 2021-02-10T11:31:34.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:34.568 7f98314b9700 1 -- 172.21.2.112:0/1284263727 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f982800c510 con 0x7f982c0e5070 2021-02-10T11:31:34.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:34.568 7f983469f700 1 -- 172.21.2.112:0/1284263727 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f982c0e5070 msgr2=0x7f982c1b3820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:34.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:34.568 7f983469f700 1 --2- 172.21.2.112:0/1284263727 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f982c0e5070 0x7f982c1b3820 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 rx=0x7f9828006650 tx=0x7f9828009a80).stop 2021-02-10T11:31:34.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:34.568 7f983469f700 1 -- 172.21.2.112:0/1284263727 shutdown_connections 2021-02-10T11:31:34.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:34.568 7f983469f700 1 --2- 172.21.2.112:0/1284263727 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f982c0e5070 0x7f982c1b3820 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:34.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:34.568 7f983469f700 1 -- 172.21.2.112:0/1284263727 >> 172.21.2.112:0/1284263727 conn(0x7f982c0ce690 msgr2=0x7f982c0e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:34.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:34.568 7f983469f700 1 -- 172.21.2.112:0/1284263727 shutdown_connections 2021-02-10T11:31:34.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:34.568 7f983469f700 1 -- 172.21.2.112:0/1284263727 wait complete. 2021-02-10T11:31:34.573 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:34.583 7f983469f700 1 Processor -- start 2021-02-10T11:31:34.573 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:34.583 7f983469f700 1 -- start start 2021-02-10T11:31:34.573 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:34.583 7f983469f700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f982c08af20 0x7f982c08a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:34.573 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:34.583 7f983469f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f982c0c0170 0x7f982c0c0570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:34.573 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:34.583 7f983469f700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f982c0c0b90 0x7f982c0a1660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:34.573 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:34.583 7f983469f700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f982c0917e0 con 0x7f982c0c0170 2021-02-10T11:31:34.573 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:34.583 7f983469f700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f982c08b360 con 0x7f982c08af20 2021-02-10T11:31:34.573 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:34.583 7f983469f700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f982c1c2510 con 0x7f982c0c0b90 2021-02-10T11:31:34.574 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:34.583 7f9831cba700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f982c0c0170 0x7f982c0c0570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:34.574 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:34.584 7f9831cba700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f982c0c0170 0x7f982c0c0570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:48144/0 (socket says 172.21.2.112:48144) 2021-02-10T11:31:34.574 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:34.584 7f9831cba700 1 -- 172.21.2.112:0/3103548918 learned_addr learned my addr 172.21.2.112:0/3103548918 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:31:34.574 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:34.584 7f9831cba700 1 -- 172.21.2.112:0/3103548918 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f982c0c0b90 msgr2=0x7f982c0a1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:34.574 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:34.584 7f9831cba700 1 --2- 172.21.2.112:0/3103548918 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f982c0c0b90 0x7f982c0a1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:34.574 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:34.584 7f9831cba700 1 -- 172.21.2.112:0/3103548918 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f982c08af20 msgr2=0x7f982c08a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:34.574 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:34.584 7f9831cba700 1 --2- 172.21.2.112:0/3103548918 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f982c08af20 0x7f982c08a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:34.574 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:34.584 7f9831cba700 1 -- 172.21.2.112:0/3103548918 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f982c08aa50 con 0x7f982c0c0170 2021-02-10T11:31:34.574 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:34.584 7f9831cba700 1 --2- 172.21.2.112:0/3103548918 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f982c0c0170 0x7f982c0c0570 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 rx=0x7f981c001400 tx=0x7f981c00a900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:34.574 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:34.584 7f9822ffd700 1 -- 172.21.2.112:0/3103548918 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f981c00acf0 con 0x7f982c0c0170 2021-02-10T11:31:34.574 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:34.584 7f9822ffd700 1 -- 172.21.2.112:0/3103548918 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f981c000c90 con 0x7f982c0c0170 2021-02-10T11:31:34.574 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:34.584 7f9822ffd700 1 -- 172.21.2.112:0/3103548918 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f981c004590 con 0x7f982c0c0170 2021-02-10T11:31:34.574 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:34.584 7f983469f700 1 -- 172.21.2.112:0/3103548918 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f982c08aa50 con 0x7f982c0c0170 2021-02-10T11:31:34.575 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:34.584 7f983469f700 1 -- 172.21.2.112:0/3103548918 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f982c08aa50 con 0x7f982c0c0170 2021-02-10T11:31:34.575 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:34.585 7f9822ffd700 1 -- 172.21.2.112:0/3103548918 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f981c0043a0 con 0x7f982c0c0170 2021-02-10T11:31:34.575 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:34.585 7f9822ffd700 1 --2- 172.21.2.112:0/3103548918 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9814018730 0x7f981401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:34.575 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:34.585 7f9822ffd700 1 -- 172.21.2.112:0/3103548918 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 4839+0+0 (secure 0 0 0) 0x7f981c032330 con 0x7f982c0c0170 2021-02-10T11:31:34.575 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:34.585 7f98324bb700 1 --2- 172.21.2.112:0/3103548918 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9814018730 0x7f981401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:34.575 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:34.585 7f98324bb700 1 --2- 172.21.2.112:0/3103548918 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9814018730 0x7f981401abc0 crc :-1 s=READY pgs=505 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:34.606 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:34.615 7f983469f700 1 -- 172.21.2.112:0/3103548918 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f982c0897d0 con 0x7f982c0c0170 2021-02-10T11:31:34.606 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:34.618 7f9822ffd700 1 -- 172.21.2.112:0/3103548918 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f981c007070 con 0x7f982c0c0170 2021-02-10T11:31:34.744 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:34.757 7f983469f700 1 -- 172.21.2.112:0/3103548918 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7f982c0ce690 con 0x7f982c0c0170 2021-02-10T11:31:35.088 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:35.100 7f9822ffd700 1 -- 172.21.2.112:0/3103548918 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v33) v1 ==== 97+0+0 (secure 0 0 0) 0x7f981c0043a0 con 0x7f982c0c0170 2021-02-10T11:31:35.110 INFO:teuthology.orchestra.run.gibba012.stderr: removed all blacklist entries 2021-02-10T11:31:35.112 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:35.124 7f983469f700 1 -- 172.21.2.112:0/3103548918 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9814018730 msgr2=0x7f981401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:35.112 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:35.124 7f983469f700 1 --2- 172.21.2.112:0/3103548918 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9814018730 0x7f981401abc0 crc :-1 s=READY pgs=505 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:35.112 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:35.124 7f983469f700 1 -- 172.21.2.112:0/3103548918 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f982c0c0170 msgr2=0x7f982c0c0570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:35.112 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:35.124 7f983469f700 1 --2- 172.21.2.112:0/3103548918 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f982c0c0170 0x7f982c0c0570 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 rx=0x7f981c001400 tx=0x7f981c00a900).stop 2021-02-10T11:31:35.113 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:35.125 7f983469f700 1 -- 172.21.2.112:0/3103548918 shutdown_connections 2021-02-10T11:31:35.113 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:35.125 7f983469f700 1 --2- 172.21.2.112:0/3103548918 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9814018730 0x7f981401abc0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:35.113 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:35.125 7f983469f700 1 --2- 172.21.2.112:0/3103548918 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f982c0c0b90 0x7f982c0a1660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:35.113 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:35.125 7f983469f700 1 --2- 172.21.2.112:0/3103548918 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f982c0c0170 0x7f982c0c0570 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:35.113 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:35.125 7f983469f700 1 --2- 172.21.2.112:0/3103548918 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f982c08af20 0x7f982c08a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:35.113 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:35.125 7f983469f700 1 -- 172.21.2.112:0/3103548918 >> 172.21.2.112:0/3103548918 conn(0x7f982c089a50 msgr2=0x7f982c094b60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:35.113 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:35.125 7f983469f700 1 -- 172.21.2.112:0/3103548918 shutdown_connections 2021-02-10T11:31:35.113 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:35.125 7f983469f700 1 -- 172.21.2.112:0/3103548918 wait complete. 2021-02-10T11:31:35.120 INFO:tasks.cephfs.cephfs_test_case:['0', '1', '2', '3'] 2021-02-10T11:31:35.121 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth ls --format=json-pretty 2021-02-10T11:31:35.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:35.543 7f6296665700 1 Processor -- start 2021-02-10T11:31:35.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:35.543 7f6296665700 1 -- start start 2021-02-10T11:31:35.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:35.544 7f6296665700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f62900e5470 0x7f629007e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:35.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:35.544 7f6296665700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f629007ed80 con 0x7f629007f300 2021-02-10T11:31:35.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:35.544 7f6296665700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62901b4010 con 0x7f62900e5070 2021-02-10T11:31:35.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:35.544 7f6296665700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62900d1790 con 0x7f62900e5470 2021-02-10T11:31:35.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:35.544 7f628f7fe700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f62900e5470 0x7f629007e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:35.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:35.544 7f628f7fe700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f62900e5470 0x7f629007e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3300/0 says I am v2:172.21.2.112:50428/0 (socket says 172.21.2.112:50428) 2021-02-10T11:31:35.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:35.544 7f628f7fe700 1 -- 172.21.2.112:0/924061717 learned_addr learned my addr 172.21.2.112:0/924061717 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:31:35.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:35.544 7f628f7fe700 1 -- 172.21.2.112:0/924061717 >> v1:172.21.2.120:6789/0 conn(0x7f62900e5070 legacy=0x7f62901b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:35.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:35.544 7f628f7fe700 1 -- 172.21.2.112:0/924061717 >> v1:172.21.2.112:6789/0 conn(0x7f629007f300 legacy=0x7f629007f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:35.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:35.544 7f628f7fe700 1 -- 172.21.2.112:0/924061717 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62900d19b0 con 0x7f62900e5470 2021-02-10T11:31:35.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:35.545 7f628f7fe700 1 --2- 172.21.2.112:0/924061717 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f62900e5470 0x7f629007e6a0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7f627c006700 tx=0x7f627c009b80).ready entity=mon.1 client_cookie=542961ff011d2da3 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:35.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:35.545 7f628effd700 1 -- 172.21.2.112:0/924061717 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f627c00c0c0 con 0x7f62900e5470 2021-02-10T11:31:35.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:35.545 7f628effd700 1 -- 172.21.2.112:0/924061717 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f627c00c440 con 0x7f62900e5470 2021-02-10T11:31:35.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:35.545 7f628effd700 1 -- 172.21.2.112:0/924061717 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f627c00c960 con 0x7f62900e5470 2021-02-10T11:31:35.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:35.545 7f6296665700 1 -- 172.21.2.112:0/924061717 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f62900e5470 msgr2=0x7f629007e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:35.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:35.545 7f6296665700 1 --2- 172.21.2.112:0/924061717 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f62900e5470 0x7f629007e6a0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7f627c006700 tx=0x7f627c009b80).stop 2021-02-10T11:31:35.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:35.545 7f6296665700 1 -- 172.21.2.112:0/924061717 shutdown_connections 2021-02-10T11:31:35.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:35.545 7f6296665700 1 --2- 172.21.2.112:0/924061717 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f62900e5470 0x7f629007e6a0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:35.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:35.545 7f6296665700 1 -- 172.21.2.112:0/924061717 >> 172.21.2.112:0/924061717 conn(0x7f62900ce690 msgr2=0x7f62900e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:35.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:35.545 7f6296665700 1 -- 172.21.2.112:0/924061717 shutdown_connections 2021-02-10T11:31:35.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:35.545 7f6296665700 1 -- 172.21.2.112:0/924061717 wait complete. 2021-02-10T11:31:35.550 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:35.560 7f6296665700 1 Processor -- start 2021-02-10T11:31:35.550 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:35.560 7f6296665700 1 -- start start 2021-02-10T11:31:35.550 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:35.561 7f6296665700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f62900befe0 0x7f62900bf3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:35.550 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:35.561 7f6296665700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6290094dd0 0x7f62900951d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:35.550 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:35.561 7f6296665700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f62900bfa70 0x7f62900bfe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:35.550 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:35.561 7f6296665700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6290095b30 con 0x7f62900befe0 2021-02-10T11:31:35.550 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:35.561 7f6296665700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6290095830 con 0x7f62900bfa70 2021-02-10T11:31:35.550 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:35.561 7f6296665700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62900a19e0 con 0x7f6290094dd0 2021-02-10T11:31:35.550 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:35.561 7f628ffff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f62900befe0 0x7f62900bf3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:35.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:35.561 7f628ffff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f62900befe0 0x7f62900bf3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:48156/0 (socket says 172.21.2.112:48156) 2021-02-10T11:31:35.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:35.561 7f628ffff700 1 -- 172.21.2.112:0/1339716273 learned_addr learned my addr 172.21.2.112:0/1339716273 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:31:35.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:35.561 7f628f7fe700 1 --2- 172.21.2.112:0/1339716273 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6290094dd0 0x7f62900951d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:35.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:35.561 7f6294c82700 1 --2- 172.21.2.112:0/1339716273 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f62900bfa70 0x7f62900bfe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:35.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:35.561 7f628ffff700 1 -- 172.21.2.112:0/1339716273 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6290094dd0 msgr2=0x7f62900951d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:35.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:35.561 7f628ffff700 1 --2- 172.21.2.112:0/1339716273 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6290094dd0 0x7f62900951d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:35.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:35.561 7f628ffff700 1 -- 172.21.2.112:0/1339716273 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f62900bfa70 msgr2=0x7f62900bfe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:35.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:35.561 7f628ffff700 1 --2- 172.21.2.112:0/1339716273 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f62900bfa70 0x7f62900bfe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:35.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:35.561 7f628ffff700 1 -- 172.21.2.112:0/1339716273 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62900c0490 con 0x7f62900befe0 2021-02-10T11:31:35.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:35.562 7f628f7fe700 1 --2- 172.21.2.112:0/1339716273 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6290094dd0 0x7f62900951d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:31:35.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:35.562 7f6294c82700 1 --2- 172.21.2.112:0/1339716273 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f62900bfa70 0x7f62900bfe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:31:35.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:35.562 7f628ffff700 1 --2- 172.21.2.112:0/1339716273 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f62900befe0 0x7f62900bf3e0 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 rx=0x7f628000b5b0 tx=0x7f628000b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:35.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:35.562 7f628cff9700 1 -- 172.21.2.112:0/1339716273 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f628000bd40 con 0x7f62900befe0 2021-02-10T11:31:35.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:35.562 7f628cff9700 1 -- 172.21.2.112:0/1339716273 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6280006020 con 0x7f62900befe0 2021-02-10T11:31:35.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:35.562 7f6296665700 1 -- 172.21.2.112:0/1339716273 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62900c0490 con 0x7f62900befe0 2021-02-10T11:31:35.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:35.562 7f628cff9700 1 -- 172.21.2.112:0/1339716273 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f62800054a0 con 0x7f62900befe0 2021-02-10T11:31:35.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:35.562 7f6296665700 1 -- 172.21.2.112:0/1339716273 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62900c0a20 con 0x7f62900befe0 2021-02-10T11:31:35.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:35.563 7f628cff9700 1 -- 172.21.2.112:0/1339716273 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6280006200 con 0x7f62900befe0 2021-02-10T11:31:35.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:35.563 7f628cff9700 1 --2- 172.21.2.112:0/1339716273 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f626c0186c0 0x7f626c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:35.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:35.563 7f628cff9700 1 -- 172.21.2.112:0/1339716273 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f6280032240 con 0x7f62900befe0 2021-02-10T11:31:35.553 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:35.563 7f628f7fe700 1 --2- 172.21.2.112:0/1339716273 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f626c0186c0 0x7f626c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:35.553 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:35.564 7f628f7fe700 1 --2- 172.21.2.112:0/1339716273 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f626c0186c0 0x7f626c01abc0 crc :-1 s=READY pgs=506 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:35.581 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:35.594 7f6296665700 1 -- 172.21.2.112:0/1339716273 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62900ce690 con 0x7f62900befe0 2021-02-10T11:31:35.583 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:35.596 7f628cff9700 1 -- 172.21.2.112:0/1339716273 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6280000af0 con 0x7f62900befe0 2021-02-10T11:31:35.717 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:35.730 7f6296665700 1 -- 172.21.2.112:0/1339716273 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f62900ce690 con 0x7f62900befe0 2021-02-10T11:31:35.718 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:35.730 7f628cff9700 1 -- 172.21.2.112:0/1339716273 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v16) v1 ==== 80+0+4513 (secure 0 0 0) 0x7f628000bd40 con 0x7f62900befe0 2021-02-10T11:31:35.718 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:31:35.718 INFO:teuthology.orchestra.run.gibba012.stdout:{ 2021-02-10T11:31:35.718 INFO:teuthology.orchestra.run.gibba012.stdout: "auth_dump": [ 2021-02-10T11:31:35.718 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:31:35.718 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "mds.a", 2021-02-10T11:31:35.718 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCKvyNgPkV4AxAAEmVNaghdccP6lWd8hvQnxw==", 2021-02-10T11:31:35.719 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:31:35.719 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:31:35.719 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:31:35.719 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:31:35.719 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:31:35.719 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:31:35.719 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:31:35.719 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:31:35.719 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "mds.b", 2021-02-10T11:31:35.719 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCKvyNguxONLhAAHIpry+R89MAwdhFJyEhVCw==", 2021-02-10T11:31:35.719 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:31:35.719 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:31:35.720 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:31:35.720 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:31:35.720 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:31:35.720 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:31:35.720 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:31:35.720 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:31:35.720 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.0", 2021-02-10T11:31:35.720 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCSvyNgXcKPLhAAP/Bj2F+06DCtPCzQDNq+cQ==", 2021-02-10T11:31:35.720 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:31:35.720 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:31:35.720 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:31:35.720 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:31:35.720 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:31:35.721 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:31:35.721 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:31:35.721 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.1", 2021-02-10T11:31:35.721 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCVvyNghfbcLhAA66PrRwR2M7o6VTF5UhU/Fg==", 2021-02-10T11:31:35.721 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:31:35.721 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:31:35.721 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:31:35.721 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:31:35.721 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:31:35.721 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:31:35.721 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:31:35.721 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.2", 2021-02-10T11:31:35.721 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCYvyNgwiPqLhAA66BVkRsgaUlGpAsQ15Nf8Q==", 2021-02-10T11:31:35.721 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:31:35.722 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:31:35.722 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:31:35.722 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:31:35.722 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:31:35.722 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:31:35.722 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:31:35.722 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.3", 2021-02-10T11:31:35.722 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCbvyNgqBZrLxAAERPe/1L3SXAEzI3d9ReHHg==", 2021-02-10T11:31:35.722 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:31:35.722 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:31:35.722 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:31:35.722 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:31:35.722 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:31:35.723 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:31:35.723 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:31:35.723 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.4", 2021-02-10T11:31:35.723 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCkvyNgDQsxIhAAHmVjVYnx20N4SDmKqvbcuA==", 2021-02-10T11:31:35.723 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:31:35.723 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:31:35.723 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:31:35.723 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:31:35.723 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:31:35.723 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:31:35.723 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:31:35.723 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.5", 2021-02-10T11:31:35.723 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCnvyNg8ITFIhAADd4aKybAV7btoGPx3kCg3A==", 2021-02-10T11:31:35.723 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:31:35.724 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:31:35.724 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:31:35.724 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:31:35.724 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:31:35.724 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:31:35.724 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:31:35.724 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.6", 2021-02-10T11:31:35.724 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCqvyNgxOhDIxAAzXixEa/nlDc1xXrg0aeLwg==", 2021-02-10T11:31:35.724 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:31:35.724 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:31:35.724 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:31:35.724 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:31:35.724 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:31:35.725 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:31:35.725 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:31:35.725 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.7", 2021-02-10T11:31:35.725 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCtvyNgxSqGIxAACxJQlwP7DpiVVCZHA3zn1A==", 2021-02-10T11:31:35.725 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:31:35.725 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:31:35.725 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:31:35.725 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:31:35.725 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:31:35.725 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:31:35.725 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:31:35.725 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.0", 2021-02-10T11:31:35.725 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCLvyNgDkNiHRAAygU8Za4PBusTqCgwAYWTiQ==", 2021-02-10T11:31:35.726 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:31:35.726 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:31:35.726 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow r", 2021-02-10T11:31:35.726 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:31:35.726 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:31:35.726 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:31:35.726 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:31:35.726 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.1", 2021-02-10T11:31:35.726 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCLvyNgxmLqMxAA1MlSDJ9LH9u4f2sL7mm8/Q==", 2021-02-10T11:31:35.726 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:31:35.726 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:31:35.727 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow r", 2021-02-10T11:31:35.727 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:31:35.727 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:31:35.727 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:31:35.727 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:31:35.727 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.2", 2021-02-10T11:31:35.727 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCMvyNgp8veDhAAniSVHYPtwwTog/pjbaRQmw==", 2021-02-10T11:31:35.727 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:31:35.727 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:31:35.727 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow r", 2021-02-10T11:31:35.727 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:31:35.727 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:31:35.727 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:31:35.727 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:31:35.728 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.3", 2021-02-10T11:31:35.728 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCMvyNgQEVoJRAApHcWdm1sldIxu+heyD4HdQ==", 2021-02-10T11:31:35.728 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:31:35.728 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:31:35.728 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow r", 2021-02-10T11:31:35.728 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:31:35.728 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:31:35.728 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:31:35.728 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:31:35.728 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.admin", 2021-02-10T11:31:35.728 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCGvyNg7jjJBRAAtot+ypUZOiTo1GbY+oVhVw==", 2021-02-10T11:31:35.728 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:31:35.728 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow *", 2021-02-10T11:31:35.728 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:31:35.729 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:31:35.729 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:31:35.729 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:31:35.729 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:31:35.729 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:31:35.729 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "mgr.x", 2021-02-10T11:31:35.729 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCJvyNgTPVwKBAAjduiBE8+RSXBv82tdPecWg==", 2021-02-10T11:31:35.729 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:31:35.729 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow *", 2021-02-10T11:31:35.729 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow profile mgr", 2021-02-10T11:31:35.729 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:31:35.729 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:31:35.729 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:31:35.730 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:31:35.730 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "mgr.y", 2021-02-10T11:31:35.730 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCJvyNgQ1tTDhAASwBssDy+Rz31ujw8fNlWow==", 2021-02-10T11:31:35.730 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:31:35.730 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow *", 2021-02-10T11:31:35.730 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow profile mgr", 2021-02-10T11:31:35.730 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:31:35.730 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:31:35.730 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:31:35.730 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:31:35.730 INFO:teuthology.orchestra.run.gibba012.stdout:} 2021-02-10T11:31:35.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:35.732 7f6296665700 1 -- 172.21.2.112:0/1339716273 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f626c0186c0 msgr2=0x7f626c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:35.732 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:35.732 7f6296665700 1 --2- 172.21.2.112:0/1339716273 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f626c0186c0 0x7f626c01abc0 crc :-1 s=READY pgs=506 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:35.732 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:35.732 7f6296665700 1 -- 172.21.2.112:0/1339716273 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f62900befe0 msgr2=0x7f62900bf3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:35.732 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:35.732 7f6296665700 1 --2- 172.21.2.112:0/1339716273 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f62900befe0 0x7f62900bf3e0 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 rx=0x7f628000b5b0 tx=0x7f628000b950).stop 2021-02-10T11:31:35.732 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:35.732 7f6296665700 1 -- 172.21.2.112:0/1339716273 shutdown_connections 2021-02-10T11:31:35.732 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:35.732 7f6296665700 1 --2- 172.21.2.112:0/1339716273 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f626c0186c0 0x7f626c01abc0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:35.732 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:35.732 7f6296665700 1 --2- 172.21.2.112:0/1339716273 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f62900bfa70 0x7f62900bfe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:35.732 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:35.732 7f6296665700 1 --2- 172.21.2.112:0/1339716273 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6290094dd0 0x7f62900951d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:35.732 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:35.732 7f6296665700 1 --2- 172.21.2.112:0/1339716273 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f62900befe0 0x7f62900bf3e0 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:35.732 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:35.733 7f6296665700 1 -- 172.21.2.112:0/1339716273 >> 172.21.2.112:0/1339716273 conn(0x7f6290106df0 msgr2=0x7f62900962d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:35.732 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:35.733 7f6296665700 1 -- 172.21.2.112:0/1339716273 shutdown_connections 2021-02-10T11:31:35.732 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:35.733 7f6296665700 1 -- 172.21.2.112:0/1339716273 wait complete. 2021-02-10T11:31:35.733 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:31:36.157 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.154 7f15eade6700 1 Processor -- start 2021-02-10T11:31:36.158 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.154 7f15eade6700 1 -- start start 2021-02-10T11:31:36.158 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.154 7f15eade6700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f15e407e6a0 0x7f15e40d1790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:36.158 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.154 7f15eade6700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15e407ed80 con 0x7f15e40e5070 2021-02-10T11:31:36.158 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.154 7f15eade6700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15e407eaa0 con 0x7f15e40e5470 2021-02-10T11:31:36.158 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.154 7f15eade6700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15e41b4080 con 0x7f15e407e6a0 2021-02-10T11:31:36.158 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.154 7f15e9403700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f15e407e6a0 0x7f15e40d1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:36.158 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.154 7f15e8c02700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f15e40e5070 0x7f15e41b3820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:47624/0 (socket says 172.21.2.112:47624) 2021-02-10T11:31:36.158 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.155 7f15e8c02700 1 -- 172.21.2.112:0/1410014635 learned_addr learned my addr 172.21.2.112:0/1410014635 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:31:36.158 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.155 7f15e9403700 1 -- 172.21.2.112:0/1410014635 >> v1:172.21.2.120:6789/0 conn(0x7f15e40e5470 legacy=0x7f15e407f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:36.158 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.155 7f15e9403700 1 -- 172.21.2.112:0/1410014635 >> v1:172.21.2.112:6789/0 conn(0x7f15e40e5070 legacy=0x7f15e41b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:36.158 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.155 7f15e37fe700 1 -- 172.21.2.112:0/1410014635 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 429183816 0 0) 0x7f15cc001100 con 0x7f15e40e5070 2021-02-10T11:31:36.159 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.155 7f15e9403700 1 -- 172.21.2.112:0/1410014635 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15e40d1db0 con 0x7f15e407e6a0 2021-02-10T11:31:36.159 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.155 7f15e9403700 1 --2- 172.21.2.112:0/1410014635 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f15e407e6a0 0x7f15e40d1790 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 rx=0x7f15dc006c30 tx=0x7f15dc009d90).ready entity=mon.0 client_cookie=cf4314e3bd17d959 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:36.159 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.155 7f15e37fe700 1 -- 172.21.2.112:0/1410014635 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f15dc000f90 con 0x7f15e407e6a0 2021-02-10T11:31:36.159 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.155 7f15e37fe700 1 -- 172.21.2.112:0/1410014635 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15dc00b300 con 0x7f15e407e6a0 2021-02-10T11:31:36.159 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.156 7f15e37fe700 1 -- 172.21.2.112:0/1410014635 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f15dc00b7e0 con 0x7f15e407e6a0 2021-02-10T11:31:36.159 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.156 7f15eade6700 1 -- 172.21.2.112:0/1410014635 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f15e407e6a0 msgr2=0x7f15e40d1790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:36.159 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.156 7f15eade6700 1 --2- 172.21.2.112:0/1410014635 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f15e407e6a0 0x7f15e40d1790 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 rx=0x7f15dc006c30 tx=0x7f15dc009d90).stop 2021-02-10T11:31:36.159 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.156 7f15eade6700 1 -- 172.21.2.112:0/1410014635 shutdown_connections 2021-02-10T11:31:36.159 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.156 7f15eade6700 1 --2- 172.21.2.112:0/1410014635 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f15e407e6a0 0x7f15e40d1790 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:36.159 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.156 7f15eade6700 1 -- 172.21.2.112:0/1410014635 >> 172.21.2.112:0/1410014635 conn(0x7f15e40ce690 msgr2=0x7f15e40e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:36.159 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.156 7f15eade6700 1 -- 172.21.2.112:0/1410014635 shutdown_connections 2021-02-10T11:31:36.159 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.156 7f15eade6700 1 -- 172.21.2.112:0/1410014635 wait complete. 2021-02-10T11:31:36.161 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.171 7f15eade6700 1 Processor -- start 2021-02-10T11:31:36.161 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.171 7f15eade6700 1 -- start start 2021-02-10T11:31:36.161 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.171 7f15eade6700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f15e408c830 0x7f15e408d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:36.161 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.171 7f15eade6700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f15e41afe20 0x7f15e40b2f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:36.161 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.171 7f15eade6700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f15e41b0220 0x7f15e40b3550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:36.161 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.171 7f15eade6700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15e408cc30 con 0x7f15e41b0220 2021-02-10T11:31:36.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.171 7f15eade6700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15e40b3b70 con 0x7f15e408c830 2021-02-10T11:31:36.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.171 7f15eade6700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15e40b3d90 con 0x7f15e41afe20 2021-02-10T11:31:36.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.171 7f15e9403700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f15e41b0220 0x7f15e40b3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:36.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.171 7f15e9403700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f15e41b0220 0x7f15e40b3550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:48170/0 (socket says 172.21.2.112:48170) 2021-02-10T11:31:36.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.171 7f15e9403700 1 -- 172.21.2.112:0/97184896 learned_addr learned my addr 172.21.2.112:0/97184896 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:31:36.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.172 7f15e8c02700 1 --2- 172.21.2.112:0/97184896 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f15e408c830 0x7f15e408d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:36.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.172 7f15e3fff700 1 --2- 172.21.2.112:0/97184896 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f15e41afe20 0x7f15e40b2f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:36.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.172 7f15e9403700 1 -- 172.21.2.112:0/97184896 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f15e41afe20 msgr2=0x7f15e40b2f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:36.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.172 7f15e9403700 1 --2- 172.21.2.112:0/97184896 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f15e41afe20 0x7f15e40b2f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:36.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.172 7f15e9403700 1 -- 172.21.2.112:0/97184896 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f15e408c830 msgr2=0x7f15e408d0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:36.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.172 7f15e9403700 1 --2- 172.21.2.112:0/97184896 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f15e408c830 0x7f15e408d0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:36.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.172 7f15e9403700 1 -- 172.21.2.112:0/97184896 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15e41b0c00 con 0x7f15e41b0220 2021-02-10T11:31:36.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.172 7f15e8c02700 1 --2- 172.21.2.112:0/97184896 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f15e408c830 0x7f15e408d0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:31:36.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.172 7f15e9403700 1 --2- 172.21.2.112:0/97184896 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f15e41b0220 0x7f15e40b3550 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 rx=0x7f15dc006b80 tx=0x7f15dc00bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:36.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.172 7f15e17fa700 1 -- 172.21.2.112:0/97184896 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f15dc000f50 con 0x7f15e41b0220 2021-02-10T11:31:36.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.172 7f15e17fa700 1 -- 172.21.2.112:0/97184896 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15dc003030 con 0x7f15e41b0220 2021-02-10T11:31:36.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.172 7f15e17fa700 1 -- 172.21.2.112:0/97184896 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f15dc0014f0 con 0x7f15e41b0220 2021-02-10T11:31:36.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.172 7f15eade6700 1 -- 172.21.2.112:0/97184896 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15e41b0c00 con 0x7f15e41b0220 2021-02-10T11:31:36.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.172 7f15eade6700 1 -- 172.21.2.112:0/97184896 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15e41b1150 con 0x7f15e41b0220 2021-02-10T11:31:36.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.173 7f15e17fa700 1 -- 172.21.2.112:0/97184896 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f15dc001270 con 0x7f15e41b0220 2021-02-10T11:31:36.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.173 7f15e17fa700 1 --2- 172.21.2.112:0/97184896 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f15c8018730 0x7f15c801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:36.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.173 7f15e17fa700 1 -- 172.21.2.112:0/97184896 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f15dc032840 con 0x7f15e41b0220 2021-02-10T11:31:36.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.173 7f15e8c02700 1 --2- 172.21.2.112:0/97184896 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f15c8018730 0x7f15c801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:36.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.174 7f15e8c02700 1 --2- 172.21.2.112:0/97184896 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f15c8018730 0x7f15c801ac10 crc :-1 s=READY pgs=507 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:36.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.204 7f15eade6700 1 -- 172.21.2.112:0/97184896 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15e4083db0 con 0x7f15e41b0220 2021-02-10T11:31:36.194 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.207 7f15e17fa700 1 -- 172.21.2.112:0/97184896 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f15dc00be30 con 0x7f15e41b0220 2021-02-10T11:31:36.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.331 7f15eade6700 1 -- 172.21.2.112:0/97184896 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f15e407ed80 con 0x7f15e41b0220 2021-02-10T11:31:36.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.332 7f15e17fa700 1 -- 172.21.2.112:0/97184896 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) v1 ==== 74+0+11144 (secure 0 0 0) 0x7f15dc001ef0 con 0x7f15e41b0220 2021-02-10T11:31:36.319 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:31:36.319 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":33,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:31:35.097492","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:31:36.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.333 7f15eade6700 1 -- 172.21.2.112:0/97184896 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f15c8018730 msgr2=0x7f15c801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:36.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.333 7f15eade6700 1 --2- 172.21.2.112:0/97184896 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f15c8018730 0x7f15c801ac10 crc :-1 s=READY pgs=507 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:36.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.333 7f15eade6700 1 -- 172.21.2.112:0/97184896 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f15e41b0220 msgr2=0x7f15e40b3550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:36.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.333 7f15eade6700 1 --2- 172.21.2.112:0/97184896 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f15e41b0220 0x7f15e40b3550 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 rx=0x7f15dc006b80 tx=0x7f15dc00bcf0).stop 2021-02-10T11:31:36.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.333 7f15eade6700 1 -- 172.21.2.112:0/97184896 shutdown_connections 2021-02-10T11:31:36.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.333 7f15eade6700 1 --2- 172.21.2.112:0/97184896 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f15c8018730 0x7f15c801ac10 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:36.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.333 7f15eade6700 1 --2- 172.21.2.112:0/97184896 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f15e41b0220 0x7f15e40b3550 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:36.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.333 7f15eade6700 1 --2- 172.21.2.112:0/97184896 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f15e41afe20 0x7f15e40b2f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:36.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.333 7f15eade6700 1 --2- 172.21.2.112:0/97184896 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f15e408c830 0x7f15e408d0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:36.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.333 7f15eade6700 1 -- 172.21.2.112:0/97184896 >> 172.21.2.112:0/97184896 conn(0x7f15e40937a0 msgr2=0x7f15e4197e80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:36.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.334 7f15eade6700 1 -- 172.21.2.112:0/97184896 shutdown_connections 2021-02-10T11:31:36.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.334 7f15eade6700 1 -- 172.21.2.112:0/97184896 wait complete. 2021-02-10T11:31:36.330 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:13:57.508758', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2021-02-10T11:31:36.331 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:31:36.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.756 7fc066ba9700 1 Processor -- start 2021-02-10T11:31:36.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.756 7fc066ba9700 1 -- start start 2021-02-10T11:31:36.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.757 7fc066ba9700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc0601b0870 0x7fc0600f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:36.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.757 7fc066ba9700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0601b0c70 con 0x7fc0601b0470 2021-02-10T11:31:36.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.757 7fc066ba9700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0600f0b50 con 0x7fc0600efe00 2021-02-10T11:31:36.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.757 7fc066ba9700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0600e5ea0 con 0x7fc0601b0870 2021-02-10T11:31:36.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.757 7fc0649c5700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fc0601b0470 0x7fc0600ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:47636/0 (socket says 172.21.2.112:47636) 2021-02-10T11:31:36.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.757 7fc0649c5700 1 -- 172.21.2.112:0/3659032601 learned_addr learned my addr 172.21.2.112:0/3659032601 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:31:36.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.757 7fc05ffff700 1 --2- 172.21.2.112:0/3659032601 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc0601b0870 0x7fc0600f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:36.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.758 7fc05f7fe700 1 -- 172.21.2.112:0/3659032601 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1842348145 0 0) 0x7fc048001180 con 0x7fc0601b0470 2021-02-10T11:31:36.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.758 7fc05f7fe700 1 -- 172.21.2.112:0/3659032601 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc04c002fe0 con 0x7fc0601b0470 2021-02-10T11:31:36.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.758 7fc05f7fe700 1 -- 172.21.2.112:0/3659032601 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1940648490 0 0) 0x7fc0580009b0 con 0x7fc0600efe00 2021-02-10T11:31:36.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.758 7fc05f7fe700 1 -- 172.21.2.112:0/3659032601 --> v1:172.21.2.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc04c003210 con 0x7fc0600efe00 2021-02-10T11:31:36.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.758 7fc05f7fe700 1 -- 172.21.2.112:0/3659032601 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3976059388 0 0) 0x7fc048001180 con 0x7fc0601b0470 2021-02-10T11:31:36.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.758 7fc05ffff700 1 -- 172.21.2.112:0/3659032601 >> v1:172.21.2.120:6789/0 conn(0x7fc0600efe00 legacy=0x7fc0600f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:36.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.758 7fc05ffff700 1 -- 172.21.2.112:0/3659032601 >> v1:172.21.2.112:6789/0 conn(0x7fc0601b0470 legacy=0x7fc0600ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:36.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.758 7fc05ffff700 1 -- 172.21.2.112:0/3659032601 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0600e52e0 con 0x7fc0601b0870 2021-02-10T11:31:36.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.758 7fc05ffff700 1 --2- 172.21.2.112:0/3659032601 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc0601b0870 0x7fc0600f20e0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7fc0500069d0 tx=0x7fc050009dc0).ready entity=mon.2 client_cookie=b8a402d34cddd639 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:36.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.758 7fc05f7fe700 1 -- 172.21.2.112:0/3659032601 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc05000b510 con 0x7fc0601b0870 2021-02-10T11:31:36.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.758 7fc05f7fe700 1 -- 172.21.2.112:0/3659032601 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc050006ac0 con 0x7fc0601b0870 2021-02-10T11:31:36.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.759 7fc05f7fe700 1 -- 172.21.2.112:0/3659032601 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc05000ba80 con 0x7fc0601b0870 2021-02-10T11:31:36.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.759 7fc066ba9700 1 -- 172.21.2.112:0/3659032601 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc0601b0870 msgr2=0x7fc0600f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:36.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.759 7fc066ba9700 1 --2- 172.21.2.112:0/3659032601 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc0601b0870 0x7fc0600f20e0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7fc0500069d0 tx=0x7fc050009dc0).stop 2021-02-10T11:31:36.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.759 7fc066ba9700 1 -- 172.21.2.112:0/3659032601 shutdown_connections 2021-02-10T11:31:36.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.759 7fc066ba9700 1 --2- 172.21.2.112:0/3659032601 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc0601b0870 0x7fc0600f20e0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:36.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.759 7fc066ba9700 1 -- 172.21.2.112:0/3659032601 >> 172.21.2.112:0/3659032601 conn(0x7fc0600ce3e0 msgr2=0x7fc0600e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:36.763 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.759 7fc066ba9700 1 -- 172.21.2.112:0/3659032601 shutdown_connections 2021-02-10T11:31:36.763 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.759 7fc066ba9700 1 -- 172.21.2.112:0/3659032601 wait complete. 2021-02-10T11:31:36.764 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.774 7fc066ba9700 1 Processor -- start 2021-02-10T11:31:36.764 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.774 7fc066ba9700 1 -- start start 2021-02-10T11:31:36.764 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.774 7fc066ba9700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc060190920 0x7fc0600b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:36.764 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.774 7fc066ba9700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc0600b8200 0x7fc0600b8600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:36.764 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.774 7fc066ba9700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc0600c1d80 0x7fc0600c2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:36.765 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.774 7fc066ba9700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc060190da0 con 0x7fc060190920 2021-02-10T11:31:36.765 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.774 7fc066ba9700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0600b8c20 con 0x7fc0600c1d80 2021-02-10T11:31:36.765 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.774 7fc066ba9700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0600b8e10 con 0x7fc0600b8200 2021-02-10T11:31:36.765 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.774 7fc0649c5700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc060190920 0x7fc0600b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:36.765 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.774 7fc0649c5700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc060190920 0x7fc0600b7be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:48178/0 (socket says 172.21.2.112:48178) 2021-02-10T11:31:36.765 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.774 7fc0649c5700 1 -- 172.21.2.112:0/2290625520 learned_addr learned my addr 172.21.2.112:0/2290625520 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:31:36.765 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.775 7fc0649c5700 1 -- 172.21.2.112:0/2290625520 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc0600b8200 msgr2=0x7fc0600b8600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:36.765 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.775 7fc0649c5700 1 --2- 172.21.2.112:0/2290625520 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc0600b8200 0x7fc0600b8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:36.765 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.775 7fc0649c5700 1 -- 172.21.2.112:0/2290625520 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc0600c1d80 msgr2=0x7fc0600c2180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:36.766 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.775 7fc0649c5700 1 --2- 172.21.2.112:0/2290625520 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc0600c1d80 0x7fc0600c2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:36.766 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.775 7fc0649c5700 1 -- 172.21.2.112:0/2290625520 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0600c27d0 con 0x7fc060190920 2021-02-10T11:31:36.766 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.775 7fc0649c5700 1 --2- 172.21.2.112:0/2290625520 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc060190920 0x7fc0600b7be0 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 rx=0x7fc048001ec0 tx=0x7fc04800a930).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:36.766 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.775 7fc05d7fa700 1 -- 172.21.2.112:0/2290625520 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc0480070d0 con 0x7fc060190920 2021-02-10T11:31:36.766 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.775 7fc05d7fa700 1 -- 172.21.2.112:0/2290625520 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc048001180 con 0x7fc060190920 2021-02-10T11:31:36.766 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.775 7fc066ba9700 1 -- 172.21.2.112:0/2290625520 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0600c27d0 con 0x7fc060190920 2021-02-10T11:31:36.766 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.775 7fc066ba9700 1 -- 172.21.2.112:0/2290625520 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0600c2d60 con 0x7fc060190920 2021-02-10T11:31:36.766 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.775 7fc05d7fa700 1 -- 172.21.2.112:0/2290625520 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc048004830 con 0x7fc060190920 2021-02-10T11:31:36.766 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.776 7fc05d7fa700 1 -- 172.21.2.112:0/2290625520 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc048009030 con 0x7fc060190920 2021-02-10T11:31:36.766 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.776 7fc05d7fa700 1 --2- 172.21.2.112:0/2290625520 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc0440186c0 0x7fc04401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:36.766 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.776 7fc05d7fa700 1 -- 172.21.2.112:0/2290625520 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 4753+0+0 (secure 0 0 0) 0x7fc0480324a0 con 0x7fc060190920 2021-02-10T11:31:36.766 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.776 7fc05ffff700 1 --2- 172.21.2.112:0/2290625520 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc0440186c0 0x7fc04401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:36.766 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.776 7fc05ffff700 1 --2- 172.21.2.112:0/2290625520 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc0440186c0 0x7fc04401abc0 crc :-1 s=READY pgs=508 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:36.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.806 7fc066ba9700 1 -- 172.21.2.112:0/2290625520 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc060105ae0 con 0x7fc060190920 2021-02-10T11:31:36.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.809 7fc05d7fa700 1 -- 172.21.2.112:0/2290625520 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc04800ad20 con 0x7fc060190920 2021-02-10T11:31:36.923 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.936 7fc066ba9700 1 -- 172.21.2.112:0/2290625520 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7fc0600f0b50 con 0x7fc060190920 2021-02-10T11:31:36.924 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.936 7fc05d7fa700 1 -- 172.21.2.112:0/2290625520 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v33) v1 ==== 92+0+10 (secure 0 0 0) 0x7fc048009030 con 0x7fc060190920 2021-02-10T11:31:36.924 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:31:36.925 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.938 7fc066ba9700 1 -- 172.21.2.112:0/2290625520 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc0440186c0 msgr2=0x7fc04401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:36.925 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.938 7fc066ba9700 1 --2- 172.21.2.112:0/2290625520 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc0440186c0 0x7fc04401abc0 crc :-1 s=READY pgs=508 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:36.926 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.938 7fc066ba9700 1 -- 172.21.2.112:0/2290625520 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc060190920 msgr2=0x7fc0600b7be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:36.926 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.938 7fc066ba9700 1 --2- 172.21.2.112:0/2290625520 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc060190920 0x7fc0600b7be0 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 rx=0x7fc048001ec0 tx=0x7fc04800a930).stop 2021-02-10T11:31:36.926 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.938 7fc066ba9700 1 -- 172.21.2.112:0/2290625520 shutdown_connections 2021-02-10T11:31:36.926 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.938 7fc066ba9700 1 --2- 172.21.2.112:0/2290625520 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc0440186c0 0x7fc04401abc0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:36.926 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.938 7fc066ba9700 1 --2- 172.21.2.112:0/2290625520 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc0600c1d80 0x7fc0600c2180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:36.926 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.938 7fc066ba9700 1 --2- 172.21.2.112:0/2290625520 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc0600b8200 0x7fc0600b8600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:36.927 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.938 7fc066ba9700 1 --2- 172.21.2.112:0/2290625520 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc060190920 0x7fc0600b7be0 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:36.927 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.938 7fc066ba9700 1 -- 172.21.2.112:0/2290625520 >> 172.21.2.112:0/2290625520 conn(0x7fc060094e10 msgr2=0x7fc0600a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:36.927 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.939 7fc066ba9700 1 -- 172.21.2.112:0/2290625520 shutdown_connections 2021-02-10T11:31:36.927 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:36.939 7fc066ba9700 1 -- 172.21.2.112:0/2290625520 wait complete. 2021-02-10T11:31:36.935 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --format=json-pretty osd lspools 2021-02-10T11:31:37.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.356 7f5f5189a700 1 Processor -- start 2021-02-10T11:31:37.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.356 7f5f5189a700 1 -- start start 2021-02-10T11:31:37.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.356 7f5f5189a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5f4c0e5070 0x7f5f4c1b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:37.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.356 7f5f5189a700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f5f4c0e5470 0x7f5f4c07d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:37.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.356 7f5f5189a700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f4c07ed80 con 0x7f5f4c1b3e40 2021-02-10T11:31:37.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.356 7f5f5189a700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f4c080310 con 0x7f5f4c0e5470 2021-02-10T11:31:37.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.356 7f5f5189a700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f4c080500 con 0x7f5f4c0e5070 2021-02-10T11:31:37.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.357 7f5f4ad73700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f5f4c0e5470 0x7f5f4c07d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:37.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.357 7f5f4ad73700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f5f4c0e5470 0x7f5f4c07d690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:48186/0 (socket says 172.21.2.112:48186) 2021-02-10T11:31:37.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.357 7f5f4ad73700 1 -- 172.21.2.112:0/1178669842 learned_addr learned my addr 172.21.2.112:0/1178669842 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:31:37.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.357 7f5f4b574700 1 --2- 172.21.2.112:0/1178669842 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5f4c0e5070 0x7f5f4c1b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:37.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.357 7f5f4ad73700 1 -- 172.21.2.112:0/1178669842 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5f4c0e5070 msgr2=0x7f5f4c1b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:37.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.357 7f5f4ad73700 1 --2- 172.21.2.112:0/1178669842 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5f4c0e5070 0x7f5f4c1b3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:37.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.357 7f5f4ad73700 1 -- 172.21.2.112:0/1178669842 >> v1:172.21.2.120:6789/0 conn(0x7f5f4c1b3e40 legacy=0x7f5f4c07dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:37.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.357 7f5f4ad73700 1 -- 172.21.2.112:0/1178669842 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f4c0806f0 con 0x7f5f4c0e5470 2021-02-10T11:31:37.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.357 7f5f4ad73700 1 --2- 172.21.2.112:0/1178669842 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5f4c0e5470 0x7f5f4c07d690 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 rx=0x7f5f40006700 tx=0x7f5f40009b80).ready entity=mon.0 client_cookie=8a458c278ed5134b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:37.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.357 7f5f4a572700 1 -- 172.21.2.112:0/1178669842 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5f4000c0c0 con 0x7f5f4c0e5470 2021-02-10T11:31:37.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.358 7f5f4a572700 1 -- 172.21.2.112:0/1178669842 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f4000c340 con 0x7f5f4c0e5470 2021-02-10T11:31:37.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.358 7f5f4a572700 1 -- 172.21.2.112:0/1178669842 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5f4000c860 con 0x7f5f4c0e5470 2021-02-10T11:31:37.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.358 7f5f5189a700 1 -- 172.21.2.112:0/1178669842 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5f4c0e5470 msgr2=0x7f5f4c07d690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:37.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.358 7f5f5189a700 1 --2- 172.21.2.112:0/1178669842 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5f4c0e5470 0x7f5f4c07d690 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 rx=0x7f5f40006700 tx=0x7f5f40009b80).stop 2021-02-10T11:31:37.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.358 7f5f5189a700 1 -- 172.21.2.112:0/1178669842 shutdown_connections 2021-02-10T11:31:37.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.358 7f5f5189a700 1 --2- 172.21.2.112:0/1178669842 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5f4c0e5470 0x7f5f4c07d690 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:37.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.358 7f5f5189a700 1 --2- 172.21.2.112:0/1178669842 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5f4c0e5070 0x7f5f4c1b3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:37.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.358 7f5f5189a700 1 -- 172.21.2.112:0/1178669842 >> 172.21.2.112:0/1178669842 conn(0x7f5f4c0ce690 msgr2=0x7f5f4c0e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:37.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.358 7f5f5189a700 1 -- 172.21.2.112:0/1178669842 shutdown_connections 2021-02-10T11:31:37.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.358 7f5f5189a700 1 -- 172.21.2.112:0/1178669842 wait complete. 2021-02-10T11:31:37.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.373 7f5f5189a700 1 Processor -- start 2021-02-10T11:31:37.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.373 7f5f5189a700 1 -- start start 2021-02-10T11:31:37.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.373 7f5f5189a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5f4c0b1970 0x7f5f4c0b1d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:37.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.373 7f5f5189a700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5f4c08acb0 0x7f5f4c08b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:37.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.373 7f5f5189a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5f4c08b6d0 0x7f5f4c1aac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:37.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.373 7f5f5189a700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f4c08bba0 con 0x7f5f4c08b6d0 2021-02-10T11:31:37.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.373 7f5f5189a700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f4c1ab280 con 0x7f5f4c08acb0 2021-02-10T11:31:37.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.373 7f5f5189a700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f4c1ab4a0 con 0x7f5f4c0b1970 2021-02-10T11:31:37.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.374 7f5f4b574700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5f4c0b1970 0x7f5f4c0b1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:37.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.374 7f5f4bd75700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5f4c08b6d0 0x7f5f4c1aac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:37.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.374 7f5f4ad73700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5f4c08acb0 0x7f5f4c08b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:37.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.374 7f5f4b574700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5f4c0b1970 0x7f5f4c0b1d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3301/0 says I am v2:172.21.2.112:47870/0 (socket says 172.21.2.112:47870) 2021-02-10T11:31:37.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.374 7f5f4b574700 1 -- 172.21.2.112:0/2512396291 learned_addr learned my addr 172.21.2.112:0/2512396291 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:31:37.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.374 7f5f4ad73700 1 -- 172.21.2.112:0/2512396291 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5f4c0b1970 msgr2=0x7f5f4c0b1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:37.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.374 7f5f4ad73700 1 --2- 172.21.2.112:0/2512396291 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5f4c0b1970 0x7f5f4c0b1d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:37.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.374 7f5f4ad73700 1 -- 172.21.2.112:0/2512396291 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5f4c08b6d0 msgr2=0x7f5f4c1aac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:37.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.374 7f5f4ad73700 1 --2- 172.21.2.112:0/2512396291 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5f4c08b6d0 0x7f5f4c1aac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:37.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.374 7f5f4ad73700 1 -- 172.21.2.112:0/2512396291 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f4c1ab6c0 con 0x7f5f4c08acb0 2021-02-10T11:31:37.365 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.374 7f5f4bd75700 1 --2- 172.21.2.112:0/2512396291 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5f4c08b6d0 0x7f5f4c1aac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:31:37.365 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.374 7f5f4ad73700 1 --2- 172.21.2.112:0/2512396291 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5f4c08acb0 0x7f5f4c08b0b0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 rx=0x7f5f400096d0 tx=0x7f5f4000b8c0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:37.365 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.374 7f5f33fff700 1 -- 172.21.2.112:0/2512396291 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5f4000bcb0 con 0x7f5f4c08acb0 2021-02-10T11:31:37.365 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.374 7f5f33fff700 1 -- 172.21.2.112:0/2512396291 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f40002030 con 0x7f5f4c08acb0 2021-02-10T11:31:37.365 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.374 7f5f5189a700 1 -- 172.21.2.112:0/2512396291 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f4c1ab6c0 con 0x7f5f4c08acb0 2021-02-10T11:31:37.365 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.374 7f5f33fff700 1 -- 172.21.2.112:0/2512396291 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5f40002440 con 0x7f5f4c08acb0 2021-02-10T11:31:37.365 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.374 7f5f5189a700 1 -- 172.21.2.112:0/2512396291 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f4c1ab6c0 con 0x7f5f4c08acb0 2021-02-10T11:31:37.365 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.375 7f5f33fff700 1 -- 172.21.2.112:0/2512396291 <== mon.1 v2:172.21.2.120:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5f40002210 con 0x7f5f4c08acb0 2021-02-10T11:31:37.365 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.375 7f5f33fff700 1 --2- 172.21.2.112:0/2512396291 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5f2c018700 0x7f5f2c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:37.365 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.375 7f5f33fff700 1 -- 172.21.2.112:0/2512396291 <== mon.1 v2:172.21.2.120:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f5f40032260 con 0x7f5f4c08acb0 2021-02-10T11:31:37.365 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.375 7f5f4b574700 1 --2- 172.21.2.112:0/2512396291 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5f2c018700 0x7f5f2c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:37.365 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.376 7f5f4b574700 1 --2- 172.21.2.112:0/2512396291 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5f2c018700 0x7f5f2c01abc0 crc :-1 s=READY pgs=509 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:37.394 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.406 7f5f5189a700 1 -- 172.21.2.112:0/2512396291 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f4c1ab280 con 0x7f5f4c08acb0 2021-02-10T11:31:37.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.410 7f5f33fff700 1 -- 172.21.2.112:0/2512396291 <== mon.1 v2:172.21.2.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5f40002030 con 0x7f5f4c08acb0 2021-02-10T11:31:37.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.539 7f5f5189a700 1 -- 172.21.2.112:0/2512396291 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f5f4c1ab280 con 0x7f5f4c08acb0 2021-02-10T11:31:37.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.539 7f5f33fff700 1 -- 172.21.2.112:0/2512396291 <== mon.1 v2:172.21.2.120:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v33) v1 ==== 84+0+64 (secure 0 0 0) 0x7f5f4000bcb0 con 0x7f5f4c08acb0 2021-02-10T11:31:37.527 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:31:37.528 INFO:teuthology.orchestra.run.gibba012.stdout:[ 2021-02-10T11:31:37.528 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:31:37.528 INFO:teuthology.orchestra.run.gibba012.stdout: "poolnum": 1, 2021-02-10T11:31:37.528 INFO:teuthology.orchestra.run.gibba012.stdout: "poolname": "rbd" 2021-02-10T11:31:37.528 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:31:37.528 INFO:teuthology.orchestra.run.gibba012.stdout:] 2021-02-10T11:31:37.529 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.541 7f5f5189a700 1 -- 172.21.2.112:0/2512396291 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5f2c018700 msgr2=0x7f5f2c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:37.529 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.541 7f5f5189a700 1 --2- 172.21.2.112:0/2512396291 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5f2c018700 0x7f5f2c01abc0 crc :-1 s=READY pgs=509 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:37.530 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.541 7f5f5189a700 1 -- 172.21.2.112:0/2512396291 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5f4c08acb0 msgr2=0x7f5f4c08b0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:37.530 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.541 7f5f5189a700 1 --2- 172.21.2.112:0/2512396291 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5f4c08acb0 0x7f5f4c08b0b0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 rx=0x7f5f400096d0 tx=0x7f5f4000b8c0).stop 2021-02-10T11:31:37.530 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.541 7f5f5189a700 1 -- 172.21.2.112:0/2512396291 shutdown_connections 2021-02-10T11:31:37.530 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.541 7f5f5189a700 1 --2- 172.21.2.112:0/2512396291 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5f2c018700 0x7f5f2c01abc0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:37.530 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.541 7f5f5189a700 1 --2- 172.21.2.112:0/2512396291 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5f4c08acb0 0x7f5f4c08b0b0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:37.530 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.541 7f5f5189a700 1 --2- 172.21.2.112:0/2512396291 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5f4c08b6d0 0x7f5f4c1aac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:37.530 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.541 7f5f5189a700 1 --2- 172.21.2.112:0/2512396291 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5f4c0b1970 0x7f5f4c0b1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:37.530 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.541 7f5f5189a700 1 -- 172.21.2.112:0/2512396291 >> 172.21.2.112:0/2512396291 conn(0x7f5f4c1ac620 msgr2=0x7f5f4c0893d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:37.531 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.541 7f5f5189a700 1 -- 172.21.2.112:0/2512396291 shutdown_connections 2021-02-10T11:31:37.531 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.541 7f5f5189a700 1 -- 172.21.2.112:0/2512396291 wait complete. 2021-02-10T11:31:37.537 INFO:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2021-02-10T11:31:37.537 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_metadata 8 2021-02-10T11:31:37.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.962 7f5eb9357700 1 Processor -- start 2021-02-10T11:31:37.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.962 7f5eb9357700 1 -- start start 2021-02-10T11:31:37.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.963 7f5eb9357700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f5eb41b0470 0x7f5eb41b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:37.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.963 7f5eb9357700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5eb40f0b50 con 0x7f5eb40ef610 2021-02-10T11:31:37.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.963 7f5eb9357700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5eb40f24e0 con 0x7f5eb40f20e0 2021-02-10T11:31:37.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.963 7f5eb9357700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5eb40e5ea0 con 0x7f5eb41b0470 2021-02-10T11:31:37.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.963 7f5eb287c700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f5eb40ef610 0x7f5eb40efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:47658/0 (socket says 172.21.2.112:47658) 2021-02-10T11:31:37.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.963 7f5eb307d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f5eb41b0470 0x7f5eb41b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:37.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.963 7f5eb287c700 1 -- 172.21.2.112:0/4215874802 learned_addr learned my addr 172.21.2.112:0/4215874802 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:31:37.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.963 7f5eb207b700 1 -- 172.21.2.112:0/4215874802 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2553381497 0 0) 0x7f5ea8001180 con 0x7f5eb40ef610 2021-02-10T11:31:37.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.963 7f5eb207b700 1 -- 172.21.2.112:0/4215874802 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e9c002fe0 con 0x7f5eb40ef610 2021-02-10T11:31:37.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.963 7f5eb307d700 1 -- 172.21.2.112:0/4215874802 >> v1:172.21.2.120:6789/0 conn(0x7f5eb40f20e0 legacy=0x7f5eb40f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:37.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.963 7f5eb307d700 1 -- 172.21.2.112:0/4215874802 >> v1:172.21.2.112:6789/0 conn(0x7f5eb40ef610 legacy=0x7f5eb40efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:37.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.963 7f5eb307d700 1 -- 172.21.2.112:0/4215874802 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5eb40e52e0 con 0x7f5eb41b0470 2021-02-10T11:31:37.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.963 7f5eb307d700 1 --2- 172.21.2.112:0/4215874802 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5eb41b0470 0x7f5eb41b0870 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 rx=0x7f5ea4006650 tx=0x7f5ea4009a80).ready entity=mon.0 client_cookie=f675aa07f4090d41 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:37.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.964 7f5eb207b700 1 -- 172.21.2.112:0/4215874802 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5ea40009b0 con 0x7f5eb41b0470 2021-02-10T11:31:37.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.964 7f5eb207b700 1 -- 172.21.2.112:0/4215874802 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ea400c0c0 con 0x7f5eb41b0470 2021-02-10T11:31:37.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.964 7f5eb207b700 1 -- 172.21.2.112:0/4215874802 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5ea400c5e0 con 0x7f5eb41b0470 2021-02-10T11:31:37.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.964 7f5eb9357700 1 -- 172.21.2.112:0/4215874802 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5eb41b0470 msgr2=0x7f5eb41b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:37.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.964 7f5eb9357700 1 --2- 172.21.2.112:0/4215874802 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5eb41b0470 0x7f5eb41b0870 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 rx=0x7f5ea4006650 tx=0x7f5ea4009a80).stop 2021-02-10T11:31:37.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.964 7f5eb9357700 1 -- 172.21.2.112:0/4215874802 shutdown_connections 2021-02-10T11:31:37.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.964 7f5eb9357700 1 --2- 172.21.2.112:0/4215874802 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5eb41b0470 0x7f5eb41b0870 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:37.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.964 7f5eb9357700 1 -- 172.21.2.112:0/4215874802 >> 172.21.2.112:0/4215874802 conn(0x7f5eb40ce3e0 msgr2=0x7f5eb40e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:37.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.965 7f5eb9357700 1 -- 172.21.2.112:0/4215874802 shutdown_connections 2021-02-10T11:31:37.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.965 7f5eb9357700 1 -- 172.21.2.112:0/4215874802 wait complete. 2021-02-10T11:31:37.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.979 7f5eb9357700 1 Processor -- start 2021-02-10T11:31:37.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.979 7f5eb9357700 1 -- start start 2021-02-10T11:31:37.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.980 7f5eb9357700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5eb4094fb0 0x7f5eb409e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:37.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.980 7f5eb9357700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5eb40c2be0 0x7f5eb40c2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:37.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.980 7f5eb9357700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5eb40c3600 0x7f5eb40a9c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:37.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.980 7f5eb9357700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5eb409ebf0 con 0x7f5eb40c2be0 2021-02-10T11:31:37.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.980 7f5eb9357700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5eb40aa2a0 con 0x7f5eb4094fb0 2021-02-10T11:31:37.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.980 7f5eb9357700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5eb40aa4c0 con 0x7f5eb40c3600 2021-02-10T11:31:37.971 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.980 7f5eb287c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5eb40c2be0 0x7f5eb40c2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:37.971 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.980 7f5eb287c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5eb40c2be0 0x7f5eb40c2fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:48204/0 (socket says 172.21.2.112:48204) 2021-02-10T11:31:37.971 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.980 7f5eb287c700 1 -- 172.21.2.112:0/2215919768 learned_addr learned my addr 172.21.2.112:0/2215919768 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:31:37.971 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.980 7f5eb387e700 1 --2- 172.21.2.112:0/2215919768 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5eb40c3600 0x7f5eb40a9c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:37.971 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.980 7f5eb307d700 1 --2- 172.21.2.112:0/2215919768 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5eb4094fb0 0x7f5eb409e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:37.971 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.980 7f5eb287c700 1 -- 172.21.2.112:0/2215919768 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5eb40c3600 msgr2=0x7f5eb40a9c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:37.971 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.980 7f5eb287c700 1 --2- 172.21.2.112:0/2215919768 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5eb40c3600 0x7f5eb40a9c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:37.971 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.980 7f5eb287c700 1 -- 172.21.2.112:0/2215919768 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5eb4094fb0 msgr2=0x7f5eb409e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:37.971 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.980 7f5eb287c700 1 --2- 172.21.2.112:0/2215919768 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5eb4094fb0 0x7f5eb409e5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:37.971 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.981 7f5eb287c700 1 -- 172.21.2.112:0/2215919768 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5eb40aa6e0 con 0x7f5eb40c2be0 2021-02-10T11:31:37.971 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.981 7f5eb287c700 1 --2- 172.21.2.112:0/2215919768 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5eb40c2be0 0x7f5eb40c2fe0 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 rx=0x7f5ea800a5d0 tx=0x7f5ea800a970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:37.971 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.981 7f5e9bfff700 1 -- 172.21.2.112:0/2215919768 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5ea8000ce0 con 0x7f5eb40c2be0 2021-02-10T11:31:37.971 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.981 7f5e9bfff700 1 -- 172.21.2.112:0/2215919768 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ea80043a0 con 0x7f5eb40c2be0 2021-02-10T11:31:37.972 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.981 7f5e9bfff700 1 -- 172.21.2.112:0/2215919768 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5ea80047f0 con 0x7f5eb40c2be0 2021-02-10T11:31:37.972 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.981 7f5eb9357700 1 -- 172.21.2.112:0/2215919768 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5eb40aa6e0 con 0x7f5eb40c2be0 2021-02-10T11:31:37.972 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.981 7f5eb9357700 1 -- 172.21.2.112:0/2215919768 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5eb40aa6e0 con 0x7f5eb40c2be0 2021-02-10T11:31:37.972 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.982 7f5e9bfff700 1 -- 172.21.2.112:0/2215919768 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5ea8004790 con 0x7f5eb40c2be0 2021-02-10T11:31:37.972 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.982 7f5e9bfff700 1 --2- 172.21.2.112:0/2215919768 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5e940186c0 0x7f5e9401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:37.972 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.982 7f5e9bfff700 1 -- 172.21.2.112:0/2215919768 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f5ea80322d0 con 0x7f5eb40c2be0 2021-02-10T11:31:37.972 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.982 7f5eb307d700 1 --2- 172.21.2.112:0/2215919768 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5e940186c0 0x7f5e9401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:37.972 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:37.982 7f5eb307d700 1 --2- 172.21.2.112:0/2215919768 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5e940186c0 0x7f5e9401abc0 crc :-1 s=READY pgs=510 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:38.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:38.013 7f5eb9357700 1 -- 172.21.2.112:0/2215919768 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5eb40a9750 con 0x7f5eb40c2be0 2021-02-10T11:31:38.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:38.015 7f5e9bfff700 1 -- 172.21.2.112:0/2215919768 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5ea80043a0 con 0x7f5eb40c2be0 2021-02-10T11:31:38.132 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:38.144 7f5eb9357700 1 -- 172.21.2.112:0/2215919768 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"} v 0) v1 -- 0x7f5eb40a9750 con 0x7f5eb40c2be0 2021-02-10T11:31:38.553 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:38.565 7f5e9bfff700 1 -- 172.21.2.112:0/2215919768 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"}]=0 pool 'cephfs_metadata' created v34) v1 ==== 133+0+0 (secure 0 0 0) 0x7f5ea80045b0 con 0x7f5eb40c2be0 2021-02-10T11:31:38.598 INFO:teuthology.orchestra.run.gibba012.stderr:pool 'cephfs_metadata' created 2021-02-10T11:31:38.600 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:38.612 7f5eb9357700 1 -- 172.21.2.112:0/2215919768 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5e940186c0 msgr2=0x7f5e9401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:38.600 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:38.612 7f5eb9357700 1 --2- 172.21.2.112:0/2215919768 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5e940186c0 0x7f5e9401abc0 crc :-1 s=READY pgs=510 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:38.600 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:38.612 7f5eb9357700 1 -- 172.21.2.112:0/2215919768 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5eb40c2be0 msgr2=0x7f5eb40c2fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:38.600 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:38.612 7f5eb9357700 1 --2- 172.21.2.112:0/2215919768 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5eb40c2be0 0x7f5eb40c2fe0 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 rx=0x7f5ea800a5d0 tx=0x7f5ea800a970).stop 2021-02-10T11:31:38.600 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:38.613 7f5eb9357700 1 -- 172.21.2.112:0/2215919768 shutdown_connections 2021-02-10T11:31:38.600 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:38.613 7f5eb9357700 1 --2- 172.21.2.112:0/2215919768 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5e940186c0 0x7f5e9401abc0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:38.600 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:38.613 7f5eb9357700 1 --2- 172.21.2.112:0/2215919768 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5eb40c3600 0x7f5eb40a9c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:38.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:38.613 7f5eb9357700 1 --2- 172.21.2.112:0/2215919768 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5eb40c2be0 0x7f5eb40c2fe0 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:38.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:38.613 7f5eb9357700 1 --2- 172.21.2.112:0/2215919768 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5eb4094fb0 0x7f5eb409e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:38.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:38.613 7f5eb9357700 1 -- 172.21.2.112:0/2215919768 >> 172.21.2.112:0/2215919768 conn(0x7f5eb409f630 msgr2=0x7f5eb40a0130 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:38.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:38.613 7f5eb9357700 1 -- 172.21.2.112:0/2215919768 shutdown_connections 2021-02-10T11:31:38.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:38.613 7f5eb9357700 1 -- 172.21.2.112:0/2215919768 wait complete. 2021-02-10T11:31:38.608 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_data 8 2021-02-10T11:31:39.038 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:39.033 7f2f7cd45700 1 Processor -- start 2021-02-10T11:31:39.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:39.033 7f2f7cd45700 1 -- start start 2021-02-10T11:31:39.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:39.034 7f2f7cd45700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2f781b0470 0x7f2f781b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:39.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:39.034 7f2f7cd45700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f2f780ef610 0x7f2f780efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:39.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:39.034 7f2f7cd45700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f780f0b50 con 0x7f2f780f0030 2021-02-10T11:31:39.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:39.034 7f2f7cd45700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f780e52e0 con 0x7f2f780ef610 2021-02-10T11:31:39.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:39.034 7f2f7cd45700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f780e5490 con 0x7f2f781b0470 2021-02-10T11:31:39.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:39.034 7f2f77314700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7f2f780f0030 0x7f2f780f10d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:41234/0 (socket says 172.21.2.112:41234) 2021-02-10T11:31:39.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:39.034 7f2f77314700 1 -- 172.21.2.112:0/1349496205 learned_addr learned my addr 172.21.2.112:0/1349496205 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:31:39.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:39.034 7f2f76312700 1 --2- 172.21.2.112:0/1349496205 >> v2:172.21.2.120:3300/0 conn(0x7f2f780ef610 0x7f2f780efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:39.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:39.034 7f2f76b13700 1 --2- 172.21.2.112:0/1349496205 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2f781b0470 0x7f2f781b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:39.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:39.035 7f2f76312700 1 -- 172.21.2.112:0/1349496205 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2f781b0470 msgr2=0x7f2f781b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:39.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:39.035 7f2f76312700 1 --2- 172.21.2.112:0/1349496205 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2f781b0470 0x7f2f781b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:39.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:39.035 7f2f76312700 1 -- 172.21.2.112:0/1349496205 >> v1:172.21.2.120:6789/0 conn(0x7f2f780f0030 legacy=0x7f2f780f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:39.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:39.035 7f2f76312700 1 -- 172.21.2.112:0/1349496205 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f780f0430 con 0x7f2f780ef610 2021-02-10T11:31:39.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:39.035 7f2f76312700 1 --2- 172.21.2.112:0/1349496205 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2f780ef610 0x7f2f780efa10 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 rx=0x7f2f68006650 tx=0x7f2f68009a80).ready entity=mon.1 client_cookie=bf250fbc37c6d33f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:39.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:39.036 7f2f75b11700 1 -- 172.21.2.112:0/1349496205 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2f6800c0c0 con 0x7f2f780ef610 2021-02-10T11:31:39.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:39.036 7f2f75b11700 1 -- 172.21.2.112:0/1349496205 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f6800c340 con 0x7f2f780ef610 2021-02-10T11:31:39.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:39.036 7f2f75b11700 1 -- 172.21.2.112:0/1349496205 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2f6800c860 con 0x7f2f780ef610 2021-02-10T11:31:39.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:39.036 7f2f7cd45700 1 -- 172.21.2.112:0/1349496205 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2f780ef610 msgr2=0x7f2f780efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:39.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:39.036 7f2f7cd45700 1 --2- 172.21.2.112:0/1349496205 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2f780ef610 0x7f2f780efa10 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 rx=0x7f2f68006650 tx=0x7f2f68009a80).stop 2021-02-10T11:31:39.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:39.037 7f2f7cd45700 1 -- 172.21.2.112:0/1349496205 shutdown_connections 2021-02-10T11:31:39.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:39.037 7f2f7cd45700 1 --2- 172.21.2.112:0/1349496205 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2f780ef610 0x7f2f780efa10 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:39.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:39.037 7f2f7cd45700 1 --2- 172.21.2.112:0/1349496205 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2f781b0470 0x7f2f781b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:39.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:39.037 7f2f7cd45700 1 -- 172.21.2.112:0/1349496205 >> 172.21.2.112:0/1349496205 conn(0x7f2f780ce3e0 msgr2=0x7f2f780e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:39.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:39.037 7f2f7cd45700 1 -- 172.21.2.112:0/1349496205 shutdown_connections 2021-02-10T11:31:39.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:39.037 7f2f7cd45700 1 -- 172.21.2.112:0/1349496205 wait complete. 2021-02-10T11:31:39.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:39.051 7f2f7cd45700 1 Processor -- start 2021-02-10T11:31:39.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:39.051 7f2f7cd45700 1 -- start start 2021-02-10T11:31:39.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:39.052 7f2f7cd45700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2f780a0420 0x7f2f780a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:39.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:39.052 7f2f7cd45700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2f78189d00 0x7f2f7818a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:39.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:39.052 7f2f7cd45700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2f7818a720 0x7f2f780b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:39.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:39.052 7f2f7cd45700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f78189780 con 0x7f2f78189d00 2021-02-10T11:31:39.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:39.052 7f2f7cd45700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f7818ab20 con 0x7f2f7818a720 2021-02-10T11:31:39.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:39.052 7f2f7cd45700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f780a1140 con 0x7f2f780a0420 2021-02-10T11:31:39.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:39.052 7f2f76312700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2f78189d00 0x7f2f7818a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:39.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:39.052 7f2f76312700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2f78189d00 0x7f2f7818a100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:48218/0 (socket says 172.21.2.112:48218) 2021-02-10T11:31:39.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:39.052 7f2f76312700 1 -- 172.21.2.112:0/3249537135 learned_addr learned my addr 172.21.2.112:0/3249537135 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:31:39.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:39.053 7f2f76b13700 1 --2- 172.21.2.112:0/3249537135 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2f780a0420 0x7f2f780a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:39.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:39.053 7f2f76312700 1 -- 172.21.2.112:0/3249537135 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2f780a0420 msgr2=0x7f2f780a0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:39.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:39.053 7f2f77314700 1 --2- 172.21.2.112:0/3249537135 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2f7818a720 0x7f2f780b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:39.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:39.053 7f2f76312700 1 --2- 172.21.2.112:0/3249537135 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2f780a0420 0x7f2f780a0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:39.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:39.053 7f2f76312700 1 -- 172.21.2.112:0/3249537135 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2f7818a720 msgr2=0x7f2f780b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:39.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:39.053 7f2f76312700 1 --2- 172.21.2.112:0/3249537135 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2f7818a720 0x7f2f780b7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:39.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:39.053 7f2f76312700 1 -- 172.21.2.112:0/3249537135 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f780b8200 con 0x7f2f78189d00 2021-02-10T11:31:39.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:39.053 7f2f76b13700 1 --2- 172.21.2.112:0/3249537135 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2f780a0420 0x7f2f780a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:31:39.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:39.053 7f2f77314700 1 --2- 172.21.2.112:0/3249537135 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2f7818a720 0x7f2f780b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:31:39.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:39.053 7f2f76312700 1 --2- 172.21.2.112:0/3249537135 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2f78189d00 0x7f2f7818a100 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 rx=0x7f2f6800b5e0 tx=0x7f2f6800b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:39.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:39.053 7f2f6f7fe700 1 -- 172.21.2.112:0/3249537135 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2f6800bd70 con 0x7f2f78189d00 2021-02-10T11:31:39.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:39.053 7f2f6f7fe700 1 -- 172.21.2.112:0/3249537135 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f68009290 con 0x7f2f78189d00 2021-02-10T11:31:39.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:39.053 7f2f7cd45700 1 -- 172.21.2.112:0/3249537135 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f780b8200 con 0x7f2f78189d00 2021-02-10T11:31:39.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:39.053 7f2f7cd45700 1 -- 172.21.2.112:0/3249537135 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f780b8790 con 0x7f2f78189d00 2021-02-10T11:31:39.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:39.053 7f2f6f7fe700 1 -- 172.21.2.112:0/3249537135 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2f68002300 con 0x7f2f78189d00 2021-02-10T11:31:39.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:39.054 7f2f6f7fe700 1 -- 172.21.2.112:0/3249537135 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2f68000c90 con 0x7f2f78189d00 2021-02-10T11:31:39.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:39.054 7f2f6f7fe700 1 --2- 172.21.2.112:0/3249537135 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2f5c018730 0x7f2f5c01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:39.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:39.054 7f2f6f7fe700 1 -- 172.21.2.112:0/3249537135 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5107+0+0 (secure 0 0 0) 0x7f2f68032460 con 0x7f2f78189d00 2021-02-10T11:31:39.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:39.054 7f2f76b13700 1 --2- 172.21.2.112:0/3249537135 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2f5c018730 0x7f2f5c01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:39.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:39.055 7f2f76b13700 1 --2- 172.21.2.112:0/3249537135 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2f5c018730 0x7f2f5c01ac10 crc :-1 s=READY pgs=511 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:39.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:39.084 7f2f7cd45700 1 -- 172.21.2.112:0/3249537135 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f780a1360 con 0x7f2f78189d00 2021-02-10T11:31:39.074 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:39.086 7f2f6f7fe700 1 -- 172.21.2.112:0/3249537135 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2f68009290 con 0x7f2f78189d00 2021-02-10T11:31:39.203 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:39.216 7f2f7cd45700 1 -- 172.21.2.112:0/3249537135 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"} v 0) v1 -- 0x7f2f780a1360 con 0x7f2f78189d00 2021-02-10T11:31:39.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:39.572 7f2f6f7fe700 1 -- 172.21.2.112:0/3249537135 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"}]=0 pool 'cephfs_data' created v36) v1 ==== 125+0+0 (secure 0 0 0) 0x7f2f680025c0 con 0x7f2f78189d00 2021-02-10T11:31:39.569 INFO:teuthology.orchestra.run.gibba012.stderr:pool 'cephfs_data' created 2021-02-10T11:31:39.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:39.583 7f2f7cd45700 1 -- 172.21.2.112:0/3249537135 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2f5c018730 msgr2=0x7f2f5c01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:39.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:39.583 7f2f7cd45700 1 --2- 172.21.2.112:0/3249537135 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2f5c018730 0x7f2f5c01ac10 crc :-1 s=READY pgs=511 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:39.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:39.583 7f2f7cd45700 1 -- 172.21.2.112:0/3249537135 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2f78189d00 msgr2=0x7f2f7818a100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:39.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:39.583 7f2f7cd45700 1 --2- 172.21.2.112:0/3249537135 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2f78189d00 0x7f2f7818a100 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 rx=0x7f2f6800b5e0 tx=0x7f2f6800b980).stop 2021-02-10T11:31:39.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:39.583 7f2f7cd45700 1 -- 172.21.2.112:0/3249537135 shutdown_connections 2021-02-10T11:31:39.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:39.583 7f2f7cd45700 1 --2- 172.21.2.112:0/3249537135 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2f5c018730 0x7f2f5c01ac10 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:39.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:39.583 7f2f7cd45700 1 --2- 172.21.2.112:0/3249537135 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2f7818a720 0x7f2f780b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:39.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:39.583 7f2f7cd45700 1 --2- 172.21.2.112:0/3249537135 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2f78189d00 0x7f2f7818a100 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:39.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:39.583 7f2f7cd45700 1 --2- 172.21.2.112:0/3249537135 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2f780a0420 0x7f2f780a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:39.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:39.583 7f2f7cd45700 1 -- 172.21.2.112:0/3249537135 >> 172.21.2.112:0/3249537135 conn(0x7f2f78094e10 msgr2=0x7f2f780a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:39.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:39.583 7f2f7cd45700 1 -- 172.21.2.112:0/3249537135 shutdown_connections 2021-02-10T11:31:39.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:39.583 7f2f7cd45700 1 -- 172.21.2.112:0/3249537135 wait complete. 2021-02-10T11:31:39.579 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2021-02-10T11:31:40.010 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.005 7fa33b81a700 1 Processor -- start 2021-02-10T11:31:40.010 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.005 7fa33b81a700 1 -- start start 2021-02-10T11:31:40.010 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.005 7fa33b81a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa3340d1870 0x7fa3341b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:40.011 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.005 7fa33b81a700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fa3341b0ae0 0x7fa334108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:40.011 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.005 7fa33b81a700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fa3340d1c70 0x7fa334108f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:40.011 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.005 7fa33b81a700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3340e5ef0 con 0x7fa3341b0ae0 2021-02-10T11:31:40.011 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.005 7fa33b81a700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa334109a60 con 0x7fa3340d1c70 2021-02-10T11:31:40.011 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.005 7fa33b81a700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa334088020 con 0x7fa3340d1870 2021-02-10T11:31:40.011 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.006 7fa338e35700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fa3341b0ae0 0x7fa334108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:40.012 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.006 7fa338e35700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fa3341b0ae0 0x7fa334108940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:48226/0 (socket says 172.21.2.112:48226) 2021-02-10T11:31:40.012 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.006 7fa338e35700 1 -- 172.21.2.112:0/3408854928 learned_addr learned my addr 172.21.2.112:0/3408854928 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:31:40.012 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.006 7fa338e35700 1 -- 172.21.2.112:0/3408854928 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa3340d1870 msgr2=0x7fa3341b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:40.012 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.006 7fa338e35700 1 --2- 172.21.2.112:0/3408854928 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa3340d1870 0x7fa3341b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:40.012 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.006 7fa338e35700 1 -- 172.21.2.112:0/3408854928 >> v2:172.21.2.120:3300/0 conn(0x7fa3340d1c70 msgr2=0x7fa334108f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:40.012 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.006 7fa338e35700 1 --2- 172.21.2.112:0/3408854928 >> v2:172.21.2.120:3300/0 conn(0x7fa3340d1c70 0x7fa334108f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:40.012 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.006 7fa338e35700 1 -- 172.21.2.112:0/3408854928 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa334109580 con 0x7fa3341b0ae0 2021-02-10T11:31:40.013 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.006 7fa338e35700 1 --2- 172.21.2.112:0/3408854928 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa3341b0ae0 0x7fa334108940 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 rx=0x7fa330006700 tx=0x7fa330009b80).ready entity=mon.0 client_cookie=408ba4a49d9f0ddb server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:40.013 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.006 7fa327fff700 1 -- 172.21.2.112:0/3408854928 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa33000c0c0 con 0x7fa3341b0ae0 2021-02-10T11:31:40.013 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.006 7fa327fff700 1 -- 172.21.2.112:0/3408854928 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa33000c340 con 0x7fa3341b0ae0 2021-02-10T11:31:40.013 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.006 7fa327fff700 1 -- 172.21.2.112:0/3408854928 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa33000c860 con 0x7fa3341b0ae0 2021-02-10T11:31:40.013 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.006 7fa33b81a700 1 -- 172.21.2.112:0/3408854928 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa3341b0ae0 msgr2=0x7fa334108940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:40.013 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.006 7fa33b81a700 1 --2- 172.21.2.112:0/3408854928 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa3341b0ae0 0x7fa334108940 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 rx=0x7fa330006700 tx=0x7fa330009b80).stop 2021-02-10T11:31:40.014 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.007 7fa33b81a700 1 -- 172.21.2.112:0/3408854928 shutdown_connections 2021-02-10T11:31:40.014 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.007 7fa33b81a700 1 --2- 172.21.2.112:0/3408854928 >> v2:172.21.2.120:3300/0 conn(0x7fa3340d1c70 0x7fa334108f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:40.014 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.007 7fa33b81a700 1 --2- 172.21.2.112:0/3408854928 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa3341b0ae0 0x7fa334108940 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:40.014 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.007 7fa33b81a700 1 --2- 172.21.2.112:0/3408854928 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa3340d1870 0x7fa3341b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:40.014 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.007 7fa33b81a700 1 -- 172.21.2.112:0/3408854928 >> 172.21.2.112:0/3408854928 conn(0x7fa3340ce420 msgr2=0x7fa3340d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:40.014 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.007 7fa33b81a700 1 -- 172.21.2.112:0/3408854928 shutdown_connections 2021-02-10T11:31:40.015 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.007 7fa33b81a700 1 -- 172.21.2.112:0/3408854928 wait complete. 2021-02-10T11:31:40.015 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.022 7fa33b81a700 1 Processor -- start 2021-02-10T11:31:40.015 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.022 7fa33b81a700 1 -- start start 2021-02-10T11:31:40.015 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.022 7fa33b81a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa334187e70 0x7fa334096460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:40.015 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.022 7fa33b81a700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa3341937f0 0x7fa334095930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:40.015 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.022 7fa33b81a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa334095f50 0x7fa3341942d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:40.015 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.022 7fa33b81a700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa334193dd0 con 0x7fa334095f50 2021-02-10T11:31:40.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.022 7fa33b81a700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa334096a80 con 0x7fa3341937f0 2021-02-10T11:31:40.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.022 7fa33b81a700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3341948f0 con 0x7fa334187e70 2021-02-10T11:31:40.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.023 7fa339e37700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa334095f50 0x7fa3341942d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:40.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.023 7fa339e37700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa334095f50 0x7fa3341942d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:48230/0 (socket says 172.21.2.112:48230) 2021-02-10T11:31:40.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.023 7fa339e37700 1 -- 172.21.2.112:0/3980332949 learned_addr learned my addr 172.21.2.112:0/3980332949 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:31:40.018 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.023 7fa339e37700 1 -- 172.21.2.112:0/3980332949 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa334187e70 msgr2=0x7fa334096460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:40.019 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.023 7fa339e37700 1 --2- 172.21.2.112:0/3980332949 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa334187e70 0x7fa334096460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:40.019 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.023 7fa339e37700 1 -- 172.21.2.112:0/3980332949 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa3341937f0 msgr2=0x7fa334095930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:40.019 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.023 7fa339e37700 1 --2- 172.21.2.112:0/3980332949 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa3341937f0 0x7fa334095930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:40.019 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.023 7fa339e37700 1 -- 172.21.2.112:0/3980332949 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa334194b10 con 0x7fa334095f50 2021-02-10T11:31:40.019 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.023 7fa339e37700 1 --2- 172.21.2.112:0/3980332949 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa334095f50 0x7fa3341942d0 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 rx=0x7fa32c009980 tx=0x7fa32c00bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:40.019 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.023 7fa325ffb700 1 -- 172.21.2.112:0/3980332949 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa32c005870 con 0x7fa334095f50 2021-02-10T11:31:40.019 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.023 7fa325ffb700 1 -- 172.21.2.112:0/3980332949 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa32c005b30 con 0x7fa334095f50 2021-02-10T11:31:40.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.023 7fa325ffb700 1 -- 172.21.2.112:0/3980332949 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa32c006030 con 0x7fa334095f50 2021-02-10T11:31:40.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.023 7fa33b81a700 1 -- 172.21.2.112:0/3980332949 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa334194b10 con 0x7fa334095f50 2021-02-10T11:31:40.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.023 7fa33b81a700 1 -- 172.21.2.112:0/3980332949 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3341950a0 con 0x7fa334095f50 2021-02-10T11:31:40.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.024 7fa325ffb700 1 -- 172.21.2.112:0/3980332949 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa32c005d10 con 0x7fa334095f50 2021-02-10T11:31:40.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.024 7fa325ffb700 1 --2- 172.21.2.112:0/3980332949 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa3140186c0 0x7fa31401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:40.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.024 7fa325ffb700 1 -- 172.21.2.112:0/3980332949 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5457+0+0 (secure 0 0 0) 0x7fa32c0328c0 con 0x7fa334095f50 2021-02-10T11:31:40.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.025 7fa339636700 1 --2- 172.21.2.112:0/3980332949 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa3140186c0 0x7fa31401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:40.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.025 7fa339636700 1 --2- 172.21.2.112:0/3980332949 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa3140186c0 0x7fa31401abc0 crc :-1 s=READY pgs=514 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:40.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.056 7fa33b81a700 1 -- 172.21.2.112:0/3980332949 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3340828f0 con 0x7fa334095f50 2021-02-10T11:31:40.045 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.058 7fa325ffb700 1 -- 172.21.2.112:0/3980332949 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa32c0062f0 con 0x7fa334095f50 2021-02-10T11:31:40.176 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.189 7fa33b81a700 1 -- 172.21.2.112:0/3980332949 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"} v 0) v1 -- 0x7fa334109a60 con 0x7fa334095f50 2021-02-10T11:31:40.179 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.192 7fa325ffb700 1 -- 172.21.2.112:0/3980332949 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"}]=0 new fs with metadata pool 6 and data pool 7 v19) v1 ==== 187+0+0 (secure 0 0 0) 0x7fa32c005c00 con 0x7fa334095f50 2021-02-10T11:31:40.183 INFO:teuthology.orchestra.run.gibba012.stderr:new fs with metadata pool 6 and data pool 7 2021-02-10T11:31:40.186 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.198 7fa33b81a700 1 -- 172.21.2.112:0/3980332949 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa3140186c0 msgr2=0x7fa31401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:40.186 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.198 7fa33b81a700 1 --2- 172.21.2.112:0/3980332949 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa3140186c0 0x7fa31401abc0 crc :-1 s=READY pgs=514 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:40.186 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.198 7fa33b81a700 1 -- 172.21.2.112:0/3980332949 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa334095f50 msgr2=0x7fa3341942d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:40.186 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.198 7fa33b81a700 1 --2- 172.21.2.112:0/3980332949 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa334095f50 0x7fa3341942d0 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 rx=0x7fa32c009980 tx=0x7fa32c00bcd0).stop 2021-02-10T11:31:40.186 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.198 7fa33b81a700 1 -- 172.21.2.112:0/3980332949 shutdown_connections 2021-02-10T11:31:40.186 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.198 7fa33b81a700 1 --2- 172.21.2.112:0/3980332949 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa3140186c0 0x7fa31401abc0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:40.187 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.198 7fa33b81a700 1 --2- 172.21.2.112:0/3980332949 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa3341937f0 0x7fa334095930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:40.187 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.198 7fa33b81a700 1 --2- 172.21.2.112:0/3980332949 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa334095f50 0x7fa3341942d0 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:40.187 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.198 7fa33b81a700 1 --2- 172.21.2.112:0/3980332949 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa334187e70 0x7fa334096460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:40.187 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.198 7fa33b81a700 1 -- 172.21.2.112:0/3980332949 >> 172.21.2.112:0/3980332949 conn(0x7fa33419f830 msgr2=0x7fa334082b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:40.187 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.198 7fa33b81a700 1 -- 172.21.2.112:0/3980332949 shutdown_connections 2021-02-10T11:31:40.187 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.198 7fa33b81a700 1 -- 172.21.2.112:0/3980332949 wait complete. 2021-02-10T11:31:40.194 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:31:40.621 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.617 7f87c269a700 1 Processor -- start 2021-02-10T11:31:40.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.617 7f87c269a700 1 -- start start 2021-02-10T11:31:40.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.618 7f87c269a700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f87bc0e5470 0x7f87bc07e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:40.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.618 7f87c269a700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f87bc07f300 0x7f87bc07f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:40.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.618 7f87c269a700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87bc07ed80 con 0x7f87bc0e5070 2021-02-10T11:31:40.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.618 7f87c269a700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87bc1b4010 con 0x7f87bc0e5470 2021-02-10T11:31:40.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.618 7f87c269a700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87bc0d1790 con 0x7f87bc07f300 2021-02-10T11:31:40.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.618 7f87bb7fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f87bc0e5470 0x7f87bc07e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:40.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.618 7f87bbfff700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f87bc0e5070 0x7f87bc1b3820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:47700/0 (socket says 172.21.2.112:47700) 2021-02-10T11:31:40.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.618 7f87bbfff700 1 -- 172.21.2.112:0/3963073512 learned_addr learned my addr 172.21.2.112:0/3963073512 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:31:40.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.618 7f87bb7fe700 1 -- 172.21.2.112:0/3963073512 >> v2:172.21.2.120:3300/0 conn(0x7f87bc07f300 msgr2=0x7f87bc07f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:40.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.618 7f87bb7fe700 1 --2- 172.21.2.112:0/3963073512 >> v2:172.21.2.120:3300/0 conn(0x7f87bc07f300 0x7f87bc07f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:40.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.618 7f87bb7fe700 1 -- 172.21.2.112:0/3963073512 >> v1:172.21.2.112:6789/0 conn(0x7f87bc0e5070 legacy=0x7f87bc1b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:40.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.618 7f87bb7fe700 1 -- 172.21.2.112:0/3963073512 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87bc0d19b0 con 0x7f87bc0e5470 2021-02-10T11:31:40.624 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.618 7f87bb7fe700 1 --2- 172.21.2.112:0/3963073512 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f87bc0e5470 0x7f87bc07e6a0 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 rx=0x7f87b0006650 tx=0x7f87b0009a80).ready entity=mon.0 client_cookie=3476e75782c15ee8 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:40.624 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.619 7f87baffd700 1 -- 172.21.2.112:0/3963073512 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f87b000c0c0 con 0x7f87bc0e5470 2021-02-10T11:31:40.624 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.619 7f87baffd700 1 -- 172.21.2.112:0/3963073512 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87b000c340 con 0x7f87bc0e5470 2021-02-10T11:31:40.624 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.619 7f87baffd700 1 -- 172.21.2.112:0/3963073512 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f87b000c860 con 0x7f87bc0e5470 2021-02-10T11:31:40.624 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.619 7f87c269a700 1 -- 172.21.2.112:0/3963073512 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f87bc0e5470 msgr2=0x7f87bc07e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:40.624 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.619 7f87c269a700 1 --2- 172.21.2.112:0/3963073512 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f87bc0e5470 0x7f87bc07e6a0 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 rx=0x7f87b0006650 tx=0x7f87b0009a80).stop 2021-02-10T11:31:40.624 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.619 7f87c269a700 1 -- 172.21.2.112:0/3963073512 shutdown_connections 2021-02-10T11:31:40.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.619 7f87c269a700 1 --2- 172.21.2.112:0/3963073512 >> v2:172.21.2.120:3300/0 conn(0x7f87bc07f300 0x7f87bc07f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:40.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.619 7f87c269a700 1 --2- 172.21.2.112:0/3963073512 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f87bc0e5470 0x7f87bc07e6a0 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:40.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.619 7f87c269a700 1 -- 172.21.2.112:0/3963073512 >> 172.21.2.112:0/3963073512 conn(0x7f87bc0ce690 msgr2=0x7f87bc0e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:40.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.619 7f87c269a700 1 -- 172.21.2.112:0/3963073512 shutdown_connections 2021-02-10T11:31:40.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.619 7f87c269a700 1 -- 172.21.2.112:0/3963073512 wait complete. 2021-02-10T11:31:40.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.634 7f87c269a700 1 Processor -- start 2021-02-10T11:31:40.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.634 7f87c269a700 1 -- start start 2021-02-10T11:31:40.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.634 7f87c269a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f87bc1b19d0 0x7f87bc08cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:40.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.634 7f87c269a700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f87bc1b0d00 0x7f87bc1b1100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:40.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.634 7f87c269a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f87bc1a2b50 0x7f87bc1a2f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:40.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.634 7f87c269a700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87bc1b1720 con 0x7f87bc1a2b50 2021-02-10T11:31:40.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.634 7f87c269a700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87bc08c470 con 0x7f87bc1b0d00 2021-02-10T11:31:40.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.634 7f87c269a700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87bc1b1f50 con 0x7f87bc1b19d0 2021-02-10T11:31:40.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.635 7f87c0cb7700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f87bc1a2b50 0x7f87bc1a2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:40.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.635 7f87c0cb7700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f87bc1a2b50 0x7f87bc1a2f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:48248/0 (socket says 172.21.2.112:48248) 2021-02-10T11:31:40.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.635 7f87c0cb7700 1 -- 172.21.2.112:0/110126305 learned_addr learned my addr 172.21.2.112:0/110126305 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:31:40.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.635 7f87c0cb7700 1 -- 172.21.2.112:0/110126305 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f87bc1b19d0 msgr2=0x7f87bc08cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:40.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.635 7f87c0cb7700 1 --2- 172.21.2.112:0/110126305 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f87bc1b19d0 0x7f87bc08cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:40.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.635 7f87c0cb7700 1 -- 172.21.2.112:0/110126305 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f87bc1b0d00 msgr2=0x7f87bc1b1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:40.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.635 7f87c0cb7700 1 --2- 172.21.2.112:0/110126305 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f87bc1b0d00 0x7f87bc1b1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:40.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.635 7f87c0cb7700 1 -- 172.21.2.112:0/110126305 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87bc1a3570 con 0x7f87bc1a2b50 2021-02-10T11:31:40.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.635 7f87c0cb7700 1 --2- 172.21.2.112:0/110126305 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f87bc1a2b50 0x7f87bc1a2f50 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 rx=0x7f87b4009980 tx=0x7f87b400bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:40.629 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.635 7f87b8ff9700 1 -- 172.21.2.112:0/110126305 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f87b4005af0 con 0x7f87bc1a2b50 2021-02-10T11:31:40.629 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.635 7f87b8ff9700 1 -- 172.21.2.112:0/110126305 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87b4005d70 con 0x7f87bc1a2b50 2021-02-10T11:31:40.629 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.635 7f87b8ff9700 1 -- 172.21.2.112:0/110126305 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f87b40062b0 con 0x7f87bc1a2b50 2021-02-10T11:31:40.629 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.635 7f87c269a700 1 -- 172.21.2.112:0/110126305 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87bc1a3570 con 0x7f87bc1a2b50 2021-02-10T11:31:40.629 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.635 7f87c269a700 1 -- 172.21.2.112:0/110126305 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87bc0b3330 con 0x7f87bc1a2b50 2021-02-10T11:31:40.629 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.636 7f87b8ff9700 1 -- 172.21.2.112:0/110126305 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f87b4006030 con 0x7f87bc1a2b50 2021-02-10T11:31:40.629 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.636 7f87b8ff9700 1 --2- 172.21.2.112:0/110126305 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f879c0186c0 0x7f879c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:40.629 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.636 7f87b8ff9700 1 -- 172.21.2.112:0/110126305 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f87b40328b0 con 0x7f87bc1a2b50 2021-02-10T11:31:40.629 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.637 7f87bbfff700 1 --2- 172.21.2.112:0/110126305 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f879c0186c0 0x7f879c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:40.629 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.637 7f87bbfff700 1 --2- 172.21.2.112:0/110126305 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f879c0186c0 0x7f879c01abc0 crc :-1 s=READY pgs=515 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:40.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.668 7f87c269a700 1 -- 172.21.2.112:0/110126305 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87bc083dd0 con 0x7f87bc1a2b50 2021-02-10T11:31:40.657 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.670 7f87b8ff9700 1 -- 172.21.2.112:0/110126305 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f87b4006570 con 0x7f87bc1a2b50 2021-02-10T11:31:40.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.795 7f87c269a700 1 -- 172.21.2.112:0/110126305 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f87bc07ed80 con 0x7f87bc1a2b50 2021-02-10T11:31:40.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.796 7f87b8ff9700 1 -- 172.21.2.112:0/110126305 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v37) v1 ==== 74+0+14006 (secure 0 0 0) 0x7f87b4006030 con 0x7f87bc1a2b50 2021-02-10T11:31:40.784 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:31:40.784 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":37,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:31:40.190178","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:31:38.162654","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":7,"pool_name":"cephfs_data","create_time":"2021-02-10 11:31:39.233986","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:31:40.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.797 7f87c269a700 1 -- 172.21.2.112:0/110126305 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f879c0186c0 msgr2=0x7f879c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:40.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.797 7f87c269a700 1 --2- 172.21.2.112:0/110126305 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f879c0186c0 0x7f879c01abc0 crc :-1 s=READY pgs=515 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:40.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.797 7f87c269a700 1 -- 172.21.2.112:0/110126305 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f87bc1a2b50 msgr2=0x7f87bc1a2f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:40.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.797 7f87c269a700 1 --2- 172.21.2.112:0/110126305 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f87bc1a2b50 0x7f87bc1a2f50 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 rx=0x7f87b4009980 tx=0x7f87b400bcd0).stop 2021-02-10T11:31:40.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.798 7f87c269a700 1 -- 172.21.2.112:0/110126305 shutdown_connections 2021-02-10T11:31:40.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.798 7f87c269a700 1 --2- 172.21.2.112:0/110126305 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f879c0186c0 0x7f879c01abc0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:40.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.798 7f87c269a700 1 --2- 172.21.2.112:0/110126305 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f87bc1b0d00 0x7f87bc1b1100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:40.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.798 7f87c269a700 1 --2- 172.21.2.112:0/110126305 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f87bc1a2b50 0x7f87bc1a2f50 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:40.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.798 7f87c269a700 1 --2- 172.21.2.112:0/110126305 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f87bc1b19d0 0x7f87bc08cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:40.786 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.798 7f87c269a700 1 -- 172.21.2.112:0/110126305 >> 172.21.2.112:0/110126305 conn(0x7f87bc093720 msgr2=0x7f87bc084b80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:40.786 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.798 7f87c269a700 1 -- 172.21.2.112:0/110126305 shutdown_connections 2021-02-10T11:31:40.786 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:40.798 7f87c269a700 1 -- 172.21.2.112:0/110126305 wait complete. 2021-02-10T11:31:40.794 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:31:41.219 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.216 7ffb606bc700 1 Processor -- start 2021-02-10T11:31:41.220 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.216 7ffb606bc700 1 -- start start 2021-02-10T11:31:41.220 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.216 7ffb606bc700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7ffb580e5070 0x7ffb581b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:41.220 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.216 7ffb606bc700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ffb581b3e40 0x7ffb5807d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:41.220 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.216 7ffb606bc700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7ffb580e5470 0x7ffb5807dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:41.220 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.216 7ffb606bc700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb5807ed80 con 0x7ffb580e5470 2021-02-10T11:31:41.220 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.216 7ffb606bc700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb5807e2d0 con 0x7ffb580e5070 2021-02-10T11:31:41.220 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.216 7ffb606bc700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb5807e4c0 con 0x7ffb581b3e40 2021-02-10T11:31:41.220 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.217 7ffb5ecd9700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7ffb580e5470 0x7ffb5807dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:41.220 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.217 7ffb5ecd9700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7ffb580e5470 0x7ffb5807dcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:48252/0 (socket says 172.21.2.112:48252) 2021-02-10T11:31:41.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.217 7ffb5ecd9700 1 -- 172.21.2.112:0/370553505 learned_addr learned my addr 172.21.2.112:0/370553505 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:31:41.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.217 7ffb5e4d8700 1 --2- 172.21.2.112:0/370553505 >> v2:172.21.2.120:3300/0 conn(0x7ffb580e5070 0x7ffb581b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:41.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.217 7ffb5dcd7700 1 --2- 172.21.2.112:0/370553505 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ffb581b3e40 0x7ffb5807d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:41.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.217 7ffb5ecd9700 1 -- 172.21.2.112:0/370553505 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ffb581b3e40 msgr2=0x7ffb5807d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:41.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.217 7ffb5ecd9700 1 --2- 172.21.2.112:0/370553505 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ffb581b3e40 0x7ffb5807d690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:41.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.217 7ffb5ecd9700 1 -- 172.21.2.112:0/370553505 >> v2:172.21.2.120:3300/0 conn(0x7ffb580e5070 msgr2=0x7ffb581b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:41.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.217 7ffb5ecd9700 1 --2- 172.21.2.112:0/370553505 >> v2:172.21.2.120:3300/0 conn(0x7ffb580e5070 0x7ffb581b3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:41.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.217 7ffb5ecd9700 1 -- 172.21.2.112:0/370553505 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb5807e6e0 con 0x7ffb580e5470 2021-02-10T11:31:41.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.217 7ffb5ecd9700 1 --2- 172.21.2.112:0/370553505 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ffb580e5470 0x7ffb5807dcb0 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 rx=0x7ffb50006f00 tx=0x7ffb50009f10).ready entity=mon.0 client_cookie=ba664868b8a175be server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:41.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.217 7ffb5d4d6700 1 -- 172.21.2.112:0/370553505 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffb5000b3b0 con 0x7ffb580e5470 2021-02-10T11:31:41.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.217 7ffb5d4d6700 1 -- 172.21.2.112:0/370553505 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb5000b630 con 0x7ffb580e5470 2021-02-10T11:31:41.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.217 7ffb5d4d6700 1 -- 172.21.2.112:0/370553505 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffb5000bb50 con 0x7ffb580e5470 2021-02-10T11:31:41.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.217 7ffb606bc700 1 -- 172.21.2.112:0/370553505 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ffb580e5470 msgr2=0x7ffb5807dcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:41.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.217 7ffb606bc700 1 --2- 172.21.2.112:0/370553505 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ffb580e5470 0x7ffb5807dcb0 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 rx=0x7ffb50006f00 tx=0x7ffb50009f10).stop 2021-02-10T11:31:41.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.218 7ffb606bc700 1 -- 172.21.2.112:0/370553505 shutdown_connections 2021-02-10T11:31:41.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.218 7ffb606bc700 1 --2- 172.21.2.112:0/370553505 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ffb580e5470 0x7ffb5807dcb0 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:41.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.218 7ffb606bc700 1 --2- 172.21.2.112:0/370553505 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ffb581b3e40 0x7ffb5807d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:41.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.218 7ffb606bc700 1 --2- 172.21.2.112:0/370553505 >> v2:172.21.2.120:3300/0 conn(0x7ffb580e5070 0x7ffb581b3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:41.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.218 7ffb606bc700 1 -- 172.21.2.112:0/370553505 >> 172.21.2.112:0/370553505 conn(0x7ffb580ce690 msgr2=0x7ffb580e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:41.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.218 7ffb606bc700 1 -- 172.21.2.112:0/370553505 shutdown_connections 2021-02-10T11:31:41.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.218 7ffb606bc700 1 -- 172.21.2.112:0/370553505 wait complete. 2021-02-10T11:31:41.224 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.233 7ffb606bc700 1 Processor -- start 2021-02-10T11:31:41.224 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.233 7ffb606bc700 1 -- start start 2021-02-10T11:31:41.224 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.233 7ffb606bc700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ffb581900e0 0x7ffb580b0c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:41.224 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.233 7ffb606bc700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ffb5818f410 0x7ffb5818f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:41.224 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.233 7ffb606bc700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ffb581ada10 0x7ffb581ade10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:41.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.233 7ffb606bc700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb5808e100 con 0x7ffb581ada10 2021-02-10T11:31:41.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.233 7ffb606bc700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb5818fe30 con 0x7ffb5818f410 2021-02-10T11:31:41.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.233 7ffb606bc700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb581ad090 con 0x7ffb581900e0 2021-02-10T11:31:41.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.233 7ffb5ecd9700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ffb581ada10 0x7ffb581ade10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:41.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.233 7ffb5ecd9700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ffb581ada10 0x7ffb581ade10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:48258/0 (socket says 172.21.2.112:48258) 2021-02-10T11:31:41.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.233 7ffb5ecd9700 1 -- 172.21.2.112:0/2152620276 learned_addr learned my addr 172.21.2.112:0/2152620276 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:31:41.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.233 7ffb5e4d8700 1 --2- 172.21.2.112:0/2152620276 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ffb581900e0 0x7ffb580b0c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:41.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.233 7ffb5ecd9700 1 -- 172.21.2.112:0/2152620276 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ffb581900e0 msgr2=0x7ffb580b0c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:41.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.233 7ffb5ecd9700 1 --2- 172.21.2.112:0/2152620276 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ffb581900e0 0x7ffb580b0c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:41.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.233 7ffb5ecd9700 1 -- 172.21.2.112:0/2152620276 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ffb5818f410 msgr2=0x7ffb5818f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:41.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.233 7ffb5dcd7700 1 --2- 172.21.2.112:0/2152620276 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ffb5818f410 0x7ffb5818f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:41.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.233 7ffb5ecd9700 1 --2- 172.21.2.112:0/2152620276 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ffb5818f410 0x7ffb5818f810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:41.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.233 7ffb5ecd9700 1 -- 172.21.2.112:0/2152620276 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb580b1240 con 0x7ffb581ada10 2021-02-10T11:31:41.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.234 7ffb5dcd7700 1 --2- 172.21.2.112:0/2152620276 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ffb5818f410 0x7ffb5818f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:31:41.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.234 7ffb5ecd9700 1 --2- 172.21.2.112:0/2152620276 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ffb581ada10 0x7ffb581ade10 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 rx=0x7ffb5000b8e0 tx=0x7ffb5000bc80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:41.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.234 7ffb4affd700 1 -- 172.21.2.112:0/2152620276 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffb50003030 con 0x7ffb581ada10 2021-02-10T11:31:41.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.234 7ffb4affd700 1 -- 172.21.2.112:0/2152620276 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb500068e0 con 0x7ffb581ada10 2021-02-10T11:31:41.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.234 7ffb4affd700 1 -- 172.21.2.112:0/2152620276 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffb5000c570 con 0x7ffb581ada10 2021-02-10T11:31:41.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.234 7ffb606bc700 1 -- 172.21.2.112:0/2152620276 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb580b1240 con 0x7ffb581ada10 2021-02-10T11:31:41.227 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.234 7ffb606bc700 1 -- 172.21.2.112:0/2152620276 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb581ae720 con 0x7ffb581ada10 2021-02-10T11:31:41.227 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.235 7ffb4affd700 1 -- 172.21.2.112:0/2152620276 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ffb50001270 con 0x7ffb581ada10 2021-02-10T11:31:41.227 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.235 7ffb4affd700 1 --2- 172.21.2.112:0/2152620276 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ffb400186c0 0x7ffb4001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:41.227 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.235 7ffb4affd700 1 -- 172.21.2.112:0/2152620276 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5573+0+0 (secure 0 0 0) 0x7ffb50032900 con 0x7ffb581ada10 2021-02-10T11:31:41.227 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.235 7ffb5e4d8700 1 --2- 172.21.2.112:0/2152620276 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ffb400186c0 0x7ffb4001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:41.227 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.235 7ffb5e4d8700 1 --2- 172.21.2.112:0/2152620276 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ffb400186c0 0x7ffb4001abc0 crc :-1 s=READY pgs=516 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:41.253 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.266 7ffb606bc700 1 -- 172.21.2.112:0/2152620276 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb5807ed80 con 0x7ffb581ada10 2021-02-10T11:31:41.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.269 7ffb4affd700 1 -- 172.21.2.112:0/2152620276 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ffb50009430 con 0x7ffb581ada10 2021-02-10T11:31:41.383 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.396 7ffb606bc700 1 -- 172.21.2.112:0/2152620276 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffb5807ed80 con 0x7ffb581ada10 2021-02-10T11:31:41.384 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.396 7ffb4affd700 1 -- 172.21.2.112:0/2152620276 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v38) v1 ==== 74+0+13993 (secure 0 0 0) 0x7ffb5001f6a0 con 0x7ffb581ada10 2021-02-10T11:31:41.384 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:31:41.384 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":38,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:31:41.192016","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:31:38.162654","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":7,"pool_name":"cephfs_data","create_time":"2021-02-10 11:31:39.233986","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:31:41.386 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.398 7ffb606bc700 1 -- 172.21.2.112:0/2152620276 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ffb400186c0 msgr2=0x7ffb4001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:41.386 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.398 7ffb606bc700 1 --2- 172.21.2.112:0/2152620276 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ffb400186c0 0x7ffb4001abc0 crc :-1 s=READY pgs=516 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:41.386 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.398 7ffb606bc700 1 -- 172.21.2.112:0/2152620276 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ffb581ada10 msgr2=0x7ffb581ade10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:41.386 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.398 7ffb606bc700 1 --2- 172.21.2.112:0/2152620276 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ffb581ada10 0x7ffb581ade10 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 rx=0x7ffb5000b8e0 tx=0x7ffb5000bc80).stop 2021-02-10T11:31:41.386 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.398 7ffb606bc700 1 -- 172.21.2.112:0/2152620276 shutdown_connections 2021-02-10T11:31:41.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.398 7ffb606bc700 1 --2- 172.21.2.112:0/2152620276 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ffb400186c0 0x7ffb4001abc0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:41.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.398 7ffb606bc700 1 --2- 172.21.2.112:0/2152620276 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ffb5818f410 0x7ffb5818f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:41.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.398 7ffb606bc700 1 --2- 172.21.2.112:0/2152620276 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ffb581ada10 0x7ffb581ade10 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:41.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.398 7ffb606bc700 1 --2- 172.21.2.112:0/2152620276 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ffb581900e0 0x7ffb580b0c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:41.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.398 7ffb606bc700 1 -- 172.21.2.112:0/2152620276 >> 172.21.2.112:0/2152620276 conn(0x7ffb581a31c0 msgr2=0x7ffb5808fa80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:41.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.399 7ffb606bc700 1 -- 172.21.2.112:0/2152620276 shutdown_connections 2021-02-10T11:31:41.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.399 7ffb606bc700 1 -- 172.21.2.112:0/2152620276 wait complete. 2021-02-10T11:31:41.395 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2021-02-10T11:31:41.826 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.822 7fce2c6c8700 1 Processor -- start 2021-02-10T11:31:41.827 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.822 7fce2c6c8700 1 -- start start 2021-02-10T11:31:41.827 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.823 7fce2c6c8700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fce241b0470 0x7fce241b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:41.827 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.823 7fce2c6c8700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fce240ef610 0x7fce240efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:41.827 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.823 7fce2c6c8700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fce240f0030 0x7fce240f0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:41.827 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.823 7fce2c6c8700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce240e5ea0 con 0x7fce241b0470 2021-02-10T11:31:41.827 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.823 7fce2c6c8700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce240f0b50 con 0x7fce240f0030 2021-02-10T11:31:41.827 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.823 7fce2c6c8700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce240e52e0 con 0x7fce240ef610 2021-02-10T11:31:41.827 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.823 7fce2a4e4700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fce241b0470 0x7fce241b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:41.827 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.823 7fce2a4e4700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fce241b0470 0x7fce241b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:48264/0 (socket says 172.21.2.112:48264) 2021-02-10T11:31:41.827 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.823 7fce2a4e4700 1 -- 172.21.2.112:0/715222707 learned_addr learned my addr 172.21.2.112:0/715222707 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:31:41.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.823 7fce2ace5700 1 --2- 172.21.2.112:0/715222707 >> v2:172.21.2.120:3300/0 conn(0x7fce240f0030 0x7fce240f0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:41.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.823 7fce29ce3700 1 --2- 172.21.2.112:0/715222707 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fce240ef610 0x7fce240efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:41.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.823 7fce2a4e4700 1 -- 172.21.2.112:0/715222707 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fce240ef610 msgr2=0x7fce240efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:41.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.823 7fce2a4e4700 1 --2- 172.21.2.112:0/715222707 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fce240ef610 0x7fce240efa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:41.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.823 7fce2a4e4700 1 -- 172.21.2.112:0/715222707 >> v2:172.21.2.120:3300/0 conn(0x7fce240f0030 msgr2=0x7fce240f0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:41.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.823 7fce2a4e4700 1 --2- 172.21.2.112:0/715222707 >> v2:172.21.2.120:3300/0 conn(0x7fce240f0030 0x7fce240f0430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:41.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.823 7fce2a4e4700 1 -- 172.21.2.112:0/715222707 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce240f10d0 con 0x7fce241b0470 2021-02-10T11:31:41.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.823 7fce2ace5700 1 --2- 172.21.2.112:0/715222707 >> v2:172.21.2.120:3300/0 conn(0x7fce240f0030 0x7fce240f0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:31:41.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.823 7fce29ce3700 1 --2- 172.21.2.112:0/715222707 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fce240ef610 0x7fce240efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:31:41.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.823 7fce2a4e4700 1 --2- 172.21.2.112:0/715222707 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fce241b0470 0x7fce241b0870 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 rx=0x7fce20006700 tx=0x7fce20009b80).ready entity=mon.0 client_cookie=b91a392a6cd810d8 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:41.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.824 7fce294e2700 1 -- 172.21.2.112:0/715222707 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fce2000c0c0 con 0x7fce241b0470 2021-02-10T11:31:41.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.824 7fce294e2700 1 -- 172.21.2.112:0/715222707 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce2000c440 con 0x7fce241b0470 2021-02-10T11:31:41.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.824 7fce294e2700 1 -- 172.21.2.112:0/715222707 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fce2000c960 con 0x7fce241b0470 2021-02-10T11:31:41.829 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.824 7fce2c6c8700 1 -- 172.21.2.112:0/715222707 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fce241b0470 msgr2=0x7fce241b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:41.829 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.824 7fce2c6c8700 1 --2- 172.21.2.112:0/715222707 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fce241b0470 0x7fce241b0870 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 rx=0x7fce20006700 tx=0x7fce20009b80).stop 2021-02-10T11:31:41.829 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.825 7fce2c6c8700 1 -- 172.21.2.112:0/715222707 shutdown_connections 2021-02-10T11:31:41.829 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.825 7fce2c6c8700 1 --2- 172.21.2.112:0/715222707 >> v2:172.21.2.120:3300/0 conn(0x7fce240f0030 0x7fce240f0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:41.829 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.825 7fce2c6c8700 1 --2- 172.21.2.112:0/715222707 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fce240ef610 0x7fce240efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:41.829 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.825 7fce2c6c8700 1 --2- 172.21.2.112:0/715222707 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fce241b0470 0x7fce241b0870 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:41.829 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.825 7fce2c6c8700 1 -- 172.21.2.112:0/715222707 >> 172.21.2.112:0/715222707 conn(0x7fce240ce3e0 msgr2=0x7fce240e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:41.829 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.825 7fce2c6c8700 1 -- 172.21.2.112:0/715222707 shutdown_connections 2021-02-10T11:31:41.829 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.825 7fce2c6c8700 1 -- 172.21.2.112:0/715222707 wait complete. 2021-02-10T11:31:41.829 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.839 7fce2c6c8700 1 Processor -- start 2021-02-10T11:31:41.829 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.839 7fce2c6c8700 1 -- start start 2021-02-10T11:31:41.830 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.840 7fce2c6c8700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fce2409b930 0x7fce240b7c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:41.830 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.840 7fce2c6c8700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fce240b8410 0x7fce240b8810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:41.831 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.840 7fce2c6c8700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fce240c1f80 0x7fce240c2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:41.831 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.840 7fce2c6c8700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce240c1a00 con 0x7fce240b8410 2021-02-10T11:31:41.831 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.840 7fce2c6c8700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce240b8e30 con 0x7fce240c1f80 2021-02-10T11:31:41.831 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.840 7fce2c6c8700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce240b9050 con 0x7fce2409b930 2021-02-10T11:31:41.831 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.840 7fce29ce3700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fce240b8410 0x7fce240b8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:41.831 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.840 7fce29ce3700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fce240b8410 0x7fce240b8810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:48270/0 (socket says 172.21.2.112:48270) 2021-02-10T11:31:41.831 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.840 7fce29ce3700 1 -- 172.21.2.112:0/3320524945 learned_addr learned my addr 172.21.2.112:0/3320524945 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:31:41.831 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.840 7fce29ce3700 1 -- 172.21.2.112:0/3320524945 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fce2409b930 msgr2=0x7fce240b7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:41.831 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.840 7fce29ce3700 1 --2- 172.21.2.112:0/3320524945 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fce2409b930 0x7fce240b7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:41.831 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.840 7fce29ce3700 1 -- 172.21.2.112:0/3320524945 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fce240c1f80 msgr2=0x7fce240c2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:41.831 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.840 7fce29ce3700 1 --2- 172.21.2.112:0/3320524945 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fce240c1f80 0x7fce240c2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:41.831 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.840 7fce29ce3700 1 -- 172.21.2.112:0/3320524945 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce240c29a0 con 0x7fce240b8410 2021-02-10T11:31:41.831 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.841 7fce29ce3700 1 --2- 172.21.2.112:0/3320524945 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fce240b8410 0x7fce240b8810 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 rx=0x7fce14006f20 tx=0x7fce1400b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:41.832 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.841 7fce1affd700 1 -- 172.21.2.112:0/3320524945 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fce14006bb0 con 0x7fce240b8410 2021-02-10T11:31:41.832 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.841 7fce1affd700 1 -- 172.21.2.112:0/3320524945 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce1400bd80 con 0x7fce240b8410 2021-02-10T11:31:41.832 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.841 7fce1affd700 1 -- 172.21.2.112:0/3320524945 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fce14002220 con 0x7fce240b8410 2021-02-10T11:31:41.832 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.841 7fce2c6c8700 1 -- 172.21.2.112:0/3320524945 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce240c29a0 con 0x7fce240b8410 2021-02-10T11:31:41.832 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.841 7fce2c6c8700 1 -- 172.21.2.112:0/3320524945 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce240c2f30 con 0x7fce240b8410 2021-02-10T11:31:41.832 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.841 7fce1affd700 1 -- 172.21.2.112:0/3320524945 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fce14000c90 con 0x7fce240b8410 2021-02-10T11:31:41.832 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.841 7fce1affd700 1 --2- 172.21.2.112:0/3320524945 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fce0c0186c0 0x7fce0c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:41.832 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.841 7fce1affd700 1 -- 172.21.2.112:0/3320524945 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fce140327e0 con 0x7fce240b8410 2021-02-10T11:31:41.832 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.842 7fce2a4e4700 1 --2- 172.21.2.112:0/3320524945 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fce0c0186c0 0x7fce0c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:41.832 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.842 7fce2a4e4700 1 --2- 172.21.2.112:0/3320524945 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fce0c0186c0 0x7fce0c01abc0 crc :-1 s=READY pgs=517 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:41.860 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.872 7fce2c6c8700 1 -- 172.21.2.112:0/3320524945 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce24105b30 con 0x7fce240b8410 2021-02-10T11:31:41.862 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:41.875 7fce1affd700 1 -- 172.21.2.112:0/3320524945 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fce140009b0 con 0x7fce240b8410 2021-02-10T11:31:41.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:42.012 7fce2c6c8700 1 -- 172.21.2.112:0/3320524945 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"} v 0) v1 -- 0x7fce240f0b50 con 0x7fce240b8410 2021-02-10T11:31:42.186 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:42.199 7fce1affd700 1 -- 172.21.2.112:0/3320524945 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"}]=0 v22) v1 ==== 118+0+0 (secure 0 0 0) 0x7fce14000c90 con 0x7fce240b8410 2021-02-10T11:31:42.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:42.230 7fce2c6c8700 1 -- 172.21.2.112:0/3320524945 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fce0c0186c0 msgr2=0x7fce0c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:42.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:42.230 7fce2c6c8700 1 --2- 172.21.2.112:0/3320524945 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fce0c0186c0 0x7fce0c01abc0 crc :-1 s=READY pgs=517 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:42.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:42.230 7fce2c6c8700 1 -- 172.21.2.112:0/3320524945 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fce240b8410 msgr2=0x7fce240b8810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:42.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:42.230 7fce2c6c8700 1 --2- 172.21.2.112:0/3320524945 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fce240b8410 0x7fce240b8810 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 rx=0x7fce14006f20 tx=0x7fce1400b990).stop 2021-02-10T11:31:42.218 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:42.230 7fce2c6c8700 1 -- 172.21.2.112:0/3320524945 shutdown_connections 2021-02-10T11:31:42.218 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:42.230 7fce2c6c8700 1 --2- 172.21.2.112:0/3320524945 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fce0c0186c0 0x7fce0c01abc0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:42.218 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:42.230 7fce2c6c8700 1 --2- 172.21.2.112:0/3320524945 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fce240c1f80 0x7fce240c2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:42.218 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:42.230 7fce2c6c8700 1 --2- 172.21.2.112:0/3320524945 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fce240b8410 0x7fce240b8810 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:42.218 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:42.230 7fce2c6c8700 1 --2- 172.21.2.112:0/3320524945 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fce2409b930 0x7fce240b7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:42.218 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:42.230 7fce2c6c8700 1 -- 172.21.2.112:0/3320524945 >> 172.21.2.112:0/3320524945 conn(0x7fce24094e10 msgr2=0x7fce240a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:42.218 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:42.230 7fce2c6c8700 1 -- 172.21.2.112:0/3320524945 shutdown_connections 2021-02-10T11:31:42.218 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:42.230 7fce2c6c8700 1 -- 172.21.2.112:0/3320524945 wait complete. 2021-02-10T11:31:42.225 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:31:42.652 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:42.648 7f2e9880f700 1 Processor -- start 2021-02-10T11:31:42.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:42.648 7f2e9880f700 1 -- start start 2021-02-10T11:31:42.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:42.649 7f2e9880f700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f2e940e5470 0x7f2e9407e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:42.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:42.649 7f2e9880f700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f2e9407f300 0x7f2e9407f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:42.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:42.649 7f2e9880f700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e9407ed80 con 0x7f2e940e5070 2021-02-10T11:31:42.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:42.649 7f2e9880f700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e941b4010 con 0x7f2e940e5470 2021-02-10T11:31:42.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:42.649 7f2e9880f700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e940d1790 con 0x7f2e9407f300 2021-02-10T11:31:42.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:42.649 7f2e9261c700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f2e940e5070 0x7f2e941b3820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:47738/0 (socket says 172.21.2.112:47738) 2021-02-10T11:31:42.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:42.649 7f2e9261c700 1 -- 172.21.2.112:0/2290652555 learned_addr learned my addr 172.21.2.112:0/2290652555 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:31:42.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:42.649 7f2e91e1b700 1 --2- 172.21.2.112:0/2290652555 >> v2:172.21.2.112:3300/0 conn(0x7f2e940e5470 0x7f2e9407e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:42.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:42.649 7f2e9161a700 1 -- 172.21.2.112:0/2290652555 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3913641169 0 0) 0x7f2e84001180 con 0x7f2e940e5070 2021-02-10T11:31:42.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:42.649 7f2e92e1d700 1 --2- 172.21.2.112:0/2290652555 >> v2:172.21.2.120:3300/0 conn(0x7f2e9407f300 0x7f2e9407f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:42.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:42.649 7f2e9161a700 1 -- 172.21.2.112:0/2290652555 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e7c002fe0 con 0x7f2e940e5070 2021-02-10T11:31:42.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:42.649 7f2e91e1b700 1 -- 172.21.2.112:0/2290652555 >> v2:172.21.2.120:3300/0 conn(0x7f2e9407f300 msgr2=0x7f2e9407f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:42.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:42.649 7f2e91e1b700 1 --2- 172.21.2.112:0/2290652555 >> v2:172.21.2.120:3300/0 conn(0x7f2e9407f300 0x7f2e9407f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:42.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:42.649 7f2e91e1b700 1 -- 172.21.2.112:0/2290652555 >> v1:172.21.2.112:6789/0 conn(0x7f2e940e5070 legacy=0x7f2e941b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:42.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:42.649 7f2e91e1b700 1 -- 172.21.2.112:0/2290652555 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e940d19b0 con 0x7f2e940e5470 2021-02-10T11:31:42.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:42.649 7f2e92e1d700 1 --2- 172.21.2.112:0/2290652555 >> v2:172.21.2.120:3300/0 conn(0x7f2e9407f300 0x7f2e9407f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:31:42.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:42.649 7f2e91e1b700 1 --2- 172.21.2.112:0/2290652555 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2e940e5470 0x7f2e9407e6a0 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 rx=0x7f2e88006650 tx=0x7f2e88009a80).ready entity=mon.0 client_cookie=aa3882f69f5cba8c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:42.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:42.650 7f2e9161a700 1 -- 172.21.2.112:0/2290652555 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2e880009b0 con 0x7f2e940e5470 2021-02-10T11:31:42.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:42.650 7f2e9161a700 1 -- 172.21.2.112:0/2290652555 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e8800c0c0 con 0x7f2e940e5470 2021-02-10T11:31:42.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:42.650 7f2e9161a700 1 -- 172.21.2.112:0/2290652555 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2e8800c5e0 con 0x7f2e940e5470 2021-02-10T11:31:42.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:42.650 7f2e9880f700 1 -- 172.21.2.112:0/2290652555 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2e940e5470 msgr2=0x7f2e9407e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:42.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:42.650 7f2e9880f700 1 --2- 172.21.2.112:0/2290652555 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2e940e5470 0x7f2e9407e6a0 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 rx=0x7f2e88006650 tx=0x7f2e88009a80).stop 2021-02-10T11:31:42.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:42.651 7f2e9880f700 1 -- 172.21.2.112:0/2290652555 shutdown_connections 2021-02-10T11:31:42.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:42.651 7f2e9880f700 1 --2- 172.21.2.112:0/2290652555 >> v2:172.21.2.120:3300/0 conn(0x7f2e9407f300 0x7f2e9407f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:42.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:42.651 7f2e9880f700 1 --2- 172.21.2.112:0/2290652555 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2e940e5470 0x7f2e9407e6a0 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:42.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:42.651 7f2e9880f700 1 -- 172.21.2.112:0/2290652555 >> 172.21.2.112:0/2290652555 conn(0x7f2e940ce690 msgr2=0x7f2e940e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:42.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:42.651 7f2e9880f700 1 -- 172.21.2.112:0/2290652555 shutdown_connections 2021-02-10T11:31:42.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:42.651 7f2e9880f700 1 -- 172.21.2.112:0/2290652555 wait complete. 2021-02-10T11:31:42.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:42.665 7f2e9880f700 1 Processor -- start 2021-02-10T11:31:42.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:42.665 7f2e9880f700 1 -- start start 2021-02-10T11:31:42.657 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:42.666 7f2e9880f700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2e9408d400 0x7f2e94081230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:42.657 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:42.666 7f2e9880f700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2e94081850 0x7f2e940acf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:42.657 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:42.666 7f2e9880f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2e94081c50 0x7f2e940ad5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:42.657 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:42.666 7f2e9880f700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e94082110 con 0x7f2e94081c50 2021-02-10T11:31:42.657 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:42.666 7f2e9880f700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e9408d800 con 0x7f2e9408d400 2021-02-10T11:31:42.657 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:42.666 7f2e9880f700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e940adbc0 con 0x7f2e94081850 2021-02-10T11:31:42.657 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:42.666 7f2e92e1d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2e94081c50 0x7f2e940ad5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:42.657 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:42.666 7f2e92e1d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2e94081c50 0x7f2e940ad5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:48282/0 (socket says 172.21.2.112:48282) 2021-02-10T11:31:42.657 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:42.666 7f2e92e1d700 1 -- 172.21.2.112:0/743975163 learned_addr learned my addr 172.21.2.112:0/743975163 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:31:42.658 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:42.666 7f2e92e1d700 1 -- 172.21.2.112:0/743975163 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2e94081850 msgr2=0x7f2e940acf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:42.658 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:42.666 7f2e92e1d700 1 --2- 172.21.2.112:0/743975163 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2e94081850 0x7f2e940acf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:42.658 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:42.666 7f2e92e1d700 1 -- 172.21.2.112:0/743975163 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2e9408d400 msgr2=0x7f2e94081230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:42.658 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:42.667 7f2e92e1d700 1 --2- 172.21.2.112:0/743975163 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2e9408d400 0x7f2e94081230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:42.658 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:42.667 7f2e92e1d700 1 -- 172.21.2.112:0/743975163 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e94187fc0 con 0x7f2e94081c50 2021-02-10T11:31:42.658 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:42.667 7f2e92e1d700 1 --2- 172.21.2.112:0/743975163 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2e94081c50 0x7f2e940ad5a0 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 rx=0x7f2e8c009980 tx=0x7f2e8c00bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:42.658 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:42.667 7f2e82ffd700 1 -- 172.21.2.112:0/743975163 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2e8c00c2f0 con 0x7f2e94081c50 2021-02-10T11:31:42.658 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:42.667 7f2e82ffd700 1 -- 172.21.2.112:0/743975163 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e8c006790 con 0x7f2e94081c50 2021-02-10T11:31:42.658 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:42.667 7f2e82ffd700 1 -- 172.21.2.112:0/743975163 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2e8c00c870 con 0x7f2e94081c50 2021-02-10T11:31:42.658 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:42.667 7f2e9880f700 1 -- 172.21.2.112:0/743975163 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e94187fc0 con 0x7f2e94081c50 2021-02-10T11:31:42.658 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:42.667 7f2e9880f700 1 -- 172.21.2.112:0/743975163 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e94187fc0 con 0x7f2e94081c50 2021-02-10T11:31:42.658 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:42.668 7f2e82ffd700 1 -- 172.21.2.112:0/743975163 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2e8c0011f0 con 0x7f2e94081c50 2021-02-10T11:31:42.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:42.668 7f2e82ffd700 1 --2- 172.21.2.112:0/743975163 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2e780186c0 0x7f2e7801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:42.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:42.668 7f2e82ffd700 1 -- 172.21.2.112:0/743975163 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f2e8c032960 con 0x7f2e94081c50 2021-02-10T11:31:42.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:42.668 7f2e9261c700 1 --2- 172.21.2.112:0/743975163 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2e780186c0 0x7f2e7801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:42.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:42.669 7f2e9261c700 1 --2- 172.21.2.112:0/743975163 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2e780186c0 0x7f2e7801abc0 crc :-1 s=READY pgs=518 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:42.686 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:42.698 7f2e9880f700 1 -- 172.21.2.112:0/743975163 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e9408d800 con 0x7f2e94081c50 2021-02-10T11:31:42.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:42.701 7f2e82ffd700 1 -- 172.21.2.112:0/743975163 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2e8c00be10 con 0x7f2e94081c50 2021-02-10T11:31:42.819 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:42.829 7f2e9880f700 1 -- 172.21.2.112:0/743975163 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2e9408d800 con 0x7f2e94081c50 2021-02-10T11:31:42.819 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:42.830 7f2e82ffd700 1 -- 172.21.2.112:0/743975163 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+2099 (secure 0 0 0) 0x7f2e8c0011f0 con 0x7f2e94081c50 2021-02-10T11:31:42.819 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 22 2021-02-10T11:31:42.819 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:42.832 7f2e9880f700 1 -- 172.21.2.112:0/743975163 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2e780186c0 msgr2=0x7f2e7801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:42.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:42.832 7f2e9880f700 1 --2- 172.21.2.112:0/743975163 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2e780186c0 0x7f2e7801abc0 crc :-1 s=READY pgs=518 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:42.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:42.832 7f2e9880f700 1 -- 172.21.2.112:0/743975163 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2e94081c50 msgr2=0x7f2e940ad5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:42.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:42.832 7f2e9880f700 1 --2- 172.21.2.112:0/743975163 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2e94081c50 0x7f2e940ad5a0 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 rx=0x7f2e8c009980 tx=0x7f2e8c00bcd0).stop 2021-02-10T11:31:42.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:42.832 7f2e9880f700 1 -- 172.21.2.112:0/743975163 shutdown_connections 2021-02-10T11:31:42.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:42.832 7f2e9880f700 1 --2- 172.21.2.112:0/743975163 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2e780186c0 0x7f2e7801abc0 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:42.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:42.832 7f2e9880f700 1 --2- 172.21.2.112:0/743975163 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2e94081c50 0x7f2e940ad5a0 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:42.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:42.832 7f2e9880f700 1 --2- 172.21.2.112:0/743975163 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2e94081850 0x7f2e940acf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:42.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:42.832 7f2e9880f700 1 --2- 172.21.2.112:0/743975163 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2e9408d400 0x7f2e94081230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:42.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:42.832 7f2e9880f700 1 -- 172.21.2.112:0/743975163 >> 172.21.2.112:0/743975163 conn(0x7f2e941ac620 msgr2=0x7f2e940896e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:42.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:42.832 7f2e9880f700 1 -- 172.21.2.112:0/743975163 shutdown_connections 2021-02-10T11:31:42.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:42.832 7f2e9880f700 1 -- 172.21.2.112:0/743975163 wait complete. 2021-02-10T11:31:42.820 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:31:42.821 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":22,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7155,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/1957863967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":1957863967},{"type":"v1","addr":"172.21.2.112:6833","nonce":1957863967}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":18}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:31:40.190262","modified":"2021-02-10 11:31:42.195824","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5534},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5534":{"gid":5534,"name":"b","rank":0,"incarnation":19,"state":"up:active","state_seq":3,"addr":"172.21.2.120:6835/1513827094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":1513827094},{"type":"v1","addr":"172.21.2.120:6835","nonce":1513827094}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2021-02-10T11:31:42.828 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:31:43.255 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.250 7ff047762700 1 Processor -- start 2021-02-10T11:31:43.255 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.250 7ff047762700 1 -- start start 2021-02-10T11:31:43.255 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.250 7ff047762700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7ff0400e5070 0x7ff0401b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:43.255 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.251 7ff047762700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff0400e5470 0x7ff040080310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:43.255 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.251 7ff047762700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff04007ed80 con 0x7ff0401b3e40 2021-02-10T11:31:43.255 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.251 7ff047762700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0400d1790 con 0x7ff0400e5070 2021-02-10T11:31:43.255 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.251 7ff047762700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0400d19b0 con 0x7ff0400e5470 2021-02-10T11:31:43.255 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.251 7ff04557e700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7ff0400e5070 0x7ff0401b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:43.255 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.251 7ff044d7d700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7ff0401b3e40 0x7ff04007d690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:41310/0 (socket says 172.21.2.112:41310) 2021-02-10T11:31:43.255 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.251 7ff044d7d700 1 -- 172.21.2.112:0/897854135 learned_addr learned my addr 172.21.2.112:0/897854135 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:31:43.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.251 7ff045d7f700 1 --2- 172.21.2.112:0/897854135 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff0400e5470 0x7ff040080310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:43.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.251 7ff04557e700 1 -- 172.21.2.112:0/897854135 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff0400e5470 msgr2=0x7ff040080310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:43.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.251 7ff04557e700 1 --2- 172.21.2.112:0/897854135 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff0400e5470 0x7ff040080310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:43.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.251 7ff04557e700 1 -- 172.21.2.112:0/897854135 >> v1:172.21.2.120:6789/0 conn(0x7ff0401b3e40 legacy=0x7ff04007d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:43.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.251 7ff04557e700 1 -- 172.21.2.112:0/897854135 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0400d1bd0 con 0x7ff0400e5070 2021-02-10T11:31:43.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.252 7ff04557e700 1 --2- 172.21.2.112:0/897854135 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff0400e5070 0x7ff0401b3820 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 rx=0x7ff03c006650 tx=0x7ff03c009a80).ready entity=mon.0 client_cookie=a354998e810b036b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:43.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.252 7ff033fff700 1 -- 172.21.2.112:0/897854135 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff03c00c0c0 con 0x7ff0400e5070 2021-02-10T11:31:43.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.252 7ff033fff700 1 -- 172.21.2.112:0/897854135 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff03c00c340 con 0x7ff0400e5070 2021-02-10T11:31:43.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.252 7ff033fff700 1 -- 172.21.2.112:0/897854135 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff03c00c860 con 0x7ff0400e5070 2021-02-10T11:31:43.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.252 7ff047762700 1 -- 172.21.2.112:0/897854135 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff0400e5070 msgr2=0x7ff0401b3820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:43.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.252 7ff047762700 1 --2- 172.21.2.112:0/897854135 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff0400e5070 0x7ff0401b3820 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 rx=0x7ff03c006650 tx=0x7ff03c009a80).stop 2021-02-10T11:31:43.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.253 7ff047762700 1 -- 172.21.2.112:0/897854135 shutdown_connections 2021-02-10T11:31:43.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.253 7ff047762700 1 --2- 172.21.2.112:0/897854135 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff0400e5470 0x7ff040080310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:43.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.253 7ff047762700 1 --2- 172.21.2.112:0/897854135 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff0400e5070 0x7ff0401b3820 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:43.257 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.253 7ff047762700 1 -- 172.21.2.112:0/897854135 >> 172.21.2.112:0/897854135 conn(0x7ff0400ce690 msgr2=0x7ff0400e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:43.257 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.253 7ff047762700 1 -- 172.21.2.112:0/897854135 shutdown_connections 2021-02-10T11:31:43.257 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.253 7ff047762700 1 -- 172.21.2.112:0/897854135 wait complete. 2021-02-10T11:31:43.258 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.268 7ff047762700 1 Processor -- start 2021-02-10T11:31:43.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.268 7ff047762700 1 -- start start 2021-02-10T11:31:43.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.268 7ff047762700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff04008fca0 0x7ff0400ad560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:43.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.268 7ff047762700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff0400adb80 0x7ff040190fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:43.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.268 7ff047762700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff040108010 0x7ff0401915f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:43.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.268 7ff047762700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff040197270 con 0x7ff04008fca0 2021-02-10T11:31:43.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.268 7ff047762700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff040191c10 con 0x7ff040108010 2021-02-10T11:31:43.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.268 7ff047762700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff040191e30 con 0x7ff0400adb80 2021-02-10T11:31:43.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.269 7ff04557e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff04008fca0 0x7ff0400ad560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:43.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.269 7ff04557e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff04008fca0 0x7ff0400ad560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:48294/0 (socket says 172.21.2.112:48294) 2021-02-10T11:31:43.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.269 7ff04557e700 1 -- 172.21.2.112:0/3589692027 learned_addr learned my addr 172.21.2.112:0/3589692027 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:31:43.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.269 7ff044d7d700 1 --2- 172.21.2.112:0/3589692027 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff0400adb80 0x7ff040190fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:43.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.269 7ff045d7f700 1 --2- 172.21.2.112:0/3589692027 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff040108010 0x7ff0401915f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:43.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.269 7ff04557e700 1 -- 172.21.2.112:0/3589692027 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff0400adb80 msgr2=0x7ff040190fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:43.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.269 7ff04557e700 1 --2- 172.21.2.112:0/3589692027 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff0400adb80 0x7ff040190fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:43.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.269 7ff04557e700 1 -- 172.21.2.112:0/3589692027 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff040108010 msgr2=0x7ff0401915f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:43.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.269 7ff04557e700 1 --2- 172.21.2.112:0/3589692027 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff040108010 0x7ff0401915f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:43.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.269 7ff04557e700 1 -- 172.21.2.112:0/3589692027 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff040192690 con 0x7ff04008fca0 2021-02-10T11:31:43.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.269 7ff04557e700 1 --2- 172.21.2.112:0/3589692027 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff04008fca0 0x7ff0400ad560 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 rx=0x7ff03c009630 tx=0x7ff03c00b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:43.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.269 7ff031ffb700 1 -- 172.21.2.112:0/3589692027 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff03c0093a0 con 0x7ff04008fca0 2021-02-10T11:31:43.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.269 7ff031ffb700 1 -- 172.21.2.112:0/3589692027 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff03c00bd70 con 0x7ff04008fca0 2021-02-10T11:31:43.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.269 7ff031ffb700 1 -- 172.21.2.112:0/3589692027 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff03c005560 con 0x7ff04008fca0 2021-02-10T11:31:43.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.270 7ff047762700 1 -- 172.21.2.112:0/3589692027 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff040192690 con 0x7ff04008fca0 2021-02-10T11:31:43.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.270 7ff047762700 1 -- 172.21.2.112:0/3589692027 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff040192c20 con 0x7ff04008fca0 2021-02-10T11:31:43.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.270 7ff031ffb700 1 -- 172.21.2.112:0/3589692027 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff03c007020 con 0x7ff04008fca0 2021-02-10T11:31:43.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.270 7ff031ffb700 1 --2- 172.21.2.112:0/3589692027 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff0200186c0 0x7ff02001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:43.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.270 7ff031ffb700 1 -- 172.21.2.112:0/3589692027 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5573+0+0 (secure 0 0 0) 0x7ff03c032570 con 0x7ff04008fca0 2021-02-10T11:31:43.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.271 7ff044d7d700 1 --2- 172.21.2.112:0/3589692027 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff0200186c0 0x7ff02001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:43.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.271 7ff044d7d700 1 --2- 172.21.2.112:0/3589692027 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff0200186c0 0x7ff02001abc0 crc :-1 s=READY pgs=519 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:43.288 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.301 7ff047762700 1 -- 172.21.2.112:0/3589692027 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff040107180 con 0x7ff04008fca0 2021-02-10T11:31:43.290 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.303 7ff031ffb700 1 -- 172.21.2.112:0/3589692027 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff03c009340 con 0x7ff04008fca0 2021-02-10T11:31:43.415 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.428 7ff047762700 1 -- 172.21.2.112:0/3589692027 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff04007ed80 con 0x7ff04008fca0 2021-02-10T11:31:43.416 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.429 7ff031ffb700 1 -- 172.21.2.112:0/3589692027 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v38) v1 ==== 74+0+13993 (secure 0 0 0) 0x7ff03c007000 con 0x7ff04008fca0 2021-02-10T11:31:43.416 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:31:43.416 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":38,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:31:41.192016","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:31:38.162654","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":7,"pool_name":"cephfs_data","create_time":"2021-02-10 11:31:39.233986","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:31:43.417 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.430 7ff047762700 1 -- 172.21.2.112:0/3589692027 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff0200186c0 msgr2=0x7ff02001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:43.418 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.430 7ff047762700 1 --2- 172.21.2.112:0/3589692027 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff0200186c0 0x7ff02001abc0 crc :-1 s=READY pgs=519 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:43.418 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.430 7ff047762700 1 -- 172.21.2.112:0/3589692027 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff04008fca0 msgr2=0x7ff0400ad560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:43.418 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.430 7ff047762700 1 --2- 172.21.2.112:0/3589692027 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff04008fca0 0x7ff0400ad560 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 rx=0x7ff03c009630 tx=0x7ff03c00b980).stop 2021-02-10T11:31:43.418 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.430 7ff047762700 1 -- 172.21.2.112:0/3589692027 shutdown_connections 2021-02-10T11:31:43.418 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.430 7ff047762700 1 --2- 172.21.2.112:0/3589692027 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff0200186c0 0x7ff02001abc0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:43.418 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.430 7ff047762700 1 --2- 172.21.2.112:0/3589692027 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff040108010 0x7ff0401915f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:43.418 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.430 7ff047762700 1 --2- 172.21.2.112:0/3589692027 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff0400adb80 0x7ff040190fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:43.418 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.430 7ff047762700 1 --2- 172.21.2.112:0/3589692027 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff04008fca0 0x7ff0400ad560 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:43.418 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.430 7ff047762700 1 -- 172.21.2.112:0/3589692027 >> 172.21.2.112:0/3589692027 conn(0x7ff0401986e0 msgr2=0x7ff040093c60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:43.418 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.430 7ff047762700 1 -- 172.21.2.112:0/3589692027 shutdown_connections 2021-02-10T11:31:43.419 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.430 7ff047762700 1 -- 172.21.2.112:0/3589692027 wait complete. 2021-02-10T11:31:43.427 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:31:43.857 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.852 7f92b25c6700 1 Processor -- start 2021-02-10T11:31:43.857 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.852 7f92b25c6700 1 -- start start 2021-02-10T11:31:43.857 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.853 7f92b25c6700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f92ac0e5470 0x7f92ac07e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:43.857 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.853 7f92b25c6700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f92ac07f300 0x7f92ac07f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:43.857 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.853 7f92b25c6700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92ac07ed80 con 0x7f92ac0e5070 2021-02-10T11:31:43.857 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.853 7f92b25c6700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92ac1b4010 con 0x7f92ac07f300 2021-02-10T11:31:43.857 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.853 7f92b25c6700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92ac0d1790 con 0x7f92ac0e5470 2021-02-10T11:31:43.857 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.853 7f92b0be3700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f92ac07f300 0x7f92ac07f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:43.857 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.853 7f92b0be3700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f92ac07f300 0x7f92ac07f700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:48306/0 (socket says 172.21.2.112:48306) 2021-02-10T11:31:43.857 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.853 7f92b0be3700 1 -- 172.21.2.112:0/2040529335 learned_addr learned my addr 172.21.2.112:0/2040529335 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:31:43.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.854 7f92ab7fe700 1 --2- 172.21.2.112:0/2040529335 >> v2:172.21.2.120:3300/0 conn(0x7f92ac0e5470 0x7f92ac07e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:43.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.854 7f92b0be3700 1 -- 172.21.2.112:0/2040529335 >> v2:172.21.2.120:3300/0 conn(0x7f92ac0e5470 msgr2=0x7f92ac07e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:43.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.854 7f92b0be3700 1 --2- 172.21.2.112:0/2040529335 >> v2:172.21.2.120:3300/0 conn(0x7f92ac0e5470 0x7f92ac07e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:43.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.854 7f92b0be3700 1 -- 172.21.2.112:0/2040529335 >> v1:172.21.2.120:6789/0 conn(0x7f92ac0e5070 legacy=0x7f92ac1b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:43.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.854 7f92b0be3700 1 -- 172.21.2.112:0/2040529335 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92ac0d19b0 con 0x7f92ac07f300 2021-02-10T11:31:43.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.854 7f92ab7fe700 1 --2- 172.21.2.112:0/2040529335 >> v2:172.21.2.120:3300/0 conn(0x7f92ac0e5470 0x7f92ac07e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:31:43.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.854 7f92b0be3700 1 --2- 172.21.2.112:0/2040529335 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f92ac07f300 0x7f92ac07f700 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 rx=0x7f92a4006b80 tx=0x7f92a4009b90).ready entity=mon.0 client_cookie=b86ca653e887daa5 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:43.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.854 7f92aaffd700 1 -- 172.21.2.112:0/2040529335 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f92a40022e0 con 0x7f92ac07f300 2021-02-10T11:31:43.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.854 7f92aaffd700 1 -- 172.21.2.112:0/2040529335 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92a4002620 con 0x7f92ac07f300 2021-02-10T11:31:43.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.854 7f92aaffd700 1 -- 172.21.2.112:0/2040529335 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f92a4002b00 con 0x7f92ac07f300 2021-02-10T11:31:43.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.854 7f92b25c6700 1 -- 172.21.2.112:0/2040529335 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f92ac07f300 msgr2=0x7f92ac07f700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:43.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.854 7f92b25c6700 1 --2- 172.21.2.112:0/2040529335 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f92ac07f300 0x7f92ac07f700 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 rx=0x7f92a4006b80 tx=0x7f92a4009b90).stop 2021-02-10T11:31:43.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.855 7f92b25c6700 1 -- 172.21.2.112:0/2040529335 shutdown_connections 2021-02-10T11:31:43.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.855 7f92b25c6700 1 --2- 172.21.2.112:0/2040529335 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f92ac07f300 0x7f92ac07f700 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:43.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.855 7f92b25c6700 1 --2- 172.21.2.112:0/2040529335 >> v2:172.21.2.120:3300/0 conn(0x7f92ac0e5470 0x7f92ac07e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:43.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.855 7f92b25c6700 1 -- 172.21.2.112:0/2040529335 >> 172.21.2.112:0/2040529335 conn(0x7f92ac0ce690 msgr2=0x7f92ac0e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:43.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.855 7f92b25c6700 1 -- 172.21.2.112:0/2040529335 shutdown_connections 2021-02-10T11:31:43.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.855 7f92b25c6700 1 -- 172.21.2.112:0/2040529335 wait complete. 2021-02-10T11:31:43.860 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.870 7f92b25c6700 1 Processor -- start 2021-02-10T11:31:43.860 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.870 7f92b25c6700 1 -- start start 2021-02-10T11:31:43.861 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.870 7f92b25c6700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f92ac08ce50 0x7f92ac093020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:43.861 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.870 7f92b25c6700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f92ac08d250 0x7f92ac093640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:43.861 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.870 7f92b25c6700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f92ac08c440 0x7f92ac087ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:43.861 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.870 7f92b25c6700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92ac08c8d0 con 0x7f92ac08c440 2021-02-10T11:31:43.861 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.870 7f92b25c6700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92ac089430 con 0x7f92ac08ce50 2021-02-10T11:31:43.861 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.870 7f92b25c6700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92ac088f40 con 0x7f92ac08d250 2021-02-10T11:31:43.861 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.871 7f92b0be3700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f92ac08c440 0x7f92ac087ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:43.861 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.871 7f92b0be3700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f92ac08c440 0x7f92ac087ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:48312/0 (socket says 172.21.2.112:48312) 2021-02-10T11:31:43.861 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.871 7f92b0be3700 1 -- 172.21.2.112:0/2383451830 learned_addr learned my addr 172.21.2.112:0/2383451830 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:31:43.861 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.871 7f92abfff700 1 --2- 172.21.2.112:0/2383451830 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f92ac08ce50 0x7f92ac093020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:43.861 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.871 7f92b0be3700 1 -- 172.21.2.112:0/2383451830 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f92ac08d250 msgr2=0x7f92ac093640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:43.861 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.871 7f92ab7fe700 1 --2- 172.21.2.112:0/2383451830 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f92ac08d250 0x7f92ac093640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:43.861 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.871 7f92b0be3700 1 --2- 172.21.2.112:0/2383451830 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f92ac08d250 0x7f92ac093640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:43.862 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.871 7f92b0be3700 1 -- 172.21.2.112:0/2383451830 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f92ac08ce50 msgr2=0x7f92ac093020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:43.862 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.871 7f92b0be3700 1 --2- 172.21.2.112:0/2383451830 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f92ac08ce50 0x7f92ac093020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:43.862 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.871 7f92b0be3700 1 -- 172.21.2.112:0/2383451830 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92ac088510 con 0x7f92ac08c440 2021-02-10T11:31:43.862 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.871 7f92ab7fe700 1 --2- 172.21.2.112:0/2383451830 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f92ac08d250 0x7f92ac093640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:31:43.862 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.871 7f92abfff700 1 --2- 172.21.2.112:0/2383451830 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f92ac08ce50 0x7f92ac093020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:31:43.862 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.871 7f92b0be3700 1 --2- 172.21.2.112:0/2383451830 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f92ac08c440 0x7f92ac087ef0 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 rx=0x7f92a4009640 tx=0x7f92a400b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:43.862 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.872 7f92a8ff9700 1 -- 172.21.2.112:0/2383451830 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f92a400bd50 con 0x7f92ac08c440 2021-02-10T11:31:43.862 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.872 7f92a8ff9700 1 -- 172.21.2.112:0/2383451830 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92a4006240 con 0x7f92ac08c440 2021-02-10T11:31:43.862 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.872 7f92a8ff9700 1 -- 172.21.2.112:0/2383451830 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f92a40054a0 con 0x7f92ac08c440 2021-02-10T11:31:43.862 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.872 7f92b25c6700 1 -- 172.21.2.112:0/2383451830 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92ac088510 con 0x7f92ac08c440 2021-02-10T11:31:43.862 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.872 7f92b25c6700 1 -- 172.21.2.112:0/2383451830 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92ac088aa0 con 0x7f92ac08c440 2021-02-10T11:31:43.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.873 7f92a8ff9700 1 -- 172.21.2.112:0/2383451830 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f92a4006420 con 0x7f92ac08c440 2021-02-10T11:31:43.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.873 7f92a8ff9700 1 --2- 172.21.2.112:0/2383451830 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f92900186c0 0x7f929001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:43.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.873 7f92a8ff9700 1 -- 172.21.2.112:0/2383451830 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f92a40325d0 con 0x7f92ac08c440 2021-02-10T11:31:43.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.874 7f92abfff700 1 --2- 172.21.2.112:0/2383451830 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f92900186c0 0x7f929001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:43.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.874 7f92abfff700 1 --2- 172.21.2.112:0/2383451830 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f92900186c0 0x7f929001abc0 crc :-1 s=READY pgs=520 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:43.890 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.903 7f92b25c6700 1 -- 172.21.2.112:0/2383451830 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92ac0893e0 con 0x7f92ac08c440 2021-02-10T11:31:43.892 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:43.905 7f92a8ff9700 1 -- 172.21.2.112:0/2383451830 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f92a4009100 con 0x7f92ac08c440 2021-02-10T11:31:44.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:44.036 7f92b25c6700 1 -- 172.21.2.112:0/2383451830 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f92ac0893e0 con 0x7f92ac08c440 2021-02-10T11:31:44.027 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:44.039 7f92a8ff9700 1 -- 172.21.2.112:0/2383451830 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v17) v1 ==== 184+0+0 (secure 0 0 0) 0x7f92a40159e0 con 0x7f92ac08c440 2021-02-10T11:31:44.033 INFO:teuthology.orchestra.run.gibba012.stderr:updated caps for client.0 2021-02-10T11:31:44.033 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:44.044 7f92b25c6700 1 -- 172.21.2.112:0/2383451830 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f92900186c0 msgr2=0x7f929001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:44.033 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:44.044 7f92b25c6700 1 --2- 172.21.2.112:0/2383451830 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f92900186c0 0x7f929001abc0 crc :-1 s=READY pgs=520 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:44.033 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:44.044 7f92b25c6700 1 -- 172.21.2.112:0/2383451830 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f92ac08c440 msgr2=0x7f92ac087ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:44.033 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:44.044 7f92b25c6700 1 --2- 172.21.2.112:0/2383451830 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f92ac08c440 0x7f92ac087ef0 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 rx=0x7f92a4009640 tx=0x7f92a400b960).stop 2021-02-10T11:31:44.034 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:44.044 7f92b25c6700 1 -- 172.21.2.112:0/2383451830 shutdown_connections 2021-02-10T11:31:44.034 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:44.044 7f92b25c6700 1 --2- 172.21.2.112:0/2383451830 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f92900186c0 0x7f929001abc0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:44.034 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:44.044 7f92b25c6700 1 --2- 172.21.2.112:0/2383451830 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f92ac08c440 0x7f92ac087ef0 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:44.034 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:44.044 7f92b25c6700 1 --2- 172.21.2.112:0/2383451830 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f92ac08d250 0x7f92ac093640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:44.034 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:44.044 7f92b25c6700 1 --2- 172.21.2.112:0/2383451830 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f92ac08ce50 0x7f92ac093020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:44.035 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:44.044 7f92b25c6700 1 -- 172.21.2.112:0/2383451830 >> 172.21.2.112:0/2383451830 conn(0x7f92ac08dde0 msgr2=0x7f92ac1ac5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:44.035 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:44.045 7f92b25c6700 1 -- 172.21.2.112:0/2383451830 shutdown_connections 2021-02-10T11:31:44.035 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:44.045 7f92b25c6700 1 -- 172.21.2.112:0/2383451830 wait complete. 2021-02-10T11:31:44.041 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:31:44.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:44.475 7fe4f43b6700 1 Processor -- start 2021-02-10T11:31:44.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:44.475 7fe4f43b6700 1 -- start start 2021-02-10T11:31:44.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:44.476 7fe4f43b6700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fe4ec1b0470 0x7fe4ec1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:44.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:44.476 7fe4f43b6700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fe4ec0ef610 0x7fe4ec0efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:44.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:44.476 7fe4f43b6700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe4ec0f0030 0x7fe4ec0f0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:44.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:44.476 7fe4f43b6700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4ec0e5ea0 con 0x7fe4ec1b0470 2021-02-10T11:31:44.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:44.476 7fe4f43b6700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4ec0f0b50 con 0x7fe4ec0ef610 2021-02-10T11:31:44.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:44.476 7fe4f43b6700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4ec0e52e0 con 0x7fe4ec0f0030 2021-02-10T11:31:44.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:44.476 7fe4f21d2700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fe4ec1b0470 0x7fe4ec1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:44.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:44.476 7fe4f21d2700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fe4ec1b0470 0x7fe4ec1b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:48316/0 (socket says 172.21.2.112:48316) 2021-02-10T11:31:44.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:44.476 7fe4f21d2700 1 -- 172.21.2.112:0/3826639749 learned_addr learned my addr 172.21.2.112:0/3826639749 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:31:44.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:44.476 7fe4f21d2700 1 -- 172.21.2.112:0/3826639749 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe4ec0f0030 msgr2=0x7fe4ec0f0430 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:31:44.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:44.476 7fe4f21d2700 1 --2- 172.21.2.112:0/3826639749 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe4ec0f0030 0x7fe4ec0f0430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:44.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:44.476 7fe4f21d2700 1 -- 172.21.2.112:0/3826639749 >> v2:172.21.2.120:3300/0 conn(0x7fe4ec0ef610 msgr2=0x7fe4ec0efa10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:31:44.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:44.476 7fe4f21d2700 1 --2- 172.21.2.112:0/3826639749 >> v2:172.21.2.120:3300/0 conn(0x7fe4ec0ef610 0x7fe4ec0efa10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:44.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:44.476 7fe4f21d2700 1 -- 172.21.2.112:0/3826639749 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4ec0f10d0 con 0x7fe4ec1b0470 2021-02-10T11:31:44.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:44.476 7fe4f21d2700 1 --2- 172.21.2.112:0/3826639749 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe4ec1b0470 0x7fe4ec1b0870 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 rx=0x7fe4e0006700 tx=0x7fe4e0009b80).ready entity=mon.0 client_cookie=94712cc552a26566 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:44.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:44.476 7fe4f11d0700 1 -- 172.21.2.112:0/3826639749 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe4e000c0c0 con 0x7fe4ec1b0470 2021-02-10T11:31:44.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:44.476 7fe4f11d0700 1 -- 172.21.2.112:0/3826639749 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4e000c440 con 0x7fe4ec1b0470 2021-02-10T11:31:44.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:44.476 7fe4f11d0700 1 -- 172.21.2.112:0/3826639749 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe4e000c920 con 0x7fe4ec1b0470 2021-02-10T11:31:44.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:44.476 7fe4f43b6700 1 -- 172.21.2.112:0/3826639749 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe4ec1b0470 msgr2=0x7fe4ec1b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:44.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:44.476 7fe4f43b6700 1 --2- 172.21.2.112:0/3826639749 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe4ec1b0470 0x7fe4ec1b0870 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 rx=0x7fe4e0006700 tx=0x7fe4e0009b80).stop 2021-02-10T11:31:44.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:44.477 7fe4f43b6700 1 -- 172.21.2.112:0/3826639749 shutdown_connections 2021-02-10T11:31:44.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:44.477 7fe4f43b6700 1 --2- 172.21.2.112:0/3826639749 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe4ec0f0030 0x7fe4ec0f0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:44.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:44.477 7fe4f43b6700 1 --2- 172.21.2.112:0/3826639749 >> v2:172.21.2.120:3300/0 conn(0x7fe4ec0ef610 0x7fe4ec0efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:44.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:44.477 7fe4f43b6700 1 --2- 172.21.2.112:0/3826639749 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe4ec1b0470 0x7fe4ec1b0870 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:44.481 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:44.477 7fe4f43b6700 1 -- 172.21.2.112:0/3826639749 >> 172.21.2.112:0/3826639749 conn(0x7fe4ec0ce3e0 msgr2=0x7fe4ec0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:44.481 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:44.477 7fe4f43b6700 1 -- 172.21.2.112:0/3826639749 shutdown_connections 2021-02-10T11:31:44.481 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:44.477 7fe4f43b6700 1 -- 172.21.2.112:0/3826639749 wait complete. 2021-02-10T11:31:44.482 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:44.492 7fe4f43b6700 1 Processor -- start 2021-02-10T11:31:44.482 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:44.492 7fe4f43b6700 1 -- start start 2021-02-10T11:31:44.482 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:44.492 7fe4f43b6700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe4ec1b6de0 0x7fe4ec1b63f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:44.482 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:44.492 7fe4f43b6700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe4ec18f3f0 0x7fe4ec18f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:44.482 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:44.492 7fe4f43b6700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe4ec18fe10 0x7fe4ec18ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:44.483 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:44.492 7fe4f43b6700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4ec18d950 con 0x7fe4ec18f3f0 2021-02-10T11:31:44.483 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:44.492 7fe4f43b6700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4ec1b71e0 con 0x7fe4ec18fe10 2021-02-10T11:31:44.483 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:44.492 7fe4f43b6700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4ec1b7400 con 0x7fe4ec1b6de0 2021-02-10T11:31:44.483 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:44.492 7fe4f19d1700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe4ec18f3f0 0x7fe4ec18f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:44.483 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:44.492 7fe4f19d1700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe4ec18f3f0 0x7fe4ec18f7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:48322/0 (socket says 172.21.2.112:48322) 2021-02-10T11:31:44.483 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:44.492 7fe4f19d1700 1 -- 172.21.2.112:0/3835163191 learned_addr learned my addr 172.21.2.112:0/3835163191 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:31:44.483 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:44.493 7fe4f19d1700 1 -- 172.21.2.112:0/3835163191 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe4ec1b6de0 msgr2=0x7fe4ec1b63f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:44.483 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:44.493 7fe4f19d1700 1 --2- 172.21.2.112:0/3835163191 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe4ec1b6de0 0x7fe4ec1b63f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:44.483 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:44.493 7fe4f19d1700 1 -- 172.21.2.112:0/3835163191 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe4ec18fe10 msgr2=0x7fe4ec18ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:44.483 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:44.493 7fe4f19d1700 1 --2- 172.21.2.112:0/3835163191 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe4ec18fe10 0x7fe4ec18ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:44.483 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:44.493 7fe4f19d1700 1 -- 172.21.2.112:0/3835163191 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4ec190210 con 0x7fe4ec18f3f0 2021-02-10T11:31:44.483 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:44.493 7fe4f19d1700 1 --2- 172.21.2.112:0/3835163191 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe4ec18f3f0 0x7fe4ec18f7f0 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 rx=0x7fe4e800b5c0 tx=0x7fe4e800b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:44.483 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:44.493 7fe4deffd700 1 -- 172.21.2.112:0/3835163191 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe4e800bd50 con 0x7fe4ec18f3f0 2021-02-10T11:31:44.484 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:44.493 7fe4deffd700 1 -- 172.21.2.112:0/3835163191 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4e8006030 con 0x7fe4ec18f3f0 2021-02-10T11:31:44.484 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:44.493 7fe4f43b6700 1 -- 172.21.2.112:0/3835163191 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4ec190210 con 0x7fe4ec18f3f0 2021-02-10T11:31:44.484 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:44.493 7fe4deffd700 1 -- 172.21.2.112:0/3835163191 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe4e80054d0 con 0x7fe4ec18f3f0 2021-02-10T11:31:44.484 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:44.493 7fe4f43b6700 1 -- 172.21.2.112:0/3835163191 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4ec18e700 con 0x7fe4ec18f3f0 2021-02-10T11:31:44.484 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:44.494 7fe4deffd700 1 -- 172.21.2.112:0/3835163191 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe4e8006210 con 0x7fe4ec18f3f0 2021-02-10T11:31:44.484 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:44.494 7fe4deffd700 1 --2- 172.21.2.112:0/3835163191 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe4d4018730 0x7fe4d401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:44.484 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:44.494 7fe4deffd700 1 -- 172.21.2.112:0/3835163191 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fe4e8032640 con 0x7fe4ec18f3f0 2021-02-10T11:31:44.484 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:44.494 7fe4f21d2700 1 --2- 172.21.2.112:0/3835163191 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe4d4018730 0x7fe4d401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:44.484 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:44.495 7fe4f21d2700 1 --2- 172.21.2.112:0/3835163191 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe4d4018730 0x7fe4d401ac10 crc :-1 s=READY pgs=521 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:44.513 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:44.526 7fe4f43b6700 1 -- 172.21.2.112:0/3835163191 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4ec1059e0 con 0x7fe4ec18f3f0 2021-02-10T11:31:44.516 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:44.529 7fe4deffd700 1 -- 172.21.2.112:0/3835163191 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe4e8000940 con 0x7fe4ec18f3f0 2021-02-10T11:31:44.649 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:44.661 7fe4f43b6700 1 -- 172.21.2.112:0/3835163191 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fe4ec0f0b50 con 0x7fe4ec18f3f0 2021-02-10T11:31:44.652 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:44.665 7fe4deffd700 1 -- 172.21.2.112:0/3835163191 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v18) v1 ==== 184+0+0 (secure 0 0 0) 0x7fe4e8006030 con 0x7fe4ec18f3f0 2021-02-10T11:31:44.656 INFO:teuthology.orchestra.run.gibba012.stderr:updated caps for client.1 2021-02-10T11:31:44.658 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:44.671 7fe4f43b6700 1 -- 172.21.2.112:0/3835163191 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe4d4018730 msgr2=0x7fe4d401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:44.658 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:44.671 7fe4f43b6700 1 --2- 172.21.2.112:0/3835163191 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe4d4018730 0x7fe4d401ac10 crc :-1 s=READY pgs=521 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:44.658 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:44.671 7fe4f43b6700 1 -- 172.21.2.112:0/3835163191 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe4ec18f3f0 msgr2=0x7fe4ec18f7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:44.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:44.671 7fe4f43b6700 1 --2- 172.21.2.112:0/3835163191 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe4ec18f3f0 0x7fe4ec18f7f0 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 rx=0x7fe4e800b5c0 tx=0x7fe4e800b960).stop 2021-02-10T11:31:44.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:44.671 7fe4f43b6700 1 -- 172.21.2.112:0/3835163191 shutdown_connections 2021-02-10T11:31:44.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:44.671 7fe4f43b6700 1 --2- 172.21.2.112:0/3835163191 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe4d4018730 0x7fe4d401ac10 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:44.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:44.671 7fe4f43b6700 1 --2- 172.21.2.112:0/3835163191 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe4ec18fe10 0x7fe4ec18ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:44.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:44.671 7fe4f43b6700 1 --2- 172.21.2.112:0/3835163191 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe4ec18f3f0 0x7fe4ec18f7f0 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:44.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:44.671 7fe4f43b6700 1 --2- 172.21.2.112:0/3835163191 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe4ec1b6de0 0x7fe4ec1b63f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:44.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:44.671 7fe4f43b6700 1 -- 172.21.2.112:0/3835163191 >> 172.21.2.112:0/3835163191 conn(0x7fe4ec1a37a0 msgr2=0x7fe4ec18c230 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:44.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:44.671 7fe4f43b6700 1 -- 172.21.2.112:0/3835163191 shutdown_connections 2021-02-10T11:31:44.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:44.671 7fe4f43b6700 1 -- 172.21.2.112:0/3835163191 wait complete. 2021-02-10T11:31:44.667 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.2 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:31:45.003 DEBUG:teuthology.orchestra.run.gibba012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:31:45.007 DEBUG:teuthology.orchestra.run.gibba020:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:31:45.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.102 7f8c23bee700 1 Processor -- start 2021-02-10T11:31:45.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.102 7f8c23bee700 1 -- start start 2021-02-10T11:31:45.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.102 7f8c23bee700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8c1c1b0870 0x7f8c1c0f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:45.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.102 7f8c23bee700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f8c1c0efe00 0x7f8c1c0f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:45.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.102 7f8c23bee700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c1c1b0c70 con 0x7f8c1c1b0470 2021-02-10T11:31:45.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.102 7f8c23bee700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c1c0f0b50 con 0x7f8c1c0efe00 2021-02-10T11:31:45.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.103 7f8c23bee700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c1c0f0820 con 0x7f8c1c1b0870 2021-02-10T11:31:45.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.103 7f8c21a0a700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f8c1c1b0470 0x7f8c1c0ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:47790/0 (socket says 172.21.2.112:47790) 2021-02-10T11:31:45.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.103 7f8c21a0a700 1 -- 172.21.2.112:0/2426981284 learned_addr learned my addr 172.21.2.112:0/2426981284 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:31:45.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.103 7f8c2220b700 1 --2- 172.21.2.112:0/2426981284 >> v2:172.21.2.112:3300/0 conn(0x7f8c1c0efe00 0x7f8c1c0f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:45.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.103 7f8c2220b700 1 -- 172.21.2.112:0/2426981284 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8c1c1b0870 msgr2=0x7f8c1c0f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:45.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.103 7f8c2220b700 1 --2- 172.21.2.112:0/2426981284 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8c1c1b0870 0x7f8c1c0f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:45.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.103 7f8c2220b700 1 -- 172.21.2.112:0/2426981284 >> v1:172.21.2.112:6789/0 conn(0x7f8c1c1b0470 legacy=0x7f8c1c0ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:45.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.103 7f8c2220b700 1 -- 172.21.2.112:0/2426981284 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c1c0e52e0 con 0x7f8c1c0efe00 2021-02-10T11:31:45.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.103 7f8c2220b700 1 --2- 172.21.2.112:0/2426981284 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8c1c0efe00 0x7f8c1c0f0200 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 rx=0x7f8c14006e30 tx=0x7f8c14009e60).ready entity=mon.0 client_cookie=8faa013805c66118 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:45.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.103 7f8c20a08700 1 -- 172.21.2.112:0/2426981284 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8c14006a50 con 0x7f8c1c0efe00 2021-02-10T11:31:45.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.103 7f8c20a08700 1 -- 172.21.2.112:0/2426981284 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c1400b300 con 0x7f8c1c0efe00 2021-02-10T11:31:45.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.103 7f8c20a08700 1 -- 172.21.2.112:0/2426981284 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8c1400b790 con 0x7f8c1c0efe00 2021-02-10T11:31:45.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.104 7f8c23bee700 1 -- 172.21.2.112:0/2426981284 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8c1c0efe00 msgr2=0x7f8c1c0f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:45.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.104 7f8c23bee700 1 --2- 172.21.2.112:0/2426981284 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8c1c0efe00 0x7f8c1c0f0200 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 rx=0x7f8c14006e30 tx=0x7f8c14009e60).stop 2021-02-10T11:31:45.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.104 7f8c23bee700 1 -- 172.21.2.112:0/2426981284 shutdown_connections 2021-02-10T11:31:45.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.104 7f8c23bee700 1 --2- 172.21.2.112:0/2426981284 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8c1c0efe00 0x7f8c1c0f0200 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:45.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.104 7f8c23bee700 1 --2- 172.21.2.112:0/2426981284 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8c1c1b0870 0x7f8c1c0f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:45.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.104 7f8c23bee700 1 -- 172.21.2.112:0/2426981284 >> 172.21.2.112:0/2426981284 conn(0x7f8c1c0ce3e0 msgr2=0x7f8c1c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:45.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.104 7f8c23bee700 1 -- 172.21.2.112:0/2426981284 shutdown_connections 2021-02-10T11:31:45.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.104 7f8c23bee700 1 -- 172.21.2.112:0/2426981284 wait complete. 2021-02-10T11:31:45.109 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.118 7f8c23bee700 1 Processor -- start 2021-02-10T11:31:45.109 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.119 7f8c23bee700 1 -- start start 2021-02-10T11:31:45.109 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.119 7f8c23bee700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8c1c18ecd0 0x7f8c1c18e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:45.109 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.119 7f8c23bee700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8c1c1a3d90 0x7f8c1c1ab630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:45.109 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.119 7f8c23bee700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8c1c1a4190 0x7f8c1c1ac5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:45.110 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.119 7f8c23bee700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c1c18f150 con 0x7f8c1c18ecd0 2021-02-10T11:31:45.110 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.119 7f8c23bee700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c1c1abc50 con 0x7f8c1c1a4190 2021-02-10T11:31:45.110 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.119 7f8c23bee700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c1c1abe70 con 0x7f8c1c1a3d90 2021-02-10T11:31:45.110 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.119 7f8c21a0a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8c1c18ecd0 0x7f8c1c18e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:45.110 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.119 7f8c21a0a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8c1c18ecd0 0x7f8c1c18e4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:48336/0 (socket says 172.21.2.112:48336) 2021-02-10T11:31:45.110 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.119 7f8c21a0a700 1 -- 172.21.2.112:0/1554897587 learned_addr learned my addr 172.21.2.112:0/1554897587 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:31:45.110 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.120 7f8c21a0a700 1 -- 172.21.2.112:0/1554897587 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8c1c1a3d90 msgr2=0x7f8c1c1ab630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:45.110 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.120 7f8c21a0a700 1 --2- 172.21.2.112:0/1554897587 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8c1c1a3d90 0x7f8c1c1ab630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:45.110 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.120 7f8c21a0a700 1 -- 172.21.2.112:0/1554897587 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8c1c1a4190 msgr2=0x7f8c1c1ac5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:45.110 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.120 7f8c21a0a700 1 --2- 172.21.2.112:0/1554897587 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8c1c1a4190 0x7f8c1c1ac5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:45.110 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.120 7f8c21a0a700 1 -- 172.21.2.112:0/1554897587 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c1c1acc30 con 0x7f8c1c18ecd0 2021-02-10T11:31:45.110 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.120 7f8c21a0a700 1 --2- 172.21.2.112:0/1554897587 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8c1c18ecd0 0x7f8c1c18e4c0 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 rx=0x7f8c10001c60 tx=0x7f8c1000a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:45.111 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.120 7f8c0e7fc700 1 -- 172.21.2.112:0/1554897587 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8c1000acc0 con 0x7f8c1c18ecd0 2021-02-10T11:31:45.111 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.120 7f8c0e7fc700 1 -- 172.21.2.112:0/1554897587 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c10004140 con 0x7f8c1c18ecd0 2021-02-10T11:31:45.111 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.120 7f8c0e7fc700 1 -- 172.21.2.112:0/1554897587 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8c100045a0 con 0x7f8c1c18ecd0 2021-02-10T11:31:45.111 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.120 7f8c23bee700 1 -- 172.21.2.112:0/1554897587 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c1c1acc30 con 0x7f8c1c18ecd0 2021-02-10T11:31:45.111 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.120 7f8c23bee700 1 -- 172.21.2.112:0/1554897587 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c1c1acc30 con 0x7f8c1c18ecd0 2021-02-10T11:31:45.111 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.121 7f8c0e7fc700 1 -- 172.21.2.112:0/1554897587 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8c10000da0 con 0x7f8c1c18ecd0 2021-02-10T11:31:45.111 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.121 7f8c0e7fc700 1 --2- 172.21.2.112:0/1554897587 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8c040186d0 0x7f8c0401abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:45.111 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.121 7f8c0e7fc700 1 -- 172.21.2.112:0/1554897587 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f8c10032680 con 0x7f8c1c18ecd0 2021-02-10T11:31:45.111 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.121 7f8c21209700 1 --2- 172.21.2.112:0/1554897587 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8c040186d0 0x7f8c0401abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:45.111 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.122 7f8c21209700 1 --2- 172.21.2.112:0/1554897587 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8c040186d0 0x7f8c0401abd0 crc :-1 s=READY pgs=522 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:45.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.152 7f8c23bee700 1 -- 172.21.2.112:0/1554897587 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c1c0edd30 con 0x7f8c1c18ecd0 2021-02-10T11:31:45.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.155 7f8c0e7fc700 1 -- 172.21.2.112:0/1554897587 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8c100009b0 con 0x7f8c1c18ecd0 2021-02-10T11:31:45.273 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.285 7f8c23bee700 1 -- 172.21.2.112:0/1554897587 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f8c1c0e52e0 con 0x7f8c1c18ecd0 2021-02-10T11:31:45.275 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.288 7f8c0e7fc700 1 -- 172.21.2.112:0/1554897587 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.2 v19) v1 ==== 184+0+0 (secure 0 0 0) 0x7f8c1001f540 con 0x7f8c1c18ecd0 2021-02-10T11:31:45.276 INFO:teuthology.orchestra.run.gibba012.stderr:updated caps for client.2 2021-02-10T11:31:45.277 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.290 7f8c23bee700 1 -- 172.21.2.112:0/1554897587 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8c040186d0 msgr2=0x7f8c0401abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:45.277 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.290 7f8c23bee700 1 --2- 172.21.2.112:0/1554897587 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8c040186d0 0x7f8c0401abd0 crc :-1 s=READY pgs=522 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:45.277 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.290 7f8c23bee700 1 -- 172.21.2.112:0/1554897587 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8c1c18ecd0 msgr2=0x7f8c1c18e4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:45.277 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.290 7f8c23bee700 1 --2- 172.21.2.112:0/1554897587 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8c1c18ecd0 0x7f8c1c18e4c0 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 rx=0x7f8c10001c60 tx=0x7f8c1000a8d0).stop 2021-02-10T11:31:45.278 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.290 7f8c23bee700 1 -- 172.21.2.112:0/1554897587 shutdown_connections 2021-02-10T11:31:45.278 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.290 7f8c23bee700 1 --2- 172.21.2.112:0/1554897587 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8c040186d0 0x7f8c0401abd0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:45.278 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.290 7f8c23bee700 1 --2- 172.21.2.112:0/1554897587 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8c1c1a4190 0x7f8c1c1ac5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:45.278 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.290 7f8c23bee700 1 --2- 172.21.2.112:0/1554897587 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8c1c1a3d90 0x7f8c1c1ab630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:45.278 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.290 7f8c23bee700 1 --2- 172.21.2.112:0/1554897587 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8c1c18ecd0 0x7f8c1c18e4c0 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:45.278 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.290 7f8c23bee700 1 -- 172.21.2.112:0/1554897587 >> 172.21.2.112:0/1554897587 conn(0x7f8c1c1a90c0 msgr2=0x7f8c1c1b38b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:45.278 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.290 7f8c23bee700 1 -- 172.21.2.112:0/1554897587 shutdown_connections 2021-02-10T11:31:45.278 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.290 7f8c23bee700 1 -- 172.21.2.112:0/1554897587 wait complete. 2021-02-10T11:31:45.286 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.3 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:31:45.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.719 7fafd2c1b700 1 Processor -- start 2021-02-10T11:31:45.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.720 7fafd2c1b700 1 -- start start 2021-02-10T11:31:45.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.720 7fafd2c1b700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fafcc1b0470 0x7fafcc1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:45.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.720 7fafd2c1b700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafcc0f0b50 con 0x7fafcc0ef610 2021-02-10T11:31:45.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.720 7fafd2c1b700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafcc0f24e0 con 0x7fafcc0f20e0 2021-02-10T11:31:45.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.720 7fafd2c1b700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafcc0e5ea0 con 0x7fafcc1b0470 2021-02-10T11:31:45.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.720 7fafcbfff700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fafcc0ef610 0x7fafcc0efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:47804/0 (socket says 172.21.2.112:47804) 2021-02-10T11:31:45.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.720 7fafcbfff700 1 -- 172.21.2.112:0/2084501174 learned_addr learned my addr 172.21.2.112:0/2084501174 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:31:45.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.720 7fafcb7fe700 1 -- 172.21.2.112:0/2084501174 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3114517821 0 0) 0x7fafbc000ce0 con 0x7fafcc0ef610 2021-02-10T11:31:45.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.720 7fafcb7fe700 1 -- 172.21.2.112:0/2084501174 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafb8002fe0 con 0x7fafcc0ef610 2021-02-10T11:31:45.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.721 7fafd0a37700 1 --2- 172.21.2.112:0/2084501174 >> v2:172.21.2.120:3300/0 conn(0x7fafcc1b0470 0x7fafcc1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:45.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.721 7fafcb7fe700 1 -- 172.21.2.112:0/2084501174 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3804775984 0 0) 0x7fafbc0010e0 con 0x7fafcc0ef610 2021-02-10T11:31:45.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.721 7fafcb7fe700 1 -- 172.21.2.112:0/2084501174 >> v2:172.21.2.120:3300/0 conn(0x7fafcc1b0470 msgr2=0x7fafcc1b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:45.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.721 7fafcb7fe700 1 --2- 172.21.2.112:0/2084501174 >> v2:172.21.2.120:3300/0 conn(0x7fafcc1b0470 0x7fafcc1b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:45.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.721 7fafcb7fe700 1 -- 172.21.2.112:0/2084501174 >> v1:172.21.2.120:6789/0 conn(0x7fafcc0f20e0 legacy=0x7fafcc0f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:45.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.721 7fafcb7fe700 1 -- 172.21.2.112:0/2084501174 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafcc0e52e0 con 0x7fafcc0ef610 2021-02-10T11:31:45.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.721 7fafcb7fe700 1 -- 172.21.2.112:0/2084501174 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7fafbc001660 con 0x7fafcc0ef610 2021-02-10T11:31:45.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.721 7fafd0a37700 1 --2- 172.21.2.112:0/2084501174 >> v2:172.21.2.120:3300/0 conn(0x7fafcc1b0470 0x7fafcc1b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:31:45.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.721 7fafcb7fe700 1 -- 172.21.2.112:0/2084501174 >> v1:172.21.2.112:6789/0 conn(0x7fafcc0ef610 legacy=0x7fafcc0efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:45.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.721 7fafcb7fe700 1 --2- 172.21.2.112:0/2084501174 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fafb8003af0 0x7fafb8005f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:45.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.721 7fafcb7fe700 1 --2- 172.21.2.112:0/2084501174 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fafb8006780 0x7fafb8008c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:45.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.721 7fafcb7fe700 1 --2- 172.21.2.112:0/2084501174 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fafb8009580 0x7fafb800baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:45.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.721 7fafcb7fe700 1 -- 172.21.2.112:0/2084501174 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafb800c2f0 con 0x7fafb8003af0 2021-02-10T11:31:45.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.721 7fafcb7fe700 1 -- 172.21.2.112:0/2084501174 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafb800c510 con 0x7fafb8009580 2021-02-10T11:31:45.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.721 7fafcb7fe700 1 -- 172.21.2.112:0/2084501174 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafb800c730 con 0x7fafb8006780 2021-02-10T11:31:45.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.721 7fafd0a37700 1 --2- 172.21.2.112:0/2084501174 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fafb8003af0 0x7fafb8005f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:45.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.721 7fafcbfff700 1 --2- 172.21.2.112:0/2084501174 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fafb8006780 0x7fafb8008c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:45.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.721 7fafd1238700 1 --2- 172.21.2.112:0/2084501174 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fafb8009580 0x7fafb800baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:45.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.721 7fafd0a37700 1 -- 172.21.2.112:0/2084501174 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fafb8006780 msgr2=0x7fafb8008c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:45.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.721 7fafd0a37700 1 --2- 172.21.2.112:0/2084501174 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fafb8006780 0x7fafb8008c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:45.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.721 7fafd0a37700 1 -- 172.21.2.112:0/2084501174 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fafb8009580 msgr2=0x7fafb800baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:45.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.721 7fafd0a37700 1 --2- 172.21.2.112:0/2084501174 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fafb8009580 0x7fafb800baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:45.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.721 7fafd0a37700 1 -- 172.21.2.112:0/2084501174 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafb800c9d0 con 0x7fafb8003af0 2021-02-10T11:31:45.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.721 7fafd1238700 1 -- 172.21.2.112:0/2084501174 reap_dead start 2021-02-10T11:31:45.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.722 7fafd0a37700 1 --2- 172.21.2.112:0/2084501174 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fafb8003af0 0x7fafb8005f00 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 rx=0x7fafb404d980 tx=0x7fafb404dd20).ready entity=mon.0 client_cookie=43bb44666823438e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:45.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.722 7fafcb7fe700 1 -- 172.21.2.112:0/2084501174 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fafb40009b0 con 0x7fafb8003af0 2021-02-10T11:31:45.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.722 7fafcb7fe700 1 -- 172.21.2.112:0/2084501174 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafb4006460 con 0x7fafb8003af0 2021-02-10T11:31:45.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.722 7fafcb7fe700 1 -- 172.21.2.112:0/2084501174 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fafb4002220 con 0x7fafb8003af0 2021-02-10T11:31:45.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.722 7fafd2c1b700 1 -- 172.21.2.112:0/2084501174 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fafb8003af0 msgr2=0x7fafb8005f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:45.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.722 7fafd2c1b700 1 --2- 172.21.2.112:0/2084501174 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fafb8003af0 0x7fafb8005f00 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 rx=0x7fafb404d980 tx=0x7fafb404dd20).stop 2021-02-10T11:31:45.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.722 7fafd2c1b700 1 -- 172.21.2.112:0/2084501174 shutdown_connections 2021-02-10T11:31:45.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.722 7fafd2c1b700 1 --2- 172.21.2.112:0/2084501174 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fafb8003af0 0x7fafb8005f00 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:45.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.722 7fafd2c1b700 1 -- 172.21.2.112:0/2084501174 >> 172.21.2.112:0/2084501174 conn(0x7fafcc0ce3e0 msgr2=0x7fafcc0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:45.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.722 7fafd2c1b700 1 -- 172.21.2.112:0/2084501174 shutdown_connections 2021-02-10T11:31:45.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.722 7fafd2c1b700 1 -- 172.21.2.112:0/2084501174 wait complete. 2021-02-10T11:31:45.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.737 7fafd2c1b700 1 Processor -- start 2021-02-10T11:31:45.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.737 7fafd2c1b700 1 -- start start 2021-02-10T11:31:45.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.737 7fafd2c1b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fafcc1a7960 0x7fafcc1a7d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:45.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.737 7fafd2c1b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fafcc08c440 0x7fafcc189710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:45.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.737 7fafd2c1b700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fafcc189d30 0x7fafcc18a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:45.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.737 7fafd2c1b700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafcc18a850 con 0x7fafcc1a7960 2021-02-10T11:31:45.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.737 7fafd2c1b700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafcc08c840 con 0x7fafcc189d30 2021-02-10T11:31:45.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.737 7fafd2c1b700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafcc08cbb0 con 0x7fafcc08c440 2021-02-10T11:31:45.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.738 7fafd0a37700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fafcc1a7960 0x7fafcc1a7d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:45.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.738 7fafd0a37700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fafcc1a7960 0x7fafcc1a7d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:48354/0 (socket says 172.21.2.112:48354) 2021-02-10T11:31:45.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.738 7fafd0a37700 1 -- 172.21.2.112:0/1131146404 learned_addr learned my addr 172.21.2.112:0/1131146404 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:31:45.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.738 7fafd0a37700 1 -- 172.21.2.112:0/1131146404 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fafcc08c440 msgr2=0x7fafcc189710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:45.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.738 7fafd0a37700 1 --2- 172.21.2.112:0/1131146404 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fafcc08c440 0x7fafcc189710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:45.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.738 7fafd0a37700 1 -- 172.21.2.112:0/1131146404 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fafcc189d30 msgr2=0x7fafcc18a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:45.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.738 7fafd0a37700 1 --2- 172.21.2.112:0/1131146404 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fafcc189d30 0x7fafcc18a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:45.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.738 7fafd0a37700 1 -- 172.21.2.112:0/1131146404 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafcc18ad50 con 0x7fafcc1a7960 2021-02-10T11:31:45.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.738 7fafd0a37700 1 --2- 172.21.2.112:0/1131146404 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fafcc1a7960 0x7fafcc1a7d60 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 rx=0x7fafb40505b0 tx=0x7fafb4050950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:45.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.738 7fafc97fa700 1 -- 172.21.2.112:0/1131146404 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fafb4000c90 con 0x7fafcc1a7960 2021-02-10T11:31:45.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.738 7fafc97fa700 1 -- 172.21.2.112:0/1131146404 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafb40020d0 con 0x7fafcc1a7960 2021-02-10T11:31:45.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.738 7fafd2c1b700 1 -- 172.21.2.112:0/1131146404 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafcc18ad50 con 0x7fafcc1a7960 2021-02-10T11:31:45.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.738 7fafc97fa700 1 -- 172.21.2.112:0/1131146404 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fafb4002520 con 0x7fafcc1a7960 2021-02-10T11:31:45.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.738 7fafd2c1b700 1 -- 172.21.2.112:0/1131146404 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafcc18ad50 con 0x7fafcc1a7960 2021-02-10T11:31:45.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.739 7fafc97fa700 1 -- 172.21.2.112:0/1131146404 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fafb40022b0 con 0x7fafcc1a7960 2021-02-10T11:31:45.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.739 7fafc97fa700 1 --2- 172.21.2.112:0/1131146404 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fafb0018730 0x7fafb001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:45.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.739 7fafc97fa700 1 -- 172.21.2.112:0/1131146404 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fafb4076670 con 0x7fafcc1a7960 2021-02-10T11:31:45.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.739 7fafcbfff700 1 --2- 172.21.2.112:0/1131146404 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fafb0018730 0x7fafb001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:45.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.740 7fafcbfff700 1 --2- 172.21.2.112:0/1131146404 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fafb0018730 0x7fafb001abc0 crc :-1 s=READY pgs=523 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:45.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.770 7fafd2c1b700 1 -- 172.21.2.112:0/1131146404 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafcc105a30 con 0x7fafcc1a7960 2021-02-10T11:31:45.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.772 7fafc97fa700 1 -- 172.21.2.112:0/1131146404 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fafb4000c90 con 0x7fafcc1a7960 2021-02-10T11:31:45.890 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.903 7fafd2c1b700 1 -- 172.21.2.112:0/1131146404 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fafcc105a30 con 0x7fafcc1a7960 2021-02-10T11:31:45.894 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.907 7fafc97fa700 1 -- 172.21.2.112:0/1131146404 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.3 v20) v1 ==== 184+0+0 (secure 0 0 0) 0x7fafb40020d0 con 0x7fafcc1a7960 2021-02-10T11:31:45.898 INFO:teuthology.orchestra.run.gibba012.stderr:updated caps for client.3 2021-02-10T11:31:45.900 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.912 7fafd2c1b700 1 -- 172.21.2.112:0/1131146404 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fafb0018730 msgr2=0x7fafb001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:45.900 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.912 7fafd2c1b700 1 --2- 172.21.2.112:0/1131146404 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fafb0018730 0x7fafb001abc0 crc :-1 s=READY pgs=523 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:45.900 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.912 7fafd2c1b700 1 -- 172.21.2.112:0/1131146404 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fafcc1a7960 msgr2=0x7fafcc1a7d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:45.900 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.912 7fafd2c1b700 1 --2- 172.21.2.112:0/1131146404 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fafcc1a7960 0x7fafcc1a7d60 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 rx=0x7fafb40505b0 tx=0x7fafb4050950).stop 2021-02-10T11:31:45.900 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.912 7fafd2c1b700 1 -- 172.21.2.112:0/1131146404 shutdown_connections 2021-02-10T11:31:45.900 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.912 7fafd2c1b700 1 --2- 172.21.2.112:0/1131146404 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fafb0018730 0x7fafb001abc0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:45.901 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.912 7fafd2c1b700 1 --2- 172.21.2.112:0/1131146404 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fafcc189d30 0x7fafcc18a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:45.901 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.912 7fafd2c1b700 1 --2- 172.21.2.112:0/1131146404 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fafcc08c440 0x7fafcc189710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:45.901 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.912 7fafd2c1b700 1 --2- 172.21.2.112:0/1131146404 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fafcc1a7960 0x7fafcc1a7d60 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:45.901 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.912 7fafd2c1b700 1 -- 172.21.2.112:0/1131146404 >> 172.21.2.112:0/1131146404 conn(0x7fafcc0b5850 msgr2=0x7fafcc091a50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:45.901 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.913 7fafd2c1b700 1 -- 172.21.2.112:0/1131146404 shutdown_connections 2021-02-10T11:31:45.901 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:45.913 7fafd2c1b700 1 -- 172.21.2.112:0/1131146404 wait complete. 2021-02-10T11:31:45.908 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:31:46.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:46.330 7f39d651e700 1 Processor -- start 2021-02-10T11:31:46.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:46.330 7f39d651e700 1 -- start start 2021-02-10T11:31:46.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:46.331 7f39d651e700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f39d00e5070 0x7f39d01b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:46.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:46.331 7f39d651e700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39d007ed80 con 0x7f39d01b3e40 2021-02-10T11:31:46.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:46.331 7f39d651e700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39d007e670 con 0x7f39d00e5470 2021-02-10T11:31:46.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:46.331 7f39d651e700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39d007e890 con 0x7f39d00e5070 2021-02-10T11:31:46.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:46.331 7f39cf7fe700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f39d01b3e40 0x7f39d007d690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:47826/0 (socket says 172.21.2.112:47826) 2021-02-10T11:31:46.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:46.331 7f39cf7fe700 1 -- 172.21.2.112:0/318114137 learned_addr learned my addr 172.21.2.112:0/318114137 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:31:46.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:46.331 7f39ceffd700 1 -- 172.21.2.112:0/318114137 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2402369618 0 0) 0x7f39c40009b0 con 0x7f39d01b3e40 2021-02-10T11:31:46.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:46.331 7f39ceffd700 1 -- 172.21.2.112:0/318114137 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39b8002fe0 con 0x7f39d01b3e40 2021-02-10T11:31:46.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:46.331 7f39cffff700 1 --2- 172.21.2.112:0/318114137 >> v2:172.21.2.120:3300/0 conn(0x7f39d00e5070 0x7f39d01b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:46.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:46.332 7f39ceffd700 1 -- 172.21.2.112:0/318114137 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4077412248 0 0) 0x7f39c4001010 con 0x7f39d01b3e40 2021-02-10T11:31:46.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:46.332 7f39ceffd700 1 -- 172.21.2.112:0/318114137 >> v2:172.21.2.120:3300/0 conn(0x7f39d00e5070 msgr2=0x7f39d01b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:46.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:46.332 7f39ceffd700 1 --2- 172.21.2.112:0/318114137 >> v2:172.21.2.120:3300/0 conn(0x7f39d00e5070 0x7f39d01b3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:46.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:46.332 7f39ceffd700 1 -- 172.21.2.112:0/318114137 >> v1:172.21.2.120:6789/0 conn(0x7f39d00e5470 legacy=0x7f39d007de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:46.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:46.332 7f39ceffd700 1 -- 172.21.2.112:0/318114137 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39d007eab0 con 0x7f39d01b3e40 2021-02-10T11:31:46.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:46.332 7f39ceffd700 1 -- 172.21.2.112:0/318114137 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f39c4001590 con 0x7f39d01b3e40 2021-02-10T11:31:46.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:46.332 7f39ceffd700 1 -- 172.21.2.112:0/318114137 >> v1:172.21.2.112:6789/0 conn(0x7f39d01b3e40 legacy=0x7f39d007d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:46.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:46.332 7f39ceffd700 1 --2- 172.21.2.112:0/318114137 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f39b8003aa0 0x7f39b8005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:46.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:46.332 7f39ceffd700 1 --2- 172.21.2.112:0/318114137 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f39b8006790 0x7f39b8008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:46.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:46.332 7f39ceffd700 1 --2- 172.21.2.112:0/318114137 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f39b8009590 0x7f39b800bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:46.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:46.332 7f39ceffd700 1 -- 172.21.2.112:0/318114137 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39b800c300 con 0x7f39b8006790 2021-02-10T11:31:46.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:46.332 7f39ceffd700 1 -- 172.21.2.112:0/318114137 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39b800c520 con 0x7f39b8009590 2021-02-10T11:31:46.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:46.332 7f39ceffd700 1 -- 172.21.2.112:0/318114137 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39b8050750 con 0x7f39b8003aa0 2021-02-10T11:31:46.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:46.332 7f39cf7fe700 1 --2- 172.21.2.112:0/318114137 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f39b8006790 0x7f39b8008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:46.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:46.332 7f39d4b3b700 1 --2- 172.21.2.112:0/318114137 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f39b8009590 0x7f39b800bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:46.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:46.332 7f39cffff700 1 --2- 172.21.2.112:0/318114137 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f39b8003aa0 0x7f39b8005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:46.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:46.332 7f39cf7fe700 1 -- 172.21.2.112:0/318114137 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f39b8003aa0 msgr2=0x7f39b8005ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:46.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:46.332 7f39cf7fe700 1 --2- 172.21.2.112:0/318114137 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f39b8003aa0 0x7f39b8005ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:46.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:46.332 7f39cf7fe700 1 -- 172.21.2.112:0/318114137 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f39b8009590 msgr2=0x7f39b800bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:46.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:46.332 7f39cf7fe700 1 --2- 172.21.2.112:0/318114137 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f39b8009590 0x7f39b800bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:46.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:46.332 7f39d4b3b700 1 -- 172.21.2.112:0/318114137 reap_dead start 2021-02-10T11:31:46.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:46.332 7f39cf7fe700 1 -- 172.21.2.112:0/318114137 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39b80509b0 con 0x7f39b8006790 2021-02-10T11:31:46.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:46.332 7f39cf7fe700 1 --2- 172.21.2.112:0/318114137 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f39b8006790 0x7f39b8008c90 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 rx=0x7f39c40017d0 tx=0x7f39c4007d30).ready entity=mon.0 client_cookie=5a33dd609d48042a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:46.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:46.333 7f39ceffd700 1 -- 172.21.2.112:0/318114137 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f39c400a310 con 0x7f39b8006790 2021-02-10T11:31:46.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:46.333 7f39ceffd700 1 -- 172.21.2.112:0/318114137 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39c40013a0 con 0x7f39b8006790 2021-02-10T11:31:46.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:46.333 7f39ceffd700 1 -- 172.21.2.112:0/318114137 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f39c400a810 con 0x7f39b8006790 2021-02-10T11:31:46.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:46.333 7f39d651e700 1 -- 172.21.2.112:0/318114137 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f39b8006790 msgr2=0x7f39b8008c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:46.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:46.333 7f39d651e700 1 --2- 172.21.2.112:0/318114137 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f39b8006790 0x7f39b8008c90 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 rx=0x7f39c40017d0 tx=0x7f39c4007d30).stop 2021-02-10T11:31:46.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:46.333 7f39d651e700 1 -- 172.21.2.112:0/318114137 shutdown_connections 2021-02-10T11:31:46.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:46.333 7f39d651e700 1 --2- 172.21.2.112:0/318114137 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f39b8006790 0x7f39b8008c90 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:46.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:46.333 7f39d651e700 1 -- 172.21.2.112:0/318114137 >> 172.21.2.112:0/318114137 conn(0x7f39d00ce690 msgr2=0x7f39d00e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:46.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:46.333 7f39d651e700 1 -- 172.21.2.112:0/318114137 shutdown_connections 2021-02-10T11:31:46.339 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:46.333 7f39d651e700 1 -- 172.21.2.112:0/318114137 wait complete. 2021-02-10T11:31:46.339 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:46.348 7f39d651e700 1 Processor -- start 2021-02-10T11:31:46.339 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:46.348 7f39d651e700 1 -- start start 2021-02-10T11:31:46.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:46.348 7f39d651e700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f39d019acd0 0x7f39d00a03d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:46.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:46.348 7f39d651e700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f39d00a09f0 0x7f39d00b3ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:46.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:46.348 7f39d651e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f39d00b40e0 0x7f39d00b44e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:46.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:46.348 7f39d651e700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39d00a1240 con 0x7f39d00b40e0 2021-02-10T11:31:46.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:46.348 7f39d651e700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39d019b1e0 con 0x7f39d00a09f0 2021-02-10T11:31:46.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:46.348 7f39d651e700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39d00a0df0 con 0x7f39d019acd0 2021-02-10T11:31:46.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:46.348 7f39d4b3b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f39d00b40e0 0x7f39d00b44e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:46.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:46.348 7f39d4b3b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f39d00b40e0 0x7f39d00b44e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:48376/0 (socket says 172.21.2.112:48376) 2021-02-10T11:31:46.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:46.348 7f39d4b3b700 1 -- 172.21.2.112:0/3540341787 learned_addr learned my addr 172.21.2.112:0/3540341787 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:31:46.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:46.349 7f39d4b3b700 1 -- 172.21.2.112:0/3540341787 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f39d019acd0 msgr2=0x7f39d00a03d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:31:46.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:46.349 7f39d4b3b700 1 --2- 172.21.2.112:0/3540341787 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f39d019acd0 0x7f39d00a03d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:46.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:46.349 7f39d4b3b700 1 -- 172.21.2.112:0/3540341787 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f39d00a09f0 msgr2=0x7f39d00b3ac0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:31:46.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:46.349 7f39d4b3b700 1 --2- 172.21.2.112:0/3540341787 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f39d00a09f0 0x7f39d00b3ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:46.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:46.349 7f39d4b3b700 1 -- 172.21.2.112:0/3540341787 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39d00a1740 con 0x7f39d00b40e0 2021-02-10T11:31:46.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:46.349 7f39d4b3b700 1 --2- 172.21.2.112:0/3540341787 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f39d00b40e0 0x7f39d00b44e0 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 rx=0x7f39c800b970 tx=0x7f39c800bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:46.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:46.349 7f39ccff9700 1 -- 172.21.2.112:0/3540341787 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f39c80066e0 con 0x7f39d00b40e0 2021-02-10T11:31:46.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:46.349 7f39ccff9700 1 -- 172.21.2.112:0/3540341787 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39c8000ce0 con 0x7f39d00b40e0 2021-02-10T11:31:46.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:46.349 7f39d651e700 1 -- 172.21.2.112:0/3540341787 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39d00a1740 con 0x7f39d00b40e0 2021-02-10T11:31:46.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:46.349 7f39d651e700 1 -- 172.21.2.112:0/3540341787 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39d00a1740 con 0x7f39d00b40e0 2021-02-10T11:31:46.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:46.349 7f39ccff9700 1 -- 172.21.2.112:0/3540341787 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f39c800c4e0 con 0x7f39d00b40e0 2021-02-10T11:31:46.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:46.350 7f39ccff9700 1 -- 172.21.2.112:0/3540341787 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f39c800c2f0 con 0x7f39d00b40e0 2021-02-10T11:31:46.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:46.350 7f39ccff9700 1 --2- 172.21.2.112:0/3540341787 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f39b0018730 0x7f39b001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:46.342 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:46.350 7f39ccff9700 1 -- 172.21.2.112:0/3540341787 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f39c8032b50 con 0x7f39d00b40e0 2021-02-10T11:31:46.342 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:46.350 7f39cffff700 1 --2- 172.21.2.112:0/3540341787 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f39b0018730 0x7f39b001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:46.342 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:46.350 7f39cffff700 1 --2- 172.21.2.112:0/3540341787 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f39b0018730 0x7f39b001ac10 crc :-1 s=READY pgs=524 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:46.368 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:46.380 7f39d651e700 1 -- 172.21.2.112:0/3540341787 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39d007ed80 con 0x7f39d00b40e0 2021-02-10T11:31:46.370 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:46.383 7f39ccff9700 1 -- 172.21.2.112:0/3540341787 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f39c8000cc0 con 0x7f39d00b40e0 2021-02-10T11:31:46.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:46.512 7f39d651e700 1 -- 172.21.2.112:0/3540341787 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f39d007ed80 con 0x7f39d00b40e0 2021-02-10T11:31:46.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:46.513 7f39ccff9700 1 -- 172.21.2.112:0/3540341787 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+2099 (secure 0 0 0) 0x7f39c8000cc0 con 0x7f39d00b40e0 2021-02-10T11:31:46.501 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:31:46.501 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":22,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7155,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/1957863967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":1957863967},{"type":"v1","addr":"172.21.2.112:6833","nonce":1957863967}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":18}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:31:40.190262","modified":"2021-02-10 11:31:42.195824","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5534},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5534":{"gid":5534,"name":"b","rank":0,"incarnation":19,"state":"up:active","state_seq":3,"addr":"172.21.2.120:6835/1513827094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":1513827094},{"type":"v1","addr":"172.21.2.120:6835","nonce":1513827094}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2021-02-10T11:31:46.501 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 22 2021-02-10T11:31:46.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:46.515 7f39d651e700 1 -- 172.21.2.112:0/3540341787 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f39b0018730 msgr2=0x7f39b001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:46.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:46.515 7f39d651e700 1 --2- 172.21.2.112:0/3540341787 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f39b0018730 0x7f39b001ac10 crc :-1 s=READY pgs=524 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:46.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:46.515 7f39d651e700 1 -- 172.21.2.112:0/3540341787 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f39d00b40e0 msgr2=0x7f39d00b44e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:46.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:46.515 7f39d651e700 1 --2- 172.21.2.112:0/3540341787 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f39d00b40e0 0x7f39d00b44e0 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 rx=0x7f39c800b970 tx=0x7f39c800bd10).stop 2021-02-10T11:31:46.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:46.515 7f39d651e700 1 -- 172.21.2.112:0/3540341787 shutdown_connections 2021-02-10T11:31:46.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:46.515 7f39d651e700 1 --2- 172.21.2.112:0/3540341787 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f39b0018730 0x7f39b001ac10 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:46.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:46.515 7f39d651e700 1 --2- 172.21.2.112:0/3540341787 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f39d00a09f0 0x7f39d00b3ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:46.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:46.515 7f39d651e700 1 --2- 172.21.2.112:0/3540341787 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f39d00b40e0 0x7f39d00b44e0 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:46.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:46.515 7f39d651e700 1 --2- 172.21.2.112:0/3540341787 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f39d019acd0 0x7f39d00a03d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:46.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:46.515 7f39d651e700 1 -- 172.21.2.112:0/3540341787 >> 172.21.2.112:0/3540341787 conn(0x7f39d01a31d0 msgr2=0x7f39d008b830 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:46.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:46.515 7f39d651e700 1 -- 172.21.2.112:0/3540341787 shutdown_connections 2021-02-10T11:31:46.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:46.515 7f39d651e700 1 -- 172.21.2.112:0/3540341787 wait complete. 2021-02-10T11:31:46.713 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:31:40.190262', 'modified': '2021-02-10 11:31:42.195824', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5534}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5534': {'gid': 5534, 'name': 'b', 'rank': 0, 'incarnation': 19, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.120:6835/1513827094', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.120:6834', 'nonce': 1513827094}, {'type': 'v1', 'addr': '172.21.2.120:6835', 'nonce': 1513827094}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [7], 'metadata_pool': 6, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:31:46.713 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:31:46.714 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:31:47.111 INFO:teuthology.orchestra.run.gibba020.stdout:{ 2021-02-10T11:31:47.112 INFO:teuthology.orchestra.run.gibba020.stdout: "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", 2021-02-10T11:31:47.112 INFO:teuthology.orchestra.run.gibba020.stdout: "whoami": 0, 2021-02-10T11:31:47.112 INFO:teuthology.orchestra.run.gibba020.stdout: "id": 5534, 2021-02-10T11:31:47.112 INFO:teuthology.orchestra.run.gibba020.stdout: "want_state": "up:active", 2021-02-10T11:31:47.112 INFO:teuthology.orchestra.run.gibba020.stdout: "state": "up:active", 2021-02-10T11:31:47.112 INFO:teuthology.orchestra.run.gibba020.stdout: "rank_uptime": 6.9328078809999996, 2021-02-10T11:31:47.112 INFO:teuthology.orchestra.run.gibba020.stdout: "mdsmap_epoch": 22, 2021-02-10T11:31:47.112 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch": 38, 2021-02-10T11:31:47.112 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch_barrier": 38, 2021-02-10T11:31:47.112 INFO:teuthology.orchestra.run.gibba020.stdout: "uptime": 12.565272795 2021-02-10T11:31:47.112 INFO:teuthology.orchestra.run.gibba020.stdout:} 2021-02-10T11:31:47.118 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", "whoami": 0, "id": 5534, "want_state": "up:active", "state": "up:active", "rank_uptime": 6.9328078809999996, "mdsmap_epoch": 22, "osdmap_epoch": 38, "osdmap_epoch_barrier": 38, "uptime": 12.565272795 } 2021-02-10T11:31:47.118 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:31:47.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:47.542 7f17ef212700 1 Processor -- start 2021-02-10T11:31:47.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:47.542 7f17ef212700 1 -- start start 2021-02-10T11:31:47.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:47.543 7f17ef212700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f17e80e5070 0x7f17e81b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:47.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:47.543 7f17ef212700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f17e81b3e40 0x7f17e807d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:47.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:47.543 7f17ef212700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17e807ed80 con 0x7f17e80e5470 2021-02-10T11:31:47.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:47.543 7f17ef212700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17e8080310 con 0x7f17e80e5070 2021-02-10T11:31:47.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:47.543 7f17ef212700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17e8080530 con 0x7f17e81b3e40 2021-02-10T11:31:47.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:47.543 7f17ed02e700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f17e80e5070 0x7f17e81b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:47.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:47.543 7f17ed02e700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f17e80e5070 0x7f17e81b3820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:48382/0 (socket says 172.21.2.112:48382) 2021-02-10T11:31:47.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:47.543 7f17ed02e700 1 -- 172.21.2.112:0/3940433244 learned_addr learned my addr 172.21.2.112:0/3940433244 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:31:47.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:47.543 7f17ec82d700 1 --2- 172.21.2.112:0/3940433244 >> v2:172.21.2.120:3300/0 conn(0x7f17e81b3e40 0x7f17e807d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:47.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:47.543 7f17ed02e700 1 -- 172.21.2.112:0/3940433244 >> v2:172.21.2.120:3300/0 conn(0x7f17e81b3e40 msgr2=0x7f17e807d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:47.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:47.543 7f17ed02e700 1 --2- 172.21.2.112:0/3940433244 >> v2:172.21.2.120:3300/0 conn(0x7f17e81b3e40 0x7f17e807d690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:47.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:47.543 7f17ed02e700 1 -- 172.21.2.112:0/3940433244 >> v1:172.21.2.120:6789/0 conn(0x7f17e80e5470 legacy=0x7f17e807dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:47.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:47.543 7f17ed02e700 1 -- 172.21.2.112:0/3940433244 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17e8080750 con 0x7f17e80e5070 2021-02-10T11:31:47.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:47.543 7f17ec82d700 1 --2- 172.21.2.112:0/3940433244 >> v2:172.21.2.120:3300/0 conn(0x7f17e81b3e40 0x7f17e807d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:31:47.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:47.544 7f17ed02e700 1 --2- 172.21.2.112:0/3940433244 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f17e80e5070 0x7f17e81b3820 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 rx=0x7f17d8006700 tx=0x7f17d8009b80).ready entity=mon.0 client_cookie=76e711c99ce702f4 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:47.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:47.544 7f17e7fff700 1 -- 172.21.2.112:0/3940433244 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f17d800c0c0 con 0x7f17e80e5070 2021-02-10T11:31:47.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:47.544 7f17e7fff700 1 -- 172.21.2.112:0/3940433244 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17d800c440 con 0x7f17e80e5070 2021-02-10T11:31:47.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:47.544 7f17e7fff700 1 -- 172.21.2.112:0/3940433244 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f17d800c960 con 0x7f17e80e5070 2021-02-10T11:31:47.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:47.544 7f17ef212700 1 -- 172.21.2.112:0/3940433244 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f17e80e5070 msgr2=0x7f17e81b3820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:47.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:47.544 7f17ef212700 1 --2- 172.21.2.112:0/3940433244 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f17e80e5070 0x7f17e81b3820 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 rx=0x7f17d8006700 tx=0x7f17d8009b80).stop 2021-02-10T11:31:47.549 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:47.545 7f17ef212700 1 -- 172.21.2.112:0/3940433244 shutdown_connections 2021-02-10T11:31:47.549 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:47.545 7f17ef212700 1 --2- 172.21.2.112:0/3940433244 >> v2:172.21.2.120:3300/0 conn(0x7f17e81b3e40 0x7f17e807d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:47.549 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:47.545 7f17ef212700 1 --2- 172.21.2.112:0/3940433244 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f17e80e5070 0x7f17e81b3820 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:47.549 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:47.545 7f17ef212700 1 -- 172.21.2.112:0/3940433244 >> 172.21.2.112:0/3940433244 conn(0x7f17e80ce690 msgr2=0x7f17e80e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:47.549 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:47.545 7f17ef212700 1 -- 172.21.2.112:0/3940433244 shutdown_connections 2021-02-10T11:31:47.549 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:47.545 7f17ef212700 1 -- 172.21.2.112:0/3940433244 wait complete. 2021-02-10T11:31:47.549 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:47.559 7f17ef212700 1 Processor -- start 2021-02-10T11:31:47.549 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:47.560 7f17ef212700 1 -- start start 2021-02-10T11:31:47.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:47.560 7f17ef212700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f17e808d1c0 0x7f17e8081230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:47.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:47.560 7f17ef212700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f17e8081850 0x7f17e80ad0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:47.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:47.560 7f17ef212700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f17e80ad6f0 0x7f17e81a1d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:47.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:47.560 7f17ef212700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17e8087740 con 0x7f17e8081850 2021-02-10T11:31:47.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:47.560 7f17ef212700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17e8081db0 con 0x7f17e80ad6f0 2021-02-10T11:31:47.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:47.560 7f17ef212700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17e808d5c0 con 0x7f17e808d1c0 2021-02-10T11:31:47.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:47.560 7f17ec82d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f17e8081850 0x7f17e80ad0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:47.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:47.560 7f17ec82d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f17e8081850 0x7f17e80ad0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:48390/0 (socket says 172.21.2.112:48390) 2021-02-10T11:31:47.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:47.560 7f17ec82d700 1 -- 172.21.2.112:0/3407111986 learned_addr learned my addr 172.21.2.112:0/3407111986 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:31:47.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:47.561 7f17ed02e700 1 --2- 172.21.2.112:0/3407111986 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f17e808d1c0 0x7f17e8081230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:47.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:47.561 7f17ed82f700 1 --2- 172.21.2.112:0/3407111986 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f17e80ad6f0 0x7f17e81a1d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:47.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:47.561 7f17ec82d700 1 -- 172.21.2.112:0/3407111986 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f17e808d1c0 msgr2=0x7f17e8081230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:47.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:47.561 7f17ec82d700 1 --2- 172.21.2.112:0/3407111986 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f17e808d1c0 0x7f17e8081230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:47.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:47.561 7f17ec82d700 1 -- 172.21.2.112:0/3407111986 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f17e80ad6f0 msgr2=0x7f17e81a1d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:47.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:47.561 7f17ec82d700 1 --2- 172.21.2.112:0/3407111986 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f17e80ad6f0 0x7f17e81a1d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:47.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:47.561 7f17ec82d700 1 -- 172.21.2.112:0/3407111986 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17e808da00 con 0x7f17e8081850 2021-02-10T11:31:47.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:47.561 7f17ed02e700 1 --2- 172.21.2.112:0/3407111986 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f17e808d1c0 0x7f17e8081230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:31:47.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:47.561 7f17ed82f700 1 --2- 172.21.2.112:0/3407111986 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f17e80ad6f0 0x7f17e81a1d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:31:47.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:47.561 7f17ec82d700 1 --2- 172.21.2.112:0/3407111986 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f17e8081850 0x7f17e80ad0d0 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 rx=0x7f17dc0099a0 tx=0x7f17dc00bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:47.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:47.561 7f17e5ffb700 1 -- 172.21.2.112:0/3407111986 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f17dc00c2f0 con 0x7f17e8081850 2021-02-10T11:31:47.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:47.561 7f17e5ffb700 1 -- 172.21.2.112:0/3407111986 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17dc00c630 con 0x7f17e8081850 2021-02-10T11:31:47.553 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:47.561 7f17e5ffb700 1 -- 172.21.2.112:0/3407111986 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f17dc00cb50 con 0x7f17e8081850 2021-02-10T11:31:47.553 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:47.561 7f17ef212700 1 -- 172.21.2.112:0/3407111986 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17e808da00 con 0x7f17e8081850 2021-02-10T11:31:47.553 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:47.561 7f17ef212700 1 -- 172.21.2.112:0/3407111986 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17e80adde0 con 0x7f17e8081850 2021-02-10T11:31:47.553 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:47.562 7f17e5ffb700 1 -- 172.21.2.112:0/3407111986 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f17dc00c2f0 con 0x7f17e8081850 2021-02-10T11:31:47.553 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:47.562 7f17e5ffb700 1 --2- 172.21.2.112:0/3407111986 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f17c80186c0 0x7f17c801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:47.553 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:47.562 7f17e5ffb700 1 -- 172.21.2.112:0/3407111986 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f17dc032900 con 0x7f17e8081850 2021-02-10T11:31:47.553 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:47.563 7f17ed02e700 1 --2- 172.21.2.112:0/3407111986 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f17c80186c0 0x7f17c801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:47.554 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:47.564 7f17ed02e700 1 --2- 172.21.2.112:0/3407111986 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f17c80186c0 0x7f17c801abc0 crc :-1 s=READY pgs=525 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:47.580 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:47.593 7f17ef212700 1 -- 172.21.2.112:0/3407111986 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17e8087740 con 0x7f17e8081850 2021-02-10T11:31:47.583 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:47.595 7f17e5ffb700 1 -- 172.21.2.112:0/3407111986 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f17dc000cc0 con 0x7f17e8081850 2021-02-10T11:31:47.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:47.721 7f17ef212700 1 -- 172.21.2.112:0/3407111986 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f17e8087740 con 0x7f17e8081850 2021-02-10T11:31:47.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:47.722 7f17e5ffb700 1 -- 172.21.2.112:0/3407111986 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v38) v1 ==== 74+0+13993 (secure 0 0 0) 0x7f17dc00c2f0 con 0x7f17e8081850 2021-02-10T11:31:47.710 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:31:47.710 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":38,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:31:41.192016","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:31:38.162654","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":7,"pool_name":"cephfs_data","create_time":"2021-02-10 11:31:39.233986","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:31:47.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:47.724 7f17ef212700 1 -- 172.21.2.112:0/3407111986 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f17c80186c0 msgr2=0x7f17c801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:47.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:47.724 7f17ef212700 1 --2- 172.21.2.112:0/3407111986 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f17c80186c0 0x7f17c801abc0 crc :-1 s=READY pgs=525 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:47.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:47.724 7f17ef212700 1 -- 172.21.2.112:0/3407111986 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f17e8081850 msgr2=0x7f17e80ad0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:47.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:47.724 7f17ef212700 1 --2- 172.21.2.112:0/3407111986 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f17e8081850 0x7f17e80ad0d0 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 rx=0x7f17dc0099a0 tx=0x7f17dc00bcf0).stop 2021-02-10T11:31:47.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:47.724 7f17ef212700 1 -- 172.21.2.112:0/3407111986 shutdown_connections 2021-02-10T11:31:47.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:47.724 7f17ef212700 1 --2- 172.21.2.112:0/3407111986 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f17c80186c0 0x7f17c801abc0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:47.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:47.724 7f17ef212700 1 --2- 172.21.2.112:0/3407111986 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f17e80ad6f0 0x7f17e81a1d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:47.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:47.724 7f17ef212700 1 --2- 172.21.2.112:0/3407111986 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f17e8081850 0x7f17e80ad0d0 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:47.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:47.724 7f17ef212700 1 --2- 172.21.2.112:0/3407111986 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f17e808d1c0 0x7f17e8081230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:47.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:47.724 7f17ef212700 1 -- 172.21.2.112:0/3407111986 >> 172.21.2.112:0/3407111986 conn(0x7f17e819b930 msgr2=0x7f17e81a0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:47.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:47.725 7f17ef212700 1 -- 172.21.2.112:0/3407111986 shutdown_connections 2021-02-10T11:31:47.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:47.725 7f17ef212700 1 -- 172.21.2.112:0/3407111986 wait complete. 2021-02-10T11:31:47.720 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:13:57.508758', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 6, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:31:38.162654', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '37', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 7, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:31:39.233986', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:31:47.720 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:31:48.150 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.145 7f64b0519700 1 Processor -- start 2021-02-10T11:31:48.150 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.145 7f64b0519700 1 -- start start 2021-02-10T11:31:48.150 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.145 7f64b0519700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f64a81b0870 0x7f64a80f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:48.150 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.146 7f64b0519700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f64a80efe00 0x7f64a80f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:48.150 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.146 7f64b0519700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64a81b0c70 con 0x7f64a81b0470 2021-02-10T11:31:48.150 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.146 7f64b0519700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64a80f0b50 con 0x7f64a81b0870 2021-02-10T11:31:48.150 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.146 7f64b0519700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64a80f0820 con 0x7f64a80efe00 2021-02-10T11:31:48.150 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.146 7f64ae335700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f64a81b0470 0x7f64a80ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:47856/0 (socket says 172.21.2.112:47856) 2021-02-10T11:31:48.150 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.146 7f64ae335700 1 -- 172.21.2.112:0/1629638583 learned_addr learned my addr 172.21.2.112:0/1629638583 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:31:48.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.146 7f64aeb36700 1 --2- 172.21.2.112:0/1629638583 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f64a80efe00 0x7f64a80f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:48.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.146 7f64adb34700 1 --2- 172.21.2.112:0/1629638583 >> v2:172.21.2.120:3300/0 conn(0x7f64a81b0870 0x7f64a80f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:48.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.146 7f64ad333700 1 -- 172.21.2.112:0/1629638583 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3499971586 0 0) 0x7f64a4001180 con 0x7f64a81b0470 2021-02-10T11:31:48.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.147 7f64ad333700 1 -- 172.21.2.112:0/1629638583 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6494002fe0 con 0x7f64a81b0470 2021-02-10T11:31:48.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.147 7f64aeb36700 1 -- 172.21.2.112:0/1629638583 >> v2:172.21.2.120:3300/0 conn(0x7f64a81b0870 msgr2=0x7f64a80f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:48.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.147 7f64ad333700 1 -- 172.21.2.112:0/1629638583 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1754445033 0 0) 0x7f64a4001180 con 0x7f64a81b0470 2021-02-10T11:31:48.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.147 7f64aeb36700 1 --2- 172.21.2.112:0/1629638583 >> v2:172.21.2.120:3300/0 conn(0x7f64a81b0870 0x7f64a80f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:48.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.147 7f64aeb36700 1 -- 172.21.2.112:0/1629638583 >> v1:172.21.2.112:6789/0 conn(0x7f64a81b0470 legacy=0x7f64a80ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:48.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.147 7f64aeb36700 1 -- 172.21.2.112:0/1629638583 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64a80e52e0 con 0x7f64a80efe00 2021-02-10T11:31:48.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.147 7f64aeb36700 1 --2- 172.21.2.112:0/1629638583 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f64a80efe00 0x7f64a80f0200 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f64a0006e50 tx=0x7f64a0009e70).ready entity=mon.2 client_cookie=6beae9d967939c50 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:48.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.147 7f64ad333700 1 -- 172.21.2.112:0/1629638583 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f64a000b300 con 0x7f64a80efe00 2021-02-10T11:31:48.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.147 7f64ad333700 1 -- 172.21.2.112:0/1629638583 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64a0003030 con 0x7f64a80efe00 2021-02-10T11:31:48.152 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.147 7f64ad333700 1 -- 172.21.2.112:0/1629638583 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f64a000b730 con 0x7f64a80efe00 2021-02-10T11:31:48.152 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.148 7f64b0519700 1 -- 172.21.2.112:0/1629638583 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f64a80efe00 msgr2=0x7f64a80f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:48.152 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.148 7f64b0519700 1 --2- 172.21.2.112:0/1629638583 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f64a80efe00 0x7f64a80f0200 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f64a0006e50 tx=0x7f64a0009e70).stop 2021-02-10T11:31:48.152 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.148 7f64b0519700 1 -- 172.21.2.112:0/1629638583 shutdown_connections 2021-02-10T11:31:48.152 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.148 7f64b0519700 1 --2- 172.21.2.112:0/1629638583 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f64a80efe00 0x7f64a80f0200 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:48.152 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.148 7f64b0519700 1 --2- 172.21.2.112:0/1629638583 >> v2:172.21.2.120:3300/0 conn(0x7f64a81b0870 0x7f64a80f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:48.152 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.148 7f64b0519700 1 -- 172.21.2.112:0/1629638583 >> 172.21.2.112:0/1629638583 conn(0x7f64a80ce3e0 msgr2=0x7f64a80e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:48.152 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.148 7f64b0519700 1 -- 172.21.2.112:0/1629638583 shutdown_connections 2021-02-10T11:31:48.152 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.148 7f64b0519700 1 -- 172.21.2.112:0/1629638583 wait complete. 2021-02-10T11:31:48.154 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.163 7f64b0519700 1 Processor -- start 2021-02-10T11:31:48.154 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.163 7f64b0519700 1 -- start start 2021-02-10T11:31:48.154 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.163 7f64b0519700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f64a80a0420 0x7f64a80a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:48.154 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.163 7f64b0519700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f64a8189d00 0x7f64a818a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:48.154 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.163 7f64b0519700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f64a818a720 0x7f64a80b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:48.154 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.163 7f64b0519700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64a8189800 con 0x7f64a818a720 2021-02-10T11:31:48.155 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.163 7f64b0519700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64a818ab20 con 0x7f64a8189d00 2021-02-10T11:31:48.155 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.163 7f64b0519700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64a80a1140 con 0x7f64a80a0420 2021-02-10T11:31:48.155 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.163 7f64aeb36700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f64a818a720 0x7f64a80b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:48.155 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.164 7f64aeb36700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f64a818a720 0x7f64a80b7be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:48404/0 (socket says 172.21.2.112:48404) 2021-02-10T11:31:48.155 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.164 7f64aeb36700 1 -- 172.21.2.112:0/2652341097 learned_addr learned my addr 172.21.2.112:0/2652341097 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:31:48.155 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.164 7f64ae335700 1 --2- 172.21.2.112:0/2652341097 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f64a80a0420 0x7f64a80a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:48.155 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.164 7f64adb34700 1 --2- 172.21.2.112:0/2652341097 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f64a8189d00 0x7f64a818a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:48.155 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.164 7f64aeb36700 1 -- 172.21.2.112:0/2652341097 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f64a80a0420 msgr2=0x7f64a80a0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:48.155 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.164 7f64aeb36700 1 --2- 172.21.2.112:0/2652341097 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f64a80a0420 0x7f64a80a0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:48.155 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.164 7f64aeb36700 1 -- 172.21.2.112:0/2652341097 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f64a8189d00 msgr2=0x7f64a818a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:48.155 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.164 7f64aeb36700 1 --2- 172.21.2.112:0/2652341097 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f64a8189d00 0x7f64a818a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:48.155 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.164 7f64aeb36700 1 -- 172.21.2.112:0/2652341097 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64a80b8200 con 0x7f64a818a720 2021-02-10T11:31:48.155 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.164 7f64ae335700 1 --2- 172.21.2.112:0/2652341097 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f64a80a0420 0x7f64a80a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:31:48.156 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.164 7f64adb34700 1 --2- 172.21.2.112:0/2652341097 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f64a8189d00 0x7f64a818a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:31:48.156 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.164 7f64aeb36700 1 --2- 172.21.2.112:0/2652341097 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f64a818a720 0x7f64a80b7be0 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 rx=0x7f64a000b940 tx=0x7f64a000bce0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:48.156 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.164 7f649affd700 1 -- 172.21.2.112:0/2652341097 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f64a0002030 con 0x7f64a818a720 2021-02-10T11:31:48.156 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.164 7f649affd700 1 -- 172.21.2.112:0/2652341097 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64a0002370 con 0x7f64a818a720 2021-02-10T11:31:48.156 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.164 7f64b0519700 1 -- 172.21.2.112:0/2652341097 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64a80b8200 con 0x7f64a818a720 2021-02-10T11:31:48.156 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.164 7f649affd700 1 -- 172.21.2.112:0/2652341097 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f64a00027d0 con 0x7f64a818a720 2021-02-10T11:31:48.156 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.164 7f64b0519700 1 -- 172.21.2.112:0/2652341097 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64a80b8790 con 0x7f64a818a720 2021-02-10T11:31:48.156 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.165 7f649affd700 1 -- 172.21.2.112:0/2652341097 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f64a0002020 con 0x7f64a818a720 2021-02-10T11:31:48.156 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.165 7f649affd700 1 --2- 172.21.2.112:0/2652341097 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6490018730 0x7f649001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:48.156 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.165 7f649affd700 1 -- 172.21.2.112:0/2652341097 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f64a00329f0 con 0x7f64a818a720 2021-02-10T11:31:48.156 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.166 7f64ae335700 1 --2- 172.21.2.112:0/2652341097 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6490018730 0x7f649001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:48.156 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.166 7f64ae335700 1 --2- 172.21.2.112:0/2652341097 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6490018730 0x7f649001ac10 crc :-1 s=READY pgs=526 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:48.183 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.195 7f64b0519700 1 -- 172.21.2.112:0/2652341097 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64a8105ae0 con 0x7f64a818a720 2021-02-10T11:31:48.185 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.198 7f649affd700 1 -- 172.21.2.112:0/2652341097 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f64a00066f0 con 0x7f64a818a720 2021-02-10T11:31:48.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.326 7f64b0519700 1 -- 172.21.2.112:0/2652341097 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f64a80f0b50 con 0x7f64a818a720 2021-02-10T11:31:48.314 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.326 7f649affd700 1 -- 172.21.2.112:0/2652341097 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v38) v1 ==== 92+0+10 (secure 0 0 0) 0x7f64a0002000 con 0x7f64a818a720 2021-02-10T11:31:48.314 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:31:48.316 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.329 7f64b0519700 1 -- 172.21.2.112:0/2652341097 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6490018730 msgr2=0x7f649001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:48.316 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.329 7f64b0519700 1 --2- 172.21.2.112:0/2652341097 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6490018730 0x7f649001ac10 crc :-1 s=READY pgs=526 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:48.316 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.329 7f64b0519700 1 -- 172.21.2.112:0/2652341097 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f64a818a720 msgr2=0x7f64a80b7be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:48.316 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.329 7f64b0519700 1 --2- 172.21.2.112:0/2652341097 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f64a818a720 0x7f64a80b7be0 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 rx=0x7f64a000b940 tx=0x7f64a000bce0).stop 2021-02-10T11:31:48.316 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.329 7f64b0519700 1 -- 172.21.2.112:0/2652341097 shutdown_connections 2021-02-10T11:31:48.316 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.329 7f64b0519700 1 --2- 172.21.2.112:0/2652341097 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6490018730 0x7f649001ac10 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:48.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.329 7f64b0519700 1 --2- 172.21.2.112:0/2652341097 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f64a8189d00 0x7f64a818a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:48.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.329 7f64b0519700 1 --2- 172.21.2.112:0/2652341097 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f64a818a720 0x7f64a80b7be0 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:48.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.329 7f64b0519700 1 --2- 172.21.2.112:0/2652341097 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f64a80a0420 0x7f64a80a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:48.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.329 7f64b0519700 1 -- 172.21.2.112:0/2652341097 >> 172.21.2.112:0/2652341097 conn(0x7f64a8094e10 msgr2=0x7f64a80a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:48.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.329 7f64b0519700 1 -- 172.21.2.112:0/2652341097 shutdown_connections 2021-02-10T11:31:48.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.329 7f64b0519700 1 -- 172.21.2.112:0/2652341097 wait complete. 2021-02-10T11:31:48.325 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:31:48.754 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.750 7f2d3a252700 1 Processor -- start 2021-02-10T11:31:48.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.750 7f2d3a252700 1 -- start start 2021-02-10T11:31:48.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.750 7f2d3a252700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f2d341b0470 0x7f2d341b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:48.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.750 7f2d3a252700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d340f0b50 con 0x7f2d340ef610 2021-02-10T11:31:48.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.750 7f2d3a252700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d340f24e0 con 0x7f2d340f20e0 2021-02-10T11:31:48.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.750 7f2d3a252700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d340e5ea0 con 0x7f2d341b0470 2021-02-10T11:31:48.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.751 7f2d337fe700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f2d340ef610 0x7f2d340efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:47870/0 (socket says 172.21.2.112:47870) 2021-02-10T11:31:48.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.751 7f2d337fe700 1 -- 172.21.2.112:0/563196941 learned_addr learned my addr 172.21.2.112:0/563196941 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:31:48.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.751 7f2d3886f700 1 --1- 172.21.2.112:0/563196941 >> v1:172.21.2.120:6789/0 conn(0x7f2d340f20e0 0x7f2d340f0200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:41428/0 (socket says 172.21.2.112:41428) 2021-02-10T11:31:48.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.751 7f2d33fff700 1 --2- 172.21.2.112:0/563196941 >> v2:172.21.2.112:3300/0 conn(0x7f2d341b0470 0x7f2d341b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:48.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.751 7f2d32ffd700 1 -- 172.21.2.112:0/563196941 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2870907375 0 0) 0x7f2d2c001440 con 0x7f2d340f20e0 2021-02-10T11:31:48.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.751 7f2d32ffd700 1 -- 172.21.2.112:0/563196941 --> v1:172.21.2.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d1c002fe0 con 0x7f2d340f20e0 2021-02-10T11:31:48.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.751 7f2d32ffd700 1 -- 172.21.2.112:0/563196941 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 628910978 0 0) 0x7f2d28000ff0 con 0x7f2d340ef610 2021-02-10T11:31:48.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.751 7f2d32ffd700 1 -- 172.21.2.112:0/563196941 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d1c003210 con 0x7f2d340ef610 2021-02-10T11:31:48.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.751 7f2d33fff700 1 -- 172.21.2.112:0/563196941 >> v1:172.21.2.120:6789/0 conn(0x7f2d340f20e0 legacy=0x7f2d340f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:48.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.751 7f2d33fff700 1 -- 172.21.2.112:0/563196941 >> v1:172.21.2.112:6789/0 conn(0x7f2d340ef610 legacy=0x7f2d340efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:48.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.751 7f2d33fff700 1 -- 172.21.2.112:0/563196941 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d340e52e0 con 0x7f2d341b0470 2021-02-10T11:31:48.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.752 7f2d33fff700 1 --2- 172.21.2.112:0/563196941 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2d341b0470 0x7f2d341b0870 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 rx=0x7f2d24006650 tx=0x7f2d24009a80).ready entity=mon.0 client_cookie=40c269c8b455767b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:48.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.752 7f2d32ffd700 1 -- 172.21.2.112:0/563196941 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2d240009b0 con 0x7f2d341b0470 2021-02-10T11:31:48.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.752 7f2d32ffd700 1 -- 172.21.2.112:0/563196941 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d2400c0c0 con 0x7f2d341b0470 2021-02-10T11:31:48.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.752 7f2d32ffd700 1 -- 172.21.2.112:0/563196941 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2d2400c5e0 con 0x7f2d341b0470 2021-02-10T11:31:48.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.752 7f2d3a252700 1 -- 172.21.2.112:0/563196941 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2d341b0470 msgr2=0x7f2d341b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:48.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.752 7f2d3a252700 1 --2- 172.21.2.112:0/563196941 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2d341b0470 0x7f2d341b0870 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 rx=0x7f2d24006650 tx=0x7f2d24009a80).stop 2021-02-10T11:31:48.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.753 7f2d3a252700 1 -- 172.21.2.112:0/563196941 shutdown_connections 2021-02-10T11:31:48.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.753 7f2d3a252700 1 --2- 172.21.2.112:0/563196941 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2d341b0470 0x7f2d341b0870 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:48.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.753 7f2d3a252700 1 -- 172.21.2.112:0/563196941 >> 172.21.2.112:0/563196941 conn(0x7f2d340ce3e0 msgr2=0x7f2d340e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:48.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.753 7f2d3a252700 1 -- 172.21.2.112:0/563196941 shutdown_connections 2021-02-10T11:31:48.757 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.753 7f2d3a252700 1 -- 172.21.2.112:0/563196941 wait complete. 2021-02-10T11:31:48.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.768 7f2d3a252700 1 Processor -- start 2021-02-10T11:31:48.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.768 7f2d3a252700 1 -- start start 2021-02-10T11:31:48.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.768 7f2d3a252700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2d340c3600 0x7f2d341a7750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:48.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.768 7f2d3a252700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2d341a7d70 0x7f2d34105d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:48.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.768 7f2d3a252700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2d34106330 0x7f2d34106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:48.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.768 7f2d3a252700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d340a75d0 con 0x7f2d34106330 2021-02-10T11:31:48.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.768 7f2d3a252700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d340a7190 con 0x7f2d340c3600 2021-02-10T11:31:48.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.768 7f2d3a252700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d340c1940 con 0x7f2d341a7d70 2021-02-10T11:31:48.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.768 7f2d3886f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2d34106330 0x7f2d34106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:48.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.769 7f2d3886f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2d34106330 0x7f2d34106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:48414/0 (socket says 172.21.2.112:48414) 2021-02-10T11:31:48.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.769 7f2d3886f700 1 -- 172.21.2.112:0/3285500067 learned_addr learned my addr 172.21.2.112:0/3285500067 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:31:48.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.769 7f2d33fff700 1 --2- 172.21.2.112:0/3285500067 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2d340c3600 0x7f2d341a7750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:48.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.769 7f2d3886f700 1 -- 172.21.2.112:0/3285500067 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2d341a7d70 msgr2=0x7f2d34105d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:48.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.769 7f2d3886f700 1 --2- 172.21.2.112:0/3285500067 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2d341a7d70 0x7f2d34105d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:48.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.769 7f2d3886f700 1 -- 172.21.2.112:0/3285500067 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2d340c3600 msgr2=0x7f2d341a7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:48.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.769 7f2d3886f700 1 --2- 172.21.2.112:0/3285500067 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2d340c3600 0x7f2d341a7750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:48.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.769 7f2d3886f700 1 -- 172.21.2.112:0/3285500067 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d340c1b60 con 0x7f2d34106330 2021-02-10T11:31:48.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.769 7f2d33fff700 1 --2- 172.21.2.112:0/3285500067 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2d340c3600 0x7f2d341a7750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:31:48.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.769 7f2d3886f700 1 --2- 172.21.2.112:0/3285500067 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2d34106330 0x7f2d34106730 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 rx=0x7f2d2c00a940 tx=0x7f2d2c00ace0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:48.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.769 7f2d30ff9700 1 -- 172.21.2.112:0/3285500067 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2d2c0048b0 con 0x7f2d34106330 2021-02-10T11:31:48.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.769 7f2d30ff9700 1 -- 172.21.2.112:0/3285500067 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d2c004c30 con 0x7f2d34106330 2021-02-10T11:31:48.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.769 7f2d30ff9700 1 -- 172.21.2.112:0/3285500067 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2d2c007350 con 0x7f2d34106330 2021-02-10T11:31:48.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.769 7f2d3a252700 1 -- 172.21.2.112:0/3285500067 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d340c1b60 con 0x7f2d34106330 2021-02-10T11:31:48.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.770 7f2d3a252700 1 -- 172.21.2.112:0/3285500067 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d340c20f0 con 0x7f2d34106330 2021-02-10T11:31:48.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.770 7f2d30ff9700 1 -- 172.21.2.112:0/3285500067 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2d2c0048b0 con 0x7f2d34106330 2021-02-10T11:31:48.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.770 7f2d30ff9700 1 --2- 172.21.2.112:0/3285500067 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2d140186c0 0x7f2d1401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:48.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.770 7f2d30ff9700 1 -- 172.21.2.112:0/3285500067 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f2d2c032b40 con 0x7f2d34106330 2021-02-10T11:31:48.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.771 7f2d33fff700 1 --2- 172.21.2.112:0/3285500067 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2d140186c0 0x7f2d1401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:48.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.771 7f2d33fff700 1 --2- 172.21.2.112:0/3285500067 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2d140186c0 0x7f2d1401abc0 crc :-1 s=READY pgs=527 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:48.787 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.800 7f2d3a252700 1 -- 172.21.2.112:0/3285500067 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d34105ae0 con 0x7f2d34106330 2021-02-10T11:31:48.790 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.803 7f2d30ff9700 1 -- 172.21.2.112:0/3285500067 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2d2c0046a0 con 0x7f2d34106330 2021-02-10T11:31:48.919 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.932 7f2d3a252700 1 -- 172.21.2.112:0/3285500067 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f2d340f0b50 con 0x7f2d34106330 2021-02-10T11:31:48.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.932 7f2d30ff9700 1 -- 172.21.2.112:0/3285500067 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v38) v1 ==== 104+0+10 (secure 0 0 0) 0x7f2d2c0048b0 con 0x7f2d34106330 2021-02-10T11:31:48.920 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:31:48.922 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.934 7f2d3a252700 1 -- 172.21.2.112:0/3285500067 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2d140186c0 msgr2=0x7f2d1401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:48.922 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.934 7f2d3a252700 1 --2- 172.21.2.112:0/3285500067 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2d140186c0 0x7f2d1401abc0 crc :-1 s=READY pgs=527 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:48.922 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.934 7f2d3a252700 1 -- 172.21.2.112:0/3285500067 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2d34106330 msgr2=0x7f2d34106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:48.922 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.934 7f2d3a252700 1 --2- 172.21.2.112:0/3285500067 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2d34106330 0x7f2d34106730 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 rx=0x7f2d2c00a940 tx=0x7f2d2c00ace0).stop 2021-02-10T11:31:48.922 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.934 7f2d3a252700 1 -- 172.21.2.112:0/3285500067 shutdown_connections 2021-02-10T11:31:48.922 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.935 7f2d3a252700 1 --2- 172.21.2.112:0/3285500067 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2d140186c0 0x7f2d1401abc0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:48.922 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.935 7f2d3a252700 1 --2- 172.21.2.112:0/3285500067 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2d34106330 0x7f2d34106730 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:48.923 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.935 7f2d3a252700 1 --2- 172.21.2.112:0/3285500067 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2d341a7d70 0x7f2d34105d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:48.923 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.935 7f2d3a252700 1 --2- 172.21.2.112:0/3285500067 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2d340c3600 0x7f2d341a7750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:48.923 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.935 7f2d3a252700 1 -- 172.21.2.112:0/3285500067 >> 172.21.2.112:0/3285500067 conn(0x7f2d340a37f0 msgr2=0x7f2d340bf050 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:48.923 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.935 7f2d3a252700 1 -- 172.21.2.112:0/3285500067 shutdown_connections 2021-02-10T11:31:48.923 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:48.935 7f2d3a252700 1 -- 172.21.2.112:0/3285500067 wait complete. 2021-02-10T11:31:48.930 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:31:49.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.356 7f93620dd700 1 Processor -- start 2021-02-10T11:31:49.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.356 7f93620dd700 1 -- start start 2021-02-10T11:31:49.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.357 7f93620dd700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f935c1b0470 0x7f935c1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:49.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.357 7f93620dd700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f935c0f0b50 con 0x7f935c0ef610 2021-02-10T11:31:49.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.357 7f93620dd700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f935c0f24e0 con 0x7f935c0f20e0 2021-02-10T11:31:49.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.357 7f93620dd700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f935c0e5ea0 con 0x7f935c1b0470 2021-02-10T11:31:49.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.357 7f935affd700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f935c0ef610 0x7f935c0efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:47882/0 (socket says 172.21.2.112:47882) 2021-02-10T11:31:49.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.357 7f935affd700 1 -- 172.21.2.112:0/2811957907 learned_addr learned my addr 172.21.2.112:0/2811957907 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:31:49.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.357 7f935a7fc700 1 -- 172.21.2.112:0/2811957907 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1972749506 0 0) 0x7f934c0009b0 con 0x7f935c0ef610 2021-02-10T11:31:49.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.357 7f935a7fc700 1 -- 172.21.2.112:0/2811957907 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9348002fe0 con 0x7f935c0ef610 2021-02-10T11:31:49.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.358 7f935b7fe700 1 --2- 172.21.2.112:0/2811957907 >> v2:172.21.2.120:3300/0 conn(0x7f935c1b0470 0x7f935c1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:49.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.358 7f935a7fc700 1 -- 172.21.2.112:0/2811957907 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3363237932 0 0) 0x7f934c001010 con 0x7f935c0ef610 2021-02-10T11:31:49.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.358 7f935a7fc700 1 -- 172.21.2.112:0/2811957907 >> v2:172.21.2.120:3300/0 conn(0x7f935c1b0470 msgr2=0x7f935c1b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:49.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.358 7f935a7fc700 1 --2- 172.21.2.112:0/2811957907 >> v2:172.21.2.120:3300/0 conn(0x7f935c1b0470 0x7f935c1b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:49.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.358 7f935a7fc700 1 -- 172.21.2.112:0/2811957907 >> v1:172.21.2.120:6789/0 conn(0x7f935c0f20e0 legacy=0x7f935c0f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:49.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.358 7f935a7fc700 1 -- 172.21.2.112:0/2811957907 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f935c0e52e0 con 0x7f935c0ef610 2021-02-10T11:31:49.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.358 7f935a7fc700 1 -- 172.21.2.112:0/2811957907 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f934c001590 con 0x7f935c0ef610 2021-02-10T11:31:49.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.358 7f935a7fc700 1 -- 172.21.2.112:0/2811957907 >> v1:172.21.2.112:6789/0 conn(0x7f935c0ef610 legacy=0x7f935c0efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:49.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.358 7f935a7fc700 1 --2- 172.21.2.112:0/2811957907 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9348003a60 0x7f9348005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:49.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.358 7f935a7fc700 1 --2- 172.21.2.112:0/2811957907 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9348006790 0x7f9348008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:49.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.358 7f935a7fc700 1 --2- 172.21.2.112:0/2811957907 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9348009590 0x7f934800bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:49.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.358 7f935a7fc700 1 -- 172.21.2.112:0/2811957907 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f934800c300 con 0x7f9348003a60 2021-02-10T11:31:49.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.358 7f935a7fc700 1 -- 172.21.2.112:0/2811957907 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f934800c520 con 0x7f9348006790 2021-02-10T11:31:49.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.358 7f935a7fc700 1 -- 172.21.2.112:0/2811957907 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f934800c740 con 0x7f9348009590 2021-02-10T11:31:49.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.358 7f935b7fe700 1 --2- 172.21.2.112:0/2811957907 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9348003a60 0x7f9348005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:49.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.358 7f935affd700 1 --2- 172.21.2.112:0/2811957907 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9348006790 0x7f9348008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:49.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.358 7f935bfff700 1 --2- 172.21.2.112:0/2811957907 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9348009590 0x7f934800bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:49.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.358 7f935b7fe700 1 -- 172.21.2.112:0/2811957907 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9348009590 msgr2=0x7f934800bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:49.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.358 7f935b7fe700 1 --2- 172.21.2.112:0/2811957907 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9348009590 0x7f934800bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:49.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.358 7f935b7fe700 1 -- 172.21.2.112:0/2811957907 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9348006790 msgr2=0x7f9348008c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:49.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.358 7f935b7fe700 1 --2- 172.21.2.112:0/2811957907 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9348006790 0x7f9348008c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:49.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.359 7f935b7fe700 1 -- 172.21.2.112:0/2811957907 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93480509b0 con 0x7f9348003a60 2021-02-10T11:31:49.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.359 7f935bfff700 1 -- 172.21.2.112:0/2811957907 reap_dead start 2021-02-10T11:31:49.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.359 7f935affd700 1 --2- 172.21.2.112:0/2811957907 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9348006790 0x7f9348008c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:31:49.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.359 7f935b7fe700 1 --2- 172.21.2.112:0/2811957907 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9348003a60 0x7f9348005ef0 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 rx=0x7f934400c8f0 tx=0x7f934400cc90).ready entity=mon.0 client_cookie=d51474bf8fab28e9 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:49.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.359 7f935a7fc700 1 -- 172.21.2.112:0/2811957907 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9344000c90 con 0x7f9348003a60 2021-02-10T11:31:49.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.359 7f935a7fc700 1 -- 172.21.2.112:0/2811957907 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9344009310 con 0x7f9348003a60 2021-02-10T11:31:49.365 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.359 7f935a7fc700 1 -- 172.21.2.112:0/2811957907 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9344009830 con 0x7f9348003a60 2021-02-10T11:31:49.365 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.359 7f93620dd700 1 -- 172.21.2.112:0/2811957907 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9348003a60 msgr2=0x7f9348005ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:49.365 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.359 7f93620dd700 1 --2- 172.21.2.112:0/2811957907 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9348003a60 0x7f9348005ef0 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 rx=0x7f934400c8f0 tx=0x7f934400cc90).stop 2021-02-10T11:31:49.365 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.360 7f93620dd700 1 -- 172.21.2.112:0/2811957907 shutdown_connections 2021-02-10T11:31:49.365 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.360 7f93620dd700 1 --2- 172.21.2.112:0/2811957907 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9348003a60 0x7f9348005ef0 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:49.365 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.360 7f93620dd700 1 -- 172.21.2.112:0/2811957907 >> 172.21.2.112:0/2811957907 conn(0x7f935c0ce3e0 msgr2=0x7f935c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:49.365 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.360 7f93620dd700 1 -- 172.21.2.112:0/2811957907 shutdown_connections 2021-02-10T11:31:49.365 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.360 7f93620dd700 1 -- 172.21.2.112:0/2811957907 wait complete. 2021-02-10T11:31:49.365 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.374 7f93620dd700 1 Processor -- start 2021-02-10T11:31:49.365 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.374 7f93620dd700 1 -- start start 2021-02-10T11:31:49.366 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.375 7f93620dd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f935c0b7db0 0x7f935c0b8260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:49.367 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.375 7f93620dd700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f935c0b8880 0x7f935c1a7450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:49.367 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.375 7f93620dd700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f935c0b8c80 0x7f935c1a7a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:49.367 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.375 7f93620dd700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f935c0b79b0 con 0x7f935c0b7db0 2021-02-10T11:31:49.367 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.375 7f93620dd700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f935c1a8090 con 0x7f935c0b8c80 2021-02-10T11:31:49.367 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.375 7f93620dd700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f935c1a82b0 con 0x7f935c0b8880 2021-02-10T11:31:49.367 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.375 7f935b7fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f935c0b7db0 0x7f935c0b8260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:49.367 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.375 7f935b7fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f935c0b7db0 0x7f935c0b8260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:48432/0 (socket says 172.21.2.112:48432) 2021-02-10T11:31:49.367 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.375 7f935b7fe700 1 -- 172.21.2.112:0/12772670 learned_addr learned my addr 172.21.2.112:0/12772670 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:31:49.367 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.375 7f935b7fe700 1 -- 172.21.2.112:0/12772670 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f935c0b8880 msgr2=0x7f935c1a7450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:49.367 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.375 7f935b7fe700 1 --2- 172.21.2.112:0/12772670 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f935c0b8880 0x7f935c1a7450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:49.367 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.375 7f935b7fe700 1 -- 172.21.2.112:0/12772670 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f935c0b8c80 msgr2=0x7f935c1a7a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:49.367 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.375 7f935b7fe700 1 --2- 172.21.2.112:0/12772670 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f935c0b8c80 0x7f935c1a7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:49.367 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.375 7f935b7fe700 1 -- 172.21.2.112:0/12772670 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f935c190c10 con 0x7f935c0b7db0 2021-02-10T11:31:49.368 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.375 7f935b7fe700 1 --2- 172.21.2.112:0/12772670 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f935c0b7db0 0x7f935c0b8260 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 rx=0x7f934400c0d0 tx=0x7f934400b300).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:49.368 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.376 7f9353fff700 1 -- 172.21.2.112:0/12772670 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9344000940 con 0x7f935c0b7db0 2021-02-10T11:31:49.368 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.376 7f9353fff700 1 -- 172.21.2.112:0/12772670 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9344000c40 con 0x7f935c0b7db0 2021-02-10T11:31:49.368 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.376 7f9353fff700 1 -- 172.21.2.112:0/12772670 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f934400c960 con 0x7f935c0b7db0 2021-02-10T11:31:49.368 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.376 7f93620dd700 1 -- 172.21.2.112:0/12772670 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f935c190c10 con 0x7f935c0b7db0 2021-02-10T11:31:49.368 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.376 7f93620dd700 1 -- 172.21.2.112:0/12772670 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f935c0c1eb0 con 0x7f935c0b7db0 2021-02-10T11:31:49.368 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.376 7f9353fff700 1 -- 172.21.2.112:0/12772670 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9344006030 con 0x7f935c0b7db0 2021-02-10T11:31:49.368 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.376 7f9353fff700 1 --2- 172.21.2.112:0/12772670 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f93400186c0 0x7f934001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:49.368 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.376 7f9353fff700 1 -- 172.21.2.112:0/12772670 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f9344032540 con 0x7f935c0b7db0 2021-02-10T11:31:49.368 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.377 7f935affd700 1 --2- 172.21.2.112:0/12772670 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f93400186c0 0x7f934001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:49.368 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.377 7f935affd700 1 --2- 172.21.2.112:0/12772670 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f93400186c0 0x7f934001abc0 crc :-1 s=READY pgs=528 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:49.394 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.407 7f93620dd700 1 -- 172.21.2.112:0/12772670 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f935c105b30 con 0x7f935c0b7db0 2021-02-10T11:31:49.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.409 7f9353fff700 1 -- 172.21.2.112:0/12772670 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f934400c470 con 0x7f935c0b7db0 2021-02-10T11:31:49.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.536 7f93620dd700 1 -- 172.21.2.112:0/12772670 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f935c0f0b50 con 0x7f935c0b7db0 2021-02-10T11:31:49.524 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.536 7f9353fff700 1 -- 172.21.2.112:0/12772670 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v38) v1 ==== 100+0+10 (secure 0 0 0) 0x7f9344000d00 con 0x7f935c0b7db0 2021-02-10T11:31:49.524 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:31:49.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.538 7f93620dd700 1 -- 172.21.2.112:0/12772670 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f93400186c0 msgr2=0x7f934001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:49.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.538 7f93620dd700 1 --2- 172.21.2.112:0/12772670 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f93400186c0 0x7f934001abc0 crc :-1 s=READY pgs=528 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:49.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.538 7f93620dd700 1 -- 172.21.2.112:0/12772670 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f935c0b7db0 msgr2=0x7f935c0b8260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:49.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.538 7f93620dd700 1 --2- 172.21.2.112:0/12772670 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f935c0b7db0 0x7f935c0b8260 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 rx=0x7f934400c0d0 tx=0x7f934400b300).stop 2021-02-10T11:31:49.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.538 7f93620dd700 1 -- 172.21.2.112:0/12772670 shutdown_connections 2021-02-10T11:31:49.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.538 7f93620dd700 1 --2- 172.21.2.112:0/12772670 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f93400186c0 0x7f934001abc0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:49.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.538 7f93620dd700 1 --2- 172.21.2.112:0/12772670 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f935c0b8c80 0x7f935c1a7a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:49.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.538 7f93620dd700 1 --2- 172.21.2.112:0/12772670 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f935c0b8880 0x7f935c1a7450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:49.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.539 7f93620dd700 1 --2- 172.21.2.112:0/12772670 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f935c0b7db0 0x7f935c0b8260 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:49.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.539 7f93620dd700 1 -- 172.21.2.112:0/12772670 >> 172.21.2.112:0/12772670 conn(0x7f935c094e10 msgr2=0x7f935c0a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:49.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.539 7f93620dd700 1 -- 172.21.2.112:0/12772670 shutdown_connections 2021-02-10T11:31:49.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.539 7f93620dd700 1 -- 172.21.2.112:0/12772670 wait complete. 2021-02-10T11:31:49.534 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:31:49.535 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:31:49.960 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.956 7f52bbdac700 1 Processor -- start 2021-02-10T11:31:49.960 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.956 7f52bbdac700 1 -- start start 2021-02-10T11:31:49.961 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.957 7f52bbdac700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f52b40e5070 0x7f52b41b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:49.961 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.957 7f52bbdac700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f52b41b3e40 0x7f52b4080310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:49.961 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.957 7f52bbdac700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52b407ed80 con 0x7f52b40e5470 2021-02-10T11:31:49.961 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.957 7f52bbdac700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52b40d1790 con 0x7f52b41b3e40 2021-02-10T11:31:49.961 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.957 7f52bbdac700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52b40d1980 con 0x7f52b40e5070 2021-02-10T11:31:49.961 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.957 7f52b93c7700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f52b40e5470 0x7f52b407d690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:47900/0 (socket says 172.21.2.112:47900) 2021-02-10T11:31:49.961 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.957 7f52b93c7700 1 -- 172.21.2.112:0/3566589229 learned_addr learned my addr 172.21.2.112:0/3566589229 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:31:49.961 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.957 7f52ba3c9700 1 --2- 172.21.2.112:0/3566589229 >> v2:172.21.2.120:3300/0 conn(0x7f52b41b3e40 0x7f52b4080310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:49.961 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.957 7f52b8bc6700 1 -- 172.21.2.112:0/3566589229 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3890064008 0 0) 0x7f52a40010b0 con 0x7f52b40e5470 2021-02-10T11:31:49.961 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.957 7f52b9bc8700 1 --2- 172.21.2.112:0/3566589229 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f52b40e5070 0x7f52b41b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:49.961 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.957 7f52b8bc6700 1 -- 172.21.2.112:0/3566589229 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52a0002fe0 con 0x7f52b40e5470 2021-02-10T11:31:49.961 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.958 7f52ba3c9700 1 -- 172.21.2.112:0/3566589229 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f52b40e5070 msgr2=0x7f52b41b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:49.962 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.958 7f52ba3c9700 1 --2- 172.21.2.112:0/3566589229 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f52b40e5070 0x7f52b41b3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:49.962 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.958 7f52ba3c9700 1 -- 172.21.2.112:0/3566589229 >> v1:172.21.2.112:6789/0 conn(0x7f52b40e5470 legacy=0x7f52b407d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:49.962 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.958 7f52ba3c9700 1 -- 172.21.2.112:0/3566589229 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52b40d1ba0 con 0x7f52b41b3e40 2021-02-10T11:31:49.962 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.958 7f52b9bc8700 1 --2- 172.21.2.112:0/3566589229 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f52b40e5070 0x7f52b41b3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:31:49.962 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.958 7f52ba3c9700 1 --2- 172.21.2.112:0/3566589229 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f52b41b3e40 0x7f52b4080310 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 rx=0x7f52ac006e50 tx=0x7f52ac009e70).ready entity=mon.1 client_cookie=beebbfe4d35a7839 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:49.962 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.958 7f52b8bc6700 1 -- 172.21.2.112:0/3566589229 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f52ac006a20 con 0x7f52b41b3e40 2021-02-10T11:31:49.962 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.958 7f52b8bc6700 1 -- 172.21.2.112:0/3566589229 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52ac006790 con 0x7f52b41b3e40 2021-02-10T11:31:49.962 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.959 7f52bbdac700 1 -- 172.21.2.112:0/3566589229 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f52b41b3e40 msgr2=0x7f52b4080310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:49.962 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.959 7f52bbdac700 1 --2- 172.21.2.112:0/3566589229 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f52b41b3e40 0x7f52b4080310 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 rx=0x7f52ac006e50 tx=0x7f52ac009e70).stop 2021-02-10T11:31:49.962 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.958 7f52b8bc6700 1 -- 172.21.2.112:0/3566589229 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f52ac00b700 con 0x7f52b41b3e40 2021-02-10T11:31:49.962 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.959 7f52bbdac700 1 -- 172.21.2.112:0/3566589229 shutdown_connections 2021-02-10T11:31:49.962 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.959 7f52bbdac700 1 --2- 172.21.2.112:0/3566589229 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f52b41b3e40 0x7f52b4080310 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:49.963 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.959 7f52bbdac700 1 --2- 172.21.2.112:0/3566589229 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f52b40e5070 0x7f52b41b3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:49.963 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.959 7f52bbdac700 1 -- 172.21.2.112:0/3566589229 >> 172.21.2.112:0/3566589229 conn(0x7f52b40ce690 msgr2=0x7f52b40e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:49.963 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.959 7f52bbdac700 1 -- 172.21.2.112:0/3566589229 shutdown_connections 2021-02-10T11:31:49.963 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.959 7f52bbdac700 1 -- 172.21.2.112:0/3566589229 wait complete. 2021-02-10T11:31:49.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.973 7f52bbdac700 1 Processor -- start 2021-02-10T11:31:49.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.974 7f52bbdac700 1 -- start start 2021-02-10T11:31:49.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.974 7f52bbdac700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f52b408d400 0x7f52b4081230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:49.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.974 7f52bbdac700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f52b4081850 0x7f52b40acf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:49.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.974 7f52bbdac700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f52b4081c50 0x7f52b40ad5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:49.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.974 7f52bbdac700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52b4082110 con 0x7f52b4081850 2021-02-10T11:31:49.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.974 7f52bbdac700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52b408d800 con 0x7f52b4081c50 2021-02-10T11:31:49.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.974 7f52bbdac700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52b40adbc0 con 0x7f52b408d400 2021-02-10T11:31:49.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.974 7f52b93c7700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f52b4081850 0x7f52b40acf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:49.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.974 7f52b93c7700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f52b4081850 0x7f52b40acf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:48446/0 (socket says 172.21.2.112:48446) 2021-02-10T11:31:49.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.974 7f52b93c7700 1 -- 172.21.2.112:0/390413200 learned_addr learned my addr 172.21.2.112:0/390413200 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:31:49.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.974 7f52b93c7700 1 -- 172.21.2.112:0/390413200 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f52b408d400 msgr2=0x7f52b4081230 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:31:49.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.974 7f52b93c7700 1 --2- 172.21.2.112:0/390413200 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f52b408d400 0x7f52b4081230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:49.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.974 7f52b93c7700 1 -- 172.21.2.112:0/390413200 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f52b4081c50 msgr2=0x7f52b40ad5a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:31:49.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.974 7f52b93c7700 1 --2- 172.21.2.112:0/390413200 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f52b4081c50 0x7f52b40ad5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:49.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.974 7f52b93c7700 1 -- 172.21.2.112:0/390413200 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52b4187fc0 con 0x7f52b4081850 2021-02-10T11:31:49.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.975 7f52b93c7700 1 --2- 172.21.2.112:0/390413200 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f52b4081850 0x7f52b40acf80 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 rx=0x7f52a4001c60 tx=0x7f52a400a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:49.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.975 7f52aa7fc700 1 -- 172.21.2.112:0/390413200 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f52a400acc0 con 0x7f52b4081850 2021-02-10T11:31:49.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.975 7f52aa7fc700 1 -- 172.21.2.112:0/390413200 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52a4004140 con 0x7f52b4081850 2021-02-10T11:31:49.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.975 7f52aa7fc700 1 -- 172.21.2.112:0/390413200 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f52a40045a0 con 0x7f52b4081850 2021-02-10T11:31:49.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.975 7f52bbdac700 1 -- 172.21.2.112:0/390413200 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52b4187fc0 con 0x7f52b4081850 2021-02-10T11:31:49.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.975 7f52bbdac700 1 -- 172.21.2.112:0/390413200 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52b4187fc0 con 0x7f52b4081850 2021-02-10T11:31:49.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.976 7f52aa7fc700 1 -- 172.21.2.112:0/390413200 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f52a4004320 con 0x7f52b4081850 2021-02-10T11:31:49.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.976 7f52aa7fc700 1 --2- 172.21.2.112:0/390413200 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f529c0186c0 0x7f529c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:49.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.976 7f52aa7fc700 1 -- 172.21.2.112:0/390413200 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f52a4032560 con 0x7f52b4081850 2021-02-10T11:31:49.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.976 7f52b9bc8700 1 --2- 172.21.2.112:0/390413200 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f529c0186c0 0x7f529c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:49.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:49.976 7f52b9bc8700 1 --2- 172.21.2.112:0/390413200 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f529c0186c0 0x7f529c01abc0 crc :-1 s=READY pgs=529 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:49.994 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:50.007 7f52bbdac700 1 -- 172.21.2.112:0/390413200 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52b407ed80 con 0x7f52b4081850 2021-02-10T11:31:49.996 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:50.009 7f52aa7fc700 1 -- 172.21.2.112:0/390413200 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f52a4025f90 con 0x7f52b4081850 2021-02-10T11:31:50.124 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:50.137 7f52bbdac700 1 -- 172.21.2.112:0/390413200 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f52b407ed80 con 0x7f52b4081850 2021-02-10T11:31:50.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:50.137 7f52aa7fc700 1 -- 172.21.2.112:0/390413200 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+2099 (secure 0 0 0) 0x7f52a4025f90 con 0x7f52b4081850 2021-02-10T11:31:50.125 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:31:50.125 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":22,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7155,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/1957863967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":1957863967},{"type":"v1","addr":"172.21.2.112:6833","nonce":1957863967}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":18}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:31:40.190262","modified":"2021-02-10 11:31:42.195824","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5534},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5534":{"gid":5534,"name":"b","rank":0,"incarnation":19,"state":"up:active","state_seq":3,"addr":"172.21.2.120:6835/1513827094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":1513827094},{"type":"v1","addr":"172.21.2.120:6835","nonce":1513827094}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2021-02-10T11:31:50.125 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 22 2021-02-10T11:31:50.126 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:50.139 7f52bbdac700 1 -- 172.21.2.112:0/390413200 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f529c0186c0 msgr2=0x7f529c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:50.127 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:50.139 7f52bbdac700 1 --2- 172.21.2.112:0/390413200 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f529c0186c0 0x7f529c01abc0 crc :-1 s=READY pgs=529 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:50.127 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:50.139 7f52bbdac700 1 -- 172.21.2.112:0/390413200 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f52b4081850 msgr2=0x7f52b40acf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:50.127 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:50.139 7f52bbdac700 1 --2- 172.21.2.112:0/390413200 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f52b4081850 0x7f52b40acf80 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 rx=0x7f52a4001c60 tx=0x7f52a400a8d0).stop 2021-02-10T11:31:50.127 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:50.139 7f52bbdac700 1 -- 172.21.2.112:0/390413200 shutdown_connections 2021-02-10T11:31:50.127 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:50.139 7f52bbdac700 1 --2- 172.21.2.112:0/390413200 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f529c0186c0 0x7f529c01abc0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:50.127 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:50.139 7f52bbdac700 1 --2- 172.21.2.112:0/390413200 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f52b4081c50 0x7f52b40ad5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:50.127 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:50.139 7f52bbdac700 1 --2- 172.21.2.112:0/390413200 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f52b4081850 0x7f52b40acf80 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:50.127 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:50.139 7f52bbdac700 1 --2- 172.21.2.112:0/390413200 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f52b408d400 0x7f52b4081230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:50.127 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:50.139 7f52bbdac700 1 -- 172.21.2.112:0/390413200 >> 172.21.2.112:0/390413200 conn(0x7f52b41ac620 msgr2=0x7f52b40896e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:50.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:50.139 7f52bbdac700 1 -- 172.21.2.112:0/390413200 shutdown_connections 2021-02-10T11:31:50.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:50.139 7f52bbdac700 1 -- 172.21.2.112:0/390413200 wait complete. 2021-02-10T11:31:50.337 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:31:40.190262', 'modified': '2021-02-10 11:31:42.195824', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5534}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5534': {'gid': 5534, 'name': 'b', 'rank': 0, 'incarnation': 19, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.120:6835/1513827094', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.120:6834', 'nonce': 1513827094}, {'type': 'v1', 'addr': '172.21.2.120:6835', 'nonce': 1513827094}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [7], 'metadata_pool': 6, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:31:50.337 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:31:50.338 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:31:50.734 INFO:teuthology.orchestra.run.gibba020.stdout:{ 2021-02-10T11:31:50.735 INFO:teuthology.orchestra.run.gibba020.stdout: "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", 2021-02-10T11:31:50.735 INFO:teuthology.orchestra.run.gibba020.stdout: "whoami": 0, 2021-02-10T11:31:50.735 INFO:teuthology.orchestra.run.gibba020.stdout: "id": 5534, 2021-02-10T11:31:50.735 INFO:teuthology.orchestra.run.gibba020.stdout: "want_state": "up:active", 2021-02-10T11:31:50.735 INFO:teuthology.orchestra.run.gibba020.stdout: "state": "up:active", 2021-02-10T11:31:50.735 INFO:teuthology.orchestra.run.gibba020.stdout: "rank_uptime": 10.555807388, 2021-02-10T11:31:50.735 INFO:teuthology.orchestra.run.gibba020.stdout: "mdsmap_epoch": 22, 2021-02-10T11:31:50.735 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch": 38, 2021-02-10T11:31:50.735 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch_barrier": 38, 2021-02-10T11:31:50.735 INFO:teuthology.orchestra.run.gibba020.stdout: "uptime": 16.188273032000001 2021-02-10T11:31:50.736 INFO:teuthology.orchestra.run.gibba020.stdout:} 2021-02-10T11:31:50.738 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", "whoami": 0, "id": 5534, "want_state": "up:active", "state": "up:active", "rank_uptime": 10.555807388, "mdsmap_epoch": 22, "osdmap_epoch": 38, "osdmap_epoch_barrier": 38, "uptime": 16.188273032000001 } 2021-02-10T11:31:50.739 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:31:50.739 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2021-02-10T11:31:50.739 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@gibba012.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2021-02-10T11:31:50.739 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:50.739 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:31:51.074 DEBUG:teuthology.orchestra.run.gibba012:> sudo modprobe fuse 2021-02-10T11:31:51.417 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:31:51.417 DEBUG:teuthology.orchestra.run.gibba012:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:31:51.760 INFO:teuthology.orchestra.run.gibba012.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:31:51.760 INFO:teuthology.orchestra.run.gibba012.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:31:51.761 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:31:51.761 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:51.761 DEBUG:teuthology.orchestra.run.gibba012:> ls /sys/fs/fuse/connections 2021-02-10T11:31:52.094 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2021-02-10T11:31:52.094 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:31:52.097 DEBUG:teuthology.orchestra.run.gibba012:> sudo modprobe fuse 2021-02-10T11:31:52.440 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:31:52.440 DEBUG:teuthology.orchestra.run.gibba012:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:31:52.505 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba012.stderr:ceph-fuse[121274]: starting ceph client 2021-02-10T11:31:52.505 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba012.stderr:2021-02-10 11:31:52.518 7f81321eff80 -1 init, newargv = 0x55a2322af0e0 newargc=9 2021-02-10T11:31:52.511 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba012.stderr:ceph-fuse[121274]: starting fuse 2021-02-10T11:31:52.782 INFO:teuthology.orchestra.run.gibba012.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:31:52.782 INFO:teuthology.orchestra.run.gibba012.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:31:52.783 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:31:52.783 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:52.783 DEBUG:teuthology.orchestra.run.gibba012:> ls /sys/fs/fuse/connections 2021-02-10T11:31:53.117 INFO:teuthology.orchestra.run.gibba012.stdout:42 2021-02-10T11:31:53.118 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2021-02-10T11:31:53.118 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:53.118 DEBUG:teuthology.orchestra.run.gibba012:> sudo python3 -c ' 2021-02-10T11:31:53.118 DEBUG:teuthology.orchestra.run.gibba012:> import glob 2021-02-10T11:31:53.118 DEBUG:teuthology.orchestra.run.gibba012:> import re 2021-02-10T11:31:53.118 DEBUG:teuthology.orchestra.run.gibba012:> import os 2021-02-10T11:31:53.119 DEBUG:teuthology.orchestra.run.gibba012:> import subprocess 2021-02-10T11:31:53.119 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:31:53.119 DEBUG:teuthology.orchestra.run.gibba012:> def find_socket(client_name): 2021-02-10T11:31:53.119 DEBUG:teuthology.orchestra.run.gibba012:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2021-02-10T11:31:53.119 DEBUG:teuthology.orchestra.run.gibba012:> files = glob.glob(asok_path) 2021-02-10T11:31:53.119 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:31:53.119 DEBUG:teuthology.orchestra.run.gibba012:> # Given a non-glob path, it better be there 2021-02-10T11:31:53.119 DEBUG:teuthology.orchestra.run.gibba012:> if "*" not in asok_path: 2021-02-10T11:31:53.119 DEBUG:teuthology.orchestra.run.gibba012:> assert(len(files) == 1) 2021-02-10T11:31:53.119 DEBUG:teuthology.orchestra.run.gibba012:> return files[0] 2021-02-10T11:31:53.119 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:31:53.119 DEBUG:teuthology.orchestra.run.gibba012:> for f in files: 2021-02-10T11:31:53.119 DEBUG:teuthology.orchestra.run.gibba012:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:31:53.120 DEBUG:teuthology.orchestra.run.gibba012:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:31:53.120 DEBUG:teuthology.orchestra.run.gibba012:> return f 2021-02-10T11:31:53.120 DEBUG:teuthology.orchestra.run.gibba012:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:31:53.120 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:31:53.120 DEBUG:teuthology.orchestra.run.gibba012:> print(find_socket("client.0")) 2021-02-10T11:31:53.120 DEBUG:teuthology.orchestra.run.gibba012:> ' 2021-02-10T11:31:53.480 INFO:teuthology.orchestra.run.gibba012.stdout:/var/run/ceph/ceph-client.0.121274.asok 2021-02-10T11:31:53.482 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.121274.asok 2021-02-10T11:31:53.483 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:53.483 DEBUG:teuthology.orchestra.run.gibba012:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.121274.asok status 2021-02-10T11:31:53.873 INFO:teuthology.orchestra.run.gibba012.stdout:{ 2021-02-10T11:31:53.873 INFO:teuthology.orchestra.run.gibba012.stdout: "metadata": { 2021-02-10T11:31:53.873 INFO:teuthology.orchestra.run.gibba012.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:31:53.873 INFO:teuthology.orchestra.run.gibba012.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:31:53.873 INFO:teuthology.orchestra.run.gibba012.stdout: "entity_id": "0", 2021-02-10T11:31:53.873 INFO:teuthology.orchestra.run.gibba012.stdout: "hostname": "gibba012", 2021-02-10T11:31:53.874 INFO:teuthology.orchestra.run.gibba012.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2021-02-10T11:31:53.874 INFO:teuthology.orchestra.run.gibba012.stdout: "pid": "121274", 2021-02-10T11:31:53.874 INFO:teuthology.orchestra.run.gibba012.stdout: "root": "/" 2021-02-10T11:31:53.874 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:31:53.874 INFO:teuthology.orchestra.run.gibba012.stdout: "dentry_count": 0, 2021-02-10T11:31:53.874 INFO:teuthology.orchestra.run.gibba012.stdout: "dentry_pinned_count": 0, 2021-02-10T11:31:53.874 INFO:teuthology.orchestra.run.gibba012.stdout: "id": 7320, 2021-02-10T11:31:53.874 INFO:teuthology.orchestra.run.gibba012.stdout: "inst": { 2021-02-10T11:31:53.874 INFO:teuthology.orchestra.run.gibba012.stdout: "name": { 2021-02-10T11:31:53.874 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "client", 2021-02-10T11:31:53.874 INFO:teuthology.orchestra.run.gibba012.stdout: "num": 7320 2021-02-10T11:31:53.875 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:31:53.875 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": { 2021-02-10T11:31:53.875 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:31:53.875 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:0", 2021-02-10T11:31:53.875 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 1001703705 2021-02-10T11:31:53.875 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:31:53.875 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:31:53.875 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": { 2021-02-10T11:31:53.875 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:31:53.875 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:0", 2021-02-10T11:31:53.875 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 1001703705 2021-02-10T11:31:53.876 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:31:53.876 INFO:teuthology.orchestra.run.gibba012.stdout: "inst_str": "client.7320 172.21.2.112:0/1001703705", 2021-02-10T11:31:53.876 INFO:teuthology.orchestra.run.gibba012.stdout: "addr_str": "172.21.2.112:0/1001703705", 2021-02-10T11:31:53.876 INFO:teuthology.orchestra.run.gibba012.stdout: "inode_count": 1, 2021-02-10T11:31:53.876 INFO:teuthology.orchestra.run.gibba012.stdout: "mds_epoch": 22, 2021-02-10T11:31:53.876 INFO:teuthology.orchestra.run.gibba012.stdout: "osd_epoch": 38, 2021-02-10T11:31:53.876 INFO:teuthology.orchestra.run.gibba012.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:31:53.876 INFO:teuthology.orchestra.run.gibba012.stdout: "blacklisted": false 2021-02-10T11:31:53.876 INFO:teuthology.orchestra.run.gibba012.stdout:} 2021-02-10T11:31:53.877 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:53.878 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:31:53.878 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:31:54.222 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:31:54.222 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:31:54.223 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:54.223 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:31:54.568 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:31:54.996 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:54.991 7f4eb3dd5700 1 Processor -- start 2021-02-10T11:31:54.996 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:54.991 7f4eb3dd5700 1 -- start start 2021-02-10T11:31:54.996 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:54.992 7f4eb3dd5700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f4eac0e5070 0x7f4eac1b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:54.996 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:54.992 7f4eb3dd5700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4eac07ed80 con 0x7f4eac1b3e40 2021-02-10T11:31:54.996 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:54.992 7f4eb3dd5700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4eac07e670 con 0x7f4eac0e5470 2021-02-10T11:31:54.996 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:54.992 7f4eb3dd5700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4eac07e890 con 0x7f4eac0e5070 2021-02-10T11:31:54.996 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:54.992 7f4eb13f0700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f4eac1b3e40 0x7f4eac07d690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:47928/0 (socket says 172.21.2.112:47928) 2021-02-10T11:31:54.996 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:54.992 7f4eb13f0700 1 -- 172.21.2.112:0/873165139 learned_addr learned my addr 172.21.2.112:0/873165139 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:31:54.996 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:54.992 7f4eb1bf1700 1 --2- 172.21.2.112:0/873165139 >> v2:172.21.2.120:3300/0 conn(0x7f4eac0e5070 0x7f4eac1b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:54.996 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:54.992 7f4eb0bef700 1 -- 172.21.2.112:0/873165139 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 294130117 0 0) 0x7f4ea00009b0 con 0x7f4eac1b3e40 2021-02-10T11:31:54.997 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:54.992 7f4eb0bef700 1 -- 172.21.2.112:0/873165139 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e98002fe0 con 0x7f4eac1b3e40 2021-02-10T11:31:54.997 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:54.992 7f4eb0bef700 1 -- 172.21.2.112:0/873165139 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3704355819 0 0) 0x7f4ea4001400 con 0x7f4eac0e5470 2021-02-10T11:31:54.997 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:54.992 7f4eb0bef700 1 -- 172.21.2.112:0/873165139 --> v1:172.21.2.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e98003210 con 0x7f4eac0e5470 2021-02-10T11:31:54.997 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:54.993 7f4eb0bef700 1 -- 172.21.2.112:0/873165139 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 850874550 0 0) 0x7f4ea0001010 con 0x7f4eac1b3e40 2021-02-10T11:31:54.997 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:54.993 7f4eb0bef700 1 -- 172.21.2.112:0/873165139 >> v2:172.21.2.120:3300/0 conn(0x7f4eac0e5070 msgr2=0x7f4eac1b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:54.997 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:54.993 7f4eb0bef700 1 --2- 172.21.2.112:0/873165139 >> v2:172.21.2.120:3300/0 conn(0x7f4eac0e5070 0x7f4eac1b3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:54.997 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:54.993 7f4eb0bef700 1 -- 172.21.2.112:0/873165139 >> v1:172.21.2.120:6789/0 conn(0x7f4eac0e5470 legacy=0x7f4eac07de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:54.997 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:54.993 7f4eb0bef700 1 -- 172.21.2.112:0/873165139 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4eac07eab0 con 0x7f4eac1b3e40 2021-02-10T11:31:54.997 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:54.993 7f4eb0bef700 1 -- 172.21.2.112:0/873165139 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f4ea0001590 con 0x7f4eac1b3e40 2021-02-10T11:31:54.997 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:54.993 7f4eb0bef700 1 -- 172.21.2.112:0/873165139 >> v1:172.21.2.112:6789/0 conn(0x7f4eac1b3e40 legacy=0x7f4eac07d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:54.997 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:54.993 7f4eb0bef700 1 --2- 172.21.2.112:0/873165139 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4e98003b90 0x7f4e98005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:54.997 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:54.993 7f4eb0bef700 1 --2- 172.21.2.112:0/873165139 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4e980067a0 0x7f4e98008ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:54.997 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:54.993 7f4eb0bef700 1 --2- 172.21.2.112:0/873165139 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4e980095a0 0x7f4e9800bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:54.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:54.993 7f4eb0bef700 1 -- 172.21.2.112:0/873165139 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e9800c310 con 0x7f4e980095a0 2021-02-10T11:31:54.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:54.993 7f4eb0bef700 1 -- 172.21.2.112:0/873165139 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e98050500 con 0x7f4e98003b90 2021-02-10T11:31:54.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:54.993 7f4eb0bef700 1 -- 172.21.2.112:0/873165139 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e98050720 con 0x7f4e980067a0 2021-02-10T11:31:54.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:54.993 7f4eb23f2700 1 --2- 172.21.2.112:0/873165139 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4e980095a0 0x7f4e9800bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:54.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:54.993 7f4eb1bf1700 1 --2- 172.21.2.112:0/873165139 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4e98003b90 0x7f4e98005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:54.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:54.993 7f4eb13f0700 1 --2- 172.21.2.112:0/873165139 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4e980067a0 0x7f4e98008ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:54.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:54.993 7f4eb23f2700 1 -- 172.21.2.112:0/873165139 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4e980067a0 msgr2=0x7f4e98008ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:54.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:54.993 7f4eb23f2700 1 --2- 172.21.2.112:0/873165139 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4e980067a0 0x7f4e98008ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:55.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:54.993 7f4eb23f2700 1 -- 172.21.2.112:0/873165139 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4e98003b90 msgr2=0x7f4e98005fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:55.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:54.993 7f4eb23f2700 1 --2- 172.21.2.112:0/873165139 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4e98003b90 0x7f4e98005fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:55.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:54.993 7f4eb23f2700 1 -- 172.21.2.112:0/873165139 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e980509c0 con 0x7f4e980095a0 2021-02-10T11:31:55.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:54.993 7f4eb23f2700 1 -- 172.21.2.112:0/873165139 reap_dead start 2021-02-10T11:31:55.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:54.993 7f4eb23f2700 1 --2- 172.21.2.112:0/873165139 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4e980095a0 0x7f4e9800bac0 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 rx=0x7f4ea40013c0 tx=0x7f4ea4009e20).ready entity=mon.0 client_cookie=2aea48094a961ceb server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:55.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:54.993 7f4eb0bef700 1 -- 172.21.2.112:0/873165139 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4ea4001400 con 0x7f4e980095a0 2021-02-10T11:31:55.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:54.993 7f4eb0bef700 1 -- 172.21.2.112:0/873165139 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ea4006a60 con 0x7f4e980095a0 2021-02-10T11:31:55.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:54.993 7f4eb0bef700 1 -- 172.21.2.112:0/873165139 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4ea4001740 con 0x7f4e980095a0 2021-02-10T11:31:55.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:54.993 7f4eb3dd5700 1 -- 172.21.2.112:0/873165139 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4e980095a0 msgr2=0x7f4e9800bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:55.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:54.993 7f4eb3dd5700 1 --2- 172.21.2.112:0/873165139 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4e980095a0 0x7f4e9800bac0 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 rx=0x7f4ea40013c0 tx=0x7f4ea4009e20).stop 2021-02-10T11:31:55.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:54.994 7f4eb3dd5700 1 -- 172.21.2.112:0/873165139 shutdown_connections 2021-02-10T11:31:55.001 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:54.994 7f4eb3dd5700 1 --2- 172.21.2.112:0/873165139 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4e980095a0 0x7f4e9800bac0 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:55.001 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:54.994 7f4eb3dd5700 1 -- 172.21.2.112:0/873165139 >> 172.21.2.112:0/873165139 conn(0x7f4eac0ce690 msgr2=0x7f4eac0e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:55.001 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:54.994 7f4eb3dd5700 1 -- 172.21.2.112:0/873165139 shutdown_connections 2021-02-10T11:31:55.001 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:54.994 7f4eb3dd5700 1 -- 172.21.2.112:0/873165139 wait complete. 2021-02-10T11:31:55.001 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:55.009 7f4eb3dd5700 1 Processor -- start 2021-02-10T11:31:55.001 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:55.009 7f4eb3dd5700 1 -- start start 2021-02-10T11:31:55.001 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:55.009 7f4eb3dd5700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4eac1ad6e0 0x7f4eac0a3210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:55.001 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:55.009 7f4eb3dd5700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4eac08da10 0x7f4eac1ae300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:55.001 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:55.009 7f4eb3dd5700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4eac08de10 0x7f4eac1ae920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:55.001 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:55.009 7f4eb3dd5700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4eac1add80 con 0x7f4eac1ad6e0 2021-02-10T11:31:55.001 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:55.009 7f4eb3dd5700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4eac1adae0 con 0x7f4eac08da10 2021-02-10T11:31:55.001 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:55.009 7f4eb3dd5700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4eac0a3830 con 0x7f4eac08de10 2021-02-10T11:31:55.001 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:55.010 7f4eb1bf1700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4eac1ad6e0 0x7f4eac0a3210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:55.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:55.010 7f4eb1bf1700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4eac1ad6e0 0x7f4eac0a3210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:48480/0 (socket says 172.21.2.112:48480) 2021-02-10T11:31:55.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:55.010 7f4eb1bf1700 1 -- 172.21.2.112:0/2253146567 learned_addr learned my addr 172.21.2.112:0/2253146567 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:31:55.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:55.010 7f4eb1bf1700 1 -- 172.21.2.112:0/2253146567 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4eac08de10 msgr2=0x7f4eac1ae920 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:31:55.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:55.010 7f4eb1bf1700 1 --2- 172.21.2.112:0/2253146567 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4eac08de10 0x7f4eac1ae920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:55.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:55.010 7f4eb1bf1700 1 -- 172.21.2.112:0/2253146567 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4eac08da10 msgr2=0x7f4eac1ae300 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:31:55.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:55.010 7f4eb1bf1700 1 --2- 172.21.2.112:0/2253146567 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4eac08da10 0x7f4eac1ae300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:55.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:55.010 7f4eb1bf1700 1 -- 172.21.2.112:0/2253146567 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4eac1aef40 con 0x7f4eac1ad6e0 2021-02-10T11:31:55.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:55.010 7f4eb1bf1700 1 --2- 172.21.2.112:0/2253146567 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4eac1ad6e0 0x7f4eac0a3210 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 rx=0x7f4ea8006f60 tx=0x7f4ea800b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:55.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:55.010 7f4e9e7fc700 1 -- 172.21.2.112:0/2253146567 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4ea800bd60 con 0x7f4eac1ad6e0 2021-02-10T11:31:55.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:55.010 7f4e9e7fc700 1 -- 172.21.2.112:0/2253146567 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ea8000c90 con 0x7f4eac1ad6e0 2021-02-10T11:31:55.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:55.010 7f4e9e7fc700 1 -- 172.21.2.112:0/2253146567 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4ea8002220 con 0x7f4eac1ad6e0 2021-02-10T11:31:55.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:55.010 7f4eb3dd5700 1 -- 172.21.2.112:0/2253146567 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4eac1aef40 con 0x7f4eac1ad6e0 2021-02-10T11:31:55.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:55.010 7f4eb3dd5700 1 -- 172.21.2.112:0/2253146567 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4eac1aef40 con 0x7f4eac1ad6e0 2021-02-10T11:31:55.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:55.011 7f4e9e7fc700 1 -- 172.21.2.112:0/2253146567 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4ea80052e0 con 0x7f4eac1ad6e0 2021-02-10T11:31:55.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:55.011 7f4e9e7fc700 1 --2- 172.21.2.112:0/2253146567 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4e940186c0 0x7f4e9401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:55.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:55.011 7f4e9e7fc700 1 -- 172.21.2.112:0/2253146567 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f4ea8032520 con 0x7f4eac1ad6e0 2021-02-10T11:31:55.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:55.011 7f4eb13f0700 1 --2- 172.21.2.112:0/2253146567 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4e940186c0 0x7f4e9401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:55.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:55.012 7f4eb13f0700 1 --2- 172.21.2.112:0/2253146567 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4e940186c0 0x7f4e9401abc0 crc :-1 s=READY pgs=530 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:55.029 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:55.042 7f4eb3dd5700 1 -- 172.21.2.112:0/2253146567 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4eac07ed80 con 0x7f4eac1ad6e0 2021-02-10T11:31:55.031 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:55.044 7f4e9e7fc700 1 -- 172.21.2.112:0/2253146567 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4ea8027820 con 0x7f4eac1ad6e0 2021-02-10T11:31:55.156 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:55.169 7f4eb3dd5700 1 -- 172.21.2.112:0/2253146567 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4eac07ed80 con 0x7f4eac1ad6e0 2021-02-10T11:31:55.157 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:55.170 7f4e9e7fc700 1 -- 172.21.2.112:0/2253146567 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v38) v1 ==== 74+0+13993 (secure 0 0 0) 0x7f4ea8027820 con 0x7f4eac1ad6e0 2021-02-10T11:31:55.157 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:31:55.158 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":38,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:31:41.192016","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:31:38.162654","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":7,"pool_name":"cephfs_data","create_time":"2021-02-10 11:31:39.233986","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:31:55.159 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:55.171 7f4eb3dd5700 1 -- 172.21.2.112:0/2253146567 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4e940186c0 msgr2=0x7f4e9401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:55.159 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:55.171 7f4eb3dd5700 1 --2- 172.21.2.112:0/2253146567 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4e940186c0 0x7f4e9401abc0 crc :-1 s=READY pgs=530 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:55.159 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:55.171 7f4eb3dd5700 1 -- 172.21.2.112:0/2253146567 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4eac1ad6e0 msgr2=0x7f4eac0a3210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:55.159 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:55.171 7f4eb3dd5700 1 --2- 172.21.2.112:0/2253146567 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4eac1ad6e0 0x7f4eac0a3210 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 rx=0x7f4ea8006f60 tx=0x7f4ea800b970).stop 2021-02-10T11:31:55.159 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:55.171 7f4eb3dd5700 1 -- 172.21.2.112:0/2253146567 shutdown_connections 2021-02-10T11:31:55.159 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:55.171 7f4eb3dd5700 1 --2- 172.21.2.112:0/2253146567 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4e940186c0 0x7f4e9401abc0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:55.159 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:55.171 7f4eb3dd5700 1 --2- 172.21.2.112:0/2253146567 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4eac08da10 0x7f4eac1ae300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:55.159 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:55.171 7f4eb3dd5700 1 --2- 172.21.2.112:0/2253146567 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4eac08de10 0x7f4eac1ae920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:55.159 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:55.171 7f4eb3dd5700 1 --2- 172.21.2.112:0/2253146567 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4eac1ad6e0 0x7f4eac0a3210 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:55.159 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:55.171 7f4eb3dd5700 1 -- 172.21.2.112:0/2253146567 >> 172.21.2.112:0/2253146567 conn(0x7f4eac1a31b0 msgr2=0x7f4eac18c8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:55.160 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:55.171 7f4eb3dd5700 1 -- 172.21.2.112:0/2253146567 shutdown_connections 2021-02-10T11:31:55.160 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:55.172 7f4eb3dd5700 1 -- 172.21.2.112:0/2253146567 wait complete. 2021-02-10T11:31:55.168 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:13:57.508758', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 6, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:31:38.162654', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '37', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 7, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:31:39.233986', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:31:55.168 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:31:55.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:55.594 7fca174bc700 1 Processor -- start 2021-02-10T11:31:55.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:55.595 7fca174bc700 1 -- start start 2021-02-10T11:31:55.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:55.595 7fca174bc700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fca101b0870 0x7fca100f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:55.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:55.595 7fca174bc700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fca100efe00 0x7fca100f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:55.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:55.595 7fca174bc700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca101b0c70 con 0x7fca101b0470 2021-02-10T11:31:55.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:55.595 7fca174bc700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca100e5ea0 con 0x7fca101b0870 2021-02-10T11:31:55.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:55.595 7fca174bc700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca100f0b50 con 0x7fca100efe00 2021-02-10T11:31:55.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:55.595 7fca14ad7700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fca101b0870 0x7fca100f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:55.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:55.595 7fca152d8700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fca101b0470 0x7fca100ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:47948/0 (socket says 172.21.2.112:47948) 2021-02-10T11:31:55.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:55.595 7fca152d8700 1 -- 172.21.2.112:0/2778805105 learned_addr learned my addr 172.21.2.112:0/2778805105 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:31:55.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:55.595 7fca15ad9700 1 --2- 172.21.2.112:0/2778805105 >> v2:172.21.2.120:3300/0 conn(0x7fca100efe00 0x7fca100f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:55.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:55.596 7fca14ad7700 1 -- 172.21.2.112:0/2778805105 >> v2:172.21.2.120:3300/0 conn(0x7fca100efe00 msgr2=0x7fca100f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:55.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:55.596 7fca14ad7700 1 --2- 172.21.2.112:0/2778805105 >> v2:172.21.2.120:3300/0 conn(0x7fca100efe00 0x7fca100f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:55.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:55.596 7fca14ad7700 1 -- 172.21.2.112:0/2778805105 >> v1:172.21.2.112:6789/0 conn(0x7fca101b0470 legacy=0x7fca100ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:55.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:55.596 7fca14ad7700 1 -- 172.21.2.112:0/2778805105 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca100f0820 con 0x7fca101b0870 2021-02-10T11:31:55.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:55.596 7fca15ad9700 1 --2- 172.21.2.112:0/2778805105 >> v2:172.21.2.120:3300/0 conn(0x7fca100efe00 0x7fca100f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:31:55.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:55.596 7fca14ad7700 1 --2- 172.21.2.112:0/2778805105 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fca101b0870 0x7fca100f20e0 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 rx=0x7fca04006650 tx=0x7fca04009a80).ready entity=mon.0 client_cookie=2f2361d6bb4bbf27 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:55.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:55.596 7fca03fff700 1 -- 172.21.2.112:0/2778805105 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fca040009b0 con 0x7fca101b0870 2021-02-10T11:31:55.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:55.596 7fca03fff700 1 -- 172.21.2.112:0/2778805105 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca0400c0c0 con 0x7fca101b0870 2021-02-10T11:31:55.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:55.596 7fca03fff700 1 -- 172.21.2.112:0/2778805105 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fca0400c520 con 0x7fca101b0870 2021-02-10T11:31:55.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:55.596 7fca174bc700 1 -- 172.21.2.112:0/2778805105 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fca101b0870 msgr2=0x7fca100f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:55.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:55.596 7fca174bc700 1 --2- 172.21.2.112:0/2778805105 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fca101b0870 0x7fca100f20e0 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 rx=0x7fca04006650 tx=0x7fca04009a80).stop 2021-02-10T11:31:55.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:55.597 7fca174bc700 1 -- 172.21.2.112:0/2778805105 shutdown_connections 2021-02-10T11:31:55.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:55.597 7fca174bc700 1 --2- 172.21.2.112:0/2778805105 >> v2:172.21.2.120:3300/0 conn(0x7fca100efe00 0x7fca100f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:55.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:55.597 7fca174bc700 1 --2- 172.21.2.112:0/2778805105 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fca101b0870 0x7fca100f20e0 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:55.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:55.597 7fca174bc700 1 -- 172.21.2.112:0/2778805105 >> 172.21.2.112:0/2778805105 conn(0x7fca100ce3e0 msgr2=0x7fca100e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:55.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:55.597 7fca174bc700 1 -- 172.21.2.112:0/2778805105 shutdown_connections 2021-02-10T11:31:55.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:55.597 7fca174bc700 1 -- 172.21.2.112:0/2778805105 wait complete. 2021-02-10T11:31:55.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:55.612 7fca174bc700 1 Processor -- start 2021-02-10T11:31:55.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:55.612 7fca174bc700 1 -- start start 2021-02-10T11:31:55.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:55.612 7fca174bc700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fca100a0420 0x7fca100a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:55.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:55.612 7fca174bc700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fca10189d00 0x7fca1018a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:55.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:55.612 7fca174bc700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fca1018a720 0x7fca100b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:55.606 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:55.612 7fca174bc700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca10189780 con 0x7fca100a0420 2021-02-10T11:31:55.606 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:55.612 7fca174bc700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca1018ab20 con 0x7fca1018a720 2021-02-10T11:31:55.606 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:55.612 7fca174bc700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca100a1140 con 0x7fca10189d00 2021-02-10T11:31:55.606 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:55.612 7fca152d8700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fca100a0420 0x7fca100a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:55.606 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:55.612 7fca152d8700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fca100a0420 0x7fca100a0820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:48492/0 (socket says 172.21.2.112:48492) 2021-02-10T11:31:55.606 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:55.612 7fca152d8700 1 -- 172.21.2.112:0/267737353 learned_addr learned my addr 172.21.2.112:0/267737353 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:31:55.606 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:55.612 7fca14ad7700 1 --2- 172.21.2.112:0/267737353 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fca10189d00 0x7fca1018a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:55.606 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:55.613 7fca15ad9700 1 --2- 172.21.2.112:0/267737353 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fca1018a720 0x7fca100b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:55.606 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:55.613 7fca152d8700 1 -- 172.21.2.112:0/267737353 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fca10189d00 msgr2=0x7fca1018a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:55.606 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:55.613 7fca152d8700 1 --2- 172.21.2.112:0/267737353 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fca10189d00 0x7fca1018a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:55.606 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:55.613 7fca152d8700 1 -- 172.21.2.112:0/267737353 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fca1018a720 msgr2=0x7fca100b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:55.606 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:55.613 7fca152d8700 1 --2- 172.21.2.112:0/267737353 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fca1018a720 0x7fca100b7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:55.607 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:55.613 7fca152d8700 1 -- 172.21.2.112:0/267737353 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca100b8200 con 0x7fca100a0420 2021-02-10T11:31:55.607 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:55.613 7fca14ad7700 1 --2- 172.21.2.112:0/267737353 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fca10189d00 0x7fca1018a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:31:55.607 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:55.613 7fca152d8700 1 --2- 172.21.2.112:0/267737353 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fca100a0420 0x7fca100a0820 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 rx=0x7fca0c001c60 tx=0x7fca0c00a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:55.607 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:55.613 7fca01ffb700 1 -- 172.21.2.112:0/267737353 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fca0c00acc0 con 0x7fca100a0420 2021-02-10T11:31:55.607 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:55.613 7fca01ffb700 1 -- 172.21.2.112:0/267737353 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca0c004140 con 0x7fca100a0420 2021-02-10T11:31:55.607 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:55.613 7fca01ffb700 1 -- 172.21.2.112:0/267737353 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fca0c004620 con 0x7fca100a0420 2021-02-10T11:31:55.607 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:55.613 7fca174bc700 1 -- 172.21.2.112:0/267737353 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca100b8200 con 0x7fca100a0420 2021-02-10T11:31:55.607 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:55.613 7fca174bc700 1 -- 172.21.2.112:0/267737353 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca100b8790 con 0x7fca100a0420 2021-02-10T11:31:55.607 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:55.614 7fca01ffb700 1 -- 172.21.2.112:0/267737353 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fca0c004320 con 0x7fca100a0420 2021-02-10T11:31:55.607 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:55.614 7fca01ffb700 1 --2- 172.21.2.112:0/267737353 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc9f00186c0 0x7fc9f001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:55.607 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:55.614 7fca01ffb700 1 -- 172.21.2.112:0/267737353 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fca0c032410 con 0x7fca100a0420 2021-02-10T11:31:55.607 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:55.614 7fca14ad7700 1 --2- 172.21.2.112:0/267737353 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc9f00186c0 0x7fc9f001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:55.607 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:55.615 7fca14ad7700 1 --2- 172.21.2.112:0/267737353 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc9f00186c0 0x7fc9f001abc0 crc :-1 s=READY pgs=531 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:55.632 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:55.645 7fca174bc700 1 -- 172.21.2.112:0/267737353 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca100a1360 con 0x7fca100a0420 2021-02-10T11:31:55.634 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:55.647 7fca01ffb700 1 -- 172.21.2.112:0/267737353 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fca0c000db0 con 0x7fca100a0420 2021-02-10T11:31:55.763 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:55.775 7fca174bc700 1 -- 172.21.2.112:0/267737353 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7fca100a1360 con 0x7fca100a0420 2021-02-10T11:31:55.763 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:55.776 7fca01ffb700 1 -- 172.21.2.112:0/267737353 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v38) v1 ==== 92+0+10 (secure 0 0 0) 0x7fca0c00acc0 con 0x7fca100a0420 2021-02-10T11:31:55.764 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:31:55.765 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:55.778 7fca174bc700 1 -- 172.21.2.112:0/267737353 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc9f00186c0 msgr2=0x7fc9f001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:55.765 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:55.778 7fca174bc700 1 --2- 172.21.2.112:0/267737353 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc9f00186c0 0x7fc9f001abc0 crc :-1 s=READY pgs=531 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:55.765 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:55.778 7fca174bc700 1 -- 172.21.2.112:0/267737353 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fca100a0420 msgr2=0x7fca100a0820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:55.765 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:55.778 7fca174bc700 1 --2- 172.21.2.112:0/267737353 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fca100a0420 0x7fca100a0820 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 rx=0x7fca0c001c60 tx=0x7fca0c00a8d0).stop 2021-02-10T11:31:55.766 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:55.778 7fca174bc700 1 -- 172.21.2.112:0/267737353 shutdown_connections 2021-02-10T11:31:55.766 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:55.778 7fca174bc700 1 --2- 172.21.2.112:0/267737353 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc9f00186c0 0x7fc9f001abc0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:55.766 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:55.778 7fca174bc700 1 --2- 172.21.2.112:0/267737353 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fca1018a720 0x7fca100b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:55.766 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:55.778 7fca174bc700 1 --2- 172.21.2.112:0/267737353 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fca10189d00 0x7fca1018a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:55.766 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:55.778 7fca174bc700 1 --2- 172.21.2.112:0/267737353 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fca100a0420 0x7fca100a0820 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:55.766 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:55.778 7fca174bc700 1 -- 172.21.2.112:0/267737353 >> 172.21.2.112:0/267737353 conn(0x7fca10094e10 msgr2=0x7fca100a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:55.766 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:55.778 7fca174bc700 1 -- 172.21.2.112:0/267737353 shutdown_connections 2021-02-10T11:31:55.766 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:55.778 7fca174bc700 1 -- 172.21.2.112:0/267737353 wait complete. 2021-02-10T11:31:55.774 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:31:56.204 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.200 7fda41b30700 1 Processor -- start 2021-02-10T11:31:56.204 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.200 7fda41b30700 1 -- start start 2021-02-10T11:31:56.204 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.200 7fda41b30700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fda3c1b0470 0x7fda3c1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:56.204 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.200 7fda41b30700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fda3c0f20e0 0x7fda3c0f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:56.204 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.200 7fda41b30700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda3c0f0b50 con 0x7fda3c0ef610 2021-02-10T11:31:56.204 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.200 7fda41b30700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda3c0f0820 con 0x7fda3c1b0470 2021-02-10T11:31:56.204 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.200 7fda41b30700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda3c0f24e0 con 0x7fda3c0f20e0 2021-02-10T11:31:56.204 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.201 7fda3affd700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fda3c0ef610 0x7fda3c0efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:47962/0 (socket says 172.21.2.112:47962) 2021-02-10T11:31:56.205 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.201 7fda3affd700 1 -- 172.21.2.112:0/1542025188 learned_addr learned my addr 172.21.2.112:0/1542025188 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:31:56.205 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.201 7fda3b7fe700 1 --2- 172.21.2.112:0/1542025188 >> v2:172.21.2.120:3300/0 conn(0x7fda3c1b0470 0x7fda3c1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:56.205 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.201 7fda3bfff700 1 --2- 172.21.2.112:0/1542025188 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fda3c0f20e0 0x7fda3c0f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:56.205 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.201 7fda3a7fc700 1 -- 172.21.2.112:0/1542025188 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2368086394 0 0) 0x7fda300010b0 con 0x7fda3c0ef610 2021-02-10T11:31:56.205 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.201 7fda3a7fc700 1 -- 172.21.2.112:0/1542025188 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda24002fe0 con 0x7fda3c0ef610 2021-02-10T11:31:56.205 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.201 7fda3bfff700 1 -- 172.21.2.112:0/1542025188 >> v2:172.21.2.120:3300/0 conn(0x7fda3c1b0470 msgr2=0x7fda3c1b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:56.205 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.201 7fda3a7fc700 1 -- 172.21.2.112:0/1542025188 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1012248117 0 0) 0x7fda300010b0 con 0x7fda3c0ef610 2021-02-10T11:31:56.205 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.201 7fda3bfff700 1 --2- 172.21.2.112:0/1542025188 >> v2:172.21.2.120:3300/0 conn(0x7fda3c1b0470 0x7fda3c1b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:56.205 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.201 7fda3bfff700 1 -- 172.21.2.112:0/1542025188 >> v1:172.21.2.112:6789/0 conn(0x7fda3c0ef610 legacy=0x7fda3c0efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:56.205 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.201 7fda3bfff700 1 -- 172.21.2.112:0/1542025188 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda3c0e52e0 con 0x7fda3c0f20e0 2021-02-10T11:31:56.205 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.201 7fda3bfff700 1 --2- 172.21.2.112:0/1542025188 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fda3c0f20e0 0x7fda3c0f0200 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7fda34006e50 tx=0x7fda34009e70).ready entity=mon.2 client_cookie=137035ff0a116e4d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:56.206 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.202 7fda3a7fc700 1 -- 172.21.2.112:0/1542025188 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fda34006a20 con 0x7fda3c0f20e0 2021-02-10T11:31:56.206 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.202 7fda3a7fc700 1 -- 172.21.2.112:0/1542025188 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda34006790 con 0x7fda3c0f20e0 2021-02-10T11:31:56.206 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.202 7fda3a7fc700 1 -- 172.21.2.112:0/1542025188 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fda3400b700 con 0x7fda3c0f20e0 2021-02-10T11:31:56.206 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.202 7fda41b30700 1 -- 172.21.2.112:0/1542025188 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fda3c0f20e0 msgr2=0x7fda3c0f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:56.206 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.202 7fda41b30700 1 --2- 172.21.2.112:0/1542025188 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fda3c0f20e0 0x7fda3c0f0200 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7fda34006e50 tx=0x7fda34009e70).stop 2021-02-10T11:31:56.206 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.202 7fda41b30700 1 -- 172.21.2.112:0/1542025188 shutdown_connections 2021-02-10T11:31:56.206 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.202 7fda41b30700 1 --2- 172.21.2.112:0/1542025188 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fda3c0f20e0 0x7fda3c0f0200 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:56.206 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.202 7fda41b30700 1 --2- 172.21.2.112:0/1542025188 >> v2:172.21.2.120:3300/0 conn(0x7fda3c1b0470 0x7fda3c1b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:56.206 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.202 7fda41b30700 1 -- 172.21.2.112:0/1542025188 >> 172.21.2.112:0/1542025188 conn(0x7fda3c0ce3e0 msgr2=0x7fda3c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:56.206 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.202 7fda41b30700 1 -- 172.21.2.112:0/1542025188 shutdown_connections 2021-02-10T11:31:56.206 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.202 7fda41b30700 1 -- 172.21.2.112:0/1542025188 wait complete. 2021-02-10T11:31:56.210 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.217 7fda41b30700 1 Processor -- start 2021-02-10T11:31:56.210 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.217 7fda41b30700 1 -- start start 2021-02-10T11:31:56.210 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.217 7fda41b30700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fda3c0a0420 0x7fda3c0a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:56.210 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.217 7fda41b30700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fda3c189d00 0x7fda3c18a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:56.210 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.217 7fda41b30700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fda3c18a720 0x7fda3c0b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:56.210 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.217 7fda41b30700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda3c189800 con 0x7fda3c189d00 2021-02-10T11:31:56.210 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.217 7fda41b30700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda3c18ab20 con 0x7fda3c0a0420 2021-02-10T11:31:56.210 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.217 7fda41b30700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda3c0a1140 con 0x7fda3c18a720 2021-02-10T11:31:56.210 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.218 7fda3affd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fda3c189d00 0x7fda3c18a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:56.211 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.218 7fda3affd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fda3c189d00 0x7fda3c18a100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:48506/0 (socket says 172.21.2.112:48506) 2021-02-10T11:31:56.211 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.218 7fda3affd700 1 -- 172.21.2.112:0/3222054494 learned_addr learned my addr 172.21.2.112:0/3222054494 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:31:56.211 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.218 7fda3affd700 1 -- 172.21.2.112:0/3222054494 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fda3c18a720 msgr2=0x7fda3c0b7be0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:31:56.211 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.218 7fda3affd700 1 --2- 172.21.2.112:0/3222054494 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fda3c18a720 0x7fda3c0b7be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:56.211 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.218 7fda3affd700 1 -- 172.21.2.112:0/3222054494 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fda3c0a0420 msgr2=0x7fda3c0a0820 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:31:56.211 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.218 7fda3affd700 1 --2- 172.21.2.112:0/3222054494 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fda3c0a0420 0x7fda3c0a0820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:56.211 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.218 7fda3affd700 1 -- 172.21.2.112:0/3222054494 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda3c0b8200 con 0x7fda3c189d00 2021-02-10T11:31:56.211 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.218 7fda3affd700 1 --2- 172.21.2.112:0/3222054494 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fda3c189d00 0x7fda3c18a100 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 rx=0x7fda30001c60 tx=0x7fda3000a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:56.211 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.218 7fda23fff700 1 -- 172.21.2.112:0/3222054494 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fda3000acc0 con 0x7fda3c189d00 2021-02-10T11:31:56.211 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.218 7fda23fff700 1 -- 172.21.2.112:0/3222054494 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda30004140 con 0x7fda3c189d00 2021-02-10T11:31:56.211 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.218 7fda41b30700 1 -- 172.21.2.112:0/3222054494 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda3c0b8200 con 0x7fda3c189d00 2021-02-10T11:31:56.211 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.218 7fda41b30700 1 -- 172.21.2.112:0/3222054494 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda3c0b8200 con 0x7fda3c189d00 2021-02-10T11:31:56.211 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.218 7fda23fff700 1 -- 172.21.2.112:0/3222054494 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fda30004620 con 0x7fda3c189d00 2021-02-10T11:31:56.211 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.219 7fda23fff700 1 -- 172.21.2.112:0/3222054494 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fda30004320 con 0x7fda3c189d00 2021-02-10T11:31:56.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.219 7fda23fff700 1 --2- 172.21.2.112:0/3222054494 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fda1c018730 0x7fda1c01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:56.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.219 7fda23fff700 1 -- 172.21.2.112:0/3222054494 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fda30032780 con 0x7fda3c189d00 2021-02-10T11:31:56.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.219 7fda3b7fe700 1 --2- 172.21.2.112:0/3222054494 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fda1c018730 0x7fda1c01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:56.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.220 7fda3b7fe700 1 --2- 172.21.2.112:0/3222054494 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fda1c018730 0x7fda1c01ac10 crc :-1 s=READY pgs=532 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:56.238 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.250 7fda41b30700 1 -- 172.21.2.112:0/3222054494 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda3c18ab20 con 0x7fda3c189d00 2021-02-10T11:31:56.240 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.253 7fda23fff700 1 -- 172.21.2.112:0/3222054494 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fda30000e20 con 0x7fda3c189d00 2021-02-10T11:31:56.369 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.382 7fda41b30700 1 -- 172.21.2.112:0/3222054494 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7fda3c18ab20 con 0x7fda3c189d00 2021-02-10T11:31:56.370 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.382 7fda23fff700 1 -- 172.21.2.112:0/3222054494 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v38) v1 ==== 104+0+10 (secure 0 0 0) 0x7fda3000acc0 con 0x7fda3c189d00 2021-02-10T11:31:56.370 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:31:56.372 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.384 7fda41b30700 1 -- 172.21.2.112:0/3222054494 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fda1c018730 msgr2=0x7fda1c01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:56.372 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.384 7fda41b30700 1 --2- 172.21.2.112:0/3222054494 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fda1c018730 0x7fda1c01ac10 crc :-1 s=READY pgs=532 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:56.372 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.384 7fda41b30700 1 -- 172.21.2.112:0/3222054494 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fda3c189d00 msgr2=0x7fda3c18a100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:56.372 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.384 7fda41b30700 1 --2- 172.21.2.112:0/3222054494 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fda3c189d00 0x7fda3c18a100 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 rx=0x7fda30001c60 tx=0x7fda3000a8d0).stop 2021-02-10T11:31:56.372 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.384 7fda41b30700 1 -- 172.21.2.112:0/3222054494 shutdown_connections 2021-02-10T11:31:56.372 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.384 7fda41b30700 1 --2- 172.21.2.112:0/3222054494 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fda1c018730 0x7fda1c01ac10 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:56.372 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.384 7fda41b30700 1 --2- 172.21.2.112:0/3222054494 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fda3c18a720 0x7fda3c0b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:56.372 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.384 7fda41b30700 1 --2- 172.21.2.112:0/3222054494 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fda3c189d00 0x7fda3c18a100 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:56.373 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.384 7fda41b30700 1 --2- 172.21.2.112:0/3222054494 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fda3c0a0420 0x7fda3c0a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:56.373 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.384 7fda41b30700 1 -- 172.21.2.112:0/3222054494 >> 172.21.2.112:0/3222054494 conn(0x7fda3c094e10 msgr2=0x7fda3c0a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:56.373 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.385 7fda41b30700 1 -- 172.21.2.112:0/3222054494 shutdown_connections 2021-02-10T11:31:56.373 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.385 7fda41b30700 1 -- 172.21.2.112:0/3222054494 wait complete. 2021-02-10T11:31:56.382 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:31:56.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.808 7f9a09595700 1 Processor -- start 2021-02-10T11:31:56.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.809 7f9a09595700 1 -- start start 2021-02-10T11:31:56.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.809 7f9a09595700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9a041b0470 0x7f9a041b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:56.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.809 7f9a09595700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f9a040f20e0 0x7f9a040f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:56.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.809 7f9a09595700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a040f0b50 con 0x7f9a040ef610 2021-02-10T11:31:56.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.809 7f9a09595700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a040f0820 con 0x7f9a040f20e0 2021-02-10T11:31:56.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.809 7f9a09595700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a040f24e0 con 0x7f9a041b0470 2021-02-10T11:31:56.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.809 7f9a03ac0700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f9a040f20e0 0x7f9a040f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:56.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.809 7f9a03ac0700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f9a040f20e0 0x7f9a040f0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:48514/0 (socket says 172.21.2.112:48514) 2021-02-10T11:31:56.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.809 7f9a03ac0700 1 -- 172.21.2.112:0/899095452 learned_addr learned my addr 172.21.2.112:0/899095452 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:31:56.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.810 7f9a03ac0700 1 -- 172.21.2.112:0/899095452 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9a041b0470 msgr2=0x7f9a041b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:56.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.810 7f9a03ac0700 1 --2- 172.21.2.112:0/899095452 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9a041b0470 0x7f9a041b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:56.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.810 7f9a03ac0700 1 -- 172.21.2.112:0/899095452 >> v1:172.21.2.112:6789/0 conn(0x7f9a040ef610 legacy=0x7f9a040efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:56.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.810 7f9a022bd700 1 -- 172.21.2.112:0/899095452 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3985663073 0 0) 0x7f99f8000ff0 con 0x7f9a040ef610 2021-02-10T11:31:56.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.810 7f9a03ac0700 1 -- 172.21.2.112:0/899095452 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a040e52e0 con 0x7f9a040f20e0 2021-02-10T11:31:56.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.810 7f9a03ac0700 1 --2- 172.21.2.112:0/899095452 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9a040f20e0 0x7f9a040f0200 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 rx=0x7f99fc006f00 tx=0x7f99fc009f10).ready entity=mon.0 client_cookie=55023184b1fabd3e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:56.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.810 7f9a022bd700 1 -- 172.21.2.112:0/899095452 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f99fc00b3b0 con 0x7f9a040f20e0 2021-02-10T11:31:56.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.810 7f9a022bd700 1 -- 172.21.2.112:0/899095452 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99fc00b630 con 0x7f9a040f20e0 2021-02-10T11:31:56.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.810 7f9a022bd700 1 -- 172.21.2.112:0/899095452 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f99fc00bb50 con 0x7f9a040f20e0 2021-02-10T11:31:56.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.810 7f9a09595700 1 -- 172.21.2.112:0/899095452 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9a040f20e0 msgr2=0x7f9a040f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:56.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.810 7f9a09595700 1 --2- 172.21.2.112:0/899095452 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9a040f20e0 0x7f9a040f0200 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 rx=0x7f99fc006f00 tx=0x7f99fc009f10).stop 2021-02-10T11:31:56.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.811 7f9a09595700 1 -- 172.21.2.112:0/899095452 shutdown_connections 2021-02-10T11:31:56.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.811 7f9a09595700 1 --2- 172.21.2.112:0/899095452 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9a040f20e0 0x7f9a040f0200 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:56.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.811 7f9a09595700 1 --2- 172.21.2.112:0/899095452 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9a041b0470 0x7f9a041b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:56.815 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.811 7f9a09595700 1 -- 172.21.2.112:0/899095452 >> 172.21.2.112:0/899095452 conn(0x7f9a040ce3e0 msgr2=0x7f9a040e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:56.815 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.811 7f9a09595700 1 -- 172.21.2.112:0/899095452 shutdown_connections 2021-02-10T11:31:56.815 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.811 7f9a09595700 1 -- 172.21.2.112:0/899095452 wait complete. 2021-02-10T11:31:56.815 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.825 7f9a09595700 1 Processor -- start 2021-02-10T11:31:56.816 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.825 7f9a09595700 1 -- start start 2021-02-10T11:31:56.816 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.826 7f9a09595700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9a04190920 0x7f9a0408f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:56.816 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.826 7f9a09595700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9a04190d20 0x7f9a0408fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:56.816 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.826 7f9a09595700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9a04193270 0x7f9a04193670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:56.816 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.826 7f9a09595700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a04090470 con 0x7f9a04190920 2021-02-10T11:31:56.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.826 7f9a09595700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a04193c90 con 0x7f9a04190d20 2021-02-10T11:31:56.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.826 7f9a09595700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a04193e70 con 0x7f9a04193270 2021-02-10T11:31:56.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.826 7f9a032bf700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9a04190920 0x7f9a0408f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:56.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.826 7f9a032bf700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9a04190920 0x7f9a0408f600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:48518/0 (socket says 172.21.2.112:48518) 2021-02-10T11:31:56.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.826 7f9a032bf700 1 -- 172.21.2.112:0/4063256249 learned_addr learned my addr 172.21.2.112:0/4063256249 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:31:56.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.826 7f9a032bf700 1 -- 172.21.2.112:0/4063256249 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9a04193270 msgr2=0x7f9a04193670 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:31:56.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.826 7f9a032bf700 1 --2- 172.21.2.112:0/4063256249 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9a04193270 0x7f9a04193670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:56.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.826 7f9a032bf700 1 -- 172.21.2.112:0/4063256249 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9a04190d20 msgr2=0x7f9a0408fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:56.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.826 7f9a032bf700 1 --2- 172.21.2.112:0/4063256249 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9a04190d20 0x7f9a0408fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:56.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.826 7f9a032bf700 1 -- 172.21.2.112:0/4063256249 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a04194090 con 0x7f9a04190920 2021-02-10T11:31:56.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.827 7f9a032bf700 1 --2- 172.21.2.112:0/4063256249 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9a04190920 0x7f9a0408f600 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 rx=0x7f99f400b530 tx=0x7f99f400b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:56.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.827 7f99ebfff700 1 -- 172.21.2.112:0/4063256249 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f99f400bcc0 con 0x7f9a04190920 2021-02-10T11:31:56.818 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.827 7f99ebfff700 1 -- 172.21.2.112:0/4063256249 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99f4000c90 con 0x7f9a04190920 2021-02-10T11:31:56.818 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.827 7f9a09595700 1 -- 172.21.2.112:0/4063256249 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a04194090 con 0x7f9a04190920 2021-02-10T11:31:56.818 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.827 7f99ebfff700 1 -- 172.21.2.112:0/4063256249 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f99f40022a0 con 0x7f9a04190920 2021-02-10T11:31:56.818 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.827 7f9a09595700 1 -- 172.21.2.112:0/4063256249 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a041920b0 con 0x7f9a04190920 2021-02-10T11:31:56.818 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.828 7f99ebfff700 1 -- 172.21.2.112:0/4063256249 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f99f4005310 con 0x7f9a04190920 2021-02-10T11:31:56.818 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.828 7f99ebfff700 1 --2- 172.21.2.112:0/4063256249 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f99e40186c0 0x7f99e401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:56.818 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.828 7f99ebfff700 1 -- 172.21.2.112:0/4063256249 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f99f4032410 con 0x7f9a04190920 2021-02-10T11:31:56.818 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.828 7f9a02abe700 1 --2- 172.21.2.112:0/4063256249 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f99e40186c0 0x7f99e401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:56.818 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.828 7f9a02abe700 1 --2- 172.21.2.112:0/4063256249 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f99e40186c0 0x7f99e401abc0 crc :-1 s=READY pgs=533 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:56.846 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.859 7f9a09595700 1 -- 172.21.2.112:0/4063256249 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a04105af0 con 0x7f9a04190920 2021-02-10T11:31:56.848 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.861 7f99ebfff700 1 -- 172.21.2.112:0/4063256249 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f99f4000960 con 0x7f9a04190920 2021-02-10T11:31:56.978 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.990 7f9a09595700 1 -- 172.21.2.112:0/4063256249 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f9a040f0b50 con 0x7f9a04190920 2021-02-10T11:31:56.978 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.991 7f99ebfff700 1 -- 172.21.2.112:0/4063256249 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v38) v1 ==== 100+0+10 (secure 0 0 0) 0x7f99f4005310 con 0x7f9a04190920 2021-02-10T11:31:56.978 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:31:56.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.992 7f9a09595700 1 -- 172.21.2.112:0/4063256249 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f99e40186c0 msgr2=0x7f99e401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:56.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.992 7f9a09595700 1 --2- 172.21.2.112:0/4063256249 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f99e40186c0 0x7f99e401abc0 crc :-1 s=READY pgs=533 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:56.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.992 7f9a09595700 1 -- 172.21.2.112:0/4063256249 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9a04190920 msgr2=0x7f9a0408f600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:56.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.992 7f9a09595700 1 --2- 172.21.2.112:0/4063256249 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9a04190920 0x7f9a0408f600 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 rx=0x7f99f400b530 tx=0x7f99f400b8d0).stop 2021-02-10T11:31:56.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.993 7f9a09595700 1 -- 172.21.2.112:0/4063256249 shutdown_connections 2021-02-10T11:31:56.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.993 7f9a09595700 1 --2- 172.21.2.112:0/4063256249 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f99e40186c0 0x7f99e401abc0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:56.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.993 7f9a09595700 1 --2- 172.21.2.112:0/4063256249 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9a04190d20 0x7f9a0408fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:56.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.993 7f9a09595700 1 --2- 172.21.2.112:0/4063256249 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9a04193270 0x7f9a04193670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:56.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.993 7f9a09595700 1 --2- 172.21.2.112:0/4063256249 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9a04190920 0x7f9a0408f600 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:56.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.993 7f9a09595700 1 -- 172.21.2.112:0/4063256249 >> 172.21.2.112:0/4063256249 conn(0x7f9a040bf4e0 msgr2=0x7f9a040be3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:56.981 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.993 7f9a09595700 1 -- 172.21.2.112:0/4063256249 shutdown_connections 2021-02-10T11:31:56.981 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:56.993 7f9a09595700 1 -- 172.21.2.112:0/4063256249 wait complete. 2021-02-10T11:31:56.988 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:31:56.989 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:31:57.413 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:57.410 7f1ccd366700 1 Processor -- start 2021-02-10T11:31:57.413 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:57.410 7f1ccd366700 1 -- start start 2021-02-10T11:31:57.414 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:57.410 7f1ccd366700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1cc80d17f0 0x7f1cc81b31e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:57.414 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:57.410 7f1ccd366700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f1cc80d1bf0 0x7f1cc81b3800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:57.414 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:57.410 7f1ccd366700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f1cc81b3e20 0x7f1cc80ceef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:57.414 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:57.410 7f1ccd366700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cc80e5eb0 con 0x7f1cc80d1bf0 2021-02-10T11:31:57.414 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:57.410 7f1ccd366700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cc80f0b20 con 0x7f1cc81b3e20 2021-02-10T11:31:57.414 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:57.410 7f1ccd366700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cc80cf510 con 0x7f1cc80d17f0 2021-02-10T11:31:57.414 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:57.410 7f1cc687c700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f1cc80d1bf0 0x7f1cc81b3800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:57.414 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:57.411 7f1cc687c700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f1cc80d1bf0 0x7f1cc81b3800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:48526/0 (socket says 172.21.2.112:48526) 2021-02-10T11:31:57.414 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:57.411 7f1cc687c700 1 -- 172.21.2.112:0/217193735 learned_addr learned my addr 172.21.2.112:0/217193735 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:31:57.414 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:57.411 7f1cc707d700 1 --2- 172.21.2.112:0/217193735 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1cc80d17f0 0x7f1cc81b31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:57.415 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:57.411 7f1cc787e700 1 --2- 172.21.2.112:0/217193735 >> v2:172.21.2.120:3300/0 conn(0x7f1cc81b3e20 0x7f1cc80ceef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:57.415 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:57.411 7f1cc687c700 1 -- 172.21.2.112:0/217193735 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1cc80d17f0 msgr2=0x7f1cc81b31e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:57.415 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:57.411 7f1cc687c700 1 --2- 172.21.2.112:0/217193735 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1cc80d17f0 0x7f1cc81b31e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:57.415 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:57.411 7f1cc687c700 1 -- 172.21.2.112:0/217193735 >> v2:172.21.2.120:3300/0 conn(0x7f1cc81b3e20 msgr2=0x7f1cc80ceef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:57.415 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:57.411 7f1cc687c700 1 --2- 172.21.2.112:0/217193735 >> v2:172.21.2.120:3300/0 conn(0x7f1cc81b3e20 0x7f1cc80ceef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:57.415 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:57.411 7f1cc687c700 1 -- 172.21.2.112:0/217193735 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cc80cf730 con 0x7f1cc80d1bf0 2021-02-10T11:31:57.415 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:57.411 7f1cc687c700 1 --2- 172.21.2.112:0/217193735 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1cc80d1bf0 0x7f1cc81b3800 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 rx=0x7f1cbc006700 tx=0x7f1cbc009b00).ready entity=mon.0 client_cookie=f5778a31a04914bb server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:57.415 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:57.411 7f1cc607b700 1 -- 172.21.2.112:0/217193735 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1cbc000c90 con 0x7f1cc80d1bf0 2021-02-10T11:31:57.415 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:57.411 7f1cc607b700 1 -- 172.21.2.112:0/217193735 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cbc00c0c0 con 0x7f1cc80d1bf0 2021-02-10T11:31:57.415 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:57.411 7f1cc607b700 1 -- 172.21.2.112:0/217193735 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1cbc00c5c0 con 0x7f1cc80d1bf0 2021-02-10T11:31:57.415 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:57.411 7f1ccd366700 1 -- 172.21.2.112:0/217193735 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1cc80d1bf0 msgr2=0x7f1cc81b3800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:57.415 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:57.411 7f1ccd366700 1 --2- 172.21.2.112:0/217193735 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1cc80d1bf0 0x7f1cc81b3800 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 rx=0x7f1cbc006700 tx=0x7f1cbc009b00).stop 2021-02-10T11:31:57.416 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:57.412 7f1ccd366700 1 -- 172.21.2.112:0/217193735 shutdown_connections 2021-02-10T11:31:57.416 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:57.412 7f1ccd366700 1 --2- 172.21.2.112:0/217193735 >> v2:172.21.2.120:3300/0 conn(0x7f1cc81b3e20 0x7f1cc80ceef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:57.416 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:57.412 7f1ccd366700 1 --2- 172.21.2.112:0/217193735 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1cc80d1bf0 0x7f1cc81b3800 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:57.416 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:57.412 7f1ccd366700 1 --2- 172.21.2.112:0/217193735 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1cc80d17f0 0x7f1cc81b31e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:57.416 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:57.412 7f1ccd366700 1 -- 172.21.2.112:0/217193735 >> 172.21.2.112:0/217193735 conn(0x7f1cc80fde40 msgr2=0x7f1cc80ce390 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:57.416 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:57.412 7f1ccd366700 1 -- 172.21.2.112:0/217193735 shutdown_connections 2021-02-10T11:31:57.416 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:57.412 7f1ccd366700 1 -- 172.21.2.112:0/217193735 wait complete. 2021-02-10T11:31:57.419 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:57.426 7f1ccd366700 1 Processor -- start 2021-02-10T11:31:57.419 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:57.427 7f1ccd366700 1 -- start start 2021-02-10T11:31:57.419 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:57.427 7f1ccd366700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1cc8107ee0 0x7f1cc808e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:57.419 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:57.427 7f1ccd366700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1cc808ee90 0x7f1cc808f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:57.419 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:57.427 7f1ccd366700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1cc80b6d20 0x7f1cc80b7120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:57.419 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:57.427 7f1ccd366700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cc80b67a0 con 0x7f1cc8107ee0 2021-02-10T11:31:57.419 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:57.427 7f1ccd366700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cc81082e0 con 0x7f1cc80b6d20 2021-02-10T11:31:57.419 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:57.427 7f1ccd366700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cc808f8b0 con 0x7f1cc808ee90 2021-02-10T11:31:57.419 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:57.427 7f1cc707d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1cc8107ee0 0x7f1cc808e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:57.419 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:57.427 7f1cc707d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1cc8107ee0 0x7f1cc808e870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:48530/0 (socket says 172.21.2.112:48530) 2021-02-10T11:31:57.419 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:57.427 7f1cc707d700 1 -- 172.21.2.112:0/1196931028 learned_addr learned my addr 172.21.2.112:0/1196931028 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:31:57.419 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:57.427 7f1cc687c700 1 --2- 172.21.2.112:0/1196931028 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1cc808ee90 0x7f1cc808f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:57.420 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:57.427 7f1cc787e700 1 --2- 172.21.2.112:0/1196931028 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1cc80b6d20 0x7f1cc80b7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:57.420 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:57.428 7f1cc707d700 1 -- 172.21.2.112:0/1196931028 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1cc808ee90 msgr2=0x7f1cc808f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:57.420 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:57.428 7f1cc707d700 1 --2- 172.21.2.112:0/1196931028 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1cc808ee90 0x7f1cc808f290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:57.420 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:57.428 7f1cc707d700 1 -- 172.21.2.112:0/1196931028 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1cc80b6d20 msgr2=0x7f1cc80b7120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:57.420 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:57.428 7f1cc707d700 1 --2- 172.21.2.112:0/1196931028 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1cc80b6d20 0x7f1cc80b7120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:57.420 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:57.428 7f1cc707d700 1 -- 172.21.2.112:0/1196931028 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cc80b7740 con 0x7f1cc8107ee0 2021-02-10T11:31:57.420 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:57.428 7f1cc707d700 1 --2- 172.21.2.112:0/1196931028 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1cc8107ee0 0x7f1cc808e870 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 rx=0x7f1cb8006f00 tx=0x7f1cb800b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:57.420 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:57.428 7f1caffff700 1 -- 172.21.2.112:0/1196931028 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1cb800bd30 con 0x7f1cc8107ee0 2021-02-10T11:31:57.420 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:57.428 7f1caffff700 1 -- 172.21.2.112:0/1196931028 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cb8002030 con 0x7f1cc8107ee0 2021-02-10T11:31:57.420 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:57.428 7f1caffff700 1 -- 172.21.2.112:0/1196931028 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1cb8002480 con 0x7f1cc8107ee0 2021-02-10T11:31:57.421 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:57.428 7f1ccd366700 1 -- 172.21.2.112:0/1196931028 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cc80b7740 con 0x7f1cc8107ee0 2021-02-10T11:31:57.421 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:57.428 7f1ccd366700 1 -- 172.21.2.112:0/1196931028 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cc80b7cd0 con 0x7f1cc8107ee0 2021-02-10T11:31:57.421 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:57.429 7f1caffff700 1 -- 172.21.2.112:0/1196931028 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1cb8000c90 con 0x7f1cc8107ee0 2021-02-10T11:31:57.421 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:57.429 7f1caffff700 1 --2- 172.21.2.112:0/1196931028 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1ca80186c0 0x7f1ca801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:57.421 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:57.429 7f1caffff700 1 -- 172.21.2.112:0/1196931028 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f1cb80327e0 con 0x7f1cc8107ee0 2021-02-10T11:31:57.421 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:57.429 7f1cc687c700 1 --2- 172.21.2.112:0/1196931028 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1ca80186c0 0x7f1ca801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:57.421 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:57.430 7f1cc687c700 1 --2- 172.21.2.112:0/1196931028 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1ca80186c0 0x7f1ca801abc0 crc :-1 s=READY pgs=534 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:57.447 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:57.460 7f1ccd366700 1 -- 172.21.2.112:0/1196931028 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cc81036a0 con 0x7f1cc8107ee0 2021-02-10T11:31:57.450 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:57.463 7f1caffff700 1 -- 172.21.2.112:0/1196931028 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1cb80009b0 con 0x7f1cc8107ee0 2021-02-10T11:31:57.579 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:57.592 7f1ccd366700 1 -- 172.21.2.112:0/1196931028 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1cc80f0b20 con 0x7f1cc8107ee0 2021-02-10T11:31:57.580 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:57.593 7f1caffff700 1 -- 172.21.2.112:0/1196931028 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+2099 (secure 0 0 0) 0x7f1cb800bd30 con 0x7f1cc8107ee0 2021-02-10T11:31:57.580 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 22 2021-02-10T11:31:57.580 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:31:57.581 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":22,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7155,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/1957863967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":1957863967},{"type":"v1","addr":"172.21.2.112:6833","nonce":1957863967}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":18}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:31:40.190262","modified":"2021-02-10 11:31:42.195824","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5534},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5534":{"gid":5534,"name":"b","rank":0,"incarnation":19,"state":"up:active","state_seq":3,"addr":"172.21.2.120:6835/1513827094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":1513827094},{"type":"v1","addr":"172.21.2.120:6835","nonce":1513827094}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2021-02-10T11:31:57.581 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:57.594 7f1ccd366700 1 -- 172.21.2.112:0/1196931028 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1ca80186c0 msgr2=0x7f1ca801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:57.582 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:57.594 7f1ccd366700 1 --2- 172.21.2.112:0/1196931028 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1ca80186c0 0x7f1ca801abc0 crc :-1 s=READY pgs=534 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:57.582 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:57.594 7f1ccd366700 1 -- 172.21.2.112:0/1196931028 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1cc8107ee0 msgr2=0x7f1cc808e870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:57.582 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:57.594 7f1ccd366700 1 --2- 172.21.2.112:0/1196931028 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1cc8107ee0 0x7f1cc808e870 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 rx=0x7f1cb8006f00 tx=0x7f1cb800b940).stop 2021-02-10T11:31:57.582 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:57.595 7f1ccd366700 1 -- 172.21.2.112:0/1196931028 shutdown_connections 2021-02-10T11:31:57.582 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:57.595 7f1ccd366700 1 --2- 172.21.2.112:0/1196931028 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1ca80186c0 0x7f1ca801abc0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:57.582 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:57.595 7f1ccd366700 1 --2- 172.21.2.112:0/1196931028 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1cc80b6d20 0x7f1cc80b7120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:57.582 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:57.595 7f1ccd366700 1 --2- 172.21.2.112:0/1196931028 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1cc808ee90 0x7f1cc808f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:57.582 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:57.595 7f1ccd366700 1 --2- 172.21.2.112:0/1196931028 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1cc8107ee0 0x7f1cc808e870 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:57.582 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:57.595 7f1ccd366700 1 -- 172.21.2.112:0/1196931028 >> 172.21.2.112:0/1196931028 conn(0x7f1cc809ca80 msgr2=0x7f1cc80c3c60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:57.583 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:57.595 7f1ccd366700 1 -- 172.21.2.112:0/1196931028 shutdown_connections 2021-02-10T11:31:57.583 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:31:57.595 7f1ccd366700 1 -- 172.21.2.112:0/1196931028 wait complete. 2021-02-10T11:31:57.792 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:31:40.190262', 'modified': '2021-02-10 11:31:42.195824', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5534}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5534': {'gid': 5534, 'name': 'b', 'rank': 0, 'incarnation': 19, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.120:6835/1513827094', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.120:6834', 'nonce': 1513827094}, {'type': 'v1', 'addr': '172.21.2.120:6835', 'nonce': 1513827094}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [7], 'metadata_pool': 6, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:31:57.792 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:31:57.792 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:31:58.190 INFO:teuthology.orchestra.run.gibba020.stdout:{ 2021-02-10T11:31:58.190 INFO:teuthology.orchestra.run.gibba020.stdout: "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", 2021-02-10T11:31:58.190 INFO:teuthology.orchestra.run.gibba020.stdout: "whoami": 0, 2021-02-10T11:31:58.190 INFO:teuthology.orchestra.run.gibba020.stdout: "id": 5534, 2021-02-10T11:31:58.190 INFO:teuthology.orchestra.run.gibba020.stdout: "want_state": "up:active", 2021-02-10T11:31:58.190 INFO:teuthology.orchestra.run.gibba020.stdout: "state": "up:active", 2021-02-10T11:31:58.190 INFO:teuthology.orchestra.run.gibba020.stdout: "rank_uptime": 18.011244017999999, 2021-02-10T11:31:58.190 INFO:teuthology.orchestra.run.gibba020.stdout: "mdsmap_epoch": 22, 2021-02-10T11:31:58.190 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch": 38, 2021-02-10T11:31:58.190 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch_barrier": 38, 2021-02-10T11:31:58.190 INFO:teuthology.orchestra.run.gibba020.stdout: "uptime": 23.643707403000001 2021-02-10T11:31:58.191 INFO:teuthology.orchestra.run.gibba020.stdout:} 2021-02-10T11:31:58.194 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", "whoami": 0, "id": 5534, "want_state": "up:active", "state": "up:active", "rank_uptime": 18.011244017999999, "mdsmap_epoch": 22, "osdmap_epoch": 38, "osdmap_epoch_barrier": 38, "uptime": 23.643707403000001 } 2021-02-10T11:31:58.194 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:31:58.194 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2021-02-10T11:31:58.194 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.1 at ubuntu@gibba012.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2021-02-10T11:31:58.194 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:58.194 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:31:58.530 DEBUG:teuthology.orchestra.run.gibba012:> sudo modprobe fuse 2021-02-10T11:31:58.874 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:31:58.874 DEBUG:teuthology.orchestra.run.gibba012:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:31:59.217 INFO:teuthology.orchestra.run.gibba012.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:31:59.217 INFO:teuthology.orchestra.run.gibba012.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:31:59.218 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:31:59.218 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:59.218 DEBUG:teuthology.orchestra.run.gibba012:> ls /sys/fs/fuse/connections 2021-02-10T11:31:59.555 INFO:teuthology.orchestra.run.gibba012.stdout:42 2021-02-10T11:31:59.555 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [42] 2021-02-10T11:31:59.555 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.1 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:31:59.557 DEBUG:teuthology.orchestra.run.gibba012:> sudo modprobe fuse 2021-02-10T11:31:59.901 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:31:59.901 DEBUG:teuthology.orchestra.run.gibba012:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:31:59.967 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba012.stderr:ceph-fuse[121914]: starting ceph client 2021-02-10T11:31:59.967 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba012.stderr:2021-02-10 11:31:59.979 7fd2533f4f80 -1 init, newargv = 0x563995372120 newargc=9 2021-02-10T11:31:59.972 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba012.stderr:ceph-fuse[121914]: starting fuse 2021-02-10T11:32:00.242 INFO:teuthology.orchestra.run.gibba012.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:32:00.243 INFO:teuthology.orchestra.run.gibba012.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:32:00.243 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:32:00.243 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:00.244 DEBUG:teuthology.orchestra.run.gibba012:> ls /sys/fs/fuse/connections 2021-02-10T11:32:00.577 INFO:teuthology.orchestra.run.gibba012.stdout:42 2021-02-10T11:32:00.577 INFO:teuthology.orchestra.run.gibba012.stdout:43 2021-02-10T11:32:00.578 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42, 43] 2021-02-10T11:32:00.578 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:00.578 DEBUG:teuthology.orchestra.run.gibba012:> sudo python3 -c ' 2021-02-10T11:32:00.578 DEBUG:teuthology.orchestra.run.gibba012:> import glob 2021-02-10T11:32:00.578 DEBUG:teuthology.orchestra.run.gibba012:> import re 2021-02-10T11:32:00.578 DEBUG:teuthology.orchestra.run.gibba012:> import os 2021-02-10T11:32:00.578 DEBUG:teuthology.orchestra.run.gibba012:> import subprocess 2021-02-10T11:32:00.578 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:32:00.579 DEBUG:teuthology.orchestra.run.gibba012:> def find_socket(client_name): 2021-02-10T11:32:00.579 DEBUG:teuthology.orchestra.run.gibba012:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2021-02-10T11:32:00.579 DEBUG:teuthology.orchestra.run.gibba012:> files = glob.glob(asok_path) 2021-02-10T11:32:00.579 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:32:00.579 DEBUG:teuthology.orchestra.run.gibba012:> # Given a non-glob path, it better be there 2021-02-10T11:32:00.579 DEBUG:teuthology.orchestra.run.gibba012:> if "*" not in asok_path: 2021-02-10T11:32:00.579 DEBUG:teuthology.orchestra.run.gibba012:> assert(len(files) == 1) 2021-02-10T11:32:00.579 DEBUG:teuthology.orchestra.run.gibba012:> return files[0] 2021-02-10T11:32:00.579 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:32:00.579 DEBUG:teuthology.orchestra.run.gibba012:> for f in files: 2021-02-10T11:32:00.579 DEBUG:teuthology.orchestra.run.gibba012:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:32:00.579 DEBUG:teuthology.orchestra.run.gibba012:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:32:00.579 DEBUG:teuthology.orchestra.run.gibba012:> return f 2021-02-10T11:32:00.579 DEBUG:teuthology.orchestra.run.gibba012:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:32:00.579 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:32:00.580 DEBUG:teuthology.orchestra.run.gibba012:> print(find_socket("client.1")) 2021-02-10T11:32:00.580 DEBUG:teuthology.orchestra.run.gibba012:> ' 2021-02-10T11:32:00.940 INFO:teuthology.orchestra.run.gibba012.stdout:/var/run/ceph/ceph-client.1.121914.asok 2021-02-10T11:32:00.943 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.121914.asok 2021-02-10T11:32:00.944 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:00.944 DEBUG:teuthology.orchestra.run.gibba012:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.121914.asok status 2021-02-10T11:32:01.331 INFO:teuthology.orchestra.run.gibba012.stdout:{ 2021-02-10T11:32:01.332 INFO:teuthology.orchestra.run.gibba012.stdout: "metadata": { 2021-02-10T11:32:01.332 INFO:teuthology.orchestra.run.gibba012.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:32:01.332 INFO:teuthology.orchestra.run.gibba012.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:32:01.332 INFO:teuthology.orchestra.run.gibba012.stdout: "entity_id": "1", 2021-02-10T11:32:01.332 INFO:teuthology.orchestra.run.gibba012.stdout: "hostname": "gibba012", 2021-02-10T11:32:01.332 INFO:teuthology.orchestra.run.gibba012.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2021-02-10T11:32:01.332 INFO:teuthology.orchestra.run.gibba012.stdout: "pid": "121914", 2021-02-10T11:32:01.332 INFO:teuthology.orchestra.run.gibba012.stdout: "root": "/" 2021-02-10T11:32:01.332 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:32:01.332 INFO:teuthology.orchestra.run.gibba012.stdout: "dentry_count": 0, 2021-02-10T11:32:01.332 INFO:teuthology.orchestra.run.gibba012.stdout: "dentry_pinned_count": 0, 2021-02-10T11:32:01.333 INFO:teuthology.orchestra.run.gibba012.stdout: "id": 7362, 2021-02-10T11:32:01.333 INFO:teuthology.orchestra.run.gibba012.stdout: "inst": { 2021-02-10T11:32:01.333 INFO:teuthology.orchestra.run.gibba012.stdout: "name": { 2021-02-10T11:32:01.333 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "client", 2021-02-10T11:32:01.333 INFO:teuthology.orchestra.run.gibba012.stdout: "num": 7362 2021-02-10T11:32:01.333 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:32:01.333 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": { 2021-02-10T11:32:01.333 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:32:01.333 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:0", 2021-02-10T11:32:01.333 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 1155012381 2021-02-10T11:32:01.333 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:32:01.333 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:32:01.333 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": { 2021-02-10T11:32:01.334 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:32:01.334 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:0", 2021-02-10T11:32:01.334 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 1155012381 2021-02-10T11:32:01.334 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:32:01.334 INFO:teuthology.orchestra.run.gibba012.stdout: "inst_str": "client.7362 172.21.2.112:0/1155012381", 2021-02-10T11:32:01.334 INFO:teuthology.orchestra.run.gibba012.stdout: "addr_str": "172.21.2.112:0/1155012381", 2021-02-10T11:32:01.334 INFO:teuthology.orchestra.run.gibba012.stdout: "inode_count": 1, 2021-02-10T11:32:01.334 INFO:teuthology.orchestra.run.gibba012.stdout: "mds_epoch": 22, 2021-02-10T11:32:01.334 INFO:teuthology.orchestra.run.gibba012.stdout: "osd_epoch": 38, 2021-02-10T11:32:01.334 INFO:teuthology.orchestra.run.gibba012.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:32:01.334 INFO:teuthology.orchestra.run.gibba012.stdout: "blacklisted": false 2021-02-10T11:32:01.334 INFO:teuthology.orchestra.run.gibba012.stdout:} 2021-02-10T11:32:01.336 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:01.336 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:32:01.336 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:32:01.671 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:32:01.671 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:32:01.672 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:01.672 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:32:02.018 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2021-02-10T11:32:02.448 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:02.443 7f3568e01700 1 Processor -- start 2021-02-10T11:32:02.448 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:02.443 7f3568e01700 1 -- start start 2021-02-10T11:32:02.448 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:02.443 7f3568e01700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f35640d1c70 0x7f35640cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:02.448 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:02.443 7f3568e01700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f35640cf580 0x7f35640cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:02.448 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:02.443 7f3568e01700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35640fe1c0 con 0x7f35640d1870 2021-02-10T11:32:02.448 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:02.443 7f3568e01700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35641b0cb0 con 0x7f35640cf580 2021-02-10T11:32:02.448 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:02.443 7f3568e01700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3564109a60 con 0x7f35640d1c70 2021-02-10T11:32:02.448 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:02.443 7f3562b13700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f35640d1870 0x7f35641b04c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:48012/0 (socket says 172.21.2.112:48012) 2021-02-10T11:32:02.449 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:02.444 7f3562b13700 1 -- 172.21.2.112:0/1125100034 learned_addr learned my addr 172.21.2.112:0/1125100034 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:32:02.449 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:02.444 7f3563314700 1 --2- 172.21.2.112:0/1125100034 >> v2:172.21.2.120:3300/0 conn(0x7f35640cf580 0x7f35640cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:02.449 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:02.444 7f3562312700 1 --2- 172.21.2.112:0/1125100034 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f35640d1c70 0x7f35640cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:02.449 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:02.444 7f3561b11700 1 -- 172.21.2.112:0/1125100034 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1704560503 0 0) 0x7f354c000ce0 con 0x7f35640d1870 2021-02-10T11:32:02.449 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:02.444 7f3561b11700 1 -- 172.21.2.112:0/1125100034 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3550002fe0 con 0x7f35640d1870 2021-02-10T11:32:02.449 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:02.444 7f3563314700 1 -- 172.21.2.112:0/1125100034 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f35640d1c70 msgr2=0x7f35640cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:02.449 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:02.444 7f3563314700 1 --2- 172.21.2.112:0/1125100034 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f35640d1c70 0x7f35640cef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:02.449 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:02.444 7f3563314700 1 -- 172.21.2.112:0/1125100034 >> v1:172.21.2.112:6789/0 conn(0x7f35640d1870 legacy=0x7f35641b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:02.449 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:02.444 7f3563314700 1 -- 172.21.2.112:0/1125100034 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3564088020 con 0x7f35640cf580 2021-02-10T11:32:02.449 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:02.444 7f3562312700 1 --2- 172.21.2.112:0/1125100034 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f35640d1c70 0x7f35640cef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:32:02.449 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:02.445 7f3563314700 1 --2- 172.21.2.112:0/1125100034 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f35640cf580 0x7f35640cf980 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 rx=0x7f355c006e50 tx=0x7f355c009e70).ready entity=mon.1 client_cookie=7186ee0bdcd2b02a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:02.449 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:02.445 7f3561b11700 1 -- 172.21.2.112:0/1125100034 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f355c00b510 con 0x7f35640cf580 2021-02-10T11:32:02.449 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:02.445 7f3561b11700 1 -- 172.21.2.112:0/1125100034 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f355c006790 con 0x7f35640cf580 2021-02-10T11:32:02.450 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:02.445 7f3568e01700 1 -- 172.21.2.112:0/1125100034 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f35640cf580 msgr2=0x7f35640cf980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:02.450 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:02.445 7f3568e01700 1 --2- 172.21.2.112:0/1125100034 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f35640cf580 0x7f35640cf980 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 rx=0x7f355c006e50 tx=0x7f355c009e70).stop 2021-02-10T11:32:02.450 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:02.445 7f3561b11700 1 -- 172.21.2.112:0/1125100034 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f355c00b940 con 0x7f35640cf580 2021-02-10T11:32:02.450 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:02.445 7f3568e01700 1 -- 172.21.2.112:0/1125100034 shutdown_connections 2021-02-10T11:32:02.450 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:02.445 7f3568e01700 1 --2- 172.21.2.112:0/1125100034 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f35640cf580 0x7f35640cf980 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:02.450 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:02.445 7f3568e01700 1 --2- 172.21.2.112:0/1125100034 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f35640d1c70 0x7f35640cef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:02.450 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:02.445 7f3568e01700 1 -- 172.21.2.112:0/1125100034 >> 172.21.2.112:0/1125100034 conn(0x7f35640ce420 msgr2=0x7f35640d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:02.450 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:02.446 7f3568e01700 1 -- 172.21.2.112:0/1125100034 shutdown_connections 2021-02-10T11:32:02.450 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:02.446 7f3568e01700 1 -- 172.21.2.112:0/1125100034 wait complete. 2021-02-10T11:32:02.452 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:02.460 7f3568e01700 1 Processor -- start 2021-02-10T11:32:02.452 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:02.460 7f3568e01700 1 -- start start 2021-02-10T11:32:02.452 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:02.460 7f3568e01700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f356408c480 0x7f3564083760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:02.452 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:02.460 7f3568e01700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3564083d80 0x7f356408d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:02.452 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:02.460 7f3568e01700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f356418ef30 0x7f356408d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:02.452 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:02.460 7f3568e01700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f356408ce30 con 0x7f356418ef30 2021-02-10T11:32:02.452 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:02.460 7f3568e01700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f356408c880 con 0x7f356408c480 2021-02-10T11:32:02.452 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:02.460 7f3568e01700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f356408ca60 con 0x7f3564083d80 2021-02-10T11:32:02.452 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:02.461 7f3563314700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f356418ef30 0x7f356408d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:02.452 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:02.461 7f3563314700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f356418ef30 0x7f356408d9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:48558/0 (socket says 172.21.2.112:48558) 2021-02-10T11:32:02.453 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:02.461 7f3563314700 1 -- 172.21.2.112:0/2964717777 learned_addr learned my addr 172.21.2.112:0/2964717777 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:32:02.453 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:02.461 7f3563314700 1 -- 172.21.2.112:0/2964717777 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3564083d80 msgr2=0x7f356408d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:02.453 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:02.461 7f3563314700 1 --2- 172.21.2.112:0/2964717777 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3564083d80 0x7f356408d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:02.453 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:02.461 7f3563314700 1 -- 172.21.2.112:0/2964717777 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f356408c480 msgr2=0x7f3564083760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:02.453 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:02.461 7f3563314700 1 --2- 172.21.2.112:0/2964717777 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f356408c480 0x7f3564083760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:02.453 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:02.461 7f3563314700 1 -- 172.21.2.112:0/2964717777 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f356408dff0 con 0x7f356418ef30 2021-02-10T11:32:02.453 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:02.461 7f3563314700 1 --2- 172.21.2.112:0/2964717777 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f356418ef30 0x7f356408d9d0 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 rx=0x7f355c00b9a0 tx=0x7f355c00bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:02.453 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:02.461 7f355b7fe700 1 -- 172.21.2.112:0/2964717777 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f355c006ce0 con 0x7f356418ef30 2021-02-10T11:32:02.453 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:02.462 7f355b7fe700 1 -- 172.21.2.112:0/2964717777 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f355c00c2f0 con 0x7f356418ef30 2021-02-10T11:32:02.453 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:02.462 7f3568e01700 1 -- 172.21.2.112:0/2964717777 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f356408dff0 con 0x7f356418ef30 2021-02-10T11:32:02.453 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:02.462 7f3568e01700 1 -- 172.21.2.112:0/2964717777 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f356408e580 con 0x7f356418ef30 2021-02-10T11:32:02.453 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:02.462 7f355b7fe700 1 -- 172.21.2.112:0/2964717777 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f355c00c810 con 0x7f356418ef30 2021-02-10T11:32:02.453 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:02.462 7f355b7fe700 1 -- 172.21.2.112:0/2964717777 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f355c007020 con 0x7f356418ef30 2021-02-10T11:32:02.453 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:02.462 7f355b7fe700 1 --2- 172.21.2.112:0/2964717777 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3548018730 0x7f354801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:02.454 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:02.462 7f355b7fe700 1 -- 172.21.2.112:0/2964717777 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f355c032bc0 con 0x7f356418ef30 2021-02-10T11:32:02.454 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:02.463 7f3562b13700 1 --2- 172.21.2.112:0/2964717777 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3548018730 0x7f354801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:02.454 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:02.464 7f3562b13700 1 --2- 172.21.2.112:0/2964717777 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3548018730 0x7f354801ac10 crc :-1 s=READY pgs=535 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:02.481 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:02.493 7f3568e01700 1 -- 172.21.2.112:0/2964717777 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3564082960 con 0x7f356418ef30 2021-02-10T11:32:02.484 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:02.496 7f355b7fe700 1 -- 172.21.2.112:0/2964717777 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f355c0093d0 con 0x7f356418ef30 2021-02-10T11:32:02.620 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:02.632 7f3568e01700 1 -- 172.21.2.112:0/2964717777 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f3564082960 con 0x7f356418ef30 2021-02-10T11:32:03.033 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:03.045 7f355b7fe700 1 -- 172.21.2.112:0/2964717777 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v23) v1 ==== 136+0+0 (secure 0 0 0) 0x7f355c006ff0 con 0x7f356418ef30 2021-02-10T11:32:03.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:03.051 7f3568e01700 1 -- 172.21.2.112:0/2964717777 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3548018730 msgr2=0x7f354801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:03.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:03.051 7f3568e01700 1 --2- 172.21.2.112:0/2964717777 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3548018730 0x7f354801ac10 crc :-1 s=READY pgs=535 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:03.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:03.051 7f3568e01700 1 -- 172.21.2.112:0/2964717777 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f356418ef30 msgr2=0x7f356408d9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:03.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:03.051 7f3568e01700 1 --2- 172.21.2.112:0/2964717777 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f356418ef30 0x7f356408d9d0 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 rx=0x7f355c00b9a0 tx=0x7f355c00bd40).stop 2021-02-10T11:32:03.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:03.051 7f3568e01700 1 -- 172.21.2.112:0/2964717777 shutdown_connections 2021-02-10T11:32:03.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:03.051 7f3568e01700 1 --2- 172.21.2.112:0/2964717777 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3548018730 0x7f354801ac10 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:03.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:03.051 7f3568e01700 1 --2- 172.21.2.112:0/2964717777 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f356418ef30 0x7f356408d9d0 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:03.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:03.051 7f3568e01700 1 --2- 172.21.2.112:0/2964717777 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3564083d80 0x7f356408d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:03.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:03.051 7f3568e01700 1 --2- 172.21.2.112:0/2964717777 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f356408c480 0x7f3564083760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:03.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:03.051 7f3568e01700 1 -- 172.21.2.112:0/2964717777 >> 172.21.2.112:0/2964717777 conn(0x7f3564085e40 msgr2=0x7f3564086290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:03.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:03.051 7f3568e01700 1 -- 172.21.2.112:0/2964717777 shutdown_connections 2021-02-10T11:32:03.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:03.051 7f3568e01700 1 -- 172.21.2.112:0/2964717777 wait complete. 2021-02-10T11:32:03.048 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs volume ls 2021-02-10T11:32:03.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:03.469 7f0a66007700 1 Processor -- start 2021-02-10T11:32:03.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:03.469 7f0a66007700 1 -- start start 2021-02-10T11:32:03.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:03.470 7f0a66007700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f0a600e5070 0x7f0a601b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:03.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:03.470 7f0a66007700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f0a600e5470 0x7f0a60080310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:03.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:03.470 7f0a66007700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a6007ed80 con 0x7f0a601b3e40 2021-02-10T11:32:03.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:03.470 7f0a66007700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a600d1790 con 0x7f0a600e5070 2021-02-10T11:32:03.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:03.470 7f0a66007700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a600d19b0 con 0x7f0a600e5470 2021-02-10T11:32:03.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:03.470 7f0a5f7fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f0a600e5070 0x7f0a601b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:03.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:03.470 7f0a5f7fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f0a600e5070 0x7f0a601b3820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:48562/0 (socket says 172.21.2.112:48562) 2021-02-10T11:32:03.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:03.470 7f0a5f7fe700 1 -- 172.21.2.112:0/330113074 learned_addr learned my addr 172.21.2.112:0/330113074 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:32:03.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:03.470 7f0a5ffff700 1 --2- 172.21.2.112:0/330113074 >> v2:172.21.2.120:3300/0 conn(0x7f0a600e5470 0x7f0a60080310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:03.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:03.470 7f0a5f7fe700 1 -- 172.21.2.112:0/330113074 >> v2:172.21.2.120:3300/0 conn(0x7f0a600e5470 msgr2=0x7f0a60080310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:03.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:03.470 7f0a5f7fe700 1 --2- 172.21.2.112:0/330113074 >> v2:172.21.2.120:3300/0 conn(0x7f0a600e5470 0x7f0a60080310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:03.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:03.470 7f0a5f7fe700 1 -- 172.21.2.112:0/330113074 >> v1:172.21.2.120:6789/0 conn(0x7f0a601b3e40 legacy=0x7f0a6007d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:03.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:03.470 7f0a5f7fe700 1 -- 172.21.2.112:0/330113074 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a600d1bd0 con 0x7f0a600e5070 2021-02-10T11:32:03.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:03.471 7f0a5f7fe700 1 --2- 172.21.2.112:0/330113074 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0a600e5070 0x7f0a601b3820 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 rx=0x7f0a50006700 tx=0x7f0a50009b80).ready entity=mon.0 client_cookie=93f76e896318b391 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:03.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:03.471 7f0a5e7fc700 1 -- 172.21.2.112:0/330113074 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0a5000c0c0 con 0x7f0a600e5070 2021-02-10T11:32:03.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:03.471 7f0a5e7fc700 1 -- 172.21.2.112:0/330113074 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a5000c340 con 0x7f0a600e5070 2021-02-10T11:32:03.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:03.471 7f0a5e7fc700 1 -- 172.21.2.112:0/330113074 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0a5000c7a0 con 0x7f0a600e5070 2021-02-10T11:32:03.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:03.471 7f0a66007700 1 -- 172.21.2.112:0/330113074 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0a600e5070 msgr2=0x7f0a601b3820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:03.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:03.471 7f0a66007700 1 --2- 172.21.2.112:0/330113074 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0a600e5070 0x7f0a601b3820 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 rx=0x7f0a50006700 tx=0x7f0a50009b80).stop 2021-02-10T11:32:03.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:03.472 7f0a66007700 1 -- 172.21.2.112:0/330113074 shutdown_connections 2021-02-10T11:32:03.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:03.472 7f0a66007700 1 --2- 172.21.2.112:0/330113074 >> v2:172.21.2.120:3300/0 conn(0x7f0a600e5470 0x7f0a60080310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:03.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:03.472 7f0a66007700 1 --2- 172.21.2.112:0/330113074 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0a600e5070 0x7f0a601b3820 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:03.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:03.472 7f0a66007700 1 -- 172.21.2.112:0/330113074 >> 172.21.2.112:0/330113074 conn(0x7f0a600ce690 msgr2=0x7f0a600e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:03.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:03.472 7f0a66007700 1 -- 172.21.2.112:0/330113074 shutdown_connections 2021-02-10T11:32:03.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:03.472 7f0a66007700 1 -- 172.21.2.112:0/330113074 wait complete. 2021-02-10T11:32:03.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:03.486 7f0a66007700 1 Processor -- start 2021-02-10T11:32:03.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:03.486 7f0a66007700 1 -- start start 2021-02-10T11:32:03.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:03.487 7f0a66007700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0a600bd700 0x7f0a600bdb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:03.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:03.487 7f0a66007700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0a600a4b10 0x7f0a600a4f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:03.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:03.487 7f0a66007700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0a600a5530 0x7f0a600a5930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:03.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:03.487 7f0a66007700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a600be230 con 0x7f0a600a4b10 2021-02-10T11:32:03.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:03.487 7f0a66007700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a600be450 con 0x7f0a600a5530 2021-02-10T11:32:03.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:03.487 7f0a66007700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a600be670 con 0x7f0a600bd700 2021-02-10T11:32:03.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:03.487 7f0a5effd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0a600a4b10 0x7f0a600a4f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:03.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:03.487 7f0a5effd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0a600a4b10 0x7f0a600a4f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:48568/0 (socket says 172.21.2.112:48568) 2021-02-10T11:32:03.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:03.487 7f0a5effd700 1 -- 172.21.2.112:0/2328966698 learned_addr learned my addr 172.21.2.112:0/2328966698 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:32:03.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:03.487 7f0a5effd700 1 -- 172.21.2.112:0/2328966698 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0a600bd700 msgr2=0x7f0a600bdb00 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:32:03.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:03.487 7f0a5effd700 1 --2- 172.21.2.112:0/2328966698 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0a600bd700 0x7f0a600bdb00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:03.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:03.487 7f0a5effd700 1 -- 172.21.2.112:0/2328966698 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0a600a5530 msgr2=0x7f0a600a5930 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:32:03.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:03.487 7f0a5effd700 1 --2- 172.21.2.112:0/2328966698 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0a600a5530 0x7f0a600a5930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:03.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:03.487 7f0a5effd700 1 -- 172.21.2.112:0/2328966698 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a600be890 con 0x7f0a600a4b10 2021-02-10T11:32:03.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:03.487 7f0a5effd700 1 --2- 172.21.2.112:0/2328966698 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0a600a4b10 0x7f0a600a4f10 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 rx=0x7f0a4c006ed0 tx=0x7f0a4c00b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:03.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:03.488 7f0a57fff700 1 -- 172.21.2.112:0/2328966698 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0a4c006020 con 0x7f0a600a4b10 2021-02-10T11:32:03.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:03.488 7f0a57fff700 1 -- 172.21.2.112:0/2328966698 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a4c006320 con 0x7f0a600a4b10 2021-02-10T11:32:03.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:03.488 7f0a66007700 1 -- 172.21.2.112:0/2328966698 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a600be890 con 0x7f0a600a4b10 2021-02-10T11:32:03.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:03.488 7f0a66007700 1 -- 172.21.2.112:0/2328966698 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a600bee20 con 0x7f0a600a4b10 2021-02-10T11:32:03.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:03.488 7f0a57fff700 1 -- 172.21.2.112:0/2328966698 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0a4c005730 con 0x7f0a600a4b10 2021-02-10T11:32:03.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:03.488 7f0a57fff700 1 -- 172.21.2.112:0/2328966698 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0a4c0054b0 con 0x7f0a600a4b10 2021-02-10T11:32:03.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:03.489 7f0a57fff700 1 --2- 172.21.2.112:0/2328966698 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0a3c0186c0 0x7f0a3c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:03.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:03.489 7f0a57fff700 1 -- 172.21.2.112:0/2328966698 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f0a4c032590 con 0x7f0a600a4b10 2021-02-10T11:32:03.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:03.489 7f0a5f7fe700 1 --2- 172.21.2.112:0/2328966698 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0a3c0186c0 0x7f0a3c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:03.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:03.490 7f0a5f7fe700 1 --2- 172.21.2.112:0/2328966698 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0a3c0186c0 0x7f0a3c01abc0 crc :-1 s=READY pgs=536 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:03.508 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:03.520 7f0a66007700 1 -- 172.21.2.112:0/2328966698 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a6007ed80 con 0x7f0a600a4b10 2021-02-10T11:32:03.510 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:03.522 7f0a57fff700 1 -- 172.21.2.112:0/2328966698 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0a4c030d40 con 0x7f0a600a4b10 2021-02-10T11:32:03.645 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:03.656 7f0a66007700 1 -- 172.21.2.112:0/2328966698 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs volume ls", "target": ["mgr", ""]}) v1 -- 0x7f0a600bee20 con 0x7f0a3c0186c0 2021-02-10T11:32:03.646 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:03.659 7f0a57fff700 1 -- 172.21.2.112:0/2328966698 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+41 (crc 0 0 0) 0x7f0a500065e0 con 0x7f0a3c0186c0 2021-02-10T11:32:03.648 INFO:teuthology.orchestra.run.gibba012.stdout:[ 2021-02-10T11:32:03.648 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:32:03.648 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "cephfs" 2021-02-10T11:32:03.648 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:32:03.648 INFO:teuthology.orchestra.run.gibba012.stdout:] 2021-02-10T11:32:03.649 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:03.661 7f0a66007700 1 -- 172.21.2.112:0/2328966698 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0a3c0186c0 msgr2=0x7f0a3c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:03.649 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:03.661 7f0a66007700 1 --2- 172.21.2.112:0/2328966698 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0a3c0186c0 0x7f0a3c01abc0 crc :-1 s=READY pgs=536 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:03.649 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:03.661 7f0a66007700 1 -- 172.21.2.112:0/2328966698 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0a600a4b10 msgr2=0x7f0a600a4f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:03.649 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:03.661 7f0a66007700 1 --2- 172.21.2.112:0/2328966698 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0a600a4b10 0x7f0a600a4f10 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 rx=0x7f0a4c006ed0 tx=0x7f0a4c00b960).stop 2021-02-10T11:32:03.649 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:03.661 7f0a66007700 1 -- 172.21.2.112:0/2328966698 shutdown_connections 2021-02-10T11:32:03.650 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:03.661 7f0a66007700 1 --2- 172.21.2.112:0/2328966698 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0a3c0186c0 0x7f0a3c01abc0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:03.650 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:03.661 7f0a66007700 1 --2- 172.21.2.112:0/2328966698 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0a600a5530 0x7f0a600a5930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:03.650 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:03.661 7f0a66007700 1 --2- 172.21.2.112:0/2328966698 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0a600a4b10 0x7f0a600a4f10 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:03.650 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:03.661 7f0a66007700 1 --2- 172.21.2.112:0/2328966698 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0a600bd700 0x7f0a600bdb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:03.650 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:03.661 7f0a66007700 1 -- 172.21.2.112:0/2328966698 >> 172.21.2.112:0/2328966698 conn(0x7f0a600c0cf0 msgr2=0x7f0a600a70a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:03.650 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:03.661 7f0a66007700 1 -- 172.21.2.112:0/2328966698 shutdown_connections 2021-02-10T11:32:03.650 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:03.661 7f0a66007700 1 -- 172.21.2.112:0/2328966698 wait complete. 2021-02-10T11:32:03.658 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:32:04.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.086 7fe4ab843700 1 Processor -- start 2021-02-10T11:32:04.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.086 7fe4ab843700 1 -- start start 2021-02-10T11:32:04.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.086 7fe4ab843700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fe4a41b0470 0x7fe4a41b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:04.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.086 7fe4ab843700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fe4a40ef610 0x7fe4a40efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:04.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.086 7fe4ab843700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe4a40f0030 0x7fe4a40f0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:04.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.086 7fe4ab843700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4a40e5ea0 con 0x7fe4a41b0470 2021-02-10T11:32:04.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.086 7fe4ab843700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4a40f0b50 con 0x7fe4a40ef610 2021-02-10T11:32:04.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.086 7fe4ab843700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4a40e52e0 con 0x7fe4a40f0030 2021-02-10T11:32:04.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.087 7fe4a9e60700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe4a40f0030 0x7fe4a40f0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:04.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.087 7fe4a965f700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fe4a41b0470 0x7fe4a41b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:04.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.087 7fe4a965f700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fe4a41b0470 0x7fe4a41b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:48574/0 (socket says 172.21.2.112:48574) 2021-02-10T11:32:04.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.087 7fe4a9e60700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe4a40f0030 0x7fe4a40f0430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3301/0 says I am v2:172.21.2.112:48254/0 (socket says 172.21.2.112:48254) 2021-02-10T11:32:04.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.087 7fe4a965f700 1 -- 172.21.2.112:0/1654443575 learned_addr learned my addr 172.21.2.112:0/1654443575 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:32:04.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.087 7fe4a8e5e700 1 --2- 172.21.2.112:0/1654443575 >> v2:172.21.2.120:3300/0 conn(0x7fe4a40ef610 0x7fe4a40efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:04.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.087 7fe4a965f700 1 -- 172.21.2.112:0/1654443575 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe4a40f0030 msgr2=0x7fe4a40f0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:04.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.087 7fe4a965f700 1 --2- 172.21.2.112:0/1654443575 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe4a40f0030 0x7fe4a40f0430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:04.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.087 7fe4a965f700 1 -- 172.21.2.112:0/1654443575 >> v2:172.21.2.120:3300/0 conn(0x7fe4a40ef610 msgr2=0x7fe4a40efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:04.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.087 7fe4a965f700 1 --2- 172.21.2.112:0/1654443575 >> v2:172.21.2.120:3300/0 conn(0x7fe4a40ef610 0x7fe4a40efa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:04.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.087 7fe4a965f700 1 -- 172.21.2.112:0/1654443575 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4a40f10d0 con 0x7fe4a41b0470 2021-02-10T11:32:04.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.087 7fe4a965f700 1 --2- 172.21.2.112:0/1654443575 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe4a41b0470 0x7fe4a41b0870 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 rx=0x7fe4a0006700 tx=0x7fe4a0009b80).ready entity=mon.0 client_cookie=336049e61fa6c67b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:04.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.087 7fe497fff700 1 -- 172.21.2.112:0/1654443575 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe4a000c0c0 con 0x7fe4a41b0470 2021-02-10T11:32:04.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.087 7fe497fff700 1 -- 172.21.2.112:0/1654443575 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4a000c3c0 con 0x7fe4a41b0470 2021-02-10T11:32:04.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.087 7fe497fff700 1 -- 172.21.2.112:0/1654443575 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe4a000c900 con 0x7fe4a41b0470 2021-02-10T11:32:04.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.087 7fe4ab843700 1 -- 172.21.2.112:0/1654443575 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe4a41b0470 msgr2=0x7fe4a41b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:04.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.087 7fe4ab843700 1 --2- 172.21.2.112:0/1654443575 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe4a41b0470 0x7fe4a41b0870 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 rx=0x7fe4a0006700 tx=0x7fe4a0009b80).stop 2021-02-10T11:32:04.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.088 7fe4ab843700 1 -- 172.21.2.112:0/1654443575 shutdown_connections 2021-02-10T11:32:04.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.088 7fe4ab843700 1 --2- 172.21.2.112:0/1654443575 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe4a40f0030 0x7fe4a40f0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:04.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.088 7fe4ab843700 1 --2- 172.21.2.112:0/1654443575 >> v2:172.21.2.120:3300/0 conn(0x7fe4a40ef610 0x7fe4a40efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:04.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.088 7fe4ab843700 1 --2- 172.21.2.112:0/1654443575 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe4a41b0470 0x7fe4a41b0870 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:04.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.088 7fe4ab843700 1 -- 172.21.2.112:0/1654443575 >> 172.21.2.112:0/1654443575 conn(0x7fe4a40ce3e0 msgr2=0x7fe4a40e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:04.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.088 7fe4ab843700 1 -- 172.21.2.112:0/1654443575 shutdown_connections 2021-02-10T11:32:04.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.088 7fe4ab843700 1 -- 172.21.2.112:0/1654443575 wait complete. 2021-02-10T11:32:04.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.103 7fe4ab843700 1 Processor -- start 2021-02-10T11:32:04.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.103 7fe4ab843700 1 -- start start 2021-02-10T11:32:04.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.103 7fe4ab843700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe4a409b930 0x7fe4a40b7c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:04.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.103 7fe4ab843700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe4a40b8410 0x7fe4a40b8810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:04.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.103 7fe4ab843700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe4a40c1f80 0x7fe4a40c2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:04.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.103 7fe4ab843700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4a40c1a00 con 0x7fe4a409b930 2021-02-10T11:32:04.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.103 7fe4ab843700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4a40b8e30 con 0x7fe4a40b8410 2021-02-10T11:32:04.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.103 7fe4ab843700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4a40b9050 con 0x7fe4a40c1f80 2021-02-10T11:32:04.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.104 7fe4a965f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe4a409b930 0x7fe4a40b7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:04.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.104 7fe4a965f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe4a409b930 0x7fe4a40b7c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:48580/0 (socket says 172.21.2.112:48580) 2021-02-10T11:32:04.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.104 7fe4a965f700 1 -- 172.21.2.112:0/3673183957 learned_addr learned my addr 172.21.2.112:0/3673183957 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:32:04.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.104 7fe4a965f700 1 -- 172.21.2.112:0/3673183957 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe4a40c1f80 msgr2=0x7fe4a40c2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:04.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.104 7fe4a965f700 1 --2- 172.21.2.112:0/3673183957 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe4a40c1f80 0x7fe4a40c2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:04.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.104 7fe4a965f700 1 -- 172.21.2.112:0/3673183957 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe4a40b8410 msgr2=0x7fe4a40b8810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:04.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.104 7fe4a965f700 1 --2- 172.21.2.112:0/3673183957 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe4a40b8410 0x7fe4a40b8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:04.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.104 7fe4a965f700 1 -- 172.21.2.112:0/3673183957 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4a40c29a0 con 0x7fe4a409b930 2021-02-10T11:32:04.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.104 7fe4a965f700 1 --2- 172.21.2.112:0/3673183957 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe4a409b930 0x7fe4a40b7c20 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 rx=0x7fe4a000b5e0 tx=0x7fe4a000b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:04.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.104 7fe495ffb700 1 -- 172.21.2.112:0/3673183957 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe4a000bd70 con 0x7fe4a409b930 2021-02-10T11:32:04.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.104 7fe495ffb700 1 -- 172.21.2.112:0/3673183957 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4a0009440 con 0x7fe4a409b930 2021-02-10T11:32:04.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.104 7fe4ab843700 1 -- 172.21.2.112:0/3673183957 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4a40c29a0 con 0x7fe4a409b930 2021-02-10T11:32:04.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.104 7fe495ffb700 1 -- 172.21.2.112:0/3673183957 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe4a0002300 con 0x7fe4a409b930 2021-02-10T11:32:04.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.104 7fe4ab843700 1 -- 172.21.2.112:0/3673183957 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4a40c2f30 con 0x7fe4a409b930 2021-02-10T11:32:04.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.105 7fe495ffb700 1 -- 172.21.2.112:0/3673183957 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe4a0000c90 con 0x7fe4a409b930 2021-02-10T11:32:04.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.105 7fe495ffb700 1 --2- 172.21.2.112:0/3673183957 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe4840186c0 0x7fe48401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:04.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.105 7fe495ffb700 1 -- 172.21.2.112:0/3673183957 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fe4a0032690 con 0x7fe4a409b930 2021-02-10T11:32:04.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.105 7fe4a8e5e700 1 --2- 172.21.2.112:0/3673183957 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe4840186c0 0x7fe48401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:04.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.106 7fe4a8e5e700 1 --2- 172.21.2.112:0/3673183957 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe4840186c0 0x7fe48401abc0 crc :-1 s=READY pgs=537 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:04.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.136 7fe4ab843700 1 -- 172.21.2.112:0/3673183957 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4a4105b30 con 0x7fe4a409b930 2021-02-10T11:32:04.127 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.139 7fe495ffb700 1 -- 172.21.2.112:0/3673183957 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe4a0009440 con 0x7fe4a409b930 2021-02-10T11:32:04.276 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.288 7fe4ab843700 1 -- 172.21.2.112:0/3673183957 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fe4a40f0b50 con 0x7fe4a409b930 2021-02-10T11:32:04.277 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.289 7fe495ffb700 1 -- 172.21.2.112:0/3673183957 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v4)=0 v4) v1 ==== 122+0+0 (secure 0 0 0) 0x7fe4a0000c90 con 0x7fe4a409b930 2021-02-10T11:32:04.279 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.291 7fe4ab843700 1 -- 172.21.2.112:0/3673183957 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe4840186c0 msgr2=0x7fe48401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:04.279 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.291 7fe4ab843700 1 --2- 172.21.2.112:0/3673183957 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe4840186c0 0x7fe48401abc0 crc :-1 s=READY pgs=537 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:04.279 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.291 7fe4ab843700 1 -- 172.21.2.112:0/3673183957 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe4a409b930 msgr2=0x7fe4a40b7c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:04.279 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.291 7fe4ab843700 1 --2- 172.21.2.112:0/3673183957 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe4a409b930 0x7fe4a40b7c20 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 rx=0x7fe4a000b5e0 tx=0x7fe4a000b980).stop 2021-02-10T11:32:04.279 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.291 7fe4ab843700 1 -- 172.21.2.112:0/3673183957 shutdown_connections 2021-02-10T11:32:04.280 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.291 7fe4ab843700 1 --2- 172.21.2.112:0/3673183957 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe4840186c0 0x7fe48401abc0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:04.280 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.291 7fe4ab843700 1 --2- 172.21.2.112:0/3673183957 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe4a40b8410 0x7fe4a40b8810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:04.280 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.291 7fe4ab843700 1 --2- 172.21.2.112:0/3673183957 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe4a40c1f80 0x7fe4a40c2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:04.280 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.291 7fe4ab843700 1 --2- 172.21.2.112:0/3673183957 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe4a409b930 0x7fe4a40b7c20 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:04.280 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.291 7fe4ab843700 1 -- 172.21.2.112:0/3673183957 >> 172.21.2.112:0/3673183957 conn(0x7fe4a4094e10 msgr2=0x7fe4a40a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:04.280 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.291 7fe4ab843700 1 -- 172.21.2.112:0/3673183957 shutdown_connections 2021-02-10T11:32:04.280 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.291 7fe4ab843700 1 -- 172.21.2.112:0/3673183957 wait complete. 2021-02-10T11:32:04.288 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth get-or-create client.guest1 mds 'allow *' osd 'allow rw' mon 'allow *' 2021-02-10T11:32:04.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.721 7f731d83f700 1 Processor -- start 2021-02-10T11:32:04.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.722 7f731d83f700 1 -- start start 2021-02-10T11:32:04.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.722 7f731d83f700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f73181b0470 0x7f73181b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:04.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.722 7f731d83f700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f73180f20e0 0x7f73180f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:04.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.722 7f731d83f700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73180f0b50 con 0x7f73180ef610 2021-02-10T11:32:04.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.722 7f731d83f700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73180f0820 con 0x7f73181b0470 2021-02-10T11:32:04.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.722 7f731d83f700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73180f24e0 con 0x7f73180f20e0 2021-02-10T11:32:04.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.722 7f7317574700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f73181b0470 0x7f73181b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:04.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.722 7f7316d73700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f73180ef610 0x7f73180efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:48048/0 (socket says 172.21.2.112:48048) 2021-02-10T11:32:04.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.722 7f7316d73700 1 -- 172.21.2.112:0/501832953 learned_addr learned my addr 172.21.2.112:0/501832953 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:32:04.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.722 7f7316572700 1 -- 172.21.2.112:0/501832953 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3524345546 0 0) 0x7f73000010b0 con 0x7f73180ef610 2021-02-10T11:32:04.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.722 7f7316572700 1 -- 172.21.2.112:0/501832953 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7304002fe0 con 0x7f73180ef610 2021-02-10T11:32:04.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.723 7f7317574700 1 -- 172.21.2.112:0/501832953 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f73180f20e0 msgr2=0x7f73180f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:04.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.723 7f7317574700 1 --2- 172.21.2.112:0/501832953 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f73180f20e0 0x7f73180f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:04.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.723 7f7317574700 1 -- 172.21.2.112:0/501832953 >> v1:172.21.2.112:6789/0 conn(0x7f73180ef610 legacy=0x7f73180efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:04.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.723 7f7317574700 1 -- 172.21.2.112:0/501832953 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73180e52e0 con 0x7f73181b0470 2021-02-10T11:32:04.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.723 7f7317574700 1 --2- 172.21.2.112:0/501832953 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f73181b0470 0x7f73181b0870 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 rx=0x7f7308006650 tx=0x7f7308009a00).ready entity=mon.0 client_cookie=ff591ee76a973d7a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:04.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.723 7f7316572700 1 -- 172.21.2.112:0/501832953 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7308009df0 con 0x7f73181b0470 2021-02-10T11:32:04.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.723 7f7316572700 1 -- 172.21.2.112:0/501832953 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73080009b0 con 0x7f73181b0470 2021-02-10T11:32:04.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.723 7f7316572700 1 -- 172.21.2.112:0/501832953 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f730800c2b0 con 0x7f73181b0470 2021-02-10T11:32:04.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.723 7f731d83f700 1 -- 172.21.2.112:0/501832953 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f73181b0470 msgr2=0x7f73181b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:04.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.723 7f731d83f700 1 --2- 172.21.2.112:0/501832953 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f73181b0470 0x7f73181b0870 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 rx=0x7f7308006650 tx=0x7f7308009a00).stop 2021-02-10T11:32:04.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.724 7f731d83f700 1 -- 172.21.2.112:0/501832953 shutdown_connections 2021-02-10T11:32:04.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.724 7f731d83f700 1 --2- 172.21.2.112:0/501832953 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f73180f20e0 0x7f73180f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:04.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.724 7f731d83f700 1 --2- 172.21.2.112:0/501832953 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f73181b0470 0x7f73181b0870 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:04.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.724 7f731d83f700 1 -- 172.21.2.112:0/501832953 >> 172.21.2.112:0/501832953 conn(0x7f73180ce3e0 msgr2=0x7f73180e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:04.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.724 7f731d83f700 1 -- 172.21.2.112:0/501832953 shutdown_connections 2021-02-10T11:32:04.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.724 7f731d83f700 1 -- 172.21.2.112:0/501832953 wait complete. 2021-02-10T11:32:04.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.738 7f731d83f700 1 Processor -- start 2021-02-10T11:32:04.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.738 7f731d83f700 1 -- start start 2021-02-10T11:32:04.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.739 7f731d83f700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f731818ed50 0x7f73180ad800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:04.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.739 7f731d83f700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f73180ade20 0x7f73181b6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:04.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.739 7f731d83f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f731819c2d0 0x7f73181b73f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:04.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.739 7f731d83f700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f731818e7d0 con 0x7f731819c2d0 2021-02-10T11:32:04.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.739 7f731d83f700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73181b7a10 con 0x7f731818ed50 2021-02-10T11:32:04.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.739 7f731d83f700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73181b7bf0 con 0x7f73180ade20 2021-02-10T11:32:04.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.739 7f7317d75700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f731819c2d0 0x7f73181b73f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:04.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.739 7f7317d75700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f731819c2d0 0x7f73181b73f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:48592/0 (socket says 172.21.2.112:48592) 2021-02-10T11:32:04.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.739 7f7317d75700 1 -- 172.21.2.112:0/1301413830 learned_addr learned my addr 172.21.2.112:0/1301413830 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:32:04.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.739 7f7317d75700 1 -- 172.21.2.112:0/1301413830 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f73180ade20 msgr2=0x7f73181b6dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:04.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.739 7f7317d75700 1 --2- 172.21.2.112:0/1301413830 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f73180ade20 0x7f73181b6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:04.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.739 7f7317d75700 1 -- 172.21.2.112:0/1301413830 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f731818ed50 msgr2=0x7f73180ad800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:04.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.739 7f7317d75700 1 --2- 172.21.2.112:0/1301413830 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f731818ed50 0x7f73180ad800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:04.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.739 7f7317d75700 1 -- 172.21.2.112:0/1301413830 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f731818cee0 con 0x7f731819c2d0 2021-02-10T11:32:04.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.740 7f7317d75700 1 --2- 172.21.2.112:0/1301413830 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f731819c2d0 0x7f73181b73f0 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 rx=0x7f7310009980 tx=0x7f731000bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:04.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.740 7f730ffff700 1 -- 172.21.2.112:0/1301413830 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7310003020 con 0x7f731819c2d0 2021-02-10T11:32:04.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.740 7f730ffff700 1 -- 172.21.2.112:0/1301413830 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7310005a80 con 0x7f731819c2d0 2021-02-10T11:32:04.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.740 7f731d83f700 1 -- 172.21.2.112:0/1301413830 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f731818cee0 con 0x7f731819c2d0 2021-02-10T11:32:04.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.740 7f731d83f700 1 -- 172.21.2.112:0/1301413830 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f731818d470 con 0x7f731819c2d0 2021-02-10T11:32:04.732 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.740 7f730ffff700 1 -- 172.21.2.112:0/1301413830 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7310006030 con 0x7f731819c2d0 2021-02-10T11:32:04.732 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.740 7f730ffff700 1 -- 172.21.2.112:0/1301413830 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7310005c60 con 0x7f731819c2d0 2021-02-10T11:32:04.732 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.740 7f730ffff700 1 --2- 172.21.2.112:0/1301413830 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f72fc0186c0 0x7f72fc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:04.732 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.740 7f730ffff700 1 -- 172.21.2.112:0/1301413830 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f73100328e0 con 0x7f731819c2d0 2021-02-10T11:32:04.732 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.741 7f7317574700 1 --2- 172.21.2.112:0/1301413830 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f72fc0186c0 0x7f72fc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:04.732 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.741 7f7317574700 1 --2- 172.21.2.112:0/1301413830 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f72fc0186c0 0x7f72fc01abc0 crc :-1 s=READY pgs=538 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:04.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.772 7f731d83f700 1 -- 172.21.2.112:0/1301413830 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73181059f0 con 0x7f731819c2d0 2021-02-10T11:32:04.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.774 7f730ffff700 1 -- 172.21.2.112:0/1301413830 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f73100062f0 con 0x7f731819c2d0 2021-02-10T11:32:04.894 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.906 7f731d83f700 1 -- 172.21.2.112:0/1301413830 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.guest1", "caps": ["mds", "allow *", "osd", "allow rw", "mon", "allow *"]} v 0) v1 -- 0x7f73180f0b50 con 0x7f731819c2d0 2021-02-10T11:32:04.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.910 7f730ffff700 1 -- 172.21.2.112:0/1301413830 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.guest1", "caps": ["mds", "allow *", "osd", "allow rw", "mon", "allow *"]}]=0 v21) v1 ==== 158+0+64 (secure 0 0 0) 0x7f7310005c60 con 0x7f731819c2d0 2021-02-10T11:32:04.901 INFO:teuthology.orchestra.run.gibba012.stdout:[client.guest1] 2021-02-10T11:32:04.901 INFO:teuthology.orchestra.run.gibba012.stdout: key = AQA0xCNg3bEjNhAA9hi8UJZVHetJC//dAy6vzw== 2021-02-10T11:32:04.904 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.914 7f731d83f700 1 -- 172.21.2.112:0/1301413830 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f72fc0186c0 msgr2=0x7f72fc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:04.904 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.914 7f731d83f700 1 --2- 172.21.2.112:0/1301413830 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f72fc0186c0 0x7f72fc01abc0 crc :-1 s=READY pgs=538 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:04.904 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.914 7f731d83f700 1 -- 172.21.2.112:0/1301413830 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f731819c2d0 msgr2=0x7f73181b73f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:04.904 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.914 7f731d83f700 1 --2- 172.21.2.112:0/1301413830 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f731819c2d0 0x7f73181b73f0 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 rx=0x7f7310009980 tx=0x7f731000bcd0).stop 2021-02-10T11:32:04.904 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.914 7f731d83f700 1 -- 172.21.2.112:0/1301413830 shutdown_connections 2021-02-10T11:32:04.905 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.914 7f731d83f700 1 --2- 172.21.2.112:0/1301413830 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f72fc0186c0 0x7f72fc01abc0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:04.905 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.914 7f731d83f700 1 --2- 172.21.2.112:0/1301413830 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f731819c2d0 0x7f73181b73f0 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:04.905 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.914 7f731d83f700 1 --2- 172.21.2.112:0/1301413830 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f73180ade20 0x7f73181b6dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:04.905 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.915 7f731d83f700 1 --2- 172.21.2.112:0/1301413830 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f731818ed50 0x7f73180ad800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:04.905 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.915 7f731d83f700 1 -- 172.21.2.112:0/1301413830 >> 172.21.2.112:0/1301413830 conn(0x7f73181a3790 msgr2=0x7f731819f670 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:04.905 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.915 7f731d83f700 1 -- 172.21.2.112:0/1301413830 shutdown_connections 2021-02-10T11:32:04.906 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:04.915 7f731d83f700 1 -- 172.21.2.112:0/1301413830 wait complete. 2021-02-10T11:32:04.912 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolumegroup create cephfs group_0000000000550574 2021-02-10T11:32:05.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.337 7fb35d27f700 1 Processor -- start 2021-02-10T11:32:05.342 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.337 7fb35d27f700 1 -- start start 2021-02-10T11:32:05.342 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.337 7fb35d27f700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fb3581b0870 0x7fb3580f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:05.342 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.337 7fb35d27f700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3581b0c70 con 0x7fb3580efe00 2021-02-10T11:32:05.342 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.337 7fb35d27f700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3580f0b50 con 0x7fb3581b0470 2021-02-10T11:32:05.342 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.337 7fb35d27f700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3580e5ea0 con 0x7fb3581b0870 2021-02-10T11:32:05.342 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.337 7fb35687c700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fb3581b0870 0x7fb3580f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:05.342 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.337 7fb35687c700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fb3581b0870 0x7fb3580f20e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:48598/0 (socket says 172.21.2.112:48598) 2021-02-10T11:32:05.342 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.338 7fb35687c700 1 -- 172.21.2.112:0/271767576 learned_addr learned my addr 172.21.2.112:0/271767576 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:32:05.342 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.338 7fb35607b700 1 -- 172.21.2.112:0/271767576 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3312467093 0 0) 0x7fb348000fe0 con 0x7fb3581b0470 2021-02-10T11:32:05.342 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.338 7fb35607b700 1 -- 172.21.2.112:0/271767576 --> v1:172.21.2.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb344002fe0 con 0x7fb3581b0470 2021-02-10T11:32:05.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.338 7fb35687c700 1 -- 172.21.2.112:0/271767576 >> v1:172.21.2.120:6789/0 conn(0x7fb3581b0470 legacy=0x7fb3580ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:05.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.338 7fb35687c700 1 -- 172.21.2.112:0/271767576 >> v1:172.21.2.112:6789/0 conn(0x7fb3580efe00 legacy=0x7fb3580f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:05.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.338 7fb35687c700 1 -- 172.21.2.112:0/271767576 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3580d2fb0 con 0x7fb3581b0870 2021-02-10T11:32:05.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.338 7fb35687c700 1 --2- 172.21.2.112:0/271767576 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb3581b0870 0x7fb3580f20e0 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 rx=0x7fb340006700 tx=0x7fb340009b80).ready entity=mon.0 client_cookie=1a4a0ff5ff93dbba server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:05.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.338 7fb35607b700 1 -- 172.21.2.112:0/271767576 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb34000c0c0 con 0x7fb3581b0870 2021-02-10T11:32:05.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.338 7fb35607b700 1 -- 172.21.2.112:0/271767576 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb34000c440 con 0x7fb3581b0870 2021-02-10T11:32:05.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.338 7fb35607b700 1 -- 172.21.2.112:0/271767576 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb34000c920 con 0x7fb3581b0870 2021-02-10T11:32:05.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.338 7fb35d27f700 1 -- 172.21.2.112:0/271767576 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb3581b0870 msgr2=0x7fb3580f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:05.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.338 7fb35d27f700 1 --2- 172.21.2.112:0/271767576 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb3581b0870 0x7fb3580f20e0 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 rx=0x7fb340006700 tx=0x7fb340009b80).stop 2021-02-10T11:32:05.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.339 7fb35d27f700 1 -- 172.21.2.112:0/271767576 shutdown_connections 2021-02-10T11:32:05.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.339 7fb35d27f700 1 --2- 172.21.2.112:0/271767576 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb3581b0870 0x7fb3580f20e0 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:05.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.339 7fb35d27f700 1 -- 172.21.2.112:0/271767576 >> 172.21.2.112:0/271767576 conn(0x7fb3580ce3e0 msgr2=0x7fb3580e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:05.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.339 7fb35d27f700 1 -- 172.21.2.112:0/271767576 shutdown_connections 2021-02-10T11:32:05.344 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.339 7fb35d27f700 1 -- 172.21.2.112:0/271767576 wait complete. 2021-02-10T11:32:05.345 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.354 7fb35d27f700 1 Processor -- start 2021-02-10T11:32:05.346 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.354 7fb35d27f700 1 -- start start 2021-02-10T11:32:05.346 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.354 7fb35d27f700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb3580c36a0 0x7fb3580c2b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:05.346 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.354 7fb35d27f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb3580c3190 0x7fb3580b1d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:05.346 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.354 7fb35d27f700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb3580b2370 0x7fb3580b2d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:05.346 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.354 7fb35d27f700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3580b27c0 con 0x7fb3580c3190 2021-02-10T11:32:05.346 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.354 7fb35d27f700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3580b3360 con 0x7fb3580b2370 2021-02-10T11:32:05.346 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.354 7fb35d27f700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3580b3540 con 0x7fb3580c36a0 2021-02-10T11:32:05.346 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.355 7fb35687c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb3580c3190 0x7fb3580b1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:05.346 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.355 7fb35707d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb3580c36a0 0x7fb3580c2b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:05.346 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.355 7fb35687c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb3580c3190 0x7fb3580b1d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:48604/0 (socket says 172.21.2.112:48604) 2021-02-10T11:32:05.346 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.355 7fb35687c700 1 -- 172.21.2.112:0/2249739464 learned_addr learned my addr 172.21.2.112:0/2249739464 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:32:05.346 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.355 7fb35787e700 1 --2- 172.21.2.112:0/2249739464 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb3580b2370 0x7fb3580b2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:05.346 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.355 7fb35687c700 1 -- 172.21.2.112:0/2249739464 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb3580c36a0 msgr2=0x7fb3580c2b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:05.347 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.355 7fb35687c700 1 --2- 172.21.2.112:0/2249739464 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb3580c36a0 0x7fb3580c2b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:05.347 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.355 7fb35687c700 1 -- 172.21.2.112:0/2249739464 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb3580b2370 msgr2=0x7fb3580b2d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:05.347 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.355 7fb35687c700 1 --2- 172.21.2.112:0/2249739464 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb3580b2370 0x7fb3580b2d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:05.347 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.355 7fb35687c700 1 -- 172.21.2.112:0/2249739464 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3580b3760 con 0x7fb3580c3190 2021-02-10T11:32:05.347 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.355 7fb35707d700 1 --2- 172.21.2.112:0/2249739464 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb3580c36a0 0x7fb3580c2b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:32:05.347 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.355 7fb35687c700 1 --2- 172.21.2.112:0/2249739464 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb3580c3190 0x7fb3580b1d50 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 rx=0x7fb34000b5e0 tx=0x7fb34000b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:05.347 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.355 7fb34ffff700 1 -- 172.21.2.112:0/2249739464 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb34000bd70 con 0x7fb3580c3190 2021-02-10T11:32:05.347 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.355 7fb34ffff700 1 -- 172.21.2.112:0/2249739464 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb340002030 con 0x7fb3580c3190 2021-02-10T11:32:05.348 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.355 7fb35d27f700 1 -- 172.21.2.112:0/2249739464 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3580b3760 con 0x7fb3580c3190 2021-02-10T11:32:05.348 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.356 7fb35d27f700 1 -- 172.21.2.112:0/2249739464 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3580b3760 con 0x7fb3580c3190 2021-02-10T11:32:05.348 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.356 7fb34ffff700 1 -- 172.21.2.112:0/2249739464 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb340002550 con 0x7fb3580c3190 2021-02-10T11:32:05.348 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.356 7fb34ffff700 1 -- 172.21.2.112:0/2249739464 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb340000c90 con 0x7fb3580c3190 2021-02-10T11:32:05.348 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.356 7fb34ffff700 1 --2- 172.21.2.112:0/2249739464 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb33c0186d0 0x7fb33c01abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:05.348 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.356 7fb34ffff700 1 -- 172.21.2.112:0/2249739464 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fb340032830 con 0x7fb3580c3190 2021-02-10T11:32:05.348 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.357 7fb35707d700 1 --2- 172.21.2.112:0/2249739464 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb33c0186d0 0x7fb33c01abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:05.348 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.357 7fb35707d700 1 --2- 172.21.2.112:0/2249739464 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb33c0186d0 0x7fb33c01abd0 crc :-1 s=READY pgs=539 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:05.375 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.387 7fb35d27f700 1 -- 172.21.2.112:0/2249739464 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3580edd10 con 0x7fb3580c3190 2021-02-10T11:32:05.378 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.390 7fb34ffff700 1 -- 172.21.2.112:0/2249739464 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb34001f570 con 0x7fb3580c3190 2021-02-10T11:32:05.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.518 7fb35d27f700 1 -- 172.21.2.112:0/2249739464 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "target": ["mgr", ""], "group_name": "group_0000000000550574"}) v1 -- 0x7fb3580e52e0 con 0x7fb33c0186d0 2021-02-10T11:32:05.511 INFO:tasks.ceph.mgr.x.gibba020.stderr:2021-02-10 11:32:05.528 7f6382eee700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:32:05.512 INFO:tasks.ceph.mgr.x.gibba020.stderr:2021-02-10 11:32:05.528 7f6382eee700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:32:05.512 INFO:tasks.ceph.mgr.x.gibba020.stderr:2021-02-10 11:32:05.528 7f6382eee700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:32:05.512 INFO:tasks.ceph.mgr.x.gibba020.stderr:2021-02-10 11:32:05.528 7f6382eee700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:32:05.512 INFO:tasks.ceph.mgr.x.gibba020.stderr:2021-02-10 11:32:05.528 7f6382eee700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:32:05.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.537 7fb34ffff700 1 -- 172.21.2.112:0/2249739464 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fb348002f50 con 0x7fb33c0186d0 2021-02-10T11:32:05.539 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.551 7fb35d27f700 1 -- 172.21.2.112:0/2249739464 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb33c0186d0 msgr2=0x7fb33c01abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:05.539 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.551 7fb35d27f700 1 --2- 172.21.2.112:0/2249739464 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb33c0186d0 0x7fb33c01abd0 crc :-1 s=READY pgs=539 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:05.539 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.551 7fb35d27f700 1 -- 172.21.2.112:0/2249739464 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb3580c3190 msgr2=0x7fb3580b1d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:05.539 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.551 7fb35d27f700 1 --2- 172.21.2.112:0/2249739464 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb3580c3190 0x7fb3580b1d50 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 rx=0x7fb34000b5e0 tx=0x7fb34000b980).stop 2021-02-10T11:32:05.540 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.551 7fb35d27f700 1 -- 172.21.2.112:0/2249739464 shutdown_connections 2021-02-10T11:32:05.540 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.551 7fb35d27f700 1 --2- 172.21.2.112:0/2249739464 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb33c0186d0 0x7fb33c01abd0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:05.540 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.551 7fb35d27f700 1 --2- 172.21.2.112:0/2249739464 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb3580b2370 0x7fb3580b2d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:05.540 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.551 7fb35d27f700 1 --2- 172.21.2.112:0/2249739464 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb3580c3190 0x7fb3580b1d50 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:05.540 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.551 7fb35d27f700 1 --2- 172.21.2.112:0/2249739464 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb3580c36a0 0x7fb3580c2b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:05.540 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.551 7fb35d27f700 1 -- 172.21.2.112:0/2249739464 >> 172.21.2.112:0/2249739464 conn(0x7fb35809f630 msgr2=0x7fb3580a0130 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:05.540 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.551 7fb35d27f700 1 -- 172.21.2.112:0/2249739464 shutdown_connections 2021-02-10T11:32:05.540 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.551 7fb35d27f700 1 -- 172.21.2.112:0/2249739464 wait complete. 2021-02-10T11:32:05.548 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000940581 --group_name group_0000000000550574 2021-02-10T11:32:05.978 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.973 7fba69245700 1 Processor -- start 2021-02-10T11:32:05.978 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.973 7fba69245700 1 -- start start 2021-02-10T11:32:05.978 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.974 7fba69245700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fba640d1860 0x7fba641b03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:05.978 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.974 7fba69245700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fba641b0a00 0x7fba640cf320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:05.978 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.974 7fba69245700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fba640d1c60 0x7fba640cf940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:05.978 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.974 7fba69245700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba640e5ec0 con 0x7fba641b0a00 2021-02-10T11:32:05.978 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.974 7fba69245700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba640f0b00 con 0x7fba640d1860 2021-02-10T11:32:05.979 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.974 7fba69245700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba640cff60 con 0x7fba640d1c60 2021-02-10T11:32:05.979 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.974 7fba6259c700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fba641b0a00 0x7fba640cf320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:05.979 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.974 7fba6259c700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fba641b0a00 0x7fba640cf320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:48612/0 (socket says 172.21.2.112:48612) 2021-02-10T11:32:05.979 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.974 7fba6259c700 1 -- 172.21.2.112:0/3030908656 learned_addr learned my addr 172.21.2.112:0/3030908656 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:32:05.979 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.974 7fba6259c700 1 -- 172.21.2.112:0/3030908656 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fba640d1c60 msgr2=0x7fba640cf940 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:32:05.979 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.974 7fba6259c700 1 --2- 172.21.2.112:0/3030908656 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fba640d1c60 0x7fba640cf940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:05.979 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.974 7fba6259c700 1 -- 172.21.2.112:0/3030908656 >> v2:172.21.2.120:3300/0 conn(0x7fba640d1860 msgr2=0x7fba641b03e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:32:05.979 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.974 7fba6259c700 1 --2- 172.21.2.112:0/3030908656 >> v2:172.21.2.120:3300/0 conn(0x7fba640d1860 0x7fba641b03e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:05.979 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.974 7fba6259c700 1 -- 172.21.2.112:0/3030908656 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba640d0140 con 0x7fba641b0a00 2021-02-10T11:32:05.979 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.974 7fba6259c700 1 --2- 172.21.2.112:0/3030908656 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fba641b0a00 0x7fba640cf320 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 rx=0x7fba54006700 tx=0x7fba54009b80).ready entity=mon.0 client_cookie=753b21b7a55a6609 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:05.979 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.974 7fba61d9b700 1 -- 172.21.2.112:0/3030908656 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fba5400c0c0 con 0x7fba641b0a00 2021-02-10T11:32:05.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.974 7fba61d9b700 1 -- 172.21.2.112:0/3030908656 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba5400c340 con 0x7fba641b0a00 2021-02-10T11:32:05.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.974 7fba61d9b700 1 -- 172.21.2.112:0/3030908656 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fba5400c820 con 0x7fba641b0a00 2021-02-10T11:32:05.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.974 7fba69245700 1 -- 172.21.2.112:0/3030908656 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fba641b0a00 msgr2=0x7fba640cf320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:05.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.974 7fba69245700 1 --2- 172.21.2.112:0/3030908656 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fba641b0a00 0x7fba640cf320 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 rx=0x7fba54006700 tx=0x7fba54009b80).stop 2021-02-10T11:32:05.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.975 7fba69245700 1 -- 172.21.2.112:0/3030908656 shutdown_connections 2021-02-10T11:32:05.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.975 7fba69245700 1 --2- 172.21.2.112:0/3030908656 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fba640d1c60 0x7fba640cf940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:05.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.975 7fba69245700 1 --2- 172.21.2.112:0/3030908656 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fba641b0a00 0x7fba640cf320 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:05.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.975 7fba69245700 1 --2- 172.21.2.112:0/3030908656 >> v2:172.21.2.120:3300/0 conn(0x7fba640d1860 0x7fba641b03e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:05.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.975 7fba69245700 1 -- 172.21.2.112:0/3030908656 >> 172.21.2.112:0/3030908656 conn(0x7fba640ce440 msgr2=0x7fba640d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:05.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.975 7fba69245700 1 -- 172.21.2.112:0/3030908656 shutdown_connections 2021-02-10T11:32:05.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.975 7fba69245700 1 -- 172.21.2.112:0/3030908656 wait complete. 2021-02-10T11:32:05.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.990 7fba69245700 1 Processor -- start 2021-02-10T11:32:05.981 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.990 7fba69245700 1 -- start start 2021-02-10T11:32:05.982 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.990 7fba69245700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fba6418ea90 0x7fba641a3040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:05.982 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.990 7fba69245700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fba641b3f10 0x7fba640aa440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:05.982 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.990 7fba69245700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fba640aaa60 0x7fba641a1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:05.982 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.990 7fba69245700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba641a2690 con 0x7fba6418ea90 2021-02-10T11:32:05.982 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.990 7fba69245700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba641a28b0 con 0x7fba641b3f10 2021-02-10T11:32:05.982 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.990 7fba69245700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba6418efa0 con 0x7fba640aaa60 2021-02-10T11:32:05.982 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.990 7fba62d9d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fba6418ea90 0x7fba641a3040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:05.983 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.990 7fba62d9d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fba6418ea90 0x7fba641a3040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:48616/0 (socket says 172.21.2.112:48616) 2021-02-10T11:32:05.983 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.990 7fba62d9d700 1 -- 172.21.2.112:0/1667263511 learned_addr learned my addr 172.21.2.112:0/1667263511 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:32:05.983 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.991 7fba62d9d700 1 -- 172.21.2.112:0/1667263511 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fba640aaa60 msgr2=0x7fba641a1fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:05.983 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.991 7fba62d9d700 1 --2- 172.21.2.112:0/1667263511 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fba640aaa60 0x7fba641a1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:05.983 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.991 7fba62d9d700 1 -- 172.21.2.112:0/1667263511 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fba641b3f10 msgr2=0x7fba640aa440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:05.983 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.991 7fba62d9d700 1 --2- 172.21.2.112:0/1667263511 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fba641b3f10 0x7fba640aa440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:05.983 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.991 7fba62d9d700 1 -- 172.21.2.112:0/1667263511 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba6418f1c0 con 0x7fba6418ea90 2021-02-10T11:32:05.983 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.991 7fba62d9d700 1 --2- 172.21.2.112:0/1667263511 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fba6418ea90 0x7fba641a3040 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 rx=0x7fba4c00b5c0 tx=0x7fba4c00b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:05.983 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.991 7fba5b7fe700 1 -- 172.21.2.112:0/1667263511 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fba4c00bd50 con 0x7fba6418ea90 2021-02-10T11:32:05.983 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.991 7fba5b7fe700 1 -- 172.21.2.112:0/1667263511 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba4c006030 con 0x7fba6418ea90 2021-02-10T11:32:05.983 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.991 7fba5b7fe700 1 -- 172.21.2.112:0/1667263511 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fba4c006360 con 0x7fba6418ea90 2021-02-10T11:32:05.983 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.991 7fba69245700 1 -- 172.21.2.112:0/1667263511 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba6418f1c0 con 0x7fba6418ea90 2021-02-10T11:32:05.984 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.991 7fba69245700 1 -- 172.21.2.112:0/1667263511 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba6418f1c0 con 0x7fba6418ea90 2021-02-10T11:32:05.984 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.992 7fba5b7fe700 1 -- 172.21.2.112:0/1667263511 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fba4c006210 con 0x7fba6418ea90 2021-02-10T11:32:05.984 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.992 7fba5b7fe700 1 --2- 172.21.2.112:0/1667263511 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fba480186c0 0x7fba4801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:05.984 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.992 7fba5b7fe700 1 -- 172.21.2.112:0/1667263511 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fba4c0325a0 con 0x7fba6418ea90 2021-02-10T11:32:05.984 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.992 7fba6259c700 1 --2- 172.21.2.112:0/1667263511 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fba480186c0 0x7fba4801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:05.984 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:05.993 7fba6259c700 1 --2- 172.21.2.112:0/1667263511 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fba480186c0 0x7fba4801abc0 crc :-1 s=READY pgs=540 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:06.012 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:06.024 7fba69245700 1 -- 172.21.2.112:0/1667263511 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba640ebc40 con 0x7fba6418ea90 2021-02-10T11:32:06.015 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:06.027 7fba5b7fe700 1 -- 172.21.2.112:0/1667263511 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fba4c006b80 con 0x7fba6418ea90 2021-02-10T11:32:06.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:06.154 7fba69245700 1 -- 172.21.2.112:0/1667263511 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000940581", "target": ["mgr", ""], "group_name": "group_0000000000550574"}) v1 -- 0x7fba640ebc40 con 0x7fba480186c0 2021-02-10T11:32:06.165 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:06.177 7fba5b7fe700 1 -- 172.21.2.112:0/1667263511 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fba54006390 con 0x7fba480186c0 2021-02-10T11:32:06.175 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:06.186 7fba69245700 1 -- 172.21.2.112:0/1667263511 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fba480186c0 msgr2=0x7fba4801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:06.175 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:06.187 7fba69245700 1 --2- 172.21.2.112:0/1667263511 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fba480186c0 0x7fba4801abc0 crc :-1 s=READY pgs=540 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:06.175 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:06.187 7fba69245700 1 -- 172.21.2.112:0/1667263511 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fba6418ea90 msgr2=0x7fba641a3040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:06.175 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:06.187 7fba69245700 1 --2- 172.21.2.112:0/1667263511 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fba6418ea90 0x7fba641a3040 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 rx=0x7fba4c00b5c0 tx=0x7fba4c00b960).stop 2021-02-10T11:32:06.175 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:06.187 7fba69245700 1 -- 172.21.2.112:0/1667263511 shutdown_connections 2021-02-10T11:32:06.175 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:06.187 7fba69245700 1 --2- 172.21.2.112:0/1667263511 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fba480186c0 0x7fba4801abc0 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:06.175 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:06.187 7fba69245700 1 --2- 172.21.2.112:0/1667263511 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fba641b3f10 0x7fba640aa440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:06.176 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:06.187 7fba69245700 1 --2- 172.21.2.112:0/1667263511 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fba640aaa60 0x7fba641a1fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:06.176 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:06.187 7fba69245700 1 --2- 172.21.2.112:0/1667263511 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fba6418ea90 0x7fba641a3040 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:06.176 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:06.187 7fba69245700 1 -- 172.21.2.112:0/1667263511 >> 172.21.2.112:0/1667263511 conn(0x7fba641a3870 msgr2=0x7fba6418c250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:06.176 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:06.187 7fba69245700 1 -- 172.21.2.112:0/1667263511 shutdown_connections 2021-02-10T11:32:06.176 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:06.187 7fba69245700 1 -- 172.21.2.112:0/1667263511 wait complete. 2021-02-10T11:32:06.184 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume authorize cephfs subvolume_0000000000940581 guest1 --group_name group_0000000000550574 --tenant_id tenant1 --allow-existing-id 2021-02-10T11:32:06.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:06.617 7f81e217b700 1 Processor -- start 2021-02-10T11:32:06.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:06.617 7f81e217b700 1 -- start start 2021-02-10T11:32:06.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:06.618 7f81e217b700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f81dc0dce80 0x7f81dc0d14f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:06.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:06.618 7f81e217b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f81dc0eb220 0x7f81dc0eb620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:06.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:06.618 7f81e217b700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81dc10a550 con 0x7f81dc10aad0 2021-02-10T11:32:06.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:06.618 7f81e217b700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81dc0881f0 con 0x7f81dc0dce80 2021-02-10T11:32:06.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:06.618 7f81e217b700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81dc0d1f30 con 0x7f81dc0eb220 2021-02-10T11:32:06.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:06.618 7f81db7fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f81dc0dce80 0x7f81dc0d14f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:06.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:06.618 7f81dbfff700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f81dc10aad0 0x7f81dc10aed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:48090/0 (socket says 172.21.2.112:48090) 2021-02-10T11:32:06.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:06.618 7f81dbfff700 1 -- 172.21.2.112:0/2501620034 learned_addr learned my addr 172.21.2.112:0/2501620034 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:32:06.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:06.618 7f81db7fe700 1 -- 172.21.2.112:0/2501620034 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f81dc0eb220 msgr2=0x7f81dc0eb620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:06.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:06.618 7f81db7fe700 1 --2- 172.21.2.112:0/2501620034 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f81dc0eb220 0x7f81dc0eb620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:06.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:06.618 7f81db7fe700 1 -- 172.21.2.112:0/2501620034 >> v1:172.21.2.112:6789/0 conn(0x7f81dc10aad0 legacy=0x7f81dc10aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:06.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:06.618 7f81db7fe700 1 -- 172.21.2.112:0/2501620034 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81dc0dd280 con 0x7f81dc0dce80 2021-02-10T11:32:06.624 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:06.619 7f81db7fe700 1 --2- 172.21.2.112:0/2501620034 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f81dc0dce80 0x7f81dc0d14f0 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 rx=0x7f81c4006650 tx=0x7f81c4009a80).ready entity=mon.0 client_cookie=1f8d6fec7f28c1e1 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:06.624 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:06.619 7f81da7fc700 1 -- 172.21.2.112:0/2501620034 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f81c40009b0 con 0x7f81dc0dce80 2021-02-10T11:32:06.624 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:06.619 7f81da7fc700 1 -- 172.21.2.112:0/2501620034 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81c400c0c0 con 0x7f81dc0dce80 2021-02-10T11:32:06.624 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:06.619 7f81da7fc700 1 -- 172.21.2.112:0/2501620034 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f81c400c5e0 con 0x7f81dc0dce80 2021-02-10T11:32:06.624 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:06.619 7f81e217b700 1 -- 172.21.2.112:0/2501620034 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f81dc0dce80 msgr2=0x7f81dc0d14f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:06.624 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:06.619 7f81e217b700 1 --2- 172.21.2.112:0/2501620034 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f81dc0dce80 0x7f81dc0d14f0 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 rx=0x7f81c4006650 tx=0x7f81c4009a80).stop 2021-02-10T11:32:06.624 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:06.619 7f81e217b700 1 -- 172.21.2.112:0/2501620034 shutdown_connections 2021-02-10T11:32:06.624 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:06.619 7f81e217b700 1 --2- 172.21.2.112:0/2501620034 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f81dc0eb220 0x7f81dc0eb620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:06.624 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:06.619 7f81e217b700 1 --2- 172.21.2.112:0/2501620034 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f81dc0dce80 0x7f81dc0d14f0 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:06.624 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:06.619 7f81e217b700 1 -- 172.21.2.112:0/2501620034 >> 172.21.2.112:0/2501620034 conn(0x7f81dc0ea630 msgr2=0x7f81dc0d0800 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:06.624 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:06.620 7f81e217b700 1 -- 172.21.2.112:0/2501620034 shutdown_connections 2021-02-10T11:32:06.624 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:06.620 7f81e217b700 1 -- 172.21.2.112:0/2501620034 wait complete. 2021-02-10T11:32:06.624 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:06.634 7f81e217b700 1 Processor -- start 2021-02-10T11:32:06.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:06.634 7f81e217b700 1 -- start start 2021-02-10T11:32:06.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:06.635 7f81e217b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f81dc0b3f30 0x7f81dc0b4330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:06.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:06.635 7f81e217b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f81dc0ff040 0x7f81dc0ff440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:06.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:06.635 7f81e217b700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f81dc0ffa60 0x7f81dc100880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:06.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:06.635 7f81e217b700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81dc100300 con 0x7f81dc0ff040 2021-02-10T11:32:06.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:06.635 7f81e217b700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81dc191b60 con 0x7f81dc0ffa60 2021-02-10T11:32:06.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:06.635 7f81e217b700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81dc197b60 con 0x7f81dc0b3f30 2021-02-10T11:32:06.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:06.635 7f81daffd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f81dc0ff040 0x7f81dc0ff440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:06.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:06.635 7f81daffd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f81dc0ff040 0x7f81dc0ff440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:48634/0 (socket says 172.21.2.112:48634) 2021-02-10T11:32:06.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:06.635 7f81daffd700 1 -- 172.21.2.112:0/3244595944 learned_addr learned my addr 172.21.2.112:0/3244595944 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:32:06.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:06.635 7f81daffd700 1 -- 172.21.2.112:0/3244595944 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f81dc0b3f30 msgr2=0x7f81dc0b4330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:06.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:06.635 7f81daffd700 1 --2- 172.21.2.112:0/3244595944 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f81dc0b3f30 0x7f81dc0b4330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:06.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:06.635 7f81daffd700 1 -- 172.21.2.112:0/3244595944 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f81dc0ffa60 msgr2=0x7f81dc100880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:06.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:06.635 7f81daffd700 1 --2- 172.21.2.112:0/3244595944 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f81dc0ffa60 0x7f81dc100880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:06.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:06.635 7f81daffd700 1 -- 172.21.2.112:0/3244595944 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81dc198020 con 0x7f81dc0ff040 2021-02-10T11:32:06.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:06.635 7f81daffd700 1 --2- 172.21.2.112:0/3244595944 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f81dc0ff040 0x7f81dc0ff440 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 rx=0x7f81cc006f00 tx=0x7f81cc00b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:06.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:06.636 7f81d3fff700 1 -- 172.21.2.112:0/3244595944 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f81cc006b90 con 0x7f81dc0ff040 2021-02-10T11:32:06.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:06.636 7f81d3fff700 1 -- 172.21.2.112:0/3244595944 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81cc00bdf0 con 0x7f81dc0ff040 2021-02-10T11:32:06.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:06.636 7f81e217b700 1 -- 172.21.2.112:0/3244595944 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81dc198020 con 0x7f81dc0ff040 2021-02-10T11:32:06.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:06.636 7f81d3fff700 1 -- 172.21.2.112:0/3244595944 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f81cc002020 con 0x7f81dc0ff040 2021-02-10T11:32:06.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:06.636 7f81e217b700 1 -- 172.21.2.112:0/3244595944 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81dc100ea0 con 0x7f81dc0ff040 2021-02-10T11:32:06.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:06.636 7f81d3fff700 1 -- 172.21.2.112:0/3244595944 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f81cc005310 con 0x7f81dc0ff040 2021-02-10T11:32:06.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:06.637 7f81d3fff700 1 --2- 172.21.2.112:0/3244595944 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f81c00186c0 0x7f81c001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:06.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:06.637 7f81d3fff700 1 -- 172.21.2.112:0/3244595944 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f81cc032570 con 0x7f81dc0ff040 2021-02-10T11:32:06.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:06.637 7f81db7fe700 1 --2- 172.21.2.112:0/3244595944 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f81c00186c0 0x7f81c001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:06.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:06.637 7f81db7fe700 1 --2- 172.21.2.112:0/3244595944 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f81c00186c0 0x7f81c001abc0 crc :-1 s=READY pgs=541 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:06.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:06.668 7f81e217b700 1 -- 172.21.2.112:0/3244595944 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81dc10a550 con 0x7f81dc0ff040 2021-02-10T11:32:06.658 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:06.670 7f81d3fff700 1 -- 172.21.2.112:0/3244595944 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f81cc000960 con 0x7f81dc0ff040 2021-02-10T11:32:06.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:06.795 7f81e217b700 1 -- 172.21.2.112:0/3244595944 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"allow_existing_id": true, "vol_name": "cephfs", "sub_name": "subvolume_0000000000940581", "target": ["mgr", ""], "tenant_id": "tenant1", "group_name": "group_0000000000550574", "prefix": "fs subvolume authorize", "auth_id": "guest1"}) v1 -- 0x7f81dc100ea0 con 0x7f81c00186c0 2021-02-10T11:32:06.789 INFO:tasks.ceph.mgr.x.gibba020.stderr:Creating meta for ID guest1 with tenant tenant1 2021-02-10T11:32:06.819 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:06.832 7f81d3fff700 1 -- 172.21.2.112:0/3244595944 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+41 (crc 0 0 0) 0x7f81c4006040 con 0x7f81c00186c0 2021-02-10T11:32:06.847 INFO:teuthology.orchestra.run.gibba012.stdout:AQA0xCNg3bEjNhAA9hi8UJZVHetJC//dAy6vzw== 2021-02-10T11:32:06.849 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:06.861 7f81e217b700 1 -- 172.21.2.112:0/3244595944 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f81c00186c0 msgr2=0x7f81c001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:06.849 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:06.861 7f81e217b700 1 --2- 172.21.2.112:0/3244595944 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f81c00186c0 0x7f81c001abc0 crc :-1 s=READY pgs=541 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:06.849 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:06.861 7f81e217b700 1 -- 172.21.2.112:0/3244595944 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f81dc0ff040 msgr2=0x7f81dc0ff440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:06.849 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:06.861 7f81e217b700 1 --2- 172.21.2.112:0/3244595944 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f81dc0ff040 0x7f81dc0ff440 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 rx=0x7f81cc006f00 tx=0x7f81cc00b940).stop 2021-02-10T11:32:06.849 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:06.861 7f81e217b700 1 -- 172.21.2.112:0/3244595944 shutdown_connections 2021-02-10T11:32:06.850 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:06.861 7f81e217b700 1 --2- 172.21.2.112:0/3244595944 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f81c00186c0 0x7f81c001abc0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:06.850 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:06.861 7f81e217b700 1 --2- 172.21.2.112:0/3244595944 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f81dc0ffa60 0x7f81dc100880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:06.850 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:06.861 7f81e217b700 1 --2- 172.21.2.112:0/3244595944 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f81dc0ff040 0x7f81dc0ff440 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:06.850 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:06.861 7f81e217b700 1 --2- 172.21.2.112:0/3244595944 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f81dc0b3f30 0x7f81dc0b4330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:06.850 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:06.861 7f81e217b700 1 -- 172.21.2.112:0/3244595944 >> 172.21.2.112:0/3244595944 conn(0x7f81dc08d660 msgr2=0x7f81dc1ab150 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:06.850 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:06.861 7f81e217b700 1 -- 172.21.2.112:0/3244595944 shutdown_connections 2021-02-10T11:32:06.850 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:06.861 7f81e217b700 1 -- 172.21.2.112:0/3244595944 wait complete. 2021-02-10T11:32:06.857 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume deauthorize cephfs subvolume_0000000000940581 guest1 --group_name group_0000000000550574 2021-02-10T11:32:07.287 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.283 7fc883c44700 1 Processor -- start 2021-02-10T11:32:07.287 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.283 7fc883c44700 1 -- start start 2021-02-10T11:32:07.288 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.283 7fc883c44700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fc87c0d1860 0x7fc87c1b04b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:07.288 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.283 7fc883c44700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc87c1b0ad0 0x7fc87c108930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:07.288 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.283 7fc883c44700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc87c0fe1b0 con 0x7fc87c0d1c60 2021-02-10T11:32:07.288 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.283 7fc883c44700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc87c109a50 con 0x7fc87c0d1860 2021-02-10T11:32:07.288 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.283 7fc883c44700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc87c109740 con 0x7fc87c1b0ad0 2021-02-10T11:32:07.288 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.283 7fc881a60700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fc87c0d1860 0x7fc87c1b04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:07.288 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.283 7fc881a60700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fc87c0d1860 0x7fc87c1b04b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:48638/0 (socket says 172.21.2.112:48638) 2021-02-10T11:32:07.288 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.283 7fc881a60700 1 -- 172.21.2.112:0/972550640 learned_addr learned my addr 172.21.2.112:0/972550640 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:32:07.288 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.284 7fc881a60700 1 -- 172.21.2.112:0/972550640 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc87c1b0ad0 msgr2=0x7fc87c108930 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:32:07.288 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.284 7fc881a60700 1 --2- 172.21.2.112:0/972550640 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc87c1b0ad0 0x7fc87c108930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:07.288 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.284 7fc881a60700 1 -- 172.21.2.112:0/972550640 >> v1:172.21.2.120:6789/0 conn(0x7fc87c0d1c60 legacy=0x7fc87c108f50 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:32:07.288 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.284 7fc881a60700 1 -- 172.21.2.112:0/972550640 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc87c088020 con 0x7fc87c0d1860 2021-02-10T11:32:07.289 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.284 7fc881a60700 1 --2- 172.21.2.112:0/972550640 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc87c0d1860 0x7fc87c1b04b0 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 rx=0x7fc870006700 tx=0x7fc870009b00).ready entity=mon.0 client_cookie=ddd5054c214c5cc server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:07.289 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.284 7fc880a5e700 1 -- 172.21.2.112:0/972550640 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc87000c0c0 con 0x7fc87c0d1860 2021-02-10T11:32:07.289 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.284 7fc880a5e700 1 -- 172.21.2.112:0/972550640 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc87000c340 con 0x7fc87c0d1860 2021-02-10T11:32:07.289 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.284 7fc880a5e700 1 -- 172.21.2.112:0/972550640 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc87000c820 con 0x7fc87c0d1860 2021-02-10T11:32:07.289 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.284 7fc883c44700 1 -- 172.21.2.112:0/972550640 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc87c0d1860 msgr2=0x7fc87c1b04b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:07.289 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.284 7fc883c44700 1 --2- 172.21.2.112:0/972550640 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc87c0d1860 0x7fc87c1b04b0 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 rx=0x7fc870006700 tx=0x7fc870009b00).stop 2021-02-10T11:32:07.289 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.285 7fc883c44700 1 -- 172.21.2.112:0/972550640 shutdown_connections 2021-02-10T11:32:07.289 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.285 7fc883c44700 1 --2- 172.21.2.112:0/972550640 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc87c1b0ad0 0x7fc87c108930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:07.289 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.285 7fc883c44700 1 --2- 172.21.2.112:0/972550640 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc87c0d1860 0x7fc87c1b04b0 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:07.289 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.285 7fc883c44700 1 -- 172.21.2.112:0/972550640 >> 172.21.2.112:0/972550640 conn(0x7fc87c0ce410 msgr2=0x7fc87c0d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:07.289 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.285 7fc883c44700 1 -- 172.21.2.112:0/972550640 shutdown_connections 2021-02-10T11:32:07.289 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.285 7fc883c44700 1 -- 172.21.2.112:0/972550640 wait complete. 2021-02-10T11:32:07.291 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.300 7fc883c44700 1 Processor -- start 2021-02-10T11:32:07.291 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.300 7fc883c44700 1 -- start start 2021-02-10T11:32:07.291 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.300 7fc883c44700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc87c19e640 0x7fc87c1a2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:07.291 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.300 7fc883c44700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc87c1a3120 0x7fc87c0b33b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:07.291 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.300 7fc883c44700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc87c0b39d0 0x7fc87c193450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:07.291 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.300 7fc883c44700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc87c19eac0 con 0x7fc87c0b39d0 2021-02-10T11:32:07.291 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.300 7fc883c44700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc87c1a3520 con 0x7fc87c19e640 2021-02-10T11:32:07.291 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.300 7fc883c44700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc87c1a3740 con 0x7fc87c1a3120 2021-02-10T11:32:07.292 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.300 7fc882261700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc87c0b39d0 0x7fc87c193450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:07.292 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.300 7fc882261700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc87c0b39d0 0x7fc87c193450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:48646/0 (socket says 172.21.2.112:48646) 2021-02-10T11:32:07.292 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.300 7fc882261700 1 -- 172.21.2.112:0/2479385491 learned_addr learned my addr 172.21.2.112:0/2479385491 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:32:07.292 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.301 7fc882261700 1 -- 172.21.2.112:0/2479385491 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc87c1a3120 msgr2=0x7fc87c0b33b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:07.292 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.301 7fc882261700 1 --2- 172.21.2.112:0/2479385491 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc87c1a3120 0x7fc87c0b33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:07.292 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.301 7fc882261700 1 -- 172.21.2.112:0/2479385491 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc87c19e640 msgr2=0x7fc87c1a2b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:07.292 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.301 7fc881a60700 1 --2- 172.21.2.112:0/2479385491 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc87c19e640 0x7fc87c1a2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:07.292 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.301 7fc882261700 1 --2- 172.21.2.112:0/2479385491 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc87c19e640 0x7fc87c1a2b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:07.292 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.301 7fc882261700 1 -- 172.21.2.112:0/2479385491 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc87c0b3dd0 con 0x7fc87c0b39d0 2021-02-10T11:32:07.292 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.301 7fc881a60700 1 --2- 172.21.2.112:0/2479385491 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc87c19e640 0x7fc87c1a2b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:32:07.292 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.301 7fc882261700 1 --2- 172.21.2.112:0/2479385491 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc87c0b39d0 0x7fc87c193450 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 rx=0x7fc87400b8e0 tx=0x7fc87400bc80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:07.292 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.301 7fc86e7fc700 1 -- 172.21.2.112:0/2479385491 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc874003030 con 0x7fc87c0b39d0 2021-02-10T11:32:07.292 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.301 7fc86e7fc700 1 -- 172.21.2.112:0/2479385491 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc874001270 con 0x7fc87c0b39d0 2021-02-10T11:32:07.292 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.301 7fc883c44700 1 -- 172.21.2.112:0/2479385491 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc87c0b3dd0 con 0x7fc87c0b39d0 2021-02-10T11:32:07.293 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.301 7fc86e7fc700 1 -- 172.21.2.112:0/2479385491 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc874001750 con 0x7fc87c0b39d0 2021-02-10T11:32:07.293 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.301 7fc883c44700 1 -- 172.21.2.112:0/2479385491 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc87c0b3dd0 con 0x7fc87c0b39d0 2021-02-10T11:32:07.293 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.302 7fc86e7fc700 1 -- 172.21.2.112:0/2479385491 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc874001450 con 0x7fc87c0b39d0 2021-02-10T11:32:07.293 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.302 7fc86e7fc700 1 --2- 172.21.2.112:0/2479385491 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc864018730 0x7fc86401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:07.293 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.302 7fc86e7fc700 1 -- 172.21.2.112:0/2479385491 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fc874032a30 con 0x7fc87c0b39d0 2021-02-10T11:32:07.293 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.302 7fc881a60700 1 --2- 172.21.2.112:0/2479385491 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc864018730 0x7fc86401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:07.293 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.303 7fc881a60700 1 --2- 172.21.2.112:0/2479385491 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc864018730 0x7fc86401ac10 crc :-1 s=READY pgs=542 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:07.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.333 7fc883c44700 1 -- 172.21.2.112:0/2479385491 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc87c07e440 con 0x7fc87c0b39d0 2021-02-10T11:32:07.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.335 7fc86e7fc700 1 -- 172.21.2.112:0/2479385491 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc87400bdc0 con 0x7fc87c0b39d0 2021-02-10T11:32:07.451 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.462 7fc883c44700 1 -- 172.21.2.112:0/2479385491 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume deauthorize", "vol_name": "cephfs", "sub_name": "subvolume_0000000000940581", "target": ["mgr", ""], "auth_id": "guest1", "group_name": "group_0000000000550574"}) v1 -- 0x7fc87c07e440 con 0x7fc864018730 2021-02-10T11:32:07.485 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.498 7fc86e7fc700 1 -- 172.21.2.112:0/2479385491 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fc870006ad0 con 0x7fc864018730 2021-02-10T11:32:07.517 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.528 7fc883c44700 1 -- 172.21.2.112:0/2479385491 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc864018730 msgr2=0x7fc86401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:07.517 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.528 7fc883c44700 1 --2- 172.21.2.112:0/2479385491 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc864018730 0x7fc86401ac10 crc :-1 s=READY pgs=542 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:07.517 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.528 7fc883c44700 1 -- 172.21.2.112:0/2479385491 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc87c0b39d0 msgr2=0x7fc87c193450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:07.517 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.528 7fc883c44700 1 --2- 172.21.2.112:0/2479385491 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc87c0b39d0 0x7fc87c193450 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 rx=0x7fc87400b8e0 tx=0x7fc87400bc80).stop 2021-02-10T11:32:07.517 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.528 7fc883c44700 1 -- 172.21.2.112:0/2479385491 shutdown_connections 2021-02-10T11:32:07.517 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.528 7fc883c44700 1 --2- 172.21.2.112:0/2479385491 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc864018730 0x7fc86401ac10 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:07.517 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.528 7fc883c44700 1 --2- 172.21.2.112:0/2479385491 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc87c0b39d0 0x7fc87c193450 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:07.517 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.528 7fc883c44700 1 --2- 172.21.2.112:0/2479385491 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc87c1a3120 0x7fc87c0b33b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:07.518 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.528 7fc883c44700 1 --2- 172.21.2.112:0/2479385491 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc87c19e640 0x7fc87c1a2b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:07.518 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.528 7fc883c44700 1 -- 172.21.2.112:0/2479385491 >> 172.21.2.112:0/2479385491 conn(0x7fc87c19f280 msgr2=0x7fc87c097a20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:07.518 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.528 7fc883c44700 1 -- 172.21.2.112:0/2479385491 shutdown_connections 2021-02-10T11:32:07.518 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.528 7fc883c44700 1 -- 172.21.2.112:0/2479385491 wait complete. 2021-02-10T11:32:07.525 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth rm client.guest1 2021-02-10T11:32:07.949 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.944 7f6880a78700 1 Processor -- start 2021-02-10T11:32:07.949 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.944 7f6880a78700 1 -- start start 2021-02-10T11:32:07.949 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.945 7f6880a78700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f687c07e6a0 0x7f687c0d1790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:07.949 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.945 7f6880a78700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f687c07ed80 con 0x7f687c0e5470 2021-02-10T11:32:07.950 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.945 7f6880a78700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f687c07eaa0 con 0x7f687c0e5070 2021-02-10T11:32:07.950 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.945 7f6880a78700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f687c1b4080 con 0x7f687c07e6a0 2021-02-10T11:32:07.950 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.945 7f687a05d700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f687c0e5470 0x7f687c07f300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:48114/0 (socket says 172.21.2.112:48114) 2021-02-10T11:32:07.950 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.945 7f687a05d700 1 -- 172.21.2.112:0/2078809208 learned_addr learned my addr 172.21.2.112:0/2078809208 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:32:07.950 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.945 7f687b05f700 1 --2- 172.21.2.112:0/2078809208 >> v2:172.21.2.112:3300/0 conn(0x7f687c07e6a0 0x7f687c0d1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:07.950 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.945 7f687985c700 1 -- 172.21.2.112:0/2078809208 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3475704543 0 0) 0x7f68700010b0 con 0x7f687c0e5470 2021-02-10T11:32:07.950 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.945 7f687985c700 1 -- 172.21.2.112:0/2078809208 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6864002fe0 con 0x7f687c0e5470 2021-02-10T11:32:07.950 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.945 7f687b05f700 1 -- 172.21.2.112:0/2078809208 >> v1:172.21.2.120:6789/0 conn(0x7f687c0e5070 legacy=0x7f687c1b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:07.950 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.945 7f687b05f700 1 -- 172.21.2.112:0/2078809208 >> v1:172.21.2.112:6789/0 conn(0x7f687c0e5470 legacy=0x7f687c07f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:07.950 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.945 7f687b05f700 1 -- 172.21.2.112:0/2078809208 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f687c0d1db0 con 0x7f687c07e6a0 2021-02-10T11:32:07.950 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.945 7f687b05f700 1 --2- 172.21.2.112:0/2078809208 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f687c07e6a0 0x7f687c0d1790 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 rx=0x7f6874006e30 tx=0x7f6874009e60).ready entity=mon.0 client_cookie=6bb0a35184a3a0c4 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:07.950 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.946 7f687985c700 1 -- 172.21.2.112:0/2078809208 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6874006a50 con 0x7f687c07e6a0 2021-02-10T11:32:07.950 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.946 7f687985c700 1 -- 172.21.2.112:0/2078809208 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f687400b300 con 0x7f687c07e6a0 2021-02-10T11:32:07.951 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.946 7f687985c700 1 -- 172.21.2.112:0/2078809208 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f687400b7e0 con 0x7f687c07e6a0 2021-02-10T11:32:07.951 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.946 7f6880a78700 1 -- 172.21.2.112:0/2078809208 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f687c07e6a0 msgr2=0x7f687c0d1790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:07.951 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.946 7f6880a78700 1 --2- 172.21.2.112:0/2078809208 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f687c07e6a0 0x7f687c0d1790 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 rx=0x7f6874006e30 tx=0x7f6874009e60).stop 2021-02-10T11:32:07.951 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.947 7f6880a78700 1 -- 172.21.2.112:0/2078809208 shutdown_connections 2021-02-10T11:32:07.951 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.947 7f6880a78700 1 --2- 172.21.2.112:0/2078809208 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f687c07e6a0 0x7f687c0d1790 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:07.951 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.947 7f6880a78700 1 -- 172.21.2.112:0/2078809208 >> 172.21.2.112:0/2078809208 conn(0x7f687c0ce690 msgr2=0x7f687c0e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:07.951 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.947 7f6880a78700 1 -- 172.21.2.112:0/2078809208 shutdown_connections 2021-02-10T11:32:07.951 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.947 7f6880a78700 1 -- 172.21.2.112:0/2078809208 wait complete. 2021-02-10T11:32:07.952 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.961 7f6880a78700 1 Processor -- start 2021-02-10T11:32:07.952 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.961 7f6880a78700 1 -- start start 2021-02-10T11:32:07.953 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.962 7f6880a78700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f687c0a2b70 0x7f687c0c0590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:07.953 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.962 7f6880a78700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f687c0c0bb0 0x7f687c0a3850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:07.953 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.962 7f6880a78700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f687c0c0fb0 0x7f687c0a3e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:07.953 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.962 7f6880a78700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f687c0a2f70 con 0x7f687c0c0fb0 2021-02-10T11:32:07.953 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.962 7f6880a78700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f687c0a1a40 con 0x7f687c0c0bb0 2021-02-10T11:32:07.953 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.962 7f6880a78700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f687c0a1c60 con 0x7f687c0a2b70 2021-02-10T11:32:07.953 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.962 7f687b05f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f687c0c0fb0 0x7f687c0a3e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:07.953 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.962 7f687b05f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f687c0c0fb0 0x7f687c0a3e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:48660/0 (socket says 172.21.2.112:48660) 2021-02-10T11:32:07.953 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.962 7f687b05f700 1 -- 172.21.2.112:0/126190461 learned_addr learned my addr 172.21.2.112:0/126190461 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:32:07.953 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.963 7f687b05f700 1 -- 172.21.2.112:0/126190461 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f687c0a2b70 msgr2=0x7f687c0c0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:07.953 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.963 7f687b05f700 1 --2- 172.21.2.112:0/126190461 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f687c0a2b70 0x7f687c0c0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:07.953 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.963 7f687b05f700 1 -- 172.21.2.112:0/126190461 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f687c0c0bb0 msgr2=0x7f687c0a3850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:07.954 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.963 7f687b05f700 1 --2- 172.21.2.112:0/126190461 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f687c0c0bb0 0x7f687c0a3850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:07.954 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.963 7f687b05f700 1 -- 172.21.2.112:0/126190461 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f687c0a4490 con 0x7f687c0c0fb0 2021-02-10T11:32:07.954 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.963 7f687b05f700 1 --2- 172.21.2.112:0/126190461 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f687c0c0fb0 0x7f687c0a3e70 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 rx=0x7f6874006ab0 tx=0x7f687400bd60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:07.954 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.963 7f686b7fe700 1 -- 172.21.2.112:0/126190461 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6874009070 con 0x7f687c0c0fb0 2021-02-10T11:32:07.954 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.963 7f686b7fe700 1 -- 172.21.2.112:0/126190461 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6874001270 con 0x7f687c0c0fb0 2021-02-10T11:32:07.954 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.963 7f686b7fe700 1 -- 172.21.2.112:0/126190461 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6874001790 con 0x7f687c0c0fb0 2021-02-10T11:32:07.954 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.963 7f6880a78700 1 -- 172.21.2.112:0/126190461 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f687c0a4490 con 0x7f687c0c0fb0 2021-02-10T11:32:07.954 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.963 7f6880a78700 1 -- 172.21.2.112:0/126190461 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f687c0a4a20 con 0x7f687c0c0fb0 2021-02-10T11:32:07.954 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.964 7f686b7fe700 1 -- 172.21.2.112:0/126190461 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6874001450 con 0x7f687c0c0fb0 2021-02-10T11:32:07.954 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.964 7f686b7fe700 1 --2- 172.21.2.112:0/126190461 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f68600186c0 0x7f686001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:07.954 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.964 7f686b7fe700 1 -- 172.21.2.112:0/126190461 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f6874032890 con 0x7f687c0c0fb0 2021-02-10T11:32:07.955 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.964 7f687a85e700 1 --2- 172.21.2.112:0/126190461 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f68600186c0 0x7f686001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:07.955 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.965 7f687a85e700 1 --2- 172.21.2.112:0/126190461 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f68600186c0 0x7f686001abc0 crc :-1 s=READY pgs=543 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:07.982 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.995 7f6880a78700 1 -- 172.21.2.112:0/126190461 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f687c083da0 con 0x7f687c0c0fb0 2021-02-10T11:32:07.985 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:07.997 7f686b7fe700 1 -- 172.21.2.112:0/126190461 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f68740093d0 con 0x7f687c0c0fb0 2021-02-10T11:32:08.116 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:08.128 7f6880a78700 1 -- 172.21.2.112:0/126190461 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "auth rm", "entity": "client.guest1"} v 0) v1 -- 0x7f687c07ed80 con 0x7f687c0c0fb0 2021-02-10T11:32:08.119 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:08.131 7f686b7fe700 1 -- 172.21.2.112:0/126190461 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth rm", "entity": "client.guest1"}]=0 updated v24) v1 ==== 89+0+0 (secure 0 0 0) 0x7f6874001450 con 0x7f687c0c0fb0 2021-02-10T11:32:08.123 INFO:teuthology.orchestra.run.gibba012.stderr:updated 2021-02-10T11:32:08.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:08.136 7f6880a78700 1 -- 172.21.2.112:0/126190461 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f68600186c0 msgr2=0x7f686001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:08.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:08.136 7f6880a78700 1 --2- 172.21.2.112:0/126190461 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f68600186c0 0x7f686001abc0 crc :-1 s=READY pgs=543 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:08.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:08.137 7f6880a78700 1 -- 172.21.2.112:0/126190461 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f687c0c0fb0 msgr2=0x7f687c0a3e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:08.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:08.137 7f6880a78700 1 --2- 172.21.2.112:0/126190461 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f687c0c0fb0 0x7f687c0a3e70 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 rx=0x7f6874006ab0 tx=0x7f687400bd60).stop 2021-02-10T11:32:08.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:08.137 7f6880a78700 1 -- 172.21.2.112:0/126190461 shutdown_connections 2021-02-10T11:32:08.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:08.137 7f6880a78700 1 --2- 172.21.2.112:0/126190461 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f68600186c0 0x7f686001abc0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:08.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:08.137 7f6880a78700 1 --2- 172.21.2.112:0/126190461 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f687c0c0bb0 0x7f687c0a3850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:08.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:08.137 7f6880a78700 1 --2- 172.21.2.112:0/126190461 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f687c0c0fb0 0x7f687c0a3e70 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:08.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:08.137 7f6880a78700 1 --2- 172.21.2.112:0/126190461 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f687c0a2b70 0x7f687c0c0590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:08.126 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:08.137 7f6880a78700 1 -- 172.21.2.112:0/126190461 >> 172.21.2.112:0/126190461 conn(0x7f687c0bde70 msgr2=0x7f687c0c3970 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:08.126 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:08.137 7f6880a78700 1 -- 172.21.2.112:0/126190461 shutdown_connections 2021-02-10T11:32:08.126 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:08.137 7f6880a78700 1 -- 172.21.2.112:0/126190461 wait complete. 2021-02-10T11:32:08.133 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000940581 --group_name group_0000000000550574 2021-02-10T11:32:08.565 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:08.559 7fe093e41700 1 Processor -- start 2021-02-10T11:32:08.565 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:08.559 7fe093e41700 1 -- start start 2021-02-10T11:32:08.566 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:08.559 7fe093e41700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fe08c0d1860 0x7fe08c1b03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:08.566 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:08.559 7fe093e41700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe08c1b0a00 0x7fe08c0cf320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:08.566 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:08.559 7fe093e41700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe08c0f0b00 con 0x7fe08c0d1c60 2021-02-10T11:32:08.566 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:08.559 7fe093e41700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe08c0e5ec0 con 0x7fe08c0d1860 2021-02-10T11:32:08.566 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:08.559 7fe093e41700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe08c0f0640 con 0x7fe08c1b0a00 2021-02-10T11:32:08.566 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:08.562 7fe091c5d700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fe08c0d1860 0x7fe08c1b03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:08.566 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:08.562 7fe09245e700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7fe08c0d1c60 0x7fe08c0cf940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:41684/0 (socket says 172.21.2.112:41684) 2021-02-10T11:32:08.566 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:08.562 7fe09145c700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe08c1b0a00 0x7fe08c0cf320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:08.566 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:08.562 7fe09245e700 1 -- 172.21.2.112:0/2805414606 learned_addr learned my addr 172.21.2.112:0/2805414606 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:32:08.566 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:08.562 7fe09145c700 1 -- 172.21.2.112:0/2805414606 >> v2:172.21.2.120:3300/0 conn(0x7fe08c0d1860 msgr2=0x7fe08c1b03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:08.566 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:08.562 7fe09145c700 1 --2- 172.21.2.112:0/2805414606 >> v2:172.21.2.120:3300/0 conn(0x7fe08c0d1860 0x7fe08c1b03e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:08.567 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:08.562 7fe09145c700 1 -- 172.21.2.112:0/2805414606 >> v1:172.21.2.120:6789/0 conn(0x7fe08c0d1c60 legacy=0x7fe08c0cf940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:08.567 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:08.562 7fe090c5b700 1 -- 172.21.2.112:0/2805414606 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2538751590 0 0) 0x7fe084001440 con 0x7fe08c0d1c60 2021-02-10T11:32:08.567 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:08.562 7fe09145c700 1 -- 172.21.2.112:0/2805414606 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe08c0f0860 con 0x7fe08c1b0a00 2021-02-10T11:32:08.567 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:08.562 7fe091c5d700 1 --2- 172.21.2.112:0/2805414606 >> v2:172.21.2.120:3300/0 conn(0x7fe08c0d1860 0x7fe08c1b03e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:32:08.567 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:08.562 7fe09145c700 1 --2- 172.21.2.112:0/2805414606 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe08c1b0a00 0x7fe08c0cf320 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7fe088006650 tx=0x7fe088009a80).ready entity=mon.2 client_cookie=ceec9279df5a2a02 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:08.567 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:08.562 7fe090c5b700 1 -- 172.21.2.112:0/2805414606 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe0880009b0 con 0x7fe08c1b0a00 2021-02-10T11:32:08.567 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:08.562 7fe090c5b700 1 -- 172.21.2.112:0/2805414606 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe08800c0c0 con 0x7fe08c1b0a00 2021-02-10T11:32:08.567 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:08.562 7fe090c5b700 1 -- 172.21.2.112:0/2805414606 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe08800c510 con 0x7fe08c1b0a00 2021-02-10T11:32:08.567 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:08.562 7fe093e41700 1 -- 172.21.2.112:0/2805414606 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe08c1b0a00 msgr2=0x7fe08c0cf320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:08.567 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:08.562 7fe093e41700 1 --2- 172.21.2.112:0/2805414606 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe08c1b0a00 0x7fe08c0cf320 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7fe088006650 tx=0x7fe088009a80).stop 2021-02-10T11:32:08.567 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:08.563 7fe093e41700 1 -- 172.21.2.112:0/2805414606 shutdown_connections 2021-02-10T11:32:08.567 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:08.563 7fe093e41700 1 --2- 172.21.2.112:0/2805414606 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe08c1b0a00 0x7fe08c0cf320 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:08.567 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:08.563 7fe093e41700 1 --2- 172.21.2.112:0/2805414606 >> v2:172.21.2.120:3300/0 conn(0x7fe08c0d1860 0x7fe08c1b03e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:08.568 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:08.563 7fe093e41700 1 -- 172.21.2.112:0/2805414606 >> 172.21.2.112:0/2805414606 conn(0x7fe08c0ce440 msgr2=0x7fe08c0d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:08.568 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:08.563 7fe093e41700 1 -- 172.21.2.112:0/2805414606 shutdown_connections 2021-02-10T11:32:08.568 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:08.563 7fe093e41700 1 -- 172.21.2.112:0/2805414606 wait complete. 2021-02-10T11:32:08.569 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:08.578 7fe093e41700 1 Processor -- start 2021-02-10T11:32:08.569 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:08.578 7fe093e41700 1 -- start start 2021-02-10T11:32:08.569 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:08.578 7fe093e41700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe08c1a49c0 0x7fe08c099c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:08.569 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:08.578 7fe093e41700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe08c09a220 0x7fe08c09a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:08.570 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:08.578 7fe093e41700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe08c08a460 0x7fe08c08a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:08.570 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:08.578 7fe093e41700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe08c08e5b0 con 0x7fe08c09a220 2021-02-10T11:32:08.570 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:08.578 7fe093e41700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe08c0a9090 con 0x7fe08c08a460 2021-02-10T11:32:08.570 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:08.578 7fe093e41700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe08c08ae80 con 0x7fe08c1a49c0 2021-02-10T11:32:08.570 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:08.578 7fe09145c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe08c09a220 0x7fe08c09a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:08.570 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:08.578 7fe09145c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe08c09a220 0x7fe08c09a620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:48672/0 (socket says 172.21.2.112:48672) 2021-02-10T11:32:08.570 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:08.578 7fe09145c700 1 -- 172.21.2.112:0/408061271 learned_addr learned my addr 172.21.2.112:0/408061271 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:32:08.570 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:08.579 7fe09145c700 1 -- 172.21.2.112:0/408061271 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe08c1a49c0 msgr2=0x7fe08c099c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:08.570 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:08.579 7fe09145c700 1 --2- 172.21.2.112:0/408061271 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe08c1a49c0 0x7fe08c099c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:08.570 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:08.579 7fe09245e700 1 --2- 172.21.2.112:0/408061271 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe08c08a460 0x7fe08c08a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:08.570 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:08.579 7fe09145c700 1 -- 172.21.2.112:0/408061271 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe08c08a460 msgr2=0x7fe08c08a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:08.570 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:08.579 7fe09145c700 1 --2- 172.21.2.112:0/408061271 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe08c08a460 0x7fe08c08a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:08.570 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:08.579 7fe09145c700 1 -- 172.21.2.112:0/408061271 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe08c08b0a0 con 0x7fe08c09a220 2021-02-10T11:32:08.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:08.579 7fe09245e700 1 --2- 172.21.2.112:0/408061271 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe08c08a460 0x7fe08c08a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:32:08.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:08.579 7fe09145c700 1 --2- 172.21.2.112:0/408061271 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe08c09a220 0x7fe08c09a620 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 rx=0x7fe088009630 tx=0x7fe08800b900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:08.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:08.579 7fe0827fc700 1 -- 172.21.2.112:0/408061271 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe0880093a0 con 0x7fe08c09a220 2021-02-10T11:32:08.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:08.579 7fe0827fc700 1 -- 172.21.2.112:0/408061271 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe08800bcf0 con 0x7fe08c09a220 2021-02-10T11:32:08.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:08.579 7fe0827fc700 1 -- 172.21.2.112:0/408061271 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe088005560 con 0x7fe08c09a220 2021-02-10T11:32:08.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:08.579 7fe093e41700 1 -- 172.21.2.112:0/408061271 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe08c08b0a0 con 0x7fe08c09a220 2021-02-10T11:32:08.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:08.579 7fe093e41700 1 -- 172.21.2.112:0/408061271 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe08c08b0a0 con 0x7fe08c09a220 2021-02-10T11:32:08.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:08.580 7fe0827fc700 1 -- 172.21.2.112:0/408061271 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe088008240 con 0x7fe08c09a220 2021-02-10T11:32:08.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:08.580 7fe0827fc700 1 --2- 172.21.2.112:0/408061271 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe0740186c0 0x7fe07401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:08.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:08.580 7fe0827fc700 1 -- 172.21.2.112:0/408061271 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fe088032520 con 0x7fe08c09a220 2021-02-10T11:32:08.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:08.580 7fe091c5d700 1 --2- 172.21.2.112:0/408061271 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe0740186c0 0x7fe07401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:08.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:08.581 7fe091c5d700 1 --2- 172.21.2.112:0/408061271 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe0740186c0 0x7fe07401abc0 crc :-1 s=READY pgs=544 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:08.597 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:08.609 7fe093e41700 1 -- 172.21.2.112:0/408061271 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe08c0f0b00 con 0x7fe08c09a220 2021-02-10T11:32:08.600 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:08.612 7fe0827fc700 1 -- 172.21.2.112:0/408061271 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe088008030 con 0x7fe08c09a220 2021-02-10T11:32:08.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:08.739 7fe093e41700 1 -- 172.21.2.112:0/408061271 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000940581", "target": ["mgr", ""], "group_name": "group_0000000000550574"}) v1 -- 0x7fe08c0a9090 con 0x7fe0740186c0 2021-02-10T11:32:08.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:08.751 7fe0827fc700 1 -- 172.21.2.112:0/408061271 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fe07c006420 con 0x7fe0740186c0 2021-02-10T11:32:08.745 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:08.757 7fe093e41700 1 -- 172.21.2.112:0/408061271 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe0740186c0 msgr2=0x7fe07401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:08.745 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:08.757 7fe093e41700 1 --2- 172.21.2.112:0/408061271 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe0740186c0 0x7fe07401abc0 crc :-1 s=READY pgs=544 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:08.745 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:08.757 7fe093e41700 1 -- 172.21.2.112:0/408061271 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe08c09a220 msgr2=0x7fe08c09a620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:08.745 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:08.757 7fe093e41700 1 --2- 172.21.2.112:0/408061271 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe08c09a220 0x7fe08c09a620 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 rx=0x7fe088009630 tx=0x7fe08800b900).stop 2021-02-10T11:32:08.746 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:08.757 7fe093e41700 1 -- 172.21.2.112:0/408061271 shutdown_connections 2021-02-10T11:32:08.746 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:08.757 7fe093e41700 1 --2- 172.21.2.112:0/408061271 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe0740186c0 0x7fe07401abc0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:08.746 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:08.757 7fe093e41700 1 --2- 172.21.2.112:0/408061271 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe08c08a460 0x7fe08c08a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:08.746 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:08.757 7fe093e41700 1 --2- 172.21.2.112:0/408061271 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe08c09a220 0x7fe08c09a620 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:08.746 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:08.757 7fe093e41700 1 --2- 172.21.2.112:0/408061271 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe08c1a49c0 0x7fe08c099c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:08.746 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:08.757 7fe093e41700 1 -- 172.21.2.112:0/408061271 >> 172.21.2.112:0/408061271 conn(0x7fe08c0bd7a0 msgr2=0x7fe08c09e160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:08.746 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:08.757 7fe093e41700 1 -- 172.21.2.112:0/408061271 shutdown_connections 2021-02-10T11:32:08.746 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:08.757 7fe093e41700 1 -- 172.21.2.112:0/408061271 wait complete. 2021-02-10T11:32:08.754 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolumegroup rm cephfs group_0000000000550574 2021-02-10T11:32:09.184 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:09.179 7fabaab1e700 1 Processor -- start 2021-02-10T11:32:09.184 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:09.179 7fabaab1e700 1 -- start start 2021-02-10T11:32:09.184 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:09.179 7fabaab1e700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7faba41b0870 0x7faba40f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:09.184 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:09.179 7fabaab1e700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faba41b0c70 con 0x7faba41b0470 2021-02-10T11:32:09.184 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:09.179 7fabaab1e700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faba40f0b50 con 0x7faba40efe00 2021-02-10T11:32:09.184 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:09.179 7fabaab1e700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faba40e5ea0 con 0x7faba41b0870 2021-02-10T11:32:09.184 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:09.179 7faba3fff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7faba41b0870 0x7faba40f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:09.184 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:09.179 7faba893a700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7faba41b0470 0x7faba40ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:48140/0 (socket says 172.21.2.112:48140) 2021-02-10T11:32:09.184 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:09.179 7faba893a700 1 -- 172.21.2.112:0/1929582165 learned_addr learned my addr 172.21.2.112:0/1929582165 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:32:09.184 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:09.180 7faba3fff700 1 -- 172.21.2.112:0/1929582165 >> v1:172.21.2.120:6789/0 conn(0x7faba40efe00 legacy=0x7faba40f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:09.185 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:09.180 7faba3fff700 1 -- 172.21.2.112:0/1929582165 >> v1:172.21.2.112:6789/0 conn(0x7faba41b0470 legacy=0x7faba40ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:09.185 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:09.180 7faba3fff700 1 -- 172.21.2.112:0/1929582165 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faba40e52e0 con 0x7faba41b0870 2021-02-10T11:32:09.185 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:09.180 7faba3fff700 1 --2- 172.21.2.112:0/1929582165 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faba41b0870 0x7faba40f20e0 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 rx=0x7fab94006650 tx=0x7fab94009a80).ready entity=mon.0 client_cookie=b0ac09a01aaa6aa9 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:09.185 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:09.180 7faba37fe700 1 -- 172.21.2.112:0/1929582165 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fab9400c0c0 con 0x7faba41b0870 2021-02-10T11:32:09.185 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:09.180 7faba37fe700 1 -- 172.21.2.112:0/1929582165 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab9400c340 con 0x7faba41b0870 2021-02-10T11:32:09.185 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:09.180 7faba37fe700 1 -- 172.21.2.112:0/1929582165 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fab9400c860 con 0x7faba41b0870 2021-02-10T11:32:09.185 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:09.180 7fabaab1e700 1 -- 172.21.2.112:0/1929582165 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faba41b0870 msgr2=0x7faba40f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:09.185 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:09.180 7fabaab1e700 1 --2- 172.21.2.112:0/1929582165 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faba41b0870 0x7faba40f20e0 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 rx=0x7fab94006650 tx=0x7fab94009a80).stop 2021-02-10T11:32:09.185 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:09.181 7fabaab1e700 1 -- 172.21.2.112:0/1929582165 shutdown_connections 2021-02-10T11:32:09.185 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:09.181 7fabaab1e700 1 --2- 172.21.2.112:0/1929582165 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faba41b0870 0x7faba40f20e0 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:09.185 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:09.181 7fabaab1e700 1 -- 172.21.2.112:0/1929582165 >> 172.21.2.112:0/1929582165 conn(0x7faba40ce3e0 msgr2=0x7faba40e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:09.185 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:09.181 7fabaab1e700 1 -- 172.21.2.112:0/1929582165 shutdown_connections 2021-02-10T11:32:09.185 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:09.181 7fabaab1e700 1 -- 172.21.2.112:0/1929582165 wait complete. 2021-02-10T11:32:09.187 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:09.196 7fabaab1e700 1 Processor -- start 2021-02-10T11:32:09.187 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:09.196 7fabaab1e700 1 -- start start 2021-02-10T11:32:09.187 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:09.196 7fabaab1e700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7faba40b9d30 0x7faba40932b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:09.187 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:09.196 7fabaab1e700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7faba40884e0 0x7faba407f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:09.187 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:09.196 7fabaab1e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faba40ba130 0x7faba4088f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:09.187 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:09.196 7fabaab1e700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faba4088a20 con 0x7faba40ba130 2021-02-10T11:32:09.187 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:09.196 7fabaab1e700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faba4080490 con 0x7faba40884e0 2021-02-10T11:32:09.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:09.196 7fabaab1e700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faba40938d0 con 0x7faba40b9d30 2021-02-10T11:32:09.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:09.197 7faba913b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faba40ba130 0x7faba4088f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:09.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:09.197 7faba913b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faba40ba130 0x7faba4088f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:48688/0 (socket says 172.21.2.112:48688) 2021-02-10T11:32:09.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:09.197 7faba913b700 1 -- 172.21.2.112:0/4242148052 learned_addr learned my addr 172.21.2.112:0/4242148052 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:32:09.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:09.197 7faba893a700 1 --2- 172.21.2.112:0/4242148052 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7faba40b9d30 0x7faba40932b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:09.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:09.197 7faba3fff700 1 --2- 172.21.2.112:0/4242148052 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7faba40884e0 0x7faba407f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:09.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:09.197 7faba913b700 1 -- 172.21.2.112:0/4242148052 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7faba40b9d30 msgr2=0x7faba40932b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:09.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:09.197 7faba913b700 1 --2- 172.21.2.112:0/4242148052 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7faba40b9d30 0x7faba40932b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:09.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:09.197 7faba913b700 1 -- 172.21.2.112:0/4242148052 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7faba40884e0 msgr2=0x7faba407f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:09.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:09.197 7faba913b700 1 --2- 172.21.2.112:0/4242148052 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7faba40884e0 0x7faba407f8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:09.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:09.197 7faba913b700 1 -- 172.21.2.112:0/4242148052 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faba407fed0 con 0x7faba40ba130 2021-02-10T11:32:09.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:09.197 7faba3fff700 1 --2- 172.21.2.112:0/4242148052 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7faba40884e0 0x7faba407f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:32:09.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:09.197 7faba913b700 1 --2- 172.21.2.112:0/4242148052 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faba40ba130 0x7faba4088f20 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 rx=0x7fab9c009940 tx=0x7fab9c00bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:09.189 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:09.198 7faba17fa700 1 -- 172.21.2.112:0/4242148052 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fab9c000ce0 con 0x7faba40ba130 2021-02-10T11:32:09.189 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:09.198 7faba17fa700 1 -- 172.21.2.112:0/4242148052 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab9c00c2f0 con 0x7faba40ba130 2021-02-10T11:32:09.189 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:09.198 7faba17fa700 1 -- 172.21.2.112:0/4242148052 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fab9c00c750 con 0x7faba40ba130 2021-02-10T11:32:09.189 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:09.198 7fabaab1e700 1 -- 172.21.2.112:0/4242148052 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faba407fed0 con 0x7faba40ba130 2021-02-10T11:32:09.189 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:09.198 7fabaab1e700 1 -- 172.21.2.112:0/4242148052 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faba4089830 con 0x7faba40ba130 2021-02-10T11:32:09.189 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:09.198 7faba17fa700 1 -- 172.21.2.112:0/4242148052 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fab9c001220 con 0x7faba40ba130 2021-02-10T11:32:09.189 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:09.198 7faba17fa700 1 --2- 172.21.2.112:0/4242148052 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fab880186c0 0x7fab8801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:09.189 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:09.198 7faba17fa700 1 -- 172.21.2.112:0/4242148052 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fab9c032900 con 0x7faba40ba130 2021-02-10T11:32:09.189 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:09.199 7faba893a700 1 --2- 172.21.2.112:0/4242148052 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fab880186c0 0x7fab8801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:09.189 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:09.199 7faba893a700 1 --2- 172.21.2.112:0/4242148052 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fab880186c0 0x7fab8801abc0 crc :-1 s=READY pgs=545 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:09.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:09.229 7fabaab1e700 1 -- 172.21.2.112:0/4242148052 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faba407e6b0 con 0x7faba40ba130 2021-02-10T11:32:09.219 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:09.232 7faba17fa700 1 -- 172.21.2.112:0/4242148052 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fab9c000cc0 con 0x7faba40ba130 2021-02-10T11:32:09.352 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:09.364 7fabaab1e700 1 -- 172.21.2.112:0/4242148052 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolumegroup rm", "vol_name": "cephfs", "target": ["mgr", ""], "group_name": "group_0000000000550574"}) v1 -- 0x7faba4089830 con 0x7fab880186c0 2021-02-10T11:32:09.356 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:09.369 7faba17fa700 1 -- 172.21.2.112:0/4242148052 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fab8c001050 con 0x7fab880186c0 2021-02-10T11:32:09.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:09.373 7fabaab1e700 1 -- 172.21.2.112:0/4242148052 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fab880186c0 msgr2=0x7fab8801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:09.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:09.373 7fabaab1e700 1 --2- 172.21.2.112:0/4242148052 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fab880186c0 0x7fab8801abc0 crc :-1 s=READY pgs=545 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:09.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:09.373 7fabaab1e700 1 -- 172.21.2.112:0/4242148052 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faba40ba130 msgr2=0x7faba4088f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:09.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:09.373 7fabaab1e700 1 --2- 172.21.2.112:0/4242148052 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faba40ba130 0x7faba4088f20 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 rx=0x7fab9c009940 tx=0x7fab9c00bcf0).stop 2021-02-10T11:32:09.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:09.373 7fabaab1e700 1 -- 172.21.2.112:0/4242148052 shutdown_connections 2021-02-10T11:32:09.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:09.373 7fabaab1e700 1 --2- 172.21.2.112:0/4242148052 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fab880186c0 0x7fab8801abc0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:09.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:09.373 7fabaab1e700 1 --2- 172.21.2.112:0/4242148052 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7faba40884e0 0x7faba407f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:09.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:09.373 7fabaab1e700 1 --2- 172.21.2.112:0/4242148052 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faba40ba130 0x7faba4088f20 unknown :-1 s=CLOSED pgs=1065 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:09.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:09.373 7fabaab1e700 1 --2- 172.21.2.112:0/4242148052 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7faba40b9d30 0x7faba40932b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:09.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:09.373 7fabaab1e700 1 -- 172.21.2.112:0/4242148052 >> 172.21.2.112:0/4242148052 conn(0x7faba40a1430 msgr2=0x7faba4083130 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:09.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:09.374 7fabaab1e700 1 -- 172.21.2.112:0/4242148052 shutdown_connections 2021-02-10T11:32:09.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:09.374 7fabaab1e700 1 -- 172.21.2.112:0/4242148052 wait complete. 2021-02-10T11:32:09.370 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:32:09.370 DEBUG:teuthology.orchestra.run.gibba012:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:32:09.715 DEBUG:teuthology.orchestra.run.gibba020:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:32:10.060 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:32:10.061 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:10.061 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:32:10.061 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:32:10.395 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:32:10.395 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:32:10.395 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba012.front.sepia.ceph.com... 2021-02-10T11:32:10.396 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:32:10.396 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:32:10.748 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba012.stderr:ceph-fuse[121274]: fuse finished with error 0 and tester_r 0 2021-02-10T11:32:10.749 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:10.750 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:32:10.750 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:32:11.083 INFO:teuthology.orchestra.run.gibba012.stdout:ext2/ext3 2021-02-10T11:32:11.084 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:32:11.084 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:32:11.084 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:32:11.418 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:11.419 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:32:11.419 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:32:11.754 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:32:11.754 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:32:11.755 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba012.front.sepia.ceph.com... 2021-02-10T11:32:11.755 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:32:11.755 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:32:12.107 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba012.stderr:ceph-fuse[121914]: fuse finished with error 0 and tester_r 0 2021-02-10T11:32:12.109 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:12.109 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:32:12.109 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:32:12.442 INFO:teuthology.orchestra.run.gibba012.stdout:ext2/ext3 2021-02-10T11:32:12.442 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:32:12.443 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:32:12.443 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:32:12.777 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:12.777 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:32:12.777 DEBUG:teuthology.orchestra.run.gibba020:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:32:13.111 INFO:teuthology.orchestra.run.gibba020.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:32:13.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:32:13.112 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:32:13.112 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:32:13.112 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:32:13.446 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:13.447 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:32:13.447 DEBUG:teuthology.orchestra.run.gibba020:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:32:13.780 INFO:teuthology.orchestra.run.gibba020.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:32:13.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:32:13.780 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:32:13.781 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:32:13.781 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:32:14.115 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config rm mon mon_allow_pool_delete 2021-02-10T11:32:14.545 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:14.541 7fdbfdbb8700 1 Processor -- start 2021-02-10T11:32:14.545 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:14.541 7fdbfdbb8700 1 -- start start 2021-02-10T11:32:14.545 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:14.541 7fdbfdbb8700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fdbf80cff00 0x7fdbf81b39d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:14.545 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:14.541 7fdbfdbb8700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fdbf80d1bf0 0x7fdbf80ef620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:14.545 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:14.541 7fdbfdbb8700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbf80f0b20 con 0x7fdbf80d17f0 2021-02-10T11:32:14.545 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:14.541 7fdbfdbb8700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbf80e5eb0 con 0x7fdbf80d1bf0 2021-02-10T11:32:14.546 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:14.541 7fdbfdbb8700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbf81b4020 con 0x7fdbf80cff00 2021-02-10T11:32:14.546 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:14.542 7fdbf77fe700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fdbf80d17f0 0x7fdbf81b31e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:48154/0 (socket says 172.21.2.112:48154) 2021-02-10T11:32:14.546 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:14.542 7fdbf77fe700 1 -- 172.21.2.112:0/4169739339 learned_addr learned my addr 172.21.2.112:0/4169739339 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:32:14.546 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:14.542 7fdbf7fff700 1 --2- 172.21.2.112:0/4169739339 >> v2:172.21.2.112:3300/0 conn(0x7fdbf80d1bf0 0x7fdbf80ef620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:14.546 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:14.542 7fdbf67fc700 1 -- 172.21.2.112:0/4169739339 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 859667958 0 0) 0x7fdbe8001180 con 0x7fdbf80d17f0 2021-02-10T11:32:14.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:14.542 7fdbf6ffd700 1 --2- 172.21.2.112:0/4169739339 >> v2:172.21.2.120:3300/0 conn(0x7fdbf80cff00 0x7fdbf81b39d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:14.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:14.542 7fdbf67fc700 1 -- 172.21.2.112:0/4169739339 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbe4002fe0 con 0x7fdbf80d17f0 2021-02-10T11:32:14.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:14.542 7fdbf7fff700 1 -- 172.21.2.112:0/4169739339 >> v2:172.21.2.120:3300/0 conn(0x7fdbf80cff00 msgr2=0x7fdbf81b39d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:14.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:14.542 7fdbf7fff700 1 --2- 172.21.2.112:0/4169739339 >> v2:172.21.2.120:3300/0 conn(0x7fdbf80cff00 0x7fdbf81b39d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:14.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:14.542 7fdbf7fff700 1 -- 172.21.2.112:0/4169739339 >> v1:172.21.2.112:6789/0 conn(0x7fdbf80d17f0 legacy=0x7fdbf81b31e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:14.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:14.542 7fdbf7fff700 1 -- 172.21.2.112:0/4169739339 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbf80f20b0 con 0x7fdbf80d1bf0 2021-02-10T11:32:14.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:14.542 7fdbf6ffd700 1 --2- 172.21.2.112:0/4169739339 >> v2:172.21.2.120:3300/0 conn(0x7fdbf80cff00 0x7fdbf81b39d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:32:14.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:14.542 7fdbf7fff700 1 --2- 172.21.2.112:0/4169739339 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdbf80d1bf0 0x7fdbf80ef620 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 rx=0x7fdbf0006ab0 tx=0x7fdbf0009b40).ready entity=mon.0 client_cookie=315eb822bd9bd90f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:14.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:14.542 7fdbf67fc700 1 -- 172.21.2.112:0/4169739339 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdbf00022e0 con 0x7fdbf80d1bf0 2021-02-10T11:32:14.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:14.543 7fdbf67fc700 1 -- 172.21.2.112:0/4169739339 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbf0002660 con 0x7fdbf80d1bf0 2021-02-10T11:32:14.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:14.543 7fdbf67fc700 1 -- 172.21.2.112:0/4169739339 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdbf0002a70 con 0x7fdbf80d1bf0 2021-02-10T11:32:14.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:14.543 7fdbfdbb8700 1 -- 172.21.2.112:0/4169739339 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdbf80d1bf0 msgr2=0x7fdbf80ef620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:14.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:14.543 7fdbfdbb8700 1 --2- 172.21.2.112:0/4169739339 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdbf80d1bf0 0x7fdbf80ef620 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 rx=0x7fdbf0006ab0 tx=0x7fdbf0009b40).stop 2021-02-10T11:32:14.549 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:14.543 7fdbfdbb8700 1 -- 172.21.2.112:0/4169739339 shutdown_connections 2021-02-10T11:32:14.549 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:14.543 7fdbfdbb8700 1 --2- 172.21.2.112:0/4169739339 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdbf80d1bf0 0x7fdbf80ef620 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:14.549 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:14.543 7fdbfdbb8700 1 --2- 172.21.2.112:0/4169739339 >> v2:172.21.2.120:3300/0 conn(0x7fdbf80cff00 0x7fdbf81b39d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:14.549 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:14.543 7fdbfdbb8700 1 -- 172.21.2.112:0/4169739339 >> 172.21.2.112:0/4169739339 conn(0x7fdbf80fde40 msgr2=0x7fdbf80ce390 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:14.549 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:14.543 7fdbfdbb8700 1 -- 172.21.2.112:0/4169739339 shutdown_connections 2021-02-10T11:32:14.549 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:14.543 7fdbfdbb8700 1 -- 172.21.2.112:0/4169739339 wait complete. 2021-02-10T11:32:14.550 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:14.558 7fdbfdbb8700 1 Processor -- start 2021-02-10T11:32:14.553 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:14.558 7fdbfdbb8700 1 -- start start 2021-02-10T11:32:14.553 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:14.558 7fdbfdbb8700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdbf81a49b0 0x7fdbf808a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:14.553 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:14.558 7fdbfdbb8700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fdbf809e080 0x7fdbf80c3100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:14.553 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:14.558 7fdbfdbb8700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdbf809e480 0x7fdbf80c3720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:14.554 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:14.558 7fdbfdbb8700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbf80c2b80 con 0x7fdbf809e480 2021-02-10T11:32:14.554 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:14.558 7fdbfdbb8700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbf80bee50 con 0x7fdbf809e080 2021-02-10T11:32:14.554 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:14.558 7fdbfdbb8700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbf80c13f0 con 0x7fdbf81a49b0 2021-02-10T11:32:14.554 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:14.559 7fdbf7fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdbf809e480 0x7fdbf80c3720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:14.554 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:14.559 7fdbf7fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdbf809e480 0x7fdbf80c3720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:48702/0 (socket says 172.21.2.112:48702) 2021-02-10T11:32:14.555 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:14.559 7fdbf7fff700 1 -- 172.21.2.112:0/1389714009 learned_addr learned my addr 172.21.2.112:0/1389714009 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:32:14.555 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:14.559 7fdbf6ffd700 1 --2- 172.21.2.112:0/1389714009 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fdbf809e080 0x7fdbf80c3100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:14.555 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:14.559 7fdbf77fe700 1 --2- 172.21.2.112:0/1389714009 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdbf81a49b0 0x7fdbf808a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:14.555 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:14.559 7fdbf7fff700 1 -- 172.21.2.112:0/1389714009 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdbf81a49b0 msgr2=0x7fdbf808a400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:14.555 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:14.559 7fdbf7fff700 1 --2- 172.21.2.112:0/1389714009 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdbf81a49b0 0x7fdbf808a400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:14.555 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:14.559 7fdbf7fff700 1 -- 172.21.2.112:0/1389714009 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fdbf809e080 msgr2=0x7fdbf80c3100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:14.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:14.559 7fdbf7fff700 1 --2- 172.21.2.112:0/1389714009 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fdbf809e080 0x7fdbf80c3100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:14.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:14.559 7fdbf7fff700 1 -- 172.21.2.112:0/1389714009 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbf80c1610 con 0x7fdbf809e480 2021-02-10T11:32:14.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:14.559 7fdbf6ffd700 1 --2- 172.21.2.112:0/1389714009 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fdbf809e080 0x7fdbf80c3100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:32:14.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:14.559 7fdbf77fe700 1 --2- 172.21.2.112:0/1389714009 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdbf81a49b0 0x7fdbf808a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:32:14.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:14.559 7fdbf7fff700 1 --2- 172.21.2.112:0/1389714009 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdbf809e480 0x7fdbf80c3720 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 rx=0x7fdbf000b5e0 tx=0x7fdbf000b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:14.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:14.560 7fdbeffff700 1 -- 172.21.2.112:0/1389714009 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdbf0006b40 con 0x7fdbf809e480 2021-02-10T11:32:14.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:14.560 7fdbeffff700 1 -- 172.21.2.112:0/1389714009 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbf0009070 con 0x7fdbf809e480 2021-02-10T11:32:14.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:14.560 7fdbeffff700 1 -- 172.21.2.112:0/1389714009 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdbf0002500 con 0x7fdbf809e480 2021-02-10T11:32:14.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:14.560 7fdbfdbb8700 1 -- 172.21.2.112:0/1389714009 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbf80c1610 con 0x7fdbf809e480 2021-02-10T11:32:14.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:14.560 7fdbfdbb8700 1 -- 172.21.2.112:0/1389714009 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbf80c1ba0 con 0x7fdbf809e480 2021-02-10T11:32:14.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:14.560 7fdbeffff700 1 -- 172.21.2.112:0/1389714009 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fdbf0006a70 con 0x7fdbf809e480 2021-02-10T11:32:14.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:14.560 7fdbeffff700 1 --2- 172.21.2.112:0/1389714009 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fdbdc0186c0 0x7fdbdc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:14.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:14.561 7fdbeffff700 1 -- 172.21.2.112:0/1389714009 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fdbf0032400 con 0x7fdbf809e480 2021-02-10T11:32:14.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:14.561 7fdbf77fe700 1 --2- 172.21.2.112:0/1389714009 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fdbdc0186c0 0x7fdbdc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:14.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:14.561 7fdbf77fe700 1 --2- 172.21.2.112:0/1389714009 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fdbdc0186c0 0x7fdbdc01abc0 crc :-1 s=READY pgs=546 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:14.578 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:14.591 7fdbfdbb8700 1 -- 172.21.2.112:0/1389714009 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbf8103690 con 0x7fdbf809e480 2021-02-10T11:32:14.581 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:14.594 7fdbeffff700 1 -- 172.21.2.112:0/1389714009 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fdbf00093e0 con 0x7fdbf809e480 2021-02-10T11:32:14.732 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:14.745 7fdbfdbb8700 1 -- 172.21.2.112:0/1389714009 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fdbf80f0b20 con 0x7fdbf809e480 2021-02-10T11:32:14.736 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:14.749 7fdbeffff700 1 -- 172.21.2.112:0/1389714009 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v5) v1 ==== 104+0+0 (secure 0 0 0) 0x7fdbf0006a70 con 0x7fdbf809e480 2021-02-10T11:32:14.741 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:14.754 7fdbfdbb8700 1 -- 172.21.2.112:0/1389714009 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fdbdc0186c0 msgr2=0x7fdbdc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:14.741 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:14.754 7fdbfdbb8700 1 --2- 172.21.2.112:0/1389714009 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fdbdc0186c0 0x7fdbdc01abc0 crc :-1 s=READY pgs=546 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:14.741 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:14.754 7fdbfdbb8700 1 -- 172.21.2.112:0/1389714009 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdbf809e480 msgr2=0x7fdbf80c3720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:14.741 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:14.754 7fdbfdbb8700 1 --2- 172.21.2.112:0/1389714009 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdbf809e480 0x7fdbf80c3720 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 rx=0x7fdbf000b5e0 tx=0x7fdbf000b980).stop 2021-02-10T11:32:14.741 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:14.754 7fdbfdbb8700 1 -- 172.21.2.112:0/1389714009 shutdown_connections 2021-02-10T11:32:14.741 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:14.754 7fdbfdbb8700 1 --2- 172.21.2.112:0/1389714009 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fdbdc0186c0 0x7fdbdc01abc0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:14.741 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:14.754 7fdbfdbb8700 1 --2- 172.21.2.112:0/1389714009 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fdbf809e080 0x7fdbf80c3100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:14.741 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:14.754 7fdbfdbb8700 1 --2- 172.21.2.112:0/1389714009 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdbf809e480 0x7fdbf80c3720 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:14.742 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:14.754 7fdbfdbb8700 1 --2- 172.21.2.112:0/1389714009 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdbf81a49b0 0x7fdbf808a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:14.742 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:14.754 7fdbfdbb8700 1 -- 172.21.2.112:0/1389714009 >> 172.21.2.112:0/1389714009 conn(0x7fdbf80a1690 msgr2=0x7fdbf80a94c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:14.742 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:14.754 7fdbfdbb8700 1 -- 172.21.2.112:0/1389714009 shutdown_connections 2021-02-10T11:32:14.742 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:14.754 7fdbfdbb8700 1 -- 172.21.2.112:0/1389714009 wait complete. 2021-02-10T11:32:14.750 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestVolumes.test_authorize_allow_existing_id_option' 2021-02-10T11:32:15.176 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:15.171 7f5441f0f700 1 Processor -- start 2021-02-10T11:32:15.176 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:15.171 7f5441f0f700 1 -- start start 2021-02-10T11:32:15.177 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:15.171 7f5441f0f700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f543c07e6a0 0x7f543c0d1790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:15.177 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:15.171 7f5441f0f700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f543c07ed80 con 0x7f543c0e5070 2021-02-10T11:32:15.177 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:15.171 7f5441f0f700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f543c07eaa0 con 0x7f543c0e5470 2021-02-10T11:32:15.177 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:15.171 7f5441f0f700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f543c1b4080 con 0x7f543c07e6a0 2021-02-10T11:32:15.177 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:15.172 7f543b7fe700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f543c0e5070 0x7f543c1b3820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:48168/0 (socket says 172.21.2.112:48168) 2021-02-10T11:32:15.177 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:15.172 7f543b7fe700 1 -- 172.21.2.112:0/2207944260 learned_addr learned my addr 172.21.2.112:0/2207944260 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:32:15.178 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:15.172 7f543bfff700 1 --2- 172.21.2.112:0/2207944260 >> v2:172.21.2.120:3300/0 conn(0x7f543c07e6a0 0x7f543c0d1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:15.178 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:15.172 7f543a7fc700 1 -- 172.21.2.112:0/2207944260 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2323583274 0 0) 0x7f54240010b0 con 0x7f543c0e5070 2021-02-10T11:32:15.178 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:15.172 7f543a7fc700 1 -- 172.21.2.112:0/2207944260 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5428002fe0 con 0x7f543c0e5070 2021-02-10T11:32:15.178 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:15.172 7f543a7fc700 1 -- 172.21.2.112:0/2207944260 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3348736098 0 0) 0x7f542c001080 con 0x7f543c0e5470 2021-02-10T11:32:15.178 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:15.172 7f543a7fc700 1 -- 172.21.2.112:0/2207944260 --> v1:172.21.2.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5428003210 con 0x7f543c0e5470 2021-02-10T11:32:15.178 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:15.173 7f543bfff700 1 -- 172.21.2.112:0/2207944260 >> v1:172.21.2.120:6789/0 conn(0x7f543c0e5470 legacy=0x7f543c07f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:15.178 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:15.173 7f543bfff700 1 -- 172.21.2.112:0/2207944260 >> v1:172.21.2.112:6789/0 conn(0x7f543c0e5070 legacy=0x7f543c1b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:15.178 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:15.173 7f543bfff700 1 -- 172.21.2.112:0/2207944260 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f543c0d1db0 con 0x7f543c07e6a0 2021-02-10T11:32:15.178 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:15.173 7f543bfff700 1 --2- 172.21.2.112:0/2207944260 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f543c07e6a0 0x7f543c0d1790 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 rx=0x7f5434006e50 tx=0x7f5434009e70).ready entity=mon.1 client_cookie=86822cfffeaa5784 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:15.179 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:15.173 7f543a7fc700 1 -- 172.21.2.112:0/2207944260 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5434006a20 con 0x7f543c07e6a0 2021-02-10T11:32:15.179 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:15.173 7f543a7fc700 1 -- 172.21.2.112:0/2207944260 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5434006790 con 0x7f543c07e6a0 2021-02-10T11:32:15.179 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:15.173 7f543a7fc700 1 -- 172.21.2.112:0/2207944260 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f543400b700 con 0x7f543c07e6a0 2021-02-10T11:32:15.179 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:15.173 7f5441f0f700 1 -- 172.21.2.112:0/2207944260 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f543c07e6a0 msgr2=0x7f543c0d1790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:15.179 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:15.173 7f5441f0f700 1 --2- 172.21.2.112:0/2207944260 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f543c07e6a0 0x7f543c0d1790 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 rx=0x7f5434006e50 tx=0x7f5434009e70).stop 2021-02-10T11:32:15.179 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:15.174 7f5441f0f700 1 -- 172.21.2.112:0/2207944260 shutdown_connections 2021-02-10T11:32:15.179 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:15.174 7f5441f0f700 1 --2- 172.21.2.112:0/2207944260 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f543c07e6a0 0x7f543c0d1790 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:15.179 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:15.174 7f5441f0f700 1 -- 172.21.2.112:0/2207944260 >> 172.21.2.112:0/2207944260 conn(0x7f543c0ce690 msgr2=0x7f543c0e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:15.179 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:15.174 7f5441f0f700 1 -- 172.21.2.112:0/2207944260 shutdown_connections 2021-02-10T11:32:15.179 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:15.174 7f5441f0f700 1 -- 172.21.2.112:0/2207944260 wait complete. 2021-02-10T11:32:15.179 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:15.189 7f5441f0f700 1 Processor -- start 2021-02-10T11:32:15.179 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:15.189 7f5441f0f700 1 -- start start 2021-02-10T11:32:15.181 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:15.189 7f5441f0f700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f543c08d400 0x7f543c096c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:15.181 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:15.189 7f5441f0f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f543c0b1450 0x7f543c0b1850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:15.181 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:15.189 7f5441f0f700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f543c0b1e70 0x7f543c1b1f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:15.181 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:15.189 7f5441f0f700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f543c097310 con 0x7f543c0b1450 2021-02-10T11:32:15.181 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:15.189 7f5441f0f700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f543c08d800 con 0x7f543c0b1e70 2021-02-10T11:32:15.181 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:15.189 7f5441f0f700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f543c08d9f0 con 0x7f543c08d400 2021-02-10T11:32:15.181 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:15.190 7f543affd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f543c0b1450 0x7f543c0b1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:15.181 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:15.190 7f543affd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f543c0b1450 0x7f543c0b1850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:48714/0 (socket says 172.21.2.112:48714) 2021-02-10T11:32:15.181 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:15.190 7f543affd700 1 -- 172.21.2.112:0/1501430915 learned_addr learned my addr 172.21.2.112:0/1501430915 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:32:15.181 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:15.190 7f543affd700 1 -- 172.21.2.112:0/1501430915 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f543c08d400 msgr2=0x7f543c096c70 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:32:15.182 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:15.190 7f543affd700 1 --2- 172.21.2.112:0/1501430915 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f543c08d400 0x7f543c096c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:15.182 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:15.190 7f543affd700 1 -- 172.21.2.112:0/1501430915 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f543c0b1e70 msgr2=0x7f543c1b1f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:15.182 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:15.190 7f543affd700 1 --2- 172.21.2.112:0/1501430915 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f543c0b1e70 0x7f543c1b1f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:15.182 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:15.190 7f543affd700 1 -- 172.21.2.112:0/1501430915 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f543c1b25f0 con 0x7f543c0b1450 2021-02-10T11:32:15.182 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:15.190 7f543affd700 1 --2- 172.21.2.112:0/1501430915 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f543c0b1450 0x7f543c0b1850 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 rx=0x7f542c006f00 tx=0x7f542c00b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:15.182 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:15.190 7f5433fff700 1 -- 172.21.2.112:0/1501430915 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f542c00bd30 con 0x7f543c0b1450 2021-02-10T11:32:15.182 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:15.190 7f5433fff700 1 -- 172.21.2.112:0/1501430915 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f542c000e20 con 0x7f543c0b1450 2021-02-10T11:32:15.182 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:15.190 7f5433fff700 1 -- 172.21.2.112:0/1501430915 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f542c0054f0 con 0x7f543c0b1450 2021-02-10T11:32:15.183 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:15.190 7f5441f0f700 1 -- 172.21.2.112:0/1501430915 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f543c1b25f0 con 0x7f543c0b1450 2021-02-10T11:32:15.183 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:15.190 7f5441f0f700 1 -- 172.21.2.112:0/1501430915 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f543c1b2b80 con 0x7f543c0b1450 2021-02-10T11:32:15.183 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:15.191 7f5433fff700 1 -- 172.21.2.112:0/1501430915 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f542c006030 con 0x7f543c0b1450 2021-02-10T11:32:15.183 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:15.191 7f5433fff700 1 --2- 172.21.2.112:0/1501430915 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f54200186c0 0x7f542001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:15.183 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:15.191 7f5433fff700 1 -- 172.21.2.112:0/1501430915 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f542c032590 con 0x7f543c0b1450 2021-02-10T11:32:15.183 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:15.192 7f543b7fe700 1 --2- 172.21.2.112:0/1501430915 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f54200186c0 0x7f542001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:15.183 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:15.193 7f543b7fe700 1 --2- 172.21.2.112:0/1501430915 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f54200186c0 0x7f542001abc0 crc :-1 s=READY pgs=547 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:15.209 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:15.222 7f5441f0f700 1 -- 172.21.2.112:0/1501430915 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f543c07ed80 con 0x7f543c0b1450 2021-02-10T11:32:15.211 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:15.224 7f5433fff700 1 -- 172.21.2.112:0/1501430915 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f542c000bc0 con 0x7f543c0b1450 2021-02-10T11:32:15.344 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:15.356 7f5441f0f700 1 -- 172.21.2.112:0/1501430915 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_authorize_allow_existing_id_option"]} v 0) v1 -- 0x7f543c07ed80 con 0x7f543c0b1450 2021-02-10T11:32:15.373 DEBUG:teuthology.orchestra.run.gibba012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:32:15.376 DEBUG:teuthology.orchestra.run.gibba020:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:32:16.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:16.053 7f5433fff700 1 -- 172.21.2.112:0/1501430915 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_authorize_allow_existing_id_option"]}]=0 v243) v1 ==== 156+0+0 (secure 0 0 0) 0x7f542c006030 con 0x7f543c0b1450 2021-02-10T11:32:16.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:16.076 7f5441f0f700 1 -- 172.21.2.112:0/1501430915 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f54200186c0 msgr2=0x7f542001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:16.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:16.076 7f5441f0f700 1 --2- 172.21.2.112:0/1501430915 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f54200186c0 0x7f542001abc0 crc :-1 s=READY pgs=547 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:16.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:16.076 7f5441f0f700 1 -- 172.21.2.112:0/1501430915 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f543c0b1450 msgr2=0x7f543c0b1850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:16.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:16.076 7f5441f0f700 1 --2- 172.21.2.112:0/1501430915 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f543c0b1450 0x7f543c0b1850 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 rx=0x7f542c006f00 tx=0x7f542c00b940).stop 2021-02-10T11:32:16.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:16.076 7f5441f0f700 1 -- 172.21.2.112:0/1501430915 shutdown_connections 2021-02-10T11:32:16.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:16.076 7f5441f0f700 1 --2- 172.21.2.112:0/1501430915 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f54200186c0 0x7f542001abc0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:16.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:16.076 7f5441f0f700 1 --2- 172.21.2.112:0/1501430915 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f543c0b1e70 0x7f543c1b1f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:16.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:16.076 7f5441f0f700 1 --2- 172.21.2.112:0/1501430915 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f543c0b1450 0x7f543c0b1850 unknown :-1 s=CLOSED pgs=1070 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:16.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:16.076 7f5441f0f700 1 --2- 172.21.2.112:0/1501430915 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f543c08d400 0x7f543c096c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:16.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:16.076 7f5441f0f700 1 -- 172.21.2.112:0/1501430915 >> 172.21.2.112:0/1501430915 conn(0x7f543c1ac620 msgr2=0x7f543c0896e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:16.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:16.077 7f5441f0f700 1 -- 172.21.2.112:0/1501430915 shutdown_connections 2021-02-10T11:32:16.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:16.077 7f5441f0f700 1 -- 172.21.2.112:0/1501430915 wait complete. 2021-02-10T11:32:16.073 INFO:tasks.cephfs_test_runner:test_authorize_allow_existing_id_option (tasks.cephfs.test_volumes.TestVolumes) ... ok 2021-02-10T11:32:16.073 INFO:tasks.cephfs_test_runner:Starting test: test_authorize_auth_id_not_created_by_mgr_volumes (tasks.cephfs.test_volumes.TestVolumes) 2021-02-10T11:32:16.074 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestVolumes.test_authorize_auth_id_not_created_by_mgr_volumes' 2021-02-10T11:32:16.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:16.496 7f954a1a8700 1 Processor -- start 2021-02-10T11:32:16.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:16.496 7f954a1a8700 1 -- start start 2021-02-10T11:32:16.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:16.496 7f954a1a8700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f954407e6a0 0x7f95440d1790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:16.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:16.496 7f954a1a8700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f954407ed80 con 0x7f95440e5070 2021-02-10T11:32:16.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:16.496 7f954a1a8700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f954407eb10 con 0x7f95440e5470 2021-02-10T11:32:16.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:16.496 7f954a1a8700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95441b4080 con 0x7f954407e6a0 2021-02-10T11:32:16.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:16.497 7f95437fe700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f95440e5070 0x7f95441b3820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:48180/0 (socket says 172.21.2.112:48180) 2021-02-10T11:32:16.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:16.497 7f95437fe700 1 -- 172.21.2.112:0/4255838297 learned_addr learned my addr 172.21.2.112:0/4255838297 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:32:16.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:16.497 7f95427fc700 1 -- 172.21.2.112:0/4255838297 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1421006828 0 0) 0x7f952c0010b0 con 0x7f95440e5070 2021-02-10T11:32:16.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:16.497 7f95427fc700 1 -- 172.21.2.112:0/4255838297 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9530002fe0 con 0x7f95440e5070 2021-02-10T11:32:16.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:16.497 7f9543fff700 1 --2- 172.21.2.112:0/4255838297 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f954407e6a0 0x7f95440d1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:16.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:16.497 7f95427fc700 1 -- 172.21.2.112:0/4255838297 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3794853909 0 0) 0x7f952c0010b0 con 0x7f95440e5070 2021-02-10T11:32:16.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:16.497 7f95427fc700 1 -- 172.21.2.112:0/4255838297 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f954407e6a0 msgr2=0x7f95440d1790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:16.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:16.497 7f95427fc700 1 --2- 172.21.2.112:0/4255838297 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f954407e6a0 0x7f95440d1790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:16.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:16.497 7f95427fc700 1 -- 172.21.2.112:0/4255838297 >> v1:172.21.2.120:6789/0 conn(0x7f95440e5470 legacy=0x7f954407f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:16.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:16.497 7f95427fc700 1 -- 172.21.2.112:0/4255838297 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95440d1db0 con 0x7f95440e5070 2021-02-10T11:32:16.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:16.497 7f95427fc700 1 -- 172.21.2.112:0/4255838297 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f952c0015b0 con 0x7f95440e5070 2021-02-10T11:32:16.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:16.497 7f95427fc700 1 -- 172.21.2.112:0/4255838297 >> v1:172.21.2.112:6789/0 conn(0x7f95440e5070 legacy=0x7f95441b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:16.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:16.497 7f95427fc700 1 --2- 172.21.2.112:0/4255838297 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9530003a40 0x7f9530005f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:16.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:16.497 7f95427fc700 1 --2- 172.21.2.112:0/4255838297 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9530006800 0x7f9530008c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:16.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:16.497 7f95427fc700 1 --2- 172.21.2.112:0/4255838297 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9530009510 0x7f953000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:16.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:16.497 7f95427fc700 1 -- 172.21.2.112:0/4255838297 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f953000c2c0 con 0x7f9530003a40 2021-02-10T11:32:16.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:16.497 7f95427fc700 1 -- 172.21.2.112:0/4255838297 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f953000c4e0 con 0x7f9530009510 2021-02-10T11:32:16.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:16.497 7f95427fc700 1 -- 172.21.2.112:0/4255838297 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f953000c700 con 0x7f9530006800 2021-02-10T11:32:16.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:16.497 7f95437fe700 1 --2- 172.21.2.112:0/4255838297 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9530003a40 0x7f9530005f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:16.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:16.498 7f9543fff700 1 --2- 172.21.2.112:0/4255838297 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9530009510 0x7f953000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:16.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:16.498 7f9542ffd700 1 --2- 172.21.2.112:0/4255838297 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9530006800 0x7f9530008c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:16.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:16.498 7f95437fe700 1 -- 172.21.2.112:0/4255838297 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9530006800 msgr2=0x7f9530008c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:16.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:16.498 7f95437fe700 1 --2- 172.21.2.112:0/4255838297 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9530006800 0x7f9530008c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:16.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:16.498 7f95437fe700 1 -- 172.21.2.112:0/4255838297 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9530009510 msgr2=0x7f953000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:16.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:16.498 7f95437fe700 1 --2- 172.21.2.112:0/4255838297 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9530009510 0x7f953000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:16.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:16.498 7f95437fe700 1 -- 172.21.2.112:0/4255838297 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f953000c9a0 con 0x7f9530003a40 2021-02-10T11:32:16.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:16.498 7f95437fe700 1 --2- 172.21.2.112:0/4255838297 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9530003a40 0x7f9530005f60 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 rx=0x7f952c001ae0 tx=0x7f952c04bcc0).ready entity=mon.0 client_cookie=e2320e3b7fc449db server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:16.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:16.498 7f95427fc700 1 -- 172.21.2.112:0/4255838297 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f952c0009b0 con 0x7f9530003a40 2021-02-10T11:32:16.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:16.498 7f95427fc700 1 -- 172.21.2.112:0/4255838297 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f952c004410 con 0x7f9530003a40 2021-02-10T11:32:16.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:16.498 7f95427fc700 1 -- 172.21.2.112:0/4255838297 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f952c004860 con 0x7f9530003a40 2021-02-10T11:32:16.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:16.498 7f954a1a8700 1 -- 172.21.2.112:0/4255838297 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9530003a40 msgr2=0x7f9530005f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:16.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:16.498 7f954a1a8700 1 --2- 172.21.2.112:0/4255838297 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9530003a40 0x7f9530005f60 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 rx=0x7f952c001ae0 tx=0x7f952c04bcc0).stop 2021-02-10T11:32:16.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:16.498 7f9543fff700 1 -- 172.21.2.112:0/4255838297 reap_dead start 2021-02-10T11:32:16.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:16.499 7f954a1a8700 1 -- 172.21.2.112:0/4255838297 shutdown_connections 2021-02-10T11:32:16.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:16.499 7f954a1a8700 1 -- 172.21.2.112:0/4255838297 >> 172.21.2.112:0/4255838297 conn(0x7f95440ce690 msgr2=0x7f95440e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:16.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:16.499 7f954a1a8700 1 -- 172.21.2.112:0/4255838297 shutdown_connections 2021-02-10T11:32:16.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:16.499 7f954a1a8700 1 -- 172.21.2.112:0/4255838297 wait complete. 2021-02-10T11:32:16.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:16.513 7f954a1a8700 1 Processor -- start 2021-02-10T11:32:16.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:16.513 7f954a1a8700 1 -- start start 2021-02-10T11:32:16.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:16.514 7f954a1a8700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9544194000 0x7f95440b0d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:16.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:16.514 7f954a1a8700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f95440b1330 0x7f9544087050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:16.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:16.514 7f954a1a8700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9544087670 0x7f9544087a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:16.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:16.514 7f954a1a8700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f954409c0e0 con 0x7f9544194000 2021-02-10T11:32:16.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:16.514 7f954a1a8700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9544194440 con 0x7f95440b1330 2021-02-10T11:32:16.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:16.514 7f954a1a8700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f954409c5e0 con 0x7f9544087670 2021-02-10T11:32:16.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:16.514 7f95437fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9544194000 0x7f95440b0d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:16.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:16.514 7f95437fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9544194000 0x7f95440b0d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:48732/0 (socket says 172.21.2.112:48732) 2021-02-10T11:32:16.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:16.514 7f95437fe700 1 -- 172.21.2.112:0/1175495285 learned_addr learned my addr 172.21.2.112:0/1175495285 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:32:16.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:16.514 7f9543fff700 1 --2- 172.21.2.112:0/1175495285 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9544087670 0x7f9544087a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:16.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:16.514 7f9542ffd700 1 --2- 172.21.2.112:0/1175495285 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f95440b1330 0x7f9544087050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:16.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:16.514 7f95437fe700 1 -- 172.21.2.112:0/1175495285 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9544087670 msgr2=0x7f9544087a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:16.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:16.514 7f95437fe700 1 --2- 172.21.2.112:0/1175495285 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9544087670 0x7f9544087a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:16.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:16.514 7f95437fe700 1 -- 172.21.2.112:0/1175495285 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f95440b1330 msgr2=0x7f9544087050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:16.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:16.514 7f95437fe700 1 --2- 172.21.2.112:0/1175495285 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f95440b1330 0x7f9544087050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:16.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:16.515 7f95437fe700 1 -- 172.21.2.112:0/1175495285 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f954409c7d0 con 0x7f9544194000 2021-02-10T11:32:16.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:16.515 7f9543fff700 1 --2- 172.21.2.112:0/1175495285 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9544087670 0x7f9544087a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:32:16.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:16.515 7f95437fe700 1 --2- 172.21.2.112:0/1175495285 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9544194000 0x7f95440b0d10 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 rx=0x7f952c001260 tx=0x7f952c04e950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:16.508 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:16.515 7f953bfff700 1 -- 172.21.2.112:0/1175495285 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f952c0009b0 con 0x7f9544194000 2021-02-10T11:32:16.508 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:16.515 7f953bfff700 1 -- 172.21.2.112:0/1175495285 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f952c04ed40 con 0x7f9544194000 2021-02-10T11:32:16.508 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:16.515 7f954a1a8700 1 -- 172.21.2.112:0/1175495285 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f954409c7d0 con 0x7f9544194000 2021-02-10T11:32:16.508 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:16.515 7f954a1a8700 1 -- 172.21.2.112:0/1175495285 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f954409cd60 con 0x7f9544194000 2021-02-10T11:32:16.508 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:16.515 7f953bfff700 1 -- 172.21.2.112:0/1175495285 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f952c050350 con 0x7f9544194000 2021-02-10T11:32:16.508 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:16.516 7f953bfff700 1 -- 172.21.2.112:0/1175495285 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f952c04c030 con 0x7f9544194000 2021-02-10T11:32:16.508 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:16.516 7f953bfff700 1 --2- 172.21.2.112:0/1175495285 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f95280186c0 0x7f952801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:16.508 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:16.516 7f953bfff700 1 -- 172.21.2.112:0/1175495285 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f952c076590 con 0x7f9544194000 2021-02-10T11:32:16.508 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:16.516 7f9542ffd700 1 --2- 172.21.2.112:0/1175495285 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f95280186c0 0x7f952801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:16.508 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:16.517 7f9542ffd700 1 --2- 172.21.2.112:0/1175495285 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f95280186c0 0x7f952801abc0 crc :-1 s=READY pgs=548 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:16.533 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:16.546 7f954a1a8700 1 -- 172.21.2.112:0/1175495285 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9544084b20 con 0x7f9544194000 2021-02-10T11:32:16.535 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:16.548 7f953bfff700 1 -- 172.21.2.112:0/1175495285 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f952c0009b0 con 0x7f9544194000 2021-02-10T11:32:16.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:16.681 7f954a1a8700 1 -- 172.21.2.112:0/1175495285 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_authorize_auth_id_not_created_by_mgr_volumes"]} v 0) v1 -- 0x7f954407ed80 con 0x7f9544194000 2021-02-10T11:32:17.045 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:17.057 7f953bfff700 1 -- 172.21.2.112:0/1175495285 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_authorize_auth_id_not_created_by_mgr_volumes"]}]=0 v244) v1 ==== 169+0+0 (secure 0 0 0) 0x7f952c04c030 con 0x7f9544194000 2021-02-10T11:32:17.087 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:17.099 7f954a1a8700 1 -- 172.21.2.112:0/1175495285 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f95280186c0 msgr2=0x7f952801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:17.087 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:17.099 7f954a1a8700 1 --2- 172.21.2.112:0/1175495285 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f95280186c0 0x7f952801abc0 crc :-1 s=READY pgs=548 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:17.087 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:17.099 7f954a1a8700 1 -- 172.21.2.112:0/1175495285 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9544194000 msgr2=0x7f95440b0d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:17.088 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:17.099 7f954a1a8700 1 --2- 172.21.2.112:0/1175495285 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9544194000 0x7f95440b0d10 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 rx=0x7f952c001260 tx=0x7f952c04e950).stop 2021-02-10T11:32:17.088 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:17.099 7f954a1a8700 1 -- 172.21.2.112:0/1175495285 shutdown_connections 2021-02-10T11:32:17.088 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:17.099 7f954a1a8700 1 --2- 172.21.2.112:0/1175495285 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f95280186c0 0x7f952801abc0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:17.088 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:17.099 7f954a1a8700 1 --2- 172.21.2.112:0/1175495285 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f95440b1330 0x7f9544087050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:17.088 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:17.099 7f954a1a8700 1 --2- 172.21.2.112:0/1175495285 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9544087670 0x7f9544087a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:17.088 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:17.099 7f954a1a8700 1 --2- 172.21.2.112:0/1175495285 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9544194000 0x7f95440b0d10 unknown :-1 s=CLOSED pgs=1073 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:17.088 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:17.099 7f954a1a8700 1 -- 172.21.2.112:0/1175495285 >> 172.21.2.112:0/1175495285 conn(0x7f9544092c20 msgr2=0x7f9544195070 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:17.088 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:17.100 7f954a1a8700 1 -- 172.21.2.112:0/1175495285 shutdown_connections 2021-02-10T11:32:17.088 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:17.100 7f954a1a8700 1 -- 172.21.2.112:0/1175495285 wait complete. 2021-02-10T11:32:17.095 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:32:17.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:17.521 7f5e4a062700 1 Processor -- start 2021-02-10T11:32:17.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:17.521 7f5e4a062700 1 -- start start 2021-02-10T11:32:17.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:17.522 7f5e4a062700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f5e441b0470 0x7f5e441b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:17.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:17.522 7f5e4a062700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5e440ef610 0x7f5e440efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:17.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:17.522 7f5e4a062700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e440f0b50 con 0x7f5e440f0030 2021-02-10T11:32:17.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:17.522 7f5e4a062700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e440e52e0 con 0x7f5e441b0470 2021-02-10T11:32:17.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:17.522 7f5e4a062700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e440e54d0 con 0x7f5e440ef610 2021-02-10T11:32:17.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:17.522 7f5e43fff700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f5e440f0030 0x7f5e440f10d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:48200/0 (socket says 172.21.2.112:48200) 2021-02-10T11:32:17.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:17.522 7f5e43fff700 1 -- 172.21.2.112:0/1720212645 learned_addr learned my addr 172.21.2.112:0/1720212645 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:32:17.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:17.522 7f5e437fe700 1 --2- 172.21.2.112:0/1720212645 >> v2:172.21.2.112:3300/0 conn(0x7f5e441b0470 0x7f5e441b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:17.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:17.522 7f5e437fe700 1 -- 172.21.2.112:0/1720212645 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5e440ef610 msgr2=0x7f5e440efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:17.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:17.522 7f5e437fe700 1 --2- 172.21.2.112:0/1720212645 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5e440ef610 0x7f5e440efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:17.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:17.522 7f5e437fe700 1 -- 172.21.2.112:0/1720212645 >> v1:172.21.2.112:6789/0 conn(0x7f5e440f0030 legacy=0x7f5e440f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:17.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:17.522 7f5e437fe700 1 -- 172.21.2.112:0/1720212645 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e440f0430 con 0x7f5e441b0470 2021-02-10T11:32:17.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:17.522 7f5e427fc700 1 -- 172.21.2.112:0/1720212645 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 868588357 0 0) 0x7f5e3c001500 con 0x7f5e440f0030 2021-02-10T11:32:17.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:17.523 7f5e437fe700 1 --2- 172.21.2.112:0/1720212645 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5e441b0470 0x7f5e441b0870 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 rx=0x7f5e2c006650 tx=0x7f5e2c009a80).ready entity=mon.0 client_cookie=7a222aa6c9b58ecc server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:17.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:17.523 7f5e427fc700 1 -- 172.21.2.112:0/1720212645 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5e2c00c0c0 con 0x7f5e441b0470 2021-02-10T11:32:17.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:17.523 7f5e427fc700 1 -- 172.21.2.112:0/1720212645 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e2c00c340 con 0x7f5e441b0470 2021-02-10T11:32:17.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:17.523 7f5e427fc700 1 -- 172.21.2.112:0/1720212645 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5e2c00c860 con 0x7f5e441b0470 2021-02-10T11:32:17.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:17.523 7f5e4a062700 1 -- 172.21.2.112:0/1720212645 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5e441b0470 msgr2=0x7f5e441b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:17.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:17.523 7f5e4a062700 1 --2- 172.21.2.112:0/1720212645 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5e441b0470 0x7f5e441b0870 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 rx=0x7f5e2c006650 tx=0x7f5e2c009a80).stop 2021-02-10T11:32:17.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:17.523 7f5e4a062700 1 -- 172.21.2.112:0/1720212645 shutdown_connections 2021-02-10T11:32:17.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:17.523 7f5e4a062700 1 --2- 172.21.2.112:0/1720212645 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5e440ef610 0x7f5e440efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:17.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:17.523 7f5e4a062700 1 --2- 172.21.2.112:0/1720212645 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5e441b0470 0x7f5e441b0870 unknown :-1 s=CLOSED pgs=1075 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:17.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:17.523 7f5e4a062700 1 -- 172.21.2.112:0/1720212645 >> 172.21.2.112:0/1720212645 conn(0x7f5e440ce3e0 msgr2=0x7f5e440e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:17.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:17.523 7f5e4a062700 1 -- 172.21.2.112:0/1720212645 shutdown_connections 2021-02-10T11:32:17.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:17.523 7f5e4a062700 1 -- 172.21.2.112:0/1720212645 wait complete. 2021-02-10T11:32:17.528 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:17.538 7f5e4a062700 1 Processor -- start 2021-02-10T11:32:17.528 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:17.538 7f5e4a062700 1 -- start start 2021-02-10T11:32:17.528 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:17.538 7f5e4a062700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5e44190920 0x7f5e4408f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:17.528 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:17.538 7f5e4a062700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5e44190d20 0x7f5e4408fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:17.528 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:17.538 7f5e4a062700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5e44193270 0x7f5e44193670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:17.528 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:17.538 7f5e4a062700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e44090470 con 0x7f5e44190d20 2021-02-10T11:32:17.529 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:17.538 7f5e4a062700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e44193c90 con 0x7f5e44193270 2021-02-10T11:32:17.529 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:17.538 7f5e4a062700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e44193e70 con 0x7f5e44190920 2021-02-10T11:32:17.529 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:17.539 7f5e42ffd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5e44190d20 0x7f5e4408fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:17.529 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:17.539 7f5e42ffd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5e44190d20 0x7f5e4408fc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:48746/0 (socket says 172.21.2.112:48746) 2021-02-10T11:32:17.529 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:17.539 7f5e42ffd700 1 -- 172.21.2.112:0/1810812844 learned_addr learned my addr 172.21.2.112:0/1810812844 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:32:17.529 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:17.539 7f5e42ffd700 1 -- 172.21.2.112:0/1810812844 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5e44190920 msgr2=0x7f5e4408f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:17.529 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:17.539 7f5e42ffd700 1 --2- 172.21.2.112:0/1810812844 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5e44190920 0x7f5e4408f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:17.529 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:17.539 7f5e42ffd700 1 -- 172.21.2.112:0/1810812844 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5e44193270 msgr2=0x7f5e44193670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:17.529 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:17.539 7f5e42ffd700 1 --2- 172.21.2.112:0/1810812844 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5e44193270 0x7f5e44193670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:17.529 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:17.539 7f5e42ffd700 1 -- 172.21.2.112:0/1810812844 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e44194090 con 0x7f5e44190d20 2021-02-10T11:32:17.529 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:17.539 7f5e42ffd700 1 --2- 172.21.2.112:0/1810812844 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5e44190d20 0x7f5e4408fc20 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 rx=0x7f5e34006f00 tx=0x7f5e3400b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:17.529 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:17.539 7f5e3bfff700 1 -- 172.21.2.112:0/1810812844 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5e3400bd30 con 0x7f5e44190d20 2021-02-10T11:32:17.529 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:17.539 7f5e3bfff700 1 -- 172.21.2.112:0/1810812844 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e34000c90 con 0x7f5e44190d20 2021-02-10T11:32:17.530 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:17.539 7f5e3bfff700 1 -- 172.21.2.112:0/1810812844 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5e340022a0 con 0x7f5e44190d20 2021-02-10T11:32:17.530 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:17.539 7f5e4a062700 1 -- 172.21.2.112:0/1810812844 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e44194090 con 0x7f5e44190d20 2021-02-10T11:32:17.530 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:17.539 7f5e4a062700 1 -- 172.21.2.112:0/1810812844 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e441920b0 con 0x7f5e44190d20 2021-02-10T11:32:17.530 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:17.540 7f5e3bfff700 1 -- 172.21.2.112:0/1810812844 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5e34005310 con 0x7f5e44190d20 2021-02-10T11:32:17.530 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:17.540 7f5e3bfff700 1 --2- 172.21.2.112:0/1810812844 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5e280186c0 0x7f5e2801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:17.530 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:17.540 7f5e3bfff700 1 -- 172.21.2.112:0/1810812844 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f5e34032550 con 0x7f5e44190d20 2021-02-10T11:32:17.530 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:17.540 7f5e437fe700 1 --2- 172.21.2.112:0/1810812844 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5e280186c0 0x7f5e2801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:17.530 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:17.541 7f5e437fe700 1 --2- 172.21.2.112:0/1810812844 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5e280186c0 0x7f5e2801abc0 crc :-1 s=READY pgs=549 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:17.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:17.572 7f5e4a062700 1 -- 172.21.2.112:0/1810812844 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e44105af0 con 0x7f5e44190d20 2021-02-10T11:32:17.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:17.574 7f5e3bfff700 1 -- 172.21.2.112:0/1810812844 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5e34000960 con 0x7f5e44190d20 2021-02-10T11:32:17.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:17.722 7f5e4a062700 1 -- 172.21.2.112:0/1810812844 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f5e440f0b50 con 0x7f5e44190d20 2021-02-10T11:32:17.714 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:17.727 7f5e3bfff700 1 -- 172.21.2.112:0/1810812844 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v6)=0 v6) v1 ==== 122+0+0 (secure 0 0 0) 0x7f5e34005310 con 0x7f5e44190d20 2021-02-10T11:32:17.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:17.731 7f5e4a062700 1 -- 172.21.2.112:0/1810812844 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5e280186c0 msgr2=0x7f5e2801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:17.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:17.731 7f5e4a062700 1 --2- 172.21.2.112:0/1810812844 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5e280186c0 0x7f5e2801abc0 crc :-1 s=READY pgs=549 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:17.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:17.731 7f5e4a062700 1 -- 172.21.2.112:0/1810812844 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5e44190d20 msgr2=0x7f5e4408fc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:17.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:17.731 7f5e4a062700 1 --2- 172.21.2.112:0/1810812844 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5e44190d20 0x7f5e4408fc20 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 rx=0x7f5e34006f00 tx=0x7f5e3400b940).stop 2021-02-10T11:32:17.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:17.732 7f5e4a062700 1 -- 172.21.2.112:0/1810812844 shutdown_connections 2021-02-10T11:32:17.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:17.732 7f5e4a062700 1 --2- 172.21.2.112:0/1810812844 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5e280186c0 0x7f5e2801abc0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:17.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:17.732 7f5e4a062700 1 --2- 172.21.2.112:0/1810812844 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5e44193270 0x7f5e44193670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:17.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:17.732 7f5e4a062700 1 --2- 172.21.2.112:0/1810812844 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5e44190d20 0x7f5e4408fc20 unknown :-1 s=CLOSED pgs=1076 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:17.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:17.732 7f5e4a062700 1 --2- 172.21.2.112:0/1810812844 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5e44190920 0x7f5e4408f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:17.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:17.732 7f5e4a062700 1 -- 172.21.2.112:0/1810812844 >> 172.21.2.112:0/1810812844 conn(0x7f5e440bf4e0 msgr2=0x7f5e440be3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:17.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:17.732 7f5e4a062700 1 -- 172.21.2.112:0/1810812844 shutdown_connections 2021-02-10T11:32:17.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:17.732 7f5e4a062700 1 -- 172.21.2.112:0/1810812844 wait complete. 2021-02-10T11:32:17.727 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:32:17.727 DEBUG:teuthology.orchestra.run.gibba012:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:32:18.072 DEBUG:teuthology.orchestra.run.gibba020:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:32:18.416 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:32:18.416 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:18.416 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:32:18.416 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:32:18.750 INFO:teuthology.orchestra.run.gibba012.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.0’: No such file or directory 2021-02-10T11:32:18.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:32:18.751 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2021-02-10T11:32:18.751 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:18.751 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:32:18.751 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:32:19.086 INFO:teuthology.orchestra.run.gibba012.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: No such file or directory 2021-02-10T11:32:19.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:32:19.086 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2021-02-10T11:32:19.086 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:19.087 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:32:19.087 DEBUG:teuthology.orchestra.run.gibba020:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:32:19.419 INFO:teuthology.orchestra.run.gibba020.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:32:19.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:32:19.419 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:32:19.420 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:19.420 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:32:19.420 DEBUG:teuthology.orchestra.run.gibba020:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:32:19.755 INFO:teuthology.orchestra.run.gibba020.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:32:19.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:32:19.756 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:32:19.756 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json-pretty 2021-02-10T11:32:20.181 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.178 7f57f66e8700 1 Processor -- start 2021-02-10T11:32:20.181 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.178 7f57f66e8700 1 -- start start 2021-02-10T11:32:20.181 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.179 7f57f66e8700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f57f00e5070 0x7f57f01b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:20.181 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.179 7f57f66e8700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f57f00e5470 0x7f57f0080310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:20.182 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.179 7f57f66e8700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57f007ed80 con 0x7f57f01b3e40 2021-02-10T11:32:20.182 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.179 7f57f66e8700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57f00d1790 con 0x7f57f00e5070 2021-02-10T11:32:20.182 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.179 7f57f66e8700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57f00d19b0 con 0x7f57f00e5470 2021-02-10T11:32:20.182 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.179 7f57effff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f57f00e5070 0x7f57f01b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:20.182 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.179 7f57effff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f57f00e5070 0x7f57f01b3820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:48750/0 (socket says 172.21.2.112:48750) 2021-02-10T11:32:20.182 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.179 7f57effff700 1 -- 172.21.2.112:0/3381129459 learned_addr learned my addr 172.21.2.112:0/3381129459 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:32:20.182 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.179 7f57effff700 1 -- 172.21.2.112:0/3381129459 >> v2:172.21.2.120:3300/0 conn(0x7f57f00e5470 msgr2=0x7f57f0080310 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:32:20.182 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.179 7f57effff700 1 --2- 172.21.2.112:0/3381129459 >> v2:172.21.2.120:3300/0 conn(0x7f57f00e5470 0x7f57f0080310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:20.182 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.179 7f57effff700 1 -- 172.21.2.112:0/3381129459 >> v1:172.21.2.120:6789/0 conn(0x7f57f01b3e40 legacy=0x7f57f007d690 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:32:20.182 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.179 7f57effff700 1 -- 172.21.2.112:0/3381129459 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57f00d1bd0 con 0x7f57f00e5070 2021-02-10T11:32:20.182 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.179 7f57effff700 1 --2- 172.21.2.112:0/3381129459 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f57f00e5070 0x7f57f01b3820 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 rx=0x7f57e0006700 tx=0x7f57e0009b00).ready entity=mon.0 client_cookie=d397e71ee531157e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:20.182 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.179 7f57eeffd700 1 -- 172.21.2.112:0/3381129459 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f57e000c0c0 con 0x7f57f00e5070 2021-02-10T11:32:20.183 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.179 7f57eeffd700 1 -- 172.21.2.112:0/3381129459 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57e000c340 con 0x7f57f00e5070 2021-02-10T11:32:20.183 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.179 7f57eeffd700 1 -- 172.21.2.112:0/3381129459 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f57e000c820 con 0x7f57f00e5070 2021-02-10T11:32:20.183 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.179 7f57f66e8700 1 -- 172.21.2.112:0/3381129459 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f57f00e5070 msgr2=0x7f57f01b3820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:20.183 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.179 7f57f66e8700 1 --2- 172.21.2.112:0/3381129459 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f57f00e5070 0x7f57f01b3820 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 rx=0x7f57e0006700 tx=0x7f57e0009b00).stop 2021-02-10T11:32:20.183 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.180 7f57f66e8700 1 -- 172.21.2.112:0/3381129459 shutdown_connections 2021-02-10T11:32:20.183 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.180 7f57f66e8700 1 --2- 172.21.2.112:0/3381129459 >> v2:172.21.2.120:3300/0 conn(0x7f57f00e5470 0x7f57f0080310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:20.183 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.180 7f57f66e8700 1 --2- 172.21.2.112:0/3381129459 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f57f00e5070 0x7f57f01b3820 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:20.183 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.180 7f57f66e8700 1 -- 172.21.2.112:0/3381129459 >> 172.21.2.112:0/3381129459 conn(0x7f57f00ce690 msgr2=0x7f57f00e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:20.183 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.180 7f57f66e8700 1 -- 172.21.2.112:0/3381129459 shutdown_connections 2021-02-10T11:32:20.183 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.180 7f57f66e8700 1 -- 172.21.2.112:0/3381129459 wait complete. 2021-02-10T11:32:20.185 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.194 7f57f66e8700 1 Processor -- start 2021-02-10T11:32:20.186 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.194 7f57f66e8700 1 -- start start 2021-02-10T11:32:20.186 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.195 7f57f66e8700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f57f01a2a90 0x7f57f01aa0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:20.186 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.195 7f57f66e8700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f57f01aa700 0x7f57f01a3a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:20.186 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.195 7f57f66e8700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f57f00a55e0 0x7f57f01a4040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:20.186 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.195 7f57f66e8700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57f01a34a0 con 0x7f57f00a55e0 2021-02-10T11:32:20.186 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.195 7f57f66e8700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57f01a2e90 con 0x7f57f01aa700 2021-02-10T11:32:20.186 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.195 7f57f66e8700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57f01a30b0 con 0x7f57f01a2a90 2021-02-10T11:32:20.186 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.195 7f57f4d05700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f57f00a55e0 0x7f57f01a4040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:20.186 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.195 7f57f4d05700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f57f00a55e0 0x7f57f01a4040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:48760/0 (socket says 172.21.2.112:48760) 2021-02-10T11:32:20.187 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.195 7f57f4d05700 1 -- 172.21.2.112:0/2493503902 learned_addr learned my addr 172.21.2.112:0/2493503902 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:32:20.187 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.195 7f57effff700 1 --2- 172.21.2.112:0/2493503902 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f57f01a2a90 0x7f57f01aa0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:20.187 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.195 7f57ef7fe700 1 --2- 172.21.2.112:0/2493503902 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f57f01aa700 0x7f57f01a3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:20.187 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.195 7f57f4d05700 1 -- 172.21.2.112:0/2493503902 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f57f01a2a90 msgr2=0x7f57f01aa0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:20.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.195 7f57f4d05700 1 --2- 172.21.2.112:0/2493503902 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f57f01a2a90 0x7f57f01aa0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:20.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.195 7f57f4d05700 1 -- 172.21.2.112:0/2493503902 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f57f01aa700 msgr2=0x7f57f01a3a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:20.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.195 7f57f4d05700 1 --2- 172.21.2.112:0/2493503902 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f57f01aa700 0x7f57f01a3a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:20.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.195 7f57f4d05700 1 -- 172.21.2.112:0/2493503902 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57f01a4660 con 0x7f57f00a55e0 2021-02-10T11:32:20.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.195 7f57ef7fe700 1 --2- 172.21.2.112:0/2493503902 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f57f01aa700 0x7f57f01a3a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:32:20.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.196 7f57f4d05700 1 --2- 172.21.2.112:0/2493503902 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f57f00a55e0 0x7f57f01a4040 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 rx=0x7f57e8009960 tx=0x7f57e800bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:20.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.196 7f57ecff9700 1 -- 172.21.2.112:0/2493503902 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f57e8003030 con 0x7f57f00a55e0 2021-02-10T11:32:20.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.196 7f57ecff9700 1 -- 172.21.2.112:0/2493503902 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57e8001310 con 0x7f57f00a55e0 2021-02-10T11:32:20.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.196 7f57ecff9700 1 -- 172.21.2.112:0/2493503902 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f57e8001770 con 0x7f57f00a55e0 2021-02-10T11:32:20.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.196 7f57f66e8700 1 -- 172.21.2.112:0/2493503902 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57f01a4660 con 0x7f57f00a55e0 2021-02-10T11:32:20.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.196 7f57f66e8700 1 -- 172.21.2.112:0/2493503902 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57f01a4bf0 con 0x7f57f00a55e0 2021-02-10T11:32:20.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.197 7f57ecff9700 1 -- 172.21.2.112:0/2493503902 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f57e80014f0 con 0x7f57f00a55e0 2021-02-10T11:32:20.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.197 7f57ecff9700 1 --2- 172.21.2.112:0/2493503902 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f57d00186c0 0x7f57d001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:20.189 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.197 7f57ecff9700 1 -- 172.21.2.112:0/2493503902 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f57e80327d0 con 0x7f57f00a55e0 2021-02-10T11:32:20.189 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.197 7f57effff700 1 --2- 172.21.2.112:0/2493503902 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f57d00186c0 0x7f57d001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:20.189 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.197 7f57effff700 1 --2- 172.21.2.112:0/2493503902 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f57d00186c0 0x7f57d001abc0 crc :-1 s=READY pgs=550 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:20.216 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.228 7f57f66e8700 1 -- 172.21.2.112:0/2493503902 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57f0083da0 con 0x7f57f00a55e0 2021-02-10T11:32:20.218 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.231 7f57ecff9700 1 -- 172.21.2.112:0/2493503902 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f57e8020f20 con 0x7f57f00a55e0 2021-02-10T11:32:20.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.356 7f57f66e8700 1 -- 172.21.2.112:0/2493503902 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f57f007ed80 con 0x7f57f00a55e0 2021-02-10T11:32:20.344 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.357 7f57ecff9700 1 -- 172.21.2.112:0/2493503902 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v38) v1 ==== 81+0+30360 (secure 0 0 0) 0x7f57e8020f20 con 0x7f57f00a55e0 2021-02-10T11:32:20.345 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:32:20.345 INFO:teuthology.orchestra.run.gibba012.stdout:{ 2021-02-10T11:32:20.345 INFO:teuthology.orchestra.run.gibba012.stdout: "epoch": 38, 2021-02-10T11:32:20.345 INFO:teuthology.orchestra.run.gibba012.stdout: "fsid": "f14f004e-0699-4584-ad81-3754258493de", 2021-02-10T11:32:20.345 INFO:teuthology.orchestra.run.gibba012.stdout: "created": "2021-02-10 11:13:26.501424", 2021-02-10T11:32:20.345 INFO:teuthology.orchestra.run.gibba012.stdout: "modified": "2021-02-10 11:31:41.192016", 2021-02-10T11:32:20.345 INFO:teuthology.orchestra.run.gibba012.stdout: "last_up_change": "2021-02-10 11:13:52.780316", 2021-02-10T11:32:20.345 INFO:teuthology.orchestra.run.gibba012.stdout: "last_in_change": "2021-02-10 11:13:52.780316", 2021-02-10T11:32:20.345 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-02-10T11:32:20.345 INFO:teuthology.orchestra.run.gibba012.stdout: "flags_num": 5799936, 2021-02-10T11:32:20.346 INFO:teuthology.orchestra.run.gibba012.stdout: "flags_set": [ 2021-02-10T11:32:20.346 INFO:teuthology.orchestra.run.gibba012.stdout: "pglog_hardlimit", 2021-02-10T11:32:20.346 INFO:teuthology.orchestra.run.gibba012.stdout: "purged_snapdirs", 2021-02-10T11:32:20.346 INFO:teuthology.orchestra.run.gibba012.stdout: "recovery_deletes", 2021-02-10T11:32:20.346 INFO:teuthology.orchestra.run.gibba012.stdout: "sortbitwise" 2021-02-10T11:32:20.346 INFO:teuthology.orchestra.run.gibba012.stdout: ], 2021-02-10T11:32:20.346 INFO:teuthology.orchestra.run.gibba012.stdout: "crush_version": 4, 2021-02-10T11:32:20.346 INFO:teuthology.orchestra.run.gibba012.stdout: "full_ratio": 0.94999998807907104, 2021-02-10T11:32:20.346 INFO:teuthology.orchestra.run.gibba012.stdout: "backfillfull_ratio": 0.89999997615814209, 2021-02-10T11:32:20.346 INFO:teuthology.orchestra.run.gibba012.stdout: "nearfull_ratio": 0.85000002384185791, 2021-02-10T11:32:20.346 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_snapshot": "", 2021-02-10T11:32:20.347 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_max": 7, 2021-02-10T11:32:20.347 INFO:teuthology.orchestra.run.gibba012.stdout: "max_osd": 8, 2021-02-10T11:32:20.347 INFO:teuthology.orchestra.run.gibba012.stdout: "require_min_compat_client": "jewel", 2021-02-10T11:32:20.347 INFO:teuthology.orchestra.run.gibba012.stdout: "min_compat_client": "jewel", 2021-02-10T11:32:20.347 INFO:teuthology.orchestra.run.gibba012.stdout: "require_osd_release": "nautilus", 2021-02-10T11:32:20.347 INFO:teuthology.orchestra.run.gibba012.stdout: "pools": [ 2021-02-10T11:32:20.347 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:32:20.347 INFO:teuthology.orchestra.run.gibba012.stdout: "pool": 1, 2021-02-10T11:32:20.347 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_name": "rbd", 2021-02-10T11:32:20.347 INFO:teuthology.orchestra.run.gibba012.stdout: "create_time": "2021-02-10 11:13:57.508758", 2021-02-10T11:32:20.347 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:32:20.347 INFO:teuthology.orchestra.run.gibba012.stdout: "flags_names": "hashpspool", 2021-02-10T11:32:20.348 INFO:teuthology.orchestra.run.gibba012.stdout: "type": 1, 2021-02-10T11:32:20.348 INFO:teuthology.orchestra.run.gibba012.stdout: "size": 2, 2021-02-10T11:32:20.348 INFO:teuthology.orchestra.run.gibba012.stdout: "min_size": 1, 2021-02-10T11:32:20.348 INFO:teuthology.orchestra.run.gibba012.stdout: "crush_rule": 0, 2021-02-10T11:32:20.348 INFO:teuthology.orchestra.run.gibba012.stdout: "object_hash": 2, 2021-02-10T11:32:20.348 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:32:20.348 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num": 8, 2021-02-10T11:32:20.348 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num": 8, 2021-02-10T11:32:20.348 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num_target": 8, 2021-02-10T11:32:20.348 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_target": 8, 2021-02-10T11:32:20.348 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_pending": 8, 2021-02-10T11:32:20.348 INFO:teuthology.orchestra.run.gibba012.stdout: "last_pg_merge_meta": { 2021-02-10T11:32:20.349 INFO:teuthology.orchestra.run.gibba012.stdout: "source_pgid": "0.0", 2021-02-10T11:32:20.349 INFO:teuthology.orchestra.run.gibba012.stdout: "ready_epoch": 0, 2021-02-10T11:32:20.349 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_started": 0, 2021-02-10T11:32:20.349 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_clean": 0, 2021-02-10T11:32:20.349 INFO:teuthology.orchestra.run.gibba012.stdout: "source_version": "0'0", 2021-02-10T11:32:20.349 INFO:teuthology.orchestra.run.gibba012.stdout: "target_version": "0'0" 2021-02-10T11:32:20.349 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:32:20.349 INFO:teuthology.orchestra.run.gibba012.stdout: "last_change": "16", 2021-02-10T11:32:20.349 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend": "0", 2021-02-10T11:32:20.349 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:32:20.349 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:32:20.349 INFO:teuthology.orchestra.run.gibba012.stdout: "auid": 0, 2021-02-10T11:32:20.349 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:32:20.350 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_seq": 0, 2021-02-10T11:32:20.350 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_epoch": 0, 2021-02-10T11:32:20.350 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_snaps": [], 2021-02-10T11:32:20.350 INFO:teuthology.orchestra.run.gibba012.stdout: "removed_snaps": "[]", 2021-02-10T11:32:20.350 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_bytes": 0, 2021-02-10T11:32:20.350 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_objects": 0, 2021-02-10T11:32:20.350 INFO:teuthology.orchestra.run.gibba012.stdout: "tiers": [], 2021-02-10T11:32:20.350 INFO:teuthology.orchestra.run.gibba012.stdout: "tier_of": -1, 2021-02-10T11:32:20.350 INFO:teuthology.orchestra.run.gibba012.stdout: "read_tier": -1, 2021-02-10T11:32:20.350 INFO:teuthology.orchestra.run.gibba012.stdout: "write_tier": -1, 2021-02-10T11:32:20.350 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_mode": "none", 2021-02-10T11:32:20.351 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_bytes": 0, 2021-02-10T11:32:20.351 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_objects": 0, 2021-02-10T11:32:20.351 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:32:20.351 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:32:20.351 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:32:20.351 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_flush_age": 0, 2021-02-10T11:32:20.351 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_evict_age": 0, 2021-02-10T11:32:20.351 INFO:teuthology.orchestra.run.gibba012.stdout: "erasure_code_profile": "", 2021-02-10T11:32:20.351 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_params": { 2021-02-10T11:32:20.351 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "none" 2021-02-10T11:32:20.351 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:32:20.351 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_period": 0, 2021-02-10T11:32:20.352 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_count": 0, 2021-02-10T11:32:20.352 INFO:teuthology.orchestra.run.gibba012.stdout: "use_gmt_hitset": true, 2021-02-10T11:32:20.352 INFO:teuthology.orchestra.run.gibba012.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:32:20.352 INFO:teuthology.orchestra.run.gibba012.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:32:20.352 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:32:20.352 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:32:20.352 INFO:teuthology.orchestra.run.gibba012.stdout: "grade_table": [], 2021-02-10T11:32:20.352 INFO:teuthology.orchestra.run.gibba012.stdout: "stripe_width": 0, 2021-02-10T11:32:20.352 INFO:teuthology.orchestra.run.gibba012.stdout: "expected_num_objects": 0, 2021-02-10T11:32:20.352 INFO:teuthology.orchestra.run.gibba012.stdout: "fast_read": false, 2021-02-10T11:32:20.352 INFO:teuthology.orchestra.run.gibba012.stdout: "options": {}, 2021-02-10T11:32:20.352 INFO:teuthology.orchestra.run.gibba012.stdout: "application_metadata": { 2021-02-10T11:32:20.353 INFO:teuthology.orchestra.run.gibba012.stdout: "rbd": {} 2021-02-10T11:32:20.353 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:32:20.353 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:32:20.353 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:32:20.353 INFO:teuthology.orchestra.run.gibba012.stdout: "pool": 6, 2021-02-10T11:32:20.353 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_name": "cephfs_metadata", 2021-02-10T11:32:20.353 INFO:teuthology.orchestra.run.gibba012.stdout: "create_time": "2021-02-10 11:31:38.162654", 2021-02-10T11:32:20.353 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:32:20.353 INFO:teuthology.orchestra.run.gibba012.stdout: "flags_names": "hashpspool", 2021-02-10T11:32:20.353 INFO:teuthology.orchestra.run.gibba012.stdout: "type": 1, 2021-02-10T11:32:20.354 INFO:teuthology.orchestra.run.gibba012.stdout: "size": 2, 2021-02-10T11:32:20.354 INFO:teuthology.orchestra.run.gibba012.stdout: "min_size": 1, 2021-02-10T11:32:20.354 INFO:teuthology.orchestra.run.gibba012.stdout: "crush_rule": 0, 2021-02-10T11:32:20.354 INFO:teuthology.orchestra.run.gibba012.stdout: "object_hash": 2, 2021-02-10T11:32:20.354 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:32:20.354 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num": 8, 2021-02-10T11:32:20.354 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num": 8, 2021-02-10T11:32:20.354 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num_target": 8, 2021-02-10T11:32:20.354 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_target": 8, 2021-02-10T11:32:20.354 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_pending": 8, 2021-02-10T11:32:20.354 INFO:teuthology.orchestra.run.gibba012.stdout: "last_pg_merge_meta": { 2021-02-10T11:32:20.355 INFO:teuthology.orchestra.run.gibba012.stdout: "source_pgid": "0.0", 2021-02-10T11:32:20.355 INFO:teuthology.orchestra.run.gibba012.stdout: "ready_epoch": 0, 2021-02-10T11:32:20.355 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_started": 0, 2021-02-10T11:32:20.355 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_clean": 0, 2021-02-10T11:32:20.355 INFO:teuthology.orchestra.run.gibba012.stdout: "source_version": "0'0", 2021-02-10T11:32:20.355 INFO:teuthology.orchestra.run.gibba012.stdout: "target_version": "0'0" 2021-02-10T11:32:20.355 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:32:20.355 INFO:teuthology.orchestra.run.gibba012.stdout: "last_change": "37", 2021-02-10T11:32:20.355 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend": "0", 2021-02-10T11:32:20.355 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:32:20.355 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:32:20.355 INFO:teuthology.orchestra.run.gibba012.stdout: "auid": 0, 2021-02-10T11:32:20.355 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:32:20.355 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_seq": 0, 2021-02-10T11:32:20.356 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_epoch": 0, 2021-02-10T11:32:20.356 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_snaps": [], 2021-02-10T11:32:20.356 INFO:teuthology.orchestra.run.gibba012.stdout: "removed_snaps": "[]", 2021-02-10T11:32:20.356 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_bytes": 0, 2021-02-10T11:32:20.356 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_objects": 0, 2021-02-10T11:32:20.356 INFO:teuthology.orchestra.run.gibba012.stdout: "tiers": [], 2021-02-10T11:32:20.356 INFO:teuthology.orchestra.run.gibba012.stdout: "tier_of": -1, 2021-02-10T11:32:20.356 INFO:teuthology.orchestra.run.gibba012.stdout: "read_tier": -1, 2021-02-10T11:32:20.356 INFO:teuthology.orchestra.run.gibba012.stdout: "write_tier": -1, 2021-02-10T11:32:20.356 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_mode": "none", 2021-02-10T11:32:20.356 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_bytes": 0, 2021-02-10T11:32:20.356 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_objects": 0, 2021-02-10T11:32:20.357 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:32:20.357 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:32:20.357 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:32:20.357 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_flush_age": 0, 2021-02-10T11:32:20.357 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_evict_age": 0, 2021-02-10T11:32:20.357 INFO:teuthology.orchestra.run.gibba012.stdout: "erasure_code_profile": "", 2021-02-10T11:32:20.357 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_params": { 2021-02-10T11:32:20.357 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "none" 2021-02-10T11:32:20.357 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:32:20.357 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_period": 0, 2021-02-10T11:32:20.357 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_count": 0, 2021-02-10T11:32:20.357 INFO:teuthology.orchestra.run.gibba012.stdout: "use_gmt_hitset": true, 2021-02-10T11:32:20.357 INFO:teuthology.orchestra.run.gibba012.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:32:20.358 INFO:teuthology.orchestra.run.gibba012.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:32:20.358 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:32:20.358 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:32:20.358 INFO:teuthology.orchestra.run.gibba012.stdout: "grade_table": [], 2021-02-10T11:32:20.358 INFO:teuthology.orchestra.run.gibba012.stdout: "stripe_width": 0, 2021-02-10T11:32:20.358 INFO:teuthology.orchestra.run.gibba012.stdout: "expected_num_objects": 0, 2021-02-10T11:32:20.358 INFO:teuthology.orchestra.run.gibba012.stdout: "fast_read": false, 2021-02-10T11:32:20.358 INFO:teuthology.orchestra.run.gibba012.stdout: "options": { 2021-02-10T11:32:20.358 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_autoscale_bias": 4, 2021-02-10T11:32:20.358 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_min": 16, 2021-02-10T11:32:20.358 INFO:teuthology.orchestra.run.gibba012.stdout: "recovery_priority": 5 2021-02-10T11:32:20.358 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:32:20.359 INFO:teuthology.orchestra.run.gibba012.stdout: "application_metadata": { 2021-02-10T11:32:20.359 INFO:teuthology.orchestra.run.gibba012.stdout: "cephfs": { 2021-02-10T11:32:20.359 INFO:teuthology.orchestra.run.gibba012.stdout: "metadata": "cephfs" 2021-02-10T11:32:20.359 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:32:20.359 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:32:20.359 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:32:20.359 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:32:20.359 INFO:teuthology.orchestra.run.gibba012.stdout: "pool": 7, 2021-02-10T11:32:20.359 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_name": "cephfs_data", 2021-02-10T11:32:20.359 INFO:teuthology.orchestra.run.gibba012.stdout: "create_time": "2021-02-10 11:31:39.233986", 2021-02-10T11:32:20.359 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:32:20.359 INFO:teuthology.orchestra.run.gibba012.stdout: "flags_names": "hashpspool", 2021-02-10T11:32:20.359 INFO:teuthology.orchestra.run.gibba012.stdout: "type": 1, 2021-02-10T11:32:20.360 INFO:teuthology.orchestra.run.gibba012.stdout: "size": 2, 2021-02-10T11:32:20.360 INFO:teuthology.orchestra.run.gibba012.stdout: "min_size": 1, 2021-02-10T11:32:20.360 INFO:teuthology.orchestra.run.gibba012.stdout: "crush_rule": 0, 2021-02-10T11:32:20.360 INFO:teuthology.orchestra.run.gibba012.stdout: "object_hash": 2, 2021-02-10T11:32:20.360 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:32:20.360 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num": 8, 2021-02-10T11:32:20.360 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num": 8, 2021-02-10T11:32:20.360 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num_target": 8, 2021-02-10T11:32:20.360 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_target": 8, 2021-02-10T11:32:20.360 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_pending": 8, 2021-02-10T11:32:20.360 INFO:teuthology.orchestra.run.gibba012.stdout: "last_pg_merge_meta": { 2021-02-10T11:32:20.360 INFO:teuthology.orchestra.run.gibba012.stdout: "source_pgid": "0.0", 2021-02-10T11:32:20.361 INFO:teuthology.orchestra.run.gibba012.stdout: "ready_epoch": 0, 2021-02-10T11:32:20.361 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_started": 0, 2021-02-10T11:32:20.361 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_clean": 0, 2021-02-10T11:32:20.361 INFO:teuthology.orchestra.run.gibba012.stdout: "source_version": "0'0", 2021-02-10T11:32:20.361 INFO:teuthology.orchestra.run.gibba012.stdout: "target_version": "0'0" 2021-02-10T11:32:20.361 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:32:20.361 INFO:teuthology.orchestra.run.gibba012.stdout: "last_change": "38", 2021-02-10T11:32:20.361 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend": "0", 2021-02-10T11:32:20.361 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:32:20.361 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:32:20.361 INFO:teuthology.orchestra.run.gibba012.stdout: "auid": 0, 2021-02-10T11:32:20.361 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:32:20.362 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_seq": 0, 2021-02-10T11:32:20.362 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_epoch": 0, 2021-02-10T11:32:20.362 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_snaps": [], 2021-02-10T11:32:20.362 INFO:teuthology.orchestra.run.gibba012.stdout: "removed_snaps": "[]", 2021-02-10T11:32:20.362 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_bytes": 0, 2021-02-10T11:32:20.362 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_objects": 0, 2021-02-10T11:32:20.362 INFO:teuthology.orchestra.run.gibba012.stdout: "tiers": [], 2021-02-10T11:32:20.362 INFO:teuthology.orchestra.run.gibba012.stdout: "tier_of": -1, 2021-02-10T11:32:20.362 INFO:teuthology.orchestra.run.gibba012.stdout: "read_tier": -1, 2021-02-10T11:32:20.362 INFO:teuthology.orchestra.run.gibba012.stdout: "write_tier": -1, 2021-02-10T11:32:20.362 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_mode": "none", 2021-02-10T11:32:20.362 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_bytes": 0, 2021-02-10T11:32:20.363 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_objects": 0, 2021-02-10T11:32:20.363 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:32:20.363 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:32:20.363 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:32:20.363 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_flush_age": 0, 2021-02-10T11:32:20.363 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_evict_age": 0, 2021-02-10T11:32:20.363 INFO:teuthology.orchestra.run.gibba012.stdout: "erasure_code_profile": "", 2021-02-10T11:32:20.363 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_params": { 2021-02-10T11:32:20.363 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "none" 2021-02-10T11:32:20.363 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:32:20.363 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_period": 0, 2021-02-10T11:32:20.363 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_count": 0, 2021-02-10T11:32:20.363 INFO:teuthology.orchestra.run.gibba012.stdout: "use_gmt_hitset": true, 2021-02-10T11:32:20.364 INFO:teuthology.orchestra.run.gibba012.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:32:20.364 INFO:teuthology.orchestra.run.gibba012.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:32:20.364 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:32:20.364 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:32:20.364 INFO:teuthology.orchestra.run.gibba012.stdout: "grade_table": [], 2021-02-10T11:32:20.364 INFO:teuthology.orchestra.run.gibba012.stdout: "stripe_width": 0, 2021-02-10T11:32:20.364 INFO:teuthology.orchestra.run.gibba012.stdout: "expected_num_objects": 0, 2021-02-10T11:32:20.364 INFO:teuthology.orchestra.run.gibba012.stdout: "fast_read": false, 2021-02-10T11:32:20.364 INFO:teuthology.orchestra.run.gibba012.stdout: "options": {}, 2021-02-10T11:32:20.364 INFO:teuthology.orchestra.run.gibba012.stdout: "application_metadata": { 2021-02-10T11:32:20.364 INFO:teuthology.orchestra.run.gibba012.stdout: "cephfs": { 2021-02-10T11:32:20.364 INFO:teuthology.orchestra.run.gibba012.stdout: "data": "cephfs" 2021-02-10T11:32:20.365 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:32:20.365 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:32:20.365 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:32:20.365 INFO:teuthology.orchestra.run.gibba012.stdout: ], 2021-02-10T11:32:20.365 INFO:teuthology.orchestra.run.gibba012.stdout: "osds": [ 2021-02-10T11:32:20.365 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:32:20.365 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 0, 2021-02-10T11:32:20.365 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "95444b8f-6370-4a1d-ae5f-df62651b77da", 2021-02-10T11:32:20.365 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:32:20.365 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:32:20.365 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:32:20.365 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:32:20.366 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:32:20.366 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:32:20.366 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:32:20.366 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 36, 2021-02-10T11:32:20.366 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:32:20.366 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:32:20.366 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:32:20.366 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:32:20.366 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:32:20.366 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:32:20.366 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6817", 2021-02-10T11:32:20.366 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:32:20.367 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:32:20.367 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:32:20.367 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:32:20.367 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6818", 2021-02-10T11:32:20.367 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:32:20.367 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:32:20.367 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:32:20.367 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:32:20.367 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:32:20.367 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:32:20.367 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:32:20.367 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:32:20.368 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6820", 2021-02-10T11:32:20.368 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:32:20.368 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:32:20.368 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:32:20.368 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:32:20.368 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6822", 2021-02-10T11:32:20.368 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:32:20.368 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:32:20.368 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:32:20.368 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:32:20.368 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:32:20.368 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:32:20.369 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:32:20.369 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:32:20.369 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6828", 2021-02-10T11:32:20.369 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:32:20.369 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:32:20.369 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:32:20.369 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:32:20.369 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6830", 2021-02-10T11:32:20.369 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:32:20.369 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:32:20.369 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:32:20.369 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:32:20.369 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:32:20.369 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:32:20.370 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:32:20.370 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:32:20.370 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6824", 2021-02-10T11:32:20.370 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:32:20.370 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:32:20.370 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:32:20.370 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:32:20.370 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6826", 2021-02-10T11:32:20.370 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:32:20.370 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:32:20.370 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:32:20.370 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:32:20.371 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.112:6818/36030", 2021-02-10T11:32:20.371 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.112:6822/36030", 2021-02-10T11:32:20.371 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.112:6830/36030", 2021-02-10T11:32:20.371 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.112:6826/36030", 2021-02-10T11:32:20.371 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:32:20.371 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:32:20.371 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:32:20.371 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:32:20.371 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:32:20.371 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:32:20.371 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 1, 2021-02-10T11:32:20.371 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "a23e217e-0d8e-486e-be07-712adfabd262", 2021-02-10T11:32:20.372 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:32:20.372 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:32:20.372 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:32:20.372 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:32:20.372 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:32:20.372 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:32:20.372 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:32:20.372 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 35, 2021-02-10T11:32:20.372 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:32:20.372 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:32:20.372 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:32:20.372 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:32:20.372 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:32:20.373 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:32:20.373 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6800", 2021-02-10T11:32:20.373 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:32:20.373 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:32:20.373 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:32:20.373 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:32:20.373 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6802", 2021-02-10T11:32:20.373 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:32:20.373 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:32:20.373 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:32:20.373 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:32:20.373 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:32:20.373 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:32:20.374 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:32:20.374 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:32:20.374 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6804", 2021-02-10T11:32:20.374 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:32:20.374 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:32:20.374 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:32:20.374 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:32:20.374 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6806", 2021-02-10T11:32:20.374 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:32:20.374 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:32:20.374 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:32:20.374 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:32:20.375 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:32:20.375 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:32:20.375 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:32:20.375 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:32:20.375 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6812", 2021-02-10T11:32:20.375 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:32:20.375 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:32:20.375 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:32:20.375 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:32:20.375 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6814", 2021-02-10T11:32:20.375 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:32:20.375 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:32:20.376 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:32:20.376 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:32:20.376 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:32:20.376 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:32:20.376 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:32:20.376 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:32:20.376 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6808", 2021-02-10T11:32:20.376 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:32:20.376 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:32:20.376 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:32:20.376 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:32:20.376 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6810", 2021-02-10T11:32:20.377 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:32:20.377 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:32:20.377 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:32:20.377 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:32:20.377 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.112:6802/36029", 2021-02-10T11:32:20.377 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.112:6806/36029", 2021-02-10T11:32:20.377 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.112:6814/36029", 2021-02-10T11:32:20.377 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.112:6810/36029", 2021-02-10T11:32:20.377 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:32:20.377 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:32:20.377 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:32:20.377 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:32:20.378 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:32:20.378 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:32:20.378 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 2, 2021-02-10T11:32:20.378 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "026afa8a-c0f1-4b35-9a76-53c00fe0c2fc", 2021-02-10T11:32:20.378 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:32:20.378 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:32:20.378 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:32:20.378 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:32:20.378 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:32:20.378 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:32:20.378 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:32:20.378 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 27, 2021-02-10T11:32:20.379 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:32:20.379 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:32:20.379 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:32:20.379 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:32:20.379 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:32:20.379 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:32:20.379 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6801", 2021-02-10T11:32:20.379 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:32:20.379 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:32:20.379 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:32:20.379 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:32:20.379 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6803", 2021-02-10T11:32:20.379 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:32:20.379 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:32:20.380 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:32:20.380 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:32:20.380 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:32:20.380 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:32:20.380 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:32:20.380 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:32:20.380 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6805", 2021-02-10T11:32:20.380 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:32:20.380 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:32:20.380 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:32:20.380 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:32:20.380 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6807", 2021-02-10T11:32:20.381 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:32:20.381 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:32:20.381 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:32:20.381 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:32:20.381 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:32:20.381 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:32:20.381 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:32:20.381 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:32:20.381 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6813", 2021-02-10T11:32:20.381 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:32:20.381 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:32:20.381 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:32:20.382 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:32:20.382 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6815", 2021-02-10T11:32:20.382 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:32:20.382 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:32:20.382 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:32:20.382 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:32:20.382 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:32:20.382 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:32:20.382 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:32:20.382 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:32:20.382 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6809", 2021-02-10T11:32:20.382 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:32:20.383 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:32:20.383 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:32:20.383 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:32:20.383 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6811", 2021-02-10T11:32:20.383 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:32:20.383 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:32:20.383 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:32:20.383 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:32:20.383 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.112:6803/36031", 2021-02-10T11:32:20.383 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.112:6807/36031", 2021-02-10T11:32:20.383 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.112:6815/36031", 2021-02-10T11:32:20.383 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.112:6811/36031", 2021-02-10T11:32:20.383 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:32:20.384 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:32:20.384 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:32:20.384 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:32:20.384 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:32:20.384 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:32:20.384 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 3, 2021-02-10T11:32:20.384 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "58795682-e00d-4173-bb84-f7075ca89a11", 2021-02-10T11:32:20.384 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:32:20.384 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:32:20.384 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:32:20.384 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:32:20.385 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:32:20.385 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:32:20.385 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:32:20.385 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 36, 2021-02-10T11:32:20.385 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:32:20.385 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:32:20.385 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:32:20.385 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:32:20.385 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:32:20.385 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:32:20.385 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6816", 2021-02-10T11:32:20.385 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:32:20.386 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:32:20.386 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:32:20.386 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:32:20.386 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6819", 2021-02-10T11:32:20.386 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:32:20.386 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:32:20.386 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:32:20.386 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:32:20.386 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:32:20.386 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:32:20.386 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:32:20.386 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:32:20.386 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6821", 2021-02-10T11:32:20.387 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:32:20.387 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:32:20.387 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:32:20.387 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:32:20.387 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6823", 2021-02-10T11:32:20.387 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:32:20.387 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:32:20.387 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:32:20.387 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:32:20.387 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:32:20.387 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:32:20.387 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:32:20.388 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:32:20.388 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6829", 2021-02-10T11:32:20.388 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:32:20.388 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:32:20.388 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:32:20.388 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:32:20.388 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6831", 2021-02-10T11:32:20.388 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:32:20.388 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:32:20.388 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:32:20.388 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:32:20.388 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:32:20.389 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:32:20.389 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:32:20.389 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:32:20.389 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6825", 2021-02-10T11:32:20.389 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:32:20.389 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:32:20.389 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:32:20.389 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:32:20.389 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6827", 2021-02-10T11:32:20.389 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:32:20.389 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:32:20.389 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:32:20.390 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:32:20.390 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.112:6819/36028", 2021-02-10T11:32:20.390 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.112:6823/36028", 2021-02-10T11:32:20.390 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.112:6831/36028", 2021-02-10T11:32:20.390 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.112:6827/36028", 2021-02-10T11:32:20.390 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:32:20.390 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:32:20.390 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:32:20.390 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:32:20.390 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:32:20.390 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:32:20.390 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 4, 2021-02-10T11:32:20.390 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "6763f8d6-53dc-4e45-bdf6-a6eabcf81e39", 2021-02-10T11:32:20.391 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:32:20.391 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:32:20.391 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:32:20.391 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:32:20.391 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:32:20.391 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:32:20.391 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:32:20.391 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 36, 2021-02-10T11:32:20.391 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:32:20.391 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:32:20.391 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:32:20.392 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:32:20.392 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:32:20.392 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:32:20.392 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6810", 2021-02-10T11:32:20.392 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:32:20.392 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:32:20.392 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:32:20.392 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:32:20.392 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6813", 2021-02-10T11:32:20.392 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:32:20.392 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:32:20.392 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:32:20.392 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:32:20.393 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:32:20.393 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:32:20.393 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:32:20.393 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:32:20.393 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6816", 2021-02-10T11:32:20.393 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:32:20.393 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:32:20.393 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:32:20.393 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:32:20.393 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6819", 2021-02-10T11:32:20.393 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:32:20.393 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:32:20.394 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:32:20.394 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:32:20.394 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:32:20.394 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:32:20.394 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:32:20.394 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:32:20.394 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6825", 2021-02-10T11:32:20.394 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:32:20.394 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:32:20.394 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:32:20.394 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:32:20.394 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6827", 2021-02-10T11:32:20.395 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:32:20.395 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:32:20.395 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:32:20.395 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:32:20.395 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:32:20.395 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:32:20.395 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:32:20.395 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:32:20.395 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6821", 2021-02-10T11:32:20.395 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:32:20.395 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:32:20.395 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:32:20.396 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:32:20.396 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6823", 2021-02-10T11:32:20.396 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:32:20.396 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:32:20.396 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:32:20.396 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:32:20.396 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.120:6813/36388", 2021-02-10T11:32:20.396 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.120:6819/36388", 2021-02-10T11:32:20.396 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.120:6827/36388", 2021-02-10T11:32:20.396 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.120:6823/36388", 2021-02-10T11:32:20.396 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:32:20.396 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:32:20.396 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:32:20.397 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:32:20.397 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:32:20.397 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:32:20.397 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 5, 2021-02-10T11:32:20.397 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "5d0e4dd6-0cb0-4bf8-8314-626acaa1b544", 2021-02-10T11:32:20.397 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:32:20.397 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:32:20.397 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:32:20.397 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:32:20.397 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:32:20.397 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:32:20.397 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:32:20.397 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 36, 2021-02-10T11:32:20.398 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:32:20.398 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:32:20.398 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:32:20.398 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:32:20.398 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:32:20.398 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:32:20.398 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6804", 2021-02-10T11:32:20.398 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:32:20.398 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:32:20.398 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:32:20.398 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:32:20.398 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6806", 2021-02-10T11:32:20.399 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:32:20.399 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:32:20.399 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:32:20.399 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:32:20.399 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:32:20.399 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:32:20.399 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:32:20.399 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:32:20.399 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6809", 2021-02-10T11:32:20.399 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:32:20.399 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:32:20.399 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:32:20.400 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:32:20.400 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6812", 2021-02-10T11:32:20.400 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:32:20.400 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:32:20.400 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:32:20.400 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:32:20.400 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:32:20.400 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:32:20.400 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:32:20.400 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:32:20.400 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6820", 2021-02-10T11:32:20.400 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:32:20.400 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:32:20.401 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:32:20.401 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:32:20.401 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6822", 2021-02-10T11:32:20.401 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:32:20.401 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:32:20.401 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:32:20.401 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:32:20.401 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:32:20.401 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:32:20.401 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:32:20.401 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:32:20.401 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6815", 2021-02-10T11:32:20.401 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:32:20.402 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:32:20.402 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:32:20.402 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:32:20.402 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6818", 2021-02-10T11:32:20.402 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:32:20.402 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:32:20.402 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:32:20.402 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:32:20.402 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.120:6806/36387", 2021-02-10T11:32:20.402 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.120:6812/36387", 2021-02-10T11:32:20.402 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.120:6822/36387", 2021-02-10T11:32:20.403 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.120:6818/36387", 2021-02-10T11:32:20.403 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:32:20.403 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:32:20.403 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:32:20.403 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:32:20.403 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:32:20.403 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:32:20.403 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 6, 2021-02-10T11:32:20.403 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "b010e187-4541-4b00-80dd-f948f6ab8f15", 2021-02-10T11:32:20.403 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:32:20.403 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:32:20.403 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:32:20.403 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:32:20.404 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:32:20.404 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:32:20.404 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:32:20.404 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 36, 2021-02-10T11:32:20.404 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:32:20.404 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:32:20.404 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:32:20.404 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:32:20.404 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:32:20.404 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:32:20.404 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6824", 2021-02-10T11:32:20.404 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:32:20.404 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:32:20.405 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:32:20.405 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:32:20.405 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6826", 2021-02-10T11:32:20.405 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:32:20.405 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:32:20.405 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:32:20.405 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:32:20.405 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:32:20.405 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:32:20.405 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:32:20.405 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:32:20.405 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6828", 2021-02-10T11:32:20.406 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:32:20.406 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:32:20.406 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:32:20.406 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:32:20.406 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6829", 2021-02-10T11:32:20.406 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:32:20.406 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:32:20.406 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:32:20.406 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:32:20.406 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:32:20.406 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:32:20.406 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:32:20.407 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:32:20.407 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6832", 2021-02-10T11:32:20.407 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:32:20.407 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:32:20.407 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:32:20.407 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:32:20.407 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6833", 2021-02-10T11:32:20.407 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:32:20.407 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:32:20.407 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:32:20.407 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:32:20.407 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:32:20.407 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:32:20.408 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:32:20.408 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:32:20.408 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6830", 2021-02-10T11:32:20.408 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:32:20.408 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:32:20.408 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:32:20.408 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:32:20.408 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6831", 2021-02-10T11:32:20.408 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:32:20.408 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:32:20.408 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:32:20.408 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:32:20.409 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.120:6826/36385", 2021-02-10T11:32:20.409 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.120:6829/36385", 2021-02-10T11:32:20.409 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.120:6833/36385", 2021-02-10T11:32:20.409 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.120:6831/36385", 2021-02-10T11:32:20.409 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:32:20.409 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:32:20.409 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:32:20.409 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:32:20.409 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:32:20.409 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:32:20.409 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 7, 2021-02-10T11:32:20.409 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "671e2772-33b7-497e-8e44-2178bd850bcf", 2021-02-10T11:32:20.409 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:32:20.410 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:32:20.410 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:32:20.410 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:32:20.410 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:32:20.410 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:32:20.410 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:32:20.410 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 36, 2021-02-10T11:32:20.410 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:32:20.410 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:32:20.410 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:32:20.410 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:32:20.410 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:32:20.411 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:32:20.411 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6802", 2021-02-10T11:32:20.411 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:32:20.411 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:32:20.411 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:32:20.411 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:32:20.411 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6803", 2021-02-10T11:32:20.411 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:32:20.411 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:32:20.411 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:32:20.411 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:32:20.411 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:32:20.411 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:32:20.412 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:32:20.412 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:32:20.412 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6805", 2021-02-10T11:32:20.412 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:32:20.412 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:32:20.412 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:32:20.412 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:32:20.412 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6807", 2021-02-10T11:32:20.412 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:32:20.412 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:32:20.412 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:32:20.412 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:32:20.413 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:32:20.413 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:32:20.413 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:32:20.413 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:32:20.413 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6814", 2021-02-10T11:32:20.413 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:32:20.413 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:32:20.413 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:32:20.413 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:32:20.413 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6817", 2021-02-10T11:32:20.413 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:32:20.414 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:32:20.414 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:32:20.414 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:32:20.414 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:32:20.414 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:32:20.414 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:32:20.414 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:32:20.414 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6808", 2021-02-10T11:32:20.414 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:32:20.414 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:32:20.414 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:32:20.414 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:32:20.414 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6811", 2021-02-10T11:32:20.415 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:32:20.415 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:32:20.415 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:32:20.415 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:32:20.415 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.120:6803/36386", 2021-02-10T11:32:20.415 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.120:6807/36386", 2021-02-10T11:32:20.415 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.120:6817/36386", 2021-02-10T11:32:20.415 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.120:6811/36386", 2021-02-10T11:32:20.415 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:32:20.415 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:32:20.415 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:32:20.415 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:32:20.416 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:32:20.416 INFO:teuthology.orchestra.run.gibba012.stdout: ], 2021-02-10T11:32:20.416 INFO:teuthology.orchestra.run.gibba012.stdout: "osd_xinfo": [ 2021-02-10T11:32:20.416 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:32:20.416 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 0, 2021-02-10T11:32:20.416 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:32:20.416 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:32:20.416 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:32:20.416 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:32:20.416 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:32:20.416 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:32:20.416 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:32:20.417 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 1, 2021-02-10T11:32:20.417 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:32:20.417 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:32:20.417 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:32:20.417 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:32:20.417 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:32:20.417 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:32:20.417 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:32:20.417 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 2, 2021-02-10T11:32:20.417 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:32:20.417 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:32:20.417 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:32:20.417 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:32:20.418 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:32:20.418 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:32:20.418 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:32:20.418 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 3, 2021-02-10T11:32:20.418 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:32:20.418 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:32:20.418 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:32:20.418 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:32:20.418 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:32:20.418 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:32:20.418 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:32:20.418 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 4, 2021-02-10T11:32:20.418 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:32:20.419 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:32:20.419 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:32:20.419 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:32:20.419 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:32:20.419 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:32:20.419 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:32:20.419 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 5, 2021-02-10T11:32:20.419 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:32:20.419 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:32:20.419 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:32:20.419 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:32:20.419 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:32:20.420 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:32:20.420 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:32:20.420 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 6, 2021-02-10T11:32:20.420 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:32:20.420 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:32:20.420 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:32:20.420 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:32:20.420 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:32:20.420 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:32:20.420 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:32:20.420 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 7, 2021-02-10T11:32:20.420 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:32:20.420 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:32:20.421 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:32:20.421 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:32:20.421 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:32:20.421 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:32:20.421 INFO:teuthology.orchestra.run.gibba012.stdout: ], 2021-02-10T11:32:20.421 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_upmap": [], 2021-02-10T11:32:20.421 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_upmap_items": [], 2021-02-10T11:32:20.421 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_temp": [], 2021-02-10T11:32:20.421 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_temp": [], 2021-02-10T11:32:20.421 INFO:teuthology.orchestra.run.gibba012.stdout: "blacklist": {}, 2021-02-10T11:32:20.421 INFO:teuthology.orchestra.run.gibba012.stdout: "erasure_code_profiles": { 2021-02-10T11:32:20.421 INFO:teuthology.orchestra.run.gibba012.stdout: "default": { 2021-02-10T11:32:20.422 INFO:teuthology.orchestra.run.gibba012.stdout: "crush-failure-domain": "osd", 2021-02-10T11:32:20.422 INFO:teuthology.orchestra.run.gibba012.stdout: "k": "2", 2021-02-10T11:32:20.422 INFO:teuthology.orchestra.run.gibba012.stdout: "m": "1", 2021-02-10T11:32:20.422 INFO:teuthology.orchestra.run.gibba012.stdout: "plugin": "jerasure", 2021-02-10T11:32:20.422 INFO:teuthology.orchestra.run.gibba012.stdout: "ruleset-failure-domain": "osd", 2021-02-10T11:32:20.422 INFO:teuthology.orchestra.run.gibba012.stdout: "technique": "reed_sol_van" 2021-02-10T11:32:20.422 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:32:20.422 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:32:20.422 INFO:teuthology.orchestra.run.gibba012.stdout: "removed_snaps_queue": [], 2021-02-10T11:32:20.422 INFO:teuthology.orchestra.run.gibba012.stdout: "new_removed_snaps": [], 2021-02-10T11:32:20.422 INFO:teuthology.orchestra.run.gibba012.stdout: "new_purged_snaps": [], 2021-02-10T11:32:20.422 INFO:teuthology.orchestra.run.gibba012.stdout: "crush_node_flags": {}, 2021-02-10T11:32:20.422 INFO:teuthology.orchestra.run.gibba012.stdout: "device_class_flags": {} 2021-02-10T11:32:20.423 INFO:teuthology.orchestra.run.gibba012.stdout:} 2021-02-10T11:32:20.424 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.358 7f57f66e8700 1 -- 172.21.2.112:0/2493503902 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f57d00186c0 msgr2=0x7f57d001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:20.424 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.358 7f57f66e8700 1 --2- 172.21.2.112:0/2493503902 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f57d00186c0 0x7f57d001abc0 crc :-1 s=READY pgs=550 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:20.424 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.358 7f57f66e8700 1 -- 172.21.2.112:0/2493503902 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f57f00a55e0 msgr2=0x7f57f01a4040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:20.424 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.358 7f57f66e8700 1 --2- 172.21.2.112:0/2493503902 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f57f00a55e0 0x7f57f01a4040 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 rx=0x7f57e8009960 tx=0x7f57e800bcf0).stop 2021-02-10T11:32:20.424 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.359 7f57f66e8700 1 -- 172.21.2.112:0/2493503902 shutdown_connections 2021-02-10T11:32:20.424 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.359 7f57f66e8700 1 --2- 172.21.2.112:0/2493503902 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f57d00186c0 0x7f57d001abc0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:20.424 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.359 7f57f66e8700 1 --2- 172.21.2.112:0/2493503902 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f57f01aa700 0x7f57f01a3a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:20.424 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.359 7f57f66e8700 1 --2- 172.21.2.112:0/2493503902 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f57f00a55e0 0x7f57f01a4040 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:20.425 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.359 7f57f66e8700 1 --2- 172.21.2.112:0/2493503902 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f57f01a2a90 0x7f57f01aa0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:20.425 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.359 7f57f66e8700 1 -- 172.21.2.112:0/2493503902 >> 172.21.2.112:0/2493503902 conn(0x7f57f008fe10 msgr2=0x7f57f019c020 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:20.425 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.359 7f57f66e8700 1 -- 172.21.2.112:0/2493503902 shutdown_connections 2021-02-10T11:32:20.425 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.359 7f57f66e8700 1 -- 172.21.2.112:0/2493503902 wait complete. 2021-02-10T11:32:20.426 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:32:20.850 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.847 7fdb77297700 1 Processor -- start 2021-02-10T11:32:20.851 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.847 7fdb77297700 1 -- start start 2021-02-10T11:32:20.851 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.847 7fdb77297700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdb700e5470 0x7fdb7007e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:20.851 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.847 7fdb77297700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fdb7007f300 0x7fdb7007f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:20.851 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.847 7fdb77297700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb7007ed80 con 0x7fdb700e5070 2021-02-10T11:32:20.851 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.847 7fdb77297700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb701b4010 con 0x7fdb7007f300 2021-02-10T11:32:20.851 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.847 7fdb77297700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb700d1790 con 0x7fdb700e5470 2021-02-10T11:32:20.851 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.848 7fdb758b4700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fdb7007f300 0x7fdb7007f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:20.851 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.848 7fdb758b4700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fdb7007f300 0x7fdb7007f700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:48768/0 (socket says 172.21.2.112:48768) 2021-02-10T11:32:20.851 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.848 7fdb758b4700 1 -- 172.21.2.112:0/4212040206 learned_addr learned my addr 172.21.2.112:0/4212040206 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:32:20.851 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.848 7fdb748b2700 1 --2- 172.21.2.112:0/4212040206 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdb700e5470 0x7fdb7007e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:20.852 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.848 7fdb758b4700 1 -- 172.21.2.112:0/4212040206 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdb700e5470 msgr2=0x7fdb7007e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:20.852 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.848 7fdb758b4700 1 --2- 172.21.2.112:0/4212040206 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdb700e5470 0x7fdb7007e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:20.852 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.848 7fdb758b4700 1 -- 172.21.2.112:0/4212040206 >> v1:172.21.2.120:6789/0 conn(0x7fdb700e5070 legacy=0x7fdb701b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:20.852 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.848 7fdb758b4700 1 -- 172.21.2.112:0/4212040206 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb700d1980 con 0x7fdb7007f300 2021-02-10T11:32:20.852 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.848 7fdb758b4700 1 --2- 172.21.2.112:0/4212040206 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdb7007f300 0x7fdb7007f700 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 rx=0x7fdb68006f00 tx=0x7fdb68009f10).ready entity=mon.0 client_cookie=cdb1f0bd83a3f2c2 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:20.852 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.849 7fdb63fff700 1 -- 172.21.2.112:0/4212040206 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdb6800b3b0 con 0x7fdb7007f300 2021-02-10T11:32:20.852 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.849 7fdb63fff700 1 -- 172.21.2.112:0/4212040206 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb6800b730 con 0x7fdb7007f300 2021-02-10T11:32:20.852 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.849 7fdb63fff700 1 -- 172.21.2.112:0/4212040206 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdb6800bb90 con 0x7fdb7007f300 2021-02-10T11:32:20.852 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.849 7fdb77297700 1 -- 172.21.2.112:0/4212040206 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdb7007f300 msgr2=0x7fdb7007f700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:20.852 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.849 7fdb77297700 1 --2- 172.21.2.112:0/4212040206 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdb7007f300 0x7fdb7007f700 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 rx=0x7fdb68006f00 tx=0x7fdb68009f10).stop 2021-02-10T11:32:20.852 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.849 7fdb77297700 1 -- 172.21.2.112:0/4212040206 shutdown_connections 2021-02-10T11:32:20.852 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.849 7fdb77297700 1 --2- 172.21.2.112:0/4212040206 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdb7007f300 0x7fdb7007f700 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:20.852 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.849 7fdb77297700 1 --2- 172.21.2.112:0/4212040206 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdb700e5470 0x7fdb7007e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:20.852 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.849 7fdb77297700 1 -- 172.21.2.112:0/4212040206 >> 172.21.2.112:0/4212040206 conn(0x7fdb700ce690 msgr2=0x7fdb700e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:20.853 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.850 7fdb77297700 1 -- 172.21.2.112:0/4212040206 shutdown_connections 2021-02-10T11:32:20.853 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.850 7fdb77297700 1 -- 172.21.2.112:0/4212040206 wait complete. 2021-02-10T11:32:20.854 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.864 7fdb77297700 1 Processor -- start 2021-02-10T11:32:20.854 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.864 7fdb77297700 1 -- start start 2021-02-10T11:32:20.854 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.864 7fdb77297700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdb70191bb0 0x7fdb700a2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:20.854 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.864 7fdb77297700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdb700a29a0 0x7fdb700a2da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:20.854 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.864 7fdb77297700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fdb700a33c0 0x7fdb700a03d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:20.854 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.864 7fdb77297700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb700adf90 con 0x7fdb700a29a0 2021-02-10T11:32:20.855 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.864 7fdb77297700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb700a1e00 con 0x7fdb700a33c0 2021-02-10T11:32:20.855 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.864 7fdb77297700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb700a09f0 con 0x7fdb70191bb0 2021-02-10T11:32:20.855 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.865 7fdb748b2700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdb700a29a0 0x7fdb700a2da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:20.855 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.865 7fdb748b2700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdb700a29a0 0x7fdb700a2da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:48774/0 (socket says 172.21.2.112:48774) 2021-02-10T11:32:20.855 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.865 7fdb748b2700 1 -- 172.21.2.112:0/3573041012 learned_addr learned my addr 172.21.2.112:0/3573041012 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:32:20.855 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.865 7fdb758b4700 1 --2- 172.21.2.112:0/3573041012 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fdb700a33c0 0x7fdb700a03d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:20.855 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.865 7fdb750b3700 1 --2- 172.21.2.112:0/3573041012 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdb70191bb0 0x7fdb700a2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:20.855 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.865 7fdb748b2700 1 -- 172.21.2.112:0/3573041012 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdb70191bb0 msgr2=0x7fdb700a2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:20.855 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.865 7fdb748b2700 1 --2- 172.21.2.112:0/3573041012 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdb70191bb0 0x7fdb700a2380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:20.855 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.865 7fdb748b2700 1 -- 172.21.2.112:0/3573041012 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fdb700a33c0 msgr2=0x7fdb700a03d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:20.855 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.865 7fdb748b2700 1 --2- 172.21.2.112:0/3573041012 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fdb700a33c0 0x7fdb700a03d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:20.855 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.865 7fdb748b2700 1 -- 172.21.2.112:0/3573041012 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb700a0bd0 con 0x7fdb700a29a0 2021-02-10T11:32:20.855 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.865 7fdb758b4700 1 --2- 172.21.2.112:0/3573041012 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fdb700a33c0 0x7fdb700a03d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:32:20.856 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.865 7fdb748b2700 1 --2- 172.21.2.112:0/3573041012 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdb700a29a0 0x7fdb700a2da0 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 rx=0x7fdb6c0067e0 tx=0x7fdb6c00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:20.856 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.866 7fdb61ffb700 1 -- 172.21.2.112:0/3573041012 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdb6c006900 con 0x7fdb700a29a0 2021-02-10T11:32:20.856 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.866 7fdb61ffb700 1 -- 172.21.2.112:0/3573041012 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb6c00be10 con 0x7fdb700a29a0 2021-02-10T11:32:20.856 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.866 7fdb61ffb700 1 -- 172.21.2.112:0/3573041012 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdb6c002030 con 0x7fdb700a29a0 2021-02-10T11:32:20.856 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.866 7fdb77297700 1 -- 172.21.2.112:0/3573041012 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb700a0bd0 con 0x7fdb700a29a0 2021-02-10T11:32:20.856 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.866 7fdb77297700 1 -- 172.21.2.112:0/3573041012 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb700a1160 con 0x7fdb700a29a0 2021-02-10T11:32:20.856 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.866 7fdb61ffb700 1 -- 172.21.2.112:0/3573041012 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fdb6c000c90 con 0x7fdb700a29a0 2021-02-10T11:32:20.856 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.866 7fdb61ffb700 1 --2- 172.21.2.112:0/3573041012 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fdb500186c0 0x7fdb5001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:20.856 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.866 7fdb61ffb700 1 -- 172.21.2.112:0/3573041012 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fdb6c032570 con 0x7fdb700a29a0 2021-02-10T11:32:20.856 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.867 7fdb750b3700 1 --2- 172.21.2.112:0/3573041012 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fdb500186c0 0x7fdb5001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:20.857 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.867 7fdb750b3700 1 --2- 172.21.2.112:0/3573041012 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fdb500186c0 0x7fdb5001abc0 crc :-1 s=READY pgs=551 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:20.885 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.897 7fdb77297700 1 -- 172.21.2.112:0/3573041012 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb700a1e00 con 0x7fdb700a29a0 2021-02-10T11:32:20.887 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:20.900 7fdb61ffb700 1 -- 172.21.2.112:0/3573041012 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fdb6c0258b0 con 0x7fdb700a29a0 2021-02-10T11:32:21.015 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:21.028 7fdb77297700 1 -- 172.21.2.112:0/3573041012 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdb700a1e00 con 0x7fdb700a29a0 2021-02-10T11:32:21.015 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:21.028 7fdb61ffb700 1 -- 172.21.2.112:0/3573041012 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+2099 (secure 0 0 0) 0x7fdb6c0258b0 con 0x7fdb700a29a0 2021-02-10T11:32:21.016 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 23 2021-02-10T11:32:21.016 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:32:21.016 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":23,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7155,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/1957863967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":1957863967},{"type":"v1","addr":"172.21.2.112:6833","nonce":1957863967}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":18}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:31:40.190262","modified":"2021-02-10 11:31:42.195824","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5534},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5534":{"gid":5534,"name":"b","rank":0,"incarnation":19,"state":"up:active","state_seq":3,"addr":"172.21.2.120:6835/1513827094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":1513827094},{"type":"v1","addr":"172.21.2.120:6835","nonce":1513827094}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2021-02-10T11:32:21.018 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:21.030 7fdb77297700 1 -- 172.21.2.112:0/3573041012 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fdb500186c0 msgr2=0x7fdb5001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:21.018 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:21.030 7fdb77297700 1 --2- 172.21.2.112:0/3573041012 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fdb500186c0 0x7fdb5001abc0 crc :-1 s=READY pgs=551 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:21.018 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:21.030 7fdb77297700 1 -- 172.21.2.112:0/3573041012 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdb700a29a0 msgr2=0x7fdb700a2da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:21.018 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:21.030 7fdb77297700 1 --2- 172.21.2.112:0/3573041012 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdb700a29a0 0x7fdb700a2da0 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 rx=0x7fdb6c0067e0 tx=0x7fdb6c00b990).stop 2021-02-10T11:32:21.018 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:21.030 7fdb77297700 1 -- 172.21.2.112:0/3573041012 shutdown_connections 2021-02-10T11:32:21.018 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:21.030 7fdb77297700 1 --2- 172.21.2.112:0/3573041012 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fdb500186c0 0x7fdb5001abc0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:21.018 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:21.030 7fdb77297700 1 --2- 172.21.2.112:0/3573041012 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fdb700a33c0 0x7fdb700a03d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:21.018 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:21.030 7fdb77297700 1 --2- 172.21.2.112:0/3573041012 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdb700a29a0 0x7fdb700a2da0 unknown :-1 s=CLOSED pgs=1080 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:21.018 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:21.030 7fdb77297700 1 --2- 172.21.2.112:0/3573041012 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdb70191bb0 0x7fdb700a2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:21.019 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:21.030 7fdb77297700 1 -- 172.21.2.112:0/3573041012 >> 172.21.2.112:0/3573041012 conn(0x7fdb701a31d0 msgr2=0x7fdb7008b830 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:21.019 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:21.030 7fdb77297700 1 -- 172.21.2.112:0/3573041012 shutdown_connections 2021-02-10T11:32:21.019 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:21.030 7fdb77297700 1 -- 172.21.2.112:0/3573041012 wait complete. 2021-02-10T11:32:21.026 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs fail cephfs 2021-02-10T11:32:21.457 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:21.447 7f66d8fcb700 1 Processor -- start 2021-02-10T11:32:21.457 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:21.447 7f66d8fcb700 1 -- start start 2021-02-10T11:32:21.458 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:21.447 7f66d8fcb700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f66d40e5470 0x7f66d407e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:21.458 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:21.447 7f66d8fcb700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f66d407f300 0x7f66d407f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:21.458 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:21.447 7f66d8fcb700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66d407ed80 con 0x7f66d40e5070 2021-02-10T11:32:21.458 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:21.447 7f66d8fcb700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66d41b4010 con 0x7f66d40e5470 2021-02-10T11:32:21.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:21.447 7f66d8fcb700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66d40d1790 con 0x7f66d407f300 2021-02-10T11:32:21.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:21.448 7f66d2d9d700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f66d40e5070 0x7f66d41b3820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:48240/0 (socket says 172.21.2.112:48240) 2021-02-10T11:32:21.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:21.448 7f66d2d9d700 1 -- 172.21.2.112:0/4072210321 learned_addr learned my addr 172.21.2.112:0/4072210321 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:32:21.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:21.448 7f66d359e700 1 --2- 172.21.2.112:0/4072210321 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f66d407f300 0x7f66d407f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:21.460 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:21.448 7f66d1d9b700 1 -- 172.21.2.112:0/4072210321 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 834634101 0 0) 0x7f66c4000ce0 con 0x7f66d40e5070 2021-02-10T11:32:21.460 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:21.448 7f66d259c700 1 --2- 172.21.2.112:0/4072210321 >> v2:172.21.2.120:3300/0 conn(0x7f66d40e5470 0x7f66d407e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:21.460 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:21.448 7f66d1d9b700 1 -- 172.21.2.112:0/4072210321 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66c0002fe0 con 0x7f66d40e5070 2021-02-10T11:32:21.460 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:21.449 7f66d259c700 1 -- 172.21.2.112:0/4072210321 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f66d407f300 msgr2=0x7f66d407f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:21.460 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:21.449 7f66d259c700 1 --2- 172.21.2.112:0/4072210321 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f66d407f300 0x7f66d407f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:21.460 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:21.449 7f66d259c700 1 -- 172.21.2.112:0/4072210321 >> v1:172.21.2.112:6789/0 conn(0x7f66d40e5070 legacy=0x7f66d41b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:21.461 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:21.449 7f66d259c700 1 -- 172.21.2.112:0/4072210321 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66d40d19b0 con 0x7f66d40e5470 2021-02-10T11:32:21.461 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:21.449 7f66d259c700 1 --2- 172.21.2.112:0/4072210321 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f66d40e5470 0x7f66d407e6a0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7f66bc006650 tx=0x7f66bc009a80).ready entity=mon.1 client_cookie=24de867e8726452f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:21.461 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:21.449 7f66d1d9b700 1 -- 172.21.2.112:0/4072210321 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f66bc0009b0 con 0x7f66d40e5470 2021-02-10T11:32:21.461 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:21.449 7f66d1d9b700 1 -- 172.21.2.112:0/4072210321 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66bc00c0c0 con 0x7f66d40e5470 2021-02-10T11:32:21.461 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:21.449 7f66d1d9b700 1 -- 172.21.2.112:0/4072210321 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f66bc00c5e0 con 0x7f66d40e5470 2021-02-10T11:32:21.461 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:21.449 7f66d8fcb700 1 -- 172.21.2.112:0/4072210321 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f66d40e5470 msgr2=0x7f66d407e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:21.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:21.449 7f66d8fcb700 1 --2- 172.21.2.112:0/4072210321 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f66d40e5470 0x7f66d407e6a0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7f66bc006650 tx=0x7f66bc009a80).stop 2021-02-10T11:32:21.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:21.450 7f66d8fcb700 1 -- 172.21.2.112:0/4072210321 shutdown_connections 2021-02-10T11:32:21.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:21.450 7f66d8fcb700 1 --2- 172.21.2.112:0/4072210321 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f66d407f300 0x7f66d407f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:21.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:21.450 7f66d8fcb700 1 --2- 172.21.2.112:0/4072210321 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f66d40e5470 0x7f66d407e6a0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:21.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:21.450 7f66d8fcb700 1 -- 172.21.2.112:0/4072210321 >> 172.21.2.112:0/4072210321 conn(0x7f66d40ce690 msgr2=0x7f66d40e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:21.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:21.450 7f66d8fcb700 1 -- 172.21.2.112:0/4072210321 shutdown_connections 2021-02-10T11:32:21.463 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:21.450 7f66d8fcb700 1 -- 172.21.2.112:0/4072210321 wait complete. 2021-02-10T11:32:21.463 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:21.465 7f66d8fcb700 1 Processor -- start 2021-02-10T11:32:21.463 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:21.465 7f66d8fcb700 1 -- start start 2021-02-10T11:32:21.463 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:21.466 7f66d8fcb700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f66d408f500 0x7f66d419f900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:21.463 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:21.466 7f66d8fcb700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f66d408fd90 0x7f66d4090190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:21.463 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:21.466 7f66d8fcb700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f66d40907b0 0x7f66d4094370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:21.464 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:21.466 7f66d8fcb700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66d408f900 con 0x7f66d408f500 2021-02-10T11:32:21.464 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:21.466 7f66d8fcb700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66d419ff20 con 0x7f66d408fd90 2021-02-10T11:32:21.464 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:21.466 7f66d8fcb700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66d41a0140 con 0x7f66d40907b0 2021-02-10T11:32:21.464 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:21.466 7f66d2d9d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f66d408f500 0x7f66d419f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:21.464 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:21.466 7f66d2d9d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f66d408f500 0x7f66d419f900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:48788/0 (socket says 172.21.2.112:48788) 2021-02-10T11:32:21.464 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:21.466 7f66d2d9d700 1 -- 172.21.2.112:0/3340323902 learned_addr learned my addr 172.21.2.112:0/3340323902 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:32:21.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:21.466 7f66d259c700 1 --2- 172.21.2.112:0/3340323902 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f66d408fd90 0x7f66d4090190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:21.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:21.466 7f66d359e700 1 --2- 172.21.2.112:0/3340323902 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f66d40907b0 0x7f66d4094370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:21.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:21.466 7f66d2d9d700 1 -- 172.21.2.112:0/3340323902 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f66d40907b0 msgr2=0x7f66d4094370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:21.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:21.466 7f66d2d9d700 1 --2- 172.21.2.112:0/3340323902 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f66d40907b0 0x7f66d4094370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:21.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:21.466 7f66d2d9d700 1 -- 172.21.2.112:0/3340323902 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f66d408fd90 msgr2=0x7f66d4090190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:21.466 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:21.466 7f66d2d9d700 1 --2- 172.21.2.112:0/3340323902 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f66d408fd90 0x7f66d4090190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:21.466 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:21.466 7f66d2d9d700 1 -- 172.21.2.112:0/3340323902 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66d4094990 con 0x7f66d408f500 2021-02-10T11:32:21.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:21.466 7f66d259c700 1 --2- 172.21.2.112:0/3340323902 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f66d408fd90 0x7f66d4090190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:32:21.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:21.466 7f66d359e700 1 --2- 172.21.2.112:0/3340323902 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f66d40907b0 0x7f66d4094370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:32:21.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:21.467 7f66d2d9d700 1 --2- 172.21.2.112:0/3340323902 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f66d408f500 0x7f66d419f900 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 rx=0x7f66c400a580 tx=0x7f66c400a920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:21.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:21.467 7f66cb7fe700 1 -- 172.21.2.112:0/3340323902 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f66c4001940 con 0x7f66d408f500 2021-02-10T11:32:21.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:21.467 7f66cb7fe700 1 -- 172.21.2.112:0/3340323902 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66c4007400 con 0x7f66d408f500 2021-02-10T11:32:21.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:21.467 7f66d8fcb700 1 -- 172.21.2.112:0/3340323902 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66d4094990 con 0x7f66d408f500 2021-02-10T11:32:21.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:21.467 7f66cb7fe700 1 -- 172.21.2.112:0/3340323902 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f66c4004620 con 0x7f66d408f500 2021-02-10T11:32:21.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:21.467 7f66d8fcb700 1 -- 172.21.2.112:0/3340323902 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66d4094990 con 0x7f66d408f500 2021-02-10T11:32:21.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:21.467 7f66cb7fe700 1 -- 172.21.2.112:0/3340323902 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f66c40043a0 con 0x7f66d408f500 2021-02-10T11:32:21.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:21.467 7f66cb7fe700 1 --2- 172.21.2.112:0/3340323902 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f66b8018730 0x7f66b801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:21.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:21.468 7f66cb7fe700 1 -- 172.21.2.112:0/3340323902 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f66c4032650 con 0x7f66d408f500 2021-02-10T11:32:21.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:21.468 7f66d259c700 1 --2- 172.21.2.112:0/3340323902 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f66b8018730 0x7f66b801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:21.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:21.468 7f66d259c700 1 --2- 172.21.2.112:0/3340323902 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f66b8018730 0x7f66b801abc0 crc :-1 s=READY pgs=552 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:21.484 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:21.497 7f66d8fcb700 1 -- 172.21.2.112:0/3340323902 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66d4093e40 con 0x7f66d408f500 2021-02-10T11:32:21.487 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:21.499 7f66cb7fe700 1 -- 172.21.2.112:0/3340323902 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f66c4001870 con 0x7f66d408f500 2021-02-10T11:32:21.614 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:21.627 7f66d8fcb700 1 -- 172.21.2.112:0/3340323902 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f66d4093e40 con 0x7f66d408f500 2021-02-10T11:32:21.723 INFO:tasks.ceph.mon.a.gibba012.stderr:2021-02-10 11:32:21.736 7fc3fe6e2700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-02-10T11:32:21.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:21.741 7f66cb7fe700 1 -- 172.21.2.112:0/3340323902 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v24) v1 ==== 161+0+0 (secure 0 0 0) 0x7f66c401f510 con 0x7f66d408f500 2021-02-10T11:32:21.779 INFO:teuthology.orchestra.run.gibba012.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2021-02-10T11:32:21.779 INFO:tasks.ceph.mds.b.gibba020.stdout:starting mds.b at 2021-02-10T11:32:21.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:21.793 7f66d8fcb700 1 -- 172.21.2.112:0/3340323902 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f66b8018730 msgr2=0x7f66b801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:21.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:21.793 7f66d8fcb700 1 --2- 172.21.2.112:0/3340323902 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f66b8018730 0x7f66b801abc0 crc :-1 s=READY pgs=552 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:21.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:21.793 7f66d8fcb700 1 -- 172.21.2.112:0/3340323902 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f66d408f500 msgr2=0x7f66d419f900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:21.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:21.793 7f66d8fcb700 1 --2- 172.21.2.112:0/3340323902 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f66d408f500 0x7f66d419f900 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 rx=0x7f66c400a580 tx=0x7f66c400a920).stop 2021-02-10T11:32:21.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:21.793 7f66d8fcb700 1 -- 172.21.2.112:0/3340323902 shutdown_connections 2021-02-10T11:32:21.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:21.793 7f66d8fcb700 1 --2- 172.21.2.112:0/3340323902 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f66b8018730 0x7f66b801abc0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:21.782 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:21.793 7f66d8fcb700 1 --2- 172.21.2.112:0/3340323902 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f66d408fd90 0x7f66d4090190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:21.782 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:21.793 7f66d8fcb700 1 --2- 172.21.2.112:0/3340323902 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f66d40907b0 0x7f66d4094370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:21.782 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:21.793 7f66d8fcb700 1 --2- 172.21.2.112:0/3340323902 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f66d408f500 0x7f66d419f900 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:21.782 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:21.793 7f66d8fcb700 1 -- 172.21.2.112:0/3340323902 >> 172.21.2.112:0/3340323902 conn(0x7f66d40b5560 msgr2=0x7f66d40a6200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:21.782 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:21.793 7f66d8fcb700 1 -- 172.21.2.112:0/3340323902 shutdown_connections 2021-02-10T11:32:21.782 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:21.793 7f66d8fcb700 1 -- 172.21.2.112:0/3340323902 wait complete. 2021-02-10T11:32:21.789 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:32:22.214 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.211 7f69fe5c1700 1 Processor -- start 2021-02-10T11:32:22.214 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.211 7f69fe5c1700 1 -- start start 2021-02-10T11:32:22.214 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.211 7f69fe5c1700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f69f80e5070 0x7f69f81b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:22.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.211 7f69fe5c1700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f69f80e5470 0x7f69f807d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:22.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.211 7f69fe5c1700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69f807ed80 con 0x7f69f81b3e40 2021-02-10T11:32:22.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.211 7f69fe5c1700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69f8080310 con 0x7f69f80e5470 2021-02-10T11:32:22.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.211 7f69fe5c1700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69f8080500 con 0x7f69f80e5070 2021-02-10T11:32:22.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.211 7f69f77fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f69f80e5470 0x7f69f807d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:22.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.211 7f69f77fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f69f80e5470 0x7f69f807d690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:48792/0 (socket says 172.21.2.112:48792) 2021-02-10T11:32:22.216 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.211 7f69f77fe700 1 -- 172.21.2.112:0/367362741 learned_addr learned my addr 172.21.2.112:0/367362741 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:32:22.216 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.211 7f69f77fe700 1 -- 172.21.2.112:0/367362741 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f69f80e5070 msgr2=0x7f69f81b3820 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:32:22.216 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.211 7f69f77fe700 1 --2- 172.21.2.112:0/367362741 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f69f80e5070 0x7f69f81b3820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:22.216 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.211 7f69f77fe700 1 -- 172.21.2.112:0/367362741 >> v1:172.21.2.120:6789/0 conn(0x7f69f81b3e40 legacy=0x7f69f807dcb0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:32:22.216 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.211 7f69f77fe700 1 -- 172.21.2.112:0/367362741 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69f80806f0 con 0x7f69f80e5470 2021-02-10T11:32:22.216 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.212 7f69f77fe700 1 --2- 172.21.2.112:0/367362741 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f69f80e5470 0x7f69f807d690 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 rx=0x7f69ec006700 tx=0x7f69ec009b00).ready entity=mon.0 client_cookie=3a5f9ef0cdfa03d7 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:22.216 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.212 7f69f6ffd700 1 -- 172.21.2.112:0/367362741 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f69ec00c0c0 con 0x7f69f80e5470 2021-02-10T11:32:22.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.212 7f69f6ffd700 1 -- 172.21.2.112:0/367362741 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69ec00c400 con 0x7f69f80e5470 2021-02-10T11:32:22.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.212 7f69f6ffd700 1 -- 172.21.2.112:0/367362741 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f69ec00c8e0 con 0x7f69f80e5470 2021-02-10T11:32:22.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.212 7f69fe5c1700 1 -- 172.21.2.112:0/367362741 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f69f80e5470 msgr2=0x7f69f807d690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:22.218 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.212 7f69fe5c1700 1 --2- 172.21.2.112:0/367362741 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f69f80e5470 0x7f69f807d690 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 rx=0x7f69ec006700 tx=0x7f69ec009b00).stop 2021-02-10T11:32:22.218 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.212 7f69fe5c1700 1 -- 172.21.2.112:0/367362741 shutdown_connections 2021-02-10T11:32:22.218 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.212 7f69fe5c1700 1 --2- 172.21.2.112:0/367362741 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f69f80e5470 0x7f69f807d690 unknown :-1 s=CLOSED pgs=1085 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:22.218 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.212 7f69fe5c1700 1 --2- 172.21.2.112:0/367362741 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f69f80e5070 0x7f69f81b3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:22.218 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.212 7f69fe5c1700 1 -- 172.21.2.112:0/367362741 >> 172.21.2.112:0/367362741 conn(0x7f69f80ce690 msgr2=0x7f69f80e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:22.218 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.212 7f69fe5c1700 1 -- 172.21.2.112:0/367362741 shutdown_connections 2021-02-10T11:32:22.218 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.212 7f69fe5c1700 1 -- 172.21.2.112:0/367362741 wait complete. 2021-02-10T11:32:22.218 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.227 7f69fe5c1700 1 Processor -- start 2021-02-10T11:32:22.219 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.227 7f69fe5c1700 1 -- start start 2021-02-10T11:32:22.220 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.227 7f69fe5c1700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f69f80ad1e0 0x7f69f81ae990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:22.220 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.227 7f69fe5c1700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f69f81aefb0 0x7f69f81adb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:22.220 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.227 7f69fe5c1700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f69f81ae190 0x7f69f8090ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:22.220 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.227 7f69fe5c1700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69f80909a0 con 0x7f69f81aefb0 2021-02-10T11:32:22.220 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.227 7f69fe5c1700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69f81ae5d0 con 0x7f69f80ad1e0 2021-02-10T11:32:22.220 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.227 7f69fe5c1700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69f80905e0 con 0x7f69f81ae190 2021-02-10T11:32:22.220 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.227 7f69f77fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f69f81aefb0 0x7f69f81adb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:22.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.228 7f69f77fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f69f81aefb0 0x7f69f81adb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:48798/0 (socket says 172.21.2.112:48798) 2021-02-10T11:32:22.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.228 7f69f77fe700 1 -- 172.21.2.112:0/3122717841 learned_addr learned my addr 172.21.2.112:0/3122717841 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:32:22.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.228 7f69fcbde700 1 --2- 172.21.2.112:0/3122717841 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f69f81ae190 0x7f69f8090ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:22.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.228 7f69f77fe700 1 -- 172.21.2.112:0/3122717841 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f69f81ae190 msgr2=0x7f69f8090ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:22.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.228 7f69f77fe700 1 --2- 172.21.2.112:0/3122717841 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f69f81ae190 0x7f69f8090ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:22.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.228 7f69f77fe700 1 -- 172.21.2.112:0/3122717841 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f69f80ad1e0 msgr2=0x7f69f81ae990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:22.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.228 7f69f77fe700 1 --2- 172.21.2.112:0/3122717841 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f69f80ad1e0 0x7f69f81ae990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:22.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.228 7f69f77fe700 1 -- 172.21.2.112:0/3122717841 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69f80914c0 con 0x7f69f81aefb0 2021-02-10T11:32:22.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.228 7f69f77fe700 1 --2- 172.21.2.112:0/3122717841 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f69f81aefb0 0x7f69f81adb70 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 rx=0x7f69ec00b660 tx=0x7f69ec00ba00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:22.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.228 7f69f4ff9700 1 -- 172.21.2.112:0/3122717841 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f69ec009440 con 0x7f69f81aefb0 2021-02-10T11:32:22.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.228 7f69f4ff9700 1 -- 172.21.2.112:0/3122717841 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69ec006390 con 0x7f69f81aefb0 2021-02-10T11:32:22.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.228 7f69f4ff9700 1 -- 172.21.2.112:0/3122717841 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f69ec0052e0 con 0x7f69f81aefb0 2021-02-10T11:32:22.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.228 7f69fe5c1700 1 -- 172.21.2.112:0/3122717841 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69f80914c0 con 0x7f69f81aefb0 2021-02-10T11:32:22.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.228 7f69fe5c1700 1 -- 172.21.2.112:0/3122717841 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69f80914c0 con 0x7f69f81aefb0 2021-02-10T11:32:22.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.229 7f69f4ff9700 1 -- 172.21.2.112:0/3122717841 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f69ec0093e0 con 0x7f69f81aefb0 2021-02-10T11:32:22.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.229 7f69f4ff9700 1 --2- 172.21.2.112:0/3122717841 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f69d80186c0 0x7f69d801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:22.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.229 7f69f4ff9700 1 -- 172.21.2.112:0/3122717841 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5659+0+0 (secure 0 0 0) 0x7f69ec032650 con 0x7f69f81aefb0 2021-02-10T11:32:22.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.229 7f69f7fff700 1 --2- 172.21.2.112:0/3122717841 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f69d80186c0 0x7f69d801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:22.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.230 7f69f7fff700 1 --2- 172.21.2.112:0/3122717841 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f69d80186c0 0x7f69d801abc0 crc :-1 s=READY pgs=553 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:22.248 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.261 7f69fe5c1700 1 -- 172.21.2.112:0/3122717841 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69f8083dd0 con 0x7f69f81aefb0 2021-02-10T11:32:22.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.264 7f69f4ff9700 1 -- 172.21.2.112:0/3122717841 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f69ec0090c0 con 0x7f69f81aefb0 2021-02-10T11:32:22.379 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.392 7f69fe5c1700 1 -- 172.21.2.112:0/3122717841 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f69f807ed80 con 0x7f69f81aefb0 2021-02-10T11:32:22.379 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.392 7f69f4ff9700 1 -- 172.21.2.112:0/3122717841 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+1754 (secure 0 0 0) 0x7f69ec00bdf0 con 0x7f69f81aefb0 2021-02-10T11:32:22.380 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 24 2021-02-10T11:32:22.380 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:32:22.380 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":24,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7155,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/1957863967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":1957863967},{"type":"v1","addr":"172.21.2.112:6833","nonce":1957863967}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":18}],"filesystems":[{"mdsmap":{"epoch":24,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:31:40.190262","modified":"2021-02-10 11:32:21.737260","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2021-02-10T11:32:22.382 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.394 7f69fe5c1700 1 -- 172.21.2.112:0/3122717841 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f69d80186c0 msgr2=0x7f69d801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:22.382 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.394 7f69fe5c1700 1 --2- 172.21.2.112:0/3122717841 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f69d80186c0 0x7f69d801abc0 crc :-1 s=READY pgs=553 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:22.382 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.394 7f69fe5c1700 1 -- 172.21.2.112:0/3122717841 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f69f81aefb0 msgr2=0x7f69f81adb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:22.382 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.394 7f69fe5c1700 1 --2- 172.21.2.112:0/3122717841 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f69f81aefb0 0x7f69f81adb70 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 rx=0x7f69ec00b660 tx=0x7f69ec00ba00).stop 2021-02-10T11:32:22.382 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.394 7f69fe5c1700 1 -- 172.21.2.112:0/3122717841 shutdown_connections 2021-02-10T11:32:22.382 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.394 7f69fe5c1700 1 --2- 172.21.2.112:0/3122717841 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f69d80186c0 0x7f69d801abc0 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:22.382 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.394 7f69fe5c1700 1 --2- 172.21.2.112:0/3122717841 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f69f81ae190 0x7f69f8090ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:22.382 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.394 7f69fe5c1700 1 --2- 172.21.2.112:0/3122717841 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f69f81aefb0 0x7f69f81adb70 unknown :-1 s=CLOSED pgs=1086 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:22.382 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.394 7f69fe5c1700 1 --2- 172.21.2.112:0/3122717841 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f69f80ad1e0 0x7f69f81ae990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:22.382 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.394 7f69fe5c1700 1 -- 172.21.2.112:0/3122717841 >> 172.21.2.112:0/3122717841 conn(0x7f69f81a3820 msgr2=0x7f69f81af970 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:22.382 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.395 7f69fe5c1700 1 -- 172.21.2.112:0/3122717841 shutdown_connections 2021-02-10T11:32:22.382 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.395 7f69fe5c1700 1 -- 172.21.2.112:0/3122717841 wait complete. 2021-02-10T11:32:22.391 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2021-02-10T11:32:22.823 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.820 7f3145d51700 1 Processor -- start 2021-02-10T11:32:22.823 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.820 7f3145d51700 1 -- start start 2021-02-10T11:32:22.823 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.820 7f3145d51700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f31400cff00 0x7f31401b39d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:22.824 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.820 7f3145d51700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31400f0b20 con 0x7f31400d17f0 2021-02-10T11:32:22.824 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.820 7f3145d51700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31401b3ff0 con 0x7f31400d1bf0 2021-02-10T11:32:22.824 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.820 7f3145d51700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31400e5eb0 con 0x7f31400cff00 2021-02-10T11:32:22.824 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.821 7f313f7fe700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f31400d17f0 0x7f31401b31e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:48266/0 (socket says 172.21.2.112:48266) 2021-02-10T11:32:22.824 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.821 7f313f7fe700 1 -- 172.21.2.112:0/740381213 learned_addr learned my addr 172.21.2.112:0/740381213 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:32:22.824 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.821 7f313effd700 1 --2- 172.21.2.112:0/740381213 >> v2:172.21.2.120:3300/0 conn(0x7f31400cff00 0x7f31401b39d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:22.824 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.821 7f313e7fc700 1 -- 172.21.2.112:0/740381213 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 44542604 0 0) 0x7f31300009b0 con 0x7f31400d17f0 2021-02-10T11:32:22.824 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.821 7f313e7fc700 1 -- 172.21.2.112:0/740381213 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f312c002fe0 con 0x7f31400d17f0 2021-02-10T11:32:22.824 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.821 7f313e7fc700 1 -- 172.21.2.112:0/740381213 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1271222175 0 0) 0x7f3138000f90 con 0x7f31400d1bf0 2021-02-10T11:32:22.824 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.821 7f313e7fc700 1 -- 172.21.2.112:0/740381213 --> v1:172.21.2.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f312c004370 con 0x7f31400d1bf0 2021-02-10T11:32:22.824 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.821 7f313effd700 1 -- 172.21.2.112:0/740381213 >> v1:172.21.2.120:6789/0 conn(0x7f31400d1bf0 legacy=0x7f31400ef620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:22.824 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.821 7f313effd700 1 -- 172.21.2.112:0/740381213 >> v1:172.21.2.112:6789/0 conn(0x7f31400d17f0 legacy=0x7f31401b31e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:22.824 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.821 7f313effd700 1 -- 172.21.2.112:0/740381213 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31400f20b0 con 0x7f31400cff00 2021-02-10T11:32:22.825 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.821 7f313effd700 1 --2- 172.21.2.112:0/740381213 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f31400cff00 0x7f31401b39d0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 rx=0x7f3128006650 tx=0x7f3128009a80).ready entity=mon.1 client_cookie=c3a59fbfcea28507 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:22.825 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.821 7f313e7fc700 1 -- 172.21.2.112:0/740381213 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f312800c0c0 con 0x7f31400cff00 2021-02-10T11:32:22.825 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.821 7f313e7fc700 1 -- 172.21.2.112:0/740381213 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f312800c0c0 con 0x7f31400cff00 2021-02-10T11:32:22.825 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.821 7f313e7fc700 1 -- 172.21.2.112:0/740381213 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f312800c3c0 con 0x7f31400cff00 2021-02-10T11:32:22.825 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.821 7f3145d51700 1 -- 172.21.2.112:0/740381213 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f31400cff00 msgr2=0x7f31401b39d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:22.825 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.821 7f3145d51700 1 --2- 172.21.2.112:0/740381213 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f31400cff00 0x7f31401b39d0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 rx=0x7f3128006650 tx=0x7f3128009a80).stop 2021-02-10T11:32:22.825 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.822 7f3145d51700 1 -- 172.21.2.112:0/740381213 shutdown_connections 2021-02-10T11:32:22.825 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.822 7f3145d51700 1 --2- 172.21.2.112:0/740381213 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f31400cff00 0x7f31401b39d0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:22.825 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.822 7f3145d51700 1 -- 172.21.2.112:0/740381213 >> 172.21.2.112:0/740381213 conn(0x7f31400fde40 msgr2=0x7f31400ce390 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:22.825 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.822 7f3145d51700 1 -- 172.21.2.112:0/740381213 shutdown_connections 2021-02-10T11:32:22.825 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.822 7f3145d51700 1 -- 172.21.2.112:0/740381213 wait complete. 2021-02-10T11:32:22.827 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.836 7f3145d51700 1 Processor -- start 2021-02-10T11:32:22.827 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.837 7f3145d51700 1 -- start start 2021-02-10T11:32:22.827 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.837 7f3145d51700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f31400ae340 0x7f31400ae740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:22.827 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.837 7f3145d51700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3140106a40 0x7f3140106e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:22.827 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.837 7f3145d51700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3140187e70 0x7f3140188270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:22.827 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.837 7f3145d51700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31401076e0 con 0x7f3140106a40 2021-02-10T11:32:22.827 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.837 7f3145d51700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3140095020 con 0x7f3140187e70 2021-02-10T11:32:22.827 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.837 7f3145d51700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3140095200 con 0x7f31400ae340 2021-02-10T11:32:22.827 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.837 7f313effd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3140106a40 0x7f3140106e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:22.827 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.837 7f313effd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3140106a40 0x7f3140106e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:48814/0 (socket says 172.21.2.112:48814) 2021-02-10T11:32:22.827 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.837 7f313effd700 1 -- 172.21.2.112:0/516274839 learned_addr learned my addr 172.21.2.112:0/516274839 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:32:22.827 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.837 7f313effd700 1 -- 172.21.2.112:0/516274839 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f31400ae340 msgr2=0x7f31400ae740 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:32:22.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.837 7f313effd700 1 --2- 172.21.2.112:0/516274839 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f31400ae340 0x7f31400ae740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:22.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.837 7f313effd700 1 -- 172.21.2.112:0/516274839 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3140187e70 msgr2=0x7f3140188270 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:32:22.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.837 7f313effd700 1 --2- 172.21.2.112:0/516274839 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3140187e70 0x7f3140188270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:22.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.837 7f313effd700 1 -- 172.21.2.112:0/516274839 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31401a8a40 con 0x7f3140106a40 2021-02-10T11:32:22.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.838 7f313effd700 1 --2- 172.21.2.112:0/516274839 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3140106a40 0x7f3140106e40 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 rx=0x7f3128009570 tx=0x7f3128002670).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:22.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.838 7f3137fff700 1 -- 172.21.2.112:0/516274839 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3128006030 con 0x7f3140106a40 2021-02-10T11:32:22.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.838 7f3137fff700 1 -- 172.21.2.112:0/516274839 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31280052e0 con 0x7f3140106a40 2021-02-10T11:32:22.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.838 7f3137fff700 1 -- 172.21.2.112:0/516274839 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f31280057c0 con 0x7f3140106a40 2021-02-10T11:32:22.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.838 7f3145d51700 1 -- 172.21.2.112:0/516274839 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31401a8a40 con 0x7f3140106a40 2021-02-10T11:32:22.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.838 7f3145d51700 1 -- 172.21.2.112:0/516274839 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31401a8a40 con 0x7f3140106a40 2021-02-10T11:32:22.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.838 7f3137fff700 1 -- 172.21.2.112:0/516274839 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f31280054c0 con 0x7f3140106a40 2021-02-10T11:32:22.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.838 7f3137fff700 1 --2- 172.21.2.112:0/516274839 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3124018730 0x7f312401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:22.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.838 7f3137fff700 1 -- 172.21.2.112:0/516274839 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5659+0+0 (secure 0 0 0) 0x7f3128032670 con 0x7f3140106a40 2021-02-10T11:32:22.829 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.839 7f313f7fe700 1 --2- 172.21.2.112:0/516274839 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3124018730 0x7f312401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:22.829 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.839 7f313f7fe700 1 --2- 172.21.2.112:0/516274839 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3124018730 0x7f312401abc0 crc :-1 s=READY pgs=554 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:22.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.870 7f3145d51700 1 -- 172.21.2.112:0/516274839 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3140094fd0 con 0x7f3140106a40 2021-02-10T11:32:22.860 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.873 7f3137fff700 1 -- 172.21.2.112:0/516274839 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3128002a60 con 0x7f3140106a40 2021-02-10T11:32:22.986 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:22.999 7f3145d51700 1 -- 172.21.2.112:0/516274839 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f3140094fd0 con 0x7f3140106a40 2021-02-10T11:32:23.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:23.750 7f3137fff700 1 -- 172.21.2.112:0/516274839 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v25) v1 ==== 104+0+0 (secure 0 0 0) 0x7f31280052e0 con 0x7f3140106a40 2021-02-10T11:32:23.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:23.770 7f3145d51700 1 -- 172.21.2.112:0/516274839 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3124018730 msgr2=0x7f312401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:23.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:23.770 7f3145d51700 1 --2- 172.21.2.112:0/516274839 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3124018730 0x7f312401abc0 crc :-1 s=READY pgs=554 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:23.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:23.770 7f3145d51700 1 -- 172.21.2.112:0/516274839 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3140106a40 msgr2=0x7f3140106e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:23.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:23.770 7f3145d51700 1 --2- 172.21.2.112:0/516274839 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3140106a40 0x7f3140106e40 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 rx=0x7f3128009570 tx=0x7f3128002670).stop 2021-02-10T11:32:23.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:23.771 7f3145d51700 1 -- 172.21.2.112:0/516274839 shutdown_connections 2021-02-10T11:32:23.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:23.771 7f3145d51700 1 --2- 172.21.2.112:0/516274839 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3124018730 0x7f312401abc0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:23.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:23.771 7f3145d51700 1 --2- 172.21.2.112:0/516274839 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3140187e70 0x7f3140188270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:23.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:23.771 7f3145d51700 1 --2- 172.21.2.112:0/516274839 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3140106a40 0x7f3140106e40 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:23.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:23.771 7f3145d51700 1 --2- 172.21.2.112:0/516274839 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f31400ae340 0x7f31400ae740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:23.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:23.771 7f3145d51700 1 -- 172.21.2.112:0/516274839 >> 172.21.2.112:0/516274839 conn(0x7f31401ba0f0 msgr2=0x7f314018c240 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:23.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:23.771 7f3145d51700 1 -- 172.21.2.112:0/516274839 shutdown_connections 2021-02-10T11:32:23.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:23.771 7f3145d51700 1 -- 172.21.2.112:0/516274839 wait complete. 2021-02-10T11:32:23.766 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2021-02-10T11:32:24.196 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:24.193 7fd8d4d83700 1 Processor -- start 2021-02-10T11:32:24.197 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:24.193 7fd8d4d83700 1 -- start start 2021-02-10T11:32:24.197 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:24.193 7fd8d4d83700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fd8d00d1870 0x7fd8d01b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:24.197 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:24.193 7fd8d4d83700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fd8d00cff70 0x7fd8d0109130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:24.197 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:24.193 7fd8d4d83700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8d00fe1c0 con 0x7fd8d01b0ae0 2021-02-10T11:32:24.197 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:24.193 7fd8d4d83700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8d0109ad0 con 0x7fd8d00cff70 2021-02-10T11:32:24.197 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:24.193 7fd8d4d83700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8d0109750 con 0x7fd8d00d1870 2021-02-10T11:32:24.197 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:24.193 7fd8cf314700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fd8d00cff70 0x7fd8d0109130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:24.197 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:24.193 7fd8cf314700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fd8d00cff70 0x7fd8d0109130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:48822/0 (socket says 172.21.2.112:48822) 2021-02-10T11:32:24.197 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:24.193 7fd8cf314700 1 -- 172.21.2.112:0/426918392 learned_addr learned my addr 172.21.2.112:0/426918392 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:32:24.197 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:24.194 7fd8cf314700 1 -- 172.21.2.112:0/426918392 >> v2:172.21.2.120:3300/0 conn(0x7fd8d00d1870 msgr2=0x7fd8d01b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:24.198 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:24.194 7fd8cf314700 1 --2- 172.21.2.112:0/426918392 >> v2:172.21.2.120:3300/0 conn(0x7fd8d00d1870 0x7fd8d01b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:24.198 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:24.194 7fd8cf314700 1 -- 172.21.2.112:0/426918392 >> v1:172.21.2.112:6789/0 conn(0x7fd8d01b0ae0 legacy=0x7fd8d0108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:24.198 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:24.194 7fd8cf314700 1 -- 172.21.2.112:0/426918392 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8d0088020 con 0x7fd8d00cff70 2021-02-10T11:32:24.198 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:24.194 7fd8cf314700 1 --2- 172.21.2.112:0/426918392 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd8d00cff70 0x7fd8d0109130 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 rx=0x7fd8c8006f00 tx=0x7fd8c8009f10).ready entity=mon.0 client_cookie=c3b3fdc82ce80d0b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:24.198 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:24.194 7fd8cdb11700 1 -- 172.21.2.112:0/426918392 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd8c800b3b0 con 0x7fd8d00cff70 2021-02-10T11:32:24.198 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:24.194 7fd8cdb11700 1 -- 172.21.2.112:0/426918392 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8c800b730 con 0x7fd8d00cff70 2021-02-10T11:32:24.198 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:24.194 7fd8cdb11700 1 -- 172.21.2.112:0/426918392 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd8c800bc50 con 0x7fd8d00cff70 2021-02-10T11:32:24.198 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:24.194 7fd8d4d83700 1 -- 172.21.2.112:0/426918392 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd8d00cff70 msgr2=0x7fd8d0109130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:24.198 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:24.194 7fd8d4d83700 1 --2- 172.21.2.112:0/426918392 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd8d00cff70 0x7fd8d0109130 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 rx=0x7fd8c8006f00 tx=0x7fd8c8009f10).stop 2021-02-10T11:32:24.198 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:24.195 7fd8d4d83700 1 -- 172.21.2.112:0/426918392 shutdown_connections 2021-02-10T11:32:24.198 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:24.195 7fd8d4d83700 1 --2- 172.21.2.112:0/426918392 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd8d00cff70 0x7fd8d0109130 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:24.198 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:24.195 7fd8d4d83700 1 --2- 172.21.2.112:0/426918392 >> v2:172.21.2.120:3300/0 conn(0x7fd8d00d1870 0x7fd8d01b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:24.199 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:24.195 7fd8d4d83700 1 -- 172.21.2.112:0/426918392 >> 172.21.2.112:0/426918392 conn(0x7fd8d00ce420 msgr2=0x7fd8d00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:24.199 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:24.195 7fd8d4d83700 1 -- 172.21.2.112:0/426918392 shutdown_connections 2021-02-10T11:32:24.199 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:24.195 7fd8d4d83700 1 -- 172.21.2.112:0/426918392 wait complete. 2021-02-10T11:32:24.200 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:24.210 7fd8d4d83700 1 Processor -- start 2021-02-10T11:32:24.200 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:24.210 7fd8d4d83700 1 -- start start 2021-02-10T11:32:24.200 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:24.210 7fd8d4d83700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd8d01a9930 0x7fd8d01a9080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:24.201 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:24.210 7fd8d4d83700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd8d01abd60 0x7fd8d0192a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:24.201 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:24.210 7fd8d4d83700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd8d00b03e0 0x7fd8d00b07e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:24.201 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:24.210 7fd8d4d83700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8d01932c0 con 0x7fd8d00b03e0 2021-02-10T11:32:24.201 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:24.210 7fd8d4d83700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8d0193840 con 0x7fd8d01abd60 2021-02-10T11:32:24.201 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:24.210 7fd8d4d83700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8d0193a20 con 0x7fd8d01a9930 2021-02-10T11:32:24.201 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:24.210 7fd8cf314700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd8d00b03e0 0x7fd8d00b07e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:24.201 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:24.210 7fd8cf314700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd8d00b03e0 0x7fd8d00b07e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:48826/0 (socket says 172.21.2.112:48826) 2021-02-10T11:32:24.201 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:24.210 7fd8cf314700 1 -- 172.21.2.112:0/725200079 learned_addr learned my addr 172.21.2.112:0/725200079 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:32:24.201 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:24.210 7fd8cf314700 1 -- 172.21.2.112:0/725200079 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd8d01a9930 msgr2=0x7fd8d01a9080 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:32:24.201 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:24.210 7fd8cf314700 1 --2- 172.21.2.112:0/725200079 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd8d01a9930 0x7fd8d01a9080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:24.201 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:24.210 7fd8cf314700 1 -- 172.21.2.112:0/725200079 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd8d01abd60 msgr2=0x7fd8d0192a60 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:32:24.201 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:24.210 7fd8cf314700 1 --2- 172.21.2.112:0/725200079 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd8d01abd60 0x7fd8d0192a60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:24.201 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:24.210 7fd8cf314700 1 -- 172.21.2.112:0/725200079 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8d0193c40 con 0x7fd8d00b03e0 2021-02-10T11:32:24.202 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:24.211 7fd8cf314700 1 --2- 172.21.2.112:0/725200079 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd8d00b03e0 0x7fd8d00b07e0 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 rx=0x7fd8c8009390 tx=0x7fd8c800bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:24.202 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:24.211 7fd8c77fe700 1 -- 172.21.2.112:0/725200079 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd8c8003030 con 0x7fd8d00b03e0 2021-02-10T11:32:24.202 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:24.211 7fd8c77fe700 1 -- 172.21.2.112:0/725200079 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8c80068e0 con 0x7fd8d00b03e0 2021-02-10T11:32:24.202 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:24.211 7fd8c77fe700 1 -- 172.21.2.112:0/725200079 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd8c8001670 con 0x7fd8d00b03e0 2021-02-10T11:32:24.202 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:24.211 7fd8d4d83700 1 -- 172.21.2.112:0/725200079 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8d0193c40 con 0x7fd8d00b03e0 2021-02-10T11:32:24.202 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:24.211 7fd8d4d83700 1 -- 172.21.2.112:0/725200079 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8d0193c40 con 0x7fd8d00b03e0 2021-02-10T11:32:24.202 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:24.211 7fd8c77fe700 1 -- 172.21.2.112:0/725200079 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd8c8001270 con 0x7fd8d00b03e0 2021-02-10T11:32:24.203 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:24.211 7fd8c77fe700 1 --2- 172.21.2.112:0/725200079 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd8b40186c0 0x7fd8b401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:24.203 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:24.211 7fd8c77fe700 1 -- 172.21.2.112:0/725200079 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5659+0+0 (secure 0 0 0) 0x7fd8c8032bb0 con 0x7fd8d00b03e0 2021-02-10T11:32:24.203 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:24.212 7fd8ceb13700 1 --2- 172.21.2.112:0/725200079 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd8b40186c0 0x7fd8b401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:24.203 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:24.212 7fd8ceb13700 1 --2- 172.21.2.112:0/725200079 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd8b40186c0 0x7fd8b401abc0 crc :-1 s=READY pgs=555 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:24.230 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:24.243 7fd8d4d83700 1 -- 172.21.2.112:0/725200079 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8d00828e0 con 0x7fd8d00b03e0 2021-02-10T11:32:24.233 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:24.246 7fd8c77fe700 1 -- 172.21.2.112:0/725200079 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd8c8000cc0 con 0x7fd8d00b03e0 2021-02-10T11:32:24.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:24.373 7fd8d4d83700 1 -- 172.21.2.112:0/725200079 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fd8d00828e0 con 0x7fd8d00b03e0 2021-02-10T11:32:24.743 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:24.755 7fd8c77fe700 1 -- 172.21.2.112:0/725200079 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v40) v1 ==== 185+0+0 (secure 0 0 0) 0x7fd8c80068e0 con 0x7fd8d00b03e0 2021-02-10T11:32:24.775 INFO:teuthology.orchestra.run.gibba012.stderr:pool 'cephfs_metadata' removed 2021-02-10T11:32:24.778 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:24.789 7fd8d4d83700 1 -- 172.21.2.112:0/725200079 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd8b40186c0 msgr2=0x7fd8b401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:24.778 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:24.789 7fd8d4d83700 1 --2- 172.21.2.112:0/725200079 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd8b40186c0 0x7fd8b401abc0 crc :-1 s=READY pgs=555 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:24.778 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:24.789 7fd8d4d83700 1 -- 172.21.2.112:0/725200079 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd8d00b03e0 msgr2=0x7fd8d00b07e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:24.778 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:24.789 7fd8d4d83700 1 --2- 172.21.2.112:0/725200079 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd8d00b03e0 0x7fd8d00b07e0 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 rx=0x7fd8c8009390 tx=0x7fd8c800bd40).stop 2021-02-10T11:32:24.778 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:24.789 7fd8d4d83700 1 -- 172.21.2.112:0/725200079 shutdown_connections 2021-02-10T11:32:24.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:24.789 7fd8d4d83700 1 --2- 172.21.2.112:0/725200079 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd8b40186c0 0x7fd8b401abc0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:24.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:24.790 7fd8d4d83700 1 --2- 172.21.2.112:0/725200079 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd8d01abd60 0x7fd8d0192a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:24.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:24.790 7fd8d4d83700 1 --2- 172.21.2.112:0/725200079 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd8d00b03e0 0x7fd8d00b07e0 unknown :-1 s=CLOSED pgs=1091 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:24.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:24.790 7fd8d4d83700 1 --2- 172.21.2.112:0/725200079 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd8d01a9930 0x7fd8d01a9080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:24.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:24.790 7fd8d4d83700 1 -- 172.21.2.112:0/725200079 >> 172.21.2.112:0/725200079 conn(0x7fd8d019f820 msgr2=0x7fd8d0082b10 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:24.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:24.790 7fd8d4d83700 1 -- 172.21.2.112:0/725200079 shutdown_connections 2021-02-10T11:32:24.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:24.790 7fd8d4d83700 1 -- 172.21.2.112:0/725200079 wait complete. 2021-02-10T11:32:24.787 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2021-02-10T11:32:25.228 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:25.223 7f6e9d218700 1 Processor -- start 2021-02-10T11:32:25.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:25.223 7f6e9d218700 1 -- start start 2021-02-10T11:32:25.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:25.224 7f6e9d218700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6e980d1860 0x7f6e981b04b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:25.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:25.224 7f6e9d218700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e98109a50 con 0x7f6e981b0ad0 2021-02-10T11:32:25.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:25.224 7f6e9d218700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e98088020 con 0x7f6e980cff60 2021-02-10T11:32:25.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:25.224 7f6e9d218700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e980e5ee0 con 0x7f6e980d1860 2021-02-10T11:32:25.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:25.224 7f6e9659c700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f6e981b0ad0 0x7f6e98108930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:48294/0 (socket says 172.21.2.112:48294) 2021-02-10T11:32:25.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:25.224 7f6e9659c700 1 -- 172.21.2.112:0/1710703660 learned_addr learned my addr 172.21.2.112:0/1710703660 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:32:25.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:25.224 7f6e95d9b700 1 -- 172.21.2.112:0/1710703660 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 597007381 0 0) 0x7f6e90001500 con 0x7f6e981b0ad0 2021-02-10T11:32:25.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:25.224 7f6e95d9b700 1 -- 172.21.2.112:0/1710703660 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e80002fe0 con 0x7f6e981b0ad0 2021-02-10T11:32:25.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:25.224 7f6e96d9d700 1 --2- 172.21.2.112:0/1710703660 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6e980d1860 0x7f6e981b04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:25.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:25.225 7f6e95d9b700 1 -- 172.21.2.112:0/1710703660 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 823750722 0 0) 0x7f6e90001460 con 0x7f6e981b0ad0 2021-02-10T11:32:25.230 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:25.225 7f6e95d9b700 1 -- 172.21.2.112:0/1710703660 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6e980d1860 msgr2=0x7f6e981b04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:25.230 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:25.225 7f6e95d9b700 1 --2- 172.21.2.112:0/1710703660 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6e980d1860 0x7f6e981b04b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:25.230 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:25.225 7f6e95d9b700 1 -- 172.21.2.112:0/1710703660 >> v1:172.21.2.120:6789/0 conn(0x7f6e980cff60 legacy=0x7f6e98109120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:25.230 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:25.225 7f6e95d9b700 1 -- 172.21.2.112:0/1710703660 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e980d1c60 con 0x7f6e981b0ad0 2021-02-10T11:32:25.230 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:25.225 7f6e95d9b700 1 -- 172.21.2.112:0/1710703660 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f6e90001980 con 0x7f6e981b0ad0 2021-02-10T11:32:25.230 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:25.225 7f6e95d9b700 1 -- 172.21.2.112:0/1710703660 >> v1:172.21.2.112:6789/0 conn(0x7f6e981b0ad0 legacy=0x7f6e98108930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:25.230 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:25.225 7f6e95d9b700 1 --2- 172.21.2.112:0/1710703660 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6e80003aa0 0x7f6e80005ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:25.230 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:25.225 7f6e95d9b700 1 --2- 172.21.2.112:0/1710703660 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6e80006770 0x7f6e80008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:25.230 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:25.225 7f6e95d9b700 1 --2- 172.21.2.112:0/1710703660 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6e80009590 0x7f6e8000ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:25.230 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:25.225 7f6e95d9b700 1 -- 172.21.2.112:0/1710703660 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e8000c2d0 con 0x7f6e80006770 2021-02-10T11:32:25.230 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:25.225 7f6e95d9b700 1 -- 172.21.2.112:0/1710703660 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e8000c4f0 con 0x7f6e80003aa0 2021-02-10T11:32:25.230 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:25.225 7f6e95d9b700 1 -- 172.21.2.112:0/1710703660 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e8000c710 con 0x7f6e80009590 2021-02-10T11:32:25.230 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:25.225 7f6e9659c700 1 --2- 172.21.2.112:0/1710703660 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6e80006770 0x7f6e80008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:25.231 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:25.225 7f6e96d9d700 1 --2- 172.21.2.112:0/1710703660 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6e80003aa0 0x7f6e80005ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:25.231 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:25.225 7f6e9759e700 1 --2- 172.21.2.112:0/1710703660 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6e80009590 0x7f6e8000ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:25.231 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:25.225 7f6e9659c700 1 -- 172.21.2.112:0/1710703660 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6e80009590 msgr2=0x7f6e8000ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:25.231 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:25.225 7f6e9659c700 1 --2- 172.21.2.112:0/1710703660 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6e80009590 0x7f6e8000ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:25.231 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:25.225 7f6e9659c700 1 -- 172.21.2.112:0/1710703660 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6e80003aa0 msgr2=0x7f6e80005ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:25.231 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:25.225 7f6e9659c700 1 --2- 172.21.2.112:0/1710703660 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6e80003aa0 0x7f6e80005ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:25.231 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:25.225 7f6e9659c700 1 -- 172.21.2.112:0/1710703660 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e80050980 con 0x7f6e80006770 2021-02-10T11:32:25.231 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:25.225 7f6e96d9d700 1 --2- 172.21.2.112:0/1710703660 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6e80003aa0 0x7f6e80005ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:32:25.231 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:25.226 7f6e9659c700 1 --2- 172.21.2.112:0/1710703660 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6e80006770 0x7f6e80008c90 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 rx=0x7f6e90007cf0 tx=0x7f6e9000a310).ready entity=mon.0 client_cookie=48802bee8b88cdd server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:25.231 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:25.226 7f6e95d9b700 1 -- 172.21.2.112:0/1710703660 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6e900070c0 con 0x7f6e80006770 2021-02-10T11:32:25.231 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:25.226 7f6e95d9b700 1 -- 172.21.2.112:0/1710703660 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e90007340 con 0x7f6e80006770 2021-02-10T11:32:25.231 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:25.226 7f6e95d9b700 1 -- 172.21.2.112:0/1710703660 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6e9000ab90 con 0x7f6e80006770 2021-02-10T11:32:25.231 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:25.226 7f6e9d218700 1 -- 172.21.2.112:0/1710703660 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6e80006770 msgr2=0x7f6e80008c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:25.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:25.226 7f6e9d218700 1 --2- 172.21.2.112:0/1710703660 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6e80006770 0x7f6e80008c90 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 rx=0x7f6e90007cf0 tx=0x7f6e9000a310).stop 2021-02-10T11:32:25.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:25.226 7f6e9759e700 1 -- 172.21.2.112:0/1710703660 reap_dead start 2021-02-10T11:32:25.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:25.227 7f6e9d218700 1 -- 172.21.2.112:0/1710703660 shutdown_connections 2021-02-10T11:32:25.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:25.227 7f6e9d218700 1 -- 172.21.2.112:0/1710703660 >> 172.21.2.112:0/1710703660 conn(0x7f6e980ce410 msgr2=0x7f6e980d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:25.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:25.227 7f6e9d218700 1 -- 172.21.2.112:0/1710703660 shutdown_connections 2021-02-10T11:32:25.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:25.227 7f6e9d218700 1 -- 172.21.2.112:0/1710703660 wait complete. 2021-02-10T11:32:25.233 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:25.242 7f6e9d218700 1 Processor -- start 2021-02-10T11:32:25.233 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:25.242 7f6e9d218700 1 -- start start 2021-02-10T11:32:25.233 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:25.242 7f6e9d218700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6e981a05d0 0x7f6e980bb0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:25.234 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:25.242 7f6e9d218700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6e9818f480 0x7f6e9818f880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:25.234 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:25.242 7f6e9d218700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6e98190200 0x7f6e98190600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:25.234 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:25.242 7f6e9d218700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e980bb760 con 0x7f6e9818f480 2021-02-10T11:32:25.234 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:25.242 7f6e9d218700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e980babb0 con 0x7f6e98190200 2021-02-10T11:32:25.234 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:25.242 7f6e9d218700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e980b7680 con 0x7f6e981a05d0 2021-02-10T11:32:25.234 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:25.242 7f6e9659c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6e9818f480 0x7f6e9818f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:25.234 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:25.242 7f6e9659c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6e9818f480 0x7f6e9818f880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:48846/0 (socket says 172.21.2.112:48846) 2021-02-10T11:32:25.234 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:25.242 7f6e9659c700 1 -- 172.21.2.112:0/3566735501 learned_addr learned my addr 172.21.2.112:0/3566735501 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:32:25.234 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:25.242 7f6e9659c700 1 -- 172.21.2.112:0/3566735501 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6e981a05d0 msgr2=0x7f6e980bb0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:25.234 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:25.242 7f6e9659c700 1 --2- 172.21.2.112:0/3566735501 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6e981a05d0 0x7f6e980bb0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:25.234 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:25.243 7f6e9659c700 1 -- 172.21.2.112:0/3566735501 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6e98190200 msgr2=0x7f6e98190600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:25.234 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:25.243 7f6e9659c700 1 --2- 172.21.2.112:0/3566735501 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6e98190200 0x7f6e98190600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:25.234 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:25.243 7f6e9659c700 1 -- 172.21.2.112:0/3566735501 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e980b78a0 con 0x7f6e9818f480 2021-02-10T11:32:25.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:25.243 7f6e9659c700 1 --2- 172.21.2.112:0/3566735501 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6e9818f480 0x7f6e9818f880 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 rx=0x7f6e900021c0 tx=0x7f6e9000ad20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:25.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:25.243 7f6e877fe700 1 -- 172.21.2.112:0/3566735501 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6e900070d0 con 0x7f6e9818f480 2021-02-10T11:32:25.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:25.243 7f6e877fe700 1 -- 172.21.2.112:0/3566735501 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e90007350 con 0x7f6e9818f480 2021-02-10T11:32:25.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:25.243 7f6e877fe700 1 -- 172.21.2.112:0/3566735501 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6e90004b20 con 0x7f6e9818f480 2021-02-10T11:32:25.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:25.243 7f6e9d218700 1 -- 172.21.2.112:0/3566735501 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e980b78a0 con 0x7f6e9818f480 2021-02-10T11:32:25.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:25.243 7f6e9d218700 1 -- 172.21.2.112:0/3566735501 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e980b7e30 con 0x7f6e9818f480 2021-02-10T11:32:25.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:25.244 7f6e877fe700 1 -- 172.21.2.112:0/3566735501 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6e90001380 con 0x7f6e9818f480 2021-02-10T11:32:25.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:25.244 7f6e877fe700 1 --2- 172.21.2.112:0/3566735501 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6e7c0186c0 0x7f6e7c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:25.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:25.244 7f6e877fe700 1 -- 172.21.2.112:0/3566735501 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 5221+0+0 (secure 0 0 0) 0x7f6e900327d0 con 0x7f6e9818f480 2021-02-10T11:32:25.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:25.244 7f6e96d9d700 1 --2- 172.21.2.112:0/3566735501 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6e7c0186c0 0x7f6e7c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:25.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:25.245 7f6e96d9d700 1 --2- 172.21.2.112:0/3566735501 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6e7c0186c0 0x7f6e7c01abc0 crc :-1 s=READY pgs=556 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:25.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:25.274 7f6e9d218700 1 -- 172.21.2.112:0/3566735501 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e980b7650 con 0x7f6e9818f480 2021-02-10T11:32:25.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:25.277 7f6e877fe700 1 -- 172.21.2.112:0/3566735501 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6e90004de0 con 0x7f6e9818f480 2021-02-10T11:32:25.390 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:25.403 7f6e9d218700 1 -- 172.21.2.112:0/3566735501 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f6e980b7650 con 0x7f6e9818f480 2021-02-10T11:32:25.747 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:25.760 7f6e877fe700 1 -- 172.21.2.112:0/3566735501 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v41) v1 ==== 173+0+0 (secure 0 0 0) 0x7f6e90001380 con 0x7f6e9818f480 2021-02-10T11:32:25.757 INFO:teuthology.orchestra.run.gibba012.stderr:pool 'cephfs_data' removed 2021-02-10T11:32:25.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:25.771 7f6e9d218700 1 -- 172.21.2.112:0/3566735501 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6e7c0186c0 msgr2=0x7f6e7c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:25.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:25.771 7f6e9d218700 1 --2- 172.21.2.112:0/3566735501 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6e7c0186c0 0x7f6e7c01abc0 crc :-1 s=READY pgs=556 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:25.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:25.771 7f6e9d218700 1 -- 172.21.2.112:0/3566735501 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6e9818f480 msgr2=0x7f6e9818f880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:25.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:25.771 7f6e9d218700 1 --2- 172.21.2.112:0/3566735501 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6e9818f480 0x7f6e9818f880 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 rx=0x7f6e900021c0 tx=0x7f6e9000ad20).stop 2021-02-10T11:32:25.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:25.771 7f6e9d218700 1 -- 172.21.2.112:0/3566735501 shutdown_connections 2021-02-10T11:32:25.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:25.771 7f6e9d218700 1 --2- 172.21.2.112:0/3566735501 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6e7c0186c0 0x7f6e7c01abc0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:25.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:25.771 7f6e9d218700 1 --2- 172.21.2.112:0/3566735501 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6e98190200 0x7f6e98190600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:25.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:25.771 7f6e9d218700 1 --2- 172.21.2.112:0/3566735501 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6e9818f480 0x7f6e9818f880 unknown :-1 s=CLOSED pgs=1094 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:25.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:25.771 7f6e9d218700 1 --2- 172.21.2.112:0/3566735501 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6e981a05d0 0x7f6e980bb0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:25.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:25.771 7f6e9d218700 1 -- 172.21.2.112:0/3566735501 >> 172.21.2.112:0/3566735501 conn(0x7f6e980873e0 msgr2=0x7f6e980b5010 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:25.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:25.771 7f6e9d218700 1 -- 172.21.2.112:0/3566735501 shutdown_connections 2021-02-10T11:32:25.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:25.771 7f6e9d218700 1 -- 172.21.2.112:0/3566735501 wait complete. 2021-02-10T11:32:25.767 INFO:tasks.ceph.mds.a:Restarting daemon 2021-02-10T11:32:25.767 INFO:tasks.ceph.mds.a:Stopping old one... 2021-02-10T11:32:25.768 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2021-02-10T11:32:25.768 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:32:25.768 INFO:tasks.ceph.mds.b:Restarting daemon 2021-02-10T11:32:25.768 INFO:tasks.ceph.mds.b:Stopping old one... 2021-02-10T11:32:25.768 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2021-02-10T11:32:25.769 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:32:25.777 INFO:tasks.ceph.mds.b:Stopped 2021-02-10T11:32:25.777 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2021-02-10T11:32:25.778 INFO:tasks.ceph.mds.a:Stopped 2021-02-10T11:32:25.779 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2021-02-10T11:32:25.824 INFO:tasks.ceph.mds.b:Started 2021-02-10T11:32:25.825 DEBUG:teuthology.parallel:result is None 2021-02-10T11:32:25.827 INFO:tasks.ceph.mds.a:Started 2021-02-10T11:32:25.828 DEBUG:teuthology.parallel:result is None 2021-02-10T11:32:25.828 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd blacklist clear 2021-02-10T11:32:26.227 INFO:tasks.ceph.mds.b.gibba020.stdout:starting mds.b at 2021-02-10T11:32:26.233 INFO:tasks.ceph.mds.a.gibba012.stdout:starting mds.a at 2021-02-10T11:32:26.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:26.252 7f839d753700 1 Processor -- start 2021-02-10T11:32:26.257 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:26.252 7f839d753700 1 -- start start 2021-02-10T11:32:26.257 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:26.253 7f839d753700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f839807e6a0 0x7f83980d1790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:26.257 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:26.253 7f839d753700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f839807ed80 con 0x7f83980e5070 2021-02-10T11:32:26.257 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:26.253 7f839d753700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f839807eaa0 con 0x7f83980e5470 2021-02-10T11:32:26.257 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:26.253 7f839d753700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83981b4080 con 0x7f839807e6a0 2021-02-10T11:32:26.257 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:26.253 7f8397549700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f83980e5070 0x7f83981b3820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:48326/0 (socket says 172.21.2.112:48326) 2021-02-10T11:32:26.257 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:26.253 7f8397549700 1 -- 172.21.2.112:0/556405503 learned_addr learned my addr 172.21.2.112:0/556405503 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:32:26.257 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:26.253 7f8397d4a700 1 --2- 172.21.2.112:0/556405503 >> v2:172.21.2.120:3300/0 conn(0x7f839807e6a0 0x7f83980d1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:26.257 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:26.253 7f8396547700 1 -- 172.21.2.112:0/556405503 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 74232876 0 0) 0x7f838c001180 con 0x7f83980e5070 2021-02-10T11:32:26.257 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:26.253 7f8396547700 1 -- 172.21.2.112:0/556405503 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8380002fe0 con 0x7f83980e5070 2021-02-10T11:32:26.257 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:26.254 7f8396547700 1 -- 172.21.2.112:0/556405503 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 559409817 0 0) 0x7f8388000c60 con 0x7f83980e5470 2021-02-10T11:32:26.258 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:26.254 7f8396547700 1 -- 172.21.2.112:0/556405503 --> v1:172.21.2.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8380003210 con 0x7f83980e5470 2021-02-10T11:32:26.258 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:26.254 7f8397d4a700 1 -- 172.21.2.112:0/556405503 >> v1:172.21.2.120:6789/0 conn(0x7f83980e5470 legacy=0x7f839807f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:26.258 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:26.254 7f8397d4a700 1 -- 172.21.2.112:0/556405503 >> v1:172.21.2.112:6789/0 conn(0x7f83980e5070 legacy=0x7f83981b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:26.258 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:26.254 7f8397d4a700 1 -- 172.21.2.112:0/556405503 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83980d1db0 con 0x7f839807e6a0 2021-02-10T11:32:26.258 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:26.254 7f8397d4a700 1 --2- 172.21.2.112:0/556405503 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f839807e6a0 0x7f83980d1790 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 rx=0x7f8390006e50 tx=0x7f8390009e70).ready entity=mon.1 client_cookie=93404c7ff7bdff96 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:26.258 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:26.254 7f8396547700 1 -- 172.21.2.112:0/556405503 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8390006a20 con 0x7f839807e6a0 2021-02-10T11:32:26.258 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:26.254 7f8396547700 1 -- 172.21.2.112:0/556405503 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8390006790 con 0x7f839807e6a0 2021-02-10T11:32:26.258 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:26.254 7f8396547700 1 -- 172.21.2.112:0/556405503 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f839000b700 con 0x7f839807e6a0 2021-02-10T11:32:26.258 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:26.254 7f839d753700 1 -- 172.21.2.112:0/556405503 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f839807e6a0 msgr2=0x7f83980d1790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:26.258 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:26.254 7f839d753700 1 --2- 172.21.2.112:0/556405503 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f839807e6a0 0x7f83980d1790 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 rx=0x7f8390006e50 tx=0x7f8390009e70).stop 2021-02-10T11:32:26.258 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:26.255 7f839d753700 1 -- 172.21.2.112:0/556405503 shutdown_connections 2021-02-10T11:32:26.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:26.255 7f839d753700 1 --2- 172.21.2.112:0/556405503 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f839807e6a0 0x7f83980d1790 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:26.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:26.255 7f839d753700 1 -- 172.21.2.112:0/556405503 >> 172.21.2.112:0/556405503 conn(0x7f83980ce690 msgr2=0x7f83980e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:26.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:26.255 7f839d753700 1 -- 172.21.2.112:0/556405503 shutdown_connections 2021-02-10T11:32:26.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:26.255 7f839d753700 1 -- 172.21.2.112:0/556405503 wait complete. 2021-02-10T11:32:26.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:26.269 7f839d753700 1 Processor -- start 2021-02-10T11:32:26.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:26.270 7f839d753700 1 -- start start 2021-02-10T11:32:26.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:26.270 7f839d753700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f83980bf710 0x7f83980c3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:26.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:26.270 7f839d753700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f83980891a0 0x7f83980895a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:26.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:26.270 7f839d753700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f83980b3d50 0x7f83980b4150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:26.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:26.270 7f839d753700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83980bfdf0 con 0x7f83980b3d50 2021-02-10T11:32:26.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:26.270 7f839d753700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83980c3650 con 0x7f83980bf710 2021-02-10T11:32:26.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:26.270 7f839d753700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8398089bc0 con 0x7f83980891a0 2021-02-10T11:32:26.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:26.270 7f8397d4a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f83980b3d50 0x7f83980b4150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:26.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:26.270 7f8397d4a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f83980b3d50 0x7f83980b4150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:48868/0 (socket says 172.21.2.112:48868) 2021-02-10T11:32:26.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:26.270 7f8397d4a700 1 -- 172.21.2.112:0/1309533834 learned_addr learned my addr 172.21.2.112:0/1309533834 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:32:26.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:26.270 7f8396d48700 1 --2- 172.21.2.112:0/1309533834 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f83980891a0 0x7f83980895a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:26.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:26.270 7f8397549700 1 --2- 172.21.2.112:0/1309533834 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f83980bf710 0x7f83980c3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:26.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:26.271 7f8397d4a700 1 -- 172.21.2.112:0/1309533834 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f83980891a0 msgr2=0x7f83980895a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:26.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:26.271 7f8397d4a700 1 --2- 172.21.2.112:0/1309533834 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f83980891a0 0x7f83980895a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:26.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:26.271 7f8397d4a700 1 -- 172.21.2.112:0/1309533834 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f83980bf710 msgr2=0x7f83980c3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:26.262 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:26.271 7f8397d4a700 1 --2- 172.21.2.112:0/1309533834 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f83980bf710 0x7f83980c3bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:26.262 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:26.271 7f8397d4a700 1 -- 172.21.2.112:0/1309533834 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83980bfb10 con 0x7f83980b3d50 2021-02-10T11:32:26.262 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:26.271 7f8396d48700 1 --2- 172.21.2.112:0/1309533834 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f83980891a0 0x7f83980895a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:32:26.262 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:26.271 7f8397d4a700 1 --2- 172.21.2.112:0/1309533834 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f83980b3d50 0x7f83980b4150 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 rx=0x7f83900099f0 tx=0x7f839000bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:26.262 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:26.271 7f837ffff700 1 -- 172.21.2.112:0/1309533834 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f83900093d0 con 0x7f83980b3d50 2021-02-10T11:32:26.262 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:26.271 7f837ffff700 1 -- 172.21.2.112:0/1309533834 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83900090c0 con 0x7f83980b3d50 2021-02-10T11:32:26.262 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:26.271 7f837ffff700 1 -- 172.21.2.112:0/1309533834 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8390001390 con 0x7f83980b3d50 2021-02-10T11:32:26.262 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:26.271 7f839d753700 1 -- 172.21.2.112:0/1309533834 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83980bfb10 con 0x7f83980b3d50 2021-02-10T11:32:26.262 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:26.271 7f839d753700 1 -- 172.21.2.112:0/1309533834 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f839819bcf0 con 0x7f83980b3d50 2021-02-10T11:32:26.262 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:26.272 7f837ffff700 1 -- 172.21.2.112:0/1309533834 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8390000df0 con 0x7f83980b3d50 2021-02-10T11:32:26.262 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:26.272 7f837ffff700 1 --2- 172.21.2.112:0/1309533834 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f83780186c0 0x7f837801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:26.262 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:26.272 7f837ffff700 1 -- 172.21.2.112:0/1309533834 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4839+0+0 (secure 0 0 0) 0x7f8390032460 con 0x7f83980b3d50 2021-02-10T11:32:26.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:26.272 7f8397549700 1 --2- 172.21.2.112:0/1309533834 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f83780186c0 0x7f837801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:26.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:26.273 7f8397549700 1 --2- 172.21.2.112:0/1309533834 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f83780186c0 0x7f837801abc0 crc :-1 s=READY pgs=557 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:26.289 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:26.301 7f839d753700 1 -- 172.21.2.112:0/1309533834 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83981071c0 con 0x7f83980b3d50 2021-02-10T11:32:26.291 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:26.304 7f837ffff700 1 -- 172.21.2.112:0/1309533834 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f83900093d0 con 0x7f83980b3d50 2021-02-10T11:32:26.429 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:26.442 7f839d753700 1 -- 172.21.2.112:0/1309533834 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7f839807ed80 con 0x7f83980b3d50 2021-02-10T11:32:26.750 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:26.763 7f837ffff700 1 -- 172.21.2.112:0/1309533834 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v42) v1 ==== 97+0+0 (secure 0 0 0) 0x7f8390000df0 con 0x7f83980b3d50 2021-02-10T11:32:26.795 INFO:teuthology.orchestra.run.gibba012.stderr: removed all blacklist entries 2021-02-10T11:32:26.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:26.810 7f839d753700 1 -- 172.21.2.112:0/1309533834 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f83780186c0 msgr2=0x7f837801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:26.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:26.810 7f839d753700 1 --2- 172.21.2.112:0/1309533834 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f83780186c0 0x7f837801abc0 crc :-1 s=READY pgs=557 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:26.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:26.810 7f839d753700 1 -- 172.21.2.112:0/1309533834 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f83980b3d50 msgr2=0x7f83980b4150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:26.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:26.810 7f839d753700 1 --2- 172.21.2.112:0/1309533834 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f83980b3d50 0x7f83980b4150 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 rx=0x7f83900099f0 tx=0x7f839000bd10).stop 2021-02-10T11:32:26.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:26.810 7f839d753700 1 -- 172.21.2.112:0/1309533834 shutdown_connections 2021-02-10T11:32:26.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:26.810 7f839d753700 1 --2- 172.21.2.112:0/1309533834 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f83780186c0 0x7f837801abc0 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:26.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:26.810 7f839d753700 1 --2- 172.21.2.112:0/1309533834 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f83980b3d50 0x7f83980b4150 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:26.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:26.810 7f839d753700 1 --2- 172.21.2.112:0/1309533834 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f83980891a0 0x7f83980895a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:26.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:26.810 7f839d753700 1 --2- 172.21.2.112:0/1309533834 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f83980bf710 0x7f83980c3bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:26.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:26.810 7f839d753700 1 -- 172.21.2.112:0/1309533834 >> 172.21.2.112:0/1309533834 conn(0x7f8398082130 msgr2=0x7f8398087150 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:26.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:26.810 7f839d753700 1 -- 172.21.2.112:0/1309533834 shutdown_connections 2021-02-10T11:32:26.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:26.810 7f839d753700 1 -- 172.21.2.112:0/1309533834 wait complete. 2021-02-10T11:32:26.806 INFO:tasks.cephfs.cephfs_test_case:['0', '1', '2', '3'] 2021-02-10T11:32:26.806 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth ls --format=json-pretty 2021-02-10T11:32:27.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.228 7f106a33d700 1 Processor -- start 2021-02-10T11:32:27.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.228 7f106a33d700 1 -- start start 2021-02-10T11:32:27.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.229 7f106a33d700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f10640e5070 0x7f10641b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:27.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.229 7f106a33d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f10641b3e40 0x7f106407d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:27.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.229 7f106a33d700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f106407ed80 con 0x7f10640e5470 2021-02-10T11:32:27.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.229 7f106a33d700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1064080310 con 0x7f10641b3e40 2021-02-10T11:32:27.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.229 7f106a33d700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1064080530 con 0x7f10640e5070 2021-02-10T11:32:27.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.229 7f10637fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f10641b3e40 0x7f106407d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:27.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.229 7f10637fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f10641b3e40 0x7f106407d690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:48880/0 (socket says 172.21.2.112:48880) 2021-02-10T11:32:27.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.229 7f10637fe700 1 -- 172.21.2.112:0/1696328863 learned_addr learned my addr 172.21.2.112:0/1696328863 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:32:27.233 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.229 7f10637fe700 1 -- 172.21.2.112:0/1696328863 >> v2:172.21.2.120:3300/0 conn(0x7f10640e5070 msgr2=0x7f10641b3820 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:32:27.233 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.229 7f10637fe700 1 --2- 172.21.2.112:0/1696328863 >> v2:172.21.2.120:3300/0 conn(0x7f10640e5070 0x7f10641b3820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:27.233 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.229 7f10637fe700 1 -- 172.21.2.112:0/1696328863 >> v1:172.21.2.120:6789/0 conn(0x7f10640e5470 legacy=0x7f106407dcb0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:32:27.233 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.229 7f10637fe700 1 -- 172.21.2.112:0/1696328863 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1064080750 con 0x7f10641b3e40 2021-02-10T11:32:27.233 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.230 7f10637fe700 1 --2- 172.21.2.112:0/1696328863 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f10641b3e40 0x7f106407d690 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 rx=0x7f1058006700 tx=0x7f1058009b80).ready entity=mon.0 client_cookie=4c8cca8a2c587f02 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:27.233 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.230 7f1062ffd700 1 -- 172.21.2.112:0/1696328863 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f105800c0c0 con 0x7f10641b3e40 2021-02-10T11:32:27.233 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.230 7f1062ffd700 1 -- 172.21.2.112:0/1696328863 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f105800c440 con 0x7f10641b3e40 2021-02-10T11:32:27.233 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.230 7f1062ffd700 1 -- 172.21.2.112:0/1696328863 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f105800c8a0 con 0x7f10641b3e40 2021-02-10T11:32:27.233 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.230 7f106a33d700 1 -- 172.21.2.112:0/1696328863 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f10641b3e40 msgr2=0x7f106407d690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:27.233 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.230 7f106a33d700 1 --2- 172.21.2.112:0/1696328863 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f10641b3e40 0x7f106407d690 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 rx=0x7f1058006700 tx=0x7f1058009b80).stop 2021-02-10T11:32:27.233 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.230 7f106a33d700 1 -- 172.21.2.112:0/1696328863 shutdown_connections 2021-02-10T11:32:27.233 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.231 7f106a33d700 1 --2- 172.21.2.112:0/1696328863 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f10641b3e40 0x7f106407d690 unknown :-1 s=CLOSED pgs=1101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:27.234 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.231 7f106a33d700 1 --2- 172.21.2.112:0/1696328863 >> v2:172.21.2.120:3300/0 conn(0x7f10640e5070 0x7f10641b3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:27.234 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.231 7f106a33d700 1 -- 172.21.2.112:0/1696328863 >> 172.21.2.112:0/1696328863 conn(0x7f10640ce690 msgr2=0x7f10640e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:27.234 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.231 7f106a33d700 1 -- 172.21.2.112:0/1696328863 shutdown_connections 2021-02-10T11:32:27.234 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.231 7f106a33d700 1 -- 172.21.2.112:0/1696328863 wait complete. 2021-02-10T11:32:27.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.245 7f106a33d700 1 Processor -- start 2021-02-10T11:32:27.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.245 7f106a33d700 1 -- start start 2021-02-10T11:32:27.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.245 7f106a33d700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f10641abb40 0x7f10640905e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:27.236 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.245 7f106a33d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f10640b49a0 0x7f106408a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:27.236 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.245 7f106a33d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f106409fe40 0x7f1064091230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:27.236 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.245 7f106a33d700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10641ac0c0 con 0x7f106409fe40 2021-02-10T11:32:27.236 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.245 7f106a33d700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1064090cb0 con 0x7f10641abb40 2021-02-10T11:32:27.236 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.245 7f106a33d700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10640a0240 con 0x7f10640b49a0 2021-02-10T11:32:27.236 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.245 7f106895a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f106409fe40 0x7f1064091230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:27.236 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.246 7f106895a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f106409fe40 0x7f1064091230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:48882/0 (socket says 172.21.2.112:48882) 2021-02-10T11:32:27.236 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.246 7f106895a700 1 -- 172.21.2.112:0/1068104405 learned_addr learned my addr 172.21.2.112:0/1068104405 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:32:27.236 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.246 7f106895a700 1 -- 172.21.2.112:0/1068104405 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f10640b49a0 msgr2=0x7f106408a460 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:32:27.236 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.246 7f106895a700 1 --2- 172.21.2.112:0/1068104405 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f10640b49a0 0x7f106408a460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:27.236 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.246 7f106895a700 1 -- 172.21.2.112:0/1068104405 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f10641abb40 msgr2=0x7f10640905e0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:32:27.236 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.246 7f106895a700 1 --2- 172.21.2.112:0/1068104405 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f10641abb40 0x7f10640905e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:27.236 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.246 7f106895a700 1 -- 172.21.2.112:0/1068104405 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10640a0460 con 0x7f106409fe40 2021-02-10T11:32:27.237 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.246 7f106895a700 1 --2- 172.21.2.112:0/1068104405 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f106409fe40 0x7f1064091230 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 rx=0x7f105c009940 tx=0x7f105c00bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:27.237 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.246 7f1060ff9700 1 -- 172.21.2.112:0/1068104405 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f105c003030 con 0x7f106409fe40 2021-02-10T11:32:27.237 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.246 7f1060ff9700 1 -- 172.21.2.112:0/1068104405 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f105c001270 con 0x7f106409fe40 2021-02-10T11:32:27.237 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.246 7f1060ff9700 1 -- 172.21.2.112:0/1068104405 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f105c001790 con 0x7f106409fe40 2021-02-10T11:32:27.237 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.246 7f106a33d700 1 -- 172.21.2.112:0/1068104405 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10640a0460 con 0x7f106409fe40 2021-02-10T11:32:27.237 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.246 7f106a33d700 1 -- 172.21.2.112:0/1068104405 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1064091b40 con 0x7f106409fe40 2021-02-10T11:32:27.237 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.247 7f1060ff9700 1 -- 172.21.2.112:0/1068104405 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f105c00a000 con 0x7f106409fe40 2021-02-10T11:32:27.237 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.247 7f1060ff9700 1 --2- 172.21.2.112:0/1068104405 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f10440186c0 0x7f104401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:27.237 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.247 7f1060ff9700 1 -- 172.21.2.112:0/1068104405 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f105c032590 con 0x7f106409fe40 2021-02-10T11:32:27.237 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.247 7f1063fff700 1 --2- 172.21.2.112:0/1068104405 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f10440186c0 0x7f104401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:27.237 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.248 7f1063fff700 1 --2- 172.21.2.112:0/1068104405 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f10440186c0 0x7f104401abc0 crc :-1 s=READY pgs=558 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:27.266 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.279 7f106a33d700 1 -- 172.21.2.112:0/1068104405 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1064083dd0 con 0x7f106409fe40 2021-02-10T11:32:27.268 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.281 7f1060ff9700 1 -- 172.21.2.112:0/1068104405 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f105c000f90 con 0x7f106409fe40 2021-02-10T11:32:27.399 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.412 7f106a33d700 1 -- 172.21.2.112:0/1068104405 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f106407ed80 con 0x7f106409fe40 2021-02-10T11:32:27.400 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.412 7f1060ff9700 1 -- 172.21.2.112:0/1068104405 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v24) v1 ==== 80+0+4513 (secure 0 0 0) 0x7f105c01f860 con 0x7f106409fe40 2021-02-10T11:32:27.400 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:32:27.400 INFO:teuthology.orchestra.run.gibba012.stdout:{ 2021-02-10T11:32:27.400 INFO:teuthology.orchestra.run.gibba012.stdout: "auth_dump": [ 2021-02-10T11:32:27.400 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:32:27.400 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "mds.a", 2021-02-10T11:32:27.400 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCKvyNgPkV4AxAAEmVNaghdccP6lWd8hvQnxw==", 2021-02-10T11:32:27.401 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:32:27.401 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:32:27.401 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:32:27.401 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:32:27.401 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:32:27.401 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:32:27.401 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:32:27.401 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:32:27.401 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "mds.b", 2021-02-10T11:32:27.401 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCKvyNguxONLhAAHIpry+R89MAwdhFJyEhVCw==", 2021-02-10T11:32:27.401 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:32:27.401 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:32:27.401 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:32:27.402 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:32:27.402 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:32:27.402 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:32:27.402 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:32:27.402 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:32:27.402 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.0", 2021-02-10T11:32:27.402 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCSvyNgXcKPLhAAP/Bj2F+06DCtPCzQDNq+cQ==", 2021-02-10T11:32:27.402 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:32:27.402 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:32:27.402 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:32:27.402 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:32:27.402 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:32:27.402 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:32:27.403 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:32:27.403 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.1", 2021-02-10T11:32:27.403 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCVvyNghfbcLhAA66PrRwR2M7o6VTF5UhU/Fg==", 2021-02-10T11:32:27.403 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:32:27.403 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:32:27.403 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:32:27.403 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:32:27.403 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:32:27.403 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:32:27.403 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:32:27.403 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.2", 2021-02-10T11:32:27.403 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCYvyNgwiPqLhAA66BVkRsgaUlGpAsQ15Nf8Q==", 2021-02-10T11:32:27.403 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:32:27.404 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:32:27.404 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:32:27.404 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:32:27.404 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:32:27.404 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:32:27.404 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:32:27.404 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.3", 2021-02-10T11:32:27.404 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCbvyNgqBZrLxAAERPe/1L3SXAEzI3d9ReHHg==", 2021-02-10T11:32:27.404 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:32:27.404 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:32:27.404 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:32:27.404 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:32:27.404 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:32:27.404 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:32:27.405 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:32:27.405 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.4", 2021-02-10T11:32:27.405 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCkvyNgDQsxIhAAHmVjVYnx20N4SDmKqvbcuA==", 2021-02-10T11:32:27.405 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:32:27.405 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:32:27.405 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:32:27.405 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:32:27.405 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:32:27.405 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:32:27.405 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:32:27.405 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.5", 2021-02-10T11:32:27.405 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCnvyNg8ITFIhAADd4aKybAV7btoGPx3kCg3A==", 2021-02-10T11:32:27.406 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:32:27.406 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:32:27.406 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:32:27.406 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:32:27.406 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:32:27.406 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:32:27.406 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:32:27.406 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.6", 2021-02-10T11:32:27.406 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCqvyNgxOhDIxAAzXixEa/nlDc1xXrg0aeLwg==", 2021-02-10T11:32:27.406 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:32:27.406 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:32:27.406 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:32:27.406 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:32:27.407 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:32:27.407 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:32:27.407 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:32:27.407 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.7", 2021-02-10T11:32:27.407 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCtvyNgxSqGIxAACxJQlwP7DpiVVCZHA3zn1A==", 2021-02-10T11:32:27.407 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:32:27.407 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:32:27.407 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:32:27.407 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:32:27.407 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:32:27.407 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:32:27.407 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:32:27.407 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.0", 2021-02-10T11:32:27.408 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCLvyNgDkNiHRAAygU8Za4PBusTqCgwAYWTiQ==", 2021-02-10T11:32:27.408 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:32:27.408 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:32:27.408 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow r", 2021-02-10T11:32:27.408 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:32:27.408 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:32:27.408 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:32:27.408 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:32:27.408 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.1", 2021-02-10T11:32:27.408 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCLvyNgxmLqMxAA1MlSDJ9LH9u4f2sL7mm8/Q==", 2021-02-10T11:32:27.408 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:32:27.408 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:32:27.408 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow r", 2021-02-10T11:32:27.409 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:32:27.409 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:32:27.409 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:32:27.409 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:32:27.409 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.2", 2021-02-10T11:32:27.409 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCMvyNgp8veDhAAniSVHYPtwwTog/pjbaRQmw==", 2021-02-10T11:32:27.409 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:32:27.409 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:32:27.409 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow r", 2021-02-10T11:32:27.409 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:32:27.409 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:32:27.409 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:32:27.409 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:32:27.410 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.3", 2021-02-10T11:32:27.410 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCMvyNgQEVoJRAApHcWdm1sldIxu+heyD4HdQ==", 2021-02-10T11:32:27.410 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:32:27.410 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:32:27.410 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow r", 2021-02-10T11:32:27.410 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:32:27.410 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:32:27.410 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:32:27.410 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:32:27.410 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.admin", 2021-02-10T11:32:27.410 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCGvyNg7jjJBRAAtot+ypUZOiTo1GbY+oVhVw==", 2021-02-10T11:32:27.410 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:32:27.411 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow *", 2021-02-10T11:32:27.411 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:32:27.411 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:32:27.411 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:32:27.411 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:32:27.411 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:32:27.411 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:32:27.411 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "mgr.x", 2021-02-10T11:32:27.411 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCJvyNgTPVwKBAAjduiBE8+RSXBv82tdPecWg==", 2021-02-10T11:32:27.411 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:32:27.411 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow *", 2021-02-10T11:32:27.411 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow profile mgr", 2021-02-10T11:32:27.411 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:32:27.411 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:32:27.412 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:32:27.412 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:32:27.412 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "mgr.y", 2021-02-10T11:32:27.412 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCJvyNgQ1tTDhAASwBssDy+Rz31ujw8fNlWow==", 2021-02-10T11:32:27.412 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:32:27.412 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow *", 2021-02-10T11:32:27.412 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow profile mgr", 2021-02-10T11:32:27.412 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:32:27.412 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:32:27.412 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:32:27.412 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:32:27.412 INFO:teuthology.orchestra.run.gibba012.stdout:} 2021-02-10T11:32:27.414 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.414 7f106a33d700 1 -- 172.21.2.112:0/1068104405 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f10440186c0 msgr2=0x7f104401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:27.414 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.414 7f106a33d700 1 --2- 172.21.2.112:0/1068104405 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f10440186c0 0x7f104401abc0 crc :-1 s=READY pgs=558 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:27.414 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.414 7f106a33d700 1 -- 172.21.2.112:0/1068104405 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f106409fe40 msgr2=0x7f1064091230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:27.414 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.414 7f106a33d700 1 --2- 172.21.2.112:0/1068104405 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f106409fe40 0x7f1064091230 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 rx=0x7f105c009940 tx=0x7f105c00bcf0).stop 2021-02-10T11:32:27.414 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.414 7f106a33d700 1 -- 172.21.2.112:0/1068104405 shutdown_connections 2021-02-10T11:32:27.414 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.414 7f106a33d700 1 --2- 172.21.2.112:0/1068104405 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f10440186c0 0x7f104401abc0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:27.414 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.414 7f106a33d700 1 --2- 172.21.2.112:0/1068104405 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f106409fe40 0x7f1064091230 unknown :-1 s=CLOSED pgs=1102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:27.414 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.414 7f106a33d700 1 --2- 172.21.2.112:0/1068104405 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f10640b49a0 0x7f106408a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:27.414 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.414 7f106a33d700 1 --2- 172.21.2.112:0/1068104405 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f10641abb40 0x7f10640905e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:27.415 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.414 7f106a33d700 1 -- 172.21.2.112:0/1068104405 >> 172.21.2.112:0/1068104405 conn(0x7f10641b2d20 msgr2=0x7f10641912a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:27.415 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.415 7f106a33d700 1 -- 172.21.2.112:0/1068104405 shutdown_connections 2021-02-10T11:32:27.415 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.415 7f106a33d700 1 -- 172.21.2.112:0/1068104405 wait complete. 2021-02-10T11:32:27.415 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:32:27.843 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.839 7f0cad27f700 1 Processor -- start 2021-02-10T11:32:27.843 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.839 7f0cad27f700 1 -- start start 2021-02-10T11:32:27.843 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.840 7f0cad27f700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f0ca80e5470 0x7f0ca807e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:27.843 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.840 7f0cad27f700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f0ca807f300 0x7f0ca807f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:27.843 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.840 7f0cad27f700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ca807ed80 con 0x7f0ca80e5070 2021-02-10T11:32:27.843 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.840 7f0cad27f700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ca81b4010 con 0x7f0ca80e5470 2021-02-10T11:32:27.844 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.840 7f0cad27f700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ca80d1790 con 0x7f0ca807f300 2021-02-10T11:32:27.844 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.840 7f0ca787e700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f0ca807f300 0x7f0ca807f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:27.844 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.840 7f0ca687c700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f0ca80e5470 0x7f0ca807e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:27.844 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.840 7f0ca707d700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7f0ca80e5070 0x7f0ca81b3820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:41910/0 (socket says 172.21.2.112:41910) 2021-02-10T11:32:27.844 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.840 7f0ca787e700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f0ca807f300 0x7f0ca807f700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3300/0 says I am v2:172.21.2.112:51174/0 (socket says 172.21.2.112:51174) 2021-02-10T11:32:27.844 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.840 7f0ca707d700 1 -- 172.21.2.112:0/1094396166 learned_addr learned my addr 172.21.2.112:0/1094396166 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:32:27.844 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.840 7f0ca787e700 1 -- 172.21.2.112:0/1094396166 >> v2:172.21.2.112:3300/0 conn(0x7f0ca80e5470 msgr2=0x7f0ca807e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:27.844 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.840 7f0ca787e700 1 --2- 172.21.2.112:0/1094396166 >> v2:172.21.2.112:3300/0 conn(0x7f0ca80e5470 0x7f0ca807e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:27.844 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.840 7f0ca787e700 1 -- 172.21.2.112:0/1094396166 >> v1:172.21.2.120:6789/0 conn(0x7f0ca80e5070 legacy=0x7f0ca81b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:27.844 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.840 7f0ca787e700 1 -- 172.21.2.112:0/1094396166 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ca80d19b0 con 0x7f0ca807f300 2021-02-10T11:32:27.844 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.840 7f0ca687c700 1 --2- 172.21.2.112:0/1094396166 >> v2:172.21.2.112:3300/0 conn(0x7f0ca80e5470 0x7f0ca807e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:32:27.844 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.840 7f0ca787e700 1 --2- 172.21.2.112:0/1094396166 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0ca807f300 0x7f0ca807f700 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 rx=0x7f0ca0006e50 tx=0x7f0ca0009e70).ready entity=mon.1 client_cookie=6489e34d1f2dcaa1 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:27.844 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.841 7f0ca607b700 1 -- 172.21.2.112:0/1094396166 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0ca00066f0 con 0x7f0ca807f300 2021-02-10T11:32:27.845 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.841 7f0ca607b700 1 -- 172.21.2.112:0/1094396166 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ca0003030 con 0x7f0ca807f300 2021-02-10T11:32:27.845 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.841 7f0ca607b700 1 -- 172.21.2.112:0/1094396166 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0ca000b700 con 0x7f0ca807f300 2021-02-10T11:32:27.845 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.841 7f0cad27f700 1 -- 172.21.2.112:0/1094396166 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0ca807f300 msgr2=0x7f0ca807f700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:27.845 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.841 7f0cad27f700 1 --2- 172.21.2.112:0/1094396166 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0ca807f300 0x7f0ca807f700 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 rx=0x7f0ca0006e50 tx=0x7f0ca0009e70).stop 2021-02-10T11:32:27.845 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.841 7f0cad27f700 1 -- 172.21.2.112:0/1094396166 shutdown_connections 2021-02-10T11:32:27.845 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.841 7f0cad27f700 1 --2- 172.21.2.112:0/1094396166 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0ca807f300 0x7f0ca807f700 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:27.845 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.841 7f0cad27f700 1 --2- 172.21.2.112:0/1094396166 >> v2:172.21.2.112:3300/0 conn(0x7f0ca80e5470 0x7f0ca807e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:27.845 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.841 7f0cad27f700 1 -- 172.21.2.112:0/1094396166 >> 172.21.2.112:0/1094396166 conn(0x7f0ca80ce690 msgr2=0x7f0ca80e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:27.845 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.841 7f0cad27f700 1 -- 172.21.2.112:0/1094396166 shutdown_connections 2021-02-10T11:32:27.845 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.841 7f0cad27f700 1 -- 172.21.2.112:0/1094396166 wait complete. 2021-02-10T11:32:27.847 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.856 7f0cad27f700 1 Processor -- start 2021-02-10T11:32:27.847 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.856 7f0cad27f700 1 -- start start 2021-02-10T11:32:27.847 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.857 7f0cad27f700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0ca808d400 0x7f0ca8096c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:27.847 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.857 7f0cad27f700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0ca80b1450 0x7f0ca80b1850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:27.847 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.857 7f0cad27f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0ca80b1e70 0x7f0ca81b1f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:27.847 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.857 7f0cad27f700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ca8097310 con 0x7f0ca80b1e70 2021-02-10T11:32:27.847 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.857 7f0cad27f700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ca808d800 con 0x7f0ca808d400 2021-02-10T11:32:27.847 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.857 7f0cad27f700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ca808d9f0 con 0x7f0ca80b1450 2021-02-10T11:32:27.847 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.857 7f0ca787e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0ca80b1e70 0x7f0ca81b1f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:27.847 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.857 7f0ca707d700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0ca808d400 0x7f0ca8096c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:27.848 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.857 7f0ca707d700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0ca808d400 0x7f0ca8096c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3300/0 says I am v2:172.21.2.112:51176/0 (socket says 172.21.2.112:51176) 2021-02-10T11:32:27.848 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.857 7f0ca787e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0ca80b1e70 0x7f0ca81b1f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:48900/0 (socket says 172.21.2.112:48900) 2021-02-10T11:32:27.848 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.857 7f0ca707d700 1 -- 172.21.2.112:0/1994486787 learned_addr learned my addr 172.21.2.112:0/1994486787 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:32:27.848 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.857 7f0ca687c700 1 --2- 172.21.2.112:0/1994486787 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0ca80b1450 0x7f0ca80b1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:27.848 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.857 7f0ca787e700 1 -- 172.21.2.112:0/1994486787 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0ca80b1450 msgr2=0x7f0ca80b1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:27.848 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.857 7f0ca787e700 1 --2- 172.21.2.112:0/1994486787 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0ca80b1450 0x7f0ca80b1850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:27.848 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.857 7f0ca787e700 1 -- 172.21.2.112:0/1994486787 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0ca808d400 msgr2=0x7f0ca8096c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:27.848 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.857 7f0ca787e700 1 --2- 172.21.2.112:0/1994486787 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0ca808d400 0x7f0ca8096c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:27.848 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.857 7f0ca787e700 1 -- 172.21.2.112:0/1994486787 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ca81b25f0 con 0x7f0ca80b1e70 2021-02-10T11:32:27.848 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.857 7f0ca707d700 1 --2- 172.21.2.112:0/1994486787 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0ca808d400 0x7f0ca8096c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:32:27.848 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.858 7f0ca787e700 1 --2- 172.21.2.112:0/1994486787 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0ca80b1e70 0x7f0ca81b1f60 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 rx=0x7f0ca000b870 tx=0x7f0ca000bc10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:27.848 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.858 7f0c9ffff700 1 -- 172.21.2.112:0/1994486787 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0ca0007240 con 0x7f0ca80b1e70 2021-02-10T11:32:27.848 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.858 7f0c9ffff700 1 -- 172.21.2.112:0/1994486787 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ca0009030 con 0x7f0ca80b1e70 2021-02-10T11:32:27.849 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.858 7f0c9ffff700 1 -- 172.21.2.112:0/1994486787 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0ca00076b0 con 0x7f0ca80b1e70 2021-02-10T11:32:27.849 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.858 7f0cad27f700 1 -- 172.21.2.112:0/1994486787 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ca81b25f0 con 0x7f0ca80b1e70 2021-02-10T11:32:27.849 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.858 7f0cad27f700 1 -- 172.21.2.112:0/1994486787 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ca81b25f0 con 0x7f0ca80b1e70 2021-02-10T11:32:27.849 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.858 7f0c9ffff700 1 -- 172.21.2.112:0/1994486787 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0ca0007240 con 0x7f0ca80b1e70 2021-02-10T11:32:27.849 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.859 7f0c9ffff700 1 --2- 172.21.2.112:0/1994486787 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0c8c018730 0x7f0c8c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:27.849 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.859 7f0c9ffff700 1 -- 172.21.2.112:0/1994486787 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f0ca0032780 con 0x7f0ca80b1e70 2021-02-10T11:32:27.849 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.859 7f0ca707d700 1 --2- 172.21.2.112:0/1994486787 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0c8c018730 0x7f0c8c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:27.849 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.859 7f0ca707d700 1 --2- 172.21.2.112:0/1994486787 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0c8c018730 0x7f0c8c01abc0 crc :-1 s=READY pgs=559 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:27.876 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.889 7f0cad27f700 1 -- 172.21.2.112:0/1994486787 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ca807ed80 con 0x7f0ca80b1e70 2021-02-10T11:32:27.879 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:27.892 7f0c9ffff700 1 -- 172.21.2.112:0/1994486787 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0ca0009010 con 0x7f0ca80b1e70 2021-02-10T11:32:28.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:28.017 7f0cad27f700 1 -- 172.21.2.112:0/1994486787 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0ca807ed80 con 0x7f0ca80b1e70 2021-02-10T11:32:28.005 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:28.018 7f0c9ffff700 1 -- 172.21.2.112:0/1994486787 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v42) v1 ==== 74+0+11144 (secure 0 0 0) 0x7f0ca0009010 con 0x7f0ca80b1e70 2021-02-10T11:32:28.005 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:32:28.005 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":42,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:32:26.759890","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:32:28.007 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:28.019 7f0cad27f700 1 -- 172.21.2.112:0/1994486787 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0c8c018730 msgr2=0x7f0c8c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:28.007 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:28.019 7f0cad27f700 1 --2- 172.21.2.112:0/1994486787 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0c8c018730 0x7f0c8c01abc0 crc :-1 s=READY pgs=559 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:28.007 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:28.019 7f0cad27f700 1 -- 172.21.2.112:0/1994486787 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0ca80b1e70 msgr2=0x7f0ca81b1f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:28.007 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:28.019 7f0cad27f700 1 --2- 172.21.2.112:0/1994486787 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0ca80b1e70 0x7f0ca81b1f60 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 rx=0x7f0ca000b870 tx=0x7f0ca000bc10).stop 2021-02-10T11:32:28.007 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:28.020 7f0cad27f700 1 -- 172.21.2.112:0/1994486787 shutdown_connections 2021-02-10T11:32:28.007 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:28.020 7f0cad27f700 1 --2- 172.21.2.112:0/1994486787 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0c8c018730 0x7f0c8c01abc0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:28.007 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:28.020 7f0cad27f700 1 --2- 172.21.2.112:0/1994486787 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0ca80b1e70 0x7f0ca81b1f60 unknown :-1 s=CLOSED pgs=1103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:28.007 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:28.020 7f0cad27f700 1 --2- 172.21.2.112:0/1994486787 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0ca80b1450 0x7f0ca80b1850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:28.007 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:28.020 7f0cad27f700 1 --2- 172.21.2.112:0/1994486787 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0ca808d400 0x7f0ca8096c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:28.007 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:28.020 7f0cad27f700 1 -- 172.21.2.112:0/1994486787 >> 172.21.2.112:0/1994486787 conn(0x7f0ca81ac620 msgr2=0x7f0ca80896e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:28.008 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:28.020 7f0cad27f700 1 -- 172.21.2.112:0/1994486787 shutdown_connections 2021-02-10T11:32:28.008 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:28.020 7f0cad27f700 1 -- 172.21.2.112:0/1994486787 wait complete. 2021-02-10T11:32:28.016 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:13:57.508758', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2021-02-10T11:32:28.016 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:32:28.445 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:28.441 7f012036d700 1 Processor -- start 2021-02-10T11:32:28.445 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:28.441 7f012036d700 1 -- start start 2021-02-10T11:32:28.445 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:28.442 7f012036d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f01181b0870 0x7f01180f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:28.446 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:28.442 7f012036d700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01181b0c70 con 0x7f01181b0470 2021-02-10T11:32:28.446 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:28.442 7f012036d700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01180f0b50 con 0x7f01180efe00 2021-02-10T11:32:28.446 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:28.442 7f012036d700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01180e5ea0 con 0x7f01181b0870 2021-02-10T11:32:28.446 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:28.442 7f011d988700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f01181b0870 0x7f01180f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:28.446 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:28.442 7f011e189700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f01181b0470 0x7f01180ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:48366/0 (socket says 172.21.2.112:48366) 2021-02-10T11:32:28.446 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:28.442 7f011e189700 1 -- 172.21.2.112:0/1156782399 learned_addr learned my addr 172.21.2.112:0/1156782399 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:32:28.446 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:28.442 7f011d187700 1 -- 172.21.2.112:0/1156782399 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2109182760 0 0) 0x7f010c001180 con 0x7f01181b0470 2021-02-10T11:32:28.446 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:28.443 7f011d187700 1 -- 172.21.2.112:0/1156782399 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0104002fe0 con 0x7f01181b0470 2021-02-10T11:32:28.446 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:28.443 7f011d988700 1 -- 172.21.2.112:0/1156782399 >> v1:172.21.2.120:6789/0 conn(0x7f01180efe00 legacy=0x7f01180f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:28.446 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:28.443 7f011d988700 1 -- 172.21.2.112:0/1156782399 >> v1:172.21.2.112:6789/0 conn(0x7f01181b0470 legacy=0x7f01180ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:28.446 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:28.443 7f011d988700 1 -- 172.21.2.112:0/1156782399 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01180e52e0 con 0x7f01181b0870 2021-02-10T11:32:28.446 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:28.443 7f011d988700 1 --2- 172.21.2.112:0/1156782399 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f01181b0870 0x7f01180f20e0 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 rx=0x7f0114006650 tx=0x7f0114009a80).ready entity=mon.0 client_cookie=883ce5d20b373bfe server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:28.447 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:28.443 7f011d187700 1 -- 172.21.2.112:0/1156782399 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f01140009b0 con 0x7f01181b0870 2021-02-10T11:32:28.447 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:28.443 7f011d187700 1 -- 172.21.2.112:0/1156782399 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f011400c0c0 con 0x7f01181b0870 2021-02-10T11:32:28.447 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:28.443 7f011d187700 1 -- 172.21.2.112:0/1156782399 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f011400c520 con 0x7f01181b0870 2021-02-10T11:32:28.447 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:28.443 7f012036d700 1 -- 172.21.2.112:0/1156782399 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f01181b0870 msgr2=0x7f01180f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:28.447 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:28.443 7f012036d700 1 --2- 172.21.2.112:0/1156782399 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f01181b0870 0x7f01180f20e0 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 rx=0x7f0114006650 tx=0x7f0114009a80).stop 2021-02-10T11:32:28.447 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:28.444 7f012036d700 1 -- 172.21.2.112:0/1156782399 shutdown_connections 2021-02-10T11:32:28.447 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:28.444 7f012036d700 1 --2- 172.21.2.112:0/1156782399 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f01181b0870 0x7f01180f20e0 unknown :-1 s=CLOSED pgs=1105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:28.447 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:28.444 7f012036d700 1 -- 172.21.2.112:0/1156782399 >> 172.21.2.112:0/1156782399 conn(0x7f01180ce3e0 msgr2=0x7f01180e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:28.447 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:28.444 7f012036d700 1 -- 172.21.2.112:0/1156782399 shutdown_connections 2021-02-10T11:32:28.447 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:28.444 7f012036d700 1 -- 172.21.2.112:0/1156782399 wait complete. 2021-02-10T11:32:28.449 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:28.459 7f012036d700 1 Processor -- start 2021-02-10T11:32:28.449 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:28.459 7f012036d700 1 -- start start 2021-02-10T11:32:28.449 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:28.459 7f012036d700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f01180a6c90 0x7f01181924c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:28.449 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:28.459 7f012036d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0118192ae0 0x7f0118192ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:28.449 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:28.459 7f012036d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f01180919e0 0x7f0118091de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:28.449 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:28.459 7f012036d700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f011819d2e0 con 0x7f01180919e0 2021-02-10T11:32:28.449 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:28.459 7f012036d700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01180914d0 con 0x7f01180a6c90 2021-02-10T11:32:28.449 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:28.459 7f012036d700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01180a6820 con 0x7f0118192ae0 2021-02-10T11:32:28.449 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:28.459 7f011e98a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f01180919e0 0x7f0118091de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:28.449 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:28.459 7f011e98a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f01180919e0 0x7f0118091de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:48912/0 (socket says 172.21.2.112:48912) 2021-02-10T11:32:28.449 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:28.459 7f011e98a700 1 -- 172.21.2.112:0/1610366340 learned_addr learned my addr 172.21.2.112:0/1610366340 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:32:28.449 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:28.460 7f011e98a700 1 -- 172.21.2.112:0/1610366340 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0118192ae0 msgr2=0x7f0118192ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:28.450 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:28.460 7f011e98a700 1 --2- 172.21.2.112:0/1610366340 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0118192ae0 0x7f0118192ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:28.450 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:28.460 7f011e98a700 1 -- 172.21.2.112:0/1610366340 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f01180a6c90 msgr2=0x7f01181924c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:28.450 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:28.460 7f011e98a700 1 --2- 172.21.2.112:0/1610366340 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f01180a6c90 0x7f01181924c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:28.450 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:28.460 7f011e98a700 1 -- 172.21.2.112:0/1610366340 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0118092400 con 0x7f01180919e0 2021-02-10T11:32:28.450 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:28.460 7f011e98a700 1 --2- 172.21.2.112:0/1610366340 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f01180919e0 0x7f0118091de0 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 rx=0x7f0110009940 tx=0x7f011000bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:28.450 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:28.460 7f010affd700 1 -- 172.21.2.112:0/1610366340 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0110000f90 con 0x7f01180919e0 2021-02-10T11:32:28.450 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:28.460 7f010affd700 1 -- 172.21.2.112:0/1610366340 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f011000c2f0 con 0x7f01180919e0 2021-02-10T11:32:28.450 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:28.460 7f012036d700 1 -- 172.21.2.112:0/1610366340 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0118092400 con 0x7f01180919e0 2021-02-10T11:32:28.450 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:28.460 7f010affd700 1 -- 172.21.2.112:0/1610366340 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f011000c750 con 0x7f01180919e0 2021-02-10T11:32:28.450 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:28.460 7f012036d700 1 -- 172.21.2.112:0/1610366340 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0118092400 con 0x7f01180919e0 2021-02-10T11:32:28.450 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:28.461 7f010affd700 1 -- 172.21.2.112:0/1610366340 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0110001220 con 0x7f01180919e0 2021-02-10T11:32:28.450 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:28.461 7f010affd700 1 --2- 172.21.2.112:0/1610366340 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f01000186c0 0x7f010001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:28.450 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:28.461 7f010affd700 1 -- 172.21.2.112:0/1610366340 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f0110032430 con 0x7f01180919e0 2021-02-10T11:32:28.451 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:28.461 7f011e189700 1 --2- 172.21.2.112:0/1610366340 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f01000186c0 0x7f010001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:28.451 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:28.462 7f011e189700 1 --2- 172.21.2.112:0/1610366340 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f01000186c0 0x7f010001abc0 crc :-1 s=READY pgs=560 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:28.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:28.492 7f012036d700 1 -- 172.21.2.112:0/1610366340 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01180e52e0 con 0x7f01180919e0 2021-02-10T11:32:28.481 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:28.494 7f010affd700 1 -- 172.21.2.112:0/1610366340 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f011000be30 con 0x7f01180919e0 2021-02-10T11:32:28.610 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:28.622 7f012036d700 1 -- 172.21.2.112:0/1610366340 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f01180e52e0 con 0x7f01180919e0 2021-02-10T11:32:28.610 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:28.623 7f010affd700 1 -- 172.21.2.112:0/1610366340 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v42) v1 ==== 92+0+10 (secure 0 0 0) 0x7f011001f860 con 0x7f01180919e0 2021-02-10T11:32:28.611 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:32:28.612 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:28.625 7f012036d700 1 -- 172.21.2.112:0/1610366340 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f01000186c0 msgr2=0x7f010001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:28.612 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:28.625 7f012036d700 1 --2- 172.21.2.112:0/1610366340 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f01000186c0 0x7f010001abc0 crc :-1 s=READY pgs=560 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:28.612 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:28.625 7f012036d700 1 -- 172.21.2.112:0/1610366340 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f01180919e0 msgr2=0x7f0118091de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:28.612 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:28.625 7f012036d700 1 --2- 172.21.2.112:0/1610366340 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f01180919e0 0x7f0118091de0 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 rx=0x7f0110009940 tx=0x7f011000bcf0).stop 2021-02-10T11:32:28.613 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:28.625 7f012036d700 1 -- 172.21.2.112:0/1610366340 shutdown_connections 2021-02-10T11:32:28.613 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:28.625 7f012036d700 1 --2- 172.21.2.112:0/1610366340 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f01000186c0 0x7f010001abc0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:28.613 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:28.625 7f012036d700 1 --2- 172.21.2.112:0/1610366340 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f01180919e0 0x7f0118091de0 unknown :-1 s=CLOSED pgs=1106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:28.613 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:28.625 7f012036d700 1 --2- 172.21.2.112:0/1610366340 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0118192ae0 0x7f0118192ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:28.613 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:28.625 7f012036d700 1 --2- 172.21.2.112:0/1610366340 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f01180a6c90 0x7f01181924c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:28.613 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:28.625 7f012036d700 1 -- 172.21.2.112:0/1610366340 >> 172.21.2.112:0/1610366340 conn(0x7f01180ac550 msgr2=0x7f01180be1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:28.613 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:28.625 7f012036d700 1 -- 172.21.2.112:0/1610366340 shutdown_connections 2021-02-10T11:32:28.613 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:28.625 7f012036d700 1 -- 172.21.2.112:0/1610366340 wait complete. 2021-02-10T11:32:28.621 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --format=json-pretty osd lspools 2021-02-10T11:32:29.045 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.042 7f6c60129700 1 Processor -- start 2021-02-10T11:32:29.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.042 7f6c60129700 1 -- start start 2021-02-10T11:32:29.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.043 7f6c60129700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f6c580cff00 0x7f6c581b39d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:29.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.043 7f6c60129700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f6c580d1bf0 0x7f6c580ef620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:29.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.043 7f6c60129700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c580f0b20 con 0x7f6c580d17f0 2021-02-10T11:32:29.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.043 7f6c60129700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c580e5eb0 con 0x7f6c580cff00 2021-02-10T11:32:29.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.043 7f6c60129700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c581b4020 con 0x7f6c580d1bf0 2021-02-10T11:32:29.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.043 7f6c5d744700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f6c580cff00 0x7f6c581b39d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:29.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.043 7f6c5d744700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f6c580cff00 0x7f6c581b39d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:48918/0 (socket says 172.21.2.112:48918) 2021-02-10T11:32:29.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.043 7f6c5d744700 1 -- 172.21.2.112:0/4002836364 learned_addr learned my addr 172.21.2.112:0/4002836364 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:32:29.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.043 7f6c5d744700 1 -- 172.21.2.112:0/4002836364 >> v2:172.21.2.120:3300/0 conn(0x7f6c580d1bf0 msgr2=0x7f6c580ef620 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:32:29.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.043 7f6c5d744700 1 --2- 172.21.2.112:0/4002836364 >> v2:172.21.2.120:3300/0 conn(0x7f6c580d1bf0 0x7f6c580ef620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:29.047 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.043 7f6c5d744700 1 -- 172.21.2.112:0/4002836364 >> v1:172.21.2.120:6789/0 conn(0x7f6c580d17f0 legacy=0x7f6c581b31e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:32:29.047 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.043 7f6c5d744700 1 -- 172.21.2.112:0/4002836364 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c580f20b0 con 0x7f6c580cff00 2021-02-10T11:32:29.047 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.043 7f6c5d744700 1 --2- 172.21.2.112:0/4002836364 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6c580cff00 0x7f6c581b39d0 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 rx=0x7f6c48006700 tx=0x7f6c48009b80).ready entity=mon.0 client_cookie=e6d414d93ffcd0a2 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:29.047 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.043 7f6c5cf43700 1 -- 172.21.2.112:0/4002836364 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6c4800c0c0 con 0x7f6c580cff00 2021-02-10T11:32:29.047 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.044 7f6c5cf43700 1 -- 172.21.2.112:0/4002836364 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c4800c440 con 0x7f6c580cff00 2021-02-10T11:32:29.047 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.044 7f6c5cf43700 1 -- 172.21.2.112:0/4002836364 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6c4800c920 con 0x7f6c580cff00 2021-02-10T11:32:29.047 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.044 7f6c60129700 1 -- 172.21.2.112:0/4002836364 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6c580cff00 msgr2=0x7f6c581b39d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:29.047 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.044 7f6c60129700 1 --2- 172.21.2.112:0/4002836364 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6c580cff00 0x7f6c581b39d0 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 rx=0x7f6c48006700 tx=0x7f6c48009b80).stop 2021-02-10T11:32:29.047 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.044 7f6c60129700 1 -- 172.21.2.112:0/4002836364 shutdown_connections 2021-02-10T11:32:29.047 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.044 7f6c60129700 1 --2- 172.21.2.112:0/4002836364 >> v2:172.21.2.120:3300/0 conn(0x7f6c580d1bf0 0x7f6c580ef620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:29.047 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.044 7f6c60129700 1 --2- 172.21.2.112:0/4002836364 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6c580cff00 0x7f6c581b39d0 unknown :-1 s=CLOSED pgs=1107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:29.047 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.044 7f6c60129700 1 -- 172.21.2.112:0/4002836364 >> 172.21.2.112:0/4002836364 conn(0x7f6c580fde40 msgr2=0x7f6c580ce390 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:29.047 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.044 7f6c60129700 1 -- 172.21.2.112:0/4002836364 shutdown_connections 2021-02-10T11:32:29.048 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.044 7f6c60129700 1 -- 172.21.2.112:0/4002836364 wait complete. 2021-02-10T11:32:29.049 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.059 7f6c60129700 1 Processor -- start 2021-02-10T11:32:29.049 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.059 7f6c60129700 1 -- start start 2021-02-10T11:32:29.049 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.059 7f6c60129700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6c580b3480 0x7f6c580b3880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:29.049 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.059 7f6c60129700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6c58106330 0x7f6c58106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:29.049 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.059 7f6c60129700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6c58106d50 0x7f6c58107150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:29.049 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.059 7f6c60129700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c581038d0 con 0x7f6c580b3480 2021-02-10T11:32:29.049 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.059 7f6c60129700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c5818d9c0 con 0x7f6c58106330 2021-02-10T11:32:29.050 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.059 7f6c60129700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c5818dbe0 con 0x7f6c58106d50 2021-02-10T11:32:29.050 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.060 7f6c5df45700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6c580b3480 0x7f6c580b3880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:29.050 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.060 7f6c5df45700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6c580b3480 0x7f6c580b3880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:48920/0 (socket says 172.21.2.112:48920) 2021-02-10T11:32:29.050 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.060 7f6c5df45700 1 -- 172.21.2.112:0/2630620326 learned_addr learned my addr 172.21.2.112:0/2630620326 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:32:29.050 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.060 7f6c5d744700 1 --2- 172.21.2.112:0/2630620326 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6c58106330 0x7f6c58106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:29.050 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.060 7f6c5e746700 1 --2- 172.21.2.112:0/2630620326 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6c58106d50 0x7f6c58107150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:29.050 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.060 7f6c5df45700 1 -- 172.21.2.112:0/2630620326 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6c58106d50 msgr2=0x7f6c58107150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:29.050 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.060 7f6c5df45700 1 --2- 172.21.2.112:0/2630620326 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6c58106d50 0x7f6c58107150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:29.050 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.060 7f6c5df45700 1 -- 172.21.2.112:0/2630620326 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6c58106330 msgr2=0x7f6c58106730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:29.050 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.060 7f6c5df45700 1 --2- 172.21.2.112:0/2630620326 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6c58106330 0x7f6c58106730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:29.050 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.060 7f6c5df45700 1 -- 172.21.2.112:0/2630620326 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c5818de00 con 0x7f6c580b3480 2021-02-10T11:32:29.050 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.060 7f6c5d744700 1 --2- 172.21.2.112:0/2630620326 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6c58106330 0x7f6c58106730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:32:29.051 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.060 7f6c5df45700 1 --2- 172.21.2.112:0/2630620326 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6c580b3480 0x7f6c580b3880 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 rx=0x7f6c5400b5b0 tx=0x7f6c5400b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:29.051 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.061 7f6c4e7fc700 1 -- 172.21.2.112:0/2630620326 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6c5400bd40 con 0x7f6c580b3480 2021-02-10T11:32:29.051 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.061 7f6c4e7fc700 1 -- 172.21.2.112:0/2630620326 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c54002080 con 0x7f6c580b3480 2021-02-10T11:32:29.051 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.061 7f6c4e7fc700 1 -- 172.21.2.112:0/2630620326 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6c54002510 con 0x7f6c580b3480 2021-02-10T11:32:29.051 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.061 7f6c60129700 1 -- 172.21.2.112:0/2630620326 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c5818de00 con 0x7f6c580b3480 2021-02-10T11:32:29.051 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.061 7f6c60129700 1 -- 172.21.2.112:0/2630620326 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c5818de00 con 0x7f6c580b3480 2021-02-10T11:32:29.051 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.061 7f6c4e7fc700 1 -- 172.21.2.112:0/2630620326 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6c54002260 con 0x7f6c580b3480 2021-02-10T11:32:29.051 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.061 7f6c4e7fc700 1 --2- 172.21.2.112:0/2630620326 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6c400186c0 0x7f6c4001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:29.051 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.062 7f6c4e7fc700 1 -- 172.21.2.112:0/2630620326 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f6c54032280 con 0x7f6c580b3480 2021-02-10T11:32:29.051 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.062 7f6c5d744700 1 --2- 172.21.2.112:0/2630620326 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6c400186c0 0x7f6c4001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:29.052 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.062 7f6c5d744700 1 --2- 172.21.2.112:0/2630620326 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6c400186c0 0x7f6c4001abc0 crc :-1 s=READY pgs=561 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:29.080 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.092 7f6c60129700 1 -- 172.21.2.112:0/2630620326 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c580f0b20 con 0x7f6c580b3480 2021-02-10T11:32:29.082 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.095 7f6c4e7fc700 1 -- 172.21.2.112:0/2630620326 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6c54000c60 con 0x7f6c580b3480 2021-02-10T11:32:29.207 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.220 7f6c60129700 1 -- 172.21.2.112:0/2630620326 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f6c580f0b20 con 0x7f6c580b3480 2021-02-10T11:32:29.207 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.220 7f6c4e7fc700 1 -- 172.21.2.112:0/2630620326 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v42) v1 ==== 84+0+64 (secure 0 0 0) 0x7f6c5401f520 con 0x7f6c580b3480 2021-02-10T11:32:29.208 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:32:29.208 INFO:teuthology.orchestra.run.gibba012.stdout:[ 2021-02-10T11:32:29.208 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:32:29.208 INFO:teuthology.orchestra.run.gibba012.stdout: "poolnum": 1, 2021-02-10T11:32:29.208 INFO:teuthology.orchestra.run.gibba012.stdout: "poolname": "rbd" 2021-02-10T11:32:29.208 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:32:29.208 INFO:teuthology.orchestra.run.gibba012.stdout:] 2021-02-10T11:32:29.209 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.222 7f6c60129700 1 -- 172.21.2.112:0/2630620326 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6c400186c0 msgr2=0x7f6c4001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:29.210 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.222 7f6c60129700 1 --2- 172.21.2.112:0/2630620326 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6c400186c0 0x7f6c4001abc0 crc :-1 s=READY pgs=561 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:29.210 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.222 7f6c60129700 1 -- 172.21.2.112:0/2630620326 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6c580b3480 msgr2=0x7f6c580b3880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:29.210 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.222 7f6c60129700 1 --2- 172.21.2.112:0/2630620326 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6c580b3480 0x7f6c580b3880 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 rx=0x7f6c5400b5b0 tx=0x7f6c5400b950).stop 2021-02-10T11:32:29.210 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.222 7f6c60129700 1 -- 172.21.2.112:0/2630620326 shutdown_connections 2021-02-10T11:32:29.210 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.222 7f6c60129700 1 --2- 172.21.2.112:0/2630620326 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6c400186c0 0x7f6c4001abc0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:29.210 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.222 7f6c60129700 1 --2- 172.21.2.112:0/2630620326 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6c58106330 0x7f6c58106730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:29.210 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.222 7f6c60129700 1 --2- 172.21.2.112:0/2630620326 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6c58106d50 0x7f6c58107150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:29.210 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.222 7f6c60129700 1 --2- 172.21.2.112:0/2630620326 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6c580b3480 0x7f6c580b3880 unknown :-1 s=CLOSED pgs=1108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:29.210 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.222 7f6c60129700 1 -- 172.21.2.112:0/2630620326 >> 172.21.2.112:0/2630620326 conn(0x7f6c581a3780 msgr2=0x7f6c581a5db0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:29.210 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.223 7f6c60129700 1 -- 172.21.2.112:0/2630620326 shutdown_connections 2021-02-10T11:32:29.210 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.223 7f6c60129700 1 -- 172.21.2.112:0/2630620326 wait complete. 2021-02-10T11:32:29.218 INFO:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2021-02-10T11:32:29.219 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_metadata 8 2021-02-10T11:32:29.649 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.644 7f1d7171e700 1 Processor -- start 2021-02-10T11:32:29.649 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.644 7f1d7171e700 1 -- start start 2021-02-10T11:32:29.649 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.645 7f1d7171e700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f1d6c1b0870 0x7f1d6c0f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:29.650 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.645 7f1d7171e700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f1d6c0efe00 0x7f1d6c0f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:29.650 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.645 7f1d7171e700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d6c1b0c70 con 0x7f1d6c1b0470 2021-02-10T11:32:29.650 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.645 7f1d7171e700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d6c0e5ea0 con 0x7f1d6c1b0870 2021-02-10T11:32:29.650 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.645 7f1d7171e700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d6c0f0b50 con 0x7f1d6c0efe00 2021-02-10T11:32:29.650 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.645 7f1d6aabe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f1d6c1b0870 0x7f1d6c0f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:29.650 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.645 7f1d6b2bf700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f1d6c1b0470 0x7f1d6c0ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:48390/0 (socket says 172.21.2.112:48390) 2021-02-10T11:32:29.650 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.645 7f1d6b2bf700 1 -- 172.21.2.112:0/1413192807 learned_addr learned my addr 172.21.2.112:0/1413192807 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:32:29.650 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.645 7f1d6aabe700 1 -- 172.21.2.112:0/1413192807 >> v2:172.21.2.120:3300/0 conn(0x7f1d6c0efe00 msgr2=0x7f1d6c0f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:29.650 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.645 7f1d6aabe700 1 --2- 172.21.2.112:0/1413192807 >> v2:172.21.2.120:3300/0 conn(0x7f1d6c0efe00 0x7f1d6c0f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:29.650 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.645 7f1d6aabe700 1 -- 172.21.2.112:0/1413192807 >> v1:172.21.2.112:6789/0 conn(0x7f1d6c1b0470 legacy=0x7f1d6c0ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:29.650 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.645 7f1d6a2bd700 1 -- 172.21.2.112:0/1413192807 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4058143837 0 0) 0x7f1d5c001180 con 0x7f1d6c1b0470 2021-02-10T11:32:29.650 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.645 7f1d6aabe700 1 -- 172.21.2.112:0/1413192807 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d6c0f0820 con 0x7f1d6c1b0870 2021-02-10T11:32:29.651 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.646 7f1d6aabe700 1 --2- 172.21.2.112:0/1413192807 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1d6c1b0870 0x7f1d6c0f20e0 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 rx=0x7f1d60006650 tx=0x7f1d60009a00).ready entity=mon.0 client_cookie=c6f16bd10bf879f8 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:29.651 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.646 7f1d6a2bd700 1 -- 172.21.2.112:0/1413192807 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1d60009df0 con 0x7f1d6c1b0870 2021-02-10T11:32:29.651 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.646 7f1d6a2bd700 1 -- 172.21.2.112:0/1413192807 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d600009b0 con 0x7f1d6c1b0870 2021-02-10T11:32:29.651 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.646 7f1d6a2bd700 1 -- 172.21.2.112:0/1413192807 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1d6000c2b0 con 0x7f1d6c1b0870 2021-02-10T11:32:29.651 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.646 7f1d7171e700 1 -- 172.21.2.112:0/1413192807 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1d6c1b0870 msgr2=0x7f1d6c0f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:29.651 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.646 7f1d7171e700 1 --2- 172.21.2.112:0/1413192807 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1d6c1b0870 0x7f1d6c0f20e0 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 rx=0x7f1d60006650 tx=0x7f1d60009a00).stop 2021-02-10T11:32:29.651 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.646 7f1d7171e700 1 -- 172.21.2.112:0/1413192807 shutdown_connections 2021-02-10T11:32:29.651 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.646 7f1d7171e700 1 --2- 172.21.2.112:0/1413192807 >> v2:172.21.2.120:3300/0 conn(0x7f1d6c0efe00 0x7f1d6c0f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:29.651 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.646 7f1d7171e700 1 --2- 172.21.2.112:0/1413192807 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1d6c1b0870 0x7f1d6c0f20e0 unknown :-1 s=CLOSED pgs=1110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:29.651 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.646 7f1d7171e700 1 -- 172.21.2.112:0/1413192807 >> 172.21.2.112:0/1413192807 conn(0x7f1d6c0ce3e0 msgr2=0x7f1d6c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:29.651 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.647 7f1d7171e700 1 -- 172.21.2.112:0/1413192807 shutdown_connections 2021-02-10T11:32:29.651 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.647 7f1d7171e700 1 -- 172.21.2.112:0/1413192807 wait complete. 2021-02-10T11:32:29.651 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.661 7f1d7171e700 1 Processor -- start 2021-02-10T11:32:29.652 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.661 7f1d7171e700 1 -- start start 2021-02-10T11:32:29.652 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.661 7f1d7171e700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1d6c190920 0x7f1d6c08f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:29.652 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.661 7f1d7171e700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1d6c190d20 0x7f1d6c08fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:29.652 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.661 7f1d7171e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1d6c193270 0x7f1d6c193670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:29.652 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.661 7f1d7171e700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d6c090470 con 0x7f1d6c193270 2021-02-10T11:32:29.652 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.661 7f1d7171e700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d6c193c90 con 0x7f1d6c190d20 2021-02-10T11:32:29.652 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.661 7f1d7171e700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d6c193e70 con 0x7f1d6c190920 2021-02-10T11:32:29.652 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.662 7f1d6bac0700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1d6c193270 0x7f1d6c193670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:29.652 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.662 7f1d6bac0700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1d6c193270 0x7f1d6c193670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:48936/0 (socket says 172.21.2.112:48936) 2021-02-10T11:32:29.652 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.662 7f1d6bac0700 1 -- 172.21.2.112:0/1456990134 learned_addr learned my addr 172.21.2.112:0/1456990134 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:32:29.652 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.662 7f1d6bac0700 1 -- 172.21.2.112:0/1456990134 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1d6c190920 msgr2=0x7f1d6c08f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:29.652 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.662 7f1d6bac0700 1 --2- 172.21.2.112:0/1456990134 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1d6c190920 0x7f1d6c08f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:29.652 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.662 7f1d6bac0700 1 -- 172.21.2.112:0/1456990134 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1d6c190d20 msgr2=0x7f1d6c08fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:29.652 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.662 7f1d6bac0700 1 --2- 172.21.2.112:0/1456990134 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1d6c190d20 0x7f1d6c08fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:29.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.662 7f1d6bac0700 1 -- 172.21.2.112:0/1456990134 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d6c194090 con 0x7f1d6c193270 2021-02-10T11:32:29.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.662 7f1d6bac0700 1 --2- 172.21.2.112:0/1456990134 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1d6c193270 0x7f1d6c193670 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 rx=0x7f1d64009980 tx=0x7f1d6400bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:29.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.662 7f1d53fff700 1 -- 172.21.2.112:0/1456990134 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1d64005870 con 0x7f1d6c193270 2021-02-10T11:32:29.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.663 7f1d53fff700 1 -- 172.21.2.112:0/1456990134 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d64005b30 con 0x7f1d6c193270 2021-02-10T11:32:29.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.663 7f1d53fff700 1 -- 172.21.2.112:0/1456990134 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1d64006030 con 0x7f1d6c193270 2021-02-10T11:32:29.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.663 7f1d7171e700 1 -- 172.21.2.112:0/1456990134 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d6c194090 con 0x7f1d6c193270 2021-02-10T11:32:29.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.663 7f1d7171e700 1 -- 172.21.2.112:0/1456990134 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d6c1920b0 con 0x7f1d6c193270 2021-02-10T11:32:29.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.663 7f1d53fff700 1 -- 172.21.2.112:0/1456990134 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1d64006030 con 0x7f1d6c193270 2021-02-10T11:32:29.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.663 7f1d53fff700 1 --2- 172.21.2.112:0/1456990134 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1d4c0186c0 0x7f1d4c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:29.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.663 7f1d53fff700 1 -- 172.21.2.112:0/1456990134 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f1d640325a0 con 0x7f1d6c193270 2021-02-10T11:32:29.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.664 7f1d6b2bf700 1 --2- 172.21.2.112:0/1456990134 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1d4c0186c0 0x7f1d4c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:29.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.664 7f1d6b2bf700 1 --2- 172.21.2.112:0/1456990134 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1d4c0186c0 0x7f1d4c01abc0 crc :-1 s=READY pgs=562 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:29.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.695 7f1d7171e700 1 -- 172.21.2.112:0/1456990134 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d6c105af0 con 0x7f1d6c193270 2021-02-10T11:32:29.684 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.697 7f1d53fff700 1 -- 172.21.2.112:0/1456990134 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1d64005d10 con 0x7f1d6c193270 2021-02-10T11:32:29.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:29.825 7f1d7171e700 1 -- 172.21.2.112:0/1456990134 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"} v 0) v1 -- 0x7f1d6c0f0b50 con 0x7f1d6c193270 2021-02-10T11:32:30.240 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:30.253 7f1d53fff700 1 -- 172.21.2.112:0/1456990134 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"}]=0 pool 'cephfs_metadata' created v43) v1 ==== 133+0+0 (secure 0 0 0) 0x7f1d64006f20 con 0x7f1d6c193270 2021-02-10T11:32:30.279 INFO:teuthology.orchestra.run.gibba012.stderr:pool 'cephfs_metadata' created 2021-02-10T11:32:30.280 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:30.293 7f1d7171e700 1 -- 172.21.2.112:0/1456990134 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1d4c0186c0 msgr2=0x7f1d4c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:30.280 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:30.293 7f1d7171e700 1 --2- 172.21.2.112:0/1456990134 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1d4c0186c0 0x7f1d4c01abc0 crc :-1 s=READY pgs=562 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:30.280 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:30.293 7f1d7171e700 1 -- 172.21.2.112:0/1456990134 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1d6c193270 msgr2=0x7f1d6c193670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:30.281 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:30.293 7f1d7171e700 1 --2- 172.21.2.112:0/1456990134 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1d6c193270 0x7f1d6c193670 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 rx=0x7f1d64009980 tx=0x7f1d6400bcd0).stop 2021-02-10T11:32:30.281 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:30.293 7f1d7171e700 1 -- 172.21.2.112:0/1456990134 shutdown_connections 2021-02-10T11:32:30.281 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:30.293 7f1d7171e700 1 --2- 172.21.2.112:0/1456990134 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1d4c0186c0 0x7f1d4c01abc0 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:30.281 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:30.293 7f1d7171e700 1 --2- 172.21.2.112:0/1456990134 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1d6c190d20 0x7f1d6c08fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:30.281 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:30.293 7f1d7171e700 1 --2- 172.21.2.112:0/1456990134 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1d6c193270 0x7f1d6c193670 unknown :-1 s=CLOSED pgs=1111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:30.281 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:30.293 7f1d7171e700 1 --2- 172.21.2.112:0/1456990134 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1d6c190920 0x7f1d6c08f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:30.281 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:30.293 7f1d7171e700 1 -- 172.21.2.112:0/1456990134 >> 172.21.2.112:0/1456990134 conn(0x7f1d6c0bf4e0 msgr2=0x7f1d6c0be3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:30.281 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:30.294 7f1d7171e700 1 -- 172.21.2.112:0/1456990134 shutdown_connections 2021-02-10T11:32:30.282 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:30.294 7f1d7171e700 1 -- 172.21.2.112:0/1456990134 wait complete. 2021-02-10T11:32:30.289 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_data 8 2021-02-10T11:32:30.718 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:30.714 7ff228cbc700 1 Processor -- start 2021-02-10T11:32:30.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:30.714 7ff228cbc700 1 -- start start 2021-02-10T11:32:30.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:30.715 7ff228cbc700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff2241b0470 0x7ff2241b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:30.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:30.715 7ff228cbc700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7ff2240f20e0 0x7ff2240f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:30.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:30.715 7ff228cbc700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2240f0b50 con 0x7ff2240ef610 2021-02-10T11:32:30.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:30.715 7ff228cbc700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2240f0820 con 0x7ff2240f20e0 2021-02-10T11:32:30.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:30.715 7ff228cbc700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2240f24e0 con 0x7ff2241b0470 2021-02-10T11:32:30.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:30.715 7ff22305f700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7ff2240f20e0 0x7ff2240f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:30.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:30.715 7ff22205d700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7ff2240ef610 0x7ff2240efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:41964/0 (socket says 172.21.2.112:41964) 2021-02-10T11:32:30.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:30.715 7ff22305f700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7ff2240f20e0 0x7ff2240f0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3300/0 says I am v2:172.21.2.112:51226/0 (socket says 172.21.2.112:51226) 2021-02-10T11:32:30.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:30.715 7ff22205d700 1 -- 172.21.2.112:0/262682637 learned_addr learned my addr 172.21.2.112:0/262682637 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:32:30.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:30.715 7ff22285e700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff2241b0470 0x7ff2241b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:30.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:30.716 7ff22305f700 1 -- 172.21.2.112:0/262682637 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff2241b0470 msgr2=0x7ff2241b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:30.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:30.716 7ff22305f700 1 --2- 172.21.2.112:0/262682637 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff2241b0470 0x7ff2241b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:30.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:30.716 7ff22305f700 1 -- 172.21.2.112:0/262682637 >> v1:172.21.2.120:6789/0 conn(0x7ff2240ef610 legacy=0x7ff2240efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:30.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:30.716 7ff22305f700 1 -- 172.21.2.112:0/262682637 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2240e52e0 con 0x7ff2240f20e0 2021-02-10T11:32:30.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:30.716 7ff22285e700 1 --2- 172.21.2.112:0/262682637 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff2241b0470 0x7ff2241b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:32:30.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:30.716 7ff22305f700 1 --2- 172.21.2.112:0/262682637 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff2240f20e0 0x7ff2240f0200 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7ff21c006e50 tx=0x7ff21c009e70).ready entity=mon.1 client_cookie=597fc6c841a971dc server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:30.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:30.716 7ff22185c700 1 -- 172.21.2.112:0/262682637 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff21c006a20 con 0x7ff2240f20e0 2021-02-10T11:32:30.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:30.716 7ff22185c700 1 -- 172.21.2.112:0/262682637 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff21c006790 con 0x7ff2240f20e0 2021-02-10T11:32:30.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:30.716 7ff22185c700 1 -- 172.21.2.112:0/262682637 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff21c00b700 con 0x7ff2240f20e0 2021-02-10T11:32:30.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:30.717 7ff228cbc700 1 -- 172.21.2.112:0/262682637 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff2240f20e0 msgr2=0x7ff2240f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:30.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:30.717 7ff228cbc700 1 --2- 172.21.2.112:0/262682637 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff2240f20e0 0x7ff2240f0200 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7ff21c006e50 tx=0x7ff21c009e70).stop 2021-02-10T11:32:30.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:30.717 7ff228cbc700 1 -- 172.21.2.112:0/262682637 shutdown_connections 2021-02-10T11:32:30.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:30.717 7ff228cbc700 1 --2- 172.21.2.112:0/262682637 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff2240f20e0 0x7ff2240f0200 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:30.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:30.717 7ff228cbc700 1 --2- 172.21.2.112:0/262682637 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff2241b0470 0x7ff2241b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:30.721 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:30.717 7ff228cbc700 1 -- 172.21.2.112:0/262682637 >> 172.21.2.112:0/262682637 conn(0x7ff2240ce3e0 msgr2=0x7ff2240e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:30.721 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:30.717 7ff228cbc700 1 -- 172.21.2.112:0/262682637 shutdown_connections 2021-02-10T11:32:30.721 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:30.717 7ff228cbc700 1 -- 172.21.2.112:0/262682637 wait complete. 2021-02-10T11:32:30.721 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:30.732 7ff228cbc700 1 Processor -- start 2021-02-10T11:32:30.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:30.732 7ff228cbc700 1 -- start start 2021-02-10T11:32:30.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:30.732 7ff228cbc700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff2240b2eb0 0x7ff2240b2330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:30.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:30.732 7ff228cbc700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff2240b2950 0x7ff2240b3380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:30.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:30.732 7ff228cbc700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff2240b39a0 0x7ff22409d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:30.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:30.732 7ff228cbc700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff22409cdc0 con 0x7ff2240b2950 2021-02-10T11:32:30.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:30.732 7ff228cbc700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff22409d960 con 0x7ff2240b2eb0 2021-02-10T11:32:30.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:30.732 7ff228cbc700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff22409db80 con 0x7ff2240b39a0 2021-02-10T11:32:30.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:30.732 7ff22205d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff2240b2950 0x7ff2240b3380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:30.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:30.732 7ff22205d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff2240b2950 0x7ff2240b3380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:48948/0 (socket says 172.21.2.112:48948) 2021-02-10T11:32:30.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:30.732 7ff22205d700 1 -- 172.21.2.112:0/4090225317 learned_addr learned my addr 172.21.2.112:0/4090225317 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:32:30.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:30.733 7ff22285e700 1 --2- 172.21.2.112:0/4090225317 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff2240b2eb0 0x7ff2240b2330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:30.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:30.733 7ff22305f700 1 --2- 172.21.2.112:0/4090225317 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff2240b39a0 0x7ff22409d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:30.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:30.733 7ff22205d700 1 -- 172.21.2.112:0/4090225317 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff2240b39a0 msgr2=0x7ff22409d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:30.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:30.733 7ff22205d700 1 --2- 172.21.2.112:0/4090225317 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff2240b39a0 0x7ff22409d340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:30.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:30.733 7ff22205d700 1 -- 172.21.2.112:0/4090225317 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff2240b2eb0 msgr2=0x7ff2240b2330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:30.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:30.733 7ff22205d700 1 --2- 172.21.2.112:0/4090225317 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff2240b2eb0 0x7ff2240b2330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:30.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:30.733 7ff22205d700 1 -- 172.21.2.112:0/4090225317 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff22409dda0 con 0x7ff2240b2950 2021-02-10T11:32:30.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:30.733 7ff22205d700 1 --2- 172.21.2.112:0/4090225317 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff2240b2950 0x7ff2240b3380 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 rx=0x7ff20c00a5c0 tx=0x7ff20c00a960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:30.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:30.733 7ff21b7fe700 1 -- 172.21.2.112:0/4090225317 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff20c00ad50 con 0x7ff2240b2950 2021-02-10T11:32:30.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:30.733 7ff21b7fe700 1 -- 172.21.2.112:0/4090225317 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff20c0070d0 con 0x7ff2240b2950 2021-02-10T11:32:30.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:30.733 7ff21b7fe700 1 -- 172.21.2.112:0/4090225317 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff20c004510 con 0x7ff2240b2950 2021-02-10T11:32:30.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:30.733 7ff228cbc700 1 -- 172.21.2.112:0/4090225317 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff22409dda0 con 0x7ff2240b2950 2021-02-10T11:32:30.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:30.733 7ff228cbc700 1 -- 172.21.2.112:0/4090225317 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff22409dda0 con 0x7ff2240b2950 2021-02-10T11:32:30.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:30.734 7ff21b7fe700 1 -- 172.21.2.112:0/4090225317 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff20c004320 con 0x7ff2240b2950 2021-02-10T11:32:30.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:30.734 7ff21b7fe700 1 --2- 172.21.2.112:0/4090225317 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff2080186d0 0x7ff20801abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:30.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:30.734 7ff21b7fe700 1 -- 172.21.2.112:0/4090225317 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5107+0+0 (secure 0 0 0) 0x7ff20c0325e0 con 0x7ff2240b2950 2021-02-10T11:32:30.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:30.734 7ff22285e700 1 --2- 172.21.2.112:0/4090225317 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff2080186d0 0x7ff20801abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:30.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:30.735 7ff22285e700 1 --2- 172.21.2.112:0/4090225317 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff2080186d0 0x7ff20801abd0 crc :-1 s=READY pgs=563 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:30.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:30.765 7ff228cbc700 1 -- 172.21.2.112:0/4090225317 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff224105b30 con 0x7ff2240b2950 2021-02-10T11:32:30.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:30.767 7ff21b7fe700 1 -- 172.21.2.112:0/4090225317 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff20c000c10 con 0x7ff2240b2950 2021-02-10T11:32:30.884 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:30.897 7ff228cbc700 1 -- 172.21.2.112:0/4090225317 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"} v 0) v1 -- 0x7ff2240f0b50 con 0x7ff2240b2950 2021-02-10T11:32:31.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:31.259 7ff21b7fe700 1 -- 172.21.2.112:0/4090225317 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"}]=0 pool 'cephfs_data' created v44) v1 ==== 125+0+0 (secure 0 0 0) 0x7ff20c01f4f0 con 0x7ff2240b2950 2021-02-10T11:32:31.248 INFO:teuthology.orchestra.run.gibba012.stderr:pool 'cephfs_data' created 2021-02-10T11:32:31.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:31.263 7ff228cbc700 1 -- 172.21.2.112:0/4090225317 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff2080186d0 msgr2=0x7ff20801abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:31.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:31.263 7ff228cbc700 1 --2- 172.21.2.112:0/4090225317 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff2080186d0 0x7ff20801abd0 crc :-1 s=READY pgs=563 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:31.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:31.263 7ff228cbc700 1 -- 172.21.2.112:0/4090225317 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff2240b2950 msgr2=0x7ff2240b3380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:31.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:31.263 7ff228cbc700 1 --2- 172.21.2.112:0/4090225317 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff2240b2950 0x7ff2240b3380 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 rx=0x7ff20c00a5c0 tx=0x7ff20c00a960).stop 2021-02-10T11:32:31.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:31.263 7ff228cbc700 1 -- 172.21.2.112:0/4090225317 shutdown_connections 2021-02-10T11:32:31.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:31.263 7ff228cbc700 1 --2- 172.21.2.112:0/4090225317 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff2080186d0 0x7ff20801abd0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:31.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:31.263 7ff228cbc700 1 --2- 172.21.2.112:0/4090225317 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff2240b39a0 0x7ff22409d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:31.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:31.263 7ff228cbc700 1 --2- 172.21.2.112:0/4090225317 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff2240b2950 0x7ff2240b3380 unknown :-1 s=CLOSED pgs=1112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:31.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:31.263 7ff228cbc700 1 --2- 172.21.2.112:0/4090225317 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff2240b2eb0 0x7ff2240b2330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:31.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:31.263 7ff228cbc700 1 -- 172.21.2.112:0/4090225317 >> 172.21.2.112:0/4090225317 conn(0x7ff22409fc90 msgr2=0x7ff224094ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:31.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:31.263 7ff228cbc700 1 -- 172.21.2.112:0/4090225317 shutdown_connections 2021-02-10T11:32:31.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:31.263 7ff228cbc700 1 -- 172.21.2.112:0/4090225317 wait complete. 2021-02-10T11:32:31.259 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2021-02-10T11:32:31.689 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:31.685 7f831e2e7700 1 Processor -- start 2021-02-10T11:32:31.689 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:31.685 7f831e2e7700 1 -- start start 2021-02-10T11:32:31.689 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:31.686 7f831e2e7700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f83180d1870 0x7f83181b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:31.689 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:31.686 7f831e2e7700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f83181b0ae0 0x7f8318108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:31.690 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:31.686 7f831e2e7700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f83180d1c70 0x7f8318108f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:31.690 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:31.686 7f831e2e7700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83180fe1c0 con 0x7f83180d1870 2021-02-10T11:32:31.690 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:31.686 7f831e2e7700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8318109a60 con 0x7f83181b0ae0 2021-02-10T11:32:31.690 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:31.686 7f831e2e7700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8318088020 con 0x7f83180d1c70 2021-02-10T11:32:31.690 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:31.686 7f8317fff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f83180d1870 0x7f83181b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:31.690 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:31.686 7f8317fff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f83180d1870 0x7f83181b04c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:48958/0 (socket says 172.21.2.112:48958) 2021-02-10T11:32:31.690 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:31.686 7f8317fff700 1 -- 172.21.2.112:0/3400014107 learned_addr learned my addr 172.21.2.112:0/3400014107 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:32:31.690 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:31.686 7f8317fff700 1 -- 172.21.2.112:0/3400014107 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f83180d1c70 msgr2=0x7f8318108f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:31.690 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:31.686 7f8317fff700 1 --2- 172.21.2.112:0/3400014107 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f83180d1c70 0x7f8318108f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:31.690 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:31.686 7f8317fff700 1 -- 172.21.2.112:0/3400014107 >> v2:172.21.2.120:3300/0 conn(0x7f83181b0ae0 msgr2=0x7f8318108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:31.690 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:31.686 7f8317fff700 1 --2- 172.21.2.112:0/3400014107 >> v2:172.21.2.120:3300/0 conn(0x7f83181b0ae0 0x7f8318108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:31.690 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:31.686 7f8317fff700 1 -- 172.21.2.112:0/3400014107 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8318109580 con 0x7f83180d1870 2021-02-10T11:32:31.690 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:31.687 7f8317fff700 1 --2- 172.21.2.112:0/3400014107 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f83180d1870 0x7f83181b04c0 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 rx=0x7f8308006700 tx=0x7f8308009b80).ready entity=mon.0 client_cookie=d012f8fc5ae89469 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:31.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:31.687 7f8316ffd700 1 -- 172.21.2.112:0/3400014107 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f830800c0c0 con 0x7f83180d1870 2021-02-10T11:32:31.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:31.687 7f8316ffd700 1 -- 172.21.2.112:0/3400014107 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f830800c380 con 0x7f83180d1870 2021-02-10T11:32:31.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:31.687 7f8316ffd700 1 -- 172.21.2.112:0/3400014107 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f830800c8c0 con 0x7f83180d1870 2021-02-10T11:32:31.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:31.687 7f831e2e7700 1 -- 172.21.2.112:0/3400014107 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f83180d1870 msgr2=0x7f83181b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:31.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:31.687 7f831e2e7700 1 --2- 172.21.2.112:0/3400014107 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f83180d1870 0x7f83181b04c0 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 rx=0x7f8308006700 tx=0x7f8308009b80).stop 2021-02-10T11:32:31.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:31.688 7f831e2e7700 1 -- 172.21.2.112:0/3400014107 shutdown_connections 2021-02-10T11:32:31.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:31.688 7f831e2e7700 1 --2- 172.21.2.112:0/3400014107 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f83180d1c70 0x7f8318108f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:31.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:31.688 7f831e2e7700 1 --2- 172.21.2.112:0/3400014107 >> v2:172.21.2.120:3300/0 conn(0x7f83181b0ae0 0x7f8318108940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:31.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:31.688 7f831e2e7700 1 --2- 172.21.2.112:0/3400014107 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f83180d1870 0x7f83181b04c0 unknown :-1 s=CLOSED pgs=1113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:31.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:31.688 7f831e2e7700 1 -- 172.21.2.112:0/3400014107 >> 172.21.2.112:0/3400014107 conn(0x7f83180ce420 msgr2=0x7f83180d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:31.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:31.688 7f831e2e7700 1 -- 172.21.2.112:0/3400014107 shutdown_connections 2021-02-10T11:32:31.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:31.688 7f831e2e7700 1 -- 172.21.2.112:0/3400014107 wait complete. 2021-02-10T11:32:31.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:31.702 7f831e2e7700 1 Processor -- start 2021-02-10T11:32:31.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:31.702 7f831e2e7700 1 -- start start 2021-02-10T11:32:31.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:31.703 7f831e2e7700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8318093fb0 0x7f83181a5670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:31.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:31.703 7f831e2e7700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f83181a1c50 0x7f83181a2050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:31.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:31.703 7f831e2e7700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f831819bbd0 0x7f831819bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:31.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:31.703 7f831e2e7700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83181a2740 con 0x7f831819bbd0 2021-02-10T11:32:31.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:31.703 7f831e2e7700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83180a76c0 con 0x7f8318093fb0 2021-02-10T11:32:31.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:31.703 7f831e2e7700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83181a5c90 con 0x7f83181a1c50 2021-02-10T11:32:31.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:31.703 7f831c904700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f831819bbd0 0x7f831819bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:31.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:31.703 7f831c904700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f831819bbd0 0x7f831819bfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:48964/0 (socket says 172.21.2.112:48964) 2021-02-10T11:32:31.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:31.703 7f831c904700 1 -- 172.21.2.112:0/2692431776 learned_addr learned my addr 172.21.2.112:0/2692431776 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:32:31.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:31.703 7f831c904700 1 -- 172.21.2.112:0/2692431776 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f83181a1c50 msgr2=0x7f83181a2050 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:32:31.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:31.703 7f831c904700 1 --2- 172.21.2.112:0/2692431776 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f83181a1c50 0x7f83181a2050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:31.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:31.703 7f831c904700 1 -- 172.21.2.112:0/2692431776 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8318093fb0 msgr2=0x7f83181a5670 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:32:31.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:31.703 7f831c904700 1 --2- 172.21.2.112:0/2692431776 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8318093fb0 0x7f83181a5670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:31.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:31.703 7f831c904700 1 -- 172.21.2.112:0/2692431776 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f831819c5f0 con 0x7f831819bbd0 2021-02-10T11:32:31.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:31.704 7f831c904700 1 --2- 172.21.2.112:0/2692431776 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f831819bbd0 0x7f831819bfd0 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 rx=0x7f8310009980 tx=0x7f831000bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:31.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:31.704 7f8314ff9700 1 -- 172.21.2.112:0/2692431776 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8310005af0 con 0x7f831819bbd0 2021-02-10T11:32:31.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:31.704 7f8314ff9700 1 -- 172.21.2.112:0/2692431776 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8310005d70 con 0x7f831819bbd0 2021-02-10T11:32:31.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:31.704 7f8314ff9700 1 -- 172.21.2.112:0/2692431776 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f83100062b0 con 0x7f831819bbd0 2021-02-10T11:32:31.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:31.704 7f831e2e7700 1 -- 172.21.2.112:0/2692431776 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f831819c5f0 con 0x7f831819bbd0 2021-02-10T11:32:31.695 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:31.704 7f831e2e7700 1 -- 172.21.2.112:0/2692431776 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f831819cb80 con 0x7f831819bbd0 2021-02-10T11:32:31.695 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:31.705 7f8314ff9700 1 -- 172.21.2.112:0/2692431776 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8310006030 con 0x7f831819bbd0 2021-02-10T11:32:31.695 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:31.705 7f8314ff9700 1 --2- 172.21.2.112:0/2692431776 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f82f80186c0 0x7f82f801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:31.695 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:31.705 7f8314ff9700 1 -- 172.21.2.112:0/2692431776 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5457+0+0 (secure 0 0 0) 0x7f8310032890 con 0x7f831819bbd0 2021-02-10T11:32:31.695 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:31.705 7f8317fff700 1 --2- 172.21.2.112:0/2692431776 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f82f80186c0 0x7f82f801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:31.695 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:31.706 7f8317fff700 1 --2- 172.21.2.112:0/2692431776 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f82f80186c0 0x7f82f801abc0 crc :-1 s=READY pgs=566 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:31.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:31.735 7f831e2e7700 1 -- 172.21.2.112:0/2692431776 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83180fe1c0 con 0x7f831819bbd0 2021-02-10T11:32:31.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:31.738 7f8314ff9700 1 -- 172.21.2.112:0/2692431776 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8310019680 con 0x7f831819bbd0 2021-02-10T11:32:31.855 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:31.868 7f831e2e7700 1 -- 172.21.2.112:0/2692431776 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"} v 0) v1 -- 0x7f83180fe1c0 con 0x7f831819bbd0 2021-02-10T11:32:31.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:31.871 7f8314ff9700 1 -- 172.21.2.112:0/2692431776 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"}]=0 new fs with metadata pool 8 and data pool 9 v27) v1 ==== 187+0+0 (secure 0 0 0) 0x7f8310019680 con 0x7f831819bbd0 2021-02-10T11:32:31.862 INFO:teuthology.orchestra.run.gibba012.stderr:new fs with metadata pool 8 and data pool 9 2021-02-10T11:32:31.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:31.877 7f831e2e7700 1 -- 172.21.2.112:0/2692431776 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f82f80186c0 msgr2=0x7f82f801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:31.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:31.877 7f831e2e7700 1 --2- 172.21.2.112:0/2692431776 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f82f80186c0 0x7f82f801abc0 crc :-1 s=READY pgs=566 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:31.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:31.877 7f831e2e7700 1 -- 172.21.2.112:0/2692431776 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f831819bbd0 msgr2=0x7f831819bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:31.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:31.877 7f831e2e7700 1 --2- 172.21.2.112:0/2692431776 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f831819bbd0 0x7f831819bfd0 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 rx=0x7f8310009980 tx=0x7f831000bcd0).stop 2021-02-10T11:32:31.865 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:31.877 7f831e2e7700 1 -- 172.21.2.112:0/2692431776 shutdown_connections 2021-02-10T11:32:31.865 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:31.877 7f831e2e7700 1 --2- 172.21.2.112:0/2692431776 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f82f80186c0 0x7f82f801abc0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:31.865 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:31.877 7f831e2e7700 1 --2- 172.21.2.112:0/2692431776 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f831819bbd0 0x7f831819bfd0 unknown :-1 s=CLOSED pgs=1114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:31.865 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:31.877 7f831e2e7700 1 --2- 172.21.2.112:0/2692431776 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f83181a1c50 0x7f83181a2050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:31.865 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:31.877 7f831e2e7700 1 --2- 172.21.2.112:0/2692431776 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8318093fb0 0x7f83181a5670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:31.865 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:31.877 7f831e2e7700 1 -- 172.21.2.112:0/2692431776 >> 172.21.2.112:0/2692431776 conn(0x7f8318094ea0 msgr2=0x7f831819acf0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:31.865 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:31.878 7f831e2e7700 1 -- 172.21.2.112:0/2692431776 shutdown_connections 2021-02-10T11:32:31.865 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:31.878 7f831e2e7700 1 -- 172.21.2.112:0/2692431776 wait complete. 2021-02-10T11:32:31.873 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:32:32.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.293 7fc7637c1700 1 Processor -- start 2021-02-10T11:32:32.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.293 7fc7637c1700 1 -- start start 2021-02-10T11:32:32.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.294 7fc7637c1700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fc75c0e5470 0x7fc75c07e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:32.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.294 7fc7637c1700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc75c07f300 0x7fc75c07f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:32.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.294 7fc7637c1700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc75c07ed80 con 0x7fc75c0e5070 2021-02-10T11:32:32.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.294 7fc7637c1700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc75c1b4010 con 0x7fc75c0e5470 2021-02-10T11:32:32.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.294 7fc7637c1700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc75c0d1790 con 0x7fc75c07f300 2021-02-10T11:32:32.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.294 7fc7615dd700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fc75c0e5070 0x7fc75c1b3820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:48432/0 (socket says 172.21.2.112:48432) 2021-02-10T11:32:32.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.294 7fc7615dd700 1 -- 172.21.2.112:0/1790698829 learned_addr learned my addr 172.21.2.112:0/1790698829 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:32:32.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.294 7fc74ffff700 1 -- 172.21.2.112:0/1790698829 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2259490970 0 0) 0x7fc7500009b0 con 0x7fc75c0e5070 2021-02-10T11:32:32.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.294 7fc74ffff700 1 -- 172.21.2.112:0/1790698829 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc748002fe0 con 0x7fc75c0e5070 2021-02-10T11:32:32.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.295 7fc760ddc700 1 --2- 172.21.2.112:0/1790698829 >> v2:172.21.2.120:3300/0 conn(0x7fc75c0e5470 0x7fc75c07e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:32.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.295 7fc74ffff700 1 -- 172.21.2.112:0/1790698829 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2000887561 0 0) 0x7fc7500010b0 con 0x7fc75c0e5070 2021-02-10T11:32:32.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.295 7fc74ffff700 1 -- 172.21.2.112:0/1790698829 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc75c07f300 msgr2=0x7fc75c07f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:32.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.295 7fc74ffff700 1 --2- 172.21.2.112:0/1790698829 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc75c07f300 0x7fc75c07f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:32.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.295 7fc74ffff700 1 -- 172.21.2.112:0/1790698829 >> v2:172.21.2.120:3300/0 conn(0x7fc75c0e5470 msgr2=0x7fc75c07e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:32.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.295 7fc74ffff700 1 --2- 172.21.2.112:0/1790698829 >> v2:172.21.2.120:3300/0 conn(0x7fc75c0e5470 0x7fc75c07e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:32.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.295 7fc74ffff700 1 -- 172.21.2.112:0/1790698829 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc75c0d19b0 con 0x7fc75c0e5070 2021-02-10T11:32:32.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.295 7fc74ffff700 1 -- 172.21.2.112:0/1790698829 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7fc7500015b0 con 0x7fc75c0e5070 2021-02-10T11:32:32.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.295 7fc74ffff700 1 -- 172.21.2.112:0/1790698829 >> v1:172.21.2.112:6789/0 conn(0x7fc75c0e5070 legacy=0x7fc75c1b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:32.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.295 7fc74ffff700 1 --2- 172.21.2.112:0/1790698829 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc748003a60 0x7fc748005f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:32.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.295 7fc74ffff700 1 --2- 172.21.2.112:0/1790698829 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc748006800 0x7fc748008d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:32.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.295 7fc74ffff700 1 --2- 172.21.2.112:0/1790698829 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc748009620 0x7fc74800bb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:32.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.295 7fc74ffff700 1 -- 172.21.2.112:0/1790698829 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc74800c360 con 0x7fc748006800 2021-02-10T11:32:32.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.295 7fc74ffff700 1 -- 172.21.2.112:0/1790698829 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc74800c580 con 0x7fc748003a60 2021-02-10T11:32:32.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.295 7fc74ffff700 1 -- 172.21.2.112:0/1790698829 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc74800c7a0 con 0x7fc748009620 2021-02-10T11:32:32.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.295 7fc760ddc700 1 --2- 172.21.2.112:0/1790698829 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc748006800 0x7fc748008d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:32.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.295 7fc7615dd700 1 --2- 172.21.2.112:0/1790698829 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc748003a60 0x7fc748005f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:32.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.295 7fc761dde700 1 --2- 172.21.2.112:0/1790698829 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc748009620 0x7fc74800bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:32.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.295 7fc760ddc700 1 -- 172.21.2.112:0/1790698829 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc748009620 msgr2=0x7fc74800bb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:32.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.295 7fc760ddc700 1 --2- 172.21.2.112:0/1790698829 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc748009620 0x7fc74800bb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:32.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.295 7fc760ddc700 1 -- 172.21.2.112:0/1790698829 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc748003a60 msgr2=0x7fc748005f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:32.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.295 7fc760ddc700 1 --2- 172.21.2.112:0/1790698829 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc748003a60 0x7fc748005f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:32.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.295 7fc760ddc700 1 -- 172.21.2.112:0/1790698829 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc74800ca40 con 0x7fc748006800 2021-02-10T11:32:32.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.295 7fc760ddc700 1 --2- 172.21.2.112:0/1790698829 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc748006800 0x7fc748008d20 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 rx=0x7fc75804d810 tx=0x7fc75804dbb0).ready entity=mon.0 client_cookie=44553a4bdd78c903 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:32.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.295 7fc74ffff700 1 -- 172.21.2.112:0/1790698829 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc758006840 con 0x7fc748006800 2021-02-10T11:32:32.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.295 7fc74ffff700 1 -- 172.21.2.112:0/1790698829 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7580501c0 con 0x7fc748006800 2021-02-10T11:32:32.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.295 7fc74ffff700 1 -- 172.21.2.112:0/1790698829 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc758050610 con 0x7fc748006800 2021-02-10T11:32:32.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.295 7fc7637c1700 1 -- 172.21.2.112:0/1790698829 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc748006800 msgr2=0x7fc748008d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:32.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.295 7fc7637c1700 1 --2- 172.21.2.112:0/1790698829 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc748006800 0x7fc748008d20 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 rx=0x7fc75804d810 tx=0x7fc75804dbb0).stop 2021-02-10T11:32:32.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.295 7fc761dde700 1 -- 172.21.2.112:0/1790698829 reap_dead start 2021-02-10T11:32:32.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.296 7fc7637c1700 1 -- 172.21.2.112:0/1790698829 shutdown_connections 2021-02-10T11:32:32.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.296 7fc7637c1700 1 -- 172.21.2.112:0/1790698829 >> 172.21.2.112:0/1790698829 conn(0x7fc75c0ce690 msgr2=0x7fc75c0e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:32.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.296 7fc7637c1700 1 -- 172.21.2.112:0/1790698829 shutdown_connections 2021-02-10T11:32:32.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.296 7fc7637c1700 1 -- 172.21.2.112:0/1790698829 wait complete. 2021-02-10T11:32:32.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.311 7fc7637c1700 1 Processor -- start 2021-02-10T11:32:32.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.311 7fc7637c1700 1 -- start start 2021-02-10T11:32:32.303 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.311 7fc7637c1700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc75c085a30 0x7fc75c0b8170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:32.303 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.311 7fc7637c1700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc75c198a40 0x7fc75c0b8790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:32.303 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.311 7fc7637c1700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc75c18a220 0x7fc75c18a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:32.303 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.311 7fc7637c1700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc75c189ca0 con 0x7fc75c198a40 2021-02-10T11:32:32.303 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.311 7fc7637c1700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc75c0b8db0 con 0x7fc75c18a220 2021-02-10T11:32:32.303 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.311 7fc7637c1700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc75c0b8fd0 con 0x7fc75c085a30 2021-02-10T11:32:32.303 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.311 7fc760ddc700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc75c198a40 0x7fc75c0b8790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:32.303 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.311 7fc760ddc700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc75c198a40 0x7fc75c0b8790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:48982/0 (socket says 172.21.2.112:48982) 2021-02-10T11:32:32.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.311 7fc760ddc700 1 -- 172.21.2.112:0/2902488348 learned_addr learned my addr 172.21.2.112:0/2902488348 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:32:32.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.312 7fc760ddc700 1 -- 172.21.2.112:0/2902488348 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc75c085a30 msgr2=0x7fc75c0b8170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:32.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.312 7fc760ddc700 1 --2- 172.21.2.112:0/2902488348 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc75c085a30 0x7fc75c0b8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:32.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.312 7fc760ddc700 1 -- 172.21.2.112:0/2902488348 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc75c18a220 msgr2=0x7fc75c18a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:32.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.312 7fc760ddc700 1 --2- 172.21.2.112:0/2902488348 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc75c18a220 0x7fc75c18a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:32.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.312 7fc760ddc700 1 -- 172.21.2.112:0/2902488348 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc75c18ac40 con 0x7fc75c198a40 2021-02-10T11:32:32.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.312 7fc760ddc700 1 --2- 172.21.2.112:0/2902488348 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc75c198a40 0x7fc75c0b8790 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 rx=0x7fc758050590 tx=0x7fc758050930).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:32.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.312 7fc74dffb700 1 -- 172.21.2.112:0/2902488348 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc758050d20 con 0x7fc75c198a40 2021-02-10T11:32:32.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.312 7fc74dffb700 1 -- 172.21.2.112:0/2902488348 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7580020d0 con 0x7fc75c198a40 2021-02-10T11:32:32.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.312 7fc74dffb700 1 -- 172.21.2.112:0/2902488348 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc7580025f0 con 0x7fc75c198a40 2021-02-10T11:32:32.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.312 7fc7637c1700 1 -- 172.21.2.112:0/2902488348 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc75c18ac40 con 0x7fc75c198a40 2021-02-10T11:32:32.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.312 7fc7637c1700 1 -- 172.21.2.112:0/2902488348 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc75c18d840 con 0x7fc75c198a40 2021-02-10T11:32:32.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.313 7fc74dffb700 1 -- 172.21.2.112:0/2902488348 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc758000c90 con 0x7fc75c198a40 2021-02-10T11:32:32.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.313 7fc74dffb700 1 --2- 172.21.2.112:0/2902488348 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc73c0186c0 0x7fc73c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:32.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.313 7fc74dffb700 1 -- 172.21.2.112:0/2902488348 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fc758076580 con 0x7fc75c198a40 2021-02-10T11:32:32.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.313 7fc7615dd700 1 --2- 172.21.2.112:0/2902488348 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc73c0186c0 0x7fc73c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:32.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.314 7fc7615dd700 1 --2- 172.21.2.112:0/2902488348 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc73c0186c0 0x7fc73c01abc0 crc :-1 s=READY pgs=567 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:32.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.344 7fc7637c1700 1 -- 172.21.2.112:0/2902488348 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc75c084b20 con 0x7fc75c198a40 2021-02-10T11:32:32.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.346 7fc74dffb700 1 -- 172.21.2.112:0/2902488348 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc758006c60 con 0x7fc75c198a40 2021-02-10T11:32:32.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.472 7fc7637c1700 1 -- 172.21.2.112:0/2902488348 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc75c07ed80 con 0x7fc75c198a40 2021-02-10T11:32:32.460 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.473 7fc74dffb700 1 -- 172.21.2.112:0/2902488348 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v45) v1 ==== 74+0+14006 (secure 0 0 0) 0x7fc758000c90 con 0x7fc75c198a40 2021-02-10T11:32:32.461 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:32:32.461 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":45,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:32:31.869603","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:32:29.843136","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":9,"pool_name":"cephfs_data","create_time":"2021-02-10 11:32:30.914946","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:32:32.461 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.474 7fc7637c1700 1 -- 172.21.2.112:0/2902488348 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc73c0186c0 msgr2=0x7fc73c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:32.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.474 7fc7637c1700 1 --2- 172.21.2.112:0/2902488348 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc73c0186c0 0x7fc73c01abc0 crc :-1 s=READY pgs=567 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:32.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.474 7fc7637c1700 1 -- 172.21.2.112:0/2902488348 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc75c198a40 msgr2=0x7fc75c0b8790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:32.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.475 7fc7637c1700 1 --2- 172.21.2.112:0/2902488348 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc75c198a40 0x7fc75c0b8790 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 rx=0x7fc758050590 tx=0x7fc758050930).stop 2021-02-10T11:32:32.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.475 7fc7637c1700 1 -- 172.21.2.112:0/2902488348 shutdown_connections 2021-02-10T11:32:32.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.475 7fc7637c1700 1 --2- 172.21.2.112:0/2902488348 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc73c0186c0 0x7fc73c01abc0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:32.463 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.475 7fc7637c1700 1 --2- 172.21.2.112:0/2902488348 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc75c18a220 0x7fc75c18a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:32.463 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.475 7fc7637c1700 1 --2- 172.21.2.112:0/2902488348 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc75c198a40 0x7fc75c0b8790 unknown :-1 s=CLOSED pgs=1117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:32.463 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.475 7fc7637c1700 1 --2- 172.21.2.112:0/2902488348 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc75c085a30 0x7fc75c0b8170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:32.463 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.475 7fc7637c1700 1 -- 172.21.2.112:0/2902488348 >> 172.21.2.112:0/2902488348 conn(0x7fc75c0855a0 msgr2=0x7fc75c1996d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:32.463 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.475 7fc7637c1700 1 -- 172.21.2.112:0/2902488348 shutdown_connections 2021-02-10T11:32:32.463 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.475 7fc7637c1700 1 -- 172.21.2.112:0/2902488348 wait complete. 2021-02-10T11:32:32.471 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:32:32.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.892 7ff75f23a700 1 Processor -- start 2021-02-10T11:32:32.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.893 7ff75f23a700 1 -- start start 2021-02-10T11:32:32.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.893 7ff75f23a700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7ff7580e5070 0x7ff7581b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:32.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.893 7ff75f23a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff7580e5470 0x7ff758080310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:32.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.893 7ff75f23a700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff75807ed80 con 0x7ff7581b3e40 2021-02-10T11:32:32.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.893 7ff75f23a700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7580d1790 con 0x7ff7580e5070 2021-02-10T11:32:32.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.893 7ff75f23a700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7580d19b0 con 0x7ff7580e5470 2021-02-10T11:32:32.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.893 7ff75d056700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7ff7580e5070 0x7ff7581b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:32.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.893 7ff75d056700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7ff7580e5070 0x7ff7581b3820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:48990/0 (socket says 172.21.2.112:48990) 2021-02-10T11:32:32.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.893 7ff75d056700 1 -- 172.21.2.112:0/89829825 learned_addr learned my addr 172.21.2.112:0/89829825 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:32:32.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.893 7ff75d857700 1 --2- 172.21.2.112:0/89829825 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff7580e5470 0x7ff758080310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:32.897 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.893 7ff75d056700 1 -- 172.21.2.112:0/89829825 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff7580e5470 msgr2=0x7ff758080310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:32.897 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.894 7ff75d056700 1 --2- 172.21.2.112:0/89829825 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff7580e5470 0x7ff758080310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:32.897 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.894 7ff75d056700 1 -- 172.21.2.112:0/89829825 >> v1:172.21.2.120:6789/0 conn(0x7ff7581b3e40 legacy=0x7ff75807d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:32.897 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.894 7ff75d056700 1 -- 172.21.2.112:0/89829825 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7580d1bd0 con 0x7ff7580e5070 2021-02-10T11:32:32.897 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.894 7ff75d056700 1 --2- 172.21.2.112:0/89829825 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff7580e5070 0x7ff7581b3820 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 rx=0x7ff754006700 tx=0x7ff754009b80).ready entity=mon.0 client_cookie=95d7f14ad0e785ed server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:32.897 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.894 7ff74ffff700 1 -- 172.21.2.112:0/89829825 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff75400c0c0 con 0x7ff7580e5070 2021-02-10T11:32:32.897 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.894 7ff74ffff700 1 -- 172.21.2.112:0/89829825 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff75400c440 con 0x7ff7580e5070 2021-02-10T11:32:32.897 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.894 7ff74ffff700 1 -- 172.21.2.112:0/89829825 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff75400c920 con 0x7ff7580e5070 2021-02-10T11:32:32.897 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.894 7ff75f23a700 1 -- 172.21.2.112:0/89829825 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff7580e5070 msgr2=0x7ff7581b3820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:32.897 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.894 7ff75f23a700 1 --2- 172.21.2.112:0/89829825 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff7580e5070 0x7ff7581b3820 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 rx=0x7ff754006700 tx=0x7ff754009b80).stop 2021-02-10T11:32:32.897 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.894 7ff75f23a700 1 -- 172.21.2.112:0/89829825 shutdown_connections 2021-02-10T11:32:32.897 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.894 7ff75f23a700 1 --2- 172.21.2.112:0/89829825 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff7580e5470 0x7ff758080310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:32.897 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.894 7ff75f23a700 1 --2- 172.21.2.112:0/89829825 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff7580e5070 0x7ff7581b3820 unknown :-1 s=CLOSED pgs=1118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:32.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.895 7ff75f23a700 1 -- 172.21.2.112:0/89829825 >> 172.21.2.112:0/89829825 conn(0x7ff7580ce690 msgr2=0x7ff7580e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:32.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.895 7ff75f23a700 1 -- 172.21.2.112:0/89829825 shutdown_connections 2021-02-10T11:32:32.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.895 7ff75f23a700 1 -- 172.21.2.112:0/89829825 wait complete. 2021-02-10T11:32:32.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.909 7ff75f23a700 1 Processor -- start 2021-02-10T11:32:32.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.909 7ff75f23a700 1 -- start start 2021-02-10T11:32:32.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.909 7ff75f23a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff758191bb0 0x7ff7580a2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:32.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.909 7ff75f23a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff7580a29a0 0x7ff7580a2da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:32.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.909 7ff75f23a700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff7580a33c0 0x7ff7580a03d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:32.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.909 7ff75f23a700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7580adf90 con 0x7ff7580a29a0 2021-02-10T11:32:32.900 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.909 7ff75f23a700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7580a1e00 con 0x7ff7580a33c0 2021-02-10T11:32:32.900 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.909 7ff75f23a700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7580a09f0 con 0x7ff758191bb0 2021-02-10T11:32:32.900 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.910 7ff75c855700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff7580a29a0 0x7ff7580a2da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:32.900 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.910 7ff75c855700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff7580a29a0 0x7ff7580a2da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:48998/0 (socket says 172.21.2.112:48998) 2021-02-10T11:32:32.900 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.910 7ff75c855700 1 -- 172.21.2.112:0/752988956 learned_addr learned my addr 172.21.2.112:0/752988956 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:32:32.900 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.910 7ff75d056700 1 --2- 172.21.2.112:0/752988956 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff758191bb0 0x7ff7580a2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:32.900 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.910 7ff75d857700 1 --2- 172.21.2.112:0/752988956 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff7580a33c0 0x7ff7580a03d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:32.900 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.910 7ff75c855700 1 -- 172.21.2.112:0/752988956 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff758191bb0 msgr2=0x7ff7580a2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:32.900 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.910 7ff75c855700 1 --2- 172.21.2.112:0/752988956 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff758191bb0 0x7ff7580a2380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:32.900 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.910 7ff75c855700 1 -- 172.21.2.112:0/752988956 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff7580a33c0 msgr2=0x7ff7580a03d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:32.900 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.910 7ff75c855700 1 --2- 172.21.2.112:0/752988956 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff7580a33c0 0x7ff7580a03d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:32.900 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.910 7ff75c855700 1 -- 172.21.2.112:0/752988956 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7580a0bd0 con 0x7ff7580a29a0 2021-02-10T11:32:32.901 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.910 7ff75c855700 1 --2- 172.21.2.112:0/752988956 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff7580a29a0 0x7ff7580a2da0 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 rx=0x7ff748006ed0 tx=0x7ff74800b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:32.901 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.910 7ff74dffb700 1 -- 172.21.2.112:0/752988956 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff748006020 con 0x7ff7580a29a0 2021-02-10T11:32:32.901 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.911 7ff74dffb700 1 -- 172.21.2.112:0/752988956 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7480009b0 con 0x7ff7580a29a0 2021-02-10T11:32:32.901 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.911 7ff74dffb700 1 -- 172.21.2.112:0/752988956 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff7480054b0 con 0x7ff7580a29a0 2021-02-10T11:32:32.901 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.911 7ff75f23a700 1 -- 172.21.2.112:0/752988956 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7580a0bd0 con 0x7ff7580a29a0 2021-02-10T11:32:32.901 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.911 7ff75f23a700 1 -- 172.21.2.112:0/752988956 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7580a1160 con 0x7ff7580a29a0 2021-02-10T11:32:32.901 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.911 7ff74dffb700 1 -- 172.21.2.112:0/752988956 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff748006020 con 0x7ff7580a29a0 2021-02-10T11:32:32.901 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.911 7ff74dffb700 1 --2- 172.21.2.112:0/752988956 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff7380186c0 0x7ff73801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:32.901 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.911 7ff74dffb700 1 -- 172.21.2.112:0/752988956 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5573+0+0 (secure 0 0 0) 0x7ff7480323f0 con 0x7ff7580a29a0 2021-02-10T11:32:32.901 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.911 7ff75d056700 1 --2- 172.21.2.112:0/752988956 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff7380186c0 0x7ff73801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:32.901 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.912 7ff75d056700 1 --2- 172.21.2.112:0/752988956 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff7380186c0 0x7ff73801abc0 crc :-1 s=READY pgs=568 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:32.930 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.943 7ff75f23a700 1 -- 172.21.2.112:0/752988956 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7580a1160 con 0x7ff7580a29a0 2021-02-10T11:32:32.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:32.945 7ff74dffb700 1 -- 172.21.2.112:0/752988956 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff748030d20 con 0x7ff7580a29a0 2021-02-10T11:32:33.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:33.070 7ff75f23a700 1 -- 172.21.2.112:0/752988956 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff7580a1160 con 0x7ff7580a29a0 2021-02-10T11:32:33.058 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:33.071 7ff74dffb700 1 -- 172.21.2.112:0/752988956 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v46) v1 ==== 74+0+13993 (secure 0 0 0) 0x7ff748006000 con 0x7ff7580a29a0 2021-02-10T11:32:33.058 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:32:33.058 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":46,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:32:32.871643","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:32:29.843136","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":9,"pool_name":"cephfs_data","create_time":"2021-02-10 11:32:30.914946","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:32:33.059 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:33.072 7ff75f23a700 1 -- 172.21.2.112:0/752988956 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff7380186c0 msgr2=0x7ff73801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:33.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:33.072 7ff75f23a700 1 --2- 172.21.2.112:0/752988956 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff7380186c0 0x7ff73801abc0 crc :-1 s=READY pgs=568 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:33.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:33.072 7ff75f23a700 1 -- 172.21.2.112:0/752988956 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff7580a29a0 msgr2=0x7ff7580a2da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:33.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:33.072 7ff75f23a700 1 --2- 172.21.2.112:0/752988956 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff7580a29a0 0x7ff7580a2da0 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 rx=0x7ff748006ed0 tx=0x7ff74800b960).stop 2021-02-10T11:32:33.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:33.073 7ff75f23a700 1 -- 172.21.2.112:0/752988956 shutdown_connections 2021-02-10T11:32:33.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:33.073 7ff75f23a700 1 --2- 172.21.2.112:0/752988956 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff7380186c0 0x7ff73801abc0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:33.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:33.073 7ff75f23a700 1 --2- 172.21.2.112:0/752988956 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff7580a33c0 0x7ff7580a03d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:33.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:33.073 7ff75f23a700 1 --2- 172.21.2.112:0/752988956 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff7580a29a0 0x7ff7580a2da0 unknown :-1 s=CLOSED pgs=1119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:33.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:33.073 7ff75f23a700 1 --2- 172.21.2.112:0/752988956 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff758191bb0 0x7ff7580a2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:33.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:33.073 7ff75f23a700 1 -- 172.21.2.112:0/752988956 >> 172.21.2.112:0/752988956 conn(0x7ff7581a31d0 msgr2=0x7ff75808b830 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:33.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:33.073 7ff75f23a700 1 -- 172.21.2.112:0/752988956 shutdown_connections 2021-02-10T11:32:33.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:33.073 7ff75f23a700 1 -- 172.21.2.112:0/752988956 wait complete. 2021-02-10T11:32:33.069 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2021-02-10T11:32:33.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:33.494 7f62584b9700 1 Processor -- start 2021-02-10T11:32:33.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:33.494 7f62584b9700 1 -- start start 2021-02-10T11:32:33.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:33.494 7f62584b9700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f62501b0470 0x7f62501b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:33.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:33.494 7f62584b9700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f62500ef610 0x7f62500efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:33.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:33.494 7f62584b9700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f62500f0030 0x7f62500f0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:33.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:33.494 7f62584b9700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62500e5ea0 con 0x7f62500ef610 2021-02-10T11:32:33.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:33.494 7f62584b9700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62500f0b50 con 0x7f62501b0470 2021-02-10T11:32:33.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:33.494 7f62584b9700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62500e52e0 con 0x7f62500f0030 2021-02-10T11:32:33.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:33.495 7f6255ad4700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f62500ef610 0x7f62500efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:33.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:33.495 7f6255ad4700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f62500ef610 0x7f62500efa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:49004/0 (socket says 172.21.2.112:49004) 2021-02-10T11:32:33.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:33.495 7f6255ad4700 1 -- 172.21.2.112:0/3793924109 learned_addr learned my addr 172.21.2.112:0/3793924109 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:32:33.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:33.495 7f6255ad4700 1 -- 172.21.2.112:0/3793924109 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f62500f0030 msgr2=0x7f62500f0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:33.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:33.495 7f6255ad4700 1 --2- 172.21.2.112:0/3793924109 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f62500f0030 0x7f62500f0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:33.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:33.495 7f6255ad4700 1 -- 172.21.2.112:0/3793924109 >> v2:172.21.2.120:3300/0 conn(0x7f62501b0470 msgr2=0x7f62501b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:33.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:33.495 7f6255ad4700 1 --2- 172.21.2.112:0/3793924109 >> v2:172.21.2.120:3300/0 conn(0x7f62501b0470 0x7f62501b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:33.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:33.495 7f6255ad4700 1 -- 172.21.2.112:0/3793924109 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62500f10d0 con 0x7f62500ef610 2021-02-10T11:32:33.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:33.495 7f6255ad4700 1 --2- 172.21.2.112:0/3793924109 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f62500ef610 0x7f62500efa10 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 rx=0x7f624c006700 tx=0x7f624c009b80).ready entity=mon.0 client_cookie=5fa1de7d79a5f178 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:33.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:33.495 7f62552d3700 1 -- 172.21.2.112:0/3793924109 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f624c00c0c0 con 0x7f62500ef610 2021-02-10T11:32:33.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:33.495 7f62552d3700 1 -- 172.21.2.112:0/3793924109 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f624c00c440 con 0x7f62500ef610 2021-02-10T11:32:33.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:33.496 7f62552d3700 1 -- 172.21.2.112:0/3793924109 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f624c00c960 con 0x7f62500ef610 2021-02-10T11:32:33.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:33.496 7f62584b9700 1 -- 172.21.2.112:0/3793924109 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f62500ef610 msgr2=0x7f62500efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:33.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:33.496 7f62584b9700 1 --2- 172.21.2.112:0/3793924109 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f62500ef610 0x7f62500efa10 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 rx=0x7f624c006700 tx=0x7f624c009b80).stop 2021-02-10T11:32:33.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:33.496 7f62584b9700 1 -- 172.21.2.112:0/3793924109 shutdown_connections 2021-02-10T11:32:33.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:33.496 7f62584b9700 1 --2- 172.21.2.112:0/3793924109 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f62500f0030 0x7f62500f0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:33.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:33.496 7f62584b9700 1 --2- 172.21.2.112:0/3793924109 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f62500ef610 0x7f62500efa10 unknown :-1 s=CLOSED pgs=1120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:33.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:33.496 7f62584b9700 1 --2- 172.21.2.112:0/3793924109 >> v2:172.21.2.120:3300/0 conn(0x7f62501b0470 0x7f62501b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:33.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:33.496 7f62584b9700 1 -- 172.21.2.112:0/3793924109 >> 172.21.2.112:0/3793924109 conn(0x7f62500ce3e0 msgr2=0x7f62500e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:33.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:33.497 7f62584b9700 1 -- 172.21.2.112:0/3793924109 shutdown_connections 2021-02-10T11:32:33.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:33.497 7f62584b9700 1 -- 172.21.2.112:0/3793924109 wait complete. 2021-02-10T11:32:33.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:33.511 7f62584b9700 1 Processor -- start 2021-02-10T11:32:33.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:33.511 7f62584b9700 1 -- start start 2021-02-10T11:32:33.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:33.511 7f62584b9700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f62500b20b0 0x7f62500ac970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:33.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:33.511 7f62584b9700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f62500b24b0 0x7f6250191dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:33.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:33.511 7f62584b9700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f62500acf90 0x7f62501923e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:33.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:33.511 7f62584b9700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62500b1b30 con 0x7f62500b24b0 2021-02-10T11:32:33.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:33.511 7f62584b9700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62501a04e0 con 0x7f62500b20b0 2021-02-10T11:32:33.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:33.511 7f62584b9700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62501a06c0 con 0x7f62500acf90 2021-02-10T11:32:33.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:33.512 7f6255ad4700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f62500b24b0 0x7f6250191dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:33.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:33.512 7f6255ad4700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f62500b24b0 0x7f6250191dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:49008/0 (socket says 172.21.2.112:49008) 2021-02-10T11:32:33.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:33.512 7f6255ad4700 1 -- 172.21.2.112:0/3528135182 learned_addr learned my addr 172.21.2.112:0/3528135182 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:32:33.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:33.512 7f6255ad4700 1 -- 172.21.2.112:0/3528135182 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f62500acf90 msgr2=0x7f62501923e0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:32:33.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:33.512 7f6255ad4700 1 --2- 172.21.2.112:0/3528135182 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f62500acf90 0x7f62501923e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:33.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:33.512 7f6255ad4700 1 -- 172.21.2.112:0/3528135182 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f62500b20b0 msgr2=0x7f62500ac970 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:32:33.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:33.512 7f6255ad4700 1 --2- 172.21.2.112:0/3528135182 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f62500b20b0 0x7f62500ac970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:33.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:33.512 7f6255ad4700 1 -- 172.21.2.112:0/3528135182 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6250192be0 con 0x7f62500b24b0 2021-02-10T11:32:33.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:33.512 7f6255ad4700 1 --2- 172.21.2.112:0/3528135182 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f62500b24b0 0x7f6250191dc0 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 rx=0x7f624c00b650 tx=0x7f624c00b9f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:33.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:33.512 7f6242ffd700 1 -- 172.21.2.112:0/3528135182 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f624c00bde0 con 0x7f62500b24b0 2021-02-10T11:32:33.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:33.512 7f6242ffd700 1 -- 172.21.2.112:0/3528135182 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f624c000c90 con 0x7f62500b24b0 2021-02-10T11:32:33.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:33.512 7f6242ffd700 1 -- 172.21.2.112:0/3528135182 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f624c0022b0 con 0x7f62500b24b0 2021-02-10T11:32:33.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:33.512 7f62584b9700 1 -- 172.21.2.112:0/3528135182 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6250192be0 con 0x7f62500b24b0 2021-02-10T11:32:33.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:33.513 7f62584b9700 1 -- 172.21.2.112:0/3528135182 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6250193170 con 0x7f62500b24b0 2021-02-10T11:32:33.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:33.513 7f6242ffd700 1 -- 172.21.2.112:0/3528135182 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f624c009f90 con 0x7f62500b24b0 2021-02-10T11:32:33.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:33.513 7f6242ffd700 1 --2- 172.21.2.112:0/3528135182 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f62380186c0 0x7f623801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:33.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:33.513 7f6242ffd700 1 -- 172.21.2.112:0/3528135182 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f624c0323f0 con 0x7f62500b24b0 2021-02-10T11:32:33.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:33.514 7f62562d5700 1 --2- 172.21.2.112:0/3528135182 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f62380186c0 0x7f623801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:33.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:33.514 7f62562d5700 1 --2- 172.21.2.112:0/3528135182 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f62380186c0 0x7f623801abc0 crc :-1 s=READY pgs=569 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:33.532 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:33.545 7f62584b9700 1 -- 172.21.2.112:0/3528135182 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6250105af0 con 0x7f62500b24b0 2021-02-10T11:32:33.534 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:33.547 7f6242ffd700 1 -- 172.21.2.112:0/3528135182 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f624c02d0e0 con 0x7f62500b24b0 2021-02-10T11:32:33.670 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:33.683 7f62584b9700 1 -- 172.21.2.112:0/3528135182 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"} v 0) v1 -- 0x7f62500f0b50 con 0x7f62500b24b0 2021-02-10T11:32:33.868 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:33.881 7f6242ffd700 1 -- 172.21.2.112:0/3528135182 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"}]=0 v30) v1 ==== 118+0+0 (secure 0 0 0) 0x7f624c02d0e0 con 0x7f62500b24b0 2021-02-10T11:32:33.888 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:33.900 7f62584b9700 1 -- 172.21.2.112:0/3528135182 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f62380186c0 msgr2=0x7f623801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:33.888 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:33.900 7f62584b9700 1 --2- 172.21.2.112:0/3528135182 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f62380186c0 0x7f623801abc0 crc :-1 s=READY pgs=569 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:33.888 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:33.900 7f62584b9700 1 -- 172.21.2.112:0/3528135182 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f62500b24b0 msgr2=0x7f6250191dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:33.888 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:33.900 7f62584b9700 1 --2- 172.21.2.112:0/3528135182 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f62500b24b0 0x7f6250191dc0 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 rx=0x7f624c00b650 tx=0x7f624c00b9f0).stop 2021-02-10T11:32:33.888 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:33.901 7f62584b9700 1 -- 172.21.2.112:0/3528135182 shutdown_connections 2021-02-10T11:32:33.888 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:33.901 7f62584b9700 1 --2- 172.21.2.112:0/3528135182 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f62380186c0 0x7f623801abc0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:33.888 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:33.901 7f62584b9700 1 --2- 172.21.2.112:0/3528135182 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f62500acf90 0x7f62501923e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:33.888 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:33.901 7f62584b9700 1 --2- 172.21.2.112:0/3528135182 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f62500b24b0 0x7f6250191dc0 unknown :-1 s=CLOSED pgs=1121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:33.889 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:33.901 7f62584b9700 1 --2- 172.21.2.112:0/3528135182 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f62500b20b0 0x7f62500ac970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:33.889 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:33.901 7f62584b9700 1 -- 172.21.2.112:0/3528135182 >> 172.21.2.112:0/3528135182 conn(0x7f62500bf4e0 msgr2=0x7f62500be3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:33.889 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:33.901 7f62584b9700 1 -- 172.21.2.112:0/3528135182 shutdown_connections 2021-02-10T11:32:33.889 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:33.901 7f62584b9700 1 -- 172.21.2.112:0/3528135182 wait complete. 2021-02-10T11:32:33.896 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:32:34.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.318 7f35bfbde700 1 Processor -- start 2021-02-10T11:32:34.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.318 7f35bfbde700 1 -- start start 2021-02-10T11:32:34.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.319 7f35bfbde700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f35b807e6a0 0x7f35b80d1790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:34.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.319 7f35bfbde700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35b807ed80 con 0x7f35b80e5070 2021-02-10T11:32:34.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.319 7f35bfbde700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35b807eaa0 con 0x7f35b80e5470 2021-02-10T11:32:34.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.319 7f35bfbde700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35b81b4080 con 0x7f35b807e6a0 2021-02-10T11:32:34.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.319 7f35bd9fa700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f35b80e5070 0x7f35b81b3820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:48478/0 (socket says 172.21.2.112:48478) 2021-02-10T11:32:34.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.319 7f35bd9fa700 1 -- 172.21.2.112:0/927314485 learned_addr learned my addr 172.21.2.112:0/927314485 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:32:34.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.319 7f35be1fb700 1 --2- 172.21.2.112:0/927314485 >> v2:172.21.2.112:3300/0 conn(0x7f35b807e6a0 0x7f35b80d1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:34.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.320 7f35be1fb700 1 -- 172.21.2.112:0/927314485 >> v1:172.21.2.120:6789/0 conn(0x7f35b80e5470 legacy=0x7f35b807f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:34.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.320 7f35be1fb700 1 -- 172.21.2.112:0/927314485 >> v1:172.21.2.112:6789/0 conn(0x7f35b80e5070 legacy=0x7f35b81b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:34.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.320 7f35be1fb700 1 -- 172.21.2.112:0/927314485 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35b80d1db0 con 0x7f35b807e6a0 2021-02-10T11:32:34.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.320 7f35be1fb700 1 --2- 172.21.2.112:0/927314485 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f35b807e6a0 0x7f35b80d1790 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 rx=0x7f35b0006e30 tx=0x7f35b0009e60).ready entity=mon.0 client_cookie=6443d5c63201cc53 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:34.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.320 7f35bc9f8700 1 -- 172.21.2.112:0/927314485 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f35b0006a50 con 0x7f35b807e6a0 2021-02-10T11:32:34.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.320 7f35bc9f8700 1 -- 172.21.2.112:0/927314485 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35b000b300 con 0x7f35b807e6a0 2021-02-10T11:32:34.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.320 7f35bc9f8700 1 -- 172.21.2.112:0/927314485 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f35b000b820 con 0x7f35b807e6a0 2021-02-10T11:32:34.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.320 7f35bfbde700 1 -- 172.21.2.112:0/927314485 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f35b807e6a0 msgr2=0x7f35b80d1790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:34.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.320 7f35bfbde700 1 --2- 172.21.2.112:0/927314485 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f35b807e6a0 0x7f35b80d1790 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 rx=0x7f35b0006e30 tx=0x7f35b0009e60).stop 2021-02-10T11:32:34.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.321 7f35bfbde700 1 -- 172.21.2.112:0/927314485 shutdown_connections 2021-02-10T11:32:34.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.321 7f35bfbde700 1 --2- 172.21.2.112:0/927314485 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f35b807e6a0 0x7f35b80d1790 unknown :-1 s=CLOSED pgs=1123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:34.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.321 7f35bfbde700 1 -- 172.21.2.112:0/927314485 >> 172.21.2.112:0/927314485 conn(0x7f35b80ce690 msgr2=0x7f35b80e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:34.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.321 7f35bfbde700 1 -- 172.21.2.112:0/927314485 shutdown_connections 2021-02-10T11:32:34.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.321 7f35bfbde700 1 -- 172.21.2.112:0/927314485 wait complete. 2021-02-10T11:32:34.326 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.336 7f35bfbde700 1 Processor -- start 2021-02-10T11:32:34.326 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.336 7f35bfbde700 1 -- start start 2021-02-10T11:32:34.326 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.336 7f35bfbde700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f35b81c55a0 0x7f35b81c59a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:34.326 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.336 7f35bfbde700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f35b80bfdc0 0x7f35b80c01c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:34.326 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.336 7f35bfbde700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f35b8099900 0x7f35b8099d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:34.326 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.336 7f35bfbde700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35b80c09f0 con 0x7f35b81c55a0 2021-02-10T11:32:34.326 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.336 7f35bfbde700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35b809a320 con 0x7f35b8099900 2021-02-10T11:32:34.326 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.336 7f35bfbde700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35b809a540 con 0x7f35b80bfdc0 2021-02-10T11:32:34.327 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.336 7f35bd9fa700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f35b81c55a0 0x7f35b81c59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:34.327 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.336 7f35bd9fa700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f35b81c55a0 0x7f35b81c59a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:49022/0 (socket says 172.21.2.112:49022) 2021-02-10T11:32:34.327 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.336 7f35bd9fa700 1 -- 172.21.2.112:0/3133051921 learned_addr learned my addr 172.21.2.112:0/3133051921 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:32:34.327 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.337 7f35bd9fa700 1 -- 172.21.2.112:0/3133051921 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f35b80bfdc0 msgr2=0x7f35b80c01c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:34.327 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.337 7f35bd9fa700 1 --2- 172.21.2.112:0/3133051921 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f35b80bfdc0 0x7f35b80c01c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:34.327 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.337 7f35bd9fa700 1 -- 172.21.2.112:0/3133051921 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f35b8099900 msgr2=0x7f35b8099d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:34.327 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.337 7f35bd9fa700 1 --2- 172.21.2.112:0/3133051921 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f35b8099900 0x7f35b8099d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:34.327 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.337 7f35bd9fa700 1 -- 172.21.2.112:0/3133051921 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35b809a760 con 0x7f35b81c55a0 2021-02-10T11:32:34.327 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.337 7f35bd9fa700 1 --2- 172.21.2.112:0/3133051921 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f35b81c55a0 0x7f35b81c59a0 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 rx=0x7f35b4001c60 tx=0x7f35b400a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:34.327 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.337 7f35ae7fc700 1 -- 172.21.2.112:0/3133051921 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f35b400acc0 con 0x7f35b81c55a0 2021-02-10T11:32:34.327 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.337 7f35ae7fc700 1 -- 172.21.2.112:0/3133051921 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35b4004140 con 0x7f35b81c55a0 2021-02-10T11:32:34.327 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.337 7f35ae7fc700 1 -- 172.21.2.112:0/3133051921 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f35b4004660 con 0x7f35b81c55a0 2021-02-10T11:32:34.327 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.337 7f35bfbde700 1 -- 172.21.2.112:0/3133051921 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35b809a760 con 0x7f35b81c55a0 2021-02-10T11:32:34.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.338 7f35bfbde700 1 -- 172.21.2.112:0/3133051921 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35b809a760 con 0x7f35b81c55a0 2021-02-10T11:32:34.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.338 7f35ae7fc700 1 -- 172.21.2.112:0/3133051921 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f35b4000ce0 con 0x7f35b81c55a0 2021-02-10T11:32:34.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.338 7f35ae7fc700 1 --2- 172.21.2.112:0/3133051921 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f35a00186c0 0x7f35a001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:34.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.338 7f35ae7fc700 1 -- 172.21.2.112:0/3133051921 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f35b4032550 con 0x7f35b81c55a0 2021-02-10T11:32:34.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.339 7f35bd1f9700 1 --2- 172.21.2.112:0/3133051921 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f35a00186c0 0x7f35a001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:34.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.339 7f35bd1f9700 1 --2- 172.21.2.112:0/3133051921 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f35a00186c0 0x7f35a001abc0 crc :-1 s=READY pgs=570 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:34.356 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.369 7f35bfbde700 1 -- 172.21.2.112:0/3133051921 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35b80ce690 con 0x7f35b81c55a0 2021-02-10T11:32:34.358 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.371 7f35ae7fc700 1 -- 172.21.2.112:0/3133051921 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f35b40009b0 con 0x7f35b81c55a0 2021-02-10T11:32:34.488 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.500 7f35bfbde700 1 -- 172.21.2.112:0/3133051921 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f35b80ce690 con 0x7f35b81c55a0 2021-02-10T11:32:34.488 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.501 7f35ae7fc700 1 -- 172.21.2.112:0/3133051921 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+2093 (secure 0 0 0) 0x7f35b4000ce0 con 0x7f35b81c55a0 2021-02-10T11:32:34.489 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 30 2021-02-10T11:32:34.489 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:32:34.489 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":30,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7536,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/340651080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":340651080},{"type":"v1","addr":"172.21.2.112:6833","nonce":340651080}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":26}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:32:31.869812","modified":"2021-02-10 11:32:33.879671","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6409},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6409":{"gid":6409,"name":"b","rank":0,"incarnation":27,"state":"up:active","state_seq":3,"addr":"172.21.2.120:6835/210055046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":210055046},{"type":"v1","addr":"172.21.2.120:6835","nonce":210055046}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":4}]} 2021-02-10T11:32:34.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.503 7f35bfbde700 1 -- 172.21.2.112:0/3133051921 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f35a00186c0 msgr2=0x7f35a001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:34.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.503 7f35bfbde700 1 --2- 172.21.2.112:0/3133051921 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f35a00186c0 0x7f35a001abc0 crc :-1 s=READY pgs=570 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:34.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.503 7f35bfbde700 1 -- 172.21.2.112:0/3133051921 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f35b81c55a0 msgr2=0x7f35b81c59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:34.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.503 7f35bfbde700 1 --2- 172.21.2.112:0/3133051921 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f35b81c55a0 0x7f35b81c59a0 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 rx=0x7f35b4001c60 tx=0x7f35b400a8d0).stop 2021-02-10T11:32:34.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.503 7f35bfbde700 1 -- 172.21.2.112:0/3133051921 shutdown_connections 2021-02-10T11:32:34.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.503 7f35bfbde700 1 --2- 172.21.2.112:0/3133051921 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f35a00186c0 0x7f35a001abc0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:34.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.503 7f35bfbde700 1 --2- 172.21.2.112:0/3133051921 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f35b8099900 0x7f35b8099d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:34.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.503 7f35bfbde700 1 --2- 172.21.2.112:0/3133051921 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f35b80bfdc0 0x7f35b80c01c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:34.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.503 7f35bfbde700 1 --2- 172.21.2.112:0/3133051921 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f35b81c55a0 0x7f35b81c59a0 unknown :-1 s=CLOSED pgs=1124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:34.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.503 7f35bfbde700 1 -- 172.21.2.112:0/3133051921 >> 172.21.2.112:0/3133051921 conn(0x7f35b80976b0 msgr2=0x7f35b80962b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:34.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.503 7f35bfbde700 1 -- 172.21.2.112:0/3133051921 shutdown_connections 2021-02-10T11:32:34.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.503 7f35bfbde700 1 -- 172.21.2.112:0/3133051921 wait complete. 2021-02-10T11:32:34.499 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:32:34.927 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.923 7f0003332700 1 Processor -- start 2021-02-10T11:32:34.927 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.923 7f0003332700 1 -- start start 2021-02-10T11:32:34.927 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.924 7f0003332700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7efffc0e5070 0x7efffc1b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:34.928 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.924 7f0003332700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7efffc1b3e40 0x7efffc07d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:34.928 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.924 7f0003332700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efffc07ed80 con 0x7efffc0e5470 2021-02-10T11:32:34.928 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.924 7f0003332700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efffc080310 con 0x7efffc1b3e40 2021-02-10T11:32:34.928 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.924 7f0003332700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efffc080530 con 0x7efffc0e5070 2021-02-10T11:32:34.928 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.924 7f000094d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7efffc1b3e40 0x7efffc07d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:34.928 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.924 7f000094d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7efffc1b3e40 0x7efffc07d690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:49030/0 (socket says 172.21.2.112:49030) 2021-02-10T11:32:34.928 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.924 7f000094d700 1 -- 172.21.2.112:0/140337246 learned_addr learned my addr 172.21.2.112:0/140337246 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:32:34.928 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.924 7f000114e700 1 --2- 172.21.2.112:0/140337246 >> v2:172.21.2.120:3300/0 conn(0x7efffc0e5070 0x7efffc1b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:34.928 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.924 7f000094d700 1 -- 172.21.2.112:0/140337246 >> v2:172.21.2.120:3300/0 conn(0x7efffc0e5070 msgr2=0x7efffc1b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:34.928 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.924 7f000094d700 1 --2- 172.21.2.112:0/140337246 >> v2:172.21.2.120:3300/0 conn(0x7efffc0e5070 0x7efffc1b3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:34.928 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.924 7f000094d700 1 -- 172.21.2.112:0/140337246 >> v1:172.21.2.120:6789/0 conn(0x7efffc0e5470 legacy=0x7efffc07dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:34.928 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.924 7f000094d700 1 -- 172.21.2.112:0/140337246 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efffc080750 con 0x7efffc1b3e40 2021-02-10T11:32:34.928 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.924 7f000094d700 1 --2- 172.21.2.112:0/140337246 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efffc1b3e40 0x7efffc07d690 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 rx=0x7efff0006700 tx=0x7efff0009b80).ready entity=mon.0 client_cookie=19d47162ad6ecbc4 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:34.929 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.924 7effeffff700 1 -- 172.21.2.112:0/140337246 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efff000c0c0 con 0x7efffc1b3e40 2021-02-10T11:32:34.929 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.925 7effeffff700 1 -- 172.21.2.112:0/140337246 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efff000c400 con 0x7efffc1b3e40 2021-02-10T11:32:34.929 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.925 7effeffff700 1 -- 172.21.2.112:0/140337246 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efff000c860 con 0x7efffc1b3e40 2021-02-10T11:32:34.929 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.925 7f0003332700 1 -- 172.21.2.112:0/140337246 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efffc1b3e40 msgr2=0x7efffc07d690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:34.929 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.925 7f0003332700 1 --2- 172.21.2.112:0/140337246 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efffc1b3e40 0x7efffc07d690 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 rx=0x7efff0006700 tx=0x7efff0009b80).stop 2021-02-10T11:32:34.929 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.925 7f0003332700 1 -- 172.21.2.112:0/140337246 shutdown_connections 2021-02-10T11:32:34.929 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.925 7f0003332700 1 --2- 172.21.2.112:0/140337246 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efffc1b3e40 0x7efffc07d690 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:34.929 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.925 7f0003332700 1 --2- 172.21.2.112:0/140337246 >> v2:172.21.2.120:3300/0 conn(0x7efffc0e5070 0x7efffc1b3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:34.929 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.925 7f0003332700 1 -- 172.21.2.112:0/140337246 >> 172.21.2.112:0/140337246 conn(0x7efffc0ce690 msgr2=0x7efffc0e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:34.929 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.926 7f0003332700 1 -- 172.21.2.112:0/140337246 shutdown_connections 2021-02-10T11:32:34.929 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.926 7f0003332700 1 -- 172.21.2.112:0/140337246 wait complete. 2021-02-10T11:32:34.929 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.940 7f0003332700 1 Processor -- start 2021-02-10T11:32:34.929 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.940 7f0003332700 1 -- start start 2021-02-10T11:32:34.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.941 7f0003332700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efffc193860 0x7efffc0a9ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:34.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.941 7f0003332700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7efffc0aa500 0x7efffc1073b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:34.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.941 7f0003332700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efffc1079d0 0x7efffc107dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:34.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.941 7f0003332700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efffc1af7b0 con 0x7efffc193860 2021-02-10T11:32:34.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.941 7f0003332700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efffc087fb0 con 0x7efffc0aa500 2021-02-10T11:32:34.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.941 7f0003332700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efffc1affc0 con 0x7efffc1079d0 2021-02-10T11:32:34.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.941 7f000114e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efffc193860 0x7efffc0a9ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:34.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.941 7f000114e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efffc193860 0x7efffc0a9ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:49034/0 (socket says 172.21.2.112:49034) 2021-02-10T11:32:34.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.941 7f000114e700 1 -- 172.21.2.112:0/2620023040 learned_addr learned my addr 172.21.2.112:0/2620023040 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:32:34.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.941 7f000094d700 1 --2- 172.21.2.112:0/2620023040 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7efffc0aa500 0x7efffc1073b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:34.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.941 7f000194f700 1 --2- 172.21.2.112:0/2620023040 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efffc1079d0 0x7efffc107dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:34.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.941 7f000114e700 1 -- 172.21.2.112:0/2620023040 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efffc1079d0 msgr2=0x7efffc107dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:34.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.941 7f000114e700 1 --2- 172.21.2.112:0/2620023040 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efffc1079d0 0x7efffc107dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:34.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.941 7f000114e700 1 -- 172.21.2.112:0/2620023040 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7efffc0aa500 msgr2=0x7efffc1073b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:34.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.941 7f000114e700 1 --2- 172.21.2.112:0/2620023040 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7efffc0aa500 0x7efffc1073b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:34.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.941 7f000114e700 1 -- 172.21.2.112:0/2620023040 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efffc1b01e0 con 0x7efffc193860 2021-02-10T11:32:34.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.941 7f000114e700 1 --2- 172.21.2.112:0/2620023040 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efffc193860 0x7efffc0a9ee0 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 rx=0x7efff8006f60 tx=0x7efff800b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:34.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.942 7effedffb700 1 -- 172.21.2.112:0/2620023040 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efff800bd60 con 0x7efffc193860 2021-02-10T11:32:34.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.942 7effedffb700 1 -- 172.21.2.112:0/2620023040 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efff8000c90 con 0x7efffc193860 2021-02-10T11:32:34.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.942 7effedffb700 1 -- 172.21.2.112:0/2620023040 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efff80022b0 con 0x7efffc193860 2021-02-10T11:32:34.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.942 7f0003332700 1 -- 172.21.2.112:0/2620023040 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efffc1b01e0 con 0x7efffc193860 2021-02-10T11:32:34.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.942 7f0003332700 1 -- 172.21.2.112:0/2620023040 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efffc1b0770 con 0x7efffc193860 2021-02-10T11:32:34.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.942 7effedffb700 1 -- 172.21.2.112:0/2620023040 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7efff8006b00 con 0x7efffc193860 2021-02-10T11:32:34.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.942 7effedffb700 1 --2- 172.21.2.112:0/2620023040 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7effdc0186c0 0x7effdc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:34.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.942 7effedffb700 1 -- 172.21.2.112:0/2620023040 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5573+0+0 (secure 0 0 0) 0x7efff80323d0 con 0x7efffc193860 2021-02-10T11:32:34.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.943 7f000094d700 1 --2- 172.21.2.112:0/2620023040 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7effdc0186c0 0x7effdc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:34.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.943 7f000094d700 1 --2- 172.21.2.112:0/2620023040 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7effdc0186c0 0x7effdc01abc0 crc :-1 s=READY pgs=571 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:34.960 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.973 7f0003332700 1 -- 172.21.2.112:0/2620023040 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efffc07ed80 con 0x7efffc193860 2021-02-10T11:32:34.963 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:34.976 7effedffb700 1 -- 172.21.2.112:0/2620023040 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7efff8030d00 con 0x7efffc193860 2021-02-10T11:32:35.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:35.101 7f0003332700 1 -- 172.21.2.112:0/2620023040 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efffc07ed80 con 0x7efffc193860 2021-02-10T11:32:35.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:35.102 7effedffb700 1 -- 172.21.2.112:0/2620023040 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v46) v1 ==== 74+0+13993 (secure 0 0 0) 0x7efff8006b00 con 0x7efffc193860 2021-02-10T11:32:35.090 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:32:35.090 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":46,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:32:32.871643","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:32:29.843136","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":9,"pool_name":"cephfs_data","create_time":"2021-02-10 11:32:30.914946","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:32:35.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:35.103 7f0003332700 1 -- 172.21.2.112:0/2620023040 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7effdc0186c0 msgr2=0x7effdc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:35.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:35.104 7f0003332700 1 --2- 172.21.2.112:0/2620023040 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7effdc0186c0 0x7effdc01abc0 crc :-1 s=READY pgs=571 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:35.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:35.104 7f0003332700 1 -- 172.21.2.112:0/2620023040 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efffc193860 msgr2=0x7efffc0a9ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:35.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:35.104 7f0003332700 1 --2- 172.21.2.112:0/2620023040 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efffc193860 0x7efffc0a9ee0 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 rx=0x7efff8006f60 tx=0x7efff800b970).stop 2021-02-10T11:32:35.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:35.104 7f0003332700 1 -- 172.21.2.112:0/2620023040 shutdown_connections 2021-02-10T11:32:35.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:35.104 7f0003332700 1 --2- 172.21.2.112:0/2620023040 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7effdc0186c0 0x7effdc01abc0 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:35.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:35.104 7f0003332700 1 --2- 172.21.2.112:0/2620023040 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7efffc0aa500 0x7efffc1073b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:35.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:35.104 7f0003332700 1 --2- 172.21.2.112:0/2620023040 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efffc1079d0 0x7efffc107dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:35.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:35.104 7f0003332700 1 --2- 172.21.2.112:0/2620023040 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efffc193860 0x7efffc0a9ee0 unknown :-1 s=CLOSED pgs=1126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:35.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:35.104 7f0003332700 1 -- 172.21.2.112:0/2620023040 >> 172.21.2.112:0/2620023040 conn(0x7efffc1aa5d0 msgr2=0x7efffc19f180 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:35.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:35.104 7f0003332700 1 -- 172.21.2.112:0/2620023040 shutdown_connections 2021-02-10T11:32:35.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:35.104 7f0003332700 1 -- 172.21.2.112:0/2620023040 wait complete. 2021-02-10T11:32:35.100 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:32:35.540 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:35.535 7f8c9052a700 1 Processor -- start 2021-02-10T11:32:35.540 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:35.535 7f8c9052a700 1 -- start start 2021-02-10T11:32:35.540 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:35.536 7f8c9052a700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f8c881b0470 0x7f8c881b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:35.540 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:35.536 7f8c9052a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8c880ef610 0x7f8c880efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:35.540 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:35.536 7f8c9052a700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c880f0b50 con 0x7f8c880f0030 2021-02-10T11:32:35.540 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:35.536 7f8c9052a700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c880e52e0 con 0x7f8c881b0470 2021-02-10T11:32:35.540 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:35.536 7f8c9052a700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c880e54d0 con 0x7f8c880ef610 2021-02-10T11:32:35.540 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:35.536 7f8c8eb47700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f8c880f0030 0x7f8c880f10d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:48502/0 (socket says 172.21.2.112:48502) 2021-02-10T11:32:35.540 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:35.536 7f8c8eb47700 1 -- 172.21.2.112:0/1765856853 learned_addr learned my addr 172.21.2.112:0/1765856853 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:32:35.540 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:35.536 7f8c8d344700 1 -- 172.21.2.112:0/1765856853 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3683878108 0 0) 0x7f8c80001500 con 0x7f8c880f0030 2021-02-10T11:32:35.541 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:35.536 7f8c8d344700 1 -- 172.21.2.112:0/1765856853 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c74002fe0 con 0x7f8c880f0030 2021-02-10T11:32:35.541 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:35.536 7f8c8d344700 1 -- 172.21.2.112:0/1765856853 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3740519671 0 0) 0x7f8c80001460 con 0x7f8c880f0030 2021-02-10T11:32:35.541 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:35.536 7f8c8d344700 1 -- 172.21.2.112:0/1765856853 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8c880ef610 msgr2=0x7f8c880efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:35.541 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:35.536 7f8c8d344700 1 --2- 172.21.2.112:0/1765856853 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8c880ef610 0x7f8c880efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:35.541 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:35.536 7f8c8d344700 1 -- 172.21.2.112:0/1765856853 >> v2:172.21.2.120:3300/0 conn(0x7f8c881b0470 msgr2=0x7f8c881b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:35.541 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:35.536 7f8c8d344700 1 --2- 172.21.2.112:0/1765856853 >> v2:172.21.2.120:3300/0 conn(0x7f8c881b0470 0x7f8c881b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:35.541 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:35.536 7f8c8d344700 1 -- 172.21.2.112:0/1765856853 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c880f0430 con 0x7f8c880f0030 2021-02-10T11:32:35.541 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:35.536 7f8c8d344700 1 -- 172.21.2.112:0/1765856853 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f8c80001980 con 0x7f8c880f0030 2021-02-10T11:32:35.541 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:35.536 7f8c8d344700 1 -- 172.21.2.112:0/1765856853 >> v1:172.21.2.112:6789/0 conn(0x7f8c880f0030 legacy=0x7f8c880f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:35.541 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:35.536 7f8c8d344700 1 --2- 172.21.2.112:0/1765856853 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8c74003a40 0x7f8c74005ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:35.541 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:35.536 7f8c8d344700 1 --2- 172.21.2.112:0/1765856853 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8c74006770 0x7f8c74008c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:35.541 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:35.537 7f8c8d344700 1 --2- 172.21.2.112:0/1765856853 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8c74009500 0x7f8c7400ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:35.541 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:35.537 7f8c8d344700 1 -- 172.21.2.112:0/1765856853 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c7400c2d0 con 0x7f8c74003a40 2021-02-10T11:32:35.541 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:35.537 7f8c8d344700 1 -- 172.21.2.112:0/1765856853 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c7400c4f0 con 0x7f8c74009500 2021-02-10T11:32:35.542 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:35.537 7f8c8d344700 1 -- 172.21.2.112:0/1765856853 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c7400c710 con 0x7f8c74006770 2021-02-10T11:32:35.542 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:35.537 7f8c8e346700 1 --2- 172.21.2.112:0/1765856853 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8c74003a40 0x7f8c74005ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:35.542 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:35.537 7f8c8db45700 1 --2- 172.21.2.112:0/1765856853 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8c74006770 0x7f8c74008c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:35.542 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:35.537 7f8c8eb47700 1 --2- 172.21.2.112:0/1765856853 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8c74009500 0x7f8c7400ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:35.542 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:35.537 7f8c8e346700 1 -- 172.21.2.112:0/1765856853 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8c74006770 msgr2=0x7f8c74008c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:35.542 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:35.537 7f8c8e346700 1 --2- 172.21.2.112:0/1765856853 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8c74006770 0x7f8c74008c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:35.542 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:35.537 7f8c8e346700 1 -- 172.21.2.112:0/1765856853 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8c74009500 msgr2=0x7f8c7400ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:35.542 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:35.537 7f8c8e346700 1 --2- 172.21.2.112:0/1765856853 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8c74009500 0x7f8c7400ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:35.542 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:35.537 7f8c8e346700 1 -- 172.21.2.112:0/1765856853 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c7400c9b0 con 0x7f8c74003a40 2021-02-10T11:32:35.542 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:35.537 7f8c8e346700 1 --2- 172.21.2.112:0/1765856853 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8c74003a40 0x7f8c74005ed0 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 rx=0x7f8c7c006c10 tx=0x7f8c7c04dc60).ready entity=mon.0 client_cookie=c7f148c7c4007dbf server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:35.542 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:35.537 7f8c8d344700 1 -- 172.21.2.112:0/1765856853 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8c7c006810 con 0x7f8c74003a40 2021-02-10T11:32:35.542 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:35.537 7f8c8d344700 1 -- 172.21.2.112:0/1765856853 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c7c000c90 con 0x7f8c74003a40 2021-02-10T11:32:35.542 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:35.537 7f8c8d344700 1 -- 172.21.2.112:0/1765856853 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8c7c0009b0 con 0x7f8c74003a40 2021-02-10T11:32:35.543 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:35.537 7f8c9052a700 1 -- 172.21.2.112:0/1765856853 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8c74003a40 msgr2=0x7f8c74005ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:35.543 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:35.537 7f8c9052a700 1 --2- 172.21.2.112:0/1765856853 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8c74003a40 0x7f8c74005ed0 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 rx=0x7f8c7c006c10 tx=0x7f8c7c04dc60).stop 2021-02-10T11:32:35.543 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:35.537 7f8c8eb47700 1 -- 172.21.2.112:0/1765856853 reap_dead start 2021-02-10T11:32:35.543 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:35.538 7f8c9052a700 1 -- 172.21.2.112:0/1765856853 shutdown_connections 2021-02-10T11:32:35.543 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:35.538 7f8c9052a700 1 -- 172.21.2.112:0/1765856853 >> 172.21.2.112:0/1765856853 conn(0x7f8c880ce3e0 msgr2=0x7f8c880e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:35.543 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:35.538 7f8c9052a700 1 -- 172.21.2.112:0/1765856853 shutdown_connections 2021-02-10T11:32:35.543 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:35.538 7f8c9052a700 1 -- 172.21.2.112:0/1765856853 wait complete. 2021-02-10T11:32:35.543 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:35.553 7f8c9052a700 1 Processor -- start 2021-02-10T11:32:35.543 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:35.553 7f8c9052a700 1 -- start start 2021-02-10T11:32:35.545 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:35.553 7f8c9052a700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8c880b8530 0x7f8c880b8930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:35.545 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:35.553 7f8c9052a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8c880b8f50 0x7f8c880b9350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:35.545 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:35.553 7f8c9052a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8c880b65c0 0x7f8c880b69c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:35.545 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:35.553 7f8c9052a700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c880b99f0 con 0x7f8c880b8f50 2021-02-10T11:32:35.545 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:35.553 7f8c9052a700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c880b8020 con 0x7f8c880b8530 2021-02-10T11:32:35.545 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:35.553 7f8c9052a700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c88091650 con 0x7f8c880b65c0 2021-02-10T11:32:35.545 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:35.553 7f8c8db45700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8c880b8f50 0x7f8c880b9350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:35.545 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:35.553 7f8c8db45700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8c880b8f50 0x7f8c880b9350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:49048/0 (socket says 172.21.2.112:49048) 2021-02-10T11:32:35.545 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:35.553 7f8c8db45700 1 -- 172.21.2.112:0/3973192188 learned_addr learned my addr 172.21.2.112:0/3973192188 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:32:35.545 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:35.553 7f8c8e346700 1 --2- 172.21.2.112:0/3973192188 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8c880b8530 0x7f8c880b8930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:35.546 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:35.554 7f8c8eb47700 1 --2- 172.21.2.112:0/3973192188 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8c880b65c0 0x7f8c880b69c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:35.546 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:35.554 7f8c8db45700 1 -- 172.21.2.112:0/3973192188 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8c880b65c0 msgr2=0x7f8c880b69c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:35.546 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:35.554 7f8c8db45700 1 --2- 172.21.2.112:0/3973192188 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8c880b65c0 0x7f8c880b69c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:35.546 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:35.554 7f8c8db45700 1 -- 172.21.2.112:0/3973192188 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8c880b8530 msgr2=0x7f8c880b8930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:35.546 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:35.554 7f8c8db45700 1 --2- 172.21.2.112:0/3973192188 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8c880b8530 0x7f8c880b8930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:35.546 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:35.554 7f8c8db45700 1 -- 172.21.2.112:0/3973192188 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c880b6fe0 con 0x7f8c880b8f50 2021-02-10T11:32:35.546 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:35.554 7f8c8eb47700 1 --2- 172.21.2.112:0/3973192188 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8c880b65c0 0x7f8c880b69c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:32:35.546 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:35.554 7f8c8db45700 1 --2- 172.21.2.112:0/3973192188 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8c880b8f50 0x7f8c880b9350 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 rx=0x7f8c84006f00 tx=0x7f8c8400b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:35.546 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:35.554 7f8c7affd700 1 -- 172.21.2.112:0/3973192188 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8c8400bd30 con 0x7f8c880b8f50 2021-02-10T11:32:35.546 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:35.554 7f8c7affd700 1 -- 172.21.2.112:0/3973192188 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c84002030 con 0x7f8c880b8f50 2021-02-10T11:32:35.546 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:35.554 7f8c9052a700 1 -- 172.21.2.112:0/3973192188 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c880b6fe0 con 0x7f8c880b8f50 2021-02-10T11:32:35.546 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:35.554 7f8c9052a700 1 -- 172.21.2.112:0/3973192188 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c880b7570 con 0x7f8c880b8f50 2021-02-10T11:32:35.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:35.554 7f8c7affd700 1 -- 172.21.2.112:0/3973192188 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8c84002490 con 0x7f8c880b8f50 2021-02-10T11:32:35.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:35.555 7f8c7affd700 1 -- 172.21.2.112:0/3973192188 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8c84000c90 con 0x7f8c880b8f50 2021-02-10T11:32:35.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:35.555 7f8c7affd700 1 --2- 172.21.2.112:0/3973192188 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8c70018730 0x7f8c7001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:35.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:35.555 7f8c7affd700 1 -- 172.21.2.112:0/3973192188 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f8c84032840 con 0x7f8c880b8f50 2021-02-10T11:32:35.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:35.555 7f8c8e346700 1 --2- 172.21.2.112:0/3973192188 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8c70018730 0x7f8c7001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:35.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:35.556 7f8c8e346700 1 --2- 172.21.2.112:0/3973192188 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8c70018730 0x7f8c7001ac10 crc :-1 s=READY pgs=572 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:35.573 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:35.585 7f8c9052a700 1 -- 172.21.2.112:0/3973192188 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c88105a30 con 0x7f8c880b8f50 2021-02-10T11:32:35.575 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:35.588 7f8c7affd700 1 -- 172.21.2.112:0/3973192188 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8c84000940 con 0x7f8c880b8f50 2021-02-10T11:32:35.704 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:35.717 7f8c9052a700 1 -- 172.21.2.112:0/3973192188 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f8c880f0b50 con 0x7f8c880b8f50 2021-02-10T11:32:35.708 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:35.720 7f8c7affd700 1 -- 172.21.2.112:0/3973192188 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v25) v1 ==== 184+0+0 (secure 0 0 0) 0x7f8c8401f580 con 0x7f8c880b8f50 2021-02-10T11:32:35.711 INFO:teuthology.orchestra.run.gibba012.stderr:updated caps for client.0 2021-02-10T11:32:35.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:35.725 7f8c9052a700 1 -- 172.21.2.112:0/3973192188 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8c70018730 msgr2=0x7f8c7001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:35.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:35.725 7f8c9052a700 1 --2- 172.21.2.112:0/3973192188 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8c70018730 0x7f8c7001ac10 crc :-1 s=READY pgs=572 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:35.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:35.725 7f8c9052a700 1 -- 172.21.2.112:0/3973192188 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8c880b8f50 msgr2=0x7f8c880b9350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:35.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:35.725 7f8c9052a700 1 --2- 172.21.2.112:0/3973192188 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8c880b8f50 0x7f8c880b9350 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 rx=0x7f8c84006f00 tx=0x7f8c8400b940).stop 2021-02-10T11:32:35.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:35.725 7f8c9052a700 1 -- 172.21.2.112:0/3973192188 shutdown_connections 2021-02-10T11:32:35.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:35.725 7f8c9052a700 1 --2- 172.21.2.112:0/3973192188 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8c70018730 0x7f8c7001ac10 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:35.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:35.725 7f8c9052a700 1 --2- 172.21.2.112:0/3973192188 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8c880b65c0 0x7f8c880b69c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:35.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:35.725 7f8c9052a700 1 --2- 172.21.2.112:0/3973192188 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8c880b8f50 0x7f8c880b9350 unknown :-1 s=CLOSED pgs=1129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:35.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:35.725 7f8c9052a700 1 --2- 172.21.2.112:0/3973192188 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8c880b8530 0x7f8c880b8930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:35.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:35.726 7f8c9052a700 1 -- 172.21.2.112:0/3973192188 >> 172.21.2.112:0/3973192188 conn(0x7f8c880b2800 msgr2=0x7f8c88091a50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:35.714 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:35.726 7f8c9052a700 1 -- 172.21.2.112:0/3973192188 shutdown_connections 2021-02-10T11:32:35.714 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:35.726 7f8c9052a700 1 -- 172.21.2.112:0/3973192188 wait complete. 2021-02-10T11:32:35.721 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:32:36.161 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.157 7f57b9d50700 1 Processor -- start 2021-02-10T11:32:36.161 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.157 7f57b9d50700 1 -- start start 2021-02-10T11:32:36.161 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.157 7f57b9d50700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f57b40e5470 0x7f57b407e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:36.161 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.157 7f57b9d50700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57b407ed80 con 0x7f57b407f300 2021-02-10T11:32:36.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.157 7f57b9d50700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57b41b4010 con 0x7f57b40e5070 2021-02-10T11:32:36.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.157 7f57b9d50700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57b40d1790 con 0x7f57b40e5470 2021-02-10T11:32:36.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.157 7f57b3fff700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f57b407f300 0x7f57b407f700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:48518/0 (socket says 172.21.2.112:48518) 2021-02-10T11:32:36.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.157 7f57b3fff700 1 -- 172.21.2.112:0/218225420 learned_addr learned my addr 172.21.2.112:0/218225420 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:32:36.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.158 7f57b2ffd700 1 --2- 172.21.2.112:0/218225420 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f57b40e5470 0x7f57b407e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:36.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.158 7f57b27fc700 1 -- 172.21.2.112:0/218225420 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1048097807 0 0) 0x7f57ac000f90 con 0x7f57b407f300 2021-02-10T11:32:36.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.158 7f57b27fc700 1 -- 172.21.2.112:0/218225420 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f579c002fe0 con 0x7f57b407f300 2021-02-10T11:32:36.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.158 7f57b2ffd700 1 -- 172.21.2.112:0/218225420 >> v1:172.21.2.120:6789/0 conn(0x7f57b40e5070 legacy=0x7f57b41b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:36.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.158 7f57b27fc700 1 -- 172.21.2.112:0/218225420 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 794003267 0 0) 0x7f57a4000ff0 con 0x7f57b40e5070 2021-02-10T11:32:36.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.158 7f57b2ffd700 1 -- 172.21.2.112:0/218225420 >> v1:172.21.2.112:6789/0 conn(0x7f57b407f300 legacy=0x7f57b407f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:36.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.158 7f57b2ffd700 1 -- 172.21.2.112:0/218225420 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57b40d1980 con 0x7f57b40e5470 2021-02-10T11:32:36.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.158 7f57b2ffd700 1 --2- 172.21.2.112:0/218225420 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f57b40e5470 0x7f57b407e6a0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f57a8006650 tx=0x7f57a8009a80).ready entity=mon.2 client_cookie=c413ab1cbe42655a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:36.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.159 7f57b27fc700 1 -- 172.21.2.112:0/218225420 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f57a800c0c0 con 0x7f57b40e5470 2021-02-10T11:32:36.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.159 7f57b27fc700 1 -- 172.21.2.112:0/218225420 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57a800c340 con 0x7f57b40e5470 2021-02-10T11:32:36.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.159 7f57b27fc700 1 -- 172.21.2.112:0/218225420 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f57a800c860 con 0x7f57b40e5470 2021-02-10T11:32:36.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.159 7f57b9d50700 1 -- 172.21.2.112:0/218225420 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f57b40e5470 msgr2=0x7f57b407e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:36.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.159 7f57b9d50700 1 --2- 172.21.2.112:0/218225420 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f57b40e5470 0x7f57b407e6a0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f57a8006650 tx=0x7f57a8009a80).stop 2021-02-10T11:32:36.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.160 7f57b9d50700 1 -- 172.21.2.112:0/218225420 shutdown_connections 2021-02-10T11:32:36.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.160 7f57b9d50700 1 --2- 172.21.2.112:0/218225420 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f57b40e5470 0x7f57b407e6a0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:36.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.160 7f57b9d50700 1 -- 172.21.2.112:0/218225420 >> 172.21.2.112:0/218225420 conn(0x7f57b40ce690 msgr2=0x7f57b40e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:36.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.160 7f57b9d50700 1 -- 172.21.2.112:0/218225420 shutdown_connections 2021-02-10T11:32:36.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.160 7f57b9d50700 1 -- 172.21.2.112:0/218225420 wait complete. 2021-02-10T11:32:36.165 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.174 7f57b9d50700 1 Processor -- start 2021-02-10T11:32:36.165 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.175 7f57b9d50700 1 -- start start 2021-02-10T11:32:36.165 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.175 7f57b9d50700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f57b41ad130 0x7f57b419f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:36.165 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.175 7f57b9d50700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f57b40ae970 0x7f57b419f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:36.165 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.175 7f57b9d50700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f57b419fd50 0x7f57b41adb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:36.165 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.175 7f57b9d50700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57b41a0150 con 0x7f57b419fd50 2021-02-10T11:32:36.165 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.175 7f57b9d50700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57b419b970 con 0x7f57b40ae970 2021-02-10T11:32:36.165 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.175 7f57b9d50700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57b41ae150 con 0x7f57b41ad130 2021-02-10T11:32:36.165 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.175 7f57b3fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f57b419fd50 0x7f57b41adb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:36.165 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.175 7f57b3fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f57b419fd50 0x7f57b41adb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:49066/0 (socket says 172.21.2.112:49066) 2021-02-10T11:32:36.165 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.175 7f57b3fff700 1 -- 172.21.2.112:0/2698559136 learned_addr learned my addr 172.21.2.112:0/2698559136 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:32:36.166 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.175 7f57b3fff700 1 -- 172.21.2.112:0/2698559136 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f57b41ad130 msgr2=0x7f57b419f110 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:32:36.166 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.175 7f57b3fff700 1 --2- 172.21.2.112:0/2698559136 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f57b41ad130 0x7f57b419f110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:36.166 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.175 7f57b3fff700 1 -- 172.21.2.112:0/2698559136 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f57b40ae970 msgr2=0x7f57b419f730 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:32:36.166 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.175 7f57b3fff700 1 --2- 172.21.2.112:0/2698559136 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f57b40ae970 0x7f57b419f730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:36.166 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.176 7f57b3fff700 1 -- 172.21.2.112:0/2698559136 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57b41ae370 con 0x7f57b419fd50 2021-02-10T11:32:36.166 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.176 7f57b3fff700 1 --2- 172.21.2.112:0/2698559136 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f57b419fd50 0x7f57b41adb30 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 rx=0x7f57ac001c10 tx=0x7f57ac00aca0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:36.166 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.176 7f579bfff700 1 -- 172.21.2.112:0/2698559136 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f57ac007ce0 con 0x7f57b419fd50 2021-02-10T11:32:36.166 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.176 7f579bfff700 1 -- 172.21.2.112:0/2698559136 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57ac0070d0 con 0x7f57b419fd50 2021-02-10T11:32:36.166 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.176 7f579bfff700 1 -- 172.21.2.112:0/2698559136 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f57ac004930 con 0x7f57b419fd50 2021-02-10T11:32:36.166 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.176 7f57b9d50700 1 -- 172.21.2.112:0/2698559136 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57b41ae370 con 0x7f57b419fd50 2021-02-10T11:32:36.166 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.176 7f57b9d50700 1 -- 172.21.2.112:0/2698559136 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57b41ae370 con 0x7f57b419fd50 2021-02-10T11:32:36.166 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.176 7f579bfff700 1 -- 172.21.2.112:0/2698559136 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f57ac0072b0 con 0x7f57b419fd50 2021-02-10T11:32:36.166 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.177 7f579bfff700 1 --2- 172.21.2.112:0/2698559136 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f57940186c0 0x7f579401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:36.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.177 7f579bfff700 1 -- 172.21.2.112:0/2698559136 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f57ac032bb0 con 0x7f57b419fd50 2021-02-10T11:32:36.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.177 7f57b37fe700 1 --2- 172.21.2.112:0/2698559136 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f57940186c0 0x7f579401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:36.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.178 7f57b37fe700 1 --2- 172.21.2.112:0/2698559136 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f57940186c0 0x7f579401abc0 crc :-1 s=READY pgs=573 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:36.194 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.207 7f57b9d50700 1 -- 172.21.2.112:0/2698559136 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57b407ed80 con 0x7f57b419fd50 2021-02-10T11:32:36.197 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.210 7f579bfff700 1 -- 172.21.2.112:0/2698559136 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f57ac007ce0 con 0x7f57b419fd50 2021-02-10T11:32:36.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.341 7f57b9d50700 1 -- 172.21.2.112:0/2698559136 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f57b407ed80 con 0x7f57b419fd50 2021-02-10T11:32:36.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.344 7f579bfff700 1 -- 172.21.2.112:0/2698559136 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v26) v1 ==== 184+0+0 (secure 0 0 0) 0x7f57ac0070b0 con 0x7f57b419fd50 2021-02-10T11:32:36.335 INFO:teuthology.orchestra.run.gibba012.stderr:updated caps for client.1 2021-02-10T11:32:36.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.350 7f57b9d50700 1 -- 172.21.2.112:0/2698559136 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f57940186c0 msgr2=0x7f579401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:36.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.350 7f57b9d50700 1 --2- 172.21.2.112:0/2698559136 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f57940186c0 0x7f579401abc0 crc :-1 s=READY pgs=573 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:36.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.350 7f57b9d50700 1 -- 172.21.2.112:0/2698559136 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f57b419fd50 msgr2=0x7f57b41adb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:36.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.350 7f57b9d50700 1 --2- 172.21.2.112:0/2698559136 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f57b419fd50 0x7f57b41adb30 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 rx=0x7f57ac001c10 tx=0x7f57ac00aca0).stop 2021-02-10T11:32:36.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.350 7f57b9d50700 1 -- 172.21.2.112:0/2698559136 shutdown_connections 2021-02-10T11:32:36.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.350 7f57b9d50700 1 --2- 172.21.2.112:0/2698559136 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f57940186c0 0x7f579401abc0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:36.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.350 7f57b9d50700 1 --2- 172.21.2.112:0/2698559136 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f57b40ae970 0x7f57b419f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:36.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.350 7f57b9d50700 1 --2- 172.21.2.112:0/2698559136 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f57b419fd50 0x7f57b41adb30 unknown :-1 s=CLOSED pgs=1131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:36.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.350 7f57b9d50700 1 --2- 172.21.2.112:0/2698559136 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f57b41ad130 0x7f57b419f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:36.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.350 7f57b9d50700 1 -- 172.21.2.112:0/2698559136 >> 172.21.2.112:0/2698559136 conn(0x7f57b408dde0 msgr2=0x7f57b41ab9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:36.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.350 7f57b9d50700 1 -- 172.21.2.112:0/2698559136 shutdown_connections 2021-02-10T11:32:36.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.350 7f57b9d50700 1 -- 172.21.2.112:0/2698559136 wait complete. 2021-02-10T11:32:36.346 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.2 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:32:36.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.779 7f957f439700 1 Processor -- start 2021-02-10T11:32:36.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.779 7f957f439700 1 -- start start 2021-02-10T11:32:36.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.780 7f957f439700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f95781b0470 0x7f95781b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:36.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.780 7f957f439700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f95780f20e0 0x7f95780f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:36.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.780 7f957f439700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95780f0b50 con 0x7f95780ef610 2021-02-10T11:32:36.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.780 7f957f439700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95780f0820 con 0x7f95781b0470 2021-02-10T11:32:36.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.780 7f957f439700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95780f24e0 con 0x7f95780f20e0 2021-02-10T11:32:36.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.780 7f957d255700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f95781b0470 0x7f95781b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:36.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.780 7f957d255700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f95781b0470 0x7f95781b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3300/0 says I am v2:172.21.2.112:51350/0 (socket says 172.21.2.112:51350) 2021-02-10T11:32:36.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.780 7f957ca54700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7f95780ef610 0x7f95780efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:42092/0 (socket says 172.21.2.112:42092) 2021-02-10T11:32:36.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.780 7f957d255700 1 -- 172.21.2.112:0/218211480 learned_addr learned my addr 172.21.2.112:0/218211480 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:32:36.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.780 7f957da56700 1 --2- 172.21.2.112:0/218211480 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f95780f20e0 0x7f95780f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:36.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.781 7f956ffff700 1 -- 172.21.2.112:0/218211480 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3215902162 0 0) 0x7f95680009b0 con 0x7f95780ef610 2021-02-10T11:32:36.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.781 7f957d255700 1 -- 172.21.2.112:0/218211480 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f95780f20e0 msgr2=0x7f95780f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:36.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.781 7f957d255700 1 --2- 172.21.2.112:0/218211480 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f95780f20e0 0x7f95780f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:36.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.781 7f957d255700 1 -- 172.21.2.112:0/218211480 >> v1:172.21.2.120:6789/0 conn(0x7f95780ef610 legacy=0x7f95780efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:36.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.781 7f957d255700 1 -- 172.21.2.112:0/218211480 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95780e52e0 con 0x7f95781b0470 2021-02-10T11:32:36.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.781 7f957da56700 1 --2- 172.21.2.112:0/218211480 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f95780f20e0 0x7f95780f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:32:36.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.781 7f957d255700 1 --2- 172.21.2.112:0/218211480 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f95781b0470 0x7f95781b0870 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7f9570006a80 tx=0x7f9570009f00).ready entity=mon.1 client_cookie=bb7118da8e6bcc1e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:36.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.781 7f956ffff700 1 -- 172.21.2.112:0/218211480 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f95700066c0 con 0x7f95781b0470 2021-02-10T11:32:36.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.781 7f956ffff700 1 -- 172.21.2.112:0/218211480 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f957000b5c0 con 0x7f95781b0470 2021-02-10T11:32:36.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.781 7f956ffff700 1 -- 172.21.2.112:0/218211480 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f957000ba20 con 0x7f95781b0470 2021-02-10T11:32:36.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.781 7f957f439700 1 -- 172.21.2.112:0/218211480 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f95781b0470 msgr2=0x7f95781b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:36.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.781 7f957f439700 1 --2- 172.21.2.112:0/218211480 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f95781b0470 0x7f95781b0870 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7f9570006a80 tx=0x7f9570009f00).stop 2021-02-10T11:32:36.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.782 7f957f439700 1 -- 172.21.2.112:0/218211480 shutdown_connections 2021-02-10T11:32:36.786 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.782 7f957f439700 1 --2- 172.21.2.112:0/218211480 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f95780f20e0 0x7f95780f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:36.786 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.782 7f957f439700 1 --2- 172.21.2.112:0/218211480 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f95781b0470 0x7f95781b0870 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:36.786 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.782 7f957f439700 1 -- 172.21.2.112:0/218211480 >> 172.21.2.112:0/218211480 conn(0x7f95780ce3e0 msgr2=0x7f95780e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:36.786 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.782 7f957f439700 1 -- 172.21.2.112:0/218211480 shutdown_connections 2021-02-10T11:32:36.786 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.782 7f957f439700 1 -- 172.21.2.112:0/218211480 wait complete. 2021-02-10T11:32:36.787 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.797 7f957f439700 1 Processor -- start 2021-02-10T11:32:36.787 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.797 7f957f439700 1 -- start start 2021-02-10T11:32:36.787 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.797 7f957f439700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f95781a0480 0x7f957808e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:36.788 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.797 7f957f439700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f957810a4b0 0x7f957808ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:36.788 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.797 7f957f439700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f957818ebf0 0x7f957818eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:36.788 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.797 7f957f439700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f957818f720 con 0x7f95781a0480 2021-02-10T11:32:36.788 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.797 7f957f439700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f957818f940 con 0x7f957818ebf0 2021-02-10T11:32:36.788 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.797 7f957f439700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f957818fb60 con 0x7f957810a4b0 2021-02-10T11:32:36.788 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.798 7f957d255700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f95781a0480 0x7f957808e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:36.788 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.798 7f957d255700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f95781a0480 0x7f957808e840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:49076/0 (socket says 172.21.2.112:49076) 2021-02-10T11:32:36.788 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.798 7f957d255700 1 -- 172.21.2.112:0/2962909507 learned_addr learned my addr 172.21.2.112:0/2962909507 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:32:36.788 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.798 7f957da56700 1 --2- 172.21.2.112:0/2962909507 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f957818ebf0 0x7f957818eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:36.788 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.798 7f957ca54700 1 --2- 172.21.2.112:0/2962909507 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f957810a4b0 0x7f957808ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:36.788 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.798 7f957d255700 1 -- 172.21.2.112:0/2962909507 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f957810a4b0 msgr2=0x7f957808ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:36.788 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.798 7f957d255700 1 --2- 172.21.2.112:0/2962909507 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f957810a4b0 0x7f957808ee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:36.788 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.798 7f957d255700 1 -- 172.21.2.112:0/2962909507 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f957818ebf0 msgr2=0x7f957818eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:36.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.798 7f957d255700 1 --2- 172.21.2.112:0/2962909507 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f957818ebf0 0x7f957818eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:36.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.798 7f957d255700 1 -- 172.21.2.112:0/2962909507 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f957818fd80 con 0x7f95781a0480 2021-02-10T11:32:36.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.798 7f957d255700 1 --2- 172.21.2.112:0/2962909507 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f95781a0480 0x7f957808e840 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 rx=0x7f957000b970 tx=0x7f957000bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:36.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.798 7f956dffb700 1 -- 172.21.2.112:0/2962909507 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9570000d30 con 0x7f95781a0480 2021-02-10T11:32:36.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.798 7f956dffb700 1 -- 172.21.2.112:0/2962909507 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9570006240 con 0x7f95781a0480 2021-02-10T11:32:36.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.798 7f956dffb700 1 -- 172.21.2.112:0/2962909507 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9570005660 con 0x7f95781a0480 2021-02-10T11:32:36.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.798 7f957f439700 1 -- 172.21.2.112:0/2962909507 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f957818fd80 con 0x7f95781a0480 2021-02-10T11:32:36.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.799 7f957f439700 1 -- 172.21.2.112:0/2962909507 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f957818fd80 con 0x7f95781a0480 2021-02-10T11:32:36.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.799 7f956dffb700 1 -- 172.21.2.112:0/2962909507 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9570006420 con 0x7f95781a0480 2021-02-10T11:32:36.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.799 7f956dffb700 1 --2- 172.21.2.112:0/2962909507 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f95580186c0 0x7f955801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:36.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.799 7f956dffb700 1 -- 172.21.2.112:0/2962909507 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f9570032790 con 0x7f95781a0480 2021-02-10T11:32:36.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.800 7f957ca54700 1 --2- 172.21.2.112:0/2962909507 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f95580186c0 0x7f955801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:36.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.800 7f957ca54700 1 --2- 172.21.2.112:0/2962909507 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f95580186c0 0x7f955801abc0 crc :-1 s=READY pgs=574 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:36.818 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.830 7f957f439700 1 -- 172.21.2.112:0/2962909507 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95781059f0 con 0x7f95781a0480 2021-02-10T11:32:36.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.832 7f956dffb700 1 -- 172.21.2.112:0/2962909507 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f957002cfa0 con 0x7f95781a0480 2021-02-10T11:32:36.950 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.963 7f957f439700 1 -- 172.21.2.112:0/2962909507 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f95780f0b50 con 0x7f95781a0480 2021-02-10T11:32:36.953 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.966 7f956dffb700 1 -- 172.21.2.112:0/2962909507 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.2 v27) v1 ==== 184+0+0 (secure 0 0 0) 0x7f957002cfa0 con 0x7f95781a0480 2021-02-10T11:32:36.953 INFO:teuthology.orchestra.run.gibba012.stderr:updated caps for client.2 2021-02-10T11:32:36.955 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.968 7f957f439700 1 -- 172.21.2.112:0/2962909507 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f95580186c0 msgr2=0x7f955801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:36.955 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.968 7f957f439700 1 --2- 172.21.2.112:0/2962909507 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f95580186c0 0x7f955801abc0 crc :-1 s=READY pgs=574 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:36.955 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.968 7f957f439700 1 -- 172.21.2.112:0/2962909507 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f95781a0480 msgr2=0x7f957808e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:36.955 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.968 7f957f439700 1 --2- 172.21.2.112:0/2962909507 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f95781a0480 0x7f957808e840 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 rx=0x7f957000b970 tx=0x7f957000bd10).stop 2021-02-10T11:32:36.955 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.968 7f957f439700 1 -- 172.21.2.112:0/2962909507 shutdown_connections 2021-02-10T11:32:36.956 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.968 7f957f439700 1 --2- 172.21.2.112:0/2962909507 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f95580186c0 0x7f955801abc0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:36.956 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.968 7f957f439700 1 --2- 172.21.2.112:0/2962909507 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f957818ebf0 0x7f957818eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:36.956 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.968 7f957f439700 1 --2- 172.21.2.112:0/2962909507 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f957810a4b0 0x7f957808ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:36.956 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.968 7f957f439700 1 --2- 172.21.2.112:0/2962909507 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f95781a0480 0x7f957808e840 unknown :-1 s=CLOSED pgs=1132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:36.956 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.968 7f957f439700 1 -- 172.21.2.112:0/2962909507 >> 172.21.2.112:0/2962909507 conn(0x7f957809cd10 msgr2=0x7f9578093a20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:36.956 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.968 7f957f439700 1 -- 172.21.2.112:0/2962909507 shutdown_connections 2021-02-10T11:32:36.956 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:36.968 7f957f439700 1 -- 172.21.2.112:0/2962909507 wait complete. 2021-02-10T11:32:36.964 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.3 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:32:37.402 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:37.399 7fdf497eb700 1 Processor -- start 2021-02-10T11:32:37.402 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:37.399 7fdf497eb700 1 -- start start 2021-02-10T11:32:37.403 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:37.400 7fdf497eb700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdf441b0470 0x7fdf441b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:37.403 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:37.400 7fdf497eb700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fdf440ef610 0x7fdf440efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:37.403 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:37.400 7fdf497eb700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fdf440f0030 0x7fdf440f0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:37.403 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:37.400 7fdf497eb700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf440e5ea0 con 0x7fdf440ef610 2021-02-10T11:32:37.403 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:37.400 7fdf497eb700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf440f0b50 con 0x7fdf440f0030 2021-02-10T11:32:37.403 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:37.400 7fdf497eb700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf440e52e0 con 0x7fdf441b0470 2021-02-10T11:32:37.403 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:37.400 7fdf42d73700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fdf440ef610 0x7fdf440efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:37.403 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:37.400 7fdf42d73700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fdf440ef610 0x7fdf440efa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:49088/0 (socket says 172.21.2.112:49088) 2021-02-10T11:32:37.403 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:37.400 7fdf42d73700 1 -- 172.21.2.112:0/4110819396 learned_addr learned my addr 172.21.2.112:0/4110819396 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:32:37.403 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:37.400 7fdf42d73700 1 -- 172.21.2.112:0/4110819396 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdf441b0470 msgr2=0x7fdf441b0870 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:32:37.403 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:37.400 7fdf42d73700 1 --2- 172.21.2.112:0/4110819396 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdf441b0470 0x7fdf441b0870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:37.403 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:37.400 7fdf42d73700 1 -- 172.21.2.112:0/4110819396 >> v2:172.21.2.120:3300/0 conn(0x7fdf440f0030 msgr2=0x7fdf440f0430 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:32:37.404 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:37.400 7fdf42d73700 1 --2- 172.21.2.112:0/4110819396 >> v2:172.21.2.120:3300/0 conn(0x7fdf440f0030 0x7fdf440f0430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:37.404 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:37.400 7fdf42d73700 1 -- 172.21.2.112:0/4110819396 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf440f10d0 con 0x7fdf440ef610 2021-02-10T11:32:37.404 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:37.400 7fdf42d73700 1 --2- 172.21.2.112:0/4110819396 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdf440ef610 0x7fdf440efa10 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 rx=0x7fdf2c006700 tx=0x7fdf2c009b80).ready entity=mon.0 client_cookie=b55db4436057aef server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:37.404 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:37.401 7fdf42572700 1 -- 172.21.2.112:0/4110819396 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdf2c00c0c0 con 0x7fdf440ef610 2021-02-10T11:32:37.404 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:37.401 7fdf42572700 1 -- 172.21.2.112:0/4110819396 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf2c00c440 con 0x7fdf440ef610 2021-02-10T11:32:37.404 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:37.401 7fdf42572700 1 -- 172.21.2.112:0/4110819396 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdf2c00c960 con 0x7fdf440ef610 2021-02-10T11:32:37.404 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:37.401 7fdf497eb700 1 -- 172.21.2.112:0/4110819396 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdf440ef610 msgr2=0x7fdf440efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:37.404 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:37.401 7fdf497eb700 1 --2- 172.21.2.112:0/4110819396 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdf440ef610 0x7fdf440efa10 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 rx=0x7fdf2c006700 tx=0x7fdf2c009b80).stop 2021-02-10T11:32:37.404 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:37.401 7fdf497eb700 1 -- 172.21.2.112:0/4110819396 shutdown_connections 2021-02-10T11:32:37.404 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:37.401 7fdf497eb700 1 --2- 172.21.2.112:0/4110819396 >> v2:172.21.2.120:3300/0 conn(0x7fdf440f0030 0x7fdf440f0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:37.404 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:37.401 7fdf497eb700 1 --2- 172.21.2.112:0/4110819396 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdf440ef610 0x7fdf440efa10 unknown :-1 s=CLOSED pgs=1133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:37.404 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:37.401 7fdf497eb700 1 --2- 172.21.2.112:0/4110819396 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdf441b0470 0x7fdf441b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:37.404 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:37.401 7fdf497eb700 1 -- 172.21.2.112:0/4110819396 >> 172.21.2.112:0/4110819396 conn(0x7fdf440ce3e0 msgr2=0x7fdf440e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:37.405 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:37.401 7fdf497eb700 1 -- 172.21.2.112:0/4110819396 shutdown_connections 2021-02-10T11:32:37.405 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:37.401 7fdf497eb700 1 -- 172.21.2.112:0/4110819396 wait complete. 2021-02-10T11:32:37.406 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:37.416 7fdf497eb700 1 Processor -- start 2021-02-10T11:32:37.406 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:37.416 7fdf497eb700 1 -- start start 2021-02-10T11:32:37.406 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:37.416 7fdf497eb700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdf441b6de0 0x7fdf441b63f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:37.406 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:37.416 7fdf497eb700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdf4418f3f0 0x7fdf4418f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:37.406 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:37.416 7fdf497eb700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fdf4418fe10 0x7fdf4418ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:37.406 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:37.416 7fdf497eb700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf4418d950 con 0x7fdf441b6de0 2021-02-10T11:32:37.407 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:37.416 7fdf497eb700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf441b71e0 con 0x7fdf4418fe10 2021-02-10T11:32:37.407 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:37.416 7fdf497eb700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf441b7400 con 0x7fdf4418f3f0 2021-02-10T11:32:37.407 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:37.416 7fdf43574700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdf441b6de0 0x7fdf441b63f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:37.407 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:37.416 7fdf43574700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdf441b6de0 0x7fdf441b63f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:49094/0 (socket says 172.21.2.112:49094) 2021-02-10T11:32:37.407 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:37.416 7fdf43574700 1 -- 172.21.2.112:0/3240510670 learned_addr learned my addr 172.21.2.112:0/3240510670 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:32:37.407 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:37.417 7fdf43574700 1 -- 172.21.2.112:0/3240510670 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdf4418f3f0 msgr2=0x7fdf4418f7f0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:32:37.407 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:37.417 7fdf43574700 1 --2- 172.21.2.112:0/3240510670 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdf4418f3f0 0x7fdf4418f7f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:37.407 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:37.417 7fdf43574700 1 -- 172.21.2.112:0/3240510670 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fdf4418fe10 msgr2=0x7fdf4418ddf0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:32:37.407 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:37.417 7fdf43574700 1 --2- 172.21.2.112:0/3240510670 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fdf4418fe10 0x7fdf4418ddf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:37.407 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:37.417 7fdf43574700 1 -- 172.21.2.112:0/3240510670 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf44190210 con 0x7fdf441b6de0 2021-02-10T11:32:37.407 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:37.417 7fdf43574700 1 --2- 172.21.2.112:0/3240510670 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdf441b6de0 0x7fdf441b63f0 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 rx=0x7fdf3400b5c0 tx=0x7fdf3400b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:37.407 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:37.417 7fdf3bfff700 1 -- 172.21.2.112:0/3240510670 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdf3400bd50 con 0x7fdf441b6de0 2021-02-10T11:32:37.407 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:37.417 7fdf3bfff700 1 -- 172.21.2.112:0/3240510670 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf34006030 con 0x7fdf441b6de0 2021-02-10T11:32:37.408 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:37.417 7fdf3bfff700 1 -- 172.21.2.112:0/3240510670 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdf340054d0 con 0x7fdf441b6de0 2021-02-10T11:32:37.408 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:37.417 7fdf497eb700 1 -- 172.21.2.112:0/3240510670 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf44190210 con 0x7fdf441b6de0 2021-02-10T11:32:37.408 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:37.417 7fdf497eb700 1 -- 172.21.2.112:0/3240510670 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf4418e700 con 0x7fdf441b6de0 2021-02-10T11:32:37.408 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:37.418 7fdf3bfff700 1 -- 172.21.2.112:0/3240510670 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fdf34006210 con 0x7fdf441b6de0 2021-02-10T11:32:37.408 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:37.418 7fdf3bfff700 1 --2- 172.21.2.112:0/3240510670 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fdf280186c0 0x7fdf2801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:37.408 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:37.418 7fdf3bfff700 1 -- 172.21.2.112:0/3240510670 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fdf340325d0 con 0x7fdf441b6de0 2021-02-10T11:32:37.408 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:37.418 7fdf42d73700 1 --2- 172.21.2.112:0/3240510670 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fdf280186c0 0x7fdf2801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:37.408 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:37.419 7fdf42d73700 1 --2- 172.21.2.112:0/3240510670 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fdf280186c0 0x7fdf2801abc0 crc :-1 s=READY pgs=575 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:37.437 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:37.449 7fdf497eb700 1 -- 172.21.2.112:0/3240510670 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf4418e700 con 0x7fdf441b6de0 2021-02-10T11:32:37.439 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:37.452 7fdf3bfff700 1 -- 172.21.2.112:0/3240510670 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fdf3401c060 con 0x7fdf441b6de0 2021-02-10T11:32:37.570 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:37.583 7fdf497eb700 1 -- 172.21.2.112:0/3240510670 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fdf4418e700 con 0x7fdf441b6de0 2021-02-10T11:32:37.574 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:37.587 7fdf3bfff700 1 -- 172.21.2.112:0/3240510670 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.3 v28) v1 ==== 184+0+0 (secure 0 0 0) 0x7fdf3401c060 con 0x7fdf441b6de0 2021-02-10T11:32:37.577 INFO:teuthology.orchestra.run.gibba012.stderr:updated caps for client.3 2021-02-10T11:32:37.579 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:37.592 7fdf497eb700 1 -- 172.21.2.112:0/3240510670 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fdf280186c0 msgr2=0x7fdf2801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:37.579 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:37.592 7fdf497eb700 1 --2- 172.21.2.112:0/3240510670 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fdf280186c0 0x7fdf2801abc0 crc :-1 s=READY pgs=575 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:37.579 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:37.592 7fdf497eb700 1 -- 172.21.2.112:0/3240510670 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdf441b6de0 msgr2=0x7fdf441b63f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:37.580 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:37.592 7fdf497eb700 1 --2- 172.21.2.112:0/3240510670 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdf441b6de0 0x7fdf441b63f0 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 rx=0x7fdf3400b5c0 tx=0x7fdf3400b960).stop 2021-02-10T11:32:37.580 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:37.592 7fdf497eb700 1 -- 172.21.2.112:0/3240510670 shutdown_connections 2021-02-10T11:32:37.580 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:37.592 7fdf497eb700 1 --2- 172.21.2.112:0/3240510670 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fdf280186c0 0x7fdf2801abc0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:37.580 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:37.592 7fdf497eb700 1 --2- 172.21.2.112:0/3240510670 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fdf4418fe10 0x7fdf4418ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:37.580 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:37.592 7fdf497eb700 1 --2- 172.21.2.112:0/3240510670 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdf4418f3f0 0x7fdf4418f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:37.580 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:37.592 7fdf497eb700 1 --2- 172.21.2.112:0/3240510670 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdf441b6de0 0x7fdf441b63f0 unknown :-1 s=CLOSED pgs=1134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:37.580 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:37.592 7fdf497eb700 1 -- 172.21.2.112:0/3240510670 >> 172.21.2.112:0/3240510670 conn(0x7fdf441a37a0 msgr2=0x7fdf4418c230 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:37.581 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:37.593 7fdf497eb700 1 -- 172.21.2.112:0/3240510670 shutdown_connections 2021-02-10T11:32:37.581 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:37.593 7fdf497eb700 1 -- 172.21.2.112:0/3240510670 wait complete. 2021-02-10T11:32:37.588 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:32:38.015 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:38.012 7fd4d7d7a700 1 Processor -- start 2021-02-10T11:32:38.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:38.012 7fd4d7d7a700 1 -- start start 2021-02-10T11:32:38.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:38.012 7fd4d7d7a700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fd4d00e5070 0x7fd4d01b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:38.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:38.012 7fd4d7d7a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd4d01b3e40 0x7fd4d007d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:38.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:38.012 7fd4d7d7a700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4d007ed80 con 0x7fd4d00e5470 2021-02-10T11:32:38.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:38.012 7fd4d7d7a700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4d0080310 con 0x7fd4d00e5070 2021-02-10T11:32:38.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:38.012 7fd4d7d7a700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4d0080500 con 0x7fd4d01b3e40 2021-02-10T11:32:38.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:38.013 7fd4d5b96700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fd4d00e5070 0x7fd4d01b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:38.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:38.013 7fd4d5b96700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fd4d00e5070 0x7fd4d01b3820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:49098/0 (socket says 172.21.2.112:49098) 2021-02-10T11:32:38.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:38.013 7fd4d5b96700 1 -- 172.21.2.112:0/435750774 learned_addr learned my addr 172.21.2.112:0/435750774 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:32:38.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:38.013 7fd4d5b96700 1 -- 172.21.2.112:0/435750774 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd4d01b3e40 msgr2=0x7fd4d007d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:38.017 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:38.013 7fd4d5b96700 1 --2- 172.21.2.112:0/435750774 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd4d01b3e40 0x7fd4d007d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:38.017 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:38.013 7fd4d5b96700 1 -- 172.21.2.112:0/435750774 >> v1:172.21.2.120:6789/0 conn(0x7fd4d00e5470 legacy=0x7fd4d007dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:38.017 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:38.013 7fd4d5b96700 1 -- 172.21.2.112:0/435750774 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4d00806f0 con 0x7fd4d00e5070 2021-02-10T11:32:38.017 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:38.013 7fd4d5b96700 1 --2- 172.21.2.112:0/435750774 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd4d00e5070 0x7fd4d01b3820 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 rx=0x7fd4c4006700 tx=0x7fd4c4009b80).ready entity=mon.0 client_cookie=10d39ab92557b763 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:38.017 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:38.013 7fd4d4b94700 1 -- 172.21.2.112:0/435750774 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd4c400c0c0 con 0x7fd4d00e5070 2021-02-10T11:32:38.017 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:38.013 7fd4d4b94700 1 -- 172.21.2.112:0/435750774 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4c400c400 con 0x7fd4d00e5070 2021-02-10T11:32:38.017 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:38.013 7fd4d4b94700 1 -- 172.21.2.112:0/435750774 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd4c400c860 con 0x7fd4d00e5070 2021-02-10T11:32:38.017 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:38.013 7fd4d7d7a700 1 -- 172.21.2.112:0/435750774 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd4d00e5070 msgr2=0x7fd4d01b3820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:38.017 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:38.013 7fd4d7d7a700 1 --2- 172.21.2.112:0/435750774 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd4d00e5070 0x7fd4d01b3820 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 rx=0x7fd4c4006700 tx=0x7fd4c4009b80).stop 2021-02-10T11:32:38.017 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:38.014 7fd4d7d7a700 1 -- 172.21.2.112:0/435750774 shutdown_connections 2021-02-10T11:32:38.017 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:38.014 7fd4d7d7a700 1 --2- 172.21.2.112:0/435750774 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd4d01b3e40 0x7fd4d007d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:38.018 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:38.014 7fd4d7d7a700 1 --2- 172.21.2.112:0/435750774 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd4d00e5070 0x7fd4d01b3820 unknown :-1 s=CLOSED pgs=1135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:38.018 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:38.014 7fd4d7d7a700 1 -- 172.21.2.112:0/435750774 >> 172.21.2.112:0/435750774 conn(0x7fd4d00ce690 msgr2=0x7fd4d00e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:38.018 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:38.014 7fd4d7d7a700 1 -- 172.21.2.112:0/435750774 shutdown_connections 2021-02-10T11:32:38.018 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:38.014 7fd4d7d7a700 1 -- 172.21.2.112:0/435750774 wait complete. 2021-02-10T11:32:38.018 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:38.028 7fd4d7d7a700 1 Processor -- start 2021-02-10T11:32:38.018 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:38.028 7fd4d7d7a700 1 -- start start 2021-02-10T11:32:38.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:38.029 7fd4d7d7a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd4d01b19d0 0x7fd4d008cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:38.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:38.029 7fd4d7d7a700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd4d01b0d00 0x7fd4d01b1100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:38.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:38.029 7fd4d7d7a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd4d01a2b50 0x7fd4d01a2f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:38.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:38.029 7fd4d7d7a700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4d01b1720 con 0x7fd4d01b19d0 2021-02-10T11:32:38.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:38.029 7fd4d7d7a700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4d008c470 con 0x7fd4d01b0d00 2021-02-10T11:32:38.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:38.029 7fd4d7d7a700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4d01b1f50 con 0x7fd4d01a2b50 2021-02-10T11:32:38.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:38.029 7fd4d5b96700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd4d01b19d0 0x7fd4d008cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:38.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:38.029 7fd4d5b96700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd4d01b19d0 0x7fd4d008cd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:49104/0 (socket says 172.21.2.112:49104) 2021-02-10T11:32:38.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:38.029 7fd4d5b96700 1 -- 172.21.2.112:0/873017810 learned_addr learned my addr 172.21.2.112:0/873017810 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:32:38.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:38.029 7fd4d5b96700 1 -- 172.21.2.112:0/873017810 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd4d01a2b50 msgr2=0x7fd4d01a2f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:38.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:38.029 7fd4d5b96700 1 --2- 172.21.2.112:0/873017810 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd4d01a2b50 0x7fd4d01a2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:38.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:38.029 7fd4d5b96700 1 -- 172.21.2.112:0/873017810 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd4d01b0d00 msgr2=0x7fd4d01b1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:38.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:38.029 7fd4d5b96700 1 --2- 172.21.2.112:0/873017810 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd4d01b0d00 0x7fd4d01b1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:38.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:38.029 7fd4d5b96700 1 -- 172.21.2.112:0/873017810 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4d01a3570 con 0x7fd4d01b19d0 2021-02-10T11:32:38.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:38.030 7fd4d5b96700 1 --2- 172.21.2.112:0/873017810 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd4d01b19d0 0x7fd4d008cd90 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 rx=0x7fd4c40066a0 tx=0x7fd4c400b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:38.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:38.030 7fd4c27fc700 1 -- 172.21.2.112:0/873017810 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd4c400bd80 con 0x7fd4d01b19d0 2021-02-10T11:32:38.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:38.030 7fd4c27fc700 1 -- 172.21.2.112:0/873017810 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4c40052e0 con 0x7fd4d01b19d0 2021-02-10T11:32:38.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:38.030 7fd4c27fc700 1 -- 172.21.2.112:0/873017810 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd4c4005740 con 0x7fd4d01b19d0 2021-02-10T11:32:38.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:38.030 7fd4d7d7a700 1 -- 172.21.2.112:0/873017810 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4d01a3570 con 0x7fd4d01b19d0 2021-02-10T11:32:38.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:38.030 7fd4d7d7a700 1 -- 172.21.2.112:0/873017810 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4d01a3570 con 0x7fd4d01b19d0 2021-02-10T11:32:38.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:38.030 7fd4c27fc700 1 -- 172.21.2.112:0/873017810 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd4c4006240 con 0x7fd4d01b19d0 2021-02-10T11:32:38.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:38.031 7fd4c27fc700 1 --2- 172.21.2.112:0/873017810 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd4b80186c0 0x7fd4b801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:38.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:38.031 7fd4c27fc700 1 -- 172.21.2.112:0/873017810 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fd4c4032560 con 0x7fd4d01b19d0 2021-02-10T11:32:38.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:38.031 7fd4d5395700 1 --2- 172.21.2.112:0/873017810 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd4b80186c0 0x7fd4b801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:38.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:38.031 7fd4d5395700 1 --2- 172.21.2.112:0/873017810 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd4b80186c0 0x7fd4b801abc0 crc :-1 s=READY pgs=576 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:38.049 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:38.062 7fd4d7d7a700 1 -- 172.21.2.112:0/873017810 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4d0083dd0 con 0x7fd4d01b19d0 2021-02-10T11:32:38.052 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:38.065 7fd4c27fc700 1 -- 172.21.2.112:0/873017810 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd4c4006030 con 0x7fd4d01b19d0 2021-02-10T11:32:38.181 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:38.194 7fd4d7d7a700 1 -- 172.21.2.112:0/873017810 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd4d0083dd0 con 0x7fd4d01b19d0 2021-02-10T11:32:38.182 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:38.194 7fd4c27fc700 1 -- 172.21.2.112:0/873017810 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+2093 (secure 0 0 0) 0x7fd4c400bd80 con 0x7fd4d01b19d0 2021-02-10T11:32:38.182 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 30 2021-02-10T11:32:38.182 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:32:38.182 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":30,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7536,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/340651080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":340651080},{"type":"v1","addr":"172.21.2.112:6833","nonce":340651080}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":26}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:32:31.869812","modified":"2021-02-10 11:32:33.879671","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6409},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6409":{"gid":6409,"name":"b","rank":0,"incarnation":27,"state":"up:active","state_seq":3,"addr":"172.21.2.120:6835/210055046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":210055046},{"type":"v1","addr":"172.21.2.120:6835","nonce":210055046}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":4}]} 2021-02-10T11:32:38.184 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:38.197 7fd4d7d7a700 1 -- 172.21.2.112:0/873017810 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd4b80186c0 msgr2=0x7fd4b801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:38.184 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:38.197 7fd4d7d7a700 1 --2- 172.21.2.112:0/873017810 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd4b80186c0 0x7fd4b801abc0 crc :-1 s=READY pgs=576 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:38.184 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:38.197 7fd4d7d7a700 1 -- 172.21.2.112:0/873017810 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd4d01b19d0 msgr2=0x7fd4d008cd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:38.184 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:38.197 7fd4d7d7a700 1 --2- 172.21.2.112:0/873017810 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd4d01b19d0 0x7fd4d008cd90 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 rx=0x7fd4c40066a0 tx=0x7fd4c400b990).stop 2021-02-10T11:32:38.184 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:38.197 7fd4d7d7a700 1 -- 172.21.2.112:0/873017810 shutdown_connections 2021-02-10T11:32:38.185 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:38.197 7fd4d7d7a700 1 --2- 172.21.2.112:0/873017810 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd4b80186c0 0x7fd4b801abc0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:38.185 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:38.197 7fd4d7d7a700 1 --2- 172.21.2.112:0/873017810 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd4d01b0d00 0x7fd4d01b1100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:38.185 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:38.197 7fd4d7d7a700 1 --2- 172.21.2.112:0/873017810 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd4d01a2b50 0x7fd4d01a2f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:38.185 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:38.197 7fd4d7d7a700 1 --2- 172.21.2.112:0/873017810 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd4d01b19d0 0x7fd4d008cd90 unknown :-1 s=CLOSED pgs=1136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:38.185 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:38.197 7fd4d7d7a700 1 -- 172.21.2.112:0/873017810 >> 172.21.2.112:0/873017810 conn(0x7fd4d0093720 msgr2=0x7fd4d0084b80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:38.185 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:38.197 7fd4d7d7a700 1 -- 172.21.2.112:0/873017810 shutdown_connections 2021-02-10T11:32:38.185 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:38.197 7fd4d7d7a700 1 -- 172.21.2.112:0/873017810 wait complete. 2021-02-10T11:32:38.394 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:32:31.869812', 'modified': '2021-02-10 11:32:33.879671', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6409}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6409': {'gid': 6409, 'name': 'b', 'rank': 0, 'incarnation': 27, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.120:6835/210055046', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.120:6834', 'nonce': 210055046}, {'type': 'v1', 'addr': '172.21.2.120:6835', 'nonce': 210055046}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [9], 'metadata_pool': 8, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:32:38.395 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:32:38.395 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:32:38.794 INFO:teuthology.orchestra.run.gibba020.stdout:{ 2021-02-10T11:32:38.794 INFO:teuthology.orchestra.run.gibba020.stdout: "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", 2021-02-10T11:32:38.794 INFO:teuthology.orchestra.run.gibba020.stdout: "whoami": 0, 2021-02-10T11:32:38.794 INFO:teuthology.orchestra.run.gibba020.stdout: "id": 6409, 2021-02-10T11:32:38.794 INFO:teuthology.orchestra.run.gibba020.stdout: "want_state": "up:active", 2021-02-10T11:32:38.794 INFO:teuthology.orchestra.run.gibba020.stdout: "state": "up:active", 2021-02-10T11:32:38.794 INFO:teuthology.orchestra.run.gibba020.stdout: "rank_uptime": 6.9355699819999996, 2021-02-10T11:32:38.794 INFO:teuthology.orchestra.run.gibba020.stdout: "mdsmap_epoch": 30, 2021-02-10T11:32:38.794 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch": 46, 2021-02-10T11:32:38.794 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch_barrier": 46, 2021-02-10T11:32:38.794 INFO:teuthology.orchestra.run.gibba020.stdout: "uptime": 12.565527267 2021-02-10T11:32:38.795 INFO:teuthology.orchestra.run.gibba020.stdout:} 2021-02-10T11:32:38.798 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", "whoami": 0, "id": 6409, "want_state": "up:active", "state": "up:active", "rank_uptime": 6.9355699819999996, "mdsmap_epoch": 30, "osdmap_epoch": 46, "osdmap_epoch_barrier": 46, "uptime": 12.565527267 } 2021-02-10T11:32:38.798 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:32:39.224 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.220 7f9fc34a9700 1 Processor -- start 2021-02-10T11:32:39.224 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.220 7f9fc34a9700 1 -- start start 2021-02-10T11:32:39.224 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.220 7f9fc34a9700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9fbc0d17f0 0x7f9fbc1b31e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:39.224 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.220 7f9fc34a9700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f9fbc0d1bf0 0x7f9fbc1b3800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:39.224 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.220 7f9fc34a9700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fbc0f0b20 con 0x7f9fbc1b3e20 2021-02-10T11:32:39.224 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.220 7f9fc34a9700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fbc0e5eb0 con 0x7f9fbc0d1bf0 2021-02-10T11:32:39.224 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.220 7f9fc34a9700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fbc0f0630 con 0x7f9fbc0d17f0 2021-02-10T11:32:39.224 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.220 7f9fc1ac6700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f9fbc1b3e20 0x7f9fbc0ceef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:48572/0 (socket says 172.21.2.112:48572) 2021-02-10T11:32:39.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.220 7f9fc1ac6700 1 -- 172.21.2.112:0/2796014887 learned_addr learned my addr 172.21.2.112:0/2796014887 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:32:39.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.221 7f9fc0ac4700 1 --2- 172.21.2.112:0/2796014887 >> v2:172.21.2.120:3300/0 conn(0x7f9fbc0d1bf0 0x7f9fbc1b3800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:39.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.221 7f9fc12c5700 1 --2- 172.21.2.112:0/2796014887 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9fbc0d17f0 0x7f9fbc1b31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:39.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.221 7f9faffff700 1 -- 172.21.2.112:0/2796014887 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 861379903 0 0) 0x7f9fb8001180 con 0x7f9fbc1b3e20 2021-02-10T11:32:39.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.221 7f9faffff700 1 -- 172.21.2.112:0/2796014887 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fa8002fe0 con 0x7f9fbc1b3e20 2021-02-10T11:32:39.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.221 7f9fc0ac4700 1 -- 172.21.2.112:0/2796014887 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9fbc0d17f0 msgr2=0x7f9fbc1b31e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:39.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.221 7f9fc0ac4700 1 --2- 172.21.2.112:0/2796014887 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9fbc0d17f0 0x7f9fbc1b31e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:39.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.221 7f9fc0ac4700 1 -- 172.21.2.112:0/2796014887 >> v1:172.21.2.112:6789/0 conn(0x7f9fbc1b3e20 legacy=0x7f9fbc0ceef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:39.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.221 7f9fc0ac4700 1 -- 172.21.2.112:0/2796014887 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fbc0f0820 con 0x7f9fbc0d1bf0 2021-02-10T11:32:39.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.221 7f9fc12c5700 1 --2- 172.21.2.112:0/2796014887 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9fbc0d17f0 0x7f9fbc1b31e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:32:39.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.221 7f9fc0ac4700 1 --2- 172.21.2.112:0/2796014887 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9fbc0d1bf0 0x7f9fbc1b3800 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 rx=0x7f9fb0006650 tx=0x7f9fb0009a80).ready entity=mon.1 client_cookie=7539feab90172090 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:39.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.222 7f9faffff700 1 -- 172.21.2.112:0/2796014887 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9fb000c0c0 con 0x7f9fbc0d1bf0 2021-02-10T11:32:39.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.222 7f9faffff700 1 -- 172.21.2.112:0/2796014887 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fb000c340 con 0x7f9fbc0d1bf0 2021-02-10T11:32:39.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.222 7f9faffff700 1 -- 172.21.2.112:0/2796014887 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9fb000c860 con 0x7f9fbc0d1bf0 2021-02-10T11:32:39.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.222 7f9fc34a9700 1 -- 172.21.2.112:0/2796014887 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9fbc0d1bf0 msgr2=0x7f9fbc1b3800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:39.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.222 7f9fc34a9700 1 --2- 172.21.2.112:0/2796014887 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9fbc0d1bf0 0x7f9fbc1b3800 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 rx=0x7f9fb0006650 tx=0x7f9fb0009a80).stop 2021-02-10T11:32:39.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.222 7f9fc34a9700 1 -- 172.21.2.112:0/2796014887 shutdown_connections 2021-02-10T11:32:39.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.222 7f9fc34a9700 1 --2- 172.21.2.112:0/2796014887 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9fbc0d1bf0 0x7f9fbc1b3800 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:39.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.222 7f9fc34a9700 1 --2- 172.21.2.112:0/2796014887 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9fbc0d17f0 0x7f9fbc1b31e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:39.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.222 7f9fc34a9700 1 -- 172.21.2.112:0/2796014887 >> 172.21.2.112:0/2796014887 conn(0x7f9fbc0fde40 msgr2=0x7f9fbc0ce390 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:39.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.223 7f9fc34a9700 1 -- 172.21.2.112:0/2796014887 shutdown_connections 2021-02-10T11:32:39.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.223 7f9fc34a9700 1 -- 172.21.2.112:0/2796014887 wait complete. 2021-02-10T11:32:39.228 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.237 7f9fc34a9700 1 Processor -- start 2021-02-10T11:32:39.228 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.237 7f9fc34a9700 1 -- start start 2021-02-10T11:32:39.228 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.237 7f9fc34a9700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9fbc091530 0x7f9fbc09bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:39.228 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.237 7f9fc34a9700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9fbc090030 0x7f9fbc090430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:39.228 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.237 7f9fc34a9700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9fbc090a50 0x7f9fbc0b6920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:39.228 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.238 7f9fc34a9700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fbc090e50 con 0x7f9fbc091530 2021-02-10T11:32:39.228 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.238 7f9fc34a9700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fbc091040 con 0x7f9fbc090a50 2021-02-10T11:32:39.228 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.238 7f9fc34a9700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fbc091260 con 0x7f9fbc090030 2021-02-10T11:32:39.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.238 7f9fc12c5700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9fbc091530 0x7f9fbc09bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:39.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.238 7f9fc12c5700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9fbc091530 0x7f9fbc09bdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:49118/0 (socket says 172.21.2.112:49118) 2021-02-10T11:32:39.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.238 7f9fc12c5700 1 -- 172.21.2.112:0/4290009196 learned_addr learned my addr 172.21.2.112:0/4290009196 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:32:39.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.238 7f9fc1ac6700 1 --2- 172.21.2.112:0/4290009196 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9fbc090a50 0x7f9fbc0b6920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:39.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.238 7f9fc0ac4700 1 --2- 172.21.2.112:0/4290009196 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9fbc090030 0x7f9fbc090430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:39.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.238 7f9fc12c5700 1 -- 172.21.2.112:0/4290009196 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9fbc090030 msgr2=0x7f9fbc090430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:39.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.238 7f9fc12c5700 1 --2- 172.21.2.112:0/4290009196 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9fbc090030 0x7f9fbc090430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:39.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.238 7f9fc12c5700 1 -- 172.21.2.112:0/4290009196 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9fbc090a50 msgr2=0x7f9fbc0b6920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:39.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.238 7f9fc12c5700 1 --2- 172.21.2.112:0/4290009196 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9fbc090a50 0x7f9fbc0b6920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:39.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.238 7f9fc12c5700 1 -- 172.21.2.112:0/4290009196 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fbc0b6f40 con 0x7f9fbc091530 2021-02-10T11:32:39.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.238 7f9fc12c5700 1 --2- 172.21.2.112:0/4290009196 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9fbc091530 0x7f9fbc09bdc0 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 rx=0x7f9fb400b970 tx=0x7f9fb400bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:39.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.238 7f9fadffb700 1 -- 172.21.2.112:0/4290009196 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9fb4000d30 con 0x7f9fbc091530 2021-02-10T11:32:39.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.238 7f9fadffb700 1 -- 172.21.2.112:0/4290009196 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fb4006240 con 0x7f9fbc091530 2021-02-10T11:32:39.230 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.238 7f9fadffb700 1 -- 172.21.2.112:0/4290009196 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9fb4005660 con 0x7f9fbc091530 2021-02-10T11:32:39.230 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.238 7f9fc34a9700 1 -- 172.21.2.112:0/4290009196 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fbc0b6f40 con 0x7f9fbc091530 2021-02-10T11:32:39.230 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.238 7f9fc34a9700 1 -- 172.21.2.112:0/4290009196 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fbc0b74d0 con 0x7f9fbc091530 2021-02-10T11:32:39.230 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.239 7f9fadffb700 1 -- 172.21.2.112:0/4290009196 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9fb4006420 con 0x7f9fbc091530 2021-02-10T11:32:39.230 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.239 7f9fadffb700 1 --2- 172.21.2.112:0/4290009196 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9f9c0186d0 0x7f9f9c01abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:39.230 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.239 7f9fadffb700 1 -- 172.21.2.112:0/4290009196 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f9fb40329e0 con 0x7f9fbc091530 2021-02-10T11:32:39.230 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.239 7f9fc0ac4700 1 --2- 172.21.2.112:0/4290009196 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9f9c0186d0 0x7f9f9c01abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:39.230 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.240 7f9fc0ac4700 1 --2- 172.21.2.112:0/4290009196 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9f9c0186d0 0x7f9f9c01abd0 crc :-1 s=READY pgs=577 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:39.258 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.270 7f9fc34a9700 1 -- 172.21.2.112:0/4290009196 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fbc1036d0 con 0x7f9fbc091530 2021-02-10T11:32:39.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.273 7f9fadffb700 1 -- 172.21.2.112:0/4290009196 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9fb4000cc0 con 0x7f9fbc091530 2021-02-10T11:32:39.385 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.397 7f9fc34a9700 1 -- 172.21.2.112:0/4290009196 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9fbc0f0b20 con 0x7f9fbc091530 2021-02-10T11:32:39.385 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.398 7f9fadffb700 1 -- 172.21.2.112:0/4290009196 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v46) v1 ==== 74+0+13993 (secure 0 0 0) 0x7f9fb4006240 con 0x7f9fbc091530 2021-02-10T11:32:39.386 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:32:39.386 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":46,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:32:32.871643","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:32:29.843136","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":9,"pool_name":"cephfs_data","create_time":"2021-02-10 11:32:30.914946","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:32:39.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.400 7f9fc34a9700 1 -- 172.21.2.112:0/4290009196 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9f9c0186d0 msgr2=0x7f9f9c01abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:39.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.400 7f9fc34a9700 1 --2- 172.21.2.112:0/4290009196 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9f9c0186d0 0x7f9f9c01abd0 crc :-1 s=READY pgs=577 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:39.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.400 7f9fc34a9700 1 -- 172.21.2.112:0/4290009196 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9fbc091530 msgr2=0x7f9fbc09bdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:39.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.400 7f9fc34a9700 1 --2- 172.21.2.112:0/4290009196 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9fbc091530 0x7f9fbc09bdc0 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 rx=0x7f9fb400b970 tx=0x7f9fb400bd10).stop 2021-02-10T11:32:39.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.400 7f9fc34a9700 1 -- 172.21.2.112:0/4290009196 shutdown_connections 2021-02-10T11:32:39.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.400 7f9fc34a9700 1 --2- 172.21.2.112:0/4290009196 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9f9c0186d0 0x7f9f9c01abd0 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:39.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.400 7f9fc34a9700 1 --2- 172.21.2.112:0/4290009196 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9fbc090a50 0x7f9fbc0b6920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:39.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.400 7f9fc34a9700 1 --2- 172.21.2.112:0/4290009196 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9fbc090030 0x7f9fbc090430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:39.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.400 7f9fc34a9700 1 --2- 172.21.2.112:0/4290009196 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9fbc091530 0x7f9fbc09bdc0 unknown :-1 s=CLOSED pgs=1138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:39.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.400 7f9fc34a9700 1 -- 172.21.2.112:0/4290009196 >> 172.21.2.112:0/4290009196 conn(0x7f9fbc0b5e30 msgr2=0x7f9fbc0c3c90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:39.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.400 7f9fc34a9700 1 -- 172.21.2.112:0/4290009196 shutdown_connections 2021-02-10T11:32:39.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.400 7f9fc34a9700 1 -- 172.21.2.112:0/4290009196 wait complete. 2021-02-10T11:32:39.396 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:13:57.508758', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:32:29.843136', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:32:30.914946', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '46', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:32:39.396 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:32:39.827 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.823 7ff9585ca700 1 Processor -- start 2021-02-10T11:32:39.827 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.823 7ff9585ca700 1 -- start start 2021-02-10T11:32:39.827 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.823 7ff9585ca700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7ff9501b0470 0x7ff9501b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:39.827 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.823 7ff9585ca700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff9500ef610 0x7ff9500efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:39.827 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.823 7ff9585ca700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7ff9500f0030 0x7ff9500f0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:39.827 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.823 7ff9585ca700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9500e5ea0 con 0x7ff9501b0470 2021-02-10T11:32:39.827 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.823 7ff9585ca700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9500f0b50 con 0x7ff9500f0030 2021-02-10T11:32:39.827 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.823 7ff9585ca700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9500e52e0 con 0x7ff9500ef610 2021-02-10T11:32:39.827 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.824 7ff9563e6700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7ff9501b0470 0x7ff9501b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:39.827 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.824 7ff9563e6700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7ff9501b0470 0x7ff9501b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:49124/0 (socket says 172.21.2.112:49124) 2021-02-10T11:32:39.827 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.824 7ff9563e6700 1 -- 172.21.2.112:0/3942889454 learned_addr learned my addr 172.21.2.112:0/3942889454 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:32:39.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.824 7ff956be7700 1 --2- 172.21.2.112:0/3942889454 >> v2:172.21.2.120:3300/0 conn(0x7ff9500f0030 0x7ff9500f0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:39.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.824 7ff955be5700 1 --2- 172.21.2.112:0/3942889454 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff9500ef610 0x7ff9500efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:39.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.824 7ff9563e6700 1 -- 172.21.2.112:0/3942889454 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff9500ef610 msgr2=0x7ff9500efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:39.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.824 7ff9563e6700 1 --2- 172.21.2.112:0/3942889454 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff9500ef610 0x7ff9500efa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:39.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.824 7ff9563e6700 1 -- 172.21.2.112:0/3942889454 >> v2:172.21.2.120:3300/0 conn(0x7ff9500f0030 msgr2=0x7ff9500f0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:39.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.824 7ff9563e6700 1 --2- 172.21.2.112:0/3942889454 >> v2:172.21.2.120:3300/0 conn(0x7ff9500f0030 0x7ff9500f0430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:39.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.824 7ff9563e6700 1 -- 172.21.2.112:0/3942889454 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9500f10d0 con 0x7ff9501b0470 2021-02-10T11:32:39.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.824 7ff955be5700 1 --2- 172.21.2.112:0/3942889454 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff9500ef610 0x7ff9500efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:32:39.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.824 7ff9563e6700 1 --2- 172.21.2.112:0/3942889454 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff9501b0470 0x7ff9501b0870 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 rx=0x7ff944006700 tx=0x7ff944009b80).ready entity=mon.0 client_cookie=d6cf9b28546c000a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:39.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.824 7ff9553e4700 1 -- 172.21.2.112:0/3942889454 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff94400c0c0 con 0x7ff9501b0470 2021-02-10T11:32:39.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.824 7ff9553e4700 1 -- 172.21.2.112:0/3942889454 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff94400c440 con 0x7ff9501b0470 2021-02-10T11:32:39.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.824 7ff9553e4700 1 -- 172.21.2.112:0/3942889454 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff94400c920 con 0x7ff9501b0470 2021-02-10T11:32:39.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.824 7ff9585ca700 1 -- 172.21.2.112:0/3942889454 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff9501b0470 msgr2=0x7ff9501b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:39.829 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.824 7ff9585ca700 1 --2- 172.21.2.112:0/3942889454 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff9501b0470 0x7ff9501b0870 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 rx=0x7ff944006700 tx=0x7ff944009b80).stop 2021-02-10T11:32:39.829 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.825 7ff9585ca700 1 -- 172.21.2.112:0/3942889454 shutdown_connections 2021-02-10T11:32:39.829 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.825 7ff9585ca700 1 --2- 172.21.2.112:0/3942889454 >> v2:172.21.2.120:3300/0 conn(0x7ff9500f0030 0x7ff9500f0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:39.829 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.825 7ff9585ca700 1 --2- 172.21.2.112:0/3942889454 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff9500ef610 0x7ff9500efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:39.829 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.825 7ff9585ca700 1 --2- 172.21.2.112:0/3942889454 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff9501b0470 0x7ff9501b0870 unknown :-1 s=CLOSED pgs=1139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:39.829 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.825 7ff9585ca700 1 -- 172.21.2.112:0/3942889454 >> 172.21.2.112:0/3942889454 conn(0x7ff9500ce3e0 msgr2=0x7ff9500e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:39.829 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.825 7ff9585ca700 1 -- 172.21.2.112:0/3942889454 shutdown_connections 2021-02-10T11:32:39.829 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.825 7ff9585ca700 1 -- 172.21.2.112:0/3942889454 wait complete. 2021-02-10T11:32:39.831 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.840 7ff9585ca700 1 Processor -- start 2021-02-10T11:32:39.831 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.840 7ff9585ca700 1 -- start start 2021-02-10T11:32:39.831 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.840 7ff9585ca700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff9500b7db0 0x7ff9500b8260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:39.831 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.840 7ff9585ca700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff9500b8880 0x7ff9501a7450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:39.831 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.840 7ff9585ca700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff9500b8c80 0x7ff9501a7a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:39.831 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.841 7ff9585ca700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9500b79b0 con 0x7ff9500b8880 2021-02-10T11:32:39.831 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.841 7ff9585ca700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9501a8090 con 0x7ff9500b7db0 2021-02-10T11:32:39.831 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.841 7ff9585ca700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9501a82b0 con 0x7ff9500b8c80 2021-02-10T11:32:39.831 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.841 7ff955be5700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff9500b8880 0x7ff9501a7450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:39.831 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.841 7ff955be5700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff9500b8880 0x7ff9501a7450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:49128/0 (socket says 172.21.2.112:49128) 2021-02-10T11:32:39.832 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.841 7ff955be5700 1 -- 172.21.2.112:0/4837590 learned_addr learned my addr 172.21.2.112:0/4837590 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:32:39.832 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.841 7ff9563e6700 1 --2- 172.21.2.112:0/4837590 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff9500b7db0 0x7ff9500b8260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:39.832 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.841 7ff956be7700 1 --2- 172.21.2.112:0/4837590 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff9500b8c80 0x7ff9501a7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:39.832 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.841 7ff955be5700 1 -- 172.21.2.112:0/4837590 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff9500b8c80 msgr2=0x7ff9501a7a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:39.832 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.841 7ff955be5700 1 --2- 172.21.2.112:0/4837590 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff9500b8c80 0x7ff9501a7a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:39.832 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.841 7ff955be5700 1 -- 172.21.2.112:0/4837590 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff9500b7db0 msgr2=0x7ff9500b8260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:39.832 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.841 7ff955be5700 1 --2- 172.21.2.112:0/4837590 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff9500b7db0 0x7ff9500b8260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:39.832 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.841 7ff955be5700 1 -- 172.21.2.112:0/4837590 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff950190c10 con 0x7ff9500b8880 2021-02-10T11:32:39.832 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.841 7ff956be7700 1 --2- 172.21.2.112:0/4837590 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff9500b8c80 0x7ff9501a7a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:32:39.832 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.841 7ff9563e6700 1 --2- 172.21.2.112:0/4837590 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff9500b7db0 0x7ff9500b8260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:32:39.832 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.841 7ff955be5700 1 --2- 172.21.2.112:0/4837590 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff9500b8880 0x7ff9501a7450 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 rx=0x7ff94c006f20 tx=0x7ff94c00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:39.832 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.841 7ff942ffd700 1 -- 172.21.2.112:0/4837590 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff94c0009b0 con 0x7ff9500b8880 2021-02-10T11:32:39.832 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.841 7ff942ffd700 1 -- 172.21.2.112:0/4837590 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff94c00bd80 con 0x7ff9500b8880 2021-02-10T11:32:39.833 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.841 7ff9585ca700 1 -- 172.21.2.112:0/4837590 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff950190c10 con 0x7ff9500b8880 2021-02-10T11:32:39.833 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.842 7ff942ffd700 1 -- 172.21.2.112:0/4837590 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff94c002220 con 0x7ff9500b8880 2021-02-10T11:32:39.833 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.842 7ff9585ca700 1 -- 172.21.2.112:0/4837590 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9500c1eb0 con 0x7ff9500b8880 2021-02-10T11:32:39.833 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.842 7ff942ffd700 1 -- 172.21.2.112:0/4837590 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff94c00a000 con 0x7ff9500b8880 2021-02-10T11:32:39.833 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.842 7ff942ffd700 1 --2- 172.21.2.112:0/4837590 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff9380186c0 0x7ff93801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:39.833 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.842 7ff942ffd700 1 -- 172.21.2.112:0/4837590 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5573+0+0 (secure 0 0 0) 0x7ff94c032580 con 0x7ff9500b8880 2021-02-10T11:32:39.833 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.842 7ff9563e6700 1 --2- 172.21.2.112:0/4837590 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff9380186c0 0x7ff93801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:39.833 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.843 7ff9563e6700 1 --2- 172.21.2.112:0/4837590 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff9380186c0 0x7ff93801abc0 crc :-1 s=READY pgs=578 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:39.861 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.873 7ff9585ca700 1 -- 172.21.2.112:0/4837590 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff950105b30 con 0x7ff9500b8880 2021-02-10T11:32:39.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:39.876 7ff942ffd700 1 -- 172.21.2.112:0/4837590 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff94c02dc50 con 0x7ff9500b8880 2021-02-10T11:32:39.991 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:40.004 7ff9585ca700 1 -- 172.21.2.112:0/4837590 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7ff9500f0b50 con 0x7ff9500b8880 2021-02-10T11:32:39.992 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:40.005 7ff942ffd700 1 -- 172.21.2.112:0/4837590 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v46) v1 ==== 92+0+10 (secure 0 0 0) 0x7ff94c02dc50 con 0x7ff9500b8880 2021-02-10T11:32:39.992 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:32:39.994 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:40.007 7ff9585ca700 1 -- 172.21.2.112:0/4837590 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff9380186c0 msgr2=0x7ff93801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:39.994 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:40.007 7ff9585ca700 1 --2- 172.21.2.112:0/4837590 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff9380186c0 0x7ff93801abc0 crc :-1 s=READY pgs=578 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:39.994 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:40.007 7ff9585ca700 1 -- 172.21.2.112:0/4837590 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff9500b8880 msgr2=0x7ff9501a7450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:39.994 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:40.007 7ff9585ca700 1 --2- 172.21.2.112:0/4837590 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff9500b8880 0x7ff9501a7450 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 rx=0x7ff94c006f20 tx=0x7ff94c00b990).stop 2021-02-10T11:32:39.994 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:40.007 7ff9585ca700 1 -- 172.21.2.112:0/4837590 shutdown_connections 2021-02-10T11:32:39.994 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:40.007 7ff9585ca700 1 --2- 172.21.2.112:0/4837590 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff9380186c0 0x7ff93801abc0 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:39.994 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:40.007 7ff9585ca700 1 --2- 172.21.2.112:0/4837590 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff9500b8c80 0x7ff9501a7a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:39.994 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:40.007 7ff9585ca700 1 --2- 172.21.2.112:0/4837590 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff9500b8880 0x7ff9501a7450 unknown :-1 s=CLOSED pgs=1140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:39.995 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:40.007 7ff9585ca700 1 --2- 172.21.2.112:0/4837590 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff9500b7db0 0x7ff9500b8260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:39.995 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:40.007 7ff9585ca700 1 -- 172.21.2.112:0/4837590 >> 172.21.2.112:0/4837590 conn(0x7ff950094e10 msgr2=0x7ff9500a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:39.995 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:40.007 7ff9585ca700 1 -- 172.21.2.112:0/4837590 shutdown_connections 2021-02-10T11:32:39.995 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:40.007 7ff9585ca700 1 -- 172.21.2.112:0/4837590 wait complete. 2021-02-10T11:32:40.003 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:32:40.433 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:40.429 7fb1378d4700 1 Processor -- start 2021-02-10T11:32:40.433 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:40.429 7fb1378d4700 1 -- start start 2021-02-10T11:32:40.433 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:40.429 7fb1378d4700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb1301b0870 0x7fb1300f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:40.433 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:40.429 7fb1378d4700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1301b0c70 con 0x7fb1300efe00 2021-02-10T11:32:40.433 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:40.429 7fb1378d4700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1300f0b50 con 0x7fb1301b0470 2021-02-10T11:32:40.434 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:40.429 7fb1378d4700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1300e5ea0 con 0x7fb1301b0870 2021-02-10T11:32:40.434 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:40.429 7fb135ef1700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fb1300efe00 0x7fb1300f0200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:48600/0 (socket says 172.21.2.112:48600) 2021-02-10T11:32:40.434 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:40.429 7fb135ef1700 1 -- 172.21.2.112:0/4007458846 learned_addr learned my addr 172.21.2.112:0/4007458846 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:32:40.434 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:40.430 7fb134eef700 1 --2- 172.21.2.112:0/4007458846 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb1301b0870 0x7fb1300f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:40.434 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:40.430 7fb123fff700 1 -- 172.21.2.112:0/4007458846 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3973605113 0 0) 0x7fb128001500 con 0x7fb1300efe00 2021-02-10T11:32:40.434 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:40.430 7fb123fff700 1 -- 172.21.2.112:0/4007458846 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb11c002fe0 con 0x7fb1300efe00 2021-02-10T11:32:40.434 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:40.430 7fb123fff700 1 -- 172.21.2.112:0/4007458846 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 867655517 0 0) 0x7fb124000c60 con 0x7fb1301b0470 2021-02-10T11:32:40.434 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:40.430 7fb134eef700 1 -- 172.21.2.112:0/4007458846 >> v1:172.21.2.120:6789/0 conn(0x7fb1301b0470 legacy=0x7fb1300ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:40.434 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:40.430 7fb134eef700 1 -- 172.21.2.112:0/4007458846 >> v1:172.21.2.112:6789/0 conn(0x7fb1300efe00 legacy=0x7fb1300f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:40.434 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:40.430 7fb134eef700 1 -- 172.21.2.112:0/4007458846 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1300e52e0 con 0x7fb1301b0870 2021-02-10T11:32:40.434 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:40.430 7fb134eef700 1 --2- 172.21.2.112:0/4007458846 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb1301b0870 0x7fb1300f20e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7fb12c006650 tx=0x7fb12c009a00).ready entity=mon.2 client_cookie=d6be1765d8ef623f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:40.434 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:40.431 7fb123fff700 1 -- 172.21.2.112:0/4007458846 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb12c0009b0 con 0x7fb1301b0870 2021-02-10T11:32:40.434 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:40.431 7fb123fff700 1 -- 172.21.2.112:0/4007458846 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb12c00c0c0 con 0x7fb1301b0870 2021-02-10T11:32:40.435 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:40.431 7fb123fff700 1 -- 172.21.2.112:0/4007458846 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb12c00c520 con 0x7fb1301b0870 2021-02-10T11:32:40.435 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:40.431 7fb1378d4700 1 -- 172.21.2.112:0/4007458846 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb1301b0870 msgr2=0x7fb1300f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:40.435 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:40.431 7fb1378d4700 1 --2- 172.21.2.112:0/4007458846 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb1301b0870 0x7fb1300f20e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7fb12c006650 tx=0x7fb12c009a00).stop 2021-02-10T11:32:40.435 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:40.431 7fb1378d4700 1 -- 172.21.2.112:0/4007458846 shutdown_connections 2021-02-10T11:32:40.435 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:40.431 7fb1378d4700 1 --2- 172.21.2.112:0/4007458846 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb1301b0870 0x7fb1300f20e0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:40.435 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:40.431 7fb1378d4700 1 -- 172.21.2.112:0/4007458846 >> 172.21.2.112:0/4007458846 conn(0x7fb1300ce3e0 msgr2=0x7fb1300e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:40.435 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:40.431 7fb1378d4700 1 -- 172.21.2.112:0/4007458846 shutdown_connections 2021-02-10T11:32:40.435 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:40.432 7fb1378d4700 1 -- 172.21.2.112:0/4007458846 wait complete. 2021-02-10T11:32:40.435 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:40.446 7fb1378d4700 1 Processor -- start 2021-02-10T11:32:40.435 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:40.446 7fb1378d4700 1 -- start start 2021-02-10T11:32:40.437 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:40.447 7fb1378d4700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb1300b20b0 0x7fb1300ac970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:40.437 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:40.447 7fb1378d4700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb1300b24b0 0x7fb130191dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:40.437 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:40.447 7fb1378d4700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb1300acf90 0x7fb1301923e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:40.437 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:40.447 7fb1378d4700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1300b1b30 con 0x7fb1300acf90 2021-02-10T11:32:40.437 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:40.447 7fb1378d4700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1301a04e0 con 0x7fb1300b20b0 2021-02-10T11:32:40.437 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:40.447 7fb1378d4700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1301a06c0 con 0x7fb1300b24b0 2021-02-10T11:32:40.437 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:40.447 7fb135ef1700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb1300acf90 0x7fb1301923e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:40.437 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:40.447 7fb135ef1700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb1300acf90 0x7fb1301923e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:49142/0 (socket says 172.21.2.112:49142) 2021-02-10T11:32:40.437 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:40.447 7fb135ef1700 1 -- 172.21.2.112:0/2500190216 learned_addr learned my addr 172.21.2.112:0/2500190216 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:32:40.437 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:40.447 7fb135ef1700 1 -- 172.21.2.112:0/2500190216 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb1300b24b0 msgr2=0x7fb130191dc0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:32:40.437 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:40.447 7fb135ef1700 1 --2- 172.21.2.112:0/2500190216 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb1300b24b0 0x7fb130191dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:40.437 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:40.447 7fb135ef1700 1 -- 172.21.2.112:0/2500190216 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb1300b20b0 msgr2=0x7fb1300ac970 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:32:40.438 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:40.447 7fb135ef1700 1 --2- 172.21.2.112:0/2500190216 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb1300b20b0 0x7fb1300ac970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:40.438 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:40.447 7fb135ef1700 1 -- 172.21.2.112:0/2500190216 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb130192be0 con 0x7fb1300acf90 2021-02-10T11:32:40.438 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:40.447 7fb135ef1700 1 --2- 172.21.2.112:0/2500190216 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb1300acf90 0x7fb1301923e0 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 rx=0x7fb128001c10 tx=0x7fb12800aca0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:40.438 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:40.447 7fb121ffb700 1 -- 172.21.2.112:0/2500190216 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb128007ce0 con 0x7fb1300acf90 2021-02-10T11:32:40.438 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:40.447 7fb121ffb700 1 -- 172.21.2.112:0/2500190216 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1280070d0 con 0x7fb1300acf90 2021-02-10T11:32:40.438 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:40.447 7fb121ffb700 1 -- 172.21.2.112:0/2500190216 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb128004930 con 0x7fb1300acf90 2021-02-10T11:32:40.438 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:40.447 7fb1378d4700 1 -- 172.21.2.112:0/2500190216 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb130192be0 con 0x7fb1300acf90 2021-02-10T11:32:40.438 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:40.447 7fb1378d4700 1 -- 172.21.2.112:0/2500190216 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb130193170 con 0x7fb1300acf90 2021-02-10T11:32:40.438 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:40.448 7fb121ffb700 1 -- 172.21.2.112:0/2500190216 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb128004720 con 0x7fb1300acf90 2021-02-10T11:32:40.438 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:40.448 7fb121ffb700 1 --2- 172.21.2.112:0/2500190216 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb1100186c0 0x7fb11001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:40.438 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:40.448 7fb121ffb700 1 -- 172.21.2.112:0/2500190216 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fb1280328f0 con 0x7fb1300acf90 2021-02-10T11:32:40.438 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:40.448 7fb1356f0700 1 --2- 172.21.2.112:0/2500190216 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb1100186c0 0x7fb11001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:40.438 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:40.449 7fb1356f0700 1 --2- 172.21.2.112:0/2500190216 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb1100186c0 0x7fb11001abc0 crc :-1 s=READY pgs=579 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:40.466 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:40.479 7fb1378d4700 1 -- 172.21.2.112:0/2500190216 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb130105af0 con 0x7fb1300acf90 2021-02-10T11:32:40.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:40.481 7fb121ffb700 1 -- 172.21.2.112:0/2500190216 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb12802df60 con 0x7fb1300acf90 2021-02-10T11:32:40.595 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:40.608 7fb1378d4700 1 -- 172.21.2.112:0/2500190216 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7fb1300f0b50 con 0x7fb1300acf90 2021-02-10T11:32:40.596 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:40.608 7fb121ffb700 1 -- 172.21.2.112:0/2500190216 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v46) v1 ==== 104+0+10 (secure 0 0 0) 0x7fb12802df60 con 0x7fb1300acf90 2021-02-10T11:32:40.597 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:32:40.597 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:40.610 7fb1378d4700 1 -- 172.21.2.112:0/2500190216 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb1100186c0 msgr2=0x7fb11001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:40.598 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:40.610 7fb1378d4700 1 --2- 172.21.2.112:0/2500190216 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb1100186c0 0x7fb11001abc0 crc :-1 s=READY pgs=579 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:40.598 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:40.610 7fb1378d4700 1 -- 172.21.2.112:0/2500190216 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb1300acf90 msgr2=0x7fb1301923e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:40.598 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:40.610 7fb1378d4700 1 --2- 172.21.2.112:0/2500190216 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb1300acf90 0x7fb1301923e0 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 rx=0x7fb128001c10 tx=0x7fb12800aca0).stop 2021-02-10T11:32:40.598 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:40.611 7fb1378d4700 1 -- 172.21.2.112:0/2500190216 shutdown_connections 2021-02-10T11:32:40.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:40.611 7fb1378d4700 1 --2- 172.21.2.112:0/2500190216 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb1100186c0 0x7fb11001abc0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:40.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:40.611 7fb1378d4700 1 --2- 172.21.2.112:0/2500190216 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb1300acf90 0x7fb1301923e0 unknown :-1 s=CLOSED pgs=1142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:40.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:40.611 7fb1378d4700 1 --2- 172.21.2.112:0/2500190216 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb1300b24b0 0x7fb130191dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:40.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:40.611 7fb1378d4700 1 --2- 172.21.2.112:0/2500190216 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb1300b20b0 0x7fb1300ac970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:40.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:40.611 7fb1378d4700 1 -- 172.21.2.112:0/2500190216 >> 172.21.2.112:0/2500190216 conn(0x7fb1300bf4e0 msgr2=0x7fb1300be3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:40.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:40.611 7fb1378d4700 1 -- 172.21.2.112:0/2500190216 shutdown_connections 2021-02-10T11:32:40.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:40.611 7fb1378d4700 1 -- 172.21.2.112:0/2500190216 wait complete. 2021-02-10T11:32:40.607 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:32:41.036 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.032 7f5ae0ce4700 1 Processor -- start 2021-02-10T11:32:41.036 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.032 7f5ae0ce4700 1 -- start start 2021-02-10T11:32:41.037 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.032 7f5ae0ce4700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f5adc1b0870 0x7f5adc0f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:41.037 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.033 7f5ae0ce4700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f5adc0efe00 0x7f5adc0f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:41.037 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.033 7f5ae0ce4700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5adc1b0c70 con 0x7f5adc1b0470 2021-02-10T11:32:41.037 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.033 7f5ae0ce4700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5adc0e5ea0 con 0x7f5adc0efe00 2021-02-10T11:32:41.037 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.033 7f5ae0ce4700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5adc0f0b50 con 0x7f5adc1b0870 2021-02-10T11:32:41.037 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.033 7f5adaae8700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f5adc1b0470 0x7f5adc0ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:48610/0 (socket says 172.21.2.112:48610) 2021-02-10T11:32:41.038 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.033 7f5adaae8700 1 -- 172.21.2.112:0/3465208547 learned_addr learned my addr 172.21.2.112:0/3465208547 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:32:41.038 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.033 7f5adb2e9700 1 --2- 172.21.2.112:0/3465208547 >> v2:172.21.2.112:3300/0 conn(0x7f5adc0efe00 0x7f5adc0f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:41.038 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.033 7f5adb2e9700 1 -- 172.21.2.112:0/3465208547 >> v2:172.21.2.120:3300/0 conn(0x7f5adc1b0870 msgr2=0x7f5adc0f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:41.038 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.033 7f5adb2e9700 1 --2- 172.21.2.112:0/3465208547 >> v2:172.21.2.120:3300/0 conn(0x7f5adc1b0870 0x7f5adc0f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:41.038 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.033 7f5adb2e9700 1 -- 172.21.2.112:0/3465208547 >> v1:172.21.2.112:6789/0 conn(0x7f5adc1b0470 legacy=0x7f5adc0ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:41.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.033 7f5ad9ae6700 1 -- 172.21.2.112:0/3465208547 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3528641792 0 0) 0x7f5acc001180 con 0x7f5adc1b0470 2021-02-10T11:32:41.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.033 7f5adb2e9700 1 -- 172.21.2.112:0/3465208547 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5adc0f0820 con 0x7f5adc0efe00 2021-02-10T11:32:41.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.033 7f5adb2e9700 1 --2- 172.21.2.112:0/3465208547 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5adc0efe00 0x7f5adc0f0200 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 rx=0x7f5ad4006e30 tx=0x7f5ad4009de0).ready entity=mon.0 client_cookie=9d437904fc70f74d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:41.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.034 7f5ad9ae6700 1 -- 172.21.2.112:0/3465208547 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5ad400b300 con 0x7f5adc0efe00 2021-02-10T11:32:41.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.034 7f5ad9ae6700 1 -- 172.21.2.112:0/3465208547 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ad400b300 con 0x7f5adc0efe00 2021-02-10T11:32:41.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.034 7f5ad9ae6700 1 -- 172.21.2.112:0/3465208547 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5ad400b580 con 0x7f5adc0efe00 2021-02-10T11:32:41.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.034 7f5ae0ce4700 1 -- 172.21.2.112:0/3465208547 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5adc0efe00 msgr2=0x7f5adc0f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:41.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.034 7f5ae0ce4700 1 --2- 172.21.2.112:0/3465208547 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5adc0efe00 0x7f5adc0f0200 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 rx=0x7f5ad4006e30 tx=0x7f5ad4009de0).stop 2021-02-10T11:32:41.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.034 7f5ae0ce4700 1 -- 172.21.2.112:0/3465208547 shutdown_connections 2021-02-10T11:32:41.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.034 7f5ae0ce4700 1 --2- 172.21.2.112:0/3465208547 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5adc0efe00 0x7f5adc0f0200 unknown :-1 s=CLOSED pgs=1144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:41.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.035 7f5ae0ce4700 1 --2- 172.21.2.112:0/3465208547 >> v2:172.21.2.120:3300/0 conn(0x7f5adc1b0870 0x7f5adc0f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:41.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.035 7f5ae0ce4700 1 -- 172.21.2.112:0/3465208547 >> 172.21.2.112:0/3465208547 conn(0x7f5adc0ce3e0 msgr2=0x7f5adc0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:41.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.035 7f5ae0ce4700 1 -- 172.21.2.112:0/3465208547 shutdown_connections 2021-02-10T11:32:41.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.035 7f5ae0ce4700 1 -- 172.21.2.112:0/3465208547 wait complete. 2021-02-10T11:32:41.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.049 7f5ae0ce4700 1 Processor -- start 2021-02-10T11:32:41.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.049 7f5ae0ce4700 1 -- start start 2021-02-10T11:32:41.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.049 7f5ae0ce4700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5adc0bf910 0x7f5adc0bfd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:41.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.049 7f5ae0ce4700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5adc0c0330 0x7f5adc0c0730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:41.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.049 7f5ae0ce4700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5adc0c13d0 0x7f5adc0c17d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:41.045 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.049 7f5ae0ce4700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5adc109930 con 0x7f5adc0bf910 2021-02-10T11:32:41.045 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.049 7f5ae0ce4700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5adc109b50 con 0x7f5adc0c13d0 2021-02-10T11:32:41.045 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.049 7f5ae0ce4700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5adc0c1df0 con 0x7f5adc0c0330 2021-02-10T11:32:41.045 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.049 7f5adaae8700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5adc0bf910 0x7f5adc0bfd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:41.045 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.050 7f5adaae8700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5adc0bf910 0x7f5adc0bfd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:49154/0 (socket says 172.21.2.112:49154) 2021-02-10T11:32:41.045 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.050 7f5adaae8700 1 -- 172.21.2.112:0/3401775942 learned_addr learned my addr 172.21.2.112:0/3401775942 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:32:41.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.050 7f5adaae8700 1 -- 172.21.2.112:0/3401775942 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5adc0c0330 msgr2=0x7f5adc0c0730 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:32:41.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.050 7f5adaae8700 1 --2- 172.21.2.112:0/3401775942 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5adc0c0330 0x7f5adc0c0730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:41.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.050 7f5adaae8700 1 -- 172.21.2.112:0/3401775942 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5adc0c13d0 msgr2=0x7f5adc0c17d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:41.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.050 7f5adb2e9700 1 --2- 172.21.2.112:0/3401775942 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5adc0c13d0 0x7f5adc0c17d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:41.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.050 7f5adaae8700 1 --2- 172.21.2.112:0/3401775942 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5adc0c13d0 0x7f5adc0c17d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:41.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.050 7f5adaae8700 1 -- 172.21.2.112:0/3401775942 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5adc0c1fd0 con 0x7f5adc0bf910 2021-02-10T11:32:41.047 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.050 7f5adb2e9700 1 --2- 172.21.2.112:0/3401775942 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5adc0c13d0 0x7f5adc0c17d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:32:41.047 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.050 7f5adaae8700 1 --2- 172.21.2.112:0/3401775942 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5adc0bf910 0x7f5adc0bfd10 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 rx=0x7f5acc001ea0 tx=0x7f5acc00a880).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:41.047 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.051 7f5ad37fe700 1 -- 172.21.2.112:0/3401775942 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5acc00ac70 con 0x7f5adc0bf910 2021-02-10T11:32:41.047 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.051 7f5ad37fe700 1 -- 172.21.2.112:0/3401775942 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5acc001180 con 0x7f5adc0bf910 2021-02-10T11:32:41.047 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.051 7f5ad37fe700 1 -- 172.21.2.112:0/3401775942 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5acc004590 con 0x7f5adc0bf910 2021-02-10T11:32:41.047 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.051 7f5ae0ce4700 1 -- 172.21.2.112:0/3401775942 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5adc0c1fd0 con 0x7f5adc0bf910 2021-02-10T11:32:41.048 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.051 7f5ae0ce4700 1 -- 172.21.2.112:0/3401775942 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5adc0c2560 con 0x7f5adc0bf910 2021-02-10T11:32:41.048 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.051 7f5ad37fe700 1 -- 172.21.2.112:0/3401775942 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5acc0043a0 con 0x7f5adc0bf910 2021-02-10T11:32:41.048 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.051 7f5ad37fe700 1 --2- 172.21.2.112:0/3401775942 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5ac00186c0 0x7f5ac001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:41.048 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.051 7f5ad37fe700 1 -- 172.21.2.112:0/3401775942 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f5acc032550 con 0x7f5adc0bf910 2021-02-10T11:32:41.048 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.052 7f5ada2e7700 1 --2- 172.21.2.112:0/3401775942 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5ac00186c0 0x7f5ac001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:41.048 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.052 7f5ada2e7700 1 --2- 172.21.2.112:0/3401775942 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5ac00186c0 0x7f5ac001abc0 crc :-1 s=READY pgs=580 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:41.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.082 7f5ae0ce4700 1 -- 172.21.2.112:0/3401775942 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5adc0f0b50 con 0x7f5adc0bf910 2021-02-10T11:32:41.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.085 7f5ad37fe700 1 -- 172.21.2.112:0/3401775942 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5acc000c90 con 0x7f5adc0bf910 2021-02-10T11:32:41.199 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.212 7f5ae0ce4700 1 -- 172.21.2.112:0/3401775942 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f5adc0f0b50 con 0x7f5adc0bf910 2021-02-10T11:32:41.200 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.212 7f5ad37fe700 1 -- 172.21.2.112:0/3401775942 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v46) v1 ==== 100+0+10 (secure 0 0 0) 0x7f5acc0043a0 con 0x7f5adc0bf910 2021-02-10T11:32:41.200 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:32:41.202 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.214 7f5ae0ce4700 1 -- 172.21.2.112:0/3401775942 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5ac00186c0 msgr2=0x7f5ac001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:41.202 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.214 7f5ae0ce4700 1 --2- 172.21.2.112:0/3401775942 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5ac00186c0 0x7f5ac001abc0 crc :-1 s=READY pgs=580 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:41.202 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.214 7f5ae0ce4700 1 -- 172.21.2.112:0/3401775942 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5adc0bf910 msgr2=0x7f5adc0bfd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:41.202 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.214 7f5ae0ce4700 1 --2- 172.21.2.112:0/3401775942 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5adc0bf910 0x7f5adc0bfd10 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 rx=0x7f5acc001ea0 tx=0x7f5acc00a880).stop 2021-02-10T11:32:41.202 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.215 7f5ae0ce4700 1 -- 172.21.2.112:0/3401775942 shutdown_connections 2021-02-10T11:32:41.202 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.215 7f5ae0ce4700 1 --2- 172.21.2.112:0/3401775942 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5ac00186c0 0x7f5ac001abc0 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:41.202 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.215 7f5ae0ce4700 1 --2- 172.21.2.112:0/3401775942 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5adc0c13d0 0x7f5adc0c17d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:41.203 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.215 7f5ae0ce4700 1 --2- 172.21.2.112:0/3401775942 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5adc0c0330 0x7f5adc0c0730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:41.203 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.215 7f5ae0ce4700 1 --2- 172.21.2.112:0/3401775942 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5adc0bf910 0x7f5adc0bfd10 unknown :-1 s=CLOSED pgs=1145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:41.203 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.215 7f5ae0ce4700 1 -- 172.21.2.112:0/3401775942 >> 172.21.2.112:0/3401775942 conn(0x7f5adc0bf4e0 msgr2=0x7f5adc191190 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:41.203 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.215 7f5ae0ce4700 1 -- 172.21.2.112:0/3401775942 shutdown_connections 2021-02-10T11:32:41.203 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.215 7f5ae0ce4700 1 -- 172.21.2.112:0/3401775942 wait complete. 2021-02-10T11:32:41.211 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:32:41.211 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:32:41.638 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.634 7f491ab46700 1 Processor -- start 2021-02-10T11:32:41.638 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.634 7f491ab46700 1 -- start start 2021-02-10T11:32:41.638 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.634 7f491ab46700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f49140e5070 0x7f49141b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:41.638 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.634 7f491ab46700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f49140e5470 0x7f491407d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:41.638 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.634 7f491ab46700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f491407ed80 con 0x7f49141b3e40 2021-02-10T11:32:41.638 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.634 7f491ab46700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4914080310 con 0x7f49140e5470 2021-02-10T11:32:41.638 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.634 7f491ab46700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4914080500 con 0x7f49140e5070 2021-02-10T11:32:41.638 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.635 7f4919163700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f49141b3e40 0x7f491407dcb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:48628/0 (socket says 172.21.2.112:48628) 2021-02-10T11:32:41.638 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.635 7f4919163700 1 -- 172.21.2.112:0/1881045026 learned_addr learned my addr 172.21.2.112:0/1881045026 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:32:41.639 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.635 7f4913fff700 1 --2- 172.21.2.112:0/1881045026 >> v2:172.21.2.112:3300/0 conn(0x7f49140e5470 0x7f491407d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:41.639 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.635 7f4918962700 1 --2- 172.21.2.112:0/1881045026 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f49140e5070 0x7f49141b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:41.639 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.635 7f4913fff700 1 -- 172.21.2.112:0/1881045026 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f49140e5070 msgr2=0x7f49141b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:41.639 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.635 7f4913fff700 1 --2- 172.21.2.112:0/1881045026 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f49140e5070 0x7f49141b3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:41.639 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.635 7f4913fff700 1 -- 172.21.2.112:0/1881045026 >> v1:172.21.2.112:6789/0 conn(0x7f49141b3e40 legacy=0x7f491407dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:41.639 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.635 7f4913fff700 1 -- 172.21.2.112:0/1881045026 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49140806f0 con 0x7f49140e5470 2021-02-10T11:32:41.639 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.635 7f4913fff700 1 --2- 172.21.2.112:0/1881045026 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f49140e5470 0x7f491407d690 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 rx=0x7f4904006650 tx=0x7f4904009a80).ready entity=mon.0 client_cookie=ffcda7cc2e53ff24 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:41.639 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.636 7f49137fe700 1 -- 172.21.2.112:0/1881045026 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f49040009b0 con 0x7f49140e5470 2021-02-10T11:32:41.639 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.636 7f49137fe700 1 -- 172.21.2.112:0/1881045026 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f490400c0c0 con 0x7f49140e5470 2021-02-10T11:32:41.639 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.636 7f49137fe700 1 -- 172.21.2.112:0/1881045026 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f490400c5e0 con 0x7f49140e5470 2021-02-10T11:32:41.639 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.636 7f491ab46700 1 -- 172.21.2.112:0/1881045026 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f49140e5470 msgr2=0x7f491407d690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:41.639 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.636 7f491ab46700 1 --2- 172.21.2.112:0/1881045026 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f49140e5470 0x7f491407d690 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 rx=0x7f4904006650 tx=0x7f4904009a80).stop 2021-02-10T11:32:41.639 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.636 7f491ab46700 1 -- 172.21.2.112:0/1881045026 shutdown_connections 2021-02-10T11:32:41.640 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.636 7f491ab46700 1 --2- 172.21.2.112:0/1881045026 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f49140e5470 0x7f491407d690 unknown :-1 s=CLOSED pgs=1147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:41.640 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.636 7f491ab46700 1 --2- 172.21.2.112:0/1881045026 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f49140e5070 0x7f49141b3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:41.640 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.636 7f491ab46700 1 -- 172.21.2.112:0/1881045026 >> 172.21.2.112:0/1881045026 conn(0x7f49140ce690 msgr2=0x7f49140e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:41.640 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.637 7f491ab46700 1 -- 172.21.2.112:0/1881045026 shutdown_connections 2021-02-10T11:32:41.640 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.637 7f491ab46700 1 -- 172.21.2.112:0/1881045026 wait complete. 2021-02-10T11:32:41.641 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.651 7f491ab46700 1 Processor -- start 2021-02-10T11:32:41.642 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.651 7f491ab46700 1 -- start start 2021-02-10T11:32:41.642 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.651 7f491ab46700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4914191bb0 0x7f49140a2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:41.642 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.651 7f491ab46700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f49140a29a0 0x7f49140a2da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:41.642 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.651 7f491ab46700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f49140a33c0 0x7f49140a03d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:41.642 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.651 7f491ab46700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49140adf90 con 0x7f49140a29a0 2021-02-10T11:32:41.642 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.651 7f491ab46700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49140a1e00 con 0x7f49140a33c0 2021-02-10T11:32:41.642 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.651 7f491ab46700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49140a09f0 con 0x7f4914191bb0 2021-02-10T11:32:41.642 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.652 7f4913fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f49140a29a0 0x7f49140a2da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:41.642 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.652 7f4913fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f49140a29a0 0x7f49140a2da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:49172/0 (socket says 172.21.2.112:49172) 2021-02-10T11:32:41.642 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.652 7f4913fff700 1 -- 172.21.2.112:0/3532304380 learned_addr learned my addr 172.21.2.112:0/3532304380 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:32:41.642 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.652 7f4913fff700 1 -- 172.21.2.112:0/3532304380 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4914191bb0 msgr2=0x7f49140a2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:41.642 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.652 7f4913fff700 1 --2- 172.21.2.112:0/3532304380 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4914191bb0 0x7f49140a2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:41.643 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.652 7f4913fff700 1 -- 172.21.2.112:0/3532304380 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f49140a33c0 msgr2=0x7f49140a03d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:41.643 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.652 7f4913fff700 1 --2- 172.21.2.112:0/3532304380 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f49140a33c0 0x7f49140a03d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:41.643 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.652 7f4913fff700 1 -- 172.21.2.112:0/3532304380 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49140a0bd0 con 0x7f49140a29a0 2021-02-10T11:32:41.643 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.652 7f4913fff700 1 --2- 172.21.2.112:0/3532304380 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f49140a29a0 0x7f49140a2da0 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 rx=0x7f4904009630 tx=0x7f490400b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:41.643 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.652 7f49117fa700 1 -- 172.21.2.112:0/3532304380 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f49040093a0 con 0x7f49140a29a0 2021-02-10T11:32:41.643 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.652 7f49117fa700 1 -- 172.21.2.112:0/3532304380 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f490400bd70 con 0x7f49140a29a0 2021-02-10T11:32:41.643 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.652 7f491ab46700 1 -- 172.21.2.112:0/3532304380 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49140a0bd0 con 0x7f49140a29a0 2021-02-10T11:32:41.643 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.653 7f491ab46700 1 -- 172.21.2.112:0/3532304380 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49140a1160 con 0x7f49140a29a0 2021-02-10T11:32:41.643 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.653 7f49117fa700 1 -- 172.21.2.112:0/3532304380 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4904005560 con 0x7f49140a29a0 2021-02-10T11:32:41.643 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.653 7f49117fa700 1 -- 172.21.2.112:0/3532304380 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f490400a000 con 0x7f49140a29a0 2021-02-10T11:32:41.643 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.653 7f49117fa700 1 --2- 172.21.2.112:0/3532304380 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f48f80186c0 0x7f48f801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:41.643 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.653 7f49117fa700 1 -- 172.21.2.112:0/3532304380 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f4904032790 con 0x7f49140a29a0 2021-02-10T11:32:41.643 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.654 7f4918962700 1 --2- 172.21.2.112:0/3532304380 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f48f80186c0 0x7f48f801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:41.644 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.654 7f4918962700 1 --2- 172.21.2.112:0/3532304380 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f48f80186c0 0x7f48f801abc0 crc :-1 s=READY pgs=581 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:41.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.685 7f491ab46700 1 -- 172.21.2.112:0/3532304380 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49140a1160 con 0x7f49140a29a0 2021-02-10T11:32:41.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.687 7f49117fa700 1 -- 172.21.2.112:0/3532304380 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4904009340 con 0x7f49140a29a0 2021-02-10T11:32:41.801 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.814 7f491ab46700 1 -- 172.21.2.112:0/3532304380 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f49140a1160 con 0x7f49140a29a0 2021-02-10T11:32:41.801 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.814 7f49117fa700 1 -- 172.21.2.112:0/3532304380 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+2093 (secure 0 0 0) 0x7f490400bd70 con 0x7f49140a29a0 2021-02-10T11:32:41.802 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 30 2021-02-10T11:32:41.802 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:32:41.802 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":30,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7536,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/340651080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":340651080},{"type":"v1","addr":"172.21.2.112:6833","nonce":340651080}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":26}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:32:31.869812","modified":"2021-02-10 11:32:33.879671","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6409},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6409":{"gid":6409,"name":"b","rank":0,"incarnation":27,"state":"up:active","state_seq":3,"addr":"172.21.2.120:6835/210055046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":210055046},{"type":"v1","addr":"172.21.2.120:6835","nonce":210055046}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":4}]} 2021-02-10T11:32:41.803 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.816 7f491ab46700 1 -- 172.21.2.112:0/3532304380 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f48f80186c0 msgr2=0x7f48f801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:41.803 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.816 7f491ab46700 1 --2- 172.21.2.112:0/3532304380 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f48f80186c0 0x7f48f801abc0 crc :-1 s=READY pgs=581 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:41.803 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.816 7f491ab46700 1 -- 172.21.2.112:0/3532304380 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f49140a29a0 msgr2=0x7f49140a2da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:41.803 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.816 7f491ab46700 1 --2- 172.21.2.112:0/3532304380 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f49140a29a0 0x7f49140a2da0 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 rx=0x7f4904009630 tx=0x7f490400b980).stop 2021-02-10T11:32:41.804 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.816 7f491ab46700 1 -- 172.21.2.112:0/3532304380 shutdown_connections 2021-02-10T11:32:41.804 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.816 7f491ab46700 1 --2- 172.21.2.112:0/3532304380 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f48f80186c0 0x7f48f801abc0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:41.804 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.816 7f491ab46700 1 --2- 172.21.2.112:0/3532304380 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f49140a33c0 0x7f49140a03d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:41.804 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.816 7f491ab46700 1 --2- 172.21.2.112:0/3532304380 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f49140a29a0 0x7f49140a2da0 unknown :-1 s=CLOSED pgs=1148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:41.804 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.816 7f491ab46700 1 --2- 172.21.2.112:0/3532304380 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4914191bb0 0x7f49140a2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:41.804 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.817 7f491ab46700 1 -- 172.21.2.112:0/3532304380 >> 172.21.2.112:0/3532304380 conn(0x7f49141a31d0 msgr2=0x7f491408b830 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:41.804 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.817 7f491ab46700 1 -- 172.21.2.112:0/3532304380 shutdown_connections 2021-02-10T11:32:41.804 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:41.817 7f491ab46700 1 -- 172.21.2.112:0/3532304380 wait complete. 2021-02-10T11:32:42.014 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:32:31.869812', 'modified': '2021-02-10 11:32:33.879671', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6409}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6409': {'gid': 6409, 'name': 'b', 'rank': 0, 'incarnation': 27, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.120:6835/210055046', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.120:6834', 'nonce': 210055046}, {'type': 'v1', 'addr': '172.21.2.120:6835', 'nonce': 210055046}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [9], 'metadata_pool': 8, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:32:42.014 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:32:42.014 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:32:42.412 INFO:teuthology.orchestra.run.gibba020.stdout:{ 2021-02-10T11:32:42.412 INFO:teuthology.orchestra.run.gibba020.stdout: "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", 2021-02-10T11:32:42.412 INFO:teuthology.orchestra.run.gibba020.stdout: "whoami": 0, 2021-02-10T11:32:42.412 INFO:teuthology.orchestra.run.gibba020.stdout: "id": 6409, 2021-02-10T11:32:42.412 INFO:teuthology.orchestra.run.gibba020.stdout: "want_state": "up:active", 2021-02-10T11:32:42.412 INFO:teuthology.orchestra.run.gibba020.stdout: "state": "up:active", 2021-02-10T11:32:42.413 INFO:teuthology.orchestra.run.gibba020.stdout: "rank_uptime": 10.553928794999999, 2021-02-10T11:32:42.413 INFO:teuthology.orchestra.run.gibba020.stdout: "mdsmap_epoch": 30, 2021-02-10T11:32:42.413 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch": 46, 2021-02-10T11:32:42.413 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch_barrier": 46, 2021-02-10T11:32:42.413 INFO:teuthology.orchestra.run.gibba020.stdout: "uptime": 16.183888434 2021-02-10T11:32:42.413 INFO:teuthology.orchestra.run.gibba020.stdout:} 2021-02-10T11:32:42.416 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", "whoami": 0, "id": 6409, "want_state": "up:active", "state": "up:active", "rank_uptime": 10.553928794999999, "mdsmap_epoch": 30, "osdmap_epoch": 46, "osdmap_epoch_barrier": 46, "uptime": 16.183888434 } 2021-02-10T11:32:42.417 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:32:42.417 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2021-02-10T11:32:42.417 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@gibba012.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2021-02-10T11:32:42.417 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:42.417 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:32:42.753 DEBUG:teuthology.orchestra.run.gibba012:> sudo modprobe fuse 2021-02-10T11:32:43.097 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:32:43.098 DEBUG:teuthology.orchestra.run.gibba012:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:32:43.441 INFO:teuthology.orchestra.run.gibba012.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:32:43.441 INFO:teuthology.orchestra.run.gibba012.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:32:43.442 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:32:43.442 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:43.442 DEBUG:teuthology.orchestra.run.gibba012:> ls /sys/fs/fuse/connections 2021-02-10T11:32:43.776 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2021-02-10T11:32:43.776 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:32:43.779 DEBUG:teuthology.orchestra.run.gibba012:> sudo modprobe fuse 2021-02-10T11:32:44.123 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:32:44.123 DEBUG:teuthology.orchestra.run.gibba012:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:32:44.188 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba012.stderr:ceph-fuse[125240]: starting ceph client 2021-02-10T11:32:44.189 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba012.stderr:2021-02-10 11:32:44.201 7f4fc3849f80 -1 init, newargv = 0x556c5640d100 newargc=9 2021-02-10T11:32:44.193 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba012.stderr:ceph-fuse[125240]: starting fuse 2021-02-10T11:32:44.466 INFO:teuthology.orchestra.run.gibba012.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:32:44.466 INFO:teuthology.orchestra.run.gibba012.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:32:44.467 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:32:44.467 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:44.467 DEBUG:teuthology.orchestra.run.gibba012:> ls /sys/fs/fuse/connections 2021-02-10T11:32:44.801 INFO:teuthology.orchestra.run.gibba012.stdout:42 2021-02-10T11:32:44.801 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2021-02-10T11:32:44.801 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:44.801 DEBUG:teuthology.orchestra.run.gibba012:> sudo python3 -c ' 2021-02-10T11:32:44.802 DEBUG:teuthology.orchestra.run.gibba012:> import glob 2021-02-10T11:32:44.802 DEBUG:teuthology.orchestra.run.gibba012:> import re 2021-02-10T11:32:44.802 DEBUG:teuthology.orchestra.run.gibba012:> import os 2021-02-10T11:32:44.802 DEBUG:teuthology.orchestra.run.gibba012:> import subprocess 2021-02-10T11:32:44.802 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:32:44.802 DEBUG:teuthology.orchestra.run.gibba012:> def find_socket(client_name): 2021-02-10T11:32:44.802 DEBUG:teuthology.orchestra.run.gibba012:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2021-02-10T11:32:44.802 DEBUG:teuthology.orchestra.run.gibba012:> files = glob.glob(asok_path) 2021-02-10T11:32:44.802 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:32:44.803 DEBUG:teuthology.orchestra.run.gibba012:> # Given a non-glob path, it better be there 2021-02-10T11:32:44.803 DEBUG:teuthology.orchestra.run.gibba012:> if "*" not in asok_path: 2021-02-10T11:32:44.803 DEBUG:teuthology.orchestra.run.gibba012:> assert(len(files) == 1) 2021-02-10T11:32:44.803 DEBUG:teuthology.orchestra.run.gibba012:> return files[0] 2021-02-10T11:32:44.803 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:32:44.803 DEBUG:teuthology.orchestra.run.gibba012:> for f in files: 2021-02-10T11:32:44.803 DEBUG:teuthology.orchestra.run.gibba012:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:32:44.803 DEBUG:teuthology.orchestra.run.gibba012:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:32:44.803 DEBUG:teuthology.orchestra.run.gibba012:> return f 2021-02-10T11:32:44.803 DEBUG:teuthology.orchestra.run.gibba012:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:32:44.803 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:32:44.803 DEBUG:teuthology.orchestra.run.gibba012:> print(find_socket("client.0")) 2021-02-10T11:32:44.803 DEBUG:teuthology.orchestra.run.gibba012:> ' 2021-02-10T11:32:45.164 INFO:teuthology.orchestra.run.gibba012.stdout:/var/run/ceph/ceph-client.0.125240.asok 2021-02-10T11:32:45.167 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.125240.asok 2021-02-10T11:32:45.167 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:45.167 DEBUG:teuthology.orchestra.run.gibba012:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.125240.asok status 2021-02-10T11:32:45.556 INFO:teuthology.orchestra.run.gibba012.stdout:{ 2021-02-10T11:32:45.556 INFO:teuthology.orchestra.run.gibba012.stdout: "metadata": { 2021-02-10T11:32:45.556 INFO:teuthology.orchestra.run.gibba012.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:32:45.556 INFO:teuthology.orchestra.run.gibba012.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:32:45.556 INFO:teuthology.orchestra.run.gibba012.stdout: "entity_id": "0", 2021-02-10T11:32:45.557 INFO:teuthology.orchestra.run.gibba012.stdout: "hostname": "gibba012", 2021-02-10T11:32:45.557 INFO:teuthology.orchestra.run.gibba012.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2021-02-10T11:32:45.557 INFO:teuthology.orchestra.run.gibba012.stdout: "pid": "125240", 2021-02-10T11:32:45.557 INFO:teuthology.orchestra.run.gibba012.stdout: "root": "/" 2021-02-10T11:32:45.557 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:32:45.557 INFO:teuthology.orchestra.run.gibba012.stdout: "dentry_count": 0, 2021-02-10T11:32:45.557 INFO:teuthology.orchestra.run.gibba012.stdout: "dentry_pinned_count": 0, 2021-02-10T11:32:45.557 INFO:teuthology.orchestra.run.gibba012.stdout: "id": 7689, 2021-02-10T11:32:45.557 INFO:teuthology.orchestra.run.gibba012.stdout: "inst": { 2021-02-10T11:32:45.557 INFO:teuthology.orchestra.run.gibba012.stdout: "name": { 2021-02-10T11:32:45.557 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "client", 2021-02-10T11:32:45.557 INFO:teuthology.orchestra.run.gibba012.stdout: "num": 7689 2021-02-10T11:32:45.557 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:32:45.557 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": { 2021-02-10T11:32:45.558 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:32:45.558 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:0", 2021-02-10T11:32:45.558 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 2150175872 2021-02-10T11:32:45.558 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:32:45.558 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:32:45.558 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": { 2021-02-10T11:32:45.558 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:32:45.558 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:0", 2021-02-10T11:32:45.558 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 2150175872 2021-02-10T11:32:45.558 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:32:45.558 INFO:teuthology.orchestra.run.gibba012.stdout: "inst_str": "client.7689 172.21.2.112:0/2150175872", 2021-02-10T11:32:45.558 INFO:teuthology.orchestra.run.gibba012.stdout: "addr_str": "172.21.2.112:0/2150175872", 2021-02-10T11:32:45.558 INFO:teuthology.orchestra.run.gibba012.stdout: "inode_count": 1, 2021-02-10T11:32:45.559 INFO:teuthology.orchestra.run.gibba012.stdout: "mds_epoch": 30, 2021-02-10T11:32:45.559 INFO:teuthology.orchestra.run.gibba012.stdout: "osd_epoch": 46, 2021-02-10T11:32:45.559 INFO:teuthology.orchestra.run.gibba012.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:32:45.559 INFO:teuthology.orchestra.run.gibba012.stdout: "blacklisted": false 2021-02-10T11:32:45.559 INFO:teuthology.orchestra.run.gibba012.stdout:} 2021-02-10T11:32:45.560 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:45.560 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:32:45.560 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:32:45.744 DEBUG:teuthology.orchestra.run.gibba012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:32:45.748 DEBUG:teuthology.orchestra.run.gibba020:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:32:45.894 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:32:45.895 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:32:45.895 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:45.895 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:32:46.240 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:32:46.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:46.663 7f353d231700 1 Processor -- start 2021-02-10T11:32:46.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:46.663 7f353d231700 1 -- start start 2021-02-10T11:32:46.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:46.664 7f353d231700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f35380e5070 0x7f35381b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:46.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:46.664 7f353d231700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f353807ed80 con 0x7f35381b3e40 2021-02-10T11:32:46.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:46.664 7f353d231700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f353807e670 con 0x7f35380e5470 2021-02-10T11:32:46.669 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:46.664 7f353d231700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f353807e940 con 0x7f35380e5070 2021-02-10T11:32:46.669 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:46.664 7f353759e700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f35381b3e40 0x7f353807de80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:48656/0 (socket says 172.21.2.112:48656) 2021-02-10T11:32:46.669 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:46.664 7f353759e700 1 -- 172.21.2.112:0/468066714 learned_addr learned my addr 172.21.2.112:0/468066714 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:32:46.669 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:46.664 7f3536d9d700 1 --2- 172.21.2.112:0/468066714 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f35380e5070 0x7f35381b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:46.669 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:46.665 7f3535d9b700 1 -- 172.21.2.112:0/468066714 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 715962555 0 0) 0x7f3530001500 con 0x7f35381b3e40 2021-02-10T11:32:46.669 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:46.665 7f3535d9b700 1 -- 172.21.2.112:0/468066714 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3520002fe0 con 0x7f35381b3e40 2021-02-10T11:32:46.669 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:46.665 7f3535d9b700 1 -- 172.21.2.112:0/468066714 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2658758553 0 0) 0x7f352c000ff0 con 0x7f35380e5470 2021-02-10T11:32:46.669 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:46.665 7f3536d9d700 1 -- 172.21.2.112:0/468066714 >> v1:172.21.2.120:6789/0 conn(0x7f35380e5470 legacy=0x7f353807d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:46.669 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:46.665 7f3536d9d700 1 -- 172.21.2.112:0/468066714 >> v1:172.21.2.112:6789/0 conn(0x7f35381b3e40 legacy=0x7f353807de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:46.669 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:46.665 7f3536d9d700 1 -- 172.21.2.112:0/468066714 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f353807eb30 con 0x7f35380e5070 2021-02-10T11:32:46.669 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:46.665 7f3536d9d700 1 --2- 172.21.2.112:0/468066714 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f35380e5070 0x7f35381b3820 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f3528006650 tx=0x7f3528009a80).ready entity=mon.2 client_cookie=6160d191b13bb4b0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:46.669 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:46.666 7f3535d9b700 1 -- 172.21.2.112:0/468066714 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f352800c0c0 con 0x7f35380e5070 2021-02-10T11:32:46.670 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:46.666 7f3535d9b700 1 -- 172.21.2.112:0/468066714 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f352800c340 con 0x7f35380e5070 2021-02-10T11:32:46.670 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:46.666 7f3535d9b700 1 -- 172.21.2.112:0/468066714 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f352800c860 con 0x7f35380e5070 2021-02-10T11:32:46.670 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:46.666 7f353d231700 1 -- 172.21.2.112:0/468066714 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f35380e5070 msgr2=0x7f35381b3820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:46.670 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:46.666 7f353d231700 1 --2- 172.21.2.112:0/468066714 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f35380e5070 0x7f35381b3820 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f3528006650 tx=0x7f3528009a80).stop 2021-02-10T11:32:46.670 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:46.666 7f353d231700 1 -- 172.21.2.112:0/468066714 shutdown_connections 2021-02-10T11:32:46.670 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:46.666 7f353d231700 1 --2- 172.21.2.112:0/468066714 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f35380e5070 0x7f35381b3820 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:46.670 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:46.666 7f353d231700 1 -- 172.21.2.112:0/468066714 >> 172.21.2.112:0/468066714 conn(0x7f35380ce690 msgr2=0x7f35380e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:46.670 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:46.667 7f353d231700 1 -- 172.21.2.112:0/468066714 shutdown_connections 2021-02-10T11:32:46.670 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:46.667 7f353d231700 1 -- 172.21.2.112:0/468066714 wait complete. 2021-02-10T11:32:46.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:46.681 7f353d231700 1 Processor -- start 2021-02-10T11:32:46.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:46.681 7f353d231700 1 -- start start 2021-02-10T11:32:46.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:46.682 7f353d231700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3538192960 0x7f3538192d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:46.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:46.682 7f353d231700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f35380ad110 0x7f35380ad510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:46.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:46.682 7f353d231700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f35380adb30 0x7f353808e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:46.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:46.682 7f353d231700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35381a03e0 con 0x7f3538192960 2021-02-10T11:32:46.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:46.682 7f353d231700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3538193380 con 0x7f35380ad110 2021-02-10T11:32:46.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:46.682 7f353d231700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f353808ea80 con 0x7f35380adb30 2021-02-10T11:32:46.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:46.682 7f3536d9d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3538192960 0x7f3538192d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:46.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:46.682 7f3536d9d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3538192960 0x7f3538192d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:49198/0 (socket says 172.21.2.112:49198) 2021-02-10T11:32:46.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:46.682 7f3536d9d700 1 -- 172.21.2.112:0/1309116070 learned_addr learned my addr 172.21.2.112:0/1309116070 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:32:46.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:46.682 7f353759e700 1 --2- 172.21.2.112:0/1309116070 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f35380adb30 0x7f353808e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:46.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:46.682 7f353659c700 1 --2- 172.21.2.112:0/1309116070 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f35380ad110 0x7f35380ad510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:46.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:46.682 7f3536d9d700 1 -- 172.21.2.112:0/1309116070 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f35380adb30 msgr2=0x7f353808e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:46.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:46.682 7f3536d9d700 1 --2- 172.21.2.112:0/1309116070 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f35380adb30 0x7f353808e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:46.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:46.682 7f3536d9d700 1 -- 172.21.2.112:0/1309116070 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f35380ad110 msgr2=0x7f35380ad510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:46.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:46.682 7f3536d9d700 1 --2- 172.21.2.112:0/1309116070 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f35380ad110 0x7f35380ad510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:46.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:46.682 7f3536d9d700 1 -- 172.21.2.112:0/1309116070 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f353808eca0 con 0x7f3538192960 2021-02-10T11:32:46.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:46.683 7f3536d9d700 1 --2- 172.21.2.112:0/1309116070 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3538192960 0x7f3538192d60 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 rx=0x7f35280096a0 tx=0x7f352800b9b0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:46.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:46.683 7f35277fe700 1 -- 172.21.2.112:0/1309116070 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3528000940 con 0x7f3538192960 2021-02-10T11:32:46.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:46.683 7f35277fe700 1 -- 172.21.2.112:0/1309116070 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35280093a0 con 0x7f3538192960 2021-02-10T11:32:46.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:46.683 7f35277fe700 1 -- 172.21.2.112:0/1309116070 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3528002220 con 0x7f3538192960 2021-02-10T11:32:46.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:46.683 7f353d231700 1 -- 172.21.2.112:0/1309116070 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f353808eca0 con 0x7f3538192960 2021-02-10T11:32:46.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:46.683 7f353d231700 1 -- 172.21.2.112:0/1309116070 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f353808f230 con 0x7f3538192960 2021-02-10T11:32:46.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:46.683 7f35277fe700 1 -- 172.21.2.112:0/1309116070 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3528002240 con 0x7f3538192960 2021-02-10T11:32:46.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:46.683 7f35277fe700 1 --2- 172.21.2.112:0/1309116070 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f351c0186c0 0x7f351c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:46.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:46.684 7f35277fe700 1 -- 172.21.2.112:0/1309116070 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f35280325c0 con 0x7f3538192960 2021-02-10T11:32:46.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:46.684 7f353659c700 1 --2- 172.21.2.112:0/1309116070 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f351c0186c0 0x7f351c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:46.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:46.684 7f353659c700 1 --2- 172.21.2.112:0/1309116070 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f351c0186c0 0x7f351c01abc0 crc :-1 s=READY pgs=582 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:46.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:46.714 7f353d231700 1 -- 172.21.2.112:0/1309116070 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3538107120 con 0x7f3538192960 2021-02-10T11:32:46.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:46.717 7f35277fe700 1 -- 172.21.2.112:0/1309116070 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3528002010 con 0x7f3538192960 2021-02-10T11:32:46.830 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:46.843 7f353d231700 1 -- 172.21.2.112:0/1309116070 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f353807ed80 con 0x7f3538192960 2021-02-10T11:32:46.831 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:46.844 7f35277fe700 1 -- 172.21.2.112:0/1309116070 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v46) v1 ==== 74+0+13993 (secure 0 0 0) 0x7f3528002220 con 0x7f3538192960 2021-02-10T11:32:46.831 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:32:46.831 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":46,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:32:32.871643","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:32:29.843136","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":9,"pool_name":"cephfs_data","create_time":"2021-02-10 11:32:30.914946","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:32:46.833 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:46.846 7f353d231700 1 -- 172.21.2.112:0/1309116070 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f351c0186c0 msgr2=0x7f351c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:46.833 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:46.846 7f353d231700 1 --2- 172.21.2.112:0/1309116070 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f351c0186c0 0x7f351c01abc0 crc :-1 s=READY pgs=582 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:46.833 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:46.846 7f353d231700 1 -- 172.21.2.112:0/1309116070 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3538192960 msgr2=0x7f3538192d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:46.833 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:46.846 7f353d231700 1 --2- 172.21.2.112:0/1309116070 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3538192960 0x7f3538192d60 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 rx=0x7f35280096a0 tx=0x7f352800b9b0).stop 2021-02-10T11:32:46.833 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:46.846 7f353d231700 1 -- 172.21.2.112:0/1309116070 shutdown_connections 2021-02-10T11:32:46.833 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:46.846 7f353d231700 1 --2- 172.21.2.112:0/1309116070 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f351c0186c0 0x7f351c01abc0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:46.833 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:46.846 7f353d231700 1 --2- 172.21.2.112:0/1309116070 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f35380ad110 0x7f35380ad510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:46.833 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:46.846 7f353d231700 1 --2- 172.21.2.112:0/1309116070 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f35380adb30 0x7f353808e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:46.833 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:46.846 7f353d231700 1 --2- 172.21.2.112:0/1309116070 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3538192960 0x7f3538192d60 unknown :-1 s=CLOSED pgs=1152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:46.834 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:46.846 7f353d231700 1 -- 172.21.2.112:0/1309116070 >> 172.21.2.112:0/1309116070 conn(0x7f35380891f0 msgr2=0x7f35381a0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:46.834 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:46.846 7f353d231700 1 -- 172.21.2.112:0/1309116070 shutdown_connections 2021-02-10T11:32:46.834 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:46.846 7f353d231700 1 -- 172.21.2.112:0/1309116070 wait complete. 2021-02-10T11:32:46.842 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:13:57.508758', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:32:29.843136', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:32:30.914946', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '46', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:32:46.842 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:32:47.272 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.269 7f35e57b9700 1 Processor -- start 2021-02-10T11:32:47.272 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.269 7f35e57b9700 1 -- start start 2021-02-10T11:32:47.273 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.269 7f35e57b9700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f35e00f05f0 0x7f35e00f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:47.273 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.269 7f35e57b9700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35e01b0c70 con 0x7f35e01b0870 2021-02-10T11:32:47.273 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.269 7f35e57b9700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35e00f0b50 con 0x7f35e01b0470 2021-02-10T11:32:47.273 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.269 7f35e57b9700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35e00e5ea0 con 0x7f35e00f05f0 2021-02-10T11:32:47.273 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.270 7f35dfd75700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f35e00f05f0 0x7f35e00f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:47.274 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.270 7f35ded73700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f35e01b0870 0x7f35e00efe00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:48666/0 (socket says 172.21.2.112:48666) 2021-02-10T11:32:47.274 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.270 7f35ded73700 1 -- 172.21.2.112:0/1613656826 learned_addr learned my addr 172.21.2.112:0/1613656826 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:32:47.274 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.270 7f35dfd75700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f35e00f05f0 0x7f35e00f20e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:49206/0 (socket says 172.21.2.112:49206) 2021-02-10T11:32:47.274 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.270 7f35dfd75700 1 -- 172.21.2.112:0/1613656826 >> v1:172.21.2.120:6789/0 conn(0x7f35e01b0470 legacy=0x7f35e00ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:47.274 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.270 7f35dfd75700 1 -- 172.21.2.112:0/1613656826 >> v1:172.21.2.112:6789/0 conn(0x7f35e01b0870 legacy=0x7f35e00efe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:47.274 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.270 7f35dfd75700 1 -- 172.21.2.112:0/1613656826 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35e00e52e0 con 0x7f35e00f05f0 2021-02-10T11:32:47.274 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.270 7f35de572700 1 -- 172.21.2.112:0/1613656826 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3136802208 0 0) 0x7f35d4000ce0 con 0x7f35e01b0870 2021-02-10T11:32:47.274 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.270 7f35dfd75700 1 --2- 172.21.2.112:0/1613656826 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f35e00f05f0 0x7f35e00f20e0 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 rx=0x7f35d8006e30 tx=0x7f35d8009e60).ready entity=mon.0 client_cookie=4eedc7997de0966e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:47.274 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.270 7f35de572700 1 -- 172.21.2.112:0/1613656826 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f35d8006a50 con 0x7f35e00f05f0 2021-02-10T11:32:47.274 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.271 7f35de572700 1 -- 172.21.2.112:0/1613656826 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35d800b300 con 0x7f35e00f05f0 2021-02-10T11:32:47.274 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.271 7f35de572700 1 -- 172.21.2.112:0/1613656826 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f35d800b750 con 0x7f35e00f05f0 2021-02-10T11:32:47.275 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.271 7f35e57b9700 1 -- 172.21.2.112:0/1613656826 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f35e00f05f0 msgr2=0x7f35e00f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:47.275 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.271 7f35e57b9700 1 --2- 172.21.2.112:0/1613656826 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f35e00f05f0 0x7f35e00f20e0 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 rx=0x7f35d8006e30 tx=0x7f35d8009e60).stop 2021-02-10T11:32:47.275 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.271 7f35e57b9700 1 -- 172.21.2.112:0/1613656826 shutdown_connections 2021-02-10T11:32:47.275 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.271 7f35e57b9700 1 --2- 172.21.2.112:0/1613656826 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f35e00f05f0 0x7f35e00f20e0 unknown :-1 s=CLOSED pgs=1154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:47.275 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.271 7f35e57b9700 1 -- 172.21.2.112:0/1613656826 >> 172.21.2.112:0/1613656826 conn(0x7f35e00ce3e0 msgr2=0x7f35e00e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:47.275 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.271 7f35e57b9700 1 -- 172.21.2.112:0/1613656826 shutdown_connections 2021-02-10T11:32:47.275 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.271 7f35e57b9700 1 -- 172.21.2.112:0/1613656826 wait complete. 2021-02-10T11:32:47.276 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.286 7f35e57b9700 1 Processor -- start 2021-02-10T11:32:47.277 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.286 7f35e57b9700 1 -- start start 2021-02-10T11:32:47.277 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.286 7f35e57b9700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f35e00bdbf0 0x7f35e01a7450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:47.277 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.286 7f35e57b9700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f35e00bdff0 0x7f35e01a7a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:47.277 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.286 7f35e57b9700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f35e00abcc0 0x7f35e00ac0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:47.277 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.286 7f35e57b9700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35e00be590 con 0x7f35e00bdbf0 2021-02-10T11:32:47.277 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.286 7f35e57b9700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35e01a8090 con 0x7f35e00abcc0 2021-02-10T11:32:47.277 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.286 7f35e57b9700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35e01a82b0 con 0x7f35e00bdff0 2021-02-10T11:32:47.277 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.286 7f35df574700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f35e00bdbf0 0x7f35e01a7450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:47.277 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.286 7f35df574700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f35e00bdbf0 0x7f35e01a7450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:49208/0 (socket says 172.21.2.112:49208) 2021-02-10T11:32:47.277 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.286 7f35df574700 1 -- 172.21.2.112:0/309061692 learned_addr learned my addr 172.21.2.112:0/309061692 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:32:47.277 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.286 7f35df574700 1 -- 172.21.2.112:0/309061692 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f35e00bdff0 msgr2=0x7f35e01a7a70 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:32:47.277 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.286 7f35df574700 1 --2- 172.21.2.112:0/309061692 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f35e00bdff0 0x7f35e01a7a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:47.278 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.286 7f35df574700 1 -- 172.21.2.112:0/309061692 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f35e00abcc0 msgr2=0x7f35e00ac0c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:32:47.278 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.286 7f35df574700 1 --2- 172.21.2.112:0/309061692 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f35e00abcc0 0x7f35e00ac0c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:47.278 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.287 7f35df574700 1 -- 172.21.2.112:0/309061692 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35e00ac750 con 0x7f35e00bdbf0 2021-02-10T11:32:47.278 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.287 7f35df574700 1 --2- 172.21.2.112:0/309061692 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f35e00bdbf0 0x7f35e01a7450 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 rx=0x7f35d000b5b0 tx=0x7f35d000b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:47.278 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.287 7f35c7fff700 1 -- 172.21.2.112:0/309061692 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f35d000bd40 con 0x7f35e00bdbf0 2021-02-10T11:32:47.279 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.287 7f35c7fff700 1 -- 172.21.2.112:0/309061692 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35d00009b0 con 0x7f35e00bdbf0 2021-02-10T11:32:47.279 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.287 7f35c7fff700 1 -- 172.21.2.112:0/309061692 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f35d0006210 con 0x7f35e00bdbf0 2021-02-10T11:32:47.279 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.287 7f35e57b9700 1 -- 172.21.2.112:0/309061692 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35e00ac750 con 0x7f35e00bdbf0 2021-02-10T11:32:47.280 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.287 7f35e57b9700 1 -- 172.21.2.112:0/309061692 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35e00ac750 con 0x7f35e00bdbf0 2021-02-10T11:32:47.280 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.287 7f35c7fff700 1 -- 172.21.2.112:0/309061692 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f35d0006020 con 0x7f35e00bdbf0 2021-02-10T11:32:47.280 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.288 7f35c7fff700 1 --2- 172.21.2.112:0/309061692 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f35c00186c0 0x7f35c001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:47.281 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.288 7f35c7fff700 1 -- 172.21.2.112:0/309061692 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f35d0032570 con 0x7f35e00bdbf0 2021-02-10T11:32:47.281 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.288 7f35ded73700 1 --2- 172.21.2.112:0/309061692 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f35c00186c0 0x7f35c001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:47.281 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.289 7f35ded73700 1 --2- 172.21.2.112:0/309061692 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f35c00186c0 0x7f35c001abc0 crc :-1 s=READY pgs=583 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:47.306 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.319 7f35e57b9700 1 -- 172.21.2.112:0/309061692 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35e01a8090 con 0x7f35e00bdbf0 2021-02-10T11:32:47.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.322 7f35c7fff700 1 -- 172.21.2.112:0/309061692 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f35d0006230 con 0x7f35e00bdbf0 2021-02-10T11:32:47.435 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.448 7f35e57b9700 1 -- 172.21.2.112:0/309061692 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f35e01a8090 con 0x7f35e00bdbf0 2021-02-10T11:32:47.436 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.449 7f35c7fff700 1 -- 172.21.2.112:0/309061692 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v46) v1 ==== 92+0+10 (secure 0 0 0) 0x7f35d0006000 con 0x7f35e00bdbf0 2021-02-10T11:32:47.436 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:32:47.437 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.450 7f35e57b9700 1 -- 172.21.2.112:0/309061692 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f35c00186c0 msgr2=0x7f35c001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:47.438 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.450 7f35e57b9700 1 --2- 172.21.2.112:0/309061692 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f35c00186c0 0x7f35c001abc0 crc :-1 s=READY pgs=583 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:47.438 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.450 7f35e57b9700 1 -- 172.21.2.112:0/309061692 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f35e00bdbf0 msgr2=0x7f35e01a7450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:47.438 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.450 7f35e57b9700 1 --2- 172.21.2.112:0/309061692 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f35e00bdbf0 0x7f35e01a7450 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 rx=0x7f35d000b5b0 tx=0x7f35d000b950).stop 2021-02-10T11:32:47.438 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.451 7f35e57b9700 1 -- 172.21.2.112:0/309061692 shutdown_connections 2021-02-10T11:32:47.438 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.451 7f35e57b9700 1 --2- 172.21.2.112:0/309061692 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f35c00186c0 0x7f35c001abc0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:47.438 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.451 7f35e57b9700 1 --2- 172.21.2.112:0/309061692 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f35e00abcc0 0x7f35e00ac0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:47.438 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.451 7f35e57b9700 1 --2- 172.21.2.112:0/309061692 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f35e00bdff0 0x7f35e01a7a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:47.438 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.451 7f35e57b9700 1 --2- 172.21.2.112:0/309061692 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f35e00bdbf0 0x7f35e01a7450 unknown :-1 s=CLOSED pgs=1155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:47.438 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.451 7f35e57b9700 1 -- 172.21.2.112:0/309061692 >> 172.21.2.112:0/309061692 conn(0x7f35e0095a70 msgr2=0x7f35e00bf040 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:47.439 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.451 7f35e57b9700 1 -- 172.21.2.112:0/309061692 shutdown_connections 2021-02-10T11:32:47.439 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.451 7f35e57b9700 1 -- 172.21.2.112:0/309061692 wait complete. 2021-02-10T11:32:47.446 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:32:47.876 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.872 7f5fa207a700 1 Processor -- start 2021-02-10T11:32:47.876 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.872 7f5fa207a700 1 -- start start 2021-02-10T11:32:47.876 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.872 7f5fa207a700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f5f9c0f05f0 0x7f5f9c0f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:47.876 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.872 7f5fa207a700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f9c1b0c70 con 0x7f5f9c1b0870 2021-02-10T11:32:47.876 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.872 7f5fa207a700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f9c0f0b50 con 0x7f5f9c1b0470 2021-02-10T11:32:47.876 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.872 7f5fa207a700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f9c0e5ea0 con 0x7f5f9c0f05f0 2021-02-10T11:32:47.876 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.873 7f5f9bfff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f5f9c0f05f0 0x7f5f9c0f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:47.876 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.873 7f5f9bfff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f5f9c0f05f0 0x7f5f9c0f20e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:49220/0 (socket says 172.21.2.112:49220) 2021-02-10T11:32:47.876 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.873 7f5f9bfff700 1 -- 172.21.2.112:0/102243948 learned_addr learned my addr 172.21.2.112:0/102243948 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:32:47.876 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.873 7f5f9bfff700 1 -- 172.21.2.112:0/102243948 >> v1:172.21.2.120:6789/0 conn(0x7f5f9c1b0470 legacy=0x7f5f9c0ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:47.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.873 7f5f9bfff700 1 -- 172.21.2.112:0/102243948 >> v1:172.21.2.112:6789/0 conn(0x7f5f9c1b0870 legacy=0x7f5f9c0efe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:47.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.873 7f5f9bfff700 1 -- 172.21.2.112:0/102243948 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f9c0e52e0 con 0x7f5f9c0f05f0 2021-02-10T11:32:47.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.873 7f5f9bfff700 1 --2- 172.21.2.112:0/102243948 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5f9c0f05f0 0x7f5f9c0f20e0 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 rx=0x7f5f94006f00 tx=0x7f5f94009e90).ready entity=mon.0 client_cookie=75654e2bc64add62 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:47.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.873 7f5f9a7fc700 1 -- 172.21.2.112:0/102243948 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5f9400b300 con 0x7f5f9c0f05f0 2021-02-10T11:32:47.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.873 7f5f9a7fc700 1 -- 172.21.2.112:0/102243948 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f9400b640 con 0x7f5f9c0f05f0 2021-02-10T11:32:47.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.873 7f5f9a7fc700 1 -- 172.21.2.112:0/102243948 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5f9400bb20 con 0x7f5f9c0f05f0 2021-02-10T11:32:47.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.873 7f5fa207a700 1 -- 172.21.2.112:0/102243948 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5f9c0f05f0 msgr2=0x7f5f9c0f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:47.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.873 7f5fa207a700 1 --2- 172.21.2.112:0/102243948 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5f9c0f05f0 0x7f5f9c0f20e0 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 rx=0x7f5f94006f00 tx=0x7f5f94009e90).stop 2021-02-10T11:32:47.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.874 7f5fa207a700 1 -- 172.21.2.112:0/102243948 shutdown_connections 2021-02-10T11:32:47.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.874 7f5fa207a700 1 --2- 172.21.2.112:0/102243948 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5f9c0f05f0 0x7f5f9c0f20e0 unknown :-1 s=CLOSED pgs=1157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:47.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.874 7f5fa207a700 1 -- 172.21.2.112:0/102243948 >> 172.21.2.112:0/102243948 conn(0x7f5f9c0ce3e0 msgr2=0x7f5f9c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:47.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.874 7f5fa207a700 1 -- 172.21.2.112:0/102243948 shutdown_connections 2021-02-10T11:32:47.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.874 7f5fa207a700 1 -- 172.21.2.112:0/102243948 wait complete. 2021-02-10T11:32:47.878 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.889 7f5fa207a700 1 Processor -- start 2021-02-10T11:32:47.879 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.889 7f5fa207a700 1 -- start start 2021-02-10T11:32:47.879 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.889 7f5fa207a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5f9c087510 0x7f5f9c0aedc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:47.879 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.889 7f5fa207a700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5f9c1c68f0 0x7f5f9c1b1f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:47.879 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.889 7f5fa207a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5f9c08c4d0 0x7f5f9c1b18a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:47.879 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.889 7f5fa207a700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f9c0a2d50 con 0x7f5f9c087510 2021-02-10T11:32:47.879 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.889 7f5fa207a700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f9c0a2f70 con 0x7f5f9c1c68f0 2021-02-10T11:32:47.879 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.889 7f5fa207a700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f9c0a3190 con 0x7f5f9c08c4d0 2021-02-10T11:32:47.879 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.890 7f5f9b7fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5f9c087510 0x7f5f9c0aedc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:47.880 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.890 7f5f9b7fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5f9c087510 0x7f5f9c0aedc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:49224/0 (socket says 172.21.2.112:49224) 2021-02-10T11:32:47.880 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.890 7f5f9b7fe700 1 -- 172.21.2.112:0/1058721504 learned_addr learned my addr 172.21.2.112:0/1058721504 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:32:47.880 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.890 7f5f9bfff700 1 --2- 172.21.2.112:0/1058721504 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5f9c08c4d0 0x7f5f9c1b18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:47.880 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.890 7f5f9affd700 1 --2- 172.21.2.112:0/1058721504 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5f9c1c68f0 0x7f5f9c1b1f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:47.880 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.890 7f5f9b7fe700 1 -- 172.21.2.112:0/1058721504 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5f9c08c4d0 msgr2=0x7f5f9c1b18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:47.880 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.890 7f5f9b7fe700 1 --2- 172.21.2.112:0/1058721504 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5f9c08c4d0 0x7f5f9c1b18a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:47.880 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.890 7f5f9b7fe700 1 -- 172.21.2.112:0/1058721504 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5f9c1c68f0 msgr2=0x7f5f9c1b1f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:47.880 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.890 7f5f9b7fe700 1 --2- 172.21.2.112:0/1058721504 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5f9c1c68f0 0x7f5f9c1b1f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:47.880 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.890 7f5f9b7fe700 1 -- 172.21.2.112:0/1058721504 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f9c0a33b0 con 0x7f5f9c087510 2021-02-10T11:32:47.880 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.890 7f5f9b7fe700 1 --2- 172.21.2.112:0/1058721504 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5f9c087510 0x7f5f9c0aedc0 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 rx=0x7f5f8c00b5b0 tx=0x7f5f8c00b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:47.880 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.890 7f5f83fff700 1 -- 172.21.2.112:0/1058721504 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5f8c00bd40 con 0x7f5f9c087510 2021-02-10T11:32:47.880 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.890 7f5f83fff700 1 -- 172.21.2.112:0/1058721504 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f8c006240 con 0x7f5f9c087510 2021-02-10T11:32:47.880 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.890 7f5f83fff700 1 -- 172.21.2.112:0/1058721504 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5f8c005720 con 0x7f5f9c087510 2021-02-10T11:32:47.881 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.890 7f5fa207a700 1 -- 172.21.2.112:0/1058721504 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f9c0a33b0 con 0x7f5f9c087510 2021-02-10T11:32:47.881 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.890 7f5fa207a700 1 -- 172.21.2.112:0/1058721504 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f9c0afca0 con 0x7f5f9c087510 2021-02-10T11:32:47.881 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.891 7f5f83fff700 1 -- 172.21.2.112:0/1058721504 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5f8c0054a0 con 0x7f5f9c087510 2021-02-10T11:32:47.881 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.891 7f5f83fff700 1 --2- 172.21.2.112:0/1058721504 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5f7c0186d0 0x7f5f7c01abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:47.881 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.891 7f5f83fff700 1 -- 172.21.2.112:0/1058721504 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f5f8c0327f0 con 0x7f5f9c087510 2021-02-10T11:32:47.881 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.891 7f5f9affd700 1 --2- 172.21.2.112:0/1058721504 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5f7c0186d0 0x7f5f7c01abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:47.881 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.892 7f5f9affd700 1 --2- 172.21.2.112:0/1058721504 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5f7c0186d0 0x7f5f7c01abd0 crc :-1 s=READY pgs=584 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:47.909 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.922 7f5fa207a700 1 -- 172.21.2.112:0/1058721504 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f9c07e6b0 con 0x7f5f9c087510 2021-02-10T11:32:47.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:47.925 7f5f83fff700 1 -- 172.21.2.112:0/1058721504 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5f8c000c60 con 0x7f5f9c087510 2021-02-10T11:32:48.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:48.054 7f5fa207a700 1 -- 172.21.2.112:0/1058721504 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f5f9c0ce3e0 con 0x7f5f9c087510 2021-02-10T11:32:48.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:48.055 7f5f83fff700 1 -- 172.21.2.112:0/1058721504 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v46) v1 ==== 104+0+10 (secure 0 0 0) 0x7f5f8c0054a0 con 0x7f5f9c087510 2021-02-10T11:32:48.044 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:32:48.045 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:48.058 7f5fa207a700 1 -- 172.21.2.112:0/1058721504 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5f7c0186d0 msgr2=0x7f5f7c01abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:48.045 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:48.058 7f5fa207a700 1 --2- 172.21.2.112:0/1058721504 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5f7c0186d0 0x7f5f7c01abd0 crc :-1 s=READY pgs=584 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:48.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:48.058 7f5fa207a700 1 -- 172.21.2.112:0/1058721504 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5f9c087510 msgr2=0x7f5f9c0aedc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:48.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:48.058 7f5fa207a700 1 --2- 172.21.2.112:0/1058721504 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5f9c087510 0x7f5f9c0aedc0 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 rx=0x7f5f8c00b5b0 tx=0x7f5f8c00b950).stop 2021-02-10T11:32:48.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:48.059 7f5fa207a700 1 -- 172.21.2.112:0/1058721504 shutdown_connections 2021-02-10T11:32:48.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:48.059 7f5fa207a700 1 --2- 172.21.2.112:0/1058721504 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5f7c0186d0 0x7f5f7c01abd0 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:48.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:48.059 7f5fa207a700 1 --2- 172.21.2.112:0/1058721504 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5f9c1c68f0 0x7f5f9c1b1f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:48.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:48.059 7f5fa207a700 1 --2- 172.21.2.112:0/1058721504 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5f9c08c4d0 0x7f5f9c1b18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:48.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:48.059 7f5fa207a700 1 --2- 172.21.2.112:0/1058721504 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5f9c087510 0x7f5f9c0aedc0 unknown :-1 s=CLOSED pgs=1158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:48.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:48.059 7f5fa207a700 1 -- 172.21.2.112:0/1058721504 >> 172.21.2.112:0/1058721504 conn(0x7f5f9c0886d0 msgr2=0x7f5f9c081c40 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:48.047 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:48.059 7f5fa207a700 1 -- 172.21.2.112:0/1058721504 shutdown_connections 2021-02-10T11:32:48.047 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:48.059 7f5fa207a700 1 -- 172.21.2.112:0/1058721504 wait complete. 2021-02-10T11:32:48.054 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:32:48.484 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:48.481 7ff638001700 1 Processor -- start 2021-02-10T11:32:48.484 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:48.481 7ff638001700 1 -- start start 2021-02-10T11:32:48.484 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:48.481 7ff638001700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7ff6300f05f0 0x7ff6300f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:48.484 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:48.481 7ff638001700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6301b0c70 con 0x7ff6301b0470 2021-02-10T11:32:48.484 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:48.481 7ff638001700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6300f0b50 con 0x7ff6301b0870 2021-02-10T11:32:48.485 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:48.481 7ff638001700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6300e5ea0 con 0x7ff6300f05f0 2021-02-10T11:32:48.485 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:48.482 7ff63661e700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7ff6300f05f0 0x7ff6300f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:48.485 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:48.482 7ff63661e700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7ff6300f05f0 0x7ff6300f20e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:49230/0 (socket says 172.21.2.112:49230) 2021-02-10T11:32:48.485 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:48.482 7ff63661e700 1 -- 172.21.2.112:0/1350687338 learned_addr learned my addr 172.21.2.112:0/1350687338 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:32:48.485 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:48.482 7ff63661e700 1 -- 172.21.2.112:0/1350687338 >> v1:172.21.2.120:6789/0 conn(0x7ff6301b0870 legacy=0x7ff6300efe00 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:32:48.485 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:48.482 7ff63661e700 1 -- 172.21.2.112:0/1350687338 >> v1:172.21.2.112:6789/0 conn(0x7ff6301b0470 legacy=0x7ff6300ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:48.485 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:48.482 7ff63661e700 1 -- 172.21.2.112:0/1350687338 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6300e52e0 con 0x7ff6300f05f0 2021-02-10T11:32:48.485 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:48.482 7ff63661e700 1 --2- 172.21.2.112:0/1350687338 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff6300f05f0 0x7ff6300f20e0 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 rx=0x7ff628006f00 tx=0x7ff628009f10).ready entity=mon.0 client_cookie=e353d98a1e923b35 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:48.485 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:48.482 7ff634e1b700 1 -- 172.21.2.112:0/1350687338 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff62800b3b0 con 0x7ff6300f05f0 2021-02-10T11:32:48.485 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:48.482 7ff634e1b700 1 -- 172.21.2.112:0/1350687338 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff628000d30 con 0x7ff6300f05f0 2021-02-10T11:32:48.485 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:48.482 7ff634e1b700 1 -- 172.21.2.112:0/1350687338 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff62800b8b0 con 0x7ff6300f05f0 2021-02-10T11:32:48.485 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:48.482 7ff638001700 1 -- 172.21.2.112:0/1350687338 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff6300f05f0 msgr2=0x7ff6300f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:48.485 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:48.482 7ff638001700 1 --2- 172.21.2.112:0/1350687338 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff6300f05f0 0x7ff6300f20e0 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 rx=0x7ff628006f00 tx=0x7ff628009f10).stop 2021-02-10T11:32:48.486 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:48.483 7ff638001700 1 -- 172.21.2.112:0/1350687338 shutdown_connections 2021-02-10T11:32:48.486 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:48.483 7ff638001700 1 --2- 172.21.2.112:0/1350687338 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff6300f05f0 0x7ff6300f20e0 unknown :-1 s=CLOSED pgs=1160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:48.486 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:48.483 7ff638001700 1 -- 172.21.2.112:0/1350687338 >> 172.21.2.112:0/1350687338 conn(0x7ff6300ce3e0 msgr2=0x7ff6300e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:48.486 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:48.483 7ff638001700 1 -- 172.21.2.112:0/1350687338 shutdown_connections 2021-02-10T11:32:48.486 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:48.483 7ff638001700 1 -- 172.21.2.112:0/1350687338 wait complete. 2021-02-10T11:32:48.487 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:48.497 7ff638001700 1 Processor -- start 2021-02-10T11:32:48.487 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:48.497 7ff638001700 1 -- start start 2021-02-10T11:32:48.488 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:48.498 7ff638001700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff630087510 0x7ff6300aedc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:48.488 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:48.498 7ff638001700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff6301c68f0 0x7ff6301b1f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:48.488 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:48.498 7ff638001700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff63008c4d0 0x7ff6301b18a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:48.488 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:48.498 7ff638001700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6300a2d50 con 0x7ff6301c68f0 2021-02-10T11:32:48.488 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:48.498 7ff638001700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6300a2f70 con 0x7ff630087510 2021-02-10T11:32:48.488 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:48.498 7ff638001700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6300a3190 con 0x7ff63008c4d0 2021-02-10T11:32:48.488 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:48.498 7ff63561c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff6301c68f0 0x7ff6301b1f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:48.488 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:48.498 7ff63561c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff6301c68f0 0x7ff6301b1f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:49238/0 (socket says 172.21.2.112:49238) 2021-02-10T11:32:48.488 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:48.498 7ff63561c700 1 -- 172.21.2.112:0/4037616978 learned_addr learned my addr 172.21.2.112:0/4037616978 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:32:48.488 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:48.498 7ff63561c700 1 -- 172.21.2.112:0/4037616978 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff63008c4d0 msgr2=0x7ff6301b18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:48.488 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:48.498 7ff63561c700 1 --2- 172.21.2.112:0/4037616978 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff63008c4d0 0x7ff6301b18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:48.488 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:48.498 7ff63561c700 1 -- 172.21.2.112:0/4037616978 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff630087510 msgr2=0x7ff6300aedc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:48.488 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:48.498 7ff63561c700 1 --2- 172.21.2.112:0/4037616978 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff630087510 0x7ff6300aedc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:48.489 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:48.498 7ff63561c700 1 -- 172.21.2.112:0/4037616978 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6300a33b0 con 0x7ff6301c68f0 2021-02-10T11:32:48.489 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:48.499 7ff63561c700 1 --2- 172.21.2.112:0/4037616978 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff6301c68f0 0x7ff6301b1f60 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 rx=0x7ff62400b5b0 tx=0x7ff62400b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:48.489 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:48.499 7ff6227fc700 1 -- 172.21.2.112:0/4037616978 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff62400bd40 con 0x7ff6301c68f0 2021-02-10T11:32:48.489 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:48.499 7ff6227fc700 1 -- 172.21.2.112:0/4037616978 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff624006020 con 0x7ff6301c68f0 2021-02-10T11:32:48.489 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:48.499 7ff6227fc700 1 -- 172.21.2.112:0/4037616978 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff6240054a0 con 0x7ff6301c68f0 2021-02-10T11:32:48.489 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:48.499 7ff638001700 1 -- 172.21.2.112:0/4037616978 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6300a33b0 con 0x7ff6301c68f0 2021-02-10T11:32:48.489 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:48.499 7ff638001700 1 -- 172.21.2.112:0/4037616978 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6300a33b0 con 0x7ff6301c68f0 2021-02-10T11:32:48.489 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:48.499 7ff6227fc700 1 -- 172.21.2.112:0/4037616978 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff624006200 con 0x7ff6301c68f0 2021-02-10T11:32:48.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:48.500 7ff6227fc700 1 --2- 172.21.2.112:0/4037616978 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff6180186c0 0x7ff61801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:48.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:48.500 7ff6227fc700 1 -- 172.21.2.112:0/4037616978 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5573+0+0 (secure 0 0 0) 0x7ff624032580 con 0x7ff6301c68f0 2021-02-10T11:32:48.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:48.500 7ff635e1d700 1 --2- 172.21.2.112:0/4037616978 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff6180186c0 0x7ff61801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:48.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:48.501 7ff635e1d700 1 --2- 172.21.2.112:0/4037616978 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff6180186c0 0x7ff61801abc0 crc :-1 s=READY pgs=585 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:48.518 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:48.531 7ff638001700 1 -- 172.21.2.112:0/4037616978 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff63007e6b0 con 0x7ff6301c68f0 2021-02-10T11:32:48.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:48.534 7ff6227fc700 1 -- 172.21.2.112:0/4037616978 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff624030d30 con 0x7ff6301c68f0 2021-02-10T11:32:48.651 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:48.664 7ff638001700 1 -- 172.21.2.112:0/4037616978 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7ff63007e6b0 con 0x7ff6301c68f0 2021-02-10T11:32:48.651 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:48.664 7ff6227fc700 1 -- 172.21.2.112:0/4037616978 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v46) v1 ==== 100+0+10 (secure 0 0 0) 0x7ff62400bd40 con 0x7ff6301c68f0 2021-02-10T11:32:48.652 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:32:48.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:48.666 7ff638001700 1 -- 172.21.2.112:0/4037616978 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff6180186c0 msgr2=0x7ff61801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:48.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:48.666 7ff638001700 1 --2- 172.21.2.112:0/4037616978 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff6180186c0 0x7ff61801abc0 crc :-1 s=READY pgs=585 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:48.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:48.666 7ff638001700 1 -- 172.21.2.112:0/4037616978 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff6301c68f0 msgr2=0x7ff6301b1f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:48.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:48.666 7ff638001700 1 --2- 172.21.2.112:0/4037616978 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff6301c68f0 0x7ff6301b1f60 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 rx=0x7ff62400b5b0 tx=0x7ff62400b950).stop 2021-02-10T11:32:48.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:48.666 7ff638001700 1 -- 172.21.2.112:0/4037616978 shutdown_connections 2021-02-10T11:32:48.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:48.666 7ff638001700 1 --2- 172.21.2.112:0/4037616978 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff6180186c0 0x7ff61801abc0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:48.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:48.666 7ff638001700 1 --2- 172.21.2.112:0/4037616978 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff63008c4d0 0x7ff6301b18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:48.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:48.666 7ff638001700 1 --2- 172.21.2.112:0/4037616978 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff6301c68f0 0x7ff6301b1f60 unknown :-1 s=CLOSED pgs=1161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:48.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:48.666 7ff638001700 1 --2- 172.21.2.112:0/4037616978 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff630087510 0x7ff6300aedc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:48.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:48.666 7ff638001700 1 -- 172.21.2.112:0/4037616978 >> 172.21.2.112:0/4037616978 conn(0x7ff6300886d0 msgr2=0x7ff630081c40 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:48.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:48.666 7ff638001700 1 -- 172.21.2.112:0/4037616978 shutdown_connections 2021-02-10T11:32:48.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:48.666 7ff638001700 1 -- 172.21.2.112:0/4037616978 wait complete. 2021-02-10T11:32:48.662 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:32:48.663 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:32:49.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:49.085 7f7d22442700 1 Processor -- start 2021-02-10T11:32:49.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:49.085 7f7d22442700 1 -- start start 2021-02-10T11:32:49.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:49.085 7f7d22442700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7d1c0e5470 0x7f7d1c07e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:49.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:49.085 7f7d22442700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d1c07ed80 con 0x7f7d1c0e5070 2021-02-10T11:32:49.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:49.085 7f7d22442700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d1c1b4010 con 0x7f7d1c07f300 2021-02-10T11:32:49.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:49.085 7f7d22442700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d1c0d1790 con 0x7f7d1c0e5470 2021-02-10T11:32:49.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:49.085 7f7d1bfff700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f7d1c0e5070 0x7f7d1c1b3820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:48704/0 (socket says 172.21.2.112:48704) 2021-02-10T11:32:49.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:49.085 7f7d1bfff700 1 -- 172.21.2.112:0/2409890963 learned_addr learned my addr 172.21.2.112:0/2409890963 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:32:49.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:49.086 7f7d1affd700 1 -- 172.21.2.112:0/2409890963 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 961070155 0 0) 0x7f7d0c001180 con 0x7f7d1c0e5070 2021-02-10T11:32:49.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:49.086 7f7d1affd700 1 -- 172.21.2.112:0/2409890963 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d04002fe0 con 0x7f7d1c0e5070 2021-02-10T11:32:49.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:49.086 7f7d1affd700 1 -- 172.21.2.112:0/2409890963 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2994125198 0 0) 0x7f7d0c0010e0 con 0x7f7d1c0e5070 2021-02-10T11:32:49.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:49.086 7f7d1affd700 1 -- 172.21.2.112:0/2409890963 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7d1c0e5470 msgr2=0x7f7d1c07e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:49.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:49.086 7f7d1affd700 1 --2- 172.21.2.112:0/2409890963 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7d1c0e5470 0x7f7d1c07e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:49.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:49.086 7f7d1affd700 1 -- 172.21.2.112:0/2409890963 >> v1:172.21.2.120:6789/0 conn(0x7f7d1c07f300 legacy=0x7f7d1c07f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:49.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:49.086 7f7d1affd700 1 -- 172.21.2.112:0/2409890963 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d1c0d1980 con 0x7f7d1c0e5070 2021-02-10T11:32:49.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:49.086 7f7d1affd700 1 -- 172.21.2.112:0/2409890963 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f7d0c001600 con 0x7f7d1c0e5070 2021-02-10T11:32:49.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:49.086 7f7d1affd700 1 -- 172.21.2.112:0/2409890963 >> v1:172.21.2.112:6789/0 conn(0x7f7d1c0e5070 legacy=0x7f7d1c1b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:49.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:49.086 7f7d1affd700 1 --2- 172.21.2.112:0/2409890963 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7d04003aa0 0x7f7d04005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:49.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:49.086 7f7d1affd700 1 --2- 172.21.2.112:0/2409890963 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7d04006790 0x7f7d04008c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:49.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:49.086 7f7d1affd700 1 --2- 172.21.2.112:0/2409890963 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7d04009500 0x7f7d0400ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:49.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:49.086 7f7d1affd700 1 -- 172.21.2.112:0/2409890963 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d0400c2d0 con 0x7f7d04003aa0 2021-02-10T11:32:49.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:49.086 7f7d1affd700 1 -- 172.21.2.112:0/2409890963 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d0400c4f0 con 0x7f7d04009500 2021-02-10T11:32:49.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:49.086 7f7d1affd700 1 -- 172.21.2.112:0/2409890963 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d0400c710 con 0x7f7d04006790 2021-02-10T11:32:49.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:49.086 7f7d1bfff700 1 --2- 172.21.2.112:0/2409890963 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7d04003aa0 0x7f7d04005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:49.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:49.086 7f7d20a5f700 1 --2- 172.21.2.112:0/2409890963 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7d04009500 0x7f7d0400ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:49.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:49.086 7f7d1bfff700 1 -- 172.21.2.112:0/2409890963 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7d04006790 msgr2=0x7f7d04008c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:49.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:49.086 7f7d1bfff700 1 --2- 172.21.2.112:0/2409890963 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7d04006790 0x7f7d04008c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:49.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:49.086 7f7d1bfff700 1 -- 172.21.2.112:0/2409890963 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7d04009500 msgr2=0x7f7d0400ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:49.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:49.086 7f7d1bfff700 1 --2- 172.21.2.112:0/2409890963 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7d04009500 0x7f7d0400ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:49.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:49.086 7f7d1bfff700 1 -- 172.21.2.112:0/2409890963 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d0400c9b0 con 0x7f7d04003aa0 2021-02-10T11:32:49.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:49.086 7f7d20a5f700 1 --2- 172.21.2.112:0/2409890963 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7d04009500 0x7f7d0400ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:32:49.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:49.086 7f7d1bfff700 1 --2- 172.21.2.112:0/2409890963 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7d04003aa0 0x7f7d04005ef0 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 rx=0x7f7d0c0017c0 tx=0x7f7d0c007d30).ready entity=mon.0 client_cookie=dd07c9c9e086259a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:49.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:49.087 7f7d1affd700 1 -- 172.21.2.112:0/2409890963 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7d0c007030 con 0x7f7d04003aa0 2021-02-10T11:32:49.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:49.087 7f7d1affd700 1 -- 172.21.2.112:0/2409890963 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d0c00a0c0 con 0x7f7d04003aa0 2021-02-10T11:32:49.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:49.087 7f7d1affd700 1 -- 172.21.2.112:0/2409890963 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7d0c00a550 con 0x7f7d04003aa0 2021-02-10T11:32:49.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:49.087 7f7d22442700 1 -- 172.21.2.112:0/2409890963 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7d04003aa0 msgr2=0x7f7d04005ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:49.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:49.087 7f7d22442700 1 --2- 172.21.2.112:0/2409890963 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7d04003aa0 0x7f7d04005ef0 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 rx=0x7f7d0c0017c0 tx=0x7f7d0c007d30).stop 2021-02-10T11:32:49.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:49.087 7f7d20a5f700 1 -- 172.21.2.112:0/2409890963 reap_dead start 2021-02-10T11:32:49.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:49.087 7f7d22442700 1 -- 172.21.2.112:0/2409890963 shutdown_connections 2021-02-10T11:32:49.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:49.087 7f7d22442700 1 -- 172.21.2.112:0/2409890963 >> 172.21.2.112:0/2409890963 conn(0x7f7d1c0ce690 msgr2=0x7f7d1c0e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:49.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:49.087 7f7d22442700 1 -- 172.21.2.112:0/2409890963 shutdown_connections 2021-02-10T11:32:49.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:49.087 7f7d22442700 1 -- 172.21.2.112:0/2409890963 wait complete. 2021-02-10T11:32:49.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:49.102 7f7d22442700 1 Processor -- start 2021-02-10T11:32:49.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:49.102 7f7d22442700 1 -- start start 2021-02-10T11:32:49.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:49.103 7f7d22442700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7d1c1ad6e0 0x7f7d1c0a3210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:49.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:49.103 7f7d22442700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7d1c08da10 0x7f7d1c1ae300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:49.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:49.103 7f7d22442700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7d1c08de10 0x7f7d1c1ae920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:49.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:49.103 7f7d22442700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d1c1add80 con 0x7f7d1c08de10 2021-02-10T11:32:49.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:49.103 7f7d22442700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d1c1adae0 con 0x7f7d1c1ad6e0 2021-02-10T11:32:49.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:49.103 7f7d22442700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d1c0a3830 con 0x7f7d1c08da10 2021-02-10T11:32:49.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:49.103 7f7d20a5f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7d1c08de10 0x7f7d1c1ae920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:49.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:49.103 7f7d20a5f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7d1c08de10 0x7f7d1c1ae920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:49254/0 (socket says 172.21.2.112:49254) 2021-02-10T11:32:49.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:49.103 7f7d20a5f700 1 -- 172.21.2.112:0/2236181724 learned_addr learned my addr 172.21.2.112:0/2236181724 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:32:49.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:49.103 7f7d20a5f700 1 -- 172.21.2.112:0/2236181724 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7d1c08da10 msgr2=0x7f7d1c1ae300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:49.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:49.103 7f7d20a5f700 1 --2- 172.21.2.112:0/2236181724 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7d1c08da10 0x7f7d1c1ae300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:49.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:49.103 7f7d20a5f700 1 -- 172.21.2.112:0/2236181724 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7d1c1ad6e0 msgr2=0x7f7d1c0a3210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:49.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:49.103 7f7d20a5f700 1 --2- 172.21.2.112:0/2236181724 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7d1c1ad6e0 0x7f7d1c0a3210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:49.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:49.103 7f7d20a5f700 1 -- 172.21.2.112:0/2236181724 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d1c1aef40 con 0x7f7d1c08de10 2021-02-10T11:32:49.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:49.104 7f7d20a5f700 1 --2- 172.21.2.112:0/2236181724 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7d1c08de10 0x7f7d1c1ae920 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 rx=0x7f7d140099e0 tx=0x7f7d1400bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:49.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:49.104 7f7d18ff9700 1 -- 172.21.2.112:0/2236181724 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7d14006950 con 0x7f7d1c08de10 2021-02-10T11:32:49.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:49.104 7f7d18ff9700 1 -- 172.21.2.112:0/2236181724 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d14006030 con 0x7f7d1c08de10 2021-02-10T11:32:49.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:49.104 7f7d18ff9700 1 -- 172.21.2.112:0/2236181724 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7d14006550 con 0x7f7d1c08de10 2021-02-10T11:32:49.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:49.104 7f7d22442700 1 -- 172.21.2.112:0/2236181724 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d1c1aef40 con 0x7f7d1c08de10 2021-02-10T11:32:49.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:49.104 7f7d22442700 1 -- 172.21.2.112:0/2236181724 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d1c1af4d0 con 0x7f7d1c08de10 2021-02-10T11:32:49.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:49.104 7f7d18ff9700 1 -- 172.21.2.112:0/2236181724 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7d14006210 con 0x7f7d1c08de10 2021-02-10T11:32:49.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:49.104 7f7d18ff9700 1 --2- 172.21.2.112:0/2236181724 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7cfc0186c0 0x7f7cfc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:49.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:49.105 7f7d18ff9700 1 -- 172.21.2.112:0/2236181724 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f7d140329e0 con 0x7f7d1c08de10 2021-02-10T11:32:49.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:49.105 7f7d1bfff700 1 --2- 172.21.2.112:0/2236181724 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7cfc0186c0 0x7f7cfc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:49.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:49.105 7f7d1bfff700 1 --2- 172.21.2.112:0/2236181724 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7cfc0186c0 0x7f7cfc01abc0 crc :-1 s=READY pgs=586 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:49.122 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:49.135 7f7d22442700 1 -- 172.21.2.112:0/2236181724 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d1c07ed80 con 0x7f7d1c08de10 2021-02-10T11:32:49.124 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:49.138 7f7d18ff9700 1 -- 172.21.2.112:0/2236181724 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7d1402def0 con 0x7f7d1c08de10 2021-02-10T11:32:49.254 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:49.267 7f7d22442700 1 -- 172.21.2.112:0/2236181724 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7d1c07ed80 con 0x7f7d1c08de10 2021-02-10T11:32:49.254 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:49.267 7f7d18ff9700 1 -- 172.21.2.112:0/2236181724 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+2093 (secure 0 0 0) 0x7f7d1402def0 con 0x7f7d1c08de10 2021-02-10T11:32:49.255 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:32:49.255 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":30,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7536,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/340651080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":340651080},{"type":"v1","addr":"172.21.2.112:6833","nonce":340651080}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":26}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:32:31.869812","modified":"2021-02-10 11:32:33.879671","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6409},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6409":{"gid":6409,"name":"b","rank":0,"incarnation":27,"state":"up:active","state_seq":3,"addr":"172.21.2.120:6835/210055046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":210055046},{"type":"v1","addr":"172.21.2.120:6835","nonce":210055046}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":4}]} 2021-02-10T11:32:49.255 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 30 2021-02-10T11:32:49.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:49.269 7f7d22442700 1 -- 172.21.2.112:0/2236181724 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7cfc0186c0 msgr2=0x7f7cfc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:49.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:49.269 7f7d22442700 1 --2- 172.21.2.112:0/2236181724 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7cfc0186c0 0x7f7cfc01abc0 crc :-1 s=READY pgs=586 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:49.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:49.269 7f7d22442700 1 -- 172.21.2.112:0/2236181724 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7d1c08de10 msgr2=0x7f7d1c1ae920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:49.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:49.269 7f7d22442700 1 --2- 172.21.2.112:0/2236181724 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7d1c08de10 0x7f7d1c1ae920 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 rx=0x7f7d140099e0 tx=0x7f7d1400bcf0).stop 2021-02-10T11:32:49.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:49.269 7f7d22442700 1 -- 172.21.2.112:0/2236181724 shutdown_connections 2021-02-10T11:32:49.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:49.269 7f7d22442700 1 --2- 172.21.2.112:0/2236181724 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7cfc0186c0 0x7f7cfc01abc0 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:49.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:49.269 7f7d22442700 1 --2- 172.21.2.112:0/2236181724 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7d1c08de10 0x7f7d1c1ae920 unknown :-1 s=CLOSED pgs=1164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:49.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:49.269 7f7d22442700 1 --2- 172.21.2.112:0/2236181724 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7d1c08da10 0x7f7d1c1ae300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:49.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:49.269 7f7d22442700 1 --2- 172.21.2.112:0/2236181724 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7d1c1ad6e0 0x7f7d1c0a3210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:49.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:49.269 7f7d22442700 1 -- 172.21.2.112:0/2236181724 >> 172.21.2.112:0/2236181724 conn(0x7f7d1c1a31b0 msgr2=0x7f7d1c18c8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:49.257 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:49.269 7f7d22442700 1 -- 172.21.2.112:0/2236181724 shutdown_connections 2021-02-10T11:32:49.257 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:49.269 7f7d22442700 1 -- 172.21.2.112:0/2236181724 wait complete. 2021-02-10T11:32:49.466 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:32:31.869812', 'modified': '2021-02-10 11:32:33.879671', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6409}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6409': {'gid': 6409, 'name': 'b', 'rank': 0, 'incarnation': 27, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.120:6835/210055046', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.120:6834', 'nonce': 210055046}, {'type': 'v1', 'addr': '172.21.2.120:6835', 'nonce': 210055046}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [9], 'metadata_pool': 8, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:32:49.466 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:32:49.466 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:32:49.863 INFO:teuthology.orchestra.run.gibba020.stdout:{ 2021-02-10T11:32:49.864 INFO:teuthology.orchestra.run.gibba020.stdout: "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", 2021-02-10T11:32:49.864 INFO:teuthology.orchestra.run.gibba020.stdout: "whoami": 0, 2021-02-10T11:32:49.864 INFO:teuthology.orchestra.run.gibba020.stdout: "id": 6409, 2021-02-10T11:32:49.864 INFO:teuthology.orchestra.run.gibba020.stdout: "want_state": "up:active", 2021-02-10T11:32:49.864 INFO:teuthology.orchestra.run.gibba020.stdout: "state": "up:active", 2021-02-10T11:32:49.864 INFO:teuthology.orchestra.run.gibba020.stdout: "rank_uptime": 18.005643804000002, 2021-02-10T11:32:49.864 INFO:teuthology.orchestra.run.gibba020.stdout: "mdsmap_epoch": 30, 2021-02-10T11:32:49.864 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch": 46, 2021-02-10T11:32:49.864 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch_barrier": 46, 2021-02-10T11:32:49.864 INFO:teuthology.orchestra.run.gibba020.stdout: "uptime": 23.635602383999998 2021-02-10T11:32:49.864 INFO:teuthology.orchestra.run.gibba020.stdout:} 2021-02-10T11:32:49.868 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", "whoami": 0, "id": 6409, "want_state": "up:active", "state": "up:active", "rank_uptime": 18.005643804000002, "mdsmap_epoch": 30, "osdmap_epoch": 46, "osdmap_epoch_barrier": 46, "uptime": 23.635602383999998 } 2021-02-10T11:32:49.868 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:32:49.868 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2021-02-10T11:32:49.868 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.1 at ubuntu@gibba012.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2021-02-10T11:32:49.868 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:49.868 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:32:50.204 DEBUG:teuthology.orchestra.run.gibba012:> sudo modprobe fuse 2021-02-10T11:32:50.548 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:32:50.648 DEBUG:teuthology.orchestra.run.gibba012:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:32:50.994 INFO:teuthology.orchestra.run.gibba012.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:32:50.994 INFO:teuthology.orchestra.run.gibba012.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:32:50.994 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:32:50.995 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:50.995 DEBUG:teuthology.orchestra.run.gibba012:> ls /sys/fs/fuse/connections 2021-02-10T11:32:51.330 INFO:teuthology.orchestra.run.gibba012.stdout:42 2021-02-10T11:32:51.330 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [42] 2021-02-10T11:32:51.330 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.1 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:32:51.334 DEBUG:teuthology.orchestra.run.gibba012:> sudo modprobe fuse 2021-02-10T11:32:51.677 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:32:51.677 DEBUG:teuthology.orchestra.run.gibba012:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:32:51.744 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba012.stderr:2021-02-10 11:32:51.756 7f7365147f80 -1 init, newargv = 0x557a4e7650f0 newargc=9 2021-02-10T11:32:51.744 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba012.stderr:ceph-fuse[125909]: starting ceph client 2021-02-10T11:32:51.749 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba012.stderr:ceph-fuse[125909]: starting fuse 2021-02-10T11:32:52.019 INFO:teuthology.orchestra.run.gibba012.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:32:52.019 INFO:teuthology.orchestra.run.gibba012.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:32:52.020 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:32:52.020 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:52.020 DEBUG:teuthology.orchestra.run.gibba012:> ls /sys/fs/fuse/connections 2021-02-10T11:32:52.360 INFO:teuthology.orchestra.run.gibba012.stdout:42 2021-02-10T11:32:52.360 INFO:teuthology.orchestra.run.gibba012.stdout:43 2021-02-10T11:32:52.360 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42, 43] 2021-02-10T11:32:52.360 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:52.361 DEBUG:teuthology.orchestra.run.gibba012:> sudo python3 -c ' 2021-02-10T11:32:52.361 DEBUG:teuthology.orchestra.run.gibba012:> import glob 2021-02-10T11:32:52.361 DEBUG:teuthology.orchestra.run.gibba012:> import re 2021-02-10T11:32:52.361 DEBUG:teuthology.orchestra.run.gibba012:> import os 2021-02-10T11:32:52.361 DEBUG:teuthology.orchestra.run.gibba012:> import subprocess 2021-02-10T11:32:52.361 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:32:52.361 DEBUG:teuthology.orchestra.run.gibba012:> def find_socket(client_name): 2021-02-10T11:32:52.361 DEBUG:teuthology.orchestra.run.gibba012:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2021-02-10T11:32:52.361 DEBUG:teuthology.orchestra.run.gibba012:> files = glob.glob(asok_path) 2021-02-10T11:32:52.361 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:32:52.361 DEBUG:teuthology.orchestra.run.gibba012:> # Given a non-glob path, it better be there 2021-02-10T11:32:52.361 DEBUG:teuthology.orchestra.run.gibba012:> if "*" not in asok_path: 2021-02-10T11:32:52.361 DEBUG:teuthology.orchestra.run.gibba012:> assert(len(files) == 1) 2021-02-10T11:32:52.361 DEBUG:teuthology.orchestra.run.gibba012:> return files[0] 2021-02-10T11:32:52.362 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:32:52.362 DEBUG:teuthology.orchestra.run.gibba012:> for f in files: 2021-02-10T11:32:52.362 DEBUG:teuthology.orchestra.run.gibba012:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:32:52.362 DEBUG:teuthology.orchestra.run.gibba012:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:32:52.362 DEBUG:teuthology.orchestra.run.gibba012:> return f 2021-02-10T11:32:52.362 DEBUG:teuthology.orchestra.run.gibba012:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:32:52.362 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:32:52.362 DEBUG:teuthology.orchestra.run.gibba012:> print(find_socket("client.1")) 2021-02-10T11:32:52.362 DEBUG:teuthology.orchestra.run.gibba012:> ' 2021-02-10T11:32:52.721 INFO:teuthology.orchestra.run.gibba012.stdout:/var/run/ceph/ceph-client.1.125909.asok 2021-02-10T11:32:52.724 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.125909.asok 2021-02-10T11:32:52.724 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:52.724 DEBUG:teuthology.orchestra.run.gibba012:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.125909.asok status 2021-02-10T11:32:53.113 INFO:teuthology.orchestra.run.gibba012.stdout:{ 2021-02-10T11:32:53.113 INFO:teuthology.orchestra.run.gibba012.stdout: "metadata": { 2021-02-10T11:32:53.113 INFO:teuthology.orchestra.run.gibba012.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:32:53.113 INFO:teuthology.orchestra.run.gibba012.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:32:53.113 INFO:teuthology.orchestra.run.gibba012.stdout: "entity_id": "1", 2021-02-10T11:32:53.114 INFO:teuthology.orchestra.run.gibba012.stdout: "hostname": "gibba012", 2021-02-10T11:32:53.114 INFO:teuthology.orchestra.run.gibba012.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2021-02-10T11:32:53.114 INFO:teuthology.orchestra.run.gibba012.stdout: "pid": "125909", 2021-02-10T11:32:53.114 INFO:teuthology.orchestra.run.gibba012.stdout: "root": "/" 2021-02-10T11:32:53.114 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:32:53.114 INFO:teuthology.orchestra.run.gibba012.stdout: "dentry_count": 0, 2021-02-10T11:32:53.114 INFO:teuthology.orchestra.run.gibba012.stdout: "dentry_pinned_count": 0, 2021-02-10T11:32:53.114 INFO:teuthology.orchestra.run.gibba012.stdout: "id": 7734, 2021-02-10T11:32:53.114 INFO:teuthology.orchestra.run.gibba012.stdout: "inst": { 2021-02-10T11:32:53.114 INFO:teuthology.orchestra.run.gibba012.stdout: "name": { 2021-02-10T11:32:53.114 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "client", 2021-02-10T11:32:53.114 INFO:teuthology.orchestra.run.gibba012.stdout: "num": 7734 2021-02-10T11:32:53.115 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:32:53.115 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": { 2021-02-10T11:32:53.115 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:32:53.115 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:0", 2021-02-10T11:32:53.115 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 921407212 2021-02-10T11:32:53.115 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:32:53.115 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:32:53.115 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": { 2021-02-10T11:32:53.115 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:32:53.115 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:0", 2021-02-10T11:32:53.115 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 921407212 2021-02-10T11:32:53.115 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:32:53.115 INFO:teuthology.orchestra.run.gibba012.stdout: "inst_str": "client.7734 172.21.2.112:0/921407212", 2021-02-10T11:32:53.116 INFO:teuthology.orchestra.run.gibba012.stdout: "addr_str": "172.21.2.112:0/921407212", 2021-02-10T11:32:53.116 INFO:teuthology.orchestra.run.gibba012.stdout: "inode_count": 1, 2021-02-10T11:32:53.116 INFO:teuthology.orchestra.run.gibba012.stdout: "mds_epoch": 30, 2021-02-10T11:32:53.116 INFO:teuthology.orchestra.run.gibba012.stdout: "osd_epoch": 46, 2021-02-10T11:32:53.116 INFO:teuthology.orchestra.run.gibba012.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:32:53.116 INFO:teuthology.orchestra.run.gibba012.stdout: "blacklisted": false 2021-02-10T11:32:53.116 INFO:teuthology.orchestra.run.gibba012.stdout:} 2021-02-10T11:32:53.117 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:53.117 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:32:53.117 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:32:53.453 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:32:53.453 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:32:53.453 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:53.454 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:32:53.800 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2021-02-10T11:32:54.228 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:54.225 7f409d8bd700 1 Processor -- start 2021-02-10T11:32:54.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:54.225 7f409d8bd700 1 -- start start 2021-02-10T11:32:54.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:54.226 7f409d8bd700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f40980d1c70 0x7f40980cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:54.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:54.226 7f409d8bd700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f40980cf580 0x7f40980cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:54.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:54.226 7f409d8bd700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40980fe1c0 con 0x7f40980d1870 2021-02-10T11:32:54.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:54.226 7f409d8bd700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40981b0cb0 con 0x7f40980d1c70 2021-02-10T11:32:54.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:54.226 7f409d8bd700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4098109a60 con 0x7f40980cf580 2021-02-10T11:32:54.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:54.226 7f4096d73700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f40980d1c70 0x7f40980cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:54.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:54.226 7f4097574700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f40980d1870 0x7f40981b04c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:48744/0 (socket says 172.21.2.112:48744) 2021-02-10T11:32:54.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:54.226 7f4097574700 1 -- 172.21.2.112:0/3893441266 learned_addr learned my addr 172.21.2.112:0/3893441266 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:32:54.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:54.226 7f4096d73700 1 -- 172.21.2.112:0/3893441266 >> v2:172.21.2.120:3300/0 conn(0x7f40980cf580 msgr2=0x7f40980cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:54.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:54.226 7f4096d73700 1 --2- 172.21.2.112:0/3893441266 >> v2:172.21.2.120:3300/0 conn(0x7f40980cf580 0x7f40980cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:54.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:54.226 7f4096d73700 1 -- 172.21.2.112:0/3893441266 >> v1:172.21.2.112:6789/0 conn(0x7f40980d1870 legacy=0x7f40981b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:54.230 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:54.226 7f4096d73700 1 -- 172.21.2.112:0/3893441266 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4098088020 con 0x7f40980d1c70 2021-02-10T11:32:54.230 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:54.226 7f4096d73700 1 --2- 172.21.2.112:0/3893441266 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f40980d1c70 0x7f40980cef60 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 rx=0x7f4080006650 tx=0x7f4080009a80).ready entity=mon.0 client_cookie=e5dd3af2a720df9 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:54.230 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:54.227 7f4096572700 1 -- 172.21.2.112:0/3893441266 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f40800009b0 con 0x7f40980d1c70 2021-02-10T11:32:54.230 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:54.227 7f4096572700 1 -- 172.21.2.112:0/3893441266 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f408000c0c0 con 0x7f40980d1c70 2021-02-10T11:32:54.230 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:54.227 7f4096572700 1 -- 172.21.2.112:0/3893441266 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f408000c5e0 con 0x7f40980d1c70 2021-02-10T11:32:54.230 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:54.227 7f409d8bd700 1 -- 172.21.2.112:0/3893441266 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f40980d1c70 msgr2=0x7f40980cef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:54.230 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:54.227 7f409d8bd700 1 --2- 172.21.2.112:0/3893441266 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f40980d1c70 0x7f40980cef60 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 rx=0x7f4080006650 tx=0x7f4080009a80).stop 2021-02-10T11:32:54.230 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:54.227 7f409d8bd700 1 -- 172.21.2.112:0/3893441266 shutdown_connections 2021-02-10T11:32:54.230 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:54.227 7f409d8bd700 1 --2- 172.21.2.112:0/3893441266 >> v2:172.21.2.120:3300/0 conn(0x7f40980cf580 0x7f40980cf980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:54.230 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:54.227 7f409d8bd700 1 --2- 172.21.2.112:0/3893441266 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f40980d1c70 0x7f40980cef60 unknown :-1 s=CLOSED pgs=1169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:54.230 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:54.227 7f409d8bd700 1 -- 172.21.2.112:0/3893441266 >> 172.21.2.112:0/3893441266 conn(0x7f40980ce420 msgr2=0x7f40980d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:54.230 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:54.227 7f409d8bd700 1 -- 172.21.2.112:0/3893441266 shutdown_connections 2021-02-10T11:32:54.230 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:54.227 7f409d8bd700 1 -- 172.21.2.112:0/3893441266 wait complete. 2021-02-10T11:32:54.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:54.242 7f409d8bd700 1 Processor -- start 2021-02-10T11:32:54.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:54.242 7f409d8bd700 1 -- start start 2021-02-10T11:32:54.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:54.242 7f409d8bd700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f40981a9930 0x7f40981a9080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:54.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:54.242 7f409d8bd700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f40981abd60 0x7f4098192a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:54.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:54.242 7f409d8bd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f40980b03e0 0x7f40980b07e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:54.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:54.242 7f409d8bd700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40981932c0 con 0x7f40980b03e0 2021-02-10T11:32:54.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:54.242 7f409d8bd700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4098193840 con 0x7f40981abd60 2021-02-10T11:32:54.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:54.242 7f409d8bd700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4098193a20 con 0x7f40981a9930 2021-02-10T11:32:54.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:54.242 7f4097d75700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f40980b03e0 0x7f40980b07e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:54.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:54.243 7f4097d75700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f40980b03e0 0x7f40980b07e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:49290/0 (socket says 172.21.2.112:49290) 2021-02-10T11:32:54.233 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:54.243 7f4097d75700 1 -- 172.21.2.112:0/2000830970 learned_addr learned my addr 172.21.2.112:0/2000830970 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:32:54.233 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:54.243 7f4097d75700 1 -- 172.21.2.112:0/2000830970 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f40981a9930 msgr2=0x7f40981a9080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:54.233 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:54.243 7f4097d75700 1 --2- 172.21.2.112:0/2000830970 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f40981a9930 0x7f40981a9080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:54.233 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:54.243 7f4097d75700 1 -- 172.21.2.112:0/2000830970 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f40981abd60 msgr2=0x7f4098192a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:54.233 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:54.243 7f4097d75700 1 --2- 172.21.2.112:0/2000830970 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f40981abd60 0x7f4098192a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:54.233 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:54.243 7f4097d75700 1 -- 172.21.2.112:0/2000830970 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4098193c40 con 0x7f40980b03e0 2021-02-10T11:32:54.233 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:54.243 7f4097d75700 1 --2- 172.21.2.112:0/2000830970 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f40980b03e0 0x7f40980b07e0 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 rx=0x7f409000b8c0 tx=0x7f409000bc60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:54.234 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:54.243 7f408ffff700 1 -- 172.21.2.112:0/2000830970 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4090005870 con 0x7f40980b03e0 2021-02-10T11:32:54.234 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:54.243 7f408ffff700 1 -- 172.21.2.112:0/2000830970 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4090005b30 con 0x7f40980b03e0 2021-02-10T11:32:54.234 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:54.243 7f408ffff700 1 -- 172.21.2.112:0/2000830970 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4090006030 con 0x7f40980b03e0 2021-02-10T11:32:54.234 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:54.243 7f409d8bd700 1 -- 172.21.2.112:0/2000830970 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4098193c40 con 0x7f40980b03e0 2021-02-10T11:32:54.234 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:54.243 7f409d8bd700 1 -- 172.21.2.112:0/2000830970 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40981941d0 con 0x7f40980b03e0 2021-02-10T11:32:54.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:54.244 7f408ffff700 1 -- 172.21.2.112:0/2000830970 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4090006030 con 0x7f40980b03e0 2021-02-10T11:32:54.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:54.244 7f408ffff700 1 --2- 172.21.2.112:0/2000830970 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f407c0186c0 0x7f407c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:54.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:54.244 7f408ffff700 1 -- 172.21.2.112:0/2000830970 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f4090032950 con 0x7f40980b03e0 2021-02-10T11:32:54.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:54.244 7f4097574700 1 --2- 172.21.2.112:0/2000830970 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f407c0186c0 0x7f407c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:54.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:54.244 7f4097574700 1 --2- 172.21.2.112:0/2000830970 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f407c0186c0 0x7f407c01abc0 crc :-1 s=READY pgs=587 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:54.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:54.276 7f409d8bd700 1 -- 172.21.2.112:0/2000830970 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40981937e0 con 0x7f40980b03e0 2021-02-10T11:32:54.265 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:54.278 7f408ffff700 1 -- 172.21.2.112:0/2000830970 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4090005d10 con 0x7f40980b03e0 2021-02-10T11:32:54.402 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:54.415 7f409d8bd700 1 -- 172.21.2.112:0/2000830970 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f40981937e0 con 0x7f40980b03e0 2021-02-10T11:32:55.048 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:55.060 7f408ffff700 1 -- 172.21.2.112:0/2000830970 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v31) v1 ==== 136+0+0 (secure 0 0 0) 0x7f4090006f20 con 0x7f40980b03e0 2021-02-10T11:32:55.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:55.084 7f409d8bd700 1 -- 172.21.2.112:0/2000830970 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f407c0186c0 msgr2=0x7f407c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:55.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:55.084 7f409d8bd700 1 --2- 172.21.2.112:0/2000830970 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f407c0186c0 0x7f407c01abc0 crc :-1 s=READY pgs=587 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:55.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:55.084 7f409d8bd700 1 -- 172.21.2.112:0/2000830970 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f40980b03e0 msgr2=0x7f40980b07e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:55.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:55.084 7f409d8bd700 1 --2- 172.21.2.112:0/2000830970 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f40980b03e0 0x7f40980b07e0 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 rx=0x7f409000b8c0 tx=0x7f409000bc60).stop 2021-02-10T11:32:55.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:55.085 7f409d8bd700 1 -- 172.21.2.112:0/2000830970 shutdown_connections 2021-02-10T11:32:55.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:55.085 7f409d8bd700 1 --2- 172.21.2.112:0/2000830970 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f407c0186c0 0x7f407c01abc0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:55.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:55.085 7f409d8bd700 1 --2- 172.21.2.112:0/2000830970 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f40981abd60 0x7f4098192a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:55.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:55.085 7f409d8bd700 1 --2- 172.21.2.112:0/2000830970 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f40980b03e0 0x7f40980b07e0 unknown :-1 s=CLOSED pgs=1170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:55.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:55.085 7f409d8bd700 1 --2- 172.21.2.112:0/2000830970 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f40981a9930 0x7f40981a9080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:55.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:55.085 7f409d8bd700 1 -- 172.21.2.112:0/2000830970 >> 172.21.2.112:0/2000830970 conn(0x7f409819f820 msgr2=0x7f4098082b10 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:55.073 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:55.085 7f409d8bd700 1 -- 172.21.2.112:0/2000830970 shutdown_connections 2021-02-10T11:32:55.073 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:55.085 7f409d8bd700 1 -- 172.21.2.112:0/2000830970 wait complete. 2021-02-10T11:32:55.080 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs volume ls 2021-02-10T11:32:55.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:55.501 7f2c749c5700 1 Processor -- start 2021-02-10T11:32:55.505 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:55.501 7f2c749c5700 1 -- start start 2021-02-10T11:32:55.505 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:55.501 7f2c749c5700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f2c700e5070 0x7f2c701b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:55.505 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:55.501 7f2c749c5700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f2c700e5470 0x7f2c70080310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:55.505 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:55.501 7f2c749c5700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c7007ed80 con 0x7f2c701b3e40 2021-02-10T11:32:55.505 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:55.501 7f2c749c5700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c700d1790 con 0x7f2c700e5070 2021-02-10T11:32:55.505 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:55.501 7f2c749c5700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c700d19b0 con 0x7f2c700e5470 2021-02-10T11:32:55.505 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:55.502 7f2c6e61c700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f2c700e5070 0x7f2c701b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:55.505 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:55.502 7f2c6e61c700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f2c700e5070 0x7f2c701b3820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:49294/0 (socket says 172.21.2.112:49294) 2021-02-10T11:32:55.505 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:55.502 7f2c6e61c700 1 -- 172.21.2.112:0/2104694013 learned_addr learned my addr 172.21.2.112:0/2104694013 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:32:55.505 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:55.502 7f2c6ee1d700 1 --2- 172.21.2.112:0/2104694013 >> v2:172.21.2.120:3300/0 conn(0x7f2c700e5470 0x7f2c70080310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:55.505 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:55.502 7f2c6e61c700 1 -- 172.21.2.112:0/2104694013 >> v2:172.21.2.120:3300/0 conn(0x7f2c700e5470 msgr2=0x7f2c70080310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:55.505 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:55.502 7f2c6e61c700 1 --2- 172.21.2.112:0/2104694013 >> v2:172.21.2.120:3300/0 conn(0x7f2c700e5470 0x7f2c70080310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:55.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:55.502 7f2c6e61c700 1 -- 172.21.2.112:0/2104694013 >> v1:172.21.2.120:6789/0 conn(0x7f2c701b3e40 legacy=0x7f2c7007d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:55.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:55.502 7f2c6e61c700 1 -- 172.21.2.112:0/2104694013 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c700d1bd0 con 0x7f2c700e5070 2021-02-10T11:32:55.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:55.502 7f2c6e61c700 1 --2- 172.21.2.112:0/2104694013 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2c700e5070 0x7f2c701b3820 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 rx=0x7f2c60006700 tx=0x7f2c60009b00).ready entity=mon.0 client_cookie=60c411e4cdd6c50d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:55.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:55.502 7f2c6d61a700 1 -- 172.21.2.112:0/2104694013 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2c6000c0c0 con 0x7f2c700e5070 2021-02-10T11:32:55.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:55.502 7f2c6d61a700 1 -- 172.21.2.112:0/2104694013 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c6000c400 con 0x7f2c700e5070 2021-02-10T11:32:55.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:55.502 7f2c6d61a700 1 -- 172.21.2.112:0/2104694013 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2c6000c920 con 0x7f2c700e5070 2021-02-10T11:32:55.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:55.502 7f2c749c5700 1 -- 172.21.2.112:0/2104694013 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2c700e5070 msgr2=0x7f2c701b3820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:55.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:55.502 7f2c749c5700 1 --2- 172.21.2.112:0/2104694013 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2c700e5070 0x7f2c701b3820 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 rx=0x7f2c60006700 tx=0x7f2c60009b00).stop 2021-02-10T11:32:55.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:55.503 7f2c749c5700 1 -- 172.21.2.112:0/2104694013 shutdown_connections 2021-02-10T11:32:55.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:55.503 7f2c749c5700 1 --2- 172.21.2.112:0/2104694013 >> v2:172.21.2.120:3300/0 conn(0x7f2c700e5470 0x7f2c70080310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:55.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:55.503 7f2c749c5700 1 --2- 172.21.2.112:0/2104694013 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2c700e5070 0x7f2c701b3820 unknown :-1 s=CLOSED pgs=1171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:55.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:55.503 7f2c749c5700 1 -- 172.21.2.112:0/2104694013 >> 172.21.2.112:0/2104694013 conn(0x7f2c700ce690 msgr2=0x7f2c700e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:55.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:55.503 7f2c749c5700 1 -- 172.21.2.112:0/2104694013 shutdown_connections 2021-02-10T11:32:55.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:55.503 7f2c749c5700 1 -- 172.21.2.112:0/2104694013 wait complete. 2021-02-10T11:32:55.508 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:55.518 7f2c749c5700 1 Processor -- start 2021-02-10T11:32:55.508 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:55.518 7f2c749c5700 1 -- start start 2021-02-10T11:32:55.508 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:55.518 7f2c749c5700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2c700b4ad0 0x7f2c700a78d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:55.508 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:55.518 7f2c749c5700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2c700a7ef0 0x7f2c701a6d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:55.508 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:55.518 7f2c749c5700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2c700a82f0 0x7f2c701a73b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:55.508 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:55.518 7f2c749c5700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c700aa3c0 con 0x7f2c700a82f0 2021-02-10T11:32:55.508 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:55.518 7f2c749c5700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c700a88c0 con 0x7f2c700b4ad0 2021-02-10T11:32:55.508 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:55.518 7f2c749c5700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c700b4ed0 con 0x7f2c700a7ef0 2021-02-10T11:32:55.508 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:55.518 7f2c6ee1d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2c700a82f0 0x7f2c701a73b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:55.509 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:55.518 7f2c6ee1d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2c700a82f0 0x7f2c701a73b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:49298/0 (socket says 172.21.2.112:49298) 2021-02-10T11:32:55.509 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:55.518 7f2c6ee1d700 1 -- 172.21.2.112:0/3385013626 learned_addr learned my addr 172.21.2.112:0/3385013626 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:32:55.509 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:55.519 7f2c6ee1d700 1 -- 172.21.2.112:0/3385013626 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2c700a7ef0 msgr2=0x7f2c701a6d90 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:32:55.509 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:55.519 7f2c6ee1d700 1 --2- 172.21.2.112:0/3385013626 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2c700a7ef0 0x7f2c701a6d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:55.509 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:55.519 7f2c6ee1d700 1 -- 172.21.2.112:0/3385013626 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2c700b4ad0 msgr2=0x7f2c700a78d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:32:55.509 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:55.519 7f2c6ee1d700 1 --2- 172.21.2.112:0/3385013626 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2c700b4ad0 0x7f2c700a78d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:55.509 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:55.519 7f2c6ee1d700 1 -- 172.21.2.112:0/3385013626 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c701a79d0 con 0x7f2c700a82f0 2021-02-10T11:32:55.509 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:55.519 7f2c6ee1d700 1 --2- 172.21.2.112:0/3385013626 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2c700a82f0 0x7f2c701a73b0 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 rx=0x7f2c68006f40 tx=0x7f2c6800bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:55.509 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:55.519 7f2c66ffd700 1 -- 172.21.2.112:0/3385013626 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2c680011f0 con 0x7f2c700a82f0 2021-02-10T11:32:55.509 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:55.519 7f2c66ffd700 1 -- 172.21.2.112:0/3385013626 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c68003030 con 0x7f2c700a82f0 2021-02-10T11:32:55.509 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:55.519 7f2c66ffd700 1 -- 172.21.2.112:0/3385013626 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2c68001770 con 0x7f2c700a82f0 2021-02-10T11:32:55.509 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:55.519 7f2c749c5700 1 -- 172.21.2.112:0/3385013626 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c701a79d0 con 0x7f2c700a82f0 2021-02-10T11:32:55.509 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:55.519 7f2c749c5700 1 -- 172.21.2.112:0/3385013626 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c700a9130 con 0x7f2c700a82f0 2021-02-10T11:32:55.509 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:55.520 7f2c66ffd700 1 -- 172.21.2.112:0/3385013626 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2c680011f0 con 0x7f2c700a82f0 2021-02-10T11:32:55.510 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:55.520 7f2c66ffd700 1 --2- 172.21.2.112:0/3385013626 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2c540186c0 0x7f2c5401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:55.510 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:55.520 7f2c66ffd700 1 -- 172.21.2.112:0/3385013626 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f2c68032890 con 0x7f2c700a82f0 2021-02-10T11:32:55.510 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:55.520 7f2c6e61c700 1 --2- 172.21.2.112:0/3385013626 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2c540186c0 0x7f2c5401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:55.510 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:55.521 7f2c6e61c700 1 --2- 172.21.2.112:0/3385013626 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2c540186c0 0x7f2c5401abc0 crc :-1 s=READY pgs=588 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:55.538 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:55.551 7f2c749c5700 1 -- 172.21.2.112:0/3385013626 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c70084a90 con 0x7f2c700a82f0 2021-02-10T11:32:55.541 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:55.554 7f2c66ffd700 1 -- 172.21.2.112:0/3385013626 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2c680090d0 con 0x7f2c700a82f0 2021-02-10T11:32:55.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:55.686 7f2c749c5700 1 -- 172.21.2.112:0/3385013626 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs volume ls", "target": ["mgr", ""]}) v1 -- 0x7f2c700b4ed0 con 0x7f2c540186c0 2021-02-10T11:32:55.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:55.688 7f2c66ffd700 1 -- 172.21.2.112:0/3385013626 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+41 (crc 0 0 0) 0x7f2c60009440 con 0x7f2c540186c0 2021-02-10T11:32:55.676 INFO:teuthology.orchestra.run.gibba012.stdout:[ 2021-02-10T11:32:55.676 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:32:55.677 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "cephfs" 2021-02-10T11:32:55.677 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:32:55.677 INFO:teuthology.orchestra.run.gibba012.stdout:] 2021-02-10T11:32:55.678 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:55.691 7f2c749c5700 1 -- 172.21.2.112:0/3385013626 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2c540186c0 msgr2=0x7f2c5401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:55.678 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:55.691 7f2c749c5700 1 --2- 172.21.2.112:0/3385013626 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2c540186c0 0x7f2c5401abc0 crc :-1 s=READY pgs=588 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:55.678 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:55.691 7f2c749c5700 1 -- 172.21.2.112:0/3385013626 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2c700a82f0 msgr2=0x7f2c701a73b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:55.678 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:55.691 7f2c749c5700 1 --2- 172.21.2.112:0/3385013626 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2c700a82f0 0x7f2c701a73b0 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 rx=0x7f2c68006f40 tx=0x7f2c6800bd40).stop 2021-02-10T11:32:55.678 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:55.691 7f2c749c5700 1 -- 172.21.2.112:0/3385013626 shutdown_connections 2021-02-10T11:32:55.678 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:55.691 7f2c749c5700 1 --2- 172.21.2.112:0/3385013626 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2c540186c0 0x7f2c5401abc0 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:55.679 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:55.691 7f2c749c5700 1 --2- 172.21.2.112:0/3385013626 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2c700a82f0 0x7f2c701a73b0 unknown :-1 s=CLOSED pgs=1172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:55.679 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:55.691 7f2c749c5700 1 --2- 172.21.2.112:0/3385013626 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2c700a7ef0 0x7f2c701a6d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:55.679 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:55.691 7f2c749c5700 1 --2- 172.21.2.112:0/3385013626 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2c700b4ad0 0x7f2c700a78d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:55.679 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:55.691 7f2c749c5700 1 -- 172.21.2.112:0/3385013626 >> 172.21.2.112:0/3385013626 conn(0x7f2c700b5560 msgr2=0x7f2c700a6200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:55.679 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:55.691 7f2c749c5700 1 -- 172.21.2.112:0/3385013626 shutdown_connections 2021-02-10T11:32:55.679 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:55.691 7f2c749c5700 1 -- 172.21.2.112:0/3385013626 wait complete. 2021-02-10T11:32:55.686 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:32:56.134 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.130 7fea9e1a2700 1 Processor -- start 2021-02-10T11:32:56.134 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.130 7fea9e1a2700 1 -- start start 2021-02-10T11:32:56.134 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.131 7fea9e1a2700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fea981b0870 0x7fea980f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:56.135 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.131 7fea9e1a2700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fea980efe00 0x7fea980f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:56.135 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.131 7fea9e1a2700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea981b0c70 con 0x7fea981b0470 2021-02-10T11:32:56.135 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.131 7fea9e1a2700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea980e5ea0 con 0x7fea981b0870 2021-02-10T11:32:56.135 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.131 7fea9e1a2700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea980f0b50 con 0x7fea980efe00 2021-02-10T11:32:56.135 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.131 7fea8effd700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fea981b0870 0x7fea980f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:56.135 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.131 7fea97fff700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fea980efe00 0x7fea980f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:56.135 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.131 7fea8effd700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fea981b0870 0x7fea980f20e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:49308/0 (socket says 172.21.2.112:49308) 2021-02-10T11:32:56.135 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.131 7fea8effd700 1 -- 172.21.2.112:0/993923677 learned_addr learned my addr 172.21.2.112:0/993923677 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:32:56.135 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.131 7fea8effd700 1 -- 172.21.2.112:0/993923677 >> v2:172.21.2.120:3300/0 conn(0x7fea980efe00 msgr2=0x7fea980f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:56.135 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.131 7fea8effd700 1 --2- 172.21.2.112:0/993923677 >> v2:172.21.2.120:3300/0 conn(0x7fea980efe00 0x7fea980f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:56.135 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.131 7fea8effd700 1 -- 172.21.2.112:0/993923677 >> v1:172.21.2.120:6789/0 conn(0x7fea981b0470 legacy=0x7fea980ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:56.135 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.131 7fea8effd700 1 -- 172.21.2.112:0/993923677 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea980f0820 con 0x7fea981b0870 2021-02-10T11:32:56.135 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.132 7fea8effd700 1 --2- 172.21.2.112:0/993923677 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fea981b0870 0x7fea980f20e0 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 rx=0x7fea80006700 tx=0x7fea80009b80).ready entity=mon.0 client_cookie=86e492f175466438 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:56.136 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.132 7fea96ffd700 1 -- 172.21.2.112:0/993923677 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fea8000c0c0 con 0x7fea981b0870 2021-02-10T11:32:56.136 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.132 7fea96ffd700 1 -- 172.21.2.112:0/993923677 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea8000c400 con 0x7fea981b0870 2021-02-10T11:32:56.136 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.132 7fea96ffd700 1 -- 172.21.2.112:0/993923677 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fea8000c880 con 0x7fea981b0870 2021-02-10T11:32:56.136 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.132 7fea9e1a2700 1 -- 172.21.2.112:0/993923677 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fea981b0870 msgr2=0x7fea980f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:56.136 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.132 7fea9e1a2700 1 --2- 172.21.2.112:0/993923677 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fea981b0870 0x7fea980f20e0 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 rx=0x7fea80006700 tx=0x7fea80009b80).stop 2021-02-10T11:32:56.136 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.133 7fea9e1a2700 1 -- 172.21.2.112:0/993923677 shutdown_connections 2021-02-10T11:32:56.136 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.133 7fea9e1a2700 1 --2- 172.21.2.112:0/993923677 >> v2:172.21.2.120:3300/0 conn(0x7fea980efe00 0x7fea980f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:56.136 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.133 7fea9e1a2700 1 --2- 172.21.2.112:0/993923677 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fea981b0870 0x7fea980f20e0 unknown :-1 s=CLOSED pgs=1173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:56.136 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.133 7fea9e1a2700 1 -- 172.21.2.112:0/993923677 >> 172.21.2.112:0/993923677 conn(0x7fea980ce3e0 msgr2=0x7fea980e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:56.136 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.133 7fea9e1a2700 1 -- 172.21.2.112:0/993923677 shutdown_connections 2021-02-10T11:32:56.136 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.133 7fea9e1a2700 1 -- 172.21.2.112:0/993923677 wait complete. 2021-02-10T11:32:56.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.147 7fea9e1a2700 1 Processor -- start 2021-02-10T11:32:56.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.148 7fea9e1a2700 1 -- start start 2021-02-10T11:32:56.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.148 7fea9e1a2700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fea98190b20 0x7fea980b3100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:56.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.148 7fea9e1a2700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fea980b3720 0x7fea980bc380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:56.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.148 7fea9e1a2700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fea980bc9a0 0x7fea980c2030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:56.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.148 7fea9e1a2700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea98190630 con 0x7fea98190b20 2021-02-10T11:32:56.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.148 7fea9e1a2700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea98190850 con 0x7fea980b3720 2021-02-10T11:32:56.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.148 7fea9e1a2700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea981a4970 con 0x7fea980bc9a0 2021-02-10T11:32:56.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.148 7fea8effd700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fea980b3720 0x7fea980bc380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:56.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.148 7fea8effd700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fea980b3720 0x7fea980bc380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3300/0 says I am v2:172.21.2.112:51590/0 (socket says 172.21.2.112:51590) 2021-02-10T11:32:56.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.148 7fea8effd700 1 -- 172.21.2.112:0/3523599589 learned_addr learned my addr 172.21.2.112:0/3523599589 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:32:56.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.148 7fea977fe700 1 --2- 172.21.2.112:0/3523599589 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fea98190b20 0x7fea980b3100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:56.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.148 7fea97fff700 1 --2- 172.21.2.112:0/3523599589 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fea980bc9a0 0x7fea980c2030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:56.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.148 7fea8effd700 1 -- 172.21.2.112:0/3523599589 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fea980bc9a0 msgr2=0x7fea980c2030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:56.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.148 7fea8effd700 1 --2- 172.21.2.112:0/3523599589 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fea980bc9a0 0x7fea980c2030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:56.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.149 7fea8effd700 1 -- 172.21.2.112:0/3523599589 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fea98190b20 msgr2=0x7fea980b3100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:56.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.149 7fea8effd700 1 --2- 172.21.2.112:0/3523599589 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fea98190b20 0x7fea980b3100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:56.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.149 7fea8effd700 1 -- 172.21.2.112:0/3523599589 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea980c2650 con 0x7fea980b3720 2021-02-10T11:32:56.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.149 7fea8effd700 1 --2- 172.21.2.112:0/3523599589 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fea980b3720 0x7fea980bc380 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7fea8000b530 tx=0x7fea8000b8d0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:56.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.149 7fea94ff9700 1 -- 172.21.2.112:0/3523599589 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fea800091b0 con 0x7fea980b3720 2021-02-10T11:32:56.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.149 7fea94ff9700 1 -- 172.21.2.112:0/3523599589 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea800091b0 con 0x7fea980b3720 2021-02-10T11:32:56.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.149 7fea9e1a2700 1 -- 172.21.2.112:0/3523599589 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea980c2650 con 0x7fea980b3720 2021-02-10T11:32:56.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.149 7fea94ff9700 1 -- 172.21.2.112:0/3523599589 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fea80002030 con 0x7fea980b3720 2021-02-10T11:32:56.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.149 7fea9e1a2700 1 -- 172.21.2.112:0/3523599589 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea980c2650 con 0x7fea980b3720 2021-02-10T11:32:56.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.149 7fea94ff9700 1 -- 172.21.2.112:0/3523599589 <== mon.1 v2:172.21.2.120:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fea800052e0 con 0x7fea980b3720 2021-02-10T11:32:56.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.149 7fea94ff9700 1 --2- 172.21.2.112:0/3523599589 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fea7c018730 0x7fea7c01abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:56.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.150 7fea94ff9700 1 -- 172.21.2.112:0/3523599589 <== mon.1 v2:172.21.2.120:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fea80032620 con 0x7fea980b3720 2021-02-10T11:32:56.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.150 7fea977fe700 1 --2- 172.21.2.112:0/3523599589 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fea7c018730 0x7fea7c01abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:56.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.150 7fea977fe700 1 --2- 172.21.2.112:0/3523599589 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fea7c018730 0x7fea7c01abf0 crc :-1 s=READY pgs=589 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:56.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.181 7fea9e1a2700 1 -- 172.21.2.112:0/3523599589 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea98105ae0 con 0x7fea980b3720 2021-02-10T11:32:56.170 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.183 7fea94ff9700 1 -- 172.21.2.112:0/3523599589 <== mon.1 v2:172.21.2.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fea800091b0 con 0x7fea980b3720 2021-02-10T11:32:56.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.334 7fea9e1a2700 1 -- 172.21.2.112:0/3523599589 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fea980f0b50 con 0x7fea980b3720 2021-02-10T11:32:56.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.336 7fea94ff9700 1 -- 172.21.2.112:0/3523599589 <== mon.1 v2:172.21.2.120:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v6)=0 v6) v1 ==== 122+0+0 (secure 0 0 0) 0x7fea800093c0 con 0x7fea980b3720 2021-02-10T11:32:56.327 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.339 7fea9e1a2700 1 -- 172.21.2.112:0/3523599589 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fea7c018730 msgr2=0x7fea7c01abf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:56.327 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.339 7fea9e1a2700 1 --2- 172.21.2.112:0/3523599589 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fea7c018730 0x7fea7c01abf0 crc :-1 s=READY pgs=589 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:56.327 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.339 7fea9e1a2700 1 -- 172.21.2.112:0/3523599589 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fea980b3720 msgr2=0x7fea980bc380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:56.327 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.339 7fea9e1a2700 1 --2- 172.21.2.112:0/3523599589 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fea980b3720 0x7fea980bc380 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7fea8000b530 tx=0x7fea8000b8d0).stop 2021-02-10T11:32:56.327 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.339 7fea9e1a2700 1 -- 172.21.2.112:0/3523599589 shutdown_connections 2021-02-10T11:32:56.327 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.339 7fea9e1a2700 1 --2- 172.21.2.112:0/3523599589 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fea7c018730 0x7fea7c01abf0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:56.327 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.339 7fea9e1a2700 1 --2- 172.21.2.112:0/3523599589 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fea980b3720 0x7fea980bc380 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:56.327 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.339 7fea9e1a2700 1 --2- 172.21.2.112:0/3523599589 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fea980bc9a0 0x7fea980c2030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:56.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.339 7fea9e1a2700 1 --2- 172.21.2.112:0/3523599589 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fea98190b20 0x7fea980b3100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:56.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.339 7fea9e1a2700 1 -- 172.21.2.112:0/3523599589 >> 172.21.2.112:0/3523599589 conn(0x7fea980bbf60 msgr2=0x7fea98105d10 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:56.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.340 7fea9e1a2700 1 -- 172.21.2.112:0/3523599589 shutdown_connections 2021-02-10T11:32:56.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.340 7fea9e1a2700 1 -- 172.21.2.112:0/3523599589 wait complete. 2021-02-10T11:32:56.335 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth get-or-create client.guest1 mds 'allow *' osd 'allow rw' mon 'allow *' 2021-02-10T11:32:56.786 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.781 7f6380287700 1 Processor -- start 2021-02-10T11:32:56.786 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.781 7f6380287700 1 -- start start 2021-02-10T11:32:56.786 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.781 7f6380287700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f63781b0470 0x7f63781b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:56.787 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.781 7f6380287700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f63780f20e0 0x7f63780f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:56.787 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.781 7f6380287700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63780f0b50 con 0x7f63780ef610 2021-02-10T11:32:56.787 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.781 7f6380287700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63780f0820 con 0x7f63781b0470 2021-02-10T11:32:56.787 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.781 7f6380287700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63780f24e0 con 0x7f63780f20e0 2021-02-10T11:32:56.787 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.782 7f637e0a3700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f63781b0470 0x7f63781b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:56.787 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.782 7f637e0a3700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f63781b0470 0x7f63781b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:49318/0 (socket says 172.21.2.112:49318) 2021-02-10T11:32:56.788 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.782 7f637e0a3700 1 -- 172.21.2.112:0/1875092481 learned_addr learned my addr 172.21.2.112:0/1875092481 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:32:56.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.782 7f637e0a3700 1 -- 172.21.2.112:0/1875092481 >> v2:172.21.2.120:3300/0 conn(0x7f63780f20e0 msgr2=0x7f63780f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:56.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.782 7f637e0a3700 1 --2- 172.21.2.112:0/1875092481 >> v2:172.21.2.120:3300/0 conn(0x7f63780f20e0 0x7f63780f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:56.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.782 7f637e0a3700 1 -- 172.21.2.112:0/1875092481 >> v1:172.21.2.120:6789/0 conn(0x7f63780ef610 legacy=0x7f63780efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:56.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.782 7f637e0a3700 1 -- 172.21.2.112:0/1875092481 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63780e52e0 con 0x7f63781b0470 2021-02-10T11:32:56.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.782 7f637e0a3700 1 --2- 172.21.2.112:0/1875092481 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f63781b0470 0x7f63781b0870 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 rx=0x7f6374006700 tx=0x7f6374009b00).ready entity=mon.0 client_cookie=c4959c726250e916 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:56.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.782 7f637d0a1700 1 -- 172.21.2.112:0/1875092481 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f637400c0c0 con 0x7f63781b0470 2021-02-10T11:32:56.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.782 7f637d0a1700 1 -- 172.21.2.112:0/1875092481 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f637400c340 con 0x7f63781b0470 2021-02-10T11:32:56.790 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.782 7f637d0a1700 1 -- 172.21.2.112:0/1875092481 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f637400c7a0 con 0x7f63781b0470 2021-02-10T11:32:56.790 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.782 7f6380287700 1 -- 172.21.2.112:0/1875092481 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f63781b0470 msgr2=0x7f63781b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:56.790 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.782 7f6380287700 1 --2- 172.21.2.112:0/1875092481 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f63781b0470 0x7f63781b0870 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 rx=0x7f6374006700 tx=0x7f6374009b00).stop 2021-02-10T11:32:56.790 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.783 7f6380287700 1 -- 172.21.2.112:0/1875092481 shutdown_connections 2021-02-10T11:32:56.790 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.783 7f6380287700 1 --2- 172.21.2.112:0/1875092481 >> v2:172.21.2.120:3300/0 conn(0x7f63780f20e0 0x7f63780f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:56.790 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.783 7f6380287700 1 --2- 172.21.2.112:0/1875092481 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f63781b0470 0x7f63781b0870 unknown :-1 s=CLOSED pgs=1174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:56.790 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.783 7f6380287700 1 -- 172.21.2.112:0/1875092481 >> 172.21.2.112:0/1875092481 conn(0x7f63780ce3e0 msgr2=0x7f63780e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:56.791 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.783 7f6380287700 1 -- 172.21.2.112:0/1875092481 shutdown_connections 2021-02-10T11:32:56.791 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.783 7f6380287700 1 -- 172.21.2.112:0/1875092481 wait complete. 2021-02-10T11:32:56.791 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.798 7f6380287700 1 Processor -- start 2021-02-10T11:32:56.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.798 7f6380287700 1 -- start start 2021-02-10T11:32:56.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.798 7f6380287700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f637818ecd0 0x7f637818e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:56.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.798 7f6380287700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f63781a3d90 0x7f63781ab630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:56.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.798 7f6380287700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f63781a4190 0x7f63781ac5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:56.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.798 7f6380287700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f637818f150 con 0x7f63781a3d90 2021-02-10T11:32:56.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.798 7f6380287700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63781abc50 con 0x7f63781a4190 2021-02-10T11:32:56.793 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.798 7f6380287700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63781abe70 con 0x7f637818ecd0 2021-02-10T11:32:56.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.799 7f637d8a2700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f63781a3d90 0x7f63781ab630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:56.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.799 7f637d8a2700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f63781a3d90 0x7f63781ab630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:49324/0 (socket says 172.21.2.112:49324) 2021-02-10T11:32:56.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.799 7f637d8a2700 1 -- 172.21.2.112:0/1445288097 learned_addr learned my addr 172.21.2.112:0/1445288097 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:32:56.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.799 7f637d8a2700 1 -- 172.21.2.112:0/1445288097 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f637818ecd0 msgr2=0x7f637818e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:56.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.799 7f637d8a2700 1 --2- 172.21.2.112:0/1445288097 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f637818ecd0 0x7f637818e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:56.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.799 7f637d8a2700 1 -- 172.21.2.112:0/1445288097 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f63781a4190 msgr2=0x7f63781ac5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:56.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.799 7f637d8a2700 1 --2- 172.21.2.112:0/1445288097 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f63781a4190 0x7f63781ac5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:56.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.799 7f637d8a2700 1 -- 172.21.2.112:0/1445288097 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63781acc30 con 0x7f63781a3d90 2021-02-10T11:32:56.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.799 7f637d8a2700 1 --2- 172.21.2.112:0/1445288097 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f63781a3d90 0x7f63781ab630 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 rx=0x7f636800b5b0 tx=0x7f636800b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:56.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.799 7f636effd700 1 -- 172.21.2.112:0/1445288097 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f636800bd40 con 0x7f63781a3d90 2021-02-10T11:32:56.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.799 7f636effd700 1 -- 172.21.2.112:0/1445288097 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6368000ce0 con 0x7f63781a3d90 2021-02-10T11:32:56.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.799 7f636effd700 1 -- 172.21.2.112:0/1445288097 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6368006210 con 0x7f63781a3d90 2021-02-10T11:32:56.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.799 7f6380287700 1 -- 172.21.2.112:0/1445288097 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63781acc30 con 0x7f63781a3d90 2021-02-10T11:32:56.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.800 7f6380287700 1 -- 172.21.2.112:0/1445288097 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63781acc30 con 0x7f63781a3d90 2021-02-10T11:32:56.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.800 7f636effd700 1 -- 172.21.2.112:0/1445288097 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f63680054a0 con 0x7f63781a3d90 2021-02-10T11:32:56.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.800 7f636effd700 1 --2- 172.21.2.112:0/1445288097 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f63600186c0 0x7f636001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:56.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.800 7f636effd700 1 -- 172.21.2.112:0/1445288097 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f6368032590 con 0x7f63781a3d90 2021-02-10T11:32:56.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.801 7f637e0a3700 1 --2- 172.21.2.112:0/1445288097 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f63600186c0 0x7f636001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:56.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.801 7f637e0a3700 1 --2- 172.21.2.112:0/1445288097 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f63600186c0 0x7f636001abc0 crc :-1 s=READY pgs=590 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:56.818 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.831 7f6380287700 1 -- 172.21.2.112:0/1445288097 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63780edd30 con 0x7f63781a3d90 2021-02-10T11:32:56.821 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.834 7f636effd700 1 -- 172.21.2.112:0/1445288097 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6368030d40 con 0x7f63781a3d90 2021-02-10T11:32:56.954 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.967 7f6380287700 1 -- 172.21.2.112:0/1445288097 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.guest1", "caps": ["mds", "allow *", "osd", "allow rw", "mon", "allow *"]} v 0) v1 -- 0x7f63780edd30 con 0x7f63781a3d90 2021-02-10T11:32:56.958 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.970 7f636effd700 1 -- 172.21.2.112:0/1445288097 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.guest1", "caps": ["mds", "allow *", "osd", "allow rw", "mon", "allow *"]}]=0 v29) v1 ==== 158+0+64 (secure 0 0 0) 0x7f636800bd40 con 0x7f63781a3d90 2021-02-10T11:32:56.962 INFO:teuthology.orchestra.run.gibba012.stdout:[client.guest1] 2021-02-10T11:32:56.962 INFO:teuthology.orchestra.run.gibba012.stdout: key = AQBoxCNgW5S4ORAA2JS7VFffl3+jdcHHPt24LA== 2021-02-10T11:32:56.963 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.976 7f6380287700 1 -- 172.21.2.112:0/1445288097 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f63600186c0 msgr2=0x7f636001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:56.963 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.976 7f6380287700 1 --2- 172.21.2.112:0/1445288097 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f63600186c0 0x7f636001abc0 crc :-1 s=READY pgs=590 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:56.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.976 7f6380287700 1 -- 172.21.2.112:0/1445288097 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f63781a3d90 msgr2=0x7f63781ab630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:56.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.976 7f6380287700 1 --2- 172.21.2.112:0/1445288097 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f63781a3d90 0x7f63781ab630 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 rx=0x7f636800b5b0 tx=0x7f636800b950).stop 2021-02-10T11:32:56.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.976 7f6380287700 1 -- 172.21.2.112:0/1445288097 shutdown_connections 2021-02-10T11:32:56.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.976 7f6380287700 1 --2- 172.21.2.112:0/1445288097 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f63600186c0 0x7f636001abc0 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:56.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.976 7f6380287700 1 --2- 172.21.2.112:0/1445288097 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f63781a4190 0x7f63781ac5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:56.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.976 7f6380287700 1 --2- 172.21.2.112:0/1445288097 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f63781a3d90 0x7f63781ab630 unknown :-1 s=CLOSED pgs=1175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:56.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.976 7f6380287700 1 --2- 172.21.2.112:0/1445288097 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f637818ecd0 0x7f637818e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:56.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.976 7f6380287700 1 -- 172.21.2.112:0/1445288097 >> 172.21.2.112:0/1445288097 conn(0x7f63781a90c0 msgr2=0x7f63781b38b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:56.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.976 7f6380287700 1 -- 172.21.2.112:0/1445288097 shutdown_connections 2021-02-10T11:32:56.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:56.976 7f6380287700 1 -- 172.21.2.112:0/1445288097 wait complete. 2021-02-10T11:32:56.971 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolumegroup create cephfs group_0000000001022050 2021-02-10T11:32:57.400 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:57.397 7f8cd8314700 1 Processor -- start 2021-02-10T11:32:57.401 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:57.397 7f8cd8314700 1 -- start start 2021-02-10T11:32:57.401 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:57.397 7f8cd8314700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f8cd01b0470 0x7f8cd01b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:57.401 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:57.397 7f8cd8314700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8cd00ef610 0x7f8cd00efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:57.401 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:57.397 7f8cd8314700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cd00f0b50 con 0x7f8cd00f0030 2021-02-10T11:32:57.401 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:57.397 7f8cd8314700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cd00e52e0 con 0x7f8cd01b0470 2021-02-10T11:32:57.401 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:57.397 7f8cd8314700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cd00e54d0 con 0x7f8cd00ef610 2021-02-10T11:32:57.401 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:57.398 7f8cd6130700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f8cd01b0470 0x7f8cd01b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:57.401 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:57.398 7f8cd6130700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f8cd01b0470 0x7f8cd01b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:49330/0 (socket says 172.21.2.112:49330) 2021-02-10T11:32:57.401 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:57.398 7f8cd6130700 1 -- 172.21.2.112:0/3116554605 learned_addr learned my addr 172.21.2.112:0/3116554605 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:32:57.402 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:57.398 7f8cd6130700 1 -- 172.21.2.112:0/3116554605 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8cd00ef610 msgr2=0x7f8cd00efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:57.402 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:57.398 7f8cd6130700 1 --2- 172.21.2.112:0/3116554605 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8cd00ef610 0x7f8cd00efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:57.402 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:57.398 7f8cd6130700 1 -- 172.21.2.112:0/3116554605 >> v1:172.21.2.120:6789/0 conn(0x7f8cd00f0030 legacy=0x7f8cd00f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:57.402 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:57.398 7f8cd6130700 1 -- 172.21.2.112:0/3116554605 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cd00f0430 con 0x7f8cd01b0470 2021-02-10T11:32:57.402 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:57.398 7f8cd6130700 1 --2- 172.21.2.112:0/3116554605 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8cd01b0470 0x7f8cd01b0870 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 rx=0x7f8cc4006700 tx=0x7f8cc4009b80).ready entity=mon.0 client_cookie=4ba7a5e6285bfaaf server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:57.402 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:57.398 7f8cd512e700 1 -- 172.21.2.112:0/3116554605 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8cc400c0c0 con 0x7f8cd01b0470 2021-02-10T11:32:57.402 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:57.398 7f8cd512e700 1 -- 172.21.2.112:0/3116554605 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cc400c400 con 0x7f8cd01b0470 2021-02-10T11:32:57.402 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:57.398 7f8cd512e700 1 -- 172.21.2.112:0/3116554605 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8cc400c920 con 0x7f8cd01b0470 2021-02-10T11:32:57.402 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:57.398 7f8cd8314700 1 -- 172.21.2.112:0/3116554605 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8cd01b0470 msgr2=0x7f8cd01b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:57.402 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:57.398 7f8cd8314700 1 --2- 172.21.2.112:0/3116554605 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8cd01b0470 0x7f8cd01b0870 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 rx=0x7f8cc4006700 tx=0x7f8cc4009b80).stop 2021-02-10T11:32:57.402 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:57.399 7f8cd8314700 1 -- 172.21.2.112:0/3116554605 shutdown_connections 2021-02-10T11:32:57.403 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:57.399 7f8cd8314700 1 --2- 172.21.2.112:0/3116554605 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8cd00ef610 0x7f8cd00efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:57.403 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:57.399 7f8cd8314700 1 --2- 172.21.2.112:0/3116554605 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8cd01b0470 0x7f8cd01b0870 unknown :-1 s=CLOSED pgs=1176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:57.403 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:57.399 7f8cd8314700 1 -- 172.21.2.112:0/3116554605 >> 172.21.2.112:0/3116554605 conn(0x7f8cd00ce3e0 msgr2=0x7f8cd00e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:57.403 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:57.399 7f8cd8314700 1 -- 172.21.2.112:0/3116554605 shutdown_connections 2021-02-10T11:32:57.403 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:57.399 7f8cd8314700 1 -- 172.21.2.112:0/3116554605 wait complete. 2021-02-10T11:32:57.405 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:57.414 7f8cd8314700 1 Processor -- start 2021-02-10T11:32:57.405 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:57.414 7f8cd8314700 1 -- start start 2021-02-10T11:32:57.405 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:57.414 7f8cd8314700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8cd00b2840 0x7f8cd01a4300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:57.405 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:57.414 7f8cd8314700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8cd00b30d0 0x7f8cd00b34d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:57.405 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:57.414 7f8cd8314700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8cd00b3af0 0x7f8cd00f0580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:57.405 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:57.414 7f8cd8314700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cd00b2c40 con 0x7f8cd00b2840 2021-02-10T11:32:57.405 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:57.414 7f8cd8314700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cd01a4920 con 0x7f8cd00b30d0 2021-02-10T11:32:57.405 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:57.414 7f8cd8314700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cd00f0ba0 con 0x7f8cd00b3af0 2021-02-10T11:32:57.406 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:57.414 7f8cd6130700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8cd00b2840 0x7f8cd01a4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:57.406 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:57.414 7f8cd6130700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8cd00b2840 0x7f8cd01a4300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:49332/0 (socket says 172.21.2.112:49332) 2021-02-10T11:32:57.406 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:57.414 7f8cd6130700 1 -- 172.21.2.112:0/820715955 learned_addr learned my addr 172.21.2.112:0/820715955 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:32:57.406 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:57.415 7f8cd592f700 1 --2- 172.21.2.112:0/820715955 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8cd00b30d0 0x7f8cd00b34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:57.406 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:57.415 7f8cd6931700 1 --2- 172.21.2.112:0/820715955 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8cd00b3af0 0x7f8cd00f0580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:57.406 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:57.415 7f8cd6130700 1 -- 172.21.2.112:0/820715955 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8cd00b3af0 msgr2=0x7f8cd00f0580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:57.406 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:57.415 7f8cd6130700 1 --2- 172.21.2.112:0/820715955 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8cd00b3af0 0x7f8cd00f0580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:57.406 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:57.415 7f8cd6130700 1 -- 172.21.2.112:0/820715955 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8cd00b30d0 msgr2=0x7f8cd00b34d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:57.406 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:57.415 7f8cd6130700 1 --2- 172.21.2.112:0/820715955 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8cd00b30d0 0x7f8cd00b34d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:57.406 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:57.415 7f8cd6130700 1 -- 172.21.2.112:0/820715955 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cd00f0dc0 con 0x7f8cd00b2840 2021-02-10T11:32:57.406 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:57.415 7f8cd6931700 1 --2- 172.21.2.112:0/820715955 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8cd00b3af0 0x7f8cd00f0580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:32:57.407 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:57.415 7f8cd6130700 1 --2- 172.21.2.112:0/820715955 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8cd00b2840 0x7f8cd01a4300 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 rx=0x7f8cc400b650 tx=0x7f8cc400b9f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:57.407 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:57.415 7f8cc2ffd700 1 -- 172.21.2.112:0/820715955 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8cc400bde0 con 0x7f8cd00b2840 2021-02-10T11:32:57.407 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:57.415 7f8cc2ffd700 1 -- 172.21.2.112:0/820715955 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cc4006570 con 0x7f8cd00b2840 2021-02-10T11:32:57.407 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:57.415 7f8cc2ffd700 1 -- 172.21.2.112:0/820715955 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8cc40022b0 con 0x7f8cd00b2840 2021-02-10T11:32:57.407 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:57.415 7f8cd8314700 1 -- 172.21.2.112:0/820715955 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cd00f0dc0 con 0x7f8cd00b2840 2021-02-10T11:32:57.407 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:57.415 7f8cd8314700 1 -- 172.21.2.112:0/820715955 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cd00f0dc0 con 0x7f8cd00b2840 2021-02-10T11:32:57.407 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:57.416 7f8cc2ffd700 1 -- 172.21.2.112:0/820715955 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8cc4009f90 con 0x7f8cd00b2840 2021-02-10T11:32:57.407 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:57.416 7f8cc2ffd700 1 --2- 172.21.2.112:0/820715955 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8cb80186c0 0x7f8cb801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:57.407 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:57.416 7f8cc2ffd700 1 -- 172.21.2.112:0/820715955 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f8cc4032670 con 0x7f8cd00b2840 2021-02-10T11:32:57.407 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:57.416 7f8cd592f700 1 --2- 172.21.2.112:0/820715955 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8cb80186c0 0x7f8cb801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:57.407 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:57.417 7f8cd592f700 1 --2- 172.21.2.112:0/820715955 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8cb80186c0 0x7f8cb801abc0 crc :-1 s=READY pgs=591 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:57.435 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:57.447 7f8cd8314700 1 -- 172.21.2.112:0/820715955 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cd00f0050 con 0x7f8cd00b2840 2021-02-10T11:32:57.436 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:57.450 7f8cc2ffd700 1 -- 172.21.2.112:0/820715955 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8cc4000c90 con 0x7f8cd00b2840 2021-02-10T11:32:57.565 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:57.578 7f8cd8314700 1 -- 172.21.2.112:0/820715955 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "target": ["mgr", ""], "group_name": "group_0000000001022050"}) v1 -- 0x7f8cd00f0050 con 0x7f8cb80186c0 2021-02-10T11:32:57.571 INFO:tasks.ceph.mgr.x.gibba020.stderr:2021-02-10 11:32:57.587 7f6382eee700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:32:57.571 INFO:tasks.ceph.mgr.x.gibba020.stderr:2021-02-10 11:32:57.587 7f6382eee700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:32:57.571 INFO:tasks.ceph.mgr.x.gibba020.stderr:2021-02-10 11:32:57.587 7f6382eee700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:32:57.571 INFO:tasks.ceph.mgr.x.gibba020.stderr:2021-02-10 11:32:57.587 7f6382eee700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:32:57.571 INFO:tasks.ceph.mgr.x.gibba020.stderr:2021-02-10 11:32:57.587 7f6382eee700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:32:57.582 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:57.595 7f8cc2ffd700 1 -- 172.21.2.112:0/820715955 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f8ccc0009b0 con 0x7f8cb80186c0 2021-02-10T11:32:57.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:57.612 7f8cd8314700 1 -- 172.21.2.112:0/820715955 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8cb80186c0 msgr2=0x7f8cb801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:57.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:57.612 7f8cd8314700 1 --2- 172.21.2.112:0/820715955 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8cb80186c0 0x7f8cb801abc0 crc :-1 s=READY pgs=591 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:57.600 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:57.612 7f8cd8314700 1 -- 172.21.2.112:0/820715955 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8cd00b2840 msgr2=0x7f8cd01a4300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:57.600 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:57.612 7f8cd8314700 1 --2- 172.21.2.112:0/820715955 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8cd00b2840 0x7f8cd01a4300 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 rx=0x7f8cc400b650 tx=0x7f8cc400b9f0).stop 2021-02-10T11:32:57.600 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:57.612 7f8cd8314700 1 -- 172.21.2.112:0/820715955 shutdown_connections 2021-02-10T11:32:57.600 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:57.612 7f8cd8314700 1 --2- 172.21.2.112:0/820715955 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8cb80186c0 0x7f8cb801abc0 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:57.600 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:57.612 7f8cd8314700 1 --2- 172.21.2.112:0/820715955 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8cd00b30d0 0x7f8cd00b34d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:57.600 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:57.612 7f8cd8314700 1 --2- 172.21.2.112:0/820715955 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8cd00b3af0 0x7f8cd00f0580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:57.600 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:57.612 7f8cd8314700 1 --2- 172.21.2.112:0/820715955 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8cd00b2840 0x7f8cd01a4300 unknown :-1 s=CLOSED pgs=1177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:57.600 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:57.612 7f8cd8314700 1 -- 172.21.2.112:0/820715955 >> 172.21.2.112:0/820715955 conn(0x7f8cd00c36f0 msgr2=0x7f8cd0094c30 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:57.600 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:57.612 7f8cd8314700 1 -- 172.21.2.112:0/820715955 shutdown_connections 2021-02-10T11:32:57.600 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:57.612 7f8cd8314700 1 -- 172.21.2.112:0/820715955 wait complete. 2021-02-10T11:32:57.607 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000592966 --group_name group_0000000001022050 2021-02-10T11:32:58.038 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.033 7f4a44388700 1 Processor -- start 2021-02-10T11:32:58.038 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.033 7f4a44388700 1 -- start start 2021-02-10T11:32:58.038 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.034 7f4a44388700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f4a3c0d0330 0x7f4a3c0ef630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:58.038 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.034 7f4a44388700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a3c0f0b00 con 0x7f4a3c0d1860 2021-02-10T11:32:58.038 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.034 7f4a44388700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a3c1b0bd0 con 0x7f4a3c0d1c60 2021-02-10T11:32:58.038 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.034 7f4a44388700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a3c0e5ec0 con 0x7f4a3c0d0330 2021-02-10T11:32:58.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.034 7f4a421a4700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f4a3c0d1860 0x7f4a3c1b03e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:48802/0 (socket says 172.21.2.112:48802) 2021-02-10T11:32:58.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.034 7f4a421a4700 1 -- 172.21.2.112:0/3406576446 learned_addr learned my addr 172.21.2.112:0/3406576446 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:32:58.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.035 7f4a419a3700 1 --2- 172.21.2.112:0/3406576446 >> v2:172.21.2.120:3300/0 conn(0x7f4a3c0d0330 0x7f4a3c0ef630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:58.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.035 7f4a411a2700 1 -- 172.21.2.112:0/3406576446 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3469315251 0 0) 0x7f4a300010b0 con 0x7f4a3c0d1860 2021-02-10T11:32:58.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.035 7f4a411a2700 1 -- 172.21.2.112:0/3406576446 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a28002fe0 con 0x7f4a3c0d1860 2021-02-10T11:32:58.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.035 7f4a411a2700 1 -- 172.21.2.112:0/3406576446 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 789058291 0 0) 0x7f4a300010b0 con 0x7f4a3c0d1860 2021-02-10T11:32:58.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.035 7f4a411a2700 1 -- 172.21.2.112:0/3406576446 >> v2:172.21.2.120:3300/0 conn(0x7f4a3c0d0330 msgr2=0x7f4a3c0ef630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:58.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.035 7f4a411a2700 1 --2- 172.21.2.112:0/3406576446 >> v2:172.21.2.120:3300/0 conn(0x7f4a3c0d0330 0x7f4a3c0ef630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:58.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.035 7f4a411a2700 1 -- 172.21.2.112:0/3406576446 >> v1:172.21.2.120:6789/0 conn(0x7f4a3c0d1c60 legacy=0x7f4a3c0efc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:58.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.035 7f4a411a2700 1 -- 172.21.2.112:0/3406576446 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a3c0f0440 con 0x7f4a3c0d1860 2021-02-10T11:32:58.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.035 7f4a411a2700 1 -- 172.21.2.112:0/3406576446 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f4a300015b0 con 0x7f4a3c0d1860 2021-02-10T11:32:58.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.035 7f4a411a2700 1 -- 172.21.2.112:0/3406576446 >> v1:172.21.2.112:6789/0 conn(0x7f4a3c0d1860 legacy=0x7f4a3c1b03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:58.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.035 7f4a411a2700 1 --2- 172.21.2.112:0/3406576446 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4a28003b10 0x7f4a28005f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:58.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.035 7f4a411a2700 1 --2- 172.21.2.112:0/3406576446 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4a28006820 0x7f4a28008d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:58.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.035 7f4a411a2700 1 --2- 172.21.2.112:0/3406576446 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4a28009620 0x7f4a2800bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:58.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.035 7f4a411a2700 1 -- 172.21.2.112:0/3406576446 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a2800c390 con 0x7f4a28006820 2021-02-10T11:32:58.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.035 7f4a411a2700 1 -- 172.21.2.112:0/3406576446 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a2800c5b0 con 0x7f4a28003b10 2021-02-10T11:32:58.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.035 7f4a411a2700 1 -- 172.21.2.112:0/3406576446 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a2800c7d0 con 0x7f4a28009620 2021-02-10T11:32:58.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.035 7f4a419a3700 1 --2- 172.21.2.112:0/3406576446 >> v2:172.21.2.120:3300/0 conn(0x7f4a3c0d0330 0x7f4a3c0ef630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:32:58.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.035 7f4a419a3700 1 --2- 172.21.2.112:0/3406576446 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4a28006820 0x7f4a28008d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:58.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.035 7f4a421a4700 1 --2- 172.21.2.112:0/3406576446 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4a28003b10 0x7f4a28005f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:58.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.035 7f4a429a5700 1 --2- 172.21.2.112:0/3406576446 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4a28009620 0x7f4a2800bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:58.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.035 7f4a419a3700 1 -- 172.21.2.112:0/3406576446 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4a28009620 msgr2=0x7f4a2800bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:58.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.035 7f4a419a3700 1 --2- 172.21.2.112:0/3406576446 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4a28009620 0x7f4a2800bb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:58.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.035 7f4a419a3700 1 -- 172.21.2.112:0/3406576446 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4a28003b10 msgr2=0x7f4a28005f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:58.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.035 7f4a419a3700 1 --2- 172.21.2.112:0/3406576446 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4a28003b10 0x7f4a28005f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:58.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.035 7f4a429a5700 1 -- 172.21.2.112:0/3406576446 reap_dead start 2021-02-10T11:32:58.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.035 7f4a419a3700 1 -- 172.21.2.112:0/3406576446 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a28050a40 con 0x7f4a28006820 2021-02-10T11:32:58.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.036 7f4a419a3700 1 --2- 172.21.2.112:0/3406576446 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4a28006820 0x7f4a28008d20 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 rx=0x7f4a38000b80 tx=0x7f4a38009d20).ready entity=mon.0 client_cookie=92afec70ecda8dac server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:58.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.036 7f4a411a2700 1 -- 172.21.2.112:0/3406576446 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4a38006cd0 con 0x7f4a28006820 2021-02-10T11:32:58.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.036 7f4a411a2700 1 -- 172.21.2.112:0/3406576446 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a38007070 con 0x7f4a28006820 2021-02-10T11:32:58.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.036 7f4a411a2700 1 -- 172.21.2.112:0/3406576446 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4a38007590 con 0x7f4a28006820 2021-02-10T11:32:58.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.036 7f4a44388700 1 -- 172.21.2.112:0/3406576446 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4a28006820 msgr2=0x7f4a28008d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:58.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.036 7f4a44388700 1 --2- 172.21.2.112:0/3406576446 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4a28006820 0x7f4a28008d20 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 rx=0x7f4a38000b80 tx=0x7f4a38009d20).stop 2021-02-10T11:32:58.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.036 7f4a44388700 1 -- 172.21.2.112:0/3406576446 shutdown_connections 2021-02-10T11:32:58.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.036 7f4a44388700 1 --2- 172.21.2.112:0/3406576446 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4a28006820 0x7f4a28008d20 unknown :-1 s=CLOSED pgs=1179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:58.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.036 7f4a44388700 1 -- 172.21.2.112:0/3406576446 >> 172.21.2.112:0/3406576446 conn(0x7f4a3c0ce440 msgr2=0x7f4a3c0d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:58.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.036 7f4a44388700 1 -- 172.21.2.112:0/3406576446 shutdown_connections 2021-02-10T11:32:58.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.036 7f4a44388700 1 -- 172.21.2.112:0/3406576446 wait complete. 2021-02-10T11:32:58.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.051 7f4a44388700 1 Processor -- start 2021-02-10T11:32:58.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.051 7f4a44388700 1 -- start start 2021-02-10T11:32:58.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.051 7f4a44388700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4a3c098b00 0x7f4a3c098f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:58.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.051 7f4a44388700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4a3c099520 0x7f4a3c099d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:58.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.051 7f4a44388700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4a3c09a330 0x7f4a3c09b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:58.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.051 7f4a44388700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a3c09adf0 con 0x7f4a3c09a330 2021-02-10T11:32:58.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.051 7f4a44388700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a3c19d2e0 con 0x7f4a3c098b00 2021-02-10T11:32:58.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.051 7f4a44388700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a3c09a730 con 0x7f4a3c099520 2021-02-10T11:32:58.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.052 7f4a429a5700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4a3c09a330 0x7f4a3c09b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:58.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.052 7f4a429a5700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4a3c09a330 0x7f4a3c09b2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:49356/0 (socket says 172.21.2.112:49356) 2021-02-10T11:32:58.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.052 7f4a429a5700 1 -- 172.21.2.112:0/3758488662 learned_addr learned my addr 172.21.2.112:0/3758488662 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:32:58.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.052 7f4a419a3700 1 --2- 172.21.2.112:0/3758488662 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4a3c099520 0x7f4a3c099d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:58.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.052 7f4a421a4700 1 --2- 172.21.2.112:0/3758488662 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4a3c098b00 0x7f4a3c098f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:58.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.052 7f4a429a5700 1 -- 172.21.2.112:0/3758488662 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4a3c099520 msgr2=0x7f4a3c099d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:58.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.052 7f4a429a5700 1 --2- 172.21.2.112:0/3758488662 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4a3c099520 0x7f4a3c099d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:58.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.052 7f4a429a5700 1 -- 172.21.2.112:0/3758488662 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4a3c098b00 msgr2=0x7f4a3c098f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:58.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.052 7f4a429a5700 1 --2- 172.21.2.112:0/3758488662 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4a3c098b00 0x7f4a3c098f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:58.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.052 7f4a429a5700 1 -- 172.21.2.112:0/3758488662 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a3c09a950 con 0x7f4a3c09a330 2021-02-10T11:32:58.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.052 7f4a421a4700 1 --2- 172.21.2.112:0/3758488662 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4a3c098b00 0x7f4a3c098f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:32:58.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.052 7f4a429a5700 1 --2- 172.21.2.112:0/3758488662 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4a3c09a330 0x7f4a3c09b2f0 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 rx=0x7f4a34006900 tx=0x7f4a3400bd20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:58.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.052 7f4a2effd700 1 -- 172.21.2.112:0/3758488662 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4a34006240 con 0x7f4a3c09a330 2021-02-10T11:32:58.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.053 7f4a2effd700 1 -- 172.21.2.112:0/3758488662 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a3400be60 con 0x7f4a3c09a330 2021-02-10T11:32:58.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.053 7f4a2effd700 1 -- 172.21.2.112:0/3758488662 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4a340056a0 con 0x7f4a3c09a330 2021-02-10T11:32:58.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.053 7f4a44388700 1 -- 172.21.2.112:0/3758488662 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a3c09a950 con 0x7f4a3c09a330 2021-02-10T11:32:58.045 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.053 7f4a44388700 1 -- 172.21.2.112:0/3758488662 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a3c09bc00 con 0x7f4a3c09a330 2021-02-10T11:32:58.045 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.053 7f4a2effd700 1 -- 172.21.2.112:0/3758488662 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4a34006030 con 0x7f4a3c09a330 2021-02-10T11:32:58.045 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.053 7f4a2effd700 1 --2- 172.21.2.112:0/3758488662 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4a240186c0 0x7f4a2401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:58.045 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.053 7f4a2effd700 1 -- 172.21.2.112:0/3758488662 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f4a34032710 con 0x7f4a3c09a330 2021-02-10T11:32:58.045 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.054 7f4a421a4700 1 --2- 172.21.2.112:0/3758488662 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4a240186c0 0x7f4a2401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:58.045 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.054 7f4a421a4700 1 --2- 172.21.2.112:0/3758488662 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4a240186c0 0x7f4a2401abc0 crc :-1 s=READY pgs=592 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:58.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.084 7f4a44388700 1 -- 172.21.2.112:0/3758488662 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a3c0ebbe0 con 0x7f4a3c09a330 2021-02-10T11:32:58.073 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.086 7f4a2effd700 1 -- 172.21.2.112:0/3758488662 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4a34006a50 con 0x7f4a3c09a330 2021-02-10T11:32:58.200 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.212 7f4a44388700 1 -- 172.21.2.112:0/3758488662 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000592966", "target": ["mgr", ""], "group_name": "group_0000000001022050"}) v1 -- 0x7f4a3c09bc00 con 0x7f4a240186c0 2021-02-10T11:32:58.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.238 7f4a2effd700 1 -- 172.21.2.112:0/3758488662 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f4a300009b0 con 0x7f4a240186c0 2021-02-10T11:32:58.234 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.247 7f4a44388700 1 -- 172.21.2.112:0/3758488662 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4a240186c0 msgr2=0x7f4a2401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:58.234 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.247 7f4a44388700 1 --2- 172.21.2.112:0/3758488662 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4a240186c0 0x7f4a2401abc0 crc :-1 s=READY pgs=592 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:58.234 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.247 7f4a44388700 1 -- 172.21.2.112:0/3758488662 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4a3c09a330 msgr2=0x7f4a3c09b2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:58.234 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.247 7f4a44388700 1 --2- 172.21.2.112:0/3758488662 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4a3c09a330 0x7f4a3c09b2f0 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 rx=0x7f4a34006900 tx=0x7f4a3400bd20).stop 2021-02-10T11:32:58.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.247 7f4a44388700 1 -- 172.21.2.112:0/3758488662 shutdown_connections 2021-02-10T11:32:58.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.247 7f4a44388700 1 --2- 172.21.2.112:0/3758488662 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4a240186c0 0x7f4a2401abc0 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:58.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.247 7f4a44388700 1 --2- 172.21.2.112:0/3758488662 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4a3c09a330 0x7f4a3c09b2f0 unknown :-1 s=CLOSED pgs=1180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:58.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.247 7f4a44388700 1 --2- 172.21.2.112:0/3758488662 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4a3c099520 0x7f4a3c099d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:58.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.247 7f4a44388700 1 --2- 172.21.2.112:0/3758488662 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4a3c098b00 0x7f4a3c098f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:58.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.247 7f4a44388700 1 -- 172.21.2.112:0/3758488662 >> 172.21.2.112:0/3758488662 conn(0x7f4a3c0b2a80 msgr2=0x7f4a3c09e3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:58.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.247 7f4a44388700 1 -- 172.21.2.112:0/3758488662 shutdown_connections 2021-02-10T11:32:58.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.247 7f4a44388700 1 -- 172.21.2.112:0/3758488662 wait complete. 2021-02-10T11:32:58.242 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume authorize cephfs subvolume_0000000000592966 guest1 --group_name group_0000000001022050 --tenant_id tenant1 2021-02-10T11:32:58.681 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.676 7fc884271700 1 Processor -- start 2021-02-10T11:32:58.681 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.676 7fc884271700 1 -- start start 2021-02-10T11:32:58.681 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.676 7fc884271700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fc87c0e5010 0x7fc87c1b37c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:58.681 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.676 7fc884271700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc87c0e5410 0x7fc87c0f6a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:58.681 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.676 7fc884271700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc87c0f5480 con 0x7fc87c1b3de0 2021-02-10T11:32:58.681 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.676 7fc884271700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc87c0d1750 con 0x7fc87c0e5010 2021-02-10T11:32:58.681 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.676 7fc884271700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc87c0d1940 con 0x7fc87c0e5410 2021-02-10T11:32:58.681 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.677 7fc88288e700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc87c0e5410 0x7fc87c0f6a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:58.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.677 7fc88208d700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fc87c0e5010 0x7fc87c1b37c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:58.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.677 7fc88188c700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7fc87c1b3de0 0x7fc87c0f3dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:42380/0 (socket says 172.21.2.112:42380) 2021-02-10T11:32:58.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.677 7fc88188c700 1 -- 172.21.2.112:0/3369319015 learned_addr learned my addr 172.21.2.112:0/3369319015 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:32:58.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.678 7fc88208d700 1 -- 172.21.2.112:0/3369319015 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc87c0e5410 msgr2=0x7fc87c0f6a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:58.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.678 7fc88208d700 1 --2- 172.21.2.112:0/3369319015 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc87c0e5410 0x7fc87c0f6a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:58.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.678 7fc88208d700 1 -- 172.21.2.112:0/3369319015 >> v1:172.21.2.120:6789/0 conn(0x7fc87c1b3de0 legacy=0x7fc87c0f3dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:58.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.678 7fc88208d700 1 -- 172.21.2.112:0/3369319015 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc87c0d1b60 con 0x7fc87c0e5010 2021-02-10T11:32:58.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.678 7fc88208d700 1 --2- 172.21.2.112:0/3369319015 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc87c0e5010 0x7fc87c1b37c0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7fc870006650 tx=0x7fc870009a80).ready entity=mon.1 client_cookie=6857663ae296b9a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:58.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.678 7fc88108b700 1 -- 172.21.2.112:0/3369319015 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc87000c0c0 con 0x7fc87c0e5010 2021-02-10T11:32:58.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.678 7fc88108b700 1 -- 172.21.2.112:0/3369319015 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc87000c340 con 0x7fc87c0e5010 2021-02-10T11:32:58.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.678 7fc88108b700 1 -- 172.21.2.112:0/3369319015 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc87000c860 con 0x7fc87c0e5010 2021-02-10T11:32:58.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.678 7fc884271700 1 -- 172.21.2.112:0/3369319015 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc87c0e5010 msgr2=0x7fc87c1b37c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:58.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.678 7fc884271700 1 --2- 172.21.2.112:0/3369319015 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc87c0e5010 0x7fc87c1b37c0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7fc870006650 tx=0x7fc870009a80).stop 2021-02-10T11:32:58.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.679 7fc884271700 1 -- 172.21.2.112:0/3369319015 shutdown_connections 2021-02-10T11:32:58.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.679 7fc884271700 1 --2- 172.21.2.112:0/3369319015 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc87c0e5410 0x7fc87c0f6a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:58.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.679 7fc884271700 1 --2- 172.21.2.112:0/3369319015 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc87c0e5010 0x7fc87c1b37c0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:58.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.679 7fc884271700 1 -- 172.21.2.112:0/3369319015 >> 172.21.2.112:0/3369319015 conn(0x7fc87c0ce670 msgr2=0x7fc87c0e44b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:58.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.679 7fc884271700 1 -- 172.21.2.112:0/3369319015 shutdown_connections 2021-02-10T11:32:58.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.679 7fc884271700 1 -- 172.21.2.112:0/3369319015 wait complete. 2021-02-10T11:32:58.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.694 7fc884271700 1 Processor -- start 2021-02-10T11:32:58.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.694 7fc884271700 1 -- start start 2021-02-10T11:32:58.685 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.694 7fc884271700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc87c1898a0 0x7fc87c0a9a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:58.685 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.694 7fc884271700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc87c189ca0 0x7fc87c0aa090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:58.685 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.694 7fc884271700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc87c0aa6b0 0x7fc87c188450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:58.685 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.694 7fc884271700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc87c1076e0 con 0x7fc87c0aa6b0 2021-02-10T11:32:58.685 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.694 7fc884271700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc87c188a70 con 0x7fc87c1898a0 2021-02-10T11:32:58.685 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.694 7fc884271700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc87c188c90 con 0x7fc87c189ca0 2021-02-10T11:32:58.685 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.694 7fc88288e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc87c0aa6b0 0x7fc87c188450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:58.685 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.695 7fc88288e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc87c0aa6b0 0x7fc87c188450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:49370/0 (socket says 172.21.2.112:49370) 2021-02-10T11:32:58.685 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.695 7fc88288e700 1 -- 172.21.2.112:0/3377663565 learned_addr learned my addr 172.21.2.112:0/3377663565 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:32:58.685 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.695 7fc88288e700 1 -- 172.21.2.112:0/3377663565 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc87c189ca0 msgr2=0x7fc87c0aa090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:58.685 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.695 7fc88288e700 1 --2- 172.21.2.112:0/3377663565 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc87c189ca0 0x7fc87c0aa090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:58.685 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.695 7fc88288e700 1 -- 172.21.2.112:0/3377663565 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc87c1898a0 msgr2=0x7fc87c0a9a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:58.686 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.695 7fc88288e700 1 --2- 172.21.2.112:0/3377663565 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc87c1898a0 0x7fc87c0a9a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:58.686 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.695 7fc88288e700 1 -- 172.21.2.112:0/3377663565 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc87c188eb0 con 0x7fc87c0aa6b0 2021-02-10T11:32:58.686 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.695 7fc88288e700 1 --2- 172.21.2.112:0/3377663565 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc87c0aa6b0 0x7fc87c188450 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 rx=0x7fc87400b970 tx=0x7fc87400bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:58.686 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.695 7fc86effd700 1 -- 172.21.2.112:0/3377663565 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc874000f90 con 0x7fc87c0aa6b0 2021-02-10T11:32:58.686 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.695 7fc86effd700 1 -- 172.21.2.112:0/3377663565 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc874006aa0 con 0x7fc87c0aa6b0 2021-02-10T11:32:58.686 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.695 7fc86effd700 1 -- 172.21.2.112:0/3377663565 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc874001680 con 0x7fc87c0aa6b0 2021-02-10T11:32:58.686 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.695 7fc884271700 1 -- 172.21.2.112:0/3377663565 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc87c188eb0 con 0x7fc87c0aa6b0 2021-02-10T11:32:58.686 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.695 7fc884271700 1 -- 172.21.2.112:0/3377663565 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc87c189440 con 0x7fc87c0aa6b0 2021-02-10T11:32:58.686 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.696 7fc86effd700 1 -- 172.21.2.112:0/3377663565 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc8740011f0 con 0x7fc87c0aa6b0 2021-02-10T11:32:58.686 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.696 7fc86effd700 1 --2- 172.21.2.112:0/3377663565 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc8640186c0 0x7fc86401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:58.686 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.696 7fc86effd700 1 -- 172.21.2.112:0/3377663565 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fc8740328f0 con 0x7fc87c0aa6b0 2021-02-10T11:32:58.686 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.696 7fc88208d700 1 --2- 172.21.2.112:0/3377663565 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc8640186c0 0x7fc86401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:58.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.697 7fc88208d700 1 --2- 172.21.2.112:0/3377663565 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc8640186c0 0x7fc86401abc0 crc :-1 s=READY pgs=593 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:58.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.726 7fc884271700 1 -- 172.21.2.112:0/3377663565 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc87c07e0f0 con 0x7fc87c0aa6b0 2021-02-10T11:32:58.716 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.729 7fc86effd700 1 -- 172.21.2.112:0/3377663565 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc874000e30 con 0x7fc87c0aa6b0 2021-02-10T11:32:58.841 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.854 7fc884271700 1 -- 172.21.2.112:0/3377663565 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume authorize", "vol_name": "cephfs", "sub_name": "subvolume_0000000000592966", "target": ["mgr", ""], "tenant_id": "tenant1", "auth_id": "guest1", "group_name": "group_0000000001022050"}) v1 -- 0x7fc87c189440 con 0x7fc8640186c0 2021-02-10T11:32:58.847 INFO:tasks.ceph.mgr.x.gibba020.stderr:2021-02-10 11:32:58.864 7f6382eee700 -1 mgr.server reply reply (1) Operation not permitted auth ID: guest1 exists and not created by mgr plugin. Not allowed to modify 2021-02-10T11:32:58.847 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.860 7fc86effd700 1 -- 172.21.2.112:0/3377663565 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: -1 auth ID: guest1 exists and not created by mgr plugin. Not allowed to modify) v1 ==== 83+0+0 (crc 0 0 0) 0x7fc870000940 con 0x7fc8640186c0 2021-02-10T11:32:58.848 INFO:teuthology.orchestra.run.gibba012.stderr:Error EPERM: auth ID: guest1 exists and not created by mgr plugin. Not allowed to modify 2021-02-10T11:32:58.850 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.863 7fc884271700 1 -- 172.21.2.112:0/3377663565 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc8640186c0 msgr2=0x7fc86401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:58.850 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.863 7fc884271700 1 --2- 172.21.2.112:0/3377663565 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc8640186c0 0x7fc86401abc0 crc :-1 s=READY pgs=593 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:58.851 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.863 7fc884271700 1 -- 172.21.2.112:0/3377663565 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc87c0aa6b0 msgr2=0x7fc87c188450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:58.851 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.863 7fc884271700 1 --2- 172.21.2.112:0/3377663565 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc87c0aa6b0 0x7fc87c188450 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 rx=0x7fc87400b970 tx=0x7fc87400bd10).stop 2021-02-10T11:32:58.851 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.863 7fc884271700 1 -- 172.21.2.112:0/3377663565 shutdown_connections 2021-02-10T11:32:58.851 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.863 7fc884271700 1 --2- 172.21.2.112:0/3377663565 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc8640186c0 0x7fc86401abc0 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:58.851 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.863 7fc884271700 1 --2- 172.21.2.112:0/3377663565 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc87c0aa6b0 0x7fc87c188450 unknown :-1 s=CLOSED pgs=1181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:58.851 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.863 7fc884271700 1 --2- 172.21.2.112:0/3377663565 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc87c189ca0 0x7fc87c0aa090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:58.851 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.863 7fc884271700 1 --2- 172.21.2.112:0/3377663565 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc87c1898a0 0x7fc87c0a9a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:58.851 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.863 7fc884271700 1 -- 172.21.2.112:0/3377663565 >> 172.21.2.112:0/3377663565 conn(0x7fc87c19e810 msgr2=0x7fc87c07e630 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:58.851 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.863 7fc884271700 1 -- 172.21.2.112:0/3377663565 shutdown_connections 2021-02-10T11:32:58.851 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:58.864 7fc884271700 1 -- 172.21.2.112:0/3377663565 wait complete. 2021-02-10T11:32:58.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:32:58.859 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth rm client.guest1 2021-02-10T11:32:59.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.281 7fe696c3d700 1 Processor -- start 2021-02-10T11:32:59.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.281 7fe696c3d700 1 -- start start 2021-02-10T11:32:59.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.281 7fe696c3d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe6900e5070 0x7fe6901b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:59.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.281 7fe696c3d700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe69007ed80 con 0x7fe6901b3e40 2021-02-10T11:32:59.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.281 7fe696c3d700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe69007e670 con 0x7fe6900e5470 2021-02-10T11:32:59.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.281 7fe696c3d700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe69007e940 con 0x7fe6900e5070 2021-02-10T11:32:59.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.282 7fe69525a700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fe6901b3e40 0x7fe69007de80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:48840/0 (socket says 172.21.2.112:48840) 2021-02-10T11:32:59.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.282 7fe69525a700 1 -- 172.21.2.112:0/1166268832 learned_addr learned my addr 172.21.2.112:0/1166268832 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:32:59.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.282 7fe694a59700 1 --2- 172.21.2.112:0/1166268832 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe6900e5070 0x7fe6901b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:59.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.282 7fe68f7fe700 1 -- 172.21.2.112:0/1166268832 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1902378052 0 0) 0x7fe688001500 con 0x7fe6901b3e40 2021-02-10T11:32:59.287 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.282 7fe68f7fe700 1 -- 172.21.2.112:0/1166268832 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe67c002fe0 con 0x7fe6901b3e40 2021-02-10T11:32:59.287 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.282 7fe68f7fe700 1 -- 172.21.2.112:0/1166268832 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1002943849 0 0) 0x7fe688001500 con 0x7fe6901b3e40 2021-02-10T11:32:59.287 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.282 7fe68f7fe700 1 -- 172.21.2.112:0/1166268832 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe6900e5070 msgr2=0x7fe6901b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:59.287 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.282 7fe68f7fe700 1 --2- 172.21.2.112:0/1166268832 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe6900e5070 0x7fe6901b3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:59.287 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.282 7fe68f7fe700 1 -- 172.21.2.112:0/1166268832 >> v1:172.21.2.120:6789/0 conn(0x7fe6900e5470 legacy=0x7fe69007d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:59.287 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.282 7fe68f7fe700 1 -- 172.21.2.112:0/1166268832 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe69007eb30 con 0x7fe6901b3e40 2021-02-10T11:32:59.287 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.282 7fe68f7fe700 1 -- 172.21.2.112:0/1166268832 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7fe688001a00 con 0x7fe6901b3e40 2021-02-10T11:32:59.287 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.282 7fe68f7fe700 1 -- 172.21.2.112:0/1166268832 >> v1:172.21.2.112:6789/0 conn(0x7fe6901b3e40 legacy=0x7fe69007de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:59.287 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.282 7fe68f7fe700 1 --2- 172.21.2.112:0/1166268832 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe67c003a60 0x7fe67c005e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:59.287 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.282 7fe68f7fe700 1 --2- 172.21.2.112:0/1166268832 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe67c006730 0x7fe67c008c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:59.287 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.282 7fe68f7fe700 1 --2- 172.21.2.112:0/1166268832 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe67c0094f0 0x7fe67c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:59.287 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.282 7fe68f7fe700 1 -- 172.21.2.112:0/1166268832 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe67c00c2c0 con 0x7fe67c006730 2021-02-10T11:32:59.287 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.282 7fe68f7fe700 1 -- 172.21.2.112:0/1166268832 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe67c00c4e0 con 0x7fe67c0094f0 2021-02-10T11:32:59.287 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.283 7fe68f7fe700 1 -- 172.21.2.112:0/1166268832 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe67c050710 con 0x7fe67c003a60 2021-02-10T11:32:59.288 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.283 7fe68ffff700 1 --2- 172.21.2.112:0/1166268832 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe67c006730 0x7fe67c008c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:59.288 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.283 7fe694a59700 1 --2- 172.21.2.112:0/1166268832 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe67c003a60 0x7fe67c005e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:59.288 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.283 7fe68ffff700 1 -- 172.21.2.112:0/1166268832 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe67c003a60 msgr2=0x7fe67c005e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:59.288 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.283 7fe69525a700 1 --2- 172.21.2.112:0/1166268832 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe67c0094f0 0x7fe67c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:59.288 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.283 7fe68ffff700 1 --2- 172.21.2.112:0/1166268832 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe67c003a60 0x7fe67c005e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:59.288 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.283 7fe68ffff700 1 -- 172.21.2.112:0/1166268832 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe67c0094f0 msgr2=0x7fe67c00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:59.288 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.283 7fe68ffff700 1 --2- 172.21.2.112:0/1166268832 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe67c0094f0 0x7fe67c00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:59.288 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.283 7fe68ffff700 1 -- 172.21.2.112:0/1166268832 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe67c050940 con 0x7fe67c006730 2021-02-10T11:32:59.288 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.283 7fe69525a700 1 --2- 172.21.2.112:0/1166268832 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe67c0094f0 0x7fe67c00ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:32:59.288 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.283 7fe68ffff700 1 --2- 172.21.2.112:0/1166268832 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe67c006730 0x7fe67c008c50 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 rx=0x7fe680006740 tx=0x7fe680009b00).ready entity=mon.0 client_cookie=31e4401038c3acbf server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:59.288 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.283 7fe68f7fe700 1 -- 172.21.2.112:0/1166268832 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe680000c10 con 0x7fe67c006730 2021-02-10T11:32:59.288 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.283 7fe68f7fe700 1 -- 172.21.2.112:0/1166268832 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe680001290 con 0x7fe67c006730 2021-02-10T11:32:59.288 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.283 7fe68f7fe700 1 -- 172.21.2.112:0/1166268832 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe680001770 con 0x7fe67c006730 2021-02-10T11:32:59.289 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.283 7fe696c3d700 1 -- 172.21.2.112:0/1166268832 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe67c006730 msgr2=0x7fe67c008c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:59.289 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.283 7fe696c3d700 1 --2- 172.21.2.112:0/1166268832 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe67c006730 0x7fe67c008c50 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 rx=0x7fe680006740 tx=0x7fe680009b00).stop 2021-02-10T11:32:59.289 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.283 7fe69525a700 1 -- 172.21.2.112:0/1166268832 reap_dead start 2021-02-10T11:32:59.289 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.284 7fe696c3d700 1 -- 172.21.2.112:0/1166268832 shutdown_connections 2021-02-10T11:32:59.289 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.284 7fe696c3d700 1 -- 172.21.2.112:0/1166268832 >> 172.21.2.112:0/1166268832 conn(0x7fe6900ce690 msgr2=0x7fe6900e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:59.289 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.284 7fe696c3d700 1 -- 172.21.2.112:0/1166268832 shutdown_connections 2021-02-10T11:32:59.289 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.284 7fe696c3d700 1 -- 172.21.2.112:0/1166268832 wait complete. 2021-02-10T11:32:59.291 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.299 7fe696c3d700 1 Processor -- start 2021-02-10T11:32:59.291 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.299 7fe696c3d700 1 -- start start 2021-02-10T11:32:59.291 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.299 7fe696c3d700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe6900b4ad0 0x7fe6900a78d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:59.291 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.299 7fe696c3d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe6900a7ef0 0x7fe6901a6d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:59.291 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.299 7fe696c3d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe6900a82f0 0x7fe6901a73b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:59.291 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.299 7fe696c3d700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6900aa3c0 con 0x7fe6900a82f0 2021-02-10T11:32:59.291 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.299 7fe696c3d700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6900a88c0 con 0x7fe6900b4ad0 2021-02-10T11:32:59.291 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.299 7fe696c3d700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6900b4ed0 con 0x7fe6900a7ef0 2021-02-10T11:32:59.291 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.300 7fe69525a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe6900a82f0 0x7fe6901a73b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:59.291 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.300 7fe69525a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe6900a82f0 0x7fe6901a73b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:49390/0 (socket says 172.21.2.112:49390) 2021-02-10T11:32:59.291 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.300 7fe69525a700 1 -- 172.21.2.112:0/1747578725 learned_addr learned my addr 172.21.2.112:0/1747578725 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:32:59.292 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.300 7fe694a59700 1 --2- 172.21.2.112:0/1747578725 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe6900b4ad0 0x7fe6900a78d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:59.292 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.300 7fe68ffff700 1 --2- 172.21.2.112:0/1747578725 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe6900a7ef0 0x7fe6901a6d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:59.292 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.300 7fe69525a700 1 -- 172.21.2.112:0/1747578725 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe6900a7ef0 msgr2=0x7fe6901a6d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:59.292 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.300 7fe69525a700 1 --2- 172.21.2.112:0/1747578725 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe6900a7ef0 0x7fe6901a6d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:59.292 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.300 7fe69525a700 1 -- 172.21.2.112:0/1747578725 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe6900b4ad0 msgr2=0x7fe6900a78d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:59.292 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.300 7fe69525a700 1 --2- 172.21.2.112:0/1747578725 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe6900b4ad0 0x7fe6900a78d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:59.292 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.300 7fe69525a700 1 -- 172.21.2.112:0/1747578725 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6901a79d0 con 0x7fe6900a82f0 2021-02-10T11:32:59.292 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.300 7fe69525a700 1 --2- 172.21.2.112:0/1747578725 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe6900a82f0 0x7fe6901a73b0 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 rx=0x7fe68800a900 tx=0x7fe68800aca0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:59.292 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.301 7fe68d7fa700 1 -- 172.21.2.112:0/1747578725 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe6880070d0 con 0x7fe6900a82f0 2021-02-10T11:32:59.292 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.301 7fe696c3d700 1 -- 172.21.2.112:0/1747578725 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6901a79d0 con 0x7fe6900a82f0 2021-02-10T11:32:59.292 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.301 7fe68d7fa700 1 -- 172.21.2.112:0/1747578725 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6880070d0 con 0x7fe6900a82f0 2021-02-10T11:32:59.292 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.301 7fe68d7fa700 1 -- 172.21.2.112:0/1747578725 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe688007390 con 0x7fe6900a82f0 2021-02-10T11:32:59.292 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.301 7fe696c3d700 1 -- 172.21.2.112:0/1747578725 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6901a79d0 con 0x7fe6900a82f0 2021-02-10T11:32:59.292 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.301 7fe68d7fa700 1 -- 172.21.2.112:0/1747578725 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe688007350 con 0x7fe6900a82f0 2021-02-10T11:32:59.293 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.301 7fe68d7fa700 1 --2- 172.21.2.112:0/1747578725 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe6740186c0 0x7fe67401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:59.293 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.301 7fe68d7fa700 1 -- 172.21.2.112:0/1747578725 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fe6880328d0 con 0x7fe6900a82f0 2021-02-10T11:32:59.293 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.302 7fe694a59700 1 --2- 172.21.2.112:0/1747578725 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe6740186c0 0x7fe67401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:59.293 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.302 7fe694a59700 1 --2- 172.21.2.112:0/1747578725 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe6740186c0 0x7fe67401abc0 crc :-1 s=READY pgs=594 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:59.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.331 7fe696c3d700 1 -- 172.21.2.112:0/1747578725 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe690084a90 con 0x7fe6900a82f0 2021-02-10T11:32:59.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.334 7fe68d7fa700 1 -- 172.21.2.112:0/1747578725 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe68802cf20 con 0x7fe6900a82f0 2021-02-10T11:32:59.451 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.464 7fe696c3d700 1 -- 172.21.2.112:0/1747578725 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "auth rm", "entity": "client.guest1"} v 0) v1 -- 0x7fe69007ed80 con 0x7fe6900a82f0 2021-02-10T11:32:59.455 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.468 7fe68d7fa700 1 -- 172.21.2.112:0/1747578725 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth rm", "entity": "client.guest1"}]=0 updated v30) v1 ==== 89+0+0 (secure 0 0 0) 0x7fe68802cf20 con 0x7fe6900a82f0 2021-02-10T11:32:59.458 INFO:teuthology.orchestra.run.gibba012.stderr:updated 2021-02-10T11:32:59.460 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.473 7fe696c3d700 1 -- 172.21.2.112:0/1747578725 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe6740186c0 msgr2=0x7fe67401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:59.460 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.473 7fe696c3d700 1 --2- 172.21.2.112:0/1747578725 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe6740186c0 0x7fe67401abc0 crc :-1 s=READY pgs=594 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:59.460 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.473 7fe696c3d700 1 -- 172.21.2.112:0/1747578725 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe6900a82f0 msgr2=0x7fe6901a73b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:59.460 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.473 7fe696c3d700 1 --2- 172.21.2.112:0/1747578725 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe6900a82f0 0x7fe6901a73b0 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 rx=0x7fe68800a900 tx=0x7fe68800aca0).stop 2021-02-10T11:32:59.461 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.473 7fe696c3d700 1 -- 172.21.2.112:0/1747578725 shutdown_connections 2021-02-10T11:32:59.461 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.473 7fe696c3d700 1 --2- 172.21.2.112:0/1747578725 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe6740186c0 0x7fe67401abc0 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:59.461 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.473 7fe696c3d700 1 --2- 172.21.2.112:0/1747578725 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe6900a82f0 0x7fe6901a73b0 unknown :-1 s=CLOSED pgs=1184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:59.461 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.473 7fe696c3d700 1 --2- 172.21.2.112:0/1747578725 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe6900a7ef0 0x7fe6901a6d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:59.461 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.473 7fe696c3d700 1 --2- 172.21.2.112:0/1747578725 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe6900b4ad0 0x7fe6900a78d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:59.461 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.473 7fe696c3d700 1 -- 172.21.2.112:0/1747578725 >> 172.21.2.112:0/1747578725 conn(0x7fe6900b5560 msgr2=0x7fe6900a6200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:59.461 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.474 7fe696c3d700 1 -- 172.21.2.112:0/1747578725 shutdown_connections 2021-02-10T11:32:59.461 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.474 7fe696c3d700 1 -- 172.21.2.112:0/1747578725 wait complete. 2021-02-10T11:32:59.469 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000592966 --group_name group_0000000001022050 2021-02-10T11:32:59.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.895 7faf927b5700 1 Processor -- start 2021-02-10T11:32:59.900 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.895 7faf927b5700 1 -- start start 2021-02-10T11:32:59.900 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.896 7faf927b5700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7faf8c0d1860 0x7faf8c1b03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:59.900 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.896 7faf927b5700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7faf8c0f0640 0x7faf8c0cfb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:59.900 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.896 7faf927b5700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf8c0f0b00 con 0x7faf8c1b0a00 2021-02-10T11:32:59.900 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.896 7faf927b5700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf8c0e5ec0 con 0x7faf8c0d1860 2021-02-10T11:32:59.900 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.896 7faf927b5700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf8c0d1c60 con 0x7faf8c0f0640 2021-02-10T11:32:59.900 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.896 7faf8bfff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7faf8c0d1860 0x7faf8c1b03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:59.900 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.896 7faf8bfff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7faf8c0d1860 0x7faf8c1b03e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:49394/0 (socket says 172.21.2.112:49394) 2021-02-10T11:32:59.900 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.896 7faf8bfff700 1 -- 172.21.2.112:0/822103817 learned_addr learned my addr 172.21.2.112:0/822103817 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:32:59.900 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.896 7faf8bfff700 1 -- 172.21.2.112:0/822103817 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7faf8c0f0640 msgr2=0x7faf8c0cfb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:59.900 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.896 7faf8bfff700 1 --2- 172.21.2.112:0/822103817 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7faf8c0f0640 0x7faf8c0cfb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:59.901 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.896 7faf8bfff700 1 -- 172.21.2.112:0/822103817 >> v1:172.21.2.120:6789/0 conn(0x7faf8c1b0a00 legacy=0x7faf8c0cf320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:59.901 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.896 7faf8bfff700 1 -- 172.21.2.112:0/822103817 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf8c0d1e80 con 0x7faf8c0d1860 2021-02-10T11:32:59.901 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.897 7faf8bfff700 1 --2- 172.21.2.112:0/822103817 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faf8c0d1860 0x7faf8c1b03e0 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 rx=0x7faf7c006700 tx=0x7faf7c009b00).ready entity=mon.0 client_cookie=5f42f593cd072118 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:59.901 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.897 7faf8affd700 1 -- 172.21.2.112:0/822103817 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faf7c00c0c0 con 0x7faf8c0d1860 2021-02-10T11:32:59.901 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.897 7faf8affd700 1 -- 172.21.2.112:0/822103817 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf7c00c400 con 0x7faf8c0d1860 2021-02-10T11:32:59.901 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.897 7faf8affd700 1 -- 172.21.2.112:0/822103817 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faf7c00c920 con 0x7faf8c0d1860 2021-02-10T11:32:59.901 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.897 7faf927b5700 1 -- 172.21.2.112:0/822103817 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faf8c0d1860 msgr2=0x7faf8c1b03e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:59.901 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.897 7faf927b5700 1 --2- 172.21.2.112:0/822103817 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faf8c0d1860 0x7faf8c1b03e0 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 rx=0x7faf7c006700 tx=0x7faf7c009b00).stop 2021-02-10T11:32:59.901 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.897 7faf927b5700 1 -- 172.21.2.112:0/822103817 shutdown_connections 2021-02-10T11:32:59.901 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.897 7faf927b5700 1 --2- 172.21.2.112:0/822103817 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7faf8c0f0640 0x7faf8c0cfb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:59.901 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.897 7faf927b5700 1 --2- 172.21.2.112:0/822103817 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faf8c0d1860 0x7faf8c1b03e0 unknown :-1 s=CLOSED pgs=1185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:59.901 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.897 7faf927b5700 1 -- 172.21.2.112:0/822103817 >> 172.21.2.112:0/822103817 conn(0x7faf8c0ce440 msgr2=0x7faf8c0d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:59.901 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.898 7faf927b5700 1 -- 172.21.2.112:0/822103817 shutdown_connections 2021-02-10T11:32:59.901 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.898 7faf927b5700 1 -- 172.21.2.112:0/822103817 wait complete. 2021-02-10T11:32:59.902 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.912 7faf927b5700 1 Processor -- start 2021-02-10T11:32:59.902 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.912 7faf927b5700 1 -- start start 2021-02-10T11:32:59.902 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.912 7faf927b5700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faf8c09f190 0x7faf8c09e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:59.902 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.913 7faf927b5700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7faf8c097240 0x7faf8c097640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:59.902 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.913 7faf927b5700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7faf8c098310 0x7faf8c098710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:59.902 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.913 7faf927b5700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf8c097d90 con 0x7faf8c09f190 2021-02-10T11:32:59.902 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.913 7faf927b5700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf8c09ec40 con 0x7faf8c098310 2021-02-10T11:32:59.902 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.913 7faf927b5700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf8c09ee20 con 0x7faf8c097240 2021-02-10T11:32:59.903 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.913 7faf8bfff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faf8c09f190 0x7faf8c09e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:59.903 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.913 7faf8bfff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faf8c09f190 0x7faf8c09e620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:49400/0 (socket says 172.21.2.112:49400) 2021-02-10T11:32:59.903 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.913 7faf8bfff700 1 -- 172.21.2.112:0/2126756559 learned_addr learned my addr 172.21.2.112:0/2126756559 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:32:59.903 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.913 7faf8bfff700 1 -- 172.21.2.112:0/2126756559 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7faf8c097240 msgr2=0x7faf8c097640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:59.904 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.913 7faf8bfff700 1 --2- 172.21.2.112:0/2126756559 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7faf8c097240 0x7faf8c097640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:59.904 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.913 7faf8bfff700 1 -- 172.21.2.112:0/2126756559 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7faf8c098310 msgr2=0x7faf8c098710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:59.904 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.913 7faf8bfff700 1 --2- 172.21.2.112:0/2126756559 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7faf8c098310 0x7faf8c098710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:59.904 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.913 7faf8bfff700 1 -- 172.21.2.112:0/2126756559 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf8c098d30 con 0x7faf8c09f190 2021-02-10T11:32:59.904 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.913 7faf8bfff700 1 --2- 172.21.2.112:0/2126756559 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faf8c09f190 0x7faf8c09e620 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 rx=0x7faf7c00b660 tx=0x7faf7c00ba00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:59.904 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.913 7faf88ff9700 1 -- 172.21.2.112:0/2126756559 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faf7c009440 con 0x7faf8c09f190 2021-02-10T11:32:59.904 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.914 7faf88ff9700 1 -- 172.21.2.112:0/2126756559 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf7c000c90 con 0x7faf8c09f190 2021-02-10T11:32:59.904 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.914 7faf88ff9700 1 -- 172.21.2.112:0/2126756559 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faf7c002030 con 0x7faf8c09f190 2021-02-10T11:32:59.904 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.914 7faf927b5700 1 -- 172.21.2.112:0/2126756559 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf8c098d30 con 0x7faf8c09f190 2021-02-10T11:32:59.904 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.914 7faf927b5700 1 -- 172.21.2.112:0/2126756559 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf8c0992c0 con 0x7faf8c09f190 2021-02-10T11:32:59.904 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.914 7faf88ff9700 1 -- 172.21.2.112:0/2126756559 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7faf7c0093e0 con 0x7faf8c09f190 2021-02-10T11:32:59.904 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.914 7faf88ff9700 1 --2- 172.21.2.112:0/2126756559 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7faf6c0186c0 0x7faf6c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:59.904 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.914 7faf88ff9700 1 -- 172.21.2.112:0/2126756559 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5573+0+0 (secure 0 0 0) 0x7faf7c032580 con 0x7faf8c09f190 2021-02-10T11:32:59.904 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.915 7faf8b7fe700 1 --2- 172.21.2.112:0/2126756559 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7faf6c0186c0 0x7faf6c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:59.905 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.915 7faf8b7fe700 1 --2- 172.21.2.112:0/2126756559 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7faf6c0186c0 0x7faf6c01abc0 crc :-1 s=READY pgs=595 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:59.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.946 7faf927b5700 1 -- 172.21.2.112:0/2126756559 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf8c0ad800 con 0x7faf8c09f190 2021-02-10T11:32:59.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:32:59.949 7faf88ff9700 1 -- 172.21.2.112:0/2126756559 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7faf7c030d30 con 0x7faf8c09f190 2021-02-10T11:33:00.063 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:00.076 7faf927b5700 1 -- 172.21.2.112:0/2126756559 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000592966", "target": ["mgr", ""], "group_name": "group_0000000001022050"}) v1 -- 0x7faf8c097d90 con 0x7faf6c0186c0 2021-02-10T11:33:00.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:00.084 7faf88ff9700 1 -- 172.21.2.112:0/2126756559 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7faf800009b0 con 0x7faf6c0186c0 2021-02-10T11:33:00.081 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:00.094 7faf927b5700 1 -- 172.21.2.112:0/2126756559 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7faf6c0186c0 msgr2=0x7faf6c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:00.081 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:00.094 7faf927b5700 1 --2- 172.21.2.112:0/2126756559 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7faf6c0186c0 0x7faf6c01abc0 crc :-1 s=READY pgs=595 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:00.081 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:00.094 7faf927b5700 1 -- 172.21.2.112:0/2126756559 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faf8c09f190 msgr2=0x7faf8c09e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:00.081 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:00.094 7faf927b5700 1 --2- 172.21.2.112:0/2126756559 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faf8c09f190 0x7faf8c09e620 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 rx=0x7faf7c00b660 tx=0x7faf7c00ba00).stop 2021-02-10T11:33:00.081 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:00.094 7faf927b5700 1 -- 172.21.2.112:0/2126756559 shutdown_connections 2021-02-10T11:33:00.081 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:00.094 7faf927b5700 1 --2- 172.21.2.112:0/2126756559 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7faf6c0186c0 0x7faf6c01abc0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:00.082 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:00.094 7faf927b5700 1 --2- 172.21.2.112:0/2126756559 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7faf8c098310 0x7faf8c098710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:00.082 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:00.094 7faf927b5700 1 --2- 172.21.2.112:0/2126756559 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7faf8c097240 0x7faf8c097640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:00.082 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:00.094 7faf927b5700 1 --2- 172.21.2.112:0/2126756559 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faf8c09f190 0x7faf8c09e620 unknown :-1 s=CLOSED pgs=1186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:00.082 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:00.094 7faf927b5700 1 -- 172.21.2.112:0/2126756559 >> 172.21.2.112:0/2126756559 conn(0x7faf8c19f890 msgr2=0x7faf8c1a6100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:00.082 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:00.094 7faf927b5700 1 -- 172.21.2.112:0/2126756559 shutdown_connections 2021-02-10T11:33:00.082 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:00.094 7faf927b5700 1 -- 172.21.2.112:0/2126756559 wait complete. 2021-02-10T11:33:00.089 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolumegroup rm cephfs group_0000000001022050 2021-02-10T11:33:00.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:00.517 7f603f5e2700 1 Processor -- start 2021-02-10T11:33:00.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:00.517 7f603f5e2700 1 -- start start 2021-02-10T11:33:00.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:00.517 7f603f5e2700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f60381b0470 0x7f60381b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:00.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:00.517 7f603f5e2700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f60380f20e0 0x7f60380f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:00.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:00.517 7f603f5e2700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60380f0b50 con 0x7f60380ef610 2021-02-10T11:33:00.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:00.517 7f603f5e2700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60380f0820 con 0x7f60380f20e0 2021-02-10T11:33:00.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:00.518 7f603f5e2700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60380f24e0 con 0x7f60381b0470 2021-02-10T11:33:00.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:00.518 7f603dbff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f60380f20e0 0x7f60380f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:00.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:00.518 7f603dbff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f60380f20e0 0x7f60380f0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:49408/0 (socket says 172.21.2.112:49408) 2021-02-10T11:33:00.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:00.518 7f603dbff700 1 -- 172.21.2.112:0/4060189863 learned_addr learned my addr 172.21.2.112:0/4060189863 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:33:00.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:00.518 7f603d3fe700 1 --2- 172.21.2.112:0/4060189863 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f60381b0470 0x7f60381b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:00.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:00.518 7f603dbff700 1 -- 172.21.2.112:0/4060189863 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f60381b0470 msgr2=0x7f60381b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:00.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:00.518 7f603dbff700 1 --2- 172.21.2.112:0/4060189863 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f60381b0470 0x7f60381b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:00.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:00.518 7f603dbff700 1 -- 172.21.2.112:0/4060189863 >> v1:172.21.2.120:6789/0 conn(0x7f60380ef610 legacy=0x7f60380efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:00.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:00.518 7f603dbff700 1 -- 172.21.2.112:0/4060189863 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60380e52e0 con 0x7f60380f20e0 2021-02-10T11:33:00.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:00.518 7f603dbff700 1 --2- 172.21.2.112:0/4060189863 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f60380f20e0 0x7f60380f0200 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 rx=0x7f6030006f00 tx=0x7f6030009f10).ready entity=mon.0 client_cookie=3fed18f4ce4f5ab1 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:00.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:00.518 7f602bfff700 1 -- 172.21.2.112:0/4060189863 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f603000b3b0 con 0x7f60380f20e0 2021-02-10T11:33:00.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:00.519 7f602bfff700 1 -- 172.21.2.112:0/4060189863 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f603000b630 con 0x7f60380f20e0 2021-02-10T11:33:00.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:00.519 7f602bfff700 1 -- 172.21.2.112:0/4060189863 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f603000bb50 con 0x7f60380f20e0 2021-02-10T11:33:00.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:00.519 7f603f5e2700 1 -- 172.21.2.112:0/4060189863 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f60380f20e0 msgr2=0x7f60380f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:00.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:00.519 7f603f5e2700 1 --2- 172.21.2.112:0/4060189863 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f60380f20e0 0x7f60380f0200 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 rx=0x7f6030006f00 tx=0x7f6030009f10).stop 2021-02-10T11:33:00.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:00.519 7f603f5e2700 1 -- 172.21.2.112:0/4060189863 shutdown_connections 2021-02-10T11:33:00.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:00.519 7f603f5e2700 1 --2- 172.21.2.112:0/4060189863 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f60380f20e0 0x7f60380f0200 unknown :-1 s=CLOSED pgs=1187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:00.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:00.519 7f603f5e2700 1 --2- 172.21.2.112:0/4060189863 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f60381b0470 0x7f60381b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:00.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:00.519 7f603f5e2700 1 -- 172.21.2.112:0/4060189863 >> 172.21.2.112:0/4060189863 conn(0x7f60380ce3e0 msgr2=0x7f60380e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:00.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:00.519 7f603f5e2700 1 -- 172.21.2.112:0/4060189863 shutdown_connections 2021-02-10T11:33:00.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:00.519 7f603f5e2700 1 -- 172.21.2.112:0/4060189863 wait complete. 2021-02-10T11:33:00.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:00.534 7f603f5e2700 1 Processor -- start 2021-02-10T11:33:00.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:00.534 7f603f5e2700 1 -- start start 2021-02-10T11:33:00.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:00.535 7f603f5e2700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f60380b7c20 0x7f60380b8410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:00.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:00.535 7f603f5e2700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f60380b8a30 0x7f60380c1f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:00.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:00.535 7f603f5e2700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f60380c25a0 0x7f60380c29a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:00.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:00.535 7f603f5e2700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60380c1a80 con 0x7f60380b8a30 2021-02-10T11:33:00.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:00.535 7f603f5e2700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60380b8020 con 0x7f60380b7c20 2021-02-10T11:33:00.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:00.535 7f603f5e2700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f603809b9c0 con 0x7f60380c25a0 2021-02-10T11:33:00.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:00.535 7f603cbfd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f60380b8a30 0x7f60380c1f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:00.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:00.535 7f603cbfd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f60380b8a30 0x7f60380c1f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:49414/0 (socket says 172.21.2.112:49414) 2021-02-10T11:33:00.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:00.535 7f603cbfd700 1 -- 172.21.2.112:0/2777693387 learned_addr learned my addr 172.21.2.112:0/2777693387 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:33:00.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:00.535 7f603d3fe700 1 --2- 172.21.2.112:0/2777693387 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f60380b7c20 0x7f60380b8410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:00.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:00.535 7f603dbff700 1 --2- 172.21.2.112:0/2777693387 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f60380c25a0 0x7f60380c29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:00.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:00.536 7f603d3fe700 1 -- 172.21.2.112:0/2777693387 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f60380c25a0 msgr2=0x7f60380c29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:00.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:00.536 7f603d3fe700 1 --2- 172.21.2.112:0/2777693387 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f60380c25a0 0x7f60380c29a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:00.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:00.536 7f603d3fe700 1 -- 172.21.2.112:0/2777693387 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f60380b8a30 msgr2=0x7f60380c1f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:00.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:00.536 7f603d3fe700 1 --2- 172.21.2.112:0/2777693387 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f60380b8a30 0x7f60380c1f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:00.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:00.536 7f603d3fe700 1 -- 172.21.2.112:0/2777693387 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60380b8e30 con 0x7f60380b7c20 2021-02-10T11:33:00.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:00.536 7f603cbfd700 1 --2- 172.21.2.112:0/2777693387 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f60380b8a30 0x7f60380c1f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:33:00.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:00.536 7f603d3fe700 1 --2- 172.21.2.112:0/2777693387 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f60380b7c20 0x7f60380b8410 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 rx=0x7f602c00b4e0 tx=0x7f602c00b880).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:00.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:00.536 7f6029ffb700 1 -- 172.21.2.112:0/2777693387 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f602c000c90 con 0x7f60380b7c20 2021-02-10T11:33:00.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:00.536 7f6029ffb700 1 -- 172.21.2.112:0/2777693387 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f602c006400 con 0x7f60380b7c20 2021-02-10T11:33:00.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:00.536 7f6029ffb700 1 -- 172.21.2.112:0/2777693387 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f602c002030 con 0x7f60380b7c20 2021-02-10T11:33:00.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:00.536 7f603f5e2700 1 -- 172.21.2.112:0/2777693387 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60380b8e30 con 0x7f60380b7c20 2021-02-10T11:33:00.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:00.536 7f603f5e2700 1 -- 172.21.2.112:0/2777693387 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60380b8e30 con 0x7f60380b7c20 2021-02-10T11:33:00.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:00.536 7f6029ffb700 1 -- 172.21.2.112:0/2777693387 <== mon.1 v2:172.21.2.120:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f602c000c90 con 0x7f60380b7c20 2021-02-10T11:33:00.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:00.536 7f6029ffb700 1 --2- 172.21.2.112:0/2777693387 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f60180186d0 0x7f601801abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:00.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:00.536 7f6029ffb700 1 -- 172.21.2.112:0/2777693387 <== mon.1 v2:172.21.2.120:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f602c0326f0 con 0x7f60380b7c20 2021-02-10T11:33:00.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:00.537 7f603cbfd700 1 --2- 172.21.2.112:0/2777693387 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f60180186d0 0x7f601801abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:00.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:00.537 7f603cbfd700 1 --2- 172.21.2.112:0/2777693387 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f60180186d0 0x7f601801abd0 crc :-1 s=READY pgs=596 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:00.554 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:00.567 7f603f5e2700 1 -- 172.21.2.112:0/2777693387 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6038105b30 con 0x7f60380b7c20 2021-02-10T11:33:00.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:00.570 7f6029ffb700 1 -- 172.21.2.112:0/2777693387 <== mon.1 v2:172.21.2.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f602c006ae0 con 0x7f60380b7c20 2021-02-10T11:33:00.684 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:00.697 7f603f5e2700 1 -- 172.21.2.112:0/2777693387 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolumegroup rm", "vol_name": "cephfs", "target": ["mgr", ""], "group_name": "group_0000000001022050"}) v1 -- 0x7f6038105b30 con 0x7f60180186d0 2021-02-10T11:33:00.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:00.702 7f6029ffb700 1 -- 172.21.2.112:0/2777693387 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f6034007930 con 0x7f60180186d0 2021-02-10T11:33:00.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:00.706 7f603f5e2700 1 -- 172.21.2.112:0/2777693387 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f60180186d0 msgr2=0x7f601801abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:00.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:00.706 7f603f5e2700 1 --2- 172.21.2.112:0/2777693387 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f60180186d0 0x7f601801abd0 crc :-1 s=READY pgs=596 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:00.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:00.706 7f603f5e2700 1 -- 172.21.2.112:0/2777693387 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f60380b7c20 msgr2=0x7f60380b8410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:00.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:00.707 7f603f5e2700 1 --2- 172.21.2.112:0/2777693387 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f60380b7c20 0x7f60380b8410 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 rx=0x7f602c00b4e0 tx=0x7f602c00b880).stop 2021-02-10T11:33:00.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:00.707 7f603f5e2700 1 -- 172.21.2.112:0/2777693387 shutdown_connections 2021-02-10T11:33:00.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:00.707 7f603f5e2700 1 --2- 172.21.2.112:0/2777693387 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f60180186d0 0x7f601801abd0 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:00.695 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:00.707 7f603f5e2700 1 --2- 172.21.2.112:0/2777693387 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f60380c25a0 0x7f60380c29a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:00.695 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:00.707 7f603f5e2700 1 --2- 172.21.2.112:0/2777693387 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f60380b8a30 0x7f60380c1f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:00.695 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:00.707 7f603f5e2700 1 --2- 172.21.2.112:0/2777693387 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f60380b7c20 0x7f60380b8410 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:00.695 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:00.707 7f603f5e2700 1 -- 172.21.2.112:0/2777693387 >> 172.21.2.112:0/2777693387 conn(0x7f6038094e10 msgr2=0x7f60380a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:00.695 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:00.707 7f603f5e2700 1 -- 172.21.2.112:0/2777693387 shutdown_connections 2021-02-10T11:33:00.695 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:00.707 7f603f5e2700 1 -- 172.21.2.112:0/2777693387 wait complete. 2021-02-10T11:33:00.702 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:33:00.702 DEBUG:teuthology.orchestra.run.gibba012:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:33:01.047 DEBUG:teuthology.orchestra.run.gibba020:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:33:01.391 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:33:01.391 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:01.391 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:33:01.391 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:33:01.726 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:33:01.726 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:33:01.726 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba012.front.sepia.ceph.com... 2021-02-10T11:33:01.726 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:33:01.727 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:33:02.076 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba012.stderr:ceph-fuse[125240]: fuse finished with error 0 and tester_r 0 2021-02-10T11:33:02.077 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:02.077 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:33:02.077 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:33:02.411 INFO:teuthology.orchestra.run.gibba012.stdout:ext2/ext3 2021-02-10T11:33:02.412 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:33:02.412 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:33:02.412 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:33:02.746 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:02.747 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:33:02.747 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:33:03.082 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:33:03.082 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:33:03.082 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba012.front.sepia.ceph.com... 2021-02-10T11:33:03.082 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:33:03.083 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:33:03.435 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba012.stderr:ceph-fuse[125909]: fuse finished with error 0 and tester_r 0 2021-02-10T11:33:03.437 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:03.437 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:33:03.437 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:33:03.772 INFO:teuthology.orchestra.run.gibba012.stdout:ext2/ext3 2021-02-10T11:33:03.773 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:33:03.773 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:33:03.773 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:33:04.108 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:04.108 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:33:04.108 DEBUG:teuthology.orchestra.run.gibba020:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:33:04.443 INFO:teuthology.orchestra.run.gibba020.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:33:04.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:33:04.443 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:33:04.444 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:33:04.444 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:33:04.779 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:04.779 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:33:04.779 DEBUG:teuthology.orchestra.run.gibba020:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:33:05.113 INFO:teuthology.orchestra.run.gibba020.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:33:05.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:33:05.114 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:33:05.114 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:33:05.114 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:33:05.449 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config rm mon mon_allow_pool_delete 2021-02-10T11:33:05.879 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:05.876 7fe750526700 1 Processor -- start 2021-02-10T11:33:05.880 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:05.876 7fe750526700 1 -- start start 2021-02-10T11:33:05.880 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:05.876 7fe750526700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe7480d17f0 0x7fe7481b31e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:05.880 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:05.876 7fe750526700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fe7480cff00 0x7fe7480ef620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:05.880 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:05.876 7fe750526700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7480f0b20 con 0x7fe7480d1bf0 2021-02-10T11:33:05.880 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:05.876 7fe750526700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7480e5eb0 con 0x7fe7480cff00 2021-02-10T11:33:05.880 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:05.876 7fe750526700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7481b3ff0 con 0x7fe7480d17f0 2021-02-10T11:33:05.880 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:05.877 7fe74eb43700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fe7480cff00 0x7fe7480ef620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:05.880 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:05.877 7fe74eb43700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fe7480cff00 0x7fe7480ef620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:49422/0 (socket says 172.21.2.112:49422) 2021-02-10T11:33:05.881 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:05.877 7fe74eb43700 1 -- 172.21.2.112:0/3640036800 learned_addr learned my addr 172.21.2.112:0/3640036800 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:33:05.881 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:05.877 7fe74e342700 1 --2- 172.21.2.112:0/3640036800 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe7480d17f0 0x7fe7481b31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:05.881 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:05.877 7fe74eb43700 1 -- 172.21.2.112:0/3640036800 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe7480d17f0 msgr2=0x7fe7481b31e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:05.881 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:05.877 7fe74eb43700 1 --2- 172.21.2.112:0/3640036800 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe7480d17f0 0x7fe7481b31e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:05.881 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:05.877 7fe74eb43700 1 -- 172.21.2.112:0/3640036800 >> v1:172.21.2.120:6789/0 conn(0x7fe7480d1bf0 legacy=0x7fe7481b3800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:05.881 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:05.877 7fe74eb43700 1 -- 172.21.2.112:0/3640036800 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7480f20b0 con 0x7fe7480cff00 2021-02-10T11:33:05.881 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:05.877 7fe74eb43700 1 --2- 172.21.2.112:0/3640036800 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe7480cff00 0x7fe7480ef620 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 rx=0x7fe740006f00 tx=0x7fe740009f10).ready entity=mon.0 client_cookie=9411f7397eb6468c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:05.881 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:05.877 7fe74d340700 1 -- 172.21.2.112:0/3640036800 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe74000b3b0 con 0x7fe7480cff00 2021-02-10T11:33:05.881 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:05.877 7fe74d340700 1 -- 172.21.2.112:0/3640036800 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe74000b730 con 0x7fe7480cff00 2021-02-10T11:33:05.881 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:05.877 7fe74d340700 1 -- 172.21.2.112:0/3640036800 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe74000bb90 con 0x7fe7480cff00 2021-02-10T11:33:05.881 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:05.877 7fe750526700 1 -- 172.21.2.112:0/3640036800 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe7480cff00 msgr2=0x7fe7480ef620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:05.882 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:05.877 7fe750526700 1 --2- 172.21.2.112:0/3640036800 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe7480cff00 0x7fe7480ef620 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 rx=0x7fe740006f00 tx=0x7fe740009f10).stop 2021-02-10T11:33:05.882 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:05.878 7fe750526700 1 -- 172.21.2.112:0/3640036800 shutdown_connections 2021-02-10T11:33:05.882 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:05.878 7fe750526700 1 --2- 172.21.2.112:0/3640036800 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe7480cff00 0x7fe7480ef620 unknown :-1 s=CLOSED pgs=1188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:05.882 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:05.878 7fe750526700 1 --2- 172.21.2.112:0/3640036800 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe7480d17f0 0x7fe7481b31e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:05.882 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:05.878 7fe750526700 1 -- 172.21.2.112:0/3640036800 >> 172.21.2.112:0/3640036800 conn(0x7fe7480fde40 msgr2=0x7fe7480ce390 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:05.882 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:05.878 7fe750526700 1 -- 172.21.2.112:0/3640036800 shutdown_connections 2021-02-10T11:33:05.882 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:05.878 7fe750526700 1 -- 172.21.2.112:0/3640036800 wait complete. 2021-02-10T11:33:05.884 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:05.893 7fe750526700 1 Processor -- start 2021-02-10T11:33:05.884 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:05.893 7fe750526700 1 -- start start 2021-02-10T11:33:05.884 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:05.893 7fe750526700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe7480ae7e0 0x7fe7481038c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:05.884 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:05.893 7fe750526700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe748103ee0 0x7fe74819dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:05.884 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:05.893 7fe750526700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe74819e340 0x7fe748098f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:05.884 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:05.893 7fe750526700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe74819d470 con 0x7fe74819e340 2021-02-10T11:33:05.884 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:05.893 7fe750526700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe748099540 con 0x7fe7480ae7e0 2021-02-10T11:33:05.884 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:05.893 7fe750526700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe748099760 con 0x7fe748103ee0 2021-02-10T11:33:05.884 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:05.893 7fe74eb43700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe74819e340 0x7fe748098f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:05.884 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:05.894 7fe74eb43700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe74819e340 0x7fe748098f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:49428/0 (socket says 172.21.2.112:49428) 2021-02-10T11:33:05.884 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:05.894 7fe74eb43700 1 -- 172.21.2.112:0/1190923749 learned_addr learned my addr 172.21.2.112:0/1190923749 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:33:05.885 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:05.894 7fe74eb43700 1 -- 172.21.2.112:0/1190923749 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe748103ee0 msgr2=0x7fe74819dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:05.885 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:05.894 7fe74eb43700 1 --2- 172.21.2.112:0/1190923749 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe748103ee0 0x7fe74819dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:05.885 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:05.894 7fe74eb43700 1 -- 172.21.2.112:0/1190923749 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe7480ae7e0 msgr2=0x7fe7481038c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:05.885 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:05.894 7fe74eb43700 1 --2- 172.21.2.112:0/1190923749 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe7480ae7e0 0x7fe7481038c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:05.885 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:05.894 7fe74eb43700 1 -- 172.21.2.112:0/1190923749 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe748099980 con 0x7fe74819e340 2021-02-10T11:33:05.885 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:05.894 7fe74eb43700 1 --2- 172.21.2.112:0/1190923749 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe74819e340 0x7fe748098f20 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 rx=0x7fe740009390 tx=0x7fe74000bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:05.885 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:05.894 7fe73affd700 1 -- 172.21.2.112:0/1190923749 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe740003030 con 0x7fe74819e340 2021-02-10T11:33:05.885 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:05.894 7fe73affd700 1 -- 172.21.2.112:0/1190923749 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7400068e0 con 0x7fe74819e340 2021-02-10T11:33:05.885 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:05.894 7fe73affd700 1 -- 172.21.2.112:0/1190923749 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe74000c570 con 0x7fe74819e340 2021-02-10T11:33:05.885 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:05.894 7fe750526700 1 -- 172.21.2.112:0/1190923749 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe748099980 con 0x7fe74819e340 2021-02-10T11:33:05.885 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:05.894 7fe750526700 1 -- 172.21.2.112:0/1190923749 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe74818f8a0 con 0x7fe74819e340 2021-02-10T11:33:05.885 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:05.895 7fe73affd700 1 -- 172.21.2.112:0/1190923749 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe740005820 con 0x7fe74819e340 2021-02-10T11:33:05.886 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:05.895 7fe73affd700 1 --2- 172.21.2.112:0/1190923749 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe7300186c0 0x7fe73001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:05.886 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:05.895 7fe73affd700 1 -- 172.21.2.112:0/1190923749 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fe7400328f0 con 0x7fe74819e340 2021-02-10T11:33:05.886 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:05.895 7fe74e342700 1 --2- 172.21.2.112:0/1190923749 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe7300186c0 0x7fe73001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:05.886 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:05.896 7fe74e342700 1 --2- 172.21.2.112:0/1190923749 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe7300186c0 0x7fe73001abc0 crc :-1 s=READY pgs=597 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:05.913 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:05.926 7fe750526700 1 -- 172.21.2.112:0/1190923749 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe748103690 con 0x7fe74819e340 2021-02-10T11:33:05.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:05.929 7fe73affd700 1 -- 172.21.2.112:0/1190923749 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe740009430 con 0x7fe74819e340 2021-02-10T11:33:06.068 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:06.081 7fe750526700 1 -- 172.21.2.112:0/1190923749 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fe7480f0b20 con 0x7fe74819e340 2021-02-10T11:33:06.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:06.084 7fe73affd700 1 -- 172.21.2.112:0/1190923749 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v7) v1 ==== 104+0+0 (secure 0 0 0) 0x7fe740005820 con 0x7fe74819e340 2021-02-10T11:33:06.073 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:06.086 7fe750526700 1 -- 172.21.2.112:0/1190923749 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe7300186c0 msgr2=0x7fe73001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:06.074 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:06.086 7fe750526700 1 --2- 172.21.2.112:0/1190923749 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe7300186c0 0x7fe73001abc0 crc :-1 s=READY pgs=597 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:06.074 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:06.086 7fe750526700 1 -- 172.21.2.112:0/1190923749 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe74819e340 msgr2=0x7fe748098f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:06.074 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:06.086 7fe750526700 1 --2- 172.21.2.112:0/1190923749 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe74819e340 0x7fe748098f20 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 rx=0x7fe740009390 tx=0x7fe74000bd40).stop 2021-02-10T11:33:06.074 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:06.086 7fe750526700 1 -- 172.21.2.112:0/1190923749 shutdown_connections 2021-02-10T11:33:06.074 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:06.086 7fe750526700 1 --2- 172.21.2.112:0/1190923749 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe7300186c0 0x7fe73001abc0 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:06.074 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:06.086 7fe750526700 1 --2- 172.21.2.112:0/1190923749 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe74819e340 0x7fe748098f20 unknown :-1 s=CLOSED pgs=1189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:06.074 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:06.086 7fe750526700 1 --2- 172.21.2.112:0/1190923749 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe748103ee0 0x7fe74819dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:06.074 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:06.086 7fe750526700 1 --2- 172.21.2.112:0/1190923749 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe7480ae7e0 0x7fe7481038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:06.074 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:06.086 7fe750526700 1 -- 172.21.2.112:0/1190923749 >> 172.21.2.112:0/1190923749 conn(0x7fe7481b04b0 msgr2=0x7fe74819aa40 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:06.074 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:06.086 7fe750526700 1 -- 172.21.2.112:0/1190923749 shutdown_connections 2021-02-10T11:33:06.074 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:06.086 7fe750526700 1 -- 172.21.2.112:0/1190923749 wait complete. 2021-02-10T11:33:06.082 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestVolumes.test_authorize_auth_id_not_created_by_mgr_volumes' 2021-02-10T11:33:06.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:06.503 7f74a97bc700 1 Processor -- start 2021-02-10T11:33:06.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:06.503 7f74a97bc700 1 -- start start 2021-02-10T11:33:06.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:06.504 7f74a97bc700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f74a40d17f0 0x7f74a41b31e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:06.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:06.504 7f74a97bc700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f74a40d1bf0 0x7f74a41b3800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:06.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:06.504 7f74a97bc700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f74a41b3e20 0x7f74a40ceef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:06.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:06.504 7f74a97bc700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74a40e5eb0 con 0x7f74a40d1bf0 2021-02-10T11:33:06.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:06.504 7f74a97bc700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74a40f0b20 con 0x7f74a41b3e20 2021-02-10T11:33:06.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:06.504 7f74a97bc700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74a40cf510 con 0x7f74a40d17f0 2021-02-10T11:33:06.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:06.504 7f74a2d73700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f74a40d1bf0 0x7f74a41b3800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:06.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:06.504 7f74a2d73700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f74a40d1bf0 0x7f74a41b3800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:49434/0 (socket says 172.21.2.112:49434) 2021-02-10T11:33:06.508 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:06.504 7f74a2d73700 1 -- 172.21.2.112:0/1765613119 learned_addr learned my addr 172.21.2.112:0/1765613119 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:33:06.508 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:06.504 7f74a3d75700 1 --2- 172.21.2.112:0/1765613119 >> v2:172.21.2.120:3300/0 conn(0x7f74a41b3e20 0x7f74a40ceef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:06.508 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:06.504 7f74a3574700 1 --2- 172.21.2.112:0/1765613119 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f74a40d17f0 0x7f74a41b31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:06.508 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:06.504 7f74a2d73700 1 -- 172.21.2.112:0/1765613119 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f74a40d17f0 msgr2=0x7f74a41b31e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:06.508 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:06.504 7f74a2d73700 1 --2- 172.21.2.112:0/1765613119 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f74a40d17f0 0x7f74a41b31e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:06.508 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:06.504 7f74a2d73700 1 -- 172.21.2.112:0/1765613119 >> v2:172.21.2.120:3300/0 conn(0x7f74a41b3e20 msgr2=0x7f74a40ceef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:06.508 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:06.504 7f74a2d73700 1 --2- 172.21.2.112:0/1765613119 >> v2:172.21.2.120:3300/0 conn(0x7f74a41b3e20 0x7f74a40ceef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:06.508 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:06.504 7f74a2d73700 1 -- 172.21.2.112:0/1765613119 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74a40cf730 con 0x7f74a40d1bf0 2021-02-10T11:33:06.508 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:06.505 7f74a2d73700 1 --2- 172.21.2.112:0/1765613119 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f74a40d1bf0 0x7f74a41b3800 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 rx=0x7f7494006700 tx=0x7f7494009b80).ready entity=mon.0 client_cookie=4b50631067ef681f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:06.508 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:06.505 7f74a2572700 1 -- 172.21.2.112:0/1765613119 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f749400c0c0 con 0x7f74a40d1bf0 2021-02-10T11:33:06.508 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:06.505 7f74a2572700 1 -- 172.21.2.112:0/1765613119 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f749400c340 con 0x7f74a40d1bf0 2021-02-10T11:33:06.508 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:06.505 7f74a2572700 1 -- 172.21.2.112:0/1765613119 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f749400c860 con 0x7f74a40d1bf0 2021-02-10T11:33:06.508 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:06.505 7f74a97bc700 1 -- 172.21.2.112:0/1765613119 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f74a40d1bf0 msgr2=0x7f74a41b3800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:06.509 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:06.505 7f74a97bc700 1 --2- 172.21.2.112:0/1765613119 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f74a40d1bf0 0x7f74a41b3800 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 rx=0x7f7494006700 tx=0x7f7494009b80).stop 2021-02-10T11:33:06.509 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:06.505 7f74a97bc700 1 -- 172.21.2.112:0/1765613119 shutdown_connections 2021-02-10T11:33:06.509 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:06.505 7f74a97bc700 1 --2- 172.21.2.112:0/1765613119 >> v2:172.21.2.120:3300/0 conn(0x7f74a41b3e20 0x7f74a40ceef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:06.509 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:06.505 7f74a97bc700 1 --2- 172.21.2.112:0/1765613119 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f74a40d1bf0 0x7f74a41b3800 unknown :-1 s=CLOSED pgs=1190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:06.509 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:06.505 7f74a97bc700 1 --2- 172.21.2.112:0/1765613119 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f74a40d17f0 0x7f74a41b31e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:06.509 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:06.505 7f74a97bc700 1 -- 172.21.2.112:0/1765613119 >> 172.21.2.112:0/1765613119 conn(0x7f74a40fde40 msgr2=0x7f74a40ce390 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:06.509 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:06.506 7f74a97bc700 1 -- 172.21.2.112:0/1765613119 shutdown_connections 2021-02-10T11:33:06.509 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:06.506 7f74a97bc700 1 -- 172.21.2.112:0/1765613119 wait complete. 2021-02-10T11:33:06.511 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:06.520 7f74a97bc700 1 Processor -- start 2021-02-10T11:33:06.511 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:06.520 7f74a97bc700 1 -- start start 2021-02-10T11:33:06.511 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:06.520 7f74a97bc700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f74a4093a40 0x7f74a4094010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:06.511 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:06.520 7f74a97bc700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f74a4189e40 0x7f74a41a7180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:06.511 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:06.520 7f74a97bc700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f74a418a240 0x7f74a41a77a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:06.512 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:06.520 7f74a97bc700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74a40947a0 con 0x7f74a4093a40 2021-02-10T11:33:06.512 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:06.520 7f74a97bc700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74a41a7dc0 con 0x7f74a4189e40 2021-02-10T11:33:06.512 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:06.520 7f74a97bc700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74a41a7fe0 con 0x7f74a418a240 2021-02-10T11:33:06.512 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:06.521 7f74a3574700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f74a4093a40 0x7f74a4094010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:06.512 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:06.521 7f74a3574700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f74a4093a40 0x7f74a4094010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:49440/0 (socket says 172.21.2.112:49440) 2021-02-10T11:33:06.512 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:06.521 7f74a3574700 1 -- 172.21.2.112:0/782833582 learned_addr learned my addr 172.21.2.112:0/782833582 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:33:06.512 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:06.521 7f74a2d73700 1 --2- 172.21.2.112:0/782833582 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f74a4189e40 0x7f74a41a7180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:06.512 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:06.521 7f74a3d75700 1 --2- 172.21.2.112:0/782833582 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f74a418a240 0x7f74a41a77a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:06.512 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:06.521 7f74a3574700 1 -- 172.21.2.112:0/782833582 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f74a418a240 msgr2=0x7f74a41a77a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:06.512 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:06.521 7f74a3574700 1 --2- 172.21.2.112:0/782833582 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f74a418a240 0x7f74a41a77a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:06.512 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:06.521 7f74a3574700 1 -- 172.21.2.112:0/782833582 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f74a4189e40 msgr2=0x7f74a41a7180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:06.512 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:06.521 7f74a3574700 1 --2- 172.21.2.112:0/782833582 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f74a4189e40 0x7f74a41a7180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:06.512 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:06.521 7f74a3574700 1 -- 172.21.2.112:0/782833582 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74a418ad20 con 0x7f74a4093a40 2021-02-10T11:33:06.513 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:06.521 7f74a3574700 1 --2- 172.21.2.112:0/782833582 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f74a4093a40 0x7f74a4094010 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 rx=0x7f748c006f00 tx=0x7f748c00b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:06.513 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:06.521 7f749bfff700 1 -- 172.21.2.112:0/782833582 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f748c00bd30 con 0x7f74a4093a40 2021-02-10T11:33:06.513 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:06.522 7f749bfff700 1 -- 172.21.2.112:0/782833582 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f748c002030 con 0x7f74a4093a40 2021-02-10T11:33:06.513 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:06.522 7f749bfff700 1 -- 172.21.2.112:0/782833582 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f748c002590 con 0x7f74a4093a40 2021-02-10T11:33:06.513 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:06.522 7f74a97bc700 1 -- 172.21.2.112:0/782833582 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74a418ad20 con 0x7f74a4093a40 2021-02-10T11:33:06.513 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:06.522 7f74a97bc700 1 -- 172.21.2.112:0/782833582 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74a418ad20 con 0x7f74a4093a40 2021-02-10T11:33:06.513 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:06.522 7f749bfff700 1 -- 172.21.2.112:0/782833582 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f748c000c90 con 0x7f74a4093a40 2021-02-10T11:33:06.513 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:06.522 7f749bfff700 1 --2- 172.21.2.112:0/782833582 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7488018700 0x7f748801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:06.513 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:06.522 7f749bfff700 1 -- 172.21.2.112:0/782833582 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f748c032660 con 0x7f74a4093a40 2021-02-10T11:33:06.513 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:06.522 7f74a2d73700 1 --2- 172.21.2.112:0/782833582 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7488018700 0x7f748801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:06.513 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:06.523 7f74a2d73700 1 --2- 172.21.2.112:0/782833582 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7488018700 0x7f748801abc0 crc :-1 s=READY pgs=598 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:06.541 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:06.553 7f74a97bc700 1 -- 172.21.2.112:0/782833582 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74a40947a0 con 0x7f74a4093a40 2021-02-10T11:33:06.543 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:06.556 7f749bfff700 1 -- 172.21.2.112:0/782833582 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f748c01f520 con 0x7f74a4093a40 2021-02-10T11:33:06.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:06.688 7f74a97bc700 1 -- 172.21.2.112:0/782833582 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_authorize_auth_id_not_created_by_mgr_volumes"]} v 0) v1 -- 0x7f74a40947a0 con 0x7f74a4093a40 2021-02-10T11:33:07.045 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:07.058 7f749bfff700 1 -- 172.21.2.112:0/782833582 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_authorize_auth_id_not_created_by_mgr_volumes"]}]=0 v278) v1 ==== 166+0+0 (secure 0 0 0) 0x7f748c000c90 con 0x7f74a4093a40 2021-02-10T11:33:07.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:07.107 7f74a97bc700 1 -- 172.21.2.112:0/782833582 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7488018700 msgr2=0x7f748801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:07.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:07.107 7f74a97bc700 1 --2- 172.21.2.112:0/782833582 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7488018700 0x7f748801abc0 crc :-1 s=READY pgs=598 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:07.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:07.107 7f74a97bc700 1 -- 172.21.2.112:0/782833582 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f74a4093a40 msgr2=0x7f74a4094010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:07.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:07.107 7f74a97bc700 1 --2- 172.21.2.112:0/782833582 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f74a4093a40 0x7f74a4094010 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 rx=0x7f748c006f00 tx=0x7f748c00b940).stop 2021-02-10T11:33:07.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:07.108 7f74a97bc700 1 -- 172.21.2.112:0/782833582 shutdown_connections 2021-02-10T11:33:07.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:07.108 7f74a97bc700 1 --2- 172.21.2.112:0/782833582 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7488018700 0x7f748801abc0 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:07.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:07.108 7f74a97bc700 1 --2- 172.21.2.112:0/782833582 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f74a4189e40 0x7f74a41a7180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:07.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:07.108 7f74a97bc700 1 --2- 172.21.2.112:0/782833582 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f74a418a240 0x7f74a41a77a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:07.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:07.108 7f74a97bc700 1 --2- 172.21.2.112:0/782833582 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f74a4093a40 0x7f74a4094010 unknown :-1 s=CLOSED pgs=1191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:07.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:07.108 7f74a97bc700 1 -- 172.21.2.112:0/782833582 >> 172.21.2.112:0/782833582 conn(0x7f74a40aa4a0 msgr2=0x7f74a40a8890 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:07.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:07.108 7f74a97bc700 1 -- 172.21.2.112:0/782833582 shutdown_connections 2021-02-10T11:33:07.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:07.108 7f74a97bc700 1 -- 172.21.2.112:0/782833582 wait complete. 2021-02-10T11:33:07.103 INFO:tasks.cephfs_test_runner:test_authorize_auth_id_not_created_by_mgr_volumes (tasks.cephfs.test_volumes.TestVolumes) ... ok 2021-02-10T11:33:07.103 INFO:tasks.cephfs_test_runner:Starting test: test_authorize_deauthorize_legacy_subvolume (tasks.cephfs.test_volumes.TestVolumes) 2021-02-10T11:33:07.103 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestVolumes.test_authorize_deauthorize_legacy_subvolume' 2021-02-10T11:33:07.528 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:07.525 7ff820009700 1 Processor -- start 2021-02-10T11:33:07.529 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:07.525 7ff820009700 1 -- start start 2021-02-10T11:33:07.529 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:07.526 7ff820009700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7ff81807e6a0 0x7ff8180d1790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:07.529 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:07.526 7ff820009700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff81807ed80 con 0x7ff8180e5470 2021-02-10T11:33:07.529 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:07.526 7ff820009700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff81807eaa0 con 0x7ff8180e5070 2021-02-10T11:33:07.529 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:07.526 7ff820009700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8181b4080 con 0x7ff81807e6a0 2021-02-10T11:33:07.529 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:07.526 7ff81d624700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7ff8180e5470 0x7ff81807f300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:48908/0 (socket says 172.21.2.112:48908) 2021-02-10T11:33:07.529 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:07.526 7ff81d624700 1 -- 172.21.2.112:0/1855124824 learned_addr learned my addr 172.21.2.112:0/1855124824 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:33:07.529 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:07.526 7ff81e626700 1 --2- 172.21.2.112:0/1855124824 >> v2:172.21.2.112:3300/0 conn(0x7ff81807e6a0 0x7ff8180d1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:07.529 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:07.526 7ff81e626700 1 -- 172.21.2.112:0/1855124824 >> v1:172.21.2.120:6789/0 conn(0x7ff8180e5070 legacy=0x7ff8181b3820 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:33:07.530 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:07.526 7ff81e626700 1 -- 172.21.2.112:0/1855124824 >> v1:172.21.2.112:6789/0 conn(0x7ff8180e5470 legacy=0x7ff81807f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:07.530 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:07.527 7ff81ce23700 1 -- 172.21.2.112:0/1855124824 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 976885439 0 0) 0x7ff80c0010b0 con 0x7ff8180e5470 2021-02-10T11:33:07.530 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:07.527 7ff81e626700 1 -- 172.21.2.112:0/1855124824 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8180d1db0 con 0x7ff81807e6a0 2021-02-10T11:33:07.530 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:07.527 7ff81e626700 1 --2- 172.21.2.112:0/1855124824 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff81807e6a0 0x7ff8180d1790 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 rx=0x7ff810006e30 tx=0x7ff810009de0).ready entity=mon.0 client_cookie=800dead9a3357e44 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:07.530 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:07.527 7ff81ce23700 1 -- 172.21.2.112:0/1855124824 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff81000b300 con 0x7ff81807e6a0 2021-02-10T11:33:07.530 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:07.527 7ff81ce23700 1 -- 172.21.2.112:0/1855124824 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff81000b680 con 0x7ff81807e6a0 2021-02-10T11:33:07.530 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:07.527 7ff81ce23700 1 -- 172.21.2.112:0/1855124824 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff81000bba0 con 0x7ff81807e6a0 2021-02-10T11:33:07.530 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:07.527 7ff820009700 1 -- 172.21.2.112:0/1855124824 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff81807e6a0 msgr2=0x7ff8180d1790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:07.530 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:07.527 7ff820009700 1 --2- 172.21.2.112:0/1855124824 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff81807e6a0 0x7ff8180d1790 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 rx=0x7ff810006e30 tx=0x7ff810009de0).stop 2021-02-10T11:33:07.531 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:07.527 7ff820009700 1 -- 172.21.2.112:0/1855124824 shutdown_connections 2021-02-10T11:33:07.531 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:07.527 7ff820009700 1 --2- 172.21.2.112:0/1855124824 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff81807e6a0 0x7ff8180d1790 unknown :-1 s=CLOSED pgs=1193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:07.531 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:07.527 7ff820009700 1 -- 172.21.2.112:0/1855124824 >> 172.21.2.112:0/1855124824 conn(0x7ff8180ce690 msgr2=0x7ff8180e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:07.531 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:07.528 7ff820009700 1 -- 172.21.2.112:0/1855124824 shutdown_connections 2021-02-10T11:33:07.531 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:07.528 7ff820009700 1 -- 172.21.2.112:0/1855124824 wait complete. 2021-02-10T11:33:07.532 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:07.542 7ff820009700 1 Processor -- start 2021-02-10T11:33:07.532 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:07.542 7ff820009700 1 -- start start 2021-02-10T11:33:07.532 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:07.542 7ff820009700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff8181ade10 0x7ff81809d250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:07.532 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:07.542 7ff820009700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff8180b4440 0x7ff8180b4840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:07.532 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:07.542 7ff820009700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff8181952d0 0x7ff8181956d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:07.533 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:07.542 7ff820009700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff818194d50 con 0x7ff8181952d0 2021-02-10T11:33:07.533 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:07.542 7ff820009700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8181932c0 con 0x7ff8180b4440 2021-02-10T11:33:07.533 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:07.542 7ff820009700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff818195cf0 con 0x7ff8181ade10 2021-02-10T11:33:07.533 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:07.543 7ff81e626700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff8181952d0 0x7ff8181956d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:07.533 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:07.543 7ff81e626700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff8181952d0 0x7ff8181956d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:49456/0 (socket says 172.21.2.112:49456) 2021-02-10T11:33:07.533 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:07.543 7ff81e626700 1 -- 172.21.2.112:0/2327245948 learned_addr learned my addr 172.21.2.112:0/2327245948 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:33:07.533 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:07.543 7ff81e626700 1 -- 172.21.2.112:0/2327245948 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff8181ade10 msgr2=0x7ff81809d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:07.533 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:07.543 7ff81e626700 1 --2- 172.21.2.112:0/2327245948 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff8181ade10 0x7ff81809d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:07.533 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:07.543 7ff81e626700 1 -- 172.21.2.112:0/2327245948 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff8180b4440 msgr2=0x7ff8180b4840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:07.533 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:07.543 7ff81e626700 1 --2- 172.21.2.112:0/2327245948 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff8180b4440 0x7ff8180b4840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:07.533 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:07.543 7ff81e626700 1 -- 172.21.2.112:0/2327245948 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff818195f10 con 0x7ff8181952d0 2021-02-10T11:33:07.534 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:07.543 7ff81e626700 1 --2- 172.21.2.112:0/2327245948 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff8181952d0 0x7ff8181956d0 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 rx=0x7ff81000b970 tx=0x7ff81000bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:07.534 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:07.543 7ff80a7fc700 1 -- 172.21.2.112:0/2327245948 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff810009380 con 0x7ff8181952d0 2021-02-10T11:33:07.534 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:07.543 7ff80a7fc700 1 -- 172.21.2.112:0/2327245948 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff810001270 con 0x7ff8181952d0 2021-02-10T11:33:07.534 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:07.544 7ff80a7fc700 1 -- 172.21.2.112:0/2327245948 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff810001790 con 0x7ff8181952d0 2021-02-10T11:33:07.534 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:07.544 7ff820009700 1 -- 172.21.2.112:0/2327245948 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff818195f10 con 0x7ff8181952d0 2021-02-10T11:33:07.534 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:07.544 7ff820009700 1 -- 172.21.2.112:0/2327245948 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff818195f10 con 0x7ff8181952d0 2021-02-10T11:33:07.534 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:07.544 7ff80a7fc700 1 -- 172.21.2.112:0/2327245948 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff810001450 con 0x7ff8181952d0 2021-02-10T11:33:07.534 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:07.544 7ff80a7fc700 1 --2- 172.21.2.112:0/2327245948 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff8000186c0 0x7ff80001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:07.534 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:07.544 7ff80a7fc700 1 -- 172.21.2.112:0/2327245948 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5573+0+0 (secure 0 0 0) 0x7ff8100328e0 con 0x7ff8181952d0 2021-02-10T11:33:07.534 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:07.545 7ff81de25700 1 --2- 172.21.2.112:0/2327245948 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff8000186c0 0x7ff80001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:07.535 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:07.546 7ff81de25700 1 --2- 172.21.2.112:0/2327245948 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff8000186c0 0x7ff80001abc0 crc :-1 s=READY pgs=599 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:07.567 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:07.576 7ff820009700 1 -- 172.21.2.112:0/2327245948 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff818083db0 con 0x7ff8181952d0 2021-02-10T11:33:07.567 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:07.579 7ff80a7fc700 1 -- 172.21.2.112:0/2327245948 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff810003070 con 0x7ff8181952d0 2021-02-10T11:33:07.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:07.709 7ff820009700 1 -- 172.21.2.112:0/2327245948 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_authorize_deauthorize_legacy_subvolume"]} v 0) v1 -- 0x7ff81807ed80 con 0x7ff8181952d0 2021-02-10T11:33:08.050 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:08.063 7ff80a7fc700 1 -- 172.21.2.112:0/2327245948 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_authorize_deauthorize_legacy_subvolume"]}]=0 v279) v1 ==== 163+0+0 (secure 0 0 0) 0x7ff810001450 con 0x7ff8181952d0 2021-02-10T11:33:08.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:08.077 7ff820009700 1 -- 172.21.2.112:0/2327245948 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff8000186c0 msgr2=0x7ff80001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:08.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:08.077 7ff820009700 1 --2- 172.21.2.112:0/2327245948 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff8000186c0 0x7ff80001abc0 crc :-1 s=READY pgs=599 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:08.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:08.077 7ff820009700 1 -- 172.21.2.112:0/2327245948 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff8181952d0 msgr2=0x7ff8181956d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:08.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:08.077 7ff820009700 1 --2- 172.21.2.112:0/2327245948 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff8181952d0 0x7ff8181956d0 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 rx=0x7ff81000b970 tx=0x7ff81000bd10).stop 2021-02-10T11:33:08.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:08.077 7ff820009700 1 -- 172.21.2.112:0/2327245948 shutdown_connections 2021-02-10T11:33:08.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:08.077 7ff820009700 1 --2- 172.21.2.112:0/2327245948 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff8000186c0 0x7ff80001abc0 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:08.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:08.077 7ff820009700 1 --2- 172.21.2.112:0/2327245948 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff8180b4440 0x7ff8180b4840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:08.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:08.077 7ff820009700 1 --2- 172.21.2.112:0/2327245948 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff8181952d0 0x7ff8181956d0 unknown :-1 s=CLOSED pgs=1194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:08.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:08.077 7ff820009700 1 --2- 172.21.2.112:0/2327245948 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff8181ade10 0x7ff81809d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:08.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:08.077 7ff820009700 1 -- 172.21.2.112:0/2327245948 >> 172.21.2.112:0/2327245948 conn(0x7ff8180937a0 msgr2=0x7ff818197e80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:08.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:08.077 7ff820009700 1 -- 172.21.2.112:0/2327245948 shutdown_connections 2021-02-10T11:33:08.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:08.077 7ff820009700 1 -- 172.21.2.112:0/2327245948 wait complete. 2021-02-10T11:33:08.072 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:33:08.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:08.498 7f15c50e3700 1 Processor -- start 2021-02-10T11:33:08.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:08.498 7f15c50e3700 1 -- start start 2021-02-10T11:33:08.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:08.498 7f15c50e3700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f15c01b0870 0x7f15c00f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:08.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:08.498 7f15c50e3700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f15c00efe00 0x7f15c00f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:08.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:08.498 7f15c50e3700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15c01b0c70 con 0x7f15c01b0470 2021-02-10T11:33:08.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:08.498 7f15c50e3700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15c00e5ea0 con 0x7f15c01b0870 2021-02-10T11:33:08.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:08.498 7f15c50e3700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15c00f0b50 con 0x7f15c00efe00 2021-02-10T11:33:08.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:08.498 7f15be59c700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f15c01b0870 0x7f15c00f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:08.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:08.498 7f15be59c700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f15c01b0870 0x7f15c00f20e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:49462/0 (socket says 172.21.2.112:49462) 2021-02-10T11:33:08.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:08.498 7f15be59c700 1 -- 172.21.2.112:0/2188233456 learned_addr learned my addr 172.21.2.112:0/2188233456 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:33:08.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:08.499 7f15be59c700 1 -- 172.21.2.112:0/2188233456 >> v2:172.21.2.120:3300/0 conn(0x7f15c00efe00 msgr2=0x7f15c00f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:08.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:08.499 7f15be59c700 1 --2- 172.21.2.112:0/2188233456 >> v2:172.21.2.120:3300/0 conn(0x7f15c00efe00 0x7f15c00f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:08.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:08.499 7f15be59c700 1 -- 172.21.2.112:0/2188233456 >> v1:172.21.2.120:6789/0 conn(0x7f15c01b0470 legacy=0x7f15c00ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:08.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:08.499 7f15be59c700 1 -- 172.21.2.112:0/2188233456 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15c00f0820 con 0x7f15c01b0870 2021-02-10T11:33:08.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:08.499 7f15be59c700 1 --2- 172.21.2.112:0/2188233456 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f15c01b0870 0x7f15c00f20e0 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 rx=0x7f15b0006700 tx=0x7f15b0009b80).ready entity=mon.0 client_cookie=760aacc695814634 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:08.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:08.499 7f15bdd9b700 1 -- 172.21.2.112:0/2188233456 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f15b000c0c0 con 0x7f15c01b0870 2021-02-10T11:33:08.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:08.499 7f15bdd9b700 1 -- 172.21.2.112:0/2188233456 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15b000c400 con 0x7f15c01b0870 2021-02-10T11:33:08.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:08.499 7f15bdd9b700 1 -- 172.21.2.112:0/2188233456 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f15b000c860 con 0x7f15c01b0870 2021-02-10T11:33:08.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:08.499 7f15c50e3700 1 -- 172.21.2.112:0/2188233456 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f15c01b0870 msgr2=0x7f15c00f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:08.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:08.499 7f15c50e3700 1 --2- 172.21.2.112:0/2188233456 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f15c01b0870 0x7f15c00f20e0 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 rx=0x7f15b0006700 tx=0x7f15b0009b80).stop 2021-02-10T11:33:08.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:08.500 7f15c50e3700 1 -- 172.21.2.112:0/2188233456 shutdown_connections 2021-02-10T11:33:08.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:08.500 7f15c50e3700 1 --2- 172.21.2.112:0/2188233456 >> v2:172.21.2.120:3300/0 conn(0x7f15c00efe00 0x7f15c00f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:08.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:08.500 7f15c50e3700 1 --2- 172.21.2.112:0/2188233456 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f15c01b0870 0x7f15c00f20e0 unknown :-1 s=CLOSED pgs=1195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:08.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:08.500 7f15c50e3700 1 -- 172.21.2.112:0/2188233456 >> 172.21.2.112:0/2188233456 conn(0x7f15c00ce3e0 msgr2=0x7f15c00e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:08.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:08.500 7f15c50e3700 1 -- 172.21.2.112:0/2188233456 shutdown_connections 2021-02-10T11:33:08.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:08.500 7f15c50e3700 1 -- 172.21.2.112:0/2188233456 wait complete. 2021-02-10T11:33:08.505 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:08.515 7f15c50e3700 1 Processor -- start 2021-02-10T11:33:08.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:08.515 7f15c50e3700 1 -- start start 2021-02-10T11:33:08.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:08.515 7f15c50e3700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f15c00a6c90 0x7f15c01924c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:08.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:08.515 7f15c50e3700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f15c0192ae0 0x7f15c0192ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:08.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:08.515 7f15c50e3700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f15c00919e0 0x7f15c0091de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:08.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:08.515 7f15c50e3700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15c019d2e0 con 0x7f15c00a6c90 2021-02-10T11:33:08.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:08.515 7f15c50e3700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15c00914d0 con 0x7f15c00919e0 2021-02-10T11:33:08.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:08.515 7f15c50e3700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15c00a6820 con 0x7f15c0192ae0 2021-02-10T11:33:08.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:08.516 7f15bed9d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f15c00a6c90 0x7f15c01924c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:08.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:08.516 7f15bed9d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f15c00a6c90 0x7f15c01924c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:49466/0 (socket says 172.21.2.112:49466) 2021-02-10T11:33:08.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:08.516 7f15bed9d700 1 -- 172.21.2.112:0/530798955 learned_addr learned my addr 172.21.2.112:0/530798955 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:33:08.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:08.516 7f15be59c700 1 --2- 172.21.2.112:0/530798955 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f15c0192ae0 0x7f15c0192ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:08.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:08.516 7f15bf59e700 1 --2- 172.21.2.112:0/530798955 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f15c00919e0 0x7f15c0091de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:08.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:08.516 7f15bed9d700 1 -- 172.21.2.112:0/530798955 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f15c0192ae0 msgr2=0x7f15c0192ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:08.508 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:08.516 7f15bed9d700 1 --2- 172.21.2.112:0/530798955 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f15c0192ae0 0x7f15c0192ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:08.508 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:08.516 7f15bed9d700 1 -- 172.21.2.112:0/530798955 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f15c00919e0 msgr2=0x7f15c0091de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:08.508 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:08.516 7f15bed9d700 1 --2- 172.21.2.112:0/530798955 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f15c00919e0 0x7f15c0091de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:08.508 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:08.516 7f15bed9d700 1 -- 172.21.2.112:0/530798955 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15c0092400 con 0x7f15c00a6c90 2021-02-10T11:33:08.508 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:08.516 7f15be59c700 1 --2- 172.21.2.112:0/530798955 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f15c0192ae0 0x7f15c0192ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:33:08.508 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:08.516 7f15bf59e700 1 --2- 172.21.2.112:0/530798955 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f15c00919e0 0x7f15c0091de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:33:08.508 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:08.516 7f15bed9d700 1 --2- 172.21.2.112:0/530798955 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f15c00a6c90 0x7f15c01924c0 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 rx=0x7f15a800b5b0 tx=0x7f15a800b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:08.508 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:08.516 7f15b77fe700 1 -- 172.21.2.112:0/530798955 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f15a800bd40 con 0x7f15c00a6c90 2021-02-10T11:33:08.508 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:08.516 7f15b77fe700 1 -- 172.21.2.112:0/530798955 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15a8006020 con 0x7f15c00a6c90 2021-02-10T11:33:08.508 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:08.516 7f15c50e3700 1 -- 172.21.2.112:0/530798955 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15c0092400 con 0x7f15c00a6c90 2021-02-10T11:33:08.508 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:08.516 7f15b77fe700 1 -- 172.21.2.112:0/530798955 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f15a80054a0 con 0x7f15c00a6c90 2021-02-10T11:33:08.508 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:08.516 7f15c50e3700 1 -- 172.21.2.112:0/530798955 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15c0092400 con 0x7f15c00a6c90 2021-02-10T11:33:08.509 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:08.517 7f15b77fe700 1 -- 172.21.2.112:0/530798955 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f15a8006200 con 0x7f15c00a6c90 2021-02-10T11:33:08.509 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:08.517 7f15b77fe700 1 --2- 172.21.2.112:0/530798955 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f15a40186c0 0x7f15a401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:08.509 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:08.517 7f15b77fe700 1 -- 172.21.2.112:0/530798955 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f15a8032620 con 0x7f15c00a6c90 2021-02-10T11:33:08.509 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:08.518 7f15be59c700 1 --2- 172.21.2.112:0/530798955 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f15a40186c0 0x7f15a401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:08.509 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:08.518 7f15be59c700 1 --2- 172.21.2.112:0/530798955 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f15a40186c0 0x7f15a401abc0 crc :-1 s=READY pgs=600 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:08.535 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:08.548 7f15c50e3700 1 -- 172.21.2.112:0/530798955 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15c00e52e0 con 0x7f15c00a6c90 2021-02-10T11:33:08.538 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:08.551 7f15b77fe700 1 -- 172.21.2.112:0/530798955 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f15a801f4e0 con 0x7f15c00a6c90 2021-02-10T11:33:08.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:08.700 7f15c50e3700 1 -- 172.21.2.112:0/530798955 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f15c00e52e0 con 0x7f15c00a6c90 2021-02-10T11:33:08.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:08.704 7f15b77fe700 1 -- 172.21.2.112:0/530798955 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v8)=0 v8) v1 ==== 122+0+0 (secure 0 0 0) 0x7f15a8006000 con 0x7f15c00a6c90 2021-02-10T11:33:08.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:08.709 7f15c50e3700 1 -- 172.21.2.112:0/530798955 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f15a40186c0 msgr2=0x7f15a401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:08.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:08.709 7f15c50e3700 1 --2- 172.21.2.112:0/530798955 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f15a40186c0 0x7f15a401abc0 crc :-1 s=READY pgs=600 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:08.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:08.709 7f15c50e3700 1 -- 172.21.2.112:0/530798955 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f15c00a6c90 msgr2=0x7f15c01924c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:08.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:08.709 7f15c50e3700 1 --2- 172.21.2.112:0/530798955 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f15c00a6c90 0x7f15c01924c0 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 rx=0x7f15a800b5b0 tx=0x7f15a800b950).stop 2021-02-10T11:33:08.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:08.709 7f15c50e3700 1 -- 172.21.2.112:0/530798955 shutdown_connections 2021-02-10T11:33:08.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:08.709 7f15c50e3700 1 --2- 172.21.2.112:0/530798955 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f15a40186c0 0x7f15a401abc0 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:08.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:08.709 7f15c50e3700 1 --2- 172.21.2.112:0/530798955 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f15c00919e0 0x7f15c0091de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:08.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:08.709 7f15c50e3700 1 --2- 172.21.2.112:0/530798955 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f15c0192ae0 0x7f15c0192ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:08.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:08.709 7f15c50e3700 1 --2- 172.21.2.112:0/530798955 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f15c00a6c90 0x7f15c01924c0 unknown :-1 s=CLOSED pgs=1196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:08.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:08.709 7f15c50e3700 1 -- 172.21.2.112:0/530798955 >> 172.21.2.112:0/530798955 conn(0x7f15c00ac550 msgr2=0x7f15c00be1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:08.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:08.709 7f15c50e3700 1 -- 172.21.2.112:0/530798955 shutdown_connections 2021-02-10T11:33:08.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:08.709 7f15c50e3700 1 -- 172.21.2.112:0/530798955 wait complete. 2021-02-10T11:33:08.705 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:33:08.706 DEBUG:teuthology.orchestra.run.gibba012:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:33:09.051 DEBUG:teuthology.orchestra.run.gibba020:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:33:09.396 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:33:09.396 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:09.396 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:33:09.396 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:33:09.731 INFO:teuthology.orchestra.run.gibba012.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.0’: No such file or directory 2021-02-10T11:33:09.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:33:09.732 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2021-02-10T11:33:09.732 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:09.732 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:33:09.732 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:33:10.067 INFO:teuthology.orchestra.run.gibba012.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: No such file or directory 2021-02-10T11:33:10.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:33:10.067 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2021-02-10T11:33:10.067 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:10.068 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:33:10.068 DEBUG:teuthology.orchestra.run.gibba020:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:33:10.401 INFO:teuthology.orchestra.run.gibba020.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:33:10.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:33:10.402 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:33:10.402 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:10.402 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:33:10.402 DEBUG:teuthology.orchestra.run.gibba020:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:33:10.735 INFO:teuthology.orchestra.run.gibba020.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:33:10.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:33:10.736 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:33:10.736 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json-pretty 2021-02-10T11:33:11.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.158 7fdf4895d700 1 Processor -- start 2021-02-10T11:33:11.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.158 7fdf4895d700 1 -- start start 2021-02-10T11:33:11.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.159 7fdf4895d700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fdf440e5070 0x7fdf441b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:11.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.159 7fdf4895d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdf441b3e40 0x7fdf4407d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:11.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.159 7fdf4895d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fdf440e5470 0x7fdf4407dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:11.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.159 7fdf4895d700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf4407ed80 con 0x7fdf440e5470 2021-02-10T11:33:11.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.159 7fdf4895d700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf4407e2d0 con 0x7fdf440e5070 2021-02-10T11:33:11.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.159 7fdf4895d700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf4407e4c0 con 0x7fdf441b3e40 2021-02-10T11:33:11.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.159 7fdf42e1d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fdf440e5470 0x7fdf4407dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:11.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.159 7fdf42e1d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fdf440e5470 0x7fdf4407dcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:49474/0 (socket says 172.21.2.112:49474) 2021-02-10T11:33:11.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.159 7fdf42e1d700 1 -- 172.21.2.112:0/756558429 learned_addr learned my addr 172.21.2.112:0/756558429 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:33:11.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.159 7fdf42e1d700 1 -- 172.21.2.112:0/756558429 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdf441b3e40 msgr2=0x7fdf4407d690 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:33:11.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.159 7fdf42e1d700 1 --2- 172.21.2.112:0/756558429 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdf441b3e40 0x7fdf4407d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:11.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.159 7fdf42e1d700 1 -- 172.21.2.112:0/756558429 >> v2:172.21.2.120:3300/0 conn(0x7fdf440e5070 msgr2=0x7fdf441b3820 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:33:11.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.159 7fdf42e1d700 1 --2- 172.21.2.112:0/756558429 >> v2:172.21.2.120:3300/0 conn(0x7fdf440e5070 0x7fdf441b3820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:11.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.159 7fdf42e1d700 1 -- 172.21.2.112:0/756558429 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf4407e6e0 con 0x7fdf440e5470 2021-02-10T11:33:11.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.160 7fdf42e1d700 1 --2- 172.21.2.112:0/756558429 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdf440e5470 0x7fdf4407dcb0 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 rx=0x7fdf3c006f00 tx=0x7fdf3c009f10).ready entity=mon.0 client_cookie=e6944cc20a2934bb server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:11.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.160 7fdf4161a700 1 -- 172.21.2.112:0/756558429 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdf3c00b3b0 con 0x7fdf440e5470 2021-02-10T11:33:11.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.160 7fdf4161a700 1 -- 172.21.2.112:0/756558429 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf3c00b730 con 0x7fdf440e5470 2021-02-10T11:33:11.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.160 7fdf4161a700 1 -- 172.21.2.112:0/756558429 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdf3c00bb90 con 0x7fdf440e5470 2021-02-10T11:33:11.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.160 7fdf4895d700 1 -- 172.21.2.112:0/756558429 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdf440e5470 msgr2=0x7fdf4407dcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:11.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.160 7fdf4895d700 1 --2- 172.21.2.112:0/756558429 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdf440e5470 0x7fdf4407dcb0 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 rx=0x7fdf3c006f00 tx=0x7fdf3c009f10).stop 2021-02-10T11:33:11.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.161 7fdf4895d700 1 -- 172.21.2.112:0/756558429 shutdown_connections 2021-02-10T11:33:11.164 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.161 7fdf4895d700 1 --2- 172.21.2.112:0/756558429 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdf440e5470 0x7fdf4407dcb0 unknown :-1 s=CLOSED pgs=1197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:11.164 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.161 7fdf4895d700 1 --2- 172.21.2.112:0/756558429 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdf441b3e40 0x7fdf4407d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:11.164 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.161 7fdf4895d700 1 --2- 172.21.2.112:0/756558429 >> v2:172.21.2.120:3300/0 conn(0x7fdf440e5070 0x7fdf441b3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:11.164 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.161 7fdf4895d700 1 -- 172.21.2.112:0/756558429 >> 172.21.2.112:0/756558429 conn(0x7fdf440ce690 msgr2=0x7fdf440e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:11.164 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.161 7fdf4895d700 1 -- 172.21.2.112:0/756558429 shutdown_connections 2021-02-10T11:33:11.164 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.161 7fdf4895d700 1 -- 172.21.2.112:0/756558429 wait complete. 2021-02-10T11:33:11.165 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.175 7fdf4895d700 1 Processor -- start 2021-02-10T11:33:11.165 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.175 7fdf4895d700 1 -- start start 2021-02-10T11:33:11.165 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.176 7fdf4895d700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fdf4418c700 0x7fdf44096c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:11.166 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.176 7fdf4895d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdf440ae400 0x7fdf440ae800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:11.166 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.176 7fdf4895d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdf44192380 0x7fdf44192780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:11.166 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.176 7fdf4895d700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf44097290 con 0x7fdf440ae400 2021-02-10T11:33:11.166 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.176 7fdf4895d700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf44192da0 con 0x7fdf4418c700 2021-02-10T11:33:11.166 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.176 7fdf4895d700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf44192f80 con 0x7fdf44192380 2021-02-10T11:33:11.166 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.176 7fdf41e1b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdf440ae400 0x7fdf440ae800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:11.166 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.176 7fdf41e1b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdf440ae400 0x7fdf440ae800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:49476/0 (socket says 172.21.2.112:49476) 2021-02-10T11:33:11.166 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.176 7fdf41e1b700 1 -- 172.21.2.112:0/2656646029 learned_addr learned my addr 172.21.2.112:0/2656646029 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:33:11.166 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.176 7fdf41e1b700 1 -- 172.21.2.112:0/2656646029 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdf44192380 msgr2=0x7fdf44192780 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:33:11.166 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.176 7fdf41e1b700 1 --2- 172.21.2.112:0/2656646029 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdf44192380 0x7fdf44192780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:11.166 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.176 7fdf41e1b700 1 -- 172.21.2.112:0/2656646029 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fdf4418c700 msgr2=0x7fdf44096c70 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:33:11.166 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.176 7fdf41e1b700 1 --2- 172.21.2.112:0/2656646029 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fdf4418c700 0x7fdf44096c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:11.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.176 7fdf41e1b700 1 -- 172.21.2.112:0/2656646029 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf441931a0 con 0x7fdf440ae400 2021-02-10T11:33:11.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.176 7fdf41e1b700 1 --2- 172.21.2.112:0/2656646029 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdf440ae400 0x7fdf440ae800 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 rx=0x7fdf3800b5c0 tx=0x7fdf3800b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:11.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.176 7fdf32ffd700 1 -- 172.21.2.112:0/2656646029 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdf3800bd50 con 0x7fdf440ae400 2021-02-10T11:33:11.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.176 7fdf32ffd700 1 -- 172.21.2.112:0/2656646029 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf38006030 con 0x7fdf440ae400 2021-02-10T11:33:11.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.176 7fdf4895d700 1 -- 172.21.2.112:0/2656646029 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf441931a0 con 0x7fdf440ae400 2021-02-10T11:33:11.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.176 7fdf4895d700 1 -- 172.21.2.112:0/2656646029 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf441931a0 con 0x7fdf440ae400 2021-02-10T11:33:11.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.176 7fdf32ffd700 1 -- 172.21.2.112:0/2656646029 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdf380054d0 con 0x7fdf440ae400 2021-02-10T11:33:11.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.177 7fdf32ffd700 1 -- 172.21.2.112:0/2656646029 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fdf38006210 con 0x7fdf440ae400 2021-02-10T11:33:11.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.177 7fdf32ffd700 1 --2- 172.21.2.112:0/2656646029 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fdf28018730 0x7fdf2801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:11.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.177 7fdf32ffd700 1 -- 172.21.2.112:0/2656646029 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fdf380327c0 con 0x7fdf440ae400 2021-02-10T11:33:11.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.178 7fdf4261c700 1 --2- 172.21.2.112:0/2656646029 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fdf28018730 0x7fdf2801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:11.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.178 7fdf4261c700 1 --2- 172.21.2.112:0/2656646029 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fdf28018730 0x7fdf2801ac10 crc :-1 s=READY pgs=601 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:11.196 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.209 7fdf4895d700 1 -- 172.21.2.112:0/2656646029 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf44107120 con 0x7fdf440ae400 2021-02-10T11:33:11.198 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.211 7fdf32ffd700 1 -- 172.21.2.112:0/2656646029 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fdf38006b80 con 0x7fdf440ae400 2021-02-10T11:33:11.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.335 7fdf4895d700 1 -- 172.21.2.112:0/2656646029 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7fdf44107120 con 0x7fdf440ae400 2021-02-10T11:33:11.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.336 7fdf32ffd700 1 -- 172.21.2.112:0/2656646029 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v46) v1 ==== 81+0+30360 (secure 0 0 0) 0x7fdf38006030 con 0x7fdf440ae400 2021-02-10T11:33:11.324 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:33:11.324 INFO:teuthology.orchestra.run.gibba012.stdout:{ 2021-02-10T11:33:11.324 INFO:teuthology.orchestra.run.gibba012.stdout: "epoch": 46, 2021-02-10T11:33:11.324 INFO:teuthology.orchestra.run.gibba012.stdout: "fsid": "f14f004e-0699-4584-ad81-3754258493de", 2021-02-10T11:33:11.324 INFO:teuthology.orchestra.run.gibba012.stdout: "created": "2021-02-10 11:13:26.501424", 2021-02-10T11:33:11.324 INFO:teuthology.orchestra.run.gibba012.stdout: "modified": "2021-02-10 11:32:32.871643", 2021-02-10T11:33:11.324 INFO:teuthology.orchestra.run.gibba012.stdout: "last_up_change": "2021-02-10 11:13:52.780316", 2021-02-10T11:33:11.324 INFO:teuthology.orchestra.run.gibba012.stdout: "last_in_change": "2021-02-10 11:13:52.780316", 2021-02-10T11:33:11.325 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-02-10T11:33:11.325 INFO:teuthology.orchestra.run.gibba012.stdout: "flags_num": 5799936, 2021-02-10T11:33:11.325 INFO:teuthology.orchestra.run.gibba012.stdout: "flags_set": [ 2021-02-10T11:33:11.325 INFO:teuthology.orchestra.run.gibba012.stdout: "pglog_hardlimit", 2021-02-10T11:33:11.325 INFO:teuthology.orchestra.run.gibba012.stdout: "purged_snapdirs", 2021-02-10T11:33:11.325 INFO:teuthology.orchestra.run.gibba012.stdout: "recovery_deletes", 2021-02-10T11:33:11.325 INFO:teuthology.orchestra.run.gibba012.stdout: "sortbitwise" 2021-02-10T11:33:11.325 INFO:teuthology.orchestra.run.gibba012.stdout: ], 2021-02-10T11:33:11.325 INFO:teuthology.orchestra.run.gibba012.stdout: "crush_version": 4, 2021-02-10T11:33:11.325 INFO:teuthology.orchestra.run.gibba012.stdout: "full_ratio": 0.94999998807907104, 2021-02-10T11:33:11.325 INFO:teuthology.orchestra.run.gibba012.stdout: "backfillfull_ratio": 0.89999997615814209, 2021-02-10T11:33:11.325 INFO:teuthology.orchestra.run.gibba012.stdout: "nearfull_ratio": 0.85000002384185791, 2021-02-10T11:33:11.325 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_snapshot": "", 2021-02-10T11:33:11.326 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_max": 9, 2021-02-10T11:33:11.326 INFO:teuthology.orchestra.run.gibba012.stdout: "max_osd": 8, 2021-02-10T11:33:11.326 INFO:teuthology.orchestra.run.gibba012.stdout: "require_min_compat_client": "jewel", 2021-02-10T11:33:11.326 INFO:teuthology.orchestra.run.gibba012.stdout: "min_compat_client": "jewel", 2021-02-10T11:33:11.326 INFO:teuthology.orchestra.run.gibba012.stdout: "require_osd_release": "nautilus", 2021-02-10T11:33:11.326 INFO:teuthology.orchestra.run.gibba012.stdout: "pools": [ 2021-02-10T11:33:11.326 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:11.326 INFO:teuthology.orchestra.run.gibba012.stdout: "pool": 1, 2021-02-10T11:33:11.326 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_name": "rbd", 2021-02-10T11:33:11.326 INFO:teuthology.orchestra.run.gibba012.stdout: "create_time": "2021-02-10 11:13:57.508758", 2021-02-10T11:33:11.326 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:33:11.326 INFO:teuthology.orchestra.run.gibba012.stdout: "flags_names": "hashpspool", 2021-02-10T11:33:11.326 INFO:teuthology.orchestra.run.gibba012.stdout: "type": 1, 2021-02-10T11:33:11.327 INFO:teuthology.orchestra.run.gibba012.stdout: "size": 2, 2021-02-10T11:33:11.327 INFO:teuthology.orchestra.run.gibba012.stdout: "min_size": 1, 2021-02-10T11:33:11.327 INFO:teuthology.orchestra.run.gibba012.stdout: "crush_rule": 0, 2021-02-10T11:33:11.327 INFO:teuthology.orchestra.run.gibba012.stdout: "object_hash": 2, 2021-02-10T11:33:11.327 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:33:11.327 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num": 8, 2021-02-10T11:33:11.327 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num": 8, 2021-02-10T11:33:11.327 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num_target": 8, 2021-02-10T11:33:11.327 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_target": 8, 2021-02-10T11:33:11.327 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_pending": 8, 2021-02-10T11:33:11.327 INFO:teuthology.orchestra.run.gibba012.stdout: "last_pg_merge_meta": { 2021-02-10T11:33:11.327 INFO:teuthology.orchestra.run.gibba012.stdout: "source_pgid": "0.0", 2021-02-10T11:33:11.327 INFO:teuthology.orchestra.run.gibba012.stdout: "ready_epoch": 0, 2021-02-10T11:33:11.327 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_started": 0, 2021-02-10T11:33:11.328 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_clean": 0, 2021-02-10T11:33:11.328 INFO:teuthology.orchestra.run.gibba012.stdout: "source_version": "0'0", 2021-02-10T11:33:11.328 INFO:teuthology.orchestra.run.gibba012.stdout: "target_version": "0'0" 2021-02-10T11:33:11.328 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:11.328 INFO:teuthology.orchestra.run.gibba012.stdout: "last_change": "16", 2021-02-10T11:33:11.328 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend": "0", 2021-02-10T11:33:11.328 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:33:11.328 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:33:11.328 INFO:teuthology.orchestra.run.gibba012.stdout: "auid": 0, 2021-02-10T11:33:11.328 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:33:11.328 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_seq": 0, 2021-02-10T11:33:11.328 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_epoch": 0, 2021-02-10T11:33:11.328 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_snaps": [], 2021-02-10T11:33:11.329 INFO:teuthology.orchestra.run.gibba012.stdout: "removed_snaps": "[]", 2021-02-10T11:33:11.329 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_bytes": 0, 2021-02-10T11:33:11.329 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_objects": 0, 2021-02-10T11:33:11.329 INFO:teuthology.orchestra.run.gibba012.stdout: "tiers": [], 2021-02-10T11:33:11.329 INFO:teuthology.orchestra.run.gibba012.stdout: "tier_of": -1, 2021-02-10T11:33:11.329 INFO:teuthology.orchestra.run.gibba012.stdout: "read_tier": -1, 2021-02-10T11:33:11.329 INFO:teuthology.orchestra.run.gibba012.stdout: "write_tier": -1, 2021-02-10T11:33:11.329 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_mode": "none", 2021-02-10T11:33:11.329 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_bytes": 0, 2021-02-10T11:33:11.329 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_objects": 0, 2021-02-10T11:33:11.329 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:33:11.329 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:33:11.329 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:33:11.329 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_flush_age": 0, 2021-02-10T11:33:11.330 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_evict_age": 0, 2021-02-10T11:33:11.330 INFO:teuthology.orchestra.run.gibba012.stdout: "erasure_code_profile": "", 2021-02-10T11:33:11.330 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_params": { 2021-02-10T11:33:11.330 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "none" 2021-02-10T11:33:11.330 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:11.330 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_period": 0, 2021-02-10T11:33:11.330 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_count": 0, 2021-02-10T11:33:11.330 INFO:teuthology.orchestra.run.gibba012.stdout: "use_gmt_hitset": true, 2021-02-10T11:33:11.330 INFO:teuthology.orchestra.run.gibba012.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:33:11.330 INFO:teuthology.orchestra.run.gibba012.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:33:11.330 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:33:11.330 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:33:11.331 INFO:teuthology.orchestra.run.gibba012.stdout: "grade_table": [], 2021-02-10T11:33:11.331 INFO:teuthology.orchestra.run.gibba012.stdout: "stripe_width": 0, 2021-02-10T11:33:11.331 INFO:teuthology.orchestra.run.gibba012.stdout: "expected_num_objects": 0, 2021-02-10T11:33:11.331 INFO:teuthology.orchestra.run.gibba012.stdout: "fast_read": false, 2021-02-10T11:33:11.331 INFO:teuthology.orchestra.run.gibba012.stdout: "options": {}, 2021-02-10T11:33:11.331 INFO:teuthology.orchestra.run.gibba012.stdout: "application_metadata": { 2021-02-10T11:33:11.331 INFO:teuthology.orchestra.run.gibba012.stdout: "rbd": {} 2021-02-10T11:33:11.331 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:33:11.331 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:11.331 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:11.331 INFO:teuthology.orchestra.run.gibba012.stdout: "pool": 8, 2021-02-10T11:33:11.332 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_name": "cephfs_metadata", 2021-02-10T11:33:11.332 INFO:teuthology.orchestra.run.gibba012.stdout: "create_time": "2021-02-10 11:32:29.843136", 2021-02-10T11:33:11.332 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:33:11.332 INFO:teuthology.orchestra.run.gibba012.stdout: "flags_names": "hashpspool", 2021-02-10T11:33:11.332 INFO:teuthology.orchestra.run.gibba012.stdout: "type": 1, 2021-02-10T11:33:11.332 INFO:teuthology.orchestra.run.gibba012.stdout: "size": 2, 2021-02-10T11:33:11.332 INFO:teuthology.orchestra.run.gibba012.stdout: "min_size": 1, 2021-02-10T11:33:11.332 INFO:teuthology.orchestra.run.gibba012.stdout: "crush_rule": 0, 2021-02-10T11:33:11.332 INFO:teuthology.orchestra.run.gibba012.stdout: "object_hash": 2, 2021-02-10T11:33:11.332 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:33:11.332 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num": 8, 2021-02-10T11:33:11.332 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num": 8, 2021-02-10T11:33:11.332 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num_target": 8, 2021-02-10T11:33:11.332 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_target": 8, 2021-02-10T11:33:11.333 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_pending": 8, 2021-02-10T11:33:11.333 INFO:teuthology.orchestra.run.gibba012.stdout: "last_pg_merge_meta": { 2021-02-10T11:33:11.333 INFO:teuthology.orchestra.run.gibba012.stdout: "source_pgid": "0.0", 2021-02-10T11:33:11.333 INFO:teuthology.orchestra.run.gibba012.stdout: "ready_epoch": 0, 2021-02-10T11:33:11.333 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_started": 0, 2021-02-10T11:33:11.333 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_clean": 0, 2021-02-10T11:33:11.333 INFO:teuthology.orchestra.run.gibba012.stdout: "source_version": "0'0", 2021-02-10T11:33:11.333 INFO:teuthology.orchestra.run.gibba012.stdout: "target_version": "0'0" 2021-02-10T11:33:11.333 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:11.333 INFO:teuthology.orchestra.run.gibba012.stdout: "last_change": "45", 2021-02-10T11:33:11.333 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend": "0", 2021-02-10T11:33:11.333 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:33:11.333 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:33:11.333 INFO:teuthology.orchestra.run.gibba012.stdout: "auid": 0, 2021-02-10T11:33:11.334 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:33:11.334 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_seq": 0, 2021-02-10T11:33:11.334 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_epoch": 0, 2021-02-10T11:33:11.334 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_snaps": [], 2021-02-10T11:33:11.334 INFO:teuthology.orchestra.run.gibba012.stdout: "removed_snaps": "[]", 2021-02-10T11:33:11.334 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_bytes": 0, 2021-02-10T11:33:11.334 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_objects": 0, 2021-02-10T11:33:11.334 INFO:teuthology.orchestra.run.gibba012.stdout: "tiers": [], 2021-02-10T11:33:11.334 INFO:teuthology.orchestra.run.gibba012.stdout: "tier_of": -1, 2021-02-10T11:33:11.334 INFO:teuthology.orchestra.run.gibba012.stdout: "read_tier": -1, 2021-02-10T11:33:11.334 INFO:teuthology.orchestra.run.gibba012.stdout: "write_tier": -1, 2021-02-10T11:33:11.334 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_mode": "none", 2021-02-10T11:33:11.335 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_bytes": 0, 2021-02-10T11:33:11.335 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_objects": 0, 2021-02-10T11:33:11.335 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:33:11.335 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:33:11.335 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:33:11.335 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_flush_age": 0, 2021-02-10T11:33:11.335 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_evict_age": 0, 2021-02-10T11:33:11.335 INFO:teuthology.orchestra.run.gibba012.stdout: "erasure_code_profile": "", 2021-02-10T11:33:11.335 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_params": { 2021-02-10T11:33:11.335 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "none" 2021-02-10T11:33:11.335 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:11.335 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_period": 0, 2021-02-10T11:33:11.336 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_count": 0, 2021-02-10T11:33:11.336 INFO:teuthology.orchestra.run.gibba012.stdout: "use_gmt_hitset": true, 2021-02-10T11:33:11.336 INFO:teuthology.orchestra.run.gibba012.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:33:11.336 INFO:teuthology.orchestra.run.gibba012.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:33:11.336 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:33:11.336 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:33:11.336 INFO:teuthology.orchestra.run.gibba012.stdout: "grade_table": [], 2021-02-10T11:33:11.336 INFO:teuthology.orchestra.run.gibba012.stdout: "stripe_width": 0, 2021-02-10T11:33:11.336 INFO:teuthology.orchestra.run.gibba012.stdout: "expected_num_objects": 0, 2021-02-10T11:33:11.336 INFO:teuthology.orchestra.run.gibba012.stdout: "fast_read": false, 2021-02-10T11:33:11.336 INFO:teuthology.orchestra.run.gibba012.stdout: "options": { 2021-02-10T11:33:11.336 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_autoscale_bias": 4, 2021-02-10T11:33:11.336 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_min": 16, 2021-02-10T11:33:11.337 INFO:teuthology.orchestra.run.gibba012.stdout: "recovery_priority": 5 2021-02-10T11:33:11.337 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:11.337 INFO:teuthology.orchestra.run.gibba012.stdout: "application_metadata": { 2021-02-10T11:33:11.337 INFO:teuthology.orchestra.run.gibba012.stdout: "cephfs": { 2021-02-10T11:33:11.337 INFO:teuthology.orchestra.run.gibba012.stdout: "metadata": "cephfs" 2021-02-10T11:33:11.338 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:33:11.338 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:33:11.338 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:11.338 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:11.338 INFO:teuthology.orchestra.run.gibba012.stdout: "pool": 9, 2021-02-10T11:33:11.338 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_name": "cephfs_data", 2021-02-10T11:33:11.338 INFO:teuthology.orchestra.run.gibba012.stdout: "create_time": "2021-02-10 11:32:30.914946", 2021-02-10T11:33:11.338 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:33:11.339 INFO:teuthology.orchestra.run.gibba012.stdout: "flags_names": "hashpspool", 2021-02-10T11:33:11.339 INFO:teuthology.orchestra.run.gibba012.stdout: "type": 1, 2021-02-10T11:33:11.339 INFO:teuthology.orchestra.run.gibba012.stdout: "size": 2, 2021-02-10T11:33:11.339 INFO:teuthology.orchestra.run.gibba012.stdout: "min_size": 1, 2021-02-10T11:33:11.339 INFO:teuthology.orchestra.run.gibba012.stdout: "crush_rule": 0, 2021-02-10T11:33:11.339 INFO:teuthology.orchestra.run.gibba012.stdout: "object_hash": 2, 2021-02-10T11:33:11.339 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:33:11.339 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num": 8, 2021-02-10T11:33:11.339 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num": 8, 2021-02-10T11:33:11.339 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num_target": 8, 2021-02-10T11:33:11.339 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_target": 8, 2021-02-10T11:33:11.339 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_pending": 8, 2021-02-10T11:33:11.339 INFO:teuthology.orchestra.run.gibba012.stdout: "last_pg_merge_meta": { 2021-02-10T11:33:11.340 INFO:teuthology.orchestra.run.gibba012.stdout: "source_pgid": "0.0", 2021-02-10T11:33:11.340 INFO:teuthology.orchestra.run.gibba012.stdout: "ready_epoch": 0, 2021-02-10T11:33:11.340 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_started": 0, 2021-02-10T11:33:11.340 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_clean": 0, 2021-02-10T11:33:11.340 INFO:teuthology.orchestra.run.gibba012.stdout: "source_version": "0'0", 2021-02-10T11:33:11.340 INFO:teuthology.orchestra.run.gibba012.stdout: "target_version": "0'0" 2021-02-10T11:33:11.340 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:11.340 INFO:teuthology.orchestra.run.gibba012.stdout: "last_change": "46", 2021-02-10T11:33:11.340 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend": "0", 2021-02-10T11:33:11.340 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:33:11.341 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:33:11.341 INFO:teuthology.orchestra.run.gibba012.stdout: "auid": 0, 2021-02-10T11:33:11.341 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:33:11.341 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_seq": 0, 2021-02-10T11:33:11.341 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_epoch": 0, 2021-02-10T11:33:11.341 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_snaps": [], 2021-02-10T11:33:11.341 INFO:teuthology.orchestra.run.gibba012.stdout: "removed_snaps": "[]", 2021-02-10T11:33:11.341 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_bytes": 0, 2021-02-10T11:33:11.341 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_objects": 0, 2021-02-10T11:33:11.341 INFO:teuthology.orchestra.run.gibba012.stdout: "tiers": [], 2021-02-10T11:33:11.341 INFO:teuthology.orchestra.run.gibba012.stdout: "tier_of": -1, 2021-02-10T11:33:11.341 INFO:teuthology.orchestra.run.gibba012.stdout: "read_tier": -1, 2021-02-10T11:33:11.341 INFO:teuthology.orchestra.run.gibba012.stdout: "write_tier": -1, 2021-02-10T11:33:11.342 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_mode": "none", 2021-02-10T11:33:11.342 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_bytes": 0, 2021-02-10T11:33:11.342 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_objects": 0, 2021-02-10T11:33:11.342 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:33:11.342 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:33:11.342 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:33:11.342 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_flush_age": 0, 2021-02-10T11:33:11.342 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_evict_age": 0, 2021-02-10T11:33:11.342 INFO:teuthology.orchestra.run.gibba012.stdout: "erasure_code_profile": "", 2021-02-10T11:33:11.342 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_params": { 2021-02-10T11:33:11.342 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "none" 2021-02-10T11:33:11.342 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:11.343 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_period": 0, 2021-02-10T11:33:11.343 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_count": 0, 2021-02-10T11:33:11.343 INFO:teuthology.orchestra.run.gibba012.stdout: "use_gmt_hitset": true, 2021-02-10T11:33:11.343 INFO:teuthology.orchestra.run.gibba012.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:33:11.343 INFO:teuthology.orchestra.run.gibba012.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:33:11.343 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:33:11.343 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:33:11.343 INFO:teuthology.orchestra.run.gibba012.stdout: "grade_table": [], 2021-02-10T11:33:11.343 INFO:teuthology.orchestra.run.gibba012.stdout: "stripe_width": 0, 2021-02-10T11:33:11.343 INFO:teuthology.orchestra.run.gibba012.stdout: "expected_num_objects": 0, 2021-02-10T11:33:11.343 INFO:teuthology.orchestra.run.gibba012.stdout: "fast_read": false, 2021-02-10T11:33:11.343 INFO:teuthology.orchestra.run.gibba012.stdout: "options": {}, 2021-02-10T11:33:11.343 INFO:teuthology.orchestra.run.gibba012.stdout: "application_metadata": { 2021-02-10T11:33:11.344 INFO:teuthology.orchestra.run.gibba012.stdout: "cephfs": { 2021-02-10T11:33:11.344 INFO:teuthology.orchestra.run.gibba012.stdout: "data": "cephfs" 2021-02-10T11:33:11.344 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:33:11.344 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:33:11.344 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:33:11.344 INFO:teuthology.orchestra.run.gibba012.stdout: ], 2021-02-10T11:33:11.344 INFO:teuthology.orchestra.run.gibba012.stdout: "osds": [ 2021-02-10T11:33:11.344 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:11.344 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 0, 2021-02-10T11:33:11.344 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "95444b8f-6370-4a1d-ae5f-df62651b77da", 2021-02-10T11:33:11.344 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:33:11.345 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:33:11.345 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:33:11.345 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:33:11.345 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:33:11.345 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:33:11.345 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:33:11.345 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 44, 2021-02-10T11:33:11.345 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:33:11.345 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:33:11.345 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:33:11.345 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:33:11.345 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:11.345 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:33:11.346 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6817", 2021-02-10T11:33:11.346 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:33:11.346 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:11.346 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:11.346 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:33:11.346 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6818", 2021-02-10T11:33:11.346 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:33:11.346 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:33:11.346 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:33:11.346 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:11.346 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:33:11.347 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:33:11.347 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:11.347 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:33:11.347 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6820", 2021-02-10T11:33:11.347 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:33:11.347 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:11.347 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:11.347 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:33:11.348 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6822", 2021-02-10T11:33:11.348 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:33:11.348 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:33:11.348 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:33:11.348 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:11.348 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:33:11.348 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:33:11.348 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:11.348 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:33:11.349 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6828", 2021-02-10T11:33:11.349 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:33:11.349 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:11.349 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:11.349 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:33:11.349 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6830", 2021-02-10T11:33:11.349 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:33:11.349 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:33:11.349 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:33:11.349 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:11.350 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:33:11.350 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:33:11.350 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:11.350 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:33:11.350 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6824", 2021-02-10T11:33:11.350 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:33:11.350 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:11.350 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:11.350 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:33:11.353 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6826", 2021-02-10T11:33:11.353 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:33:11.353 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:33:11.353 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:33:11.353 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:11.353 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.112:6818/36030", 2021-02-10T11:33:11.353 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.112:6822/36030", 2021-02-10T11:33:11.354 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.112:6830/36030", 2021-02-10T11:33:11.354 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.112:6826/36030", 2021-02-10T11:33:11.354 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:33:11.354 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:33:11.354 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:33:11.354 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:33:11.354 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:11.354 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:11.354 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 1, 2021-02-10T11:33:11.354 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "a23e217e-0d8e-486e-be07-712adfabd262", 2021-02-10T11:33:11.354 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:33:11.354 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:33:11.355 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:33:11.355 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:33:11.355 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:33:11.355 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:33:11.355 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:33:11.355 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 44, 2021-02-10T11:33:11.355 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:33:11.355 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:33:11.355 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:33:11.355 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:33:11.355 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:11.355 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:33:11.356 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6800", 2021-02-10T11:33:11.356 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:33:11.356 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:11.356 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:11.356 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:33:11.356 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6802", 2021-02-10T11:33:11.356 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:33:11.356 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:33:11.356 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:33:11.356 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:11.356 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:33:11.356 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:33:11.356 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:11.356 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:33:11.357 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6804", 2021-02-10T11:33:11.357 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:33:11.357 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:11.357 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:11.357 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:33:11.357 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6806", 2021-02-10T11:33:11.357 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:33:11.357 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:33:11.357 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:33:11.357 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:11.357 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:33:11.357 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:33:11.358 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:11.358 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:33:11.358 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6812", 2021-02-10T11:33:11.358 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:33:11.358 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:11.358 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:11.358 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:33:11.358 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6814", 2021-02-10T11:33:11.358 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:33:11.358 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:33:11.358 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:33:11.358 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:11.359 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:33:11.359 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:33:11.359 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:11.359 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:33:11.359 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6808", 2021-02-10T11:33:11.359 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:33:11.359 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:11.359 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:11.359 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:33:11.359 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6810", 2021-02-10T11:33:11.359 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:33:11.359 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:33:11.359 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:33:11.360 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:11.360 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.112:6802/36029", 2021-02-10T11:33:11.360 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.112:6806/36029", 2021-02-10T11:33:11.360 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.112:6814/36029", 2021-02-10T11:33:11.360 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.112:6810/36029", 2021-02-10T11:33:11.360 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:33:11.360 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:33:11.360 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:33:11.360 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:33:11.360 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:11.360 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:11.360 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 2, 2021-02-10T11:33:11.360 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "026afa8a-c0f1-4b35-9a76-53c00fe0c2fc", 2021-02-10T11:33:11.360 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:33:11.361 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:33:11.361 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:33:11.361 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:33:11.361 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:33:11.361 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:33:11.361 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:33:11.361 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 27, 2021-02-10T11:33:11.361 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:33:11.361 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:33:11.361 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:33:11.361 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:33:11.361 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:11.361 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:33:11.362 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6801", 2021-02-10T11:33:11.362 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:33:11.362 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:11.362 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:11.362 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:33:11.362 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6803", 2021-02-10T11:33:11.362 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:33:11.362 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:33:11.362 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:33:11.362 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:11.362 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:33:11.362 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:33:11.363 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:11.363 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:33:11.363 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6805", 2021-02-10T11:33:11.363 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:33:11.363 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:11.363 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:11.363 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:33:11.363 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6807", 2021-02-10T11:33:11.363 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:33:11.363 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:33:11.363 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:33:11.363 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:11.363 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:33:11.364 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:33:11.364 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:11.364 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:33:11.364 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6813", 2021-02-10T11:33:11.364 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:33:11.364 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:11.364 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:11.364 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:33:11.364 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6815", 2021-02-10T11:33:11.364 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:33:11.364 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:33:11.364 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:33:11.364 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:11.365 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:33:11.365 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:33:11.365 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:11.365 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:33:11.365 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6809", 2021-02-10T11:33:11.365 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:33:11.365 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:11.365 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:11.365 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:33:11.365 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6811", 2021-02-10T11:33:11.365 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:33:11.365 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:33:11.366 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:33:11.366 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:11.366 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.112:6803/36031", 2021-02-10T11:33:11.366 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.112:6807/36031", 2021-02-10T11:33:11.366 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.112:6815/36031", 2021-02-10T11:33:11.366 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.112:6811/36031", 2021-02-10T11:33:11.366 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:33:11.366 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:33:11.366 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:33:11.366 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:33:11.367 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:11.367 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:11.367 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 3, 2021-02-10T11:33:11.367 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "58795682-e00d-4173-bb84-f7075ca89a11", 2021-02-10T11:33:11.367 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:33:11.367 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:33:11.367 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:33:11.367 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:33:11.367 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:33:11.367 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:33:11.367 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:33:11.367 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 44, 2021-02-10T11:33:11.368 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:33:11.368 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:33:11.368 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:33:11.368 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:33:11.368 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:11.368 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:33:11.368 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6816", 2021-02-10T11:33:11.368 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:33:11.368 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:11.368 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:11.368 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:33:11.368 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6819", 2021-02-10T11:33:11.368 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:33:11.369 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:33:11.369 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:33:11.369 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:11.369 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:33:11.369 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:33:11.369 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:11.369 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:33:11.370 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6821", 2021-02-10T11:33:11.370 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:33:11.370 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:11.370 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:11.370 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:33:11.370 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6823", 2021-02-10T11:33:11.370 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:33:11.370 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:33:11.370 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:33:11.371 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:11.371 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:33:11.371 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:33:11.371 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:11.371 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:33:11.371 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6829", 2021-02-10T11:33:11.371 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:33:11.371 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:11.371 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:11.371 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:33:11.371 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6831", 2021-02-10T11:33:11.371 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:33:11.372 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:33:11.372 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:33:11.372 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:11.372 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:33:11.372 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:33:11.372 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:11.372 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:33:11.372 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6825", 2021-02-10T11:33:11.372 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:33:11.372 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:11.373 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:11.373 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:33:11.373 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6827", 2021-02-10T11:33:11.373 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:33:11.373 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:33:11.373 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:33:11.373 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:11.373 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.112:6819/36028", 2021-02-10T11:33:11.373 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.112:6823/36028", 2021-02-10T11:33:11.373 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.112:6831/36028", 2021-02-10T11:33:11.373 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.112:6827/36028", 2021-02-10T11:33:11.373 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:33:11.373 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:33:11.374 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:33:11.374 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:33:11.374 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:11.374 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:11.374 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 4, 2021-02-10T11:33:11.374 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "6763f8d6-53dc-4e45-bdf6-a6eabcf81e39", 2021-02-10T11:33:11.374 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:33:11.374 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:33:11.374 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:33:11.374 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:33:11.374 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:33:11.375 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:33:11.375 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:33:11.375 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 43, 2021-02-10T11:33:11.375 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:33:11.375 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:33:11.375 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:33:11.375 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:33:11.375 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:11.375 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:33:11.375 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6810", 2021-02-10T11:33:11.375 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:33:11.375 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:11.376 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:11.376 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:33:11.376 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6813", 2021-02-10T11:33:11.376 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:33:11.376 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:33:11.376 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:33:11.376 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:11.376 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:33:11.376 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:33:11.376 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:11.376 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:33:11.376 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6816", 2021-02-10T11:33:11.377 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:33:11.377 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:11.377 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:11.377 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:33:11.377 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6819", 2021-02-10T11:33:11.377 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:33:11.377 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:33:11.377 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:33:11.377 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:11.377 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:33:11.377 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:33:11.377 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:11.378 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:33:11.378 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6825", 2021-02-10T11:33:11.378 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:33:11.378 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:11.378 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:11.378 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:33:11.378 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6827", 2021-02-10T11:33:11.378 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:33:11.378 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:33:11.378 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:33:11.378 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:11.379 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:33:11.379 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:33:11.379 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:11.379 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:33:11.379 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6821", 2021-02-10T11:33:11.379 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:33:11.379 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:11.379 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:11.379 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:33:11.379 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6823", 2021-02-10T11:33:11.379 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:33:11.379 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:33:11.379 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:33:11.379 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:11.380 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.120:6813/36388", 2021-02-10T11:33:11.380 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.120:6819/36388", 2021-02-10T11:33:11.380 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.120:6827/36388", 2021-02-10T11:33:11.380 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.120:6823/36388", 2021-02-10T11:33:11.380 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:33:11.380 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:33:11.380 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:33:11.380 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:33:11.380 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:11.380 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:11.380 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 5, 2021-02-10T11:33:11.380 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "5d0e4dd6-0cb0-4bf8-8314-626acaa1b544", 2021-02-10T11:33:11.380 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:33:11.381 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:33:11.381 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:33:11.381 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:33:11.381 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:33:11.381 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:33:11.381 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:33:11.381 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 44, 2021-02-10T11:33:11.381 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:33:11.381 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:33:11.381 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:33:11.381 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:33:11.381 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:11.382 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:33:11.382 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6804", 2021-02-10T11:33:11.382 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:33:11.382 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:11.382 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:11.382 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:33:11.382 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6806", 2021-02-10T11:33:11.382 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:33:11.382 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:33:11.382 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:33:11.382 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:11.382 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:33:11.383 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:33:11.383 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:11.383 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:33:11.383 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6809", 2021-02-10T11:33:11.383 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:33:11.383 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:11.383 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:11.383 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:33:11.383 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6812", 2021-02-10T11:33:11.383 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:33:11.383 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:33:11.383 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:33:11.383 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:11.384 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:33:11.384 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:33:11.384 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:11.384 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:33:11.384 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6820", 2021-02-10T11:33:11.384 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:33:11.384 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:11.384 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:11.384 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:33:11.384 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6822", 2021-02-10T11:33:11.384 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:33:11.384 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:33:11.384 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:33:11.384 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:11.385 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:33:11.385 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:33:11.385 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:11.385 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:33:11.385 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6815", 2021-02-10T11:33:11.385 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:33:11.385 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:11.385 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:11.385 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:33:11.385 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6818", 2021-02-10T11:33:11.385 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:33:11.385 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:33:11.385 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:33:11.386 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:11.386 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.120:6806/36387", 2021-02-10T11:33:11.386 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.120:6812/36387", 2021-02-10T11:33:11.386 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.120:6822/36387", 2021-02-10T11:33:11.386 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.120:6818/36387", 2021-02-10T11:33:11.386 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:33:11.386 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:33:11.386 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:33:11.386 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:33:11.386 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:11.386 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:11.386 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 6, 2021-02-10T11:33:11.387 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "b010e187-4541-4b00-80dd-f948f6ab8f15", 2021-02-10T11:33:11.387 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:33:11.387 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:33:11.387 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:33:11.387 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:33:11.387 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:33:11.387 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:33:11.387 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:33:11.387 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 43, 2021-02-10T11:33:11.387 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:33:11.387 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:33:11.387 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:33:11.387 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:33:11.387 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:11.388 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:33:11.388 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6824", 2021-02-10T11:33:11.388 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:33:11.388 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:11.388 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:11.388 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:33:11.388 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6826", 2021-02-10T11:33:11.388 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:33:11.388 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:33:11.388 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:33:11.388 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:11.388 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:33:11.388 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:33:11.389 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:11.389 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:33:11.389 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6828", 2021-02-10T11:33:11.389 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:33:11.389 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:11.389 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:11.389 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:33:11.389 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6829", 2021-02-10T11:33:11.389 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:33:11.389 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:33:11.389 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:33:11.389 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:11.389 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:33:11.389 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:33:11.390 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:11.390 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:33:11.390 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6832", 2021-02-10T11:33:11.390 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:33:11.390 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:11.390 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:11.390 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:33:11.390 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6833", 2021-02-10T11:33:11.390 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:33:11.390 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:33:11.390 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:33:11.390 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:11.391 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:33:11.391 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:33:11.391 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:11.391 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:33:11.391 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6830", 2021-02-10T11:33:11.391 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:33:11.391 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:11.391 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:11.391 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:33:11.391 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6831", 2021-02-10T11:33:11.391 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:33:11.391 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:33:11.392 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:33:11.392 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:11.392 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.120:6826/36385", 2021-02-10T11:33:11.392 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.120:6829/36385", 2021-02-10T11:33:11.392 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.120:6833/36385", 2021-02-10T11:33:11.392 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.120:6831/36385", 2021-02-10T11:33:11.392 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:33:11.392 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:33:11.392 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:33:11.392 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:33:11.392 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:11.392 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:11.392 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 7, 2021-02-10T11:33:11.393 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "671e2772-33b7-497e-8e44-2178bd850bcf", 2021-02-10T11:33:11.393 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:33:11.393 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:33:11.393 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:33:11.393 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:33:11.393 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:33:11.393 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:33:11.393 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:33:11.393 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 44, 2021-02-10T11:33:11.393 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:33:11.393 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:33:11.393 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:33:11.393 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:33:11.394 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:11.394 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:33:11.394 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6802", 2021-02-10T11:33:11.394 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:33:11.394 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:11.394 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:11.394 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:33:11.394 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6803", 2021-02-10T11:33:11.394 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:33:11.394 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:33:11.394 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:33:11.395 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:11.395 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:33:11.395 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:33:11.395 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:11.395 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:33:11.395 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6805", 2021-02-10T11:33:11.395 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:33:11.395 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:11.395 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:11.395 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:33:11.395 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6807", 2021-02-10T11:33:11.396 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:33:11.396 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:33:11.396 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:33:11.396 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:11.396 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:33:11.396 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:33:11.396 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:11.396 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:33:11.396 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6814", 2021-02-10T11:33:11.397 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:33:11.397 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:11.397 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:11.397 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:33:11.397 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6817", 2021-02-10T11:33:11.397 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:33:11.397 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:33:11.397 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:33:11.397 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:11.397 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:33:11.397 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:33:11.397 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:11.397 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:33:11.398 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6808", 2021-02-10T11:33:11.398 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:33:11.398 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:11.398 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:11.398 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:33:11.398 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6811", 2021-02-10T11:33:11.398 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:33:11.398 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:33:11.398 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:33:11.398 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:11.398 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.120:6803/36386", 2021-02-10T11:33:11.399 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.120:6807/36386", 2021-02-10T11:33:11.399 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.120:6817/36386", 2021-02-10T11:33:11.399 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.120:6811/36386", 2021-02-10T11:33:11.399 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:33:11.399 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:33:11.399 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:33:11.399 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:33:11.399 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:33:11.399 INFO:teuthology.orchestra.run.gibba012.stdout: ], 2021-02-10T11:33:11.399 INFO:teuthology.orchestra.run.gibba012.stdout: "osd_xinfo": [ 2021-02-10T11:33:11.399 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:11.399 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 0, 2021-02-10T11:33:11.399 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:33:11.400 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:33:11.400 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:33:11.400 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:33:11.400 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:33:11.400 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:11.400 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:11.400 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 1, 2021-02-10T11:33:11.400 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:33:11.400 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:33:11.400 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:33:11.400 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:33:11.400 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:33:11.401 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:11.401 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:11.401 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 2, 2021-02-10T11:33:11.401 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:33:11.401 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:33:11.401 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:33:11.401 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:33:11.401 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:33:11.401 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:11.401 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:11.401 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 3, 2021-02-10T11:33:11.401 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:33:11.402 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:33:11.402 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:33:11.402 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:33:11.402 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:33:11.402 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:11.402 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:11.402 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 4, 2021-02-10T11:33:11.402 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:33:11.402 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:33:11.402 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:33:11.402 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:33:11.403 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:33:11.403 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:11.403 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:11.403 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 5, 2021-02-10T11:33:11.403 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:33:11.403 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:33:11.403 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:33:11.403 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:33:11.403 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:33:11.403 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:11.403 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:11.403 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 6, 2021-02-10T11:33:11.403 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:33:11.404 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:33:11.404 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:33:11.404 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:33:11.404 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:33:11.404 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:11.404 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:11.404 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 7, 2021-02-10T11:33:11.404 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:33:11.404 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:33:11.404 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:33:11.404 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:33:11.404 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:33:11.404 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:33:11.405 INFO:teuthology.orchestra.run.gibba012.stdout: ], 2021-02-10T11:33:11.405 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_upmap": [], 2021-02-10T11:33:11.405 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_upmap_items": [], 2021-02-10T11:33:11.405 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_temp": [], 2021-02-10T11:33:11.405 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_temp": [], 2021-02-10T11:33:11.405 INFO:teuthology.orchestra.run.gibba012.stdout: "blacklist": {}, 2021-02-10T11:33:11.405 INFO:teuthology.orchestra.run.gibba012.stdout: "erasure_code_profiles": { 2021-02-10T11:33:11.405 INFO:teuthology.orchestra.run.gibba012.stdout: "default": { 2021-02-10T11:33:11.405 INFO:teuthology.orchestra.run.gibba012.stdout: "crush-failure-domain": "osd", 2021-02-10T11:33:11.405 INFO:teuthology.orchestra.run.gibba012.stdout: "k": "2", 2021-02-10T11:33:11.405 INFO:teuthology.orchestra.run.gibba012.stdout: "m": "1", 2021-02-10T11:33:11.405 INFO:teuthology.orchestra.run.gibba012.stdout: "plugin": "jerasure", 2021-02-10T11:33:11.405 INFO:teuthology.orchestra.run.gibba012.stdout: "ruleset-failure-domain": "osd", 2021-02-10T11:33:11.406 INFO:teuthology.orchestra.run.gibba012.stdout: "technique": "reed_sol_van" 2021-02-10T11:33:11.406 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:33:11.406 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:11.406 INFO:teuthology.orchestra.run.gibba012.stdout: "removed_snaps_queue": [], 2021-02-10T11:33:11.406 INFO:teuthology.orchestra.run.gibba012.stdout: "new_removed_snaps": [], 2021-02-10T11:33:11.406 INFO:teuthology.orchestra.run.gibba012.stdout: "new_purged_snaps": [], 2021-02-10T11:33:11.406 INFO:teuthology.orchestra.run.gibba012.stdout: "crush_node_flags": {}, 2021-02-10T11:33:11.406 INFO:teuthology.orchestra.run.gibba012.stdout: "device_class_flags": {} 2021-02-10T11:33:11.406 INFO:teuthology.orchestra.run.gibba012.stdout:} 2021-02-10T11:33:11.408 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.338 7fdf4895d700 1 -- 172.21.2.112:0/2656646029 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fdf28018730 msgr2=0x7fdf2801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:11.408 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.338 7fdf4895d700 1 --2- 172.21.2.112:0/2656646029 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fdf28018730 0x7fdf2801ac10 crc :-1 s=READY pgs=601 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:11.408 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.338 7fdf4895d700 1 -- 172.21.2.112:0/2656646029 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdf440ae400 msgr2=0x7fdf440ae800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:11.408 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.338 7fdf4895d700 1 --2- 172.21.2.112:0/2656646029 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdf440ae400 0x7fdf440ae800 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 rx=0x7fdf3800b5c0 tx=0x7fdf3800b960).stop 2021-02-10T11:33:11.408 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.338 7fdf4895d700 1 -- 172.21.2.112:0/2656646029 shutdown_connections 2021-02-10T11:33:11.408 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.338 7fdf4895d700 1 --2- 172.21.2.112:0/2656646029 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fdf28018730 0x7fdf2801ac10 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:11.408 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.338 7fdf4895d700 1 --2- 172.21.2.112:0/2656646029 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdf44192380 0x7fdf44192780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:11.408 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.338 7fdf4895d700 1 --2- 172.21.2.112:0/2656646029 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdf440ae400 0x7fdf440ae800 unknown :-1 s=CLOSED pgs=1198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:11.408 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.338 7fdf4895d700 1 --2- 172.21.2.112:0/2656646029 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fdf4418c700 0x7fdf44096c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:11.409 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.338 7fdf4895d700 1 -- 172.21.2.112:0/2656646029 >> 172.21.2.112:0/2656646029 conn(0x7fdf440891f0 msgr2=0x7fdf441a0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:11.409 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.338 7fdf4895d700 1 -- 172.21.2.112:0/2656646029 shutdown_connections 2021-02-10T11:33:11.409 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.338 7fdf4895d700 1 -- 172.21.2.112:0/2656646029 wait complete. 2021-02-10T11:33:11.409 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:33:11.836 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.832 7fa35e8f4700 1 Processor -- start 2021-02-10T11:33:11.836 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.833 7fa35e8f4700 1 -- start start 2021-02-10T11:33:11.836 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.833 7fa35e8f4700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fa3580e5070 0x7fa3581b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:11.836 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.833 7fa35e8f4700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa35807ed80 con 0x7fa3580e5470 2021-02-10T11:33:11.836 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.833 7fa35e8f4700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa35807e670 con 0x7fa3581b3e40 2021-02-10T11:33:11.836 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.833 7fa35e8f4700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa35807e890 con 0x7fa3580e5070 2021-02-10T11:33:11.837 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.833 7fa35cf11700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fa3580e5470 0x7fa35807de80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:48946/0 (socket says 172.21.2.112:48946) 2021-02-10T11:33:11.837 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.833 7fa35cf11700 1 -- 172.21.2.112:0/1814806676 learned_addr learned my addr 172.21.2.112:0/1814806676 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:33:11.837 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.833 7fa357fff700 1 --2- 172.21.2.112:0/1814806676 >> v2:172.21.2.112:3300/0 conn(0x7fa3580e5070 0x7fa3581b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:11.837 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.834 7fa357fff700 1 -- 172.21.2.112:0/1814806676 >> v1:172.21.2.120:6789/0 conn(0x7fa3581b3e40 legacy=0x7fa35807d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:11.837 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.834 7fa357fff700 1 -- 172.21.2.112:0/1814806676 >> v1:172.21.2.112:6789/0 conn(0x7fa3580e5470 legacy=0x7fa35807de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:11.837 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.834 7fa357fff700 1 -- 172.21.2.112:0/1814806676 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa35807eab0 con 0x7fa3580e5070 2021-02-10T11:33:11.837 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.834 7fa357fff700 1 --2- 172.21.2.112:0/1814806676 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa3580e5070 0x7fa3581b3820 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 rx=0x7fa348006650 tx=0x7fa348009a80).ready entity=mon.0 client_cookie=307ce190eb871bf3 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:11.837 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.834 7fa356ffd700 1 -- 172.21.2.112:0/1814806676 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa3480009b0 con 0x7fa3580e5070 2021-02-10T11:33:11.837 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.834 7fa356ffd700 1 -- 172.21.2.112:0/1814806676 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa34800c0c0 con 0x7fa3580e5070 2021-02-10T11:33:11.837 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.834 7fa356ffd700 1 -- 172.21.2.112:0/1814806676 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa34800c510 con 0x7fa3580e5070 2021-02-10T11:33:11.837 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.834 7fa35e8f4700 1 -- 172.21.2.112:0/1814806676 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa3580e5070 msgr2=0x7fa3581b3820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:11.837 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.834 7fa35e8f4700 1 --2- 172.21.2.112:0/1814806676 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa3580e5070 0x7fa3581b3820 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 rx=0x7fa348006650 tx=0x7fa348009a80).stop 2021-02-10T11:33:11.837 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.835 7fa35e8f4700 1 -- 172.21.2.112:0/1814806676 shutdown_connections 2021-02-10T11:33:11.838 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.835 7fa35e8f4700 1 --2- 172.21.2.112:0/1814806676 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa3580e5070 0x7fa3581b3820 unknown :-1 s=CLOSED pgs=1200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:11.838 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.835 7fa35e8f4700 1 -- 172.21.2.112:0/1814806676 >> 172.21.2.112:0/1814806676 conn(0x7fa3580ce690 msgr2=0x7fa3580e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:11.838 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.835 7fa35e8f4700 1 -- 172.21.2.112:0/1814806676 shutdown_connections 2021-02-10T11:33:11.838 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.835 7fa35e8f4700 1 -- 172.21.2.112:0/1814806676 wait complete. 2021-02-10T11:33:11.839 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.849 7fa35e8f4700 1 Processor -- start 2021-02-10T11:33:11.839 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.850 7fa35e8f4700 1 -- start start 2021-02-10T11:33:11.839 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.850 7fa35e8f4700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa358092dc0 0x7fa3580931c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:11.839 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.850 7fa35e8f4700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa3580937e0 0x7fa3580ab250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:11.840 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.850 7fa35e8f4700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa3580ab870 0x7fa3580942e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:11.840 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.850 7fa35e8f4700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa358093de0 con 0x7fa358092dc0 2021-02-10T11:33:11.840 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.850 7fa35e8f4700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3581c2500 con 0x7fa3580937e0 2021-02-10T11:33:11.840 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.850 7fa35e8f4700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3581c2720 con 0x7fa3580ab870 2021-02-10T11:33:11.840 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.850 7fa357fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa358092dc0 0x7fa3580931c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:11.840 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.850 7fa357fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa358092dc0 0x7fa3580931c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:49486/0 (socket says 172.21.2.112:49486) 2021-02-10T11:33:11.840 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.850 7fa357fff700 1 -- 172.21.2.112:0/1732154826 learned_addr learned my addr 172.21.2.112:0/1732154826 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:33:11.840 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.851 7fa357fff700 1 -- 172.21.2.112:0/1732154826 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa3580ab870 msgr2=0x7fa3580942e0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:33:11.840 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.851 7fa357fff700 1 --2- 172.21.2.112:0/1732154826 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa3580ab870 0x7fa3580942e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:11.840 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.851 7fa357fff700 1 -- 172.21.2.112:0/1732154826 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa3580937e0 msgr2=0x7fa3580ab250 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:33:11.840 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.851 7fa357fff700 1 --2- 172.21.2.112:0/1732154826 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa3580937e0 0x7fa3580ab250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:11.840 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.851 7fa357fff700 1 -- 172.21.2.112:0/1732154826 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3580abce0 con 0x7fa358092dc0 2021-02-10T11:33:11.841 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.851 7fa357fff700 1 --2- 172.21.2.112:0/1732154826 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa358092dc0 0x7fa3580931c0 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 rx=0x7fa348009630 tx=0x7fa34800b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:11.841 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.851 7fa354ff9700 1 -- 172.21.2.112:0/1732154826 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa3480093a0 con 0x7fa358092dc0 2021-02-10T11:33:11.841 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.851 7fa354ff9700 1 -- 172.21.2.112:0/1732154826 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa34800bd70 con 0x7fa358092dc0 2021-02-10T11:33:11.841 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.851 7fa354ff9700 1 -- 172.21.2.112:0/1732154826 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa348005560 con 0x7fa358092dc0 2021-02-10T11:33:11.841 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.851 7fa35e8f4700 1 -- 172.21.2.112:0/1732154826 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3580abce0 con 0x7fa358092dc0 2021-02-10T11:33:11.841 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.851 7fa35e8f4700 1 -- 172.21.2.112:0/1732154826 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3580abce0 con 0x7fa358092dc0 2021-02-10T11:33:11.841 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.851 7fa354ff9700 1 -- 172.21.2.112:0/1732154826 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa3480052e0 con 0x7fa358092dc0 2021-02-10T11:33:11.841 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.852 7fa354ff9700 1 --2- 172.21.2.112:0/1732154826 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa338018730 0x7fa33801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:11.841 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.852 7fa354ff9700 1 -- 172.21.2.112:0/1732154826 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fa3480327a0 con 0x7fa358092dc0 2021-02-10T11:33:11.841 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.852 7fa3577fe700 1 --2- 172.21.2.112:0/1732154826 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa338018730 0x7fa33801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:11.842 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.853 7fa3577fe700 1 --2- 172.21.2.112:0/1732154826 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa338018730 0x7fa33801abc0 crc :-1 s=READY pgs=602 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:11.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.883 7fa35e8f4700 1 -- 172.21.2.112:0/1732154826 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3580ce690 con 0x7fa358092dc0 2021-02-10T11:33:11.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:11.885 7fa354ff9700 1 -- 172.21.2.112:0/1732154826 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa34801f4e0 con 0x7fa358092dc0 2021-02-10T11:33:12.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:12.013 7fa35e8f4700 1 -- 172.21.2.112:0/1732154826 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa3580ce690 con 0x7fa358092dc0 2021-02-10T11:33:12.001 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:12.014 7fa354ff9700 1 -- 172.21.2.112:0/1732154826 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+2093 (secure 0 0 0) 0x7fa34800bd70 con 0x7fa358092dc0 2021-02-10T11:33:12.002 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:33:12.002 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":31,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7536,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/340651080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":340651080},{"type":"v1","addr":"172.21.2.112:6833","nonce":340651080}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":26}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:32:31.869812","modified":"2021-02-10 11:32:33.879671","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6409},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6409":{"gid":6409,"name":"b","rank":0,"incarnation":27,"state":"up:active","state_seq":3,"addr":"172.21.2.120:6835/210055046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":210055046},{"type":"v1","addr":"172.21.2.120:6835","nonce":210055046}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":4}]} 2021-02-10T11:33:12.002 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 31 2021-02-10T11:33:12.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:12.016 7fa35e8f4700 1 -- 172.21.2.112:0/1732154826 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa338018730 msgr2=0x7fa33801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:12.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:12.016 7fa35e8f4700 1 --2- 172.21.2.112:0/1732154826 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa338018730 0x7fa33801abc0 crc :-1 s=READY pgs=602 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:12.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:12.016 7fa35e8f4700 1 -- 172.21.2.112:0/1732154826 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa358092dc0 msgr2=0x7fa3580931c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:12.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:12.016 7fa35e8f4700 1 --2- 172.21.2.112:0/1732154826 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa358092dc0 0x7fa3580931c0 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 rx=0x7fa348009630 tx=0x7fa34800b980).stop 2021-02-10T11:33:12.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:12.016 7fa35e8f4700 1 -- 172.21.2.112:0/1732154826 shutdown_connections 2021-02-10T11:33:12.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:12.016 7fa35e8f4700 1 --2- 172.21.2.112:0/1732154826 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa338018730 0x7fa33801abc0 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:12.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:12.016 7fa35e8f4700 1 --2- 172.21.2.112:0/1732154826 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa3580937e0 0x7fa3580ab250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:12.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:12.016 7fa35e8f4700 1 --2- 172.21.2.112:0/1732154826 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa3580ab870 0x7fa3580942e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:12.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:12.016 7fa35e8f4700 1 --2- 172.21.2.112:0/1732154826 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa358092dc0 0x7fa3580931c0 unknown :-1 s=CLOSED pgs=1201 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:12.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:12.016 7fa35e8f4700 1 -- 172.21.2.112:0/1732154826 >> 172.21.2.112:0/1732154826 conn(0x7fa358106df0 msgr2=0x7fa3580962d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:12.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:12.016 7fa35e8f4700 1 -- 172.21.2.112:0/1732154826 shutdown_connections 2021-02-10T11:33:12.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:12.016 7fa35e8f4700 1 -- 172.21.2.112:0/1732154826 wait complete. 2021-02-10T11:33:12.012 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs fail cephfs 2021-02-10T11:33:12.438 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:12.433 7fbb8cf05700 1 Processor -- start 2021-02-10T11:33:12.439 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:12.434 7fbb8cf05700 1 -- start start 2021-02-10T11:33:12.439 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:12.434 7fbb8cf05700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fbb880e5070 0x7fbb881b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:12.439 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:12.434 7fbb8cf05700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb8807ed80 con 0x7fbb880e5470 2021-02-10T11:33:12.439 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:12.434 7fbb8cf05700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb8807e670 con 0x7fbb881b3e40 2021-02-10T11:33:12.439 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:12.434 7fbb8cf05700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb8807e890 con 0x7fbb880e5070 2021-02-10T11:33:12.439 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:12.434 7fbb87314700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fbb880e5470 0x7fbb8807de80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:48960/0 (socket says 172.21.2.112:48960) 2021-02-10T11:33:12.439 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:12.434 7fbb87314700 1 -- 172.21.2.112:0/2470687407 learned_addr learned my addr 172.21.2.112:0/2470687407 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:33:12.439 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:12.435 7fbb85b11700 1 -- 172.21.2.112:0/2470687407 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2086305411 0 0) 0x7fbb80001500 con 0x7fbb880e5470 2021-02-10T11:33:12.439 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:12.435 7fbb85b11700 1 -- 172.21.2.112:0/2470687407 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb74002fe0 con 0x7fbb880e5470 2021-02-10T11:33:12.439 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:12.435 7fbb86b13700 1 --2- 172.21.2.112:0/2470687407 >> v2:172.21.2.120:3300/0 conn(0x7fbb880e5070 0x7fbb881b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:12.439 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:12.435 7fbb85b11700 1 -- 172.21.2.112:0/2470687407 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1838990810 0 0) 0x7fbb80001460 con 0x7fbb880e5470 2021-02-10T11:33:12.439 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:12.435 7fbb85b11700 1 -- 172.21.2.112:0/2470687407 >> v2:172.21.2.120:3300/0 conn(0x7fbb880e5070 msgr2=0x7fbb881b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:12.440 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:12.435 7fbb85b11700 1 --2- 172.21.2.112:0/2470687407 >> v2:172.21.2.120:3300/0 conn(0x7fbb880e5070 0x7fbb881b3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:12.440 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:12.435 7fbb85b11700 1 -- 172.21.2.112:0/2470687407 >> v1:172.21.2.120:6789/0 conn(0x7fbb881b3e40 legacy=0x7fbb8807d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:12.440 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:12.435 7fbb85b11700 1 -- 172.21.2.112:0/2470687407 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb8807eab0 con 0x7fbb880e5470 2021-02-10T11:33:12.440 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:12.435 7fbb85b11700 1 -- 172.21.2.112:0/2470687407 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7fbb80001980 con 0x7fbb880e5470 2021-02-10T11:33:12.440 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:12.435 7fbb85b11700 1 -- 172.21.2.112:0/2470687407 >> v1:172.21.2.112:6789/0 conn(0x7fbb880e5470 legacy=0x7fbb8807de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:12.440 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:12.435 7fbb85b11700 1 --2- 172.21.2.112:0/2470687407 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbb74003ae0 0x7fbb74005f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:12.440 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:12.435 7fbb85b11700 1 --2- 172.21.2.112:0/2470687407 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbb74006830 0x7fbb74008d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:12.440 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:12.435 7fbb85b11700 1 --2- 172.21.2.112:0/2470687407 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fbb74009630 0x7fbb7400bb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:12.440 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:12.435 7fbb85b11700 1 -- 172.21.2.112:0/2470687407 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb7400c3a0 con 0x7fbb74006830 2021-02-10T11:33:12.440 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:12.435 7fbb85b11700 1 -- 172.21.2.112:0/2470687407 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb7400c5c0 con 0x7fbb74009630 2021-02-10T11:33:12.440 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:12.435 7fbb85b11700 1 -- 172.21.2.112:0/2470687407 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb7400c7e0 con 0x7fbb74003ae0 2021-02-10T11:33:12.440 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:12.435 7fbb86b13700 1 --2- 172.21.2.112:0/2470687407 >> v2:172.21.2.120:3300/0 conn(0x7fbb880e5070 0x7fbb881b3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:33:12.440 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:12.435 7fbb86312700 1 --2- 172.21.2.112:0/2470687407 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbb74006830 0x7fbb74008d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:12.440 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:12.435 7fbb87314700 1 --2- 172.21.2.112:0/2470687407 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fbb74009630 0x7fbb7400bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:12.441 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:12.436 7fbb86b13700 1 --2- 172.21.2.112:0/2470687407 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbb74003ae0 0x7fbb74005f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:12.441 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:12.436 7fbb86312700 1 -- 172.21.2.112:0/2470687407 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbb74003ae0 msgr2=0x7fbb74005f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:12.441 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:12.436 7fbb86312700 1 --2- 172.21.2.112:0/2470687407 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbb74003ae0 0x7fbb74005f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:12.441 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:12.436 7fbb86312700 1 -- 172.21.2.112:0/2470687407 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fbb74009630 msgr2=0x7fbb7400bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:12.441 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:12.436 7fbb86312700 1 --2- 172.21.2.112:0/2470687407 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fbb74009630 0x7fbb7400bb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:12.441 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:12.436 7fbb87314700 1 -- 172.21.2.112:0/2470687407 reap_dead start 2021-02-10T11:33:12.441 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:12.436 7fbb86312700 1 -- 172.21.2.112:0/2470687407 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb7400ca80 con 0x7fbb74006830 2021-02-10T11:33:12.441 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:12.436 7fbb86312700 1 --2- 172.21.2.112:0/2470687407 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbb74006830 0x7fbb74008d30 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 rx=0x7fbb78006ab0 tx=0x7fbb7804db20).ready entity=mon.0 client_cookie=408674a87490f958 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:12.441 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:12.436 7fbb85b11700 1 -- 172.21.2.112:0/2470687407 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbb780066a0 con 0x7fbb74006830 2021-02-10T11:33:12.441 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:12.436 7fbb85b11700 1 -- 172.21.2.112:0/2470687407 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb78000e20 con 0x7fbb74006830 2021-02-10T11:33:12.441 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:12.436 7fbb85b11700 1 -- 172.21.2.112:0/2470687407 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbb780022c0 con 0x7fbb74006830 2021-02-10T11:33:12.441 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:12.436 7fbb8cf05700 1 -- 172.21.2.112:0/2470687407 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbb74006830 msgr2=0x7fbb74008d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:12.441 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:12.436 7fbb8cf05700 1 --2- 172.21.2.112:0/2470687407 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbb74006830 0x7fbb74008d30 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 rx=0x7fbb78006ab0 tx=0x7fbb7804db20).stop 2021-02-10T11:33:12.442 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:12.437 7fbb8cf05700 1 -- 172.21.2.112:0/2470687407 shutdown_connections 2021-02-10T11:33:12.442 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:12.437 7fbb8cf05700 1 --2- 172.21.2.112:0/2470687407 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbb74006830 0x7fbb74008d30 unknown :-1 s=CLOSED pgs=1203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:12.442 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:12.437 7fbb8cf05700 1 -- 172.21.2.112:0/2470687407 >> 172.21.2.112:0/2470687407 conn(0x7fbb880ce690 msgr2=0x7fbb880e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:12.442 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:12.437 7fbb8cf05700 1 -- 172.21.2.112:0/2470687407 shutdown_connections 2021-02-10T11:33:12.442 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:12.437 7fbb8cf05700 1 -- 172.21.2.112:0/2470687407 wait complete. 2021-02-10T11:33:12.442 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:12.451 7fbb8cf05700 1 Processor -- start 2021-02-10T11:33:12.442 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:12.451 7fbb8cf05700 1 -- start start 2021-02-10T11:33:12.444 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:12.452 7fbb8cf05700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fbb880b6570 0x7fbb880a6af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:12.444 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:12.452 7fbb8cf05700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbb8809edf0 0x7fbb8809f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:12.444 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:12.452 7fbb8cf05700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbb881a6d50 0x7fbb881a7150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:12.444 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:12.452 7fbb8cf05700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb8809f810 con 0x7fbb881a6d50 2021-02-10T11:33:12.444 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:12.452 7fbb8cf05700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb880a6570 con 0x7fbb880b6570 2021-02-10T11:33:12.445 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:12.452 7fbb8cf05700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb881a7770 con 0x7fbb8809edf0 2021-02-10T11:33:12.445 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:12.452 7fbb87314700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbb881a6d50 0x7fbb881a7150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:12.445 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:12.452 7fbb87314700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbb881a6d50 0x7fbb881a7150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:49508/0 (socket says 172.21.2.112:49508) 2021-02-10T11:33:12.445 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:12.452 7fbb87314700 1 -- 172.21.2.112:0/4075467956 learned_addr learned my addr 172.21.2.112:0/4075467956 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:33:12.445 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:12.452 7fbb87314700 1 -- 172.21.2.112:0/4075467956 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbb8809edf0 msgr2=0x7fbb8809f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:12.445 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:12.452 7fbb86b13700 1 --2- 172.21.2.112:0/4075467956 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fbb880b6570 0x7fbb880a6af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:12.446 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:12.452 7fbb86312700 1 --2- 172.21.2.112:0/4075467956 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbb8809edf0 0x7fbb8809f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:12.446 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:12.452 7fbb87314700 1 --2- 172.21.2.112:0/4075467956 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbb8809edf0 0x7fbb8809f1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:12.446 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:12.452 7fbb87314700 1 -- 172.21.2.112:0/4075467956 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fbb880b6570 msgr2=0x7fbb880a6af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:12.446 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:12.452 7fbb87314700 1 --2- 172.21.2.112:0/4075467956 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fbb880b6570 0x7fbb880a6af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:12.446 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:12.452 7fbb87314700 1 -- 172.21.2.112:0/4075467956 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb881a7950 con 0x7fbb881a6d50 2021-02-10T11:33:12.446 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:12.453 7fbb86b13700 1 --2- 172.21.2.112:0/4075467956 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fbb880b6570 0x7fbb880a6af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:33:12.446 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:12.453 7fbb86312700 1 --2- 172.21.2.112:0/4075467956 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbb8809edf0 0x7fbb8809f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:33:12.447 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:12.453 7fbb87314700 1 --2- 172.21.2.112:0/4075467956 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbb881a6d50 0x7fbb881a7150 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 rx=0x7fbb8000a900 tx=0x7fbb8000aca0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:12.447 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:12.453 7fbb7f7fe700 1 -- 172.21.2.112:0/4075467956 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbb800070d0 con 0x7fbb881a6d50 2021-02-10T11:33:12.447 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:12.453 7fbb7f7fe700 1 -- 172.21.2.112:0/4075467956 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb800073d0 con 0x7fbb881a6d50 2021-02-10T11:33:12.447 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:12.453 7fbb7f7fe700 1 -- 172.21.2.112:0/4075467956 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbb80004bb0 con 0x7fbb881a6d50 2021-02-10T11:33:12.447 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:12.453 7fbb8cf05700 1 -- 172.21.2.112:0/4075467956 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb881a7950 con 0x7fbb881a6d50 2021-02-10T11:33:12.447 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:12.453 7fbb8cf05700 1 -- 172.21.2.112:0/4075467956 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb880a3350 con 0x7fbb881a6d50 2021-02-10T11:33:12.448 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:12.454 7fbb7f7fe700 1 -- 172.21.2.112:0/4075467956 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fbb80004720 con 0x7fbb881a6d50 2021-02-10T11:33:12.448 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:12.454 7fbb7f7fe700 1 --2- 172.21.2.112:0/4075467956 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fbb6c0186c0 0x7fbb6c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:12.448 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:12.454 7fbb7f7fe700 1 -- 172.21.2.112:0/4075467956 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fbb800328b0 con 0x7fbb881a6d50 2021-02-10T11:33:12.448 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:12.454 7fbb86b13700 1 --2- 172.21.2.112:0/4075467956 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fbb6c0186c0 0x7fbb6c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:12.448 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:12.456 7fbb86b13700 1 --2- 172.21.2.112:0/4075467956 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fbb6c0186c0 0x7fbb6c01abc0 crc :-1 s=READY pgs=603 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:12.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:12.484 7fbb8cf05700 1 -- 172.21.2.112:0/4075467956 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb88084a90 con 0x7fbb881a6d50 2021-02-10T11:33:12.473 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:12.486 7fbb7f7fe700 1 -- 172.21.2.112:0/4075467956 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fbb80021350 con 0x7fbb881a6d50 2021-02-10T11:33:12.600 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:12.613 7fbb8cf05700 1 -- 172.21.2.112:0/4075467956 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fbb8807ed80 con 0x7fbb881a6d50 2021-02-10T11:33:13.382 INFO:tasks.ceph.mon.a.gibba012.stderr:2021-02-10 11:33:13.395 7fc3fe6e2700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-02-10T11:33:13.386 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:13.400 7fbb7f7fe700 1 -- 172.21.2.112:0/4075467956 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v32) v1 ==== 161+0+0 (secure 0 0 0) 0x7fbb80021350 con 0x7fbb881a6d50 2021-02-10T11:33:13.420 INFO:teuthology.orchestra.run.gibba012.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2021-02-10T11:33:13.422 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:13.435 7fbb8cf05700 1 -- 172.21.2.112:0/4075467956 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fbb6c0186c0 msgr2=0x7fbb6c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:13.422 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:13.435 7fbb8cf05700 1 --2- 172.21.2.112:0/4075467956 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fbb6c0186c0 0x7fbb6c01abc0 crc :-1 s=READY pgs=603 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:13.422 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:13.435 7fbb8cf05700 1 -- 172.21.2.112:0/4075467956 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbb881a6d50 msgr2=0x7fbb881a7150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:13.422 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:13.435 7fbb8cf05700 1 --2- 172.21.2.112:0/4075467956 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbb881a6d50 0x7fbb881a7150 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 rx=0x7fbb8000a900 tx=0x7fbb8000aca0).stop 2021-02-10T11:33:13.422 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:13.435 7fbb8cf05700 1 -- 172.21.2.112:0/4075467956 shutdown_connections 2021-02-10T11:33:13.422 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:13.435 7fbb8cf05700 1 --2- 172.21.2.112:0/4075467956 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fbb6c0186c0 0x7fbb6c01abc0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:13.422 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:13.435 7fbb8cf05700 1 --2- 172.21.2.112:0/4075467956 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbb881a6d50 0x7fbb881a7150 unknown :-1 s=CLOSED pgs=1204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:13.423 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:13.435 7fbb8cf05700 1 --2- 172.21.2.112:0/4075467956 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbb8809edf0 0x7fbb8809f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:13.423 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:13.435 7fbb8cf05700 1 --2- 172.21.2.112:0/4075467956 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fbb880b6570 0x7fbb880a6af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:13.423 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:13.435 7fbb8cf05700 1 -- 172.21.2.112:0/4075467956 >> 172.21.2.112:0/4075467956 conn(0x7fbb8809be90 msgr2=0x7fbb8809cc10 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:13.423 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:13.435 7fbb8cf05700 1 -- 172.21.2.112:0/4075467956 shutdown_connections 2021-02-10T11:33:13.423 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:13.435 7fbb8cf05700 1 -- 172.21.2.112:0/4075467956 wait complete. 2021-02-10T11:33:13.431 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:33:13.438 INFO:tasks.ceph.mds.b.gibba020.stdout:starting mds.b at 2021-02-10T11:33:13.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:13.855 7f6422950700 1 Processor -- start 2021-02-10T11:33:13.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:13.855 7f6422950700 1 -- start start 2021-02-10T11:33:13.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:13.856 7f6422950700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f641c0e5070 0x7f641c1b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:13.860 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:13.856 7f6422950700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f641c0e5470 0x7f641c07d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:13.860 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:13.856 7f6422950700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f641c07ed80 con 0x7f641c1b3e40 2021-02-10T11:33:13.860 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:13.856 7f6422950700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f641c080310 con 0x7f641c0e5470 2021-02-10T11:33:13.860 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:13.856 7f6422950700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f641c080500 con 0x7f641c0e5070 2021-02-10T11:33:13.860 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:13.856 7f641b7fe700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f641c0e5470 0x7f641c07d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:13.860 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:13.856 7f6420f6d700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7f641c1b3e40 0x7f641c07dcb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:42538/0 (socket says 172.21.2.112:42538) 2021-02-10T11:33:13.860 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:13.856 7f6420f6d700 1 -- 172.21.2.112:0/4199196494 learned_addr learned my addr 172.21.2.112:0/4199196494 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:33:13.860 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:13.856 7f641bfff700 1 --2- 172.21.2.112:0/4199196494 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f641c0e5070 0x7f641c1b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:13.860 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:13.857 7f641b7fe700 1 -- 172.21.2.112:0/4199196494 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f641c0e5070 msgr2=0x7f641c1b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:13.860 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:13.857 7f641b7fe700 1 --2- 172.21.2.112:0/4199196494 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f641c0e5070 0x7f641c1b3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:13.860 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:13.857 7f641b7fe700 1 -- 172.21.2.112:0/4199196494 >> v1:172.21.2.120:6789/0 conn(0x7f641c1b3e40 legacy=0x7f641c07dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:13.860 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:13.857 7f641b7fe700 1 -- 172.21.2.112:0/4199196494 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f641c0806f0 con 0x7f641c0e5470 2021-02-10T11:33:13.860 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:13.857 7f641b7fe700 1 --2- 172.21.2.112:0/4199196494 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f641c0e5470 0x7f641c07d690 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 rx=0x7f640c006650 tx=0x7f640c009a80).ready entity=mon.1 client_cookie=decfac932d5ad22d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:13.861 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:13.857 7f641affd700 1 -- 172.21.2.112:0/4199196494 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f640c0009b0 con 0x7f641c0e5470 2021-02-10T11:33:13.861 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:13.857 7f641affd700 1 -- 172.21.2.112:0/4199196494 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f640c00c0c0 con 0x7f641c0e5470 2021-02-10T11:33:13.861 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:13.857 7f641affd700 1 -- 172.21.2.112:0/4199196494 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f640c00c5e0 con 0x7f641c0e5470 2021-02-10T11:33:13.861 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:13.857 7f6422950700 1 -- 172.21.2.112:0/4199196494 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f641c0e5470 msgr2=0x7f641c07d690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:13.861 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:13.857 7f6422950700 1 --2- 172.21.2.112:0/4199196494 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f641c0e5470 0x7f641c07d690 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 rx=0x7f640c006650 tx=0x7f640c009a80).stop 2021-02-10T11:33:13.861 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:13.858 7f6422950700 1 -- 172.21.2.112:0/4199196494 shutdown_connections 2021-02-10T11:33:13.861 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:13.858 7f6422950700 1 --2- 172.21.2.112:0/4199196494 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f641c0e5470 0x7f641c07d690 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:13.861 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:13.858 7f6422950700 1 --2- 172.21.2.112:0/4199196494 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f641c0e5070 0x7f641c1b3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:13.861 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:13.858 7f6422950700 1 -- 172.21.2.112:0/4199196494 >> 172.21.2.112:0/4199196494 conn(0x7f641c0ce690 msgr2=0x7f641c0e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:13.861 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:13.858 7f6422950700 1 -- 172.21.2.112:0/4199196494 shutdown_connections 2021-02-10T11:33:13.861 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:13.858 7f6422950700 1 -- 172.21.2.112:0/4199196494 wait complete. 2021-02-10T11:33:13.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:13.873 7f6422950700 1 Processor -- start 2021-02-10T11:33:13.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:13.873 7f6422950700 1 -- start start 2021-02-10T11:33:13.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:13.873 7f6422950700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f641c08d400 0x7f641c081230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:13.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:13.873 7f6422950700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f641c081850 0x7f641c0acf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:13.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:13.873 7f6422950700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f641c081c50 0x7f641c0ad5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:13.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:13.873 7f6422950700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f641c082110 con 0x7f641c08d400 2021-02-10T11:33:13.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:13.873 7f6422950700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f641c08d800 con 0x7f641c081c50 2021-02-10T11:33:13.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:13.873 7f6422950700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f641c0adbc0 con 0x7f641c081850 2021-02-10T11:33:13.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:13.873 7f641bfff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f641c08d400 0x7f641c081230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:13.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:13.874 7f641bfff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f641c08d400 0x7f641c081230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:49522/0 (socket says 172.21.2.112:49522) 2021-02-10T11:33:13.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:13.874 7f641bfff700 1 -- 172.21.2.112:0/2852527976 learned_addr learned my addr 172.21.2.112:0/2852527976 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:33:13.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:13.874 7f641b7fe700 1 --2- 172.21.2.112:0/2852527976 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f641c081850 0x7f641c0acf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:13.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:13.874 7f6420f6d700 1 --2- 172.21.2.112:0/2852527976 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f641c081c50 0x7f641c0ad5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:13.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:13.874 7f641bfff700 1 -- 172.21.2.112:0/2852527976 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f641c081850 msgr2=0x7f641c0acf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:13.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:13.874 7f641bfff700 1 --2- 172.21.2.112:0/2852527976 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f641c081850 0x7f641c0acf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:13.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:13.874 7f641bfff700 1 -- 172.21.2.112:0/2852527976 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f641c081c50 msgr2=0x7f641c0ad5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:13.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:13.874 7f641bfff700 1 --2- 172.21.2.112:0/2852527976 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f641c081c50 0x7f641c0ad5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:13.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:13.874 7f641bfff700 1 -- 172.21.2.112:0/2852527976 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f641c187fc0 con 0x7f641c08d400 2021-02-10T11:33:13.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:13.874 7f641bfff700 1 --2- 172.21.2.112:0/2852527976 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f641c08d400 0x7f641c081230 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 rx=0x7f6404006f60 tx=0x7f640400b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:13.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:13.874 7f6418ff9700 1 -- 172.21.2.112:0/2852527976 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f64040009b0 con 0x7f641c08d400 2021-02-10T11:33:13.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:13.874 7f6418ff9700 1 -- 172.21.2.112:0/2852527976 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f640400bdf0 con 0x7f641c08d400 2021-02-10T11:33:13.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:13.874 7f6418ff9700 1 -- 172.21.2.112:0/2852527976 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6404002030 con 0x7f641c08d400 2021-02-10T11:33:13.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:13.874 7f6422950700 1 -- 172.21.2.112:0/2852527976 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f641c187fc0 con 0x7f641c08d400 2021-02-10T11:33:13.865 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:13.874 7f6422950700 1 -- 172.21.2.112:0/2852527976 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f641c188550 con 0x7f641c08d400 2021-02-10T11:33:13.865 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:13.875 7f6418ff9700 1 -- 172.21.2.112:0/2852527976 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6404000c90 con 0x7f641c08d400 2021-02-10T11:33:13.865 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:13.875 7f6418ff9700 1 --2- 172.21.2.112:0/2852527976 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f64000186c0 0x7f640001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:13.865 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:13.875 7f6418ff9700 1 -- 172.21.2.112:0/2852527976 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5659+0+0 (secure 0 0 0) 0x7f6404032600 con 0x7f641c08d400 2021-02-10T11:33:13.865 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:13.875 7f641b7fe700 1 --2- 172.21.2.112:0/2852527976 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f64000186c0 0x7f640001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:13.865 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:13.876 7f641b7fe700 1 --2- 172.21.2.112:0/2852527976 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f64000186c0 0x7f640001abc0 crc :-1 s=READY pgs=604 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:13.892 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:13.905 7f6422950700 1 -- 172.21.2.112:0/2852527976 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f641c08d800 con 0x7f641c08d400 2021-02-10T11:33:13.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:13.908 7f6418ff9700 1 -- 172.21.2.112:0/2852527976 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6404006b00 con 0x7f641c08d400 2021-02-10T11:33:14.024 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:14.037 7f6422950700 1 -- 172.21.2.112:0/2852527976 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f641c08d800 con 0x7f641c08d400 2021-02-10T11:33:14.024 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:14.037 7f6418ff9700 1 -- 172.21.2.112:0/2852527976 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+1751 (secure 0 0 0) 0x7f6404000c90 con 0x7f641c08d400 2021-02-10T11:33:14.025 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:33:14.025 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":32,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7536,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/340651080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":340651080},{"type":"v1","addr":"172.21.2.112:6833","nonce":340651080}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":26}],"filesystems":[{"mdsmap":{"epoch":32,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:32:31.869812","modified":"2021-02-10 11:33:13.395909","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":4}]} 2021-02-10T11:33:14.025 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 32 2021-02-10T11:33:14.026 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:14.039 7f6422950700 1 -- 172.21.2.112:0/2852527976 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f64000186c0 msgr2=0x7f640001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:14.026 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:14.039 7f6422950700 1 --2- 172.21.2.112:0/2852527976 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f64000186c0 0x7f640001abc0 crc :-1 s=READY pgs=604 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:14.026 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:14.039 7f6422950700 1 -- 172.21.2.112:0/2852527976 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f641c08d400 msgr2=0x7f641c081230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:14.027 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:14.039 7f6422950700 1 --2- 172.21.2.112:0/2852527976 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f641c08d400 0x7f641c081230 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 rx=0x7f6404006f60 tx=0x7f640400b970).stop 2021-02-10T11:33:14.027 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:14.039 7f6422950700 1 -- 172.21.2.112:0/2852527976 shutdown_connections 2021-02-10T11:33:14.027 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:14.039 7f6422950700 1 --2- 172.21.2.112:0/2852527976 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f64000186c0 0x7f640001abc0 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:14.027 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:14.039 7f6422950700 1 --2- 172.21.2.112:0/2852527976 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f641c081c50 0x7f641c0ad5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:14.027 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:14.039 7f6422950700 1 --2- 172.21.2.112:0/2852527976 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f641c081850 0x7f641c0acf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:14.027 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:14.039 7f6422950700 1 --2- 172.21.2.112:0/2852527976 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f641c08d400 0x7f641c081230 unknown :-1 s=CLOSED pgs=1205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:14.027 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:14.039 7f6422950700 1 -- 172.21.2.112:0/2852527976 >> 172.21.2.112:0/2852527976 conn(0x7f641c1ac620 msgr2=0x7f641c0896e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:14.027 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:14.040 7f6422950700 1 -- 172.21.2.112:0/2852527976 shutdown_connections 2021-02-10T11:33:14.028 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:14.040 7f6422950700 1 -- 172.21.2.112:0/2852527976 wait complete. 2021-02-10T11:33:14.036 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2021-02-10T11:33:14.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:14.463 7eff2d377700 1 Processor -- start 2021-02-10T11:33:14.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:14.463 7eff2d377700 1 -- start start 2021-02-10T11:33:14.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:14.464 7eff2d377700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7eff280d17f0 0x7eff281b31e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:14.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:14.464 7eff2d377700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7eff280d1bf0 0x7eff281b3800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:14.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:14.464 7eff2d377700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff280f0b20 con 0x7eff281b3e20 2021-02-10T11:33:14.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:14.464 7eff2d377700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff280e5eb0 con 0x7eff280d1bf0 2021-02-10T11:33:14.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:14.464 7eff2d377700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff280f0630 con 0x7eff280d17f0 2021-02-10T11:33:14.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:14.464 7eff2687c700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7eff280d1bf0 0x7eff281b3800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:14.469 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:14.464 7eff2687c700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7eff280d1bf0 0x7eff281b3800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:49528/0 (socket says 172.21.2.112:49528) 2021-02-10T11:33:14.469 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:14.464 7eff2687c700 1 -- 172.21.2.112:0/1033704828 learned_addr learned my addr 172.21.2.112:0/1033704828 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:33:14.469 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:14.465 7eff2687c700 1 -- 172.21.2.112:0/1033704828 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7eff280d17f0 msgr2=0x7eff281b31e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:14.469 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:14.465 7eff2687c700 1 --2- 172.21.2.112:0/1033704828 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7eff280d17f0 0x7eff281b31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:14.469 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:14.465 7eff2687c700 1 -- 172.21.2.112:0/1033704828 >> v1:172.21.2.120:6789/0 conn(0x7eff281b3e20 legacy=0x7eff280ceef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:14.469 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:14.465 7eff2687c700 1 -- 172.21.2.112:0/1033704828 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff280f0820 con 0x7eff280d1bf0 2021-02-10T11:33:14.469 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:14.465 7eff2687c700 1 --2- 172.21.2.112:0/1033704828 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7eff280d1bf0 0x7eff281b3800 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 rx=0x7eff18006700 tx=0x7eff18009b80).ready entity=mon.0 client_cookie=1ee35f8d479f68bd server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:14.469 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:14.465 7eff2607b700 1 -- 172.21.2.112:0/1033704828 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7eff1800c0c0 con 0x7eff280d1bf0 2021-02-10T11:33:14.469 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:14.465 7eff2607b700 1 -- 172.21.2.112:0/1033704828 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff1800c340 con 0x7eff280d1bf0 2021-02-10T11:33:14.469 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:14.465 7eff2607b700 1 -- 172.21.2.112:0/1033704828 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7eff1800c860 con 0x7eff280d1bf0 2021-02-10T11:33:14.469 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:14.465 7eff2d377700 1 -- 172.21.2.112:0/1033704828 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7eff280d1bf0 msgr2=0x7eff281b3800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:14.469 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:14.465 7eff2d377700 1 --2- 172.21.2.112:0/1033704828 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7eff280d1bf0 0x7eff281b3800 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 rx=0x7eff18006700 tx=0x7eff18009b80).stop 2021-02-10T11:33:14.469 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:14.466 7eff2d377700 1 -- 172.21.2.112:0/1033704828 shutdown_connections 2021-02-10T11:33:14.470 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:14.466 7eff2d377700 1 --2- 172.21.2.112:0/1033704828 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7eff280d1bf0 0x7eff281b3800 unknown :-1 s=CLOSED pgs=1206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:14.470 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:14.466 7eff2d377700 1 --2- 172.21.2.112:0/1033704828 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7eff280d17f0 0x7eff281b31e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:14.470 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:14.466 7eff2d377700 1 -- 172.21.2.112:0/1033704828 >> 172.21.2.112:0/1033704828 conn(0x7eff280fde40 msgr2=0x7eff280ce390 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:14.470 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:14.466 7eff2d377700 1 -- 172.21.2.112:0/1033704828 shutdown_connections 2021-02-10T11:33:14.470 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:14.466 7eff2d377700 1 -- 172.21.2.112:0/1033704828 wait complete. 2021-02-10T11:33:14.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:14.481 7eff2d377700 1 Processor -- start 2021-02-10T11:33:14.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:14.481 7eff2d377700 1 -- start start 2021-02-10T11:33:14.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:14.482 7eff2d377700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7eff281aa810 0x7eff281aa010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:14.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:14.482 7eff2d377700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7eff2809c960 0x7eff28092ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:14.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:14.482 7eff2d377700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7eff2809cd60 0x7eff280930d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:14.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:14.482 7eff2d377700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff281aac10 con 0x7eff2809cd60 2021-02-10T11:33:14.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:14.482 7eff2d377700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff280936f0 con 0x7eff281aa810 2021-02-10T11:33:14.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:14.482 7eff2d377700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff280b7400 con 0x7eff2809c960 2021-02-10T11:33:14.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:14.482 7eff2787e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7eff2809cd60 0x7eff280930d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:14.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:14.482 7eff2687c700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7eff2809c960 0x7eff28092ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:14.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:14.482 7eff2687c700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7eff2809c960 0x7eff28092ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3301/0 says I am v2:172.21.2.112:49216/0 (socket says 172.21.2.112:49216) 2021-02-10T11:33:14.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:14.482 7eff2687c700 1 -- 172.21.2.112:0/2323297004 learned_addr learned my addr 172.21.2.112:0/2323297004 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:33:14.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:14.482 7eff2787e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7eff2809cd60 0x7eff280930d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:49536/0 (socket says 172.21.2.112:49536) 2021-02-10T11:33:14.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:14.482 7eff2707d700 1 --2- 172.21.2.112:0/2323297004 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7eff281aa810 0x7eff281aa010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:14.473 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:14.482 7eff2687c700 1 -- 172.21.2.112:0/2323297004 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7eff281aa810 msgr2=0x7eff281aa010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:14.473 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:14.482 7eff2687c700 1 --2- 172.21.2.112:0/2323297004 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7eff281aa810 0x7eff281aa010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:14.473 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:14.482 7eff2687c700 1 -- 172.21.2.112:0/2323297004 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7eff2809cd60 msgr2=0x7eff280930d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:14.473 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:14.482 7eff2687c700 1 --2- 172.21.2.112:0/2323297004 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7eff2809cd60 0x7eff280930d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:14.473 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:14.482 7eff2687c700 1 -- 172.21.2.112:0/2323297004 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff280b7620 con 0x7eff2809c960 2021-02-10T11:33:14.473 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:14.483 7eff2687c700 1 --2- 172.21.2.112:0/2323297004 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7eff2809c960 0x7eff28092ab0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7eff1800b5e0 tx=0x7eff1800b980).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:14.473 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:14.483 7eff1ffff700 1 -- 172.21.2.112:0/2323297004 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7eff1800bd70 con 0x7eff2809c960 2021-02-10T11:33:14.473 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:14.483 7eff1ffff700 1 -- 172.21.2.112:0/2323297004 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff18002030 con 0x7eff2809c960 2021-02-10T11:33:14.473 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:14.483 7eff1ffff700 1 -- 172.21.2.112:0/2323297004 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7eff180024d0 con 0x7eff2809c960 2021-02-10T11:33:14.473 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:14.483 7eff2d377700 1 -- 172.21.2.112:0/2323297004 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff280b7620 con 0x7eff2809c960 2021-02-10T11:33:14.473 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:14.483 7eff2d377700 1 -- 172.21.2.112:0/2323297004 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff280b7620 con 0x7eff2809c960 2021-02-10T11:33:14.473 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:14.484 7eff1ffff700 1 -- 172.21.2.112:0/2323297004 <== mon.2 v2:172.21.2.120:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7eff18000c90 con 0x7eff2809c960 2021-02-10T11:33:14.473 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:14.484 7eff1ffff700 1 --2- 172.21.2.112:0/2323297004 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7eff0c0186c0 0x7eff0c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:14.473 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:14.484 7eff1ffff700 1 -- 172.21.2.112:0/2323297004 <== mon.2 v2:172.21.2.120:3301/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5659+0+0 (secure 0 0 0) 0x7eff18032440 con 0x7eff2809c960 2021-02-10T11:33:14.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:14.484 7eff2707d700 1 --2- 172.21.2.112:0/2323297004 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7eff0c0186c0 0x7eff0c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:14.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:14.484 7eff2707d700 1 --2- 172.21.2.112:0/2323297004 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7eff0c0186c0 0x7eff0c01abc0 crc :-1 s=READY pgs=605 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:14.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:14.514 7eff2d377700 1 -- 172.21.2.112:0/2323297004 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff281036f0 con 0x7eff2809c960 2021-02-10T11:33:14.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:14.517 7eff1ffff700 1 -- 172.21.2.112:0/2323297004 <== mon.2 v2:172.21.2.120:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7eff18030d20 con 0x7eff2809c960 2021-02-10T11:33:14.633 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:14.646 7eff2d377700 1 -- 172.21.2.112:0/2323297004 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7eff281036f0 con 0x7eff2809c960 2021-02-10T11:33:15.394 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:15.407 7eff1ffff700 1 -- 172.21.2.112:0/2323297004 <== mon.2 v2:172.21.2.120:3301/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v33) v1 ==== 104+0+0 (secure 0 0 0) 0x7eff18000c90 con 0x7eff2809c960 2021-02-10T11:33:15.406 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:15.419 7eff2d377700 1 -- 172.21.2.112:0/2323297004 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7eff0c0186c0 msgr2=0x7eff0c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:15.406 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:15.419 7eff2d377700 1 --2- 172.21.2.112:0/2323297004 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7eff0c0186c0 0x7eff0c01abc0 crc :-1 s=READY pgs=605 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:15.406 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:15.419 7eff2d377700 1 -- 172.21.2.112:0/2323297004 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7eff2809c960 msgr2=0x7eff28092ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:15.406 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:15.419 7eff2d377700 1 --2- 172.21.2.112:0/2323297004 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7eff2809c960 0x7eff28092ab0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7eff1800b5e0 tx=0x7eff1800b980).stop 2021-02-10T11:33:15.406 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:15.419 7eff2d377700 1 -- 172.21.2.112:0/2323297004 shutdown_connections 2021-02-10T11:33:15.406 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:15.419 7eff2d377700 1 --2- 172.21.2.112:0/2323297004 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7eff0c0186c0 0x7eff0c01abc0 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:15.406 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:15.419 7eff2d377700 1 --2- 172.21.2.112:0/2323297004 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7eff2809cd60 0x7eff280930d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:15.407 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:15.419 7eff2d377700 1 --2- 172.21.2.112:0/2323297004 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7eff2809c960 0x7eff28092ab0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:15.407 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:15.419 7eff2d377700 1 --2- 172.21.2.112:0/2323297004 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7eff281aa810 0x7eff281aa010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:15.407 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:15.419 7eff2d377700 1 -- 172.21.2.112:0/2323297004 >> 172.21.2.112:0/2323297004 conn(0x7eff28104170 msgr2=0x7eff28107720 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:15.407 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:15.419 7eff2d377700 1 -- 172.21.2.112:0/2323297004 shutdown_connections 2021-02-10T11:33:15.407 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:15.419 7eff2d377700 1 -- 172.21.2.112:0/2323297004 wait complete. 2021-02-10T11:33:15.415 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2021-02-10T11:33:15.846 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:15.842 7f8e16089700 1 Processor -- start 2021-02-10T11:33:15.847 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:15.842 7f8e16089700 1 -- start start 2021-02-10T11:33:15.847 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:15.843 7f8e16089700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f8e100d1c70 0x7f8e100cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:15.847 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:15.843 7f8e16089700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f8e100cf580 0x7f8e100cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:15.847 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:15.843 7f8e16089700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e100fe1c0 con 0x7f8e100d1870 2021-02-10T11:33:15.847 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:15.843 7f8e16089700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e101b0cb0 con 0x7f8e100cf580 2021-02-10T11:33:15.847 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:15.843 7f8e16089700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e10109a60 con 0x7f8e100d1c70 2021-02-10T11:33:15.847 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:15.843 7f8e0f7fe700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f8e100d1870 0x7f8e101b04c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:49002/0 (socket says 172.21.2.112:49002) 2021-02-10T11:33:15.847 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:15.843 7f8e0f7fe700 1 -- 172.21.2.112:0/3437997689 learned_addr learned my addr 172.21.2.112:0/3437997689 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:33:15.847 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:15.843 7f8e0ffff700 1 --2- 172.21.2.112:0/3437997689 >> v2:172.21.2.112:3300/0 conn(0x7f8e100cf580 0x7f8e100cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:15.847 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:15.844 7f8e0e7fc700 1 -- 172.21.2.112:0/3437997689 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4258514600 0 0) 0x7f8df8001180 con 0x7f8e100d1870 2021-02-10T11:33:15.847 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:15.844 7f8e0e7fc700 1 -- 172.21.2.112:0/3437997689 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8dfc002fe0 con 0x7f8e100d1870 2021-02-10T11:33:15.847 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:15.844 7f8e0ffff700 1 -- 172.21.2.112:0/3437997689 >> v2:172.21.2.120:3300/0 conn(0x7f8e100d1c70 msgr2=0x7f8e100cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:15.848 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:15.844 7f8e0ffff700 1 --2- 172.21.2.112:0/3437997689 >> v2:172.21.2.120:3300/0 conn(0x7f8e100d1c70 0x7f8e100cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:15.848 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:15.844 7f8e0ffff700 1 -- 172.21.2.112:0/3437997689 >> v1:172.21.2.112:6789/0 conn(0x7f8e100d1870 legacy=0x7f8e101b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:15.848 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:15.844 7f8e0ffff700 1 -- 172.21.2.112:0/3437997689 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e10088020 con 0x7f8e100cf580 2021-02-10T11:33:15.848 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:15.844 7f8e0ffff700 1 --2- 172.21.2.112:0/3437997689 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8e100cf580 0x7f8e100cf980 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 rx=0x7f8e08006e30 tx=0x7f8e08009e60).ready entity=mon.0 client_cookie=2e640abe9a6b4973 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:15.848 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:15.844 7f8e0e7fc700 1 -- 172.21.2.112:0/3437997689 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8e08006a50 con 0x7f8e100cf580 2021-02-10T11:33:15.848 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:15.844 7f8e0e7fc700 1 -- 172.21.2.112:0/3437997689 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e0800b300 con 0x7f8e100cf580 2021-02-10T11:33:15.848 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:15.844 7f8e0e7fc700 1 -- 172.21.2.112:0/3437997689 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8e0800b820 con 0x7f8e100cf580 2021-02-10T11:33:15.848 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:15.844 7f8e16089700 1 -- 172.21.2.112:0/3437997689 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8e100cf580 msgr2=0x7f8e100cf980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:15.848 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:15.844 7f8e16089700 1 --2- 172.21.2.112:0/3437997689 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8e100cf580 0x7f8e100cf980 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 rx=0x7f8e08006e30 tx=0x7f8e08009e60).stop 2021-02-10T11:33:15.848 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:15.845 7f8e16089700 1 -- 172.21.2.112:0/3437997689 shutdown_connections 2021-02-10T11:33:15.848 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:15.845 7f8e16089700 1 --2- 172.21.2.112:0/3437997689 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8e100cf580 0x7f8e100cf980 unknown :-1 s=CLOSED pgs=1208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:15.848 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:15.845 7f8e16089700 1 --2- 172.21.2.112:0/3437997689 >> v2:172.21.2.120:3300/0 conn(0x7f8e100d1c70 0x7f8e100cef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:15.848 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:15.845 7f8e16089700 1 -- 172.21.2.112:0/3437997689 >> 172.21.2.112:0/3437997689 conn(0x7f8e100ce420 msgr2=0x7f8e100d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:15.848 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:15.845 7f8e16089700 1 -- 172.21.2.112:0/3437997689 shutdown_connections 2021-02-10T11:33:15.849 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:15.845 7f8e16089700 1 -- 172.21.2.112:0/3437997689 wait complete. 2021-02-10T11:33:15.850 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:15.860 7f8e16089700 1 Processor -- start 2021-02-10T11:33:15.850 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:15.860 7f8e16089700 1 -- start start 2021-02-10T11:33:15.850 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:15.860 7f8e16089700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8e100b8070 0x7f8e100b77c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:15.850 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:15.860 7f8e16089700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8e100ba4a0 0x7f8e100ba8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:15.850 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:15.860 7f8e16089700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8e101a2fd0 0x7f8e101a33d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:15.850 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:15.860 7f8e16089700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e100baec0 con 0x7f8e101a2fd0 2021-02-10T11:33:15.850 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:15.860 7f8e16089700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e101a39f0 con 0x7f8e100ba4a0 2021-02-10T11:33:15.851 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:15.860 7f8e16089700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e101a3c10 con 0x7f8e100b8070 2021-02-10T11:33:15.851 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:15.861 7f8e0ffff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8e101a2fd0 0x7f8e101a33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:15.851 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:15.861 7f8e0ffff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8e101a2fd0 0x7f8e101a33d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:49550/0 (socket says 172.21.2.112:49550) 2021-02-10T11:33:15.851 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:15.861 7f8e0ffff700 1 -- 172.21.2.112:0/3868366871 learned_addr learned my addr 172.21.2.112:0/3868366871 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:33:15.851 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:15.861 7f8e0effd700 1 --2- 172.21.2.112:0/3868366871 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8e100ba4a0 0x7f8e100ba8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:15.851 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:15.861 7f8e0ffff700 1 -- 172.21.2.112:0/3868366871 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8e100b8070 msgr2=0x7f8e100b77c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:15.851 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:15.861 7f8e0f7fe700 1 --2- 172.21.2.112:0/3868366871 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8e100b8070 0x7f8e100b77c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:15.851 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:15.861 7f8e0ffff700 1 --2- 172.21.2.112:0/3868366871 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8e100b8070 0x7f8e100b77c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:15.851 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:15.861 7f8e0ffff700 1 -- 172.21.2.112:0/3868366871 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8e100ba4a0 msgr2=0x7f8e100ba8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:15.851 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:15.861 7f8e0ffff700 1 --2- 172.21.2.112:0/3868366871 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8e100ba4a0 0x7f8e100ba8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:15.851 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:15.861 7f8e0ffff700 1 -- 172.21.2.112:0/3868366871 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e100ad700 con 0x7f8e101a2fd0 2021-02-10T11:33:15.851 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:15.861 7f8e0effd700 1 --2- 172.21.2.112:0/3868366871 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8e100ba4a0 0x7f8e100ba8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:33:15.851 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:15.861 7f8e0f7fe700 1 --2- 172.21.2.112:0/3868366871 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8e100b8070 0x7f8e100b77c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:33:15.852 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:15.862 7f8e0ffff700 1 --2- 172.21.2.112:0/3868366871 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8e101a2fd0 0x7f8e101a33d0 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 rx=0x7f8e0800b970 tx=0x7f8e0800bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:15.852 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:15.862 7f8e07fff700 1 -- 172.21.2.112:0/3868366871 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8e08009380 con 0x7f8e101a2fd0 2021-02-10T11:33:15.852 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:15.862 7f8e07fff700 1 -- 172.21.2.112:0/3868366871 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e08005820 con 0x7f8e101a2fd0 2021-02-10T11:33:15.852 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:15.862 7f8e07fff700 1 -- 172.21.2.112:0/3868366871 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8e08005d40 con 0x7f8e101a2fd0 2021-02-10T11:33:15.852 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:15.862 7f8e16089700 1 -- 172.21.2.112:0/3868366871 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e100ad700 con 0x7f8e101a2fd0 2021-02-10T11:33:15.852 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:15.862 7f8e16089700 1 -- 172.21.2.112:0/3868366871 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e100adc90 con 0x7f8e101a2fd0 2021-02-10T11:33:15.852 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:15.863 7f8e07fff700 1 -- 172.21.2.112:0/3868366871 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8e08005cd0 con 0x7f8e101a2fd0 2021-02-10T11:33:15.852 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:15.863 7f8e07fff700 1 --2- 172.21.2.112:0/3868366871 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8df40186c0 0x7f8df401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:15.852 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:15.863 7f8e07fff700 1 -- 172.21.2.112:0/3868366871 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5659+0+0 (secure 0 0 0) 0x7f8e080329c0 con 0x7f8e101a2fd0 2021-02-10T11:33:15.853 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:15.864 7f8e0f7fe700 1 --2- 172.21.2.112:0/3868366871 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8df40186c0 0x7f8df401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:15.853 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:15.865 7f8e0f7fe700 1 --2- 172.21.2.112:0/3868366871 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8df40186c0 0x7f8df401abc0 crc :-1 s=READY pgs=606 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:15.880 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:15.893 7f8e16089700 1 -- 172.21.2.112:0/3868366871 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e100828e0 con 0x7f8e101a2fd0 2021-02-10T11:33:15.883 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:15.896 7f8e07fff700 1 -- 172.21.2.112:0/3868366871 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8e08005ac0 con 0x7f8e101a2fd0 2021-02-10T11:33:16.009 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:16.022 7f8e16089700 1 -- 172.21.2.112:0/3868366871 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f8e100828e0 con 0x7f8e101a2fd0 2021-02-10T11:33:16.115 DEBUG:teuthology.orchestra.run.gibba012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:33:16.119 DEBUG:teuthology.orchestra.run.gibba020:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:33:16.402 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:16.415 7f8e07fff700 1 -- 172.21.2.112:0/3868366871 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v48) v1 ==== 185+0+0 (secure 0 0 0) 0x7f8e08005cd0 con 0x7f8e101a2fd0 2021-02-10T11:33:16.425 INFO:teuthology.orchestra.run.gibba012.stderr:pool 'cephfs_metadata' removed 2021-02-10T11:33:16.427 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:16.440 7f8e16089700 1 -- 172.21.2.112:0/3868366871 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8df40186c0 msgr2=0x7f8df401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:16.427 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:16.440 7f8e16089700 1 --2- 172.21.2.112:0/3868366871 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8df40186c0 0x7f8df401abc0 crc :-1 s=READY pgs=606 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:16.428 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:16.440 7f8e16089700 1 -- 172.21.2.112:0/3868366871 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8e101a2fd0 msgr2=0x7f8e101a33d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:16.428 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:16.440 7f8e16089700 1 --2- 172.21.2.112:0/3868366871 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8e101a2fd0 0x7f8e101a33d0 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 rx=0x7f8e0800b970 tx=0x7f8e0800bd10).stop 2021-02-10T11:33:16.428 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:16.440 7f8e16089700 1 -- 172.21.2.112:0/3868366871 shutdown_connections 2021-02-10T11:33:16.428 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:16.440 7f8e16089700 1 --2- 172.21.2.112:0/3868366871 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8df40186c0 0x7f8df401abc0 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:16.428 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:16.440 7f8e16089700 1 --2- 172.21.2.112:0/3868366871 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8e100ba4a0 0x7f8e100ba8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:16.428 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:16.440 7f8e16089700 1 --2- 172.21.2.112:0/3868366871 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8e101a2fd0 0x7f8e101a33d0 unknown :-1 s=CLOSED pgs=1209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:16.428 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:16.440 7f8e16089700 1 --2- 172.21.2.112:0/3868366871 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8e100b8070 0x7f8e100b77c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:16.428 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:16.440 7f8e16089700 1 -- 172.21.2.112:0/3868366871 >> 172.21.2.112:0/3868366871 conn(0x7f8e100b58d0 msgr2=0x7f8e100b5010 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:16.428 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:16.441 7f8e16089700 1 -- 172.21.2.112:0/3868366871 shutdown_connections 2021-02-10T11:33:16.428 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:16.441 7f8e16089700 1 -- 172.21.2.112:0/3868366871 wait complete. 2021-02-10T11:33:16.436 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2021-02-10T11:33:16.868 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:16.863 7f0d8eca0700 1 Processor -- start 2021-02-10T11:33:16.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:16.863 7f0d8eca0700 1 -- start start 2021-02-10T11:33:16.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:16.864 7f0d8eca0700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f0d880d1c60 0x7f0d880cef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:16.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:16.864 7f0d8eca0700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d880d0160 con 0x7f0d880d1860 2021-02-10T11:33:16.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:16.864 7f0d8eca0700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d880fe1b0 con 0x7f0d880cf570 2021-02-10T11:33:16.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:16.864 7f0d8eca0700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d881b0ca0 con 0x7f0d880d1c60 2021-02-10T11:33:16.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:16.864 7f0d8cabc700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f0d880d1860 0x7f0d881b04b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:49016/0 (socket says 172.21.2.112:49016) 2021-02-10T11:33:16.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:16.864 7f0d8cabc700 1 -- 172.21.2.112:0/1947217983 learned_addr learned my addr 172.21.2.112:0/1947217983 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:33:16.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:16.865 7f0d877fe700 1 -- 172.21.2.112:0/1947217983 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1817035352 0 0) 0x7f0d70000ce0 con 0x7f0d880d1860 2021-02-10T11:33:16.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:16.865 7f0d877fe700 1 -- 172.21.2.112:0/1947217983 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d74002fe0 con 0x7f0d880d1860 2021-02-10T11:33:16.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:16.865 7f0d87fff700 1 --2- 172.21.2.112:0/1947217983 >> v2:172.21.2.120:3300/0 conn(0x7f0d880d1c60 0x7f0d880cef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:16.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:16.865 7f0d877fe700 1 -- 172.21.2.112:0/1947217983 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 421191022 0 0) 0x7f0d700010e0 con 0x7f0d880d1860 2021-02-10T11:33:16.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:16.865 7f0d877fe700 1 -- 172.21.2.112:0/1947217983 >> v2:172.21.2.120:3300/0 conn(0x7f0d880d1c60 msgr2=0x7f0d880cef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:16.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:16.865 7f0d877fe700 1 --2- 172.21.2.112:0/1947217983 >> v2:172.21.2.120:3300/0 conn(0x7f0d880d1c60 0x7f0d880cef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:16.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:16.865 7f0d877fe700 1 -- 172.21.2.112:0/1947217983 >> v1:172.21.2.120:6789/0 conn(0x7f0d880cf570 legacy=0x7f0d880cf970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:16.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:16.865 7f0d877fe700 1 -- 172.21.2.112:0/1947217983 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d88109a50 con 0x7f0d880d1860 2021-02-10T11:33:16.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:16.865 7f0d877fe700 1 -- 172.21.2.112:0/1947217983 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f0d70001660 con 0x7f0d880d1860 2021-02-10T11:33:16.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:16.865 7f0d877fe700 1 -- 172.21.2.112:0/1947217983 >> v1:172.21.2.112:6789/0 conn(0x7f0d880d1860 legacy=0x7f0d881b04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:16.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:16.865 7f0d877fe700 1 --2- 172.21.2.112:0/1947217983 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0d74003af0 0x7f0d74005f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:16.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:16.865 7f0d877fe700 1 --2- 172.21.2.112:0/1947217983 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0d74006780 0x7f0d74008c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:16.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:16.865 7f0d877fe700 1 --2- 172.21.2.112:0/1947217983 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0d74009580 0x7f0d7400baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:16.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:16.865 7f0d877fe700 1 -- 172.21.2.112:0/1947217983 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d7400c2f0 con 0x7f0d74009580 2021-02-10T11:33:16.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:16.865 7f0d877fe700 1 -- 172.21.2.112:0/1947217983 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d7400c510 con 0x7f0d74003af0 2021-02-10T11:33:16.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:16.865 7f0d877fe700 1 -- 172.21.2.112:0/1947217983 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d7400c730 con 0x7f0d74006780 2021-02-10T11:33:16.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:16.865 7f0d8d2bd700 1 --2- 172.21.2.112:0/1947217983 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0d74009580 0x7f0d7400baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:16.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:16.865 7f0d8cabc700 1 --2- 172.21.2.112:0/1947217983 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0d74003af0 0x7f0d74005f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:16.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:16.866 7f0d87fff700 1 --2- 172.21.2.112:0/1947217983 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0d74006780 0x7f0d74008c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:16.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:16.866 7f0d8d2bd700 1 -- 172.21.2.112:0/1947217983 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0d74006780 msgr2=0x7f0d74008c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:16.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:16.866 7f0d8d2bd700 1 --2- 172.21.2.112:0/1947217983 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0d74006780 0x7f0d74008c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:16.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:16.866 7f0d8d2bd700 1 -- 172.21.2.112:0/1947217983 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0d74003af0 msgr2=0x7f0d74005f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:16.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:16.866 7f0d8d2bd700 1 --2- 172.21.2.112:0/1947217983 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0d74003af0 0x7f0d74005f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:16.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:16.866 7f0d8d2bd700 1 -- 172.21.2.112:0/1947217983 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d740509a0 con 0x7f0d74009580 2021-02-10T11:33:16.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:16.866 7f0d8d2bd700 1 -- 172.21.2.112:0/1947217983 reap_dead start 2021-02-10T11:33:16.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:16.866 7f0d8d2bd700 1 --2- 172.21.2.112:0/1947217983 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0d74009580 0x7f0d7400baa0 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 rx=0x7f0d80006ac0 tx=0x7f0d80009af0).ready entity=mon.0 client_cookie=29ed6b741c9e19d6 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:16.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:16.866 7f0d877fe700 1 -- 172.21.2.112:0/1947217983 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0d8000c0c0 con 0x7f0d74009580 2021-02-10T11:33:16.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:16.866 7f0d877fe700 1 -- 172.21.2.112:0/1947217983 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d8000c3c0 con 0x7f0d74009580 2021-02-10T11:33:16.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:16.866 7f0d877fe700 1 -- 172.21.2.112:0/1947217983 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0d8000c820 con 0x7f0d74009580 2021-02-10T11:33:16.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:16.866 7f0d8eca0700 1 -- 172.21.2.112:0/1947217983 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0d74009580 msgr2=0x7f0d7400baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:16.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:16.866 7f0d8eca0700 1 --2- 172.21.2.112:0/1947217983 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0d74009580 0x7f0d7400baa0 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 rx=0x7f0d80006ac0 tx=0x7f0d80009af0).stop 2021-02-10T11:33:16.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:16.867 7f0d8eca0700 1 -- 172.21.2.112:0/1947217983 shutdown_connections 2021-02-10T11:33:16.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:16.867 7f0d8eca0700 1 --2- 172.21.2.112:0/1947217983 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0d74009580 0x7f0d7400baa0 unknown :-1 s=CLOSED pgs=1211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:16.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:16.867 7f0d8eca0700 1 -- 172.21.2.112:0/1947217983 >> 172.21.2.112:0/1947217983 conn(0x7f0d880ce410 msgr2=0x7f0d880d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:16.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:16.867 7f0d8eca0700 1 -- 172.21.2.112:0/1947217983 shutdown_connections 2021-02-10T11:33:16.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:16.867 7f0d8eca0700 1 -- 172.21.2.112:0/1947217983 wait complete. 2021-02-10T11:33:16.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:16.881 7f0d8eca0700 1 Processor -- start 2021-02-10T11:33:16.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:16.882 7f0d8eca0700 1 -- start start 2021-02-10T11:33:16.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:16.882 7f0d8eca0700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0d881a05d0 0x7f0d880bb0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:16.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:16.882 7f0d8eca0700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0d8818f480 0x7f0d8818f880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:16.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:16.882 7f0d8eca0700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0d88190200 0x7f0d88190600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:16.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:16.882 7f0d8eca0700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d880bb760 con 0x7f0d88190200 2021-02-10T11:33:16.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:16.882 7f0d8eca0700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d880babb0 con 0x7f0d881a05d0 2021-02-10T11:33:16.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:16.882 7f0d8eca0700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d880b7680 con 0x7f0d8818f480 2021-02-10T11:33:16.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:16.882 7f0d8d2bd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0d88190200 0x7f0d88190600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:16.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:16.882 7f0d8d2bd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0d88190200 0x7f0d88190600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:49570/0 (socket says 172.21.2.112:49570) 2021-02-10T11:33:16.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:16.882 7f0d8d2bd700 1 -- 172.21.2.112:0/1300963954 learned_addr learned my addr 172.21.2.112:0/1300963954 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:33:16.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:16.883 7f0d8d2bd700 1 -- 172.21.2.112:0/1300963954 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0d8818f480 msgr2=0x7f0d8818f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:16.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:16.883 7f0d8d2bd700 1 --2- 172.21.2.112:0/1300963954 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0d8818f480 0x7f0d8818f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:16.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:16.883 7f0d8d2bd700 1 -- 172.21.2.112:0/1300963954 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0d881a05d0 msgr2=0x7f0d880bb0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:16.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:16.883 7f0d8d2bd700 1 --2- 172.21.2.112:0/1300963954 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0d881a05d0 0x7f0d880bb0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:16.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:16.883 7f0d8d2bd700 1 -- 172.21.2.112:0/1300963954 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d880b78a0 con 0x7f0d88190200 2021-02-10T11:33:16.875 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:16.883 7f0d8d2bd700 1 --2- 172.21.2.112:0/1300963954 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0d88190200 0x7f0d88190600 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 rx=0x7f0d800069e0 tx=0x7f0d8000b9e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:16.875 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:16.883 7f0d857fa700 1 -- 172.21.2.112:0/1300963954 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0d80000a10 con 0x7f0d88190200 2021-02-10T11:33:16.875 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:16.883 7f0d857fa700 1 -- 172.21.2.112:0/1300963954 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d80006240 con 0x7f0d88190200 2021-02-10T11:33:16.875 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:16.883 7f0d8eca0700 1 -- 172.21.2.112:0/1300963954 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d880b78a0 con 0x7f0d88190200 2021-02-10T11:33:16.875 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:16.883 7f0d8eca0700 1 -- 172.21.2.112:0/1300963954 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d880b7e30 con 0x7f0d88190200 2021-02-10T11:33:16.875 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:16.883 7f0d857fa700 1 -- 172.21.2.112:0/1300963954 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0d80005730 con 0x7f0d88190200 2021-02-10T11:33:16.875 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:16.884 7f0d857fa700 1 -- 172.21.2.112:0/1300963954 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0d80005c00 con 0x7f0d88190200 2021-02-10T11:33:16.875 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:16.884 7f0d857fa700 1 --2- 172.21.2.112:0/1300963954 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0d6c0186c0 0x7f0d6c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:16.875 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:16.884 7f0d857fa700 1 -- 172.21.2.112:0/1300963954 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5221+0+0 (secure 0 0 0) 0x7f0d80032480 con 0x7f0d88190200 2021-02-10T11:33:16.875 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:16.884 7f0d8cabc700 1 --2- 172.21.2.112:0/1300963954 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0d6c0186c0 0x7f0d6c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:16.875 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:16.885 7f0d8cabc700 1 --2- 172.21.2.112:0/1300963954 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0d6c0186c0 0x7f0d6c01abc0 crc :-1 s=READY pgs=607 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:16.901 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:16.914 7f0d8eca0700 1 -- 172.21.2.112:0/1300963954 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d60002fd0 con 0x7f0d88190200 2021-02-10T11:33:16.903 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:16.916 7f0d857fa700 1 -- 172.21.2.112:0/1300963954 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0d800059d0 con 0x7f0d88190200 2021-02-10T11:33:17.029 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:17.042 7f0d8eca0700 1 -- 172.21.2.112:0/1300963954 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f0d60002ef0 con 0x7f0d88190200 2021-02-10T11:33:17.407 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:17.420 7f0d857fa700 1 -- 172.21.2.112:0/1300963954 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v49) v1 ==== 173+0+0 (secure 0 0 0) 0x7f0d80005c00 con 0x7f0d88190200 2021-02-10T11:33:17.444 INFO:teuthology.orchestra.run.gibba012.stderr:pool 'cephfs_data' removed 2021-02-10T11:33:17.446 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:17.459 7f0d8eca0700 1 -- 172.21.2.112:0/1300963954 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0d6c0186c0 msgr2=0x7f0d6c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:17.446 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:17.459 7f0d8eca0700 1 --2- 172.21.2.112:0/1300963954 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0d6c0186c0 0x7f0d6c01abc0 crc :-1 s=READY pgs=607 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:17.446 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:17.459 7f0d8eca0700 1 -- 172.21.2.112:0/1300963954 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0d88190200 msgr2=0x7f0d88190600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:17.446 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:17.459 7f0d8eca0700 1 --2- 172.21.2.112:0/1300963954 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0d88190200 0x7f0d88190600 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 rx=0x7f0d800069e0 tx=0x7f0d8000b9e0).stop 2021-02-10T11:33:17.447 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:17.459 7f0d8eca0700 1 -- 172.21.2.112:0/1300963954 shutdown_connections 2021-02-10T11:33:17.447 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:17.459 7f0d8eca0700 1 --2- 172.21.2.112:0/1300963954 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0d6c0186c0 0x7f0d6c01abc0 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:17.447 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:17.459 7f0d8eca0700 1 --2- 172.21.2.112:0/1300963954 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0d88190200 0x7f0d88190600 unknown :-1 s=CLOSED pgs=1212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:17.447 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:17.459 7f0d8eca0700 1 --2- 172.21.2.112:0/1300963954 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0d8818f480 0x7f0d8818f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:17.447 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:17.459 7f0d8eca0700 1 --2- 172.21.2.112:0/1300963954 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0d881a05d0 0x7f0d880bb0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:17.447 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:17.459 7f0d8eca0700 1 -- 172.21.2.112:0/1300963954 >> 172.21.2.112:0/1300963954 conn(0x7f0d880873e0 msgr2=0x7f0d880b5010 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:17.447 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:17.460 7f0d8eca0700 1 -- 172.21.2.112:0/1300963954 shutdown_connections 2021-02-10T11:33:17.447 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:17.460 7f0d8eca0700 1 -- 172.21.2.112:0/1300963954 wait complete. 2021-02-10T11:33:17.455 INFO:tasks.ceph.mds.a:Restarting daemon 2021-02-10T11:33:17.455 INFO:tasks.ceph.mds.a:Stopping old one... 2021-02-10T11:33:17.455 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2021-02-10T11:33:17.456 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:33:17.456 INFO:tasks.ceph.mds.b:Restarting daemon 2021-02-10T11:33:17.456 INFO:tasks.ceph.mds.b:Stopping old one... 2021-02-10T11:33:17.456 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2021-02-10T11:33:17.456 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:33:17.461 INFO:tasks.ceph.mds.a:Stopped 2021-02-10T11:33:17.462 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2021-02-10T11:33:17.463 INFO:tasks.ceph.mds.b:Stopped 2021-02-10T11:33:17.463 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2021-02-10T11:33:17.503 INFO:tasks.ceph.mds.a:Started 2021-02-10T11:33:17.503 DEBUG:teuthology.parallel:result is None 2021-02-10T11:33:17.505 INFO:tasks.ceph.mds.b:Started 2021-02-10T11:33:17.505 DEBUG:teuthology.parallel:result is None 2021-02-10T11:33:17.505 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd blacklist clear 2021-02-10T11:33:17.909 INFO:tasks.ceph.mds.b.gibba020.stdout:starting mds.b at 2021-02-10T11:33:17.909 INFO:tasks.ceph.mds.a.gibba012.stdout:starting mds.a at 2021-02-10T11:33:17.929 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:17.925 7f1a5356f700 1 Processor -- start 2021-02-10T11:33:17.929 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:17.925 7f1a5356f700 1 -- start start 2021-02-10T11:33:17.929 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:17.926 7f1a5356f700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f1a4c0e5070 0x7f1a4c1b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:17.929 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:17.926 7f1a5356f700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f1a4c1b3e40 0x7f1a4c07d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:17.930 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:17.926 7f1a5356f700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a4c07ed80 con 0x7f1a4c0e5470 2021-02-10T11:33:17.930 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:17.926 7f1a5356f700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a4c080310 con 0x7f1a4c0e5070 2021-02-10T11:33:17.930 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:17.926 7f1a5356f700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a4c080530 con 0x7f1a4c1b3e40 2021-02-10T11:33:17.930 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:17.926 7f1a5138b700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f1a4c0e5070 0x7f1a4c1b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:17.930 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:17.926 7f1a50b8a700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f1a4c1b3e40 0x7f1a4c07d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:17.930 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:17.926 7f1a5138b700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f1a4c0e5070 0x7f1a4c1b3820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:49586/0 (socket says 172.21.2.112:49586) 2021-02-10T11:33:17.930 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:17.926 7f1a5138b700 1 -- 172.21.2.112:0/2864993170 learned_addr learned my addr 172.21.2.112:0/2864993170 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:33:17.930 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:17.926 7f1a5138b700 1 -- 172.21.2.112:0/2864993170 >> v2:172.21.2.120:3300/0 conn(0x7f1a4c1b3e40 msgr2=0x7f1a4c07d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:17.930 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:17.926 7f1a5138b700 1 --2- 172.21.2.112:0/2864993170 >> v2:172.21.2.120:3300/0 conn(0x7f1a4c1b3e40 0x7f1a4c07d690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:17.930 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:17.926 7f1a5138b700 1 -- 172.21.2.112:0/2864993170 >> v1:172.21.2.120:6789/0 conn(0x7f1a4c0e5470 legacy=0x7f1a4c07dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:17.930 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:17.926 7f1a5138b700 1 -- 172.21.2.112:0/2864993170 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a4c080750 con 0x7f1a4c0e5070 2021-02-10T11:33:17.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:17.926 7f1a5138b700 1 --2- 172.21.2.112:0/2864993170 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1a4c0e5070 0x7f1a4c1b3820 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 rx=0x7f1a48006700 tx=0x7f1a48009b80).ready entity=mon.0 client_cookie=d24e773271b0b44f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:17.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:17.927 7f1a43fff700 1 -- 172.21.2.112:0/2864993170 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1a4800c0c0 con 0x7f1a4c0e5070 2021-02-10T11:33:17.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:17.927 7f1a43fff700 1 -- 172.21.2.112:0/2864993170 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a4800c440 con 0x7f1a4c0e5070 2021-02-10T11:33:17.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:17.927 7f1a43fff700 1 -- 172.21.2.112:0/2864993170 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1a4800c8a0 con 0x7f1a4c0e5070 2021-02-10T11:33:17.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:17.927 7f1a5356f700 1 -- 172.21.2.112:0/2864993170 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1a4c0e5070 msgr2=0x7f1a4c1b3820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:17.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:17.927 7f1a5356f700 1 --2- 172.21.2.112:0/2864993170 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1a4c0e5070 0x7f1a4c1b3820 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 rx=0x7f1a48006700 tx=0x7f1a48009b80).stop 2021-02-10T11:33:17.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:17.927 7f1a5356f700 1 -- 172.21.2.112:0/2864993170 shutdown_connections 2021-02-10T11:33:17.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:17.927 7f1a5356f700 1 --2- 172.21.2.112:0/2864993170 >> v2:172.21.2.120:3300/0 conn(0x7f1a4c1b3e40 0x7f1a4c07d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:17.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:17.927 7f1a5356f700 1 --2- 172.21.2.112:0/2864993170 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1a4c0e5070 0x7f1a4c1b3820 unknown :-1 s=CLOSED pgs=1215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:17.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:17.927 7f1a5356f700 1 -- 172.21.2.112:0/2864993170 >> 172.21.2.112:0/2864993170 conn(0x7f1a4c0ce690 msgr2=0x7f1a4c0e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:17.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:17.927 7f1a5356f700 1 -- 172.21.2.112:0/2864993170 shutdown_connections 2021-02-10T11:33:17.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:17.927 7f1a5356f700 1 -- 172.21.2.112:0/2864993170 wait complete. 2021-02-10T11:33:17.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:17.942 7f1a5356f700 1 Processor -- start 2021-02-10T11:33:17.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:17.942 7f1a5356f700 1 -- start start 2021-02-10T11:33:17.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:17.943 7f1a5356f700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1a4c0b4ad0 0x7f1a4c0a78d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:17.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:17.943 7f1a5356f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1a4c0a7ef0 0x7f1a4c1a6d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:17.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:17.943 7f1a5356f700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1a4c0a82f0 0x7f1a4c1a73b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:17.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:17.943 7f1a5356f700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a4c0aa3c0 con 0x7f1a4c0a7ef0 2021-02-10T11:33:17.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:17.943 7f1a5356f700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a4c0a88c0 con 0x7f1a4c0a82f0 2021-02-10T11:33:17.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:17.943 7f1a5356f700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a4c0b4ed0 con 0x7f1a4c0b4ad0 2021-02-10T11:33:17.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:17.943 7f1a50b8a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1a4c0a7ef0 0x7f1a4c1a6d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:17.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:17.943 7f1a50b8a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1a4c0a7ef0 0x7f1a4c1a6d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:49590/0 (socket says 172.21.2.112:49590) 2021-02-10T11:33:17.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:17.943 7f1a50b8a700 1 -- 172.21.2.112:0/2753070419 learned_addr learned my addr 172.21.2.112:0/2753070419 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:33:17.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:17.943 7f1a5138b700 1 --2- 172.21.2.112:0/2753070419 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1a4c0b4ad0 0x7f1a4c0a78d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:17.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:17.943 7f1a51b8c700 1 --2- 172.21.2.112:0/2753070419 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1a4c0a82f0 0x7f1a4c1a73b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:17.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:17.943 7f1a50b8a700 1 -- 172.21.2.112:0/2753070419 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1a4c0b4ad0 msgr2=0x7f1a4c0a78d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:17.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:17.943 7f1a50b8a700 1 --2- 172.21.2.112:0/2753070419 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1a4c0b4ad0 0x7f1a4c0a78d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:17.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:17.943 7f1a50b8a700 1 -- 172.21.2.112:0/2753070419 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1a4c0a82f0 msgr2=0x7f1a4c1a73b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:17.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:17.943 7f1a50b8a700 1 --2- 172.21.2.112:0/2753070419 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1a4c0a82f0 0x7f1a4c1a73b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:17.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:17.943 7f1a50b8a700 1 -- 172.21.2.112:0/2753070419 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a4c1a79d0 con 0x7f1a4c0a7ef0 2021-02-10T11:33:17.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:17.943 7f1a51b8c700 1 --2- 172.21.2.112:0/2753070419 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1a4c0a82f0 0x7f1a4c1a73b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:33:17.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:17.943 7f1a50b8a700 1 --2- 172.21.2.112:0/2753070419 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1a4c0a7ef0 0x7f1a4c1a6d90 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 rx=0x7f1a3c006f60 tx=0x7f1a3c00b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:17.935 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:17.943 7f1a41ffb700 1 -- 172.21.2.112:0/2753070419 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1a3c00bd60 con 0x7f1a4c0a7ef0 2021-02-10T11:33:17.935 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:17.943 7f1a41ffb700 1 -- 172.21.2.112:0/2753070419 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a3c000c90 con 0x7f1a4c0a7ef0 2021-02-10T11:33:17.935 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:17.943 7f1a41ffb700 1 -- 172.21.2.112:0/2753070419 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1a3c002220 con 0x7f1a4c0a7ef0 2021-02-10T11:33:17.935 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:17.943 7f1a5356f700 1 -- 172.21.2.112:0/2753070419 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a4c1a79d0 con 0x7f1a4c0a7ef0 2021-02-10T11:33:17.935 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:17.943 7f1a5356f700 1 -- 172.21.2.112:0/2753070419 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a4c1a79d0 con 0x7f1a4c0a7ef0 2021-02-10T11:33:17.935 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:17.944 7f1a41ffb700 1 -- 172.21.2.112:0/2753070419 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1a3c0052e0 con 0x7f1a4c0a7ef0 2021-02-10T11:33:17.935 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:17.944 7f1a41ffb700 1 --2- 172.21.2.112:0/2753070419 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1a2c0186d0 0x7f1a2c01abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:17.935 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:17.944 7f1a41ffb700 1 -- 172.21.2.112:0/2753070419 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 4839+0+0 (secure 0 0 0) 0x7f1a3c0324b0 con 0x7f1a4c0a7ef0 2021-02-10T11:33:17.935 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:17.944 7f1a5138b700 1 --2- 172.21.2.112:0/2753070419 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1a2c0186d0 0x7f1a2c01abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:17.935 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:17.945 7f1a5138b700 1 --2- 172.21.2.112:0/2753070419 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1a2c0186d0 0x7f1a2c01abd0 crc :-1 s=READY pgs=608 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:17.962 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:17.975 7f1a5356f700 1 -- 172.21.2.112:0/2753070419 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a4c084a90 con 0x7f1a4c0a7ef0 2021-02-10T11:33:17.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:17.978 7f1a41ffb700 1 -- 172.21.2.112:0/2753070419 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1a3c0052e0 con 0x7f1a4c0a7ef0 2021-02-10T11:33:18.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:18.118 7f1a5356f700 1 -- 172.21.2.112:0/2753070419 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7f1a4c07ed80 con 0x7f1a4c0a7ef0 2021-02-10T11:33:18.448 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:18.460 7f1a41ffb700 1 -- 172.21.2.112:0/2753070419 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v50) v1 ==== 97+0+0 (secure 0 0 0) 0x7f1a3c000c90 con 0x7f1a4c0a7ef0 2021-02-10T11:33:18.472 INFO:teuthology.orchestra.run.gibba012.stderr: removed all blacklist entries 2021-02-10T11:33:18.473 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:18.486 7f1a5356f700 1 -- 172.21.2.112:0/2753070419 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1a2c0186d0 msgr2=0x7f1a2c01abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:18.473 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:18.486 7f1a5356f700 1 --2- 172.21.2.112:0/2753070419 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1a2c0186d0 0x7f1a2c01abd0 crc :-1 s=READY pgs=608 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:18.473 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:18.486 7f1a5356f700 1 -- 172.21.2.112:0/2753070419 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1a4c0a7ef0 msgr2=0x7f1a4c1a6d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:18.473 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:18.486 7f1a5356f700 1 --2- 172.21.2.112:0/2753070419 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1a4c0a7ef0 0x7f1a4c1a6d90 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 rx=0x7f1a3c006f60 tx=0x7f1a3c00b970).stop 2021-02-10T11:33:18.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:18.487 7f1a5356f700 1 -- 172.21.2.112:0/2753070419 shutdown_connections 2021-02-10T11:33:18.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:18.487 7f1a5356f700 1 --2- 172.21.2.112:0/2753070419 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1a2c0186d0 0x7f1a2c01abd0 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:18.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:18.487 7f1a5356f700 1 --2- 172.21.2.112:0/2753070419 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1a4c0a82f0 0x7f1a4c1a73b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:18.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:18.487 7f1a5356f700 1 --2- 172.21.2.112:0/2753070419 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1a4c0a7ef0 0x7f1a4c1a6d90 unknown :-1 s=CLOSED pgs=1216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:18.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:18.487 7f1a5356f700 1 --2- 172.21.2.112:0/2753070419 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1a4c0b4ad0 0x7f1a4c0a78d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:18.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:18.487 7f1a5356f700 1 -- 172.21.2.112:0/2753070419 >> 172.21.2.112:0/2753070419 conn(0x7f1a4c0b5560 msgr2=0x7f1a4c0a6200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:18.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:18.487 7f1a5356f700 1 -- 172.21.2.112:0/2753070419 shutdown_connections 2021-02-10T11:33:18.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:18.487 7f1a5356f700 1 -- 172.21.2.112:0/2753070419 wait complete. 2021-02-10T11:33:18.482 INFO:tasks.cephfs.cephfs_test_case:['0', '1', '2', '3'] 2021-02-10T11:33:18.482 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth ls --format=json-pretty 2021-02-10T11:33:18.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:18.904 7fd4d6a75700 1 Processor -- start 2021-02-10T11:33:18.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:18.904 7fd4d6a75700 1 -- start start 2021-02-10T11:33:18.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:18.905 7fd4d6a75700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fd4d00e5070 0x7fd4d01b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:18.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:18.905 7fd4d6a75700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4d007ed80 con 0x7fd4d00e5470 2021-02-10T11:33:18.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:18.905 7fd4d6a75700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4d007e670 con 0x7fd4d01b3e40 2021-02-10T11:33:18.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:18.905 7fd4d6a75700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4d007e890 con 0x7fd4d00e5070 2021-02-10T11:33:18.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:18.905 7fd4d5092700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fd4d00e5470 0x7fd4d007de80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:49060/0 (socket says 172.21.2.112:49060) 2021-02-10T11:33:18.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:18.905 7fd4d5092700 1 -- 172.21.2.112:0/3345811698 learned_addr learned my addr 172.21.2.112:0/3345811698 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:33:18.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:18.905 7fd4d4891700 1 --2- 172.21.2.112:0/3345811698 >> v2:172.21.2.112:3300/0 conn(0x7fd4d00e5070 0x7fd4d01b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:18.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:18.905 7fd4cf7fe700 1 -- 172.21.2.112:0/3345811698 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1490539299 0 0) 0x7fd4c8001180 con 0x7fd4d00e5470 2021-02-10T11:33:18.909 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:18.905 7fd4cf7fe700 1 -- 172.21.2.112:0/3345811698 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4bc002fe0 con 0x7fd4d00e5470 2021-02-10T11:33:18.909 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:18.905 7fd4d4891700 1 -- 172.21.2.112:0/3345811698 >> v1:172.21.2.120:6789/0 conn(0x7fd4d01b3e40 legacy=0x7fd4d007d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:18.909 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:18.905 7fd4d4891700 1 -- 172.21.2.112:0/3345811698 >> v1:172.21.2.112:6789/0 conn(0x7fd4d00e5470 legacy=0x7fd4d007de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:18.909 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:18.906 7fd4d4891700 1 -- 172.21.2.112:0/3345811698 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4d007eab0 con 0x7fd4d00e5070 2021-02-10T11:33:18.909 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:18.906 7fd4d4891700 1 --2- 172.21.2.112:0/3345811698 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd4d00e5070 0x7fd4d01b3820 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 rx=0x7fd4b8006650 tx=0x7fd4b8009a80).ready entity=mon.0 client_cookie=e0c9bc0b358db4e4 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:18.909 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:18.906 7fd4cf7fe700 1 -- 172.21.2.112:0/3345811698 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd4b800c0c0 con 0x7fd4d00e5070 2021-02-10T11:33:18.909 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:18.906 7fd4cf7fe700 1 -- 172.21.2.112:0/3345811698 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4b800c340 con 0x7fd4d00e5070 2021-02-10T11:33:18.909 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:18.906 7fd4cf7fe700 1 -- 172.21.2.112:0/3345811698 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd4b800c860 con 0x7fd4d00e5070 2021-02-10T11:33:18.909 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:18.906 7fd4d6a75700 1 -- 172.21.2.112:0/3345811698 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd4d00e5070 msgr2=0x7fd4d01b3820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:18.909 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:18.906 7fd4d6a75700 1 --2- 172.21.2.112:0/3345811698 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd4d00e5070 0x7fd4d01b3820 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 rx=0x7fd4b8006650 tx=0x7fd4b8009a80).stop 2021-02-10T11:33:18.909 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:18.906 7fd4d6a75700 1 -- 172.21.2.112:0/3345811698 shutdown_connections 2021-02-10T11:33:18.909 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:18.906 7fd4d6a75700 1 --2- 172.21.2.112:0/3345811698 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd4d00e5070 0x7fd4d01b3820 unknown :-1 s=CLOSED pgs=1218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:18.910 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:18.906 7fd4d6a75700 1 -- 172.21.2.112:0/3345811698 >> 172.21.2.112:0/3345811698 conn(0x7fd4d00ce690 msgr2=0x7fd4d00e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:18.910 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:18.907 7fd4d6a75700 1 -- 172.21.2.112:0/3345811698 shutdown_connections 2021-02-10T11:33:18.910 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:18.907 7fd4d6a75700 1 -- 172.21.2.112:0/3345811698 wait complete. 2021-02-10T11:33:18.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:18.921 7fd4d6a75700 1 Processor -- start 2021-02-10T11:33:18.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:18.921 7fd4d6a75700 1 -- start start 2021-02-10T11:33:18.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:18.922 7fd4d6a75700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd4d01b19d0 0x7fd4d008cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:18.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:18.922 7fd4d6a75700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd4d01b0d00 0x7fd4d01b1100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:18.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:18.922 7fd4d6a75700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd4d01a2b50 0x7fd4d01a2f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:18.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:18.922 7fd4d6a75700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4d01b1720 con 0x7fd4d01b19d0 2021-02-10T11:33:18.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:18.922 7fd4d6a75700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4d008c470 con 0x7fd4d01b0d00 2021-02-10T11:33:18.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:18.922 7fd4d6a75700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4d01b1f50 con 0x7fd4d01a2b50 2021-02-10T11:33:18.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:18.922 7fd4d4891700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd4d01b19d0 0x7fd4d008cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:18.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:18.922 7fd4d4891700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd4d01b19d0 0x7fd4d008cd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:49604/0 (socket says 172.21.2.112:49604) 2021-02-10T11:33:18.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:18.922 7fd4d4891700 1 -- 172.21.2.112:0/1727404165 learned_addr learned my addr 172.21.2.112:0/1727404165 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:33:18.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:18.922 7fd4d4891700 1 -- 172.21.2.112:0/1727404165 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd4d01a2b50 msgr2=0x7fd4d01a2f50 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:33:18.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:18.922 7fd4d4891700 1 --2- 172.21.2.112:0/1727404165 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd4d01a2b50 0x7fd4d01a2f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:18.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:18.922 7fd4d4891700 1 -- 172.21.2.112:0/1727404165 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd4d01b0d00 msgr2=0x7fd4d01b1100 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:33:18.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:18.922 7fd4d4891700 1 --2- 172.21.2.112:0/1727404165 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd4d01b0d00 0x7fd4d01b1100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:18.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:18.922 7fd4d4891700 1 -- 172.21.2.112:0/1727404165 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4d01a3570 con 0x7fd4d01b19d0 2021-02-10T11:33:18.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:18.922 7fd4d4891700 1 --2- 172.21.2.112:0/1727404165 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd4d01b19d0 0x7fd4d008cd90 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 rx=0x7fd4b8009630 tx=0x7fd4b800b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:18.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:18.923 7fd4cd7fa700 1 -- 172.21.2.112:0/1727404165 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd4b80093a0 con 0x7fd4d01b19d0 2021-02-10T11:33:18.913 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:18.923 7fd4cd7fa700 1 -- 172.21.2.112:0/1727404165 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4b800bd70 con 0x7fd4d01b19d0 2021-02-10T11:33:18.913 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:18.923 7fd4cd7fa700 1 -- 172.21.2.112:0/1727404165 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd4b80054d0 con 0x7fd4d01b19d0 2021-02-10T11:33:18.913 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:18.923 7fd4d6a75700 1 -- 172.21.2.112:0/1727404165 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4d01a3570 con 0x7fd4d01b19d0 2021-02-10T11:33:18.913 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:18.923 7fd4d6a75700 1 -- 172.21.2.112:0/1727404165 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4d00b3330 con 0x7fd4d01b19d0 2021-02-10T11:33:18.913 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:18.923 7fd4cd7fa700 1 -- 172.21.2.112:0/1727404165 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd4b8007020 con 0x7fd4d01b19d0 2021-02-10T11:33:18.913 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:18.923 7fd4cd7fa700 1 --2- 172.21.2.112:0/1727404165 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd4b40186c0 0x7fd4b401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:18.913 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:18.923 7fd4cd7fa700 1 -- 172.21.2.112:0/1727404165 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 4753+0+0 (secure 0 0 0) 0x7fd4b8032250 con 0x7fd4d01b19d0 2021-02-10T11:33:18.913 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:18.924 7fd4cffff700 1 --2- 172.21.2.112:0/1727404165 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd4b40186c0 0x7fd4b401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:18.914 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:18.925 7fd4cffff700 1 --2- 172.21.2.112:0/1727404165 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd4b40186c0 0x7fd4b401abc0 crc :-1 s=READY pgs=609 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:18.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:18.955 7fd4d6a75700 1 -- 172.21.2.112:0/1727404165 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4d0083dd0 con 0x7fd4d01b19d0 2021-02-10T11:33:18.944 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:18.957 7fd4cd7fa700 1 -- 172.21.2.112:0/1727404165 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd4b8009340 con 0x7fd4d01b19d0 2021-02-10T11:33:19.076 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:19.089 7fd4d6a75700 1 -- 172.21.2.112:0/1727404165 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fd4d0083dd0 con 0x7fd4d01b19d0 2021-02-10T11:33:19.076 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:19.089 7fd4cd7fa700 1 -- 172.21.2.112:0/1727404165 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v30) v1 ==== 80+0+4513 (secure 0 0 0) 0x7fd4b8007000 con 0x7fd4d01b19d0 2021-02-10T11:33:19.076 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:33:19.077 INFO:teuthology.orchestra.run.gibba012.stdout:{ 2021-02-10T11:33:19.077 INFO:teuthology.orchestra.run.gibba012.stdout: "auth_dump": [ 2021-02-10T11:33:19.077 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:19.077 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "mds.a", 2021-02-10T11:33:19.077 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCKvyNgPkV4AxAAEmVNaghdccP6lWd8hvQnxw==", 2021-02-10T11:33:19.077 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:33:19.077 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:33:19.077 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:33:19.077 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:33:19.077 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:33:19.077 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:33:19.077 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:19.078 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:19.078 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "mds.b", 2021-02-10T11:33:19.078 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCKvyNguxONLhAAHIpry+R89MAwdhFJyEhVCw==", 2021-02-10T11:33:19.078 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:33:19.078 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:33:19.078 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:33:19.078 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:33:19.078 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:33:19.078 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:33:19.078 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:19.078 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:19.078 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.0", 2021-02-10T11:33:19.078 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCSvyNgXcKPLhAAP/Bj2F+06DCtPCzQDNq+cQ==", 2021-02-10T11:33:19.079 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:33:19.079 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:33:19.079 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:33:19.079 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:33:19.079 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:33:19.079 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:19.079 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:19.079 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.1", 2021-02-10T11:33:19.079 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCVvyNghfbcLhAA66PrRwR2M7o6VTF5UhU/Fg==", 2021-02-10T11:33:19.080 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:33:19.080 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:33:19.080 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:33:19.080 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:33:19.080 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:33:19.080 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:19.080 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:19.080 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.2", 2021-02-10T11:33:19.080 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCYvyNgwiPqLhAA66BVkRsgaUlGpAsQ15Nf8Q==", 2021-02-10T11:33:19.080 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:33:19.080 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:33:19.080 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:33:19.081 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:33:19.081 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:33:19.081 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:19.081 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:19.081 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.3", 2021-02-10T11:33:19.081 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCbvyNgqBZrLxAAERPe/1L3SXAEzI3d9ReHHg==", 2021-02-10T11:33:19.081 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:33:19.081 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:33:19.081 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:33:19.081 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:33:19.081 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:33:19.081 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:19.081 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:19.081 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.4", 2021-02-10T11:33:19.082 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCkvyNgDQsxIhAAHmVjVYnx20N4SDmKqvbcuA==", 2021-02-10T11:33:19.082 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:33:19.082 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:33:19.082 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:33:19.082 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:33:19.082 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:33:19.082 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:19.082 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:19.082 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.5", 2021-02-10T11:33:19.082 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCnvyNg8ITFIhAADd4aKybAV7btoGPx3kCg3A==", 2021-02-10T11:33:19.082 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:33:19.082 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:33:19.082 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:33:19.083 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:33:19.083 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:33:19.083 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:19.083 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:19.083 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.6", 2021-02-10T11:33:19.083 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCqvyNgxOhDIxAAzXixEa/nlDc1xXrg0aeLwg==", 2021-02-10T11:33:19.083 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:33:19.083 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:33:19.083 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:33:19.083 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:33:19.083 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:33:19.083 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:19.083 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:19.084 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.7", 2021-02-10T11:33:19.084 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCtvyNgxSqGIxAACxJQlwP7DpiVVCZHA3zn1A==", 2021-02-10T11:33:19.084 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:33:19.084 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:33:19.084 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:33:19.084 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:33:19.084 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:33:19.084 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:19.084 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:19.084 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.0", 2021-02-10T11:33:19.084 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCLvyNgDkNiHRAAygU8Za4PBusTqCgwAYWTiQ==", 2021-02-10T11:33:19.084 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:33:19.084 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:33:19.085 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow r", 2021-02-10T11:33:19.085 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:33:19.085 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:33:19.085 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:19.085 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:19.085 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.1", 2021-02-10T11:33:19.085 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCLvyNgxmLqMxAA1MlSDJ9LH9u4f2sL7mm8/Q==", 2021-02-10T11:33:19.085 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:33:19.085 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:33:19.085 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow r", 2021-02-10T11:33:19.085 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:33:19.085 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:33:19.085 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:19.085 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:19.086 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.2", 2021-02-10T11:33:19.086 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCMvyNgp8veDhAAniSVHYPtwwTog/pjbaRQmw==", 2021-02-10T11:33:19.086 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:33:19.086 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:33:19.086 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow r", 2021-02-10T11:33:19.086 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:33:19.086 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:33:19.086 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:19.086 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:19.086 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.3", 2021-02-10T11:33:19.086 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCMvyNgQEVoJRAApHcWdm1sldIxu+heyD4HdQ==", 2021-02-10T11:33:19.086 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:33:19.086 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:33:19.087 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow r", 2021-02-10T11:33:19.087 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:33:19.087 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:33:19.087 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:19.087 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:19.087 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.admin", 2021-02-10T11:33:19.087 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCGvyNg7jjJBRAAtot+ypUZOiTo1GbY+oVhVw==", 2021-02-10T11:33:19.087 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:33:19.087 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow *", 2021-02-10T11:33:19.087 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:33:19.087 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:33:19.087 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:33:19.087 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:33:19.088 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:19.088 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:19.088 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "mgr.x", 2021-02-10T11:33:19.088 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCJvyNgTPVwKBAAjduiBE8+RSXBv82tdPecWg==", 2021-02-10T11:33:19.088 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:33:19.088 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow *", 2021-02-10T11:33:19.088 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow profile mgr", 2021-02-10T11:33:19.088 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:33:19.088 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:33:19.088 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:19.088 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:19.088 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "mgr.y", 2021-02-10T11:33:19.088 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCJvyNgQ1tTDhAASwBssDy+Rz31ujw8fNlWow==", 2021-02-10T11:33:19.088 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:33:19.089 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow *", 2021-02-10T11:33:19.089 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow profile mgr", 2021-02-10T11:33:19.089 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:33:19.089 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:33:19.089 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:33:19.089 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:33:19.089 INFO:teuthology.orchestra.run.gibba012.stdout:} 2021-02-10T11:33:19.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:19.091 7fd4d6a75700 1 -- 172.21.2.112:0/1727404165 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd4b40186c0 msgr2=0x7fd4b401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:19.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:19.091 7fd4d6a75700 1 --2- 172.21.2.112:0/1727404165 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd4b40186c0 0x7fd4b401abc0 crc :-1 s=READY pgs=609 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:19.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:19.091 7fd4d6a75700 1 -- 172.21.2.112:0/1727404165 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd4d01b19d0 msgr2=0x7fd4d008cd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:19.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:19.091 7fd4d6a75700 1 --2- 172.21.2.112:0/1727404165 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd4d01b19d0 0x7fd4d008cd90 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 rx=0x7fd4b8009630 tx=0x7fd4b800b980).stop 2021-02-10T11:33:19.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:19.091 7fd4d6a75700 1 -- 172.21.2.112:0/1727404165 shutdown_connections 2021-02-10T11:33:19.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:19.091 7fd4d6a75700 1 --2- 172.21.2.112:0/1727404165 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd4b40186c0 0x7fd4b401abc0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:19.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:19.091 7fd4d6a75700 1 --2- 172.21.2.112:0/1727404165 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd4d01b0d00 0x7fd4d01b1100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:19.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:19.091 7fd4d6a75700 1 --2- 172.21.2.112:0/1727404165 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd4d01a2b50 0x7fd4d01a2f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:19.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:19.091 7fd4d6a75700 1 --2- 172.21.2.112:0/1727404165 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd4d01b19d0 0x7fd4d008cd90 unknown :-1 s=CLOSED pgs=1219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:19.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:19.091 7fd4d6a75700 1 -- 172.21.2.112:0/1727404165 >> 172.21.2.112:0/1727404165 conn(0x7fd4d0093720 msgr2=0x7fd4d0084b80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:19.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:19.091 7fd4d6a75700 1 -- 172.21.2.112:0/1727404165 shutdown_connections 2021-02-10T11:33:19.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:19.091 7fd4d6a75700 1 -- 172.21.2.112:0/1727404165 wait complete. 2021-02-10T11:33:19.092 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:33:19.518 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:19.514 7f23bb43e700 1 Processor -- start 2021-02-10T11:33:19.518 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:19.514 7f23bb43e700 1 -- start start 2021-02-10T11:33:19.518 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:19.515 7f23bb43e700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f23b40e5470 0x7f23b407e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:19.518 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:19.515 7f23bb43e700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f23b407f300 0x7f23b407f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:19.518 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:19.515 7f23bb43e700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23b407ed80 con 0x7f23b40e5070 2021-02-10T11:33:19.518 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:19.515 7f23bb43e700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23b41b4010 con 0x7f23b407f300 2021-02-10T11:33:19.518 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:19.515 7f23bb43e700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23b40d1790 con 0x7f23b40e5470 2021-02-10T11:33:19.518 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:19.515 7f23b925a700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f23b40e5070 0x7f23b41b3820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:49072/0 (socket says 172.21.2.112:49072) 2021-02-10T11:33:19.518 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:19.515 7f23b925a700 1 -- 172.21.2.112:0/893853084 learned_addr learned my addr 172.21.2.112:0/893853084 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:33:19.519 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:19.515 7f23b9a5b700 1 --2- 172.21.2.112:0/893853084 >> v2:172.21.2.112:3300/0 conn(0x7f23b407f300 0x7f23b407f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:19.519 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:19.515 7f23a7fff700 1 -- 172.21.2.112:0/893853084 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2056534281 0 0) 0x7f23b0001180 con 0x7f23b40e5070 2021-02-10T11:33:19.519 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:19.515 7f23b9a5b700 1 -- 172.21.2.112:0/893853084 >> v2:172.21.2.120:3300/0 conn(0x7f23b40e5470 msgr2=0x7f23b407e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:19.519 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:19.515 7f23b9a5b700 1 --2- 172.21.2.112:0/893853084 >> v2:172.21.2.120:3300/0 conn(0x7f23b40e5470 0x7f23b407e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:19.519 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:19.515 7f23b9a5b700 1 -- 172.21.2.112:0/893853084 >> v1:172.21.2.112:6789/0 conn(0x7f23b40e5070 legacy=0x7f23b41b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:19.519 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:19.515 7f23b9a5b700 1 -- 172.21.2.112:0/893853084 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23b40d19b0 con 0x7f23b407f300 2021-02-10T11:33:19.519 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:19.516 7f23b9a5b700 1 --2- 172.21.2.112:0/893853084 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f23b407f300 0x7f23b407f700 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 rx=0x7f23ac006e30 tx=0x7f23ac009e60).ready entity=mon.0 client_cookie=4749c7bb5c89bcc6 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:19.519 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:19.516 7f23a7fff700 1 -- 172.21.2.112:0/893853084 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f23ac003030 con 0x7f23b407f300 2021-02-10T11:33:19.519 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:19.516 7f23a7fff700 1 -- 172.21.2.112:0/893853084 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23ac00b300 con 0x7f23b407f300 2021-02-10T11:33:19.519 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:19.516 7f23a7fff700 1 -- 172.21.2.112:0/893853084 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f23ac000f90 con 0x7f23b407f300 2021-02-10T11:33:19.519 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:19.516 7f23bb43e700 1 -- 172.21.2.112:0/893853084 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f23b407f300 msgr2=0x7f23b407f700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:19.519 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:19.516 7f23bb43e700 1 --2- 172.21.2.112:0/893853084 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f23b407f300 0x7f23b407f700 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 rx=0x7f23ac006e30 tx=0x7f23ac009e60).stop 2021-02-10T11:33:19.519 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:19.516 7f23bb43e700 1 -- 172.21.2.112:0/893853084 shutdown_connections 2021-02-10T11:33:19.520 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:19.516 7f23bb43e700 1 --2- 172.21.2.112:0/893853084 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f23b407f300 0x7f23b407f700 unknown :-1 s=CLOSED pgs=1221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:19.520 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:19.516 7f23bb43e700 1 --2- 172.21.2.112:0/893853084 >> v2:172.21.2.120:3300/0 conn(0x7f23b40e5470 0x7f23b407e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:19.520 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:19.516 7f23bb43e700 1 -- 172.21.2.112:0/893853084 >> 172.21.2.112:0/893853084 conn(0x7f23b40ce690 msgr2=0x7f23b40e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:19.520 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:19.517 7f23bb43e700 1 -- 172.21.2.112:0/893853084 shutdown_connections 2021-02-10T11:33:19.520 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:19.517 7f23bb43e700 1 -- 172.21.2.112:0/893853084 wait complete. 2021-02-10T11:33:19.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:19.531 7f23bb43e700 1 Processor -- start 2021-02-10T11:33:19.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:19.531 7f23bb43e700 1 -- start start 2021-02-10T11:33:19.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:19.532 7f23bb43e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f23b408d1c0 0x7f23b4081230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:19.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:19.532 7f23bb43e700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f23b4081850 0x7f23b40ad0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:19.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:19.532 7f23bb43e700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f23b40ad6f0 0x7f23b41a1d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:19.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:19.532 7f23bb43e700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23b4087740 con 0x7f23b408d1c0 2021-02-10T11:33:19.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:19.532 7f23bb43e700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23b4081db0 con 0x7f23b4081850 2021-02-10T11:33:19.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:19.532 7f23bb43e700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23b408d5c0 con 0x7f23b40ad6f0 2021-02-10T11:33:19.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:19.532 7f23b925a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f23b408d1c0 0x7f23b4081230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:19.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:19.532 7f23b925a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f23b408d1c0 0x7f23b4081230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:49618/0 (socket says 172.21.2.112:49618) 2021-02-10T11:33:19.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:19.532 7f23b925a700 1 -- 172.21.2.112:0/4240831127 learned_addr learned my addr 172.21.2.112:0/4240831127 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:33:19.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:19.532 7f23b925a700 1 -- 172.21.2.112:0/4240831127 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f23b40ad6f0 msgr2=0x7f23b41a1d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:19.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:19.532 7f23b925a700 1 --2- 172.21.2.112:0/4240831127 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f23b40ad6f0 0x7f23b41a1d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:19.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:19.532 7f23b925a700 1 -- 172.21.2.112:0/4240831127 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f23b4081850 msgr2=0x7f23b40ad0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:19.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:19.532 7f23b925a700 1 --2- 172.21.2.112:0/4240831127 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f23b4081850 0x7f23b40ad0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:19.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:19.532 7f23b925a700 1 -- 172.21.2.112:0/4240831127 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23b408da00 con 0x7f23b408d1c0 2021-02-10T11:33:19.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:19.532 7f23b925a700 1 --2- 172.21.2.112:0/4240831127 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f23b408d1c0 0x7f23b4081230 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 rx=0x7f23b000a570 tx=0x7f23b000a910).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:19.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:19.533 7f23a5ffb700 1 -- 172.21.2.112:0/4240831127 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f23b000ad00 con 0x7f23b408d1c0 2021-02-10T11:33:19.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:19.533 7f23a5ffb700 1 -- 172.21.2.112:0/4240831127 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23b0001180 con 0x7f23b408d1c0 2021-02-10T11:33:19.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:19.533 7f23a5ffb700 1 -- 172.21.2.112:0/4240831127 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f23b00070d0 con 0x7f23b408d1c0 2021-02-10T11:33:19.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:19.533 7f23bb43e700 1 -- 172.21.2.112:0/4240831127 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23b408da00 con 0x7f23b408d1c0 2021-02-10T11:33:19.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:19.533 7f23bb43e700 1 -- 172.21.2.112:0/4240831127 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23b40adde0 con 0x7f23b408d1c0 2021-02-10T11:33:19.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:19.533 7f23a5ffb700 1 -- 172.21.2.112:0/4240831127 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f23b0009030 con 0x7f23b408d1c0 2021-02-10T11:33:19.524 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:19.533 7f23a5ffb700 1 --2- 172.21.2.112:0/4240831127 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f23940186c0 0x7f239401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:19.524 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:19.533 7f23a5ffb700 1 -- 172.21.2.112:0/4240831127 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f23b0032210 con 0x7f23b408d1c0 2021-02-10T11:33:19.524 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:19.534 7f23b8a59700 1 --2- 172.21.2.112:0/4240831127 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f23940186c0 0x7f239401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:19.524 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:19.534 7f23b8a59700 1 --2- 172.21.2.112:0/4240831127 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f23940186c0 0x7f239401abc0 crc :-1 s=READY pgs=610 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:19.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:19.565 7f23bb43e700 1 -- 172.21.2.112:0/4240831127 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23b407ed80 con 0x7f23b408d1c0 2021-02-10T11:33:19.554 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:19.567 7f23a5ffb700 1 -- 172.21.2.112:0/4240831127 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f23b00070d0 con 0x7f23b408d1c0 2021-02-10T11:33:19.679 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:19.692 7f23bb43e700 1 -- 172.21.2.112:0/4240831127 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f23b407ed80 con 0x7f23b408d1c0 2021-02-10T11:33:19.680 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:19.693 7f23a5ffb700 1 -- 172.21.2.112:0/4240831127 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v50) v1 ==== 74+0+11144 (secure 0 0 0) 0x7f23b0009030 con 0x7f23b408d1c0 2021-02-10T11:33:19.680 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:33:19.681 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":50,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:33:18.457054","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:33:19.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:19.694 7f23bb43e700 1 -- 172.21.2.112:0/4240831127 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f23940186c0 msgr2=0x7f239401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:19.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:19.694 7f23bb43e700 1 --2- 172.21.2.112:0/4240831127 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f23940186c0 0x7f239401abc0 crc :-1 s=READY pgs=610 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:19.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:19.695 7f23bb43e700 1 -- 172.21.2.112:0/4240831127 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f23b408d1c0 msgr2=0x7f23b4081230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:19.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:19.695 7f23bb43e700 1 --2- 172.21.2.112:0/4240831127 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f23b408d1c0 0x7f23b4081230 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 rx=0x7f23b000a570 tx=0x7f23b000a910).stop 2021-02-10T11:33:19.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:19.695 7f23bb43e700 1 -- 172.21.2.112:0/4240831127 shutdown_connections 2021-02-10T11:33:19.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:19.695 7f23bb43e700 1 --2- 172.21.2.112:0/4240831127 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f23940186c0 0x7f239401abc0 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:19.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:19.695 7f23bb43e700 1 --2- 172.21.2.112:0/4240831127 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f23b4081850 0x7f23b40ad0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:19.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:19.695 7f23bb43e700 1 --2- 172.21.2.112:0/4240831127 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f23b40ad6f0 0x7f23b41a1d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:19.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:19.695 7f23bb43e700 1 --2- 172.21.2.112:0/4240831127 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f23b408d1c0 0x7f23b4081230 unknown :-1 s=CLOSED pgs=1222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:19.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:19.695 7f23bb43e700 1 -- 172.21.2.112:0/4240831127 >> 172.21.2.112:0/4240831127 conn(0x7f23b419b930 msgr2=0x7f23b41a0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:19.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:19.695 7f23bb43e700 1 -- 172.21.2.112:0/4240831127 shutdown_connections 2021-02-10T11:33:19.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:19.695 7f23bb43e700 1 -- 172.21.2.112:0/4240831127 wait complete. 2021-02-10T11:33:19.691 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:13:57.508758', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2021-02-10T11:33:19.691 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:33:20.123 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.119 7f4bcd3d2700 1 Processor -- start 2021-02-10T11:33:20.123 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.119 7f4bcd3d2700 1 -- start start 2021-02-10T11:33:20.123 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.119 7f4bcd3d2700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f4bc81b0870 0x7f4bc80f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:20.124 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.119 7f4bcd3d2700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4bc80efe00 0x7f4bc80f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:20.124 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.119 7f4bcd3d2700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bc81b0c70 con 0x7f4bc81b0470 2021-02-10T11:33:20.124 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.119 7f4bcd3d2700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bc80f0b50 con 0x7f4bc81b0870 2021-02-10T11:33:20.124 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.119 7f4bcd3d2700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bc80f0820 con 0x7f4bc80efe00 2021-02-10T11:33:20.124 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.119 7f4bc707d700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f4bc81b0470 0x7f4bc80ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:49086/0 (socket says 172.21.2.112:49086) 2021-02-10T11:33:20.124 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.119 7f4bc707d700 1 -- 172.21.2.112:0/1825243492 learned_addr learned my addr 172.21.2.112:0/1825243492 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:33:20.124 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.120 7f4bc607b700 1 -- 172.21.2.112:0/1825243492 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1556451633 0 0) 0x7f4bb80010b0 con 0x7f4bc81b0470 2021-02-10T11:33:20.124 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.120 7f4bc607b700 1 -- 172.21.2.112:0/1825243492 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4bb4002fe0 con 0x7f4bc81b0470 2021-02-10T11:33:20.124 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.120 7f4bc687c700 1 --2- 172.21.2.112:0/1825243492 >> v2:172.21.2.120:3300/0 conn(0x7f4bc81b0870 0x7f4bc80f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:20.124 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.120 7f4bc787e700 1 --2- 172.21.2.112:0/1825243492 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4bc80efe00 0x7f4bc80f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:20.124 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.120 7f4bc607b700 1 -- 172.21.2.112:0/1825243492 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4293379836 0 0) 0x7f4bb80010b0 con 0x7f4bc81b0470 2021-02-10T11:33:20.124 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.120 7f4bc607b700 1 -- 172.21.2.112:0/1825243492 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4bc80efe00 msgr2=0x7f4bc80f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:20.124 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.120 7f4bc607b700 1 --2- 172.21.2.112:0/1825243492 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4bc80efe00 0x7f4bc80f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:20.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.120 7f4bc607b700 1 -- 172.21.2.112:0/1825243492 >> v2:172.21.2.120:3300/0 conn(0x7f4bc81b0870 msgr2=0x7f4bc80f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:20.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.120 7f4bc607b700 1 --2- 172.21.2.112:0/1825243492 >> v2:172.21.2.120:3300/0 conn(0x7f4bc81b0870 0x7f4bc80f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:20.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.120 7f4bc607b700 1 -- 172.21.2.112:0/1825243492 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bc80e52e0 con 0x7f4bc81b0470 2021-02-10T11:33:20.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.120 7f4bc607b700 1 -- 172.21.2.112:0/1825243492 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f4bb80015b0 con 0x7f4bc81b0470 2021-02-10T11:33:20.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.120 7f4bc687c700 1 --2- 172.21.2.112:0/1825243492 >> v2:172.21.2.120:3300/0 conn(0x7f4bc81b0870 0x7f4bc80f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:33:20.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.120 7f4bc607b700 1 -- 172.21.2.112:0/1825243492 >> v1:172.21.2.112:6789/0 conn(0x7f4bc81b0470 legacy=0x7f4bc80ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:20.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.120 7f4bc607b700 1 --2- 172.21.2.112:0/1825243492 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4bb4003a60 0x7f4bb4005f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:20.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.120 7f4bc607b700 1 --2- 172.21.2.112:0/1825243492 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4bb40067d0 0x7f4bb4008c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:20.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.121 7f4bc607b700 1 --2- 172.21.2.112:0/1825243492 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4bb404d530 0x7f4bb404fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:20.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.121 7f4bc607b700 1 -- 172.21.2.112:0/1825243492 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bb40502e0 con 0x7f4bb40067d0 2021-02-10T11:33:20.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.121 7f4bc607b700 1 -- 172.21.2.112:0/1825243492 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bb4050500 con 0x7f4bb404d530 2021-02-10T11:33:20.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.121 7f4bc607b700 1 -- 172.21.2.112:0/1825243492 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bb4050720 con 0x7f4bb4003a60 2021-02-10T11:33:20.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.121 7f4bc687c700 1 --2- 172.21.2.112:0/1825243492 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4bb40067d0 0x7f4bb4008c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:20.126 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.121 7f4bc707d700 1 --2- 172.21.2.112:0/1825243492 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4bb4003a60 0x7f4bb4005f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:20.126 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.121 7f4bc687c700 1 -- 172.21.2.112:0/1825243492 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4bb4003a60 msgr2=0x7f4bb4005f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:20.126 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.121 7f4bc687c700 1 --2- 172.21.2.112:0/1825243492 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4bb4003a60 0x7f4bb4005f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:20.126 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.121 7f4bc687c700 1 -- 172.21.2.112:0/1825243492 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4bb404d530 msgr2=0x7f4bb404fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:20.126 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.121 7f4bc687c700 1 --2- 172.21.2.112:0/1825243492 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4bb404d530 0x7f4bb404fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:20.126 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.121 7f4bc687c700 1 -- 172.21.2.112:0/1825243492 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bb40509c0 con 0x7f4bb40067d0 2021-02-10T11:33:20.126 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.121 7f4bc687c700 1 --2- 172.21.2.112:0/1825243492 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4bb40067d0 0x7f4bb4008c80 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 rx=0x7f4bb0009960 tx=0x7f4bb0009d00).ready entity=mon.0 client_cookie=1f903632eb6c25c5 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:20.126 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.121 7f4bc607b700 1 -- 172.21.2.112:0/1825243492 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4bb0006ce0 con 0x7f4bb40067d0 2021-02-10T11:33:20.126 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.121 7f4bc607b700 1 -- 172.21.2.112:0/1825243492 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bb0006930 con 0x7f4bb40067d0 2021-02-10T11:33:20.126 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.121 7f4bc607b700 1 -- 172.21.2.112:0/1825243492 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4bb0002030 con 0x7f4bb40067d0 2021-02-10T11:33:20.126 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.121 7f4bcd3d2700 1 -- 172.21.2.112:0/1825243492 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4bb40067d0 msgr2=0x7f4bb4008c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:20.126 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.121 7f4bcd3d2700 1 --2- 172.21.2.112:0/1825243492 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4bb40067d0 0x7f4bb4008c80 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 rx=0x7f4bb0009960 tx=0x7f4bb0009d00).stop 2021-02-10T11:33:20.127 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.121 7f4bc787e700 1 -- 172.21.2.112:0/1825243492 reap_dead start 2021-02-10T11:33:20.127 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.121 7f4bcd3d2700 1 -- 172.21.2.112:0/1825243492 shutdown_connections 2021-02-10T11:33:20.127 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.121 7f4bcd3d2700 1 -- 172.21.2.112:0/1825243492 >> 172.21.2.112:0/1825243492 conn(0x7f4bc80ce3e0 msgr2=0x7f4bc80e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:20.127 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.122 7f4bcd3d2700 1 -- 172.21.2.112:0/1825243492 shutdown_connections 2021-02-10T11:33:20.127 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.122 7f4bcd3d2700 1 -- 172.21.2.112:0/1825243492 wait complete. 2021-02-10T11:33:20.127 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.136 7f4bcd3d2700 1 Processor -- start 2021-02-10T11:33:20.127 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.136 7f4bcd3d2700 1 -- start start 2021-02-10T11:33:20.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.137 7f4bcd3d2700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4bc809b930 0x7f4bc80b7c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:20.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.137 7f4bcd3d2700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4bc80b8410 0x7f4bc80b8810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:20.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.137 7f4bcd3d2700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4bc80c1f80 0x7f4bc80c2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:20.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.137 7f4bcd3d2700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bc80c1a00 con 0x7f4bc809b930 2021-02-10T11:33:20.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.137 7f4bcd3d2700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bc80b8e30 con 0x7f4bc80b8410 2021-02-10T11:33:20.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.137 7f4bcd3d2700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bc80b9050 con 0x7f4bc80c1f80 2021-02-10T11:33:20.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.137 7f4bc707d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4bc809b930 0x7f4bc80b7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:20.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.137 7f4bc707d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4bc809b930 0x7f4bc80b7c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:49636/0 (socket says 172.21.2.112:49636) 2021-02-10T11:33:20.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.137 7f4bc707d700 1 -- 172.21.2.112:0/598231065 learned_addr learned my addr 172.21.2.112:0/598231065 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:33:20.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.137 7f4bc707d700 1 -- 172.21.2.112:0/598231065 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4bc80c1f80 msgr2=0x7f4bc80c2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:20.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.137 7f4bc707d700 1 --2- 172.21.2.112:0/598231065 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4bc80c1f80 0x7f4bc80c2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:20.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.137 7f4bc707d700 1 -- 172.21.2.112:0/598231065 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4bc80b8410 msgr2=0x7f4bc80b8810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:20.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.137 7f4bc707d700 1 --2- 172.21.2.112:0/598231065 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4bc80b8410 0x7f4bc80b8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:20.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.137 7f4bc707d700 1 -- 172.21.2.112:0/598231065 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bc80c29a0 con 0x7f4bc809b930 2021-02-10T11:33:20.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.138 7f4bc707d700 1 --2- 172.21.2.112:0/598231065 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4bc809b930 0x7f4bc80b7c20 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 rx=0x7f4bb8007650 tx=0x7f4bb800a910).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:20.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.138 7f4bbffff700 1 -- 172.21.2.112:0/598231065 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4bb800ad00 con 0x7f4bc809b930 2021-02-10T11:33:20.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.138 7f4bbffff700 1 -- 172.21.2.112:0/598231065 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bb8004350 con 0x7f4bc809b930 2021-02-10T11:33:20.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.138 7f4bbffff700 1 -- 172.21.2.112:0/598231065 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4bb8004870 con 0x7f4bc809b930 2021-02-10T11:33:20.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.138 7f4bcd3d2700 1 -- 172.21.2.112:0/598231065 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bc80c29a0 con 0x7f4bc809b930 2021-02-10T11:33:20.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.138 7f4bcd3d2700 1 -- 172.21.2.112:0/598231065 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bc80c2f30 con 0x7f4bc809b930 2021-02-10T11:33:20.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.138 7f4bbffff700 1 -- 172.21.2.112:0/598231065 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4bb8004530 con 0x7f4bc809b930 2021-02-10T11:33:20.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.139 7f4bbffff700 1 --2- 172.21.2.112:0/598231065 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4bac0186c0 0x7f4bac01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:20.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.139 7f4bbffff700 1 -- 172.21.2.112:0/598231065 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f4bb80320a0 con 0x7f4bc809b930 2021-02-10T11:33:20.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.139 7f4bc687c700 1 --2- 172.21.2.112:0/598231065 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4bac0186c0 0x7f4bac01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:20.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.139 7f4bc687c700 1 --2- 172.21.2.112:0/598231065 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4bac0186c0 0x7f4bac01abc0 crc :-1 s=READY pgs=611 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:20.156 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.169 7f4bcd3d2700 1 -- 172.21.2.112:0/598231065 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bc8105b30 con 0x7f4bc809b930 2021-02-10T11:33:20.159 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.172 7f4bbffff700 1 -- 172.21.2.112:0/598231065 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4bb80009b0 con 0x7f4bc809b930 2021-02-10T11:33:20.291 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.304 7f4bcd3d2700 1 -- 172.21.2.112:0/598231065 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f4bc80f0b50 con 0x7f4bc809b930 2021-02-10T11:33:20.291 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.304 7f4bbffff700 1 -- 172.21.2.112:0/598231065 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v50) v1 ==== 92+0+10 (secure 0 0 0) 0x7f4bb8004530 con 0x7f4bc809b930 2021-02-10T11:33:20.292 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:33:20.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.306 7f4bcd3d2700 1 -- 172.21.2.112:0/598231065 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4bac0186c0 msgr2=0x7f4bac01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:20.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.306 7f4bcd3d2700 1 --2- 172.21.2.112:0/598231065 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4bac0186c0 0x7f4bac01abc0 crc :-1 s=READY pgs=611 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:20.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.306 7f4bcd3d2700 1 -- 172.21.2.112:0/598231065 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4bc809b930 msgr2=0x7f4bc80b7c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:20.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.306 7f4bcd3d2700 1 --2- 172.21.2.112:0/598231065 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4bc809b930 0x7f4bc80b7c20 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 rx=0x7f4bb8007650 tx=0x7f4bb800a910).stop 2021-02-10T11:33:20.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.306 7f4bcd3d2700 1 -- 172.21.2.112:0/598231065 shutdown_connections 2021-02-10T11:33:20.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.306 7f4bcd3d2700 1 --2- 172.21.2.112:0/598231065 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4bac0186c0 0x7f4bac01abc0 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:20.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.306 7f4bcd3d2700 1 --2- 172.21.2.112:0/598231065 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4bc80b8410 0x7f4bc80b8810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:20.295 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.306 7f4bcd3d2700 1 --2- 172.21.2.112:0/598231065 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4bc80c1f80 0x7f4bc80c2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:20.295 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.306 7f4bcd3d2700 1 --2- 172.21.2.112:0/598231065 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4bc809b930 0x7f4bc80b7c20 unknown :-1 s=CLOSED pgs=1225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:20.295 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.306 7f4bcd3d2700 1 -- 172.21.2.112:0/598231065 >> 172.21.2.112:0/598231065 conn(0x7f4bc8094e10 msgr2=0x7f4bc80a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:20.295 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.307 7f4bcd3d2700 1 -- 172.21.2.112:0/598231065 shutdown_connections 2021-02-10T11:33:20.295 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.307 7f4bcd3d2700 1 -- 172.21.2.112:0/598231065 wait complete. 2021-02-10T11:33:20.302 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --format=json-pretty osd lspools 2021-02-10T11:33:20.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.722 7f7a29569700 1 Processor -- start 2021-02-10T11:33:20.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.722 7f7a29569700 1 -- start start 2021-02-10T11:33:20.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.723 7f7a29569700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f7a240e5070 0x7f7a241b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:20.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.723 7f7a29569700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f7a240e5470 0x7f7a24080310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:20.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.723 7f7a29569700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a2407ed80 con 0x7f7a241b3e40 2021-02-10T11:33:20.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.723 7f7a29569700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a240d1790 con 0x7f7a240e5470 2021-02-10T11:33:20.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.723 7f7a29569700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a240d19b0 con 0x7f7a240e5070 2021-02-10T11:33:20.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.723 7f7a23ac0700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f7a240e5470 0x7f7a24080310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:20.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.723 7f7a23ac0700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f7a240e5470 0x7f7a24080310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:49644/0 (socket says 172.21.2.112:49644) 2021-02-10T11:33:20.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.723 7f7a23ac0700 1 -- 172.21.2.112:0/650960245 learned_addr learned my addr 172.21.2.112:0/650960245 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:33:20.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.723 7f7a232bf700 1 --2- 172.21.2.112:0/650960245 >> v2:172.21.2.120:3300/0 conn(0x7f7a240e5070 0x7f7a241b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:20.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.723 7f7a23ac0700 1 -- 172.21.2.112:0/650960245 >> v2:172.21.2.120:3300/0 conn(0x7f7a240e5070 msgr2=0x7f7a241b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:20.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.723 7f7a23ac0700 1 --2- 172.21.2.112:0/650960245 >> v2:172.21.2.120:3300/0 conn(0x7f7a240e5070 0x7f7a241b3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:20.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.723 7f7a23ac0700 1 -- 172.21.2.112:0/650960245 >> v1:172.21.2.120:6789/0 conn(0x7f7a241b3e40 legacy=0x7f7a2407d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:20.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.723 7f7a23ac0700 1 -- 172.21.2.112:0/650960245 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a240d1bd0 con 0x7f7a240e5470 2021-02-10T11:33:20.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.724 7f7a23ac0700 1 --2- 172.21.2.112:0/650960245 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7a240e5470 0x7f7a24080310 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 rx=0x7f7a1c006f00 tx=0x7f7a1c009e90).ready entity=mon.0 client_cookie=e699a13a45ed0770 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:20.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.724 7f7a222bd700 1 -- 172.21.2.112:0/650960245 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7a1c00b300 con 0x7f7a240e5470 2021-02-10T11:33:20.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.724 7f7a222bd700 1 -- 172.21.2.112:0/650960245 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a1c00b640 con 0x7f7a240e5470 2021-02-10T11:33:20.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.724 7f7a222bd700 1 -- 172.21.2.112:0/650960245 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7a1c00baa0 con 0x7f7a240e5470 2021-02-10T11:33:20.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.724 7f7a29569700 1 -- 172.21.2.112:0/650960245 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7a240e5470 msgr2=0x7f7a24080310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:20.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.724 7f7a29569700 1 --2- 172.21.2.112:0/650960245 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7a240e5470 0x7f7a24080310 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 rx=0x7f7a1c006f00 tx=0x7f7a1c009e90).stop 2021-02-10T11:33:20.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.724 7f7a29569700 1 -- 172.21.2.112:0/650960245 shutdown_connections 2021-02-10T11:33:20.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.724 7f7a29569700 1 --2- 172.21.2.112:0/650960245 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7a240e5470 0x7f7a24080310 unknown :-1 s=CLOSED pgs=1226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:20.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.724 7f7a29569700 1 --2- 172.21.2.112:0/650960245 >> v2:172.21.2.120:3300/0 conn(0x7f7a240e5070 0x7f7a241b3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:20.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.724 7f7a29569700 1 -- 172.21.2.112:0/650960245 >> 172.21.2.112:0/650960245 conn(0x7f7a240ce690 msgr2=0x7f7a240e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:20.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.725 7f7a29569700 1 -- 172.21.2.112:0/650960245 shutdown_connections 2021-02-10T11:33:20.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.725 7f7a29569700 1 -- 172.21.2.112:0/650960245 wait complete. 2021-02-10T11:33:20.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.739 7f7a29569700 1 Processor -- start 2021-02-10T11:33:20.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.739 7f7a29569700 1 -- start start 2021-02-10T11:33:20.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.740 7f7a29569700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7a2408d400 0x7f7a24081230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:20.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.740 7f7a29569700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7a24081850 0x7f7a240acf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:20.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.740 7f7a29569700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7a24081c50 0x7f7a240ad5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:20.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.740 7f7a29569700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a24082110 con 0x7f7a24081850 2021-02-10T11:33:20.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.740 7f7a29569700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a2408d800 con 0x7f7a2408d400 2021-02-10T11:33:20.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.740 7f7a29569700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a240adbc0 con 0x7f7a24081c50 2021-02-10T11:33:20.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.740 7f7a22abe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7a24081850 0x7f7a240acf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:20.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.740 7f7a22abe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7a24081850 0x7f7a240acf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:49652/0 (socket says 172.21.2.112:49652) 2021-02-10T11:33:20.732 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.740 7f7a22abe700 1 -- 172.21.2.112:0/270047865 learned_addr learned my addr 172.21.2.112:0/270047865 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:33:20.732 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.740 7f7a232bf700 1 --2- 172.21.2.112:0/270047865 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7a2408d400 0x7f7a24081230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:20.732 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.740 7f7a23ac0700 1 --2- 172.21.2.112:0/270047865 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7a24081c50 0x7f7a240ad5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:20.732 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.740 7f7a22abe700 1 -- 172.21.2.112:0/270047865 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7a24081c50 msgr2=0x7f7a240ad5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:20.732 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.741 7f7a22abe700 1 --2- 172.21.2.112:0/270047865 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7a24081c50 0x7f7a240ad5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:20.732 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.741 7f7a22abe700 1 -- 172.21.2.112:0/270047865 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7a2408d400 msgr2=0x7f7a24081230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:20.732 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.741 7f7a22abe700 1 --2- 172.21.2.112:0/270047865 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7a2408d400 0x7f7a24081230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:20.732 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.741 7f7a22abe700 1 -- 172.21.2.112:0/270047865 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a24187fc0 con 0x7f7a24081850 2021-02-10T11:33:20.732 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.741 7f7a23ac0700 1 --2- 172.21.2.112:0/270047865 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7a24081c50 0x7f7a240ad5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:33:20.732 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.741 7f7a232bf700 1 --2- 172.21.2.112:0/270047865 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7a2408d400 0x7f7a24081230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:33:20.732 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.741 7f7a22abe700 1 --2- 172.21.2.112:0/270047865 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7a24081850 0x7f7a240acf80 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 rx=0x7f7a0c006ed0 tx=0x7f7a0c00b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:20.732 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.741 7f7a1bfff700 1 -- 172.21.2.112:0/270047865 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7a0c006020 con 0x7f7a24081850 2021-02-10T11:33:20.732 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.741 7f7a1bfff700 1 -- 172.21.2.112:0/270047865 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a0c0009b0 con 0x7f7a24081850 2021-02-10T11:33:20.733 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.741 7f7a29569700 1 -- 172.21.2.112:0/270047865 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a24187fc0 con 0x7f7a24081850 2021-02-10T11:33:20.733 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.741 7f7a29569700 1 -- 172.21.2.112:0/270047865 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a24187fc0 con 0x7f7a24081850 2021-02-10T11:33:20.733 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.741 7f7a1bfff700 1 -- 172.21.2.112:0/270047865 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7a0c0054b0 con 0x7f7a24081850 2021-02-10T11:33:20.733 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.742 7f7a1bfff700 1 -- 172.21.2.112:0/270047865 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7a0c006020 con 0x7f7a24081850 2021-02-10T11:33:20.733 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.742 7f7a1bfff700 1 --2- 172.21.2.112:0/270047865 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7a080186c0 0x7f7a0801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:20.733 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.742 7f7a1bfff700 1 -- 172.21.2.112:0/270047865 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f7a0c0323f0 con 0x7f7a24081850 2021-02-10T11:33:20.733 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.744 7f7a232bf700 1 --2- 172.21.2.112:0/270047865 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7a080186c0 0x7f7a0801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:20.733 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.744 7f7a232bf700 1 --2- 172.21.2.112:0/270047865 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7a080186c0 0x7f7a0801abc0 crc :-1 s=READY pgs=612 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:20.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.773 7f7a29569700 1 -- 172.21.2.112:0/270047865 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a2408d800 con 0x7f7a24081850 2021-02-10T11:33:20.763 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.775 7f7a1bfff700 1 -- 172.21.2.112:0/270047865 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7a0c005730 con 0x7f7a24081850 2021-02-10T11:33:20.885 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.897 7f7a29569700 1 -- 172.21.2.112:0/270047865 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f7a2408d800 con 0x7f7a24081850 2021-02-10T11:33:20.885 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.897 7f7a1bfff700 1 -- 172.21.2.112:0/270047865 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v50) v1 ==== 84+0+64 (secure 0 0 0) 0x7f7a0c00bd50 con 0x7f7a24081850 2021-02-10T11:33:20.885 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:33:20.886 INFO:teuthology.orchestra.run.gibba012.stdout:[ 2021-02-10T11:33:20.886 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:20.886 INFO:teuthology.orchestra.run.gibba012.stdout: "poolnum": 1, 2021-02-10T11:33:20.886 INFO:teuthology.orchestra.run.gibba012.stdout: "poolname": "rbd" 2021-02-10T11:33:20.886 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:33:20.886 INFO:teuthology.orchestra.run.gibba012.stdout:] 2021-02-10T11:33:20.887 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.899 7f7a29569700 1 -- 172.21.2.112:0/270047865 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7a080186c0 msgr2=0x7f7a0801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:20.887 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.899 7f7a29569700 1 --2- 172.21.2.112:0/270047865 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7a080186c0 0x7f7a0801abc0 crc :-1 s=READY pgs=612 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:20.887 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.899 7f7a29569700 1 -- 172.21.2.112:0/270047865 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7a24081850 msgr2=0x7f7a240acf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:20.887 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.899 7f7a29569700 1 --2- 172.21.2.112:0/270047865 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7a24081850 0x7f7a240acf80 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 rx=0x7f7a0c006ed0 tx=0x7f7a0c00b960).stop 2021-02-10T11:33:20.887 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.899 7f7a29569700 1 -- 172.21.2.112:0/270047865 shutdown_connections 2021-02-10T11:33:20.887 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.899 7f7a29569700 1 --2- 172.21.2.112:0/270047865 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7a080186c0 0x7f7a0801abc0 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:20.887 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.899 7f7a29569700 1 --2- 172.21.2.112:0/270047865 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7a24081c50 0x7f7a240ad5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:20.888 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.899 7f7a29569700 1 --2- 172.21.2.112:0/270047865 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7a24081850 0x7f7a240acf80 unknown :-1 s=CLOSED pgs=1227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:20.888 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.899 7f7a29569700 1 --2- 172.21.2.112:0/270047865 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7a2408d400 0x7f7a24081230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:20.888 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.899 7f7a29569700 1 -- 172.21.2.112:0/270047865 >> 172.21.2.112:0/270047865 conn(0x7f7a241ac620 msgr2=0x7f7a240896e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:20.888 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.899 7f7a29569700 1 -- 172.21.2.112:0/270047865 shutdown_connections 2021-02-10T11:33:20.888 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:20.899 7f7a29569700 1 -- 172.21.2.112:0/270047865 wait complete. 2021-02-10T11:33:20.895 INFO:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2021-02-10T11:33:20.896 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_metadata 8 2021-02-10T11:33:21.326 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:21.320 7fae6397f700 1 Processor -- start 2021-02-10T11:33:21.327 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:21.321 7fae6397f700 1 -- start start 2021-02-10T11:33:21.327 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:21.321 7fae6397f700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fae5c1b0470 0x7fae5c1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:21.327 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:21.321 7fae6397f700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae5c0f0b50 con 0x7fae5c0ef610 2021-02-10T11:33:21.327 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:21.321 7fae6397f700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae5c0f24e0 con 0x7fae5c0f20e0 2021-02-10T11:33:21.327 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:21.321 7fae6397f700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae5c0e5ea0 con 0x7fae5c1b0470 2021-02-10T11:33:21.327 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:21.321 7fae60f9a700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fae5c0ef610 0x7fae5c0efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:49120/0 (socket says 172.21.2.112:49120) 2021-02-10T11:33:21.327 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:21.321 7fae60f9a700 1 -- 172.21.2.112:0/119968705 learned_addr learned my addr 172.21.2.112:0/119968705 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:33:21.327 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:21.322 7fae4ffff700 1 -- 172.21.2.112:0/119968705 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2092465535 0 0) 0x7fae580010b0 con 0x7fae5c0ef610 2021-02-10T11:33:21.327 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:21.322 7fae4ffff700 1 -- 172.21.2.112:0/119968705 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae48002fe0 con 0x7fae5c0ef610 2021-02-10T11:33:21.327 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:21.322 7fae6179b700 1 --2- 172.21.2.112:0/119968705 >> v2:172.21.2.120:3300/0 conn(0x7fae5c1b0470 0x7fae5c1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:21.327 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:21.322 7fae4ffff700 1 -- 172.21.2.112:0/119968705 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 43357624 0 0) 0x7fae580010b0 con 0x7fae5c0ef610 2021-02-10T11:33:21.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:21.322 7fae4ffff700 1 -- 172.21.2.112:0/119968705 >> v2:172.21.2.120:3300/0 conn(0x7fae5c1b0470 msgr2=0x7fae5c1b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:21.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:21.322 7fae4ffff700 1 --2- 172.21.2.112:0/119968705 >> v2:172.21.2.120:3300/0 conn(0x7fae5c1b0470 0x7fae5c1b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:21.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:21.322 7fae4ffff700 1 -- 172.21.2.112:0/119968705 >> v1:172.21.2.120:6789/0 conn(0x7fae5c0f20e0 legacy=0x7fae5c0f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:21.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:21.322 7fae4ffff700 1 -- 172.21.2.112:0/119968705 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae5c0e52e0 con 0x7fae5c0ef610 2021-02-10T11:33:21.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:21.322 7fae4ffff700 1 -- 172.21.2.112:0/119968705 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7fae580015b0 con 0x7fae5c0ef610 2021-02-10T11:33:21.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:21.322 7fae4ffff700 1 -- 172.21.2.112:0/119968705 >> v1:172.21.2.112:6789/0 conn(0x7fae5c0ef610 legacy=0x7fae5c0efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:21.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:21.322 7fae4ffff700 1 --2- 172.21.2.112:0/119968705 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fae48003ae0 0x7fae48006000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:21.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:21.322 7fae4ffff700 1 --2- 172.21.2.112:0/119968705 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fae48006820 0x7fae48008d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:21.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:21.322 7fae4ffff700 1 --2- 172.21.2.112:0/119968705 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fae48009640 0x7fae4800bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:21.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:21.322 7fae4ffff700 1 -- 172.21.2.112:0/119968705 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae4800c390 con 0x7fae48006820 2021-02-10T11:33:21.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:21.322 7fae4ffff700 1 -- 172.21.2.112:0/119968705 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae4800c5b0 con 0x7fae48003ae0 2021-02-10T11:33:21.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:21.322 7fae4ffff700 1 -- 172.21.2.112:0/119968705 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae4800c7d0 con 0x7fae48009640 2021-02-10T11:33:21.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:21.322 7fae6179b700 1 --2- 172.21.2.112:0/119968705 >> v2:172.21.2.120:3300/0 conn(0x7fae5c1b0470 0x7fae5c1b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:33:21.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:21.322 7fae60f9a700 1 --2- 172.21.2.112:0/119968705 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fae48006820 0x7fae48008d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:21.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:21.322 7fae60f9a700 1 -- 172.21.2.112:0/119968705 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fae48009640 msgr2=0x7fae4800bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:21.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:21.322 7fae60f9a700 1 --2- 172.21.2.112:0/119968705 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fae48009640 0x7fae4800bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:21.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:21.322 7fae60f9a700 1 -- 172.21.2.112:0/119968705 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fae48003ae0 msgr2=0x7fae48006000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:21.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:21.322 7fae60f9a700 1 --2- 172.21.2.112:0/119968705 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fae48003ae0 0x7fae48006000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:21.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:21.323 7fae60f9a700 1 -- 172.21.2.112:0/119968705 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae4800ca70 con 0x7fae48006820 2021-02-10T11:33:21.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:21.323 7fae61f9c700 1 -- 172.21.2.112:0/119968705 reap_dead start 2021-02-10T11:33:21.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:21.323 7fae60f9a700 1 --2- 172.21.2.112:0/119968705 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fae48006820 0x7fae48008d40 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 rx=0x7fae580017d0 tx=0x7fae5804bd40).ready entity=mon.0 client_cookie=f3623e83a067284c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:21.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:21.323 7fae4ffff700 1 -- 172.21.2.112:0/119968705 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fae580013a0 con 0x7fae48006820 2021-02-10T11:33:21.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:21.323 7fae4ffff700 1 -- 172.21.2.112:0/119968705 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae5804e310 con 0x7fae48006820 2021-02-10T11:33:21.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:21.323 7fae4ffff700 1 -- 172.21.2.112:0/119968705 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fae5804e770 con 0x7fae48006820 2021-02-10T11:33:21.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:21.323 7fae6397f700 1 -- 172.21.2.112:0/119968705 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fae48006820 msgr2=0x7fae48008d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:21.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:21.323 7fae6397f700 1 --2- 172.21.2.112:0/119968705 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fae48006820 0x7fae48008d40 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 rx=0x7fae580017d0 tx=0x7fae5804bd40).stop 2021-02-10T11:33:21.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:21.324 7fae6397f700 1 -- 172.21.2.112:0/119968705 shutdown_connections 2021-02-10T11:33:21.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:21.324 7fae6397f700 1 --2- 172.21.2.112:0/119968705 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fae48006820 0x7fae48008d40 unknown :-1 s=CLOSED pgs=1229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:21.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:21.324 7fae6397f700 1 -- 172.21.2.112:0/119968705 >> 172.21.2.112:0/119968705 conn(0x7fae5c0ce3e0 msgr2=0x7fae5c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:21.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:21.324 7fae6397f700 1 -- 172.21.2.112:0/119968705 shutdown_connections 2021-02-10T11:33:21.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:21.324 7fae6397f700 1 -- 172.21.2.112:0/119968705 wait complete. 2021-02-10T11:33:21.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:21.338 7fae6397f700 1 Processor -- start 2021-02-10T11:33:21.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:21.339 7fae6397f700 1 -- start start 2021-02-10T11:33:21.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:21.339 7fae6397f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fae5c09e560 0x7fae5c0a0130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:21.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:21.339 7fae6397f700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fae5c09e960 0x7fae5c0a0750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:21.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:21.339 7fae6397f700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fae5c0a0d70 0x7fae5c0a1af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:21.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:21.339 7fae6397f700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae5c0c38f0 con 0x7fae5c09e560 2021-02-10T11:33:21.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:21.339 7fae6397f700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae5c0a1170 con 0x7fae5c0a0d70 2021-02-10T11:33:21.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:21.339 7fae6397f700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae5c0a1390 con 0x7fae5c09e960 2021-02-10T11:33:21.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:21.339 7fae6179b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fae5c09e560 0x7fae5c0a0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:21.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:21.339 7fae6179b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fae5c09e560 0x7fae5c0a0130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:49666/0 (socket says 172.21.2.112:49666) 2021-02-10T11:33:21.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:21.339 7fae6179b700 1 -- 172.21.2.112:0/894150746 learned_addr learned my addr 172.21.2.112:0/894150746 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:33:21.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:21.340 7fae61f9c700 1 --2- 172.21.2.112:0/894150746 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fae5c0a0d70 0x7fae5c0a1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:21.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:21.340 7fae60f9a700 1 --2- 172.21.2.112:0/894150746 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fae5c09e960 0x7fae5c0a0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:21.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:21.340 7fae6179b700 1 -- 172.21.2.112:0/894150746 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fae5c09e960 msgr2=0x7fae5c0a0750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:21.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:21.340 7fae6179b700 1 --2- 172.21.2.112:0/894150746 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fae5c09e960 0x7fae5c0a0750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:21.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:21.340 7fae6179b700 1 -- 172.21.2.112:0/894150746 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fae5c0a0d70 msgr2=0x7fae5c0a1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:21.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:21.340 7fae6179b700 1 --2- 172.21.2.112:0/894150746 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fae5c0a0d70 0x7fae5c0a1af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:21.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:21.340 7fae6179b700 1 -- 172.21.2.112:0/894150746 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae5c0a2110 con 0x7fae5c09e560 2021-02-10T11:33:21.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:21.340 7fae6179b700 1 --2- 172.21.2.112:0/894150746 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fae5c09e560 0x7fae5c0a0130 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 rx=0x7fae50006ee0 tx=0x7fae5000b9b0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:21.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:21.340 7fae4dffb700 1 -- 172.21.2.112:0/894150746 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fae5000bda0 con 0x7fae5c09e560 2021-02-10T11:33:21.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:21.340 7fae4dffb700 1 -- 172.21.2.112:0/894150746 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae500052e0 con 0x7fae5c09e560 2021-02-10T11:33:21.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:21.340 7fae6397f700 1 -- 172.21.2.112:0/894150746 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae5c0a2110 con 0x7fae5c09e560 2021-02-10T11:33:21.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:21.340 7fae4dffb700 1 -- 172.21.2.112:0/894150746 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fae50005740 con 0x7fae5c09e560 2021-02-10T11:33:21.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:21.340 7fae6397f700 1 -- 172.21.2.112:0/894150746 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae5c0a26a0 con 0x7fae5c09e560 2021-02-10T11:33:21.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:21.341 7fae4dffb700 1 -- 172.21.2.112:0/894150746 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fae500056d0 con 0x7fae5c09e560 2021-02-10T11:33:21.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:21.341 7fae4dffb700 1 --2- 172.21.2.112:0/894150746 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fae3c0186d0 0x7fae3c01abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:21.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:21.341 7fae4dffb700 1 -- 172.21.2.112:0/894150746 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 4753+0+0 (secure 0 0 0) 0x7fae50032300 con 0x7fae5c09e560 2021-02-10T11:33:21.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:21.341 7fae60f9a700 1 --2- 172.21.2.112:0/894150746 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fae3c0186d0 0x7fae3c01abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:21.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:21.342 7fae60f9a700 1 --2- 172.21.2.112:0/894150746 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fae3c0186d0 0x7fae3c01abd0 crc :-1 s=READY pgs=613 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:21.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:21.371 7fae6397f700 1 -- 172.21.2.112:0/894150746 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae5c105b30 con 0x7fae5c09e560 2021-02-10T11:33:21.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:21.373 7fae4dffb700 1 -- 172.21.2.112:0/894150746 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fae500052e0 con 0x7fae5c09e560 2021-02-10T11:33:21.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:21.504 7fae6397f700 1 -- 172.21.2.112:0/894150746 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"} v 0) v1 -- 0x7fae5c0f0b50 con 0x7fae5c09e560 2021-02-10T11:33:21.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:21.928 7fae4dffb700 1 -- 172.21.2.112:0/894150746 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"}]=0 pool 'cephfs_metadata' created v51) v1 ==== 133+0+0 (secure 0 0 0) 0x7fae500054f0 con 0x7fae5c09e560 2021-02-10T11:33:21.959 INFO:teuthology.orchestra.run.gibba012.stderr:pool 'cephfs_metadata' created 2021-02-10T11:33:21.960 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:21.972 7fae6397f700 1 -- 172.21.2.112:0/894150746 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fae3c0186d0 msgr2=0x7fae3c01abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:21.960 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:21.972 7fae6397f700 1 --2- 172.21.2.112:0/894150746 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fae3c0186d0 0x7fae3c01abd0 crc :-1 s=READY pgs=613 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:21.960 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:21.972 7fae6397f700 1 -- 172.21.2.112:0/894150746 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fae5c09e560 msgr2=0x7fae5c0a0130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:21.960 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:21.972 7fae6397f700 1 --2- 172.21.2.112:0/894150746 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fae5c09e560 0x7fae5c0a0130 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 rx=0x7fae50006ee0 tx=0x7fae5000b9b0).stop 2021-02-10T11:33:21.961 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:21.972 7fae6397f700 1 -- 172.21.2.112:0/894150746 shutdown_connections 2021-02-10T11:33:21.961 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:21.972 7fae6397f700 1 --2- 172.21.2.112:0/894150746 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fae3c0186d0 0x7fae3c01abd0 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:21.961 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:21.972 7fae6397f700 1 --2- 172.21.2.112:0/894150746 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fae5c0a0d70 0x7fae5c0a1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:21.961 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:21.972 7fae6397f700 1 --2- 172.21.2.112:0/894150746 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fae5c09e960 0x7fae5c0a0750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:21.961 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:21.972 7fae6397f700 1 --2- 172.21.2.112:0/894150746 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fae5c09e560 0x7fae5c0a0130 unknown :-1 s=CLOSED pgs=1230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:21.962 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:21.972 7fae6397f700 1 -- 172.21.2.112:0/894150746 >> 172.21.2.112:0/894150746 conn(0x7fae5c09fc10 msgr2=0x7fae5c094ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:21.962 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:21.973 7fae6397f700 1 -- 172.21.2.112:0/894150746 shutdown_connections 2021-02-10T11:33:21.962 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:21.973 7fae6397f700 1 -- 172.21.2.112:0/894150746 wait complete. 2021-02-10T11:33:21.969 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_data 8 2021-02-10T11:33:22.399 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:22.394 7fd7958b8700 1 Processor -- start 2021-02-10T11:33:22.400 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:22.394 7fd7958b8700 1 -- start start 2021-02-10T11:33:22.400 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:22.394 7fd7958b8700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fd7901b0870 0x7fd7900f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:22.400 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:22.394 7fd7958b8700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7901b0c70 con 0x7fd7901b0470 2021-02-10T11:33:22.400 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:22.394 7fd7958b8700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7900f0b50 con 0x7fd7900efe00 2021-02-10T11:33:22.400 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:22.394 7fd7958b8700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7900e5ea0 con 0x7fd7901b0870 2021-02-10T11:33:22.400 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:22.394 7fd78f574700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fd7901b0470 0x7fd7900ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:49136/0 (socket says 172.21.2.112:49136) 2021-02-10T11:33:22.400 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:22.394 7fd78f574700 1 -- 172.21.2.112:0/3751931479 learned_addr learned my addr 172.21.2.112:0/3751931479 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:33:22.400 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:22.395 7fd78e572700 1 -- 172.21.2.112:0/3751931479 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3599065916 0 0) 0x7fd778000ce0 con 0x7fd7901b0470 2021-02-10T11:33:22.400 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:22.395 7fd78e572700 1 -- 172.21.2.112:0/3751931479 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd77c002fe0 con 0x7fd7901b0470 2021-02-10T11:33:22.400 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:22.395 7fd78ed73700 1 --2- 172.21.2.112:0/3751931479 >> v2:172.21.2.120:3300/0 conn(0x7fd7901b0870 0x7fd7900f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:22.400 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:22.395 7fd78e572700 1 -- 172.21.2.112:0/3751931479 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 732229652 0 0) 0x7fd7780010e0 con 0x7fd7901b0470 2021-02-10T11:33:22.400 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:22.395 7fd78e572700 1 -- 172.21.2.112:0/3751931479 >> v2:172.21.2.120:3300/0 conn(0x7fd7901b0870 msgr2=0x7fd7900f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:22.401 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:22.395 7fd78e572700 1 --2- 172.21.2.112:0/3751931479 >> v2:172.21.2.120:3300/0 conn(0x7fd7901b0870 0x7fd7900f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:22.401 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:22.395 7fd78e572700 1 -- 172.21.2.112:0/3751931479 >> v1:172.21.2.120:6789/0 conn(0x7fd7900efe00 legacy=0x7fd7900f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:22.401 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:22.395 7fd78e572700 1 -- 172.21.2.112:0/3751931479 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7900e52e0 con 0x7fd7901b0470 2021-02-10T11:33:22.401 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:22.395 7fd78e572700 1 -- 172.21.2.112:0/3751931479 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7fd778001660 con 0x7fd7901b0470 2021-02-10T11:33:22.401 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:22.395 7fd78e572700 1 -- 172.21.2.112:0/3751931479 >> v1:172.21.2.112:6789/0 conn(0x7fd7901b0470 legacy=0x7fd7900ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:22.401 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:22.395 7fd78e572700 1 --2- 172.21.2.112:0/3751931479 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd77c003ac0 0x7fd77c005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:22.401 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:22.395 7fd78e572700 1 --2- 172.21.2.112:0/3751931479 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd77c006790 0x7fd77c008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:22.401 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:22.395 7fd78e572700 1 --2- 172.21.2.112:0/3751931479 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd77c009590 0x7fd77c00bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:22.401 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:22.395 7fd78e572700 1 -- 172.21.2.112:0/3751931479 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd77c00c300 con 0x7fd77c006790 2021-02-10T11:33:22.401 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:22.395 7fd78e572700 1 -- 172.21.2.112:0/3751931479 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd77c00c520 con 0x7fd77c003ac0 2021-02-10T11:33:22.401 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:22.395 7fd78e572700 1 -- 172.21.2.112:0/3751931479 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd77c00c740 con 0x7fd77c009590 2021-02-10T11:33:22.401 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:22.395 7fd78ed73700 1 --2- 172.21.2.112:0/3751931479 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd77c006790 0x7fd77c008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:22.401 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:22.396 7fd78fd75700 1 --2- 172.21.2.112:0/3751931479 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd77c009590 0x7fd77c00bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:22.402 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:22.396 7fd78f574700 1 --2- 172.21.2.112:0/3751931479 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd77c003ac0 0x7fd77c005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:22.402 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:22.396 7fd78fd75700 1 -- 172.21.2.112:0/3751931479 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd77c003ac0 msgr2=0x7fd77c005ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:22.402 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:22.396 7fd78fd75700 1 --2- 172.21.2.112:0/3751931479 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd77c003ac0 0x7fd77c005ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:22.402 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:22.396 7fd78fd75700 1 -- 172.21.2.112:0/3751931479 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd77c006790 msgr2=0x7fd77c008c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:22.402 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:22.396 7fd78fd75700 1 --2- 172.21.2.112:0/3751931479 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd77c006790 0x7fd77c008c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:22.402 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:22.396 7fd78fd75700 1 -- 172.21.2.112:0/3751931479 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd77c00c9e0 con 0x7fd77c009590 2021-02-10T11:33:22.402 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:22.396 7fd78fd75700 1 -- 172.21.2.112:0/3751931479 reap_dead start 2021-02-10T11:33:22.402 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:22.396 7fd78fd75700 1 --2- 172.21.2.112:0/3751931479 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd77c009590 0x7fd77c00bab0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7fd788006e40 tx=0x7fd788009db0).ready entity=mon.2 client_cookie=632d83e5f5a88076 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:22.402 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:22.396 7fd78e572700 1 -- 172.21.2.112:0/3751931479 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd78800b300 con 0x7fd77c009590 2021-02-10T11:33:22.402 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:22.396 7fd78e572700 1 -- 172.21.2.112:0/3751931479 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd788006780 con 0x7fd77c009590 2021-02-10T11:33:22.402 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:22.396 7fd78e572700 1 -- 172.21.2.112:0/3751931479 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd788006a10 con 0x7fd77c009590 2021-02-10T11:33:22.402 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:22.396 7fd7958b8700 1 -- 172.21.2.112:0/3751931479 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd77c009590 msgr2=0x7fd77c00bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:22.402 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:22.396 7fd7958b8700 1 --2- 172.21.2.112:0/3751931479 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd77c009590 0x7fd77c00bab0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7fd788006e40 tx=0x7fd788009db0).stop 2021-02-10T11:33:22.403 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:22.397 7fd7958b8700 1 -- 172.21.2.112:0/3751931479 shutdown_connections 2021-02-10T11:33:22.403 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:22.397 7fd7958b8700 1 --2- 172.21.2.112:0/3751931479 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd77c009590 0x7fd77c00bab0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:22.403 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:22.397 7fd7958b8700 1 -- 172.21.2.112:0/3751931479 >> 172.21.2.112:0/3751931479 conn(0x7fd7900ce3e0 msgr2=0x7fd7900e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:22.403 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:22.397 7fd7958b8700 1 -- 172.21.2.112:0/3751931479 shutdown_connections 2021-02-10T11:33:22.403 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:22.397 7fd7958b8700 1 -- 172.21.2.112:0/3751931479 wait complete. 2021-02-10T11:33:22.403 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:22.412 7fd7958b8700 1 Processor -- start 2021-02-10T11:33:22.404 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:22.412 7fd7958b8700 1 -- start start 2021-02-10T11:33:22.404 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:22.412 7fd7958b8700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd7900b7db0 0x7fd7900b8260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:22.405 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:22.412 7fd7958b8700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd7900b8880 0x7fd7901a7450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:22.405 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:22.412 7fd7958b8700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd7900b8c80 0x7fd7901a7a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:22.405 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:22.412 7fd7958b8700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7900b79b0 con 0x7fd7900b8880 2021-02-10T11:33:22.405 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:22.412 7fd7958b8700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7901a8090 con 0x7fd7900b7db0 2021-02-10T11:33:22.405 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:22.412 7fd7958b8700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7901a82b0 con 0x7fd7900b8c80 2021-02-10T11:33:22.405 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:22.412 7fd78ed73700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd7900b8880 0x7fd7901a7450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:22.405 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:22.412 7fd78ed73700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd7900b8880 0x7fd7901a7450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:49684/0 (socket says 172.21.2.112:49684) 2021-02-10T11:33:22.405 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:22.412 7fd78ed73700 1 -- 172.21.2.112:0/2096413653 learned_addr learned my addr 172.21.2.112:0/2096413653 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:33:22.405 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:22.413 7fd78ed73700 1 -- 172.21.2.112:0/2096413653 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd7900b8c80 msgr2=0x7fd7901a7a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:22.405 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:22.413 7fd78ed73700 1 --2- 172.21.2.112:0/2096413653 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd7900b8c80 0x7fd7901a7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:22.405 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:22.413 7fd78ed73700 1 -- 172.21.2.112:0/2096413653 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd7900b7db0 msgr2=0x7fd7900b8260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:22.405 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:22.413 7fd78ed73700 1 --2- 172.21.2.112:0/2096413653 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd7900b7db0 0x7fd7900b8260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:22.405 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:22.413 7fd78ed73700 1 -- 172.21.2.112:0/2096413653 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd790190c10 con 0x7fd7900b8880 2021-02-10T11:33:22.406 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:22.413 7fd78ed73700 1 --2- 172.21.2.112:0/2096413653 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd7900b8880 0x7fd7901a7450 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 rx=0x7fd780050570 tx=0x7fd780050910).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:22.406 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:22.413 7fd787fff700 1 -- 172.21.2.112:0/2096413653 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd7800009b0 con 0x7fd7900b8880 2021-02-10T11:33:22.406 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:22.413 7fd787fff700 1 -- 172.21.2.112:0/2096413653 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd780050d00 con 0x7fd7900b8880 2021-02-10T11:33:22.406 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:22.413 7fd787fff700 1 -- 172.21.2.112:0/2096413653 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd7800022b0 con 0x7fd7900b8880 2021-02-10T11:33:22.406 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:22.413 7fd7958b8700 1 -- 172.21.2.112:0/2096413653 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd790190c10 con 0x7fd7900b8880 2021-02-10T11:33:22.406 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:22.413 7fd7958b8700 1 -- 172.21.2.112:0/2096413653 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd790190c10 con 0x7fd7900b8880 2021-02-10T11:33:22.406 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:22.414 7fd787fff700 1 -- 172.21.2.112:0/2096413653 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd780000c90 con 0x7fd7900b8880 2021-02-10T11:33:22.406 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:22.414 7fd787fff700 1 --2- 172.21.2.112:0/2096413653 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd7740186c0 0x7fd77401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:22.406 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:22.414 7fd787fff700 1 -- 172.21.2.112:0/2096413653 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 5107+0+0 (secure 0 0 0) 0x7fd780076410 con 0x7fd7900b8880 2021-02-10T11:33:22.407 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:22.414 7fd78f574700 1 --2- 172.21.2.112:0/2096413653 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd7740186c0 0x7fd77401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:22.408 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:22.415 7fd78f574700 1 --2- 172.21.2.112:0/2096413653 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd7740186c0 0x7fd77401abc0 crc :-1 s=READY pgs=614 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:22.433 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:22.444 7fd7958b8700 1 -- 172.21.2.112:0/2096413653 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd790105b30 con 0x7fd7900b8880 2021-02-10T11:33:22.435 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:22.447 7fd787fff700 1 -- 172.21.2.112:0/2096413653 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd780073de0 con 0x7fd7900b8880 2021-02-10T11:33:22.564 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:22.576 7fd7958b8700 1 -- 172.21.2.112:0/2096413653 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"} v 0) v1 -- 0x7fd7900f0b50 con 0x7fd7900b8880 2021-02-10T11:33:22.923 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:22.935 7fd787fff700 1 -- 172.21.2.112:0/2096413653 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"}]=0 pool 'cephfs_data' created v53) v1 ==== 125+0+0 (secure 0 0 0) 0x7fd780073de0 con 0x7fd7900b8880 2021-02-10T11:33:22.929 INFO:teuthology.orchestra.run.gibba012.stderr:pool 'cephfs_data' created 2021-02-10T11:33:22.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:22.943 7fd7958b8700 1 -- 172.21.2.112:0/2096413653 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd7740186c0 msgr2=0x7fd77401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:22.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:22.943 7fd7958b8700 1 --2- 172.21.2.112:0/2096413653 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd7740186c0 0x7fd77401abc0 crc :-1 s=READY pgs=614 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:22.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:22.943 7fd7958b8700 1 -- 172.21.2.112:0/2096413653 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd7900b8880 msgr2=0x7fd7901a7450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:22.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:22.943 7fd7958b8700 1 --2- 172.21.2.112:0/2096413653 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd7900b8880 0x7fd7901a7450 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 rx=0x7fd780050570 tx=0x7fd780050910).stop 2021-02-10T11:33:22.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:22.943 7fd7958b8700 1 -- 172.21.2.112:0/2096413653 shutdown_connections 2021-02-10T11:33:22.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:22.943 7fd7958b8700 1 --2- 172.21.2.112:0/2096413653 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd7740186c0 0x7fd77401abc0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:22.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:22.943 7fd7958b8700 1 --2- 172.21.2.112:0/2096413653 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd7900b8c80 0x7fd7901a7a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:22.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:22.943 7fd7958b8700 1 --2- 172.21.2.112:0/2096413653 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd7900b8880 0x7fd7901a7450 unknown :-1 s=CLOSED pgs=1232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:22.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:22.943 7fd7958b8700 1 --2- 172.21.2.112:0/2096413653 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd7900b7db0 0x7fd7900b8260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:22.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:22.943 7fd7958b8700 1 -- 172.21.2.112:0/2096413653 >> 172.21.2.112:0/2096413653 conn(0x7fd790094e10 msgr2=0x7fd7900a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:22.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:22.943 7fd7958b8700 1 -- 172.21.2.112:0/2096413653 shutdown_connections 2021-02-10T11:33:22.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:22.943 7fd7958b8700 1 -- 172.21.2.112:0/2096413653 wait complete. 2021-02-10T11:33:22.940 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2021-02-10T11:33:23.372 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.365 7fe4fa0f3700 1 Processor -- start 2021-02-10T11:33:23.372 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.365 7fe4fa0f3700 1 -- start start 2021-02-10T11:33:23.372 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.366 7fe4fa0f3700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe4f40d1870 0x7fe4f41b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:23.372 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.366 7fe4fa0f3700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fe4f41b0ae0 0x7fe4f4108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:23.372 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.366 7fe4fa0f3700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4f4109a60 con 0x7fe4f40d1c70 2021-02-10T11:33:23.372 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.366 7fe4fa0f3700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4f40e5ef0 con 0x7fe4f41b0ae0 2021-02-10T11:33:23.372 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.366 7fe4fa0f3700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4f4109750 con 0x7fe4f40d1870 2021-02-10T11:33:23.372 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.366 7fe4f3fff700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fe4f40d1c70 0x7fe4f4108f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:49156/0 (socket says 172.21.2.112:49156) 2021-02-10T11:33:23.373 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.366 7fe4f3fff700 1 -- 172.21.2.112:0/1967809015 learned_addr learned my addr 172.21.2.112:0/1967809015 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:33:23.373 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.366 7fe4f2ffd700 1 --2- 172.21.2.112:0/1967809015 >> v2:172.21.2.120:3300/0 conn(0x7fe4f41b0ae0 0x7fe4f4108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:23.373 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.366 7fe4f27fc700 1 -- 172.21.2.112:0/1967809015 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1263514258 0 0) 0x7fe4ec001500 con 0x7fe4f40d1c70 2021-02-10T11:33:23.373 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.366 7fe4f37fe700 1 --2- 172.21.2.112:0/1967809015 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe4f40d1870 0x7fe4f41b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:23.373 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.366 7fe4f27fc700 1 -- 172.21.2.112:0/1967809015 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4e0002fe0 con 0x7fe4f40d1c70 2021-02-10T11:33:23.373 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.367 7fe4f27fc700 1 -- 172.21.2.112:0/1967809015 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1536234687 0 0) 0x7fe4ec001500 con 0x7fe4f40d1c70 2021-02-10T11:33:23.373 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.367 7fe4f27fc700 1 -- 172.21.2.112:0/1967809015 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe4f40d1870 msgr2=0x7fe4f41b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:23.373 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.367 7fe4f27fc700 1 --2- 172.21.2.112:0/1967809015 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe4f40d1870 0x7fe4f41b04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:23.373 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.367 7fe4f27fc700 1 -- 172.21.2.112:0/1967809015 >> v2:172.21.2.120:3300/0 conn(0x7fe4f41b0ae0 msgr2=0x7fe4f4108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:23.373 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.367 7fe4f27fc700 1 --2- 172.21.2.112:0/1967809015 >> v2:172.21.2.120:3300/0 conn(0x7fe4f41b0ae0 0x7fe4f4108940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:23.373 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.367 7fe4f27fc700 1 -- 172.21.2.112:0/1967809015 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4f4088020 con 0x7fe4f40d1c70 2021-02-10T11:33:23.373 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.367 7fe4f27fc700 1 -- 172.21.2.112:0/1967809015 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7fe4ec001a00 con 0x7fe4f40d1c70 2021-02-10T11:33:23.373 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.367 7fe4f27fc700 1 -- 172.21.2.112:0/1967809015 >> v1:172.21.2.112:6789/0 conn(0x7fe4f40d1c70 legacy=0x7fe4f4108f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:23.374 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.367 7fe4f27fc700 1 --2- 172.21.2.112:0/1967809015 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe4e0003b10 0x7fe4e0005f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:23.374 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.367 7fe4f27fc700 1 --2- 172.21.2.112:0/1967809015 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe4e00067a0 0x7fe4e0008c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:23.374 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.367 7fe4f27fc700 1 --2- 172.21.2.112:0/1967809015 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe4e00094f0 0x7fe4e000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:23.374 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.367 7fe4f27fc700 1 -- 172.21.2.112:0/1967809015 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4e000c2c0 con 0x7fe4e00094f0 2021-02-10T11:33:23.374 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.367 7fe4f27fc700 1 -- 172.21.2.112:0/1967809015 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4e00504b0 con 0x7fe4e0003b10 2021-02-10T11:33:23.374 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.367 7fe4f27fc700 1 -- 172.21.2.112:0/1967809015 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4e00506d0 con 0x7fe4e00067a0 2021-02-10T11:33:23.374 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.367 7fe4f3fff700 1 --2- 172.21.2.112:0/1967809015 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe4e00094f0 0x7fe4e000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:23.374 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.367 7fe4f2ffd700 1 --2- 172.21.2.112:0/1967809015 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe4e00067a0 0x7fe4e0008c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:23.374 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.367 7fe4f37fe700 1 --2- 172.21.2.112:0/1967809015 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe4e0003b10 0x7fe4e0005f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:23.374 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.368 7fe4f3fff700 1 -- 172.21.2.112:0/1967809015 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe4e00067a0 msgr2=0x7fe4e0008c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:23.374 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.368 7fe4f3fff700 1 --2- 172.21.2.112:0/1967809015 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe4e00067a0 0x7fe4e0008c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:23.374 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.368 7fe4f3fff700 1 -- 172.21.2.112:0/1967809015 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe4e0003b10 msgr2=0x7fe4e0005f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:23.374 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.368 7fe4f3fff700 1 --2- 172.21.2.112:0/1967809015 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe4e0003b10 0x7fe4e0005f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:23.374 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.368 7fe4f3fff700 1 -- 172.21.2.112:0/1967809015 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4e0050970 con 0x7fe4e00094f0 2021-02-10T11:33:23.375 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.368 7fe4f2ffd700 1 --2- 172.21.2.112:0/1967809015 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe4e00067a0 0x7fe4e0008c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:33:23.375 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.368 7fe4f3fff700 1 --2- 172.21.2.112:0/1967809015 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe4e00094f0 0x7fe4e000ba10 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 rx=0x7fe4ec001130 tx=0x7fe4ec00a310).ready entity=mon.0 client_cookie=d33de689bce0ab10 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:23.375 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.368 7fe4f27fc700 1 -- 172.21.2.112:0/1967809015 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe4ec001460 con 0x7fe4e00094f0 2021-02-10T11:33:23.375 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.368 7fe4f27fc700 1 -- 172.21.2.112:0/1967809015 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4ec0070c0 con 0x7fe4e00094f0 2021-02-10T11:33:23.375 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.368 7fe4f27fc700 1 -- 172.21.2.112:0/1967809015 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe4ec001460 con 0x7fe4e00094f0 2021-02-10T11:33:23.375 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.368 7fe4fa0f3700 1 -- 172.21.2.112:0/1967809015 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe4e00094f0 msgr2=0x7fe4e000ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:23.375 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.368 7fe4fa0f3700 1 --2- 172.21.2.112:0/1967809015 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe4e00094f0 0x7fe4e000ba10 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 rx=0x7fe4ec001130 tx=0x7fe4ec00a310).stop 2021-02-10T11:33:23.375 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.368 7fe4f3fff700 1 -- 172.21.2.112:0/1967809015 reap_dead start 2021-02-10T11:33:23.375 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.369 7fe4fa0f3700 1 -- 172.21.2.112:0/1967809015 shutdown_connections 2021-02-10T11:33:23.375 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.369 7fe4fa0f3700 1 -- 172.21.2.112:0/1967809015 >> 172.21.2.112:0/1967809015 conn(0x7fe4f40ce420 msgr2=0x7fe4f40d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:23.375 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.369 7fe4fa0f3700 1 -- 172.21.2.112:0/1967809015 shutdown_connections 2021-02-10T11:33:23.375 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.369 7fe4fa0f3700 1 -- 172.21.2.112:0/1967809015 wait complete. 2021-02-10T11:33:23.375 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.384 7fe4fa0f3700 1 Processor -- start 2021-02-10T11:33:23.376 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.384 7fe4fa0f3700 1 -- start start 2021-02-10T11:33:23.377 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.384 7fe4fa0f3700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe4f41a7a60 0x7fe4f41a7e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:23.377 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.384 7fe4fa0f3700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe4f409c620 0x7fe4f409ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:23.377 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.384 7fe4fa0f3700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe4f409d040 0x7fe4f40be800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:23.377 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.384 7fe4fa0f3700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4f409d440 con 0x7fe4f409d040 2021-02-10T11:33:23.377 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.384 7fe4fa0f3700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4f40b0960 con 0x7fe4f409c620 2021-02-10T11:33:23.377 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.384 7fe4fa0f3700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4f40c0690 con 0x7fe4f41a7a60 2021-02-10T11:33:23.377 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.384 7fe4f3fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe4f409d040 0x7fe4f40be800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:23.377 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.384 7fe4f3fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe4f409d040 0x7fe4f40be800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:49704/0 (socket says 172.21.2.112:49704) 2021-02-10T11:33:23.377 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.384 7fe4f3fff700 1 -- 172.21.2.112:0/470065837 learned_addr learned my addr 172.21.2.112:0/470065837 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:33:23.377 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.385 7fe4f3fff700 1 -- 172.21.2.112:0/470065837 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe4f41a7a60 msgr2=0x7fe4f41a7e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:23.377 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.385 7fe4f3fff700 1 --2- 172.21.2.112:0/470065837 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe4f41a7a60 0x7fe4f41a7e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:23.378 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.385 7fe4f3fff700 1 -- 172.21.2.112:0/470065837 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe4f409c620 msgr2=0x7fe4f409ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:23.378 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.385 7fe4f3fff700 1 --2- 172.21.2.112:0/470065837 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe4f409c620 0x7fe4f409ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:23.378 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.385 7fe4f3fff700 1 -- 172.21.2.112:0/470065837 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4f40bee20 con 0x7fe4f409d040 2021-02-10T11:33:23.378 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.385 7fe4f3fff700 1 --2- 172.21.2.112:0/470065837 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe4f409d040 0x7fe4f40be800 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 rx=0x7fe4ec001e50 tx=0x7fe4ec00acb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:23.378 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.385 7fe4ebfff700 1 -- 172.21.2.112:0/470065837 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe4ec001760 con 0x7fe4f409d040 2021-02-10T11:33:23.378 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.385 7fe4ebfff700 1 -- 172.21.2.112:0/470065837 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4ec004930 con 0x7fe4f409d040 2021-02-10T11:33:23.378 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.385 7fe4ebfff700 1 -- 172.21.2.112:0/470065837 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe4ec007020 con 0x7fe4f409d040 2021-02-10T11:33:23.378 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.385 7fe4fa0f3700 1 -- 172.21.2.112:0/470065837 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4f40bee20 con 0x7fe4f409d040 2021-02-10T11:33:23.378 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.385 7fe4fa0f3700 1 -- 172.21.2.112:0/470065837 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4f40bee20 con 0x7fe4f409d040 2021-02-10T11:33:23.378 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.386 7fe4ebfff700 1 -- 172.21.2.112:0/470065837 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe4ec00adf0 con 0x7fe4f409d040 2021-02-10T11:33:23.378 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.386 7fe4ebfff700 1 --2- 172.21.2.112:0/470065837 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe4d80186c0 0x7fe4d801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:23.378 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.386 7fe4ebfff700 1 -- 172.21.2.112:0/470065837 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 5457+0+0 (secure 0 0 0) 0x7fe4ec0328b0 con 0x7fe4f409d040 2021-02-10T11:33:23.378 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.386 7fe4f37fe700 1 --2- 172.21.2.112:0/470065837 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe4d80186c0 0x7fe4d801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:23.378 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.387 7fe4f37fe700 1 --2- 172.21.2.112:0/470065837 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe4d80186c0 0x7fe4d801abc0 crc :-1 s=READY pgs=617 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:23.404 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.416 7fe4fa0f3700 1 -- 172.21.2.112:0/470065837 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4f4082920 con 0x7fe4f409d040 2021-02-10T11:33:23.406 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.418 7fe4ebfff700 1 -- 172.21.2.112:0/470065837 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe4ec020ef0 con 0x7fe4f409d040 2021-02-10T11:33:23.536 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.548 7fe4fa0f3700 1 -- 172.21.2.112:0/470065837 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"} v 0) v1 -- 0x7fe4f4082920 con 0x7fe4f409d040 2021-02-10T11:33:23.540 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.552 7fe4ebfff700 1 -- 172.21.2.112:0/470065837 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"}]=0 new fs with metadata pool 10 and data pool 11 v36) v1 ==== 189+0+0 (secure 0 0 0) 0x7fe4ec00adf0 con 0x7fe4f409d040 2021-02-10T11:33:23.545 INFO:teuthology.orchestra.run.gibba012.stderr:new fs with metadata pool 10 and data pool 11 2021-02-10T11:33:23.546 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.558 7fe4fa0f3700 1 -- 172.21.2.112:0/470065837 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe4d80186c0 msgr2=0x7fe4d801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:23.546 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.558 7fe4fa0f3700 1 --2- 172.21.2.112:0/470065837 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe4d80186c0 0x7fe4d801abc0 crc :-1 s=READY pgs=617 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:23.546 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.558 7fe4fa0f3700 1 -- 172.21.2.112:0/470065837 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe4f409d040 msgr2=0x7fe4f40be800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:23.546 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.558 7fe4fa0f3700 1 --2- 172.21.2.112:0/470065837 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe4f409d040 0x7fe4f40be800 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 rx=0x7fe4ec001e50 tx=0x7fe4ec00acb0).stop 2021-02-10T11:33:23.546 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.558 7fe4fa0f3700 1 -- 172.21.2.112:0/470065837 shutdown_connections 2021-02-10T11:33:23.546 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.558 7fe4fa0f3700 1 --2- 172.21.2.112:0/470065837 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe4d80186c0 0x7fe4d801abc0 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:23.546 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.558 7fe4fa0f3700 1 --2- 172.21.2.112:0/470065837 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe4f409c620 0x7fe4f409ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:23.546 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.558 7fe4fa0f3700 1 --2- 172.21.2.112:0/470065837 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe4f409d040 0x7fe4f40be800 unknown :-1 s=CLOSED pgs=1235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:23.546 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.558 7fe4fa0f3700 1 --2- 172.21.2.112:0/470065837 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe4f41a7a60 0x7fe4f41a7e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:23.546 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.558 7fe4fa0f3700 1 -- 172.21.2.112:0/470065837 >> 172.21.2.112:0/470065837 conn(0x7fe4f40c0dc0 msgr2=0x7fe4f40ad400 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:23.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.558 7fe4fa0f3700 1 -- 172.21.2.112:0/470065837 shutdown_connections 2021-02-10T11:33:23.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.558 7fe4fa0f3700 1 -- 172.21.2.112:0/470065837 wait complete. 2021-02-10T11:33:23.554 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:33:23.979 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.974 7f3bf4f8c700 1 Processor -- start 2021-02-10T11:33:23.979 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.974 7f3bf4f8c700 1 -- start start 2021-02-10T11:33:23.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.975 7f3bf4f8c700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f3bf00e5470 0x7f3bf007e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:23.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.975 7f3bf4f8c700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3bf007f300 0x7f3bf007f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:23.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.975 7f3bf4f8c700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bf007ed80 con 0x7f3bf00e5070 2021-02-10T11:33:23.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.975 7f3bf4f8c700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3bf01b4010 con 0x7f3bf00e5470 2021-02-10T11:33:23.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.975 7f3bf4f8c700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bf00d1790 con 0x7f3bf007f300 2021-02-10T11:33:23.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.975 7f3bee59c700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f3bf00e5470 0x7f3bf007e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:23.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.975 7f3bee59c700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f3bf00e5470 0x7f3bf007e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3300/0 says I am v2:172.21.2.112:51994/0 (socket says 172.21.2.112:51994) 2021-02-10T11:33:23.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.976 7f3bee59c700 1 -- 172.21.2.112:0/3321329265 learned_addr learned my addr 172.21.2.112:0/3321329265 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:33:23.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.976 7f3bef59e700 1 --2- 172.21.2.112:0/3321329265 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3bf007f300 0x7f3bf007f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:23.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.976 7f3bee59c700 1 -- 172.21.2.112:0/3321329265 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3bf007f300 msgr2=0x7f3bf007f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:23.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.976 7f3bee59c700 1 --2- 172.21.2.112:0/3321329265 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3bf007f300 0x7f3bf007f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:23.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.976 7f3bee59c700 1 -- 172.21.2.112:0/3321329265 >> v1:172.21.2.120:6789/0 conn(0x7f3bf00e5070 legacy=0x7f3bf01b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:23.981 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.976 7f3bee59c700 1 -- 172.21.2.112:0/3321329265 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bf00d19b0 con 0x7f3bf00e5470 2021-02-10T11:33:23.981 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.976 7f3bef59e700 1 --2- 172.21.2.112:0/3321329265 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3bf007f300 0x7f3bf007f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:33:23.981 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.976 7f3bee59c700 1 --2- 172.21.2.112:0/3321329265 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3bf00e5470 0x7f3bf007e6a0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 rx=0x7f3be0006700 tx=0x7f3be0009b80).ready entity=mon.1 client_cookie=cf0d4d7da88303a6 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:23.981 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.977 7f3bedd9b700 1 -- 172.21.2.112:0/3321329265 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3be0006340 con 0x7f3bf00e5470 2021-02-10T11:33:23.981 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.977 7f3bedd9b700 1 -- 172.21.2.112:0/3321329265 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3be000c0c0 con 0x7f3bf00e5470 2021-02-10T11:33:23.981 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.977 7f3bedd9b700 1 -- 172.21.2.112:0/3321329265 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3be000c5e0 con 0x7f3bf00e5470 2021-02-10T11:33:23.981 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.977 7f3bf4f8c700 1 -- 172.21.2.112:0/3321329265 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3bf00e5470 msgr2=0x7f3bf007e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:23.981 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.977 7f3bf4f8c700 1 --2- 172.21.2.112:0/3321329265 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3bf00e5470 0x7f3bf007e6a0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 rx=0x7f3be0006700 tx=0x7f3be0009b80).stop 2021-02-10T11:33:23.981 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.977 7f3bf4f8c700 1 -- 172.21.2.112:0/3321329265 shutdown_connections 2021-02-10T11:33:23.981 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.977 7f3bf4f8c700 1 --2- 172.21.2.112:0/3321329265 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3bf00e5470 0x7f3bf007e6a0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:23.981 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.977 7f3bf4f8c700 1 --2- 172.21.2.112:0/3321329265 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3bf007f300 0x7f3bf007f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:23.981 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.977 7f3bf4f8c700 1 -- 172.21.2.112:0/3321329265 >> 172.21.2.112:0/3321329265 conn(0x7f3bf00ce690 msgr2=0x7f3bf00e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:23.981 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.977 7f3bf4f8c700 1 -- 172.21.2.112:0/3321329265 shutdown_connections 2021-02-10T11:33:23.981 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.977 7f3bf4f8c700 1 -- 172.21.2.112:0/3321329265 wait complete. 2021-02-10T11:33:23.983 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.992 7f3bf4f8c700 1 Processor -- start 2021-02-10T11:33:23.983 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.992 7f3bf4f8c700 1 -- start start 2021-02-10T11:33:23.983 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.992 7f3bf4f8c700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3bf01918e0 0x7f3bf0189c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:23.983 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.992 7f3bf4f8c700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3bf018a2a0 0x7f3bf018a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:23.984 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.992 7f3bf4f8c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3bf018acc0 0x7f3bf0190070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:23.984 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.992 7f3bf4f8c700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bf01a2f00 con 0x7f3bf018acc0 2021-02-10T11:33:23.984 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.992 7f3bf4f8c700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bf0191ce0 con 0x7f3bf01918e0 2021-02-10T11:33:23.984 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.992 7f3bf4f8c700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bf0191f00 con 0x7f3bf018a2a0 2021-02-10T11:33:23.984 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.992 7f3bef59e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3bf018acc0 0x7f3bf0190070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:23.984 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.992 7f3bef59e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3bf018acc0 0x7f3bf0190070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:49722/0 (socket says 172.21.2.112:49722) 2021-02-10T11:33:23.984 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.992 7f3bef59e700 1 -- 172.21.2.112:0/181375088 learned_addr learned my addr 172.21.2.112:0/181375088 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:33:23.984 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.993 7f3bef59e700 1 -- 172.21.2.112:0/181375088 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3bf018a2a0 msgr2=0x7f3bf018a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:23.984 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.993 7f3bef59e700 1 --2- 172.21.2.112:0/181375088 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3bf018a2a0 0x7f3bf018a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:23.984 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.993 7f3bef59e700 1 -- 172.21.2.112:0/181375088 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3bf01918e0 msgr2=0x7f3bf0189c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:23.984 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.993 7f3bef59e700 1 --2- 172.21.2.112:0/181375088 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3bf01918e0 0x7f3bf0189c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:23.984 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.993 7f3bef59e700 1 -- 172.21.2.112:0/181375088 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bf0190690 con 0x7f3bf018acc0 2021-02-10T11:33:23.984 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.993 7f3bef59e700 1 --2- 172.21.2.112:0/181375088 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3bf018acc0 0x7f3bf0190070 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 rx=0x7f3be800b970 tx=0x7f3be800bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:23.984 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.993 7f3be77fe700 1 -- 172.21.2.112:0/181375088 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3be8000f90 con 0x7f3bf018acc0 2021-02-10T11:33:23.985 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.993 7f3be77fe700 1 -- 172.21.2.112:0/181375088 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3be8006790 con 0x7f3bf018acc0 2021-02-10T11:33:23.985 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.993 7f3be77fe700 1 -- 172.21.2.112:0/181375088 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3be80015f0 con 0x7f3bf018acc0 2021-02-10T11:33:23.985 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.993 7f3bf4f8c700 1 -- 172.21.2.112:0/181375088 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bf0190690 con 0x7f3bf018acc0 2021-02-10T11:33:23.985 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.993 7f3bf4f8c700 1 -- 172.21.2.112:0/181375088 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bf0190c20 con 0x7f3bf018acc0 2021-02-10T11:33:23.985 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.994 7f3be77fe700 1 -- 172.21.2.112:0/181375088 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3be8000df0 con 0x7f3bf018acc0 2021-02-10T11:33:23.985 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.994 7f3be77fe700 1 --2- 172.21.2.112:0/181375088 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3bd40186c0 0x7f3bd401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:23.985 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.994 7f3be77fe700 1 -- 172.21.2.112:0/181375088 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f3be80328f0 con 0x7f3bf018acc0 2021-02-10T11:33:23.985 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.994 7f3beed9d700 1 --2- 172.21.2.112:0/181375088 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3bd40186c0 0x7f3bd401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:23.985 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:23.995 7f3beed9d700 1 --2- 172.21.2.112:0/181375088 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3bd40186c0 0x7f3bd401abc0 crc :-1 s=READY pgs=618 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:24.013 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:24.025 7f3bf4f8c700 1 -- 172.21.2.112:0/181375088 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bf007ed80 con 0x7f3bf018acc0 2021-02-10T11:33:24.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:24.028 7f3be77fe700 1 -- 172.21.2.112:0/181375088 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3be8006af0 con 0x7f3bf018acc0 2021-02-10T11:33:24.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:24.153 7f3bf4f8c700 1 -- 172.21.2.112:0/181375088 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3bf007ed80 con 0x7f3bf018acc0 2021-02-10T11:33:24.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:24.154 7f3be77fe700 1 -- 172.21.2.112:0/181375088 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v54) v1 ==== 74+0+14009 (secure 0 0 0) 0x7f3be8001ef0 con 0x7f3bf018acc0 2021-02-10T11:33:24.142 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:33:24.142 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":54,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:33:23.550705","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:33:21.523665","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":11,"pool_name":"cephfs_data","create_time":"2021-02-10 11:33:22.595892","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:33:24.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:24.155 7f3bf4f8c700 1 -- 172.21.2.112:0/181375088 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3bd40186c0 msgr2=0x7f3bd401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:24.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:24.155 7f3bf4f8c700 1 --2- 172.21.2.112:0/181375088 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3bd40186c0 0x7f3bd401abc0 crc :-1 s=READY pgs=618 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:24.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:24.155 7f3bf4f8c700 1 -- 172.21.2.112:0/181375088 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3bf018acc0 msgr2=0x7f3bf0190070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:24.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:24.155 7f3bf4f8c700 1 --2- 172.21.2.112:0/181375088 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3bf018acc0 0x7f3bf0190070 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 rx=0x7f3be800b970 tx=0x7f3be800bd10).stop 2021-02-10T11:33:24.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:24.155 7f3bf4f8c700 1 -- 172.21.2.112:0/181375088 shutdown_connections 2021-02-10T11:33:24.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:24.155 7f3bf4f8c700 1 --2- 172.21.2.112:0/181375088 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3bd40186c0 0x7f3bd401abc0 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:24.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:24.155 7f3bf4f8c700 1 --2- 172.21.2.112:0/181375088 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3bf018acc0 0x7f3bf0190070 unknown :-1 s=CLOSED pgs=1236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:24.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:24.155 7f3bf4f8c700 1 --2- 172.21.2.112:0/181375088 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3bf018a2a0 0x7f3bf018a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:24.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:24.155 7f3bf4f8c700 1 --2- 172.21.2.112:0/181375088 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3bf01918e0 0x7f3bf0189c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:24.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:24.155 7f3bf4f8c700 1 -- 172.21.2.112:0/181375088 >> 172.21.2.112:0/181375088 conn(0x7f3bf01a31d0 msgr2=0x7f3bf008b830 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:24.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:24.156 7f3bf4f8c700 1 -- 172.21.2.112:0/181375088 shutdown_connections 2021-02-10T11:33:24.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:24.156 7f3bf4f8c700 1 -- 172.21.2.112:0/181375088 wait complete. 2021-02-10T11:33:24.153 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:33:24.580 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:24.575 7f3a104e0700 1 Processor -- start 2021-02-10T11:33:24.581 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:24.575 7f3a104e0700 1 -- start start 2021-02-10T11:33:24.581 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:24.576 7f3a104e0700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f3a080e5070 0x7f3a081b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:24.581 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:24.576 7f3a104e0700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f3a080e5470 0x7f3a08080310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:24.581 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:24.576 7f3a104e0700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a0807ed80 con 0x7f3a081b3e40 2021-02-10T11:33:24.581 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:24.576 7f3a104e0700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a080d1790 con 0x7f3a080e5070 2021-02-10T11:33:24.581 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:24.576 7f3a104e0700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a080d19b0 con 0x7f3a080e5470 2021-02-10T11:33:24.581 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:24.576 7f3a0dafb700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f3a081b3e40 0x7f3a0807d690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:49188/0 (socket says 172.21.2.112:49188) 2021-02-10T11:33:24.581 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:24.576 7f3a0dafb700 1 -- 172.21.2.112:0/1573602388 learned_addr learned my addr 172.21.2.112:0/1573602388 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:33:24.581 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:24.576 7f3a0e2fc700 1 --2- 172.21.2.112:0/1573602388 >> v2:172.21.2.112:3300/0 conn(0x7f3a080e5070 0x7f3a081b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:24.581 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:24.576 7f3a0eafd700 1 --2- 172.21.2.112:0/1573602388 >> v2:172.21.2.120:3300/0 conn(0x7f3a080e5470 0x7f3a08080310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:24.581 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:24.576 7f3a0d2fa700 1 -- 172.21.2.112:0/1573602388 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3121909264 0 0) 0x7f3a040010b0 con 0x7f3a081b3e40 2021-02-10T11:33:24.582 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:24.576 7f3a0d2fa700 1 -- 172.21.2.112:0/1573602388 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39f4002fe0 con 0x7f3a081b3e40 2021-02-10T11:33:24.582 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:24.576 7f3a0e2fc700 1 -- 172.21.2.112:0/1573602388 >> v2:172.21.2.120:3300/0 conn(0x7f3a080e5470 msgr2=0x7f3a08080310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:24.582 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:24.576 7f3a0e2fc700 1 --2- 172.21.2.112:0/1573602388 >> v2:172.21.2.120:3300/0 conn(0x7f3a080e5470 0x7f3a08080310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:24.582 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:24.576 7f3a0e2fc700 1 -- 172.21.2.112:0/1573602388 >> v1:172.21.2.112:6789/0 conn(0x7f3a081b3e40 legacy=0x7f3a0807d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:24.582 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:24.576 7f3a0e2fc700 1 -- 172.21.2.112:0/1573602388 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a080d1bd0 con 0x7f3a080e5070 2021-02-10T11:33:24.582 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:24.576 7f3a0eafd700 1 --2- 172.21.2.112:0/1573602388 >> v2:172.21.2.120:3300/0 conn(0x7f3a080e5470 0x7f3a08080310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:33:24.582 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:24.577 7f3a0e2fc700 1 --2- 172.21.2.112:0/1573602388 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3a080e5070 0x7f3a081b3820 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 rx=0x7f39fc006650 tx=0x7f39fc009a80).ready entity=mon.0 client_cookie=e0d1a2f6c300f77f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:24.582 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:24.577 7f3a0d2fa700 1 -- 172.21.2.112:0/1573602388 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f39fc0009b0 con 0x7f3a080e5070 2021-02-10T11:33:24.582 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:24.577 7f3a0d2fa700 1 -- 172.21.2.112:0/1573602388 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39fc00c0c0 con 0x7f3a080e5070 2021-02-10T11:33:24.582 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:24.577 7f3a0d2fa700 1 -- 172.21.2.112:0/1573602388 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f39fc00c520 con 0x7f3a080e5070 2021-02-10T11:33:24.582 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:24.577 7f3a104e0700 1 -- 172.21.2.112:0/1573602388 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3a080e5070 msgr2=0x7f3a081b3820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:24.582 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:24.577 7f3a104e0700 1 --2- 172.21.2.112:0/1573602388 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3a080e5070 0x7f3a081b3820 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 rx=0x7f39fc006650 tx=0x7f39fc009a80).stop 2021-02-10T11:33:24.582 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:24.578 7f3a104e0700 1 -- 172.21.2.112:0/1573602388 shutdown_connections 2021-02-10T11:33:24.583 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:24.578 7f3a104e0700 1 --2- 172.21.2.112:0/1573602388 >> v2:172.21.2.120:3300/0 conn(0x7f3a080e5470 0x7f3a08080310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:24.583 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:24.578 7f3a104e0700 1 --2- 172.21.2.112:0/1573602388 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3a080e5070 0x7f3a081b3820 unknown :-1 s=CLOSED pgs=1238 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:24.583 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:24.578 7f3a104e0700 1 -- 172.21.2.112:0/1573602388 >> 172.21.2.112:0/1573602388 conn(0x7f3a080ce690 msgr2=0x7f3a080e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:24.583 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:24.578 7f3a104e0700 1 -- 172.21.2.112:0/1573602388 shutdown_connections 2021-02-10T11:33:24.583 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:24.578 7f3a104e0700 1 -- 172.21.2.112:0/1573602388 wait complete. 2021-02-10T11:33:24.586 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:24.593 7f3a104e0700 1 Processor -- start 2021-02-10T11:33:24.586 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:24.593 7f3a104e0700 1 -- start start 2021-02-10T11:33:24.586 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:24.593 7f3a104e0700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3a0809dbf0 0x7f3a08189ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:24.586 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:24.593 7f3a104e0700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3a0809dff0 0x7f3a0818a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:24.587 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:24.593 7f3a104e0700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3a080b7f50 0x7f3a080b8350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:24.587 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:24.593 7f3a104e0700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a0818ad50 con 0x7f3a0809dff0 2021-02-10T11:33:24.587 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:24.593 7f3a104e0700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a080b8970 con 0x7f3a080b7f50 2021-02-10T11:33:24.587 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:24.593 7f3a104e0700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a080b8b90 con 0x7f3a0809dbf0 2021-02-10T11:33:24.587 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:24.594 7f3a0dafb700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3a0809dff0 0x7f3a0818a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:24.587 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:24.594 7f3a0dafb700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3a0809dff0 0x7f3a0818a610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:49730/0 (socket says 172.21.2.112:49730) 2021-02-10T11:33:24.587 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:24.594 7f3a0dafb700 1 -- 172.21.2.112:0/1794352439 learned_addr learned my addr 172.21.2.112:0/1794352439 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:33:24.587 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:24.594 7f3a0e2fc700 1 --2- 172.21.2.112:0/1794352439 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3a0809dbf0 0x7f3a08189ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:24.587 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:24.594 7f3a0eafd700 1 --2- 172.21.2.112:0/1794352439 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3a080b7f50 0x7f3a080b8350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:24.587 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:24.594 7f3a0dafb700 1 -- 172.21.2.112:0/1794352439 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3a0809dbf0 msgr2=0x7f3a08189ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:24.588 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:24.594 7f3a0dafb700 1 --2- 172.21.2.112:0/1794352439 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3a0809dbf0 0x7f3a08189ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:24.588 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:24.594 7f3a0dafb700 1 -- 172.21.2.112:0/1794352439 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3a080b7f50 msgr2=0x7f3a080b8350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:24.588 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:24.594 7f3a0dafb700 1 --2- 172.21.2.112:0/1794352439 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3a080b7f50 0x7f3a080b8350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:24.588 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:24.594 7f3a0dafb700 1 -- 172.21.2.112:0/1794352439 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a080b8db0 con 0x7f3a0809dff0 2021-02-10T11:33:24.588 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:24.594 7f3a0dafb700 1 --2- 172.21.2.112:0/1794352439 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3a0809dff0 0x7f3a0818a610 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 rx=0x7f3a04001c60 tx=0x7f3a0400a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:24.589 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:24.594 7f39faffd700 1 -- 172.21.2.112:0/1794352439 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3a0400acc0 con 0x7f3a0809dff0 2021-02-10T11:33:24.589 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:24.594 7f39faffd700 1 -- 172.21.2.112:0/1794352439 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a04004350 con 0x7f3a0809dff0 2021-02-10T11:33:24.589 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:24.594 7f3a104e0700 1 -- 172.21.2.112:0/1794352439 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a080b8db0 con 0x7f3a0809dff0 2021-02-10T11:33:24.589 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:24.594 7f39faffd700 1 -- 172.21.2.112:0/1794352439 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3a040047b0 con 0x7f3a0809dff0 2021-02-10T11:33:24.589 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:24.594 7f3a104e0700 1 -- 172.21.2.112:0/1794352439 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a080b8db0 con 0x7f3a0809dff0 2021-02-10T11:33:24.589 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:24.595 7f39faffd700 1 -- 172.21.2.112:0/1794352439 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3a04004530 con 0x7f3a0809dff0 2021-02-10T11:33:24.590 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:24.595 7f39faffd700 1 --2- 172.21.2.112:0/1794352439 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f39f0018730 0x7f39f001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:24.590 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:24.595 7f39faffd700 1 -- 172.21.2.112:0/1794352439 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f3a040327c0 con 0x7f3a0809dff0 2021-02-10T11:33:24.590 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:24.595 7f3a0e2fc700 1 --2- 172.21.2.112:0/1794352439 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f39f0018730 0x7f39f001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:24.590 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:24.595 7f3a0e2fc700 1 --2- 172.21.2.112:0/1794352439 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f39f0018730 0x7f39f001ac10 crc :-1 s=READY pgs=619 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:24.619 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:24.624 7f3a104e0700 1 -- 172.21.2.112:0/1794352439 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a08084b50 con 0x7f3a0809dff0 2021-02-10T11:33:24.619 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:24.627 7f39faffd700 1 -- 172.21.2.112:0/1794352439 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3a04004140 con 0x7f3a0809dff0 2021-02-10T11:33:24.741 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:24.753 7f3a104e0700 1 -- 172.21.2.112:0/1794352439 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3a0807ed80 con 0x7f3a0809dff0 2021-02-10T11:33:24.741 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:24.754 7f39faffd700 1 -- 172.21.2.112:0/1794352439 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v55) v1 ==== 74+0+13996 (secure 0 0 0) 0x7f3a04004350 con 0x7f3a0809dff0 2021-02-10T11:33:24.742 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:33:24.742 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":55,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:33:24.553170","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:33:21.523665","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":11,"pool_name":"cephfs_data","create_time":"2021-02-10 11:33:22.595892","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:33:24.743 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:24.755 7f3a104e0700 1 -- 172.21.2.112:0/1794352439 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f39f0018730 msgr2=0x7f39f001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:24.744 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:24.755 7f3a104e0700 1 --2- 172.21.2.112:0/1794352439 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f39f0018730 0x7f39f001ac10 crc :-1 s=READY pgs=619 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:24.744 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:24.755 7f3a104e0700 1 -- 172.21.2.112:0/1794352439 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3a0809dff0 msgr2=0x7f3a0818a610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:24.744 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:24.755 7f3a104e0700 1 --2- 172.21.2.112:0/1794352439 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3a0809dff0 0x7f3a0818a610 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 rx=0x7f3a04001c60 tx=0x7f3a0400a8d0).stop 2021-02-10T11:33:24.744 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:24.756 7f3a104e0700 1 -- 172.21.2.112:0/1794352439 shutdown_connections 2021-02-10T11:33:24.744 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:24.756 7f3a104e0700 1 --2- 172.21.2.112:0/1794352439 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f39f0018730 0x7f39f001ac10 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:24.744 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:24.756 7f3a104e0700 1 --2- 172.21.2.112:0/1794352439 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3a080b7f50 0x7f3a080b8350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:24.744 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:24.756 7f3a104e0700 1 --2- 172.21.2.112:0/1794352439 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3a0809dff0 0x7f3a0818a610 unknown :-1 s=CLOSED pgs=1239 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:24.744 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:24.756 7f3a104e0700 1 --2- 172.21.2.112:0/1794352439 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3a0809dbf0 0x7f3a08189ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:24.744 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:24.756 7f3a104e0700 1 -- 172.21.2.112:0/1794352439 >> 172.21.2.112:0/1794352439 conn(0x7f3a080855d0 msgr2=0x7f3a081ab840 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:24.744 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:24.756 7f3a104e0700 1 -- 172.21.2.112:0/1794352439 shutdown_connections 2021-02-10T11:33:24.744 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:24.756 7f3a104e0700 1 -- 172.21.2.112:0/1794352439 wait complete. 2021-02-10T11:33:24.752 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2021-02-10T11:33:25.182 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:25.177 7febfded4700 1 Processor -- start 2021-02-10T11:33:25.182 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:25.177 7febfded4700 1 -- start start 2021-02-10T11:33:25.182 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:25.178 7febfded4700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7febf81b0470 0x7febf81b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:25.182 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:25.178 7febfded4700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7febf80ef610 0x7febf80efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:25.182 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:25.178 7febfded4700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febf80f0b50 con 0x7febf80f0030 2021-02-10T11:33:25.182 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:25.178 7febfded4700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febf80e52e0 con 0x7febf80ef610 2021-02-10T11:33:25.182 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:25.178 7febfded4700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febf80e5490 con 0x7febf81b0470 2021-02-10T11:33:25.182 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:25.178 7febf6ffd700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7febf80ef610 0x7febf80efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:25.183 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:25.178 7febf6ffd700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7febf80ef610 0x7febf80efa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:49736/0 (socket says 172.21.2.112:49736) 2021-02-10T11:33:25.183 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:25.178 7febf6ffd700 1 -- 172.21.2.112:0/179764606 learned_addr learned my addr 172.21.2.112:0/179764606 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:33:25.183 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:25.178 7febf6ffd700 1 -- 172.21.2.112:0/179764606 >> v2:172.21.2.120:3300/0 conn(0x7febf81b0470 msgr2=0x7febf81b0870 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:33:25.183 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:25.178 7febf6ffd700 1 --2- 172.21.2.112:0/179764606 >> v2:172.21.2.120:3300/0 conn(0x7febf81b0470 0x7febf81b0870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:25.183 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:25.178 7febf6ffd700 1 -- 172.21.2.112:0/179764606 >> v1:172.21.2.120:6789/0 conn(0x7febf80f0030 legacy=0x7febf80f10d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:33:25.183 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:25.178 7febf6ffd700 1 -- 172.21.2.112:0/179764606 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febf80f0430 con 0x7febf80ef610 2021-02-10T11:33:25.183 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:25.178 7febf6ffd700 1 --2- 172.21.2.112:0/179764606 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7febf80ef610 0x7febf80efa10 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 rx=0x7febe8006700 tx=0x7febe8009b00).ready entity=mon.0 client_cookie=a8a00d51a5174837 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:25.183 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:25.179 7febf67fc700 1 -- 172.21.2.112:0/179764606 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7febe800c0c0 con 0x7febf80ef610 2021-02-10T11:33:25.183 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:25.179 7febf67fc700 1 -- 172.21.2.112:0/179764606 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febe800c340 con 0x7febf80ef610 2021-02-10T11:33:25.183 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:25.179 7febf67fc700 1 -- 172.21.2.112:0/179764606 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7febe800c820 con 0x7febf80ef610 2021-02-10T11:33:25.184 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:25.179 7febfded4700 1 -- 172.21.2.112:0/179764606 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7febf80ef610 msgr2=0x7febf80efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:25.184 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:25.179 7febfded4700 1 --2- 172.21.2.112:0/179764606 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7febf80ef610 0x7febf80efa10 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 rx=0x7febe8006700 tx=0x7febe8009b00).stop 2021-02-10T11:33:25.184 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:25.179 7febfded4700 1 -- 172.21.2.112:0/179764606 shutdown_connections 2021-02-10T11:33:25.184 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:25.179 7febfded4700 1 --2- 172.21.2.112:0/179764606 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7febf80ef610 0x7febf80efa10 unknown :-1 s=CLOSED pgs=1240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:25.184 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:25.179 7febfded4700 1 --2- 172.21.2.112:0/179764606 >> v2:172.21.2.120:3300/0 conn(0x7febf81b0470 0x7febf81b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:25.184 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:25.179 7febfded4700 1 -- 172.21.2.112:0/179764606 >> 172.21.2.112:0/179764606 conn(0x7febf80ce3e0 msgr2=0x7febf80e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:25.184 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:25.179 7febfded4700 1 -- 172.21.2.112:0/179764606 shutdown_connections 2021-02-10T11:33:25.184 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:25.179 7febfded4700 1 -- 172.21.2.112:0/179764606 wait complete. 2021-02-10T11:33:25.185 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:25.194 7febfded4700 1 Processor -- start 2021-02-10T11:33:25.186 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:25.194 7febfded4700 1 -- start start 2021-02-10T11:33:25.186 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:25.195 7febfded4700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7febf80c36a0 0x7febf80c2b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:25.186 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:25.195 7febfded4700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7febf80c3190 0x7febf80b1d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:25.186 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:25.195 7febfded4700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7febf80b2370 0x7febf80b2d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:25.186 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:25.195 7febfded4700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febf80b27c0 con 0x7febf80c36a0 2021-02-10T11:33:25.186 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:25.195 7febfded4700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febf80b3360 con 0x7febf80b2370 2021-02-10T11:33:25.186 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:25.195 7febfded4700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febf80b3540 con 0x7febf80c3190 2021-02-10T11:33:25.186 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:25.195 7febf77fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7febf80c36a0 0x7febf80c2b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:25.186 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:25.195 7febf77fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7febf80c36a0 0x7febf80c2b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:49738/0 (socket says 172.21.2.112:49738) 2021-02-10T11:33:25.186 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:25.195 7febf77fe700 1 -- 172.21.2.112:0/1030448523 learned_addr learned my addr 172.21.2.112:0/1030448523 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:33:25.187 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:25.195 7febf77fe700 1 -- 172.21.2.112:0/1030448523 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7febf80c3190 msgr2=0x7febf80b1d50 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:33:25.187 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:25.195 7febf77fe700 1 --2- 172.21.2.112:0/1030448523 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7febf80c3190 0x7febf80b1d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:25.187 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:25.195 7febf77fe700 1 -- 172.21.2.112:0/1030448523 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7febf80b2370 msgr2=0x7febf80b2d40 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:33:25.187 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:25.195 7febf77fe700 1 --2- 172.21.2.112:0/1030448523 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7febf80b2370 0x7febf80b2d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:25.187 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:25.195 7febf77fe700 1 -- 172.21.2.112:0/1030448523 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febf80b3760 con 0x7febf80c36a0 2021-02-10T11:33:25.187 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:25.195 7febf77fe700 1 --2- 172.21.2.112:0/1030448523 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7febf80c36a0 0x7febf80c2b70 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 rx=0x7febe000b5c0 tx=0x7febe000b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:25.187 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:25.195 7febeffff700 1 -- 172.21.2.112:0/1030448523 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7febe000bd50 con 0x7febf80c36a0 2021-02-10T11:33:25.187 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:25.195 7febeffff700 1 -- 172.21.2.112:0/1030448523 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febe0006030 con 0x7febf80c36a0 2021-02-10T11:33:25.187 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:25.195 7febfded4700 1 -- 172.21.2.112:0/1030448523 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febf80b3760 con 0x7febf80c36a0 2021-02-10T11:33:25.187 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:25.195 7febfded4700 1 -- 172.21.2.112:0/1030448523 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febf80b3cf0 con 0x7febf80c36a0 2021-02-10T11:33:25.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:25.195 7febeffff700 1 -- 172.21.2.112:0/1030448523 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7febe00054d0 con 0x7febf80c36a0 2021-02-10T11:33:25.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:25.196 7febeffff700 1 -- 172.21.2.112:0/1030448523 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7febe0006210 con 0x7febf80c36a0 2021-02-10T11:33:25.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:25.196 7febeffff700 1 --2- 172.21.2.112:0/1030448523 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7febdc0186c0 0x7febdc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:25.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:25.196 7febeffff700 1 -- 172.21.2.112:0/1030448523 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5573+0+0 (secure 0 0 0) 0x7febe00325b0 con 0x7febf80c36a0 2021-02-10T11:33:25.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:25.197 7febf6ffd700 1 --2- 172.21.2.112:0/1030448523 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7febdc0186c0 0x7febdc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:25.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:25.199 7febf6ffd700 1 --2- 172.21.2.112:0/1030448523 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7febdc0186c0 0x7febdc01abc0 crc :-1 s=READY pgs=620 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:25.216 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:25.228 7febfded4700 1 -- 172.21.2.112:0/1030448523 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febf80edd10 con 0x7febf80c36a0 2021-02-10T11:33:25.218 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:25.230 7febeffff700 1 -- 172.21.2.112:0/1030448523 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7febe0000940 con 0x7febf80c36a0 2021-02-10T11:33:25.354 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:25.366 7febfded4700 1 -- 172.21.2.112:0/1030448523 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"} v 0) v1 -- 0x7febf80e52e0 con 0x7febf80c36a0 2021-02-10T11:33:25.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:25.563 7febeffff700 1 -- 172.21.2.112:0/1030448523 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"}]=0 v39) v1 ==== 118+0+0 (secure 0 0 0) 0x7febe0006210 con 0x7febf80c36a0 2021-02-10T11:33:25.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:25.582 7febfded4700 1 -- 172.21.2.112:0/1030448523 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7febdc0186c0 msgr2=0x7febdc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:25.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:25.582 7febfded4700 1 --2- 172.21.2.112:0/1030448523 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7febdc0186c0 0x7febdc01abc0 crc :-1 s=READY pgs=620 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:25.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:25.582 7febfded4700 1 -- 172.21.2.112:0/1030448523 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7febf80c36a0 msgr2=0x7febf80c2b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:25.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:25.582 7febfded4700 1 --2- 172.21.2.112:0/1030448523 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7febf80c36a0 0x7febf80c2b70 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 rx=0x7febe000b5c0 tx=0x7febe000b960).stop 2021-02-10T11:33:25.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:25.582 7febfded4700 1 -- 172.21.2.112:0/1030448523 shutdown_connections 2021-02-10T11:33:25.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:25.582 7febfded4700 1 --2- 172.21.2.112:0/1030448523 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7febdc0186c0 0x7febdc01abc0 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:25.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:25.582 7febfded4700 1 --2- 172.21.2.112:0/1030448523 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7febf80b2370 0x7febf80b2d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:25.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:25.582 7febfded4700 1 --2- 172.21.2.112:0/1030448523 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7febf80c3190 0x7febf80b1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:25.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:25.582 7febfded4700 1 --2- 172.21.2.112:0/1030448523 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7febf80c36a0 0x7febf80c2b70 unknown :-1 s=CLOSED pgs=1241 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:25.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:25.582 7febfded4700 1 -- 172.21.2.112:0/1030448523 >> 172.21.2.112:0/1030448523 conn(0x7febf809f630 msgr2=0x7febf80a0130 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:25.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:25.583 7febfded4700 1 -- 172.21.2.112:0/1030448523 shutdown_connections 2021-02-10T11:33:25.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:25.583 7febfded4700 1 -- 172.21.2.112:0/1030448523 wait complete. 2021-02-10T11:33:25.579 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:33:26.006 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.000 7fa73d215700 1 Processor -- start 2021-02-10T11:33:26.006 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.000 7fa73d215700 1 -- start start 2021-02-10T11:33:26.006 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.001 7fa73d215700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fa7380d17f0 0x7fa7381b31e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:26.006 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.001 7fa73d215700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fa7380d1bf0 0x7fa7381b3800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:26.006 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.001 7fa73d215700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa7381b3e20 0x7fa7380ceef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:26.006 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.001 7fa73d215700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7380e5eb0 con 0x7fa7380d1bf0 2021-02-10T11:33:26.006 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.001 7fa73d215700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7380f0b20 con 0x7fa7380d17f0 2021-02-10T11:33:26.006 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.001 7fa73d215700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7380cf510 con 0x7fa7381b3e20 2021-02-10T11:33:26.006 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.001 7fa73659c700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fa7380d1bf0 0x7fa7381b3800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:26.006 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.001 7fa73659c700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fa7380d1bf0 0x7fa7381b3800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:49746/0 (socket says 172.21.2.112:49746) 2021-02-10T11:33:26.007 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.001 7fa73659c700 1 -- 172.21.2.112:0/3298375064 learned_addr learned my addr 172.21.2.112:0/3298375064 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:33:26.007 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.001 7fa73659c700 1 -- 172.21.2.112:0/3298375064 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa7381b3e20 msgr2=0x7fa7380ceef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:26.007 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.001 7fa73659c700 1 --2- 172.21.2.112:0/3298375064 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa7381b3e20 0x7fa7380ceef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:26.007 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.001 7fa73659c700 1 -- 172.21.2.112:0/3298375064 >> v2:172.21.2.120:3300/0 conn(0x7fa7380d17f0 msgr2=0x7fa7381b31e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:26.007 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.001 7fa73659c700 1 --2- 172.21.2.112:0/3298375064 >> v2:172.21.2.120:3300/0 conn(0x7fa7380d17f0 0x7fa7381b31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:26.007 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.001 7fa73659c700 1 -- 172.21.2.112:0/3298375064 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7380cf6f0 con 0x7fa7380d1bf0 2021-02-10T11:33:26.007 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.002 7fa73659c700 1 --2- 172.21.2.112:0/3298375064 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa7380d1bf0 0x7fa7381b3800 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 rx=0x7fa72c006700 tx=0x7fa72c009b80).ready entity=mon.0 client_cookie=1c4fd79e0a6a8c5d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:26.008 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.002 7fa735d9b700 1 -- 172.21.2.112:0/3298375064 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa72c0009b0 con 0x7fa7380d1bf0 2021-02-10T11:33:26.008 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.002 7fa735d9b700 1 -- 172.21.2.112:0/3298375064 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa72c00c0c0 con 0x7fa7380d1bf0 2021-02-10T11:33:26.008 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.002 7fa735d9b700 1 -- 172.21.2.112:0/3298375064 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa72c00c600 con 0x7fa7380d1bf0 2021-02-10T11:33:26.008 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.002 7fa73d215700 1 -- 172.21.2.112:0/3298375064 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa7380d1bf0 msgr2=0x7fa7381b3800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:26.008 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.002 7fa73d215700 1 --2- 172.21.2.112:0/3298375064 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa7380d1bf0 0x7fa7381b3800 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 rx=0x7fa72c006700 tx=0x7fa72c009b80).stop 2021-02-10T11:33:26.008 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.002 7fa73d215700 1 -- 172.21.2.112:0/3298375064 shutdown_connections 2021-02-10T11:33:26.008 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.003 7fa73d215700 1 --2- 172.21.2.112:0/3298375064 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa7381b3e20 0x7fa7380ceef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:26.009 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.003 7fa73d215700 1 --2- 172.21.2.112:0/3298375064 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa7380d1bf0 0x7fa7381b3800 unknown :-1 s=CLOSED pgs=1242 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:26.009 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.003 7fa73d215700 1 --2- 172.21.2.112:0/3298375064 >> v2:172.21.2.120:3300/0 conn(0x7fa7380d17f0 0x7fa7381b31e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:26.009 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.003 7fa73d215700 1 -- 172.21.2.112:0/3298375064 >> 172.21.2.112:0/3298375064 conn(0x7fa7380fde40 msgr2=0x7fa7380ce390 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:26.009 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.003 7fa73d215700 1 -- 172.21.2.112:0/3298375064 shutdown_connections 2021-02-10T11:33:26.009 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.003 7fa73d215700 1 -- 172.21.2.112:0/3298375064 wait complete. 2021-02-10T11:33:26.009 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.017 7fa73d215700 1 Processor -- start 2021-02-10T11:33:26.009 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.017 7fa73d215700 1 -- start start 2021-02-10T11:33:26.009 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.018 7fa73d215700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa7381b2c30 0x7fa7381a65c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:26.009 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.018 7fa73d215700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa7381a7110 0x7fa7381b1b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:26.009 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.018 7fa73d215700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa7381b2130 0x7fa73819fed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:26.010 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.018 7fa73d215700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7381a0ff0 con 0x7fa7381b2c30 2021-02-10T11:33:26.010 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.018 7fa73d215700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7381b26e0 con 0x7fa7381a7110 2021-02-10T11:33:26.010 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.018 7fa73d215700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7381a6be0 con 0x7fa7381b2130 2021-02-10T11:33:26.011 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.018 7fa736d9d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa7381b2c30 0x7fa7381a65c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:26.011 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.018 7fa736d9d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa7381b2c30 0x7fa7381a65c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:49754/0 (socket says 172.21.2.112:49754) 2021-02-10T11:33:26.011 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.018 7fa736d9d700 1 -- 172.21.2.112:0/1578493717 learned_addr learned my addr 172.21.2.112:0/1578493717 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:33:26.011 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.018 7fa73659c700 1 --2- 172.21.2.112:0/1578493717 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa7381a7110 0x7fa7381b1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:26.012 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.018 7fa73759e700 1 --2- 172.21.2.112:0/1578493717 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa7381b2130 0x7fa73819fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:26.012 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.018 7fa736d9d700 1 -- 172.21.2.112:0/1578493717 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa7381b2130 msgr2=0x7fa73819fed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:26.012 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.018 7fa736d9d700 1 --2- 172.21.2.112:0/1578493717 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa7381b2130 0x7fa73819fed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:26.012 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.019 7fa736d9d700 1 -- 172.21.2.112:0/1578493717 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa7381a7110 msgr2=0x7fa7381b1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:26.012 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.019 7fa736d9d700 1 --2- 172.21.2.112:0/1578493717 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa7381a7110 0x7fa7381b1b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:26.012 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.019 7fa736d9d700 1 -- 172.21.2.112:0/1578493717 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7381a04f0 con 0x7fa7381b2c30 2021-02-10T11:33:26.012 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.019 7fa736d9d700 1 --2- 172.21.2.112:0/1578493717 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa7381b2c30 0x7fa7381a65c0 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 rx=0x7fa730009980 tx=0x7fa73000bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:26.012 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.019 7fa7277fe700 1 -- 172.21.2.112:0/1578493717 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa7300066d0 con 0x7fa7381b2c30 2021-02-10T11:33:26.012 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.019 7fa7277fe700 1 -- 172.21.2.112:0/1578493717 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa73000c2f0 con 0x7fa7381b2c30 2021-02-10T11:33:26.012 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.019 7fa7277fe700 1 -- 172.21.2.112:0/1578493717 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa73000c810 con 0x7fa7381b2c30 2021-02-10T11:33:26.012 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.019 7fa73d215700 1 -- 172.21.2.112:0/1578493717 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7381a04f0 con 0x7fa7381b2c30 2021-02-10T11:33:26.012 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.019 7fa73d215700 1 -- 172.21.2.112:0/1578493717 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7381a0a80 con 0x7fa7381b2c30 2021-02-10T11:33:26.013 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.020 7fa7277fe700 1 -- 172.21.2.112:0/1578493717 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa730001010 con 0x7fa7381b2c30 2021-02-10T11:33:26.013 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.020 7fa7277fe700 1 --2- 172.21.2.112:0/1578493717 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa71c0186c0 0x7fa71c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:26.013 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.020 7fa7277fe700 1 -- 172.21.2.112:0/1578493717 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fa730032900 con 0x7fa7381b2c30 2021-02-10T11:33:26.013 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.021 7fa73659c700 1 --2- 172.21.2.112:0/1578493717 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa71c0186c0 0x7fa71c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:26.013 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.021 7fa73659c700 1 --2- 172.21.2.112:0/1578493717 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa71c0186c0 0x7fa71c01abc0 crc :-1 s=READY pgs=621 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:26.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.051 7fa73d215700 1 -- 172.21.2.112:0/1578493717 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7381036d0 con 0x7fa7381b2c30 2021-02-10T11:33:26.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.053 7fa7277fe700 1 -- 172.21.2.112:0/1578493717 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa730000cc0 con 0x7fa7381b2c30 2021-02-10T11:33:26.170 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.182 7fa73d215700 1 -- 172.21.2.112:0/1578493717 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa7380f0b20 con 0x7fa7381b2c30 2021-02-10T11:33:26.170 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.182 7fa7277fe700 1 -- 172.21.2.112:0/1578493717 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) v1 ==== 94+0+2101 (secure 0 0 0) 0x7fa730001010 con 0x7fa7381b2c30 2021-02-10T11:33:26.171 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 39 2021-02-10T11:33:26.171 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:33:26.171 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":39,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7881,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/1476007789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":1476007789},{"type":"v1","addr":"172.21.2.112:6833","nonce":1476007789}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":35}],"filesystems":[{"mdsmap":{"epoch":39,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:33:23.550791","modified":"2021-02-10 11:33:25.562811","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6595":{"gid":6595,"name":"b","rank":0,"incarnation":36,"state":"up:active","state_seq":3,"addr":"172.21.2.120:6835/4241229406","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":4241229406},{"type":"v1","addr":"172.21.2.120:6835","nonce":4241229406}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2021-02-10T11:33:26.172 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.184 7fa73d215700 1 -- 172.21.2.112:0/1578493717 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa71c0186c0 msgr2=0x7fa71c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:26.172 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.184 7fa73d215700 1 --2- 172.21.2.112:0/1578493717 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa71c0186c0 0x7fa71c01abc0 crc :-1 s=READY pgs=621 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:26.172 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.184 7fa73d215700 1 -- 172.21.2.112:0/1578493717 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa7381b2c30 msgr2=0x7fa7381a65c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:26.172 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.184 7fa73d215700 1 --2- 172.21.2.112:0/1578493717 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa7381b2c30 0x7fa7381a65c0 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 rx=0x7fa730009980 tx=0x7fa73000bcd0).stop 2021-02-10T11:33:26.172 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.184 7fa73d215700 1 -- 172.21.2.112:0/1578493717 shutdown_connections 2021-02-10T11:33:26.172 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.184 7fa73d215700 1 --2- 172.21.2.112:0/1578493717 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa71c0186c0 0x7fa71c01abc0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:26.173 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.184 7fa73d215700 1 --2- 172.21.2.112:0/1578493717 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa7381a7110 0x7fa7381b1b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:26.173 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.184 7fa73d215700 1 --2- 172.21.2.112:0/1578493717 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa7381b2130 0x7fa73819fed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:26.173 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.184 7fa73d215700 1 --2- 172.21.2.112:0/1578493717 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa7381b2c30 0x7fa7381a65c0 unknown :-1 s=CLOSED pgs=1243 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:26.173 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.184 7fa73d215700 1 -- 172.21.2.112:0/1578493717 >> 172.21.2.112:0/1578493717 conn(0x7fa7381a37b0 msgr2=0x7fa738103900 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:26.173 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.184 7fa73d215700 1 -- 172.21.2.112:0/1578493717 shutdown_connections 2021-02-10T11:33:26.173 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.184 7fa73d215700 1 -- 172.21.2.112:0/1578493717 wait complete. 2021-02-10T11:33:26.180 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:33:26.607 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.603 7f11106b3700 1 Processor -- start 2021-02-10T11:33:26.608 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.603 7f11106b3700 1 -- start start 2021-02-10T11:33:26.608 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.604 7f11106b3700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f11080e5470 0x7f110807e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:26.608 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.604 7f11106b3700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f110807f300 0x7f110807f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:26.608 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.604 7f11106b3700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f110807ed80 con 0x7f11080e5070 2021-02-10T11:33:26.608 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.604 7f11106b3700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11081b4010 con 0x7f11080e5470 2021-02-10T11:33:26.608 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.604 7f11106b3700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11080d1790 con 0x7f110807f300 2021-02-10T11:33:26.608 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.604 7f110dcce700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f11080e5470 0x7f110807e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:26.608 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.604 7f110dcce700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f11080e5470 0x7f110807e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:49760/0 (socket says 172.21.2.112:49760) 2021-02-10T11:33:26.608 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.604 7f110dcce700 1 -- 172.21.2.112:0/2422288183 learned_addr learned my addr 172.21.2.112:0/2422288183 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:33:26.608 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.604 7f110dcce700 1 -- 172.21.2.112:0/2422288183 >> v2:172.21.2.120:3300/0 conn(0x7f110807f300 msgr2=0x7f110807f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:26.608 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.604 7f110dcce700 1 --2- 172.21.2.112:0/2422288183 >> v2:172.21.2.120:3300/0 conn(0x7f110807f300 0x7f110807f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:26.609 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.604 7f110dcce700 1 -- 172.21.2.112:0/2422288183 >> v1:172.21.2.120:6789/0 conn(0x7f11080e5070 legacy=0x7f11081b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:26.609 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.604 7f110dcce700 1 -- 172.21.2.112:0/2422288183 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11080d19b0 con 0x7f11080e5470 2021-02-10T11:33:26.609 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.604 7f110dcce700 1 --2- 172.21.2.112:0/2422288183 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f11080e5470 0x7f110807e6a0 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 rx=0x7f1104006700 tx=0x7f1104009b00).ready entity=mon.0 client_cookie=9e058156fec955c8 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:26.609 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.605 7f110d4cd700 1 -- 172.21.2.112:0/2422288183 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f110400c0c0 con 0x7f11080e5470 2021-02-10T11:33:26.609 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.605 7f110d4cd700 1 -- 172.21.2.112:0/2422288183 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f110400c340 con 0x7f11080e5470 2021-02-10T11:33:26.609 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.605 7f110d4cd700 1 -- 172.21.2.112:0/2422288183 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f110400c820 con 0x7f11080e5470 2021-02-10T11:33:26.609 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.605 7f11106b3700 1 -- 172.21.2.112:0/2422288183 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f11080e5470 msgr2=0x7f110807e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:26.609 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.605 7f11106b3700 1 --2- 172.21.2.112:0/2422288183 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f11080e5470 0x7f110807e6a0 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 rx=0x7f1104006700 tx=0x7f1104009b00).stop 2021-02-10T11:33:26.609 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.605 7f11106b3700 1 -- 172.21.2.112:0/2422288183 shutdown_connections 2021-02-10T11:33:26.609 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.605 7f11106b3700 1 --2- 172.21.2.112:0/2422288183 >> v2:172.21.2.120:3300/0 conn(0x7f110807f300 0x7f110807f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:26.609 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.605 7f11106b3700 1 --2- 172.21.2.112:0/2422288183 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f11080e5470 0x7f110807e6a0 unknown :-1 s=CLOSED pgs=1244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:26.609 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.605 7f11106b3700 1 -- 172.21.2.112:0/2422288183 >> 172.21.2.112:0/2422288183 conn(0x7f11080ce690 msgr2=0x7f11080e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:26.609 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.605 7f11106b3700 1 -- 172.21.2.112:0/2422288183 shutdown_connections 2021-02-10T11:33:26.610 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.605 7f11106b3700 1 -- 172.21.2.112:0/2422288183 wait complete. 2021-02-10T11:33:26.611 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.620 7f11106b3700 1 Processor -- start 2021-02-10T11:33:26.611 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.620 7f11106b3700 1 -- start start 2021-02-10T11:33:26.611 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.620 7f11106b3700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f11081a2a90 0x7f11081aa0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:26.611 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.620 7f11106b3700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f11081aa700 0x7f11081a3a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:26.612 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.620 7f11106b3700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f11080a55e0 0x7f11081a4040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:26.612 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.620 7f11106b3700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11081a34a0 con 0x7f11081aa700 2021-02-10T11:33:26.612 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.620 7f11106b3700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11081a2e90 con 0x7f11081a2a90 2021-02-10T11:33:26.612 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.620 7f11106b3700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11081a30b0 con 0x7f11080a55e0 2021-02-10T11:33:26.612 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.620 7f110dcce700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f11081aa700 0x7f11081a3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:26.612 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.620 7f110e4cf700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f11081a2a90 0x7f11081aa0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:26.612 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.620 7f110dcce700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f11081aa700 0x7f11081a3a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:49768/0 (socket says 172.21.2.112:49768) 2021-02-10T11:33:26.612 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.620 7f110dcce700 1 -- 172.21.2.112:0/3121852737 learned_addr learned my addr 172.21.2.112:0/3121852737 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:33:26.612 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.621 7f110ecd0700 1 --2- 172.21.2.112:0/3121852737 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f11080a55e0 0x7f11081a4040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:26.612 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.621 7f110dcce700 1 -- 172.21.2.112:0/3121852737 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f11080a55e0 msgr2=0x7f11081a4040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:26.612 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.621 7f110dcce700 1 --2- 172.21.2.112:0/3121852737 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f11080a55e0 0x7f11081a4040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:26.612 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.621 7f110dcce700 1 -- 172.21.2.112:0/3121852737 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f11081a2a90 msgr2=0x7f11081aa0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:26.612 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.621 7f110dcce700 1 --2- 172.21.2.112:0/3121852737 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f11081a2a90 0x7f11081aa0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:26.613 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.621 7f110dcce700 1 -- 172.21.2.112:0/3121852737 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11081a4660 con 0x7f11081aa700 2021-02-10T11:33:26.613 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.621 7f110e4cf700 1 --2- 172.21.2.112:0/3121852737 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f11081a2a90 0x7f11081aa0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:33:26.613 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.621 7f110dcce700 1 --2- 172.21.2.112:0/3121852737 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f11081aa700 0x7f11081a3a20 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 rx=0x7f1104006f20 tx=0x7f110400b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:26.613 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.621 7f10faffd700 1 -- 172.21.2.112:0/3121852737 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1104000c90 con 0x7f11081aa700 2021-02-10T11:33:26.613 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.621 7f10faffd700 1 -- 172.21.2.112:0/3121852737 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1104009100 con 0x7f11081aa700 2021-02-10T11:33:26.613 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.621 7f10faffd700 1 -- 172.21.2.112:0/3121852737 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1104002220 con 0x7f11081aa700 2021-02-10T11:33:26.613 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.621 7f11106b3700 1 -- 172.21.2.112:0/3121852737 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11081a4660 con 0x7f11081aa700 2021-02-10T11:33:26.613 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.621 7f11106b3700 1 -- 172.21.2.112:0/3121852737 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11081a4bf0 con 0x7f11081aa700 2021-02-10T11:33:26.613 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.622 7f10faffd700 1 -- 172.21.2.112:0/3121852737 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f110400a000 con 0x7f11081aa700 2021-02-10T11:33:26.613 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.622 7f10faffd700 1 --2- 172.21.2.112:0/3121852737 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f10f00186c0 0x7f10f001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:26.613 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.622 7f10faffd700 1 -- 172.21.2.112:0/3121852737 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f11040323e0 con 0x7f11081aa700 2021-02-10T11:33:26.613 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.622 7f110e4cf700 1 --2- 172.21.2.112:0/3121852737 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f10f00186c0 0x7f10f001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:26.614 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.623 7f110e4cf700 1 --2- 172.21.2.112:0/3121852737 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f10f00186c0 0x7f10f001abc0 crc :-1 s=READY pgs=622 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:26.642 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.654 7f11106b3700 1 -- 172.21.2.112:0/3121852737 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1108083da0 con 0x7f11081aa700 2021-02-10T11:33:26.644 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.656 7f10faffd700 1 -- 172.21.2.112:0/3121852737 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1104000c90 con 0x7f11081aa700 2021-02-10T11:33:26.769 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.781 7f11106b3700 1 -- 172.21.2.112:0/3121852737 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f110807ed80 con 0x7f11081aa700 2021-02-10T11:33:26.769 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.781 7f10faffd700 1 -- 172.21.2.112:0/3121852737 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v55) v1 ==== 74+0+13996 (secure 0 0 0) 0x7f110400a000 con 0x7f11081aa700 2021-02-10T11:33:26.774 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:33:26.774 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":55,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:33:24.553170","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:33:21.523665","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":11,"pool_name":"cephfs_data","create_time":"2021-02-10 11:33:22.595892","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:33:26.774 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.783 7f11106b3700 1 -- 172.21.2.112:0/3121852737 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f10f00186c0 msgr2=0x7f10f001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:26.774 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.783 7f11106b3700 1 --2- 172.21.2.112:0/3121852737 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f10f00186c0 0x7f10f001abc0 crc :-1 s=READY pgs=622 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:26.774 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.783 7f11106b3700 1 -- 172.21.2.112:0/3121852737 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f11081aa700 msgr2=0x7f11081a3a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:26.774 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.783 7f11106b3700 1 --2- 172.21.2.112:0/3121852737 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f11081aa700 0x7f11081a3a20 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 rx=0x7f1104006f20 tx=0x7f110400b990).stop 2021-02-10T11:33:26.774 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.783 7f11106b3700 1 -- 172.21.2.112:0/3121852737 shutdown_connections 2021-02-10T11:33:26.774 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.783 7f11106b3700 1 --2- 172.21.2.112:0/3121852737 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f10f00186c0 0x7f10f001abc0 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:26.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.783 7f11106b3700 1 --2- 172.21.2.112:0/3121852737 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f11080a55e0 0x7f11081a4040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:26.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.783 7f11106b3700 1 --2- 172.21.2.112:0/3121852737 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f11081aa700 0x7f11081a3a20 unknown :-1 s=CLOSED pgs=1245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:26.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.783 7f11106b3700 1 --2- 172.21.2.112:0/3121852737 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f11081a2a90 0x7f11081aa0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:26.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.783 7f11106b3700 1 -- 172.21.2.112:0/3121852737 >> 172.21.2.112:0/3121852737 conn(0x7f110808fe10 msgr2=0x7f110819c020 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:26.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.783 7f11106b3700 1 -- 172.21.2.112:0/3121852737 shutdown_connections 2021-02-10T11:33:26.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:26.783 7f11106b3700 1 -- 172.21.2.112:0/3121852737 wait complete. 2021-02-10T11:33:26.780 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:33:27.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.221 7fc32043b700 1 Processor -- start 2021-02-10T11:33:27.227 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.222 7fc32043b700 1 -- start start 2021-02-10T11:33:27.227 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.222 7fc32043b700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fc3181b0470 0x7fc3181b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:27.227 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.222 7fc32043b700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fc3180ef610 0x7fc3180efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:27.227 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.222 7fc32043b700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3180f0b50 con 0x7fc3180f0030 2021-02-10T11:33:27.227 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.222 7fc32043b700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3180e52e0 con 0x7fc3181b0470 2021-02-10T11:33:27.227 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.222 7fc32043b700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3180e5490 con 0x7fc3180ef610 2021-02-10T11:33:27.227 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.222 7fc31ea58700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fc3180f0030 0x7fc3180f10d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:49234/0 (socket says 172.21.2.112:49234) 2021-02-10T11:33:27.227 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.222 7fc31ea58700 1 -- 172.21.2.112:0/2452729085 learned_addr learned my addr 172.21.2.112:0/2452729085 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:33:27.227 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.222 7fc31e257700 1 --2- 172.21.2.112:0/2452729085 >> v2:172.21.2.112:3300/0 conn(0x7fc3181b0470 0x7fc3181b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:27.227 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.223 7fc31d255700 1 -- 172.21.2.112:0/2452729085 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 607117361 0 0) 0x7fc310001500 con 0x7fc3180f0030 2021-02-10T11:33:27.227 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.223 7fc31e257700 1 -- 172.21.2.112:0/2452729085 >> v2:172.21.2.120:3300/0 conn(0x7fc3180ef610 msgr2=0x7fc3180efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:27.227 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.223 7fc31e257700 1 --2- 172.21.2.112:0/2452729085 >> v2:172.21.2.120:3300/0 conn(0x7fc3180ef610 0x7fc3180efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:27.228 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.223 7fc31e257700 1 -- 172.21.2.112:0/2452729085 >> v1:172.21.2.112:6789/0 conn(0x7fc3180f0030 legacy=0x7fc3180f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:27.228 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.223 7fc31e257700 1 -- 172.21.2.112:0/2452729085 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3180f0430 con 0x7fc3181b0470 2021-02-10T11:33:27.228 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.223 7fc31e257700 1 --2- 172.21.2.112:0/2452729085 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc3181b0470 0x7fc3181b0870 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 rx=0x7fc30c006650 tx=0x7fc30c009a80).ready entity=mon.0 client_cookie=83866deecf1f24e5 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:27.228 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.223 7fc31d255700 1 -- 172.21.2.112:0/2452729085 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc30c00c0c0 con 0x7fc3181b0470 2021-02-10T11:33:27.228 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.223 7fc31d255700 1 -- 172.21.2.112:0/2452729085 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc30c00c340 con 0x7fc3181b0470 2021-02-10T11:33:27.228 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.223 7fc31d255700 1 -- 172.21.2.112:0/2452729085 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc30c00c790 con 0x7fc3181b0470 2021-02-10T11:33:27.228 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.223 7fc32043b700 1 -- 172.21.2.112:0/2452729085 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc3181b0470 msgr2=0x7fc3181b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:27.228 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.223 7fc32043b700 1 --2- 172.21.2.112:0/2452729085 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc3181b0470 0x7fc3181b0870 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 rx=0x7fc30c006650 tx=0x7fc30c009a80).stop 2021-02-10T11:33:27.228 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.225 7fc32043b700 1 -- 172.21.2.112:0/2452729085 shutdown_connections 2021-02-10T11:33:27.228 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.225 7fc32043b700 1 --2- 172.21.2.112:0/2452729085 >> v2:172.21.2.120:3300/0 conn(0x7fc3180ef610 0x7fc3180efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:27.228 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.225 7fc32043b700 1 --2- 172.21.2.112:0/2452729085 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc3181b0470 0x7fc3181b0870 unknown :-1 s=CLOSED pgs=1247 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:27.228 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.225 7fc32043b700 1 -- 172.21.2.112:0/2452729085 >> 172.21.2.112:0/2452729085 conn(0x7fc3180ce3e0 msgr2=0x7fc3180e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:27.228 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.225 7fc32043b700 1 -- 172.21.2.112:0/2452729085 shutdown_connections 2021-02-10T11:33:27.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.225 7fc32043b700 1 -- 172.21.2.112:0/2452729085 wait complete. 2021-02-10T11:33:27.230 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.239 7fc32043b700 1 Processor -- start 2021-02-10T11:33:27.230 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.239 7fc32043b700 1 -- start start 2021-02-10T11:33:27.230 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.239 7fc32043b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc318190e90 0x7fc31810a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:27.230 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.239 7fc32043b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc318191290 0x7fc318108bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:27.230 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.239 7fc32043b700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc318109210 0x7fc318109610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:27.231 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.239 7fc32043b700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3181916c0 con 0x7fc318190e90 2021-02-10T11:33:27.231 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.239 7fc32043b700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc318109c30 con 0x7fc318109210 2021-02-10T11:33:27.231 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.239 7fc32043b700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3181b67e0 con 0x7fc318191290 2021-02-10T11:33:27.231 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.240 7fc31e257700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc318190e90 0x7fc31810a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:27.231 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.240 7fc31e257700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc318190e90 0x7fc31810a0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:49776/0 (socket says 172.21.2.112:49776) 2021-02-10T11:33:27.231 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.240 7fc31e257700 1 -- 172.21.2.112:0/1439658237 learned_addr learned my addr 172.21.2.112:0/1439658237 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:33:27.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.240 7fc31e257700 1 -- 172.21.2.112:0/1439658237 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc318191290 msgr2=0x7fc318108bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:27.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.240 7fc31e257700 1 --2- 172.21.2.112:0/1439658237 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc318191290 0x7fc318108bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:27.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.240 7fc31e257700 1 -- 172.21.2.112:0/1439658237 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc318109210 msgr2=0x7fc318109610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:27.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.240 7fc31e257700 1 --2- 172.21.2.112:0/1439658237 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc318109210 0x7fc318109610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:27.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.240 7fc31e257700 1 -- 172.21.2.112:0/1439658237 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3181b6a00 con 0x7fc318190e90 2021-02-10T11:33:27.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.240 7fc31e257700 1 --2- 172.21.2.112:0/1439658237 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc318190e90 0x7fc31810a0a0 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 rx=0x7fc30c006f70 tx=0x7fc30c00b9b0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:27.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.240 7fc30affd700 1 -- 172.21.2.112:0/1439658237 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc30c0009b0 con 0x7fc318190e90 2021-02-10T11:33:27.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.240 7fc30affd700 1 -- 172.21.2.112:0/1439658237 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc30c00bda0 con 0x7fc318190e90 2021-02-10T11:33:27.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.240 7fc30affd700 1 -- 172.21.2.112:0/1439658237 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc30c0022b0 con 0x7fc318190e90 2021-02-10T11:33:27.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.240 7fc32043b700 1 -- 172.21.2.112:0/1439658237 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3181b6a00 con 0x7fc318190e90 2021-02-10T11:33:27.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.240 7fc32043b700 1 -- 172.21.2.112:0/1439658237 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3181b6f90 con 0x7fc318190e90 2021-02-10T11:33:27.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.241 7fc30affd700 1 -- 172.21.2.112:0/1439658237 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc30c0052e0 con 0x7fc318190e90 2021-02-10T11:33:27.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.241 7fc30affd700 1 --2- 172.21.2.112:0/1439658237 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc3000186c0 0x7fc30001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:27.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.241 7fc30affd700 1 -- 172.21.2.112:0/1439658237 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fc30c032500 con 0x7fc318190e90 2021-02-10T11:33:27.233 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.242 7fc31da56700 1 --2- 172.21.2.112:0/1439658237 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc3000186c0 0x7fc30001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:27.233 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.243 7fc31da56700 1 --2- 172.21.2.112:0/1439658237 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc3000186c0 0x7fc30001abc0 crc :-1 s=READY pgs=623 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:27.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.272 7fc32043b700 1 -- 172.21.2.112:0/1439658237 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc31810a6c0 con 0x7fc318190e90 2021-02-10T11:33:27.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.275 7fc30affd700 1 -- 172.21.2.112:0/1439658237 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc30c006510 con 0x7fc318190e90 2021-02-10T11:33:27.391 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.403 7fc32043b700 1 -- 172.21.2.112:0/1439658237 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fc31810a6c0 con 0x7fc318190e90 2021-02-10T11:33:27.395 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.407 7fc30affd700 1 -- 172.21.2.112:0/1439658237 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v31) v1 ==== 184+0+0 (secure 0 0 0) 0x7fc30c0052e0 con 0x7fc318190e90 2021-02-10T11:33:27.398 INFO:teuthology.orchestra.run.gibba012.stderr:updated caps for client.0 2021-02-10T11:33:27.401 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.412 7fc32043b700 1 -- 172.21.2.112:0/1439658237 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc3000186c0 msgr2=0x7fc30001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:27.401 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.412 7fc32043b700 1 --2- 172.21.2.112:0/1439658237 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc3000186c0 0x7fc30001abc0 crc :-1 s=READY pgs=623 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:27.402 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.412 7fc32043b700 1 -- 172.21.2.112:0/1439658237 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc318190e90 msgr2=0x7fc31810a0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:27.402 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.412 7fc32043b700 1 --2- 172.21.2.112:0/1439658237 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc318190e90 0x7fc31810a0a0 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 rx=0x7fc30c006f70 tx=0x7fc30c00b9b0).stop 2021-02-10T11:33:27.402 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.412 7fc32043b700 1 -- 172.21.2.112:0/1439658237 shutdown_connections 2021-02-10T11:33:27.402 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.412 7fc32043b700 1 --2- 172.21.2.112:0/1439658237 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc3000186c0 0x7fc30001abc0 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:27.402 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.412 7fc32043b700 1 --2- 172.21.2.112:0/1439658237 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc318109210 0x7fc318109610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:27.402 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.412 7fc32043b700 1 --2- 172.21.2.112:0/1439658237 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc318191290 0x7fc318108bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:27.402 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.412 7fc32043b700 1 --2- 172.21.2.112:0/1439658237 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc318190e90 0x7fc31810a0a0 unknown :-1 s=CLOSED pgs=1248 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:27.402 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.412 7fc32043b700 1 -- 172.21.2.112:0/1439658237 >> 172.21.2.112:0/1439658237 conn(0x7fc31808c1b0 msgr2=0x7fc318090d60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:27.402 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.412 7fc32043b700 1 -- 172.21.2.112:0/1439658237 shutdown_connections 2021-02-10T11:33:27.402 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.413 7fc32043b700 1 -- 172.21.2.112:0/1439658237 wait complete. 2021-02-10T11:33:27.409 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:33:27.848 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.844 7f5482a3f700 1 Processor -- start 2021-02-10T11:33:27.849 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.844 7f5482a3f700 1 -- start start 2021-02-10T11:33:27.849 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.844 7f5482a3f700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f547c1b0470 0x7f547c1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:27.849 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.844 7f5482a3f700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f547c0ef610 0x7f547c0efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:27.849 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.844 7f5482a3f700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f547c0f0030 0x7f547c0f0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:27.849 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.844 7f5482a3f700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f547c0e5ea0 con 0x7f547c1b0470 2021-02-10T11:33:27.849 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.844 7f5482a3f700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f547c0f0b50 con 0x7f547c0ef610 2021-02-10T11:33:27.849 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.844 7f5482a3f700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f547c0e52e0 con 0x7f547c0f0030 2021-02-10T11:33:27.849 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.844 7f548085b700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f547c1b0470 0x7f547c1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:27.849 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.844 7f548085b700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f547c1b0470 0x7f547c1b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:49782/0 (socket says 172.21.2.112:49782) 2021-02-10T11:33:27.850 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.844 7f548085b700 1 -- 172.21.2.112:0/470384186 learned_addr learned my addr 172.21.2.112:0/470384186 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:33:27.850 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.845 7f548085b700 1 -- 172.21.2.112:0/470384186 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f547c0f0030 msgr2=0x7f547c0f0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:27.850 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.845 7f548085b700 1 --2- 172.21.2.112:0/470384186 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f547c0f0030 0x7f547c0f0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:27.850 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.845 7f548085b700 1 -- 172.21.2.112:0/470384186 >> v2:172.21.2.120:3300/0 conn(0x7f547c0ef610 msgr2=0x7f547c0efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:27.850 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.845 7f548085b700 1 --2- 172.21.2.112:0/470384186 >> v2:172.21.2.120:3300/0 conn(0x7f547c0ef610 0x7f547c0efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:27.850 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.845 7f548085b700 1 -- 172.21.2.112:0/470384186 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f547c0f10d0 con 0x7f547c1b0470 2021-02-10T11:33:27.850 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.845 7f548085b700 1 --2- 172.21.2.112:0/470384186 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f547c1b0470 0x7f547c1b0870 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 rx=0x7f5464006700 tx=0x7f5464009b80).ready entity=mon.0 client_cookie=d1b62184d1d33ae server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:27.850 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.845 7f547b7fe700 1 -- 172.21.2.112:0/470384186 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f546400c0c0 con 0x7f547c1b0470 2021-02-10T11:33:27.850 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.845 7f547b7fe700 1 -- 172.21.2.112:0/470384186 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f546400c3c0 con 0x7f547c1b0470 2021-02-10T11:33:27.851 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.845 7f547b7fe700 1 -- 172.21.2.112:0/470384186 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f546400c8e0 con 0x7f547c1b0470 2021-02-10T11:33:27.851 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.845 7f5482a3f700 1 -- 172.21.2.112:0/470384186 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f547c1b0470 msgr2=0x7f547c1b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:27.851 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.845 7f5482a3f700 1 --2- 172.21.2.112:0/470384186 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f547c1b0470 0x7f547c1b0870 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 rx=0x7f5464006700 tx=0x7f5464009b80).stop 2021-02-10T11:33:27.851 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.846 7f5482a3f700 1 -- 172.21.2.112:0/470384186 shutdown_connections 2021-02-10T11:33:27.851 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.846 7f5482a3f700 1 --2- 172.21.2.112:0/470384186 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f547c0f0030 0x7f547c0f0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:27.851 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.846 7f5482a3f700 1 --2- 172.21.2.112:0/470384186 >> v2:172.21.2.120:3300/0 conn(0x7f547c0ef610 0x7f547c0efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:27.851 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.846 7f5482a3f700 1 --2- 172.21.2.112:0/470384186 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f547c1b0470 0x7f547c1b0870 unknown :-1 s=CLOSED pgs=1249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:27.851 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.846 7f5482a3f700 1 -- 172.21.2.112:0/470384186 >> 172.21.2.112:0/470384186 conn(0x7f547c0ce3e0 msgr2=0x7f547c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:27.851 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.846 7f5482a3f700 1 -- 172.21.2.112:0/470384186 shutdown_connections 2021-02-10T11:33:27.851 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.846 7f5482a3f700 1 -- 172.21.2.112:0/470384186 wait complete. 2021-02-10T11:33:27.855 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.861 7f5482a3f700 1 Processor -- start 2021-02-10T11:33:27.855 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.861 7f5482a3f700 1 -- start start 2021-02-10T11:33:27.855 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.861 7f5482a3f700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f547c1ac0b0 0x7f547c0ad920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:27.855 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.861 7f5482a3f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f547c0adf40 0x7f547c1b7040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:27.855 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.861 7f5482a3f700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f547c1b7660 0x7f547c1b7a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:27.855 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.861 7f5482a3f700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f547c1b8080 con 0x7f547c0adf40 2021-02-10T11:33:27.855 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.861 7f5482a3f700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f547c1b8270 con 0x7f547c1b7660 2021-02-10T11:33:27.855 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.861 7f5482a3f700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f547c18d950 con 0x7f547c1ac0b0 2021-02-10T11:33:27.856 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.862 7f547bfff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f547c0adf40 0x7f547c1b7040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:27.856 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.862 7f547bfff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f547c0adf40 0x7f547c1b7040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:49788/0 (socket says 172.21.2.112:49788) 2021-02-10T11:33:27.856 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.862 7f547bfff700 1 -- 172.21.2.112:0/4094752658 learned_addr learned my addr 172.21.2.112:0/4094752658 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:33:27.856 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.862 7f548105c700 1 --2- 172.21.2.112:0/4094752658 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f547c1b7660 0x7f547c1b7a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:27.856 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.862 7f548085b700 1 --2- 172.21.2.112:0/4094752658 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f547c1ac0b0 0x7f547c0ad920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:27.856 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.862 7f547bfff700 1 -- 172.21.2.112:0/4094752658 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f547c1ac0b0 msgr2=0x7f547c0ad920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:27.856 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.862 7f547bfff700 1 --2- 172.21.2.112:0/4094752658 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f547c1ac0b0 0x7f547c0ad920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:27.856 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.862 7f547bfff700 1 -- 172.21.2.112:0/4094752658 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f547c1b7660 msgr2=0x7f547c1b7a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:27.856 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.862 7f547bfff700 1 --2- 172.21.2.112:0/4094752658 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f547c1b7660 0x7f547c1b7a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:27.856 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.862 7f547bfff700 1 -- 172.21.2.112:0/4094752658 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f547c18db70 con 0x7f547c0adf40 2021-02-10T11:33:27.856 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.862 7f547bfff700 1 --2- 172.21.2.112:0/4094752658 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f547c0adf40 0x7f547c1b7040 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 rx=0x7f546c006f00 tx=0x7f546c00b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:27.857 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.862 7f54797fa700 1 -- 172.21.2.112:0/4094752658 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f546c006350 con 0x7f547c0adf40 2021-02-10T11:33:27.857 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.862 7f54797fa700 1 -- 172.21.2.112:0/4094752658 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f546c006b90 con 0x7f547c0adf40 2021-02-10T11:33:27.857 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.862 7f54797fa700 1 -- 172.21.2.112:0/4094752658 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f546c000c90 con 0x7f547c0adf40 2021-02-10T11:33:27.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.862 7f5482a3f700 1 -- 172.21.2.112:0/4094752658 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f547c18db70 con 0x7f547c0adf40 2021-02-10T11:33:27.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.863 7f5482a3f700 1 -- 172.21.2.112:0/4094752658 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f547c18e100 con 0x7f547c0adf40 2021-02-10T11:33:27.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.863 7f54797fa700 1 -- 172.21.2.112:0/4094752658 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f546c007000 con 0x7f547c0adf40 2021-02-10T11:33:27.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.863 7f54797fa700 1 --2- 172.21.2.112:0/4094752658 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f54600186c0 0x7f546001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:27.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.863 7f54797fa700 1 -- 172.21.2.112:0/4094752658 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f546c032580 con 0x7f547c0adf40 2021-02-10T11:33:27.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.864 7f548085b700 1 --2- 172.21.2.112:0/4094752658 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f54600186c0 0x7f546001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:27.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.864 7f548085b700 1 --2- 172.21.2.112:0/4094752658 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f54600186c0 0x7f546001abc0 crc :-1 s=READY pgs=624 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:27.882 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.894 7f5482a3f700 1 -- 172.21.2.112:0/4094752658 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f547c1059e0 con 0x7f547c0adf40 2021-02-10T11:33:27.884 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:27.896 7f54797fa700 1 -- 172.21.2.112:0/4094752658 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f546c0009b0 con 0x7f547c0adf40 2021-02-10T11:33:28.015 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:28.027 7f5482a3f700 1 -- 172.21.2.112:0/4094752658 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f547c0f0b50 con 0x7f547c0adf40 2021-02-10T11:33:28.018 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:28.030 7f54797fa700 1 -- 172.21.2.112:0/4094752658 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v32) v1 ==== 184+0+0 (secure 0 0 0) 0x7f546c007000 con 0x7f547c0adf40 2021-02-10T11:33:28.022 INFO:teuthology.orchestra.run.gibba012.stderr:updated caps for client.1 2021-02-10T11:33:28.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:28.035 7f5482a3f700 1 -- 172.21.2.112:0/4094752658 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f54600186c0 msgr2=0x7f546001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:28.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:28.035 7f5482a3f700 1 --2- 172.21.2.112:0/4094752658 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f54600186c0 0x7f546001abc0 crc :-1 s=READY pgs=624 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:28.024 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:28.035 7f5482a3f700 1 -- 172.21.2.112:0/4094752658 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f547c0adf40 msgr2=0x7f547c1b7040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:28.024 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:28.035 7f5482a3f700 1 --2- 172.21.2.112:0/4094752658 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f547c0adf40 0x7f547c1b7040 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 rx=0x7f546c006f00 tx=0x7f546c00b940).stop 2021-02-10T11:33:28.024 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:28.036 7f5482a3f700 1 -- 172.21.2.112:0/4094752658 shutdown_connections 2021-02-10T11:33:28.024 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:28.036 7f5482a3f700 1 --2- 172.21.2.112:0/4094752658 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f54600186c0 0x7f546001abc0 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:28.024 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:28.036 7f5482a3f700 1 --2- 172.21.2.112:0/4094752658 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f547c1b7660 0x7f547c1b7a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:28.024 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:28.036 7f5482a3f700 1 --2- 172.21.2.112:0/4094752658 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f547c0adf40 0x7f547c1b7040 unknown :-1 s=CLOSED pgs=1250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:28.024 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:28.036 7f5482a3f700 1 --2- 172.21.2.112:0/4094752658 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f547c1ac0b0 0x7f547c0ad920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:28.025 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:28.036 7f5482a3f700 1 -- 172.21.2.112:0/4094752658 >> 172.21.2.112:0/4094752658 conn(0x7f547c1a37a0 msgr2=0x7f547c18c230 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:28.025 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:28.036 7f5482a3f700 1 -- 172.21.2.112:0/4094752658 shutdown_connections 2021-02-10T11:33:28.025 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:28.036 7f5482a3f700 1 -- 172.21.2.112:0/4094752658 wait complete. 2021-02-10T11:33:28.033 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.2 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:33:28.470 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:28.465 7fc0a1497700 1 Processor -- start 2021-02-10T11:33:28.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:28.465 7fc0a1497700 1 -- start start 2021-02-10T11:33:28.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:28.466 7fc0a1497700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc09c1b0870 0x7fc09c0f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:28.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:28.466 7fc0a1497700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc09c1b0c70 con 0x7fc09c0efe00 2021-02-10T11:33:28.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:28.466 7fc0a1497700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc09c0f0b50 con 0x7fc09c1b0470 2021-02-10T11:33:28.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:28.466 7fc0a1497700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc09c0e5ea0 con 0x7fc09c1b0870 2021-02-10T11:33:28.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:28.466 7fc09b87e700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fc09c0efe00 0x7fc09c0f0200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:49262/0 (socket says 172.21.2.112:49262) 2021-02-10T11:33:28.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:28.466 7fc09b87e700 1 -- 172.21.2.112:0/1368124866 learned_addr learned my addr 172.21.2.112:0/1368124866 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:33:28.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:28.466 7fc09a87c700 1 --2- 172.21.2.112:0/1368124866 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc09c1b0870 0x7fc09c0f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:28.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:28.467 7fc09a07b700 1 -- 172.21.2.112:0/1368124866 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1682270755 0 0) 0x7fc094000f90 con 0x7fc09c0efe00 2021-02-10T11:33:28.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:28.467 7fc09a07b700 1 -- 172.21.2.112:0/1368124866 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc088002fe0 con 0x7fc09c0efe00 2021-02-10T11:33:28.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:28.467 7fc09a07b700 1 -- 172.21.2.112:0/1368124866 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 612346025 0 0) 0x7fc08c001080 con 0x7fc09c1b0470 2021-02-10T11:33:28.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:28.467 7fc09a07b700 1 -- 172.21.2.112:0/1368124866 --> v1:172.21.2.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc088004370 con 0x7fc09c1b0470 2021-02-10T11:33:28.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:28.467 7fc09a87c700 1 -- 172.21.2.112:0/1368124866 >> v1:172.21.2.120:6789/0 conn(0x7fc09c1b0470 legacy=0x7fc09c0ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:28.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:28.467 7fc09a87c700 1 -- 172.21.2.112:0/1368124866 >> v1:172.21.2.112:6789/0 conn(0x7fc09c0efe00 legacy=0x7fc09c0f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:28.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:28.467 7fc09a87c700 1 -- 172.21.2.112:0/1368124866 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc09c0d2fb0 con 0x7fc09c1b0870 2021-02-10T11:33:28.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:28.467 7fc09a87c700 1 --2- 172.21.2.112:0/1368124866 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc09c1b0870 0x7fc09c0f20e0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7fc084006650 tx=0x7fc084009a80).ready entity=mon.2 client_cookie=3b56ed08226c9e20 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:28.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:28.467 7fc09a07b700 1 -- 172.21.2.112:0/1368124866 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc0840009b0 con 0x7fc09c1b0870 2021-02-10T11:33:28.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:28.467 7fc09a07b700 1 -- 172.21.2.112:0/1368124866 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc08400c0c0 con 0x7fc09c1b0870 2021-02-10T11:33:28.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:28.467 7fc09a07b700 1 -- 172.21.2.112:0/1368124866 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc08400c5e0 con 0x7fc09c1b0870 2021-02-10T11:33:28.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:28.468 7fc0a1497700 1 -- 172.21.2.112:0/1368124866 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc09c1b0870 msgr2=0x7fc09c0f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:28.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:28.468 7fc0a1497700 1 --2- 172.21.2.112:0/1368124866 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc09c1b0870 0x7fc09c0f20e0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7fc084006650 tx=0x7fc084009a80).stop 2021-02-10T11:33:28.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:28.468 7fc0a1497700 1 -- 172.21.2.112:0/1368124866 shutdown_connections 2021-02-10T11:33:28.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:28.468 7fc0a1497700 1 --2- 172.21.2.112:0/1368124866 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc09c1b0870 0x7fc09c0f20e0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:28.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:28.468 7fc0a1497700 1 -- 172.21.2.112:0/1368124866 >> 172.21.2.112:0/1368124866 conn(0x7fc09c0ce3e0 msgr2=0x7fc09c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:28.473 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:28.468 7fc0a1497700 1 -- 172.21.2.112:0/1368124866 shutdown_connections 2021-02-10T11:33:28.473 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:28.468 7fc0a1497700 1 -- 172.21.2.112:0/1368124866 wait complete. 2021-02-10T11:33:28.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:28.483 7fc0a1497700 1 Processor -- start 2021-02-10T11:33:28.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:28.483 7fc0a1497700 1 -- start start 2021-02-10T11:33:28.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:28.483 7fc0a1497700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc09c18ed50 0x7fc09c0ad800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:28.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:28.483 7fc0a1497700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc09c0ade20 0x7fc09c1b6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:28.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:28.483 7fc0a1497700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc09c19c2d0 0x7fc09c1b73f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:28.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:28.483 7fc0a1497700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc09c18e7d0 con 0x7fc09c0ade20 2021-02-10T11:33:28.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:28.483 7fc0a1497700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc09c1b7a10 con 0x7fc09c18ed50 2021-02-10T11:33:28.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:28.483 7fc0a1497700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc09c1b7bf0 con 0x7fc09c19c2d0 2021-02-10T11:33:28.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:28.483 7fc09a87c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc09c0ade20 0x7fc09c1b6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:28.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:28.483 7fc09a87c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc09c0ade20 0x7fc09c1b6dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:49802/0 (socket says 172.21.2.112:49802) 2021-02-10T11:33:28.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:28.483 7fc09a87c700 1 -- 172.21.2.112:0/3529034454 learned_addr learned my addr 172.21.2.112:0/3529034454 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:33:28.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:28.484 7fc09b87e700 1 --2- 172.21.2.112:0/3529034454 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc09c19c2d0 0x7fc09c1b73f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:28.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:28.484 7fc09b07d700 1 --2- 172.21.2.112:0/3529034454 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc09c18ed50 0x7fc09c0ad800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:28.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:28.484 7fc09a87c700 1 -- 172.21.2.112:0/3529034454 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc09c19c2d0 msgr2=0x7fc09c1b73f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:28.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:28.484 7fc09a87c700 1 --2- 172.21.2.112:0/3529034454 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc09c19c2d0 0x7fc09c1b73f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:28.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:28.484 7fc09a87c700 1 -- 172.21.2.112:0/3529034454 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc09c18ed50 msgr2=0x7fc09c0ad800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:28.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:28.484 7fc09a87c700 1 --2- 172.21.2.112:0/3529034454 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc09c18ed50 0x7fc09c0ad800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:28.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:28.484 7fc09a87c700 1 -- 172.21.2.112:0/3529034454 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc09c18cee0 con 0x7fc09c0ade20 2021-02-10T11:33:28.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:28.484 7fc09a87c700 1 --2- 172.21.2.112:0/3529034454 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc09c0ade20 0x7fc09c1b6dd0 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 rx=0x7fc084009650 tx=0x7fc08400b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:28.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:28.484 7fc093fff700 1 -- 172.21.2.112:0/3529034454 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc0840093a0 con 0x7fc09c0ade20 2021-02-10T11:33:28.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:28.484 7fc093fff700 1 -- 172.21.2.112:0/3529034454 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc08400bd50 con 0x7fc09c0ade20 2021-02-10T11:33:28.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:28.484 7fc0a1497700 1 -- 172.21.2.112:0/3529034454 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc09c18cee0 con 0x7fc09c0ade20 2021-02-10T11:33:28.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:28.484 7fc0a1497700 1 -- 172.21.2.112:0/3529034454 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc09c18cee0 con 0x7fc09c0ade20 2021-02-10T11:33:28.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:28.484 7fc093fff700 1 -- 172.21.2.112:0/3529034454 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc0840052e0 con 0x7fc09c0ade20 2021-02-10T11:33:28.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:28.485 7fc093fff700 1 -- 172.21.2.112:0/3529034454 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc084006030 con 0x7fc09c0ade20 2021-02-10T11:33:28.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:28.485 7fc093fff700 1 --2- 172.21.2.112:0/3529034454 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc080018730 0x7fc08001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:28.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:28.485 7fc093fff700 1 -- 172.21.2.112:0/3529034454 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fc0840327b0 con 0x7fc09c0ade20 2021-02-10T11:33:28.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:28.485 7fc09b07d700 1 --2- 172.21.2.112:0/3529034454 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc080018730 0x7fc08001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:28.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:28.485 7fc09b07d700 1 --2- 172.21.2.112:0/3529034454 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc080018730 0x7fc08001abc0 crc :-1 s=READY pgs=625 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:28.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:28.516 7fc0a1497700 1 -- 172.21.2.112:0/3529034454 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc09c1b7bf0 con 0x7fc09c0ade20 2021-02-10T11:33:28.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:28.518 7fc093fff700 1 -- 172.21.2.112:0/3529034454 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc0840009c0 con 0x7fc09c0ade20 2021-02-10T11:33:28.637 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:28.649 7fc0a1497700 1 -- 172.21.2.112:0/3529034454 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fc09c1b7bf0 con 0x7fc09c0ade20 2021-02-10T11:33:28.640 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:28.652 7fc093fff700 1 -- 172.21.2.112:0/3529034454 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.2 v33) v1 ==== 184+0+0 (secure 0 0 0) 0x7fc084006030 con 0x7fc09c0ade20 2021-02-10T11:33:28.640 INFO:teuthology.orchestra.run.gibba012.stderr:updated caps for client.2 2021-02-10T11:33:28.642 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:28.654 7fc0a1497700 1 -- 172.21.2.112:0/3529034454 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc080018730 msgr2=0x7fc08001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:28.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:28.654 7fc0a1497700 1 --2- 172.21.2.112:0/3529034454 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc080018730 0x7fc08001abc0 crc :-1 s=READY pgs=625 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:28.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:28.654 7fc0a1497700 1 -- 172.21.2.112:0/3529034454 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc09c0ade20 msgr2=0x7fc09c1b6dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:28.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:28.654 7fc0a1497700 1 --2- 172.21.2.112:0/3529034454 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc09c0ade20 0x7fc09c1b6dd0 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 rx=0x7fc084009650 tx=0x7fc08400b960).stop 2021-02-10T11:33:28.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:28.654 7fc0a1497700 1 -- 172.21.2.112:0/3529034454 shutdown_connections 2021-02-10T11:33:28.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:28.654 7fc0a1497700 1 --2- 172.21.2.112:0/3529034454 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc080018730 0x7fc08001abc0 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:28.648 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:28.654 7fc0a1497700 1 --2- 172.21.2.112:0/3529034454 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc09c19c2d0 0x7fc09c1b73f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:28.648 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:28.654 7fc0a1497700 1 --2- 172.21.2.112:0/3529034454 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc09c0ade20 0x7fc09c1b6dd0 unknown :-1 s=CLOSED pgs=1252 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:28.648 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:28.654 7fc0a1497700 1 --2- 172.21.2.112:0/3529034454 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc09c18ed50 0x7fc09c0ad800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:28.648 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:28.654 7fc0a1497700 1 -- 172.21.2.112:0/3529034454 >> 172.21.2.112:0/3529034454 conn(0x7fc09c1a3790 msgr2=0x7fc09c19f670 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:28.648 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:28.654 7fc0a1497700 1 -- 172.21.2.112:0/3529034454 shutdown_connections 2021-02-10T11:33:28.648 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:28.654 7fc0a1497700 1 -- 172.21.2.112:0/3529034454 wait complete. 2021-02-10T11:33:28.651 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.3 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:33:29.088 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.083 7f0e8dfac700 1 Processor -- start 2021-02-10T11:33:29.088 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.083 7f0e8dfac700 1 -- start start 2021-02-10T11:33:29.088 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.084 7f0e8dfac700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f0e881b0470 0x7f0e881b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:29.088 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.084 7f0e8dfac700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f0e880ef610 0x7f0e880efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:29.088 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.084 7f0e8dfac700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e880f0b50 con 0x7f0e880f0030 2021-02-10T11:33:29.088 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.084 7f0e8dfac700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e880e52e0 con 0x7f0e881b0470 2021-02-10T11:33:29.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.084 7f0e8dfac700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e880e5490 con 0x7f0e880ef610 2021-02-10T11:33:29.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.084 7f0e877fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f0e881b0470 0x7f0e881b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:29.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.084 7f0e877fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f0e881b0470 0x7f0e881b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:49810/0 (socket says 172.21.2.112:49810) 2021-02-10T11:33:29.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.084 7f0e877fe700 1 -- 172.21.2.112:0/2362387608 learned_addr learned my addr 172.21.2.112:0/2362387608 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:33:29.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.084 7f0e877fe700 1 -- 172.21.2.112:0/2362387608 >> v2:172.21.2.120:3300/0 conn(0x7f0e880ef610 msgr2=0x7f0e880efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:29.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.084 7f0e877fe700 1 --2- 172.21.2.112:0/2362387608 >> v2:172.21.2.120:3300/0 conn(0x7f0e880ef610 0x7f0e880efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:29.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.084 7f0e877fe700 1 -- 172.21.2.112:0/2362387608 >> v1:172.21.2.120:6789/0 conn(0x7f0e880f0030 legacy=0x7f0e880f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:29.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.084 7f0e877fe700 1 -- 172.21.2.112:0/2362387608 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e880f0430 con 0x7f0e881b0470 2021-02-10T11:33:29.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.084 7f0e877fe700 1 --2- 172.21.2.112:0/2362387608 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0e881b0470 0x7f0e881b0870 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 rx=0x7f0e70006700 tx=0x7f0e70009b80).ready entity=mon.0 client_cookie=52a3998f9ae02d58 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:29.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.085 7f0e867fc700 1 -- 172.21.2.112:0/2362387608 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0e7000c0c0 con 0x7f0e881b0470 2021-02-10T11:33:29.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.085 7f0e867fc700 1 -- 172.21.2.112:0/2362387608 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e7000c440 con 0x7f0e881b0470 2021-02-10T11:33:29.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.085 7f0e867fc700 1 -- 172.21.2.112:0/2362387608 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0e7000c960 con 0x7f0e881b0470 2021-02-10T11:33:29.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.085 7f0e8dfac700 1 -- 172.21.2.112:0/2362387608 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0e881b0470 msgr2=0x7f0e881b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:29.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.085 7f0e8dfac700 1 --2- 172.21.2.112:0/2362387608 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0e881b0470 0x7f0e881b0870 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 rx=0x7f0e70006700 tx=0x7f0e70009b80).stop 2021-02-10T11:33:29.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.085 7f0e8dfac700 1 -- 172.21.2.112:0/2362387608 shutdown_connections 2021-02-10T11:33:29.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.085 7f0e8dfac700 1 --2- 172.21.2.112:0/2362387608 >> v2:172.21.2.120:3300/0 conn(0x7f0e880ef610 0x7f0e880efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:29.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.085 7f0e8dfac700 1 --2- 172.21.2.112:0/2362387608 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0e881b0470 0x7f0e881b0870 unknown :-1 s=CLOSED pgs=1253 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:29.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.085 7f0e8dfac700 1 -- 172.21.2.112:0/2362387608 >> 172.21.2.112:0/2362387608 conn(0x7f0e880ce3e0 msgr2=0x7f0e880e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:29.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.086 7f0e8dfac700 1 -- 172.21.2.112:0/2362387608 shutdown_connections 2021-02-10T11:33:29.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.086 7f0e8dfac700 1 -- 172.21.2.112:0/2362387608 wait complete. 2021-02-10T11:33:29.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.100 7f0e8dfac700 1 Processor -- start 2021-02-10T11:33:29.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.100 7f0e8dfac700 1 -- start start 2021-02-10T11:33:29.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.101 7f0e8dfac700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0e8809ca50 0x7f0e880b47b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:29.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.101 7f0e8dfac700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0e880b4dd0 0x7f0e8818e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:29.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.101 7f0e8dfac700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0e8809ce50 0x7f0e8818eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:29.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.101 7f0e8dfac700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e88189980 con 0x7f0e880b4dd0 2021-02-10T11:33:29.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.101 7f0e8dfac700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e881a2f80 con 0x7f0e8809ca50 2021-02-10T11:33:29.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.101 7f0e8dfac700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e88189f00 con 0x7f0e8809ce50 2021-02-10T11:33:29.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.101 7f0e86ffd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0e880b4dd0 0x7f0e8818e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:29.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.101 7f0e86ffd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0e880b4dd0 0x7f0e8818e4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:49812/0 (socket says 172.21.2.112:49812) 2021-02-10T11:33:29.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.101 7f0e86ffd700 1 -- 172.21.2.112:0/913921570 learned_addr learned my addr 172.21.2.112:0/913921570 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:33:29.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.101 7f0e86ffd700 1 -- 172.21.2.112:0/913921570 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0e8809ce50 msgr2=0x7f0e8818eae0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:33:29.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.101 7f0e86ffd700 1 --2- 172.21.2.112:0/913921570 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0e8809ce50 0x7f0e8818eae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:29.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.101 7f0e86ffd700 1 -- 172.21.2.112:0/913921570 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0e8809ca50 msgr2=0x7f0e880b47b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:33:29.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.101 7f0e86ffd700 1 --2- 172.21.2.112:0/913921570 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0e8809ca50 0x7f0e880b47b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:29.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.101 7f0e86ffd700 1 -- 172.21.2.112:0/913921570 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e8818a120 con 0x7f0e880b4dd0 2021-02-10T11:33:29.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.101 7f0e86ffd700 1 --2- 172.21.2.112:0/913921570 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0e880b4dd0 0x7f0e8818e4c0 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 rx=0x7f0e78006f00 tx=0x7f0e7800b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:29.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.101 7f0e7ffff700 1 -- 172.21.2.112:0/913921570 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0e7800bd30 con 0x7f0e880b4dd0 2021-02-10T11:33:29.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.101 7f0e7ffff700 1 -- 172.21.2.112:0/913921570 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e78000c90 con 0x7f0e880b4dd0 2021-02-10T11:33:29.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.101 7f0e8dfac700 1 -- 172.21.2.112:0/913921570 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e8818a120 con 0x7f0e880b4dd0 2021-02-10T11:33:29.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.101 7f0e8dfac700 1 -- 172.21.2.112:0/913921570 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e8818a6b0 con 0x7f0e880b4dd0 2021-02-10T11:33:29.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.101 7f0e7ffff700 1 -- 172.21.2.112:0/913921570 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0e780022a0 con 0x7f0e880b4dd0 2021-02-10T11:33:29.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.102 7f0e7ffff700 1 -- 172.21.2.112:0/913921570 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0e78005310 con 0x7f0e880b4dd0 2021-02-10T11:33:29.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.102 7f0e7ffff700 1 --2- 172.21.2.112:0/913921570 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0e6c0186c0 0x7f0e6c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:29.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.102 7f0e7ffff700 1 -- 172.21.2.112:0/913921570 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f0e78032500 con 0x7f0e880b4dd0 2021-02-10T11:33:29.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.103 7f0e877fe700 1 --2- 172.21.2.112:0/913921570 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0e6c0186c0 0x7f0e6c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:29.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.105 7f0e877fe700 1 --2- 172.21.2.112:0/913921570 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0e6c0186c0 0x7f0e6c01abc0 crc :-1 s=READY pgs=626 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:29.121 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.134 7f0e8dfac700 1 -- 172.21.2.112:0/913921570 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e880edd30 con 0x7f0e880b4dd0 2021-02-10T11:33:29.124 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.136 7f0e7ffff700 1 -- 172.21.2.112:0/913921570 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0e78000940 con 0x7f0e880b4dd0 2021-02-10T11:33:29.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.268 7f0e8dfac700 1 -- 172.21.2.112:0/913921570 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f0e880e52e0 con 0x7f0e880b4dd0 2021-02-10T11:33:29.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.271 7f0e7ffff700 1 -- 172.21.2.112:0/913921570 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.3 v34) v1 ==== 184+0+0 (secure 0 0 0) 0x7f0e78005310 con 0x7f0e880b4dd0 2021-02-10T11:33:29.263 INFO:teuthology.orchestra.run.gibba012.stderr:updated caps for client.3 2021-02-10T11:33:29.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.276 7f0e8dfac700 1 -- 172.21.2.112:0/913921570 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0e6c0186c0 msgr2=0x7f0e6c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:29.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.276 7f0e8dfac700 1 --2- 172.21.2.112:0/913921570 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0e6c0186c0 0x7f0e6c01abc0 crc :-1 s=READY pgs=626 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:29.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.276 7f0e8dfac700 1 -- 172.21.2.112:0/913921570 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0e880b4dd0 msgr2=0x7f0e8818e4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:29.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.276 7f0e8dfac700 1 --2- 172.21.2.112:0/913921570 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0e880b4dd0 0x7f0e8818e4c0 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 rx=0x7f0e78006f00 tx=0x7f0e7800b940).stop 2021-02-10T11:33:29.265 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.277 7f0e8dfac700 1 -- 172.21.2.112:0/913921570 shutdown_connections 2021-02-10T11:33:29.265 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.277 7f0e8dfac700 1 --2- 172.21.2.112:0/913921570 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0e6c0186c0 0x7f0e6c01abc0 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:29.265 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.277 7f0e8dfac700 1 --2- 172.21.2.112:0/913921570 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0e8809ce50 0x7f0e8818eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:29.265 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.277 7f0e8dfac700 1 --2- 172.21.2.112:0/913921570 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0e880b4dd0 0x7f0e8818e4c0 unknown :-1 s=CLOSED pgs=1254 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:29.265 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.277 7f0e8dfac700 1 --2- 172.21.2.112:0/913921570 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0e8809ca50 0x7f0e880b47b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:29.265 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.277 7f0e8dfac700 1 -- 172.21.2.112:0/913921570 >> 172.21.2.112:0/913921570 conn(0x7f0e881a90c0 msgr2=0x7f0e881b38b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:29.265 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.277 7f0e8dfac700 1 -- 172.21.2.112:0/913921570 shutdown_connections 2021-02-10T11:33:29.265 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.277 7f0e8dfac700 1 -- 172.21.2.112:0/913921570 wait complete. 2021-02-10T11:33:29.275 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:33:29.704 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.700 7f262c92f700 1 Processor -- start 2021-02-10T11:33:29.704 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.700 7f262c92f700 1 -- start start 2021-02-10T11:33:29.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.700 7f262c92f700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f26280e5470 0x7f262807e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:29.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.700 7f262c92f700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f262807f300 0x7f262807f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:29.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.700 7f262c92f700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f262807ed80 con 0x7f26280e5070 2021-02-10T11:33:29.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.700 7f262c92f700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26281b4010 con 0x7f262807f300 2021-02-10T11:33:29.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.700 7f262c92f700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26280d1790 con 0x7f26280e5470 2021-02-10T11:33:29.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.701 7f2626e1d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f262807f300 0x7f262807f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:29.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.701 7f2626e1d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f262807f300 0x7f262807f700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:49824/0 (socket says 172.21.2.112:49824) 2021-02-10T11:33:29.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.701 7f2626e1d700 1 -- 172.21.2.112:0/4036369589 learned_addr learned my addr 172.21.2.112:0/4036369589 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:33:29.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.701 7f2626e1d700 1 -- 172.21.2.112:0/4036369589 >> v2:172.21.2.120:3300/0 conn(0x7f26280e5470 msgr2=0x7f262807e6a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:33:29.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.701 7f2626e1d700 1 --2- 172.21.2.112:0/4036369589 >> v2:172.21.2.120:3300/0 conn(0x7f26280e5470 0x7f262807e6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:29.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.701 7f2626e1d700 1 -- 172.21.2.112:0/4036369589 >> v1:172.21.2.120:6789/0 conn(0x7f26280e5070 legacy=0x7f26281b3820 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:33:29.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.701 7f2626e1d700 1 -- 172.21.2.112:0/4036369589 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26280d19b0 con 0x7f262807f300 2021-02-10T11:33:29.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.701 7f2626e1d700 1 --2- 172.21.2.112:0/4036369589 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f262807f300 0x7f262807f700 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 rx=0x7f2620006f00 tx=0x7f2620009f10).ready entity=mon.0 client_cookie=946204680c98e1cf server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:29.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.701 7f262561a700 1 -- 172.21.2.112:0/4036369589 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f262000b3b0 con 0x7f262807f300 2021-02-10T11:33:29.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.701 7f262561a700 1 -- 172.21.2.112:0/4036369589 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f262000b630 con 0x7f262807f300 2021-02-10T11:33:29.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.701 7f262561a700 1 -- 172.21.2.112:0/4036369589 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f262000bb50 con 0x7f262807f300 2021-02-10T11:33:29.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.701 7f262c92f700 1 -- 172.21.2.112:0/4036369589 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f262807f300 msgr2=0x7f262807f700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:29.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.701 7f262c92f700 1 --2- 172.21.2.112:0/4036369589 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f262807f300 0x7f262807f700 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 rx=0x7f2620006f00 tx=0x7f2620009f10).stop 2021-02-10T11:33:29.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.702 7f262c92f700 1 -- 172.21.2.112:0/4036369589 shutdown_connections 2021-02-10T11:33:29.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.702 7f262c92f700 1 --2- 172.21.2.112:0/4036369589 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f262807f300 0x7f262807f700 unknown :-1 s=CLOSED pgs=1255 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:29.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.702 7f262c92f700 1 --2- 172.21.2.112:0/4036369589 >> v2:172.21.2.120:3300/0 conn(0x7f26280e5470 0x7f262807e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:29.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.702 7f262c92f700 1 -- 172.21.2.112:0/4036369589 >> 172.21.2.112:0/4036369589 conn(0x7f26280ce690 msgr2=0x7f26280e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:29.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.702 7f262c92f700 1 -- 172.21.2.112:0/4036369589 shutdown_connections 2021-02-10T11:33:29.707 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.702 7f262c92f700 1 -- 172.21.2.112:0/4036369589 wait complete. 2021-02-10T11:33:29.708 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.717 7f262c92f700 1 Processor -- start 2021-02-10T11:33:29.708 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.717 7f262c92f700 1 -- start start 2021-02-10T11:33:29.708 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.717 7f262c92f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f26281a2a90 0x7f26281aa0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:29.708 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.717 7f262c92f700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f26281aa700 0x7f26281a3a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:29.708 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.717 7f262c92f700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f26280a55e0 0x7f26281a4040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:29.708 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.717 7f262c92f700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26281a34a0 con 0x7f26281a2a90 2021-02-10T11:33:29.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.717 7f262c92f700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26281a2e90 con 0x7f26281aa700 2021-02-10T11:33:29.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.717 7f262c92f700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26281a30b0 con 0x7f26280a55e0 2021-02-10T11:33:29.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.717 7f262661c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f26281a2a90 0x7f26281aa0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:29.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.717 7f262661c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f26281a2a90 0x7f26281aa0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:49830/0 (socket says 172.21.2.112:49830) 2021-02-10T11:33:29.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.717 7f262661c700 1 -- 172.21.2.112:0/1561897030 learned_addr learned my addr 172.21.2.112:0/1561897030 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:33:29.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.718 7f262661c700 1 -- 172.21.2.112:0/1561897030 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f26280a55e0 msgr2=0x7f26281a4040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:29.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.718 7f262661c700 1 --2- 172.21.2.112:0/1561897030 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f26280a55e0 0x7f26281a4040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:29.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.718 7f262661c700 1 -- 172.21.2.112:0/1561897030 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f26281aa700 msgr2=0x7f26281a3a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:29.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.718 7f262661c700 1 --2- 172.21.2.112:0/1561897030 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f26281aa700 0x7f26281a3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:29.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.718 7f262661c700 1 -- 172.21.2.112:0/1561897030 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26281a4660 con 0x7f26281a2a90 2021-02-10T11:33:29.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.718 7f262661c700 1 --2- 172.21.2.112:0/1561897030 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f26281a2a90 0x7f26281aa0e0 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 rx=0x7f261800b5b0 tx=0x7f261800b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:29.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.718 7f261effd700 1 -- 172.21.2.112:0/1561897030 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f261800bd40 con 0x7f26281a2a90 2021-02-10T11:33:29.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.718 7f261effd700 1 -- 172.21.2.112:0/1561897030 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2618006020 con 0x7f26281a2a90 2021-02-10T11:33:29.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.718 7f261effd700 1 -- 172.21.2.112:0/1561897030 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f26180054a0 con 0x7f26281a2a90 2021-02-10T11:33:29.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.718 7f262c92f700 1 -- 172.21.2.112:0/1561897030 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26281a4660 con 0x7f26281a2a90 2021-02-10T11:33:29.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.718 7f262c92f700 1 -- 172.21.2.112:0/1561897030 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26281a4bf0 con 0x7f26281a2a90 2021-02-10T11:33:29.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.719 7f261effd700 1 -- 172.21.2.112:0/1561897030 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2618006200 con 0x7f26281a2a90 2021-02-10T11:33:29.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.719 7f261effd700 1 --2- 172.21.2.112:0/1561897030 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f260c0186c0 0x7f260c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:29.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.719 7f261effd700 1 -- 172.21.2.112:0/1561897030 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f2618032590 con 0x7f26281a2a90 2021-02-10T11:33:29.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.719 7f2625e1b700 1 --2- 172.21.2.112:0/1561897030 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f260c0186c0 0x7f260c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:29.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.721 7f2625e1b700 1 --2- 172.21.2.112:0/1561897030 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f260c0186c0 0x7f260c01abc0 crc :-1 s=READY pgs=627 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:29.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.750 7f262c92f700 1 -- 172.21.2.112:0/1561897030 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2628083da0 con 0x7f26281a2a90 2021-02-10T11:33:29.741 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.753 7f261effd700 1 -- 172.21.2.112:0/1561897030 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f26180009b0 con 0x7f26281a2a90 2021-02-10T11:33:29.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.882 7f262c92f700 1 -- 172.21.2.112:0/1561897030 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f262807ed80 con 0x7f26281a2a90 2021-02-10T11:33:29.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.883 7f261effd700 1 -- 172.21.2.112:0/1561897030 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) v1 ==== 94+0+2101 (secure 0 0 0) 0x7f261800bd40 con 0x7f26281a2a90 2021-02-10T11:33:29.871 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 39 2021-02-10T11:33:29.872 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:33:29.872 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":39,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7881,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/1476007789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":1476007789},{"type":"v1","addr":"172.21.2.112:6833","nonce":1476007789}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":35}],"filesystems":[{"mdsmap":{"epoch":39,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:33:23.550791","modified":"2021-02-10 11:33:25.562811","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6595":{"gid":6595,"name":"b","rank":0,"incarnation":36,"state":"up:active","state_seq":3,"addr":"172.21.2.120:6835/4241229406","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":4241229406},{"type":"v1","addr":"172.21.2.120:6835","nonce":4241229406}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2021-02-10T11:33:29.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.885 7f262c92f700 1 -- 172.21.2.112:0/1561897030 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f260c0186c0 msgr2=0x7f260c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:29.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.885 7f262c92f700 1 --2- 172.21.2.112:0/1561897030 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f260c0186c0 0x7f260c01abc0 crc :-1 s=READY pgs=627 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:29.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.885 7f262c92f700 1 -- 172.21.2.112:0/1561897030 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f26281a2a90 msgr2=0x7f26281aa0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:29.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.885 7f262c92f700 1 --2- 172.21.2.112:0/1561897030 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f26281a2a90 0x7f26281aa0e0 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 rx=0x7f261800b5b0 tx=0x7f261800b950).stop 2021-02-10T11:33:29.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.885 7f262c92f700 1 -- 172.21.2.112:0/1561897030 shutdown_connections 2021-02-10T11:33:29.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.885 7f262c92f700 1 --2- 172.21.2.112:0/1561897030 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f260c0186c0 0x7f260c01abc0 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:29.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.885 7f262c92f700 1 --2- 172.21.2.112:0/1561897030 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f26281aa700 0x7f26281a3a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:29.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.885 7f262c92f700 1 --2- 172.21.2.112:0/1561897030 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f26280a55e0 0x7f26281a4040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:29.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.885 7f262c92f700 1 --2- 172.21.2.112:0/1561897030 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f26281a2a90 0x7f26281aa0e0 unknown :-1 s=CLOSED pgs=1256 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:29.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.885 7f262c92f700 1 -- 172.21.2.112:0/1561897030 >> 172.21.2.112:0/1561897030 conn(0x7f262808fe10 msgr2=0x7f262819c020 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:29.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.885 7f262c92f700 1 -- 172.21.2.112:0/1561897030 shutdown_connections 2021-02-10T11:33:29.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:29.885 7f262c92f700 1 -- 172.21.2.112:0/1561897030 wait complete. 2021-02-10T11:33:30.083 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 39, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:33:23.550791', 'modified': '2021-02-10 11:33:25.562811', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6595}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6595': {'gid': 6595, 'name': 'b', 'rank': 0, 'incarnation': 36, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.120:6835/4241229406', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.120:6834', 'nonce': 4241229406}, {'type': 'v1', 'addr': '172.21.2.120:6835', 'nonce': 4241229406}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [11], 'metadata_pool': 10, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:33:30.083 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:33:30.084 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:33:30.481 INFO:teuthology.orchestra.run.gibba020.stdout:{ 2021-02-10T11:33:30.482 INFO:teuthology.orchestra.run.gibba020.stdout: "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", 2021-02-10T11:33:30.482 INFO:teuthology.orchestra.run.gibba020.stdout: "whoami": 0, 2021-02-10T11:33:30.482 INFO:teuthology.orchestra.run.gibba020.stdout: "id": 6595, 2021-02-10T11:33:30.482 INFO:teuthology.orchestra.run.gibba020.stdout: "want_state": "up:active", 2021-02-10T11:33:30.482 INFO:teuthology.orchestra.run.gibba020.stdout: "state": "up:active", 2021-02-10T11:33:30.482 INFO:teuthology.orchestra.run.gibba020.stdout: "rank_uptime": 6.9418402600000002, 2021-02-10T11:33:30.482 INFO:teuthology.orchestra.run.gibba020.stdout: "mdsmap_epoch": 39, 2021-02-10T11:33:30.482 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch": 55, 2021-02-10T11:33:30.482 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch_barrier": 55, 2021-02-10T11:33:30.482 INFO:teuthology.orchestra.run.gibba020.stdout: "uptime": 12.571592872 2021-02-10T11:33:30.482 INFO:teuthology.orchestra.run.gibba020.stdout:} 2021-02-10T11:33:30.486 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", "whoami": 0, "id": 6595, "want_state": "up:active", "state": "up:active", "rank_uptime": 6.9418402600000002, "mdsmap_epoch": 39, "osdmap_epoch": 55, "osdmap_epoch_barrier": 55, "uptime": 12.571592872 } 2021-02-10T11:33:30.486 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:33:30.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:30.907 7f1872fce700 1 Processor -- start 2021-02-10T11:33:30.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:30.907 7f1872fce700 1 -- start start 2021-02-10T11:33:30.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:30.908 7f1872fce700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f186c0e5470 0x7f186c07e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:30.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:30.908 7f1872fce700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f186c07f300 0x7f186c07f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:30.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:30.908 7f1872fce700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f186c07ed80 con 0x7f186c0e5070 2021-02-10T11:33:30.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:30.908 7f1872fce700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f186c1b4010 con 0x7f186c0e5470 2021-02-10T11:33:30.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:30.908 7f1872fce700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f186c0d1790 con 0x7f186c07f300 2021-02-10T11:33:30.913 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:30.909 7f186bfff700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f186c0e5470 0x7f186c07e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:30.913 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:30.909 7f1870dea700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7f186c0e5070 0x7f186c1b3820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:42856/0 (socket says 172.21.2.112:42856) 2021-02-10T11:33:30.913 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:30.909 7f1870dea700 1 -- 172.21.2.112:0/2942933727 learned_addr learned my addr 172.21.2.112:0/2942933727 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:33:30.913 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:30.909 7f18715eb700 1 --2- 172.21.2.112:0/2942933727 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f186c07f300 0x7f186c07f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:30.913 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:30.909 7f18715eb700 1 -- 172.21.2.112:0/2942933727 >> v2:172.21.2.120:3300/0 conn(0x7f186c0e5470 msgr2=0x7f186c07e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:30.913 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:30.909 7f18715eb700 1 --2- 172.21.2.112:0/2942933727 >> v2:172.21.2.120:3300/0 conn(0x7f186c0e5470 0x7f186c07e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:30.913 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:30.909 7f18715eb700 1 -- 172.21.2.112:0/2942933727 >> v1:172.21.2.120:6789/0 conn(0x7f186c0e5070 legacy=0x7f186c1b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:30.913 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:30.909 7f18715eb700 1 -- 172.21.2.112:0/2942933727 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f186c0d19b0 con 0x7f186c07f300 2021-02-10T11:33:30.913 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:30.909 7f186bfff700 1 --2- 172.21.2.112:0/2942933727 >> v2:172.21.2.120:3300/0 conn(0x7f186c0e5470 0x7f186c07e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:33:30.913 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:30.909 7f18715eb700 1 --2- 172.21.2.112:0/2942933727 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f186c07f300 0x7f186c07f700 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f1864006e50 tx=0x7f1864009e70).ready entity=mon.2 client_cookie=bd01f9a1a552dde5 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:30.913 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:30.910 7f186b7fe700 1 -- 172.21.2.112:0/2942933727 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1864000f90 con 0x7f186c07f300 2021-02-10T11:33:30.914 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:30.910 7f186b7fe700 1 -- 172.21.2.112:0/2942933727 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18640066f0 con 0x7f186c07f300 2021-02-10T11:33:30.914 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:30.910 7f186b7fe700 1 -- 172.21.2.112:0/2942933727 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f186400b700 con 0x7f186c07f300 2021-02-10T11:33:30.914 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:30.910 7f1872fce700 1 -- 172.21.2.112:0/2942933727 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f186c07f300 msgr2=0x7f186c07f700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:30.914 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:30.910 7f1872fce700 1 --2- 172.21.2.112:0/2942933727 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f186c07f300 0x7f186c07f700 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f1864006e50 tx=0x7f1864009e70).stop 2021-02-10T11:33:30.914 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:30.910 7f1872fce700 1 -- 172.21.2.112:0/2942933727 shutdown_connections 2021-02-10T11:33:30.914 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:30.910 7f1872fce700 1 --2- 172.21.2.112:0/2942933727 >> v2:172.21.2.120:3300/0 conn(0x7f186c0e5470 0x7f186c07e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:30.914 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:30.910 7f1872fce700 1 --2- 172.21.2.112:0/2942933727 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f186c07f300 0x7f186c07f700 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:30.914 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:30.910 7f1872fce700 1 -- 172.21.2.112:0/2942933727 >> 172.21.2.112:0/2942933727 conn(0x7f186c0ce690 msgr2=0x7f186c0e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:30.914 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:30.910 7f1872fce700 1 -- 172.21.2.112:0/2942933727 shutdown_connections 2021-02-10T11:33:30.914 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:30.910 7f1872fce700 1 -- 172.21.2.112:0/2942933727 wait complete. 2021-02-10T11:33:30.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:30.925 7f1872fce700 1 Processor -- start 2021-02-10T11:33:30.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:30.925 7f1872fce700 1 -- start start 2021-02-10T11:33:30.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:30.925 7f1872fce700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f186c08d400 0x7f186c081230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:30.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:30.925 7f1872fce700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f186c081850 0x7f186c0acf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:30.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:30.925 7f1872fce700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f186c081c50 0x7f186c0ad5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:30.917 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:30.925 7f1872fce700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f186c082110 con 0x7f186c081850 2021-02-10T11:33:30.917 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:30.925 7f1872fce700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f186c08d800 con 0x7f186c081c50 2021-02-10T11:33:30.917 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:30.925 7f1872fce700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f186c0adbc0 con 0x7f186c08d400 2021-02-10T11:33:30.917 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:30.926 7f186bfff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f186c081850 0x7f186c0acf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:30.917 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:30.926 7f186bfff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f186c081850 0x7f186c0acf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:49842/0 (socket says 172.21.2.112:49842) 2021-02-10T11:33:30.918 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:30.926 7f186bfff700 1 -- 172.21.2.112:0/3076080789 learned_addr learned my addr 172.21.2.112:0/3076080789 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:33:30.918 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:30.926 7f186bfff700 1 -- 172.21.2.112:0/3076080789 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f186c08d400 msgr2=0x7f186c081230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:30.918 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:30.926 7f186bfff700 1 --2- 172.21.2.112:0/3076080789 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f186c08d400 0x7f186c081230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:30.919 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:30.926 7f186bfff700 1 -- 172.21.2.112:0/3076080789 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f186c081c50 msgr2=0x7f186c0ad5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:30.919 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:30.926 7f186bfff700 1 --2- 172.21.2.112:0/3076080789 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f186c081c50 0x7f186c0ad5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:30.919 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:30.926 7f186bfff700 1 -- 172.21.2.112:0/3076080789 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f186c187fc0 con 0x7f186c081850 2021-02-10T11:33:30.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:30.926 7f186bfff700 1 --2- 172.21.2.112:0/3076080789 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f186c081850 0x7f186c0acf80 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 rx=0x7f185c006f40 tx=0x7f185c00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:30.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:30.926 7f18697fa700 1 -- 172.21.2.112:0/3076080789 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f185c006bd0 con 0x7f186c081850 2021-02-10T11:33:30.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:30.926 7f18697fa700 1 -- 172.21.2.112:0/3076080789 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f185c00bd80 con 0x7f186c081850 2021-02-10T11:33:30.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:30.926 7f18697fa700 1 -- 172.21.2.112:0/3076080789 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f185c0022b0 con 0x7f186c081850 2021-02-10T11:33:30.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:30.926 7f1872fce700 1 -- 172.21.2.112:0/3076080789 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f186c187fc0 con 0x7f186c081850 2021-02-10T11:33:30.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:30.926 7f1872fce700 1 -- 172.21.2.112:0/3076080789 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f186c187fc0 con 0x7f186c081850 2021-02-10T11:33:30.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:30.927 7f18697fa700 1 -- 172.21.2.112:0/3076080789 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f185c00a000 con 0x7f186c081850 2021-02-10T11:33:30.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:30.927 7f18697fa700 1 --2- 172.21.2.112:0/3076080789 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1850018730 0x7f185001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:30.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:30.927 7f18697fa700 1 -- 172.21.2.112:0/3076080789 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f185c032660 con 0x7f186c081850 2021-02-10T11:33:30.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:30.927 7f1870dea700 1 --2- 172.21.2.112:0/3076080789 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1850018730 0x7f185001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:30.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:30.928 7f1870dea700 1 --2- 172.21.2.112:0/3076080789 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1850018730 0x7f185001ac10 crc :-1 s=READY pgs=628 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:30.944 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:30.957 7f1872fce700 1 -- 172.21.2.112:0/3076080789 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f186c08d800 con 0x7f186c081850 2021-02-10T11:33:30.947 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:30.960 7f18697fa700 1 -- 172.21.2.112:0/3076080789 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f185c000c90 con 0x7f186c081850 2021-02-10T11:33:31.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:31.085 7f1872fce700 1 -- 172.21.2.112:0/3076080789 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f186c08d800 con 0x7f186c081850 2021-02-10T11:33:31.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:31.086 7f18697fa700 1 -- 172.21.2.112:0/3076080789 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v55) v1 ==== 74+0+13996 (secure 0 0 0) 0x7f185c000ee0 con 0x7f186c081850 2021-02-10T11:33:31.073 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:33:31.073 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":55,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:33:24.553170","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:33:21.523665","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":11,"pool_name":"cephfs_data","create_time":"2021-02-10 11:33:22.595892","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:33:31.074 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:31.087 7f1872fce700 1 -- 172.21.2.112:0/3076080789 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1850018730 msgr2=0x7f185001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:31.074 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:31.087 7f1872fce700 1 --2- 172.21.2.112:0/3076080789 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1850018730 0x7f185001ac10 crc :-1 s=READY pgs=628 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:31.074 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:31.087 7f1872fce700 1 -- 172.21.2.112:0/3076080789 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f186c081850 msgr2=0x7f186c0acf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:31.074 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:31.087 7f1872fce700 1 --2- 172.21.2.112:0/3076080789 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f186c081850 0x7f186c0acf80 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 rx=0x7f185c006f40 tx=0x7f185c00b990).stop 2021-02-10T11:33:31.075 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:31.087 7f1872fce700 1 -- 172.21.2.112:0/3076080789 shutdown_connections 2021-02-10T11:33:31.075 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:31.087 7f1872fce700 1 --2- 172.21.2.112:0/3076080789 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1850018730 0x7f185001ac10 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:31.075 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:31.087 7f1872fce700 1 --2- 172.21.2.112:0/3076080789 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f186c081c50 0x7f186c0ad5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:31.075 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:31.087 7f1872fce700 1 --2- 172.21.2.112:0/3076080789 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f186c081850 0x7f186c0acf80 unknown :-1 s=CLOSED pgs=1257 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:31.075 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:31.087 7f1872fce700 1 --2- 172.21.2.112:0/3076080789 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f186c08d400 0x7f186c081230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:31.075 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:31.087 7f1872fce700 1 -- 172.21.2.112:0/3076080789 >> 172.21.2.112:0/3076080789 conn(0x7f186c1ac620 msgr2=0x7f186c0896e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:31.075 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:31.088 7f1872fce700 1 -- 172.21.2.112:0/3076080789 shutdown_connections 2021-02-10T11:33:31.075 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:31.088 7f1872fce700 1 -- 172.21.2.112:0/3076080789 wait complete. 2021-02-10T11:33:31.083 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:13:57.508758', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 10, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:33:21.523665', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '54', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 11, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:33:22.595892', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '55', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:33:31.083 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:33:31.515 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:31.509 7f1208871700 1 Processor -- start 2021-02-10T11:33:31.515 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:31.509 7f1208871700 1 -- start start 2021-02-10T11:33:31.515 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:31.510 7f1208871700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f12041b0870 0x7f12040f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:31.515 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:31.510 7f1208871700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12041b0c70 con 0x7f12040efe00 2021-02-10T11:33:31.515 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:31.510 7f1208871700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12040f0b50 con 0x7f12041b0470 2021-02-10T11:33:31.515 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:31.510 7f1208871700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12040e5ea0 con 0x7f12041b0870 2021-02-10T11:33:31.515 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:31.510 7f1202e1d700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f12040efe00 0x7f12040f0200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:49314/0 (socket says 172.21.2.112:49314) 2021-02-10T11:33:31.515 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:31.510 7f1202e1d700 1 -- 172.21.2.112:0/1629192383 learned_addr learned my addr 172.21.2.112:0/1629192383 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:33:31.515 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:31.510 7f120161a700 1 -- 172.21.2.112:0/1629192383 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2973810330 0 0) 0x7f11fc001500 con 0x7f12040efe00 2021-02-10T11:33:31.515 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:31.510 7f1201e1b700 1 --2- 172.21.2.112:0/1629192383 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f12041b0870 0x7f12040f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:31.516 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:31.510 7f120161a700 1 -- 172.21.2.112:0/1629192383 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11f0002fe0 con 0x7f12040efe00 2021-02-10T11:33:31.516 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:31.510 7f120161a700 1 -- 172.21.2.112:0/1629192383 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1009616230 0 0) 0x7f11fc001460 con 0x7f12040efe00 2021-02-10T11:33:31.516 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:31.510 7f120161a700 1 -- 172.21.2.112:0/1629192383 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f12041b0870 msgr2=0x7f12040f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:31.516 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:31.510 7f120161a700 1 --2- 172.21.2.112:0/1629192383 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f12041b0870 0x7f12040f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:31.516 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:31.510 7f120161a700 1 -- 172.21.2.112:0/1629192383 >> v1:172.21.2.120:6789/0 conn(0x7f12041b0470 legacy=0x7f12040ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:31.516 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:31.510 7f120161a700 1 -- 172.21.2.112:0/1629192383 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12040e52e0 con 0x7f12040efe00 2021-02-10T11:33:31.516 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:31.510 7f120161a700 1 -- 172.21.2.112:0/1629192383 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f11fc001980 con 0x7f12040efe00 2021-02-10T11:33:31.516 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:31.511 7f120161a700 1 -- 172.21.2.112:0/1629192383 >> v1:172.21.2.112:6789/0 conn(0x7f12040efe00 legacy=0x7f12040f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:31.516 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:31.511 7f120161a700 1 --2- 172.21.2.112:0/1629192383 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f11f0003aa0 0x7f11f0005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:31.516 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:31.511 7f120161a700 1 --2- 172.21.2.112:0/1629192383 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f11f0006790 0x7f11f0008c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:31.516 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:31.511 7f120161a700 1 --2- 172.21.2.112:0/1629192383 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f11f0009500 0x7f11f000ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:31.516 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:31.511 7f120161a700 1 -- 172.21.2.112:0/1629192383 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11f000c2d0 con 0x7f11f0003aa0 2021-02-10T11:33:31.516 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:31.511 7f120161a700 1 -- 172.21.2.112:0/1629192383 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11f000c4f0 con 0x7f11f0009500 2021-02-10T11:33:31.517 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:31.511 7f120161a700 1 -- 172.21.2.112:0/1629192383 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11f0050720 con 0x7f11f0006790 2021-02-10T11:33:31.517 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:31.511 7f120261c700 1 --2- 172.21.2.112:0/1629192383 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f11f0003aa0 0x7f11f0005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:31.517 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:31.511 7f1201e1b700 1 --2- 172.21.2.112:0/1629192383 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f11f0006790 0x7f11f0008c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:31.517 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:31.511 7f1202e1d700 1 --2- 172.21.2.112:0/1629192383 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f11f0009500 0x7f11f000ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:31.517 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:31.511 7f120261c700 1 -- 172.21.2.112:0/1629192383 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f11f0006790 msgr2=0x7f11f0008c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:31.517 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:31.511 7f120261c700 1 --2- 172.21.2.112:0/1629192383 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f11f0006790 0x7f11f0008c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:31.518 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:31.511 7f120261c700 1 -- 172.21.2.112:0/1629192383 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f11f0009500 msgr2=0x7f11f000ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:31.518 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:31.511 7f120261c700 1 --2- 172.21.2.112:0/1629192383 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f11f0009500 0x7f11f000ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:31.518 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:31.511 7f120261c700 1 -- 172.21.2.112:0/1629192383 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11f0050980 con 0x7f11f0003aa0 2021-02-10T11:33:31.518 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:31.511 7f120261c700 1 --2- 172.21.2.112:0/1629192383 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f11f0003aa0 0x7f11f0005ef0 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 rx=0x7f11ec006ab0 tx=0x7f11ec009b40).ready entity=mon.0 client_cookie=12dc033c0942c3f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:31.518 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:31.511 7f120161a700 1 -- 172.21.2.112:0/1629192383 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f11ec00b2e0 con 0x7f11f0003aa0 2021-02-10T11:33:31.518 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:31.511 7f120161a700 1 -- 172.21.2.112:0/1629192383 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11ec00b5e0 con 0x7f11f0003aa0 2021-02-10T11:33:31.518 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:31.511 7f120161a700 1 -- 172.21.2.112:0/1629192383 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f11ec00ba30 con 0x7f11f0003aa0 2021-02-10T11:33:31.518 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:31.511 7f1208871700 1 -- 172.21.2.112:0/1629192383 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f11f0003aa0 msgr2=0x7f11f0005ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:31.518 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:31.511 7f1208871700 1 --2- 172.21.2.112:0/1629192383 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f11f0003aa0 0x7f11f0005ef0 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 rx=0x7f11ec006ab0 tx=0x7f11ec009b40).stop 2021-02-10T11:33:31.518 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:31.511 7f1202e1d700 1 -- 172.21.2.112:0/1629192383 reap_dead start 2021-02-10T11:33:31.518 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:31.512 7f1208871700 1 -- 172.21.2.112:0/1629192383 shutdown_connections 2021-02-10T11:33:31.519 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:31.512 7f1208871700 1 -- 172.21.2.112:0/1629192383 >> 172.21.2.112:0/1629192383 conn(0x7f12040ce3e0 msgr2=0x7f12040e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:31.519 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:31.512 7f1208871700 1 -- 172.21.2.112:0/1629192383 shutdown_connections 2021-02-10T11:33:31.519 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:31.512 7f1208871700 1 -- 172.21.2.112:0/1629192383 wait complete. 2021-02-10T11:33:31.519 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:31.527 7f1208871700 1 Processor -- start 2021-02-10T11:33:31.519 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:31.527 7f1208871700 1 -- start start 2021-02-10T11:33:31.519 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:31.527 7f1208871700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f120409b930 0x7f12040b7c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:31.519 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:31.527 7f1208871700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f12040b8410 0x7f12040b8810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:31.519 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:31.527 7f1208871700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f12040c1f80 0x7f12040c2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:31.519 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:31.527 7f1208871700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12040c1a00 con 0x7f12040b8410 2021-02-10T11:33:31.519 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:31.527 7f1208871700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12040b8e30 con 0x7f12040c1f80 2021-02-10T11:33:31.519 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:31.527 7f1208871700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12040b9050 con 0x7f120409b930 2021-02-10T11:33:31.519 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:31.527 7f1201e1b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f12040b8410 0x7f12040b8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:31.519 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:31.527 7f1201e1b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f12040b8410 0x7f12040b8810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:49864/0 (socket says 172.21.2.112:49864) 2021-02-10T11:33:31.520 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:31.527 7f1201e1b700 1 -- 172.21.2.112:0/1972153975 learned_addr learned my addr 172.21.2.112:0/1972153975 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:33:31.520 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:31.527 7f120261c700 1 --2- 172.21.2.112:0/1972153975 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f120409b930 0x7f12040b7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:31.520 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:31.528 7f1202e1d700 1 --2- 172.21.2.112:0/1972153975 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f12040c1f80 0x7f12040c2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:31.520 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:31.528 7f1201e1b700 1 -- 172.21.2.112:0/1972153975 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f120409b930 msgr2=0x7f12040b7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:31.520 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:31.528 7f1201e1b700 1 --2- 172.21.2.112:0/1972153975 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f120409b930 0x7f12040b7c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:31.520 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:31.528 7f1201e1b700 1 -- 172.21.2.112:0/1972153975 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f12040c1f80 msgr2=0x7f12040c2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:31.520 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:31.528 7f1201e1b700 1 --2- 172.21.2.112:0/1972153975 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f12040c1f80 0x7f12040c2380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:31.520 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:31.528 7f1201e1b700 1 -- 172.21.2.112:0/1972153975 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12040c29a0 con 0x7f12040b8410 2021-02-10T11:33:31.520 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:31.528 7f120261c700 1 --2- 172.21.2.112:0/1972153975 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f120409b930 0x7f12040b7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:33:31.520 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:31.528 7f1202e1d700 1 --2- 172.21.2.112:0/1972153975 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f12040c1f80 0x7f12040c2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:33:31.520 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:31.528 7f1201e1b700 1 --2- 172.21.2.112:0/1972153975 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f12040b8410 0x7f12040b8810 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 rx=0x7f11f4006f60 tx=0x7f11f400b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:31.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:31.528 7f11faffd700 1 -- 172.21.2.112:0/1972153975 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f11f400bd60 con 0x7f12040b8410 2021-02-10T11:33:31.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:31.528 7f11faffd700 1 -- 172.21.2.112:0/1972153975 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11f4000c90 con 0x7f12040b8410 2021-02-10T11:33:31.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:31.528 7f11faffd700 1 -- 172.21.2.112:0/1972153975 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f11f4002270 con 0x7f12040b8410 2021-02-10T11:33:31.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:31.528 7f1208871700 1 -- 172.21.2.112:0/1972153975 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12040c29a0 con 0x7f12040b8410 2021-02-10T11:33:31.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:31.528 7f1208871700 1 -- 172.21.2.112:0/1972153975 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12040c29a0 con 0x7f12040b8410 2021-02-10T11:33:31.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:31.529 7f11faffd700 1 -- 172.21.2.112:0/1972153975 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f11f4002020 con 0x7f12040b8410 2021-02-10T11:33:31.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:31.529 7f11faffd700 1 --2- 172.21.2.112:0/1972153975 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f11e80186c0 0x7f11e801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:31.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:31.529 7f11faffd700 1 -- 172.21.2.112:0/1972153975 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f11f4032580 con 0x7f12040b8410 2021-02-10T11:33:31.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:31.529 7f120261c700 1 --2- 172.21.2.112:0/1972153975 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f11e80186c0 0x7f11e801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:31.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:31.530 7f120261c700 1 --2- 172.21.2.112:0/1972153975 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f11e80186c0 0x7f11e801abc0 crc :-1 s=READY pgs=629 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:31.546 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:31.559 7f1208871700 1 -- 172.21.2.112:0/1972153975 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1204105b30 con 0x7f12040b8410 2021-02-10T11:33:31.549 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:31.562 7f11faffd700 1 -- 172.21.2.112:0/1972153975 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f11f402dee0 con 0x7f12040b8410 2021-02-10T11:33:31.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:31.690 7f1208871700 1 -- 172.21.2.112:0/1972153975 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f1204105b30 con 0x7f12040b8410 2021-02-10T11:33:31.678 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:31.691 7f11faffd700 1 -- 172.21.2.112:0/1972153975 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v55) v1 ==== 92+0+10 (secure 0 0 0) 0x7f11f402dee0 con 0x7f12040b8410 2021-02-10T11:33:31.678 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:33:31.680 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:31.693 7f1208871700 1 -- 172.21.2.112:0/1972153975 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f11e80186c0 msgr2=0x7f11e801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:31.680 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:31.693 7f1208871700 1 --2- 172.21.2.112:0/1972153975 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f11e80186c0 0x7f11e801abc0 crc :-1 s=READY pgs=629 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:31.680 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:31.693 7f1208871700 1 -- 172.21.2.112:0/1972153975 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f12040b8410 msgr2=0x7f12040b8810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:31.680 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:31.693 7f1208871700 1 --2- 172.21.2.112:0/1972153975 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f12040b8410 0x7f12040b8810 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 rx=0x7f11f4006f60 tx=0x7f11f400b970).stop 2021-02-10T11:33:31.680 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:31.693 7f1208871700 1 -- 172.21.2.112:0/1972153975 shutdown_connections 2021-02-10T11:33:31.680 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:31.693 7f1208871700 1 --2- 172.21.2.112:0/1972153975 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f11e80186c0 0x7f11e801abc0 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:31.680 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:31.693 7f1208871700 1 --2- 172.21.2.112:0/1972153975 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f12040c1f80 0x7f12040c2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:31.680 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:31.693 7f1208871700 1 --2- 172.21.2.112:0/1972153975 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f12040b8410 0x7f12040b8810 unknown :-1 s=CLOSED pgs=1260 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:31.681 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:31.693 7f1208871700 1 --2- 172.21.2.112:0/1972153975 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f120409b930 0x7f12040b7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:31.681 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:31.693 7f1208871700 1 -- 172.21.2.112:0/1972153975 >> 172.21.2.112:0/1972153975 conn(0x7f1204094e10 msgr2=0x7f12040a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:31.681 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:31.693 7f1208871700 1 -- 172.21.2.112:0/1972153975 shutdown_connections 2021-02-10T11:33:31.681 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:31.693 7f1208871700 1 -- 172.21.2.112:0/1972153975 wait complete. 2021-02-10T11:33:31.689 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:33:32.118 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.115 7fe3e3b94700 1 Processor -- start 2021-02-10T11:33:32.119 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.115 7fe3e3b94700 1 -- start start 2021-02-10T11:33:32.119 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.116 7fe3e3b94700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fe3dc1b0470 0x7fe3dc1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:32.119 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.116 7fe3e3b94700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fe3dc0ef610 0x7fe3dc0efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:32.119 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.116 7fe3e3b94700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3dc0f0b50 con 0x7fe3dc0f0030 2021-02-10T11:33:32.119 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.116 7fe3e3b94700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3dc0e52e0 con 0x7fe3dc1b0470 2021-02-10T11:33:32.119 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.116 7fe3e3b94700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3dc0e5490 con 0x7fe3dc0ef610 2021-02-10T11:33:32.119 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.116 7fe3e21b1700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fe3dc0f0030 0x7fe3dc0f10d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:49330/0 (socket says 172.21.2.112:49330) 2021-02-10T11:33:32.119 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.116 7fe3e21b1700 1 -- 172.21.2.112:0/4067720696 learned_addr learned my addr 172.21.2.112:0/4067720696 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:33:32.119 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.116 7fe3e19b0700 1 --2- 172.21.2.112:0/4067720696 >> v2:172.21.2.112:3300/0 conn(0x7fe3dc1b0470 0x7fe3dc1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:32.119 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.116 7fe3e09ae700 1 -- 172.21.2.112:0/4067720696 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2516887129 0 0) 0x7fe3d4001500 con 0x7fe3dc0f0030 2021-02-10T11:33:32.119 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.116 7fe3e09ae700 1 -- 172.21.2.112:0/4067720696 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3c8002fe0 con 0x7fe3dc0f0030 2021-02-10T11:33:32.119 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.116 7fe3e19b0700 1 -- 172.21.2.112:0/4067720696 >> v2:172.21.2.120:3300/0 conn(0x7fe3dc0ef610 msgr2=0x7fe3dc0efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:32.119 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.116 7fe3e19b0700 1 --2- 172.21.2.112:0/4067720696 >> v2:172.21.2.120:3300/0 conn(0x7fe3dc0ef610 0x7fe3dc0efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:32.120 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.116 7fe3e19b0700 1 -- 172.21.2.112:0/4067720696 >> v1:172.21.2.112:6789/0 conn(0x7fe3dc0f0030 legacy=0x7fe3dc0f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:32.120 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.117 7fe3e19b0700 1 -- 172.21.2.112:0/4067720696 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3dc0f0430 con 0x7fe3dc1b0470 2021-02-10T11:33:32.120 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.117 7fe3e19b0700 1 --2- 172.21.2.112:0/4067720696 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe3dc1b0470 0x7fe3dc1b0870 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 rx=0x7fe3d8006650 tx=0x7fe3d8009a80).ready entity=mon.0 client_cookie=b79069b175d72b4a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:32.120 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.117 7fe3e09ae700 1 -- 172.21.2.112:0/4067720696 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe3d800c0c0 con 0x7fe3dc1b0470 2021-02-10T11:33:32.120 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.117 7fe3e09ae700 1 -- 172.21.2.112:0/4067720696 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3d800c340 con 0x7fe3dc1b0470 2021-02-10T11:33:32.120 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.117 7fe3e09ae700 1 -- 172.21.2.112:0/4067720696 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe3d800c860 con 0x7fe3dc1b0470 2021-02-10T11:33:32.120 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.117 7fe3e3b94700 1 -- 172.21.2.112:0/4067720696 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe3dc1b0470 msgr2=0x7fe3dc1b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:32.120 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.117 7fe3e3b94700 1 --2- 172.21.2.112:0/4067720696 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe3dc1b0470 0x7fe3dc1b0870 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 rx=0x7fe3d8006650 tx=0x7fe3d8009a80).stop 2021-02-10T11:33:32.120 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.117 7fe3e3b94700 1 -- 172.21.2.112:0/4067720696 shutdown_connections 2021-02-10T11:33:32.120 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.117 7fe3e3b94700 1 --2- 172.21.2.112:0/4067720696 >> v2:172.21.2.120:3300/0 conn(0x7fe3dc0ef610 0x7fe3dc0efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:32.120 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.117 7fe3e3b94700 1 --2- 172.21.2.112:0/4067720696 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe3dc1b0470 0x7fe3dc1b0870 unknown :-1 s=CLOSED pgs=1262 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:32.120 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.117 7fe3e3b94700 1 -- 172.21.2.112:0/4067720696 >> 172.21.2.112:0/4067720696 conn(0x7fe3dc0ce3e0 msgr2=0x7fe3dc0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:32.120 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.118 7fe3e3b94700 1 -- 172.21.2.112:0/4067720696 shutdown_connections 2021-02-10T11:33:32.120 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.118 7fe3e3b94700 1 -- 172.21.2.112:0/4067720696 wait complete. 2021-02-10T11:33:32.122 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.132 7fe3e3b94700 1 Processor -- start 2021-02-10T11:33:32.122 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.132 7fe3e3b94700 1 -- start start 2021-02-10T11:33:32.122 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.132 7fe3e3b94700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe3dc0a0420 0x7fe3dc0a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:32.122 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.132 7fe3e3b94700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe3dc189d00 0x7fe3dc18a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:32.122 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.132 7fe3e3b94700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe3dc18a720 0x7fe3dc0b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:32.122 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.132 7fe3e3b94700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3dc189780 con 0x7fe3dc18a720 2021-02-10T11:33:32.122 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.132 7fe3e3b94700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3dc18ab20 con 0x7fe3dc0a0420 2021-02-10T11:33:32.123 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.132 7fe3e3b94700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3dc0a1140 con 0x7fe3dc189d00 2021-02-10T11:33:32.123 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.133 7fe3e21b1700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe3dc18a720 0x7fe3dc0b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:32.123 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.133 7fe3e21b1700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe3dc18a720 0x7fe3dc0b7be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:49872/0 (socket says 172.21.2.112:49872) 2021-02-10T11:33:32.123 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.133 7fe3e21b1700 1 -- 172.21.2.112:0/3793325803 learned_addr learned my addr 172.21.2.112:0/3793325803 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:33:32.123 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.133 7fe3e21b1700 1 -- 172.21.2.112:0/3793325803 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe3dc189d00 msgr2=0x7fe3dc18a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:32.123 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.133 7fe3e21b1700 1 --2- 172.21.2.112:0/3793325803 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe3dc189d00 0x7fe3dc18a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:32.123 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.133 7fe3e21b1700 1 -- 172.21.2.112:0/3793325803 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe3dc0a0420 msgr2=0x7fe3dc0a0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:32.123 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.133 7fe3e21b1700 1 --2- 172.21.2.112:0/3793325803 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe3dc0a0420 0x7fe3dc0a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:32.123 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.133 7fe3e21b1700 1 -- 172.21.2.112:0/3793325803 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3dc0b8200 con 0x7fe3dc18a720 2021-02-10T11:33:32.123 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.133 7fe3e21b1700 1 --2- 172.21.2.112:0/3793325803 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe3dc18a720 0x7fe3dc0b7be0 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 rx=0x7fe3d4009980 tx=0x7fe3d400bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:32.123 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.133 7fe3ce7fc700 1 -- 172.21.2.112:0/3793325803 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe3d400be10 con 0x7fe3dc18a720 2021-02-10T11:33:32.123 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.133 7fe3ce7fc700 1 -- 172.21.2.112:0/3793325803 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3d4006080 con 0x7fe3dc18a720 2021-02-10T11:33:32.123 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.133 7fe3ce7fc700 1 -- 172.21.2.112:0/3793325803 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe3d4004720 con 0x7fe3dc18a720 2021-02-10T11:33:32.124 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.133 7fe3e3b94700 1 -- 172.21.2.112:0/3793325803 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3dc0b8200 con 0x7fe3dc18a720 2021-02-10T11:33:32.124 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.134 7fe3e3b94700 1 -- 172.21.2.112:0/3793325803 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3dc0b8200 con 0x7fe3dc18a720 2021-02-10T11:33:32.124 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.134 7fe3ce7fc700 1 -- 172.21.2.112:0/3793325803 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe3d4006260 con 0x7fe3dc18a720 2021-02-10T11:33:32.124 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.134 7fe3ce7fc700 1 --2- 172.21.2.112:0/3793325803 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe3c4018730 0x7fe3c401abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:32.124 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.134 7fe3ce7fc700 1 -- 172.21.2.112:0/3793325803 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fe3d4032b70 con 0x7fe3dc18a720 2021-02-10T11:33:32.124 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.135 7fe3e19b0700 1 --2- 172.21.2.112:0/3793325803 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe3c4018730 0x7fe3c401abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:32.124 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.135 7fe3e19b0700 1 --2- 172.21.2.112:0/3793325803 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe3c4018730 0x7fe3c401abf0 crc :-1 s=READY pgs=630 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:32.152 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.166 7fe3e3b94700 1 -- 172.21.2.112:0/3793325803 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3dc105ae0 con 0x7fe3dc18a720 2021-02-10T11:33:32.155 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.168 7fe3ce7fc700 1 -- 172.21.2.112:0/3793325803 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe3d4006290 con 0x7fe3dc18a720 2021-02-10T11:33:32.282 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.295 7fe3e3b94700 1 -- 172.21.2.112:0/3793325803 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7fe3dc105ae0 con 0x7fe3dc18a720 2021-02-10T11:33:32.283 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.296 7fe3ce7fc700 1 -- 172.21.2.112:0/3793325803 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v55) v1 ==== 104+0+10 (secure 0 0 0) 0x7fe3d4006000 con 0x7fe3dc18a720 2021-02-10T11:33:32.283 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:33:32.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.298 7fe3e3b94700 1 -- 172.21.2.112:0/3793325803 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe3c4018730 msgr2=0x7fe3c401abf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:32.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.298 7fe3e3b94700 1 --2- 172.21.2.112:0/3793325803 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe3c4018730 0x7fe3c401abf0 crc :-1 s=READY pgs=630 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:32.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.298 7fe3e3b94700 1 -- 172.21.2.112:0/3793325803 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe3dc18a720 msgr2=0x7fe3dc0b7be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:32.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.298 7fe3e3b94700 1 --2- 172.21.2.112:0/3793325803 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe3dc18a720 0x7fe3dc0b7be0 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 rx=0x7fe3d4009980 tx=0x7fe3d400bcd0).stop 2021-02-10T11:33:32.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.298 7fe3e3b94700 1 -- 172.21.2.112:0/3793325803 shutdown_connections 2021-02-10T11:33:32.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.298 7fe3e3b94700 1 --2- 172.21.2.112:0/3793325803 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe3c4018730 0x7fe3c401abf0 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:32.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.298 7fe3e3b94700 1 --2- 172.21.2.112:0/3793325803 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe3dc18a720 0x7fe3dc0b7be0 unknown :-1 s=CLOSED pgs=1263 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:32.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.298 7fe3e3b94700 1 --2- 172.21.2.112:0/3793325803 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe3dc189d00 0x7fe3dc18a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:32.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.298 7fe3e3b94700 1 --2- 172.21.2.112:0/3793325803 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe3dc0a0420 0x7fe3dc0a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:32.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.298 7fe3e3b94700 1 -- 172.21.2.112:0/3793325803 >> 172.21.2.112:0/3793325803 conn(0x7fe3dc094e10 msgr2=0x7fe3dc0a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:32.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.298 7fe3e3b94700 1 -- 172.21.2.112:0/3793325803 shutdown_connections 2021-02-10T11:33:32.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.298 7fe3e3b94700 1 -- 172.21.2.112:0/3793325803 wait complete. 2021-02-10T11:33:32.293 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:33:32.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.718 7fa75a247700 1 Processor -- start 2021-02-10T11:33:32.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.718 7fa75a247700 1 -- start start 2021-02-10T11:33:32.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.719 7fa75a247700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa7541b0870 0x7fa7540f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:32.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.719 7fa75a247700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fa7540efe00 0x7fa7540f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:32.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.719 7fa75a247700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7541b0c70 con 0x7fa7541b0470 2021-02-10T11:33:32.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.719 7fa75a247700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7540f0b50 con 0x7fa7540efe00 2021-02-10T11:33:32.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.719 7fa75a247700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7540f0820 con 0x7fa7541b0870 2021-02-10T11:33:32.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.719 7fa753fff700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fa7541b0470 0x7fa7540ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:49340/0 (socket says 172.21.2.112:49340) 2021-02-10T11:33:32.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.719 7fa753fff700 1 -- 172.21.2.112:0/3494000987 learned_addr learned my addr 172.21.2.112:0/3494000987 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:33:32.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.719 7fa758864700 1 --2- 172.21.2.112:0/3494000987 >> v2:172.21.2.120:3300/0 conn(0x7fa7540efe00 0x7fa7540f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:32.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.719 7fa7537fe700 1 --2- 172.21.2.112:0/3494000987 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa7541b0870 0x7fa7540f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:32.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.719 7fa752ffd700 1 -- 172.21.2.112:0/3494000987 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 283647891 0 0) 0x7fa744000ce0 con 0x7fa7541b0470 2021-02-10T11:33:32.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.719 7fa752ffd700 1 -- 172.21.2.112:0/3494000987 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa73c002fe0 con 0x7fa7541b0470 2021-02-10T11:33:32.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.720 7fa758864700 1 -- 172.21.2.112:0/3494000987 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa7541b0870 msgr2=0x7fa7540f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:32.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.720 7fa758864700 1 --2- 172.21.2.112:0/3494000987 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa7541b0870 0x7fa7540f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:32.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.720 7fa758864700 1 -- 172.21.2.112:0/3494000987 >> v1:172.21.2.112:6789/0 conn(0x7fa7541b0470 legacy=0x7fa7540ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:32.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.720 7fa752ffd700 1 -- 172.21.2.112:0/3494000987 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2257109764 0 0) 0x7fa744000ed0 con 0x7fa7541b0470 2021-02-10T11:33:32.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.720 7fa758864700 1 -- 172.21.2.112:0/3494000987 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7540e52e0 con 0x7fa7540efe00 2021-02-10T11:33:32.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.720 7fa758864700 1 --2- 172.21.2.112:0/3494000987 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa7540efe00 0x7fa7540f0200 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 rx=0x7fa74c006e50 tx=0x7fa74c009e70).ready entity=mon.1 client_cookie=b2e3b22839fefde5 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:32.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.720 7fa752ffd700 1 -- 172.21.2.112:0/3494000987 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa74c006a20 con 0x7fa7540efe00 2021-02-10T11:33:32.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.720 7fa752ffd700 1 -- 172.21.2.112:0/3494000987 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa74c006790 con 0x7fa7540efe00 2021-02-10T11:33:32.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.720 7fa752ffd700 1 -- 172.21.2.112:0/3494000987 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa74c00b700 con 0x7fa7540efe00 2021-02-10T11:33:32.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.720 7fa75a247700 1 -- 172.21.2.112:0/3494000987 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa7540efe00 msgr2=0x7fa7540f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:32.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.720 7fa75a247700 1 --2- 172.21.2.112:0/3494000987 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa7540efe00 0x7fa7540f0200 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 rx=0x7fa74c006e50 tx=0x7fa74c009e70).stop 2021-02-10T11:33:32.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.721 7fa75a247700 1 -- 172.21.2.112:0/3494000987 shutdown_connections 2021-02-10T11:33:32.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.721 7fa75a247700 1 --2- 172.21.2.112:0/3494000987 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa7540efe00 0x7fa7540f0200 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:32.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.721 7fa75a247700 1 --2- 172.21.2.112:0/3494000987 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa7541b0870 0x7fa7540f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:32.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.721 7fa75a247700 1 -- 172.21.2.112:0/3494000987 >> 172.21.2.112:0/3494000987 conn(0x7fa7540ce3e0 msgr2=0x7fa7540e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:32.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.721 7fa75a247700 1 -- 172.21.2.112:0/3494000987 shutdown_connections 2021-02-10T11:33:32.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.721 7fa75a247700 1 -- 172.21.2.112:0/3494000987 wait complete. 2021-02-10T11:33:32.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.735 7fa75a247700 1 Processor -- start 2021-02-10T11:33:32.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.735 7fa75a247700 1 -- start start 2021-02-10T11:33:32.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.736 7fa75a247700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa75409f140 0x7fa7540c1940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:32.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.736 7fa75a247700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa7540c1f60 0x7fa75409e300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:32.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.736 7fa75a247700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa75409e920 0x7fa7540c2960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:32.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.736 7fa75a247700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7540c23e0 con 0x7fa7540c1f60 2021-02-10T11:33:32.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.736 7fa75a247700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa75409ed20 con 0x7fa75409f140 2021-02-10T11:33:32.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.736 7fa75a247700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa754105d10 con 0x7fa75409e920 2021-02-10T11:33:32.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.736 7fa7537fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa7540c1f60 0x7fa75409e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:32.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.736 7fa7537fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa7540c1f60 0x7fa75409e300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:49884/0 (socket says 172.21.2.112:49884) 2021-02-10T11:33:32.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.736 7fa7537fe700 1 -- 172.21.2.112:0/997796585 learned_addr learned my addr 172.21.2.112:0/997796585 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:33:32.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.736 7fa758864700 1 --2- 172.21.2.112:0/997796585 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa75409e920 0x7fa7540c2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:32.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.736 7fa753fff700 1 --2- 172.21.2.112:0/997796585 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa75409f140 0x7fa7540c1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:32.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.737 7fa7537fe700 1 -- 172.21.2.112:0/997796585 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa75409e920 msgr2=0x7fa7540c2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:32.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.737 7fa7537fe700 1 --2- 172.21.2.112:0/997796585 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa75409e920 0x7fa7540c2960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:32.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.737 7fa7537fe700 1 -- 172.21.2.112:0/997796585 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa75409f140 msgr2=0x7fa7540c1940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:32.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.737 7fa7537fe700 1 --2- 172.21.2.112:0/997796585 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa75409f140 0x7fa7540c1940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:32.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.737 7fa7537fe700 1 -- 172.21.2.112:0/997796585 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7541a4910 con 0x7fa7540c1f60 2021-02-10T11:33:32.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.737 7fa7537fe700 1 --2- 172.21.2.112:0/997796585 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa7540c1f60 0x7fa75409e300 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 rx=0x7fa748006f60 tx=0x7fa74800b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:32.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.737 7fa750ff9700 1 -- 172.21.2.112:0/997796585 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa74800bd60 con 0x7fa7540c1f60 2021-02-10T11:33:32.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.737 7fa750ff9700 1 -- 172.21.2.112:0/997796585 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa748000c90 con 0x7fa7540c1f60 2021-02-10T11:33:32.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.737 7fa750ff9700 1 -- 172.21.2.112:0/997796585 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa748002220 con 0x7fa7540c1f60 2021-02-10T11:33:32.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.737 7fa75a247700 1 -- 172.21.2.112:0/997796585 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7541a4910 con 0x7fa7540c1f60 2021-02-10T11:33:32.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.737 7fa75a247700 1 -- 172.21.2.112:0/997796585 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7541a4910 con 0x7fa7540c1f60 2021-02-10T11:33:32.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.738 7fa750ff9700 1 -- 172.21.2.112:0/997796585 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa748006b00 con 0x7fa7540c1f60 2021-02-10T11:33:32.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.738 7fa750ff9700 1 --2- 172.21.2.112:0/997796585 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa7340186c0 0x7fa73401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:32.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.738 7fa750ff9700 1 -- 172.21.2.112:0/997796585 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fa748032420 con 0x7fa7540c1f60 2021-02-10T11:33:32.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.738 7fa753fff700 1 --2- 172.21.2.112:0/997796585 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa7340186c0 0x7fa73401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:32.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.738 7fa753fff700 1 --2- 172.21.2.112:0/997796585 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa7340186c0 0x7fa73401abc0 crc :-1 s=READY pgs=631 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:32.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.769 7fa75a247700 1 -- 172.21.2.112:0/997796585 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7540f0b50 con 0x7fa7540c1f60 2021-02-10T11:33:32.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.771 7fa750ff9700 1 -- 172.21.2.112:0/997796585 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa748030d50 con 0x7fa7540c1f60 2021-02-10T11:33:32.886 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.899 7fa75a247700 1 -- 172.21.2.112:0/997796585 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7fa7540f0b50 con 0x7fa7540c1f60 2021-02-10T11:33:32.886 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.899 7fa750ff9700 1 -- 172.21.2.112:0/997796585 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v55) v1 ==== 100+0+10 (secure 0 0 0) 0x7fa74800bd60 con 0x7fa7540c1f60 2021-02-10T11:33:32.887 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:33:32.888 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.901 7fa75a247700 1 -- 172.21.2.112:0/997796585 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa7340186c0 msgr2=0x7fa73401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:32.888 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.901 7fa75a247700 1 --2- 172.21.2.112:0/997796585 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa7340186c0 0x7fa73401abc0 crc :-1 s=READY pgs=631 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:32.888 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.901 7fa75a247700 1 -- 172.21.2.112:0/997796585 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa7540c1f60 msgr2=0x7fa75409e300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:32.889 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.901 7fa75a247700 1 --2- 172.21.2.112:0/997796585 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa7540c1f60 0x7fa75409e300 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 rx=0x7fa748006f60 tx=0x7fa74800b970).stop 2021-02-10T11:33:32.889 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.901 7fa75a247700 1 -- 172.21.2.112:0/997796585 shutdown_connections 2021-02-10T11:33:32.889 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.901 7fa75a247700 1 --2- 172.21.2.112:0/997796585 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa7340186c0 0x7fa73401abc0 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:32.889 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.901 7fa75a247700 1 --2- 172.21.2.112:0/997796585 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa75409e920 0x7fa7540c2960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:32.889 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.901 7fa75a247700 1 --2- 172.21.2.112:0/997796585 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa7540c1f60 0x7fa75409e300 unknown :-1 s=CLOSED pgs=1265 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:32.889 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.901 7fa75a247700 1 --2- 172.21.2.112:0/997796585 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa75409f140 0x7fa7540c1940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:32.890 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.901 7fa75a247700 1 -- 172.21.2.112:0/997796585 >> 172.21.2.112:0/997796585 conn(0x7fa75409fc10 msgr2=0x7fa7540b3250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:32.890 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.901 7fa75a247700 1 -- 172.21.2.112:0/997796585 shutdown_connections 2021-02-10T11:33:32.890 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:32.901 7fa75a247700 1 -- 172.21.2.112:0/997796585 wait complete. 2021-02-10T11:33:32.897 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:33:32.897 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:33:33.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:33.321 7f5b34ffd700 1 Processor -- start 2021-02-10T11:33:33.326 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:33.321 7f5b34ffd700 1 -- start start 2021-02-10T11:33:33.326 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:33.321 7f5b34ffd700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5b300e5070 0x7f5b301b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:33.326 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:33.321 7f5b34ffd700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f5b300e5470 0x7f5b3007d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:33.326 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:33.321 7f5b34ffd700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b3007ed80 con 0x7f5b301b3e40 2021-02-10T11:33:33.326 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:33.321 7f5b34ffd700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b30080310 con 0x7f5b300e5470 2021-02-10T11:33:33.326 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:33.321 7f5b34ffd700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b30080500 con 0x7f5b300e5070 2021-02-10T11:33:33.326 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:33.322 7f5b2ed9d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5b300e5070 0x7f5b301b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:33.326 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:33.322 7f5b2f59e700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7f5b301b3e40 0x7f5b3007dcb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:42916/0 (socket says 172.21.2.112:42916) 2021-02-10T11:33:33.326 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:33.322 7f5b2f59e700 1 -- 172.21.2.112:0/1746178315 learned_addr learned my addr 172.21.2.112:0/1746178315 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:33:33.326 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:33.322 7f5b2e59c700 1 --2- 172.21.2.112:0/1746178315 >> v2:172.21.2.120:3300/0 conn(0x7f5b300e5470 0x7f5b3007d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:33.327 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:33.323 7f5b2ed9d700 1 -- 172.21.2.112:0/1746178315 >> v2:172.21.2.120:3300/0 conn(0x7f5b300e5470 msgr2=0x7f5b3007d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:33.327 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:33.323 7f5b2ed9d700 1 --2- 172.21.2.112:0/1746178315 >> v2:172.21.2.120:3300/0 conn(0x7f5b300e5470 0x7f5b3007d690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:33.327 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:33.323 7f5b2ed9d700 1 -- 172.21.2.112:0/1746178315 >> v1:172.21.2.120:6789/0 conn(0x7f5b301b3e40 legacy=0x7f5b3007dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:33.327 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:33.323 7f5b2dd9b700 1 -- 172.21.2.112:0/1746178315 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3306554126 0 0) 0x7f5b28001510 con 0x7f5b301b3e40 2021-02-10T11:33:33.327 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:33.323 7f5b2ed9d700 1 -- 172.21.2.112:0/1746178315 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b300806f0 con 0x7f5b300e5070 2021-02-10T11:33:33.327 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:33.323 7f5b2e59c700 1 --2- 172.21.2.112:0/1746178315 >> v2:172.21.2.120:3300/0 conn(0x7f5b300e5470 0x7f5b3007d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:33:33.327 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:33.323 7f5b2ed9d700 1 --2- 172.21.2.112:0/1746178315 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5b300e5070 0x7f5b301b3820 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f5b18006650 tx=0x7f5b18009a80).ready entity=mon.2 client_cookie=fe1c96589536d785 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:33.327 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:33.323 7f5b2dd9b700 1 -- 172.21.2.112:0/1746178315 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5b1800c0c0 con 0x7f5b300e5070 2021-02-10T11:33:33.327 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:33.323 7f5b2dd9b700 1 -- 172.21.2.112:0/1746178315 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b1800c340 con 0x7f5b300e5070 2021-02-10T11:33:33.327 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:33.323 7f5b2dd9b700 1 -- 172.21.2.112:0/1746178315 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5b1800c860 con 0x7f5b300e5070 2021-02-10T11:33:33.327 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:33.324 7f5b34ffd700 1 -- 172.21.2.112:0/1746178315 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5b300e5070 msgr2=0x7f5b301b3820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:33.327 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:33.324 7f5b34ffd700 1 --2- 172.21.2.112:0/1746178315 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5b300e5070 0x7f5b301b3820 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f5b18006650 tx=0x7f5b18009a80).stop 2021-02-10T11:33:33.327 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:33.324 7f5b34ffd700 1 -- 172.21.2.112:0/1746178315 shutdown_connections 2021-02-10T11:33:33.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:33.324 7f5b34ffd700 1 --2- 172.21.2.112:0/1746178315 >> v2:172.21.2.120:3300/0 conn(0x7f5b300e5470 0x7f5b3007d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:33.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:33.324 7f5b34ffd700 1 --2- 172.21.2.112:0/1746178315 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5b300e5070 0x7f5b301b3820 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:33.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:33.324 7f5b34ffd700 1 -- 172.21.2.112:0/1746178315 >> 172.21.2.112:0/1746178315 conn(0x7f5b300ce690 msgr2=0x7f5b300e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:33.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:33.324 7f5b34ffd700 1 -- 172.21.2.112:0/1746178315 shutdown_connections 2021-02-10T11:33:33.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:33.325 7f5b34ffd700 1 -- 172.21.2.112:0/1746178315 wait complete. 2021-02-10T11:33:33.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:33.339 7f5b34ffd700 1 Processor -- start 2021-02-10T11:33:33.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:33.339 7f5b34ffd700 1 -- start start 2021-02-10T11:33:33.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:33.339 7f5b34ffd700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5b30187eb0 0x7f5b300b86b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:33.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:33.339 7f5b34ffd700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5b300b8cd0 0x7f5b30188c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:33.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:33.339 7f5b34ffd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5b30189250 0x7f5b30189650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:33.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:33.339 7f5b34ffd700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b301882b0 con 0x7f5b30189250 2021-02-10T11:33:33.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:33.339 7f5b34ffd700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b301884a0 con 0x7f5b30187eb0 2021-02-10T11:33:33.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:33.339 7f5b34ffd700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b30189c70 con 0x7f5b300b8cd0 2021-02-10T11:33:33.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:33.340 7f5b2f59e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5b30189250 0x7f5b30189650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:33.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:33.340 7f5b2f59e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5b30189250 0x7f5b30189650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:49900/0 (socket says 172.21.2.112:49900) 2021-02-10T11:33:33.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:33.340 7f5b2f59e700 1 -- 172.21.2.112:0/1582868779 learned_addr learned my addr 172.21.2.112:0/1582868779 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:33:33.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:33.340 7f5b2f59e700 1 -- 172.21.2.112:0/1582868779 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5b300b8cd0 msgr2=0x7f5b30188c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:33.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:33.340 7f5b2f59e700 1 --2- 172.21.2.112:0/1582868779 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5b300b8cd0 0x7f5b30188c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:33.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:33.340 7f5b2f59e700 1 -- 172.21.2.112:0/1582868779 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5b30187eb0 msgr2=0x7f5b300b86b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:33.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:33.340 7f5b2f59e700 1 --2- 172.21.2.112:0/1582868779 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5b30187eb0 0x7f5b300b86b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:33.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:33.340 7f5b2f59e700 1 -- 172.21.2.112:0/1582868779 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b30189e90 con 0x7f5b30189250 2021-02-10T11:33:33.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:33.340 7f5b2f59e700 1 --2- 172.21.2.112:0/1582868779 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5b30189250 0x7f5b30189650 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 rx=0x7f5b2800a900 tx=0x7f5b2800aca0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:33.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:33.341 7f5b277fe700 1 -- 172.21.2.112:0/1582868779 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5b28000f40 con 0x7f5b30189250 2021-02-10T11:33:33.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:33.341 7f5b277fe700 1 -- 172.21.2.112:0/1582868779 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b28007020 con 0x7f5b30189250 2021-02-10T11:33:33.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:33.341 7f5b34ffd700 1 -- 172.21.2.112:0/1582868779 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b30189e90 con 0x7f5b30189250 2021-02-10T11:33:33.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:33.341 7f5b34ffd700 1 -- 172.21.2.112:0/1582868779 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b30189e90 con 0x7f5b30189250 2021-02-10T11:33:33.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:33.341 7f5b277fe700 1 -- 172.21.2.112:0/1582868779 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5b280046a0 con 0x7f5b30189250 2021-02-10T11:33:33.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:33.341 7f5b277fe700 1 -- 172.21.2.112:0/1582868779 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5b28007200 con 0x7f5b30189250 2021-02-10T11:33:33.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:33.341 7f5b277fe700 1 --2- 172.21.2.112:0/1582868779 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5b14018730 0x7f5b1401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:33.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:33.341 7f5b277fe700 1 -- 172.21.2.112:0/1582868779 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f5b28032b20 con 0x7f5b30189250 2021-02-10T11:33:33.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:33.342 7f5b2ed9d700 1 --2- 172.21.2.112:0/1582868779 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5b14018730 0x7f5b1401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:33.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:33.342 7f5b2ed9d700 1 --2- 172.21.2.112:0/1582868779 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5b14018730 0x7f5b1401ac10 crc :-1 s=READY pgs=632 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:33.358 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:33.371 7f5b34ffd700 1 -- 172.21.2.112:0/1582868779 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b3007ed80 con 0x7f5b30189250 2021-02-10T11:33:33.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:33.373 7f5b277fe700 1 -- 172.21.2.112:0/1582868779 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5b28007c10 con 0x7f5b30189250 2021-02-10T11:33:33.488 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:33.501 7f5b34ffd700 1 -- 172.21.2.112:0/1582868779 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5b3007ed80 con 0x7f5b30189250 2021-02-10T11:33:33.488 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:33.501 7f5b277fe700 1 -- 172.21.2.112:0/1582868779 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) v1 ==== 94+0+2101 (secure 0 0 0) 0x7f5b280011f0 con 0x7f5b30189250 2021-02-10T11:33:33.489 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:33:33.489 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":39,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7881,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/1476007789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":1476007789},{"type":"v1","addr":"172.21.2.112:6833","nonce":1476007789}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":35}],"filesystems":[{"mdsmap":{"epoch":39,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:33:23.550791","modified":"2021-02-10 11:33:25.562811","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6595":{"gid":6595,"name":"b","rank":0,"incarnation":36,"state":"up:active","state_seq":3,"addr":"172.21.2.120:6835/4241229406","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":4241229406},{"type":"v1","addr":"172.21.2.120:6835","nonce":4241229406}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2021-02-10T11:33:33.489 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 39 2021-02-10T11:33:33.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:33.503 7f5b34ffd700 1 -- 172.21.2.112:0/1582868779 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5b14018730 msgr2=0x7f5b1401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:33.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:33.503 7f5b34ffd700 1 --2- 172.21.2.112:0/1582868779 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5b14018730 0x7f5b1401ac10 crc :-1 s=READY pgs=632 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:33.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:33.503 7f5b34ffd700 1 -- 172.21.2.112:0/1582868779 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5b30189250 msgr2=0x7f5b30189650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:33.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:33.503 7f5b34ffd700 1 --2- 172.21.2.112:0/1582868779 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5b30189250 0x7f5b30189650 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 rx=0x7f5b2800a900 tx=0x7f5b2800aca0).stop 2021-02-10T11:33:33.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:33.504 7f5b34ffd700 1 -- 172.21.2.112:0/1582868779 shutdown_connections 2021-02-10T11:33:33.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:33.504 7f5b34ffd700 1 --2- 172.21.2.112:0/1582868779 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5b14018730 0x7f5b1401ac10 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:33.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:33.504 7f5b34ffd700 1 --2- 172.21.2.112:0/1582868779 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5b30189250 0x7f5b30189650 unknown :-1 s=CLOSED pgs=1266 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:33.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:33.504 7f5b34ffd700 1 --2- 172.21.2.112:0/1582868779 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5b300b8cd0 0x7f5b30188c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:33.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:33.504 7f5b34ffd700 1 --2- 172.21.2.112:0/1582868779 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5b30187eb0 0x7f5b300b86b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:33.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:33.504 7f5b34ffd700 1 -- 172.21.2.112:0/1582868779 >> 172.21.2.112:0/1582868779 conn(0x7f5b3009b790 msgr2=0x7f5b30089700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:33.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:33.504 7f5b34ffd700 1 -- 172.21.2.112:0/1582868779 shutdown_connections 2021-02-10T11:33:33.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:33.504 7f5b34ffd700 1 -- 172.21.2.112:0/1582868779 wait complete. 2021-02-10T11:33:33.701 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 39, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:33:23.550791', 'modified': '2021-02-10 11:33:25.562811', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6595}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6595': {'gid': 6595, 'name': 'b', 'rank': 0, 'incarnation': 36, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.120:6835/4241229406', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.120:6834', 'nonce': 4241229406}, {'type': 'v1', 'addr': '172.21.2.120:6835', 'nonce': 4241229406}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [11], 'metadata_pool': 10, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:33:33.701 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:33:33.701 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:33:34.098 INFO:teuthology.orchestra.run.gibba020.stdout:{ 2021-02-10T11:33:34.099 INFO:teuthology.orchestra.run.gibba020.stdout: "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", 2021-02-10T11:33:34.099 INFO:teuthology.orchestra.run.gibba020.stdout: "whoami": 0, 2021-02-10T11:33:34.099 INFO:teuthology.orchestra.run.gibba020.stdout: "id": 6595, 2021-02-10T11:33:34.099 INFO:teuthology.orchestra.run.gibba020.stdout: "want_state": "up:active", 2021-02-10T11:33:34.099 INFO:teuthology.orchestra.run.gibba020.stdout: "state": "up:active", 2021-02-10T11:33:34.099 INFO:teuthology.orchestra.run.gibba020.stdout: "rank_uptime": 10.559016602, 2021-02-10T11:33:34.099 INFO:teuthology.orchestra.run.gibba020.stdout: "mdsmap_epoch": 39, 2021-02-10T11:33:34.099 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch": 55, 2021-02-10T11:33:34.099 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch_barrier": 55, 2021-02-10T11:33:34.099 INFO:teuthology.orchestra.run.gibba020.stdout: "uptime": 16.188771925000001 2021-02-10T11:33:34.100 INFO:teuthology.orchestra.run.gibba020.stdout:} 2021-02-10T11:33:34.103 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", "whoami": 0, "id": 6595, "want_state": "up:active", "state": "up:active", "rank_uptime": 10.559016602, "mdsmap_epoch": 39, "osdmap_epoch": 55, "osdmap_epoch_barrier": 55, "uptime": 16.188771925000001 } 2021-02-10T11:33:34.103 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:33:34.103 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2021-02-10T11:33:34.103 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@gibba012.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2021-02-10T11:33:34.103 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:34.103 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:33:34.439 DEBUG:teuthology.orchestra.run.gibba012:> sudo modprobe fuse 2021-02-10T11:33:34.784 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:33:34.784 DEBUG:teuthology.orchestra.run.gibba012:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:33:35.124 INFO:teuthology.orchestra.run.gibba012.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:33:35.125 INFO:teuthology.orchestra.run.gibba012.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:33:35.125 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:33:35.126 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:35.126 DEBUG:teuthology.orchestra.run.gibba012:> ls /sys/fs/fuse/connections 2021-02-10T11:33:35.459 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2021-02-10T11:33:35.459 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:33:35.462 DEBUG:teuthology.orchestra.run.gibba012:> sudo modprobe fuse 2021-02-10T11:33:35.805 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:33:35.806 DEBUG:teuthology.orchestra.run.gibba012:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:33:35.873 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba012.stderr:2021-02-10 11:33:35.885 7fdbc0feaf80 -1 init, newargv = 0x558ed701bb10 newargc=9 2021-02-10T11:33:35.873 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba012.stderr:ceph-fuse[129175]: starting ceph client 2021-02-10T11:33:35.878 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba012.stderr:ceph-fuse[129175]: starting fuse 2021-02-10T11:33:36.148 INFO:teuthology.orchestra.run.gibba012.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:33:36.149 INFO:teuthology.orchestra.run.gibba012.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:33:36.149 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:33:36.150 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:36.150 DEBUG:teuthology.orchestra.run.gibba012:> ls /sys/fs/fuse/connections 2021-02-10T11:33:36.483 INFO:teuthology.orchestra.run.gibba012.stdout:42 2021-02-10T11:33:36.483 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2021-02-10T11:33:36.483 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:36.484 DEBUG:teuthology.orchestra.run.gibba012:> sudo python3 -c ' 2021-02-10T11:33:36.484 DEBUG:teuthology.orchestra.run.gibba012:> import glob 2021-02-10T11:33:36.484 DEBUG:teuthology.orchestra.run.gibba012:> import re 2021-02-10T11:33:36.484 DEBUG:teuthology.orchestra.run.gibba012:> import os 2021-02-10T11:33:36.484 DEBUG:teuthology.orchestra.run.gibba012:> import subprocess 2021-02-10T11:33:36.484 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:33:36.484 DEBUG:teuthology.orchestra.run.gibba012:> def find_socket(client_name): 2021-02-10T11:33:36.484 DEBUG:teuthology.orchestra.run.gibba012:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2021-02-10T11:33:36.484 DEBUG:teuthology.orchestra.run.gibba012:> files = glob.glob(asok_path) 2021-02-10T11:33:36.484 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:33:36.484 DEBUG:teuthology.orchestra.run.gibba012:> # Given a non-glob path, it better be there 2021-02-10T11:33:36.484 DEBUG:teuthology.orchestra.run.gibba012:> if "*" not in asok_path: 2021-02-10T11:33:36.484 DEBUG:teuthology.orchestra.run.gibba012:> assert(len(files) == 1) 2021-02-10T11:33:36.485 DEBUG:teuthology.orchestra.run.gibba012:> return files[0] 2021-02-10T11:33:36.485 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:33:36.485 DEBUG:teuthology.orchestra.run.gibba012:> for f in files: 2021-02-10T11:33:36.485 DEBUG:teuthology.orchestra.run.gibba012:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:33:36.485 DEBUG:teuthology.orchestra.run.gibba012:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:33:36.485 DEBUG:teuthology.orchestra.run.gibba012:> return f 2021-02-10T11:33:36.485 DEBUG:teuthology.orchestra.run.gibba012:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:33:36.485 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:33:36.485 DEBUG:teuthology.orchestra.run.gibba012:> print(find_socket("client.0")) 2021-02-10T11:33:36.485 DEBUG:teuthology.orchestra.run.gibba012:> ' 2021-02-10T11:33:36.846 INFO:teuthology.orchestra.run.gibba012.stdout:/var/run/ceph/ceph-client.0.129175.asok 2021-02-10T11:33:36.849 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.129175.asok 2021-02-10T11:33:36.849 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:36.849 DEBUG:teuthology.orchestra.run.gibba012:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.129175.asok status 2021-02-10T11:33:37.238 INFO:teuthology.orchestra.run.gibba012.stdout:{ 2021-02-10T11:33:37.238 INFO:teuthology.orchestra.run.gibba012.stdout: "metadata": { 2021-02-10T11:33:37.238 INFO:teuthology.orchestra.run.gibba012.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:33:37.238 INFO:teuthology.orchestra.run.gibba012.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:33:37.238 INFO:teuthology.orchestra.run.gibba012.stdout: "entity_id": "0", 2021-02-10T11:33:37.238 INFO:teuthology.orchestra.run.gibba012.stdout: "hostname": "gibba012", 2021-02-10T11:33:37.238 INFO:teuthology.orchestra.run.gibba012.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2021-02-10T11:33:37.238 INFO:teuthology.orchestra.run.gibba012.stdout: "pid": "129175", 2021-02-10T11:33:37.238 INFO:teuthology.orchestra.run.gibba012.stdout: "root": "/" 2021-02-10T11:33:37.238 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:37.239 INFO:teuthology.orchestra.run.gibba012.stdout: "dentry_count": 0, 2021-02-10T11:33:37.239 INFO:teuthology.orchestra.run.gibba012.stdout: "dentry_pinned_count": 0, 2021-02-10T11:33:37.239 INFO:teuthology.orchestra.run.gibba012.stdout: "id": 8031, 2021-02-10T11:33:37.239 INFO:teuthology.orchestra.run.gibba012.stdout: "inst": { 2021-02-10T11:33:37.239 INFO:teuthology.orchestra.run.gibba012.stdout: "name": { 2021-02-10T11:33:37.239 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "client", 2021-02-10T11:33:37.239 INFO:teuthology.orchestra.run.gibba012.stdout: "num": 8031 2021-02-10T11:33:37.239 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:37.239 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": { 2021-02-10T11:33:37.239 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:33:37.239 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:0", 2021-02-10T11:33:37.239 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 3816346115 2021-02-10T11:33:37.239 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:33:37.240 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:37.240 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": { 2021-02-10T11:33:37.240 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:33:37.240 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:0", 2021-02-10T11:33:37.240 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 3816346115 2021-02-10T11:33:37.240 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:37.240 INFO:teuthology.orchestra.run.gibba012.stdout: "inst_str": "client.8031 172.21.2.112:0/3816346115", 2021-02-10T11:33:37.240 INFO:teuthology.orchestra.run.gibba012.stdout: "addr_str": "172.21.2.112:0/3816346115", 2021-02-10T11:33:37.240 INFO:teuthology.orchestra.run.gibba012.stdout: "inode_count": 1, 2021-02-10T11:33:37.240 INFO:teuthology.orchestra.run.gibba012.stdout: "mds_epoch": 39, 2021-02-10T11:33:37.240 INFO:teuthology.orchestra.run.gibba012.stdout: "osd_epoch": 55, 2021-02-10T11:33:37.240 INFO:teuthology.orchestra.run.gibba012.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:33:37.240 INFO:teuthology.orchestra.run.gibba012.stdout: "blacklisted": false 2021-02-10T11:33:37.241 INFO:teuthology.orchestra.run.gibba012.stdout:} 2021-02-10T11:33:37.242 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:37.242 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:33:37.242 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:33:37.577 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:33:37.577 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:33:37.577 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:37.577 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:33:37.924 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:33:38.350 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.346 7f47800c6700 1 Processor -- start 2021-02-10T11:33:38.351 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.346 7f47800c6700 1 -- start start 2021-02-10T11:33:38.351 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.347 7f47800c6700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f47780e5470 0x7f477807e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:38.351 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.347 7f47800c6700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f477807f300 0x7f477807f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:38.351 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.347 7f47800c6700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f477807ed80 con 0x7f47780e5070 2021-02-10T11:33:38.351 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.347 7f47800c6700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47781b4010 con 0x7f47780e5470 2021-02-10T11:33:38.351 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.347 7f47800c6700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47780d1790 con 0x7f477807f300 2021-02-10T11:33:38.351 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.347 7f477dee2700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f47780e5070 0x7f47781b3820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:49386/0 (socket says 172.21.2.112:49386) 2021-02-10T11:33:38.351 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.347 7f477dee2700 1 -- 172.21.2.112:0/3348567070 learned_addr learned my addr 172.21.2.112:0/3348567070 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:33:38.351 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.347 7f477cee0700 1 -- 172.21.2.112:0/3348567070 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1140474404 0 0) 0x7f476c0010b0 con 0x7f47780e5070 2021-02-10T11:33:38.351 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.347 7f477cee0700 1 -- 172.21.2.112:0/3348567070 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4764002fe0 con 0x7f47780e5070 2021-02-10T11:33:38.352 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.347 7f477d6e1700 1 --2- 172.21.2.112:0/3348567070 >> v2:172.21.2.120:3300/0 conn(0x7f47780e5470 0x7f477807e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:38.352 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.347 7f477e6e3700 1 --2- 172.21.2.112:0/3348567070 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f477807f300 0x7f477807f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:38.352 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.348 7f477cee0700 1 -- 172.21.2.112:0/3348567070 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 552049198 0 0) 0x7f476c0010b0 con 0x7f47780e5070 2021-02-10T11:33:38.352 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.348 7f477cee0700 1 -- 172.21.2.112:0/3348567070 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f477807f300 msgr2=0x7f477807f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:38.352 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.348 7f477cee0700 1 --2- 172.21.2.112:0/3348567070 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f477807f300 0x7f477807f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:38.352 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.348 7f477cee0700 1 -- 172.21.2.112:0/3348567070 >> v2:172.21.2.120:3300/0 conn(0x7f47780e5470 msgr2=0x7f477807e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:38.352 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.348 7f477cee0700 1 --2- 172.21.2.112:0/3348567070 >> v2:172.21.2.120:3300/0 conn(0x7f47780e5470 0x7f477807e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:38.352 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.348 7f477cee0700 1 -- 172.21.2.112:0/3348567070 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47780d19b0 con 0x7f47780e5070 2021-02-10T11:33:38.352 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.348 7f477cee0700 1 -- 172.21.2.112:0/3348567070 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f476c0015b0 con 0x7f47780e5070 2021-02-10T11:33:38.352 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.348 7f477cee0700 1 -- 172.21.2.112:0/3348567070 >> v1:172.21.2.112:6789/0 conn(0x7f47780e5070 legacy=0x7f47781b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:38.352 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.348 7f477cee0700 1 --2- 172.21.2.112:0/3348567070 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4764003b90 0x7f4764005fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:38.352 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.348 7f477cee0700 1 --2- 172.21.2.112:0/3348567070 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f476404a7c0 0x7f476404cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:38.352 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.348 7f477cee0700 1 --2- 172.21.2.112:0/3348567070 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f476404d500 0x7f476404fa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:38.353 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.348 7f477cee0700 1 -- 172.21.2.112:0/3348567070 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47640502d0 con 0x7f476404a7c0 2021-02-10T11:33:38.353 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.348 7f477cee0700 1 -- 172.21.2.112:0/3348567070 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47640504f0 con 0x7f476404d500 2021-02-10T11:33:38.353 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.348 7f477cee0700 1 -- 172.21.2.112:0/3348567070 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4764050710 con 0x7f4764003b90 2021-02-10T11:33:38.353 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.348 7f477d6e1700 1 --2- 172.21.2.112:0/3348567070 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f476404a7c0 0x7f476404cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:38.353 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.348 7f477e6e3700 1 --2- 172.21.2.112:0/3348567070 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f476404d500 0x7f476404fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:38.353 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.348 7f477d6e1700 1 -- 172.21.2.112:0/3348567070 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4764003b90 msgr2=0x7f4764005fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:38.353 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.348 7f477d6e1700 1 --2- 172.21.2.112:0/3348567070 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4764003b90 0x7f4764005fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:38.353 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.348 7f477d6e1700 1 -- 172.21.2.112:0/3348567070 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f476404d500 msgr2=0x7f476404fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:38.353 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.348 7f477d6e1700 1 --2- 172.21.2.112:0/3348567070 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f476404d500 0x7f476404fa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:38.353 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.348 7f477d6e1700 1 -- 172.21.2.112:0/3348567070 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47640509b0 con 0x7f476404a7c0 2021-02-10T11:33:38.353 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.348 7f477e6e3700 1 --2- 172.21.2.112:0/3348567070 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f476404d500 0x7f476404fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:33:38.353 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.348 7f477d6e1700 1 --2- 172.21.2.112:0/3348567070 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f476404a7c0 0x7f476404cc60 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 rx=0x7f477400c8f0 tx=0x7f477400cc90).ready entity=mon.0 client_cookie=303cbc6805f862d7 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:38.353 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.348 7f477cee0700 1 -- 172.21.2.112:0/3348567070 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f477400cdd0 con 0x7f476404a7c0 2021-02-10T11:33:38.353 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.348 7f477cee0700 1 -- 172.21.2.112:0/3348567070 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4774009310 con 0x7f476404a7c0 2021-02-10T11:33:38.354 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.348 7f477cee0700 1 -- 172.21.2.112:0/3348567070 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f47740097f0 con 0x7f476404a7c0 2021-02-10T11:33:38.354 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.348 7f47800c6700 1 -- 172.21.2.112:0/3348567070 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f476404a7c0 msgr2=0x7f476404cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:38.354 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.348 7f47800c6700 1 --2- 172.21.2.112:0/3348567070 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f476404a7c0 0x7f476404cc60 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 rx=0x7f477400c8f0 tx=0x7f477400cc90).stop 2021-02-10T11:33:38.354 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.348 7f477e6e3700 1 -- 172.21.2.112:0/3348567070 reap_dead start 2021-02-10T11:33:38.354 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.349 7f47800c6700 1 -- 172.21.2.112:0/3348567070 shutdown_connections 2021-02-10T11:33:38.354 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.349 7f47800c6700 1 -- 172.21.2.112:0/3348567070 >> 172.21.2.112:0/3348567070 conn(0x7f47780ce690 msgr2=0x7f47780e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:38.354 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.349 7f47800c6700 1 -- 172.21.2.112:0/3348567070 shutdown_connections 2021-02-10T11:33:38.354 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.349 7f47800c6700 1 -- 172.21.2.112:0/3348567070 wait complete. 2021-02-10T11:33:38.354 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.364 7f47800c6700 1 Processor -- start 2021-02-10T11:33:38.354 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.364 7f47800c6700 1 -- start start 2021-02-10T11:33:38.355 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.364 7f47800c6700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4778194000 0x7f477819f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:38.356 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.364 7f47800c6700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f477819fde0 0x7f477809bed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:38.356 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.364 7f47800c6700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f47781a01e0 0x7f47780895c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:38.356 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.364 7f47800c6700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4778194400 con 0x7f4778194000 2021-02-10T11:33:38.356 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.364 7f47800c6700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f477809c4f0 con 0x7f477819fde0 2021-02-10T11:33:38.356 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.364 7f47800c6700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4778089be0 con 0x7f47781a01e0 2021-02-10T11:33:38.356 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.364 7f477dee2700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4778194000 0x7f477819f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:38.356 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.364 7f477dee2700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4778194000 0x7f477819f680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:49936/0 (socket says 172.21.2.112:49936) 2021-02-10T11:33:38.356 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.364 7f477dee2700 1 -- 172.21.2.112:0/516585511 learned_addr learned my addr 172.21.2.112:0/516585511 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:33:38.356 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.365 7f477dee2700 1 -- 172.21.2.112:0/516585511 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f47781a01e0 msgr2=0x7f47780895c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:33:38.356 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.365 7f477dee2700 1 --2- 172.21.2.112:0/516585511 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f47781a01e0 0x7f47780895c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:38.356 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.365 7f477dee2700 1 -- 172.21.2.112:0/516585511 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f477819fde0 msgr2=0x7f477809bed0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:33:38.356 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.365 7f477dee2700 1 --2- 172.21.2.112:0/516585511 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f477819fde0 0x7f477809bed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:38.357 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.365 7f477dee2700 1 -- 172.21.2.112:0/516585511 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4778089e00 con 0x7f4778194000 2021-02-10T11:33:38.357 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.365 7f477dee2700 1 --2- 172.21.2.112:0/516585511 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4778194000 0x7f477819f680 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 rx=0x7f476c001c60 tx=0x7f476c00a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:38.357 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.365 7f476a7fc700 1 -- 172.21.2.112:0/516585511 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f476c00acc0 con 0x7f4778194000 2021-02-10T11:33:38.357 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.365 7f476a7fc700 1 -- 172.21.2.112:0/516585511 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f476c004350 con 0x7f4778194000 2021-02-10T11:33:38.357 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.365 7f476a7fc700 1 -- 172.21.2.112:0/516585511 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f476c004830 con 0x7f4778194000 2021-02-10T11:33:38.357 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.365 7f47800c6700 1 -- 172.21.2.112:0/516585511 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4778089e00 con 0x7f4778194000 2021-02-10T11:33:38.357 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.365 7f47800c6700 1 -- 172.21.2.112:0/516585511 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4778089e00 con 0x7f4778194000 2021-02-10T11:33:38.357 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.366 7f476a7fc700 1 -- 172.21.2.112:0/516585511 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f476c000dd0 con 0x7f4778194000 2021-02-10T11:33:38.357 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.366 7f476a7fc700 1 --2- 172.21.2.112:0/516585511 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f47600186c0 0x7f476001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:38.357 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.366 7f476a7fc700 1 -- 172.21.2.112:0/516585511 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f476c0325b0 con 0x7f4778194000 2021-02-10T11:33:38.357 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.366 7f477d6e1700 1 --2- 172.21.2.112:0/516585511 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f47600186c0 0x7f476001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:38.357 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.367 7f477d6e1700 1 --2- 172.21.2.112:0/516585511 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f47600186c0 0x7f476001abc0 crc :-1 s=READY pgs=633 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:38.383 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.396 7f47800c6700 1 -- 172.21.2.112:0/516585511 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f477809c710 con 0x7f4778194000 2021-02-10T11:33:38.386 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.399 7f476a7fc700 1 -- 172.21.2.112:0/516585511 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f476c004140 con 0x7f4778194000 2021-02-10T11:33:38.509 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.522 7f47800c6700 1 -- 172.21.2.112:0/516585511 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4778089090 con 0x7f4778194000 2021-02-10T11:33:38.510 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.523 7f476a7fc700 1 -- 172.21.2.112:0/516585511 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v55) v1 ==== 74+0+13996 (secure 0 0 0) 0x7f476c000da0 con 0x7f4778194000 2021-02-10T11:33:38.510 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:33:38.510 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":55,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:33:24.553170","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:33:21.523665","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":11,"pool_name":"cephfs_data","create_time":"2021-02-10 11:33:22.595892","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:33:38.511 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.524 7f47800c6700 1 -- 172.21.2.112:0/516585511 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f47600186c0 msgr2=0x7f476001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:38.511 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.524 7f47800c6700 1 --2- 172.21.2.112:0/516585511 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f47600186c0 0x7f476001abc0 crc :-1 s=READY pgs=633 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:38.511 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.524 7f47800c6700 1 -- 172.21.2.112:0/516585511 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4778194000 msgr2=0x7f477819f680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:38.512 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.524 7f47800c6700 1 --2- 172.21.2.112:0/516585511 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4778194000 0x7f477819f680 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 rx=0x7f476c001c60 tx=0x7f476c00a8d0).stop 2021-02-10T11:33:38.512 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.525 7f47800c6700 1 -- 172.21.2.112:0/516585511 shutdown_connections 2021-02-10T11:33:38.512 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.525 7f47800c6700 1 --2- 172.21.2.112:0/516585511 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f47600186c0 0x7f476001abc0 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:38.512 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.525 7f47800c6700 1 --2- 172.21.2.112:0/516585511 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f477819fde0 0x7f477809bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:38.512 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.525 7f47800c6700 1 --2- 172.21.2.112:0/516585511 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f47781a01e0 0x7f47780895c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:38.512 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.525 7f47800c6700 1 --2- 172.21.2.112:0/516585511 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4778194000 0x7f477819f680 unknown :-1 s=CLOSED pgs=1272 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:38.512 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.525 7f47800c6700 1 -- 172.21.2.112:0/516585511 >> 172.21.2.112:0/516585511 conn(0x7f4778092c20 msgr2=0x7f4778195070 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:38.513 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.525 7f47800c6700 1 -- 172.21.2.112:0/516585511 shutdown_connections 2021-02-10T11:33:38.513 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.525 7f47800c6700 1 -- 172.21.2.112:0/516585511 wait complete. 2021-02-10T11:33:38.520 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:13:57.508758', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 10, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:33:21.523665', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '54', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 11, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:33:22.595892', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '55', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:33:38.521 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:33:38.953 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.948 7fdbc520f700 1 Processor -- start 2021-02-10T11:33:38.954 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.948 7fdbc520f700 1 -- start start 2021-02-10T11:33:38.954 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.949 7fdbc520f700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fdbc01b0470 0x7fdbc01b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:38.954 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.949 7fdbc520f700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbc00f0b50 con 0x7fdbc00f20e0 2021-02-10T11:33:38.954 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.949 7fdbc520f700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbc00f24e0 con 0x7fdbc00ef610 2021-02-10T11:33:38.954 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.949 7fdbc520f700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbc00e5ea0 con 0x7fdbc01b0470 2021-02-10T11:33:38.954 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.949 7fdbbf59e700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fdbc00f20e0 0x7fdbc00f0200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:49408/0 (socket says 172.21.2.112:49408) 2021-02-10T11:33:38.954 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.949 7fdbbf59e700 1 -- 172.21.2.112:0/1811681584 learned_addr learned my addr 172.21.2.112:0/1811681584 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:33:38.954 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.949 7fdbbdd9b700 1 -- 172.21.2.112:0/1811681584 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3488851058 0 0) 0x7fdbb8001500 con 0x7fdbc00f20e0 2021-02-10T11:33:38.954 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.950 7fdbbdd9b700 1 -- 172.21.2.112:0/1811681584 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbac002fe0 con 0x7fdbc00f20e0 2021-02-10T11:33:38.954 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.950 7fdbbed9d700 1 --2- 172.21.2.112:0/1811681584 >> v2:172.21.2.120:3300/0 conn(0x7fdbc01b0470 0x7fdbc01b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:38.955 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.950 7fdbbdd9b700 1 -- 172.21.2.112:0/1811681584 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4208256421 0 0) 0x7fdbb8001500 con 0x7fdbc00f20e0 2021-02-10T11:33:38.955 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.950 7fdbbdd9b700 1 -- 172.21.2.112:0/1811681584 >> v2:172.21.2.120:3300/0 conn(0x7fdbc01b0470 msgr2=0x7fdbc01b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:38.955 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.950 7fdbbdd9b700 1 --2- 172.21.2.112:0/1811681584 >> v2:172.21.2.120:3300/0 conn(0x7fdbc01b0470 0x7fdbc01b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:38.955 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.950 7fdbbdd9b700 1 -- 172.21.2.112:0/1811681584 >> v1:172.21.2.120:6789/0 conn(0x7fdbc00ef610 legacy=0x7fdbc00efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:38.955 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.950 7fdbbdd9b700 1 -- 172.21.2.112:0/1811681584 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbc00e52e0 con 0x7fdbc00f20e0 2021-02-10T11:33:38.955 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.950 7fdbbdd9b700 1 -- 172.21.2.112:0/1811681584 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7fdbb8001a00 con 0x7fdbc00f20e0 2021-02-10T11:33:38.955 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.950 7fdbbdd9b700 1 -- 172.21.2.112:0/1811681584 >> v1:172.21.2.112:6789/0 conn(0x7fdbc00f20e0 legacy=0x7fdbc00f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:38.955 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.950 7fdbbdd9b700 1 --2- 172.21.2.112:0/1811681584 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdbac003af0 0x7fdbac005f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:38.955 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.950 7fdbbdd9b700 1 --2- 172.21.2.112:0/1811681584 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdbac006820 0x7fdbac008d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:38.955 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.950 7fdbbdd9b700 1 --2- 172.21.2.112:0/1811681584 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fdbac009620 0x7fdbac00bad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:38.955 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.950 7fdbbdd9b700 1 -- 172.21.2.112:0/1811681584 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbac00c320 con 0x7fdbac006820 2021-02-10T11:33:38.955 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.950 7fdbbdd9b700 1 -- 172.21.2.112:0/1811681584 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbac00c540 con 0x7fdbac009620 2021-02-10T11:33:38.955 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.950 7fdbbdd9b700 1 -- 172.21.2.112:0/1811681584 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbac00c760 con 0x7fdbac003af0 2021-02-10T11:33:38.956 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.950 7fdbbe59c700 1 --2- 172.21.2.112:0/1811681584 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdbac006820 0x7fdbac008d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:38.956 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.950 7fdbbed9d700 1 --2- 172.21.2.112:0/1811681584 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdbac003af0 0x7fdbac005f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:38.956 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.950 7fdbbf59e700 1 --2- 172.21.2.112:0/1811681584 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fdbac009620 0x7fdbac00bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:38.956 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.950 7fdbbe59c700 1 -- 172.21.2.112:0/1811681584 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdbac003af0 msgr2=0x7fdbac005f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:38.956 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.950 7fdbbe59c700 1 --2- 172.21.2.112:0/1811681584 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdbac003af0 0x7fdbac005f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:38.956 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.950 7fdbbe59c700 1 -- 172.21.2.112:0/1811681584 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fdbac009620 msgr2=0x7fdbac00bad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:38.956 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.950 7fdbbe59c700 1 --2- 172.21.2.112:0/1811681584 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fdbac009620 0x7fdbac00bad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:38.956 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.950 7fdbbe59c700 1 -- 172.21.2.112:0/1811681584 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbac00ca00 con 0x7fdbac006820 2021-02-10T11:33:38.956 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.950 7fdbbf59e700 1 -- 172.21.2.112:0/1811681584 reap_dead start 2021-02-10T11:33:38.956 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.950 7fdbbed9d700 1 --2- 172.21.2.112:0/1811681584 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdbac003af0 0x7fdbac005f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:33:38.956 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.951 7fdbbe59c700 1 --2- 172.21.2.112:0/1811681584 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdbac006820 0x7fdbac008d20 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 rx=0x7fdbb0006ad0 tx=0x7fdbb004db50).ready entity=mon.0 client_cookie=5de8779d1a6992f4 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:38.956 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.951 7fdbbdd9b700 1 -- 172.21.2.112:0/1811681584 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdbb0006680 con 0x7fdbac006820 2021-02-10T11:33:38.956 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.951 7fdbbdd9b700 1 -- 172.21.2.112:0/1811681584 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbb0002280 con 0x7fdbac006820 2021-02-10T11:33:38.956 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.951 7fdbbdd9b700 1 -- 172.21.2.112:0/1811681584 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdbb00026e0 con 0x7fdbac006820 2021-02-10T11:33:38.957 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.951 7fdbc520f700 1 -- 172.21.2.112:0/1811681584 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdbac006820 msgr2=0x7fdbac008d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:38.957 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.951 7fdbc520f700 1 --2- 172.21.2.112:0/1811681584 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdbac006820 0x7fdbac008d20 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 rx=0x7fdbb0006ad0 tx=0x7fdbb004db50).stop 2021-02-10T11:33:38.957 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.952 7fdbc520f700 1 -- 172.21.2.112:0/1811681584 shutdown_connections 2021-02-10T11:33:38.957 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.952 7fdbc520f700 1 --2- 172.21.2.112:0/1811681584 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdbac006820 0x7fdbac008d20 unknown :-1 s=CLOSED pgs=1274 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:38.957 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.952 7fdbc520f700 1 -- 172.21.2.112:0/1811681584 >> 172.21.2.112:0/1811681584 conn(0x7fdbc00ce3e0 msgr2=0x7fdbc00e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:38.957 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.952 7fdbc520f700 1 -- 172.21.2.112:0/1811681584 shutdown_connections 2021-02-10T11:33:38.957 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.952 7fdbc520f700 1 -- 172.21.2.112:0/1811681584 wait complete. 2021-02-10T11:33:38.957 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.966 7fdbc520f700 1 Processor -- start 2021-02-10T11:33:38.957 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.967 7fdbc520f700 1 -- start start 2021-02-10T11:33:38.959 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.967 7fdbc520f700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdbc00b7db0 0x7fdbc00b8260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:38.959 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.967 7fdbc520f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdbc00b8880 0x7fdbc01a7450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:38.959 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.967 7fdbc520f700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fdbc00b8c80 0x7fdbc01a7a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:38.959 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.967 7fdbc520f700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc00b79b0 con 0x7fdbc00b8880 2021-02-10T11:33:38.959 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.967 7fdbc520f700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc01a8090 con 0x7fdbc00b8c80 2021-02-10T11:33:38.959 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.967 7fdbc520f700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc01a82b0 con 0x7fdbc00b7db0 2021-02-10T11:33:38.959 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.967 7fdbbe59c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdbc00b8880 0x7fdbc01a7450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:38.959 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.967 7fdbbe59c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdbc00b8880 0x7fdbc01a7450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:49954/0 (socket says 172.21.2.112:49954) 2021-02-10T11:33:38.959 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.967 7fdbbe59c700 1 -- 172.21.2.112:0/3126310899 learned_addr learned my addr 172.21.2.112:0/3126310899 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:33:38.959 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.968 7fdbbed9d700 1 --2- 172.21.2.112:0/3126310899 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdbc00b7db0 0x7fdbc00b8260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:38.959 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.968 7fdbbf59e700 1 --2- 172.21.2.112:0/3126310899 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fdbc00b8c80 0x7fdbc01a7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:38.960 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.968 7fdbbe59c700 1 -- 172.21.2.112:0/3126310899 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdbc00b7db0 msgr2=0x7fdbc00b8260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:38.960 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.968 7fdbbe59c700 1 --2- 172.21.2.112:0/3126310899 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdbc00b7db0 0x7fdbc00b8260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:38.960 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.968 7fdbbe59c700 1 -- 172.21.2.112:0/3126310899 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fdbc00b8c80 msgr2=0x7fdbc01a7a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:38.960 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.968 7fdbbe59c700 1 --2- 172.21.2.112:0/3126310899 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fdbc00b8c80 0x7fdbc01a7a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:38.960 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.968 7fdbbe59c700 1 -- 172.21.2.112:0/3126310899 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbc0190c10 con 0x7fdbc00b8880 2021-02-10T11:33:38.960 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.968 7fdbbed9d700 1 --2- 172.21.2.112:0/3126310899 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdbc00b7db0 0x7fdbc00b8260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:33:38.960 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.968 7fdbbe59c700 1 --2- 172.21.2.112:0/3126310899 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdbc00b8880 0x7fdbc01a7450 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 rx=0x7fdbb0002490 tx=0x7fdbb0002830).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:38.960 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.968 7fdbb77fe700 1 -- 172.21.2.112:0/3126310899 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdbb0006030 con 0x7fdbc00b8880 2021-02-10T11:33:38.960 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.968 7fdbb77fe700 1 -- 172.21.2.112:0/3126310899 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbb0006330 con 0x7fdbc00b8880 2021-02-10T11:33:38.960 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.968 7fdbb77fe700 1 -- 172.21.2.112:0/3126310899 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdbb0005720 con 0x7fdbc00b8880 2021-02-10T11:33:38.960 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.968 7fdbc520f700 1 -- 172.21.2.112:0/3126310899 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbc0190c10 con 0x7fdbc00b8880 2021-02-10T11:33:38.960 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.968 7fdbc520f700 1 -- 172.21.2.112:0/3126310899 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbc0190c10 con 0x7fdbc00b8880 2021-02-10T11:33:38.960 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.969 7fdbb77fe700 1 -- 172.21.2.112:0/3126310899 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fdbb0006030 con 0x7fdbc00b8880 2021-02-10T11:33:38.961 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.969 7fdbb77fe700 1 --2- 172.21.2.112:0/3126310899 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fdba4018730 0x7fdba401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:38.961 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.969 7fdbb77fe700 1 -- 172.21.2.112:0/3126310899 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fdbb0076680 con 0x7fdbc00b8880 2021-02-10T11:33:38.961 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.969 7fdbbed9d700 1 --2- 172.21.2.112:0/3126310899 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fdba4018730 0x7fdba401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:38.961 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.969 7fdbbed9d700 1 --2- 172.21.2.112:0/3126310899 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fdba4018730 0x7fdba401abc0 crc :-1 s=READY pgs=634 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:38.986 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:38.999 7fdbc520f700 1 -- 172.21.2.112:0/3126310899 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbc0105b30 con 0x7fdbc00b8880 2021-02-10T11:33:38.988 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:39.001 7fdbb77fe700 1 -- 172.21.2.112:0/3126310899 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fdbb0002c20 con 0x7fdbc00b8880 2021-02-10T11:33:39.119 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:39.132 7fdbc520f700 1 -- 172.21.2.112:0/3126310899 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7fdbc00f0b50 con 0x7fdbc00b8880 2021-02-10T11:33:39.120 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:39.133 7fdbb77fe700 1 -- 172.21.2.112:0/3126310899 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v55) v1 ==== 92+0+10 (secure 0 0 0) 0x7fdbb0006030 con 0x7fdbc00b8880 2021-02-10T11:33:39.120 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:33:39.121 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:39.135 7fdbc520f700 1 -- 172.21.2.112:0/3126310899 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fdba4018730 msgr2=0x7fdba401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:39.122 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:39.135 7fdbc520f700 1 --2- 172.21.2.112:0/3126310899 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fdba4018730 0x7fdba401abc0 crc :-1 s=READY pgs=634 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:39.122 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:39.135 7fdbc520f700 1 -- 172.21.2.112:0/3126310899 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdbc00b8880 msgr2=0x7fdbc01a7450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:39.122 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:39.135 7fdbc520f700 1 --2- 172.21.2.112:0/3126310899 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdbc00b8880 0x7fdbc01a7450 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 rx=0x7fdbb0002490 tx=0x7fdbb0002830).stop 2021-02-10T11:33:39.122 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:39.135 7fdbc520f700 1 -- 172.21.2.112:0/3126310899 shutdown_connections 2021-02-10T11:33:39.122 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:39.135 7fdbc520f700 1 --2- 172.21.2.112:0/3126310899 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fdba4018730 0x7fdba401abc0 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:39.122 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:39.135 7fdbc520f700 1 --2- 172.21.2.112:0/3126310899 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fdbc00b8c80 0x7fdbc01a7a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:39.122 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:39.135 7fdbc520f700 1 --2- 172.21.2.112:0/3126310899 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdbc00b8880 0x7fdbc01a7450 unknown :-1 s=CLOSED pgs=1275 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:39.122 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:39.135 7fdbc520f700 1 --2- 172.21.2.112:0/3126310899 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdbc00b7db0 0x7fdbc00b8260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:39.122 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:39.135 7fdbc520f700 1 -- 172.21.2.112:0/3126310899 >> 172.21.2.112:0/3126310899 conn(0x7fdbc0094e10 msgr2=0x7fdbc00a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:39.123 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:39.135 7fdbc520f700 1 -- 172.21.2.112:0/3126310899 shutdown_connections 2021-02-10T11:33:39.123 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:39.135 7fdbc520f700 1 -- 172.21.2.112:0/3126310899 wait complete. 2021-02-10T11:33:39.130 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:33:39.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:39.556 7f53bc359700 1 Processor -- start 2021-02-10T11:33:39.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:39.556 7f53bc359700 1 -- start start 2021-02-10T11:33:39.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:39.557 7f53bc359700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f53b41b0870 0x7f53b40f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:39.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:39.557 7f53bc359700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f53b40efe00 0x7f53b40f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:39.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:39.557 7f53bc359700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53b41b0c70 con 0x7f53b41b0470 2021-02-10T11:33:39.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:39.557 7f53bc359700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53b40e5ea0 con 0x7f53b41b0870 2021-02-10T11:33:39.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:39.557 7f53bc359700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53b40f0b50 con 0x7f53b40efe00 2021-02-10T11:33:39.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:39.557 7f53b9974700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f53b41b0870 0x7f53b40f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:39.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:39.557 7f53ba175700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f53b41b0470 0x7f53b40ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:49424/0 (socket says 172.21.2.112:49424) 2021-02-10T11:33:39.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:39.557 7f53ba175700 1 -- 172.21.2.112:0/218809068 learned_addr learned my addr 172.21.2.112:0/218809068 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:33:39.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:39.557 7f53ba976700 1 --2- 172.21.2.112:0/218809068 >> v2:172.21.2.120:3300/0 conn(0x7f53b40efe00 0x7f53b40f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:39.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:39.557 7f53b9974700 1 -- 172.21.2.112:0/218809068 >> v2:172.21.2.120:3300/0 conn(0x7f53b40efe00 msgr2=0x7f53b40f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:39.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:39.557 7f53b9974700 1 --2- 172.21.2.112:0/218809068 >> v2:172.21.2.120:3300/0 conn(0x7f53b40efe00 0x7f53b40f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:39.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:39.557 7f53b9974700 1 -- 172.21.2.112:0/218809068 >> v1:172.21.2.112:6789/0 conn(0x7f53b41b0470 legacy=0x7f53b40ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:39.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:39.558 7f53b9974700 1 -- 172.21.2.112:0/218809068 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53b40f0820 con 0x7f53b41b0870 2021-02-10T11:33:39.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:39.558 7f53ba976700 1 --2- 172.21.2.112:0/218809068 >> v2:172.21.2.120:3300/0 conn(0x7f53b40efe00 0x7f53b40f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:33:39.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:39.558 7f53b9974700 1 --2- 172.21.2.112:0/218809068 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f53b41b0870 0x7f53b40f20e0 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 rx=0x7f53a8006650 tx=0x7f53a8009a80).ready entity=mon.0 client_cookie=a6c374ae393a906 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:39.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:39.558 7f53b9173700 1 -- 172.21.2.112:0/218809068 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f53a80009b0 con 0x7f53b41b0870 2021-02-10T11:33:39.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:39.558 7f53b9173700 1 -- 172.21.2.112:0/218809068 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53a800c0c0 con 0x7f53b41b0870 2021-02-10T11:33:39.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:39.558 7f53b9173700 1 -- 172.21.2.112:0/218809068 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f53a800c520 con 0x7f53b41b0870 2021-02-10T11:33:39.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:39.558 7f53bc359700 1 -- 172.21.2.112:0/218809068 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f53b41b0870 msgr2=0x7f53b40f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:39.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:39.558 7f53bc359700 1 --2- 172.21.2.112:0/218809068 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f53b41b0870 0x7f53b40f20e0 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 rx=0x7f53a8006650 tx=0x7f53a8009a80).stop 2021-02-10T11:33:39.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:39.559 7f53bc359700 1 -- 172.21.2.112:0/218809068 shutdown_connections 2021-02-10T11:33:39.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:39.559 7f53bc359700 1 --2- 172.21.2.112:0/218809068 >> v2:172.21.2.120:3300/0 conn(0x7f53b40efe00 0x7f53b40f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:39.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:39.559 7f53bc359700 1 --2- 172.21.2.112:0/218809068 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f53b41b0870 0x7f53b40f20e0 unknown :-1 s=CLOSED pgs=1277 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:39.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:39.559 7f53bc359700 1 -- 172.21.2.112:0/218809068 >> 172.21.2.112:0/218809068 conn(0x7f53b40ce3e0 msgr2=0x7f53b40e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:39.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:39.559 7f53bc359700 1 -- 172.21.2.112:0/218809068 shutdown_connections 2021-02-10T11:33:39.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:39.559 7f53bc359700 1 -- 172.21.2.112:0/218809068 wait complete. 2021-02-10T11:33:39.564 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:39.573 7f53bc359700 1 Processor -- start 2021-02-10T11:33:39.564 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:39.573 7f53bc359700 1 -- start start 2021-02-10T11:33:39.564 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:39.574 7f53bc359700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f53b409f140 0x7f53b40c1940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:39.564 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:39.574 7f53bc359700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f53b40c1f60 0x7f53b409e300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:39.564 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:39.574 7f53bc359700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f53b409e920 0x7f53b40c2960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:39.565 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:39.574 7f53bc359700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53b40c23e0 con 0x7f53b40c1f60 2021-02-10T11:33:39.565 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:39.574 7f53bc359700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53b409ed20 con 0x7f53b409f140 2021-02-10T11:33:39.565 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:39.574 7f53bc359700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53b4105d10 con 0x7f53b409e920 2021-02-10T11:33:39.565 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:39.574 7f53b9974700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f53b40c1f60 0x7f53b409e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:39.565 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:39.574 7f53b9974700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f53b40c1f60 0x7f53b409e300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:49966/0 (socket says 172.21.2.112:49966) 2021-02-10T11:33:39.566 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:39.574 7f53b9974700 1 -- 172.21.2.112:0/1223485036 learned_addr learned my addr 172.21.2.112:0/1223485036 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:33:39.566 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:39.574 7f53b9974700 1 -- 172.21.2.112:0/1223485036 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f53b409e920 msgr2=0x7f53b40c2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:39.566 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:39.574 7f53b9974700 1 --2- 172.21.2.112:0/1223485036 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f53b409e920 0x7f53b40c2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:39.566 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:39.574 7f53b9974700 1 -- 172.21.2.112:0/1223485036 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f53b409f140 msgr2=0x7f53b40c1940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:39.566 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:39.574 7f53b9974700 1 --2- 172.21.2.112:0/1223485036 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f53b409f140 0x7f53b40c1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:39.566 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:39.574 7f53b9974700 1 -- 172.21.2.112:0/1223485036 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53b41a4910 con 0x7f53b40c1f60 2021-02-10T11:33:39.566 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:39.574 7f53b9974700 1 --2- 172.21.2.112:0/1223485036 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f53b40c1f60 0x7f53b409e300 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 rx=0x7f53a8009650 tx=0x7f53a800b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:39.566 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:39.575 7f53a6ffd700 1 -- 172.21.2.112:0/1223485036 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f53a800bd50 con 0x7f53b40c1f60 2021-02-10T11:33:39.566 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:39.575 7f53a6ffd700 1 -- 172.21.2.112:0/1223485036 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53a8006030 con 0x7f53b40c1f60 2021-02-10T11:33:39.566 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:39.575 7f53bc359700 1 -- 172.21.2.112:0/1223485036 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53b41a4910 con 0x7f53b40c1f60 2021-02-10T11:33:39.566 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:39.575 7f53bc359700 1 -- 172.21.2.112:0/1223485036 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53b40c3270 con 0x7f53b40c1f60 2021-02-10T11:33:39.567 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:39.575 7f53a6ffd700 1 -- 172.21.2.112:0/1223485036 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f53a8005560 con 0x7f53b40c1f60 2021-02-10T11:33:39.567 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:39.576 7f53a6ffd700 1 -- 172.21.2.112:0/1223485036 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f53a80052e0 con 0x7f53b40c1f60 2021-02-10T11:33:39.567 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:39.576 7f53a6ffd700 1 --2- 172.21.2.112:0/1223485036 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f539c018700 0x7f539c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:39.567 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:39.576 7f53a6ffd700 1 -- 172.21.2.112:0/1223485036 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f53a8032780 con 0x7f53b40c1f60 2021-02-10T11:33:39.567 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:39.576 7f53ba175700 1 --2- 172.21.2.112:0/1223485036 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f539c018700 0x7f539c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:39.567 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:39.576 7f53ba175700 1 --2- 172.21.2.112:0/1223485036 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f539c018700 0x7f539c01abc0 crc :-1 s=READY pgs=635 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:39.594 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:39.607 7f53bc359700 1 -- 172.21.2.112:0/1223485036 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53b40f0b50 con 0x7f53b40c1f60 2021-02-10T11:33:39.597 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:39.610 7f53a6ffd700 1 -- 172.21.2.112:0/1223485036 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f53a80052e0 con 0x7f53b40c1f60 2021-02-10T11:33:39.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:39.737 7f53bc359700 1 -- 172.21.2.112:0/1223485036 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f53b40f0b50 con 0x7f53b40c1f60 2021-02-10T11:33:39.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:39.738 7f53a6ffd700 1 -- 172.21.2.112:0/1223485036 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v55) v1 ==== 104+0+10 (secure 0 0 0) 0x7f53a8006030 con 0x7f53b40c1f60 2021-02-10T11:33:39.725 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:33:39.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:39.740 7f53bc359700 1 -- 172.21.2.112:0/1223485036 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f539c018700 msgr2=0x7f539c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:39.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:39.740 7f53bc359700 1 --2- 172.21.2.112:0/1223485036 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f539c018700 0x7f539c01abc0 crc :-1 s=READY pgs=635 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:39.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:39.740 7f53bc359700 1 -- 172.21.2.112:0/1223485036 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f53b40c1f60 msgr2=0x7f53b409e300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:39.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:39.740 7f53bc359700 1 --2- 172.21.2.112:0/1223485036 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f53b40c1f60 0x7f53b409e300 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 rx=0x7f53a8009650 tx=0x7f53a800b960).stop 2021-02-10T11:33:39.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:39.740 7f53bc359700 1 -- 172.21.2.112:0/1223485036 shutdown_connections 2021-02-10T11:33:39.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:39.740 7f53bc359700 1 --2- 172.21.2.112:0/1223485036 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f539c018700 0x7f539c01abc0 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:39.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:39.740 7f53bc359700 1 --2- 172.21.2.112:0/1223485036 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f53b409e920 0x7f53b40c2960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:39.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:39.740 7f53bc359700 1 --2- 172.21.2.112:0/1223485036 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f53b40c1f60 0x7f53b409e300 unknown :-1 s=CLOSED pgs=1278 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:39.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:39.740 7f53bc359700 1 --2- 172.21.2.112:0/1223485036 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f53b409f140 0x7f53b40c1940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:39.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:39.740 7f53bc359700 1 -- 172.21.2.112:0/1223485036 >> 172.21.2.112:0/1223485036 conn(0x7f53b409fc10 msgr2=0x7f53b40b3250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:39.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:39.740 7f53bc359700 1 -- 172.21.2.112:0/1223485036 shutdown_connections 2021-02-10T11:33:39.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:39.740 7f53bc359700 1 -- 172.21.2.112:0/1223485036 wait complete. 2021-02-10T11:33:39.735 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:33:40.166 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.162 7f8bf8c0d700 1 Processor -- start 2021-02-10T11:33:40.166 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.162 7f8bf8c0d700 1 -- start start 2021-02-10T11:33:40.166 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.162 7f8bf8c0d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f8bf41b0870 0x7f8bf40f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:40.166 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.162 7f8bf8c0d700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bf41b0c70 con 0x7f8bf41b0470 2021-02-10T11:33:40.166 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.162 7f8bf8c0d700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bf40f0b50 con 0x7f8bf40efe00 2021-02-10T11:33:40.166 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.162 7f8bf8c0d700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8bf40e5ea0 con 0x7f8bf41b0870 2021-02-10T11:33:40.166 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.162 7f8bf205d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f8bf41b0870 0x7f8bf40f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:40.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.163 7f8bf205d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f8bf41b0870 0x7f8bf40f20e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:49974/0 (socket says 172.21.2.112:49974) 2021-02-10T11:33:40.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.163 7f8bf205d700 1 -- 172.21.2.112:0/406613199 learned_addr learned my addr 172.21.2.112:0/406613199 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:33:40.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.163 7f8bf205d700 1 -- 172.21.2.112:0/406613199 >> v1:172.21.2.120:6789/0 conn(0x7f8bf40efe00 legacy=0x7f8bf40f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:40.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.163 7f8bf205d700 1 -- 172.21.2.112:0/406613199 >> v1:172.21.2.112:6789/0 conn(0x7f8bf41b0470 legacy=0x7f8bf40ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:40.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.163 7f8bf205d700 1 -- 172.21.2.112:0/406613199 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bf40e52e0 con 0x7f8bf41b0870 2021-02-10T11:33:40.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.163 7f8bf205d700 1 --2- 172.21.2.112:0/406613199 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8bf41b0870 0x7f8bf40f20e0 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 rx=0x7f8be4006700 tx=0x7f8be4009b80).ready entity=mon.0 client_cookie=de158808b6d961e5 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:40.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.163 7f8bf185c700 1 -- 172.21.2.112:0/406613199 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8be400c0c0 con 0x7f8bf41b0870 2021-02-10T11:33:40.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.164 7f8bf185c700 1 -- 172.21.2.112:0/406613199 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8be400c440 con 0x7f8bf41b0870 2021-02-10T11:33:40.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.164 7f8bf185c700 1 -- 172.21.2.112:0/406613199 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8be400c920 con 0x7f8bf41b0870 2021-02-10T11:33:40.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.164 7f8bf8c0d700 1 -- 172.21.2.112:0/406613199 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8bf41b0870 msgr2=0x7f8bf40f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:40.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.164 7f8bf8c0d700 1 --2- 172.21.2.112:0/406613199 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8bf41b0870 0x7f8bf40f20e0 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 rx=0x7f8be4006700 tx=0x7f8be4009b80).stop 2021-02-10T11:33:40.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.164 7f8bf8c0d700 1 -- 172.21.2.112:0/406613199 shutdown_connections 2021-02-10T11:33:40.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.164 7f8bf8c0d700 1 --2- 172.21.2.112:0/406613199 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8bf41b0870 0x7f8bf40f20e0 unknown :-1 s=CLOSED pgs=1280 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:40.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.164 7f8bf8c0d700 1 -- 172.21.2.112:0/406613199 >> 172.21.2.112:0/406613199 conn(0x7f8bf40ce3e0 msgr2=0x7f8bf40e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:40.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.164 7f8bf8c0d700 1 -- 172.21.2.112:0/406613199 shutdown_connections 2021-02-10T11:33:40.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.164 7f8bf8c0d700 1 -- 172.21.2.112:0/406613199 wait complete. 2021-02-10T11:33:40.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.179 7f8bf8c0d700 1 Processor -- start 2021-02-10T11:33:40.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.179 7f8bf8c0d700 1 -- start start 2021-02-10T11:33:40.169 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.179 7f8bf8c0d700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8bf4080810 0x7f8bf40bf340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:40.169 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.179 7f8bf8c0d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8bf407fef0 0x7f8bf40b0100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:40.169 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.179 7f8bf8c0d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8bf40802f0 0x7f8bf40a6650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:40.169 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.179 7f8bf8c0d700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bf40a5e70 con 0x7f8bf40802f0 2021-02-10T11:33:40.169 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.179 7f8bf8c0d700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bf40a60d0 con 0x7f8bf4080810 2021-02-10T11:33:40.170 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.179 7f8bf8c0d700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bf40b0720 con 0x7f8bf407fef0 2021-02-10T11:33:40.170 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.180 7f8bf305f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8bf40802f0 0x7f8bf40a6650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:40.170 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.180 7f8bf305f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8bf40802f0 0x7f8bf40a6650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:49982/0 (socket says 172.21.2.112:49982) 2021-02-10T11:33:40.170 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.180 7f8bf305f700 1 -- 172.21.2.112:0/4139252441 learned_addr learned my addr 172.21.2.112:0/4139252441 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:33:40.170 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.180 7f8bf305f700 1 -- 172.21.2.112:0/4139252441 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8bf407fef0 msgr2=0x7f8bf40b0100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:40.170 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.180 7f8bf305f700 1 --2- 172.21.2.112:0/4139252441 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8bf407fef0 0x7f8bf40b0100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:40.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.180 7f8bf305f700 1 -- 172.21.2.112:0/4139252441 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8bf4080810 msgr2=0x7f8bf40bf340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:40.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.180 7f8bf305f700 1 --2- 172.21.2.112:0/4139252441 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8bf4080810 0x7f8bf40bf340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:40.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.180 7f8bf305f700 1 -- 172.21.2.112:0/4139252441 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bf40868c0 con 0x7f8bf40802f0 2021-02-10T11:33:40.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.180 7f8bf305f700 1 --2- 172.21.2.112:0/4139252441 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8bf40802f0 0x7f8bf40a6650 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 rx=0x7f8bec006ed0 tx=0x7f8bec00b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:40.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.180 7f8beb7fe700 1 -- 172.21.2.112:0/4139252441 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8bec00bd50 con 0x7f8bf40802f0 2021-02-10T11:33:40.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.181 7f8beb7fe700 1 -- 172.21.2.112:0/4139252441 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bec006020 con 0x7f8bf40802f0 2021-02-10T11:33:40.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.181 7f8beb7fe700 1 -- 172.21.2.112:0/4139252441 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8bec0054b0 con 0x7f8bf40802f0 2021-02-10T11:33:40.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.181 7f8bf8c0d700 1 -- 172.21.2.112:0/4139252441 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bf40868c0 con 0x7f8bf40802f0 2021-02-10T11:33:40.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.181 7f8bf8c0d700 1 -- 172.21.2.112:0/4139252441 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bf40a6f60 con 0x7f8bf40802f0 2021-02-10T11:33:40.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.181 7f8beb7fe700 1 -- 172.21.2.112:0/4139252441 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8bec006200 con 0x7f8bf40802f0 2021-02-10T11:33:40.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.181 7f8beb7fe700 1 --2- 172.21.2.112:0/4139252441 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8bd80186c0 0x7f8bd801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:40.172 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.181 7f8beb7fe700 1 -- 172.21.2.112:0/4139252441 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f8bec032570 con 0x7f8bf40802f0 2021-02-10T11:33:40.172 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.182 7f8bf285e700 1 --2- 172.21.2.112:0/4139252441 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8bd80186c0 0x7f8bd801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:40.172 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.182 7f8bf285e700 1 --2- 172.21.2.112:0/4139252441 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8bd80186c0 0x7f8bd801abc0 crc :-1 s=READY pgs=636 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:40.200 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.212 7f8bf8c0d700 1 -- 172.21.2.112:0/4139252441 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bf40a6f60 con 0x7f8bf40802f0 2021-02-10T11:33:40.202 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.215 7f8beb7fe700 1 -- 172.21.2.112:0/4139252441 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8bec000bc0 con 0x7f8bf40802f0 2021-02-10T11:33:40.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.343 7f8bf8c0d700 1 -- 172.21.2.112:0/4139252441 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f8bf40a6f60 con 0x7f8bf40802f0 2021-02-10T11:33:40.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.343 7f8beb7fe700 1 -- 172.21.2.112:0/4139252441 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v55) v1 ==== 100+0+10 (secure 0 0 0) 0x7f8bec00bd50 con 0x7f8bf40802f0 2021-02-10T11:33:40.331 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:33:40.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.345 7f8bf8c0d700 1 -- 172.21.2.112:0/4139252441 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8bd80186c0 msgr2=0x7f8bd801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:40.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.345 7f8bf8c0d700 1 --2- 172.21.2.112:0/4139252441 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8bd80186c0 0x7f8bd801abc0 crc :-1 s=READY pgs=636 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:40.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.345 7f8bf8c0d700 1 -- 172.21.2.112:0/4139252441 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8bf40802f0 msgr2=0x7f8bf40a6650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:40.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.346 7f8bf8c0d700 1 --2- 172.21.2.112:0/4139252441 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8bf40802f0 0x7f8bf40a6650 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 rx=0x7f8bec006ed0 tx=0x7f8bec00b960).stop 2021-02-10T11:33:40.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.346 7f8bf8c0d700 1 -- 172.21.2.112:0/4139252441 shutdown_connections 2021-02-10T11:33:40.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.346 7f8bf8c0d700 1 --2- 172.21.2.112:0/4139252441 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8bd80186c0 0x7f8bd801abc0 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:40.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.346 7f8bf8c0d700 1 --2- 172.21.2.112:0/4139252441 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8bf40802f0 0x7f8bf40a6650 unknown :-1 s=CLOSED pgs=1281 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:40.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.346 7f8bf8c0d700 1 --2- 172.21.2.112:0/4139252441 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8bf407fef0 0x7f8bf40b0100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:40.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.346 7f8bf8c0d700 1 --2- 172.21.2.112:0/4139252441 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8bf4080810 0x7f8bf40bf340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:40.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.346 7f8bf8c0d700 1 -- 172.21.2.112:0/4139252441 >> 172.21.2.112:0/4139252441 conn(0x7f8bf40859f0 msgr2=0x7f8bf40c3bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:40.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.346 7f8bf8c0d700 1 -- 172.21.2.112:0/4139252441 shutdown_connections 2021-02-10T11:33:40.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.346 7f8bf8c0d700 1 -- 172.21.2.112:0/4139252441 wait complete. 2021-02-10T11:33:40.342 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:33:40.342 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:33:40.769 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.765 7f9ef7ff6700 1 Processor -- start 2021-02-10T11:33:40.769 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.766 7f9ef7ff6700 1 -- start start 2021-02-10T11:33:40.769 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.766 7f9ef7ff6700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f9ef00e5470 0x7f9ef007e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:40.770 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.766 7f9ef7ff6700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ef007ed80 con 0x7f9ef007f300 2021-02-10T11:33:40.770 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.766 7f9ef7ff6700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ef01b4010 con 0x7f9ef00e5070 2021-02-10T11:33:40.770 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.766 7f9ef7ff6700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ef00d1790 con 0x7f9ef00e5470 2021-02-10T11:33:40.770 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.766 7f9ef5611700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f9ef00e5470 0x7f9ef007e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:40.770 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.766 7f9ef5611700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f9ef00e5470 0x7f9ef007e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:49986/0 (socket says 172.21.2.112:49986) 2021-02-10T11:33:40.770 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.766 7f9ef5611700 1 -- 172.21.2.112:0/865730117 learned_addr learned my addr 172.21.2.112:0/865730117 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:33:40.770 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.767 7f9ef5611700 1 -- 172.21.2.112:0/865730117 >> v1:172.21.2.120:6789/0 conn(0x7f9ef00e5070 legacy=0x7f9ef01b3820 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:33:40.770 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.767 7f9ef5611700 1 -- 172.21.2.112:0/865730117 >> v1:172.21.2.112:6789/0 conn(0x7f9ef007f300 legacy=0x7f9ef007f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:40.770 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.767 7f9ef4e10700 1 -- 172.21.2.112:0/865730117 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3024935683 0 0) 0x7f9ee8000f90 con 0x7f9ef007f300 2021-02-10T11:33:40.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.767 7f9ef5611700 1 -- 172.21.2.112:0/865730117 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ef00d19b0 con 0x7f9ef00e5470 2021-02-10T11:33:40.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.767 7f9ef5611700 1 --2- 172.21.2.112:0/865730117 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9ef00e5470 0x7f9ef007e6a0 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 rx=0x7f9ee4006700 tx=0x7f9ee4009b00).ready entity=mon.0 client_cookie=d0d8817028d52116 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:40.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.767 7f9ef4e10700 1 -- 172.21.2.112:0/865730117 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9ee400c0c0 con 0x7f9ef00e5470 2021-02-10T11:33:40.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.767 7f9ef4e10700 1 -- 172.21.2.112:0/865730117 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ee400c400 con 0x7f9ef00e5470 2021-02-10T11:33:40.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.767 7f9ef4e10700 1 -- 172.21.2.112:0/865730117 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9ee400c860 con 0x7f9ef00e5470 2021-02-10T11:33:40.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.767 7f9ef7ff6700 1 -- 172.21.2.112:0/865730117 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9ef00e5470 msgr2=0x7f9ef007e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:40.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.767 7f9ef7ff6700 1 --2- 172.21.2.112:0/865730117 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9ef00e5470 0x7f9ef007e6a0 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 rx=0x7f9ee4006700 tx=0x7f9ee4009b00).stop 2021-02-10T11:33:40.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.768 7f9ef7ff6700 1 -- 172.21.2.112:0/865730117 shutdown_connections 2021-02-10T11:33:40.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.768 7f9ef7ff6700 1 --2- 172.21.2.112:0/865730117 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9ef00e5470 0x7f9ef007e6a0 unknown :-1 s=CLOSED pgs=1283 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:40.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.768 7f9ef7ff6700 1 -- 172.21.2.112:0/865730117 >> 172.21.2.112:0/865730117 conn(0x7f9ef00ce690 msgr2=0x7f9ef00e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:40.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.768 7f9ef7ff6700 1 -- 172.21.2.112:0/865730117 shutdown_connections 2021-02-10T11:33:40.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.768 7f9ef7ff6700 1 -- 172.21.2.112:0/865730117 wait complete. 2021-02-10T11:33:40.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.782 7f9ef7ff6700 1 Processor -- start 2021-02-10T11:33:40.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.782 7f9ef7ff6700 1 -- start start 2021-02-10T11:33:40.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.783 7f9ef7ff6700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9ef01ade10 0x7f9ef009d250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:40.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.783 7f9ef7ff6700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9ef00b4440 0x7f9ef00b4840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:40.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.783 7f9ef7ff6700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9ef01952d0 0x7f9ef01956d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:40.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.783 7f9ef7ff6700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ef0194d50 con 0x7f9ef00b4440 2021-02-10T11:33:40.774 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.783 7f9ef7ff6700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ef01932c0 con 0x7f9ef01ade10 2021-02-10T11:33:40.774 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.783 7f9ef7ff6700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ef0195cf0 con 0x7f9ef01952d0 2021-02-10T11:33:40.774 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.783 7f9ef5611700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9ef00b4440 0x7f9ef00b4840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:40.774 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.783 7f9ef5611700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9ef00b4440 0x7f9ef00b4840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:49996/0 (socket says 172.21.2.112:49996) 2021-02-10T11:33:40.774 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.783 7f9ef5611700 1 -- 172.21.2.112:0/2505811972 learned_addr learned my addr 172.21.2.112:0/2505811972 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:33:40.774 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.783 7f9ef5611700 1 -- 172.21.2.112:0/2505811972 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9ef01952d0 msgr2=0x7f9ef01956d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:33:40.774 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.783 7f9ef5611700 1 --2- 172.21.2.112:0/2505811972 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9ef01952d0 0x7f9ef01956d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:40.774 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.783 7f9ef5611700 1 -- 172.21.2.112:0/2505811972 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9ef01ade10 msgr2=0x7f9ef009d250 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:33:40.774 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.783 7f9ef5611700 1 --2- 172.21.2.112:0/2505811972 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9ef01ade10 0x7f9ef009d250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:40.774 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.783 7f9ef5611700 1 -- 172.21.2.112:0/2505811972 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ef0195f10 con 0x7f9ef00b4440 2021-02-10T11:33:40.774 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.784 7f9ef5611700 1 --2- 172.21.2.112:0/2505811972 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9ef00b4440 0x7f9ef00b4840 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 rx=0x7f9ee400b660 tx=0x7f9ee400ba00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:40.774 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.784 7f9ee27fc700 1 -- 172.21.2.112:0/2505811972 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9ee4009440 con 0x7f9ef00b4440 2021-02-10T11:33:40.774 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.784 7f9ee27fc700 1 -- 172.21.2.112:0/2505811972 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ee4006030 con 0x7f9ef00b4440 2021-02-10T11:33:40.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.784 7f9ee27fc700 1 -- 172.21.2.112:0/2505811972 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9ee4005370 con 0x7f9ef00b4440 2021-02-10T11:33:40.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.784 7f9ef7ff6700 1 -- 172.21.2.112:0/2505811972 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ef0195f10 con 0x7f9ef00b4440 2021-02-10T11:33:40.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.784 7f9ef7ff6700 1 -- 172.21.2.112:0/2505811972 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ef0195f10 con 0x7f9ef00b4440 2021-02-10T11:33:40.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.785 7f9ee27fc700 1 -- 172.21.2.112:0/2505811972 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9ee40093e0 con 0x7f9ef00b4440 2021-02-10T11:33:40.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.785 7f9ee27fc700 1 --2- 172.21.2.112:0/2505811972 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9ed80186c0 0x7f9ed801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:40.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.785 7f9ee27fc700 1 -- 172.21.2.112:0/2505811972 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f9ee4032590 con 0x7f9ef00b4440 2021-02-10T11:33:40.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.785 7f9ef5e12700 1 --2- 172.21.2.112:0/2505811972 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9ed80186c0 0x7f9ed801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:40.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.786 7f9ef5e12700 1 --2- 172.21.2.112:0/2505811972 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9ed80186c0 0x7f9ed801abc0 crc :-1 s=READY pgs=637 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:40.803 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.816 7f9ef7ff6700 1 -- 172.21.2.112:0/2505811972 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ef0083db0 con 0x7f9ef00b4440 2021-02-10T11:33:40.805 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.818 7f9ee27fc700 1 -- 172.21.2.112:0/2505811972 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9ee40090c0 con 0x7f9ef00b4440 2021-02-10T11:33:40.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.946 7f9ef7ff6700 1 -- 172.21.2.112:0/2505811972 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9ef007ed80 con 0x7f9ef00b4440 2021-02-10T11:33:40.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.946 7f9ee27fc700 1 -- 172.21.2.112:0/2505811972 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) v1 ==== 94+0+2101 (secure 0 0 0) 0x7f9ee400bdf0 con 0x7f9ef00b4440 2021-02-10T11:33:40.934 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:33:40.935 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":39,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7881,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/1476007789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":1476007789},{"type":"v1","addr":"172.21.2.112:6833","nonce":1476007789}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":35}],"filesystems":[{"mdsmap":{"epoch":39,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:33:23.550791","modified":"2021-02-10 11:33:25.562811","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6595":{"gid":6595,"name":"b","rank":0,"incarnation":36,"state":"up:active","state_seq":3,"addr":"172.21.2.120:6835/4241229406","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":4241229406},{"type":"v1","addr":"172.21.2.120:6835","nonce":4241229406}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2021-02-10T11:33:40.935 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 39 2021-02-10T11:33:40.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.948 7f9ef7ff6700 1 -- 172.21.2.112:0/2505811972 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9ed80186c0 msgr2=0x7f9ed801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:40.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.948 7f9ef7ff6700 1 --2- 172.21.2.112:0/2505811972 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9ed80186c0 0x7f9ed801abc0 crc :-1 s=READY pgs=637 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:40.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.948 7f9ef7ff6700 1 -- 172.21.2.112:0/2505811972 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9ef00b4440 msgr2=0x7f9ef00b4840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:40.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.948 7f9ef7ff6700 1 --2- 172.21.2.112:0/2505811972 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9ef00b4440 0x7f9ef00b4840 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 rx=0x7f9ee400b660 tx=0x7f9ee400ba00).stop 2021-02-10T11:33:40.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.948 7f9ef7ff6700 1 -- 172.21.2.112:0/2505811972 shutdown_connections 2021-02-10T11:33:40.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.948 7f9ef7ff6700 1 --2- 172.21.2.112:0/2505811972 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9ed80186c0 0x7f9ed801abc0 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:40.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.948 7f9ef7ff6700 1 --2- 172.21.2.112:0/2505811972 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9ef01952d0 0x7f9ef01956d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:40.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.948 7f9ef7ff6700 1 --2- 172.21.2.112:0/2505811972 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9ef00b4440 0x7f9ef00b4840 unknown :-1 s=CLOSED pgs=1284 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:40.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.948 7f9ef7ff6700 1 --2- 172.21.2.112:0/2505811972 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9ef01ade10 0x7f9ef009d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:40.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.948 7f9ef7ff6700 1 -- 172.21.2.112:0/2505811972 >> 172.21.2.112:0/2505811972 conn(0x7f9ef00937a0 msgr2=0x7f9ef0197e80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:40.938 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.948 7f9ef7ff6700 1 -- 172.21.2.112:0/2505811972 shutdown_connections 2021-02-10T11:33:40.938 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:40.948 7f9ef7ff6700 1 -- 172.21.2.112:0/2505811972 wait complete. 2021-02-10T11:33:41.146 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 39, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:33:23.550791', 'modified': '2021-02-10 11:33:25.562811', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6595}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6595': {'gid': 6595, 'name': 'b', 'rank': 0, 'incarnation': 36, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.120:6835/4241229406', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.120:6834', 'nonce': 4241229406}, {'type': 'v1', 'addr': '172.21.2.120:6835', 'nonce': 4241229406}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [11], 'metadata_pool': 10, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:33:41.146 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:33:41.146 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:33:41.545 INFO:teuthology.orchestra.run.gibba020.stdout:{ 2021-02-10T11:33:41.545 INFO:teuthology.orchestra.run.gibba020.stdout: "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", 2021-02-10T11:33:41.545 INFO:teuthology.orchestra.run.gibba020.stdout: "whoami": 0, 2021-02-10T11:33:41.545 INFO:teuthology.orchestra.run.gibba020.stdout: "id": 6595, 2021-02-10T11:33:41.545 INFO:teuthology.orchestra.run.gibba020.stdout: "want_state": "up:active", 2021-02-10T11:33:41.545 INFO:teuthology.orchestra.run.gibba020.stdout: "state": "up:active", 2021-02-10T11:33:41.546 INFO:teuthology.orchestra.run.gibba020.stdout: "rank_uptime": 18.005011916000001, 2021-02-10T11:33:41.546 INFO:teuthology.orchestra.run.gibba020.stdout: "mdsmap_epoch": 39, 2021-02-10T11:33:41.546 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch": 55, 2021-02-10T11:33:41.546 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch_barrier": 55, 2021-02-10T11:33:41.546 INFO:teuthology.orchestra.run.gibba020.stdout: "uptime": 23.634766556999999 2021-02-10T11:33:41.546 INFO:teuthology.orchestra.run.gibba020.stdout:} 2021-02-10T11:33:41.549 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", "whoami": 0, "id": 6595, "want_state": "up:active", "state": "up:active", "rank_uptime": 18.005011916000001, "mdsmap_epoch": 39, "osdmap_epoch": 55, "osdmap_epoch_barrier": 55, "uptime": 23.634766556999999 } 2021-02-10T11:33:41.549 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:33:41.549 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2021-02-10T11:33:41.549 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.1 at ubuntu@gibba012.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2021-02-10T11:33:41.549 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:41.549 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:33:41.884 DEBUG:teuthology.orchestra.run.gibba012:> sudo modprobe fuse 2021-02-10T11:33:42.227 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:33:42.227 DEBUG:teuthology.orchestra.run.gibba012:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:33:42.570 INFO:teuthology.orchestra.run.gibba012.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:33:42.570 INFO:teuthology.orchestra.run.gibba012.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:33:42.571 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:33:42.571 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:42.571 DEBUG:teuthology.orchestra.run.gibba012:> ls /sys/fs/fuse/connections 2021-02-10T11:33:42.904 INFO:teuthology.orchestra.run.gibba012.stdout:42 2021-02-10T11:33:42.904 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [42] 2021-02-10T11:33:42.905 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.1 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:33:42.907 DEBUG:teuthology.orchestra.run.gibba012:> sudo modprobe fuse 2021-02-10T11:33:43.252 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:33:43.252 DEBUG:teuthology.orchestra.run.gibba012:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:33:43.317 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba012.stderr:ceph-fuse[129816]: starting ceph client 2021-02-10T11:33:43.318 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba012.stderr:2021-02-10 11:33:43.330 7f17972cef80 -1 init, newargv = 0x558dbc2b30b0 newargc=9 2021-02-10T11:33:43.323 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba012.stderr:ceph-fuse[129816]: starting fuse 2021-02-10T11:33:43.595 INFO:teuthology.orchestra.run.gibba012.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:33:43.595 INFO:teuthology.orchestra.run.gibba012.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:33:43.596 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:33:43.596 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:43.596 DEBUG:teuthology.orchestra.run.gibba012:> ls /sys/fs/fuse/connections 2021-02-10T11:33:43.930 INFO:teuthology.orchestra.run.gibba012.stdout:42 2021-02-10T11:33:43.930 INFO:teuthology.orchestra.run.gibba012.stdout:43 2021-02-10T11:33:43.931 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42, 43] 2021-02-10T11:33:43.931 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:43.931 DEBUG:teuthology.orchestra.run.gibba012:> sudo python3 -c ' 2021-02-10T11:33:43.931 DEBUG:teuthology.orchestra.run.gibba012:> import glob 2021-02-10T11:33:43.931 DEBUG:teuthology.orchestra.run.gibba012:> import re 2021-02-10T11:33:43.931 DEBUG:teuthology.orchestra.run.gibba012:> import os 2021-02-10T11:33:43.931 DEBUG:teuthology.orchestra.run.gibba012:> import subprocess 2021-02-10T11:33:43.931 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:33:43.931 DEBUG:teuthology.orchestra.run.gibba012:> def find_socket(client_name): 2021-02-10T11:33:43.931 DEBUG:teuthology.orchestra.run.gibba012:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2021-02-10T11:33:43.932 DEBUG:teuthology.orchestra.run.gibba012:> files = glob.glob(asok_path) 2021-02-10T11:33:43.932 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:33:43.932 DEBUG:teuthology.orchestra.run.gibba012:> # Given a non-glob path, it better be there 2021-02-10T11:33:43.932 DEBUG:teuthology.orchestra.run.gibba012:> if "*" not in asok_path: 2021-02-10T11:33:43.932 DEBUG:teuthology.orchestra.run.gibba012:> assert(len(files) == 1) 2021-02-10T11:33:43.932 DEBUG:teuthology.orchestra.run.gibba012:> return files[0] 2021-02-10T11:33:43.932 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:33:43.932 DEBUG:teuthology.orchestra.run.gibba012:> for f in files: 2021-02-10T11:33:43.932 DEBUG:teuthology.orchestra.run.gibba012:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:33:43.932 DEBUG:teuthology.orchestra.run.gibba012:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:33:43.932 DEBUG:teuthology.orchestra.run.gibba012:> return f 2021-02-10T11:33:43.932 DEBUG:teuthology.orchestra.run.gibba012:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:33:43.932 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:33:43.932 DEBUG:teuthology.orchestra.run.gibba012:> print(find_socket("client.1")) 2021-02-10T11:33:43.932 DEBUG:teuthology.orchestra.run.gibba012:> ' 2021-02-10T11:33:44.292 INFO:teuthology.orchestra.run.gibba012.stdout:/var/run/ceph/ceph-client.1.129816.asok 2021-02-10T11:33:44.295 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.129816.asok 2021-02-10T11:33:44.295 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:44.296 DEBUG:teuthology.orchestra.run.gibba012:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.129816.asok status 2021-02-10T11:33:44.684 INFO:teuthology.orchestra.run.gibba012.stdout:{ 2021-02-10T11:33:44.684 INFO:teuthology.orchestra.run.gibba012.stdout: "metadata": { 2021-02-10T11:33:44.684 INFO:teuthology.orchestra.run.gibba012.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:33:44.685 INFO:teuthology.orchestra.run.gibba012.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:33:44.685 INFO:teuthology.orchestra.run.gibba012.stdout: "entity_id": "1", 2021-02-10T11:33:44.685 INFO:teuthology.orchestra.run.gibba012.stdout: "hostname": "gibba012", 2021-02-10T11:33:44.685 INFO:teuthology.orchestra.run.gibba012.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2021-02-10T11:33:44.685 INFO:teuthology.orchestra.run.gibba012.stdout: "pid": "129816", 2021-02-10T11:33:44.685 INFO:teuthology.orchestra.run.gibba012.stdout: "root": "/" 2021-02-10T11:33:44.685 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:44.685 INFO:teuthology.orchestra.run.gibba012.stdout: "dentry_count": 0, 2021-02-10T11:33:44.685 INFO:teuthology.orchestra.run.gibba012.stdout: "dentry_pinned_count": 0, 2021-02-10T11:33:44.685 INFO:teuthology.orchestra.run.gibba012.stdout: "id": 8079, 2021-02-10T11:33:44.685 INFO:teuthology.orchestra.run.gibba012.stdout: "inst": { 2021-02-10T11:33:44.685 INFO:teuthology.orchestra.run.gibba012.stdout: "name": { 2021-02-10T11:33:44.685 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "client", 2021-02-10T11:33:44.686 INFO:teuthology.orchestra.run.gibba012.stdout: "num": 8079 2021-02-10T11:33:44.686 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:44.686 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": { 2021-02-10T11:33:44.686 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:33:44.686 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:0", 2021-02-10T11:33:44.686 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 1696597564 2021-02-10T11:33:44.686 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:33:44.686 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:44.686 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": { 2021-02-10T11:33:44.686 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:33:44.686 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:0", 2021-02-10T11:33:44.686 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 1696597564 2021-02-10T11:33:44.687 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:44.687 INFO:teuthology.orchestra.run.gibba012.stdout: "inst_str": "client.8079 172.21.2.112:0/1696597564", 2021-02-10T11:33:44.687 INFO:teuthology.orchestra.run.gibba012.stdout: "addr_str": "172.21.2.112:0/1696597564", 2021-02-10T11:33:44.687 INFO:teuthology.orchestra.run.gibba012.stdout: "inode_count": 1, 2021-02-10T11:33:44.687 INFO:teuthology.orchestra.run.gibba012.stdout: "mds_epoch": 39, 2021-02-10T11:33:44.687 INFO:teuthology.orchestra.run.gibba012.stdout: "osd_epoch": 55, 2021-02-10T11:33:44.687 INFO:teuthology.orchestra.run.gibba012.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:33:44.687 INFO:teuthology.orchestra.run.gibba012.stdout: "blacklisted": false 2021-02-10T11:33:44.687 INFO:teuthology.orchestra.run.gibba012.stdout:} 2021-02-10T11:33:44.689 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:44.689 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:33:44.689 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:33:45.023 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:33:45.023 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:33:45.023 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:45.023 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:33:45.368 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2021-02-10T11:33:45.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:45.795 7f78fa3c1700 1 Processor -- start 2021-02-10T11:33:45.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:45.795 7f78fa3c1700 1 -- start start 2021-02-10T11:33:45.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:45.796 7f78fa3c1700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f78f40d1c70 0x7f78f40cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:45.800 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:45.796 7f78fa3c1700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f78f40cf580 0x7f78f40cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:45.800 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:45.796 7f78fa3c1700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78f40fe1c0 con 0x7f78f40d1870 2021-02-10T11:33:45.800 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:45.796 7f78fa3c1700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78f41b0cb0 con 0x7f78f40d1c70 2021-02-10T11:33:45.800 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:45.796 7f78fa3c1700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78f4109a60 con 0x7f78f40cf580 2021-02-10T11:33:45.800 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:45.796 7f78f3fff700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7f78f40d1870 0x7f78f41b04c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:43034/0 (socket says 172.21.2.112:43034) 2021-02-10T11:33:45.800 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:45.796 7f78f37fe700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f78f40d1c70 0x7f78f40cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:45.801 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:45.796 7f78f3fff700 1 -- 172.21.2.112:0/3997527484 learned_addr learned my addr 172.21.2.112:0/3997527484 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:33:45.801 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:45.796 7f78f89de700 1 --2- 172.21.2.112:0/3997527484 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f78f40cf580 0x7f78f40cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:45.801 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:45.796 7f78f37fe700 1 -- 172.21.2.112:0/3997527484 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f78f40cf580 msgr2=0x7f78f40cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:45.801 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:45.796 7f78f37fe700 1 --2- 172.21.2.112:0/3997527484 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f78f40cf580 0x7f78f40cf980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:45.801 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:45.796 7f78f37fe700 1 -- 172.21.2.112:0/3997527484 >> v1:172.21.2.120:6789/0 conn(0x7f78f40d1870 legacy=0x7f78f41b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:45.801 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:45.796 7f78f37fe700 1 -- 172.21.2.112:0/3997527484 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78f40d3360 con 0x7f78f40d1c70 2021-02-10T11:33:45.802 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:45.797 7f78f37fe700 1 --2- 172.21.2.112:0/3997527484 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f78f40d1c70 0x7f78f40cef60 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 rx=0x7f78dc006650 tx=0x7f78dc009a80).ready entity=mon.1 client_cookie=1f6b66290a532661 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:45.802 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:45.797 7f78f2ffd700 1 -- 172.21.2.112:0/3997527484 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f78dc00c0c0 con 0x7f78f40d1c70 2021-02-10T11:33:45.802 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:45.797 7f78f2ffd700 1 -- 172.21.2.112:0/3997527484 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78dc00c340 con 0x7f78f40d1c70 2021-02-10T11:33:45.802 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:45.797 7f78f2ffd700 1 -- 172.21.2.112:0/3997527484 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f78dc00c7a0 con 0x7f78f40d1c70 2021-02-10T11:33:45.802 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:45.797 7f78fa3c1700 1 -- 172.21.2.112:0/3997527484 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f78f40d1c70 msgr2=0x7f78f40cef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:45.802 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:45.797 7f78fa3c1700 1 --2- 172.21.2.112:0/3997527484 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f78f40d1c70 0x7f78f40cef60 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 rx=0x7f78dc006650 tx=0x7f78dc009a80).stop 2021-02-10T11:33:45.802 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:45.797 7f78fa3c1700 1 -- 172.21.2.112:0/3997527484 shutdown_connections 2021-02-10T11:33:45.803 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:45.797 7f78fa3c1700 1 --2- 172.21.2.112:0/3997527484 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f78f40d1c70 0x7f78f40cef60 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:45.803 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:45.797 7f78fa3c1700 1 --2- 172.21.2.112:0/3997527484 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f78f40cf580 0x7f78f40cf980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:45.803 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:45.797 7f78fa3c1700 1 -- 172.21.2.112:0/3997527484 >> 172.21.2.112:0/3997527484 conn(0x7f78f40ce420 msgr2=0x7f78f40d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:45.803 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:45.797 7f78fa3c1700 1 -- 172.21.2.112:0/3997527484 shutdown_connections 2021-02-10T11:33:45.803 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:45.798 7f78fa3c1700 1 -- 172.21.2.112:0/3997527484 wait complete. 2021-02-10T11:33:45.803 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:45.812 7f78fa3c1700 1 Processor -- start 2021-02-10T11:33:45.803 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:45.812 7f78fa3c1700 1 -- start start 2021-02-10T11:33:45.805 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:45.812 7f78fa3c1700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f78f40b8070 0x7f78f40b77c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:45.805 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:45.812 7f78fa3c1700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f78f40ba4a0 0x7f78f40ba8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:45.805 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:45.812 7f78fa3c1700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f78f41a2fd0 0x7f78f41a33d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:45.805 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:45.813 7f78fa3c1700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78f40baec0 con 0x7f78f40b8070 2021-02-10T11:33:45.805 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:45.813 7f78fa3c1700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78f41a39f0 con 0x7f78f41a2fd0 2021-02-10T11:33:45.805 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:45.813 7f78fa3c1700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78f41a3c10 con 0x7f78f40ba4a0 2021-02-10T11:33:45.805 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:45.813 7f78f3fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f78f40b8070 0x7f78f40b77c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:45.805 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:45.813 7f78f3fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f78f40b8070 0x7f78f40b77c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:50020/0 (socket says 172.21.2.112:50020) 2021-02-10T11:33:45.805 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:45.813 7f78f3fff700 1 -- 172.21.2.112:0/2715502886 learned_addr learned my addr 172.21.2.112:0/2715502886 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:33:45.806 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:45.813 7f78f37fe700 1 --2- 172.21.2.112:0/2715502886 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f78f40ba4a0 0x7f78f40ba8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:45.806 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:45.813 7f78f89de700 1 --2- 172.21.2.112:0/2715502886 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f78f41a2fd0 0x7f78f41a33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:45.806 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:45.813 7f78f37fe700 1 -- 172.21.2.112:0/2715502886 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f78f41a2fd0 msgr2=0x7f78f41a33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:45.806 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:45.813 7f78f37fe700 1 --2- 172.21.2.112:0/2715502886 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f78f41a2fd0 0x7f78f41a33d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:45.806 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:45.813 7f78f37fe700 1 -- 172.21.2.112:0/2715502886 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f78f40b8070 msgr2=0x7f78f40b77c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:45.806 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:45.813 7f78f37fe700 1 --2- 172.21.2.112:0/2715502886 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f78f40b8070 0x7f78f40b77c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:45.806 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:45.813 7f78f37fe700 1 -- 172.21.2.112:0/2715502886 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78f40ad700 con 0x7f78f40ba4a0 2021-02-10T11:33:45.806 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:45.813 7f78f89de700 1 --2- 172.21.2.112:0/2715502886 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f78f41a2fd0 0x7f78f41a33d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:33:45.806 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:45.813 7f78f37fe700 1 --2- 172.21.2.112:0/2715502886 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f78f40ba4a0 0x7f78f40ba8a0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f78dc00b500 tx=0x7f78dc00b8a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:45.807 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:45.813 7f78f0ff9700 1 -- 172.21.2.112:0/2715502886 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f78dc0063a0 con 0x7f78f40ba4a0 2021-02-10T11:33:45.807 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:45.813 7f78fa3c1700 1 -- 172.21.2.112:0/2715502886 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78f40ad700 con 0x7f78f40ba4a0 2021-02-10T11:33:45.807 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:45.813 7f78fa3c1700 1 -- 172.21.2.112:0/2715502886 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78f40adc90 con 0x7f78f40ba4a0 2021-02-10T11:33:45.807 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:45.813 7f78f0ff9700 1 -- 172.21.2.112:0/2715502886 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78dc0063a0 con 0x7f78f40ba4a0 2021-02-10T11:33:45.807 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:45.814 7f78f0ff9700 1 -- 172.21.2.112:0/2715502886 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f78dc002030 con 0x7f78f40ba4a0 2021-02-10T11:33:45.807 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:45.814 7f78f0ff9700 1 -- 172.21.2.112:0/2715502886 <== mon.2 v2:172.21.2.120:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f78dc002440 con 0x7f78f40ba4a0 2021-02-10T11:33:45.807 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:45.814 7f78f0ff9700 1 --2- 172.21.2.112:0/2715502886 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f78d80186c0 0x7f78d801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:45.807 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:45.814 7f78f0ff9700 1 -- 172.21.2.112:0/2715502886 <== mon.2 v2:172.21.2.120:3301/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f78dc032340 con 0x7f78f40ba4a0 2021-02-10T11:33:45.807 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:45.814 7f78f3fff700 1 --2- 172.21.2.112:0/2715502886 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f78d80186c0 0x7f78d801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:45.807 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:45.815 7f78f3fff700 1 --2- 172.21.2.112:0/2715502886 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f78d80186c0 0x7f78d801abc0 crc :-1 s=READY pgs=638 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:45.832 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:45.845 7f78fa3c1700 1 -- 172.21.2.112:0/2715502886 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78f40828e0 con 0x7f78f40ba4a0 2021-02-10T11:33:45.836 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:45.849 7f78f0ff9700 1 -- 172.21.2.112:0/2715502886 <== mon.2 v2:172.21.2.120:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f78dc025ef0 con 0x7f78f40ba4a0 2021-02-10T11:33:45.975 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:45.988 7f78fa3c1700 1 -- 172.21.2.112:0/2715502886 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f78f40828e0 con 0x7f78f40ba4a0 2021-02-10T11:33:46.034 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:46.047 7f78f0ff9700 1 -- 172.21.2.112:0/2715502886 <== mon.2 v2:172.21.2.120:3301/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v40) v1 ==== 136+0+0 (secure 0 0 0) 0x7f78dc025ef0 con 0x7f78f40ba4a0 2021-02-10T11:33:46.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:46.055 7f78fa3c1700 1 -- 172.21.2.112:0/2715502886 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f78d80186c0 msgr2=0x7f78d801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:46.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:46.055 7f78fa3c1700 1 --2- 172.21.2.112:0/2715502886 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f78d80186c0 0x7f78d801abc0 crc :-1 s=READY pgs=638 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:46.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:46.055 7f78fa3c1700 1 -- 172.21.2.112:0/2715502886 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f78f40ba4a0 msgr2=0x7f78f40ba8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:46.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:46.055 7f78fa3c1700 1 --2- 172.21.2.112:0/2715502886 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f78f40ba4a0 0x7f78f40ba8a0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f78dc00b500 tx=0x7f78dc00b8a0).stop 2021-02-10T11:33:46.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:46.055 7f78fa3c1700 1 -- 172.21.2.112:0/2715502886 shutdown_connections 2021-02-10T11:33:46.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:46.055 7f78fa3c1700 1 --2- 172.21.2.112:0/2715502886 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f78d80186c0 0x7f78d801abc0 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:46.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:46.055 7f78fa3c1700 1 --2- 172.21.2.112:0/2715502886 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f78f41a2fd0 0x7f78f41a33d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:46.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:46.055 7f78fa3c1700 1 --2- 172.21.2.112:0/2715502886 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f78f40ba4a0 0x7f78f40ba8a0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:46.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:46.055 7f78fa3c1700 1 --2- 172.21.2.112:0/2715502886 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f78f40b8070 0x7f78f40b77c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:46.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:46.055 7f78fa3c1700 1 -- 172.21.2.112:0/2715502886 >> 172.21.2.112:0/2715502886 conn(0x7f78f40b58d0 msgr2=0x7f78f40b5010 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:46.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:46.055 7f78fa3c1700 1 -- 172.21.2.112:0/2715502886 shutdown_connections 2021-02-10T11:33:46.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:46.055 7f78fa3c1700 1 -- 172.21.2.112:0/2715502886 wait complete. 2021-02-10T11:33:46.051 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs volume ls 2021-02-10T11:33:46.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:46.473 7fce158fd700 1 Processor -- start 2021-02-10T11:33:46.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:46.473 7fce158fd700 1 -- start start 2021-02-10T11:33:46.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:46.473 7fce158fd700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fce100e5470 0x7fce1007e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:46.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:46.473 7fce158fd700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fce1007f300 0x7fce1007f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:46.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:46.473 7fce158fd700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce1007ed80 con 0x7fce100e5070 2021-02-10T11:33:46.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:46.473 7fce158fd700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce101b4010 con 0x7fce1007f300 2021-02-10T11:33:46.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:46.473 7fce158fd700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce100d1790 con 0x7fce100e5470 2021-02-10T11:33:46.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:46.473 7fce0fd75700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fce1007f300 0x7fce1007f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:46.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:46.473 7fce0fd75700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fce1007f300 0x7fce1007f700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:50032/0 (socket says 172.21.2.112:50032) 2021-02-10T11:33:46.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:46.473 7fce0fd75700 1 -- 172.21.2.112:0/2681340547 learned_addr learned my addr 172.21.2.112:0/2681340547 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:33:46.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:46.474 7fce0fd75700 1 -- 172.21.2.112:0/2681340547 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fce100e5470 msgr2=0x7fce1007e6a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:33:46.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:46.474 7fce0fd75700 1 --2- 172.21.2.112:0/2681340547 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fce100e5470 0x7fce1007e6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:46.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:46.474 7fce0fd75700 1 -- 172.21.2.112:0/2681340547 >> v1:172.21.2.120:6789/0 conn(0x7fce100e5070 legacy=0x7fce101b3820 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:33:46.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:46.474 7fce0fd75700 1 -- 172.21.2.112:0/2681340547 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce100d1980 con 0x7fce1007f300 2021-02-10T11:33:46.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:46.474 7fce0fd75700 1 --2- 172.21.2.112:0/2681340547 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fce1007f300 0x7fce1007f700 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 rx=0x7fce08006f00 tx=0x7fce08009f10).ready entity=mon.0 client_cookie=6627bb89dc754346 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:46.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:46.474 7fce0e572700 1 -- 172.21.2.112:0/2681340547 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fce0800b3b0 con 0x7fce1007f300 2021-02-10T11:33:46.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:46.474 7fce0e572700 1 -- 172.21.2.112:0/2681340547 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce0800b730 con 0x7fce1007f300 2021-02-10T11:33:46.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:46.474 7fce0e572700 1 -- 172.21.2.112:0/2681340547 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fce0800bc50 con 0x7fce1007f300 2021-02-10T11:33:46.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:46.474 7fce158fd700 1 -- 172.21.2.112:0/2681340547 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fce1007f300 msgr2=0x7fce1007f700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:46.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:46.474 7fce158fd700 1 --2- 172.21.2.112:0/2681340547 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fce1007f300 0x7fce1007f700 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 rx=0x7fce08006f00 tx=0x7fce08009f10).stop 2021-02-10T11:33:46.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:46.474 7fce158fd700 1 -- 172.21.2.112:0/2681340547 shutdown_connections 2021-02-10T11:33:46.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:46.474 7fce158fd700 1 --2- 172.21.2.112:0/2681340547 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fce1007f300 0x7fce1007f700 unknown :-1 s=CLOSED pgs=1288 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:46.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:46.474 7fce158fd700 1 --2- 172.21.2.112:0/2681340547 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fce100e5470 0x7fce1007e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:46.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:46.474 7fce158fd700 1 -- 172.21.2.112:0/2681340547 >> 172.21.2.112:0/2681340547 conn(0x7fce100ce690 msgr2=0x7fce100e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:46.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:46.474 7fce158fd700 1 -- 172.21.2.112:0/2681340547 shutdown_connections 2021-02-10T11:33:46.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:46.475 7fce158fd700 1 -- 172.21.2.112:0/2681340547 wait complete. 2021-02-10T11:33:46.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:46.489 7fce158fd700 1 Processor -- start 2021-02-10T11:33:46.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:46.489 7fce158fd700 1 -- start start 2021-02-10T11:33:46.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:46.489 7fce158fd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fce100a2b70 0x7fce100c0590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:46.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:46.489 7fce158fd700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fce100c0bb0 0x7fce100a3850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:46.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:46.489 7fce158fd700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fce100c0fb0 0x7fce100a3e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:46.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:46.489 7fce158fd700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce100a2f70 con 0x7fce100a2b70 2021-02-10T11:33:46.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:46.489 7fce158fd700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce100a1a40 con 0x7fce100c0bb0 2021-02-10T11:33:46.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:46.489 7fce158fd700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce100a1c60 con 0x7fce100c0fb0 2021-02-10T11:33:46.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:46.490 7fce0f574700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fce100a2b70 0x7fce100c0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:46.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:46.490 7fce0f574700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fce100a2b70 0x7fce100c0590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:50036/0 (socket says 172.21.2.112:50036) 2021-02-10T11:33:46.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:46.490 7fce0f574700 1 -- 172.21.2.112:0/4036981646 learned_addr learned my addr 172.21.2.112:0/4036981646 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:33:46.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:46.490 7fce0f574700 1 -- 172.21.2.112:0/4036981646 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fce100c0fb0 msgr2=0x7fce100a3e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:46.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:46.490 7fce0fd75700 1 --2- 172.21.2.112:0/4036981646 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fce100c0fb0 0x7fce100a3e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:46.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:46.490 7fce0f574700 1 --2- 172.21.2.112:0/4036981646 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fce100c0fb0 0x7fce100a3e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:46.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:46.490 7fce0f574700 1 -- 172.21.2.112:0/4036981646 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fce100c0bb0 msgr2=0x7fce100a3850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:46.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:46.490 7fce0ed73700 1 --2- 172.21.2.112:0/4036981646 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fce100c0bb0 0x7fce100a3850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:46.481 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:46.490 7fce0f574700 1 --2- 172.21.2.112:0/4036981646 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fce100c0bb0 0x7fce100a3850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:46.481 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:46.490 7fce0f574700 1 -- 172.21.2.112:0/4036981646 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce100a4490 con 0x7fce100a2b70 2021-02-10T11:33:46.481 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:46.490 7fce0ed73700 1 --2- 172.21.2.112:0/4036981646 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fce100c0bb0 0x7fce100a3850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:33:46.481 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:46.490 7fce0f574700 1 --2- 172.21.2.112:0/4036981646 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fce100a2b70 0x7fce100c0590 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 rx=0x7fcdf800b530 tx=0x7fcdf800b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:46.481 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:46.490 7fce07fff700 1 -- 172.21.2.112:0/4036981646 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcdf800bcc0 con 0x7fce100a2b70 2021-02-10T11:33:46.481 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:46.491 7fce07fff700 1 -- 172.21.2.112:0/4036981646 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdf8000c60 con 0x7fce100a2b70 2021-02-10T11:33:46.481 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:46.491 7fce07fff700 1 -- 172.21.2.112:0/4036981646 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcdf80062a0 con 0x7fce100a2b70 2021-02-10T11:33:46.481 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:46.491 7fce158fd700 1 -- 172.21.2.112:0/4036981646 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce100a4490 con 0x7fce100a2b70 2021-02-10T11:33:46.481 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:46.491 7fce158fd700 1 -- 172.21.2.112:0/4036981646 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce100a4a20 con 0x7fce100a2b70 2021-02-10T11:33:46.481 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:46.491 7fce07fff700 1 -- 172.21.2.112:0/4036981646 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fcdf8006020 con 0x7fce100a2b70 2021-02-10T11:33:46.481 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:46.491 7fce07fff700 1 --2- 172.21.2.112:0/4036981646 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcdf40186c0 0x7fcdf401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:46.481 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:46.491 7fce07fff700 1 -- 172.21.2.112:0/4036981646 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fcdf80325e0 con 0x7fce100a2b70 2021-02-10T11:33:46.481 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:46.492 7fce0ed73700 1 --2- 172.21.2.112:0/4036981646 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcdf40186c0 0x7fcdf401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:46.482 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:46.492 7fce0ed73700 1 --2- 172.21.2.112:0/4036981646 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcdf40186c0 0x7fcdf401abc0 crc :-1 s=READY pgs=639 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:46.496 DEBUG:teuthology.orchestra.run.gibba012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:33:46.499 DEBUG:teuthology.orchestra.run.gibba020:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:33:46.510 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:46.523 7fce158fd700 1 -- 172.21.2.112:0/4036981646 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce10083da0 con 0x7fce100a2b70 2021-02-10T11:33:46.512 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:46.526 7fce07fff700 1 -- 172.21.2.112:0/4036981646 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fcdf8006230 con 0x7fce100a2b70 2021-02-10T11:33:46.648 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:46.661 7fce158fd700 1 -- 172.21.2.112:0/4036981646 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs volume ls", "target": ["mgr", ""]}) v1 -- 0x7fce1007ed80 con 0x7fcdf40186c0 2021-02-10T11:33:46.649 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:46.662 7fce07fff700 1 -- 172.21.2.112:0/4036981646 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+41 (crc 0 0 0) 0x7fce00006580 con 0x7fcdf40186c0 2021-02-10T11:33:46.651 INFO:teuthology.orchestra.run.gibba012.stdout:[ 2021-02-10T11:33:46.651 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:46.651 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "cephfs" 2021-02-10T11:33:46.652 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:33:46.652 INFO:teuthology.orchestra.run.gibba012.stdout:] 2021-02-10T11:33:46.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:46.666 7fce158fd700 1 -- 172.21.2.112:0/4036981646 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcdf40186c0 msgr2=0x7fcdf401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:46.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:46.666 7fce158fd700 1 --2- 172.21.2.112:0/4036981646 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcdf40186c0 0x7fcdf401abc0 crc :-1 s=READY pgs=639 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:46.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:46.666 7fce158fd700 1 -- 172.21.2.112:0/4036981646 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fce100a2b70 msgr2=0x7fce100c0590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:46.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:46.666 7fce158fd700 1 --2- 172.21.2.112:0/4036981646 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fce100a2b70 0x7fce100c0590 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 rx=0x7fcdf800b530 tx=0x7fcdf800b8d0).stop 2021-02-10T11:33:46.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:46.666 7fce158fd700 1 -- 172.21.2.112:0/4036981646 shutdown_connections 2021-02-10T11:33:46.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:46.666 7fce158fd700 1 --2- 172.21.2.112:0/4036981646 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcdf40186c0 0x7fcdf401abc0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:46.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:46.666 7fce158fd700 1 --2- 172.21.2.112:0/4036981646 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fce100c0bb0 0x7fce100a3850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:46.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:46.666 7fce158fd700 1 --2- 172.21.2.112:0/4036981646 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fce100c0fb0 0x7fce100a3e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:46.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:46.666 7fce158fd700 1 --2- 172.21.2.112:0/4036981646 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fce100a2b70 0x7fce100c0590 unknown :-1 s=CLOSED pgs=1289 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:46.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:46.666 7fce158fd700 1 -- 172.21.2.112:0/4036981646 >> 172.21.2.112:0/4036981646 conn(0x7fce100bde70 msgr2=0x7fce100c3970 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:46.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:46.667 7fce158fd700 1 -- 172.21.2.112:0/4036981646 shutdown_connections 2021-02-10T11:33:46.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:46.667 7fce158fd700 1 -- 172.21.2.112:0/4036981646 wait complete. 2021-02-10T11:33:46.662 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:33:47.131 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:47.127 7fb23e840700 1 Processor -- start 2021-02-10T11:33:47.132 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:47.128 7fb23e840700 1 -- start start 2021-02-10T11:33:47.132 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:47.128 7fb23e840700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fb2381b0870 0x7fb2380f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:47.132 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:47.128 7fb23e840700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2381b0c70 con 0x7fb2380efe00 2021-02-10T11:33:47.132 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:47.128 7fb23e840700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2380f0b50 con 0x7fb2381b0470 2021-02-10T11:33:47.132 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:47.128 7fb23e840700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2380e5ea0 con 0x7fb2381b0870 2021-02-10T11:33:47.132 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:47.128 7fb23ce5d700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fb2380efe00 0x7fb2380f0200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:49508/0 (socket says 172.21.2.112:49508) 2021-02-10T11:33:47.132 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:47.128 7fb23ce5d700 1 -- 172.21.2.112:0/1021726366 learned_addr learned my addr 172.21.2.112:0/1021726366 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:33:47.132 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:47.129 7fb236ffd700 1 -- 172.21.2.112:0/1021726366 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1969777727 0 0) 0x7fb230001500 con 0x7fb2380efe00 2021-02-10T11:33:47.132 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:47.129 7fb236ffd700 1 -- 172.21.2.112:0/1021726366 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb220002fe0 con 0x7fb2380efe00 2021-02-10T11:33:47.132 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:47.129 7fb2377fe700 1 --2- 172.21.2.112:0/1021726366 >> v2:172.21.2.120:3300/0 conn(0x7fb2381b0870 0x7fb2380f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:47.132 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:47.129 7fb236ffd700 1 -- 172.21.2.112:0/1021726366 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2423441410 0 0) 0x7fb230001460 con 0x7fb2380efe00 2021-02-10T11:33:47.132 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:47.129 7fb236ffd700 1 -- 172.21.2.112:0/1021726366 >> v2:172.21.2.120:3300/0 conn(0x7fb2381b0870 msgr2=0x7fb2380f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:47.132 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:47.129 7fb236ffd700 1 --2- 172.21.2.112:0/1021726366 >> v2:172.21.2.120:3300/0 conn(0x7fb2381b0870 0x7fb2380f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:47.133 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:47.129 7fb236ffd700 1 -- 172.21.2.112:0/1021726366 >> v1:172.21.2.120:6789/0 conn(0x7fb2381b0470 legacy=0x7fb2380ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:47.133 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:47.129 7fb236ffd700 1 -- 172.21.2.112:0/1021726366 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2380e52e0 con 0x7fb2380efe00 2021-02-10T11:33:47.133 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:47.129 7fb236ffd700 1 -- 172.21.2.112:0/1021726366 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7fb230001980 con 0x7fb2380efe00 2021-02-10T11:33:47.133 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:47.129 7fb236ffd700 1 -- 172.21.2.112:0/1021726366 >> v1:172.21.2.112:6789/0 conn(0x7fb2380efe00 legacy=0x7fb2380f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:47.133 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:47.129 7fb236ffd700 1 --2- 172.21.2.112:0/1021726366 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb220003a70 0x7fb220005ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:47.133 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:47.129 7fb236ffd700 1 --2- 172.21.2.112:0/1021726366 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb220006780 0x7fb220008ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:47.133 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:47.129 7fb236ffd700 1 --2- 172.21.2.112:0/1021726366 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb2200095a0 0x7fb22000bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:47.133 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:47.129 7fb236ffd700 1 -- 172.21.2.112:0/1021726366 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb22000c310 con 0x7fb220003a70 2021-02-10T11:33:47.133 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:47.129 7fb236ffd700 1 -- 172.21.2.112:0/1021726366 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb22000c530 con 0x7fb2200095a0 2021-02-10T11:33:47.133 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:47.129 7fb236ffd700 1 -- 172.21.2.112:0/1021726366 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb22000c750 con 0x7fb220006780 2021-02-10T11:33:47.133 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:47.129 7fb237fff700 1 --2- 172.21.2.112:0/1021726366 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb220003a70 0x7fb220005ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:47.133 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:47.129 7fb2377fe700 1 --2- 172.21.2.112:0/1021726366 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb220006780 0x7fb220008ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:47.133 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:47.129 7fb237fff700 1 -- 172.21.2.112:0/1021726366 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb220006780 msgr2=0x7fb220008ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:47.133 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:47.129 7fb237fff700 1 --2- 172.21.2.112:0/1021726366 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb220006780 0x7fb220008ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:47.134 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:47.129 7fb237fff700 1 -- 172.21.2.112:0/1021726366 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb2200095a0 msgr2=0x7fb22000bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:47.134 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:47.129 7fb23ce5d700 1 --2- 172.21.2.112:0/1021726366 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb2200095a0 0x7fb22000bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:47.134 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:47.129 7fb237fff700 1 --2- 172.21.2.112:0/1021726366 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb2200095a0 0x7fb22000bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:47.134 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:47.129 7fb237fff700 1 -- 172.21.2.112:0/1021726366 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb22000c9f0 con 0x7fb220003a70 2021-02-10T11:33:47.134 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:47.129 7fb23ce5d700 1 --2- 172.21.2.112:0/1021726366 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb2200095a0 0x7fb22000bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:33:47.134 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:47.129 7fb23ce5d700 1 -- 172.21.2.112:0/1021726366 reap_dead start 2021-02-10T11:33:47.134 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:47.129 7fb237fff700 1 --2- 172.21.2.112:0/1021726366 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb220003a70 0x7fb220005ee0 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 rx=0x7fb22804aad0 tx=0x7fb22804da50).ready entity=mon.0 client_cookie=64433393ded31239 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:47.134 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:47.130 7fb236ffd700 1 -- 172.21.2.112:0/1021726366 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb22804b280 con 0x7fb220003a70 2021-02-10T11:33:47.134 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:47.130 7fb236ffd700 1 -- 172.21.2.112:0/1021726366 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb22804b5c0 con 0x7fb220003a70 2021-02-10T11:33:47.134 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:47.130 7fb236ffd700 1 -- 172.21.2.112:0/1021726366 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb22804baa0 con 0x7fb220003a70 2021-02-10T11:33:47.134 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:47.130 7fb23e840700 1 -- 172.21.2.112:0/1021726366 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb220003a70 msgr2=0x7fb220005ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:47.134 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:47.130 7fb23e840700 1 --2- 172.21.2.112:0/1021726366 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb220003a70 0x7fb220005ee0 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 rx=0x7fb22804aad0 tx=0x7fb22804da50).stop 2021-02-10T11:33:47.134 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:47.130 7fb23e840700 1 -- 172.21.2.112:0/1021726366 shutdown_connections 2021-02-10T11:33:47.135 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:47.130 7fb23e840700 1 --2- 172.21.2.112:0/1021726366 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb220003a70 0x7fb220005ee0 unknown :-1 s=CLOSED pgs=1291 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:47.135 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:47.130 7fb23e840700 1 -- 172.21.2.112:0/1021726366 >> 172.21.2.112:0/1021726366 conn(0x7fb2380ce3e0 msgr2=0x7fb2380e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:47.135 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:47.130 7fb23e840700 1 -- 172.21.2.112:0/1021726366 shutdown_connections 2021-02-10T11:33:47.135 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:47.130 7fb23e840700 1 -- 172.21.2.112:0/1021726366 wait complete. 2021-02-10T11:33:47.135 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:47.145 7fb23e840700 1 Processor -- start 2021-02-10T11:33:47.135 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:47.145 7fb23e840700 1 -- start start 2021-02-10T11:33:47.136 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:47.145 7fb23e840700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb23809b930 0x7fb2380b7c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:47.136 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:47.145 7fb23e840700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb2380b8410 0x7fb2380b8810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:47.136 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:47.145 7fb23e840700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb2380c1f80 0x7fb2380c2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:47.136 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:47.145 7fb23e840700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2380c1a00 con 0x7fb2380b8410 2021-02-10T11:33:47.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:47.145 7fb23e840700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2380b8e30 con 0x7fb23809b930 2021-02-10T11:33:47.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:47.145 7fb23e840700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2380b9050 con 0x7fb2380c1f80 2021-02-10T11:33:47.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:47.145 7fb2377fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb2380b8410 0x7fb2380b8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:47.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:47.145 7fb2377fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb2380b8410 0x7fb2380b8810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:50054/0 (socket says 172.21.2.112:50054) 2021-02-10T11:33:47.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:47.145 7fb2377fe700 1 -- 172.21.2.112:0/3098729818 learned_addr learned my addr 172.21.2.112:0/3098729818 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:33:47.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:47.145 7fb237fff700 1 --2- 172.21.2.112:0/3098729818 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb23809b930 0x7fb2380b7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:47.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:47.145 7fb23ce5d700 1 --2- 172.21.2.112:0/3098729818 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb2380c1f80 0x7fb2380c2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:47.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:47.146 7fb2377fe700 1 -- 172.21.2.112:0/3098729818 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb2380c1f80 msgr2=0x7fb2380c2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:47.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:47.146 7fb2377fe700 1 --2- 172.21.2.112:0/3098729818 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb2380c1f80 0x7fb2380c2380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:47.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:47.146 7fb2377fe700 1 -- 172.21.2.112:0/3098729818 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb23809b930 msgr2=0x7fb2380b7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:47.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:47.146 7fb2377fe700 1 --2- 172.21.2.112:0/3098729818 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb23809b930 0x7fb2380b7c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:47.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:47.146 7fb2377fe700 1 -- 172.21.2.112:0/3098729818 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2380c29a0 con 0x7fb2380b8410 2021-02-10T11:33:47.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:47.146 7fb2377fe700 1 --2- 172.21.2.112:0/3098729818 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb2380b8410 0x7fb2380b8810 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 rx=0x7fb22c006f00 tx=0x7fb22c00b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:47.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:47.146 7fb234ff9700 1 -- 172.21.2.112:0/3098729818 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb22c006b90 con 0x7fb2380b8410 2021-02-10T11:33:47.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:47.146 7fb234ff9700 1 -- 172.21.2.112:0/3098729818 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb22c00bd30 con 0x7fb2380b8410 2021-02-10T11:33:47.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:47.146 7fb234ff9700 1 -- 172.21.2.112:0/3098729818 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb22c0022b0 con 0x7fb2380b8410 2021-02-10T11:33:47.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:47.146 7fb23e840700 1 -- 172.21.2.112:0/3098729818 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2380c29a0 con 0x7fb2380b8410 2021-02-10T11:33:47.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:47.146 7fb23e840700 1 -- 172.21.2.112:0/3098729818 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2380c2f30 con 0x7fb2380b8410 2021-02-10T11:33:47.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:47.147 7fb234ff9700 1 -- 172.21.2.112:0/3098729818 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb22c0052e0 con 0x7fb2380b8410 2021-02-10T11:33:47.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:47.147 7fb234ff9700 1 --2- 172.21.2.112:0/3098729818 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb2180186c0 0x7fb21801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:47.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:47.147 7fb234ff9700 1 -- 172.21.2.112:0/3098729818 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fb22c0325a0 con 0x7fb2380b8410 2021-02-10T11:33:47.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:47.147 7fb237fff700 1 --2- 172.21.2.112:0/3098729818 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb2180186c0 0x7fb21801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:47.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:47.148 7fb237fff700 1 --2- 172.21.2.112:0/3098729818 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb2180186c0 0x7fb21801abc0 crc :-1 s=READY pgs=640 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:47.165 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:47.178 7fb23e840700 1 -- 172.21.2.112:0/3098729818 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb238105b30 con 0x7fb2380b8410 2021-02-10T11:33:47.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:47.181 7fb234ff9700 1 -- 172.21.2.112:0/3098729818 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb22c000c90 con 0x7fb2380b8410 2021-02-10T11:33:47.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:47.329 7fb23e840700 1 -- 172.21.2.112:0/3098729818 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fb2380f0b50 con 0x7fb2380b8410 2021-02-10T11:33:47.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:47.330 7fb234ff9700 1 -- 172.21.2.112:0/3098729818 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v8)=0 v8) v1 ==== 122+0+0 (secure 0 0 0) 0x7fb22c01f340 con 0x7fb2380b8410 2021-02-10T11:33:47.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:47.332 7fb23e840700 1 -- 172.21.2.112:0/3098729818 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb2180186c0 msgr2=0x7fb21801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:47.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:47.332 7fb23e840700 1 --2- 172.21.2.112:0/3098729818 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb2180186c0 0x7fb21801abc0 crc :-1 s=READY pgs=640 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:47.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:47.332 7fb23e840700 1 -- 172.21.2.112:0/3098729818 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb2380b8410 msgr2=0x7fb2380b8810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:47.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:47.332 7fb23e840700 1 --2- 172.21.2.112:0/3098729818 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb2380b8410 0x7fb2380b8810 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 rx=0x7fb22c006f00 tx=0x7fb22c00b940).stop 2021-02-10T11:33:47.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:47.332 7fb23e840700 1 -- 172.21.2.112:0/3098729818 shutdown_connections 2021-02-10T11:33:47.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:47.332 7fb23e840700 1 --2- 172.21.2.112:0/3098729818 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb2180186c0 0x7fb21801abc0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:47.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:47.332 7fb23e840700 1 --2- 172.21.2.112:0/3098729818 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb2380c1f80 0x7fb2380c2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:47.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:47.332 7fb23e840700 1 --2- 172.21.2.112:0/3098729818 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb2380b8410 0x7fb2380b8810 unknown :-1 s=CLOSED pgs=1292 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:47.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:47.332 7fb23e840700 1 --2- 172.21.2.112:0/3098729818 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb23809b930 0x7fb2380b7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:47.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:47.332 7fb23e840700 1 -- 172.21.2.112:0/3098729818 >> 172.21.2.112:0/3098729818 conn(0x7fb238094e10 msgr2=0x7fb2380a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:47.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:47.332 7fb23e840700 1 -- 172.21.2.112:0/3098729818 shutdown_connections 2021-02-10T11:33:47.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:47.332 7fb23e840700 1 -- 172.21.2.112:0/3098729818 wait complete. 2021-02-10T11:33:47.328 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:47.328 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:33:47.328 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:33:47.663 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:33:47.664 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:33:47.664 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba012.front.sepia.ceph.com... 2021-02-10T11:33:47.664 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:33:47.664 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:33:48.017 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba012.stderr:ceph-fuse[129816]: fuse finished with error 0 and tester_r 0 2021-02-10T11:33:48.018 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:48.019 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:33:48.019 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:33:48.353 INFO:teuthology.orchestra.run.gibba012.stdout:ext2/ext3 2021-02-10T11:33:48.354 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:33:48.354 INFO:teuthology.orchestra.run:waiting for 900 2021-02-10T11:33:48.355 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:33:48.355 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:33:48.691 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo mkdir -p ./volumes/group_0000000000544641/subvolume_0000000000670007 2021-02-10T11:33:49.044 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo getfattr --only-values -n ceph.dir.layout.pool . 2021-02-10T11:33:49.390 INFO:teuthology.orchestra.run.gibba012.stdout:cephfs_data 2021-02-10T11:33:49.391 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo setfattr -n ceph.dir.layout.pool -v cephfs_data ./volumes/group_0000000000544641/subvolume_0000000000670007 2021-02-10T11:33:49.759 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume authorize cephfs subvolume_0000000000670007 alice --group_name group_0000000000544641 --tenant_id tenant_id 2021-02-10T11:33:50.197 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.193 7f478d569700 1 Processor -- start 2021-02-10T11:33:50.197 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.193 7f478d569700 1 -- start start 2021-02-10T11:33:50.197 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.194 7f478d569700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f47880e5010 0x7f47881b37c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:50.198 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.194 7f478d569700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f47880e5410 0x7f47880f4de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:50.198 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.194 7f478d569700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47880f5480 con 0x7f47881b3de0 2021-02-10T11:33:50.198 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.194 7f478d569700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47880e5e40 con 0x7f47880e5410 2021-02-10T11:33:50.198 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.194 7f478d569700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47880d1750 con 0x7f47880e5010 2021-02-10T11:33:50.198 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.194 7f47872bf700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f47880e5010 0x7f47881b37c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:50.198 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.194 7f4787ac0700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f47880e5410 0x7f47880f4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:50.198 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.194 7f4787ac0700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f47880e5410 0x7f47880f4de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3300/0 says I am v2:172.21.2.112:52342/0 (socket says 172.21.2.112:52342) 2021-02-10T11:33:50.198 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.194 7f4787ac0700 1 -- 172.21.2.112:0/3711008995 learned_addr learned my addr 172.21.2.112:0/3711008995 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:33:50.198 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.195 7f47872bf700 1 -- 172.21.2.112:0/3711008995 >> v2:172.21.2.120:3300/0 conn(0x7f47880e5410 msgr2=0x7f47880f4de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:50.198 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.195 7f47872bf700 1 --2- 172.21.2.112:0/3711008995 >> v2:172.21.2.120:3300/0 conn(0x7f47880e5410 0x7f47880f4de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:50.198 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.195 7f47872bf700 1 -- 172.21.2.112:0/3711008995 >> v1:172.21.2.120:6789/0 conn(0x7f47881b3de0 legacy=0x7f47880f5a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:50.198 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.195 7f47872bf700 1 -- 172.21.2.112:0/3711008995 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47880d1940 con 0x7f47880e5010 2021-02-10T11:33:50.198 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.195 7f4787ac0700 1 --2- 172.21.2.112:0/3711008995 >> v2:172.21.2.120:3300/0 conn(0x7f47880e5410 0x7f47880f4de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:33:50.199 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.195 7f47872bf700 1 --2- 172.21.2.112:0/3711008995 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f47880e5010 0x7f47881b37c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f4770006650 tx=0x7f4770009a80).ready entity=mon.2 client_cookie=e0a9182f7844c32a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:50.199 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.195 7f4786abe700 1 -- 172.21.2.112:0/3711008995 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f477000c0c0 con 0x7f47880e5010 2021-02-10T11:33:50.199 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.195 7f4786abe700 1 -- 172.21.2.112:0/3711008995 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f477000c440 con 0x7f47880e5010 2021-02-10T11:33:50.199 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.195 7f4786abe700 1 -- 172.21.2.112:0/3711008995 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f477000c920 con 0x7f47880e5010 2021-02-10T11:33:50.199 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.195 7f478d569700 1 -- 172.21.2.112:0/3711008995 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f47880e5010 msgr2=0x7f47881b37c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:50.199 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.195 7f478d569700 1 --2- 172.21.2.112:0/3711008995 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f47880e5010 0x7f47881b37c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f4770006650 tx=0x7f4770009a80).stop 2021-02-10T11:33:50.199 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.196 7f478d569700 1 -- 172.21.2.112:0/3711008995 shutdown_connections 2021-02-10T11:33:50.199 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.196 7f478d569700 1 --2- 172.21.2.112:0/3711008995 >> v2:172.21.2.120:3300/0 conn(0x7f47880e5410 0x7f47880f4de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:50.199 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.196 7f478d569700 1 --2- 172.21.2.112:0/3711008995 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f47880e5010 0x7f47881b37c0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:50.199 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.196 7f478d569700 1 -- 172.21.2.112:0/3711008995 >> 172.21.2.112:0/3711008995 conn(0x7f47880ce670 msgr2=0x7f47880e44b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:50.199 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.196 7f478d569700 1 -- 172.21.2.112:0/3711008995 shutdown_connections 2021-02-10T11:33:50.199 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.196 7f478d569700 1 -- 172.21.2.112:0/3711008995 wait complete. 2021-02-10T11:33:50.201 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.211 7f478d569700 1 Processor -- start 2021-02-10T11:33:50.201 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.211 7f478d569700 1 -- start start 2021-02-10T11:33:50.201 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.211 7f478d569700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f47880ae990 0x7f47881afc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:50.201 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.211 7f478d569700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f47881b0260 0x7f4788095080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:50.201 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.211 7f478d569700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f47880956a0 0x7f4788095aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:50.201 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.211 7f478d569700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47881b2360 con 0x7f47881b0260 2021-02-10T11:33:50.201 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.211 7f478d569700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47881b0860 con 0x7f47880956a0 2021-02-10T11:33:50.201 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.211 7f478d569700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47880960c0 con 0x7f47880ae990 2021-02-10T11:33:50.201 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.211 7f477eabe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f47881b0260 0x7f4788095080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:50.202 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.211 7f477eabe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f47881b0260 0x7f4788095080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:50068/0 (socket says 172.21.2.112:50068) 2021-02-10T11:33:50.202 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.211 7f477eabe700 1 -- 172.21.2.112:0/99652256 learned_addr learned my addr 172.21.2.112:0/99652256 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:33:50.202 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.211 7f47872bf700 1 --2- 172.21.2.112:0/99652256 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f47880ae990 0x7f47881afc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:50.202 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.211 7f4787ac0700 1 --2- 172.21.2.112:0/99652256 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f47880956a0 0x7f4788095aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:50.202 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.212 7f477eabe700 1 -- 172.21.2.112:0/99652256 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f47880ae990 msgr2=0x7f47881afc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:50.202 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.212 7f477eabe700 1 --2- 172.21.2.112:0/99652256 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f47880ae990 0x7f47881afc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:50.202 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.212 7f477eabe700 1 -- 172.21.2.112:0/99652256 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f47880956a0 msgr2=0x7f4788095aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:50.202 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.212 7f477eabe700 1 --2- 172.21.2.112:0/99652256 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f47880956a0 0x7f4788095aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:50.202 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.212 7f477eabe700 1 -- 172.21.2.112:0/99652256 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47881b0de0 con 0x7f47881b0260 2021-02-10T11:33:50.202 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.212 7f4787ac0700 1 --2- 172.21.2.112:0/99652256 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f47880956a0 0x7f4788095aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:33:50.202 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.212 7f477eabe700 1 --2- 172.21.2.112:0/99652256 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f47881b0260 0x7f4788095080 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 rx=0x7f477800b530 tx=0x7f477800b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:50.202 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.212 7f4784aba700 1 -- 172.21.2.112:0/99652256 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f477800bcc0 con 0x7f47881b0260 2021-02-10T11:33:50.202 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.212 7f4784aba700 1 -- 172.21.2.112:0/99652256 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4778000c60 con 0x7f47881b0260 2021-02-10T11:33:50.202 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.212 7f4784aba700 1 -- 172.21.2.112:0/99652256 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f47780062a0 con 0x7f47881b0260 2021-02-10T11:33:50.203 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.212 7f478d569700 1 -- 172.21.2.112:0/99652256 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47881b0de0 con 0x7f47881b0260 2021-02-10T11:33:50.203 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.212 7f478d569700 1 -- 172.21.2.112:0/99652256 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47881b0de0 con 0x7f47881b0260 2021-02-10T11:33:50.203 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.212 7f4784aba700 1 -- 172.21.2.112:0/99652256 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f477800a000 con 0x7f47881b0260 2021-02-10T11:33:50.203 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.213 7f4784aba700 1 --2- 172.21.2.112:0/99652256 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f47600186c0 0x7f476001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:50.203 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.213 7f4784aba700 1 -- 172.21.2.112:0/99652256 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f47780323e0 con 0x7f47881b0260 2021-02-10T11:33:50.203 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.213 7f47872bf700 1 --2- 172.21.2.112:0/99652256 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f47600186c0 0x7f476001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:50.203 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.213 7f47872bf700 1 --2- 172.21.2.112:0/99652256 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f47600186c0 0x7f476001abc0 crc :-1 s=READY pgs=641 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:50.230 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.244 7f478d569700 1 -- 172.21.2.112:0/99652256 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47881b0860 con 0x7f47881b0260 2021-02-10T11:33:50.233 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.246 7f4784aba700 1 -- 172.21.2.112:0/99652256 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f477800bcc0 con 0x7f47881b0260 2021-02-10T11:33:50.358 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.371 7f478d569700 1 -- 172.21.2.112:0/99652256 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume authorize", "vol_name": "cephfs", "sub_name": "subvolume_0000000000670007", "target": ["mgr", ""], "tenant_id": "tenant_id", "auth_id": "alice", "group_name": "group_0000000000544641"}) v1 -- 0x7f47880960c0 con 0x7f47600186c0 2021-02-10T11:33:50.362 INFO:tasks.ceph.mgr.x.gibba020.stderr:2021-02-10 11:33:50.379 7f6382eee700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:33:50.362 INFO:tasks.ceph.mgr.x.gibba020.stderr:2021-02-10 11:33:50.379 7f6382eee700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:33:50.362 INFO:tasks.ceph.mgr.x.gibba020.stderr:2021-02-10 11:33:50.379 7f6382eee700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:33:50.362 INFO:tasks.ceph.mgr.x.gibba020.stderr:2021-02-10 11:33:50.379 7f6382eee700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:33:50.362 INFO:tasks.ceph.mgr.x.gibba020.stderr:2021-02-10 11:33:50.379 7f6382eee700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:33:50.386 INFO:tasks.ceph.mgr.x.gibba020.stderr:Creating meta for ID alice with tenant tenant_id 2021-02-10T11:33:50.414 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.427 7f4784aba700 1 -- 172.21.2.112:0/99652256 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+41 (crc 0 0 0) 0x7f47700063a0 con 0x7f47600186c0 2021-02-10T11:33:50.421 INFO:teuthology.orchestra.run.gibba012.stdout:AQCexCNgfqx8GBAANue1hs5gTtkZO/pClbOpEw== 2021-02-10T11:33:50.423 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.436 7f478d569700 1 -- 172.21.2.112:0/99652256 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f47600186c0 msgr2=0x7f476001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:50.423 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.436 7f478d569700 1 --2- 172.21.2.112:0/99652256 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f47600186c0 0x7f476001abc0 crc :-1 s=READY pgs=641 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:50.423 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.436 7f478d569700 1 -- 172.21.2.112:0/99652256 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f47881b0260 msgr2=0x7f4788095080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:50.423 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.436 7f478d569700 1 --2- 172.21.2.112:0/99652256 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f47881b0260 0x7f4788095080 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 rx=0x7f477800b530 tx=0x7f477800b8d0).stop 2021-02-10T11:33:50.423 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.436 7f478d569700 1 -- 172.21.2.112:0/99652256 shutdown_connections 2021-02-10T11:33:50.423 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.436 7f478d569700 1 --2- 172.21.2.112:0/99652256 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f47600186c0 0x7f476001abc0 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:50.423 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.436 7f478d569700 1 --2- 172.21.2.112:0/99652256 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f47880956a0 0x7f4788095aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:50.424 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.436 7f478d569700 1 --2- 172.21.2.112:0/99652256 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f47881b0260 0x7f4788095080 unknown :-1 s=CLOSED pgs=1293 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:50.424 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.436 7f478d569700 1 --2- 172.21.2.112:0/99652256 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f47880ae990 0x7f47881afc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:50.424 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.436 7f478d569700 1 -- 172.21.2.112:0/99652256 >> 172.21.2.112:0/99652256 conn(0x7f47881b2dc0 msgr2=0x7f478807fd60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:50.424 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.437 7f478d569700 1 -- 172.21.2.112:0/99652256 shutdown_connections 2021-02-10T11:33:50.424 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.437 7f478d569700 1 -- 172.21.2.112:0/99652256 wait complete. 2021-02-10T11:33:50.432 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth ls --format=json-pretty 2021-02-10T11:33:50.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.855 7f22e6346700 1 Processor -- start 2021-02-10T11:33:50.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.855 7f22e6346700 1 -- start start 2021-02-10T11:33:50.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.855 7f22e6346700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f22e00e5470 0x7f22e007e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:50.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.855 7f22e6346700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f22e007f300 0x7f22e007f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:50.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.855 7f22e6346700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22e007ed80 con 0x7f22e00e5070 2021-02-10T11:33:50.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.855 7f22e6346700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22e01b4010 con 0x7f22e00e5470 2021-02-10T11:33:50.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.855 7f22e6346700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22e00d1790 con 0x7f22e007f300 2021-02-10T11:33:50.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.856 7f22df7fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f22e00e5470 0x7f22e007e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:50.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.856 7f22dffff700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f22e00e5070 0x7f22e01b3820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:49536/0 (socket says 172.21.2.112:49536) 2021-02-10T11:33:50.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.856 7f22dffff700 1 -- 172.21.2.112:0/3532784238 learned_addr learned my addr 172.21.2.112:0/3532784238 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:33:50.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.856 7f22df7fe700 1 -- 172.21.2.112:0/3532784238 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f22e007f300 msgr2=0x7f22e007f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:50.860 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.856 7f22df7fe700 1 --2- 172.21.2.112:0/3532784238 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f22e007f300 0x7f22e007f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:50.860 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.856 7f22df7fe700 1 -- 172.21.2.112:0/3532784238 >> v1:172.21.2.112:6789/0 conn(0x7f22e00e5070 legacy=0x7f22e01b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:50.860 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.856 7f22df7fe700 1 -- 172.21.2.112:0/3532784238 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22e00d19b0 con 0x7f22e00e5470 2021-02-10T11:33:50.860 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.856 7f22df7fe700 1 --2- 172.21.2.112:0/3532784238 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f22e00e5470 0x7f22e007e6a0 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 rx=0x7f22d4006650 tx=0x7f22d4009a80).ready entity=mon.0 client_cookie=fb4e7b51cdce6f9e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:50.860 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.857 7f22deffd700 1 -- 172.21.2.112:0/3532784238 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f22d40009b0 con 0x7f22e00e5470 2021-02-10T11:33:50.860 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.857 7f22deffd700 1 -- 172.21.2.112:0/3532784238 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22d400c0c0 con 0x7f22e00e5470 2021-02-10T11:33:50.860 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.857 7f22deffd700 1 -- 172.21.2.112:0/3532784238 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f22d400c5e0 con 0x7f22e00e5470 2021-02-10T11:33:50.860 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.857 7f22e6346700 1 -- 172.21.2.112:0/3532784238 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f22e00e5470 msgr2=0x7f22e007e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:50.860 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.857 7f22e6346700 1 --2- 172.21.2.112:0/3532784238 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f22e00e5470 0x7f22e007e6a0 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 rx=0x7f22d4006650 tx=0x7f22d4009a80).stop 2021-02-10T11:33:50.860 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.857 7f22e6346700 1 -- 172.21.2.112:0/3532784238 shutdown_connections 2021-02-10T11:33:50.860 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.857 7f22e6346700 1 --2- 172.21.2.112:0/3532784238 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f22e007f300 0x7f22e007f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:50.860 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.857 7f22e6346700 1 --2- 172.21.2.112:0/3532784238 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f22e00e5470 0x7f22e007e6a0 unknown :-1 s=CLOSED pgs=1295 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:50.860 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.857 7f22e6346700 1 -- 172.21.2.112:0/3532784238 >> 172.21.2.112:0/3532784238 conn(0x7f22e00ce690 msgr2=0x7f22e00e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:50.861 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.857 7f22e6346700 1 -- 172.21.2.112:0/3532784238 shutdown_connections 2021-02-10T11:33:50.861 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.857 7f22e6346700 1 -- 172.21.2.112:0/3532784238 wait complete. 2021-02-10T11:33:50.862 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.872 7f22e6346700 1 Processor -- start 2021-02-10T11:33:50.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.872 7f22e6346700 1 -- start start 2021-02-10T11:33:50.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.873 7f22e6346700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f22e01b19d0 0x7f22e008cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:50.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.873 7f22e6346700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f22e01b0d00 0x7f22e01b1100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:50.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.873 7f22e6346700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f22e01a2b50 0x7f22e01a2f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:50.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.873 7f22e6346700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22e01b1720 con 0x7f22e01a2b50 2021-02-10T11:33:50.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.873 7f22e6346700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22e008c470 con 0x7f22e01b0d00 2021-02-10T11:33:50.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.873 7f22e6346700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22e01b1f50 con 0x7f22e01b19d0 2021-02-10T11:33:50.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.873 7f22e4963700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f22e01a2b50 0x7f22e01a2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:50.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.873 7f22e4963700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f22e01a2b50 0x7f22e01a2f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:50084/0 (socket says 172.21.2.112:50084) 2021-02-10T11:33:50.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.873 7f22e4963700 1 -- 172.21.2.112:0/284990379 learned_addr learned my addr 172.21.2.112:0/284990379 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:33:50.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.873 7f22e4963700 1 -- 172.21.2.112:0/284990379 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f22e01b19d0 msgr2=0x7f22e008cd90 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:33:50.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.873 7f22e4963700 1 --2- 172.21.2.112:0/284990379 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f22e01b19d0 0x7f22e008cd90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:50.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.873 7f22e4963700 1 -- 172.21.2.112:0/284990379 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f22e01b0d00 msgr2=0x7f22e01b1100 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:33:50.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.873 7f22e4963700 1 --2- 172.21.2.112:0/284990379 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f22e01b0d00 0x7f22e01b1100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:50.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.873 7f22e4963700 1 -- 172.21.2.112:0/284990379 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22e01a3570 con 0x7f22e01a2b50 2021-02-10T11:33:50.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.873 7f22e4963700 1 --2- 172.21.2.112:0/284990379 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f22e01a2b50 0x7f22e01a2f50 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 rx=0x7f22d8009980 tx=0x7f22d800bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:50.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.874 7f22dcff9700 1 -- 172.21.2.112:0/284990379 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f22d8003020 con 0x7f22e01a2b50 2021-02-10T11:33:50.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.874 7f22dcff9700 1 -- 172.21.2.112:0/284990379 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22d8005a80 con 0x7f22e01a2b50 2021-02-10T11:33:50.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.874 7f22dcff9700 1 -- 172.21.2.112:0/284990379 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f22d8006030 con 0x7f22e01a2b50 2021-02-10T11:33:50.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.874 7f22e6346700 1 -- 172.21.2.112:0/284990379 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22e01a3570 con 0x7f22e01a2b50 2021-02-10T11:33:50.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.874 7f22e6346700 1 -- 172.21.2.112:0/284990379 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22e00b3330 con 0x7f22e01a2b50 2021-02-10T11:33:50.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.874 7f22dcff9700 1 -- 172.21.2.112:0/284990379 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f22d8005c60 con 0x7f22e01a2b50 2021-02-10T11:33:50.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.874 7f22dcff9700 1 --2- 172.21.2.112:0/284990379 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f22c00186c0 0x7f22c001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:50.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.874 7f22dcff9700 1 -- 172.21.2.112:0/284990379 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f22d80327a0 con 0x7f22e01a2b50 2021-02-10T11:33:50.865 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.875 7f22dffff700 1 --2- 172.21.2.112:0/284990379 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f22c00186c0 0x7f22c001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:50.865 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.875 7f22dffff700 1 --2- 172.21.2.112:0/284990379 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f22c00186c0 0x7f22c001abc0 crc :-1 s=READY pgs=642 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:50.892 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.905 7f22e6346700 1 -- 172.21.2.112:0/284990379 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22e0083dd0 con 0x7f22e01a2b50 2021-02-10T11:33:50.894 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:50.908 7f22dcff9700 1 -- 172.21.2.112:0/284990379 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f22d8003020 con 0x7f22e01a2b50 2021-02-10T11:33:51.027 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:51.040 7f22e6346700 1 -- 172.21.2.112:0/284990379 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f22e007ed80 con 0x7f22e01a2b50 2021-02-10T11:33:51.027 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:51.040 7f22dcff9700 1 -- 172.21.2.112:0/284990379 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v35) v1 ==== 80+0+4855 (secure 0 0 0) 0x7f22d8005c60 con 0x7f22e01a2b50 2021-02-10T11:33:51.028 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:33:51.028 INFO:teuthology.orchestra.run.gibba012.stdout:{ 2021-02-10T11:33:51.028 INFO:teuthology.orchestra.run.gibba012.stdout: "auth_dump": [ 2021-02-10T11:33:51.028 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:51.028 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "mds.a", 2021-02-10T11:33:51.028 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCKvyNgPkV4AxAAEmVNaghdccP6lWd8hvQnxw==", 2021-02-10T11:33:51.028 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:33:51.028 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:33:51.028 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:33:51.028 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:33:51.028 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:33:51.029 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:33:51.029 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:51.029 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:51.029 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "mds.b", 2021-02-10T11:33:51.029 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCKvyNguxONLhAAHIpry+R89MAwdhFJyEhVCw==", 2021-02-10T11:33:51.029 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:33:51.029 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:33:51.029 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:33:51.029 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:33:51.029 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:33:51.029 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:33:51.029 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:51.029 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:51.030 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.0", 2021-02-10T11:33:51.030 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCSvyNgXcKPLhAAP/Bj2F+06DCtPCzQDNq+cQ==", 2021-02-10T11:33:51.030 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:33:51.030 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:33:51.030 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:33:51.030 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:33:51.030 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:33:51.030 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:51.030 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:51.030 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.1", 2021-02-10T11:33:51.030 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCVvyNghfbcLhAA66PrRwR2M7o6VTF5UhU/Fg==", 2021-02-10T11:33:51.030 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:33:51.030 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:33:51.031 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:33:51.031 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:33:51.031 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:33:51.031 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:51.031 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:51.031 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.2", 2021-02-10T11:33:51.031 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCYvyNgwiPqLhAA66BVkRsgaUlGpAsQ15Nf8Q==", 2021-02-10T11:33:51.031 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:33:51.031 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:33:51.031 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:33:51.031 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:33:51.031 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:33:51.032 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:51.032 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:51.032 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.3", 2021-02-10T11:33:51.032 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCbvyNgqBZrLxAAERPe/1L3SXAEzI3d9ReHHg==", 2021-02-10T11:33:51.032 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:33:51.033 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:33:51.033 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:33:51.033 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:33:51.033 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:33:51.033 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:51.033 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:51.033 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.4", 2021-02-10T11:33:51.033 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCkvyNgDQsxIhAAHmVjVYnx20N4SDmKqvbcuA==", 2021-02-10T11:33:51.033 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:33:51.034 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:33:51.034 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:33:51.034 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:33:51.034 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:33:51.034 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:51.034 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:51.034 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.5", 2021-02-10T11:33:51.034 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCnvyNg8ITFIhAADd4aKybAV7btoGPx3kCg3A==", 2021-02-10T11:33:51.034 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:33:51.034 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:33:51.034 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:33:51.034 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:33:51.034 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:33:51.035 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:51.035 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:51.035 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.6", 2021-02-10T11:33:51.035 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCqvyNgxOhDIxAAzXixEa/nlDc1xXrg0aeLwg==", 2021-02-10T11:33:51.035 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:33:51.035 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:33:51.035 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:33:51.035 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:33:51.035 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:33:51.035 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:51.035 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:51.035 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.7", 2021-02-10T11:33:51.035 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCtvyNgxSqGIxAACxJQlwP7DpiVVCZHA3zn1A==", 2021-02-10T11:33:51.035 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:33:51.036 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:33:51.036 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:33:51.036 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:33:51.036 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:33:51.036 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:51.036 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:51.036 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.0", 2021-02-10T11:33:51.036 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCLvyNgDkNiHRAAygU8Za4PBusTqCgwAYWTiQ==", 2021-02-10T11:33:51.036 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:33:51.036 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:33:51.036 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow r", 2021-02-10T11:33:51.036 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:33:51.036 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:33:51.037 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:51.037 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:51.037 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.1", 2021-02-10T11:33:51.037 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCLvyNgxmLqMxAA1MlSDJ9LH9u4f2sL7mm8/Q==", 2021-02-10T11:33:51.037 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:33:51.037 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:33:51.037 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow r", 2021-02-10T11:33:51.037 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:33:51.037 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:33:51.037 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:51.037 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:51.037 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.2", 2021-02-10T11:33:51.037 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCMvyNgp8veDhAAniSVHYPtwwTog/pjbaRQmw==", 2021-02-10T11:33:51.038 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:33:51.038 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:33:51.038 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow r", 2021-02-10T11:33:51.038 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:33:51.038 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:33:51.038 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:51.038 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:51.038 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.3", 2021-02-10T11:33:51.038 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCMvyNgQEVoJRAApHcWdm1sldIxu+heyD4HdQ==", 2021-02-10T11:33:51.038 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:33:51.038 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:33:51.038 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow r", 2021-02-10T11:33:51.038 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:33:51.039 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:33:51.039 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:51.039 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:51.039 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.admin", 2021-02-10T11:33:51.039 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCGvyNg7jjJBRAAtot+ypUZOiTo1GbY+oVhVw==", 2021-02-10T11:33:51.039 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:33:51.039 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow *", 2021-02-10T11:33:51.039 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:33:51.039 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:33:51.039 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:33:51.039 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:33:51.039 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:51.039 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:51.040 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.alice", 2021-02-10T11:33:51.040 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCexCNgfqx8GBAANue1hs5gTtkZO/pClbOpEw==", 2021-02-10T11:33:51.040 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:33:51.040 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow rw path=/volumes/group_0000000000544641/subvolume_0000000000670007", 2021-02-10T11:33:51.040 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow r", 2021-02-10T11:33:51.040 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:33:51.040 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:33:51.040 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:51.040 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:51.040 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "mgr.x", 2021-02-10T11:33:51.040 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCJvyNgTPVwKBAAjduiBE8+RSXBv82tdPecWg==", 2021-02-10T11:33:51.040 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:33:51.040 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow *", 2021-02-10T11:33:51.041 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow profile mgr", 2021-02-10T11:33:51.041 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:33:51.041 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:33:51.041 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:51.041 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:33:51.041 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "mgr.y", 2021-02-10T11:33:51.041 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCJvyNgQ1tTDhAASwBssDy+Rz31ujw8fNlWow==", 2021-02-10T11:33:51.041 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:33:51.041 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow *", 2021-02-10T11:33:51.041 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow profile mgr", 2021-02-10T11:33:51.041 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:33:51.041 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:33:51.041 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:33:51.041 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:33:51.042 INFO:teuthology.orchestra.run.gibba012.stdout:} 2021-02-10T11:33:51.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:51.042 7f22e6346700 1 -- 172.21.2.112:0/284990379 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f22c00186c0 msgr2=0x7f22c001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:51.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:51.042 7f22e6346700 1 --2- 172.21.2.112:0/284990379 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f22c00186c0 0x7f22c001abc0 crc :-1 s=READY pgs=642 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:51.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:51.042 7f22e6346700 1 -- 172.21.2.112:0/284990379 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f22e01a2b50 msgr2=0x7f22e01a2f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:51.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:51.042 7f22e6346700 1 --2- 172.21.2.112:0/284990379 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f22e01a2b50 0x7f22e01a2f50 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 rx=0x7f22d8009980 tx=0x7f22d800bcd0).stop 2021-02-10T11:33:51.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:51.042 7f22e6346700 1 -- 172.21.2.112:0/284990379 shutdown_connections 2021-02-10T11:33:51.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:51.042 7f22e6346700 1 --2- 172.21.2.112:0/284990379 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f22c00186c0 0x7f22c001abc0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:51.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:51.042 7f22e6346700 1 --2- 172.21.2.112:0/284990379 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f22e01b0d00 0x7f22e01b1100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:51.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:51.042 7f22e6346700 1 --2- 172.21.2.112:0/284990379 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f22e01a2b50 0x7f22e01a2f50 unknown :-1 s=CLOSED pgs=1296 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:51.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:51.042 7f22e6346700 1 --2- 172.21.2.112:0/284990379 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f22e01b19d0 0x7f22e008cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:51.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:51.042 7f22e6346700 1 -- 172.21.2.112:0/284990379 >> 172.21.2.112:0/284990379 conn(0x7f22e0093720 msgr2=0x7f22e0084b80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:51.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:51.042 7f22e6346700 1 -- 172.21.2.112:0/284990379 shutdown_connections 2021-02-10T11:33:51.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:51.043 7f22e6346700 1 -- 172.21.2.112:0/284990379 wait complete. 2021-02-10T11:33:51.045 DEBUG:teuthology.orchestra.run.gibba012:> set -ex 2021-02-10T11:33:51.045 DEBUG:teuthology.orchestra.run.gibba012:> sudo dd of=/etc/ceph/ceph.client.alice.keyring 2021-02-10T11:33:51.388 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set client.alice debug_client 20 2021-02-10T11:33:51.819 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:51.816 7f72860d2700 1 Processor -- start 2021-02-10T11:33:51.819 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:51.816 7f72860d2700 1 -- start start 2021-02-10T11:33:51.819 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:51.816 7f72860d2700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f72801b0870 0x7f72800f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:51.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:51.816 7f72860d2700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f72800efe00 0x7f72800f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:51.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:51.817 7f72860d2700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72801b0c70 con 0x7f72801b0470 2021-02-10T11:33:51.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:51.817 7f72860d2700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72800f0b50 con 0x7f72800efe00 2021-02-10T11:33:51.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:51.817 7f72860d2700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72800f0820 con 0x7f72801b0870 2021-02-10T11:33:51.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:51.817 7f727ffff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f72800efe00 0x7f72800f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:51.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:51.817 7f727ffff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f72800efe00 0x7f72800f0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:50090/0 (socket says 172.21.2.112:50090) 2021-02-10T11:33:51.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:51.817 7f727ffff700 1 -- 172.21.2.112:0/1646922308 learned_addr learned my addr 172.21.2.112:0/1646922308 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:33:51.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:51.817 7f727effd700 1 --2- 172.21.2.112:0/1646922308 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f72801b0870 0x7f72800f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:51.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:51.817 7f727ffff700 1 -- 172.21.2.112:0/1646922308 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f72801b0870 msgr2=0x7f72800f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:51.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:51.817 7f727ffff700 1 --2- 172.21.2.112:0/1646922308 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f72801b0870 0x7f72800f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:51.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:51.817 7f727ffff700 1 -- 172.21.2.112:0/1646922308 >> v1:172.21.2.120:6789/0 conn(0x7f72801b0470 legacy=0x7f72800ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:51.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:51.817 7f727ffff700 1 -- 172.21.2.112:0/1646922308 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72800e52e0 con 0x7f72800efe00 2021-02-10T11:33:51.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:51.817 7f727ffff700 1 --2- 172.21.2.112:0/1646922308 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f72800efe00 0x7f72800f0200 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 rx=0x7f7278006f00 tx=0x7f7278009f10).ready entity=mon.0 client_cookie=2ac8ea192ddf2569 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:51.821 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:51.817 7f727e7fc700 1 -- 172.21.2.112:0/1646922308 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f727800b3b0 con 0x7f72800efe00 2021-02-10T11:33:51.821 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:51.818 7f727e7fc700 1 -- 172.21.2.112:0/1646922308 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f727800b730 con 0x7f72800efe00 2021-02-10T11:33:51.821 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:51.818 7f727e7fc700 1 -- 172.21.2.112:0/1646922308 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f727800bc10 con 0x7f72800efe00 2021-02-10T11:33:51.821 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:51.818 7f72860d2700 1 -- 172.21.2.112:0/1646922308 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f72800efe00 msgr2=0x7f72800f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:51.821 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:51.818 7f72860d2700 1 --2- 172.21.2.112:0/1646922308 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f72800efe00 0x7f72800f0200 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 rx=0x7f7278006f00 tx=0x7f7278009f10).stop 2021-02-10T11:33:51.821 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:51.818 7f72860d2700 1 -- 172.21.2.112:0/1646922308 shutdown_connections 2021-02-10T11:33:51.821 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:51.818 7f72860d2700 1 --2- 172.21.2.112:0/1646922308 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f72800efe00 0x7f72800f0200 unknown :-1 s=CLOSED pgs=1297 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:51.821 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:51.818 7f72860d2700 1 --2- 172.21.2.112:0/1646922308 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f72801b0870 0x7f72800f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:51.821 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:51.818 7f72860d2700 1 -- 172.21.2.112:0/1646922308 >> 172.21.2.112:0/1646922308 conn(0x7f72800ce3e0 msgr2=0x7f72800e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:51.821 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:51.818 7f72860d2700 1 -- 172.21.2.112:0/1646922308 shutdown_connections 2021-02-10T11:33:51.821 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:51.818 7f72860d2700 1 -- 172.21.2.112:0/1646922308 wait complete. 2021-02-10T11:33:51.821 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:51.832 7f72860d2700 1 Processor -- start 2021-02-10T11:33:51.821 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:51.833 7f72860d2700 1 -- start start 2021-02-10T11:33:51.823 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:51.833 7f72860d2700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f728019dc90 0x7f728019e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:51.823 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:51.833 7f72860d2700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f72800bc7b0 0x7f72801062f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:51.823 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:51.833 7f72860d2700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7280106910 0x7f72800c0780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:51.823 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:51.833 7f72860d2700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72800bfeb0 con 0x7f7280106910 2021-02-10T11:33:51.823 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:51.833 7f72860d2700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72800933b0 con 0x7f728019dc90 2021-02-10T11:33:51.823 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:51.833 7f72860d2700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72800c0da0 con 0x7f72800bc7b0 2021-02-10T11:33:51.823 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:51.833 7f727ffff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7280106910 0x7f72800c0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:51.823 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:51.833 7f727ffff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7280106910 0x7f72800c0780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:50094/0 (socket says 172.21.2.112:50094) 2021-02-10T11:33:51.823 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:51.833 7f727ffff700 1 -- 172.21.2.112:0/2295724999 learned_addr learned my addr 172.21.2.112:0/2295724999 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:33:51.823 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:51.833 7f727ffff700 1 -- 172.21.2.112:0/2295724999 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f72800bc7b0 msgr2=0x7f72801062f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:51.823 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:51.833 7f727ffff700 1 --2- 172.21.2.112:0/2295724999 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f72800bc7b0 0x7f72801062f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:51.823 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:51.833 7f727ffff700 1 -- 172.21.2.112:0/2295724999 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f728019dc90 msgr2=0x7f728019e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:51.824 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:51.833 7f727ffff700 1 --2- 172.21.2.112:0/2295724999 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f728019dc90 0x7f728019e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:51.824 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:51.833 7f727ffff700 1 -- 172.21.2.112:0/2295724999 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72800c0fc0 con 0x7f7280106910 2021-02-10T11:33:51.824 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:51.834 7f727ffff700 1 --2- 172.21.2.112:0/2295724999 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7280106910 0x7f72800c0780 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 rx=0x7f7278009390 tx=0x7f727800bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:51.824 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:51.834 7f7267fff700 1 -- 172.21.2.112:0/2295724999 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7278003030 con 0x7f7280106910 2021-02-10T11:33:51.824 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:51.834 7f7267fff700 1 -- 172.21.2.112:0/2295724999 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f727800c2f0 con 0x7f7280106910 2021-02-10T11:33:51.824 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:51.834 7f72860d2700 1 -- 172.21.2.112:0/2295724999 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72800c0fc0 con 0x7f7280106910 2021-02-10T11:33:51.824 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:51.834 7f72860d2700 1 -- 172.21.2.112:0/2295724999 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72800c1550 con 0x7f7280106910 2021-02-10T11:33:51.824 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:51.834 7f7267fff700 1 -- 172.21.2.112:0/2295724999 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f727800c810 con 0x7f7280106910 2021-02-10T11:33:51.824 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:51.835 7f7267fff700 1 -- 172.21.2.112:0/2295724999 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7278005820 con 0x7f7280106910 2021-02-10T11:33:51.824 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:51.835 7f7267fff700 1 --2- 172.21.2.112:0/2295724999 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f72600186c0 0x7f726001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:51.824 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:51.835 7f7267fff700 1 -- 172.21.2.112:0/2295724999 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f7278032900 con 0x7f7280106910 2021-02-10T11:33:51.824 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:51.835 7f727f7fe700 1 --2- 172.21.2.112:0/2295724999 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f72600186c0 0x7f726001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:51.824 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:51.836 7f727f7fe700 1 --2- 172.21.2.112:0/2295724999 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f72600186c0 0x7f726001abc0 crc :-1 s=READY pgs=643 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:51.853 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:51.867 7f72860d2700 1 -- 172.21.2.112:0/2295724999 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7280105ae0 con 0x7f7280106910 2021-02-10T11:33:51.856 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:51.869 7f7267fff700 1 -- 172.21.2.112:0/2295724999 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f72780260a0 con 0x7f7280106910 2021-02-10T11:33:52.007 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:52.020 7f72860d2700 1 -- 172.21.2.112:0/2295724999 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command([{prefix=config set, name=debug_client}] v 0) v1 -- 0x7f72800f0b50 con 0x7f7280106910 2021-02-10T11:33:52.010 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:52.023 7f7267fff700 1 -- 172.21.2.112:0/2295724999 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{prefix=config set, name=debug_client}]=0 v9)=0 v9) v1 ==== 120+0+0 (secure 0 0 0) 0x7f72780260a0 con 0x7f7280106910 2021-02-10T11:33:52.015 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:52.028 7f72860d2700 1 -- 172.21.2.112:0/2295724999 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f72600186c0 msgr2=0x7f726001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:52.015 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:52.028 7f72860d2700 1 --2- 172.21.2.112:0/2295724999 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f72600186c0 0x7f726001abc0 crc :-1 s=READY pgs=643 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:52.015 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:52.028 7f72860d2700 1 -- 172.21.2.112:0/2295724999 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7280106910 msgr2=0x7f72800c0780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:52.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:52.028 7f72860d2700 1 --2- 172.21.2.112:0/2295724999 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7280106910 0x7f72800c0780 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 rx=0x7f7278009390 tx=0x7f727800bd40).stop 2021-02-10T11:33:52.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:52.028 7f72860d2700 1 -- 172.21.2.112:0/2295724999 shutdown_connections 2021-02-10T11:33:52.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:52.028 7f72860d2700 1 --2- 172.21.2.112:0/2295724999 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f72600186c0 0x7f726001abc0 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:52.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:52.028 7f72860d2700 1 --2- 172.21.2.112:0/2295724999 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7280106910 0x7f72800c0780 unknown :-1 s=CLOSED pgs=1298 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:52.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:52.028 7f72860d2700 1 --2- 172.21.2.112:0/2295724999 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f72800bc7b0 0x7f72801062f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:52.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:52.028 7f72860d2700 1 --2- 172.21.2.112:0/2295724999 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f728019dc90 0x7f728019e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:52.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:52.028 7f72860d2700 1 -- 172.21.2.112:0/2295724999 >> 172.21.2.112:0/2295724999 conn(0x7f728009dfd0 msgr2=0x7f728009abf0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:52.017 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:52.029 7f72860d2700 1 -- 172.21.2.112:0/2295724999 shutdown_connections 2021-02-10T11:33:52.017 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:52.029 7f72860d2700 1 -- 172.21.2.112:0/2295724999 wait complete. 2021-02-10T11:33:52.024 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set client.alice debug_objecter 20 2021-02-10T11:33:52.451 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:52.449 7fae2692d700 1 Processor -- start 2021-02-10T11:33:52.452 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:52.449 7fae2692d700 1 -- start start 2021-02-10T11:33:52.452 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:52.449 7fae2692d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fae201b0470 0x7fae201b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:52.452 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:52.449 7fae2692d700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae200f0b50 con 0x7fae200ef610 2021-02-10T11:33:52.452 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:52.449 7fae2692d700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae200f24e0 con 0x7fae200f20e0 2021-02-10T11:33:52.452 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:52.449 7fae2692d700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae200e5ea0 con 0x7fae201b0470 2021-02-10T11:33:52.452 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:52.449 7fae1ffff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fae201b0470 0x7fae201b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:52.452 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:52.450 7fae1f7fe700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fae200ef610 0x7fae200efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:49562/0 (socket says 172.21.2.112:49562) 2021-02-10T11:33:52.452 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:52.450 7fae1f7fe700 1 -- 172.21.2.112:0/3410004302 learned_addr learned my addr 172.21.2.112:0/3410004302 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:33:52.452 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:52.450 7fae1ffff700 1 -- 172.21.2.112:0/3410004302 >> v1:172.21.2.120:6789/0 conn(0x7fae200f20e0 legacy=0x7fae200f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:52.452 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:52.450 7fae1ffff700 1 -- 172.21.2.112:0/3410004302 >> v1:172.21.2.112:6789/0 conn(0x7fae200ef610 legacy=0x7fae200efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:52.452 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:52.450 7fae1ffff700 1 -- 172.21.2.112:0/3410004302 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae200e52e0 con 0x7fae201b0470 2021-02-10T11:33:52.453 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:52.450 7fae1ffff700 1 --2- 172.21.2.112:0/3410004302 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fae201b0470 0x7fae201b0870 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 rx=0x7fae10006650 tx=0x7fae10009a80).ready entity=mon.0 client_cookie=7809fcc04f3b5485 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:52.453 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:52.450 7fae1effd700 1 -- 172.21.2.112:0/3410004302 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fae100009b0 con 0x7fae201b0470 2021-02-10T11:33:52.453 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:52.450 7fae1effd700 1 -- 172.21.2.112:0/3410004302 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae10006740 con 0x7fae201b0470 2021-02-10T11:33:52.453 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:52.450 7fae1effd700 1 -- 172.21.2.112:0/3410004302 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fae1000c0c0 con 0x7fae201b0470 2021-02-10T11:33:52.453 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:52.450 7fae2692d700 1 -- 172.21.2.112:0/3410004302 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fae201b0470 msgr2=0x7fae201b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:52.453 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:52.450 7fae2692d700 1 --2- 172.21.2.112:0/3410004302 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fae201b0470 0x7fae201b0870 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 rx=0x7fae10006650 tx=0x7fae10009a80).stop 2021-02-10T11:33:52.453 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:52.451 7fae2692d700 1 -- 172.21.2.112:0/3410004302 shutdown_connections 2021-02-10T11:33:52.453 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:52.451 7fae2692d700 1 --2- 172.21.2.112:0/3410004302 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fae201b0470 0x7fae201b0870 unknown :-1 s=CLOSED pgs=1300 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:52.453 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:52.451 7fae2692d700 1 -- 172.21.2.112:0/3410004302 >> 172.21.2.112:0/3410004302 conn(0x7fae200ce3e0 msgr2=0x7fae200e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:52.453 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:52.451 7fae2692d700 1 -- 172.21.2.112:0/3410004302 shutdown_connections 2021-02-10T11:33:52.453 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:52.451 7fae2692d700 1 -- 172.21.2.112:0/3410004302 wait complete. 2021-02-10T11:33:52.455 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:52.465 7fae2692d700 1 Processor -- start 2021-02-10T11:33:52.455 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:52.465 7fae2692d700 1 -- start start 2021-02-10T11:33:52.455 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:52.466 7fae2692d700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fae201be110 0x7fae20086bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:52.455 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:52.466 7fae2692d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fae20087210 0x7fae20085910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:52.455 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:52.466 7fae2692d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fae200a4ef0 0x7fae20085f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:52.455 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:52.466 7fae2692d700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae200828e0 con 0x7fae20087210 2021-02-10T11:33:52.455 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:52.466 7fae2692d700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae20086550 con 0x7fae201be110 2021-02-10T11:33:52.455 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:52.466 7fae2692d700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae20086740 con 0x7fae200a4ef0 2021-02-10T11:33:52.456 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:52.466 7fae1f7fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fae20087210 0x7fae20085910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:52.456 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:52.466 7fae1f7fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fae20087210 0x7fae20085910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:50104/0 (socket says 172.21.2.112:50104) 2021-02-10T11:33:52.456 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:52.466 7fae1f7fe700 1 -- 172.21.2.112:0/69456974 learned_addr learned my addr 172.21.2.112:0/69456974 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:33:52.456 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:52.466 7fae1f7fe700 1 -- 172.21.2.112:0/69456974 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fae200a4ef0 msgr2=0x7fae20085f30 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:33:52.456 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:52.466 7fae1f7fe700 1 --2- 172.21.2.112:0/69456974 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fae200a4ef0 0x7fae20085f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:52.456 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:52.466 7fae1f7fe700 1 -- 172.21.2.112:0/69456974 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fae201be110 msgr2=0x7fae20086bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:52.456 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:52.466 7fae1f7fe700 1 --2- 172.21.2.112:0/69456974 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fae201be110 0x7fae20086bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:52.456 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:52.466 7fae1f7fe700 1 -- 172.21.2.112:0/69456974 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae20086960 con 0x7fae20087210 2021-02-10T11:33:52.456 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:52.467 7fae1f7fe700 1 --2- 172.21.2.112:0/69456974 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fae20087210 0x7fae20085910 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 rx=0x7fae14001c60 tx=0x7fae1400a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:52.456 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:52.467 7fae1cff9700 1 -- 172.21.2.112:0/69456974 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fae1400acc0 con 0x7fae20087210 2021-02-10T11:33:52.456 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:52.467 7fae1cff9700 1 -- 172.21.2.112:0/69456974 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae14004140 con 0x7fae20087210 2021-02-10T11:33:52.456 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:52.467 7fae1cff9700 1 -- 172.21.2.112:0/69456974 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fae14004620 con 0x7fae20087210 2021-02-10T11:33:52.456 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:52.467 7fae2692d700 1 -- 172.21.2.112:0/69456974 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae20086960 con 0x7fae20087210 2021-02-10T11:33:52.456 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:52.467 7fae2692d700 1 -- 172.21.2.112:0/69456974 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae20086960 con 0x7fae20087210 2021-02-10T11:33:52.457 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:52.468 7fae1cff9700 1 -- 172.21.2.112:0/69456974 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fae14000d90 con 0x7fae20087210 2021-02-10T11:33:52.457 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:52.468 7fae1cff9700 1 --2- 172.21.2.112:0/69456974 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fae000186c0 0x7fae0001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:52.457 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:52.468 7fae1cff9700 1 -- 172.21.2.112:0/69456974 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fae14032550 con 0x7fae20087210 2021-02-10T11:33:52.457 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:52.468 7fae1ffff700 1 --2- 172.21.2.112:0/69456974 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fae000186c0 0x7fae0001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:52.457 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:52.469 7fae1ffff700 1 --2- 172.21.2.112:0/69456974 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fae000186c0 0x7fae0001abc0 crc :-1 s=READY pgs=644 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:52.487 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:52.499 7fae2692d700 1 -- 172.21.2.112:0/69456974 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae2007e6b0 con 0x7fae20087210 2021-02-10T11:33:52.549 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:52.502 7fae1cff9700 1 -- 172.21.2.112:0/69456974 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fae140009b0 con 0x7fae20087210 2021-02-10T11:33:52.637 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:52.650 7fae2692d700 1 -- 172.21.2.112:0/69456974 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command([{prefix=config set, name=debug_objecter}] v 0) v1 -- 0x7fae200ce3e0 con 0x7fae20087210 2021-02-10T11:33:52.641 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:52.654 7fae1cff9700 1 -- 172.21.2.112:0/69456974 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{prefix=config set, name=debug_objecter}]=0 v10)=0 v10) v1 ==== 122+0+0 (secure 0 0 0) 0x7fae14000d90 con 0x7fae20087210 2021-02-10T11:33:52.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:52.659 7fae2692d700 1 -- 172.21.2.112:0/69456974 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fae000186c0 msgr2=0x7fae0001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:52.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:52.659 7fae2692d700 1 --2- 172.21.2.112:0/69456974 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fae000186c0 0x7fae0001abc0 crc :-1 s=READY pgs=644 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:52.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:52.659 7fae2692d700 1 -- 172.21.2.112:0/69456974 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fae20087210 msgr2=0x7fae20085910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:52.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:52.659 7fae2692d700 1 --2- 172.21.2.112:0/69456974 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fae20087210 0x7fae20085910 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 rx=0x7fae14001c60 tx=0x7fae1400a8d0).stop 2021-02-10T11:33:52.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:52.659 7fae2692d700 1 -- 172.21.2.112:0/69456974 shutdown_connections 2021-02-10T11:33:52.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:52.659 7fae2692d700 1 --2- 172.21.2.112:0/69456974 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fae000186c0 0x7fae0001abc0 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:52.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:52.659 7fae2692d700 1 --2- 172.21.2.112:0/69456974 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fae200a4ef0 0x7fae20085f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:52.648 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:52.659 7fae2692d700 1 --2- 172.21.2.112:0/69456974 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fae20087210 0x7fae20085910 unknown :-1 s=CLOSED pgs=1301 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:52.648 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:52.659 7fae2692d700 1 --2- 172.21.2.112:0/69456974 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fae201be110 0x7fae20086bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:52.648 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:52.659 7fae2692d700 1 -- 172.21.2.112:0/69456974 >> 172.21.2.112:0/69456974 conn(0x7fae2008c540 msgr2=0x7fae200ae2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:52.648 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:52.660 7fae2692d700 1 -- 172.21.2.112:0/69456974 shutdown_connections 2021-02-10T11:33:52.648 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:52.660 7fae2692d700 1 -- 172.21.2.112:0/69456974 wait complete. 2021-02-10T11:33:52.658 DEBUG:teuthology.orchestra.run.gibba012:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2021-02-10T11:33:52.665 DEBUG:teuthology.orchestra.run.gibba020:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2021-02-10T11:33:53.023 INFO:teuthology.orchestra.run.gibba012.stdout:[global] 2021-02-10T11:33:53.023 INFO:teuthology.orchestra.run.gibba012.stdout: chdir = "" 2021-02-10T11:33:53.024 INFO:teuthology.orchestra.run.gibba012.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2021-02-10T11:33:53.024 INFO:teuthology.orchestra.run.gibba012.stdout: auth supported = cephx 2021-02-10T11:33:53.024 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:33:53.024 INFO:teuthology.orchestra.run.gibba012.stdout: filestore xattr use omap = true 2021-02-10T11:33:53.024 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:33:53.024 INFO:teuthology.orchestra.run.gibba012.stdout: mon clock drift allowed = 1.000 2021-02-10T11:33:53.024 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:33:53.024 INFO:teuthology.orchestra.run.gibba012.stdout: osd crush chooseleaf type = 0 2021-02-10T11:33:53.024 INFO:teuthology.orchestra.run.gibba012.stdout: auth debug = true 2021-02-10T11:33:53.024 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:33:53.024 INFO:teuthology.orchestra.run.gibba012.stdout: ms die on old message = true 2021-02-10T11:33:53.024 INFO:teuthology.orchestra.run.gibba012.stdout: ms die on bug = true 2021-02-10T11:33:53.024 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:33:53.024 INFO:teuthology.orchestra.run.gibba012.stdout: mon max pg per osd = 10000 # >= luminous 2021-02-10T11:33:53.025 INFO:teuthology.orchestra.run.gibba012.stdout: mon pg warn max object skew = 0 2021-02-10T11:33:53.025 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:33:53.025 INFO:teuthology.orchestra.run.gibba012.stdout: osd pool default size = 2 2021-02-10T11:33:53.025 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:33:53.025 INFO:teuthology.orchestra.run.gibba012.stdout: mon osd allow primary affinity = true 2021-02-10T11:33:53.025 INFO:teuthology.orchestra.run.gibba012.stdout: mon osd allow pg remap = true 2021-02-10T11:33:53.025 INFO:teuthology.orchestra.run.gibba012.stdout: mon warn on legacy crush tunables = false 2021-02-10T11:33:53.025 INFO:teuthology.orchestra.run.gibba012.stdout: mon warn on crush straw calc version zero = false 2021-02-10T11:33:53.025 INFO:teuthology.orchestra.run.gibba012.stdout: mon warn on no sortbitwise = false 2021-02-10T11:33:53.025 INFO:teuthology.orchestra.run.gibba012.stdout: mon warn on osd down out interval zero = false 2021-02-10T11:33:53.025 INFO:teuthology.orchestra.run.gibba012.stdout: mon warn on too few osds = false 2021-02-10T11:33:53.025 INFO:teuthology.orchestra.run.gibba012.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2021-02-10T11:33:53.025 INFO:teuthology.orchestra.run.gibba012.stdout: mon_warn_on_pool_no_redundancy = false 2021-02-10T11:33:53.026 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:33:53.026 INFO:teuthology.orchestra.run.gibba012.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2021-02-10T11:33:53.026 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:33:53.026 INFO:teuthology.orchestra.run.gibba012.stdout: osd default data pool replay window = 5 2021-02-10T11:33:53.026 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:33:53.026 INFO:teuthology.orchestra.run.gibba012.stdout: mon allow pool delete = true 2021-02-10T11:33:53.026 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:33:53.026 INFO:teuthology.orchestra.run.gibba012.stdout: mon cluster log file level = debug 2021-02-10T11:33:53.026 INFO:teuthology.orchestra.run.gibba012.stdout: debug asserts on shutdown = true 2021-02-10T11:33:53.026 INFO:teuthology.orchestra.run.gibba012.stdout: mon health detail to clog = false 2021-02-10T11:33:53.026 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:33:53.026 INFO:teuthology.orchestra.run.gibba012.stdout: # we see this fail in qa on *nautilus*; bump up retries 2021-02-10T11:33:53.026 INFO:teuthology.orchestra.run.gibba012.stdout: mon_client_directed_command_retry = 4 2021-02-10T11:33:53.027 INFO:teuthology.orchestra.run.gibba012.stdout: mon host = "172.21.2.112,172.21.2.120,[v2:172.21.2.120:3301,v1:172.21.2.120:6790]" 2021-02-10T11:33:53.027 INFO:teuthology.orchestra.run.gibba012.stdout: fsid = f14f004e-0699-4584-ad81-3754258493de 2021-02-10T11:33:53.027 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:33:53.027 INFO:teuthology.orchestra.run.gibba012.stdout:[osd] 2021-02-10T11:33:53.027 INFO:teuthology.orchestra.run.gibba012.stdout: osd journal size = 100 2021-02-10T11:33:53.027 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:33:53.027 INFO:teuthology.orchestra.run.gibba012.stdout: osd scrub load threshold = 5.0 2021-02-10T11:33:53.027 INFO:teuthology.orchestra.run.gibba012.stdout: osd scrub max interval = 600 2021-02-10T11:33:53.027 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:33:53.027 INFO:teuthology.orchestra.run.gibba012.stdout: osd recover clone overlap = true 2021-02-10T11:33:53.027 INFO:teuthology.orchestra.run.gibba012.stdout: osd recovery max chunk = 1048576 2021-02-10T11:33:53.027 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:33:53.027 INFO:teuthology.orchestra.run.gibba012.stdout: osd debug shutdown = true 2021-02-10T11:33:53.028 INFO:teuthology.orchestra.run.gibba012.stdout: osd debug op order = true 2021-02-10T11:33:53.028 INFO:teuthology.orchestra.run.gibba012.stdout: osd debug verify stray on activate = true 2021-02-10T11:33:53.028 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:33:53.028 INFO:teuthology.orchestra.run.gibba012.stdout: osd open classes on start = true 2021-02-10T11:33:53.028 INFO:teuthology.orchestra.run.gibba012.stdout: osd debug pg log writeout = true 2021-02-10T11:33:53.028 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:33:53.028 INFO:teuthology.orchestra.run.gibba012.stdout: osd deep scrub update digest min age = 30 2021-02-10T11:33:53.028 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:33:53.028 INFO:teuthology.orchestra.run.gibba012.stdout: osd map max advance = 10 2021-02-10T11:33:53.028 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:33:53.028 INFO:teuthology.orchestra.run.gibba012.stdout: journal zero on create = true 2021-02-10T11:33:53.028 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:33:53.028 INFO:teuthology.orchestra.run.gibba012.stdout: filestore ondisk finisher threads = 3 2021-02-10T11:33:53.029 INFO:teuthology.orchestra.run.gibba012.stdout: filestore apply finisher threads = 3 2021-02-10T11:33:53.029 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:33:53.029 INFO:teuthology.orchestra.run.gibba012.stdout: bdev debug aio = true 2021-02-10T11:33:53.029 INFO:teuthology.orchestra.run.gibba012.stdout: osd debug misdirected ops = true 2021-02-10T11:33:53.029 INFO:teuthology.orchestra.run.gibba012.stdout: bdev async discard = True 2021-02-10T11:33:53.029 INFO:teuthology.orchestra.run.gibba012.stdout: bdev enable discard = True 2021-02-10T11:33:53.029 INFO:teuthology.orchestra.run.gibba012.stdout: bluestore allocator = bitmap 2021-02-10T11:33:53.029 INFO:teuthology.orchestra.run.gibba012.stdout: bluestore block size = 96636764160 2021-02-10T11:33:53.029 INFO:teuthology.orchestra.run.gibba012.stdout: bluestore fsck on mount = True 2021-02-10T11:33:53.029 INFO:teuthology.orchestra.run.gibba012.stdout: debug bluefs = 20 2021-02-10T11:33:53.029 INFO:teuthology.orchestra.run.gibba012.stdout: debug bluestore = 20 2021-02-10T11:33:53.029 INFO:teuthology.orchestra.run.gibba012.stdout: debug filestore = 20 2021-02-10T11:33:53.029 INFO:teuthology.orchestra.run.gibba012.stdout: debug journal = 20 2021-02-10T11:33:53.029 INFO:teuthology.orchestra.run.gibba012.stdout: debug ms = 20 2021-02-10T11:33:53.030 INFO:teuthology.orchestra.run.gibba012.stdout: debug osd = 25 2021-02-10T11:33:53.030 INFO:teuthology.orchestra.run.gibba012.stdout: debug rocksdb = 10 2021-02-10T11:33:53.030 INFO:teuthology.orchestra.run.gibba012.stdout: mon osd backfillfull_ratio = 0.85 2021-02-10T11:33:53.030 INFO:teuthology.orchestra.run.gibba012.stdout: mon osd full ratio = 0.9 2021-02-10T11:33:53.030 INFO:teuthology.orchestra.run.gibba012.stdout: mon osd nearfull ratio = 0.8 2021-02-10T11:33:53.030 INFO:teuthology.orchestra.run.gibba012.stdout: osd failsafe full ratio = 0.95 2021-02-10T11:33:53.030 INFO:teuthology.orchestra.run.gibba012.stdout: osd objectstore = bluestore 2021-02-10T11:33:53.030 INFO:teuthology.orchestra.run.gibba012.stdout: osd op complaint time = 180 2021-02-10T11:33:53.030 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:33:53.030 INFO:teuthology.orchestra.run.gibba012.stdout:[mgr] 2021-02-10T11:33:53.030 INFO:teuthology.orchestra.run.gibba012.stdout: debug ms = 1 2021-02-10T11:33:53.030 INFO:teuthology.orchestra.run.gibba012.stdout: debug mgr = 20 2021-02-10T11:33:53.031 INFO:teuthology.orchestra.run.gibba012.stdout: debug mon = 20 2021-02-10T11:33:53.031 INFO:teuthology.orchestra.run.gibba012.stdout: debug auth = 20 2021-02-10T11:33:53.031 INFO:teuthology.orchestra.run.gibba012.stdout: mon reweight min pgs per osd = 4 2021-02-10T11:33:53.031 INFO:teuthology.orchestra.run.gibba012.stdout: mon reweight min bytes per osd = 10 2021-02-10T11:33:53.031 INFO:teuthology.orchestra.run.gibba012.stdout: debug client = 10 2021-02-10T11:33:53.031 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:33:53.031 INFO:teuthology.orchestra.run.gibba012.stdout:[mon] 2021-02-10T11:33:53.031 INFO:teuthology.orchestra.run.gibba012.stdout: debug ms = 1 2021-02-10T11:33:53.031 INFO:teuthology.orchestra.run.gibba012.stdout: debug mon = 20 2021-02-10T11:33:53.031 INFO:teuthology.orchestra.run.gibba012.stdout: debug paxos = 20 2021-02-10T11:33:53.031 INFO:teuthology.orchestra.run.gibba012.stdout: debug auth = 20 2021-02-10T11:33:53.031 INFO:teuthology.orchestra.run.gibba012.stdout: mon data avail warn = 5 2021-02-10T11:33:53.031 INFO:teuthology.orchestra.run.gibba012.stdout: mon mgr mkfs grace = 240 2021-02-10T11:33:53.032 INFO:teuthology.orchestra.run.gibba012.stdout: mon reweight min pgs per osd = 4 2021-02-10T11:33:53.032 INFO:teuthology.orchestra.run.gibba012.stdout: mon osd reporter subtree level = osd 2021-02-10T11:33:53.032 INFO:teuthology.orchestra.run.gibba012.stdout: mon osd prime pg temp = true 2021-02-10T11:33:53.032 INFO:teuthology.orchestra.run.gibba012.stdout: mon reweight min bytes per osd = 10 2021-02-10T11:33:53.032 INFO:teuthology.orchestra.run.gibba012.stdout: mon op complaint time = 120 2021-02-10T11:33:53.032 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:33:53.032 INFO:teuthology.orchestra.run.gibba012.stdout:[client] 2021-02-10T11:33:53.032 INFO:teuthology.orchestra.run.gibba012.stdout: rgw cache enabled = true 2021-02-10T11:33:53.032 INFO:teuthology.orchestra.run.gibba012.stdout: rgw enable ops log = true 2021-02-10T11:33:53.032 INFO:teuthology.orchestra.run.gibba012.stdout: rgw enable usage log = true 2021-02-10T11:33:53.032 INFO:teuthology.orchestra.run.gibba012.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2021-02-10T11:33:53.032 INFO:teuthology.orchestra.run.gibba012.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2021-02-10T11:33:53.032 INFO:teuthology.orchestra.run.gibba012.stdout: client mount timeout = 600 2021-02-10T11:33:53.032 INFO:teuthology.orchestra.run.gibba012.stdout: debug client = 20 2021-02-10T11:33:53.033 INFO:teuthology.orchestra.run.gibba012.stdout: debug ms = 1 2021-02-10T11:33:53.033 INFO:teuthology.orchestra.run.gibba012.stdout: pid file = "" 2021-02-10T11:33:53.033 INFO:teuthology.orchestra.run.gibba012.stdout:[mon.a] 2021-02-10T11:33:53.033 INFO:teuthology.orchestra.run.gibba012.stdout:[mon.b] 2021-02-10T11:33:53.033 INFO:teuthology.orchestra.run.gibba012.stdout:[mon.c] 2021-02-10T11:33:53.033 INFO:teuthology.orchestra.run.gibba012.stdout:[mds.a] 2021-02-10T11:33:53.033 INFO:teuthology.orchestra.run.gibba012.stdout:[mds.b] 2021-02-10T11:33:53.033 INFO:teuthology.orchestra.run.gibba012.stdout:[mds] 2021-02-10T11:33:53.033 INFO:teuthology.orchestra.run.gibba012.stdout: debug mds = 20 2021-02-10T11:33:53.033 INFO:teuthology.orchestra.run.gibba012.stdout: debug ms = 1 2021-02-10T11:33:53.033 INFO:teuthology.orchestra.run.gibba012.stdout: mds bal frag = True 2021-02-10T11:33:53.033 INFO:teuthology.orchestra.run.gibba012.stdout: mds bal fragment size max = 10000 2021-02-10T11:33:53.033 INFO:teuthology.orchestra.run.gibba012.stdout: mds bal merge size = 5 2021-02-10T11:33:53.034 INFO:teuthology.orchestra.run.gibba012.stdout: mds bal split bits = 3 2021-02-10T11:33:53.034 INFO:teuthology.orchestra.run.gibba012.stdout: mds bal split size = 100 2021-02-10T11:33:53.034 INFO:teuthology.orchestra.run.gibba012.stdout: mds debug frag = True 2021-02-10T11:33:53.034 INFO:teuthology.orchestra.run.gibba012.stdout: mds debug scatterstat = True 2021-02-10T11:33:53.034 INFO:teuthology.orchestra.run.gibba012.stdout: mds op complaint time = 180 2021-02-10T11:33:53.034 INFO:teuthology.orchestra.run.gibba012.stdout: mds verify scatter = True 2021-02-10T11:33:53.034 INFO:teuthology.orchestra.run.gibba012.stdout: osd op complaint time = 180 2021-02-10T11:33:53.034 INFO:teuthology.orchestra.run.gibba012.stdout:[client.alice] 2021-02-10T11:33:53.034 INFO:teuthology.orchestra.run.gibba012.stdout: keyring = /etc/ceph/ceph.client.alice.keyring 2021-02-10T11:33:53.035 INFO:teuthology.orchestra.run.gibba020.stdout:[global] 2021-02-10T11:33:53.035 INFO:teuthology.orchestra.run.gibba020.stdout: chdir = "" 2021-02-10T11:33:53.035 INFO:teuthology.orchestra.run.gibba020.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2021-02-10T11:33:53.035 INFO:teuthology.orchestra.run.gibba020.stdout: auth supported = cephx 2021-02-10T11:33:53.035 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:33:53.036 INFO:teuthology.orchestra.run.gibba020.stdout: filestore xattr use omap = true 2021-02-10T11:33:53.036 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:33:53.036 INFO:teuthology.orchestra.run.gibba020.stdout: mon clock drift allowed = 1.000 2021-02-10T11:33:53.036 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:33:53.036 INFO:teuthology.orchestra.run.gibba020.stdout: osd crush chooseleaf type = 0 2021-02-10T11:33:53.036 INFO:teuthology.orchestra.run.gibba020.stdout: auth debug = true 2021-02-10T11:33:53.036 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:33:53.036 INFO:teuthology.orchestra.run.gibba020.stdout: ms die on old message = true 2021-02-10T11:33:53.036 INFO:teuthology.orchestra.run.gibba020.stdout: ms die on bug = true 2021-02-10T11:33:53.036 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:33:53.036 INFO:teuthology.orchestra.run.gibba020.stdout: mon max pg per osd = 10000 # >= luminous 2021-02-10T11:33:53.036 INFO:teuthology.orchestra.run.gibba020.stdout: mon pg warn max object skew = 0 2021-02-10T11:33:53.036 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:33:53.036 INFO:teuthology.orchestra.run.gibba020.stdout: osd pool default size = 2 2021-02-10T11:33:53.037 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:33:53.037 INFO:teuthology.orchestra.run.gibba020.stdout: mon osd allow primary affinity = true 2021-02-10T11:33:53.037 INFO:teuthology.orchestra.run.gibba020.stdout: mon osd allow pg remap = true 2021-02-10T11:33:53.037 INFO:teuthology.orchestra.run.gibba020.stdout: mon warn on legacy crush tunables = false 2021-02-10T11:33:53.037 INFO:teuthology.orchestra.run.gibba020.stdout: mon warn on crush straw calc version zero = false 2021-02-10T11:33:53.037 INFO:teuthology.orchestra.run.gibba020.stdout: mon warn on no sortbitwise = false 2021-02-10T11:33:53.037 INFO:teuthology.orchestra.run.gibba020.stdout: mon warn on osd down out interval zero = false 2021-02-10T11:33:53.037 INFO:teuthology.orchestra.run.gibba020.stdout: mon warn on too few osds = false 2021-02-10T11:33:53.037 INFO:teuthology.orchestra.run.gibba020.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2021-02-10T11:33:53.037 INFO:teuthology.orchestra.run.gibba020.stdout: mon_warn_on_pool_no_redundancy = false 2021-02-10T11:33:53.037 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:33:53.037 INFO:teuthology.orchestra.run.gibba020.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2021-02-10T11:33:53.037 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:33:53.038 INFO:teuthology.orchestra.run.gibba020.stdout: osd default data pool replay window = 5 2021-02-10T11:33:53.038 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:33:53.038 INFO:teuthology.orchestra.run.gibba020.stdout: mon allow pool delete = true 2021-02-10T11:33:53.038 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:33:53.038 INFO:teuthology.orchestra.run.gibba020.stdout: mon cluster log file level = debug 2021-02-10T11:33:53.038 INFO:teuthology.orchestra.run.gibba020.stdout: debug asserts on shutdown = true 2021-02-10T11:33:53.038 INFO:teuthology.orchestra.run.gibba020.stdout: mon health detail to clog = false 2021-02-10T11:33:53.038 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:33:53.038 INFO:teuthology.orchestra.run.gibba020.stdout: # we see this fail in qa on *nautilus*; bump up retries 2021-02-10T11:33:53.038 INFO:teuthology.orchestra.run.gibba020.stdout: mon_client_directed_command_retry = 4 2021-02-10T11:33:53.038 INFO:teuthology.orchestra.run.gibba020.stdout: mon host = "172.21.2.112,172.21.2.120,[v2:172.21.2.120:3301,v1:172.21.2.120:6790]" 2021-02-10T11:33:53.038 INFO:teuthology.orchestra.run.gibba020.stdout: fsid = f14f004e-0699-4584-ad81-3754258493de 2021-02-10T11:33:53.038 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:33:53.039 INFO:teuthology.orchestra.run.gibba020.stdout:[osd] 2021-02-10T11:33:53.039 INFO:teuthology.orchestra.run.gibba020.stdout: osd journal size = 100 2021-02-10T11:33:53.039 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:33:53.039 INFO:teuthology.orchestra.run.gibba020.stdout: osd scrub load threshold = 5.0 2021-02-10T11:33:53.039 INFO:teuthology.orchestra.run.gibba020.stdout: osd scrub max interval = 600 2021-02-10T11:33:53.039 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:33:53.039 INFO:teuthology.orchestra.run.gibba020.stdout: osd recover clone overlap = true 2021-02-10T11:33:53.039 INFO:teuthology.orchestra.run.gibba020.stdout: osd recovery max chunk = 1048576 2021-02-10T11:33:53.039 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:33:53.039 INFO:teuthology.orchestra.run.gibba020.stdout: osd debug shutdown = true 2021-02-10T11:33:53.039 INFO:teuthology.orchestra.run.gibba020.stdout: osd debug op order = true 2021-02-10T11:33:53.039 INFO:teuthology.orchestra.run.gibba020.stdout: osd debug verify stray on activate = true 2021-02-10T11:33:53.040 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:33:53.040 INFO:teuthology.orchestra.run.gibba020.stdout: osd open classes on start = true 2021-02-10T11:33:53.040 INFO:teuthology.orchestra.run.gibba020.stdout: osd debug pg log writeout = true 2021-02-10T11:33:53.040 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:33:53.040 INFO:teuthology.orchestra.run.gibba020.stdout: osd deep scrub update digest min age = 30 2021-02-10T11:33:53.040 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:33:53.040 INFO:teuthology.orchestra.run.gibba020.stdout: osd map max advance = 10 2021-02-10T11:33:53.040 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:33:53.040 INFO:teuthology.orchestra.run.gibba020.stdout: journal zero on create = true 2021-02-10T11:33:53.040 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:33:53.040 INFO:teuthology.orchestra.run.gibba020.stdout: filestore ondisk finisher threads = 3 2021-02-10T11:33:53.040 INFO:teuthology.orchestra.run.gibba020.stdout: filestore apply finisher threads = 3 2021-02-10T11:33:53.040 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:33:53.040 INFO:teuthology.orchestra.run.gibba020.stdout: bdev debug aio = true 2021-02-10T11:33:53.041 INFO:teuthology.orchestra.run.gibba020.stdout: osd debug misdirected ops = true 2021-02-10T11:33:53.041 INFO:teuthology.orchestra.run.gibba020.stdout: bdev async discard = True 2021-02-10T11:33:53.041 INFO:teuthology.orchestra.run.gibba020.stdout: bdev enable discard = True 2021-02-10T11:33:53.041 INFO:teuthology.orchestra.run.gibba020.stdout: bluestore allocator = bitmap 2021-02-10T11:33:53.041 INFO:teuthology.orchestra.run.gibba020.stdout: bluestore block size = 96636764160 2021-02-10T11:33:53.041 INFO:teuthology.orchestra.run.gibba020.stdout: bluestore fsck on mount = True 2021-02-10T11:33:53.041 INFO:teuthology.orchestra.run.gibba020.stdout: debug bluefs = 20 2021-02-10T11:33:53.041 INFO:teuthology.orchestra.run.gibba020.stdout: debug bluestore = 20 2021-02-10T11:33:53.041 INFO:teuthology.orchestra.run.gibba020.stdout: debug filestore = 20 2021-02-10T11:33:53.041 INFO:teuthology.orchestra.run.gibba020.stdout: debug journal = 20 2021-02-10T11:33:53.041 INFO:teuthology.orchestra.run.gibba020.stdout: debug ms = 20 2021-02-10T11:33:53.041 INFO:teuthology.orchestra.run.gibba020.stdout: debug osd = 25 2021-02-10T11:33:53.041 INFO:teuthology.orchestra.run.gibba020.stdout: debug rocksdb = 10 2021-02-10T11:33:53.041 INFO:teuthology.orchestra.run.gibba020.stdout: mon osd backfillfull_ratio = 0.85 2021-02-10T11:33:53.042 INFO:teuthology.orchestra.run.gibba020.stdout: mon osd full ratio = 0.9 2021-02-10T11:33:53.042 INFO:teuthology.orchestra.run.gibba020.stdout: mon osd nearfull ratio = 0.8 2021-02-10T11:33:53.042 INFO:teuthology.orchestra.run.gibba020.stdout: osd failsafe full ratio = 0.95 2021-02-10T11:33:53.042 INFO:teuthology.orchestra.run.gibba020.stdout: osd objectstore = bluestore 2021-02-10T11:33:53.042 INFO:teuthology.orchestra.run.gibba020.stdout: osd op complaint time = 180 2021-02-10T11:33:53.042 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:33:53.042 INFO:teuthology.orchestra.run.gibba020.stdout:[mgr] 2021-02-10T11:33:53.042 INFO:teuthology.orchestra.run.gibba020.stdout: debug ms = 1 2021-02-10T11:33:53.042 INFO:teuthology.orchestra.run.gibba020.stdout: debug mgr = 20 2021-02-10T11:33:53.042 INFO:teuthology.orchestra.run.gibba020.stdout: debug mon = 20 2021-02-10T11:33:53.042 INFO:teuthology.orchestra.run.gibba020.stdout: debug auth = 20 2021-02-10T11:33:53.042 INFO:teuthology.orchestra.run.gibba020.stdout: mon reweight min pgs per osd = 4 2021-02-10T11:33:53.042 INFO:teuthology.orchestra.run.gibba020.stdout: mon reweight min bytes per osd = 10 2021-02-10T11:33:53.042 INFO:teuthology.orchestra.run.gibba020.stdout: debug client = 10 2021-02-10T11:33:53.043 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:33:53.043 INFO:teuthology.orchestra.run.gibba020.stdout:[mon] 2021-02-10T11:33:53.043 INFO:teuthology.orchestra.run.gibba020.stdout: debug ms = 1 2021-02-10T11:33:53.043 INFO:teuthology.orchestra.run.gibba020.stdout: debug mon = 20 2021-02-10T11:33:53.043 INFO:teuthology.orchestra.run.gibba020.stdout: debug paxos = 20 2021-02-10T11:33:53.043 INFO:teuthology.orchestra.run.gibba020.stdout: debug auth = 20 2021-02-10T11:33:53.043 INFO:teuthology.orchestra.run.gibba020.stdout: mon data avail warn = 5 2021-02-10T11:33:53.043 INFO:teuthology.orchestra.run.gibba020.stdout: mon mgr mkfs grace = 240 2021-02-10T11:33:53.043 INFO:teuthology.orchestra.run.gibba020.stdout: mon reweight min pgs per osd = 4 2021-02-10T11:33:53.043 INFO:teuthology.orchestra.run.gibba020.stdout: mon osd reporter subtree level = osd 2021-02-10T11:33:53.043 INFO:teuthology.orchestra.run.gibba020.stdout: mon osd prime pg temp = true 2021-02-10T11:33:53.043 INFO:teuthology.orchestra.run.gibba020.stdout: mon reweight min bytes per osd = 10 2021-02-10T11:33:53.043 INFO:teuthology.orchestra.run.gibba020.stdout: mon op complaint time = 120 2021-02-10T11:33:53.044 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:33:53.044 INFO:teuthology.orchestra.run.gibba020.stdout:[client] 2021-02-10T11:33:53.044 INFO:teuthology.orchestra.run.gibba020.stdout: rgw cache enabled = true 2021-02-10T11:33:53.044 INFO:teuthology.orchestra.run.gibba020.stdout: rgw enable ops log = true 2021-02-10T11:33:53.044 INFO:teuthology.orchestra.run.gibba020.stdout: rgw enable usage log = true 2021-02-10T11:33:53.044 INFO:teuthology.orchestra.run.gibba020.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2021-02-10T11:33:53.044 INFO:teuthology.orchestra.run.gibba020.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2021-02-10T11:33:53.044 INFO:teuthology.orchestra.run.gibba020.stdout: client mount timeout = 600 2021-02-10T11:33:53.044 INFO:teuthology.orchestra.run.gibba020.stdout: debug client = 20 2021-02-10T11:33:53.044 INFO:teuthology.orchestra.run.gibba020.stdout: debug ms = 1 2021-02-10T11:33:53.044 INFO:teuthology.orchestra.run.gibba020.stdout: pid file = "" 2021-02-10T11:33:53.044 INFO:teuthology.orchestra.run.gibba020.stdout:[mon.a] 2021-02-10T11:33:53.044 INFO:teuthology.orchestra.run.gibba020.stdout:[mon.b] 2021-02-10T11:33:53.044 INFO:teuthology.orchestra.run.gibba020.stdout:[mon.c] 2021-02-10T11:33:53.045 INFO:teuthology.orchestra.run.gibba020.stdout:[mds.a] 2021-02-10T11:33:53.045 INFO:teuthology.orchestra.run.gibba020.stdout:[mds.b] 2021-02-10T11:33:53.045 INFO:teuthology.orchestra.run.gibba020.stdout:[mds] 2021-02-10T11:33:53.045 INFO:teuthology.orchestra.run.gibba020.stdout: debug mds = 20 2021-02-10T11:33:53.045 INFO:teuthology.orchestra.run.gibba020.stdout: debug ms = 1 2021-02-10T11:33:53.045 INFO:teuthology.orchestra.run.gibba020.stdout: mds bal frag = True 2021-02-10T11:33:53.045 INFO:teuthology.orchestra.run.gibba020.stdout: mds bal fragment size max = 10000 2021-02-10T11:33:53.045 INFO:teuthology.orchestra.run.gibba020.stdout: mds bal merge size = 5 2021-02-10T11:33:53.045 INFO:teuthology.orchestra.run.gibba020.stdout: mds bal split bits = 3 2021-02-10T11:33:53.045 INFO:teuthology.orchestra.run.gibba020.stdout: mds bal split size = 100 2021-02-10T11:33:53.045 INFO:teuthology.orchestra.run.gibba020.stdout: mds debug frag = True 2021-02-10T11:33:53.045 INFO:teuthology.orchestra.run.gibba020.stdout: mds debug scatterstat = True 2021-02-10T11:33:53.045 INFO:teuthology.orchestra.run.gibba020.stdout: mds op complaint time = 180 2021-02-10T11:33:53.045 INFO:teuthology.orchestra.run.gibba020.stdout: mds verify scatter = True 2021-02-10T11:33:53.046 INFO:teuthology.orchestra.run.gibba020.stdout: osd op complaint time = 180 2021-02-10T11:33:53.046 INFO:teuthology.orchestra.run.gibba020.stdout:[client.alice] 2021-02-10T11:33:53.046 INFO:teuthology.orchestra.run.gibba020.stdout: keyring = /etc/ceph/ceph.client.alice.keyring 2021-02-10T11:33:53.047 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:33:53.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:53.470 7f7f9a7da700 1 Processor -- start 2021-02-10T11:33:53.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:53.470 7f7f9a7da700 1 -- start start 2021-02-10T11:33:53.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:53.471 7f7f9a7da700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f7f940e53e0 0x7f7f941b0550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:53.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:53.471 7f7f9a7da700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f7f940842e0 0x7f7f94082100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:53.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:53.471 7f7f9a7da700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f94082d50 con 0x7f7f941b0b70 2021-02-10T11:33:53.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:53.471 7f7f9a7da700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f940e5fe0 con 0x7f7f940e53e0 2021-02-10T11:33:53.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:53.471 7f7f9a7da700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f940e57e0 con 0x7f7f940842e0 2021-02-10T11:33:53.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:53.471 7f7f93fff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f7f940e53e0 0x7f7f941b0550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:53.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:53.471 7f7f93fff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f7f940e53e0 0x7f7f941b0550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:50112/0 (socket says 172.21.2.112:50112) 2021-02-10T11:33:53.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:53.471 7f7f93fff700 1 -- 172.21.2.112:0/1889900044 learned_addr learned my addr 172.21.2.112:0/1889900044 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:33:53.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:53.471 7f7f98df7700 1 --2- 172.21.2.112:0/1889900044 >> v2:172.21.2.120:3300/0 conn(0x7f7f940842e0 0x7f7f94082100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:53.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:53.471 7f7f93fff700 1 -- 172.21.2.112:0/1889900044 >> v2:172.21.2.120:3300/0 conn(0x7f7f940842e0 msgr2=0x7f7f94082100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:53.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:53.471 7f7f93fff700 1 --2- 172.21.2.112:0/1889900044 >> v2:172.21.2.120:3300/0 conn(0x7f7f940842e0 0x7f7f94082100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:53.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:53.471 7f7f93fff700 1 -- 172.21.2.112:0/1889900044 >> v1:172.21.2.120:6789/0 conn(0x7f7f941b0b70 legacy=0x7f7f94081910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:53.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:53.472 7f7f93fff700 1 -- 172.21.2.112:0/1889900044 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f940846e0 con 0x7f7f940e53e0 2021-02-10T11:33:53.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:53.472 7f7f98df7700 1 --2- 172.21.2.112:0/1889900044 >> v2:172.21.2.120:3300/0 conn(0x7f7f940842e0 0x7f7f94082100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:33:53.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:53.472 7f7f93fff700 1 --2- 172.21.2.112:0/1889900044 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7f940e53e0 0x7f7f941b0550 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 rx=0x7f7f84006700 tx=0x7f7f84009b80).ready entity=mon.0 client_cookie=c5b172e4f319c078 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:53.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:53.472 7f7f92ffd700 1 -- 172.21.2.112:0/1889900044 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7f84006340 con 0x7f7f940e53e0 2021-02-10T11:33:53.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:53.472 7f7f92ffd700 1 -- 172.21.2.112:0/1889900044 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f8400c0c0 con 0x7f7f940e53e0 2021-02-10T11:33:53.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:53.472 7f7f92ffd700 1 -- 172.21.2.112:0/1889900044 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7f8400c5e0 con 0x7f7f940e53e0 2021-02-10T11:33:53.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:53.472 7f7f9a7da700 1 -- 172.21.2.112:0/1889900044 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7f940e53e0 msgr2=0x7f7f941b0550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:53.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:53.472 7f7f9a7da700 1 --2- 172.21.2.112:0/1889900044 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7f940e53e0 0x7f7f941b0550 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 rx=0x7f7f84006700 tx=0x7f7f84009b80).stop 2021-02-10T11:33:53.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:53.473 7f7f9a7da700 1 -- 172.21.2.112:0/1889900044 shutdown_connections 2021-02-10T11:33:53.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:53.473 7f7f9a7da700 1 --2- 172.21.2.112:0/1889900044 >> v2:172.21.2.120:3300/0 conn(0x7f7f940842e0 0x7f7f94082100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:53.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:53.473 7f7f9a7da700 1 --2- 172.21.2.112:0/1889900044 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7f940e53e0 0x7f7f941b0550 unknown :-1 s=CLOSED pgs=1302 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:53.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:53.473 7f7f9a7da700 1 -- 172.21.2.112:0/1889900044 >> 172.21.2.112:0/1889900044 conn(0x7f7f940ce500 msgr2=0x7f7f940e4610 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:53.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:53.473 7f7f9a7da700 1 -- 172.21.2.112:0/1889900044 shutdown_connections 2021-02-10T11:33:53.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:53.473 7f7f9a7da700 1 -- 172.21.2.112:0/1889900044 wait complete. 2021-02-10T11:33:53.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:53.487 7f7f9a7da700 1 Processor -- start 2021-02-10T11:33:53.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:53.488 7f7f9a7da700 1 -- start start 2021-02-10T11:33:53.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:53.488 7f7f9a7da700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7f9418c030 0x7f7f9418c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:53.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:53.488 7f7f9a7da700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7f940b19d0 0x7f7f940b1dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:53.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:53.488 7f7f9a7da700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7f940b23f0 0x7f7f94188940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:53.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:53.488 7f7f9a7da700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f940b27f0 con 0x7f7f940b23f0 2021-02-10T11:33:53.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:53.488 7f7f9a7da700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f941883c0 con 0x7f7f940b19d0 2021-02-10T11:33:53.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:53.488 7f7f9a7da700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f9418f220 con 0x7f7f9418c030 2021-02-10T11:33:53.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:53.488 7f7f98df7700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7f940b23f0 0x7f7f94188940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:53.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:53.488 7f7f98df7700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7f940b23f0 0x7f7f94188940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:50114/0 (socket says 172.21.2.112:50114) 2021-02-10T11:33:53.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:53.488 7f7f98df7700 1 -- 172.21.2.112:0/2902042682 learned_addr learned my addr 172.21.2.112:0/2902042682 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:33:53.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:53.488 7f7f937fe700 1 --2- 172.21.2.112:0/2902042682 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7f940b19d0 0x7f7f940b1dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:53.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:53.488 7f7f93fff700 1 --2- 172.21.2.112:0/2902042682 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7f9418c030 0x7f7f9418c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:53.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:53.488 7f7f98df7700 1 -- 172.21.2.112:0/2902042682 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7f9418c030 msgr2=0x7f7f9418c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:53.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:53.488 7f7f98df7700 1 --2- 172.21.2.112:0/2902042682 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7f9418c030 0x7f7f9418c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:53.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:53.488 7f7f98df7700 1 -- 172.21.2.112:0/2902042682 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7f940b19d0 msgr2=0x7f7f940b1dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:53.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:53.488 7f7f98df7700 1 --2- 172.21.2.112:0/2902042682 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7f940b19d0 0x7f7f940b1dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:53.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:53.488 7f7f98df7700 1 -- 172.21.2.112:0/2902042682 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f9418ca50 con 0x7f7f940b23f0 2021-02-10T11:33:53.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:53.489 7f7f98df7700 1 --2- 172.21.2.112:0/2902042682 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7f940b23f0 0x7f7f94188940 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 rx=0x7f7f8c009980 tx=0x7f7f8c00bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:53.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:53.489 7f7f90ff9700 1 -- 172.21.2.112:0/2902042682 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7f8c00c2f0 con 0x7f7f940b23f0 2021-02-10T11:33:53.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:53.489 7f7f90ff9700 1 -- 172.21.2.112:0/2902042682 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f8c006790 con 0x7f7f940b23f0 2021-02-10T11:33:53.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:53.489 7f7f90ff9700 1 -- 172.21.2.112:0/2902042682 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7f8c00c7e0 con 0x7f7f940b23f0 2021-02-10T11:33:53.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:53.489 7f7f9a7da700 1 -- 172.21.2.112:0/2902042682 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f9418ca50 con 0x7f7f940b23f0 2021-02-10T11:33:53.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:53.489 7f7f9a7da700 1 -- 172.21.2.112:0/2902042682 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f9418ca50 con 0x7f7f940b23f0 2021-02-10T11:33:53.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:53.489 7f7f90ff9700 1 -- 172.21.2.112:0/2902042682 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7f8c0011f0 con 0x7f7f940b23f0 2021-02-10T11:33:53.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:53.489 7f7f90ff9700 1 --2- 172.21.2.112:0/2902042682 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7f74018730 0x7f7f7401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:53.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:53.489 7f7f90ff9700 1 -- 172.21.2.112:0/2902042682 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f7f8c032a40 con 0x7f7f940b23f0 2021-02-10T11:33:53.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:53.490 7f7f93fff700 1 --2- 172.21.2.112:0/2902042682 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7f74018730 0x7f7f7401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:53.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:53.490 7f7f93fff700 1 --2- 172.21.2.112:0/2902042682 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7f74018730 0x7f7f7401abc0 crc :-1 s=READY pgs=645 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:53.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:53.521 7f7f9a7da700 1 -- 172.21.2.112:0/2902042682 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f940ed660 con 0x7f7f940b23f0 2021-02-10T11:33:53.510 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:53.523 7f7f90ff9700 1 -- 172.21.2.112:0/2902042682 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7f8c00be10 con 0x7f7f940b23f0 2021-02-10T11:33:53.637 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:53.650 7f7f9a7da700 1 -- 172.21.2.112:0/2902042682 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7f94082d50 con 0x7f7f940b23f0 2021-02-10T11:33:53.637 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:53.651 7f7f90ff9700 1 -- 172.21.2.112:0/2902042682 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v55) v1 ==== 74+0+13996 (secure 0 0 0) 0x7f7f8c006670 con 0x7f7f940b23f0 2021-02-10T11:33:53.638 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:33:53.638 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":55,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:33:24.553170","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:33:21.523665","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":11,"pool_name":"cephfs_data","create_time":"2021-02-10 11:33:22.595892","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:33:53.639 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:53.652 7f7f9a7da700 1 -- 172.21.2.112:0/2902042682 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7f74018730 msgr2=0x7f7f7401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:53.639 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:53.652 7f7f9a7da700 1 --2- 172.21.2.112:0/2902042682 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7f74018730 0x7f7f7401abc0 crc :-1 s=READY pgs=645 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:53.639 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:53.652 7f7f9a7da700 1 -- 172.21.2.112:0/2902042682 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7f940b23f0 msgr2=0x7f7f94188940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:53.640 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:53.652 7f7f9a7da700 1 --2- 172.21.2.112:0/2902042682 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7f940b23f0 0x7f7f94188940 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 rx=0x7f7f8c009980 tx=0x7f7f8c00bcd0).stop 2021-02-10T11:33:53.640 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:53.652 7f7f9a7da700 1 -- 172.21.2.112:0/2902042682 shutdown_connections 2021-02-10T11:33:53.640 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:53.652 7f7f9a7da700 1 --2- 172.21.2.112:0/2902042682 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7f74018730 0x7f7f7401abc0 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:53.640 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:53.652 7f7f9a7da700 1 --2- 172.21.2.112:0/2902042682 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7f940b19d0 0x7f7f940b1dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:53.640 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:53.652 7f7f9a7da700 1 --2- 172.21.2.112:0/2902042682 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7f940b23f0 0x7f7f94188940 unknown :-1 s=CLOSED pgs=1303 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:53.640 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:53.652 7f7f9a7da700 1 --2- 172.21.2.112:0/2902042682 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7f9418c030 0x7f7f9418c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:53.640 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:53.652 7f7f9a7da700 1 -- 172.21.2.112:0/2902042682 >> 172.21.2.112:0/2902042682 conn(0x7f7f940f2240 msgr2=0x7f7f94094840 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:53.640 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:53.653 7f7f9a7da700 1 -- 172.21.2.112:0/2902042682 shutdown_connections 2021-02-10T11:33:53.640 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:53.653 7f7f9a7da700 1 -- 172.21.2.112:0/2902042682 wait complete. 2021-02-10T11:33:53.648 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:13:57.508758', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 10, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:33:21.523665', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '54', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 11, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:33:22.595892', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '55', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:33:53.648 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:33:54.081 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.078 7f64c93e0700 1 Processor -- start 2021-02-10T11:33:54.081 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.078 7f64c93e0700 1 -- start start 2021-02-10T11:33:54.081 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.078 7f64c93e0700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f64c40e5570 0x7f64c4078520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:54.081 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.078 7f64c93e0700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f64c4079110 0x7f64c4079510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:54.081 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.078 7f64c93e0700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64c4078b90 con 0x7f64c40e5170 2021-02-10T11:33:54.081 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.078 7f64c93e0700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64c41b40d0 con 0x7f64c4079110 2021-02-10T11:33:54.081 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.078 7f64c93e0700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64c40d1900 con 0x7f64c40e5570 2021-02-10T11:33:54.081 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.078 7f64c387e700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f64c4079110 0x7f64c4079510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:54.081 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.078 7f64c387e700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f64c4079110 0x7f64c4079510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:50126/0 (socket says 172.21.2.112:50126) 2021-02-10T11:33:54.081 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.078 7f64c387e700 1 -- 172.21.2.112:0/2659543784 learned_addr learned my addr 172.21.2.112:0/2659543784 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:33:54.081 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.079 7f64c287c700 1 --2- 172.21.2.112:0/2659543784 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f64c40e5570 0x7f64c4078520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:54.082 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.079 7f64c387e700 1 -- 172.21.2.112:0/2659543784 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f64c40e5570 msgr2=0x7f64c4078520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:54.082 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.079 7f64c387e700 1 --2- 172.21.2.112:0/2659543784 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f64c40e5570 0x7f64c4078520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:54.082 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.079 7f64c387e700 1 -- 172.21.2.112:0/2659543784 >> v1:172.21.2.120:6789/0 conn(0x7f64c40e5170 legacy=0x7f64c41b38e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:54.082 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.079 7f64c387e700 1 -- 172.21.2.112:0/2659543784 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64c40d1b20 con 0x7f64c4079110 2021-02-10T11:33:54.082 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.079 7f64c287c700 1 --2- 172.21.2.112:0/2659543784 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f64c40e5570 0x7f64c4078520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:33:54.082 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.079 7f64c387e700 1 --2- 172.21.2.112:0/2659543784 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f64c4079110 0x7f64c4079510 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 rx=0x7f64bc006f00 tx=0x7f64bc009f10).ready entity=mon.0 client_cookie=3e7ca78bd9c75fd server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:54.082 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.079 7f64c207b700 1 -- 172.21.2.112:0/2659543784 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f64bc00b3b0 con 0x7f64c4079110 2021-02-10T11:33:54.082 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.079 7f64c207b700 1 -- 172.21.2.112:0/2659543784 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64bc000d30 con 0x7f64c4079110 2021-02-10T11:33:54.082 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.079 7f64c207b700 1 -- 172.21.2.112:0/2659543784 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f64bc00b8e0 con 0x7f64c4079110 2021-02-10T11:33:54.082 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.079 7f64c93e0700 1 -- 172.21.2.112:0/2659543784 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f64c4079110 msgr2=0x7f64c4079510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:54.082 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.079 7f64c93e0700 1 --2- 172.21.2.112:0/2659543784 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f64c4079110 0x7f64c4079510 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 rx=0x7f64bc006f00 tx=0x7f64bc009f10).stop 2021-02-10T11:33:54.082 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.080 7f64c93e0700 1 -- 172.21.2.112:0/2659543784 shutdown_connections 2021-02-10T11:33:54.083 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.080 7f64c93e0700 1 --2- 172.21.2.112:0/2659543784 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f64c4079110 0x7f64c4079510 unknown :-1 s=CLOSED pgs=1304 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:54.083 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.080 7f64c93e0700 1 --2- 172.21.2.112:0/2659543784 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f64c40e5570 0x7f64c4078520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:54.083 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.080 7f64c93e0700 1 -- 172.21.2.112:0/2659543784 >> 172.21.2.112:0/2659543784 conn(0x7f64c40ce820 msgr2=0x7f64c40e4660 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:54.083 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.080 7f64c93e0700 1 -- 172.21.2.112:0/2659543784 shutdown_connections 2021-02-10T11:33:54.083 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.080 7f64c93e0700 1 -- 172.21.2.112:0/2659543784 wait complete. 2021-02-10T11:33:54.085 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.094 7f64c93e0700 1 Processor -- start 2021-02-10T11:33:54.085 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.094 7f64c93e0700 1 -- start start 2021-02-10T11:33:54.085 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.095 7f64c93e0700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f64c41117f0 0x7f64c40aeb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:54.085 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.095 7f64c93e0700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f64c40afa10 0x7f64c40afe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:54.085 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.095 7f64c93e0700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f64c40b0430 0x7f64c40b0830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:54.085 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.095 7f64c93e0700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64c40af160 con 0x7f64c40b0430 2021-02-10T11:33:54.085 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.095 7f64c93e0700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64c4109b20 con 0x7f64c41117f0 2021-02-10T11:33:54.085 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.095 7f64c93e0700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64c4111bf0 con 0x7f64c40afa10 2021-02-10T11:33:54.085 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.095 7f64c387e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f64c40b0430 0x7f64c40b0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:54.085 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.095 7f64c387e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f64c40b0430 0x7f64c40b0830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:50132/0 (socket says 172.21.2.112:50132) 2021-02-10T11:33:54.085 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.095 7f64c387e700 1 -- 172.21.2.112:0/3150440177 learned_addr learned my addr 172.21.2.112:0/3150440177 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:33:54.085 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.095 7f64c307d700 1 --2- 172.21.2.112:0/3150440177 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f64c41117f0 0x7f64c40aeb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:54.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.095 7f64c387e700 1 -- 172.21.2.112:0/3150440177 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f64c40afa10 msgr2=0x7f64c40afe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:54.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.095 7f64c287c700 1 --2- 172.21.2.112:0/3150440177 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f64c40afa10 0x7f64c40afe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:54.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.095 7f64c387e700 1 --2- 172.21.2.112:0/3150440177 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f64c40afa10 0x7f64c40afe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:54.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.095 7f64c387e700 1 -- 172.21.2.112:0/3150440177 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f64c41117f0 msgr2=0x7f64c40aeb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:54.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.095 7f64c387e700 1 --2- 172.21.2.112:0/3150440177 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f64c41117f0 0x7f64c40aeb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:54.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.095 7f64c387e700 1 -- 172.21.2.112:0/3150440177 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64c40954b0 con 0x7f64c40b0430 2021-02-10T11:33:54.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.095 7f64c307d700 1 --2- 172.21.2.112:0/3150440177 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f64c41117f0 0x7f64c40aeb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:33:54.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.095 7f64c287c700 1 --2- 172.21.2.112:0/3150440177 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f64c40afa10 0x7f64c40afe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:33:54.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.096 7f64c387e700 1 --2- 172.21.2.112:0/3150440177 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f64c40b0430 0x7f64c40b0830 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 rx=0x7f64bc00a950 tx=0x7f64bc00acf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:54.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.096 7f64bbfff700 1 -- 172.21.2.112:0/3150440177 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f64bc00c300 con 0x7f64c40b0430 2021-02-10T11:33:54.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.096 7f64bbfff700 1 -- 172.21.2.112:0/3150440177 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64bc00c580 con 0x7f64c40b0430 2021-02-10T11:33:54.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.096 7f64c93e0700 1 -- 172.21.2.112:0/3150440177 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64c40954b0 con 0x7f64c40b0430 2021-02-10T11:33:54.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.096 7f64c93e0700 1 -- 172.21.2.112:0/3150440177 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64c41a0370 con 0x7f64c40b0430 2021-02-10T11:33:54.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.096 7f64bbfff700 1 -- 172.21.2.112:0/3150440177 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f64bc00caa0 con 0x7f64c40b0430 2021-02-10T11:33:54.087 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.096 7f64bbfff700 1 -- 172.21.2.112:0/3150440177 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f64bc00c760 con 0x7f64c40b0430 2021-02-10T11:33:54.087 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.097 7f64bbfff700 1 --2- 172.21.2.112:0/3150440177 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f64a80186c0 0x7f64a801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:54.087 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.097 7f64bbfff700 1 -- 172.21.2.112:0/3150440177 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f64bc032b90 con 0x7f64c40b0430 2021-02-10T11:33:54.087 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.097 7f64c307d700 1 --2- 172.21.2.112:0/3150440177 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f64a80186c0 0x7f64a801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:54.087 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.097 7f64c307d700 1 --2- 172.21.2.112:0/3150440177 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f64a80186c0 0x7f64a801abc0 crc :-1 s=READY pgs=646 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:54.115 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.128 7f64c93e0700 1 -- 172.21.2.112:0/3150440177 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64c4105c20 con 0x7f64c40b0430 2021-02-10T11:33:54.118 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.131 7f64bbfff700 1 -- 172.21.2.112:0/3150440177 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f64bc00c770 con 0x7f64c40b0430 2021-02-10T11:33:54.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.260 7f64c93e0700 1 -- 172.21.2.112:0/3150440177 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f64c40fe290 con 0x7f64c40b0430 2021-02-10T11:33:54.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.260 7f64bbfff700 1 -- 172.21.2.112:0/3150440177 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v55) v1 ==== 92+0+10 (secure 0 0 0) 0x7f64bc00c560 con 0x7f64c40b0430 2021-02-10T11:33:54.247 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:33:54.249 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.262 7f64c93e0700 1 -- 172.21.2.112:0/3150440177 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f64a80186c0 msgr2=0x7f64a801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:54.249 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.262 7f64c93e0700 1 --2- 172.21.2.112:0/3150440177 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f64a80186c0 0x7f64a801abc0 crc :-1 s=READY pgs=646 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:54.249 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.262 7f64c93e0700 1 -- 172.21.2.112:0/3150440177 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f64c40b0430 msgr2=0x7f64c40b0830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:54.249 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.262 7f64c93e0700 1 --2- 172.21.2.112:0/3150440177 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f64c40b0430 0x7f64c40b0830 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 rx=0x7f64bc00a950 tx=0x7f64bc00acf0).stop 2021-02-10T11:33:54.249 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.262 7f64c93e0700 1 -- 172.21.2.112:0/3150440177 shutdown_connections 2021-02-10T11:33:54.249 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.262 7f64c93e0700 1 --2- 172.21.2.112:0/3150440177 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f64a80186c0 0x7f64a801abc0 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:54.249 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.262 7f64c93e0700 1 --2- 172.21.2.112:0/3150440177 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f64c40b0430 0x7f64c40b0830 unknown :-1 s=CLOSED pgs=1305 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:54.249 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.262 7f64c93e0700 1 --2- 172.21.2.112:0/3150440177 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f64c40afa10 0x7f64c40afe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:54.250 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.262 7f64c93e0700 1 --2- 172.21.2.112:0/3150440177 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f64c41117f0 0x7f64c40aeb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:54.250 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.262 7f64c93e0700 1 -- 172.21.2.112:0/3150440177 >> 172.21.2.112:0/3150440177 conn(0x7f64c4105e50 msgr2=0x7f64c40b22a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:54.250 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.263 7f64c93e0700 1 -- 172.21.2.112:0/3150440177 shutdown_connections 2021-02-10T11:33:54.250 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.263 7f64c93e0700 1 -- 172.21.2.112:0/3150440177 wait complete. 2021-02-10T11:33:54.257 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:33:54.686 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.683 7f4f8b154700 1 Processor -- start 2021-02-10T11:33:54.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.683 7f4f8b154700 1 -- start start 2021-02-10T11:33:54.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.683 7f4f8b154700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4f840e5570 0x7f4f84078520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:54.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.683 7f4f8b154700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f84078b90 con 0x7f4f840e5170 2021-02-10T11:33:54.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.683 7f4f8b154700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f841b40d0 con 0x7f4f84079110 2021-02-10T11:33:54.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.683 7f4f8b154700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f840d1900 con 0x7f4f840e5570 2021-02-10T11:33:54.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.684 7f4f88f70700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f4f840e5170 0x7f4f841b38e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:49598/0 (socket says 172.21.2.112:49598) 2021-02-10T11:33:54.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.684 7f4f88f70700 1 -- 172.21.2.112:0/2005563334 learned_addr learned my addr 172.21.2.112:0/2005563334 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:33:54.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.684 7f4f83fff700 1 --2- 172.21.2.112:0/2005563334 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4f840e5570 0x7f4f84078520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:54.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.684 7f4f837fe700 1 -- 172.21.2.112:0/2005563334 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1402072345 0 0) 0x7f4f6c001180 con 0x7f4f840e5170 2021-02-10T11:33:54.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.684 7f4f837fe700 1 -- 172.21.2.112:0/2005563334 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f70002fe0 con 0x7f4f840e5170 2021-02-10T11:33:54.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.684 7f4f83fff700 1 -- 172.21.2.112:0/2005563334 >> v1:172.21.2.120:6789/0 conn(0x7f4f84079110 legacy=0x7f4f84079510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:54.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.684 7f4f83fff700 1 -- 172.21.2.112:0/2005563334 >> v1:172.21.2.112:6789/0 conn(0x7f4f840e5170 legacy=0x7f4f841b38e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:54.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.684 7f4f83fff700 1 -- 172.21.2.112:0/2005563334 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f840d1ae0 con 0x7f4f840e5570 2021-02-10T11:33:54.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.684 7f4f83fff700 1 --2- 172.21.2.112:0/2005563334 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4f840e5570 0x7f4f84078520 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f4f74006650 tx=0x7f4f74009a80).ready entity=mon.2 client_cookie=78cda0eda4cc12f5 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:54.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.685 7f4f837fe700 1 -- 172.21.2.112:0/2005563334 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4f7400c0c0 con 0x7f4f840e5570 2021-02-10T11:33:54.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.685 7f4f837fe700 1 -- 172.21.2.112:0/2005563334 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f7400c340 con 0x7f4f840e5570 2021-02-10T11:33:54.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.685 7f4f837fe700 1 -- 172.21.2.112:0/2005563334 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4f7400c860 con 0x7f4f840e5570 2021-02-10T11:33:54.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.685 7f4f8b154700 1 -- 172.21.2.112:0/2005563334 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4f840e5570 msgr2=0x7f4f84078520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:54.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.685 7f4f8b154700 1 --2- 172.21.2.112:0/2005563334 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4f840e5570 0x7f4f84078520 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f4f74006650 tx=0x7f4f74009a80).stop 2021-02-10T11:33:54.689 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.685 7f4f8b154700 1 -- 172.21.2.112:0/2005563334 shutdown_connections 2021-02-10T11:33:54.689 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.685 7f4f8b154700 1 --2- 172.21.2.112:0/2005563334 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4f840e5570 0x7f4f84078520 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:54.689 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.685 7f4f8b154700 1 -- 172.21.2.112:0/2005563334 >> 172.21.2.112:0/2005563334 conn(0x7f4f840ce820 msgr2=0x7f4f840e4660 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:54.689 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.685 7f4f8b154700 1 -- 172.21.2.112:0/2005563334 shutdown_connections 2021-02-10T11:33:54.689 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.685 7f4f8b154700 1 -- 172.21.2.112:0/2005563334 wait complete. 2021-02-10T11:33:54.689 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.700 7f4f8b154700 1 Processor -- start 2021-02-10T11:33:54.689 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.700 7f4f8b154700 1 -- start start 2021-02-10T11:33:54.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.700 7f4f8b154700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4f840b6d40 0x7f4f841a6640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:54.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.700 7f4f8b154700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4f840b9510 0x7f4f840b34c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:54.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.700 7f4f8b154700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4f841a6c60 0x7f4f840b3ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:54.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.700 7f4f8b154700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f840b4300 con 0x7f4f840b6d40 2021-02-10T11:33:54.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.700 7f4f8b154700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f84111c50 con 0x7f4f841a6c60 2021-02-10T11:33:54.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.700 7f4f8b154700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f840b7140 con 0x7f4f840b9510 2021-02-10T11:33:54.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.700 7f4f88f70700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4f840b6d40 0x7f4f841a6640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:54.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.700 7f4f88f70700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4f840b6d40 0x7f4f841a6640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:50142/0 (socket says 172.21.2.112:50142) 2021-02-10T11:33:54.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.700 7f4f88f70700 1 -- 172.21.2.112:0/3465097624 learned_addr learned my addr 172.21.2.112:0/3465097624 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:33:54.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.701 7f4f88f70700 1 -- 172.21.2.112:0/3465097624 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4f840b9510 msgr2=0x7f4f840b34c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:33:54.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.701 7f4f88f70700 1 --2- 172.21.2.112:0/3465097624 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4f840b9510 0x7f4f840b34c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:54.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.701 7f4f88f70700 1 -- 172.21.2.112:0/3465097624 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4f841a6c60 msgr2=0x7f4f840b3ae0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:33:54.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.701 7f4f88f70700 1 --2- 172.21.2.112:0/3465097624 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4f841a6c60 0x7f4f840b3ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:54.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.701 7f4f88f70700 1 -- 172.21.2.112:0/3465097624 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f840b4800 con 0x7f4f840b6d40 2021-02-10T11:33:54.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.701 7f4f88f70700 1 --2- 172.21.2.112:0/3465097624 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4f840b6d40 0x7f4f841a6640 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 rx=0x7f4f6c00a5d0 tx=0x7f4f6c00a970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:54.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.701 7f4f817fa700 1 -- 172.21.2.112:0/3465097624 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4f6c000ce0 con 0x7f4f840b6d40 2021-02-10T11:33:54.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.701 7f4f817fa700 1 -- 172.21.2.112:0/3465097624 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f6c0043a0 con 0x7f4f840b6d40 2021-02-10T11:33:54.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.701 7f4f8b154700 1 -- 172.21.2.112:0/3465097624 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f840b4800 con 0x7f4f840b6d40 2021-02-10T11:33:54.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.701 7f4f8b154700 1 -- 172.21.2.112:0/3465097624 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f840b4d90 con 0x7f4f840b6d40 2021-02-10T11:33:54.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.701 7f4f817fa700 1 -- 172.21.2.112:0/3465097624 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4f6c0048c0 con 0x7f4f840b6d40 2021-02-10T11:33:54.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.702 7f4f817fa700 1 -- 172.21.2.112:0/3465097624 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4f6c000ce0 con 0x7f4f840b6d40 2021-02-10T11:33:54.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.702 7f4f817fa700 1 --2- 172.21.2.112:0/3465097624 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4f68018730 0x7f4f6801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:54.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.702 7f4f817fa700 1 -- 172.21.2.112:0/3465097624 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f4f6c032790 con 0x7f4f840b6d40 2021-02-10T11:33:54.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.703 7f4f83fff700 1 --2- 172.21.2.112:0/3465097624 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4f68018730 0x7f4f6801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:54.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.703 7f4f83fff700 1 --2- 172.21.2.112:0/3465097624 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4f68018730 0x7f4f6801ac10 crc :-1 s=READY pgs=647 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:54.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.733 7f4f8b154700 1 -- 172.21.2.112:0/3465097624 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f8410bc00 con 0x7f4f840b6d40 2021-02-10T11:33:54.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.736 7f4f817fa700 1 -- 172.21.2.112:0/3465097624 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4f6c01f4d0 con 0x7f4f840b6d40 2021-02-10T11:33:54.855 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.865 7f4f8b154700 1 -- 172.21.2.112:0/3465097624 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f4f840fe290 con 0x7f4f840b6d40 2021-02-10T11:33:54.855 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.865 7f4f817fa700 1 -- 172.21.2.112:0/3465097624 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v55) v1 ==== 104+0+10 (secure 0 0 0) 0x7f4f6c00ad60 con 0x7f4f840b6d40 2021-02-10T11:33:54.856 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:33:54.856 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.867 7f4f8b154700 1 -- 172.21.2.112:0/3465097624 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4f68018730 msgr2=0x7f4f6801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:54.856 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.867 7f4f8b154700 1 --2- 172.21.2.112:0/3465097624 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4f68018730 0x7f4f6801ac10 crc :-1 s=READY pgs=647 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:54.857 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.867 7f4f8b154700 1 -- 172.21.2.112:0/3465097624 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4f840b6d40 msgr2=0x7f4f841a6640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:54.857 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.867 7f4f8b154700 1 --2- 172.21.2.112:0/3465097624 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4f840b6d40 0x7f4f841a6640 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 rx=0x7f4f6c00a5d0 tx=0x7f4f6c00a970).stop 2021-02-10T11:33:54.857 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.867 7f4f8b154700 1 -- 172.21.2.112:0/3465097624 shutdown_connections 2021-02-10T11:33:54.857 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.867 7f4f8b154700 1 --2- 172.21.2.112:0/3465097624 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4f68018730 0x7f4f6801ac10 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:54.857 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.867 7f4f8b154700 1 --2- 172.21.2.112:0/3465097624 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4f841a6c60 0x7f4f840b3ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:54.857 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.867 7f4f8b154700 1 --2- 172.21.2.112:0/3465097624 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4f840b9510 0x7f4f840b34c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:54.857 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.867 7f4f8b154700 1 --2- 172.21.2.112:0/3465097624 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4f840b6d40 0x7f4f841a6640 unknown :-1 s=CLOSED pgs=1307 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:54.857 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.867 7f4f8b154700 1 -- 172.21.2.112:0/3465097624 >> 172.21.2.112:0/3465097624 conn(0x7f4f840bd380 msgr2=0x7f4f840bb9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:54.857 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.867 7f4f8b154700 1 -- 172.21.2.112:0/3465097624 shutdown_connections 2021-02-10T11:33:54.857 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:54.867 7f4f8b154700 1 -- 172.21.2.112:0/3465097624 wait complete. 2021-02-10T11:33:54.863 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:33:55.293 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.290 7fd76a74e700 1 Processor -- start 2021-02-10T11:33:55.293 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.290 7fd76a74e700 1 -- start start 2021-02-10T11:33:55.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.291 7fd76a74e700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fd764082920 0x7fd7640832d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:55.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.291 7fd76a74e700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd7640838f0 0x7fd764083cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:55.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.291 7fd76a74e700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd764082d50 con 0x7fd7640e53d0 2021-02-10T11:33:55.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.291 7fd76a74e700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7640e57d0 con 0x7fd764082920 2021-02-10T11:33:55.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.291 7fd76a74e700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7641b0d30 con 0x7fd7640838f0 2021-02-10T11:33:55.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.291 7fd7637fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fd764082920 0x7fd7640832d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:55.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.291 7fd7637fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fd764082920 0x7fd7640832d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:50152/0 (socket says 172.21.2.112:50152) 2021-02-10T11:33:55.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.291 7fd7637fe700 1 -- 172.21.2.112:0/2143039169 learned_addr learned my addr 172.21.2.112:0/2143039169 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:33:55.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.291 7fd768d6b700 1 --2- 172.21.2.112:0/2143039169 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd7640838f0 0x7fd764083cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:55.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.291 7fd762ffd700 1 -- 172.21.2.112:0/2143039169 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2001355972 0 0) 0x7fd75c001060 con 0x7fd7640e53d0 2021-02-10T11:33:55.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.291 7fd762ffd700 1 -- 172.21.2.112:0/2143039169 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd74c002fe0 con 0x7fd7640e53d0 2021-02-10T11:33:55.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.291 7fd7637fe700 1 -- 172.21.2.112:0/2143039169 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd7640838f0 msgr2=0x7fd764083cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:55.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.291 7fd7637fe700 1 --2- 172.21.2.112:0/2143039169 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd7640838f0 0x7fd764083cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:55.295 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.291 7fd7637fe700 1 -- 172.21.2.112:0/2143039169 >> v1:172.21.2.112:6789/0 conn(0x7fd7640e53d0 legacy=0x7fd7641b0540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:55.295 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.291 7fd7637fe700 1 -- 172.21.2.112:0/2143039169 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd764084310 con 0x7fd764082920 2021-02-10T11:33:55.295 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.292 7fd7637fe700 1 --2- 172.21.2.112:0/2143039169 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd764082920 0x7fd7640832d0 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 rx=0x7fd758006700 tx=0x7fd758009b80).ready entity=mon.0 client_cookie=2397e76705534095 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:55.295 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.292 7fd762ffd700 1 -- 172.21.2.112:0/2143039169 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd75800c0c0 con 0x7fd764082920 2021-02-10T11:33:55.295 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.292 7fd762ffd700 1 -- 172.21.2.112:0/2143039169 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd75800c440 con 0x7fd764082920 2021-02-10T11:33:55.295 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.292 7fd762ffd700 1 -- 172.21.2.112:0/2143039169 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd75800c8a0 con 0x7fd764082920 2021-02-10T11:33:55.295 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.292 7fd76a74e700 1 -- 172.21.2.112:0/2143039169 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd764082920 msgr2=0x7fd7640832d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:55.295 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.292 7fd76a74e700 1 --2- 172.21.2.112:0/2143039169 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd764082920 0x7fd7640832d0 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 rx=0x7fd758006700 tx=0x7fd758009b80).stop 2021-02-10T11:33:55.295 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.292 7fd76a74e700 1 -- 172.21.2.112:0/2143039169 shutdown_connections 2021-02-10T11:33:55.295 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.292 7fd76a74e700 1 --2- 172.21.2.112:0/2143039169 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd7640838f0 0x7fd764083cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:55.295 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.292 7fd76a74e700 1 --2- 172.21.2.112:0/2143039169 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd764082920 0x7fd7640832d0 unknown :-1 s=CLOSED pgs=1309 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:55.295 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.292 7fd76a74e700 1 -- 172.21.2.112:0/2143039169 >> 172.21.2.112:0/2143039169 conn(0x7fd7640ce4f0 msgr2=0x7fd7640e4600 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:55.295 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.292 7fd76a74e700 1 -- 172.21.2.112:0/2143039169 shutdown_connections 2021-02-10T11:33:55.295 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.292 7fd76a74e700 1 -- 172.21.2.112:0/2143039169 wait complete. 2021-02-10T11:33:55.297 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.307 7fd76a74e700 1 Processor -- start 2021-02-10T11:33:55.297 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.307 7fd76a74e700 1 -- start start 2021-02-10T11:33:55.297 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.307 7fd76a74e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd7640f17d0 0x7fd76409bd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:55.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.307 7fd76a74e700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd76409c9f0 0x7fd76409d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:55.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.307 7fd76a74e700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd76409cdf0 0x7fd76409df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:55.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.307 7fd76a74e700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd76409eee0 con 0x7fd7640f17d0 2021-02-10T11:33:55.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.307 7fd76a74e700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd76409d3e0 con 0x7fd76409cdf0 2021-02-10T11:33:55.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.307 7fd76a74e700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd76409c600 con 0x7fd76409c9f0 2021-02-10T11:33:55.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.307 7fd763fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd7640f17d0 0x7fd76409bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:55.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.307 7fd763fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd7640f17d0 0x7fd76409bd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:50158/0 (socket says 172.21.2.112:50158) 2021-02-10T11:33:55.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.307 7fd763fff700 1 -- 172.21.2.112:0/2604399538 learned_addr learned my addr 172.21.2.112:0/2604399538 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:33:55.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.307 7fd7637fe700 1 --2- 172.21.2.112:0/2604399538 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd76409c9f0 0x7fd76409d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:55.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.308 7fd768d6b700 1 --2- 172.21.2.112:0/2604399538 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd76409cdf0 0x7fd76409df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:55.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.308 7fd763fff700 1 -- 172.21.2.112:0/2604399538 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd76409c9f0 msgr2=0x7fd76409d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:55.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.308 7fd763fff700 1 --2- 172.21.2.112:0/2604399538 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd76409c9f0 0x7fd76409d960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:55.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.308 7fd763fff700 1 -- 172.21.2.112:0/2604399538 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd76409cdf0 msgr2=0x7fd76409df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:55.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.308 7fd763fff700 1 --2- 172.21.2.112:0/2604399538 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd76409cdf0 0x7fd76409df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:55.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.308 7fd763fff700 1 -- 172.21.2.112:0/2604399538 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd76409e5a0 con 0x7fd7640f17d0 2021-02-10T11:33:55.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.308 7fd768d6b700 1 --2- 172.21.2.112:0/2604399538 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd76409cdf0 0x7fd76409df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:33:55.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.308 7fd763fff700 1 --2- 172.21.2.112:0/2604399538 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd7640f17d0 0x7fd76409bd70 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 rx=0x7fd75c009980 tx=0x7fd75c00bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:55.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.308 7fd760ff9700 1 -- 172.21.2.112:0/2604399538 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd75c003080 con 0x7fd7640f17d0 2021-02-10T11:33:55.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.308 7fd760ff9700 1 -- 172.21.2.112:0/2604399538 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd75c001460 con 0x7fd7640f17d0 2021-02-10T11:33:55.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.308 7fd760ff9700 1 -- 172.21.2.112:0/2604399538 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd75c001940 con 0x7fd7640f17d0 2021-02-10T11:33:55.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.308 7fd76a74e700 1 -- 172.21.2.112:0/2604399538 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd76409e5a0 con 0x7fd7640f17d0 2021-02-10T11:33:55.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.308 7fd76a74e700 1 -- 172.21.2.112:0/2604399538 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd76409e5a0 con 0x7fd7640f17d0 2021-02-10T11:33:55.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.309 7fd760ff9700 1 -- 172.21.2.112:0/2604399538 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd75c000fd0 con 0x7fd7640f17d0 2021-02-10T11:33:55.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.309 7fd760ff9700 1 --2- 172.21.2.112:0/2604399538 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd744018730 0x7fd74401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:55.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.309 7fd760ff9700 1 -- 172.21.2.112:0/2604399538 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fd75c032a20 con 0x7fd7640f17d0 2021-02-10T11:33:55.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.309 7fd7637fe700 1 --2- 172.21.2.112:0/2604399538 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd744018730 0x7fd74401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:55.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.310 7fd7637fe700 1 --2- 172.21.2.112:0/2604399538 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd744018730 0x7fd74401ac10 crc :-1 s=READY pgs=648 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:55.327 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.340 7fd76a74e700 1 -- 172.21.2.112:0/2604399538 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7640ed630 con 0x7fd7640f17d0 2021-02-10T11:33:55.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.343 7fd760ff9700 1 -- 172.21.2.112:0/2604399538 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd75c00be10 con 0x7fd7640f17d0 2021-02-10T11:33:55.457 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.470 7fd76a74e700 1 -- 172.21.2.112:0/2604399538 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7fd7640f3a00 con 0x7fd7640f17d0 2021-02-10T11:33:55.458 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.471 7fd760ff9700 1 -- 172.21.2.112:0/2604399538 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v55) v1 ==== 100+0+10 (secure 0 0 0) 0x7fd75c000fd0 con 0x7fd7640f17d0 2021-02-10T11:33:55.458 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:33:55.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.473 7fd76a74e700 1 -- 172.21.2.112:0/2604399538 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd744018730 msgr2=0x7fd74401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:55.460 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.473 7fd76a74e700 1 --2- 172.21.2.112:0/2604399538 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd744018730 0x7fd74401ac10 crc :-1 s=READY pgs=648 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:55.460 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.473 7fd76a74e700 1 -- 172.21.2.112:0/2604399538 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd7640f17d0 msgr2=0x7fd76409bd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:55.460 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.473 7fd76a74e700 1 --2- 172.21.2.112:0/2604399538 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd7640f17d0 0x7fd76409bd70 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 rx=0x7fd75c009980 tx=0x7fd75c00bcd0).stop 2021-02-10T11:33:55.461 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.473 7fd76a74e700 1 -- 172.21.2.112:0/2604399538 shutdown_connections 2021-02-10T11:33:55.461 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.473 7fd76a74e700 1 --2- 172.21.2.112:0/2604399538 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd744018730 0x7fd74401ac10 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:55.461 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.473 7fd76a74e700 1 --2- 172.21.2.112:0/2604399538 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd76409cdf0 0x7fd76409df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:55.461 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.473 7fd76a74e700 1 --2- 172.21.2.112:0/2604399538 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd76409c9f0 0x7fd76409d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:55.461 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.473 7fd76a74e700 1 --2- 172.21.2.112:0/2604399538 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd7640f17d0 0x7fd76409bd70 unknown :-1 s=CLOSED pgs=1310 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:55.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.473 7fd76a74e700 1 -- 172.21.2.112:0/2604399538 >> 172.21.2.112:0/2604399538 conn(0x7fd7640917b0 msgr2=0x7fd7640b5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:55.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.473 7fd76a74e700 1 -- 172.21.2.112:0/2604399538 shutdown_connections 2021-02-10T11:33:55.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.473 7fd76a74e700 1 -- 172.21.2.112:0/2604399538 wait complete. 2021-02-10T11:33:55.469 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:33:55.469 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:33:55.894 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.891 7fd77fec1700 1 Processor -- start 2021-02-10T11:33:55.894 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.891 7fd77fec1700 1 -- start start 2021-02-10T11:33:55.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.892 7fd77fec1700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd7780d1ad0 0x7fd7780e10d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:55.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.892 7fd77fec1700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fd7781b5c20 0x7fd7781b8030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:55.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.892 7fd77fec1700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7780dfb40 con 0x7fd7780d0b80 2021-02-10T11:33:55.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.892 7fd77fec1700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7780d0f80 con 0x7fd7781b5c20 2021-02-10T11:33:55.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.892 7fd77fec1700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7780f73f0 con 0x7fd7780d1ad0 2021-02-10T11:33:55.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.892 7fd77e4de700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fd7781b5c20 0x7fd7781b8030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:55.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.892 7fd77e4de700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fd7781b5c20 0x7fd7781b8030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:50168/0 (socket says 172.21.2.112:50168) 2021-02-10T11:33:55.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.892 7fd77e4de700 1 -- 172.21.2.112:0/2582497081 learned_addr learned my addr 172.21.2.112:0/2582497081 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:33:55.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.892 7fd77e4de700 1 -- 172.21.2.112:0/2582497081 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd7780d1ad0 msgr2=0x7fd7780e10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:55.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.892 7fd77e4de700 1 --2- 172.21.2.112:0/2582497081 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd7780d1ad0 0x7fd7780e10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:55.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.892 7fd77e4de700 1 -- 172.21.2.112:0/2582497081 >> v1:172.21.2.120:6789/0 conn(0x7fd7780d0b80 legacy=0x7fd7780d12e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:55.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.892 7fd77e4de700 1 -- 172.21.2.112:0/2582497081 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7780df760 con 0x7fd7781b5c20 2021-02-10T11:33:55.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.892 7fd77e4de700 1 --2- 172.21.2.112:0/2582497081 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd7781b5c20 0x7fd7781b8030 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 rx=0x7fd770006f00 tx=0x7fd770009f10).ready entity=mon.0 client_cookie=6dbbc8e65017beef server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:55.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.893 7fd77ccdb700 1 -- 172.21.2.112:0/2582497081 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd77000b3b0 con 0x7fd7781b5c20 2021-02-10T11:33:55.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.893 7fd77ccdb700 1 -- 172.21.2.112:0/2582497081 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd770000d30 con 0x7fd7781b5c20 2021-02-10T11:33:55.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.893 7fd77ccdb700 1 -- 172.21.2.112:0/2582497081 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd77000b920 con 0x7fd7781b5c20 2021-02-10T11:33:55.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.893 7fd77fec1700 1 -- 172.21.2.112:0/2582497081 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd7781b5c20 msgr2=0x7fd7781b8030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:55.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.893 7fd77fec1700 1 --2- 172.21.2.112:0/2582497081 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd7781b5c20 0x7fd7781b8030 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 rx=0x7fd770006f00 tx=0x7fd770009f10).stop 2021-02-10T11:33:55.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.893 7fd77fec1700 1 -- 172.21.2.112:0/2582497081 shutdown_connections 2021-02-10T11:33:55.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.893 7fd77fec1700 1 --2- 172.21.2.112:0/2582497081 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd7781b5c20 0x7fd7781b8030 unknown :-1 s=CLOSED pgs=1311 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:55.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.893 7fd77fec1700 1 --2- 172.21.2.112:0/2582497081 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd7780d1ad0 0x7fd7780e10d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:55.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.893 7fd77fec1700 1 -- 172.21.2.112:0/2582497081 >> 172.21.2.112:0/2582497081 conn(0x7fd7780ea6c0 msgr2=0x7fd7780dcba0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:55.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.894 7fd77fec1700 1 -- 172.21.2.112:0/2582497081 shutdown_connections 2021-02-10T11:33:55.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.894 7fd77fec1700 1 -- 172.21.2.112:0/2582497081 wait complete. 2021-02-10T11:33:55.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.908 7fd77fec1700 1 Processor -- start 2021-02-10T11:33:55.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.908 7fd77fec1700 1 -- start start 2021-02-10T11:33:55.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.908 7fd77fec1700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd77819ead0 0x7fd7781ae850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:55.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.908 7fd77fec1700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd7781aee70 0x7fd7781ad6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:55.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.908 7fd77fec1700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd7781adce0 0x7fd7780b7d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:55.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.908 7fd77fec1700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd77819f060 con 0x7fd7781adce0 2021-02-10T11:33:55.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.908 7fd77fec1700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7781ae0e0 con 0x7fd77819ead0 2021-02-10T11:33:55.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.908 7fd77fec1700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7781ae300 con 0x7fd7781aee70 2021-02-10T11:33:55.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.908 7fd77e4de700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd7781adce0 0x7fd7780b7d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:55.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.908 7fd77e4de700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd7781adce0 0x7fd7780b7d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:50172/0 (socket says 172.21.2.112:50172) 2021-02-10T11:33:55.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.908 7fd77e4de700 1 -- 172.21.2.112:0/3104245982 learned_addr learned my addr 172.21.2.112:0/3104245982 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:33:55.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.909 7fd77e4de700 1 -- 172.21.2.112:0/3104245982 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd7781aee70 msgr2=0x7fd7781ad6c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:33:55.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.909 7fd77e4de700 1 --2- 172.21.2.112:0/3104245982 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd7781aee70 0x7fd7781ad6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:55.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.909 7fd77e4de700 1 -- 172.21.2.112:0/3104245982 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd77819ead0 msgr2=0x7fd7781ae850 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:33:55.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.909 7fd77e4de700 1 --2- 172.21.2.112:0/3104245982 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd77819ead0 0x7fd7781ae850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:55.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.909 7fd77e4de700 1 -- 172.21.2.112:0/3104245982 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7780b8390 con 0x7fd7781adce0 2021-02-10T11:33:55.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.909 7fd77e4de700 1 --2- 172.21.2.112:0/3104245982 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd7781adce0 0x7fd7780b7d70 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 rx=0x7fd770006f30 tx=0x7fd770002d20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:55.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.909 7fd76a7fc700 1 -- 172.21.2.112:0/3104245982 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd77000c300 con 0x7fd7781adce0 2021-02-10T11:33:55.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.909 7fd76a7fc700 1 -- 172.21.2.112:0/3104245982 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd77000c580 con 0x7fd7781adce0 2021-02-10T11:33:55.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.909 7fd76a7fc700 1 -- 172.21.2.112:0/3104245982 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd77000caa0 con 0x7fd7781adce0 2021-02-10T11:33:55.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.909 7fd77fec1700 1 -- 172.21.2.112:0/3104245982 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7780b8390 con 0x7fd7781adce0 2021-02-10T11:33:55.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.909 7fd77fec1700 1 -- 172.21.2.112:0/3104245982 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7780b8390 con 0x7fd7781adce0 2021-02-10T11:33:55.900 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.910 7fd76a7fc700 1 -- 172.21.2.112:0/3104245982 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd770001270 con 0x7fd7781adce0 2021-02-10T11:33:55.900 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.910 7fd76a7fc700 1 --2- 172.21.2.112:0/3104245982 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd7600186c0 0x7fd76001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:55.900 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.910 7fd76a7fc700 1 -- 172.21.2.112:0/3104245982 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fd7700328b0 con 0x7fd7781adce0 2021-02-10T11:33:55.900 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.910 7fd77dcdd700 1 --2- 172.21.2.112:0/3104245982 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd7600186c0 0x7fd76001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:55.900 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.911 7fd77dcdd700 1 --2- 172.21.2.112:0/3104245982 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd7600186c0 0x7fd76001abc0 crc :-1 s=READY pgs=649 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:55.929 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.942 7fd77fec1700 1 -- 172.21.2.112:0/3104245982 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd77807b2f0 con 0x7fd7781adce0 2021-02-10T11:33:55.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:55.944 7fd76a7fc700 1 -- 172.21.2.112:0/3104245982 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd770000cc0 con 0x7fd7781adce0 2021-02-10T11:33:56.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:56.074 7fd77fec1700 1 -- 172.21.2.112:0/3104245982 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd7780f6a30 con 0x7fd7781adce0 2021-02-10T11:33:56.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:56.075 7fd76a7fc700 1 -- 172.21.2.112:0/3104245982 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+2101 (secure 0 0 0) 0x7fd770001270 con 0x7fd7781adce0 2021-02-10T11:33:56.062 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:33:56.062 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":40,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7881,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/1476007789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":1476007789},{"type":"v1","addr":"172.21.2.112:6833","nonce":1476007789}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":35}],"filesystems":[{"mdsmap":{"epoch":39,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:33:23.550791","modified":"2021-02-10 11:33:25.562811","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6595":{"gid":6595,"name":"b","rank":0,"incarnation":36,"state":"up:active","state_seq":3,"addr":"172.21.2.120:6835/4241229406","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":4241229406},{"type":"v1","addr":"172.21.2.120:6835","nonce":4241229406}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2021-02-10T11:33:56.062 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 40 2021-02-10T11:33:56.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:56.077 7fd77fec1700 1 -- 172.21.2.112:0/3104245982 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd7600186c0 msgr2=0x7fd76001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:56.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:56.077 7fd77fec1700 1 --2- 172.21.2.112:0/3104245982 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd7600186c0 0x7fd76001abc0 crc :-1 s=READY pgs=649 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:56.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:56.077 7fd77fec1700 1 -- 172.21.2.112:0/3104245982 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd7781adce0 msgr2=0x7fd7780b7d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:56.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:56.077 7fd77fec1700 1 --2- 172.21.2.112:0/3104245982 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd7781adce0 0x7fd7780b7d70 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 rx=0x7fd770006f30 tx=0x7fd770002d20).stop 2021-02-10T11:33:56.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:56.077 7fd77fec1700 1 -- 172.21.2.112:0/3104245982 shutdown_connections 2021-02-10T11:33:56.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:56.077 7fd77fec1700 1 --2- 172.21.2.112:0/3104245982 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd7600186c0 0x7fd76001abc0 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:56.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:56.077 7fd77fec1700 1 --2- 172.21.2.112:0/3104245982 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd7781adce0 0x7fd7780b7d70 unknown :-1 s=CLOSED pgs=1312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:56.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:56.077 7fd77fec1700 1 --2- 172.21.2.112:0/3104245982 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd7781aee70 0x7fd7781ad6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:56.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:56.077 7fd77fec1700 1 --2- 172.21.2.112:0/3104245982 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd77819ead0 0x7fd7781ae850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:56.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:56.077 7fd77fec1700 1 -- 172.21.2.112:0/3104245982 >> 172.21.2.112:0/3104245982 conn(0x7fd7781a38f0 msgr2=0x7fd778195cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:56.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:56.077 7fd77fec1700 1 -- 172.21.2.112:0/3104245982 shutdown_connections 2021-02-10T11:33:56.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:33:56.077 7fd77fec1700 1 -- 172.21.2.112:0/3104245982 wait complete. 2021-02-10T11:33:56.274 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 39, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:33:23.550791', 'modified': '2021-02-10 11:33:25.562811', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6595}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6595': {'gid': 6595, 'name': 'b', 'rank': 0, 'incarnation': 36, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.120:6835/4241229406', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.120:6834', 'nonce': 4241229406}, {'type': 'v1', 'addr': '172.21.2.120:6835', 'nonce': 4241229406}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [11], 'metadata_pool': 10, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:33:56.274 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:33:56.274 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:33:56.671 INFO:teuthology.orchestra.run.gibba020.stdout:{ 2021-02-10T11:33:56.671 INFO:teuthology.orchestra.run.gibba020.stdout: "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", 2021-02-10T11:33:56.671 INFO:teuthology.orchestra.run.gibba020.stdout: "whoami": 0, 2021-02-10T11:33:56.672 INFO:teuthology.orchestra.run.gibba020.stdout: "id": 6595, 2021-02-10T11:33:56.672 INFO:teuthology.orchestra.run.gibba020.stdout: "want_state": "up:active", 2021-02-10T11:33:56.672 INFO:teuthology.orchestra.run.gibba020.stdout: "state": "up:active", 2021-02-10T11:33:56.672 INFO:teuthology.orchestra.run.gibba020.stdout: "rank_uptime": 33.131773656, 2021-02-10T11:33:56.672 INFO:teuthology.orchestra.run.gibba020.stdout: "mdsmap_epoch": 39, 2021-02-10T11:33:56.672 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch": 55, 2021-02-10T11:33:56.672 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch_barrier": 55, 2021-02-10T11:33:56.672 INFO:teuthology.orchestra.run.gibba020.stdout: "uptime": 38.761528849000001 2021-02-10T11:33:56.672 INFO:teuthology.orchestra.run.gibba020.stdout:} 2021-02-10T11:33:56.675 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", "whoami": 0, "id": 6595, "want_state": "up:active", "state": "up:active", "rank_uptime": 33.131773656, "mdsmap_epoch": 39, "osdmap_epoch": 55, "osdmap_epoch_barrier": 55, "uptime": 38.761528849000001 } 2021-02-10T11:33:56.675 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:33:56.675 INFO:tasks.cephfs.fuse_mount:Client client.alice config is {} 2021-02-10T11:33:56.676 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.alice at ubuntu@gibba012.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2021-02-10T11:33:56.676 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:56.676 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:33:57.010 DEBUG:teuthology.orchestra.run.gibba012:> sudo modprobe fuse 2021-02-10T11:33:57.355 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:33:57.355 DEBUG:teuthology.orchestra.run.gibba012:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:33:57.698 INFO:teuthology.orchestra.run.gibba012.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:33:57.698 INFO:teuthology.orchestra.run.gibba012.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:33:57.699 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:33:57.699 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:57.699 DEBUG:teuthology.orchestra.run.gibba012:> ls /sys/fs/fuse/connections 2021-02-10T11:33:58.034 INFO:teuthology.orchestra.run.gibba012.stdout:42 2021-02-10T11:33:58.034 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [42] 2021-02-10T11:33:58.035 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --client_mountpoint=/volumes/group_0000000000544641/subvolume_0000000000670007 --name client.alice /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:33:58.037 DEBUG:teuthology.orchestra.run.gibba012:> sudo modprobe fuse 2021-02-10T11:33:58.378 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:33:58.378 DEBUG:teuthology.orchestra.run.gibba012:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:33:58.445 INFO:tasks.cephfs.fuse_mount.ceph-fuse.alice.gibba012.stderr:ceph-fuse[131139]: starting ceph client 2021-02-10T11:33:58.445 INFO:tasks.cephfs.fuse_mount.ceph-fuse.alice.gibba012.stderr:2021-02-10 11:33:58.458 7efc902daf80 -1 init, newargv = 0x56494277e080 newargc=9 2021-02-10T11:33:58.451 INFO:tasks.cephfs.fuse_mount.ceph-fuse.alice.gibba012.stderr:ceph-fuse[131139]: starting fuse 2021-02-10T11:33:58.720 INFO:teuthology.orchestra.run.gibba012.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:33:58.721 INFO:teuthology.orchestra.run.gibba012.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:33:58.721 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:33:58.722 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:58.722 DEBUG:teuthology.orchestra.run.gibba012:> ls /sys/fs/fuse/connections 2021-02-10T11:33:59.057 INFO:teuthology.orchestra.run.gibba012.stdout:42 2021-02-10T11:33:59.057 INFO:teuthology.orchestra.run.gibba012.stdout:43 2021-02-10T11:33:59.057 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42, 43] 2021-02-10T11:33:59.057 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:59.058 DEBUG:teuthology.orchestra.run.gibba012:> sudo python3 -c ' 2021-02-10T11:33:59.058 DEBUG:teuthology.orchestra.run.gibba012:> import glob 2021-02-10T11:33:59.058 DEBUG:teuthology.orchestra.run.gibba012:> import re 2021-02-10T11:33:59.058 DEBUG:teuthology.orchestra.run.gibba012:> import os 2021-02-10T11:33:59.058 DEBUG:teuthology.orchestra.run.gibba012:> import subprocess 2021-02-10T11:33:59.058 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:33:59.058 DEBUG:teuthology.orchestra.run.gibba012:> def find_socket(client_name): 2021-02-10T11:33:59.058 DEBUG:teuthology.orchestra.run.gibba012:> asok_path = "/var/run/ceph/ceph-client.alice.*.asok" 2021-02-10T11:33:59.058 DEBUG:teuthology.orchestra.run.gibba012:> files = glob.glob(asok_path) 2021-02-10T11:33:59.058 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:33:59.058 DEBUG:teuthology.orchestra.run.gibba012:> # Given a non-glob path, it better be there 2021-02-10T11:33:59.058 DEBUG:teuthology.orchestra.run.gibba012:> if "*" not in asok_path: 2021-02-10T11:33:59.058 DEBUG:teuthology.orchestra.run.gibba012:> assert(len(files) == 1) 2021-02-10T11:33:59.058 DEBUG:teuthology.orchestra.run.gibba012:> return files[0] 2021-02-10T11:33:59.059 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:33:59.059 DEBUG:teuthology.orchestra.run.gibba012:> for f in files: 2021-02-10T11:33:59.059 DEBUG:teuthology.orchestra.run.gibba012:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:33:59.059 DEBUG:teuthology.orchestra.run.gibba012:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:33:59.059 DEBUG:teuthology.orchestra.run.gibba012:> return f 2021-02-10T11:33:59.059 DEBUG:teuthology.orchestra.run.gibba012:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:33:59.059 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:33:59.059 DEBUG:teuthology.orchestra.run.gibba012:> print(find_socket("client.alice")) 2021-02-10T11:33:59.059 DEBUG:teuthology.orchestra.run.gibba012:> ' 2021-02-10T11:33:59.421 INFO:teuthology.orchestra.run.gibba012.stdout:/var/run/ceph/ceph-client.alice.131139.asok 2021-02-10T11:33:59.423 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.alice.131139.asok 2021-02-10T11:33:59.423 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:59.424 DEBUG:teuthology.orchestra.run.gibba012:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.alice.131139.asok status 2021-02-10T11:33:59.812 INFO:teuthology.orchestra.run.gibba012.stdout:{ 2021-02-10T11:33:59.812 INFO:teuthology.orchestra.run.gibba012.stdout: "metadata": { 2021-02-10T11:33:59.812 INFO:teuthology.orchestra.run.gibba012.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:33:59.812 INFO:teuthology.orchestra.run.gibba012.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:33:59.812 INFO:teuthology.orchestra.run.gibba012.stdout: "entity_id": "alice", 2021-02-10T11:33:59.812 INFO:teuthology.orchestra.run.gibba012.stdout: "hostname": "gibba012", 2021-02-10T11:33:59.812 INFO:teuthology.orchestra.run.gibba012.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2021-02-10T11:33:59.812 INFO:teuthology.orchestra.run.gibba012.stdout: "pid": "131139", 2021-02-10T11:33:59.812 INFO:teuthology.orchestra.run.gibba012.stdout: "root": "/volumes/group_0000000000544641/subvolume_0000000000670007" 2021-02-10T11:33:59.812 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:59.813 INFO:teuthology.orchestra.run.gibba012.stdout: "dentry_count": 0, 2021-02-10T11:33:59.813 INFO:teuthology.orchestra.run.gibba012.stdout: "dentry_pinned_count": 0, 2021-02-10T11:33:59.813 INFO:teuthology.orchestra.run.gibba012.stdout: "id": 8163, 2021-02-10T11:33:59.813 INFO:teuthology.orchestra.run.gibba012.stdout: "inst": { 2021-02-10T11:33:59.813 INFO:teuthology.orchestra.run.gibba012.stdout: "name": { 2021-02-10T11:33:59.813 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "client", 2021-02-10T11:33:59.813 INFO:teuthology.orchestra.run.gibba012.stdout: "num": 8163 2021-02-10T11:33:59.813 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:59.813 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": { 2021-02-10T11:33:59.813 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:33:59.813 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:0", 2021-02-10T11:33:59.813 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 1116442191 2021-02-10T11:33:59.813 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:33:59.814 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:59.814 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": { 2021-02-10T11:33:59.814 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:33:59.814 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:0", 2021-02-10T11:33:59.814 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 1116442191 2021-02-10T11:33:59.814 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:33:59.814 INFO:teuthology.orchestra.run.gibba012.stdout: "inst_str": "client.8163 172.21.2.112:0/1116442191", 2021-02-10T11:33:59.814 INFO:teuthology.orchestra.run.gibba012.stdout: "addr_str": "172.21.2.112:0/1116442191", 2021-02-10T11:33:59.814 INFO:teuthology.orchestra.run.gibba012.stdout: "inode_count": 1, 2021-02-10T11:33:59.814 INFO:teuthology.orchestra.run.gibba012.stdout: "mds_epoch": 39, 2021-02-10T11:33:59.814 INFO:teuthology.orchestra.run.gibba012.stdout: "osd_epoch": 55, 2021-02-10T11:33:59.814 INFO:teuthology.orchestra.run.gibba012.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:33:59.815 INFO:teuthology.orchestra.run.gibba012.stdout: "blacklisted": false 2021-02-10T11:33:59.815 INFO:teuthology.orchestra.run.gibba012.stdout:} 2021-02-10T11:33:59.816 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:59.816 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:33:59.816 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:34:00.152 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:34:00.152 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:34:00.152 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.1 && sudo dd if=/dev/urandom of=data.bin bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:34:00.526 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:34:00.526 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:34:00.526 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0254478 s, 41.2 MB/s 2021-02-10T11:34:00.527 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume authorize cephfs subvolume_0000000000670007 alice --group_name group_0000000000544641 --tenant_id tenant_id --access_level r 2021-02-10T11:34:00.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:00.962 7fd8f1ed3700 1 Processor -- start 2021-02-10T11:34:00.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:00.962 7fd8f1ed3700 1 -- start start 2021-02-10T11:34:00.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:00.963 7fd8f1ed3700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd8ec0e5170 0x7fd8ec1b38e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:00.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:00.963 7fd8f1ed3700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fd8ec1b3f00 0x7fd8ec077510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:00.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:00.963 7fd8f1ed3700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fd8ec0e5570 0x7fd8ec077b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:00.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:00.963 7fd8f1ed3700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8ec078b90 con 0x7fd8ec1b3f00 2021-02-10T11:34:00.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:00.963 7fd8f1ed3700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8ec078150 con 0x7fd8ec0e5570 2021-02-10T11:34:00.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:00.963 7fd8f1ed3700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8ec078340 con 0x7fd8ec0e5170 2021-02-10T11:34:00.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:00.963 7fd8eaffd700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fd8ec1b3f00 0x7fd8ec077510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:00.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:00.963 7fd8eaffd700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fd8ec1b3f00 0x7fd8ec077510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:50194/0 (socket says 172.21.2.112:50194) 2021-02-10T11:34:00.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:00.963 7fd8eaffd700 1 -- 172.21.2.112:0/4113706912 learned_addr learned my addr 172.21.2.112:0/4113706912 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:34:00.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:00.963 7fd8eaffd700 1 -- 172.21.2.112:0/4113706912 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd8ec0e5170 msgr2=0x7fd8ec1b38e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:34:00.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:00.963 7fd8eaffd700 1 --2- 172.21.2.112:0/4113706912 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd8ec0e5170 0x7fd8ec1b38e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:00.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:00.963 7fd8eaffd700 1 -- 172.21.2.112:0/4113706912 >> v2:172.21.2.120:3300/0 conn(0x7fd8ec0e5570 msgr2=0x7fd8ec077b30 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:34:00.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:00.963 7fd8eaffd700 1 --2- 172.21.2.112:0/4113706912 >> v2:172.21.2.120:3300/0 conn(0x7fd8ec0e5570 0x7fd8ec077b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:00.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:00.963 7fd8eaffd700 1 -- 172.21.2.112:0/4113706912 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8ec078560 con 0x7fd8ec1b3f00 2021-02-10T11:34:00.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:00.963 7fd8eaffd700 1 --2- 172.21.2.112:0/4113706912 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd8ec1b3f00 0x7fd8ec077510 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 rx=0x7fd8e0006700 tx=0x7fd8e0009b00).ready entity=mon.0 client_cookie=844978cfd613fe4 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:00.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:00.964 7fd8ea7fc700 1 -- 172.21.2.112:0/4113706912 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd8e000c0c0 con 0x7fd8ec1b3f00 2021-02-10T11:34:00.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:00.964 7fd8ea7fc700 1 -- 172.21.2.112:0/4113706912 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8e000c400 con 0x7fd8ec1b3f00 2021-02-10T11:34:00.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:00.964 7fd8ea7fc700 1 -- 172.21.2.112:0/4113706912 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd8e000c860 con 0x7fd8ec1b3f00 2021-02-10T11:34:00.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:00.964 7fd8f1ed3700 1 -- 172.21.2.112:0/4113706912 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd8ec1b3f00 msgr2=0x7fd8ec077510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:00.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:00.964 7fd8f1ed3700 1 --2- 172.21.2.112:0/4113706912 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd8ec1b3f00 0x7fd8ec077510 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 rx=0x7fd8e0006700 tx=0x7fd8e0009b00).stop 2021-02-10T11:34:00.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:00.964 7fd8f1ed3700 1 -- 172.21.2.112:0/4113706912 shutdown_connections 2021-02-10T11:34:00.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:00.964 7fd8f1ed3700 1 --2- 172.21.2.112:0/4113706912 >> v2:172.21.2.120:3300/0 conn(0x7fd8ec0e5570 0x7fd8ec077b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:00.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:00.964 7fd8f1ed3700 1 --2- 172.21.2.112:0/4113706912 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd8ec1b3f00 0x7fd8ec077510 unknown :-1 s=CLOSED pgs=1315 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:00.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:00.964 7fd8f1ed3700 1 --2- 172.21.2.112:0/4113706912 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd8ec0e5170 0x7fd8ec1b38e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:00.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:00.964 7fd8f1ed3700 1 -- 172.21.2.112:0/4113706912 >> 172.21.2.112:0/4113706912 conn(0x7fd8ec0ce820 msgr2=0x7fd8ec0e4660 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:00.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:00.964 7fd8f1ed3700 1 -- 172.21.2.112:0/4113706912 shutdown_connections 2021-02-10T11:34:00.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:00.964 7fd8f1ed3700 1 -- 172.21.2.112:0/4113706912 wait complete. 2021-02-10T11:34:00.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:00.979 7fd8f1ed3700 1 Processor -- start 2021-02-10T11:34:00.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:00.979 7fd8f1ed3700 1 -- start start 2021-02-10T11:34:00.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:00.979 7fd8f1ed3700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd8ec1a59d0 0x7fd8ec1a83e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:00.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:00.979 7fd8f1ed3700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd8ec0ad8a0 0x7fd8ec0adca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:00.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:00.979 7fd8f1ed3700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd8ec10e290 0x7fd8ec10e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:00.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:00.979 7fd8f1ed3700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8ec1a7e60 con 0x7fd8ec1a59d0 2021-02-10T11:34:00.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:00.979 7fd8f1ed3700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8ec1b1440 con 0x7fd8ec0ad8a0 2021-02-10T11:34:00.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:00.979 7fd8f1ed3700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8ec1b1620 con 0x7fd8ec10e290 2021-02-10T11:34:00.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:00.980 7fd8eb7fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd8ec1a59d0 0x7fd8ec1a83e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:00.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:00.980 7fd8eb7fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd8ec1a59d0 0x7fd8ec1a83e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:50200/0 (socket says 172.21.2.112:50200) 2021-02-10T11:34:00.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:00.980 7fd8eb7fe700 1 -- 172.21.2.112:0/3463574639 learned_addr learned my addr 172.21.2.112:0/3463574639 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:34:00.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:00.980 7fd8eb7fe700 1 -- 172.21.2.112:0/3463574639 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd8ec10e290 msgr2=0x7fd8ec10e690 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:34:00.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:00.980 7fd8eb7fe700 1 --2- 172.21.2.112:0/3463574639 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd8ec10e290 0x7fd8ec10e690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:00.971 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:00.980 7fd8eb7fe700 1 -- 172.21.2.112:0/3463574639 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd8ec0ad8a0 msgr2=0x7fd8ec0adca0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:34:00.971 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:00.980 7fd8eb7fe700 1 --2- 172.21.2.112:0/3463574639 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd8ec0ad8a0 0x7fd8ec0adca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:00.971 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:00.980 7fd8eb7fe700 1 -- 172.21.2.112:0/3463574639 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8ec1b1840 con 0x7fd8ec1a59d0 2021-02-10T11:34:00.971 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:00.980 7fd8eb7fe700 1 --2- 172.21.2.112:0/3463574639 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd8ec1a59d0 0x7fd8ec1a83e0 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 rx=0x7fd8dc00b5c0 tx=0x7fd8dc00b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:00.971 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:00.980 7fd8d3fff700 1 -- 172.21.2.112:0/3463574639 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd8dc0009b0 con 0x7fd8ec1a59d0 2021-02-10T11:34:00.971 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:00.980 7fd8d3fff700 1 -- 172.21.2.112:0/3463574639 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8dc00bd50 con 0x7fd8ec1a59d0 2021-02-10T11:34:00.971 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:00.980 7fd8f1ed3700 1 -- 172.21.2.112:0/3463574639 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8ec1b1840 con 0x7fd8ec1a59d0 2021-02-10T11:34:00.971 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:00.980 7fd8f1ed3700 1 -- 172.21.2.112:0/3463574639 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8ec1b1dd0 con 0x7fd8ec1a59d0 2021-02-10T11:34:00.971 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:00.981 7fd8d3fff700 1 -- 172.21.2.112:0/3463574639 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd8dc0062d0 con 0x7fd8ec1a59d0 2021-02-10T11:34:00.971 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:00.981 7fd8d3fff700 1 -- 172.21.2.112:0/3463574639 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd8dc006030 con 0x7fd8ec1a59d0 2021-02-10T11:34:00.971 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:00.981 7fd8d3fff700 1 --2- 172.21.2.112:0/3463574639 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd8cc0186d0 0x7fd8cc01abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:00.971 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:00.981 7fd8d3fff700 1 -- 172.21.2.112:0/3463574639 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fd8dc032670 con 0x7fd8ec1a59d0 2021-02-10T11:34:00.971 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:00.982 7fd8eaffd700 1 --2- 172.21.2.112:0/3463574639 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd8cc0186d0 0x7fd8cc01abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:00.971 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:00.982 7fd8eaffd700 1 --2- 172.21.2.112:0/3463574639 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd8cc0186d0 0x7fd8cc01abd0 crc :-1 s=READY pgs=650 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:01.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:01.013 7fd8f1ed3700 1 -- 172.21.2.112:0/3463574639 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8ec0fe290 con 0x7fd8ec1a59d0 2021-02-10T11:34:01.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:01.015 7fd8d3fff700 1 -- 172.21.2.112:0/3463574639 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd8dc000990 con 0x7fd8ec1a59d0 2021-02-10T11:34:01.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:01.141 7fd8f1ed3700 1 -- 172.21.2.112:0/3463574639 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"vol_name": "cephfs", "sub_name": "subvolume_0000000000670007", "target": ["mgr", ""], "tenant_id": "tenant_id", "group_name": "group_0000000000544641", "access_level": "r", "prefix": "fs subvolume authorize", "auth_id": "alice"}) v1 -- 0x7fd8ec1b1dd0 con 0x7fd8cc0186d0 2021-02-10T11:34:01.172 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:01.185 7fd8d3fff700 1 -- 172.21.2.112:0/3463574639 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+41 (crc 0 0 0) 0x7fd8e0009440 con 0x7fd8cc0186d0 2021-02-10T11:34:01.192 INFO:teuthology.orchestra.run.gibba012.stdout:AQCexCNgfqx8GBAANue1hs5gTtkZO/pClbOpEw== 2021-02-10T11:34:01.193 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:01.206 7fd8f1ed3700 1 -- 172.21.2.112:0/3463574639 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd8cc0186d0 msgr2=0x7fd8cc01abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:01.194 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:01.206 7fd8f1ed3700 1 --2- 172.21.2.112:0/3463574639 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd8cc0186d0 0x7fd8cc01abd0 crc :-1 s=READY pgs=650 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:01.194 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:01.206 7fd8f1ed3700 1 -- 172.21.2.112:0/3463574639 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd8ec1a59d0 msgr2=0x7fd8ec1a83e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:01.194 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:01.206 7fd8f1ed3700 1 --2- 172.21.2.112:0/3463574639 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd8ec1a59d0 0x7fd8ec1a83e0 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 rx=0x7fd8dc00b5c0 tx=0x7fd8dc00b960).stop 2021-02-10T11:34:01.194 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:01.207 7fd8f1ed3700 1 -- 172.21.2.112:0/3463574639 shutdown_connections 2021-02-10T11:34:01.194 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:01.207 7fd8f1ed3700 1 --2- 172.21.2.112:0/3463574639 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd8cc0186d0 0x7fd8cc01abd0 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:01.194 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:01.207 7fd8f1ed3700 1 --2- 172.21.2.112:0/3463574639 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd8ec0ad8a0 0x7fd8ec0adca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:01.194 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:01.207 7fd8f1ed3700 1 --2- 172.21.2.112:0/3463574639 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd8ec10e290 0x7fd8ec10e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:01.194 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:01.207 7fd8f1ed3700 1 --2- 172.21.2.112:0/3463574639 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd8ec1a59d0 0x7fd8ec1a83e0 unknown :-1 s=CLOSED pgs=1316 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:01.194 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:01.207 7fd8f1ed3700 1 -- 172.21.2.112:0/3463574639 >> 172.21.2.112:0/3463574639 conn(0x7fd8ec1a4620 msgr2=0x7fd8ec1af390 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:01.194 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:01.207 7fd8f1ed3700 1 -- 172.21.2.112:0/3463574639 shutdown_connections 2021-02-10T11:34:01.194 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:01.207 7fd8f1ed3700 1 -- 172.21.2.112:0/3463574639 wait complete. 2021-02-10T11:34:01.202 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:01.202 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:34:01.202 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:34:01.537 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:34:01.538 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:34:01.538 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba012.front.sepia.ceph.com... 2021-02-10T11:34:01.538 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:34:01.538 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:34:01.890 INFO:tasks.cephfs.fuse_mount.ceph-fuse.alice.gibba012.stderr:ceph-fuse[131139]: fuse finished with error 0 and tester_r 0 2021-02-10T11:34:01.891 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:01.891 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:34:01.891 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:34:02.225 INFO:teuthology.orchestra.run.gibba012.stdout:ext2/ext3 2021-02-10T11:34:02.225 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:34:02.226 INFO:teuthology.orchestra.run:waiting for 900 2021-02-10T11:34:08.328 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:34:08.328 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:34:08.663 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:34:09.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.087 7f96eaf02700 1 Processor -- start 2021-02-10T11:34:09.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.087 7f96eaf02700 1 -- start start 2021-02-10T11:34:09.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.088 7f96eaf02700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f96e40e53e0 0x7f96e41b0550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:09.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.088 7f96eaf02700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96e4082d50 con 0x7f96e40842e0 2021-02-10T11:34:09.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.088 7f96eaf02700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96e40e57e0 con 0x7f96e41b0b70 2021-02-10T11:34:09.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.088 7f96eaf02700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96e4084850 con 0x7f96e40e53e0 2021-02-10T11:34:09.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.088 7f96e951f700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f96e40842e0 0x7f96e4082100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:49670/0 (socket says 172.21.2.112:49670) 2021-02-10T11:34:09.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.088 7f96e951f700 1 -- 172.21.2.112:0/3108461194 learned_addr learned my addr 172.21.2.112:0/3108461194 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:34:09.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.088 7f96e8d1e700 1 --2- 172.21.2.112:0/3108461194 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f96e40e53e0 0x7f96e41b0550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:09.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.088 7f96e37fe700 1 -- 172.21.2.112:0/3108461194 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3880373992 0 0) 0x7f96dc001500 con 0x7f96e40842e0 2021-02-10T11:34:09.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.088 7f96e37fe700 1 -- 172.21.2.112:0/3108461194 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96d0002fe0 con 0x7f96e40842e0 2021-02-10T11:34:09.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.088 7f96e37fe700 1 -- 172.21.2.112:0/3108461194 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1482302313 0 0) 0x7f96dc001500 con 0x7f96e40842e0 2021-02-10T11:34:09.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.088 7f96e37fe700 1 -- 172.21.2.112:0/3108461194 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f96e40e53e0 msgr2=0x7f96e41b0550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:09.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.088 7f96e37fe700 1 --2- 172.21.2.112:0/3108461194 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f96e40e53e0 0x7f96e41b0550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:09.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.088 7f96e37fe700 1 -- 172.21.2.112:0/3108461194 >> v1:172.21.2.120:6789/0 conn(0x7f96e41b0b70 legacy=0x7f96e4081910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:09.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.088 7f96e37fe700 1 -- 172.21.2.112:0/3108461194 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96e40e5fe0 con 0x7f96e40842e0 2021-02-10T11:34:09.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.088 7f96e37fe700 1 -- 172.21.2.112:0/3108461194 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f96dc001a00 con 0x7f96e40842e0 2021-02-10T11:34:09.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.088 7f96e37fe700 1 -- 172.21.2.112:0/3108461194 >> v1:172.21.2.112:6789/0 conn(0x7f96e40842e0 legacy=0x7f96e4082100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:09.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.088 7f96e37fe700 1 --2- 172.21.2.112:0/3108461194 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f96d0003a70 0x7f96d0005f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:09.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.088 7f96e37fe700 1 --2- 172.21.2.112:0/3108461194 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f96d00067a0 0x7f96d0008ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:09.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.088 7f96e37fe700 1 --2- 172.21.2.112:0/3108461194 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f96d00095a0 0x7f96d000ba90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:09.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.088 7f96e37fe700 1 -- 172.21.2.112:0/3108461194 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96d000c2e0 con 0x7f96d00067a0 2021-02-10T11:34:09.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.088 7f96e37fe700 1 -- 172.21.2.112:0/3108461194 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96d000c500 con 0x7f96d0003a70 2021-02-10T11:34:09.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.089 7f96e37fe700 1 -- 172.21.2.112:0/3108461194 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96d0050730 con 0x7f96d00095a0 2021-02-10T11:34:09.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.089 7f96e3fff700 1 --2- 172.21.2.112:0/3108461194 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f96d00067a0 0x7f96d0008ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:09.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.089 7f96e8d1e700 1 --2- 172.21.2.112:0/3108461194 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f96d0003a70 0x7f96d0005f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:09.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.089 7f96e951f700 1 --2- 172.21.2.112:0/3108461194 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f96d00095a0 0x7f96d000ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:09.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.089 7f96e3fff700 1 -- 172.21.2.112:0/3108461194 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f96d00095a0 msgr2=0x7f96d000ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:09.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.089 7f96e3fff700 1 --2- 172.21.2.112:0/3108461194 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f96d00095a0 0x7f96d000ba90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:09.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.089 7f96e3fff700 1 -- 172.21.2.112:0/3108461194 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f96d0003a70 msgr2=0x7f96d0005f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:09.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.089 7f96e3fff700 1 --2- 172.21.2.112:0/3108461194 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f96d0003a70 0x7f96d0005f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:09.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.089 7f96e3fff700 1 -- 172.21.2.112:0/3108461194 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96d0050990 con 0x7f96d00067a0 2021-02-10T11:34:09.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.089 7f96e3fff700 1 --2- 172.21.2.112:0/3108461194 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f96d00067a0 0x7f96d0008ca0 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 rx=0x7f96d4006ac0 tx=0x7f96d4009af0).ready entity=mon.0 client_cookie=63e1f94b01e51f61 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:09.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.089 7f96e37fe700 1 -- 172.21.2.112:0/3108461194 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f96d4007280 con 0x7f96d00067a0 2021-02-10T11:34:09.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.089 7f96e37fe700 1 -- 172.21.2.112:0/3108461194 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96d4007580 con 0x7f96d00067a0 2021-02-10T11:34:09.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.089 7f96e37fe700 1 -- 172.21.2.112:0/3108461194 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f96d4007a60 con 0x7f96d00067a0 2021-02-10T11:34:09.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.089 7f96eaf02700 1 -- 172.21.2.112:0/3108461194 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f96d00067a0 msgr2=0x7f96d0008ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:09.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.089 7f96eaf02700 1 --2- 172.21.2.112:0/3108461194 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f96d00067a0 0x7f96d0008ca0 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 rx=0x7f96d4006ac0 tx=0x7f96d4009af0).stop 2021-02-10T11:34:09.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.089 7f96e951f700 1 -- 172.21.2.112:0/3108461194 reap_dead start 2021-02-10T11:34:09.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.090 7f96eaf02700 1 -- 172.21.2.112:0/3108461194 shutdown_connections 2021-02-10T11:34:09.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.090 7f96eaf02700 1 -- 172.21.2.112:0/3108461194 >> 172.21.2.112:0/3108461194 conn(0x7f96e40ce500 msgr2=0x7f96e40e4610 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:09.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.090 7f96eaf02700 1 -- 172.21.2.112:0/3108461194 shutdown_connections 2021-02-10T11:34:09.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.090 7f96eaf02700 1 -- 172.21.2.112:0/3108461194 wait complete. 2021-02-10T11:34:09.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.104 7f96eaf02700 1 Processor -- start 2021-02-10T11:34:09.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.104 7f96eaf02700 1 -- start start 2021-02-10T11:34:09.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.105 7f96eaf02700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f96e40c22f0 0x7f96e4191340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:09.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.105 7f96eaf02700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f96e40c17c0 0x7f96e40c1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:09.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.105 7f96eaf02700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f96e40b6ef0 0x7f96e40b72f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:09.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.105 7f96eaf02700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96e4191d70 con 0x7f96e40c22f0 2021-02-10T11:34:09.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.105 7f96eaf02700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96e4191960 con 0x7f96e40b6ef0 2021-02-10T11:34:09.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.105 7f96eaf02700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96e4191b40 con 0x7f96e40c17c0 2021-02-10T11:34:09.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.105 7f96e8d1e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f96e40c22f0 0x7f96e4191340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:09.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.105 7f96e8d1e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f96e40c22f0 0x7f96e4191340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:50222/0 (socket says 172.21.2.112:50222) 2021-02-10T11:34:09.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.105 7f96e8d1e700 1 -- 172.21.2.112:0/1576017705 learned_addr learned my addr 172.21.2.112:0/1576017705 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:34:09.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.105 7f96e8d1e700 1 -- 172.21.2.112:0/1576017705 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f96e40c17c0 msgr2=0x7f96e40c1bc0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:34:09.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.105 7f96e8d1e700 1 --2- 172.21.2.112:0/1576017705 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f96e40c17c0 0x7f96e40c1bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:09.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.105 7f96e8d1e700 1 -- 172.21.2.112:0/1576017705 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f96e40b6ef0 msgr2=0x7f96e40b72f0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:34:09.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.105 7f96e8d1e700 1 --2- 172.21.2.112:0/1576017705 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f96e40b6ef0 0x7f96e40b72f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:09.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.105 7f96e8d1e700 1 -- 172.21.2.112:0/1576017705 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96e40b7910 con 0x7f96e40c22f0 2021-02-10T11:34:09.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.105 7f96e8d1e700 1 --2- 172.21.2.112:0/1576017705 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f96e40c22f0 0x7f96e4191340 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 rx=0x7f96cc006f20 tx=0x7f96cc00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:09.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.105 7f96e17fa700 1 -- 172.21.2.112:0/1576017705 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f96cc006be0 con 0x7f96e40c22f0 2021-02-10T11:34:09.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.105 7f96e17fa700 1 -- 172.21.2.112:0/1576017705 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96cc00bd80 con 0x7f96e40c22f0 2021-02-10T11:34:09.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.106 7f96e17fa700 1 -- 172.21.2.112:0/1576017705 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f96cc0052e0 con 0x7f96e40c22f0 2021-02-10T11:34:09.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.106 7f96eaf02700 1 -- 172.21.2.112:0/1576017705 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96e40b7910 con 0x7f96e40c22f0 2021-02-10T11:34:09.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.106 7f96eaf02700 1 -- 172.21.2.112:0/1576017705 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96e40b7910 con 0x7f96e40c22f0 2021-02-10T11:34:09.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.106 7f96e17fa700 1 -- 172.21.2.112:0/1576017705 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f96cc006740 con 0x7f96e40c22f0 2021-02-10T11:34:09.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.106 7f96e17fa700 1 --2- 172.21.2.112:0/1576017705 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f96c80186c0 0x7f96c801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:09.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.106 7f96e17fa700 1 -- 172.21.2.112:0/1576017705 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f96cc0327a0 con 0x7f96e40c22f0 2021-02-10T11:34:09.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.107 7f96e3fff700 1 --2- 172.21.2.112:0/1576017705 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f96c80186c0 0x7f96c801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:09.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.107 7f96e3fff700 1 --2- 172.21.2.112:0/1576017705 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f96c80186c0 0x7f96c801abc0 crc :-1 s=READY pgs=651 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:09.124 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.137 7f96eaf02700 1 -- 172.21.2.112:0/1576017705 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96e40ed5d0 con 0x7f96e40c22f0 2021-02-10T11:34:09.126 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.140 7f96e17fa700 1 -- 172.21.2.112:0/1576017705 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f96cc0009b0 con 0x7f96e40c22f0 2021-02-10T11:34:09.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.264 7f96eaf02700 1 -- 172.21.2.112:0/1576017705 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f96e40ed5d0 con 0x7f96e40c22f0 2021-02-10T11:34:09.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.265 7f96e17fa700 1 -- 172.21.2.112:0/1576017705 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v55) v1 ==== 74+0+13996 (secure 0 0 0) 0x7f96cc006740 con 0x7f96e40c22f0 2021-02-10T11:34:09.252 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:34:09.252 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":55,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:33:24.553170","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:33:21.523665","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":11,"pool_name":"cephfs_data","create_time":"2021-02-10 11:33:22.595892","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:34:09.253 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.266 7f96eaf02700 1 -- 172.21.2.112:0/1576017705 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f96c80186c0 msgr2=0x7f96c801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:09.253 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.266 7f96eaf02700 1 --2- 172.21.2.112:0/1576017705 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f96c80186c0 0x7f96c801abc0 crc :-1 s=READY pgs=651 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:09.253 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.266 7f96eaf02700 1 -- 172.21.2.112:0/1576017705 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f96e40c22f0 msgr2=0x7f96e4191340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:09.253 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.266 7f96eaf02700 1 --2- 172.21.2.112:0/1576017705 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f96e40c22f0 0x7f96e4191340 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 rx=0x7f96cc006f20 tx=0x7f96cc00b990).stop 2021-02-10T11:34:09.254 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.267 7f96eaf02700 1 -- 172.21.2.112:0/1576017705 shutdown_connections 2021-02-10T11:34:09.254 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.267 7f96eaf02700 1 --2- 172.21.2.112:0/1576017705 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f96c80186c0 0x7f96c801abc0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:09.254 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.267 7f96eaf02700 1 --2- 172.21.2.112:0/1576017705 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f96e40b6ef0 0x7f96e40b72f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:09.254 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.267 7f96eaf02700 1 --2- 172.21.2.112:0/1576017705 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f96e40c17c0 0x7f96e40c1bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:09.254 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.267 7f96eaf02700 1 --2- 172.21.2.112:0/1576017705 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f96e40c22f0 0x7f96e4191340 unknown :-1 s=CLOSED pgs=1319 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:09.254 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.267 7f96eaf02700 1 -- 172.21.2.112:0/1576017705 >> 172.21.2.112:0/1576017705 conn(0x7f96e40ad1b0 msgr2=0x7f96e40b50a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:09.254 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.267 7f96eaf02700 1 -- 172.21.2.112:0/1576017705 shutdown_connections 2021-02-10T11:34:09.254 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.267 7f96eaf02700 1 -- 172.21.2.112:0/1576017705 wait complete. 2021-02-10T11:34:09.263 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:13:57.508758', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 10, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:33:21.523665', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '54', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 11, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:33:22.595892', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '55', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:34:09.263 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:34:09.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.689 7fdfb499e700 1 Processor -- start 2021-02-10T11:34:09.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.689 7fdfb499e700 1 -- start start 2021-02-10T11:34:09.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.690 7fdfb499e700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fdfb00e5570 0x7fdfb0078520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:09.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.690 7fdfb499e700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fdfb0079110 0x7fdfb0079510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:09.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.690 7fdfb499e700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfb0078b90 con 0x7fdfb00e5170 2021-02-10T11:34:09.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.690 7fdfb499e700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfb01b40d0 con 0x7fdfb0079110 2021-02-10T11:34:09.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.690 7fdfb499e700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfb00d1900 con 0x7fdfb00e5570 2021-02-10T11:34:09.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.690 7fdfaee1d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fdfb0079110 0x7fdfb0079510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:09.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.690 7fdfaee1d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fdfb0079110 0x7fdfb0079510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:50230/0 (socket says 172.21.2.112:50230) 2021-02-10T11:34:09.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.690 7fdfaee1d700 1 -- 172.21.2.112:0/3003384907 learned_addr learned my addr 172.21.2.112:0/3003384907 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:34:09.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.690 7fdfaee1d700 1 -- 172.21.2.112:0/3003384907 >> v2:172.21.2.120:3300/0 conn(0x7fdfb00e5570 msgr2=0x7fdfb0078520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:09.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.690 7fdfaee1d700 1 --2- 172.21.2.112:0/3003384907 >> v2:172.21.2.120:3300/0 conn(0x7fdfb00e5570 0x7fdfb0078520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:09.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.690 7fdfaee1d700 1 -- 172.21.2.112:0/3003384907 >> v1:172.21.2.120:6789/0 conn(0x7fdfb00e5170 legacy=0x7fdfb01b38e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:09.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.690 7fdfaee1d700 1 -- 172.21.2.112:0/3003384907 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfb00d1b20 con 0x7fdfb0079110 2021-02-10T11:34:09.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.691 7fdfaee1d700 1 --2- 172.21.2.112:0/3003384907 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdfb0079110 0x7fdfb0079510 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 rx=0x7fdfa8006f00 tx=0x7fdfa8009e90).ready entity=mon.0 client_cookie=8df4ceaace879b96 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:09.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.691 7fdfad61a700 1 -- 172.21.2.112:0/3003384907 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdfa800b300 con 0x7fdfb0079110 2021-02-10T11:34:09.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.691 7fdfad61a700 1 -- 172.21.2.112:0/3003384907 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfa800b640 con 0x7fdfb0079110 2021-02-10T11:34:09.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.691 7fdfad61a700 1 -- 172.21.2.112:0/3003384907 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdfa800baa0 con 0x7fdfb0079110 2021-02-10T11:34:09.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.691 7fdfb499e700 1 -- 172.21.2.112:0/3003384907 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdfb0079110 msgr2=0x7fdfb0079510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:09.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.691 7fdfb499e700 1 --2- 172.21.2.112:0/3003384907 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdfb0079110 0x7fdfb0079510 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 rx=0x7fdfa8006f00 tx=0x7fdfa8009e90).stop 2021-02-10T11:34:09.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.691 7fdfb499e700 1 -- 172.21.2.112:0/3003384907 shutdown_connections 2021-02-10T11:34:09.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.691 7fdfb499e700 1 --2- 172.21.2.112:0/3003384907 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdfb0079110 0x7fdfb0079510 unknown :-1 s=CLOSED pgs=1320 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:09.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.691 7fdfb499e700 1 --2- 172.21.2.112:0/3003384907 >> v2:172.21.2.120:3300/0 conn(0x7fdfb00e5570 0x7fdfb0078520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:09.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.691 7fdfb499e700 1 -- 172.21.2.112:0/3003384907 >> 172.21.2.112:0/3003384907 conn(0x7fdfb00ce820 msgr2=0x7fdfb00e4660 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:09.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.691 7fdfb499e700 1 -- 172.21.2.112:0/3003384907 shutdown_connections 2021-02-10T11:34:09.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.691 7fdfb499e700 1 -- 172.21.2.112:0/3003384907 wait complete. 2021-02-10T11:34:09.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.706 7fdfb499e700 1 Processor -- start 2021-02-10T11:34:09.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.706 7fdfb499e700 1 -- start start 2021-02-10T11:34:09.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.706 7fdfb499e700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdfb00b7860 0x7fdfb0099700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:09.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.706 7fdfb499e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdfb01a2360 0x7fdfb01a2760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:09.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.706 7fdfb499e700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fdfb00984b0 0x7fdfb00988b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:09.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.706 7fdfb499e700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfb0098ed0 con 0x7fdfb01a2360 2021-02-10T11:34:09.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.706 7fdfb499e700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfb00990f0 con 0x7fdfb00984b0 2021-02-10T11:34:09.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.706 7fdfb499e700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfb0099310 con 0x7fdfb00b7860 2021-02-10T11:34:09.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.706 7fdfade1b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdfb01a2360 0x7fdfb01a2760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:09.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.706 7fdfade1b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdfb01a2360 0x7fdfb01a2760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:50232/0 (socket says 172.21.2.112:50232) 2021-02-10T11:34:09.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.706 7fdfade1b700 1 -- 172.21.2.112:0/3904606877 learned_addr learned my addr 172.21.2.112:0/3904606877 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:34:09.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.706 7fdfae61c700 1 --2- 172.21.2.112:0/3904606877 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdfb00b7860 0x7fdfb0099700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:09.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.707 7fdfaee1d700 1 --2- 172.21.2.112:0/3904606877 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fdfb00984b0 0x7fdfb00988b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:09.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.707 7fdfade1b700 1 -- 172.21.2.112:0/3904606877 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdfb00b7860 msgr2=0x7fdfb0099700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:09.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.707 7fdfade1b700 1 --2- 172.21.2.112:0/3904606877 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdfb00b7860 0x7fdfb0099700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:09.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.707 7fdfade1b700 1 -- 172.21.2.112:0/3904606877 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fdfb00984b0 msgr2=0x7fdfb00988b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:09.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.707 7fdfade1b700 1 --2- 172.21.2.112:0/3904606877 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fdfb00984b0 0x7fdfb00988b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:09.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.707 7fdfade1b700 1 -- 172.21.2.112:0/3904606877 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfb009a350 con 0x7fdfb01a2360 2021-02-10T11:34:09.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.707 7fdfaee1d700 1 --2- 172.21.2.112:0/3904606877 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fdfb00984b0 0x7fdfb00988b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:34:09.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.707 7fdfade1b700 1 --2- 172.21.2.112:0/3904606877 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdfb01a2360 0x7fdfb01a2760 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 rx=0x7fdfa0006f00 tx=0x7fdfa000b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:09.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.707 7fdfa6ffd700 1 -- 172.21.2.112:0/3904606877 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdfa000bd30 con 0x7fdfb01a2360 2021-02-10T11:34:09.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.707 7fdfa6ffd700 1 -- 172.21.2.112:0/3904606877 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfa0000c90 con 0x7fdfb01a2360 2021-02-10T11:34:09.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.707 7fdfb499e700 1 -- 172.21.2.112:0/3904606877 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfb009a350 con 0x7fdfb01a2360 2021-02-10T11:34:09.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.707 7fdfa6ffd700 1 -- 172.21.2.112:0/3904606877 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdfa00022a0 con 0x7fdfb01a2360 2021-02-10T11:34:09.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.707 7fdfb499e700 1 -- 172.21.2.112:0/3904606877 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfb009a350 con 0x7fdfb01a2360 2021-02-10T11:34:09.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.708 7fdfa6ffd700 1 -- 172.21.2.112:0/3904606877 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fdfa0005310 con 0x7fdfb01a2360 2021-02-10T11:34:09.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.708 7fdfa6ffd700 1 --2- 172.21.2.112:0/3904606877 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fdf94018730 0x7fdf9401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:09.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.708 7fdfa6ffd700 1 -- 172.21.2.112:0/3904606877 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fdfa0032620 con 0x7fdfb01a2360 2021-02-10T11:34:09.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.708 7fdfae61c700 1 --2- 172.21.2.112:0/3904606877 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fdf94018730 0x7fdf9401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:09.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.709 7fdfae61c700 1 --2- 172.21.2.112:0/3904606877 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fdf94018730 0x7fdf9401abc0 crc :-1 s=READY pgs=652 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:09.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.740 7fdfb499e700 1 -- 172.21.2.112:0/3904606877 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfb00fe290 con 0x7fdfb01a2360 2021-02-10T11:34:09.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.742 7fdfa6ffd700 1 -- 172.21.2.112:0/3904606877 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fdfa0006b90 con 0x7fdfb01a2360 2021-02-10T11:34:09.856 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.870 7fdfb499e700 1 -- 172.21.2.112:0/3904606877 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7fdfb00fe290 con 0x7fdfb01a2360 2021-02-10T11:34:09.857 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.870 7fdfa6ffd700 1 -- 172.21.2.112:0/3904606877 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v55) v1 ==== 92+0+10 (secure 0 0 0) 0x7fdfa000bd30 con 0x7fdfb01a2360 2021-02-10T11:34:09.857 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:34:09.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.872 7fdfb499e700 1 -- 172.21.2.112:0/3904606877 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fdf94018730 msgr2=0x7fdf9401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:09.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.872 7fdfb499e700 1 --2- 172.21.2.112:0/3904606877 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fdf94018730 0x7fdf9401abc0 crc :-1 s=READY pgs=652 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:09.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.872 7fdfb499e700 1 -- 172.21.2.112:0/3904606877 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdfb01a2360 msgr2=0x7fdfb01a2760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:09.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.872 7fdfb499e700 1 --2- 172.21.2.112:0/3904606877 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdfb01a2360 0x7fdfb01a2760 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 rx=0x7fdfa0006f00 tx=0x7fdfa000b940).stop 2021-02-10T11:34:09.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.872 7fdfb499e700 1 -- 172.21.2.112:0/3904606877 shutdown_connections 2021-02-10T11:34:09.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.872 7fdfb499e700 1 --2- 172.21.2.112:0/3904606877 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fdf94018730 0x7fdf9401abc0 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:09.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.872 7fdfb499e700 1 --2- 172.21.2.112:0/3904606877 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fdfb00984b0 0x7fdfb00988b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:09.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.872 7fdfb499e700 1 --2- 172.21.2.112:0/3904606877 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdfb01a2360 0x7fdfb01a2760 unknown :-1 s=CLOSED pgs=1321 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:09.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.872 7fdfb499e700 1 --2- 172.21.2.112:0/3904606877 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdfb00b7860 0x7fdfb0099700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:09.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.872 7fdfb499e700 1 -- 172.21.2.112:0/3904606877 >> 172.21.2.112:0/3904606877 conn(0x7fdfb00b85f0 msgr2=0x7fdfb00ada70 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:09.860 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.872 7fdfb499e700 1 -- 172.21.2.112:0/3904606877 shutdown_connections 2021-02-10T11:34:09.860 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:09.872 7fdfb499e700 1 -- 172.21.2.112:0/3904606877 wait complete. 2021-02-10T11:34:09.867 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:34:10.295 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.292 7f530c452700 1 Processor -- start 2021-02-10T11:34:10.295 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.292 7f530c452700 1 -- start start 2021-02-10T11:34:10.296 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.292 7f530c452700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f53040e5170 0x7f53041b38e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:10.296 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.292 7f530c452700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f53040e5570 0x7f530407a120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:10.296 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.292 7f530c452700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5304078b90 con 0x7f53041b3f00 2021-02-10T11:34:10.296 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.292 7f530c452700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53040d1900 con 0x7f53040e5170 2021-02-10T11:34:10.296 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.292 7f530c452700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53040d1b20 con 0x7f53040e5570 2021-02-10T11:34:10.296 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.293 7f530aa6f700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f53040e5570 0x7f530407a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:10.296 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.293 7f530a26e700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f53040e5170 0x7f53041b38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:10.296 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.293 7f5309a6d700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7f53041b3f00 0x7f5304077510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:43260/0 (socket says 172.21.2.112:43260) 2021-02-10T11:34:10.296 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.293 7f5309a6d700 1 -- 172.21.2.112:0/3751835643 learned_addr learned my addr 172.21.2.112:0/3751835643 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:34:10.296 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.293 7f530aa6f700 1 -- 172.21.2.112:0/3751835643 >> v2:172.21.2.120:3300/0 conn(0x7f53040e5170 msgr2=0x7f53041b38e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:10.296 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.293 7f530aa6f700 1 --2- 172.21.2.112:0/3751835643 >> v2:172.21.2.120:3300/0 conn(0x7f53040e5170 0x7f53041b38e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:10.296 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.293 7f530aa6f700 1 -- 172.21.2.112:0/3751835643 >> v1:172.21.2.120:6789/0 conn(0x7f53041b3f00 legacy=0x7f5304077510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:10.297 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.293 7f530aa6f700 1 -- 172.21.2.112:0/3751835643 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53040d1d40 con 0x7f53040e5570 2021-02-10T11:34:10.297 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.294 7f530aa6f700 1 --2- 172.21.2.112:0/3751835643 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f53040e5570 0x7f530407a120 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f52fc006e50 tx=0x7f52fc009e70).ready entity=mon.2 client_cookie=f9bf2255ffd5c12c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:10.297 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.294 7f530926c700 1 -- 172.21.2.112:0/3751835643 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f52fc00b510 con 0x7f53040e5570 2021-02-10T11:34:10.297 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.294 7f530926c700 1 -- 172.21.2.112:0/3751835643 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52fc006790 con 0x7f53040e5570 2021-02-10T11:34:10.297 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.294 7f530926c700 1 -- 172.21.2.112:0/3751835643 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f52fc00b940 con 0x7f53040e5570 2021-02-10T11:34:10.297 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.294 7f530c452700 1 -- 172.21.2.112:0/3751835643 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f53040e5570 msgr2=0x7f530407a120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:10.297 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.294 7f530c452700 1 --2- 172.21.2.112:0/3751835643 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f53040e5570 0x7f530407a120 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f52fc006e50 tx=0x7f52fc009e70).stop 2021-02-10T11:34:10.297 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.294 7f530c452700 1 -- 172.21.2.112:0/3751835643 shutdown_connections 2021-02-10T11:34:10.297 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.294 7f530c452700 1 --2- 172.21.2.112:0/3751835643 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f53040e5570 0x7f530407a120 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:10.297 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.294 7f530c452700 1 --2- 172.21.2.112:0/3751835643 >> v2:172.21.2.120:3300/0 conn(0x7f53040e5170 0x7f53041b38e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:10.297 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.294 7f530c452700 1 -- 172.21.2.112:0/3751835643 >> 172.21.2.112:0/3751835643 conn(0x7f53040ce820 msgr2=0x7f53040e4660 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:10.297 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.295 7f530c452700 1 -- 172.21.2.112:0/3751835643 shutdown_connections 2021-02-10T11:34:10.297 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.295 7f530c452700 1 -- 172.21.2.112:0/3751835643 wait complete. 2021-02-10T11:34:10.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.309 7f530c452700 1 Processor -- start 2021-02-10T11:34:10.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.309 7f530c452700 1 -- start start 2021-02-10T11:34:10.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.309 7f530c452700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f53041116d0 0x7f53040b8ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:10.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.309 7f530c452700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f53040b94f0 0x7f53040b98f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:10.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.309 7f530c452700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f53040ba640 0x7f53040baa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:10.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.309 7f530c452700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53041960f0 con 0x7f53041116d0 2021-02-10T11:34:10.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.309 7f530c452700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53040ba0c0 con 0x7f53040b94f0 2021-02-10T11:34:10.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.309 7f530c452700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5304111230 con 0x7f53040ba640 2021-02-10T11:34:10.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.310 7f530a26e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f53041116d0 0x7f53040b8ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:10.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.310 7f530a26e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f53041116d0 0x7f53040b8ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:50246/0 (socket says 172.21.2.112:50246) 2021-02-10T11:34:10.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.310 7f530a26e700 1 -- 172.21.2.112:0/1208147264 learned_addr learned my addr 172.21.2.112:0/1208147264 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:34:10.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.310 7f5309a6d700 1 --2- 172.21.2.112:0/1208147264 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f53040b94f0 0x7f53040b98f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:10.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.310 7f530aa6f700 1 --2- 172.21.2.112:0/1208147264 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f53040ba640 0x7f53040baa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:10.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.310 7f530a26e700 1 -- 172.21.2.112:0/1208147264 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f53040ba640 msgr2=0x7f53040baa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:10.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.310 7f530a26e700 1 --2- 172.21.2.112:0/1208147264 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f53040ba640 0x7f53040baa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:10.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.310 7f530a26e700 1 -- 172.21.2.112:0/1208147264 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f53040b94f0 msgr2=0x7f53040b98f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:10.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.310 7f530a26e700 1 --2- 172.21.2.112:0/1208147264 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f53040b94f0 0x7f53040b98f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:10.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.310 7f530a26e700 1 -- 172.21.2.112:0/1208147264 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53040bb060 con 0x7f53041116d0 2021-02-10T11:34:10.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.310 7f530a26e700 1 --2- 172.21.2.112:0/1208147264 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f53041116d0 0x7f53040b8ed0 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 rx=0x7f5300006be0 tx=0x7f530000b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:10.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.310 7f52f6ffd700 1 -- 172.21.2.112:0/1208147264 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f53000064d0 con 0x7f53041116d0 2021-02-10T11:34:10.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.310 7f52f6ffd700 1 -- 172.21.2.112:0/1208147264 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f530000bd80 con 0x7f53041116d0 2021-02-10T11:34:10.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.310 7f52f6ffd700 1 -- 172.21.2.112:0/1208147264 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5300002220 con 0x7f53041116d0 2021-02-10T11:34:10.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.310 7f530c452700 1 -- 172.21.2.112:0/1208147264 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53040bb060 con 0x7f53041116d0 2021-02-10T11:34:10.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.310 7f530c452700 1 -- 172.21.2.112:0/1208147264 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53040bb060 con 0x7f53041116d0 2021-02-10T11:34:10.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.311 7f52f6ffd700 1 -- 172.21.2.112:0/1208147264 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5300006c90 con 0x7f53041116d0 2021-02-10T11:34:10.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.311 7f52f6ffd700 1 --2- 172.21.2.112:0/1208147264 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f52ec0186c0 0x7f52ec01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:10.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.311 7f52f6ffd700 1 -- 172.21.2.112:0/1208147264 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f5300032400 con 0x7f53041116d0 2021-02-10T11:34:10.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.311 7f5309a6d700 1 --2- 172.21.2.112:0/1208147264 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f52ec0186c0 0x7f52ec01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:10.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.312 7f5309a6d700 1 --2- 172.21.2.112:0/1208147264 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f52ec0186c0 0x7f52ec01abc0 crc :-1 s=READY pgs=653 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:10.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.342 7f530c452700 1 -- 172.21.2.112:0/1208147264 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5304111230 con 0x7f53041116d0 2021-02-10T11:34:10.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.345 7f52f6ffd700 1 -- 172.21.2.112:0/1208147264 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f53000064d0 con 0x7f53041116d0 2021-02-10T11:34:10.461 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.474 7f530c452700 1 -- 172.21.2.112:0/1208147264 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f5304111230 con 0x7f53041116d0 2021-02-10T11:34:10.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.475 7f52f6ffd700 1 -- 172.21.2.112:0/1208147264 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v55) v1 ==== 104+0+10 (secure 0 0 0) 0x7f5300006c90 con 0x7f53041116d0 2021-02-10T11:34:10.462 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:34:10.464 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.477 7f530c452700 1 -- 172.21.2.112:0/1208147264 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f52ec0186c0 msgr2=0x7f52ec01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:10.464 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.477 7f530c452700 1 --2- 172.21.2.112:0/1208147264 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f52ec0186c0 0x7f52ec01abc0 crc :-1 s=READY pgs=653 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:10.464 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.477 7f530c452700 1 -- 172.21.2.112:0/1208147264 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f53041116d0 msgr2=0x7f53040b8ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:10.464 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.477 7f530c452700 1 --2- 172.21.2.112:0/1208147264 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f53041116d0 0x7f53040b8ed0 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 rx=0x7f5300006be0 tx=0x7f530000b990).stop 2021-02-10T11:34:10.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.477 7f530c452700 1 -- 172.21.2.112:0/1208147264 shutdown_connections 2021-02-10T11:34:10.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.477 7f530c452700 1 --2- 172.21.2.112:0/1208147264 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f52ec0186c0 0x7f52ec01abc0 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:10.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.477 7f530c452700 1 --2- 172.21.2.112:0/1208147264 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f53040b94f0 0x7f53040b98f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:10.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.477 7f530c452700 1 --2- 172.21.2.112:0/1208147264 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f53040ba640 0x7f53040baa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:10.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.477 7f530c452700 1 --2- 172.21.2.112:0/1208147264 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f53041116d0 0x7f53040b8ed0 unknown :-1 s=CLOSED pgs=1322 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:10.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.477 7f530c452700 1 -- 172.21.2.112:0/1208147264 >> 172.21.2.112:0/1208147264 conn(0x7f530410fc20 msgr2=0x7f53040a3830 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:10.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.477 7f530c452700 1 -- 172.21.2.112:0/1208147264 shutdown_connections 2021-02-10T11:34:10.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.477 7f530c452700 1 -- 172.21.2.112:0/1208147264 wait complete. 2021-02-10T11:34:10.472 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:34:10.901 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.898 7faf956d7700 1 Processor -- start 2021-02-10T11:34:10.901 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.898 7faf956d7700 1 -- start start 2021-02-10T11:34:10.901 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.898 7faf956d7700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7faf900e5170 0x7faf901b38e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:10.901 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.898 7faf956d7700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7faf901b3f00 0x7faf90077510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:10.901 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.898 7faf956d7700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7faf900e5570 0x7faf90077b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:10.901 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.898 7faf956d7700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf90078b90 con 0x7faf901b3f00 2021-02-10T11:34:10.902 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.898 7faf956d7700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf90078150 con 0x7faf900e5170 2021-02-10T11:34:10.902 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.898 7faf956d7700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf90078370 con 0x7faf900e5570 2021-02-10T11:34:10.902 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.899 7faf8eabe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7faf901b3f00 0x7faf90077510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:10.902 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.899 7faf8eabe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7faf901b3f00 0x7faf90077510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:50252/0 (socket says 172.21.2.112:50252) 2021-02-10T11:34:10.902 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.899 7faf8eabe700 1 -- 172.21.2.112:0/3295805166 learned_addr learned my addr 172.21.2.112:0/3295805166 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:34:10.902 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.899 7faf8eabe700 1 -- 172.21.2.112:0/3295805166 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7faf900e5570 msgr2=0x7faf90077b30 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:34:10.902 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.899 7faf8eabe700 1 --2- 172.21.2.112:0/3295805166 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7faf900e5570 0x7faf90077b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:10.902 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.899 7faf8eabe700 1 -- 172.21.2.112:0/3295805166 >> v2:172.21.2.120:3300/0 conn(0x7faf900e5170 msgr2=0x7faf901b38e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:34:10.902 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.899 7faf8eabe700 1 --2- 172.21.2.112:0/3295805166 >> v2:172.21.2.120:3300/0 conn(0x7faf900e5170 0x7faf901b38e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:10.902 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.899 7faf8eabe700 1 -- 172.21.2.112:0/3295805166 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf90078590 con 0x7faf901b3f00 2021-02-10T11:34:10.902 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.899 7faf8eabe700 1 --2- 172.21.2.112:0/3295805166 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faf901b3f00 0x7faf90077510 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 rx=0x7faf80006700 tx=0x7faf80009b80).ready entity=mon.0 client_cookie=530002e67f7e25dd server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:10.903 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.899 7faf8e2bd700 1 -- 172.21.2.112:0/3295805166 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faf8000c0c0 con 0x7faf901b3f00 2021-02-10T11:34:10.903 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.899 7faf8e2bd700 1 -- 172.21.2.112:0/3295805166 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf8000c340 con 0x7faf901b3f00 2021-02-10T11:34:10.903 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.899 7faf8e2bd700 1 -- 172.21.2.112:0/3295805166 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faf8000c860 con 0x7faf901b3f00 2021-02-10T11:34:10.903 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.899 7faf956d7700 1 -- 172.21.2.112:0/3295805166 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faf901b3f00 msgr2=0x7faf90077510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:10.903 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.899 7faf956d7700 1 --2- 172.21.2.112:0/3295805166 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faf901b3f00 0x7faf90077510 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 rx=0x7faf80006700 tx=0x7faf80009b80).stop 2021-02-10T11:34:10.903 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.900 7faf956d7700 1 -- 172.21.2.112:0/3295805166 shutdown_connections 2021-02-10T11:34:10.903 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.900 7faf956d7700 1 --2- 172.21.2.112:0/3295805166 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7faf900e5570 0x7faf90077b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:10.903 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.900 7faf956d7700 1 --2- 172.21.2.112:0/3295805166 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faf901b3f00 0x7faf90077510 unknown :-1 s=CLOSED pgs=1323 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:10.903 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.900 7faf956d7700 1 --2- 172.21.2.112:0/3295805166 >> v2:172.21.2.120:3300/0 conn(0x7faf900e5170 0x7faf901b38e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:10.903 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.900 7faf956d7700 1 -- 172.21.2.112:0/3295805166 >> 172.21.2.112:0/3295805166 conn(0x7faf900ce820 msgr2=0x7faf900e4660 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:10.903 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.900 7faf956d7700 1 -- 172.21.2.112:0/3295805166 shutdown_connections 2021-02-10T11:34:10.903 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.900 7faf956d7700 1 -- 172.21.2.112:0/3295805166 wait complete. 2021-02-10T11:34:10.905 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.915 7faf956d7700 1 Processor -- start 2021-02-10T11:34:10.905 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.915 7faf956d7700 1 -- start start 2021-02-10T11:34:10.905 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.915 7faf956d7700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faf901117f0 0x7faf900aeb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:10.905 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.915 7faf956d7700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7faf900af160 0x7faf900af560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:10.905 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.915 7faf956d7700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7faf900bb520 0x7faf900bb920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:10.905 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.915 7faf956d7700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf9010da90 con 0x7faf901117f0 2021-02-10T11:34:10.905 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.915 7faf956d7700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf900afb80 con 0x7faf900af160 2021-02-10T11:34:10.906 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.915 7faf956d7700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf90111bf0 con 0x7faf900bb520 2021-02-10T11:34:10.906 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.915 7faf8f2bf700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faf901117f0 0x7faf900aeb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:10.906 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.915 7faf8f2bf700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faf901117f0 0x7faf900aeb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:50256/0 (socket says 172.21.2.112:50256) 2021-02-10T11:34:10.906 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.915 7faf8f2bf700 1 -- 172.21.2.112:0/125086024 learned_addr learned my addr 172.21.2.112:0/125086024 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:34:10.906 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.916 7faf8f2bf700 1 -- 172.21.2.112:0/125086024 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7faf900bb520 msgr2=0x7faf900bb920 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:34:10.906 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.916 7faf8f2bf700 1 --2- 172.21.2.112:0/125086024 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7faf900bb520 0x7faf900bb920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:10.906 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.916 7faf8f2bf700 1 -- 172.21.2.112:0/125086024 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7faf900af160 msgr2=0x7faf900af560 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:34:10.906 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.916 7faf8f2bf700 1 --2- 172.21.2.112:0/125086024 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7faf900af160 0x7faf900af560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:10.906 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.916 7faf8f2bf700 1 -- 172.21.2.112:0/125086024 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf900bc9d0 con 0x7faf901117f0 2021-02-10T11:34:10.907 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.916 7faf8f2bf700 1 --2- 172.21.2.112:0/125086024 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faf901117f0 0x7faf900aeb40 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 rx=0x7faf7800b5c0 tx=0x7faf7800b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:10.907 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.916 7faf87fff700 1 -- 172.21.2.112:0/125086024 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faf7800bd50 con 0x7faf901117f0 2021-02-10T11:34:10.907 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.916 7faf87fff700 1 -- 172.21.2.112:0/125086024 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf78006030 con 0x7faf901117f0 2021-02-10T11:34:10.907 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.916 7faf87fff700 1 -- 172.21.2.112:0/125086024 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faf780054d0 con 0x7faf901117f0 2021-02-10T11:34:10.907 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.916 7faf956d7700 1 -- 172.21.2.112:0/125086024 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf900bc9d0 con 0x7faf901117f0 2021-02-10T11:34:10.907 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.916 7faf956d7700 1 -- 172.21.2.112:0/125086024 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf900bc230 con 0x7faf901117f0 2021-02-10T11:34:10.907 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.917 7faf87fff700 1 -- 172.21.2.112:0/125086024 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7faf78006210 con 0x7faf901117f0 2021-02-10T11:34:10.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.917 7faf87fff700 1 --2- 172.21.2.112:0/125086024 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7faf740186c0 0x7faf7401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:10.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.917 7faf87fff700 1 -- 172.21.2.112:0/125086024 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5573+0+0 (secure 0 0 0) 0x7faf780325e0 con 0x7faf901117f0 2021-02-10T11:34:10.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.918 7faf8eabe700 1 --2- 172.21.2.112:0/125086024 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7faf740186c0 0x7faf7401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:10.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.918 7faf8eabe700 1 --2- 172.21.2.112:0/125086024 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7faf740186c0 0x7faf7401abc0 crc :-1 s=READY pgs=654 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:10.935 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.948 7faf956d7700 1 -- 172.21.2.112:0/125086024 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf90105c20 con 0x7faf901117f0 2021-02-10T11:34:10.938 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:10.951 7faf87fff700 1 -- 172.21.2.112:0/125086024 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7faf78030d90 con 0x7faf901117f0 2021-02-10T11:34:11.068 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:11.081 7faf956d7700 1 -- 172.21.2.112:0/125086024 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7faf90105c20 con 0x7faf901117f0 2021-02-10T11:34:11.068 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:11.081 7faf87fff700 1 -- 172.21.2.112:0/125086024 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v55) v1 ==== 100+0+10 (secure 0 0 0) 0x7faf7800bd50 con 0x7faf901117f0 2021-02-10T11:34:11.069 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:34:11.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:11.083 7faf956d7700 1 -- 172.21.2.112:0/125086024 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7faf740186c0 msgr2=0x7faf7401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:11.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:11.083 7faf956d7700 1 --2- 172.21.2.112:0/125086024 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7faf740186c0 0x7faf7401abc0 crc :-1 s=READY pgs=654 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:11.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:11.083 7faf956d7700 1 -- 172.21.2.112:0/125086024 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faf901117f0 msgr2=0x7faf900aeb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:11.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:11.083 7faf956d7700 1 --2- 172.21.2.112:0/125086024 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faf901117f0 0x7faf900aeb40 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 rx=0x7faf7800b5c0 tx=0x7faf7800b960).stop 2021-02-10T11:34:11.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:11.083 7faf956d7700 1 -- 172.21.2.112:0/125086024 shutdown_connections 2021-02-10T11:34:11.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:11.083 7faf956d7700 1 --2- 172.21.2.112:0/125086024 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7faf740186c0 0x7faf7401abc0 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:11.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:11.083 7faf956d7700 1 --2- 172.21.2.112:0/125086024 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7faf900af160 0x7faf900af560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:11.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:11.083 7faf956d7700 1 --2- 172.21.2.112:0/125086024 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7faf900bb520 0x7faf900bb920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:11.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:11.083 7faf956d7700 1 --2- 172.21.2.112:0/125086024 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faf901117f0 0x7faf900aeb40 unknown :-1 s=CLOSED pgs=1324 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:11.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:11.083 7faf956d7700 1 -- 172.21.2.112:0/125086024 >> 172.21.2.112:0/125086024 conn(0x7faf90105e50 msgr2=0x7faf900b22a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:11.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:11.083 7faf956d7700 1 -- 172.21.2.112:0/125086024 shutdown_connections 2021-02-10T11:34:11.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:11.083 7faf956d7700 1 -- 172.21.2.112:0/125086024 wait complete. 2021-02-10T11:34:11.078 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:34:11.079 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:34:11.505 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:11.502 7f758444c700 1 Processor -- start 2021-02-10T11:34:11.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:11.502 7f758444c700 1 -- start start 2021-02-10T11:34:11.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:11.503 7f758444c700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f757c0e53e0 0x7f757c1b0550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:11.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:11.503 7f758444c700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f757c1b0b70 0x7f757c081910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:11.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:11.503 7f758444c700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f757c082d50 con 0x7f757c081f30 2021-02-10T11:34:11.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:11.503 7f758444c700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f757c082b20 con 0x7f757c0e53e0 2021-02-10T11:34:11.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:11.503 7f758444c700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f757c0e5fe0 con 0x7f757c1b0b70 2021-02-10T11:34:11.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:11.503 7f7582268700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f757c0e53e0 0x7f757c1b0550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:11.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:11.503 7f7582268700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f757c0e53e0 0x7f757c1b0550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:50262/0 (socket says 172.21.2.112:50262) 2021-02-10T11:34:11.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:11.503 7f7582268700 1 -- 172.21.2.112:0/3864552905 learned_addr learned my addr 172.21.2.112:0/3864552905 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:34:11.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:11.503 7f7582268700 1 -- 172.21.2.112:0/3864552905 >> v2:172.21.2.120:3300/0 conn(0x7f757c1b0b70 msgr2=0x7f757c081910 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:34:11.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:11.503 7f7582268700 1 --2- 172.21.2.112:0/3864552905 >> v2:172.21.2.120:3300/0 conn(0x7f757c1b0b70 0x7f757c081910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:11.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:11.503 7f7582268700 1 -- 172.21.2.112:0/3864552905 >> v1:172.21.2.120:6789/0 conn(0x7f757c081f30 legacy=0x7f757c082330 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:34:11.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:11.503 7f7582268700 1 -- 172.21.2.112:0/3864552905 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f757c0e57e0 con 0x7f757c0e53e0 2021-02-10T11:34:11.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:11.503 7f7582268700 1 --2- 172.21.2.112:0/3864552905 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f757c0e53e0 0x7f757c1b0550 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 rx=0x7f7578006700 tx=0x7f7578009b00).ready entity=mon.0 client_cookie=7259191425983951 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:11.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:11.504 7f7581266700 1 -- 172.21.2.112:0/3864552905 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f757800c0c0 con 0x7f757c0e53e0 2021-02-10T11:34:11.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:11.504 7f7581266700 1 -- 172.21.2.112:0/3864552905 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f757800c400 con 0x7f757c0e53e0 2021-02-10T11:34:11.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:11.504 7f7581266700 1 -- 172.21.2.112:0/3864552905 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f757800c920 con 0x7f757c0e53e0 2021-02-10T11:34:11.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:11.504 7f758444c700 1 -- 172.21.2.112:0/3864552905 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f757c0e53e0 msgr2=0x7f757c1b0550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:11.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:11.504 7f758444c700 1 --2- 172.21.2.112:0/3864552905 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f757c0e53e0 0x7f757c1b0550 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 rx=0x7f7578006700 tx=0x7f7578009b00).stop 2021-02-10T11:34:11.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:11.504 7f758444c700 1 -- 172.21.2.112:0/3864552905 shutdown_connections 2021-02-10T11:34:11.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:11.504 7f758444c700 1 --2- 172.21.2.112:0/3864552905 >> v2:172.21.2.120:3300/0 conn(0x7f757c1b0b70 0x7f757c081910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:11.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:11.504 7f758444c700 1 --2- 172.21.2.112:0/3864552905 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f757c0e53e0 0x7f757c1b0550 unknown :-1 s=CLOSED pgs=1325 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:11.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:11.504 7f758444c700 1 -- 172.21.2.112:0/3864552905 >> 172.21.2.112:0/3864552905 conn(0x7f757c0ce500 msgr2=0x7f757c0e4610 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:11.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:11.505 7f758444c700 1 -- 172.21.2.112:0/3864552905 shutdown_connections 2021-02-10T11:34:11.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:11.505 7f758444c700 1 -- 172.21.2.112:0/3864552905 wait complete. 2021-02-10T11:34:11.509 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:11.519 7f758444c700 1 Processor -- start 2021-02-10T11:34:11.509 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:11.519 7f758444c700 1 -- start start 2021-02-10T11:34:11.509 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:11.520 7f758444c700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f757c19e3b0 0x7f757c1b19f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:11.509 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:11.520 7f758444c700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f757c1b2010 0x7f757c1b2410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:11.509 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:11.520 7f758444c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f757c199d80 0x7f757c19a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:11.509 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:11.520 7f758444c700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f757c19e7b0 con 0x7f757c199d80 2021-02-10T11:34:11.509 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:11.520 7f758444c700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f757c1af520 con 0x7f757c1b2010 2021-02-10T11:34:11.509 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:11.520 7f758444c700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f757c19a7a0 con 0x7f757c19e3b0 2021-02-10T11:34:11.509 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:11.520 7f7582a69700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f757c199d80 0x7f757c19a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:11.509 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:11.520 7f7582a69700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f757c199d80 0x7f757c19a180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:50266/0 (socket says 172.21.2.112:50266) 2021-02-10T11:34:11.509 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:11.520 7f7582a69700 1 -- 172.21.2.112:0/1889668641 learned_addr learned my addr 172.21.2.112:0/1889668641 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:34:11.509 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:11.520 7f7582a69700 1 -- 172.21.2.112:0/1889668641 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f757c19e3b0 msgr2=0x7f757c1b19f0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:34:11.510 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:11.520 7f7582a69700 1 --2- 172.21.2.112:0/1889668641 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f757c19e3b0 0x7f757c1b19f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:11.510 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:11.520 7f7582a69700 1 -- 172.21.2.112:0/1889668641 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f757c1b2010 msgr2=0x7f757c1b2410 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:34:11.510 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:11.520 7f7582a69700 1 --2- 172.21.2.112:0/1889668641 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f757c1b2010 0x7f757c1b2410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:11.510 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:11.520 7f7582a69700 1 -- 172.21.2.112:0/1889668641 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f757c19a9c0 con 0x7f757c199d80 2021-02-10T11:34:11.510 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:11.520 7f7582a69700 1 --2- 172.21.2.112:0/1889668641 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f757c199d80 0x7f757c19a180 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 rx=0x7f7574009940 tx=0x7f757400bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:11.510 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:11.520 7f756effd700 1 -- 172.21.2.112:0/1889668641 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7574000f90 con 0x7f757c199d80 2021-02-10T11:34:11.510 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:11.520 7f756effd700 1 -- 172.21.2.112:0/1889668641 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f757400c2f0 con 0x7f757c199d80 2021-02-10T11:34:11.510 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:11.520 7f758444c700 1 -- 172.21.2.112:0/1889668641 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f757c19a9c0 con 0x7f757c199d80 2021-02-10T11:34:11.510 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:11.521 7f756effd700 1 -- 172.21.2.112:0/1889668641 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f757400c7d0 con 0x7f757c199d80 2021-02-10T11:34:11.510 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:11.521 7f758444c700 1 -- 172.21.2.112:0/1889668641 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f757c19af50 con 0x7f757c199d80 2021-02-10T11:34:11.511 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:11.521 7f756effd700 1 -- 172.21.2.112:0/1889668641 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7574001220 con 0x7f757c199d80 2021-02-10T11:34:11.511 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:11.521 7f756effd700 1 --2- 172.21.2.112:0/1889668641 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f75640186c0 0x7f756401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:11.511 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:11.521 7f756effd700 1 -- 172.21.2.112:0/1889668641 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f7574032750 con 0x7f757c199d80 2021-02-10T11:34:11.511 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:11.522 7f7582268700 1 --2- 172.21.2.112:0/1889668641 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f75640186c0 0x7f756401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:11.511 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:11.522 7f7582268700 1 --2- 172.21.2.112:0/1889668641 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f75640186c0 0x7f756401abc0 crc :-1 s=READY pgs=655 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:11.540 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:11.553 7f758444c700 1 -- 172.21.2.112:0/1889668641 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f757c0ed660 con 0x7f757c199d80 2021-02-10T11:34:11.542 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:11.555 7f756effd700 1 -- 172.21.2.112:0/1889668641 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7574000f40 con 0x7f757c199d80 2021-02-10T11:34:11.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:11.684 7f758444c700 1 -- 172.21.2.112:0/1889668641 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f757c082d50 con 0x7f757c199d80 2021-02-10T11:34:11.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:11.685 7f756effd700 1 -- 172.21.2.112:0/1889668641 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+2101 (secure 0 0 0) 0x7f7574001220 con 0x7f757c199d80 2021-02-10T11:34:11.672 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 40 2021-02-10T11:34:11.673 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:34:11.673 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":40,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7881,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/1476007789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":1476007789},{"type":"v1","addr":"172.21.2.112:6833","nonce":1476007789}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":35}],"filesystems":[{"mdsmap":{"epoch":39,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:33:23.550791","modified":"2021-02-10 11:33:25.562811","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6595":{"gid":6595,"name":"b","rank":0,"incarnation":36,"state":"up:active","state_seq":3,"addr":"172.21.2.120:6835/4241229406","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":4241229406},{"type":"v1","addr":"172.21.2.120:6835","nonce":4241229406}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2021-02-10T11:34:11.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:11.687 7f758444c700 1 -- 172.21.2.112:0/1889668641 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f75640186c0 msgr2=0x7f756401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:11.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:11.687 7f758444c700 1 --2- 172.21.2.112:0/1889668641 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f75640186c0 0x7f756401abc0 crc :-1 s=READY pgs=655 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:11.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:11.687 7f758444c700 1 -- 172.21.2.112:0/1889668641 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f757c199d80 msgr2=0x7f757c19a180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:11.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:11.687 7f758444c700 1 --2- 172.21.2.112:0/1889668641 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f757c199d80 0x7f757c19a180 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 rx=0x7f7574009940 tx=0x7f757400bcf0).stop 2021-02-10T11:34:11.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:11.687 7f758444c700 1 -- 172.21.2.112:0/1889668641 shutdown_connections 2021-02-10T11:34:11.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:11.687 7f758444c700 1 --2- 172.21.2.112:0/1889668641 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f75640186c0 0x7f756401abc0 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:11.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:11.687 7f758444c700 1 --2- 172.21.2.112:0/1889668641 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f757c1b2010 0x7f757c1b2410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:11.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:11.687 7f758444c700 1 --2- 172.21.2.112:0/1889668641 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f757c199d80 0x7f757c19a180 unknown :-1 s=CLOSED pgs=1326 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:11.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:11.687 7f758444c700 1 --2- 172.21.2.112:0/1889668641 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f757c19e3b0 0x7f757c1b19f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:11.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:11.687 7f758444c700 1 -- 172.21.2.112:0/1889668641 >> 172.21.2.112:0/1889668641 conn(0x7f757c1b2f00 msgr2=0x7f757c1a1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:11.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:11.687 7f758444c700 1 -- 172.21.2.112:0/1889668641 shutdown_connections 2021-02-10T11:34:11.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:11.687 7f758444c700 1 -- 172.21.2.112:0/1889668641 wait complete. 2021-02-10T11:34:11.884 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 39, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:33:23.550791', 'modified': '2021-02-10 11:33:25.562811', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6595}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6595': {'gid': 6595, 'name': 'b', 'rank': 0, 'incarnation': 36, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.120:6835/4241229406', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.120:6834', 'nonce': 4241229406}, {'type': 'v1', 'addr': '172.21.2.120:6835', 'nonce': 4241229406}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [11], 'metadata_pool': 10, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:34:11.884 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:34:11.884 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:34:12.281 INFO:teuthology.orchestra.run.gibba020.stdout:{ 2021-02-10T11:34:12.281 INFO:teuthology.orchestra.run.gibba020.stdout: "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", 2021-02-10T11:34:12.281 INFO:teuthology.orchestra.run.gibba020.stdout: "whoami": 0, 2021-02-10T11:34:12.282 INFO:teuthology.orchestra.run.gibba020.stdout: "id": 6595, 2021-02-10T11:34:12.282 INFO:teuthology.orchestra.run.gibba020.stdout: "want_state": "up:active", 2021-02-10T11:34:12.282 INFO:teuthology.orchestra.run.gibba020.stdout: "state": "up:active", 2021-02-10T11:34:12.282 INFO:teuthology.orchestra.run.gibba020.stdout: "rank_uptime": 48.741000395, 2021-02-10T11:34:12.282 INFO:teuthology.orchestra.run.gibba020.stdout: "mdsmap_epoch": 39, 2021-02-10T11:34:12.282 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch": 55, 2021-02-10T11:34:12.282 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch_barrier": 55, 2021-02-10T11:34:12.282 INFO:teuthology.orchestra.run.gibba020.stdout: "uptime": 54.370755504000002 2021-02-10T11:34:12.282 INFO:teuthology.orchestra.run.gibba020.stdout:} 2021-02-10T11:34:12.285 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", "whoami": 0, "id": 6595, "want_state": "up:active", "state": "up:active", "rank_uptime": 48.741000395, "mdsmap_epoch": 39, "osdmap_epoch": 55, "osdmap_epoch_barrier": 55, "uptime": 54.370755504000002 } 2021-02-10T11:34:12.285 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:34:12.286 INFO:tasks.cephfs.fuse_mount:Client client.alice config is {} 2021-02-10T11:34:12.286 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.alice at ubuntu@gibba012.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2021-02-10T11:34:12.286 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:12.286 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:34:12.622 DEBUG:teuthology.orchestra.run.gibba012:> sudo modprobe fuse 2021-02-10T11:34:12.966 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:34:12.966 DEBUG:teuthology.orchestra.run.gibba012:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:34:13.310 INFO:teuthology.orchestra.run.gibba012.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:34:13.311 INFO:teuthology.orchestra.run.gibba012.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:34:13.311 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:34:13.312 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:13.312 DEBUG:teuthology.orchestra.run.gibba012:> ls /sys/fs/fuse/connections 2021-02-10T11:34:13.657 INFO:teuthology.orchestra.run.gibba012.stdout:42 2021-02-10T11:34:13.658 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [42] 2021-02-10T11:34:13.658 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --client_mountpoint=/volumes/group_0000000000544641/subvolume_0000000000670007 --name client.alice /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:34:13.662 DEBUG:teuthology.orchestra.run.gibba012:> sudo modprobe fuse 2021-02-10T11:34:14.004 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:34:14.004 DEBUG:teuthology.orchestra.run.gibba012:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:34:14.071 INFO:tasks.cephfs.fuse_mount.ceph-fuse.alice.gibba012.stderr:ceph-fuse[131944]: starting ceph client 2021-02-10T11:34:14.072 INFO:tasks.cephfs.fuse_mount.ceph-fuse.alice.gibba012.stderr:2021-02-10 11:34:14.084 7f14022daf80 -1 init, newargv = 0x55fabb15ee40 newargc=9 2021-02-10T11:34:14.077 INFO:tasks.cephfs.fuse_mount.ceph-fuse.alice.gibba012.stderr:ceph-fuse[131944]: starting fuse 2021-02-10T11:34:14.346 INFO:teuthology.orchestra.run.gibba012.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:34:14.346 INFO:teuthology.orchestra.run.gibba012.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:34:14.346 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:34:14.347 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:14.347 DEBUG:teuthology.orchestra.run.gibba012:> ls /sys/fs/fuse/connections 2021-02-10T11:34:14.681 INFO:teuthology.orchestra.run.gibba012.stdout:42 2021-02-10T11:34:14.681 INFO:teuthology.orchestra.run.gibba012.stdout:43 2021-02-10T11:34:14.681 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42, 43] 2021-02-10T11:34:14.682 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:14.682 DEBUG:teuthology.orchestra.run.gibba012:> sudo python3 -c ' 2021-02-10T11:34:14.682 DEBUG:teuthology.orchestra.run.gibba012:> import glob 2021-02-10T11:34:14.682 DEBUG:teuthology.orchestra.run.gibba012:> import re 2021-02-10T11:34:14.682 DEBUG:teuthology.orchestra.run.gibba012:> import os 2021-02-10T11:34:14.682 DEBUG:teuthology.orchestra.run.gibba012:> import subprocess 2021-02-10T11:34:14.682 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:34:14.682 DEBUG:teuthology.orchestra.run.gibba012:> def find_socket(client_name): 2021-02-10T11:34:14.682 DEBUG:teuthology.orchestra.run.gibba012:> asok_path = "/var/run/ceph/ceph-client.alice.*.asok" 2021-02-10T11:34:14.682 DEBUG:teuthology.orchestra.run.gibba012:> files = glob.glob(asok_path) 2021-02-10T11:34:14.682 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:34:14.683 DEBUG:teuthology.orchestra.run.gibba012:> # Given a non-glob path, it better be there 2021-02-10T11:34:14.683 DEBUG:teuthology.orchestra.run.gibba012:> if "*" not in asok_path: 2021-02-10T11:34:14.683 DEBUG:teuthology.orchestra.run.gibba012:> assert(len(files) == 1) 2021-02-10T11:34:14.683 DEBUG:teuthology.orchestra.run.gibba012:> return files[0] 2021-02-10T11:34:14.683 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:34:14.683 DEBUG:teuthology.orchestra.run.gibba012:> for f in files: 2021-02-10T11:34:14.683 DEBUG:teuthology.orchestra.run.gibba012:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:34:14.683 DEBUG:teuthology.orchestra.run.gibba012:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:34:14.683 DEBUG:teuthology.orchestra.run.gibba012:> return f 2021-02-10T11:34:14.683 DEBUG:teuthology.orchestra.run.gibba012:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:34:14.683 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:34:14.683 DEBUG:teuthology.orchestra.run.gibba012:> print(find_socket("client.alice")) 2021-02-10T11:34:14.683 DEBUG:teuthology.orchestra.run.gibba012:> ' 2021-02-10T11:34:15.044 INFO:teuthology.orchestra.run.gibba012.stdout:/var/run/ceph/ceph-client.alice.131944.asok 2021-02-10T11:34:15.046 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.alice.131944.asok 2021-02-10T11:34:15.047 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:15.047 DEBUG:teuthology.orchestra.run.gibba012:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.alice.131944.asok status 2021-02-10T11:34:15.437 INFO:teuthology.orchestra.run.gibba012.stdout:{ 2021-02-10T11:34:15.437 INFO:teuthology.orchestra.run.gibba012.stdout: "metadata": { 2021-02-10T11:34:15.437 INFO:teuthology.orchestra.run.gibba012.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:34:15.437 INFO:teuthology.orchestra.run.gibba012.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:34:15.438 INFO:teuthology.orchestra.run.gibba012.stdout: "entity_id": "alice", 2021-02-10T11:34:15.438 INFO:teuthology.orchestra.run.gibba012.stdout: "hostname": "gibba012", 2021-02-10T11:34:15.438 INFO:teuthology.orchestra.run.gibba012.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2021-02-10T11:34:15.438 INFO:teuthology.orchestra.run.gibba012.stdout: "pid": "131944", 2021-02-10T11:34:15.438 INFO:teuthology.orchestra.run.gibba012.stdout: "root": "/volumes/group_0000000000544641/subvolume_0000000000670007" 2021-02-10T11:34:15.438 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:15.438 INFO:teuthology.orchestra.run.gibba012.stdout: "dentry_count": 0, 2021-02-10T11:34:15.438 INFO:teuthology.orchestra.run.gibba012.stdout: "dentry_pinned_count": 0, 2021-02-10T11:34:15.438 INFO:teuthology.orchestra.run.gibba012.stdout: "id": 8199, 2021-02-10T11:34:15.438 INFO:teuthology.orchestra.run.gibba012.stdout: "inst": { 2021-02-10T11:34:15.438 INFO:teuthology.orchestra.run.gibba012.stdout: "name": { 2021-02-10T11:34:15.438 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "client", 2021-02-10T11:34:15.439 INFO:teuthology.orchestra.run.gibba012.stdout: "num": 8199 2021-02-10T11:34:15.439 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:15.439 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": { 2021-02-10T11:34:15.439 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:34:15.439 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:0", 2021-02-10T11:34:15.439 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 4087411699 2021-02-10T11:34:15.439 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:34:15.439 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:15.439 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": { 2021-02-10T11:34:15.439 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:34:15.439 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:0", 2021-02-10T11:34:15.439 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 4087411699 2021-02-10T11:34:15.440 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:15.440 INFO:teuthology.orchestra.run.gibba012.stdout: "inst_str": "client.8199 172.21.2.112:0/4087411699", 2021-02-10T11:34:15.440 INFO:teuthology.orchestra.run.gibba012.stdout: "addr_str": "172.21.2.112:0/4087411699", 2021-02-10T11:34:15.440 INFO:teuthology.orchestra.run.gibba012.stdout: "inode_count": 1, 2021-02-10T11:34:15.440 INFO:teuthology.orchestra.run.gibba012.stdout: "mds_epoch": 39, 2021-02-10T11:34:15.440 INFO:teuthology.orchestra.run.gibba012.stdout: "osd_epoch": 55, 2021-02-10T11:34:15.440 INFO:teuthology.orchestra.run.gibba012.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:34:15.440 INFO:teuthology.orchestra.run.gibba012.stdout: "blacklisted": false 2021-02-10T11:34:15.440 INFO:teuthology.orchestra.run.gibba012.stdout:} 2021-02-10T11:34:15.442 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.1 && sudo ls /home/ubuntu/cephtest/mnt.1 2021-02-10T11:34:15.788 INFO:teuthology.orchestra.run.gibba012.stdout:data.bin 2021-02-10T11:34:15.789 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:15.789 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:34:15.789 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:34:16.124 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:34:16.124 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:34:16.125 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.1 && sudo dd if=/dev/urandom of=rogue.bin bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:34:16.470 INFO:teuthology.orchestra.run.gibba012.stderr:dd: failed to open ‘rogue.bin’: Permission denied 2021-02-10T11:34:16.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:34:16.472 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:16.472 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:34:16.472 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:34:16.807 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:34:16.807 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:34:16.807 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba012.front.sepia.ceph.com... 2021-02-10T11:34:16.807 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:34:16.807 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:34:17.030 DEBUG:teuthology.orchestra.run.gibba012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:34:17.034 DEBUG:teuthology.orchestra.run.gibba020:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:34:17.158 INFO:tasks.cephfs.fuse_mount.ceph-fuse.alice.gibba012.stderr:ceph-fuse[131944]: fuse finished with error 0 and tester_r 0 2021-02-10T11:34:17.159 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:17.159 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:34:17.159 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:34:17.494 INFO:teuthology.orchestra.run.gibba012.stdout:ext2/ext3 2021-02-10T11:34:17.494 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:34:17.494 INFO:teuthology.orchestra.run:waiting for 900 2021-02-10T11:34:17.494 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:34:17.495 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:34:17.829 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume deauthorize cephfs subvolume_0000000000670007 alice --group_name group_0000000000544641 2021-02-10T11:34:18.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.256 7fba22d3e700 1 Processor -- start 2021-02-10T11:34:18.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.257 7fba22d3e700 1 -- start start 2021-02-10T11:34:18.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.257 7fba22d3e700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fba1c0e55a0 0x7fba1c078520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:18.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.257 7fba22d3e700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fba1c079110 0x7fba1c079510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:18.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.257 7fba22d3e700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba1c078b90 con 0x7fba1c0e51a0 2021-02-10T11:34:18.262 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.257 7fba22d3e700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba1c1b4100 con 0x7fba1c079110 2021-02-10T11:34:18.262 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.257 7fba22d3e700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba1c0d1930 con 0x7fba1c0e55a0 2021-02-10T11:34:18.262 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.257 7fba20b5a700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fba1c0e51a0 0x7fba1c1b3910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:49748/0 (socket says 172.21.2.112:49748) 2021-02-10T11:34:18.262 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.257 7fba20b5a700 1 -- 172.21.2.112:0/3500527659 learned_addr learned my addr 172.21.2.112:0/3500527659 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:34:18.262 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.258 7fba1b7fe700 1 -- 172.21.2.112:0/3500527659 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 894401065 0 0) 0x7fba0c0010b0 con 0x7fba1c0e51a0 2021-02-10T11:34:18.262 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.258 7fba1b7fe700 1 -- 172.21.2.112:0/3500527659 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba08002fe0 con 0x7fba1c0e51a0 2021-02-10T11:34:18.262 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.258 7fba2135b700 1 --2- 172.21.2.112:0/3500527659 >> v2:172.21.2.120:3300/0 conn(0x7fba1c079110 0x7fba1c079510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:18.262 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.258 7fba1bfff700 1 --2- 172.21.2.112:0/3500527659 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fba1c0e55a0 0x7fba1c078520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:18.262 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.258 7fba1b7fe700 1 -- 172.21.2.112:0/3500527659 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4185845680 0 0) 0x7fba0c0010b0 con 0x7fba1c0e51a0 2021-02-10T11:34:18.262 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.258 7fba1b7fe700 1 -- 172.21.2.112:0/3500527659 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fba1c0e55a0 msgr2=0x7fba1c078520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:18.262 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.258 7fba1b7fe700 1 --2- 172.21.2.112:0/3500527659 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fba1c0e55a0 0x7fba1c078520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:18.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.258 7fba1b7fe700 1 -- 172.21.2.112:0/3500527659 >> v2:172.21.2.120:3300/0 conn(0x7fba1c079110 msgr2=0x7fba1c079510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:18.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.258 7fba1b7fe700 1 --2- 172.21.2.112:0/3500527659 >> v2:172.21.2.120:3300/0 conn(0x7fba1c079110 0x7fba1c079510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:18.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.258 7fba1b7fe700 1 -- 172.21.2.112:0/3500527659 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba1c0d1b50 con 0x7fba1c0e51a0 2021-02-10T11:34:18.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.258 7fba1b7fe700 1 -- 172.21.2.112:0/3500527659 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7fba0c0015b0 con 0x7fba1c0e51a0 2021-02-10T11:34:18.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.258 7fba1b7fe700 1 -- 172.21.2.112:0/3500527659 >> v1:172.21.2.112:6789/0 conn(0x7fba1c0e51a0 legacy=0x7fba1c1b3910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:18.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.258 7fba1b7fe700 1 --2- 172.21.2.112:0/3500527659 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fba08003ac0 0x7fba08005f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:18.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.258 7fba1b7fe700 1 --2- 172.21.2.112:0/3500527659 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fba0804a790 0x7fba0804cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:18.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.258 7fba1b7fe700 1 --2- 172.21.2.112:0/3500527659 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fba0804d590 0x7fba0804fa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:18.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.258 7fba1b7fe700 1 -- 172.21.2.112:0/3500527659 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba080502d0 con 0x7fba0804a790 2021-02-10T11:34:18.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.258 7fba1b7fe700 1 -- 172.21.2.112:0/3500527659 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba080504f0 con 0x7fba08003ac0 2021-02-10T11:34:18.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.258 7fba1b7fe700 1 -- 172.21.2.112:0/3500527659 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba08050710 con 0x7fba0804d590 2021-02-10T11:34:18.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.258 7fba1bfff700 1 --2- 172.21.2.112:0/3500527659 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fba0804a790 0x7fba0804cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:18.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.259 7fba20b5a700 1 --2- 172.21.2.112:0/3500527659 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fba08003ac0 0x7fba08005f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:18.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.259 7fba2135b700 1 --2- 172.21.2.112:0/3500527659 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fba0804d590 0x7fba0804fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:18.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.259 7fba1bfff700 1 -- 172.21.2.112:0/3500527659 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fba0804d590 msgr2=0x7fba0804fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:18.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.259 7fba1bfff700 1 --2- 172.21.2.112:0/3500527659 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fba0804d590 0x7fba0804fa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:18.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.259 7fba1bfff700 1 -- 172.21.2.112:0/3500527659 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fba08003ac0 msgr2=0x7fba08005f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:18.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.259 7fba1bfff700 1 --2- 172.21.2.112:0/3500527659 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fba08003ac0 0x7fba08005f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:18.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.259 7fba1bfff700 1 -- 172.21.2.112:0/3500527659 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba080509b0 con 0x7fba0804a790 2021-02-10T11:34:18.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.259 7fba20b5a700 1 --2- 172.21.2.112:0/3500527659 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fba08003ac0 0x7fba08005f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:34:18.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.259 7fba2135b700 1 --2- 172.21.2.112:0/3500527659 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fba0804d590 0x7fba0804fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:34:18.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.259 7fba1bfff700 1 --2- 172.21.2.112:0/3500527659 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fba0804a790 0x7fba0804cc90 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 rx=0x7fba04006ac0 tx=0x7fba04009ab0).ready entity=mon.0 client_cookie=9fefd1be73a98c2a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:18.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.259 7fba1b7fe700 1 -- 172.21.2.112:0/3500527659 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fba0400c0c0 con 0x7fba0804a790 2021-02-10T11:34:18.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.259 7fba1b7fe700 1 -- 172.21.2.112:0/3500527659 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba0400c3c0 con 0x7fba0804a790 2021-02-10T11:34:18.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.259 7fba1b7fe700 1 -- 172.21.2.112:0/3500527659 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fba0400c8a0 con 0x7fba0804a790 2021-02-10T11:34:18.265 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.259 7fba22d3e700 1 -- 172.21.2.112:0/3500527659 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fba0804a790 msgr2=0x7fba0804cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:18.265 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.259 7fba22d3e700 1 --2- 172.21.2.112:0/3500527659 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fba0804a790 0x7fba0804cc90 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 rx=0x7fba04006ac0 tx=0x7fba04009ab0).stop 2021-02-10T11:34:18.265 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.259 7fba2135b700 1 -- 172.21.2.112:0/3500527659 reap_dead start 2021-02-10T11:34:18.265 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.260 7fba22d3e700 1 -- 172.21.2.112:0/3500527659 shutdown_connections 2021-02-10T11:34:18.265 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.260 7fba22d3e700 1 -- 172.21.2.112:0/3500527659 >> 172.21.2.112:0/3500527659 conn(0x7fba1c0ce850 msgr2=0x7fba1c0e4690 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:18.265 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.260 7fba22d3e700 1 -- 172.21.2.112:0/3500527659 shutdown_connections 2021-02-10T11:34:18.265 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.260 7fba22d3e700 1 -- 172.21.2.112:0/3500527659 wait complete. 2021-02-10T11:34:18.265 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.274 7fba22d3e700 1 Processor -- start 2021-02-10T11:34:18.265 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.274 7fba22d3e700 1 -- start start 2021-02-10T11:34:18.266 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.275 7fba22d3e700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fba1c08d810 0x7fba1c09d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:18.267 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.275 7fba22d3e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fba1c09dc40 0x7fba1c091ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:18.267 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.275 7fba22d3e700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fba1c0922c0 0x7fba1c090a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:18.267 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.275 7fba22d3e700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba1c0917a0 con 0x7fba1c09dc40 2021-02-10T11:34:18.267 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.275 7fba22d3e700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba1c09cc00 con 0x7fba1c08d810 2021-02-10T11:34:18.267 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.275 7fba22d3e700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba1c08dc10 con 0x7fba1c0922c0 2021-02-10T11:34:18.267 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.275 7fba1bfff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fba1c09dc40 0x7fba1c091ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:18.267 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.275 7fba1bfff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fba1c09dc40 0x7fba1c091ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:50296/0 (socket says 172.21.2.112:50296) 2021-02-10T11:34:18.267 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.275 7fba1bfff700 1 -- 172.21.2.112:0/209784400 learned_addr learned my addr 172.21.2.112:0/209784400 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:34:18.267 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.275 7fba1bfff700 1 -- 172.21.2.112:0/209784400 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fba1c0922c0 msgr2=0x7fba1c090a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:18.267 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.275 7fba1bfff700 1 --2- 172.21.2.112:0/209784400 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fba1c0922c0 0x7fba1c090a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:18.267 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.275 7fba1bfff700 1 -- 172.21.2.112:0/209784400 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fba1c08d810 msgr2=0x7fba1c09d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:18.267 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.275 7fba1bfff700 1 --2- 172.21.2.112:0/209784400 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fba1c08d810 0x7fba1c09d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:18.268 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.275 7fba1bfff700 1 -- 172.21.2.112:0/209784400 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba1c08de30 con 0x7fba1c09dc40 2021-02-10T11:34:18.268 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.276 7fba1bfff700 1 --2- 172.21.2.112:0/209784400 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fba1c09dc40 0x7fba1c091ca0 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 rx=0x7fba040066c0 tx=0x7fba0400b9c0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:18.268 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.276 7fba197fa700 1 -- 172.21.2.112:0/209784400 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fba0400bdb0 con 0x7fba1c09dc40 2021-02-10T11:34:18.268 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.276 7fba197fa700 1 -- 172.21.2.112:0/209784400 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba040009d0 con 0x7fba1c09dc40 2021-02-10T11:34:18.268 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.276 7fba197fa700 1 -- 172.21.2.112:0/209784400 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fba04002030 con 0x7fba1c09dc40 2021-02-10T11:34:18.268 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.276 7fba22d3e700 1 -- 172.21.2.112:0/209784400 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba1c08de30 con 0x7fba1c09dc40 2021-02-10T11:34:18.268 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.276 7fba22d3e700 1 -- 172.21.2.112:0/209784400 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba1c091350 con 0x7fba1c09dc40 2021-02-10T11:34:18.268 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.277 7fba197fa700 1 -- 172.21.2.112:0/209784400 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fba04000c90 con 0x7fba1c09dc40 2021-02-10T11:34:18.268 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.277 7fba197fa700 1 --2- 172.21.2.112:0/209784400 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fba000186c0 0x7fba0001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:18.268 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.277 7fba197fa700 1 -- 172.21.2.112:0/209784400 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fba04032580 con 0x7fba1c09dc40 2021-02-10T11:34:18.268 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.277 7fba20b5a700 1 --2- 172.21.2.112:0/209784400 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fba000186c0 0x7fba0001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:18.268 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.277 7fba20b5a700 1 --2- 172.21.2.112:0/209784400 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fba000186c0 0x7fba0001abc0 crc :-1 s=READY pgs=656 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:18.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.307 7fba22d3e700 1 -- 172.21.2.112:0/209784400 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba1c10a8c0 con 0x7fba1c09dc40 2021-02-10T11:34:18.296 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.310 7fba197fa700 1 -- 172.21.2.112:0/209784400 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fba0400a300 con 0x7fba1c09dc40 2021-02-10T11:34:18.422 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.435 7fba22d3e700 1 -- 172.21.2.112:0/209784400 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume deauthorize", "vol_name": "cephfs", "sub_name": "subvolume_0000000000670007", "target": ["mgr", ""], "auth_id": "alice", "group_name": "group_0000000000544641"}) v1 -- 0x7fba1c0fe2c0 con 0x7fba000186c0 2021-02-10T11:34:18.460 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.473 7fba197fa700 1 -- 172.21.2.112:0/209784400 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fba0c000e20 con 0x7fba000186c0 2021-02-10T11:34:18.488 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.501 7fba22d3e700 1 -- 172.21.2.112:0/209784400 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fba000186c0 msgr2=0x7fba0001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:18.488 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.501 7fba22d3e700 1 --2- 172.21.2.112:0/209784400 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fba000186c0 0x7fba0001abc0 crc :-1 s=READY pgs=656 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:18.488 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.501 7fba22d3e700 1 -- 172.21.2.112:0/209784400 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fba1c09dc40 msgr2=0x7fba1c091ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:18.488 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.501 7fba22d3e700 1 --2- 172.21.2.112:0/209784400 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fba1c09dc40 0x7fba1c091ca0 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 rx=0x7fba040066c0 tx=0x7fba0400b9c0).stop 2021-02-10T11:34:18.489 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.501 7fba22d3e700 1 -- 172.21.2.112:0/209784400 shutdown_connections 2021-02-10T11:34:18.489 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.501 7fba22d3e700 1 --2- 172.21.2.112:0/209784400 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fba000186c0 0x7fba0001abc0 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:18.489 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.501 7fba22d3e700 1 --2- 172.21.2.112:0/209784400 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fba1c0922c0 0x7fba1c090a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:18.489 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.501 7fba22d3e700 1 --2- 172.21.2.112:0/209784400 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fba1c09dc40 0x7fba1c091ca0 unknown :-1 s=CLOSED pgs=1330 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:18.489 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.501 7fba22d3e700 1 --2- 172.21.2.112:0/209784400 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fba1c08d810 0x7fba1c09d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:18.489 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.501 7fba22d3e700 1 -- 172.21.2.112:0/209784400 >> 172.21.2.112:0/209784400 conn(0x7fba1c1a1310 msgr2=0x7fba1c1b2690 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:18.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.501 7fba22d3e700 1 -- 172.21.2.112:0/209784400 shutdown_connections 2021-02-10T11:34:18.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.501 7fba22d3e700 1 -- 172.21.2.112:0/209784400 wait complete. 2021-02-10T11:34:18.496 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth ls --format=json-pretty 2021-02-10T11:34:18.921 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.918 7f0770e67700 1 Processor -- start 2021-02-10T11:34:18.922 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.918 7f0770e67700 1 -- start start 2021-02-10T11:34:18.922 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.919 7f0770e67700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f076c082920 0x7f076c0832d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:18.922 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.919 7f0770e67700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f076c0838f0 0x7f076c083cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:18.922 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.919 7f0770e67700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f076c082d50 con 0x7f076c0e53e0 2021-02-10T11:34:18.922 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.919 7f0770e67700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f076c0e57e0 con 0x7f076c082920 2021-02-10T11:34:18.922 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.919 7f0770e67700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f076c1b0d40 con 0x7f076c0838f0 2021-02-10T11:34:18.922 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.919 7f076a312700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f076c082920 0x7f076c0832d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:18.922 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.919 7f076a312700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f076c082920 0x7f076c0832d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:50304/0 (socket says 172.21.2.112:50304) 2021-02-10T11:34:18.922 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.919 7f076a312700 1 -- 172.21.2.112:0/2384932186 learned_addr learned my addr 172.21.2.112:0/2384932186 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:34:18.922 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.919 7f076a312700 1 -- 172.21.2.112:0/2384932186 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f076c0838f0 msgr2=0x7f076c083cf0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:34:18.922 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.919 7f076a312700 1 --2- 172.21.2.112:0/2384932186 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f076c0838f0 0x7f076c083cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:18.923 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.919 7f076a312700 1 -- 172.21.2.112:0/2384932186 >> v1:172.21.2.120:6789/0 conn(0x7f076c0e53e0 legacy=0x7f076c1b0550 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:34:18.923 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.919 7f076a312700 1 -- 172.21.2.112:0/2384932186 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f076c084310 con 0x7f076c082920 2021-02-10T11:34:18.923 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.920 7f076a312700 1 --2- 172.21.2.112:0/2384932186 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f076c082920 0x7f076c0832d0 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 rx=0x7f075c006700 tx=0x7f075c009b80).ready entity=mon.0 client_cookie=fbf263e05d3094f7 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:18.923 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.920 7f0769b11700 1 -- 172.21.2.112:0/2384932186 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f075c00c0c0 con 0x7f076c082920 2021-02-10T11:34:18.923 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.920 7f0769b11700 1 -- 172.21.2.112:0/2384932186 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f075c00c340 con 0x7f076c082920 2021-02-10T11:34:18.923 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.920 7f0769b11700 1 -- 172.21.2.112:0/2384932186 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f075c00c820 con 0x7f076c082920 2021-02-10T11:34:18.923 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.920 7f0770e67700 1 -- 172.21.2.112:0/2384932186 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f076c082920 msgr2=0x7f076c0832d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:18.923 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.920 7f0770e67700 1 --2- 172.21.2.112:0/2384932186 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f076c082920 0x7f076c0832d0 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 rx=0x7f075c006700 tx=0x7f075c009b80).stop 2021-02-10T11:34:18.923 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.921 7f0770e67700 1 -- 172.21.2.112:0/2384932186 shutdown_connections 2021-02-10T11:34:18.923 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.921 7f0770e67700 1 --2- 172.21.2.112:0/2384932186 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f076c082920 0x7f076c0832d0 unknown :-1 s=CLOSED pgs=1331 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:18.923 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.921 7f0770e67700 1 --2- 172.21.2.112:0/2384932186 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f076c0838f0 0x7f076c083cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:18.923 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.921 7f0770e67700 1 -- 172.21.2.112:0/2384932186 >> 172.21.2.112:0/2384932186 conn(0x7f076c0ce500 msgr2=0x7f076c0e4610 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:18.924 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.921 7f0770e67700 1 -- 172.21.2.112:0/2384932186 shutdown_connections 2021-02-10T11:34:18.924 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.921 7f0770e67700 1 -- 172.21.2.112:0/2384932186 wait complete. 2021-02-10T11:34:18.925 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.935 7f0770e67700 1 Processor -- start 2021-02-10T11:34:18.926 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.935 7f0770e67700 1 -- start start 2021-02-10T11:34:18.926 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.935 7f0770e67700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f076c18f230 0x7f076c18f630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:18.926 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.935 7f0770e67700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f076c18fc50 0x7f076c190050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:18.926 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.935 7f0770e67700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f076c18d3c0 0x7f076c18d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:18.926 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.935 7f0770e67700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f076c1af490 con 0x7f076c18d3c0 2021-02-10T11:34:18.926 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.935 7f0770e67700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f076c18dde0 con 0x7f076c18fc50 2021-02-10T11:34:18.926 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.935 7f0770e67700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f076c18e000 con 0x7f076c18f230 2021-02-10T11:34:18.926 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.935 7f076b314700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f076c18d3c0 0x7f076c18d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:18.926 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.935 7f076b314700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f076c18d3c0 0x7f076c18d7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:50310/0 (socket says 172.21.2.112:50310) 2021-02-10T11:34:18.926 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.935 7f076b314700 1 -- 172.21.2.112:0/1767192972 learned_addr learned my addr 172.21.2.112:0/1767192972 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:34:18.927 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.936 7f076b314700 1 -- 172.21.2.112:0/1767192972 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f076c18f230 msgr2=0x7f076c18f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:18.927 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.936 7f076b314700 1 --2- 172.21.2.112:0/1767192972 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f076c18f230 0x7f076c18f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:18.927 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.936 7f076b314700 1 -- 172.21.2.112:0/1767192972 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f076c18fc50 msgr2=0x7f076c190050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:18.927 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.936 7f076b314700 1 --2- 172.21.2.112:0/1767192972 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f076c18fc50 0x7f076c190050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:18.927 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.936 7f076b314700 1 -- 172.21.2.112:0/1767192972 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f076c18e220 con 0x7f076c18d3c0 2021-02-10T11:34:18.927 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.936 7f076b314700 1 --2- 172.21.2.112:0/1767192972 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f076c18d3c0 0x7f076c18d7c0 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 rx=0x7f0764009960 tx=0x7f076400bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:18.927 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.936 7f07637fe700 1 -- 172.21.2.112:0/1767192972 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0764003030 con 0x7f076c18d3c0 2021-02-10T11:34:18.927 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.936 7f07637fe700 1 -- 172.21.2.112:0/1767192972 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0764001310 con 0x7f076c18d3c0 2021-02-10T11:34:18.927 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.936 7f07637fe700 1 -- 172.21.2.112:0/1767192972 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0764001830 con 0x7f076c18d3c0 2021-02-10T11:34:18.927 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.936 7f0770e67700 1 -- 172.21.2.112:0/1767192972 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f076c18e220 con 0x7f076c18d3c0 2021-02-10T11:34:18.927 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.936 7f0770e67700 1 -- 172.21.2.112:0/1767192972 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f076c18e220 con 0x7f076c18d3c0 2021-02-10T11:34:18.927 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.937 7f07637fe700 1 -- 172.21.2.112:0/1767192972 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0764000d10 con 0x7f076c18d3c0 2021-02-10T11:34:18.927 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.937 7f07637fe700 1 --2- 172.21.2.112:0/1767192972 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f07500186c0 0x7f075001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:18.927 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.937 7f07637fe700 1 -- 172.21.2.112:0/1767192972 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f0764032940 con 0x7f076c18d3c0 2021-02-10T11:34:18.928 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.937 7f076ab13700 1 --2- 172.21.2.112:0/1767192972 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f07500186c0 0x7f075001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:18.928 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.938 7f076ab13700 1 --2- 172.21.2.112:0/1767192972 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f07500186c0 0x7f075001abc0 crc :-1 s=READY pgs=657 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:18.955 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.969 7f0770e67700 1 -- 172.21.2.112:0/1767192972 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f076c18ed00 con 0x7f076c18d3c0 2021-02-10T11:34:18.958 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:18.971 7f07637fe700 1 -- 172.21.2.112:0/1767192972 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f076400be30 con 0x7f076c18d3c0 2021-02-10T11:34:19.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:19.102 7f0770e67700 1 -- 172.21.2.112:0/1767192972 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f076c18ed00 con 0x7f076c18d3c0 2021-02-10T11:34:19.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:19.102 7f07637fe700 1 -- 172.21.2.112:0/1767192972 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v37) v1 ==== 80+0+4513 (secure 0 0 0) 0x7f0764000d10 con 0x7f076c18d3c0 2021-02-10T11:34:19.090 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:34:19.090 INFO:teuthology.orchestra.run.gibba012.stdout:{ 2021-02-10T11:34:19.090 INFO:teuthology.orchestra.run.gibba012.stdout: "auth_dump": [ 2021-02-10T11:34:19.090 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:19.090 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "mds.a", 2021-02-10T11:34:19.090 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCKvyNgPkV4AxAAEmVNaghdccP6lWd8hvQnxw==", 2021-02-10T11:34:19.090 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:34:19.090 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:34:19.090 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:34:19.091 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:34:19.091 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:34:19.091 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:34:19.091 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:19.091 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:19.091 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "mds.b", 2021-02-10T11:34:19.091 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCKvyNguxONLhAAHIpry+R89MAwdhFJyEhVCw==", 2021-02-10T11:34:19.091 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:34:19.091 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:34:19.091 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:34:19.091 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:34:19.091 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:34:19.091 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:34:19.092 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:19.092 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:19.092 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.0", 2021-02-10T11:34:19.092 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCSvyNgXcKPLhAAP/Bj2F+06DCtPCzQDNq+cQ==", 2021-02-10T11:34:19.092 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:34:19.092 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:34:19.092 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:34:19.092 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:34:19.093 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:34:19.093 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:19.093 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:19.093 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.1", 2021-02-10T11:34:19.093 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCVvyNghfbcLhAA66PrRwR2M7o6VTF5UhU/Fg==", 2021-02-10T11:34:19.094 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:34:19.094 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:34:19.094 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:34:19.094 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:34:19.094 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:34:19.094 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:19.094 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:19.094 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.2", 2021-02-10T11:34:19.094 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCYvyNgwiPqLhAA66BVkRsgaUlGpAsQ15Nf8Q==", 2021-02-10T11:34:19.095 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:34:19.095 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:34:19.095 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:34:19.095 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:34:19.095 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:34:19.095 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:19.095 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:19.095 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.3", 2021-02-10T11:34:19.095 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCbvyNgqBZrLxAAERPe/1L3SXAEzI3d9ReHHg==", 2021-02-10T11:34:19.095 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:34:19.095 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:34:19.095 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:34:19.095 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:34:19.096 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:34:19.096 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:19.096 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:19.096 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.4", 2021-02-10T11:34:19.096 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCkvyNgDQsxIhAAHmVjVYnx20N4SDmKqvbcuA==", 2021-02-10T11:34:19.096 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:34:19.096 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:34:19.096 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:34:19.096 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:34:19.096 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:34:19.096 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:19.096 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:19.097 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.5", 2021-02-10T11:34:19.097 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCnvyNg8ITFIhAADd4aKybAV7btoGPx3kCg3A==", 2021-02-10T11:34:19.097 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:34:19.097 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:34:19.097 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:34:19.097 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:34:19.097 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:34:19.097 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:19.097 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:19.097 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.6", 2021-02-10T11:34:19.097 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCqvyNgxOhDIxAAzXixEa/nlDc1xXrg0aeLwg==", 2021-02-10T11:34:19.097 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:34:19.097 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:34:19.098 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:34:19.098 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:34:19.098 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:34:19.098 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:19.098 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:19.098 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.7", 2021-02-10T11:34:19.098 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCtvyNgxSqGIxAACxJQlwP7DpiVVCZHA3zn1A==", 2021-02-10T11:34:19.098 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:34:19.098 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:34:19.098 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:34:19.098 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:34:19.099 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:34:19.099 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:19.099 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:19.099 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.0", 2021-02-10T11:34:19.099 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCLvyNgDkNiHRAAygU8Za4PBusTqCgwAYWTiQ==", 2021-02-10T11:34:19.099 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:34:19.099 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:34:19.099 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow r", 2021-02-10T11:34:19.099 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:34:19.099 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:34:19.099 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:19.099 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:19.100 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.1", 2021-02-10T11:34:19.100 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCLvyNgxmLqMxAA1MlSDJ9LH9u4f2sL7mm8/Q==", 2021-02-10T11:34:19.100 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:34:19.100 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:34:19.100 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow r", 2021-02-10T11:34:19.100 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:34:19.100 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:34:19.100 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:19.101 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:19.101 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.2", 2021-02-10T11:34:19.101 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCMvyNgp8veDhAAniSVHYPtwwTog/pjbaRQmw==", 2021-02-10T11:34:19.101 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:34:19.101 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:34:19.101 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow r", 2021-02-10T11:34:19.101 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:34:19.101 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:34:19.101 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:19.101 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:19.101 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.3", 2021-02-10T11:34:19.101 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCMvyNgQEVoJRAApHcWdm1sldIxu+heyD4HdQ==", 2021-02-10T11:34:19.102 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:34:19.102 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:34:19.102 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow r", 2021-02-10T11:34:19.102 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:34:19.102 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:34:19.102 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:19.102 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:19.102 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.admin", 2021-02-10T11:34:19.102 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCGvyNg7jjJBRAAtot+ypUZOiTo1GbY+oVhVw==", 2021-02-10T11:34:19.102 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:34:19.103 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow *", 2021-02-10T11:34:19.103 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:34:19.103 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:34:19.103 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:34:19.103 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:34:19.103 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:19.103 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:19.103 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "mgr.x", 2021-02-10T11:34:19.103 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCJvyNgTPVwKBAAjduiBE8+RSXBv82tdPecWg==", 2021-02-10T11:34:19.103 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:34:19.103 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow *", 2021-02-10T11:34:19.103 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow profile mgr", 2021-02-10T11:34:19.103 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:34:19.104 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:34:19.104 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:19.104 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:19.104 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "mgr.y", 2021-02-10T11:34:19.104 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCJvyNgQ1tTDhAASwBssDy+Rz31ujw8fNlWow==", 2021-02-10T11:34:19.104 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:34:19.104 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow *", 2021-02-10T11:34:19.104 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow profile mgr", 2021-02-10T11:34:19.104 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:34:19.105 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:34:19.105 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:34:19.105 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:34:19.105 INFO:teuthology.orchestra.run.gibba012.stdout:} 2021-02-10T11:34:19.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:19.104 7f0770e67700 1 -- 172.21.2.112:0/1767192972 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f07500186c0 msgr2=0x7f075001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:19.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:19.104 7f0770e67700 1 --2- 172.21.2.112:0/1767192972 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f07500186c0 0x7f075001abc0 crc :-1 s=READY pgs=657 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:19.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:19.104 7f0770e67700 1 -- 172.21.2.112:0/1767192972 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f076c18d3c0 msgr2=0x7f076c18d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:19.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:19.104 7f0770e67700 1 --2- 172.21.2.112:0/1767192972 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f076c18d3c0 0x7f076c18d7c0 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 rx=0x7f0764009960 tx=0x7f076400bcf0).stop 2021-02-10T11:34:19.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:19.104 7f0770e67700 1 -- 172.21.2.112:0/1767192972 shutdown_connections 2021-02-10T11:34:19.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:19.104 7f0770e67700 1 --2- 172.21.2.112:0/1767192972 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f07500186c0 0x7f075001abc0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:19.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:19.104 7f0770e67700 1 --2- 172.21.2.112:0/1767192972 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f076c18fc50 0x7f076c190050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:19.109 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:19.104 7f0770e67700 1 --2- 172.21.2.112:0/1767192972 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f076c18d3c0 0x7f076c18d7c0 unknown :-1 s=CLOSED pgs=1332 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:19.109 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:19.104 7f0770e67700 1 --2- 172.21.2.112:0/1767192972 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f076c18f230 0x7f076c18f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:19.109 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:19.104 7f0770e67700 1 -- 172.21.2.112:0/1767192972 >> 172.21.2.112:0/1767192972 conn(0x7f076c18cf70 msgr2=0x7f076c1a71b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:19.109 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:19.104 7f0770e67700 1 -- 172.21.2.112:0/1767192972 shutdown_connections 2021-02-10T11:34:19.109 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:19.104 7f0770e67700 1 -- 172.21.2.112:0/1767192972 wait complete. 2021-02-10T11:34:19.110 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000670007 --group_name group_0000000000544641 2021-02-10T11:34:19.541 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:19.536 7f997bd0d700 1 Processor -- start 2021-02-10T11:34:19.541 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:19.536 7f997bd0d700 1 -- start start 2021-02-10T11:34:19.541 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:19.537 7f997bd0d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9974078520 0x7f99740d1910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:19.541 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:19.537 7f997bd0d700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9974078920 con 0x7f99740e5580 2021-02-10T11:34:19.541 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:19.537 7f997bd0d700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9974078b90 con 0x7f99740e5180 2021-02-10T11:34:19.541 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:19.537 7f997bd0d700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99741b4150 con 0x7f9974078520 2021-02-10T11:34:19.541 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:19.537 7f9979328700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f99740e5580 0x7f9974079110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:49778/0 (socket says 172.21.2.112:49778) 2021-02-10T11:34:19.541 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:19.537 7f9979328700 1 -- 172.21.2.112:0/210356199 learned_addr learned my addr 172.21.2.112:0/210356199 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:34:19.541 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:19.537 7f9978b27700 1 -- 172.21.2.112:0/210356199 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2280159601 0 0) 0x7f99700009b0 con 0x7f99740e5580 2021-02-10T11:34:19.541 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:19.537 7f9978b27700 1 -- 172.21.2.112:0/210356199 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9960002fe0 con 0x7f99740e5580 2021-02-10T11:34:19.542 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:19.538 7f997a32a700 1 --2- 172.21.2.112:0/210356199 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9974078520 0x7f99740d1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:19.542 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:19.538 7f9978b27700 1 -- 172.21.2.112:0/210356199 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3024593231 0 0) 0x7f9970001010 con 0x7f99740e5580 2021-02-10T11:34:19.542 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:19.538 7f9978b27700 1 -- 172.21.2.112:0/210356199 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9974078520 msgr2=0x7f99740d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:19.542 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:19.538 7f9978b27700 1 --2- 172.21.2.112:0/210356199 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9974078520 0x7f99740d1910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:19.542 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:19.538 7f9978b27700 1 -- 172.21.2.112:0/210356199 >> v1:172.21.2.120:6789/0 conn(0x7f99740e5180 legacy=0x7f99741b38f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:19.542 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:19.538 7f9978b27700 1 -- 172.21.2.112:0/210356199 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99740d1f30 con 0x7f99740e5580 2021-02-10T11:34:19.542 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:19.538 7f9978b27700 1 -- 172.21.2.112:0/210356199 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f9970001590 con 0x7f99740e5580 2021-02-10T11:34:19.542 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:19.538 7f9978b27700 1 -- 172.21.2.112:0/210356199 >> v1:172.21.2.112:6789/0 conn(0x7f99740e5580 legacy=0x7f9974079110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:19.542 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:19.538 7f9978b27700 1 --2- 172.21.2.112:0/210356199 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9960003af0 0x7f9960005f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:19.542 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:19.538 7f9978b27700 1 --2- 172.21.2.112:0/210356199 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9960006800 0x7f9960008d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:19.542 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:19.538 7f9978b27700 1 --2- 172.21.2.112:0/210356199 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f99600095a0 0x7f996000bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:19.543 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:19.538 7f9978b27700 1 -- 172.21.2.112:0/210356199 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f996000c370 con 0x7f99600095a0 2021-02-10T11:34:19.543 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:19.538 7f9978b27700 1 -- 172.21.2.112:0/210356199 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f996000c590 con 0x7f9960006800 2021-02-10T11:34:19.543 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:19.538 7f9978b27700 1 -- 172.21.2.112:0/210356199 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f996000c7b0 con 0x7f9960003af0 2021-02-10T11:34:19.543 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:19.538 7f997a32a700 1 --2- 172.21.2.112:0/210356199 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f99600095a0 0x7f996000bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:19.543 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:19.538 7f9979b29700 1 --2- 172.21.2.112:0/210356199 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9960003af0 0x7f9960005f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:19.543 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:19.538 7f9979328700 1 --2- 172.21.2.112:0/210356199 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9960006800 0x7f9960008d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:19.543 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:19.539 7f997a32a700 1 -- 172.21.2.112:0/210356199 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9960003af0 msgr2=0x7f9960005f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:19.543 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:19.539 7f997a32a700 1 --2- 172.21.2.112:0/210356199 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9960003af0 0x7f9960005f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:19.543 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:19.539 7f997a32a700 1 -- 172.21.2.112:0/210356199 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9960006800 msgr2=0x7f9960008d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:19.543 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:19.539 7f997a32a700 1 --2- 172.21.2.112:0/210356199 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9960006800 0x7f9960008d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:19.543 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:19.539 7f997a32a700 1 -- 172.21.2.112:0/210356199 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f996000ca50 con 0x7f99600095a0 2021-02-10T11:34:19.543 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:19.539 7f997a32a700 1 --2- 172.21.2.112:0/210356199 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f99600095a0 0x7f996000bac0 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 rx=0x7f996c04da40 tx=0x7f996c04dde0).ready entity=mon.0 client_cookie=30ef2da1897d8aad server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:19.544 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:19.539 7f9978b27700 1 -- 172.21.2.112:0/210356199 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f996c006a70 con 0x7f99600095a0 2021-02-10T11:34:19.544 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:19.539 7f9978b27700 1 -- 172.21.2.112:0/210356199 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f996c04f300 con 0x7f99600095a0 2021-02-10T11:34:19.544 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:19.539 7f9978b27700 1 -- 172.21.2.112:0/210356199 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f996c04f760 con 0x7f99600095a0 2021-02-10T11:34:19.544 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:19.539 7f997bd0d700 1 -- 172.21.2.112:0/210356199 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f99600095a0 msgr2=0x7f996000bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:19.544 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:19.539 7f997bd0d700 1 --2- 172.21.2.112:0/210356199 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f99600095a0 0x7f996000bac0 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 rx=0x7f996c04da40 tx=0x7f996c04dde0).stop 2021-02-10T11:34:19.544 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:19.539 7f997a32a700 1 -- 172.21.2.112:0/210356199 reap_dead start 2021-02-10T11:34:19.544 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:19.539 7f997bd0d700 1 -- 172.21.2.112:0/210356199 shutdown_connections 2021-02-10T11:34:19.544 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:19.539 7f997bd0d700 1 -- 172.21.2.112:0/210356199 >> 172.21.2.112:0/210356199 conn(0x7f99740ce830 msgr2=0x7f99740e4670 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:19.544 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:19.540 7f997bd0d700 1 -- 172.21.2.112:0/210356199 shutdown_connections 2021-02-10T11:34:19.544 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:19.540 7f997bd0d700 1 -- 172.21.2.112:0/210356199 wait complete. 2021-02-10T11:34:19.546 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:19.554 7f997bd0d700 1 Processor -- start 2021-02-10T11:34:19.546 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:19.554 7f997bd0d700 1 -- start start 2021-02-10T11:34:19.546 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:19.555 7f997bd0d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f99741a3100 0x7f99741a3500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:19.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:19.555 7f997bd0d700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f997410cc70 0x7f997410d070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:19.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:19.555 7f997bd0d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f99740b6010 0x7f99740b6410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:19.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:19.555 7f997bd0d700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f997410d9e0 con 0x7f99741a3100 2021-02-10T11:34:19.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:19.555 7f997bd0d700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f997410d690 con 0x7f997410cc70 2021-02-10T11:34:19.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:19.555 7f997bd0d700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99740b6a30 con 0x7f99740b6010 2021-02-10T11:34:19.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:19.555 7f9979b29700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f99741a3100 0x7f99741a3500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:19.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:19.555 7f9979b29700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f99741a3100 0x7f99741a3500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:50330/0 (socket says 172.21.2.112:50330) 2021-02-10T11:34:19.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:19.555 7f9979b29700 1 -- 172.21.2.112:0/3646960550 learned_addr learned my addr 172.21.2.112:0/3646960550 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:34:19.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:19.555 7f9979b29700 1 -- 172.21.2.112:0/3646960550 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f99740b6010 msgr2=0x7f99740b6410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:19.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:19.555 7f9979b29700 1 --2- 172.21.2.112:0/3646960550 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f99740b6010 0x7f99740b6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:19.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:19.555 7f9979b29700 1 -- 172.21.2.112:0/3646960550 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f997410cc70 msgr2=0x7f997410d070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:19.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:19.555 7f9979b29700 1 --2- 172.21.2.112:0/3646960550 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f997410cc70 0x7f997410d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:19.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:19.555 7f9979b29700 1 -- 172.21.2.112:0/3646960550 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99740b6c50 con 0x7f99741a3100 2021-02-10T11:34:19.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:19.555 7f9979b29700 1 --2- 172.21.2.112:0/3646960550 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f99741a3100 0x7f99741a3500 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 rx=0x7f996800b590 tx=0x7f996800b930).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:19.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:19.555 7f99667fc700 1 -- 172.21.2.112:0/3646960550 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f996800bd20 con 0x7f99741a3100 2021-02-10T11:34:19.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:19.555 7f99667fc700 1 -- 172.21.2.112:0/3646960550 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9968000ca0 con 0x7f99741a3100 2021-02-10T11:34:19.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:19.555 7f99667fc700 1 -- 172.21.2.112:0/3646960550 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9968002220 con 0x7f99741a3100 2021-02-10T11:34:19.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:19.555 7f997bd0d700 1 -- 172.21.2.112:0/3646960550 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99740b6c50 con 0x7f99741a3100 2021-02-10T11:34:19.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:19.556 7f997bd0d700 1 -- 172.21.2.112:0/3646960550 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99740b6c50 con 0x7f99741a3100 2021-02-10T11:34:19.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:19.556 7f99667fc700 1 -- 172.21.2.112:0/3646960550 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f99680052e0 con 0x7f99741a3100 2021-02-10T11:34:19.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:19.556 7f99667fc700 1 --2- 172.21.2.112:0/3646960550 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f995c0186c0 0x7f995c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:19.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:19.556 7f99667fc700 1 -- 172.21.2.112:0/3646960550 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f9968032610 con 0x7f99741a3100 2021-02-10T11:34:19.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:19.556 7f9979328700 1 --2- 172.21.2.112:0/3646960550 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f995c0186c0 0x7f995c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:19.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:19.557 7f9979328700 1 --2- 172.21.2.112:0/3646960550 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f995c0186c0 0x7f995c01abc0 crc :-1 s=READY pgs=658 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:19.573 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:19.587 7f997bd0d700 1 -- 172.21.2.112:0/3646960550 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f997410ca40 con 0x7f99741a3100 2021-02-10T11:34:19.576 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:19.589 7f99667fc700 1 -- 172.21.2.112:0/3646960550 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9968006bb0 con 0x7f99741a3100 2021-02-10T11:34:19.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:19.716 7f997bd0d700 1 -- 172.21.2.112:0/3646960550 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000670007", "target": ["mgr", ""], "group_name": "group_0000000000544641"}) v1 -- 0x7f99740fe2a0 con 0x7f995c0186c0 2021-02-10T11:34:19.714 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:19.727 7f99667fc700 1 -- 172.21.2.112:0/3646960550 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f9970000e20 con 0x7f995c0186c0 2021-02-10T11:34:19.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:19.732 7f997bd0d700 1 -- 172.21.2.112:0/3646960550 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f995c0186c0 msgr2=0x7f995c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:19.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:19.732 7f997bd0d700 1 --2- 172.21.2.112:0/3646960550 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f995c0186c0 0x7f995c01abc0 crc :-1 s=READY pgs=658 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:19.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:19.732 7f997bd0d700 1 -- 172.21.2.112:0/3646960550 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f99741a3100 msgr2=0x7f99741a3500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:19.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:19.732 7f997bd0d700 1 --2- 172.21.2.112:0/3646960550 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f99741a3100 0x7f99741a3500 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 rx=0x7f996800b590 tx=0x7f996800b930).stop 2021-02-10T11:34:19.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:19.732 7f997bd0d700 1 -- 172.21.2.112:0/3646960550 shutdown_connections 2021-02-10T11:34:19.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:19.732 7f997bd0d700 1 --2- 172.21.2.112:0/3646960550 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f995c0186c0 0x7f995c01abc0 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:19.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:19.732 7f997bd0d700 1 --2- 172.21.2.112:0/3646960550 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f997410cc70 0x7f997410d070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:19.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:19.732 7f997bd0d700 1 --2- 172.21.2.112:0/3646960550 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f99740b6010 0x7f99740b6410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:19.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:19.732 7f997bd0d700 1 --2- 172.21.2.112:0/3646960550 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f99741a3100 0x7f99741a3500 unknown :-1 s=CLOSED pgs=1335 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:19.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:19.732 7f997bd0d700 1 -- 172.21.2.112:0/3646960550 >> 172.21.2.112:0/3646960550 conn(0x7f99740a3880 msgr2=0x7f99740ad320 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:19.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:19.733 7f997bd0d700 1 -- 172.21.2.112:0/3646960550 shutdown_connections 2021-02-10T11:34:19.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:19.733 7f997bd0d700 1 -- 172.21.2.112:0/3646960550 wait complete. 2021-02-10T11:34:19.728 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolumegroup rm cephfs group_0000000000544641 2021-02-10T11:34:20.160 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:20.157 7f308e1f7700 1 Processor -- start 2021-02-10T11:34:20.160 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:20.157 7f308e1f7700 1 -- start start 2021-02-10T11:34:20.161 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:20.157 7f308e1f7700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f30880e5170 0x7f30881b38e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:20.161 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:20.157 7f308e1f7700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f30881b3f00 0x7f3088077510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:20.161 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:20.157 7f308e1f7700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3088078b90 con 0x7f30880e5570 2021-02-10T11:34:20.161 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:20.157 7f308e1f7700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f308807a120 con 0x7f30881b3f00 2021-02-10T11:34:20.161 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:20.157 7f308e1f7700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f308807a310 con 0x7f30880e5170 2021-02-10T11:34:20.161 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:20.158 7f3087fff700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f30880e5170 0x7f30881b38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:20.161 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:20.158 7f30877fe700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f30881b3f00 0x7f3088077510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:20.161 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:20.158 7f30877fe700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f30881b3f00 0x7f3088077510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3300/0 says I am v2:172.21.2.112:52616/0 (socket says 172.21.2.112:52616) 2021-02-10T11:34:20.161 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:20.158 7f30877fe700 1 -- 172.21.2.112:0/1509513680 learned_addr learned my addr 172.21.2.112:0/1509513680 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:34:20.161 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:20.158 7f3087fff700 1 -- 172.21.2.112:0/1509513680 >> v2:172.21.2.120:3300/0 conn(0x7f30881b3f00 msgr2=0x7f3088077510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:20.161 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:20.158 7f3087fff700 1 --2- 172.21.2.112:0/1509513680 >> v2:172.21.2.120:3300/0 conn(0x7f30881b3f00 0x7f3088077510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:20.161 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:20.158 7f3087fff700 1 -- 172.21.2.112:0/1509513680 >> v1:172.21.2.120:6789/0 conn(0x7f30880e5570 legacy=0x7f3088077b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:20.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:20.158 7f3087fff700 1 -- 172.21.2.112:0/1509513680 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f308807a500 con 0x7f30880e5170 2021-02-10T11:34:20.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:20.158 7f30877fe700 1 --2- 172.21.2.112:0/1509513680 >> v2:172.21.2.120:3300/0 conn(0x7f30881b3f00 0x7f3088077510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:34:20.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:20.158 7f3087fff700 1 --2- 172.21.2.112:0/1509513680 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f30880e5170 0x7f30881b38e0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f3078006650 tx=0x7f3078009a80).ready entity=mon.2 client_cookie=c418af6d50ead624 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:20.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:20.159 7f3086ffd700 1 -- 172.21.2.112:0/1509513680 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f30780009b0 con 0x7f30880e5170 2021-02-10T11:34:20.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:20.159 7f3086ffd700 1 -- 172.21.2.112:0/1509513680 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f307800c0c0 con 0x7f30880e5170 2021-02-10T11:34:20.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:20.159 7f3086ffd700 1 -- 172.21.2.112:0/1509513680 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f307800c5e0 con 0x7f30880e5170 2021-02-10T11:34:20.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:20.159 7f308e1f7700 1 -- 172.21.2.112:0/1509513680 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f30880e5170 msgr2=0x7f30881b38e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:20.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:20.159 7f308e1f7700 1 --2- 172.21.2.112:0/1509513680 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f30880e5170 0x7f30881b38e0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f3078006650 tx=0x7f3078009a80).stop 2021-02-10T11:34:20.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:20.159 7f308e1f7700 1 -- 172.21.2.112:0/1509513680 shutdown_connections 2021-02-10T11:34:20.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:20.159 7f308e1f7700 1 --2- 172.21.2.112:0/1509513680 >> v2:172.21.2.120:3300/0 conn(0x7f30881b3f00 0x7f3088077510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:20.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:20.159 7f308e1f7700 1 --2- 172.21.2.112:0/1509513680 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f30880e5170 0x7f30881b38e0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:20.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:20.159 7f308e1f7700 1 -- 172.21.2.112:0/1509513680 >> 172.21.2.112:0/1509513680 conn(0x7f30880ce820 msgr2=0x7f30880e4660 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:20.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:20.159 7f308e1f7700 1 -- 172.21.2.112:0/1509513680 shutdown_connections 2021-02-10T11:34:20.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:20.160 7f308e1f7700 1 -- 172.21.2.112:0/1509513680 wait complete. 2021-02-10T11:34:20.164 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:20.174 7f308e1f7700 1 Processor -- start 2021-02-10T11:34:20.165 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:20.174 7f308e1f7700 1 -- start start 2021-02-10T11:34:20.165 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:20.174 7f308e1f7700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f308809e350 0x7f30880be490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:20.165 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:20.174 7f308e1f7700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f30880ab5d0 0x7f30880ab9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:20.165 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:20.174 7f308e1f7700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f30880abff0 0x7f3088110900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:20.165 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:20.174 7f308e1f7700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30880beab0 con 0x7f30880ab5d0 2021-02-10T11:34:20.165 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:20.174 7f308e1f7700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f308809ddd0 con 0x7f30880abff0 2021-02-10T11:34:20.165 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:20.174 7f308e1f7700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f308809e750 con 0x7f308809e350 2021-02-10T11:34:20.165 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:20.175 7f30877fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f30880ab5d0 0x7f30880ab9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:20.165 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:20.175 7f30877fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f30880ab5d0 0x7f30880ab9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:50340/0 (socket says 172.21.2.112:50340) 2021-02-10T11:34:20.165 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:20.175 7f30877fe700 1 -- 172.21.2.112:0/2865971772 learned_addr learned my addr 172.21.2.112:0/2865971772 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:34:20.165 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:20.175 7f3087fff700 1 --2- 172.21.2.112:0/2865971772 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f308809e350 0x7f30880be490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:20.165 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:20.175 7f308c814700 1 --2- 172.21.2.112:0/2865971772 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f30880abff0 0x7f3088110900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:20.166 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:20.175 7f30877fe700 1 -- 172.21.2.112:0/2865971772 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f308809e350 msgr2=0x7f30880be490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:20.166 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:20.175 7f30877fe700 1 --2- 172.21.2.112:0/2865971772 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f308809e350 0x7f30880be490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:20.166 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:20.175 7f30877fe700 1 -- 172.21.2.112:0/2865971772 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f30880abff0 msgr2=0x7f3088110900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:20.166 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:20.175 7f30877fe700 1 --2- 172.21.2.112:0/2865971772 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f30880abff0 0x7f3088110900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:20.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:20.175 7f30877fe700 1 -- 172.21.2.112:0/2865971772 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3088110f20 con 0x7f30880ab5d0 2021-02-10T11:34:20.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:20.175 7f308c814700 1 --2- 172.21.2.112:0/2865971772 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f30880abff0 0x7f3088110900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:34:20.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:20.175 7f30877fe700 1 --2- 172.21.2.112:0/2865971772 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f30880ab5d0 0x7f30880ab9d0 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 rx=0x7f307c006ee0 tx=0x7f307c00b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:20.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:20.175 7f3084ff9700 1 -- 172.21.2.112:0/2865971772 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f307c00bd50 con 0x7f30880ab5d0 2021-02-10T11:34:20.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:20.176 7f3084ff9700 1 -- 172.21.2.112:0/2865971772 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f307c0052e0 con 0x7f30880ab5d0 2021-02-10T11:34:20.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:20.176 7f308e1f7700 1 -- 172.21.2.112:0/2865971772 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3088110f20 con 0x7f30880ab5d0 2021-02-10T11:34:20.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:20.176 7f3084ff9700 1 -- 172.21.2.112:0/2865971772 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f307c005800 con 0x7f30880ab5d0 2021-02-10T11:34:20.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:20.176 7f308e1f7700 1 -- 172.21.2.112:0/2865971772 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30881114b0 con 0x7f30880ab5d0 2021-02-10T11:34:20.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:20.176 7f3084ff9700 1 -- 172.21.2.112:0/2865971772 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f307c0054c0 con 0x7f30880ab5d0 2021-02-10T11:34:20.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:20.176 7f3084ff9700 1 --2- 172.21.2.112:0/2865971772 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f30680186c0 0x7f306801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:20.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:20.176 7f3084ff9700 1 -- 172.21.2.112:0/2865971772 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f307c032420 con 0x7f30880ab5d0 2021-02-10T11:34:20.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:20.177 7f3087fff700 1 --2- 172.21.2.112:0/2865971772 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f30680186c0 0x7f306801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:20.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:20.177 7f3087fff700 1 --2- 172.21.2.112:0/2865971772 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f30680186c0 0x7f306801abc0 crc :-1 s=READY pgs=659 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:20.194 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:20.207 7f308e1f7700 1 -- 172.21.2.112:0/2865971772 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3088105c50 con 0x7f30880ab5d0 2021-02-10T11:34:20.196 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:20.210 7f3084ff9700 1 -- 172.21.2.112:0/2865971772 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f307c000940 con 0x7f30880ab5d0 2021-02-10T11:34:20.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:20.337 7f308e1f7700 1 -- 172.21.2.112:0/2865971772 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolumegroup rm", "vol_name": "cephfs", "target": ["mgr", ""], "group_name": "group_0000000000544641"}) v1 -- 0x7f3088111a50 con 0x7f30680186c0 2021-02-10T11:34:20.327 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:20.340 7f3084ff9700 1 -- 172.21.2.112:0/2865971772 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f3078000c90 con 0x7f30680186c0 2021-02-10T11:34:20.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:20.345 7f308e1f7700 1 -- 172.21.2.112:0/2865971772 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f30680186c0 msgr2=0x7f306801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:20.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:20.345 7f308e1f7700 1 --2- 172.21.2.112:0/2865971772 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f30680186c0 0x7f306801abc0 crc :-1 s=READY pgs=659 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:20.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:20.345 7f308e1f7700 1 -- 172.21.2.112:0/2865971772 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f30880ab5d0 msgr2=0x7f30880ab9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:20.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:20.345 7f308e1f7700 1 --2- 172.21.2.112:0/2865971772 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f30880ab5d0 0x7f30880ab9d0 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 rx=0x7f307c006ee0 tx=0x7f307c00b960).stop 2021-02-10T11:34:20.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:20.346 7f308e1f7700 1 -- 172.21.2.112:0/2865971772 shutdown_connections 2021-02-10T11:34:20.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:20.346 7f308e1f7700 1 --2- 172.21.2.112:0/2865971772 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f30680186c0 0x7f306801abc0 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:20.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:20.346 7f308e1f7700 1 --2- 172.21.2.112:0/2865971772 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f30880abff0 0x7f3088110900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:20.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:20.346 7f308e1f7700 1 --2- 172.21.2.112:0/2865971772 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f30880ab5d0 0x7f30880ab9d0 unknown :-1 s=CLOSED pgs=1336 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:20.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:20.346 7f308e1f7700 1 --2- 172.21.2.112:0/2865971772 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f308809e350 0x7f30880be490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:20.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:20.346 7f308e1f7700 1 -- 172.21.2.112:0/2865971772 >> 172.21.2.112:0/2865971772 conn(0x7f30880a3ee0 msgr2=0x7f3088093ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:20.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:20.346 7f308e1f7700 1 -- 172.21.2.112:0/2865971772 shutdown_connections 2021-02-10T11:34:20.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:20.346 7f308e1f7700 1 -- 172.21.2.112:0/2865971772 wait complete. 2021-02-10T11:34:20.342 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:34:20.342 DEBUG:teuthology.orchestra.run.gibba012:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:34:20.688 DEBUG:teuthology.orchestra.run.gibba020:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:34:21.031 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:34:21.031 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:21.032 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:34:21.032 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:34:21.367 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:34:21.367 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:34:21.367 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba012.front.sepia.ceph.com... 2021-02-10T11:34:21.367 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:34:21.367 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:34:21.720 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba012.stderr:ceph-fuse[129175]: fuse finished with error 0 and tester_r 0 2021-02-10T11:34:21.721 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:21.722 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:34:21.722 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:34:22.057 INFO:teuthology.orchestra.run.gibba012.stdout:ext2/ext3 2021-02-10T11:34:22.057 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:34:22.057 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:34:22.057 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:34:22.392 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:22.392 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:34:22.392 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:34:22.727 INFO:teuthology.orchestra.run.gibba012.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: No such file or directory 2021-02-10T11:34:22.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:34:22.728 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2021-02-10T11:34:22.728 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:34:22.728 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:34:23.063 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:23.064 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:34:23.064 DEBUG:teuthology.orchestra.run.gibba020:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:34:23.399 INFO:teuthology.orchestra.run.gibba020.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:34:23.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:34:23.399 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:34:23.400 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:34:23.400 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:34:23.733 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:24.583 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:34:24.584 DEBUG:teuthology.orchestra.run.gibba020:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:34:24.919 INFO:teuthology.orchestra.run.gibba020.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:34:24.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:34:24.919 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:34:24.919 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:34:24.920 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:34:25.255 DEBUG:teuthology.orchestra.run.gibba012:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2021-02-10T11:34:25.258 DEBUG:teuthology.orchestra.run.gibba020:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2021-02-10T11:34:25.617 INFO:teuthology.orchestra.run.gibba012.stdout:[global] 2021-02-10T11:34:25.617 INFO:teuthology.orchestra.run.gibba012.stdout: chdir = "" 2021-02-10T11:34:25.617 INFO:teuthology.orchestra.run.gibba012.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2021-02-10T11:34:25.617 INFO:teuthology.orchestra.run.gibba012.stdout: auth supported = cephx 2021-02-10T11:34:25.618 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:34:25.618 INFO:teuthology.orchestra.run.gibba012.stdout: filestore xattr use omap = true 2021-02-10T11:34:25.618 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:34:25.618 INFO:teuthology.orchestra.run.gibba012.stdout: mon clock drift allowed = 1.000 2021-02-10T11:34:25.618 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:34:25.618 INFO:teuthology.orchestra.run.gibba012.stdout: osd crush chooseleaf type = 0 2021-02-10T11:34:25.618 INFO:teuthology.orchestra.run.gibba012.stdout: auth debug = true 2021-02-10T11:34:25.618 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:34:25.618 INFO:teuthology.orchestra.run.gibba012.stdout: ms die on old message = true 2021-02-10T11:34:25.618 INFO:teuthology.orchestra.run.gibba012.stdout: ms die on bug = true 2021-02-10T11:34:25.618 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:34:25.618 INFO:teuthology.orchestra.run.gibba012.stdout: mon max pg per osd = 10000 # >= luminous 2021-02-10T11:34:25.619 INFO:teuthology.orchestra.run.gibba012.stdout: mon pg warn max object skew = 0 2021-02-10T11:34:25.619 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:34:25.619 INFO:teuthology.orchestra.run.gibba012.stdout: osd pool default size = 2 2021-02-10T11:34:25.619 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:34:25.619 INFO:teuthology.orchestra.run.gibba012.stdout: mon osd allow primary affinity = true 2021-02-10T11:34:25.619 INFO:teuthology.orchestra.run.gibba012.stdout: mon osd allow pg remap = true 2021-02-10T11:34:25.619 INFO:teuthology.orchestra.run.gibba012.stdout: mon warn on legacy crush tunables = false 2021-02-10T11:34:25.619 INFO:teuthology.orchestra.run.gibba012.stdout: mon warn on crush straw calc version zero = false 2021-02-10T11:34:25.619 INFO:teuthology.orchestra.run.gibba012.stdout: mon warn on no sortbitwise = false 2021-02-10T11:34:25.619 INFO:teuthology.orchestra.run.gibba012.stdout: mon warn on osd down out interval zero = false 2021-02-10T11:34:25.619 INFO:teuthology.orchestra.run.gibba012.stdout: mon warn on too few osds = false 2021-02-10T11:34:25.619 INFO:teuthology.orchestra.run.gibba012.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2021-02-10T11:34:25.619 INFO:teuthology.orchestra.run.gibba012.stdout: mon_warn_on_pool_no_redundancy = false 2021-02-10T11:34:25.620 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:34:25.620 INFO:teuthology.orchestra.run.gibba012.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2021-02-10T11:34:25.620 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:34:25.620 INFO:teuthology.orchestra.run.gibba012.stdout: osd default data pool replay window = 5 2021-02-10T11:34:25.620 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:34:25.620 INFO:teuthology.orchestra.run.gibba012.stdout: mon allow pool delete = true 2021-02-10T11:34:25.620 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:34:25.620 INFO:teuthology.orchestra.run.gibba012.stdout: mon cluster log file level = debug 2021-02-10T11:34:25.620 INFO:teuthology.orchestra.run.gibba012.stdout: debug asserts on shutdown = true 2021-02-10T11:34:25.620 INFO:teuthology.orchestra.run.gibba012.stdout: mon health detail to clog = false 2021-02-10T11:34:25.620 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:34:25.620 INFO:teuthology.orchestra.run.gibba012.stdout: # we see this fail in qa on *nautilus*; bump up retries 2021-02-10T11:34:25.620 INFO:teuthology.orchestra.run.gibba012.stdout: mon_client_directed_command_retry = 4 2021-02-10T11:34:25.621 INFO:teuthology.orchestra.run.gibba012.stdout: mon host = "172.21.2.112,172.21.2.120,[v2:172.21.2.120:3301,v1:172.21.2.120:6790]" 2021-02-10T11:34:25.621 INFO:teuthology.orchestra.run.gibba012.stdout: fsid = f14f004e-0699-4584-ad81-3754258493de 2021-02-10T11:34:25.621 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:34:25.621 INFO:teuthology.orchestra.run.gibba012.stdout:[osd] 2021-02-10T11:34:25.621 INFO:teuthology.orchestra.run.gibba012.stdout: osd journal size = 100 2021-02-10T11:34:25.621 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:34:25.621 INFO:teuthology.orchestra.run.gibba012.stdout: osd scrub load threshold = 5.0 2021-02-10T11:34:25.621 INFO:teuthology.orchestra.run.gibba012.stdout: osd scrub max interval = 600 2021-02-10T11:34:25.621 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:34:25.621 INFO:teuthology.orchestra.run.gibba012.stdout: osd recover clone overlap = true 2021-02-10T11:34:25.621 INFO:teuthology.orchestra.run.gibba012.stdout: osd recovery max chunk = 1048576 2021-02-10T11:34:25.621 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:34:25.621 INFO:teuthology.orchestra.run.gibba012.stdout: osd debug shutdown = true 2021-02-10T11:34:25.622 INFO:teuthology.orchestra.run.gibba012.stdout: osd debug op order = true 2021-02-10T11:34:25.622 INFO:teuthology.orchestra.run.gibba012.stdout: osd debug verify stray on activate = true 2021-02-10T11:34:25.622 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:34:25.622 INFO:teuthology.orchestra.run.gibba012.stdout: osd open classes on start = true 2021-02-10T11:34:25.622 INFO:teuthology.orchestra.run.gibba012.stdout: osd debug pg log writeout = true 2021-02-10T11:34:25.622 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:34:25.622 INFO:teuthology.orchestra.run.gibba012.stdout: osd deep scrub update digest min age = 30 2021-02-10T11:34:25.622 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:34:25.622 INFO:teuthology.orchestra.run.gibba012.stdout: osd map max advance = 10 2021-02-10T11:34:25.622 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:34:25.622 INFO:teuthology.orchestra.run.gibba012.stdout: journal zero on create = true 2021-02-10T11:34:25.622 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:34:25.622 INFO:teuthology.orchestra.run.gibba012.stdout: filestore ondisk finisher threads = 3 2021-02-10T11:34:25.623 INFO:teuthology.orchestra.run.gibba012.stdout: filestore apply finisher threads = 3 2021-02-10T11:34:25.623 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:34:25.623 INFO:teuthology.orchestra.run.gibba012.stdout: bdev debug aio = true 2021-02-10T11:34:25.623 INFO:teuthology.orchestra.run.gibba012.stdout: osd debug misdirected ops = true 2021-02-10T11:34:25.623 INFO:teuthology.orchestra.run.gibba012.stdout: bdev async discard = True 2021-02-10T11:34:25.623 INFO:teuthology.orchestra.run.gibba012.stdout: bdev enable discard = True 2021-02-10T11:34:25.623 INFO:teuthology.orchestra.run.gibba012.stdout: bluestore allocator = bitmap 2021-02-10T11:34:25.623 INFO:teuthology.orchestra.run.gibba012.stdout: bluestore block size = 96636764160 2021-02-10T11:34:25.623 INFO:teuthology.orchestra.run.gibba012.stdout: bluestore fsck on mount = True 2021-02-10T11:34:25.623 INFO:teuthology.orchestra.run.gibba012.stdout: debug bluefs = 20 2021-02-10T11:34:25.623 INFO:teuthology.orchestra.run.gibba012.stdout: debug bluestore = 20 2021-02-10T11:34:25.623 INFO:teuthology.orchestra.run.gibba012.stdout: debug filestore = 20 2021-02-10T11:34:25.623 INFO:teuthology.orchestra.run.gibba012.stdout: debug journal = 20 2021-02-10T11:34:25.624 INFO:teuthology.orchestra.run.gibba012.stdout: debug ms = 20 2021-02-10T11:34:25.624 INFO:teuthology.orchestra.run.gibba012.stdout: debug osd = 25 2021-02-10T11:34:25.624 INFO:teuthology.orchestra.run.gibba012.stdout: debug rocksdb = 10 2021-02-10T11:34:25.624 INFO:teuthology.orchestra.run.gibba012.stdout: mon osd backfillfull_ratio = 0.85 2021-02-10T11:34:25.624 INFO:teuthology.orchestra.run.gibba012.stdout: mon osd full ratio = 0.9 2021-02-10T11:34:25.624 INFO:teuthology.orchestra.run.gibba012.stdout: mon osd nearfull ratio = 0.8 2021-02-10T11:34:25.624 INFO:teuthology.orchestra.run.gibba012.stdout: osd failsafe full ratio = 0.95 2021-02-10T11:34:25.624 INFO:teuthology.orchestra.run.gibba012.stdout: osd objectstore = bluestore 2021-02-10T11:34:25.624 INFO:teuthology.orchestra.run.gibba012.stdout: osd op complaint time = 180 2021-02-10T11:34:25.624 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:34:25.624 INFO:teuthology.orchestra.run.gibba012.stdout:[mgr] 2021-02-10T11:34:25.624 INFO:teuthology.orchestra.run.gibba012.stdout: debug ms = 1 2021-02-10T11:34:25.624 INFO:teuthology.orchestra.run.gibba012.stdout: debug mgr = 20 2021-02-10T11:34:25.625 INFO:teuthology.orchestra.run.gibba012.stdout: debug mon = 20 2021-02-10T11:34:25.625 INFO:teuthology.orchestra.run.gibba012.stdout: debug auth = 20 2021-02-10T11:34:25.625 INFO:teuthology.orchestra.run.gibba012.stdout: mon reweight min pgs per osd = 4 2021-02-10T11:34:25.625 INFO:teuthology.orchestra.run.gibba012.stdout: mon reweight min bytes per osd = 10 2021-02-10T11:34:25.625 INFO:teuthology.orchestra.run.gibba012.stdout: debug client = 10 2021-02-10T11:34:25.625 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:34:25.625 INFO:teuthology.orchestra.run.gibba012.stdout:[mon] 2021-02-10T11:34:25.625 INFO:teuthology.orchestra.run.gibba012.stdout: debug ms = 1 2021-02-10T11:34:25.625 INFO:teuthology.orchestra.run.gibba012.stdout: debug mon = 20 2021-02-10T11:34:25.625 INFO:teuthology.orchestra.run.gibba012.stdout: debug paxos = 20 2021-02-10T11:34:25.625 INFO:teuthology.orchestra.run.gibba012.stdout: debug auth = 20 2021-02-10T11:34:25.625 INFO:teuthology.orchestra.run.gibba012.stdout: mon data avail warn = 5 2021-02-10T11:34:25.625 INFO:teuthology.orchestra.run.gibba012.stdout: mon mgr mkfs grace = 240 2021-02-10T11:34:25.626 INFO:teuthology.orchestra.run.gibba012.stdout: mon reweight min pgs per osd = 4 2021-02-10T11:34:25.626 INFO:teuthology.orchestra.run.gibba012.stdout: mon osd reporter subtree level = osd 2021-02-10T11:34:25.626 INFO:teuthology.orchestra.run.gibba012.stdout: mon osd prime pg temp = true 2021-02-10T11:34:25.626 INFO:teuthology.orchestra.run.gibba012.stdout: mon reweight min bytes per osd = 10 2021-02-10T11:34:25.626 INFO:teuthology.orchestra.run.gibba012.stdout: mon op complaint time = 120 2021-02-10T11:34:25.626 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:34:25.626 INFO:teuthology.orchestra.run.gibba012.stdout:[client] 2021-02-10T11:34:25.626 INFO:teuthology.orchestra.run.gibba012.stdout: rgw cache enabled = true 2021-02-10T11:34:25.626 INFO:teuthology.orchestra.run.gibba012.stdout: rgw enable ops log = true 2021-02-10T11:34:25.626 INFO:teuthology.orchestra.run.gibba012.stdout: rgw enable usage log = true 2021-02-10T11:34:25.626 INFO:teuthology.orchestra.run.gibba012.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2021-02-10T11:34:25.626 INFO:teuthology.orchestra.run.gibba012.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2021-02-10T11:34:25.626 INFO:teuthology.orchestra.run.gibba012.stdout: client mount timeout = 600 2021-02-10T11:34:25.627 INFO:teuthology.orchestra.run.gibba012.stdout: debug client = 20 2021-02-10T11:34:25.627 INFO:teuthology.orchestra.run.gibba012.stdout: debug ms = 1 2021-02-10T11:34:25.627 INFO:teuthology.orchestra.run.gibba012.stdout: pid file = "" 2021-02-10T11:34:25.627 INFO:teuthology.orchestra.run.gibba012.stdout:[mon.a] 2021-02-10T11:34:25.627 INFO:teuthology.orchestra.run.gibba012.stdout:[mon.b] 2021-02-10T11:34:25.627 INFO:teuthology.orchestra.run.gibba012.stdout:[mon.c] 2021-02-10T11:34:25.627 INFO:teuthology.orchestra.run.gibba012.stdout:[mds.a] 2021-02-10T11:34:25.627 INFO:teuthology.orchestra.run.gibba012.stdout:[mds.b] 2021-02-10T11:34:25.627 INFO:teuthology.orchestra.run.gibba012.stdout:[mds] 2021-02-10T11:34:25.627 INFO:teuthology.orchestra.run.gibba012.stdout: debug mds = 20 2021-02-10T11:34:25.627 INFO:teuthology.orchestra.run.gibba012.stdout: debug ms = 1 2021-02-10T11:34:25.627 INFO:teuthology.orchestra.run.gibba012.stdout: mds bal frag = True 2021-02-10T11:34:25.627 INFO:teuthology.orchestra.run.gibba012.stdout: mds bal fragment size max = 10000 2021-02-10T11:34:25.628 INFO:teuthology.orchestra.run.gibba012.stdout: mds bal merge size = 5 2021-02-10T11:34:25.628 INFO:teuthology.orchestra.run.gibba012.stdout: mds bal split bits = 3 2021-02-10T11:34:25.628 INFO:teuthology.orchestra.run.gibba012.stdout: mds bal split size = 100 2021-02-10T11:34:25.628 INFO:teuthology.orchestra.run.gibba012.stdout: mds debug frag = True 2021-02-10T11:34:25.628 INFO:teuthology.orchestra.run.gibba012.stdout: mds debug scatterstat = True 2021-02-10T11:34:25.628 INFO:teuthology.orchestra.run.gibba012.stdout: mds op complaint time = 180 2021-02-10T11:34:25.628 INFO:teuthology.orchestra.run.gibba012.stdout: mds verify scatter = True 2021-02-10T11:34:25.628 INFO:teuthology.orchestra.run.gibba012.stdout: osd op complaint time = 180 2021-02-10T11:34:25.628 INFO:teuthology.orchestra.run.gibba012.stdout:[client.alice] 2021-02-10T11:34:25.629 INFO:teuthology.orchestra.run.gibba020.stdout:[global] 2021-02-10T11:34:25.629 INFO:teuthology.orchestra.run.gibba020.stdout: chdir = "" 2021-02-10T11:34:25.629 INFO:teuthology.orchestra.run.gibba020.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2021-02-10T11:34:25.629 INFO:teuthology.orchestra.run.gibba020.stdout: auth supported = cephx 2021-02-10T11:34:25.629 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:34:25.630 INFO:teuthology.orchestra.run.gibba020.stdout: filestore xattr use omap = true 2021-02-10T11:34:25.630 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:34:25.630 INFO:teuthology.orchestra.run.gibba020.stdout: mon clock drift allowed = 1.000 2021-02-10T11:34:25.630 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:34:25.630 INFO:teuthology.orchestra.run.gibba020.stdout: osd crush chooseleaf type = 0 2021-02-10T11:34:25.630 INFO:teuthology.orchestra.run.gibba020.stdout: auth debug = true 2021-02-10T11:34:25.630 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:34:25.630 INFO:teuthology.orchestra.run.gibba020.stdout: ms die on old message = true 2021-02-10T11:34:25.630 INFO:teuthology.orchestra.run.gibba020.stdout: ms die on bug = true 2021-02-10T11:34:25.630 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:34:25.630 INFO:teuthology.orchestra.run.gibba020.stdout: mon max pg per osd = 10000 # >= luminous 2021-02-10T11:34:25.630 INFO:teuthology.orchestra.run.gibba020.stdout: mon pg warn max object skew = 0 2021-02-10T11:34:25.630 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:34:25.631 INFO:teuthology.orchestra.run.gibba020.stdout: osd pool default size = 2 2021-02-10T11:34:25.631 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:34:25.631 INFO:teuthology.orchestra.run.gibba020.stdout: mon osd allow primary affinity = true 2021-02-10T11:34:25.631 INFO:teuthology.orchestra.run.gibba020.stdout: mon osd allow pg remap = true 2021-02-10T11:34:25.631 INFO:teuthology.orchestra.run.gibba020.stdout: mon warn on legacy crush tunables = false 2021-02-10T11:34:25.631 INFO:teuthology.orchestra.run.gibba020.stdout: mon warn on crush straw calc version zero = false 2021-02-10T11:34:25.631 INFO:teuthology.orchestra.run.gibba020.stdout: mon warn on no sortbitwise = false 2021-02-10T11:34:25.631 INFO:teuthology.orchestra.run.gibba020.stdout: mon warn on osd down out interval zero = false 2021-02-10T11:34:25.631 INFO:teuthology.orchestra.run.gibba020.stdout: mon warn on too few osds = false 2021-02-10T11:34:25.631 INFO:teuthology.orchestra.run.gibba020.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2021-02-10T11:34:25.631 INFO:teuthology.orchestra.run.gibba020.stdout: mon_warn_on_pool_no_redundancy = false 2021-02-10T11:34:25.631 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:34:25.631 INFO:teuthology.orchestra.run.gibba020.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2021-02-10T11:34:25.632 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:34:25.632 INFO:teuthology.orchestra.run.gibba020.stdout: osd default data pool replay window = 5 2021-02-10T11:34:25.632 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:34:25.632 INFO:teuthology.orchestra.run.gibba020.stdout: mon allow pool delete = true 2021-02-10T11:34:25.632 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:34:25.632 INFO:teuthology.orchestra.run.gibba020.stdout: mon cluster log file level = debug 2021-02-10T11:34:25.632 INFO:teuthology.orchestra.run.gibba020.stdout: debug asserts on shutdown = true 2021-02-10T11:34:25.632 INFO:teuthology.orchestra.run.gibba020.stdout: mon health detail to clog = false 2021-02-10T11:34:25.632 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:34:25.632 INFO:teuthology.orchestra.run.gibba020.stdout: # we see this fail in qa on *nautilus*; bump up retries 2021-02-10T11:34:25.632 INFO:teuthology.orchestra.run.gibba020.stdout: mon_client_directed_command_retry = 4 2021-02-10T11:34:25.632 INFO:teuthology.orchestra.run.gibba020.stdout: mon host = "172.21.2.112,172.21.2.120,[v2:172.21.2.120:3301,v1:172.21.2.120:6790]" 2021-02-10T11:34:25.632 INFO:teuthology.orchestra.run.gibba020.stdout: fsid = f14f004e-0699-4584-ad81-3754258493de 2021-02-10T11:34:25.632 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:34:25.633 INFO:teuthology.orchestra.run.gibba020.stdout:[osd] 2021-02-10T11:34:25.633 INFO:teuthology.orchestra.run.gibba020.stdout: osd journal size = 100 2021-02-10T11:34:25.633 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:34:25.633 INFO:teuthology.orchestra.run.gibba020.stdout: osd scrub load threshold = 5.0 2021-02-10T11:34:25.633 INFO:teuthology.orchestra.run.gibba020.stdout: osd scrub max interval = 600 2021-02-10T11:34:25.633 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:34:25.633 INFO:teuthology.orchestra.run.gibba020.stdout: osd recover clone overlap = true 2021-02-10T11:34:25.633 INFO:teuthology.orchestra.run.gibba020.stdout: osd recovery max chunk = 1048576 2021-02-10T11:34:25.633 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:34:25.633 INFO:teuthology.orchestra.run.gibba020.stdout: osd debug shutdown = true 2021-02-10T11:34:25.633 INFO:teuthology.orchestra.run.gibba020.stdout: osd debug op order = true 2021-02-10T11:34:25.633 INFO:teuthology.orchestra.run.gibba020.stdout: osd debug verify stray on activate = true 2021-02-10T11:34:25.634 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:34:25.634 INFO:teuthology.orchestra.run.gibba020.stdout: osd open classes on start = true 2021-02-10T11:34:25.634 INFO:teuthology.orchestra.run.gibba020.stdout: osd debug pg log writeout = true 2021-02-10T11:34:25.634 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:34:25.634 INFO:teuthology.orchestra.run.gibba020.stdout: osd deep scrub update digest min age = 30 2021-02-10T11:34:25.634 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:34:25.634 INFO:teuthology.orchestra.run.gibba020.stdout: osd map max advance = 10 2021-02-10T11:34:25.634 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:34:25.634 INFO:teuthology.orchestra.run.gibba020.stdout: journal zero on create = true 2021-02-10T11:34:25.634 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:34:25.634 INFO:teuthology.orchestra.run.gibba020.stdout: filestore ondisk finisher threads = 3 2021-02-10T11:34:25.634 INFO:teuthology.orchestra.run.gibba020.stdout: filestore apply finisher threads = 3 2021-02-10T11:34:25.635 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:34:25.635 INFO:teuthology.orchestra.run.gibba020.stdout: bdev debug aio = true 2021-02-10T11:34:25.635 INFO:teuthology.orchestra.run.gibba020.stdout: osd debug misdirected ops = true 2021-02-10T11:34:25.635 INFO:teuthology.orchestra.run.gibba020.stdout: bdev async discard = True 2021-02-10T11:34:25.635 INFO:teuthology.orchestra.run.gibba020.stdout: bdev enable discard = True 2021-02-10T11:34:25.635 INFO:teuthology.orchestra.run.gibba020.stdout: bluestore allocator = bitmap 2021-02-10T11:34:25.635 INFO:teuthology.orchestra.run.gibba020.stdout: bluestore block size = 96636764160 2021-02-10T11:34:25.635 INFO:teuthology.orchestra.run.gibba020.stdout: bluestore fsck on mount = True 2021-02-10T11:34:25.635 INFO:teuthology.orchestra.run.gibba020.stdout: debug bluefs = 20 2021-02-10T11:34:25.635 INFO:teuthology.orchestra.run.gibba020.stdout: debug bluestore = 20 2021-02-10T11:34:25.635 INFO:teuthology.orchestra.run.gibba020.stdout: debug filestore = 20 2021-02-10T11:34:25.635 INFO:teuthology.orchestra.run.gibba020.stdout: debug journal = 20 2021-02-10T11:34:25.635 INFO:teuthology.orchestra.run.gibba020.stdout: debug ms = 20 2021-02-10T11:34:25.636 INFO:teuthology.orchestra.run.gibba020.stdout: debug osd = 25 2021-02-10T11:34:25.636 INFO:teuthology.orchestra.run.gibba020.stdout: debug rocksdb = 10 2021-02-10T11:34:25.636 INFO:teuthology.orchestra.run.gibba020.stdout: mon osd backfillfull_ratio = 0.85 2021-02-10T11:34:25.636 INFO:teuthology.orchestra.run.gibba020.stdout: mon osd full ratio = 0.9 2021-02-10T11:34:25.636 INFO:teuthology.orchestra.run.gibba020.stdout: mon osd nearfull ratio = 0.8 2021-02-10T11:34:25.636 INFO:teuthology.orchestra.run.gibba020.stdout: osd failsafe full ratio = 0.95 2021-02-10T11:34:25.636 INFO:teuthology.orchestra.run.gibba020.stdout: osd objectstore = bluestore 2021-02-10T11:34:25.636 INFO:teuthology.orchestra.run.gibba020.stdout: osd op complaint time = 180 2021-02-10T11:34:25.636 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:34:25.636 INFO:teuthology.orchestra.run.gibba020.stdout:[mgr] 2021-02-10T11:34:25.636 INFO:teuthology.orchestra.run.gibba020.stdout: debug ms = 1 2021-02-10T11:34:25.636 INFO:teuthology.orchestra.run.gibba020.stdout: debug mgr = 20 2021-02-10T11:34:25.636 INFO:teuthology.orchestra.run.gibba020.stdout: debug mon = 20 2021-02-10T11:34:25.637 INFO:teuthology.orchestra.run.gibba020.stdout: debug auth = 20 2021-02-10T11:34:25.637 INFO:teuthology.orchestra.run.gibba020.stdout: mon reweight min pgs per osd = 4 2021-02-10T11:34:25.637 INFO:teuthology.orchestra.run.gibba020.stdout: mon reweight min bytes per osd = 10 2021-02-10T11:34:25.637 INFO:teuthology.orchestra.run.gibba020.stdout: debug client = 10 2021-02-10T11:34:25.637 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:34:25.637 INFO:teuthology.orchestra.run.gibba020.stdout:[mon] 2021-02-10T11:34:25.637 INFO:teuthology.orchestra.run.gibba020.stdout: debug ms = 1 2021-02-10T11:34:25.637 INFO:teuthology.orchestra.run.gibba020.stdout: debug mon = 20 2021-02-10T11:34:25.637 INFO:teuthology.orchestra.run.gibba020.stdout: debug paxos = 20 2021-02-10T11:34:25.637 INFO:teuthology.orchestra.run.gibba020.stdout: debug auth = 20 2021-02-10T11:34:25.637 INFO:teuthology.orchestra.run.gibba020.stdout: mon data avail warn = 5 2021-02-10T11:34:25.637 INFO:teuthology.orchestra.run.gibba020.stdout: mon mgr mkfs grace = 240 2021-02-10T11:34:25.637 INFO:teuthology.orchestra.run.gibba020.stdout: mon reweight min pgs per osd = 4 2021-02-10T11:34:25.638 INFO:teuthology.orchestra.run.gibba020.stdout: mon osd reporter subtree level = osd 2021-02-10T11:34:25.638 INFO:teuthology.orchestra.run.gibba020.stdout: mon osd prime pg temp = true 2021-02-10T11:34:25.638 INFO:teuthology.orchestra.run.gibba020.stdout: mon reweight min bytes per osd = 10 2021-02-10T11:34:25.638 INFO:teuthology.orchestra.run.gibba020.stdout: mon op complaint time = 120 2021-02-10T11:34:25.638 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:34:25.638 INFO:teuthology.orchestra.run.gibba020.stdout:[client] 2021-02-10T11:34:25.638 INFO:teuthology.orchestra.run.gibba020.stdout: rgw cache enabled = true 2021-02-10T11:34:25.638 INFO:teuthology.orchestra.run.gibba020.stdout: rgw enable ops log = true 2021-02-10T11:34:25.638 INFO:teuthology.orchestra.run.gibba020.stdout: rgw enable usage log = true 2021-02-10T11:34:25.638 INFO:teuthology.orchestra.run.gibba020.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2021-02-10T11:34:25.638 INFO:teuthology.orchestra.run.gibba020.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2021-02-10T11:34:25.638 INFO:teuthology.orchestra.run.gibba020.stdout: client mount timeout = 600 2021-02-10T11:34:25.638 INFO:teuthology.orchestra.run.gibba020.stdout: debug client = 20 2021-02-10T11:34:25.639 INFO:teuthology.orchestra.run.gibba020.stdout: debug ms = 1 2021-02-10T11:34:25.639 INFO:teuthology.orchestra.run.gibba020.stdout: pid file = "" 2021-02-10T11:34:25.639 INFO:teuthology.orchestra.run.gibba020.stdout:[mon.a] 2021-02-10T11:34:25.639 INFO:teuthology.orchestra.run.gibba020.stdout:[mon.b] 2021-02-10T11:34:25.639 INFO:teuthology.orchestra.run.gibba020.stdout:[mon.c] 2021-02-10T11:34:25.639 INFO:teuthology.orchestra.run.gibba020.stdout:[mds.a] 2021-02-10T11:34:25.639 INFO:teuthology.orchestra.run.gibba020.stdout:[mds.b] 2021-02-10T11:34:25.639 INFO:teuthology.orchestra.run.gibba020.stdout:[mds] 2021-02-10T11:34:25.639 INFO:teuthology.orchestra.run.gibba020.stdout: debug mds = 20 2021-02-10T11:34:25.639 INFO:teuthology.orchestra.run.gibba020.stdout: debug ms = 1 2021-02-10T11:34:25.639 INFO:teuthology.orchestra.run.gibba020.stdout: mds bal frag = True 2021-02-10T11:34:25.639 INFO:teuthology.orchestra.run.gibba020.stdout: mds bal fragment size max = 10000 2021-02-10T11:34:25.639 INFO:teuthology.orchestra.run.gibba020.stdout: mds bal merge size = 5 2021-02-10T11:34:25.639 INFO:teuthology.orchestra.run.gibba020.stdout: mds bal split bits = 3 2021-02-10T11:34:25.640 INFO:teuthology.orchestra.run.gibba020.stdout: mds bal split size = 100 2021-02-10T11:34:25.640 INFO:teuthology.orchestra.run.gibba020.stdout: mds debug frag = True 2021-02-10T11:34:25.640 INFO:teuthology.orchestra.run.gibba020.stdout: mds debug scatterstat = True 2021-02-10T11:34:25.640 INFO:teuthology.orchestra.run.gibba020.stdout: mds op complaint time = 180 2021-02-10T11:34:25.640 INFO:teuthology.orchestra.run.gibba020.stdout: mds verify scatter = True 2021-02-10T11:34:25.640 INFO:teuthology.orchestra.run.gibba020.stdout: osd op complaint time = 180 2021-02-10T11:34:25.640 INFO:teuthology.orchestra.run.gibba020.stdout:[client.alice] 2021-02-10T11:34:25.641 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config rm client.alice debug_client 2021-02-10T11:34:26.078 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.075 7f9b4796e700 1 Processor -- start 2021-02-10T11:34:26.078 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.075 7f9b4796e700 1 -- start start 2021-02-10T11:34:26.078 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.075 7f9b4796e700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f9b400d1c70 0x7f9b400f20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:26.079 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.075 7f9b4796e700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b400f0b10 con 0x7f9b400d1870 2021-02-10T11:34:26.079 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.075 7f9b4796e700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b401b0be0 con 0x7f9b400d0340 2021-02-10T11:34:26.079 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.075 7f9b4796e700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b400e5ed0 con 0x7f9b400d1c70 2021-02-10T11:34:26.079 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.076 7f9b45f8b700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f9b400d1c70 0x7f9b400f20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:26.079 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.076 7f9b45f8b700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f9b400d1c70 0x7f9b400f20a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:50346/0 (socket says 172.21.2.112:50346) 2021-02-10T11:34:26.079 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.076 7f9b45f8b700 1 -- 172.21.2.112:0/878921506 learned_addr learned my addr 172.21.2.112:0/878921506 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:34:26.079 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.076 7f9b45f8b700 1 -- 172.21.2.112:0/878921506 >> v1:172.21.2.120:6789/0 conn(0x7f9b400d0340 legacy=0x7f9b400ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:26.079 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.076 7f9b45f8b700 1 -- 172.21.2.112:0/878921506 >> v1:172.21.2.112:6789/0 conn(0x7f9b400d1870 legacy=0x7f9b401b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:26.079 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.076 7f9b33fff700 1 -- 172.21.2.112:0/878921506 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 517745682 0 0) 0x7f9b3c000ce0 con 0x7f9b400d1870 2021-02-10T11:34:26.079 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.076 7f9b45f8b700 1 -- 172.21.2.112:0/878921506 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b400efe30 con 0x7f9b400d1c70 2021-02-10T11:34:26.079 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.076 7f9b45f8b700 1 --2- 172.21.2.112:0/878921506 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9b400d1c70 0x7f9b400f20a0 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 rx=0x7f9b38006f00 tx=0x7f9b38009e90).ready entity=mon.0 client_cookie=daf45426d1b3b95b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:26.079 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.076 7f9b33fff700 1 -- 172.21.2.112:0/878921506 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9b3800b300 con 0x7f9b400d1c70 2021-02-10T11:34:26.079 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.076 7f9b33fff700 1 -- 172.21.2.112:0/878921506 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b3800b640 con 0x7f9b400d1c70 2021-02-10T11:34:26.080 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.076 7f9b33fff700 1 -- 172.21.2.112:0/878921506 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9b3800baa0 con 0x7f9b400d1c70 2021-02-10T11:34:26.080 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.076 7f9b4796e700 1 -- 172.21.2.112:0/878921506 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9b400d1c70 msgr2=0x7f9b400f20a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:26.080 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.076 7f9b4796e700 1 --2- 172.21.2.112:0/878921506 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9b400d1c70 0x7f9b400f20a0 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 rx=0x7f9b38006f00 tx=0x7f9b38009e90).stop 2021-02-10T11:34:26.080 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.077 7f9b4796e700 1 -- 172.21.2.112:0/878921506 shutdown_connections 2021-02-10T11:34:26.080 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.077 7f9b4796e700 1 --2- 172.21.2.112:0/878921506 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9b400d1c70 0x7f9b400f20a0 unknown :-1 s=CLOSED pgs=1338 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:26.080 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.077 7f9b4796e700 1 -- 172.21.2.112:0/878921506 >> 172.21.2.112:0/878921506 conn(0x7f9b400ce450 msgr2=0x7f9b400d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:26.080 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.077 7f9b4796e700 1 -- 172.21.2.112:0/878921506 shutdown_connections 2021-02-10T11:34:26.080 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.077 7f9b4796e700 1 -- 172.21.2.112:0/878921506 wait complete. 2021-02-10T11:34:26.081 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.092 7f9b4796e700 1 Processor -- start 2021-02-10T11:34:26.081 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.092 7f9b4796e700 1 -- start start 2021-02-10T11:34:26.082 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.092 7f9b4796e700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9b400bbcf0 0x7f9b401a4400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:26.082 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.092 7f9b4796e700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9b401a4a20 0x7f9b400bb020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:26.082 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.092 7f9b4796e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9b400ae5e0 0x7f9b400bb640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:26.082 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.092 7f9b4796e700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b400bc2f0 con 0x7f9b400ae5e0 2021-02-10T11:34:26.082 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.092 7f9b4796e700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b40095700 con 0x7f9b400bbcf0 2021-02-10T11:34:26.082 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.092 7f9b4796e700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b40095920 con 0x7f9b401a4a20 2021-02-10T11:34:26.082 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.092 7f9b45f8b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9b400ae5e0 0x7f9b400bb640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:26.082 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.092 7f9b45f8b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9b400ae5e0 0x7f9b400bb640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:50356/0 (socket says 172.21.2.112:50356) 2021-02-10T11:34:26.082 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.092 7f9b45f8b700 1 -- 172.21.2.112:0/315249151 learned_addr learned my addr 172.21.2.112:0/315249151 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:34:26.082 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.093 7f9b45f8b700 1 -- 172.21.2.112:0/315249151 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9b401a4a20 msgr2=0x7f9b400bb020 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:34:26.082 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.093 7f9b45f8b700 1 --2- 172.21.2.112:0/315249151 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9b401a4a20 0x7f9b400bb020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:26.082 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.093 7f9b45f8b700 1 -- 172.21.2.112:0/315249151 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9b400bbcf0 msgr2=0x7f9b401a4400 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:34:26.083 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.093 7f9b45f8b700 1 --2- 172.21.2.112:0/315249151 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9b400bbcf0 0x7f9b401a4400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:26.083 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.093 7f9b45f8b700 1 -- 172.21.2.112:0/315249151 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b40095b40 con 0x7f9b400ae5e0 2021-02-10T11:34:26.083 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.093 7f9b45f8b700 1 --2- 172.21.2.112:0/315249151 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9b400ae5e0 0x7f9b400bb640 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 rx=0x7f9b3800b940 tx=0x7f9b3800bce0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:26.083 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.093 7f9b31ffb700 1 -- 172.21.2.112:0/315249151 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9b38000f90 con 0x7f9b400ae5e0 2021-02-10T11:34:26.083 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.093 7f9b31ffb700 1 -- 172.21.2.112:0/315249151 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b38003030 con 0x7f9b400ae5e0 2021-02-10T11:34:26.083 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.093 7f9b4796e700 1 -- 172.21.2.112:0/315249151 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b40095b40 con 0x7f9b400ae5e0 2021-02-10T11:34:26.083 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.093 7f9b4796e700 1 -- 172.21.2.112:0/315249151 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b400960d0 con 0x7f9b400ae5e0 2021-02-10T11:34:26.083 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.093 7f9b31ffb700 1 -- 172.21.2.112:0/315249151 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9b38001460 con 0x7f9b400ae5e0 2021-02-10T11:34:26.083 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.094 7f9b31ffb700 1 -- 172.21.2.112:0/315249151 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9b38001270 con 0x7f9b400ae5e0 2021-02-10T11:34:26.083 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.094 7f9b31ffb700 1 --2- 172.21.2.112:0/315249151 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9b20018730 0x7f9b2001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:26.083 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.094 7f9b31ffb700 1 -- 172.21.2.112:0/315249151 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f9b38032b40 con 0x7f9b400ae5e0 2021-02-10T11:34:26.083 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.094 7f9b4578a700 1 --2- 172.21.2.112:0/315249151 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9b20018730 0x7f9b2001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:26.083 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.095 7f9b4578a700 1 --2- 172.21.2.112:0/315249151 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9b20018730 0x7f9b2001ac10 crc :-1 s=READY pgs=660 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:26.112 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.125 7f9b4796e700 1 -- 172.21.2.112:0/315249151 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b400ce450 con 0x7f9b400ae5e0 2021-02-10T11:34:26.114 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.127 7f9b31ffb700 1 -- 172.21.2.112:0/315249151 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9b38000f90 con 0x7f9b400ae5e0 2021-02-10T11:34:26.262 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.275 7f9b4796e700 1 -- 172.21.2.112:0/315249151 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "config rm", "who": "client.alice", "name": "debug_client"} v 0) v1 -- 0x7f9b400ce450 con 0x7f9b400ae5e0 2021-02-10T11:34:26.266 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.280 7f9b31ffb700 1 -- 172.21.2.112:0/315249151 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "client.alice", "name": "debug_client"}]=0 v11) v1 ==== 104+0+0 (secure 0 0 0) 0x7f9b38001270 con 0x7f9b400ae5e0 2021-02-10T11:34:26.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.284 7f9b4796e700 1 -- 172.21.2.112:0/315249151 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9b20018730 msgr2=0x7f9b2001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:26.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.284 7f9b4796e700 1 --2- 172.21.2.112:0/315249151 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9b20018730 0x7f9b2001ac10 crc :-1 s=READY pgs=660 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:26.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.284 7f9b4796e700 1 -- 172.21.2.112:0/315249151 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9b400ae5e0 msgr2=0x7f9b400bb640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:26.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.284 7f9b4796e700 1 --2- 172.21.2.112:0/315249151 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9b400ae5e0 0x7f9b400bb640 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 rx=0x7f9b3800b940 tx=0x7f9b3800bce0).stop 2021-02-10T11:34:26.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.284 7f9b4796e700 1 -- 172.21.2.112:0/315249151 shutdown_connections 2021-02-10T11:34:26.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.284 7f9b4796e700 1 --2- 172.21.2.112:0/315249151 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9b20018730 0x7f9b2001ac10 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:26.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.284 7f9b4796e700 1 --2- 172.21.2.112:0/315249151 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9b400ae5e0 0x7f9b400bb640 unknown :-1 s=CLOSED pgs=1339 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:26.272 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.284 7f9b4796e700 1 --2- 172.21.2.112:0/315249151 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9b401a4a20 0x7f9b400bb020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:26.272 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.284 7f9b4796e700 1 --2- 172.21.2.112:0/315249151 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9b400bbcf0 0x7f9b401a4400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:26.272 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.284 7f9b4796e700 1 -- 172.21.2.112:0/315249151 >> 172.21.2.112:0/315249151 conn(0x7f9b400a6620 msgr2=0x7f9b400b17a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:26.272 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.284 7f9b4796e700 1 -- 172.21.2.112:0/315249151 shutdown_connections 2021-02-10T11:34:26.272 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.284 7f9b4796e700 1 -- 172.21.2.112:0/315249151 wait complete. 2021-02-10T11:34:26.279 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config rm client.alice debug_objecter 2021-02-10T11:34:26.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.705 7f6f8ffce700 1 Processor -- start 2021-02-10T11:34:26.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.705 7f6f8ffce700 1 -- start start 2021-02-10T11:34:26.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.706 7f6f8ffce700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6f880d1870 0x7f6f881b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:26.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.706 7f6f8ffce700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f6f881b0a10 0x7f6f880cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:26.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.706 7f6f8ffce700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f880f0b10 con 0x7f6f880d1c70 2021-02-10T11:34:26.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.706 7f6f8ffce700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f880fe230 con 0x7f6f881b0a10 2021-02-10T11:34:26.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.706 7f6f8ffce700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f880e5ed0 con 0x7f6f880d1870 2021-02-10T11:34:26.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.706 7f6f8e5eb700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f6f880d1c70 0x7f6f880cf950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:49822/0 (socket says 172.21.2.112:49822) 2021-02-10T11:34:26.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.706 7f6f8e5eb700 1 -- 172.21.2.112:0/553205219 learned_addr learned my addr 172.21.2.112:0/553205219 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:34:26.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.706 7f6f8d5e9700 1 --2- 172.21.2.112:0/553205219 >> v2:172.21.2.112:3300/0 conn(0x7f6f881b0a10 0x7f6f880cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:26.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.706 7f6f8d5e9700 1 -- 172.21.2.112:0/553205219 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6f880d1870 msgr2=0x7f6f881b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:26.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.706 7f6f8ddea700 1 --2- 172.21.2.112:0/553205219 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6f880d1870 0x7f6f881b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:26.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.706 7f6f8d5e9700 1 --2- 172.21.2.112:0/553205219 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6f880d1870 0x7f6f881b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:26.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.706 7f6f8d5e9700 1 -- 172.21.2.112:0/553205219 >> v1:172.21.2.112:6789/0 conn(0x7f6f880d1c70 legacy=0x7f6f880cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:26.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.706 7f6f8cde8700 1 -- 172.21.2.112:0/553205219 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2695001614 0 0) 0x7f6f80001500 con 0x7f6f880d1c70 2021-02-10T11:34:26.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.706 7f6f8d5e9700 1 -- 172.21.2.112:0/553205219 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f880f0650 con 0x7f6f881b0a10 2021-02-10T11:34:26.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.706 7f6f8ddea700 1 --2- 172.21.2.112:0/553205219 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6f880d1870 0x7f6f881b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:34:26.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.707 7f6f8d5e9700 1 --2- 172.21.2.112:0/553205219 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6f881b0a10 0x7f6f880cf330 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 rx=0x7f6f7c006650 tx=0x7f6f7c009a80).ready entity=mon.0 client_cookie=e6fe377edbe58524 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:26.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.707 7f6f8cde8700 1 -- 172.21.2.112:0/553205219 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6f7c0009b0 con 0x7f6f881b0a10 2021-02-10T11:34:26.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.707 7f6f8cde8700 1 -- 172.21.2.112:0/553205219 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f7c00c0c0 con 0x7f6f881b0a10 2021-02-10T11:34:26.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.707 7f6f8cde8700 1 -- 172.21.2.112:0/553205219 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6f7c00c5e0 con 0x7f6f881b0a10 2021-02-10T11:34:26.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.707 7f6f8ffce700 1 -- 172.21.2.112:0/553205219 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6f881b0a10 msgr2=0x7f6f880cf330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:26.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.707 7f6f8ffce700 1 --2- 172.21.2.112:0/553205219 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6f881b0a10 0x7f6f880cf330 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 rx=0x7f6f7c006650 tx=0x7f6f7c009a80).stop 2021-02-10T11:34:26.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.708 7f6f8ffce700 1 -- 172.21.2.112:0/553205219 shutdown_connections 2021-02-10T11:34:26.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.708 7f6f8ffce700 1 --2- 172.21.2.112:0/553205219 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6f881b0a10 0x7f6f880cf330 unknown :-1 s=CLOSED pgs=1341 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:26.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.708 7f6f8ffce700 1 --2- 172.21.2.112:0/553205219 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6f880d1870 0x7f6f881b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:26.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.708 7f6f8ffce700 1 -- 172.21.2.112:0/553205219 >> 172.21.2.112:0/553205219 conn(0x7f6f880ce450 msgr2=0x7f6f880d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:26.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.708 7f6f8ffce700 1 -- 172.21.2.112:0/553205219 shutdown_connections 2021-02-10T11:34:26.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.708 7f6f8ffce700 1 -- 172.21.2.112:0/553205219 wait complete. 2021-02-10T11:34:26.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.722 7f6f8ffce700 1 Processor -- start 2021-02-10T11:34:26.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.722 7f6f8ffce700 1 -- start start 2021-02-10T11:34:26.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.723 7f6f8ffce700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6f881a06c0 0x7f6f880ebe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:26.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.723 7f6f8ffce700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6f880ec490 0x7f6f880c30f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:26.714 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.723 7f6f8ffce700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6f880c3710 0x7f6f880bc070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:26.714 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.723 7f6f8ffce700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f880c3b10 con 0x7f6f880c3710 2021-02-10T11:34:26.714 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.723 7f6f8ffce700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f880eca90 con 0x7f6f881a06c0 2021-02-10T11:34:26.714 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.723 7f6f8ffce700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f880bc690 con 0x7f6f880ec490 2021-02-10T11:34:26.714 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.723 7f6f8e5eb700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6f880c3710 0x7f6f880bc070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:26.714 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.723 7f6f8e5eb700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6f880c3710 0x7f6f880bc070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:50368/0 (socket says 172.21.2.112:50368) 2021-02-10T11:34:26.714 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.723 7f6f8e5eb700 1 -- 172.21.2.112:0/4266757083 learned_addr learned my addr 172.21.2.112:0/4266757083 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:34:26.714 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.723 7f6f8e5eb700 1 -- 172.21.2.112:0/4266757083 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6f880ec490 msgr2=0x7f6f880c30f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:26.714 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.723 7f6f8e5eb700 1 --2- 172.21.2.112:0/4266757083 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6f880ec490 0x7f6f880c30f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:26.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.723 7f6f8e5eb700 1 -- 172.21.2.112:0/4266757083 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6f881a06c0 msgr2=0x7f6f880ebe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:26.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.723 7f6f8e5eb700 1 --2- 172.21.2.112:0/4266757083 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6f881a06c0 0x7f6f880ebe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:26.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.723 7f6f8e5eb700 1 -- 172.21.2.112:0/4266757083 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f880bc870 con 0x7f6f880c3710 2021-02-10T11:34:26.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.724 7f6f8e5eb700 1 --2- 172.21.2.112:0/4266757083 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6f880c3710 0x7f6f880bc070 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 rx=0x7f6f80001c10 tx=0x7f6f8000ac60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:26.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.724 7f6f7a7fc700 1 -- 172.21.2.112:0/4266757083 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6f80000f90 con 0x7f6f880c3710 2021-02-10T11:34:26.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.724 7f6f7a7fc700 1 -- 172.21.2.112:0/4266757083 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f800070d0 con 0x7f6f880c3710 2021-02-10T11:34:26.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.724 7f6f7a7fc700 1 -- 172.21.2.112:0/4266757083 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6f80004720 con 0x7f6f880c3710 2021-02-10T11:34:26.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.724 7f6f8ffce700 1 -- 172.21.2.112:0/4266757083 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f880bc870 con 0x7f6f880c3710 2021-02-10T11:34:26.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.724 7f6f8ffce700 1 -- 172.21.2.112:0/4266757083 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f880bc870 con 0x7f6f880c3710 2021-02-10T11:34:26.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.725 7f6f7a7fc700 1 -- 172.21.2.112:0/4266757083 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6f800072b0 con 0x7f6f880c3710 2021-02-10T11:34:26.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.725 7f6f7a7fc700 1 --2- 172.21.2.112:0/4266757083 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6f700186c0 0x7f6f7001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:26.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.725 7f6f7a7fc700 1 -- 172.21.2.112:0/4266757083 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f6f800328e0 con 0x7f6f880c3710 2021-02-10T11:34:26.716 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.725 7f6f8ddea700 1 --2- 172.21.2.112:0/4266757083 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6f700186c0 0x7f6f7001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:26.716 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.726 7f6f8ddea700 1 --2- 172.21.2.112:0/4266757083 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6f700186c0 0x7f6f7001abc0 crc :-1 s=READY pgs=661 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:26.743 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.756 7f6f8ffce700 1 -- 172.21.2.112:0/4266757083 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f880ebc40 con 0x7f6f880c3710 2021-02-10T11:34:26.745 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.758 7f6f7a7fc700 1 -- 172.21.2.112:0/4266757083 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6f80023b50 con 0x7f6f880c3710 2021-02-10T11:34:26.894 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.907 7f6f8ffce700 1 -- 172.21.2.112:0/4266757083 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "config rm", "who": "client.alice", "name": "debug_objecter"} v 0) v1 -- 0x7f6f880f0b10 con 0x7f6f880c3710 2021-02-10T11:34:26.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.911 7f6f7a7fc700 1 -- 172.21.2.112:0/4266757083 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "client.alice", "name": "debug_objecter"}]=0 v12) v1 ==== 106+0+0 (secure 0 0 0) 0x7f6f80023b50 con 0x7f6f880c3710 2021-02-10T11:34:26.903 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.916 7f6f8ffce700 1 -- 172.21.2.112:0/4266757083 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6f700186c0 msgr2=0x7f6f7001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:26.903 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.916 7f6f8ffce700 1 --2- 172.21.2.112:0/4266757083 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6f700186c0 0x7f6f7001abc0 crc :-1 s=READY pgs=661 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:26.903 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.916 7f6f8ffce700 1 -- 172.21.2.112:0/4266757083 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6f880c3710 msgr2=0x7f6f880bc070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:26.904 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.916 7f6f8ffce700 1 --2- 172.21.2.112:0/4266757083 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6f880c3710 0x7f6f880bc070 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 rx=0x7f6f80001c10 tx=0x7f6f8000ac60).stop 2021-02-10T11:34:26.904 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.916 7f6f8ffce700 1 -- 172.21.2.112:0/4266757083 shutdown_connections 2021-02-10T11:34:26.904 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.916 7f6f8ffce700 1 --2- 172.21.2.112:0/4266757083 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6f700186c0 0x7f6f7001abc0 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:26.904 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.916 7f6f8ffce700 1 --2- 172.21.2.112:0/4266757083 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6f880c3710 0x7f6f880bc070 unknown :-1 s=CLOSED pgs=1342 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:26.904 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.916 7f6f8ffce700 1 --2- 172.21.2.112:0/4266757083 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6f880ec490 0x7f6f880c30f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:26.904 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.916 7f6f8ffce700 1 --2- 172.21.2.112:0/4266757083 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6f881a06c0 0x7f6f880ebe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:26.904 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.917 7f6f8ffce700 1 -- 172.21.2.112:0/4266757083 >> 172.21.2.112:0/4266757083 conn(0x7f6f880a6bb0 msgr2=0x7f6f88095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:26.904 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.917 7f6f8ffce700 1 -- 172.21.2.112:0/4266757083 shutdown_connections 2021-02-10T11:34:26.904 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:26.917 7f6f8ffce700 1 -- 172.21.2.112:0/4266757083 wait complete. 2021-02-10T11:34:26.912 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config rm mon mon_allow_pool_delete 2021-02-10T11:34:27.342 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.339 7f1e3050d700 1 Processor -- start 2021-02-10T11:34:27.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.339 7f1e3050d700 1 -- start start 2021-02-10T11:34:27.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.339 7f1e3050d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1e280d0340 0x7f1e280ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:27.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.339 7f1e3050d700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e280f0b10 con 0x7f1e280d1870 2021-02-10T11:34:27.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.339 7f1e3050d700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e281b0be0 con 0x7f1e280d1c70 2021-02-10T11:34:27.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.339 7f1e3050d700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e280e5ed0 con 0x7f1e280d0340 2021-02-10T11:34:27.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.340 7f1e2e329700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f1e280d1870 0x7f1e281b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:49836/0 (socket says 172.21.2.112:49836) 2021-02-10T11:34:27.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.340 7f1e2e329700 1 -- 172.21.2.112:0/1652250821 learned_addr learned my addr 172.21.2.112:0/1652250821 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:34:27.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.340 7f1e2eb2a700 1 --1- 172.21.2.112:0/1652250821 >> v1:172.21.2.120:6789/0 conn(0x7f1e280d1c70 0x7f1e280efc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:43398/0 (socket says 172.21.2.112:43398) 2021-02-10T11:34:27.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.340 7f1e2db28700 1 --2- 172.21.2.112:0/1652250821 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1e280d0340 0x7f1e280ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:27.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.340 7f1e2d327700 1 -- 172.21.2.112:0/1652250821 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3194689660 0 0) 0x7f1e1c001180 con 0x7f1e280d1870 2021-02-10T11:34:27.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.340 7f1e2db28700 1 -- 172.21.2.112:0/1652250821 >> v1:172.21.2.120:6789/0 conn(0x7f1e280d1c70 legacy=0x7f1e280efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:27.344 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.340 7f1e2db28700 1 -- 172.21.2.112:0/1652250821 >> v1:172.21.2.112:6789/0 conn(0x7f1e280d1870 legacy=0x7f1e281b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:27.344 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.340 7f1e2db28700 1 -- 172.21.2.112:0/1652250821 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e280f0450 con 0x7f1e280d0340 2021-02-10T11:34:27.344 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.340 7f1e2db28700 1 --2- 172.21.2.112:0/1652250821 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1e280d0340 0x7f1e280ef640 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f1e24006650 tx=0x7f1e24009a00).ready entity=mon.2 client_cookie=e55381ed05a392e0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:27.344 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.341 7f1e2d327700 1 -- 172.21.2.112:0/1652250821 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1e240009b0 con 0x7f1e280d0340 2021-02-10T11:34:27.344 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.341 7f1e2d327700 1 -- 172.21.2.112:0/1652250821 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e2400c0c0 con 0x7f1e280d0340 2021-02-10T11:34:27.344 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.341 7f1e2d327700 1 -- 172.21.2.112:0/1652250821 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1e2400c5a0 con 0x7f1e280d0340 2021-02-10T11:34:27.344 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.341 7f1e3050d700 1 -- 172.21.2.112:0/1652250821 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1e280d0340 msgr2=0x7f1e280ef640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:27.344 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.341 7f1e3050d700 1 --2- 172.21.2.112:0/1652250821 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1e280d0340 0x7f1e280ef640 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f1e24006650 tx=0x7f1e24009a00).stop 2021-02-10T11:34:27.344 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.341 7f1e3050d700 1 -- 172.21.2.112:0/1652250821 shutdown_connections 2021-02-10T11:34:27.344 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.341 7f1e3050d700 1 --2- 172.21.2.112:0/1652250821 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1e280d0340 0x7f1e280ef640 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:27.344 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.341 7f1e3050d700 1 -- 172.21.2.112:0/1652250821 >> 172.21.2.112:0/1652250821 conn(0x7f1e280ce450 msgr2=0x7f1e280d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:27.344 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.341 7f1e3050d700 1 -- 172.21.2.112:0/1652250821 shutdown_connections 2021-02-10T11:34:27.344 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.341 7f1e3050d700 1 -- 172.21.2.112:0/1652250821 wait complete. 2021-02-10T11:34:27.345 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.356 7f1e3050d700 1 Processor -- start 2021-02-10T11:34:27.345 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.356 7f1e3050d700 1 -- start start 2021-02-10T11:34:27.346 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.356 7f1e3050d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1e28095750 0x7f1e2818c730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:27.346 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.356 7f1e3050d700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1e28095c50 0x7f1e281a0ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:27.346 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.356 7f1e3050d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1e281a12c0 0x7f1e281a16c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:27.346 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.356 7f1e3050d700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e2809fba0 con 0x7f1e28095750 2021-02-10T11:34:27.346 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.356 7f1e3050d700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e2818cd50 con 0x7f1e28095c50 2021-02-10T11:34:27.346 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.356 7f1e3050d700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e28094190 con 0x7f1e281a12c0 2021-02-10T11:34:27.346 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.356 7f1e2e329700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1e28095750 0x7f1e2818c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:27.347 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.357 7f1e2eb2a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1e281a12c0 0x7f1e281a16c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:27.347 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.357 7f1e2e329700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1e28095750 0x7f1e2818c730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:50382/0 (socket says 172.21.2.112:50382) 2021-02-10T11:34:27.347 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.357 7f1e2e329700 1 -- 172.21.2.112:0/3038863830 learned_addr learned my addr 172.21.2.112:0/3038863830 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:34:27.347 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.357 7f1e2db28700 1 --2- 172.21.2.112:0/3038863830 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1e28095c50 0x7f1e281a0ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:27.347 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.357 7f1e2eb2a700 1 -- 172.21.2.112:0/3038863830 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1e28095c50 msgr2=0x7f1e281a0ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:27.347 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.357 7f1e2eb2a700 1 --2- 172.21.2.112:0/3038863830 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1e28095c50 0x7f1e281a0ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:27.347 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.357 7f1e2eb2a700 1 -- 172.21.2.112:0/3038863830 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1e28095750 msgr2=0x7f1e2818c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:27.347 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.357 7f1e2eb2a700 1 --2- 172.21.2.112:0/3038863830 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1e28095750 0x7f1e2818c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:27.347 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.357 7f1e2eb2a700 1 -- 172.21.2.112:0/3038863830 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e280a0120 con 0x7f1e281a12c0 2021-02-10T11:34:27.347 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.357 7f1e2eb2a700 1 --2- 172.21.2.112:0/3038863830 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1e281a12c0 0x7f1e281a16c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f1e20007fc0 tx=0x7f1e2000ac20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:27.347 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.357 7f1e1affd700 1 -- 172.21.2.112:0/3038863830 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1e20001400 con 0x7f1e281a12c0 2021-02-10T11:34:27.347 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.357 7f1e1affd700 1 -- 172.21.2.112:0/3038863830 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e200048b0 con 0x7f1e281a12c0 2021-02-10T11:34:27.347 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.357 7f1e1affd700 1 -- 172.21.2.112:0/3038863830 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1e20004dd0 con 0x7f1e281a12c0 2021-02-10T11:34:27.348 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.357 7f1e3050d700 1 -- 172.21.2.112:0/3038863830 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e280a0120 con 0x7f1e281a12c0 2021-02-10T11:34:27.348 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.357 7f1e3050d700 1 -- 172.21.2.112:0/3038863830 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e280a06b0 con 0x7f1e281a12c0 2021-02-10T11:34:27.348 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.358 7f1e1affd700 1 -- 172.21.2.112:0/3038863830 <== mon.2 v2:172.21.2.120:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1e2000ad60 con 0x7f1e281a12c0 2021-02-10T11:34:27.348 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.358 7f1e1affd700 1 --2- 172.21.2.112:0/3038863830 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1e100186c0 0x7f1e1001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:27.348 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.358 7f1e1affd700 1 -- 172.21.2.112:0/3038863830 <== mon.2 v2:172.21.2.120:3301/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f1e20032850 con 0x7f1e281a12c0 2021-02-10T11:34:27.348 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.358 7f1e2e329700 1 --2- 172.21.2.112:0/3038863830 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1e100186c0 0x7f1e1001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:27.348 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.359 7f1e2e329700 1 --2- 172.21.2.112:0/3038863830 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1e100186c0 0x7f1e1001abc0 crc :-1 s=READY pgs=662 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:27.376 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.389 7f1e3050d700 1 -- 172.21.2.112:0/3038863830 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e28094170 con 0x7f1e281a12c0 2021-02-10T11:34:27.379 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.393 7f1e1affd700 1 -- 172.21.2.112:0/3038863830 <== mon.2 v2:172.21.2.120:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1e20001400 con 0x7f1e281a12c0 2021-02-10T11:34:27.532 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.545 7f1e3050d700 1 -- 172.21.2.112:0/3038863830 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f1e28094170 con 0x7f1e281a12c0 2021-02-10T11:34:27.537 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.550 7f1e1affd700 1 -- 172.21.2.112:0/3038863830 <== mon.2 v2:172.21.2.120:3301/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v13) v1 ==== 104+0+0 (secure 0 0 0) 0x7f1e2000ad60 con 0x7f1e281a12c0 2021-02-10T11:34:27.540 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.554 7f1e3050d700 1 -- 172.21.2.112:0/3038863830 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1e100186c0 msgr2=0x7f1e1001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:27.541 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.554 7f1e3050d700 1 --2- 172.21.2.112:0/3038863830 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1e100186c0 0x7f1e1001abc0 crc :-1 s=READY pgs=662 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:27.541 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.554 7f1e3050d700 1 -- 172.21.2.112:0/3038863830 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1e281a12c0 msgr2=0x7f1e281a16c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:27.541 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.554 7f1e3050d700 1 --2- 172.21.2.112:0/3038863830 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1e281a12c0 0x7f1e281a16c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f1e20007fc0 tx=0x7f1e2000ac20).stop 2021-02-10T11:34:27.541 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.554 7f1e3050d700 1 -- 172.21.2.112:0/3038863830 shutdown_connections 2021-02-10T11:34:27.541 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.554 7f1e3050d700 1 --2- 172.21.2.112:0/3038863830 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1e100186c0 0x7f1e1001abc0 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:27.541 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.554 7f1e3050d700 1 --2- 172.21.2.112:0/3038863830 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1e28095c50 0x7f1e281a0ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:27.541 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.554 7f1e3050d700 1 --2- 172.21.2.112:0/3038863830 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1e281a12c0 0x7f1e281a16c0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:27.541 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.554 7f1e3050d700 1 --2- 172.21.2.112:0/3038863830 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1e28095750 0x7f1e2818c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:27.541 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.554 7f1e3050d700 1 -- 172.21.2.112:0/3038863830 >> 172.21.2.112:0/3038863830 conn(0x7f1e280b4980 msgr2=0x7f1e2809ead0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:27.542 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.554 7f1e3050d700 1 -- 172.21.2.112:0/3038863830 shutdown_connections 2021-02-10T11:34:27.542 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.554 7f1e3050d700 1 -- 172.21.2.112:0/3038863830 wait complete. 2021-02-10T11:34:27.549 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestVolumes.test_authorize_deauthorize_legacy_subvolume' 2021-02-10T11:34:27.977 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.974 7f2a8ca3e700 1 Processor -- start 2021-02-10T11:34:27.978 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.975 7f2a8ca3e700 1 -- start start 2021-02-10T11:34:27.978 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.975 7f2a8ca3e700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f2a880d0340 0x7f2a880ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:27.978 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.975 7f2a8ca3e700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f2a880d1c70 0x7f2a880efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:27.978 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.975 7f2a8ca3e700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a880f0b10 con 0x7f2a880d1870 2021-02-10T11:34:27.978 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.975 7f2a8ca3e700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a880e5ed0 con 0x7f2a880d1c70 2021-02-10T11:34:27.978 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.975 7f2a8ca3e700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a881b0be0 con 0x7f2a880d0340 2021-02-10T11:34:27.979 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.975 7f2a86e1d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f2a880d1c70 0x7f2a880efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:27.979 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.975 7f2a86e1d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f2a880d1c70 0x7f2a880efc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:50392/0 (socket says 172.21.2.112:50392) 2021-02-10T11:34:27.979 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.975 7f2a86e1d700 1 -- 172.21.2.112:0/2894317680 learned_addr learned my addr 172.21.2.112:0/2894317680 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:34:27.979 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.975 7f2a85e1b700 1 --2- 172.21.2.112:0/2894317680 >> v2:172.21.2.120:3300/0 conn(0x7f2a880d0340 0x7f2a880ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:27.979 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.976 7f2a86e1d700 1 -- 172.21.2.112:0/2894317680 >> v2:172.21.2.120:3300/0 conn(0x7f2a880d0340 msgr2=0x7f2a880ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:27.979 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.976 7f2a86e1d700 1 --2- 172.21.2.112:0/2894317680 >> v2:172.21.2.120:3300/0 conn(0x7f2a880d0340 0x7f2a880ef640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:27.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.976 7f2a86e1d700 1 -- 172.21.2.112:0/2894317680 >> v1:172.21.2.120:6789/0 conn(0x7f2a880d1870 legacy=0x7f2a881b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:27.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.976 7f2a86e1d700 1 -- 172.21.2.112:0/2894317680 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a880f20a0 con 0x7f2a880d1c70 2021-02-10T11:34:27.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.976 7f2a86e1d700 1 --2- 172.21.2.112:0/2894317680 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2a880d1c70 0x7f2a880efc60 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 rx=0x7f2a80006f00 tx=0x7f2a80009f10).ready entity=mon.0 client_cookie=b74dc9354a75532e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:27.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.976 7f2a8561a700 1 -- 172.21.2.112:0/2894317680 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2a8000b3b0 con 0x7f2a880d1c70 2021-02-10T11:34:27.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.976 7f2a8561a700 1 -- 172.21.2.112:0/2894317680 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a8000b730 con 0x7f2a880d1c70 2021-02-10T11:34:27.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.976 7f2a8561a700 1 -- 172.21.2.112:0/2894317680 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2a8000bb90 con 0x7f2a880d1c70 2021-02-10T11:34:27.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.976 7f2a8ca3e700 1 -- 172.21.2.112:0/2894317680 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2a880d1c70 msgr2=0x7f2a880efc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:27.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.976 7f2a8ca3e700 1 --2- 172.21.2.112:0/2894317680 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2a880d1c70 0x7f2a880efc60 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 rx=0x7f2a80006f00 tx=0x7f2a80009f10).stop 2021-02-10T11:34:27.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.977 7f2a8ca3e700 1 -- 172.21.2.112:0/2894317680 shutdown_connections 2021-02-10T11:34:27.981 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.977 7f2a8ca3e700 1 --2- 172.21.2.112:0/2894317680 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2a880d1c70 0x7f2a880efc60 unknown :-1 s=CLOSED pgs=1344 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:27.981 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.977 7f2a8ca3e700 1 --2- 172.21.2.112:0/2894317680 >> v2:172.21.2.120:3300/0 conn(0x7f2a880d0340 0x7f2a880ef640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:27.981 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.977 7f2a8ca3e700 1 -- 172.21.2.112:0/2894317680 >> 172.21.2.112:0/2894317680 conn(0x7f2a880ce450 msgr2=0x7f2a880d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:27.981 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.977 7f2a8ca3e700 1 -- 172.21.2.112:0/2894317680 shutdown_connections 2021-02-10T11:34:27.981 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.977 7f2a8ca3e700 1 -- 172.21.2.112:0/2894317680 wait complete. 2021-02-10T11:34:27.983 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.991 7f2a8ca3e700 1 Processor -- start 2021-02-10T11:34:27.983 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.991 7f2a8ca3e700 1 -- start start 2021-02-10T11:34:27.983 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.992 7f2a8ca3e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2a880970b0 0x7f2a880ebe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:27.983 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.992 7f2a8ca3e700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2a88098810 0x7f2a88098c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:27.983 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.992 7f2a8ca3e700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2a88099230 0x7f2a880b18c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:27.983 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.992 7f2a8ca3e700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a880ec6d0 con 0x7f2a880970b0 2021-02-10T11:34:27.983 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.992 7f2a8ca3e700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a880974b0 con 0x7f2a88098810 2021-02-10T11:34:27.984 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.992 7f2a8ca3e700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a880b1ee0 con 0x7f2a88099230 2021-02-10T11:34:27.984 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.992 7f2a8661c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2a880970b0 0x7f2a880ebe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:27.984 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.992 7f2a86e1d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2a88099230 0x7f2a880b18c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:27.984 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.992 7f2a8661c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2a880970b0 0x7f2a880ebe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:50396/0 (socket says 172.21.2.112:50396) 2021-02-10T11:34:27.984 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.992 7f2a8661c700 1 -- 172.21.2.112:0/3955544245 learned_addr learned my addr 172.21.2.112:0/3955544245 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:34:27.984 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.992 7f2a85e1b700 1 --2- 172.21.2.112:0/3955544245 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2a88098810 0x7f2a88098c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:27.984 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.992 7f2a86e1d700 1 -- 172.21.2.112:0/3955544245 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2a88098810 msgr2=0x7f2a88098c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:27.984 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.992 7f2a86e1d700 1 --2- 172.21.2.112:0/3955544245 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2a88098810 0x7f2a88098c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:27.984 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.992 7f2a86e1d700 1 -- 172.21.2.112:0/3955544245 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2a880970b0 msgr2=0x7f2a880ebe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:27.984 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.992 7f2a86e1d700 1 --2- 172.21.2.112:0/3955544245 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2a880970b0 0x7f2a880ebe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:27.984 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.992 7f2a86e1d700 1 -- 172.21.2.112:0/3955544245 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a880b2100 con 0x7f2a88099230 2021-02-10T11:34:27.984 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.992 7f2a8661c700 1 --2- 172.21.2.112:0/3955544245 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2a880970b0 0x7f2a880ebe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:34:27.985 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.992 7f2a86e1d700 1 --2- 172.21.2.112:0/3955544245 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2a88099230 0x7f2a880b18c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f2a80006f30 tx=0x7f2a8000bca0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:27.985 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.993 7f2a7effd700 1 -- 172.21.2.112:0/3955544245 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2a8000bde0 con 0x7f2a88099230 2021-02-10T11:34:27.985 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.993 7f2a7effd700 1 -- 172.21.2.112:0/3955544245 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a80000f90 con 0x7f2a88099230 2021-02-10T11:34:27.985 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.993 7f2a7effd700 1 -- 172.21.2.112:0/3955544245 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2a8000c570 con 0x7f2a88099230 2021-02-10T11:34:27.985 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.993 7f2a8ca3e700 1 -- 172.21.2.112:0/3955544245 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a880b2100 con 0x7f2a88099230 2021-02-10T11:34:27.985 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.993 7f2a8ca3e700 1 -- 172.21.2.112:0/3955544245 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a880b2690 con 0x7f2a88099230 2021-02-10T11:34:27.985 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.993 7f2a7effd700 1 -- 172.21.2.112:0/3955544245 <== mon.2 v2:172.21.2.120:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2a80001270 con 0x7f2a88099230 2021-02-10T11:34:27.986 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.993 7f2a7effd700 1 --2- 172.21.2.112:0/3955544245 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2a6c0186c0 0x7f2a6c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:27.986 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.993 7f2a7effd700 1 -- 172.21.2.112:0/3955544245 <== mon.2 v2:172.21.2.120:3301/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f2a80032850 con 0x7f2a88099230 2021-02-10T11:34:27.986 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.994 7f2a8661c700 1 --2- 172.21.2.112:0/3955544245 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2a6c0186c0 0x7f2a6c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:27.986 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:27.994 7f2a8661c700 1 --2- 172.21.2.112:0/3955544245 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2a6c0186c0 0x7f2a6c01abc0 crc :-1 s=READY pgs=663 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:28.012 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:28.025 7f2a8ca3e700 1 -- 172.21.2.112:0/3955544245 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a880ebc00 con 0x7f2a88099230 2021-02-10T11:34:28.015 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:28.028 7f2a7effd700 1 -- 172.21.2.112:0/3955544245 <== mon.2 v2:172.21.2.120:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2a80009450 con 0x7f2a88099230 2021-02-10T11:34:28.150 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:28.163 7f2a8ca3e700 1 -- 172.21.2.112:0/3955544245 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_authorize_deauthorize_legacy_subvolume"]} v 0) v1 -- 0x7f2a880f0b10 con 0x7f2a88099230 2021-02-10T11:34:29.069 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:29.082 7f2a7effd700 1 -- 172.21.2.112:0/3955544245 <== mon.2 v2:172.21.2.120:3301/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_authorize_deauthorize_legacy_subvolume"]}]=0 v328) v1 ==== 160+0+0 (secure 0 0 0) 0x7f2a8001f7e0 con 0x7f2a88099230 2021-02-10T11:34:29.120 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:29.133 7f2a8ca3e700 1 -- 172.21.2.112:0/3955544245 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2a6c0186c0 msgr2=0x7f2a6c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:29.121 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:29.133 7f2a8ca3e700 1 --2- 172.21.2.112:0/3955544245 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2a6c0186c0 0x7f2a6c01abc0 crc :-1 s=READY pgs=663 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:29.121 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:29.133 7f2a8ca3e700 1 -- 172.21.2.112:0/3955544245 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2a88099230 msgr2=0x7f2a880b18c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:29.121 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:29.133 7f2a8ca3e700 1 --2- 172.21.2.112:0/3955544245 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2a88099230 0x7f2a880b18c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f2a80006f30 tx=0x7f2a8000bca0).stop 2021-02-10T11:34:29.121 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:29.133 7f2a8ca3e700 1 -- 172.21.2.112:0/3955544245 shutdown_connections 2021-02-10T11:34:29.121 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:29.133 7f2a8ca3e700 1 --2- 172.21.2.112:0/3955544245 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2a6c0186c0 0x7f2a6c01abc0 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:29.121 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:29.133 7f2a8ca3e700 1 --2- 172.21.2.112:0/3955544245 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2a88098810 0x7f2a88098c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:29.121 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:29.133 7f2a8ca3e700 1 --2- 172.21.2.112:0/3955544245 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2a88099230 0x7f2a880b18c0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:29.121 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:29.133 7f2a8ca3e700 1 --2- 172.21.2.112:0/3955544245 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2a880970b0 0x7f2a880ebe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:29.121 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:29.133 7f2a8ca3e700 1 -- 172.21.2.112:0/3955544245 >> 172.21.2.112:0/3955544245 conn(0x7f2a8819a730 msgr2=0x7f2a8819e480 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:29.121 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:29.134 7f2a8ca3e700 1 -- 172.21.2.112:0/3955544245 shutdown_connections 2021-02-10T11:34:29.121 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:29.134 7f2a8ca3e700 1 -- 172.21.2.112:0/3955544245 wait complete. 2021-02-10T11:34:29.131 INFO:tasks.cephfs_test_runner:test_authorize_deauthorize_legacy_subvolume (tasks.cephfs.test_volumes.TestVolumes) ... ok 2021-02-10T11:34:29.131 INFO:tasks.cephfs_test_runner:Starting test: test_authorize_deauthorize_subvolume (tasks.cephfs.test_volumes.TestVolumes) 2021-02-10T11:34:29.132 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestVolumes.test_authorize_deauthorize_subvolume' 2021-02-10T11:34:29.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:29.555 7f3f3e1a9700 1 Processor -- start 2021-02-10T11:34:29.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:29.555 7f3f3e1a9700 1 -- start start 2021-02-10T11:34:29.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:29.556 7f3f3e1a9700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f3f380d1870 0x7f3f381b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:29.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:29.556 7f3f3e1a9700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f3f381b0a10 0x7f3f380cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:29.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:29.556 7f3f3e1a9700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f380f0b10 con 0x7f3f380d1c70 2021-02-10T11:34:29.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:29.556 7f3f3e1a9700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f380e5ed0 con 0x7f3f381b0a10 2021-02-10T11:34:29.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:29.556 7f3f3e1a9700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f380f0650 con 0x7f3f380d1870 2021-02-10T11:34:29.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:29.556 7f3f36ffd700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f3f381b0a10 0x7f3f380cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:29.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:29.556 7f3f36ffd700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f3f381b0a10 0x7f3f380cf330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:50402/0 (socket says 172.21.2.112:50402) 2021-02-10T11:34:29.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:29.556 7f3f36ffd700 1 -- 172.21.2.112:0/3089133055 learned_addr learned my addr 172.21.2.112:0/3089133055 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:34:29.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:29.556 7f3f377fe700 1 --2- 172.21.2.112:0/3089133055 >> v2:172.21.2.120:3300/0 conn(0x7f3f380d1870 0x7f3f381b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:29.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:29.556 7f3f36ffd700 1 -- 172.21.2.112:0/3089133055 >> v2:172.21.2.120:3300/0 conn(0x7f3f380d1870 msgr2=0x7f3f381b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:29.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:29.556 7f3f36ffd700 1 --2- 172.21.2.112:0/3089133055 >> v2:172.21.2.120:3300/0 conn(0x7f3f380d1870 0x7f3f381b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:29.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:29.556 7f3f36ffd700 1 -- 172.21.2.112:0/3089133055 >> v1:172.21.2.120:6789/0 conn(0x7f3f380d1c70 legacy=0x7f3f380cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:29.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:29.556 7f3f36ffd700 1 -- 172.21.2.112:0/3089133055 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f380f0840 con 0x7f3f381b0a10 2021-02-10T11:34:29.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:29.556 7f3f377fe700 1 --2- 172.21.2.112:0/3089133055 >> v2:172.21.2.120:3300/0 conn(0x7f3f380d1870 0x7f3f381b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:34:29.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:29.557 7f3f36ffd700 1 --2- 172.21.2.112:0/3089133055 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3f381b0a10 0x7f3f380cf330 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 rx=0x7f3f2c006700 tx=0x7f3f2c009b80).ready entity=mon.0 client_cookie=792d96615b23213a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:29.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:29.557 7f3f367fc700 1 -- 172.21.2.112:0/3089133055 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3f2c00c0c0 con 0x7f3f381b0a10 2021-02-10T11:34:29.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:29.557 7f3f367fc700 1 -- 172.21.2.112:0/3089133055 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f2c00c400 con 0x7f3f381b0a10 2021-02-10T11:34:29.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:29.557 7f3f367fc700 1 -- 172.21.2.112:0/3089133055 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3f2c00c920 con 0x7f3f381b0a10 2021-02-10T11:34:29.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:29.557 7f3f3e1a9700 1 -- 172.21.2.112:0/3089133055 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3f381b0a10 msgr2=0x7f3f380cf330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:29.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:29.557 7f3f3e1a9700 1 --2- 172.21.2.112:0/3089133055 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3f381b0a10 0x7f3f380cf330 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 rx=0x7f3f2c006700 tx=0x7f3f2c009b80).stop 2021-02-10T11:34:29.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:29.558 7f3f3e1a9700 1 -- 172.21.2.112:0/3089133055 shutdown_connections 2021-02-10T11:34:29.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:29.558 7f3f3e1a9700 1 --2- 172.21.2.112:0/3089133055 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3f381b0a10 0x7f3f380cf330 unknown :-1 s=CLOSED pgs=1345 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:29.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:29.558 7f3f3e1a9700 1 --2- 172.21.2.112:0/3089133055 >> v2:172.21.2.120:3300/0 conn(0x7f3f380d1870 0x7f3f381b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:29.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:29.558 7f3f3e1a9700 1 -- 172.21.2.112:0/3089133055 >> 172.21.2.112:0/3089133055 conn(0x7f3f380ce450 msgr2=0x7f3f380d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:29.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:29.558 7f3f3e1a9700 1 -- 172.21.2.112:0/3089133055 shutdown_connections 2021-02-10T11:34:29.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:29.558 7f3f3e1a9700 1 -- 172.21.2.112:0/3089133055 wait complete. 2021-02-10T11:34:29.563 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:29.573 7f3f3e1a9700 1 Processor -- start 2021-02-10T11:34:29.563 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:29.573 7f3f3e1a9700 1 -- start start 2021-02-10T11:34:29.563 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:29.573 7f3f3e1a9700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3f38199110 0x7f3f38199510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:29.563 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:29.573 7f3f3e1a9700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3f381b1c50 0x7f3f3819a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:29.563 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:29.573 7f3f3e1a9700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3f3819aae0 0x7f3f3819aee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:29.563 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:29.573 7f3f3e1a9700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f38199f40 con 0x7f3f3819aae0 2021-02-10T11:34:29.563 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:29.573 7f3f3e1a9700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f3818f160 con 0x7f3f38199110 2021-02-10T11:34:29.563 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:29.573 7f3f3e1a9700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f3818ecc0 con 0x7f3f381b1c50 2021-02-10T11:34:29.563 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:29.573 7f3f37fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3f3819aae0 0x7f3f3819aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:29.563 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:29.573 7f3f37fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3f3819aae0 0x7f3f3819aee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:50408/0 (socket says 172.21.2.112:50408) 2021-02-10T11:34:29.563 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:29.573 7f3f37fff700 1 -- 172.21.2.112:0/3898523906 learned_addr learned my addr 172.21.2.112:0/3898523906 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:34:29.564 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:29.574 7f3f37fff700 1 -- 172.21.2.112:0/3898523906 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3f381b1c50 msgr2=0x7f3f3819a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:29.564 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:29.574 7f3f37fff700 1 --2- 172.21.2.112:0/3898523906 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3f381b1c50 0x7f3f3819a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:29.564 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:29.574 7f3f37fff700 1 -- 172.21.2.112:0/3898523906 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3f38199110 msgr2=0x7f3f38199510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:29.564 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:29.574 7f3f37fff700 1 --2- 172.21.2.112:0/3898523906 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3f38199110 0x7f3f38199510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:29.564 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:29.574 7f3f37fff700 1 -- 172.21.2.112:0/3898523906 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f38199b30 con 0x7f3f3819aae0 2021-02-10T11:34:29.564 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:29.574 7f3f37fff700 1 --2- 172.21.2.112:0/3898523906 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3f3819aae0 0x7f3f3819aee0 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 rx=0x7f3f3000b8e0 tx=0x7f3f3000bc80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:29.564 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:29.574 7f3f1ffff700 1 -- 172.21.2.112:0/3898523906 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3f30000f90 con 0x7f3f3819aae0 2021-02-10T11:34:29.564 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:29.574 7f3f1ffff700 1 -- 172.21.2.112:0/3898523906 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f3000c2f0 con 0x7f3f3819aae0 2021-02-10T11:34:29.564 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:29.574 7f3f1ffff700 1 -- 172.21.2.112:0/3898523906 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3f3000c810 con 0x7f3f3819aae0 2021-02-10T11:34:29.564 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:29.574 7f3f3e1a9700 1 -- 172.21.2.112:0/3898523906 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f38199b30 con 0x7f3f3819aae0 2021-02-10T11:34:29.564 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:29.574 7f3f3e1a9700 1 -- 172.21.2.112:0/3898523906 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f3819b7f0 con 0x7f3f3819aae0 2021-02-10T11:34:29.564 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:29.575 7f3f1ffff700 1 -- 172.21.2.112:0/3898523906 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3f30001220 con 0x7f3f3819aae0 2021-02-10T11:34:29.564 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:29.575 7f3f1ffff700 1 --2- 172.21.2.112:0/3898523906 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3f180186c0 0x7f3f1801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:29.565 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:29.575 7f3f1ffff700 1 -- 172.21.2.112:0/3898523906 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f3f30032900 con 0x7f3f3819aae0 2021-02-10T11:34:29.565 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:29.575 7f3f377fe700 1 --2- 172.21.2.112:0/3898523906 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3f180186c0 0x7f3f1801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:29.565 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:29.575 7f3f377fe700 1 --2- 172.21.2.112:0/3898523906 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3f180186c0 0x7f3f1801abc0 crc :-1 s=READY pgs=664 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:29.592 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:29.605 7f3f3e1a9700 1 -- 172.21.2.112:0/3898523906 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f3818f110 con 0x7f3f3819aae0 2021-02-10T11:34:29.595 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:29.608 7f3f1ffff700 1 -- 172.21.2.112:0/3898523906 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3f30000f40 con 0x7f3f3819aae0 2021-02-10T11:34:29.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:29.739 7f3f3e1a9700 1 -- 172.21.2.112:0/3898523906 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_authorize_deauthorize_subvolume"]} v 0) v1 -- 0x7f3f3818f110 con 0x7f3f3819aae0 2021-02-10T11:34:30.074 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:30.087 7f3f1ffff700 1 -- 172.21.2.112:0/3898523906 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_authorize_deauthorize_subvolume"]}]=0 v329) v1 ==== 156+0+0 (secure 0 0 0) 0x7f3f3001f860 con 0x7f3f3819aae0 2021-02-10T11:34:30.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:30.107 7f3f3e1a9700 1 -- 172.21.2.112:0/3898523906 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3f180186c0 msgr2=0x7f3f1801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:30.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:30.107 7f3f3e1a9700 1 --2- 172.21.2.112:0/3898523906 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3f180186c0 0x7f3f1801abc0 crc :-1 s=READY pgs=664 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:30.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:30.107 7f3f3e1a9700 1 -- 172.21.2.112:0/3898523906 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3f3819aae0 msgr2=0x7f3f3819aee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:30.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:30.107 7f3f3e1a9700 1 --2- 172.21.2.112:0/3898523906 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3f3819aae0 0x7f3f3819aee0 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 rx=0x7f3f3000b8e0 tx=0x7f3f3000bc80).stop 2021-02-10T11:34:30.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:30.107 7f3f3e1a9700 1 -- 172.21.2.112:0/3898523906 shutdown_connections 2021-02-10T11:34:30.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:30.107 7f3f3e1a9700 1 --2- 172.21.2.112:0/3898523906 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3f180186c0 0x7f3f1801abc0 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:30.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:30.107 7f3f3e1a9700 1 --2- 172.21.2.112:0/3898523906 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3f3819aae0 0x7f3f3819aee0 unknown :-1 s=CLOSED pgs=1346 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:30.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:30.107 7f3f3e1a9700 1 --2- 172.21.2.112:0/3898523906 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3f381b1c50 0x7f3f3819a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:30.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:30.107 7f3f3e1a9700 1 --2- 172.21.2.112:0/3898523906 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3f38199110 0x7f3f38199510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:30.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:30.107 7f3f3e1a9700 1 -- 172.21.2.112:0/3898523906 >> 172.21.2.112:0/3898523906 conn(0x7f3f3819f880 msgr2=0x7f3f38198620 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:30.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:30.107 7f3f3e1a9700 1 -- 172.21.2.112:0/3898523906 shutdown_connections 2021-02-10T11:34:30.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:30.107 7f3f3e1a9700 1 -- 172.21.2.112:0/3898523906 wait complete. 2021-02-10T11:34:30.102 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:34:30.531 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:30.527 7f94b5471700 1 Processor -- start 2021-02-10T11:34:30.531 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:30.527 7f94b5471700 1 -- start start 2021-02-10T11:34:30.531 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:30.528 7f94b5471700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f94b00d1870 0x7f94b01b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:30.531 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:30.528 7f94b5471700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f94b01b0ae0 0x7f94b0108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:30.532 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:30.528 7f94b5471700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94b00fe1c0 con 0x7f94b00d1c70 2021-02-10T11:34:30.532 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:30.528 7f94b5471700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94b0109a60 con 0x7f94b00d1870 2021-02-10T11:34:30.532 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:30.528 7f94b5471700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94b0109750 con 0x7f94b01b0ae0 2021-02-10T11:34:30.532 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:30.528 7f94af07d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f94b00d1870 0x7f94b01b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:30.532 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:30.528 7f94af07d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f94b00d1870 0x7f94b01b04c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:50414/0 (socket says 172.21.2.112:50414) 2021-02-10T11:34:30.532 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:30.528 7f94af07d700 1 -- 172.21.2.112:0/2129072254 learned_addr learned my addr 172.21.2.112:0/2129072254 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:34:30.532 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:30.529 7f94af07d700 1 -- 172.21.2.112:0/2129072254 >> v2:172.21.2.120:3300/0 conn(0x7f94b01b0ae0 msgr2=0x7f94b0108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:30.532 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:30.529 7f94af07d700 1 --2- 172.21.2.112:0/2129072254 >> v2:172.21.2.120:3300/0 conn(0x7f94b01b0ae0 0x7f94b0108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:30.532 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:30.529 7f94af07d700 1 -- 172.21.2.112:0/2129072254 >> v1:172.21.2.120:6789/0 conn(0x7f94b00d1c70 legacy=0x7f94b0108f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:30.532 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:30.529 7f94af07d700 1 -- 172.21.2.112:0/2129072254 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94b0088070 con 0x7f94b00d1870 2021-02-10T11:34:30.532 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:30.529 7f94af07d700 1 --2- 172.21.2.112:0/2129072254 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f94b00d1870 0x7f94b01b04c0 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 rx=0x7f94a0006700 tx=0x7f94a0009b80).ready entity=mon.0 client_cookie=11ba1f7e5deb29e7 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:30.532 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:30.529 7f94ae07b700 1 -- 172.21.2.112:0/2129072254 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f94a00009b0 con 0x7f94b00d1870 2021-02-10T11:34:30.533 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:30.529 7f94ae07b700 1 -- 172.21.2.112:0/2129072254 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94a000c0c0 con 0x7f94b00d1870 2021-02-10T11:34:30.533 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:30.529 7f94ae07b700 1 -- 172.21.2.112:0/2129072254 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f94a000c5e0 con 0x7f94b00d1870 2021-02-10T11:34:30.533 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:30.529 7f94b5471700 1 -- 172.21.2.112:0/2129072254 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f94b00d1870 msgr2=0x7f94b01b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:30.533 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:30.529 7f94b5471700 1 --2- 172.21.2.112:0/2129072254 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f94b00d1870 0x7f94b01b04c0 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 rx=0x7f94a0006700 tx=0x7f94a0009b80).stop 2021-02-10T11:34:30.533 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:30.530 7f94b5471700 1 -- 172.21.2.112:0/2129072254 shutdown_connections 2021-02-10T11:34:30.533 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:30.530 7f94b5471700 1 --2- 172.21.2.112:0/2129072254 >> v2:172.21.2.120:3300/0 conn(0x7f94b01b0ae0 0x7f94b0108940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:30.533 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:30.530 7f94b5471700 1 --2- 172.21.2.112:0/2129072254 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f94b00d1870 0x7f94b01b04c0 unknown :-1 s=CLOSED pgs=1347 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:30.533 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:30.530 7f94b5471700 1 -- 172.21.2.112:0/2129072254 >> 172.21.2.112:0/2129072254 conn(0x7f94b00ce420 msgr2=0x7f94b00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:30.533 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:30.530 7f94b5471700 1 -- 172.21.2.112:0/2129072254 shutdown_connections 2021-02-10T11:34:30.533 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:30.530 7f94b5471700 1 -- 172.21.2.112:0/2129072254 wait complete. 2021-02-10T11:34:30.535 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:30.545 7f94b5471700 1 Processor -- start 2021-02-10T11:34:30.535 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:30.545 7f94b5471700 1 -- start start 2021-02-10T11:34:30.535 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:30.545 7f94b5471700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f94b00c3820 0x7f94b0084cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:30.535 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:30.545 7f94b5471700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f94b00a0290 0x7f94b00852d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:30.535 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:30.545 7f94b5471700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f94b0086270 0x7f94b0086670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:30.535 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:30.545 7f94b5471700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94b00c0700 con 0x7f94b0086270 2021-02-10T11:34:30.535 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:30.545 7f94b5471700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94b00858f0 con 0x7f94b00a0290 2021-02-10T11:34:30.535 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:30.545 7f94b5471700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94b0085b10 con 0x7f94b00c3820 2021-02-10T11:34:30.535 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:30.545 7f94af87e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f94b0086270 0x7f94b0086670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:30.535 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:30.545 7f94af87e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f94b0086270 0x7f94b0086670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:50422/0 (socket says 172.21.2.112:50422) 2021-02-10T11:34:30.535 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:30.545 7f94af87e700 1 -- 172.21.2.112:0/415236705 learned_addr learned my addr 172.21.2.112:0/415236705 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:34:30.536 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:30.546 7f94af87e700 1 -- 172.21.2.112:0/415236705 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f94b00c3820 msgr2=0x7f94b0084cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:30.536 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:30.546 7f94af87e700 1 --2- 172.21.2.112:0/415236705 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f94b00c3820 0x7f94b0084cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:30.536 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:30.546 7f94af87e700 1 -- 172.21.2.112:0/415236705 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f94b00a0290 msgr2=0x7f94b00852d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:30.536 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:30.546 7f94af87e700 1 --2- 172.21.2.112:0/415236705 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f94b00a0290 0x7f94b00852d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:30.536 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:30.546 7f94af87e700 1 -- 172.21.2.112:0/415236705 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94b0086c90 con 0x7f94b0086270 2021-02-10T11:34:30.536 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:30.546 7f94af87e700 1 --2- 172.21.2.112:0/415236705 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f94b0086270 0x7f94b0086670 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 rx=0x7f94a8009940 tx=0x7f94a800bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:30.536 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:30.546 7f94a7fff700 1 -- 172.21.2.112:0/415236705 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f94a8000f90 con 0x7f94b0086270 2021-02-10T11:34:30.536 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:30.546 7f94a7fff700 1 -- 172.21.2.112:0/415236705 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94a8000d30 con 0x7f94b0086270 2021-02-10T11:34:30.536 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:30.546 7f94a7fff700 1 -- 172.21.2.112:0/415236705 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f94a800c570 con 0x7f94b0086270 2021-02-10T11:34:30.536 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:30.546 7f94b5471700 1 -- 172.21.2.112:0/415236705 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94b0086c90 con 0x7f94b0086270 2021-02-10T11:34:30.536 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:30.546 7f94b5471700 1 -- 172.21.2.112:0/415236705 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94b01a2f00 con 0x7f94b0086270 2021-02-10T11:34:30.536 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:30.547 7f94a7fff700 1 -- 172.21.2.112:0/415236705 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f94a8001220 con 0x7f94b0086270 2021-02-10T11:34:30.536 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:30.547 7f94a7fff700 1 --2- 172.21.2.112:0/415236705 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f94940186c0 0x7f949401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:30.537 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:30.547 7f94a7fff700 1 -- 172.21.2.112:0/415236705 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f94a80328f0 con 0x7f94b0086270 2021-02-10T11:34:30.537 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:30.548 7f94af07d700 1 --2- 172.21.2.112:0/415236705 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f94940186c0 0x7f949401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:30.537 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:30.548 7f94af07d700 1 --2- 172.21.2.112:0/415236705 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f94940186c0 0x7f949401abc0 crc :-1 s=READY pgs=665 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:30.565 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:30.578 7f94b5471700 1 -- 172.21.2.112:0/415236705 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94b0082a70 con 0x7f94b0086270 2021-02-10T11:34:30.567 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:30.581 7f94a7fff700 1 -- 172.21.2.112:0/415236705 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f94a8027270 con 0x7f94b0086270 2021-02-10T11:34:30.714 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:30.727 7f94b5471700 1 -- 172.21.2.112:0/415236705 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f94b0082a70 con 0x7f94b0086270 2021-02-10T11:34:30.718 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:30.732 7f94a7fff700 1 -- 172.21.2.112:0/415236705 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v14)=0 v14) v1 ==== 122+0+0 (secure 0 0 0) 0x7f94a8027270 con 0x7f94b0086270 2021-02-10T11:34:30.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:30.736 7f94b5471700 1 -- 172.21.2.112:0/415236705 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f94940186c0 msgr2=0x7f949401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:30.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:30.736 7f94b5471700 1 --2- 172.21.2.112:0/415236705 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f94940186c0 0x7f949401abc0 crc :-1 s=READY pgs=665 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:30.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:30.736 7f94b5471700 1 -- 172.21.2.112:0/415236705 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f94b0086270 msgr2=0x7f94b0086670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:30.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:30.736 7f94b5471700 1 --2- 172.21.2.112:0/415236705 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f94b0086270 0x7f94b0086670 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 rx=0x7f94a8009940 tx=0x7f94a800bcf0).stop 2021-02-10T11:34:30.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:30.736 7f94b5471700 1 -- 172.21.2.112:0/415236705 shutdown_connections 2021-02-10T11:34:30.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:30.736 7f94b5471700 1 --2- 172.21.2.112:0/415236705 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f94940186c0 0x7f949401abc0 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:30.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:30.736 7f94b5471700 1 --2- 172.21.2.112:0/415236705 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f94b00a0290 0x7f94b00852d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:30.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:30.736 7f94b5471700 1 --2- 172.21.2.112:0/415236705 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f94b0086270 0x7f94b0086670 unknown :-1 s=CLOSED pgs=1348 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:30.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:30.736 7f94b5471700 1 --2- 172.21.2.112:0/415236705 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f94b00c3820 0x7f94b0084cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:30.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:30.737 7f94b5471700 1 -- 172.21.2.112:0/415236705 >> 172.21.2.112:0/415236705 conn(0x7f94b00872c0 msgr2=0x7f94b0095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:30.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:30.737 7f94b5471700 1 -- 172.21.2.112:0/415236705 shutdown_connections 2021-02-10T11:34:30.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:30.737 7f94b5471700 1 -- 172.21.2.112:0/415236705 wait complete. 2021-02-10T11:34:30.732 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:34:30.732 DEBUG:teuthology.orchestra.run.gibba012:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:34:31.077 DEBUG:teuthology.orchestra.run.gibba020:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:34:31.420 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:34:31.421 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:31.421 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:34:31.421 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:34:31.757 INFO:teuthology.orchestra.run.gibba012.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.0’: No such file or directory 2021-02-10T11:34:31.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:34:31.757 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2021-02-10T11:34:31.758 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:31.758 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:34:31.758 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:34:32.094 INFO:teuthology.orchestra.run.gibba012.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: No such file or directory 2021-02-10T11:34:32.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:34:32.094 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2021-02-10T11:34:32.095 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:32.095 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:34:32.095 DEBUG:teuthology.orchestra.run.gibba020:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:34:32.428 INFO:teuthology.orchestra.run.gibba020.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:34:32.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:34:32.429 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:34:32.429 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:32.429 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:34:32.429 DEBUG:teuthology.orchestra.run.gibba020:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:34:32.763 INFO:teuthology.orchestra.run.gibba020.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:34:32.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:34:32.764 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:34:32.764 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json-pretty 2021-02-10T11:34:33.191 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.188 7ff515fbf700 1 Processor -- start 2021-02-10T11:34:33.191 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.188 7ff515fbf700 1 -- start start 2021-02-10T11:34:33.191 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.188 7ff515fbf700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7ff5100d1870 0x7ff5101b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:33.191 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.189 7ff515fbf700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7ff5101b0a10 0x7ff5100cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:33.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.189 7ff515fbf700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5100f0b10 con 0x7ff5100d1c70 2021-02-10T11:34:33.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.189 7ff515fbf700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5100e5ed0 con 0x7ff5100d1870 2021-02-10T11:34:33.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.189 7ff515fbf700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5100f0650 con 0x7ff5101b0a10 2021-02-10T11:34:33.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.189 7ff50f7fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7ff5100d1870 0x7ff5101b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:33.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.189 7ff50f7fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7ff5100d1870 0x7ff5101b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:50426/0 (socket says 172.21.2.112:50426) 2021-02-10T11:34:33.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.189 7ff50f7fe700 1 -- 172.21.2.112:0/258585539 learned_addr learned my addr 172.21.2.112:0/258585539 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:34:33.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.189 7ff506ffd700 1 --2- 172.21.2.112:0/258585539 >> v2:172.21.2.120:3300/0 conn(0x7ff5101b0a10 0x7ff5100cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:33.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.189 7ff50f7fe700 1 -- 172.21.2.112:0/258585539 >> v2:172.21.2.120:3300/0 conn(0x7ff5101b0a10 msgr2=0x7ff5100cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:33.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.189 7ff50f7fe700 1 --2- 172.21.2.112:0/258585539 >> v2:172.21.2.120:3300/0 conn(0x7ff5101b0a10 0x7ff5100cf330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:33.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.189 7ff50f7fe700 1 -- 172.21.2.112:0/258585539 >> v1:172.21.2.120:6789/0 conn(0x7ff5100d1c70 legacy=0x7ff5100cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:33.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.189 7ff50f7fe700 1 -- 172.21.2.112:0/258585539 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5100f0840 con 0x7ff5100d1870 2021-02-10T11:34:33.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.189 7ff50f7fe700 1 --2- 172.21.2.112:0/258585539 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff5100d1870 0x7ff5101b03f0 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 rx=0x7ff500006700 tx=0x7ff500009b00).ready entity=mon.0 client_cookie=d02803a0256f5241 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:33.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.189 7ff50effd700 1 -- 172.21.2.112:0/258585539 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff50000c0c0 con 0x7ff5100d1870 2021-02-10T11:34:33.193 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.190 7ff50effd700 1 -- 172.21.2.112:0/258585539 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff50000c400 con 0x7ff5100d1870 2021-02-10T11:34:33.193 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.190 7ff50effd700 1 -- 172.21.2.112:0/258585539 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff50000c8e0 con 0x7ff5100d1870 2021-02-10T11:34:33.193 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.190 7ff515fbf700 1 -- 172.21.2.112:0/258585539 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff5100d1870 msgr2=0x7ff5101b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:33.193 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.190 7ff515fbf700 1 --2- 172.21.2.112:0/258585539 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff5100d1870 0x7ff5101b03f0 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 rx=0x7ff500006700 tx=0x7ff500009b00).stop 2021-02-10T11:34:33.193 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.190 7ff515fbf700 1 -- 172.21.2.112:0/258585539 shutdown_connections 2021-02-10T11:34:33.193 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.190 7ff515fbf700 1 --2- 172.21.2.112:0/258585539 >> v2:172.21.2.120:3300/0 conn(0x7ff5101b0a10 0x7ff5100cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:33.193 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.190 7ff515fbf700 1 --2- 172.21.2.112:0/258585539 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff5100d1870 0x7ff5101b03f0 unknown :-1 s=CLOSED pgs=1349 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:33.193 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.190 7ff515fbf700 1 -- 172.21.2.112:0/258585539 >> 172.21.2.112:0/258585539 conn(0x7ff5100ce450 msgr2=0x7ff5100d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:33.193 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.190 7ff515fbf700 1 -- 172.21.2.112:0/258585539 shutdown_connections 2021-02-10T11:34:33.193 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.190 7ff515fbf700 1 -- 172.21.2.112:0/258585539 wait complete. 2021-02-10T11:34:33.195 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.205 7ff515fbf700 1 Processor -- start 2021-02-10T11:34:33.195 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.205 7ff515fbf700 1 -- start start 2021-02-10T11:34:33.195 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.205 7ff515fbf700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff5100edc10 0x7ff5100ee2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:33.195 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.205 7ff515fbf700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff510091130 0x7ff51019d140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:33.195 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.205 7ff515fbf700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff51019d760 0x7ff51019db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:33.195 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.205 7ff515fbf700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff51019e6c0 con 0x7ff510091130 2021-02-10T11:34:33.195 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.205 7ff515fbf700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff51019cbf0 con 0x7ff5100edc10 2021-02-10T11:34:33.196 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.205 7ff515fbf700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5100ee010 con 0x7ff51019d760 2021-02-10T11:34:33.196 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.205 7ff506ffd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff510091130 0x7ff51019d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:33.196 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.205 7ff506ffd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff510091130 0x7ff51019d140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:50432/0 (socket says 172.21.2.112:50432) 2021-02-10T11:34:33.196 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.205 7ff506ffd700 1 -- 172.21.2.112:0/3769834560 learned_addr learned my addr 172.21.2.112:0/3769834560 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:34:33.196 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.206 7ff50ffff700 1 --2- 172.21.2.112:0/3769834560 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff51019d760 0x7ff51019db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:33.196 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.206 7ff50f7fe700 1 --2- 172.21.2.112:0/3769834560 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff5100edc10 0x7ff5100ee2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:33.196 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.206 7ff506ffd700 1 -- 172.21.2.112:0/3769834560 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff51019d760 msgr2=0x7ff51019db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:33.196 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.206 7ff506ffd700 1 --2- 172.21.2.112:0/3769834560 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff51019d760 0x7ff51019db60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:33.196 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.206 7ff506ffd700 1 -- 172.21.2.112:0/3769834560 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff5100edc10 msgr2=0x7ff5100ee2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:33.196 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.206 7ff506ffd700 1 --2- 172.21.2.112:0/3769834560 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff5100edc10 0x7ff5100ee2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:33.196 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.206 7ff506ffd700 1 -- 172.21.2.112:0/3769834560 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5100ee910 con 0x7ff510091130 2021-02-10T11:34:33.196 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.206 7ff50f7fe700 1 --2- 172.21.2.112:0/3769834560 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff5100edc10 0x7ff5100ee2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:34:33.196 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.206 7ff506ffd700 1 --2- 172.21.2.112:0/3769834560 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff510091130 0x7ff51019d140 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 rx=0x7ff4f8006be0 tx=0x7ff4f800b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:33.196 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.206 7ff50cff9700 1 -- 172.21.2.112:0/3769834560 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff4f80064d0 con 0x7ff510091130 2021-02-10T11:34:33.197 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.206 7ff50cff9700 1 -- 172.21.2.112:0/3769834560 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4f800bd80 con 0x7ff510091130 2021-02-10T11:34:33.197 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.206 7ff515fbf700 1 -- 172.21.2.112:0/3769834560 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5100ee910 con 0x7ff510091130 2021-02-10T11:34:33.197 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.206 7ff50cff9700 1 -- 172.21.2.112:0/3769834560 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff4f8002030 con 0x7ff510091130 2021-02-10T11:34:33.197 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.206 7ff515fbf700 1 -- 172.21.2.112:0/3769834560 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff51019e180 con 0x7ff510091130 2021-02-10T11:34:33.197 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.207 7ff50cff9700 1 -- 172.21.2.112:0/3769834560 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff4f80052e0 con 0x7ff510091130 2021-02-10T11:34:33.197 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.207 7ff50cff9700 1 --2- 172.21.2.112:0/3769834560 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff4f40186d0 0x7ff4f401abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:33.197 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.207 7ff50cff9700 1 -- 172.21.2.112:0/3769834560 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5573+0+0 (secure 0 0 0) 0x7ff4f8032780 con 0x7ff510091130 2021-02-10T11:34:33.197 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.207 7ff50f7fe700 1 --2- 172.21.2.112:0/3769834560 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff4f40186d0 0x7ff4f401abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:33.197 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.207 7ff50f7fe700 1 --2- 172.21.2.112:0/3769834560 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff4f40186d0 0x7ff4f401abd0 crc :-1 s=READY pgs=666 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:33.224 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.238 7ff515fbf700 1 -- 172.21.2.112:0/3769834560 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5100ebc60 con 0x7ff510091130 2021-02-10T11:34:33.227 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.240 7ff50cff9700 1 -- 172.21.2.112:0/3769834560 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff4f8006d00 con 0x7ff510091130 2021-02-10T11:34:33.354 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.367 7ff515fbf700 1 -- 172.21.2.112:0/3769834560 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7ff5100f0b10 con 0x7ff510091130 2021-02-10T11:34:33.354 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.368 7ff50cff9700 1 -- 172.21.2.112:0/3769834560 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v55) v1 ==== 81+0+30363 (secure 0 0 0) 0x7ff4f801f4c0 con 0x7ff510091130 2021-02-10T11:34:33.355 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:34:33.355 INFO:teuthology.orchestra.run.gibba012.stdout:{ 2021-02-10T11:34:33.355 INFO:teuthology.orchestra.run.gibba012.stdout: "epoch": 55, 2021-02-10T11:34:33.355 INFO:teuthology.orchestra.run.gibba012.stdout: "fsid": "f14f004e-0699-4584-ad81-3754258493de", 2021-02-10T11:34:33.355 INFO:teuthology.orchestra.run.gibba012.stdout: "created": "2021-02-10 11:13:26.501424", 2021-02-10T11:34:33.355 INFO:teuthology.orchestra.run.gibba012.stdout: "modified": "2021-02-10 11:33:24.553170", 2021-02-10T11:34:33.355 INFO:teuthology.orchestra.run.gibba012.stdout: "last_up_change": "2021-02-10 11:13:52.780316", 2021-02-10T11:34:33.355 INFO:teuthology.orchestra.run.gibba012.stdout: "last_in_change": "2021-02-10 11:13:52.780316", 2021-02-10T11:34:33.355 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-02-10T11:34:33.356 INFO:teuthology.orchestra.run.gibba012.stdout: "flags_num": 5799936, 2021-02-10T11:34:33.356 INFO:teuthology.orchestra.run.gibba012.stdout: "flags_set": [ 2021-02-10T11:34:33.356 INFO:teuthology.orchestra.run.gibba012.stdout: "pglog_hardlimit", 2021-02-10T11:34:33.356 INFO:teuthology.orchestra.run.gibba012.stdout: "purged_snapdirs", 2021-02-10T11:34:33.356 INFO:teuthology.orchestra.run.gibba012.stdout: "recovery_deletes", 2021-02-10T11:34:33.356 INFO:teuthology.orchestra.run.gibba012.stdout: "sortbitwise" 2021-02-10T11:34:33.356 INFO:teuthology.orchestra.run.gibba012.stdout: ], 2021-02-10T11:34:33.356 INFO:teuthology.orchestra.run.gibba012.stdout: "crush_version": 4, 2021-02-10T11:34:33.356 INFO:teuthology.orchestra.run.gibba012.stdout: "full_ratio": 0.94999998807907104, 2021-02-10T11:34:33.356 INFO:teuthology.orchestra.run.gibba012.stdout: "backfillfull_ratio": 0.89999997615814209, 2021-02-10T11:34:33.356 INFO:teuthology.orchestra.run.gibba012.stdout: "nearfull_ratio": 0.85000002384185791, 2021-02-10T11:34:33.356 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_snapshot": "", 2021-02-10T11:34:33.357 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_max": 11, 2021-02-10T11:34:33.357 INFO:teuthology.orchestra.run.gibba012.stdout: "max_osd": 8, 2021-02-10T11:34:33.357 INFO:teuthology.orchestra.run.gibba012.stdout: "require_min_compat_client": "jewel", 2021-02-10T11:34:33.357 INFO:teuthology.orchestra.run.gibba012.stdout: "min_compat_client": "jewel", 2021-02-10T11:34:33.357 INFO:teuthology.orchestra.run.gibba012.stdout: "require_osd_release": "nautilus", 2021-02-10T11:34:33.357 INFO:teuthology.orchestra.run.gibba012.stdout: "pools": [ 2021-02-10T11:34:33.357 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:33.357 INFO:teuthology.orchestra.run.gibba012.stdout: "pool": 1, 2021-02-10T11:34:33.357 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_name": "rbd", 2021-02-10T11:34:33.357 INFO:teuthology.orchestra.run.gibba012.stdout: "create_time": "2021-02-10 11:13:57.508758", 2021-02-10T11:34:33.357 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:34:33.357 INFO:teuthology.orchestra.run.gibba012.stdout: "flags_names": "hashpspool", 2021-02-10T11:34:33.357 INFO:teuthology.orchestra.run.gibba012.stdout: "type": 1, 2021-02-10T11:34:33.357 INFO:teuthology.orchestra.run.gibba012.stdout: "size": 2, 2021-02-10T11:34:33.358 INFO:teuthology.orchestra.run.gibba012.stdout: "min_size": 1, 2021-02-10T11:34:33.358 INFO:teuthology.orchestra.run.gibba012.stdout: "crush_rule": 0, 2021-02-10T11:34:33.358 INFO:teuthology.orchestra.run.gibba012.stdout: "object_hash": 2, 2021-02-10T11:34:33.358 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:34:33.358 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num": 8, 2021-02-10T11:34:33.358 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num": 8, 2021-02-10T11:34:33.358 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num_target": 8, 2021-02-10T11:34:33.358 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_target": 8, 2021-02-10T11:34:33.358 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_pending": 8, 2021-02-10T11:34:33.358 INFO:teuthology.orchestra.run.gibba012.stdout: "last_pg_merge_meta": { 2021-02-10T11:34:33.358 INFO:teuthology.orchestra.run.gibba012.stdout: "source_pgid": "0.0", 2021-02-10T11:34:33.358 INFO:teuthology.orchestra.run.gibba012.stdout: "ready_epoch": 0, 2021-02-10T11:34:33.359 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_started": 0, 2021-02-10T11:34:33.359 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_clean": 0, 2021-02-10T11:34:33.359 INFO:teuthology.orchestra.run.gibba012.stdout: "source_version": "0'0", 2021-02-10T11:34:33.359 INFO:teuthology.orchestra.run.gibba012.stdout: "target_version": "0'0" 2021-02-10T11:34:33.359 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:33.359 INFO:teuthology.orchestra.run.gibba012.stdout: "last_change": "16", 2021-02-10T11:34:33.359 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend": "0", 2021-02-10T11:34:33.359 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:34:33.359 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:34:33.359 INFO:teuthology.orchestra.run.gibba012.stdout: "auid": 0, 2021-02-10T11:34:33.359 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:34:33.359 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_seq": 0, 2021-02-10T11:34:33.360 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_epoch": 0, 2021-02-10T11:34:33.360 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_snaps": [], 2021-02-10T11:34:33.360 INFO:teuthology.orchestra.run.gibba012.stdout: "removed_snaps": "[]", 2021-02-10T11:34:33.360 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_bytes": 0, 2021-02-10T11:34:33.360 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_objects": 0, 2021-02-10T11:34:33.360 INFO:teuthology.orchestra.run.gibba012.stdout: "tiers": [], 2021-02-10T11:34:33.360 INFO:teuthology.orchestra.run.gibba012.stdout: "tier_of": -1, 2021-02-10T11:34:33.360 INFO:teuthology.orchestra.run.gibba012.stdout: "read_tier": -1, 2021-02-10T11:34:33.360 INFO:teuthology.orchestra.run.gibba012.stdout: "write_tier": -1, 2021-02-10T11:34:33.360 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_mode": "none", 2021-02-10T11:34:33.360 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_bytes": 0, 2021-02-10T11:34:33.360 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_objects": 0, 2021-02-10T11:34:33.361 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:34:33.361 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:34:33.361 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:34:33.361 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_flush_age": 0, 2021-02-10T11:34:33.361 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_evict_age": 0, 2021-02-10T11:34:33.361 INFO:teuthology.orchestra.run.gibba012.stdout: "erasure_code_profile": "", 2021-02-10T11:34:33.361 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_params": { 2021-02-10T11:34:33.361 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "none" 2021-02-10T11:34:33.361 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:33.361 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_period": 0, 2021-02-10T11:34:33.361 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_count": 0, 2021-02-10T11:34:33.361 INFO:teuthology.orchestra.run.gibba012.stdout: "use_gmt_hitset": true, 2021-02-10T11:34:33.361 INFO:teuthology.orchestra.run.gibba012.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:34:33.361 INFO:teuthology.orchestra.run.gibba012.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:34:33.362 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:34:33.362 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:34:33.362 INFO:teuthology.orchestra.run.gibba012.stdout: "grade_table": [], 2021-02-10T11:34:33.362 INFO:teuthology.orchestra.run.gibba012.stdout: "stripe_width": 0, 2021-02-10T11:34:33.362 INFO:teuthology.orchestra.run.gibba012.stdout: "expected_num_objects": 0, 2021-02-10T11:34:33.362 INFO:teuthology.orchestra.run.gibba012.stdout: "fast_read": false, 2021-02-10T11:34:33.362 INFO:teuthology.orchestra.run.gibba012.stdout: "options": {}, 2021-02-10T11:34:33.362 INFO:teuthology.orchestra.run.gibba012.stdout: "application_metadata": { 2021-02-10T11:34:33.362 INFO:teuthology.orchestra.run.gibba012.stdout: "rbd": {} 2021-02-10T11:34:33.362 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:34:33.362 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:33.363 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:33.363 INFO:teuthology.orchestra.run.gibba012.stdout: "pool": 10, 2021-02-10T11:34:33.363 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_name": "cephfs_metadata", 2021-02-10T11:34:33.363 INFO:teuthology.orchestra.run.gibba012.stdout: "create_time": "2021-02-10 11:33:21.523665", 2021-02-10T11:34:33.363 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:34:33.363 INFO:teuthology.orchestra.run.gibba012.stdout: "flags_names": "hashpspool", 2021-02-10T11:34:33.363 INFO:teuthology.orchestra.run.gibba012.stdout: "type": 1, 2021-02-10T11:34:33.363 INFO:teuthology.orchestra.run.gibba012.stdout: "size": 2, 2021-02-10T11:34:33.363 INFO:teuthology.orchestra.run.gibba012.stdout: "min_size": 1, 2021-02-10T11:34:33.363 INFO:teuthology.orchestra.run.gibba012.stdout: "crush_rule": 0, 2021-02-10T11:34:33.363 INFO:teuthology.orchestra.run.gibba012.stdout: "object_hash": 2, 2021-02-10T11:34:33.363 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:34:33.363 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num": 8, 2021-02-10T11:34:33.364 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num": 8, 2021-02-10T11:34:33.364 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num_target": 8, 2021-02-10T11:34:33.364 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_target": 8, 2021-02-10T11:34:33.364 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_pending": 8, 2021-02-10T11:34:33.364 INFO:teuthology.orchestra.run.gibba012.stdout: "last_pg_merge_meta": { 2021-02-10T11:34:33.364 INFO:teuthology.orchestra.run.gibba012.stdout: "source_pgid": "0.0", 2021-02-10T11:34:33.364 INFO:teuthology.orchestra.run.gibba012.stdout: "ready_epoch": 0, 2021-02-10T11:34:33.364 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_started": 0, 2021-02-10T11:34:33.364 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_clean": 0, 2021-02-10T11:34:33.364 INFO:teuthology.orchestra.run.gibba012.stdout: "source_version": "0'0", 2021-02-10T11:34:33.364 INFO:teuthology.orchestra.run.gibba012.stdout: "target_version": "0'0" 2021-02-10T11:34:33.364 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:33.365 INFO:teuthology.orchestra.run.gibba012.stdout: "last_change": "54", 2021-02-10T11:34:33.365 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend": "0", 2021-02-10T11:34:33.365 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:34:33.365 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:34:33.365 INFO:teuthology.orchestra.run.gibba012.stdout: "auid": 0, 2021-02-10T11:34:33.365 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:34:33.365 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_seq": 0, 2021-02-10T11:34:33.365 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_epoch": 0, 2021-02-10T11:34:33.365 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_snaps": [], 2021-02-10T11:34:33.365 INFO:teuthology.orchestra.run.gibba012.stdout: "removed_snaps": "[]", 2021-02-10T11:34:33.365 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_bytes": 0, 2021-02-10T11:34:33.365 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_objects": 0, 2021-02-10T11:34:33.365 INFO:teuthology.orchestra.run.gibba012.stdout: "tiers": [], 2021-02-10T11:34:33.365 INFO:teuthology.orchestra.run.gibba012.stdout: "tier_of": -1, 2021-02-10T11:34:33.366 INFO:teuthology.orchestra.run.gibba012.stdout: "read_tier": -1, 2021-02-10T11:34:33.366 INFO:teuthology.orchestra.run.gibba012.stdout: "write_tier": -1, 2021-02-10T11:34:33.366 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_mode": "none", 2021-02-10T11:34:33.366 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_bytes": 0, 2021-02-10T11:34:33.366 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_objects": 0, 2021-02-10T11:34:33.366 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:34:33.366 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:34:33.366 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:34:33.366 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_flush_age": 0, 2021-02-10T11:34:33.366 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_evict_age": 0, 2021-02-10T11:34:33.366 INFO:teuthology.orchestra.run.gibba012.stdout: "erasure_code_profile": "", 2021-02-10T11:34:33.366 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_params": { 2021-02-10T11:34:33.367 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "none" 2021-02-10T11:34:33.367 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:33.367 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_period": 0, 2021-02-10T11:34:33.367 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_count": 0, 2021-02-10T11:34:33.367 INFO:teuthology.orchestra.run.gibba012.stdout: "use_gmt_hitset": true, 2021-02-10T11:34:33.367 INFO:teuthology.orchestra.run.gibba012.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:34:33.367 INFO:teuthology.orchestra.run.gibba012.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:34:33.367 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:34:33.367 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:34:33.367 INFO:teuthology.orchestra.run.gibba012.stdout: "grade_table": [], 2021-02-10T11:34:33.367 INFO:teuthology.orchestra.run.gibba012.stdout: "stripe_width": 0, 2021-02-10T11:34:33.367 INFO:teuthology.orchestra.run.gibba012.stdout: "expected_num_objects": 0, 2021-02-10T11:34:33.367 INFO:teuthology.orchestra.run.gibba012.stdout: "fast_read": false, 2021-02-10T11:34:33.368 INFO:teuthology.orchestra.run.gibba012.stdout: "options": { 2021-02-10T11:34:33.368 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_autoscale_bias": 4, 2021-02-10T11:34:33.368 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_min": 16, 2021-02-10T11:34:33.368 INFO:teuthology.orchestra.run.gibba012.stdout: "recovery_priority": 5 2021-02-10T11:34:33.368 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:33.368 INFO:teuthology.orchestra.run.gibba012.stdout: "application_metadata": { 2021-02-10T11:34:33.368 INFO:teuthology.orchestra.run.gibba012.stdout: "cephfs": { 2021-02-10T11:34:33.368 INFO:teuthology.orchestra.run.gibba012.stdout: "metadata": "cephfs" 2021-02-10T11:34:33.368 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:34:33.368 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:34:33.368 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:33.368 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:33.368 INFO:teuthology.orchestra.run.gibba012.stdout: "pool": 11, 2021-02-10T11:34:33.369 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_name": "cephfs_data", 2021-02-10T11:34:33.369 INFO:teuthology.orchestra.run.gibba012.stdout: "create_time": "2021-02-10 11:33:22.595892", 2021-02-10T11:34:33.369 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:34:33.369 INFO:teuthology.orchestra.run.gibba012.stdout: "flags_names": "hashpspool", 2021-02-10T11:34:33.369 INFO:teuthology.orchestra.run.gibba012.stdout: "type": 1, 2021-02-10T11:34:33.369 INFO:teuthology.orchestra.run.gibba012.stdout: "size": 2, 2021-02-10T11:34:33.369 INFO:teuthology.orchestra.run.gibba012.stdout: "min_size": 1, 2021-02-10T11:34:33.369 INFO:teuthology.orchestra.run.gibba012.stdout: "crush_rule": 0, 2021-02-10T11:34:33.369 INFO:teuthology.orchestra.run.gibba012.stdout: "object_hash": 2, 2021-02-10T11:34:33.369 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:34:33.369 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num": 8, 2021-02-10T11:34:33.369 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num": 8, 2021-02-10T11:34:33.369 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num_target": 8, 2021-02-10T11:34:33.370 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_target": 8, 2021-02-10T11:34:33.370 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_pending": 8, 2021-02-10T11:34:33.370 INFO:teuthology.orchestra.run.gibba012.stdout: "last_pg_merge_meta": { 2021-02-10T11:34:33.370 INFO:teuthology.orchestra.run.gibba012.stdout: "source_pgid": "0.0", 2021-02-10T11:34:33.370 INFO:teuthology.orchestra.run.gibba012.stdout: "ready_epoch": 0, 2021-02-10T11:34:33.370 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_started": 0, 2021-02-10T11:34:33.370 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_clean": 0, 2021-02-10T11:34:33.370 INFO:teuthology.orchestra.run.gibba012.stdout: "source_version": "0'0", 2021-02-10T11:34:33.370 INFO:teuthology.orchestra.run.gibba012.stdout: "target_version": "0'0" 2021-02-10T11:34:33.370 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:33.370 INFO:teuthology.orchestra.run.gibba012.stdout: "last_change": "55", 2021-02-10T11:34:33.370 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend": "0", 2021-02-10T11:34:33.370 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:34:33.371 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:34:33.371 INFO:teuthology.orchestra.run.gibba012.stdout: "auid": 0, 2021-02-10T11:34:33.371 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:34:33.371 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_seq": 0, 2021-02-10T11:34:33.371 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_epoch": 0, 2021-02-10T11:34:33.371 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_snaps": [], 2021-02-10T11:34:33.371 INFO:teuthology.orchestra.run.gibba012.stdout: "removed_snaps": "[]", 2021-02-10T11:34:33.371 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_bytes": 0, 2021-02-10T11:34:33.371 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_objects": 0, 2021-02-10T11:34:33.371 INFO:teuthology.orchestra.run.gibba012.stdout: "tiers": [], 2021-02-10T11:34:33.371 INFO:teuthology.orchestra.run.gibba012.stdout: "tier_of": -1, 2021-02-10T11:34:33.371 INFO:teuthology.orchestra.run.gibba012.stdout: "read_tier": -1, 2021-02-10T11:34:33.371 INFO:teuthology.orchestra.run.gibba012.stdout: "write_tier": -1, 2021-02-10T11:34:33.372 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_mode": "none", 2021-02-10T11:34:33.372 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_bytes": 0, 2021-02-10T11:34:33.372 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_objects": 0, 2021-02-10T11:34:33.372 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:34:33.372 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:34:33.372 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:34:33.372 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_flush_age": 0, 2021-02-10T11:34:33.372 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_evict_age": 0, 2021-02-10T11:34:33.372 INFO:teuthology.orchestra.run.gibba012.stdout: "erasure_code_profile": "", 2021-02-10T11:34:33.372 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_params": { 2021-02-10T11:34:33.372 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "none" 2021-02-10T11:34:33.372 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:33.372 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_period": 0, 2021-02-10T11:34:33.372 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_count": 0, 2021-02-10T11:34:33.373 INFO:teuthology.orchestra.run.gibba012.stdout: "use_gmt_hitset": true, 2021-02-10T11:34:33.373 INFO:teuthology.orchestra.run.gibba012.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:34:33.373 INFO:teuthology.orchestra.run.gibba012.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:34:33.373 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:34:33.373 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:34:33.373 INFO:teuthology.orchestra.run.gibba012.stdout: "grade_table": [], 2021-02-10T11:34:33.373 INFO:teuthology.orchestra.run.gibba012.stdout: "stripe_width": 0, 2021-02-10T11:34:33.373 INFO:teuthology.orchestra.run.gibba012.stdout: "expected_num_objects": 0, 2021-02-10T11:34:33.373 INFO:teuthology.orchestra.run.gibba012.stdout: "fast_read": false, 2021-02-10T11:34:33.373 INFO:teuthology.orchestra.run.gibba012.stdout: "options": {}, 2021-02-10T11:34:33.373 INFO:teuthology.orchestra.run.gibba012.stdout: "application_metadata": { 2021-02-10T11:34:33.373 INFO:teuthology.orchestra.run.gibba012.stdout: "cephfs": { 2021-02-10T11:34:33.373 INFO:teuthology.orchestra.run.gibba012.stdout: "data": "cephfs" 2021-02-10T11:34:33.374 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:34:33.374 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:34:33.374 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:34:33.374 INFO:teuthology.orchestra.run.gibba012.stdout: ], 2021-02-10T11:34:33.374 INFO:teuthology.orchestra.run.gibba012.stdout: "osds": [ 2021-02-10T11:34:33.374 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:33.374 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 0, 2021-02-10T11:34:33.374 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "95444b8f-6370-4a1d-ae5f-df62651b77da", 2021-02-10T11:34:33.374 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:34:33.374 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:34:33.374 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:34:33.374 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:34:33.375 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:34:33.375 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:34:33.375 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:34:33.375 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 53, 2021-02-10T11:34:33.375 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:34:33.375 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:34:33.375 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:34:33.375 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:34:33.375 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:33.375 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:34:33.375 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6817", 2021-02-10T11:34:33.375 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:34:33.375 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:33.376 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:33.376 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:34:33.376 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6818", 2021-02-10T11:34:33.376 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:34:33.376 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:34:33.376 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:34:33.376 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:33.376 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:34:33.376 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:34:33.376 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:33.376 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:34:33.376 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6820", 2021-02-10T11:34:33.376 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:34:33.377 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:33.377 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:33.377 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:34:33.377 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6822", 2021-02-10T11:34:33.377 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:34:33.377 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:34:33.377 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:34:33.377 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:33.377 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:34:33.377 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:34:33.377 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:33.377 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:34:33.378 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6828", 2021-02-10T11:34:33.378 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:34:33.378 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:33.378 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:33.378 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:34:33.378 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6830", 2021-02-10T11:34:33.378 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:34:33.378 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:34:33.378 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:34:33.378 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:33.378 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:34:33.378 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:34:33.378 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:33.379 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:34:33.379 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6824", 2021-02-10T11:34:33.379 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:34:33.379 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:33.379 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:33.379 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:34:33.379 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6826", 2021-02-10T11:34:33.379 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:34:33.379 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:34:33.379 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:34:33.379 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:33.379 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.112:6818/36030", 2021-02-10T11:34:33.379 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.112:6822/36030", 2021-02-10T11:34:33.380 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.112:6830/36030", 2021-02-10T11:34:33.380 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.112:6826/36030", 2021-02-10T11:34:33.380 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:34:33.380 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:34:33.380 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:34:33.380 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:34:33.380 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:33.380 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:33.380 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 1, 2021-02-10T11:34:33.380 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "a23e217e-0d8e-486e-be07-712adfabd262", 2021-02-10T11:34:33.380 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:34:33.380 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:34:33.380 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:34:33.381 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:34:33.381 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:34:33.381 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:34:33.381 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:34:33.381 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 53, 2021-02-10T11:34:33.381 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:34:33.381 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:34:33.381 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:34:33.381 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:34:33.381 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:33.381 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:34:33.381 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6800", 2021-02-10T11:34:33.381 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:34:33.382 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:33.382 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:33.382 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:34:33.382 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6802", 2021-02-10T11:34:33.382 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:34:33.382 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:34:33.382 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:34:33.382 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:33.382 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:34:33.382 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:34:33.382 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:33.382 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:34:33.382 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6804", 2021-02-10T11:34:33.383 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:34:33.383 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:33.383 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:33.383 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:34:33.383 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6806", 2021-02-10T11:34:33.383 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:34:33.383 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:34:33.383 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:34:33.383 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:33.383 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:34:33.383 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:34:33.383 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:33.383 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:34:33.384 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6812", 2021-02-10T11:34:33.384 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:34:33.384 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:33.384 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:33.384 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:34:33.384 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6814", 2021-02-10T11:34:33.384 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:34:33.384 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:34:33.384 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:34:33.384 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:33.384 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:34:33.384 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:34:33.384 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:33.384 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:34:33.385 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6808", 2021-02-10T11:34:33.385 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:34:33.385 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:33.385 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:33.385 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:34:33.385 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6810", 2021-02-10T11:34:33.385 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:34:33.385 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:34:33.385 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:34:33.385 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:33.385 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.112:6802/36029", 2021-02-10T11:34:33.385 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.112:6806/36029", 2021-02-10T11:34:33.385 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.112:6814/36029", 2021-02-10T11:34:33.386 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.112:6810/36029", 2021-02-10T11:34:33.386 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:34:33.386 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:34:33.386 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:34:33.386 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:34:33.386 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:33.386 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:33.386 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 2, 2021-02-10T11:34:33.386 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "026afa8a-c0f1-4b35-9a76-53c00fe0c2fc", 2021-02-10T11:34:33.386 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:34:33.386 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:34:33.386 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:34:33.386 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:34:33.387 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:34:33.387 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:34:33.387 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:34:33.387 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 53, 2021-02-10T11:34:33.387 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:34:33.387 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:34:33.387 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:34:33.387 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:34:33.387 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:33.387 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:34:33.387 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6801", 2021-02-10T11:34:33.387 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:34:33.387 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:33.388 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:33.388 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:34:33.388 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6803", 2021-02-10T11:34:33.388 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:34:33.388 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:34:33.388 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:34:33.388 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:33.388 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:34:33.388 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:34:33.388 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:33.388 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:34:33.388 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6805", 2021-02-10T11:34:33.388 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:34:33.389 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:33.389 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:33.389 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:34:33.389 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6807", 2021-02-10T11:34:33.389 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:34:33.389 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:34:33.389 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:34:33.389 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:33.389 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:34:33.389 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:34:33.389 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:33.389 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:34:33.389 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6813", 2021-02-10T11:34:33.390 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:34:33.390 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:33.390 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:33.390 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:34:33.390 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6815", 2021-02-10T11:34:33.390 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:34:33.390 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:34:33.390 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:34:33.390 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:33.390 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:34:33.390 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:34:33.390 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:33.390 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:34:33.391 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6809", 2021-02-10T11:34:33.391 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:34:33.391 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:33.391 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:33.391 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:34:33.391 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6811", 2021-02-10T11:34:33.391 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:34:33.391 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:34:33.391 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:34:33.391 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:33.391 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.112:6803/36031", 2021-02-10T11:34:33.391 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.112:6807/36031", 2021-02-10T11:34:33.391 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.112:6815/36031", 2021-02-10T11:34:33.391 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.112:6811/36031", 2021-02-10T11:34:33.392 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:34:33.392 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:34:33.392 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:34:33.392 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:34:33.392 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:33.392 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:33.392 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 3, 2021-02-10T11:34:33.392 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "58795682-e00d-4173-bb84-f7075ca89a11", 2021-02-10T11:34:33.392 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:34:33.392 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:34:33.393 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:34:33.393 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:34:33.393 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:34:33.393 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:34:33.393 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:34:33.393 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 44, 2021-02-10T11:34:33.393 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:34:33.393 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:34:33.393 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:34:33.393 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:34:33.393 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:33.393 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:34:33.393 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6816", 2021-02-10T11:34:33.393 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:34:33.394 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:33.394 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:33.394 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:34:33.394 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6819", 2021-02-10T11:34:33.394 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:34:33.394 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:34:33.394 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:34:33.394 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:33.394 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:34:33.394 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:34:33.394 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:33.394 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:34:33.395 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6821", 2021-02-10T11:34:33.395 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:34:33.395 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:33.395 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:33.395 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:34:33.395 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6823", 2021-02-10T11:34:33.395 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:34:33.395 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:34:33.395 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:34:33.395 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:33.395 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:34:33.395 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:34:33.395 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:33.396 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:34:33.396 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6829", 2021-02-10T11:34:33.396 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:34:33.396 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:33.396 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:33.396 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:34:33.396 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6831", 2021-02-10T11:34:33.396 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:34:33.396 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:34:33.396 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:34:33.396 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:33.396 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:34:33.396 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:34:33.397 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:33.397 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:34:33.397 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6825", 2021-02-10T11:34:33.397 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:34:33.397 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:33.397 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:33.397 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:34:33.397 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6827", 2021-02-10T11:34:33.397 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:34:33.397 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:34:33.397 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:34:33.397 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:33.397 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.112:6819/36028", 2021-02-10T11:34:33.398 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.112:6823/36028", 2021-02-10T11:34:33.398 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.112:6831/36028", 2021-02-10T11:34:33.398 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.112:6827/36028", 2021-02-10T11:34:33.398 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:34:33.398 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:34:33.398 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:34:33.398 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:34:33.398 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:33.398 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:33.398 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 4, 2021-02-10T11:34:33.398 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "6763f8d6-53dc-4e45-bdf6-a6eabcf81e39", 2021-02-10T11:34:33.398 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:34:33.398 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:34:33.399 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:34:33.399 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:34:33.399 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:34:33.399 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:34:33.399 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:34:33.399 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 51, 2021-02-10T11:34:33.399 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:34:33.399 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:34:33.399 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:34:33.399 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:34:33.399 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:33.399 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:34:33.400 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6810", 2021-02-10T11:34:33.400 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:34:33.400 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:33.400 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:33.400 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:34:33.400 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6813", 2021-02-10T11:34:33.400 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:34:33.400 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:34:33.400 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:34:33.400 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:33.400 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:34:33.400 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:34:33.400 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:33.400 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:34:33.401 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6816", 2021-02-10T11:34:33.401 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:34:33.401 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:33.401 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:33.401 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:34:33.401 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6819", 2021-02-10T11:34:33.401 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:34:33.401 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:34:33.401 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:34:33.401 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:33.401 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:34:33.401 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:34:33.401 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:33.402 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:34:33.402 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6825", 2021-02-10T11:34:33.402 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:34:33.402 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:33.402 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:33.402 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:34:33.402 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6827", 2021-02-10T11:34:33.402 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:34:33.402 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:34:33.402 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:34:33.402 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:33.402 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:34:33.402 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:34:33.403 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:33.403 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:34:33.403 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6821", 2021-02-10T11:34:33.403 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:34:33.403 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:33.403 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:33.403 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:34:33.403 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6823", 2021-02-10T11:34:33.403 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:34:33.403 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:34:33.403 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:34:33.403 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:33.403 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.120:6813/36388", 2021-02-10T11:34:33.404 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.120:6819/36388", 2021-02-10T11:34:33.404 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.120:6827/36388", 2021-02-10T11:34:33.404 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.120:6823/36388", 2021-02-10T11:34:33.404 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:34:33.404 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:34:33.404 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:34:33.404 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:34:33.404 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:33.404 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:33.404 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 5, 2021-02-10T11:34:33.404 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "5d0e4dd6-0cb0-4bf8-8314-626acaa1b544", 2021-02-10T11:34:33.404 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:34:33.404 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:34:33.405 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:34:33.405 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:34:33.405 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:34:33.405 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:34:33.405 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:34:33.405 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 53, 2021-02-10T11:34:33.405 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:34:33.405 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:34:33.405 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:34:33.405 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:34:33.405 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:33.405 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:34:33.405 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6804", 2021-02-10T11:34:33.406 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:34:33.406 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:33.406 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:33.406 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:34:33.406 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6806", 2021-02-10T11:34:33.406 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:34:33.406 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:34:33.406 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:34:33.406 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:33.406 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:34:33.406 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:34:33.406 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:33.407 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:34:33.407 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6809", 2021-02-10T11:34:33.407 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:34:33.407 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:33.407 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:33.407 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:34:33.407 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6812", 2021-02-10T11:34:33.407 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:34:33.407 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:34:33.407 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:34:33.407 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:33.407 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:34:33.407 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:34:33.408 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:33.408 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:34:33.408 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6820", 2021-02-10T11:34:33.408 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:34:33.408 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:33.408 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:33.408 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:34:33.408 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6822", 2021-02-10T11:34:33.408 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:34:33.408 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:34:33.408 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:34:33.408 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:33.408 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:34:33.409 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:34:33.409 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:33.409 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:34:33.409 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6815", 2021-02-10T11:34:33.409 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:34:33.409 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:33.409 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:33.409 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:34:33.409 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6818", 2021-02-10T11:34:33.409 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:34:33.409 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:34:33.409 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:34:33.409 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:33.410 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.120:6806/36387", 2021-02-10T11:34:33.410 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.120:6812/36387", 2021-02-10T11:34:33.410 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.120:6822/36387", 2021-02-10T11:34:33.410 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.120:6818/36387", 2021-02-10T11:34:33.410 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:34:33.410 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:34:33.410 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:34:33.410 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:34:33.410 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:33.410 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:33.410 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 6, 2021-02-10T11:34:33.410 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "b010e187-4541-4b00-80dd-f948f6ab8f15", 2021-02-10T11:34:33.410 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:34:33.411 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:34:33.411 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:34:33.411 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:34:33.411 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:34:33.411 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:34:33.411 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:34:33.411 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 53, 2021-02-10T11:34:33.411 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:34:33.411 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:34:33.411 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:34:33.411 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:34:33.411 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:33.411 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:34:33.412 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6824", 2021-02-10T11:34:33.412 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:34:33.412 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:33.412 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:33.412 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:34:33.412 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6826", 2021-02-10T11:34:33.412 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:34:33.412 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:34:33.412 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:34:33.412 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:33.412 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:34:33.412 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:34:33.412 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:33.413 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:34:33.413 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6828", 2021-02-10T11:34:33.413 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:34:33.413 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:33.413 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:33.413 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:34:33.413 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6829", 2021-02-10T11:34:33.413 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:34:33.413 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:34:33.413 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:34:33.413 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:33.413 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:34:33.413 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:34:33.414 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:33.414 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:34:33.414 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6832", 2021-02-10T11:34:33.414 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:34:33.414 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:33.414 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:33.414 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:34:33.414 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6833", 2021-02-10T11:34:33.414 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:34:33.414 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:34:33.414 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:34:33.414 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:33.415 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:34:33.415 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:34:33.415 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:33.415 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:34:33.415 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6830", 2021-02-10T11:34:33.415 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:34:33.415 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:33.415 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:33.415 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:34:33.415 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6831", 2021-02-10T11:34:33.415 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:34:33.415 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:34:33.415 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:34:33.416 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:33.416 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.120:6826/36385", 2021-02-10T11:34:33.416 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.120:6829/36385", 2021-02-10T11:34:33.416 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.120:6833/36385", 2021-02-10T11:34:33.416 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.120:6831/36385", 2021-02-10T11:34:33.416 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:34:33.416 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:34:33.416 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:34:33.416 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:34:33.416 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:33.416 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:33.416 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 7, 2021-02-10T11:34:33.416 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "671e2772-33b7-497e-8e44-2178bd850bcf", 2021-02-10T11:34:33.417 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:34:33.417 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:34:33.417 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:34:33.417 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:34:33.417 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:34:33.417 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:34:33.417 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:34:33.417 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 53, 2021-02-10T11:34:33.417 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:34:33.417 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:34:33.417 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:34:33.417 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:34:33.417 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:33.418 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:34:33.418 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6802", 2021-02-10T11:34:33.418 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:34:33.418 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:33.418 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:33.418 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:34:33.418 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6803", 2021-02-10T11:34:33.418 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:34:33.418 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:34:33.418 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:34:33.418 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:33.418 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:34:33.419 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:34:33.419 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:33.419 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:34:33.419 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6805", 2021-02-10T11:34:33.419 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:34:33.419 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:33.419 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:33.419 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:34:33.419 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6807", 2021-02-10T11:34:33.419 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:34:33.419 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:34:33.419 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:34:33.419 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:33.419 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:34:33.420 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:34:33.420 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:33.420 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:34:33.420 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6814", 2021-02-10T11:34:33.420 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:34:33.420 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:33.420 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:33.420 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:34:33.420 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6817", 2021-02-10T11:34:33.420 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:34:33.420 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:34:33.420 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:34:33.421 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:33.421 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:34:33.421 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:34:33.421 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:33.421 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:34:33.421 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6808", 2021-02-10T11:34:33.421 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:34:33.421 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:33.421 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:33.421 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:34:33.421 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6811", 2021-02-10T11:34:33.421 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:34:33.421 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:34:33.422 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:34:33.422 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:33.422 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.120:6803/36386", 2021-02-10T11:34:33.422 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.120:6807/36386", 2021-02-10T11:34:33.422 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.120:6817/36386", 2021-02-10T11:34:33.422 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.120:6811/36386", 2021-02-10T11:34:33.422 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:34:33.422 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:34:33.422 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:34:33.422 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:34:33.422 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:34:33.423 INFO:teuthology.orchestra.run.gibba012.stdout: ], 2021-02-10T11:34:33.423 INFO:teuthology.orchestra.run.gibba012.stdout: "osd_xinfo": [ 2021-02-10T11:34:33.423 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:33.423 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 0, 2021-02-10T11:34:33.423 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:34:33.423 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:34:33.423 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:34:33.423 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:34:33.423 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:34:33.423 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:33.423 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:33.423 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 1, 2021-02-10T11:34:33.423 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:34:33.424 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:34:33.424 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:34:33.424 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:34:33.424 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:34:33.424 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:33.424 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:33.424 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 2, 2021-02-10T11:34:33.424 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:34:33.424 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:34:33.424 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:34:33.424 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:34:33.424 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:34:33.424 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:33.425 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:33.425 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 3, 2021-02-10T11:34:33.425 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:34:33.425 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:34:33.425 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:34:33.425 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:34:33.425 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:34:33.425 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:33.425 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:33.425 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 4, 2021-02-10T11:34:33.425 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:34:33.425 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:34:33.425 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:34:33.426 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:34:33.426 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:34:33.426 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:33.426 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:33.426 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 5, 2021-02-10T11:34:33.426 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:34:33.426 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:34:33.426 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:34:33.426 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:34:33.426 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:34:33.426 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:33.426 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:33.426 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 6, 2021-02-10T11:34:33.427 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:34:33.427 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:34:33.427 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:34:33.427 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:34:33.427 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:34:33.427 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:33.427 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:33.427 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 7, 2021-02-10T11:34:33.427 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:34:33.427 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:34:33.427 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:34:33.427 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:34:33.427 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:34:33.428 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:34:33.428 INFO:teuthology.orchestra.run.gibba012.stdout: ], 2021-02-10T11:34:33.428 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_upmap": [], 2021-02-10T11:34:33.428 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_upmap_items": [], 2021-02-10T11:34:33.428 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_temp": [], 2021-02-10T11:34:33.428 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_temp": [], 2021-02-10T11:34:33.428 INFO:teuthology.orchestra.run.gibba012.stdout: "blacklist": {}, 2021-02-10T11:34:33.428 INFO:teuthology.orchestra.run.gibba012.stdout: "erasure_code_profiles": { 2021-02-10T11:34:33.428 INFO:teuthology.orchestra.run.gibba012.stdout: "default": { 2021-02-10T11:34:33.428 INFO:teuthology.orchestra.run.gibba012.stdout: "crush-failure-domain": "osd", 2021-02-10T11:34:33.428 INFO:teuthology.orchestra.run.gibba012.stdout: "k": "2", 2021-02-10T11:34:33.428 INFO:teuthology.orchestra.run.gibba012.stdout: "m": "1", 2021-02-10T11:34:33.428 INFO:teuthology.orchestra.run.gibba012.stdout: "plugin": "jerasure", 2021-02-10T11:34:33.429 INFO:teuthology.orchestra.run.gibba012.stdout: "ruleset-failure-domain": "osd", 2021-02-10T11:34:33.429 INFO:teuthology.orchestra.run.gibba012.stdout: "technique": "reed_sol_van" 2021-02-10T11:34:33.429 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:34:33.429 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:33.429 INFO:teuthology.orchestra.run.gibba012.stdout: "removed_snaps_queue": [], 2021-02-10T11:34:33.429 INFO:teuthology.orchestra.run.gibba012.stdout: "new_removed_snaps": [], 2021-02-10T11:34:33.429 INFO:teuthology.orchestra.run.gibba012.stdout: "new_purged_snaps": [], 2021-02-10T11:34:33.429 INFO:teuthology.orchestra.run.gibba012.stdout: "crush_node_flags": {}, 2021-02-10T11:34:33.429 INFO:teuthology.orchestra.run.gibba012.stdout: "device_class_flags": {} 2021-02-10T11:34:33.429 INFO:teuthology.orchestra.run.gibba012.stdout:} 2021-02-10T11:34:33.431 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.369 7ff515fbf700 1 -- 172.21.2.112:0/3769834560 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff4f40186d0 msgr2=0x7ff4f401abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:33.431 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.369 7ff515fbf700 1 --2- 172.21.2.112:0/3769834560 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff4f40186d0 0x7ff4f401abd0 crc :-1 s=READY pgs=666 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:33.431 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.369 7ff515fbf700 1 -- 172.21.2.112:0/3769834560 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff510091130 msgr2=0x7ff51019d140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:33.431 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.369 7ff515fbf700 1 --2- 172.21.2.112:0/3769834560 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff510091130 0x7ff51019d140 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 rx=0x7ff4f8006be0 tx=0x7ff4f800b990).stop 2021-02-10T11:34:33.431 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.369 7ff515fbf700 1 -- 172.21.2.112:0/3769834560 shutdown_connections 2021-02-10T11:34:33.431 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.369 7ff515fbf700 1 --2- 172.21.2.112:0/3769834560 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff4f40186d0 0x7ff4f401abd0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:33.431 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.369 7ff515fbf700 1 --2- 172.21.2.112:0/3769834560 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff51019d760 0x7ff51019db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:33.431 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.369 7ff515fbf700 1 --2- 172.21.2.112:0/3769834560 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff510091130 0x7ff51019d140 unknown :-1 s=CLOSED pgs=1350 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:33.431 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.369 7ff515fbf700 1 --2- 172.21.2.112:0/3769834560 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff5100edc10 0x7ff5100ee2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:33.432 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.369 7ff515fbf700 1 -- 172.21.2.112:0/3769834560 >> 172.21.2.112:0/3769834560 conn(0x7ff51019c5a0 msgr2=0x7ff5100ebe90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:33.432 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.369 7ff515fbf700 1 -- 172.21.2.112:0/3769834560 shutdown_connections 2021-02-10T11:34:33.432 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.369 7ff515fbf700 1 -- 172.21.2.112:0/3769834560 wait complete. 2021-02-10T11:34:33.432 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:34:33.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.855 7f8b81c35700 1 Processor -- start 2021-02-10T11:34:33.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.855 7f8b81c35700 1 -- start start 2021-02-10T11:34:33.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.856 7f8b81c35700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8b7c0d0340 0x7f8b7c0ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:33.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.856 7f8b81c35700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f8b7c0d1c70 0x7f8b7c0efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:33.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.856 7f8b81c35700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b7c0f0b10 con 0x7f8b7c0d1870 2021-02-10T11:34:33.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.856 7f8b81c35700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b7c0e5ed0 con 0x7f8b7c0d1c70 2021-02-10T11:34:33.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.856 7f8b81c35700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b7c1b0be0 con 0x7f8b7c0d0340 2021-02-10T11:34:33.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.856 7f8b7bfff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f8b7c0d1c70 0x7f8b7c0efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:33.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.856 7f8b7b7fe700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f8b7c0d1870 0x7f8b7c1b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:49900/0 (socket says 172.21.2.112:49900) 2021-02-10T11:34:33.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.856 7f8b7b7fe700 1 -- 172.21.2.112:0/1105589385 learned_addr learned my addr 172.21.2.112:0/1105589385 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:34:33.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.856 7f8b7bfff700 1 -- 172.21.2.112:0/1105589385 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8b7c0d0340 msgr2=0x7f8b7c0ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:33.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.856 7f8b7bfff700 1 --2- 172.21.2.112:0/1105589385 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8b7c0d0340 0x7f8b7c0ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:33.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.856 7f8b7bfff700 1 -- 172.21.2.112:0/1105589385 >> v1:172.21.2.112:6789/0 conn(0x7f8b7c0d1870 legacy=0x7f8b7c1b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:33.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.856 7f8b7bfff700 1 -- 172.21.2.112:0/1105589385 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b7c0f20a0 con 0x7f8b7c0d1c70 2021-02-10T11:34:33.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.856 7f8b7bfff700 1 --2- 172.21.2.112:0/1105589385 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8b7c0d1c70 0x7f8b7c0efc60 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 rx=0x7f8b74006e30 tx=0x7f8b74009e60).ready entity=mon.0 client_cookie=a0242b6eccb0cae5 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:33.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.857 7f8b7a7fc700 1 -- 172.21.2.112:0/1105589385 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8b74006a50 con 0x7f8b7c0d1c70 2021-02-10T11:34:33.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.857 7f8b7a7fc700 1 -- 172.21.2.112:0/1105589385 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b7400b300 con 0x7f8b7c0d1c70 2021-02-10T11:34:33.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.857 7f8b7a7fc700 1 -- 172.21.2.112:0/1105589385 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8b7400b820 con 0x7f8b7c0d1c70 2021-02-10T11:34:33.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.857 7f8b81c35700 1 -- 172.21.2.112:0/1105589385 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8b7c0d1c70 msgr2=0x7f8b7c0efc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:33.860 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.857 7f8b81c35700 1 --2- 172.21.2.112:0/1105589385 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8b7c0d1c70 0x7f8b7c0efc60 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 rx=0x7f8b74006e30 tx=0x7f8b74009e60).stop 2021-02-10T11:34:33.860 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.857 7f8b81c35700 1 -- 172.21.2.112:0/1105589385 shutdown_connections 2021-02-10T11:34:33.860 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.857 7f8b81c35700 1 --2- 172.21.2.112:0/1105589385 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8b7c0d1c70 0x7f8b7c0efc60 unknown :-1 s=CLOSED pgs=1352 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:33.860 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.857 7f8b81c35700 1 --2- 172.21.2.112:0/1105589385 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8b7c0d0340 0x7f8b7c0ef640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:33.860 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.857 7f8b81c35700 1 -- 172.21.2.112:0/1105589385 >> 172.21.2.112:0/1105589385 conn(0x7f8b7c0ce450 msgr2=0x7f8b7c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:33.860 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.857 7f8b81c35700 1 -- 172.21.2.112:0/1105589385 shutdown_connections 2021-02-10T11:34:33.860 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.857 7f8b81c35700 1 -- 172.21.2.112:0/1105589385 wait complete. 2021-02-10T11:34:33.861 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.872 7f8b81c35700 1 Processor -- start 2021-02-10T11:34:33.861 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.872 7f8b81c35700 1 -- start start 2021-02-10T11:34:33.861 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.872 7f8b81c35700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8b7c199690 0x7f8b7c0af0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:33.861 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.872 7f8b81c35700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8b7c0af710 0x7f8b7c1aca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:33.861 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.872 7f8b81c35700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8b7c0afb10 0x7f8b7c1ad060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:33.861 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.872 7f8b81c35700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b7c09a590 con 0x7f8b7c0af710 2021-02-10T11:34:33.861 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.872 7f8b81c35700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b7c09a7b0 con 0x7f8b7c199690 2021-02-10T11:34:33.861 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.872 7f8b81c35700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b7c1991e0 con 0x7f8b7c0afb10 2021-02-10T11:34:33.862 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.872 7f8b7affd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8b7c0af710 0x7f8b7c1aca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:33.862 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.872 7f8b7affd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8b7c0af710 0x7f8b7c1aca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:50448/0 (socket says 172.21.2.112:50448) 2021-02-10T11:34:33.862 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.872 7f8b7affd700 1 -- 172.21.2.112:0/1910957922 learned_addr learned my addr 172.21.2.112:0/1910957922 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:34:33.862 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.873 7f8b7affd700 1 -- 172.21.2.112:0/1910957922 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8b7c0afb10 msgr2=0x7f8b7c1ad060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:33.862 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.873 7f8b7affd700 1 --2- 172.21.2.112:0/1910957922 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8b7c0afb10 0x7f8b7c1ad060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:33.862 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.873 7f8b7affd700 1 -- 172.21.2.112:0/1910957922 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8b7c199690 msgr2=0x7f8b7c0af0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:33.862 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.873 7f8b7affd700 1 --2- 172.21.2.112:0/1910957922 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8b7c199690 0x7f8b7c0af0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:33.862 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.873 7f8b7affd700 1 -- 172.21.2.112:0/1910957922 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b7c1ad680 con 0x7f8b7c0af710 2021-02-10T11:34:33.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.873 7f8b7affd700 1 --2- 172.21.2.112:0/1910957922 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8b7c0af710 0x7f8b7c1aca40 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 rx=0x7f8b70006f00 tx=0x7f8b7000b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:33.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.873 7f8b63fff700 1 -- 172.21.2.112:0/1910957922 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8b7000bd30 con 0x7f8b7c0af710 2021-02-10T11:34:33.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.873 7f8b63fff700 1 -- 172.21.2.112:0/1910957922 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b70000c90 con 0x7f8b7c0af710 2021-02-10T11:34:33.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.873 7f8b63fff700 1 -- 172.21.2.112:0/1910957922 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8b700022a0 con 0x7f8b7c0af710 2021-02-10T11:34:33.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.873 7f8b81c35700 1 -- 172.21.2.112:0/1910957922 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b7c1ad680 con 0x7f8b7c0af710 2021-02-10T11:34:33.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.873 7f8b81c35700 1 -- 172.21.2.112:0/1910957922 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b7c1ae4a0 con 0x7f8b7c0af710 2021-02-10T11:34:33.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.874 7f8b63fff700 1 -- 172.21.2.112:0/1910957922 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8b70005310 con 0x7f8b7c0af710 2021-02-10T11:34:33.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.874 7f8b63fff700 1 --2- 172.21.2.112:0/1910957922 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8b5c018700 0x7f8b5c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:33.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.874 7f8b63fff700 1 -- 172.21.2.112:0/1910957922 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f8b70032780 con 0x7f8b7c0af710 2021-02-10T11:34:33.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.874 7f8b7b7fe700 1 --2- 172.21.2.112:0/1910957922 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8b5c018700 0x7f8b5c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:33.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.875 7f8b7b7fe700 1 --2- 172.21.2.112:0/1910957922 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8b5c018700 0x7f8b5c01abc0 crc :-1 s=READY pgs=667 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:33.892 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.905 7f8b81c35700 1 -- 172.21.2.112:0/1910957922 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b7c0ebc00 con 0x7f8b7c0af710 2021-02-10T11:34:33.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:33.908 7f8b63fff700 1 -- 172.21.2.112:0/1910957922 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8b70006b90 con 0x7f8b7c0af710 2021-02-10T11:34:34.024 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:34.037 7f8b81c35700 1 -- 172.21.2.112:0/1910957922 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8b7c0f0b10 con 0x7f8b7c0af710 2021-02-10T11:34:34.024 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:34.037 7f8b63fff700 1 -- 172.21.2.112:0/1910957922 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+2101 (secure 0 0 0) 0x7f8b7000bd30 con 0x7f8b7c0af710 2021-02-10T11:34:34.024 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 40 2021-02-10T11:34:34.025 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:34:34.025 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":40,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7881,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/1476007789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":1476007789},{"type":"v1","addr":"172.21.2.112:6833","nonce":1476007789}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":35}],"filesystems":[{"mdsmap":{"epoch":39,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:33:23.550791","modified":"2021-02-10 11:33:25.562811","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6595":{"gid":6595,"name":"b","rank":0,"incarnation":36,"state":"up:active","state_seq":3,"addr":"172.21.2.120:6835/4241229406","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":4241229406},{"type":"v1","addr":"172.21.2.120:6835","nonce":4241229406}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2021-02-10T11:34:34.026 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:34.039 7f8b81c35700 1 -- 172.21.2.112:0/1910957922 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8b5c018700 msgr2=0x7f8b5c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:34.026 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:34.039 7f8b81c35700 1 --2- 172.21.2.112:0/1910957922 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8b5c018700 0x7f8b5c01abc0 crc :-1 s=READY pgs=667 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:34.026 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:34.039 7f8b81c35700 1 -- 172.21.2.112:0/1910957922 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8b7c0af710 msgr2=0x7f8b7c1aca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:34.026 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:34.039 7f8b81c35700 1 --2- 172.21.2.112:0/1910957922 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8b7c0af710 0x7f8b7c1aca40 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 rx=0x7f8b70006f00 tx=0x7f8b7000b940).stop 2021-02-10T11:34:34.027 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:34.039 7f8b81c35700 1 -- 172.21.2.112:0/1910957922 shutdown_connections 2021-02-10T11:34:34.027 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:34.039 7f8b81c35700 1 --2- 172.21.2.112:0/1910957922 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8b5c018700 0x7f8b5c01abc0 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:34.027 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:34.039 7f8b81c35700 1 --2- 172.21.2.112:0/1910957922 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8b7c0afb10 0x7f8b7c1ad060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:34.027 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:34.039 7f8b81c35700 1 --2- 172.21.2.112:0/1910957922 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8b7c0af710 0x7f8b7c1aca40 unknown :-1 s=CLOSED pgs=1353 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:34.027 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:34.039 7f8b81c35700 1 --2- 172.21.2.112:0/1910957922 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8b7c199690 0x7f8b7c0af0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:34.027 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:34.039 7f8b81c35700 1 -- 172.21.2.112:0/1910957922 >> 172.21.2.112:0/1910957922 conn(0x7f8b7c1a63c0 msgr2=0x7f8b7c094040 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:34.027 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:34.039 7f8b81c35700 1 -- 172.21.2.112:0/1910957922 shutdown_connections 2021-02-10T11:34:34.027 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:34.039 7f8b81c35700 1 -- 172.21.2.112:0/1910957922 wait complete. 2021-02-10T11:34:34.035 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs fail cephfs 2021-02-10T11:34:34.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:34.459 7fc80da6e700 1 Processor -- start 2021-02-10T11:34:34.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:34.459 7fc80da6e700 1 -- start start 2021-02-10T11:34:34.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:34.460 7fc80da6e700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fc8080d1870 0x7fc8081b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:34.463 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:34.460 7fc80da6e700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc8081b0a10 0x7fc8080cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:34.463 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:34.460 7fc80da6e700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fc8080d1c70 0x7fc8080cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:34.463 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:34.460 7fc80da6e700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8080fe200 con 0x7fc8080d1c70 2021-02-10T11:34:34.463 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:34.460 7fc80da6e700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8080e5f10 con 0x7fc8080d1870 2021-02-10T11:34:34.463 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:34.460 7fc80da6e700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8080f0b50 con 0x7fc8081b0a10 2021-02-10T11:34:34.463 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:34.460 7fc807fff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fc8080d1c70 0x7fc8080cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:34.463 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:34.460 7fc807fff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fc8080d1c70 0x7fc8080cf950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:50454/0 (socket says 172.21.2.112:50454) 2021-02-10T11:34:34.463 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:34.460 7fc807fff700 1 -- 172.21.2.112:0/128770050 learned_addr learned my addr 172.21.2.112:0/128770050 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:34:34.463 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:34.460 7fc807fff700 1 -- 172.21.2.112:0/128770050 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc8081b0a10 msgr2=0x7fc8080cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:34.463 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:34.460 7fc807fff700 1 --2- 172.21.2.112:0/128770050 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc8081b0a10 0x7fc8080cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:34.463 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:34.460 7fc807fff700 1 -- 172.21.2.112:0/128770050 >> v2:172.21.2.120:3300/0 conn(0x7fc8080d1870 msgr2=0x7fc8081b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:34.463 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:34.460 7fc807fff700 1 --2- 172.21.2.112:0/128770050 >> v2:172.21.2.120:3300/0 conn(0x7fc8080d1870 0x7fc8081b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:34.464 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:34.460 7fc807fff700 1 -- 172.21.2.112:0/128770050 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8080cff70 con 0x7fc8080d1c70 2021-02-10T11:34:34.464 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:34.460 7fc807fff700 1 --2- 172.21.2.112:0/128770050 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc8080d1c70 0x7fc8080cf950 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 rx=0x7fc800006f00 tx=0x7fc800009f10).ready entity=mon.0 client_cookie=783913797578912f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:34.464 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:34.461 7fc8067fc700 1 -- 172.21.2.112:0/128770050 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc80000b3b0 con 0x7fc8080d1c70 2021-02-10T11:34:34.464 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:34.461 7fc8067fc700 1 -- 172.21.2.112:0/128770050 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc80000b730 con 0x7fc8080d1c70 2021-02-10T11:34:34.464 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:34.461 7fc8067fc700 1 -- 172.21.2.112:0/128770050 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc80000bc50 con 0x7fc8080d1c70 2021-02-10T11:34:34.464 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:34.461 7fc80da6e700 1 -- 172.21.2.112:0/128770050 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc8080d1c70 msgr2=0x7fc8080cf950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:34.464 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:34.461 7fc80da6e700 1 --2- 172.21.2.112:0/128770050 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc8080d1c70 0x7fc8080cf950 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 rx=0x7fc800006f00 tx=0x7fc800009f10).stop 2021-02-10T11:34:34.464 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:34.461 7fc80da6e700 1 -- 172.21.2.112:0/128770050 shutdown_connections 2021-02-10T11:34:34.464 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:34.461 7fc80da6e700 1 --2- 172.21.2.112:0/128770050 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc8080d1c70 0x7fc8080cf950 unknown :-1 s=CLOSED pgs=1354 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:34.464 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:34.461 7fc80da6e700 1 --2- 172.21.2.112:0/128770050 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc8081b0a10 0x7fc8080cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:34.464 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:34.461 7fc80da6e700 1 --2- 172.21.2.112:0/128770050 >> v2:172.21.2.120:3300/0 conn(0x7fc8080d1870 0x7fc8081b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:34.464 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:34.461 7fc80da6e700 1 -- 172.21.2.112:0/128770050 >> 172.21.2.112:0/128770050 conn(0x7fc8080ce450 msgr2=0x7fc8080d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:34.464 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:34.461 7fc80da6e700 1 -- 172.21.2.112:0/128770050 shutdown_connections 2021-02-10T11:34:34.464 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:34.462 7fc80da6e700 1 -- 172.21.2.112:0/128770050 wait complete. 2021-02-10T11:34:34.466 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:34.476 7fc80da6e700 1 Processor -- start 2021-02-10T11:34:34.466 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:34.476 7fc80da6e700 1 -- start start 2021-02-10T11:34:34.466 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:34.476 7fc80da6e700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc80818d0d0 0x7fc80818f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:34.466 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:34.476 7fc80da6e700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc8080ef250 0x7fc8081a0050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:34.466 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:34.476 7fc80da6e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc8080ef650 0x7fc808191840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:34.466 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:34.476 7fc80da6e700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc808191340 con 0x7fc8080ef650 2021-02-10T11:34:34.466 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:34.476 7fc80da6e700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc80818d510 con 0x7fc80818d0d0 2021-02-10T11:34:34.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:34.476 7fc80da6e700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8081a0670 con 0x7fc8080ef250 2021-02-10T11:34:34.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:34.477 7fc807fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc8080ef650 0x7fc808191840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:34.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:34.477 7fc807fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc8080ef650 0x7fc808191840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:50458/0 (socket says 172.21.2.112:50458) 2021-02-10T11:34:34.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:34.477 7fc807fff700 1 -- 172.21.2.112:0/2832257434 learned_addr learned my addr 172.21.2.112:0/2832257434 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:34:34.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:34.477 7fc807fff700 1 -- 172.21.2.112:0/2832257434 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc8080ef250 msgr2=0x7fc8081a0050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:34.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:34.477 7fc807fff700 1 --2- 172.21.2.112:0/2832257434 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc8080ef250 0x7fc8081a0050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:34.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:34.477 7fc807fff700 1 -- 172.21.2.112:0/2832257434 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc80818d0d0 msgr2=0x7fc80818f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:34.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:34.477 7fc807fff700 1 --2- 172.21.2.112:0/2832257434 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc80818d0d0 0x7fc80818f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:34.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:34.477 7fc807fff700 1 -- 172.21.2.112:0/2832257434 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8081a0890 con 0x7fc8080ef650 2021-02-10T11:34:34.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:34.477 7fc807fff700 1 --2- 172.21.2.112:0/2832257434 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc8080ef650 0x7fc808191840 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 rx=0x7fc800009980 tx=0x7fc80000bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:34.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:34.477 7fc7effff700 1 -- 172.21.2.112:0/2832257434 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc800005820 con 0x7fc8080ef650 2021-02-10T11:34:34.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:34.477 7fc7effff700 1 -- 172.21.2.112:0/2832257434 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc800005b20 con 0x7fc8080ef650 2021-02-10T11:34:34.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:34.477 7fc7effff700 1 -- 172.21.2.112:0/2832257434 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc800006030 con 0x7fc8080ef650 2021-02-10T11:34:34.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:34.477 7fc80da6e700 1 -- 172.21.2.112:0/2832257434 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8081a0890 con 0x7fc8080ef650 2021-02-10T11:34:34.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:34.477 7fc80da6e700 1 -- 172.21.2.112:0/2832257434 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc808192150 con 0x7fc8080ef650 2021-02-10T11:34:34.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:34.478 7fc7effff700 1 -- 172.21.2.112:0/2832257434 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc800005d00 con 0x7fc8080ef650 2021-02-10T11:34:34.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:34.478 7fc7effff700 1 --2- 172.21.2.112:0/2832257434 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc7e80186c0 0x7fc7e801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:34.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:34.478 7fc7effff700 1 -- 172.21.2.112:0/2832257434 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fc8000328f0 con 0x7fc8080ef650 2021-02-10T11:34:34.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:34.478 7fc8077fe700 1 --2- 172.21.2.112:0/2832257434 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc7e80186c0 0x7fc7e801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:34.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:34.479 7fc8077fe700 1 --2- 172.21.2.112:0/2832257434 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc7e80186c0 0x7fc7e801abc0 crc :-1 s=READY pgs=668 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:34.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:34.509 7fc80da6e700 1 -- 172.21.2.112:0/2832257434 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8080ee080 con 0x7fc8080ef650 2021-02-10T11:34:34.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:34.512 7fc7effff700 1 -- 172.21.2.112:0/2832257434 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc800027750 con 0x7fc8080ef650 2021-02-10T11:34:34.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:34.639 7fc80da6e700 1 -- 172.21.2.112:0/2832257434 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fc8080ce450 con 0x7fc8080ef650 2021-02-10T11:34:35.413 INFO:tasks.ceph.mon.a.gibba012.stderr:2021-02-10 11:34:35.426 7fc3fe6e2700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-02-10T11:34:35.418 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:35.432 7fc7effff700 1 -- 172.21.2.112:0/2832257434 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v41) v1 ==== 161+0+0 (secure 0 0 0) 0x7fc800027750 con 0x7fc8080ef650 2021-02-10T11:34:35.445 INFO:teuthology.orchestra.run.gibba012.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2021-02-10T11:34:35.447 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:35.460 7fc80da6e700 1 -- 172.21.2.112:0/2832257434 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc7e80186c0 msgr2=0x7fc7e801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:35.447 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:35.460 7fc80da6e700 1 --2- 172.21.2.112:0/2832257434 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc7e80186c0 0x7fc7e801abc0 crc :-1 s=READY pgs=668 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:35.447 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:35.460 7fc80da6e700 1 -- 172.21.2.112:0/2832257434 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc8080ef650 msgr2=0x7fc808191840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:35.447 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:35.460 7fc80da6e700 1 --2- 172.21.2.112:0/2832257434 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc8080ef650 0x7fc808191840 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 rx=0x7fc800009980 tx=0x7fc80000bcf0).stop 2021-02-10T11:34:35.447 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:35.460 7fc80da6e700 1 -- 172.21.2.112:0/2832257434 shutdown_connections 2021-02-10T11:34:35.447 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:35.460 7fc80da6e700 1 --2- 172.21.2.112:0/2832257434 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc7e80186c0 0x7fc7e801abc0 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:35.448 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:35.460 7fc80da6e700 1 --2- 172.21.2.112:0/2832257434 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc8080ef650 0x7fc808191840 unknown :-1 s=CLOSED pgs=1355 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:35.448 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:35.460 7fc80da6e700 1 --2- 172.21.2.112:0/2832257434 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc8080ef250 0x7fc8081a0050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:35.448 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:35.460 7fc80da6e700 1 --2- 172.21.2.112:0/2832257434 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc80818d0d0 0x7fc80818f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:35.448 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:35.460 7fc80da6e700 1 -- 172.21.2.112:0/2832257434 >> 172.21.2.112:0/2832257434 conn(0x7fc8080efc90 msgr2=0x7fc8080bf0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:35.448 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:35.461 7fc80da6e700 1 -- 172.21.2.112:0/2832257434 shutdown_connections 2021-02-10T11:34:35.448 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:35.461 7fc80da6e700 1 -- 172.21.2.112:0/2832257434 wait complete. 2021-02-10T11:34:35.456 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:34:35.469 INFO:tasks.ceph.mds.b.gibba020.stdout:starting mds.b at 2021-02-10T11:34:35.883 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:35.880 7f14ef3db700 1 Processor -- start 2021-02-10T11:34:35.883 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:35.880 7f14ef3db700 1 -- start start 2021-02-10T11:34:35.883 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:35.880 7f14ef3db700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f14e81b0870 0x7f14e80f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:35.883 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:35.880 7f14ef3db700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14e81b0c70 con 0x7f14e81b0470 2021-02-10T11:34:35.883 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:35.880 7f14ef3db700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14e80f0b50 con 0x7f14e80efe00 2021-02-10T11:34:35.883 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:35.880 7f14ef3db700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14e80e5ea0 con 0x7f14e81b0870 2021-02-10T11:34:35.883 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:35.880 7f14ec9f6700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f14e81b0870 0x7f14e80f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:35.883 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:35.880 7f14ed1f7700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f14e81b0470 0x7f14e80ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:49926/0 (socket says 172.21.2.112:49926) 2021-02-10T11:34:35.883 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:35.880 7f14ed1f7700 1 -- 172.21.2.112:0/15825240 learned_addr learned my addr 172.21.2.112:0/15825240 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:34:35.883 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:35.881 7f14ec9f6700 1 -- 172.21.2.112:0/15825240 >> v1:172.21.2.120:6789/0 conn(0x7f14e80efe00 legacy=0x7f14e80f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:35.884 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:35.881 7f14ec9f6700 1 -- 172.21.2.112:0/15825240 >> v1:172.21.2.112:6789/0 conn(0x7f14e81b0470 legacy=0x7f14e80ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:35.884 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:35.881 7f14ec9f6700 1 -- 172.21.2.112:0/15825240 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14e80e52e0 con 0x7f14e81b0870 2021-02-10T11:34:35.884 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:35.881 7f14ec9f6700 1 --2- 172.21.2.112:0/15825240 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f14e81b0870 0x7f14e80f20e0 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 rx=0x7f14d8006650 tx=0x7f14d8009a80).ready entity=mon.0 client_cookie=9b4901c3bf04397c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:35.884 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:35.881 7f14dffff700 1 -- 172.21.2.112:0/15825240 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f14d800c0c0 con 0x7f14e81b0870 2021-02-10T11:34:35.884 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:35.881 7f14dffff700 1 -- 172.21.2.112:0/15825240 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14d800c3c0 con 0x7f14e81b0870 2021-02-10T11:34:35.884 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:35.881 7f14dffff700 1 -- 172.21.2.112:0/15825240 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f14d800c8e0 con 0x7f14e81b0870 2021-02-10T11:34:35.884 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:35.881 7f14ef3db700 1 -- 172.21.2.112:0/15825240 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f14e81b0870 msgr2=0x7f14e80f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:35.884 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:35.881 7f14ef3db700 1 --2- 172.21.2.112:0/15825240 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f14e81b0870 0x7f14e80f20e0 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 rx=0x7f14d8006650 tx=0x7f14d8009a80).stop 2021-02-10T11:34:35.884 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:35.882 7f14ef3db700 1 -- 172.21.2.112:0/15825240 shutdown_connections 2021-02-10T11:34:35.884 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:35.882 7f14ef3db700 1 --2- 172.21.2.112:0/15825240 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f14e81b0870 0x7f14e80f20e0 unknown :-1 s=CLOSED pgs=1357 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:35.884 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:35.882 7f14ef3db700 1 -- 172.21.2.112:0/15825240 >> 172.21.2.112:0/15825240 conn(0x7f14e80ce3e0 msgr2=0x7f14e80e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:35.884 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:35.882 7f14ef3db700 1 -- 172.21.2.112:0/15825240 shutdown_connections 2021-02-10T11:34:35.885 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:35.882 7f14ef3db700 1 -- 172.21.2.112:0/15825240 wait complete. 2021-02-10T11:34:35.886 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:35.896 7f14ef3db700 1 Processor -- start 2021-02-10T11:34:35.886 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:35.896 7f14ef3db700 1 -- start start 2021-02-10T11:34:35.886 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:35.897 7f14ef3db700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f14e8092e30 0x7f14e81c6320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:35.886 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:35.897 7f14ef3db700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f14e80964a0 0x7f14e80968a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:35.887 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:35.897 7f14ef3db700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f14e8096ec0 0x7f14e8084f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:35.887 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:35.897 7f14ef3db700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14e80932b0 con 0x7f14e8096ec0 2021-02-10T11:34:35.887 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:35.897 7f14ef3db700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14e81b5f70 con 0x7f14e8092e30 2021-02-10T11:34:35.887 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:35.897 7f14ef3db700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14e81b6150 con 0x7f14e80964a0 2021-02-10T11:34:35.887 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:35.897 7f14ed1f7700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f14e8092e30 0x7f14e81c6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:35.887 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:35.897 7f14ed9f8700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f14e8096ec0 0x7f14e8084f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:35.887 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:35.897 7f14ed1f7700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f14e8092e30 0x7f14e81c6320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3300/0 says I am v2:172.21.2.112:52754/0 (socket says 172.21.2.112:52754) 2021-02-10T11:34:35.887 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:35.897 7f14ed1f7700 1 -- 172.21.2.112:0/1746781255 learned_addr learned my addr 172.21.2.112:0/1746781255 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:34:35.887 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:35.897 7f14ec9f6700 1 --2- 172.21.2.112:0/1746781255 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f14e80964a0 0x7f14e80968a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:35.887 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:35.897 7f14ed9f8700 1 -- 172.21.2.112:0/1746781255 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f14e80964a0 msgr2=0x7f14e80968a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:35.887 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:35.897 7f14ed9f8700 1 --2- 172.21.2.112:0/1746781255 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f14e80964a0 0x7f14e80968a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:35.887 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:35.897 7f14ed9f8700 1 -- 172.21.2.112:0/1746781255 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f14e8092e30 msgr2=0x7f14e81c6320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:35.887 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:35.897 7f14ed9f8700 1 --2- 172.21.2.112:0/1746781255 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f14e8092e30 0x7f14e81c6320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:35.888 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:35.897 7f14ed9f8700 1 -- 172.21.2.112:0/1746781255 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14e81c6970 con 0x7f14e8096ec0 2021-02-10T11:34:35.888 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:35.897 7f14ed1f7700 1 --2- 172.21.2.112:0/1746781255 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f14e8092e30 0x7f14e81c6320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:34:35.888 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:35.898 7f14ed9f8700 1 --2- 172.21.2.112:0/1746781255 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f14e8096ec0 0x7f14e8084f60 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 rx=0x7f14e0009fc0 tx=0x7f14e000bc90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:35.888 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:35.898 7f14ddffb700 1 -- 172.21.2.112:0/1746781255 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f14e0000ce0 con 0x7f14e8096ec0 2021-02-10T11:34:35.888 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:35.898 7f14ddffb700 1 -- 172.21.2.112:0/1746781255 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14e0006a30 con 0x7f14e8096ec0 2021-02-10T11:34:35.888 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:35.898 7f14ddffb700 1 -- 172.21.2.112:0/1746781255 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f14e000c4e0 con 0x7f14e8096ec0 2021-02-10T11:34:35.888 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:35.898 7f14ef3db700 1 -- 172.21.2.112:0/1746781255 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14e81c6970 con 0x7f14e8096ec0 2021-02-10T11:34:35.888 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:35.898 7f14ef3db700 1 -- 172.21.2.112:0/1746781255 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14e81c6970 con 0x7f14e8096ec0 2021-02-10T11:34:35.888 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:35.898 7f14ddffb700 1 -- 172.21.2.112:0/1746781255 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f14e0001220 con 0x7f14e8096ec0 2021-02-10T11:34:35.888 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:35.898 7f14ddffb700 1 --2- 172.21.2.112:0/1746781255 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f14c80186d0 0x7f14c801abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:35.888 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:35.898 7f14ddffb700 1 -- 172.21.2.112:0/1746781255 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5659+0+0 (secure 0 0 0) 0x7f14e0032b00 con 0x7f14e8096ec0 2021-02-10T11:34:35.888 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:35.899 7f14ed1f7700 1 --2- 172.21.2.112:0/1746781255 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f14c80186d0 0x7f14c801abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:35.889 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:35.899 7f14ed1f7700 1 --2- 172.21.2.112:0/1746781255 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f14c80186d0 0x7f14c801abd0 crc :-1 s=READY pgs=669 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:35.917 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:35.930 7f14ef3db700 1 -- 172.21.2.112:0/1746781255 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14e807ed50 con 0x7f14e8096ec0 2021-02-10T11:34:35.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:35.932 7f14ddffb700 1 -- 172.21.2.112:0/1746781255 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f14e000bdd0 con 0x7f14e8096ec0 2021-02-10T11:34:36.047 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:36.060 7f14ef3db700 1 -- 172.21.2.112:0/1746781255 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f14e80ce3e0 con 0x7f14e8096ec0 2021-02-10T11:34:36.047 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:36.061 7f14ddffb700 1 -- 172.21.2.112:0/1746781255 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+1756 (secure 0 0 0) 0x7f14e0001220 con 0x7f14e8096ec0 2021-02-10T11:34:36.048 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:34:36.048 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":41,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7881,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/1476007789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":1476007789},{"type":"v1","addr":"172.21.2.112:6833","nonce":1476007789}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":35}],"filesystems":[{"mdsmap":{"epoch":41,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:33:23.550791","modified":"2021-02-10 11:34:35.427181","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2021-02-10T11:34:36.048 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 41 2021-02-10T11:34:36.049 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:36.062 7f14ef3db700 1 -- 172.21.2.112:0/1746781255 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f14c80186d0 msgr2=0x7f14c801abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:36.049 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:36.062 7f14ef3db700 1 --2- 172.21.2.112:0/1746781255 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f14c80186d0 0x7f14c801abd0 crc :-1 s=READY pgs=669 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:36.049 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:36.062 7f14ef3db700 1 -- 172.21.2.112:0/1746781255 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f14e8096ec0 msgr2=0x7f14e8084f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:36.049 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:36.062 7f14ef3db700 1 --2- 172.21.2.112:0/1746781255 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f14e8096ec0 0x7f14e8084f60 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 rx=0x7f14e0009fc0 tx=0x7f14e000bc90).stop 2021-02-10T11:34:36.050 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:36.063 7f14ef3db700 1 -- 172.21.2.112:0/1746781255 shutdown_connections 2021-02-10T11:34:36.050 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:36.063 7f14ef3db700 1 --2- 172.21.2.112:0/1746781255 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f14c80186d0 0x7f14c801abd0 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:36.050 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:36.063 7f14ef3db700 1 --2- 172.21.2.112:0/1746781255 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f14e8096ec0 0x7f14e8084f60 unknown :-1 s=CLOSED pgs=1358 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:36.050 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:36.063 7f14ef3db700 1 --2- 172.21.2.112:0/1746781255 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f14e80964a0 0x7f14e80968a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:36.050 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:36.063 7f14ef3db700 1 --2- 172.21.2.112:0/1746781255 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f14e8092e30 0x7f14e81c6320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:36.050 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:36.063 7f14ef3db700 1 -- 172.21.2.112:0/1746781255 >> 172.21.2.112:0/1746781255 conn(0x7f14e80c3bd0 msgr2=0x7f14e8089210 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:36.050 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:36.063 7f14ef3db700 1 -- 172.21.2.112:0/1746781255 shutdown_connections 2021-02-10T11:34:36.050 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:36.063 7f14ef3db700 1 -- 172.21.2.112:0/1746781255 wait complete. 2021-02-10T11:34:36.059 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2021-02-10T11:34:36.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:36.487 7fbc615e0700 1 Processor -- start 2021-02-10T11:34:36.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:36.487 7fbc615e0700 1 -- start start 2021-02-10T11:34:36.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:36.487 7fbc615e0700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fbc5c0d1870 0x7fbc5c1b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:36.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:36.487 7fbc615e0700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc5c0f0b10 con 0x7fbc5c0f0650 2021-02-10T11:34:36.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:36.487 7fbc615e0700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc5c0d0310 con 0x7fbc5c1b0a10 2021-02-10T11:34:36.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:36.487 7fbc615e0700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc5c0d0530 con 0x7fbc5c0d1870 2021-02-10T11:34:36.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:36.487 7fbc5bac0700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fbc5c0f0650 0x7fbc5c0cfb20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:49940/0 (socket says 172.21.2.112:49940) 2021-02-10T11:34:36.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:36.487 7fbc5bac0700 1 -- 172.21.2.112:0/3739727922 learned_addr learned my addr 172.21.2.112:0/3739727922 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:34:36.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:36.488 7fbc5a2bd700 1 -- 172.21.2.112:0/3739727922 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3980288867 0 0) 0x7fbc54001500 con 0x7fbc5c0f0650 2021-02-10T11:34:36.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:36.488 7fbc5a2bd700 1 -- 172.21.2.112:0/3739727922 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc48002fe0 con 0x7fbc5c0f0650 2021-02-10T11:34:36.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:36.488 7fbc5b2bf700 1 --2- 172.21.2.112:0/3739727922 >> v2:172.21.2.120:3300/0 conn(0x7fbc5c0d1870 0x7fbc5c1b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:36.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:36.488 7fbc5a2bd700 1 -- 172.21.2.112:0/3739727922 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1303469080 0 0) 0x7fbc54001270 con 0x7fbc5c0f0650 2021-02-10T11:34:36.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:36.488 7fbc5a2bd700 1 -- 172.21.2.112:0/3739727922 >> v2:172.21.2.120:3300/0 conn(0x7fbc5c0d1870 msgr2=0x7fbc5c1b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:36.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:36.488 7fbc5a2bd700 1 --2- 172.21.2.112:0/3739727922 >> v2:172.21.2.120:3300/0 conn(0x7fbc5c0d1870 0x7fbc5c1b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:36.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:36.488 7fbc5a2bd700 1 -- 172.21.2.112:0/3739727922 >> v1:172.21.2.120:6789/0 conn(0x7fbc5c1b0a10 legacy=0x7fbc5c0cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:36.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:36.488 7fbc5a2bd700 1 -- 172.21.2.112:0/3739727922 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc5c0d1c70 con 0x7fbc5c0f0650 2021-02-10T11:34:36.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:36.488 7fbc5a2bd700 1 -- 172.21.2.112:0/3739727922 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7fbc54001960 con 0x7fbc5c0f0650 2021-02-10T11:34:36.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:36.488 7fbc5a2bd700 1 -- 172.21.2.112:0/3739727922 >> v1:172.21.2.112:6789/0 conn(0x7fbc5c0f0650 legacy=0x7fbc5c0cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:36.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:36.488 7fbc5a2bd700 1 --2- 172.21.2.112:0/3739727922 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fbc48003ac0 0x7fbc48005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:36.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:36.488 7fbc5a2bd700 1 --2- 172.21.2.112:0/3739727922 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbc48006770 0x7fbc48008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:36.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:36.488 7fbc5a2bd700 1 --2- 172.21.2.112:0/3739727922 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbc48009590 0x7fbc4800bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:36.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:36.488 7fbc5a2bd700 1 -- 172.21.2.112:0/3739727922 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc4800c300 con 0x7fbc48006770 2021-02-10T11:34:36.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:36.488 7fbc5a2bd700 1 -- 172.21.2.112:0/3739727922 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc4800c520 con 0x7fbc48003ac0 2021-02-10T11:34:36.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:36.488 7fbc5a2bd700 1 -- 172.21.2.112:0/3739727922 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc4800c740 con 0x7fbc48009590 2021-02-10T11:34:36.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:36.488 7fbc5aabe700 1 --2- 172.21.2.112:0/3739727922 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbc48006770 0x7fbc48008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:36.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:36.488 7fbc5b2bf700 1 --2- 172.21.2.112:0/3739727922 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fbc48003ac0 0x7fbc48005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:36.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:36.488 7fbc5bac0700 1 --2- 172.21.2.112:0/3739727922 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbc48009590 0x7fbc4800bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:36.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:36.489 7fbc5aabe700 1 -- 172.21.2.112:0/3739727922 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbc48009590 msgr2=0x7fbc4800bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:36.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:36.489 7fbc5aabe700 1 --2- 172.21.2.112:0/3739727922 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbc48009590 0x7fbc4800bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:36.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:36.489 7fbc5aabe700 1 -- 172.21.2.112:0/3739727922 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fbc48003ac0 msgr2=0x7fbc48005ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:36.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:36.489 7fbc5aabe700 1 --2- 172.21.2.112:0/3739727922 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fbc48003ac0 0x7fbc48005ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:36.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:36.489 7fbc5bac0700 1 -- 172.21.2.112:0/3739727922 reap_dead start 2021-02-10T11:34:36.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:36.489 7fbc5aabe700 1 -- 172.21.2.112:0/3739727922 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc480509b0 con 0x7fbc48006770 2021-02-10T11:34:36.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:36.489 7fbc5aabe700 1 --2- 172.21.2.112:0/3739727922 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbc48006770 0x7fbc48008c90 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 rx=0x7fbc44006ac0 tx=0x7fbc44009a40).ready entity=mon.0 client_cookie=ffb6f36a8975f65e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:36.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:36.489 7fbc5a2bd700 1 -- 172.21.2.112:0/3739727922 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbc44007280 con 0x7fbc48006770 2021-02-10T11:34:36.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:36.489 7fbc5a2bd700 1 -- 172.21.2.112:0/3739727922 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc44007600 con 0x7fbc48006770 2021-02-10T11:34:36.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:36.489 7fbc5a2bd700 1 -- 172.21.2.112:0/3739727922 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbc44007a50 con 0x7fbc48006770 2021-02-10T11:34:36.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:36.489 7fbc615e0700 1 -- 172.21.2.112:0/3739727922 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbc48006770 msgr2=0x7fbc48008c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:36.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:36.489 7fbc615e0700 1 --2- 172.21.2.112:0/3739727922 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbc48006770 0x7fbc48008c90 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 rx=0x7fbc44006ac0 tx=0x7fbc44009a40).stop 2021-02-10T11:34:36.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:36.489 7fbc615e0700 1 -- 172.21.2.112:0/3739727922 shutdown_connections 2021-02-10T11:34:36.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:36.489 7fbc615e0700 1 --2- 172.21.2.112:0/3739727922 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbc48006770 0x7fbc48008c90 unknown :-1 s=CLOSED pgs=1360 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:36.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:36.489 7fbc615e0700 1 -- 172.21.2.112:0/3739727922 >> 172.21.2.112:0/3739727922 conn(0x7fbc5c0ce450 msgr2=0x7fbc5c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:36.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:36.489 7fbc615e0700 1 -- 172.21.2.112:0/3739727922 shutdown_connections 2021-02-10T11:34:36.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:36.489 7fbc615e0700 1 -- 172.21.2.112:0/3739727922 wait complete. 2021-02-10T11:34:36.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:36.504 7fbc615e0700 1 Processor -- start 2021-02-10T11:34:36.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:36.504 7fbc615e0700 1 -- start start 2021-02-10T11:34:36.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:36.505 7fbc615e0700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fbc5c094310 0x7fbc5c0ebe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:36.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:36.505 7fbc615e0700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbc5c094710 0x7fbc5c0bca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:36.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:36.505 7fbc615e0700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbc5c0ec490 0x7fbc5c0bd050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:36.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:36.505 7fbc615e0700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc5c093d90 con 0x7fbc5c0ec490 2021-02-10T11:34:36.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:36.505 7fbc615e0700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc5c0bd670 con 0x7fbc5c094310 2021-02-10T11:34:36.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:36.505 7fbc615e0700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc5c0939f0 con 0x7fbc5c094710 2021-02-10T11:34:36.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:36.505 7fbc5bac0700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbc5c0ec490 0x7fbc5c0bd050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:36.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:36.505 7fbc5bac0700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbc5c0ec490 0x7fbc5c0bd050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:50486/0 (socket says 172.21.2.112:50486) 2021-02-10T11:34:36.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:36.505 7fbc5bac0700 1 -- 172.21.2.112:0/767313572 learned_addr learned my addr 172.21.2.112:0/767313572 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:34:36.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:36.505 7fbc5bac0700 1 -- 172.21.2.112:0/767313572 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbc5c094710 msgr2=0x7fbc5c0bca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:36.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:36.505 7fbc5bac0700 1 --2- 172.21.2.112:0/767313572 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbc5c094710 0x7fbc5c0bca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:36.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:36.505 7fbc5bac0700 1 -- 172.21.2.112:0/767313572 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fbc5c094310 msgr2=0x7fbc5c0ebe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:36.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:36.505 7fbc5bac0700 1 --2- 172.21.2.112:0/767313572 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fbc5c094310 0x7fbc5c0ebe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:36.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:36.505 7fbc5bac0700 1 -- 172.21.2.112:0/767313572 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc5c0bda80 con 0x7fbc5c0ec490 2021-02-10T11:34:36.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:36.505 7fbc5bac0700 1 --2- 172.21.2.112:0/767313572 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbc5c0ec490 0x7fbc5c0bd050 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 rx=0x7fbc54001520 tx=0x7fbc5400acb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:36.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:36.506 7fbc53fff700 1 -- 172.21.2.112:0/767313572 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbc54001d90 con 0x7fbc5c0ec490 2021-02-10T11:34:36.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:36.506 7fbc53fff700 1 -- 172.21.2.112:0/767313572 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc54004930 con 0x7fbc5c0ec490 2021-02-10T11:34:36.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:36.506 7fbc53fff700 1 -- 172.21.2.112:0/767313572 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbc54007020 con 0x7fbc5c0ec490 2021-02-10T11:34:36.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:36.506 7fbc615e0700 1 -- 172.21.2.112:0/767313572 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc5c0bda80 con 0x7fbc5c0ec490 2021-02-10T11:34:36.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:36.506 7fbc615e0700 1 -- 172.21.2.112:0/767313572 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc5c1a76d0 con 0x7fbc5c0ec490 2021-02-10T11:34:36.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:36.506 7fbc53fff700 1 -- 172.21.2.112:0/767313572 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fbc54001d60 con 0x7fbc5c0ec490 2021-02-10T11:34:36.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:36.506 7fbc53fff700 1 --2- 172.21.2.112:0/767313572 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fbc400186c0 0x7fbc4001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:36.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:36.506 7fbc53fff700 1 -- 172.21.2.112:0/767313572 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5659+0+0 (secure 0 0 0) 0x7fbc54032980 con 0x7fbc5c0ec490 2021-02-10T11:34:36.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:36.507 7fbc5b2bf700 1 --2- 172.21.2.112:0/767313572 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fbc400186c0 0x7fbc4001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:36.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:36.507 7fbc5b2bf700 1 --2- 172.21.2.112:0/767313572 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fbc400186c0 0x7fbc4001abc0 crc :-1 s=READY pgs=670 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:36.524 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:36.537 7fbc615e0700 1 -- 172.21.2.112:0/767313572 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc5c0ebc40 con 0x7fbc5c0ec490 2021-02-10T11:34:36.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:36.540 7fbc53fff700 1 -- 172.21.2.112:0/767313572 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fbc54004bd0 con 0x7fbc5c0ec490 2021-02-10T11:34:36.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:36.666 7fbc615e0700 1 -- 172.21.2.112:0/767313572 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fbc5c0f0b10 con 0x7fbc5c0ec490 2021-02-10T11:34:37.426 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:37.439 7fbc53fff700 1 -- 172.21.2.112:0/767313572 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v42) v1 ==== 104+0+0 (secure 0 0 0) 0x7fbc54001d60 con 0x7fbc5c0ec490 2021-02-10T11:34:37.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:37.487 7fbc615e0700 1 -- 172.21.2.112:0/767313572 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fbc400186c0 msgr2=0x7fbc4001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:37.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:37.487 7fbc615e0700 1 --2- 172.21.2.112:0/767313572 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fbc400186c0 0x7fbc4001abc0 crc :-1 s=READY pgs=670 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:37.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:37.488 7fbc615e0700 1 -- 172.21.2.112:0/767313572 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbc5c0ec490 msgr2=0x7fbc5c0bd050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:37.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:37.488 7fbc615e0700 1 --2- 172.21.2.112:0/767313572 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbc5c0ec490 0x7fbc5c0bd050 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 rx=0x7fbc54001520 tx=0x7fbc5400acb0).stop 2021-02-10T11:34:37.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:37.488 7fbc615e0700 1 -- 172.21.2.112:0/767313572 shutdown_connections 2021-02-10T11:34:37.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:37.488 7fbc615e0700 1 --2- 172.21.2.112:0/767313572 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fbc400186c0 0x7fbc4001abc0 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:37.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:37.488 7fbc615e0700 1 --2- 172.21.2.112:0/767313572 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbc5c0ec490 0x7fbc5c0bd050 unknown :-1 s=CLOSED pgs=1361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:37.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:37.488 7fbc615e0700 1 --2- 172.21.2.112:0/767313572 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbc5c094710 0x7fbc5c0bca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:37.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:37.488 7fbc615e0700 1 --2- 172.21.2.112:0/767313572 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fbc5c094310 0x7fbc5c0ebe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:37.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:37.488 7fbc615e0700 1 -- 172.21.2.112:0/767313572 >> 172.21.2.112:0/767313572 conn(0x7fbc5c08fa10 msgr2=0x7fbc5c0bfa60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:37.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:37.488 7fbc615e0700 1 -- 172.21.2.112:0/767313572 shutdown_connections 2021-02-10T11:34:37.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:37.488 7fbc615e0700 1 -- 172.21.2.112:0/767313572 wait complete. 2021-02-10T11:34:37.483 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2021-02-10T11:34:37.913 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:37.909 7f1d80b10700 1 Processor -- start 2021-02-10T11:34:37.913 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:37.909 7f1d80b10700 1 -- start start 2021-02-10T11:34:37.913 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:37.910 7f1d80b10700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f1d7c0d1870 0x7f1d7c1b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:37.913 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:37.910 7f1d80b10700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1d7c1b0ae0 0x7f1d7c108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:37.913 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:37.910 7f1d80b10700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f1d7c0d1c70 0x7f1d7c108f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:37.913 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:37.910 7f1d80b10700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d7c0fe1c0 con 0x7f1d7c0d1870 2021-02-10T11:34:37.914 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:37.910 7f1d80b10700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d7c109a60 con 0x7f1d7c0d1c70 2021-02-10T11:34:37.914 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:37.910 7f1d80b10700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d7c088070 con 0x7f1d7c1b0ae0 2021-02-10T11:34:37.914 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:37.910 7f1d7a85e700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f1d7c0d1870 0x7f1d7c1b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:37.914 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:37.910 7f1d7a85e700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f1d7c0d1870 0x7f1d7c1b04c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:50494/0 (socket says 172.21.2.112:50494) 2021-02-10T11:34:37.914 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:37.910 7f1d7a85e700 1 -- 172.21.2.112:0/1866452300 learned_addr learned my addr 172.21.2.112:0/1866452300 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:34:37.914 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:37.911 7f1d7a05d700 1 --2- 172.21.2.112:0/1866452300 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1d7c1b0ae0 0x7f1d7c108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:37.914 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:37.911 7f1d7a85e700 1 -- 172.21.2.112:0/1866452300 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1d7c1b0ae0 msgr2=0x7f1d7c108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:37.914 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:37.911 7f1d7a85e700 1 --2- 172.21.2.112:0/1866452300 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1d7c1b0ae0 0x7f1d7c108940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:37.914 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:37.911 7f1d7a85e700 1 -- 172.21.2.112:0/1866452300 >> v2:172.21.2.120:3300/0 conn(0x7f1d7c0d1c70 msgr2=0x7f1d7c108f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:37.914 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:37.911 7f1d7b05f700 1 --2- 172.21.2.112:0/1866452300 >> v2:172.21.2.120:3300/0 conn(0x7f1d7c0d1c70 0x7f1d7c108f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:37.914 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:37.911 7f1d7a85e700 1 --2- 172.21.2.112:0/1866452300 >> v2:172.21.2.120:3300/0 conn(0x7f1d7c0d1c70 0x7f1d7c108f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:37.914 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:37.911 7f1d7a85e700 1 -- 172.21.2.112:0/1866452300 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d7c109580 con 0x7f1d7c0d1870 2021-02-10T11:34:37.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:37.911 7f1d7a05d700 1 --2- 172.21.2.112:0/1866452300 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1d7c1b0ae0 0x7f1d7c108940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:34:37.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:37.911 7f1d7b05f700 1 --2- 172.21.2.112:0/1866452300 >> v2:172.21.2.120:3300/0 conn(0x7f1d7c0d1c70 0x7f1d7c108f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:34:37.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:37.911 7f1d7a85e700 1 --2- 172.21.2.112:0/1866452300 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1d7c0d1870 0x7f1d7c1b04c0 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 rx=0x7f1d6c006700 tx=0x7f1d6c009b80).ready entity=mon.0 client_cookie=69d0fdc088448fee server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:37.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:37.911 7f1d7985c700 1 -- 172.21.2.112:0/1866452300 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1d6c006340 con 0x7f1d7c0d1870 2021-02-10T11:34:37.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:37.911 7f1d7985c700 1 -- 172.21.2.112:0/1866452300 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d6c00c0c0 con 0x7f1d7c0d1870 2021-02-10T11:34:37.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:37.911 7f1d7985c700 1 -- 172.21.2.112:0/1866452300 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1d6c00c5e0 con 0x7f1d7c0d1870 2021-02-10T11:34:37.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:37.911 7f1d80b10700 1 -- 172.21.2.112:0/1866452300 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1d7c0d1870 msgr2=0x7f1d7c1b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:37.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:37.911 7f1d80b10700 1 --2- 172.21.2.112:0/1866452300 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1d7c0d1870 0x7f1d7c1b04c0 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 rx=0x7f1d6c006700 tx=0x7f1d6c009b80).stop 2021-02-10T11:34:37.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:37.912 7f1d80b10700 1 -- 172.21.2.112:0/1866452300 shutdown_connections 2021-02-10T11:34:37.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:37.912 7f1d80b10700 1 --2- 172.21.2.112:0/1866452300 >> v2:172.21.2.120:3300/0 conn(0x7f1d7c0d1c70 0x7f1d7c108f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:37.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:37.912 7f1d80b10700 1 --2- 172.21.2.112:0/1866452300 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1d7c1b0ae0 0x7f1d7c108940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:37.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:37.912 7f1d80b10700 1 --2- 172.21.2.112:0/1866452300 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1d7c0d1870 0x7f1d7c1b04c0 unknown :-1 s=CLOSED pgs=1362 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:37.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:37.912 7f1d80b10700 1 -- 172.21.2.112:0/1866452300 >> 172.21.2.112:0/1866452300 conn(0x7f1d7c0ce420 msgr2=0x7f1d7c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:37.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:37.912 7f1d80b10700 1 -- 172.21.2.112:0/1866452300 shutdown_connections 2021-02-10T11:34:37.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:37.912 7f1d80b10700 1 -- 172.21.2.112:0/1866452300 wait complete. 2021-02-10T11:34:37.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:37.926 7f1d80b10700 1 Processor -- start 2021-02-10T11:34:37.917 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:37.926 7f1d80b10700 1 -- start start 2021-02-10T11:34:37.917 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:37.927 7f1d80b10700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1d7c19f040 0x7f1d7c1a7510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:37.918 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:37.927 7f1d80b10700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1d7c0b1fb0 0x7f1d7c0b23b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:37.918 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:37.927 7f1d80b10700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1d7c082800 0x7f1d7c082c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:37.918 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:37.927 7f1d80b10700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d7c0833a0 con 0x7f1d7c0b1fb0 2021-02-10T11:34:37.918 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:37.927 7f1d80b10700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d7c1a6f90 con 0x7f1d7c082800 2021-02-10T11:34:37.918 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:37.927 7f1d80b10700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d7c19f650 con 0x7f1d7c19f040 2021-02-10T11:34:37.918 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:37.927 7f1d7a05d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1d7c0b1fb0 0x7f1d7c0b23b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:37.918 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:37.927 7f1d7a05d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1d7c0b1fb0 0x7f1d7c0b23b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:50498/0 (socket says 172.21.2.112:50498) 2021-02-10T11:34:37.918 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:37.927 7f1d7a05d700 1 -- 172.21.2.112:0/2033898072 learned_addr learned my addr 172.21.2.112:0/2033898072 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:34:37.918 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:37.928 7f1d7a05d700 1 -- 172.21.2.112:0/2033898072 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1d7c19f040 msgr2=0x7f1d7c1a7510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:37.918 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:37.928 7f1d7a85e700 1 --2- 172.21.2.112:0/2033898072 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1d7c19f040 0x7f1d7c1a7510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:37.918 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:37.928 7f1d7b05f700 1 --2- 172.21.2.112:0/2033898072 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1d7c082800 0x7f1d7c082c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:37.919 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:37.928 7f1d7a05d700 1 --2- 172.21.2.112:0/2033898072 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1d7c19f040 0x7f1d7c1a7510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:37.919 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:37.928 7f1d7a05d700 1 -- 172.21.2.112:0/2033898072 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1d7c082800 msgr2=0x7f1d7c082c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:37.919 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:37.928 7f1d7a05d700 1 --2- 172.21.2.112:0/2033898072 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1d7c082800 0x7f1d7c082c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:37.919 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:37.928 7f1d7a05d700 1 -- 172.21.2.112:0/2033898072 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d7c1a7b30 con 0x7f1d7c0b1fb0 2021-02-10T11:34:37.919 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:37.928 7f1d7a85e700 1 --2- 172.21.2.112:0/2033898072 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1d7c19f040 0x7f1d7c1a7510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:34:37.919 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:37.928 7f1d7b05f700 1 --2- 172.21.2.112:0/2033898072 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1d7c082800 0x7f1d7c082c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:34:37.919 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:37.928 7f1d7a05d700 1 --2- 172.21.2.112:0/2033898072 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1d7c0b1fb0 0x7f1d7c0b23b0 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 rx=0x7f1d64006f20 tx=0x7f1d6400b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:37.919 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:37.928 7f1d737fe700 1 -- 172.21.2.112:0/2033898072 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1d64006bb0 con 0x7f1d7c0b1fb0 2021-02-10T11:34:37.919 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:37.928 7f1d737fe700 1 -- 172.21.2.112:0/2033898072 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d6400bd80 con 0x7f1d7c0b1fb0 2021-02-10T11:34:37.919 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:37.928 7f1d80b10700 1 -- 172.21.2.112:0/2033898072 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d7c1a7b30 con 0x7f1d7c0b1fb0 2021-02-10T11:34:37.919 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:37.928 7f1d737fe700 1 -- 172.21.2.112:0/2033898072 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1d64002220 con 0x7f1d7c0b1fb0 2021-02-10T11:34:37.919 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:37.928 7f1d80b10700 1 -- 172.21.2.112:0/2033898072 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d7c1a5f60 con 0x7f1d7c0b1fb0 2021-02-10T11:34:37.919 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:37.929 7f1d737fe700 1 -- 172.21.2.112:0/2033898072 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1d64000c90 con 0x7f1d7c0b1fb0 2021-02-10T11:34:37.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:37.929 7f1d737fe700 1 --2- 172.21.2.112:0/2033898072 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1d600186c0 0x7f1d6001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:37.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:37.929 7f1d737fe700 1 -- 172.21.2.112:0/2033898072 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5659+0+0 (secure 0 0 0) 0x7f1d64032490 con 0x7f1d7c0b1fb0 2021-02-10T11:34:37.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:37.929 7f1d7a85e700 1 --2- 172.21.2.112:0/2033898072 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1d600186c0 0x7f1d6001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:37.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:37.930 7f1d7a85e700 1 --2- 172.21.2.112:0/2033898072 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1d600186c0 0x7f1d6001abc0 crc :-1 s=READY pgs=671 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:37.946 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:37.960 7f1d80b10700 1 -- 172.21.2.112:0/2033898072 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d7c0825d0 con 0x7f1d7c0b1fb0 2021-02-10T11:34:37.949 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:37.962 7f1d737fe700 1 -- 172.21.2.112:0/2033898072 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1d6402cf40 con 0x7f1d7c0b1fb0 2021-02-10T11:34:38.075 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:38.088 7f1d80b10700 1 -- 172.21.2.112:0/2033898072 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f1d7c0825d0 con 0x7f1d7c0b1fb0 2021-02-10T11:34:38.430 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:38.443 7f1d737fe700 1 -- 172.21.2.112:0/2033898072 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v57) v1 ==== 185+0+0 (secure 0 0 0) 0x7f1d6402cf40 con 0x7f1d7c0b1fb0 2021-02-10T11:34:38.442 INFO:teuthology.orchestra.run.gibba012.stderr:pool 'cephfs_metadata' removed 2021-02-10T11:34:38.445 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:38.458 7f1d80b10700 1 -- 172.21.2.112:0/2033898072 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1d600186c0 msgr2=0x7f1d6001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:38.445 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:38.458 7f1d80b10700 1 --2- 172.21.2.112:0/2033898072 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1d600186c0 0x7f1d6001abc0 crc :-1 s=READY pgs=671 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:38.445 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:38.458 7f1d80b10700 1 -- 172.21.2.112:0/2033898072 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1d7c0b1fb0 msgr2=0x7f1d7c0b23b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:38.445 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:38.458 7f1d80b10700 1 --2- 172.21.2.112:0/2033898072 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1d7c0b1fb0 0x7f1d7c0b23b0 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 rx=0x7f1d64006f20 tx=0x7f1d6400b990).stop 2021-02-10T11:34:38.445 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:38.458 7f1d80b10700 1 -- 172.21.2.112:0/2033898072 shutdown_connections 2021-02-10T11:34:38.445 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:38.458 7f1d80b10700 1 --2- 172.21.2.112:0/2033898072 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1d600186c0 0x7f1d6001abc0 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:38.446 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:38.458 7f1d80b10700 1 --2- 172.21.2.112:0/2033898072 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1d7c082800 0x7f1d7c082c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:38.446 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:38.458 7f1d80b10700 1 --2- 172.21.2.112:0/2033898072 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1d7c0b1fb0 0x7f1d7c0b23b0 unknown :-1 s=CLOSED pgs=1363 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:38.446 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:38.458 7f1d80b10700 1 --2- 172.21.2.112:0/2033898072 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1d7c19f040 0x7f1d7c1a7510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:38.446 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:38.458 7f1d80b10700 1 -- 172.21.2.112:0/2033898072 >> 172.21.2.112:0/2033898072 conn(0x7f1d7c1a2f90 msgr2=0x7f1d7c09aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:38.446 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:38.458 7f1d80b10700 1 -- 172.21.2.112:0/2033898072 shutdown_connections 2021-02-10T11:34:38.446 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:38.458 7f1d80b10700 1 -- 172.21.2.112:0/2033898072 wait complete. 2021-02-10T11:34:38.453 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2021-02-10T11:34:38.883 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:38.880 7f16d7113700 1 Processor -- start 2021-02-10T11:34:38.883 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:38.880 7f16d7113700 1 -- start start 2021-02-10T11:34:38.884 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:38.880 7f16d7113700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f16d00d1880 0x7f16d01b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:38.884 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:38.880 7f16d7113700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f16d00cff80 0x7f16d0109140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:38.884 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:38.880 7f16d7113700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16d0109a70 con 0x7f16d01b0af0 2021-02-10T11:34:38.884 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:38.880 7f16d7113700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16d0109760 con 0x7f16d00d1880 2021-02-10T11:34:38.884 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:38.880 7f16d7113700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16d0088070 con 0x7f16d00cff80 2021-02-10T11:34:38.884 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:38.880 7f16d4f2f700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f16d00d1880 0x7f16d01b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:38.884 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:38.880 7f16d4f2f700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f16d00d1880 0x7f16d01b04d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:50508/0 (socket says 172.21.2.112:50508) 2021-02-10T11:34:38.884 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:38.880 7f16d4f2f700 1 -- 172.21.2.112:0/2380349079 learned_addr learned my addr 172.21.2.112:0/2380349079 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:34:38.885 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:38.880 7f16d5730700 1 --2- 172.21.2.112:0/2380349079 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f16d00cff80 0x7f16d0109140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:38.885 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:38.881 7f16d4f2f700 1 -- 172.21.2.112:0/2380349079 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f16d00cff80 msgr2=0x7f16d0109140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:38.885 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:38.881 7f16d4f2f700 1 --2- 172.21.2.112:0/2380349079 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f16d00cff80 0x7f16d0109140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:38.885 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:38.881 7f16d4f2f700 1 -- 172.21.2.112:0/2380349079 >> v1:172.21.2.120:6789/0 conn(0x7f16d01b0af0 legacy=0x7f16d0108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:38.885 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:38.881 7f16d4f2f700 1 -- 172.21.2.112:0/2380349079 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16d00d1c80 con 0x7f16d00d1880 2021-02-10T11:34:38.885 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:38.881 7f16d4f2f700 1 --2- 172.21.2.112:0/2380349079 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f16d00d1880 0x7f16d01b04d0 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 rx=0x7f16cc006700 tx=0x7f16cc009b80).ready entity=mon.0 client_cookie=4e602677db7bddae server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:38.885 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:38.881 7f16c77fe700 1 -- 172.21.2.112:0/2380349079 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f16cc00c0c0 con 0x7f16d00d1880 2021-02-10T11:34:38.885 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:38.881 7f16c77fe700 1 -- 172.21.2.112:0/2380349079 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16cc00c340 con 0x7f16d00d1880 2021-02-10T11:34:38.885 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:38.881 7f16c77fe700 1 -- 172.21.2.112:0/2380349079 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f16cc00c860 con 0x7f16d00d1880 2021-02-10T11:34:38.885 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:38.881 7f16d7113700 1 -- 172.21.2.112:0/2380349079 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f16d00d1880 msgr2=0x7f16d01b04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:38.885 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:38.881 7f16d7113700 1 --2- 172.21.2.112:0/2380349079 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f16d00d1880 0x7f16d01b04d0 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 rx=0x7f16cc006700 tx=0x7f16cc009b80).stop 2021-02-10T11:34:38.886 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:38.882 7f16d7113700 1 -- 172.21.2.112:0/2380349079 shutdown_connections 2021-02-10T11:34:38.886 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:38.882 7f16d7113700 1 --2- 172.21.2.112:0/2380349079 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f16d00cff80 0x7f16d0109140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:38.886 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:38.882 7f16d7113700 1 --2- 172.21.2.112:0/2380349079 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f16d00d1880 0x7f16d01b04d0 unknown :-1 s=CLOSED pgs=1364 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:38.886 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:38.882 7f16d7113700 1 -- 172.21.2.112:0/2380349079 >> 172.21.2.112:0/2380349079 conn(0x7f16d00ce430 msgr2=0x7f16d00d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:38.886 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:38.882 7f16d7113700 1 -- 172.21.2.112:0/2380349079 shutdown_connections 2021-02-10T11:34:38.886 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:38.882 7f16d7113700 1 -- 172.21.2.112:0/2380349079 wait complete. 2021-02-10T11:34:38.886 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:38.896 7f16d7113700 1 Processor -- start 2021-02-10T11:34:38.886 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:38.896 7f16d7113700 1 -- start start 2021-02-10T11:34:38.888 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:38.897 7f16d7113700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f16d01b58d0 0x7f16d00aa4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:38.888 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:38.897 7f16d7113700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f16d00aaaf0 0x7f16d01b80a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:38.888 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:38.897 7f16d7113700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f16d01b86c0 0x7f16d019de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:38.888 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:38.897 7f16d7113700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16d01b8ac0 con 0x7f16d01b86c0 2021-02-10T11:34:38.888 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:38.897 7f16d7113700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16d01b8e20 con 0x7f16d01b58d0 2021-02-10T11:34:38.888 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:38.897 7f16d7113700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16d01b5cd0 con 0x7f16d00aaaf0 2021-02-10T11:34:38.888 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:38.897 7f16d5730700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f16d01b86c0 0x7f16d019de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:38.888 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:38.897 7f16d5730700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f16d01b86c0 0x7f16d019de70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:50514/0 (socket says 172.21.2.112:50514) 2021-02-10T11:34:38.888 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:38.897 7f16d5730700 1 -- 172.21.2.112:0/3085268336 learned_addr learned my addr 172.21.2.112:0/3085268336 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:34:38.888 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:38.897 7f16d5730700 1 -- 172.21.2.112:0/3085268336 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f16d00aaaf0 msgr2=0x7f16d01b80a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:34:38.889 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:38.897 7f16d5730700 1 --2- 172.21.2.112:0/3085268336 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f16d00aaaf0 0x7f16d01b80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:38.889 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:38.897 7f16d5730700 1 -- 172.21.2.112:0/3085268336 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f16d01b58d0 msgr2=0x7f16d00aa4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:38.889 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:38.897 7f16d5730700 1 --2- 172.21.2.112:0/3085268336 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f16d01b58d0 0x7f16d00aa4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:38.889 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:38.897 7f16d5730700 1 -- 172.21.2.112:0/3085268336 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16d0095710 con 0x7f16d01b86c0 2021-02-10T11:34:38.889 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:38.897 7f16d5730700 1 --2- 172.21.2.112:0/3085268336 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f16d01b86c0 0x7f16d019de70 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 rx=0x7f16c800b970 tx=0x7f16c800bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:38.889 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:38.898 7f16c57fa700 1 -- 172.21.2.112:0/3085268336 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f16c8000f90 con 0x7f16d01b86c0 2021-02-10T11:34:38.889 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:38.898 7f16c57fa700 1 -- 172.21.2.112:0/3085268336 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16c8001400 con 0x7f16d01b86c0 2021-02-10T11:34:38.889 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:38.898 7f16c57fa700 1 -- 172.21.2.112:0/3085268336 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f16c8001860 con 0x7f16d01b86c0 2021-02-10T11:34:38.889 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:38.898 7f16d7113700 1 -- 172.21.2.112:0/3085268336 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16d0095710 con 0x7f16d01b86c0 2021-02-10T11:34:38.889 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:38.898 7f16d7113700 1 -- 172.21.2.112:0/3085268336 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16d0095710 con 0x7f16d01b86c0 2021-02-10T11:34:38.889 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:38.898 7f16c57fa700 1 -- 172.21.2.112:0/3085268336 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f16c8000df0 con 0x7f16d01b86c0 2021-02-10T11:34:38.889 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:38.898 7f16c57fa700 1 --2- 172.21.2.112:0/3085268336 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f16b00186c0 0x7f16b001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:38.890 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:38.898 7f16c57fa700 1 -- 172.21.2.112:0/3085268336 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5221+0+0 (secure 0 0 0) 0x7f16c8032860 con 0x7f16d01b86c0 2021-02-10T11:34:38.890 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:38.899 7f16d4f2f700 1 --2- 172.21.2.112:0/3085268336 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f16b00186c0 0x7f16b001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:38.890 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:38.899 7f16d4f2f700 1 --2- 172.21.2.112:0/3085268336 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f16b00186c0 0x7f16b001abc0 crc :-1 s=READY pgs=672 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:38.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:38.930 7f16d7113700 1 -- 172.21.2.112:0/3085268336 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16d0082aa0 con 0x7f16d01b86c0 2021-02-10T11:34:38.919 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:38.932 7f16c57fa700 1 -- 172.21.2.112:0/3085268336 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f16c801f880 con 0x7f16d01b86c0 2021-02-10T11:34:39.045 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:39.057 7f16d7113700 1 -- 172.21.2.112:0/3085268336 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f16d0082aa0 con 0x7f16d01b86c0 2021-02-10T11:34:39.436 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:39.448 7f16c57fa700 1 -- 172.21.2.112:0/3085268336 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v58) v1 ==== 173+0+0 (secure 0 0 0) 0x7f16c8000df0 con 0x7f16d01b86c0 2021-02-10T11:34:39.462 INFO:teuthology.orchestra.run.gibba012.stderr:pool 'cephfs_data' removed 2021-02-10T11:34:39.463 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:39.476 7f16d7113700 1 -- 172.21.2.112:0/3085268336 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f16b00186c0 msgr2=0x7f16b001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:39.464 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:39.476 7f16d7113700 1 --2- 172.21.2.112:0/3085268336 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f16b00186c0 0x7f16b001abc0 crc :-1 s=READY pgs=672 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:39.464 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:39.476 7f16d7113700 1 -- 172.21.2.112:0/3085268336 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f16d01b86c0 msgr2=0x7f16d019de70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:39.464 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:39.476 7f16d7113700 1 --2- 172.21.2.112:0/3085268336 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f16d01b86c0 0x7f16d019de70 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 rx=0x7f16c800b970 tx=0x7f16c800bd10).stop 2021-02-10T11:34:39.464 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:39.476 7f16d7113700 1 -- 172.21.2.112:0/3085268336 shutdown_connections 2021-02-10T11:34:39.464 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:39.476 7f16d7113700 1 --2- 172.21.2.112:0/3085268336 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f16b00186c0 0x7f16b001abc0 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:39.464 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:39.476 7f16d7113700 1 --2- 172.21.2.112:0/3085268336 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f16d01b86c0 0x7f16d019de70 unknown :-1 s=CLOSED pgs=1365 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:39.464 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:39.476 7f16d7113700 1 --2- 172.21.2.112:0/3085268336 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f16d00aaaf0 0x7f16d01b80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:39.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:39.476 7f16d7113700 1 --2- 172.21.2.112:0/3085268336 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f16d01b58d0 0x7f16d00aa4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:39.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:39.476 7f16d7113700 1 -- 172.21.2.112:0/3085268336 >> 172.21.2.112:0/3085268336 conn(0x7f16d019f940 msgr2=0x7f16d018c290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:39.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:39.476 7f16d7113700 1 -- 172.21.2.112:0/3085268336 shutdown_connections 2021-02-10T11:34:39.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:39.476 7f16d7113700 1 -- 172.21.2.112:0/3085268336 wait complete. 2021-02-10T11:34:39.473 INFO:tasks.ceph.mds.a:Restarting daemon 2021-02-10T11:34:39.473 INFO:tasks.ceph.mds.a:Stopping old one... 2021-02-10T11:34:39.473 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2021-02-10T11:34:39.473 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:34:39.473 INFO:tasks.ceph.mds.b:Restarting daemon 2021-02-10T11:34:39.473 INFO:tasks.ceph.mds.b:Stopping old one... 2021-02-10T11:34:39.473 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2021-02-10T11:34:39.474 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:34:39.479 INFO:tasks.ceph.mds.a:Stopped 2021-02-10T11:34:39.479 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2021-02-10T11:34:39.480 INFO:tasks.ceph.mds.b:Stopped 2021-02-10T11:34:39.480 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2021-02-10T11:34:39.522 INFO:tasks.ceph.mds.a:Started 2021-02-10T11:34:39.523 DEBUG:teuthology.parallel:result is None 2021-02-10T11:34:39.523 INFO:tasks.ceph.mds.b:Started 2021-02-10T11:34:39.523 DEBUG:teuthology.parallel:result is None 2021-02-10T11:34:39.524 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd blacklist clear 2021-02-10T11:34:39.926 INFO:tasks.ceph.mds.b.gibba020.stdout:starting mds.b at 2021-02-10T11:34:39.929 INFO:tasks.ceph.mds.a.gibba012.stdout:starting mds.a at 2021-02-10T11:34:39.955 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:39.948 7fd4105dc700 1 Processor -- start 2021-02-10T11:34:39.955 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:39.948 7fd4105dc700 1 -- start start 2021-02-10T11:34:39.956 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:39.948 7fd4105dc700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fd4080d1870 0x7fd4081b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:39.956 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:39.948 7fd4105dc700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4080f0b10 con 0x7fd4081b0a10 2021-02-10T11:34:39.956 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:39.948 7fd4105dc700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4080d0310 con 0x7fd4080f0650 2021-02-10T11:34:39.956 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:39.948 7fd4105dc700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4080d0530 con 0x7fd4080d1870 2021-02-10T11:34:39.956 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:39.948 7fd40e3f8700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fd4080d1870 0x7fd4081b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:39.956 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:39.948 7fd40dbf7700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fd4081b0a10 0x7fd4080cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:49996/0 (socket says 172.21.2.112:49996) 2021-02-10T11:34:39.956 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:39.948 7fd40dbf7700 1 -- 172.21.2.112:0/2439564522 learned_addr learned my addr 172.21.2.112:0/2439564522 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:34:39.957 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:39.949 7fd40e3f8700 1 -- 172.21.2.112:0/2439564522 >> v1:172.21.2.120:6789/0 conn(0x7fd4080f0650 legacy=0x7fd4080cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:39.957 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:39.949 7fd40e3f8700 1 -- 172.21.2.112:0/2439564522 >> v1:172.21.2.112:6789/0 conn(0x7fd4081b0a10 legacy=0x7fd4080cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:39.957 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:39.949 7fd40d3f6700 1 -- 172.21.2.112:0/2439564522 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1424480934 0 0) 0x7fd3f80009b0 con 0x7fd4081b0a10 2021-02-10T11:34:39.957 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:39.949 7fd40e3f8700 1 -- 172.21.2.112:0/2439564522 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4080d1c70 con 0x7fd4080d1870 2021-02-10T11:34:39.957 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:39.949 7fd40e3f8700 1 --2- 172.21.2.112:0/2439564522 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd4080d1870 0x7fd4081b03f0 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 rx=0x7fd404006650 tx=0x7fd404009a00).ready entity=mon.0 client_cookie=2ed42e221cf16f29 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:39.957 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:39.949 7fd40d3f6700 1 -- 172.21.2.112:0/2439564522 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd404009df0 con 0x7fd4080d1870 2021-02-10T11:34:39.957 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:39.949 7fd40d3f6700 1 -- 172.21.2.112:0/2439564522 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4040009b0 con 0x7fd4080d1870 2021-02-10T11:34:39.957 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:39.949 7fd40d3f6700 1 -- 172.21.2.112:0/2439564522 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd40400c2b0 con 0x7fd4080d1870 2021-02-10T11:34:39.958 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:39.949 7fd4105dc700 1 -- 172.21.2.112:0/2439564522 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd4080d1870 msgr2=0x7fd4081b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:39.958 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:39.949 7fd4105dc700 1 --2- 172.21.2.112:0/2439564522 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd4080d1870 0x7fd4081b03f0 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 rx=0x7fd404006650 tx=0x7fd404009a00).stop 2021-02-10T11:34:39.958 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:39.950 7fd4105dc700 1 -- 172.21.2.112:0/2439564522 shutdown_connections 2021-02-10T11:34:39.958 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:39.950 7fd4105dc700 1 --2- 172.21.2.112:0/2439564522 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd4080d1870 0x7fd4081b03f0 unknown :-1 s=CLOSED pgs=1370 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:39.958 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:39.950 7fd4105dc700 1 -- 172.21.2.112:0/2439564522 >> 172.21.2.112:0/2439564522 conn(0x7fd4080ce450 msgr2=0x7fd4080d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:39.958 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:39.950 7fd4105dc700 1 -- 172.21.2.112:0/2439564522 shutdown_connections 2021-02-10T11:34:39.958 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:39.950 7fd4105dc700 1 -- 172.21.2.112:0/2439564522 wait complete. 2021-02-10T11:34:39.958 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:39.965 7fd4105dc700 1 Processor -- start 2021-02-10T11:34:39.958 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:39.965 7fd4105dc700 1 -- start start 2021-02-10T11:34:39.958 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:39.965 7fd4105dc700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd4080bb5d0 0x7fd4080bfac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:39.958 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:39.965 7fd4105dc700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd4080a9b50 0x7fd4080a9f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:39.959 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:39.965 7fd4105dc700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd4081a7550 0x7fd4081a7950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:39.959 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:39.965 7fd4105dc700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4080bb9d0 con 0x7fd4081a7550 2021-02-10T11:34:39.959 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:39.965 7fd4105dc700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4080aa570 con 0x7fd4080a9b50 2021-02-10T11:34:39.959 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:39.965 7fd4105dc700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4080aa790 con 0x7fd4080bb5d0 2021-02-10T11:34:39.959 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:39.965 7fd40ebf9700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd4081a7550 0x7fd4081a7950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:39.959 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:39.965 7fd40ebf9700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd4081a7550 0x7fd4081a7950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:50542/0 (socket says 172.21.2.112:50542) 2021-02-10T11:34:39.959 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:39.965 7fd40ebf9700 1 -- 172.21.2.112:0/2373675966 learned_addr learned my addr 172.21.2.112:0/2373675966 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:34:39.960 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:39.966 7fd40e3f8700 1 --2- 172.21.2.112:0/2373675966 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd4080bb5d0 0x7fd4080bfac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:39.960 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:39.966 7fd40dbf7700 1 --2- 172.21.2.112:0/2373675966 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd4080a9b50 0x7fd4080a9f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:39.960 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:39.966 7fd40ebf9700 1 -- 172.21.2.112:0/2373675966 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd4080bb5d0 msgr2=0x7fd4080bfac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:39.960 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:39.966 7fd40ebf9700 1 --2- 172.21.2.112:0/2373675966 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd4080bb5d0 0x7fd4080bfac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:39.960 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:39.966 7fd40ebf9700 1 -- 172.21.2.112:0/2373675966 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd4080a9b50 msgr2=0x7fd4080a9f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:39.960 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:39.966 7fd40ebf9700 1 --2- 172.21.2.112:0/2373675966 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd4080a9b50 0x7fd4080a9f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:39.960 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:39.966 7fd40ebf9700 1 -- 172.21.2.112:0/2373675966 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4081a7f70 con 0x7fd4081a7550 2021-02-10T11:34:39.961 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:39.966 7fd40ebf9700 1 --2- 172.21.2.112:0/2373675966 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd4081a7550 0x7fd4081a7950 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 rx=0x7fd400009940 tx=0x7fd40000bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:39.962 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:39.966 7fd3feffd700 1 -- 172.21.2.112:0/2373675966 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd400000f40 con 0x7fd4081a7550 2021-02-10T11:34:39.962 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:39.966 7fd3feffd700 1 -- 172.21.2.112:0/2373675966 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd40000c2f0 con 0x7fd4081a7550 2021-02-10T11:34:39.962 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:39.966 7fd3feffd700 1 -- 172.21.2.112:0/2373675966 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd40000c810 con 0x7fd4081a7550 2021-02-10T11:34:39.962 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:39.966 7fd4105dc700 1 -- 172.21.2.112:0/2373675966 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4081a7f70 con 0x7fd4081a7550 2021-02-10T11:34:39.962 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:39.966 7fd4105dc700 1 -- 172.21.2.112:0/2373675966 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4080bbe60 con 0x7fd4081a7550 2021-02-10T11:34:39.962 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:39.967 7fd3feffd700 1 -- 172.21.2.112:0/2373675966 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd400001220 con 0x7fd4081a7550 2021-02-10T11:34:39.962 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:39.967 7fd3feffd700 1 --2- 172.21.2.112:0/2373675966 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd3f00186c0 0x7fd3f001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:39.962 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:39.967 7fd3feffd700 1 -- 172.21.2.112:0/2373675966 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 4839+0+0 (secure 0 0 0) 0x7fd400032600 con 0x7fd4081a7550 2021-02-10T11:34:39.963 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:39.967 7fd40e3f8700 1 --2- 172.21.2.112:0/2373675966 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd3f00186c0 0x7fd3f001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:39.963 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:39.968 7fd40e3f8700 1 --2- 172.21.2.112:0/2373675966 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd3f00186c0 0x7fd3f001abc0 crc :-1 s=READY pgs=673 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:39.985 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:39.997 7fd4105dc700 1 -- 172.21.2.112:0/2373675966 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4080ee0b0 con 0x7fd4081a7550 2021-02-10T11:34:39.987 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:40.000 7fd3feffd700 1 -- 172.21.2.112:0/2373675966 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd400000f40 con 0x7fd4081a7550 2021-02-10T11:34:40.126 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:40.138 7fd4105dc700 1 -- 172.21.2.112:0/2373675966 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7fd4080ee0b0 con 0x7fd4081a7550 2021-02-10T11:34:40.481 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:40.493 7fd3feffd700 1 -- 172.21.2.112:0/2373675966 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v59) v1 ==== 97+0+0 (secure 0 0 0) 0x7fd40001f860 con 0x7fd4081a7550 2021-02-10T11:34:40.492 INFO:teuthology.orchestra.run.gibba012.stderr: removed all blacklist entries 2021-02-10T11:34:40.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:40.507 7fd4105dc700 1 -- 172.21.2.112:0/2373675966 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd3f00186c0 msgr2=0x7fd3f001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:40.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:40.507 7fd4105dc700 1 --2- 172.21.2.112:0/2373675966 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd3f00186c0 0x7fd3f001abc0 crc :-1 s=READY pgs=673 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:40.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:40.507 7fd4105dc700 1 -- 172.21.2.112:0/2373675966 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd4081a7550 msgr2=0x7fd4081a7950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:40.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:40.507 7fd4105dc700 1 --2- 172.21.2.112:0/2373675966 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd4081a7550 0x7fd4081a7950 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 rx=0x7fd400009940 tx=0x7fd40000bcf0).stop 2021-02-10T11:34:40.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:40.507 7fd4105dc700 1 -- 172.21.2.112:0/2373675966 shutdown_connections 2021-02-10T11:34:40.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:40.507 7fd4105dc700 1 --2- 172.21.2.112:0/2373675966 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd3f00186c0 0x7fd3f001abc0 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:40.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:40.507 7fd4105dc700 1 --2- 172.21.2.112:0/2373675966 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd4080a9b50 0x7fd4080a9f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:40.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:40.507 7fd4105dc700 1 --2- 172.21.2.112:0/2373675966 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd4081a7550 0x7fd4081a7950 unknown :-1 s=CLOSED pgs=1371 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:40.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:40.507 7fd4105dc700 1 --2- 172.21.2.112:0/2373675966 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd4080bb5d0 0x7fd4080bfac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:40.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:40.507 7fd4105dc700 1 -- 172.21.2.112:0/2373675966 >> 172.21.2.112:0/2373675966 conn(0x7fd4080a6eb0 msgr2=0x7fd4080bf2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:40.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:40.507 7fd4105dc700 1 -- 172.21.2.112:0/2373675966 shutdown_connections 2021-02-10T11:34:40.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:40.507 7fd4105dc700 1 -- 172.21.2.112:0/2373675966 wait complete. 2021-02-10T11:34:40.503 INFO:tasks.cephfs.cephfs_test_case:['0', '1', '2', '3'] 2021-02-10T11:34:40.504 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth ls --format=json-pretty 2021-02-10T11:34:40.929 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:40.924 7f34a442d700 1 Processor -- start 2021-02-10T11:34:40.929 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:40.924 7f34a442d700 1 -- start start 2021-02-10T11:34:40.929 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:40.925 7f34a442d700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f349c0d1870 0x7f349c1b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:40.930 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:40.925 7f34a442d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f349c1b0a10 0x7f349c0cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:40.930 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:40.925 7f34a442d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f349c0d1c70 0x7f349c0cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:40.930 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:40.925 7f34a442d700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f349c0e5ed0 con 0x7f349c1b0a10 2021-02-10T11:34:40.930 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:40.925 7f34a442d700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f349c0f0b10 con 0x7f349c0d1870 2021-02-10T11:34:40.930 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:40.925 7f34a442d700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f349c0cff70 con 0x7f349c0d1c70 2021-02-10T11:34:40.930 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:40.925 7f34a1a48700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f349c1b0a10 0x7f349c0cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:40.930 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:40.925 7f34a1a48700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f349c1b0a10 0x7f349c0cf330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:50546/0 (socket says 172.21.2.112:50546) 2021-02-10T11:34:40.930 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:40.925 7f34a1a48700 1 -- 172.21.2.112:0/3120721706 learned_addr learned my addr 172.21.2.112:0/3120721706 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:34:40.930 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:40.925 7f34a1a48700 1 -- 172.21.2.112:0/3120721706 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f349c0d1c70 msgr2=0x7f349c0cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:40.930 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:40.925 7f34a1a48700 1 --2- 172.21.2.112:0/3120721706 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f349c0d1c70 0x7f349c0cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:40.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:40.925 7f34a1a48700 1 -- 172.21.2.112:0/3120721706 >> v2:172.21.2.120:3300/0 conn(0x7f349c0d1870 msgr2=0x7f349c1b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:40.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:40.925 7f34a1a48700 1 --2- 172.21.2.112:0/3120721706 >> v2:172.21.2.120:3300/0 conn(0x7f349c0d1870 0x7f349c1b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:40.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:40.925 7f34a1a48700 1 -- 172.21.2.112:0/3120721706 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f349c0d0150 con 0x7f349c1b0a10 2021-02-10T11:34:40.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:40.925 7f34a1a48700 1 --2- 172.21.2.112:0/3120721706 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f349c1b0a10 0x7f349c0cf330 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 rx=0x7f3498006700 tx=0x7f3498009b80).ready entity=mon.0 client_cookie=9baabac15ced6ba7 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:40.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:40.926 7f34a1247700 1 -- 172.21.2.112:0/3120721706 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f349800c0c0 con 0x7f349c1b0a10 2021-02-10T11:34:40.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:40.926 7f34a1247700 1 -- 172.21.2.112:0/3120721706 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f349800c380 con 0x7f349c1b0a10 2021-02-10T11:34:40.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:40.926 7f34a1247700 1 -- 172.21.2.112:0/3120721706 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f349800c800 con 0x7f349c1b0a10 2021-02-10T11:34:40.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:40.926 7f34a442d700 1 -- 172.21.2.112:0/3120721706 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f349c1b0a10 msgr2=0x7f349c0cf330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:40.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:40.926 7f34a442d700 1 --2- 172.21.2.112:0/3120721706 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f349c1b0a10 0x7f349c0cf330 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 rx=0x7f3498006700 tx=0x7f3498009b80).stop 2021-02-10T11:34:40.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:40.927 7f34a442d700 1 -- 172.21.2.112:0/3120721706 shutdown_connections 2021-02-10T11:34:40.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:40.927 7f34a442d700 1 --2- 172.21.2.112:0/3120721706 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f349c0d1c70 0x7f349c0cf950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:40.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:40.927 7f34a442d700 1 --2- 172.21.2.112:0/3120721706 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f349c1b0a10 0x7f349c0cf330 unknown :-1 s=CLOSED pgs=1372 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:40.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:40.927 7f34a442d700 1 --2- 172.21.2.112:0/3120721706 >> v2:172.21.2.120:3300/0 conn(0x7f349c0d1870 0x7f349c1b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:40.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:40.927 7f34a442d700 1 -- 172.21.2.112:0/3120721706 >> 172.21.2.112:0/3120721706 conn(0x7f349c0ce450 msgr2=0x7f349c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:40.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:40.927 7f34a442d700 1 -- 172.21.2.112:0/3120721706 shutdown_connections 2021-02-10T11:34:40.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:40.927 7f34a442d700 1 -- 172.21.2.112:0/3120721706 wait complete. 2021-02-10T11:34:40.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:40.941 7f34a442d700 1 Processor -- start 2021-02-10T11:34:40.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:40.941 7f34a442d700 1 -- start start 2021-02-10T11:34:40.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:40.942 7f34a442d700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f349c1ac980 0x7f349c1ace80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:40.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:40.942 7f34a442d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f349c0aa400 0x7f349c1b72d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:40.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:40.942 7f34a442d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f349c0aa800 0x7f349c1b78f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:40.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:40.942 7f34a442d700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f349c1b6d50 con 0x7f349c0aa400 2021-02-10T11:34:40.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:40.942 7f34a442d700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f349c1ad4a0 con 0x7f349c1ac980 2021-02-10T11:34:40.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:40.942 7f34a442d700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f349c18f0e0 con 0x7f349c0aa800 2021-02-10T11:34:40.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:40.942 7f34a1a48700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f349c0aa400 0x7f349c1b72d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:40.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:40.942 7f34a1a48700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f349c0aa400 0x7f349c1b72d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:50554/0 (socket says 172.21.2.112:50554) 2021-02-10T11:34:40.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:40.942 7f34a1a48700 1 -- 172.21.2.112:0/1833793214 learned_addr learned my addr 172.21.2.112:0/1833793214 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:34:40.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:40.942 7f34a1a48700 1 -- 172.21.2.112:0/1833793214 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f349c0aa800 msgr2=0x7f349c1b78f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:40.935 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:40.942 7f34a1a48700 1 --2- 172.21.2.112:0/1833793214 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f349c0aa800 0x7f349c1b78f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:40.935 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:40.942 7f34a1a48700 1 -- 172.21.2.112:0/1833793214 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f349c1ac980 msgr2=0x7f349c1ace80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:40.935 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:40.942 7f34a1a48700 1 --2- 172.21.2.112:0/1833793214 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f349c1ac980 0x7f349c1ace80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:40.935 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:40.942 7f34a1a48700 1 -- 172.21.2.112:0/1833793214 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f349c1ac4c0 con 0x7f349c0aa400 2021-02-10T11:34:40.935 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:40.943 7f34a1a48700 1 --2- 172.21.2.112:0/1833793214 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f349c0aa400 0x7f349c1b72d0 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 rx=0x7f349800b5e0 tx=0x7f349800b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:40.935 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:40.943 7f348effd700 1 -- 172.21.2.112:0/1833793214 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f349800bd70 con 0x7f349c0aa400 2021-02-10T11:34:40.935 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:40.943 7f348effd700 1 -- 172.21.2.112:0/1833793214 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3498009440 con 0x7f349c0aa400 2021-02-10T11:34:40.935 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:40.943 7f34a442d700 1 -- 172.21.2.112:0/1833793214 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f349c1ac4c0 con 0x7f349c0aa400 2021-02-10T11:34:40.935 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:40.943 7f34a442d700 1 -- 172.21.2.112:0/1833793214 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f349c1b8200 con 0x7f349c0aa400 2021-02-10T11:34:40.935 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:40.943 7f348effd700 1 -- 172.21.2.112:0/1833793214 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3498002300 con 0x7f349c0aa400 2021-02-10T11:34:40.935 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:40.944 7f348effd700 1 -- 172.21.2.112:0/1833793214 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3498000c90 con 0x7f349c0aa400 2021-02-10T11:34:40.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:40.944 7f348effd700 1 --2- 172.21.2.112:0/1833793214 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3484018730 0x7f348401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:40.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:40.944 7f348effd700 1 -- 172.21.2.112:0/1833793214 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f34980324e0 con 0x7f349c0aa400 2021-02-10T11:34:40.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:40.944 7f34a2249700 1 --2- 172.21.2.112:0/1833793214 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3484018730 0x7f348401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:40.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:40.944 7f34a2249700 1 --2- 172.21.2.112:0/1833793214 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3484018730 0x7f348401ac10 crc :-1 s=READY pgs=674 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:40.962 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:40.975 7f34a442d700 1 -- 172.21.2.112:0/1833793214 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f349c0ebc60 con 0x7f349c0aa400 2021-02-10T11:34:40.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:40.977 7f348effd700 1 -- 172.21.2.112:0/1833793214 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3498006320 con 0x7f349c0aa400 2021-02-10T11:34:41.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:41.108 7f34a442d700 1 -- 172.21.2.112:0/1833793214 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f349c0f0b10 con 0x7f349c0aa400 2021-02-10T11:34:41.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:41.108 7f348effd700 1 -- 172.21.2.112:0/1833793214 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v37) v1 ==== 80+0+4513 (secure 0 0 0) 0x7f349800bd70 con 0x7f349c0aa400 2021-02-10T11:34:41.096 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:34:41.097 INFO:teuthology.orchestra.run.gibba012.stdout:{ 2021-02-10T11:34:41.097 INFO:teuthology.orchestra.run.gibba012.stdout: "auth_dump": [ 2021-02-10T11:34:41.097 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:41.097 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "mds.a", 2021-02-10T11:34:41.097 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCKvyNgPkV4AxAAEmVNaghdccP6lWd8hvQnxw==", 2021-02-10T11:34:41.097 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:34:41.097 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:34:41.097 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:34:41.097 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:34:41.098 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:34:41.098 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:34:41.098 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:41.098 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:41.098 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "mds.b", 2021-02-10T11:34:41.098 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCKvyNguxONLhAAHIpry+R89MAwdhFJyEhVCw==", 2021-02-10T11:34:41.098 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:34:41.098 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:34:41.099 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:34:41.099 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:34:41.099 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:34:41.099 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:34:41.099 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:41.099 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:41.099 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.0", 2021-02-10T11:34:41.099 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCSvyNgXcKPLhAAP/Bj2F+06DCtPCzQDNq+cQ==", 2021-02-10T11:34:41.099 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:34:41.099 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:34:41.099 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:34:41.099 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:34:41.100 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:34:41.100 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:41.100 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:41.100 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.1", 2021-02-10T11:34:41.100 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCVvyNghfbcLhAA66PrRwR2M7o6VTF5UhU/Fg==", 2021-02-10T11:34:41.100 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:34:41.100 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:34:41.100 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:34:41.101 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:34:41.101 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:34:41.101 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:41.101 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:41.101 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.2", 2021-02-10T11:34:41.101 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCYvyNgwiPqLhAA66BVkRsgaUlGpAsQ15Nf8Q==", 2021-02-10T11:34:41.101 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:34:41.101 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:34:41.101 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:34:41.101 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:34:41.101 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:34:41.101 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:41.102 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:41.102 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.3", 2021-02-10T11:34:41.102 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCbvyNgqBZrLxAAERPe/1L3SXAEzI3d9ReHHg==", 2021-02-10T11:34:41.102 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:34:41.102 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:34:41.102 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:34:41.102 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:34:41.103 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:34:41.103 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:41.103 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:41.103 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.4", 2021-02-10T11:34:41.103 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCkvyNgDQsxIhAAHmVjVYnx20N4SDmKqvbcuA==", 2021-02-10T11:34:41.103 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:34:41.104 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:34:41.104 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:34:41.104 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:34:41.104 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:34:41.104 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:41.104 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:41.104 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.5", 2021-02-10T11:34:41.104 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCnvyNg8ITFIhAADd4aKybAV7btoGPx3kCg3A==", 2021-02-10T11:34:41.104 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:34:41.104 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:34:41.104 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:34:41.104 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:34:41.104 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:34:41.105 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:41.105 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:41.105 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.6", 2021-02-10T11:34:41.105 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCqvyNgxOhDIxAAzXixEa/nlDc1xXrg0aeLwg==", 2021-02-10T11:34:41.105 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:34:41.105 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:34:41.106 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:34:41.106 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:34:41.106 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:34:41.106 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:41.106 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:41.106 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.7", 2021-02-10T11:34:41.106 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCtvyNgxSqGIxAACxJQlwP7DpiVVCZHA3zn1A==", 2021-02-10T11:34:41.106 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:34:41.106 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:34:41.106 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:34:41.106 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:34:41.107 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:34:41.107 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:41.107 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:41.107 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.0", 2021-02-10T11:34:41.107 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCLvyNgDkNiHRAAygU8Za4PBusTqCgwAYWTiQ==", 2021-02-10T11:34:41.107 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:34:41.107 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:34:41.107 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow r", 2021-02-10T11:34:41.107 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:34:41.107 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:34:41.107 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:41.108 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:41.108 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.1", 2021-02-10T11:34:41.108 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCLvyNgxmLqMxAA1MlSDJ9LH9u4f2sL7mm8/Q==", 2021-02-10T11:34:41.108 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:34:41.108 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:34:41.108 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow r", 2021-02-10T11:34:41.108 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:34:41.108 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:34:41.108 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:41.108 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:41.108 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.2", 2021-02-10T11:34:41.108 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCMvyNgp8veDhAAniSVHYPtwwTog/pjbaRQmw==", 2021-02-10T11:34:41.108 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:34:41.109 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:34:41.109 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow r", 2021-02-10T11:34:41.109 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:34:41.109 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:34:41.109 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:41.109 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:41.109 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.3", 2021-02-10T11:34:41.109 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCMvyNgQEVoJRAApHcWdm1sldIxu+heyD4HdQ==", 2021-02-10T11:34:41.109 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:34:41.109 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:34:41.110 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow r", 2021-02-10T11:34:41.110 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:34:41.110 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:34:41.110 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:41.110 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:41.110 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.admin", 2021-02-10T11:34:41.110 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCGvyNg7jjJBRAAtot+ypUZOiTo1GbY+oVhVw==", 2021-02-10T11:34:41.111 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:34:41.111 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow *", 2021-02-10T11:34:41.111 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:34:41.111 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:34:41.111 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:34:41.111 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:34:41.111 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:41.111 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:41.111 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "mgr.x", 2021-02-10T11:34:41.111 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCJvyNgTPVwKBAAjduiBE8+RSXBv82tdPecWg==", 2021-02-10T11:34:41.111 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:34:41.112 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow *", 2021-02-10T11:34:41.112 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow profile mgr", 2021-02-10T11:34:41.112 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:34:41.112 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:34:41.112 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:41.112 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:41.112 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "mgr.y", 2021-02-10T11:34:41.112 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCJvyNgQ1tTDhAASwBssDy+Rz31ujw8fNlWow==", 2021-02-10T11:34:41.112 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:34:41.112 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow *", 2021-02-10T11:34:41.112 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow profile mgr", 2021-02-10T11:34:41.112 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:34:41.112 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:34:41.113 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:34:41.113 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:34:41.113 INFO:teuthology.orchestra.run.gibba012.stdout:} 2021-02-10T11:34:41.114 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:41.110 7f34a442d700 1 -- 172.21.2.112:0/1833793214 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3484018730 msgr2=0x7f348401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:41.114 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:41.110 7f34a442d700 1 --2- 172.21.2.112:0/1833793214 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3484018730 0x7f348401ac10 crc :-1 s=READY pgs=674 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:41.114 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:41.110 7f34a442d700 1 -- 172.21.2.112:0/1833793214 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f349c0aa400 msgr2=0x7f349c1b72d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:41.114 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:41.110 7f34a442d700 1 --2- 172.21.2.112:0/1833793214 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f349c0aa400 0x7f349c1b72d0 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 rx=0x7f349800b5e0 tx=0x7f349800b980).stop 2021-02-10T11:34:41.114 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:41.110 7f34a442d700 1 -- 172.21.2.112:0/1833793214 shutdown_connections 2021-02-10T11:34:41.114 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:41.110 7f34a442d700 1 --2- 172.21.2.112:0/1833793214 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3484018730 0x7f348401ac10 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:41.115 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:41.110 7f34a442d700 1 --2- 172.21.2.112:0/1833793214 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f349c0aa800 0x7f349c1b78f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:41.115 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:41.110 7f34a442d700 1 --2- 172.21.2.112:0/1833793214 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f349c0aa400 0x7f349c1b72d0 unknown :-1 s=CLOSED pgs=1373 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:41.115 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:41.110 7f34a442d700 1 --2- 172.21.2.112:0/1833793214 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f349c1ac980 0x7f349c1ace80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:41.115 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:41.110 7f34a442d700 1 -- 172.21.2.112:0/1833793214 >> 172.21.2.112:0/1833793214 conn(0x7f349c1ac040 msgr2=0x7f349c094980 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:41.115 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:41.110 7f34a442d700 1 -- 172.21.2.112:0/1833793214 shutdown_connections 2021-02-10T11:34:41.115 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:41.110 7f34a442d700 1 -- 172.21.2.112:0/1833793214 wait complete. 2021-02-10T11:34:41.116 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:34:41.545 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:41.539 7f0d1bbf3700 1 Processor -- start 2021-02-10T11:34:41.545 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:41.539 7f0d1bbf3700 1 -- start start 2021-02-10T11:34:41.545 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:41.540 7f0d1bbf3700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0d140d1870 0x7f0d141b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:41.545 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:41.540 7f0d1bbf3700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d140f0b10 con 0x7f0d141b0a10 2021-02-10T11:34:41.545 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:41.540 7f0d1bbf3700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d140d0310 con 0x7f0d140f0650 2021-02-10T11:34:41.545 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:41.540 7f0d1bbf3700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d140e5ed0 con 0x7f0d140d1870 2021-02-10T11:34:41.545 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:41.540 7f0d1920e700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f0d141b0a10 0x7f0d140cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:50020/0 (socket says 172.21.2.112:50020) 2021-02-10T11:34:41.545 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:41.540 7f0d1920e700 1 -- 172.21.2.112:0/2629670323 learned_addr learned my addr 172.21.2.112:0/2629670323 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:34:41.545 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:41.540 7f0d19a0f700 1 --2- 172.21.2.112:0/2629670323 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0d140d1870 0x7f0d141b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:41.546 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:41.540 7f0d18a0d700 1 -- 172.21.2.112:0/2629670323 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3188160581 0 0) 0x7f0d100010b0 con 0x7f0d141b0a10 2021-02-10T11:34:41.546 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:41.540 7f0d18a0d700 1 -- 172.21.2.112:0/2629670323 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d00002fe0 con 0x7f0d141b0a10 2021-02-10T11:34:41.546 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:41.541 7f0d18a0d700 1 -- 172.21.2.112:0/2629670323 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1346057523 0 0) 0x7f0d100010b0 con 0x7f0d141b0a10 2021-02-10T11:34:41.546 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:41.541 7f0d18a0d700 1 -- 172.21.2.112:0/2629670323 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0d140d1870 msgr2=0x7f0d141b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:41.546 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:41.541 7f0d18a0d700 1 --2- 172.21.2.112:0/2629670323 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0d140d1870 0x7f0d141b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:41.546 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:41.541 7f0d18a0d700 1 -- 172.21.2.112:0/2629670323 >> v1:172.21.2.120:6789/0 conn(0x7f0d140f0650 legacy=0x7f0d140cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:41.546 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:41.541 7f0d18a0d700 1 -- 172.21.2.112:0/2629670323 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d140d0530 con 0x7f0d141b0a10 2021-02-10T11:34:41.546 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:41.541 7f0d18a0d700 1 -- 172.21.2.112:0/2629670323 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f0d100015b0 con 0x7f0d141b0a10 2021-02-10T11:34:41.546 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:41.541 7f0d18a0d700 1 -- 172.21.2.112:0/2629670323 >> v1:172.21.2.112:6789/0 conn(0x7f0d141b0a10 legacy=0x7f0d140cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:41.546 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:41.541 7f0d18a0d700 1 --2- 172.21.2.112:0/2629670323 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0d00003af0 0x7f0d00005f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:41.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:41.541 7f0d18a0d700 1 --2- 172.21.2.112:0/2629670323 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0d000067a0 0x7f0d00008c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:41.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:41.541 7f0d18a0d700 1 --2- 172.21.2.112:0/2629670323 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0d00009510 0x7f0d0000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:41.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:41.541 7f0d18a0d700 1 -- 172.21.2.112:0/2629670323 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d0000c2c0 con 0x7f0d00009510 2021-02-10T11:34:41.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:41.541 7f0d18a0d700 1 -- 172.21.2.112:0/2629670323 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d0000c4e0 con 0x7f0d00003af0 2021-02-10T11:34:41.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:41.541 7f0d18a0d700 1 -- 172.21.2.112:0/2629670323 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d0000c700 con 0x7f0d000067a0 2021-02-10T11:34:41.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:41.541 7f0d1a210700 1 --2- 172.21.2.112:0/2629670323 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0d00009510 0x7f0d0000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:41.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:41.541 7f0d19a0f700 1 --2- 172.21.2.112:0/2629670323 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0d00003af0 0x7f0d00005f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:41.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:41.541 7f0d1920e700 1 --2- 172.21.2.112:0/2629670323 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0d000067a0 0x7f0d00008c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:41.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:41.541 7f0d1a210700 1 -- 172.21.2.112:0/2629670323 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0d000067a0 msgr2=0x7f0d00008c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:41.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:41.541 7f0d1a210700 1 --2- 172.21.2.112:0/2629670323 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0d000067a0 0x7f0d00008c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:41.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:41.541 7f0d1a210700 1 -- 172.21.2.112:0/2629670323 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0d00003af0 msgr2=0x7f0d00005f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:41.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:41.541 7f0d1a210700 1 --2- 172.21.2.112:0/2629670323 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0d00003af0 0x7f0d00005f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:41.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:41.541 7f0d1a210700 1 -- 172.21.2.112:0/2629670323 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d00050970 con 0x7f0d00009510 2021-02-10T11:34:41.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:41.541 7f0d1a210700 1 --2- 172.21.2.112:0/2629670323 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0d00009510 0x7f0d0000ba10 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 rx=0x7f0d0c006e30 tx=0x7f0d0c009e60).ready entity=mon.0 client_cookie=b71cb5c11f5115ce server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:41.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:41.542 7f0d18a0d700 1 -- 172.21.2.112:0/2629670323 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0d0c00b300 con 0x7f0d00009510 2021-02-10T11:34:41.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:41.542 7f0d18a0d700 1 -- 172.21.2.112:0/2629670323 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d0c00b5c0 con 0x7f0d00009510 2021-02-10T11:34:41.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:41.542 7f0d18a0d700 1 -- 172.21.2.112:0/2629670323 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0d0c00ba20 con 0x7f0d00009510 2021-02-10T11:34:41.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:41.542 7f0d1bbf3700 1 -- 172.21.2.112:0/2629670323 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0d00009510 msgr2=0x7f0d0000ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:41.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:41.542 7f0d1bbf3700 1 --2- 172.21.2.112:0/2629670323 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0d00009510 0x7f0d0000ba10 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 rx=0x7f0d0c006e30 tx=0x7f0d0c009e60).stop 2021-02-10T11:34:41.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:41.542 7f0d1a210700 1 -- 172.21.2.112:0/2629670323 reap_dead start 2021-02-10T11:34:41.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:41.542 7f0d1bbf3700 1 -- 172.21.2.112:0/2629670323 shutdown_connections 2021-02-10T11:34:41.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:41.542 7f0d1bbf3700 1 -- 172.21.2.112:0/2629670323 >> 172.21.2.112:0/2629670323 conn(0x7f0d140ce450 msgr2=0x7f0d140d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:41.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:41.542 7f0d1bbf3700 1 -- 172.21.2.112:0/2629670323 shutdown_connections 2021-02-10T11:34:41.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:41.542 7f0d1bbf3700 1 -- 172.21.2.112:0/2629670323 wait complete. 2021-02-10T11:34:41.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:41.557 7f0d1bbf3700 1 Processor -- start 2021-02-10T11:34:41.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:41.557 7f0d1bbf3700 1 -- start start 2021-02-10T11:34:41.550 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:41.557 7f0d1bbf3700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0d140c37f0 0x7f0d140907f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:41.550 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:41.557 7f0d1bbf3700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0d14090e10 0x7f0d140a3bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:41.550 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:41.557 7f0d1bbf3700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0d14091210 0x7f0d140a4210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:41.550 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:41.557 7f0d1bbf3700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d14091830 con 0x7f0d14091210 2021-02-10T11:34:41.550 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:41.557 7f0d1bbf3700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d1409e5d0 con 0x7f0d140c37f0 2021-02-10T11:34:41.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:41.557 7f0d1bbf3700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d140c3bf0 con 0x7f0d14090e10 2021-02-10T11:34:41.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:41.558 7f0d1a210700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0d14091210 0x7f0d140a4210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:41.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:41.558 7f0d1920e700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0d14090e10 0x7f0d140a3bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:41.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:41.558 7f0d1a210700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0d14091210 0x7f0d140a4210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:50570/0 (socket says 172.21.2.112:50570) 2021-02-10T11:34:41.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:41.558 7f0d1920e700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0d14090e10 0x7f0d140a3bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3301/0 says I am v2:172.21.2.112:50248/0 (socket says 172.21.2.112:50248) 2021-02-10T11:34:41.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:41.558 7f0d1a210700 1 -- 172.21.2.112:0/260014911 learned_addr learned my addr 172.21.2.112:0/260014911 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:34:41.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:41.558 7f0d19a0f700 1 --2- 172.21.2.112:0/260014911 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0d140c37f0 0x7f0d140907f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:41.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:41.558 7f0d1a210700 1 -- 172.21.2.112:0/260014911 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0d14090e10 msgr2=0x7f0d140a3bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:41.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:41.558 7f0d1a210700 1 --2- 172.21.2.112:0/260014911 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0d14090e10 0x7f0d140a3bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:41.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:41.558 7f0d1a210700 1 -- 172.21.2.112:0/260014911 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0d140c37f0 msgr2=0x7f0d140907f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:41.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:41.558 7f0d1a210700 1 --2- 172.21.2.112:0/260014911 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0d140c37f0 0x7f0d140907f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:41.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:41.558 7f0d1a210700 1 -- 172.21.2.112:0/260014911 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d140a4830 con 0x7f0d14091210 2021-02-10T11:34:41.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:41.558 7f0d1920e700 1 --2- 172.21.2.112:0/260014911 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0d14090e10 0x7f0d140a3bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:34:41.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:41.558 7f0d19a0f700 1 --2- 172.21.2.112:0/260014911 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0d140c37f0 0x7f0d140907f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:34:41.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:41.558 7f0d1a210700 1 --2- 172.21.2.112:0/260014911 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0d14091210 0x7f0d140a4210 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 rx=0x7f0d0c00b970 tx=0x7f0d0c00bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:41.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:41.559 7f0d067fc700 1 -- 172.21.2.112:0/260014911 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0d0c0093d0 con 0x7f0d14091210 2021-02-10T11:34:41.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:41.559 7f0d067fc700 1 -- 172.21.2.112:0/260014911 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d0c007240 con 0x7f0d14091210 2021-02-10T11:34:41.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:41.559 7f0d1bbf3700 1 -- 172.21.2.112:0/260014911 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d140a4830 con 0x7f0d14091210 2021-02-10T11:34:41.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:41.559 7f0d067fc700 1 -- 172.21.2.112:0/260014911 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0d0c0076e0 con 0x7f0d14091210 2021-02-10T11:34:41.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:41.559 7f0d1bbf3700 1 -- 172.21.2.112:0/260014911 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d140a4830 con 0x7f0d14091210 2021-02-10T11:34:41.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:41.559 7f0d067fc700 1 -- 172.21.2.112:0/260014911 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0d0c007420 con 0x7f0d14091210 2021-02-10T11:34:41.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:41.559 7f0d067fc700 1 --2- 172.21.2.112:0/260014911 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0cfc018730 0x7f0cfc01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:41.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:41.560 7f0d067fc700 1 -- 172.21.2.112:0/260014911 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f0d0c032840 con 0x7f0d14091210 2021-02-10T11:34:41.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:41.560 7f0d19a0f700 1 --2- 172.21.2.112:0/260014911 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0cfc018730 0x7f0cfc01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:41.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:41.560 7f0d19a0f700 1 --2- 172.21.2.112:0/260014911 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0cfc018730 0x7f0cfc01ac10 crc :-1 s=READY pgs=675 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:41.577 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:41.590 7f0d1bbf3700 1 -- 172.21.2.112:0/260014911 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d140ebc80 con 0x7f0d14091210 2021-02-10T11:34:41.580 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:41.592 7f0d067fc700 1 -- 172.21.2.112:0/260014911 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0d0c009080 con 0x7f0d14091210 2021-02-10T11:34:41.704 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:41.716 7f0d1bbf3700 1 -- 172.21.2.112:0/260014911 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0d140f0b10 con 0x7f0d14091210 2021-02-10T11:34:41.704 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:41.717 7f0d067fc700 1 -- 172.21.2.112:0/260014911 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v59) v1 ==== 74+0+11145 (secure 0 0 0) 0x7f0d0c01f8b0 con 0x7f0d14091210 2021-02-10T11:34:41.706 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:34:41.706 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":59,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:34:40.490124","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:34:41.707 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:41.719 7f0d1bbf3700 1 -- 172.21.2.112:0/260014911 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0cfc018730 msgr2=0x7f0cfc01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:41.707 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:41.719 7f0d1bbf3700 1 --2- 172.21.2.112:0/260014911 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0cfc018730 0x7f0cfc01ac10 crc :-1 s=READY pgs=675 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:41.707 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:41.719 7f0d1bbf3700 1 -- 172.21.2.112:0/260014911 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0d14091210 msgr2=0x7f0d140a4210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:41.707 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:41.719 7f0d1bbf3700 1 --2- 172.21.2.112:0/260014911 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0d14091210 0x7f0d140a4210 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 rx=0x7f0d0c00b970 tx=0x7f0d0c00bd10).stop 2021-02-10T11:34:41.707 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:41.719 7f0d1bbf3700 1 -- 172.21.2.112:0/260014911 shutdown_connections 2021-02-10T11:34:41.707 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:41.719 7f0d1bbf3700 1 --2- 172.21.2.112:0/260014911 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0cfc018730 0x7f0cfc01ac10 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:41.707 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:41.719 7f0d1bbf3700 1 --2- 172.21.2.112:0/260014911 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0d14091210 0x7f0d140a4210 unknown :-1 s=CLOSED pgs=1376 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:41.707 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:41.719 7f0d1bbf3700 1 --2- 172.21.2.112:0/260014911 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0d14090e10 0x7f0d140a3bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:41.707 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:41.719 7f0d1bbf3700 1 --2- 172.21.2.112:0/260014911 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0d140c37f0 0x7f0d140907f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:41.707 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:41.719 7f0d1bbf3700 1 -- 172.21.2.112:0/260014911 >> 172.21.2.112:0/260014911 conn(0x7f0d140a2b90 msgr2=0x7f0d1408b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:41.708 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:41.719 7f0d1bbf3700 1 -- 172.21.2.112:0/260014911 shutdown_connections 2021-02-10T11:34:41.708 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:41.719 7f0d1bbf3700 1 -- 172.21.2.112:0/260014911 wait complete. 2021-02-10T11:34:41.715 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:13:57.508758', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2021-02-10T11:34:41.716 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:34:42.146 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.141 7feb5f5da700 1 Processor -- start 2021-02-10T11:34:42.146 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.141 7feb5f5da700 1 -- start start 2021-02-10T11:34:42.146 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.142 7feb5f5da700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7feb580d1870 0x7feb581b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:42.146 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.142 7feb5f5da700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7feb580cff70 0x7feb58109130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:42.146 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.142 7feb5f5da700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb58109a60 con 0x7feb581b0ae0 2021-02-10T11:34:42.146 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.142 7feb5f5da700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb58109750 con 0x7feb580cff70 2021-02-10T11:34:42.146 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.142 7feb5f5da700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb58088070 con 0x7feb580d1870 2021-02-10T11:34:42.146 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.142 7feb5d3f6700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7feb580d1870 0x7feb581b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:42.146 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.142 7feb5cbf5700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7feb581b0ae0 0x7feb58108940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:43596/0 (socket says 172.21.2.112:43596) 2021-02-10T11:34:42.147 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.142 7feb5cbf5700 1 -- 172.21.2.112:0/3605756711 learned_addr learned my addr 172.21.2.112:0/3605756711 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:34:42.147 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.142 7feb5dbf7700 1 --2- 172.21.2.112:0/3605756711 >> v2:172.21.2.120:3300/0 conn(0x7feb580cff70 0x7feb58109130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:42.147 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.143 7feb4ffff700 1 -- 172.21.2.112:0/3605756711 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1907019802 0 0) 0x7feb480010b0 con 0x7feb581b0ae0 2021-02-10T11:34:42.147 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.143 7feb4ffff700 1 -- 172.21.2.112:0/3605756711 --> v1:172.21.2.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb44002fe0 con 0x7feb581b0ae0 2021-02-10T11:34:42.147 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.143 7feb5dbf7700 1 -- 172.21.2.112:0/3605756711 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7feb580d1870 msgr2=0x7feb581b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:42.147 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.143 7feb5dbf7700 1 --2- 172.21.2.112:0/3605756711 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7feb580d1870 0x7feb581b04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:42.147 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.143 7feb5dbf7700 1 -- 172.21.2.112:0/3605756711 >> v1:172.21.2.120:6789/0 conn(0x7feb581b0ae0 legacy=0x7feb58108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:42.147 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.143 7feb5dbf7700 1 -- 172.21.2.112:0/3605756711 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb58088260 con 0x7feb580cff70 2021-02-10T11:34:42.147 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.143 7feb5dbf7700 1 --2- 172.21.2.112:0/3605756711 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7feb580cff70 0x7feb58109130 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 rx=0x7feb54006ad0 tx=0x7feb54009ad0).ready entity=mon.1 client_cookie=b46378db0b9bc687 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:42.147 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.143 7feb4ffff700 1 -- 172.21.2.112:0/3605756711 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feb540022e0 con 0x7feb580cff70 2021-02-10T11:34:42.147 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.143 7feb4ffff700 1 -- 172.21.2.112:0/3605756711 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb54006410 con 0x7feb580cff70 2021-02-10T11:34:42.147 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.143 7feb4ffff700 1 -- 172.21.2.112:0/3605756711 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feb54002620 con 0x7feb580cff70 2021-02-10T11:34:42.147 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.143 7feb5f5da700 1 -- 172.21.2.112:0/3605756711 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7feb580cff70 msgr2=0x7feb58109130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:42.148 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.143 7feb5f5da700 1 --2- 172.21.2.112:0/3605756711 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7feb580cff70 0x7feb58109130 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 rx=0x7feb54006ad0 tx=0x7feb54009ad0).stop 2021-02-10T11:34:42.148 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.144 7feb5f5da700 1 -- 172.21.2.112:0/3605756711 shutdown_connections 2021-02-10T11:34:42.148 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.144 7feb5f5da700 1 --2- 172.21.2.112:0/3605756711 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7feb580cff70 0x7feb58109130 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:42.148 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.144 7feb5f5da700 1 --2- 172.21.2.112:0/3605756711 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7feb580d1870 0x7feb581b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:42.148 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.144 7feb5f5da700 1 -- 172.21.2.112:0/3605756711 >> 172.21.2.112:0/3605756711 conn(0x7feb580ce420 msgr2=0x7feb580d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:42.148 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.144 7feb5f5da700 1 -- 172.21.2.112:0/3605756711 shutdown_connections 2021-02-10T11:34:42.148 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.144 7feb5f5da700 1 -- 172.21.2.112:0/3605756711 wait complete. 2021-02-10T11:34:42.150 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.158 7feb5f5da700 1 Processor -- start 2021-02-10T11:34:42.150 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.158 7feb5f5da700 1 -- start start 2021-02-10T11:34:42.150 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.159 7feb5f5da700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7feb580c1850 0x7feb580c1f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:42.150 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.159 7feb5f5da700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7feb580c2810 0x7feb580c2c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:42.150 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.159 7feb5f5da700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7feb580c3230 0x7feb580c3630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:42.150 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.159 7feb5f5da700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb580c1c50 con 0x7feb580c2810 2021-02-10T11:34:42.150 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.159 7feb5f5da700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb58084920 con 0x7feb580c1850 2021-02-10T11:34:42.150 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.159 7feb5f5da700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb58084b40 con 0x7feb580c3230 2021-02-10T11:34:42.150 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.159 7feb5d3f6700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7feb580c1850 0x7feb580c1f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:42.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.159 7feb5cbf5700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7feb580c2810 0x7feb580c2c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:42.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.159 7feb5d3f6700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7feb580c1850 0x7feb580c1f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3300/0 says I am v2:172.21.2.112:52860/0 (socket says 172.21.2.112:52860) 2021-02-10T11:34:42.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.159 7feb5d3f6700 1 -- 172.21.2.112:0/1053703473 learned_addr learned my addr 172.21.2.112:0/1053703473 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:34:42.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.159 7feb5dbf7700 1 --2- 172.21.2.112:0/1053703473 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7feb580c3230 0x7feb580c3630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:42.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.159 7feb5d3f6700 1 -- 172.21.2.112:0/1053703473 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7feb580c3230 msgr2=0x7feb580c3630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:42.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.159 7feb5d3f6700 1 --2- 172.21.2.112:0/1053703473 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7feb580c3230 0x7feb580c3630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:42.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.159 7feb5d3f6700 1 -- 172.21.2.112:0/1053703473 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7feb580c2810 msgr2=0x7feb580c2c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:42.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.159 7feb5d3f6700 1 --2- 172.21.2.112:0/1053703473 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7feb580c2810 0x7feb580c2c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:42.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.159 7feb5d3f6700 1 -- 172.21.2.112:0/1053703473 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb58084d60 con 0x7feb580c1850 2021-02-10T11:34:42.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.159 7feb5cbf5700 1 --2- 172.21.2.112:0/1053703473 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7feb580c2810 0x7feb580c2c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:34:42.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.160 7feb5dbf7700 1 --2- 172.21.2.112:0/1053703473 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7feb580c3230 0x7feb580c3630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:34:42.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.160 7feb5d3f6700 1 --2- 172.21.2.112:0/1053703473 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7feb580c1850 0x7feb580c1f70 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 rx=0x7feb50009fc0 tx=0x7feb5000bcd0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:42.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.160 7feb4dffb700 1 -- 172.21.2.112:0/1053703473 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feb50006030 con 0x7feb580c1850 2021-02-10T11:34:42.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.160 7feb4dffb700 1 -- 172.21.2.112:0/1053703473 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb500062f0 con 0x7feb580c1850 2021-02-10T11:34:42.152 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.160 7feb4dffb700 1 -- 172.21.2.112:0/1053703473 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feb50000d30 con 0x7feb580c1850 2021-02-10T11:34:42.152 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.160 7feb5f5da700 1 -- 172.21.2.112:0/1053703473 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb58084d60 con 0x7feb580c1850 2021-02-10T11:34:42.152 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.160 7feb5f5da700 1 -- 172.21.2.112:0/1053703473 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb58084d60 con 0x7feb580c1850 2021-02-10T11:34:42.152 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.161 7feb4dffb700 1 -- 172.21.2.112:0/1053703473 <== mon.1 v2:172.21.2.120:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7feb50005660 con 0x7feb580c1850 2021-02-10T11:34:42.152 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.161 7feb4dffb700 1 --2- 172.21.2.112:0/1053703473 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7feb380186c0 0x7feb3801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:42.152 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.161 7feb4dffb700 1 -- 172.21.2.112:0/1053703473 <== mon.1 v2:172.21.2.120:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 4753+0+0 (secure 0 0 0) 0x7feb50032870 con 0x7feb580c1850 2021-02-10T11:34:42.152 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.161 7feb5cbf5700 1 --2- 172.21.2.112:0/1053703473 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7feb380186c0 0x7feb3801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:42.152 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.162 7feb5cbf5700 1 --2- 172.21.2.112:0/1053703473 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7feb380186c0 0x7feb3801abc0 crc :-1 s=READY pgs=676 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:42.180 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.192 7feb5f5da700 1 -- 172.21.2.112:0/1053703473 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb58109a60 con 0x7feb580c1850 2021-02-10T11:34:42.183 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.196 7feb4dffb700 1 -- 172.21.2.112:0/1053703473 <== mon.1 v2:172.21.2.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7feb5000be10 con 0x7feb580c1850 2021-02-10T11:34:42.314 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.326 7feb5f5da700 1 -- 172.21.2.112:0/1053703473 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7feb58109a60 con 0x7feb580c1850 2021-02-10T11:34:42.315 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.328 7feb4dffb700 1 -- 172.21.2.112:0/1053703473 <== mon.1 v2:172.21.2.120:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v59) v1 ==== 92+0+10 (secure 0 0 0) 0x7feb500061f0 con 0x7feb580c1850 2021-02-10T11:34:42.317 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:34:42.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.330 7feb5f5da700 1 -- 172.21.2.112:0/1053703473 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7feb380186c0 msgr2=0x7feb3801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:42.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.330 7feb5f5da700 1 --2- 172.21.2.112:0/1053703473 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7feb380186c0 0x7feb3801abc0 crc :-1 s=READY pgs=676 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:42.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.330 7feb5f5da700 1 -- 172.21.2.112:0/1053703473 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7feb580c1850 msgr2=0x7feb580c1f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:42.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.330 7feb5f5da700 1 --2- 172.21.2.112:0/1053703473 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7feb580c1850 0x7feb580c1f70 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 rx=0x7feb50009fc0 tx=0x7feb5000bcd0).stop 2021-02-10T11:34:42.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.331 7feb5f5da700 1 -- 172.21.2.112:0/1053703473 shutdown_connections 2021-02-10T11:34:42.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.331 7feb5f5da700 1 --2- 172.21.2.112:0/1053703473 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7feb380186c0 0x7feb3801abc0 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:42.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.331 7feb5f5da700 1 --2- 172.21.2.112:0/1053703473 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7feb580c3230 0x7feb580c3630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:42.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.331 7feb5f5da700 1 --2- 172.21.2.112:0/1053703473 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7feb580c2810 0x7feb580c2c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:42.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.331 7feb5f5da700 1 --2- 172.21.2.112:0/1053703473 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7feb580c1850 0x7feb580c1f70 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:42.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.331 7feb5f5da700 1 -- 172.21.2.112:0/1053703473 >> 172.21.2.112:0/1053703473 conn(0x7feb580af690 msgr2=0x7feb58094c40 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:42.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.331 7feb5f5da700 1 -- 172.21.2.112:0/1053703473 shutdown_connections 2021-02-10T11:34:42.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.331 7feb5f5da700 1 -- 172.21.2.112:0/1053703473 wait complete. 2021-02-10T11:34:42.327 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --format=json-pretty osd lspools 2021-02-10T11:34:42.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.748 7f1e324cc700 1 Processor -- start 2021-02-10T11:34:42.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.748 7f1e324cc700 1 -- start start 2021-02-10T11:34:42.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.749 7f1e324cc700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f1e2c1b0870 0x7f1e2c0f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:42.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.749 7f1e324cc700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e2c1b0c70 con 0x7f1e2c1b0470 2021-02-10T11:34:42.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.749 7f1e324cc700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e2c0f0b50 con 0x7f1e2c0efe00 2021-02-10T11:34:42.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.749 7f1e324cc700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e2c0e5ea0 con 0x7f1e2c1b0870 2021-02-10T11:34:42.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.749 7f1e2bfff700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f1e2c1b0470 0x7f1e2c0ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:50050/0 (socket says 172.21.2.112:50050) 2021-02-10T11:34:42.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.749 7f1e2bfff700 1 -- 172.21.2.112:0/660153331 learned_addr learned my addr 172.21.2.112:0/660153331 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:34:42.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.749 7f1e2b7fe700 1 --2- 172.21.2.112:0/660153331 >> v2:172.21.2.112:3300/0 conn(0x7f1e2c1b0870 0x7f1e2c0f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:42.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.749 7f1e2affd700 1 -- 172.21.2.112:0/660153331 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3695091170 0 0) 0x7f1e1c0010b0 con 0x7f1e2c1b0470 2021-02-10T11:34:42.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.749 7f1e2affd700 1 -- 172.21.2.112:0/660153331 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e14002fe0 con 0x7f1e2c1b0470 2021-02-10T11:34:42.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.749 7f1e2b7fe700 1 -- 172.21.2.112:0/660153331 >> v1:172.21.2.120:6789/0 conn(0x7f1e2c0efe00 legacy=0x7f1e2c0f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:42.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.749 7f1e2b7fe700 1 -- 172.21.2.112:0/660153331 >> v1:172.21.2.112:6789/0 conn(0x7f1e2c1b0470 legacy=0x7f1e2c0ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:42.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.749 7f1e2b7fe700 1 -- 172.21.2.112:0/660153331 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e2c0e52e0 con 0x7f1e2c1b0870 2021-02-10T11:34:42.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.750 7f1e2b7fe700 1 --2- 172.21.2.112:0/660153331 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1e2c1b0870 0x7f1e2c0f20e0 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 rx=0x7f1e20006650 tx=0x7f1e20009a80).ready entity=mon.0 client_cookie=f8599e8ba60d495a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:42.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.750 7f1e2affd700 1 -- 172.21.2.112:0/660153331 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1e200009b0 con 0x7f1e2c1b0870 2021-02-10T11:34:42.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.750 7f1e2affd700 1 -- 172.21.2.112:0/660153331 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e2000c0c0 con 0x7f1e2c1b0870 2021-02-10T11:34:42.754 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.750 7f1e2affd700 1 -- 172.21.2.112:0/660153331 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1e2000c5e0 con 0x7f1e2c1b0870 2021-02-10T11:34:42.754 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.750 7f1e324cc700 1 -- 172.21.2.112:0/660153331 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1e2c1b0870 msgr2=0x7f1e2c0f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:42.754 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.750 7f1e324cc700 1 --2- 172.21.2.112:0/660153331 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1e2c1b0870 0x7f1e2c0f20e0 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 rx=0x7f1e20006650 tx=0x7f1e20009a80).stop 2021-02-10T11:34:42.754 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.750 7f1e324cc700 1 -- 172.21.2.112:0/660153331 shutdown_connections 2021-02-10T11:34:42.754 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.750 7f1e324cc700 1 --2- 172.21.2.112:0/660153331 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1e2c1b0870 0x7f1e2c0f20e0 unknown :-1 s=CLOSED pgs=1378 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:42.754 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.750 7f1e324cc700 1 -- 172.21.2.112:0/660153331 >> 172.21.2.112:0/660153331 conn(0x7f1e2c0ce3e0 msgr2=0x7f1e2c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:42.754 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.750 7f1e324cc700 1 -- 172.21.2.112:0/660153331 shutdown_connections 2021-02-10T11:34:42.754 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.750 7f1e324cc700 1 -- 172.21.2.112:0/660153331 wait complete. 2021-02-10T11:34:42.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.765 7f1e324cc700 1 Processor -- start 2021-02-10T11:34:42.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.765 7f1e324cc700 1 -- start start 2021-02-10T11:34:42.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.765 7f1e324cc700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1e2c092e40 0x7f1e2c093240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:42.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.765 7f1e324cc700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1e2c093860 0x7f1e2c1a83d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:42.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.765 7f1e324cc700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1e2c1a89f0 0x7f1e2c1a8df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:42.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.765 7f1e324cc700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e2c195ad0 con 0x7f1e2c092e40 2021-02-10T11:34:42.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.765 7f1e324cc700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e2c1a9450 con 0x7f1e2c093860 2021-02-10T11:34:42.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.765 7f1e324cc700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e2c0b1b00 con 0x7f1e2c1a89f0 2021-02-10T11:34:42.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.766 7f1e2bfff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1e2c092e40 0x7f1e2c093240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:42.757 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.766 7f1e2bfff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1e2c092e40 0x7f1e2c093240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:50592/0 (socket says 172.21.2.112:50592) 2021-02-10T11:34:42.757 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.766 7f1e2bfff700 1 -- 172.21.2.112:0/3367107304 learned_addr learned my addr 172.21.2.112:0/3367107304 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:34:42.757 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.766 7f1e2bfff700 1 -- 172.21.2.112:0/3367107304 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1e2c1a89f0 msgr2=0x7f1e2c1a8df0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:34:42.757 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.766 7f1e2bfff700 1 --2- 172.21.2.112:0/3367107304 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1e2c1a89f0 0x7f1e2c1a8df0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:42.757 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.766 7f1e2bfff700 1 -- 172.21.2.112:0/3367107304 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1e2c093860 msgr2=0x7f1e2c1a83d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:34:42.757 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.766 7f1e2bfff700 1 --2- 172.21.2.112:0/3367107304 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1e2c093860 0x7f1e2c1a83d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:42.757 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.766 7f1e2bfff700 1 -- 172.21.2.112:0/3367107304 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e2c1a99d0 con 0x7f1e2c092e40 2021-02-10T11:34:42.757 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.766 7f1e2bfff700 1 --2- 172.21.2.112:0/3367107304 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1e2c092e40 0x7f1e2c093240 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 rx=0x7f1e1c00a4c0 tx=0x7f1e1c00a860).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:42.757 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.766 7f1e28ff9700 1 -- 172.21.2.112:0/3367107304 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1e1c00ac50 con 0x7f1e2c092e40 2021-02-10T11:34:42.757 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.766 7f1e28ff9700 1 -- 172.21.2.112:0/3367107304 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e1c004350 con 0x7f1e2c092e40 2021-02-10T11:34:42.757 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.766 7f1e28ff9700 1 -- 172.21.2.112:0/3367107304 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1e1c004870 con 0x7f1e2c092e40 2021-02-10T11:34:42.757 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.766 7f1e324cc700 1 -- 172.21.2.112:0/3367107304 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e2c1a99d0 con 0x7f1e2c092e40 2021-02-10T11:34:42.757 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.766 7f1e324cc700 1 -- 172.21.2.112:0/3367107304 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e2c1a9f60 con 0x7f1e2c092e40 2021-02-10T11:34:42.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.767 7f1e28ff9700 1 -- 172.21.2.112:0/3367107304 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1e1c004530 con 0x7f1e2c092e40 2021-02-10T11:34:42.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.767 7f1e28ff9700 1 --2- 172.21.2.112:0/3367107304 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1e0c0186c0 0x7f1e0c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:42.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.767 7f1e28ff9700 1 -- 172.21.2.112:0/3367107304 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f1e1c0321f0 con 0x7f1e2c092e40 2021-02-10T11:34:42.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.768 7f1e2b7fe700 1 --2- 172.21.2.112:0/3367107304 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1e0c0186c0 0x7f1e0c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:42.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.768 7f1e2b7fe700 1 --2- 172.21.2.112:0/3367107304 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1e0c0186c0 0x7f1e0c01abc0 crc :-1 s=READY pgs=677 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:42.786 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.799 7f1e324cc700 1 -- 172.21.2.112:0/3367107304 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e2c0b1b00 con 0x7f1e2c092e40 2021-02-10T11:34:42.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.801 7f1e28ff9700 1 -- 172.21.2.112:0/3367107304 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1e1c000ea0 con 0x7f1e2c092e40 2021-02-10T11:34:42.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.924 7f1e324cc700 1 -- 172.21.2.112:0/3367107304 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f1e2c0b1b00 con 0x7f1e2c092e40 2021-02-10T11:34:42.913 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.925 7f1e28ff9700 1 -- 172.21.2.112:0/3367107304 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v59) v1 ==== 84+0+64 (secure 0 0 0) 0x7f1e1c004530 con 0x7f1e2c092e40 2021-02-10T11:34:42.913 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:34:42.913 INFO:teuthology.orchestra.run.gibba012.stdout:[ 2021-02-10T11:34:42.913 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:34:42.914 INFO:teuthology.orchestra.run.gibba012.stdout: "poolnum": 1, 2021-02-10T11:34:42.914 INFO:teuthology.orchestra.run.gibba012.stdout: "poolname": "rbd" 2021-02-10T11:34:42.914 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:34:42.914 INFO:teuthology.orchestra.run.gibba012.stdout:] 2021-02-10T11:34:42.914 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.927 7f1e324cc700 1 -- 172.21.2.112:0/3367107304 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1e0c0186c0 msgr2=0x7f1e0c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:42.914 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.927 7f1e324cc700 1 --2- 172.21.2.112:0/3367107304 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1e0c0186c0 0x7f1e0c01abc0 crc :-1 s=READY pgs=677 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:42.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.927 7f1e324cc700 1 -- 172.21.2.112:0/3367107304 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1e2c092e40 msgr2=0x7f1e2c093240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:42.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.927 7f1e324cc700 1 --2- 172.21.2.112:0/3367107304 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1e2c092e40 0x7f1e2c093240 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 rx=0x7f1e1c00a4c0 tx=0x7f1e1c00a860).stop 2021-02-10T11:34:42.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.927 7f1e324cc700 1 -- 172.21.2.112:0/3367107304 shutdown_connections 2021-02-10T11:34:42.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.927 7f1e324cc700 1 --2- 172.21.2.112:0/3367107304 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1e0c0186c0 0x7f1e0c01abc0 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:42.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.927 7f1e324cc700 1 --2- 172.21.2.112:0/3367107304 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1e2c093860 0x7f1e2c1a83d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:42.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.927 7f1e324cc700 1 --2- 172.21.2.112:0/3367107304 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1e2c1a89f0 0x7f1e2c1a8df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:42.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.927 7f1e324cc700 1 --2- 172.21.2.112:0/3367107304 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1e2c092e40 0x7f1e2c093240 unknown :-1 s=CLOSED pgs=1379 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:42.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.927 7f1e324cc700 1 -- 172.21.2.112:0/3367107304 >> 172.21.2.112:0/3367107304 conn(0x7f1e2c19f7a0 msgr2=0x7f1e2c0940e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:42.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.927 7f1e324cc700 1 -- 172.21.2.112:0/3367107304 shutdown_connections 2021-02-10T11:34:42.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:42.927 7f1e324cc700 1 -- 172.21.2.112:0/3367107304 wait complete. 2021-02-10T11:34:42.923 INFO:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2021-02-10T11:34:42.924 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_metadata 8 2021-02-10T11:34:43.355 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:43.350 7f6d4c442700 1 Processor -- start 2021-02-10T11:34:43.355 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:43.350 7f6d4c442700 1 -- start start 2021-02-10T11:34:43.355 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:43.350 7f6d4c442700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f6d440d1c70 0x7f6d440cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:43.355 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:43.350 7f6d4c442700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d440d0170 con 0x7f6d440cf580 2021-02-10T11:34:43.355 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:43.350 7f6d4c442700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d440fe1c0 con 0x7f6d440d1870 2021-02-10T11:34:43.355 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:43.350 7f6d4c442700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d441b0cb0 con 0x7f6d440d1c70 2021-02-10T11:34:43.355 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:43.351 7f6d4aa5f700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f6d440cf580 0x7f6d440cf980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:50066/0 (socket says 172.21.2.112:50066) 2021-02-10T11:34:43.355 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:43.351 7f6d4aa5f700 1 -- 172.21.2.112:0/1043162234 learned_addr learned my addr 172.21.2.112:0/1043162234 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:34:43.355 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:43.351 7f6d49a5d700 1 --2- 172.21.2.112:0/1043162234 >> v2:172.21.2.112:3300/0 conn(0x7f6d440d1c70 0x7f6d440cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:43.355 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:43.351 7f6d49a5d700 1 -- 172.21.2.112:0/1043162234 >> v1:172.21.2.120:6789/0 conn(0x7f6d440d1870 legacy=0x7f6d441b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:43.356 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:43.351 7f6d49a5d700 1 -- 172.21.2.112:0/1043162234 >> v1:172.21.2.112:6789/0 conn(0x7f6d440cf580 legacy=0x7f6d440cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:43.356 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:43.351 7f6d49a5d700 1 -- 172.21.2.112:0/1043162234 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d44109a60 con 0x7f6d440d1c70 2021-02-10T11:34:43.356 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:43.352 7f6d49a5d700 1 --2- 172.21.2.112:0/1043162234 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6d440d1c70 0x7f6d440cef60 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 rx=0x7f6d34006650 tx=0x7f6d34009a00).ready entity=mon.0 client_cookie=e224e1573dfc6240 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:43.356 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:43.352 7f6d4925c700 1 -- 172.21.2.112:0/1043162234 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6d34009df0 con 0x7f6d440d1c70 2021-02-10T11:34:43.356 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:43.352 7f6d4925c700 1 -- 172.21.2.112:0/1043162234 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d340009b0 con 0x7f6d440d1c70 2021-02-10T11:34:43.356 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:43.352 7f6d4925c700 1 -- 172.21.2.112:0/1043162234 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6d3400c2b0 con 0x7f6d440d1c70 2021-02-10T11:34:43.356 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:43.352 7f6d4c442700 1 -- 172.21.2.112:0/1043162234 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6d440d1c70 msgr2=0x7f6d440cef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:43.356 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:43.352 7f6d4c442700 1 --2- 172.21.2.112:0/1043162234 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6d440d1c70 0x7f6d440cef60 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 rx=0x7f6d34006650 tx=0x7f6d34009a00).stop 2021-02-10T11:34:43.356 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:43.353 7f6d4c442700 1 -- 172.21.2.112:0/1043162234 shutdown_connections 2021-02-10T11:34:43.356 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:43.353 7f6d4c442700 1 --2- 172.21.2.112:0/1043162234 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6d440d1c70 0x7f6d440cef60 unknown :-1 s=CLOSED pgs=1381 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:43.356 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:43.353 7f6d4c442700 1 -- 172.21.2.112:0/1043162234 >> 172.21.2.112:0/1043162234 conn(0x7f6d440ce420 msgr2=0x7f6d440d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:43.357 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:43.353 7f6d4c442700 1 -- 172.21.2.112:0/1043162234 shutdown_connections 2021-02-10T11:34:43.357 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:43.353 7f6d4c442700 1 -- 172.21.2.112:0/1043162234 wait complete. 2021-02-10T11:34:43.358 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:43.367 7f6d4c442700 1 Processor -- start 2021-02-10T11:34:43.358 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:43.367 7f6d4c442700 1 -- start start 2021-02-10T11:34:43.358 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:43.368 7f6d4c442700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6d440b7bb0 0x7f6d441c10e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:43.358 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:43.368 7f6d4c442700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6d441c1700 0x7f6d44103c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:43.358 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:43.368 7f6d4c442700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6d44104220 0x7f6d44104620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:43.358 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:43.368 7f6d4c442700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d44104c40 con 0x7f6d440b7bb0 2021-02-10T11:34:43.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:43.368 7f6d4c442700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d44104e60 con 0x7f6d441c1700 2021-02-10T11:34:43.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:43.368 7f6d4c442700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d44105080 con 0x7f6d44104220 2021-02-10T11:34:43.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:43.368 7f6d4a25e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6d440b7bb0 0x7f6d441c10e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:43.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:43.368 7f6d4a25e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6d440b7bb0 0x7f6d441c10e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:50608/0 (socket says 172.21.2.112:50608) 2021-02-10T11:34:43.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:43.368 7f6d4a25e700 1 -- 172.21.2.112:0/1265256165 learned_addr learned my addr 172.21.2.112:0/1265256165 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:34:43.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:43.368 7f6d4aa5f700 1 --2- 172.21.2.112:0/1265256165 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6d44104220 0x7f6d44104620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:43.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:43.368 7f6d49a5d700 1 --2- 172.21.2.112:0/1265256165 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6d441c1700 0x7f6d44103c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:43.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:43.369 7f6d4a25e700 1 -- 172.21.2.112:0/1265256165 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6d44104220 msgr2=0x7f6d44104620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:43.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:43.369 7f6d4a25e700 1 --2- 172.21.2.112:0/1265256165 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6d44104220 0x7f6d44104620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:43.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:43.369 7f6d4a25e700 1 -- 172.21.2.112:0/1265256165 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6d441c1700 msgr2=0x7f6d44103c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:43.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:43.369 7f6d4a25e700 1 --2- 172.21.2.112:0/1265256165 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6d441c1700 0x7f6d44103c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:43.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:43.369 7f6d4a25e700 1 -- 172.21.2.112:0/1265256165 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d44101d20 con 0x7f6d440b7bb0 2021-02-10T11:34:43.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:43.369 7f6d4a25e700 1 --2- 172.21.2.112:0/1265256165 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6d440b7bb0 0x7f6d441c10e0 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 rx=0x7f6d4000b5b0 tx=0x7f6d4000b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:43.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:43.369 7f6d3affd700 1 -- 172.21.2.112:0/1265256165 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6d4000bd40 con 0x7f6d440b7bb0 2021-02-10T11:34:43.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:43.369 7f6d3affd700 1 -- 172.21.2.112:0/1265256165 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d40006020 con 0x7f6d440b7bb0 2021-02-10T11:34:43.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:43.369 7f6d3affd700 1 -- 172.21.2.112:0/1265256165 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6d400054a0 con 0x7f6d440b7bb0 2021-02-10T11:34:43.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:43.369 7f6d4c442700 1 -- 172.21.2.112:0/1265256165 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d44101d20 con 0x7f6d440b7bb0 2021-02-10T11:34:43.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:43.369 7f6d4c442700 1 -- 172.21.2.112:0/1265256165 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d44101d20 con 0x7f6d440b7bb0 2021-02-10T11:34:43.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:43.370 7f6d3affd700 1 -- 172.21.2.112:0/1265256165 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6d40006200 con 0x7f6d440b7bb0 2021-02-10T11:34:43.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:43.370 7f6d3affd700 1 --2- 172.21.2.112:0/1265256165 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6d2c018730 0x7f6d2c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:43.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:43.370 7f6d3affd700 1 -- 172.21.2.112:0/1265256165 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f6d40032300 con 0x7f6d440b7bb0 2021-02-10T11:34:43.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:43.370 7f6d49a5d700 1 --2- 172.21.2.112:0/1265256165 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6d2c018730 0x7f6d2c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:43.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:43.371 7f6d49a5d700 1 --2- 172.21.2.112:0/1265256165 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6d2c018730 0x7f6d2c01abc0 crc :-1 s=READY pgs=678 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:43.389 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:43.401 7f6d4c442700 1 -- 172.21.2.112:0/1265256165 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d440807d0 con 0x7f6d440b7bb0 2021-02-10T11:34:43.391 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:43.403 7f6d3affd700 1 -- 172.21.2.112:0/1265256165 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6d4000bd40 con 0x7f6d440b7bb0 2021-02-10T11:34:43.519 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:43.531 7f6d4c442700 1 -- 172.21.2.112:0/1265256165 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"} v 0) v1 -- 0x7f6d440807d0 con 0x7f6d440b7bb0 2021-02-10T11:34:43.935 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:43.947 7f6d3affd700 1 -- 172.21.2.112:0/1265256165 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"}]=0 pool 'cephfs_metadata' created v60) v1 ==== 133+0+0 (secure 0 0 0) 0x7f6d40006020 con 0x7f6d440b7bb0 2021-02-10T11:34:43.935 INFO:teuthology.orchestra.run.gibba012.stderr:pool 'cephfs_metadata' created 2021-02-10T11:34:43.938 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:43.949 7f6d4c442700 1 -- 172.21.2.112:0/1265256165 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6d2c018730 msgr2=0x7f6d2c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:43.938 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:43.949 7f6d4c442700 1 --2- 172.21.2.112:0/1265256165 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6d2c018730 0x7f6d2c01abc0 crc :-1 s=READY pgs=678 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:43.938 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:43.949 7f6d4c442700 1 -- 172.21.2.112:0/1265256165 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6d440b7bb0 msgr2=0x7f6d441c10e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:43.938 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:43.949 7f6d4c442700 1 --2- 172.21.2.112:0/1265256165 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6d440b7bb0 0x7f6d441c10e0 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 rx=0x7f6d4000b5b0 tx=0x7f6d4000b950).stop 2021-02-10T11:34:43.938 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:43.949 7f6d4c442700 1 -- 172.21.2.112:0/1265256165 shutdown_connections 2021-02-10T11:34:43.938 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:43.949 7f6d4c442700 1 --2- 172.21.2.112:0/1265256165 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6d2c018730 0x7f6d2c01abc0 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:43.938 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:43.949 7f6d4c442700 1 --2- 172.21.2.112:0/1265256165 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6d441c1700 0x7f6d44103c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:43.938 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:43.949 7f6d4c442700 1 --2- 172.21.2.112:0/1265256165 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6d44104220 0x7f6d44104620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:43.938 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:43.949 7f6d4c442700 1 --2- 172.21.2.112:0/1265256165 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6d440b7bb0 0x7f6d441c10e0 unknown :-1 s=CLOSED pgs=1382 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:43.938 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:43.949 7f6d4c442700 1 -- 172.21.2.112:0/1265256165 >> 172.21.2.112:0/1265256165 conn(0x7f6d44080ef0 msgr2=0x7f6d440ffa50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:43.939 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:43.949 7f6d4c442700 1 -- 172.21.2.112:0/1265256165 shutdown_connections 2021-02-10T11:34:43.939 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:43.949 7f6d4c442700 1 -- 172.21.2.112:0/1265256165 wait complete. 2021-02-10T11:34:43.947 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_data 8 2021-02-10T11:34:44.377 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:44.373 7f9f755d4700 1 Processor -- start 2021-02-10T11:34:44.377 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:44.373 7f9f755d4700 1 -- start start 2021-02-10T11:34:44.377 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:44.373 7f9f755d4700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f9f700d1870 0x7f9f701b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:44.377 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:44.373 7f9f755d4700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f9f700cff70 0x7f9f70109130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:44.377 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:44.373 7f9f755d4700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f700fe1c0 con 0x7f9f701b0ae0 2021-02-10T11:34:44.378 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:44.373 7f9f755d4700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f70109ad0 con 0x7f9f700d1870 2021-02-10T11:34:44.378 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:44.373 7f9f755d4700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f70109750 con 0x7f9f700cff70 2021-02-10T11:34:44.378 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:44.373 7f9f6f2bf700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f9f700d1870 0x7f9f701b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:44.378 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:44.373 7f9f6f2bf700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f9f700d1870 0x7f9f701b04c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:50614/0 (socket says 172.21.2.112:50614) 2021-02-10T11:34:44.378 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:44.373 7f9f6f2bf700 1 -- 172.21.2.112:0/2752464919 learned_addr learned my addr 172.21.2.112:0/2752464919 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:34:44.378 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:44.373 7f9f6fac0700 1 --2- 172.21.2.112:0/2752464919 >> v2:172.21.2.120:3300/0 conn(0x7f9f700cff70 0x7f9f70109130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:44.378 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:44.374 7f9f6f2bf700 1 -- 172.21.2.112:0/2752464919 >> v2:172.21.2.120:3300/0 conn(0x7f9f700cff70 msgr2=0x7f9f70109130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:44.378 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:44.374 7f9f6f2bf700 1 --2- 172.21.2.112:0/2752464919 >> v2:172.21.2.120:3300/0 conn(0x7f9f700cff70 0x7f9f70109130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:44.378 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:44.374 7f9f6f2bf700 1 -- 172.21.2.112:0/2752464919 >> v1:172.21.2.120:6789/0 conn(0x7f9f701b0ae0 legacy=0x7f9f70108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:44.378 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:44.374 7f9f6f2bf700 1 -- 172.21.2.112:0/2752464919 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f70088070 con 0x7f9f700d1870 2021-02-10T11:34:44.378 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:44.374 7f9f6f2bf700 1 --2- 172.21.2.112:0/2752464919 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9f700d1870 0x7f9f701b04c0 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 rx=0x7f9f60006700 tx=0x7f9f60009b80).ready entity=mon.0 client_cookie=802a3f592aa329bc server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:44.378 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:44.374 7f9f6e2bd700 1 -- 172.21.2.112:0/2752464919 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9f6000c0c0 con 0x7f9f700d1870 2021-02-10T11:34:44.379 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:44.374 7f9f6e2bd700 1 -- 172.21.2.112:0/2752464919 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f6000c440 con 0x7f9f700d1870 2021-02-10T11:34:44.379 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:44.374 7f9f6e2bd700 1 -- 172.21.2.112:0/2752464919 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9f6000c960 con 0x7f9f700d1870 2021-02-10T11:34:44.379 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:44.374 7f9f755d4700 1 -- 172.21.2.112:0/2752464919 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9f700d1870 msgr2=0x7f9f701b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:44.379 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:44.374 7f9f755d4700 1 --2- 172.21.2.112:0/2752464919 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9f700d1870 0x7f9f701b04c0 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 rx=0x7f9f60006700 tx=0x7f9f60009b80).stop 2021-02-10T11:34:44.379 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:44.375 7f9f755d4700 1 -- 172.21.2.112:0/2752464919 shutdown_connections 2021-02-10T11:34:44.379 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:44.375 7f9f755d4700 1 --2- 172.21.2.112:0/2752464919 >> v2:172.21.2.120:3300/0 conn(0x7f9f700cff70 0x7f9f70109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:44.379 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:44.375 7f9f755d4700 1 --2- 172.21.2.112:0/2752464919 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9f700d1870 0x7f9f701b04c0 unknown :-1 s=CLOSED pgs=1383 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:44.379 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:44.375 7f9f755d4700 1 -- 172.21.2.112:0/2752464919 >> 172.21.2.112:0/2752464919 conn(0x7f9f700ce420 msgr2=0x7f9f700d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:44.379 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:44.375 7f9f755d4700 1 -- 172.21.2.112:0/2752464919 shutdown_connections 2021-02-10T11:34:44.379 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:44.375 7f9f755d4700 1 -- 172.21.2.112:0/2752464919 wait complete. 2021-02-10T11:34:44.379 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:44.389 7f9f755d4700 1 Processor -- start 2021-02-10T11:34:44.381 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:44.390 7f9f755d4700 1 -- start start 2021-02-10T11:34:44.381 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:44.390 7f9f755d4700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9f70191b10 0x7f9f700ae640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:44.381 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:44.390 7f9f755d4700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9f70191f10 0x7f9f7019ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:44.381 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:44.390 7f9f755d4700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9f701a05f0 0x7f9f70082ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:44.381 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:44.390 7f9f755d4700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f70192440 con 0x7f9f70191b10 2021-02-10T11:34:44.381 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:44.390 7f9f755d4700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f701916e0 con 0x7f9f701a05f0 2021-02-10T11:34:44.381 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:44.390 7f9f755d4700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f70191900 con 0x7f9f70191f10 2021-02-10T11:34:44.382 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:44.390 7f9f6f2bf700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9f70191b10 0x7f9f700ae640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:44.382 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:44.390 7f9f6f2bf700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9f70191b10 0x7f9f700ae640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:50618/0 (socket says 172.21.2.112:50618) 2021-02-10T11:34:44.382 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:44.390 7f9f6f2bf700 1 -- 172.21.2.112:0/234291566 learned_addr learned my addr 172.21.2.112:0/234291566 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:34:44.382 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:44.390 7f9f6fac0700 1 --2- 172.21.2.112:0/234291566 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9f701a05f0 0x7f9f70082ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:44.382 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:44.390 7f9f6eabe700 1 --2- 172.21.2.112:0/234291566 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9f70191f10 0x7f9f7019ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:44.382 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:44.391 7f9f6fac0700 1 -- 172.21.2.112:0/234291566 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9f70191f10 msgr2=0x7f9f7019ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:44.382 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:44.391 7f9f6fac0700 1 --2- 172.21.2.112:0/234291566 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9f70191f10 0x7f9f7019ffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:44.382 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:44.391 7f9f6fac0700 1 -- 172.21.2.112:0/234291566 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9f70191b10 msgr2=0x7f9f700ae640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:44.382 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:44.391 7f9f6fac0700 1 --2- 172.21.2.112:0/234291566 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9f70191b10 0x7f9f700ae640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:44.382 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:44.391 7f9f6fac0700 1 -- 172.21.2.112:0/234291566 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f700834f0 con 0x7f9f701a05f0 2021-02-10T11:34:44.382 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:44.391 7f9f6f2bf700 1 --2- 172.21.2.112:0/234291566 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9f70191b10 0x7f9f700ae640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:34:44.382 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:44.391 7f9f6fac0700 1 --2- 172.21.2.112:0/234291566 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9f701a05f0 0x7f9f70082ed0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 rx=0x7f9f6800b8a0 tx=0x7f9f6800bc40).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:44.383 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:44.391 7f9f57fff700 1 -- 172.21.2.112:0/234291566 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9f68000f90 con 0x7f9f701a05f0 2021-02-10T11:34:44.383 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:44.391 7f9f57fff700 1 -- 172.21.2.112:0/234291566 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f6800bd80 con 0x7f9f701a05f0 2021-02-10T11:34:44.383 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:44.391 7f9f755d4700 1 -- 172.21.2.112:0/234291566 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f700834f0 con 0x7f9f701a05f0 2021-02-10T11:34:44.383 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:44.391 7f9f755d4700 1 -- 172.21.2.112:0/234291566 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f700834f0 con 0x7f9f701a05f0 2021-02-10T11:34:44.383 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:44.391 7f9f57fff700 1 -- 172.21.2.112:0/234291566 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9f68001470 con 0x7f9f701a05f0 2021-02-10T11:34:44.383 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:44.392 7f9f57fff700 1 -- 172.21.2.112:0/234291566 <== mon.1 v2:172.21.2.120:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9f680011f0 con 0x7f9f701a05f0 2021-02-10T11:34:44.383 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:44.392 7f9f57fff700 1 --2- 172.21.2.112:0/234291566 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9f500186c0 0x7f9f5001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:44.383 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:44.392 7f9f57fff700 1 -- 172.21.2.112:0/234291566 <== mon.1 v2:172.21.2.120:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5107+0+0 (secure 0 0 0) 0x7f9f680326c0 con 0x7f9f701a05f0 2021-02-10T11:34:44.383 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:44.392 7f9f6f2bf700 1 --2- 172.21.2.112:0/234291566 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9f500186c0 0x7f9f5001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:44.383 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:44.393 7f9f6f2bf700 1 --2- 172.21.2.112:0/234291566 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9f500186c0 0x7f9f5001abc0 crc :-1 s=READY pgs=679 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:44.411 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:44.423 7f9f755d4700 1 -- 172.21.2.112:0/234291566 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f70192440 con 0x7f9f701a05f0 2021-02-10T11:34:44.414 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:44.427 7f9f57fff700 1 -- 172.21.2.112:0/234291566 <== mon.1 v2:172.21.2.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9f68028000 con 0x7f9f701a05f0 2021-02-10T11:34:44.546 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:44.558 7f9f755d4700 1 -- 172.21.2.112:0/234291566 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"} v 0) v1 -- 0x7f9f70192440 con 0x7f9f701a05f0 2021-02-10T11:34:44.946 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:44.958 7f9f57fff700 1 -- 172.21.2.112:0/234291566 <== mon.1 v2:172.21.2.120:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"}]=0 pool 'cephfs_data' created v62) v1 ==== 125+0+0 (secure 0 0 0) 0x7f9f68028000 con 0x7f9f701a05f0 2021-02-10T11:34:44.962 INFO:teuthology.orchestra.run.gibba012.stderr:pool 'cephfs_data' created 2021-02-10T11:34:44.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:44.976 7f9f755d4700 1 -- 172.21.2.112:0/234291566 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9f500186c0 msgr2=0x7f9f5001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:44.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:44.976 7f9f755d4700 1 --2- 172.21.2.112:0/234291566 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9f500186c0 0x7f9f5001abc0 crc :-1 s=READY pgs=679 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:44.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:44.976 7f9f755d4700 1 -- 172.21.2.112:0/234291566 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9f701a05f0 msgr2=0x7f9f70082ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:44.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:44.976 7f9f755d4700 1 --2- 172.21.2.112:0/234291566 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9f701a05f0 0x7f9f70082ed0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 rx=0x7f9f6800b8a0 tx=0x7f9f6800bc40).stop 2021-02-10T11:34:44.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:44.976 7f9f755d4700 1 -- 172.21.2.112:0/234291566 shutdown_connections 2021-02-10T11:34:44.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:44.976 7f9f755d4700 1 --2- 172.21.2.112:0/234291566 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9f500186c0 0x7f9f5001abc0 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:44.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:44.976 7f9f755d4700 1 --2- 172.21.2.112:0/234291566 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9f701a05f0 0x7f9f70082ed0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:44.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:44.976 7f9f755d4700 1 --2- 172.21.2.112:0/234291566 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9f70191f10 0x7f9f7019ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:44.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:44.976 7f9f755d4700 1 --2- 172.21.2.112:0/234291566 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9f70191b10 0x7f9f700ae640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:44.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:44.976 7f9f755d4700 1 -- 172.21.2.112:0/234291566 >> 172.21.2.112:0/234291566 conn(0x7f9f700a02d0 msgr2=0x7f9f7009f200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:44.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:44.976 7f9f755d4700 1 -- 172.21.2.112:0/234291566 shutdown_connections 2021-02-10T11:34:44.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:44.976 7f9f755d4700 1 -- 172.21.2.112:0/234291566 wait complete. 2021-02-10T11:34:44.973 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2021-02-10T11:34:45.403 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:45.398 7f3150317700 1 Processor -- start 2021-02-10T11:34:45.403 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:45.398 7f3150317700 1 -- start start 2021-02-10T11:34:45.404 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:45.398 7f3150317700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f31480d1880 0x7f31481b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:45.404 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:45.399 7f3150317700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f31481b0af0 0x7f3148108950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:45.404 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:45.399 7f3150317700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31480fe1d0 con 0x7f31480d1c80 2021-02-10T11:34:45.404 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:45.399 7f3150317700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3148109a70 con 0x7f31480d1880 2021-02-10T11:34:45.404 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:45.399 7f3150317700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3148109760 con 0x7f31481b0af0 2021-02-10T11:34:45.404 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:45.399 7f314e934700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f31480d1c80 0x7f3148108f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:50094/0 (socket says 172.21.2.112:50094) 2021-02-10T11:34:45.404 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:45.399 7f314e934700 1 -- 172.21.2.112:0/3203787194 learned_addr learned my addr 172.21.2.112:0/3203787194 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:34:45.404 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:45.399 7f314d131700 1 -- 172.21.2.112:0/3203787194 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1001171043 0 0) 0x7f3140001500 con 0x7f31480d1c80 2021-02-10T11:34:45.404 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:45.399 7f314e133700 1 --2- 172.21.2.112:0/3203787194 >> v2:172.21.2.120:3300/0 conn(0x7f31480d1880 0x7f31481b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:45.404 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:45.399 7f314d131700 1 -- 172.21.2.112:0/3203787194 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3134002fe0 con 0x7f31480d1c80 2021-02-10T11:34:45.405 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:45.399 7f314d932700 1 --2- 172.21.2.112:0/3203787194 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f31481b0af0 0x7f3148108950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:45.405 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:45.399 7f314d131700 1 -- 172.21.2.112:0/3203787194 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3393575820 0 0) 0x7f3140001460 con 0x7f31480d1c80 2021-02-10T11:34:45.405 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:45.399 7f314d131700 1 -- 172.21.2.112:0/3203787194 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f31481b0af0 msgr2=0x7f3148108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:45.405 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:45.399 7f314d131700 1 --2- 172.21.2.112:0/3203787194 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f31481b0af0 0x7f3148108950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:45.405 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:45.399 7f314d131700 1 -- 172.21.2.112:0/3203787194 >> v2:172.21.2.120:3300/0 conn(0x7f31480d1880 msgr2=0x7f31481b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:45.405 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:45.399 7f314d131700 1 --2- 172.21.2.112:0/3203787194 >> v2:172.21.2.120:3300/0 conn(0x7f31480d1880 0x7f31481b04d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:45.405 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:45.399 7f314d131700 1 -- 172.21.2.112:0/3203787194 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3148088070 con 0x7f31480d1c80 2021-02-10T11:34:45.405 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:45.399 7f314d131700 1 -- 172.21.2.112:0/3203787194 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f3140001980 con 0x7f31480d1c80 2021-02-10T11:34:45.405 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:45.399 7f314d131700 1 -- 172.21.2.112:0/3203787194 >> v1:172.21.2.112:6789/0 conn(0x7f31480d1c80 legacy=0x7f3148108f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:45.405 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:45.399 7f314d131700 1 --2- 172.21.2.112:0/3203787194 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3134003af0 0x7f3134005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:45.405 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:45.399 7f314d131700 1 --2- 172.21.2.112:0/3203787194 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f313404a7c0 0x7f313404cca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:45.405 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:45.399 7f314d131700 1 --2- 172.21.2.112:0/3203787194 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f313404d5a0 0x7f313404fa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:45.406 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:45.399 7f314d131700 1 -- 172.21.2.112:0/3203787194 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31340502e0 con 0x7f313404a7c0 2021-02-10T11:34:45.406 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:45.399 7f314d131700 1 -- 172.21.2.112:0/3203787194 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3134050500 con 0x7f3134003af0 2021-02-10T11:34:45.406 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:45.399 7f314d131700 1 -- 172.21.2.112:0/3203787194 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3134050720 con 0x7f313404d5a0 2021-02-10T11:34:45.406 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:45.400 7f314d932700 1 --2- 172.21.2.112:0/3203787194 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f313404a7c0 0x7f313404cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:45.406 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:45.400 7f314e133700 1 --2- 172.21.2.112:0/3203787194 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3134003af0 0x7f3134005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:45.406 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:45.400 7f314e934700 1 --2- 172.21.2.112:0/3203787194 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f313404d5a0 0x7f313404fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:45.406 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:45.400 7f314d932700 1 -- 172.21.2.112:0/3203787194 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f313404d5a0 msgr2=0x7f313404fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:45.406 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:45.400 7f314d932700 1 --2- 172.21.2.112:0/3203787194 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f313404d5a0 0x7f313404fa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:45.406 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:45.400 7f314d932700 1 -- 172.21.2.112:0/3203787194 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3134003af0 msgr2=0x7f3134005fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:45.406 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:45.400 7f314d932700 1 --2- 172.21.2.112:0/3203787194 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3134003af0 0x7f3134005fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:45.406 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:45.400 7f314d932700 1 -- 172.21.2.112:0/3203787194 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31340509c0 con 0x7f313404a7c0 2021-02-10T11:34:45.406 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:45.400 7f314d932700 1 --2- 172.21.2.112:0/3203787194 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f313404a7c0 0x7f313404cca0 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 rx=0x7f3138006ac0 tx=0x7f3138009ab0).ready entity=mon.0 client_cookie=12f9e9a15ecb824d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:45.407 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:45.400 7f314d131700 1 -- 172.21.2.112:0/3203787194 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f313800c0c0 con 0x7f313404a7c0 2021-02-10T11:34:45.407 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:45.400 7f314d131700 1 -- 172.21.2.112:0/3203787194 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3138006540 con 0x7f313404a7c0 2021-02-10T11:34:45.407 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:45.400 7f314d131700 1 -- 172.21.2.112:0/3203787194 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f313800c600 con 0x7f313404a7c0 2021-02-10T11:34:45.407 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:45.400 7f3150317700 1 -- 172.21.2.112:0/3203787194 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f313404a7c0 msgr2=0x7f313404cca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:45.407 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:45.400 7f3150317700 1 --2- 172.21.2.112:0/3203787194 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f313404a7c0 0x7f313404cca0 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 rx=0x7f3138006ac0 tx=0x7f3138009ab0).stop 2021-02-10T11:34:45.407 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:45.400 7f314e934700 1 -- 172.21.2.112:0/3203787194 reap_dead start 2021-02-10T11:34:45.407 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:45.401 7f3150317700 1 -- 172.21.2.112:0/3203787194 shutdown_connections 2021-02-10T11:34:45.407 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:45.401 7f3150317700 1 -- 172.21.2.112:0/3203787194 >> 172.21.2.112:0/3203787194 conn(0x7f31480ce430 msgr2=0x7f31480d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:45.407 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:45.401 7f3150317700 1 -- 172.21.2.112:0/3203787194 shutdown_connections 2021-02-10T11:34:45.407 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:45.401 7f3150317700 1 -- 172.21.2.112:0/3203787194 wait complete. 2021-02-10T11:34:45.407 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:45.416 7f3150317700 1 Processor -- start 2021-02-10T11:34:45.407 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:45.416 7f3150317700 1 -- start start 2021-02-10T11:34:45.408 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:45.416 7f3150317700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f31481a3570 0x7f31481a3970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:45.409 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:45.416 7f3150317700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f314809c9f0 0x7f314809cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:45.409 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:45.416 7f3150317700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f314809d410 0x7f3148098170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:45.409 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:45.416 7f3150317700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f314809c470 con 0x7f31481a3570 2021-02-10T11:34:45.409 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:45.416 7f3150317700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31480b2840 con 0x7f314809d410 2021-02-10T11:34:45.409 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:45.416 7f3150317700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31480b2a20 con 0x7f314809c9f0 2021-02-10T11:34:45.409 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:45.416 7f314e133700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f31481a3570 0x7f31481a3970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:45.409 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:45.416 7f314e133700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f31481a3570 0x7f31481a3970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:50644/0 (socket says 172.21.2.112:50644) 2021-02-10T11:34:45.409 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:45.416 7f314e133700 1 -- 172.21.2.112:0/3554156048 learned_addr learned my addr 172.21.2.112:0/3554156048 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:34:45.409 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:45.416 7f314e133700 1 -- 172.21.2.112:0/3554156048 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f314809c9f0 msgr2=0x7f314809cdf0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:34:45.409 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:45.416 7f314e133700 1 --2- 172.21.2.112:0/3554156048 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f314809c9f0 0x7f314809cdf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:45.409 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:45.416 7f314e133700 1 -- 172.21.2.112:0/3554156048 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f314809d410 msgr2=0x7f3148098170 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:34:45.409 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:45.416 7f314e133700 1 --2- 172.21.2.112:0/3554156048 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f314809d410 0x7f3148098170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:45.410 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:45.417 7f314e133700 1 -- 172.21.2.112:0/3554156048 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f314809d880 con 0x7f31481a3570 2021-02-10T11:34:45.410 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:45.417 7f314e133700 1 --2- 172.21.2.112:0/3554156048 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f31481a3570 0x7f31481a3970 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 rx=0x7f3144006f60 tx=0x7f314400b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:45.410 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:45.417 7f313effd700 1 -- 172.21.2.112:0/3554156048 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f314400bd60 con 0x7f31481a3570 2021-02-10T11:34:45.410 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:45.417 7f313effd700 1 -- 172.21.2.112:0/3554156048 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3144000c90 con 0x7f31481a3570 2021-02-10T11:34:45.410 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:45.417 7f313effd700 1 -- 172.21.2.112:0/3554156048 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3144002220 con 0x7f31481a3570 2021-02-10T11:34:45.410 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:45.417 7f3150317700 1 -- 172.21.2.112:0/3554156048 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f314809d880 con 0x7f31481a3570 2021-02-10T11:34:45.410 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:45.417 7f3150317700 1 -- 172.21.2.112:0/3554156048 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f314809d880 con 0x7f31481a3570 2021-02-10T11:34:45.410 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:45.418 7f313effd700 1 -- 172.21.2.112:0/3554156048 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f31440052e0 con 0x7f31481a3570 2021-02-10T11:34:45.410 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:45.418 7f313effd700 1 --2- 172.21.2.112:0/3554156048 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f31300186d0 0x7f313001abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:45.410 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:45.418 7f313effd700 1 -- 172.21.2.112:0/3554156048 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 5457+0+0 (secure 0 0 0) 0x7f3144032730 con 0x7f31481a3570 2021-02-10T11:34:45.410 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:45.418 7f314d932700 1 --2- 172.21.2.112:0/3554156048 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f31300186d0 0x7f313001abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:45.410 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:45.419 7f314d932700 1 --2- 172.21.2.112:0/3554156048 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f31300186d0 0x7f313001abd0 crc :-1 s=READY pgs=682 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:45.436 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:45.448 7f3150317700 1 -- 172.21.2.112:0/3554156048 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3148082a70 con 0x7f31481a3570 2021-02-10T11:34:45.438 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:45.451 7f313effd700 1 -- 172.21.2.112:0/3554156048 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3144006b00 con 0x7f31481a3570 2021-02-10T11:34:45.567 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:45.579 7f3150317700 1 -- 172.21.2.112:0/3554156048 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"} v 0) v1 -- 0x7f3148082a70 con 0x7f31481a3570 2021-02-10T11:34:45.572 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:45.584 7f313effd700 1 -- 172.21.2.112:0/3554156048 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"}]=0 new fs with metadata pool 12 and data pool 13 v45) v1 ==== 189+0+0 (secure 0 0 0) 0x7f31440052e0 con 0x7f31481a3570 2021-02-10T11:34:45.574 INFO:teuthology.orchestra.run.gibba012.stderr:new fs with metadata pool 12 and data pool 13 2021-02-10T11:34:45.577 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:45.589 7f3150317700 1 -- 172.21.2.112:0/3554156048 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f31300186d0 msgr2=0x7f313001abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:45.577 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:45.589 7f3150317700 1 --2- 172.21.2.112:0/3554156048 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f31300186d0 0x7f313001abd0 crc :-1 s=READY pgs=682 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:45.577 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:45.589 7f3150317700 1 -- 172.21.2.112:0/3554156048 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f31481a3570 msgr2=0x7f31481a3970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:45.577 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:45.589 7f3150317700 1 --2- 172.21.2.112:0/3554156048 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f31481a3570 0x7f31481a3970 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 rx=0x7f3144006f60 tx=0x7f314400b970).stop 2021-02-10T11:34:45.577 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:45.589 7f3150317700 1 -- 172.21.2.112:0/3554156048 shutdown_connections 2021-02-10T11:34:45.577 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:45.589 7f3150317700 1 --2- 172.21.2.112:0/3554156048 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f31300186d0 0x7f313001abd0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:45.577 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:45.589 7f3150317700 1 --2- 172.21.2.112:0/3554156048 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f314809d410 0x7f3148098170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:45.578 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:45.589 7f3150317700 1 --2- 172.21.2.112:0/3554156048 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f314809c9f0 0x7f314809cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:45.578 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:45.589 7f3150317700 1 --2- 172.21.2.112:0/3554156048 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f31481a3570 0x7f31481a3970 unknown :-1 s=CLOSED pgs=1386 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:45.578 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:45.589 7f3150317700 1 -- 172.21.2.112:0/3554156048 >> 172.21.2.112:0/3554156048 conn(0x7f31480c2cd0 msgr2=0x7f31480b5950 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:45.578 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:45.589 7f3150317700 1 -- 172.21.2.112:0/3554156048 shutdown_connections 2021-02-10T11:34:45.578 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:45.589 7f3150317700 1 -- 172.21.2.112:0/3554156048 wait complete. 2021-02-10T11:34:45.585 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:34:46.009 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.004 7f5393a3a700 1 Processor -- start 2021-02-10T11:34:46.009 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.005 7f5393a3a700 1 -- start start 2021-02-10T11:34:46.009 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.005 7f5393a3a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f538c0d1870 0x7f538c1b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:46.009 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.005 7f5393a3a700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f538c0f0650 0x7f538c0cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:46.009 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.005 7f5393a3a700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f538c0f0b10 con 0x7f538c1b0a10 2021-02-10T11:34:46.009 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.005 7f5393a3a700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f538c0fe230 con 0x7f538c0f0650 2021-02-10T11:34:46.010 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.005 7f5393a3a700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f538c0e5ed0 con 0x7f538c0d1870 2021-02-10T11:34:46.010 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.005 7f5391055700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f538c1b0a10 0x7f538c0cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:50110/0 (socket says 172.21.2.112:50110) 2021-02-10T11:34:46.010 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.005 7f5391055700 1 -- 172.21.2.112:0/3068213748 learned_addr learned my addr 172.21.2.112:0/3068213748 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:34:46.010 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.005 7f5392057700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f538c0f0650 0x7f538c0cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:46.010 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.006 7f5392057700 1 -- 172.21.2.112:0/3068213748 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f538c0d1870 msgr2=0x7f538c1b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:46.010 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.006 7f5392057700 1 --2- 172.21.2.112:0/3068213748 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f538c0d1870 0x7f538c1b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:46.010 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.006 7f5392057700 1 -- 172.21.2.112:0/3068213748 >> v1:172.21.2.112:6789/0 conn(0x7f538c1b0a10 legacy=0x7f538c0cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:46.010 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.006 7f5392057700 1 -- 172.21.2.112:0/3068213748 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f538c0f20a0 con 0x7f538c0f0650 2021-02-10T11:34:46.010 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.006 7f5392057700 1 --2- 172.21.2.112:0/3068213748 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f538c0f0650 0x7f538c0cfb20 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 rx=0x7f5384006e30 tx=0x7f5384009e60).ready entity=mon.0 client_cookie=b13011f9511aa190 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:46.010 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.006 7f5390854700 1 -- 172.21.2.112:0/3068213748 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5384006a50 con 0x7f538c0f0650 2021-02-10T11:34:46.010 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.006 7f5390854700 1 -- 172.21.2.112:0/3068213748 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f538400b300 con 0x7f538c0f0650 2021-02-10T11:34:46.011 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.006 7f5390854700 1 -- 172.21.2.112:0/3068213748 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f538400b820 con 0x7f538c0f0650 2021-02-10T11:34:46.011 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.006 7f5393a3a700 1 -- 172.21.2.112:0/3068213748 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f538c0f0650 msgr2=0x7f538c0cfb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:46.011 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.006 7f5393a3a700 1 --2- 172.21.2.112:0/3068213748 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f538c0f0650 0x7f538c0cfb20 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 rx=0x7f5384006e30 tx=0x7f5384009e60).stop 2021-02-10T11:34:46.011 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.007 7f5393a3a700 1 -- 172.21.2.112:0/3068213748 shutdown_connections 2021-02-10T11:34:46.011 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.007 7f5393a3a700 1 --2- 172.21.2.112:0/3068213748 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f538c0f0650 0x7f538c0cfb20 unknown :-1 s=CLOSED pgs=1388 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:46.011 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.007 7f5393a3a700 1 --2- 172.21.2.112:0/3068213748 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f538c0d1870 0x7f538c1b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:46.011 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.007 7f5393a3a700 1 -- 172.21.2.112:0/3068213748 >> 172.21.2.112:0/3068213748 conn(0x7f538c0ce450 msgr2=0x7f538c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:46.011 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.007 7f5393a3a700 1 -- 172.21.2.112:0/3068213748 shutdown_connections 2021-02-10T11:34:46.011 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.007 7f5393a3a700 1 -- 172.21.2.112:0/3068213748 wait complete. 2021-02-10T11:34:46.013 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.021 7f5393a3a700 1 Processor -- start 2021-02-10T11:34:46.013 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.022 7f5393a3a700 1 -- start start 2021-02-10T11:34:46.013 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.022 7f5393a3a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f538c19af00 0x7f538c19b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:46.013 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.022 7f5393a3a700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f538c19ba80 0x7f538c19be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:46.013 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.022 7f5393a3a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f538c19c4a0 0x7f538c08a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:46.014 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.022 7f5393a3a700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f538c08fb50 con 0x7f538c19c4a0 2021-02-10T11:34:46.014 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.022 7f5393a3a700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f538c08afb0 con 0x7f538c19ba80 2021-02-10T11:34:46.014 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.022 7f5393a3a700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f538c08b1a0 con 0x7f538c19af00 2021-02-10T11:34:46.014 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.022 7f5392057700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f538c19c4a0 0x7f538c08a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:46.014 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.022 7f5392057700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f538c19c4a0 0x7f538c08a990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:50654/0 (socket says 172.21.2.112:50654) 2021-02-10T11:34:46.014 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.022 7f5392057700 1 -- 172.21.2.112:0/235277976 learned_addr learned my addr 172.21.2.112:0/235277976 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:34:46.014 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.022 7f5392057700 1 -- 172.21.2.112:0/235277976 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f538c19af00 msgr2=0x7f538c19b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:46.014 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.022 7f5392057700 1 --2- 172.21.2.112:0/235277976 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f538c19af00 0x7f538c19b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:46.014 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.022 7f5392057700 1 -- 172.21.2.112:0/235277976 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f538c19ba80 msgr2=0x7f538c19be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:46.014 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.022 7f5392057700 1 --2- 172.21.2.112:0/235277976 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f538c19ba80 0x7f538c19be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:46.014 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.022 7f5392057700 1 -- 172.21.2.112:0/235277976 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f538c08b3c0 con 0x7f538c19c4a0 2021-02-10T11:34:46.015 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.023 7f5392057700 1 --2- 172.21.2.112:0/235277976 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f538c19c4a0 0x7f538c08a990 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 rx=0x7f53840099f0 tx=0x7f538400bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:46.015 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.023 7f537e7fc700 1 -- 172.21.2.112:0/235277976 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f53840090d0 con 0x7f538c19c4a0 2021-02-10T11:34:46.015 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.023 7f537e7fc700 1 -- 172.21.2.112:0/235277976 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5384005820 con 0x7f538c19c4a0 2021-02-10T11:34:46.015 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.023 7f5393a3a700 1 -- 172.21.2.112:0/235277976 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f538c08b3c0 con 0x7f538c19c4a0 2021-02-10T11:34:46.015 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.023 7f537e7fc700 1 -- 172.21.2.112:0/235277976 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5384005c70 con 0x7f538c19c4a0 2021-02-10T11:34:46.015 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.023 7f5393a3a700 1 -- 172.21.2.112:0/235277976 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f538c08b3c0 con 0x7f538c19c4a0 2021-02-10T11:34:46.015 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.024 7f537e7fc700 1 -- 172.21.2.112:0/235277976 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5384005a00 con 0x7f538c19c4a0 2021-02-10T11:34:46.015 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.024 7f537e7fc700 1 --2- 172.21.2.112:0/235277976 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f53740186d0 0x7f537401abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:46.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.024 7f537e7fc700 1 -- 172.21.2.112:0/235277976 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f5384032b50 con 0x7f538c19c4a0 2021-02-10T11:34:46.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.024 7f5391856700 1 --2- 172.21.2.112:0/235277976 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f53740186d0 0x7f537401abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:46.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.024 7f5391856700 1 --2- 172.21.2.112:0/235277976 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f53740186d0 0x7f537401abd0 crc :-1 s=READY pgs=683 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:46.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.055 7f5393a3a700 1 -- 172.21.2.112:0/235277976 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f538c0ee030 con 0x7f538c19c4a0 2021-02-10T11:34:46.045 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.058 7f537e7fc700 1 -- 172.21.2.112:0/235277976 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f538400be30 con 0x7f538c19c4a0 2021-02-10T11:34:46.170 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.182 7f5393a3a700 1 -- 172.21.2.112:0/235277976 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f538c0ee030 con 0x7f538c19c4a0 2021-02-10T11:34:46.170 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.183 7f537e7fc700 1 -- 172.21.2.112:0/235277976 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v63) v1 ==== 74+0+14009 (secure 0 0 0) 0x7f5384005820 con 0x7f538c19c4a0 2021-02-10T11:34:46.171 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:34:46.171 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":63,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:34:45.582091","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:34:43.550220","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":13,"pool_name":"cephfs_data","create_time":"2021-02-10 11:34:44.578113","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:34:46.172 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.184 7f5393a3a700 1 -- 172.21.2.112:0/235277976 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f53740186d0 msgr2=0x7f537401abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:46.172 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.184 7f5393a3a700 1 --2- 172.21.2.112:0/235277976 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f53740186d0 0x7f537401abd0 crc :-1 s=READY pgs=683 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:46.172 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.184 7f5393a3a700 1 -- 172.21.2.112:0/235277976 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f538c19c4a0 msgr2=0x7f538c08a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:46.172 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.184 7f5393a3a700 1 --2- 172.21.2.112:0/235277976 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f538c19c4a0 0x7f538c08a990 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 rx=0x7f53840099f0 tx=0x7f538400bcf0).stop 2021-02-10T11:34:46.173 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.185 7f5393a3a700 1 -- 172.21.2.112:0/235277976 shutdown_connections 2021-02-10T11:34:46.173 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.185 7f5393a3a700 1 --2- 172.21.2.112:0/235277976 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f53740186d0 0x7f537401abd0 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:46.173 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.185 7f5393a3a700 1 --2- 172.21.2.112:0/235277976 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f538c19ba80 0x7f538c19be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:46.173 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.185 7f5393a3a700 1 --2- 172.21.2.112:0/235277976 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f538c19c4a0 0x7f538c08a990 unknown :-1 s=CLOSED pgs=1389 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:46.173 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.185 7f5393a3a700 1 --2- 172.21.2.112:0/235277976 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f538c19af00 0x7f538c19b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:46.173 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.185 7f5393a3a700 1 -- 172.21.2.112:0/235277976 >> 172.21.2.112:0/235277976 conn(0x7f538c19f8c0 msgr2=0x7f538c1a5fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:46.173 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.185 7f5393a3a700 1 -- 172.21.2.112:0/235277976 shutdown_connections 2021-02-10T11:34:46.174 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.185 7f5393a3a700 1 -- 172.21.2.112:0/235277976 wait complete. 2021-02-10T11:34:46.181 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:34:46.606 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.603 7f3eda8ba700 1 Processor -- start 2021-02-10T11:34:46.606 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.603 7f3eda8ba700 1 -- start start 2021-02-10T11:34:46.606 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.603 7f3eda8ba700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f3ed40d1c70 0x7f3ed40f20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:46.606 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.603 7f3eda8ba700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ed40f0b10 con 0x7f3ed40d0340 2021-02-10T11:34:46.607 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.603 7f3eda8ba700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ed41b0be0 con 0x7f3ed40d1870 2021-02-10T11:34:46.607 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.603 7f3eda8ba700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ed40e5ed0 con 0x7f3ed40d1c70 2021-02-10T11:34:46.607 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.603 7f3ed37fe700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f3ed40d0340 0x7f3ed40ef640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:50124/0 (socket says 172.21.2.112:50124) 2021-02-10T11:34:46.607 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.603 7f3ed37fe700 1 -- 172.21.2.112:0/639943081 learned_addr learned my addr 172.21.2.112:0/639943081 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:34:46.607 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.603 7f3ed8ed7700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f3ed40d1c70 0x7f3ed40f20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:46.607 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.604 7f3ed8ed7700 1 -- 172.21.2.112:0/639943081 >> v1:172.21.2.120:6789/0 conn(0x7f3ed40d1870 legacy=0x7f3ed41b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:46.607 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.604 7f3ed8ed7700 1 -- 172.21.2.112:0/639943081 >> v1:172.21.2.112:6789/0 conn(0x7f3ed40d0340 legacy=0x7f3ed40ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:46.607 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.604 7f3ed8ed7700 1 -- 172.21.2.112:0/639943081 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ed40efe30 con 0x7f3ed40d1c70 2021-02-10T11:34:46.607 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.604 7f3ed8ed7700 1 --2- 172.21.2.112:0/639943081 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3ed40d1c70 0x7f3ed40f20a0 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 rx=0x7f3ecc006e30 tx=0x7f3ecc009de0).ready entity=mon.0 client_cookie=8eaf5cea7b3f3930 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:46.607 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.604 7f3ed2ffd700 1 -- 172.21.2.112:0/639943081 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3ecc00b300 con 0x7f3ed40d1c70 2021-02-10T11:34:46.607 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.604 7f3ed2ffd700 1 -- 172.21.2.112:0/639943081 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ecc000d30 con 0x7f3ed40d1c70 2021-02-10T11:34:46.607 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.604 7f3ed2ffd700 1 -- 172.21.2.112:0/639943081 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3ecc00b900 con 0x7f3ed40d1c70 2021-02-10T11:34:46.607 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.604 7f3eda8ba700 1 -- 172.21.2.112:0/639943081 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3ed40d1c70 msgr2=0x7f3ed40f20a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:46.608 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.604 7f3eda8ba700 1 --2- 172.21.2.112:0/639943081 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3ed40d1c70 0x7f3ed40f20a0 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 rx=0x7f3ecc006e30 tx=0x7f3ecc009de0).stop 2021-02-10T11:34:46.608 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.604 7f3eda8ba700 1 -- 172.21.2.112:0/639943081 shutdown_connections 2021-02-10T11:34:46.608 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.604 7f3eda8ba700 1 --2- 172.21.2.112:0/639943081 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3ed40d1c70 0x7f3ed40f20a0 unknown :-1 s=CLOSED pgs=1391 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:46.608 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.604 7f3eda8ba700 1 -- 172.21.2.112:0/639943081 >> 172.21.2.112:0/639943081 conn(0x7f3ed40ce450 msgr2=0x7f3ed40d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:46.608 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.605 7f3eda8ba700 1 -- 172.21.2.112:0/639943081 shutdown_connections 2021-02-10T11:34:46.608 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.605 7f3eda8ba700 1 -- 172.21.2.112:0/639943081 wait complete. 2021-02-10T11:34:46.610 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.619 7f3eda8ba700 1 Processor -- start 2021-02-10T11:34:46.610 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.619 7f3eda8ba700 1 -- start start 2021-02-10T11:34:46.610 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.619 7f3eda8ba700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3ed40af040 0x7f3ed40af440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:46.610 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.619 7f3eda8ba700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3ed40afa60 0x7f3ed40ada20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:46.610 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.619 7f3eda8ba700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3ed40afe60 0x7f3ed40ae040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:46.610 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.619 7f3eda8ba700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ed40923c0 con 0x7f3ed40afa60 2021-02-10T11:34:46.610 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.619 7f3eda8ba700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ed40b0410 con 0x7f3ed40af040 2021-02-10T11:34:46.610 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.619 7f3eda8ba700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ed4091e40 con 0x7f3ed40afe60 2021-02-10T11:34:46.610 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.620 7f3ed37fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3ed40afa60 0x7f3ed40ada20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:46.610 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.620 7f3ed37fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3ed40afa60 0x7f3ed40ada20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:50666/0 (socket says 172.21.2.112:50666) 2021-02-10T11:34:46.610 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.620 7f3ed37fe700 1 -- 172.21.2.112:0/321868995 learned_addr learned my addr 172.21.2.112:0/321868995 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:34:46.611 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.620 7f3ed8ed7700 1 --2- 172.21.2.112:0/321868995 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3ed40afe60 0x7f3ed40ae040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:46.611 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.620 7f3ed3fff700 1 --2- 172.21.2.112:0/321868995 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3ed40af040 0x7f3ed40af440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:46.611 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.620 7f3ed37fe700 1 -- 172.21.2.112:0/321868995 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3ed40afe60 msgr2=0x7f3ed40ae040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:46.611 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.620 7f3ed37fe700 1 --2- 172.21.2.112:0/321868995 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3ed40afe60 0x7f3ed40ae040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:46.611 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.620 7f3ed37fe700 1 -- 172.21.2.112:0/321868995 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3ed40af040 msgr2=0x7f3ed40af440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:46.611 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.620 7f3ed37fe700 1 --2- 172.21.2.112:0/321868995 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3ed40af040 0x7f3ed40af440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:46.611 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.620 7f3ed37fe700 1 -- 172.21.2.112:0/321868995 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ed408f6d0 con 0x7f3ed40afa60 2021-02-10T11:34:46.611 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.620 7f3ed37fe700 1 --2- 172.21.2.112:0/321868995 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3ed40afa60 0x7f3ed40ada20 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 rx=0x7f3ec8001c60 tx=0x7f3ec800a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:46.611 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.620 7f3ed0ff9700 1 -- 172.21.2.112:0/321868995 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3ec800acc0 con 0x7f3ed40afa60 2021-02-10T11:34:46.611 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.620 7f3ed0ff9700 1 -- 172.21.2.112:0/321868995 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ec8004140 con 0x7f3ed40afa60 2021-02-10T11:34:46.611 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.620 7f3ed0ff9700 1 -- 172.21.2.112:0/321868995 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3ec8004620 con 0x7f3ed40afa60 2021-02-10T11:34:46.611 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.620 7f3eda8ba700 1 -- 172.21.2.112:0/321868995 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ed408f6d0 con 0x7f3ed40afa60 2021-02-10T11:34:46.611 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.620 7f3eda8ba700 1 -- 172.21.2.112:0/321868995 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ed408f6d0 con 0x7f3ed40afa60 2021-02-10T11:34:46.611 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.621 7f3ed0ff9700 1 -- 172.21.2.112:0/321868995 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3ec8000ce0 con 0x7f3ed40afa60 2021-02-10T11:34:46.612 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.621 7f3ed0ff9700 1 --2- 172.21.2.112:0/321868995 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3eb40186d0 0x7f3eb401abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:46.612 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.621 7f3ed0ff9700 1 -- 172.21.2.112:0/321868995 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f3ec80327a0 con 0x7f3ed40afa60 2021-02-10T11:34:46.612 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.621 7f3ed3fff700 1 --2- 172.21.2.112:0/321868995 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3eb40186d0 0x7f3eb401abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:46.612 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.622 7f3ed3fff700 1 --2- 172.21.2.112:0/321868995 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3eb40186d0 0x7f3eb401abd0 crc :-1 s=READY pgs=684 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:46.641 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.653 7f3eda8ba700 1 -- 172.21.2.112:0/321868995 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ed40e7df0 con 0x7f3ed40afa60 2021-02-10T11:34:46.643 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.656 7f3ed0ff9700 1 -- 172.21.2.112:0/321868995 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3ec80009b0 con 0x7f3ed40afa60 2021-02-10T11:34:46.767 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.779 7f3eda8ba700 1 -- 172.21.2.112:0/321868995 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3ed40ce450 con 0x7f3ed40afa60 2021-02-10T11:34:46.768 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.780 7f3ed0ff9700 1 -- 172.21.2.112:0/321868995 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v64) v1 ==== 74+0+13996 (secure 0 0 0) 0x7f3ec8000ce0 con 0x7f3ed40afa60 2021-02-10T11:34:46.769 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:34:46.769 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":64,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:34:46.584716","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:34:43.550220","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":13,"pool_name":"cephfs_data","create_time":"2021-02-10 11:34:44.578113","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:34:46.770 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.782 7f3eda8ba700 1 -- 172.21.2.112:0/321868995 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3eb40186d0 msgr2=0x7f3eb401abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:46.770 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.782 7f3eda8ba700 1 --2- 172.21.2.112:0/321868995 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3eb40186d0 0x7f3eb401abd0 crc :-1 s=READY pgs=684 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:46.770 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.782 7f3eda8ba700 1 -- 172.21.2.112:0/321868995 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3ed40afa60 msgr2=0x7f3ed40ada20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:46.770 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.782 7f3eda8ba700 1 --2- 172.21.2.112:0/321868995 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3ed40afa60 0x7f3ed40ada20 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 rx=0x7f3ec8001c60 tx=0x7f3ec800a8d0).stop 2021-02-10T11:34:46.770 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.782 7f3eda8ba700 1 -- 172.21.2.112:0/321868995 shutdown_connections 2021-02-10T11:34:46.770 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.782 7f3eda8ba700 1 --2- 172.21.2.112:0/321868995 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3eb40186d0 0x7f3eb401abd0 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:46.770 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.782 7f3eda8ba700 1 --2- 172.21.2.112:0/321868995 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3ed40afe60 0x7f3ed40ae040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:46.770 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.782 7f3eda8ba700 1 --2- 172.21.2.112:0/321868995 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3ed40afa60 0x7f3ed40ada20 unknown :-1 s=CLOSED pgs=1392 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:46.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.782 7f3eda8ba700 1 --2- 172.21.2.112:0/321868995 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3ed40af040 0x7f3ed40af440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:46.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.782 7f3eda8ba700 1 -- 172.21.2.112:0/321868995 >> 172.21.2.112:0/321868995 conn(0x7f3ed40a0030 msgr2=0x7f3ed40f22b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:46.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.783 7f3eda8ba700 1 -- 172.21.2.112:0/321868995 shutdown_connections 2021-02-10T11:34:46.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:46.783 7f3eda8ba700 1 -- 172.21.2.112:0/321868995 wait complete. 2021-02-10T11:34:46.779 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2021-02-10T11:34:47.210 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:47.204 7f396c92e700 1 Processor -- start 2021-02-10T11:34:47.210 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:47.204 7f396c92e700 1 -- start start 2021-02-10T11:34:47.210 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:47.204 7f396c92e700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f39680d1c70 0x7f39680cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:47.211 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:47.204 7f396c92e700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39680d0170 con 0x7f39680d1870 2021-02-10T11:34:47.211 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:47.204 7f396c92e700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39680fe1c0 con 0x7f39680cf580 2021-02-10T11:34:47.211 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:47.204 7f396c92e700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39681b0cb0 con 0x7f39680d1c70 2021-02-10T11:34:47.211 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:47.205 7f396661c700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f39680d1870 0x7f39681b04c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:50136/0 (socket says 172.21.2.112:50136) 2021-02-10T11:34:47.211 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:47.205 7f396661c700 1 -- 172.21.2.112:0/2972985088 learned_addr learned my addr 172.21.2.112:0/2972985088 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:34:47.211 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:47.205 7f396561a700 1 -- 172.21.2.112:0/2972985088 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4246788558 0 0) 0x7f3958001180 con 0x7f39680d1870 2021-02-10T11:34:47.211 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:47.205 7f396561a700 1 -- 172.21.2.112:0/2972985088 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3950002fe0 con 0x7f39680d1870 2021-02-10T11:34:47.211 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:47.205 7f3965e1b700 1 --2- 172.21.2.112:0/2972985088 >> v2:172.21.2.120:3300/0 conn(0x7f39680d1c70 0x7f39680cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:47.211 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:47.205 7f396561a700 1 -- 172.21.2.112:0/2972985088 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3045385872 0 0) 0x7f3958001180 con 0x7f39680d1870 2021-02-10T11:34:47.211 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:47.205 7f396561a700 1 -- 172.21.2.112:0/2972985088 >> v2:172.21.2.120:3300/0 conn(0x7f39680d1c70 msgr2=0x7f39680cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:47.211 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:47.205 7f396561a700 1 --2- 172.21.2.112:0/2972985088 >> v2:172.21.2.120:3300/0 conn(0x7f39680d1c70 0x7f39680cef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:47.211 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:47.205 7f396561a700 1 -- 172.21.2.112:0/2972985088 >> v1:172.21.2.120:6789/0 conn(0x7f39680cf580 legacy=0x7f39680cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:47.211 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:47.205 7f396561a700 1 -- 172.21.2.112:0/2972985088 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3968109a60 con 0x7f39680d1870 2021-02-10T11:34:47.211 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:47.205 7f396561a700 1 -- 172.21.2.112:0/2972985088 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f3958001680 con 0x7f39680d1870 2021-02-10T11:34:47.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:47.205 7f396561a700 1 -- 172.21.2.112:0/2972985088 >> v1:172.21.2.112:6789/0 conn(0x7f39680d1870 legacy=0x7f39681b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:47.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:47.205 7f396561a700 1 --2- 172.21.2.112:0/2972985088 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3950003ac0 0x7f3950005ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:47.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:47.205 7f396561a700 1 --2- 172.21.2.112:0/2972985088 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3950006770 0x7f3950008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:47.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:47.206 7f396561a700 1 --2- 172.21.2.112:0/2972985088 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3950009590 0x7f395000bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:47.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:47.206 7f396561a700 1 -- 172.21.2.112:0/2972985088 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f395000c300 con 0x7f3950006770 2021-02-10T11:34:47.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:47.206 7f396561a700 1 -- 172.21.2.112:0/2972985088 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f395000c520 con 0x7f3950009590 2021-02-10T11:34:47.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:47.206 7f396561a700 1 -- 172.21.2.112:0/2972985088 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3950050750 con 0x7f3950003ac0 2021-02-10T11:34:47.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:47.206 7f3965e1b700 1 --2- 172.21.2.112:0/2972985088 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3950006770 0x7f3950008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:47.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:47.206 7f396661c700 1 --2- 172.21.2.112:0/2972985088 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3950003ac0 0x7f3950005ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:47.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:47.206 7f3966e1d700 1 --2- 172.21.2.112:0/2972985088 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3950009590 0x7f395000bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:47.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:47.206 7f3965e1b700 1 -- 172.21.2.112:0/2972985088 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3950003ac0 msgr2=0x7f3950005ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:47.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:47.206 7f3965e1b700 1 --2- 172.21.2.112:0/2972985088 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3950003ac0 0x7f3950005ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:47.213 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:47.206 7f3965e1b700 1 -- 172.21.2.112:0/2972985088 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3950009590 msgr2=0x7f395000bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:47.213 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:47.206 7f3965e1b700 1 --2- 172.21.2.112:0/2972985088 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3950009590 0x7f395000bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:47.213 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:47.206 7f3965e1b700 1 -- 172.21.2.112:0/2972985088 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39500509b0 con 0x7f3950006770 2021-02-10T11:34:47.213 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:47.206 7f3966e1d700 1 -- 172.21.2.112:0/2972985088 reap_dead start 2021-02-10T11:34:47.213 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:47.206 7f3965e1b700 1 --2- 172.21.2.112:0/2972985088 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3950006770 0x7f3950008c90 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 rx=0x7f395c00c8f0 tx=0x7f395c006c20).ready entity=mon.0 client_cookie=17fb3f247ff1d24 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:47.213 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:47.206 7f396561a700 1 -- 172.21.2.112:0/2972985088 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f395c0090d0 con 0x7f3950006770 2021-02-10T11:34:47.213 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:47.206 7f396561a700 1 -- 172.21.2.112:0/2972985088 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f395c009450 con 0x7f3950006770 2021-02-10T11:34:47.213 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:47.206 7f396561a700 1 -- 172.21.2.112:0/2972985088 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f395c0090d0 con 0x7f3950006770 2021-02-10T11:34:47.213 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:47.206 7f396c92e700 1 -- 172.21.2.112:0/2972985088 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3950006770 msgr2=0x7f3950008c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:47.213 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:47.206 7f396c92e700 1 --2- 172.21.2.112:0/2972985088 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3950006770 0x7f3950008c90 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 rx=0x7f395c00c8f0 tx=0x7f395c006c20).stop 2021-02-10T11:34:47.213 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:47.207 7f396c92e700 1 -- 172.21.2.112:0/2972985088 shutdown_connections 2021-02-10T11:34:47.213 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:47.207 7f396c92e700 1 --2- 172.21.2.112:0/2972985088 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3950006770 0x7f3950008c90 unknown :-1 s=CLOSED pgs=1394 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:47.213 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:47.207 7f396c92e700 1 -- 172.21.2.112:0/2972985088 >> 172.21.2.112:0/2972985088 conn(0x7f39680ce420 msgr2=0x7f39680d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:47.213 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:47.207 7f396c92e700 1 -- 172.21.2.112:0/2972985088 shutdown_connections 2021-02-10T11:34:47.214 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:47.207 7f396c92e700 1 -- 172.21.2.112:0/2972985088 wait complete. 2021-02-10T11:34:47.214 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:47.221 7f396c92e700 1 Processor -- start 2021-02-10T11:34:47.214 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:47.221 7f396c92e700 1 -- start start 2021-02-10T11:34:47.214 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:47.222 7f396c92e700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f39680af880 0x7f39680830b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:47.214 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:47.222 7f396c92e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f39680836d0 0x7f39680926d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:47.214 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:47.222 7f396c92e700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3968092cf0 0x7f39680930f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:47.214 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:47.222 7f396c92e700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39680bedf0 con 0x7f39680836d0 2021-02-10T11:34:47.214 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:47.222 7f396c92e700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39680afc80 con 0x7f3968092cf0 2021-02-10T11:34:47.214 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:47.222 7f396c92e700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3968083ad0 con 0x7f39680af880 2021-02-10T11:34:47.214 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:47.222 7f3965e1b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f39680836d0 0x7f39680926d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:47.214 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:47.222 7f3965e1b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f39680836d0 0x7f39680926d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:50686/0 (socket says 172.21.2.112:50686) 2021-02-10T11:34:47.214 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:47.222 7f3965e1b700 1 -- 172.21.2.112:0/3839826545 learned_addr learned my addr 172.21.2.112:0/3839826545 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:34:47.214 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:47.222 7f3965e1b700 1 -- 172.21.2.112:0/3839826545 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f39680af880 msgr2=0x7f39680830b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:34:47.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:47.222 7f3965e1b700 1 --2- 172.21.2.112:0/3839826545 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f39680af880 0x7f39680830b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:47.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:47.222 7f3965e1b700 1 -- 172.21.2.112:0/3839826545 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3968092cf0 msgr2=0x7f39680930f0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:34:47.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:47.222 7f3965e1b700 1 --2- 172.21.2.112:0/3839826545 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3968092cf0 0x7f39680930f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:47.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:47.222 7f3965e1b700 1 -- 172.21.2.112:0/3839826545 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3968082ca0 con 0x7f39680836d0 2021-02-10T11:34:47.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:47.222 7f3965e1b700 1 --2- 172.21.2.112:0/3839826545 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f39680836d0 0x7f39680926d0 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 rx=0x7f395c006da0 tx=0x7f395c00b300).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:47.216 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:47.223 7f3956ffd700 1 -- 172.21.2.112:0/3839826545 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f395c000c90 con 0x7f39680836d0 2021-02-10T11:34:47.216 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:47.223 7f3956ffd700 1 -- 172.21.2.112:0/3839826545 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f395c010b80 con 0x7f39680836d0 2021-02-10T11:34:47.216 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:47.223 7f3956ffd700 1 -- 172.21.2.112:0/3839826545 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f395c00b6f0 con 0x7f39680836d0 2021-02-10T11:34:47.216 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:47.223 7f396c92e700 1 -- 172.21.2.112:0/3839826545 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3968082ca0 con 0x7f39680836d0 2021-02-10T11:34:47.216 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:47.223 7f396c92e700 1 -- 172.21.2.112:0/3839826545 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39680bde00 con 0x7f39680836d0 2021-02-10T11:34:47.216 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:47.223 7f3956ffd700 1 -- 172.21.2.112:0/3839826545 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f395c000c90 con 0x7f39680836d0 2021-02-10T11:34:47.216 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:47.223 7f3956ffd700 1 --2- 172.21.2.112:0/3839826545 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f394c0186c0 0x7f394c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:47.216 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:47.223 7f3956ffd700 1 -- 172.21.2.112:0/3839826545 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f395c0367f0 con 0x7f39680836d0 2021-02-10T11:34:47.216 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:47.226 7f396661c700 1 --2- 172.21.2.112:0/3839826545 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f394c0186c0 0x7f394c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:47.216 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:47.226 7f396661c700 1 --2- 172.21.2.112:0/3839826545 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f394c0186c0 0x7f394c01abc0 crc :-1 s=READY pgs=685 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:47.242 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:47.255 7f396c92e700 1 -- 172.21.2.112:0/3839826545 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39680bed70 con 0x7f39680836d0 2021-02-10T11:34:47.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:47.257 7f3956ffd700 1 -- 172.21.2.112:0/3839826545 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f395c006740 con 0x7f39680836d0 2021-02-10T11:34:47.381 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:47.393 7f396c92e700 1 -- 172.21.2.112:0/3839826545 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"} v 0) v1 -- 0x7f39680bed70 con 0x7f39680836d0 2021-02-10T11:34:47.431 DEBUG:teuthology.orchestra.run.gibba012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:34:47.434 DEBUG:teuthology.orchestra.run.gibba020:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:34:47.581 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:47.593 7f3956ffd700 1 -- 172.21.2.112:0/3839826545 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"}]=0 v47) v1 ==== 118+0+0 (secure 0 0 0) 0x7f395c000c90 con 0x7f39680836d0 2021-02-10T11:34:47.598 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:47.609 7f396c92e700 1 -- 172.21.2.112:0/3839826545 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f394c0186c0 msgr2=0x7f394c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:47.598 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:47.609 7f396c92e700 1 --2- 172.21.2.112:0/3839826545 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f394c0186c0 0x7f394c01abc0 crc :-1 s=READY pgs=685 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:47.598 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:47.609 7f396c92e700 1 -- 172.21.2.112:0/3839826545 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f39680836d0 msgr2=0x7f39680926d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:47.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:47.609 7f396c92e700 1 --2- 172.21.2.112:0/3839826545 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f39680836d0 0x7f39680926d0 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 rx=0x7f395c006da0 tx=0x7f395c00b300).stop 2021-02-10T11:34:47.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:47.610 7f396c92e700 1 -- 172.21.2.112:0/3839826545 shutdown_connections 2021-02-10T11:34:47.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:47.610 7f396c92e700 1 --2- 172.21.2.112:0/3839826545 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f394c0186c0 0x7f394c01abc0 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:47.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:47.610 7f396c92e700 1 --2- 172.21.2.112:0/3839826545 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3968092cf0 0x7f39680930f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:47.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:47.610 7f396c92e700 1 --2- 172.21.2.112:0/3839826545 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f39680836d0 0x7f39680926d0 unknown :-1 s=CLOSED pgs=1395 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:47.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:47.610 7f396c92e700 1 --2- 172.21.2.112:0/3839826545 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f39680af880 0x7f39680830b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:47.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:47.610 7f396c92e700 1 -- 172.21.2.112:0/3839826545 >> 172.21.2.112:0/3839826545 conn(0x7f39680affc0 msgr2=0x7f396809ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:47.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:47.610 7f396c92e700 1 -- 172.21.2.112:0/3839826545 shutdown_connections 2021-02-10T11:34:47.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:47.610 7f396c92e700 1 -- 172.21.2.112:0/3839826545 wait complete. 2021-02-10T11:34:47.606 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:34:48.033 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.028 7f3d97872700 1 Processor -- start 2021-02-10T11:34:48.034 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.028 7f3d97872700 1 -- start start 2021-02-10T11:34:48.034 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.029 7f3d97872700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f3d900d1c70 0x7f3d900f20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:48.034 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.029 7f3d97872700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d900f0b10 con 0x7f3d900d0340 2021-02-10T11:34:48.034 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.029 7f3d97872700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d901b0be0 con 0x7f3d900d1870 2021-02-10T11:34:48.035 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.029 7f3d97872700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d900e5ed0 con 0x7f3d900d1c70 2021-02-10T11:34:48.035 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.029 7f3d94e8d700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f3d900d0340 0x7f3d900ef640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:50156/0 (socket says 172.21.2.112:50156) 2021-02-10T11:34:48.035 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.029 7f3d94e8d700 1 -- 172.21.2.112:0/1212954859 learned_addr learned my addr 172.21.2.112:0/1212954859 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:34:48.035 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.029 7f3d95e8f700 1 --2- 172.21.2.112:0/1212954859 >> v2:172.21.2.120:3300/0 conn(0x7f3d900d1c70 0x7f3d900f20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:48.035 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.029 7f3d83fff700 1 -- 172.21.2.112:0/1212954859 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3044013652 0 0) 0x7f3d84001060 con 0x7f3d900d0340 2021-02-10T11:34:48.035 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.030 7f3d83fff700 1 -- 172.21.2.112:0/1212954859 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d7c002fe0 con 0x7f3d900d0340 2021-02-10T11:34:48.036 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.030 7f3d95e8f700 1 -- 172.21.2.112:0/1212954859 >> v1:172.21.2.120:6789/0 conn(0x7f3d900d1870 legacy=0x7f3d901b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:48.036 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.030 7f3d95e8f700 1 -- 172.21.2.112:0/1212954859 >> v1:172.21.2.112:6789/0 conn(0x7f3d900d0340 legacy=0x7f3d900ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:48.036 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.030 7f3d83fff700 1 -- 172.21.2.112:0/1212954859 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3630873292 0 0) 0x7f3d84001460 con 0x7f3d900d0340 2021-02-10T11:34:48.036 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.030 7f3d95e8f700 1 -- 172.21.2.112:0/1212954859 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d900efe30 con 0x7f3d900d1c70 2021-02-10T11:34:48.036 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.030 7f3d95e8f700 1 --2- 172.21.2.112:0/1212954859 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3d900d1c70 0x7f3d900f20a0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 rx=0x7f3d88006ad0 tx=0x7f3d88009ad0).ready entity=mon.1 client_cookie=37e7d913cd1c426 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:48.036 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.030 7f3d83fff700 1 -- 172.21.2.112:0/1212954859 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3d880022e0 con 0x7f3d900d1c70 2021-02-10T11:34:48.036 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.030 7f3d83fff700 1 -- 172.21.2.112:0/1212954859 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d88006410 con 0x7f3d900d1c70 2021-02-10T11:34:48.036 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.030 7f3d83fff700 1 -- 172.21.2.112:0/1212954859 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3d88002620 con 0x7f3d900d1c70 2021-02-10T11:34:48.037 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.030 7f3d97872700 1 -- 172.21.2.112:0/1212954859 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3d900d1c70 msgr2=0x7f3d900f20a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:48.037 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.030 7f3d97872700 1 --2- 172.21.2.112:0/1212954859 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3d900d1c70 0x7f3d900f20a0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 rx=0x7f3d88006ad0 tx=0x7f3d88009ad0).stop 2021-02-10T11:34:48.037 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.031 7f3d97872700 1 -- 172.21.2.112:0/1212954859 shutdown_connections 2021-02-10T11:34:48.037 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.031 7f3d97872700 1 --2- 172.21.2.112:0/1212954859 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3d900d1c70 0x7f3d900f20a0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:48.037 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.031 7f3d97872700 1 -- 172.21.2.112:0/1212954859 >> 172.21.2.112:0/1212954859 conn(0x7f3d900ce450 msgr2=0x7f3d900d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:48.037 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.031 7f3d97872700 1 -- 172.21.2.112:0/1212954859 shutdown_connections 2021-02-10T11:34:48.037 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.031 7f3d97872700 1 -- 172.21.2.112:0/1212954859 wait complete. 2021-02-10T11:34:48.038 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.045 7f3d97872700 1 Processor -- start 2021-02-10T11:34:48.038 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.045 7f3d97872700 1 -- start start 2021-02-10T11:34:48.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.046 7f3d97872700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3d901b8ad0 0x7f3d90187e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:48.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.046 7f3d97872700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3d901b8ed0 0x7f3d900b1af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:48.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.046 7f3d97872700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3d900b2110 0x7f3d901b7a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:48.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.046 7f3d97872700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d90188470 con 0x7f3d901b8ed0 2021-02-10T11:34:48.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.046 7f3d97872700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d9018be10 con 0x7f3d900b2110 2021-02-10T11:34:48.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.046 7f3d97872700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d9018c030 con 0x7f3d901b8ad0 2021-02-10T11:34:48.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.046 7f3d94e8d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3d901b8ed0 0x7f3d900b1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:48.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.046 7f3d94e8d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3d901b8ed0 0x7f3d900b1af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:50704/0 (socket says 172.21.2.112:50704) 2021-02-10T11:34:48.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.046 7f3d94e8d700 1 -- 172.21.2.112:0/520288476 learned_addr learned my addr 172.21.2.112:0/520288476 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:34:48.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.047 7f3d95e8f700 1 --2- 172.21.2.112:0/520288476 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3d900b2110 0x7f3d901b7a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:48.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.047 7f3d9568e700 1 --2- 172.21.2.112:0/520288476 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3d901b8ad0 0x7f3d90187e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:48.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.047 7f3d94e8d700 1 -- 172.21.2.112:0/520288476 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3d901b8ad0 msgr2=0x7f3d90187e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:48.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.047 7f3d94e8d700 1 --2- 172.21.2.112:0/520288476 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3d901b8ad0 0x7f3d90187e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:48.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.047 7f3d94e8d700 1 -- 172.21.2.112:0/520288476 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3d900b2110 msgr2=0x7f3d901b7a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:48.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.047 7f3d94e8d700 1 --2- 172.21.2.112:0/520288476 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3d900b2110 0x7f3d901b7a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:48.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.047 7f3d94e8d700 1 -- 172.21.2.112:0/520288476 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d900b2510 con 0x7f3d901b8ed0 2021-02-10T11:34:48.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.047 7f3d95e8f700 1 --2- 172.21.2.112:0/520288476 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3d900b2110 0x7f3d901b7a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:34:48.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.047 7f3d9568e700 1 --2- 172.21.2.112:0/520288476 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3d901b8ad0 0x7f3d90187e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:34:48.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.047 7f3d94e8d700 1 --2- 172.21.2.112:0/520288476 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3d901b8ed0 0x7f3d900b1af0 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 rx=0x7f3d8400a920 tx=0x7f3d8400acc0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:48.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.047 7f3d81ffb700 1 -- 172.21.2.112:0/520288476 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3d84007020 con 0x7f3d901b8ed0 2021-02-10T11:34:48.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.047 7f3d81ffb700 1 -- 172.21.2.112:0/520288476 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d840072e0 con 0x7f3d901b8ed0 2021-02-10T11:34:48.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.047 7f3d97872700 1 -- 172.21.2.112:0/520288476 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d900b2510 con 0x7f3d901b8ed0 2021-02-10T11:34:48.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.047 7f3d81ffb700 1 -- 172.21.2.112:0/520288476 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3d84004720 con 0x7f3d901b8ed0 2021-02-10T11:34:48.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.047 7f3d97872700 1 -- 172.21.2.112:0/520288476 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d900b2510 con 0x7f3d901b8ed0 2021-02-10T11:34:48.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.047 7f3d81ffb700 1 -- 172.21.2.112:0/520288476 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3d84001cc0 con 0x7f3d901b8ed0 2021-02-10T11:34:48.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.048 7f3d81ffb700 1 --2- 172.21.2.112:0/520288476 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3d70018700 0x7f3d7001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:48.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.048 7f3d81ffb700 1 -- 172.21.2.112:0/520288476 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f3d84032990 con 0x7f3d901b8ed0 2021-02-10T11:34:48.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.048 7f3d9568e700 1 --2- 172.21.2.112:0/520288476 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3d70018700 0x7f3d7001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:48.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.049 7f3d9568e700 1 --2- 172.21.2.112:0/520288476 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3d70018700 0x7f3d7001abc0 crc :-1 s=READY pgs=686 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:48.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.079 7f3d97872700 1 -- 172.21.2.112:0/520288476 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d900ebc00 con 0x7f3d901b8ed0 2021-02-10T11:34:48.069 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.081 7f3d81ffb700 1 -- 172.21.2.112:0/520288476 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3d84001cc0 con 0x7f3d901b8ed0 2021-02-10T11:34:48.197 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.209 7f3d97872700 1 -- 172.21.2.112:0/520288476 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3d900f0b10 con 0x7f3d901b8ed0 2021-02-10T11:34:48.198 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.210 7f3d81ffb700 1 -- 172.21.2.112:0/520288476 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) v1 ==== 94+0+2101 (secure 0 0 0) 0x7f3d840070b0 con 0x7f3d901b8ed0 2021-02-10T11:34:48.198 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:34:48.198 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":47,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8334,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/3370618110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":3370618110},{"type":"v1","addr":"172.21.2.112:6833","nonce":3370618110}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":44}],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:34:45.582177","modified":"2021-02-10 11:34:47.589305","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5921},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5921":{"gid":5921,"name":"b","rank":0,"incarnation":45,"state":"up:active","state_seq":3,"addr":"172.21.2.120:6835/2436135065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":2436135065},{"type":"v1","addr":"172.21.2.120:6835","nonce":2436135065}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2021-02-10T11:34:48.199 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 47 2021-02-10T11:34:48.200 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.212 7f3d97872700 1 -- 172.21.2.112:0/520288476 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3d70018700 msgr2=0x7f3d7001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:48.200 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.212 7f3d97872700 1 --2- 172.21.2.112:0/520288476 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3d70018700 0x7f3d7001abc0 crc :-1 s=READY pgs=686 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:48.200 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.212 7f3d97872700 1 -- 172.21.2.112:0/520288476 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3d901b8ed0 msgr2=0x7f3d900b1af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:48.200 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.212 7f3d97872700 1 --2- 172.21.2.112:0/520288476 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3d901b8ed0 0x7f3d900b1af0 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 rx=0x7f3d8400a920 tx=0x7f3d8400acc0).stop 2021-02-10T11:34:48.200 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.212 7f3d97872700 1 -- 172.21.2.112:0/520288476 shutdown_connections 2021-02-10T11:34:48.200 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.212 7f3d97872700 1 --2- 172.21.2.112:0/520288476 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3d70018700 0x7f3d7001abc0 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:48.200 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.212 7f3d97872700 1 --2- 172.21.2.112:0/520288476 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3d900b2110 0x7f3d901b7a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:48.200 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.212 7f3d97872700 1 --2- 172.21.2.112:0/520288476 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3d901b8ed0 0x7f3d900b1af0 unknown :-1 s=CLOSED pgs=1397 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:48.200 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.212 7f3d97872700 1 --2- 172.21.2.112:0/520288476 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3d901b8ad0 0x7f3d90187e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:48.201 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.212 7f3d97872700 1 -- 172.21.2.112:0/520288476 >> 172.21.2.112:0/520288476 conn(0x7f3d900ed810 msgr2=0x7f3d90094980 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:48.201 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.212 7f3d97872700 1 -- 172.21.2.112:0/520288476 shutdown_connections 2021-02-10T11:34:48.201 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.212 7f3d97872700 1 -- 172.21.2.112:0/520288476 wait complete. 2021-02-10T11:34:48.208 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:34:48.638 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.632 7fcc52d45700 1 Processor -- start 2021-02-10T11:34:48.638 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.632 7fcc52d45700 1 -- start start 2021-02-10T11:34:48.638 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.633 7fcc52d45700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcc4c1b0870 0x7fcc4c0f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:48.638 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.633 7fcc52d45700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc4c1b0c70 con 0x7fcc4c1b0470 2021-02-10T11:34:48.638 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.633 7fcc52d45700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc4c0f0b50 con 0x7fcc4c0efe00 2021-02-10T11:34:48.638 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.633 7fcc52d45700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc4c0e5ea0 con 0x7fcc4c1b0870 2021-02-10T11:34:48.638 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.633 7fcc50b61700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fcc4c1b0470 0x7fcc4c0ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:50170/0 (socket says 172.21.2.112:50170) 2021-02-10T11:34:48.638 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.633 7fcc50b61700 1 -- 172.21.2.112:0/366905607 learned_addr learned my addr 172.21.2.112:0/366905607 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:34:48.639 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.633 7fcc4b7fe700 1 -- 172.21.2.112:0/366905607 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2328709188 0 0) 0x7fcc340009b0 con 0x7fcc4c1b0470 2021-02-10T11:34:48.639 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.633 7fcc4b7fe700 1 -- 172.21.2.112:0/366905607 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc38002fe0 con 0x7fcc4c1b0470 2021-02-10T11:34:48.639 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.633 7fcc4bfff700 1 --2- 172.21.2.112:0/366905607 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcc4c1b0870 0x7fcc4c0f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:48.639 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.634 7fcc4b7fe700 1 -- 172.21.2.112:0/366905607 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3785711549 0 0) 0x7fcc34001010 con 0x7fcc4c1b0470 2021-02-10T11:34:48.639 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.634 7fcc4b7fe700 1 -- 172.21.2.112:0/366905607 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcc4c1b0870 msgr2=0x7fcc4c0f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:48.639 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.634 7fcc4b7fe700 1 --2- 172.21.2.112:0/366905607 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcc4c1b0870 0x7fcc4c0f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:48.639 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.634 7fcc4b7fe700 1 -- 172.21.2.112:0/366905607 >> v1:172.21.2.120:6789/0 conn(0x7fcc4c0efe00 legacy=0x7fcc4c0f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:48.639 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.634 7fcc4b7fe700 1 -- 172.21.2.112:0/366905607 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc4c0e52e0 con 0x7fcc4c1b0470 2021-02-10T11:34:48.639 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.634 7fcc4b7fe700 1 -- 172.21.2.112:0/366905607 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7fcc34001530 con 0x7fcc4c1b0470 2021-02-10T11:34:48.639 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.634 7fcc4b7fe700 1 -- 172.21.2.112:0/366905607 >> v1:172.21.2.112:6789/0 conn(0x7fcc4c1b0470 legacy=0x7fcc4c0ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:48.639 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.634 7fcc4b7fe700 1 --2- 172.21.2.112:0/366905607 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcc38003a40 0x7fcc38005ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:48.639 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.634 7fcc4b7fe700 1 --2- 172.21.2.112:0/366905607 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcc38006770 0x7fcc38008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:48.639 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.634 7fcc4b7fe700 1 --2- 172.21.2.112:0/366905607 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcc38009590 0x7fcc3800ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:48.639 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.634 7fcc4b7fe700 1 -- 172.21.2.112:0/366905607 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc3800c2d0 con 0x7fcc38006770 2021-02-10T11:34:48.640 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.634 7fcc4b7fe700 1 -- 172.21.2.112:0/366905607 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc3800c4f0 con 0x7fcc38003a40 2021-02-10T11:34:48.640 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.634 7fcc4b7fe700 1 -- 172.21.2.112:0/366905607 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc3800c710 con 0x7fcc38009590 2021-02-10T11:34:48.640 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.634 7fcc4bfff700 1 --2- 172.21.2.112:0/366905607 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcc38006770 0x7fcc38008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:48.640 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.634 7fcc50b61700 1 --2- 172.21.2.112:0/366905607 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcc38003a40 0x7fcc38005ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:48.640 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.634 7fcc51362700 1 --2- 172.21.2.112:0/366905607 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcc38009590 0x7fcc3800ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:48.640 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.634 7fcc4bfff700 1 -- 172.21.2.112:0/366905607 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcc38009590 msgr2=0x7fcc3800ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:48.640 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.634 7fcc4bfff700 1 --2- 172.21.2.112:0/366905607 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcc38009590 0x7fcc3800ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:48.640 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.634 7fcc4bfff700 1 -- 172.21.2.112:0/366905607 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcc38003a40 msgr2=0x7fcc38005ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:48.640 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.634 7fcc4bfff700 1 --2- 172.21.2.112:0/366905607 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcc38003a40 0x7fcc38005ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:48.640 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.634 7fcc4bfff700 1 -- 172.21.2.112:0/366905607 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc3800c9b0 con 0x7fcc38006770 2021-02-10T11:34:48.640 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.634 7fcc51362700 1 --2- 172.21.2.112:0/366905607 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcc38009590 0x7fcc3800ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:34:48.640 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.634 7fcc4bfff700 1 --2- 172.21.2.112:0/366905607 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcc38006770 0x7fcc38008c90 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 rx=0x7fcc3c04aaa0 tx=0x7fcc3c04daa0).ready entity=mon.0 client_cookie=1ba1b42abbf0736d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:48.641 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.635 7fcc4b7fe700 1 -- 172.21.2.112:0/366905607 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcc3c0500c0 con 0x7fcc38006770 2021-02-10T11:34:48.641 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.635 7fcc4b7fe700 1 -- 172.21.2.112:0/366905607 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc3c050440 con 0x7fcc38006770 2021-02-10T11:34:48.641 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.635 7fcc4b7fe700 1 -- 172.21.2.112:0/366905607 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcc3c0009b0 con 0x7fcc38006770 2021-02-10T11:34:48.641 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.635 7fcc52d45700 1 -- 172.21.2.112:0/366905607 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcc38006770 msgr2=0x7fcc38008c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:48.641 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.635 7fcc52d45700 1 --2- 172.21.2.112:0/366905607 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcc38006770 0x7fcc38008c90 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 rx=0x7fcc3c04aaa0 tx=0x7fcc3c04daa0).stop 2021-02-10T11:34:48.641 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.635 7fcc51362700 1 -- 172.21.2.112:0/366905607 reap_dead start 2021-02-10T11:34:48.641 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.635 7fcc52d45700 1 -- 172.21.2.112:0/366905607 shutdown_connections 2021-02-10T11:34:48.641 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.635 7fcc52d45700 1 -- 172.21.2.112:0/366905607 >> 172.21.2.112:0/366905607 conn(0x7fcc4c0ce3e0 msgr2=0x7fcc4c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:48.641 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.635 7fcc52d45700 1 -- 172.21.2.112:0/366905607 shutdown_connections 2021-02-10T11:34:48.641 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.635 7fcc52d45700 1 -- 172.21.2.112:0/366905607 wait complete. 2021-02-10T11:34:48.641 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.650 7fcc52d45700 1 Processor -- start 2021-02-10T11:34:48.641 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.650 7fcc52d45700 1 -- start start 2021-02-10T11:34:48.643 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.650 7fcc52d45700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcc4c108ce0 0x7fcc4c09da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:48.643 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.650 7fcc52d45700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcc4c1090e0 0x7fcc4c09e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:48.643 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.650 7fcc52d45700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcc4c09eee0 0x7fcc4c09f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:48.643 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.650 7fcc52d45700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc4c1094e0 con 0x7fcc4c09eee0 2021-02-10T11:34:48.643 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.650 7fcc52d45700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc4c09e960 con 0x7fcc4c1090e0 2021-02-10T11:34:48.643 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.650 7fcc52d45700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc4c09e690 con 0x7fcc4c108ce0 2021-02-10T11:34:48.643 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.651 7fcc51362700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcc4c09eee0 0x7fcc4c09f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:48.644 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.651 7fcc51362700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcc4c09eee0 0x7fcc4c09f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:50722/0 (socket says 172.21.2.112:50722) 2021-02-10T11:34:48.644 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.651 7fcc51362700 1 -- 172.21.2.112:0/1796039400 learned_addr learned my addr 172.21.2.112:0/1796039400 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:34:48.644 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.651 7fcc4bfff700 1 --2- 172.21.2.112:0/1796039400 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcc4c1090e0 0x7fcc4c09e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:48.644 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.651 7fcc50b61700 1 --2- 172.21.2.112:0/1796039400 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcc4c108ce0 0x7fcc4c09da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:48.644 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.651 7fcc51362700 1 -- 172.21.2.112:0/1796039400 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcc4c108ce0 msgr2=0x7fcc4c09da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:48.644 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.651 7fcc51362700 1 --2- 172.21.2.112:0/1796039400 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcc4c108ce0 0x7fcc4c09da50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:48.644 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.651 7fcc51362700 1 -- 172.21.2.112:0/1796039400 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcc4c1090e0 msgr2=0x7fcc4c09e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:48.644 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.651 7fcc51362700 1 --2- 172.21.2.112:0/1796039400 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcc4c1090e0 0x7fcc4c09e070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:48.644 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.651 7fcc51362700 1 -- 172.21.2.112:0/1796039400 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc4c09f900 con 0x7fcc4c09eee0 2021-02-10T11:34:48.644 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.651 7fcc51362700 1 --2- 172.21.2.112:0/1796039400 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcc4c09eee0 0x7fcc4c09f2e0 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 rx=0x7fcc44009980 tx=0x7fcc4400bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:48.644 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.651 7fcc497fa700 1 -- 172.21.2.112:0/1796039400 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcc44006030 con 0x7fcc4c09eee0 2021-02-10T11:34:48.644 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.651 7fcc497fa700 1 -- 172.21.2.112:0/1796039400 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc44006370 con 0x7fcc4c09eee0 2021-02-10T11:34:48.644 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.651 7fcc52d45700 1 -- 172.21.2.112:0/1796039400 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc4c09f900 con 0x7fcc4c09eee0 2021-02-10T11:34:48.644 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.653 7fcc497fa700 1 -- 172.21.2.112:0/1796039400 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcc44005930 con 0x7fcc4c09eee0 2021-02-10T11:34:48.645 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.653 7fcc52d45700 1 -- 172.21.2.112:0/1796039400 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc4c09fe90 con 0x7fcc4c09eee0 2021-02-10T11:34:48.645 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.653 7fcc497fa700 1 -- 172.21.2.112:0/1796039400 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fcc44006550 con 0x7fcc4c09eee0 2021-02-10T11:34:48.645 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.653 7fcc497fa700 1 --2- 172.21.2.112:0/1796039400 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcc300186c0 0x7fcc3001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:48.645 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.653 7fcc497fa700 1 -- 172.21.2.112:0/1796039400 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fcc44032800 con 0x7fcc4c09eee0 2021-02-10T11:34:48.645 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.653 7fcc50b61700 1 --2- 172.21.2.112:0/1796039400 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcc300186c0 0x7fcc3001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:48.645 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.653 7fcc50b61700 1 --2- 172.21.2.112:0/1796039400 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcc300186c0 0x7fcc3001abc0 crc :-1 s=READY pgs=687 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:48.670 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.683 7fcc52d45700 1 -- 172.21.2.112:0/1796039400 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc4c0f0b50 con 0x7fcc4c09eee0 2021-02-10T11:34:48.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.685 7fcc497fa700 1 -- 172.21.2.112:0/1796039400 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fcc4402dff0 con 0x7fcc4c09eee0 2021-02-10T11:34:48.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.810 7fcc52d45700 1 -- 172.21.2.112:0/1796039400 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcc4c0f0b50 con 0x7fcc4c09eee0 2021-02-10T11:34:48.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.810 7fcc497fa700 1 -- 172.21.2.112:0/1796039400 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v64) v1 ==== 74+0+13996 (secure 0 0 0) 0x7fcc4402dff0 con 0x7fcc4c09eee0 2021-02-10T11:34:48.798 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:34:48.799 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":64,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:34:46.584716","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:34:43.550220","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":13,"pool_name":"cephfs_data","create_time":"2021-02-10 11:34:44.578113","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:34:48.800 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.812 7fcc52d45700 1 -- 172.21.2.112:0/1796039400 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcc300186c0 msgr2=0x7fcc3001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:48.800 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.812 7fcc52d45700 1 --2- 172.21.2.112:0/1796039400 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcc300186c0 0x7fcc3001abc0 crc :-1 s=READY pgs=687 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:48.800 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.812 7fcc52d45700 1 -- 172.21.2.112:0/1796039400 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcc4c09eee0 msgr2=0x7fcc4c09f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:48.800 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.812 7fcc52d45700 1 --2- 172.21.2.112:0/1796039400 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcc4c09eee0 0x7fcc4c09f2e0 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 rx=0x7fcc44009980 tx=0x7fcc4400bcd0).stop 2021-02-10T11:34:48.801 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.812 7fcc52d45700 1 -- 172.21.2.112:0/1796039400 shutdown_connections 2021-02-10T11:34:48.801 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.812 7fcc52d45700 1 --2- 172.21.2.112:0/1796039400 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcc300186c0 0x7fcc3001abc0 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:48.801 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.812 7fcc52d45700 1 --2- 172.21.2.112:0/1796039400 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcc4c1090e0 0x7fcc4c09e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:48.801 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.812 7fcc52d45700 1 --2- 172.21.2.112:0/1796039400 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcc4c09eee0 0x7fcc4c09f2e0 unknown :-1 s=CLOSED pgs=1400 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:48.801 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.812 7fcc52d45700 1 --2- 172.21.2.112:0/1796039400 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcc4c108ce0 0x7fcc4c09da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:48.801 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.812 7fcc52d45700 1 -- 172.21.2.112:0/1796039400 >> 172.21.2.112:0/1796039400 conn(0x7fcc4c0bc6c0 msgr2=0x7fcc4c0b1f70 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:48.801 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.812 7fcc52d45700 1 -- 172.21.2.112:0/1796039400 shutdown_connections 2021-02-10T11:34:48.801 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:48.812 7fcc52d45700 1 -- 172.21.2.112:0/1796039400 wait complete. 2021-02-10T11:34:48.809 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:34:49.252 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.243 7febbe938700 1 Processor -- start 2021-02-10T11:34:49.252 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.243 7febbe938700 1 -- start start 2021-02-10T11:34:49.252 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.244 7febbe938700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7febb80d1c60 0x7febb80cef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:49.252 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.244 7febbe938700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7febb80cf570 0x7febb80cf970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:49.252 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.244 7febbe938700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febb80fe1b0 con 0x7febb80d1860 2021-02-10T11:34:49.253 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.244 7febbe938700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febb81b0ca0 con 0x7febb80cf570 2021-02-10T11:34:49.253 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.244 7febbe938700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febb8109a50 con 0x7febb80d1c60 2021-02-10T11:34:49.253 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.244 7febb7fff700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7febb80d1860 0x7febb81b04b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:50190/0 (socket says 172.21.2.112:50190) 2021-02-10T11:34:49.253 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.244 7febb7fff700 1 -- 172.21.2.112:0/1152064887 learned_addr learned my addr 172.21.2.112:0/1152064887 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:34:49.253 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.244 7febb6ffd700 1 -- 172.21.2.112:0/1152064887 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2647070665 0 0) 0x7feba8000ce0 con 0x7febb80d1860 2021-02-10T11:34:49.254 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.245 7febb6ffd700 1 -- 172.21.2.112:0/1152064887 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb9c002fe0 con 0x7febb80d1860 2021-02-10T11:34:49.254 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.245 7febb77fe700 1 --2- 172.21.2.112:0/1152064887 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7febb80d1c60 0x7febb80cef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:49.254 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.245 7febbcf55700 1 --2- 172.21.2.112:0/1152064887 >> v2:172.21.2.120:3300/0 conn(0x7febb80cf570 0x7febb80cf970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:49.254 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.245 7febb6ffd700 1 -- 172.21.2.112:0/1152064887 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 34371112 0 0) 0x7feba80010e0 con 0x7febb80d1860 2021-02-10T11:34:49.254 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.245 7febb6ffd700 1 -- 172.21.2.112:0/1152064887 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7febb80d1c60 msgr2=0x7febb80cef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:49.254 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.245 7febb6ffd700 1 --2- 172.21.2.112:0/1152064887 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7febb80d1c60 0x7febb80cef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:49.255 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.245 7febb6ffd700 1 -- 172.21.2.112:0/1152064887 >> v2:172.21.2.120:3300/0 conn(0x7febb80cf570 msgr2=0x7febb80cf970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:49.255 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.245 7febb6ffd700 1 --2- 172.21.2.112:0/1152064887 >> v2:172.21.2.120:3300/0 conn(0x7febb80cf570 0x7febb80cf970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:49.255 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.245 7febb6ffd700 1 -- 172.21.2.112:0/1152064887 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febb8088070 con 0x7febb80d1860 2021-02-10T11:34:49.255 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.245 7febb6ffd700 1 -- 172.21.2.112:0/1152064887 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7feba8001660 con 0x7febb80d1860 2021-02-10T11:34:49.255 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.245 7febb6ffd700 1 -- 172.21.2.112:0/1152064887 >> v1:172.21.2.112:6789/0 conn(0x7febb80d1860 legacy=0x7febb81b04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:49.255 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.245 7febb6ffd700 1 --2- 172.21.2.112:0/1152064887 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7feb9c003a60 0x7feb9c005f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:49.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.245 7febb6ffd700 1 --2- 172.21.2.112:0/1152064887 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7feb9c04a760 0x7feb9c04cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:49.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.245 7febb6ffd700 1 --2- 172.21.2.112:0/1152064887 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7feb9c04d500 0x7feb9c04fa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:49.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.245 7febb6ffd700 1 -- 172.21.2.112:0/1152064887 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb9c0502d0 con 0x7feb9c04a760 2021-02-10T11:34:49.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.245 7febb6ffd700 1 -- 172.21.2.112:0/1152064887 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb9c0504f0 con 0x7feb9c04d500 2021-02-10T11:34:49.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.245 7febb6ffd700 1 -- 172.21.2.112:0/1152064887 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb9c050710 con 0x7feb9c003a60 2021-02-10T11:34:49.257 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.245 7febb77fe700 1 --2- 172.21.2.112:0/1152064887 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7feb9c04a760 0x7feb9c04cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:49.257 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.245 7febb7fff700 1 --2- 172.21.2.112:0/1152064887 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7feb9c003a60 0x7feb9c005f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:49.257 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.245 7febbcf55700 1 --2- 172.21.2.112:0/1152064887 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7feb9c04d500 0x7feb9c04fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:49.257 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.246 7febb77fe700 1 -- 172.21.2.112:0/1152064887 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7feb9c003a60 msgr2=0x7feb9c005f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:49.257 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.246 7febb77fe700 1 --2- 172.21.2.112:0/1152064887 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7feb9c003a60 0x7feb9c005f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:49.257 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.246 7febb77fe700 1 -- 172.21.2.112:0/1152064887 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7feb9c04d500 msgr2=0x7feb9c04fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:49.258 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.246 7febb77fe700 1 --2- 172.21.2.112:0/1152064887 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7feb9c04d500 0x7feb9c04fa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:49.258 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.246 7febb77fe700 1 -- 172.21.2.112:0/1152064887 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb9c0509b0 con 0x7feb9c04a760 2021-02-10T11:34:49.258 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.246 7febb77fe700 1 --2- 172.21.2.112:0/1152064887 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7feb9c04a760 0x7feb9c04cc60 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 rx=0x7feba4009310 tx=0x7feba4009a90).ready entity=mon.0 client_cookie=fbc209c291196fbf server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:49.258 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.246 7febb6ffd700 1 -- 172.21.2.112:0/1152064887 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feba400c0c0 con 0x7feb9c04a760 2021-02-10T11:34:49.258 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.246 7febb6ffd700 1 -- 172.21.2.112:0/1152064887 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feba4006400 con 0x7feb9c04a760 2021-02-10T11:34:49.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.246 7febb6ffd700 1 -- 172.21.2.112:0/1152064887 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feba400c400 con 0x7feb9c04a760 2021-02-10T11:34:49.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.246 7febbe938700 1 -- 172.21.2.112:0/1152064887 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7feb9c04a760 msgr2=0x7feb9c04cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:49.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.246 7febbe938700 1 --2- 172.21.2.112:0/1152064887 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7feb9c04a760 0x7feb9c04cc60 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 rx=0x7feba4009310 tx=0x7feba4009a90).stop 2021-02-10T11:34:49.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.246 7febbcf55700 1 -- 172.21.2.112:0/1152064887 reap_dead start 2021-02-10T11:34:49.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.247 7febbe938700 1 -- 172.21.2.112:0/1152064887 shutdown_connections 2021-02-10T11:34:49.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.247 7febbe938700 1 -- 172.21.2.112:0/1152064887 >> 172.21.2.112:0/1152064887 conn(0x7febb80ce410 msgr2=0x7febb80d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:49.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.247 7febbe938700 1 -- 172.21.2.112:0/1152064887 shutdown_connections 2021-02-10T11:34:49.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.247 7febbe938700 1 -- 172.21.2.112:0/1152064887 wait complete. 2021-02-10T11:34:49.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.261 7febbe938700 1 Processor -- start 2021-02-10T11:34:49.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.262 7febbe938700 1 -- start start 2021-02-10T11:34:49.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.262 7febbe938700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7febb80b5fb0 0x7febb80a81f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:49.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.262 7febbe938700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7febb80a8810 0x7febb80a2540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:49.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.262 7febbe938700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7febb80a8c10 0x7febb80a2b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:49.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.262 7febbe938700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febb80b63b0 con 0x7febb80a8810 2021-02-10T11:34:49.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.262 7febbe938700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febb80a9210 con 0x7febb80a8c10 2021-02-10T11:34:49.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.262 7febbe938700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febb80b2650 con 0x7febb80b5fb0 2021-02-10T11:34:49.262 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.262 7febb77fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7febb80a8810 0x7febb80a2540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:49.262 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.262 7febb77fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7febb80a8810 0x7febb80a2540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:50738/0 (socket says 172.21.2.112:50738) 2021-02-10T11:34:49.262 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.262 7febb77fe700 1 -- 172.21.2.112:0/1283309642 learned_addr learned my addr 172.21.2.112:0/1283309642 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:34:49.262 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.263 7febb7fff700 1 --2- 172.21.2.112:0/1283309642 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7febb80b5fb0 0x7febb80a81f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:49.262 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.263 7febb77fe700 1 -- 172.21.2.112:0/1283309642 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7febb80b5fb0 msgr2=0x7febb80a81f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:49.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.263 7febb77fe700 1 --2- 172.21.2.112:0/1283309642 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7febb80b5fb0 0x7febb80a81f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:49.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.263 7febb77fe700 1 -- 172.21.2.112:0/1283309642 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7febb80a8c10 msgr2=0x7febb80a2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:49.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.263 7febb77fe700 1 --2- 172.21.2.112:0/1283309642 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7febb80a8c10 0x7febb80a2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:49.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.263 7febb77fe700 1 -- 172.21.2.112:0/1283309642 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febb80a3180 con 0x7febb80a8810 2021-02-10T11:34:49.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.263 7febb7fff700 1 --2- 172.21.2.112:0/1283309642 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7febb80b5fb0 0x7febb80a81f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:34:49.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.263 7febb77fe700 1 --2- 172.21.2.112:0/1283309642 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7febb80a8810 0x7febb80a2540 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 rx=0x7feba400b610 tx=0x7feba400b9b0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:49.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.263 7febb4ff9700 1 -- 172.21.2.112:0/1283309642 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feba40093d0 con 0x7febb80a8810 2021-02-10T11:34:49.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.263 7febb4ff9700 1 -- 172.21.2.112:0/1283309642 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feba4000940 con 0x7febb80a8810 2021-02-10T11:34:49.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.263 7febb4ff9700 1 -- 172.21.2.112:0/1283309642 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feba4002220 con 0x7febb80a8810 2021-02-10T11:34:49.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.263 7febbe938700 1 -- 172.21.2.112:0/1283309642 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febb80a3180 con 0x7febb80a8810 2021-02-10T11:34:49.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.263 7febbe938700 1 -- 172.21.2.112:0/1283309642 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febb80a3710 con 0x7febb80a8810 2021-02-10T11:34:49.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.264 7febb4ff9700 1 -- 172.21.2.112:0/1283309642 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7feba4000c90 con 0x7febb80a8810 2021-02-10T11:34:49.265 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.264 7febb4ff9700 1 --2- 172.21.2.112:0/1283309642 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7feb940186c0 0x7feb9401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:49.265 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.264 7febb4ff9700 1 -- 172.21.2.112:0/1283309642 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5573+0+0 (secure 0 0 0) 0x7feba40323f0 con 0x7febb80a8810 2021-02-10T11:34:49.265 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.264 7febb7fff700 1 --2- 172.21.2.112:0/1283309642 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7feb940186c0 0x7feb9401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:49.265 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.265 7febb7fff700 1 --2- 172.21.2.112:0/1283309642 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7feb940186c0 0x7feb9401abc0 crc :-1 s=READY pgs=688 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:49.281 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.294 7febbe938700 1 -- 172.21.2.112:0/1283309642 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febb80fe1b0 con 0x7febb80a8810 2021-02-10T11:34:49.283 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.297 7febb4ff9700 1 -- 172.21.2.112:0/1283309642 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7feba402c040 con 0x7febb80a8810 2021-02-10T11:34:49.412 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.425 7febbe938700 1 -- 172.21.2.112:0/1283309642 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7febb80b63b0 con 0x7febb80a8810 2021-02-10T11:34:49.416 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.429 7febb4ff9700 1 -- 172.21.2.112:0/1283309642 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v38) v1 ==== 184+0+0 (secure 0 0 0) 0x7feba402c040 con 0x7febb80a8810 2021-02-10T11:34:49.419 INFO:teuthology.orchestra.run.gibba012.stderr:updated caps for client.0 2021-02-10T11:34:49.420 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.434 7febbe938700 1 -- 172.21.2.112:0/1283309642 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7feb940186c0 msgr2=0x7feb9401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:49.421 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.434 7febbe938700 1 --2- 172.21.2.112:0/1283309642 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7feb940186c0 0x7feb9401abc0 crc :-1 s=READY pgs=688 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:49.421 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.434 7febbe938700 1 -- 172.21.2.112:0/1283309642 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7febb80a8810 msgr2=0x7febb80a2540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:49.421 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.434 7febbe938700 1 --2- 172.21.2.112:0/1283309642 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7febb80a8810 0x7febb80a2540 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 rx=0x7feba400b610 tx=0x7feba400b9b0).stop 2021-02-10T11:34:49.421 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.434 7febbe938700 1 -- 172.21.2.112:0/1283309642 shutdown_connections 2021-02-10T11:34:49.421 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.434 7febbe938700 1 --2- 172.21.2.112:0/1283309642 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7feb940186c0 0x7feb9401abc0 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:49.421 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.434 7febbe938700 1 --2- 172.21.2.112:0/1283309642 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7febb80a8c10 0x7febb80a2b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:49.421 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.434 7febbe938700 1 --2- 172.21.2.112:0/1283309642 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7febb80a8810 0x7febb80a2540 unknown :-1 s=CLOSED pgs=1403 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:49.421 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.434 7febbe938700 1 --2- 172.21.2.112:0/1283309642 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7febb80b5fb0 0x7febb80a81f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:49.421 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.434 7febbe938700 1 -- 172.21.2.112:0/1283309642 >> 172.21.2.112:0/1283309642 conn(0x7febb809e660 msgr2=0x7febb80a9550 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:49.422 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.434 7febbe938700 1 -- 172.21.2.112:0/1283309642 shutdown_connections 2021-02-10T11:34:49.422 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.434 7febbe938700 1 -- 172.21.2.112:0/1283309642 wait complete. 2021-02-10T11:34:49.429 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:34:49.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.866 7f51778ea700 1 Processor -- start 2021-02-10T11:34:49.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.866 7f51778ea700 1 -- start start 2021-02-10T11:34:49.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.866 7f51778ea700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f51700d1c80 0x7f51700cef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:49.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.866 7f51778ea700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51700d0180 con 0x7f51700d1880 2021-02-10T11:34:49.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.866 7f51778ea700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51700fe1d0 con 0x7f51700cf590 2021-02-10T11:34:49.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.866 7f51778ea700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51701b0cc0 con 0x7f51700d1c80 2021-02-10T11:34:49.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.867 7f5175706700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f51700d1880 0x7f51701b04d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:50208/0 (socket says 172.21.2.112:50208) 2021-02-10T11:34:49.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.867 7f5175706700 1 -- 172.21.2.112:0/2737894468 learned_addr learned my addr 172.21.2.112:0/2737894468 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:34:49.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.867 7f5163fff700 1 -- 172.21.2.112:0/2737894468 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2889958667 0 0) 0x7f5164000ce0 con 0x7f51700d1880 2021-02-10T11:34:49.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.867 7f5163fff700 1 -- 172.21.2.112:0/2737894468 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f515c002fe0 con 0x7f51700d1880 2021-02-10T11:34:49.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.867 7f5174f05700 1 --2- 172.21.2.112:0/2737894468 >> v2:172.21.2.120:3300/0 conn(0x7f51700d1c80 0x7f51700cef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:49.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.868 7f5163fff700 1 -- 172.21.2.112:0/2737894468 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 364921993 0 0) 0x7f5168000f90 con 0x7f51700cf590 2021-02-10T11:34:49.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.868 7f5163fff700 1 -- 172.21.2.112:0/2737894468 --> v1:172.21.2.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f515c003210 con 0x7f51700cf590 2021-02-10T11:34:49.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.868 7f5163fff700 1 -- 172.21.2.112:0/2737894468 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 397860022 0 0) 0x7f51640010e0 con 0x7f51700d1880 2021-02-10T11:34:49.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.868 7f5163fff700 1 -- 172.21.2.112:0/2737894468 >> v2:172.21.2.120:3300/0 conn(0x7f51700d1c80 msgr2=0x7f51700cef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:49.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.868 7f5163fff700 1 --2- 172.21.2.112:0/2737894468 >> v2:172.21.2.120:3300/0 conn(0x7f51700d1c80 0x7f51700cef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:49.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.868 7f5163fff700 1 -- 172.21.2.112:0/2737894468 >> v1:172.21.2.120:6789/0 conn(0x7f51700cf590 legacy=0x7f51700cf990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:49.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.868 7f5163fff700 1 -- 172.21.2.112:0/2737894468 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5170109a70 con 0x7f51700d1880 2021-02-10T11:34:49.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.868 7f5163fff700 1 -- 172.21.2.112:0/2737894468 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f51640015e0 con 0x7f51700d1880 2021-02-10T11:34:49.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.868 7f5163fff700 1 -- 172.21.2.112:0/2737894468 >> v1:172.21.2.112:6789/0 conn(0x7f51700d1880 legacy=0x7f51701b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:49.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.868 7f5163fff700 1 --2- 172.21.2.112:0/2737894468 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f515c0038e0 0x7f515c003ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:49.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.868 7f5163fff700 1 --2- 172.21.2.112:0/2737894468 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f515c0067e0 0x7f515c008c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:49.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.868 7f5163fff700 1 --2- 172.21.2.112:0/2737894468 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f515c009580 0x7f515c00baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:49.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.868 7f5163fff700 1 -- 172.21.2.112:0/2737894468 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f515c00c2f0 con 0x7f515c0038e0 2021-02-10T11:34:49.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.868 7f5163fff700 1 -- 172.21.2.112:0/2737894468 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f515c0504e0 con 0x7f515c0067e0 2021-02-10T11:34:49.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.868 7f5163fff700 1 -- 172.21.2.112:0/2737894468 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f515c050700 con 0x7f515c009580 2021-02-10T11:34:49.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.868 7f5175706700 1 --2- 172.21.2.112:0/2737894468 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f515c0038e0 0x7f515c003ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:49.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.868 7f5174f05700 1 --2- 172.21.2.112:0/2737894468 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f515c0067e0 0x7f515c008c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:49.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.868 7f5175f07700 1 --2- 172.21.2.112:0/2737894468 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f515c009580 0x7f515c00baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:49.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.868 7f5175706700 1 -- 172.21.2.112:0/2737894468 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f515c009580 msgr2=0x7f515c00baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:49.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.868 7f5175706700 1 --2- 172.21.2.112:0/2737894468 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f515c009580 0x7f515c00baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:49.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.868 7f5175706700 1 -- 172.21.2.112:0/2737894468 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f515c0067e0 msgr2=0x7f515c008c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:49.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.868 7f5175706700 1 --2- 172.21.2.112:0/2737894468 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f515c0067e0 0x7f515c008c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:49.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.868 7f5175706700 1 -- 172.21.2.112:0/2737894468 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f515c0509a0 con 0x7f515c0038e0 2021-02-10T11:34:49.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.868 7f5175f07700 1 -- 172.21.2.112:0/2737894468 reap_dead start 2021-02-10T11:34:49.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.868 7f5175706700 1 --2- 172.21.2.112:0/2737894468 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f515c0038e0 0x7f515c003ce0 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 rx=0x7f5164001790 tx=0x7f5164007d00).ready entity=mon.0 client_cookie=43034fa610b84352 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:49.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.869 7f5163fff700 1 -- 172.21.2.112:0/2737894468 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f51640013d0 con 0x7f515c0038e0 2021-02-10T11:34:49.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.869 7f5163fff700 1 -- 172.21.2.112:0/2737894468 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51640070c0 con 0x7f515c0038e0 2021-02-10T11:34:49.875 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.869 7f5163fff700 1 -- 172.21.2.112:0/2737894468 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f516400a7a0 con 0x7f515c0038e0 2021-02-10T11:34:49.875 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.869 7f51778ea700 1 -- 172.21.2.112:0/2737894468 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f515c0038e0 msgr2=0x7f515c003ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:49.875 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.869 7f51778ea700 1 --2- 172.21.2.112:0/2737894468 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f515c0038e0 0x7f515c003ce0 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 rx=0x7f5164001790 tx=0x7f5164007d00).stop 2021-02-10T11:34:49.875 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.870 7f51778ea700 1 -- 172.21.2.112:0/2737894468 shutdown_connections 2021-02-10T11:34:49.875 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.870 7f51778ea700 1 --2- 172.21.2.112:0/2737894468 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f515c0038e0 0x7f515c003ce0 unknown :-1 s=CLOSED pgs=1405 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:49.875 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.870 7f51778ea700 1 -- 172.21.2.112:0/2737894468 >> 172.21.2.112:0/2737894468 conn(0x7f51700ce430 msgr2=0x7f51700d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:49.875 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.870 7f51778ea700 1 -- 172.21.2.112:0/2737894468 shutdown_connections 2021-02-10T11:34:49.875 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.870 7f51778ea700 1 -- 172.21.2.112:0/2737894468 wait complete. 2021-02-10T11:34:49.875 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.885 7f51778ea700 1 Processor -- start 2021-02-10T11:34:49.875 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.885 7f51778ea700 1 -- start start 2021-02-10T11:34:49.876 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.885 7f51778ea700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f51701a0580 0x7f5170086fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:49.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.885 7f51778ea700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f51700b66b0 0x7f51700b6ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:49.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.885 7f51778ea700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5170082bb0 0x7f5170082fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:49.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.885 7f51778ea700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51700875d0 con 0x7f51700b66b0 2021-02-10T11:34:49.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.885 7f51778ea700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51700835d0 con 0x7f5170082bb0 2021-02-10T11:34:49.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.885 7f51778ea700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51700837f0 con 0x7f51701a0580 2021-02-10T11:34:49.878 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.885 7f5174f05700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f51700b66b0 0x7f51700b6ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:49.878 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.885 7f5174f05700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f51700b66b0 0x7f51700b6ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:50758/0 (socket says 172.21.2.112:50758) 2021-02-10T11:34:49.878 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.885 7f5174f05700 1 -- 172.21.2.112:0/3101222013 learned_addr learned my addr 172.21.2.112:0/3101222013 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:34:49.878 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.885 7f5175706700 1 --2- 172.21.2.112:0/3101222013 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f51701a0580 0x7f5170086fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:49.878 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.886 7f5175f07700 1 --2- 172.21.2.112:0/3101222013 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5170082bb0 0x7f5170082fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:49.878 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.886 7f5174f05700 1 -- 172.21.2.112:0/3101222013 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f51701a0580 msgr2=0x7f5170086fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:49.878 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.886 7f5174f05700 1 --2- 172.21.2.112:0/3101222013 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f51701a0580 0x7f5170086fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:49.878 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.886 7f5174f05700 1 -- 172.21.2.112:0/3101222013 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5170082bb0 msgr2=0x7f5170082fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:49.878 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.886 7f5174f05700 1 --2- 172.21.2.112:0/3101222013 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5170082bb0 0x7f5170082fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:49.878 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.886 7f5174f05700 1 -- 172.21.2.112:0/3101222013 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5170083a10 con 0x7f51700b66b0 2021-02-10T11:34:49.879 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.886 7f5175f07700 1 --2- 172.21.2.112:0/3101222013 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5170082bb0 0x7f5170082fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:34:49.879 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.886 7f5174f05700 1 --2- 172.21.2.112:0/3101222013 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f51700b66b0 0x7f51700b6ab0 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 rx=0x7f516c006f60 tx=0x7f516c00b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:49.879 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.886 7f5161ffb700 1 -- 172.21.2.112:0/3101222013 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f516c00bd60 con 0x7f51700b66b0 2021-02-10T11:34:49.879 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.886 7f5161ffb700 1 -- 172.21.2.112:0/3101222013 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f516c000c90 con 0x7f51700b66b0 2021-02-10T11:34:49.879 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.886 7f5161ffb700 1 -- 172.21.2.112:0/3101222013 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f516c002220 con 0x7f51700b66b0 2021-02-10T11:34:49.879 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.886 7f51778ea700 1 -- 172.21.2.112:0/3101222013 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5170083a10 con 0x7f51700b66b0 2021-02-10T11:34:49.879 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.886 7f51778ea700 1 -- 172.21.2.112:0/3101222013 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5170083fa0 con 0x7f51700b66b0 2021-02-10T11:34:49.879 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.887 7f5161ffb700 1 -- 172.21.2.112:0/3101222013 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f516c00a000 con 0x7f51700b66b0 2021-02-10T11:34:49.880 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.887 7f5161ffb700 1 --2- 172.21.2.112:0/3101222013 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f51500186c0 0x7f515001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:49.880 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.887 7f5161ffb700 1 -- 172.21.2.112:0/3101222013 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f516c032590 con 0x7f51700b66b0 2021-02-10T11:34:49.880 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.887 7f5175706700 1 --2- 172.21.2.112:0/3101222013 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f51500186c0 0x7f515001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:49.880 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.887 7f5175706700 1 --2- 172.21.2.112:0/3101222013 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f51500186c0 0x7f515001abc0 crc :-1 s=READY pgs=689 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:49.903 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.917 7f51778ea700 1 -- 172.21.2.112:0/3101222013 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5170082980 con 0x7f51700b66b0 2021-02-10T11:34:49.906 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:49.919 7f5161ffb700 1 -- 172.21.2.112:0/3101222013 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f516c02def0 con 0x7f51700b66b0 2021-02-10T11:34:50.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:50.052 7f51778ea700 1 -- 172.21.2.112:0/3101222013 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f5170082980 con 0x7f51700b66b0 2021-02-10T11:34:50.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:50.056 7f5161ffb700 1 -- 172.21.2.112:0/3101222013 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v39) v1 ==== 184+0+0 (secure 0 0 0) 0x7f516c00a000 con 0x7f51700b66b0 2021-02-10T11:34:50.046 INFO:teuthology.orchestra.run.gibba012.stderr:updated caps for client.1 2021-02-10T11:34:50.048 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:50.061 7f51778ea700 1 -- 172.21.2.112:0/3101222013 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f51500186c0 msgr2=0x7f515001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:50.048 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:50.061 7f51778ea700 1 --2- 172.21.2.112:0/3101222013 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f51500186c0 0x7f515001abc0 crc :-1 s=READY pgs=689 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:50.048 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:50.061 7f51778ea700 1 -- 172.21.2.112:0/3101222013 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f51700b66b0 msgr2=0x7f51700b6ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:50.048 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:50.061 7f51778ea700 1 --2- 172.21.2.112:0/3101222013 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f51700b66b0 0x7f51700b6ab0 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 rx=0x7f516c006f60 tx=0x7f516c00b970).stop 2021-02-10T11:34:50.048 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:50.061 7f51778ea700 1 -- 172.21.2.112:0/3101222013 shutdown_connections 2021-02-10T11:34:50.048 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:50.061 7f51778ea700 1 --2- 172.21.2.112:0/3101222013 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f51500186c0 0x7f515001abc0 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:50.048 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:50.061 7f51778ea700 1 --2- 172.21.2.112:0/3101222013 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5170082bb0 0x7f5170082fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:50.048 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:50.061 7f51778ea700 1 --2- 172.21.2.112:0/3101222013 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f51700b66b0 0x7f51700b6ab0 unknown :-1 s=CLOSED pgs=1406 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:50.048 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:50.061 7f51778ea700 1 --2- 172.21.2.112:0/3101222013 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f51701a0580 0x7f5170086fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:50.048 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:50.061 7f51778ea700 1 -- 172.21.2.112:0/3101222013 >> 172.21.2.112:0/3101222013 conn(0x7f51700afc60 msgr2=0x7f51700b50d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:50.049 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:50.061 7f51778ea700 1 -- 172.21.2.112:0/3101222013 shutdown_connections 2021-02-10T11:34:50.049 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:50.061 7f51778ea700 1 -- 172.21.2.112:0/3101222013 wait complete. 2021-02-10T11:34:50.056 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.2 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:34:50.486 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:50.483 7f4465ffb700 1 Processor -- start 2021-02-10T11:34:50.487 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:50.483 7f4465ffb700 1 -- start start 2021-02-10T11:34:50.487 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:50.483 7f4465ffb700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f44600d1880 0x7f44601b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:50.487 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:50.483 7f4465ffb700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f44600cff80 0x7f4460109140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:50.487 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:50.483 7f4465ffb700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4460109a70 con 0x7f44601b0af0 2021-02-10T11:34:50.487 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:50.483 7f4465ffb700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4460109760 con 0x7f44600cff80 2021-02-10T11:34:50.487 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:50.483 7f4465ffb700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4460088070 con 0x7f44600d1880 2021-02-10T11:34:50.487 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:50.483 7f445effd700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f44601b0af0 0x7f4460108950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:50226/0 (socket says 172.21.2.112:50226) 2021-02-10T11:34:50.487 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:50.483 7f445effd700 1 -- 172.21.2.112:0/999569676 learned_addr learned my addr 172.21.2.112:0/999569676 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:34:50.487 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:50.483 7f445ffff700 1 --2- 172.21.2.112:0/999569676 >> v2:172.21.2.112:3300/0 conn(0x7f44600cff80 0x7f4460109140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:50.487 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:50.484 7f445f7fe700 1 --2- 172.21.2.112:0/999569676 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f44600d1880 0x7f44601b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:50.487 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:50.484 7f445e7fc700 1 -- 172.21.2.112:0/999569676 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1689999225 0 0) 0x7f4458001500 con 0x7f44601b0af0 2021-02-10T11:34:50.487 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:50.484 7f445e7fc700 1 -- 172.21.2.112:0/999569676 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4448002fe0 con 0x7f44601b0af0 2021-02-10T11:34:50.488 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:50.484 7f445ffff700 1 -- 172.21.2.112:0/999569676 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f44600d1880 msgr2=0x7f44601b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:50.488 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:50.484 7f445ffff700 1 --2- 172.21.2.112:0/999569676 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f44600d1880 0x7f44601b04d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:50.488 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:50.484 7f445ffff700 1 -- 172.21.2.112:0/999569676 >> v1:172.21.2.112:6789/0 conn(0x7f44601b0af0 legacy=0x7f4460108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:50.488 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:50.484 7f445ffff700 1 -- 172.21.2.112:0/999569676 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4460088260 con 0x7f44600cff80 2021-02-10T11:34:50.488 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:50.484 7f445f7fe700 1 --2- 172.21.2.112:0/999569676 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f44600d1880 0x7f44601b04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:34:50.488 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:50.484 7f445ffff700 1 --2- 172.21.2.112:0/999569676 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f44600cff80 0x7f4460109140 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 rx=0x7f4450006ab0 tx=0x7f4450009b40).ready entity=mon.0 client_cookie=d653886c32f86133 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:50.488 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:50.484 7f445e7fc700 1 -- 172.21.2.112:0/999569676 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f44500022e0 con 0x7f44600cff80 2021-02-10T11:34:50.488 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:50.484 7f445e7fc700 1 -- 172.21.2.112:0/999569676 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4450002560 con 0x7f44600cff80 2021-02-10T11:34:50.488 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:50.484 7f445e7fc700 1 -- 172.21.2.112:0/999569676 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f44500029f0 con 0x7f44600cff80 2021-02-10T11:34:50.488 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:50.484 7f4465ffb700 1 -- 172.21.2.112:0/999569676 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f44600cff80 msgr2=0x7f4460109140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:50.488 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:50.484 7f4465ffb700 1 --2- 172.21.2.112:0/999569676 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f44600cff80 0x7f4460109140 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 rx=0x7f4450006ab0 tx=0x7f4450009b40).stop 2021-02-10T11:34:50.488 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:50.485 7f4465ffb700 1 -- 172.21.2.112:0/999569676 shutdown_connections 2021-02-10T11:34:50.488 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:50.485 7f4465ffb700 1 --2- 172.21.2.112:0/999569676 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f44600cff80 0x7f4460109140 unknown :-1 s=CLOSED pgs=1408 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:50.489 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:50.485 7f4465ffb700 1 --2- 172.21.2.112:0/999569676 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f44600d1880 0x7f44601b04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:50.489 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:50.485 7f4465ffb700 1 -- 172.21.2.112:0/999569676 >> 172.21.2.112:0/999569676 conn(0x7f44600ce430 msgr2=0x7f44600d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:50.489 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:50.485 7f4465ffb700 1 -- 172.21.2.112:0/999569676 shutdown_connections 2021-02-10T11:34:50.489 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:50.485 7f4465ffb700 1 -- 172.21.2.112:0/999569676 wait complete. 2021-02-10T11:34:50.489 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:50.500 7f4465ffb700 1 Processor -- start 2021-02-10T11:34:50.489 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:50.500 7f4465ffb700 1 -- start start 2021-02-10T11:34:50.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:50.500 7f4465ffb700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4460085300 0x7f44600bf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:50.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:50.500 7f4465ffb700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f44600bf7d0 0x7f4460085fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:50.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:50.500 7f4465ffb700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f44600bfbd0 0x7f44600865d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:50.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:50.500 7f4465ffb700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4460084d80 con 0x7f4460085300 2021-02-10T11:34:50.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:50.500 7f4465ffb700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4460085700 con 0x7f44600bf7d0 2021-02-10T11:34:50.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:50.500 7f4465ffb700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4460086bf0 con 0x7f44600bfbd0 2021-02-10T11:34:50.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:50.500 7f445f7fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4460085300 0x7f44600bf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:50.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:50.500 7f445f7fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4460085300 0x7f44600bf1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:50774/0 (socket says 172.21.2.112:50774) 2021-02-10T11:34:50.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:50.500 7f445f7fe700 1 -- 172.21.2.112:0/1099276409 learned_addr learned my addr 172.21.2.112:0/1099276409 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:34:50.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:50.501 7f445f7fe700 1 -- 172.21.2.112:0/1099276409 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f44600bfbd0 msgr2=0x7f44600865d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:50.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:50.501 7f445f7fe700 1 --2- 172.21.2.112:0/1099276409 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f44600bfbd0 0x7f44600865d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:50.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:50.501 7f445f7fe700 1 -- 172.21.2.112:0/1099276409 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f44600bf7d0 msgr2=0x7f4460085fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:50.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:50.501 7f445f7fe700 1 --2- 172.21.2.112:0/1099276409 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f44600bf7d0 0x7f4460085fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:50.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:50.501 7f445f7fe700 1 -- 172.21.2.112:0/1099276409 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4460086e10 con 0x7f4460085300 2021-02-10T11:34:50.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:50.501 7f445f7fe700 1 --2- 172.21.2.112:0/1099276409 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4460085300 0x7f44600bf1b0 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 rx=0x7f4454006f20 tx=0x7f445400b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:50.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:50.501 7f4447fff700 1 -- 172.21.2.112:0/1099276409 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f445400bd80 con 0x7f4460085300 2021-02-10T11:34:50.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:50.501 7f4447fff700 1 -- 172.21.2.112:0/1099276409 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4454000c90 con 0x7f4460085300 2021-02-10T11:34:50.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:50.501 7f4447fff700 1 -- 172.21.2.112:0/1099276409 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f44540022b0 con 0x7f4460085300 2021-02-10T11:34:50.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:50.501 7f4465ffb700 1 -- 172.21.2.112:0/1099276409 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4460086e10 con 0x7f4460085300 2021-02-10T11:34:50.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:50.501 7f4465ffb700 1 -- 172.21.2.112:0/1099276409 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4460086e10 con 0x7f4460085300 2021-02-10T11:34:50.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:50.502 7f4447fff700 1 -- 172.21.2.112:0/1099276409 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f445400a000 con 0x7f4460085300 2021-02-10T11:34:50.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:50.502 7f4447fff700 1 --2- 172.21.2.112:0/1099276409 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f44400186d0 0x7f444001abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:50.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:50.502 7f4447fff700 1 -- 172.21.2.112:0/1099276409 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f44540327b0 con 0x7f4460085300 2021-02-10T11:34:50.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:50.502 7f445effd700 1 --2- 172.21.2.112:0/1099276409 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f44400186d0 0x7f444001abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:50.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:50.503 7f445effd700 1 --2- 172.21.2.112:0/1099276409 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f44400186d0 0x7f444001abd0 crc :-1 s=READY pgs=690 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:50.520 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:50.533 7f4465ffb700 1 -- 172.21.2.112:0/1099276409 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44600829a0 con 0x7f4460085300 2021-02-10T11:34:50.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:50.536 7f4447fff700 1 -- 172.21.2.112:0/1099276409 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4454006bb0 con 0x7f4460085300 2021-02-10T11:34:50.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:50.666 7f4465ffb700 1 -- 172.21.2.112:0/1099276409 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f4460109a70 con 0x7f4460085300 2021-02-10T11:34:50.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:50.669 7f4447fff700 1 -- 172.21.2.112:0/1099276409 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.2 v40) v1 ==== 184+0+0 (secure 0 0 0) 0x7f4454000c90 con 0x7f4460085300 2021-02-10T11:34:50.656 INFO:teuthology.orchestra.run.gibba012.stderr:updated caps for client.2 2021-02-10T11:34:50.657 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:50.670 7f4465ffb700 1 -- 172.21.2.112:0/1099276409 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f44400186d0 msgr2=0x7f444001abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:50.657 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:50.670 7f4465ffb700 1 --2- 172.21.2.112:0/1099276409 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f44400186d0 0x7f444001abd0 crc :-1 s=READY pgs=690 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:50.657 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:50.670 7f4465ffb700 1 -- 172.21.2.112:0/1099276409 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4460085300 msgr2=0x7f44600bf1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:50.657 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:50.670 7f4465ffb700 1 --2- 172.21.2.112:0/1099276409 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4460085300 0x7f44600bf1b0 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 rx=0x7f4454006f20 tx=0x7f445400b990).stop 2021-02-10T11:34:50.658 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:50.671 7f4465ffb700 1 -- 172.21.2.112:0/1099276409 shutdown_connections 2021-02-10T11:34:50.658 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:50.671 7f4465ffb700 1 --2- 172.21.2.112:0/1099276409 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f44400186d0 0x7f444001abd0 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:50.658 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:50.671 7f4465ffb700 1 --2- 172.21.2.112:0/1099276409 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f44600bf7d0 0x7f4460085fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:50.658 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:50.671 7f4465ffb700 1 --2- 172.21.2.112:0/1099276409 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f44600bfbd0 0x7f44600865d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:50.658 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:50.671 7f4465ffb700 1 --2- 172.21.2.112:0/1099276409 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4460085300 0x7f44600bf1b0 unknown :-1 s=CLOSED pgs=1409 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:50.658 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:50.671 7f4465ffb700 1 -- 172.21.2.112:0/1099276409 >> 172.21.2.112:0/1099276409 conn(0x7f44600bd7a0 msgr2=0x7f44600b5900 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:50.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:50.671 7f4465ffb700 1 -- 172.21.2.112:0/1099276409 shutdown_connections 2021-02-10T11:34:50.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:50.671 7f4465ffb700 1 -- 172.21.2.112:0/1099276409 wait complete. 2021-02-10T11:34:50.667 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.3 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:34:51.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.101 7f57c1f72700 1 Processor -- start 2021-02-10T11:34:51.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.101 7f57c1f72700 1 -- start start 2021-02-10T11:34:51.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.102 7f57c1f72700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f57bc0d1880 0x7f57bc1b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:51.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.102 7f57c1f72700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f57bc0cff80 0x7f57bc109140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:51.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.102 7f57c1f72700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57bc109a70 con 0x7f57bc1b0af0 2021-02-10T11:34:51.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.102 7f57c1f72700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57bc109760 con 0x7f57bc0cff80 2021-02-10T11:34:51.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.102 7f57c1f72700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57bc088070 con 0x7f57bc0d1880 2021-02-10T11:34:51.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.102 7f57baffd700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f57bc1b0af0 0x7f57bc108950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:50240/0 (socket says 172.21.2.112:50240) 2021-02-10T11:34:51.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.102 7f57baffd700 1 -- 172.21.2.112:0/2772340187 learned_addr learned my addr 172.21.2.112:0/2772340187 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:34:51.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.102 7f57bbfff700 1 --2- 172.21.2.112:0/2772340187 >> v2:172.21.2.120:3300/0 conn(0x7f57bc0cff80 0x7f57bc109140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:51.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.102 7f57bb7fe700 1 --2- 172.21.2.112:0/2772340187 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f57bc0d1880 0x7f57bc1b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:51.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.102 7f57ba7fc700 1 -- 172.21.2.112:0/2772340187 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 310860963 0 0) 0x7f57a40010b0 con 0x7f57bc1b0af0 2021-02-10T11:34:51.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.102 7f57ba7fc700 1 -- 172.21.2.112:0/2772340187 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57a8002fe0 con 0x7f57bc1b0af0 2021-02-10T11:34:51.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.103 7f57bbfff700 1 -- 172.21.2.112:0/2772340187 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f57bc0d1880 msgr2=0x7f57bc1b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:51.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.103 7f57bbfff700 1 --2- 172.21.2.112:0/2772340187 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f57bc0d1880 0x7f57bc1b04d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:51.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.103 7f57bbfff700 1 -- 172.21.2.112:0/2772340187 >> v1:172.21.2.112:6789/0 conn(0x7f57bc1b0af0 legacy=0x7f57bc108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:51.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.103 7f57bbfff700 1 -- 172.21.2.112:0/2772340187 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57bc088260 con 0x7f57bc0cff80 2021-02-10T11:34:51.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.103 7f57bbfff700 1 --2- 172.21.2.112:0/2772340187 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f57bc0cff80 0x7f57bc109140 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 rx=0x7f57b4006e50 tx=0x7f57b4009e70).ready entity=mon.1 client_cookie=ca73e502d3b3da60 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:51.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.103 7f57ba7fc700 1 -- 172.21.2.112:0/2772340187 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f57b400b300 con 0x7f57bc0cff80 2021-02-10T11:34:51.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.103 7f57ba7fc700 1 -- 172.21.2.112:0/2772340187 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57b4003030 con 0x7f57bc0cff80 2021-02-10T11:34:51.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.103 7f57ba7fc700 1 -- 172.21.2.112:0/2772340187 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f57b400b770 con 0x7f57bc0cff80 2021-02-10T11:34:51.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.103 7f57c1f72700 1 -- 172.21.2.112:0/2772340187 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f57bc0cff80 msgr2=0x7f57bc109140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:51.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.103 7f57c1f72700 1 --2- 172.21.2.112:0/2772340187 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f57bc0cff80 0x7f57bc109140 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 rx=0x7f57b4006e50 tx=0x7f57b4009e70).stop 2021-02-10T11:34:51.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.104 7f57c1f72700 1 -- 172.21.2.112:0/2772340187 shutdown_connections 2021-02-10T11:34:51.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.104 7f57c1f72700 1 --2- 172.21.2.112:0/2772340187 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f57bc0cff80 0x7f57bc109140 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:51.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.104 7f57c1f72700 1 --2- 172.21.2.112:0/2772340187 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f57bc0d1880 0x7f57bc1b04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:51.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.104 7f57c1f72700 1 -- 172.21.2.112:0/2772340187 >> 172.21.2.112:0/2772340187 conn(0x7f57bc0ce430 msgr2=0x7f57bc0d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:51.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.104 7f57c1f72700 1 -- 172.21.2.112:0/2772340187 shutdown_connections 2021-02-10T11:34:51.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.104 7f57c1f72700 1 -- 172.21.2.112:0/2772340187 wait complete. 2021-02-10T11:34:51.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.119 7f57c1f72700 1 Processor -- start 2021-02-10T11:34:51.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.119 7f57c1f72700 1 -- start start 2021-02-10T11:34:51.109 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.119 7f57c1f72700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f57bc085300 0x7f57bc0bf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:51.109 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.119 7f57c1f72700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f57bc0bf7d0 0x7f57bc085fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:51.109 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.119 7f57c1f72700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f57bc0bfbd0 0x7f57bc0865d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:51.109 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.119 7f57c1f72700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57bc084d80 con 0x7f57bc0bfbd0 2021-02-10T11:34:51.110 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.119 7f57c1f72700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57bc085700 con 0x7f57bc0bf7d0 2021-02-10T11:34:51.110 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.119 7f57c1f72700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57bc086bf0 con 0x7f57bc085300 2021-02-10T11:34:51.110 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.119 7f57bbfff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f57bc0bfbd0 0x7f57bc0865d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:51.110 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.119 7f57bbfff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f57bc0bfbd0 0x7f57bc0865d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:50786/0 (socket says 172.21.2.112:50786) 2021-02-10T11:34:51.110 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.119 7f57bbfff700 1 -- 172.21.2.112:0/2550203608 learned_addr learned my addr 172.21.2.112:0/2550203608 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:34:51.110 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.120 7f57bbfff700 1 -- 172.21.2.112:0/2550203608 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f57bc085300 msgr2=0x7f57bc0bf1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:51.110 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.120 7f57bbfff700 1 --2- 172.21.2.112:0/2550203608 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f57bc085300 0x7f57bc0bf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:51.110 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.120 7f57bbfff700 1 -- 172.21.2.112:0/2550203608 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f57bc0bf7d0 msgr2=0x7f57bc085fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:51.110 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.120 7f57bbfff700 1 --2- 172.21.2.112:0/2550203608 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f57bc0bf7d0 0x7f57bc085fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:51.110 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.120 7f57bbfff700 1 -- 172.21.2.112:0/2550203608 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57bc086e10 con 0x7f57bc0bfbd0 2021-02-10T11:34:51.110 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.120 7f57bbfff700 1 --2- 172.21.2.112:0/2550203608 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f57bc0bfbd0 0x7f57bc0865d0 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 rx=0x7f57b40096f0 tx=0x7f57b400bdc0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:51.110 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.120 7f57b3fff700 1 -- 172.21.2.112:0/2550203608 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f57b40011a0 con 0x7f57bc0bfbd0 2021-02-10T11:34:51.111 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.120 7f57b3fff700 1 -- 172.21.2.112:0/2550203608 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57b4003030 con 0x7f57bc0bfbd0 2021-02-10T11:34:51.111 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.120 7f57b3fff700 1 -- 172.21.2.112:0/2550203608 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f57b40015f0 con 0x7f57bc0bfbd0 2021-02-10T11:34:51.111 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.120 7f57c1f72700 1 -- 172.21.2.112:0/2550203608 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57bc086e10 con 0x7f57bc0bfbd0 2021-02-10T11:34:51.111 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.120 7f57c1f72700 1 -- 172.21.2.112:0/2550203608 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57bc0873a0 con 0x7f57bc0bfbd0 2021-02-10T11:34:51.111 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.121 7f57b3fff700 1 -- 172.21.2.112:0/2550203608 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f57b4001180 con 0x7f57bc0bfbd0 2021-02-10T11:34:51.111 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.121 7f57b3fff700 1 --2- 172.21.2.112:0/2550203608 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f57a00186c0 0x7f57a001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:51.111 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.121 7f57b3fff700 1 -- 172.21.2.112:0/2550203608 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f57b4032920 con 0x7f57bc0bfbd0 2021-02-10T11:34:51.111 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.121 7f57bb7fe700 1 --2- 172.21.2.112:0/2550203608 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f57a00186c0 0x7f57a001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:51.111 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.122 7f57bb7fe700 1 --2- 172.21.2.112:0/2550203608 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f57a00186c0 0x7f57a001abc0 crc :-1 s=READY pgs=691 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:51.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.152 7f57c1f72700 1 -- 172.21.2.112:0/2550203608 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57bc0829a0 con 0x7f57bc0bfbd0 2021-02-10T11:34:51.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.154 7f57b3fff700 1 -- 172.21.2.112:0/2550203608 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f57b40018b0 con 0x7f57bc0bfbd0 2021-02-10T11:34:51.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.284 7f57c1f72700 1 -- 172.21.2.112:0/2550203608 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f57bc109a70 con 0x7f57bc0bfbd0 2021-02-10T11:34:51.274 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.288 7f57b3fff700 1 -- 172.21.2.112:0/2550203608 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.3 v41) v1 ==== 184+0+0 (secure 0 0 0) 0x7f57b4001180 con 0x7f57bc0bfbd0 2021-02-10T11:34:51.277 INFO:teuthology.orchestra.run.gibba012.stderr:updated caps for client.3 2021-02-10T11:34:51.279 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.292 7f57c1f72700 1 -- 172.21.2.112:0/2550203608 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f57a00186c0 msgr2=0x7f57a001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:51.279 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.292 7f57c1f72700 1 --2- 172.21.2.112:0/2550203608 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f57a00186c0 0x7f57a001abc0 crc :-1 s=READY pgs=691 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:51.279 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.292 7f57c1f72700 1 -- 172.21.2.112:0/2550203608 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f57bc0bfbd0 msgr2=0x7f57bc0865d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:51.279 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.292 7f57c1f72700 1 --2- 172.21.2.112:0/2550203608 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f57bc0bfbd0 0x7f57bc0865d0 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 rx=0x7f57b40096f0 tx=0x7f57b400bdc0).stop 2021-02-10T11:34:51.279 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.292 7f57c1f72700 1 -- 172.21.2.112:0/2550203608 shutdown_connections 2021-02-10T11:34:51.279 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.292 7f57c1f72700 1 --2- 172.21.2.112:0/2550203608 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f57a00186c0 0x7f57a001abc0 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:51.279 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.292 7f57c1f72700 1 --2- 172.21.2.112:0/2550203608 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f57bc0bf7d0 0x7f57bc085fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:51.279 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.292 7f57c1f72700 1 --2- 172.21.2.112:0/2550203608 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f57bc0bfbd0 0x7f57bc0865d0 unknown :-1 s=CLOSED pgs=1411 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:51.280 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.292 7f57c1f72700 1 --2- 172.21.2.112:0/2550203608 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f57bc085300 0x7f57bc0bf1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:51.280 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.292 7f57c1f72700 1 -- 172.21.2.112:0/2550203608 >> 172.21.2.112:0/2550203608 conn(0x7f57bc0bd7a0 msgr2=0x7f57bc0b5900 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:51.280 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.293 7f57c1f72700 1 -- 172.21.2.112:0/2550203608 shutdown_connections 2021-02-10T11:34:51.280 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.293 7f57c1f72700 1 -- 172.21.2.112:0/2550203608 wait complete. 2021-02-10T11:34:51.288 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:34:51.714 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.711 7f926a76c700 1 Processor -- start 2021-02-10T11:34:51.714 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.711 7f926a76c700 1 -- start start 2021-02-10T11:34:51.714 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.711 7f926a76c700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f92640d0340 0x7f92640ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:51.714 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.711 7f926a76c700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92640f0b10 con 0x7f92640d1c70 2021-02-10T11:34:51.714 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.711 7f926a76c700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92641b0be0 con 0x7f92640d1870 2021-02-10T11:34:51.714 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.711 7f926a76c700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92640e5ed0 con 0x7f92640d0340 2021-02-10T11:34:51.714 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.712 7f9268d89700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f92640d1c70 0x7f92640efc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:50254/0 (socket says 172.21.2.112:50254) 2021-02-10T11:34:51.714 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.712 7f9268d89700 1 -- 172.21.2.112:0/3776204704 learned_addr learned my addr 172.21.2.112:0/3776204704 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:34:51.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.712 7f9263fff700 1 --1- 172.21.2.112:0/3776204704 >> v1:172.21.2.120:6789/0 conn(0x7f92640d1870 0x7f92641b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:43810/0 (socket says 172.21.2.112:43810) 2021-02-10T11:34:51.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.712 7f92637fe700 1 --2- 172.21.2.112:0/3776204704 >> v2:172.21.2.120:3300/0 conn(0x7f92640d0340 0x7f92640ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:51.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.712 7f92637fe700 1 -- 172.21.2.112:0/3776204704 >> v1:172.21.2.120:6789/0 conn(0x7f92640d1870 legacy=0x7f92641b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:51.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.712 7f92637fe700 1 -- 172.21.2.112:0/3776204704 >> v1:172.21.2.112:6789/0 conn(0x7f92640d1c70 legacy=0x7f92640efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:51.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.712 7f92637fe700 1 -- 172.21.2.112:0/3776204704 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92640f0450 con 0x7f92640d0340 2021-02-10T11:34:51.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.712 7f92637fe700 1 --2- 172.21.2.112:0/3776204704 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f92640d0340 0x7f92640ef640 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 rx=0x7f9250006650 tx=0x7f9250009a00).ready entity=mon.1 client_cookie=48784f0e833c96a0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:51.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.712 7f9262ffd700 1 -- 172.21.2.112:0/3776204704 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f92500009b0 con 0x7f92640d0340 2021-02-10T11:34:51.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.712 7f9262ffd700 1 -- 172.21.2.112:0/3776204704 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f925000c0c0 con 0x7f92640d0340 2021-02-10T11:34:51.716 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.712 7f9262ffd700 1 -- 172.21.2.112:0/3776204704 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f925000c5a0 con 0x7f92640d0340 2021-02-10T11:34:51.716 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.712 7f926a76c700 1 -- 172.21.2.112:0/3776204704 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f92640d0340 msgr2=0x7f92640ef640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:51.716 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.712 7f926a76c700 1 --2- 172.21.2.112:0/3776204704 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f92640d0340 0x7f92640ef640 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 rx=0x7f9250006650 tx=0x7f9250009a00).stop 2021-02-10T11:34:51.716 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.713 7f926a76c700 1 -- 172.21.2.112:0/3776204704 shutdown_connections 2021-02-10T11:34:51.716 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.713 7f926a76c700 1 --2- 172.21.2.112:0/3776204704 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f92640d0340 0x7f92640ef640 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:51.716 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.713 7f926a76c700 1 -- 172.21.2.112:0/3776204704 >> 172.21.2.112:0/3776204704 conn(0x7f92640ce450 msgr2=0x7f92640d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:51.716 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.713 7f926a76c700 1 -- 172.21.2.112:0/3776204704 shutdown_connections 2021-02-10T11:34:51.716 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.713 7f926a76c700 1 -- 172.21.2.112:0/3776204704 wait complete. 2021-02-10T11:34:51.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.727 7f926a76c700 1 Processor -- start 2021-02-10T11:34:51.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.728 7f926a76c700 1 -- start start 2021-02-10T11:34:51.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.728 7f926a76c700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9264103520 0x7f92640c3640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:51.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.728 7f926a76c700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9264103920 0x7f92640b32d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:51.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.728 7f926a76c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f926408a450 0x7f926408a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:51.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.728 7f926a76c700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92640c3c60 con 0x7f926408a450 2021-02-10T11:34:51.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.728 7f926a76c700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92640b38f0 con 0x7f9264103920 2021-02-10T11:34:51.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.728 7f926a76c700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92640c0be0 con 0x7f9264103520 2021-02-10T11:34:51.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.728 7f9268d89700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f926408a450 0x7f926408a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:51.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.728 7f9268d89700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f926408a450 0x7f926408a850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:50798/0 (socket says 172.21.2.112:50798) 2021-02-10T11:34:51.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.728 7f9268d89700 1 -- 172.21.2.112:0/1844342111 learned_addr learned my addr 172.21.2.112:0/1844342111 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:34:51.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.728 7f9263fff700 1 --2- 172.21.2.112:0/1844342111 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9264103520 0x7f92640c3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:51.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.728 7f92637fe700 1 --2- 172.21.2.112:0/1844342111 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9264103920 0x7f92640b32d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:51.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.729 7f9268d89700 1 -- 172.21.2.112:0/1844342111 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9264103520 msgr2=0x7f92640c3640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:51.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.729 7f9268d89700 1 --2- 172.21.2.112:0/1844342111 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9264103520 0x7f92640c3640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:51.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.729 7f9268d89700 1 -- 172.21.2.112:0/1844342111 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9264103920 msgr2=0x7f92640b32d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:51.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.729 7f9268d89700 1 --2- 172.21.2.112:0/1844342111 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9264103920 0x7f92640b32d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:51.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.729 7f9268d89700 1 -- 172.21.2.112:0/1844342111 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f926408ae70 con 0x7f926408a450 2021-02-10T11:34:51.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.729 7f92637fe700 1 --2- 172.21.2.112:0/1844342111 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9264103920 0x7f92640b32d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:34:51.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.729 7f9268d89700 1 --2- 172.21.2.112:0/1844342111 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f926408a450 0x7f926408a850 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 rx=0x7f925c001bf0 tx=0x7f925c00ac50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:51.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.729 7f9260ff9700 1 -- 172.21.2.112:0/1844342111 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f925c0070d0 con 0x7f926408a450 2021-02-10T11:34:51.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.729 7f9260ff9700 1 -- 172.21.2.112:0/1844342111 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f925c007350 con 0x7f926408a450 2021-02-10T11:34:51.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.729 7f9260ff9700 1 -- 172.21.2.112:0/1844342111 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f925c004720 con 0x7f926408a450 2021-02-10T11:34:51.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.729 7f926a76c700 1 -- 172.21.2.112:0/1844342111 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f926408ae70 con 0x7f926408a450 2021-02-10T11:34:51.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.729 7f926a76c700 1 -- 172.21.2.112:0/1844342111 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f926408b400 con 0x7f926408a450 2021-02-10T11:34:51.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.730 7f9260ff9700 1 -- 172.21.2.112:0/1844342111 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f925c007530 con 0x7f926408a450 2021-02-10T11:34:51.721 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.730 7f9260ff9700 1 --2- 172.21.2.112:0/1844342111 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f92400186c0 0x7f924001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:51.721 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.730 7f9260ff9700 1 -- 172.21.2.112:0/1844342111 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f925c032880 con 0x7f926408a450 2021-02-10T11:34:51.721 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.730 7f9263fff700 1 --2- 172.21.2.112:0/1844342111 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f92400186c0 0x7f924001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:51.721 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.731 7f9263fff700 1 --2- 172.21.2.112:0/1844342111 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f92400186c0 0x7f924001abc0 crc :-1 s=READY pgs=692 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:51.748 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.761 7f926a76c700 1 -- 172.21.2.112:0/1844342111 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92640b38f0 con 0x7f926408a450 2021-02-10T11:34:51.750 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.764 7f9260ff9700 1 -- 172.21.2.112:0/1844342111 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f925c001500 con 0x7f926408a450 2021-02-10T11:34:51.879 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.892 7f926a76c700 1 -- 172.21.2.112:0/1844342111 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f92640b38f0 con 0x7f926408a450 2021-02-10T11:34:51.880 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.893 7f9260ff9700 1 -- 172.21.2.112:0/1844342111 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) v1 ==== 94+0+2101 (secure 0 0 0) 0x7f925c007530 con 0x7f926408a450 2021-02-10T11:34:51.880 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:34:51.880 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":47,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8334,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/3370618110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":3370618110},{"type":"v1","addr":"172.21.2.112:6833","nonce":3370618110}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":44}],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:34:45.582177","modified":"2021-02-10 11:34:47.589305","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5921},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5921":{"gid":5921,"name":"b","rank":0,"incarnation":45,"state":"up:active","state_seq":3,"addr":"172.21.2.120:6835/2436135065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":2436135065},{"type":"v1","addr":"172.21.2.120:6835","nonce":2436135065}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2021-02-10T11:34:51.880 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 47 2021-02-10T11:34:51.881 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.894 7f926a76c700 1 -- 172.21.2.112:0/1844342111 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f92400186c0 msgr2=0x7f924001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:51.881 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.894 7f926a76c700 1 --2- 172.21.2.112:0/1844342111 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f92400186c0 0x7f924001abc0 crc :-1 s=READY pgs=692 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:51.881 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.894 7f926a76c700 1 -- 172.21.2.112:0/1844342111 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f926408a450 msgr2=0x7f926408a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:51.881 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.894 7f926a76c700 1 --2- 172.21.2.112:0/1844342111 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f926408a450 0x7f926408a850 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 rx=0x7f925c001bf0 tx=0x7f925c00ac50).stop 2021-02-10T11:34:51.882 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.895 7f926a76c700 1 -- 172.21.2.112:0/1844342111 shutdown_connections 2021-02-10T11:34:51.882 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.895 7f926a76c700 1 --2- 172.21.2.112:0/1844342111 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f92400186c0 0x7f924001abc0 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:51.882 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.895 7f926a76c700 1 --2- 172.21.2.112:0/1844342111 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9264103920 0x7f92640b32d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:51.882 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.895 7f926a76c700 1 --2- 172.21.2.112:0/1844342111 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f926408a450 0x7f926408a850 unknown :-1 s=CLOSED pgs=1413 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:51.882 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.895 7f926a76c700 1 --2- 172.21.2.112:0/1844342111 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9264103520 0x7f92640c3640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:51.882 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.895 7f926a76c700 1 -- 172.21.2.112:0/1844342111 >> 172.21.2.112:0/1844342111 conn(0x7f926410a820 msgr2=0x7f9264108150 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:51.882 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.895 7f926a76c700 1 -- 172.21.2.112:0/1844342111 shutdown_connections 2021-02-10T11:34:51.882 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:51.895 7f926a76c700 1 -- 172.21.2.112:0/1844342111 wait complete. 2021-02-10T11:34:52.092 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 47, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:34:45.582177', 'modified': '2021-02-10 11:34:47.589305', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5921}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5921': {'gid': 5921, 'name': 'b', 'rank': 0, 'incarnation': 45, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.120:6835/2436135065', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.120:6834', 'nonce': 2436135065}, {'type': 'v1', 'addr': '172.21.2.120:6835', 'nonce': 2436135065}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:34:52.092 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:34:52.092 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:34:52.489 INFO:teuthology.orchestra.run.gibba020.stdout:{ 2021-02-10T11:34:52.489 INFO:teuthology.orchestra.run.gibba020.stdout: "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", 2021-02-10T11:34:52.489 INFO:teuthology.orchestra.run.gibba020.stdout: "whoami": 0, 2021-02-10T11:34:52.489 INFO:teuthology.orchestra.run.gibba020.stdout: "id": 5921, 2021-02-10T11:34:52.489 INFO:teuthology.orchestra.run.gibba020.stdout: "want_state": "up:active", 2021-02-10T11:34:52.489 INFO:teuthology.orchestra.run.gibba020.stdout: "state": "up:active", 2021-02-10T11:34:52.490 INFO:teuthology.orchestra.run.gibba020.stdout: "rank_uptime": 6.9174382239999996, 2021-02-10T11:34:52.490 INFO:teuthology.orchestra.run.gibba020.stdout: "mdsmap_epoch": 47, 2021-02-10T11:34:52.490 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch": 64, 2021-02-10T11:34:52.490 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch_barrier": 64, 2021-02-10T11:34:52.490 INFO:teuthology.orchestra.run.gibba020.stdout: "uptime": 12.561312797999999 2021-02-10T11:34:52.490 INFO:teuthology.orchestra.run.gibba020.stdout:} 2021-02-10T11:34:52.493 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", "whoami": 0, "id": 5921, "want_state": "up:active", "state": "up:active", "rank_uptime": 6.9174382239999996, "mdsmap_epoch": 47, "osdmap_epoch": 64, "osdmap_epoch_barrier": 64, "uptime": 12.561312797999999 } 2021-02-10T11:34:52.493 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:34:52.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:52.916 7f73e82e8700 1 Processor -- start 2021-02-10T11:34:52.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:52.916 7f73e82e8700 1 -- start start 2021-02-10T11:34:52.921 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:52.917 7f73e82e8700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f73e00d0340 0x7f73e00ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:52.921 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:52.917 7f73e82e8700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73e00f0b10 con 0x7f73e00d1c70 2021-02-10T11:34:52.921 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:52.917 7f73e82e8700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73e01b0be0 con 0x7f73e00d1870 2021-02-10T11:34:52.921 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:52.917 7f73e82e8700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73e00e5ed0 con 0x7f73e00d0340 2021-02-10T11:34:52.921 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:52.917 7f73e6905700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f73e00d1c70 0x7f73e00efc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:50264/0 (socket says 172.21.2.112:50264) 2021-02-10T11:34:52.921 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:52.917 7f73e6905700 1 -- 172.21.2.112:0/3964092673 learned_addr learned my addr 172.21.2.112:0/3964092673 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:34:52.921 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:52.917 7f73e5102700 1 -- 172.21.2.112:0/3964092673 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1635961151 0 0) 0x7f73d8001500 con 0x7f73e00d1c70 2021-02-10T11:34:52.921 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:52.917 7f73e5102700 1 -- 172.21.2.112:0/3964092673 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73cc002fe0 con 0x7f73e00d1c70 2021-02-10T11:34:52.922 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:52.917 7f73e5102700 1 -- 172.21.2.112:0/3964092673 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1340695783 0 0) 0x7f73d8001270 con 0x7f73e00d1c70 2021-02-10T11:34:52.922 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:52.917 7f73e5903700 1 --2- 172.21.2.112:0/3964092673 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f73e00d0340 0x7f73e00ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:52.922 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:52.917 7f73e5102700 1 -- 172.21.2.112:0/3964092673 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f73e00d0340 msgr2=0x7f73e00ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:52.922 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:52.917 7f73e5102700 1 --2- 172.21.2.112:0/3964092673 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f73e00d0340 0x7f73e00ef640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:52.922 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:52.917 7f73e5102700 1 -- 172.21.2.112:0/3964092673 >> v1:172.21.2.120:6789/0 conn(0x7f73e00d1870 legacy=0x7f73e01b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:52.922 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:52.917 7f73e5102700 1 -- 172.21.2.112:0/3964092673 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73e00f0450 con 0x7f73e00d1c70 2021-02-10T11:34:52.922 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:52.917 7f73e5102700 1 -- 172.21.2.112:0/3964092673 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f73d8001b50 con 0x7f73e00d1c70 2021-02-10T11:34:52.922 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:52.917 7f73e5903700 1 --2- 172.21.2.112:0/3964092673 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f73e00d0340 0x7f73e00ef640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:34:52.922 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:52.917 7f73e5102700 1 -- 172.21.2.112:0/3964092673 >> v1:172.21.2.112:6789/0 conn(0x7f73e00d1c70 legacy=0x7f73e00efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:52.922 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:52.918 7f73e5102700 1 --2- 172.21.2.112:0/3964092673 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f73cc003a60 0x7f73cc005eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:52.922 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:52.918 7f73e5102700 1 --2- 172.21.2.112:0/3964092673 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f73cc006750 0x7f73cc008c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:52.922 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:52.918 7f73e5102700 1 --2- 172.21.2.112:0/3964092673 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f73cc0094f0 0x7f73cc00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:52.922 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:52.918 7f73e5102700 1 -- 172.21.2.112:0/3964092673 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73cc00c2c0 con 0x7f73cc006750 2021-02-10T11:34:52.923 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:52.918 7f73e5102700 1 -- 172.21.2.112:0/3964092673 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73cc00c4e0 con 0x7f73cc0094f0 2021-02-10T11:34:52.923 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:52.918 7f73e5102700 1 -- 172.21.2.112:0/3964092673 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73cc00c700 con 0x7f73cc003a60 2021-02-10T11:34:52.923 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:52.918 7f73e5903700 1 --2- 172.21.2.112:0/3964092673 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f73cc006750 0x7f73cc008c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:52.923 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:52.918 7f73e6104700 1 --2- 172.21.2.112:0/3964092673 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f73cc003a60 0x7f73cc005eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:52.923 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:52.918 7f73e6905700 1 --2- 172.21.2.112:0/3964092673 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f73cc0094f0 0x7f73cc00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:52.923 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:52.918 7f73e5903700 1 -- 172.21.2.112:0/3964092673 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f73cc003a60 msgr2=0x7f73cc005eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:52.923 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:52.918 7f73e5903700 1 --2- 172.21.2.112:0/3964092673 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f73cc003a60 0x7f73cc005eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:52.923 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:52.918 7f73e5903700 1 -- 172.21.2.112:0/3964092673 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f73cc0094f0 msgr2=0x7f73cc00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:52.923 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:52.918 7f73e5903700 1 --2- 172.21.2.112:0/3964092673 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f73cc0094f0 0x7f73cc00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:52.923 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:52.918 7f73e5903700 1 -- 172.21.2.112:0/3964092673 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73cc00c9a0 con 0x7f73cc006750 2021-02-10T11:34:52.923 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:52.918 7f73e5903700 1 --2- 172.21.2.112:0/3964092673 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f73cc006750 0x7f73cc008c50 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 rx=0x7f73d4006670 tx=0x7f73d404db60).ready entity=mon.0 client_cookie=34ee6b5fcddde358 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:52.923 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:52.918 7f73e5102700 1 -- 172.21.2.112:0/3964092673 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f73d40066e0 con 0x7f73cc006750 2021-02-10T11:34:52.923 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:52.918 7f73e5102700 1 -- 172.21.2.112:0/3964092673 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73d40500c0 con 0x7f73cc006750 2021-02-10T11:34:52.924 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:52.918 7f73e5102700 1 -- 172.21.2.112:0/3964092673 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f73d40505e0 con 0x7f73cc006750 2021-02-10T11:34:52.924 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:52.918 7f73e82e8700 1 -- 172.21.2.112:0/3964092673 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f73cc006750 msgr2=0x7f73cc008c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:52.924 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:52.918 7f73e82e8700 1 --2- 172.21.2.112:0/3964092673 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f73cc006750 0x7f73cc008c50 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 rx=0x7f73d4006670 tx=0x7f73d404db60).stop 2021-02-10T11:34:52.924 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:52.918 7f73e6905700 1 -- 172.21.2.112:0/3964092673 reap_dead start 2021-02-10T11:34:52.924 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:52.919 7f73e82e8700 1 -- 172.21.2.112:0/3964092673 shutdown_connections 2021-02-10T11:34:52.924 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:52.919 7f73e82e8700 1 -- 172.21.2.112:0/3964092673 >> 172.21.2.112:0/3964092673 conn(0x7f73e00ce450 msgr2=0x7f73e00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:52.924 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:52.919 7f73e82e8700 1 -- 172.21.2.112:0/3964092673 shutdown_connections 2021-02-10T11:34:52.924 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:52.919 7f73e82e8700 1 -- 172.21.2.112:0/3964092673 wait complete. 2021-02-10T11:34:52.924 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:52.934 7f73e82e8700 1 Processor -- start 2021-02-10T11:34:52.924 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:52.934 7f73e82e8700 1 -- start start 2021-02-10T11:34:52.925 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:52.934 7f73e82e8700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f73e01a4a60 0x7f73e00c1c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:52.926 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:52.934 7f73e82e8700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f73e00c2290 0x7f73e00c2690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:52.926 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:52.934 7f73e82e8700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f73e00c2cb0 0x7f73e00bfd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:52.926 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:52.934 7f73e82e8700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73e00f05b0 con 0x7f73e01a4a60 2021-02-10T11:34:52.926 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:52.934 7f73e82e8700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73e00f07d0 con 0x7f73e00c2290 2021-02-10T11:34:52.926 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:52.934 7f73e82e8700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73e00c0340 con 0x7f73e00c2cb0 2021-02-10T11:34:52.926 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:52.934 7f73e6104700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f73e01a4a60 0x7f73e00c1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:52.926 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:52.934 7f73e6104700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f73e01a4a60 0x7f73e00c1c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:50812/0 (socket says 172.21.2.112:50812) 2021-02-10T11:34:52.926 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:52.934 7f73e6104700 1 -- 172.21.2.112:0/2847287642 learned_addr learned my addr 172.21.2.112:0/2847287642 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:34:52.926 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:52.935 7f73e6104700 1 -- 172.21.2.112:0/2847287642 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f73e00c2cb0 msgr2=0x7f73e00bfd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:52.926 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:52.935 7f73e6104700 1 --2- 172.21.2.112:0/2847287642 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f73e00c2cb0 0x7f73e00bfd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:52.926 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:52.935 7f73e6104700 1 -- 172.21.2.112:0/2847287642 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f73e00c2290 msgr2=0x7f73e00c2690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:52.927 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:52.935 7f73e6104700 1 --2- 172.21.2.112:0/2847287642 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f73e00c2290 0x7f73e00c2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:52.927 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:52.935 7f73e6104700 1 -- 172.21.2.112:0/2847287642 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73e00c0560 con 0x7f73e01a4a60 2021-02-10T11:34:52.927 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:52.935 7f73e6104700 1 --2- 172.21.2.112:0/2847287642 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f73e01a4a60 0x7f73e00c1c70 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 rx=0x7f73dc005a20 tx=0x7f73dc00a980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:52.927 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:52.935 7f73d2ffd700 1 -- 172.21.2.112:0/2847287642 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f73dc00ad70 con 0x7f73e01a4a60 2021-02-10T11:34:52.927 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:52.935 7f73d2ffd700 1 -- 172.21.2.112:0/2847287642 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73dc00b070 con 0x7f73e01a4a60 2021-02-10T11:34:52.927 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:52.935 7f73d2ffd700 1 -- 172.21.2.112:0/2847287642 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f73dc00b510 con 0x7f73e01a4a60 2021-02-10T11:34:52.927 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:52.935 7f73e82e8700 1 -- 172.21.2.112:0/2847287642 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73e00c0560 con 0x7f73e01a4a60 2021-02-10T11:34:52.927 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:52.935 7f73e82e8700 1 -- 172.21.2.112:0/2847287642 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73e00c0560 con 0x7f73e01a4a60 2021-02-10T11:34:52.927 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:52.936 7f73d2ffd700 1 -- 172.21.2.112:0/2847287642 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f73dc006740 con 0x7f73e01a4a60 2021-02-10T11:34:52.927 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:52.936 7f73d2ffd700 1 --2- 172.21.2.112:0/2847287642 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f73c80186c0 0x7f73c801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:52.927 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:52.936 7f73d2ffd700 1 -- 172.21.2.112:0/2847287642 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f73dc032400 con 0x7f73e01a4a60 2021-02-10T11:34:52.927 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:52.936 7f73e5903700 1 --2- 172.21.2.112:0/2847287642 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f73c80186c0 0x7f73c801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:52.927 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:52.937 7f73e5903700 1 --2- 172.21.2.112:0/2847287642 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f73c80186c0 0x7f73c801abc0 crc :-1 s=READY pgs=693 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:52.953 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:52.967 7f73e82e8700 1 -- 172.21.2.112:0/2847287642 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73e00f0af0 con 0x7f73e01a4a60 2021-02-10T11:34:52.956 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:52.969 7f73d2ffd700 1 -- 172.21.2.112:0/2847287642 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f73dc01eff0 con 0x7f73e01a4a60 2021-02-10T11:34:53.080 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:53.093 7f73e82e8700 1 -- 172.21.2.112:0/2847287642 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f73e00f0af0 con 0x7f73e01a4a60 2021-02-10T11:34:53.080 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:53.094 7f73d2ffd700 1 -- 172.21.2.112:0/2847287642 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v64) v1 ==== 74+0+13996 (secure 0 0 0) 0x7f73dc01eff0 con 0x7f73e01a4a60 2021-02-10T11:34:53.081 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:34:53.081 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":64,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:34:46.584716","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:34:43.550220","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":13,"pool_name":"cephfs_data","create_time":"2021-02-10 11:34:44.578113","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:34:53.082 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:53.095 7f73e82e8700 1 -- 172.21.2.112:0/2847287642 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f73c80186c0 msgr2=0x7f73c801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:53.082 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:53.095 7f73e82e8700 1 --2- 172.21.2.112:0/2847287642 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f73c80186c0 0x7f73c801abc0 crc :-1 s=READY pgs=693 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:53.083 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:53.095 7f73e82e8700 1 -- 172.21.2.112:0/2847287642 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f73e01a4a60 msgr2=0x7f73e00c1c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:53.083 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:53.095 7f73e82e8700 1 --2- 172.21.2.112:0/2847287642 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f73e01a4a60 0x7f73e00c1c70 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 rx=0x7f73dc005a20 tx=0x7f73dc00a980).stop 2021-02-10T11:34:53.083 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:53.095 7f73e82e8700 1 -- 172.21.2.112:0/2847287642 shutdown_connections 2021-02-10T11:34:53.083 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:53.095 7f73e82e8700 1 --2- 172.21.2.112:0/2847287642 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f73c80186c0 0x7f73c801abc0 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:53.083 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:53.095 7f73e82e8700 1 --2- 172.21.2.112:0/2847287642 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f73e00c2290 0x7f73e00c2690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:53.083 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:53.095 7f73e82e8700 1 --2- 172.21.2.112:0/2847287642 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f73e00c2cb0 0x7f73e00bfd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:53.083 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:53.095 7f73e82e8700 1 --2- 172.21.2.112:0/2847287642 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f73e01a4a60 0x7f73e00c1c70 unknown :-1 s=CLOSED pgs=1416 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:53.083 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:53.095 7f73e82e8700 1 -- 172.21.2.112:0/2847287642 >> 172.21.2.112:0/2847287642 conn(0x7f73e00ebe30 msgr2=0x7f73e00bf200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:53.083 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:53.095 7f73e82e8700 1 -- 172.21.2.112:0/2847287642 shutdown_connections 2021-02-10T11:34:53.083 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:53.095 7f73e82e8700 1 -- 172.21.2.112:0/2847287642 wait complete. 2021-02-10T11:34:53.091 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:13:57.508758', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 12, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:34:43.550220', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 13, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:34:44.578113', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '64', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:34:53.092 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:34:53.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:53.517 7f607bd80700 1 Processor -- start 2021-02-10T11:34:53.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:53.518 7f607bd80700 1 -- start start 2021-02-10T11:34:53.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:53.518 7f607bd80700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f60740d1c70 0x7f60740cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:53.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:53.518 7f607bd80700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60740d0170 con 0x7f60740d1870 2021-02-10T11:34:53.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:53.518 7f607bd80700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60740fe1c0 con 0x7f60740cf580 2021-02-10T11:34:53.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:53.518 7f607bd80700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60741b0cb0 con 0x7f60740d1c70 2021-02-10T11:34:53.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:53.518 7f6079b9c700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f60740d1870 0x7f60741b04c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:50282/0 (socket says 172.21.2.112:50282) 2021-02-10T11:34:53.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:53.518 7f6079b9c700 1 -- 172.21.2.112:0/2056295578 learned_addr learned my addr 172.21.2.112:0/2056295578 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:34:53.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:53.518 7f607939b700 1 --2- 172.21.2.112:0/2056295578 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f60740d1c70 0x7f60740cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:53.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:53.519 7f607939b700 1 -- 172.21.2.112:0/2056295578 >> v1:172.21.2.120:6789/0 conn(0x7f60740cf580 legacy=0x7f60740cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:53.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:53.519 7f607939b700 1 -- 172.21.2.112:0/2056295578 >> v1:172.21.2.112:6789/0 conn(0x7f60740d1870 legacy=0x7f60741b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:53.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:53.519 7f607939b700 1 -- 172.21.2.112:0/2056295578 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6074109a60 con 0x7f60740d1c70 2021-02-10T11:34:53.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:53.519 7f607939b700 1 --2- 172.21.2.112:0/2056295578 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f60740d1c70 0x7f60740cef60 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f6070006650 tx=0x7f6070009a80).ready entity=mon.2 client_cookie=2bb34f216356868 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:53.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:53.519 7f6078b9a700 1 -- 172.21.2.112:0/2056295578 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f60700009b0 con 0x7f60740d1c70 2021-02-10T11:34:53.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:53.519 7f6078b9a700 1 -- 172.21.2.112:0/2056295578 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f607000c0c0 con 0x7f60740d1c70 2021-02-10T11:34:53.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:53.519 7f6078b9a700 1 -- 172.21.2.112:0/2056295578 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f607000c5e0 con 0x7f60740d1c70 2021-02-10T11:34:53.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:53.519 7f607bd80700 1 -- 172.21.2.112:0/2056295578 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f60740d1c70 msgr2=0x7f60740cef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:53.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:53.519 7f607bd80700 1 --2- 172.21.2.112:0/2056295578 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f60740d1c70 0x7f60740cef60 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f6070006650 tx=0x7f6070009a80).stop 2021-02-10T11:34:53.524 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:53.520 7f607bd80700 1 -- 172.21.2.112:0/2056295578 shutdown_connections 2021-02-10T11:34:53.524 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:53.520 7f607bd80700 1 --2- 172.21.2.112:0/2056295578 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f60740d1c70 0x7f60740cef60 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:53.524 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:53.520 7f607bd80700 1 -- 172.21.2.112:0/2056295578 >> 172.21.2.112:0/2056295578 conn(0x7f60740ce420 msgr2=0x7f60740d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:53.524 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:53.520 7f607bd80700 1 -- 172.21.2.112:0/2056295578 shutdown_connections 2021-02-10T11:34:53.524 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:53.520 7f607bd80700 1 -- 172.21.2.112:0/2056295578 wait complete. 2021-02-10T11:34:53.524 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:53.535 7f607bd80700 1 Processor -- start 2021-02-10T11:34:53.524 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:53.535 7f607bd80700 1 -- start start 2021-02-10T11:34:53.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:53.535 7f607bd80700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f60740c0740 0x7f60741c5580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:53.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:53.535 7f607bd80700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f60741c5ba0 0x7f60740a5c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:53.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:53.535 7f607bd80700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f60740b6e60 0x7f60740a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:53.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:53.535 7f607bd80700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60740a6880 con 0x7f60741c5ba0 2021-02-10T11:34:53.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:53.535 7f607bd80700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60740a6aa0 con 0x7f60740b6e60 2021-02-10T11:34:53.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:53.535 7f607bd80700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60740a6cc0 con 0x7f60740c0740 2021-02-10T11:34:53.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:53.536 7f607939b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f60741c5ba0 0x7f60740a5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:53.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:53.536 7f607939b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f60741c5ba0 0x7f60740a5c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:50828/0 (socket says 172.21.2.112:50828) 2021-02-10T11:34:53.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:53.536 7f607939b700 1 -- 172.21.2.112:0/3073593566 learned_addr learned my addr 172.21.2.112:0/3073593566 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:34:53.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:53.536 7f6079b9c700 1 --2- 172.21.2.112:0/3073593566 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f60740c0740 0x7f60741c5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:53.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:53.536 7f607939b700 1 -- 172.21.2.112:0/3073593566 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f60740c0740 msgr2=0x7f60741c5580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:53.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:53.536 7f607939b700 1 --2- 172.21.2.112:0/3073593566 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f60740c0740 0x7f60741c5580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:53.528 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:53.536 7f607939b700 1 -- 172.21.2.112:0/3073593566 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f60740b6e60 msgr2=0x7f60740a6260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:53.528 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:53.536 7f607a39d700 1 --2- 172.21.2.112:0/3073593566 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f60740b6e60 0x7f60740a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:53.528 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:53.536 7f607939b700 1 --2- 172.21.2.112:0/3073593566 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f60740b6e60 0x7f60740a6260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:53.528 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:53.536 7f607939b700 1 -- 172.21.2.112:0/3073593566 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60740a6ee0 con 0x7f60741c5ba0 2021-02-10T11:34:53.528 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:53.536 7f6079b9c700 1 --2- 172.21.2.112:0/3073593566 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f60740c0740 0x7f60741c5580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:34:53.528 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:53.536 7f607a39d700 1 --2- 172.21.2.112:0/3073593566 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f60740b6e60 0x7f60740a6260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:34:53.528 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:53.536 7f607939b700 1 --2- 172.21.2.112:0/3073593566 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f60741c5ba0 0x7f60740a5c40 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 rx=0x7f6070009630 tx=0x7f607000b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:53.528 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:53.537 7f60667fc700 1 -- 172.21.2.112:0/3073593566 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f60700093a0 con 0x7f60741c5ba0 2021-02-10T11:34:53.528 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:53.537 7f60667fc700 1 -- 172.21.2.112:0/3073593566 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f607000bd70 con 0x7f60741c5ba0 2021-02-10T11:34:53.528 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:53.537 7f60667fc700 1 -- 172.21.2.112:0/3073593566 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f60700054d0 con 0x7f60741c5ba0 2021-02-10T11:34:53.528 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:53.537 7f607bd80700 1 -- 172.21.2.112:0/3073593566 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60740a6ee0 con 0x7f60741c5ba0 2021-02-10T11:34:53.528 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:53.537 7f607bd80700 1 -- 172.21.2.112:0/3073593566 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60740a6ee0 con 0x7f60741c5ba0 2021-02-10T11:34:53.528 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:53.537 7f60667fc700 1 -- 172.21.2.112:0/3073593566 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6070000c90 con 0x7f60741c5ba0 2021-02-10T11:34:53.529 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:53.537 7f60667fc700 1 --2- 172.21.2.112:0/3073593566 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f605c0186c0 0x7f605c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:53.529 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:53.537 7f60667fc700 1 -- 172.21.2.112:0/3073593566 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f60700325d0 con 0x7f60741c5ba0 2021-02-10T11:34:53.529 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:53.538 7f6079b9c700 1 --2- 172.21.2.112:0/3073593566 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f605c0186c0 0x7f605c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:53.529 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:53.538 7f6079b9c700 1 --2- 172.21.2.112:0/3073593566 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f605c0186c0 0x7f605c01abc0 crc :-1 s=READY pgs=694 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:53.555 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:53.568 7f607bd80700 1 -- 172.21.2.112:0/3073593566 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60740807d0 con 0x7f60741c5ba0 2021-02-10T11:34:53.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:53.571 7f60667fc700 1 -- 172.21.2.112:0/3073593566 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f607001f4f0 con 0x7f60741c5ba0 2021-02-10T11:34:53.685 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:53.698 7f607bd80700 1 -- 172.21.2.112:0/3073593566 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f6074088070 con 0x7f60741c5ba0 2021-02-10T11:34:53.686 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:53.699 7f60667fc700 1 -- 172.21.2.112:0/3073593566 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v64) v1 ==== 92+0+10 (secure 0 0 0) 0x7f607000bd70 con 0x7f60741c5ba0 2021-02-10T11:34:53.686 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:34:53.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:53.701 7f607bd80700 1 -- 172.21.2.112:0/3073593566 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f605c0186c0 msgr2=0x7f605c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:53.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:53.701 7f607bd80700 1 --2- 172.21.2.112:0/3073593566 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f605c0186c0 0x7f605c01abc0 crc :-1 s=READY pgs=694 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:53.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:53.701 7f607bd80700 1 -- 172.21.2.112:0/3073593566 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f60741c5ba0 msgr2=0x7f60740a5c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:53.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:53.701 7f607bd80700 1 --2- 172.21.2.112:0/3073593566 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f60741c5ba0 0x7f60740a5c40 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 rx=0x7f6070009630 tx=0x7f607000b980).stop 2021-02-10T11:34:53.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:53.701 7f607bd80700 1 -- 172.21.2.112:0/3073593566 shutdown_connections 2021-02-10T11:34:53.689 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:53.701 7f607bd80700 1 --2- 172.21.2.112:0/3073593566 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f605c0186c0 0x7f605c01abc0 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:53.689 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:53.701 7f607bd80700 1 --2- 172.21.2.112:0/3073593566 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f60740b6e60 0x7f60740a6260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:53.689 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:53.701 7f607bd80700 1 --2- 172.21.2.112:0/3073593566 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f60741c5ba0 0x7f60740a5c40 unknown :-1 s=CLOSED pgs=1418 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:53.689 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:53.701 7f607bd80700 1 --2- 172.21.2.112:0/3073593566 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f60740c0740 0x7f60741c5580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:53.689 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:53.701 7f607bd80700 1 -- 172.21.2.112:0/3073593566 >> 172.21.2.112:0/3073593566 conn(0x7f60740c4120 msgr2=0x7f60740ffd60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:53.690 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:53.701 7f607bd80700 1 -- 172.21.2.112:0/3073593566 shutdown_connections 2021-02-10T11:34:53.690 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:53.701 7f607bd80700 1 -- 172.21.2.112:0/3073593566 wait complete. 2021-02-10T11:34:53.696 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:34:54.126 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.123 7f34cf78a700 1 Processor -- start 2021-02-10T11:34:54.127 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.123 7f34cf78a700 1 -- start start 2021-02-10T11:34:54.127 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.123 7f34cf78a700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f34c80d1c70 0x7f34c80cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:54.127 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.123 7f34cf78a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f34c80cf580 0x7f34c80cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:54.127 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.123 7f34cf78a700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34c80fe1c0 con 0x7f34c80d1870 2021-02-10T11:34:54.127 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.123 7f34cf78a700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34c81b0cb0 con 0x7f34c80d1c70 2021-02-10T11:34:54.127 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.123 7f34cf78a700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34c8109a60 con 0x7f34c80cf580 2021-02-10T11:34:54.127 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.124 7f34ccda5700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f34c80d1c70 0x7f34c80cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:54.127 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.124 7f34cd5a6700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f34c80d1870 0x7f34c81b04c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:50296/0 (socket says 172.21.2.112:50296) 2021-02-10T11:34:54.127 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.124 7f34cdda7700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f34c80cf580 0x7f34c80cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:54.127 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.124 7f34cd5a6700 1 -- 172.21.2.112:0/3760244950 learned_addr learned my addr 172.21.2.112:0/3760244950 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:34:54.127 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.124 7f34ccda5700 1 -- 172.21.2.112:0/3760244950 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f34c80cf580 msgr2=0x7f34c80cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:54.127 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.124 7f34ccda5700 1 --2- 172.21.2.112:0/3760244950 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f34c80cf580 0x7f34c80cf980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:54.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.124 7f34ccda5700 1 -- 172.21.2.112:0/3760244950 >> v1:172.21.2.112:6789/0 conn(0x7f34c80d1870 legacy=0x7f34c81b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:54.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.124 7f34ccda5700 1 -- 172.21.2.112:0/3760244950 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34c8088070 con 0x7f34c80d1c70 2021-02-10T11:34:54.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.124 7f34ccda5700 1 --2- 172.21.2.112:0/3760244950 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f34c80d1c70 0x7f34c80cef60 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 rx=0x7f34c4006650 tx=0x7f34c4009a80).ready entity=mon.0 client_cookie=2ed00b54fdf078e3 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:54.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.125 7f34bbfff700 1 -- 172.21.2.112:0/3760244950 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f34c40009b0 con 0x7f34c80d1c70 2021-02-10T11:34:54.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.125 7f34bbfff700 1 -- 172.21.2.112:0/3760244950 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34c4006740 con 0x7f34c80d1c70 2021-02-10T11:34:54.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.125 7f34bbfff700 1 -- 172.21.2.112:0/3760244950 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f34c400c0c0 con 0x7f34c80d1c70 2021-02-10T11:34:54.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.125 7f34cf78a700 1 -- 172.21.2.112:0/3760244950 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f34c80d1c70 msgr2=0x7f34c80cef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:54.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.125 7f34cf78a700 1 --2- 172.21.2.112:0/3760244950 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f34c80d1c70 0x7f34c80cef60 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 rx=0x7f34c4006650 tx=0x7f34c4009a80).stop 2021-02-10T11:34:54.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.125 7f34cf78a700 1 -- 172.21.2.112:0/3760244950 shutdown_connections 2021-02-10T11:34:54.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.125 7f34cf78a700 1 --2- 172.21.2.112:0/3760244950 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f34c80cf580 0x7f34c80cf980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:54.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.125 7f34cf78a700 1 --2- 172.21.2.112:0/3760244950 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f34c80d1c70 0x7f34c80cef60 unknown :-1 s=CLOSED pgs=1420 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:54.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.125 7f34cf78a700 1 -- 172.21.2.112:0/3760244950 >> 172.21.2.112:0/3760244950 conn(0x7f34c80ce420 msgr2=0x7f34c80d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:54.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.126 7f34cf78a700 1 -- 172.21.2.112:0/3760244950 shutdown_connections 2021-02-10T11:34:54.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.126 7f34cf78a700 1 -- 172.21.2.112:0/3760244950 wait complete. 2021-02-10T11:34:54.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.140 7f34cf78a700 1 Processor -- start 2021-02-10T11:34:54.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.140 7f34cf78a700 1 -- start start 2021-02-10T11:34:54.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.140 7f34cf78a700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f34c8191b10 0x7f34c80ae640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:54.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.141 7f34cf78a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f34c8191f10 0x7f34c819ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:54.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.141 7f34cf78a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f34c81a05f0 0x7f34c8082ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:54.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.141 7f34cf78a700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34c8192440 con 0x7f34c8191f10 2021-02-10T11:34:54.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.141 7f34cf78a700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34c81916e0 con 0x7f34c8191b10 2021-02-10T11:34:54.131 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.141 7f34cf78a700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34c8191900 con 0x7f34c81a05f0 2021-02-10T11:34:54.131 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.141 7f34ccda5700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f34c8191f10 0x7f34c819ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:54.131 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.141 7f34ccda5700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f34c8191f10 0x7f34c819ffd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:50844/0 (socket says 172.21.2.112:50844) 2021-02-10T11:34:54.131 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.141 7f34ccda5700 1 -- 172.21.2.112:0/1905486840 learned_addr learned my addr 172.21.2.112:0/1905486840 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:34:54.131 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.141 7f34cd5a6700 1 --2- 172.21.2.112:0/1905486840 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f34c8191b10 0x7f34c80ae640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:54.131 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.141 7f34cdda7700 1 --2- 172.21.2.112:0/1905486840 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f34c81a05f0 0x7f34c8082ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:54.131 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.141 7f34ccda5700 1 -- 172.21.2.112:0/1905486840 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f34c81a05f0 msgr2=0x7f34c8082ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:54.131 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.141 7f34ccda5700 1 --2- 172.21.2.112:0/1905486840 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f34c81a05f0 0x7f34c8082ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:54.131 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.141 7f34ccda5700 1 -- 172.21.2.112:0/1905486840 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f34c8191b10 msgr2=0x7f34c80ae640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:54.131 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.141 7f34ccda5700 1 --2- 172.21.2.112:0/1905486840 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f34c8191b10 0x7f34c80ae640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:54.131 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.141 7f34ccda5700 1 -- 172.21.2.112:0/1905486840 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34c80834f0 con 0x7f34c8191f10 2021-02-10T11:34:54.131 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.141 7f34ccda5700 1 --2- 172.21.2.112:0/1905486840 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f34c8191f10 0x7f34c819ffd0 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 rx=0x7f34c4009650 tx=0x7f34c400b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:54.131 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.141 7f34b9ffb700 1 -- 172.21.2.112:0/1905486840 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f34c400bd50 con 0x7f34c8191f10 2021-02-10T11:34:54.132 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.142 7f34b9ffb700 1 -- 172.21.2.112:0/1905486840 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34c4006030 con 0x7f34c8191f10 2021-02-10T11:34:54.132 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.142 7f34b9ffb700 1 -- 172.21.2.112:0/1905486840 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f34c4005560 con 0x7f34c8191f10 2021-02-10T11:34:54.132 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.142 7f34cf78a700 1 -- 172.21.2.112:0/1905486840 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34c80834f0 con 0x7f34c8191f10 2021-02-10T11:34:54.132 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.142 7f34cf78a700 1 -- 172.21.2.112:0/1905486840 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34c8083a80 con 0x7f34c8191f10 2021-02-10T11:34:54.132 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.142 7f34b9ffb700 1 -- 172.21.2.112:0/1905486840 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f34c40052e0 con 0x7f34c8191f10 2021-02-10T11:34:54.132 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.142 7f34b9ffb700 1 --2- 172.21.2.112:0/1905486840 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f34a80186c0 0x7f34a801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:54.132 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.142 7f34b9ffb700 1 -- 172.21.2.112:0/1905486840 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f34c40323f0 con 0x7f34c8191f10 2021-02-10T11:34:54.132 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.143 7f34cd5a6700 1 --2- 172.21.2.112:0/1905486840 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f34a80186c0 0x7f34a801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:54.132 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.143 7f34cd5a6700 1 --2- 172.21.2.112:0/1905486840 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f34a80186c0 0x7f34a801abc0 crc :-1 s=READY pgs=695 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:54.160 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.173 7f34cf78a700 1 -- 172.21.2.112:0/1905486840 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34c80fe1c0 con 0x7f34c8191f10 2021-02-10T11:34:54.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.176 7f34b9ffb700 1 -- 172.21.2.112:0/1905486840 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f34c4009300 con 0x7f34c8191f10 2021-02-10T11:34:54.291 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.304 7f34cf78a700 1 -- 172.21.2.112:0/1905486840 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f34c80fe1c0 con 0x7f34c8191f10 2021-02-10T11:34:54.291 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.304 7f34b9ffb700 1 -- 172.21.2.112:0/1905486840 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v64) v1 ==== 104+0+10 (secure 0 0 0) 0x7f34c40052e0 con 0x7f34c8191f10 2021-02-10T11:34:54.292 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:34:54.293 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.306 7f34cf78a700 1 -- 172.21.2.112:0/1905486840 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f34a80186c0 msgr2=0x7f34a801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:54.293 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.306 7f34cf78a700 1 --2- 172.21.2.112:0/1905486840 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f34a80186c0 0x7f34a801abc0 crc :-1 s=READY pgs=695 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:54.293 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.306 7f34cf78a700 1 -- 172.21.2.112:0/1905486840 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f34c8191f10 msgr2=0x7f34c819ffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:54.293 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.306 7f34cf78a700 1 --2- 172.21.2.112:0/1905486840 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f34c8191f10 0x7f34c819ffd0 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 rx=0x7f34c4009650 tx=0x7f34c400b960).stop 2021-02-10T11:34:54.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.307 7f34cf78a700 1 -- 172.21.2.112:0/1905486840 shutdown_connections 2021-02-10T11:34:54.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.307 7f34cf78a700 1 --2- 172.21.2.112:0/1905486840 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f34a80186c0 0x7f34a801abc0 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:54.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.307 7f34cf78a700 1 --2- 172.21.2.112:0/1905486840 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f34c81a05f0 0x7f34c8082ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:54.295 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.307 7f34cf78a700 1 --2- 172.21.2.112:0/1905486840 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f34c8191f10 0x7f34c819ffd0 unknown :-1 s=CLOSED pgs=1421 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:54.295 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.307 7f34cf78a700 1 --2- 172.21.2.112:0/1905486840 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f34c8191b10 0x7f34c80ae640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:54.295 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.307 7f34cf78a700 1 -- 172.21.2.112:0/1905486840 >> 172.21.2.112:0/1905486840 conn(0x7f34c80a02d0 msgr2=0x7f34c809f200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:54.295 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.307 7f34cf78a700 1 -- 172.21.2.112:0/1905486840 shutdown_connections 2021-02-10T11:34:54.295 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.307 7f34cf78a700 1 -- 172.21.2.112:0/1905486840 wait complete. 2021-02-10T11:34:54.302 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:34:54.733 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.728 7f195c46a700 1 Processor -- start 2021-02-10T11:34:54.733 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.728 7f195c46a700 1 -- start start 2021-02-10T11:34:54.733 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.729 7f195c46a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f19540d1870 0x7f19541b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:54.733 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.729 7f195c46a700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1954109a60 con 0x7f19541b0ae0 2021-02-10T11:34:54.733 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.729 7f195c46a700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1954088070 con 0x7f19540cff70 2021-02-10T11:34:54.733 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.729 7f195c46a700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19540e5ef0 con 0x7f19540d1870 2021-02-10T11:34:54.733 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.729 7f1959a85700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f19541b0ae0 0x7f1954108940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:50310/0 (socket says 172.21.2.112:50310) 2021-02-10T11:34:54.733 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.729 7f1959a85700 1 -- 172.21.2.112:0/154724568 learned_addr learned my addr 172.21.2.112:0/154724568 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:34:54.733 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.729 7f1959284700 1 -- 172.21.2.112:0/154724568 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 287714579 0 0) 0x7f19480010b0 con 0x7f19541b0ae0 2021-02-10T11:34:54.734 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.729 7f1959284700 1 -- 172.21.2.112:0/154724568 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1940002fe0 con 0x7f19541b0ae0 2021-02-10T11:34:54.734 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.729 7f195a286700 1 --2- 172.21.2.112:0/154724568 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f19540d1870 0x7f19541b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:54.734 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.729 7f1959284700 1 -- 172.21.2.112:0/154724568 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1417333001 0 0) 0x7f1948001010 con 0x7f19541b0ae0 2021-02-10T11:34:54.734 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.730 7f1959284700 1 -- 172.21.2.112:0/154724568 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f19540d1870 msgr2=0x7f19541b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:54.734 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.730 7f1959284700 1 --2- 172.21.2.112:0/154724568 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f19540d1870 0x7f19541b04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:54.734 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.730 7f1959284700 1 -- 172.21.2.112:0/154724568 >> v1:172.21.2.120:6789/0 conn(0x7f19540cff70 legacy=0x7f1954109130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:54.734 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.730 7f1959284700 1 -- 172.21.2.112:0/154724568 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19540d1c70 con 0x7f19541b0ae0 2021-02-10T11:34:54.734 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.730 7f1959284700 1 -- 172.21.2.112:0/154724568 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f1948001530 con 0x7f19541b0ae0 2021-02-10T11:34:54.734 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.730 7f1959284700 1 -- 172.21.2.112:0/154724568 >> v1:172.21.2.112:6789/0 conn(0x7f19541b0ae0 legacy=0x7f1954108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:54.734 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.730 7f1959284700 1 --2- 172.21.2.112:0/154724568 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1940003ac0 0x7f1940005ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:54.734 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.730 7f1959284700 1 --2- 172.21.2.112:0/154724568 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1940006770 0x7f1940008c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:54.734 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.730 7f1959284700 1 --2- 172.21.2.112:0/154724568 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1940009510 0x7f194000ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:54.735 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.730 7f1959284700 1 -- 172.21.2.112:0/154724568 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f194000c2e0 con 0x7f1940006770 2021-02-10T11:34:54.735 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.730 7f1959284700 1 -- 172.21.2.112:0/154724568 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f194000c500 con 0x7f1940009510 2021-02-10T11:34:54.735 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.730 7f1959284700 1 -- 172.21.2.112:0/154724568 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f194000c720 con 0x7f1940003ac0 2021-02-10T11:34:54.735 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.730 7f195a286700 1 --2- 172.21.2.112:0/154724568 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1940003ac0 0x7f1940005ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:54.735 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.730 7f1959a85700 1 --2- 172.21.2.112:0/154724568 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1940006770 0x7f1940008c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:54.735 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.730 7f195aa87700 1 --2- 172.21.2.112:0/154724568 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1940009510 0x7f194000ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:54.735 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.730 7f195a286700 1 -- 172.21.2.112:0/154724568 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1940009510 msgr2=0x7f194000ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:54.735 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.730 7f195a286700 1 --2- 172.21.2.112:0/154724568 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1940009510 0x7f194000ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:54.735 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.730 7f195a286700 1 -- 172.21.2.112:0/154724568 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1940006770 msgr2=0x7f1940008c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:54.735 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.730 7f195a286700 1 --2- 172.21.2.112:0/154724568 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1940006770 0x7f1940008c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:54.735 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.730 7f195a286700 1 -- 172.21.2.112:0/154724568 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f194000c9c0 con 0x7f1940003ac0 2021-02-10T11:34:54.735 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.730 7f195a286700 1 --2- 172.21.2.112:0/154724568 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1940003ac0 0x7f1940005ed0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f1950006ab0 tx=0x7f1950009b40).ready entity=mon.2 client_cookie=a51609557f2829be server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:54.735 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.731 7f1959284700 1 -- 172.21.2.112:0/154724568 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1950006680 con 0x7f1940003ac0 2021-02-10T11:34:54.735 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.731 7f1959284700 1 -- 172.21.2.112:0/154724568 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19500063f0 con 0x7f1940003ac0 2021-02-10T11:34:54.736 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.731 7f1959284700 1 -- 172.21.2.112:0/154724568 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f195000c2f0 con 0x7f1940003ac0 2021-02-10T11:34:54.736 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.731 7f195c46a700 1 -- 172.21.2.112:0/154724568 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1940003ac0 msgr2=0x7f1940005ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:54.736 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.731 7f195c46a700 1 --2- 172.21.2.112:0/154724568 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1940003ac0 0x7f1940005ed0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f1950006ab0 tx=0x7f1950009b40).stop 2021-02-10T11:34:54.736 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.731 7f195aa87700 1 -- 172.21.2.112:0/154724568 reap_dead start 2021-02-10T11:34:54.736 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.731 7f195c46a700 1 -- 172.21.2.112:0/154724568 shutdown_connections 2021-02-10T11:34:54.736 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.731 7f195c46a700 1 -- 172.21.2.112:0/154724568 >> 172.21.2.112:0/154724568 conn(0x7f19540ce420 msgr2=0x7f19540d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:54.736 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.731 7f195c46a700 1 -- 172.21.2.112:0/154724568 shutdown_connections 2021-02-10T11:34:54.736 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.731 7f195c46a700 1 -- 172.21.2.112:0/154724568 wait complete. 2021-02-10T11:34:54.736 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.746 7f195c46a700 1 Processor -- start 2021-02-10T11:34:54.736 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.746 7f195c46a700 1 -- start start 2021-02-10T11:34:54.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.746 7f195c46a700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1954085bc0 0x7f19540863e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:54.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.746 7f195c46a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f19540954c0 0x7f1954096300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:54.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.746 7f195c46a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f19540958c0 0x7f1954096920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:54.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.746 7f195c46a700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1954095d80 con 0x7f19540954c0 2021-02-10T11:34:54.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.746 7f195c46a700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19540bfa60 con 0x7f1954085bc0 2021-02-10T11:34:54.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.746 7f195c46a700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1954085fc0 con 0x7f19540958c0 2021-02-10T11:34:54.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.747 7f1959a85700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f19540954c0 0x7f1954096300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:54.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.747 7f1959a85700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f19540954c0 0x7f1954096300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:50860/0 (socket says 172.21.2.112:50860) 2021-02-10T11:34:54.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.747 7f1959a85700 1 -- 172.21.2.112:0/1071753483 learned_addr learned my addr 172.21.2.112:0/1071753483 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:34:54.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.747 7f1959a85700 1 -- 172.21.2.112:0/1071753483 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f19540958c0 msgr2=0x7f1954096920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:54.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.747 7f1959a85700 1 --2- 172.21.2.112:0/1071753483 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f19540958c0 0x7f1954096920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:54.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.747 7f1959a85700 1 -- 172.21.2.112:0/1071753483 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1954085bc0 msgr2=0x7f19540863e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:54.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.747 7f1959a85700 1 --2- 172.21.2.112:0/1071753483 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1954085bc0 0x7f19540863e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:54.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.747 7f1959a85700 1 -- 172.21.2.112:0/1071753483 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1954086c10 con 0x7f19540954c0 2021-02-10T11:34:54.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.747 7f1959a85700 1 --2- 172.21.2.112:0/1071753483 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f19540954c0 0x7f1954096300 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 rx=0x7f194804b5f0 tx=0x7f194804e8f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:54.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.747 7f1946ffd700 1 -- 172.21.2.112:0/1071753483 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f194804ece0 con 0x7f19540954c0 2021-02-10T11:34:54.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.747 7f1946ffd700 1 -- 172.21.2.112:0/1071753483 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f194804f070 con 0x7f19540954c0 2021-02-10T11:34:54.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.747 7f195c46a700 1 -- 172.21.2.112:0/1071753483 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1954086c10 con 0x7f19540954c0 2021-02-10T11:34:54.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.747 7f1946ffd700 1 -- 172.21.2.112:0/1071753483 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f194804f4d0 con 0x7f19540954c0 2021-02-10T11:34:54.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.748 7f195c46a700 1 -- 172.21.2.112:0/1071753483 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1954086c10 con 0x7f19540954c0 2021-02-10T11:34:54.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.748 7f1946ffd700 1 -- 172.21.2.112:0/1071753483 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f194804a230 con 0x7f19540954c0 2021-02-10T11:34:54.740 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.748 7f1946ffd700 1 --2- 172.21.2.112:0/1071753483 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f193c0186c0 0x7f193c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:54.740 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.748 7f1946ffd700 1 -- 172.21.2.112:0/1071753483 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f19480765b0 con 0x7f19540954c0 2021-02-10T11:34:54.740 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.748 7f195a286700 1 --2- 172.21.2.112:0/1071753483 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f193c0186c0 0x7f193c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:54.740 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.749 7f195a286700 1 --2- 172.21.2.112:0/1071753483 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f193c0186c0 0x7f193c01abc0 crc :-1 s=READY pgs=696 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:54.765 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.778 7f195c46a700 1 -- 172.21.2.112:0/1071753483 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19540fe1c0 con 0x7f19540954c0 2021-02-10T11:34:54.768 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.781 7f1946ffd700 1 -- 172.21.2.112:0/1071753483 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1948071f40 con 0x7f19540954c0 2021-02-10T11:34:54.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.909 7f195c46a700 1 -- 172.21.2.112:0/1071753483 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f19540fe1c0 con 0x7f19540954c0 2021-02-10T11:34:54.897 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.910 7f1946ffd700 1 -- 172.21.2.112:0/1071753483 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v64) v1 ==== 100+0+10 (secure 0 0 0) 0x7f1948071f40 con 0x7f19540954c0 2021-02-10T11:34:54.897 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:34:54.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.912 7f195c46a700 1 -- 172.21.2.112:0/1071753483 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f193c0186c0 msgr2=0x7f193c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:54.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.912 7f195c46a700 1 --2- 172.21.2.112:0/1071753483 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f193c0186c0 0x7f193c01abc0 crc :-1 s=READY pgs=696 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:54.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.912 7f195c46a700 1 -- 172.21.2.112:0/1071753483 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f19540954c0 msgr2=0x7f1954096300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:54.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.912 7f195c46a700 1 --2- 172.21.2.112:0/1071753483 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f19540954c0 0x7f1954096300 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 rx=0x7f194804b5f0 tx=0x7f194804e8f0).stop 2021-02-10T11:34:54.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.912 7f195c46a700 1 -- 172.21.2.112:0/1071753483 shutdown_connections 2021-02-10T11:34:54.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.912 7f195c46a700 1 --2- 172.21.2.112:0/1071753483 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f193c0186c0 0x7f193c01abc0 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:54.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.912 7f195c46a700 1 --2- 172.21.2.112:0/1071753483 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f19540958c0 0x7f1954096920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:54.900 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.912 7f195c46a700 1 --2- 172.21.2.112:0/1071753483 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f19540954c0 0x7f1954096300 unknown :-1 s=CLOSED pgs=1423 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:54.900 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.912 7f195c46a700 1 --2- 172.21.2.112:0/1071753483 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1954085bc0 0x7f19540863e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:54.900 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.912 7f195c46a700 1 -- 172.21.2.112:0/1071753483 >> 172.21.2.112:0/1071753483 conn(0x7f19540bef00 msgr2=0x7f195409d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:54.900 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.912 7f195c46a700 1 -- 172.21.2.112:0/1071753483 shutdown_connections 2021-02-10T11:34:54.900 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:54.912 7f195c46a700 1 -- 172.21.2.112:0/1071753483 wait complete. 2021-02-10T11:34:54.907 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:34:54.907 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:34:55.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:55.331 7fd0dbb23700 1 Processor -- start 2021-02-10T11:34:55.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:55.331 7fd0dbb23700 1 -- start start 2021-02-10T11:34:55.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:55.332 7fd0dbb23700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd0d40d1870 0x7fd0d41b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:55.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:55.332 7fd0dbb23700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fd0d41b0a10 0x7fd0d40cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:55.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:55.332 7fd0dbb23700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0d40f0b10 con 0x7fd0d40d1c70 2021-02-10T11:34:55.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:55.332 7fd0dbb23700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0d40fe230 con 0x7fd0d41b0a10 2021-02-10T11:34:55.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:55.332 7fd0dbb23700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0d40e5ed0 con 0x7fd0d40d1870 2021-02-10T11:34:55.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:55.332 7fd0d913e700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fd0d41b0a10 0x7fd0d40cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:55.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:55.332 7fd0da140700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fd0d40d1c70 0x7fd0d40cf950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:50330/0 (socket says 172.21.2.112:50330) 2021-02-10T11:34:55.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:55.332 7fd0da140700 1 -- 172.21.2.112:0/2931832173 learned_addr learned my addr 172.21.2.112:0/2931832173 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:34:55.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:55.332 7fd0d913e700 1 -- 172.21.2.112:0/2931832173 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd0d40d1870 msgr2=0x7fd0d41b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:55.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:55.332 7fd0d913e700 1 --2- 172.21.2.112:0/2931832173 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd0d40d1870 0x7fd0d41b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:55.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:55.332 7fd0d913e700 1 -- 172.21.2.112:0/2931832173 >> v1:172.21.2.112:6789/0 conn(0x7fd0d40d1c70 legacy=0x7fd0d40cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:55.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:55.332 7fd0d913e700 1 -- 172.21.2.112:0/2931832173 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0d40f0650 con 0x7fd0d41b0a10 2021-02-10T11:34:55.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:55.332 7fd0d913e700 1 --2- 172.21.2.112:0/2931832173 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd0d41b0a10 0x7fd0d40cf330 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 rx=0x7fd0d0006650 tx=0x7fd0d0009a00).ready entity=mon.0 client_cookie=1557d2431bfb67e9 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:55.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:55.333 7fd0d893d700 1 -- 172.21.2.112:0/2931832173 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd0d00009b0 con 0x7fd0d41b0a10 2021-02-10T11:34:55.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:55.333 7fd0d893d700 1 -- 172.21.2.112:0/2931832173 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0d000c0c0 con 0x7fd0d41b0a10 2021-02-10T11:34:55.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:55.333 7fd0d893d700 1 -- 172.21.2.112:0/2931832173 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd0d000c510 con 0x7fd0d41b0a10 2021-02-10T11:34:55.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:55.333 7fd0dbb23700 1 -- 172.21.2.112:0/2931832173 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd0d41b0a10 msgr2=0x7fd0d40cf330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:55.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:55.333 7fd0dbb23700 1 --2- 172.21.2.112:0/2931832173 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd0d41b0a10 0x7fd0d40cf330 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 rx=0x7fd0d0006650 tx=0x7fd0d0009a00).stop 2021-02-10T11:34:55.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:55.333 7fd0dbb23700 1 -- 172.21.2.112:0/2931832173 shutdown_connections 2021-02-10T11:34:55.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:55.333 7fd0dbb23700 1 --2- 172.21.2.112:0/2931832173 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd0d41b0a10 0x7fd0d40cf330 unknown :-1 s=CLOSED pgs=1425 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:55.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:55.333 7fd0dbb23700 1 --2- 172.21.2.112:0/2931832173 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd0d40d1870 0x7fd0d41b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:55.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:55.333 7fd0dbb23700 1 -- 172.21.2.112:0/2931832173 >> 172.21.2.112:0/2931832173 conn(0x7fd0d40ce450 msgr2=0x7fd0d40d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:55.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:55.334 7fd0dbb23700 1 -- 172.21.2.112:0/2931832173 shutdown_connections 2021-02-10T11:34:55.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:55.334 7fd0dbb23700 1 -- 172.21.2.112:0/2931832173 wait complete. 2021-02-10T11:34:55.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:55.348 7fd0dbb23700 1 Processor -- start 2021-02-10T11:34:55.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:55.348 7fd0dbb23700 1 -- start start 2021-02-10T11:34:55.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:55.348 7fd0dbb23700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd0d40ad770 0x7fd0d40ef9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:55.339 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:55.348 7fd0dbb23700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd0d409af60 0x7fd0d409b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:55.339 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:55.348 7fd0dbb23700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd0d409b980 0x7fd0d409bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:55.339 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:55.348 7fd0dbb23700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0d409a9e0 con 0x7fd0d409af60 2021-02-10T11:34:55.339 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:55.348 7fd0dbb23700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0d4187e50 con 0x7fd0d40ad770 2021-02-10T11:34:55.339 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:55.348 7fd0dbb23700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0d4188030 con 0x7fd0d409b980 2021-02-10T11:34:55.339 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:55.349 7fd0d913e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd0d409af60 0x7fd0d409b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:55.339 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:55.349 7fd0d913e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd0d409af60 0x7fd0d409b360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:50872/0 (socket says 172.21.2.112:50872) 2021-02-10T11:34:55.339 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:55.349 7fd0d913e700 1 -- 172.21.2.112:0/2507825961 learned_addr learned my addr 172.21.2.112:0/2507825961 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:34:55.339 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:55.349 7fd0d993f700 1 --2- 172.21.2.112:0/2507825961 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd0d40ad770 0x7fd0d40ef9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:55.339 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:55.349 7fd0da140700 1 --2- 172.21.2.112:0/2507825961 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd0d409b980 0x7fd0d409bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:55.339 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:55.349 7fd0d913e700 1 -- 172.21.2.112:0/2507825961 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd0d409b980 msgr2=0x7fd0d409bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:55.339 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:55.349 7fd0d913e700 1 --2- 172.21.2.112:0/2507825961 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd0d409b980 0x7fd0d409bd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:55.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:55.349 7fd0d913e700 1 -- 172.21.2.112:0/2507825961 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd0d40ad770 msgr2=0x7fd0d40ef9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:55.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:55.349 7fd0d913e700 1 --2- 172.21.2.112:0/2507825961 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd0d40ad770 0x7fd0d40ef9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:55.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:55.349 7fd0d913e700 1 -- 172.21.2.112:0/2507825961 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0d40992b0 con 0x7fd0d409af60 2021-02-10T11:34:55.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:55.349 7fd0d913e700 1 --2- 172.21.2.112:0/2507825961 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd0d409af60 0x7fd0d409b360 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 rx=0x7fd0d000b640 tx=0x7fd0d000b9e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:55.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:55.350 7fd0c67fc700 1 -- 172.21.2.112:0/2507825961 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd0d000bdd0 con 0x7fd0d409af60 2021-02-10T11:34:55.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:55.350 7fd0c67fc700 1 -- 172.21.2.112:0/2507825961 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0d0000c90 con 0x7fd0d409af60 2021-02-10T11:34:55.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:55.350 7fd0c67fc700 1 -- 172.21.2.112:0/2507825961 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd0d0002220 con 0x7fd0d409af60 2021-02-10T11:34:55.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:55.350 7fd0dbb23700 1 -- 172.21.2.112:0/2507825961 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0d40992b0 con 0x7fd0d409af60 2021-02-10T11:34:55.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:55.350 7fd0dbb23700 1 -- 172.21.2.112:0/2507825961 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0d40992b0 con 0x7fd0d409af60 2021-02-10T11:34:55.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:55.350 7fd0c67fc700 1 -- 172.21.2.112:0/2507825961 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd0d0006a10 con 0x7fd0d409af60 2021-02-10T11:34:55.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:55.350 7fd0c67fc700 1 --2- 172.21.2.112:0/2507825961 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd0bc01a8b0 0x7fd0bc01cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:55.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:55.350 7fd0c67fc700 1 -- 172.21.2.112:0/2507825961 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fd0d0032460 con 0x7fd0d409af60 2021-02-10T11:34:55.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:55.351 7fd0d993f700 1 --2- 172.21.2.112:0/2507825961 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd0bc01a8b0 0x7fd0bc01cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:55.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:55.351 7fd0d993f700 1 --2- 172.21.2.112:0/2507825961 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd0bc01a8b0 0x7fd0bc01cdd0 crc :-1 s=READY pgs=697 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:55.368 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:55.382 7fd0dbb23700 1 -- 172.21.2.112:0/2507825961 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0d40ee060 con 0x7fd0d409af60 2021-02-10T11:34:55.371 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:55.384 7fd0c67fc700 1 -- 172.21.2.112:0/2507825961 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd0d0030d90 con 0x7fd0d409af60 2021-02-10T11:34:55.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:55.511 7fd0dbb23700 1 -- 172.21.2.112:0/2507825961 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd0d40ce450 con 0x7fd0d409af60 2021-02-10T11:34:55.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:55.511 7fd0c67fc700 1 -- 172.21.2.112:0/2507825961 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) v1 ==== 94+0+2101 (secure 0 0 0) 0x7fd0d000bdd0 con 0x7fd0d409af60 2021-02-10T11:34:55.499 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 47 2021-02-10T11:34:55.499 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:34:55.499 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":47,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8334,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/3370618110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":3370618110},{"type":"v1","addr":"172.21.2.112:6833","nonce":3370618110}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":44}],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:34:45.582177","modified":"2021-02-10 11:34:47.589305","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5921},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5921":{"gid":5921,"name":"b","rank":0,"incarnation":45,"state":"up:active","state_seq":3,"addr":"172.21.2.120:6835/2436135065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":2436135065},{"type":"v1","addr":"172.21.2.120:6835","nonce":2436135065}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2021-02-10T11:34:55.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:55.513 7fd0dbb23700 1 -- 172.21.2.112:0/2507825961 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd0bc01a8b0 msgr2=0x7fd0bc01cdd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:55.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:55.513 7fd0dbb23700 1 --2- 172.21.2.112:0/2507825961 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd0bc01a8b0 0x7fd0bc01cdd0 crc :-1 s=READY pgs=697 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:55.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:55.513 7fd0dbb23700 1 -- 172.21.2.112:0/2507825961 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd0d409af60 msgr2=0x7fd0d409b360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:55.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:55.513 7fd0dbb23700 1 --2- 172.21.2.112:0/2507825961 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd0d409af60 0x7fd0d409b360 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 rx=0x7fd0d000b640 tx=0x7fd0d000b9e0).stop 2021-02-10T11:34:55.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:55.513 7fd0dbb23700 1 -- 172.21.2.112:0/2507825961 shutdown_connections 2021-02-10T11:34:55.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:55.513 7fd0dbb23700 1 --2- 172.21.2.112:0/2507825961 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd0bc01a8b0 0x7fd0bc01cdd0 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:55.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:55.513 7fd0dbb23700 1 --2- 172.21.2.112:0/2507825961 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd0d409b980 0x7fd0d409bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:55.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:55.513 7fd0dbb23700 1 --2- 172.21.2.112:0/2507825961 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd0d409af60 0x7fd0d409b360 unknown :-1 s=CLOSED pgs=1426 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:55.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:55.513 7fd0dbb23700 1 --2- 172.21.2.112:0/2507825961 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd0d40ad770 0x7fd0d40ef9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:55.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:55.513 7fd0dbb23700 1 -- 172.21.2.112:0/2507825961 >> 172.21.2.112:0/2507825961 conn(0x7fd0d41a2810 msgr2=0x7fd0d4189af0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:55.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:55.514 7fd0dbb23700 1 -- 172.21.2.112:0/2507825961 shutdown_connections 2021-02-10T11:34:55.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:34:55.514 7fd0dbb23700 1 -- 172.21.2.112:0/2507825961 wait complete. 2021-02-10T11:34:55.710 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 47, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:34:45.582177', 'modified': '2021-02-10 11:34:47.589305', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5921}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5921': {'gid': 5921, 'name': 'b', 'rank': 0, 'incarnation': 45, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.120:6835/2436135065', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.120:6834', 'nonce': 2436135065}, {'type': 'v1', 'addr': '172.21.2.120:6835', 'nonce': 2436135065}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:34:55.710 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:34:55.711 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:34:56.106 INFO:teuthology.orchestra.run.gibba020.stdout:{ 2021-02-10T11:34:56.106 INFO:teuthology.orchestra.run.gibba020.stdout: "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", 2021-02-10T11:34:56.106 INFO:teuthology.orchestra.run.gibba020.stdout: "whoami": 0, 2021-02-10T11:34:56.107 INFO:teuthology.orchestra.run.gibba020.stdout: "id": 5921, 2021-02-10T11:34:56.107 INFO:teuthology.orchestra.run.gibba020.stdout: "want_state": "up:active", 2021-02-10T11:34:56.107 INFO:teuthology.orchestra.run.gibba020.stdout: "state": "up:active", 2021-02-10T11:34:56.107 INFO:teuthology.orchestra.run.gibba020.stdout: "rank_uptime": 10.534467963000001, 2021-02-10T11:34:56.107 INFO:teuthology.orchestra.run.gibba020.stdout: "mdsmap_epoch": 47, 2021-02-10T11:34:56.107 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch": 64, 2021-02-10T11:34:56.107 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch_barrier": 64, 2021-02-10T11:34:56.107 INFO:teuthology.orchestra.run.gibba020.stdout: "uptime": 16.178343854000001 2021-02-10T11:34:56.107 INFO:teuthology.orchestra.run.gibba020.stdout:} 2021-02-10T11:34:56.110 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", "whoami": 0, "id": 5921, "want_state": "up:active", "state": "up:active", "rank_uptime": 10.534467963000001, "mdsmap_epoch": 47, "osdmap_epoch": 64, "osdmap_epoch_barrier": 64, "uptime": 16.178343854000001 } 2021-02-10T11:34:56.110 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:34:56.111 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2021-02-10T11:34:56.111 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@gibba012.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2021-02-10T11:34:56.111 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:56.111 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:34:56.446 DEBUG:teuthology.orchestra.run.gibba012:> sudo modprobe fuse 2021-02-10T11:34:56.790 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:34:56.790 DEBUG:teuthology.orchestra.run.gibba012:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:34:57.133 INFO:teuthology.orchestra.run.gibba012.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:34:57.133 INFO:teuthology.orchestra.run.gibba012.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:34:57.134 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:34:57.134 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:57.134 DEBUG:teuthology.orchestra.run.gibba012:> ls /sys/fs/fuse/connections 2021-02-10T11:34:57.469 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2021-02-10T11:34:57.469 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:34:57.472 DEBUG:teuthology.orchestra.run.gibba012:> sudo modprobe fuse 2021-02-10T11:34:57.814 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:34:57.815 DEBUG:teuthology.orchestra.run.gibba012:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:34:57.880 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba012.stderr:2021-02-10 11:34:57.893 7f17d38b8f80 -1 init, newargv = 0x5648628a6200 newargc=9 2021-02-10T11:34:57.880 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba012.stderr:ceph-fuse[135129]: starting ceph client 2021-02-10T11:34:57.885 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba012.stderr:ceph-fuse[135129]: starting fuse 2021-02-10T11:34:58.159 INFO:teuthology.orchestra.run.gibba012.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:34:58.159 INFO:teuthology.orchestra.run.gibba012.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:34:58.159 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:34:58.160 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:58.160 DEBUG:teuthology.orchestra.run.gibba012:> ls /sys/fs/fuse/connections 2021-02-10T11:34:58.493 INFO:teuthology.orchestra.run.gibba012.stdout:42 2021-02-10T11:34:58.494 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2021-02-10T11:34:58.494 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:58.494 DEBUG:teuthology.orchestra.run.gibba012:> sudo python3 -c ' 2021-02-10T11:34:58.494 DEBUG:teuthology.orchestra.run.gibba012:> import glob 2021-02-10T11:34:58.494 DEBUG:teuthology.orchestra.run.gibba012:> import re 2021-02-10T11:34:58.494 DEBUG:teuthology.orchestra.run.gibba012:> import os 2021-02-10T11:34:58.494 DEBUG:teuthology.orchestra.run.gibba012:> import subprocess 2021-02-10T11:34:58.494 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:34:58.495 DEBUG:teuthology.orchestra.run.gibba012:> def find_socket(client_name): 2021-02-10T11:34:58.495 DEBUG:teuthology.orchestra.run.gibba012:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2021-02-10T11:34:58.495 DEBUG:teuthology.orchestra.run.gibba012:> files = glob.glob(asok_path) 2021-02-10T11:34:58.495 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:34:58.495 DEBUG:teuthology.orchestra.run.gibba012:> # Given a non-glob path, it better be there 2021-02-10T11:34:58.495 DEBUG:teuthology.orchestra.run.gibba012:> if "*" not in asok_path: 2021-02-10T11:34:58.495 DEBUG:teuthology.orchestra.run.gibba012:> assert(len(files) == 1) 2021-02-10T11:34:58.495 DEBUG:teuthology.orchestra.run.gibba012:> return files[0] 2021-02-10T11:34:58.495 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:34:58.495 DEBUG:teuthology.orchestra.run.gibba012:> for f in files: 2021-02-10T11:34:58.495 DEBUG:teuthology.orchestra.run.gibba012:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:34:58.495 DEBUG:teuthology.orchestra.run.gibba012:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:34:58.495 DEBUG:teuthology.orchestra.run.gibba012:> return f 2021-02-10T11:34:58.495 DEBUG:teuthology.orchestra.run.gibba012:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:34:58.496 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:34:58.496 DEBUG:teuthology.orchestra.run.gibba012:> print(find_socket("client.0")) 2021-02-10T11:34:58.496 DEBUG:teuthology.orchestra.run.gibba012:> ' 2021-02-10T11:34:58.857 INFO:teuthology.orchestra.run.gibba012.stdout:/var/run/ceph/ceph-client.0.135129.asok 2021-02-10T11:34:58.859 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.135129.asok 2021-02-10T11:34:58.859 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:58.860 DEBUG:teuthology.orchestra.run.gibba012:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.135129.asok status 2021-02-10T11:34:59.249 INFO:teuthology.orchestra.run.gibba012.stdout:{ 2021-02-10T11:34:59.249 INFO:teuthology.orchestra.run.gibba012.stdout: "metadata": { 2021-02-10T11:34:59.249 INFO:teuthology.orchestra.run.gibba012.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:34:59.249 INFO:teuthology.orchestra.run.gibba012.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:34:59.249 INFO:teuthology.orchestra.run.gibba012.stdout: "entity_id": "0", 2021-02-10T11:34:59.250 INFO:teuthology.orchestra.run.gibba012.stdout: "hostname": "gibba012", 2021-02-10T11:34:59.250 INFO:teuthology.orchestra.run.gibba012.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2021-02-10T11:34:59.250 INFO:teuthology.orchestra.run.gibba012.stdout: "pid": "135129", 2021-02-10T11:34:59.250 INFO:teuthology.orchestra.run.gibba012.stdout: "root": "/" 2021-02-10T11:34:59.250 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:59.250 INFO:teuthology.orchestra.run.gibba012.stdout: "dentry_count": 0, 2021-02-10T11:34:59.250 INFO:teuthology.orchestra.run.gibba012.stdout: "dentry_pinned_count": 0, 2021-02-10T11:34:59.250 INFO:teuthology.orchestra.run.gibba012.stdout: "id": 8502, 2021-02-10T11:34:59.250 INFO:teuthology.orchestra.run.gibba012.stdout: "inst": { 2021-02-10T11:34:59.250 INFO:teuthology.orchestra.run.gibba012.stdout: "name": { 2021-02-10T11:34:59.250 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "client", 2021-02-10T11:34:59.250 INFO:teuthology.orchestra.run.gibba012.stdout: "num": 8502 2021-02-10T11:34:59.251 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:59.251 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": { 2021-02-10T11:34:59.251 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:34:59.251 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:0", 2021-02-10T11:34:59.251 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 3400286955 2021-02-10T11:34:59.251 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:34:59.251 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:59.251 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": { 2021-02-10T11:34:59.251 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:34:59.251 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:0", 2021-02-10T11:34:59.251 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 3400286955 2021-02-10T11:34:59.251 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:34:59.251 INFO:teuthology.orchestra.run.gibba012.stdout: "inst_str": "client.8502 172.21.2.112:0/3400286955", 2021-02-10T11:34:59.252 INFO:teuthology.orchestra.run.gibba012.stdout: "addr_str": "172.21.2.112:0/3400286955", 2021-02-10T11:34:59.252 INFO:teuthology.orchestra.run.gibba012.stdout: "inode_count": 1, 2021-02-10T11:34:59.252 INFO:teuthology.orchestra.run.gibba012.stdout: "mds_epoch": 47, 2021-02-10T11:34:59.252 INFO:teuthology.orchestra.run.gibba012.stdout: "osd_epoch": 64, 2021-02-10T11:34:59.252 INFO:teuthology.orchestra.run.gibba012.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:34:59.252 INFO:teuthology.orchestra.run.gibba012.stdout: "blacklisted": false 2021-02-10T11:34:59.252 INFO:teuthology.orchestra.run.gibba012.stdout:} 2021-02-10T11:34:59.253 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:59.253 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:34:59.254 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:34:59.588 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:34:59.589 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:34:59.589 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:59.589 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:34:59.935 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:35:00.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.359 7f5b685c4700 1 Processor -- start 2021-02-10T11:35:00.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.359 7f5b685c4700 1 -- start start 2021-02-10T11:35:00.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.359 7f5b685c4700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f5b601b0870 0x7f5b600f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:00.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.359 7f5b685c4700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f5b600efe00 0x7f5b600f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:00.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.359 7f5b685c4700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b601b0c70 con 0x7f5b601b0470 2021-02-10T11:35:00.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.359 7f5b685c4700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b600e5ea0 con 0x7f5b600efe00 2021-02-10T11:35:00.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.359 7f5b685c4700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b600f0b50 con 0x7f5b601b0870 2021-02-10T11:35:00.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.359 7f5b66be1700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f5b600efe00 0x7f5b600f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:00.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.359 7f5b663e0700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f5b601b0470 0x7f5b600ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:50352/0 (socket says 172.21.2.112:50352) 2021-02-10T11:35:00.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.359 7f5b66be1700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f5b600efe00 0x7f5b600f0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:50892/0 (socket says 172.21.2.112:50892) 2021-02-10T11:35:00.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.359 7f5b663e0700 1 -- 172.21.2.112:0/1525010563 learned_addr learned my addr 172.21.2.112:0/1525010563 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:35:00.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.360 7f5b65bdf700 1 --2- 172.21.2.112:0/1525010563 >> v2:172.21.2.120:3300/0 conn(0x7f5b601b0870 0x7f5b600f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:00.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.360 7f5b66be1700 1 -- 172.21.2.112:0/1525010563 >> v2:172.21.2.120:3300/0 conn(0x7f5b601b0870 msgr2=0x7f5b600f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:00.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.360 7f5b66be1700 1 --2- 172.21.2.112:0/1525010563 >> v2:172.21.2.120:3300/0 conn(0x7f5b601b0870 0x7f5b600f20e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:00.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.360 7f5b66be1700 1 -- 172.21.2.112:0/1525010563 >> v1:172.21.2.112:6789/0 conn(0x7f5b601b0470 legacy=0x7f5b600ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:00.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.360 7f5b66be1700 1 -- 172.21.2.112:0/1525010563 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b600f0820 con 0x7f5b600efe00 2021-02-10T11:35:00.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.360 7f5b66be1700 1 --2- 172.21.2.112:0/1525010563 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5b600efe00 0x7f5b600f0200 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 rx=0x7f5b50006ab0 tx=0x7f5b50009ac0).ready entity=mon.0 client_cookie=966f467ded3f4046 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:00.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.360 7f5b653de700 1 -- 172.21.2.112:0/1525010563 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5b500066d0 con 0x7f5b600efe00 2021-02-10T11:35:00.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.360 7f5b653de700 1 -- 172.21.2.112:0/1525010563 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b500022e0 con 0x7f5b600efe00 2021-02-10T11:35:00.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.360 7f5b653de700 1 -- 172.21.2.112:0/1525010563 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5b500027c0 con 0x7f5b600efe00 2021-02-10T11:35:00.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.361 7f5b685c4700 1 -- 172.21.2.112:0/1525010563 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5b600efe00 msgr2=0x7f5b600f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:00.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.361 7f5b685c4700 1 --2- 172.21.2.112:0/1525010563 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5b600efe00 0x7f5b600f0200 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 rx=0x7f5b50006ab0 tx=0x7f5b50009ac0).stop 2021-02-10T11:35:00.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.361 7f5b685c4700 1 -- 172.21.2.112:0/1525010563 shutdown_connections 2021-02-10T11:35:00.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.361 7f5b685c4700 1 --2- 172.21.2.112:0/1525010563 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5b600efe00 0x7f5b600f0200 unknown :-1 s=CLOSED pgs=1431 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:00.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.361 7f5b685c4700 1 --2- 172.21.2.112:0/1525010563 >> v2:172.21.2.120:3300/0 conn(0x7f5b601b0870 0x7f5b600f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:00.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.361 7f5b685c4700 1 -- 172.21.2.112:0/1525010563 >> 172.21.2.112:0/1525010563 conn(0x7f5b600ce3e0 msgr2=0x7f5b600e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:00.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.361 7f5b685c4700 1 -- 172.21.2.112:0/1525010563 shutdown_connections 2021-02-10T11:35:00.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.361 7f5b685c4700 1 -- 172.21.2.112:0/1525010563 wait complete. 2021-02-10T11:35:00.365 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.376 7f5b685c4700 1 Processor -- start 2021-02-10T11:35:00.365 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.376 7f5b685c4700 1 -- start start 2021-02-10T11:35:00.366 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.376 7f5b685c4700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5b600aa8d0 0x7f5b6019a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:00.366 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.376 7f5b685c4700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5b6019e9d0 0x7f5b6019a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:00.366 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.376 7f5b685c4700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5b6019ad40 0x7f5b6019b770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:00.366 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.376 7f5b685c4700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b6019b1f0 con 0x7f5b600aa8d0 2021-02-10T11:35:00.366 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.376 7f5b685c4700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b60109b30 con 0x7f5b6019e9d0 2021-02-10T11:35:00.366 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.376 7f5b685c4700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b60099620 con 0x7f5b6019ad40 2021-02-10T11:35:00.366 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.376 7f5b663e0700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5b600aa8d0 0x7f5b6019a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:00.366 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.376 7f5b663e0700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5b600aa8d0 0x7f5b6019a100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:50894/0 (socket says 172.21.2.112:50894) 2021-02-10T11:35:00.366 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.376 7f5b663e0700 1 -- 172.21.2.112:0/2256658916 learned_addr learned my addr 172.21.2.112:0/2256658916 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:35:00.366 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.377 7f5b663e0700 1 -- 172.21.2.112:0/2256658916 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5b6019ad40 msgr2=0x7f5b6019b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:00.366 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.377 7f5b663e0700 1 --2- 172.21.2.112:0/2256658916 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5b6019ad40 0x7f5b6019b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:00.366 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.377 7f5b663e0700 1 -- 172.21.2.112:0/2256658916 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5b6019e9d0 msgr2=0x7f5b6019a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:00.366 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.377 7f5b663e0700 1 --2- 172.21.2.112:0/2256658916 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5b6019e9d0 0x7f5b6019a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:00.367 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.377 7f5b663e0700 1 -- 172.21.2.112:0/2256658916 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b60109640 con 0x7f5b600aa8d0 2021-02-10T11:35:00.367 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.377 7f5b663e0700 1 --2- 172.21.2.112:0/2256658916 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5b600aa8d0 0x7f5b6019a100 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 rx=0x7f5b54006f00 tx=0x7f5b5400b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:00.367 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.377 7f5b5effd700 1 -- 172.21.2.112:0/2256658916 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5b5400bd30 con 0x7f5b600aa8d0 2021-02-10T11:35:00.367 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.377 7f5b5effd700 1 -- 172.21.2.112:0/2256658916 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b54006030 con 0x7f5b600aa8d0 2021-02-10T11:35:00.367 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.377 7f5b685c4700 1 -- 172.21.2.112:0/2256658916 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b60109640 con 0x7f5b600aa8d0 2021-02-10T11:35:00.367 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.377 7f5b685c4700 1 -- 172.21.2.112:0/2256658916 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b60109640 con 0x7f5b600aa8d0 2021-02-10T11:35:00.367 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.377 7f5b5effd700 1 -- 172.21.2.112:0/2256658916 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5b54004620 con 0x7f5b600aa8d0 2021-02-10T11:35:00.367 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.377 7f5b5effd700 1 -- 172.21.2.112:0/2256658916 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5b540043a0 con 0x7f5b600aa8d0 2021-02-10T11:35:00.367 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.378 7f5b5effd700 1 --2- 172.21.2.112:0/2256658916 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5b40018730 0x7f5b4001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:00.367 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.378 7f5b5effd700 1 -- 172.21.2.112:0/2256658916 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f5b54032780 con 0x7f5b600aa8d0 2021-02-10T11:35:00.367 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.378 7f5b65bdf700 1 --2- 172.21.2.112:0/2256658916 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5b40018730 0x7f5b4001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:00.368 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.379 7f5b65bdf700 1 --2- 172.21.2.112:0/2256658916 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5b40018730 0x7f5b4001ac10 crc :-1 s=READY pgs=698 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:00.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.409 7f5b685c4700 1 -- 172.21.2.112:0/2256658916 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b600f0b50 con 0x7f5b600aa8d0 2021-02-10T11:35:00.399 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.412 7f5b5effd700 1 -- 172.21.2.112:0/2256658916 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5b540009b0 con 0x7f5b600aa8d0 2021-02-10T11:35:00.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.536 7f5b685c4700 1 -- 172.21.2.112:0/2256658916 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5b60109ae0 con 0x7f5b600aa8d0 2021-02-10T11:35:00.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.537 7f5b5effd700 1 -- 172.21.2.112:0/2256658916 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v64) v1 ==== 74+0+13996 (secure 0 0 0) 0x7f5b5400bd30 con 0x7f5b600aa8d0 2021-02-10T11:35:00.524 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:35:00.524 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":64,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:34:46.584716","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:34:43.550220","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":13,"pool_name":"cephfs_data","create_time":"2021-02-10 11:34:44.578113","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:35:00.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.538 7f5b685c4700 1 -- 172.21.2.112:0/2256658916 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5b40018730 msgr2=0x7f5b4001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:00.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.538 7f5b685c4700 1 --2- 172.21.2.112:0/2256658916 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5b40018730 0x7f5b4001ac10 crc :-1 s=READY pgs=698 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:00.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.538 7f5b685c4700 1 -- 172.21.2.112:0/2256658916 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5b600aa8d0 msgr2=0x7f5b6019a100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:00.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.538 7f5b685c4700 1 --2- 172.21.2.112:0/2256658916 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5b600aa8d0 0x7f5b6019a100 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 rx=0x7f5b54006f00 tx=0x7f5b5400b940).stop 2021-02-10T11:35:00.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.539 7f5b685c4700 1 -- 172.21.2.112:0/2256658916 shutdown_connections 2021-02-10T11:35:00.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.539 7f5b685c4700 1 --2- 172.21.2.112:0/2256658916 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5b40018730 0x7f5b4001ac10 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:00.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.539 7f5b685c4700 1 --2- 172.21.2.112:0/2256658916 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5b6019e9d0 0x7f5b6019a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:00.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.539 7f5b685c4700 1 --2- 172.21.2.112:0/2256658916 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5b6019ad40 0x7f5b6019b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:00.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.539 7f5b685c4700 1 --2- 172.21.2.112:0/2256658916 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5b600aa8d0 0x7f5b6019a100 unknown :-1 s=CLOSED pgs=1432 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:00.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.539 7f5b685c4700 1 -- 172.21.2.112:0/2256658916 >> 172.21.2.112:0/2256658916 conn(0x7f5b6019f7b0 msgr2=0x7f5b6018f380 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:00.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.539 7f5b685c4700 1 -- 172.21.2.112:0/2256658916 shutdown_connections 2021-02-10T11:35:00.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.539 7f5b685c4700 1 -- 172.21.2.112:0/2256658916 wait complete. 2021-02-10T11:35:00.534 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:13:57.508758', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 12, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:34:43.550220', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 13, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:34:44.578113', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '64', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:35:00.534 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:35:00.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.961 7f807b707700 1 Processor -- start 2021-02-10T11:35:00.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.961 7f807b707700 1 -- start start 2021-02-10T11:35:00.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.961 7f807b707700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f80740d1c60 0x7f80740cef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:00.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.961 7f807b707700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f80740cf570 0x7f80740cf970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:00.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.961 7f807b707700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80740fe1b0 con 0x7f80740d1860 2021-02-10T11:35:00.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.961 7f807b707700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80741b0ca0 con 0x7f80740cf570 2021-02-10T11:35:00.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.961 7f807b707700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8074109a50 con 0x7f80740d1c60 2021-02-10T11:35:00.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.962 7f8079d24700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f80740cf570 0x7f80740cf970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:00.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.962 7f8079d24700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f80740cf570 0x7f80740cf970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:50902/0 (socket says 172.21.2.112:50902) 2021-02-10T11:35:00.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.962 7f8079d24700 1 -- 172.21.2.112:0/826695073 learned_addr learned my addr 172.21.2.112:0/826695073 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:35:00.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.962 7f8079523700 1 --1- 172.21.2.112:0/826695073 >> v1:172.21.2.112:6789/0 conn(0x7f80740d1860 0x7f80741b04b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:50364/0 (socket says 172.21.2.112:50364) 2021-02-10T11:35:00.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.962 7f8078d22700 1 --2- 172.21.2.112:0/826695073 >> v2:172.21.2.120:3300/0 conn(0x7f80740d1c60 0x7f80740cef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:00.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.962 7f8079d24700 1 -- 172.21.2.112:0/826695073 >> v2:172.21.2.120:3300/0 conn(0x7f80740d1c60 msgr2=0x7f80740cef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:00.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.962 7f8079d24700 1 --2- 172.21.2.112:0/826695073 >> v2:172.21.2.120:3300/0 conn(0x7f80740d1c60 0x7f80740cef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:00.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.962 7f8079d24700 1 -- 172.21.2.112:0/826695073 >> v1:172.21.2.112:6789/0 conn(0x7f80740d1860 legacy=0x7f80741b04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:00.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.962 7f8079d24700 1 -- 172.21.2.112:0/826695073 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8074088070 con 0x7f80740cf570 2021-02-10T11:35:00.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.962 7f8079d24700 1 --2- 172.21.2.112:0/826695073 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f80740cf570 0x7f80740cf970 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 rx=0x7f806c006f00 tx=0x7f806c009f10).ready entity=mon.0 client_cookie=969c0fcce534de4a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:00.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.962 7f806bfff700 1 -- 172.21.2.112:0/826695073 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f806c003030 con 0x7f80740cf570 2021-02-10T11:35:00.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.962 7f806bfff700 1 -- 172.21.2.112:0/826695073 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f806c00b3b0 con 0x7f80740cf570 2021-02-10T11:35:00.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.962 7f806bfff700 1 -- 172.21.2.112:0/826695073 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f806c00b8d0 con 0x7f80740cf570 2021-02-10T11:35:00.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.962 7f807b707700 1 -- 172.21.2.112:0/826695073 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f80740cf570 msgr2=0x7f80740cf970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:00.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.962 7f807b707700 1 --2- 172.21.2.112:0/826695073 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f80740cf570 0x7f80740cf970 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 rx=0x7f806c006f00 tx=0x7f806c009f10).stop 2021-02-10T11:35:00.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.963 7f807b707700 1 -- 172.21.2.112:0/826695073 shutdown_connections 2021-02-10T11:35:00.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.963 7f807b707700 1 --2- 172.21.2.112:0/826695073 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f80740cf570 0x7f80740cf970 unknown :-1 s=CLOSED pgs=1434 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:00.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.963 7f807b707700 1 --2- 172.21.2.112:0/826695073 >> v2:172.21.2.120:3300/0 conn(0x7f80740d1c60 0x7f80740cef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:00.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.963 7f807b707700 1 -- 172.21.2.112:0/826695073 >> 172.21.2.112:0/826695073 conn(0x7f80740ce410 msgr2=0x7f80740d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:00.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.963 7f807b707700 1 -- 172.21.2.112:0/826695073 shutdown_connections 2021-02-10T11:35:00.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.963 7f807b707700 1 -- 172.21.2.112:0/826695073 wait complete. 2021-02-10T11:35:00.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.978 7f807b707700 1 Processor -- start 2021-02-10T11:35:00.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.978 7f807b707700 1 -- start start 2021-02-10T11:35:00.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.978 7f807b707700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8074085270 0x7f8074085670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:00.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.978 7f807b707700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8074085c90 0x7f8074086090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:00.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.978 7f807b707700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f80740c2740 0x7f80740c2b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:00.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.978 7f807b707700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8074084cf0 con 0x7f8074085270 2021-02-10T11:35:00.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.978 7f807b707700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80740be1b0 con 0x7f8074085c90 2021-02-10T11:35:00.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.978 7f807b707700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80740be390 con 0x7f80740c2740 2021-02-10T11:35:00.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.978 7f8079523700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8074085270 0x7f8074085670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:00.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.978 7f8079523700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8074085270 0x7f8074085670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:50906/0 (socket says 172.21.2.112:50906) 2021-02-10T11:35:00.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.978 7f8079523700 1 -- 172.21.2.112:0/3081978966 learned_addr learned my addr 172.21.2.112:0/3081978966 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:35:00.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.979 7f8078d22700 1 --2- 172.21.2.112:0/3081978966 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8074085c90 0x7f8074086090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:00.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.979 7f8079d24700 1 --2- 172.21.2.112:0/3081978966 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f80740c2740 0x7f80740c2b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:00.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.979 7f8079523700 1 -- 172.21.2.112:0/3081978966 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f80740c2740 msgr2=0x7f80740c2b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:00.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.979 7f8079523700 1 --2- 172.21.2.112:0/3081978966 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f80740c2740 0x7f80740c2b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:00.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.979 7f8079523700 1 -- 172.21.2.112:0/3081978966 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8074085c90 msgr2=0x7f8074086090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:00.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.979 7f8079523700 1 --2- 172.21.2.112:0/3081978966 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8074085c90 0x7f8074086090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:00.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.979 7f8079523700 1 -- 172.21.2.112:0/3081978966 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80740c3160 con 0x7f8074085270 2021-02-10T11:35:00.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.979 7f8079d24700 1 --2- 172.21.2.112:0/3081978966 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f80740c2740 0x7f80740c2b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:35:00.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.979 7f8079523700 1 --2- 172.21.2.112:0/3081978966 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8074085270 0x7f8074085670 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 rx=0x7f8070001c60 tx=0x7f807000a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:00.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.979 7f8069ffb700 1 -- 172.21.2.112:0/3081978966 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f807000acc0 con 0x7f8074085270 2021-02-10T11:35:00.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.979 7f8069ffb700 1 -- 172.21.2.112:0/3081978966 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8070004140 con 0x7f8074085270 2021-02-10T11:35:00.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.979 7f8069ffb700 1 -- 172.21.2.112:0/3081978966 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8070004510 con 0x7f8074085270 2021-02-10T11:35:00.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.979 7f807b707700 1 -- 172.21.2.112:0/3081978966 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80740c3160 con 0x7f8074085270 2021-02-10T11:35:00.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.979 7f807b707700 1 -- 172.21.2.112:0/3081978966 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80740c36c0 con 0x7f8074085270 2021-02-10T11:35:00.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.980 7f8069ffb700 1 -- 172.21.2.112:0/3081978966 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8070004320 con 0x7f8074085270 2021-02-10T11:35:00.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.980 7f8069ffb700 1 --2- 172.21.2.112:0/3081978966 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8054018700 0x7f805401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:00.971 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.980 7f8069ffb700 1 -- 172.21.2.112:0/3081978966 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f8070032630 con 0x7f8074085270 2021-02-10T11:35:00.971 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.980 7f8078d22700 1 --2- 172.21.2.112:0/3081978966 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8054018700 0x7f805401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:00.971 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:00.981 7f8078d22700 1 --2- 172.21.2.112:0/3081978966 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8054018700 0x7f805401abc0 crc :-1 s=READY pgs=699 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:00.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:01.011 7f807b707700 1 -- 172.21.2.112:0/3081978966 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8074082a20 con 0x7f8074085270 2021-02-10T11:35:01.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:01.014 7f8069ffb700 1 -- 172.21.2.112:0/3081978966 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8070000dd0 con 0x7f8074085270 2021-02-10T11:35:01.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:01.143 7f807b707700 1 -- 172.21.2.112:0/3081978966 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f8074082a20 con 0x7f8074085270 2021-02-10T11:35:01.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:01.143 7f8069ffb700 1 -- 172.21.2.112:0/3081978966 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v64) v1 ==== 92+0+10 (secure 0 0 0) 0x7f8070004140 con 0x7f8074085270 2021-02-10T11:35:01.130 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:35:01.131 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:01.145 7f807b707700 1 -- 172.21.2.112:0/3081978966 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8054018700 msgr2=0x7f805401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:01.131 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:01.145 7f807b707700 1 --2- 172.21.2.112:0/3081978966 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8054018700 0x7f805401abc0 crc :-1 s=READY pgs=699 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:01.131 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:01.145 7f807b707700 1 -- 172.21.2.112:0/3081978966 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8074085270 msgr2=0x7f8074085670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:01.132 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:01.145 7f807b707700 1 --2- 172.21.2.112:0/3081978966 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8074085270 0x7f8074085670 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 rx=0x7f8070001c60 tx=0x7f807000a8d0).stop 2021-02-10T11:35:01.132 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:01.145 7f807b707700 1 -- 172.21.2.112:0/3081978966 shutdown_connections 2021-02-10T11:35:01.132 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:01.145 7f807b707700 1 --2- 172.21.2.112:0/3081978966 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8054018700 0x7f805401abc0 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:01.132 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:01.145 7f807b707700 1 --2- 172.21.2.112:0/3081978966 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8074085c90 0x7f8074086090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:01.132 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:01.145 7f807b707700 1 --2- 172.21.2.112:0/3081978966 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f80740c2740 0x7f80740c2b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:01.132 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:01.145 7f807b707700 1 --2- 172.21.2.112:0/3081978966 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8074085270 0x7f8074085670 unknown :-1 s=CLOSED pgs=1435 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:01.132 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:01.145 7f807b707700 1 -- 172.21.2.112:0/3081978966 >> 172.21.2.112:0/3081978966 conn(0x7f80740b5f10 msgr2=0x7f807409f200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:01.133 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:01.145 7f807b707700 1 -- 172.21.2.112:0/3081978966 shutdown_connections 2021-02-10T11:35:01.133 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:01.145 7f807b707700 1 -- 172.21.2.112:0/3081978966 wait complete. 2021-02-10T11:35:01.140 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:35:01.570 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:01.567 7ff9b8543700 1 Processor -- start 2021-02-10T11:35:01.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:01.567 7ff9b8543700 1 -- start start 2021-02-10T11:35:01.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:01.568 7ff9b8543700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7ff9b00d1870 0x7ff9b01b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:01.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:01.568 7ff9b8543700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7ff9b01b0ae0 0x7ff9b0108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:01.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:01.568 7ff9b8543700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9b00fe1c0 con 0x7ff9b00d1c70 2021-02-10T11:35:01.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:01.568 7ff9b8543700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9b0109a60 con 0x7ff9b01b0ae0 2021-02-10T11:35:01.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:01.568 7ff9b8543700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9b0109750 con 0x7ff9b00d1870 2021-02-10T11:35:01.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:01.568 7ff9b5b5e700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7ff9b01b0ae0 0x7ff9b0108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:01.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:01.568 7ff9b5b5e700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7ff9b01b0ae0 0x7ff9b0108940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:50914/0 (socket says 172.21.2.112:50914) 2021-02-10T11:35:01.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:01.568 7ff9b5b5e700 1 -- 172.21.2.112:0/122119214 learned_addr learned my addr 172.21.2.112:0/122119214 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:35:01.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:01.568 7ff9b5b5e700 1 -- 172.21.2.112:0/122119214 >> v2:172.21.2.120:3300/0 conn(0x7ff9b00d1870 msgr2=0x7ff9b01b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:01.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:01.568 7ff9b5b5e700 1 --2- 172.21.2.112:0/122119214 >> v2:172.21.2.120:3300/0 conn(0x7ff9b00d1870 0x7ff9b01b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:01.572 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:01.568 7ff9b5b5e700 1 -- 172.21.2.112:0/122119214 >> v1:172.21.2.120:6789/0 conn(0x7ff9b00d1c70 legacy=0x7ff9b0108f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:01.572 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:01.568 7ff9b5b5e700 1 -- 172.21.2.112:0/122119214 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9b0088070 con 0x7ff9b01b0ae0 2021-02-10T11:35:01.572 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:01.568 7ff9b5b5e700 1 --2- 172.21.2.112:0/122119214 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff9b01b0ae0 0x7ff9b0108940 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 rx=0x7ff9a0006700 tx=0x7ff9a0009b00).ready entity=mon.0 client_cookie=b7a8f1fe0aadb700 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:01.572 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:01.569 7ff9b535d700 1 -- 172.21.2.112:0/122119214 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff9a000c0c0 con 0x7ff9b01b0ae0 2021-02-10T11:35:01.572 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:01.569 7ff9b535d700 1 -- 172.21.2.112:0/122119214 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9a000c400 con 0x7ff9b01b0ae0 2021-02-10T11:35:01.572 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:01.569 7ff9b535d700 1 -- 172.21.2.112:0/122119214 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff9a000c920 con 0x7ff9b01b0ae0 2021-02-10T11:35:01.572 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:01.569 7ff9b8543700 1 -- 172.21.2.112:0/122119214 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff9b01b0ae0 msgr2=0x7ff9b0108940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:01.572 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:01.569 7ff9b8543700 1 --2- 172.21.2.112:0/122119214 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff9b01b0ae0 0x7ff9b0108940 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 rx=0x7ff9a0006700 tx=0x7ff9a0009b00).stop 2021-02-10T11:35:01.572 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:01.570 7ff9b8543700 1 -- 172.21.2.112:0/122119214 shutdown_connections 2021-02-10T11:35:01.572 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:01.570 7ff9b8543700 1 --2- 172.21.2.112:0/122119214 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff9b01b0ae0 0x7ff9b0108940 unknown :-1 s=CLOSED pgs=1436 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:01.572 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:01.570 7ff9b8543700 1 --2- 172.21.2.112:0/122119214 >> v2:172.21.2.120:3300/0 conn(0x7ff9b00d1870 0x7ff9b01b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:01.572 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:01.570 7ff9b8543700 1 -- 172.21.2.112:0/122119214 >> 172.21.2.112:0/122119214 conn(0x7ff9b00ce420 msgr2=0x7ff9b00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:01.572 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:01.570 7ff9b8543700 1 -- 172.21.2.112:0/122119214 shutdown_connections 2021-02-10T11:35:01.573 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:01.570 7ff9b8543700 1 -- 172.21.2.112:0/122119214 wait complete. 2021-02-10T11:35:01.574 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:01.584 7ff9b8543700 1 Processor -- start 2021-02-10T11:35:01.574 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:01.584 7ff9b8543700 1 -- start start 2021-02-10T11:35:01.574 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:01.584 7ff9b8543700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff9b00c3820 0x7ff9b0084cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:01.574 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:01.584 7ff9b8543700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff9b00a0290 0x7ff9b00852d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:01.574 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:01.585 7ff9b8543700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff9b0086270 0x7ff9b0086670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:01.574 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:01.585 7ff9b8543700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9b00c0700 con 0x7ff9b0086270 2021-02-10T11:35:01.575 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:01.585 7ff9b8543700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9b00858f0 con 0x7ff9b00c3820 2021-02-10T11:35:01.575 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:01.585 7ff9b8543700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9b0085b10 con 0x7ff9b00a0290 2021-02-10T11:35:01.575 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:01.585 7ff9b6b60700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff9b0086270 0x7ff9b0086670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:01.575 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:01.585 7ff9b6b60700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff9b0086270 0x7ff9b0086670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:50924/0 (socket says 172.21.2.112:50924) 2021-02-10T11:35:01.575 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:01.585 7ff9b6b60700 1 -- 172.21.2.112:0/2615216378 learned_addr learned my addr 172.21.2.112:0/2615216378 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:35:01.575 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:01.585 7ff9b6b60700 1 -- 172.21.2.112:0/2615216378 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff9b00a0290 msgr2=0x7ff9b00852d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:01.575 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:01.585 7ff9b6b60700 1 --2- 172.21.2.112:0/2615216378 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff9b00a0290 0x7ff9b00852d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:01.575 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:01.585 7ff9b6b60700 1 -- 172.21.2.112:0/2615216378 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff9b00c3820 msgr2=0x7ff9b0084cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:01.575 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:01.585 7ff9b6b60700 1 --2- 172.21.2.112:0/2615216378 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff9b00c3820 0x7ff9b0084cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:01.575 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:01.585 7ff9b6b60700 1 -- 172.21.2.112:0/2615216378 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9b0086c90 con 0x7ff9b0086270 2021-02-10T11:35:01.575 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:01.585 7ff9b6b60700 1 --2- 172.21.2.112:0/2615216378 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff9b0086270 0x7ff9b0086670 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 rx=0x7ff9a8009940 tx=0x7ff9a800bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:01.575 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:01.585 7ff9a6ffd700 1 -- 172.21.2.112:0/2615216378 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff9a8000f90 con 0x7ff9b0086270 2021-02-10T11:35:01.575 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:01.585 7ff9a6ffd700 1 -- 172.21.2.112:0/2615216378 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9a800c2f0 con 0x7ff9b0086270 2021-02-10T11:35:01.576 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:01.585 7ff9a6ffd700 1 -- 172.21.2.112:0/2615216378 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff9a800c750 con 0x7ff9b0086270 2021-02-10T11:35:01.576 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:01.585 7ff9b8543700 1 -- 172.21.2.112:0/2615216378 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9b0086c90 con 0x7ff9b0086270 2021-02-10T11:35:01.576 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:01.585 7ff9b8543700 1 -- 172.21.2.112:0/2615216378 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9b01a2f00 con 0x7ff9b0086270 2021-02-10T11:35:01.576 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:01.586 7ff9a6ffd700 1 -- 172.21.2.112:0/2615216378 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff9a8001220 con 0x7ff9b0086270 2021-02-10T11:35:01.576 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:01.586 7ff9a6ffd700 1 --2- 172.21.2.112:0/2615216378 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff9980186c0 0x7ff99801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:01.576 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:01.586 7ff9a6ffd700 1 -- 172.21.2.112:0/2615216378 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5573+0+0 (secure 0 0 0) 0x7ff9a8032b20 con 0x7ff9b0086270 2021-02-10T11:35:01.576 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:01.587 7ff9b635f700 1 --2- 172.21.2.112:0/2615216378 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff9980186c0 0x7ff99801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:01.576 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:01.587 7ff9b635f700 1 --2- 172.21.2.112:0/2615216378 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff9980186c0 0x7ff99801abc0 crc :-1 s=READY pgs=700 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:01.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:01.618 7ff9b8543700 1 -- 172.21.2.112:0/2615216378 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9b0082a70 con 0x7ff9b0086270 2021-02-10T11:35:01.607 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:01.620 7ff9a6ffd700 1 -- 172.21.2.112:0/2615216378 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff9a8000f40 con 0x7ff9b0086270 2021-02-10T11:35:01.736 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:01.749 7ff9b8543700 1 -- 172.21.2.112:0/2615216378 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7ff9b0082a70 con 0x7ff9b0086270 2021-02-10T11:35:01.736 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:01.749 7ff9a6ffd700 1 -- 172.21.2.112:0/2615216378 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v64) v1 ==== 104+0+10 (secure 0 0 0) 0x7ff9a801f860 con 0x7ff9b0086270 2021-02-10T11:35:01.737 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:35:01.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:01.751 7ff9b8543700 1 -- 172.21.2.112:0/2615216378 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff9980186c0 msgr2=0x7ff99801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:01.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:01.751 7ff9b8543700 1 --2- 172.21.2.112:0/2615216378 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff9980186c0 0x7ff99801abc0 crc :-1 s=READY pgs=700 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:01.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:01.751 7ff9b8543700 1 -- 172.21.2.112:0/2615216378 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff9b0086270 msgr2=0x7ff9b0086670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:01.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:01.751 7ff9b8543700 1 --2- 172.21.2.112:0/2615216378 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff9b0086270 0x7ff9b0086670 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 rx=0x7ff9a8009940 tx=0x7ff9a800bcf0).stop 2021-02-10T11:35:01.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:01.751 7ff9b8543700 1 -- 172.21.2.112:0/2615216378 shutdown_connections 2021-02-10T11:35:01.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:01.751 7ff9b8543700 1 --2- 172.21.2.112:0/2615216378 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff9980186c0 0x7ff99801abc0 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:01.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:01.751 7ff9b8543700 1 --2- 172.21.2.112:0/2615216378 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff9b0086270 0x7ff9b0086670 unknown :-1 s=CLOSED pgs=1437 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:01.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:01.752 7ff9b8543700 1 --2- 172.21.2.112:0/2615216378 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff9b00a0290 0x7ff9b00852d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:01.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:01.752 7ff9b8543700 1 --2- 172.21.2.112:0/2615216378 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff9b00c3820 0x7ff9b0084cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:01.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:01.752 7ff9b8543700 1 -- 172.21.2.112:0/2615216378 >> 172.21.2.112:0/2615216378 conn(0x7ff9b00872c0 msgr2=0x7ff9b0095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:01.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:01.752 7ff9b8543700 1 -- 172.21.2.112:0/2615216378 shutdown_connections 2021-02-10T11:35:01.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:01.752 7ff9b8543700 1 -- 172.21.2.112:0/2615216378 wait complete. 2021-02-10T11:35:01.747 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:35:02.177 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.173 7f4a64847700 1 Processor -- start 2021-02-10T11:35:02.177 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.173 7f4a64847700 1 -- start start 2021-02-10T11:35:02.178 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.173 7f4a64847700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f4a6010aff0 0x7f4a600cf750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:02.178 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.173 7f4a64847700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a600fe1c0 con 0x7f4a600d1c70 2021-02-10T11:35:02.178 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.173 7f4a64847700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a601b0cb0 con 0x7f4a600d1870 2021-02-10T11:35:02.178 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.173 7f4a64847700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a60109a60 con 0x7f4a6010aff0 2021-02-10T11:35:02.178 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.174 7f4a5de1b700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f4a600d1c70 0x7f4a600cef60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:50390/0 (socket says 172.21.2.112:50390) 2021-02-10T11:35:02.178 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.174 7f4a5de1b700 1 -- 172.21.2.112:0/723331831 learned_addr learned my addr 172.21.2.112:0/723331831 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:35:02.178 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.174 7f4a5d61a700 1 -- 172.21.2.112:0/723331831 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 113174515 0 0) 0x7f4a50001180 con 0x7f4a600d1c70 2021-02-10T11:35:02.178 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.174 7f4a5d61a700 1 -- 172.21.2.112:0/723331831 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a4c002fe0 con 0x7f4a600d1c70 2021-02-10T11:35:02.178 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.174 7f4a5d61a700 1 -- 172.21.2.112:0/723331831 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 335758170 0 0) 0x7f4a50001180 con 0x7f4a600d1c70 2021-02-10T11:35:02.178 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.174 7f4a5d61a700 1 -- 172.21.2.112:0/723331831 >> v2:172.21.2.120:3300/0 conn(0x7f4a6010aff0 msgr2=0x7f4a600cf750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:02.178 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.174 7f4a5ee1d700 1 --2- 172.21.2.112:0/723331831 >> v2:172.21.2.120:3300/0 conn(0x7f4a6010aff0 0x7f4a600cf750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:02.178 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.174 7f4a5d61a700 1 --2- 172.21.2.112:0/723331831 >> v2:172.21.2.120:3300/0 conn(0x7f4a6010aff0 0x7f4a600cf750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:02.178 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.174 7f4a5d61a700 1 -- 172.21.2.112:0/723331831 >> v1:172.21.2.120:6789/0 conn(0x7f4a600d1870 legacy=0x7f4a601b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:02.179 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.174 7f4a5d61a700 1 -- 172.21.2.112:0/723331831 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a6010b3f0 con 0x7f4a600d1c70 2021-02-10T11:35:02.179 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.174 7f4a5d61a700 1 -- 172.21.2.112:0/723331831 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f4a50001680 con 0x7f4a600d1c70 2021-02-10T11:35:02.179 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.174 7f4a5d61a700 1 -- 172.21.2.112:0/723331831 >> v1:172.21.2.112:6789/0 conn(0x7f4a600d1c70 legacy=0x7f4a600cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:02.179 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.174 7f4a5d61a700 1 --2- 172.21.2.112:0/723331831 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4a4c003af0 0x7f4a4c005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:02.179 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.174 7f4a5d61a700 1 --2- 172.21.2.112:0/723331831 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4a4c006840 0x7f4a4c008d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:02.179 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.174 7f4a5d61a700 1 --2- 172.21.2.112:0/723331831 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4a4c009640 0x7f4a4c00bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:02.179 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.174 7f4a5d61a700 1 -- 172.21.2.112:0/723331831 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a4c00c3b0 con 0x7f4a4c009640 2021-02-10T11:35:02.179 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.174 7f4a5d61a700 1 -- 172.21.2.112:0/723331831 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a4c00c5d0 con 0x7f4a4c003af0 2021-02-10T11:35:02.179 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.174 7f4a5d61a700 1 -- 172.21.2.112:0/723331831 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a4c00c7f0 con 0x7f4a4c006840 2021-02-10T11:35:02.179 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.174 7f4a5ee1d700 1 --2- 172.21.2.112:0/723331831 >> v2:172.21.2.120:3300/0 conn(0x7f4a6010aff0 0x7f4a600cf750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:35:02.179 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.175 7f4a5ee1d700 1 --2- 172.21.2.112:0/723331831 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4a4c009640 0x7f4a4c00bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:02.179 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.175 7f4a5e61c700 1 --2- 172.21.2.112:0/723331831 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4a4c003af0 0x7f4a4c005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:02.179 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.175 7f4a5de1b700 1 --2- 172.21.2.112:0/723331831 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4a4c006840 0x7f4a4c008d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:02.180 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.175 7f4a5ee1d700 1 -- 172.21.2.112:0/723331831 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4a4c006840 msgr2=0x7f4a4c008d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:02.180 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.175 7f4a5ee1d700 1 --2- 172.21.2.112:0/723331831 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4a4c006840 0x7f4a4c008d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:02.180 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.175 7f4a5ee1d700 1 -- 172.21.2.112:0/723331831 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4a4c003af0 msgr2=0x7f4a4c005fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:02.180 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.175 7f4a5ee1d700 1 --2- 172.21.2.112:0/723331831 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4a4c003af0 0x7f4a4c005fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:02.180 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.175 7f4a5ee1d700 1 -- 172.21.2.112:0/723331831 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a4c00ca90 con 0x7f4a4c009640 2021-02-10T11:35:02.180 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.175 7f4a5ee1d700 1 -- 172.21.2.112:0/723331831 reap_dead start 2021-02-10T11:35:02.180 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.175 7f4a5ee1d700 1 --2- 172.21.2.112:0/723331831 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4a4c009640 0x7f4a4c00bb60 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 rx=0x7f4a58006cc0 tx=0x7f4a5804f050).ready entity=mon.0 client_cookie=6d867a45f972e2d3 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:02.180 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.175 7f4a5d61a700 1 -- 172.21.2.112:0/723331831 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4a5804f440 con 0x7f4a4c009640 2021-02-10T11:35:02.180 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.175 7f4a5d61a700 1 -- 172.21.2.112:0/723331831 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a5804f740 con 0x7f4a4c009640 2021-02-10T11:35:02.180 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.175 7f4a5d61a700 1 -- 172.21.2.112:0/723331831 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4a5804fc20 con 0x7f4a4c009640 2021-02-10T11:35:02.180 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.175 7f4a64847700 1 -- 172.21.2.112:0/723331831 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4a4c009640 msgr2=0x7f4a4c00bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:02.180 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.175 7f4a64847700 1 --2- 172.21.2.112:0/723331831 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4a4c009640 0x7f4a4c00bb60 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 rx=0x7f4a58006cc0 tx=0x7f4a5804f050).stop 2021-02-10T11:35:02.180 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.176 7f4a64847700 1 -- 172.21.2.112:0/723331831 shutdown_connections 2021-02-10T11:35:02.181 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.176 7f4a64847700 1 --2- 172.21.2.112:0/723331831 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4a4c009640 0x7f4a4c00bb60 unknown :-1 s=CLOSED pgs=1439 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:02.181 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.176 7f4a64847700 1 -- 172.21.2.112:0/723331831 >> 172.21.2.112:0/723331831 conn(0x7f4a600ce420 msgr2=0x7f4a600d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:02.181 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.176 7f4a64847700 1 -- 172.21.2.112:0/723331831 shutdown_connections 2021-02-10T11:35:02.181 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.176 7f4a64847700 1 -- 172.21.2.112:0/723331831 wait complete. 2021-02-10T11:35:02.181 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.191 7f4a64847700 1 Processor -- start 2021-02-10T11:35:02.181 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.191 7f4a64847700 1 -- start start 2021-02-10T11:35:02.182 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.191 7f4a64847700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4a60085bc0 0x7f4a600863e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:02.182 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.191 7f4a64847700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4a600954c0 0x7f4a60096300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:02.182 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.191 7f4a64847700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4a600958c0 0x7f4a60096920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:02.182 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.191 7f4a64847700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a60095d80 con 0x7f4a600958c0 2021-02-10T11:35:02.182 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.191 7f4a64847700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a600bfa60 con 0x7f4a60085bc0 2021-02-10T11:35:02.183 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.191 7f4a64847700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a60085fc0 con 0x7f4a600954c0 2021-02-10T11:35:02.183 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.191 7f4a5ee1d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4a600958c0 0x7f4a60096920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:02.183 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.191 7f4a5ee1d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4a600958c0 0x7f4a60096920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:50944/0 (socket says 172.21.2.112:50944) 2021-02-10T11:35:02.183 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.191 7f4a5ee1d700 1 -- 172.21.2.112:0/4027422502 learned_addr learned my addr 172.21.2.112:0/4027422502 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:35:02.183 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.191 7f4a5e61c700 1 --2- 172.21.2.112:0/4027422502 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4a60085bc0 0x7f4a600863e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:02.183 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.191 7f4a5de1b700 1 --2- 172.21.2.112:0/4027422502 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4a600954c0 0x7f4a60096300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:02.183 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.192 7f4a5ee1d700 1 -- 172.21.2.112:0/4027422502 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4a600954c0 msgr2=0x7f4a60096300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:02.183 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.192 7f4a5ee1d700 1 --2- 172.21.2.112:0/4027422502 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4a600954c0 0x7f4a60096300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:02.183 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.192 7f4a5ee1d700 1 -- 172.21.2.112:0/4027422502 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4a60085bc0 msgr2=0x7f4a600863e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:02.183 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.192 7f4a5ee1d700 1 --2- 172.21.2.112:0/4027422502 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4a60085bc0 0x7f4a600863e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:02.183 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.192 7f4a5ee1d700 1 -- 172.21.2.112:0/4027422502 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a60086c10 con 0x7f4a600958c0 2021-02-10T11:35:02.183 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.192 7f4a5ee1d700 1 --2- 172.21.2.112:0/4027422502 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4a600958c0 0x7f4a60096920 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 rx=0x7f4a580508f0 tx=0x7f4a58050c90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:02.183 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.192 7f4a56ffd700 1 -- 172.21.2.112:0/4027422502 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4a58050dd0 con 0x7f4a600958c0 2021-02-10T11:35:02.184 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.192 7f4a56ffd700 1 -- 172.21.2.112:0/4027422502 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a58006030 con 0x7f4a600958c0 2021-02-10T11:35:02.184 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.192 7f4a56ffd700 1 -- 172.21.2.112:0/4027422502 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4a580064c0 con 0x7f4a600958c0 2021-02-10T11:35:02.184 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.192 7f4a64847700 1 -- 172.21.2.112:0/4027422502 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a60086c10 con 0x7f4a600958c0 2021-02-10T11:35:02.184 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.192 7f4a64847700 1 -- 172.21.2.112:0/4027422502 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a60097230 con 0x7f4a600958c0 2021-02-10T11:35:02.184 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.193 7f4a56ffd700 1 -- 172.21.2.112:0/4027422502 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4a58006930 con 0x7f4a600958c0 2021-02-10T11:35:02.184 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.193 7f4a56ffd700 1 --2- 172.21.2.112:0/4027422502 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4a440186c0 0x7f4a4401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:02.184 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.193 7f4a56ffd700 1 -- 172.21.2.112:0/4027422502 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f4a58076950 con 0x7f4a600958c0 2021-02-10T11:35:02.184 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.193 7f4a5e61c700 1 --2- 172.21.2.112:0/4027422502 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4a440186c0 0x7f4a4401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:02.184 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.194 7f4a5e61c700 1 --2- 172.21.2.112:0/4027422502 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4a440186c0 0x7f4a4401abc0 crc :-1 s=READY pgs=701 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:02.210 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.224 7f4a64847700 1 -- 172.21.2.112:0/4027422502 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a600fe1c0 con 0x7f4a600958c0 2021-02-10T11:35:02.213 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.226 7f4a56ffd700 1 -- 172.21.2.112:0/4027422502 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4a5804d5c0 con 0x7f4a600958c0 2021-02-10T11:35:02.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.353 7f4a64847700 1 -- 172.21.2.112:0/4027422502 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f4a600fe1c0 con 0x7f4a600958c0 2021-02-10T11:35:02.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.354 7f4a56ffd700 1 -- 172.21.2.112:0/4027422502 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v64) v1 ==== 100+0+10 (secure 0 0 0) 0x7f4a58006930 con 0x7f4a600958c0 2021-02-10T11:35:02.341 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:35:02.342 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.356 7f4a64847700 1 -- 172.21.2.112:0/4027422502 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4a440186c0 msgr2=0x7f4a4401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:02.342 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.356 7f4a64847700 1 --2- 172.21.2.112:0/4027422502 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4a440186c0 0x7f4a4401abc0 crc :-1 s=READY pgs=701 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:02.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.356 7f4a64847700 1 -- 172.21.2.112:0/4027422502 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4a600958c0 msgr2=0x7f4a60096920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:02.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.356 7f4a64847700 1 --2- 172.21.2.112:0/4027422502 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4a600958c0 0x7f4a60096920 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 rx=0x7f4a580508f0 tx=0x7f4a58050c90).stop 2021-02-10T11:35:02.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.356 7f4a64847700 1 -- 172.21.2.112:0/4027422502 shutdown_connections 2021-02-10T11:35:02.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.356 7f4a64847700 1 --2- 172.21.2.112:0/4027422502 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4a440186c0 0x7f4a4401abc0 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:02.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.356 7f4a64847700 1 --2- 172.21.2.112:0/4027422502 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4a600958c0 0x7f4a60096920 unknown :-1 s=CLOSED pgs=1440 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:02.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.356 7f4a64847700 1 --2- 172.21.2.112:0/4027422502 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4a600954c0 0x7f4a60096300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:02.344 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.356 7f4a64847700 1 --2- 172.21.2.112:0/4027422502 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4a60085bc0 0x7f4a600863e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:02.344 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.356 7f4a64847700 1 -- 172.21.2.112:0/4027422502 >> 172.21.2.112:0/4027422502 conn(0x7f4a600bef00 msgr2=0x7f4a6009d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:02.344 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.356 7f4a64847700 1 -- 172.21.2.112:0/4027422502 shutdown_connections 2021-02-10T11:35:02.344 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.356 7f4a64847700 1 -- 172.21.2.112:0/4027422502 wait complete. 2021-02-10T11:35:02.351 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:35:02.351 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:35:02.777 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.774 7fa9ca08f700 1 Processor -- start 2021-02-10T11:35:02.777 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.774 7fa9ca08f700 1 -- start start 2021-02-10T11:35:02.777 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.774 7fa9ca08f700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fa9c40d0340 0x7fa9c40ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:02.777 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.774 7fa9ca08f700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fa9c40d1c70 0x7fa9c40efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:02.777 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.774 7fa9ca08f700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9c40f0b10 con 0x7fa9c40d1870 2021-02-10T11:35:02.777 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.774 7fa9ca08f700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9c40e5ed0 con 0x7fa9c40d1c70 2021-02-10T11:35:02.778 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.774 7fa9ca08f700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9c41b0be0 con 0x7fa9c40d0340 2021-02-10T11:35:02.778 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.775 7fa9c3fff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fa9c40d1c70 0x7fa9c40efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:02.778 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.775 7fa9c3fff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fa9c40d1c70 0x7fa9c40efc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:50952/0 (socket says 172.21.2.112:50952) 2021-02-10T11:35:02.778 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.775 7fa9c3fff700 1 -- 172.21.2.112:0/667573268 learned_addr learned my addr 172.21.2.112:0/667573268 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:35:02.778 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.775 7fa9c3fff700 1 -- 172.21.2.112:0/667573268 >> v2:172.21.2.120:3300/0 conn(0x7fa9c40d0340 msgr2=0x7fa9c40ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:02.778 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.775 7fa9c3fff700 1 --2- 172.21.2.112:0/667573268 >> v2:172.21.2.120:3300/0 conn(0x7fa9c40d0340 0x7fa9c40ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:02.778 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.775 7fa9c3fff700 1 -- 172.21.2.112:0/667573268 >> v1:172.21.2.112:6789/0 conn(0x7fa9c40d1870 legacy=0x7fa9c41b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:02.778 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.775 7fa9c3fff700 1 -- 172.21.2.112:0/667573268 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9c40f20a0 con 0x7fa9c40d1c70 2021-02-10T11:35:02.778 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.775 7fa9c3fff700 1 --2- 172.21.2.112:0/667573268 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa9c40d1c70 0x7fa9c40efc60 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 rx=0x7fa9bc006f00 tx=0x7fa9bc009f10).ready entity=mon.0 client_cookie=97677b8e3826098a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:02.778 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.775 7fa9c27fc700 1 -- 172.21.2.112:0/667573268 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa9bc00b3b0 con 0x7fa9c40d1c70 2021-02-10T11:35:02.778 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.775 7fa9c27fc700 1 -- 172.21.2.112:0/667573268 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9bc000d30 con 0x7fa9c40d1c70 2021-02-10T11:35:02.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.775 7fa9c27fc700 1 -- 172.21.2.112:0/667573268 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa9bc00b970 con 0x7fa9c40d1c70 2021-02-10T11:35:02.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.775 7fa9ca08f700 1 -- 172.21.2.112:0/667573268 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa9c40d1c70 msgr2=0x7fa9c40efc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:02.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.775 7fa9ca08f700 1 --2- 172.21.2.112:0/667573268 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa9c40d1c70 0x7fa9c40efc60 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 rx=0x7fa9bc006f00 tx=0x7fa9bc009f10).stop 2021-02-10T11:35:02.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.776 7fa9ca08f700 1 -- 172.21.2.112:0/667573268 shutdown_connections 2021-02-10T11:35:02.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.776 7fa9ca08f700 1 --2- 172.21.2.112:0/667573268 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa9c40d1c70 0x7fa9c40efc60 unknown :-1 s=CLOSED pgs=1442 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:02.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.776 7fa9ca08f700 1 --2- 172.21.2.112:0/667573268 >> v2:172.21.2.120:3300/0 conn(0x7fa9c40d0340 0x7fa9c40ef640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:02.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.776 7fa9ca08f700 1 -- 172.21.2.112:0/667573268 >> 172.21.2.112:0/667573268 conn(0x7fa9c40ce450 msgr2=0x7fa9c40d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:02.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.776 7fa9ca08f700 1 -- 172.21.2.112:0/667573268 shutdown_connections 2021-02-10T11:35:02.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.776 7fa9ca08f700 1 -- 172.21.2.112:0/667573268 wait complete. 2021-02-10T11:35:02.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.791 7fa9ca08f700 1 Processor -- start 2021-02-10T11:35:02.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.791 7fa9ca08f700 1 -- start start 2021-02-10T11:35:02.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.791 7fa9ca08f700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa9c418e810 0x7fa9c40ef200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:02.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.791 7fa9ca08f700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa9c40ef820 0x7fa9c418d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:02.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.791 7fa9ca08f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa9c418ec10 0x7fa9c418ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:02.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.791 7fa9ca08f700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9c419e010 con 0x7fa9c418ec10 2021-02-10T11:35:02.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.791 7fa9ca08f700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9c40ac6c0 con 0x7fa9c418e810 2021-02-10T11:35:02.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.791 7fa9ca08f700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9c418e3f0 con 0x7fa9c40ef820 2021-02-10T11:35:02.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.791 7fa9c3fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa9c418ec10 0x7fa9c418ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:02.782 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.791 7fa9c3fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa9c418ec10 0x7fa9c418ddd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:50956/0 (socket says 172.21.2.112:50956) 2021-02-10T11:35:02.782 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.791 7fa9c3fff700 1 -- 172.21.2.112:0/3698609661 learned_addr learned my addr 172.21.2.112:0/3698609661 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:35:02.782 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.792 7fa9c3fff700 1 -- 172.21.2.112:0/3698609661 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa9c40ef820 msgr2=0x7fa9c418d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:02.782 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.792 7fa9c3fff700 1 --2- 172.21.2.112:0/3698609661 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa9c40ef820 0x7fa9c418d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:02.782 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.792 7fa9c3fff700 1 -- 172.21.2.112:0/3698609661 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa9c418e810 msgr2=0x7fa9c40ef200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:02.782 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.792 7fa9c3fff700 1 --2- 172.21.2.112:0/3698609661 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa9c418e810 0x7fa9c40ef200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:02.782 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.792 7fa9c3fff700 1 -- 172.21.2.112:0/3698609661 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9c4095fe0 con 0x7fa9c418ec10 2021-02-10T11:35:02.782 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.792 7fa9c3fff700 1 --2- 172.21.2.112:0/3698609661 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa9c418ec10 0x7fa9c418ddd0 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 rx=0x7fa9bc0099c0 tx=0x7fa9bc00bc90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:02.782 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.792 7fa9bbfff700 1 -- 172.21.2.112:0/3698609661 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa9bc00bdd0 con 0x7fa9c418ec10 2021-02-10T11:35:02.782 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.792 7fa9bbfff700 1 -- 172.21.2.112:0/3698609661 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9bc006cc0 con 0x7fa9c418ec10 2021-02-10T11:35:02.782 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.792 7fa9bbfff700 1 -- 172.21.2.112:0/3698609661 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa9bc0014f0 con 0x7fa9c418ec10 2021-02-10T11:35:02.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.792 7fa9ca08f700 1 -- 172.21.2.112:0/3698609661 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9c4095fe0 con 0x7fa9c418ec10 2021-02-10T11:35:02.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.792 7fa9ca08f700 1 -- 172.21.2.112:0/3698609661 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9c4096570 con 0x7fa9c418ec10 2021-02-10T11:35:02.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.793 7fa9bbfff700 1 -- 172.21.2.112:0/3698609661 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa9bc001270 con 0x7fa9c418ec10 2021-02-10T11:35:02.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.793 7fa9bbfff700 1 --2- 172.21.2.112:0/3698609661 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa9a80186c0 0x7fa9a801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:02.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.793 7fa9bbfff700 1 -- 172.21.2.112:0/3698609661 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fa9bc032920 con 0x7fa9c418ec10 2021-02-10T11:35:02.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.793 7fa9c37fe700 1 --2- 172.21.2.112:0/3698609661 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa9a80186c0 0x7fa9a801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:02.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.794 7fa9c37fe700 1 --2- 172.21.2.112:0/3698609661 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa9a80186c0 0x7fa9a801abc0 crc :-1 s=READY pgs=702 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:02.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.824 7fa9ca08f700 1 -- 172.21.2.112:0/3698609661 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9c40ee0e0 con 0x7fa9c418ec10 2021-02-10T11:35:02.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.827 7fa9bbfff700 1 -- 172.21.2.112:0/3698609661 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa9bc02ffc0 con 0x7fa9c418ec10 2021-02-10T11:35:02.943 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.956 7fa9ca08f700 1 -- 172.21.2.112:0/3698609661 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa9c40ce450 con 0x7fa9c418ec10 2021-02-10T11:35:02.943 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.956 7fa9bbfff700 1 -- 172.21.2.112:0/3698609661 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) v1 ==== 94+0+2101 (secure 0 0 0) 0x7fa9bc02ffc0 con 0x7fa9c418ec10 2021-02-10T11:35:02.944 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:35:02.944 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":47,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8334,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/3370618110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":3370618110},{"type":"v1","addr":"172.21.2.112:6833","nonce":3370618110}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":44}],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:34:45.582177","modified":"2021-02-10 11:34:47.589305","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5921},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5921":{"gid":5921,"name":"b","rank":0,"incarnation":45,"state":"up:active","state_seq":3,"addr":"172.21.2.120:6835/2436135065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":2436135065},{"type":"v1","addr":"172.21.2.120:6835","nonce":2436135065}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2021-02-10T11:35:02.944 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 47 2021-02-10T11:35:02.945 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.958 7fa9ca08f700 1 -- 172.21.2.112:0/3698609661 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa9a80186c0 msgr2=0x7fa9a801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:02.946 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.958 7fa9ca08f700 1 --2- 172.21.2.112:0/3698609661 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa9a80186c0 0x7fa9a801abc0 crc :-1 s=READY pgs=702 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:02.946 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.958 7fa9ca08f700 1 -- 172.21.2.112:0/3698609661 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa9c418ec10 msgr2=0x7fa9c418ddd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:02.946 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.958 7fa9ca08f700 1 --2- 172.21.2.112:0/3698609661 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa9c418ec10 0x7fa9c418ddd0 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 rx=0x7fa9bc0099c0 tx=0x7fa9bc00bc90).stop 2021-02-10T11:35:02.946 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.958 7fa9ca08f700 1 -- 172.21.2.112:0/3698609661 shutdown_connections 2021-02-10T11:35:02.946 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.958 7fa9ca08f700 1 --2- 172.21.2.112:0/3698609661 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa9a80186c0 0x7fa9a801abc0 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:02.946 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.958 7fa9ca08f700 1 --2- 172.21.2.112:0/3698609661 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa9c418ec10 0x7fa9c418ddd0 unknown :-1 s=CLOSED pgs=1443 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:02.946 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.958 7fa9ca08f700 1 --2- 172.21.2.112:0/3698609661 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa9c40ef820 0x7fa9c418d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:02.946 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.958 7fa9ca08f700 1 --2- 172.21.2.112:0/3698609661 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa9c418e810 0x7fa9c40ef200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:02.946 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.958 7fa9ca08f700 1 -- 172.21.2.112:0/3698609661 >> 172.21.2.112:0/3698609661 conn(0x7fa9c40efcf0 msgr2=0x7fa9c40f1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:02.946 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.959 7fa9ca08f700 1 -- 172.21.2.112:0/3698609661 shutdown_connections 2021-02-10T11:35:02.946 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:02.959 7fa9ca08f700 1 -- 172.21.2.112:0/3698609661 wait complete. 2021-02-10T11:35:03.155 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 47, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:34:45.582177', 'modified': '2021-02-10 11:34:47.589305', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5921}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5921': {'gid': 5921, 'name': 'b', 'rank': 0, 'incarnation': 45, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.120:6835/2436135065', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.120:6834', 'nonce': 2436135065}, {'type': 'v1', 'addr': '172.21.2.120:6835', 'nonce': 2436135065}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:35:03.155 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:35:03.156 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:35:03.554 INFO:teuthology.orchestra.run.gibba020.stdout:{ 2021-02-10T11:35:03.554 INFO:teuthology.orchestra.run.gibba020.stdout: "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", 2021-02-10T11:35:03.554 INFO:teuthology.orchestra.run.gibba020.stdout: "whoami": 0, 2021-02-10T11:35:03.554 INFO:teuthology.orchestra.run.gibba020.stdout: "id": 5921, 2021-02-10T11:35:03.554 INFO:teuthology.orchestra.run.gibba020.stdout: "want_state": "up:active", 2021-02-10T11:35:03.554 INFO:teuthology.orchestra.run.gibba020.stdout: "state": "up:active", 2021-02-10T11:35:03.554 INFO:teuthology.orchestra.run.gibba020.stdout: "rank_uptime": 17.981782813999999, 2021-02-10T11:35:03.555 INFO:teuthology.orchestra.run.gibba020.stdout: "mdsmap_epoch": 47, 2021-02-10T11:35:03.555 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch": 64, 2021-02-10T11:35:03.555 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch_barrier": 64, 2021-02-10T11:35:03.555 INFO:teuthology.orchestra.run.gibba020.stdout: "uptime": 23.625658427000001 2021-02-10T11:35:03.555 INFO:teuthology.orchestra.run.gibba020.stdout:} 2021-02-10T11:35:03.558 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", "whoami": 0, "id": 5921, "want_state": "up:active", "state": "up:active", "rank_uptime": 17.981782813999999, "mdsmap_epoch": 47, "osdmap_epoch": 64, "osdmap_epoch_barrier": 64, "uptime": 23.625658427000001 } 2021-02-10T11:35:03.558 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:35:03.558 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2021-02-10T11:35:03.558 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.1 at ubuntu@gibba012.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2021-02-10T11:35:03.558 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:03.559 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:35:03.894 DEBUG:teuthology.orchestra.run.gibba012:> sudo modprobe fuse 2021-02-10T11:35:04.239 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:35:04.239 DEBUG:teuthology.orchestra.run.gibba012:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:35:04.583 INFO:teuthology.orchestra.run.gibba012.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:35:04.583 INFO:teuthology.orchestra.run.gibba012.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:35:04.584 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:35:04.584 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:04.584 DEBUG:teuthology.orchestra.run.gibba012:> ls /sys/fs/fuse/connections 2021-02-10T11:35:04.919 INFO:teuthology.orchestra.run.gibba012.stdout:42 2021-02-10T11:35:04.919 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [42] 2021-02-10T11:35:04.920 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.1 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:35:04.922 DEBUG:teuthology.orchestra.run.gibba012:> sudo modprobe fuse 2021-02-10T11:35:05.267 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:35:05.267 DEBUG:teuthology.orchestra.run.gibba012:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:35:05.333 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba012.stderr:ceph-fuse[135769]: starting ceph client 2021-02-10T11:35:05.333 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba012.stderr:2021-02-10 11:35:05.346 7f20cb373f80 -1 init, newargv = 0x5614ce7fd1a0 newargc=9 2021-02-10T11:35:05.338 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba012.stderr:ceph-fuse[135769]: starting fuse 2021-02-10T11:35:05.608 INFO:teuthology.orchestra.run.gibba012.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:35:05.609 INFO:teuthology.orchestra.run.gibba012.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:35:05.609 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:35:05.610 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:05.610 DEBUG:teuthology.orchestra.run.gibba012:> ls /sys/fs/fuse/connections 2021-02-10T11:35:05.944 INFO:teuthology.orchestra.run.gibba012.stdout:42 2021-02-10T11:35:05.944 INFO:teuthology.orchestra.run.gibba012.stdout:43 2021-02-10T11:35:05.944 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42, 43] 2021-02-10T11:35:05.945 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:05.945 DEBUG:teuthology.orchestra.run.gibba012:> sudo python3 -c ' 2021-02-10T11:35:05.945 DEBUG:teuthology.orchestra.run.gibba012:> import glob 2021-02-10T11:35:05.945 DEBUG:teuthology.orchestra.run.gibba012:> import re 2021-02-10T11:35:05.945 DEBUG:teuthology.orchestra.run.gibba012:> import os 2021-02-10T11:35:05.945 DEBUG:teuthology.orchestra.run.gibba012:> import subprocess 2021-02-10T11:35:05.945 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:35:05.945 DEBUG:teuthology.orchestra.run.gibba012:> def find_socket(client_name): 2021-02-10T11:35:05.945 DEBUG:teuthology.orchestra.run.gibba012:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2021-02-10T11:35:05.945 DEBUG:teuthology.orchestra.run.gibba012:> files = glob.glob(asok_path) 2021-02-10T11:35:05.945 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:35:05.945 DEBUG:teuthology.orchestra.run.gibba012:> # Given a non-glob path, it better be there 2021-02-10T11:35:05.945 DEBUG:teuthology.orchestra.run.gibba012:> if "*" not in asok_path: 2021-02-10T11:35:05.946 DEBUG:teuthology.orchestra.run.gibba012:> assert(len(files) == 1) 2021-02-10T11:35:05.946 DEBUG:teuthology.orchestra.run.gibba012:> return files[0] 2021-02-10T11:35:05.946 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:35:05.946 DEBUG:teuthology.orchestra.run.gibba012:> for f in files: 2021-02-10T11:35:05.946 DEBUG:teuthology.orchestra.run.gibba012:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:35:05.946 DEBUG:teuthology.orchestra.run.gibba012:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:35:05.946 DEBUG:teuthology.orchestra.run.gibba012:> return f 2021-02-10T11:35:05.946 DEBUG:teuthology.orchestra.run.gibba012:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:35:05.946 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:35:05.946 DEBUG:teuthology.orchestra.run.gibba012:> print(find_socket("client.1")) 2021-02-10T11:35:05.946 DEBUG:teuthology.orchestra.run.gibba012:> ' 2021-02-10T11:35:06.307 INFO:teuthology.orchestra.run.gibba012.stdout:/var/run/ceph/ceph-client.1.135769.asok 2021-02-10T11:35:06.310 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.135769.asok 2021-02-10T11:35:06.310 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:06.310 DEBUG:teuthology.orchestra.run.gibba012:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.135769.asok status 2021-02-10T11:35:06.698 INFO:teuthology.orchestra.run.gibba012.stdout:{ 2021-02-10T11:35:06.699 INFO:teuthology.orchestra.run.gibba012.stdout: "metadata": { 2021-02-10T11:35:06.699 INFO:teuthology.orchestra.run.gibba012.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:35:06.699 INFO:teuthology.orchestra.run.gibba012.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:35:06.699 INFO:teuthology.orchestra.run.gibba012.stdout: "entity_id": "1", 2021-02-10T11:35:06.699 INFO:teuthology.orchestra.run.gibba012.stdout: "hostname": "gibba012", 2021-02-10T11:35:06.699 INFO:teuthology.orchestra.run.gibba012.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2021-02-10T11:35:06.699 INFO:teuthology.orchestra.run.gibba012.stdout: "pid": "135769", 2021-02-10T11:35:06.699 INFO:teuthology.orchestra.run.gibba012.stdout: "root": "/" 2021-02-10T11:35:06.699 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:06.699 INFO:teuthology.orchestra.run.gibba012.stdout: "dentry_count": 0, 2021-02-10T11:35:06.699 INFO:teuthology.orchestra.run.gibba012.stdout: "dentry_pinned_count": 0, 2021-02-10T11:35:06.699 INFO:teuthology.orchestra.run.gibba012.stdout: "id": 6937, 2021-02-10T11:35:06.700 INFO:teuthology.orchestra.run.gibba012.stdout: "inst": { 2021-02-10T11:35:06.700 INFO:teuthology.orchestra.run.gibba012.stdout: "name": { 2021-02-10T11:35:06.700 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "client", 2021-02-10T11:35:06.700 INFO:teuthology.orchestra.run.gibba012.stdout: "num": 6937 2021-02-10T11:35:06.700 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:06.700 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": { 2021-02-10T11:35:06.700 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:35:06.700 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:0", 2021-02-10T11:35:06.700 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 2204985408 2021-02-10T11:35:06.700 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:35:06.700 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:06.700 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": { 2021-02-10T11:35:06.700 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:35:06.701 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:0", 2021-02-10T11:35:06.701 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 2204985408 2021-02-10T11:35:06.701 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:06.701 INFO:teuthology.orchestra.run.gibba012.stdout: "inst_str": "client.6937 172.21.2.112:0/2204985408", 2021-02-10T11:35:06.701 INFO:teuthology.orchestra.run.gibba012.stdout: "addr_str": "172.21.2.112:0/2204985408", 2021-02-10T11:35:06.701 INFO:teuthology.orchestra.run.gibba012.stdout: "inode_count": 1, 2021-02-10T11:35:06.701 INFO:teuthology.orchestra.run.gibba012.stdout: "mds_epoch": 47, 2021-02-10T11:35:06.701 INFO:teuthology.orchestra.run.gibba012.stdout: "osd_epoch": 64, 2021-02-10T11:35:06.701 INFO:teuthology.orchestra.run.gibba012.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:35:06.701 INFO:teuthology.orchestra.run.gibba012.stdout: "blacklisted": false 2021-02-10T11:35:06.701 INFO:teuthology.orchestra.run.gibba012.stdout:} 2021-02-10T11:35:06.703 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:06.703 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:35:06.703 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:35:07.038 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:35:07.038 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:35:07.038 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:07.038 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:35:07.383 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2021-02-10T11:35:07.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:07.809 7fa3e3781700 1 Processor -- start 2021-02-10T11:35:07.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:07.809 7fa3e3781700 1 -- start start 2021-02-10T11:35:07.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:07.810 7fa3e3781700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fa3dc0d1880 0x7fa3dc1b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:07.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:07.810 7fa3e3781700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3dc0fe1d0 con 0x7fa3dc0cff80 2021-02-10T11:35:07.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:07.810 7fa3e3781700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3dc109a70 con 0x7fa3dc1b0af0 2021-02-10T11:35:07.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:07.810 7fa3e3781700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3dc088070 con 0x7fa3dc0d1880 2021-02-10T11:35:07.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:07.810 7fa3e159d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fa3dc0d1880 0x7fa3dc1b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:07.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:07.810 7fa3e159d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fa3dc0d1880 0x7fa3dc1b04d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:50976/0 (socket says 172.21.2.112:50976) 2021-02-10T11:35:07.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:07.810 7fa3e159d700 1 -- 172.21.2.112:0/2415810930 learned_addr learned my addr 172.21.2.112:0/2415810930 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:35:07.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:07.810 7fa3e159d700 1 -- 172.21.2.112:0/2415810930 >> v1:172.21.2.120:6789/0 conn(0x7fa3dc1b0af0 legacy=0x7fa3dc108950 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:35:07.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:07.810 7fa3e159d700 1 -- 172.21.2.112:0/2415810930 >> v1:172.21.2.112:6789/0 conn(0x7fa3dc0cff80 legacy=0x7fa3dc109140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:07.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:07.810 7fa3e159d700 1 -- 172.21.2.112:0/2415810930 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3dc088260 con 0x7fa3dc0d1880 2021-02-10T11:35:07.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:07.811 7fa3e159d700 1 --2- 172.21.2.112:0/2415810930 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa3dc0d1880 0x7fa3dc1b04d0 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 rx=0x7fa3d0006700 tx=0x7fa3d0009b80).ready entity=mon.0 client_cookie=75c6aa9cb4bd6970 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:07.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:07.811 7fa3cffff700 1 -- 172.21.2.112:0/2415810930 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa3d000c0c0 con 0x7fa3dc0d1880 2021-02-10T11:35:07.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:07.811 7fa3cffff700 1 -- 172.21.2.112:0/2415810930 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3d000c440 con 0x7fa3dc0d1880 2021-02-10T11:35:07.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:07.811 7fa3cffff700 1 -- 172.21.2.112:0/2415810930 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa3d000c960 con 0x7fa3dc0d1880 2021-02-10T11:35:07.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:07.811 7fa3e3781700 1 -- 172.21.2.112:0/2415810930 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa3dc0d1880 msgr2=0x7fa3dc1b04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:07.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:07.811 7fa3e3781700 1 --2- 172.21.2.112:0/2415810930 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa3dc0d1880 0x7fa3dc1b04d0 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 rx=0x7fa3d0006700 tx=0x7fa3d0009b80).stop 2021-02-10T11:35:07.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:07.812 7fa3e3781700 1 -- 172.21.2.112:0/2415810930 shutdown_connections 2021-02-10T11:35:07.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:07.812 7fa3e3781700 1 --2- 172.21.2.112:0/2415810930 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa3dc0d1880 0x7fa3dc1b04d0 unknown :-1 s=CLOSED pgs=1445 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:07.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:07.812 7fa3e3781700 1 -- 172.21.2.112:0/2415810930 >> 172.21.2.112:0/2415810930 conn(0x7fa3dc0ce430 msgr2=0x7fa3dc0d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:07.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:07.812 7fa3e3781700 1 -- 172.21.2.112:0/2415810930 shutdown_connections 2021-02-10T11:35:07.815 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:07.812 7fa3e3781700 1 -- 172.21.2.112:0/2415810930 wait complete. 2021-02-10T11:35:07.816 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:07.826 7fa3e3781700 1 Processor -- start 2021-02-10T11:35:07.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:07.826 7fa3e3781700 1 -- start start 2021-02-10T11:35:07.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:07.827 7fa3e3781700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa3dc09fbb0 0x7fa3dc1b5a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:07.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:07.827 7fa3e3781700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa3dc09ffb0 0x7fa3dc08a930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:07.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:07.827 7fa3e3781700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa3dc08af50 0x7fa3dc09c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:07.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:07.827 7fa3e3781700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3dc09f6b0 con 0x7fa3dc08af50 2021-02-10T11:35:07.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:07.827 7fa3e3781700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3dc08b390 con 0x7fa3dc09fbb0 2021-02-10T11:35:07.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:07.827 7fa3e3781700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3dc1b6060 con 0x7fa3dc09ffb0 2021-02-10T11:35:07.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:07.827 7fa3e1d9e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa3dc08af50 0x7fa3dc09c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:07.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:07.827 7fa3e1d9e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa3dc08af50 0x7fa3dc09c410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:50982/0 (socket says 172.21.2.112:50982) 2021-02-10T11:35:07.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:07.827 7fa3e1d9e700 1 -- 172.21.2.112:0/1221659735 learned_addr learned my addr 172.21.2.112:0/1221659735 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:35:07.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:07.827 7fa3e159d700 1 --2- 172.21.2.112:0/1221659735 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa3dc09fbb0 0x7fa3dc1b5a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:07.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:07.827 7fa3e0d9c700 1 --2- 172.21.2.112:0/1221659735 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa3dc09ffb0 0x7fa3dc08a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:07.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:07.828 7fa3e1d9e700 1 -- 172.21.2.112:0/1221659735 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa3dc09ffb0 msgr2=0x7fa3dc08a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:07.818 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:07.828 7fa3e1d9e700 1 --2- 172.21.2.112:0/1221659735 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa3dc09ffb0 0x7fa3dc08a930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:07.818 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:07.828 7fa3e1d9e700 1 -- 172.21.2.112:0/1221659735 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa3dc09fbb0 msgr2=0x7fa3dc1b5a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:07.818 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:07.828 7fa3e1d9e700 1 --2- 172.21.2.112:0/1221659735 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa3dc09fbb0 0x7fa3dc1b5a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:07.818 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:07.828 7fa3e1d9e700 1 -- 172.21.2.112:0/1221659735 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3dc09ca30 con 0x7fa3dc08af50 2021-02-10T11:35:07.818 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:07.828 7fa3e159d700 1 --2- 172.21.2.112:0/1221659735 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa3dc09fbb0 0x7fa3dc1b5a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:35:07.818 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:07.828 7fa3e1d9e700 1 --2- 172.21.2.112:0/1221659735 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa3dc08af50 0x7fa3dc09c410 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 rx=0x7fa3d4009980 tx=0x7fa3d400bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:07.818 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:07.828 7fa3cdffb700 1 -- 172.21.2.112:0/1221659735 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa3d4005870 con 0x7fa3dc08af50 2021-02-10T11:35:07.818 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:07.828 7fa3cdffb700 1 -- 172.21.2.112:0/1221659735 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3d4005a70 con 0x7fa3dc08af50 2021-02-10T11:35:07.818 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:07.828 7fa3cdffb700 1 -- 172.21.2.112:0/1221659735 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa3d4006030 con 0x7fa3dc08af50 2021-02-10T11:35:07.818 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:07.828 7fa3e3781700 1 -- 172.21.2.112:0/1221659735 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3dc09ca30 con 0x7fa3dc08af50 2021-02-10T11:35:07.818 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:07.828 7fa3e3781700 1 -- 172.21.2.112:0/1221659735 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3dc09ca30 con 0x7fa3dc08af50 2021-02-10T11:35:07.818 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:07.829 7fa3cdffb700 1 -- 172.21.2.112:0/1221659735 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa3d4005c50 con 0x7fa3dc08af50 2021-02-10T11:35:07.819 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:07.829 7fa3cdffb700 1 --2- 172.21.2.112:0/1221659735 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa3bc0186c0 0x7fa3bc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:07.819 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:07.829 7fa3cdffb700 1 -- 172.21.2.112:0/1221659735 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fa3d4032950 con 0x7fa3dc08af50 2021-02-10T11:35:07.819 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:07.829 7fa3e159d700 1 --2- 172.21.2.112:0/1221659735 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa3bc0186c0 0x7fa3bc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:07.819 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:07.830 7fa3e159d700 1 --2- 172.21.2.112:0/1221659735 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa3bc0186c0 0x7fa3bc01abc0 crc :-1 s=READY pgs=703 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:07.847 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:07.860 7fa3e3781700 1 -- 172.21.2.112:0/1221659735 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3dc09d2b0 con 0x7fa3dc08af50 2021-02-10T11:35:07.849 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:07.863 7fa3cdffb700 1 -- 172.21.2.112:0/1221659735 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa3d402cfa0 con 0x7fa3dc08af50 2021-02-10T11:35:07.985 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:07.999 7fa3e3781700 1 -- 172.21.2.112:0/1221659735 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fa3dc0ce430 con 0x7fa3dc08af50 2021-02-10T11:35:08.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:08.052 7fa3cdffb700 1 -- 172.21.2.112:0/1221659735 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v48) v1 ==== 136+0+0 (secure 0 0 0) 0x7fa3d402cfa0 con 0x7fa3dc08af50 2021-02-10T11:35:08.050 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:08.064 7fa3e3781700 1 -- 172.21.2.112:0/1221659735 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa3bc0186c0 msgr2=0x7fa3bc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:08.051 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:08.064 7fa3e3781700 1 --2- 172.21.2.112:0/1221659735 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa3bc0186c0 0x7fa3bc01abc0 crc :-1 s=READY pgs=703 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:08.051 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:08.064 7fa3e3781700 1 -- 172.21.2.112:0/1221659735 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa3dc08af50 msgr2=0x7fa3dc09c410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:08.051 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:08.064 7fa3e3781700 1 --2- 172.21.2.112:0/1221659735 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa3dc08af50 0x7fa3dc09c410 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 rx=0x7fa3d4009980 tx=0x7fa3d400bcd0).stop 2021-02-10T11:35:08.051 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:08.064 7fa3e3781700 1 -- 172.21.2.112:0/1221659735 shutdown_connections 2021-02-10T11:35:08.051 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:08.064 7fa3e3781700 1 --2- 172.21.2.112:0/1221659735 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa3bc0186c0 0x7fa3bc01abc0 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:08.051 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:08.064 7fa3e3781700 1 --2- 172.21.2.112:0/1221659735 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa3dc08af50 0x7fa3dc09c410 unknown :-1 s=CLOSED pgs=1446 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:08.052 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:08.064 7fa3e3781700 1 --2- 172.21.2.112:0/1221659735 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa3dc09ffb0 0x7fa3dc08a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:08.052 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:08.064 7fa3e3781700 1 --2- 172.21.2.112:0/1221659735 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa3dc09fbb0 0x7fa3dc1b5a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:08.052 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:08.064 7fa3e3781700 1 -- 172.21.2.112:0/1221659735 >> 172.21.2.112:0/1221659735 conn(0x7fa3dc09bad0 msgr2=0x7fa3dc0804e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:08.052 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:08.064 7fa3e3781700 1 -- 172.21.2.112:0/1221659735 shutdown_connections 2021-02-10T11:35:08.052 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:08.064 7fa3e3781700 1 -- 172.21.2.112:0/1221659735 wait complete. 2021-02-10T11:35:08.059 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs volume ls 2021-02-10T11:35:08.486 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:08.482 7fd698eaf700 1 Processor -- start 2021-02-10T11:35:08.486 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:08.482 7fd698eaf700 1 -- start start 2021-02-10T11:35:08.486 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:08.483 7fd698eaf700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fd6940d1c70 0x7fd6940f20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:08.486 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:08.483 7fd698eaf700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6940f0b10 con 0x7fd6940d1870 2021-02-10T11:35:08.487 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:08.483 7fd698eaf700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6941b0be0 con 0x7fd6940d0340 2021-02-10T11:35:08.487 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:08.483 7fd698eaf700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6940e5ed0 con 0x7fd6940d1c70 2021-02-10T11:35:08.487 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:08.483 7fd693314700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fd6940d1c70 0x7fd6940f20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:08.487 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:08.483 7fd693314700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fd6940d1c70 0x7fd6940f20a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:50990/0 (socket says 172.21.2.112:50990) 2021-02-10T11:35:08.487 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:08.483 7fd693314700 1 -- 172.21.2.112:0/4162582708 learned_addr learned my addr 172.21.2.112:0/4162582708 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:35:08.487 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:08.484 7fd693314700 1 -- 172.21.2.112:0/4162582708 >> v1:172.21.2.120:6789/0 conn(0x7fd6940d0340 legacy=0x7fd6940ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:08.487 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:08.484 7fd693314700 1 -- 172.21.2.112:0/4162582708 >> v1:172.21.2.112:6789/0 conn(0x7fd6940d1870 legacy=0x7fd6941b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:08.487 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:08.484 7fd693314700 1 -- 172.21.2.112:0/4162582708 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6940efe30 con 0x7fd6940d1c70 2021-02-10T11:35:08.487 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:08.484 7fd693314700 1 --2- 172.21.2.112:0/4162582708 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd6940d1c70 0x7fd6940f20a0 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 rx=0x7fd68c006f00 tx=0x7fd68c009e90).ready entity=mon.0 client_cookie=c1c33d5b6ad8a48b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:08.487 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:08.484 7fd691b11700 1 -- 172.21.2.112:0/4162582708 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd68c00b300 con 0x7fd6940d1c70 2021-02-10T11:35:08.487 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:08.484 7fd691b11700 1 -- 172.21.2.112:0/4162582708 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd68c00b640 con 0x7fd6940d1c70 2021-02-10T11:35:08.488 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:08.484 7fd691b11700 1 -- 172.21.2.112:0/4162582708 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd68c00bb60 con 0x7fd6940d1c70 2021-02-10T11:35:08.488 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:08.484 7fd698eaf700 1 -- 172.21.2.112:0/4162582708 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd6940d1c70 msgr2=0x7fd6940f20a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:08.488 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:08.484 7fd698eaf700 1 --2- 172.21.2.112:0/4162582708 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd6940d1c70 0x7fd6940f20a0 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 rx=0x7fd68c006f00 tx=0x7fd68c009e90).stop 2021-02-10T11:35:08.488 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:08.485 7fd698eaf700 1 -- 172.21.2.112:0/4162582708 shutdown_connections 2021-02-10T11:35:08.488 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:08.485 7fd698eaf700 1 --2- 172.21.2.112:0/4162582708 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd6940d1c70 0x7fd6940f20a0 unknown :-1 s=CLOSED pgs=1448 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:08.488 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:08.485 7fd698eaf700 1 -- 172.21.2.112:0/4162582708 >> 172.21.2.112:0/4162582708 conn(0x7fd6940ce450 msgr2=0x7fd6940d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:08.488 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:08.485 7fd698eaf700 1 -- 172.21.2.112:0/4162582708 shutdown_connections 2021-02-10T11:35:08.488 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:08.485 7fd698eaf700 1 -- 172.21.2.112:0/4162582708 wait complete. 2021-02-10T11:35:08.488 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:08.499 7fd698eaf700 1 Processor -- start 2021-02-10T11:35:08.488 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:08.499 7fd698eaf700 1 -- start start 2021-02-10T11:35:08.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:08.500 7fd698eaf700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd6940ba6c0 0x7fd69409ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:08.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:08.500 7fd698eaf700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd6940baac0 0x7fd69409f340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:08.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:08.500 7fd698eaf700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd69409f960 0x7fd6940970a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:08.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:08.500 7fd698eaf700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd694096b20 con 0x7fd69409f960 2021-02-10T11:35:08.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:08.500 7fd698eaf700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd694096710 con 0x7fd6940ba6c0 2021-02-10T11:35:08.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:08.500 7fd698eaf700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd694096930 con 0x7fd6940baac0 2021-02-10T11:35:08.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:08.500 7fd693314700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd69409f960 0x7fd6940970a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:08.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:08.500 7fd693314700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd69409f960 0x7fd6940970a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:50996/0 (socket says 172.21.2.112:50996) 2021-02-10T11:35:08.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:08.500 7fd693314700 1 -- 172.21.2.112:0/1242723810 learned_addr learned my addr 172.21.2.112:0/1242723810 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:35:08.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:08.500 7fd693314700 1 -- 172.21.2.112:0/1242723810 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd6940baac0 msgr2=0x7fd69409f340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:08.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:08.500 7fd693314700 1 --2- 172.21.2.112:0/1242723810 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd6940baac0 0x7fd69409f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:08.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:08.500 7fd693314700 1 -- 172.21.2.112:0/1242723810 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd6940ba6c0 msgr2=0x7fd69409ed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:08.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:08.500 7fd693314700 1 --2- 172.21.2.112:0/1242723810 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd6940ba6c0 0x7fd69409ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:08.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:08.500 7fd693314700 1 -- 172.21.2.112:0/1242723810 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6940976c0 con 0x7fd69409f960 2021-02-10T11:35:08.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:08.501 7fd693314700 1 --2- 172.21.2.112:0/1242723810 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd69409f960 0x7fd6940970a0 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 rx=0x7fd68c00b940 tx=0x7fd68c00bce0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:08.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:08.501 7fd68b7fe700 1 -- 172.21.2.112:0/1242723810 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd68c00be20 con 0x7fd69409f960 2021-02-10T11:35:08.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:08.501 7fd68b7fe700 1 -- 172.21.2.112:0/1242723810 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd68c003030 con 0x7fd69409f960 2021-02-10T11:35:08.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:08.501 7fd68b7fe700 1 -- 172.21.2.112:0/1242723810 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd68c001460 con 0x7fd69409f960 2021-02-10T11:35:08.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:08.501 7fd698eaf700 1 -- 172.21.2.112:0/1242723810 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6940976c0 con 0x7fd69409f960 2021-02-10T11:35:08.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:08.501 7fd698eaf700 1 -- 172.21.2.112:0/1242723810 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd694097c50 con 0x7fd69409f960 2021-02-10T11:35:08.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:08.502 7fd68b7fe700 1 -- 172.21.2.112:0/1242723810 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd68c005820 con 0x7fd69409f960 2021-02-10T11:35:08.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:08.502 7fd68b7fe700 1 --2- 172.21.2.112:0/1242723810 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd6780186c0 0x7fd67801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:08.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:08.502 7fd68b7fe700 1 -- 172.21.2.112:0/1242723810 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fd68c032930 con 0x7fd69409f960 2021-02-10T11:35:08.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:08.502 7fd692b13700 1 --2- 172.21.2.112:0/1242723810 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd6780186c0 0x7fd67801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:08.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:08.502 7fd692b13700 1 --2- 172.21.2.112:0/1242723810 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd6780186c0 0x7fd67801abc0 crc :-1 s=READY pgs=704 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:08.520 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:08.533 7fd698eaf700 1 -- 172.21.2.112:0/1242723810 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd694098420 con 0x7fd69409f960 2021-02-10T11:35:08.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:08.536 7fd68b7fe700 1 -- 172.21.2.112:0/1242723810 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd68c025f00 con 0x7fd69409f960 2021-02-10T11:35:08.658 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:08.671 7fd698eaf700 1 -- 172.21.2.112:0/1242723810 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs volume ls", "target": ["mgr", ""]}) v1 -- 0x7fd694097c50 con 0x7fd6780186c0 2021-02-10T11:35:08.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:08.672 7fd68b7fe700 1 -- 172.21.2.112:0/1242723810 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+41 (crc 0 0 0) 0x7fd684006a30 con 0x7fd6780186c0 2021-02-10T11:35:08.661 INFO:teuthology.orchestra.run.gibba012.stdout:[ 2021-02-10T11:35:08.661 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:08.661 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "cephfs" 2021-02-10T11:35:08.661 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:35:08.661 INFO:teuthology.orchestra.run.gibba012.stdout:] 2021-02-10T11:35:08.663 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:08.676 7fd698eaf700 1 -- 172.21.2.112:0/1242723810 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd6780186c0 msgr2=0x7fd67801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:08.663 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:08.676 7fd698eaf700 1 --2- 172.21.2.112:0/1242723810 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd6780186c0 0x7fd67801abc0 crc :-1 s=READY pgs=704 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:08.663 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:08.676 7fd698eaf700 1 -- 172.21.2.112:0/1242723810 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd69409f960 msgr2=0x7fd6940970a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:08.663 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:08.676 7fd698eaf700 1 --2- 172.21.2.112:0/1242723810 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd69409f960 0x7fd6940970a0 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 rx=0x7fd68c00b940 tx=0x7fd68c00bce0).stop 2021-02-10T11:35:08.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:08.677 7fd698eaf700 1 -- 172.21.2.112:0/1242723810 shutdown_connections 2021-02-10T11:35:08.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:08.677 7fd698eaf700 1 --2- 172.21.2.112:0/1242723810 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd6780186c0 0x7fd67801abc0 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:08.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:08.677 7fd698eaf700 1 --2- 172.21.2.112:0/1242723810 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd69409f960 0x7fd6940970a0 unknown :-1 s=CLOSED pgs=1449 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:08.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:08.677 7fd698eaf700 1 --2- 172.21.2.112:0/1242723810 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd6940baac0 0x7fd69409f340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:08.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:08.677 7fd698eaf700 1 --2- 172.21.2.112:0/1242723810 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd6940ba6c0 0x7fd69409ed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:08.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:08.677 7fd698eaf700 1 -- 172.21.2.112:0/1242723810 >> 172.21.2.112:0/1242723810 conn(0x7fd694099fd0 msgr2=0x7fd6940f23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:08.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:08.677 7fd698eaf700 1 -- 172.21.2.112:0/1242723810 shutdown_connections 2021-02-10T11:35:08.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:08.677 7fd698eaf700 1 -- 172.21.2.112:0/1242723810 wait complete. 2021-02-10T11:35:08.671 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:35:09.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:09.099 7f9822f22700 1 Processor -- start 2021-02-10T11:35:09.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:09.099 7f9822f22700 1 -- start start 2021-02-10T11:35:09.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:09.100 7f9822f22700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f981c0d1c70 0x7f981c0cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:09.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:09.100 7f9822f22700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f981c0cf580 0x7f981c0cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:09.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:09.100 7f9822f22700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f981c0fe1c0 con 0x7f981c0d1870 2021-02-10T11:35:09.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:09.100 7f9822f22700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f981c1b0cb0 con 0x7f981c0d1c70 2021-02-10T11:35:09.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:09.100 7f9822f22700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f981c109a60 con 0x7f981c0cf580 2021-02-10T11:35:09.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:09.100 7f981bfff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f981c0d1c70 0x7f981c0cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:09.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:09.100 7f981bfff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f981c0d1c70 0x7f981c0cef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:51000/0 (socket says 172.21.2.112:51000) 2021-02-10T11:35:09.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:09.100 7f981bfff700 1 -- 172.21.2.112:0/147619473 learned_addr learned my addr 172.21.2.112:0/147619473 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:35:09.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:09.100 7f982153f700 1 --2- 172.21.2.112:0/147619473 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f981c0cf580 0x7f981c0cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:09.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:09.100 7f981bfff700 1 -- 172.21.2.112:0/147619473 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f981c0cf580 msgr2=0x7f981c0cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:09.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:09.100 7f981bfff700 1 --2- 172.21.2.112:0/147619473 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f981c0cf580 0x7f981c0cf980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:09.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:09.100 7f981bfff700 1 -- 172.21.2.112:0/147619473 >> v1:172.21.2.120:6789/0 conn(0x7f981c0d1870 legacy=0x7f981c1b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:09.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:09.100 7f981bfff700 1 -- 172.21.2.112:0/147619473 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f981c088070 con 0x7f981c0d1c70 2021-02-10T11:35:09.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:09.100 7f982153f700 1 --2- 172.21.2.112:0/147619473 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f981c0cf580 0x7f981c0cf980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:35:09.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:09.101 7f981bfff700 1 --2- 172.21.2.112:0/147619473 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f981c0d1c70 0x7f981c0cef60 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 rx=0x7f980c006a80 tx=0x7f980c009f00).ready entity=mon.0 client_cookie=9766a06147bea787 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:09.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:09.101 7f981b7fe700 1 -- 172.21.2.112:0/147619473 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f980c00b5c0 con 0x7f981c0d1c70 2021-02-10T11:35:09.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:09.101 7f981b7fe700 1 -- 172.21.2.112:0/147619473 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f980c00b800 con 0x7f981c0d1c70 2021-02-10T11:35:09.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:09.101 7f981b7fe700 1 -- 172.21.2.112:0/147619473 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f980c00bd20 con 0x7f981c0d1c70 2021-02-10T11:35:09.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:09.101 7f9822f22700 1 -- 172.21.2.112:0/147619473 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f981c0d1c70 msgr2=0x7f981c0cef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:09.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:09.101 7f9822f22700 1 --2- 172.21.2.112:0/147619473 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f981c0d1c70 0x7f981c0cef60 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 rx=0x7f980c006a80 tx=0x7f980c009f00).stop 2021-02-10T11:35:09.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:09.101 7f9822f22700 1 -- 172.21.2.112:0/147619473 shutdown_connections 2021-02-10T11:35:09.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:09.101 7f9822f22700 1 --2- 172.21.2.112:0/147619473 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f981c0d1c70 0x7f981c0cef60 unknown :-1 s=CLOSED pgs=1450 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:09.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:09.101 7f9822f22700 1 --2- 172.21.2.112:0/147619473 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f981c0cf580 0x7f981c0cf980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:09.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:09.101 7f9822f22700 1 -- 172.21.2.112:0/147619473 >> 172.21.2.112:0/147619473 conn(0x7f981c0ce420 msgr2=0x7f981c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:09.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:09.102 7f9822f22700 1 -- 172.21.2.112:0/147619473 shutdown_connections 2021-02-10T11:35:09.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:09.102 7f9822f22700 1 -- 172.21.2.112:0/147619473 wait complete. 2021-02-10T11:35:09.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:09.116 7f9822f22700 1 Processor -- start 2021-02-10T11:35:09.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:09.116 7f9822f22700 1 -- start start 2021-02-10T11:35:09.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:09.117 7f9822f22700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f981c0c3820 0x7f981c0a7950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:09.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:09.117 7f9822f22700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f981c1a44c0 0x7f981c1a48c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:09.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:09.117 7f9822f22700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f981c0863d0 0x7f981c0867d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:09.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:09.117 7f9822f22700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f981c085ed0 con 0x7f981c1a44c0 2021-02-10T11:35:09.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:09.117 7f9822f22700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f981c0c0740 con 0x7f981c0863d0 2021-02-10T11:35:09.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:09.117 7f9822f22700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f981c086df0 con 0x7f981c0c3820 2021-02-10T11:35:09.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:09.117 7f981bfff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f981c1a44c0 0x7f981c1a48c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:09.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:09.117 7f981bfff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f981c1a44c0 0x7f981c1a48c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:51010/0 (socket says 172.21.2.112:51010) 2021-02-10T11:35:09.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:09.117 7f981bfff700 1 -- 172.21.2.112:0/1461146594 learned_addr learned my addr 172.21.2.112:0/1461146594 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:35:09.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:09.117 7f981bfff700 1 -- 172.21.2.112:0/1461146594 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f981c0c3820 msgr2=0x7f981c0a7950 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:35:09.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:09.117 7f981bfff700 1 --2- 172.21.2.112:0/1461146594 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f981c0c3820 0x7f981c0a7950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:09.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:09.117 7f981bfff700 1 -- 172.21.2.112:0/1461146594 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f981c0863d0 msgr2=0x7f981c0867d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:09.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:09.117 7f981bfff700 1 --2- 172.21.2.112:0/1461146594 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f981c0863d0 0x7f981c0867d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:09.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:09.117 7f981bfff700 1 -- 172.21.2.112:0/1461146594 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f981c087010 con 0x7f981c1a44c0 2021-02-10T11:35:09.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:09.117 7f981bfff700 1 --2- 172.21.2.112:0/1461146594 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f981c1a44c0 0x7f981c1a48c0 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 rx=0x7f980c00b970 tx=0x7f980c00bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:09.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:09.118 7f98197fa700 1 -- 172.21.2.112:0/1461146594 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f980c000d30 con 0x7f981c1a44c0 2021-02-10T11:35:09.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:09.118 7f98197fa700 1 -- 172.21.2.112:0/1461146594 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f980c00c2f0 con 0x7f981c1a44c0 2021-02-10T11:35:09.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:09.118 7f9822f22700 1 -- 172.21.2.112:0/1461146594 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f981c087010 con 0x7f981c1a44c0 2021-02-10T11:35:09.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:09.118 7f9822f22700 1 -- 172.21.2.112:0/1461146594 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f981c083000 con 0x7f981c1a44c0 2021-02-10T11:35:09.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:09.118 7f98197fa700 1 -- 172.21.2.112:0/1461146594 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f980c00c810 con 0x7f981c1a44c0 2021-02-10T11:35:09.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:09.118 7f98197fa700 1 -- 172.21.2.112:0/1461146594 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f980c001010 con 0x7f981c1a44c0 2021-02-10T11:35:09.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:09.118 7f98197fa700 1 --2- 172.21.2.112:0/1461146594 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f98000186c0 0x7f980001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:09.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:09.118 7f98197fa700 1 -- 172.21.2.112:0/1461146594 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f980c032980 con 0x7f981c1a44c0 2021-02-10T11:35:09.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:09.119 7f9820d3e700 1 --2- 172.21.2.112:0/1461146594 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f98000186c0 0x7f980001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:09.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:09.119 7f9820d3e700 1 --2- 172.21.2.112:0/1461146594 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f98000186c0 0x7f980001abc0 crc :-1 s=READY pgs=705 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:09.136 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:09.149 7f9822f22700 1 -- 172.21.2.112:0/1461146594 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f981c085e50 con 0x7f981c1a44c0 2021-02-10T11:35:09.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:09.152 7f98197fa700 1 -- 172.21.2.112:0/1461146594 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f980c000d30 con 0x7f981c1a44c0 2021-02-10T11:35:09.287 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:09.300 7f9822f22700 1 -- 172.21.2.112:0/1461146594 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f981c085e50 con 0x7f981c1a44c0 2021-02-10T11:35:09.287 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:09.300 7f98197fa700 1 -- 172.21.2.112:0/1461146594 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v14)=0 v14) v1 ==== 122+0+0 (secure 0 0 0) 0x7f980c001010 con 0x7f981c1a44c0 2021-02-10T11:35:09.289 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:09.302 7f9822f22700 1 -- 172.21.2.112:0/1461146594 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f98000186c0 msgr2=0x7f980001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:09.289 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:09.302 7f9822f22700 1 --2- 172.21.2.112:0/1461146594 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f98000186c0 0x7f980001abc0 crc :-1 s=READY pgs=705 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:09.289 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:09.302 7f9822f22700 1 -- 172.21.2.112:0/1461146594 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f981c1a44c0 msgr2=0x7f981c1a48c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:09.289 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:09.302 7f9822f22700 1 --2- 172.21.2.112:0/1461146594 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f981c1a44c0 0x7f981c1a48c0 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 rx=0x7f980c00b970 tx=0x7f980c00bd10).stop 2021-02-10T11:35:09.290 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:09.303 7f9822f22700 1 -- 172.21.2.112:0/1461146594 shutdown_connections 2021-02-10T11:35:09.290 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:09.303 7f9822f22700 1 --2- 172.21.2.112:0/1461146594 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f98000186c0 0x7f980001abc0 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:09.290 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:09.303 7f9822f22700 1 --2- 172.21.2.112:0/1461146594 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f981c0863d0 0x7f981c0867d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:09.290 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:09.303 7f9822f22700 1 --2- 172.21.2.112:0/1461146594 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f981c1a44c0 0x7f981c1a48c0 unknown :-1 s=CLOSED pgs=1451 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:09.290 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:09.303 7f9822f22700 1 --2- 172.21.2.112:0/1461146594 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f981c0c3820 0x7f981c0a7950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:09.290 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:09.303 7f9822f22700 1 -- 172.21.2.112:0/1461146594 >> 172.21.2.112:0/1461146594 conn(0x7f981c0872c0 msgr2=0x7f981c095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:09.290 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:09.303 7f9822f22700 1 -- 172.21.2.112:0/1461146594 shutdown_connections 2021-02-10T11:35:09.291 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:09.303 7f9822f22700 1 -- 172.21.2.112:0/1461146594 wait complete. 2021-02-10T11:35:09.298 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:09.298 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:35:09.298 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:35:09.634 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:35:09.634 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:35:09.635 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba012.front.sepia.ceph.com... 2021-02-10T11:35:09.635 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:35:09.635 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:35:09.986 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba012.stderr:ceph-fuse[135769]: fuse finished with error 0 and tester_r 0 2021-02-10T11:35:09.987 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:09.987 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:35:09.987 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:35:10.320 INFO:teuthology.orchestra.run.gibba012.stdout:ext2/ext3 2021-02-10T11:35:10.320 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:35:10.320 INFO:teuthology.orchestra.run:waiting for 900 2021-02-10T11:35:10.320 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:35:10.321 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:35:10.656 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolumegroup create cephfs group_0000000000485992 2021-02-10T11:35:11.088 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.084 7f1f63dc5700 1 Processor -- start 2021-02-10T11:35:11.088 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.084 7f1f63dc5700 1 -- start start 2021-02-10T11:35:11.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.085 7f1f63dc5700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f1f5c0d1870 0x7f1f5c1b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:11.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.085 7f1f63dc5700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1f5c0cff70 0x7f1f5c109130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:11.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.085 7f1f63dc5700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f5c109a60 con 0x7f1f5c1b0ae0 2021-02-10T11:35:11.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.085 7f1f63dc5700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f5c109750 con 0x7f1f5c0d1870 2021-02-10T11:35:11.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.085 7f1f63dc5700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f5c088070 con 0x7f1f5c0cff70 2021-02-10T11:35:11.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.085 7f1f613e0700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f1f5c1b0ae0 0x7f1f5c108940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:50476/0 (socket says 172.21.2.112:50476) 2021-02-10T11:35:11.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.085 7f1f613e0700 1 -- 172.21.2.112:0/7275354 learned_addr learned my addr 172.21.2.112:0/7275354 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:35:11.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.085 7f1f61be1700 1 --2- 172.21.2.112:0/7275354 >> v2:172.21.2.120:3300/0 conn(0x7f1f5c0d1870 0x7f1f5c1b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:11.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.085 7f1f623e2700 1 --2- 172.21.2.112:0/7275354 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1f5c0cff70 0x7f1f5c109130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:11.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.085 7f1f60bdf700 1 -- 172.21.2.112:0/7275354 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3800895487 0 0) 0x7f1f4c0009b0 con 0x7f1f5c1b0ae0 2021-02-10T11:35:11.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.085 7f1f60bdf700 1 -- 172.21.2.112:0/7275354 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f48002fe0 con 0x7f1f5c1b0ae0 2021-02-10T11:35:11.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.085 7f1f61be1700 1 -- 172.21.2.112:0/7275354 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1f5c0cff70 msgr2=0x7f1f5c109130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:11.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.085 7f1f61be1700 1 --2- 172.21.2.112:0/7275354 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1f5c0cff70 0x7f1f5c109130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:11.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.085 7f1f61be1700 1 -- 172.21.2.112:0/7275354 >> v1:172.21.2.112:6789/0 conn(0x7f1f5c1b0ae0 legacy=0x7f1f5c108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:11.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.086 7f1f61be1700 1 -- 172.21.2.112:0/7275354 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f5c0d1c70 con 0x7f1f5c0d1870 2021-02-10T11:35:11.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.086 7f1f623e2700 1 --2- 172.21.2.112:0/7275354 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1f5c0cff70 0x7f1f5c109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:35:11.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.086 7f1f61be1700 1 --2- 172.21.2.112:0/7275354 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1f5c0d1870 0x7f1f5c1b04c0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 rx=0x7f1f58006650 tx=0x7f1f58009a80).ready entity=mon.1 client_cookie=2a8b61d99f5d444f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:11.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.086 7f1f60bdf700 1 -- 172.21.2.112:0/7275354 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1f580009b0 con 0x7f1f5c0d1870 2021-02-10T11:35:11.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.086 7f1f60bdf700 1 -- 172.21.2.112:0/7275354 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f5800c0c0 con 0x7f1f5c0d1870 2021-02-10T11:35:11.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.086 7f1f60bdf700 1 -- 172.21.2.112:0/7275354 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1f5800c5e0 con 0x7f1f5c0d1870 2021-02-10T11:35:11.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.086 7f1f63dc5700 1 -- 172.21.2.112:0/7275354 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1f5c0d1870 msgr2=0x7f1f5c1b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:11.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.086 7f1f63dc5700 1 --2- 172.21.2.112:0/7275354 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1f5c0d1870 0x7f1f5c1b04c0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 rx=0x7f1f58006650 tx=0x7f1f58009a80).stop 2021-02-10T11:35:11.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.087 7f1f63dc5700 1 -- 172.21.2.112:0/7275354 shutdown_connections 2021-02-10T11:35:11.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.087 7f1f63dc5700 1 --2- 172.21.2.112:0/7275354 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1f5c0cff70 0x7f1f5c109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:11.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.087 7f1f63dc5700 1 --2- 172.21.2.112:0/7275354 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1f5c0d1870 0x7f1f5c1b04c0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:11.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.087 7f1f63dc5700 1 -- 172.21.2.112:0/7275354 >> 172.21.2.112:0/7275354 conn(0x7f1f5c0ce420 msgr2=0x7f1f5c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:11.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.087 7f1f63dc5700 1 -- 172.21.2.112:0/7275354 shutdown_connections 2021-02-10T11:35:11.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.087 7f1f63dc5700 1 -- 172.21.2.112:0/7275354 wait complete. 2021-02-10T11:35:11.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.101 7f1f63dc5700 1 Processor -- start 2021-02-10T11:35:11.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.101 7f1f63dc5700 1 -- start start 2021-02-10T11:35:11.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.102 7f1f63dc5700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1f5c1a79a0 0x7f1f5c1a7da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:11.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.102 7f1f63dc5700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1f5c0bdb40 0x7f1f5c0bdf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:11.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.102 7f1f63dc5700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1f5c082ca0 0x7f1f5c0830a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:11.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.102 7f1f63dc5700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f5c19d990 con 0x7f1f5c0bdb40 2021-02-10T11:35:11.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.102 7f1f63dc5700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f5c19dbb0 con 0x7f1f5c1a79a0 2021-02-10T11:35:11.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.102 7f1f63dc5700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f5c0836c0 con 0x7f1f5c082ca0 2021-02-10T11:35:11.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.102 7f1f613e0700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1f5c0bdb40 0x7f1f5c0bdf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:11.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.102 7f1f613e0700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1f5c0bdb40 0x7f1f5c0bdf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:51020/0 (socket says 172.21.2.112:51020) 2021-02-10T11:35:11.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.102 7f1f613e0700 1 -- 172.21.2.112:0/3633633127 learned_addr learned my addr 172.21.2.112:0/3633633127 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:35:11.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.102 7f1f613e0700 1 -- 172.21.2.112:0/3633633127 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1f5c082ca0 msgr2=0x7f1f5c0830a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:11.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.102 7f1f613e0700 1 --2- 172.21.2.112:0/3633633127 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1f5c082ca0 0x7f1f5c0830a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:11.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.103 7f1f613e0700 1 -- 172.21.2.112:0/3633633127 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1f5c1a79a0 msgr2=0x7f1f5c1a7da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:11.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.103 7f1f613e0700 1 --2- 172.21.2.112:0/3633633127 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1f5c1a79a0 0x7f1f5c1a7da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:11.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.103 7f1f613e0700 1 -- 172.21.2.112:0/3633633127 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f5c0838e0 con 0x7f1f5c0bdb40 2021-02-10T11:35:11.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.103 7f1f613e0700 1 --2- 172.21.2.112:0/3633633127 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1f5c0bdb40 0x7f1f5c0bdf40 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 rx=0x7f1f4c007650 tx=0x7f1f4c00a8e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:11.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.103 7f1f527fc700 1 -- 172.21.2.112:0/3633633127 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1f4c00acd0 con 0x7f1f5c0bdb40 2021-02-10T11:35:11.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.103 7f1f527fc700 1 -- 172.21.2.112:0/3633633127 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f4c004350 con 0x7f1f5c0bdb40 2021-02-10T11:35:11.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.103 7f1f63dc5700 1 -- 172.21.2.112:0/3633633127 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f5c0838e0 con 0x7f1f5c0bdb40 2021-02-10T11:35:11.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.103 7f1f527fc700 1 -- 172.21.2.112:0/3633633127 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1f4c004870 con 0x7f1f5c0bdb40 2021-02-10T11:35:11.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.103 7f1f63dc5700 1 -- 172.21.2.112:0/3633633127 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f5c0838e0 con 0x7f1f5c0bdb40 2021-02-10T11:35:11.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.104 7f1f527fc700 1 -- 172.21.2.112:0/3633633127 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1f4c000cc0 con 0x7f1f5c0bdb40 2021-02-10T11:35:11.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.104 7f1f527fc700 1 --2- 172.21.2.112:0/3633633127 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1f44018730 0x7f1f4401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:11.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.104 7f1f527fc700 1 -- 172.21.2.112:0/3633633127 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f1f4c032600 con 0x7f1f5c0bdb40 2021-02-10T11:35:11.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.104 7f1f61be1700 1 --2- 172.21.2.112:0/3633633127 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1f44018730 0x7f1f4401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:11.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.105 7f1f61be1700 1 --2- 172.21.2.112:0/3633633127 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1f44018730 0x7f1f4401ac10 crc :-1 s=READY pgs=706 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:11.121 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.135 7f1f63dc5700 1 -- 172.21.2.112:0/3633633127 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f5c082a70 con 0x7f1f5c0bdb40 2021-02-10T11:35:11.124 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.137 7f1f527fc700 1 -- 172.21.2.112:0/3633633127 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1f4c01f4c0 con 0x7f1f5c0bdb40 2021-02-10T11:35:11.252 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.265 7f1f63dc5700 1 -- 172.21.2.112:0/3633633127 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "target": ["mgr", ""], "group_name": "group_0000000000485992"}) v1 -- 0x7f1f5c082a70 con 0x7f1f44018730 2021-02-10T11:35:11.254 INFO:tasks.ceph.mgr.x.gibba020.stderr:2021-02-10 11:35:11.268 7f6382eee700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:35:11.254 INFO:tasks.ceph.mgr.x.gibba020.stderr:2021-02-10 11:35:11.268 7f6382eee700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:35:11.255 INFO:tasks.ceph.mgr.x.gibba020.stderr:2021-02-10 11:35:11.268 7f6382eee700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:35:11.255 INFO:tasks.ceph.mgr.x.gibba020.stderr:2021-02-10 11:35:11.268 7f6382eee700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:35:11.255 INFO:tasks.ceph.mgr.x.gibba020.stderr:2021-02-10 11:35:11.268 7f6382eee700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:35:11.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.282 7f1f527fc700 1 -- 172.21.2.112:0/3633633127 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f1f58000940 con 0x7f1f44018730 2021-02-10T11:35:11.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.298 7f1f63dc5700 1 -- 172.21.2.112:0/3633633127 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1f44018730 msgr2=0x7f1f4401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:11.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.298 7f1f63dc5700 1 --2- 172.21.2.112:0/3633633127 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1f44018730 0x7f1f4401ac10 crc :-1 s=READY pgs=706 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:11.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.298 7f1f63dc5700 1 -- 172.21.2.112:0/3633633127 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1f5c0bdb40 msgr2=0x7f1f5c0bdf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:11.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.298 7f1f63dc5700 1 --2- 172.21.2.112:0/3633633127 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1f5c0bdb40 0x7f1f5c0bdf40 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 rx=0x7f1f4c007650 tx=0x7f1f4c00a8e0).stop 2021-02-10T11:35:11.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.298 7f1f63dc5700 1 -- 172.21.2.112:0/3633633127 shutdown_connections 2021-02-10T11:35:11.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.298 7f1f63dc5700 1 --2- 172.21.2.112:0/3633633127 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1f44018730 0x7f1f4401ac10 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:11.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.298 7f1f63dc5700 1 --2- 172.21.2.112:0/3633633127 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1f5c082ca0 0x7f1f5c0830a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:11.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.298 7f1f63dc5700 1 --2- 172.21.2.112:0/3633633127 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1f5c0bdb40 0x7f1f5c0bdf40 unknown :-1 s=CLOSED pgs=1453 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:11.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.298 7f1f63dc5700 1 --2- 172.21.2.112:0/3633633127 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1f5c1a79a0 0x7f1f5c1a7da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:11.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.298 7f1f63dc5700 1 -- 172.21.2.112:0/3633633127 >> 172.21.2.112:0/3633633127 conn(0x7f1f5c0bef00 msgr2=0x7f1f5c09f200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:11.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.298 7f1f63dc5700 1 -- 172.21.2.112:0/3633633127 shutdown_connections 2021-02-10T11:35:11.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.298 7f1f63dc5700 1 -- 172.21.2.112:0/3633633127 wait complete. 2021-02-10T11:35:11.293 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000513667 --group_name group_0000000000485992 2021-02-10T11:35:11.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.719 7fa5c840b700 1 Processor -- start 2021-02-10T11:35:11.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.719 7fa5c840b700 1 -- start start 2021-02-10T11:35:11.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.720 7fa5c840b700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fa5c00d1880 0x7fa5c01b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:11.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.720 7fa5c840b700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fa5c01b0af0 0x7fa5c0108950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:11.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.720 7fa5c840b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa5c00d1c80 0x7fa5c0108f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:11.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.720 7fa5c840b700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5c00fe1d0 con 0x7fa5c00d1880 2021-02-10T11:35:11.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.720 7fa5c840b700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5c0109a70 con 0x7fa5c01b0af0 2021-02-10T11:35:11.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.720 7fa5c840b700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5c0088070 con 0x7fa5c00d1c80 2021-02-10T11:35:11.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.720 7fa5c6227700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fa5c00d1880 0x7fa5c01b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:11.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.720 7fa5c6a28700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa5c00d1c80 0x7fa5c0108f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:11.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.720 7fa5c6227700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fa5c00d1880 0x7fa5c01b04d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:51028/0 (socket says 172.21.2.112:51028) 2021-02-10T11:35:11.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.720 7fa5c6227700 1 -- 172.21.2.112:0/2174043665 learned_addr learned my addr 172.21.2.112:0/2174043665 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:35:11.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.720 7fa5c5a26700 1 --2- 172.21.2.112:0/2174043665 >> v2:172.21.2.120:3300/0 conn(0x7fa5c01b0af0 0x7fa5c0108950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:11.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.720 7fa5c6a28700 1 -- 172.21.2.112:0/2174043665 >> v2:172.21.2.120:3300/0 conn(0x7fa5c01b0af0 msgr2=0x7fa5c0108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:11.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.720 7fa5c6a28700 1 --2- 172.21.2.112:0/2174043665 >> v2:172.21.2.120:3300/0 conn(0x7fa5c01b0af0 0x7fa5c0108950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:11.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.720 7fa5c6a28700 1 -- 172.21.2.112:0/2174043665 >> v2:172.21.2.112:3300/0 conn(0x7fa5c00d1880 msgr2=0x7fa5c01b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:11.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.720 7fa5c6a28700 1 --2- 172.21.2.112:0/2174043665 >> v2:172.21.2.112:3300/0 conn(0x7fa5c00d1880 0x7fa5c01b04d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:11.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.720 7fa5c6a28700 1 -- 172.21.2.112:0/2174043665 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5c0109590 con 0x7fa5c00d1c80 2021-02-10T11:35:11.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.720 7fa5c6a28700 1 --2- 172.21.2.112:0/2174043665 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa5c00d1c80 0x7fa5c0108f70 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7fa5b8006e50 tx=0x7fa5b8009e70).ready entity=mon.2 client_cookie=376e4559735d9487 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:11.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.721 7fa5c5225700 1 -- 172.21.2.112:0/2174043665 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa5b800b510 con 0x7fa5c00d1c80 2021-02-10T11:35:11.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.721 7fa5c5225700 1 -- 172.21.2.112:0/2174043665 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5b8006790 con 0x7fa5c00d1c80 2021-02-10T11:35:11.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.721 7fa5c5225700 1 -- 172.21.2.112:0/2174043665 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa5b800b940 con 0x7fa5c00d1c80 2021-02-10T11:35:11.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.721 7fa5c840b700 1 -- 172.21.2.112:0/2174043665 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa5c00d1c80 msgr2=0x7fa5c0108f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:11.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.721 7fa5c840b700 1 --2- 172.21.2.112:0/2174043665 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa5c00d1c80 0x7fa5c0108f70 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7fa5b8006e50 tx=0x7fa5b8009e70).stop 2021-02-10T11:35:11.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.722 7fa5c840b700 1 -- 172.21.2.112:0/2174043665 shutdown_connections 2021-02-10T11:35:11.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.722 7fa5c840b700 1 --2- 172.21.2.112:0/2174043665 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa5c00d1c80 0x7fa5c0108f70 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:11.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.722 7fa5c840b700 1 --2- 172.21.2.112:0/2174043665 >> v2:172.21.2.120:3300/0 conn(0x7fa5c01b0af0 0x7fa5c0108950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:11.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.722 7fa5c840b700 1 --2- 172.21.2.112:0/2174043665 >> v2:172.21.2.112:3300/0 conn(0x7fa5c00d1880 0x7fa5c01b04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:11.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.722 7fa5c840b700 1 -- 172.21.2.112:0/2174043665 >> 172.21.2.112:0/2174043665 conn(0x7fa5c00ce430 msgr2=0x7fa5c00d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:11.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.722 7fa5c840b700 1 -- 172.21.2.112:0/2174043665 shutdown_connections 2021-02-10T11:35:11.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.722 7fa5c840b700 1 -- 172.21.2.112:0/2174043665 wait complete. 2021-02-10T11:35:11.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.737 7fa5c840b700 1 Processor -- start 2021-02-10T11:35:11.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.737 7fa5c840b700 1 -- start start 2021-02-10T11:35:11.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.737 7fa5c840b700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa5c00913d0 0x7fa5c01a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:11.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.737 7fa5c840b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa5c01a3c30 0x7fa5c00b1170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:11.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.737 7fa5c840b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa5c00b1790 0x7fa5c00b1b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:11.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.737 7fa5c840b700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5c0091850 con 0x7fa5c01a3c30 2021-02-10T11:35:11.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.737 7fa5c840b700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5c00b21b0 con 0x7fa5c00913d0 2021-02-10T11:35:11.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.737 7fa5c840b700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5c00b23d0 con 0x7fa5c00b1790 2021-02-10T11:35:11.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.737 7fa5c5a26700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa5c01a3c30 0x7fa5c00b1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:11.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.737 7fa5c5a26700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa5c01a3c30 0x7fa5c00b1170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:51032/0 (socket says 172.21.2.112:51032) 2021-02-10T11:35:11.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.737 7fa5c5a26700 1 -- 172.21.2.112:0/815874934 learned_addr learned my addr 172.21.2.112:0/815874934 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:35:11.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.737 7fa5c6a28700 1 --2- 172.21.2.112:0/815874934 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa5c00b1790 0x7fa5c00b1b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:11.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.737 7fa5c6227700 1 --2- 172.21.2.112:0/815874934 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa5c00913d0 0x7fa5c01a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:11.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.738 7fa5c5a26700 1 -- 172.21.2.112:0/815874934 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa5c00b1790 msgr2=0x7fa5c00b1b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:11.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.738 7fa5c5a26700 1 --2- 172.21.2.112:0/815874934 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa5c00b1790 0x7fa5c00b1b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:11.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.738 7fa5c5a26700 1 -- 172.21.2.112:0/815874934 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa5c00913d0 msgr2=0x7fa5c01a3610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:11.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.738 7fa5c5a26700 1 --2- 172.21.2.112:0/815874934 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa5c00913d0 0x7fa5c01a3610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:11.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.738 7fa5c5a26700 1 -- 172.21.2.112:0/815874934 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5c00af2a0 con 0x7fa5c01a3c30 2021-02-10T11:35:11.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.738 7fa5c5a26700 1 --2- 172.21.2.112:0/815874934 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa5c01a3c30 0x7fa5c00b1170 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 rx=0x7fa5bc006f00 tx=0x7fa5bc00b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:11.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.738 7fa5b2ffd700 1 -- 172.21.2.112:0/815874934 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa5bc00bd30 con 0x7fa5c01a3c30 2021-02-10T11:35:11.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.738 7fa5b2ffd700 1 -- 172.21.2.112:0/815874934 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5bc002030 con 0x7fa5c01a3c30 2021-02-10T11:35:11.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.738 7fa5b2ffd700 1 -- 172.21.2.112:0/815874934 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa5bc002550 con 0x7fa5c01a3c30 2021-02-10T11:35:11.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.738 7fa5c840b700 1 -- 172.21.2.112:0/815874934 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5c00af2a0 con 0x7fa5c01a3c30 2021-02-10T11:35:11.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.738 7fa5c840b700 1 -- 172.21.2.112:0/815874934 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5c00af830 con 0x7fa5c01a3c30 2021-02-10T11:35:11.732 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.739 7fa5b2ffd700 1 -- 172.21.2.112:0/815874934 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa5bc000c90 con 0x7fa5c01a3c30 2021-02-10T11:35:11.732 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.739 7fa5b2ffd700 1 --2- 172.21.2.112:0/815874934 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa5a80186c0 0x7fa5a801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:11.732 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.739 7fa5b2ffd700 1 -- 172.21.2.112:0/815874934 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fa5bc032400 con 0x7fa5c01a3c30 2021-02-10T11:35:11.732 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.739 7fa5c6227700 1 --2- 172.21.2.112:0/815874934 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa5a80186c0 0x7fa5a801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:11.732 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.740 7fa5c6227700 1 --2- 172.21.2.112:0/815874934 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa5a80186c0 0x7fa5a801abc0 crc :-1 s=READY pgs=707 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:11.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.769 7fa5c840b700 1 -- 172.21.2.112:0/815874934 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5c0082aa0 con 0x7fa5c01a3c30 2021-02-10T11:35:11.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.772 7fa5b2ffd700 1 -- 172.21.2.112:0/815874934 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa5bc01e460 con 0x7fa5c01a3c30 2021-02-10T11:35:11.886 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.899 7fa5c840b700 1 -- 172.21.2.112:0/815874934 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000513667", "target": ["mgr", ""], "group_name": "group_0000000000485992"}) v1 -- 0x7fa5c00af830 con 0x7fa5a80186c0 2021-02-10T11:35:11.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.926 7fa5b2ffd700 1 -- 172.21.2.112:0/815874934 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fa5b4006a80 con 0x7fa5a80186c0 2021-02-10T11:35:11.919 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.933 7fa5c840b700 1 -- 172.21.2.112:0/815874934 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa5a80186c0 msgr2=0x7fa5a801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:11.919 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.933 7fa5c840b700 1 --2- 172.21.2.112:0/815874934 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa5a80186c0 0x7fa5a801abc0 crc :-1 s=READY pgs=707 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:11.919 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.933 7fa5c840b700 1 -- 172.21.2.112:0/815874934 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa5c01a3c30 msgr2=0x7fa5c00b1170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:11.919 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.933 7fa5c840b700 1 --2- 172.21.2.112:0/815874934 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa5c01a3c30 0x7fa5c00b1170 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 rx=0x7fa5bc006f00 tx=0x7fa5bc00b940).stop 2021-02-10T11:35:11.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.933 7fa5c840b700 1 -- 172.21.2.112:0/815874934 shutdown_connections 2021-02-10T11:35:11.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.933 7fa5c840b700 1 --2- 172.21.2.112:0/815874934 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa5a80186c0 0x7fa5a801abc0 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:11.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.933 7fa5c840b700 1 --2- 172.21.2.112:0/815874934 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa5c00b1790 0x7fa5c00b1b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:11.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.933 7fa5c840b700 1 --2- 172.21.2.112:0/815874934 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa5c01a3c30 0x7fa5c00b1170 unknown :-1 s=CLOSED pgs=1454 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:11.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.933 7fa5c840b700 1 --2- 172.21.2.112:0/815874934 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa5c00913d0 0x7fa5c01a3610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:11.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.933 7fa5c840b700 1 -- 172.21.2.112:0/815874934 >> 172.21.2.112:0/815874934 conn(0x7fa5c0087360 msgr2=0x7fa5c00b5980 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:11.921 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.933 7fa5c840b700 1 -- 172.21.2.112:0/815874934 shutdown_connections 2021-02-10T11:35:11.921 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:11.933 7fa5c840b700 1 -- 172.21.2.112:0/815874934 wait complete. 2021-02-10T11:35:11.928 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000513667 --group_name group_0000000000485992 2021-02-10T11:35:12.358 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.354 7f00093f3700 1 Processor -- start 2021-02-10T11:35:12.358 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.354 7f00093f3700 1 -- start start 2021-02-10T11:35:12.358 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.355 7f00093f3700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f00040d1c80 0x7f00040cef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:12.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.355 7f00093f3700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f00040cf590 0x7f00040cf990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:12.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.355 7f00093f3700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00040fe1d0 con 0x7f00040d1880 2021-02-10T11:35:12.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.355 7f00093f3700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00041b0cc0 con 0x7f00040cf590 2021-02-10T11:35:12.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.355 7f00093f3700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0004109a70 con 0x7f00040d1c80 2021-02-10T11:35:12.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.355 7f000307d700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f00040d1880 0x7f00041b04d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:50500/0 (socket says 172.21.2.112:50500) 2021-02-10T11:35:12.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.355 7f000307d700 1 -- 172.21.2.112:0/2480206421 learned_addr learned my addr 172.21.2.112:0/2480206421 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:35:12.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.355 7f000387e700 1 --2- 172.21.2.112:0/2480206421 >> v2:172.21.2.112:3300/0 conn(0x7f00040cf590 0x7f00040cf990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:12.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.355 7f000287c700 1 --2- 172.21.2.112:0/2480206421 >> v2:172.21.2.120:3300/0 conn(0x7f00040d1c80 0x7f00040cef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:12.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.355 7f000207b700 1 -- 172.21.2.112:0/2480206421 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3930617356 0 0) 0x7efff8001180 con 0x7f00040d1880 2021-02-10T11:35:12.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.355 7f000287c700 1 -- 172.21.2.112:0/2480206421 >> v2:172.21.2.112:3300/0 conn(0x7f00040cf590 msgr2=0x7f00040cf990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:12.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.355 7f000287c700 1 --2- 172.21.2.112:0/2480206421 >> v2:172.21.2.112:3300/0 conn(0x7f00040cf590 0x7f00040cf990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:12.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.355 7f000287c700 1 -- 172.21.2.112:0/2480206421 >> v1:172.21.2.112:6789/0 conn(0x7f00040d1880 legacy=0x7f00041b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:12.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.355 7f000287c700 1 -- 172.21.2.112:0/2480206421 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0004088070 con 0x7f00040d1c80 2021-02-10T11:35:12.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.356 7f000287c700 1 --2- 172.21.2.112:0/2480206421 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f00040d1c80 0x7f00040cef70 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 rx=0x7efff4006650 tx=0x7efff4009a80).ready entity=mon.1 client_cookie=b5df3bf1ce36f1b3 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:12.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.356 7f000207b700 1 -- 172.21.2.112:0/2480206421 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efff40009b0 con 0x7f00040d1c80 2021-02-10T11:35:12.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.356 7f000207b700 1 -- 172.21.2.112:0/2480206421 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efff400c0c0 con 0x7f00040d1c80 2021-02-10T11:35:12.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.356 7f000207b700 1 -- 172.21.2.112:0/2480206421 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efff400c510 con 0x7f00040d1c80 2021-02-10T11:35:12.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.356 7f00093f3700 1 -- 172.21.2.112:0/2480206421 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f00040d1c80 msgr2=0x7f00040cef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:12.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.356 7f00093f3700 1 --2- 172.21.2.112:0/2480206421 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f00040d1c80 0x7f00040cef70 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 rx=0x7efff4006650 tx=0x7efff4009a80).stop 2021-02-10T11:35:12.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.357 7f00093f3700 1 -- 172.21.2.112:0/2480206421 shutdown_connections 2021-02-10T11:35:12.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.357 7f00093f3700 1 --2- 172.21.2.112:0/2480206421 >> v2:172.21.2.112:3300/0 conn(0x7f00040cf590 0x7f00040cf990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:12.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.357 7f00093f3700 1 --2- 172.21.2.112:0/2480206421 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f00040d1c80 0x7f00040cef70 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:12.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.357 7f00093f3700 1 -- 172.21.2.112:0/2480206421 >> 172.21.2.112:0/2480206421 conn(0x7f00040ce430 msgr2=0x7f00040d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:12.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.357 7f00093f3700 1 -- 172.21.2.112:0/2480206421 shutdown_connections 2021-02-10T11:35:12.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.357 7f00093f3700 1 -- 172.21.2.112:0/2480206421 wait complete. 2021-02-10T11:35:12.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.371 7f00093f3700 1 Processor -- start 2021-02-10T11:35:12.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.371 7f00093f3700 1 -- start start 2021-02-10T11:35:12.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.372 7f00093f3700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f00040aef00 0x7f0004099c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:12.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.372 7f00093f3700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f000409a260 0x7f000409a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:12.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.372 7f00093f3700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f000409ac80 0x7f000419a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:12.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.372 7f00093f3700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f000419bdd0 con 0x7f000409a260 2021-02-10T11:35:12.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.372 7f00093f3700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f000419de70 con 0x7f000409ac80 2021-02-10T11:35:12.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.372 7f00093f3700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f000419a2d0 con 0x7f00040aef00 2021-02-10T11:35:12.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.372 7f000287c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f000409a260 0x7f000409a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:12.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.372 7f000287c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f000409a260 0x7f000409a660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:51044/0 (socket says 172.21.2.112:51044) 2021-02-10T11:35:12.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.372 7f000287c700 1 -- 172.21.2.112:0/856250838 learned_addr learned my addr 172.21.2.112:0/856250838 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:35:12.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.372 7f000307d700 1 --2- 172.21.2.112:0/856250838 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f00040aef00 0x7f0004099c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:12.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.372 7f000387e700 1 --2- 172.21.2.112:0/856250838 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f000409ac80 0x7f000419a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:12.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.373 7f000287c700 1 -- 172.21.2.112:0/856250838 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f00040aef00 msgr2=0x7f0004099c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:12.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.373 7f000287c700 1 --2- 172.21.2.112:0/856250838 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f00040aef00 0x7f0004099c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:12.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.373 7f000287c700 1 -- 172.21.2.112:0/856250838 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f000409ac80 msgr2=0x7f000419a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:12.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.373 7f000287c700 1 --2- 172.21.2.112:0/856250838 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f000409ac80 0x7f000419a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:12.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.373 7f000287c700 1 -- 172.21.2.112:0/856250838 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0004199f50 con 0x7f000409a260 2021-02-10T11:35:12.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.373 7f000287c700 1 --2- 172.21.2.112:0/856250838 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f000409a260 0x7f000409a660 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 rx=0x7efff4009630 tx=0x7efff400b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:12.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.373 7effebfff700 1 -- 172.21.2.112:0/856250838 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efff40093a0 con 0x7f000409a260 2021-02-10T11:35:12.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.373 7effebfff700 1 -- 172.21.2.112:0/856250838 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efff400bd70 con 0x7f000409a260 2021-02-10T11:35:12.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.373 7effebfff700 1 -- 172.21.2.112:0/856250838 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efff4005560 con 0x7f000409a260 2021-02-10T11:35:12.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.373 7f00093f3700 1 -- 172.21.2.112:0/856250838 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0004199f50 con 0x7f000409a260 2021-02-10T11:35:12.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.373 7f00093f3700 1 -- 172.21.2.112:0/856250838 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0004199f50 con 0x7f000409a260 2021-02-10T11:35:12.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.373 7effebfff700 1 -- 172.21.2.112:0/856250838 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7efff400a000 con 0x7f000409a260 2021-02-10T11:35:12.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.373 7effebfff700 1 --2- 172.21.2.112:0/856250838 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7effe40186c0 0x7effe401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:12.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.374 7effebfff700 1 -- 172.21.2.112:0/856250838 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5573+0+0 (secure 0 0 0) 0x7efff4032410 con 0x7f000409a260 2021-02-10T11:35:12.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.374 7f000307d700 1 --2- 172.21.2.112:0/856250838 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7effe40186c0 0x7effe401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:12.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.374 7f000307d700 1 --2- 172.21.2.112:0/856250838 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7effe40186c0 0x7effe401abc0 crc :-1 s=READY pgs=708 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:12.391 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.404 7f00093f3700 1 -- 172.21.2.112:0/856250838 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0004082a70 con 0x7f000409a260 2021-02-10T11:35:12.393 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.407 7effebfff700 1 -- 172.21.2.112:0/856250838 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7efff4026660 con 0x7f000409a260 2021-02-10T11:35:12.524 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.538 7f00093f3700 1 -- 172.21.2.112:0/856250838 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000513667", "target": ["mgr", ""], "group_name": "group_0000000000485992"}) v1 -- 0x7f0004199f50 con 0x7effe40186c0 2021-02-10T11:35:12.528 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.541 7effebfff700 1 -- 172.21.2.112:0/856250838 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+96 (crc 0 0 0) 0x7efff8001180 con 0x7effe40186c0 2021-02-10T11:35:12.531 INFO:teuthology.orchestra.run.gibba012.stdout:/volumes/group_0000000000485992/subvolume_0000000000513667/7bc8264f-d88f-49e9-b22f-5fa8b916f683 2021-02-10T11:35:12.533 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.546 7f00093f3700 1 -- 172.21.2.112:0/856250838 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7effe40186c0 msgr2=0x7effe401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:12.533 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.546 7f00093f3700 1 --2- 172.21.2.112:0/856250838 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7effe40186c0 0x7effe401abc0 crc :-1 s=READY pgs=708 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:12.533 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.546 7f00093f3700 1 -- 172.21.2.112:0/856250838 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f000409a260 msgr2=0x7f000409a660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:12.533 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.546 7f00093f3700 1 --2- 172.21.2.112:0/856250838 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f000409a260 0x7f000409a660 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 rx=0x7efff4009630 tx=0x7efff400b980).stop 2021-02-10T11:35:12.533 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.546 7f00093f3700 1 -- 172.21.2.112:0/856250838 shutdown_connections 2021-02-10T11:35:12.533 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.546 7f00093f3700 1 --2- 172.21.2.112:0/856250838 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7effe40186c0 0x7effe401abc0 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:12.533 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.546 7f00093f3700 1 --2- 172.21.2.112:0/856250838 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f000409ac80 0x7f000419a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:12.533 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.546 7f00093f3700 1 --2- 172.21.2.112:0/856250838 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f000409a260 0x7f000409a660 unknown :-1 s=CLOSED pgs=1456 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:12.533 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.546 7f00093f3700 1 --2- 172.21.2.112:0/856250838 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f00040aef00 0x7f0004099c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:12.533 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.546 7f00093f3700 1 -- 172.21.2.112:0/856250838 >> 172.21.2.112:0/856250838 conn(0x7f000418e810 msgr2=0x7f000418c290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:12.534 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.546 7f00093f3700 1 -- 172.21.2.112:0/856250838 shutdown_connections 2021-02-10T11:35:12.534 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.547 7f00093f3700 1 -- 172.21.2.112:0/856250838 wait complete. 2021-02-10T11:35:12.541 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume authorize cephfs subvolume_0000000000513667 alice --group_name group_0000000000485992 --tenant_id tenant_id 2021-02-10T11:35:12.978 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.974 7fc914014700 1 Processor -- start 2021-02-10T11:35:12.978 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.974 7fc914014700 1 -- start start 2021-02-10T11:35:12.979 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.975 7fc914014700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc90c10a040 0x7fc90c10a440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:12.979 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.975 7fc914014700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fc90c10aa60 0x7fc90c10ae60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:12.979 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.975 7fc914014700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc90c1b0d40 con 0x7fc90c0e53a0 2021-02-10T11:35:12.979 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.975 7fc914014700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc90c109ac0 con 0x7fc90c10aa60 2021-02-10T11:35:12.979 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.975 7fc914014700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc90c0e57a0 con 0x7fc90c10a040 2021-02-10T11:35:12.979 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.975 7fc911e30700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fc90c0e53a0 0x7fc90c1b0550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:50514/0 (socket says 172.21.2.112:50514) 2021-02-10T11:35:12.979 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.975 7fc911e30700 1 -- 172.21.2.112:0/3726521007 learned_addr learned my addr 172.21.2.112:0/3726521007 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:35:12.979 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.975 7fc910e2e700 1 -- 172.21.2.112:0/3726521007 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3239682744 0 0) 0x7fc9000010b0 con 0x7fc90c0e53a0 2021-02-10T11:35:12.979 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.975 7fc910e2e700 1 -- 172.21.2.112:0/3726521007 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8f8002fe0 con 0x7fc90c0e53a0 2021-02-10T11:35:12.979 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.975 7fc910e2e700 1 -- 172.21.2.112:0/3726521007 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4255772131 0 0) 0x7fc9000010b0 con 0x7fc90c0e53a0 2021-02-10T11:35:12.979 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.975 7fc910e2e700 1 -- 172.21.2.112:0/3726521007 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc90c10a040 msgr2=0x7fc90c10a440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:12.979 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.975 7fc91162f700 1 --2- 172.21.2.112:0/3726521007 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc90c10a040 0x7fc90c10a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:12.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.975 7fc910e2e700 1 --2- 172.21.2.112:0/3726521007 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc90c10a040 0x7fc90c10a440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:12.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.975 7fc910e2e700 1 -- 172.21.2.112:0/3726521007 >> v2:172.21.2.120:3300/0 conn(0x7fc90c10aa60 msgr2=0x7fc90c10ae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:12.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.975 7fc910e2e700 1 --2- 172.21.2.112:0/3726521007 >> v2:172.21.2.120:3300/0 conn(0x7fc90c10aa60 0x7fc90c10ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:12.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.975 7fc910e2e700 1 -- 172.21.2.112:0/3726521007 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc90c10b480 con 0x7fc90c0e53a0 2021-02-10T11:35:12.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.976 7fc910e2e700 1 -- 172.21.2.112:0/3726521007 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7fc9000015b0 con 0x7fc90c0e53a0 2021-02-10T11:35:12.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.976 7fc910e2e700 1 -- 172.21.2.112:0/3726521007 >> v1:172.21.2.112:6789/0 conn(0x7fc90c0e53a0 legacy=0x7fc90c1b0550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:12.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.976 7fc910e2e700 1 --2- 172.21.2.112:0/3726521007 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc8f8003ae0 0x7fc8f8005f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:12.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.976 7fc910e2e700 1 --2- 172.21.2.112:0/3726521007 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc8f8006810 0x7fc8f8008d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:12.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.976 7fc910e2e700 1 --2- 172.21.2.112:0/3726521007 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc8f8009630 0x7fc8f800bb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:12.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.976 7fc910e2e700 1 -- 172.21.2.112:0/3726521007 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f800c370 con 0x7fc8f8003ae0 2021-02-10T11:35:12.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.976 7fc910e2e700 1 -- 172.21.2.112:0/3726521007 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f800c590 con 0x7fc8f8006810 2021-02-10T11:35:12.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.976 7fc910e2e700 1 -- 172.21.2.112:0/3726521007 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f800c7b0 con 0x7fc8f8009630 2021-02-10T11:35:12.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.976 7fc911e30700 1 --2- 172.21.2.112:0/3726521007 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc8f8003ae0 0x7fc8f8005f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:12.981 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.976 7fc91162f700 1 --2- 172.21.2.112:0/3726521007 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc90c10a040 0x7fc90c10a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:35:12.981 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.976 7fc912631700 1 --2- 172.21.2.112:0/3726521007 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc8f8009630 0x7fc8f800bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:12.981 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.976 7fc911e30700 1 -- 172.21.2.112:0/3726521007 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc8f8009630 msgr2=0x7fc8f800bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:12.981 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.976 7fc911e30700 1 --2- 172.21.2.112:0/3726521007 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc8f8009630 0x7fc8f800bb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:12.981 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.976 7fc911e30700 1 -- 172.21.2.112:0/3726521007 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc8f8006810 msgr2=0x7fc8f8008d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:12.981 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.976 7fc911e30700 1 --2- 172.21.2.112:0/3726521007 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc8f8006810 0x7fc8f8008d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:12.981 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.976 7fc911e30700 1 -- 172.21.2.112:0/3726521007 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8f800ca50 con 0x7fc8f8003ae0 2021-02-10T11:35:12.981 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.976 7fc911e30700 1 --2- 172.21.2.112:0/3726521007 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc8f8003ae0 0x7fc8f8005f90 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 rx=0x7fc900001c60 tx=0x7fc900007d40).ready entity=mon.0 client_cookie=b1e1ce31f7cbf41b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:12.981 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.976 7fc910e2e700 1 -- 172.21.2.112:0/3726521007 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc9000010b0 con 0x7fc8f8003ae0 2021-02-10T11:35:12.981 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.976 7fc910e2e700 1 -- 172.21.2.112:0/3726521007 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc90004e530 con 0x7fc8f8003ae0 2021-02-10T11:35:12.981 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.976 7fc910e2e700 1 -- 172.21.2.112:0/3726521007 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc90004e9c0 con 0x7fc8f8003ae0 2021-02-10T11:35:12.981 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.976 7fc914014700 1 -- 172.21.2.112:0/3726521007 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc8f8003ae0 msgr2=0x7fc8f8005f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:12.981 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.976 7fc914014700 1 --2- 172.21.2.112:0/3726521007 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc8f8003ae0 0x7fc8f8005f90 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 rx=0x7fc900001c60 tx=0x7fc900007d40).stop 2021-02-10T11:35:12.982 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.976 7fc912631700 1 -- 172.21.2.112:0/3726521007 reap_dead start 2021-02-10T11:35:12.982 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.977 7fc914014700 1 -- 172.21.2.112:0/3726521007 shutdown_connections 2021-02-10T11:35:12.982 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.977 7fc914014700 1 -- 172.21.2.112:0/3726521007 >> 172.21.2.112:0/3726521007 conn(0x7fc90c0ce4b0 msgr2=0x7fc90c0e45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:12.982 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.977 7fc914014700 1 -- 172.21.2.112:0/3726521007 shutdown_connections 2021-02-10T11:35:12.982 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.977 7fc914014700 1 -- 172.21.2.112:0/3726521007 wait complete. 2021-02-10T11:35:12.982 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.992 7fc914014700 1 Processor -- start 2021-02-10T11:35:12.982 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.992 7fc914014700 1 -- start start 2021-02-10T11:35:12.983 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.992 7fc914014700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc90c0a8f70 0x7fc90c095e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:12.984 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.992 7fc914014700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc90c0964a0 0x7fc90c0968a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:12.984 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.992 7fc914014700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc90c096ec0 0x7fc90c106520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:12.984 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.992 7fc914014700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc90c095900 con 0x7fc90c096ec0 2021-02-10T11:35:12.984 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.992 7fc914014700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc90c107be0 con 0x7fc90c0a8f70 2021-02-10T11:35:12.984 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.992 7fc914014700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc90c106b40 con 0x7fc90c0964a0 2021-02-10T11:35:12.984 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.992 7fc912631700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc90c096ec0 0x7fc90c106520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:12.984 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.992 7fc912631700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc90c096ec0 0x7fc90c106520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:51066/0 (socket says 172.21.2.112:51066) 2021-02-10T11:35:12.984 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.992 7fc912631700 1 -- 172.21.2.112:0/152706539 learned_addr learned my addr 172.21.2.112:0/152706539 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:35:12.984 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.992 7fc911e30700 1 --2- 172.21.2.112:0/152706539 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc90c0a8f70 0x7fc90c095e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:12.984 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.992 7fc91162f700 1 --2- 172.21.2.112:0/152706539 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc90c0964a0 0x7fc90c0968a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:12.984 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.993 7fc912631700 1 -- 172.21.2.112:0/152706539 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc90c0964a0 msgr2=0x7fc90c0968a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:12.984 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.993 7fc912631700 1 --2- 172.21.2.112:0/152706539 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc90c0964a0 0x7fc90c0968a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:12.985 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.993 7fc912631700 1 -- 172.21.2.112:0/152706539 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc90c0a8f70 msgr2=0x7fc90c095e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:12.985 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.993 7fc912631700 1 --2- 172.21.2.112:0/152706539 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc90c0a8f70 0x7fc90c095e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:12.985 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.993 7fc912631700 1 -- 172.21.2.112:0/152706539 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc90c106d60 con 0x7fc90c096ec0 2021-02-10T11:35:12.985 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.993 7fc912631700 1 --2- 172.21.2.112:0/152706539 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc90c096ec0 0x7fc90c106520 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 rx=0x7fc904009920 tx=0x7fc90400bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:12.985 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.993 7fc8fe7fc700 1 -- 172.21.2.112:0/152706539 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc904006df0 con 0x7fc90c096ec0 2021-02-10T11:35:12.985 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.993 7fc8fe7fc700 1 -- 172.21.2.112:0/152706539 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc904006030 con 0x7fc90c096ec0 2021-02-10T11:35:12.985 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.993 7fc8fe7fc700 1 -- 172.21.2.112:0/152706539 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc904006400 con 0x7fc90c096ec0 2021-02-10T11:35:12.985 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.993 7fc914014700 1 -- 172.21.2.112:0/152706539 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc90c106d60 con 0x7fc90c096ec0 2021-02-10T11:35:12.985 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.993 7fc914014700 1 -- 172.21.2.112:0/152706539 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc90c106d60 con 0x7fc90c096ec0 2021-02-10T11:35:12.985 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.994 7fc8fe7fc700 1 -- 172.21.2.112:0/152706539 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc904006b70 con 0x7fc90c096ec0 2021-02-10T11:35:12.985 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.994 7fc8fe7fc700 1 --2- 172.21.2.112:0/152706539 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc8f40186d0 0x7fc8f401abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:12.985 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.994 7fc8fe7fc700 1 -- 172.21.2.112:0/152706539 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fc9040329e0 con 0x7fc90c096ec0 2021-02-10T11:35:12.985 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.994 7fc911e30700 1 --2- 172.21.2.112:0/152706539 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc8f40186d0 0x7fc8f401abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:12.986 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:12.995 7fc911e30700 1 --2- 172.21.2.112:0/152706539 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc8f40186d0 0x7fc8f401abd0 crc :-1 s=READY pgs=709 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:13.011 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:13.025 7fc914014700 1 -- 172.21.2.112:0/152706539 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc90c103750 con 0x7fc90c096ec0 2021-02-10T11:35:13.014 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:13.027 7fc8fe7fc700 1 -- 172.21.2.112:0/152706539 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc904006f00 con 0x7fc90c096ec0 2021-02-10T11:35:13.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:13.153 7fc914014700 1 -- 172.21.2.112:0/152706539 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume authorize", "vol_name": "cephfs", "sub_name": "subvolume_0000000000513667", "target": ["mgr", ""], "tenant_id": "tenant_id", "auth_id": "alice", "group_name": "group_0000000000485992"}) v1 -- 0x7fc90c103750 con 0x7fc8f40186d0 2021-02-10T11:35:13.145 INFO:tasks.ceph.mgr.x.gibba020.stderr:Creating meta for ID alice with tenant tenant_id 2021-02-10T11:35:13.174 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:13.187 7fc8fe7fc700 1 -- 172.21.2.112:0/152706539 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+41 (crc 0 0 0) 0x7fc900001490 con 0x7fc8f40186d0 2021-02-10T11:35:13.203 INFO:teuthology.orchestra.run.gibba012.stdout:AQDxxCNg5hUmChAArexyCsZP7RgHhs6rSnmt2Q== 2021-02-10T11:35:13.205 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:13.218 7fc914014700 1 -- 172.21.2.112:0/152706539 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc8f40186d0 msgr2=0x7fc8f401abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:13.205 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:13.218 7fc914014700 1 --2- 172.21.2.112:0/152706539 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc8f40186d0 0x7fc8f401abd0 crc :-1 s=READY pgs=709 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:13.205 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:13.218 7fc914014700 1 -- 172.21.2.112:0/152706539 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc90c096ec0 msgr2=0x7fc90c106520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:13.205 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:13.218 7fc914014700 1 --2- 172.21.2.112:0/152706539 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc90c096ec0 0x7fc90c106520 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 rx=0x7fc904009920 tx=0x7fc90400bd40).stop 2021-02-10T11:35:13.205 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:13.218 7fc914014700 1 -- 172.21.2.112:0/152706539 shutdown_connections 2021-02-10T11:35:13.206 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:13.218 7fc914014700 1 --2- 172.21.2.112:0/152706539 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc8f40186d0 0x7fc8f401abd0 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:13.206 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:13.218 7fc914014700 1 --2- 172.21.2.112:0/152706539 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc90c096ec0 0x7fc90c106520 unknown :-1 s=CLOSED pgs=1459 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:13.206 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:13.218 7fc914014700 1 --2- 172.21.2.112:0/152706539 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc90c0964a0 0x7fc90c0968a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:13.206 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:13.218 7fc914014700 1 --2- 172.21.2.112:0/152706539 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc90c0a8f70 0x7fc90c095e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:13.206 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:13.218 7fc914014700 1 -- 172.21.2.112:0/152706539 >> 172.21.2.112:0/152706539 conn(0x7fc90c09e480 msgr2=0x7fc90c107f30 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:13.206 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:13.218 7fc914014700 1 -- 172.21.2.112:0/152706539 shutdown_connections 2021-02-10T11:35:13.206 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:13.219 7fc914014700 1 -- 172.21.2.112:0/152706539 wait complete. 2021-02-10T11:35:13.214 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth ls --format=json-pretty 2021-02-10T11:35:13.642 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:13.638 7f0d6d1af700 1 Processor -- start 2021-02-10T11:35:13.643 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:13.638 7f0d6d1af700 1 -- start start 2021-02-10T11:35:13.643 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:13.639 7f0d6d1af700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f0d680d0340 0x7f0d680ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:13.643 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:13.639 7f0d6d1af700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0d680d1c70 0x7f0d680efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:13.643 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:13.639 7f0d6d1af700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d680f0b10 con 0x7f0d680d1870 2021-02-10T11:35:13.643 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:13.639 7f0d6d1af700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d680e5ed0 con 0x7f0d680d0340 2021-02-10T11:35:13.643 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:13.639 7f0d6d1af700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d681b0be0 con 0x7f0d680d1c70 2021-02-10T11:35:13.643 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:13.639 7f0d66d9d700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f0d680d1870 0x7f0d681b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:50534/0 (socket says 172.21.2.112:50534) 2021-02-10T11:35:13.643 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:13.639 7f0d66d9d700 1 -- 172.21.2.112:0/722491840 learned_addr learned my addr 172.21.2.112:0/722491840 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:35:13.643 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:13.639 7f0d65d9b700 1 -- 172.21.2.112:0/722491840 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2534262722 0 0) 0x7f0d58001180 con 0x7f0d680d1870 2021-02-10T11:35:13.643 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:13.639 7f0d6759e700 1 --2- 172.21.2.112:0/722491840 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0d680d1c70 0x7f0d680efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:13.644 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:13.639 7f0d65d9b700 1 -- 172.21.2.112:0/722491840 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d54002fe0 con 0x7f0d680d1870 2021-02-10T11:35:13.644 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:13.639 7f0d6659c700 1 --2- 172.21.2.112:0/722491840 >> v2:172.21.2.120:3300/0 conn(0x7f0d680d0340 0x7f0d680ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:13.644 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:13.640 7f0d65d9b700 1 -- 172.21.2.112:0/722491840 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2056864780 0 0) 0x7f0d580010e0 con 0x7f0d680d1870 2021-02-10T11:35:13.644 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:13.640 7f0d65d9b700 1 -- 172.21.2.112:0/722491840 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0d680d1c70 msgr2=0x7f0d680efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:13.644 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:13.640 7f0d65d9b700 1 --2- 172.21.2.112:0/722491840 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0d680d1c70 0x7f0d680efc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:13.644 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:13.640 7f0d65d9b700 1 -- 172.21.2.112:0/722491840 >> v2:172.21.2.120:3300/0 conn(0x7f0d680d0340 msgr2=0x7f0d680ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:13.644 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:13.640 7f0d65d9b700 1 --2- 172.21.2.112:0/722491840 >> v2:172.21.2.120:3300/0 conn(0x7f0d680d0340 0x7f0d680ef640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:13.644 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:13.640 7f0d65d9b700 1 -- 172.21.2.112:0/722491840 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d680f20a0 con 0x7f0d680d1870 2021-02-10T11:35:13.644 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:13.640 7f0d65d9b700 1 -- 172.21.2.112:0/722491840 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f0d58001600 con 0x7f0d680d1870 2021-02-10T11:35:13.644 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:13.640 7f0d65d9b700 1 -- 172.21.2.112:0/722491840 >> v1:172.21.2.112:6789/0 conn(0x7f0d680d1870 legacy=0x7f0d681b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:13.644 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:13.640 7f0d65d9b700 1 --2- 172.21.2.112:0/722491840 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0d54003b10 0x7f0d54005fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:13.644 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:13.640 7f0d65d9b700 1 --2- 172.21.2.112:0/722491840 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0d5404a800 0x7f0d5404cca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:13.644 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:13.640 7f0d65d9b700 1 --2- 172.21.2.112:0/722491840 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0d5404d5a0 0x7f0d5404fa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:13.644 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:13.640 7f0d65d9b700 1 -- 172.21.2.112:0/722491840 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d540502e0 con 0x7f0d5404a800 2021-02-10T11:35:13.645 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:13.640 7f0d65d9b700 1 -- 172.21.2.112:0/722491840 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d54050500 con 0x7f0d54003b10 2021-02-10T11:35:13.645 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:13.640 7f0d65d9b700 1 -- 172.21.2.112:0/722491840 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d54050720 con 0x7f0d5404d5a0 2021-02-10T11:35:13.645 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:13.640 7f0d6659c700 1 --2- 172.21.2.112:0/722491840 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0d5404a800 0x7f0d5404cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:13.645 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:13.640 7f0d6759e700 1 --2- 172.21.2.112:0/722491840 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0d5404d5a0 0x7f0d5404fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:13.645 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:13.640 7f0d66d9d700 1 --2- 172.21.2.112:0/722491840 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0d54003b10 0x7f0d54005fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:13.645 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:13.640 7f0d6659c700 1 -- 172.21.2.112:0/722491840 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0d5404d5a0 msgr2=0x7f0d5404fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:13.645 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:13.640 7f0d6659c700 1 --2- 172.21.2.112:0/722491840 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0d5404d5a0 0x7f0d5404fa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:13.645 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:13.640 7f0d6659c700 1 -- 172.21.2.112:0/722491840 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0d54003b10 msgr2=0x7f0d54005fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:13.645 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:13.640 7f0d6659c700 1 --2- 172.21.2.112:0/722491840 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0d54003b10 0x7f0d54005fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:13.645 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:13.640 7f0d6659c700 1 -- 172.21.2.112:0/722491840 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d540509c0 con 0x7f0d5404a800 2021-02-10T11:35:13.645 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:13.641 7f0d6659c700 1 --2- 172.21.2.112:0/722491840 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0d5404a800 0x7f0d5404cca0 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 rx=0x7f0d5000c870 tx=0x7f0d5000cc10).ready entity=mon.0 client_cookie=fb9bd7e1fa88e18d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:13.645 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:13.641 7f0d65d9b700 1 -- 172.21.2.112:0/722491840 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0d500090d0 con 0x7f0d5404a800 2021-02-10T11:35:13.646 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:13.641 7f0d65d9b700 1 -- 172.21.2.112:0/722491840 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d50009410 con 0x7f0d5404a800 2021-02-10T11:35:13.646 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:13.641 7f0d65d9b700 1 -- 172.21.2.112:0/722491840 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0d500098f0 con 0x7f0d5404a800 2021-02-10T11:35:13.646 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:13.641 7f0d6d1af700 1 -- 172.21.2.112:0/722491840 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0d5404a800 msgr2=0x7f0d5404cca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:13.646 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:13.641 7f0d6d1af700 1 --2- 172.21.2.112:0/722491840 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0d5404a800 0x7f0d5404cca0 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 rx=0x7f0d5000c870 tx=0x7f0d5000cc10).stop 2021-02-10T11:35:13.646 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:13.641 7f0d6759e700 1 -- 172.21.2.112:0/722491840 reap_dead start 2021-02-10T11:35:13.646 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:13.641 7f0d6d1af700 1 -- 172.21.2.112:0/722491840 shutdown_connections 2021-02-10T11:35:13.646 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:13.641 7f0d6d1af700 1 -- 172.21.2.112:0/722491840 >> 172.21.2.112:0/722491840 conn(0x7f0d680ce450 msgr2=0x7f0d680d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:13.646 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:13.641 7f0d6d1af700 1 -- 172.21.2.112:0/722491840 shutdown_connections 2021-02-10T11:35:13.646 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:13.641 7f0d6d1af700 1 -- 172.21.2.112:0/722491840 wait complete. 2021-02-10T11:35:13.646 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:13.656 7f0d6d1af700 1 Processor -- start 2021-02-10T11:35:13.646 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:13.656 7f0d6d1af700 1 -- start start 2021-02-10T11:35:13.648 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:13.656 7f0d6d1af700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0d681a4a60 0x7f0d680c1c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:13.648 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:13.656 7f0d6d1af700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0d680c2290 0x7f0d680c2690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:13.648 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:13.656 7f0d6d1af700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0d680c2cb0 0x7f0d680bfd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:13.648 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:13.656 7f0d6d1af700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d680c31c0 con 0x7f0d681a4a60 2021-02-10T11:35:13.648 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:13.656 7f0d6d1af700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d680f05f0 con 0x7f0d680c2290 2021-02-10T11:35:13.648 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:13.656 7f0d6d1af700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d680c0340 con 0x7f0d680c2cb0 2021-02-10T11:35:13.648 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:13.657 7f0d66d9d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0d681a4a60 0x7f0d680c1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:13.649 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:13.657 7f0d66d9d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0d681a4a60 0x7f0d680c1c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:51086/0 (socket says 172.21.2.112:51086) 2021-02-10T11:35:13.649 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:13.657 7f0d66d9d700 1 -- 172.21.2.112:0/3845879244 learned_addr learned my addr 172.21.2.112:0/3845879244 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:35:13.649 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:13.657 7f0d66d9d700 1 -- 172.21.2.112:0/3845879244 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0d680c2cb0 msgr2=0x7f0d680bfd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:13.649 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:13.657 7f0d66d9d700 1 --2- 172.21.2.112:0/3845879244 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0d680c2cb0 0x7f0d680bfd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:13.649 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:13.657 7f0d66d9d700 1 -- 172.21.2.112:0/3845879244 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0d680c2290 msgr2=0x7f0d680c2690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:13.649 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:13.657 7f0d66d9d700 1 --2- 172.21.2.112:0/3845879244 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0d680c2290 0x7f0d680c2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:13.649 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:13.657 7f0d66d9d700 1 -- 172.21.2.112:0/3845879244 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d680c0560 con 0x7f0d681a4a60 2021-02-10T11:35:13.649 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:13.657 7f0d66d9d700 1 --2- 172.21.2.112:0/3845879244 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0d681a4a60 0x7f0d680c1c70 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 rx=0x7f0d5800a5d0 tx=0x7f0d5800a970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:13.649 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:13.657 7f0d5f7fe700 1 -- 172.21.2.112:0/3845879244 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0d58001c40 con 0x7f0d681a4a60 2021-02-10T11:35:13.649 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:13.657 7f0d5f7fe700 1 -- 172.21.2.112:0/3845879244 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d5800ad60 con 0x7f0d681a4a60 2021-02-10T11:35:13.649 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:13.657 7f0d5f7fe700 1 -- 172.21.2.112:0/3845879244 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0d580045b0 con 0x7f0d681a4a60 2021-02-10T11:35:13.649 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:13.657 7f0d6d1af700 1 -- 172.21.2.112:0/3845879244 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d680c0560 con 0x7f0d681a4a60 2021-02-10T11:35:13.649 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:13.658 7f0d6d1af700 1 -- 172.21.2.112:0/3845879244 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d680c0560 con 0x7f0d681a4a60 2021-02-10T11:35:13.649 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:13.658 7f0d5f7fe700 1 -- 172.21.2.112:0/3845879244 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0d580043a0 con 0x7f0d681a4a60 2021-02-10T11:35:13.650 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:13.658 7f0d5f7fe700 1 --2- 172.21.2.112:0/3845879244 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0d4c0186c0 0x7f0d4c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:13.650 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:13.658 7f0d5f7fe700 1 -- 172.21.2.112:0/3845879244 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f0d58032580 con 0x7f0d681a4a60 2021-02-10T11:35:13.650 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:13.659 7f0d6659c700 1 --2- 172.21.2.112:0/3845879244 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0d4c0186c0 0x7f0d4c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:13.650 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:13.661 7f0d6659c700 1 --2- 172.21.2.112:0/3845879244 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0d4c0186c0 0x7f0d4c01abc0 crc :-1 s=READY pgs=710 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:13.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:13.689 7f0d6d1af700 1 -- 172.21.2.112:0/3845879244 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d680f0b10 con 0x7f0d681a4a60 2021-02-10T11:35:13.678 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:13.691 7f0d5f7fe700 1 -- 172.21.2.112:0/3845879244 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0d5801c2d0 con 0x7f0d681a4a60 2021-02-10T11:35:13.808 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:13.821 7f0d6d1af700 1 -- 172.21.2.112:0/3845879244 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f0d680f0b10 con 0x7f0d681a4a60 2021-02-10T11:35:13.808 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:13.822 7f0d5f7fe700 1 -- 172.21.2.112:0/3845879244 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v42) v1 ==== 80+0+4892 (secure 0 0 0) 0x7f0d5801c2d0 con 0x7f0d681a4a60 2021-02-10T11:35:13.809 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:35:13.809 INFO:teuthology.orchestra.run.gibba012.stdout:{ 2021-02-10T11:35:13.809 INFO:teuthology.orchestra.run.gibba012.stdout: "auth_dump": [ 2021-02-10T11:35:13.809 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:13.809 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "mds.a", 2021-02-10T11:35:13.809 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCKvyNgPkV4AxAAEmVNaghdccP6lWd8hvQnxw==", 2021-02-10T11:35:13.809 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:35:13.810 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:35:13.810 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:35:13.810 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:35:13.810 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:35:13.810 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:35:13.811 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:13.811 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:13.811 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "mds.b", 2021-02-10T11:35:13.811 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCKvyNguxONLhAAHIpry+R89MAwdhFJyEhVCw==", 2021-02-10T11:35:13.811 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:35:13.811 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:35:13.812 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:35:13.812 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:35:13.812 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:35:13.812 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:35:13.812 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:13.812 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:13.812 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.0", 2021-02-10T11:35:13.813 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCSvyNgXcKPLhAAP/Bj2F+06DCtPCzQDNq+cQ==", 2021-02-10T11:35:13.813 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:35:13.813 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:35:13.813 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:35:13.813 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:35:13.813 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:35:13.814 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:13.814 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:13.814 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.1", 2021-02-10T11:35:13.814 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCVvyNghfbcLhAA66PrRwR2M7o6VTF5UhU/Fg==", 2021-02-10T11:35:13.814 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:35:13.815 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:35:13.815 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:35:13.815 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:35:13.815 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:35:13.815 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:13.816 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:13.816 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.2", 2021-02-10T11:35:13.816 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCYvyNgwiPqLhAA66BVkRsgaUlGpAsQ15Nf8Q==", 2021-02-10T11:35:13.816 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:35:13.816 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:35:13.816 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:35:13.817 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:35:13.817 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:35:13.817 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:13.817 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:13.817 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.3", 2021-02-10T11:35:13.817 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCbvyNgqBZrLxAAERPe/1L3SXAEzI3d9ReHHg==", 2021-02-10T11:35:13.818 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:35:13.818 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:35:13.818 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:35:13.818 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:35:13.818 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:35:13.818 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:13.819 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:13.819 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.4", 2021-02-10T11:35:13.819 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCkvyNgDQsxIhAAHmVjVYnx20N4SDmKqvbcuA==", 2021-02-10T11:35:13.819 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:35:13.819 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:35:13.819 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:35:13.820 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:35:13.820 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:35:13.820 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:13.820 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:13.820 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.5", 2021-02-10T11:35:13.820 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCnvyNg8ITFIhAADd4aKybAV7btoGPx3kCg3A==", 2021-02-10T11:35:13.821 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:35:13.821 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:35:13.821 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:35:13.821 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:35:13.821 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:35:13.821 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:13.822 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:13.822 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.6", 2021-02-10T11:35:13.822 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCqvyNgxOhDIxAAzXixEa/nlDc1xXrg0aeLwg==", 2021-02-10T11:35:13.822 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:35:13.822 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:35:13.822 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:35:13.823 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:35:13.823 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:35:13.823 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:13.823 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:13.823 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.7", 2021-02-10T11:35:13.823 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCtvyNgxSqGIxAACxJQlwP7DpiVVCZHA3zn1A==", 2021-02-10T11:35:13.823 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:35:13.824 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:35:13.824 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:35:13.824 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:35:13.824 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:35:13.824 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:13.824 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:13.825 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.0", 2021-02-10T11:35:13.825 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCLvyNgDkNiHRAAygU8Za4PBusTqCgwAYWTiQ==", 2021-02-10T11:35:13.825 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:35:13.825 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:35:13.825 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow r", 2021-02-10T11:35:13.825 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:35:13.826 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:35:13.826 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:13.826 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:13.826 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.1", 2021-02-10T11:35:13.826 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCLvyNgxmLqMxAA1MlSDJ9LH9u4f2sL7mm8/Q==", 2021-02-10T11:35:13.826 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:35:13.827 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:35:13.827 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow r", 2021-02-10T11:35:13.827 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:35:13.827 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:35:13.827 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:13.827 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:13.828 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.2", 2021-02-10T11:35:13.828 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCMvyNgp8veDhAAniSVHYPtwwTog/pjbaRQmw==", 2021-02-10T11:35:13.828 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:35:13.828 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:35:13.828 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow r", 2021-02-10T11:35:13.828 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:35:13.829 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:35:13.829 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:13.829 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:13.829 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.3", 2021-02-10T11:35:13.829 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCMvyNgQEVoJRAApHcWdm1sldIxu+heyD4HdQ==", 2021-02-10T11:35:13.829 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:35:13.829 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:35:13.830 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow r", 2021-02-10T11:35:13.830 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:35:13.830 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:35:13.830 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:13.830 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:13.830 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.admin", 2021-02-10T11:35:13.830 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCGvyNg7jjJBRAAtot+ypUZOiTo1GbY+oVhVw==", 2021-02-10T11:35:13.830 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:35:13.830 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow *", 2021-02-10T11:35:13.831 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:35:13.831 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:35:13.831 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:35:13.831 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:35:13.831 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:13.831 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:13.831 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.alice", 2021-02-10T11:35:13.831 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQDxxCNg5hUmChAArexyCsZP7RgHhs6rSnmt2Q==", 2021-02-10T11:35:13.831 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:35:13.831 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow rw path=/volumes/group_0000000000485992/subvolume_0000000000513667/7bc8264f-d88f-49e9-b22f-5fa8b916f683", 2021-02-10T11:35:13.831 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow r", 2021-02-10T11:35:13.831 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:35:13.831 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:35:13.832 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:13.832 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:13.832 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "mgr.x", 2021-02-10T11:35:13.832 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCJvyNgTPVwKBAAjduiBE8+RSXBv82tdPecWg==", 2021-02-10T11:35:13.832 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:35:13.832 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow *", 2021-02-10T11:35:13.832 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow profile mgr", 2021-02-10T11:35:13.832 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:35:13.832 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:35:13.832 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:13.832 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:13.832 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "mgr.y", 2021-02-10T11:35:13.832 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCJvyNgQ1tTDhAASwBssDy+Rz31ujw8fNlWow==", 2021-02-10T11:35:13.832 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:35:13.833 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow *", 2021-02-10T11:35:13.833 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow profile mgr", 2021-02-10T11:35:13.833 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:35:13.833 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:35:13.833 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:35:13.833 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:35:13.833 INFO:teuthology.orchestra.run.gibba012.stdout:} 2021-02-10T11:35:13.835 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:13.823 7f0d6d1af700 1 -- 172.21.2.112:0/3845879244 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0d4c0186c0 msgr2=0x7f0d4c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:13.835 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:13.823 7f0d6d1af700 1 --2- 172.21.2.112:0/3845879244 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0d4c0186c0 0x7f0d4c01abc0 crc :-1 s=READY pgs=710 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:13.835 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:13.824 7f0d6d1af700 1 -- 172.21.2.112:0/3845879244 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0d681a4a60 msgr2=0x7f0d680c1c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:13.835 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:13.824 7f0d6d1af700 1 --2- 172.21.2.112:0/3845879244 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0d681a4a60 0x7f0d680c1c70 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 rx=0x7f0d5800a5d0 tx=0x7f0d5800a970).stop 2021-02-10T11:35:13.835 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:13.824 7f0d6d1af700 1 -- 172.21.2.112:0/3845879244 shutdown_connections 2021-02-10T11:35:13.836 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:13.824 7f0d6d1af700 1 --2- 172.21.2.112:0/3845879244 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0d4c0186c0 0x7f0d4c01abc0 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:13.836 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:13.824 7f0d6d1af700 1 --2- 172.21.2.112:0/3845879244 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0d680c2290 0x7f0d680c2690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:13.836 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:13.824 7f0d6d1af700 1 --2- 172.21.2.112:0/3845879244 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0d680c2cb0 0x7f0d680bfd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:13.836 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:13.824 7f0d6d1af700 1 --2- 172.21.2.112:0/3845879244 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0d681a4a60 0x7f0d680c1c70 unknown :-1 s=CLOSED pgs=1462 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:13.836 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:13.824 7f0d6d1af700 1 -- 172.21.2.112:0/3845879244 >> 172.21.2.112:0/3845879244 conn(0x7f0d680ebe30 msgr2=0x7f0d680bf200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:13.836 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:13.824 7f0d6d1af700 1 -- 172.21.2.112:0/3845879244 shutdown_connections 2021-02-10T11:35:13.837 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:13.824 7f0d6d1af700 1 -- 172.21.2.112:0/3845879244 wait complete. 2021-02-10T11:35:13.837 DEBUG:teuthology.orchestra.run.gibba012:> set -ex 2021-02-10T11:35:13.838 DEBUG:teuthology.orchestra.run.gibba012:> sudo dd of=/etc/ceph/ceph.client.alice.keyring 2021-02-10T11:35:14.183 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set client.alice debug_client 20 2021-02-10T11:35:14.615 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:14.612 7f11d42e8700 1 Processor -- start 2021-02-10T11:35:14.616 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:14.612 7f11d42e8700 1 -- start start 2021-02-10T11:35:14.616 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:14.613 7f11d42e8700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f11cc0d1c70 0x7f11cc0cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:14.616 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:14.613 7f11d42e8700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f11cc0cf580 0x7f11cc0cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:14.616 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:14.613 7f11d42e8700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11cc0fe1c0 con 0x7f11cc0d1870 2021-02-10T11:35:14.616 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:14.613 7f11d42e8700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11cc1b0cb0 con 0x7f11cc0cf580 2021-02-10T11:35:14.616 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:14.613 7f11d42e8700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11cc109a60 con 0x7f11cc0d1c70 2021-02-10T11:35:14.616 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:14.613 7f11d2905700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f11cc0cf580 0x7f11cc0cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:14.616 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:14.613 7f11d2104700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f11cc0d1870 0x7f11cc1b04c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:50554/0 (socket says 172.21.2.112:50554) 2021-02-10T11:35:14.616 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:14.613 7f11d2104700 1 -- 172.21.2.112:0/614579597 learned_addr learned my addr 172.21.2.112:0/614579597 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:35:14.617 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:14.613 7f11d1903700 1 --2- 172.21.2.112:0/614579597 >> v2:172.21.2.120:3300/0 conn(0x7f11cc0d1c70 0x7f11cc0cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:14.617 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:14.614 7f11d2905700 1 -- 172.21.2.112:0/614579597 >> v2:172.21.2.120:3300/0 conn(0x7f11cc0d1c70 msgr2=0x7f11cc0cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:14.617 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:14.614 7f11d2905700 1 --2- 172.21.2.112:0/614579597 >> v2:172.21.2.120:3300/0 conn(0x7f11cc0d1c70 0x7f11cc0cef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:14.617 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:14.614 7f11d2905700 1 -- 172.21.2.112:0/614579597 >> v1:172.21.2.112:6789/0 conn(0x7f11cc0d1870 legacy=0x7f11cc1b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:14.617 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:14.614 7f11d2905700 1 -- 172.21.2.112:0/614579597 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11cc088070 con 0x7f11cc0cf580 2021-02-10T11:35:14.617 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:14.614 7f11d1903700 1 --2- 172.21.2.112:0/614579597 >> v2:172.21.2.120:3300/0 conn(0x7f11cc0d1c70 0x7f11cc0cef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:35:14.617 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:14.614 7f11d2905700 1 --2- 172.21.2.112:0/614579597 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f11cc0cf580 0x7f11cc0cf980 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 rx=0x7f11c4006e30 tx=0x7f11c4009e60).ready entity=mon.0 client_cookie=287fb87990522c76 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:14.617 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:14.614 7f11d1102700 1 -- 172.21.2.112:0/614579597 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f11c4006a50 con 0x7f11cc0cf580 2021-02-10T11:35:14.617 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:14.614 7f11d1102700 1 -- 172.21.2.112:0/614579597 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11c400b300 con 0x7f11cc0cf580 2021-02-10T11:35:14.617 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:14.614 7f11d1102700 1 -- 172.21.2.112:0/614579597 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f11c400b820 con 0x7f11cc0cf580 2021-02-10T11:35:14.617 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:14.614 7f11d42e8700 1 -- 172.21.2.112:0/614579597 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f11cc0cf580 msgr2=0x7f11cc0cf980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:14.617 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:14.614 7f11d42e8700 1 --2- 172.21.2.112:0/614579597 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f11cc0cf580 0x7f11cc0cf980 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 rx=0x7f11c4006e30 tx=0x7f11c4009e60).stop 2021-02-10T11:35:14.618 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:14.615 7f11d42e8700 1 -- 172.21.2.112:0/614579597 shutdown_connections 2021-02-10T11:35:14.618 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:14.615 7f11d42e8700 1 --2- 172.21.2.112:0/614579597 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f11cc0cf580 0x7f11cc0cf980 unknown :-1 s=CLOSED pgs=1464 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:14.618 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:14.615 7f11d42e8700 1 --2- 172.21.2.112:0/614579597 >> v2:172.21.2.120:3300/0 conn(0x7f11cc0d1c70 0x7f11cc0cef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:14.618 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:14.615 7f11d42e8700 1 -- 172.21.2.112:0/614579597 >> 172.21.2.112:0/614579597 conn(0x7f11cc0ce420 msgr2=0x7f11cc0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:14.618 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:14.615 7f11d42e8700 1 -- 172.21.2.112:0/614579597 shutdown_connections 2021-02-10T11:35:14.618 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:14.615 7f11d42e8700 1 -- 172.21.2.112:0/614579597 wait complete. 2021-02-10T11:35:14.620 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:14.629 7f11d42e8700 1 Processor -- start 2021-02-10T11:35:14.620 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:14.629 7f11d42e8700 1 -- start start 2021-02-10T11:35:14.620 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:14.630 7f11d42e8700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f11cc0a7950 0x7f11cc0b8df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:14.620 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:14.630 7f11d42e8700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f11cc1a7490 0x7f11cc1a7890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:14.620 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:14.630 7f11d42e8700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f11cc1a7eb0 0x7f11cc0ac630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:14.620 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:14.630 7f11d42e8700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11cc0a7fb0 con 0x7f11cc1a7eb0 2021-02-10T11:35:14.620 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:14.630 7f11d42e8700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11cc0acc50 con 0x7f11cc1a7490 2021-02-10T11:35:14.620 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:14.630 7f11d42e8700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11cc0ace30 con 0x7f11cc0a7950 2021-02-10T11:35:14.620 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:14.630 7f11d2905700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f11cc1a7eb0 0x7f11cc0ac630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:14.620 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:14.630 7f11d2905700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f11cc1a7eb0 0x7f11cc0ac630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:51098/0 (socket says 172.21.2.112:51098) 2021-02-10T11:35:14.621 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:14.630 7f11d2905700 1 -- 172.21.2.112:0/3972040423 learned_addr learned my addr 172.21.2.112:0/3972040423 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:35:14.621 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:14.630 7f11d2905700 1 -- 172.21.2.112:0/3972040423 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f11cc0a7950 msgr2=0x7f11cc0b8df0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:35:14.621 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:14.630 7f11d2905700 1 --2- 172.21.2.112:0/3972040423 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f11cc0a7950 0x7f11cc0b8df0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:14.621 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:14.630 7f11d2905700 1 -- 172.21.2.112:0/3972040423 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f11cc1a7490 msgr2=0x7f11cc1a7890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:35:14.621 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:14.630 7f11d2905700 1 --2- 172.21.2.112:0/3972040423 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f11cc1a7490 0x7f11cc1a7890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:14.621 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:14.630 7f11d2905700 1 -- 172.21.2.112:0/3972040423 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11cc0ad050 con 0x7f11cc1a7eb0 2021-02-10T11:35:14.621 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:14.630 7f11d2905700 1 --2- 172.21.2.112:0/3972040423 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f11cc1a7eb0 0x7f11cc0ac630 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 rx=0x7f11c400b970 tx=0x7f11c400bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:14.621 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:14.631 7f11beffd700 1 -- 172.21.2.112:0/3972040423 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f11c4009380 con 0x7f11cc1a7eb0 2021-02-10T11:35:14.621 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:14.631 7f11beffd700 1 -- 172.21.2.112:0/3972040423 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11c4005820 con 0x7f11cc1a7eb0 2021-02-10T11:35:14.621 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:14.631 7f11beffd700 1 -- 172.21.2.112:0/3972040423 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f11c4005d00 con 0x7f11cc1a7eb0 2021-02-10T11:35:14.621 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:14.631 7f11d42e8700 1 -- 172.21.2.112:0/3972040423 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11cc0ad050 con 0x7f11cc1a7eb0 2021-02-10T11:35:14.621 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:14.631 7f11d42e8700 1 -- 172.21.2.112:0/3972040423 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11cc0a8530 con 0x7f11cc1a7eb0 2021-02-10T11:35:14.621 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:14.631 7f11beffd700 1 -- 172.21.2.112:0/3972040423 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f11c4005a00 con 0x7f11cc1a7eb0 2021-02-10T11:35:14.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:14.631 7f11beffd700 1 --2- 172.21.2.112:0/3972040423 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f11b40186c0 0x7f11b401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:14.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:14.631 7f11beffd700 1 -- 172.21.2.112:0/3972040423 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f11c4032950 con 0x7f11cc1a7eb0 2021-02-10T11:35:14.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:14.632 7f11d2104700 1 --2- 172.21.2.112:0/3972040423 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f11b40186c0 0x7f11b401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:14.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:14.632 7f11d2104700 1 --2- 172.21.2.112:0/3972040423 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f11b40186c0 0x7f11b401abc0 crc :-1 s=READY pgs=711 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:14.649 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:14.663 7f11d42e8700 1 -- 172.21.2.112:0/3972040423 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11cc082a70 con 0x7f11cc1a7eb0 2021-02-10T11:35:14.652 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:14.665 7f11beffd700 1 -- 172.21.2.112:0/3972040423 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f11c4006030 con 0x7f11cc1a7eb0 2021-02-10T11:35:14.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:14.813 7f11d42e8700 1 -- 172.21.2.112:0/3972040423 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command([{prefix=config set, name=debug_client}] v 0) v1 -- 0x7f11cc0a84f0 con 0x7f11cc1a7eb0 2021-02-10T11:35:14.803 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:14.817 7f11beffd700 1 -- 172.21.2.112:0/3972040423 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{prefix=config set, name=debug_client}]=0 v15)=0 v15) v1 ==== 120+0+0 (secure 0 0 0) 0x7f11c4006ee0 con 0x7f11cc1a7eb0 2021-02-10T11:35:14.808 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:14.822 7f11d42e8700 1 -- 172.21.2.112:0/3972040423 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f11b40186c0 msgr2=0x7f11b401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:14.809 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:14.822 7f11d42e8700 1 --2- 172.21.2.112:0/3972040423 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f11b40186c0 0x7f11b401abc0 crc :-1 s=READY pgs=711 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:14.809 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:14.822 7f11d42e8700 1 -- 172.21.2.112:0/3972040423 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f11cc1a7eb0 msgr2=0x7f11cc0ac630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:14.809 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:14.822 7f11d42e8700 1 --2- 172.21.2.112:0/3972040423 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f11cc1a7eb0 0x7f11cc0ac630 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 rx=0x7f11c400b970 tx=0x7f11c400bd10).stop 2021-02-10T11:35:14.809 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:14.822 7f11d42e8700 1 -- 172.21.2.112:0/3972040423 shutdown_connections 2021-02-10T11:35:14.809 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:14.822 7f11d42e8700 1 --2- 172.21.2.112:0/3972040423 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f11b40186c0 0x7f11b401abc0 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:14.809 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:14.822 7f11d42e8700 1 --2- 172.21.2.112:0/3972040423 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f11cc1a7490 0x7f11cc1a7890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:14.810 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:14.822 7f11d42e8700 1 --2- 172.21.2.112:0/3972040423 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f11cc1a7eb0 0x7f11cc0ac630 unknown :-1 s=CLOSED pgs=1465 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:14.810 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:14.822 7f11d42e8700 1 --2- 172.21.2.112:0/3972040423 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f11cc0a7950 0x7f11cc0b8df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:14.810 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:14.822 7f11d42e8700 1 -- 172.21.2.112:0/3972040423 >> 172.21.2.112:0/3972040423 conn(0x7f11cc0a02d0 msgr2=0x7f11cc09f200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:14.810 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:14.822 7f11d42e8700 1 -- 172.21.2.112:0/3972040423 shutdown_connections 2021-02-10T11:35:14.810 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:14.822 7f11d42e8700 1 -- 172.21.2.112:0/3972040423 wait complete. 2021-02-10T11:35:14.817 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set client.alice debug_objecter 20 2021-02-10T11:35:15.245 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:15.243 7f80ce944700 1 Processor -- start 2021-02-10T11:35:15.245 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:15.243 7f80ce944700 1 -- start start 2021-02-10T11:35:15.245 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:15.243 7f80ce944700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f80c80d1870 0x7f80c81b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:15.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:15.243 7f80ce944700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f80c81b0ae0 0x7f80c8108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:15.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:15.243 7f80ce944700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80c80fe1c0 con 0x7f80c80d1c70 2021-02-10T11:35:15.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:15.243 7f80ce944700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80c8109a60 con 0x7f80c80d1870 2021-02-10T11:35:15.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:15.243 7f80ce944700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80c8109750 con 0x7f80c81b0ae0 2021-02-10T11:35:15.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:15.243 7f80c7fff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f80c80d1870 0x7f80c81b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:15.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:15.243 7f80c7fff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f80c80d1870 0x7f80c81b04c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:51106/0 (socket says 172.21.2.112:51106) 2021-02-10T11:35:15.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:15.243 7f80c7fff700 1 -- 172.21.2.112:0/1526048848 learned_addr learned my addr 172.21.2.112:0/1526048848 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:35:15.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:15.244 7f80c7fff700 1 -- 172.21.2.112:0/1526048848 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f80c81b0ae0 msgr2=0x7f80c8108940 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:35:15.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:15.244 7f80c7fff700 1 --2- 172.21.2.112:0/1526048848 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f80c81b0ae0 0x7f80c8108940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:15.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:15.244 7f80c7fff700 1 -- 172.21.2.112:0/1526048848 >> v1:172.21.2.120:6789/0 conn(0x7f80c80d1c70 legacy=0x7f80c8108f60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:35:15.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:15.244 7f80c7fff700 1 -- 172.21.2.112:0/1526048848 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80c8088070 con 0x7f80c80d1870 2021-02-10T11:35:15.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:15.244 7f80c7fff700 1 --2- 172.21.2.112:0/1526048848 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f80c80d1870 0x7f80c81b04c0 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 rx=0x7f80b8006700 tx=0x7f80b8009b00).ready entity=mon.0 client_cookie=b6928090be262561 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:15.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:15.244 7f80c6ffd700 1 -- 172.21.2.112:0/1526048848 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f80b8000c90 con 0x7f80c80d1870 2021-02-10T11:35:15.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:15.244 7f80c6ffd700 1 -- 172.21.2.112:0/1526048848 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80b800c0c0 con 0x7f80c80d1870 2021-02-10T11:35:15.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:15.244 7f80c6ffd700 1 -- 172.21.2.112:0/1526048848 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f80b800c5e0 con 0x7f80c80d1870 2021-02-10T11:35:15.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:15.244 7f80ce944700 1 -- 172.21.2.112:0/1526048848 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f80c80d1870 msgr2=0x7f80c81b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:15.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:15.244 7f80ce944700 1 --2- 172.21.2.112:0/1526048848 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f80c80d1870 0x7f80c81b04c0 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 rx=0x7f80b8006700 tx=0x7f80b8009b00).stop 2021-02-10T11:35:15.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:15.245 7f80ce944700 1 -- 172.21.2.112:0/1526048848 shutdown_connections 2021-02-10T11:35:15.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:15.245 7f80ce944700 1 --2- 172.21.2.112:0/1526048848 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f80c81b0ae0 0x7f80c8108940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:15.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:15.245 7f80ce944700 1 --2- 172.21.2.112:0/1526048848 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f80c80d1870 0x7f80c81b04c0 unknown :-1 s=CLOSED pgs=1466 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:15.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:15.245 7f80ce944700 1 -- 172.21.2.112:0/1526048848 >> 172.21.2.112:0/1526048848 conn(0x7f80c80ce420 msgr2=0x7f80c80d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:15.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:15.245 7f80ce944700 1 -- 172.21.2.112:0/1526048848 shutdown_connections 2021-02-10T11:35:15.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:15.245 7f80ce944700 1 -- 172.21.2.112:0/1526048848 wait complete. 2021-02-10T11:35:15.249 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:15.259 7f80ce944700 1 Processor -- start 2021-02-10T11:35:15.249 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:15.259 7f80ce944700 1 -- start start 2021-02-10T11:35:15.249 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:15.259 7f80ce944700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f80c80a6e70 0x7f80c80bc4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:15.249 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:15.259 7f80ce944700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f80c80ba4d0 0x7f80c80c2f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:15.249 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:15.259 7f80ce944700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f80c80c3520 0x7f80c80bb480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:15.249 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:15.259 7f80ce944700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80c80baf00 con 0x7f80c80c3520 2021-02-10T11:35:15.250 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:15.259 7f80ce944700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80c80bcb00 con 0x7f80c80ba4d0 2021-02-10T11:35:15.250 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:15.259 7f80ce944700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80c80bbaa0 con 0x7f80c80a6e70 2021-02-10T11:35:15.250 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:15.260 7f80ccf61700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f80c80c3520 0x7f80c80bb480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:15.250 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:15.260 7f80ccf61700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f80c80c3520 0x7f80c80bb480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:51112/0 (socket says 172.21.2.112:51112) 2021-02-10T11:35:15.250 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:15.260 7f80ccf61700 1 -- 172.21.2.112:0/309923683 learned_addr learned my addr 172.21.2.112:0/309923683 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:35:15.250 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:15.260 7f80c7fff700 1 --2- 172.21.2.112:0/309923683 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f80c80a6e70 0x7f80c80bc4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:15.250 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:15.260 7f80c77fe700 1 --2- 172.21.2.112:0/309923683 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f80c80ba4d0 0x7f80c80c2f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:15.250 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:15.260 7f80ccf61700 1 -- 172.21.2.112:0/309923683 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f80c80a6e70 msgr2=0x7f80c80bc4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:15.250 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:15.260 7f80ccf61700 1 --2- 172.21.2.112:0/309923683 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f80c80a6e70 0x7f80c80bc4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:15.250 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:15.260 7f80ccf61700 1 -- 172.21.2.112:0/309923683 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f80c80ba4d0 msgr2=0x7f80c80c2f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:15.250 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:15.260 7f80ccf61700 1 --2- 172.21.2.112:0/309923683 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f80c80ba4d0 0x7f80c80c2f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:15.250 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:15.260 7f80ccf61700 1 -- 172.21.2.112:0/309923683 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80c80bbcc0 con 0x7f80c80c3520 2021-02-10T11:35:15.250 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:15.260 7f80c7fff700 1 --2- 172.21.2.112:0/309923683 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f80c80a6e70 0x7f80c80bc4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:35:15.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:15.260 7f80c77fe700 1 --2- 172.21.2.112:0/309923683 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f80c80ba4d0 0x7f80c80c2f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:35:15.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:15.260 7f80ccf61700 1 --2- 172.21.2.112:0/309923683 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f80c80c3520 0x7f80c80bb480 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 rx=0x7f80c0009940 tx=0x7f80c000bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:15.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:15.260 7f80c4ff9700 1 -- 172.21.2.112:0/309923683 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f80c0003030 con 0x7f80c80c3520 2021-02-10T11:35:15.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:15.260 7f80c4ff9700 1 -- 172.21.2.112:0/309923683 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80c0001270 con 0x7f80c80c3520 2021-02-10T11:35:15.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:15.260 7f80c4ff9700 1 -- 172.21.2.112:0/309923683 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f80c0001750 con 0x7f80c80c3520 2021-02-10T11:35:15.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:15.260 7f80ce944700 1 -- 172.21.2.112:0/309923683 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80c80bbcc0 con 0x7f80c80c3520 2021-02-10T11:35:15.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:15.261 7f80ce944700 1 -- 172.21.2.112:0/309923683 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80c80bbcc0 con 0x7f80c80c3520 2021-02-10T11:35:15.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:15.261 7f80c4ff9700 1 -- 172.21.2.112:0/309923683 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f80c000a000 con 0x7f80c80c3520 2021-02-10T11:35:15.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:15.261 7f80c4ff9700 1 --2- 172.21.2.112:0/309923683 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f80a80186d0 0x7f80a801abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:15.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:15.261 7f80c4ff9700 1 -- 172.21.2.112:0/309923683 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f80c00329f0 con 0x7f80c80c3520 2021-02-10T11:35:15.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:15.261 7f80c7fff700 1 --2- 172.21.2.112:0/309923683 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f80a80186d0 0x7f80a801abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:15.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:15.262 7f80c7fff700 1 --2- 172.21.2.112:0/309923683 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f80a80186d0 0x7f80a801abd0 crc :-1 s=READY pgs=712 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:15.279 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:15.293 7f80ce944700 1 -- 172.21.2.112:0/309923683 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80c8082a70 con 0x7f80c80c3520 2021-02-10T11:35:15.283 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:15.295 7f80c4ff9700 1 -- 172.21.2.112:0/309923683 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f80c000be30 con 0x7f80c80c3520 2021-02-10T11:35:15.429 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:15.442 7f80ce944700 1 -- 172.21.2.112:0/309923683 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command([{prefix=config set, name=debug_objecter}] v 0) v1 -- 0x7f80c8082a70 con 0x7f80c80c3520 2021-02-10T11:35:15.433 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:15.446 7f80c4ff9700 1 -- 172.21.2.112:0/309923683 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{prefix=config set, name=debug_objecter}]=0 v16)=0 v16) v1 ==== 122+0+0 (secure 0 0 0) 0x7f80c000a000 con 0x7f80c80c3520 2021-02-10T11:35:15.438 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:15.450 7f80ce944700 1 -- 172.21.2.112:0/309923683 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f80a80186d0 msgr2=0x7f80a801abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:15.438 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:15.451 7f80ce944700 1 --2- 172.21.2.112:0/309923683 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f80a80186d0 0x7f80a801abd0 crc :-1 s=READY pgs=712 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:15.439 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:15.451 7f80ce944700 1 -- 172.21.2.112:0/309923683 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f80c80c3520 msgr2=0x7f80c80bb480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:15.439 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:15.451 7f80ce944700 1 --2- 172.21.2.112:0/309923683 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f80c80c3520 0x7f80c80bb480 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 rx=0x7f80c0009940 tx=0x7f80c000bcf0).stop 2021-02-10T11:35:15.439 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:15.451 7f80ce944700 1 -- 172.21.2.112:0/309923683 shutdown_connections 2021-02-10T11:35:15.439 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:15.451 7f80ce944700 1 --2- 172.21.2.112:0/309923683 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f80a80186d0 0x7f80a801abd0 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:15.439 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:15.451 7f80ce944700 1 --2- 172.21.2.112:0/309923683 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f80c80ba4d0 0x7f80c80c2f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:15.439 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:15.451 7f80ce944700 1 --2- 172.21.2.112:0/309923683 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f80c80c3520 0x7f80c80bb480 unknown :-1 s=CLOSED pgs=1467 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:15.439 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:15.451 7f80ce944700 1 --2- 172.21.2.112:0/309923683 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f80c80a6e70 0x7f80c80bc4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:15.439 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:15.451 7f80ce944700 1 -- 172.21.2.112:0/309923683 >> 172.21.2.112:0/309923683 conn(0x7f80c80c3a90 msgr2=0x7f80c80926d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:15.440 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:15.451 7f80ce944700 1 -- 172.21.2.112:0/309923683 shutdown_connections 2021-02-10T11:35:15.440 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:15.451 7f80ce944700 1 -- 172.21.2.112:0/309923683 wait complete. 2021-02-10T11:35:15.447 DEBUG:teuthology.orchestra.run.gibba012:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2021-02-10T11:35:15.450 DEBUG:teuthology.orchestra.run.gibba020:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2021-02-10T11:35:15.811 INFO:teuthology.orchestra.run.gibba012.stdout:[global] 2021-02-10T11:35:15.811 INFO:teuthology.orchestra.run.gibba012.stdout: chdir = "" 2021-02-10T11:35:15.811 INFO:teuthology.orchestra.run.gibba012.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2021-02-10T11:35:15.811 INFO:teuthology.orchestra.run.gibba012.stdout: auth supported = cephx 2021-02-10T11:35:15.811 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:35:15.811 INFO:teuthology.orchestra.run.gibba012.stdout: filestore xattr use omap = true 2021-02-10T11:35:15.812 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:35:15.812 INFO:teuthology.orchestra.run.gibba012.stdout: mon clock drift allowed = 1.000 2021-02-10T11:35:15.812 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:35:15.812 INFO:teuthology.orchestra.run.gibba012.stdout: osd crush chooseleaf type = 0 2021-02-10T11:35:15.812 INFO:teuthology.orchestra.run.gibba012.stdout: auth debug = true 2021-02-10T11:35:15.812 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:35:15.812 INFO:teuthology.orchestra.run.gibba012.stdout: ms die on old message = true 2021-02-10T11:35:15.812 INFO:teuthology.orchestra.run.gibba012.stdout: ms die on bug = true 2021-02-10T11:35:15.812 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:35:15.812 INFO:teuthology.orchestra.run.gibba012.stdout: mon max pg per osd = 10000 # >= luminous 2021-02-10T11:35:15.812 INFO:teuthology.orchestra.run.gibba012.stdout: mon pg warn max object skew = 0 2021-02-10T11:35:15.812 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:35:15.812 INFO:teuthology.orchestra.run.gibba012.stdout: osd pool default size = 2 2021-02-10T11:35:15.812 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:35:15.813 INFO:teuthology.orchestra.run.gibba012.stdout: mon osd allow primary affinity = true 2021-02-10T11:35:15.813 INFO:teuthology.orchestra.run.gibba012.stdout: mon osd allow pg remap = true 2021-02-10T11:35:15.813 INFO:teuthology.orchestra.run.gibba012.stdout: mon warn on legacy crush tunables = false 2021-02-10T11:35:15.813 INFO:teuthology.orchestra.run.gibba012.stdout: mon warn on crush straw calc version zero = false 2021-02-10T11:35:15.813 INFO:teuthology.orchestra.run.gibba012.stdout: mon warn on no sortbitwise = false 2021-02-10T11:35:15.813 INFO:teuthology.orchestra.run.gibba012.stdout: mon warn on osd down out interval zero = false 2021-02-10T11:35:15.813 INFO:teuthology.orchestra.run.gibba012.stdout: mon warn on too few osds = false 2021-02-10T11:35:15.813 INFO:teuthology.orchestra.run.gibba012.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2021-02-10T11:35:15.813 INFO:teuthology.orchestra.run.gibba012.stdout: mon_warn_on_pool_no_redundancy = false 2021-02-10T11:35:15.813 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:35:15.813 INFO:teuthology.orchestra.run.gibba012.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2021-02-10T11:35:15.813 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:35:15.813 INFO:teuthology.orchestra.run.gibba012.stdout: osd default data pool replay window = 5 2021-02-10T11:35:15.814 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:35:15.814 INFO:teuthology.orchestra.run.gibba012.stdout: mon allow pool delete = true 2021-02-10T11:35:15.814 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:35:15.814 INFO:teuthology.orchestra.run.gibba012.stdout: mon cluster log file level = debug 2021-02-10T11:35:15.814 INFO:teuthology.orchestra.run.gibba012.stdout: debug asserts on shutdown = true 2021-02-10T11:35:15.814 INFO:teuthology.orchestra.run.gibba012.stdout: mon health detail to clog = false 2021-02-10T11:35:15.814 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:35:15.814 INFO:teuthology.orchestra.run.gibba012.stdout: # we see this fail in qa on *nautilus*; bump up retries 2021-02-10T11:35:15.814 INFO:teuthology.orchestra.run.gibba012.stdout: mon_client_directed_command_retry = 4 2021-02-10T11:35:15.814 INFO:teuthology.orchestra.run.gibba012.stdout: mon host = "172.21.2.112,172.21.2.120,[v2:172.21.2.120:3301,v1:172.21.2.120:6790]" 2021-02-10T11:35:15.814 INFO:teuthology.orchestra.run.gibba012.stdout: fsid = f14f004e-0699-4584-ad81-3754258493de 2021-02-10T11:35:15.814 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:35:15.814 INFO:teuthology.orchestra.run.gibba012.stdout:[osd] 2021-02-10T11:35:15.815 INFO:teuthology.orchestra.run.gibba012.stdout: osd journal size = 100 2021-02-10T11:35:15.815 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:35:15.815 INFO:teuthology.orchestra.run.gibba012.stdout: osd scrub load threshold = 5.0 2021-02-10T11:35:15.815 INFO:teuthology.orchestra.run.gibba012.stdout: osd scrub max interval = 600 2021-02-10T11:35:15.815 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:35:15.815 INFO:teuthology.orchestra.run.gibba012.stdout: osd recover clone overlap = true 2021-02-10T11:35:15.815 INFO:teuthology.orchestra.run.gibba012.stdout: osd recovery max chunk = 1048576 2021-02-10T11:35:15.815 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:35:15.815 INFO:teuthology.orchestra.run.gibba012.stdout: osd debug shutdown = true 2021-02-10T11:35:15.815 INFO:teuthology.orchestra.run.gibba012.stdout: osd debug op order = true 2021-02-10T11:35:15.815 INFO:teuthology.orchestra.run.gibba012.stdout: osd debug verify stray on activate = true 2021-02-10T11:35:15.815 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:35:15.815 INFO:teuthology.orchestra.run.gibba012.stdout: osd open classes on start = true 2021-02-10T11:35:15.815 INFO:teuthology.orchestra.run.gibba012.stdout: osd debug pg log writeout = true 2021-02-10T11:35:15.816 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:35:15.816 INFO:teuthology.orchestra.run.gibba012.stdout: osd deep scrub update digest min age = 30 2021-02-10T11:35:15.816 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:35:15.816 INFO:teuthology.orchestra.run.gibba012.stdout: osd map max advance = 10 2021-02-10T11:35:15.816 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:35:15.816 INFO:teuthology.orchestra.run.gibba012.stdout: journal zero on create = true 2021-02-10T11:35:15.816 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:35:15.816 INFO:teuthology.orchestra.run.gibba012.stdout: filestore ondisk finisher threads = 3 2021-02-10T11:35:15.816 INFO:teuthology.orchestra.run.gibba012.stdout: filestore apply finisher threads = 3 2021-02-10T11:35:15.816 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:35:15.816 INFO:teuthology.orchestra.run.gibba012.stdout: bdev debug aio = true 2021-02-10T11:35:15.816 INFO:teuthology.orchestra.run.gibba012.stdout: osd debug misdirected ops = true 2021-02-10T11:35:15.816 INFO:teuthology.orchestra.run.gibba012.stdout: bdev async discard = True 2021-02-10T11:35:15.816 INFO:teuthology.orchestra.run.gibba012.stdout: bdev enable discard = True 2021-02-10T11:35:15.816 INFO:teuthology.orchestra.run.gibba012.stdout: bluestore allocator = bitmap 2021-02-10T11:35:15.817 INFO:teuthology.orchestra.run.gibba012.stdout: bluestore block size = 96636764160 2021-02-10T11:35:15.817 INFO:teuthology.orchestra.run.gibba012.stdout: bluestore fsck on mount = True 2021-02-10T11:35:15.817 INFO:teuthology.orchestra.run.gibba012.stdout: debug bluefs = 20 2021-02-10T11:35:15.817 INFO:teuthology.orchestra.run.gibba012.stdout: debug bluestore = 20 2021-02-10T11:35:15.817 INFO:teuthology.orchestra.run.gibba012.stdout: debug filestore = 20 2021-02-10T11:35:15.817 INFO:teuthology.orchestra.run.gibba012.stdout: debug journal = 20 2021-02-10T11:35:15.817 INFO:teuthology.orchestra.run.gibba012.stdout: debug ms = 20 2021-02-10T11:35:15.817 INFO:teuthology.orchestra.run.gibba012.stdout: debug osd = 25 2021-02-10T11:35:15.817 INFO:teuthology.orchestra.run.gibba012.stdout: debug rocksdb = 10 2021-02-10T11:35:15.817 INFO:teuthology.orchestra.run.gibba012.stdout: mon osd backfillfull_ratio = 0.85 2021-02-10T11:35:15.817 INFO:teuthology.orchestra.run.gibba012.stdout: mon osd full ratio = 0.9 2021-02-10T11:35:15.817 INFO:teuthology.orchestra.run.gibba012.stdout: mon osd nearfull ratio = 0.8 2021-02-10T11:35:15.817 INFO:teuthology.orchestra.run.gibba012.stdout: osd failsafe full ratio = 0.95 2021-02-10T11:35:15.818 INFO:teuthology.orchestra.run.gibba012.stdout: osd objectstore = bluestore 2021-02-10T11:35:15.818 INFO:teuthology.orchestra.run.gibba012.stdout: osd op complaint time = 180 2021-02-10T11:35:15.818 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:35:15.818 INFO:teuthology.orchestra.run.gibba012.stdout:[mgr] 2021-02-10T11:35:15.818 INFO:teuthology.orchestra.run.gibba012.stdout: debug ms = 1 2021-02-10T11:35:15.818 INFO:teuthology.orchestra.run.gibba012.stdout: debug mgr = 20 2021-02-10T11:35:15.818 INFO:teuthology.orchestra.run.gibba012.stdout: debug mon = 20 2021-02-10T11:35:15.818 INFO:teuthology.orchestra.run.gibba012.stdout: debug auth = 20 2021-02-10T11:35:15.818 INFO:teuthology.orchestra.run.gibba012.stdout: mon reweight min pgs per osd = 4 2021-02-10T11:35:15.818 INFO:teuthology.orchestra.run.gibba012.stdout: mon reweight min bytes per osd = 10 2021-02-10T11:35:15.818 INFO:teuthology.orchestra.run.gibba012.stdout: debug client = 10 2021-02-10T11:35:15.818 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:35:15.818 INFO:teuthology.orchestra.run.gibba012.stdout:[mon] 2021-02-10T11:35:15.819 INFO:teuthology.orchestra.run.gibba012.stdout: debug ms = 1 2021-02-10T11:35:15.819 INFO:teuthology.orchestra.run.gibba012.stdout: debug mon = 20 2021-02-10T11:35:15.819 INFO:teuthology.orchestra.run.gibba012.stdout: debug paxos = 20 2021-02-10T11:35:15.819 INFO:teuthology.orchestra.run.gibba012.stdout: debug auth = 20 2021-02-10T11:35:15.819 INFO:teuthology.orchestra.run.gibba012.stdout: mon data avail warn = 5 2021-02-10T11:35:15.819 INFO:teuthology.orchestra.run.gibba012.stdout: mon mgr mkfs grace = 240 2021-02-10T11:35:15.819 INFO:teuthology.orchestra.run.gibba012.stdout: mon reweight min pgs per osd = 4 2021-02-10T11:35:15.819 INFO:teuthology.orchestra.run.gibba012.stdout: mon osd reporter subtree level = osd 2021-02-10T11:35:15.819 INFO:teuthology.orchestra.run.gibba012.stdout: mon osd prime pg temp = true 2021-02-10T11:35:15.819 INFO:teuthology.orchestra.run.gibba012.stdout: mon reweight min bytes per osd = 10 2021-02-10T11:35:15.819 INFO:teuthology.orchestra.run.gibba012.stdout: mon op complaint time = 120 2021-02-10T11:35:15.819 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:35:15.819 INFO:teuthology.orchestra.run.gibba012.stdout:[client] 2021-02-10T11:35:15.819 INFO:teuthology.orchestra.run.gibba012.stdout: rgw cache enabled = true 2021-02-10T11:35:15.820 INFO:teuthology.orchestra.run.gibba012.stdout: rgw enable ops log = true 2021-02-10T11:35:15.820 INFO:teuthology.orchestra.run.gibba012.stdout: rgw enable usage log = true 2021-02-10T11:35:15.820 INFO:teuthology.orchestra.run.gibba012.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2021-02-10T11:35:15.821 INFO:teuthology.orchestra.run.gibba012.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2021-02-10T11:35:15.821 INFO:teuthology.orchestra.run.gibba012.stdout: client mount timeout = 600 2021-02-10T11:35:15.821 INFO:teuthology.orchestra.run.gibba012.stdout: debug client = 20 2021-02-10T11:35:15.821 INFO:teuthology.orchestra.run.gibba012.stdout: debug ms = 1 2021-02-10T11:35:15.821 INFO:teuthology.orchestra.run.gibba012.stdout: pid file = "" 2021-02-10T11:35:15.822 INFO:teuthology.orchestra.run.gibba012.stdout:[mon.a] 2021-02-10T11:35:15.822 INFO:teuthology.orchestra.run.gibba012.stdout:[mon.b] 2021-02-10T11:35:15.823 INFO:teuthology.orchestra.run.gibba012.stdout:[mon.c] 2021-02-10T11:35:15.823 INFO:teuthology.orchestra.run.gibba012.stdout:[mds.a] 2021-02-10T11:35:15.823 INFO:teuthology.orchestra.run.gibba012.stdout:[mds.b] 2021-02-10T11:35:15.823 INFO:teuthology.orchestra.run.gibba012.stdout:[mds] 2021-02-10T11:35:15.823 INFO:teuthology.orchestra.run.gibba012.stdout: debug mds = 20 2021-02-10T11:35:15.823 INFO:teuthology.orchestra.run.gibba012.stdout: debug ms = 1 2021-02-10T11:35:15.823 INFO:teuthology.orchestra.run.gibba012.stdout: mds bal frag = True 2021-02-10T11:35:15.823 INFO:teuthology.orchestra.run.gibba012.stdout: mds bal fragment size max = 10000 2021-02-10T11:35:15.824 INFO:teuthology.orchestra.run.gibba012.stdout: mds bal merge size = 5 2021-02-10T11:35:15.824 INFO:teuthology.orchestra.run.gibba012.stdout: mds bal split bits = 3 2021-02-10T11:35:15.824 INFO:teuthology.orchestra.run.gibba012.stdout: mds bal split size = 100 2021-02-10T11:35:15.824 INFO:teuthology.orchestra.run.gibba012.stdout: mds debug frag = True 2021-02-10T11:35:15.824 INFO:teuthology.orchestra.run.gibba012.stdout: mds debug scatterstat = True 2021-02-10T11:35:15.824 INFO:teuthology.orchestra.run.gibba012.stdout: mds op complaint time = 180 2021-02-10T11:35:15.824 INFO:teuthology.orchestra.run.gibba012.stdout: mds verify scatter = True 2021-02-10T11:35:15.824 INFO:teuthology.orchestra.run.gibba012.stdout: osd op complaint time = 180 2021-02-10T11:35:15.824 INFO:teuthology.orchestra.run.gibba012.stdout:[client.alice] 2021-02-10T11:35:15.824 INFO:teuthology.orchestra.run.gibba012.stdout: keyring = /etc/ceph/ceph.client.alice.keyring 2021-02-10T11:35:15.825 INFO:teuthology.orchestra.run.gibba020.stdout:[global] 2021-02-10T11:35:15.825 INFO:teuthology.orchestra.run.gibba020.stdout: chdir = "" 2021-02-10T11:35:15.826 INFO:teuthology.orchestra.run.gibba020.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2021-02-10T11:35:15.826 INFO:teuthology.orchestra.run.gibba020.stdout: auth supported = cephx 2021-02-10T11:35:15.826 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:35:15.826 INFO:teuthology.orchestra.run.gibba020.stdout: filestore xattr use omap = true 2021-02-10T11:35:15.826 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:35:15.826 INFO:teuthology.orchestra.run.gibba020.stdout: mon clock drift allowed = 1.000 2021-02-10T11:35:15.826 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:35:15.826 INFO:teuthology.orchestra.run.gibba020.stdout: osd crush chooseleaf type = 0 2021-02-10T11:35:15.826 INFO:teuthology.orchestra.run.gibba020.stdout: auth debug = true 2021-02-10T11:35:15.826 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:35:15.826 INFO:teuthology.orchestra.run.gibba020.stdout: ms die on old message = true 2021-02-10T11:35:15.826 INFO:teuthology.orchestra.run.gibba020.stdout: ms die on bug = true 2021-02-10T11:35:15.826 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:35:15.827 INFO:teuthology.orchestra.run.gibba020.stdout: mon max pg per osd = 10000 # >= luminous 2021-02-10T11:35:15.827 INFO:teuthology.orchestra.run.gibba020.stdout: mon pg warn max object skew = 0 2021-02-10T11:35:15.827 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:35:15.827 INFO:teuthology.orchestra.run.gibba020.stdout: osd pool default size = 2 2021-02-10T11:35:15.827 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:35:15.827 INFO:teuthology.orchestra.run.gibba020.stdout: mon osd allow primary affinity = true 2021-02-10T11:35:15.827 INFO:teuthology.orchestra.run.gibba020.stdout: mon osd allow pg remap = true 2021-02-10T11:35:15.827 INFO:teuthology.orchestra.run.gibba020.stdout: mon warn on legacy crush tunables = false 2021-02-10T11:35:15.827 INFO:teuthology.orchestra.run.gibba020.stdout: mon warn on crush straw calc version zero = false 2021-02-10T11:35:15.827 INFO:teuthology.orchestra.run.gibba020.stdout: mon warn on no sortbitwise = false 2021-02-10T11:35:15.827 INFO:teuthology.orchestra.run.gibba020.stdout: mon warn on osd down out interval zero = false 2021-02-10T11:35:15.827 INFO:teuthology.orchestra.run.gibba020.stdout: mon warn on too few osds = false 2021-02-10T11:35:15.827 INFO:teuthology.orchestra.run.gibba020.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2021-02-10T11:35:15.827 INFO:teuthology.orchestra.run.gibba020.stdout: mon_warn_on_pool_no_redundancy = false 2021-02-10T11:35:15.828 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:35:15.828 INFO:teuthology.orchestra.run.gibba020.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2021-02-10T11:35:15.828 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:35:15.828 INFO:teuthology.orchestra.run.gibba020.stdout: osd default data pool replay window = 5 2021-02-10T11:35:15.828 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:35:15.828 INFO:teuthology.orchestra.run.gibba020.stdout: mon allow pool delete = true 2021-02-10T11:35:15.828 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:35:15.828 INFO:teuthology.orchestra.run.gibba020.stdout: mon cluster log file level = debug 2021-02-10T11:35:15.828 INFO:teuthology.orchestra.run.gibba020.stdout: debug asserts on shutdown = true 2021-02-10T11:35:15.828 INFO:teuthology.orchestra.run.gibba020.stdout: mon health detail to clog = false 2021-02-10T11:35:15.828 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:35:15.828 INFO:teuthology.orchestra.run.gibba020.stdout: # we see this fail in qa on *nautilus*; bump up retries 2021-02-10T11:35:15.828 INFO:teuthology.orchestra.run.gibba020.stdout: mon_client_directed_command_retry = 4 2021-02-10T11:35:15.828 INFO:teuthology.orchestra.run.gibba020.stdout: mon host = "172.21.2.112,172.21.2.120,[v2:172.21.2.120:3301,v1:172.21.2.120:6790]" 2021-02-10T11:35:15.829 INFO:teuthology.orchestra.run.gibba020.stdout: fsid = f14f004e-0699-4584-ad81-3754258493de 2021-02-10T11:35:15.829 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:35:15.829 INFO:teuthology.orchestra.run.gibba020.stdout:[osd] 2021-02-10T11:35:15.829 INFO:teuthology.orchestra.run.gibba020.stdout: osd journal size = 100 2021-02-10T11:35:15.829 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:35:15.829 INFO:teuthology.orchestra.run.gibba020.stdout: osd scrub load threshold = 5.0 2021-02-10T11:35:15.829 INFO:teuthology.orchestra.run.gibba020.stdout: osd scrub max interval = 600 2021-02-10T11:35:15.829 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:35:15.829 INFO:teuthology.orchestra.run.gibba020.stdout: osd recover clone overlap = true 2021-02-10T11:35:15.829 INFO:teuthology.orchestra.run.gibba020.stdout: osd recovery max chunk = 1048576 2021-02-10T11:35:15.829 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:35:15.829 INFO:teuthology.orchestra.run.gibba020.stdout: osd debug shutdown = true 2021-02-10T11:35:15.829 INFO:teuthology.orchestra.run.gibba020.stdout: osd debug op order = true 2021-02-10T11:35:15.829 INFO:teuthology.orchestra.run.gibba020.stdout: osd debug verify stray on activate = true 2021-02-10T11:35:15.830 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:35:15.830 INFO:teuthology.orchestra.run.gibba020.stdout: osd open classes on start = true 2021-02-10T11:35:15.830 INFO:teuthology.orchestra.run.gibba020.stdout: osd debug pg log writeout = true 2021-02-10T11:35:15.830 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:35:15.830 INFO:teuthology.orchestra.run.gibba020.stdout: osd deep scrub update digest min age = 30 2021-02-10T11:35:15.830 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:35:15.830 INFO:teuthology.orchestra.run.gibba020.stdout: osd map max advance = 10 2021-02-10T11:35:15.830 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:35:15.830 INFO:teuthology.orchestra.run.gibba020.stdout: journal zero on create = true 2021-02-10T11:35:15.830 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:35:15.830 INFO:teuthology.orchestra.run.gibba020.stdout: filestore ondisk finisher threads = 3 2021-02-10T11:35:15.830 INFO:teuthology.orchestra.run.gibba020.stdout: filestore apply finisher threads = 3 2021-02-10T11:35:15.831 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:35:15.831 INFO:teuthology.orchestra.run.gibba020.stdout: bdev debug aio = true 2021-02-10T11:35:15.831 INFO:teuthology.orchestra.run.gibba020.stdout: osd debug misdirected ops = true 2021-02-10T11:35:15.831 INFO:teuthology.orchestra.run.gibba020.stdout: bdev async discard = True 2021-02-10T11:35:15.831 INFO:teuthology.orchestra.run.gibba020.stdout: bdev enable discard = True 2021-02-10T11:35:15.831 INFO:teuthology.orchestra.run.gibba020.stdout: bluestore allocator = bitmap 2021-02-10T11:35:15.831 INFO:teuthology.orchestra.run.gibba020.stdout: bluestore block size = 96636764160 2021-02-10T11:35:15.831 INFO:teuthology.orchestra.run.gibba020.stdout: bluestore fsck on mount = True 2021-02-10T11:35:15.831 INFO:teuthology.orchestra.run.gibba020.stdout: debug bluefs = 20 2021-02-10T11:35:15.831 INFO:teuthology.orchestra.run.gibba020.stdout: debug bluestore = 20 2021-02-10T11:35:15.831 INFO:teuthology.orchestra.run.gibba020.stdout: debug filestore = 20 2021-02-10T11:35:15.831 INFO:teuthology.orchestra.run.gibba020.stdout: debug journal = 20 2021-02-10T11:35:15.831 INFO:teuthology.orchestra.run.gibba020.stdout: debug ms = 20 2021-02-10T11:35:15.831 INFO:teuthology.orchestra.run.gibba020.stdout: debug osd = 25 2021-02-10T11:35:15.832 INFO:teuthology.orchestra.run.gibba020.stdout: debug rocksdb = 10 2021-02-10T11:35:15.832 INFO:teuthology.orchestra.run.gibba020.stdout: mon osd backfillfull_ratio = 0.85 2021-02-10T11:35:15.832 INFO:teuthology.orchestra.run.gibba020.stdout: mon osd full ratio = 0.9 2021-02-10T11:35:15.832 INFO:teuthology.orchestra.run.gibba020.stdout: mon osd nearfull ratio = 0.8 2021-02-10T11:35:15.832 INFO:teuthology.orchestra.run.gibba020.stdout: osd failsafe full ratio = 0.95 2021-02-10T11:35:15.832 INFO:teuthology.orchestra.run.gibba020.stdout: osd objectstore = bluestore 2021-02-10T11:35:15.832 INFO:teuthology.orchestra.run.gibba020.stdout: osd op complaint time = 180 2021-02-10T11:35:15.832 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:35:15.832 INFO:teuthology.orchestra.run.gibba020.stdout:[mgr] 2021-02-10T11:35:15.832 INFO:teuthology.orchestra.run.gibba020.stdout: debug ms = 1 2021-02-10T11:35:15.832 INFO:teuthology.orchestra.run.gibba020.stdout: debug mgr = 20 2021-02-10T11:35:15.832 INFO:teuthology.orchestra.run.gibba020.stdout: debug mon = 20 2021-02-10T11:35:15.832 INFO:teuthology.orchestra.run.gibba020.stdout: debug auth = 20 2021-02-10T11:35:15.832 INFO:teuthology.orchestra.run.gibba020.stdout: mon reweight min pgs per osd = 4 2021-02-10T11:35:15.833 INFO:teuthology.orchestra.run.gibba020.stdout: mon reweight min bytes per osd = 10 2021-02-10T11:35:15.833 INFO:teuthology.orchestra.run.gibba020.stdout: debug client = 10 2021-02-10T11:35:15.833 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:35:15.833 INFO:teuthology.orchestra.run.gibba020.stdout:[mon] 2021-02-10T11:35:15.833 INFO:teuthology.orchestra.run.gibba020.stdout: debug ms = 1 2021-02-10T11:35:15.833 INFO:teuthology.orchestra.run.gibba020.stdout: debug mon = 20 2021-02-10T11:35:15.833 INFO:teuthology.orchestra.run.gibba020.stdout: debug paxos = 20 2021-02-10T11:35:15.833 INFO:teuthology.orchestra.run.gibba020.stdout: debug auth = 20 2021-02-10T11:35:15.833 INFO:teuthology.orchestra.run.gibba020.stdout: mon data avail warn = 5 2021-02-10T11:35:15.833 INFO:teuthology.orchestra.run.gibba020.stdout: mon mgr mkfs grace = 240 2021-02-10T11:35:15.833 INFO:teuthology.orchestra.run.gibba020.stdout: mon reweight min pgs per osd = 4 2021-02-10T11:35:15.833 INFO:teuthology.orchestra.run.gibba020.stdout: mon osd reporter subtree level = osd 2021-02-10T11:35:15.833 INFO:teuthology.orchestra.run.gibba020.stdout: mon osd prime pg temp = true 2021-02-10T11:35:15.834 INFO:teuthology.orchestra.run.gibba020.stdout: mon reweight min bytes per osd = 10 2021-02-10T11:35:15.834 INFO:teuthology.orchestra.run.gibba020.stdout: mon op complaint time = 120 2021-02-10T11:35:15.834 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:35:15.834 INFO:teuthology.orchestra.run.gibba020.stdout:[client] 2021-02-10T11:35:15.834 INFO:teuthology.orchestra.run.gibba020.stdout: rgw cache enabled = true 2021-02-10T11:35:15.834 INFO:teuthology.orchestra.run.gibba020.stdout: rgw enable ops log = true 2021-02-10T11:35:15.834 INFO:teuthology.orchestra.run.gibba020.stdout: rgw enable usage log = true 2021-02-10T11:35:15.834 INFO:teuthology.orchestra.run.gibba020.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2021-02-10T11:35:15.834 INFO:teuthology.orchestra.run.gibba020.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2021-02-10T11:35:15.834 INFO:teuthology.orchestra.run.gibba020.stdout: client mount timeout = 600 2021-02-10T11:35:15.834 INFO:teuthology.orchestra.run.gibba020.stdout: debug client = 20 2021-02-10T11:35:15.834 INFO:teuthology.orchestra.run.gibba020.stdout: debug ms = 1 2021-02-10T11:35:15.834 INFO:teuthology.orchestra.run.gibba020.stdout: pid file = "" 2021-02-10T11:35:15.834 INFO:teuthology.orchestra.run.gibba020.stdout:[mon.a] 2021-02-10T11:35:15.835 INFO:teuthology.orchestra.run.gibba020.stdout:[mon.b] 2021-02-10T11:35:15.835 INFO:teuthology.orchestra.run.gibba020.stdout:[mon.c] 2021-02-10T11:35:15.835 INFO:teuthology.orchestra.run.gibba020.stdout:[mds.a] 2021-02-10T11:35:15.835 INFO:teuthology.orchestra.run.gibba020.stdout:[mds.b] 2021-02-10T11:35:15.835 INFO:teuthology.orchestra.run.gibba020.stdout:[mds] 2021-02-10T11:35:15.835 INFO:teuthology.orchestra.run.gibba020.stdout: debug mds = 20 2021-02-10T11:35:15.835 INFO:teuthology.orchestra.run.gibba020.stdout: debug ms = 1 2021-02-10T11:35:15.835 INFO:teuthology.orchestra.run.gibba020.stdout: mds bal frag = True 2021-02-10T11:35:15.835 INFO:teuthology.orchestra.run.gibba020.stdout: mds bal fragment size max = 10000 2021-02-10T11:35:15.835 INFO:teuthology.orchestra.run.gibba020.stdout: mds bal merge size = 5 2021-02-10T11:35:15.835 INFO:teuthology.orchestra.run.gibba020.stdout: mds bal split bits = 3 2021-02-10T11:35:15.835 INFO:teuthology.orchestra.run.gibba020.stdout: mds bal split size = 100 2021-02-10T11:35:15.835 INFO:teuthology.orchestra.run.gibba020.stdout: mds debug frag = True 2021-02-10T11:35:15.835 INFO:teuthology.orchestra.run.gibba020.stdout: mds debug scatterstat = True 2021-02-10T11:35:15.836 INFO:teuthology.orchestra.run.gibba020.stdout: mds op complaint time = 180 2021-02-10T11:35:15.836 INFO:teuthology.orchestra.run.gibba020.stdout: mds verify scatter = True 2021-02-10T11:35:15.836 INFO:teuthology.orchestra.run.gibba020.stdout: osd op complaint time = 180 2021-02-10T11:35:15.836 INFO:teuthology.orchestra.run.gibba020.stdout:[client.alice] 2021-02-10T11:35:15.836 INFO:teuthology.orchestra.run.gibba020.stdout: keyring = /etc/ceph/ceph.client.alice.keyring 2021-02-10T11:35:15.837 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:35:16.265 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.261 7fa8fe936700 1 Processor -- start 2021-02-10T11:35:16.265 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.261 7fa8fe936700 1 -- start start 2021-02-10T11:35:16.265 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.262 7fa8fe936700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa8f80d18d0 0x7fa8f8083ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:16.265 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.262 7fa8fe936700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8f8082d50 con 0x7fa8f8082920 2021-02-10T11:35:16.265 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.262 7fa8fe936700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8f80e57e0 con 0x7fa8f80e53e0 2021-02-10T11:35:16.265 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.262 7fa8fe936700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8f81b0d40 con 0x7fa8f80d18d0 2021-02-10T11:35:16.265 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.262 7fa8f77fe700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fa8f8082920 0x7fa8f80832d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:50580/0 (socket says 172.21.2.112:50580) 2021-02-10T11:35:16.265 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.262 7fa8f77fe700 1 -- 172.21.2.112:0/73893727 learned_addr learned my addr 172.21.2.112:0/73893727 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:35:16.266 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.262 7fa8fcf53700 1 --2- 172.21.2.112:0/73893727 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa8f80d18d0 0x7fa8f8083ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:16.266 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.262 7fa8f6ffd700 1 -- 172.21.2.112:0/73893727 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1391275210 0 0) 0x7fa8ec001180 con 0x7fa8f8082920 2021-02-10T11:35:16.266 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.262 7fa8f6ffd700 1 -- 172.21.2.112:0/73893727 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8e0002fe0 con 0x7fa8f8082920 2021-02-10T11:35:16.266 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.262 7fa8f6ffd700 1 -- 172.21.2.112:0/73893727 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3455280853 0 0) 0x7fa8e8001080 con 0x7fa8f80e53e0 2021-02-10T11:35:16.266 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.263 7fa8f6ffd700 1 -- 172.21.2.112:0/73893727 --> v1:172.21.2.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8e0003210 con 0x7fa8f80e53e0 2021-02-10T11:35:16.266 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.263 7fa8fcf53700 1 -- 172.21.2.112:0/73893727 >> v1:172.21.2.120:6789/0 conn(0x7fa8f80e53e0 legacy=0x7fa8f81b0550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:16.266 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.263 7fa8fcf53700 1 -- 172.21.2.112:0/73893727 >> v1:172.21.2.112:6789/0 conn(0x7fa8f8082920 legacy=0x7fa8f80832d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:16.266 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.263 7fa8fcf53700 1 -- 172.21.2.112:0/73893727 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8f80d1cd0 con 0x7fa8f80d18d0 2021-02-10T11:35:16.266 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.263 7fa8fcf53700 1 --2- 172.21.2.112:0/73893727 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa8f80d18d0 0x7fa8f8083ac0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7fa8f0006e50 tx=0x7fa8f0009e70).ready entity=mon.2 client_cookie=bcd32cc824f85a56 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:16.266 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.263 7fa8f6ffd700 1 -- 172.21.2.112:0/73893727 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa8f0006a20 con 0x7fa8f80d18d0 2021-02-10T11:35:16.266 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.263 7fa8f6ffd700 1 -- 172.21.2.112:0/73893727 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8f0006790 con 0x7fa8f80d18d0 2021-02-10T11:35:16.266 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.263 7fa8f6ffd700 1 -- 172.21.2.112:0/73893727 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa8f000b700 con 0x7fa8f80d18d0 2021-02-10T11:35:16.266 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.263 7fa8fe936700 1 -- 172.21.2.112:0/73893727 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa8f80d18d0 msgr2=0x7fa8f8083ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:16.267 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.263 7fa8fe936700 1 --2- 172.21.2.112:0/73893727 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa8f80d18d0 0x7fa8f8083ac0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7fa8f0006e50 tx=0x7fa8f0009e70).stop 2021-02-10T11:35:16.267 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.264 7fa8fe936700 1 -- 172.21.2.112:0/73893727 shutdown_connections 2021-02-10T11:35:16.267 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.264 7fa8fe936700 1 --2- 172.21.2.112:0/73893727 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa8f80d18d0 0x7fa8f8083ac0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:16.267 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.264 7fa8fe936700 1 -- 172.21.2.112:0/73893727 >> 172.21.2.112:0/73893727 conn(0x7fa8f80ce500 msgr2=0x7fa8f80e4610 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:16.267 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.264 7fa8fe936700 1 -- 172.21.2.112:0/73893727 shutdown_connections 2021-02-10T11:35:16.267 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.264 7fa8fe936700 1 -- 172.21.2.112:0/73893727 wait complete. 2021-02-10T11:35:16.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.279 7fa8fe936700 1 Processor -- start 2021-02-10T11:35:16.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.279 7fa8fe936700 1 -- start start 2021-02-10T11:35:16.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.279 7fa8fe936700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa8f81b2ba0 0x7fa8f819b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:16.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.279 7fa8fe936700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa8f81b2070 0x7fa8f81b2470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:16.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.279 7fa8fe936700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa8f819c520 0x7fa8f819c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:16.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.279 7fa8fe936700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8f81b30c0 con 0x7fa8f81b2ba0 2021-02-10T11:35:16.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.279 7fa8fe936700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8f819bfa0 con 0x7fa8f81b2070 2021-02-10T11:35:16.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.279 7fa8fe936700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8f819cf40 con 0x7fa8f819c520 2021-02-10T11:35:16.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.279 7fa8f7fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa8f81b2ba0 0x7fa8f819b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:16.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.279 7fa8f7fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa8f81b2ba0 0x7fa8f819b7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:51128/0 (socket says 172.21.2.112:51128) 2021-02-10T11:35:16.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.279 7fa8f7fff700 1 -- 172.21.2.112:0/1808124553 learned_addr learned my addr 172.21.2.112:0/1808124553 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:35:16.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.279 7fa8fcf53700 1 --2- 172.21.2.112:0/1808124553 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa8f819c520 0x7fa8f819c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:16.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.279 7fa8f77fe700 1 --2- 172.21.2.112:0/1808124553 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa8f81b2070 0x7fa8f81b2470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:16.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.280 7fa8f7fff700 1 -- 172.21.2.112:0/1808124553 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa8f819c520 msgr2=0x7fa8f819c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:16.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.280 7fa8f7fff700 1 --2- 172.21.2.112:0/1808124553 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa8f819c520 0x7fa8f819c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:16.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.280 7fa8f7fff700 1 -- 172.21.2.112:0/1808124553 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa8f81b2070 msgr2=0x7fa8f81b2470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:16.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.280 7fa8f7fff700 1 --2- 172.21.2.112:0/1808124553 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa8f81b2070 0x7fa8f81b2470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:16.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.280 7fa8f7fff700 1 -- 172.21.2.112:0/1808124553 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8f819d160 con 0x7fa8f81b2ba0 2021-02-10T11:35:16.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.280 7fa8f7fff700 1 --2- 172.21.2.112:0/1808124553 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa8f81b2ba0 0x7fa8f819b7b0 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 rx=0x7fa8e8006f00 tx=0x7fa8e800b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:16.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.280 7fa8f4ff9700 1 -- 172.21.2.112:0/1808124553 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa8e800bd30 con 0x7fa8f81b2ba0 2021-02-10T11:35:16.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.280 7fa8f4ff9700 1 -- 172.21.2.112:0/1808124553 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8e8000e20 con 0x7fa8f81b2ba0 2021-02-10T11:35:16.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.280 7fa8f4ff9700 1 -- 172.21.2.112:0/1808124553 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa8e80054f0 con 0x7fa8f81b2ba0 2021-02-10T11:35:16.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.280 7fa8fe936700 1 -- 172.21.2.112:0/1808124553 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8f819d160 con 0x7fa8f81b2ba0 2021-02-10T11:35:16.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.280 7fa8fe936700 1 -- 172.21.2.112:0/1808124553 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8f819d6f0 con 0x7fa8f81b2ba0 2021-02-10T11:35:16.272 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.281 7fa8f4ff9700 1 -- 172.21.2.112:0/1808124553 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa8e8006030 con 0x7fa8f81b2ba0 2021-02-10T11:35:16.272 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.281 7fa8f4ff9700 1 --2- 172.21.2.112:0/1808124553 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa8d80186c0 0x7fa8d801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:16.272 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.281 7fa8f4ff9700 1 -- 172.21.2.112:0/1808124553 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fa8e8032630 con 0x7fa8f81b2ba0 2021-02-10T11:35:16.272 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.281 7fa8f77fe700 1 --2- 172.21.2.112:0/1808124553 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa8d80186c0 0x7fa8d801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:16.272 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.281 7fa8f77fe700 1 --2- 172.21.2.112:0/1808124553 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa8d80186c0 0x7fa8d801abc0 crc :-1 s=READY pgs=713 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:16.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.311 7fa8fe936700 1 -- 172.21.2.112:0/1808124553 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8f808c580 con 0x7fa8f81b2ba0 2021-02-10T11:35:16.302 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.314 7fa8f4ff9700 1 -- 172.21.2.112:0/1808124553 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa8e8000bc0 con 0x7fa8f81b2ba0 2021-02-10T11:35:16.425 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.438 7fa8fe936700 1 -- 172.21.2.112:0/1808124553 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa8f808c580 con 0x7fa8f81b2ba0 2021-02-10T11:35:16.425 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.439 7fa8f4ff9700 1 -- 172.21.2.112:0/1808124553 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v64) v1 ==== 74+0+13996 (secure 0 0 0) 0x7fa8e8000e20 con 0x7fa8f81b2ba0 2021-02-10T11:35:16.426 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:35:16.426 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":64,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:34:46.584716","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:34:43.550220","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":13,"pool_name":"cephfs_data","create_time":"2021-02-10 11:34:44.578113","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:35:16.427 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.441 7fa8fe936700 1 -- 172.21.2.112:0/1808124553 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa8d80186c0 msgr2=0x7fa8d801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:16.427 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.441 7fa8fe936700 1 --2- 172.21.2.112:0/1808124553 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa8d80186c0 0x7fa8d801abc0 crc :-1 s=READY pgs=713 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:16.427 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.441 7fa8fe936700 1 -- 172.21.2.112:0/1808124553 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa8f81b2ba0 msgr2=0x7fa8f819b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:16.427 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.441 7fa8fe936700 1 --2- 172.21.2.112:0/1808124553 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa8f81b2ba0 0x7fa8f819b7b0 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 rx=0x7fa8e8006f00 tx=0x7fa8e800b940).stop 2021-02-10T11:35:16.428 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.441 7fa8fe936700 1 -- 172.21.2.112:0/1808124553 shutdown_connections 2021-02-10T11:35:16.428 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.441 7fa8fe936700 1 --2- 172.21.2.112:0/1808124553 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa8d80186c0 0x7fa8d801abc0 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:16.428 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.441 7fa8fe936700 1 --2- 172.21.2.112:0/1808124553 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa8f81b2070 0x7fa8f81b2470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:16.428 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.441 7fa8fe936700 1 --2- 172.21.2.112:0/1808124553 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa8f819c520 0x7fa8f819c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:16.428 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.441 7fa8fe936700 1 --2- 172.21.2.112:0/1808124553 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa8f81b2ba0 0x7fa8f819b7b0 unknown :-1 s=CLOSED pgs=1469 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:16.428 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.441 7fa8fe936700 1 -- 172.21.2.112:0/1808124553 >> 172.21.2.112:0/1808124553 conn(0x7fa8f819aed0 msgr2=0x7fa8f81ab290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:16.428 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.441 7fa8fe936700 1 -- 172.21.2.112:0/1808124553 shutdown_connections 2021-02-10T11:35:16.429 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.441 7fa8fe936700 1 -- 172.21.2.112:0/1808124553 wait complete. 2021-02-10T11:35:16.436 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:13:57.508758', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 12, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:34:43.550220', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 13, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:34:44.578113', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '64', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:35:16.437 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:35:16.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.865 7f66c3b21700 1 Processor -- start 2021-02-10T11:35:16.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.865 7f66c3b21700 1 -- start start 2021-02-10T11:35:16.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.866 7f66c3b21700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f66bc0e5170 0x7f66bc1b38e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:16.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.866 7f66c3b21700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66bc078b90 con 0x7f66bc0e5570 2021-02-10T11:35:16.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.866 7f66c3b21700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66bc0784f0 con 0x7f66bc1b3f00 2021-02-10T11:35:16.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.866 7f66c3b21700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66bc078710 con 0x7f66bc0e5170 2021-02-10T11:35:16.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.866 7f66c213e700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f66bc0e5570 0x7f66bc077d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:50598/0 (socket says 172.21.2.112:50598) 2021-02-10T11:35:16.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.866 7f66c213e700 1 -- 172.21.2.112:0/2442490924 learned_addr learned my addr 172.21.2.112:0/2442490924 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:35:16.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.866 7f66c093b700 1 -- 172.21.2.112:0/2442490924 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2083660601 0 0) 0x7f66b4001500 con 0x7f66bc0e5570 2021-02-10T11:35:16.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.866 7f66c193d700 1 --2- 172.21.2.112:0/2442490924 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f66bc0e5170 0x7f66bc1b38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:16.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.866 7f66c093b700 1 -- 172.21.2.112:0/2442490924 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66a8002fe0 con 0x7f66bc0e5570 2021-02-10T11:35:16.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.867 7f66c093b700 1 -- 172.21.2.112:0/2442490924 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2698883021 0 0) 0x7f66ac000ff0 con 0x7f66bc1b3f00 2021-02-10T11:35:16.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.867 7f66c093b700 1 -- 172.21.2.112:0/2442490924 --> v1:172.21.2.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66a8003210 con 0x7f66bc1b3f00 2021-02-10T11:35:16.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.867 7f66c093b700 1 -- 172.21.2.112:0/2442490924 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2814222692 0 0) 0x7f66b4001500 con 0x7f66bc0e5570 2021-02-10T11:35:16.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.867 7f66c093b700 1 -- 172.21.2.112:0/2442490924 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f66bc0e5170 msgr2=0x7f66bc1b38e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:16.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.867 7f66c093b700 1 --2- 172.21.2.112:0/2442490924 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f66bc0e5170 0x7f66bc1b38e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:16.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.867 7f66c093b700 1 -- 172.21.2.112:0/2442490924 >> v1:172.21.2.120:6789/0 conn(0x7f66bc1b3f00 legacy=0x7f66bc077510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:16.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.867 7f66c093b700 1 -- 172.21.2.112:0/2442490924 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66bc0788c0 con 0x7f66bc0e5570 2021-02-10T11:35:16.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.867 7f66c093b700 1 -- 172.21.2.112:0/2442490924 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f66b4001a00 con 0x7f66bc0e5570 2021-02-10T11:35:16.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.867 7f66c193d700 1 --2- 172.21.2.112:0/2442490924 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f66bc0e5170 0x7f66bc1b38e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:35:16.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.867 7f66c093b700 1 -- 172.21.2.112:0/2442490924 >> v1:172.21.2.112:6789/0 conn(0x7f66bc0e5570 legacy=0x7f66bc077d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:16.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.867 7f66c093b700 1 --2- 172.21.2.112:0/2442490924 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f66a80038e0 0x7f66a8003ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:16.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.867 7f66c093b700 1 --2- 172.21.2.112:0/2442490924 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f66a80067e0 0x7f66a8008c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:16.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.867 7f66c093b700 1 --2- 172.21.2.112:0/2442490924 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f66a8009580 0x7f66a800ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:16.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.867 7f66c093b700 1 -- 172.21.2.112:0/2442490924 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66a80502d0 con 0x7f66a80067e0 2021-02-10T11:35:16.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.867 7f66c093b700 1 -- 172.21.2.112:0/2442490924 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66a80504b0 con 0x7f66a80038e0 2021-02-10T11:35:16.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.867 7f66c093b700 1 -- 172.21.2.112:0/2442490924 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66a80506d0 con 0x7f66a8009580 2021-02-10T11:35:16.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.867 7f66c113c700 1 --2- 172.21.2.112:0/2442490924 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f66a80067e0 0x7f66a8008c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:16.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.867 7f66c213e700 1 --2- 172.21.2.112:0/2442490924 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f66a8009580 0x7f66a800ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:16.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.867 7f66c193d700 1 --2- 172.21.2.112:0/2442490924 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f66a80038e0 0x7f66a8003ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:16.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.867 7f66c113c700 1 -- 172.21.2.112:0/2442490924 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f66a8009580 msgr2=0x7f66a800ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:16.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.867 7f66c113c700 1 --2- 172.21.2.112:0/2442490924 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f66a8009580 0x7f66a800ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:16.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.867 7f66c113c700 1 -- 172.21.2.112:0/2442490924 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f66a80038e0 msgr2=0x7f66a8003ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:16.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.867 7f66c113c700 1 --2- 172.21.2.112:0/2442490924 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f66a80038e0 0x7f66a8003ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:16.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.867 7f66c113c700 1 -- 172.21.2.112:0/2442490924 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66a8050970 con 0x7f66a80067e0 2021-02-10T11:35:16.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.867 7f66c193d700 1 --2- 172.21.2.112:0/2442490924 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f66a80038e0 0x7f66a8003ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:35:16.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.868 7f66c113c700 1 --2- 172.21.2.112:0/2442490924 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f66a80067e0 0x7f66a8008c80 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 rx=0x7f66ac000fb0 tx=0x7f66ac009a90).ready entity=mon.0 client_cookie=d55f3aa9e88f5e7f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:16.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.868 7f66c093b700 1 -- 172.21.2.112:0/2442490924 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f66ac0066d0 con 0x7f66a80067e0 2021-02-10T11:35:16.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.868 7f66c093b700 1 -- 172.21.2.112:0/2442490924 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66ac007280 con 0x7f66a80067e0 2021-02-10T11:35:16.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.868 7f66c093b700 1 -- 172.21.2.112:0/2442490924 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f66ac007710 con 0x7f66a80067e0 2021-02-10T11:35:16.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.868 7f66c3b21700 1 -- 172.21.2.112:0/2442490924 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f66a80067e0 msgr2=0x7f66a8008c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:16.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.868 7f66c3b21700 1 --2- 172.21.2.112:0/2442490924 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f66a80067e0 0x7f66a8008c80 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 rx=0x7f66ac000fb0 tx=0x7f66ac009a90).stop 2021-02-10T11:35:16.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.868 7f66c213e700 1 -- 172.21.2.112:0/2442490924 reap_dead start 2021-02-10T11:35:16.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.868 7f66c3b21700 1 -- 172.21.2.112:0/2442490924 shutdown_connections 2021-02-10T11:35:16.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.868 7f66c3b21700 1 -- 172.21.2.112:0/2442490924 >> 172.21.2.112:0/2442490924 conn(0x7f66bc0ce820 msgr2=0x7f66bc0e4660 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:16.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.869 7f66c3b21700 1 -- 172.21.2.112:0/2442490924 shutdown_connections 2021-02-10T11:35:16.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.869 7f66c3b21700 1 -- 172.21.2.112:0/2442490924 wait complete. 2021-02-10T11:35:16.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.883 7f66c3b21700 1 Processor -- start 2021-02-10T11:35:16.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.883 7f66c3b21700 1 -- start start 2021-02-10T11:35:16.876 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.884 7f66c3b21700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f66bc0a0c00 0x7f66bc093490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:16.876 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.884 7f66c3b21700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f66bc0a5580 0x7f66bc0a5980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:16.876 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.884 7f66c3b21700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f66bc0ac950 0x7f66bc0acd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:16.876 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.884 7f66c3b21700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66bc090ac0 con 0x7f66bc0a5580 2021-02-10T11:35:16.876 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.884 7f66c3b21700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66bc0a5fa0 con 0x7f66bc0a0c00 2021-02-10T11:35:16.876 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.884 7f66c3b21700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66bc0ad370 con 0x7f66bc0ac950 2021-02-10T11:35:16.876 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.884 7f66c113c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f66bc0a5580 0x7f66bc0a5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:16.876 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.884 7f66c113c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f66bc0a5580 0x7f66bc0a5980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:51144/0 (socket says 172.21.2.112:51144) 2021-02-10T11:35:16.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.884 7f66c113c700 1 -- 172.21.2.112:0/3556456677 learned_addr learned my addr 172.21.2.112:0/3556456677 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:35:16.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.884 7f66c213e700 1 --2- 172.21.2.112:0/3556456677 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f66bc0ac950 0x7f66bc0acd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:16.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.884 7f66c113c700 1 -- 172.21.2.112:0/3556456677 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f66bc0ac950 msgr2=0x7f66bc0acd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:16.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.884 7f66c113c700 1 --2- 172.21.2.112:0/3556456677 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f66bc0ac950 0x7f66bc0acd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:16.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.884 7f66c193d700 1 --2- 172.21.2.112:0/3556456677 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f66bc0a0c00 0x7f66bc093490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:16.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.884 7f66c113c700 1 -- 172.21.2.112:0/3556456677 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f66bc0a0c00 msgr2=0x7f66bc093490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:16.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.884 7f66c113c700 1 --2- 172.21.2.112:0/3556456677 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f66bc0a0c00 0x7f66bc093490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:16.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.884 7f66c113c700 1 -- 172.21.2.112:0/3556456677 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66bc0ad590 con 0x7f66bc0a5580 2021-02-10T11:35:16.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.884 7f66c193d700 1 --2- 172.21.2.112:0/3556456677 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f66bc0a0c00 0x7f66bc093490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:35:16.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.884 7f66c113c700 1 --2- 172.21.2.112:0/3556456677 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f66bc0a5580 0x7f66bc0a5980 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 rx=0x7f66ac00b590 tx=0x7f66ac00b930).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:16.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.885 7f66b27fc700 1 -- 172.21.2.112:0/3556456677 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f66ac006b40 con 0x7f66bc0a5580 2021-02-10T11:35:16.878 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.885 7f66b27fc700 1 -- 172.21.2.112:0/3556456677 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66ac00bdf0 con 0x7f66bc0a5580 2021-02-10T11:35:16.878 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.885 7f66b27fc700 1 -- 172.21.2.112:0/3556456677 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f66ac000c90 con 0x7f66bc0a5580 2021-02-10T11:35:16.878 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.885 7f66c3b21700 1 -- 172.21.2.112:0/3556456677 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66bc0ad590 con 0x7f66bc0a5580 2021-02-10T11:35:16.878 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.885 7f66c3b21700 1 -- 172.21.2.112:0/3556456677 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66bc0ad590 con 0x7f66bc0a5580 2021-02-10T11:35:16.878 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.885 7f66b27fc700 1 -- 172.21.2.112:0/3556456677 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f66ac00a000 con 0x7f66bc0a5580 2021-02-10T11:35:16.878 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.885 7f66b27fc700 1 --2- 172.21.2.112:0/3556456677 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f66a40186c0 0x7f66a401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:16.878 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.885 7f66b27fc700 1 -- 172.21.2.112:0/3556456677 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f66ac0325c0 con 0x7f66bc0a5580 2021-02-10T11:35:16.878 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.886 7f66c193d700 1 --2- 172.21.2.112:0/3556456677 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f66a40186c0 0x7f66a401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:16.878 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.886 7f66c193d700 1 --2- 172.21.2.112:0/3556456677 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f66a40186c0 0x7f66a401abc0 crc :-1 s=READY pgs=714 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:16.902 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.915 7f66c3b21700 1 -- 172.21.2.112:0/3556456677 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66bc10ca30 con 0x7f66bc0a5580 2021-02-10T11:35:16.905 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:16.918 7f66b27fc700 1 -- 172.21.2.112:0/3556456677 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f66ac006b40 con 0x7f66bc0a5580 2021-02-10T11:35:17.035 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:17.048 7f66c3b21700 1 -- 172.21.2.112:0/3556456677 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f66bc0fe290 con 0x7f66bc0a5580 2021-02-10T11:35:17.035 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:17.048 7f66b27fc700 1 -- 172.21.2.112:0/3556456677 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v64) v1 ==== 92+0+10 (secure 0 0 0) 0x7f66ac00a000 con 0x7f66bc0a5580 2021-02-10T11:35:17.036 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:35:17.038 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:17.050 7f66c3b21700 1 -- 172.21.2.112:0/3556456677 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f66a40186c0 msgr2=0x7f66a401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:17.038 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:17.050 7f66c3b21700 1 --2- 172.21.2.112:0/3556456677 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f66a40186c0 0x7f66a401abc0 crc :-1 s=READY pgs=714 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:17.038 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:17.050 7f66c3b21700 1 -- 172.21.2.112:0/3556456677 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f66bc0a5580 msgr2=0x7f66bc0a5980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:17.038 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:17.050 7f66c3b21700 1 --2- 172.21.2.112:0/3556456677 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f66bc0a5580 0x7f66bc0a5980 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 rx=0x7f66ac00b590 tx=0x7f66ac00b930).stop 2021-02-10T11:35:17.038 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:17.051 7f66c3b21700 1 -- 172.21.2.112:0/3556456677 shutdown_connections 2021-02-10T11:35:17.038 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:17.051 7f66c3b21700 1 --2- 172.21.2.112:0/3556456677 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f66a40186c0 0x7f66a401abc0 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:17.038 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:17.051 7f66c3b21700 1 --2- 172.21.2.112:0/3556456677 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f66bc0ac950 0x7f66bc0acd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:17.038 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:17.051 7f66c3b21700 1 --2- 172.21.2.112:0/3556456677 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f66bc0a5580 0x7f66bc0a5980 unknown :-1 s=CLOSED pgs=1472 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:17.038 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:17.051 7f66c3b21700 1 --2- 172.21.2.112:0/3556456677 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f66bc0a0c00 0x7f66bc093490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:17.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:17.051 7f66c3b21700 1 -- 172.21.2.112:0/3556456677 >> 172.21.2.112:0/3556456677 conn(0x7f66bc188880 msgr2=0x7f66bc0a4380 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:17.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:17.051 7f66c3b21700 1 -- 172.21.2.112:0/3556456677 shutdown_connections 2021-02-10T11:35:17.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:17.051 7f66c3b21700 1 -- 172.21.2.112:0/3556456677 wait complete. 2021-02-10T11:35:17.046 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:35:17.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:17.472 7f5c5903f700 1 Processor -- start 2021-02-10T11:35:17.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:17.472 7f5c5903f700 1 -- start start 2021-02-10T11:35:17.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:17.473 7f5c5903f700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5c540e5170 0x7f5c541b38e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:17.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:17.473 7f5c5903f700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f5c541b3f00 0x7f5c54077510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:17.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:17.473 7f5c5903f700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c54078b90 con 0x7f5c540e5570 2021-02-10T11:35:17.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:17.473 7f5c5903f700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c5407a120 con 0x7f5c541b3f00 2021-02-10T11:35:17.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:17.473 7f5c5903f700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c5407a310 con 0x7f5c540e5170 2021-02-10T11:35:17.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:17.473 7f5c5259c700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f5c541b3f00 0x7f5c54077510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:17.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:17.473 7f5c5259c700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f5c541b3f00 0x7f5c54077510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:51154/0 (socket says 172.21.2.112:51154) 2021-02-10T11:35:17.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:17.473 7f5c5259c700 1 -- 172.21.2.112:0/2318491419 learned_addr learned my addr 172.21.2.112:0/2318491419 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:35:17.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:17.473 7f5c5259c700 1 -- 172.21.2.112:0/2318491419 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5c540e5170 msgr2=0x7f5c541b38e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:35:17.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:17.473 7f5c5259c700 1 --2- 172.21.2.112:0/2318491419 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5c540e5170 0x7f5c541b38e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:17.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:17.473 7f5c5259c700 1 -- 172.21.2.112:0/2318491419 >> v1:172.21.2.112:6789/0 conn(0x7f5c540e5570 legacy=0x7f5c54077b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:17.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:17.473 7f5c5259c700 1 -- 172.21.2.112:0/2318491419 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c5407a500 con 0x7f5c541b3f00 2021-02-10T11:35:17.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:17.473 7f5c5259c700 1 --2- 172.21.2.112:0/2318491419 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5c541b3f00 0x7f5c54077510 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 rx=0x7f5c48006700 tx=0x7f5c48009b80).ready entity=mon.0 client_cookie=d0fc97efaac6e096 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:17.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:17.474 7f5c51d9b700 1 -- 172.21.2.112:0/2318491419 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5c4800c0c0 con 0x7f5c541b3f00 2021-02-10T11:35:17.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:17.474 7f5c51d9b700 1 -- 172.21.2.112:0/2318491419 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c4800c400 con 0x7f5c541b3f00 2021-02-10T11:35:17.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:17.474 7f5c51d9b700 1 -- 172.21.2.112:0/2318491419 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5c4800c920 con 0x7f5c541b3f00 2021-02-10T11:35:17.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:17.474 7f5c5903f700 1 -- 172.21.2.112:0/2318491419 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5c541b3f00 msgr2=0x7f5c54077510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:17.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:17.474 7f5c5903f700 1 --2- 172.21.2.112:0/2318491419 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5c541b3f00 0x7f5c54077510 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 rx=0x7f5c48006700 tx=0x7f5c48009b80).stop 2021-02-10T11:35:17.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:17.474 7f5c5903f700 1 -- 172.21.2.112:0/2318491419 shutdown_connections 2021-02-10T11:35:17.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:17.474 7f5c5903f700 1 --2- 172.21.2.112:0/2318491419 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5c541b3f00 0x7f5c54077510 unknown :-1 s=CLOSED pgs=1474 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:17.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:17.474 7f5c5903f700 1 --2- 172.21.2.112:0/2318491419 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5c540e5170 0x7f5c541b38e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:17.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:17.474 7f5c5903f700 1 -- 172.21.2.112:0/2318491419 >> 172.21.2.112:0/2318491419 conn(0x7f5c540ce820 msgr2=0x7f5c540e4660 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:17.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:17.474 7f5c5903f700 1 -- 172.21.2.112:0/2318491419 shutdown_connections 2021-02-10T11:35:17.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:17.474 7f5c5903f700 1 -- 172.21.2.112:0/2318491419 wait complete. 2021-02-10T11:35:17.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:17.489 7f5c5903f700 1 Processor -- start 2021-02-10T11:35:17.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:17.489 7f5c5903f700 1 -- start start 2021-02-10T11:35:17.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:17.489 7f5c5903f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5c5410d0f0 0x7f5c5410d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:17.481 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:17.489 7f5c5903f700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5c540c40f0 0x7f5c540b1120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:17.481 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:17.489 7f5c5903f700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5c541a1d10 0x7f5c541a2110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:17.481 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:17.489 7f5c5903f700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c540b2af0 con 0x7f5c5410d0f0 2021-02-10T11:35:17.481 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:17.489 7f5c5903f700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c541a2730 con 0x7f5c540c40f0 2021-02-10T11:35:17.481 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:17.489 7f5c5903f700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c541a2910 con 0x7f5c541a1d10 2021-02-10T11:35:17.481 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:17.489 7f5c52d9d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5c5410d0f0 0x7f5c5410d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:17.481 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:17.489 7f5c52d9d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5c5410d0f0 0x7f5c5410d550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:51162/0 (socket says 172.21.2.112:51162) 2021-02-10T11:35:17.481 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:17.489 7f5c52d9d700 1 -- 172.21.2.112:0/844173282 learned_addr learned my addr 172.21.2.112:0/844173282 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:35:17.481 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:17.489 7f5c52d9d700 1 -- 172.21.2.112:0/844173282 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5c541a1d10 msgr2=0x7f5c541a2110 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:35:17.481 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:17.489 7f5c52d9d700 1 --2- 172.21.2.112:0/844173282 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5c541a1d10 0x7f5c541a2110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:17.481 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:17.489 7f5c52d9d700 1 -- 172.21.2.112:0/844173282 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5c540c40f0 msgr2=0x7f5c540b1120 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:35:17.482 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:17.489 7f5c52d9d700 1 --2- 172.21.2.112:0/844173282 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5c540c40f0 0x7f5c540b1120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:17.482 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:17.489 7f5c52d9d700 1 -- 172.21.2.112:0/844173282 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c541a2b30 con 0x7f5c5410d0f0 2021-02-10T11:35:17.482 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:17.490 7f5c52d9d700 1 --2- 172.21.2.112:0/844173282 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5c5410d0f0 0x7f5c5410d550 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 rx=0x7f5c4400b560 tx=0x7f5c4400b900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:17.482 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:17.490 7f5c437fe700 1 -- 172.21.2.112:0/844173282 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5c4400bcf0 con 0x7f5c5410d0f0 2021-02-10T11:35:17.482 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:17.490 7f5c437fe700 1 -- 172.21.2.112:0/844173282 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c44006030 con 0x7f5c5410d0f0 2021-02-10T11:35:17.482 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:17.490 7f5c5903f700 1 -- 172.21.2.112:0/844173282 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c541a2b30 con 0x7f5c5410d0f0 2021-02-10T11:35:17.482 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:17.490 7f5c437fe700 1 -- 172.21.2.112:0/844173282 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5c440054d0 con 0x7f5c5410d0f0 2021-02-10T11:35:17.482 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:17.490 7f5c5903f700 1 -- 172.21.2.112:0/844173282 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c540b33a0 con 0x7f5c5410d0f0 2021-02-10T11:35:17.482 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:17.490 7f5c437fe700 1 -- 172.21.2.112:0/844173282 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5c44006210 con 0x7f5c5410d0f0 2021-02-10T11:35:17.482 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:17.490 7f5c437fe700 1 --2- 172.21.2.112:0/844173282 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5c38018730 0x7f5c3801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:17.482 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:17.490 7f5c437fe700 1 -- 172.21.2.112:0/844173282 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f5c440327a0 con 0x7f5c5410d0f0 2021-02-10T11:35:17.482 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:17.491 7f5c5259c700 1 --2- 172.21.2.112:0/844173282 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5c38018730 0x7f5c3801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:17.483 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:17.491 7f5c5259c700 1 --2- 172.21.2.112:0/844173282 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5c38018730 0x7f5c3801ac10 crc :-1 s=READY pgs=715 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:17.510 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:17.523 7f5c5903f700 1 -- 172.21.2.112:0/844173282 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c5410bc00 con 0x7f5c5410d0f0 2021-02-10T11:35:17.512 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:17.525 7f5c437fe700 1 -- 172.21.2.112:0/844173282 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5c440009b0 con 0x7f5c5410d0f0 2021-02-10T11:35:17.641 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:17.654 7f5c5903f700 1 -- 172.21.2.112:0/844173282 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f5c540fe290 con 0x7f5c5410d0f0 2021-02-10T11:35:17.641 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:17.654 7f5c437fe700 1 -- 172.21.2.112:0/844173282 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v64) v1 ==== 104+0+10 (secure 0 0 0) 0x7f5c44000c40 con 0x7f5c5410d0f0 2021-02-10T11:35:17.642 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:35:17.644 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:17.656 7f5c5903f700 1 -- 172.21.2.112:0/844173282 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5c38018730 msgr2=0x7f5c3801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:17.644 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:17.656 7f5c5903f700 1 --2- 172.21.2.112:0/844173282 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5c38018730 0x7f5c3801ac10 crc :-1 s=READY pgs=715 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:17.644 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:17.656 7f5c5903f700 1 -- 172.21.2.112:0/844173282 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5c5410d0f0 msgr2=0x7f5c5410d550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:17.644 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:17.656 7f5c5903f700 1 --2- 172.21.2.112:0/844173282 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5c5410d0f0 0x7f5c5410d550 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 rx=0x7f5c4400b560 tx=0x7f5c4400b900).stop 2021-02-10T11:35:17.644 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:17.657 7f5c5903f700 1 -- 172.21.2.112:0/844173282 shutdown_connections 2021-02-10T11:35:17.644 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:17.657 7f5c5903f700 1 --2- 172.21.2.112:0/844173282 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5c38018730 0x7f5c3801ac10 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:17.644 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:17.657 7f5c5903f700 1 --2- 172.21.2.112:0/844173282 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5c540c40f0 0x7f5c540b1120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:17.644 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:17.657 7f5c5903f700 1 --2- 172.21.2.112:0/844173282 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5c541a1d10 0x7f5c541a2110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:17.644 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:17.657 7f5c5903f700 1 --2- 172.21.2.112:0/844173282 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5c5410d0f0 0x7f5c5410d550 unknown :-1 s=CLOSED pgs=1475 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:17.645 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:17.657 7f5c5903f700 1 -- 172.21.2.112:0/844173282 >> 172.21.2.112:0/844173282 conn(0x7f5c5408e180 msgr2=0x7f5c5410be30 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:17.645 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:17.657 7f5c5903f700 1 -- 172.21.2.112:0/844173282 shutdown_connections 2021-02-10T11:35:17.645 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:17.657 7f5c5903f700 1 -- 172.21.2.112:0/844173282 wait complete. 2021-02-10T11:35:17.652 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:35:17.810 DEBUG:teuthology.orchestra.run.gibba012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:35:17.813 DEBUG:teuthology.orchestra.run.gibba020:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:35:18.084 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.079 7fcb8500e700 1 Processor -- start 2021-02-10T11:35:18.084 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.079 7fcb8500e700 1 -- start start 2021-02-10T11:35:18.084 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.079 7fcb8500e700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcb80078520 0x7fcb800d1900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:18.084 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.079 7fcb8500e700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb80078920 con 0x7fcb800e5170 2021-02-10T11:35:18.084 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.079 7fcb8500e700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb80078b90 con 0x7fcb800e5570 2021-02-10T11:35:18.084 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.079 7fcb8500e700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb801b4140 con 0x7fcb80078520 2021-02-10T11:35:18.084 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.080 7fcb7ed9d700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fcb800e5170 0x7fcb801b38e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:50628/0 (socket says 172.21.2.112:50628) 2021-02-10T11:35:18.084 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.080 7fcb7ed9d700 1 -- 172.21.2.112:0/2219282333 learned_addr learned my addr 172.21.2.112:0/2219282333 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:35:18.084 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.080 7fcb7dd9b700 1 -- 172.21.2.112:0/2219282333 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 677255028 0 0) 0x7fcb70001180 con 0x7fcb800e5170 2021-02-10T11:35:18.085 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.080 7fcb7dd9b700 1 -- 172.21.2.112:0/2219282333 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb6c002fe0 con 0x7fcb800e5170 2021-02-10T11:35:18.085 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.080 7fcb7f59e700 1 --2- 172.21.2.112:0/2219282333 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcb80078520 0x7fcb800d1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:18.085 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.080 7fcb7dd9b700 1 -- 172.21.2.112:0/2219282333 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 23235894 0 0) 0x7fcb70001180 con 0x7fcb800e5170 2021-02-10T11:35:18.085 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.080 7fcb7dd9b700 1 -- 172.21.2.112:0/2219282333 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcb80078520 msgr2=0x7fcb800d1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:18.085 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.080 7fcb7dd9b700 1 --2- 172.21.2.112:0/2219282333 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcb80078520 0x7fcb800d1900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:18.085 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.080 7fcb7dd9b700 1 -- 172.21.2.112:0/2219282333 >> v1:172.21.2.120:6789/0 conn(0x7fcb800e5570 legacy=0x7fcb80079110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:18.085 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.080 7fcb7dd9b700 1 -- 172.21.2.112:0/2219282333 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb800d1f20 con 0x7fcb800e5170 2021-02-10T11:35:18.085 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.080 7fcb7dd9b700 1 -- 172.21.2.112:0/2219282333 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7fcb70001680 con 0x7fcb800e5170 2021-02-10T11:35:18.085 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.081 7fcb7dd9b700 1 -- 172.21.2.112:0/2219282333 >> v1:172.21.2.112:6789/0 conn(0x7fcb800e5170 legacy=0x7fcb801b38e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:18.085 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.081 7fcb7dd9b700 1 --2- 172.21.2.112:0/2219282333 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcb6c003a40 0x7fcb6c005f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:18.085 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.081 7fcb7dd9b700 1 --2- 172.21.2.112:0/2219282333 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcb6c006800 0x7fcb6c008cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:18.085 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.081 7fcb7dd9b700 1 --2- 172.21.2.112:0/2219282333 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcb6c009510 0x7fcb6c00ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:18.085 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.081 7fcb7dd9b700 1 -- 172.21.2.112:0/2219282333 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb6c00c2e0 con 0x7fcb6c003a40 2021-02-10T11:35:18.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.081 7fcb7dd9b700 1 -- 172.21.2.112:0/2219282333 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb6c00c500 con 0x7fcb6c009510 2021-02-10T11:35:18.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.081 7fcb7dd9b700 1 -- 172.21.2.112:0/2219282333 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb6c00c720 con 0x7fcb6c006800 2021-02-10T11:35:18.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.081 7fcb7ed9d700 1 --2- 172.21.2.112:0/2219282333 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcb6c003a40 0x7fcb6c005f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:18.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.081 7fcb7f59e700 1 --2- 172.21.2.112:0/2219282333 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcb6c009510 0x7fcb6c00ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:18.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.081 7fcb7ed9d700 1 -- 172.21.2.112:0/2219282333 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcb6c006800 msgr2=0x7fcb6c008cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:18.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.081 7fcb7ed9d700 1 --2- 172.21.2.112:0/2219282333 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcb6c006800 0x7fcb6c008cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:18.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.081 7fcb7ed9d700 1 -- 172.21.2.112:0/2219282333 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcb6c009510 msgr2=0x7fcb6c00ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:18.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.081 7fcb7ed9d700 1 --2- 172.21.2.112:0/2219282333 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcb6c009510 0x7fcb6c00ba30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:18.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.081 7fcb7ed9d700 1 -- 172.21.2.112:0/2219282333 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb6c050960 con 0x7fcb6c003a40 2021-02-10T11:35:18.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.081 7fcb7ed9d700 1 --2- 172.21.2.112:0/2219282333 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcb6c003a40 0x7fcb6c005f60 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 rx=0x7fcb70007970 tx=0x7fcb70007d10).ready entity=mon.0 client_cookie=df67aa3aa016dbb4 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:18.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.081 7fcb7dd9b700 1 -- 172.21.2.112:0/2219282333 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcb700070c0 con 0x7fcb6c003a40 2021-02-10T11:35:18.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.081 7fcb7dd9b700 1 -- 172.21.2.112:0/2219282333 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb70001180 con 0x7fcb6c003a40 2021-02-10T11:35:18.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.081 7fcb7dd9b700 1 -- 172.21.2.112:0/2219282333 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcb7000a520 con 0x7fcb6c003a40 2021-02-10T11:35:18.087 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.081 7fcb8500e700 1 -- 172.21.2.112:0/2219282333 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcb6c003a40 msgr2=0x7fcb6c005f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:18.087 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.081 7fcb8500e700 1 --2- 172.21.2.112:0/2219282333 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcb6c003a40 0x7fcb6c005f60 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 rx=0x7fcb70007970 tx=0x7fcb70007d10).stop 2021-02-10T11:35:18.087 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.081 7fcb7f59e700 1 -- 172.21.2.112:0/2219282333 reap_dead start 2021-02-10T11:35:18.087 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.082 7fcb8500e700 1 -- 172.21.2.112:0/2219282333 shutdown_connections 2021-02-10T11:35:18.087 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.082 7fcb8500e700 1 -- 172.21.2.112:0/2219282333 >> 172.21.2.112:0/2219282333 conn(0x7fcb800ce820 msgr2=0x7fcb800e4660 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:18.087 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.082 7fcb8500e700 1 -- 172.21.2.112:0/2219282333 shutdown_connections 2021-02-10T11:35:18.087 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.082 7fcb8500e700 1 -- 172.21.2.112:0/2219282333 wait complete. 2021-02-10T11:35:18.087 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.096 7fcb8500e700 1 Processor -- start 2021-02-10T11:35:18.087 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.096 7fcb8500e700 1 -- start start 2021-02-10T11:35:18.087 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.097 7fcb8500e700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcb800a0c00 0x7fcb80093490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:18.087 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.097 7fcb8500e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcb800a5580 0x7fcb800a5980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:18.087 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.097 7fcb8500e700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcb800ac950 0x7fcb800acd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:18.087 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.097 7fcb8500e700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb80090ac0 con 0x7fcb800a5580 2021-02-10T11:35:18.088 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.097 7fcb8500e700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb800a5fa0 con 0x7fcb800ac950 2021-02-10T11:35:18.088 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.097 7fcb8500e700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb800ad370 con 0x7fcb800a0c00 2021-02-10T11:35:18.088 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.097 7fcb7e59c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcb800a5580 0x7fcb800a5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:18.088 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.097 7fcb7e59c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcb800a5580 0x7fcb800a5980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:51182/0 (socket says 172.21.2.112:51182) 2021-02-10T11:35:18.088 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.097 7fcb7e59c700 1 -- 172.21.2.112:0/1529345091 learned_addr learned my addr 172.21.2.112:0/1529345091 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:35:18.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.097 7fcb7e59c700 1 -- 172.21.2.112:0/1529345091 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcb800a0c00 msgr2=0x7fcb80093490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:18.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.097 7fcb7e59c700 1 --2- 172.21.2.112:0/1529345091 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcb800a0c00 0x7fcb80093490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:18.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.097 7fcb7e59c700 1 -- 172.21.2.112:0/1529345091 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcb800ac950 msgr2=0x7fcb800acd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:18.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.097 7fcb7e59c700 1 --2- 172.21.2.112:0/1529345091 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcb800ac950 0x7fcb800acd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:18.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.097 7fcb7e59c700 1 -- 172.21.2.112:0/1529345091 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb800ad590 con 0x7fcb800a5580 2021-02-10T11:35:18.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.098 7fcb7e59c700 1 --2- 172.21.2.112:0/1529345091 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcb800a5580 0x7fcb800a5980 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 rx=0x7fcb68006ed0 tx=0x7fcb6800b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:18.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.098 7fcb777fe700 1 -- 172.21.2.112:0/1529345091 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcb68006240 con 0x7fcb800a5580 2021-02-10T11:35:18.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.098 7fcb777fe700 1 -- 172.21.2.112:0/1529345091 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb680054b0 con 0x7fcb800a5580 2021-02-10T11:35:18.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.098 7fcb777fe700 1 -- 172.21.2.112:0/1529345091 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcb68005990 con 0x7fcb800a5580 2021-02-10T11:35:18.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.098 7fcb8500e700 1 -- 172.21.2.112:0/1529345091 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb800ad590 con 0x7fcb800a5580 2021-02-10T11:35:18.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.098 7fcb8500e700 1 -- 172.21.2.112:0/1529345091 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb801a7080 con 0x7fcb800a5580 2021-02-10T11:35:18.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.099 7fcb777fe700 1 -- 172.21.2.112:0/1529345091 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fcb68006030 con 0x7fcb800a5580 2021-02-10T11:35:18.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.099 7fcb777fe700 1 --2- 172.21.2.112:0/1529345091 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcb640186c0 0x7fcb6401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:18.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.099 7fcb777fe700 1 -- 172.21.2.112:0/1529345091 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fcb68032560 con 0x7fcb800a5580 2021-02-10T11:35:18.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.099 7fcb7ed9d700 1 --2- 172.21.2.112:0/1529345091 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcb640186c0 0x7fcb6401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:18.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.100 7fcb7ed9d700 1 --2- 172.21.2.112:0/1529345091 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcb640186c0 0x7fcb6401abc0 crc :-1 s=READY pgs=716 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:18.116 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.129 7fcb8500e700 1 -- 172.21.2.112:0/1529345091 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb800a61c0 con 0x7fcb800a5580 2021-02-10T11:35:18.118 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.132 7fcb777fe700 1 -- 172.21.2.112:0/1529345091 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fcb6802cee0 con 0x7fcb800a5580 2021-02-10T11:35:18.248 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.261 7fcb8500e700 1 -- 172.21.2.112:0/1529345091 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7fcb800a61c0 con 0x7fcb800a5580 2021-02-10T11:35:18.248 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.262 7fcb777fe700 1 -- 172.21.2.112:0/1529345091 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v64) v1 ==== 100+0+10 (secure 0 0 0) 0x7fcb6802cee0 con 0x7fcb800a5580 2021-02-10T11:35:18.249 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:35:18.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.264 7fcb8500e700 1 -- 172.21.2.112:0/1529345091 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcb640186c0 msgr2=0x7fcb6401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:18.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.264 7fcb8500e700 1 --2- 172.21.2.112:0/1529345091 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcb640186c0 0x7fcb6401abc0 crc :-1 s=READY pgs=716 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:18.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.264 7fcb8500e700 1 -- 172.21.2.112:0/1529345091 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcb800a5580 msgr2=0x7fcb800a5980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:18.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.264 7fcb8500e700 1 --2- 172.21.2.112:0/1529345091 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcb800a5580 0x7fcb800a5980 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 rx=0x7fcb68006ed0 tx=0x7fcb6800b960).stop 2021-02-10T11:35:18.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.264 7fcb8500e700 1 -- 172.21.2.112:0/1529345091 shutdown_connections 2021-02-10T11:35:18.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.264 7fcb8500e700 1 --2- 172.21.2.112:0/1529345091 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcb640186c0 0x7fcb6401abc0 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:18.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.264 7fcb8500e700 1 --2- 172.21.2.112:0/1529345091 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcb800ac950 0x7fcb800acd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:18.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.264 7fcb8500e700 1 --2- 172.21.2.112:0/1529345091 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcb800a5580 0x7fcb800a5980 unknown :-1 s=CLOSED pgs=1478 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:18.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.264 7fcb8500e700 1 --2- 172.21.2.112:0/1529345091 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcb800a0c00 0x7fcb80093490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:18.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.264 7fcb8500e700 1 -- 172.21.2.112:0/1529345091 >> 172.21.2.112:0/1529345091 conn(0x7fcb80188880 msgr2=0x7fcb800a4380 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:18.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.264 7fcb8500e700 1 -- 172.21.2.112:0/1529345091 shutdown_connections 2021-02-10T11:35:18.252 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.264 7fcb8500e700 1 -- 172.21.2.112:0/1529345091 wait complete. 2021-02-10T11:35:18.260 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:35:18.260 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:35:18.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.683 7f8f4705a700 1 Processor -- start 2021-02-10T11:35:18.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.684 7f8f4705a700 1 -- start start 2021-02-10T11:35:18.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.684 7f8f4705a700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f8f400e53e0 0x7f8f401b0550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:18.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.684 7f8f4705a700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f8f401b0b70 0x7f8f40081910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:18.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.684 7f8f4705a700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f40082d50 con 0x7f8f40081f30 2021-02-10T11:35:18.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.684 7f8f4705a700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f40082b20 con 0x7f8f400e53e0 2021-02-10T11:35:18.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.684 7f8f4705a700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f400e5fe0 con 0x7f8f401b0b70 2021-02-10T11:35:18.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.684 7f8f44e76700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f8f400e53e0 0x7f8f401b0550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:18.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.684 7f8f44e76700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f8f400e53e0 0x7f8f401b0550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:51190/0 (socket says 172.21.2.112:51190) 2021-02-10T11:35:18.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.684 7f8f44e76700 1 -- 172.21.2.112:0/297682829 learned_addr learned my addr 172.21.2.112:0/297682829 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:35:18.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.684 7f8f3ffff700 1 --2- 172.21.2.112:0/297682829 >> v2:172.21.2.120:3300/0 conn(0x7f8f401b0b70 0x7f8f40081910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:18.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.685 7f8f44e76700 1 -- 172.21.2.112:0/297682829 >> v2:172.21.2.120:3300/0 conn(0x7f8f401b0b70 msgr2=0x7f8f40081910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:18.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.685 7f8f44e76700 1 --2- 172.21.2.112:0/297682829 >> v2:172.21.2.120:3300/0 conn(0x7f8f401b0b70 0x7f8f40081910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:18.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.685 7f8f44e76700 1 -- 172.21.2.112:0/297682829 >> v1:172.21.2.120:6789/0 conn(0x7f8f40081f30 legacy=0x7f8f40082330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:18.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.685 7f8f44e76700 1 -- 172.21.2.112:0/297682829 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f400e57e0 con 0x7f8f400e53e0 2021-02-10T11:35:18.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.685 7f8f44e76700 1 --2- 172.21.2.112:0/297682829 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8f400e53e0 0x7f8f401b0550 secure :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 rx=0x7f8f28006700 tx=0x7f8f28009b00).ready entity=mon.0 client_cookie=54a8dd09e1ed6054 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:18.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.685 7f8f3f7fe700 1 -- 172.21.2.112:0/297682829 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8f2800c0c0 con 0x7f8f400e53e0 2021-02-10T11:35:18.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.685 7f8f3f7fe700 1 -- 172.21.2.112:0/297682829 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f2800c400 con 0x7f8f400e53e0 2021-02-10T11:35:18.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.685 7f8f3f7fe700 1 -- 172.21.2.112:0/297682829 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8f2800c8e0 con 0x7f8f400e53e0 2021-02-10T11:35:18.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.685 7f8f4705a700 1 -- 172.21.2.112:0/297682829 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8f400e53e0 msgr2=0x7f8f401b0550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:18.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.685 7f8f4705a700 1 --2- 172.21.2.112:0/297682829 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8f400e53e0 0x7f8f401b0550 secure :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 rx=0x7f8f28006700 tx=0x7f8f28009b00).stop 2021-02-10T11:35:18.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.686 7f8f4705a700 1 -- 172.21.2.112:0/297682829 shutdown_connections 2021-02-10T11:35:18.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.686 7f8f4705a700 1 --2- 172.21.2.112:0/297682829 >> v2:172.21.2.120:3300/0 conn(0x7f8f401b0b70 0x7f8f40081910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:18.689 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.686 7f8f4705a700 1 --2- 172.21.2.112:0/297682829 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8f400e53e0 0x7f8f401b0550 unknown :-1 s=CLOSED pgs=1479 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:18.689 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.686 7f8f4705a700 1 -- 172.21.2.112:0/297682829 >> 172.21.2.112:0/297682829 conn(0x7f8f400ce500 msgr2=0x7f8f400e4610 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:18.689 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.686 7f8f4705a700 1 -- 172.21.2.112:0/297682829 shutdown_connections 2021-02-10T11:35:18.689 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.686 7f8f4705a700 1 -- 172.21.2.112:0/297682829 wait complete. 2021-02-10T11:35:18.690 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.700 7f8f4705a700 1 Processor -- start 2021-02-10T11:35:18.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.700 7f8f4705a700 1 -- start start 2021-02-10T11:35:18.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.701 7f8f4705a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8f400908f0 0x7f8f4008fd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:18.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.701 7f8f4705a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8f40090390 0x7f8f40090dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:18.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.701 7f8f4705a700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8f400913e0 0x7f8f400b8b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:18.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.701 7f8f4705a700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f400917e0 con 0x7f8f40090390 2021-02-10T11:35:18.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.701 7f8f4705a700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f400b9170 con 0x7f8f400913e0 2021-02-10T11:35:18.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.701 7f8f4705a700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f400b9390 con 0x7f8f400908f0 2021-02-10T11:35:18.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.701 7f8f3ffff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8f40090390 0x7f8f40090dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:18.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.701 7f8f3ffff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8f40090390 0x7f8f40090dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:51192/0 (socket says 172.21.2.112:51192) 2021-02-10T11:35:18.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.701 7f8f3ffff700 1 -- 172.21.2.112:0/2449092085 learned_addr learned my addr 172.21.2.112:0/2449092085 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:35:18.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.701 7f8f3ffff700 1 -- 172.21.2.112:0/2449092085 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8f400908f0 msgr2=0x7f8f4008fd70 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:35:18.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.701 7f8f3ffff700 1 --2- 172.21.2.112:0/2449092085 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8f400908f0 0x7f8f4008fd70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:18.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.701 7f8f3ffff700 1 -- 172.21.2.112:0/2449092085 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8f400913e0 msgr2=0x7f8f400b8b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:18.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.702 7f8f3ffff700 1 --2- 172.21.2.112:0/2449092085 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8f400913e0 0x7f8f400b8b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:18.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.702 7f8f3ffff700 1 -- 172.21.2.112:0/2449092085 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f400b95b0 con 0x7f8f40090390 2021-02-10T11:35:18.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.702 7f8f3ffff700 1 --2- 172.21.2.112:0/2449092085 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8f40090390 0x7f8f40090dc0 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 rx=0x7f8f3000b530 tx=0x7f8f3000b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:18.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.702 7f8f3d7fa700 1 -- 172.21.2.112:0/2449092085 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8f3000bcc0 con 0x7f8f40090390 2021-02-10T11:35:18.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.702 7f8f3d7fa700 1 -- 172.21.2.112:0/2449092085 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f30002030 con 0x7f8f40090390 2021-02-10T11:35:18.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.702 7f8f4705a700 1 -- 172.21.2.112:0/2449092085 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f400b95b0 con 0x7f8f40090390 2021-02-10T11:35:18.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.702 7f8f4705a700 1 -- 172.21.2.112:0/2449092085 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f400b9b40 con 0x7f8f40090390 2021-02-10T11:35:18.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.702 7f8f3d7fa700 1 -- 172.21.2.112:0/2449092085 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8f30002550 con 0x7f8f40090390 2021-02-10T11:35:18.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.703 7f8f3d7fa700 1 -- 172.21.2.112:0/2449092085 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8f30002210 con 0x7f8f40090390 2021-02-10T11:35:18.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.703 7f8f3d7fa700 1 --2- 172.21.2.112:0/2449092085 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8f240186c0 0x7f8f2401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:18.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.703 7f8f3d7fa700 1 -- 172.21.2.112:0/2449092085 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f8f30032540 con 0x7f8f40090390 2021-02-10T11:35:18.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.703 7f8f44e76700 1 --2- 172.21.2.112:0/2449092085 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8f240186c0 0x7f8f2401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:18.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.704 7f8f44e76700 1 --2- 172.21.2.112:0/2449092085 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8f240186c0 0x7f8f2401abc0 crc :-1 s=READY pgs=717 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:18.721 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.734 7f8f4705a700 1 -- 172.21.2.112:0/2449092085 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f400ed630 con 0x7f8f40090390 2021-02-10T11:35:18.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.737 7f8f3d7fa700 1 -- 172.21.2.112:0/2449092085 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8f30000960 con 0x7f8f40090390 2021-02-10T11:35:18.854 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.867 7f8f4705a700 1 -- 172.21.2.112:0/2449092085 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8f400f3a10 con 0x7f8f40090390 2021-02-10T11:35:18.855 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.868 7f8f3d7fa700 1 -- 172.21.2.112:0/2449092085 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) v1 ==== 94+0+2101 (secure 0 0 0) 0x7f8f300021f0 con 0x7f8f40090390 2021-02-10T11:35:18.855 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 48 2021-02-10T11:35:18.855 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:35:18.855 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":48,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8334,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/3370618110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":3370618110},{"type":"v1","addr":"172.21.2.112:6833","nonce":3370618110}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":44}],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:34:45.582177","modified":"2021-02-10 11:34:47.589305","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5921},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5921":{"gid":5921,"name":"b","rank":0,"incarnation":45,"state":"up:active","state_seq":3,"addr":"172.21.2.120:6835/2436135065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":2436135065},{"type":"v1","addr":"172.21.2.120:6835","nonce":2436135065}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2021-02-10T11:35:18.856 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.870 7f8f4705a700 1 -- 172.21.2.112:0/2449092085 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8f240186c0 msgr2=0x7f8f2401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:18.856 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.870 7f8f4705a700 1 --2- 172.21.2.112:0/2449092085 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8f240186c0 0x7f8f2401abc0 crc :-1 s=READY pgs=717 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:18.856 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.870 7f8f4705a700 1 -- 172.21.2.112:0/2449092085 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8f40090390 msgr2=0x7f8f40090dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:18.857 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.870 7f8f4705a700 1 --2- 172.21.2.112:0/2449092085 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8f40090390 0x7f8f40090dc0 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 rx=0x7f8f3000b530 tx=0x7f8f3000b8d0).stop 2021-02-10T11:35:18.857 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.870 7f8f4705a700 1 -- 172.21.2.112:0/2449092085 shutdown_connections 2021-02-10T11:35:18.857 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.870 7f8f4705a700 1 --2- 172.21.2.112:0/2449092085 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8f240186c0 0x7f8f2401abc0 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:18.857 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.870 7f8f4705a700 1 --2- 172.21.2.112:0/2449092085 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8f400913e0 0x7f8f400b8b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:18.857 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.870 7f8f4705a700 1 --2- 172.21.2.112:0/2449092085 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8f40090390 0x7f8f40090dc0 unknown :-1 s=CLOSED pgs=1480 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:18.857 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.870 7f8f4705a700 1 --2- 172.21.2.112:0/2449092085 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8f400908f0 0x7f8f4008fd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:18.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.870 7f8f4705a700 1 -- 172.21.2.112:0/2449092085 >> 172.21.2.112:0/2449092085 conn(0x7f8f400ac9c0 msgr2=0x7f8f400c3d60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:18.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.870 7f8f4705a700 1 -- 172.21.2.112:0/2449092085 shutdown_connections 2021-02-10T11:35:18.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:18.870 7f8f4705a700 1 -- 172.21.2.112:0/2449092085 wait complete. 2021-02-10T11:35:19.067 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 47, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:34:45.582177', 'modified': '2021-02-10 11:34:47.589305', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5921}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5921': {'gid': 5921, 'name': 'b', 'rank': 0, 'incarnation': 45, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.120:6835/2436135065', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.120:6834', 'nonce': 2436135065}, {'type': 'v1', 'addr': '172.21.2.120:6835', 'nonce': 2436135065}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:35:19.067 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:35:19.067 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:35:19.465 INFO:teuthology.orchestra.run.gibba020.stdout:{ 2021-02-10T11:35:19.466 INFO:teuthology.orchestra.run.gibba020.stdout: "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", 2021-02-10T11:35:19.466 INFO:teuthology.orchestra.run.gibba020.stdout: "whoami": 0, 2021-02-10T11:35:19.466 INFO:teuthology.orchestra.run.gibba020.stdout: "id": 5921, 2021-02-10T11:35:19.466 INFO:teuthology.orchestra.run.gibba020.stdout: "want_state": "up:active", 2021-02-10T11:35:19.466 INFO:teuthology.orchestra.run.gibba020.stdout: "state": "up:active", 2021-02-10T11:35:19.466 INFO:teuthology.orchestra.run.gibba020.stdout: "rank_uptime": 33.892637571999998, 2021-02-10T11:35:19.466 INFO:teuthology.orchestra.run.gibba020.stdout: "mdsmap_epoch": 47, 2021-02-10T11:35:19.466 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch": 64, 2021-02-10T11:35:19.466 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch_barrier": 64, 2021-02-10T11:35:19.466 INFO:teuthology.orchestra.run.gibba020.stdout: "uptime": 39.536513061999997 2021-02-10T11:35:19.467 INFO:teuthology.orchestra.run.gibba020.stdout:} 2021-02-10T11:35:19.470 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", "whoami": 0, "id": 5921, "want_state": "up:active", "state": "up:active", "rank_uptime": 33.892637571999998, "mdsmap_epoch": 47, "osdmap_epoch": 64, "osdmap_epoch_barrier": 64, "uptime": 39.536513061999997 } 2021-02-10T11:35:19.470 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:35:19.470 INFO:tasks.cephfs.fuse_mount:Client client.alice config is {} 2021-02-10T11:35:19.470 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.alice at ubuntu@gibba012.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2021-02-10T11:35:19.470 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:19.471 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:35:19.805 DEBUG:teuthology.orchestra.run.gibba012:> sudo modprobe fuse 2021-02-10T11:35:20.150 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:35:20.150 DEBUG:teuthology.orchestra.run.gibba012:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:35:20.494 INFO:teuthology.orchestra.run.gibba012.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:35:20.494 INFO:teuthology.orchestra.run.gibba012.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:35:20.495 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:35:20.495 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:20.495 DEBUG:teuthology.orchestra.run.gibba012:> ls /sys/fs/fuse/connections 2021-02-10T11:35:20.830 INFO:teuthology.orchestra.run.gibba012.stdout:42 2021-02-10T11:35:20.830 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [42] 2021-02-10T11:35:20.830 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --client_mountpoint=/volumes/group_0000000000485992/subvolume_0000000000513667/7bc8264f-d88f-49e9-b22f-5fa8b916f683 --name client.alice /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:35:20.833 DEBUG:teuthology.orchestra.run.gibba012:> sudo modprobe fuse 2021-02-10T11:35:21.176 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:35:21.176 DEBUG:teuthology.orchestra.run.gibba012:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:35:21.242 INFO:tasks.cephfs.fuse_mount.ceph-fuse.alice.gibba012.stderr:ceph-fuse[137177]: starting ceph client 2021-02-10T11:35:21.242 INFO:tasks.cephfs.fuse_mount.ceph-fuse.alice.gibba012.stderr:2021-02-10 11:35:21.255 7f1033d00f80 -1 init, newargv = 0x55a3af892450 newargc=9 2021-02-10T11:35:21.254 INFO:tasks.cephfs.fuse_mount.ceph-fuse.alice.gibba012.stderr:ceph-fuse[137177]: starting fuse 2021-02-10T11:35:21.519 INFO:teuthology.orchestra.run.gibba012.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:35:21.519 INFO:teuthology.orchestra.run.gibba012.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:35:21.520 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:35:21.520 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:21.520 DEBUG:teuthology.orchestra.run.gibba012:> ls /sys/fs/fuse/connections 2021-02-10T11:35:21.856 INFO:teuthology.orchestra.run.gibba012.stdout:42 2021-02-10T11:35:21.856 INFO:teuthology.orchestra.run.gibba012.stdout:43 2021-02-10T11:35:21.856 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42, 43] 2021-02-10T11:35:21.856 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:21.856 DEBUG:teuthology.orchestra.run.gibba012:> sudo python3 -c ' 2021-02-10T11:35:21.856 DEBUG:teuthology.orchestra.run.gibba012:> import glob 2021-02-10T11:35:21.857 DEBUG:teuthology.orchestra.run.gibba012:> import re 2021-02-10T11:35:21.857 DEBUG:teuthology.orchestra.run.gibba012:> import os 2021-02-10T11:35:21.857 DEBUG:teuthology.orchestra.run.gibba012:> import subprocess 2021-02-10T11:35:21.857 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:35:21.857 DEBUG:teuthology.orchestra.run.gibba012:> def find_socket(client_name): 2021-02-10T11:35:21.857 DEBUG:teuthology.orchestra.run.gibba012:> asok_path = "/var/run/ceph/ceph-client.alice.*.asok" 2021-02-10T11:35:21.857 DEBUG:teuthology.orchestra.run.gibba012:> files = glob.glob(asok_path) 2021-02-10T11:35:21.857 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:35:21.857 DEBUG:teuthology.orchestra.run.gibba012:> # Given a non-glob path, it better be there 2021-02-10T11:35:21.857 DEBUG:teuthology.orchestra.run.gibba012:> if "*" not in asok_path: 2021-02-10T11:35:21.857 DEBUG:teuthology.orchestra.run.gibba012:> assert(len(files) == 1) 2021-02-10T11:35:21.857 DEBUG:teuthology.orchestra.run.gibba012:> return files[0] 2021-02-10T11:35:21.857 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:35:21.857 DEBUG:teuthology.orchestra.run.gibba012:> for f in files: 2021-02-10T11:35:21.858 DEBUG:teuthology.orchestra.run.gibba012:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:35:21.858 DEBUG:teuthology.orchestra.run.gibba012:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:35:21.858 DEBUG:teuthology.orchestra.run.gibba012:> return f 2021-02-10T11:35:21.858 DEBUG:teuthology.orchestra.run.gibba012:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:35:21.858 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:35:21.858 DEBUG:teuthology.orchestra.run.gibba012:> print(find_socket("client.alice")) 2021-02-10T11:35:21.858 DEBUG:teuthology.orchestra.run.gibba012:> ' 2021-02-10T11:35:22.220 INFO:teuthology.orchestra.run.gibba012.stdout:/var/run/ceph/ceph-client.alice.137177.asok 2021-02-10T11:35:22.222 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.alice.137177.asok 2021-02-10T11:35:22.222 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:22.223 DEBUG:teuthology.orchestra.run.gibba012:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.alice.137177.asok status 2021-02-10T11:35:22.613 INFO:teuthology.orchestra.run.gibba012.stdout:{ 2021-02-10T11:35:22.613 INFO:teuthology.orchestra.run.gibba012.stdout: "metadata": { 2021-02-10T11:35:22.613 INFO:teuthology.orchestra.run.gibba012.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:35:22.613 INFO:teuthology.orchestra.run.gibba012.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:35:22.613 INFO:teuthology.orchestra.run.gibba012.stdout: "entity_id": "alice", 2021-02-10T11:35:22.613 INFO:teuthology.orchestra.run.gibba012.stdout: "hostname": "gibba012", 2021-02-10T11:35:22.614 INFO:teuthology.orchestra.run.gibba012.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2021-02-10T11:35:22.614 INFO:teuthology.orchestra.run.gibba012.stdout: "pid": "137177", 2021-02-10T11:35:22.614 INFO:teuthology.orchestra.run.gibba012.stdout: "root": "/volumes/group_0000000000485992/subvolume_0000000000513667/7bc8264f-d88f-49e9-b22f-5fa8b916f683" 2021-02-10T11:35:22.614 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:22.614 INFO:teuthology.orchestra.run.gibba012.stdout: "dentry_count": 0, 2021-02-10T11:35:22.614 INFO:teuthology.orchestra.run.gibba012.stdout: "dentry_pinned_count": 0, 2021-02-10T11:35:22.614 INFO:teuthology.orchestra.run.gibba012.stdout: "id": 8658, 2021-02-10T11:35:22.614 INFO:teuthology.orchestra.run.gibba012.stdout: "inst": { 2021-02-10T11:35:22.614 INFO:teuthology.orchestra.run.gibba012.stdout: "name": { 2021-02-10T11:35:22.614 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "client", 2021-02-10T11:35:22.614 INFO:teuthology.orchestra.run.gibba012.stdout: "num": 8658 2021-02-10T11:35:22.614 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:22.614 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": { 2021-02-10T11:35:22.615 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:35:22.615 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:0", 2021-02-10T11:35:22.615 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 3783778992 2021-02-10T11:35:22.615 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:35:22.615 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:22.615 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": { 2021-02-10T11:35:22.615 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:35:22.615 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:0", 2021-02-10T11:35:22.615 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 3783778992 2021-02-10T11:35:22.615 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:22.615 INFO:teuthology.orchestra.run.gibba012.stdout: "inst_str": "client.8658 172.21.2.112:0/3783778992", 2021-02-10T11:35:22.615 INFO:teuthology.orchestra.run.gibba012.stdout: "addr_str": "172.21.2.112:0/3783778992", 2021-02-10T11:35:22.615 INFO:teuthology.orchestra.run.gibba012.stdout: "inode_count": 1, 2021-02-10T11:35:22.616 INFO:teuthology.orchestra.run.gibba012.stdout: "mds_epoch": 47, 2021-02-10T11:35:22.616 INFO:teuthology.orchestra.run.gibba012.stdout: "osd_epoch": 64, 2021-02-10T11:35:22.616 INFO:teuthology.orchestra.run.gibba012.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:35:22.616 INFO:teuthology.orchestra.run.gibba012.stdout: "blacklisted": false 2021-02-10T11:35:22.616 INFO:teuthology.orchestra.run.gibba012.stdout:} 2021-02-10T11:35:22.617 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:22.617 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:35:22.618 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:35:22.952 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:35:22.952 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:35:22.952 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.1 && sudo dd if=/dev/urandom of=data.bin bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:35:23.330 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:35:23.330 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:35:23.330 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0274642 s, 38.2 MB/s 2021-02-10T11:35:23.331 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume authorize cephfs subvolume_0000000000513667 alice --group_name group_0000000000485992 --tenant_id tenant_id --access_level r 2021-02-10T11:35:23.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:23.767 7fcf27049700 1 Processor -- start 2021-02-10T11:35:23.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:23.768 7fcf27049700 1 -- start start 2021-02-10T11:35:23.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:23.768 7fcf27049700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcf200e5170 0x7fcf201b38e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:23.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:23.768 7fcf27049700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fcf200e5570 0x7fcf2007a120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:23.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:23.768 7fcf27049700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf20078b90 con 0x7fcf201b3f00 2021-02-10T11:35:23.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:23.768 7fcf27049700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf200d1900 con 0x7fcf200e5570 2021-02-10T11:35:23.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:23.768 7fcf27049700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf200d1af0 con 0x7fcf200e5170 2021-02-10T11:35:23.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:23.768 7fcf25666700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fcf200e5570 0x7fcf2007a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:23.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:23.768 7fcf25666700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fcf200e5570 0x7fcf2007a120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:51216/0 (socket says 172.21.2.112:51216) 2021-02-10T11:35:23.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:23.768 7fcf25666700 1 -- 172.21.2.112:0/1849832973 learned_addr learned my addr 172.21.2.112:0/1849832973 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:35:23.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:23.769 7fcf25666700 1 -- 172.21.2.112:0/1849832973 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcf200e5170 msgr2=0x7fcf201b38e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:23.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:23.769 7fcf25666700 1 --2- 172.21.2.112:0/1849832973 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcf200e5170 0x7fcf201b38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:23.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:23.769 7fcf25666700 1 -- 172.21.2.112:0/1849832973 >> v1:172.21.2.120:6789/0 conn(0x7fcf201b3f00 legacy=0x7fcf20077510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:23.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:23.769 7fcf25666700 1 -- 172.21.2.112:0/1849832973 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf200d1ce0 con 0x7fcf200e5570 2021-02-10T11:35:23.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:23.769 7fcf25666700 1 --2- 172.21.2.112:0/1849832973 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcf200e5570 0x7fcf2007a120 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 rx=0x7fcf18006b80 tx=0x7fcf18009b90).ready entity=mon.0 client_cookie=c191028ea83cffbb server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:23.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:23.769 7fcf1f7fe700 1 -- 172.21.2.112:0/1849832973 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcf180022e0 con 0x7fcf200e5570 2021-02-10T11:35:23.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:23.769 7fcf1f7fe700 1 -- 172.21.2.112:0/1849832973 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf18002620 con 0x7fcf200e5570 2021-02-10T11:35:23.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:23.769 7fcf1f7fe700 1 -- 172.21.2.112:0/1849832973 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcf18002b00 con 0x7fcf200e5570 2021-02-10T11:35:23.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:23.769 7fcf27049700 1 -- 172.21.2.112:0/1849832973 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcf200e5570 msgr2=0x7fcf2007a120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:23.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:23.769 7fcf27049700 1 --2- 172.21.2.112:0/1849832973 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcf200e5570 0x7fcf2007a120 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 rx=0x7fcf18006b80 tx=0x7fcf18009b90).stop 2021-02-10T11:35:23.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:23.770 7fcf27049700 1 -- 172.21.2.112:0/1849832973 shutdown_connections 2021-02-10T11:35:23.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:23.770 7fcf27049700 1 --2- 172.21.2.112:0/1849832973 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcf200e5570 0x7fcf2007a120 unknown :-1 s=CLOSED pgs=1483 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:23.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:23.770 7fcf27049700 1 --2- 172.21.2.112:0/1849832973 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcf200e5170 0x7fcf201b38e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:23.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:23.770 7fcf27049700 1 -- 172.21.2.112:0/1849832973 >> 172.21.2.112:0/1849832973 conn(0x7fcf200ce820 msgr2=0x7fcf200e4660 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:23.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:23.770 7fcf27049700 1 -- 172.21.2.112:0/1849832973 shutdown_connections 2021-02-10T11:35:23.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:23.770 7fcf27049700 1 -- 172.21.2.112:0/1849832973 wait complete. 2021-02-10T11:35:23.774 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:23.785 7fcf27049700 1 Processor -- start 2021-02-10T11:35:23.774 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:23.785 7fcf27049700 1 -- start start 2021-02-10T11:35:23.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:23.785 7fcf27049700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcf2010cde0 0x7fcf200b7550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:23.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:23.785 7fcf27049700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcf200b7b70 0x7fcf201a7230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:23.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:23.785 7fcf27049700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcf200b7f70 0x7fcf201a7850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:23.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:23.785 7fcf27049700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf201aaa20 con 0x7fcf2010cde0 2021-02-10T11:35:23.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:23.785 7fcf27049700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf2010d1e0 con 0x7fcf200b7b70 2021-02-10T11:35:23.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:23.785 7fcf27049700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf201a7e70 con 0x7fcf200b7f70 2021-02-10T11:35:23.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:23.786 7fcf24e65700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcf2010cde0 0x7fcf200b7550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:23.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:23.786 7fcf24e65700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcf2010cde0 0x7fcf200b7550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:51218/0 (socket says 172.21.2.112:51218) 2021-02-10T11:35:23.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:23.786 7fcf24e65700 1 -- 172.21.2.112:0/79519752 learned_addr learned my addr 172.21.2.112:0/79519752 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:35:23.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:23.786 7fcf24e65700 1 -- 172.21.2.112:0/79519752 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcf200b7f70 msgr2=0x7fcf201a7850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:23.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:23.786 7fcf24e65700 1 --2- 172.21.2.112:0/79519752 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcf200b7f70 0x7fcf201a7850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:23.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:23.786 7fcf24e65700 1 -- 172.21.2.112:0/79519752 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcf200b7b70 msgr2=0x7fcf201a7230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:23.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:23.786 7fcf24e65700 1 --2- 172.21.2.112:0/79519752 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcf200b7b70 0x7fcf201a7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:23.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:23.786 7fcf24e65700 1 -- 172.21.2.112:0/79519752 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf201a8090 con 0x7fcf2010cde0 2021-02-10T11:35:23.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:23.786 7fcf24e65700 1 --2- 172.21.2.112:0/79519752 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcf2010cde0 0x7fcf200b7550 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 rx=0x7fcf08006f00 tx=0x7fcf0800b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:23.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:23.786 7fcf1d7fa700 1 -- 172.21.2.112:0/79519752 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcf0800bd30 con 0x7fcf2010cde0 2021-02-10T11:35:23.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:23.786 7fcf1d7fa700 1 -- 172.21.2.112:0/79519752 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf08000c90 con 0x7fcf2010cde0 2021-02-10T11:35:23.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:23.786 7fcf1d7fa700 1 -- 172.21.2.112:0/79519752 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcf080022a0 con 0x7fcf2010cde0 2021-02-10T11:35:23.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:23.786 7fcf27049700 1 -- 172.21.2.112:0/79519752 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf201a8090 con 0x7fcf2010cde0 2021-02-10T11:35:23.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:23.786 7fcf27049700 1 -- 172.21.2.112:0/79519752 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf201a8090 con 0x7fcf2010cde0 2021-02-10T11:35:23.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:23.787 7fcf1d7fa700 1 -- 172.21.2.112:0/79519752 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fcf08005310 con 0x7fcf2010cde0 2021-02-10T11:35:23.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:23.787 7fcf1d7fa700 1 --2- 172.21.2.112:0/79519752 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcf040186c0 0x7fcf0401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:23.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:23.787 7fcf1d7fa700 1 -- 172.21.2.112:0/79519752 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fcf08032540 con 0x7fcf2010cde0 2021-02-10T11:35:23.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:23.787 7fcf1ffff700 1 --2- 172.21.2.112:0/79519752 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcf040186c0 0x7fcf0401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:23.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:23.788 7fcf1ffff700 1 --2- 172.21.2.112:0/79519752 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcf040186c0 0x7fcf0401abc0 crc :-1 s=READY pgs=718 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:23.804 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:23.818 7fcf27049700 1 -- 172.21.2.112:0/79519752 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf2010bc10 con 0x7fcf2010cde0 2021-02-10T11:35:23.807 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:23.820 7fcf1d7fa700 1 -- 172.21.2.112:0/79519752 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fcf08006b90 con 0x7fcf2010cde0 2021-02-10T11:35:23.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:23.945 7fcf27049700 1 -- 172.21.2.112:0/79519752 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"vol_name": "cephfs", "sub_name": "subvolume_0000000000513667", "target": ["mgr", ""], "tenant_id": "tenant_id", "group_name": "group_0000000000485992", "access_level": "r", "prefix": "fs subvolume authorize", "auth_id": "alice"}) v1 -- 0x7fcf200b8370 con 0x7fcf040186c0 2021-02-10T11:35:23.975 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:23.988 7fcf1d7fa700 1 -- 172.21.2.112:0/79519752 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+41 (crc 0 0 0) 0x7fcf10006e00 con 0x7fcf040186c0 2021-02-10T11:35:23.996 INFO:teuthology.orchestra.run.gibba012.stdout:AQDxxCNg5hUmChAArexyCsZP7RgHhs6rSnmt2Q== 2021-02-10T11:35:23.997 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:24.011 7fcf27049700 1 -- 172.21.2.112:0/79519752 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcf040186c0 msgr2=0x7fcf0401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:23.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:24.011 7fcf27049700 1 --2- 172.21.2.112:0/79519752 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcf040186c0 0x7fcf0401abc0 crc :-1 s=READY pgs=718 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:23.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:24.011 7fcf27049700 1 -- 172.21.2.112:0/79519752 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcf2010cde0 msgr2=0x7fcf200b7550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:23.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:24.011 7fcf27049700 1 --2- 172.21.2.112:0/79519752 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcf2010cde0 0x7fcf200b7550 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 rx=0x7fcf08006f00 tx=0x7fcf0800b940).stop 2021-02-10T11:35:23.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:24.011 7fcf27049700 1 -- 172.21.2.112:0/79519752 shutdown_connections 2021-02-10T11:35:23.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:24.011 7fcf27049700 1 --2- 172.21.2.112:0/79519752 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcf040186c0 0x7fcf0401abc0 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:23.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:24.011 7fcf27049700 1 --2- 172.21.2.112:0/79519752 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcf200b7b70 0x7fcf201a7230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:23.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:24.011 7fcf27049700 1 --2- 172.21.2.112:0/79519752 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcf200b7f70 0x7fcf201a7850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:23.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:24.011 7fcf27049700 1 --2- 172.21.2.112:0/79519752 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcf2010cde0 0x7fcf200b7550 unknown :-1 s=CLOSED pgs=1484 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:23.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:24.011 7fcf27049700 1 -- 172.21.2.112:0/79519752 >> 172.21.2.112:0/79519752 conn(0x7fcf2010d820 msgr2=0x7fcf200a3b80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:23.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:24.011 7fcf27049700 1 -- 172.21.2.112:0/79519752 shutdown_connections 2021-02-10T11:35:23.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:24.011 7fcf27049700 1 -- 172.21.2.112:0/79519752 wait complete. 2021-02-10T11:35:24.006 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:24.006 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:35:24.006 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:35:24.342 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:35:24.343 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:35:24.343 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba012.front.sepia.ceph.com... 2021-02-10T11:35:24.343 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:35:24.343 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:35:24.698 INFO:tasks.cephfs.fuse_mount.ceph-fuse.alice.gibba012.stderr:ceph-fuse[137177]: fuse finished with error 0 and tester_r 0 2021-02-10T11:35:24.699 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:24.699 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:35:24.699 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:35:25.033 INFO:teuthology.orchestra.run.gibba012.stdout:ext2/ext3 2021-02-10T11:35:25.034 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:35:25.034 INFO:teuthology.orchestra.run:waiting for 900 2021-02-10T11:35:25.034 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:35:25.034 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:35:25.405 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:35:25.893 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:25.889 7f4e98057700 1 Processor -- start 2021-02-10T11:35:25.893 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:25.889 7f4e98057700 1 -- start start 2021-02-10T11:35:25.893 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:25.890 7f4e98057700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4e900e53e0 0x7f4e901b0550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:25.893 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:25.890 7f4e98057700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f4e900842e0 0x7f4e90082100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:25.893 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:25.890 7f4e98057700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e90082d50 con 0x7f4e901b0b70 2021-02-10T11:35:25.893 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:25.890 7f4e98057700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e90084850 con 0x7f4e900842e0 2021-02-10T11:35:25.893 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:25.890 7f4e98057700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e900e5fe0 con 0x7f4e900e53e0 2021-02-10T11:35:25.893 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:25.890 7f4e96674700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f4e900842e0 0x7f4e90082100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:25.894 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:25.890 7f4e95672700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f4e901b0b70 0x7f4e90081910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:50688/0 (socket says 172.21.2.112:50688) 2021-02-10T11:35:25.894 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:25.890 7f4e96674700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f4e900842e0 0x7f4e90082100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:51226/0 (socket says 172.21.2.112:51226) 2021-02-10T11:35:25.894 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:25.890 7f4e95672700 1 -- 172.21.2.112:0/849443937 learned_addr learned my addr 172.21.2.112:0/849443937 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:35:25.894 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:25.890 7f4e95e73700 1 --2- 172.21.2.112:0/849443937 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4e900e53e0 0x7f4e901b0550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:25.894 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:25.890 7f4e94e71700 1 -- 172.21.2.112:0/849443937 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3212314100 0 0) 0x7f4e8c0010b0 con 0x7f4e901b0b70 2021-02-10T11:35:25.894 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:25.890 7f4e94e71700 1 -- 172.21.2.112:0/849443937 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e7c002fe0 con 0x7f4e901b0b70 2021-02-10T11:35:25.894 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:25.891 7f4e96674700 1 -- 172.21.2.112:0/849443937 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4e900e53e0 msgr2=0x7f4e901b0550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:25.894 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:25.891 7f4e96674700 1 --2- 172.21.2.112:0/849443937 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4e900e53e0 0x7f4e901b0550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:25.894 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:25.891 7f4e96674700 1 -- 172.21.2.112:0/849443937 >> v1:172.21.2.112:6789/0 conn(0x7f4e901b0b70 legacy=0x7f4e90081910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:25.894 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:25.891 7f4e96674700 1 -- 172.21.2.112:0/849443937 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e900e57e0 con 0x7f4e900842e0 2021-02-10T11:35:25.894 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:25.891 7f4e95e73700 1 --2- 172.21.2.112:0/849443937 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4e900e53e0 0x7f4e901b0550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:35:25.894 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:25.891 7f4e96674700 1 --2- 172.21.2.112:0/849443937 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4e900842e0 0x7f4e90082100 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 rx=0x7f4e88006e30 tx=0x7f4e88009e60).ready entity=mon.0 client_cookie=4e7b4d4cc834165c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:25.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:25.891 7f4e94e71700 1 -- 172.21.2.112:0/849443937 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4e88006a50 con 0x7f4e900842e0 2021-02-10T11:35:25.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:25.891 7f4e94e71700 1 -- 172.21.2.112:0/849443937 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e8800b300 con 0x7f4e900842e0 2021-02-10T11:35:25.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:25.891 7f4e94e71700 1 -- 172.21.2.112:0/849443937 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4e8800b7e0 con 0x7f4e900842e0 2021-02-10T11:35:25.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:25.891 7f4e98057700 1 -- 172.21.2.112:0/849443937 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4e900842e0 msgr2=0x7f4e90082100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:25.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:25.891 7f4e98057700 1 --2- 172.21.2.112:0/849443937 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4e900842e0 0x7f4e90082100 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 rx=0x7f4e88006e30 tx=0x7f4e88009e60).stop 2021-02-10T11:35:25.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:25.892 7f4e98057700 1 -- 172.21.2.112:0/849443937 shutdown_connections 2021-02-10T11:35:25.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:25.892 7f4e98057700 1 --2- 172.21.2.112:0/849443937 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4e900842e0 0x7f4e90082100 unknown :-1 s=CLOSED pgs=1486 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:25.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:25.892 7f4e98057700 1 --2- 172.21.2.112:0/849443937 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4e900e53e0 0x7f4e901b0550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:25.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:25.892 7f4e98057700 1 -- 172.21.2.112:0/849443937 >> 172.21.2.112:0/849443937 conn(0x7f4e900ce500 msgr2=0x7f4e900e4610 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:25.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:25.892 7f4e98057700 1 -- 172.21.2.112:0/849443937 shutdown_connections 2021-02-10T11:35:25.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:25.892 7f4e98057700 1 -- 172.21.2.112:0/849443937 wait complete. 2021-02-10T11:35:25.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:25.906 7f4e98057700 1 Processor -- start 2021-02-10T11:35:25.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:25.906 7f4e98057700 1 -- start start 2021-02-10T11:35:25.897 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:25.907 7f4e98057700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4e90096fb0 0x7f4e900bf130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:25.897 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:25.907 7f4e98057700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4e900bf750 0x7f4e901a7500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:25.897 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:25.907 7f4e98057700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4e901a7b20 0x7f4e901889e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:25.897 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:25.907 7f4e98057700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e90188460 con 0x7f4e900bf750 2021-02-10T11:35:25.897 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:25.907 7f4e98057700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e900bfb50 con 0x7f4e901a7b20 2021-02-10T11:35:25.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:25.907 7f4e98057700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e900ab630 con 0x7f4e90096fb0 2021-02-10T11:35:25.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:25.907 7f4e95672700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4e900bf750 0x7f4e901a7500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:25.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:25.907 7f4e95672700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4e900bf750 0x7f4e901a7500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:51234/0 (socket says 172.21.2.112:51234) 2021-02-10T11:35:25.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:25.907 7f4e95672700 1 -- 172.21.2.112:0/937777218 learned_addr learned my addr 172.21.2.112:0/937777218 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:35:25.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:25.907 7f4e96674700 1 --2- 172.21.2.112:0/937777218 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4e901a7b20 0x7f4e901889e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:25.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:25.907 7f4e95e73700 1 --2- 172.21.2.112:0/937777218 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4e90096fb0 0x7f4e900bf130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:25.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:25.907 7f4e95672700 1 -- 172.21.2.112:0/937777218 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4e90096fb0 msgr2=0x7f4e900bf130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:25.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:25.907 7f4e95672700 1 --2- 172.21.2.112:0/937777218 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4e90096fb0 0x7f4e900bf130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:25.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:25.907 7f4e95672700 1 -- 172.21.2.112:0/937777218 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4e901a7b20 msgr2=0x7f4e901889e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:25.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:25.907 7f4e95672700 1 --2- 172.21.2.112:0/937777218 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4e901a7b20 0x7f4e901889e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:25.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:25.907 7f4e95672700 1 -- 172.21.2.112:0/937777218 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e901a7f20 con 0x7f4e900bf750 2021-02-10T11:35:25.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:25.907 7f4e95672700 1 --2- 172.21.2.112:0/937777218 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4e900bf750 0x7f4e901a7500 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 rx=0x7f4e8c001c60 tx=0x7f4e8c00a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:25.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:25.908 7f4e827fc700 1 -- 172.21.2.112:0/937777218 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4e8c000e00 con 0x7f4e900bf750 2021-02-10T11:35:25.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:25.908 7f4e827fc700 1 -- 172.21.2.112:0/937777218 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e8c00acc0 con 0x7f4e900bf750 2021-02-10T11:35:25.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:25.908 7f4e827fc700 1 -- 172.21.2.112:0/937777218 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4e8c0045d0 con 0x7f4e900bf750 2021-02-10T11:35:25.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:25.908 7f4e98057700 1 -- 172.21.2.112:0/937777218 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e901a7f20 con 0x7f4e900bf750 2021-02-10T11:35:25.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:25.908 7f4e98057700 1 -- 172.21.2.112:0/937777218 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e901a7f20 con 0x7f4e900bf750 2021-02-10T11:35:25.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:25.909 7f4e827fc700 1 -- 172.21.2.112:0/937777218 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4e8c004140 con 0x7f4e900bf750 2021-02-10T11:35:25.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:25.909 7f4e827fc700 1 --2- 172.21.2.112:0/937777218 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4e780186c0 0x7f4e7801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:25.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:25.909 7f4e827fc700 1 -- 172.21.2.112:0/937777218 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f4e8c032540 con 0x7f4e900bf750 2021-02-10T11:35:25.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:25.909 7f4e95e73700 1 --2- 172.21.2.112:0/937777218 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4e780186c0 0x7f4e7801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:25.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:25.909 7f4e95e73700 1 --2- 172.21.2.112:0/937777218 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4e780186c0 0x7f4e7801abc0 crc :-1 s=READY pgs=719 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:25.926 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:25.940 7f4e98057700 1 -- 172.21.2.112:0/937777218 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e900ed660 con 0x7f4e900bf750 2021-02-10T11:35:25.929 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:25.942 7f4e827fc700 1 -- 172.21.2.112:0/937777218 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4e8c000e00 con 0x7f4e900bf750 2021-02-10T11:35:26.054 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:26.067 7f4e98057700 1 -- 172.21.2.112:0/937777218 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4e90082d50 con 0x7f4e900bf750 2021-02-10T11:35:26.054 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:26.068 7f4e827fc700 1 -- 172.21.2.112:0/937777218 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v64) v1 ==== 74+0+13996 (secure 0 0 0) 0x7f4e8c004140 con 0x7f4e900bf750 2021-02-10T11:35:26.055 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:35:26.055 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":64,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:34:46.584716","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:34:43.550220","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":13,"pool_name":"cephfs_data","create_time":"2021-02-10 11:34:44.578113","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:35:26.056 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:26.069 7f4e98057700 1 -- 172.21.2.112:0/937777218 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4e780186c0 msgr2=0x7f4e7801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:26.056 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:26.069 7f4e98057700 1 --2- 172.21.2.112:0/937777218 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4e780186c0 0x7f4e7801abc0 crc :-1 s=READY pgs=719 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:26.056 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:26.069 7f4e98057700 1 -- 172.21.2.112:0/937777218 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4e900bf750 msgr2=0x7f4e901a7500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:26.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:26.069 7f4e98057700 1 --2- 172.21.2.112:0/937777218 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4e900bf750 0x7f4e901a7500 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 rx=0x7f4e8c001c60 tx=0x7f4e8c00a8d0).stop 2021-02-10T11:35:26.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:26.070 7f4e98057700 1 -- 172.21.2.112:0/937777218 shutdown_connections 2021-02-10T11:35:26.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:26.070 7f4e98057700 1 --2- 172.21.2.112:0/937777218 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4e780186c0 0x7f4e7801abc0 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:26.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:26.070 7f4e98057700 1 --2- 172.21.2.112:0/937777218 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4e901a7b20 0x7f4e901889e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:26.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:26.070 7f4e98057700 1 --2- 172.21.2.112:0/937777218 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4e900bf750 0x7f4e901a7500 unknown :-1 s=CLOSED pgs=1487 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:26.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:26.070 7f4e98057700 1 --2- 172.21.2.112:0/937777218 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4e90096fb0 0x7f4e900bf130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:26.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:26.070 7f4e98057700 1 -- 172.21.2.112:0/937777218 >> 172.21.2.112:0/937777218 conn(0x7f4e900bfe10 msgr2=0x7f4e90097a80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:26.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:26.070 7f4e98057700 1 -- 172.21.2.112:0/937777218 shutdown_connections 2021-02-10T11:35:26.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:26.070 7f4e98057700 1 -- 172.21.2.112:0/937777218 wait complete. 2021-02-10T11:35:26.065 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:13:57.508758', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 12, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:34:43.550220', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 13, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:34:44.578113', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '64', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:35:26.065 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:35:26.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:26.492 7f29ffb61700 1 Processor -- start 2021-02-10T11:35:26.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:26.492 7f29ffb61700 1 -- start start 2021-02-10T11:35:26.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:26.492 7f29ffb61700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f29f80e5170 0x7f29f81b38e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:26.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:26.492 7f29ffb61700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f29f81b3f00 0x7f29f8077510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:26.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:26.492 7f29ffb61700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29f8078b90 con 0x7f29f80e5570 2021-02-10T11:35:26.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:26.492 7f29ffb61700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29f807a120 con 0x7f29f81b3f00 2021-02-10T11:35:26.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:26.492 7f29ffb61700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29f807a310 con 0x7f29f80e5170 2021-02-10T11:35:26.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:26.493 7f29fd97d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f29f80e5170 0x7f29f81b38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:26.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:26.493 7f29fd17c700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f29f81b3f00 0x7f29f8077510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:26.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:26.493 7f29fd97d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f29f80e5170 0x7f29f81b38e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3301/0 says I am v2:172.21.2.112:50922/0 (socket says 172.21.2.112:50922) 2021-02-10T11:35:26.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:26.493 7f29fd97d700 1 -- 172.21.2.112:0/1761730838 learned_addr learned my addr 172.21.2.112:0/1761730838 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:35:26.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:26.494 7f29fd97d700 1 -- 172.21.2.112:0/1761730838 >> v2:172.21.2.120:3300/0 conn(0x7f29f81b3f00 msgr2=0x7f29f8077510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:26.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:26.494 7f29fd97d700 1 --2- 172.21.2.112:0/1761730838 >> v2:172.21.2.120:3300/0 conn(0x7f29f81b3f00 0x7f29f8077510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:26.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:26.494 7f29fd97d700 1 -- 172.21.2.112:0/1761730838 >> v1:172.21.2.120:6789/0 conn(0x7f29f80e5570 legacy=0x7f29f8077b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:26.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:26.494 7f29fd97d700 1 -- 172.21.2.112:0/1761730838 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29f807a500 con 0x7f29f80e5170 2021-02-10T11:35:26.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:26.494 7f29fd97d700 1 --2- 172.21.2.112:0/1761730838 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f29f80e5170 0x7f29f81b38e0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f29e8006a80 tx=0x7f29e8009f00).ready entity=mon.2 client_cookie=253ffc4002945402 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:26.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:26.494 7f29fc97b700 1 -- 172.21.2.112:0/1761730838 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f29e800b3b0 con 0x7f29f80e5170 2021-02-10T11:35:26.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:26.494 7f29fc97b700 1 -- 172.21.2.112:0/1761730838 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29e800b630 con 0x7f29f80e5170 2021-02-10T11:35:26.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:26.494 7f29fc97b700 1 -- 172.21.2.112:0/1761730838 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f29e800ba80 con 0x7f29f80e5170 2021-02-10T11:35:26.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:26.495 7f29ffb61700 1 -- 172.21.2.112:0/1761730838 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f29f80e5170 msgr2=0x7f29f81b38e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:26.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:26.495 7f29ffb61700 1 --2- 172.21.2.112:0/1761730838 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f29f80e5170 0x7f29f81b38e0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f29e8006a80 tx=0x7f29e8009f00).stop 2021-02-10T11:35:26.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:26.495 7f29ffb61700 1 -- 172.21.2.112:0/1761730838 shutdown_connections 2021-02-10T11:35:26.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:26.495 7f29ffb61700 1 --2- 172.21.2.112:0/1761730838 >> v2:172.21.2.120:3300/0 conn(0x7f29f81b3f00 0x7f29f8077510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:26.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:26.495 7f29ffb61700 1 --2- 172.21.2.112:0/1761730838 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f29f80e5170 0x7f29f81b38e0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:26.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:26.495 7f29ffb61700 1 -- 172.21.2.112:0/1761730838 >> 172.21.2.112:0/1761730838 conn(0x7f29f80ce820 msgr2=0x7f29f80e4660 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:26.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:26.495 7f29ffb61700 1 -- 172.21.2.112:0/1761730838 shutdown_connections 2021-02-10T11:35:26.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:26.495 7f29ffb61700 1 -- 172.21.2.112:0/1761730838 wait complete. 2021-02-10T11:35:26.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:26.509 7f29ffb61700 1 Processor -- start 2021-02-10T11:35:26.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:26.510 7f29ffb61700 1 -- start start 2021-02-10T11:35:26.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:26.510 7f29ffb61700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f29f810da30 0x7f29f8108a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:26.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:26.510 7f29ffb61700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f29f8094b60 0x7f29f8094f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:26.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:26.510 7f29ffb61700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f29f810c1c0 0x7f29f810c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:26.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:26.510 7f29ffb61700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29f8095580 con 0x7f29f8094b60 2021-02-10T11:35:26.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:26.510 7f29ffb61700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29f80945e0 con 0x7f29f810c1c0 2021-02-10T11:35:26.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:26.510 7f29ffb61700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29f8109030 con 0x7f29f810da30 2021-02-10T11:35:26.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:26.510 7f29fd17c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f29f8094b60 0x7f29f8094f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:26.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:26.510 7f29fd17c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f29f8094b60 0x7f29f8094f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:51246/0 (socket says 172.21.2.112:51246) 2021-02-10T11:35:26.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:26.510 7f29fd17c700 1 -- 172.21.2.112:0/1201627405 learned_addr learned my addr 172.21.2.112:0/1201627405 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:35:26.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:26.510 7f29fd97d700 1 --2- 172.21.2.112:0/1201627405 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f29f810da30 0x7f29f8108a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:26.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:26.511 7f29fe17e700 1 --2- 172.21.2.112:0/1201627405 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f29f810c1c0 0x7f29f810c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:26.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:26.511 7f29fd17c700 1 -- 172.21.2.112:0/1201627405 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f29f810da30 msgr2=0x7f29f8108a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:26.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:26.511 7f29fd17c700 1 --2- 172.21.2.112:0/1201627405 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f29f810da30 0x7f29f8108a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:26.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:26.511 7f29fd17c700 1 -- 172.21.2.112:0/1201627405 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f29f810c1c0 msgr2=0x7f29f810c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:26.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:26.511 7f29fd17c700 1 --2- 172.21.2.112:0/1201627405 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f29f810c1c0 0x7f29f810c5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:26.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:26.511 7f29fd17c700 1 -- 172.21.2.112:0/1201627405 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29f810de30 con 0x7f29f8094b60 2021-02-10T11:35:26.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:26.511 7f29fd17c700 1 --2- 172.21.2.112:0/1201627405 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f29f8094b60 0x7f29f8094f60 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 rx=0x7f29ec00b560 tx=0x7f29ec00b900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:26.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:26.511 7f29f67fc700 1 -- 172.21.2.112:0/1201627405 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f29ec0009b0 con 0x7f29f8094b60 2021-02-10T11:35:26.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:26.511 7f29f67fc700 1 -- 172.21.2.112:0/1201627405 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29ec00bdc0 con 0x7f29f8094b60 2021-02-10T11:35:26.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:26.511 7f29f67fc700 1 -- 172.21.2.112:0/1201627405 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f29ec002030 con 0x7f29f8094b60 2021-02-10T11:35:26.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:26.511 7f29ffb61700 1 -- 172.21.2.112:0/1201627405 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29f810de30 con 0x7f29f8094b60 2021-02-10T11:35:26.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:26.511 7f29ffb61700 1 -- 172.21.2.112:0/1201627405 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29f810de30 con 0x7f29f8094b60 2021-02-10T11:35:26.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:26.512 7f29f67fc700 1 -- 172.21.2.112:0/1201627405 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f29ec000c90 con 0x7f29f8094b60 2021-02-10T11:35:26.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:26.512 7f29f67fc700 1 --2- 172.21.2.112:0/1201627405 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f29d80186c0 0x7f29d801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:26.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:26.512 7f29f67fc700 1 -- 172.21.2.112:0/1201627405 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f29ec032640 con 0x7f29f8094b60 2021-02-10T11:35:26.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:26.512 7f29fd97d700 1 --2- 172.21.2.112:0/1201627405 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f29d80186c0 0x7f29d801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:26.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:26.512 7f29fd97d700 1 --2- 172.21.2.112:0/1201627405 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f29d80186c0 0x7f29d801abc0 crc :-1 s=READY pgs=720 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:26.529 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:26.543 7f29ffb61700 1 -- 172.21.2.112:0/1201627405 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29f8105bc0 con 0x7f29f8094b60 2021-02-10T11:35:26.532 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:26.545 7f29f67fc700 1 -- 172.21.2.112:0/1201627405 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f29ec006b00 con 0x7f29f8094b60 2021-02-10T11:35:26.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:26.673 7f29ffb61700 1 -- 172.21.2.112:0/1201627405 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f29f8078b90 con 0x7f29f8094b60 2021-02-10T11:35:26.660 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:26.673 7f29f67fc700 1 -- 172.21.2.112:0/1201627405 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v64) v1 ==== 92+0+10 (secure 0 0 0) 0x7f29ec00bdc0 con 0x7f29f8094b60 2021-02-10T11:35:26.660 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:35:26.661 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:26.675 7f29ffb61700 1 -- 172.21.2.112:0/1201627405 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f29d80186c0 msgr2=0x7f29d801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:26.662 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:26.675 7f29ffb61700 1 --2- 172.21.2.112:0/1201627405 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f29d80186c0 0x7f29d801abc0 crc :-1 s=READY pgs=720 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:26.662 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:26.675 7f29ffb61700 1 -- 172.21.2.112:0/1201627405 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f29f8094b60 msgr2=0x7f29f8094f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:26.662 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:26.675 7f29ffb61700 1 --2- 172.21.2.112:0/1201627405 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f29f8094b60 0x7f29f8094f60 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 rx=0x7f29ec00b560 tx=0x7f29ec00b900).stop 2021-02-10T11:35:26.662 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:26.675 7f29ffb61700 1 -- 172.21.2.112:0/1201627405 shutdown_connections 2021-02-10T11:35:26.662 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:26.675 7f29ffb61700 1 --2- 172.21.2.112:0/1201627405 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f29d80186c0 0x7f29d801abc0 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:26.662 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:26.675 7f29ffb61700 1 --2- 172.21.2.112:0/1201627405 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f29f810c1c0 0x7f29f810c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:26.662 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:26.675 7f29ffb61700 1 --2- 172.21.2.112:0/1201627405 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f29f8094b60 0x7f29f8094f60 unknown :-1 s=CLOSED pgs=1488 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:26.662 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:26.675 7f29ffb61700 1 --2- 172.21.2.112:0/1201627405 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f29f810da30 0x7f29f8108a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:26.662 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:26.675 7f29ffb61700 1 -- 172.21.2.112:0/1201627405 >> 172.21.2.112:0/1201627405 conn(0x7f29f80a2e80 msgr2=0x7f29f80b1d20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:26.662 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:26.675 7f29ffb61700 1 -- 172.21.2.112:0/1201627405 shutdown_connections 2021-02-10T11:35:26.662 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:26.675 7f29ffb61700 1 -- 172.21.2.112:0/1201627405 wait complete. 2021-02-10T11:35:26.670 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:35:27.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.097 7f3623ffe700 1 Processor -- start 2021-02-10T11:35:27.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.097 7f3623ffe700 1 -- start start 2021-02-10T11:35:27.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.098 7f3623ffe700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f361c0e5170 0x7f361c1b38e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:27.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.098 7f3623ffe700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f361c1b3f00 0x7f361c077510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:27.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.098 7f3623ffe700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f361c078b90 con 0x7f361c0e5570 2021-02-10T11:35:27.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.098 7f3623ffe700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f361c07a120 con 0x7f361c0e5170 2021-02-10T11:35:27.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.098 7f3623ffe700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f361c07a310 con 0x7f361c1b3f00 2021-02-10T11:35:27.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.099 7f3621619700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f361c1b3f00 0x7f361c077510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:27.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.099 7f3621e1a700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f361c0e5170 0x7f361c1b38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:27.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.099 7f3621619700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f361c1b3f00 0x7f361c077510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3301/0 says I am v2:172.21.2.112:50934/0 (socket says 172.21.2.112:50934) 2021-02-10T11:35:27.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.099 7f3621619700 1 -- 172.21.2.112:0/645076245 learned_addr learned my addr 172.21.2.112:0/645076245 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:35:27.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.099 7f3621e1a700 1 -- 172.21.2.112:0/645076245 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f361c1b3f00 msgr2=0x7f361c077510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:27.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.099 7f3621e1a700 1 --2- 172.21.2.112:0/645076245 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f361c1b3f00 0x7f361c077510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:27.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.099 7f3621e1a700 1 -- 172.21.2.112:0/645076245 >> v1:172.21.2.120:6789/0 conn(0x7f361c0e5570 legacy=0x7f361c077b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:27.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.099 7f3621e1a700 1 -- 172.21.2.112:0/645076245 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f361c07a530 con 0x7f361c0e5170 2021-02-10T11:35:27.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.099 7f3621619700 1 --2- 172.21.2.112:0/645076245 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f361c1b3f00 0x7f361c077510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:35:27.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.100 7f3621e1a700 1 --2- 172.21.2.112:0/645076245 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f361c0e5170 0x7f361c1b38e0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 rx=0x7f3610006650 tx=0x7f3610009a80).ready entity=mon.1 client_cookie=49ede12018f7c5ae server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:27.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.100 7f3620e18700 1 -- 172.21.2.112:0/645076245 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f36100009b0 con 0x7f361c0e5170 2021-02-10T11:35:27.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.100 7f3620e18700 1 -- 172.21.2.112:0/645076245 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f361000c0c0 con 0x7f361c0e5170 2021-02-10T11:35:27.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.100 7f3620e18700 1 -- 172.21.2.112:0/645076245 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f361000c490 con 0x7f361c0e5170 2021-02-10T11:35:27.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.100 7f3623ffe700 1 -- 172.21.2.112:0/645076245 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f361c0e5170 msgr2=0x7f361c1b38e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:27.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.100 7f3623ffe700 1 --2- 172.21.2.112:0/645076245 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f361c0e5170 0x7f361c1b38e0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 rx=0x7f3610006650 tx=0x7f3610009a80).stop 2021-02-10T11:35:27.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.100 7f3623ffe700 1 -- 172.21.2.112:0/645076245 shutdown_connections 2021-02-10T11:35:27.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.100 7f3623ffe700 1 --2- 172.21.2.112:0/645076245 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f361c1b3f00 0x7f361c077510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:27.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.100 7f3623ffe700 1 --2- 172.21.2.112:0/645076245 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f361c0e5170 0x7f361c1b38e0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:27.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.101 7f3623ffe700 1 -- 172.21.2.112:0/645076245 >> 172.21.2.112:0/645076245 conn(0x7f361c0ce820 msgr2=0x7f361c0e4660 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:27.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.101 7f3623ffe700 1 -- 172.21.2.112:0/645076245 shutdown_connections 2021-02-10T11:35:27.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.101 7f3623ffe700 1 -- 172.21.2.112:0/645076245 wait complete. 2021-02-10T11:35:27.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.115 7f3623ffe700 1 Processor -- start 2021-02-10T11:35:27.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.115 7f3623ffe700 1 -- start start 2021-02-10T11:35:27.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.115 7f3623ffe700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f361c09adb0 0x7f361c0c15a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:27.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.115 7f3623ffe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f361c09b1b0 0x7f361c0c1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:27.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.115 7f3623ffe700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f361c0c2a80 0x7f361c0c2e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:27.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.115 7f3623ffe700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f361c0c22b0 con 0x7f361c09b1b0 2021-02-10T11:35:27.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.115 7f3623ffe700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f361c187fc0 con 0x7f361c0c2a80 2021-02-10T11:35:27.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.115 7f3623ffe700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f361c1881b0 con 0x7f361c09adb0 2021-02-10T11:35:27.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.116 7f3621619700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f361c09b1b0 0x7f361c0c1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:27.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.116 7f3621619700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f361c09b1b0 0x7f361c0c1bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:51256/0 (socket says 172.21.2.112:51256) 2021-02-10T11:35:27.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.116 7f3621619700 1 -- 172.21.2.112:0/355538653 learned_addr learned my addr 172.21.2.112:0/355538653 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:35:27.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.116 7f3621e1a700 1 --2- 172.21.2.112:0/355538653 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f361c09adb0 0x7f361c0c15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:27.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.116 7f362261b700 1 --2- 172.21.2.112:0/355538653 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f361c0c2a80 0x7f361c0c2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:27.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.116 7f3621619700 1 -- 172.21.2.112:0/355538653 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f361c09adb0 msgr2=0x7f361c0c15a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:27.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.116 7f3621619700 1 --2- 172.21.2.112:0/355538653 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f361c09adb0 0x7f361c0c15a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:27.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.116 7f3621619700 1 -- 172.21.2.112:0/355538653 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f361c0c2a80 msgr2=0x7f361c0c2e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:27.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.116 7f3621619700 1 --2- 172.21.2.112:0/355538653 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f361c0c2a80 0x7f361c0c2e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:27.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.116 7f3621619700 1 -- 172.21.2.112:0/355538653 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f361c0c34d0 con 0x7f361c09b1b0 2021-02-10T11:35:27.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.116 7f362261b700 1 --2- 172.21.2.112:0/355538653 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f361c0c2a80 0x7f361c0c2e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:35:27.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.116 7f3621619700 1 --2- 172.21.2.112:0/355538653 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f361c09b1b0 0x7f361c0c1bc0 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 rx=0x7f3618006ee0 tx=0x7f361800b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:27.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.116 7f360e7fc700 1 -- 172.21.2.112:0/355538653 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f361800bd50 con 0x7f361c09b1b0 2021-02-10T11:35:27.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.116 7f360e7fc700 1 -- 172.21.2.112:0/355538653 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36180052e0 con 0x7f361c09b1b0 2021-02-10T11:35:27.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.116 7f3623ffe700 1 -- 172.21.2.112:0/355538653 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f361c0c34d0 con 0x7f361c09b1b0 2021-02-10T11:35:27.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.116 7f360e7fc700 1 -- 172.21.2.112:0/355538653 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3618005730 con 0x7f361c09b1b0 2021-02-10T11:35:27.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.117 7f3623ffe700 1 -- 172.21.2.112:0/355538653 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f361c0c34d0 con 0x7f361c09b1b0 2021-02-10T11:35:27.109 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.117 7f360e7fc700 1 -- 172.21.2.112:0/355538653 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f36180054c0 con 0x7f361c09b1b0 2021-02-10T11:35:27.109 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.117 7f360e7fc700 1 --2- 172.21.2.112:0/355538653 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3604018730 0x7f360401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:27.109 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.117 7f360e7fc700 1 -- 172.21.2.112:0/355538653 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f3618032630 con 0x7f361c09b1b0 2021-02-10T11:35:27.109 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.118 7f3621e1a700 1 --2- 172.21.2.112:0/355538653 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3604018730 0x7f360401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:27.109 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.118 7f3621e1a700 1 --2- 172.21.2.112:0/355538653 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3604018730 0x7f360401ac10 crc :-1 s=READY pgs=721 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:27.134 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.147 7f3623ffe700 1 -- 172.21.2.112:0/355538653 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f361c078b90 con 0x7f361c09b1b0 2021-02-10T11:35:27.136 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.150 7f360e7fc700 1 -- 172.21.2.112:0/355538653 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f361801f4f0 con 0x7f361c09b1b0 2021-02-10T11:35:27.266 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.279 7f3623ffe700 1 -- 172.21.2.112:0/355538653 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f361c078b90 con 0x7f361c09b1b0 2021-02-10T11:35:27.266 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.279 7f360e7fc700 1 -- 172.21.2.112:0/355538653 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v64) v1 ==== 104+0+10 (secure 0 0 0) 0x7f361800bd50 con 0x7f361c09b1b0 2021-02-10T11:35:27.266 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:35:27.268 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.281 7f3623ffe700 1 -- 172.21.2.112:0/355538653 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3604018730 msgr2=0x7f360401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:27.268 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.281 7f3623ffe700 1 --2- 172.21.2.112:0/355538653 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3604018730 0x7f360401ac10 crc :-1 s=READY pgs=721 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:27.268 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.281 7f3623ffe700 1 -- 172.21.2.112:0/355538653 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f361c09b1b0 msgr2=0x7f361c0c1bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:27.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.281 7f3623ffe700 1 --2- 172.21.2.112:0/355538653 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f361c09b1b0 0x7f361c0c1bc0 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 rx=0x7f3618006ee0 tx=0x7f361800b960).stop 2021-02-10T11:35:27.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.281 7f3623ffe700 1 -- 172.21.2.112:0/355538653 shutdown_connections 2021-02-10T11:35:27.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.281 7f3623ffe700 1 --2- 172.21.2.112:0/355538653 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3604018730 0x7f360401ac10 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:27.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.282 7f3623ffe700 1 --2- 172.21.2.112:0/355538653 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f361c0c2a80 0x7f361c0c2e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:27.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.282 7f3623ffe700 1 --2- 172.21.2.112:0/355538653 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f361c09b1b0 0x7f361c0c1bc0 unknown :-1 s=CLOSED pgs=1489 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:27.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.282 7f3623ffe700 1 --2- 172.21.2.112:0/355538653 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f361c09adb0 0x7f361c0c15a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:27.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.282 7f3623ffe700 1 -- 172.21.2.112:0/355538653 >> 172.21.2.112:0/355538653 conn(0x7f361c10e5a0 msgr2=0x7f361c0b8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:27.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.282 7f3623ffe700 1 -- 172.21.2.112:0/355538653 shutdown_connections 2021-02-10T11:35:27.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.282 7f3623ffe700 1 -- 172.21.2.112:0/355538653 wait complete. 2021-02-10T11:35:27.277 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:35:27.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.704 7f89270f2700 1 Processor -- start 2021-02-10T11:35:27.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.704 7f89270f2700 1 -- start start 2021-02-10T11:35:27.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.705 7f89270f2700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f89200e5170 0x7f89201b38e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:27.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.705 7f89270f2700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f89201b3f00 0x7f8920077510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:27.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.705 7f89270f2700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8920078b90 con 0x7f89200e5570 2021-02-10T11:35:27.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.705 7f89270f2700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f892007a120 con 0x7f89201b3f00 2021-02-10T11:35:27.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.705 7f89270f2700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f892007a310 con 0x7f89200e5170 2021-02-10T11:35:27.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.705 7f892570f700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f89200e5570 0x7f8920077b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:50730/0 (socket says 172.21.2.112:50730) 2021-02-10T11:35:27.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.705 7f892570f700 1 -- 172.21.2.112:0/2607047783 learned_addr learned my addr 172.21.2.112:0/2607047783 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:35:27.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.705 7f891f7fe700 1 -- 172.21.2.112:0/2607047783 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1333360502 0 0) 0x7f8918001500 con 0x7f89200e5570 2021-02-10T11:35:27.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.705 7f891f7fe700 1 -- 172.21.2.112:0/2607047783 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f890c002fe0 con 0x7f89200e5570 2021-02-10T11:35:27.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.705 7f8924f0e700 1 --2- 172.21.2.112:0/2607047783 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f89200e5170 0x7f89201b38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:27.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.705 7f891ffff700 1 --2- 172.21.2.112:0/2607047783 >> v2:172.21.2.120:3300/0 conn(0x7f89201b3f00 0x7f8920077510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:27.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.705 7f891f7fe700 1 -- 172.21.2.112:0/2607047783 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3155834743 0 0) 0x7f8918001270 con 0x7f89200e5570 2021-02-10T11:35:27.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.705 7f891f7fe700 1 -- 172.21.2.112:0/2607047783 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f89200e5170 msgr2=0x7f89201b38e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:27.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.705 7f891f7fe700 1 --2- 172.21.2.112:0/2607047783 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f89200e5170 0x7f89201b38e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:27.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.705 7f891f7fe700 1 -- 172.21.2.112:0/2607047783 >> v2:172.21.2.120:3300/0 conn(0x7f89201b3f00 msgr2=0x7f8920077510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:27.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.705 7f891f7fe700 1 --2- 172.21.2.112:0/2607047783 >> v2:172.21.2.120:3300/0 conn(0x7f89201b3f00 0x7f8920077510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:27.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.705 7f891f7fe700 1 -- 172.21.2.112:0/2607047783 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f892007a500 con 0x7f89200e5570 2021-02-10T11:35:27.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.705 7f891f7fe700 1 -- 172.21.2.112:0/2607047783 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f89180019e0 con 0x7f89200e5570 2021-02-10T11:35:27.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.705 7f891f7fe700 1 -- 172.21.2.112:0/2607047783 >> v1:172.21.2.112:6789/0 conn(0x7f89200e5570 legacy=0x7f8920077b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:27.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.706 7f891f7fe700 1 --2- 172.21.2.112:0/2607047783 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f890c003b90 0x7f890c006000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:27.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.706 7f891f7fe700 1 --2- 172.21.2.112:0/2607047783 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f890c006790 0x7f890c008cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:27.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.706 7f891f7fe700 1 --2- 172.21.2.112:0/2607047783 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f890c009550 0x7f890c00ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:27.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.706 7f891f7fe700 1 -- 172.21.2.112:0/2607047783 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f890c0502f0 con 0x7f890c006790 2021-02-10T11:35:27.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.706 7f891f7fe700 1 -- 172.21.2.112:0/2607047783 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f890c0504a0 con 0x7f890c009550 2021-02-10T11:35:27.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.706 7f891f7fe700 1 -- 172.21.2.112:0/2607047783 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f890c0506c0 con 0x7f890c003b90 2021-02-10T11:35:27.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.706 7f891ffff700 1 --2- 172.21.2.112:0/2607047783 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f890c006790 0x7f890c008cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:27.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.706 7f8924f0e700 1 --2- 172.21.2.112:0/2607047783 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f890c003b90 0x7f890c006000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:27.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.706 7f892570f700 1 --2- 172.21.2.112:0/2607047783 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f890c009550 0x7f890c00ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:27.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.706 7f891ffff700 1 -- 172.21.2.112:0/2607047783 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f890c003b90 msgr2=0x7f890c006000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:27.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.706 7f891ffff700 1 --2- 172.21.2.112:0/2607047783 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f890c003b90 0x7f890c006000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:27.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.706 7f891ffff700 1 -- 172.21.2.112:0/2607047783 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f890c009550 msgr2=0x7f890c00ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:27.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.706 7f891ffff700 1 --2- 172.21.2.112:0/2607047783 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f890c009550 0x7f890c00ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:27.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.706 7f891ffff700 1 -- 172.21.2.112:0/2607047783 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f890c050960 con 0x7f890c006790 2021-02-10T11:35:27.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.706 7f892570f700 1 --2- 172.21.2.112:0/2607047783 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f890c009550 0x7f890c00ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:35:27.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.706 7f891ffff700 1 --2- 172.21.2.112:0/2607047783 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f890c006790 0x7f890c008cb0 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 rx=0x7f8910006360 tx=0x7f8910009c30).ready entity=mon.0 client_cookie=f46f11bcfe32a47f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:27.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.706 7f891f7fe700 1 -- 172.21.2.112:0/2607047783 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f89100009b0 con 0x7f890c006790 2021-02-10T11:35:27.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.706 7f891f7fe700 1 -- 172.21.2.112:0/2607047783 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8910000c90 con 0x7f890c006790 2021-02-10T11:35:27.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.706 7f891f7fe700 1 -- 172.21.2.112:0/2607047783 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f89100009b0 con 0x7f890c006790 2021-02-10T11:35:27.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.707 7f89270f2700 1 -- 172.21.2.112:0/2607047783 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f890c006790 msgr2=0x7f890c008cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:27.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.707 7f89270f2700 1 --2- 172.21.2.112:0/2607047783 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f890c006790 0x7f890c008cb0 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 rx=0x7f8910006360 tx=0x7f8910009c30).stop 2021-02-10T11:35:27.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.707 7f892570f700 1 -- 172.21.2.112:0/2607047783 reap_dead start 2021-02-10T11:35:27.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.707 7f89270f2700 1 -- 172.21.2.112:0/2607047783 shutdown_connections 2021-02-10T11:35:27.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.707 7f89270f2700 1 -- 172.21.2.112:0/2607047783 >> 172.21.2.112:0/2607047783 conn(0x7f89200ce820 msgr2=0x7f89200e4660 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:27.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.707 7f89270f2700 1 -- 172.21.2.112:0/2607047783 shutdown_connections 2021-02-10T11:35:27.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.707 7f89270f2700 1 -- 172.21.2.112:0/2607047783 wait complete. 2021-02-10T11:35:27.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.722 7f89270f2700 1 Processor -- start 2021-02-10T11:35:27.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.722 7f89270f2700 1 -- start start 2021-02-10T11:35:27.714 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.723 7f89270f2700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f89200ab3f0 0x7f89201a6f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:27.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.723 7f89270f2700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f89200a0be0 0x7f89201a7560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:27.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.723 7f89270f2700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f89200af470 0x7f89200af870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:27.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.723 7f89270f2700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89200ab9e0 con 0x7f89200a0be0 2021-02-10T11:35:27.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.723 7f89270f2700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89201a7b80 con 0x7f89200af470 2021-02-10T11:35:27.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.723 7f89270f2700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89201a7d60 con 0x7f89200ab3f0 2021-02-10T11:35:27.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.723 7f891ffff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f89200a0be0 0x7f89201a7560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:27.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.723 7f891ffff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f89200a0be0 0x7f89201a7560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:51280/0 (socket says 172.21.2.112:51280) 2021-02-10T11:35:27.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.723 7f891ffff700 1 -- 172.21.2.112:0/1160395857 learned_addr learned my addr 172.21.2.112:0/1160395857 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:35:27.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.723 7f891ffff700 1 -- 172.21.2.112:0/1160395857 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f89200ab3f0 msgr2=0x7f89201a6f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:27.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.723 7f891ffff700 1 --2- 172.21.2.112:0/1160395857 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f89200ab3f0 0x7f89201a6f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:27.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.723 7f891ffff700 1 -- 172.21.2.112:0/1160395857 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f89200af470 msgr2=0x7f89200af870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:27.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.723 7f891ffff700 1 --2- 172.21.2.112:0/1160395857 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f89200af470 0x7f89200af870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:27.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.723 7f891ffff700 1 -- 172.21.2.112:0/1160395857 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89200afe90 con 0x7f89200a0be0 2021-02-10T11:35:27.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.723 7f891ffff700 1 --2- 172.21.2.112:0/1160395857 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f89200a0be0 0x7f89201a7560 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 rx=0x7f89100068c0 tx=0x7f891000b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:27.716 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.724 7f891d7fa700 1 -- 172.21.2.112:0/1160395857 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8910002030 con 0x7f89200a0be0 2021-02-10T11:35:27.716 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.724 7f891d7fa700 1 -- 172.21.2.112:0/1160395857 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8910009530 con 0x7f89200a0be0 2021-02-10T11:35:27.716 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.724 7f891d7fa700 1 -- 172.21.2.112:0/1160395857 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f89100024e0 con 0x7f89200a0be0 2021-02-10T11:35:27.716 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.724 7f89270f2700 1 -- 172.21.2.112:0/1160395857 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89200afe90 con 0x7f89200a0be0 2021-02-10T11:35:27.716 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.724 7f89270f2700 1 -- 172.21.2.112:0/1160395857 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89200abf60 con 0x7f89200a0be0 2021-02-10T11:35:27.716 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.724 7f891d7fa700 1 -- 172.21.2.112:0/1160395857 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8910000c90 con 0x7f89200a0be0 2021-02-10T11:35:27.716 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.725 7f891d7fa700 1 --2- 172.21.2.112:0/1160395857 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f89040186c0 0x7f890401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:27.716 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.725 7f891d7fa700 1 -- 172.21.2.112:0/1160395857 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f89100325b0 con 0x7f89200a0be0 2021-02-10T11:35:27.716 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.725 7f8924f0e700 1 --2- 172.21.2.112:0/1160395857 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f89040186c0 0x7f890401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:27.716 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.726 7f8924f0e700 1 --2- 172.21.2.112:0/1160395857 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f89040186c0 0x7f890401abc0 crc :-1 s=READY pgs=722 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:27.741 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.754 7f89270f2700 1 -- 172.21.2.112:0/1160395857 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f892010c970 con 0x7f89200a0be0 2021-02-10T11:35:27.743 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.757 7f891d7fa700 1 -- 172.21.2.112:0/1160395857 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8910030d60 con 0x7f89200a0be0 2021-02-10T11:35:27.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.884 7f89270f2700 1 -- 172.21.2.112:0/1160395857 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f89200fe290 con 0x7f89200a0be0 2021-02-10T11:35:27.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.885 7f891d7fa700 1 -- 172.21.2.112:0/1160395857 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v64) v1 ==== 100+0+10 (secure 0 0 0) 0x7f891000bd60 con 0x7f89200a0be0 2021-02-10T11:35:27.872 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:35:27.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.887 7f89270f2700 1 -- 172.21.2.112:0/1160395857 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f89040186c0 msgr2=0x7f890401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:27.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.887 7f89270f2700 1 --2- 172.21.2.112:0/1160395857 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f89040186c0 0x7f890401abc0 crc :-1 s=READY pgs=722 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:27.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.887 7f89270f2700 1 -- 172.21.2.112:0/1160395857 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f89200a0be0 msgr2=0x7f89201a7560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:27.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.887 7f89270f2700 1 --2- 172.21.2.112:0/1160395857 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f89200a0be0 0x7f89201a7560 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 rx=0x7f89100068c0 tx=0x7f891000b970).stop 2021-02-10T11:35:27.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.887 7f89270f2700 1 -- 172.21.2.112:0/1160395857 shutdown_connections 2021-02-10T11:35:27.875 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.887 7f89270f2700 1 --2- 172.21.2.112:0/1160395857 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f89040186c0 0x7f890401abc0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:27.875 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.887 7f89270f2700 1 --2- 172.21.2.112:0/1160395857 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f89200af470 0x7f89200af870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:27.875 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.887 7f89270f2700 1 --2- 172.21.2.112:0/1160395857 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f89200a0be0 0x7f89201a7560 unknown :-1 s=CLOSED pgs=1492 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:27.875 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.887 7f89270f2700 1 --2- 172.21.2.112:0/1160395857 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f89200ab3f0 0x7f89201a6f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:27.875 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.887 7f89270f2700 1 -- 172.21.2.112:0/1160395857 >> 172.21.2.112:0/1160395857 conn(0x7f89200a1220 msgr2=0x7f89200a27f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:27.875 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.887 7f89270f2700 1 -- 172.21.2.112:0/1160395857 shutdown_connections 2021-02-10T11:35:27.875 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:27.887 7f89270f2700 1 -- 172.21.2.112:0/1160395857 wait complete. 2021-02-10T11:35:27.882 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:35:27.883 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:35:28.308 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:28.305 7f573abf2700 1 Processor -- start 2021-02-10T11:35:28.308 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:28.305 7f573abf2700 1 -- start start 2021-02-10T11:35:28.308 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:28.305 7f573abf2700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f57340d18d0 0x7f5734083ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:28.308 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:28.305 7f573abf2700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5734082d50 con 0x7f57340e53e0 2021-02-10T11:35:28.308 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:28.305 7f573abf2700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57340e57e0 con 0x7f5734082920 2021-02-10T11:35:28.308 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:28.305 7f573abf2700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57341b0d40 con 0x7f57340d18d0 2021-02-10T11:35:28.308 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:28.305 7f573920f700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f57340d18d0 0x7f5734083ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:28.308 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:28.305 7f573920f700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f57340d18d0 0x7f5734083ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:51284/0 (socket says 172.21.2.112:51284) 2021-02-10T11:35:28.308 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:28.305 7f573920f700 1 -- 172.21.2.112:0/3904015762 learned_addr learned my addr 172.21.2.112:0/3904015762 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:35:28.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:28.306 7f573920f700 1 -- 172.21.2.112:0/3904015762 >> v1:172.21.2.120:6789/0 conn(0x7f5734082920 legacy=0x7f57340832d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:28.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:28.306 7f573920f700 1 -- 172.21.2.112:0/3904015762 >> v1:172.21.2.112:6789/0 conn(0x7f57340e53e0 legacy=0x7f57341b0550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:28.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:28.306 7f573920f700 1 -- 172.21.2.112:0/3904015762 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57340d1cd0 con 0x7f57340d18d0 2021-02-10T11:35:28.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:28.306 7f573920f700 1 --2- 172.21.2.112:0/3904015762 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f57340d18d0 0x7f5734083ac0 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 rx=0x7f572c006b80 tx=0x7f572c009b10).ready entity=mon.0 client_cookie=c89f1408e53fc3e1 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:28.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:28.306 7f57337fe700 1 -- 172.21.2.112:0/3904015762 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f572c0022e0 con 0x7f57340d18d0 2021-02-10T11:35:28.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:28.306 7f57337fe700 1 -- 172.21.2.112:0/3904015762 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f572c002620 con 0x7f57340d18d0 2021-02-10T11:35:28.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:28.306 7f57337fe700 1 -- 172.21.2.112:0/3904015762 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f572c002b00 con 0x7f57340d18d0 2021-02-10T11:35:28.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:28.306 7f573abf2700 1 -- 172.21.2.112:0/3904015762 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f57340d18d0 msgr2=0x7f5734083ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:28.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:28.306 7f573abf2700 1 --2- 172.21.2.112:0/3904015762 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f57340d18d0 0x7f5734083ac0 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 rx=0x7f572c006b80 tx=0x7f572c009b10).stop 2021-02-10T11:35:28.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:28.307 7f573abf2700 1 -- 172.21.2.112:0/3904015762 shutdown_connections 2021-02-10T11:35:28.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:28.307 7f573abf2700 1 --2- 172.21.2.112:0/3904015762 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f57340d18d0 0x7f5734083ac0 unknown :-1 s=CLOSED pgs=1494 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:28.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:28.307 7f573abf2700 1 -- 172.21.2.112:0/3904015762 >> 172.21.2.112:0/3904015762 conn(0x7f57340ce500 msgr2=0x7f57340e4610 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:28.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:28.307 7f573abf2700 1 -- 172.21.2.112:0/3904015762 shutdown_connections 2021-02-10T11:35:28.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:28.307 7f573abf2700 1 -- 172.21.2.112:0/3904015762 wait complete. 2021-02-10T11:35:28.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:28.321 7f573abf2700 1 Processor -- start 2021-02-10T11:35:28.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:28.321 7f573abf2700 1 -- start start 2021-02-10T11:35:28.311 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:28.322 7f573abf2700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f57341a8630 0x7f573418e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:28.311 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:28.322 7f573abf2700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f57340eadf0 0x7f57340eb1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:28.311 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:28.322 7f573abf2700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f57340eb810 0x7f573418ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:28.311 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:28.322 7f573abf2700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57340b17c0 con 0x7f57341a8630 2021-02-10T11:35:28.311 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:28.322 7f573abf2700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57341a8a30 con 0x7f57340eadf0 2021-02-10T11:35:28.311 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:28.322 7f573abf2700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f573418e7a0 con 0x7f57340eb810 2021-02-10T11:35:28.311 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:28.322 7f5738a0e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f57341a8630 0x7f573418e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:28.311 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:28.322 7f5738a0e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f57341a8630 0x7f573418e010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:51294/0 (socket says 172.21.2.112:51294) 2021-02-10T11:35:28.311 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:28.322 7f5738a0e700 1 -- 172.21.2.112:0/3555493285 learned_addr learned my addr 172.21.2.112:0/3555493285 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:35:28.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:28.322 7f5738a0e700 1 -- 172.21.2.112:0/3555493285 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f57340eb810 msgr2=0x7f573418ed20 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:35:28.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:28.322 7f5738a0e700 1 --2- 172.21.2.112:0/3555493285 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f57340eb810 0x7f573418ed20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:28.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:28.322 7f5738a0e700 1 -- 172.21.2.112:0/3555493285 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f57340eadf0 msgr2=0x7f57340eb1f0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:35:28.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:28.322 7f5738a0e700 1 --2- 172.21.2.112:0/3555493285 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f57340eadf0 0x7f57340eb1f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:28.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:28.322 7f5738a0e700 1 -- 172.21.2.112:0/3555493285 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f573418dd40 con 0x7f57341a8630 2021-02-10T11:35:28.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:28.322 7f5738a0e700 1 --2- 172.21.2.112:0/3555493285 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f57341a8630 0x7f573418e010 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 rx=0x7f571c006ed0 tx=0x7f571c00b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:28.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:28.323 7f57317fa700 1 -- 172.21.2.112:0/3555493285 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f571c006240 con 0x7f57341a8630 2021-02-10T11:35:28.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:28.323 7f57317fa700 1 -- 172.21.2.112:0/3555493285 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f571c0054b0 con 0x7f57341a8630 2021-02-10T11:35:28.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:28.323 7f57317fa700 1 -- 172.21.2.112:0/3555493285 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f571c0059d0 con 0x7f57341a8630 2021-02-10T11:35:28.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:28.323 7f573abf2700 1 -- 172.21.2.112:0/3555493285 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f573418dd40 con 0x7f57341a8630 2021-02-10T11:35:28.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:28.323 7f573abf2700 1 -- 172.21.2.112:0/3555493285 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f573418dd40 con 0x7f57341a8630 2021-02-10T11:35:28.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:28.323 7f57317fa700 1 -- 172.21.2.112:0/3555493285 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f571c006030 con 0x7f57341a8630 2021-02-10T11:35:28.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:28.323 7f57317fa700 1 --2- 172.21.2.112:0/3555493285 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5718018730 0x7f571801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:28.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:28.323 7f57317fa700 1 -- 172.21.2.112:0/3555493285 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f571c0326b0 con 0x7f57341a8630 2021-02-10T11:35:28.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:28.324 7f5733fff700 1 --2- 172.21.2.112:0/3555493285 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5718018730 0x7f571801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:28.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:28.324 7f5733fff700 1 --2- 172.21.2.112:0/3555493285 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5718018730 0x7f571801ac10 crc :-1 s=READY pgs=723 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:28.342 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:28.355 7f573abf2700 1 -- 172.21.2.112:0/3555493285 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57340e7c10 con 0x7f57341a8630 2021-02-10T11:35:28.344 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:28.358 7f57317fa700 1 -- 172.21.2.112:0/3555493285 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f571c000de0 con 0x7f57341a8630 2021-02-10T11:35:28.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:28.485 7f573abf2700 1 -- 172.21.2.112:0/3555493285 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f57340e60c0 con 0x7f57341a8630 2021-02-10T11:35:28.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:28.486 7f57317fa700 1 -- 172.21.2.112:0/3555493285 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) v1 ==== 94+0+2101 (secure 0 0 0) 0x7f571c00bd50 con 0x7f57341a8630 2021-02-10T11:35:28.473 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:35:28.473 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":48,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8334,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/3370618110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":3370618110},{"type":"v1","addr":"172.21.2.112:6833","nonce":3370618110}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":44}],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:34:45.582177","modified":"2021-02-10 11:34:47.589305","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5921},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5921":{"gid":5921,"name":"b","rank":0,"incarnation":45,"state":"up:active","state_seq":3,"addr":"172.21.2.120:6835/2436135065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":2436135065},{"type":"v1","addr":"172.21.2.120:6835","nonce":2436135065}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2021-02-10T11:35:28.473 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 48 2021-02-10T11:35:28.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:28.487 7f573abf2700 1 -- 172.21.2.112:0/3555493285 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5718018730 msgr2=0x7f571801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:28.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:28.487 7f573abf2700 1 --2- 172.21.2.112:0/3555493285 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5718018730 0x7f571801ac10 crc :-1 s=READY pgs=723 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:28.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:28.487 7f573abf2700 1 -- 172.21.2.112:0/3555493285 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f57341a8630 msgr2=0x7f573418e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:28.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:28.487 7f573abf2700 1 --2- 172.21.2.112:0/3555493285 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f57341a8630 0x7f573418e010 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 rx=0x7f571c006ed0 tx=0x7f571c00b960).stop 2021-02-10T11:35:28.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:28.488 7f573abf2700 1 -- 172.21.2.112:0/3555493285 shutdown_connections 2021-02-10T11:35:28.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:28.488 7f573abf2700 1 --2- 172.21.2.112:0/3555493285 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5718018730 0x7f571801ac10 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:28.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:28.488 7f573abf2700 1 --2- 172.21.2.112:0/3555493285 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f57340eadf0 0x7f57340eb1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:28.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:28.488 7f573abf2700 1 --2- 172.21.2.112:0/3555493285 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f57340eb810 0x7f573418ed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:28.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:28.488 7f573abf2700 1 --2- 172.21.2.112:0/3555493285 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f57341a8630 0x7f573418e010 unknown :-1 s=CLOSED pgs=1495 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:28.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:28.488 7f573abf2700 1 -- 172.21.2.112:0/3555493285 >> 172.21.2.112:0/3555493285 conn(0x7f573419d630 msgr2=0x7f57340e8a80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:28.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:28.488 7f573abf2700 1 -- 172.21.2.112:0/3555493285 shutdown_connections 2021-02-10T11:35:28.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:28.488 7f573abf2700 1 -- 172.21.2.112:0/3555493285 wait complete. 2021-02-10T11:35:28.684 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 47, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:34:45.582177', 'modified': '2021-02-10 11:34:47.589305', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5921}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5921': {'gid': 5921, 'name': 'b', 'rank': 0, 'incarnation': 45, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.120:6835/2436135065', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.120:6834', 'nonce': 2436135065}, {'type': 'v1', 'addr': '172.21.2.120:6835', 'nonce': 2436135065}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:35:28.684 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:35:28.684 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:35:29.083 INFO:teuthology.orchestra.run.gibba020.stdout:{ 2021-02-10T11:35:29.083 INFO:teuthology.orchestra.run.gibba020.stdout: "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", 2021-02-10T11:35:29.083 INFO:teuthology.orchestra.run.gibba020.stdout: "whoami": 0, 2021-02-10T11:35:29.083 INFO:teuthology.orchestra.run.gibba020.stdout: "id": 5921, 2021-02-10T11:35:29.083 INFO:teuthology.orchestra.run.gibba020.stdout: "want_state": "up:active", 2021-02-10T11:35:29.083 INFO:teuthology.orchestra.run.gibba020.stdout: "state": "up:active", 2021-02-10T11:35:29.083 INFO:teuthology.orchestra.run.gibba020.stdout: "rank_uptime": 43.509157360000003, 2021-02-10T11:35:29.083 INFO:teuthology.orchestra.run.gibba020.stdout: "mdsmap_epoch": 47, 2021-02-10T11:35:29.083 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch": 64, 2021-02-10T11:35:29.083 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch_barrier": 64, 2021-02-10T11:35:29.084 INFO:teuthology.orchestra.run.gibba020.stdout: "uptime": 49.153033188999999 2021-02-10T11:35:29.084 INFO:teuthology.orchestra.run.gibba020.stdout:} 2021-02-10T11:35:29.087 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", "whoami": 0, "id": 5921, "want_state": "up:active", "state": "up:active", "rank_uptime": 43.509157360000003, "mdsmap_epoch": 47, "osdmap_epoch": 64, "osdmap_epoch_barrier": 64, "uptime": 49.153033188999999 } 2021-02-10T11:35:29.087 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:35:29.087 INFO:tasks.cephfs.fuse_mount:Client client.alice config is {} 2021-02-10T11:35:29.087 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.alice at ubuntu@gibba012.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2021-02-10T11:35:29.088 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:29.088 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:35:29.424 DEBUG:teuthology.orchestra.run.gibba012:> sudo modprobe fuse 2021-02-10T11:35:29.768 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:35:29.769 DEBUG:teuthology.orchestra.run.gibba012:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:35:30.113 INFO:teuthology.orchestra.run.gibba012.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:35:30.113 INFO:teuthology.orchestra.run.gibba012.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:35:30.114 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:35:30.115 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:30.115 DEBUG:teuthology.orchestra.run.gibba012:> ls /sys/fs/fuse/connections 2021-02-10T11:35:30.450 INFO:teuthology.orchestra.run.gibba012.stdout:42 2021-02-10T11:35:30.451 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [42] 2021-02-10T11:35:30.451 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --client_mountpoint=/volumes/group_0000000000485992/subvolume_0000000000513667/7bc8264f-d88f-49e9-b22f-5fa8b916f683 --name client.alice /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:35:30.453 DEBUG:teuthology.orchestra.run.gibba012:> sudo modprobe fuse 2021-02-10T11:35:30.797 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:35:30.797 DEBUG:teuthology.orchestra.run.gibba012:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:35:30.863 INFO:tasks.cephfs.fuse_mount.ceph-fuse.alice.gibba012.stderr:ceph-fuse[137982]: starting ceph client 2021-02-10T11:35:30.863 INFO:tasks.cephfs.fuse_mount.ceph-fuse.alice.gibba012.stderr:2021-02-10 11:35:30.876 7fe8dbf0ff80 -1 init, newargv = 0x55d0e6c81490 newargc=9 2021-02-10T11:35:30.870 INFO:tasks.cephfs.fuse_mount.ceph-fuse.alice.gibba012.stderr:ceph-fuse[137982]: starting fuse 2021-02-10T11:35:31.140 INFO:teuthology.orchestra.run.gibba012.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:35:31.140 INFO:teuthology.orchestra.run.gibba012.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:35:31.141 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:35:31.141 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:31.141 DEBUG:teuthology.orchestra.run.gibba012:> ls /sys/fs/fuse/connections 2021-02-10T11:35:31.475 INFO:teuthology.orchestra.run.gibba012.stdout:42 2021-02-10T11:35:31.475 INFO:teuthology.orchestra.run.gibba012.stdout:43 2021-02-10T11:35:31.475 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42, 43] 2021-02-10T11:35:31.475 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:31.475 DEBUG:teuthology.orchestra.run.gibba012:> sudo python3 -c ' 2021-02-10T11:35:31.476 DEBUG:teuthology.orchestra.run.gibba012:> import glob 2021-02-10T11:35:31.476 DEBUG:teuthology.orchestra.run.gibba012:> import re 2021-02-10T11:35:31.476 DEBUG:teuthology.orchestra.run.gibba012:> import os 2021-02-10T11:35:31.476 DEBUG:teuthology.orchestra.run.gibba012:> import subprocess 2021-02-10T11:35:31.476 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:35:31.476 DEBUG:teuthology.orchestra.run.gibba012:> def find_socket(client_name): 2021-02-10T11:35:31.476 DEBUG:teuthology.orchestra.run.gibba012:> asok_path = "/var/run/ceph/ceph-client.alice.*.asok" 2021-02-10T11:35:31.476 DEBUG:teuthology.orchestra.run.gibba012:> files = glob.glob(asok_path) 2021-02-10T11:35:31.476 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:35:31.476 DEBUG:teuthology.orchestra.run.gibba012:> # Given a non-glob path, it better be there 2021-02-10T11:35:31.476 DEBUG:teuthology.orchestra.run.gibba012:> if "*" not in asok_path: 2021-02-10T11:35:31.476 DEBUG:teuthology.orchestra.run.gibba012:> assert(len(files) == 1) 2021-02-10T11:35:31.476 DEBUG:teuthology.orchestra.run.gibba012:> return files[0] 2021-02-10T11:35:31.477 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:35:31.477 DEBUG:teuthology.orchestra.run.gibba012:> for f in files: 2021-02-10T11:35:31.477 DEBUG:teuthology.orchestra.run.gibba012:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:35:31.477 DEBUG:teuthology.orchestra.run.gibba012:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:35:31.477 DEBUG:teuthology.orchestra.run.gibba012:> return f 2021-02-10T11:35:31.477 DEBUG:teuthology.orchestra.run.gibba012:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:35:31.477 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:35:31.477 DEBUG:teuthology.orchestra.run.gibba012:> print(find_socket("client.alice")) 2021-02-10T11:35:31.477 DEBUG:teuthology.orchestra.run.gibba012:> ' 2021-02-10T11:35:31.838 INFO:teuthology.orchestra.run.gibba012.stdout:/var/run/ceph/ceph-client.alice.137982.asok 2021-02-10T11:35:31.841 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.alice.137982.asok 2021-02-10T11:35:31.841 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:31.841 DEBUG:teuthology.orchestra.run.gibba012:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.alice.137982.asok status 2021-02-10T11:35:32.229 INFO:teuthology.orchestra.run.gibba012.stdout:{ 2021-02-10T11:35:32.230 INFO:teuthology.orchestra.run.gibba012.stdout: "metadata": { 2021-02-10T11:35:32.230 INFO:teuthology.orchestra.run.gibba012.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:35:32.230 INFO:teuthology.orchestra.run.gibba012.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:35:32.230 INFO:teuthology.orchestra.run.gibba012.stdout: "entity_id": "alice", 2021-02-10T11:35:32.230 INFO:teuthology.orchestra.run.gibba012.stdout: "hostname": "gibba012", 2021-02-10T11:35:32.230 INFO:teuthology.orchestra.run.gibba012.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2021-02-10T11:35:32.230 INFO:teuthology.orchestra.run.gibba012.stdout: "pid": "137982", 2021-02-10T11:35:32.230 INFO:teuthology.orchestra.run.gibba012.stdout: "root": "/volumes/group_0000000000485992/subvolume_0000000000513667/7bc8264f-d88f-49e9-b22f-5fa8b916f683" 2021-02-10T11:35:32.230 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:32.230 INFO:teuthology.orchestra.run.gibba012.stdout: "dentry_count": 0, 2021-02-10T11:35:32.231 INFO:teuthology.orchestra.run.gibba012.stdout: "dentry_pinned_count": 0, 2021-02-10T11:35:32.231 INFO:teuthology.orchestra.run.gibba012.stdout: "id": 8700, 2021-02-10T11:35:32.231 INFO:teuthology.orchestra.run.gibba012.stdout: "inst": { 2021-02-10T11:35:32.231 INFO:teuthology.orchestra.run.gibba012.stdout: "name": { 2021-02-10T11:35:32.231 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "client", 2021-02-10T11:35:32.231 INFO:teuthology.orchestra.run.gibba012.stdout: "num": 8700 2021-02-10T11:35:32.231 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:32.231 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": { 2021-02-10T11:35:32.231 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:35:32.231 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:0", 2021-02-10T11:35:32.231 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 3957479471 2021-02-10T11:35:32.231 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:35:32.231 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:32.232 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": { 2021-02-10T11:35:32.232 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:35:32.232 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:0", 2021-02-10T11:35:32.232 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 3957479471 2021-02-10T11:35:32.232 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:32.232 INFO:teuthology.orchestra.run.gibba012.stdout: "inst_str": "client.8700 172.21.2.112:0/3957479471", 2021-02-10T11:35:32.232 INFO:teuthology.orchestra.run.gibba012.stdout: "addr_str": "172.21.2.112:0/3957479471", 2021-02-10T11:35:32.232 INFO:teuthology.orchestra.run.gibba012.stdout: "inode_count": 1, 2021-02-10T11:35:32.232 INFO:teuthology.orchestra.run.gibba012.stdout: "mds_epoch": 47, 2021-02-10T11:35:32.232 INFO:teuthology.orchestra.run.gibba012.stdout: "osd_epoch": 64, 2021-02-10T11:35:32.232 INFO:teuthology.orchestra.run.gibba012.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:35:32.232 INFO:teuthology.orchestra.run.gibba012.stdout: "blacklisted": false 2021-02-10T11:35:32.232 INFO:teuthology.orchestra.run.gibba012.stdout:} 2021-02-10T11:35:32.234 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.1 && sudo ls /home/ubuntu/cephtest/mnt.1 2021-02-10T11:35:32.580 INFO:teuthology.orchestra.run.gibba012.stdout:data.bin 2021-02-10T11:35:32.581 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:32.581 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:35:32.581 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:35:32.917 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:35:32.918 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:35:32.918 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.1 && sudo dd if=/dev/urandom of=rogue.bin bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:35:33.262 INFO:teuthology.orchestra.run.gibba012.stderr:dd: failed to open ‘rogue.bin’: Permission denied 2021-02-10T11:35:33.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:35:33.263 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:33.264 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:35:33.264 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:35:33.599 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:35:33.600 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:35:33.600 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba012.front.sepia.ceph.com... 2021-02-10T11:35:33.600 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:35:33.600 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:35:33.953 INFO:tasks.cephfs.fuse_mount.ceph-fuse.alice.gibba012.stderr:ceph-fuse[137982]: fuse finished with error 0 and tester_r 0 2021-02-10T11:35:33.954 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:33.954 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:35:33.954 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:35:34.289 INFO:teuthology.orchestra.run.gibba012.stdout:ext2/ext3 2021-02-10T11:35:34.289 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:35:34.289 INFO:teuthology.orchestra.run:waiting for 900 2021-02-10T11:35:34.290 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:35:34.290 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:35:34.625 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume deauthorize cephfs subvolume_0000000000513667 alice --group_name group_0000000000485992 2021-02-10T11:35:35.055 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.053 7faab96d3700 1 Processor -- start 2021-02-10T11:35:35.056 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.053 7faab96d3700 1 -- start start 2021-02-10T11:35:35.056 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.053 7faab96d3700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7faab40e51a0 0x7faab41b3910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:35.056 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.053 7faab96d3700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7faab41b3f30 0x7faab4077510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:35.056 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.053 7faab96d3700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faab4078b90 con 0x7faab40e55a0 2021-02-10T11:35:35.056 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.053 7faab96d3700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faab407a120 con 0x7faab41b3f30 2021-02-10T11:35:35.056 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.053 7faab96d3700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faab407a2d0 con 0x7faab40e51a0 2021-02-10T11:35:35.056 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.053 7faab2abe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7faab41b3f30 0x7faab4077510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:35.056 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.053 7faab2abe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7faab41b3f30 0x7faab4077510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:51310/0 (socket says 172.21.2.112:51310) 2021-02-10T11:35:35.056 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.053 7faab2abe700 1 -- 172.21.2.112:0/379154798 learned_addr learned my addr 172.21.2.112:0/379154798 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:35:35.056 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.054 7faab2abe700 1 -- 172.21.2.112:0/379154798 >> v2:172.21.2.120:3300/0 conn(0x7faab40e51a0 msgr2=0x7faab41b3910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:35.056 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.054 7faab2abe700 1 --2- 172.21.2.112:0/379154798 >> v2:172.21.2.120:3300/0 conn(0x7faab40e51a0 0x7faab41b3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:35.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.054 7faab2abe700 1 -- 172.21.2.112:0/379154798 >> v1:172.21.2.120:6789/0 conn(0x7faab40e55a0 legacy=0x7faab4077b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:35.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.054 7faab2abe700 1 -- 172.21.2.112:0/379154798 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faab407a4f0 con 0x7faab41b3f30 2021-02-10T11:35:35.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.054 7faab2abe700 1 --2- 172.21.2.112:0/379154798 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faab41b3f30 0x7faab4077510 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 rx=0x7faaa8006700 tx=0x7faaa8009b80).ready entity=mon.0 client_cookie=adf0aefaf11fc52a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:35.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.054 7faab22bd700 1 -- 172.21.2.112:0/379154798 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faaa800c0c0 con 0x7faab41b3f30 2021-02-10T11:35:35.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.054 7faab22bd700 1 -- 172.21.2.112:0/379154798 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faaa800c440 con 0x7faab41b3f30 2021-02-10T11:35:35.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.054 7faab22bd700 1 -- 172.21.2.112:0/379154798 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faaa800c960 con 0x7faab41b3f30 2021-02-10T11:35:35.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.054 7faab96d3700 1 -- 172.21.2.112:0/379154798 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faab41b3f30 msgr2=0x7faab4077510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:35.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.054 7faab96d3700 1 --2- 172.21.2.112:0/379154798 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faab41b3f30 0x7faab4077510 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 rx=0x7faaa8006700 tx=0x7faaa8009b80).stop 2021-02-10T11:35:35.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.055 7faab96d3700 1 -- 172.21.2.112:0/379154798 shutdown_connections 2021-02-10T11:35:35.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.055 7faab96d3700 1 --2- 172.21.2.112:0/379154798 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faab41b3f30 0x7faab4077510 unknown :-1 s=CLOSED pgs=1498 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:35.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.055 7faab96d3700 1 --2- 172.21.2.112:0/379154798 >> v2:172.21.2.120:3300/0 conn(0x7faab40e51a0 0x7faab41b3910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:35.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.055 7faab96d3700 1 -- 172.21.2.112:0/379154798 >> 172.21.2.112:0/379154798 conn(0x7faab40ce850 msgr2=0x7faab40e4690 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:35.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.055 7faab96d3700 1 -- 172.21.2.112:0/379154798 shutdown_connections 2021-02-10T11:35:35.058 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.055 7faab96d3700 1 -- 172.21.2.112:0/379154798 wait complete. 2021-02-10T11:35:35.059 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.069 7faab96d3700 1 Processor -- start 2021-02-10T11:35:35.059 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.070 7faab96d3700 1 -- start start 2021-02-10T11:35:35.059 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.070 7faab96d3700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7faab4091ec0 0x7faab41991d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:35.059 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.070 7faab96d3700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faab4097fa0 0x7faab40983a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:35.059 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.070 7faab96d3700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7faab40b1c60 0x7faab40b2060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:35.059 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.070 7faab96d3700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faab4098d60 con 0x7faab4097fa0 2021-02-10T11:35:35.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.070 7faab96d3700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faab40989c0 con 0x7faab40b1c60 2021-02-10T11:35:35.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.070 7faab96d3700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faab41998c0 con 0x7faab4091ec0 2021-02-10T11:35:35.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.070 7faab2abe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faab4097fa0 0x7faab40983a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:35.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.070 7faab2abe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faab4097fa0 0x7faab40983a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:51316/0 (socket says 172.21.2.112:51316) 2021-02-10T11:35:35.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.070 7faab2abe700 1 -- 172.21.2.112:0/2530839335 learned_addr learned my addr 172.21.2.112:0/2530839335 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:35:35.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.070 7faab2abe700 1 -- 172.21.2.112:0/2530839335 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7faab4091ec0 msgr2=0x7faab41991d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:35:35.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.070 7faab2abe700 1 --2- 172.21.2.112:0/2530839335 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7faab4091ec0 0x7faab41991d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:35.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.070 7faab2abe700 1 -- 172.21.2.112:0/2530839335 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7faab40b1c60 msgr2=0x7faab40b2060 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:35:35.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.070 7faab2abe700 1 --2- 172.21.2.112:0/2530839335 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7faab40b1c60 0x7faab40b2060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:35.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.070 7faab2abe700 1 -- 172.21.2.112:0/2530839335 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faab40b2680 con 0x7faab4097fa0 2021-02-10T11:35:35.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.071 7faab2abe700 1 --2- 172.21.2.112:0/2530839335 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faab4097fa0 0x7faab40983a0 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 rx=0x7faaa800b5e0 tx=0x7faaa800b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:35.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.071 7faa9bfff700 1 -- 172.21.2.112:0/2530839335 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faaa800bd70 con 0x7faab4097fa0 2021-02-10T11:35:35.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.071 7faa9bfff700 1 -- 172.21.2.112:0/2530839335 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faaa8002030 con 0x7faab4097fa0 2021-02-10T11:35:35.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.071 7faa9bfff700 1 -- 172.21.2.112:0/2530839335 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faaa8002550 con 0x7faab4097fa0 2021-02-10T11:35:35.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.071 7faab96d3700 1 -- 172.21.2.112:0/2530839335 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faab40b2680 con 0x7faab4097fa0 2021-02-10T11:35:35.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.071 7faab96d3700 1 -- 172.21.2.112:0/2530839335 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faab40b2680 con 0x7faab4097fa0 2021-02-10T11:35:35.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.071 7faa9bfff700 1 -- 172.21.2.112:0/2530839335 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7faaa8002210 con 0x7faab4097fa0 2021-02-10T11:35:35.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.072 7faa9bfff700 1 --2- 172.21.2.112:0/2530839335 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7faa940186c0 0x7faa9401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:35.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.072 7faa9bfff700 1 -- 172.21.2.112:0/2530839335 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5573+0+0 (secure 0 0 0) 0x7faaa80325c0 con 0x7faab4097fa0 2021-02-10T11:35:35.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.072 7faab32bf700 1 --2- 172.21.2.112:0/2530839335 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7faa940186c0 0x7faa9401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:35.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.073 7faab32bf700 1 --2- 172.21.2.112:0/2530839335 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7faa940186c0 0x7faa9401abc0 crc :-1 s=READY pgs=724 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:35.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.103 7faab96d3700 1 -- 172.21.2.112:0/2530839335 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faab40989c0 con 0x7faab4097fa0 2021-02-10T11:35:35.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.106 7faa9bfff700 1 -- 172.21.2.112:0/2530839335 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7faaa8030df0 con 0x7faab4097fa0 2021-02-10T11:35:35.218 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.232 7faab96d3700 1 -- 172.21.2.112:0/2530839335 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume deauthorize", "vol_name": "cephfs", "sub_name": "subvolume_0000000000513667", "target": ["mgr", ""], "auth_id": "alice", "group_name": "group_0000000000485992"}) v1 -- 0x7faab40989c0 con 0x7faa940186c0 2021-02-10T11:35:35.257 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.271 7faa9bfff700 1 -- 172.21.2.112:0/2530839335 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7faaa40009b0 con 0x7faa940186c0 2021-02-10T11:35:35.284 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.297 7faab96d3700 1 -- 172.21.2.112:0/2530839335 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7faa940186c0 msgr2=0x7faa9401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:35.284 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.297 7faab96d3700 1 --2- 172.21.2.112:0/2530839335 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7faa940186c0 0x7faa9401abc0 crc :-1 s=READY pgs=724 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:35.284 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.297 7faab96d3700 1 -- 172.21.2.112:0/2530839335 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faab4097fa0 msgr2=0x7faab40983a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:35.284 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.297 7faab96d3700 1 --2- 172.21.2.112:0/2530839335 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faab4097fa0 0x7faab40983a0 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 rx=0x7faaa800b5e0 tx=0x7faaa800b980).stop 2021-02-10T11:35:35.284 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.297 7faab96d3700 1 -- 172.21.2.112:0/2530839335 shutdown_connections 2021-02-10T11:35:35.284 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.297 7faab96d3700 1 --2- 172.21.2.112:0/2530839335 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7faa940186c0 0x7faa9401abc0 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:35.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.297 7faab96d3700 1 --2- 172.21.2.112:0/2530839335 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7faab40b1c60 0x7faab40b2060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:35.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.297 7faab96d3700 1 --2- 172.21.2.112:0/2530839335 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faab4097fa0 0x7faab40983a0 unknown :-1 s=CLOSED pgs=1499 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:35.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.297 7faab96d3700 1 --2- 172.21.2.112:0/2530839335 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7faab4091ec0 0x7faab41991d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:35.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.297 7faab96d3700 1 -- 172.21.2.112:0/2530839335 >> 172.21.2.112:0/2530839335 conn(0x7faab41acfe0 msgr2=0x7faab419a100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:35.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.297 7faab96d3700 1 -- 172.21.2.112:0/2530839335 shutdown_connections 2021-02-10T11:35:35.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.297 7faab96d3700 1 -- 172.21.2.112:0/2530839335 wait complete. 2021-02-10T11:35:35.292 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth ls --format=json-pretty 2021-02-10T11:35:35.718 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.715 7fba3e5d6700 1 Processor -- start 2021-02-10T11:35:35.718 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.715 7fba3e5d6700 1 -- start start 2021-02-10T11:35:35.718 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.716 7fba3e5d6700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fba380e53e0 0x7fba381b0550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:35.718 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.716 7fba3e5d6700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba38082d50 con 0x7fba380842e0 2021-02-10T11:35:35.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.716 7fba3e5d6700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba380e57e0 con 0x7fba381b0b70 2021-02-10T11:35:35.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.716 7fba3e5d6700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba380e5fe0 con 0x7fba380e53e0 2021-02-10T11:35:35.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.716 7fba3cbf3700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fba380842e0 0x7fba38082100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:50786/0 (socket says 172.21.2.112:50786) 2021-02-10T11:35:35.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.716 7fba3cbf3700 1 -- 172.21.2.112:0/2093089421 learned_addr learned my addr 172.21.2.112:0/2093089421 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:35:35.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.716 7fba37fff700 1 --2- 172.21.2.112:0/2093089421 >> v2:172.21.2.120:3300/0 conn(0x7fba380e53e0 0x7fba381b0550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:35.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.716 7fba37fff700 1 -- 172.21.2.112:0/2093089421 >> v1:172.21.2.120:6789/0 conn(0x7fba381b0b70 legacy=0x7fba38081910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:35.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.716 7fba37fff700 1 -- 172.21.2.112:0/2093089421 >> v1:172.21.2.112:6789/0 conn(0x7fba380842e0 legacy=0x7fba38082100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:35.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.716 7fba36ffd700 1 -- 172.21.2.112:0/2093089421 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3896008923 0 0) 0x7fba2c000ff0 con 0x7fba381b0b70 2021-02-10T11:35:35.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.716 7fba37fff700 1 -- 172.21.2.112:0/2093089421 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba380846e0 con 0x7fba380e53e0 2021-02-10T11:35:35.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.717 7fba37fff700 1 --2- 172.21.2.112:0/2093089421 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fba380e53e0 0x7fba381b0550 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 rx=0x7fba28006650 tx=0x7fba28009a80).ready entity=mon.1 client_cookie=109c700eda906407 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:35.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.717 7fba36ffd700 1 -- 172.21.2.112:0/2093089421 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fba2800c0c0 con 0x7fba380e53e0 2021-02-10T11:35:35.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.717 7fba36ffd700 1 -- 172.21.2.112:0/2093089421 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba2800c340 con 0x7fba380e53e0 2021-02-10T11:35:35.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.717 7fba36ffd700 1 -- 172.21.2.112:0/2093089421 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fba2800c860 con 0x7fba380e53e0 2021-02-10T11:35:35.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.717 7fba3e5d6700 1 -- 172.21.2.112:0/2093089421 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fba380e53e0 msgr2=0x7fba381b0550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:35.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.717 7fba3e5d6700 1 --2- 172.21.2.112:0/2093089421 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fba380e53e0 0x7fba381b0550 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 rx=0x7fba28006650 tx=0x7fba28009a80).stop 2021-02-10T11:35:35.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.717 7fba3e5d6700 1 -- 172.21.2.112:0/2093089421 shutdown_connections 2021-02-10T11:35:35.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.717 7fba3e5d6700 1 --2- 172.21.2.112:0/2093089421 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fba380e53e0 0x7fba381b0550 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:35.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.717 7fba3e5d6700 1 -- 172.21.2.112:0/2093089421 >> 172.21.2.112:0/2093089421 conn(0x7fba380ce500 msgr2=0x7fba380e4610 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:35.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.717 7fba3e5d6700 1 -- 172.21.2.112:0/2093089421 shutdown_connections 2021-02-10T11:35:35.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.717 7fba3e5d6700 1 -- 172.21.2.112:0/2093089421 wait complete. 2021-02-10T11:35:35.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.732 7fba3e5d6700 1 Processor -- start 2021-02-10T11:35:35.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.732 7fba3e5d6700 1 -- start start 2021-02-10T11:35:35.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.732 7fba3e5d6700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fba380b7ff0 0x7fba380b83f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:35.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.732 7fba3e5d6700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fba381989c0 0x7fba38198dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:35.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.732 7fba3e5d6700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fba38190c50 0x7fba38191050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:35.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.732 7fba3e5d6700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba3809fe70 con 0x7fba381989c0 2021-02-10T11:35:35.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.732 7fba3e5d6700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba3818ea50 con 0x7fba380b7ff0 2021-02-10T11:35:35.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.732 7fba3e5d6700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba3818ec70 con 0x7fba38190c50 2021-02-10T11:35:35.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.732 7fba377fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fba381989c0 0x7fba38198dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:35.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.733 7fba37fff700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fba380b7ff0 0x7fba380b83f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:35.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.733 7fba377fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fba381989c0 0x7fba38198dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:51334/0 (socket says 172.21.2.112:51334) 2021-02-10T11:35:35.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.733 7fba377fe700 1 -- 172.21.2.112:0/152266431 learned_addr learned my addr 172.21.2.112:0/152266431 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:35:35.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.733 7fba3cbf3700 1 --2- 172.21.2.112:0/152266431 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fba38190c50 0x7fba38191050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:35.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.733 7fba3cbf3700 1 -- 172.21.2.112:0/152266431 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fba380b7ff0 msgr2=0x7fba380b83f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:35.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.733 7fba3cbf3700 1 --2- 172.21.2.112:0/152266431 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fba380b7ff0 0x7fba380b83f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:35.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.733 7fba3cbf3700 1 -- 172.21.2.112:0/152266431 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fba381989c0 msgr2=0x7fba38198dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:35.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.733 7fba3cbf3700 1 --2- 172.21.2.112:0/152266431 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fba381989c0 0x7fba38198dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:35.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.733 7fba3cbf3700 1 -- 172.21.2.112:0/152266431 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba3818ee90 con 0x7fba38190c50 2021-02-10T11:35:35.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.733 7fba377fe700 1 --2- 172.21.2.112:0/152266431 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fba381989c0 0x7fba38198dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:35:35.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.733 7fba3cbf3700 1 --2- 172.21.2.112:0/152266431 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fba38190c50 0x7fba38191050 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7fba30001a60 tx=0x7fba3000abc0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:35.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.733 7fba34ff9700 1 -- 172.21.2.112:0/152266431 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fba30007280 con 0x7fba38190c50 2021-02-10T11:35:35.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.733 7fba34ff9700 1 -- 172.21.2.112:0/152266431 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba30001500 con 0x7fba38190c50 2021-02-10T11:35:35.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.733 7fba3e5d6700 1 -- 172.21.2.112:0/152266431 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba3818ee90 con 0x7fba38190c50 2021-02-10T11:35:35.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.733 7fba34ff9700 1 -- 172.21.2.112:0/152266431 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fba30007280 con 0x7fba38190c50 2021-02-10T11:35:35.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.733 7fba3e5d6700 1 -- 172.21.2.112:0/152266431 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba3818ee90 con 0x7fba38190c50 2021-02-10T11:35:35.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.734 7fba34ff9700 1 -- 172.21.2.112:0/152266431 <== mon.2 v2:172.21.2.120:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fba30007540 con 0x7fba38190c50 2021-02-10T11:35:35.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.734 7fba34ff9700 1 --2- 172.21.2.112:0/152266431 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fba18018730 0x7fba1801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:35.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.734 7fba34ff9700 1 -- 172.21.2.112:0/152266431 <== mon.2 v2:172.21.2.120:3301/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fba30032a90 con 0x7fba38190c50 2021-02-10T11:35:35.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.734 7fba37fff700 1 --2- 172.21.2.112:0/152266431 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fba18018730 0x7fba1801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:35.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.735 7fba37fff700 1 --2- 172.21.2.112:0/152266431 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fba18018730 0x7fba1801ac10 crc :-1 s=READY pgs=725 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:35.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.766 7fba3e5d6700 1 -- 172.21.2.112:0/152266431 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba38191670 con 0x7fba38190c50 2021-02-10T11:35:35.757 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.769 7fba34ff9700 1 -- 172.21.2.112:0/152266431 <== mon.2 v2:172.21.2.120:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fba30007070 con 0x7fba38190c50 2021-02-10T11:35:35.889 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.902 7fba3e5d6700 1 -- 172.21.2.112:0/152266431 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fba38191670 con 0x7fba38190c50 2021-02-10T11:35:35.890 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.904 7fba34ff9700 1 -- 172.21.2.112:0/152266431 <== mon.2 v2:172.21.2.120:3301/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v44) v1 ==== 80+0+4513 (secure 0 0 0) 0x7fba30001500 con 0x7fba38190c50 2021-02-10T11:35:35.892 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:35:35.892 INFO:teuthology.orchestra.run.gibba012.stdout:{ 2021-02-10T11:35:35.892 INFO:teuthology.orchestra.run.gibba012.stdout: "auth_dump": [ 2021-02-10T11:35:35.892 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:35.892 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "mds.a", 2021-02-10T11:35:35.892 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCKvyNgPkV4AxAAEmVNaghdccP6lWd8hvQnxw==", 2021-02-10T11:35:35.892 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:35:35.893 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:35:35.893 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:35:35.893 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:35:35.893 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:35:35.893 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:35:35.893 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:35.893 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:35.893 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "mds.b", 2021-02-10T11:35:35.893 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCKvyNguxONLhAAHIpry+R89MAwdhFJyEhVCw==", 2021-02-10T11:35:35.893 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:35:35.893 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:35:35.893 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:35:35.893 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:35:35.894 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:35:35.894 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:35:35.894 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:35.894 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:35.894 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.0", 2021-02-10T11:35:35.894 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCSvyNgXcKPLhAAP/Bj2F+06DCtPCzQDNq+cQ==", 2021-02-10T11:35:35.894 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:35:35.894 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:35:35.894 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:35:35.894 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:35:35.894 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:35:35.894 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:35.894 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:35.895 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.1", 2021-02-10T11:35:35.895 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCVvyNghfbcLhAA66PrRwR2M7o6VTF5UhU/Fg==", 2021-02-10T11:35:35.895 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:35:35.895 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:35:35.895 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:35:35.895 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:35:35.895 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:35:35.895 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:35.895 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:35.895 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.2", 2021-02-10T11:35:35.895 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCYvyNgwiPqLhAA66BVkRsgaUlGpAsQ15Nf8Q==", 2021-02-10T11:35:35.895 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:35:35.895 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:35:35.895 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:35:35.896 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:35:35.896 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:35:35.896 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:35.896 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:35.896 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.3", 2021-02-10T11:35:35.896 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCbvyNgqBZrLxAAERPe/1L3SXAEzI3d9ReHHg==", 2021-02-10T11:35:35.896 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:35:35.896 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:35:35.896 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:35:35.896 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:35:35.896 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:35:35.896 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:35.896 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:35.897 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.4", 2021-02-10T11:35:35.897 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCkvyNgDQsxIhAAHmVjVYnx20N4SDmKqvbcuA==", 2021-02-10T11:35:35.897 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:35:35.897 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:35:35.897 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:35:35.897 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:35:35.897 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:35:35.897 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:35.897 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:35.897 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.5", 2021-02-10T11:35:35.897 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCnvyNg8ITFIhAADd4aKybAV7btoGPx3kCg3A==", 2021-02-10T11:35:35.897 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:35:35.897 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:35:35.897 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:35:35.898 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:35:35.898 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:35:35.898 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:35.898 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:35.898 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.6", 2021-02-10T11:35:35.898 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCqvyNgxOhDIxAAzXixEa/nlDc1xXrg0aeLwg==", 2021-02-10T11:35:35.898 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:35:35.898 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:35:35.898 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:35:35.898 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:35:35.898 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:35:35.898 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:35.899 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:35.899 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.7", 2021-02-10T11:35:35.899 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCtvyNgxSqGIxAACxJQlwP7DpiVVCZHA3zn1A==", 2021-02-10T11:35:35.899 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:35:35.899 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:35:35.899 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:35:35.899 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:35:35.899 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:35:35.899 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:35.899 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:35.899 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.0", 2021-02-10T11:35:35.899 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCLvyNgDkNiHRAAygU8Za4PBusTqCgwAYWTiQ==", 2021-02-10T11:35:35.899 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:35:35.900 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:35:35.900 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow r", 2021-02-10T11:35:35.900 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:35:35.900 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:35:35.900 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:35.900 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:35.900 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.1", 2021-02-10T11:35:35.900 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCLvyNgxmLqMxAA1MlSDJ9LH9u4f2sL7mm8/Q==", 2021-02-10T11:35:35.900 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:35:35.900 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:35:35.900 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow r", 2021-02-10T11:35:35.900 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:35:35.900 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:35:35.901 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:35.901 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:35.901 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.2", 2021-02-10T11:35:35.901 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCMvyNgp8veDhAAniSVHYPtwwTog/pjbaRQmw==", 2021-02-10T11:35:35.901 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:35:35.901 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:35:35.901 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow r", 2021-02-10T11:35:35.902 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:35:35.902 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:35:35.902 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:35.902 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:35.902 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.3", 2021-02-10T11:35:35.902 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCMvyNgQEVoJRAApHcWdm1sldIxu+heyD4HdQ==", 2021-02-10T11:35:35.902 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:35:35.902 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:35:35.902 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow r", 2021-02-10T11:35:35.902 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:35:35.902 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:35:35.902 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:35.902 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:35.903 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.admin", 2021-02-10T11:35:35.903 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCGvyNg7jjJBRAAtot+ypUZOiTo1GbY+oVhVw==", 2021-02-10T11:35:35.903 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:35:35.903 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow *", 2021-02-10T11:35:35.903 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:35:35.903 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:35:35.903 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:35:35.903 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:35:35.903 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:35.903 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:35.903 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "mgr.x", 2021-02-10T11:35:35.903 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCJvyNgTPVwKBAAjduiBE8+RSXBv82tdPecWg==", 2021-02-10T11:35:35.903 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:35:35.904 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow *", 2021-02-10T11:35:35.904 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow profile mgr", 2021-02-10T11:35:35.904 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:35:35.904 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:35:35.904 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:35.904 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:35.904 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "mgr.y", 2021-02-10T11:35:35.904 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCJvyNgQ1tTDhAASwBssDy+Rz31ujw8fNlWow==", 2021-02-10T11:35:35.904 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:35:35.904 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow *", 2021-02-10T11:35:35.904 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow profile mgr", 2021-02-10T11:35:35.904 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:35:35.904 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:35:35.905 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:35:35.905 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:35:35.905 INFO:teuthology.orchestra.run.gibba012.stdout:} 2021-02-10T11:35:35.906 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.906 7fba3e5d6700 1 -- 172.21.2.112:0/152266431 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fba18018730 msgr2=0x7fba1801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:35.906 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.906 7fba3e5d6700 1 --2- 172.21.2.112:0/152266431 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fba18018730 0x7fba1801ac10 crc :-1 s=READY pgs=725 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:35.906 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.906 7fba3e5d6700 1 -- 172.21.2.112:0/152266431 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fba38190c50 msgr2=0x7fba38191050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:35.906 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.906 7fba3e5d6700 1 --2- 172.21.2.112:0/152266431 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fba38190c50 0x7fba38191050 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7fba30001a60 tx=0x7fba3000abc0).stop 2021-02-10T11:35:35.906 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.907 7fba3e5d6700 1 -- 172.21.2.112:0/152266431 shutdown_connections 2021-02-10T11:35:35.906 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.907 7fba3e5d6700 1 --2- 172.21.2.112:0/152266431 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fba18018730 0x7fba1801ac10 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:35.906 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.907 7fba3e5d6700 1 --2- 172.21.2.112:0/152266431 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fba38190c50 0x7fba38191050 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:35.907 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.907 7fba3e5d6700 1 --2- 172.21.2.112:0/152266431 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fba381989c0 0x7fba38198dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:35.907 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.907 7fba3e5d6700 1 --2- 172.21.2.112:0/152266431 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fba380b7ff0 0x7fba380b83f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:35.907 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.907 7fba3e5d6700 1 -- 172.21.2.112:0/152266431 >> 172.21.2.112:0/152266431 conn(0x7fba3809dbb0 msgr2=0x7fba381b2700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:35.907 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.907 7fba3e5d6700 1 -- 172.21.2.112:0/152266431 shutdown_connections 2021-02-10T11:35:35.907 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:35.907 7fba3e5d6700 1 -- 172.21.2.112:0/152266431 wait complete. 2021-02-10T11:35:35.907 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000513667 --group_name group_0000000000485992 2021-02-10T11:35:36.339 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.335 7fe729314700 1 Processor -- start 2021-02-10T11:35:36.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.335 7fe729314700 1 -- start start 2021-02-10T11:35:36.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.335 7fe729314700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe724078520 0x7fe7240d1910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:36.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.335 7fe729314700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe724078920 con 0x7fe7240e5580 2021-02-10T11:35:36.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.335 7fe729314700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe724078b90 con 0x7fe7240e5180 2021-02-10T11:35:36.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.335 7fe729314700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7241b4150 con 0x7fe724078520 2021-02-10T11:35:36.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.335 7fe72287c700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fe7240e5580 0x7fe724079110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:50800/0 (socket says 172.21.2.112:50800) 2021-02-10T11:35:36.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.335 7fe72287c700 1 -- 172.21.2.112:0/3756661451 learned_addr learned my addr 172.21.2.112:0/3756661451 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:35:36.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.336 7fe72207b700 1 -- 172.21.2.112:0/3756661451 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3557370608 0 0) 0x7fe714001180 con 0x7fe7240e5580 2021-02-10T11:35:36.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.336 7fe72387e700 1 --2- 172.21.2.112:0/3756661451 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe724078520 0x7fe7240d1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:36.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.336 7fe72207b700 1 -- 172.21.2.112:0/3756661451 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe710002fe0 con 0x7fe7240e5580 2021-02-10T11:35:36.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.336 7fe72207b700 1 -- 172.21.2.112:0/3756661451 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1737795118 0 0) 0x7fe714001180 con 0x7fe7240e5580 2021-02-10T11:35:36.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.336 7fe72207b700 1 -- 172.21.2.112:0/3756661451 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe724078520 msgr2=0x7fe7240d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:36.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.336 7fe72207b700 1 --2- 172.21.2.112:0/3756661451 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe724078520 0x7fe7240d1910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:36.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.336 7fe72207b700 1 -- 172.21.2.112:0/3756661451 >> v1:172.21.2.120:6789/0 conn(0x7fe7240e5180 legacy=0x7fe7241b38f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:36.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.336 7fe72207b700 1 -- 172.21.2.112:0/3756661451 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7240d1f30 con 0x7fe7240e5580 2021-02-10T11:35:36.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.336 7fe72207b700 1 -- 172.21.2.112:0/3756661451 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7fe714001680 con 0x7fe7240e5580 2021-02-10T11:35:36.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.336 7fe72207b700 1 -- 172.21.2.112:0/3756661451 >> v1:172.21.2.112:6789/0 conn(0x7fe7240e5580 legacy=0x7fe724079110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:36.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.336 7fe72207b700 1 --2- 172.21.2.112:0/3756661451 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe710003aa0 0x7fe710005eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:36.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.336 7fe72207b700 1 --2- 172.21.2.112:0/3756661451 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe710006750 0x7fe710008c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:36.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.336 7fe72207b700 1 --2- 172.21.2.112:0/3756661451 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe7100094f0 0x7fe71000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:36.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.336 7fe72207b700 1 -- 172.21.2.112:0/3756661451 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe71000c2c0 con 0x7fe7100094f0 2021-02-10T11:35:36.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.336 7fe72207b700 1 -- 172.21.2.112:0/3756661451 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe71000c4e0 con 0x7fe710006750 2021-02-10T11:35:36.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.336 7fe72207b700 1 -- 172.21.2.112:0/3756661451 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe710050710 con 0x7fe710003aa0 2021-02-10T11:35:36.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.336 7fe72387e700 1 --2- 172.21.2.112:0/3756661451 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe7100094f0 0x7fe71000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:36.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.336 7fe72287c700 1 --2- 172.21.2.112:0/3756661451 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe710006750 0x7fe710008c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:36.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.336 7fe72307d700 1 --2- 172.21.2.112:0/3756661451 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe710003aa0 0x7fe710005eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:36.342 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.337 7fe72387e700 1 -- 172.21.2.112:0/3756661451 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe710003aa0 msgr2=0x7fe710005eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:36.342 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.337 7fe72387e700 1 --2- 172.21.2.112:0/3756661451 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe710003aa0 0x7fe710005eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:36.342 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.337 7fe72387e700 1 -- 172.21.2.112:0/3756661451 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe710006750 msgr2=0x7fe710008c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:36.342 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.337 7fe72387e700 1 --2- 172.21.2.112:0/3756661451 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe710006750 0x7fe710008c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:36.342 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.337 7fe72387e700 1 -- 172.21.2.112:0/3756661451 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe710050940 con 0x7fe7100094f0 2021-02-10T11:35:36.342 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.337 7fe72387e700 1 --2- 172.21.2.112:0/3756661451 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe7100094f0 0x7fe71000ba10 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 rx=0x7fe71c009a40 tx=0x7fe71c009de0).ready entity=mon.0 client_cookie=ee652a2eadd7c39c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:36.342 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.337 7fe72207b700 1 -- 172.21.2.112:0/3756661451 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe71c006a90 con 0x7fe7100094f0 2021-02-10T11:35:36.342 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.337 7fe72207b700 1 -- 172.21.2.112:0/3756661451 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe71c003030 con 0x7fe7100094f0 2021-02-10T11:35:36.342 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.337 7fe72207b700 1 -- 172.21.2.112:0/3756661451 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe71c00b790 con 0x7fe7100094f0 2021-02-10T11:35:36.342 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.337 7fe729314700 1 -- 172.21.2.112:0/3756661451 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe7100094f0 msgr2=0x7fe71000ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:36.342 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.337 7fe729314700 1 --2- 172.21.2.112:0/3756661451 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe7100094f0 0x7fe71000ba10 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 rx=0x7fe71c009a40 tx=0x7fe71c009de0).stop 2021-02-10T11:35:36.342 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.337 7fe72387e700 1 -- 172.21.2.112:0/3756661451 reap_dead start 2021-02-10T11:35:36.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.338 7fe729314700 1 -- 172.21.2.112:0/3756661451 shutdown_connections 2021-02-10T11:35:36.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.338 7fe729314700 1 -- 172.21.2.112:0/3756661451 >> 172.21.2.112:0/3756661451 conn(0x7fe7240ce830 msgr2=0x7fe7240e4670 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:36.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.338 7fe729314700 1 -- 172.21.2.112:0/3756661451 shutdown_connections 2021-02-10T11:35:36.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.338 7fe729314700 1 -- 172.21.2.112:0/3756661451 wait complete. 2021-02-10T11:35:36.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.352 7fe729314700 1 Processor -- start 2021-02-10T11:35:36.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.353 7fe729314700 1 -- start start 2021-02-10T11:35:36.344 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.353 7fe729314700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe7240b6b10 0x7fe7241a2910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:36.344 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.353 7fe729314700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe7241a2f30 0x7fe7241a3330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:36.344 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.353 7fe729314700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe7240a3ec0 0x7fe7240a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:36.345 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.353 7fe729314700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7240a48e0 con 0x7fe7240b6b10 2021-02-10T11:35:36.345 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.353 7fe729314700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7240a4b00 con 0x7fe7240a3ec0 2021-02-10T11:35:36.345 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.353 7fe729314700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7240a4d20 con 0x7fe7241a2f30 2021-02-10T11:35:36.345 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.353 7fe72307d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe7240b6b10 0x7fe7241a2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:36.345 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.353 7fe72307d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe7240b6b10 0x7fe7241a2910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:51352/0 (socket says 172.21.2.112:51352) 2021-02-10T11:35:36.345 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.353 7fe72307d700 1 -- 172.21.2.112:0/1560009865 learned_addr learned my addr 172.21.2.112:0/1560009865 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:35:36.345 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.354 7fe72307d700 1 -- 172.21.2.112:0/1560009865 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe7241a2f30 msgr2=0x7fe7241a3330 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:35:36.345 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.354 7fe72307d700 1 --2- 172.21.2.112:0/1560009865 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe7241a2f30 0x7fe7241a3330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:36.345 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.354 7fe72307d700 1 -- 172.21.2.112:0/1560009865 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe7240a3ec0 msgr2=0x7fe7240a42c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:35:36.345 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.354 7fe72307d700 1 --2- 172.21.2.112:0/1560009865 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe7240a3ec0 0x7fe7240a42c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:36.345 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.354 7fe72307d700 1 -- 172.21.2.112:0/1560009865 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7240a4f40 con 0x7fe7240b6b10 2021-02-10T11:35:36.345 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.354 7fe72307d700 1 --2- 172.21.2.112:0/1560009865 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe7240b6b10 0x7fe7241a2910 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 rx=0x7fe70c006f10 tx=0x7fe70c00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:36.345 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.354 7fe71bfff700 1 -- 172.21.2.112:0/1560009865 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe70c00bd80 con 0x7fe7240b6b10 2021-02-10T11:35:36.346 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.354 7fe71bfff700 1 -- 172.21.2.112:0/1560009865 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe70c0060c0 con 0x7fe7240b6b10 2021-02-10T11:35:36.346 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.354 7fe729314700 1 -- 172.21.2.112:0/1560009865 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7240a4f40 con 0x7fe7240b6b10 2021-02-10T11:35:36.346 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.354 7fe729314700 1 -- 172.21.2.112:0/1560009865 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7240a7ca0 con 0x7fe7240b6b10 2021-02-10T11:35:36.346 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.354 7fe71bfff700 1 -- 172.21.2.112:0/1560009865 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe70c0052e0 con 0x7fe7240b6b10 2021-02-10T11:35:36.346 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.355 7fe71bfff700 1 -- 172.21.2.112:0/1560009865 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe70c0062a0 con 0x7fe7240b6b10 2021-02-10T11:35:36.346 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.355 7fe71bfff700 1 --2- 172.21.2.112:0/1560009865 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe7080186c0 0x7fe70801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:36.346 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.355 7fe71bfff700 1 -- 172.21.2.112:0/1560009865 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fe70c0325a0 con 0x7fe7240b6b10 2021-02-10T11:35:36.346 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.356 7fe72287c700 1 --2- 172.21.2.112:0/1560009865 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe7080186c0 0x7fe70801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:36.346 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.357 7fe72287c700 1 --2- 172.21.2.112:0/1560009865 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe7080186c0 0x7fe70801abc0 crc :-1 s=READY pgs=726 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:36.372 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.385 7fe729314700 1 -- 172.21.2.112:0/1560009865 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe72410ca50 con 0x7fe7240b6b10 2021-02-10T11:35:36.375 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.388 7fe71bfff700 1 -- 172.21.2.112:0/1560009865 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe70c030dd0 con 0x7fe7240b6b10 2021-02-10T11:35:36.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.516 7fe729314700 1 -- 172.21.2.112:0/1560009865 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000513667", "target": ["mgr", ""], "group_name": "group_0000000000485992"}) v1 -- 0x7fe7240a7ca0 con 0x7fe7080186c0 2021-02-10T11:35:36.513 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.527 7fe71bfff700 1 -- 172.21.2.112:0/1560009865 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fe714000ef0 con 0x7fe7080186c0 2021-02-10T11:35:36.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.534 7fe729314700 1 -- 172.21.2.112:0/1560009865 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe7080186c0 msgr2=0x7fe70801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:36.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.534 7fe729314700 1 --2- 172.21.2.112:0/1560009865 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe7080186c0 0x7fe70801abc0 crc :-1 s=READY pgs=726 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:36.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.534 7fe729314700 1 -- 172.21.2.112:0/1560009865 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe7240b6b10 msgr2=0x7fe7241a2910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:36.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.534 7fe729314700 1 --2- 172.21.2.112:0/1560009865 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe7240b6b10 0x7fe7241a2910 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 rx=0x7fe70c006f10 tx=0x7fe70c00b990).stop 2021-02-10T11:35:36.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.534 7fe729314700 1 -- 172.21.2.112:0/1560009865 shutdown_connections 2021-02-10T11:35:36.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.534 7fe729314700 1 --2- 172.21.2.112:0/1560009865 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe7080186c0 0x7fe70801abc0 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:36.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.534 7fe729314700 1 --2- 172.21.2.112:0/1560009865 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe7240a3ec0 0x7fe7240a42c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:36.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.534 7fe729314700 1 --2- 172.21.2.112:0/1560009865 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe7241a2f30 0x7fe7241a3330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:36.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.534 7fe729314700 1 --2- 172.21.2.112:0/1560009865 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe7240b6b10 0x7fe7241a2910 unknown :-1 s=CLOSED pgs=1503 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:36.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.534 7fe729314700 1 -- 172.21.2.112:0/1560009865 >> 172.21.2.112:0/1560009865 conn(0x7fe7240ad750 msgr2=0x7fe72418c8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:36.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.534 7fe729314700 1 -- 172.21.2.112:0/1560009865 shutdown_connections 2021-02-10T11:35:36.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.534 7fe729314700 1 -- 172.21.2.112:0/1560009865 wait complete. 2021-02-10T11:35:36.529 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolumegroup rm cephfs group_0000000000485992 2021-02-10T11:35:36.960 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.955 7f15ab2bc700 1 Processor -- start 2021-02-10T11:35:36.960 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.955 7f15ab2bc700 1 -- start start 2021-02-10T11:35:36.960 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.956 7f15ab2bc700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f15a40e5170 0x7f15a41b38e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:36.960 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.956 7f15ab2bc700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f15a40e5570 0x7f15a407a120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:36.960 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.956 7f15ab2bc700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15a4078b90 con 0x7f15a41b3f00 2021-02-10T11:35:36.960 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.956 7f15ab2bc700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15a40d1900 con 0x7f15a40e5170 2021-02-10T11:35:36.960 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.956 7f15ab2bc700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15a40d1b20 con 0x7f15a40e5570 2021-02-10T11:35:36.961 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.957 7f15a98d9700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f15a40e5570 0x7f15a407a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:36.961 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.957 7f15a88d7700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7f15a41b3f00 0x7f15a4077510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:44378/0 (socket says 172.21.2.112:44378) 2021-02-10T11:35:36.961 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.957 7f15a90d8700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f15a40e5170 0x7f15a41b38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:36.961 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.957 7f15a88d7700 1 -- 172.21.2.112:0/3009300663 learned_addr learned my addr 172.21.2.112:0/3009300663 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:35:36.961 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.958 7f15a90d8700 1 -- 172.21.2.112:0/3009300663 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f15a40e5570 msgr2=0x7f15a407a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:36.961 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.958 7f15a90d8700 1 --2- 172.21.2.112:0/3009300663 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f15a40e5570 0x7f15a407a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:36.961 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.958 7f15a90d8700 1 -- 172.21.2.112:0/3009300663 >> v1:172.21.2.120:6789/0 conn(0x7f15a41b3f00 legacy=0x7f15a4077510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:36.961 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.958 7f159bfff700 1 -- 172.21.2.112:0/3009300663 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2218318088 0 0) 0x7f15940010b0 con 0x7f15a41b3f00 2021-02-10T11:35:36.961 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.958 7f15a90d8700 1 -- 172.21.2.112:0/3009300663 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15a40d1d40 con 0x7f15a40e5170 2021-02-10T11:35:36.961 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.958 7f15a98d9700 1 --2- 172.21.2.112:0/3009300663 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f15a40e5570 0x7f15a407a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:35:36.961 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.958 7f15a90d8700 1 --2- 172.21.2.112:0/3009300663 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f15a40e5170 0x7f15a41b38e0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 rx=0x7f15a0006650 tx=0x7f15a0009a80).ready entity=mon.1 client_cookie=7e737e69a7149298 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:36.961 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.958 7f159bfff700 1 -- 172.21.2.112:0/3009300663 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f15a000c0c0 con 0x7f15a40e5170 2021-02-10T11:35:36.961 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.959 7f159bfff700 1 -- 172.21.2.112:0/3009300663 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15a000c0c0 con 0x7f15a40e5170 2021-02-10T11:35:36.962 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.959 7f159bfff700 1 -- 172.21.2.112:0/3009300663 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f15a000c340 con 0x7f15a40e5170 2021-02-10T11:35:36.962 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.959 7f15ab2bc700 1 -- 172.21.2.112:0/3009300663 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f15a40e5170 msgr2=0x7f15a41b38e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:36.962 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.959 7f15ab2bc700 1 --2- 172.21.2.112:0/3009300663 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f15a40e5170 0x7f15a41b38e0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 rx=0x7f15a0006650 tx=0x7f15a0009a80).stop 2021-02-10T11:35:36.962 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.959 7f15ab2bc700 1 -- 172.21.2.112:0/3009300663 shutdown_connections 2021-02-10T11:35:36.962 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.959 7f15ab2bc700 1 --2- 172.21.2.112:0/3009300663 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f15a40e5570 0x7f15a407a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:36.962 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.959 7f15ab2bc700 1 --2- 172.21.2.112:0/3009300663 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f15a40e5170 0x7f15a41b38e0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:36.962 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.959 7f15ab2bc700 1 -- 172.21.2.112:0/3009300663 >> 172.21.2.112:0/3009300663 conn(0x7f15a40ce820 msgr2=0x7f15a40e4660 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:36.962 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.959 7f15ab2bc700 1 -- 172.21.2.112:0/3009300663 shutdown_connections 2021-02-10T11:35:36.962 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.959 7f15ab2bc700 1 -- 172.21.2.112:0/3009300663 wait complete. 2021-02-10T11:35:36.963 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.974 7f15ab2bc700 1 Processor -- start 2021-02-10T11:35:36.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.974 7f15ab2bc700 1 -- start start 2021-02-10T11:35:36.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.974 7f15ab2bc700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f15a40adef0 0x7f15a40af470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:36.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.974 7f15ab2bc700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f15a40a0be0 0x7f15a41a6d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:36.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.974 7f15ab2bc700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f15a41a7390 0x7f15a41a7790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:36.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.974 7f15ab2bc700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15a408c550 con 0x7f15a40adef0 2021-02-10T11:35:36.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.974 7f15ab2bc700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15a40afd50 con 0x7f15a41a7390 2021-02-10T11:35:36.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.974 7f15ab2bc700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15a40afa90 con 0x7f15a40a0be0 2021-02-10T11:35:36.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.974 7f15a90d8700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f15a40adef0 0x7f15a40af470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:36.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.974 7f15a90d8700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f15a40adef0 0x7f15a40af470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:51364/0 (socket says 172.21.2.112:51364) 2021-02-10T11:35:36.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.974 7f15a90d8700 1 -- 172.21.2.112:0/3251741590 learned_addr learned my addr 172.21.2.112:0/3251741590 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:35:36.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.975 7f15a90d8700 1 -- 172.21.2.112:0/3251741590 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f15a40a0be0 msgr2=0x7f15a41a6d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:36.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.975 7f15a90d8700 1 --2- 172.21.2.112:0/3251741590 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f15a40a0be0 0x7f15a41a6d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:36.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.975 7f15a90d8700 1 -- 172.21.2.112:0/3251741590 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f15a41a7390 msgr2=0x7f15a41a7790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:36.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.975 7f15a90d8700 1 --2- 172.21.2.112:0/3251741590 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f15a41a7390 0x7f15a41a7790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:36.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.975 7f15a90d8700 1 -- 172.21.2.112:0/3251741590 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15a40b02d0 con 0x7f15a40adef0 2021-02-10T11:35:36.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.975 7f15a90d8700 1 --2- 172.21.2.112:0/3251741590 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f15a40adef0 0x7f15a40af470 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 rx=0x7f15a0009630 tx=0x7f15a000b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:36.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.975 7f1599ffb700 1 -- 172.21.2.112:0/3251741590 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f15a000bd70 con 0x7f15a40adef0 2021-02-10T11:35:36.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.975 7f1599ffb700 1 -- 172.21.2.112:0/3251741590 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15a0000960 con 0x7f15a40adef0 2021-02-10T11:35:36.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.975 7f15ab2bc700 1 -- 172.21.2.112:0/3251741590 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15a40b02d0 con 0x7f15a40adef0 2021-02-10T11:35:36.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.975 7f15ab2bc700 1 -- 172.21.2.112:0/3251741590 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15a40b0860 con 0x7f15a40adef0 2021-02-10T11:35:36.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.975 7f1599ffb700 1 -- 172.21.2.112:0/3251741590 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f15a00054d0 con 0x7f15a40adef0 2021-02-10T11:35:36.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.976 7f1599ffb700 1 -- 172.21.2.112:0/3251741590 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f15a000a000 con 0x7f15a40adef0 2021-02-10T11:35:36.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.976 7f1599ffb700 1 --2- 172.21.2.112:0/3251741590 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f15840186c0 0x7f158401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:36.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.976 7f1599ffb700 1 -- 172.21.2.112:0/3251741590 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f15a00327d0 con 0x7f15a40adef0 2021-02-10T11:35:36.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.976 7f15a88d7700 1 --2- 172.21.2.112:0/3251741590 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f15840186c0 0x7f158401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:36.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:36.977 7f15a88d7700 1 --2- 172.21.2.112:0/3251741590 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f15840186c0 0x7f158401abc0 crc :-1 s=READY pgs=727 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:36.993 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:37.007 7f15ab2bc700 1 -- 172.21.2.112:0/3251741590 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f157c002fd0 con 0x7f15a40adef0 2021-02-10T11:35:36.996 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:37.009 7f1599ffb700 1 -- 172.21.2.112:0/3251741590 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f15a0000940 con 0x7f15a40adef0 2021-02-10T11:35:37.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:37.138 7f15ab2bc700 1 -- 172.21.2.112:0/3251741590 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolumegroup rm", "vol_name": "cephfs", "target": ["mgr", ""], "group_name": "group_0000000000485992"}) v1 -- 0x7f157c002f80 con 0x7f15840186c0 2021-02-10T11:35:37.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:37.143 7f1599ffb700 1 -- 172.21.2.112:0/3251741590 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f1594000940 con 0x7f15840186c0 2021-02-10T11:35:37.134 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:37.148 7f15ab2bc700 1 -- 172.21.2.112:0/3251741590 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f15840186c0 msgr2=0x7f158401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:37.134 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:37.148 7f15ab2bc700 1 --2- 172.21.2.112:0/3251741590 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f15840186c0 0x7f158401abc0 crc :-1 s=READY pgs=727 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:37.134 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:37.148 7f15ab2bc700 1 -- 172.21.2.112:0/3251741590 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f15a40adef0 msgr2=0x7f15a40af470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:37.135 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:37.148 7f15ab2bc700 1 --2- 172.21.2.112:0/3251741590 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f15a40adef0 0x7f15a40af470 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 rx=0x7f15a0009630 tx=0x7f15a000b980).stop 2021-02-10T11:35:37.135 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:37.148 7f15ab2bc700 1 -- 172.21.2.112:0/3251741590 shutdown_connections 2021-02-10T11:35:37.135 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:37.148 7f15ab2bc700 1 --2- 172.21.2.112:0/3251741590 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f15840186c0 0x7f158401abc0 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:37.135 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:37.148 7f15ab2bc700 1 --2- 172.21.2.112:0/3251741590 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f15a41a7390 0x7f15a41a7790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:37.135 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:37.148 7f15ab2bc700 1 --2- 172.21.2.112:0/3251741590 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f15a40a0be0 0x7f15a41a6d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:37.135 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:37.148 7f15ab2bc700 1 --2- 172.21.2.112:0/3251741590 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f15a40adef0 0x7f15a40af470 unknown :-1 s=CLOSED pgs=1504 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:37.136 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:37.148 7f15ab2bc700 1 -- 172.21.2.112:0/3251741590 >> 172.21.2.112:0/3251741590 conn(0x7f15a40a1220 msgr2=0x7f15a40b4e00 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:37.136 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:37.148 7f15ab2bc700 1 -- 172.21.2.112:0/3251741590 shutdown_connections 2021-02-10T11:35:37.136 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:37.148 7f15ab2bc700 1 -- 172.21.2.112:0/3251741590 wait complete. 2021-02-10T11:35:37.143 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:35:37.143 DEBUG:teuthology.orchestra.run.gibba012:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:35:37.487 DEBUG:teuthology.orchestra.run.gibba020:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:35:37.831 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:35:37.831 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:37.831 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:35:37.831 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:35:38.166 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:35:38.167 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:35:38.167 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba012.front.sepia.ceph.com... 2021-02-10T11:35:38.167 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:35:38.167 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:35:38.516 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba012.stderr:ceph-fuse[135129]: fuse finished with error 0 and tester_r 0 2021-02-10T11:35:38.518 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:38.518 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:35:38.518 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:35:38.853 INFO:teuthology.orchestra.run.gibba012.stdout:ext2/ext3 2021-02-10T11:35:38.853 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:35:38.853 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:35:38.854 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:35:39.188 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:39.188 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:35:39.188 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:35:39.523 INFO:teuthology.orchestra.run.gibba012.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: No such file or directory 2021-02-10T11:35:39.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:35:39.523 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2021-02-10T11:35:39.524 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:35:39.524 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:35:39.858 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:39.858 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:35:39.858 DEBUG:teuthology.orchestra.run.gibba020:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:35:40.193 INFO:teuthology.orchestra.run.gibba020.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:35:40.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:35:40.193 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:35:40.194 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:35:40.194 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:35:40.529 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:40.529 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:35:40.529 DEBUG:teuthology.orchestra.run.gibba020:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:35:40.864 INFO:teuthology.orchestra.run.gibba020.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:35:40.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:35:40.864 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:35:40.865 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:35:40.865 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:35:41.199 DEBUG:teuthology.orchestra.run.gibba012:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2021-02-10T11:35:41.202 DEBUG:teuthology.orchestra.run.gibba020:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2021-02-10T11:35:41.560 INFO:teuthology.orchestra.run.gibba012.stdout:[global] 2021-02-10T11:35:41.560 INFO:teuthology.orchestra.run.gibba012.stdout: chdir = "" 2021-02-10T11:35:41.561 INFO:teuthology.orchestra.run.gibba012.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2021-02-10T11:35:41.561 INFO:teuthology.orchestra.run.gibba012.stdout: auth supported = cephx 2021-02-10T11:35:41.561 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:35:41.561 INFO:teuthology.orchestra.run.gibba012.stdout: filestore xattr use omap = true 2021-02-10T11:35:41.561 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:35:41.561 INFO:teuthology.orchestra.run.gibba012.stdout: mon clock drift allowed = 1.000 2021-02-10T11:35:41.561 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:35:41.561 INFO:teuthology.orchestra.run.gibba012.stdout: osd crush chooseleaf type = 0 2021-02-10T11:35:41.561 INFO:teuthology.orchestra.run.gibba012.stdout: auth debug = true 2021-02-10T11:35:41.561 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:35:41.561 INFO:teuthology.orchestra.run.gibba012.stdout: ms die on old message = true 2021-02-10T11:35:41.562 INFO:teuthology.orchestra.run.gibba012.stdout: ms die on bug = true 2021-02-10T11:35:41.562 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:35:41.562 INFO:teuthology.orchestra.run.gibba012.stdout: mon max pg per osd = 10000 # >= luminous 2021-02-10T11:35:41.562 INFO:teuthology.orchestra.run.gibba012.stdout: mon pg warn max object skew = 0 2021-02-10T11:35:41.562 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:35:41.562 INFO:teuthology.orchestra.run.gibba012.stdout: osd pool default size = 2 2021-02-10T11:35:41.562 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:35:41.562 INFO:teuthology.orchestra.run.gibba012.stdout: mon osd allow primary affinity = true 2021-02-10T11:35:41.562 INFO:teuthology.orchestra.run.gibba012.stdout: mon osd allow pg remap = true 2021-02-10T11:35:41.562 INFO:teuthology.orchestra.run.gibba012.stdout: mon warn on legacy crush tunables = false 2021-02-10T11:35:41.562 INFO:teuthology.orchestra.run.gibba012.stdout: mon warn on crush straw calc version zero = false 2021-02-10T11:35:41.562 INFO:teuthology.orchestra.run.gibba012.stdout: mon warn on no sortbitwise = false 2021-02-10T11:35:41.563 INFO:teuthology.orchestra.run.gibba012.stdout: mon warn on osd down out interval zero = false 2021-02-10T11:35:41.563 INFO:teuthology.orchestra.run.gibba012.stdout: mon warn on too few osds = false 2021-02-10T11:35:41.563 INFO:teuthology.orchestra.run.gibba012.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2021-02-10T11:35:41.563 INFO:teuthology.orchestra.run.gibba012.stdout: mon_warn_on_pool_no_redundancy = false 2021-02-10T11:35:41.563 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:35:41.563 INFO:teuthology.orchestra.run.gibba012.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2021-02-10T11:35:41.563 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:35:41.563 INFO:teuthology.orchestra.run.gibba012.stdout: osd default data pool replay window = 5 2021-02-10T11:35:41.563 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:35:41.563 INFO:teuthology.orchestra.run.gibba012.stdout: mon allow pool delete = true 2021-02-10T11:35:41.563 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:35:41.563 INFO:teuthology.orchestra.run.gibba012.stdout: mon cluster log file level = debug 2021-02-10T11:35:41.563 INFO:teuthology.orchestra.run.gibba012.stdout: debug asserts on shutdown = true 2021-02-10T11:35:41.564 INFO:teuthology.orchestra.run.gibba012.stdout: mon health detail to clog = false 2021-02-10T11:35:41.564 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:35:41.564 INFO:teuthology.orchestra.run.gibba012.stdout: # we see this fail in qa on *nautilus*; bump up retries 2021-02-10T11:35:41.564 INFO:teuthology.orchestra.run.gibba012.stdout: mon_client_directed_command_retry = 4 2021-02-10T11:35:41.564 INFO:teuthology.orchestra.run.gibba012.stdout: mon host = "172.21.2.112,172.21.2.120,[v2:172.21.2.120:3301,v1:172.21.2.120:6790]" 2021-02-10T11:35:41.564 INFO:teuthology.orchestra.run.gibba012.stdout: fsid = f14f004e-0699-4584-ad81-3754258493de 2021-02-10T11:35:41.564 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:35:41.564 INFO:teuthology.orchestra.run.gibba012.stdout:[osd] 2021-02-10T11:35:41.564 INFO:teuthology.orchestra.run.gibba012.stdout: osd journal size = 100 2021-02-10T11:35:41.564 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:35:41.564 INFO:teuthology.orchestra.run.gibba012.stdout: osd scrub load threshold = 5.0 2021-02-10T11:35:41.564 INFO:teuthology.orchestra.run.gibba012.stdout: osd scrub max interval = 600 2021-02-10T11:35:41.564 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:35:41.565 INFO:teuthology.orchestra.run.gibba012.stdout: osd recover clone overlap = true 2021-02-10T11:35:41.565 INFO:teuthology.orchestra.run.gibba012.stdout: osd recovery max chunk = 1048576 2021-02-10T11:35:41.565 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:35:41.565 INFO:teuthology.orchestra.run.gibba012.stdout: osd debug shutdown = true 2021-02-10T11:35:41.565 INFO:teuthology.orchestra.run.gibba012.stdout: osd debug op order = true 2021-02-10T11:35:41.565 INFO:teuthology.orchestra.run.gibba012.stdout: osd debug verify stray on activate = true 2021-02-10T11:35:41.565 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:35:41.565 INFO:teuthology.orchestra.run.gibba012.stdout: osd open classes on start = true 2021-02-10T11:35:41.565 INFO:teuthology.orchestra.run.gibba012.stdout: osd debug pg log writeout = true 2021-02-10T11:35:41.566 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:35:41.566 INFO:teuthology.orchestra.run.gibba012.stdout: osd deep scrub update digest min age = 30 2021-02-10T11:35:41.566 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:35:41.566 INFO:teuthology.orchestra.run.gibba012.stdout: osd map max advance = 10 2021-02-10T11:35:41.566 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:35:41.566 INFO:teuthology.orchestra.run.gibba012.stdout: journal zero on create = true 2021-02-10T11:35:41.567 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:35:41.567 INFO:teuthology.orchestra.run.gibba012.stdout: filestore ondisk finisher threads = 3 2021-02-10T11:35:41.567 INFO:teuthology.orchestra.run.gibba012.stdout: filestore apply finisher threads = 3 2021-02-10T11:35:41.567 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:35:41.567 INFO:teuthology.orchestra.run.gibba012.stdout: bdev debug aio = true 2021-02-10T11:35:41.567 INFO:teuthology.orchestra.run.gibba012.stdout: osd debug misdirected ops = true 2021-02-10T11:35:41.567 INFO:teuthology.orchestra.run.gibba012.stdout: bdev async discard = True 2021-02-10T11:35:41.567 INFO:teuthology.orchestra.run.gibba012.stdout: bdev enable discard = True 2021-02-10T11:35:41.567 INFO:teuthology.orchestra.run.gibba012.stdout: bluestore allocator = bitmap 2021-02-10T11:35:41.567 INFO:teuthology.orchestra.run.gibba012.stdout: bluestore block size = 96636764160 2021-02-10T11:35:41.567 INFO:teuthology.orchestra.run.gibba012.stdout: bluestore fsck on mount = True 2021-02-10T11:35:41.567 INFO:teuthology.orchestra.run.gibba012.stdout: debug bluefs = 20 2021-02-10T11:35:41.568 INFO:teuthology.orchestra.run.gibba012.stdout: debug bluestore = 20 2021-02-10T11:35:41.568 INFO:teuthology.orchestra.run.gibba012.stdout: debug filestore = 20 2021-02-10T11:35:41.568 INFO:teuthology.orchestra.run.gibba012.stdout: debug journal = 20 2021-02-10T11:35:41.568 INFO:teuthology.orchestra.run.gibba012.stdout: debug ms = 20 2021-02-10T11:35:41.568 INFO:teuthology.orchestra.run.gibba012.stdout: debug osd = 25 2021-02-10T11:35:41.568 INFO:teuthology.orchestra.run.gibba012.stdout: debug rocksdb = 10 2021-02-10T11:35:41.568 INFO:teuthology.orchestra.run.gibba012.stdout: mon osd backfillfull_ratio = 0.85 2021-02-10T11:35:41.568 INFO:teuthology.orchestra.run.gibba012.stdout: mon osd full ratio = 0.9 2021-02-10T11:35:41.568 INFO:teuthology.orchestra.run.gibba012.stdout: mon osd nearfull ratio = 0.8 2021-02-10T11:35:41.569 INFO:teuthology.orchestra.run.gibba012.stdout: osd failsafe full ratio = 0.95 2021-02-10T11:35:41.569 INFO:teuthology.orchestra.run.gibba012.stdout: osd objectstore = bluestore 2021-02-10T11:35:41.569 INFO:teuthology.orchestra.run.gibba012.stdout: osd op complaint time = 180 2021-02-10T11:35:41.569 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:35:41.569 INFO:teuthology.orchestra.run.gibba012.stdout:[mgr] 2021-02-10T11:35:41.569 INFO:teuthology.orchestra.run.gibba012.stdout: debug ms = 1 2021-02-10T11:35:41.569 INFO:teuthology.orchestra.run.gibba012.stdout: debug mgr = 20 2021-02-10T11:35:41.569 INFO:teuthology.orchestra.run.gibba012.stdout: debug mon = 20 2021-02-10T11:35:41.569 INFO:teuthology.orchestra.run.gibba012.stdout: debug auth = 20 2021-02-10T11:35:41.569 INFO:teuthology.orchestra.run.gibba012.stdout: mon reweight min pgs per osd = 4 2021-02-10T11:35:41.569 INFO:teuthology.orchestra.run.gibba012.stdout: mon reweight min bytes per osd = 10 2021-02-10T11:35:41.569 INFO:teuthology.orchestra.run.gibba012.stdout: debug client = 10 2021-02-10T11:35:41.569 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:35:41.570 INFO:teuthology.orchestra.run.gibba012.stdout:[mon] 2021-02-10T11:35:41.570 INFO:teuthology.orchestra.run.gibba012.stdout: debug ms = 1 2021-02-10T11:35:41.570 INFO:teuthology.orchestra.run.gibba012.stdout: debug mon = 20 2021-02-10T11:35:41.570 INFO:teuthology.orchestra.run.gibba012.stdout: debug paxos = 20 2021-02-10T11:35:41.570 INFO:teuthology.orchestra.run.gibba012.stdout: debug auth = 20 2021-02-10T11:35:41.570 INFO:teuthology.orchestra.run.gibba012.stdout: mon data avail warn = 5 2021-02-10T11:35:41.570 INFO:teuthology.orchestra.run.gibba012.stdout: mon mgr mkfs grace = 240 2021-02-10T11:35:41.570 INFO:teuthology.orchestra.run.gibba012.stdout: mon reweight min pgs per osd = 4 2021-02-10T11:35:41.570 INFO:teuthology.orchestra.run.gibba012.stdout: mon osd reporter subtree level = osd 2021-02-10T11:35:41.570 INFO:teuthology.orchestra.run.gibba012.stdout: mon osd prime pg temp = true 2021-02-10T11:35:41.570 INFO:teuthology.orchestra.run.gibba012.stdout: mon reweight min bytes per osd = 10 2021-02-10T11:35:41.570 INFO:teuthology.orchestra.run.gibba012.stdout: mon op complaint time = 120 2021-02-10T11:35:41.570 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:35:41.571 INFO:teuthology.orchestra.run.gibba012.stdout:[client] 2021-02-10T11:35:41.571 INFO:teuthology.orchestra.run.gibba012.stdout: rgw cache enabled = true 2021-02-10T11:35:41.571 INFO:teuthology.orchestra.run.gibba012.stdout: rgw enable ops log = true 2021-02-10T11:35:41.571 INFO:teuthology.orchestra.run.gibba012.stdout: rgw enable usage log = true 2021-02-10T11:35:41.571 INFO:teuthology.orchestra.run.gibba012.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2021-02-10T11:35:41.571 INFO:teuthology.orchestra.run.gibba012.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2021-02-10T11:35:41.571 INFO:teuthology.orchestra.run.gibba012.stdout: client mount timeout = 600 2021-02-10T11:35:41.571 INFO:teuthology.orchestra.run.gibba012.stdout: debug client = 20 2021-02-10T11:35:41.571 INFO:teuthology.orchestra.run.gibba012.stdout: debug ms = 1 2021-02-10T11:35:41.571 INFO:teuthology.orchestra.run.gibba012.stdout: pid file = "" 2021-02-10T11:35:41.571 INFO:teuthology.orchestra.run.gibba012.stdout:[mon.a] 2021-02-10T11:35:41.571 INFO:teuthology.orchestra.run.gibba012.stdout:[mon.b] 2021-02-10T11:35:41.571 INFO:teuthology.orchestra.run.gibba012.stdout:[mon.c] 2021-02-10T11:35:41.572 INFO:teuthology.orchestra.run.gibba012.stdout:[mds.a] 2021-02-10T11:35:41.572 INFO:teuthology.orchestra.run.gibba012.stdout:[mds.b] 2021-02-10T11:35:41.572 INFO:teuthology.orchestra.run.gibba012.stdout:[mds] 2021-02-10T11:35:41.572 INFO:teuthology.orchestra.run.gibba012.stdout: debug mds = 20 2021-02-10T11:35:41.572 INFO:teuthology.orchestra.run.gibba012.stdout: debug ms = 1 2021-02-10T11:35:41.572 INFO:teuthology.orchestra.run.gibba012.stdout: mds bal frag = True 2021-02-10T11:35:41.572 INFO:teuthology.orchestra.run.gibba012.stdout: mds bal fragment size max = 10000 2021-02-10T11:35:41.572 INFO:teuthology.orchestra.run.gibba012.stdout: mds bal merge size = 5 2021-02-10T11:35:41.572 INFO:teuthology.orchestra.run.gibba012.stdout: mds bal split bits = 3 2021-02-10T11:35:41.572 INFO:teuthology.orchestra.run.gibba012.stdout: mds bal split size = 100 2021-02-10T11:35:41.572 INFO:teuthology.orchestra.run.gibba012.stdout: mds debug frag = True 2021-02-10T11:35:41.572 INFO:teuthology.orchestra.run.gibba012.stdout: mds debug scatterstat = True 2021-02-10T11:35:41.572 INFO:teuthology.orchestra.run.gibba012.stdout: mds op complaint time = 180 2021-02-10T11:35:41.572 INFO:teuthology.orchestra.run.gibba012.stdout: mds verify scatter = True 2021-02-10T11:35:41.573 INFO:teuthology.orchestra.run.gibba012.stdout: osd op complaint time = 180 2021-02-10T11:35:41.573 INFO:teuthology.orchestra.run.gibba012.stdout:[client.alice] 2021-02-10T11:35:41.574 INFO:teuthology.orchestra.run.gibba020.stdout:[global] 2021-02-10T11:35:41.574 INFO:teuthology.orchestra.run.gibba020.stdout: chdir = "" 2021-02-10T11:35:41.574 INFO:teuthology.orchestra.run.gibba020.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2021-02-10T11:35:41.574 INFO:teuthology.orchestra.run.gibba020.stdout: auth supported = cephx 2021-02-10T11:35:41.574 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:35:41.574 INFO:teuthology.orchestra.run.gibba020.stdout: filestore xattr use omap = true 2021-02-10T11:35:41.574 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:35:41.575 INFO:teuthology.orchestra.run.gibba020.stdout: mon clock drift allowed = 1.000 2021-02-10T11:35:41.575 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:35:41.575 INFO:teuthology.orchestra.run.gibba020.stdout: osd crush chooseleaf type = 0 2021-02-10T11:35:41.575 INFO:teuthology.orchestra.run.gibba020.stdout: auth debug = true 2021-02-10T11:35:41.575 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:35:41.575 INFO:teuthology.orchestra.run.gibba020.stdout: ms die on old message = true 2021-02-10T11:35:41.575 INFO:teuthology.orchestra.run.gibba020.stdout: ms die on bug = true 2021-02-10T11:35:41.575 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:35:41.575 INFO:teuthology.orchestra.run.gibba020.stdout: mon max pg per osd = 10000 # >= luminous 2021-02-10T11:35:41.575 INFO:teuthology.orchestra.run.gibba020.stdout: mon pg warn max object skew = 0 2021-02-10T11:35:41.575 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:35:41.575 INFO:teuthology.orchestra.run.gibba020.stdout: osd pool default size = 2 2021-02-10T11:35:41.575 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:35:41.576 INFO:teuthology.orchestra.run.gibba020.stdout: mon osd allow primary affinity = true 2021-02-10T11:35:41.576 INFO:teuthology.orchestra.run.gibba020.stdout: mon osd allow pg remap = true 2021-02-10T11:35:41.576 INFO:teuthology.orchestra.run.gibba020.stdout: mon warn on legacy crush tunables = false 2021-02-10T11:35:41.576 INFO:teuthology.orchestra.run.gibba020.stdout: mon warn on crush straw calc version zero = false 2021-02-10T11:35:41.576 INFO:teuthology.orchestra.run.gibba020.stdout: mon warn on no sortbitwise = false 2021-02-10T11:35:41.576 INFO:teuthology.orchestra.run.gibba020.stdout: mon warn on osd down out interval zero = false 2021-02-10T11:35:41.576 INFO:teuthology.orchestra.run.gibba020.stdout: mon warn on too few osds = false 2021-02-10T11:35:41.576 INFO:teuthology.orchestra.run.gibba020.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2021-02-10T11:35:41.576 INFO:teuthology.orchestra.run.gibba020.stdout: mon_warn_on_pool_no_redundancy = false 2021-02-10T11:35:41.576 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:35:41.576 INFO:teuthology.orchestra.run.gibba020.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2021-02-10T11:35:41.576 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:35:41.576 INFO:teuthology.orchestra.run.gibba020.stdout: osd default data pool replay window = 5 2021-02-10T11:35:41.576 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:35:41.577 INFO:teuthology.orchestra.run.gibba020.stdout: mon allow pool delete = true 2021-02-10T11:35:41.577 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:35:41.577 INFO:teuthology.orchestra.run.gibba020.stdout: mon cluster log file level = debug 2021-02-10T11:35:41.577 INFO:teuthology.orchestra.run.gibba020.stdout: debug asserts on shutdown = true 2021-02-10T11:35:41.577 INFO:teuthology.orchestra.run.gibba020.stdout: mon health detail to clog = false 2021-02-10T11:35:41.577 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:35:41.577 INFO:teuthology.orchestra.run.gibba020.stdout: # we see this fail in qa on *nautilus*; bump up retries 2021-02-10T11:35:41.577 INFO:teuthology.orchestra.run.gibba020.stdout: mon_client_directed_command_retry = 4 2021-02-10T11:35:41.577 INFO:teuthology.orchestra.run.gibba020.stdout: mon host = "172.21.2.112,172.21.2.120,[v2:172.21.2.120:3301,v1:172.21.2.120:6790]" 2021-02-10T11:35:41.577 INFO:teuthology.orchestra.run.gibba020.stdout: fsid = f14f004e-0699-4584-ad81-3754258493de 2021-02-10T11:35:41.577 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:35:41.577 INFO:teuthology.orchestra.run.gibba020.stdout:[osd] 2021-02-10T11:35:41.577 INFO:teuthology.orchestra.run.gibba020.stdout: osd journal size = 100 2021-02-10T11:35:41.578 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:35:41.578 INFO:teuthology.orchestra.run.gibba020.stdout: osd scrub load threshold = 5.0 2021-02-10T11:35:41.578 INFO:teuthology.orchestra.run.gibba020.stdout: osd scrub max interval = 600 2021-02-10T11:35:41.578 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:35:41.578 INFO:teuthology.orchestra.run.gibba020.stdout: osd recover clone overlap = true 2021-02-10T11:35:41.578 INFO:teuthology.orchestra.run.gibba020.stdout: osd recovery max chunk = 1048576 2021-02-10T11:35:41.578 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:35:41.578 INFO:teuthology.orchestra.run.gibba020.stdout: osd debug shutdown = true 2021-02-10T11:35:41.578 INFO:teuthology.orchestra.run.gibba020.stdout: osd debug op order = true 2021-02-10T11:35:41.578 INFO:teuthology.orchestra.run.gibba020.stdout: osd debug verify stray on activate = true 2021-02-10T11:35:41.578 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:35:41.578 INFO:teuthology.orchestra.run.gibba020.stdout: osd open classes on start = true 2021-02-10T11:35:41.578 INFO:teuthology.orchestra.run.gibba020.stdout: osd debug pg log writeout = true 2021-02-10T11:35:41.579 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:35:41.579 INFO:teuthology.orchestra.run.gibba020.stdout: osd deep scrub update digest min age = 30 2021-02-10T11:35:41.579 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:35:41.579 INFO:teuthology.orchestra.run.gibba020.stdout: osd map max advance = 10 2021-02-10T11:35:41.579 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:35:41.579 INFO:teuthology.orchestra.run.gibba020.stdout: journal zero on create = true 2021-02-10T11:35:41.579 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:35:41.579 INFO:teuthology.orchestra.run.gibba020.stdout: filestore ondisk finisher threads = 3 2021-02-10T11:35:41.579 INFO:teuthology.orchestra.run.gibba020.stdout: filestore apply finisher threads = 3 2021-02-10T11:35:41.579 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:35:41.579 INFO:teuthology.orchestra.run.gibba020.stdout: bdev debug aio = true 2021-02-10T11:35:41.579 INFO:teuthology.orchestra.run.gibba020.stdout: osd debug misdirected ops = true 2021-02-10T11:35:41.579 INFO:teuthology.orchestra.run.gibba020.stdout: bdev async discard = True 2021-02-10T11:35:41.579 INFO:teuthology.orchestra.run.gibba020.stdout: bdev enable discard = True 2021-02-10T11:35:41.580 INFO:teuthology.orchestra.run.gibba020.stdout: bluestore allocator = bitmap 2021-02-10T11:35:41.580 INFO:teuthology.orchestra.run.gibba020.stdout: bluestore block size = 96636764160 2021-02-10T11:35:41.580 INFO:teuthology.orchestra.run.gibba020.stdout: bluestore fsck on mount = True 2021-02-10T11:35:41.580 INFO:teuthology.orchestra.run.gibba020.stdout: debug bluefs = 20 2021-02-10T11:35:41.580 INFO:teuthology.orchestra.run.gibba020.stdout: debug bluestore = 20 2021-02-10T11:35:41.580 INFO:teuthology.orchestra.run.gibba020.stdout: debug filestore = 20 2021-02-10T11:35:41.580 INFO:teuthology.orchestra.run.gibba020.stdout: debug journal = 20 2021-02-10T11:35:41.580 INFO:teuthology.orchestra.run.gibba020.stdout: debug ms = 20 2021-02-10T11:35:41.580 INFO:teuthology.orchestra.run.gibba020.stdout: debug osd = 25 2021-02-10T11:35:41.580 INFO:teuthology.orchestra.run.gibba020.stdout: debug rocksdb = 10 2021-02-10T11:35:41.580 INFO:teuthology.orchestra.run.gibba020.stdout: mon osd backfillfull_ratio = 0.85 2021-02-10T11:35:41.580 INFO:teuthology.orchestra.run.gibba020.stdout: mon osd full ratio = 0.9 2021-02-10T11:35:41.580 INFO:teuthology.orchestra.run.gibba020.stdout: mon osd nearfull ratio = 0.8 2021-02-10T11:35:41.580 INFO:teuthology.orchestra.run.gibba020.stdout: osd failsafe full ratio = 0.95 2021-02-10T11:35:41.581 INFO:teuthology.orchestra.run.gibba020.stdout: osd objectstore = bluestore 2021-02-10T11:35:41.581 INFO:teuthology.orchestra.run.gibba020.stdout: osd op complaint time = 180 2021-02-10T11:35:41.581 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:35:41.581 INFO:teuthology.orchestra.run.gibba020.stdout:[mgr] 2021-02-10T11:35:41.581 INFO:teuthology.orchestra.run.gibba020.stdout: debug ms = 1 2021-02-10T11:35:41.581 INFO:teuthology.orchestra.run.gibba020.stdout: debug mgr = 20 2021-02-10T11:35:41.581 INFO:teuthology.orchestra.run.gibba020.stdout: debug mon = 20 2021-02-10T11:35:41.581 INFO:teuthology.orchestra.run.gibba020.stdout: debug auth = 20 2021-02-10T11:35:41.581 INFO:teuthology.orchestra.run.gibba020.stdout: mon reweight min pgs per osd = 4 2021-02-10T11:35:41.581 INFO:teuthology.orchestra.run.gibba020.stdout: mon reweight min bytes per osd = 10 2021-02-10T11:35:41.581 INFO:teuthology.orchestra.run.gibba020.stdout: debug client = 10 2021-02-10T11:35:41.581 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:35:41.581 INFO:teuthology.orchestra.run.gibba020.stdout:[mon] 2021-02-10T11:35:41.582 INFO:teuthology.orchestra.run.gibba020.stdout: debug ms = 1 2021-02-10T11:35:41.582 INFO:teuthology.orchestra.run.gibba020.stdout: debug mon = 20 2021-02-10T11:35:41.582 INFO:teuthology.orchestra.run.gibba020.stdout: debug paxos = 20 2021-02-10T11:35:41.582 INFO:teuthology.orchestra.run.gibba020.stdout: debug auth = 20 2021-02-10T11:35:41.582 INFO:teuthology.orchestra.run.gibba020.stdout: mon data avail warn = 5 2021-02-10T11:35:41.582 INFO:teuthology.orchestra.run.gibba020.stdout: mon mgr mkfs grace = 240 2021-02-10T11:35:41.582 INFO:teuthology.orchestra.run.gibba020.stdout: mon reweight min pgs per osd = 4 2021-02-10T11:35:41.582 INFO:teuthology.orchestra.run.gibba020.stdout: mon osd reporter subtree level = osd 2021-02-10T11:35:41.582 INFO:teuthology.orchestra.run.gibba020.stdout: mon osd prime pg temp = true 2021-02-10T11:35:41.582 INFO:teuthology.orchestra.run.gibba020.stdout: mon reweight min bytes per osd = 10 2021-02-10T11:35:41.582 INFO:teuthology.orchestra.run.gibba020.stdout: mon op complaint time = 120 2021-02-10T11:35:41.582 INFO:teuthology.orchestra.run.gibba020.stdout: 2021-02-10T11:35:41.582 INFO:teuthology.orchestra.run.gibba020.stdout:[client] 2021-02-10T11:35:41.583 INFO:teuthology.orchestra.run.gibba020.stdout: rgw cache enabled = true 2021-02-10T11:35:41.583 INFO:teuthology.orchestra.run.gibba020.stdout: rgw enable ops log = true 2021-02-10T11:35:41.583 INFO:teuthology.orchestra.run.gibba020.stdout: rgw enable usage log = true 2021-02-10T11:35:41.583 INFO:teuthology.orchestra.run.gibba020.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2021-02-10T11:35:41.583 INFO:teuthology.orchestra.run.gibba020.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2021-02-10T11:35:41.583 INFO:teuthology.orchestra.run.gibba020.stdout: client mount timeout = 600 2021-02-10T11:35:41.583 INFO:teuthology.orchestra.run.gibba020.stdout: debug client = 20 2021-02-10T11:35:41.583 INFO:teuthology.orchestra.run.gibba020.stdout: debug ms = 1 2021-02-10T11:35:41.583 INFO:teuthology.orchestra.run.gibba020.stdout: pid file = "" 2021-02-10T11:35:41.583 INFO:teuthology.orchestra.run.gibba020.stdout:[mon.a] 2021-02-10T11:35:41.583 INFO:teuthology.orchestra.run.gibba020.stdout:[mon.b] 2021-02-10T11:35:41.583 INFO:teuthology.orchestra.run.gibba020.stdout:[mon.c] 2021-02-10T11:35:41.583 INFO:teuthology.orchestra.run.gibba020.stdout:[mds.a] 2021-02-10T11:35:41.583 INFO:teuthology.orchestra.run.gibba020.stdout:[mds.b] 2021-02-10T11:35:41.584 INFO:teuthology.orchestra.run.gibba020.stdout:[mds] 2021-02-10T11:35:41.584 INFO:teuthology.orchestra.run.gibba020.stdout: debug mds = 20 2021-02-10T11:35:41.584 INFO:teuthology.orchestra.run.gibba020.stdout: debug ms = 1 2021-02-10T11:35:41.584 INFO:teuthology.orchestra.run.gibba020.stdout: mds bal frag = True 2021-02-10T11:35:41.584 INFO:teuthology.orchestra.run.gibba020.stdout: mds bal fragment size max = 10000 2021-02-10T11:35:41.584 INFO:teuthology.orchestra.run.gibba020.stdout: mds bal merge size = 5 2021-02-10T11:35:41.584 INFO:teuthology.orchestra.run.gibba020.stdout: mds bal split bits = 3 2021-02-10T11:35:41.584 INFO:teuthology.orchestra.run.gibba020.stdout: mds bal split size = 100 2021-02-10T11:35:41.584 INFO:teuthology.orchestra.run.gibba020.stdout: mds debug frag = True 2021-02-10T11:35:41.584 INFO:teuthology.orchestra.run.gibba020.stdout: mds debug scatterstat = True 2021-02-10T11:35:41.584 INFO:teuthology.orchestra.run.gibba020.stdout: mds op complaint time = 180 2021-02-10T11:35:41.584 INFO:teuthology.orchestra.run.gibba020.stdout: mds verify scatter = True 2021-02-10T11:35:41.584 INFO:teuthology.orchestra.run.gibba020.stdout: osd op complaint time = 180 2021-02-10T11:35:41.585 INFO:teuthology.orchestra.run.gibba020.stdout:[client.alice] 2021-02-10T11:35:41.585 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config rm client.alice debug_client 2021-02-10T11:35:42.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.012 7f5f91080700 1 Processor -- start 2021-02-10T11:35:42.017 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.012 7f5f91080700 1 -- start start 2021-02-10T11:35:42.017 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.013 7f5f91080700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f5f8c0d1870 0x7f5f8c1b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:42.017 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.013 7f5f91080700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f5f8c0f0650 0x7f5f8c0cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:42.017 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.013 7f5f91080700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f8c0f0b10 con 0x7f5f8c1b0a10 2021-02-10T11:35:42.017 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.013 7f5f91080700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f8c0e5ed0 con 0x7f5f8c0f0650 2021-02-10T11:35:42.017 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.013 7f5f91080700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f8c0d1c70 con 0x7f5f8c0d1870 2021-02-10T11:35:42.017 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.013 7f5f8ad9d700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f5f8c0d1870 0x7f5f8c1b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:42.017 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.013 7f5f8a59c700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7f5f8c1b0a10 0x7f5f8c0cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:44390/0 (socket says 172.21.2.112:44390) 2021-02-10T11:35:42.017 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.013 7f5f8b59e700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f5f8c0f0650 0x7f5f8c0cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:42.018 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.013 7f5f8a59c700 1 -- 172.21.2.112:0/49117523 learned_addr learned my addr 172.21.2.112:0/49117523 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:35:42.018 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.014 7f5f8b59e700 1 -- 172.21.2.112:0/49117523 >> v2:172.21.2.120:3300/0 conn(0x7f5f8c0d1870 msgr2=0x7f5f8c1b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:42.018 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.014 7f5f8b59e700 1 --2- 172.21.2.112:0/49117523 >> v2:172.21.2.120:3300/0 conn(0x7f5f8c0d1870 0x7f5f8c1b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:42.018 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.014 7f5f8b59e700 1 -- 172.21.2.112:0/49117523 >> v1:172.21.2.120:6789/0 conn(0x7f5f8c1b0a10 legacy=0x7f5f8c0cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:42.018 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.014 7f5f8b59e700 1 -- 172.21.2.112:0/49117523 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f8c0d1e90 con 0x7f5f8c0f0650 2021-02-10T11:35:42.019 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.014 7f5f8b59e700 1 --2- 172.21.2.112:0/49117523 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5f8c0f0650 0x7f5f8c0cfb20 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 rx=0x7f5f7c006ab0 tx=0x7f5f7c009b40).ready entity=mon.0 client_cookie=141406d50d494ed1 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:42.019 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.014 7f5f89d9b700 1 -- 172.21.2.112:0/49117523 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5f7c0022e0 con 0x7f5f8c0f0650 2021-02-10T11:35:42.019 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.014 7f5f89d9b700 1 -- 172.21.2.112:0/49117523 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f7c0009b0 con 0x7f5f8c0f0650 2021-02-10T11:35:42.019 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.014 7f5f89d9b700 1 -- 172.21.2.112:0/49117523 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5f7c0026d0 con 0x7f5f8c0f0650 2021-02-10T11:35:42.019 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.014 7f5f91080700 1 -- 172.21.2.112:0/49117523 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5f8c0f0650 msgr2=0x7f5f8c0cfb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:42.019 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.014 7f5f91080700 1 --2- 172.21.2.112:0/49117523 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5f8c0f0650 0x7f5f8c0cfb20 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 rx=0x7f5f7c006ab0 tx=0x7f5f7c009b40).stop 2021-02-10T11:35:42.019 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.015 7f5f91080700 1 -- 172.21.2.112:0/49117523 shutdown_connections 2021-02-10T11:35:42.019 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.015 7f5f91080700 1 --2- 172.21.2.112:0/49117523 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5f8c0f0650 0x7f5f8c0cfb20 unknown :-1 s=CLOSED pgs=1505 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:42.019 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.015 7f5f91080700 1 --2- 172.21.2.112:0/49117523 >> v2:172.21.2.120:3300/0 conn(0x7f5f8c0d1870 0x7f5f8c1b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:42.019 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.015 7f5f91080700 1 -- 172.21.2.112:0/49117523 >> 172.21.2.112:0/49117523 conn(0x7f5f8c0ce450 msgr2=0x7f5f8c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:42.019 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.015 7f5f91080700 1 -- 172.21.2.112:0/49117523 shutdown_connections 2021-02-10T11:35:42.019 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.015 7f5f91080700 1 -- 172.21.2.112:0/49117523 wait complete. 2021-02-10T11:35:42.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.030 7f5f91080700 1 Processor -- start 2021-02-10T11:35:42.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.030 7f5f91080700 1 -- start start 2021-02-10T11:35:42.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.030 7f5f91080700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5f8c1a06c0 0x7f5f8c0ebe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:42.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.030 7f5f91080700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5f8c0ec490 0x7f5f8c0c30f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:42.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.030 7f5f91080700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5f8c0c3710 0x7f5f8c0bc070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:42.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.030 7f5f91080700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f8c0c3b10 con 0x7f5f8c1a06c0 2021-02-10T11:35:42.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.030 7f5f91080700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f8c0eca90 con 0x7f5f8c0ec490 2021-02-10T11:35:42.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.030 7f5f91080700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f8c0bc690 con 0x7f5f8c0c3710 2021-02-10T11:35:42.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.031 7f5f8ad9d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5f8c1a06c0 0x7f5f8c0ebe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:42.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.031 7f5f8ad9d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5f8c1a06c0 0x7f5f8c0ebe70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:51376/0 (socket says 172.21.2.112:51376) 2021-02-10T11:35:42.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.031 7f5f8a59c700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5f8c0ec490 0x7f5f8c0c30f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:42.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.031 7f5f8ad9d700 1 -- 172.21.2.112:0/2653825370 learned_addr learned my addr 172.21.2.112:0/2653825370 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:35:42.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.031 7f5f8b59e700 1 --2- 172.21.2.112:0/2653825370 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5f8c0c3710 0x7f5f8c0bc070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:42.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.031 7f5f8ad9d700 1 -- 172.21.2.112:0/2653825370 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5f8c0c3710 msgr2=0x7f5f8c0bc070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:42.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.031 7f5f8ad9d700 1 --2- 172.21.2.112:0/2653825370 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5f8c0c3710 0x7f5f8c0bc070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:42.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.031 7f5f8ad9d700 1 -- 172.21.2.112:0/2653825370 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5f8c0ec490 msgr2=0x7f5f8c0c30f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:42.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.031 7f5f8ad9d700 1 --2- 172.21.2.112:0/2653825370 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5f8c0ec490 0x7f5f8c0c30f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:42.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.031 7f5f8ad9d700 1 -- 172.21.2.112:0/2653825370 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f8c0bc870 con 0x7f5f8c1a06c0 2021-02-10T11:35:42.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.031 7f5f8ad9d700 1 --2- 172.21.2.112:0/2653825370 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5f8c1a06c0 0x7f5f8c0ebe70 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 rx=0x7f5f80006f60 tx=0x7f5f8000b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:42.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.031 7f5f7b7fe700 1 -- 172.21.2.112:0/2653825370 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5f8000bd60 con 0x7f5f8c1a06c0 2021-02-10T11:35:42.024 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.031 7f5f7b7fe700 1 -- 172.21.2.112:0/2653825370 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f80000c90 con 0x7f5f8c1a06c0 2021-02-10T11:35:42.024 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.031 7f5f91080700 1 -- 172.21.2.112:0/2653825370 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f8c0bc870 con 0x7f5f8c1a06c0 2021-02-10T11:35:42.024 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.031 7f5f7b7fe700 1 -- 172.21.2.112:0/2653825370 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5f800022b0 con 0x7f5f8c1a06c0 2021-02-10T11:35:42.024 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.031 7f5f91080700 1 -- 172.21.2.112:0/2653825370 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f8c0bc870 con 0x7f5f8c1a06c0 2021-02-10T11:35:42.024 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.032 7f5f7b7fe700 1 -- 172.21.2.112:0/2653825370 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5f80006b00 con 0x7f5f8c1a06c0 2021-02-10T11:35:42.024 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.032 7f5f7b7fe700 1 --2- 172.21.2.112:0/2653825370 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5f700186d0 0x7f5f7001abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:42.024 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.032 7f5f7b7fe700 1 -- 172.21.2.112:0/2653825370 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f5f80032610 con 0x7f5f8c1a06c0 2021-02-10T11:35:42.024 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.032 7f5f8a59c700 1 --2- 172.21.2.112:0/2653825370 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5f700186d0 0x7f5f7001abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:42.024 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.032 7f5f8a59c700 1 --2- 172.21.2.112:0/2653825370 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5f700186d0 0x7f5f7001abd0 crc :-1 s=READY pgs=728 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:42.049 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.063 7f5f91080700 1 -- 172.21.2.112:0/2653825370 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f8c0ebc40 con 0x7f5f8c1a06c0 2021-02-10T11:35:42.052 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.065 7f5f7b7fe700 1 -- 172.21.2.112:0/2653825370 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5f80000c90 con 0x7f5f8c1a06c0 2021-02-10T11:35:42.202 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.216 7f5f91080700 1 -- 172.21.2.112:0/2653825370 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "config rm", "who": "client.alice", "name": "debug_client"} v 0) v1 -- 0x7f5f8c0ebc40 con 0x7f5f8c1a06c0 2021-02-10T11:35:42.206 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.219 7f5f7b7fe700 1 -- 172.21.2.112:0/2653825370 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "client.alice", "name": "debug_client"}]=0 v17) v1 ==== 104+0+0 (secure 0 0 0) 0x7f5f80006b00 con 0x7f5f8c1a06c0 2021-02-10T11:35:42.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.224 7f5f91080700 1 -- 172.21.2.112:0/2653825370 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5f700186d0 msgr2=0x7f5f7001abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:42.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.225 7f5f91080700 1 --2- 172.21.2.112:0/2653825370 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5f700186d0 0x7f5f7001abd0 crc :-1 s=READY pgs=728 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:42.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.225 7f5f91080700 1 -- 172.21.2.112:0/2653825370 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5f8c1a06c0 msgr2=0x7f5f8c0ebe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:42.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.225 7f5f91080700 1 --2- 172.21.2.112:0/2653825370 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5f8c1a06c0 0x7f5f8c0ebe70 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 rx=0x7f5f80006f60 tx=0x7f5f8000b970).stop 2021-02-10T11:35:42.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.225 7f5f91080700 1 -- 172.21.2.112:0/2653825370 shutdown_connections 2021-02-10T11:35:42.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.225 7f5f91080700 1 --2- 172.21.2.112:0/2653825370 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5f700186d0 0x7f5f7001abd0 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:42.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.225 7f5f91080700 1 --2- 172.21.2.112:0/2653825370 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5f8c0ec490 0x7f5f8c0c30f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:42.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.225 7f5f91080700 1 --2- 172.21.2.112:0/2653825370 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5f8c0c3710 0x7f5f8c0bc070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:42.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.225 7f5f91080700 1 --2- 172.21.2.112:0/2653825370 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5f8c1a06c0 0x7f5f8c0ebe70 unknown :-1 s=CLOSED pgs=1506 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:42.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.225 7f5f91080700 1 -- 172.21.2.112:0/2653825370 >> 172.21.2.112:0/2653825370 conn(0x7f5f8c0a6bb0 msgr2=0x7f5f8c095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:42.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.225 7f5f91080700 1 -- 172.21.2.112:0/2653825370 shutdown_connections 2021-02-10T11:35:42.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.225 7f5f91080700 1 -- 172.21.2.112:0/2653825370 wait complete. 2021-02-10T11:35:42.220 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config rm client.alice debug_objecter 2021-02-10T11:35:42.652 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.647 7f22c7771700 1 Processor -- start 2021-02-10T11:35:42.652 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.647 7f22c7771700 1 -- start start 2021-02-10T11:35:42.652 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.648 7f22c7771700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f22c00d1860 0x7f22c01b03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:42.652 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.648 7f22c7771700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f22c01b0a00 0x7f22c00cf320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:42.652 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.648 7f22c7771700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22c00f0b00 con 0x7f22c00d1c60 2021-02-10T11:35:42.652 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.648 7f22c7771700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22c00e5ec0 con 0x7f22c00d1860 2021-02-10T11:35:42.652 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.648 7f22c7771700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22c00f0640 con 0x7f22c01b0a00 2021-02-10T11:35:42.652 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.648 7f22c5d8e700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7f22c00d1c60 0x7f22c00cf940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:44402/0 (socket says 172.21.2.112:44402) 2021-02-10T11:35:42.652 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.648 7f22c5d8e700 1 -- 172.21.2.112:0/2532143486 learned_addr learned my addr 172.21.2.112:0/2532143486 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:35:42.652 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.648 7f22c558d700 1 --2- 172.21.2.112:0/2532143486 >> v2:172.21.2.120:3300/0 conn(0x7f22c00d1860 0x7f22c01b03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:42.652 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.649 7f22c4d8c700 1 --2- 172.21.2.112:0/2532143486 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f22c01b0a00 0x7f22c00cf320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:42.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.649 7f22c4d8c700 1 -- 172.21.2.112:0/2532143486 >> v2:172.21.2.120:3300/0 conn(0x7f22c00d1860 msgr2=0x7f22c01b03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:42.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.649 7f22c4d8c700 1 --2- 172.21.2.112:0/2532143486 >> v2:172.21.2.120:3300/0 conn(0x7f22c00d1860 0x7f22c01b03e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:42.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.649 7f22c4d8c700 1 -- 172.21.2.112:0/2532143486 >> v1:172.21.2.120:6789/0 conn(0x7f22c00d1c60 legacy=0x7f22c00cf940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:42.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.649 7f22c4d8c700 1 -- 172.21.2.112:0/2532143486 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22c00f0860 con 0x7f22c01b0a00 2021-02-10T11:35:42.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.649 7f22c558d700 1 --2- 172.21.2.112:0/2532143486 >> v2:172.21.2.120:3300/0 conn(0x7f22c00d1860 0x7f22c01b03e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:35:42.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.650 7f22c4d8c700 1 --2- 172.21.2.112:0/2532143486 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f22c01b0a00 0x7f22c00cf320 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f22bc0069d0 tx=0x7f22bc009e40).ready entity=mon.2 client_cookie=e7a04c0a7b431a43 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:42.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.650 7f22b7fff700 1 -- 172.21.2.112:0/2532143486 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f22bc00b510 con 0x7f22c01b0a00 2021-02-10T11:35:42.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.650 7f22b7fff700 1 -- 172.21.2.112:0/2532143486 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22bc00b790 con 0x7f22c01b0a00 2021-02-10T11:35:42.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.650 7f22b7fff700 1 -- 172.21.2.112:0/2532143486 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f22bc00bcb0 con 0x7f22c01b0a00 2021-02-10T11:35:42.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.650 7f22c7771700 1 -- 172.21.2.112:0/2532143486 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f22c01b0a00 msgr2=0x7f22c00cf320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:42.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.650 7f22c7771700 1 --2- 172.21.2.112:0/2532143486 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f22c01b0a00 0x7f22c00cf320 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f22bc0069d0 tx=0x7f22bc009e40).stop 2021-02-10T11:35:42.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.651 7f22c7771700 1 -- 172.21.2.112:0/2532143486 shutdown_connections 2021-02-10T11:35:42.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.651 7f22c7771700 1 --2- 172.21.2.112:0/2532143486 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f22c01b0a00 0x7f22c00cf320 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:42.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.651 7f22c7771700 1 --2- 172.21.2.112:0/2532143486 >> v2:172.21.2.120:3300/0 conn(0x7f22c00d1860 0x7f22c01b03e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:42.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.651 7f22c7771700 1 -- 172.21.2.112:0/2532143486 >> 172.21.2.112:0/2532143486 conn(0x7f22c00ce440 msgr2=0x7f22c00d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:42.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.651 7f22c7771700 1 -- 172.21.2.112:0/2532143486 shutdown_connections 2021-02-10T11:35:42.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.651 7f22c7771700 1 -- 172.21.2.112:0/2532143486 wait complete. 2021-02-10T11:35:42.657 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.665 7f22c7771700 1 Processor -- start 2021-02-10T11:35:42.658 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.665 7f22c7771700 1 -- start start 2021-02-10T11:35:42.658 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.666 7f22c7771700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f22c009eaf0 0x7f22c00ebdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:42.658 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.666 7f22c7771700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f22c00b72e0 0x7f22c009f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:42.658 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.666 7f22c7771700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f22c00b7dd0 0x7f22c00b81d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:42.658 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.666 7f22c7771700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22c00b7850 con 0x7f22c00b72e0 2021-02-10T11:35:42.658 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.666 7f22c7771700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22c009eef0 con 0x7f22c00b7dd0 2021-02-10T11:35:42.658 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.666 7f22c7771700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22c00ec3f0 con 0x7f22c009eaf0 2021-02-10T11:35:42.658 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.666 7f22c4d8c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f22c00b72e0 0x7f22c009f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:42.658 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.666 7f22c558d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f22c009eaf0 0x7f22c00ebdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:42.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.666 7f22c4d8c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f22c00b72e0 0x7f22c009f1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:51390/0 (socket says 172.21.2.112:51390) 2021-02-10T11:35:42.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.666 7f22c4d8c700 1 -- 172.21.2.112:0/2778516034 learned_addr learned my addr 172.21.2.112:0/2778516034 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:35:42.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.666 7f22c5d8e700 1 --2- 172.21.2.112:0/2778516034 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f22c00b7dd0 0x7f22c00b81d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:42.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.667 7f22c4d8c700 1 -- 172.21.2.112:0/2778516034 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f22c009eaf0 msgr2=0x7f22c00ebdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:42.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.667 7f22c4d8c700 1 --2- 172.21.2.112:0/2778516034 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f22c009eaf0 0x7f22c00ebdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:42.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.667 7f22c4d8c700 1 -- 172.21.2.112:0/2778516034 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f22c00b7dd0 msgr2=0x7f22c00b81d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:42.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.667 7f22c4d8c700 1 --2- 172.21.2.112:0/2778516034 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f22c00b7dd0 0x7f22c00b81d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:42.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.667 7f22c4d8c700 1 -- 172.21.2.112:0/2778516034 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22c009f7c0 con 0x7f22c00b72e0 2021-02-10T11:35:42.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.667 7f22c4d8c700 1 --2- 172.21.2.112:0/2778516034 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f22c00b72e0 0x7f22c009f1a0 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 rx=0x7f22bc009f40 tx=0x7f22bc00bd50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:42.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.667 7f22b5ffb700 1 -- 172.21.2.112:0/2778516034 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f22bc006240 con 0x7f22c00b72e0 2021-02-10T11:35:42.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.667 7f22b5ffb700 1 -- 172.21.2.112:0/2778516034 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22bc0065c0 con 0x7f22c00b72e0 2021-02-10T11:35:42.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.667 7f22b5ffb700 1 -- 172.21.2.112:0/2778516034 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f22bc0058e0 con 0x7f22c00b72e0 2021-02-10T11:35:42.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.667 7f22c7771700 1 -- 172.21.2.112:0/2778516034 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22c009f7c0 con 0x7f22c00b72e0 2021-02-10T11:35:42.660 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.667 7f22c7771700 1 -- 172.21.2.112:0/2778516034 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22c009f7c0 con 0x7f22c00b72e0 2021-02-10T11:35:42.660 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.668 7f22b5ffb700 1 -- 172.21.2.112:0/2778516034 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f22bc005660 con 0x7f22c00b72e0 2021-02-10T11:35:42.660 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.668 7f22b5ffb700 1 --2- 172.21.2.112:0/2778516034 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f22a0018730 0x7f22a001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:42.660 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.668 7f22b5ffb700 1 -- 172.21.2.112:0/2778516034 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f22bc0329b0 con 0x7f22c00b72e0 2021-02-10T11:35:42.660 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.668 7f22c558d700 1 --2- 172.21.2.112:0/2778516034 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f22a0018730 0x7f22a001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:42.660 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.668 7f22c558d700 1 --2- 172.21.2.112:0/2778516034 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f22a0018730 0x7f22a001abc0 crc :-1 s=READY pgs=729 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:42.684 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.698 7f22c7771700 1 -- 172.21.2.112:0/2778516034 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22c00ebba0 con 0x7f22c00b72e0 2021-02-10T11:35:42.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.700 7f22b5ffb700 1 -- 172.21.2.112:0/2778516034 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f22bc006ca0 con 0x7f22c00b72e0 2021-02-10T11:35:42.833 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.846 7f22c7771700 1 -- 172.21.2.112:0/2778516034 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "config rm", "who": "client.alice", "name": "debug_objecter"} v 0) v1 -- 0x7f22c00f0b00 con 0x7f22c00b72e0 2021-02-10T11:35:42.836 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.850 7f22b5ffb700 1 -- 172.21.2.112:0/2778516034 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "client.alice", "name": "debug_objecter"}]=0 v18) v1 ==== 106+0+0 (secure 0 0 0) 0x7f22bc005660 con 0x7f22c00b72e0 2021-02-10T11:35:42.841 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.855 7f22c7771700 1 -- 172.21.2.112:0/2778516034 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f22a0018730 msgr2=0x7f22a001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:42.842 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.855 7f22c7771700 1 --2- 172.21.2.112:0/2778516034 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f22a0018730 0x7f22a001abc0 crc :-1 s=READY pgs=729 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:42.842 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.855 7f22c7771700 1 -- 172.21.2.112:0/2778516034 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f22c00b72e0 msgr2=0x7f22c009f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:42.842 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.855 7f22c7771700 1 --2- 172.21.2.112:0/2778516034 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f22c00b72e0 0x7f22c009f1a0 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 rx=0x7f22bc009f40 tx=0x7f22bc00bd50).stop 2021-02-10T11:35:42.842 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.855 7f22c7771700 1 -- 172.21.2.112:0/2778516034 shutdown_connections 2021-02-10T11:35:42.842 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.855 7f22c7771700 1 --2- 172.21.2.112:0/2778516034 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f22a0018730 0x7f22a001abc0 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:42.842 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.855 7f22c7771700 1 --2- 172.21.2.112:0/2778516034 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f22c00b7dd0 0x7f22c00b81d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:42.842 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.855 7f22c7771700 1 --2- 172.21.2.112:0/2778516034 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f22c00b72e0 0x7f22c009f1a0 unknown :-1 s=CLOSED pgs=1507 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:42.843 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.855 7f22c7771700 1 --2- 172.21.2.112:0/2778516034 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f22c009eaf0 0x7f22c00ebdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:42.843 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.855 7f22c7771700 1 -- 172.21.2.112:0/2778516034 >> 172.21.2.112:0/2778516034 conn(0x7f22c00b8df0 msgr2=0x7f22c00add50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:42.843 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.855 7f22c7771700 1 -- 172.21.2.112:0/2778516034 shutdown_connections 2021-02-10T11:35:42.843 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:42.856 7f22c7771700 1 -- 172.21.2.112:0/2778516034 wait complete. 2021-02-10T11:35:42.851 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config rm mon mon_allow_pool_delete 2021-02-10T11:35:43.281 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.278 7fefdcb91700 1 Processor -- start 2021-02-10T11:35:43.281 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.278 7fefdcb91700 1 -- start start 2021-02-10T11:35:43.282 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.278 7fefdcb91700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fefd80d1860 0x7fefd81b03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:43.282 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.278 7fefdcb91700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fefd80f0640 0x7fefd80cfb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:43.282 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.278 7fefdcb91700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefd80f0b00 con 0x7fefd81b0a00 2021-02-10T11:35:43.282 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.278 7fefdcb91700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fefd80e5ec0 con 0x7fefd80f0640 2021-02-10T11:35:43.282 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.278 7fefdcb91700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fefd80d1c60 con 0x7fefd80d1860 2021-02-10T11:35:43.282 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.279 7fefd705f700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fefd80f0640 0x7fefd80cfb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:43.282 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.279 7fefd605d700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fefd81b0a00 0x7fefd80cf320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:50858/0 (socket says 172.21.2.112:50858) 2021-02-10T11:35:43.282 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.279 7fefd605d700 1 -- 172.21.2.112:0/3584264609 learned_addr learned my addr 172.21.2.112:0/3584264609 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:35:43.282 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.279 7fefd705f700 1 -- 172.21.2.112:0/3584264609 >> v2:172.21.2.120:3300/0 conn(0x7fefd80d1860 msgr2=0x7fefd81b03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:43.282 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.279 7fefd705f700 1 --2- 172.21.2.112:0/3584264609 >> v2:172.21.2.120:3300/0 conn(0x7fefd80d1860 0x7fefd81b03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:43.283 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.279 7fefd705f700 1 -- 172.21.2.112:0/3584264609 >> v1:172.21.2.112:6789/0 conn(0x7fefd81b0a00 legacy=0x7fefd80cf320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:43.283 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.279 7fefd705f700 1 -- 172.21.2.112:0/3584264609 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefd80d1e80 con 0x7fefd80f0640 2021-02-10T11:35:43.283 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.279 7fefd705f700 1 --2- 172.21.2.112:0/3584264609 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fefd80f0640 0x7fefd80cfb10 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 rx=0x7fefd0006e30 tx=0x7fefd0009de0).ready entity=mon.0 client_cookie=21e97a65c71ff7ec server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:43.283 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.279 7fefd585c700 1 -- 172.21.2.112:0/3584264609 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fefd000b300 con 0x7fefd80f0640 2021-02-10T11:35:43.283 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.279 7fefd585c700 1 -- 172.21.2.112:0/3584264609 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefd000b680 con 0x7fefd80f0640 2021-02-10T11:35:43.283 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.279 7fefd585c700 1 -- 172.21.2.112:0/3584264609 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fefd000bba0 con 0x7fefd80f0640 2021-02-10T11:35:43.283 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.280 7fefdcb91700 1 -- 172.21.2.112:0/3584264609 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fefd80f0640 msgr2=0x7fefd80cfb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:43.283 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.280 7fefdcb91700 1 --2- 172.21.2.112:0/3584264609 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fefd80f0640 0x7fefd80cfb10 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 rx=0x7fefd0006e30 tx=0x7fefd0009de0).stop 2021-02-10T11:35:43.283 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.280 7fefdcb91700 1 -- 172.21.2.112:0/3584264609 shutdown_connections 2021-02-10T11:35:43.283 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.280 7fefdcb91700 1 --2- 172.21.2.112:0/3584264609 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fefd80f0640 0x7fefd80cfb10 unknown :-1 s=CLOSED pgs=1509 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:43.283 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.280 7fefdcb91700 1 --2- 172.21.2.112:0/3584264609 >> v2:172.21.2.120:3300/0 conn(0x7fefd80d1860 0x7fefd81b03e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:43.283 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.280 7fefdcb91700 1 -- 172.21.2.112:0/3584264609 >> 172.21.2.112:0/3584264609 conn(0x7fefd80ce440 msgr2=0x7fefd80d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:43.284 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.280 7fefdcb91700 1 -- 172.21.2.112:0/3584264609 shutdown_connections 2021-02-10T11:35:43.284 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.280 7fefdcb91700 1 -- 172.21.2.112:0/3584264609 wait complete. 2021-02-10T11:35:43.284 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.295 7fefdcb91700 1 Processor -- start 2021-02-10T11:35:43.284 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.295 7fefdcb91700 1 -- start start 2021-02-10T11:35:43.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.295 7fefdcb91700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fefd80a2ac0 0x7fefd818e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:43.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.295 7fefdcb91700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fefd80bafe0 0x7fefd818e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:43.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.295 7fefdcb91700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fefd8090fe0 0x7fefd80913e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:43.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.295 7fefdcb91700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefd8091a00 con 0x7fefd80a2ac0 2021-02-10T11:35:43.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.295 7fefdcb91700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefd80babd0 con 0x7fefd8090fe0 2021-02-10T11:35:43.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.295 7fefdcb91700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefd80eef10 con 0x7fefd80bafe0 2021-02-10T11:35:43.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.295 7fefd685e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fefd80a2ac0 0x7fefd818e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:43.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.295 7fefd685e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fefd80a2ac0 0x7fefd818e0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:51400/0 (socket says 172.21.2.112:51400) 2021-02-10T11:35:43.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.295 7fefd685e700 1 -- 172.21.2.112:0/2442172696 learned_addr learned my addr 172.21.2.112:0/2442172696 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:35:43.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.296 7fefd685e700 1 -- 172.21.2.112:0/2442172696 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fefd80bafe0 msgr2=0x7fefd818e6d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:35:43.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.296 7fefd685e700 1 --2- 172.21.2.112:0/2442172696 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fefd80bafe0 0x7fefd818e6d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:43.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.296 7fefd685e700 1 -- 172.21.2.112:0/2442172696 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fefd8090fe0 msgr2=0x7fefd80913e0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:35:43.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.296 7fefd685e700 1 --2- 172.21.2.112:0/2442172696 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fefd8090fe0 0x7fefd80913e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:43.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.296 7fefd685e700 1 -- 172.21.2.112:0/2442172696 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefd80ef130 con 0x7fefd80a2ac0 2021-02-10T11:35:43.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.296 7fefd685e700 1 --2- 172.21.2.112:0/2442172696 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fefd80a2ac0 0x7fefd818e0b0 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 rx=0x7fefc0006f00 tx=0x7fefc000b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:43.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.296 7fefcf7fe700 1 -- 172.21.2.112:0/2442172696 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fefc000bd30 con 0x7fefd80a2ac0 2021-02-10T11:35:43.287 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.296 7fefcf7fe700 1 -- 172.21.2.112:0/2442172696 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefc0006c20 con 0x7fefd80a2ac0 2021-02-10T11:35:43.287 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.296 7fefcf7fe700 1 -- 172.21.2.112:0/2442172696 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fefc0002020 con 0x7fefd80a2ac0 2021-02-10T11:35:43.287 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.296 7fefdcb91700 1 -- 172.21.2.112:0/2442172696 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefd80ef130 con 0x7fefd80a2ac0 2021-02-10T11:35:43.287 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.296 7fefdcb91700 1 -- 172.21.2.112:0/2442172696 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefd80ef130 con 0x7fefd80a2ac0 2021-02-10T11:35:43.287 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.297 7fefcf7fe700 1 -- 172.21.2.112:0/2442172696 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fefc0005310 con 0x7fefd80a2ac0 2021-02-10T11:35:43.287 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.297 7fefcf7fe700 1 --2- 172.21.2.112:0/2442172696 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fefbc0186c0 0x7fefbc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:43.287 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.297 7fefcf7fe700 1 -- 172.21.2.112:0/2442172696 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fefc0032550 con 0x7fefd80a2ac0 2021-02-10T11:35:43.287 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.297 7fefd605d700 1 --2- 172.21.2.112:0/2442172696 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fefbc0186c0 0x7fefbc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:43.287 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.298 7fefd605d700 1 --2- 172.21.2.112:0/2442172696 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fefbc0186c0 0x7fefbc01abc0 crc :-1 s=READY pgs=730 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:43.315 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.328 7fefdcb91700 1 -- 172.21.2.112:0/2442172696 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefd8091a00 con 0x7fefd80a2ac0 2021-02-10T11:35:43.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.331 7fefcf7fe700 1 -- 172.21.2.112:0/2442172696 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fefc0022820 con 0x7fefd80a2ac0 2021-02-10T11:35:43.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.480 7fefdcb91700 1 -- 172.21.2.112:0/2442172696 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fefd8091a00 con 0x7fefd80a2ac0 2021-02-10T11:35:43.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.485 7fefcf7fe700 1 -- 172.21.2.112:0/2442172696 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v19) v1 ==== 104+0+0 (secure 0 0 0) 0x7fefc0022820 con 0x7fefd80a2ac0 2021-02-10T11:35:43.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.489 7fefdcb91700 1 -- 172.21.2.112:0/2442172696 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fefbc0186c0 msgr2=0x7fefbc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:43.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.489 7fefdcb91700 1 --2- 172.21.2.112:0/2442172696 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fefbc0186c0 0x7fefbc01abc0 crc :-1 s=READY pgs=730 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:43.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.489 7fefdcb91700 1 -- 172.21.2.112:0/2442172696 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fefd80a2ac0 msgr2=0x7fefd818e0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:43.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.489 7fefdcb91700 1 --2- 172.21.2.112:0/2442172696 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fefd80a2ac0 0x7fefd818e0b0 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 rx=0x7fefc0006f00 tx=0x7fefc000b940).stop 2021-02-10T11:35:43.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.490 7fefdcb91700 1 -- 172.21.2.112:0/2442172696 shutdown_connections 2021-02-10T11:35:43.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.490 7fefdcb91700 1 --2- 172.21.2.112:0/2442172696 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fefbc0186c0 0x7fefbc01abc0 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:43.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.490 7fefdcb91700 1 --2- 172.21.2.112:0/2442172696 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fefd8090fe0 0x7fefd80913e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:43.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.490 7fefdcb91700 1 --2- 172.21.2.112:0/2442172696 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fefd80bafe0 0x7fefd818e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:43.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.490 7fefdcb91700 1 --2- 172.21.2.112:0/2442172696 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fefd80a2ac0 0x7fefd818e0b0 unknown :-1 s=CLOSED pgs=1510 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:43.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.490 7fefdcb91700 1 -- 172.21.2.112:0/2442172696 >> 172.21.2.112:0/2442172696 conn(0x7fefd80bc040 msgr2=0x7fefd80ee230 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:43.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.490 7fefdcb91700 1 -- 172.21.2.112:0/2442172696 shutdown_connections 2021-02-10T11:35:43.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.490 7fefdcb91700 1 -- 172.21.2.112:0/2442172696 wait complete. 2021-02-10T11:35:43.485 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestVolumes.test_authorize_deauthorize_subvolume' 2021-02-10T11:35:43.910 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.907 7fd5c6ceb700 1 Processor -- start 2021-02-10T11:35:43.910 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.907 7fd5c6ceb700 1 -- start start 2021-02-10T11:35:43.910 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.907 7fd5c6ceb700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fd5c00d1870 0x7fd5c01b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:43.910 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.907 7fd5c6ceb700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd5c01b0a10 0x7fd5c00cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:43.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.907 7fd5c6ceb700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5c00f0b10 con 0x7fd5c00d1c70 2021-02-10T11:35:43.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.907 7fd5c6ceb700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5c00e5ed0 con 0x7fd5c00d1870 2021-02-10T11:35:43.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.907 7fd5c6ceb700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5c00f0650 con 0x7fd5c01b0a10 2021-02-10T11:35:43.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.908 7fd5c4b07700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fd5c00d1870 0x7fd5c01b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:43.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.908 7fd5c5308700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fd5c00d1c70 0x7fd5c00cf950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:50874/0 (socket says 172.21.2.112:50874) 2021-02-10T11:35:43.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.908 7fd5c5308700 1 -- 172.21.2.112:0/3738709610 learned_addr learned my addr 172.21.2.112:0/3738709610 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:35:43.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.908 7fd5bffff700 1 --2- 172.21.2.112:0/3738709610 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd5c01b0a10 0x7fd5c00cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:43.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.908 7fd5c4b07700 1 -- 172.21.2.112:0/3738709610 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd5c01b0a10 msgr2=0x7fd5c00cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:43.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.908 7fd5c4b07700 1 --2- 172.21.2.112:0/3738709610 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd5c01b0a10 0x7fd5c00cf330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:43.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.908 7fd5c4b07700 1 -- 172.21.2.112:0/3738709610 >> v1:172.21.2.112:6789/0 conn(0x7fd5c00d1c70 legacy=0x7fd5c00cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:43.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.908 7fd5c4b07700 1 -- 172.21.2.112:0/3738709610 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5c00f0870 con 0x7fd5c00d1870 2021-02-10T11:35:43.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.908 7fd5c4b07700 1 --2- 172.21.2.112:0/3738709610 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd5c00d1870 0x7fd5c01b03f0 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 rx=0x7fd5a8006650 tx=0x7fd5a8009a80).ready entity=mon.0 client_cookie=a424988e2ca7789b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:43.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.909 7fd5bf7fe700 1 -- 172.21.2.112:0/3738709610 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd5a800c0c0 con 0x7fd5c00d1870 2021-02-10T11:35:43.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.909 7fd5bf7fe700 1 -- 172.21.2.112:0/3738709610 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5a800c340 con 0x7fd5c00d1870 2021-02-10T11:35:43.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.909 7fd5bf7fe700 1 -- 172.21.2.112:0/3738709610 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd5a800c860 con 0x7fd5c00d1870 2021-02-10T11:35:43.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.909 7fd5c6ceb700 1 -- 172.21.2.112:0/3738709610 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd5c00d1870 msgr2=0x7fd5c01b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:43.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.909 7fd5c6ceb700 1 --2- 172.21.2.112:0/3738709610 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd5c00d1870 0x7fd5c01b03f0 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 rx=0x7fd5a8006650 tx=0x7fd5a8009a80).stop 2021-02-10T11:35:43.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.909 7fd5c6ceb700 1 -- 172.21.2.112:0/3738709610 shutdown_connections 2021-02-10T11:35:43.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.909 7fd5c6ceb700 1 --2- 172.21.2.112:0/3738709610 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd5c01b0a10 0x7fd5c00cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:43.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.909 7fd5c6ceb700 1 --2- 172.21.2.112:0/3738709610 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd5c00d1870 0x7fd5c01b03f0 unknown :-1 s=CLOSED pgs=1512 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:43.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.909 7fd5c6ceb700 1 -- 172.21.2.112:0/3738709610 >> 172.21.2.112:0/3738709610 conn(0x7fd5c00ce450 msgr2=0x7fd5c00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:43.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.909 7fd5c6ceb700 1 -- 172.21.2.112:0/3738709610 shutdown_connections 2021-02-10T11:35:43.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.909 7fd5c6ceb700 1 -- 172.21.2.112:0/3738709610 wait complete. 2021-02-10T11:35:43.914 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.924 7fd5c6ceb700 1 Processor -- start 2021-02-10T11:35:43.914 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.924 7fd5c6ceb700 1 -- start start 2021-02-10T11:35:43.914 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.924 7fd5c6ceb700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd5c018d870 0x7fd5c00b19b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:43.914 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.924 7fd5c6ceb700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd5c00b1fd0 0x7fd5c0099ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:43.914 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.924 7fd5c6ceb700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd5c00b23d0 0x7fd5c009a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:43.914 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.924 7fd5c6ceb700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5c018dc70 con 0x7fd5c00b23d0 2021-02-10T11:35:43.914 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.924 7fd5c6ceb700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5c0199e60 con 0x7fd5c00b1fd0 2021-02-10T11:35:43.914 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.924 7fd5c6ceb700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5c009ab30 con 0x7fd5c018d870 2021-02-10T11:35:43.914 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.925 7fd5c5308700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd5c00b23d0 0x7fd5c009a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:43.914 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.925 7fd5c5308700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd5c00b23d0 0x7fd5c009a510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:51416/0 (socket says 172.21.2.112:51416) 2021-02-10T11:35:43.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.925 7fd5c5308700 1 -- 172.21.2.112:0/2737954993 learned_addr learned my addr 172.21.2.112:0/2737954993 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:35:43.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.925 7fd5c4b07700 1 --2- 172.21.2.112:0/2737954993 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd5c018d870 0x7fd5c00b19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:43.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.925 7fd5c5308700 1 -- 172.21.2.112:0/2737954993 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd5c018d870 msgr2=0x7fd5c00b19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:43.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.925 7fd5bffff700 1 --2- 172.21.2.112:0/2737954993 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd5c00b1fd0 0x7fd5c0099ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:43.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.925 7fd5c5308700 1 --2- 172.21.2.112:0/2737954993 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd5c018d870 0x7fd5c00b19b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:43.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.925 7fd5c5308700 1 -- 172.21.2.112:0/2737954993 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd5c00b1fd0 msgr2=0x7fd5c0099ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:43.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.925 7fd5c5308700 1 --2- 172.21.2.112:0/2737954993 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd5c00b1fd0 0x7fd5c0099ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:43.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.925 7fd5c5308700 1 -- 172.21.2.112:0/2737954993 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5c009ad50 con 0x7fd5c00b23d0 2021-02-10T11:35:43.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.925 7fd5c4b07700 1 --2- 172.21.2.112:0/2737954993 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd5c018d870 0x7fd5c00b19b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:35:43.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.925 7fd5bffff700 1 --2- 172.21.2.112:0/2737954993 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd5c00b1fd0 0x7fd5c0099ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:35:43.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.925 7fd5c5308700 1 --2- 172.21.2.112:0/2737954993 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd5c00b23d0 0x7fd5c009a510 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 rx=0x7fd5b800a940 tx=0x7fd5b800ace0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:43.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.925 7fd5bd7fa700 1 -- 172.21.2.112:0/2737954993 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd5b8001270 con 0x7fd5c00b23d0 2021-02-10T11:35:43.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.925 7fd5bd7fa700 1 -- 172.21.2.112:0/2737954993 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5b8000d30 con 0x7fd5c00b23d0 2021-02-10T11:35:43.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.925 7fd5c6ceb700 1 -- 172.21.2.112:0/2737954993 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5c009ad50 con 0x7fd5c00b23d0 2021-02-10T11:35:43.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.925 7fd5bd7fa700 1 -- 172.21.2.112:0/2737954993 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd5b8004bb0 con 0x7fd5c00b23d0 2021-02-10T11:35:43.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.925 7fd5c6ceb700 1 -- 172.21.2.112:0/2737954993 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5c009ad50 con 0x7fd5c00b23d0 2021-02-10T11:35:43.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.926 7fd5bd7fa700 1 -- 172.21.2.112:0/2737954993 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd5b800ae20 con 0x7fd5c00b23d0 2021-02-10T11:35:43.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.926 7fd5bd7fa700 1 --2- 172.21.2.112:0/2737954993 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd5a40186d0 0x7fd5a401abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:43.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.926 7fd5bd7fa700 1 -- 172.21.2.112:0/2737954993 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fd5b8032b10 con 0x7fd5c00b23d0 2021-02-10T11:35:43.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.926 7fd5c4b07700 1 --2- 172.21.2.112:0/2737954993 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd5a40186d0 0x7fd5a401abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:43.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.927 7fd5c4b07700 1 --2- 172.21.2.112:0/2737954993 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd5a40186d0 0x7fd5a401abd0 crc :-1 s=READY pgs=731 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:43.944 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.957 7fd5c6ceb700 1 -- 172.21.2.112:0/2737954993 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5c00999c0 con 0x7fd5c00b23d0 2021-02-10T11:35:43.946 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:43.960 7fd5bd7fa700 1 -- 172.21.2.112:0/2737954993 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd5b8007cb0 con 0x7fd5c00b23d0 2021-02-10T11:35:44.077 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:44.091 7fd5c6ceb700 1 -- 172.21.2.112:0/2737954993 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_authorize_deauthorize_subvolume"]} v 0) v1 -- 0x7fd5c00999c0 con 0x7fd5c00b23d0 2021-02-10T11:35:44.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:44.104 7fd5bd7fa700 1 -- 172.21.2.112:0/2737954993 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_authorize_deauthorize_subvolume"]}]=0 v379) v1 ==== 153+0+0 (secure 0 0 0) 0x7fd5b800ae20 con 0x7fd5c00b23d0 2021-02-10T11:35:44.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:44.108 7fd5c6ceb700 1 -- 172.21.2.112:0/2737954993 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd5a40186d0 msgr2=0x7fd5a401abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:44.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:44.108 7fd5c6ceb700 1 --2- 172.21.2.112:0/2737954993 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd5a40186d0 0x7fd5a401abd0 crc :-1 s=READY pgs=731 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:44.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:44.108 7fd5c6ceb700 1 -- 172.21.2.112:0/2737954993 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd5c00b23d0 msgr2=0x7fd5c009a510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:44.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:44.108 7fd5c6ceb700 1 --2- 172.21.2.112:0/2737954993 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd5c00b23d0 0x7fd5c009a510 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 rx=0x7fd5b800a940 tx=0x7fd5b800ace0).stop 2021-02-10T11:35:44.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:44.108 7fd5c6ceb700 1 -- 172.21.2.112:0/2737954993 shutdown_connections 2021-02-10T11:35:44.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:44.108 7fd5c6ceb700 1 --2- 172.21.2.112:0/2737954993 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd5a40186d0 0x7fd5a401abd0 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:44.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:44.108 7fd5c6ceb700 1 --2- 172.21.2.112:0/2737954993 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd5c00b1fd0 0x7fd5c0099ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:44.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:44.108 7fd5c6ceb700 1 --2- 172.21.2.112:0/2737954993 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd5c00b23d0 0x7fd5c009a510 unknown :-1 s=CLOSED pgs=1513 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:44.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:44.108 7fd5c6ceb700 1 --2- 172.21.2.112:0/2737954993 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd5c018d870 0x7fd5c00b19b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:44.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:44.108 7fd5c6ceb700 1 -- 172.21.2.112:0/2737954993 >> 172.21.2.112:0/2737954993 conn(0x7fd5c00f0550 msgr2=0x7fd5c0191730 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:44.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:44.108 7fd5c6ceb700 1 -- 172.21.2.112:0/2737954993 shutdown_connections 2021-02-10T11:35:44.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:44.108 7fd5c6ceb700 1 -- 172.21.2.112:0/2737954993 wait complete. 2021-02-10T11:35:44.103 INFO:tasks.cephfs_test_runner:test_authorize_deauthorize_subvolume (tasks.cephfs.test_volumes.TestVolumes) ... ok 2021-02-10T11:35:44.103 INFO:tasks.cephfs_test_runner:Starting test: test_clone_subvolume_info (tasks.cephfs.test_volumes.TestVolumes) 2021-02-10T11:35:44.104 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestVolumes.test_clone_subvolume_info' 2021-02-10T11:35:44.531 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:44.528 7f2230dd5700 1 Processor -- start 2021-02-10T11:35:44.531 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:44.528 7f2230dd5700 1 -- start start 2021-02-10T11:35:44.531 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:44.528 7f2230dd5700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f222c0d1870 0x7f222c1b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:44.531 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:44.529 7f2230dd5700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f222c0f0650 0x7f222c0cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:44.531 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:44.529 7f2230dd5700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f222c0f0b10 con 0x7f222c1b0a10 2021-02-10T11:35:44.531 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:44.529 7f2230dd5700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f222c0e5ed0 con 0x7f222c0f0650 2021-02-10T11:35:44.531 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:44.529 7f2230dd5700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f222c0d1c70 con 0x7f222c0d1870 2021-02-10T11:35:44.531 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:44.529 7f222b314700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f222c0f0650 0x7f222c0cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:44.531 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:44.529 7f222b314700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f222c0f0650 0x7f222c0cfb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:51426/0 (socket says 172.21.2.112:51426) 2021-02-10T11:35:44.532 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:44.529 7f222b314700 1 -- 172.21.2.112:0/3959025795 learned_addr learned my addr 172.21.2.112:0/3959025795 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:35:44.532 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:44.529 7f222a312700 1 --1- 172.21.2.112:0/3959025795 >> v1:172.21.2.112:6789/0 conn(0x7f222c1b0a10 0x7f222c0cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:50886/0 (socket says 172.21.2.112:50886) 2021-02-10T11:35:44.532 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:44.529 7f222b314700 1 -- 172.21.2.112:0/3959025795 >> v2:172.21.2.120:3300/0 conn(0x7f222c0d1870 msgr2=0x7f222c1b03f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:35:44.532 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:44.529 7f222b314700 1 --2- 172.21.2.112:0/3959025795 >> v2:172.21.2.120:3300/0 conn(0x7f222c0d1870 0x7f222c1b03f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:44.532 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:44.529 7f222b314700 1 -- 172.21.2.112:0/3959025795 >> v1:172.21.2.112:6789/0 conn(0x7f222c1b0a10 legacy=0x7f222c0cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:44.532 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:44.529 7f222b314700 1 -- 172.21.2.112:0/3959025795 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f222c0d1e90 con 0x7f222c0f0650 2021-02-10T11:35:44.532 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:44.529 7f222b314700 1 --2- 172.21.2.112:0/3959025795 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f222c0f0650 0x7f222c0cfb20 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 rx=0x7f2224006f00 tx=0x7f2224009e90).ready entity=mon.0 client_cookie=244bdf08511038d7 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:44.532 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:44.529 7f2229b11700 1 -- 172.21.2.112:0/3959025795 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f222400b300 con 0x7f222c0f0650 2021-02-10T11:35:44.532 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:44.529 7f2229b11700 1 -- 172.21.2.112:0/3959025795 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f222400b600 con 0x7f222c0f0650 2021-02-10T11:35:44.532 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:44.529 7f2229b11700 1 -- 172.21.2.112:0/3959025795 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f222400bae0 con 0x7f222c0f0650 2021-02-10T11:35:44.532 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:44.529 7f2230dd5700 1 -- 172.21.2.112:0/3959025795 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f222c0f0650 msgr2=0x7f222c0cfb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:44.532 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:44.529 7f2230dd5700 1 --2- 172.21.2.112:0/3959025795 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f222c0f0650 0x7f222c0cfb20 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 rx=0x7f2224006f00 tx=0x7f2224009e90).stop 2021-02-10T11:35:44.533 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:44.530 7f2230dd5700 1 -- 172.21.2.112:0/3959025795 shutdown_connections 2021-02-10T11:35:44.533 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:44.530 7f2230dd5700 1 --2- 172.21.2.112:0/3959025795 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f222c0f0650 0x7f222c0cfb20 unknown :-1 s=CLOSED pgs=1515 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:44.533 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:44.530 7f2230dd5700 1 --2- 172.21.2.112:0/3959025795 >> v2:172.21.2.120:3300/0 conn(0x7f222c0d1870 0x7f222c1b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:44.533 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:44.530 7f2230dd5700 1 -- 172.21.2.112:0/3959025795 >> 172.21.2.112:0/3959025795 conn(0x7f222c0ce450 msgr2=0x7f222c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:44.533 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:44.530 7f2230dd5700 1 -- 172.21.2.112:0/3959025795 shutdown_connections 2021-02-10T11:35:44.533 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:44.530 7f2230dd5700 1 -- 172.21.2.112:0/3959025795 wait complete. 2021-02-10T11:35:44.535 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:44.544 7f2230dd5700 1 Processor -- start 2021-02-10T11:35:44.535 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:44.545 7f2230dd5700 1 -- start start 2021-02-10T11:35:44.535 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:44.545 7f2230dd5700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f222c1b7cf0 0x7f222c19cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:44.535 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:44.545 7f2230dd5700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f222c1b80f0 0x7f222c1b84f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:44.535 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:44.545 7f2230dd5700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f222c19da20 0x7f222c19de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:44.535 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:44.545 7f2230dd5700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f222c0a77c0 con 0x7f222c1b80f0 2021-02-10T11:35:44.535 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:44.545 7f2230dd5700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f222c19d4a0 con 0x7f222c1b7cf0 2021-02-10T11:35:44.535 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:44.545 7f2230dd5700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f222c0954f0 con 0x7f222c19da20 2021-02-10T11:35:44.535 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:44.545 7f222a312700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f222c1b80f0 0x7f222c1b84f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:44.536 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:44.545 7f222a312700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f222c1b80f0 0x7f222c1b84f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:51430/0 (socket says 172.21.2.112:51430) 2021-02-10T11:35:44.536 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:44.545 7f222a312700 1 -- 172.21.2.112:0/823622467 learned_addr learned my addr 172.21.2.112:0/823622467 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:35:44.536 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:44.546 7f222a312700 1 -- 172.21.2.112:0/823622467 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f222c19da20 msgr2=0x7f222c19de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:44.536 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:44.546 7f222a312700 1 --2- 172.21.2.112:0/823622467 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f222c19da20 0x7f222c19de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:44.536 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:44.546 7f222a312700 1 -- 172.21.2.112:0/823622467 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f222c1b7cf0 msgr2=0x7f222c19cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:44.536 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:44.546 7f222a312700 1 --2- 172.21.2.112:0/823622467 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f222c1b7cf0 0x7f222c19cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:44.536 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:44.546 7f222a312700 1 -- 172.21.2.112:0/823622467 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f222c1b8d60 con 0x7f222c1b80f0 2021-02-10T11:35:44.537 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:44.546 7f222a312700 1 --2- 172.21.2.112:0/823622467 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f222c1b80f0 0x7f222c1b84f0 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 rx=0x7f221c001c60 tx=0x7f221c00a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:44.537 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:44.546 7f22237fe700 1 -- 172.21.2.112:0/823622467 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f221c00acc0 con 0x7f222c1b80f0 2021-02-10T11:35:44.537 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:44.546 7f22237fe700 1 -- 172.21.2.112:0/823622467 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f221c004140 con 0x7f222c1b80f0 2021-02-10T11:35:44.537 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:44.546 7f22237fe700 1 -- 172.21.2.112:0/823622467 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f221c004620 con 0x7f222c1b80f0 2021-02-10T11:35:44.537 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:44.546 7f2230dd5700 1 -- 172.21.2.112:0/823622467 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f222c1b8d60 con 0x7f222c1b80f0 2021-02-10T11:35:44.537 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:44.546 7f2230dd5700 1 -- 172.21.2.112:0/823622467 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f222c19e730 con 0x7f222c1b80f0 2021-02-10T11:35:44.537 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:44.547 7f22237fe700 1 -- 172.21.2.112:0/823622467 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f221c004320 con 0x7f222c1b80f0 2021-02-10T11:35:44.537 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:44.547 7f22237fe700 1 --2- 172.21.2.112:0/823622467 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f22100186c0 0x7f221001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:44.537 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:44.547 7f22237fe700 1 -- 172.21.2.112:0/823622467 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f221c032510 con 0x7f222c1b80f0 2021-02-10T11:35:44.537 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:44.548 7f222ab13700 1 --2- 172.21.2.112:0/823622467 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f22100186c0 0x7f221001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:44.537 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:44.548 7f222ab13700 1 --2- 172.21.2.112:0/823622467 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f22100186c0 0x7f221001abc0 crc :-1 s=READY pgs=732 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:44.565 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:44.578 7f2230dd5700 1 -- 172.21.2.112:0/823622467 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f222c0ebc00 con 0x7f222c1b80f0 2021-02-10T11:35:44.567 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:44.581 7f22237fe700 1 -- 172.21.2.112:0/823622467 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f221c000dd0 con 0x7f222c1b80f0 2021-02-10T11:35:44.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:44.712 7f2230dd5700 1 -- 172.21.2.112:0/823622467 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_clone_subvolume_info"]} v 0) v1 -- 0x7f222c0f0b10 con 0x7f222c1b80f0 2021-02-10T11:35:45.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:45.106 7f22237fe700 1 -- 172.21.2.112:0/823622467 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_clone_subvolume_info"]}]=0 v380) v1 ==== 145+0+0 (secure 0 0 0) 0x7f221c004320 con 0x7f222c1b80f0 2021-02-10T11:35:45.117 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:45.130 7f2230dd5700 1 -- 172.21.2.112:0/823622467 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f22100186c0 msgr2=0x7f221001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:45.118 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:45.130 7f2230dd5700 1 --2- 172.21.2.112:0/823622467 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f22100186c0 0x7f221001abc0 crc :-1 s=READY pgs=732 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:45.118 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:45.130 7f2230dd5700 1 -- 172.21.2.112:0/823622467 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f222c1b80f0 msgr2=0x7f222c1b84f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:45.118 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:45.130 7f2230dd5700 1 --2- 172.21.2.112:0/823622467 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f222c1b80f0 0x7f222c1b84f0 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 rx=0x7f221c001c60 tx=0x7f221c00a8d0).stop 2021-02-10T11:35:45.118 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:45.130 7f2230dd5700 1 -- 172.21.2.112:0/823622467 shutdown_connections 2021-02-10T11:35:45.118 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:45.130 7f2230dd5700 1 --2- 172.21.2.112:0/823622467 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f22100186c0 0x7f221001abc0 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:45.118 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:45.131 7f2230dd5700 1 --2- 172.21.2.112:0/823622467 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f222c19da20 0x7f222c19de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:45.118 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:45.131 7f2230dd5700 1 --2- 172.21.2.112:0/823622467 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f222c1b80f0 0x7f222c1b84f0 unknown :-1 s=CLOSED pgs=1516 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:45.118 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:45.131 7f2230dd5700 1 --2- 172.21.2.112:0/823622467 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f222c1b7cf0 0x7f222c19cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:45.118 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:45.131 7f2230dd5700 1 -- 172.21.2.112:0/823622467 >> 172.21.2.112:0/823622467 conn(0x7f222c0ed810 msgr2=0x7f222c094980 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:45.118 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:45.131 7f2230dd5700 1 -- 172.21.2.112:0/823622467 shutdown_connections 2021-02-10T11:35:45.118 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:45.131 7f2230dd5700 1 -- 172.21.2.112:0/823622467 wait complete. 2021-02-10T11:35:45.126 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:35:45.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:45.553 7f44c6598700 1 Processor -- start 2021-02-10T11:35:45.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:45.553 7f44c6598700 1 -- start start 2021-02-10T11:35:45.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:45.554 7f44c6598700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f44c00d1870 0x7f44c01b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:45.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:45.554 7f44c6598700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f44c01b0ae0 0x7f44c0108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:45.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:45.554 7f44c6598700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44c0109a60 con 0x7f44c00d1c70 2021-02-10T11:35:45.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:45.554 7f44c6598700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44c00e5ef0 con 0x7f44c01b0ae0 2021-02-10T11:35:45.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:45.554 7f44c6598700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44c0109750 con 0x7f44c00d1870 2021-02-10T11:35:45.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:45.554 7f44bffff700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f44c00d1870 0x7f44c01b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:45.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:45.554 7f44bf7fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f44c01b0ae0 0x7f44c0108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:45.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:45.554 7f44bffff700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f44c00d1870 0x7f44c01b04c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3301/0 says I am v2:172.21.2.112:51118/0 (socket says 172.21.2.112:51118) 2021-02-10T11:35:45.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:45.554 7f44c4bb5700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7f44c00d1c70 0x7f44c0108f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:44456/0 (socket says 172.21.2.112:44456) 2021-02-10T11:35:45.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:45.554 7f44bffff700 1 -- 172.21.2.112:0/2637514433 learned_addr learned my addr 172.21.2.112:0/2637514433 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:35:45.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:45.554 7f44bf7fe700 1 -- 172.21.2.112:0/2637514433 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f44c00d1870 msgr2=0x7f44c01b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:45.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:45.554 7f44bf7fe700 1 --2- 172.21.2.112:0/2637514433 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f44c00d1870 0x7f44c01b04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:45.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:45.554 7f44bf7fe700 1 -- 172.21.2.112:0/2637514433 >> v1:172.21.2.120:6789/0 conn(0x7f44c00d1c70 legacy=0x7f44c0108f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:45.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:45.554 7f44bf7fe700 1 -- 172.21.2.112:0/2637514433 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44c0088070 con 0x7f44c01b0ae0 2021-02-10T11:35:45.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:45.555 7f44bf7fe700 1 --2- 172.21.2.112:0/2637514433 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f44c01b0ae0 0x7f44c0108940 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 rx=0x7f44b4006650 tx=0x7f44b4009a80).ready entity=mon.0 client_cookie=b6074f7edbcdbf9e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:45.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:45.555 7f44beffd700 1 -- 172.21.2.112:0/2637514433 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f44b40009b0 con 0x7f44c01b0ae0 2021-02-10T11:35:45.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:45.555 7f44beffd700 1 -- 172.21.2.112:0/2637514433 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44b400c0c0 con 0x7f44c01b0ae0 2021-02-10T11:35:45.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:45.555 7f44beffd700 1 -- 172.21.2.112:0/2637514433 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f44b400c5e0 con 0x7f44c01b0ae0 2021-02-10T11:35:45.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:45.555 7f44c6598700 1 -- 172.21.2.112:0/2637514433 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f44c01b0ae0 msgr2=0x7f44c0108940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:45.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:45.555 7f44c6598700 1 --2- 172.21.2.112:0/2637514433 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f44c01b0ae0 0x7f44c0108940 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 rx=0x7f44b4006650 tx=0x7f44b4009a80).stop 2021-02-10T11:35:45.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:45.556 7f44c6598700 1 -- 172.21.2.112:0/2637514433 shutdown_connections 2021-02-10T11:35:45.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:45.556 7f44c6598700 1 --2- 172.21.2.112:0/2637514433 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f44c01b0ae0 0x7f44c0108940 unknown :-1 s=CLOSED pgs=1517 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:45.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:45.556 7f44c6598700 1 --2- 172.21.2.112:0/2637514433 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f44c00d1870 0x7f44c01b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:45.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:45.556 7f44c6598700 1 -- 172.21.2.112:0/2637514433 >> 172.21.2.112:0/2637514433 conn(0x7f44c00ce420 msgr2=0x7f44c00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:45.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:45.556 7f44c6598700 1 -- 172.21.2.112:0/2637514433 shutdown_connections 2021-02-10T11:35:45.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:45.556 7f44c6598700 1 -- 172.21.2.112:0/2637514433 wait complete. 2021-02-10T11:35:45.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:45.570 7f44c6598700 1 Processor -- start 2021-02-10T11:35:45.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:45.570 7f44c6598700 1 -- start start 2021-02-10T11:35:45.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:45.571 7f44c6598700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f44c00963c0 0x7f44c00967c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:45.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:45.571 7f44c6598700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f44c00a63f0 0x7f44c00a67f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:45.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:45.571 7f44c6598700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f44c00a6e10 0x7f44c00a7210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:45.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:45.571 7f44c6598700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44c0087530 con 0x7f44c00a63f0 2021-02-10T11:35:45.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:45.571 7f44c6598700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44c019c1f0 con 0x7f44c00963c0 2021-02-10T11:35:45.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:45.571 7f44c6598700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44c00974b0 con 0x7f44c00a6e10 2021-02-10T11:35:45.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:45.571 7f44bf7fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f44c00a63f0 0x7f44c00a67f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:45.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:45.571 7f44bf7fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f44c00a63f0 0x7f44c00a67f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:51444/0 (socket says 172.21.2.112:51444) 2021-02-10T11:35:45.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:45.571 7f44bf7fe700 1 -- 172.21.2.112:0/256665395 learned_addr learned my addr 172.21.2.112:0/256665395 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:35:45.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:45.571 7f44bffff700 1 --2- 172.21.2.112:0/256665395 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f44c00963c0 0x7f44c00967c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:45.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:45.571 7f44c4bb5700 1 --2- 172.21.2.112:0/256665395 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f44c00a6e10 0x7f44c00a7210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:45.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:45.571 7f44bf7fe700 1 -- 172.21.2.112:0/256665395 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f44c00a6e10 msgr2=0x7f44c00a7210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:45.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:45.571 7f44bf7fe700 1 --2- 172.21.2.112:0/256665395 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f44c00a6e10 0x7f44c00a7210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:45.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:45.571 7f44bf7fe700 1 -- 172.21.2.112:0/256665395 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f44c00963c0 msgr2=0x7f44c00967c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:45.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:45.571 7f44bf7fe700 1 --2- 172.21.2.112:0/256665395 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f44c00963c0 0x7f44c00967c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:45.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:45.571 7f44bf7fe700 1 -- 172.21.2.112:0/256665395 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44c00976d0 con 0x7f44c00a63f0 2021-02-10T11:35:45.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:45.571 7f44c4bb5700 1 --2- 172.21.2.112:0/256665395 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f44c00a6e10 0x7f44c00a7210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:35:45.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:45.571 7f44bf7fe700 1 --2- 172.21.2.112:0/256665395 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f44c00a63f0 0x7f44c00a67f0 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 rx=0x7f44b4009630 tx=0x7f44b400b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:45.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:45.572 7f44bcff9700 1 -- 172.21.2.112:0/256665395 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f44b40093a0 con 0x7f44c00a63f0 2021-02-10T11:35:45.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:45.572 7f44bcff9700 1 -- 172.21.2.112:0/256665395 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44b400bd70 con 0x7f44c00a63f0 2021-02-10T11:35:45.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:45.572 7f44bcff9700 1 -- 172.21.2.112:0/256665395 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f44b4005560 con 0x7f44c00a63f0 2021-02-10T11:35:45.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:45.572 7f44c6598700 1 -- 172.21.2.112:0/256665395 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44c00976d0 con 0x7f44c00a63f0 2021-02-10T11:35:45.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:45.572 7f44c6598700 1 -- 172.21.2.112:0/256665395 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44c0097c60 con 0x7f44c00a63f0 2021-02-10T11:35:45.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:45.573 7f44bcff9700 1 -- 172.21.2.112:0/256665395 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f44b40052e0 con 0x7f44c00a63f0 2021-02-10T11:35:45.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:45.573 7f44bcff9700 1 --2- 172.21.2.112:0/256665395 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f44a00186c0 0x7f44a001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:45.563 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:45.573 7f44bcff9700 1 -- 172.21.2.112:0/256665395 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f44b4032560 con 0x7f44c00a63f0 2021-02-10T11:35:45.563 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:45.573 7f44bffff700 1 --2- 172.21.2.112:0/256665395 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f44a00186c0 0x7f44a001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:45.563 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:45.574 7f44bffff700 1 --2- 172.21.2.112:0/256665395 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f44a00186c0 0x7f44a001abc0 crc :-1 s=READY pgs=733 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:45.590 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:45.604 7f44c6598700 1 -- 172.21.2.112:0/256665395 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44c0082980 con 0x7f44c00a63f0 2021-02-10T11:35:45.593 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:45.606 7f44bcff9700 1 -- 172.21.2.112:0/256665395 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f44b4009340 con 0x7f44c00a63f0 2021-02-10T11:35:45.747 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:45.760 7f44c6598700 1 -- 172.21.2.112:0/256665395 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f44c0109a60 con 0x7f44c00a63f0 2021-02-10T11:35:45.750 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:45.764 7f44bcff9700 1 -- 172.21.2.112:0/256665395 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v20)=0 v20) v1 ==== 122+0+0 (secure 0 0 0) 0x7f44b40052e0 con 0x7f44c00a63f0 2021-02-10T11:35:45.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:45.768 7f44c6598700 1 -- 172.21.2.112:0/256665395 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f44a00186c0 msgr2=0x7f44a001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:45.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:45.768 7f44c6598700 1 --2- 172.21.2.112:0/256665395 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f44a00186c0 0x7f44a001abc0 crc :-1 s=READY pgs=733 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:45.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:45.768 7f44c6598700 1 -- 172.21.2.112:0/256665395 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f44c00a63f0 msgr2=0x7f44c00a67f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:45.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:45.768 7f44c6598700 1 --2- 172.21.2.112:0/256665395 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f44c00a63f0 0x7f44c00a67f0 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 rx=0x7f44b4009630 tx=0x7f44b400b980).stop 2021-02-10T11:35:45.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:45.769 7f44c6598700 1 -- 172.21.2.112:0/256665395 shutdown_connections 2021-02-10T11:35:45.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:45.769 7f44c6598700 1 --2- 172.21.2.112:0/256665395 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f44a00186c0 0x7f44a001abc0 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:45.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:45.769 7f44c6598700 1 --2- 172.21.2.112:0/256665395 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f44c00a6e10 0x7f44c00a7210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:45.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:45.769 7f44c6598700 1 --2- 172.21.2.112:0/256665395 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f44c00a63f0 0x7f44c00a67f0 unknown :-1 s=CLOSED pgs=1518 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:45.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:45.769 7f44c6598700 1 --2- 172.21.2.112:0/256665395 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f44c00963c0 0x7f44c00967c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:45.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:45.769 7f44c6598700 1 -- 172.21.2.112:0/256665395 >> 172.21.2.112:0/256665395 conn(0x7f44c00be970 msgr2=0x7f44c00919c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:45.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:45.769 7f44c6598700 1 -- 172.21.2.112:0/256665395 shutdown_connections 2021-02-10T11:35:45.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:45.769 7f44c6598700 1 -- 172.21.2.112:0/256665395 wait complete. 2021-02-10T11:35:45.764 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:35:45.764 DEBUG:teuthology.orchestra.run.gibba012:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:35:46.110 DEBUG:teuthology.orchestra.run.gibba020:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:35:46.470 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:35:46.470 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:46.471 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:35:46.471 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:35:46.805 INFO:teuthology.orchestra.run.gibba012.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.0’: No such file or directory 2021-02-10T11:35:46.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:35:46.805 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2021-02-10T11:35:46.805 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:46.805 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:35:46.805 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:35:47.141 INFO:teuthology.orchestra.run.gibba012.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: No such file or directory 2021-02-10T11:35:47.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:35:47.141 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2021-02-10T11:35:47.142 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:47.142 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:35:47.142 DEBUG:teuthology.orchestra.run.gibba020:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:35:47.477 INFO:teuthology.orchestra.run.gibba020.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:35:47.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:35:47.478 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:35:47.478 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:47.478 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:35:47.478 DEBUG:teuthology.orchestra.run.gibba020:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:35:47.811 INFO:teuthology.orchestra.run.gibba020.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:35:47.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:35:47.812 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:35:47.813 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json-pretty 2021-02-10T11:35:48.181 DEBUG:teuthology.orchestra.run.gibba012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:35:48.184 DEBUG:teuthology.orchestra.run.gibba020:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:35:48.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.243 7f72f2129700 1 Processor -- start 2021-02-10T11:35:48.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.243 7f72f2129700 1 -- start start 2021-02-10T11:35:48.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.243 7f72f2129700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f72ec0d1870 0x7f72ec1b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:48.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.243 7f72f2129700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f72ec0f0650 0x7f72ec0cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:48.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.243 7f72f2129700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72ec0f0b10 con 0x7f72ec1b0a10 2021-02-10T11:35:48.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.243 7f72f2129700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72ec0e5ed0 con 0x7f72ec0f0650 2021-02-10T11:35:48.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.243 7f72f2129700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72ec0d1c70 con 0x7f72ec0d1870 2021-02-10T11:35:48.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.243 7f72ebfff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f72ec0f0650 0x7f72ec0cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:48.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.243 7f72ebfff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f72ec0f0650 0x7f72ec0cfb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:51448/0 (socket says 172.21.2.112:51448) 2021-02-10T11:35:48.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.243 7f72ebfff700 1 -- 172.21.2.112:0/3200680189 learned_addr learned my addr 172.21.2.112:0/3200680189 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:35:48.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.244 7f72ebfff700 1 -- 172.21.2.112:0/3200680189 >> v2:172.21.2.120:3300/0 conn(0x7f72ec0d1870 msgr2=0x7f72ec1b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:48.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.244 7f72ebfff700 1 --2- 172.21.2.112:0/3200680189 >> v2:172.21.2.120:3300/0 conn(0x7f72ec0d1870 0x7f72ec1b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:48.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.244 7f72ebfff700 1 -- 172.21.2.112:0/3200680189 >> v1:172.21.2.120:6789/0 conn(0x7f72ec1b0a10 legacy=0x7f72ec0cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:48.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.244 7f72ebfff700 1 -- 172.21.2.112:0/3200680189 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72ec0d1e90 con 0x7f72ec0f0650 2021-02-10T11:35:48.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.244 7f72ebfff700 1 --2- 172.21.2.112:0/3200680189 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f72ec0f0650 0x7f72ec0cfb20 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 rx=0x7f72e4006f00 tx=0x7f72e4009f10).ready entity=mon.0 client_cookie=f2ef15c291429d0c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:48.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.244 7f72ea7fc700 1 -- 172.21.2.112:0/3200680189 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f72e400b3b0 con 0x7f72ec0f0650 2021-02-10T11:35:48.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.244 7f72ea7fc700 1 -- 172.21.2.112:0/3200680189 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72e400b630 con 0x7f72ec0f0650 2021-02-10T11:35:48.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.244 7f72ea7fc700 1 -- 172.21.2.112:0/3200680189 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f72e400bb50 con 0x7f72ec0f0650 2021-02-10T11:35:48.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.244 7f72f2129700 1 -- 172.21.2.112:0/3200680189 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f72ec0f0650 msgr2=0x7f72ec0cfb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:48.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.244 7f72f2129700 1 --2- 172.21.2.112:0/3200680189 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f72ec0f0650 0x7f72ec0cfb20 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 rx=0x7f72e4006f00 tx=0x7f72e4009f10).stop 2021-02-10T11:35:48.248 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.245 7f72f2129700 1 -- 172.21.2.112:0/3200680189 shutdown_connections 2021-02-10T11:35:48.248 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.245 7f72f2129700 1 --2- 172.21.2.112:0/3200680189 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f72ec0f0650 0x7f72ec0cfb20 unknown :-1 s=CLOSED pgs=1519 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:48.248 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.245 7f72f2129700 1 --2- 172.21.2.112:0/3200680189 >> v2:172.21.2.120:3300/0 conn(0x7f72ec0d1870 0x7f72ec1b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:48.248 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.245 7f72f2129700 1 -- 172.21.2.112:0/3200680189 >> 172.21.2.112:0/3200680189 conn(0x7f72ec0ce450 msgr2=0x7f72ec0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:48.248 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.245 7f72f2129700 1 -- 172.21.2.112:0/3200680189 shutdown_connections 2021-02-10T11:35:48.248 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.245 7f72f2129700 1 -- 172.21.2.112:0/3200680189 wait complete. 2021-02-10T11:35:48.248 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.259 7f72f2129700 1 Processor -- start 2021-02-10T11:35:48.248 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.259 7f72f2129700 1 -- start start 2021-02-10T11:35:48.250 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.260 7f72f2129700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f72ec0ae8a0 0x7f72ec1ae550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:48.250 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.260 7f72f2129700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f72ec0eef70 0x7f72ec0ef370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:48.250 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.260 7f72f2129700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f72ec1ab380 0x7f72ec1ab780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:48.250 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.260 7f72f2129700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72ec0ef990 con 0x7f72ec0eef70 2021-02-10T11:35:48.250 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.260 7f72f2129700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72ec1aeb70 con 0x7f72ec1ab380 2021-02-10T11:35:48.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.260 7f72f2129700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72ec1abda0 con 0x7f72ec0ae8a0 2021-02-10T11:35:48.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.260 7f72eaffd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f72ec0eef70 0x7f72ec0ef370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:48.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.260 7f72eaffd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f72ec0eef70 0x7f72ec0ef370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:51450/0 (socket says 172.21.2.112:51450) 2021-02-10T11:35:48.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.260 7f72eaffd700 1 -- 172.21.2.112:0/2926847761 learned_addr learned my addr 172.21.2.112:0/2926847761 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:35:48.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.260 7f72eaffd700 1 -- 172.21.2.112:0/2926847761 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f72ec0ae8a0 msgr2=0x7f72ec1ae550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:48.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.260 7f72eb7fe700 1 --2- 172.21.2.112:0/2926847761 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f72ec0ae8a0 0x7f72ec1ae550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:48.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.260 7f72eaffd700 1 --2- 172.21.2.112:0/2926847761 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f72ec0ae8a0 0x7f72ec1ae550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:48.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.260 7f72eaffd700 1 -- 172.21.2.112:0/2926847761 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f72ec1ab380 msgr2=0x7f72ec1ab780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:48.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.260 7f72ebfff700 1 --2- 172.21.2.112:0/2926847761 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f72ec1ab380 0x7f72ec1ab780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:48.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.260 7f72eaffd700 1 --2- 172.21.2.112:0/2926847761 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f72ec1ab380 0x7f72ec1ab780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:48.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.260 7f72eaffd700 1 -- 172.21.2.112:0/2926847761 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72ec1abfc0 con 0x7f72ec0eef70 2021-02-10T11:35:48.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.260 7f72eb7fe700 1 --2- 172.21.2.112:0/2926847761 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f72ec0ae8a0 0x7f72ec1ae550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:35:48.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.260 7f72ebfff700 1 --2- 172.21.2.112:0/2926847761 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f72ec1ab380 0x7f72ec1ab780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:35:48.252 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.261 7f72eaffd700 1 --2- 172.21.2.112:0/2926847761 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f72ec0eef70 0x7f72ec0ef370 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 rx=0x7f72d400b5b0 tx=0x7f72d400b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:48.252 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.261 7f72e3fff700 1 -- 172.21.2.112:0/2926847761 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f72d400bd40 con 0x7f72ec0eef70 2021-02-10T11:35:48.252 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.261 7f72e3fff700 1 -- 172.21.2.112:0/2926847761 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72d4002030 con 0x7f72ec0eef70 2021-02-10T11:35:48.252 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.261 7f72e3fff700 1 -- 172.21.2.112:0/2926847761 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f72d4002510 con 0x7f72ec0eef70 2021-02-10T11:35:48.252 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.261 7f72f2129700 1 -- 172.21.2.112:0/2926847761 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72ec1abfc0 con 0x7f72ec0eef70 2021-02-10T11:35:48.252 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.261 7f72f2129700 1 -- 172.21.2.112:0/2926847761 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72ec1abfc0 con 0x7f72ec0eef70 2021-02-10T11:35:48.252 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.262 7f72e3fff700 1 -- 172.21.2.112:0/2926847761 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f72d4002210 con 0x7f72ec0eef70 2021-02-10T11:35:48.252 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.262 7f72e3fff700 1 --2- 172.21.2.112:0/2926847761 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f72d00186c0 0x7f72d001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:48.252 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.262 7f72e3fff700 1 -- 172.21.2.112:0/2926847761 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f72d40325a0 con 0x7f72ec0eef70 2021-02-10T11:35:48.252 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.262 7f72eb7fe700 1 --2- 172.21.2.112:0/2926847761 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f72d00186c0 0x7f72d001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:48.252 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.262 7f72eb7fe700 1 --2- 172.21.2.112:0/2926847761 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f72d00186c0 0x7f72d001abc0 crc :-1 s=READY pgs=734 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:48.279 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.293 7f72f2129700 1 -- 172.21.2.112:0/2926847761 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72ec1b0d30 con 0x7f72ec0eef70 2021-02-10T11:35:48.281 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.295 7f72e3fff700 1 -- 172.21.2.112:0/2926847761 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f72d4000ce0 con 0x7f72ec0eef70 2021-02-10T11:35:48.405 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.418 7f72f2129700 1 -- 172.21.2.112:0/2926847761 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f72ec1b0d30 con 0x7f72ec0eef70 2021-02-10T11:35:48.406 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.419 7f72e3fff700 1 -- 172.21.2.112:0/2926847761 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v64) v1 ==== 81+0+30363 (secure 0 0 0) 0x7f72d4030d50 con 0x7f72ec0eef70 2021-02-10T11:35:48.406 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:35:48.407 INFO:teuthology.orchestra.run.gibba012.stdout:{ 2021-02-10T11:35:48.407 INFO:teuthology.orchestra.run.gibba012.stdout: "epoch": 64, 2021-02-10T11:35:48.407 INFO:teuthology.orchestra.run.gibba012.stdout: "fsid": "f14f004e-0699-4584-ad81-3754258493de", 2021-02-10T11:35:48.407 INFO:teuthology.orchestra.run.gibba012.stdout: "created": "2021-02-10 11:13:26.501424", 2021-02-10T11:35:48.407 INFO:teuthology.orchestra.run.gibba012.stdout: "modified": "2021-02-10 11:34:46.584716", 2021-02-10T11:35:48.407 INFO:teuthology.orchestra.run.gibba012.stdout: "last_up_change": "2021-02-10 11:13:52.780316", 2021-02-10T11:35:48.407 INFO:teuthology.orchestra.run.gibba012.stdout: "last_in_change": "2021-02-10 11:13:52.780316", 2021-02-10T11:35:48.407 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-02-10T11:35:48.407 INFO:teuthology.orchestra.run.gibba012.stdout: "flags_num": 5799936, 2021-02-10T11:35:48.407 INFO:teuthology.orchestra.run.gibba012.stdout: "flags_set": [ 2021-02-10T11:35:48.407 INFO:teuthology.orchestra.run.gibba012.stdout: "pglog_hardlimit", 2021-02-10T11:35:48.408 INFO:teuthology.orchestra.run.gibba012.stdout: "purged_snapdirs", 2021-02-10T11:35:48.408 INFO:teuthology.orchestra.run.gibba012.stdout: "recovery_deletes", 2021-02-10T11:35:48.408 INFO:teuthology.orchestra.run.gibba012.stdout: "sortbitwise" 2021-02-10T11:35:48.408 INFO:teuthology.orchestra.run.gibba012.stdout: ], 2021-02-10T11:35:48.408 INFO:teuthology.orchestra.run.gibba012.stdout: "crush_version": 4, 2021-02-10T11:35:48.408 INFO:teuthology.orchestra.run.gibba012.stdout: "full_ratio": 0.94999998807907104, 2021-02-10T11:35:48.408 INFO:teuthology.orchestra.run.gibba012.stdout: "backfillfull_ratio": 0.89999997615814209, 2021-02-10T11:35:48.408 INFO:teuthology.orchestra.run.gibba012.stdout: "nearfull_ratio": 0.85000002384185791, 2021-02-10T11:35:48.408 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_snapshot": "", 2021-02-10T11:35:48.408 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_max": 13, 2021-02-10T11:35:48.408 INFO:teuthology.orchestra.run.gibba012.stdout: "max_osd": 8, 2021-02-10T11:35:48.408 INFO:teuthology.orchestra.run.gibba012.stdout: "require_min_compat_client": "jewel", 2021-02-10T11:35:48.409 INFO:teuthology.orchestra.run.gibba012.stdout: "min_compat_client": "jewel", 2021-02-10T11:35:48.409 INFO:teuthology.orchestra.run.gibba012.stdout: "require_osd_release": "nautilus", 2021-02-10T11:35:48.409 INFO:teuthology.orchestra.run.gibba012.stdout: "pools": [ 2021-02-10T11:35:48.409 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:48.409 INFO:teuthology.orchestra.run.gibba012.stdout: "pool": 1, 2021-02-10T11:35:48.409 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_name": "rbd", 2021-02-10T11:35:48.409 INFO:teuthology.orchestra.run.gibba012.stdout: "create_time": "2021-02-10 11:13:57.508758", 2021-02-10T11:35:48.409 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:35:48.409 INFO:teuthology.orchestra.run.gibba012.stdout: "flags_names": "hashpspool", 2021-02-10T11:35:48.409 INFO:teuthology.orchestra.run.gibba012.stdout: "type": 1, 2021-02-10T11:35:48.409 INFO:teuthology.orchestra.run.gibba012.stdout: "size": 2, 2021-02-10T11:35:48.409 INFO:teuthology.orchestra.run.gibba012.stdout: "min_size": 1, 2021-02-10T11:35:48.410 INFO:teuthology.orchestra.run.gibba012.stdout: "crush_rule": 0, 2021-02-10T11:35:48.410 INFO:teuthology.orchestra.run.gibba012.stdout: "object_hash": 2, 2021-02-10T11:35:48.410 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:35:48.410 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num": 8, 2021-02-10T11:35:48.410 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num": 8, 2021-02-10T11:35:48.410 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num_target": 8, 2021-02-10T11:35:48.410 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_target": 8, 2021-02-10T11:35:48.410 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_pending": 8, 2021-02-10T11:35:48.410 INFO:teuthology.orchestra.run.gibba012.stdout: "last_pg_merge_meta": { 2021-02-10T11:35:48.410 INFO:teuthology.orchestra.run.gibba012.stdout: "source_pgid": "0.0", 2021-02-10T11:35:48.410 INFO:teuthology.orchestra.run.gibba012.stdout: "ready_epoch": 0, 2021-02-10T11:35:48.410 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_started": 0, 2021-02-10T11:35:48.410 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_clean": 0, 2021-02-10T11:35:48.411 INFO:teuthology.orchestra.run.gibba012.stdout: "source_version": "0'0", 2021-02-10T11:35:48.411 INFO:teuthology.orchestra.run.gibba012.stdout: "target_version": "0'0" 2021-02-10T11:35:48.411 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:48.411 INFO:teuthology.orchestra.run.gibba012.stdout: "last_change": "16", 2021-02-10T11:35:48.411 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend": "0", 2021-02-10T11:35:48.411 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:35:48.411 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:35:48.411 INFO:teuthology.orchestra.run.gibba012.stdout: "auid": 0, 2021-02-10T11:35:48.411 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:35:48.411 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_seq": 0, 2021-02-10T11:35:48.412 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_epoch": 0, 2021-02-10T11:35:48.412 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_snaps": [], 2021-02-10T11:35:48.412 INFO:teuthology.orchestra.run.gibba012.stdout: "removed_snaps": "[]", 2021-02-10T11:35:48.412 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_bytes": 0, 2021-02-10T11:35:48.412 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_objects": 0, 2021-02-10T11:35:48.412 INFO:teuthology.orchestra.run.gibba012.stdout: "tiers": [], 2021-02-10T11:35:48.412 INFO:teuthology.orchestra.run.gibba012.stdout: "tier_of": -1, 2021-02-10T11:35:48.412 INFO:teuthology.orchestra.run.gibba012.stdout: "read_tier": -1, 2021-02-10T11:35:48.412 INFO:teuthology.orchestra.run.gibba012.stdout: "write_tier": -1, 2021-02-10T11:35:48.412 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_mode": "none", 2021-02-10T11:35:48.413 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_bytes": 0, 2021-02-10T11:35:48.413 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_objects": 0, 2021-02-10T11:35:48.413 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:35:48.413 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:35:48.413 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:35:48.413 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_flush_age": 0, 2021-02-10T11:35:48.413 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_evict_age": 0, 2021-02-10T11:35:48.413 INFO:teuthology.orchestra.run.gibba012.stdout: "erasure_code_profile": "", 2021-02-10T11:35:48.413 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_params": { 2021-02-10T11:35:48.413 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "none" 2021-02-10T11:35:48.413 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:48.413 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_period": 0, 2021-02-10T11:35:48.413 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_count": 0, 2021-02-10T11:35:48.413 INFO:teuthology.orchestra.run.gibba012.stdout: "use_gmt_hitset": true, 2021-02-10T11:35:48.414 INFO:teuthology.orchestra.run.gibba012.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:35:48.414 INFO:teuthology.orchestra.run.gibba012.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:35:48.414 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:35:48.414 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:35:48.414 INFO:teuthology.orchestra.run.gibba012.stdout: "grade_table": [], 2021-02-10T11:35:48.414 INFO:teuthology.orchestra.run.gibba012.stdout: "stripe_width": 0, 2021-02-10T11:35:48.414 INFO:teuthology.orchestra.run.gibba012.stdout: "expected_num_objects": 0, 2021-02-10T11:35:48.414 INFO:teuthology.orchestra.run.gibba012.stdout: "fast_read": false, 2021-02-10T11:35:48.414 INFO:teuthology.orchestra.run.gibba012.stdout: "options": {}, 2021-02-10T11:35:48.414 INFO:teuthology.orchestra.run.gibba012.stdout: "application_metadata": { 2021-02-10T11:35:48.414 INFO:teuthology.orchestra.run.gibba012.stdout: "rbd": {} 2021-02-10T11:35:48.414 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:35:48.415 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:48.415 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:48.415 INFO:teuthology.orchestra.run.gibba012.stdout: "pool": 12, 2021-02-10T11:35:48.415 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_name": "cephfs_metadata", 2021-02-10T11:35:48.415 INFO:teuthology.orchestra.run.gibba012.stdout: "create_time": "2021-02-10 11:34:43.550220", 2021-02-10T11:35:48.415 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:35:48.415 INFO:teuthology.orchestra.run.gibba012.stdout: "flags_names": "hashpspool", 2021-02-10T11:35:48.415 INFO:teuthology.orchestra.run.gibba012.stdout: "type": 1, 2021-02-10T11:35:48.415 INFO:teuthology.orchestra.run.gibba012.stdout: "size": 2, 2021-02-10T11:35:48.415 INFO:teuthology.orchestra.run.gibba012.stdout: "min_size": 1, 2021-02-10T11:35:48.415 INFO:teuthology.orchestra.run.gibba012.stdout: "crush_rule": 0, 2021-02-10T11:35:48.415 INFO:teuthology.orchestra.run.gibba012.stdout: "object_hash": 2, 2021-02-10T11:35:48.415 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:35:48.415 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num": 8, 2021-02-10T11:35:48.416 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num": 8, 2021-02-10T11:35:48.416 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num_target": 8, 2021-02-10T11:35:48.416 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_target": 8, 2021-02-10T11:35:48.416 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_pending": 8, 2021-02-10T11:35:48.416 INFO:teuthology.orchestra.run.gibba012.stdout: "last_pg_merge_meta": { 2021-02-10T11:35:48.416 INFO:teuthology.orchestra.run.gibba012.stdout: "source_pgid": "0.0", 2021-02-10T11:35:48.416 INFO:teuthology.orchestra.run.gibba012.stdout: "ready_epoch": 0, 2021-02-10T11:35:48.416 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_started": 0, 2021-02-10T11:35:48.416 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_clean": 0, 2021-02-10T11:35:48.416 INFO:teuthology.orchestra.run.gibba012.stdout: "source_version": "0'0", 2021-02-10T11:35:48.416 INFO:teuthology.orchestra.run.gibba012.stdout: "target_version": "0'0" 2021-02-10T11:35:48.416 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:48.416 INFO:teuthology.orchestra.run.gibba012.stdout: "last_change": "63", 2021-02-10T11:35:48.416 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend": "0", 2021-02-10T11:35:48.417 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:35:48.417 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:35:48.417 INFO:teuthology.orchestra.run.gibba012.stdout: "auid": 0, 2021-02-10T11:35:48.417 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:35:48.417 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_seq": 0, 2021-02-10T11:35:48.417 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_epoch": 0, 2021-02-10T11:35:48.417 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_snaps": [], 2021-02-10T11:35:48.417 INFO:teuthology.orchestra.run.gibba012.stdout: "removed_snaps": "[]", 2021-02-10T11:35:48.417 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_bytes": 0, 2021-02-10T11:35:48.417 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_objects": 0, 2021-02-10T11:35:48.417 INFO:teuthology.orchestra.run.gibba012.stdout: "tiers": [], 2021-02-10T11:35:48.417 INFO:teuthology.orchestra.run.gibba012.stdout: "tier_of": -1, 2021-02-10T11:35:48.417 INFO:teuthology.orchestra.run.gibba012.stdout: "read_tier": -1, 2021-02-10T11:35:48.418 INFO:teuthology.orchestra.run.gibba012.stdout: "write_tier": -1, 2021-02-10T11:35:48.418 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_mode": "none", 2021-02-10T11:35:48.418 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_bytes": 0, 2021-02-10T11:35:48.418 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_objects": 0, 2021-02-10T11:35:48.418 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:35:48.418 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:35:48.418 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:35:48.418 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_flush_age": 0, 2021-02-10T11:35:48.418 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_evict_age": 0, 2021-02-10T11:35:48.418 INFO:teuthology.orchestra.run.gibba012.stdout: "erasure_code_profile": "", 2021-02-10T11:35:48.418 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_params": { 2021-02-10T11:35:48.418 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "none" 2021-02-10T11:35:48.418 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:48.419 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_period": 0, 2021-02-10T11:35:48.419 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_count": 0, 2021-02-10T11:35:48.419 INFO:teuthology.orchestra.run.gibba012.stdout: "use_gmt_hitset": true, 2021-02-10T11:35:48.419 INFO:teuthology.orchestra.run.gibba012.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:35:48.419 INFO:teuthology.orchestra.run.gibba012.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:35:48.419 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:35:48.419 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:35:48.419 INFO:teuthology.orchestra.run.gibba012.stdout: "grade_table": [], 2021-02-10T11:35:48.419 INFO:teuthology.orchestra.run.gibba012.stdout: "stripe_width": 0, 2021-02-10T11:35:48.420 INFO:teuthology.orchestra.run.gibba012.stdout: "expected_num_objects": 0, 2021-02-10T11:35:48.420 INFO:teuthology.orchestra.run.gibba012.stdout: "fast_read": false, 2021-02-10T11:35:48.420 INFO:teuthology.orchestra.run.gibba012.stdout: "options": { 2021-02-10T11:35:48.420 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_autoscale_bias": 4, 2021-02-10T11:35:48.420 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_min": 16, 2021-02-10T11:35:48.420 INFO:teuthology.orchestra.run.gibba012.stdout: "recovery_priority": 5 2021-02-10T11:35:48.420 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:48.420 INFO:teuthology.orchestra.run.gibba012.stdout: "application_metadata": { 2021-02-10T11:35:48.420 INFO:teuthology.orchestra.run.gibba012.stdout: "cephfs": { 2021-02-10T11:35:48.420 INFO:teuthology.orchestra.run.gibba012.stdout: "metadata": "cephfs" 2021-02-10T11:35:48.420 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:35:48.420 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:35:48.420 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:48.421 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:48.421 INFO:teuthology.orchestra.run.gibba012.stdout: "pool": 13, 2021-02-10T11:35:48.421 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_name": "cephfs_data", 2021-02-10T11:35:48.421 INFO:teuthology.orchestra.run.gibba012.stdout: "create_time": "2021-02-10 11:34:44.578113", 2021-02-10T11:35:48.421 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:35:48.421 INFO:teuthology.orchestra.run.gibba012.stdout: "flags_names": "hashpspool", 2021-02-10T11:35:48.421 INFO:teuthology.orchestra.run.gibba012.stdout: "type": 1, 2021-02-10T11:35:48.421 INFO:teuthology.orchestra.run.gibba012.stdout: "size": 2, 2021-02-10T11:35:48.421 INFO:teuthology.orchestra.run.gibba012.stdout: "min_size": 1, 2021-02-10T11:35:48.421 INFO:teuthology.orchestra.run.gibba012.stdout: "crush_rule": 0, 2021-02-10T11:35:48.421 INFO:teuthology.orchestra.run.gibba012.stdout: "object_hash": 2, 2021-02-10T11:35:48.421 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:35:48.421 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num": 8, 2021-02-10T11:35:48.422 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num": 8, 2021-02-10T11:35:48.422 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num_target": 8, 2021-02-10T11:35:48.422 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_target": 8, 2021-02-10T11:35:48.422 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_pending": 8, 2021-02-10T11:35:48.422 INFO:teuthology.orchestra.run.gibba012.stdout: "last_pg_merge_meta": { 2021-02-10T11:35:48.422 INFO:teuthology.orchestra.run.gibba012.stdout: "source_pgid": "0.0", 2021-02-10T11:35:48.422 INFO:teuthology.orchestra.run.gibba012.stdout: "ready_epoch": 0, 2021-02-10T11:35:48.422 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_started": 0, 2021-02-10T11:35:48.422 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_clean": 0, 2021-02-10T11:35:48.422 INFO:teuthology.orchestra.run.gibba012.stdout: "source_version": "0'0", 2021-02-10T11:35:48.422 INFO:teuthology.orchestra.run.gibba012.stdout: "target_version": "0'0" 2021-02-10T11:35:48.422 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:48.422 INFO:teuthology.orchestra.run.gibba012.stdout: "last_change": "64", 2021-02-10T11:35:48.423 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend": "0", 2021-02-10T11:35:48.423 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:35:48.423 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:35:48.423 INFO:teuthology.orchestra.run.gibba012.stdout: "auid": 0, 2021-02-10T11:35:48.423 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:35:48.423 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_seq": 0, 2021-02-10T11:35:48.423 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_epoch": 0, 2021-02-10T11:35:48.423 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_snaps": [], 2021-02-10T11:35:48.423 INFO:teuthology.orchestra.run.gibba012.stdout: "removed_snaps": "[]", 2021-02-10T11:35:48.423 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_bytes": 0, 2021-02-10T11:35:48.423 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_objects": 0, 2021-02-10T11:35:48.423 INFO:teuthology.orchestra.run.gibba012.stdout: "tiers": [], 2021-02-10T11:35:48.423 INFO:teuthology.orchestra.run.gibba012.stdout: "tier_of": -1, 2021-02-10T11:35:48.423 INFO:teuthology.orchestra.run.gibba012.stdout: "read_tier": -1, 2021-02-10T11:35:48.424 INFO:teuthology.orchestra.run.gibba012.stdout: "write_tier": -1, 2021-02-10T11:35:48.424 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_mode": "none", 2021-02-10T11:35:48.424 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_bytes": 0, 2021-02-10T11:35:48.424 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_objects": 0, 2021-02-10T11:35:48.424 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:35:48.424 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:35:48.424 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:35:48.424 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_flush_age": 0, 2021-02-10T11:35:48.424 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_evict_age": 0, 2021-02-10T11:35:48.424 INFO:teuthology.orchestra.run.gibba012.stdout: "erasure_code_profile": "", 2021-02-10T11:35:48.424 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_params": { 2021-02-10T11:35:48.424 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "none" 2021-02-10T11:35:48.424 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:48.424 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_period": 0, 2021-02-10T11:35:48.425 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_count": 0, 2021-02-10T11:35:48.425 INFO:teuthology.orchestra.run.gibba012.stdout: "use_gmt_hitset": true, 2021-02-10T11:35:48.425 INFO:teuthology.orchestra.run.gibba012.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:35:48.425 INFO:teuthology.orchestra.run.gibba012.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:35:48.425 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:35:48.425 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:35:48.425 INFO:teuthology.orchestra.run.gibba012.stdout: "grade_table": [], 2021-02-10T11:35:48.425 INFO:teuthology.orchestra.run.gibba012.stdout: "stripe_width": 0, 2021-02-10T11:35:48.425 INFO:teuthology.orchestra.run.gibba012.stdout: "expected_num_objects": 0, 2021-02-10T11:35:48.425 INFO:teuthology.orchestra.run.gibba012.stdout: "fast_read": false, 2021-02-10T11:35:48.425 INFO:teuthology.orchestra.run.gibba012.stdout: "options": {}, 2021-02-10T11:35:48.425 INFO:teuthology.orchestra.run.gibba012.stdout: "application_metadata": { 2021-02-10T11:35:48.425 INFO:teuthology.orchestra.run.gibba012.stdout: "cephfs": { 2021-02-10T11:35:48.426 INFO:teuthology.orchestra.run.gibba012.stdout: "data": "cephfs" 2021-02-10T11:35:48.426 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:35:48.426 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:35:48.426 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:35:48.426 INFO:teuthology.orchestra.run.gibba012.stdout: ], 2021-02-10T11:35:48.426 INFO:teuthology.orchestra.run.gibba012.stdout: "osds": [ 2021-02-10T11:35:48.426 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:48.426 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 0, 2021-02-10T11:35:48.426 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "95444b8f-6370-4a1d-ae5f-df62651b77da", 2021-02-10T11:35:48.426 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:35:48.426 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:35:48.426 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:35:48.426 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:35:48.427 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:35:48.427 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:35:48.427 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:35:48.427 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 62, 2021-02-10T11:35:48.427 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:35:48.427 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:35:48.427 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:35:48.427 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:35:48.427 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:48.427 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:35:48.427 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6817", 2021-02-10T11:35:48.427 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:35:48.427 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:48.427 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:48.428 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:35:48.428 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6818", 2021-02-10T11:35:48.428 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:35:48.428 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:35:48.428 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:35:48.428 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:48.428 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:35:48.428 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:35:48.428 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:48.428 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:35:48.428 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6820", 2021-02-10T11:35:48.428 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:35:48.428 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:48.429 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:48.429 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:35:48.429 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6822", 2021-02-10T11:35:48.429 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:35:48.429 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:35:48.429 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:35:48.429 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:48.429 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:35:48.429 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:35:48.429 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:48.429 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:35:48.429 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6828", 2021-02-10T11:35:48.429 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:35:48.429 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:48.430 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:48.430 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:35:48.430 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6830", 2021-02-10T11:35:48.430 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:35:48.430 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:35:48.430 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:35:48.430 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:48.430 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:35:48.430 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:35:48.430 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:48.430 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:35:48.430 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6824", 2021-02-10T11:35:48.430 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:35:48.431 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:48.431 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:48.431 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:35:48.431 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6826", 2021-02-10T11:35:48.431 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:35:48.431 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:35:48.431 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:35:48.431 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:48.431 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.112:6818/36030", 2021-02-10T11:35:48.431 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.112:6822/36030", 2021-02-10T11:35:48.431 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.112:6830/36030", 2021-02-10T11:35:48.431 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.112:6826/36030", 2021-02-10T11:35:48.432 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:35:48.432 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:35:48.432 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:35:48.432 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:35:48.432 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:48.432 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:48.432 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 1, 2021-02-10T11:35:48.432 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "a23e217e-0d8e-486e-be07-712adfabd262", 2021-02-10T11:35:48.432 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:35:48.432 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:35:48.432 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:35:48.432 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:35:48.432 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:35:48.432 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:35:48.433 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:35:48.433 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 60, 2021-02-10T11:35:48.433 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:35:48.433 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:35:48.433 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:35:48.433 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:35:48.433 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:48.433 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:35:48.433 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6800", 2021-02-10T11:35:48.433 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:35:48.433 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:48.433 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:48.433 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:35:48.434 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6802", 2021-02-10T11:35:48.434 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:35:48.434 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:35:48.434 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:35:48.434 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:48.434 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:35:48.434 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:35:48.434 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:48.434 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:35:48.434 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6804", 2021-02-10T11:35:48.434 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:35:48.434 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:48.434 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:48.435 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:35:48.435 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6806", 2021-02-10T11:35:48.435 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:35:48.435 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:35:48.435 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:35:48.435 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:48.435 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:35:48.435 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:35:48.435 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:48.435 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:35:48.435 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6812", 2021-02-10T11:35:48.435 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:35:48.435 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:48.436 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:48.436 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:35:48.436 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6814", 2021-02-10T11:35:48.436 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:35:48.436 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:35:48.436 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:35:48.436 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:48.436 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:35:48.436 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:35:48.436 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:48.436 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:35:48.436 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6808", 2021-02-10T11:35:48.436 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:35:48.436 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:48.437 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:48.437 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:35:48.437 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6810", 2021-02-10T11:35:48.437 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:35:48.437 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:35:48.437 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:35:48.437 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:48.437 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.112:6802/36029", 2021-02-10T11:35:48.437 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.112:6806/36029", 2021-02-10T11:35:48.437 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.112:6814/36029", 2021-02-10T11:35:48.437 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.112:6810/36029", 2021-02-10T11:35:48.437 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:35:48.437 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:35:48.437 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:35:48.438 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:35:48.438 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:48.438 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:48.438 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 2, 2021-02-10T11:35:48.438 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "026afa8a-c0f1-4b35-9a76-53c00fe0c2fc", 2021-02-10T11:35:48.438 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:35:48.438 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:35:48.438 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:35:48.438 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:35:48.438 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:35:48.438 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:35:48.438 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:35:48.439 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 53, 2021-02-10T11:35:48.439 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:35:48.439 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:35:48.439 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:35:48.439 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:35:48.439 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:48.439 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:35:48.439 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6801", 2021-02-10T11:35:48.439 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:35:48.439 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:48.439 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:48.439 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:35:48.439 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6803", 2021-02-10T11:35:48.439 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:35:48.440 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:35:48.440 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:35:48.440 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:48.440 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:35:48.440 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:35:48.440 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:48.440 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:35:48.440 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6805", 2021-02-10T11:35:48.440 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:35:48.440 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:48.440 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:48.440 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:35:48.440 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6807", 2021-02-10T11:35:48.441 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:35:48.441 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:35:48.441 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:35:48.441 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:48.441 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:35:48.441 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:35:48.441 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:48.441 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:35:48.441 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6813", 2021-02-10T11:35:48.441 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:35:48.441 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:48.441 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:48.441 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:35:48.441 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6815", 2021-02-10T11:35:48.442 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:35:48.442 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:35:48.442 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:35:48.442 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:48.442 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:35:48.442 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:35:48.442 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:48.442 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:35:48.442 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6809", 2021-02-10T11:35:48.442 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:35:48.442 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:48.442 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:48.442 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:35:48.443 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6811", 2021-02-10T11:35:48.443 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:35:48.443 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:35:48.443 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:35:48.443 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:48.443 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.112:6803/36031", 2021-02-10T11:35:48.443 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.112:6807/36031", 2021-02-10T11:35:48.443 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.112:6815/36031", 2021-02-10T11:35:48.443 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.112:6811/36031", 2021-02-10T11:35:48.443 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:35:48.443 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:35:48.443 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:35:48.443 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:35:48.444 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:48.444 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:48.444 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 3, 2021-02-10T11:35:48.444 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "58795682-e00d-4173-bb84-f7075ca89a11", 2021-02-10T11:35:48.444 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:35:48.444 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:35:48.444 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:35:48.444 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:35:48.444 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:35:48.444 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:35:48.444 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:35:48.444 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 44, 2021-02-10T11:35:48.444 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:35:48.444 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:35:48.445 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:35:48.445 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:35:48.445 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:48.445 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:35:48.445 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6816", 2021-02-10T11:35:48.445 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:35:48.445 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:48.445 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:48.445 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:35:48.445 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6819", 2021-02-10T11:35:48.445 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:35:48.445 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:35:48.445 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:35:48.446 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:48.446 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:35:48.446 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:35:48.446 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:48.446 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:35:48.446 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6821", 2021-02-10T11:35:48.446 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:35:48.446 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:48.446 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:48.446 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:35:48.446 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6823", 2021-02-10T11:35:48.446 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:35:48.446 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:35:48.447 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:35:48.447 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:48.447 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:35:48.447 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:35:48.447 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:48.447 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:35:48.447 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6829", 2021-02-10T11:35:48.448 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:35:48.448 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:48.448 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:48.448 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:35:48.448 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6831", 2021-02-10T11:35:48.448 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:35:48.448 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:35:48.448 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:35:48.448 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:48.448 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:35:48.448 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:35:48.448 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:48.448 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:35:48.448 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6825", 2021-02-10T11:35:48.449 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:35:48.449 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:48.449 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:48.449 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:35:48.449 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6827", 2021-02-10T11:35:48.449 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:35:48.449 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:35:48.449 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:35:48.449 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:48.449 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.112:6819/36028", 2021-02-10T11:35:48.449 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.112:6823/36028", 2021-02-10T11:35:48.449 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.112:6831/36028", 2021-02-10T11:35:48.449 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.112:6827/36028", 2021-02-10T11:35:48.450 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:35:48.450 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:35:48.450 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:35:48.450 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:35:48.450 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:48.450 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:48.450 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 4, 2021-02-10T11:35:48.450 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "6763f8d6-53dc-4e45-bdf6-a6eabcf81e39", 2021-02-10T11:35:48.450 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:35:48.450 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:35:48.450 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:35:48.450 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:35:48.450 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:35:48.451 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:35:48.451 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:35:48.451 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 62, 2021-02-10T11:35:48.451 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:35:48.451 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:35:48.451 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:35:48.451 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:35:48.451 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:48.451 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:35:48.451 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6810", 2021-02-10T11:35:48.451 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:35:48.451 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:48.451 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:48.451 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:35:48.452 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6813", 2021-02-10T11:35:48.452 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:35:48.452 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:35:48.452 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:35:48.452 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:48.452 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:35:48.452 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:35:48.452 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:48.452 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:35:48.452 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6816", 2021-02-10T11:35:48.452 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:35:48.452 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:48.452 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:48.452 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:35:48.453 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6819", 2021-02-10T11:35:48.453 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:35:48.453 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:35:48.453 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:35:48.453 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:48.453 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:35:48.453 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:35:48.453 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:48.453 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:35:48.453 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6825", 2021-02-10T11:35:48.453 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:35:48.453 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:48.453 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:48.453 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:35:48.454 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6827", 2021-02-10T11:35:48.454 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:35:48.454 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:35:48.454 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:35:48.454 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:48.454 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:35:48.454 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:35:48.454 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:48.454 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:35:48.454 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6821", 2021-02-10T11:35:48.454 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:35:48.454 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:48.454 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:48.455 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:35:48.455 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6823", 2021-02-10T11:35:48.455 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:35:48.455 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:35:48.455 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:35:48.455 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:48.455 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.120:6813/36388", 2021-02-10T11:35:48.455 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.120:6819/36388", 2021-02-10T11:35:48.455 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.120:6827/36388", 2021-02-10T11:35:48.455 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.120:6823/36388", 2021-02-10T11:35:48.455 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:35:48.455 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:35:48.455 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:35:48.455 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:35:48.456 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:48.456 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:48.456 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 5, 2021-02-10T11:35:48.456 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "5d0e4dd6-0cb0-4bf8-8314-626acaa1b544", 2021-02-10T11:35:48.456 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:35:48.456 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:35:48.456 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:35:48.456 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:35:48.456 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:35:48.456 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:35:48.456 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:35:48.456 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 62, 2021-02-10T11:35:48.456 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:35:48.456 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:35:48.457 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:35:48.457 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:35:48.457 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:48.457 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:35:48.457 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6804", 2021-02-10T11:35:48.457 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:35:48.457 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:48.457 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:48.457 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:35:48.457 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6806", 2021-02-10T11:35:48.457 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:35:48.457 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:35:48.457 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:35:48.457 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:48.458 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:35:48.458 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:35:48.458 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:48.458 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:35:48.458 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6809", 2021-02-10T11:35:48.458 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:35:48.458 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:48.458 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:48.458 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:35:48.458 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6812", 2021-02-10T11:35:48.458 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:35:48.458 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:35:48.458 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:35:48.459 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:48.459 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:35:48.459 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:35:48.459 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:48.459 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:35:48.459 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6820", 2021-02-10T11:35:48.459 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:35:48.459 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:48.459 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:48.459 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:35:48.459 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6822", 2021-02-10T11:35:48.459 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:35:48.459 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:35:48.459 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:35:48.460 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:48.460 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:35:48.460 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:35:48.460 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:48.460 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:35:48.460 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6815", 2021-02-10T11:35:48.460 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:35:48.460 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:48.460 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:48.460 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:35:48.461 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6818", 2021-02-10T11:35:48.461 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:35:48.461 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:35:48.461 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:35:48.461 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:48.461 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.120:6806/36387", 2021-02-10T11:35:48.461 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.120:6812/36387", 2021-02-10T11:35:48.461 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.120:6822/36387", 2021-02-10T11:35:48.461 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.120:6818/36387", 2021-02-10T11:35:48.462 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:35:48.462 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:35:48.462 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:35:48.462 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:35:48.462 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:48.462 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:48.462 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 6, 2021-02-10T11:35:48.462 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "b010e187-4541-4b00-80dd-f948f6ab8f15", 2021-02-10T11:35:48.462 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:35:48.462 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:35:48.462 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:35:48.462 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:35:48.462 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:35:48.463 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:35:48.463 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:35:48.463 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 62, 2021-02-10T11:35:48.463 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:35:48.463 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:35:48.463 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:35:48.463 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:35:48.463 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:48.463 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:35:48.463 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6824", 2021-02-10T11:35:48.463 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:35:48.463 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:48.463 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:48.463 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:35:48.464 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6826", 2021-02-10T11:35:48.464 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:35:48.464 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:35:48.464 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:35:48.464 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:48.464 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:35:48.464 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:35:48.464 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:48.464 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:35:48.464 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6828", 2021-02-10T11:35:48.464 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:35:48.464 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:48.465 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:48.465 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:35:48.465 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6829", 2021-02-10T11:35:48.465 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:35:48.465 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:35:48.465 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:35:48.465 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:48.465 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:35:48.465 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:35:48.465 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:48.465 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:35:48.465 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6832", 2021-02-10T11:35:48.465 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:35:48.466 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:48.466 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:48.466 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:35:48.466 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6833", 2021-02-10T11:35:48.466 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:35:48.466 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:35:48.466 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:35:48.466 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:48.466 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:35:48.466 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:35:48.466 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:48.466 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:35:48.466 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6830", 2021-02-10T11:35:48.467 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:35:48.467 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:48.467 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:48.467 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:35:48.467 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6831", 2021-02-10T11:35:48.467 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:35:48.467 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:35:48.467 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:35:48.468 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:48.468 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.120:6826/36385", 2021-02-10T11:35:48.468 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.120:6829/36385", 2021-02-10T11:35:48.468 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.120:6833/36385", 2021-02-10T11:35:48.468 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.120:6831/36385", 2021-02-10T11:35:48.468 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:35:48.468 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:35:48.468 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:35:48.468 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:35:48.468 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:48.468 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:48.468 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 7, 2021-02-10T11:35:48.468 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "671e2772-33b7-497e-8e44-2178bd850bcf", 2021-02-10T11:35:48.469 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:35:48.469 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:35:48.469 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:35:48.469 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:35:48.469 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:35:48.469 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:35:48.469 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:35:48.469 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 62, 2021-02-10T11:35:48.469 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:35:48.469 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:35:48.469 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:35:48.469 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:35:48.469 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:48.469 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:35:48.470 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6802", 2021-02-10T11:35:48.470 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:35:48.470 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:48.470 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:48.470 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:35:48.470 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6803", 2021-02-10T11:35:48.470 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:35:48.470 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:35:48.470 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:35:48.470 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:48.470 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:35:48.470 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:35:48.470 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:48.471 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:35:48.471 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6805", 2021-02-10T11:35:48.471 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:35:48.471 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:48.471 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:48.471 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:35:48.471 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6807", 2021-02-10T11:35:48.471 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:35:48.471 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:35:48.471 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:35:48.471 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:48.471 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:35:48.471 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:35:48.471 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:48.472 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:35:48.472 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6814", 2021-02-10T11:35:48.472 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:35:48.472 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:48.472 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:48.472 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:35:48.472 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6817", 2021-02-10T11:35:48.472 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:35:48.472 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:35:48.472 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:35:48.472 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:48.472 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:35:48.472 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:35:48.473 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:48.473 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:35:48.473 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6808", 2021-02-10T11:35:48.473 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:35:48.473 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:48.473 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:48.473 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:35:48.473 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6811", 2021-02-10T11:35:48.473 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:35:48.473 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:35:48.473 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:35:48.473 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:48.473 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.120:6803/36386", 2021-02-10T11:35:48.473 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.120:6807/36386", 2021-02-10T11:35:48.474 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.120:6817/36386", 2021-02-10T11:35:48.474 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.120:6811/36386", 2021-02-10T11:35:48.474 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:35:48.474 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:35:48.474 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:35:48.474 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:35:48.474 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:35:48.474 INFO:teuthology.orchestra.run.gibba012.stdout: ], 2021-02-10T11:35:48.474 INFO:teuthology.orchestra.run.gibba012.stdout: "osd_xinfo": [ 2021-02-10T11:35:48.474 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:48.474 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 0, 2021-02-10T11:35:48.474 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:35:48.474 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:35:48.475 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:35:48.475 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:35:48.475 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:35:48.475 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:48.475 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:48.475 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 1, 2021-02-10T11:35:48.475 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:35:48.475 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:35:48.475 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:35:48.475 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:35:48.475 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:35:48.476 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:48.476 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:48.476 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 2, 2021-02-10T11:35:48.476 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:35:48.476 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:35:48.476 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:35:48.476 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:35:48.476 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:35:48.476 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:48.476 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:48.476 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 3, 2021-02-10T11:35:48.477 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:35:48.477 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:35:48.477 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:35:48.477 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:35:48.477 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:35:48.477 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:48.477 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:48.477 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 4, 2021-02-10T11:35:48.477 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:35:48.478 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:35:48.478 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:35:48.478 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:35:48.478 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:35:48.478 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:48.478 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:48.478 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 5, 2021-02-10T11:35:48.478 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:35:48.478 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:35:48.478 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:35:48.478 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:35:48.478 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:35:48.478 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:48.479 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:48.479 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 6, 2021-02-10T11:35:48.479 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:35:48.479 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:35:48.479 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:35:48.479 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:35:48.479 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:35:48.479 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:48.479 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:48.479 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 7, 2021-02-10T11:35:48.479 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:35:48.479 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:35:48.479 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:35:48.479 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:35:48.480 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:35:48.480 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:35:48.480 INFO:teuthology.orchestra.run.gibba012.stdout: ], 2021-02-10T11:35:48.480 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_upmap": [], 2021-02-10T11:35:48.480 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_upmap_items": [], 2021-02-10T11:35:48.480 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_temp": [], 2021-02-10T11:35:48.480 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_temp": [], 2021-02-10T11:35:48.480 INFO:teuthology.orchestra.run.gibba012.stdout: "blacklist": {}, 2021-02-10T11:35:48.480 INFO:teuthology.orchestra.run.gibba012.stdout: "erasure_code_profiles": { 2021-02-10T11:35:48.480 INFO:teuthology.orchestra.run.gibba012.stdout: "default": { 2021-02-10T11:35:48.480 INFO:teuthology.orchestra.run.gibba012.stdout: "crush-failure-domain": "osd", 2021-02-10T11:35:48.480 INFO:teuthology.orchestra.run.gibba012.stdout: "k": "2", 2021-02-10T11:35:48.480 INFO:teuthology.orchestra.run.gibba012.stdout: "m": "1", 2021-02-10T11:35:48.481 INFO:teuthology.orchestra.run.gibba012.stdout: "plugin": "jerasure", 2021-02-10T11:35:48.481 INFO:teuthology.orchestra.run.gibba012.stdout: "ruleset-failure-domain": "osd", 2021-02-10T11:35:48.481 INFO:teuthology.orchestra.run.gibba012.stdout: "technique": "reed_sol_van" 2021-02-10T11:35:48.481 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:35:48.481 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:48.481 INFO:teuthology.orchestra.run.gibba012.stdout: "removed_snaps_queue": [], 2021-02-10T11:35:48.481 INFO:teuthology.orchestra.run.gibba012.stdout: "new_removed_snaps": [], 2021-02-10T11:35:48.481 INFO:teuthology.orchestra.run.gibba012.stdout: "new_purged_snaps": [], 2021-02-10T11:35:48.481 INFO:teuthology.orchestra.run.gibba012.stdout: "crush_node_flags": {}, 2021-02-10T11:35:48.481 INFO:teuthology.orchestra.run.gibba012.stdout: "device_class_flags": {} 2021-02-10T11:35:48.481 INFO:teuthology.orchestra.run.gibba012.stdout:} 2021-02-10T11:35:48.483 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.421 7f72f2129700 1 -- 172.21.2.112:0/2926847761 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f72d00186c0 msgr2=0x7f72d001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:48.483 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.421 7f72f2129700 1 --2- 172.21.2.112:0/2926847761 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f72d00186c0 0x7f72d001abc0 crc :-1 s=READY pgs=734 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:48.483 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.421 7f72f2129700 1 -- 172.21.2.112:0/2926847761 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f72ec0eef70 msgr2=0x7f72ec0ef370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:48.483 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.421 7f72f2129700 1 --2- 172.21.2.112:0/2926847761 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f72ec0eef70 0x7f72ec0ef370 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 rx=0x7f72d400b5b0 tx=0x7f72d400b950).stop 2021-02-10T11:35:48.483 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.421 7f72f2129700 1 -- 172.21.2.112:0/2926847761 shutdown_connections 2021-02-10T11:35:48.483 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.421 7f72f2129700 1 --2- 172.21.2.112:0/2926847761 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f72d00186c0 0x7f72d001abc0 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:48.483 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.421 7f72f2129700 1 --2- 172.21.2.112:0/2926847761 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f72ec1ab380 0x7f72ec1ab780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:48.483 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.421 7f72f2129700 1 --2- 172.21.2.112:0/2926847761 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f72ec0eef70 0x7f72ec0ef370 unknown :-1 s=CLOSED pgs=1520 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:48.483 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.421 7f72f2129700 1 --2- 172.21.2.112:0/2926847761 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f72ec0ae8a0 0x7f72ec1ae550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:48.483 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.421 7f72f2129700 1 -- 172.21.2.112:0/2926847761 >> 172.21.2.112:0/2926847761 conn(0x7f72ec1a8c90 msgr2=0x7f72ec1b38a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:48.483 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.421 7f72f2129700 1 -- 172.21.2.112:0/2926847761 shutdown_connections 2021-02-10T11:35:48.483 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.421 7f72f2129700 1 -- 172.21.2.112:0/2926847761 wait complete. 2021-02-10T11:35:48.484 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:35:48.910 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.907 7fda28884700 1 Processor -- start 2021-02-10T11:35:48.910 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.907 7fda28884700 1 -- start start 2021-02-10T11:35:48.910 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.907 7fda28884700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fda240d1870 0x7fda241b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:48.910 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.907 7fda28884700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fda241b0a10 0x7fda240cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:48.910 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.907 7fda28884700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda240f0b10 con 0x7fda240d1c70 2021-02-10T11:35:48.910 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.907 7fda28884700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda240e5ed0 con 0x7fda241b0a10 2021-02-10T11:35:48.910 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.907 7fda28884700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda240f0650 con 0x7fda240d1870 2021-02-10T11:35:48.910 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.907 7fda21e1b700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fda241b0a10 0x7fda240cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:48.910 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.907 7fda21e1b700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fda241b0a10 0x7fda240cf330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:51458/0 (socket says 172.21.2.112:51458) 2021-02-10T11:35:48.910 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.907 7fda21e1b700 1 -- 172.21.2.112:0/1490243088 learned_addr learned my addr 172.21.2.112:0/1490243088 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:35:48.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.908 7fda21e1b700 1 -- 172.21.2.112:0/1490243088 >> v2:172.21.2.120:3300/0 conn(0x7fda240d1870 msgr2=0x7fda241b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:48.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.908 7fda21e1b700 1 --2- 172.21.2.112:0/1490243088 >> v2:172.21.2.120:3300/0 conn(0x7fda240d1870 0x7fda241b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:48.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.908 7fda21e1b700 1 -- 172.21.2.112:0/1490243088 >> v1:172.21.2.120:6789/0 conn(0x7fda240d1c70 legacy=0x7fda240cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:48.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.908 7fda21e1b700 1 -- 172.21.2.112:0/1490243088 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda240f0840 con 0x7fda241b0a10 2021-02-10T11:35:48.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.908 7fda21e1b700 1 --2- 172.21.2.112:0/1490243088 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fda241b0a10 0x7fda240cf330 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 rx=0x7fda14006700 tx=0x7fda14009b80).ready entity=mon.0 client_cookie=9b5724cd7f0836c9 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:48.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.908 7fda2161a700 1 -- 172.21.2.112:0/1490243088 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fda1400c0c0 con 0x7fda241b0a10 2021-02-10T11:35:48.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.908 7fda2161a700 1 -- 172.21.2.112:0/1490243088 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda1400c340 con 0x7fda241b0a10 2021-02-10T11:35:48.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.908 7fda2161a700 1 -- 172.21.2.112:0/1490243088 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fda1400c7a0 con 0x7fda241b0a10 2021-02-10T11:35:48.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.908 7fda28884700 1 -- 172.21.2.112:0/1490243088 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fda241b0a10 msgr2=0x7fda240cf330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:48.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.908 7fda28884700 1 --2- 172.21.2.112:0/1490243088 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fda241b0a10 0x7fda240cf330 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 rx=0x7fda14006700 tx=0x7fda14009b80).stop 2021-02-10T11:35:48.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.909 7fda28884700 1 -- 172.21.2.112:0/1490243088 shutdown_connections 2021-02-10T11:35:48.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.909 7fda28884700 1 --2- 172.21.2.112:0/1490243088 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fda241b0a10 0x7fda240cf330 unknown :-1 s=CLOSED pgs=1521 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:48.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.909 7fda28884700 1 --2- 172.21.2.112:0/1490243088 >> v2:172.21.2.120:3300/0 conn(0x7fda240d1870 0x7fda241b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:48.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.909 7fda28884700 1 -- 172.21.2.112:0/1490243088 >> 172.21.2.112:0/1490243088 conn(0x7fda240ce450 msgr2=0x7fda240d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:48.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.909 7fda28884700 1 -- 172.21.2.112:0/1490243088 shutdown_connections 2021-02-10T11:35:48.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.909 7fda28884700 1 -- 172.21.2.112:0/1490243088 wait complete. 2021-02-10T11:35:48.913 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.923 7fda28884700 1 Processor -- start 2021-02-10T11:35:48.913 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.923 7fda28884700 1 -- start start 2021-02-10T11:35:48.913 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.924 7fda28884700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fda240b4f30 0x7fda2418b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:48.913 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.924 7fda28884700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fda24093f40 0x7fda24094340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:48.914 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.924 7fda28884700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fda24094960 0x7fda24187e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:48.914 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.924 7fda28884700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda24094e90 con 0x7fda24094960 2021-02-10T11:35:48.914 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.924 7fda28884700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda2418bff0 con 0x7fda24093f40 2021-02-10T11:35:48.914 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.924 7fda28884700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda2408f780 con 0x7fda240b4f30 2021-02-10T11:35:48.914 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.924 7fda22e1d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fda24094960 0x7fda24187e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:48.914 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.924 7fda22e1d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fda24094960 0x7fda24187e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:51464/0 (socket says 172.21.2.112:51464) 2021-02-10T11:35:48.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.924 7fda22e1d700 1 -- 172.21.2.112:0/217127856 learned_addr learned my addr 172.21.2.112:0/217127856 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:35:48.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.924 7fda22e1d700 1 -- 172.21.2.112:0/217127856 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fda240b4f30 msgr2=0x7fda2418b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:48.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.924 7fda22e1d700 1 --2- 172.21.2.112:0/217127856 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fda240b4f30 0x7fda2418b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:48.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.924 7fda22e1d700 1 -- 172.21.2.112:0/217127856 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fda24093f40 msgr2=0x7fda24094340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:48.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.924 7fda22e1d700 1 --2- 172.21.2.112:0/217127856 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fda24093f40 0x7fda24094340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:48.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.924 7fda22e1d700 1 -- 172.21.2.112:0/217127856 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda24188470 con 0x7fda24094960 2021-02-10T11:35:48.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.925 7fda22e1d700 1 --2- 172.21.2.112:0/217127856 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fda24094960 0x7fda24187e50 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 rx=0x7fda1c009940 tx=0x7fda1c00bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:48.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.925 7fda12ffd700 1 -- 172.21.2.112:0/217127856 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fda1c000f90 con 0x7fda24094960 2021-02-10T11:35:48.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.925 7fda12ffd700 1 -- 172.21.2.112:0/217127856 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda1c00c2f0 con 0x7fda24094960 2021-02-10T11:35:48.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.925 7fda12ffd700 1 -- 172.21.2.112:0/217127856 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fda1c00c750 con 0x7fda24094960 2021-02-10T11:35:48.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.925 7fda28884700 1 -- 172.21.2.112:0/217127856 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda24188470 con 0x7fda24094960 2021-02-10T11:35:48.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.925 7fda28884700 1 -- 172.21.2.112:0/217127856 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda24188470 con 0x7fda24094960 2021-02-10T11:35:48.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.925 7fda12ffd700 1 -- 172.21.2.112:0/217127856 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fda1c001220 con 0x7fda24094960 2021-02-10T11:35:48.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.925 7fda12ffd700 1 --2- 172.21.2.112:0/217127856 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fda080186c0 0x7fda0801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:48.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.926 7fda12ffd700 1 -- 172.21.2.112:0/217127856 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fda1c032930 con 0x7fda24094960 2021-02-10T11:35:48.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.926 7fda2261c700 1 --2- 172.21.2.112:0/217127856 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fda080186c0 0x7fda0801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:48.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.927 7fda2261c700 1 --2- 172.21.2.112:0/217127856 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fda080186c0 0x7fda0801abc0 crc :-1 s=READY pgs=735 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:48.944 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.957 7fda28884700 1 -- 172.21.2.112:0/217127856 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda2418bff0 con 0x7fda24094960 2021-02-10T11:35:48.946 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:48.960 7fda12ffd700 1 -- 172.21.2.112:0/217127856 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fda1c00be30 con 0x7fda24094960 2021-02-10T11:35:49.073 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:49.087 7fda28884700 1 -- 172.21.2.112:0/217127856 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fda2418bff0 con 0x7fda24094960 2021-02-10T11:35:49.074 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:49.087 7fda12ffd700 1 -- 172.21.2.112:0/217127856 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) v1 ==== 94+0+2101 (secure 0 0 0) 0x7fda1c01f8c0 con 0x7fda24094960 2021-02-10T11:35:49.074 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 48 2021-02-10T11:35:49.075 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:35:49.075 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":48,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8334,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/3370618110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":3370618110},{"type":"v1","addr":"172.21.2.112:6833","nonce":3370618110}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":44}],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:34:45.582177","modified":"2021-02-10 11:34:47.589305","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5921},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5921":{"gid":5921,"name":"b","rank":0,"incarnation":45,"state":"up:active","state_seq":3,"addr":"172.21.2.120:6835/2436135065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":2436135065},{"type":"v1","addr":"172.21.2.120:6835","nonce":2436135065}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2021-02-10T11:35:49.076 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:49.089 7fda28884700 1 -- 172.21.2.112:0/217127856 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fda080186c0 msgr2=0x7fda0801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:49.076 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:49.089 7fda28884700 1 --2- 172.21.2.112:0/217127856 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fda080186c0 0x7fda0801abc0 crc :-1 s=READY pgs=735 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:49.076 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:49.089 7fda28884700 1 -- 172.21.2.112:0/217127856 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fda24094960 msgr2=0x7fda24187e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:49.076 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:49.089 7fda28884700 1 --2- 172.21.2.112:0/217127856 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fda24094960 0x7fda24187e50 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 rx=0x7fda1c009940 tx=0x7fda1c00bcf0).stop 2021-02-10T11:35:49.076 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:49.089 7fda28884700 1 -- 172.21.2.112:0/217127856 shutdown_connections 2021-02-10T11:35:49.076 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:49.090 7fda28884700 1 --2- 172.21.2.112:0/217127856 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fda080186c0 0x7fda0801abc0 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:49.077 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:49.090 7fda28884700 1 --2- 172.21.2.112:0/217127856 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fda24093f40 0x7fda24094340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:49.077 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:49.090 7fda28884700 1 --2- 172.21.2.112:0/217127856 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fda24094960 0x7fda24187e50 unknown :-1 s=CLOSED pgs=1522 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:49.077 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:49.090 7fda28884700 1 --2- 172.21.2.112:0/217127856 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fda240b4f30 0x7fda2418b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:49.077 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:49.090 7fda28884700 1 -- 172.21.2.112:0/217127856 >> 172.21.2.112:0/217127856 conn(0x7fda2419f8f0 msgr2=0x7fda241a5f80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:49.077 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:49.090 7fda28884700 1 -- 172.21.2.112:0/217127856 shutdown_connections 2021-02-10T11:35:49.077 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:49.090 7fda28884700 1 -- 172.21.2.112:0/217127856 wait complete. 2021-02-10T11:35:49.085 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs fail cephfs 2021-02-10T11:35:49.512 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:49.509 7fe8c1554700 1 Processor -- start 2021-02-10T11:35:49.512 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:49.509 7fe8c1554700 1 -- start start 2021-02-10T11:35:49.513 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:49.509 7fe8c1554700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe8bc0d0340 0x7fe8bc0ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:49.513 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:49.509 7fe8c1554700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8bc0f0b10 con 0x7fe8bc0d1c70 2021-02-10T11:35:49.513 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:49.509 7fe8c1554700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8bc1b0be0 con 0x7fe8bc0d1870 2021-02-10T11:35:49.513 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:49.509 7fe8c1554700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8bc0e5ed0 con 0x7fe8bc0d0340 2021-02-10T11:35:49.513 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:49.509 7fe8bbac0700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fe8bc0d1c70 0x7fe8bc0efc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:50934/0 (socket says 172.21.2.112:50934) 2021-02-10T11:35:49.513 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:49.509 7fe8bbac0700 1 -- 172.21.2.112:0/923585543 learned_addr learned my addr 172.21.2.112:0/923585543 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:35:49.513 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:49.509 7fe8ba2bd700 1 -- 172.21.2.112:0/923585543 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2232688954 0 0) 0x7fe8b4001500 con 0x7fe8bc0d1c70 2021-02-10T11:35:49.513 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:49.510 7fe8ba2bd700 1 -- 172.21.2.112:0/923585543 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8a8002fe0 con 0x7fe8bc0d1c70 2021-02-10T11:35:49.513 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:49.510 7fe8ba2bd700 1 -- 172.21.2.112:0/923585543 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2000242903 0 0) 0x7fe8b4001460 con 0x7fe8bc0d1c70 2021-02-10T11:35:49.513 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:49.510 7fe8ba2bd700 1 -- 172.21.2.112:0/923585543 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe8bc0d0340 msgr2=0x7fe8bc0ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:49.513 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:49.510 7fe8ba2bd700 1 --2- 172.21.2.112:0/923585543 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe8bc0d0340 0x7fe8bc0ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:49.513 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:49.510 7fe8ba2bd700 1 -- 172.21.2.112:0/923585543 >> v1:172.21.2.120:6789/0 conn(0x7fe8bc0d1870 legacy=0x7fe8bc1b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:49.514 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:49.510 7fe8ba2bd700 1 -- 172.21.2.112:0/923585543 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8bc0f0450 con 0x7fe8bc0d1c70 2021-02-10T11:35:49.514 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:49.510 7fe8ba2bd700 1 -- 172.21.2.112:0/923585543 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7fe8b4001980 con 0x7fe8bc0d1c70 2021-02-10T11:35:49.514 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:49.510 7fe8ba2bd700 1 -- 172.21.2.112:0/923585543 >> v1:172.21.2.112:6789/0 conn(0x7fe8bc0d1c70 legacy=0x7fe8bc0efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:49.514 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:49.510 7fe8ba2bd700 1 --2- 172.21.2.112:0/923585543 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe8a8003a60 0x7fe8a8005ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:49.514 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:49.510 7fe8ba2bd700 1 --2- 172.21.2.112:0/923585543 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe8a8006770 0x7fe8a8008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:49.514 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:49.510 7fe8ba2bd700 1 --2- 172.21.2.112:0/923585543 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe8a8009590 0x7fe8a800ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:49.514 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:49.510 7fe8ba2bd700 1 -- 172.21.2.112:0/923585543 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8a800c2d0 con 0x7fe8a8006770 2021-02-10T11:35:49.514 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:49.510 7fe8ba2bd700 1 -- 172.21.2.112:0/923585543 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8a800c4f0 con 0x7fe8a8003a60 2021-02-10T11:35:49.514 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:49.510 7fe8ba2bd700 1 -- 172.21.2.112:0/923585543 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8a800c710 con 0x7fe8a8009590 2021-02-10T11:35:49.514 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:49.510 7fe8baabe700 1 --2- 172.21.2.112:0/923585543 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe8a8006770 0x7fe8a8008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:49.514 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:49.510 7fe8bb2bf700 1 --2- 172.21.2.112:0/923585543 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe8a8003a60 0x7fe8a8005ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:49.514 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:49.510 7fe8bbac0700 1 --2- 172.21.2.112:0/923585543 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe8a8009590 0x7fe8a800ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:49.514 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:49.510 7fe8baabe700 1 -- 172.21.2.112:0/923585543 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe8a8009590 msgr2=0x7fe8a800ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:49.515 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:49.510 7fe8baabe700 1 --2- 172.21.2.112:0/923585543 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe8a8009590 0x7fe8a800ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:49.515 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:49.510 7fe8baabe700 1 -- 172.21.2.112:0/923585543 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe8a8003a60 msgr2=0x7fe8a8005ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:49.515 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:49.510 7fe8baabe700 1 --2- 172.21.2.112:0/923585543 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe8a8003a60 0x7fe8a8005ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:49.515 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:49.510 7fe8baabe700 1 -- 172.21.2.112:0/923585543 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8a800c9b0 con 0x7fe8a8006770 2021-02-10T11:35:49.515 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:49.510 7fe8bbac0700 1 --2- 172.21.2.112:0/923585543 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe8a8009590 0x7fe8a800ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:35:49.515 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:49.510 7fe8baabe700 1 --2- 172.21.2.112:0/923585543 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe8a8006770 0x7fe8a8008c90 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 rx=0x7fe8a4006ab0 tx=0x7fe8a404fbc0).ready entity=mon.0 client_cookie=d62a4cff0da42ae1 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:49.515 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:49.511 7fe8ba2bd700 1 -- 172.21.2.112:0/923585543 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe8a4002080 con 0x7fe8a8006770 2021-02-10T11:35:49.515 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:49.511 7fe8ba2bd700 1 -- 172.21.2.112:0/923585543 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8a40063f0 con 0x7fe8a8006770 2021-02-10T11:35:49.515 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:49.511 7fe8ba2bd700 1 -- 172.21.2.112:0/923585543 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe8a4002380 con 0x7fe8a8006770 2021-02-10T11:35:49.515 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:49.511 7fe8c1554700 1 -- 172.21.2.112:0/923585543 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe8a8006770 msgr2=0x7fe8a8008c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:49.515 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:49.511 7fe8c1554700 1 --2- 172.21.2.112:0/923585543 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe8a8006770 0x7fe8a8008c90 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 rx=0x7fe8a4006ab0 tx=0x7fe8a404fbc0).stop 2021-02-10T11:35:49.516 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:49.511 7fe8bbac0700 1 -- 172.21.2.112:0/923585543 reap_dead start 2021-02-10T11:35:49.516 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:49.511 7fe8c1554700 1 -- 172.21.2.112:0/923585543 shutdown_connections 2021-02-10T11:35:49.516 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:49.511 7fe8c1554700 1 -- 172.21.2.112:0/923585543 >> 172.21.2.112:0/923585543 conn(0x7fe8bc0ce450 msgr2=0x7fe8bc0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:49.516 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:49.511 7fe8c1554700 1 -- 172.21.2.112:0/923585543 shutdown_connections 2021-02-10T11:35:49.516 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:49.511 7fe8c1554700 1 -- 172.21.2.112:0/923585543 wait complete. 2021-02-10T11:35:49.516 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:49.526 7fe8c1554700 1 Processor -- start 2021-02-10T11:35:49.516 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:49.526 7fe8c1554700 1 -- start start 2021-02-10T11:35:49.517 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:49.526 7fe8c1554700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe8bc19ffa0 0x7fe8bc0eda10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:49.517 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:49.526 7fe8c1554700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe8bc1a03a0 0x7fe8bc0ee030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:49.518 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:49.526 7fe8c1554700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe8bc0ee650 0x7fe8bc0ebe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:49.518 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:49.526 7fe8c1554700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8bc0ed490 con 0x7fe8bc0ee650 2021-02-10T11:35:49.518 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:49.526 7fe8c1554700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8bc0eea50 con 0x7fe8bc1a03a0 2021-02-10T11:35:49.518 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:49.526 7fe8c1554700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8bc0eec30 con 0x7fe8bc19ffa0 2021-02-10T11:35:49.518 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:49.527 7fe8bbac0700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe8bc0ee650 0x7fe8bc0ebe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:49.518 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:49.527 7fe8bbac0700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe8bc0ee650 0x7fe8bc0ebe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:51484/0 (socket says 172.21.2.112:51484) 2021-02-10T11:35:49.518 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:49.527 7fe8bbac0700 1 -- 172.21.2.112:0/605545452 learned_addr learned my addr 172.21.2.112:0/605545452 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:35:49.518 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:49.527 7fe8bbac0700 1 -- 172.21.2.112:0/605545452 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe8bc19ffa0 msgr2=0x7fe8bc0eda10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:49.518 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:49.527 7fe8bbac0700 1 --2- 172.21.2.112:0/605545452 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe8bc19ffa0 0x7fe8bc0eda10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:49.518 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:49.527 7fe8bbac0700 1 -- 172.21.2.112:0/605545452 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe8bc1a03a0 msgr2=0x7fe8bc0ee030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:49.518 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:49.527 7fe8bbac0700 1 --2- 172.21.2.112:0/605545452 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe8bc1a03a0 0x7fe8bc0ee030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:49.518 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:49.527 7fe8bbac0700 1 -- 172.21.2.112:0/605545452 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8bc0ec4a0 con 0x7fe8bc0ee650 2021-02-10T11:35:49.518 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:49.527 7fe8bbac0700 1 --2- 172.21.2.112:0/605545452 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe8bc0ee650 0x7fe8bc0ebe80 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 rx=0x7fe8b400a920 tx=0x7fe8b400acc0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:49.519 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:49.527 7fe8b3fff700 1 -- 172.21.2.112:0/605545452 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe8b40070d0 con 0x7fe8bc0ee650 2021-02-10T11:35:49.519 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:49.527 7fe8b3fff700 1 -- 172.21.2.112:0/605545452 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8b4001af0 con 0x7fe8bc0ee650 2021-02-10T11:35:49.519 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:49.527 7fe8b3fff700 1 -- 172.21.2.112:0/605545452 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe8b4004930 con 0x7fe8bc0ee650 2021-02-10T11:35:49.519 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:49.527 7fe8c1554700 1 -- 172.21.2.112:0/605545452 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8bc0ec4a0 con 0x7fe8bc0ee650 2021-02-10T11:35:49.519 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:49.527 7fe8c1554700 1 -- 172.21.2.112:0/605545452 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8bc0eca30 con 0x7fe8bc0ee650 2021-02-10T11:35:49.519 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:49.528 7fe8b3fff700 1 -- 172.21.2.112:0/605545452 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe8b4004720 con 0x7fe8bc0ee650 2021-02-10T11:35:49.519 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:49.528 7fe8b3fff700 1 --2- 172.21.2.112:0/605545452 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe8a00186c0 0x7fe8a001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:49.519 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:49.528 7fe8b3fff700 1 -- 172.21.2.112:0/605545452 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fe8b4032900 con 0x7fe8bc0ee650 2021-02-10T11:35:49.519 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:49.528 7fe8bb2bf700 1 --2- 172.21.2.112:0/605545452 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe8a00186c0 0x7fe8a001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:49.519 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:49.529 7fe8bb2bf700 1 --2- 172.21.2.112:0/605545452 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe8a00186c0 0x7fe8a001abc0 crc :-1 s=READY pgs=736 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:49.546 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:49.559 7fe8c1554700 1 -- 172.21.2.112:0/605545452 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8bc0ebc50 con 0x7fe8bc0ee650 2021-02-10T11:35:49.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:49.562 7fe8b3fff700 1 -- 172.21.2.112:0/605545452 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe8b4001d00 con 0x7fe8bc0ee650 2021-02-10T11:35:49.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:49.691 7fe8c1554700 1 -- 172.21.2.112:0/605545452 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fe8bc0ebc50 con 0x7fe8bc0ee650 2021-02-10T11:35:50.088 INFO:tasks.ceph.mon.a.gibba012.stderr:2021-02-10 11:35:50.102 7fc3fe6e2700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-02-10T11:35:50.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:50.104 7fe8b3fff700 1 -- 172.21.2.112:0/605545452 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v49) v1 ==== 161+0+0 (secure 0 0 0) 0x7fe8b4004720 con 0x7fe8bc0ee650 2021-02-10T11:35:50.093 INFO:teuthology.orchestra.run.gibba012.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2021-02-10T11:35:50.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:50.108 7fe8c1554700 1 -- 172.21.2.112:0/605545452 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe8a00186c0 msgr2=0x7fe8a001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:50.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:50.108 7fe8c1554700 1 --2- 172.21.2.112:0/605545452 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe8a00186c0 0x7fe8a001abc0 crc :-1 s=READY pgs=736 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:50.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:50.108 7fe8c1554700 1 -- 172.21.2.112:0/605545452 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe8bc0ee650 msgr2=0x7fe8bc0ebe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:50.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:50.108 7fe8c1554700 1 --2- 172.21.2.112:0/605545452 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe8bc0ee650 0x7fe8bc0ebe80 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 rx=0x7fe8b400a920 tx=0x7fe8b400acc0).stop 2021-02-10T11:35:50.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:50.108 7fe8c1554700 1 -- 172.21.2.112:0/605545452 shutdown_connections 2021-02-10T11:35:50.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:50.108 7fe8c1554700 1 --2- 172.21.2.112:0/605545452 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe8a00186c0 0x7fe8a001abc0 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:50.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:50.108 7fe8c1554700 1 --2- 172.21.2.112:0/605545452 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe8bc1a03a0 0x7fe8bc0ee030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:50.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:50.108 7fe8c1554700 1 --2- 172.21.2.112:0/605545452 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe8bc0ee650 0x7fe8bc0ebe80 unknown :-1 s=CLOSED pgs=1525 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:50.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:50.108 7fe8c1554700 1 --2- 172.21.2.112:0/605545452 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe8bc19ffa0 0x7fe8bc0eda10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:50.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:50.108 7fe8c1554700 1 -- 172.21.2.112:0/605545452 >> 172.21.2.112:0/605545452 conn(0x7fe8bc094d90 msgr2=0x7fe8bc08a420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:50.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:50.108 7fe8c1554700 1 -- 172.21.2.112:0/605545452 shutdown_connections 2021-02-10T11:35:50.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:50.108 7fe8c1554700 1 -- 172.21.2.112:0/605545452 wait complete. 2021-02-10T11:35:50.104 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:35:50.143 INFO:tasks.ceph.mds.b.gibba020.stdout:starting mds.b at 2021-02-10T11:35:50.530 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:50.527 7f7336808700 1 Processor -- start 2021-02-10T11:35:50.530 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:50.528 7f7336808700 1 -- start start 2021-02-10T11:35:50.530 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:50.528 7f7336808700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f73300d1870 0x7f73301b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:50.530 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:50.528 7f7336808700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f73301b0a10 0x7f73300cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:50.530 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:50.528 7f7336808700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f73300d1c70 0x7f73300cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:50.530 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:50.528 7f7336808700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73300e5ed0 con 0x7f73300d1870 2021-02-10T11:35:50.530 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:50.528 7f7336808700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73300f0b10 con 0x7f73301b0a10 2021-02-10T11:35:50.530 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:50.528 7f7336808700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73300cff70 con 0x7f73300d1c70 2021-02-10T11:35:50.531 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:50.528 7f732ffff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f73300d1870 0x7f73301b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:50.531 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:50.528 7f732ffff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f73300d1870 0x7f73301b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:51488/0 (socket says 172.21.2.112:51488) 2021-02-10T11:35:50.531 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:50.528 7f732ffff700 1 -- 172.21.2.112:0/3878455633 learned_addr learned my addr 172.21.2.112:0/3878455633 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:35:50.531 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:50.528 7f7334e25700 1 --2- 172.21.2.112:0/3878455633 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f73300d1c70 0x7f73300cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:50.531 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:50.528 7f732ffff700 1 -- 172.21.2.112:0/3878455633 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f73300d1c70 msgr2=0x7f73300cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:50.531 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:50.528 7f732ffff700 1 --2- 172.21.2.112:0/3878455633 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f73300d1c70 0x7f73300cf950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:50.531 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:50.528 7f732ffff700 1 -- 172.21.2.112:0/3878455633 >> v2:172.21.2.120:3300/0 conn(0x7f73301b0a10 msgr2=0x7f73300cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:50.531 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:50.528 7f732ffff700 1 --2- 172.21.2.112:0/3878455633 >> v2:172.21.2.120:3300/0 conn(0x7f73301b0a10 0x7f73300cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:50.531 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:50.528 7f732ffff700 1 -- 172.21.2.112:0/3878455633 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73300d0150 con 0x7f73300d1870 2021-02-10T11:35:50.531 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:50.528 7f7334e25700 1 --2- 172.21.2.112:0/3878455633 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f73300d1c70 0x7f73300cf950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:35:50.531 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:50.529 7f732ffff700 1 --2- 172.21.2.112:0/3878455633 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f73300d1870 0x7f73301b03f0 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 rx=0x7f7320006700 tx=0x7f7320009b80).ready entity=mon.0 client_cookie=57442cc39c9e5b1f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:50.531 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:50.529 7f732effd700 1 -- 172.21.2.112:0/3878455633 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f732000c0c0 con 0x7f73300d1870 2021-02-10T11:35:50.531 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:50.529 7f732effd700 1 -- 172.21.2.112:0/3878455633 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f732000c440 con 0x7f73300d1870 2021-02-10T11:35:50.532 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:50.529 7f732effd700 1 -- 172.21.2.112:0/3878455633 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f732000c8a0 con 0x7f73300d1870 2021-02-10T11:35:50.532 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:50.529 7f7336808700 1 -- 172.21.2.112:0/3878455633 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f73300d1870 msgr2=0x7f73301b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:50.532 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:50.529 7f7336808700 1 --2- 172.21.2.112:0/3878455633 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f73300d1870 0x7f73301b03f0 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 rx=0x7f7320006700 tx=0x7f7320009b80).stop 2021-02-10T11:35:50.532 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:50.529 7f7336808700 1 -- 172.21.2.112:0/3878455633 shutdown_connections 2021-02-10T11:35:50.532 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:50.529 7f7336808700 1 --2- 172.21.2.112:0/3878455633 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f73300d1c70 0x7f73300cf950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:50.532 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:50.529 7f7336808700 1 --2- 172.21.2.112:0/3878455633 >> v2:172.21.2.120:3300/0 conn(0x7f73301b0a10 0x7f73300cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:50.532 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:50.529 7f7336808700 1 --2- 172.21.2.112:0/3878455633 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f73300d1870 0x7f73301b03f0 unknown :-1 s=CLOSED pgs=1526 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:50.532 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:50.529 7f7336808700 1 -- 172.21.2.112:0/3878455633 >> 172.21.2.112:0/3878455633 conn(0x7f73300ce450 msgr2=0x7f73300d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:50.532 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:50.529 7f7336808700 1 -- 172.21.2.112:0/3878455633 shutdown_connections 2021-02-10T11:35:50.532 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:50.529 7f7336808700 1 -- 172.21.2.112:0/3878455633 wait complete. 2021-02-10T11:35:50.534 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:50.544 7f7336808700 1 Processor -- start 2021-02-10T11:35:50.534 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:50.544 7f7336808700 1 -- start start 2021-02-10T11:35:50.534 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:50.544 7f7336808700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f73301942b0 0x7f73300b3550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:50.534 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:50.544 7f7336808700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7330195070 0x7f7330195470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:50.534 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:50.544 7f7336808700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f73300bb8e0 0x7f73300bbce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:50.534 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:50.544 7f7336808700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73301946b0 con 0x7f73300bb8e0 2021-02-10T11:35:50.534 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:50.544 7f7336808700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73300b3b70 con 0x7f7330195070 2021-02-10T11:35:50.534 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:50.544 7f7336808700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73300b3d90 con 0x7f73301942b0 2021-02-10T11:35:50.534 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:50.544 7f7334e25700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f73300bb8e0 0x7f73300bbce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:50.534 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:50.544 7f7334e25700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f73300bb8e0 0x7f73300bbce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:51494/0 (socket says 172.21.2.112:51494) 2021-02-10T11:35:50.535 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:50.544 7f7334e25700 1 -- 172.21.2.112:0/2463730615 learned_addr learned my addr 172.21.2.112:0/2463730615 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:35:50.535 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:50.545 7f7334e25700 1 -- 172.21.2.112:0/2463730615 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f73301942b0 msgr2=0x7f73300b3550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:50.535 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:50.545 7f7334e25700 1 --2- 172.21.2.112:0/2463730615 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f73301942b0 0x7f73300b3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:50.535 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:50.545 7f7334e25700 1 -- 172.21.2.112:0/2463730615 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7330195070 msgr2=0x7f7330195470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:50.535 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:50.545 7f732f7fe700 1 --2- 172.21.2.112:0/2463730615 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7330195070 0x7f7330195470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:50.535 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:50.545 7f7334e25700 1 --2- 172.21.2.112:0/2463730615 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7330195070 0x7f7330195470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:50.535 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:50.545 7f7334e25700 1 -- 172.21.2.112:0/2463730615 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73301a49e0 con 0x7f73300bb8e0 2021-02-10T11:35:50.535 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:50.545 7f732f7fe700 1 --2- 172.21.2.112:0/2463730615 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7330195070 0x7f7330195470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:35:50.535 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:50.545 7f7334e25700 1 --2- 172.21.2.112:0/2463730615 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f73300bb8e0 0x7f73300bbce0 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 rx=0x7f73280099a0 tx=0x7f732800bc70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:50.535 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:50.545 7f732cff9700 1 -- 172.21.2.112:0/2463730615 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7328006030 con 0x7f73300bb8e0 2021-02-10T11:35:50.535 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:50.545 7f732cff9700 1 -- 172.21.2.112:0/2463730615 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7328006270 con 0x7f73300bb8e0 2021-02-10T11:35:50.535 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:50.545 7f7336808700 1 -- 172.21.2.112:0/2463730615 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73301a49e0 con 0x7f73300bb8e0 2021-02-10T11:35:50.535 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:50.545 7f732cff9700 1 -- 172.21.2.112:0/2463730615 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f73280056b0 con 0x7f73300bb8e0 2021-02-10T11:35:50.535 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:50.545 7f7336808700 1 -- 172.21.2.112:0/2463730615 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73301a49e0 con 0x7f73300bb8e0 2021-02-10T11:35:50.536 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:50.545 7f732cff9700 1 -- 172.21.2.112:0/2463730615 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7328006950 con 0x7f73300bb8e0 2021-02-10T11:35:50.536 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:50.545 7f732cff9700 1 --2- 172.21.2.112:0/2463730615 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7310018730 0x7f731001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:50.536 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:50.546 7f732cff9700 1 -- 172.21.2.112:0/2463730615 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5659+0+0 (secure 0 0 0) 0x7f7328032a00 con 0x7f73300bb8e0 2021-02-10T11:35:50.536 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:50.546 7f732ffff700 1 --2- 172.21.2.112:0/2463730615 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7310018730 0x7f731001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:50.536 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:50.546 7f732ffff700 1 --2- 172.21.2.112:0/2463730615 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7310018730 0x7f731001ac10 crc :-1 s=READY pgs=737 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:50.564 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:50.578 7f7336808700 1 -- 172.21.2.112:0/2463730615 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73300bc5f0 con 0x7f73300bb8e0 2021-02-10T11:35:50.567 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:50.580 7f732cff9700 1 -- 172.21.2.112:0/2463730615 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f732800bdb0 con 0x7f73300bb8e0 2021-02-10T11:35:50.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:50.709 7f7336808700 1 -- 172.21.2.112:0/2463730615 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f73300bc5f0 con 0x7f73300bb8e0 2021-02-10T11:35:50.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:50.709 7f732cff9700 1 -- 172.21.2.112:0/2463730615 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) v1 ==== 94+0+1756 (secure 0 0 0) 0x7f7328006270 con 0x7f73300bb8e0 2021-02-10T11:35:50.697 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:35:50.697 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":49,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8334,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/3370618110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":3370618110},{"type":"v1","addr":"172.21.2.112:6833","nonce":3370618110}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":44}],"filesystems":[{"mdsmap":{"epoch":49,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:34:45.582177","modified":"2021-02-10 11:35:50.102388","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2021-02-10T11:35:50.697 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 49 2021-02-10T11:35:50.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:50.711 7f7336808700 1 -- 172.21.2.112:0/2463730615 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7310018730 msgr2=0x7f731001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:50.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:50.711 7f7336808700 1 --2- 172.21.2.112:0/2463730615 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7310018730 0x7f731001ac10 crc :-1 s=READY pgs=737 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:50.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:50.711 7f7336808700 1 -- 172.21.2.112:0/2463730615 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f73300bb8e0 msgr2=0x7f73300bbce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:50.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:50.711 7f7336808700 1 --2- 172.21.2.112:0/2463730615 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f73300bb8e0 0x7f73300bbce0 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 rx=0x7f73280099a0 tx=0x7f732800bc70).stop 2021-02-10T11:35:50.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:50.711 7f7336808700 1 -- 172.21.2.112:0/2463730615 shutdown_connections 2021-02-10T11:35:50.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:50.711 7f7336808700 1 --2- 172.21.2.112:0/2463730615 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7310018730 0x7f731001ac10 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:50.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:50.711 7f7336808700 1 --2- 172.21.2.112:0/2463730615 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7330195070 0x7f7330195470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:50.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:50.712 7f7336808700 1 --2- 172.21.2.112:0/2463730615 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f73300bb8e0 0x7f73300bbce0 unknown :-1 s=CLOSED pgs=1527 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:50.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:50.712 7f7336808700 1 --2- 172.21.2.112:0/2463730615 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f73301942b0 0x7f73300b3550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:50.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:50.712 7f7336808700 1 -- 172.21.2.112:0/2463730615 >> 172.21.2.112:0/2463730615 conn(0x7f73300b3040 msgr2=0x7f73300c3250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:50.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:50.712 7f7336808700 1 -- 172.21.2.112:0/2463730615 shutdown_connections 2021-02-10T11:35:50.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:50.712 7f7336808700 1 -- 172.21.2.112:0/2463730615 wait complete. 2021-02-10T11:35:50.706 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2021-02-10T11:35:51.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:51.134 7f33bd2e7700 1 Processor -- start 2021-02-10T11:35:51.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:51.134 7f33bd2e7700 1 -- start start 2021-02-10T11:35:51.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:51.135 7f33bd2e7700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f33b80d1870 0x7f33b81b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:51.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:51.135 7f33bd2e7700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f33b81b0a10 0x7f33b80cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:51.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:51.135 7f33bd2e7700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f33b80d1c70 0x7f33b80cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:51.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:51.135 7f33bd2e7700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33b80fe200 con 0x7f33b80d1c70 2021-02-10T11:35:51.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:51.135 7f33bd2e7700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33b80e5f10 con 0x7f33b80d1870 2021-02-10T11:35:51.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:51.135 7f33bd2e7700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33b80f0b50 con 0x7f33b81b0a10 2021-02-10T11:35:51.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:51.135 7f33b707d700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f33b80d1870 0x7f33b81b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:51.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:51.135 7f33b707d700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f33b80d1870 0x7f33b81b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3300/0 says I am v2:172.21.2.112:53780/0 (socket says 172.21.2.112:53780) 2021-02-10T11:35:51.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:51.135 7f33b787e700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f33b80d1c70 0x7f33b80cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:51.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:51.135 7f33b707d700 1 -- 172.21.2.112:0/3359108868 learned_addr learned my addr 172.21.2.112:0/3359108868 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:35:51.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:51.135 7f33ae87c700 1 --2- 172.21.2.112:0/3359108868 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f33b81b0a10 0x7f33b80cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:51.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:51.135 7f33b707d700 1 -- 172.21.2.112:0/3359108868 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f33b81b0a10 msgr2=0x7f33b80cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:51.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:51.135 7f33b707d700 1 --2- 172.21.2.112:0/3359108868 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f33b81b0a10 0x7f33b80cf330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:51.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:51.135 7f33b707d700 1 -- 172.21.2.112:0/3359108868 >> v2:172.21.2.112:3300/0 conn(0x7f33b80d1c70 msgr2=0x7f33b80cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:51.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:51.135 7f33b707d700 1 --2- 172.21.2.112:0/3359108868 >> v2:172.21.2.112:3300/0 conn(0x7f33b80d1c70 0x7f33b80cf950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:51.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:51.135 7f33b707d700 1 -- 172.21.2.112:0/3359108868 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33b80cff70 con 0x7f33b80d1870 2021-02-10T11:35:51.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:51.135 7f33b707d700 1 --2- 172.21.2.112:0/3359108868 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f33b80d1870 0x7f33b81b03f0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 rx=0x7f33a8006700 tx=0x7f33a8009b80).ready entity=mon.1 client_cookie=f8765605603d7f16 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:51.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:51.136 7f33b687c700 1 -- 172.21.2.112:0/3359108868 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f33a800c0c0 con 0x7f33b80d1870 2021-02-10T11:35:51.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:51.136 7f33b687c700 1 -- 172.21.2.112:0/3359108868 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33a800c440 con 0x7f33b80d1870 2021-02-10T11:35:51.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:51.136 7f33b687c700 1 -- 172.21.2.112:0/3359108868 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f33a800c960 con 0x7f33b80d1870 2021-02-10T11:35:51.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:51.136 7f33bd2e7700 1 -- 172.21.2.112:0/3359108868 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f33b80d1870 msgr2=0x7f33b81b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:51.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:51.136 7f33bd2e7700 1 --2- 172.21.2.112:0/3359108868 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f33b80d1870 0x7f33b81b03f0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 rx=0x7f33a8006700 tx=0x7f33a8009b80).stop 2021-02-10T11:35:51.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:51.136 7f33bd2e7700 1 -- 172.21.2.112:0/3359108868 shutdown_connections 2021-02-10T11:35:51.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:51.136 7f33bd2e7700 1 --2- 172.21.2.112:0/3359108868 >> v2:172.21.2.112:3300/0 conn(0x7f33b80d1c70 0x7f33b80cf950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:51.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:51.136 7f33bd2e7700 1 --2- 172.21.2.112:0/3359108868 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f33b81b0a10 0x7f33b80cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:51.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:51.136 7f33bd2e7700 1 --2- 172.21.2.112:0/3359108868 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f33b80d1870 0x7f33b81b03f0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:51.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:51.136 7f33bd2e7700 1 -- 172.21.2.112:0/3359108868 >> 172.21.2.112:0/3359108868 conn(0x7f33b80ce450 msgr2=0x7f33b80d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:51.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:51.136 7f33bd2e7700 1 -- 172.21.2.112:0/3359108868 shutdown_connections 2021-02-10T11:35:51.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:51.137 7f33bd2e7700 1 -- 172.21.2.112:0/3359108868 wait complete. 2021-02-10T11:35:51.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:51.151 7f33bd2e7700 1 Processor -- start 2021-02-10T11:35:51.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:51.151 7f33bd2e7700 1 -- start start 2021-02-10T11:35:51.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:51.151 7f33bd2e7700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f33b8094310 0x7f33b80ebe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:51.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:51.151 7f33bd2e7700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f33b8094710 0x7f33b80bca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:51.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:51.151 7f33bd2e7700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f33b80ec490 0x7f33b80bd050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:51.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:51.151 7f33bd2e7700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33b8093d90 con 0x7f33b8094710 2021-02-10T11:35:51.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:51.151 7f33bd2e7700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33b80bd670 con 0x7f33b80ec490 2021-02-10T11:35:51.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:51.151 7f33bd2e7700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33b80939f0 con 0x7f33b8094310 2021-02-10T11:35:51.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:51.152 7f33ae87c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f33b8094710 0x7f33b80bca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:51.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:51.152 7f33b787e700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f33b80ec490 0x7f33b80bd050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:51.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:51.152 7f33ae87c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f33b8094710 0x7f33b80bca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:51510/0 (socket says 172.21.2.112:51510) 2021-02-10T11:35:51.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:51.152 7f33ae87c700 1 -- 172.21.2.112:0/1374411123 learned_addr learned my addr 172.21.2.112:0/1374411123 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:35:51.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:51.152 7f33b707d700 1 --2- 172.21.2.112:0/1374411123 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f33b8094310 0x7f33b80ebe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:51.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:51.152 7f33b787e700 1 -- 172.21.2.112:0/1374411123 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f33b8094310 msgr2=0x7f33b80ebe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:51.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:51.152 7f33b787e700 1 --2- 172.21.2.112:0/1374411123 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f33b8094310 0x7f33b80ebe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:51.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:51.152 7f33b787e700 1 -- 172.21.2.112:0/1374411123 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f33b8094710 msgr2=0x7f33b80bca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:51.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:51.152 7f33b787e700 1 --2- 172.21.2.112:0/1374411123 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f33b8094710 0x7f33b80bca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:51.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:51.152 7f33b787e700 1 -- 172.21.2.112:0/1374411123 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33b80bda80 con 0x7f33b80ec490 2021-02-10T11:35:51.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:51.152 7f33b787e700 1 --2- 172.21.2.112:0/1374411123 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f33b80ec490 0x7f33b80bd050 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 rx=0x7f33b000a840 tx=0x7f33b000abe0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:51.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:51.152 7f33b4878700 1 -- 172.21.2.112:0/1374411123 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f33b000b070 con 0x7f33b80ec490 2021-02-10T11:35:51.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:51.152 7f33b4878700 1 -- 172.21.2.112:0/1374411123 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33b000b330 con 0x7f33b80ec490 2021-02-10T11:35:51.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:51.152 7f33b4878700 1 -- 172.21.2.112:0/1374411123 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f33b000b850 con 0x7f33b80ec490 2021-02-10T11:35:51.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:51.152 7f33bd2e7700 1 -- 172.21.2.112:0/1374411123 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33b80bda80 con 0x7f33b80ec490 2021-02-10T11:35:51.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:51.152 7f33bd2e7700 1 -- 172.21.2.112:0/1374411123 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33b80bda80 con 0x7f33b80ec490 2021-02-10T11:35:51.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:51.153 7f33b4878700 1 -- 172.21.2.112:0/1374411123 <== mon.1 v2:172.21.2.120:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f33b0000d30 con 0x7f33b80ec490 2021-02-10T11:35:51.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:51.153 7f33b4878700 1 --2- 172.21.2.112:0/1374411123 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3390018730 0x7f339001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:51.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:51.153 7f33b4878700 1 -- 172.21.2.112:0/1374411123 <== mon.1 v2:172.21.2.120:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5659+0+0 (secure 0 0 0) 0x7f33b0032980 con 0x7f33b80ec490 2021-02-10T11:35:51.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:51.153 7f33b707d700 1 --2- 172.21.2.112:0/1374411123 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3390018730 0x7f339001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:51.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:51.153 7f33b707d700 1 --2- 172.21.2.112:0/1374411123 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3390018730 0x7f339001ac10 crc :-1 s=READY pgs=738 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:51.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:51.184 7f33bd2e7700 1 -- 172.21.2.112:0/1374411123 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33b80ebc40 con 0x7f33b80ec490 2021-02-10T11:35:51.173 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:51.187 7f33b4878700 1 -- 172.21.2.112:0/1374411123 <== mon.1 v2:172.21.2.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f33b001f7f0 con 0x7f33b80ec490 2021-02-10T11:35:51.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:51.313 7f33bd2e7700 1 -- 172.21.2.112:0/1374411123 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f33b80f0b10 con 0x7f33b80ec490 2021-02-10T11:35:52.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:52.117 7f33b4878700 1 -- 172.21.2.112:0/1374411123 <== mon.1 v2:172.21.2.120:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v50) v1 ==== 104+0+0 (secure 0 0 0) 0x7f33b0000d30 con 0x7f33b80ec490 2021-02-10T11:35:52.119 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:52.132 7f33bd2e7700 1 -- 172.21.2.112:0/1374411123 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3390018730 msgr2=0x7f339001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:52.119 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:52.132 7f33bd2e7700 1 --2- 172.21.2.112:0/1374411123 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3390018730 0x7f339001ac10 crc :-1 s=READY pgs=738 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:52.119 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:52.132 7f33bd2e7700 1 -- 172.21.2.112:0/1374411123 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f33b80ec490 msgr2=0x7f33b80bd050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:52.119 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:52.132 7f33bd2e7700 1 --2- 172.21.2.112:0/1374411123 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f33b80ec490 0x7f33b80bd050 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 rx=0x7f33b000a840 tx=0x7f33b000abe0).stop 2021-02-10T11:35:52.119 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:52.133 7f33bd2e7700 1 -- 172.21.2.112:0/1374411123 shutdown_connections 2021-02-10T11:35:52.120 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:52.133 7f33bd2e7700 1 --2- 172.21.2.112:0/1374411123 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3390018730 0x7f339001ac10 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:52.120 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:52.133 7f33bd2e7700 1 --2- 172.21.2.112:0/1374411123 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f33b80ec490 0x7f33b80bd050 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:52.120 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:52.133 7f33bd2e7700 1 --2- 172.21.2.112:0/1374411123 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f33b8094710 0x7f33b80bca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:52.120 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:52.133 7f33bd2e7700 1 --2- 172.21.2.112:0/1374411123 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f33b8094310 0x7f33b80ebe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:52.120 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:52.133 7f33bd2e7700 1 -- 172.21.2.112:0/1374411123 >> 172.21.2.112:0/1374411123 conn(0x7f33b808fa10 msgr2=0x7f33b80bfa60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:52.120 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:52.133 7f33bd2e7700 1 -- 172.21.2.112:0/1374411123 shutdown_connections 2021-02-10T11:35:52.120 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:52.133 7f33bd2e7700 1 -- 172.21.2.112:0/1374411123 wait complete. 2021-02-10T11:35:52.128 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2021-02-10T11:35:52.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:52.556 7f654293e700 1 Processor -- start 2021-02-10T11:35:52.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:52.556 7f654293e700 1 -- start start 2021-02-10T11:35:52.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:52.557 7f654293e700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f653c10b000 0x7f653c0cf760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:52.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:52.557 7f654293e700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f653c0fe1d0 con 0x7f653c0d1c80 2021-02-10T11:35:52.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:52.557 7f654293e700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f653c1b0cc0 con 0x7f653c0d1880 2021-02-10T11:35:52.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:52.557 7f654293e700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f653c109a70 con 0x7f653c10b000 2021-02-10T11:35:52.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:52.557 7f653b7fe700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f653c0d1c80 0x7f653c0cef70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:50976/0 (socket says 172.21.2.112:50976) 2021-02-10T11:35:52.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:52.557 7f653b7fe700 1 -- 172.21.2.112:0/4187878053 learned_addr learned my addr 172.21.2.112:0/4187878053 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:35:52.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:52.557 7f6540f5b700 1 --2- 172.21.2.112:0/4187878053 >> v2:172.21.2.120:3300/0 conn(0x7f653c10b000 0x7f653c0cf760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:52.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:52.558 7f653affd700 1 -- 172.21.2.112:0/4187878053 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1184407178 0 0) 0x7f65300009b0 con 0x7f653c0d1c80 2021-02-10T11:35:52.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:52.558 7f653affd700 1 -- 172.21.2.112:0/4187878053 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6524002fe0 con 0x7f653c0d1c80 2021-02-10T11:35:52.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:52.558 7f653affd700 1 -- 172.21.2.112:0/4187878053 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2100814019 0 0) 0x7f652c000ff0 con 0x7f653c0d1880 2021-02-10T11:35:52.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:52.558 7f653affd700 1 -- 172.21.2.112:0/4187878053 --> v1:172.21.2.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6524004370 con 0x7f653c0d1880 2021-02-10T11:35:52.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:52.558 7f6540f5b700 1 -- 172.21.2.112:0/4187878053 >> v1:172.21.2.120:6789/0 conn(0x7f653c0d1880 legacy=0x7f653c1b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:52.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:52.558 7f6540f5b700 1 -- 172.21.2.112:0/4187878053 >> v1:172.21.2.112:6789/0 conn(0x7f653c0d1c80 legacy=0x7f653c0cef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:52.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:52.558 7f6540f5b700 1 -- 172.21.2.112:0/4187878053 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f653c10b400 con 0x7f653c10b000 2021-02-10T11:35:52.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:52.558 7f6540f5b700 1 --2- 172.21.2.112:0/4187878053 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f653c10b000 0x7f653c0cf760 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 rx=0x7f6534006e50 tx=0x7f6534009e70).ready entity=mon.1 client_cookie=98a8dd0889eda617 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:52.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:52.558 7f653affd700 1 -- 172.21.2.112:0/4187878053 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6534006aa0 con 0x7f653c10b000 2021-02-10T11:35:52.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:52.558 7f653affd700 1 -- 172.21.2.112:0/4187878053 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6534006790 con 0x7f653c10b000 2021-02-10T11:35:52.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:52.558 7f653affd700 1 -- 172.21.2.112:0/4187878053 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f653400b700 con 0x7f653c10b000 2021-02-10T11:35:52.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:52.558 7f654293e700 1 -- 172.21.2.112:0/4187878053 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f653c10b000 msgr2=0x7f653c0cf760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:52.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:52.558 7f654293e700 1 --2- 172.21.2.112:0/4187878053 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f653c10b000 0x7f653c0cf760 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 rx=0x7f6534006e50 tx=0x7f6534009e70).stop 2021-02-10T11:35:52.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:52.559 7f654293e700 1 -- 172.21.2.112:0/4187878053 shutdown_connections 2021-02-10T11:35:52.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:52.559 7f654293e700 1 --2- 172.21.2.112:0/4187878053 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f653c10b000 0x7f653c0cf760 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:52.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:52.559 7f654293e700 1 -- 172.21.2.112:0/4187878053 >> 172.21.2.112:0/4187878053 conn(0x7f653c0ce430 msgr2=0x7f653c0d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:52.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:52.559 7f654293e700 1 -- 172.21.2.112:0/4187878053 shutdown_connections 2021-02-10T11:35:52.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:52.559 7f654293e700 1 -- 172.21.2.112:0/4187878053 wait complete. 2021-02-10T11:35:52.564 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:52.574 7f654293e700 1 Processor -- start 2021-02-10T11:35:52.564 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:52.574 7f654293e700 1 -- start start 2021-02-10T11:35:52.564 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:52.574 7f654293e700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f653c1891e0 0x7f653c19c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:52.564 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:52.574 7f654293e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f653c18e660 0x7f653c18ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:52.564 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:52.574 7f654293e700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f653c18f080 0x7f653c196640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:52.564 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:52.574 7f654293e700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f653c189700 con 0x7f653c18e660 2021-02-10T11:35:52.564 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:52.574 7f654293e700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f653c196c60 con 0x7f653c18f080 2021-02-10T11:35:52.564 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:52.574 7f654293e700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f653c196e80 con 0x7f653c1891e0 2021-02-10T11:35:52.564 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:52.575 7f653b7fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f653c18e660 0x7f653c18ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:52.565 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:52.575 7f653b7fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f653c18e660 0x7f653c18ea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:51522/0 (socket says 172.21.2.112:51522) 2021-02-10T11:35:52.565 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:52.575 7f653b7fe700 1 -- 172.21.2.112:0/3233532407 learned_addr learned my addr 172.21.2.112:0/3233532407 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:35:52.565 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:52.575 7f6540f5b700 1 --2- 172.21.2.112:0/3233532407 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f653c18f080 0x7f653c196640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:52.565 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:52.575 7f653bfff700 1 --2- 172.21.2.112:0/3233532407 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f653c1891e0 0x7f653c19c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:52.565 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:52.575 7f653b7fe700 1 -- 172.21.2.112:0/3233532407 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f653c1891e0 msgr2=0x7f653c19c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:52.565 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:52.575 7f653b7fe700 1 --2- 172.21.2.112:0/3233532407 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f653c1891e0 0x7f653c19c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:52.565 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:52.575 7f653b7fe700 1 -- 172.21.2.112:0/3233532407 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f653c18f080 msgr2=0x7f653c196640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:52.565 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:52.575 7f653b7fe700 1 --2- 172.21.2.112:0/3233532407 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f653c18f080 0x7f653c196640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:52.565 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:52.575 7f653b7fe700 1 -- 172.21.2.112:0/3233532407 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f653c1970a0 con 0x7f653c18e660 2021-02-10T11:35:52.565 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:52.575 7f653b7fe700 1 --2- 172.21.2.112:0/3233532407 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f653c18e660 0x7f653c18ea60 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 rx=0x7f6530001c60 tx=0x7f653000a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:52.565 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:52.575 7f6538ff9700 1 -- 172.21.2.112:0/3233532407 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f653000acc0 con 0x7f653c18e660 2021-02-10T11:35:52.565 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:52.575 7f6538ff9700 1 -- 172.21.2.112:0/3233532407 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6530004140 con 0x7f653c18e660 2021-02-10T11:35:52.565 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:52.575 7f6538ff9700 1 -- 172.21.2.112:0/3233532407 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6530004660 con 0x7f653c18e660 2021-02-10T11:35:52.566 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:52.576 7f654293e700 1 -- 172.21.2.112:0/3233532407 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f653c1970a0 con 0x7f653c18e660 2021-02-10T11:35:52.566 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:52.576 7f654293e700 1 -- 172.21.2.112:0/3233532407 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f653c195890 con 0x7f653c18e660 2021-02-10T11:35:52.566 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:52.576 7f6538ff9700 1 -- 172.21.2.112:0/3233532407 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6530004320 con 0x7f653c18e660 2021-02-10T11:35:52.566 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:52.576 7f6538ff9700 1 --2- 172.21.2.112:0/3233532407 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f651c0186c0 0x7f651c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:52.566 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:52.576 7f6538ff9700 1 -- 172.21.2.112:0/3233532407 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5659+0+0 (secure 0 0 0) 0x7f65300325a0 con 0x7f653c18e660 2021-02-10T11:35:52.566 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:52.576 7f653bfff700 1 --2- 172.21.2.112:0/3233532407 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f651c0186c0 0x7f651c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:52.566 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:52.577 7f653bfff700 1 --2- 172.21.2.112:0/3233532407 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f651c0186c0 0x7f651c01abc0 crc :-1 s=READY pgs=739 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:52.593 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:52.607 7f654293e700 1 -- 172.21.2.112:0/3233532407 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f653c0fe1d0 con 0x7f653c18e660 2021-02-10T11:35:52.596 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:52.609 7f6538ff9700 1 -- 172.21.2.112:0/3233532407 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6530000940 con 0x7f653c18e660 2021-02-10T11:35:52.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:52.736 7f654293e700 1 -- 172.21.2.112:0/3233532407 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f653c0fe1d0 con 0x7f653c18e660 2021-02-10T11:35:53.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:53.121 7f6538ff9700 1 -- 172.21.2.112:0/3233532407 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v66) v1 ==== 185+0+0 (secure 0 0 0) 0x7f653001f4f0 con 0x7f653c18e660 2021-02-10T11:35:53.139 INFO:teuthology.orchestra.run.gibba012.stderr:pool 'cephfs_metadata' removed 2021-02-10T11:35:53.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:53.154 7f654293e700 1 -- 172.21.2.112:0/3233532407 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f651c0186c0 msgr2=0x7f651c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:53.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:53.154 7f654293e700 1 --2- 172.21.2.112:0/3233532407 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f651c0186c0 0x7f651c01abc0 crc :-1 s=READY pgs=739 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:53.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:53.154 7f654293e700 1 -- 172.21.2.112:0/3233532407 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f653c18e660 msgr2=0x7f653c18ea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:53.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:53.154 7f654293e700 1 --2- 172.21.2.112:0/3233532407 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f653c18e660 0x7f653c18ea60 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 rx=0x7f6530001c60 tx=0x7f653000a8d0).stop 2021-02-10T11:35:53.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:53.155 7f654293e700 1 -- 172.21.2.112:0/3233532407 shutdown_connections 2021-02-10T11:35:53.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:53.155 7f654293e700 1 --2- 172.21.2.112:0/3233532407 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f651c0186c0 0x7f651c01abc0 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:53.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:53.155 7f654293e700 1 --2- 172.21.2.112:0/3233532407 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f653c18f080 0x7f653c196640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:53.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:53.155 7f654293e700 1 --2- 172.21.2.112:0/3233532407 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f653c18e660 0x7f653c18ea60 unknown :-1 s=CLOSED pgs=1529 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:53.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:53.155 7f654293e700 1 --2- 172.21.2.112:0/3233532407 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f653c1891e0 0x7f653c19c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:53.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:53.155 7f654293e700 1 -- 172.21.2.112:0/3233532407 >> 172.21.2.112:0/3233532407 conn(0x7f653c19f940 msgr2=0x7f653c19ba90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:53.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:53.155 7f654293e700 1 -- 172.21.2.112:0/3233532407 shutdown_connections 2021-02-10T11:35:53.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:53.155 7f654293e700 1 -- 172.21.2.112:0/3233532407 wait complete. 2021-02-10T11:35:53.150 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2021-02-10T11:35:53.580 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:53.577 7f51255e2700 1 Processor -- start 2021-02-10T11:35:53.580 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:53.577 7f51255e2700 1 -- start start 2021-02-10T11:35:53.580 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:53.578 7f51255e2700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f51200d1880 0x7f51201b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:53.580 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:53.578 7f51255e2700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f51200cff80 0x7f5120109140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:53.580 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:53.578 7f51255e2700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51200fe1d0 con 0x7f51201b0af0 2021-02-10T11:35:53.581 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:53.578 7f51255e2700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5120109ae0 con 0x7f51200d1880 2021-02-10T11:35:53.581 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:53.578 7f51255e2700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5120109760 con 0x7f51200cff80 2021-02-10T11:35:53.581 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:53.578 7f511f2bf700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f51200d1880 0x7f51201b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:53.581 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:53.578 7f511f2bf700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f51200d1880 0x7f51201b04d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:51528/0 (socket says 172.21.2.112:51528) 2021-02-10T11:35:53.581 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:53.578 7f511f2bf700 1 -- 172.21.2.112:0/1666590861 learned_addr learned my addr 172.21.2.112:0/1666590861 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:35:53.581 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:53.578 7f511f2bf700 1 -- 172.21.2.112:0/1666590861 >> v2:172.21.2.120:3300/0 conn(0x7f51200cff80 msgr2=0x7f5120109140 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:35:53.581 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:53.578 7f511f2bf700 1 --2- 172.21.2.112:0/1666590861 >> v2:172.21.2.120:3300/0 conn(0x7f51200cff80 0x7f5120109140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:53.581 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:53.578 7f511f2bf700 1 -- 172.21.2.112:0/1666590861 >> v1:172.21.2.120:6789/0 conn(0x7f51201b0af0 legacy=0x7f5120108950 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:35:53.581 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:53.578 7f511f2bf700 1 -- 172.21.2.112:0/1666590861 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5120088070 con 0x7f51200d1880 2021-02-10T11:35:53.581 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:53.579 7f511f2bf700 1 --2- 172.21.2.112:0/1666590861 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f51200d1880 0x7f51201b04d0 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 rx=0x7f5110006700 tx=0x7f5110009b80).ready entity=mon.0 client_cookie=811521dea072f5c4 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:53.581 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:53.579 7f511e2bd700 1 -- 172.21.2.112:0/1666590861 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f511000c0c0 con 0x7f51200d1880 2021-02-10T11:35:53.581 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:53.579 7f511e2bd700 1 -- 172.21.2.112:0/1666590861 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f511000c340 con 0x7f51200d1880 2021-02-10T11:35:53.582 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:53.579 7f511e2bd700 1 -- 172.21.2.112:0/1666590861 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f511000c860 con 0x7f51200d1880 2021-02-10T11:35:53.582 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:53.579 7f51255e2700 1 -- 172.21.2.112:0/1666590861 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f51200d1880 msgr2=0x7f51201b04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:53.582 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:53.579 7f51255e2700 1 --2- 172.21.2.112:0/1666590861 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f51200d1880 0x7f51201b04d0 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 rx=0x7f5110006700 tx=0x7f5110009b80).stop 2021-02-10T11:35:53.582 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:53.579 7f51255e2700 1 -- 172.21.2.112:0/1666590861 shutdown_connections 2021-02-10T11:35:53.582 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:53.579 7f51255e2700 1 --2- 172.21.2.112:0/1666590861 >> v2:172.21.2.120:3300/0 conn(0x7f51200cff80 0x7f5120109140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:53.582 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:53.579 7f51255e2700 1 --2- 172.21.2.112:0/1666590861 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f51200d1880 0x7f51201b04d0 unknown :-1 s=CLOSED pgs=1530 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:53.582 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:53.579 7f51255e2700 1 -- 172.21.2.112:0/1666590861 >> 172.21.2.112:0/1666590861 conn(0x7f51200ce430 msgr2=0x7f51200d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:53.582 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:53.580 7f51255e2700 1 -- 172.21.2.112:0/1666590861 shutdown_connections 2021-02-10T11:35:53.582 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:53.580 7f51255e2700 1 -- 172.21.2.112:0/1666590861 wait complete. 2021-02-10T11:35:53.584 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:53.594 7f51255e2700 1 Processor -- start 2021-02-10T11:35:53.584 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:53.594 7f51255e2700 1 -- start start 2021-02-10T11:35:53.584 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:53.594 7f51255e2700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5120187e80 0x7f5120188280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:53.584 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:53.594 7f51255e2700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f51201888a0 0x7f5120189490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:53.584 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:53.594 7f51255e2700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5120189ab0 0x7f5120189eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:53.584 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:53.594 7f51255e2700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5120188ca0 con 0x7f51201888a0 2021-02-10T11:35:53.584 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:53.594 7f51255e2700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5120188f10 con 0x7f5120189ab0 2021-02-10T11:35:53.584 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:53.594 7f51255e2700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5120099520 con 0x7f5120187e80 2021-02-10T11:35:53.584 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:53.595 7f511eabe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f51201888a0 0x7f5120189490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:53.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:53.595 7f511eabe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f51201888a0 0x7f5120189490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:51534/0 (socket says 172.21.2.112:51534) 2021-02-10T11:35:53.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:53.595 7f511eabe700 1 -- 172.21.2.112:0/4208146481 learned_addr learned my addr 172.21.2.112:0/4208146481 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:35:53.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:53.595 7f511f2bf700 1 --2- 172.21.2.112:0/4208146481 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5120187e80 0x7f5120188280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:53.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:53.595 7f511fac0700 1 --2- 172.21.2.112:0/4208146481 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5120189ab0 0x7f5120189eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:53.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:53.595 7f511eabe700 1 -- 172.21.2.112:0/4208146481 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5120187e80 msgr2=0x7f5120188280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:53.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:53.595 7f511eabe700 1 --2- 172.21.2.112:0/4208146481 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5120187e80 0x7f5120188280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:53.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:53.595 7f511eabe700 1 -- 172.21.2.112:0/4208146481 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5120189ab0 msgr2=0x7f5120189eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:53.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:53.595 7f511eabe700 1 --2- 172.21.2.112:0/4208146481 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5120189ab0 0x7f5120189eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:53.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:53.595 7f511eabe700 1 -- 172.21.2.112:0/4208146481 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51200999c0 con 0x7f51201888a0 2021-02-10T11:35:53.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:53.595 7f511eabe700 1 --2- 172.21.2.112:0/4208146481 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f51201888a0 0x7f5120189490 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 rx=0x7f510800b5b0 tx=0x7f510800b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:53.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:53.596 7f5117fff700 1 -- 172.21.2.112:0/4208146481 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f510800bd40 con 0x7f51201888a0 2021-02-10T11:35:53.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:53.596 7f5117fff700 1 -- 172.21.2.112:0/4208146481 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5108006240 con 0x7f51201888a0 2021-02-10T11:35:53.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:53.596 7f5117fff700 1 -- 172.21.2.112:0/4208146481 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5108005720 con 0x7f51201888a0 2021-02-10T11:35:53.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:53.596 7f51255e2700 1 -- 172.21.2.112:0/4208146481 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51200999c0 con 0x7f51201888a0 2021-02-10T11:35:53.586 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:53.596 7f51255e2700 1 -- 172.21.2.112:0/4208146481 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f512018a7c0 con 0x7f51201888a0 2021-02-10T11:35:53.586 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:53.596 7f5117fff700 1 -- 172.21.2.112:0/4208146481 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f51080054a0 con 0x7f51201888a0 2021-02-10T11:35:53.586 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:53.596 7f5117fff700 1 --2- 172.21.2.112:0/4208146481 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f51040186c0 0x7f510401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:53.586 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:53.596 7f5117fff700 1 -- 172.21.2.112:0/4208146481 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5221+0+0 (secure 0 0 0) 0x7f5108032440 con 0x7f51201888a0 2021-02-10T11:35:53.586 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:53.597 7f511f2bf700 1 --2- 172.21.2.112:0/4208146481 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f51040186c0 0x7f510401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:53.586 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:53.597 7f511f2bf700 1 --2- 172.21.2.112:0/4208146481 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f51040186c0 0x7f510401abc0 crc :-1 s=READY pgs=740 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:53.614 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:53.628 7f51255e2700 1 -- 172.21.2.112:0/4208146481 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5120195bd0 con 0x7f51201888a0 2021-02-10T11:35:53.617 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:53.631 7f5117fff700 1 -- 172.21.2.112:0/4208146481 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f51080009b0 con 0x7f51201888a0 2021-02-10T11:35:53.745 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:53.758 7f51255e2700 1 -- 172.21.2.112:0/4208146481 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f5120195bd0 con 0x7f51201888a0 2021-02-10T11:35:54.114 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:54.127 7f5117fff700 1 -- 172.21.2.112:0/4208146481 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v67) v1 ==== 173+0+0 (secure 0 0 0) 0x7f51080054a0 con 0x7f51201888a0 2021-02-10T11:35:54.160 INFO:teuthology.orchestra.run.gibba012.stderr:pool 'cephfs_data' removed 2021-02-10T11:35:54.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:54.175 7f51255e2700 1 -- 172.21.2.112:0/4208146481 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f51040186c0 msgr2=0x7f510401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:54.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:54.175 7f51255e2700 1 --2- 172.21.2.112:0/4208146481 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f51040186c0 0x7f510401abc0 crc :-1 s=READY pgs=740 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:54.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:54.175 7f51255e2700 1 -- 172.21.2.112:0/4208146481 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f51201888a0 msgr2=0x7f5120189490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:54.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:54.175 7f51255e2700 1 --2- 172.21.2.112:0/4208146481 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f51201888a0 0x7f5120189490 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 rx=0x7f510800b5b0 tx=0x7f510800b950).stop 2021-02-10T11:35:54.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:54.176 7f51255e2700 1 -- 172.21.2.112:0/4208146481 shutdown_connections 2021-02-10T11:35:54.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:54.176 7f51255e2700 1 --2- 172.21.2.112:0/4208146481 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f51040186c0 0x7f510401abc0 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:54.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:54.176 7f51255e2700 1 --2- 172.21.2.112:0/4208146481 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5120189ab0 0x7f5120189eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:54.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:54.176 7f51255e2700 1 --2- 172.21.2.112:0/4208146481 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f51201888a0 0x7f5120189490 unknown :-1 s=CLOSED pgs=1531 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:54.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:54.176 7f51255e2700 1 --2- 172.21.2.112:0/4208146481 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5120187e80 0x7f5120188280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:54.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:54.176 7f51255e2700 1 -- 172.21.2.112:0/4208146481 >> 172.21.2.112:0/4208146481 conn(0x7f512019f940 msgr2=0x7f512018c290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:54.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:54.176 7f51255e2700 1 -- 172.21.2.112:0/4208146481 shutdown_connections 2021-02-10T11:35:54.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:54.176 7f51255e2700 1 -- 172.21.2.112:0/4208146481 wait complete. 2021-02-10T11:35:54.171 INFO:tasks.ceph.mds.a:Restarting daemon 2021-02-10T11:35:54.171 INFO:tasks.ceph.mds.a:Stopping old one... 2021-02-10T11:35:54.171 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2021-02-10T11:35:54.171 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:35:54.171 INFO:tasks.ceph.mds.b:Restarting daemon 2021-02-10T11:35:54.171 INFO:tasks.ceph.mds.b:Stopping old one... 2021-02-10T11:35:54.172 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2021-02-10T11:35:54.172 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:35:54.177 INFO:tasks.ceph.mds.a:Stopped 2021-02-10T11:35:54.177 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2021-02-10T11:35:54.178 INFO:tasks.ceph.mds.b:Stopped 2021-02-10T11:35:54.179 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2021-02-10T11:35:54.220 INFO:tasks.ceph.mds.a:Started 2021-02-10T11:35:54.220 DEBUG:teuthology.parallel:result is None 2021-02-10T11:35:54.221 INFO:tasks.ceph.mds.b:Started 2021-02-10T11:35:54.221 DEBUG:teuthology.parallel:result is None 2021-02-10T11:35:54.221 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd blacklist clear 2021-02-10T11:35:54.623 INFO:tasks.ceph.mds.b.gibba020.stdout:starting mds.b at 2021-02-10T11:35:54.628 INFO:tasks.ceph.mds.a.gibba012.stdout:starting mds.a at 2021-02-10T11:35:54.652 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:54.649 7fc0bddba700 1 Processor -- start 2021-02-10T11:35:54.652 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:54.649 7fc0bddba700 1 -- start start 2021-02-10T11:35:54.652 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:54.649 7fc0bddba700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc0b80d1870 0x7fc0b81b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:54.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:54.649 7fc0bddba700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fc0b81b0a10 0x7fc0b80cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:54.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:54.649 7fc0bddba700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0b80f0b10 con 0x7fc0b80d1c70 2021-02-10T11:35:54.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:54.649 7fc0bddba700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0b80fe230 con 0x7fc0b81b0a10 2021-02-10T11:35:54.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:54.649 7fc0bddba700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0b80e5ed0 con 0x7fc0b80d1870 2021-02-10T11:35:54.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:54.649 7fc0b7fff700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fc0b80d1c70 0x7fc0b80cf950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:51018/0 (socket says 172.21.2.112:51018) 2021-02-10T11:35:54.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:54.649 7fc0b7fff700 1 -- 172.21.2.112:0/275318663 learned_addr learned my addr 172.21.2.112:0/275318663 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:35:54.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:54.649 7fc0b6ffd700 1 -- 172.21.2.112:0/275318663 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2463885275 0 0) 0x7fc0b0001500 con 0x7fc0b80d1c70 2021-02-10T11:35:54.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:54.649 7fc0affff700 1 --2- 172.21.2.112:0/275318663 >> v2:172.21.2.120:3300/0 conn(0x7fc0b81b0a10 0x7fc0b80cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:54.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:54.649 7fc0b6ffd700 1 -- 172.21.2.112:0/275318663 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0a4002fe0 con 0x7fc0b80d1c70 2021-02-10T11:35:54.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:54.649 7fc0b77fe700 1 --2- 172.21.2.112:0/275318663 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc0b80d1870 0x7fc0b81b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:54.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:54.650 7fc0b6ffd700 1 -- 172.21.2.112:0/275318663 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 173664797 0 0) 0x7fc0b0001270 con 0x7fc0b80d1c70 2021-02-10T11:35:54.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:54.650 7fc0affff700 1 -- 172.21.2.112:0/275318663 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc0b80d1870 msgr2=0x7fc0b81b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:54.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:54.650 7fc0affff700 1 --2- 172.21.2.112:0/275318663 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc0b80d1870 0x7fc0b81b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:54.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:54.650 7fc0affff700 1 -- 172.21.2.112:0/275318663 >> v1:172.21.2.112:6789/0 conn(0x7fc0b80d1c70 legacy=0x7fc0b80cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:54.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:54.650 7fc0affff700 1 -- 172.21.2.112:0/275318663 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0b80d3360 con 0x7fc0b81b0a10 2021-02-10T11:35:54.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:54.650 7fc0affff700 1 --2- 172.21.2.112:0/275318663 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc0b81b0a10 0x7fc0b80cf330 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 rx=0x7fc0a8006650 tx=0x7fc0a8009a80).ready entity=mon.1 client_cookie=e5d0f2f45abcd466 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:54.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:54.650 7fc0b6ffd700 1 -- 172.21.2.112:0/275318663 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc0a800c0c0 con 0x7fc0b81b0a10 2021-02-10T11:35:54.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:54.650 7fc0b6ffd700 1 -- 172.21.2.112:0/275318663 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0a800c340 con 0x7fc0b81b0a10 2021-02-10T11:35:54.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:54.650 7fc0b6ffd700 1 -- 172.21.2.112:0/275318663 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc0a800c860 con 0x7fc0b81b0a10 2021-02-10T11:35:54.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:54.650 7fc0bddba700 1 -- 172.21.2.112:0/275318663 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc0b81b0a10 msgr2=0x7fc0b80cf330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:54.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:54.650 7fc0bddba700 1 --2- 172.21.2.112:0/275318663 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc0b81b0a10 0x7fc0b80cf330 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 rx=0x7fc0a8006650 tx=0x7fc0a8009a80).stop 2021-02-10T11:35:54.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:54.651 7fc0bddba700 1 -- 172.21.2.112:0/275318663 shutdown_connections 2021-02-10T11:35:54.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:54.651 7fc0bddba700 1 --2- 172.21.2.112:0/275318663 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc0b81b0a10 0x7fc0b80cf330 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:54.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:54.651 7fc0bddba700 1 --2- 172.21.2.112:0/275318663 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc0b80d1870 0x7fc0b81b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:54.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:54.651 7fc0bddba700 1 -- 172.21.2.112:0/275318663 >> 172.21.2.112:0/275318663 conn(0x7fc0b80ce450 msgr2=0x7fc0b80d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:54.657 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:54.651 7fc0bddba700 1 -- 172.21.2.112:0/275318663 shutdown_connections 2021-02-10T11:35:54.657 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:54.651 7fc0bddba700 1 -- 172.21.2.112:0/275318663 wait complete. 2021-02-10T11:35:54.658 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:54.666 7fc0bddba700 1 Processor -- start 2021-02-10T11:35:54.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:54.666 7fc0bddba700 1 -- start start 2021-02-10T11:35:54.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:54.666 7fc0bddba700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc0b80a43d0 0x7fc0b80926d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:54.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:54.666 7fc0bddba700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc0b8092f70 0x7fc0b80a10d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:54.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:54.666 7fc0bddba700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc0b8093370 0x7fc0b80a16f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:54.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:54.666 7fc0bddba700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0b80a1d10 con 0x7fc0b80a43d0 2021-02-10T11:35:54.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:54.666 7fc0bddba700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0b8092cf0 con 0x7fc0b8093370 2021-02-10T11:35:54.660 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:54.666 7fc0bddba700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0b80a1f40 con 0x7fc0b8092f70 2021-02-10T11:35:54.660 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:54.666 7fc0b77fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc0b80a43d0 0x7fc0b80926d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:54.660 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:54.666 7fc0b77fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc0b80a43d0 0x7fc0b80926d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:51562/0 (socket says 172.21.2.112:51562) 2021-02-10T11:35:54.660 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:54.666 7fc0b77fe700 1 -- 172.21.2.112:0/1135565389 learned_addr learned my addr 172.21.2.112:0/1135565389 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:35:54.660 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:54.667 7fc0b77fe700 1 -- 172.21.2.112:0/1135565389 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc0b8092f70 msgr2=0x7fc0b80a10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:54.661 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:54.667 7fc0b77fe700 1 --2- 172.21.2.112:0/1135565389 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc0b8092f70 0x7fc0b80a10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:54.661 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:54.667 7fc0b77fe700 1 -- 172.21.2.112:0/1135565389 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc0b8093370 msgr2=0x7fc0b80a16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:54.661 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:54.667 7fc0b77fe700 1 --2- 172.21.2.112:0/1135565389 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc0b8093370 0x7fc0b80a16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:54.661 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:54.667 7fc0b77fe700 1 -- 172.21.2.112:0/1135565389 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0b80a24c0 con 0x7fc0b80a43d0 2021-02-10T11:35:54.661 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:54.667 7fc0b77fe700 1 --2- 172.21.2.112:0/1135565389 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc0b80a43d0 0x7fc0b80926d0 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 rx=0x7fc0a0006f60 tx=0x7fc0a000b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:54.661 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:54.667 7fc0b4ff9700 1 -- 172.21.2.112:0/1135565389 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc0a00009b0 con 0x7fc0b80a43d0 2021-02-10T11:35:54.662 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:54.667 7fc0b4ff9700 1 -- 172.21.2.112:0/1135565389 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0a000bdf0 con 0x7fc0b80a43d0 2021-02-10T11:35:54.662 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:54.667 7fc0b4ff9700 1 -- 172.21.2.112:0/1135565389 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc0a0002030 con 0x7fc0b80a43d0 2021-02-10T11:35:54.662 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:54.667 7fc0bddba700 1 -- 172.21.2.112:0/1135565389 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0b80a24c0 con 0x7fc0b80a43d0 2021-02-10T11:35:54.662 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:54.667 7fc0bddba700 1 -- 172.21.2.112:0/1135565389 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0b80a24c0 con 0x7fc0b80a43d0 2021-02-10T11:35:54.662 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:54.668 7fc0b4ff9700 1 -- 172.21.2.112:0/1135565389 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc0a0006b00 con 0x7fc0b80a43d0 2021-02-10T11:35:54.662 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:54.668 7fc0b4ff9700 1 --2- 172.21.2.112:0/1135565389 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc09c0186c0 0x7fc09c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:54.663 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:54.668 7fc0b4ff9700 1 -- 172.21.2.112:0/1135565389 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 4839+0+0 (secure 0 0 0) 0x7fc0a0032250 con 0x7fc0b80a43d0 2021-02-10T11:35:54.663 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:54.668 7fc0affff700 1 --2- 172.21.2.112:0/1135565389 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc09c0186c0 0x7fc09c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:54.663 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:54.668 7fc0affff700 1 --2- 172.21.2.112:0/1135565389 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc09c0186c0 0x7fc09c01abc0 crc :-1 s=READY pgs=741 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:54.685 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:54.698 7fc0bddba700 1 -- 172.21.2.112:0/1135565389 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0b80a1d10 con 0x7fc0b80a43d0 2021-02-10T11:35:54.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:54.701 7fc0b4ff9700 1 -- 172.21.2.112:0/1135565389 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc0a002dec0 con 0x7fc0b80a43d0 2021-02-10T11:35:54.824 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:54.837 7fc0bddba700 1 -- 172.21.2.112:0/1135565389 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7fc0b80a1d10 con 0x7fc0b80a43d0 2021-02-10T11:35:55.152 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:55.165 7fc0b4ff9700 1 -- 172.21.2.112:0/1135565389 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v68) v1 ==== 97+0+0 (secure 0 0 0) 0x7fc0a002dec0 con 0x7fc0b80a43d0 2021-02-10T11:35:55.190 INFO:teuthology.orchestra.run.gibba012.stderr: removed all blacklist entries 2021-02-10T11:35:55.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:55.205 7fc0bddba700 1 -- 172.21.2.112:0/1135565389 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc09c0186c0 msgr2=0x7fc09c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:55.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:55.205 7fc0bddba700 1 --2- 172.21.2.112:0/1135565389 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc09c0186c0 0x7fc09c01abc0 crc :-1 s=READY pgs=741 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:55.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:55.205 7fc0bddba700 1 -- 172.21.2.112:0/1135565389 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc0b80a43d0 msgr2=0x7fc0b80926d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:55.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:55.205 7fc0bddba700 1 --2- 172.21.2.112:0/1135565389 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc0b80a43d0 0x7fc0b80926d0 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 rx=0x7fc0a0006f60 tx=0x7fc0a000b970).stop 2021-02-10T11:35:55.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:55.206 7fc0bddba700 1 -- 172.21.2.112:0/1135565389 shutdown_connections 2021-02-10T11:35:55.193 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:55.206 7fc0bddba700 1 --2- 172.21.2.112:0/1135565389 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc09c0186c0 0x7fc09c01abc0 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:55.193 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:55.206 7fc0bddba700 1 --2- 172.21.2.112:0/1135565389 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc0b8093370 0x7fc0b80a16f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:55.193 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:55.206 7fc0bddba700 1 --2- 172.21.2.112:0/1135565389 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc0b8092f70 0x7fc0b80a10d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:55.193 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:55.206 7fc0bddba700 1 --2- 172.21.2.112:0/1135565389 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc0b80a43d0 0x7fc0b80926d0 unknown :-1 s=CLOSED pgs=1536 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:55.193 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:55.206 7fc0bddba700 1 -- 172.21.2.112:0/1135565389 >> 172.21.2.112:0/1135565389 conn(0x7fc0b80bc090 msgr2=0x7fc0b808f8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:55.193 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:55.206 7fc0bddba700 1 -- 172.21.2.112:0/1135565389 shutdown_connections 2021-02-10T11:35:55.193 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:55.206 7fc0bddba700 1 -- 172.21.2.112:0/1135565389 wait complete. 2021-02-10T11:35:55.201 INFO:tasks.cephfs.cephfs_test_case:['0', '1', '2', '3'] 2021-02-10T11:35:55.201 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth ls --format=json-pretty 2021-02-10T11:35:55.629 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:55.625 7f109367c700 1 Processor -- start 2021-02-10T11:35:55.629 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:55.625 7f109367c700 1 -- start start 2021-02-10T11:35:55.629 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:55.625 7f109367c700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f108c0d1870 0x7f108c1b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:55.629 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:55.625 7f109367c700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f108c0f0b10 con 0x7f108c1b0a10 2021-02-10T11:35:55.629 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:55.625 7f109367c700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f108c0d0310 con 0x7f108c0f0650 2021-02-10T11:35:55.630 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:55.625 7f109367c700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f108c0d0530 con 0x7f108c0d1870 2021-02-10T11:35:55.630 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:55.626 7f1090c97700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f108c1b0a10 0x7f108c0cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:51028/0 (socket says 172.21.2.112:51028) 2021-02-10T11:35:55.630 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:55.626 7f1090c97700 1 -- 172.21.2.112:0/3425312161 learned_addr learned my addr 172.21.2.112:0/3425312161 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:35:55.630 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:55.626 7f1091498700 1 --2- 172.21.2.112:0/3425312161 >> v2:172.21.2.120:3300/0 conn(0x7f108c0d1870 0x7f108c1b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:55.630 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:55.626 7f107ffff700 1 -- 172.21.2.112:0/3425312161 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2045732340 0 0) 0x7f10800009b0 con 0x7f108c1b0a10 2021-02-10T11:35:55.630 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:55.626 7f107ffff700 1 -- 172.21.2.112:0/3425312161 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1078002fe0 con 0x7f108c1b0a10 2021-02-10T11:35:55.630 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:55.626 7f1091498700 1 -- 172.21.2.112:0/3425312161 >> v1:172.21.2.120:6789/0 conn(0x7f108c0f0650 legacy=0x7f108c0cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:55.630 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:55.626 7f1091498700 1 -- 172.21.2.112:0/3425312161 >> v1:172.21.2.112:6789/0 conn(0x7f108c1b0a10 legacy=0x7f108c0cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:55.630 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:55.626 7f1091498700 1 -- 172.21.2.112:0/3425312161 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f108c0d1c70 con 0x7f108c0d1870 2021-02-10T11:35:55.630 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:55.627 7f1091498700 1 --2- 172.21.2.112:0/3425312161 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f108c0d1870 0x7f108c1b03f0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 rx=0x7f1088006650 tx=0x7f1088009a80).ready entity=mon.1 client_cookie=fd800852cb48b728 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:55.630 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:55.627 7f107ffff700 1 -- 172.21.2.112:0/3425312161 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f10880009b0 con 0x7f108c0d1870 2021-02-10T11:35:55.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:55.627 7f107ffff700 1 -- 172.21.2.112:0/3425312161 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f108800c0c0 con 0x7f108c0d1870 2021-02-10T11:35:55.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:55.627 7f107ffff700 1 -- 172.21.2.112:0/3425312161 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f108800c5e0 con 0x7f108c0d1870 2021-02-10T11:35:55.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:55.627 7f109367c700 1 -- 172.21.2.112:0/3425312161 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f108c0d1870 msgr2=0x7f108c1b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:55.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:55.627 7f109367c700 1 --2- 172.21.2.112:0/3425312161 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f108c0d1870 0x7f108c1b03f0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 rx=0x7f1088006650 tx=0x7f1088009a80).stop 2021-02-10T11:35:55.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:55.628 7f109367c700 1 -- 172.21.2.112:0/3425312161 shutdown_connections 2021-02-10T11:35:55.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:55.628 7f109367c700 1 --2- 172.21.2.112:0/3425312161 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f108c0d1870 0x7f108c1b03f0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:55.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:55.628 7f109367c700 1 -- 172.21.2.112:0/3425312161 >> 172.21.2.112:0/3425312161 conn(0x7f108c0ce450 msgr2=0x7f108c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:55.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:55.628 7f109367c700 1 -- 172.21.2.112:0/3425312161 shutdown_connections 2021-02-10T11:35:55.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:55.628 7f109367c700 1 -- 172.21.2.112:0/3425312161 wait complete. 2021-02-10T11:35:55.633 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:55.643 7f109367c700 1 Processor -- start 2021-02-10T11:35:55.634 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:55.643 7f109367c700 1 -- start start 2021-02-10T11:35:55.634 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:55.643 7f109367c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f108c0b0ad0 0x7f108c091f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:55.634 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:55.643 7f109367c700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f108c1a8850 0x7f108c1a8c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:55.634 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:55.643 7f109367c700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f108c1a6fa0 0x7f108c1a73a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:55.634 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:55.643 7f109367c700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f108c1abfe0 con 0x7f108c0b0ad0 2021-02-10T11:35:55.634 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:55.643 7f109367c700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f108c1a9c40 con 0x7f108c1a6fa0 2021-02-10T11:35:55.634 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:55.643 7f109367c700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f108c1a9730 con 0x7f108c1a8850 2021-02-10T11:35:55.634 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:55.643 7f1091498700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f108c0b0ad0 0x7f108c091f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:55.634 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:55.643 7f1091498700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f108c0b0ad0 0x7f108c091f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:51572/0 (socket says 172.21.2.112:51572) 2021-02-10T11:35:55.634 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:55.644 7f1091498700 1 -- 172.21.2.112:0/2489381206 learned_addr learned my addr 172.21.2.112:0/2489381206 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:35:55.634 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:55.644 7f1091498700 1 -- 172.21.2.112:0/2489381206 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f108c1a8850 msgr2=0x7f108c1a8c50 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:35:55.634 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:55.644 7f1091498700 1 --2- 172.21.2.112:0/2489381206 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f108c1a8850 0x7f108c1a8c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:55.634 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:55.644 7f1091498700 1 -- 172.21.2.112:0/2489381206 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f108c1a6fa0 msgr2=0x7f108c1a73a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:35:55.635 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:55.644 7f1091498700 1 --2- 172.21.2.112:0/2489381206 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f108c1a6fa0 0x7f108c1a73a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:55.635 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:55.644 7f1091498700 1 -- 172.21.2.112:0/2489381206 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f108c1a9270 con 0x7f108c0b0ad0 2021-02-10T11:35:55.635 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:55.644 7f1091498700 1 --2- 172.21.2.112:0/2489381206 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f108c0b0ad0 0x7f108c091f30 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 rx=0x7f1088009630 tx=0x7f108800b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:55.635 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:55.644 7f107dffb700 1 -- 172.21.2.112:0/2489381206 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f10880093a0 con 0x7f108c0b0ad0 2021-02-10T11:35:55.635 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:55.644 7f107dffb700 1 -- 172.21.2.112:0/2489381206 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f108800bd70 con 0x7f108c0b0ad0 2021-02-10T11:35:55.635 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:55.644 7f107dffb700 1 -- 172.21.2.112:0/2489381206 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1088005560 con 0x7f108c0b0ad0 2021-02-10T11:35:55.635 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:55.644 7f109367c700 1 -- 172.21.2.112:0/2489381206 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f108c1a9270 con 0x7f108c0b0ad0 2021-02-10T11:35:55.635 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:55.644 7f109367c700 1 -- 172.21.2.112:0/2489381206 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f108c1a9270 con 0x7f108c0b0ad0 2021-02-10T11:35:55.635 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:55.645 7f107dffb700 1 -- 172.21.2.112:0/2489381206 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f108800a000 con 0x7f108c0b0ad0 2021-02-10T11:35:55.635 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:55.645 7f107dffb700 1 --2- 172.21.2.112:0/2489381206 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f106c0186c0 0x7f106c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:55.635 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:55.645 7f107dffb700 1 -- 172.21.2.112:0/2489381206 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f1088032250 con 0x7f108c0b0ad0 2021-02-10T11:35:55.635 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:55.647 7f1090c97700 1 --2- 172.21.2.112:0/2489381206 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f106c0186c0 0x7f106c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:55.635 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:55.647 7f1090c97700 1 --2- 172.21.2.112:0/2489381206 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f106c0186c0 0x7f106c01abc0 crc :-1 s=READY pgs=742 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:55.662 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:55.675 7f109367c700 1 -- 172.21.2.112:0/2489381206 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f108c1abfe0 con 0x7f108c0b0ad0 2021-02-10T11:35:55.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:55.678 7f107dffb700 1 -- 172.21.2.112:0/2489381206 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f108802ef00 con 0x7f108c0b0ad0 2021-02-10T11:35:55.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:55.809 7f109367c700 1 -- 172.21.2.112:0/2489381206 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f108c1abfe0 con 0x7f108c0b0ad0 2021-02-10T11:35:55.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:55.810 7f107dffb700 1 -- 172.21.2.112:0/2489381206 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v44) v1 ==== 80+0+4513 (secure 0 0 0) 0x7f108802ef00 con 0x7f108c0b0ad0 2021-02-10T11:35:55.797 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:35:55.797 INFO:teuthology.orchestra.run.gibba012.stdout:{ 2021-02-10T11:35:55.797 INFO:teuthology.orchestra.run.gibba012.stdout: "auth_dump": [ 2021-02-10T11:35:55.797 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:55.797 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "mds.a", 2021-02-10T11:35:55.797 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCKvyNgPkV4AxAAEmVNaghdccP6lWd8hvQnxw==", 2021-02-10T11:35:55.797 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:35:55.797 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:35:55.797 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:35:55.798 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:35:55.798 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:35:55.798 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:35:55.798 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:55.798 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:55.798 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "mds.b", 2021-02-10T11:35:55.798 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCKvyNguxONLhAAHIpry+R89MAwdhFJyEhVCw==", 2021-02-10T11:35:55.798 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:35:55.798 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:35:55.798 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:35:55.798 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:35:55.798 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:35:55.799 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:35:55.799 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:55.799 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:55.799 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.0", 2021-02-10T11:35:55.799 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCSvyNgXcKPLhAAP/Bj2F+06DCtPCzQDNq+cQ==", 2021-02-10T11:35:55.799 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:35:55.799 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:35:55.799 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:35:55.799 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:35:55.799 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:35:55.799 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:55.799 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:55.800 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.1", 2021-02-10T11:35:55.800 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCVvyNghfbcLhAA66PrRwR2M7o6VTF5UhU/Fg==", 2021-02-10T11:35:55.800 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:35:55.800 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:35:55.800 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:35:55.800 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:35:55.800 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:35:55.800 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:55.800 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:55.800 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.2", 2021-02-10T11:35:55.800 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCYvyNgwiPqLhAA66BVkRsgaUlGpAsQ15Nf8Q==", 2021-02-10T11:35:55.800 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:35:55.800 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:35:55.801 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:35:55.801 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:35:55.801 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:35:55.801 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:55.801 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:55.801 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.3", 2021-02-10T11:35:55.801 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCbvyNgqBZrLxAAERPe/1L3SXAEzI3d9ReHHg==", 2021-02-10T11:35:55.801 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:35:55.801 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:35:55.801 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:35:55.801 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:35:55.802 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:35:55.802 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:55.802 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:55.802 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.4", 2021-02-10T11:35:55.802 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCkvyNgDQsxIhAAHmVjVYnx20N4SDmKqvbcuA==", 2021-02-10T11:35:55.802 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:35:55.802 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:35:55.802 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:35:55.802 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:35:55.802 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:35:55.802 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:55.802 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:55.802 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.5", 2021-02-10T11:35:55.803 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCnvyNg8ITFIhAADd4aKybAV7btoGPx3kCg3A==", 2021-02-10T11:35:55.803 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:35:55.803 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:35:55.803 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:35:55.803 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:35:55.803 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:35:55.803 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:55.803 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:55.803 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.6", 2021-02-10T11:35:55.803 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCqvyNgxOhDIxAAzXixEa/nlDc1xXrg0aeLwg==", 2021-02-10T11:35:55.803 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:35:55.803 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:35:55.803 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:35:55.804 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:35:55.804 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:35:55.804 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:55.804 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:55.804 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.7", 2021-02-10T11:35:55.804 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCtvyNgxSqGIxAACxJQlwP7DpiVVCZHA3zn1A==", 2021-02-10T11:35:55.804 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:35:55.804 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:35:55.804 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:35:55.804 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:35:55.805 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:35:55.805 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:55.805 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:55.805 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.0", 2021-02-10T11:35:55.805 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCLvyNgDkNiHRAAygU8Za4PBusTqCgwAYWTiQ==", 2021-02-10T11:35:55.805 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:35:55.805 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:35:55.805 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow r", 2021-02-10T11:35:55.805 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:35:55.805 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:35:55.806 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:55.806 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:55.806 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.1", 2021-02-10T11:35:55.806 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCLvyNgxmLqMxAA1MlSDJ9LH9u4f2sL7mm8/Q==", 2021-02-10T11:35:55.806 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:35:55.806 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:35:55.806 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow r", 2021-02-10T11:35:55.806 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:35:55.806 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:35:55.806 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:55.806 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:55.806 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.2", 2021-02-10T11:35:55.806 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCMvyNgp8veDhAAniSVHYPtwwTog/pjbaRQmw==", 2021-02-10T11:35:55.807 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:35:55.807 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:35:55.807 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow r", 2021-02-10T11:35:55.807 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:35:55.807 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:35:55.807 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:55.807 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:55.807 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.3", 2021-02-10T11:35:55.807 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCMvyNgQEVoJRAApHcWdm1sldIxu+heyD4HdQ==", 2021-02-10T11:35:55.807 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:35:55.807 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:35:55.807 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow r", 2021-02-10T11:35:55.807 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:35:55.808 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:35:55.808 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:55.808 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:55.808 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.admin", 2021-02-10T11:35:55.808 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCGvyNg7jjJBRAAtot+ypUZOiTo1GbY+oVhVw==", 2021-02-10T11:35:55.808 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:35:55.808 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow *", 2021-02-10T11:35:55.808 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:35:55.808 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:35:55.808 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:35:55.808 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:35:55.808 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:55.808 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:55.809 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "mgr.x", 2021-02-10T11:35:55.809 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCJvyNgTPVwKBAAjduiBE8+RSXBv82tdPecWg==", 2021-02-10T11:35:55.809 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:35:55.809 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow *", 2021-02-10T11:35:55.809 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow profile mgr", 2021-02-10T11:35:55.809 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:35:55.809 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:35:55.809 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:35:55.809 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:55.809 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "mgr.y", 2021-02-10T11:35:55.809 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCJvyNgQ1tTDhAASwBssDy+Rz31ujw8fNlWow==", 2021-02-10T11:35:55.809 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:35:55.809 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow *", 2021-02-10T11:35:55.810 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow profile mgr", 2021-02-10T11:35:55.810 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:35:55.810 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:35:55.810 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:35:55.810 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:35:55.810 INFO:teuthology.orchestra.run.gibba012.stdout:} 2021-02-10T11:35:55.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:55.811 7f109367c700 1 -- 172.21.2.112:0/2489381206 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f106c0186c0 msgr2=0x7f106c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:55.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:55.811 7f109367c700 1 --2- 172.21.2.112:0/2489381206 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f106c0186c0 0x7f106c01abc0 crc :-1 s=READY pgs=742 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:55.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:55.811 7f109367c700 1 -- 172.21.2.112:0/2489381206 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f108c0b0ad0 msgr2=0x7f108c091f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:55.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:55.811 7f109367c700 1 --2- 172.21.2.112:0/2489381206 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f108c0b0ad0 0x7f108c091f30 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 rx=0x7f1088009630 tx=0x7f108800b980).stop 2021-02-10T11:35:55.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:55.812 7f109367c700 1 -- 172.21.2.112:0/2489381206 shutdown_connections 2021-02-10T11:35:55.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:55.812 7f109367c700 1 --2- 172.21.2.112:0/2489381206 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f106c0186c0 0x7f106c01abc0 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:55.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:55.812 7f109367c700 1 --2- 172.21.2.112:0/2489381206 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f108c1a6fa0 0x7f108c1a73a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:55.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:55.812 7f109367c700 1 --2- 172.21.2.112:0/2489381206 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f108c1a8850 0x7f108c1a8c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:55.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:55.812 7f109367c700 1 --2- 172.21.2.112:0/2489381206 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f108c0b0ad0 0x7f108c091f30 unknown :-1 s=CLOSED pgs=1538 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:55.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:55.812 7f109367c700 1 -- 172.21.2.112:0/2489381206 >> 172.21.2.112:0/2489381206 conn(0x7f108c19f8f0 msgr2=0x7f108c1a5f80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:55.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:55.812 7f109367c700 1 -- 172.21.2.112:0/2489381206 shutdown_connections 2021-02-10T11:35:55.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:55.812 7f109367c700 1 -- 172.21.2.112:0/2489381206 wait complete. 2021-02-10T11:35:55.813 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:35:56.241 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.238 7f62ac755700 1 Processor -- start 2021-02-10T11:35:56.242 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.238 7f62ac755700 1 -- start start 2021-02-10T11:35:56.242 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.239 7f62ac755700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f62a40d1c70 0x7f62a40f20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:56.242 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.239 7f62ac755700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62a40f0b10 con 0x7f62a40d0340 2021-02-10T11:35:56.242 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.239 7f62ac755700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62a41b0be0 con 0x7f62a40d1870 2021-02-10T11:35:56.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.239 7f62ac755700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62a40e5ed0 con 0x7f62a40d1c70 2021-02-10T11:35:56.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.239 7f62a9d70700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f62a40d0340 0x7f62a40ef640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:51042/0 (socket says 172.21.2.112:51042) 2021-02-10T11:35:56.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.239 7f62a9d70700 1 -- 172.21.2.112:0/1933898308 learned_addr learned my addr 172.21.2.112:0/1933898308 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:35:56.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.239 7f62aad72700 1 --2- 172.21.2.112:0/1933898308 >> v2:172.21.2.120:3300/0 conn(0x7f62a40d1c70 0x7f62a40f20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:56.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.239 7f62a956f700 1 -- 172.21.2.112:0/1933898308 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2903068138 0 0) 0x7f62a00010b0 con 0x7f62a40d0340 2021-02-10T11:35:56.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.239 7f62a956f700 1 -- 172.21.2.112:0/1933898308 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6290002fe0 con 0x7f62a40d0340 2021-02-10T11:35:56.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.239 7f62a956f700 1 -- 172.21.2.112:0/1933898308 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2959314356 0 0) 0x7f6298000ff0 con 0x7f62a40d1870 2021-02-10T11:35:56.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.239 7f62a956f700 1 -- 172.21.2.112:0/1933898308 --> v1:172.21.2.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62900042c0 con 0x7f62a40d1870 2021-02-10T11:35:56.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.239 7f62aad72700 1 -- 172.21.2.112:0/1933898308 >> v1:172.21.2.120:6789/0 conn(0x7f62a40d1870 legacy=0x7f62a41b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:56.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.239 7f62aad72700 1 -- 172.21.2.112:0/1933898308 >> v1:172.21.2.112:6789/0 conn(0x7f62a40d0340 legacy=0x7f62a40ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:56.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.239 7f62aad72700 1 -- 172.21.2.112:0/1933898308 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62a40efe30 con 0x7f62a40d1c70 2021-02-10T11:35:56.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.239 7f62aad72700 1 --2- 172.21.2.112:0/1933898308 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f62a40d1c70 0x7f62a40f20a0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 rx=0x7f629c006e50 tx=0x7f629c009df0).ready entity=mon.1 client_cookie=29b95193e450fee1 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:56.245 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.240 7f62a956f700 1 -- 172.21.2.112:0/1933898308 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f629c00b510 con 0x7f62a40d1c70 2021-02-10T11:35:56.245 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.240 7f62a956f700 1 -- 172.21.2.112:0/1933898308 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f629c003030 con 0x7f62a40d1c70 2021-02-10T11:35:56.245 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.240 7f62a956f700 1 -- 172.21.2.112:0/1933898308 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f629c000f90 con 0x7f62a40d1c70 2021-02-10T11:35:56.245 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.240 7f62ac755700 1 -- 172.21.2.112:0/1933898308 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f62a40d1c70 msgr2=0x7f62a40f20a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:56.245 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.240 7f62ac755700 1 --2- 172.21.2.112:0/1933898308 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f62a40d1c70 0x7f62a40f20a0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 rx=0x7f629c006e50 tx=0x7f629c009df0).stop 2021-02-10T11:35:56.245 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.240 7f62ac755700 1 -- 172.21.2.112:0/1933898308 shutdown_connections 2021-02-10T11:35:56.245 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.240 7f62ac755700 1 --2- 172.21.2.112:0/1933898308 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f62a40d1c70 0x7f62a40f20a0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:56.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.240 7f62ac755700 1 -- 172.21.2.112:0/1933898308 >> 172.21.2.112:0/1933898308 conn(0x7f62a40ce450 msgr2=0x7f62a40d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:56.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.240 7f62ac755700 1 -- 172.21.2.112:0/1933898308 shutdown_connections 2021-02-10T11:35:56.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.240 7f62ac755700 1 -- 172.21.2.112:0/1933898308 wait complete. 2021-02-10T11:35:56.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.255 7f62ac755700 1 Processor -- start 2021-02-10T11:35:56.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.255 7f62ac755700 1 -- start start 2021-02-10T11:35:56.249 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.255 7f62ac755700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f62a41b7cf0 0x7f62a419cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:56.249 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.255 7f62ac755700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f62a41b80f0 0x7f62a41b84f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:56.249 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.255 7f62ac755700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f62a419da20 0x7f62a419de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:56.249 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.255 7f62ac755700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62a40a77c0 con 0x7f62a419da20 2021-02-10T11:35:56.249 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.255 7f62ac755700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62a419d4a0 con 0x7f62a41b7cf0 2021-02-10T11:35:56.249 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.255 7f62ac755700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62a40954f0 con 0x7f62a41b80f0 2021-02-10T11:35:56.249 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.255 7f62aad72700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f62a419da20 0x7f62a419de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:56.250 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.255 7f62aad72700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f62a419da20 0x7f62a419de20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:51582/0 (socket says 172.21.2.112:51582) 2021-02-10T11:35:56.250 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.255 7f62aad72700 1 -- 172.21.2.112:0/1711782969 learned_addr learned my addr 172.21.2.112:0/1711782969 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:35:56.250 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.256 7f62aad72700 1 -- 172.21.2.112:0/1711782969 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f62a41b80f0 msgr2=0x7f62a41b84f0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:35:56.250 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.256 7f62aad72700 1 --2- 172.21.2.112:0/1711782969 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f62a41b80f0 0x7f62a41b84f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:56.250 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.256 7f62aad72700 1 -- 172.21.2.112:0/1711782969 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f62a41b7cf0 msgr2=0x7f62a419cdb0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:35:56.250 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.256 7f62aad72700 1 --2- 172.21.2.112:0/1711782969 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f62a41b7cf0 0x7f62a419cdb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:56.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.256 7f62aad72700 1 -- 172.21.2.112:0/1711782969 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62a41b8d60 con 0x7f62a419da20 2021-02-10T11:35:56.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.256 7f62aad72700 1 --2- 172.21.2.112:0/1711782969 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f62a419da20 0x7f62a419de20 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 rx=0x7f629c009310 tx=0x7f629c00bcc0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:56.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.256 7f6296ffd700 1 -- 172.21.2.112:0/1711782969 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f629c00c2f0 con 0x7f62a419da20 2021-02-10T11:35:56.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.256 7f6296ffd700 1 -- 172.21.2.112:0/1711782969 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f629c00c670 con 0x7f62a419da20 2021-02-10T11:35:56.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.256 7f6296ffd700 1 -- 172.21.2.112:0/1711782969 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f629c00cac0 con 0x7f62a419da20 2021-02-10T11:35:56.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.256 7f62ac755700 1 -- 172.21.2.112:0/1711782969 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62a41b8d60 con 0x7f62a419da20 2021-02-10T11:35:56.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.256 7f62ac755700 1 -- 172.21.2.112:0/1711782969 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62a41b8d60 con 0x7f62a419da20 2021-02-10T11:35:56.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.257 7f6296ffd700 1 -- 172.21.2.112:0/1711782969 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f629c006910 con 0x7f62a419da20 2021-02-10T11:35:56.252 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.257 7f6296ffd700 1 --2- 172.21.2.112:0/1711782969 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f628c018700 0x7f628c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:56.252 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.257 7f6296ffd700 1 -- 172.21.2.112:0/1711782969 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f629c032690 con 0x7f62a419da20 2021-02-10T11:35:56.252 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.257 7f62aa571700 1 --2- 172.21.2.112:0/1711782969 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f628c018700 0x7f628c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:56.252 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.258 7f62aa571700 1 --2- 172.21.2.112:0/1711782969 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f628c018700 0x7f628c01abc0 crc :-1 s=READY pgs=743 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:56.275 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.288 7f62ac755700 1 -- 172.21.2.112:0/1711782969 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62a40ebc00 con 0x7f62a419da20 2021-02-10T11:35:56.278 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.291 7f6296ffd700 1 -- 172.21.2.112:0/1711782969 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f629c00be00 con 0x7f62a419da20 2021-02-10T11:35:56.402 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.416 7f62ac755700 1 -- 172.21.2.112:0/1711782969 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f62a40f0b10 con 0x7f62a419da20 2021-02-10T11:35:56.429 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.416 7f6296ffd700 1 -- 172.21.2.112:0/1711782969 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v68) v1 ==== 74+0+11145 (secure 0 0 0) 0x7f629c006910 con 0x7f62a419da20 2021-02-10T11:35:56.430 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:35:56.431 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":68,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:35:55.162503","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:35:56.431 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.418 7f62ac755700 1 -- 172.21.2.112:0/1711782969 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f628c018700 msgr2=0x7f628c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:56.431 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.418 7f62ac755700 1 --2- 172.21.2.112:0/1711782969 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f628c018700 0x7f628c01abc0 crc :-1 s=READY pgs=743 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:56.431 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.418 7f62ac755700 1 -- 172.21.2.112:0/1711782969 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f62a419da20 msgr2=0x7f62a419de20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:56.431 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.418 7f62ac755700 1 --2- 172.21.2.112:0/1711782969 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f62a419da20 0x7f62a419de20 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 rx=0x7f629c009310 tx=0x7f629c00bcc0).stop 2021-02-10T11:35:56.431 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.418 7f62ac755700 1 -- 172.21.2.112:0/1711782969 shutdown_connections 2021-02-10T11:35:56.431 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.418 7f62ac755700 1 --2- 172.21.2.112:0/1711782969 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f628c018700 0x7f628c01abc0 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:56.431 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.419 7f62ac755700 1 --2- 172.21.2.112:0/1711782969 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f62a419da20 0x7f62a419de20 unknown :-1 s=CLOSED pgs=1540 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:56.431 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.419 7f62ac755700 1 --2- 172.21.2.112:0/1711782969 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f62a41b80f0 0x7f62a41b84f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:56.431 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.419 7f62ac755700 1 --2- 172.21.2.112:0/1711782969 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f62a41b7cf0 0x7f62a419cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:56.432 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.419 7f62ac755700 1 -- 172.21.2.112:0/1711782969 >> 172.21.2.112:0/1711782969 conn(0x7f62a40ed810 msgr2=0x7f62a4094980 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:56.432 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.419 7f62ac755700 1 -- 172.21.2.112:0/1711782969 shutdown_connections 2021-02-10T11:35:56.432 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.419 7f62ac755700 1 -- 172.21.2.112:0/1711782969 wait complete. 2021-02-10T11:35:56.432 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:13:57.508758', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2021-02-10T11:35:56.432 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:35:56.862 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.858 7f7f40974700 1 Processor -- start 2021-02-10T11:35:56.862 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.859 7f7f40974700 1 -- start start 2021-02-10T11:35:56.862 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.859 7f7f40974700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f7f3c10aff0 0x7f7f3c0cf750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:56.862 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.859 7f7f40974700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f3c0fe1c0 con 0x7f7f3c0d1c70 2021-02-10T11:35:56.862 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.859 7f7f40974700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f3c1b0cb0 con 0x7f7f3c0d1870 2021-02-10T11:35:56.862 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.859 7f7f40974700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f3c109a60 con 0x7f7f3c10aff0 2021-02-10T11:35:56.862 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.859 7f7f39e1b700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f7f3c0d1c70 0x7f7f3c0cef60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:51052/0 (socket says 172.21.2.112:51052) 2021-02-10T11:35:56.862 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.859 7f7f39e1b700 1 -- 172.21.2.112:0/2945526350 learned_addr learned my addr 172.21.2.112:0/2945526350 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:35:56.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.859 7f7f3ae1d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f7f3c10aff0 0x7f7f3c0cf750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:56.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.860 7f7f3ae1d700 1 -- 172.21.2.112:0/2945526350 >> v1:172.21.2.120:6789/0 conn(0x7f7f3c0d1870 legacy=0x7f7f3c1b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:56.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.860 7f7f3ae1d700 1 -- 172.21.2.112:0/2945526350 >> v1:172.21.2.112:6789/0 conn(0x7f7f3c0d1c70 legacy=0x7f7f3c0cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:56.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.860 7f7f3ae1d700 1 -- 172.21.2.112:0/2945526350 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f3c10b3f0 con 0x7f7f3c10aff0 2021-02-10T11:35:56.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.860 7f7f3ae1d700 1 --2- 172.21.2.112:0/2945526350 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7f3c10aff0 0x7f7f3c0cf750 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 rx=0x7f7f34006e30 tx=0x7f7f34009e60).ready entity=mon.0 client_cookie=b36ac574ffeab80c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:56.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.860 7f7f3961a700 1 -- 172.21.2.112:0/2945526350 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7f34006a50 con 0x7f7f3c10aff0 2021-02-10T11:35:56.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.860 7f7f3961a700 1 -- 172.21.2.112:0/2945526350 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f3400b300 con 0x7f7f3c10aff0 2021-02-10T11:35:56.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.860 7f7f3961a700 1 -- 172.21.2.112:0/2945526350 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7f3400b820 con 0x7f7f3c10aff0 2021-02-10T11:35:56.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.860 7f7f40974700 1 -- 172.21.2.112:0/2945526350 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7f3c10aff0 msgr2=0x7f7f3c0cf750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:56.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.860 7f7f40974700 1 --2- 172.21.2.112:0/2945526350 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7f3c10aff0 0x7f7f3c0cf750 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 rx=0x7f7f34006e30 tx=0x7f7f34009e60).stop 2021-02-10T11:35:56.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.861 7f7f40974700 1 -- 172.21.2.112:0/2945526350 shutdown_connections 2021-02-10T11:35:56.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.861 7f7f40974700 1 --2- 172.21.2.112:0/2945526350 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7f3c10aff0 0x7f7f3c0cf750 unknown :-1 s=CLOSED pgs=1542 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:56.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.861 7f7f40974700 1 -- 172.21.2.112:0/2945526350 >> 172.21.2.112:0/2945526350 conn(0x7f7f3c0ce420 msgr2=0x7f7f3c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:56.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.861 7f7f40974700 1 -- 172.21.2.112:0/2945526350 shutdown_connections 2021-02-10T11:35:56.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.861 7f7f40974700 1 -- 172.21.2.112:0/2945526350 wait complete. 2021-02-10T11:35:56.865 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.876 7f7f40974700 1 Processor -- start 2021-02-10T11:35:56.865 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.876 7f7f40974700 1 -- start start 2021-02-10T11:35:56.865 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.876 7f7f40974700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7f3c0c0740 0x7f7f3c1c5580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:56.865 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.876 7f7f40974700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7f3c1c5ba0 0x7f7f3c0a5c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:56.865 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.876 7f7f40974700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7f3c0b6e60 0x7f7f3c0a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:56.865 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.876 7f7f40974700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f3c0a6880 con 0x7f7f3c1c5ba0 2021-02-10T11:35:56.866 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.876 7f7f40974700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f3c0a6aa0 con 0x7f7f3c0c0740 2021-02-10T11:35:56.866 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.876 7f7f40974700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f3c0a6cc0 con 0x7f7f3c0b6e60 2021-02-10T11:35:56.866 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.876 7f7f39e1b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7f3c1c5ba0 0x7f7f3c0a5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:56.866 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.876 7f7f39e1b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7f3c1c5ba0 0x7f7f3c0a5c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:51596/0 (socket says 172.21.2.112:51596) 2021-02-10T11:35:56.866 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.876 7f7f39e1b700 1 -- 172.21.2.112:0/2988503489 learned_addr learned my addr 172.21.2.112:0/2988503489 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:35:56.866 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.877 7f7f39e1b700 1 -- 172.21.2.112:0/2988503489 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7f3c0b6e60 msgr2=0x7f7f3c0a6260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:56.866 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.877 7f7f39e1b700 1 --2- 172.21.2.112:0/2988503489 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7f3c0b6e60 0x7f7f3c0a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:56.866 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.877 7f7f39e1b700 1 -- 172.21.2.112:0/2988503489 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7f3c0c0740 msgr2=0x7f7f3c1c5580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:56.866 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.877 7f7f39e1b700 1 --2- 172.21.2.112:0/2988503489 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7f3c0c0740 0x7f7f3c1c5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:56.866 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.877 7f7f39e1b700 1 -- 172.21.2.112:0/2988503489 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f3c0a6ee0 con 0x7f7f3c1c5ba0 2021-02-10T11:35:56.866 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.877 7f7f39e1b700 1 --2- 172.21.2.112:0/2988503489 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7f3c1c5ba0 0x7f7f3c0a5c40 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 rx=0x7f7f2c001400 tx=0x7f7f2c00a900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:56.866 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.877 7f7f2affd700 1 -- 172.21.2.112:0/2988503489 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7f2c00acf0 con 0x7f7f3c1c5ba0 2021-02-10T11:35:56.867 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.877 7f7f2affd700 1 -- 172.21.2.112:0/2988503489 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f2c000c90 con 0x7f7f3c1c5ba0 2021-02-10T11:35:56.867 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.877 7f7f2affd700 1 -- 172.21.2.112:0/2988503489 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7f2c004590 con 0x7f7f3c1c5ba0 2021-02-10T11:35:56.867 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.877 7f7f40974700 1 -- 172.21.2.112:0/2988503489 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f3c0a6ee0 con 0x7f7f3c1c5ba0 2021-02-10T11:35:56.867 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.877 7f7f40974700 1 -- 172.21.2.112:0/2988503489 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f3c0b5650 con 0x7f7f3c1c5ba0 2021-02-10T11:35:56.867 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.878 7f7f2affd700 1 -- 172.21.2.112:0/2988503489 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7f2c0043a0 con 0x7f7f3c1c5ba0 2021-02-10T11:35:56.867 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.878 7f7f2affd700 1 --2- 172.21.2.112:0/2988503489 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7f200186c0 0x7f7f2001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:56.867 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.878 7f7f2affd700 1 -- 172.21.2.112:0/2988503489 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f7f2c032290 con 0x7f7f3c1c5ba0 2021-02-10T11:35:56.867 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.878 7f7f3a61c700 1 --2- 172.21.2.112:0/2988503489 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7f200186c0 0x7f7f2001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:56.868 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.879 7f7f3a61c700 1 --2- 172.21.2.112:0/2988503489 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7f200186c0 0x7f7f2001abc0 crc :-1 s=READY pgs=744 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:56.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.909 7f7f40974700 1 -- 172.21.2.112:0/2988503489 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f3c0807d0 con 0x7f7f3c1c5ba0 2021-02-10T11:35:56.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:56.912 7f7f2affd700 1 -- 172.21.2.112:0/2988503489 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7f2c030d70 con 0x7f7f3c1c5ba0 2021-02-10T11:35:57.024 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:57.038 7f7f40974700 1 -- 172.21.2.112:0/2988503489 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f7f3c088070 con 0x7f7f3c1c5ba0 2021-02-10T11:35:57.025 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:57.038 7f7f2affd700 1 -- 172.21.2.112:0/2988503489 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v68) v1 ==== 92+0+10 (secure 0 0 0) 0x7f7f2c00acf0 con 0x7f7f3c1c5ba0 2021-02-10T11:35:57.025 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:35:57.026 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:57.040 7f7f40974700 1 -- 172.21.2.112:0/2988503489 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7f200186c0 msgr2=0x7f7f2001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:57.026 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:57.040 7f7f40974700 1 --2- 172.21.2.112:0/2988503489 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7f200186c0 0x7f7f2001abc0 crc :-1 s=READY pgs=744 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:57.027 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:57.040 7f7f40974700 1 -- 172.21.2.112:0/2988503489 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7f3c1c5ba0 msgr2=0x7f7f3c0a5c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:57.027 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:57.040 7f7f40974700 1 --2- 172.21.2.112:0/2988503489 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7f3c1c5ba0 0x7f7f3c0a5c40 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 rx=0x7f7f2c001400 tx=0x7f7f2c00a900).stop 2021-02-10T11:35:57.027 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:57.040 7f7f40974700 1 -- 172.21.2.112:0/2988503489 shutdown_connections 2021-02-10T11:35:57.027 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:57.040 7f7f40974700 1 --2- 172.21.2.112:0/2988503489 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7f200186c0 0x7f7f2001abc0 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:57.027 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:57.040 7f7f40974700 1 --2- 172.21.2.112:0/2988503489 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7f3c0b6e60 0x7f7f3c0a6260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:57.027 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:57.040 7f7f40974700 1 --2- 172.21.2.112:0/2988503489 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7f3c1c5ba0 0x7f7f3c0a5c40 unknown :-1 s=CLOSED pgs=1543 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:57.027 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:57.040 7f7f40974700 1 --2- 172.21.2.112:0/2988503489 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7f3c0c0740 0x7f7f3c1c5580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:57.027 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:57.040 7f7f40974700 1 -- 172.21.2.112:0/2988503489 >> 172.21.2.112:0/2988503489 conn(0x7f7f3c0c4120 msgr2=0x7f7f3c0ffd60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:57.027 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:57.040 7f7f40974700 1 -- 172.21.2.112:0/2988503489 shutdown_connections 2021-02-10T11:35:57.027 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:57.040 7f7f40974700 1 -- 172.21.2.112:0/2988503489 wait complete. 2021-02-10T11:35:57.035 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --format=json-pretty osd lspools 2021-02-10T11:35:57.464 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:57.458 7f21b3b95700 1 Processor -- start 2021-02-10T11:35:57.464 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:57.458 7f21b3b95700 1 -- start start 2021-02-10T11:35:57.464 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:57.459 7f21b3b95700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f21ac0d1870 0x7f21ac1b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:57.464 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:57.459 7f21b3b95700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f21ac0f0650 0x7f21ac0cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:57.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:57.459 7f21b3b95700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21ac0f0b10 con 0x7f21ac1b0a10 2021-02-10T11:35:57.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:57.459 7f21b3b95700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21ac0fe230 con 0x7f21ac0f0650 2021-02-10T11:35:57.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:57.459 7f21b3b95700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21ac0e5ed0 con 0x7f21ac0d1870 2021-02-10T11:35:57.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:57.459 7f21b11b0700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f21ac1b0a10 0x7f21ac0cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:51066/0 (socket says 172.21.2.112:51066) 2021-02-10T11:35:57.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:57.459 7f21b11b0700 1 -- 172.21.2.112:0/3135471130 learned_addr learned my addr 172.21.2.112:0/3135471130 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:35:57.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:57.460 7f21b09af700 1 -- 172.21.2.112:0/3135471130 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1807686341 0 0) 0x7f219c0010b0 con 0x7f21ac1b0a10 2021-02-10T11:35:57.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:57.460 7f21b09af700 1 -- 172.21.2.112:0/3135471130 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2198002fe0 con 0x7f21ac1b0a10 2021-02-10T11:35:57.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:57.460 7f21b21b2700 1 --2- 172.21.2.112:0/3135471130 >> v2:172.21.2.120:3300/0 conn(0x7f21ac0f0650 0x7f21ac0cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:57.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:57.460 7f21b19b1700 1 --2- 172.21.2.112:0/3135471130 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f21ac0d1870 0x7f21ac1b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:57.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:57.460 7f21b09af700 1 -- 172.21.2.112:0/3135471130 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1570284846 0 0) 0x7f219c0010b0 con 0x7f21ac1b0a10 2021-02-10T11:35:57.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:57.460 7f21b09af700 1 -- 172.21.2.112:0/3135471130 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f21ac0d1870 msgr2=0x7f21ac1b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:57.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:57.460 7f21b09af700 1 --2- 172.21.2.112:0/3135471130 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f21ac0d1870 0x7f21ac1b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:57.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:57.460 7f21b09af700 1 -- 172.21.2.112:0/3135471130 >> v2:172.21.2.120:3300/0 conn(0x7f21ac0f0650 msgr2=0x7f21ac0cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:57.466 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:57.460 7f21b09af700 1 --2- 172.21.2.112:0/3135471130 >> v2:172.21.2.120:3300/0 conn(0x7f21ac0f0650 0x7f21ac0cfb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:57.466 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:57.460 7f21b09af700 1 -- 172.21.2.112:0/3135471130 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21ac0f20a0 con 0x7f21ac1b0a10 2021-02-10T11:35:57.466 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:57.460 7f21b09af700 1 -- 172.21.2.112:0/3135471130 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f219c0015b0 con 0x7f21ac1b0a10 2021-02-10T11:35:57.466 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:57.460 7f21b09af700 1 -- 172.21.2.112:0/3135471130 >> v1:172.21.2.112:6789/0 conn(0x7f21ac1b0a10 legacy=0x7f21ac0cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:57.466 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:57.460 7f21b09af700 1 --2- 172.21.2.112:0/3135471130 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2198003b90 0x7f2198006040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:57.466 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:57.460 7f21b09af700 1 --2- 172.21.2.112:0/3135471130 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f21980068e0 0x7f2198008de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:57.466 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:57.460 7f21b09af700 1 --2- 172.21.2.112:0/3135471130 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f21980096e0 0x7f219800bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:57.466 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:57.460 7f21b09af700 1 -- 172.21.2.112:0/3135471130 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f219800c420 con 0x7f2198003b90 2021-02-10T11:35:57.466 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:57.460 7f21b09af700 1 -- 172.21.2.112:0/3135471130 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2198050610 con 0x7f21980068e0 2021-02-10T11:35:57.466 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:57.460 7f21b09af700 1 -- 172.21.2.112:0/3135471130 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2198050830 con 0x7f21980096e0 2021-02-10T11:35:57.466 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:57.460 7f21b19b1700 1 --2- 172.21.2.112:0/3135471130 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f21ac0d1870 0x7f21ac1b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:35:57.466 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:57.460 7f21b21b2700 1 --2- 172.21.2.112:0/3135471130 >> v2:172.21.2.120:3300/0 conn(0x7f21ac0f0650 0x7f21ac0cfb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:35:57.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:57.460 7f21b19b1700 1 --2- 172.21.2.112:0/3135471130 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2198003b90 0x7f2198006040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:57.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:57.461 7f21b19b1700 1 -- 172.21.2.112:0/3135471130 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f21980096e0 msgr2=0x7f219800bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:57.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:57.461 7f21b19b1700 1 --2- 172.21.2.112:0/3135471130 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f21980096e0 0x7f219800bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:57.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:57.461 7f21b19b1700 1 -- 172.21.2.112:0/3135471130 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f21980068e0 msgr2=0x7f2198008de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:57.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:57.461 7f21b19b1700 1 --2- 172.21.2.112:0/3135471130 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f21980068e0 0x7f2198008de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:57.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:57.461 7f21b19b1700 1 -- 172.21.2.112:0/3135471130 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2198050ad0 con 0x7f2198003b90 2021-02-10T11:35:57.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:57.461 7f21b19b1700 1 --2- 172.21.2.112:0/3135471130 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2198003b90 0x7f2198006040 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 rx=0x7f21a8006890 tx=0x7f21a8009aa0).ready entity=mon.0 client_cookie=58fff662b5134382 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:57.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:57.461 7f21b09af700 1 -- 172.21.2.112:0/3135471130 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f21a800c0c0 con 0x7f2198003b90 2021-02-10T11:35:57.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:57.461 7f21b09af700 1 -- 172.21.2.112:0/3135471130 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21a800c340 con 0x7f2198003b90 2021-02-10T11:35:57.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:57.461 7f21b09af700 1 -- 172.21.2.112:0/3135471130 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f21a800c820 con 0x7f2198003b90 2021-02-10T11:35:57.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:57.461 7f21b3b95700 1 -- 172.21.2.112:0/3135471130 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2198003b90 msgr2=0x7f2198006040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:57.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:57.461 7f21b3b95700 1 --2- 172.21.2.112:0/3135471130 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2198003b90 0x7f2198006040 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 rx=0x7f21a8006890 tx=0x7f21a8009aa0).stop 2021-02-10T11:35:57.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:57.461 7f21b21b2700 1 -- 172.21.2.112:0/3135471130 reap_dead start 2021-02-10T11:35:57.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:57.462 7f21b3b95700 1 -- 172.21.2.112:0/3135471130 shutdown_connections 2021-02-10T11:35:57.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:57.462 7f21b3b95700 1 -- 172.21.2.112:0/3135471130 >> 172.21.2.112:0/3135471130 conn(0x7f21ac0ce450 msgr2=0x7f21ac0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:57.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:57.462 7f21b3b95700 1 -- 172.21.2.112:0/3135471130 shutdown_connections 2021-02-10T11:35:57.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:57.462 7f21b3b95700 1 -- 172.21.2.112:0/3135471130 wait complete. 2021-02-10T11:35:57.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:57.477 7f21b3b95700 1 Processor -- start 2021-02-10T11:35:57.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:57.477 7f21b3b95700 1 -- start start 2021-02-10T11:35:57.470 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:57.477 7f21b3b95700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f21ac1942b0 0x7f21ac0b3550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:57.470 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:57.477 7f21b3b95700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f21ac195070 0x7f21ac195470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:57.470 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:57.477 7f21b3b95700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f21ac0bb8e0 0x7f21ac0bbce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:57.470 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:57.477 7f21b3b95700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21ac1946b0 con 0x7f21ac195070 2021-02-10T11:35:57.470 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:57.477 7f21b3b95700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21ac0b3b70 con 0x7f21ac1942b0 2021-02-10T11:35:57.470 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:57.477 7f21b3b95700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21ac0b3d90 con 0x7f21ac0bb8e0 2021-02-10T11:35:57.470 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:57.477 7f21b11b0700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f21ac195070 0x7f21ac195470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:57.470 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:57.477 7f21b11b0700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f21ac195070 0x7f21ac195470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:51620/0 (socket says 172.21.2.112:51620) 2021-02-10T11:35:57.470 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:57.477 7f21b11b0700 1 -- 172.21.2.112:0/1944899221 learned_addr learned my addr 172.21.2.112:0/1944899221 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:35:57.470 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:57.478 7f21b11b0700 1 -- 172.21.2.112:0/1944899221 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f21ac0bb8e0 msgr2=0x7f21ac0bbce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:57.470 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:57.478 7f21b11b0700 1 --2- 172.21.2.112:0/1944899221 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f21ac0bb8e0 0x7f21ac0bbce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:57.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:57.478 7f21b11b0700 1 -- 172.21.2.112:0/1944899221 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f21ac1942b0 msgr2=0x7f21ac0b3550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:57.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:57.478 7f21b11b0700 1 --2- 172.21.2.112:0/1944899221 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f21ac1942b0 0x7f21ac0b3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:57.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:57.478 7f21b11b0700 1 -- 172.21.2.112:0/1944899221 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21ac1a49e0 con 0x7f21ac195070 2021-02-10T11:35:57.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:57.478 7f21b11b0700 1 --2- 172.21.2.112:0/1944899221 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f21ac195070 0x7f21ac195470 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 rx=0x7f219c001c60 tx=0x7f219c00a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:57.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:57.478 7f21a27fc700 1 -- 172.21.2.112:0/1944899221 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f219c00acc0 con 0x7f21ac195070 2021-02-10T11:35:57.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:57.478 7f21a27fc700 1 -- 172.21.2.112:0/1944899221 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f219c004350 con 0x7f21ac195070 2021-02-10T11:35:57.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:57.478 7f21a27fc700 1 -- 172.21.2.112:0/1944899221 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f219c0047e0 con 0x7f21ac195070 2021-02-10T11:35:57.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:57.478 7f21b3b95700 1 -- 172.21.2.112:0/1944899221 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21ac1a49e0 con 0x7f21ac195070 2021-02-10T11:35:57.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:57.478 7f21b3b95700 1 -- 172.21.2.112:0/1944899221 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21ac1a49e0 con 0x7f21ac195070 2021-02-10T11:35:57.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:57.479 7f21a27fc700 1 -- 172.21.2.112:0/1944899221 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f219c000d60 con 0x7f21ac195070 2021-02-10T11:35:57.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:57.479 7f21a27fc700 1 --2- 172.21.2.112:0/1944899221 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f21940186c0 0x7f219401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:57.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:57.479 7f21a27fc700 1 -- 172.21.2.112:0/1944899221 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f219c032240 con 0x7f21ac195070 2021-02-10T11:35:57.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:57.479 7f21b19b1700 1 --2- 172.21.2.112:0/1944899221 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f21940186c0 0x7f219401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:57.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:57.480 7f21b19b1700 1 --2- 172.21.2.112:0/1944899221 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f21940186c0 0x7f219401abc0 crc :-1 s=READY pgs=745 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:57.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:57.509 7f21b3b95700 1 -- 172.21.2.112:0/1944899221 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21ac1a49e0 con 0x7f21ac195070 2021-02-10T11:35:57.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:57.511 7f21a27fc700 1 -- 172.21.2.112:0/1944899221 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f219c025fa0 con 0x7f21ac195070 2021-02-10T11:35:57.624 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:57.637 7f21b3b95700 1 -- 172.21.2.112:0/1944899221 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f21ac1a49e0 con 0x7f21ac195070 2021-02-10T11:35:57.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:57.637 7f21a27fc700 1 -- 172.21.2.112:0/1944899221 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v68) v1 ==== 84+0+64 (secure 0 0 0) 0x7f219c025fa0 con 0x7f21ac195070 2021-02-10T11:35:57.625 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:35:57.625 INFO:teuthology.orchestra.run.gibba012.stdout:[ 2021-02-10T11:35:57.625 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:35:57.625 INFO:teuthology.orchestra.run.gibba012.stdout: "poolnum": 1, 2021-02-10T11:35:57.625 INFO:teuthology.orchestra.run.gibba012.stdout: "poolname": "rbd" 2021-02-10T11:35:57.625 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:35:57.625 INFO:teuthology.orchestra.run.gibba012.stdout:] 2021-02-10T11:35:57.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:57.639 7f21b3b95700 1 -- 172.21.2.112:0/1944899221 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f21940186c0 msgr2=0x7f219401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:57.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:57.639 7f21b3b95700 1 --2- 172.21.2.112:0/1944899221 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f21940186c0 0x7f219401abc0 crc :-1 s=READY pgs=745 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:57.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:57.639 7f21b3b95700 1 -- 172.21.2.112:0/1944899221 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f21ac195070 msgr2=0x7f21ac195470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:57.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:57.639 7f21b3b95700 1 --2- 172.21.2.112:0/1944899221 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f21ac195070 0x7f21ac195470 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 rx=0x7f219c001c60 tx=0x7f219c00a8d0).stop 2021-02-10T11:35:57.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:57.639 7f21b3b95700 1 -- 172.21.2.112:0/1944899221 shutdown_connections 2021-02-10T11:35:57.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:57.639 7f21b3b95700 1 --2- 172.21.2.112:0/1944899221 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f21940186c0 0x7f219401abc0 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:57.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:57.639 7f21b3b95700 1 --2- 172.21.2.112:0/1944899221 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f21ac0bb8e0 0x7f21ac0bbce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:57.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:57.639 7f21b3b95700 1 --2- 172.21.2.112:0/1944899221 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f21ac195070 0x7f21ac195470 unknown :-1 s=CLOSED pgs=1546 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:57.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:57.639 7f21b3b95700 1 --2- 172.21.2.112:0/1944899221 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f21ac1942b0 0x7f21ac0b3550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:57.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:57.639 7f21b3b95700 1 -- 172.21.2.112:0/1944899221 >> 172.21.2.112:0/1944899221 conn(0x7f21ac0b3040 msgr2=0x7f21ac0c3250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:57.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:57.639 7f21b3b95700 1 -- 172.21.2.112:0/1944899221 shutdown_connections 2021-02-10T11:35:57.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:57.639 7f21b3b95700 1 -- 172.21.2.112:0/1944899221 wait complete. 2021-02-10T11:35:57.635 INFO:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2021-02-10T11:35:57.635 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_metadata 8 2021-02-10T11:35:58.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:58.062 7ffab25f7700 1 Processor -- start 2021-02-10T11:35:58.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:58.062 7ffab25f7700 1 -- start start 2021-02-10T11:35:58.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:58.062 7ffab25f7700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ffaac0d1870 0x7ffaac1b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:58.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:58.062 7ffab25f7700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7ffaac0cff70 0x7ffaac109130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:58.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:58.062 7ffab25f7700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffaac109a60 con 0x7ffaac1b0ae0 2021-02-10T11:35:58.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:58.062 7ffab25f7700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffaac109750 con 0x7ffaac0cff70 2021-02-10T11:35:58.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:58.062 7ffab25f7700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffaac088070 con 0x7ffaac0d1870 2021-02-10T11:35:58.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:58.063 7ffaab7fe700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7ffaac1b0ae0 0x7ffaac108940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:51086/0 (socket says 172.21.2.112:51086) 2021-02-10T11:35:58.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:58.063 7ffaab7fe700 1 -- 172.21.2.112:0/1280243595 learned_addr learned my addr 172.21.2.112:0/1280243595 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:35:58.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:58.063 7ffab0c14700 1 --2- 172.21.2.112:0/1280243595 >> v2:172.21.2.112:3300/0 conn(0x7ffaac0cff70 0x7ffaac109130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:58.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:58.063 7ffab0c14700 1 -- 172.21.2.112:0/1280243595 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ffaac0d1870 msgr2=0x7ffaac1b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:58.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:58.063 7ffab0c14700 1 --2- 172.21.2.112:0/1280243595 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ffaac0d1870 0x7ffaac1b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:58.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:58.063 7ffab0c14700 1 -- 172.21.2.112:0/1280243595 >> v1:172.21.2.112:6789/0 conn(0x7ffaac1b0ae0 legacy=0x7ffaac108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:58.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:58.063 7ffab0c14700 1 -- 172.21.2.112:0/1280243595 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaac088260 con 0x7ffaac0cff70 2021-02-10T11:35:58.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:58.063 7ffab0c14700 1 --2- 172.21.2.112:0/1280243595 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ffaac0cff70 0x7ffaac109130 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 rx=0x7ffaa4006e30 tx=0x7ffaa4009e60).ready entity=mon.0 client_cookie=20670cae194758d5 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:58.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:58.063 7ffaaaffd700 1 -- 172.21.2.112:0/1280243595 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffaa4005820 con 0x7ffaac0cff70 2021-02-10T11:35:58.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:58.063 7ffaaaffd700 1 -- 172.21.2.112:0/1280243595 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffaa4005a60 con 0x7ffaac0cff70 2021-02-10T11:35:58.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:58.063 7ffaaaffd700 1 -- 172.21.2.112:0/1280243595 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffaa4006030 con 0x7ffaac0cff70 2021-02-10T11:35:58.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:58.063 7ffab25f7700 1 -- 172.21.2.112:0/1280243595 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ffaac0cff70 msgr2=0x7ffaac109130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:58.068 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:58.063 7ffab25f7700 1 --2- 172.21.2.112:0/1280243595 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ffaac0cff70 0x7ffaac109130 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 rx=0x7ffaa4006e30 tx=0x7ffaa4009e60).stop 2021-02-10T11:35:58.068 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:58.064 7ffab25f7700 1 -- 172.21.2.112:0/1280243595 shutdown_connections 2021-02-10T11:35:58.068 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:58.064 7ffab25f7700 1 --2- 172.21.2.112:0/1280243595 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ffaac0cff70 0x7ffaac109130 unknown :-1 s=CLOSED pgs=1548 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:58.068 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:58.064 7ffab25f7700 1 --2- 172.21.2.112:0/1280243595 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ffaac0d1870 0x7ffaac1b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:58.068 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:58.064 7ffab25f7700 1 -- 172.21.2.112:0/1280243595 >> 172.21.2.112:0/1280243595 conn(0x7ffaac0ce420 msgr2=0x7ffaac0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:58.068 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:58.064 7ffab25f7700 1 -- 172.21.2.112:0/1280243595 shutdown_connections 2021-02-10T11:35:58.068 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:58.064 7ffab25f7700 1 -- 172.21.2.112:0/1280243595 wait complete. 2021-02-10T11:35:58.069 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:58.079 7ffab25f7700 1 Processor -- start 2021-02-10T11:35:58.069 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:58.079 7ffab25f7700 1 -- start start 2021-02-10T11:35:58.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:58.079 7ffab25f7700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ffaac0c3820 0x7ffaac0a7950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:58.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:58.079 7ffab25f7700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ffaac1a44c0 0x7ffaac1a48c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:58.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:58.079 7ffab25f7700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ffaac0863d0 0x7ffaac0867d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:58.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:58.079 7ffab25f7700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaac085ed0 con 0x7ffaac0c3820 2021-02-10T11:35:58.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:58.079 7ffab25f7700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaac0c0740 con 0x7ffaac0863d0 2021-02-10T11:35:58.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:58.079 7ffab25f7700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffaac086df0 con 0x7ffaac1a44c0 2021-02-10T11:35:58.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:58.080 7ffaabfff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ffaac0c3820 0x7ffaac0a7950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:58.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:58.080 7ffaabfff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ffaac0c3820 0x7ffaac0a7950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:51632/0 (socket says 172.21.2.112:51632) 2021-02-10T11:35:58.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:58.080 7ffaabfff700 1 -- 172.21.2.112:0/3241901711 learned_addr learned my addr 172.21.2.112:0/3241901711 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:35:58.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:58.080 7ffaabfff700 1 -- 172.21.2.112:0/3241901711 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ffaac1a44c0 msgr2=0x7ffaac1a48c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:58.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:58.080 7ffaabfff700 1 --2- 172.21.2.112:0/3241901711 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ffaac1a44c0 0x7ffaac1a48c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:58.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:58.080 7ffaabfff700 1 -- 172.21.2.112:0/3241901711 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ffaac0863d0 msgr2=0x7ffaac0867d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:58.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:58.080 7ffaabfff700 1 --2- 172.21.2.112:0/3241901711 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ffaac0863d0 0x7ffaac0867d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:58.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:58.080 7ffaabfff700 1 -- 172.21.2.112:0/3241901711 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaac087010 con 0x7ffaac0c3820 2021-02-10T11:35:58.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:58.080 7ffaabfff700 1 --2- 172.21.2.112:0/3241901711 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ffaac0c3820 0x7ffaac0a7950 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 rx=0x7ffa9c006f00 tx=0x7ffa9c00b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:58.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:58.080 7ffaa8ff9700 1 -- 172.21.2.112:0/3241901711 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffa9c006b90 con 0x7ffaac0c3820 2021-02-10T11:35:58.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:58.080 7ffaa8ff9700 1 -- 172.21.2.112:0/3241901711 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa9c00bd30 con 0x7ffaac0c3820 2021-02-10T11:35:58.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:58.080 7ffaa8ff9700 1 -- 172.21.2.112:0/3241901711 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffa9c0020d0 con 0x7ffaac0c3820 2021-02-10T11:35:58.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:58.080 7ffab25f7700 1 -- 172.21.2.112:0/3241901711 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffaac087010 con 0x7ffaac0c3820 2021-02-10T11:35:58.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:58.080 7ffab25f7700 1 -- 172.21.2.112:0/3241901711 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffaac083000 con 0x7ffaac0c3820 2021-02-10T11:35:58.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:58.081 7ffaa8ff9700 1 -- 172.21.2.112:0/3241901711 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ffa9c000c90 con 0x7ffaac0c3820 2021-02-10T11:35:58.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:58.081 7ffaa8ff9700 1 --2- 172.21.2.112:0/3241901711 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ffa8c0186c0 0x7ffa8c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:58.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:58.081 7ffaa8ff9700 1 -- 172.21.2.112:0/3241901711 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 4753+0+0 (secure 0 0 0) 0x7ffa9c032260 con 0x7ffaac0c3820 2021-02-10T11:35:58.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:58.081 7ffaab7fe700 1 --2- 172.21.2.112:0/3241901711 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ffa8c0186c0 0x7ffa8c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:58.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:58.082 7ffaab7fe700 1 --2- 172.21.2.112:0/3241901711 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ffa8c0186c0 0x7ffa8c01abc0 crc :-1 s=READY pgs=746 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:58.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:58.112 7ffab25f7700 1 -- 172.21.2.112:0/3241901711 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffaac085e50 con 0x7ffaac0c3820 2021-02-10T11:35:58.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:58.114 7ffaa8ff9700 1 -- 172.21.2.112:0/3241901711 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ffa9c000940 con 0x7ffaac0c3820 2021-02-10T11:35:58.231 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:58.244 7ffab25f7700 1 -- 172.21.2.112:0/3241901711 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"} v 0) v1 -- 0x7ffaac085e50 con 0x7ffaac0c3820 2021-02-10T11:35:58.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:58.643 7ffaa8ff9700 1 -- 172.21.2.112:0/3241901711 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"}]=0 pool 'cephfs_metadata' created v69) v1 ==== 133+0+0 (secure 0 0 0) 0x7ffa9c000c90 con 0x7ffaac0c3820 2021-02-10T11:35:58.647 INFO:teuthology.orchestra.run.gibba012.stderr:pool 'cephfs_metadata' created 2021-02-10T11:35:58.649 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:58.661 7ffab25f7700 1 -- 172.21.2.112:0/3241901711 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ffa8c0186c0 msgr2=0x7ffa8c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:58.649 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:58.661 7ffab25f7700 1 --2- 172.21.2.112:0/3241901711 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ffa8c0186c0 0x7ffa8c01abc0 crc :-1 s=READY pgs=746 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:58.649 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:58.661 7ffab25f7700 1 -- 172.21.2.112:0/3241901711 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ffaac0c3820 msgr2=0x7ffaac0a7950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:58.649 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:58.661 7ffab25f7700 1 --2- 172.21.2.112:0/3241901711 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ffaac0c3820 0x7ffaac0a7950 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 rx=0x7ffa9c006f00 tx=0x7ffa9c00b940).stop 2021-02-10T11:35:58.649 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:58.661 7ffab25f7700 1 -- 172.21.2.112:0/3241901711 shutdown_connections 2021-02-10T11:35:58.649 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:58.661 7ffab25f7700 1 --2- 172.21.2.112:0/3241901711 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ffa8c0186c0 0x7ffa8c01abc0 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:58.650 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:58.661 7ffab25f7700 1 --2- 172.21.2.112:0/3241901711 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ffaac0863d0 0x7ffaac0867d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:58.650 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:58.661 7ffab25f7700 1 --2- 172.21.2.112:0/3241901711 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ffaac1a44c0 0x7ffaac1a48c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:58.650 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:58.661 7ffab25f7700 1 --2- 172.21.2.112:0/3241901711 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ffaac0c3820 0x7ffaac0a7950 unknown :-1 s=CLOSED pgs=1549 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:58.650 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:58.661 7ffab25f7700 1 -- 172.21.2.112:0/3241901711 >> 172.21.2.112:0/3241901711 conn(0x7ffaac0872c0 msgr2=0x7ffaac095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:58.650 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:58.661 7ffab25f7700 1 -- 172.21.2.112:0/3241901711 shutdown_connections 2021-02-10T11:35:58.650 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:58.661 7ffab25f7700 1 -- 172.21.2.112:0/3241901711 wait complete. 2021-02-10T11:35:58.658 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_data 8 2021-02-10T11:35:59.088 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:59.084 7f7b26b7a700 1 Processor -- start 2021-02-10T11:35:59.088 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:59.084 7f7b26b7a700 1 -- start start 2021-02-10T11:35:59.088 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:59.085 7f7b26b7a700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f7b200d1870 0x7f7b201b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:59.088 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:59.085 7f7b26b7a700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f7b200cff70 0x7f7b20109130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:59.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:59.085 7f7b26b7a700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b200fe1c0 con 0x7f7b201b0ae0 2021-02-10T11:35:59.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:59.085 7f7b26b7a700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b20109ad0 con 0x7f7b200d1870 2021-02-10T11:35:59.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:59.085 7f7b26b7a700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b20109750 con 0x7f7b200cff70 2021-02-10T11:35:59.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:59.085 7f7b24996700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f7b200d1870 0x7f7b201b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:59.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:59.085 7f7b24996700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f7b200d1870 0x7f7b201b04c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:51640/0 (socket says 172.21.2.112:51640) 2021-02-10T11:35:59.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:59.085 7f7b24996700 1 -- 172.21.2.112:0/721178954 learned_addr learned my addr 172.21.2.112:0/721178954 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:35:59.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:59.085 7f7b25197700 1 --2- 172.21.2.112:0/721178954 >> v2:172.21.2.120:3300/0 conn(0x7f7b200cff70 0x7f7b20109130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:59.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:59.085 7f7b24996700 1 -- 172.21.2.112:0/721178954 >> v2:172.21.2.120:3300/0 conn(0x7f7b200cff70 msgr2=0x7f7b20109130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:59.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:59.085 7f7b24996700 1 --2- 172.21.2.112:0/721178954 >> v2:172.21.2.120:3300/0 conn(0x7f7b200cff70 0x7f7b20109130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:59.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:59.085 7f7b24996700 1 -- 172.21.2.112:0/721178954 >> v1:172.21.2.120:6789/0 conn(0x7f7b201b0ae0 legacy=0x7f7b20108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:59.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:59.085 7f7b24996700 1 -- 172.21.2.112:0/721178954 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b20088070 con 0x7f7b200d1870 2021-02-10T11:35:59.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:59.086 7f7b24996700 1 --2- 172.21.2.112:0/721178954 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7b200d1870 0x7f7b201b04c0 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 rx=0x7f7b08006700 tx=0x7f7b08009b80).ready entity=mon.0 client_cookie=6fdc8ae830d69a0c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:59.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:59.086 7f7b1f7fe700 1 -- 172.21.2.112:0/721178954 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7b0800c0c0 con 0x7f7b200d1870 2021-02-10T11:35:59.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:59.086 7f7b1f7fe700 1 -- 172.21.2.112:0/721178954 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b0800c440 con 0x7f7b200d1870 2021-02-10T11:35:59.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:59.086 7f7b1f7fe700 1 -- 172.21.2.112:0/721178954 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7b0800c920 con 0x7f7b200d1870 2021-02-10T11:35:59.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:59.086 7f7b26b7a700 1 -- 172.21.2.112:0/721178954 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7b200d1870 msgr2=0x7f7b201b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:59.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:59.086 7f7b26b7a700 1 --2- 172.21.2.112:0/721178954 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7b200d1870 0x7f7b201b04c0 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 rx=0x7f7b08006700 tx=0x7f7b08009b80).stop 2021-02-10T11:35:59.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:59.086 7f7b26b7a700 1 -- 172.21.2.112:0/721178954 shutdown_connections 2021-02-10T11:35:59.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:59.086 7f7b26b7a700 1 --2- 172.21.2.112:0/721178954 >> v2:172.21.2.120:3300/0 conn(0x7f7b200cff70 0x7f7b20109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:59.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:59.086 7f7b26b7a700 1 --2- 172.21.2.112:0/721178954 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7b200d1870 0x7f7b201b04c0 unknown :-1 s=CLOSED pgs=1550 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:59.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:59.086 7f7b26b7a700 1 -- 172.21.2.112:0/721178954 >> 172.21.2.112:0/721178954 conn(0x7f7b200ce420 msgr2=0x7f7b200d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:59.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:59.087 7f7b26b7a700 1 -- 172.21.2.112:0/721178954 shutdown_connections 2021-02-10T11:35:59.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:59.087 7f7b26b7a700 1 -- 172.21.2.112:0/721178954 wait complete. 2021-02-10T11:35:59.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:59.101 7f7b26b7a700 1 Processor -- start 2021-02-10T11:35:59.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:59.101 7f7b26b7a700 1 -- start start 2021-02-10T11:35:59.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:59.102 7f7b26b7a700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7b200a7950 0x7f7b200b8df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:59.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:59.102 7f7b26b7a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7b201a7490 0x7f7b201a7890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:59.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:59.102 7f7b26b7a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7b201a7eb0 0x7f7b200ac630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:59.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:59.102 7f7b26b7a700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b200a7fb0 con 0x7f7b201a7490 2021-02-10T11:35:59.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:59.102 7f7b26b7a700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b200acc50 con 0x7f7b200a7950 2021-02-10T11:35:59.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:59.102 7f7b26b7a700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b200ace30 con 0x7f7b201a7eb0 2021-02-10T11:35:59.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:59.102 7f7b1ffff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7b201a7490 0x7f7b201a7890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:59.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:59.102 7f7b1ffff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7b201a7490 0x7f7b201a7890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:51644/0 (socket says 172.21.2.112:51644) 2021-02-10T11:35:59.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:59.102 7f7b1ffff700 1 -- 172.21.2.112:0/256453438 learned_addr learned my addr 172.21.2.112:0/256453438 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:35:59.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:59.102 7f7b25197700 1 --2- 172.21.2.112:0/256453438 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7b201a7eb0 0x7f7b200ac630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:59.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:59.102 7f7b24996700 1 --2- 172.21.2.112:0/256453438 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7b200a7950 0x7f7b200b8df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:59.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:59.102 7f7b1ffff700 1 -- 172.21.2.112:0/256453438 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7b201a7eb0 msgr2=0x7f7b200ac630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:59.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:59.102 7f7b1ffff700 1 --2- 172.21.2.112:0/256453438 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7b201a7eb0 0x7f7b200ac630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:59.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:59.102 7f7b1ffff700 1 -- 172.21.2.112:0/256453438 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7b200a7950 msgr2=0x7f7b200b8df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:59.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:59.102 7f7b1ffff700 1 --2- 172.21.2.112:0/256453438 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7b200a7950 0x7f7b200b8df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:59.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:59.102 7f7b1ffff700 1 -- 172.21.2.112:0/256453438 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b200ad050 con 0x7f7b201a7490 2021-02-10T11:35:59.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:59.102 7f7b1ffff700 1 --2- 172.21.2.112:0/256453438 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7b201a7490 0x7f7b201a7890 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 rx=0x7f7b1000b5b0 tx=0x7f7b1000b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:59.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:59.103 7f7b1d7fa700 1 -- 172.21.2.112:0/256453438 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7b10000960 con 0x7f7b201a7490 2021-02-10T11:35:59.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:59.103 7f7b1d7fa700 1 -- 172.21.2.112:0/256453438 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b1000bd40 con 0x7f7b201a7490 2021-02-10T11:35:59.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:59.103 7f7b1d7fa700 1 -- 172.21.2.112:0/256453438 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7b100054a0 con 0x7f7b201a7490 2021-02-10T11:35:59.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:59.103 7f7b26b7a700 1 -- 172.21.2.112:0/256453438 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b200ad050 con 0x7f7b201a7490 2021-02-10T11:35:59.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:59.103 7f7b26b7a700 1 -- 172.21.2.112:0/256453438 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b200a8530 con 0x7f7b201a7490 2021-02-10T11:35:59.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:59.103 7f7b1d7fa700 1 -- 172.21.2.112:0/256453438 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7b10006030 con 0x7f7b201a7490 2021-02-10T11:35:59.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:59.103 7f7b1d7fa700 1 --2- 172.21.2.112:0/256453438 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7b040186c0 0x7f7b0401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:59.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:59.103 7f7b1d7fa700 1 -- 172.21.2.112:0/256453438 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5107+0+0 (secure 0 0 0) 0x7f7b100323d0 con 0x7f7b201a7490 2021-02-10T11:35:59.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:59.103 7f7b24996700 1 --2- 172.21.2.112:0/256453438 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7b040186c0 0x7f7b0401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:59.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:59.104 7f7b24996700 1 --2- 172.21.2.112:0/256453438 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7b040186c0 0x7f7b0401abc0 crc :-1 s=READY pgs=747 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:59.122 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:59.134 7f7b26b7a700 1 -- 172.21.2.112:0/256453438 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b20082a70 con 0x7f7b201a7490 2021-02-10T11:35:59.124 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:59.137 7f7b1d7fa700 1 -- 172.21.2.112:0/256453438 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7b10000c60 con 0x7f7b201a7490 2021-02-10T11:35:59.253 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:59.265 7f7b26b7a700 1 -- 172.21.2.112:0/256453438 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"} v 0) v1 -- 0x7f7b20082a70 con 0x7f7b201a7490 2021-02-10T11:35:59.638 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:59.650 7f7b1d7fa700 1 -- 172.21.2.112:0/256453438 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"}]=0 pool 'cephfs_data' created v71) v1 ==== 125+0+0 (secure 0 0 0) 0x7f7b10006030 con 0x7f7b201a7490 2021-02-10T11:35:59.669 INFO:teuthology.orchestra.run.gibba012.stderr:pool 'cephfs_data' created 2021-02-10T11:35:59.670 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:59.683 7f7b26b7a700 1 -- 172.21.2.112:0/256453438 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7b040186c0 msgr2=0x7f7b0401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:59.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:59.683 7f7b26b7a700 1 --2- 172.21.2.112:0/256453438 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7b040186c0 0x7f7b0401abc0 crc :-1 s=READY pgs=747 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:59.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:59.683 7f7b26b7a700 1 -- 172.21.2.112:0/256453438 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7b201a7490 msgr2=0x7f7b201a7890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:59.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:59.683 7f7b26b7a700 1 --2- 172.21.2.112:0/256453438 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7b201a7490 0x7f7b201a7890 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 rx=0x7f7b1000b5b0 tx=0x7f7b1000b950).stop 2021-02-10T11:35:59.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:59.683 7f7b26b7a700 1 -- 172.21.2.112:0/256453438 shutdown_connections 2021-02-10T11:35:59.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:59.683 7f7b26b7a700 1 --2- 172.21.2.112:0/256453438 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7b040186c0 0x7f7b0401abc0 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:59.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:59.683 7f7b26b7a700 1 --2- 172.21.2.112:0/256453438 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7b201a7eb0 0x7f7b200ac630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:59.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:59.683 7f7b26b7a700 1 --2- 172.21.2.112:0/256453438 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7b201a7490 0x7f7b201a7890 unknown :-1 s=CLOSED pgs=1551 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:59.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:59.683 7f7b26b7a700 1 --2- 172.21.2.112:0/256453438 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7b200a7950 0x7f7b200b8df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:59.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:59.683 7f7b26b7a700 1 -- 172.21.2.112:0/256453438 >> 172.21.2.112:0/256453438 conn(0x7f7b200a02d0 msgr2=0x7f7b2009f200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:59.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:59.683 7f7b26b7a700 1 -- 172.21.2.112:0/256453438 shutdown_connections 2021-02-10T11:35:59.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:35:59.683 7f7b26b7a700 1 -- 172.21.2.112:0/256453438 wait complete. 2021-02-10T11:35:59.679 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2021-02-10T11:36:00.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.104 7f823b335700 1 Processor -- start 2021-02-10T11:36:00.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.104 7f823b335700 1 -- start start 2021-02-10T11:36:00.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.105 7f823b335700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f82340d1880 0x7f82341b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:00.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.105 7f823b335700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f82340cff80 0x7f8234109140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:00.109 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.105 7f823b335700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8234109a70 con 0x7f82341b0af0 2021-02-10T11:36:00.109 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.105 7f823b335700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8234109760 con 0x7f82340d1880 2021-02-10T11:36:00.109 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.105 7f823b335700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8234088070 con 0x7f82340cff80 2021-02-10T11:36:00.109 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.105 7f8239151700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f82340d1880 0x7f82341b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:00.109 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.105 7f8239151700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f82340d1880 0x7f82341b04d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:51654/0 (socket says 172.21.2.112:51654) 2021-02-10T11:36:00.109 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.105 7f8239151700 1 -- 172.21.2.112:0/2837539088 learned_addr learned my addr 172.21.2.112:0/2837539088 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:36:00.109 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.105 7f8239151700 1 -- 172.21.2.112:0/2837539088 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f82340cff80 msgr2=0x7f8234109140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:00.109 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.105 7f8239151700 1 --2- 172.21.2.112:0/2837539088 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f82340cff80 0x7f8234109140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:00.109 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.105 7f8239151700 1 -- 172.21.2.112:0/2837539088 >> v1:172.21.2.120:6789/0 conn(0x7f82341b0af0 legacy=0x7f8234108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:00.109 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.105 7f8239151700 1 -- 172.21.2.112:0/2837539088 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82340d1c80 con 0x7f82340d1880 2021-02-10T11:36:00.109 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.105 7f8239151700 1 --2- 172.21.2.112:0/2837539088 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f82340d1880 0x7f82341b04d0 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 rx=0x7f8228006700 tx=0x7f8228009b00).ready entity=mon.0 client_cookie=68d60e22d71d7140 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:00.109 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.106 7f8227fff700 1 -- 172.21.2.112:0/2837539088 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f822800c0c0 con 0x7f82340d1880 2021-02-10T11:36:00.109 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.106 7f8227fff700 1 -- 172.21.2.112:0/2837539088 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f822800c340 con 0x7f82340d1880 2021-02-10T11:36:00.110 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.106 7f8227fff700 1 -- 172.21.2.112:0/2837539088 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f822800c820 con 0x7f82340d1880 2021-02-10T11:36:00.110 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.106 7f823b335700 1 -- 172.21.2.112:0/2837539088 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f82340d1880 msgr2=0x7f82341b04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:00.110 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.106 7f823b335700 1 --2- 172.21.2.112:0/2837539088 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f82340d1880 0x7f82341b04d0 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 rx=0x7f8228006700 tx=0x7f8228009b00).stop 2021-02-10T11:36:00.110 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.106 7f823b335700 1 -- 172.21.2.112:0/2837539088 shutdown_connections 2021-02-10T11:36:00.110 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.106 7f823b335700 1 --2- 172.21.2.112:0/2837539088 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f82340cff80 0x7f8234109140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:00.110 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.106 7f823b335700 1 --2- 172.21.2.112:0/2837539088 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f82340d1880 0x7f82341b04d0 unknown :-1 s=CLOSED pgs=1552 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:00.110 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.106 7f823b335700 1 -- 172.21.2.112:0/2837539088 >> 172.21.2.112:0/2837539088 conn(0x7f82340ce430 msgr2=0x7f82340d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:00.110 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.107 7f823b335700 1 -- 172.21.2.112:0/2837539088 shutdown_connections 2021-02-10T11:36:00.110 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.107 7f823b335700 1 -- 172.21.2.112:0/2837539088 wait complete. 2021-02-10T11:36:00.112 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.121 7f823b335700 1 Processor -- start 2021-02-10T11:36:00.112 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.121 7f823b335700 1 -- start start 2021-02-10T11:36:00.112 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.121 7f823b335700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f823419e5b0 0x7f82341afc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:00.112 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.121 7f823b335700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f823419e9b0 0x7f82341b0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:00.112 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.121 7f823b335700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f82341b0250 0x7f82341b13e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:00.112 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.122 7f823b335700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8234195bd0 con 0x7f823419e9b0 2021-02-10T11:36:00.112 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.122 7f823b335700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82341b0840 con 0x7f823419e5b0 2021-02-10T11:36:00.113 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.122 7f823b335700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f823419eed0 con 0x7f82341b0250 2021-02-10T11:36:00.113 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.122 7f8238950700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f823419e9b0 0x7f82341b0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:00.113 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.122 7f8238950700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f823419e9b0 0x7f82341b0dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:51658/0 (socket says 172.21.2.112:51658) 2021-02-10T11:36:00.113 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.122 7f8238950700 1 -- 172.21.2.112:0/2367314074 learned_addr learned my addr 172.21.2.112:0/2367314074 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:36:00.113 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.122 7f8238950700 1 -- 172.21.2.112:0/2367314074 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f82341b0250 msgr2=0x7f82341b13e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:00.113 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.122 7f8238950700 1 --2- 172.21.2.112:0/2367314074 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f82341b0250 0x7f82341b13e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:00.113 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.122 7f8238950700 1 -- 172.21.2.112:0/2367314074 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f823419e5b0 msgr2=0x7f82341afc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:00.114 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.122 7f8238950700 1 --2- 172.21.2.112:0/2367314074 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f823419e5b0 0x7f82341afc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:00.114 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.122 7f8238950700 1 -- 172.21.2.112:0/2367314074 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f823419f0b0 con 0x7f823419e9b0 2021-02-10T11:36:00.114 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.122 7f8238950700 1 --2- 172.21.2.112:0/2367314074 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f823419e9b0 0x7f82341b0dc0 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 rx=0x7f823000b5b0 tx=0x7f823000b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:00.114 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.122 7f8225ffb700 1 -- 172.21.2.112:0/2367314074 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f823000bd40 con 0x7f823419e9b0 2021-02-10T11:36:00.114 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.123 7f8225ffb700 1 -- 172.21.2.112:0/2367314074 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8230000e20 con 0x7f823419e9b0 2021-02-10T11:36:00.114 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.123 7f823b335700 1 -- 172.21.2.112:0/2367314074 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f823419f0b0 con 0x7f823419e9b0 2021-02-10T11:36:00.114 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.123 7f8225ffb700 1 -- 172.21.2.112:0/2367314074 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8230006210 con 0x7f823419e9b0 2021-02-10T11:36:00.114 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.123 7f823b335700 1 -- 172.21.2.112:0/2367314074 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f823419f0b0 con 0x7f823419e9b0 2021-02-10T11:36:00.114 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.123 7f8225ffb700 1 -- 172.21.2.112:0/2367314074 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8230008280 con 0x7f823419e9b0 2021-02-10T11:36:00.114 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.123 7f8225ffb700 1 --2- 172.21.2.112:0/2367314074 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f82140186c0 0x7f821401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:00.115 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.123 7f8225ffb700 1 -- 172.21.2.112:0/2367314074 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5457+0+0 (secure 0 0 0) 0x7f8230032330 con 0x7f823419e9b0 2021-02-10T11:36:00.115 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.124 7f8239151700 1 --2- 172.21.2.112:0/2367314074 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f82140186c0 0x7f821401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:00.115 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.124 7f8239151700 1 --2- 172.21.2.112:0/2367314074 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f82140186c0 0x7f821401abc0 crc :-1 s=READY pgs=750 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:00.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.155 7f823b335700 1 -- 172.21.2.112:0/2367314074 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f823419f0b0 con 0x7f823419e9b0 2021-02-10T11:36:00.145 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.158 7f8225ffb700 1 -- 172.21.2.112:0/2367314074 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f823002bff0 con 0x7f823419e9b0 2021-02-10T11:36:00.274 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.286 7f823b335700 1 -- 172.21.2.112:0/2367314074 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"} v 0) v1 -- 0x7f823419f0b0 con 0x7f823419e9b0 2021-02-10T11:36:00.277 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.290 7f8225ffb700 1 -- 172.21.2.112:0/2367314074 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"}]=0 new fs with metadata pool 14 and data pool 15 v53) v1 ==== 189+0+0 (secure 0 0 0) 0x7f8230008280 con 0x7f823419e9b0 2021-02-10T11:36:00.281 INFO:teuthology.orchestra.run.gibba012.stderr:new fs with metadata pool 14 and data pool 15 2021-02-10T11:36:00.284 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.296 7f823b335700 1 -- 172.21.2.112:0/2367314074 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f82140186c0 msgr2=0x7f821401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:00.284 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.296 7f823b335700 1 --2- 172.21.2.112:0/2367314074 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f82140186c0 0x7f821401abc0 crc :-1 s=READY pgs=750 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:00.284 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.296 7f823b335700 1 -- 172.21.2.112:0/2367314074 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f823419e9b0 msgr2=0x7f82341b0dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:00.284 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.296 7f823b335700 1 --2- 172.21.2.112:0/2367314074 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f823419e9b0 0x7f82341b0dc0 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 rx=0x7f823000b5b0 tx=0x7f823000b950).stop 2021-02-10T11:36:00.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.297 7f823b335700 1 -- 172.21.2.112:0/2367314074 shutdown_connections 2021-02-10T11:36:00.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.297 7f823b335700 1 --2- 172.21.2.112:0/2367314074 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f82140186c0 0x7f821401abc0 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:00.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.297 7f823b335700 1 --2- 172.21.2.112:0/2367314074 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f82341b0250 0x7f82341b13e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:00.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.297 7f823b335700 1 --2- 172.21.2.112:0/2367314074 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f823419e9b0 0x7f82341b0dc0 unknown :-1 s=CLOSED pgs=1553 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:00.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.297 7f823b335700 1 --2- 172.21.2.112:0/2367314074 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f823419e5b0 0x7f82341afc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:00.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.297 7f823b335700 1 -- 172.21.2.112:0/2367314074 >> 172.21.2.112:0/2367314074 conn(0x7f823419f940 msgr2=0x7f823418c290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:00.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.297 7f823b335700 1 -- 172.21.2.112:0/2367314074 shutdown_connections 2021-02-10T11:36:00.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.297 7f823b335700 1 -- 172.21.2.112:0/2367314074 wait complete. 2021-02-10T11:36:00.292 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:36:00.717 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.714 7fecf2362700 1 Processor -- start 2021-02-10T11:36:00.717 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.714 7fecf2362700 1 -- start start 2021-02-10T11:36:00.717 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.714 7fecf2362700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fecec0d0340 0x7fecec0ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:00.718 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.714 7fecf2362700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fecec0d1c70 0x7fecec0efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:00.718 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.714 7fecf2362700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecec0f0b10 con 0x7fecec0d1870 2021-02-10T11:36:00.718 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.714 7fecf2362700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fecec0e5ed0 con 0x7fecec0d0340 2021-02-10T11:36:00.718 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.714 7fecf2362700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecec1b0be0 con 0x7fecec0d1c70 2021-02-10T11:36:00.718 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.714 7feceb7fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fecec0d0340 0x7fecec0ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:00.718 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.714 7feceb7fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fecec0d0340 0x7fecec0ef640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:51668/0 (socket says 172.21.2.112:51668) 2021-02-10T11:36:00.718 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.714 7feceb7fe700 1 -- 172.21.2.112:0/302387520 learned_addr learned my addr 172.21.2.112:0/302387520 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:36:00.718 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.715 7feceb7fe700 1 -- 172.21.2.112:0/302387520 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fecec0d1c70 msgr2=0x7fecec0efc60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:36:00.718 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.715 7feceb7fe700 1 --2- 172.21.2.112:0/302387520 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fecec0d1c70 0x7fecec0efc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:00.718 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.715 7feceb7fe700 1 -- 172.21.2.112:0/302387520 >> v1:172.21.2.120:6789/0 conn(0x7fecec0d1870 legacy=0x7fecec1b03f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:36:00.718 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.715 7feceb7fe700 1 -- 172.21.2.112:0/302387520 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecec0f20a0 con 0x7fecec0d0340 2021-02-10T11:36:00.718 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.715 7feceb7fe700 1 --2- 172.21.2.112:0/302387520 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fecec0d0340 0x7fecec0ef640 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 rx=0x7fece0006700 tx=0x7fece0009b80).ready entity=mon.0 client_cookie=e63a802ffc855c2b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:00.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.715 7feceaffd700 1 -- 172.21.2.112:0/302387520 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fece000c0c0 con 0x7fecec0d0340 2021-02-10T11:36:00.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.715 7feceaffd700 1 -- 172.21.2.112:0/302387520 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fece000c400 con 0x7fecec0d0340 2021-02-10T11:36:00.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.715 7feceaffd700 1 -- 172.21.2.112:0/302387520 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fece000c920 con 0x7fecec0d0340 2021-02-10T11:36:00.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.715 7fecf2362700 1 -- 172.21.2.112:0/302387520 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fecec0d0340 msgr2=0x7fecec0ef640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:00.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.715 7fecf2362700 1 --2- 172.21.2.112:0/302387520 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fecec0d0340 0x7fecec0ef640 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 rx=0x7fece0006700 tx=0x7fece0009b80).stop 2021-02-10T11:36:00.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.716 7fecf2362700 1 -- 172.21.2.112:0/302387520 shutdown_connections 2021-02-10T11:36:00.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.716 7fecf2362700 1 --2- 172.21.2.112:0/302387520 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fecec0d0340 0x7fecec0ef640 unknown :-1 s=CLOSED pgs=1554 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:00.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.716 7fecf2362700 1 --2- 172.21.2.112:0/302387520 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fecec0d1c70 0x7fecec0efc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:00.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.716 7fecf2362700 1 -- 172.21.2.112:0/302387520 >> 172.21.2.112:0/302387520 conn(0x7fecec0ce450 msgr2=0x7fecec0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:00.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.716 7fecf2362700 1 -- 172.21.2.112:0/302387520 shutdown_connections 2021-02-10T11:36:00.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.716 7fecf2362700 1 -- 172.21.2.112:0/302387520 wait complete. 2021-02-10T11:36:00.721 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.730 7fecf2362700 1 Processor -- start 2021-02-10T11:36:00.721 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.730 7fecf2362700 1 -- start start 2021-02-10T11:36:00.721 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.731 7fecf2362700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fecec0b4f30 0x7fecec18b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:00.721 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.731 7fecf2362700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fecec093f40 0x7fecec094340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:00.721 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.731 7fecf2362700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fecec094960 0x7fecec187e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:00.721 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.731 7fecf2362700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecec094e90 con 0x7fecec0b4f30 2021-02-10T11:36:00.721 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.731 7fecf2362700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecec18bff0 con 0x7fecec093f40 2021-02-10T11:36:00.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.731 7fecf2362700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecec08f780 con 0x7fecec094960 2021-02-10T11:36:00.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.731 7fecebfff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fecec0b4f30 0x7fecec18b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:00.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.731 7fecebfff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fecec0b4f30 0x7fecec18b9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:51674/0 (socket says 172.21.2.112:51674) 2021-02-10T11:36:00.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.731 7fecebfff700 1 -- 172.21.2.112:0/1431960379 learned_addr learned my addr 172.21.2.112:0/1431960379 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:36:00.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.731 7feceb7fe700 1 --2- 172.21.2.112:0/1431960379 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fecec093f40 0x7fecec094340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:00.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.731 7fecf097f700 1 --2- 172.21.2.112:0/1431960379 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fecec094960 0x7fecec187e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:00.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.731 7fecebfff700 1 -- 172.21.2.112:0/1431960379 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fecec094960 msgr2=0x7fecec187e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:00.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.731 7fecebfff700 1 --2- 172.21.2.112:0/1431960379 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fecec094960 0x7fecec187e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:00.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.731 7fecebfff700 1 -- 172.21.2.112:0/1431960379 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fecec093f40 msgr2=0x7fecec094340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:00.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.731 7fecebfff700 1 --2- 172.21.2.112:0/1431960379 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fecec093f40 0x7fecec094340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:00.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.731 7fecebfff700 1 -- 172.21.2.112:0/1431960379 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecec188470 con 0x7fecec0b4f30 2021-02-10T11:36:00.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.732 7fecebfff700 1 --2- 172.21.2.112:0/1431960379 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fecec0b4f30 0x7fecec18b9d0 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 rx=0x7fecdc00b5b0 tx=0x7fecdc00b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:00.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.732 7fece8ff9700 1 -- 172.21.2.112:0/1431960379 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fecdc00bd40 con 0x7fecec0b4f30 2021-02-10T11:36:00.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.732 7fece8ff9700 1 -- 172.21.2.112:0/1431960379 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecdc006020 con 0x7fecec0b4f30 2021-02-10T11:36:00.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.732 7fece8ff9700 1 -- 172.21.2.112:0/1431960379 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fecdc0054a0 con 0x7fecec0b4f30 2021-02-10T11:36:00.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.732 7fecf2362700 1 -- 172.21.2.112:0/1431960379 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecec188470 con 0x7fecec0b4f30 2021-02-10T11:36:00.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.732 7fecf2362700 1 -- 172.21.2.112:0/1431960379 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecec188470 con 0x7fecec0b4f30 2021-02-10T11:36:00.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.732 7fece8ff9700 1 -- 172.21.2.112:0/1431960379 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fecdc006200 con 0x7fecec0b4f30 2021-02-10T11:36:00.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.732 7fece8ff9700 1 --2- 172.21.2.112:0/1431960379 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7feccc0186c0 0x7feccc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:00.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.732 7fece8ff9700 1 -- 172.21.2.112:0/1431960379 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fecdc0327d0 con 0x7fecec0b4f30 2021-02-10T11:36:00.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.733 7feceb7fe700 1 --2- 172.21.2.112:0/1431960379 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7feccc0186c0 0x7feccc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:00.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.733 7feceb7fe700 1 --2- 172.21.2.112:0/1431960379 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7feccc0186c0 0x7feccc01abc0 crc :-1 s=READY pgs=751 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:00.751 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.764 7fecf2362700 1 -- 172.21.2.112:0/1431960379 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fecec0ce450 con 0x7fecec0b4f30 2021-02-10T11:36:00.754 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.767 7fece8ff9700 1 -- 172.21.2.112:0/1431960379 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fecdc006230 con 0x7fecec0b4f30 2021-02-10T11:36:00.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.889 7fecf2362700 1 -- 172.21.2.112:0/1431960379 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fecec0ce450 con 0x7fecec0b4f30 2021-02-10T11:36:00.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.890 7fece8ff9700 1 -- 172.21.2.112:0/1431960379 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v72) v1 ==== 74+0+14009 (secure 0 0 0) 0x7fecdc006020 con 0x7fecec0b4f30 2021-02-10T11:36:00.878 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:36:00.878 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":72,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:36:00.289002","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:35:58.263396","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":15,"pool_name":"cephfs_data","create_time":"2021-02-10 11:35:59.285157","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:36:00.879 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.892 7fecf2362700 1 -- 172.21.2.112:0/1431960379 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7feccc0186c0 msgr2=0x7feccc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:00.879 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.892 7fecf2362700 1 --2- 172.21.2.112:0/1431960379 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7feccc0186c0 0x7feccc01abc0 crc :-1 s=READY pgs=751 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:00.879 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.892 7fecf2362700 1 -- 172.21.2.112:0/1431960379 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fecec0b4f30 msgr2=0x7fecec18b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:00.879 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.892 7fecf2362700 1 --2- 172.21.2.112:0/1431960379 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fecec0b4f30 0x7fecec18b9d0 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 rx=0x7fecdc00b5b0 tx=0x7fecdc00b950).stop 2021-02-10T11:36:00.879 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.892 7fecf2362700 1 -- 172.21.2.112:0/1431960379 shutdown_connections 2021-02-10T11:36:00.880 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.892 7fecf2362700 1 --2- 172.21.2.112:0/1431960379 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7feccc0186c0 0x7feccc01abc0 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:00.880 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.892 7fecf2362700 1 --2- 172.21.2.112:0/1431960379 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fecec093f40 0x7fecec094340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:00.880 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.892 7fecf2362700 1 --2- 172.21.2.112:0/1431960379 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fecec094960 0x7fecec187e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:00.880 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.892 7fecf2362700 1 --2- 172.21.2.112:0/1431960379 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fecec0b4f30 0x7fecec18b9d0 unknown :-1 s=CLOSED pgs=1555 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:00.880 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.892 7fecf2362700 1 -- 172.21.2.112:0/1431960379 >> 172.21.2.112:0/1431960379 conn(0x7fecec19f8f0 msgr2=0x7fecec1a5f80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:00.880 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.892 7fecf2362700 1 -- 172.21.2.112:0/1431960379 shutdown_connections 2021-02-10T11:36:00.880 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:00.892 7fecf2362700 1 -- 172.21.2.112:0/1431960379 wait complete. 2021-02-10T11:36:00.888 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:36:01.314 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.309 7f1ae01ed700 1 Processor -- start 2021-02-10T11:36:01.314 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.309 7f1ae01ed700 1 -- start start 2021-02-10T11:36:01.314 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.310 7f1ae01ed700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1ad80d1870 0x7f1ad81b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:01.314 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.310 7f1ae01ed700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f1ad81b0a10 0x7f1ad80cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:01.314 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.310 7f1ae01ed700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ad80f0b10 con 0x7f1ad80d1c70 2021-02-10T11:36:01.314 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.310 7f1ae01ed700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ad80fe230 con 0x7f1ad81b0a10 2021-02-10T11:36:01.315 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.310 7f1ae01ed700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ad80e5ed0 con 0x7f1ad80d1870 2021-02-10T11:36:01.315 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.310 7f1add808700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f1ad81b0a10 0x7f1ad80cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:01.315 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.310 7f1add808700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f1ad81b0a10 0x7f1ad80cf330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:51680/0 (socket says 172.21.2.112:51680) 2021-02-10T11:36:01.315 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.310 7f1add808700 1 -- 172.21.2.112:0/719669595 learned_addr learned my addr 172.21.2.112:0/719669595 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:36:01.315 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.310 7f1ade009700 1 --2- 172.21.2.112:0/719669595 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1ad80d1870 0x7f1ad81b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:01.315 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.310 7f1add808700 1 -- 172.21.2.112:0/719669595 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1ad80d1870 msgr2=0x7f1ad81b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:01.315 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.310 7f1add808700 1 --2- 172.21.2.112:0/719669595 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1ad80d1870 0x7f1ad81b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:01.315 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.310 7f1add808700 1 -- 172.21.2.112:0/719669595 >> v1:172.21.2.120:6789/0 conn(0x7f1ad80d1c70 legacy=0x7f1ad80cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:01.315 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.310 7f1add808700 1 -- 172.21.2.112:0/719669595 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ad80f0650 con 0x7f1ad81b0a10 2021-02-10T11:36:01.315 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.310 7f1add808700 1 --2- 172.21.2.112:0/719669595 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1ad81b0a10 0x7f1ad80cf330 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 rx=0x7f1ac8006700 tx=0x7f1ac8009b80).ready entity=mon.0 client_cookie=b97d15ef2064b130 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:01.315 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.311 7f1add007700 1 -- 172.21.2.112:0/719669595 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1ac800c0c0 con 0x7f1ad81b0a10 2021-02-10T11:36:01.316 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.311 7f1add007700 1 -- 172.21.2.112:0/719669595 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ac800c400 con 0x7f1ad81b0a10 2021-02-10T11:36:01.316 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.311 7f1add007700 1 -- 172.21.2.112:0/719669595 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1ac800c920 con 0x7f1ad81b0a10 2021-02-10T11:36:01.316 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.311 7f1ae01ed700 1 -- 172.21.2.112:0/719669595 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1ad81b0a10 msgr2=0x7f1ad80cf330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:01.316 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.311 7f1ae01ed700 1 --2- 172.21.2.112:0/719669595 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1ad81b0a10 0x7f1ad80cf330 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 rx=0x7f1ac8006700 tx=0x7f1ac8009b80).stop 2021-02-10T11:36:01.316 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.312 7f1ae01ed700 1 -- 172.21.2.112:0/719669595 shutdown_connections 2021-02-10T11:36:01.316 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.312 7f1ae01ed700 1 --2- 172.21.2.112:0/719669595 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1ad81b0a10 0x7f1ad80cf330 unknown :-1 s=CLOSED pgs=1556 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:01.316 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.312 7f1ae01ed700 1 --2- 172.21.2.112:0/719669595 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1ad80d1870 0x7f1ad81b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:01.316 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.312 7f1ae01ed700 1 -- 172.21.2.112:0/719669595 >> 172.21.2.112:0/719669595 conn(0x7f1ad80ce450 msgr2=0x7f1ad80d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:01.316 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.312 7f1ae01ed700 1 -- 172.21.2.112:0/719669595 shutdown_connections 2021-02-10T11:36:01.316 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.312 7f1ae01ed700 1 -- 172.21.2.112:0/719669595 wait complete. 2021-02-10T11:36:01.316 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.326 7f1ae01ed700 1 Processor -- start 2021-02-10T11:36:01.316 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.326 7f1ae01ed700 1 -- start start 2021-02-10T11:36:01.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.327 7f1ae01ed700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1ad80f04d0 0x7f1ad819c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:01.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.327 7f1ae01ed700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1ad819cfa0 0x7f1ad80b1a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:01.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.327 7f1ae01ed700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1ad819d3a0 0x7f1ad80b2060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:01.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.327 7f1ae01ed700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ad80b2680 con 0x7f1ad80f04d0 2021-02-10T11:36:01.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.327 7f1ae01ed700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ad80b28a0 con 0x7f1ad819d3a0 2021-02-10T11:36:01.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.327 7f1ae01ed700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ad818b130 con 0x7f1ad819cfa0 2021-02-10T11:36:01.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.327 7f1ade009700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1ad80f04d0 0x7f1ad819c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:01.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.327 7f1ade009700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1ad80f04d0 0x7f1ad819c980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:51686/0 (socket says 172.21.2.112:51686) 2021-02-10T11:36:01.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.327 7f1ade009700 1 -- 172.21.2.112:0/1958827636 learned_addr learned my addr 172.21.2.112:0/1958827636 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:36:01.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.327 7f1add808700 1 --2- 172.21.2.112:0/1958827636 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1ad819cfa0 0x7f1ad80b1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:01.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.327 7f1ade80a700 1 --2- 172.21.2.112:0/1958827636 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1ad819d3a0 0x7f1ad80b2060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:01.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.327 7f1add808700 1 -- 172.21.2.112:0/1958827636 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1ad819d3a0 msgr2=0x7f1ad80b2060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:01.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.327 7f1add808700 1 --2- 172.21.2.112:0/1958827636 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1ad819d3a0 0x7f1ad80b2060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:01.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.327 7f1add808700 1 -- 172.21.2.112:0/1958827636 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1ad80f04d0 msgr2=0x7f1ad819c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:01.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.327 7f1add808700 1 --2- 172.21.2.112:0/1958827636 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1ad80f04d0 0x7f1ad819c980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:01.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.327 7f1add808700 1 -- 172.21.2.112:0/1958827636 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ad818b350 con 0x7f1ad819cfa0 2021-02-10T11:36:01.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.328 7f1add808700 1 --2- 172.21.2.112:0/1958827636 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1ad819cfa0 0x7f1ad80b1a40 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f1ac800b530 tx=0x7f1ac800b8d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:01.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.328 7f1aceffd700 1 -- 172.21.2.112:0/1958827636 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1ac8009440 con 0x7f1ad819cfa0 2021-02-10T11:36:01.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.328 7f1ae01ed700 1 -- 172.21.2.112:0/1958827636 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ad818b350 con 0x7f1ad819cfa0 2021-02-10T11:36:01.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.328 7f1ae01ed700 1 -- 172.21.2.112:0/1958827636 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ad818b350 con 0x7f1ad819cfa0 2021-02-10T11:36:01.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.328 7f1aceffd700 1 -- 172.21.2.112:0/1958827636 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ac8009440 con 0x7f1ad819cfa0 2021-02-10T11:36:01.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.328 7f1aceffd700 1 -- 172.21.2.112:0/1958827636 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1ac80052e0 con 0x7f1ad819cfa0 2021-02-10T11:36:01.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.328 7f1aceffd700 1 -- 172.21.2.112:0/1958827636 <== mon.2 v2:172.21.2.120:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1ac80052e0 con 0x7f1ad819cfa0 2021-02-10T11:36:01.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.328 7f1aceffd700 1 --2- 172.21.2.112:0/1958827636 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1ac0018730 0x7f1ac001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:01.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.328 7f1aceffd700 1 -- 172.21.2.112:0/1958827636 <== mon.2 v2:172.21.2.120:3301/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f1ac8032730 con 0x7f1ad819cfa0 2021-02-10T11:36:01.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.328 7f1ade009700 1 --2- 172.21.2.112:0/1958827636 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1ac0018730 0x7f1ac001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:01.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.329 7f1ade009700 1 --2- 172.21.2.112:0/1958827636 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1ac0018730 0x7f1ac001ac10 crc :-1 s=READY pgs=752 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:01.347 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.359 7f1ae01ed700 1 -- 172.21.2.112:0/1958827636 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ad80b2680 con 0x7f1ad819cfa0 2021-02-10T11:36:01.350 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.363 7f1aceffd700 1 -- 172.21.2.112:0/1958827636 <== mon.2 v2:172.21.2.120:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1ac8006030 con 0x7f1ad819cfa0 2021-02-10T11:36:01.481 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.494 7f1ae01ed700 1 -- 172.21.2.112:0/1958827636 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1ad80b2680 con 0x7f1ad819cfa0 2021-02-10T11:36:01.482 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.495 7f1aceffd700 1 -- 172.21.2.112:0/1958827636 <== mon.2 v2:172.21.2.120:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v73) v1 ==== 74+0+13996 (secure 0 0 0) 0x7f1ac8009440 con 0x7f1ad819cfa0 2021-02-10T11:36:01.484 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:36:01.485 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":73,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:36:01.291077","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:35:58.263396","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":15,"pool_name":"cephfs_data","create_time":"2021-02-10 11:35:59.285157","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:36:01.486 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.498 7f1ae01ed700 1 -- 172.21.2.112:0/1958827636 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1ac0018730 msgr2=0x7f1ac001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:01.486 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.498 7f1ae01ed700 1 --2- 172.21.2.112:0/1958827636 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1ac0018730 0x7f1ac001ac10 crc :-1 s=READY pgs=752 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:01.487 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.498 7f1ae01ed700 1 -- 172.21.2.112:0/1958827636 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1ad819cfa0 msgr2=0x7f1ad80b1a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:01.487 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.498 7f1ae01ed700 1 --2- 172.21.2.112:0/1958827636 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1ad819cfa0 0x7f1ad80b1a40 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f1ac800b530 tx=0x7f1ac800b8d0).stop 2021-02-10T11:36:01.487 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.499 7f1ae01ed700 1 -- 172.21.2.112:0/1958827636 shutdown_connections 2021-02-10T11:36:01.487 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.499 7f1ae01ed700 1 --2- 172.21.2.112:0/1958827636 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1ac0018730 0x7f1ac001ac10 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:01.487 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.499 7f1ae01ed700 1 --2- 172.21.2.112:0/1958827636 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1ad819d3a0 0x7f1ad80b2060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:01.487 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.499 7f1ae01ed700 1 --2- 172.21.2.112:0/1958827636 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1ad819cfa0 0x7f1ad80b1a40 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:01.487 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.499 7f1ae01ed700 1 --2- 172.21.2.112:0/1958827636 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1ad80f04d0 0x7f1ad819c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:01.487 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.499 7f1ae01ed700 1 -- 172.21.2.112:0/1958827636 >> 172.21.2.112:0/1958827636 conn(0x7f1ad819c560 msgr2=0x7f1ad819fbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:01.487 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.499 7f1ae01ed700 1 -- 172.21.2.112:0/1958827636 shutdown_connections 2021-02-10T11:36:01.487 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.499 7f1ae01ed700 1 -- 172.21.2.112:0/1958827636 wait complete. 2021-02-10T11:36:01.495 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2021-02-10T11:36:01.928 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.925 7f4a35bda700 1 Processor -- start 2021-02-10T11:36:01.929 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.925 7f4a35bda700 1 -- start start 2021-02-10T11:36:01.929 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.925 7f4a35bda700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4a300d1870 0x7f4a301b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:01.929 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.925 7f4a35bda700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f4a301b0ae0 0x7f4a30108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:01.929 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.925 7f4a35bda700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f4a300d1c70 0x7f4a30108f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:01.929 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.925 7f4a35bda700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a300e5ef0 con 0x7f4a301b0ae0 2021-02-10T11:36:01.929 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.925 7f4a35bda700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a30109a60 con 0x7f4a300d1c70 2021-02-10T11:36:01.929 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.925 7f4a35bda700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a30088070 con 0x7f4a300d1870 2021-02-10T11:36:01.929 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.925 7f4a2effd700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f4a301b0ae0 0x7f4a30108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:01.929 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.925 7f4a2effd700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f4a301b0ae0 0x7f4a30108940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:51692/0 (socket says 172.21.2.112:51692) 2021-02-10T11:36:01.929 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.925 7f4a2effd700 1 -- 172.21.2.112:0/2291309739 learned_addr learned my addr 172.21.2.112:0/2291309739 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:36:01.929 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.926 7f4a2effd700 1 -- 172.21.2.112:0/2291309739 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4a300d1870 msgr2=0x7f4a301b04c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:36:01.929 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.926 7f4a2effd700 1 --2- 172.21.2.112:0/2291309739 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4a300d1870 0x7f4a301b04c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:01.930 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.926 7f4a2effd700 1 -- 172.21.2.112:0/2291309739 >> v2:172.21.2.120:3300/0 conn(0x7f4a300d1c70 msgr2=0x7f4a30108f60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:36:01.930 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.926 7f4a2effd700 1 --2- 172.21.2.112:0/2291309739 >> v2:172.21.2.120:3300/0 conn(0x7f4a300d1c70 0x7f4a30108f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:01.930 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.926 7f4a2effd700 1 -- 172.21.2.112:0/2291309739 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a30109580 con 0x7f4a301b0ae0 2021-02-10T11:36:01.930 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.926 7f4a2effd700 1 --2- 172.21.2.112:0/2291309739 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4a301b0ae0 0x7f4a30108940 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 rx=0x7f4a20006700 tx=0x7f4a20009b00).ready entity=mon.0 client_cookie=8961b0d033b9e496 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:01.930 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.926 7f4a2e7fc700 1 -- 172.21.2.112:0/2291309739 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4a2000c0c0 con 0x7f4a301b0ae0 2021-02-10T11:36:01.930 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.926 7f4a2e7fc700 1 -- 172.21.2.112:0/2291309739 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a2000c340 con 0x7f4a301b0ae0 2021-02-10T11:36:01.930 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.926 7f4a2e7fc700 1 -- 172.21.2.112:0/2291309739 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4a2000c7a0 con 0x7f4a301b0ae0 2021-02-10T11:36:01.930 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.926 7f4a35bda700 1 -- 172.21.2.112:0/2291309739 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4a301b0ae0 msgr2=0x7f4a30108940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:01.930 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.926 7f4a35bda700 1 --2- 172.21.2.112:0/2291309739 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4a301b0ae0 0x7f4a30108940 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 rx=0x7f4a20006700 tx=0x7f4a20009b00).stop 2021-02-10T11:36:01.930 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.927 7f4a35bda700 1 -- 172.21.2.112:0/2291309739 shutdown_connections 2021-02-10T11:36:01.930 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.927 7f4a35bda700 1 --2- 172.21.2.112:0/2291309739 >> v2:172.21.2.120:3300/0 conn(0x7f4a300d1c70 0x7f4a30108f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:01.930 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.927 7f4a35bda700 1 --2- 172.21.2.112:0/2291309739 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4a301b0ae0 0x7f4a30108940 unknown :-1 s=CLOSED pgs=1557 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:01.930 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.927 7f4a35bda700 1 --2- 172.21.2.112:0/2291309739 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4a300d1870 0x7f4a301b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:01.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.927 7f4a35bda700 1 -- 172.21.2.112:0/2291309739 >> 172.21.2.112:0/2291309739 conn(0x7f4a300ce420 msgr2=0x7f4a300d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:01.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.927 7f4a35bda700 1 -- 172.21.2.112:0/2291309739 shutdown_connections 2021-02-10T11:36:01.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.927 7f4a35bda700 1 -- 172.21.2.112:0/2291309739 wait complete. 2021-02-10T11:36:01.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.941 7f4a35bda700 1 Processor -- start 2021-02-10T11:36:01.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.941 7f4a35bda700 1 -- start start 2021-02-10T11:36:01.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.942 7f4a35bda700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4a300872c0 0x7f4a301a7580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:01.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.942 7f4a35bda700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4a301a7ba0 0x7f4a300a6860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:01.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.942 7f4a35bda700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4a301a7fa0 0x7f4a300a6e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:01.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.942 7f4a35bda700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a30086e20 con 0x7f4a301a7fa0 2021-02-10T11:36:01.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.942 7f4a35bda700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a30190c60 con 0x7f4a300872c0 2021-02-10T11:36:01.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.942 7f4a35bda700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a300a74a0 con 0x7f4a301a7ba0 2021-02-10T11:36:01.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.942 7f4a2ffff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4a301a7fa0 0x7f4a300a6e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:01.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.942 7f4a2ffff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4a301a7fa0 0x7f4a300a6e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:51694/0 (socket says 172.21.2.112:51694) 2021-02-10T11:36:01.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.942 7f4a2ffff700 1 -- 172.21.2.112:0/2084040677 learned_addr learned my addr 172.21.2.112:0/2084040677 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:36:01.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.942 7f4a2ffff700 1 -- 172.21.2.112:0/2084040677 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4a301a7ba0 msgr2=0x7f4a300a6860 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:36:01.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.942 7f4a2ffff700 1 --2- 172.21.2.112:0/2084040677 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4a301a7ba0 0x7f4a300a6860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:01.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.942 7f4a2ffff700 1 -- 172.21.2.112:0/2084040677 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4a300872c0 msgr2=0x7f4a301a7580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:36:01.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.942 7f4a2ffff700 1 --2- 172.21.2.112:0/2084040677 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4a300872c0 0x7f4a301a7580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:01.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.942 7f4a2ffff700 1 -- 172.21.2.112:0/2084040677 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a300a76c0 con 0x7f4a301a7fa0 2021-02-10T11:36:01.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.942 7f4a2ffff700 1 --2- 172.21.2.112:0/2084040677 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4a301a7fa0 0x7f4a300a6e80 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 rx=0x7f4a28009960 tx=0x7f4a2800bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:01.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.943 7f4a27fff700 1 -- 172.21.2.112:0/2084040677 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4a28000d10 con 0x7f4a301a7fa0 2021-02-10T11:36:01.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.943 7f4a27fff700 1 -- 172.21.2.112:0/2084040677 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a28003020 con 0x7f4a301a7fa0 2021-02-10T11:36:01.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.943 7f4a27fff700 1 -- 172.21.2.112:0/2084040677 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4a28005ad0 con 0x7f4a301a7fa0 2021-02-10T11:36:01.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.943 7f4a35bda700 1 -- 172.21.2.112:0/2084040677 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a300a76c0 con 0x7f4a301a7fa0 2021-02-10T11:36:01.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.943 7f4a35bda700 1 -- 172.21.2.112:0/2084040677 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a300a76c0 con 0x7f4a301a7fa0 2021-02-10T11:36:01.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.943 7f4a27fff700 1 -- 172.21.2.112:0/2084040677 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4a28000d10 con 0x7f4a301a7fa0 2021-02-10T11:36:01.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.943 7f4a27fff700 1 --2- 172.21.2.112:0/2084040677 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4a140186c0 0x7f4a1401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:01.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.943 7f4a27fff700 1 -- 172.21.2.112:0/2084040677 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f4a28032900 con 0x7f4a301a7fa0 2021-02-10T11:36:01.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.944 7f4a2f7fe700 1 --2- 172.21.2.112:0/2084040677 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4a140186c0 0x7f4a1401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:01.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.945 7f4a2f7fe700 1 --2- 172.21.2.112:0/2084040677 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4a140186c0 0x7f4a1401abc0 crc :-1 s=READY pgs=753 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:01.963 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.975 7f4a35bda700 1 -- 172.21.2.112:0/2084040677 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a30086e20 con 0x7f4a301a7fa0 2021-02-10T11:36:01.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:01.978 7f4a27fff700 1 -- 172.21.2.112:0/2084040677 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4a2800be30 con 0x7f4a301a7fa0 2021-02-10T11:36:02.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:02.117 7f4a35bda700 1 -- 172.21.2.112:0/2084040677 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"} v 0) v1 -- 0x7f4a30086e20 con 0x7f4a301a7fa0 2021-02-10T11:36:02.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:02.297 7f4a27fff700 1 -- 172.21.2.112:0/2084040677 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"}]=0 v56) v1 ==== 118+0+0 (secure 0 0 0) 0x7f4a28000d10 con 0x7f4a301a7fa0 2021-02-10T11:36:02.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:02.335 7f4a35bda700 1 -- 172.21.2.112:0/2084040677 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4a140186c0 msgr2=0x7f4a1401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:02.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:02.335 7f4a35bda700 1 --2- 172.21.2.112:0/2084040677 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4a140186c0 0x7f4a1401abc0 crc :-1 s=READY pgs=753 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:02.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:02.335 7f4a35bda700 1 -- 172.21.2.112:0/2084040677 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4a301a7fa0 msgr2=0x7f4a300a6e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:02.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:02.335 7f4a35bda700 1 --2- 172.21.2.112:0/2084040677 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4a301a7fa0 0x7f4a300a6e80 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 rx=0x7f4a28009960 tx=0x7f4a2800bcf0).stop 2021-02-10T11:36:02.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:02.335 7f4a35bda700 1 -- 172.21.2.112:0/2084040677 shutdown_connections 2021-02-10T11:36:02.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:02.335 7f4a35bda700 1 --2- 172.21.2.112:0/2084040677 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4a140186c0 0x7f4a1401abc0 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:02.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:02.335 7f4a35bda700 1 --2- 172.21.2.112:0/2084040677 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4a301a7fa0 0x7f4a300a6e80 unknown :-1 s=CLOSED pgs=1558 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:02.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:02.335 7f4a35bda700 1 --2- 172.21.2.112:0/2084040677 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4a301a7ba0 0x7f4a300a6860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:02.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:02.335 7f4a35bda700 1 --2- 172.21.2.112:0/2084040677 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4a300872c0 0x7f4a301a7580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:02.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:02.335 7f4a35bda700 1 -- 172.21.2.112:0/2084040677 >> 172.21.2.112:0/2084040677 conn(0x7f4a300be980 msgr2=0x7f4a30092cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:02.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:02.336 7f4a35bda700 1 -- 172.21.2.112:0/2084040677 shutdown_connections 2021-02-10T11:36:02.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:02.336 7f4a35bda700 1 -- 172.21.2.112:0/2084040677 wait complete. 2021-02-10T11:36:02.332 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:36:02.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:02.754 7f935596b700 1 Processor -- start 2021-02-10T11:36:02.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:02.754 7f935596b700 1 -- start start 2021-02-10T11:36:02.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:02.754 7f935596b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f93500d1870 0x7f93501b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:02.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:02.754 7f935596b700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f93501b0a10 0x7f93500cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:02.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:02.754 7f935596b700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f93500d1c70 0x7f93500cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:02.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:02.754 7f935596b700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93500fe230 con 0x7f93500d1c70 2021-02-10T11:36:02.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:02.754 7f935596b700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93500e5ed0 con 0x7f93501b0a10 2021-02-10T11:36:02.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:02.754 7f935596b700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93500f0b10 con 0x7f93500d1870 2021-02-10T11:36:02.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:02.755 7f934fd75700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f93500d1c70 0x7f93500cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:02.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:02.755 7f934fd75700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f93500d1c70 0x7f93500cf950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:51702/0 (socket says 172.21.2.112:51702) 2021-02-10T11:36:02.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:02.755 7f934fd75700 1 -- 172.21.2.112:0/3148259605 learned_addr learned my addr 172.21.2.112:0/3148259605 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:36:02.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:02.755 7f934f574700 1 --2- 172.21.2.112:0/3148259605 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f93500d1870 0x7f93501b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:02.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:02.755 7f934ed73700 1 --2- 172.21.2.112:0/3148259605 >> v2:172.21.2.120:3300/0 conn(0x7f93501b0a10 0x7f93500cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:02.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:02.755 7f934fd75700 1 -- 172.21.2.112:0/3148259605 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f93500d1870 msgr2=0x7f93501b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:02.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:02.755 7f934fd75700 1 --2- 172.21.2.112:0/3148259605 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f93500d1870 0x7f93501b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:02.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:02.755 7f934fd75700 1 -- 172.21.2.112:0/3148259605 >> v2:172.21.2.120:3300/0 conn(0x7f93501b0a10 msgr2=0x7f93500cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:02.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:02.755 7f934fd75700 1 --2- 172.21.2.112:0/3148259605 >> v2:172.21.2.120:3300/0 conn(0x7f93501b0a10 0x7f93500cf330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:02.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:02.755 7f934fd75700 1 -- 172.21.2.112:0/3148259605 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93500cff70 con 0x7f93500d1c70 2021-02-10T11:36:02.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:02.755 7f934fd75700 1 --2- 172.21.2.112:0/3148259605 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f93500d1c70 0x7f93500cf950 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 rx=0x7f9348006f00 tx=0x7f9348009f10).ready entity=mon.0 client_cookie=b3dc734dccbd043b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:02.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:02.755 7f934e572700 1 -- 172.21.2.112:0/3148259605 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f934800b3b0 con 0x7f93500d1c70 2021-02-10T11:36:02.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:02.756 7f934e572700 1 -- 172.21.2.112:0/3148259605 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f934800b630 con 0x7f93500d1c70 2021-02-10T11:36:02.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:02.756 7f934e572700 1 -- 172.21.2.112:0/3148259605 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f934800ba90 con 0x7f93500d1c70 2021-02-10T11:36:02.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:02.756 7f935596b700 1 -- 172.21.2.112:0/3148259605 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f93500d1c70 msgr2=0x7f93500cf950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:02.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:02.756 7f935596b700 1 --2- 172.21.2.112:0/3148259605 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f93500d1c70 0x7f93500cf950 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 rx=0x7f9348006f00 tx=0x7f9348009f10).stop 2021-02-10T11:36:02.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:02.756 7f935596b700 1 -- 172.21.2.112:0/3148259605 shutdown_connections 2021-02-10T11:36:02.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:02.756 7f935596b700 1 --2- 172.21.2.112:0/3148259605 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f93500d1c70 0x7f93500cf950 unknown :-1 s=CLOSED pgs=1559 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:02.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:02.756 7f935596b700 1 --2- 172.21.2.112:0/3148259605 >> v2:172.21.2.120:3300/0 conn(0x7f93501b0a10 0x7f93500cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:02.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:02.756 7f935596b700 1 --2- 172.21.2.112:0/3148259605 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f93500d1870 0x7f93501b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:02.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:02.756 7f935596b700 1 -- 172.21.2.112:0/3148259605 >> 172.21.2.112:0/3148259605 conn(0x7f93500ce450 msgr2=0x7f93500d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:02.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:02.756 7f935596b700 1 -- 172.21.2.112:0/3148259605 shutdown_connections 2021-02-10T11:36:02.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:02.756 7f935596b700 1 -- 172.21.2.112:0/3148259605 wait complete. 2021-02-10T11:36:02.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:02.771 7f935596b700 1 Processor -- start 2021-02-10T11:36:02.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:02.771 7f935596b700 1 -- start start 2021-02-10T11:36:02.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:02.771 7f935596b700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9350195650 0x7f93500b5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:02.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:02.771 7f935596b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f93500b5b80 0x7f93501a2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:02.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:02.771 7f935596b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f93500acfa0 0x7f93501a3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:02.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:02.771 7f935596b700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93500b6010 con 0x7f93500acfa0 2021-02-10T11:36:02.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:02.771 7f935596b700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93501a3a60 con 0x7f9350195650 2021-02-10T11:36:02.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:02.771 7f935596b700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93501a3c40 con 0x7f93500b5b80 2021-02-10T11:36:02.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:02.772 7f934fd75700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f93500acfa0 0x7f93501a3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:02.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:02.772 7f934fd75700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f93500acfa0 0x7f93501a3440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:51708/0 (socket says 172.21.2.112:51708) 2021-02-10T11:36:02.763 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:02.772 7f934fd75700 1 -- 172.21.2.112:0/3613799306 learned_addr learned my addr 172.21.2.112:0/3613799306 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:36:02.763 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:02.772 7f934ed73700 1 --2- 172.21.2.112:0/3613799306 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f93500b5b80 0x7f93501a2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:02.763 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:02.772 7f934f574700 1 --2- 172.21.2.112:0/3613799306 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9350195650 0x7f93500b5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:02.763 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:02.772 7f934fd75700 1 -- 172.21.2.112:0/3613799306 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f93500b5b80 msgr2=0x7f93501a2e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:02.763 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:02.772 7f934fd75700 1 --2- 172.21.2.112:0/3613799306 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f93500b5b80 0x7f93501a2e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:02.763 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:02.772 7f934fd75700 1 -- 172.21.2.112:0/3613799306 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9350195650 msgr2=0x7f93500b5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:02.763 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:02.772 7f934fd75700 1 --2- 172.21.2.112:0/3613799306 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9350195650 0x7f93500b5560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:02.763 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:02.772 7f934fd75700 1 -- 172.21.2.112:0/3613799306 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93500bcae0 con 0x7f93500acfa0 2021-02-10T11:36:02.763 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:02.772 7f934fd75700 1 --2- 172.21.2.112:0/3613799306 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f93500acfa0 0x7f93501a3440 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 rx=0x7f9348009390 tx=0x7f934800bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:02.763 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:02.772 7f9347fff700 1 -- 172.21.2.112:0/3613799306 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9348003030 con 0x7f93500acfa0 2021-02-10T11:36:02.763 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:02.772 7f9347fff700 1 -- 172.21.2.112:0/3613799306 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93480068e0 con 0x7f93500acfa0 2021-02-10T11:36:02.763 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:02.772 7f9347fff700 1 -- 172.21.2.112:0/3613799306 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f934800c570 con 0x7f93500acfa0 2021-02-10T11:36:02.764 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:02.772 7f935596b700 1 -- 172.21.2.112:0/3613799306 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93500bcae0 con 0x7f93500acfa0 2021-02-10T11:36:02.764 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:02.773 7f935596b700 1 -- 172.21.2.112:0/3613799306 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93500bd070 con 0x7f93500acfa0 2021-02-10T11:36:02.764 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:02.773 7f9347fff700 1 -- 172.21.2.112:0/3613799306 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9348001270 con 0x7f93500acfa0 2021-02-10T11:36:02.764 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:02.773 7f9347fff700 1 --2- 172.21.2.112:0/3613799306 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f93340186c0 0x7f933401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:02.764 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:02.773 7f9347fff700 1 -- 172.21.2.112:0/3613799306 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f93480328a0 con 0x7f93500acfa0 2021-02-10T11:36:02.764 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:02.773 7f934f574700 1 --2- 172.21.2.112:0/3613799306 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f93340186c0 0x7f933401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:02.764 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:02.774 7f934f574700 1 --2- 172.21.2.112:0/3613799306 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f93340186c0 0x7f933401abc0 crc :-1 s=READY pgs=754 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:02.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:02.804 7f935596b700 1 -- 172.21.2.112:0/3613799306 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93500f0ab0 con 0x7f93500acfa0 2021-02-10T11:36:02.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:02.807 7f9347fff700 1 -- 172.21.2.112:0/3613799306 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9348009430 con 0x7f93500acfa0 2021-02-10T11:36:02.922 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:02.934 7f935596b700 1 -- 172.21.2.112:0/3613799306 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f93500f0ab0 con 0x7f93500acfa0 2021-02-10T11:36:02.922 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:02.935 7f9347fff700 1 -- 172.21.2.112:0/3613799306 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+2101 (secure 0 0 0) 0x7f934801f880 con 0x7f93500acfa0 2021-02-10T11:36:02.923 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:36:02.923 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":56,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8820,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/3613517698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":3613517698},{"type":"v1","addr":"172.21.2.112:6833","nonce":3613517698}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":52}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:36:00.289087","modified":"2021-02-10 11:36:02.295150","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7087},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7087":{"gid":7087,"name":"b","rank":0,"incarnation":53,"state":"up:active","state_seq":3,"addr":"172.21.2.120:6835/1870802251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":1870802251},{"type":"v1","addr":"172.21.2.120:6835","nonce":1870802251}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-02-10T11:36:02.923 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 56 2021-02-10T11:36:02.924 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:02.936 7f935596b700 1 -- 172.21.2.112:0/3613799306 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f93340186c0 msgr2=0x7f933401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:02.924 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:02.936 7f935596b700 1 --2- 172.21.2.112:0/3613799306 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f93340186c0 0x7f933401abc0 crc :-1 s=READY pgs=754 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:02.924 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:02.936 7f935596b700 1 -- 172.21.2.112:0/3613799306 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f93500acfa0 msgr2=0x7f93501a3440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:02.924 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:02.936 7f935596b700 1 --2- 172.21.2.112:0/3613799306 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f93500acfa0 0x7f93501a3440 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 rx=0x7f9348009390 tx=0x7f934800bd40).stop 2021-02-10T11:36:02.924 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:02.937 7f935596b700 1 -- 172.21.2.112:0/3613799306 shutdown_connections 2021-02-10T11:36:02.925 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:02.937 7f935596b700 1 --2- 172.21.2.112:0/3613799306 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f93340186c0 0x7f933401abc0 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:02.925 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:02.937 7f935596b700 1 --2- 172.21.2.112:0/3613799306 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f93500acfa0 0x7f93501a3440 unknown :-1 s=CLOSED pgs=1560 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:02.925 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:02.937 7f935596b700 1 --2- 172.21.2.112:0/3613799306 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f93500b5b80 0x7f93501a2e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:02.925 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:02.937 7f935596b700 1 --2- 172.21.2.112:0/3613799306 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9350195650 0x7f93500b5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:02.925 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:02.937 7f935596b700 1 -- 172.21.2.112:0/3613799306 >> 172.21.2.112:0/3613799306 conn(0x7f93500b3040 msgr2=0x7f93500c3250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:02.925 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:02.937 7f935596b700 1 -- 172.21.2.112:0/3613799306 shutdown_connections 2021-02-10T11:36:02.925 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:02.937 7f935596b700 1 -- 172.21.2.112:0/3613799306 wait complete. 2021-02-10T11:36:02.933 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:36:03.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.357 7f626a1e6700 1 Processor -- start 2021-02-10T11:36:03.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.357 7f626a1e6700 1 -- start start 2021-02-10T11:36:03.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.358 7f626a1e6700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f62640d0340 0x7f62640ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:03.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.358 7f626a1e6700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f62640d1c70 0x7f62640efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:03.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.358 7f626a1e6700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62640f0b10 con 0x7f62640d1870 2021-02-10T11:36:03.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.358 7f626a1e6700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62640e5ed0 con 0x7f62640d0340 2021-02-10T11:36:03.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.358 7f626a1e6700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62641b0be0 con 0x7f62640d1c70 2021-02-10T11:36:03.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.358 7f6263fff700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f62640d1870 0x7f62641b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:51174/0 (socket says 172.21.2.112:51174) 2021-02-10T11:36:03.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.358 7f62637fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f62640d0340 0x7f62640ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:03.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.358 7f6263fff700 1 -- 172.21.2.112:0/426491955 learned_addr learned my addr 172.21.2.112:0/426491955 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:36:03.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.358 7f62637fe700 1 -- 172.21.2.112:0/426491955 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f62640d1c70 msgr2=0x7f62640efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:03.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.358 7f62637fe700 1 --2- 172.21.2.112:0/426491955 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f62640d1c70 0x7f62640efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:03.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.358 7f62637fe700 1 -- 172.21.2.112:0/426491955 >> v1:172.21.2.112:6789/0 conn(0x7f62640d1870 legacy=0x7f62641b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:03.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.358 7f62637fe700 1 -- 172.21.2.112:0/426491955 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62640f20a0 con 0x7f62640d0340 2021-02-10T11:36:03.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.359 7f62637fe700 1 --2- 172.21.2.112:0/426491955 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f62640d0340 0x7f62640ef640 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 rx=0x7f6258006650 tx=0x7f6258009a80).ready entity=mon.0 client_cookie=e96ee5ef39bb85aa server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:03.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.359 7f6262ffd700 1 -- 172.21.2.112:0/426491955 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f62580009b0 con 0x7f62640d0340 2021-02-10T11:36:03.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.359 7f6262ffd700 1 -- 172.21.2.112:0/426491955 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f625800c0c0 con 0x7f62640d0340 2021-02-10T11:36:03.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.359 7f6262ffd700 1 -- 172.21.2.112:0/426491955 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f625800c5e0 con 0x7f62640d0340 2021-02-10T11:36:03.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.359 7f626a1e6700 1 -- 172.21.2.112:0/426491955 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f62640d0340 msgr2=0x7f62640ef640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:03.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.359 7f626a1e6700 1 --2- 172.21.2.112:0/426491955 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f62640d0340 0x7f62640ef640 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 rx=0x7f6258006650 tx=0x7f6258009a80).stop 2021-02-10T11:36:03.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.360 7f626a1e6700 1 -- 172.21.2.112:0/426491955 shutdown_connections 2021-02-10T11:36:03.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.360 7f626a1e6700 1 --2- 172.21.2.112:0/426491955 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f62640d1c70 0x7f62640efc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:03.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.360 7f626a1e6700 1 --2- 172.21.2.112:0/426491955 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f62640d0340 0x7f62640ef640 unknown :-1 s=CLOSED pgs=1562 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:03.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.360 7f626a1e6700 1 -- 172.21.2.112:0/426491955 >> 172.21.2.112:0/426491955 conn(0x7f62640ce450 msgr2=0x7f62640d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:03.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.360 7f626a1e6700 1 -- 172.21.2.112:0/426491955 shutdown_connections 2021-02-10T11:36:03.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.360 7f626a1e6700 1 -- 172.21.2.112:0/426491955 wait complete. 2021-02-10T11:36:03.365 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.374 7f626a1e6700 1 Processor -- start 2021-02-10T11:36:03.365 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.375 7f626a1e6700 1 -- start start 2021-02-10T11:36:03.365 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.375 7f626a1e6700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f62640b0ad0 0x7f6264091f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:03.365 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.375 7f626a1e6700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f62641a8850 0x7f62641a8c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:03.365 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.375 7f626a1e6700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f62641a6fa0 0x7f62641a73a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:03.366 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.375 7f626a1e6700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62641abfe0 con 0x7f62641a8850 2021-02-10T11:36:03.366 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.375 7f626a1e6700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62641a9c40 con 0x7f62640b0ad0 2021-02-10T11:36:03.366 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.375 7f626a1e6700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62641a9730 con 0x7f62641a6fa0 2021-02-10T11:36:03.366 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.375 7f62637fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f62641a8850 0x7f62641a8c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:03.366 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.375 7f62637fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f62641a8850 0x7f62641a8c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:51718/0 (socket says 172.21.2.112:51718) 2021-02-10T11:36:03.366 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.375 7f62637fe700 1 -- 172.21.2.112:0/533639864 learned_addr learned my addr 172.21.2.112:0/533639864 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:36:03.366 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.375 7f62637fe700 1 -- 172.21.2.112:0/533639864 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f62641a6fa0 msgr2=0x7f62641a73a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:03.366 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.375 7f62637fe700 1 --2- 172.21.2.112:0/533639864 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f62641a6fa0 0x7f62641a73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:03.366 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.375 7f62637fe700 1 -- 172.21.2.112:0/533639864 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f62640b0ad0 msgr2=0x7f6264091f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:03.366 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.375 7f62637fe700 1 --2- 172.21.2.112:0/533639864 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f62640b0ad0 0x7f6264091f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:03.366 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.375 7f62637fe700 1 -- 172.21.2.112:0/533639864 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62641a9270 con 0x7f62641a8850 2021-02-10T11:36:03.366 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.376 7f62637fe700 1 --2- 172.21.2.112:0/533639864 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f62641a8850 0x7f62641a8c50 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 rx=0x7f6258009630 tx=0x7f625800b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:03.366 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.376 7f6260ff9700 1 -- 172.21.2.112:0/533639864 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f62580093a0 con 0x7f62641a8850 2021-02-10T11:36:03.367 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.376 7f6260ff9700 1 -- 172.21.2.112:0/533639864 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f625800bd70 con 0x7f62641a8850 2021-02-10T11:36:03.367 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.376 7f6260ff9700 1 -- 172.21.2.112:0/533639864 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6258005560 con 0x7f62641a8850 2021-02-10T11:36:03.367 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.376 7f626a1e6700 1 -- 172.21.2.112:0/533639864 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62641a9270 con 0x7f62641a8850 2021-02-10T11:36:03.367 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.376 7f626a1e6700 1 -- 172.21.2.112:0/533639864 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62641a7cb0 con 0x7f62641a8850 2021-02-10T11:36:03.367 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.376 7f6260ff9700 1 -- 172.21.2.112:0/533639864 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f62580052e0 con 0x7f62641a8850 2021-02-10T11:36:03.367 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.376 7f6260ff9700 1 --2- 172.21.2.112:0/533639864 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6244018730 0x7f624401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:03.367 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.377 7f6260ff9700 1 -- 172.21.2.112:0/533639864 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f6258032620 con 0x7f62641a8850 2021-02-10T11:36:03.367 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.377 7f6263fff700 1 --2- 172.21.2.112:0/533639864 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6244018730 0x7f624401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:03.367 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.377 7f6263fff700 1 --2- 172.21.2.112:0/533639864 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6244018730 0x7f624401abc0 crc :-1 s=READY pgs=755 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:03.395 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.407 7f626a1e6700 1 -- 172.21.2.112:0/533639864 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62641a9710 con 0x7f62641a8850 2021-02-10T11:36:03.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.410 7f6260ff9700 1 -- 172.21.2.112:0/533639864 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6258009340 con 0x7f62641a8850 2021-02-10T11:36:03.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.536 7f626a1e6700 1 -- 172.21.2.112:0/533639864 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f62641a9710 con 0x7f62641a8850 2021-02-10T11:36:03.524 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.537 7f6260ff9700 1 -- 172.21.2.112:0/533639864 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v73) v1 ==== 74+0+13996 (secure 0 0 0) 0x7f62580052e0 con 0x7f62641a8850 2021-02-10T11:36:03.524 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:36:03.525 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":73,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:36:01.291077","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:35:58.263396","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":15,"pool_name":"cephfs_data","create_time":"2021-02-10 11:35:59.285157","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:36:03.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.538 7f626a1e6700 1 -- 172.21.2.112:0/533639864 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6244018730 msgr2=0x7f624401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:03.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.538 7f626a1e6700 1 --2- 172.21.2.112:0/533639864 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6244018730 0x7f624401abc0 crc :-1 s=READY pgs=755 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:03.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.538 7f626a1e6700 1 -- 172.21.2.112:0/533639864 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f62641a8850 msgr2=0x7f62641a8c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:03.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.538 7f626a1e6700 1 --2- 172.21.2.112:0/533639864 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f62641a8850 0x7f62641a8c50 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 rx=0x7f6258009630 tx=0x7f625800b980).stop 2021-02-10T11:36:03.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.538 7f626a1e6700 1 -- 172.21.2.112:0/533639864 shutdown_connections 2021-02-10T11:36:03.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.538 7f626a1e6700 1 --2- 172.21.2.112:0/533639864 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6244018730 0x7f624401abc0 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:03.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.538 7f626a1e6700 1 --2- 172.21.2.112:0/533639864 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f62641a6fa0 0x7f62641a73a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:03.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.538 7f626a1e6700 1 --2- 172.21.2.112:0/533639864 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f62641a8850 0x7f62641a8c50 unknown :-1 s=CLOSED pgs=1563 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:03.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.538 7f626a1e6700 1 --2- 172.21.2.112:0/533639864 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f62640b0ad0 0x7f6264091f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:03.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.538 7f626a1e6700 1 -- 172.21.2.112:0/533639864 >> 172.21.2.112:0/533639864 conn(0x7f626419f8f0 msgr2=0x7f62641a5f80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:03.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.539 7f626a1e6700 1 -- 172.21.2.112:0/533639864 shutdown_connections 2021-02-10T11:36:03.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.539 7f626a1e6700 1 -- 172.21.2.112:0/533639864 wait complete. 2021-02-10T11:36:03.535 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:36:03.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.960 7f8e2fd3b700 1 Processor -- start 2021-02-10T11:36:03.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.960 7f8e2fd3b700 1 -- start start 2021-02-10T11:36:03.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.961 7f8e2fd3b700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f8e280d1880 0x7f8e281b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:03.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.961 7f8e2fd3b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8e281b0af0 0x7f8e28108950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:03.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.961 7f8e2fd3b700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e280fe1d0 con 0x7f8e280d1c80 2021-02-10T11:36:03.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.961 7f8e2fd3b700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e28109a70 con 0x7f8e280d1880 2021-02-10T11:36:03.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.961 7f8e2fd3b700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e28109760 con 0x7f8e281b0af0 2021-02-10T11:36:03.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.961 7f8e2e358700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f8e280d1c80 0x7f8e28108f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:51186/0 (socket says 172.21.2.112:51186) 2021-02-10T11:36:03.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.961 7f8e2e358700 1 -- 172.21.2.112:0/2658663216 learned_addr learned my addr 172.21.2.112:0/2658663216 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:36:03.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.961 7f8e2d356700 1 --2- 172.21.2.112:0/2658663216 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8e281b0af0 0x7f8e28108950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:03.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.961 7f8e2db57700 1 --2- 172.21.2.112:0/2658663216 >> v2:172.21.2.120:3300/0 conn(0x7f8e280d1880 0x7f8e281b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:03.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.962 7f8e2cb55700 1 -- 172.21.2.112:0/2658663216 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3667250721 0 0) 0x7f8e20001500 con 0x7f8e280d1c80 2021-02-10T11:36:03.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.962 7f8e2cb55700 1 -- 172.21.2.112:0/2658663216 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e14002fe0 con 0x7f8e280d1c80 2021-02-10T11:36:03.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.962 7f8e2db57700 1 -- 172.21.2.112:0/2658663216 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8e281b0af0 msgr2=0x7f8e28108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:03.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.962 7f8e2db57700 1 --2- 172.21.2.112:0/2658663216 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8e281b0af0 0x7f8e28108950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:03.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.962 7f8e2db57700 1 -- 172.21.2.112:0/2658663216 >> v1:172.21.2.112:6789/0 conn(0x7f8e280d1c80 legacy=0x7f8e28108f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:03.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.962 7f8e2db57700 1 -- 172.21.2.112:0/2658663216 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e28088070 con 0x7f8e280d1880 2021-02-10T11:36:03.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.962 7f8e2db57700 1 --2- 172.21.2.112:0/2658663216 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8e280d1880 0x7f8e281b04d0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 rx=0x7f8e24006650 tx=0x7f8e24009a80).ready entity=mon.1 client_cookie=fee84b488daff5b6 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:03.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.962 7f8e2cb55700 1 -- 172.21.2.112:0/2658663216 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8e2400c0c0 con 0x7f8e280d1880 2021-02-10T11:36:03.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.963 7f8e2cb55700 1 -- 172.21.2.112:0/2658663216 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e2400c340 con 0x7f8e280d1880 2021-02-10T11:36:03.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.963 7f8e2cb55700 1 -- 172.21.2.112:0/2658663216 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8e2400c860 con 0x7f8e280d1880 2021-02-10T11:36:03.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.963 7f8e2fd3b700 1 -- 172.21.2.112:0/2658663216 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8e280d1880 msgr2=0x7f8e281b04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:03.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.963 7f8e2fd3b700 1 --2- 172.21.2.112:0/2658663216 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8e280d1880 0x7f8e281b04d0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 rx=0x7f8e24006650 tx=0x7f8e24009a80).stop 2021-02-10T11:36:03.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.963 7f8e2fd3b700 1 -- 172.21.2.112:0/2658663216 shutdown_connections 2021-02-10T11:36:03.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.963 7f8e2fd3b700 1 --2- 172.21.2.112:0/2658663216 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8e281b0af0 0x7f8e28108950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:03.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.963 7f8e2fd3b700 1 --2- 172.21.2.112:0/2658663216 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8e280d1880 0x7f8e281b04d0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:03.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.963 7f8e2fd3b700 1 -- 172.21.2.112:0/2658663216 >> 172.21.2.112:0/2658663216 conn(0x7f8e280ce430 msgr2=0x7f8e280d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:03.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.963 7f8e2fd3b700 1 -- 172.21.2.112:0/2658663216 shutdown_connections 2021-02-10T11:36:03.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.963 7f8e2fd3b700 1 -- 172.21.2.112:0/2658663216 wait complete. 2021-02-10T11:36:03.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.978 7f8e2fd3b700 1 Processor -- start 2021-02-10T11:36:03.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.978 7f8e2fd3b700 1 -- start start 2021-02-10T11:36:03.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.978 7f8e2fd3b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8e280abd30 0x7f8e28082bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:03.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.978 7f8e2fd3b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8e280831d0 0x7f8e2808a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:03.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.978 7f8e2fd3b700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8e280ac130 0x7f8e2808aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:03.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.978 7f8e2fd3b700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e28083700 con 0x7f8e280abd30 2021-02-10T11:36:03.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.978 7f8e2fd3b700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e2808b0c0 con 0x7f8e280ac130 2021-02-10T11:36:03.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.978 7f8e2fd3b700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e2808b2a0 con 0x7f8e280831d0 2021-02-10T11:36:03.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.979 7f8e2db57700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8e280abd30 0x7f8e28082bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:03.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.979 7f8e2db57700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8e280abd30 0x7f8e28082bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:51730/0 (socket says 172.21.2.112:51730) 2021-02-10T11:36:03.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.979 7f8e2db57700 1 -- 172.21.2.112:0/2575630962 learned_addr learned my addr 172.21.2.112:0/2575630962 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:36:03.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.979 7f8e2db57700 1 -- 172.21.2.112:0/2575630962 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8e280831d0 msgr2=0x7f8e2808a480 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:36:03.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.979 7f8e2db57700 1 --2- 172.21.2.112:0/2575630962 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8e280831d0 0x7f8e2808a480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:03.971 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.979 7f8e2db57700 1 -- 172.21.2.112:0/2575630962 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8e280ac130 msgr2=0x7f8e2808aaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:03.971 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.979 7f8e2db57700 1 --2- 172.21.2.112:0/2575630962 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8e280ac130 0x7f8e2808aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:03.971 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.979 7f8e2db57700 1 -- 172.21.2.112:0/2575630962 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e2808b5f0 con 0x7f8e280abd30 2021-02-10T11:36:03.971 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.979 7f8e2db57700 1 --2- 172.21.2.112:0/2575630962 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8e280abd30 0x7f8e28082bb0 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 rx=0x7f8e24009630 tx=0x7f8e2400b900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:03.971 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.979 7f8e1e7fc700 1 -- 172.21.2.112:0/2575630962 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8e240093a0 con 0x7f8e280abd30 2021-02-10T11:36:03.971 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.979 7f8e1e7fc700 1 -- 172.21.2.112:0/2575630962 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e2400bcf0 con 0x7f8e280abd30 2021-02-10T11:36:03.971 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.979 7f8e2fd3b700 1 -- 172.21.2.112:0/2575630962 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e2808b5f0 con 0x7f8e280abd30 2021-02-10T11:36:03.971 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.979 7f8e1e7fc700 1 -- 172.21.2.112:0/2575630962 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8e240054d0 con 0x7f8e280abd30 2021-02-10T11:36:03.971 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.979 7f8e2fd3b700 1 -- 172.21.2.112:0/2575630962 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e2808b5f0 con 0x7f8e280abd30 2021-02-10T11:36:03.971 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.980 7f8e1e7fc700 1 -- 172.21.2.112:0/2575630962 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8e240054f0 con 0x7f8e280abd30 2021-02-10T11:36:03.971 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.980 7f8e1e7fc700 1 --2- 172.21.2.112:0/2575630962 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8e10018730 0x7f8e1001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:03.972 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.980 7f8e1e7fc700 1 -- 172.21.2.112:0/2575630962 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f8e240327b0 con 0x7f8e280abd30 2021-02-10T11:36:03.972 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.980 7f8e2d356700 1 --2- 172.21.2.112:0/2575630962 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8e10018730 0x7f8e1001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:03.972 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:03.981 7f8e2d356700 1 --2- 172.21.2.112:0/2575630962 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8e10018730 0x7f8e1001abc0 crc :-1 s=READY pgs=756 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:03.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:04.011 7f8e2fd3b700 1 -- 172.21.2.112:0/2575630962 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e28082980 con 0x7f8e280abd30 2021-02-10T11:36:04.001 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:04.014 7f8e1e7fc700 1 -- 172.21.2.112:0/2575630962 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8e240052e0 con 0x7f8e280abd30 2021-02-10T11:36:04.132 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:04.144 7f8e2fd3b700 1 -- 172.21.2.112:0/2575630962 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f8e28082980 con 0x7f8e280abd30 2021-02-10T11:36:04.135 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:04.147 7f8e1e7fc700 1 -- 172.21.2.112:0/2575630962 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v45) v1 ==== 184+0+0 (secure 0 0 0) 0x7f8e240054f0 con 0x7f8e280abd30 2021-02-10T11:36:04.135 INFO:teuthology.orchestra.run.gibba012.stderr:updated caps for client.0 2021-02-10T11:36:04.136 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:04.149 7f8e2fd3b700 1 -- 172.21.2.112:0/2575630962 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8e10018730 msgr2=0x7f8e1001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:04.136 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:04.149 7f8e2fd3b700 1 --2- 172.21.2.112:0/2575630962 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8e10018730 0x7f8e1001abc0 crc :-1 s=READY pgs=756 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:04.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:04.149 7f8e2fd3b700 1 -- 172.21.2.112:0/2575630962 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8e280abd30 msgr2=0x7f8e28082bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:04.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:04.149 7f8e2fd3b700 1 --2- 172.21.2.112:0/2575630962 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8e280abd30 0x7f8e28082bb0 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 rx=0x7f8e24009630 tx=0x7f8e2400b900).stop 2021-02-10T11:36:04.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:04.149 7f8e2fd3b700 1 -- 172.21.2.112:0/2575630962 shutdown_connections 2021-02-10T11:36:04.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:04.149 7f8e2fd3b700 1 --2- 172.21.2.112:0/2575630962 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8e10018730 0x7f8e1001abc0 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:04.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:04.149 7f8e2fd3b700 1 --2- 172.21.2.112:0/2575630962 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8e280ac130 0x7f8e2808aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:04.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:04.149 7f8e2fd3b700 1 --2- 172.21.2.112:0/2575630962 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8e280831d0 0x7f8e2808a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:04.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:04.149 7f8e2fd3b700 1 --2- 172.21.2.112:0/2575630962 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8e280abd30 0x7f8e28082bb0 unknown :-1 s=CLOSED pgs=1565 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:04.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:04.149 7f8e2fd3b700 1 -- 172.21.2.112:0/2575630962 >> 172.21.2.112:0/2575630962 conn(0x7f8e280afc60 msgr2=0x7f8e280b50d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:04.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:04.149 7f8e2fd3b700 1 -- 172.21.2.112:0/2575630962 shutdown_connections 2021-02-10T11:36:04.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:04.149 7f8e2fd3b700 1 -- 172.21.2.112:0/2575630962 wait complete. 2021-02-10T11:36:04.145 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:36:04.586 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:04.581 7f58a042b700 1 Processor -- start 2021-02-10T11:36:04.587 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:04.581 7f58a042b700 1 -- start start 2021-02-10T11:36:04.587 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:04.581 7f58a042b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f58980d1c60 0x7f58980cef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:04.587 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:04.581 7f58a042b700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f58980cf570 0x7f58980cf970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:04.587 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:04.581 7f58a042b700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58980fe1b0 con 0x7f58980d1860 2021-02-10T11:36:04.587 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:04.581 7f58a042b700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58981b0ca0 con 0x7f58980cf570 2021-02-10T11:36:04.587 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:04.581 7f58a042b700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5898109a50 con 0x7f58980d1c60 2021-02-10T11:36:04.587 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:04.582 7f589e247700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f58980d1860 0x7f58981b04b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:51196/0 (socket says 172.21.2.112:51196) 2021-02-10T11:36:04.587 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:04.582 7f589e247700 1 -- 172.21.2.112:0/780579999 learned_addr learned my addr 172.21.2.112:0/780579999 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:36:04.587 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:04.582 7f589d245700 1 -- 172.21.2.112:0/780579999 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3152847609 0 0) 0x7f588c0010b0 con 0x7f58980d1860 2021-02-10T11:36:04.587 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:04.582 7f589d245700 1 -- 172.21.2.112:0/780579999 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5884002fe0 con 0x7f58980d1860 2021-02-10T11:36:04.588 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:04.582 7f589da46700 1 --2- 172.21.2.112:0/780579999 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f58980d1c60 0x7f58980cef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:04.588 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:04.582 7f589ea48700 1 --2- 172.21.2.112:0/780579999 >> v2:172.21.2.120:3300/0 conn(0x7f58980cf570 0x7f58980cf970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:04.588 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:04.582 7f589d245700 1 -- 172.21.2.112:0/780579999 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 836935654 0 0) 0x7f588c0010b0 con 0x7f58980d1860 2021-02-10T11:36:04.588 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:04.582 7f589d245700 1 -- 172.21.2.112:0/780579999 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f58980d1c60 msgr2=0x7f58980cef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:04.588 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:04.582 7f589d245700 1 --2- 172.21.2.112:0/780579999 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f58980d1c60 0x7f58980cef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:04.588 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:04.582 7f589d245700 1 -- 172.21.2.112:0/780579999 >> v2:172.21.2.120:3300/0 conn(0x7f58980cf570 msgr2=0x7f58980cf970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:04.588 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:04.582 7f589d245700 1 --2- 172.21.2.112:0/780579999 >> v2:172.21.2.120:3300/0 conn(0x7f58980cf570 0x7f58980cf970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:04.588 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:04.582 7f589d245700 1 -- 172.21.2.112:0/780579999 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5898088070 con 0x7f58980d1860 2021-02-10T11:36:04.588 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:04.582 7f589d245700 1 -- 172.21.2.112:0/780579999 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f588c0015b0 con 0x7f58980d1860 2021-02-10T11:36:04.588 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:04.582 7f589d245700 1 -- 172.21.2.112:0/780579999 >> v1:172.21.2.112:6789/0 conn(0x7f58980d1860 legacy=0x7f58981b04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:04.588 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:04.583 7f589d245700 1 --2- 172.21.2.112:0/780579999 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5884003b10 0x7f5884005f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:04.588 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:04.583 7f589d245700 1 --2- 172.21.2.112:0/780579999 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5884006780 0x7f5884008c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:04.589 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:04.583 7f589d245700 1 --2- 172.21.2.112:0/780579999 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f588404d510 0x7f588404fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:04.589 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:04.583 7f589d245700 1 -- 172.21.2.112:0/780579999 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58840502e0 con 0x7f588404d510 2021-02-10T11:36:04.589 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:04.583 7f589d245700 1 -- 172.21.2.112:0/780579999 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5884050500 con 0x7f5884006780 2021-02-10T11:36:04.589 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:04.583 7f589d245700 1 -- 172.21.2.112:0/780579999 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5884050720 con 0x7f5884003b10 2021-02-10T11:36:04.589 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:04.583 7f589e247700 1 --2- 172.21.2.112:0/780579999 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5884003b10 0x7f5884005f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:04.589 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:04.583 7f589ea48700 1 --2- 172.21.2.112:0/780579999 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f588404d510 0x7f588404fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:04.589 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:04.583 7f589da46700 1 --2- 172.21.2.112:0/780579999 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5884006780 0x7f5884008c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:04.589 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:04.583 7f589ea48700 1 -- 172.21.2.112:0/780579999 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5884003b10 msgr2=0x7f5884005f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:04.589 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:04.583 7f589ea48700 1 --2- 172.21.2.112:0/780579999 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5884003b10 0x7f5884005f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:04.589 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:04.583 7f589ea48700 1 -- 172.21.2.112:0/780579999 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5884006780 msgr2=0x7f5884008c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:04.589 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:04.583 7f589ea48700 1 --2- 172.21.2.112:0/780579999 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5884006780 0x7f5884008c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:04.589 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:04.583 7f589ea48700 1 -- 172.21.2.112:0/780579999 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58840509c0 con 0x7f588404d510 2021-02-10T11:36:04.589 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:04.583 7f589ea48700 1 --2- 172.21.2.112:0/780579999 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f588404d510 0x7f588404fa30 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 rx=0x7f5890006a40 tx=0x7f589000c0c0).ready entity=mon.0 client_cookie=de2c98f10b301f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:04.590 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:04.583 7f589d245700 1 -- 172.21.2.112:0/780579999 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f589000c4b0 con 0x7f588404d510 2021-02-10T11:36:04.590 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:04.583 7f589d245700 1 -- 172.21.2.112:0/780579999 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5890000f90 con 0x7f588404d510 2021-02-10T11:36:04.590 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:04.583 7f589d245700 1 -- 172.21.2.112:0/780579999 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f589000c770 con 0x7f588404d510 2021-02-10T11:36:04.590 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:04.584 7f58a042b700 1 -- 172.21.2.112:0/780579999 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f588404d510 msgr2=0x7f588404fa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:04.590 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:04.584 7f58a042b700 1 --2- 172.21.2.112:0/780579999 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f588404d510 0x7f588404fa30 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 rx=0x7f5890006a40 tx=0x7f589000c0c0).stop 2021-02-10T11:36:04.590 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:04.584 7f589ea48700 1 -- 172.21.2.112:0/780579999 reap_dead start 2021-02-10T11:36:04.590 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:04.584 7f58a042b700 1 -- 172.21.2.112:0/780579999 shutdown_connections 2021-02-10T11:36:04.590 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:04.584 7f58a042b700 1 -- 172.21.2.112:0/780579999 >> 172.21.2.112:0/780579999 conn(0x7f58980ce410 msgr2=0x7f58980d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:04.590 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:04.584 7f58a042b700 1 -- 172.21.2.112:0/780579999 shutdown_connections 2021-02-10T11:36:04.590 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:04.584 7f58a042b700 1 -- 172.21.2.112:0/780579999 wait complete. 2021-02-10T11:36:04.590 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:04.599 7f58a042b700 1 Processor -- start 2021-02-10T11:36:04.590 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:04.599 7f58a042b700 1 -- start start 2021-02-10T11:36:04.592 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:04.599 7f58a042b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5898189030 0x7f5898189430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:04.592 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:04.599 7f58a042b700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5898189a50 0x7f5898098a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:04.592 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:04.599 7f58a042b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5898189e50 0x7f5898099070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:04.592 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:04.599 7f58a042b700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5898098270 con 0x7f5898189030 2021-02-10T11:36:04.592 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:04.599 7f58a042b700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58980984d0 con 0x7f5898189a50 2021-02-10T11:36:04.592 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:04.599 7f58a042b700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f589818a590 con 0x7f5898189e50 2021-02-10T11:36:04.592 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:04.600 7f589e247700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5898189030 0x7f5898189430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:04.592 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:04.600 7f589e247700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5898189030 0x7f5898189430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:51744/0 (socket says 172.21.2.112:51744) 2021-02-10T11:36:04.592 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:04.600 7f589e247700 1 -- 172.21.2.112:0/3394934005 learned_addr learned my addr 172.21.2.112:0/3394934005 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:36:04.593 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:04.600 7f589e247700 1 -- 172.21.2.112:0/3394934005 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5898189e50 msgr2=0x7f5898099070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:04.593 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:04.600 7f589e247700 1 --2- 172.21.2.112:0/3394934005 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5898189e50 0x7f5898099070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:04.593 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:04.600 7f589e247700 1 -- 172.21.2.112:0/3394934005 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5898189a50 msgr2=0x7f5898098a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:04.593 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:04.600 7f589e247700 1 --2- 172.21.2.112:0/3394934005 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5898189a50 0x7f5898098a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:04.593 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:04.600 7f589e247700 1 -- 172.21.2.112:0/3394934005 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5898099690 con 0x7f5898189030 2021-02-10T11:36:04.593 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:04.600 7f589e247700 1 --2- 172.21.2.112:0/3394934005 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5898189030 0x7f5898189430 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 rx=0x7f588c001660 tx=0x7f588c00a930).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:04.593 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:04.600 7f588affd700 1 -- 172.21.2.112:0/3394934005 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f588c00ad20 con 0x7f5898189030 2021-02-10T11:36:04.593 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:04.600 7f588affd700 1 -- 172.21.2.112:0/3394934005 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f588c0016a0 con 0x7f5898189030 2021-02-10T11:36:04.593 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:04.600 7f588affd700 1 -- 172.21.2.112:0/3394934005 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f588c0043c0 con 0x7f5898189030 2021-02-10T11:36:04.593 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:04.600 7f58a042b700 1 -- 172.21.2.112:0/3394934005 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5898099690 con 0x7f5898189030 2021-02-10T11:36:04.593 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:04.600 7f58a042b700 1 -- 172.21.2.112:0/3394934005 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5898099690 con 0x7f5898189030 2021-02-10T11:36:04.594 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:04.601 7f588affd700 1 -- 172.21.2.112:0/3394934005 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f588c004140 con 0x7f5898189030 2021-02-10T11:36:04.594 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:04.601 7f588affd700 1 --2- 172.21.2.112:0/3394934005 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f58800186c0 0x7f588001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:04.594 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:04.601 7f588affd700 1 -- 172.21.2.112:0/3394934005 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f588c032600 con 0x7f5898189030 2021-02-10T11:36:04.594 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:04.601 7f589da46700 1 --2- 172.21.2.112:0/3394934005 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f58800186c0 0x7f588001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:04.594 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:04.602 7f589da46700 1 --2- 172.21.2.112:0/3394934005 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f58800186c0 0x7f588001abc0 crc :-1 s=READY pgs=757 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:04.619 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:04.631 7f58a042b700 1 -- 172.21.2.112:0/3394934005 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5898082960 con 0x7f5898189030 2021-02-10T11:36:04.621 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:04.634 7f588affd700 1 -- 172.21.2.112:0/3394934005 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f588c001180 con 0x7f5898189030 2021-02-10T11:36:04.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:04.765 7f58a042b700 1 -- 172.21.2.112:0/3394934005 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f5898098270 con 0x7f5898189030 2021-02-10T11:36:04.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:04.768 7f588affd700 1 -- 172.21.2.112:0/3394934005 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v46) v1 ==== 184+0+0 (secure 0 0 0) 0x7f588c004140 con 0x7f5898189030 2021-02-10T11:36:04.760 INFO:teuthology.orchestra.run.gibba012.stderr:updated caps for client.1 2021-02-10T11:36:04.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:04.774 7f58a042b700 1 -- 172.21.2.112:0/3394934005 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f58800186c0 msgr2=0x7f588001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:04.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:04.774 7f58a042b700 1 --2- 172.21.2.112:0/3394934005 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f58800186c0 0x7f588001abc0 crc :-1 s=READY pgs=757 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:04.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:04.774 7f58a042b700 1 -- 172.21.2.112:0/3394934005 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5898189030 msgr2=0x7f5898189430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:04.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:04.774 7f58a042b700 1 --2- 172.21.2.112:0/3394934005 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5898189030 0x7f5898189430 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 rx=0x7f588c001660 tx=0x7f588c00a930).stop 2021-02-10T11:36:04.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:04.774 7f58a042b700 1 -- 172.21.2.112:0/3394934005 shutdown_connections 2021-02-10T11:36:04.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:04.774 7f58a042b700 1 --2- 172.21.2.112:0/3394934005 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f58800186c0 0x7f588001abc0 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:04.763 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:04.774 7f58a042b700 1 --2- 172.21.2.112:0/3394934005 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5898189a50 0x7f5898098a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:04.763 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:04.774 7f58a042b700 1 --2- 172.21.2.112:0/3394934005 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5898189e50 0x7f5898099070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:04.763 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:04.774 7f58a042b700 1 --2- 172.21.2.112:0/3394934005 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5898189030 0x7f5898189430 unknown :-1 s=CLOSED pgs=1568 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:04.763 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:04.774 7f58a042b700 1 -- 172.21.2.112:0/3394934005 >> 172.21.2.112:0/3394934005 conn(0x7f58980b5f40 msgr2=0x7f58980c2da0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:04.763 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:04.774 7f58a042b700 1 -- 172.21.2.112:0/3394934005 shutdown_connections 2021-02-10T11:36:04.763 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:04.774 7f58a042b700 1 -- 172.21.2.112:0/3394934005 wait complete. 2021-02-10T11:36:04.770 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.2 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:36:05.210 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.205 7fef6299a700 1 Processor -- start 2021-02-10T11:36:05.210 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.205 7fef6299a700 1 -- start start 2021-02-10T11:36:05.210 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.205 7fef6299a700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fef5c0d1880 0x7fef5c1b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:05.210 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.205 7fef6299a700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fef5c1b0af0 0x7fef5c108950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:05.210 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.205 7fef6299a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fef5c0d1c80 0x7fef5c108f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:05.210 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.205 7fef6299a700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef5c0fe1d0 con 0x7fef5c0d1880 2021-02-10T11:36:05.210 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.205 7fef6299a700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef5c109a70 con 0x7fef5c1b0af0 2021-02-10T11:36:05.211 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.205 7fef6299a700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef5c088070 con 0x7fef5c0d1c80 2021-02-10T11:36:05.211 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.206 7fef5bfff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fef5c0d1880 0x7fef5c1b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:05.211 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.206 7fef5bfff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fef5c0d1880 0x7fef5c1b04d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:51752/0 (socket says 172.21.2.112:51752) 2021-02-10T11:36:05.211 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.206 7fef5bfff700 1 -- 172.21.2.112:0/1636010395 learned_addr learned my addr 172.21.2.112:0/1636010395 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:36:05.211 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.206 7fef5bfff700 1 -- 172.21.2.112:0/1636010395 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fef5c0d1c80 msgr2=0x7fef5c108f70 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:36:05.211 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.206 7fef5bfff700 1 --2- 172.21.2.112:0/1636010395 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fef5c0d1c80 0x7fef5c108f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:05.211 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.206 7fef5bfff700 1 -- 172.21.2.112:0/1636010395 >> v2:172.21.2.120:3300/0 conn(0x7fef5c1b0af0 msgr2=0x7fef5c108950 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:36:05.211 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.206 7fef5bfff700 1 --2- 172.21.2.112:0/1636010395 >> v2:172.21.2.120:3300/0 conn(0x7fef5c1b0af0 0x7fef5c108950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:05.211 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.206 7fef5bfff700 1 -- 172.21.2.112:0/1636010395 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef5c109590 con 0x7fef5c0d1880 2021-02-10T11:36:05.211 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.206 7fef5bfff700 1 --2- 172.21.2.112:0/1636010395 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fef5c0d1880 0x7fef5c1b04d0 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 rx=0x7fef4c006700 tx=0x7fef4c009b80).ready entity=mon.0 client_cookie=9a8d2e2feb7453f3 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:05.211 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.206 7fef5affd700 1 -- 172.21.2.112:0/1636010395 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fef4c00c0c0 con 0x7fef5c0d1880 2021-02-10T11:36:05.211 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.206 7fef5affd700 1 -- 172.21.2.112:0/1636010395 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef4c00c440 con 0x7fef5c0d1880 2021-02-10T11:36:05.211 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.206 7fef5affd700 1 -- 172.21.2.112:0/1636010395 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fef4c00c960 con 0x7fef5c0d1880 2021-02-10T11:36:05.211 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.206 7fef6299a700 1 -- 172.21.2.112:0/1636010395 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fef5c0d1880 msgr2=0x7fef5c1b04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:05.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.206 7fef6299a700 1 --2- 172.21.2.112:0/1636010395 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fef5c0d1880 0x7fef5c1b04d0 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 rx=0x7fef4c006700 tx=0x7fef4c009b80).stop 2021-02-10T11:36:05.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.208 7fef6299a700 1 -- 172.21.2.112:0/1636010395 shutdown_connections 2021-02-10T11:36:05.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.208 7fef6299a700 1 --2- 172.21.2.112:0/1636010395 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fef5c0d1c80 0x7fef5c108f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:05.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.208 7fef6299a700 1 --2- 172.21.2.112:0/1636010395 >> v2:172.21.2.120:3300/0 conn(0x7fef5c1b0af0 0x7fef5c108950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:05.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.208 7fef6299a700 1 --2- 172.21.2.112:0/1636010395 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fef5c0d1880 0x7fef5c1b04d0 unknown :-1 s=CLOSED pgs=1569 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:05.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.208 7fef6299a700 1 -- 172.21.2.112:0/1636010395 >> 172.21.2.112:0/1636010395 conn(0x7fef5c0ce430 msgr2=0x7fef5c0d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:05.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.208 7fef6299a700 1 -- 172.21.2.112:0/1636010395 shutdown_connections 2021-02-10T11:36:05.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.208 7fef6299a700 1 -- 172.21.2.112:0/1636010395 wait complete. 2021-02-10T11:36:05.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.222 7fef6299a700 1 Processor -- start 2021-02-10T11:36:05.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.222 7fef6299a700 1 -- start start 2021-02-10T11:36:05.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.222 7fef6299a700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fef5c1b9470 0x7fef5c1ae6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:05.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.222 7fef6299a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fef5c1aecf0 0x7fef5c0957f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:05.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.222 7fef6299a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fef5c1af0f0 0x7fef5c095e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:05.213 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.222 7fef6299a700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef5c096d70 con 0x7fef5c1af0f0 2021-02-10T11:36:05.213 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.222 7fef6299a700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef5c0a7780 con 0x7fef5c1b9470 2021-02-10T11:36:05.213 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.222 7fef6299a700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef5c1b9870 con 0x7fef5c1aecf0 2021-02-10T11:36:05.213 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.222 7fef60fb7700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fef5c1af0f0 0x7fef5c095e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:05.213 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.223 7fef60fb7700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fef5c1af0f0 0x7fef5c095e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:51760/0 (socket says 172.21.2.112:51760) 2021-02-10T11:36:05.213 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.223 7fef60fb7700 1 -- 172.21.2.112:0/2399142819 learned_addr learned my addr 172.21.2.112:0/2399142819 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:36:05.214 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.223 7fef60fb7700 1 -- 172.21.2.112:0/2399142819 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fef5c1aecf0 msgr2=0x7fef5c0957f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:05.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.223 7fef60fb7700 1 --2- 172.21.2.112:0/2399142819 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fef5c1aecf0 0x7fef5c0957f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:05.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.223 7fef60fb7700 1 -- 172.21.2.112:0/2399142819 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fef5c1b9470 msgr2=0x7fef5c1ae6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:05.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.223 7fef60fb7700 1 --2- 172.21.2.112:0/2399142819 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fef5c1b9470 0x7fef5c1ae6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:05.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.223 7fef60fb7700 1 -- 172.21.2.112:0/2399142819 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef5c19e6b0 con 0x7fef5c1af0f0 2021-02-10T11:36:05.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.223 7fef60fb7700 1 --2- 172.21.2.112:0/2399142819 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fef5c1af0f0 0x7fef5c095e10 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 rx=0x7fef54009960 tx=0x7fef5400bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:05.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.223 7fef58ff9700 1 -- 172.21.2.112:0/2399142819 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fef54003030 con 0x7fef5c1af0f0 2021-02-10T11:36:05.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.223 7fef58ff9700 1 -- 172.21.2.112:0/2399142819 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef54001310 con 0x7fef5c1af0f0 2021-02-10T11:36:05.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.223 7fef58ff9700 1 -- 172.21.2.112:0/2399142819 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fef54001830 con 0x7fef5c1af0f0 2021-02-10T11:36:05.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.223 7fef6299a700 1 -- 172.21.2.112:0/2399142819 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef5c19e6b0 con 0x7fef5c1af0f0 2021-02-10T11:36:05.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.223 7fef6299a700 1 -- 172.21.2.112:0/2399142819 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef5c096720 con 0x7fef5c1af0f0 2021-02-10T11:36:05.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.224 7fef58ff9700 1 -- 172.21.2.112:0/2399142819 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fef5400a030 con 0x7fef5c1af0f0 2021-02-10T11:36:05.216 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.224 7fef58ff9700 1 --2- 172.21.2.112:0/2399142819 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fef3c0186c0 0x7fef3c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:05.216 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.224 7fef58ff9700 1 -- 172.21.2.112:0/2399142819 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fef54032b20 con 0x7fef5c1af0f0 2021-02-10T11:36:05.216 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.226 7fef5bfff700 1 --2- 172.21.2.112:0/2399142819 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fef3c0186c0 0x7fef3c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:05.216 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.227 7fef5bfff700 1 --2- 172.21.2.112:0/2399142819 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fef3c0186c0 0x7fef3c01abc0 crc :-1 s=READY pgs=758 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:05.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.255 7fef6299a700 1 -- 172.21.2.112:0/2399142819 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef5c0829a0 con 0x7fef5c1af0f0 2021-02-10T11:36:05.245 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.258 7fef58ff9700 1 -- 172.21.2.112:0/2399142819 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fef54000f20 con 0x7fef5c1af0f0 2021-02-10T11:36:05.376 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.388 7fef6299a700 1 -- 172.21.2.112:0/2399142819 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fef5c0a7780 con 0x7fef5c1af0f0 2021-02-10T11:36:05.379 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.391 7fef58ff9700 1 -- 172.21.2.112:0/2399142819 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.2 v47) v1 ==== 184+0+0 (secure 0 0 0) 0x7fef54001310 con 0x7fef5c1af0f0 2021-02-10T11:36:05.379 INFO:teuthology.orchestra.run.gibba012.stderr:updated caps for client.2 2021-02-10T11:36:05.381 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.393 7fef6299a700 1 -- 172.21.2.112:0/2399142819 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fef3c0186c0 msgr2=0x7fef3c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:05.381 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.393 7fef6299a700 1 --2- 172.21.2.112:0/2399142819 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fef3c0186c0 0x7fef3c01abc0 crc :-1 s=READY pgs=758 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:05.382 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.393 7fef6299a700 1 -- 172.21.2.112:0/2399142819 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fef5c1af0f0 msgr2=0x7fef5c095e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:05.383 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.393 7fef6299a700 1 --2- 172.21.2.112:0/2399142819 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fef5c1af0f0 0x7fef5c095e10 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 rx=0x7fef54009960 tx=0x7fef5400bcf0).stop 2021-02-10T11:36:05.383 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.393 7fef6299a700 1 -- 172.21.2.112:0/2399142819 shutdown_connections 2021-02-10T11:36:05.383 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.393 7fef6299a700 1 --2- 172.21.2.112:0/2399142819 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fef3c0186c0 0x7fef3c01abc0 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:05.383 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.393 7fef6299a700 1 --2- 172.21.2.112:0/2399142819 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fef5c1af0f0 0x7fef5c095e10 unknown :-1 s=CLOSED pgs=1570 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:05.383 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.393 7fef6299a700 1 --2- 172.21.2.112:0/2399142819 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fef5c1aecf0 0x7fef5c0957f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:05.383 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.393 7fef6299a700 1 --2- 172.21.2.112:0/2399142819 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fef5c1b9470 0x7fef5c1ae6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:05.383 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.393 7fef6299a700 1 -- 172.21.2.112:0/2399142819 >> 172.21.2.112:0/2399142819 conn(0x7fef5c0845b0 msgr2=0x7fef5c18c290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:05.383 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.393 7fef6299a700 1 -- 172.21.2.112:0/2399142819 shutdown_connections 2021-02-10T11:36:05.383 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.393 7fef6299a700 1 -- 172.21.2.112:0/2399142819 wait complete. 2021-02-10T11:36:05.389 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.3 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:36:05.826 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.822 7f08233ef700 1 Processor -- start 2021-02-10T11:36:05.827 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.822 7f08233ef700 1 -- start start 2021-02-10T11:36:05.827 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.823 7f08233ef700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f081c0d1880 0x7f081c1b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:05.827 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.823 7f08233ef700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f081c1b0af0 0x7f081c108950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:05.827 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.823 7f08233ef700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f081c0fe1d0 con 0x7f081c0d1c80 2021-02-10T11:36:05.827 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.823 7f08233ef700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f081c109a70 con 0x7f081c0d1880 2021-02-10T11:36:05.827 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.823 7f08233ef700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f081c109760 con 0x7f081c1b0af0 2021-02-10T11:36:05.827 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.823 7f082120b700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f081c0d1880 0x7f081c1b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:05.827 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.823 7f082120b700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f081c0d1880 0x7f081c1b04d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:51766/0 (socket says 172.21.2.112:51766) 2021-02-10T11:36:05.827 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.823 7f082120b700 1 -- 172.21.2.112:0/2322380575 learned_addr learned my addr 172.21.2.112:0/2322380575 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:36:05.827 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.823 7f082120b700 1 -- 172.21.2.112:0/2322380575 >> v2:172.21.2.120:3300/0 conn(0x7f081c1b0af0 msgr2=0x7f081c108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:05.827 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.823 7f082120b700 1 --2- 172.21.2.112:0/2322380575 >> v2:172.21.2.120:3300/0 conn(0x7f081c1b0af0 0x7f081c108950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:05.827 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.823 7f082120b700 1 -- 172.21.2.112:0/2322380575 >> v1:172.21.2.120:6789/0 conn(0x7f081c0d1c80 legacy=0x7f081c108f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:05.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.823 7f082120b700 1 -- 172.21.2.112:0/2322380575 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f081c088070 con 0x7f081c0d1880 2021-02-10T11:36:05.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.824 7f082120b700 1 --2- 172.21.2.112:0/2322380575 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f081c0d1880 0x7f081c1b04d0 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 rx=0x7f0818006700 tx=0x7f0818009b80).ready entity=mon.0 client_cookie=d2829169aa2a6d6a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:05.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.824 7f080ffff700 1 -- 172.21.2.112:0/2322380575 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f081800c0c0 con 0x7f081c0d1880 2021-02-10T11:36:05.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.824 7f080ffff700 1 -- 172.21.2.112:0/2322380575 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f081800c340 con 0x7f081c0d1880 2021-02-10T11:36:05.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.824 7f080ffff700 1 -- 172.21.2.112:0/2322380575 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f081800c860 con 0x7f081c0d1880 2021-02-10T11:36:05.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.824 7f08233ef700 1 -- 172.21.2.112:0/2322380575 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f081c0d1880 msgr2=0x7f081c1b04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:05.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.824 7f08233ef700 1 --2- 172.21.2.112:0/2322380575 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f081c0d1880 0x7f081c1b04d0 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 rx=0x7f0818006700 tx=0x7f0818009b80).stop 2021-02-10T11:36:05.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.826 7f08233ef700 1 -- 172.21.2.112:0/2322380575 shutdown_connections 2021-02-10T11:36:05.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.826 7f08233ef700 1 --2- 172.21.2.112:0/2322380575 >> v2:172.21.2.120:3300/0 conn(0x7f081c1b0af0 0x7f081c108950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:05.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.826 7f08233ef700 1 --2- 172.21.2.112:0/2322380575 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f081c0d1880 0x7f081c1b04d0 unknown :-1 s=CLOSED pgs=1571 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:05.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.826 7f08233ef700 1 -- 172.21.2.112:0/2322380575 >> 172.21.2.112:0/2322380575 conn(0x7f081c0ce430 msgr2=0x7f081c0d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:05.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.826 7f08233ef700 1 -- 172.21.2.112:0/2322380575 shutdown_connections 2021-02-10T11:36:05.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.826 7f08233ef700 1 -- 172.21.2.112:0/2322380575 wait complete. 2021-02-10T11:36:05.830 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.839 7f08233ef700 1 Processor -- start 2021-02-10T11:36:05.830 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.839 7f08233ef700 1 -- start start 2021-02-10T11:36:05.830 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.840 7f08233ef700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f081c086520 0x7f081c1a1cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:05.830 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.840 7f08233ef700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f081c086920 0x7f081c187e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:05.830 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.840 7f08233ef700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f081c1a22d0 0x7f081c1884a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:05.830 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.840 7f08233ef700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f081c099600 con 0x7f081c1a22d0 2021-02-10T11:36:05.830 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.840 7f08233ef700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f081c0999f0 con 0x7f081c086920 2021-02-10T11:36:05.830 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.840 7f08233ef700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f081c188ac0 con 0x7f081c086520 2021-02-10T11:36:05.830 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.840 7f0821a0c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f081c1a22d0 0x7f081c1884a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:05.831 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.840 7f0821a0c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f081c1a22d0 0x7f081c1884a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:51772/0 (socket says 172.21.2.112:51772) 2021-02-10T11:36:05.831 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.840 7f0821a0c700 1 -- 172.21.2.112:0/2788038725 learned_addr learned my addr 172.21.2.112:0/2788038725 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:36:05.831 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.840 7f0821a0c700 1 -- 172.21.2.112:0/2788038725 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f081c086520 msgr2=0x7f081c1a1cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:05.831 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.840 7f0821a0c700 1 --2- 172.21.2.112:0/2788038725 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f081c086520 0x7f081c1a1cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:05.831 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.840 7f0821a0c700 1 -- 172.21.2.112:0/2788038725 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f081c086920 msgr2=0x7f081c187e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:05.831 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.840 7f0821a0c700 1 --2- 172.21.2.112:0/2788038725 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f081c086920 0x7f081c187e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:05.831 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.840 7f0821a0c700 1 -- 172.21.2.112:0/2788038725 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f081c188ce0 con 0x7f081c1a22d0 2021-02-10T11:36:05.831 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.841 7f0821a0c700 1 --2- 172.21.2.112:0/2788038725 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f081c1a22d0 0x7f081c1884a0 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 rx=0x7f0814009940 tx=0x7f081400bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:05.831 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.841 7f080dffb700 1 -- 172.21.2.112:0/2788038725 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0814000f90 con 0x7f081c1a22d0 2021-02-10T11:36:05.831 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.841 7f080dffb700 1 -- 172.21.2.112:0/2788038725 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f081400c2f0 con 0x7f081c1a22d0 2021-02-10T11:36:05.831 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.841 7f080dffb700 1 -- 172.21.2.112:0/2788038725 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f081400c750 con 0x7f081c1a22d0 2021-02-10T11:36:05.831 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.841 7f08233ef700 1 -- 172.21.2.112:0/2788038725 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f081c188ce0 con 0x7f081c1a22d0 2021-02-10T11:36:05.831 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.841 7f08233ef700 1 -- 172.21.2.112:0/2788038725 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f081c189270 con 0x7f081c1a22d0 2021-02-10T11:36:05.831 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.842 7f080dffb700 1 -- 172.21.2.112:0/2788038725 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0814001220 con 0x7f081c1a22d0 2021-02-10T11:36:05.832 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.842 7f080dffb700 1 --2- 172.21.2.112:0/2788038725 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f07fc0186c0 0x7f07fc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:05.832 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.842 7f080dffb700 1 -- 172.21.2.112:0/2788038725 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f0814032900 con 0x7f081c1a22d0 2021-02-10T11:36:05.832 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.842 7f082120b700 1 --2- 172.21.2.112:0/2788038725 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f07fc0186c0 0x7f07fc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:05.832 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.843 7f082120b700 1 --2- 172.21.2.112:0/2788038725 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f07fc0186c0 0x7f07fc01abc0 crc :-1 s=READY pgs=759 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:05.860 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.873 7f08233ef700 1 -- 172.21.2.112:0/2788038725 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f081c0fe1d0 con 0x7f081c1a22d0 2021-02-10T11:36:05.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:05.876 7f080dffb700 1 -- 172.21.2.112:0/2788038725 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f081402df70 con 0x7f081c1a22d0 2021-02-10T11:36:05.995 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:06.004 7f08233ef700 1 -- 172.21.2.112:0/2788038725 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f081c0999a0 con 0x7f081c1a22d0 2021-02-10T11:36:05.995 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:06.007 7f080dffb700 1 -- 172.21.2.112:0/2788038725 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.3 v48) v1 ==== 184+0+0 (secure 0 0 0) 0x7f081401f860 con 0x7f081c1a22d0 2021-02-10T11:36:05.999 INFO:teuthology.orchestra.run.gibba012.stderr:updated caps for client.3 2021-02-10T11:36:06.001 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:06.014 7f08233ef700 1 -- 172.21.2.112:0/2788038725 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f07fc0186c0 msgr2=0x7f07fc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:06.001 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:06.014 7f08233ef700 1 --2- 172.21.2.112:0/2788038725 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f07fc0186c0 0x7f07fc01abc0 crc :-1 s=READY pgs=759 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:06.001 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:06.014 7f08233ef700 1 -- 172.21.2.112:0/2788038725 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f081c1a22d0 msgr2=0x7f081c1884a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:06.001 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:06.014 7f08233ef700 1 --2- 172.21.2.112:0/2788038725 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f081c1a22d0 0x7f081c1884a0 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 rx=0x7f0814009940 tx=0x7f081400bcf0).stop 2021-02-10T11:36:06.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:06.014 7f08233ef700 1 -- 172.21.2.112:0/2788038725 shutdown_connections 2021-02-10T11:36:06.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:06.014 7f08233ef700 1 --2- 172.21.2.112:0/2788038725 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f07fc0186c0 0x7f07fc01abc0 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:06.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:06.014 7f08233ef700 1 --2- 172.21.2.112:0/2788038725 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f081c086920 0x7f081c187e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:06.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:06.014 7f08233ef700 1 --2- 172.21.2.112:0/2788038725 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f081c1a22d0 0x7f081c1884a0 unknown :-1 s=CLOSED pgs=1572 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:06.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:06.014 7f08233ef700 1 --2- 172.21.2.112:0/2788038725 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f081c086520 0x7f081c1a1cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:06.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:06.014 7f08233ef700 1 -- 172.21.2.112:0/2788038725 >> 172.21.2.112:0/2788038725 conn(0x7f081c19f880 msgr2=0x7f081c08f890 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:06.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:06.014 7f08233ef700 1 -- 172.21.2.112:0/2788038725 shutdown_connections 2021-02-10T11:36:06.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:06.014 7f08233ef700 1 -- 172.21.2.112:0/2788038725 wait complete. 2021-02-10T11:36:06.010 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:36:06.437 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:06.432 7f3e4e77b700 1 Processor -- start 2021-02-10T11:36:06.437 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:06.432 7f3e4e77b700 1 -- start start 2021-02-10T11:36:06.437 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:06.433 7f3e4e77b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3e480d1870 0x7f3e481b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:06.437 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:06.433 7f3e4e77b700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f3e481b0a10 0x7f3e480cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:06.437 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:06.433 7f3e4e77b700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f3e480d1c70 0x7f3e480cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:06.437 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:06.433 7f3e4e77b700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e480fe230 con 0x7f3e481b0a10 2021-02-10T11:36:06.437 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:06.433 7f3e4e77b700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e480e5ed0 con 0x7f3e480d1c70 2021-02-10T11:36:06.437 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:06.433 7f3e4e77b700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e480f0b10 con 0x7f3e480d1870 2021-02-10T11:36:06.437 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:06.433 7f3e477fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f3e481b0a10 0x7f3e480cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:06.437 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:06.433 7f3e477fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f3e481b0a10 0x7f3e480cf330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:51780/0 (socket says 172.21.2.112:51780) 2021-02-10T11:36:06.438 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:06.433 7f3e477fe700 1 -- 172.21.2.112:0/3756544695 learned_addr learned my addr 172.21.2.112:0/3756544695 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:36:06.438 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:06.433 7f3e477fe700 1 -- 172.21.2.112:0/3756544695 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3e480d1870 msgr2=0x7f3e481b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:06.438 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:06.433 7f3e477fe700 1 --2- 172.21.2.112:0/3756544695 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3e480d1870 0x7f3e481b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:06.438 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:06.433 7f3e477fe700 1 -- 172.21.2.112:0/3756544695 >> v2:172.21.2.120:3300/0 conn(0x7f3e480d1c70 msgr2=0x7f3e480cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:06.438 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:06.433 7f3e477fe700 1 --2- 172.21.2.112:0/3756544695 >> v2:172.21.2.120:3300/0 conn(0x7f3e480d1c70 0x7f3e480cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:06.438 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:06.433 7f3e477fe700 1 -- 172.21.2.112:0/3756544695 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e480cff70 con 0x7f3e481b0a10 2021-02-10T11:36:06.438 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:06.433 7f3e477fe700 1 --2- 172.21.2.112:0/3756544695 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3e481b0a10 0x7f3e480cf330 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 rx=0x7f3e34006700 tx=0x7f3e34009b80).ready entity=mon.0 client_cookie=cd9a0f1a44460ee6 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:06.438 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:06.434 7f3e46ffd700 1 -- 172.21.2.112:0/3756544695 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3e3400c0c0 con 0x7f3e481b0a10 2021-02-10T11:36:06.438 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:06.434 7f3e46ffd700 1 -- 172.21.2.112:0/3756544695 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e3400c400 con 0x7f3e481b0a10 2021-02-10T11:36:06.438 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:06.434 7f3e46ffd700 1 -- 172.21.2.112:0/3756544695 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3e3400c920 con 0x7f3e481b0a10 2021-02-10T11:36:06.438 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:06.434 7f3e4e77b700 1 -- 172.21.2.112:0/3756544695 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3e481b0a10 msgr2=0x7f3e480cf330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:06.438 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:06.434 7f3e4e77b700 1 --2- 172.21.2.112:0/3756544695 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3e481b0a10 0x7f3e480cf330 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 rx=0x7f3e34006700 tx=0x7f3e34009b80).stop 2021-02-10T11:36:06.438 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:06.434 7f3e4e77b700 1 -- 172.21.2.112:0/3756544695 shutdown_connections 2021-02-10T11:36:06.438 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:06.434 7f3e4e77b700 1 --2- 172.21.2.112:0/3756544695 >> v2:172.21.2.120:3300/0 conn(0x7f3e480d1c70 0x7f3e480cf950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:06.439 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:06.434 7f3e4e77b700 1 --2- 172.21.2.112:0/3756544695 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3e481b0a10 0x7f3e480cf330 unknown :-1 s=CLOSED pgs=1573 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:06.439 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:06.434 7f3e4e77b700 1 --2- 172.21.2.112:0/3756544695 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3e480d1870 0x7f3e481b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:06.439 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:06.434 7f3e4e77b700 1 -- 172.21.2.112:0/3756544695 >> 172.21.2.112:0/3756544695 conn(0x7f3e480ce450 msgr2=0x7f3e480d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:06.439 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:06.435 7f3e4e77b700 1 -- 172.21.2.112:0/3756544695 shutdown_connections 2021-02-10T11:36:06.439 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:06.435 7f3e4e77b700 1 -- 172.21.2.112:0/3756544695 wait complete. 2021-02-10T11:36:06.439 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:06.449 7f3e4e77b700 1 Processor -- start 2021-02-10T11:36:06.439 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:06.449 7f3e4e77b700 1 -- start start 2021-02-10T11:36:06.440 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:06.450 7f3e4e77b700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3e481b7cf0 0x7f3e4819cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:06.440 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:06.450 7f3e4e77b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3e481b80f0 0x7f3e481b84f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:06.440 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:06.450 7f3e4e77b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3e4819da20 0x7f3e4819de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:06.440 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:06.450 7f3e4e77b700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e480a77c0 con 0x7f3e481b80f0 2021-02-10T11:36:06.440 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:06.450 7f3e4e77b700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e4819d4a0 con 0x7f3e481b7cf0 2021-02-10T11:36:06.441 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:06.450 7f3e4e77b700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e480954f0 con 0x7f3e4819da20 2021-02-10T11:36:06.441 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:06.450 7f3e477fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3e481b80f0 0x7f3e481b84f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:06.441 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:06.450 7f3e477fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3e481b80f0 0x7f3e481b84f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:51786/0 (socket says 172.21.2.112:51786) 2021-02-10T11:36:06.441 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:06.450 7f3e477fe700 1 -- 172.21.2.112:0/1817230775 learned_addr learned my addr 172.21.2.112:0/1817230775 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:36:06.441 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:06.450 7f3e477fe700 1 -- 172.21.2.112:0/1817230775 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3e4819da20 msgr2=0x7f3e4819de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:06.441 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:06.450 7f3e477fe700 1 --2- 172.21.2.112:0/1817230775 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3e4819da20 0x7f3e4819de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:06.441 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:06.450 7f3e477fe700 1 -- 172.21.2.112:0/1817230775 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3e481b7cf0 msgr2=0x7f3e4819cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:06.441 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:06.450 7f3e477fe700 1 --2- 172.21.2.112:0/1817230775 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3e481b7cf0 0x7f3e4819cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:06.441 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:06.450 7f3e477fe700 1 -- 172.21.2.112:0/1817230775 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e481b8d60 con 0x7f3e481b80f0 2021-02-10T11:36:06.441 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:06.450 7f3e477fe700 1 --2- 172.21.2.112:0/1817230775 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3e481b80f0 0x7f3e481b84f0 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 rx=0x7f3e340066a0 tx=0x7f3e3400b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:06.441 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:06.452 7f3e44ff9700 1 -- 172.21.2.112:0/1817230775 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3e3400bd80 con 0x7f3e481b80f0 2021-02-10T11:36:06.441 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:06.452 7f3e44ff9700 1 -- 172.21.2.112:0/1817230775 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e340052e0 con 0x7f3e481b80f0 2021-02-10T11:36:06.441 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:06.452 7f3e44ff9700 1 -- 172.21.2.112:0/1817230775 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3e34005800 con 0x7f3e481b80f0 2021-02-10T11:36:06.441 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:06.452 7f3e4e77b700 1 -- 172.21.2.112:0/1817230775 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e481b8d60 con 0x7f3e481b80f0 2021-02-10T11:36:06.442 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:06.452 7f3e4e77b700 1 -- 172.21.2.112:0/1817230775 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e481b8d60 con 0x7f3e481b80f0 2021-02-10T11:36:06.442 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:06.452 7f3e44ff9700 1 -- 172.21.2.112:0/1817230775 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3e34006030 con 0x7f3e481b80f0 2021-02-10T11:36:06.442 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:06.452 7f3e44ff9700 1 --2- 172.21.2.112:0/1817230775 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3e240186c0 0x7f3e2401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:06.442 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:06.452 7f3e44ff9700 1 -- 172.21.2.112:0/1817230775 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f3e34032810 con 0x7f3e481b80f0 2021-02-10T11:36:06.442 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:06.452 7f3e47fff700 1 --2- 172.21.2.112:0/1817230775 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3e240186c0 0x7f3e2401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:06.442 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:06.452 7f3e47fff700 1 --2- 172.21.2.112:0/1817230775 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3e240186c0 0x7f3e2401abc0 crc :-1 s=READY pgs=760 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:06.470 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:06.483 7f3e4e77b700 1 -- 172.21.2.112:0/1817230775 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e480ebc00 con 0x7f3e481b80f0 2021-02-10T11:36:06.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:06.485 7f3e44ff9700 1 -- 172.21.2.112:0/1817230775 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3e3401f550 con 0x7f3e481b80f0 2021-02-10T11:36:06.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:06.613 7f3e4e77b700 1 -- 172.21.2.112:0/1817230775 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3e480f0b10 con 0x7f3e481b80f0 2021-02-10T11:36:06.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:06.613 7f3e44ff9700 1 -- 172.21.2.112:0/1817230775 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+2101 (secure 0 0 0) 0x7f3e3400bd80 con 0x7f3e481b80f0 2021-02-10T11:36:06.601 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:36:06.601 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":56,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8820,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/3613517698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":3613517698},{"type":"v1","addr":"172.21.2.112:6833","nonce":3613517698}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":52}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:36:00.289087","modified":"2021-02-10 11:36:02.295150","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7087},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7087":{"gid":7087,"name":"b","rank":0,"incarnation":53,"state":"up:active","state_seq":3,"addr":"172.21.2.120:6835/1870802251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":1870802251},{"type":"v1","addr":"172.21.2.120:6835","nonce":1870802251}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-02-10T11:36:06.602 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 56 2021-02-10T11:36:06.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:06.616 7f3e4e77b700 1 -- 172.21.2.112:0/1817230775 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3e240186c0 msgr2=0x7f3e2401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:06.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:06.616 7f3e4e77b700 1 --2- 172.21.2.112:0/1817230775 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3e240186c0 0x7f3e2401abc0 crc :-1 s=READY pgs=760 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:06.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:06.616 7f3e4e77b700 1 -- 172.21.2.112:0/1817230775 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3e481b80f0 msgr2=0x7f3e481b84f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:06.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:06.616 7f3e4e77b700 1 --2- 172.21.2.112:0/1817230775 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3e481b80f0 0x7f3e481b84f0 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 rx=0x7f3e340066a0 tx=0x7f3e3400b990).stop 2021-02-10T11:36:06.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:06.616 7f3e4e77b700 1 -- 172.21.2.112:0/1817230775 shutdown_connections 2021-02-10T11:36:06.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:06.616 7f3e4e77b700 1 --2- 172.21.2.112:0/1817230775 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3e240186c0 0x7f3e2401abc0 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:06.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:06.616 7f3e4e77b700 1 --2- 172.21.2.112:0/1817230775 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3e4819da20 0x7f3e4819de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:06.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:06.616 7f3e4e77b700 1 --2- 172.21.2.112:0/1817230775 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3e481b80f0 0x7f3e481b84f0 unknown :-1 s=CLOSED pgs=1574 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:06.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:06.616 7f3e4e77b700 1 --2- 172.21.2.112:0/1817230775 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3e481b7cf0 0x7f3e4819cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:06.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:06.616 7f3e4e77b700 1 -- 172.21.2.112:0/1817230775 >> 172.21.2.112:0/1817230775 conn(0x7f3e480ed810 msgr2=0x7f3e48094980 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:06.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:06.616 7f3e4e77b700 1 -- 172.21.2.112:0/1817230775 shutdown_connections 2021-02-10T11:36:06.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:06.616 7f3e4e77b700 1 -- 172.21.2.112:0/1817230775 wait complete. 2021-02-10T11:36:06.814 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 56, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:36:00.289087', 'modified': '2021-02-10 11:36:02.295150', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7087}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7087': {'gid': 7087, 'name': 'b', 'rank': 0, 'incarnation': 53, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.120:6835/1870802251', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.120:6834', 'nonce': 1870802251}, {'type': 'v1', 'addr': '172.21.2.120:6835', 'nonce': 1870802251}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:36:06.814 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:36:06.814 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:36:07.212 INFO:teuthology.orchestra.run.gibba020.stdout:{ 2021-02-10T11:36:07.212 INFO:teuthology.orchestra.run.gibba020.stdout: "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", 2021-02-10T11:36:07.212 INFO:teuthology.orchestra.run.gibba020.stdout: "whoami": 0, 2021-02-10T11:36:07.212 INFO:teuthology.orchestra.run.gibba020.stdout: "id": 7087, 2021-02-10T11:36:07.212 INFO:teuthology.orchestra.run.gibba020.stdout: "want_state": "up:active", 2021-02-10T11:36:07.213 INFO:teuthology.orchestra.run.gibba020.stdout: "state": "up:active", 2021-02-10T11:36:07.213 INFO:teuthology.orchestra.run.gibba020.stdout: "rank_uptime": 6.934248771, 2021-02-10T11:36:07.213 INFO:teuthology.orchestra.run.gibba020.stdout: "mdsmap_epoch": 56, 2021-02-10T11:36:07.213 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch": 73, 2021-02-10T11:36:07.213 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch_barrier": 73, 2021-02-10T11:36:07.214 INFO:teuthology.orchestra.run.gibba020.stdout: "uptime": 12.586741518 2021-02-10T11:36:07.214 INFO:teuthology.orchestra.run.gibba020.stdout:} 2021-02-10T11:36:07.216 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", "whoami": 0, "id": 7087, "want_state": "up:active", "state": "up:active", "rank_uptime": 6.934248771, "mdsmap_epoch": 56, "osdmap_epoch": 73, "osdmap_epoch_barrier": 73, "uptime": 12.586741518 } 2021-02-10T11:36:07.217 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:36:07.642 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:07.639 7fa47f229700 1 Processor -- start 2021-02-10T11:36:07.643 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:07.640 7fa47f229700 1 -- start start 2021-02-10T11:36:07.643 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:07.640 7fa47f229700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa4780d0340 0x7fa4780ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:07.643 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:07.640 7fa47f229700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fa4780d1c70 0x7fa4780efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:07.643 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:07.640 7fa47f229700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4780f0b10 con 0x7fa4780d1870 2021-02-10T11:36:07.643 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:07.640 7fa47f229700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4780e5ed0 con 0x7fa4780d1c70 2021-02-10T11:36:07.643 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:07.640 7fa47f229700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4781b0be0 con 0x7fa4780d0340 2021-02-10T11:36:07.643 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:07.640 7fa47d045700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fa4780d1870 0x7fa4781b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:51254/0 (socket says 172.21.2.112:51254) 2021-02-10T11:36:07.643 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:07.640 7fa47d045700 1 -- 172.21.2.112:0/279811120 learned_addr learned my addr 172.21.2.112:0/279811120 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:36:07.643 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:07.640 7fa47d846700 1 --2- 172.21.2.112:0/279811120 >> v2:172.21.2.112:3300/0 conn(0x7fa4780d1c70 0x7fa4780efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:07.643 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:07.641 7fa47c844700 1 --2- 172.21.2.112:0/279811120 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa4780d0340 0x7fa4780ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:07.643 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:07.641 7fa47d846700 1 -- 172.21.2.112:0/279811120 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa4780d0340 msgr2=0x7fa4780ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:07.644 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:07.641 7fa47d846700 1 --2- 172.21.2.112:0/279811120 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa4780d0340 0x7fa4780ef640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:07.644 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:07.641 7fa47d846700 1 -- 172.21.2.112:0/279811120 >> v1:172.21.2.112:6789/0 conn(0x7fa4780d1870 legacy=0x7fa4781b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:07.644 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:07.641 7fa47d846700 1 -- 172.21.2.112:0/279811120 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4780f20a0 con 0x7fa4780d1c70 2021-02-10T11:36:07.644 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:07.641 7fa47d846700 1 --2- 172.21.2.112:0/279811120 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa4780d1c70 0x7fa4780efc60 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 rx=0x7fa470006ab0 tx=0x7fa470009b40).ready entity=mon.0 client_cookie=9406915ea64bc432 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:07.644 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:07.641 7fa46bfff700 1 -- 172.21.2.112:0/279811120 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa4700022e0 con 0x7fa4780d1c70 2021-02-10T11:36:07.644 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:07.641 7fa46bfff700 1 -- 172.21.2.112:0/279811120 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa470002660 con 0x7fa4780d1c70 2021-02-10T11:36:07.644 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:07.641 7fa46bfff700 1 -- 172.21.2.112:0/279811120 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa470002a70 con 0x7fa4780d1c70 2021-02-10T11:36:07.644 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:07.641 7fa47f229700 1 -- 172.21.2.112:0/279811120 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa4780d1c70 msgr2=0x7fa4780efc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:07.644 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:07.641 7fa47f229700 1 --2- 172.21.2.112:0/279811120 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa4780d1c70 0x7fa4780efc60 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 rx=0x7fa470006ab0 tx=0x7fa470009b40).stop 2021-02-10T11:36:07.644 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:07.642 7fa47f229700 1 -- 172.21.2.112:0/279811120 shutdown_connections 2021-02-10T11:36:07.644 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:07.642 7fa47f229700 1 --2- 172.21.2.112:0/279811120 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa4780d1c70 0x7fa4780efc60 unknown :-1 s=CLOSED pgs=1576 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:07.644 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:07.642 7fa47f229700 1 --2- 172.21.2.112:0/279811120 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa4780d0340 0x7fa4780ef640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:07.644 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:07.642 7fa47f229700 1 -- 172.21.2.112:0/279811120 >> 172.21.2.112:0/279811120 conn(0x7fa4780ce450 msgr2=0x7fa4780d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:07.645 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:07.642 7fa47f229700 1 -- 172.21.2.112:0/279811120 shutdown_connections 2021-02-10T11:36:07.645 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:07.642 7fa47f229700 1 -- 172.21.2.112:0/279811120 wait complete. 2021-02-10T11:36:07.646 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:07.656 7fa47f229700 1 Processor -- start 2021-02-10T11:36:07.646 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:07.656 7fa47f229700 1 -- start start 2021-02-10T11:36:07.646 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:07.657 7fa47f229700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa4780b1ea0 0x7fa4780b22a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:07.646 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:07.657 7fa47f229700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa4780ebe30 0x7fa4780ec230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:07.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:07.657 7fa47f229700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa4780ec850 0x7fa4780ecc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:07.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:07.657 7fa47f229700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4780ed300 con 0x7fa4780ec850 2021-02-10T11:36:07.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:07.657 7fa47f229700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4781afe10 con 0x7fa4780ebe30 2021-02-10T11:36:07.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:07.657 7fa47f229700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4781af800 con 0x7fa4780b1ea0 2021-02-10T11:36:07.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:07.657 7fa47d846700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa4780ec850 0x7fa4780ecc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:07.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:07.657 7fa47d846700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa4780ec850 0x7fa4780ecc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:51800/0 (socket says 172.21.2.112:51800) 2021-02-10T11:36:07.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:07.657 7fa47d846700 1 -- 172.21.2.112:0/2020489308 learned_addr learned my addr 172.21.2.112:0/2020489308 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:36:07.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:07.658 7fa47c844700 1 --2- 172.21.2.112:0/2020489308 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa4780ebe30 0x7fa4780ec230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:07.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:07.658 7fa47d846700 1 -- 172.21.2.112:0/2020489308 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa4780b1ea0 msgr2=0x7fa4780b22a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:07.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:07.658 7fa47d045700 1 --2- 172.21.2.112:0/2020489308 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa4780b1ea0 0x7fa4780b22a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:07.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:07.658 7fa47d846700 1 --2- 172.21.2.112:0/2020489308 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa4780b1ea0 0x7fa4780b22a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:07.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:07.658 7fa47d846700 1 -- 172.21.2.112:0/2020489308 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa4780ebe30 msgr2=0x7fa4780ec230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:07.648 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:07.658 7fa47d846700 1 --2- 172.21.2.112:0/2020489308 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa4780ebe30 0x7fa4780ec230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:07.648 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:07.658 7fa47d846700 1 -- 172.21.2.112:0/2020489308 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4780ed880 con 0x7fa4780ec850 2021-02-10T11:36:07.648 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:07.658 7fa47c844700 1 --2- 172.21.2.112:0/2020489308 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa4780ebe30 0x7fa4780ec230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:36:07.648 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:07.658 7fa47d045700 1 --2- 172.21.2.112:0/2020489308 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa4780b1ea0 0x7fa4780b22a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:36:07.648 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:07.658 7fa47d846700 1 --2- 172.21.2.112:0/2020489308 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa4780ec850 0x7fa4780ecc50 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 rx=0x7fa47000b5e0 tx=0x7fa47000b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:07.648 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:07.658 7fa469ffb700 1 -- 172.21.2.112:0/2020489308 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa470006a70 con 0x7fa4780ec850 2021-02-10T11:36:07.648 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:07.658 7fa469ffb700 1 -- 172.21.2.112:0/2020489308 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa470009070 con 0x7fa4780ec850 2021-02-10T11:36:07.648 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:07.658 7fa469ffb700 1 -- 172.21.2.112:0/2020489308 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa470002470 con 0x7fa4780ec850 2021-02-10T11:36:07.648 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:07.658 7fa47f229700 1 -- 172.21.2.112:0/2020489308 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4780ed880 con 0x7fa4780ec850 2021-02-10T11:36:07.648 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:07.658 7fa47f229700 1 -- 172.21.2.112:0/2020489308 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4780ed880 con 0x7fa4780ec850 2021-02-10T11:36:07.648 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:07.659 7fa469ffb700 1 -- 172.21.2.112:0/2020489308 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa470006a20 con 0x7fa4780ec850 2021-02-10T11:36:07.648 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:07.659 7fa469ffb700 1 --2- 172.21.2.112:0/2020489308 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa4580186c0 0x7fa45801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:07.648 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:07.659 7fa469ffb700 1 -- 172.21.2.112:0/2020489308 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fa4700327c0 con 0x7fa4780ec850 2021-02-10T11:36:07.649 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:07.660 7fa47d045700 1 --2- 172.21.2.112:0/2020489308 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa4580186c0 0x7fa45801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:07.649 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:07.660 7fa47d045700 1 --2- 172.21.2.112:0/2020489308 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa4580186c0 0x7fa45801abc0 crc :-1 s=READY pgs=761 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:07.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:07.690 7fa47f229700 1 -- 172.21.2.112:0/2020489308 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4780ebc00 con 0x7fa4780ec850 2021-02-10T11:36:07.679 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:07.693 7fa469ffb700 1 -- 172.21.2.112:0/2020489308 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa470008f60 con 0x7fa4780ec850 2021-02-10T11:36:07.803 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:07.817 7fa47f229700 1 -- 172.21.2.112:0/2020489308 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa4780f0b10 con 0x7fa4780ec850 2021-02-10T11:36:07.804 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:07.817 7fa469ffb700 1 -- 172.21.2.112:0/2020489308 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v73) v1 ==== 74+0+13996 (secure 0 0 0) 0x7fa470006a20 con 0x7fa4780ec850 2021-02-10T11:36:07.804 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:36:07.804 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":73,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:36:01.291077","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:35:58.263396","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":15,"pool_name":"cephfs_data","create_time":"2021-02-10 11:35:59.285157","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:36:07.805 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:07.819 7fa47f229700 1 -- 172.21.2.112:0/2020489308 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa4580186c0 msgr2=0x7fa45801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:07.805 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:07.819 7fa47f229700 1 --2- 172.21.2.112:0/2020489308 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa4580186c0 0x7fa45801abc0 crc :-1 s=READY pgs=761 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:07.805 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:07.819 7fa47f229700 1 -- 172.21.2.112:0/2020489308 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa4780ec850 msgr2=0x7fa4780ecc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:07.805 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:07.819 7fa47f229700 1 --2- 172.21.2.112:0/2020489308 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa4780ec850 0x7fa4780ecc50 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 rx=0x7fa47000b5e0 tx=0x7fa47000b980).stop 2021-02-10T11:36:07.806 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:07.819 7fa47f229700 1 -- 172.21.2.112:0/2020489308 shutdown_connections 2021-02-10T11:36:07.806 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:07.819 7fa47f229700 1 --2- 172.21.2.112:0/2020489308 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa4580186c0 0x7fa45801abc0 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:07.806 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:07.819 7fa47f229700 1 --2- 172.21.2.112:0/2020489308 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa4780ebe30 0x7fa4780ec230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:07.806 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:07.819 7fa47f229700 1 --2- 172.21.2.112:0/2020489308 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa4780ec850 0x7fa4780ecc50 unknown :-1 s=CLOSED pgs=1577 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:07.806 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:07.819 7fa47f229700 1 --2- 172.21.2.112:0/2020489308 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa4780b1ea0 0x7fa4780b22a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:07.806 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:07.819 7fa47f229700 1 -- 172.21.2.112:0/2020489308 >> 172.21.2.112:0/2020489308 conn(0x7fa47819f120 msgr2=0x7fa47819e480 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:07.806 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:07.819 7fa47f229700 1 -- 172.21.2.112:0/2020489308 shutdown_connections 2021-02-10T11:36:07.807 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:07.819 7fa47f229700 1 -- 172.21.2.112:0/2020489308 wait complete. 2021-02-10T11:36:07.814 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:13:57.508758', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 14, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:35:58.263396', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '72', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 15, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:35:59.285157', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '73', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:36:07.815 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:36:08.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.243 7f10e5f65700 1 Processor -- start 2021-02-10T11:36:08.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.243 7f10e5f65700 1 -- start start 2021-02-10T11:36:08.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.243 7f10e5f65700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f10e00d1870 0x7f10e01b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:08.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.243 7f10e5f65700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f10e01b0ae0 0x7f10e0108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:08.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.243 7f10e5f65700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10e0109a60 con 0x7f10e00d1c70 2021-02-10T11:36:08.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.243 7f10e5f65700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10e00e5ef0 con 0x7f10e01b0ae0 2021-02-10T11:36:08.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.243 7f10e5f65700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10e0109750 con 0x7f10e00d1870 2021-02-10T11:36:08.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.243 7f10dffff700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f10e00d1c70 0x7f10e0108f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:51268/0 (socket says 172.21.2.112:51268) 2021-02-10T11:36:08.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.243 7f10dffff700 1 -- 172.21.2.112:0/151413007 learned_addr learned my addr 172.21.2.112:0/151413007 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:36:08.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.243 7f10deffd700 1 --2- 172.21.2.112:0/151413007 >> v2:172.21.2.112:3300/0 conn(0x7f10e01b0ae0 0x7f10e0108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:08.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.244 7f10deffd700 1 -- 172.21.2.112:0/151413007 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f10e00d1870 msgr2=0x7f10e01b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:08.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.244 7f10deffd700 1 --2- 172.21.2.112:0/151413007 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f10e00d1870 0x7f10e01b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:08.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.244 7f10deffd700 1 -- 172.21.2.112:0/151413007 >> v1:172.21.2.112:6789/0 conn(0x7f10e00d1c70 legacy=0x7f10e0108f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:08.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.244 7f10deffd700 1 -- 172.21.2.112:0/151413007 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10e0088070 con 0x7f10e01b0ae0 2021-02-10T11:36:08.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.244 7f10de7fc700 1 -- 172.21.2.112:0/151413007 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2965577494 0 0) 0x7f10d8001500 con 0x7f10e00d1c70 2021-02-10T11:36:08.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.244 7f10deffd700 1 --2- 172.21.2.112:0/151413007 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f10e01b0ae0 0x7f10e0108940 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 rx=0x7f10d0006650 tx=0x7f10d0009a80).ready entity=mon.0 client_cookie=d684d82dfd0973d4 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:08.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.244 7f10de7fc700 1 -- 172.21.2.112:0/151413007 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f10d00009b0 con 0x7f10e01b0ae0 2021-02-10T11:36:08.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.244 7f10de7fc700 1 -- 172.21.2.112:0/151413007 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10d000c0c0 con 0x7f10e01b0ae0 2021-02-10T11:36:08.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.244 7f10de7fc700 1 -- 172.21.2.112:0/151413007 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f10d00009b0 con 0x7f10e01b0ae0 2021-02-10T11:36:08.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.244 7f10e5f65700 1 -- 172.21.2.112:0/151413007 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f10e01b0ae0 msgr2=0x7f10e0108940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:08.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.244 7f10e5f65700 1 --2- 172.21.2.112:0/151413007 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f10e01b0ae0 0x7f10e0108940 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 rx=0x7f10d0006650 tx=0x7f10d0009a80).stop 2021-02-10T11:36:08.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.245 7f10e5f65700 1 -- 172.21.2.112:0/151413007 shutdown_connections 2021-02-10T11:36:08.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.245 7f10e5f65700 1 --2- 172.21.2.112:0/151413007 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f10e01b0ae0 0x7f10e0108940 unknown :-1 s=CLOSED pgs=1579 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:08.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.245 7f10e5f65700 1 --2- 172.21.2.112:0/151413007 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f10e00d1870 0x7f10e01b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:08.248 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.245 7f10e5f65700 1 -- 172.21.2.112:0/151413007 >> 172.21.2.112:0/151413007 conn(0x7f10e00ce420 msgr2=0x7f10e00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:08.248 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.245 7f10e5f65700 1 -- 172.21.2.112:0/151413007 shutdown_connections 2021-02-10T11:36:08.248 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.245 7f10e5f65700 1 -- 172.21.2.112:0/151413007 wait complete. 2021-02-10T11:36:08.250 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.259 7f10e5f65700 1 Processor -- start 2021-02-10T11:36:08.250 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.260 7f10e5f65700 1 -- start start 2021-02-10T11:36:08.250 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.260 7f10e5f65700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f10e00a73c0 0x7f10e00a6b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:08.250 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.260 7f10e5f65700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f10e00add40 0x7f10e00ae140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:08.250 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.260 7f10e5f65700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f10e00ae760 0x7f10e00af180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:08.250 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.260 7f10e5f65700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10e00aec00 con 0x7f10e00ae760 2021-02-10T11:36:08.250 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.260 7f10e5f65700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10e00875d0 con 0x7f10e00a73c0 2021-02-10T11:36:08.250 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.260 7f10e5f65700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10e00af7a0 con 0x7f10e00add40 2021-02-10T11:36:08.250 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.260 7f10dffff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f10e00ae760 0x7f10e00af180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:08.250 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.260 7f10dffff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f10e00ae760 0x7f10e00af180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:51814/0 (socket says 172.21.2.112:51814) 2021-02-10T11:36:08.250 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.260 7f10dffff700 1 -- 172.21.2.112:0/393308417 learned_addr learned my addr 172.21.2.112:0/393308417 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:36:08.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.260 7f10deffd700 1 --2- 172.21.2.112:0/393308417 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f10e00add40 0x7f10e00ae140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:08.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.260 7f10df7fe700 1 --2- 172.21.2.112:0/393308417 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f10e00a73c0 0x7f10e00a6b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:08.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.261 7f10dffff700 1 -- 172.21.2.112:0/393308417 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f10e00add40 msgr2=0x7f10e00ae140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:08.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.261 7f10dffff700 1 --2- 172.21.2.112:0/393308417 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f10e00add40 0x7f10e00ae140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:08.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.261 7f10dffff700 1 -- 172.21.2.112:0/393308417 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f10e00a73c0 msgr2=0x7f10e00a6b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:08.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.261 7f10dffff700 1 --2- 172.21.2.112:0/393308417 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f10e00a73c0 0x7f10e00a6b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:08.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.261 7f10dffff700 1 -- 172.21.2.112:0/393308417 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10e00af9c0 con 0x7f10e00ae760 2021-02-10T11:36:08.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.261 7f10deffd700 1 --2- 172.21.2.112:0/393308417 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f10e00add40 0x7f10e00ae140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:36:08.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.261 7f10dffff700 1 --2- 172.21.2.112:0/393308417 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f10e00ae760 0x7f10e00af180 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 rx=0x7f10d8001c10 tx=0x7f10d800ac60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:08.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.261 7f10d7fff700 1 -- 172.21.2.112:0/393308417 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f10d80070d0 con 0x7f10e00ae760 2021-02-10T11:36:08.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.261 7f10d7fff700 1 -- 172.21.2.112:0/393308417 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10d8007350 con 0x7f10e00ae760 2021-02-10T11:36:08.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.261 7f10d7fff700 1 -- 172.21.2.112:0/393308417 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f10d80049a0 con 0x7f10e00ae760 2021-02-10T11:36:08.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.261 7f10e5f65700 1 -- 172.21.2.112:0/393308417 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10e00af9c0 con 0x7f10e00ae760 2021-02-10T11:36:08.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.261 7f10e5f65700 1 -- 172.21.2.112:0/393308417 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10e00aff50 con 0x7f10e00ae760 2021-02-10T11:36:08.252 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.262 7f10d7fff700 1 -- 172.21.2.112:0/393308417 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f10d8004720 con 0x7f10e00ae760 2021-02-10T11:36:08.252 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.262 7f10d7fff700 1 --2- 172.21.2.112:0/393308417 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f10c40186c0 0x7f10c401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:08.252 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.262 7f10d7fff700 1 -- 172.21.2.112:0/393308417 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f10d80328c0 con 0x7f10e00ae760 2021-02-10T11:36:08.252 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.262 7f10df7fe700 1 --2- 172.21.2.112:0/393308417 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f10c40186c0 0x7f10c401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:08.252 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.263 7f10df7fe700 1 --2- 172.21.2.112:0/393308417 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f10c40186c0 0x7f10c401abc0 crc :-1 s=READY pgs=762 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:08.279 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.293 7f10e5f65700 1 -- 172.21.2.112:0/393308417 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10e00b04f0 con 0x7f10e00ae760 2021-02-10T11:36:08.282 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.295 7f10d7fff700 1 -- 172.21.2.112:0/393308417 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f10d8020ff0 con 0x7f10e00ae760 2021-02-10T11:36:08.409 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.423 7f10e5f65700 1 -- 172.21.2.112:0/393308417 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f10e00b04f0 con 0x7f10e00ae760 2021-02-10T11:36:08.410 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.423 7f10d7fff700 1 -- 172.21.2.112:0/393308417 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v73) v1 ==== 92+0+10 (secure 0 0 0) 0x7f10d8020ff0 con 0x7f10e00ae760 2021-02-10T11:36:08.410 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:36:08.412 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.425 7f10e5f65700 1 -- 172.21.2.112:0/393308417 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f10c40186c0 msgr2=0x7f10c401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:08.412 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.425 7f10e5f65700 1 --2- 172.21.2.112:0/393308417 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f10c40186c0 0x7f10c401abc0 crc :-1 s=READY pgs=762 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:08.412 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.425 7f10e5f65700 1 -- 172.21.2.112:0/393308417 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f10e00ae760 msgr2=0x7f10e00af180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:08.412 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.425 7f10e5f65700 1 --2- 172.21.2.112:0/393308417 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f10e00ae760 0x7f10e00af180 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 rx=0x7f10d8001c10 tx=0x7f10d800ac60).stop 2021-02-10T11:36:08.412 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.425 7f10e5f65700 1 -- 172.21.2.112:0/393308417 shutdown_connections 2021-02-10T11:36:08.412 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.425 7f10e5f65700 1 --2- 172.21.2.112:0/393308417 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f10c40186c0 0x7f10c401abc0 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:08.413 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.425 7f10e5f65700 1 --2- 172.21.2.112:0/393308417 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f10e00ae760 0x7f10e00af180 unknown :-1 s=CLOSED pgs=1580 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:08.413 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.425 7f10e5f65700 1 --2- 172.21.2.112:0/393308417 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f10e00add40 0x7f10e00ae140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:08.413 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.425 7f10e5f65700 1 --2- 172.21.2.112:0/393308417 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f10e00a73c0 0x7f10e00a6b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:08.413 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.425 7f10e5f65700 1 -- 172.21.2.112:0/393308417 >> 172.21.2.112:0/393308417 conn(0x7f10e00be980 msgr2=0x7f10e0092cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:08.413 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.426 7f10e5f65700 1 -- 172.21.2.112:0/393308417 shutdown_connections 2021-02-10T11:36:08.413 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.426 7f10e5f65700 1 -- 172.21.2.112:0/393308417 wait complete. 2021-02-10T11:36:08.421 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:36:08.851 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.848 7f65b8a8f700 1 Processor -- start 2021-02-10T11:36:08.851 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.848 7f65b8a8f700 1 -- start start 2021-02-10T11:36:08.851 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.848 7f65b8a8f700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f65b40d1860 0x7f65b41b04b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:08.851 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.848 7f65b8a8f700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f65b41b0ad0 0x7f65b4108930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:08.851 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.848 7f65b8a8f700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65b40fe1b0 con 0x7f65b40d1c60 2021-02-10T11:36:08.851 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.848 7f65b8a8f700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65b4109a50 con 0x7f65b40d1860 2021-02-10T11:36:08.851 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.848 7f65b8a8f700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65b4109740 con 0x7f65b41b0ad0 2021-02-10T11:36:08.852 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.849 7f65b285e700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f65b40d1860 0x7f65b41b04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:08.852 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.849 7f65b305f700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f65b40d1c60 0x7f65b4108f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:51280/0 (socket says 172.21.2.112:51280) 2021-02-10T11:36:08.852 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.849 7f65b285e700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f65b40d1860 0x7f65b41b04b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:51818/0 (socket says 172.21.2.112:51818) 2021-02-10T11:36:08.852 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.849 7f65b305f700 1 -- 172.21.2.112:0/6205080 learned_addr learned my addr 172.21.2.112:0/6205080 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:36:08.852 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.849 7f65b205d700 1 --2- 172.21.2.112:0/6205080 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f65b41b0ad0 0x7f65b4108930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:08.852 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.849 7f65b285e700 1 -- 172.21.2.112:0/6205080 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f65b41b0ad0 msgr2=0x7f65b4108930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:08.852 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.849 7f65b285e700 1 --2- 172.21.2.112:0/6205080 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f65b41b0ad0 0x7f65b4108930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:08.852 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.849 7f65b285e700 1 -- 172.21.2.112:0/6205080 >> v1:172.21.2.112:6789/0 conn(0x7f65b40d1c60 legacy=0x7f65b4108f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:08.852 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.849 7f65b285e700 1 -- 172.21.2.112:0/6205080 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65b4088070 con 0x7f65b40d1860 2021-02-10T11:36:08.852 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.849 7f65b285e700 1 --2- 172.21.2.112:0/6205080 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f65b40d1860 0x7f65b41b04b0 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 rx=0x7f65ac0069d0 tx=0x7f65ac009e40).ready entity=mon.0 client_cookie=5112290cba429cde server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:08.852 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.849 7f65b185c700 1 -- 172.21.2.112:0/6205080 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f65ac00b510 con 0x7f65b40d1860 2021-02-10T11:36:08.852 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.849 7f65b185c700 1 -- 172.21.2.112:0/6205080 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65ac00b790 con 0x7f65b40d1860 2021-02-10T11:36:08.853 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.849 7f65b185c700 1 -- 172.21.2.112:0/6205080 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f65ac00bcb0 con 0x7f65b40d1860 2021-02-10T11:36:08.853 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.849 7f65b8a8f700 1 -- 172.21.2.112:0/6205080 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f65b40d1860 msgr2=0x7f65b41b04b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:08.853 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.849 7f65b8a8f700 1 --2- 172.21.2.112:0/6205080 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f65b40d1860 0x7f65b41b04b0 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 rx=0x7f65ac0069d0 tx=0x7f65ac009e40).stop 2021-02-10T11:36:08.853 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.850 7f65b8a8f700 1 -- 172.21.2.112:0/6205080 shutdown_connections 2021-02-10T11:36:08.853 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.850 7f65b8a8f700 1 --2- 172.21.2.112:0/6205080 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f65b41b0ad0 0x7f65b4108930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:08.853 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.850 7f65b8a8f700 1 --2- 172.21.2.112:0/6205080 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f65b40d1860 0x7f65b41b04b0 unknown :-1 s=CLOSED pgs=1582 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:08.853 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.850 7f65b8a8f700 1 -- 172.21.2.112:0/6205080 >> 172.21.2.112:0/6205080 conn(0x7f65b40ce410 msgr2=0x7f65b40d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:08.853 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.850 7f65b8a8f700 1 -- 172.21.2.112:0/6205080 shutdown_connections 2021-02-10T11:36:08.853 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.850 7f65b8a8f700 1 -- 172.21.2.112:0/6205080 wait complete. 2021-02-10T11:36:08.855 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.865 7f65b8a8f700 1 Processor -- start 2021-02-10T11:36:08.855 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.865 7f65b8a8f700 1 -- start start 2021-02-10T11:36:08.855 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.865 7f65b8a8f700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f65b41a05e0 0x7f65b418b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:08.855 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.865 7f65b8a8f700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f65b40b8780 0x7f65b40bfa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:08.855 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.865 7f65b8a8f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f65b40c0030 0x7f65b4189e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:08.855 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.865 7f65b8a8f700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65b40b82a0 con 0x7f65b40c0030 2021-02-10T11:36:08.855 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.865 7f65b8a8f700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65b40c0430 con 0x7f65b41a05e0 2021-02-10T11:36:08.855 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.865 7f65b8a8f700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65b40c0650 con 0x7f65b40b8780 2021-02-10T11:36:08.855 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.866 7f65b305f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f65b40c0030 0x7f65b4189e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:08.856 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.866 7f65b305f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f65b40c0030 0x7f65b4189e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:51828/0 (socket says 172.21.2.112:51828) 2021-02-10T11:36:08.856 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.866 7f65b305f700 1 -- 172.21.2.112:0/3591017990 learned_addr learned my addr 172.21.2.112:0/3591017990 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:36:08.856 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.866 7f65b285e700 1 --2- 172.21.2.112:0/3591017990 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f65b41a05e0 0x7f65b418b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:08.856 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.866 7f65b205d700 1 --2- 172.21.2.112:0/3591017990 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f65b40b8780 0x7f65b40bfa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:08.856 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.866 7f65b305f700 1 -- 172.21.2.112:0/3591017990 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f65b40b8780 msgr2=0x7f65b40bfa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:08.856 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.866 7f65b305f700 1 --2- 172.21.2.112:0/3591017990 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f65b40b8780 0x7f65b40bfa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:08.856 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.866 7f65b305f700 1 -- 172.21.2.112:0/3591017990 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f65b41a05e0 msgr2=0x7f65b418b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:08.856 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.866 7f65b305f700 1 --2- 172.21.2.112:0/3591017990 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f65b41a05e0 0x7f65b418b970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:08.856 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.866 7f65b305f700 1 -- 172.21.2.112:0/3591017990 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65b418a490 con 0x7f65b40c0030 2021-02-10T11:36:08.856 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.866 7f65b305f700 1 --2- 172.21.2.112:0/3591017990 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f65b40c0030 0x7f65b4189e70 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 rx=0x7f65a4007fc0 tx=0x7f65a400a8f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:08.856 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.866 7f65a37fe700 1 -- 172.21.2.112:0/3591017990 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f65a4001180 con 0x7f65b40c0030 2021-02-10T11:36:08.856 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.867 7f65a37fe700 1 -- 172.21.2.112:0/3591017990 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65a4007030 con 0x7f65b40c0030 2021-02-10T11:36:08.856 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.867 7f65b8a8f700 1 -- 172.21.2.112:0/3591017990 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65b418a490 con 0x7f65b40c0030 2021-02-10T11:36:08.857 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.867 7f65a37fe700 1 -- 172.21.2.112:0/3591017990 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f65a4004620 con 0x7f65b40c0030 2021-02-10T11:36:08.857 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.867 7f65b8a8f700 1 -- 172.21.2.112:0/3591017990 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65b418aa20 con 0x7f65b40c0030 2021-02-10T11:36:08.857 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.867 7f65a37fe700 1 -- 172.21.2.112:0/3591017990 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f65a4001160 con 0x7f65b40c0030 2021-02-10T11:36:08.857 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.867 7f65a37fe700 1 --2- 172.21.2.112:0/3591017990 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f65980186c0 0x7f659801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:08.857 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.867 7f65a37fe700 1 -- 172.21.2.112:0/3591017990 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f65a4032430 con 0x7f65b40c0030 2021-02-10T11:36:08.857 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.868 7f65b285e700 1 --2- 172.21.2.112:0/3591017990 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f65980186c0 0x7f659801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:08.857 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.868 7f65b285e700 1 --2- 172.21.2.112:0/3591017990 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f65980186c0 0x7f659801abc0 crc :-1 s=READY pgs=763 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:08.884 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.898 7f65b8a8f700 1 -- 172.21.2.112:0/3591017990 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65b4082a20 con 0x7f65b40c0030 2021-02-10T11:36:08.887 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:08.901 7f65a37fe700 1 -- 172.21.2.112:0/3591017990 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f65a4030d60 con 0x7f65b40c0030 2021-02-10T11:36:09.017 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:09.031 7f65b8a8f700 1 -- 172.21.2.112:0/3591017990 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f65b4082a20 con 0x7f65b40c0030 2021-02-10T11:36:09.018 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:09.031 7f65a37fe700 1 -- 172.21.2.112:0/3591017990 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v73) v1 ==== 104+0+10 (secure 0 0 0) 0x7f65a4001160 con 0x7f65b40c0030 2021-02-10T11:36:09.018 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:36:09.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:09.033 7f65b8a8f700 1 -- 172.21.2.112:0/3591017990 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f65980186c0 msgr2=0x7f659801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:09.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:09.033 7f65b8a8f700 1 --2- 172.21.2.112:0/3591017990 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f65980186c0 0x7f659801abc0 crc :-1 s=READY pgs=763 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:09.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:09.033 7f65b8a8f700 1 -- 172.21.2.112:0/3591017990 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f65b40c0030 msgr2=0x7f65b4189e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:09.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:09.033 7f65b8a8f700 1 --2- 172.21.2.112:0/3591017990 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f65b40c0030 0x7f65b4189e70 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 rx=0x7f65a4007fc0 tx=0x7f65a400a8f0).stop 2021-02-10T11:36:09.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:09.033 7f65b8a8f700 1 -- 172.21.2.112:0/3591017990 shutdown_connections 2021-02-10T11:36:09.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:09.033 7f65b8a8f700 1 --2- 172.21.2.112:0/3591017990 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f65980186c0 0x7f659801abc0 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:09.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:09.033 7f65b8a8f700 1 --2- 172.21.2.112:0/3591017990 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f65b40c0030 0x7f65b4189e70 unknown :-1 s=CLOSED pgs=1583 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:09.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:09.033 7f65b8a8f700 1 --2- 172.21.2.112:0/3591017990 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f65b40b8780 0x7f65b40bfa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:09.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:09.033 7f65b8a8f700 1 --2- 172.21.2.112:0/3591017990 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f65b41a05e0 0x7f65b418b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:09.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:09.033 7f65b8a8f700 1 -- 172.21.2.112:0/3591017990 >> 172.21.2.112:0/3591017990 conn(0x7f65b40a6a80 msgr2=0x7f65b40ae110 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:09.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:09.033 7f65b8a8f700 1 -- 172.21.2.112:0/3591017990 shutdown_connections 2021-02-10T11:36:09.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:09.033 7f65b8a8f700 1 -- 172.21.2.112:0/3591017990 wait complete. 2021-02-10T11:36:09.029 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:36:09.458 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:09.455 7f969e1ec700 1 Processor -- start 2021-02-10T11:36:09.458 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:09.455 7f969e1ec700 1 -- start start 2021-02-10T11:36:09.458 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:09.455 7f969e1ec700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f96980d1c70 0x7f96980cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:09.458 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:09.455 7f969e1ec700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f96980cf580 0x7f96980cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:09.458 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:09.455 7f969e1ec700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96980fe1c0 con 0x7f96980d1870 2021-02-10T11:36:09.458 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:09.455 7f969e1ec700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96981b0cb0 con 0x7f96980cf580 2021-02-10T11:36:09.458 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:09.455 7f969e1ec700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9698109a60 con 0x7f96980d1c70 2021-02-10T11:36:09.458 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:09.456 7f969c809700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f96980cf580 0x7f96980cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:09.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:09.456 7f969c809700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f96980cf580 0x7f96980cf980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:51836/0 (socket says 172.21.2.112:51836) 2021-02-10T11:36:09.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:09.456 7f969c809700 1 -- 172.21.2.112:0/199988585 learned_addr learned my addr 172.21.2.112:0/199988585 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:36:09.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:09.456 7f969c809700 1 -- 172.21.2.112:0/199988585 >> v2:172.21.2.120:3300/0 conn(0x7f96980d1c70 msgr2=0x7f96980cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:09.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:09.456 7f96977fe700 1 --2- 172.21.2.112:0/199988585 >> v2:172.21.2.120:3300/0 conn(0x7f96980d1c70 0x7f96980cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:09.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:09.456 7f969c809700 1 --2- 172.21.2.112:0/199988585 >> v2:172.21.2.120:3300/0 conn(0x7f96980d1c70 0x7f96980cef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:09.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:09.456 7f969c809700 1 -- 172.21.2.112:0/199988585 >> v1:172.21.2.120:6789/0 conn(0x7f96980d1870 legacy=0x7f96981b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:09.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:09.456 7f969c809700 1 -- 172.21.2.112:0/199988585 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9698088070 con 0x7f96980cf580 2021-02-10T11:36:09.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:09.456 7f96977fe700 1 --2- 172.21.2.112:0/199988585 >> v2:172.21.2.120:3300/0 conn(0x7f96980d1c70 0x7f96980cef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:36:09.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:09.456 7f969c809700 1 --2- 172.21.2.112:0/199988585 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f96980cf580 0x7f96980cf980 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 rx=0x7f9690006f00 tx=0x7f9690009f10).ready entity=mon.0 client_cookie=d0fc7d789d345530 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:09.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:09.457 7f9696ffd700 1 -- 172.21.2.112:0/199988585 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f969000b3b0 con 0x7f96980cf580 2021-02-10T11:36:09.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:09.457 7f9696ffd700 1 -- 172.21.2.112:0/199988585 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f969000b730 con 0x7f96980cf580 2021-02-10T11:36:09.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:09.457 7f9696ffd700 1 -- 172.21.2.112:0/199988585 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f969000bc50 con 0x7f96980cf580 2021-02-10T11:36:09.460 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:09.457 7f969e1ec700 1 -- 172.21.2.112:0/199988585 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f96980cf580 msgr2=0x7f96980cf980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:09.460 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:09.457 7f969e1ec700 1 --2- 172.21.2.112:0/199988585 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f96980cf580 0x7f96980cf980 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 rx=0x7f9690006f00 tx=0x7f9690009f10).stop 2021-02-10T11:36:09.460 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:09.457 7f969e1ec700 1 -- 172.21.2.112:0/199988585 shutdown_connections 2021-02-10T11:36:09.460 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:09.457 7f969e1ec700 1 --2- 172.21.2.112:0/199988585 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f96980cf580 0x7f96980cf980 unknown :-1 s=CLOSED pgs=1584 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:09.460 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:09.457 7f969e1ec700 1 --2- 172.21.2.112:0/199988585 >> v2:172.21.2.120:3300/0 conn(0x7f96980d1c70 0x7f96980cef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:09.460 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:09.457 7f969e1ec700 1 -- 172.21.2.112:0/199988585 >> 172.21.2.112:0/199988585 conn(0x7f96980ce420 msgr2=0x7f96980d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:09.460 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:09.457 7f969e1ec700 1 -- 172.21.2.112:0/199988585 shutdown_connections 2021-02-10T11:36:09.460 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:09.457 7f969e1ec700 1 -- 172.21.2.112:0/199988585 wait complete. 2021-02-10T11:36:09.461 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:09.472 7f969e1ec700 1 Processor -- start 2021-02-10T11:36:09.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:09.472 7f969e1ec700 1 -- start start 2021-02-10T11:36:09.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:09.472 7f969e1ec700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f96980c3820 0x7f96980a7950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:09.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:09.472 7f969e1ec700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f96981a44c0 0x7f96981a48c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:09.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:09.472 7f969e1ec700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f96980863d0 0x7f96980867d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:09.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:09.472 7f969e1ec700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9698085ed0 con 0x7f96980c3820 2021-02-10T11:36:09.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:09.472 7f969e1ec700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96980c0740 con 0x7f96981a44c0 2021-02-10T11:36:09.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:09.472 7f969e1ec700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9698086df0 con 0x7f96980863d0 2021-02-10T11:36:09.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:09.472 7f9697fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f96980c3820 0x7f96980a7950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:09.463 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:09.473 7f9697fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f96980c3820 0x7f96980a7950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:51840/0 (socket says 172.21.2.112:51840) 2021-02-10T11:36:09.463 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:09.473 7f9697fff700 1 -- 172.21.2.112:0/1392000124 learned_addr learned my addr 172.21.2.112:0/1392000124 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:36:09.463 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:09.473 7f969c809700 1 --2- 172.21.2.112:0/1392000124 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f96980863d0 0x7f96980867d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:09.463 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:09.473 7f96977fe700 1 --2- 172.21.2.112:0/1392000124 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f96981a44c0 0x7f96981a48c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:09.463 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:09.473 7f9697fff700 1 -- 172.21.2.112:0/1392000124 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f96980863d0 msgr2=0x7f96980867d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:09.463 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:09.473 7f9697fff700 1 --2- 172.21.2.112:0/1392000124 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f96980863d0 0x7f96980867d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:09.463 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:09.473 7f9697fff700 1 -- 172.21.2.112:0/1392000124 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f96981a44c0 msgr2=0x7f96981a48c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:09.463 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:09.473 7f9697fff700 1 --2- 172.21.2.112:0/1392000124 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f96981a44c0 0x7f96981a48c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:09.463 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:09.473 7f9697fff700 1 -- 172.21.2.112:0/1392000124 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9698087010 con 0x7f96980c3820 2021-02-10T11:36:09.463 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:09.473 7f9697fff700 1 --2- 172.21.2.112:0/1392000124 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f96980c3820 0x7f96980a7950 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 rx=0x7f968800b5b0 tx=0x7f968800b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:09.463 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:09.473 7f9694ff9700 1 -- 172.21.2.112:0/1392000124 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f968800bd40 con 0x7f96980c3820 2021-02-10T11:36:09.463 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:09.473 7f9694ff9700 1 -- 172.21.2.112:0/1392000124 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9688006240 con 0x7f96980c3820 2021-02-10T11:36:09.463 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:09.473 7f9694ff9700 1 -- 172.21.2.112:0/1392000124 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9688005720 con 0x7f96980c3820 2021-02-10T11:36:09.464 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:09.473 7f969e1ec700 1 -- 172.21.2.112:0/1392000124 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9698087010 con 0x7f96980c3820 2021-02-10T11:36:09.464 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:09.474 7f969e1ec700 1 -- 172.21.2.112:0/1392000124 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9698083000 con 0x7f96980c3820 2021-02-10T11:36:09.464 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:09.474 7f9694ff9700 1 -- 172.21.2.112:0/1392000124 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f96880054a0 con 0x7f96980c3820 2021-02-10T11:36:09.464 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:09.474 7f9694ff9700 1 --2- 172.21.2.112:0/1392000124 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f96780186c0 0x7f967801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:09.464 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:09.474 7f9694ff9700 1 -- 172.21.2.112:0/1392000124 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f9688032590 con 0x7f96980c3820 2021-02-10T11:36:09.464 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:09.474 7f96977fe700 1 --2- 172.21.2.112:0/1392000124 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f96780186c0 0x7f967801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:09.464 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:09.475 7f96977fe700 1 --2- 172.21.2.112:0/1392000124 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f96780186c0 0x7f967801abc0 crc :-1 s=READY pgs=764 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:09.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:09.505 7f969e1ec700 1 -- 172.21.2.112:0/1392000124 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9698085e50 con 0x7f96980c3820 2021-02-10T11:36:09.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:09.508 7f9694ff9700 1 -- 172.21.2.112:0/1392000124 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9688000c60 con 0x7f96980c3820 2021-02-10T11:36:09.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:09.638 7f969e1ec700 1 -- 172.21.2.112:0/1392000124 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f96980fe1c0 con 0x7f96980c3820 2021-02-10T11:36:09.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:09.638 7f9694ff9700 1 -- 172.21.2.112:0/1392000124 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v73) v1 ==== 100+0+10 (secure 0 0 0) 0x7f968800bd40 con 0x7f96980c3820 2021-02-10T11:36:09.625 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:36:09.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:09.640 7f969e1ec700 1 -- 172.21.2.112:0/1392000124 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f96780186c0 msgr2=0x7f967801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:09.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:09.640 7f969e1ec700 1 --2- 172.21.2.112:0/1392000124 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f96780186c0 0x7f967801abc0 crc :-1 s=READY pgs=764 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:09.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:09.640 7f969e1ec700 1 -- 172.21.2.112:0/1392000124 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f96980c3820 msgr2=0x7f96980a7950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:09.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:09.640 7f969e1ec700 1 --2- 172.21.2.112:0/1392000124 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f96980c3820 0x7f96980a7950 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 rx=0x7f968800b5b0 tx=0x7f968800b950).stop 2021-02-10T11:36:09.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:09.640 7f969e1ec700 1 -- 172.21.2.112:0/1392000124 shutdown_connections 2021-02-10T11:36:09.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:09.640 7f969e1ec700 1 --2- 172.21.2.112:0/1392000124 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f96780186c0 0x7f967801abc0 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:09.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:09.640 7f969e1ec700 1 --2- 172.21.2.112:0/1392000124 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f96981a44c0 0x7f96981a48c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:09.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:09.640 7f969e1ec700 1 --2- 172.21.2.112:0/1392000124 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f96980863d0 0x7f96980867d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:09.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:09.640 7f969e1ec700 1 --2- 172.21.2.112:0/1392000124 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f96980c3820 0x7f96980a7950 unknown :-1 s=CLOSED pgs=1585 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:09.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:09.640 7f969e1ec700 1 -- 172.21.2.112:0/1392000124 >> 172.21.2.112:0/1392000124 conn(0x7f96980872c0 msgr2=0x7f9698095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:09.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:09.640 7f969e1ec700 1 -- 172.21.2.112:0/1392000124 shutdown_connections 2021-02-10T11:36:09.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:09.641 7f969e1ec700 1 -- 172.21.2.112:0/1392000124 wait complete. 2021-02-10T11:36:09.635 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:36:09.635 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:36:10.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:10.060 7f2907c3b700 1 Processor -- start 2021-02-10T11:36:10.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:10.060 7f2907c3b700 1 -- start start 2021-02-10T11:36:10.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:10.061 7f2907c3b700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f29000d1c70 0x7f29000f20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:10.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:10.061 7f2907c3b700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29000f0b10 con 0x7f29000d1870 2021-02-10T11:36:10.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:10.061 7f2907c3b700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29001b0be0 con 0x7f29000d0340 2021-02-10T11:36:10.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:10.061 7f2907c3b700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29000e5ed0 con 0x7f29000d1c70 2021-02-10T11:36:10.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:10.061 7f2905a57700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f29000d1870 0x7f29001b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:51306/0 (socket says 172.21.2.112:51306) 2021-02-10T11:36:10.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:10.061 7f2905a57700 1 -- 172.21.2.112:0/183817889 learned_addr learned my addr 172.21.2.112:0/183817889 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:36:10.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:10.061 7f2906258700 1 --2- 172.21.2.112:0/183817889 >> v2:172.21.2.120:3300/0 conn(0x7f29000d1c70 0x7f29000f20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:10.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:10.061 7f2904a55700 1 -- 172.21.2.112:0/183817889 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2906202350 0 0) 0x7f28fc000ce0 con 0x7f29000d1870 2021-02-10T11:36:10.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:10.061 7f2904a55700 1 -- 172.21.2.112:0/183817889 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28ec002fe0 con 0x7f29000d1870 2021-02-10T11:36:10.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:10.061 7f2906258700 1 -- 172.21.2.112:0/183817889 >> v1:172.21.2.120:6789/0 conn(0x7f29000d0340 legacy=0x7f29000ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:10.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:10.061 7f2906258700 1 -- 172.21.2.112:0/183817889 >> v1:172.21.2.112:6789/0 conn(0x7f29000d1870 legacy=0x7f29001b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:10.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:10.061 7f2906258700 1 -- 172.21.2.112:0/183817889 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29000efe30 con 0x7f29000d1c70 2021-02-10T11:36:10.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:10.062 7f2906258700 1 --2- 172.21.2.112:0/183817889 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f29000d1c70 0x7f29000f20a0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 rx=0x7f28f8006e50 tx=0x7f28f8009e70).ready entity=mon.1 client_cookie=c7840111b8b7a0b3 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:10.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:10.062 7f2904a55700 1 -- 172.21.2.112:0/183817889 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f28f800b510 con 0x7f29000d1c70 2021-02-10T11:36:10.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:10.062 7f2904a55700 1 -- 172.21.2.112:0/183817889 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28f8006790 con 0x7f29000d1c70 2021-02-10T11:36:10.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:10.062 7f2904a55700 1 -- 172.21.2.112:0/183817889 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f28f800b940 con 0x7f29000d1c70 2021-02-10T11:36:10.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:10.062 7f2907c3b700 1 -- 172.21.2.112:0/183817889 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f29000d1c70 msgr2=0x7f29000f20a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:10.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:10.062 7f2907c3b700 1 --2- 172.21.2.112:0/183817889 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f29000d1c70 0x7f29000f20a0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 rx=0x7f28f8006e50 tx=0x7f28f8009e70).stop 2021-02-10T11:36:10.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:10.063 7f2907c3b700 1 -- 172.21.2.112:0/183817889 shutdown_connections 2021-02-10T11:36:10.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:10.063 7f2907c3b700 1 --2- 172.21.2.112:0/183817889 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f29000d1c70 0x7f29000f20a0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:10.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:10.063 7f2907c3b700 1 -- 172.21.2.112:0/183817889 >> 172.21.2.112:0/183817889 conn(0x7f29000ce450 msgr2=0x7f29000d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:10.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:10.063 7f2907c3b700 1 -- 172.21.2.112:0/183817889 shutdown_connections 2021-02-10T11:36:10.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:10.063 7f2907c3b700 1 -- 172.21.2.112:0/183817889 wait complete. 2021-02-10T11:36:10.068 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:10.078 7f2907c3b700 1 Processor -- start 2021-02-10T11:36:10.068 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:10.078 7f2907c3b700 1 -- start start 2021-02-10T11:36:10.068 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:10.078 7f2907c3b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f290018f0a0 0x7f2900196ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:10.068 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:10.078 7f2907c3b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f29001970d0 0x7f29001974d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:10.068 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:10.078 7f2907c3b700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2900094650 0x7f2900094a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:10.068 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:10.078 7f2907c3b700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2900095bd0 con 0x7f29001970d0 2021-02-10T11:36:10.068 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:10.078 7f2907c3b700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29000ad7c0 con 0x7f2900094650 2021-02-10T11:36:10.068 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:10.078 7f2907c3b700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f290018ec70 con 0x7f290018f0a0 2021-02-10T11:36:10.068 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:10.078 7f2905256700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f29001970d0 0x7f29001974d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:10.069 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:10.078 7f2905256700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f29001970d0 0x7f29001974d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:51848/0 (socket says 172.21.2.112:51848) 2021-02-10T11:36:10.069 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:10.078 7f2905256700 1 -- 172.21.2.112:0/1571625630 learned_addr learned my addr 172.21.2.112:0/1571625630 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:36:10.069 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:10.079 7f2905256700 1 -- 172.21.2.112:0/1571625630 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f290018f0a0 msgr2=0x7f2900196ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:10.069 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:10.079 7f2905256700 1 --2- 172.21.2.112:0/1571625630 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f290018f0a0 0x7f2900196ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:10.069 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:10.079 7f2905256700 1 -- 172.21.2.112:0/1571625630 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2900094650 msgr2=0x7f2900094a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:10.069 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:10.079 7f2905256700 1 --2- 172.21.2.112:0/1571625630 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2900094650 0x7f2900094a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:10.069 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:10.079 7f2905256700 1 -- 172.21.2.112:0/1571625630 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2900095070 con 0x7f29001970d0 2021-02-10T11:36:10.069 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:10.079 7f2905256700 1 --2- 172.21.2.112:0/1571625630 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f29001970d0 0x7f29001974d0 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 rx=0x7f28f000b5b0 tx=0x7f28f000b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:10.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:10.079 7f28f67fc700 1 -- 172.21.2.112:0/1571625630 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f28f000bd40 con 0x7f29001970d0 2021-02-10T11:36:10.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:10.079 7f28f67fc700 1 -- 172.21.2.112:0/1571625630 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28f0000c60 con 0x7f29001970d0 2021-02-10T11:36:10.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:10.079 7f28f67fc700 1 -- 172.21.2.112:0/1571625630 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f28f00062a0 con 0x7f29001970d0 2021-02-10T11:36:10.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:10.079 7f2907c3b700 1 -- 172.21.2.112:0/1571625630 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2900095070 con 0x7f29001970d0 2021-02-10T11:36:10.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:10.079 7f2907c3b700 1 -- 172.21.2.112:0/1571625630 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2900095070 con 0x7f29001970d0 2021-02-10T11:36:10.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:10.080 7f28f67fc700 1 -- 172.21.2.112:0/1571625630 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f28f00054a0 con 0x7f29001970d0 2021-02-10T11:36:10.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:10.080 7f28f67fc700 1 --2- 172.21.2.112:0/1571625630 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f28e80186d0 0x7f28e801abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:10.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:10.080 7f28f67fc700 1 -- 172.21.2.112:0/1571625630 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f28f0032680 con 0x7f29001970d0 2021-02-10T11:36:10.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:10.080 7f2905a57700 1 --2- 172.21.2.112:0/1571625630 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f28e80186d0 0x7f28e801abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:10.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:10.081 7f2905a57700 1 --2- 172.21.2.112:0/1571625630 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f28e80186d0 0x7f28e801abd0 crc :-1 s=READY pgs=765 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:10.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:10.110 7f2907c3b700 1 -- 172.21.2.112:0/1571625630 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29000ad790 con 0x7f29001970d0 2021-02-10T11:36:10.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:10.113 7f28f67fc700 1 -- 172.21.2.112:0/1571625630 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f28f0006270 con 0x7f29001970d0 2021-02-10T11:36:10.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:10.243 7f2907c3b700 1 -- 172.21.2.112:0/1571625630 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f29000ad790 con 0x7f29001970d0 2021-02-10T11:36:10.230 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:10.243 7f28f67fc700 1 -- 172.21.2.112:0/1571625630 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+2101 (secure 0 0 0) 0x7f28f000bd40 con 0x7f29001970d0 2021-02-10T11:36:10.231 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:36:10.231 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":56,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8820,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/3613517698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":3613517698},{"type":"v1","addr":"172.21.2.112:6833","nonce":3613517698}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":52}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:36:00.289087","modified":"2021-02-10 11:36:02.295150","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7087},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7087":{"gid":7087,"name":"b","rank":0,"incarnation":53,"state":"up:active","state_seq":3,"addr":"172.21.2.120:6835/1870802251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":1870802251},{"type":"v1","addr":"172.21.2.120:6835","nonce":1870802251}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-02-10T11:36:10.231 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 56 2021-02-10T11:36:10.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:10.245 7f2907c3b700 1 -- 172.21.2.112:0/1571625630 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f28e80186d0 msgr2=0x7f28e801abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:10.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:10.245 7f2907c3b700 1 --2- 172.21.2.112:0/1571625630 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f28e80186d0 0x7f28e801abd0 crc :-1 s=READY pgs=765 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:10.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:10.245 7f2907c3b700 1 -- 172.21.2.112:0/1571625630 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f29001970d0 msgr2=0x7f29001974d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:10.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:10.245 7f2907c3b700 1 --2- 172.21.2.112:0/1571625630 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f29001970d0 0x7f29001974d0 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 rx=0x7f28f000b5b0 tx=0x7f28f000b950).stop 2021-02-10T11:36:10.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:10.245 7f2907c3b700 1 -- 172.21.2.112:0/1571625630 shutdown_connections 2021-02-10T11:36:10.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:10.245 7f2907c3b700 1 --2- 172.21.2.112:0/1571625630 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f28e80186d0 0x7f28e801abd0 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:10.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:10.245 7f2907c3b700 1 --2- 172.21.2.112:0/1571625630 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2900094650 0x7f2900094a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:10.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:10.245 7f2907c3b700 1 --2- 172.21.2.112:0/1571625630 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f29001970d0 0x7f29001974d0 unknown :-1 s=CLOSED pgs=1587 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:10.233 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:10.245 7f2907c3b700 1 --2- 172.21.2.112:0/1571625630 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f290018f0a0 0x7f2900196ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:10.233 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:10.245 7f2907c3b700 1 -- 172.21.2.112:0/1571625630 >> 172.21.2.112:0/1571625630 conn(0x7f29001a38c0 msgr2=0x7f29001a2f00 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:10.233 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:10.246 7f2907c3b700 1 -- 172.21.2.112:0/1571625630 shutdown_connections 2021-02-10T11:36:10.233 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:10.246 7f2907c3b700 1 -- 172.21.2.112:0/1571625630 wait complete. 2021-02-10T11:36:10.442 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 56, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:36:00.289087', 'modified': '2021-02-10 11:36:02.295150', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7087}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7087': {'gid': 7087, 'name': 'b', 'rank': 0, 'incarnation': 53, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.120:6835/1870802251', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.120:6834', 'nonce': 1870802251}, {'type': 'v1', 'addr': '172.21.2.120:6835', 'nonce': 1870802251}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:36:10.443 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:36:10.443 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:36:10.841 INFO:teuthology.orchestra.run.gibba020.stdout:{ 2021-02-10T11:36:10.841 INFO:teuthology.orchestra.run.gibba020.stdout: "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", 2021-02-10T11:36:10.841 INFO:teuthology.orchestra.run.gibba020.stdout: "whoami": 0, 2021-02-10T11:36:10.841 INFO:teuthology.orchestra.run.gibba020.stdout: "id": 7087, 2021-02-10T11:36:10.842 INFO:teuthology.orchestra.run.gibba020.stdout: "want_state": "up:active", 2021-02-10T11:36:10.842 INFO:teuthology.orchestra.run.gibba020.stdout: "state": "up:active", 2021-02-10T11:36:10.842 INFO:teuthology.orchestra.run.gibba020.stdout: "rank_uptime": 10.563674516000001, 2021-02-10T11:36:10.842 INFO:teuthology.orchestra.run.gibba020.stdout: "mdsmap_epoch": 56, 2021-02-10T11:36:10.842 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch": 73, 2021-02-10T11:36:10.842 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch_barrier": 73, 2021-02-10T11:36:10.842 INFO:teuthology.orchestra.run.gibba020.stdout: "uptime": 16.216166951000002 2021-02-10T11:36:10.842 INFO:teuthology.orchestra.run.gibba020.stdout:} 2021-02-10T11:36:10.846 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", "whoami": 0, "id": 7087, "want_state": "up:active", "state": "up:active", "rank_uptime": 10.563674516000001, "mdsmap_epoch": 56, "osdmap_epoch": 73, "osdmap_epoch_barrier": 73, "uptime": 16.216166951000002 } 2021-02-10T11:36:10.846 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:36:10.846 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2021-02-10T11:36:10.846 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@gibba012.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2021-02-10T11:36:10.846 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:10.846 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:36:11.181 DEBUG:teuthology.orchestra.run.gibba012:> sudo modprobe fuse 2021-02-10T11:36:11.525 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:36:11.525 DEBUG:teuthology.orchestra.run.gibba012:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:36:11.868 INFO:teuthology.orchestra.run.gibba012.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:36:11.869 INFO:teuthology.orchestra.run.gibba012.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:36:11.869 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:36:11.870 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:11.870 DEBUG:teuthology.orchestra.run.gibba012:> ls /sys/fs/fuse/connections 2021-02-10T11:36:12.204 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2021-02-10T11:36:12.204 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:36:12.207 DEBUG:teuthology.orchestra.run.gibba012:> sudo modprobe fuse 2021-02-10T11:36:12.550 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:36:12.550 DEBUG:teuthology.orchestra.run.gibba012:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:36:12.615 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba012.stderr:ceph-fuse[141138]: starting ceph client 2021-02-10T11:36:12.616 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba012.stderr:2021-02-10 11:36:12.629 7fb5d79c8f80 -1 init, newargv = 0x5635630111a0 newargc=9 2021-02-10T11:36:12.622 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba012.stderr:ceph-fuse[141138]: starting fuse 2021-02-10T11:36:12.892 INFO:teuthology.orchestra.run.gibba012.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:36:12.893 INFO:teuthology.orchestra.run.gibba012.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:36:12.893 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:36:12.894 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:12.894 DEBUG:teuthology.orchestra.run.gibba012:> ls /sys/fs/fuse/connections 2021-02-10T11:36:13.229 INFO:teuthology.orchestra.run.gibba012.stdout:42 2021-02-10T11:36:13.229 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2021-02-10T11:36:13.229 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:13.229 DEBUG:teuthology.orchestra.run.gibba012:> sudo python3 -c ' 2021-02-10T11:36:13.229 DEBUG:teuthology.orchestra.run.gibba012:> import glob 2021-02-10T11:36:13.230 DEBUG:teuthology.orchestra.run.gibba012:> import re 2021-02-10T11:36:13.230 DEBUG:teuthology.orchestra.run.gibba012:> import os 2021-02-10T11:36:13.230 DEBUG:teuthology.orchestra.run.gibba012:> import subprocess 2021-02-10T11:36:13.230 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:36:13.230 DEBUG:teuthology.orchestra.run.gibba012:> def find_socket(client_name): 2021-02-10T11:36:13.230 DEBUG:teuthology.orchestra.run.gibba012:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2021-02-10T11:36:13.230 DEBUG:teuthology.orchestra.run.gibba012:> files = glob.glob(asok_path) 2021-02-10T11:36:13.230 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:36:13.230 DEBUG:teuthology.orchestra.run.gibba012:> # Given a non-glob path, it better be there 2021-02-10T11:36:13.230 DEBUG:teuthology.orchestra.run.gibba012:> if "*" not in asok_path: 2021-02-10T11:36:13.230 DEBUG:teuthology.orchestra.run.gibba012:> assert(len(files) == 1) 2021-02-10T11:36:13.230 DEBUG:teuthology.orchestra.run.gibba012:> return files[0] 2021-02-10T11:36:13.231 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:36:13.231 DEBUG:teuthology.orchestra.run.gibba012:> for f in files: 2021-02-10T11:36:13.231 DEBUG:teuthology.orchestra.run.gibba012:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:36:13.231 DEBUG:teuthology.orchestra.run.gibba012:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:36:13.231 DEBUG:teuthology.orchestra.run.gibba012:> return f 2021-02-10T11:36:13.231 DEBUG:teuthology.orchestra.run.gibba012:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:36:13.231 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:36:13.231 DEBUG:teuthology.orchestra.run.gibba012:> print(find_socket("client.0")) 2021-02-10T11:36:13.231 DEBUG:teuthology.orchestra.run.gibba012:> ' 2021-02-10T11:36:13.591 INFO:teuthology.orchestra.run.gibba012.stdout:/var/run/ceph/ceph-client.0.141138.asok 2021-02-10T11:36:13.594 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.141138.asok 2021-02-10T11:36:13.594 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:13.594 DEBUG:teuthology.orchestra.run.gibba012:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.141138.asok status 2021-02-10T11:36:13.982 INFO:teuthology.orchestra.run.gibba012.stdout:{ 2021-02-10T11:36:13.983 INFO:teuthology.orchestra.run.gibba012.stdout: "metadata": { 2021-02-10T11:36:13.983 INFO:teuthology.orchestra.run.gibba012.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:36:13.983 INFO:teuthology.orchestra.run.gibba012.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:36:13.983 INFO:teuthology.orchestra.run.gibba012.stdout: "entity_id": "0", 2021-02-10T11:36:13.983 INFO:teuthology.orchestra.run.gibba012.stdout: "hostname": "gibba012", 2021-02-10T11:36:13.983 INFO:teuthology.orchestra.run.gibba012.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2021-02-10T11:36:13.983 INFO:teuthology.orchestra.run.gibba012.stdout: "pid": "141138", 2021-02-10T11:36:13.983 INFO:teuthology.orchestra.run.gibba012.stdout: "root": "/" 2021-02-10T11:36:13.983 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:36:13.983 INFO:teuthology.orchestra.run.gibba012.stdout: "dentry_count": 0, 2021-02-10T11:36:13.984 INFO:teuthology.orchestra.run.gibba012.stdout: "dentry_pinned_count": 0, 2021-02-10T11:36:13.984 INFO:teuthology.orchestra.run.gibba012.stdout: "id": 8985, 2021-02-10T11:36:13.984 INFO:teuthology.orchestra.run.gibba012.stdout: "inst": { 2021-02-10T11:36:13.984 INFO:teuthology.orchestra.run.gibba012.stdout: "name": { 2021-02-10T11:36:13.984 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "client", 2021-02-10T11:36:13.984 INFO:teuthology.orchestra.run.gibba012.stdout: "num": 8985 2021-02-10T11:36:13.984 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:36:13.984 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": { 2021-02-10T11:36:13.984 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:36:13.984 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:0", 2021-02-10T11:36:13.984 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 890297597 2021-02-10T11:36:13.984 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:36:13.984 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:36:13.985 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": { 2021-02-10T11:36:13.985 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:36:13.985 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:0", 2021-02-10T11:36:13.985 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 890297597 2021-02-10T11:36:13.985 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:36:13.985 INFO:teuthology.orchestra.run.gibba012.stdout: "inst_str": "client.8985 172.21.2.112:0/890297597", 2021-02-10T11:36:13.985 INFO:teuthology.orchestra.run.gibba012.stdout: "addr_str": "172.21.2.112:0/890297597", 2021-02-10T11:36:13.985 INFO:teuthology.orchestra.run.gibba012.stdout: "inode_count": 1, 2021-02-10T11:36:13.985 INFO:teuthology.orchestra.run.gibba012.stdout: "mds_epoch": 56, 2021-02-10T11:36:13.985 INFO:teuthology.orchestra.run.gibba012.stdout: "osd_epoch": 73, 2021-02-10T11:36:13.985 INFO:teuthology.orchestra.run.gibba012.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:36:13.985 INFO:teuthology.orchestra.run.gibba012.stdout: "blacklisted": false 2021-02-10T11:36:13.985 INFO:teuthology.orchestra.run.gibba012.stdout:} 2021-02-10T11:36:13.987 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:13.987 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:36:13.987 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:36:14.322 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:36:14.323 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:36:14.323 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:14.323 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:36:14.668 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:36:15.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.093 7f7d85ad1700 1 Processor -- start 2021-02-10T11:36:15.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.093 7f7d85ad1700 1 -- start start 2021-02-10T11:36:15.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.094 7f7d85ad1700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7d800d1870 0x7f7d801b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:15.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.094 7f7d85ad1700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d800f0b10 con 0x7f7d801b0a10 2021-02-10T11:36:15.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.094 7f7d85ad1700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d800d0310 con 0x7f7d800f0650 2021-02-10T11:36:15.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.094 7f7d85ad1700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d800e5ed0 con 0x7f7d800d1870 2021-02-10T11:36:15.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.094 7f7d7effd700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f7d801b0a10 0x7f7d800cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:51332/0 (socket says 172.21.2.112:51332) 2021-02-10T11:36:15.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.094 7f7d7effd700 1 -- 172.21.2.112:0/2145899438 learned_addr learned my addr 172.21.2.112:0/2145899438 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:36:15.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.094 7f7d7e7fc700 1 -- 172.21.2.112:0/2145899438 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1122837812 0 0) 0x7f7d680010b0 con 0x7f7d801b0a10 2021-02-10T11:36:15.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.094 7f7d7e7fc700 1 -- 172.21.2.112:0/2145899438 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d6c002fe0 con 0x7f7d801b0a10 2021-02-10T11:36:15.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.094 7f7d7f7fe700 1 --2- 172.21.2.112:0/2145899438 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7d800d1870 0x7f7d801b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:15.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.095 7f7d7e7fc700 1 -- 172.21.2.112:0/2145899438 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 309736095 0 0) 0x7f7d680010b0 con 0x7f7d801b0a10 2021-02-10T11:36:15.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.095 7f7d7e7fc700 1 -- 172.21.2.112:0/2145899438 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7d800d1870 msgr2=0x7f7d801b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:15.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.095 7f7d7e7fc700 1 --2- 172.21.2.112:0/2145899438 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7d800d1870 0x7f7d801b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:15.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.095 7f7d7e7fc700 1 -- 172.21.2.112:0/2145899438 >> v1:172.21.2.120:6789/0 conn(0x7f7d800f0650 legacy=0x7f7d800cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:15.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.095 7f7d7e7fc700 1 -- 172.21.2.112:0/2145899438 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d800d0530 con 0x7f7d801b0a10 2021-02-10T11:36:15.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.095 7f7d7e7fc700 1 -- 172.21.2.112:0/2145899438 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f7d680015b0 con 0x7f7d801b0a10 2021-02-10T11:36:15.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.095 7f7d7e7fc700 1 -- 172.21.2.112:0/2145899438 >> v1:172.21.2.112:6789/0 conn(0x7f7d801b0a10 legacy=0x7f7d800cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:15.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.095 7f7d7e7fc700 1 --2- 172.21.2.112:0/2145899438 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7d6c003a60 0x7f7d6c005eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:15.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.095 7f7d7e7fc700 1 --2- 172.21.2.112:0/2145899438 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7d6c006750 0x7f7d6c008c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:15.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.095 7f7d7e7fc700 1 --2- 172.21.2.112:0/2145899438 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7d6c0094f0 0x7f7d6c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:15.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.095 7f7d7e7fc700 1 -- 172.21.2.112:0/2145899438 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d6c00c2c0 con 0x7f7d6c006750 2021-02-10T11:36:15.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.095 7f7d7e7fc700 1 -- 172.21.2.112:0/2145899438 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d6c00c4e0 con 0x7f7d6c0094f0 2021-02-10T11:36:15.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.095 7f7d7e7fc700 1 -- 172.21.2.112:0/2145899438 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d6c00c700 con 0x7f7d6c003a60 2021-02-10T11:36:15.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.095 7f7d7effd700 1 --2- 172.21.2.112:0/2145899438 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7d6c006750 0x7f7d6c008c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:15.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.095 7f7d7f7fe700 1 --2- 172.21.2.112:0/2145899438 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7d6c003a60 0x7f7d6c005eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:15.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.095 7f7d7effd700 1 -- 172.21.2.112:0/2145899438 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7d6c003a60 msgr2=0x7f7d6c005eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:15.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.095 7f7d7effd700 1 --2- 172.21.2.112:0/2145899438 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7d6c003a60 0x7f7d6c005eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:15.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.095 7f7d7effd700 1 -- 172.21.2.112:0/2145899438 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7d6c0094f0 msgr2=0x7f7d6c00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:15.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.095 7f7d7ffff700 1 --2- 172.21.2.112:0/2145899438 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7d6c0094f0 0x7f7d6c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:15.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.095 7f7d7effd700 1 --2- 172.21.2.112:0/2145899438 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7d6c0094f0 0x7f7d6c00ba10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:15.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.095 7f7d7effd700 1 -- 172.21.2.112:0/2145899438 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d6c00c9a0 con 0x7f7d6c006750 2021-02-10T11:36:15.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.095 7f7d7effd700 1 --2- 172.21.2.112:0/2145899438 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7d6c006750 0x7f7d6c008c50 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 rx=0x7f7d68000b10 tx=0x7f7d68007d60).ready entity=mon.0 client_cookie=bedef1b742259685 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:15.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.095 7f7d7e7fc700 1 -- 172.21.2.112:0/2145899438 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7d6804e320 con 0x7f7d6c006750 2021-02-10T11:36:15.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.095 7f7d7e7fc700 1 -- 172.21.2.112:0/2145899438 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d6804e6a0 con 0x7f7d6c006750 2021-02-10T11:36:15.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.095 7f7d7e7fc700 1 -- 172.21.2.112:0/2145899438 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7d68001010 con 0x7f7d6c006750 2021-02-10T11:36:15.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.095 7f7d85ad1700 1 -- 172.21.2.112:0/2145899438 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7d6c006750 msgr2=0x7f7d6c008c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:15.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.095 7f7d85ad1700 1 --2- 172.21.2.112:0/2145899438 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7d6c006750 0x7f7d6c008c50 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 rx=0x7f7d68000b10 tx=0x7f7d68007d60).stop 2021-02-10T11:36:15.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.095 7f7d7ffff700 1 -- 172.21.2.112:0/2145899438 reap_dead start 2021-02-10T11:36:15.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.096 7f7d85ad1700 1 -- 172.21.2.112:0/2145899438 shutdown_connections 2021-02-10T11:36:15.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.096 7f7d85ad1700 1 -- 172.21.2.112:0/2145899438 >> 172.21.2.112:0/2145899438 conn(0x7f7d800ce450 msgr2=0x7f7d800d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:15.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.096 7f7d85ad1700 1 -- 172.21.2.112:0/2145899438 shutdown_connections 2021-02-10T11:36:15.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.096 7f7d85ad1700 1 -- 172.21.2.112:0/2145899438 wait complete. 2021-02-10T11:36:15.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.111 7f7d85ad1700 1 Processor -- start 2021-02-10T11:36:15.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.111 7f7d85ad1700 1 -- start start 2021-02-10T11:36:15.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.111 7f7d85ad1700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7d800b25d0 0x7f7d800a8d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:15.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.111 7f7d85ad1700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7d800a9340 0x7f7d800a63b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:15.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.111 7f7d85ad1700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7d800a69d0 0x7f7d800a6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:15.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.111 7f7d85ad1700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d800b29d0 con 0x7f7d800b25d0 2021-02-10T11:36:15.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.111 7f7d85ad1700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d800a73f0 con 0x7f7d800a69d0 2021-02-10T11:36:15.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.111 7f7d85ad1700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d800a7870 con 0x7f7d800a9340 2021-02-10T11:36:15.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.111 7f7d7f7fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7d800b25d0 0x7f7d800a8d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:15.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.111 7f7d7f7fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7d800b25d0 0x7f7d800a8d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:51884/0 (socket says 172.21.2.112:51884) 2021-02-10T11:36:15.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.111 7f7d7f7fe700 1 -- 172.21.2.112:0/3681858133 learned_addr learned my addr 172.21.2.112:0/3681858133 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:36:15.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.112 7f7d7f7fe700 1 -- 172.21.2.112:0/3681858133 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7d800a9340 msgr2=0x7f7d800a63b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:15.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.112 7f7d7f7fe700 1 --2- 172.21.2.112:0/3681858133 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7d800a9340 0x7f7d800a63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:15.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.112 7f7d7f7fe700 1 -- 172.21.2.112:0/3681858133 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7d800a69d0 msgr2=0x7f7d800a6dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:15.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.112 7f7d7f7fe700 1 --2- 172.21.2.112:0/3681858133 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7d800a69d0 0x7f7d800a6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:15.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.112 7f7d7f7fe700 1 -- 172.21.2.112:0/3681858133 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d800a7a90 con 0x7f7d800b25d0 2021-02-10T11:36:15.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.112 7f7d7f7fe700 1 --2- 172.21.2.112:0/3681858133 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7d800b25d0 0x7f7d800a8d20 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 rx=0x7f7d70006f10 tx=0x7f7d7000b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:15.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.112 7f7d77fff700 1 -- 172.21.2.112:0/3681858133 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7d7000bd80 con 0x7f7d800b25d0 2021-02-10T11:36:15.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.112 7f7d77fff700 1 -- 172.21.2.112:0/3681858133 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d70000c90 con 0x7f7d800b25d0 2021-02-10T11:36:15.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.112 7f7d77fff700 1 -- 172.21.2.112:0/3681858133 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7d70002220 con 0x7f7d800b25d0 2021-02-10T11:36:15.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.112 7f7d85ad1700 1 -- 172.21.2.112:0/3681858133 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d800a7a90 con 0x7f7d800b25d0 2021-02-10T11:36:15.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.112 7f7d85ad1700 1 -- 172.21.2.112:0/3681858133 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d800a8020 con 0x7f7d800b25d0 2021-02-10T11:36:15.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.113 7f7d77fff700 1 -- 172.21.2.112:0/3681858133 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7d700052e0 con 0x7f7d800b25d0 2021-02-10T11:36:15.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.113 7f7d77fff700 1 --2- 172.21.2.112:0/3681858133 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7d640186c0 0x7f7d6401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:15.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.113 7f7d77fff700 1 -- 172.21.2.112:0/3681858133 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f7d70032550 con 0x7f7d800b25d0 2021-02-10T11:36:15.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.114 7f7d7effd700 1 --2- 172.21.2.112:0/3681858133 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7d640186c0 0x7f7d6401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:15.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.114 7f7d7effd700 1 --2- 172.21.2.112:0/3681858133 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7d640186c0 0x7f7d6401abc0 crc :-1 s=READY pgs=766 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:15.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.144 7f7d85ad1700 1 -- 172.21.2.112:0/3681858133 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d800ebc80 con 0x7f7d800b25d0 2021-02-10T11:36:15.132 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.146 7f7d77fff700 1 -- 172.21.2.112:0/3681858133 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7d70000940 con 0x7f7d800b25d0 2021-02-10T11:36:15.258 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.271 7f7d85ad1700 1 -- 172.21.2.112:0/3681858133 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7d800f0b10 con 0x7f7d800b25d0 2021-02-10T11:36:15.258 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.272 7f7d77fff700 1 -- 172.21.2.112:0/3681858133 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v73) v1 ==== 74+0+13996 (secure 0 0 0) 0x7f7d7001f2f0 con 0x7f7d800b25d0 2021-02-10T11:36:15.259 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:36:15.259 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":73,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:36:01.291077","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:35:58.263396","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":15,"pool_name":"cephfs_data","create_time":"2021-02-10 11:35:59.285157","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:36:15.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.273 7f7d85ad1700 1 -- 172.21.2.112:0/3681858133 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7d640186c0 msgr2=0x7f7d6401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:15.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.273 7f7d85ad1700 1 --2- 172.21.2.112:0/3681858133 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7d640186c0 0x7f7d6401abc0 crc :-1 s=READY pgs=766 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:15.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.273 7f7d85ad1700 1 -- 172.21.2.112:0/3681858133 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7d800b25d0 msgr2=0x7f7d800a8d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:15.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.273 7f7d85ad1700 1 --2- 172.21.2.112:0/3681858133 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7d800b25d0 0x7f7d800a8d20 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 rx=0x7f7d70006f10 tx=0x7f7d7000b990).stop 2021-02-10T11:36:15.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.274 7f7d85ad1700 1 -- 172.21.2.112:0/3681858133 shutdown_connections 2021-02-10T11:36:15.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.274 7f7d85ad1700 1 --2- 172.21.2.112:0/3681858133 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7d640186c0 0x7f7d6401abc0 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:15.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.274 7f7d85ad1700 1 --2- 172.21.2.112:0/3681858133 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7d800a69d0 0x7f7d800a6dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:15.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.274 7f7d85ad1700 1 --2- 172.21.2.112:0/3681858133 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7d800a9340 0x7f7d800a63b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:15.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.274 7f7d85ad1700 1 --2- 172.21.2.112:0/3681858133 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7d800b25d0 0x7f7d800a8d20 unknown :-1 s=CLOSED pgs=1593 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:15.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.274 7f7d85ad1700 1 -- 172.21.2.112:0/3681858133 >> 172.21.2.112:0/3681858133 conn(0x7f7d800bbb50 msgr2=0x7f7d800ad8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:15.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.274 7f7d85ad1700 1 -- 172.21.2.112:0/3681858133 shutdown_connections 2021-02-10T11:36:15.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.274 7f7d85ad1700 1 -- 172.21.2.112:0/3681858133 wait complete. 2021-02-10T11:36:15.269 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:13:57.508758', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 14, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:35:58.263396', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '72', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 15, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:35:59.285157', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '73', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:36:15.269 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:36:15.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.698 7f596e38f700 1 Processor -- start 2021-02-10T11:36:15.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.698 7f596e38f700 1 -- start start 2021-02-10T11:36:15.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.698 7f596e38f700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f59680d1870 0x7f59681b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:15.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.698 7f596e38f700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f59681b0ae0 0x7f5968108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:15.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.698 7f596e38f700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5968109a60 con 0x7f59680d1c70 2021-02-10T11:36:15.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.698 7f596e38f700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59680e5ef0 con 0x7f59681b0ae0 2021-02-10T11:36:15.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.698 7f596e38f700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5968109750 con 0x7f59680d1870 2021-02-10T11:36:15.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.699 7f59677fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f59681b0ae0 0x7f5968108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:15.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.699 7f59677fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f59681b0ae0 0x7f5968108940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:51890/0 (socket says 172.21.2.112:51890) 2021-02-10T11:36:15.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.699 7f59677fe700 1 -- 172.21.2.112:0/2030785566 learned_addr learned my addr 172.21.2.112:0/2030785566 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:36:15.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.699 7f5967fff700 1 --2- 172.21.2.112:0/2030785566 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f59680d1870 0x7f59681b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:15.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.699 7f59677fe700 1 -- 172.21.2.112:0/2030785566 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f59680d1870 msgr2=0x7f59681b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:15.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.699 7f59677fe700 1 --2- 172.21.2.112:0/2030785566 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f59680d1870 0x7f59681b04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:15.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.699 7f59677fe700 1 -- 172.21.2.112:0/2030785566 >> v1:172.21.2.112:6789/0 conn(0x7f59680d1c70 legacy=0x7f5968108f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:15.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.699 7f59677fe700 1 -- 172.21.2.112:0/2030785566 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5968088070 con 0x7f59681b0ae0 2021-02-10T11:36:15.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.699 7f59677fe700 1 --2- 172.21.2.112:0/2030785566 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f59681b0ae0 0x7f5968108940 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 rx=0x7f595c006700 tx=0x7f595c009b80).ready entity=mon.0 client_cookie=abed4d5f2132b801 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:15.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.700 7f5966ffd700 1 -- 172.21.2.112:0/2030785566 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f595c00c0c0 con 0x7f59681b0ae0 2021-02-10T11:36:15.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.700 7f5966ffd700 1 -- 172.21.2.112:0/2030785566 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f595c00c440 con 0x7f59681b0ae0 2021-02-10T11:36:15.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.700 7f5966ffd700 1 -- 172.21.2.112:0/2030785566 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f595c00c960 con 0x7f59681b0ae0 2021-02-10T11:36:15.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.700 7f596e38f700 1 -- 172.21.2.112:0/2030785566 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f59681b0ae0 msgr2=0x7f5968108940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:15.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.700 7f596e38f700 1 --2- 172.21.2.112:0/2030785566 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f59681b0ae0 0x7f5968108940 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 rx=0x7f595c006700 tx=0x7f595c009b80).stop 2021-02-10T11:36:15.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.700 7f596e38f700 1 -- 172.21.2.112:0/2030785566 shutdown_connections 2021-02-10T11:36:15.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.700 7f596e38f700 1 --2- 172.21.2.112:0/2030785566 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f59681b0ae0 0x7f5968108940 unknown :-1 s=CLOSED pgs=1595 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:15.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.700 7f596e38f700 1 --2- 172.21.2.112:0/2030785566 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f59680d1870 0x7f59681b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:15.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.700 7f596e38f700 1 -- 172.21.2.112:0/2030785566 >> 172.21.2.112:0/2030785566 conn(0x7f59680ce420 msgr2=0x7f59680d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:15.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.700 7f596e38f700 1 -- 172.21.2.112:0/2030785566 shutdown_connections 2021-02-10T11:36:15.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.700 7f596e38f700 1 -- 172.21.2.112:0/2030785566 wait complete. 2021-02-10T11:36:15.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.715 7f596e38f700 1 Processor -- start 2021-02-10T11:36:15.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.715 7f596e38f700 1 -- start start 2021-02-10T11:36:15.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.715 7f596e38f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f59680abd60 0x7f59680ac160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:15.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.715 7f596e38f700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f59680ac780 0x7f59680acb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:15.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.715 7f596e38f700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5968083910 0x7f5968083d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:15.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.715 7f596e38f700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5968087180 con 0x7f59680abd60 2021-02-10T11:36:15.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.715 7f596e38f700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f596818fcd0 con 0x7f5968083910 2021-02-10T11:36:15.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.715 7f596e38f700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5968084330 con 0x7f59680ac780 2021-02-10T11:36:15.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.716 7f5967fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f59680abd60 0x7f59680ac160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:15.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.716 7f5967fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f59680abd60 0x7f59680ac160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:51896/0 (socket says 172.21.2.112:51896) 2021-02-10T11:36:15.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.716 7f5967fff700 1 -- 172.21.2.112:0/1325460538 learned_addr learned my addr 172.21.2.112:0/1325460538 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:36:15.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.716 7f5967fff700 1 -- 172.21.2.112:0/1325460538 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f59680ac780 msgr2=0x7f59680acb80 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:36:15.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.716 7f5967fff700 1 --2- 172.21.2.112:0/1325460538 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f59680ac780 0x7f59680acb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:15.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.716 7f5967fff700 1 -- 172.21.2.112:0/1325460538 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5968083910 msgr2=0x7f5968083d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:15.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.716 7f5967fff700 1 --2- 172.21.2.112:0/1325460538 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5968083910 0x7f5968083d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:15.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.716 7f5967fff700 1 -- 172.21.2.112:0/1325460538 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5968084550 con 0x7f59680abd60 2021-02-10T11:36:15.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.716 7f5967fff700 1 --2- 172.21.2.112:0/1325460538 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f59680abd60 0x7f59680ac160 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 rx=0x7f5958006f60 tx=0x7f595800b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:15.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.716 7f5964ff9700 1 -- 172.21.2.112:0/1325460538 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f595800bd60 con 0x7f59680abd60 2021-02-10T11:36:15.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.716 7f5964ff9700 1 -- 172.21.2.112:0/1325460538 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5958000c90 con 0x7f59680abd60 2021-02-10T11:36:15.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.716 7f5964ff9700 1 -- 172.21.2.112:0/1325460538 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f59580022b0 con 0x7f59680abd60 2021-02-10T11:36:15.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.716 7f596e38f700 1 -- 172.21.2.112:0/1325460538 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5968084550 con 0x7f59680abd60 2021-02-10T11:36:15.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.717 7f596e38f700 1 -- 172.21.2.112:0/1325460538 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5968084550 con 0x7f59680abd60 2021-02-10T11:36:15.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.717 7f5964ff9700 1 -- 172.21.2.112:0/1325460538 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5958006b00 con 0x7f59680abd60 2021-02-10T11:36:15.707 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.717 7f5964ff9700 1 --2- 172.21.2.112:0/1325460538 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f59480186c0 0x7f594801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:15.707 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.717 7f5964ff9700 1 -- 172.21.2.112:0/1325460538 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f5958032550 con 0x7f59680abd60 2021-02-10T11:36:15.707 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.718 7f59677fe700 1 --2- 172.21.2.112:0/1325460538 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f59480186c0 0x7f594801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:15.707 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.718 7f59677fe700 1 --2- 172.21.2.112:0/1325460538 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f59480186c0 0x7f594801abc0 crc :-1 s=READY pgs=767 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:15.734 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.748 7f596e38f700 1 -- 172.21.2.112:0/1325460538 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5968082980 con 0x7f59680abd60 2021-02-10T11:36:15.737 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.750 7f5964ff9700 1 -- 172.21.2.112:0/1325460538 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5958030d00 con 0x7f59680abd60 2021-02-10T11:36:15.865 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.879 7f596e38f700 1 -- 172.21.2.112:0/1325460538 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f5968109a60 con 0x7f59680abd60 2021-02-10T11:36:15.866 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.879 7f5964ff9700 1 -- 172.21.2.112:0/1325460538 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v73) v1 ==== 92+0+10 (secure 0 0 0) 0x7f595800bd60 con 0x7f59680abd60 2021-02-10T11:36:15.866 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:36:15.868 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.881 7f596e38f700 1 -- 172.21.2.112:0/1325460538 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f59480186c0 msgr2=0x7f594801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:15.868 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.881 7f596e38f700 1 --2- 172.21.2.112:0/1325460538 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f59480186c0 0x7f594801abc0 crc :-1 s=READY pgs=767 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:15.868 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.881 7f596e38f700 1 -- 172.21.2.112:0/1325460538 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f59680abd60 msgr2=0x7f59680ac160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:15.868 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.881 7f596e38f700 1 --2- 172.21.2.112:0/1325460538 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f59680abd60 0x7f59680ac160 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 rx=0x7f5958006f60 tx=0x7f595800b970).stop 2021-02-10T11:36:15.868 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.881 7f596e38f700 1 -- 172.21.2.112:0/1325460538 shutdown_connections 2021-02-10T11:36:15.868 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.881 7f596e38f700 1 --2- 172.21.2.112:0/1325460538 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f59480186c0 0x7f594801abc0 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:15.868 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.881 7f596e38f700 1 --2- 172.21.2.112:0/1325460538 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5968083910 0x7f5968083d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:15.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.881 7f596e38f700 1 --2- 172.21.2.112:0/1325460538 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f59680ac780 0x7f59680acb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:15.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.881 7f596e38f700 1 --2- 172.21.2.112:0/1325460538 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f59680abd60 0x7f59680ac160 unknown :-1 s=CLOSED pgs=1596 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:15.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.882 7f596e38f700 1 -- 172.21.2.112:0/1325460538 >> 172.21.2.112:0/1325460538 conn(0x7f59680a2b30 msgr2=0x7f596809ce60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:15.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.882 7f596e38f700 1 -- 172.21.2.112:0/1325460538 shutdown_connections 2021-02-10T11:36:15.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:15.882 7f596e38f700 1 -- 172.21.2.112:0/1325460538 wait complete. 2021-02-10T11:36:15.877 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:36:16.307 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.305 7f1eb40c2700 1 Processor -- start 2021-02-10T11:36:16.307 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.305 7f1eb40c2700 1 -- start start 2021-02-10T11:36:16.307 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.305 7f1eb40c2700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f1eac0d1870 0x7f1eac1b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:16.307 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.305 7f1eb40c2700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1eac1b0ae0 0x7f1eac108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:16.307 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.305 7f1eb40c2700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f1eac0d1c70 0x7f1eac108f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:16.308 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.305 7f1eb40c2700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1eac0fe1c0 con 0x7f1eac0d1c70 2021-02-10T11:36:16.308 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.305 7f1eb40c2700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1eac109a60 con 0x7f1eac0d1870 2021-02-10T11:36:16.308 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.305 7f1eb40c2700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1eac088070 con 0x7f1eac1b0ae0 2021-02-10T11:36:16.308 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.305 7f1eb26df700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f1eac0d1c70 0x7f1eac108f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:16.308 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.305 7f1eb26df700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f1eac0d1c70 0x7f1eac108f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:51902/0 (socket says 172.21.2.112:51902) 2021-02-10T11:36:16.308 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.305 7f1eb26df700 1 -- 172.21.2.112:0/2329511630 learned_addr learned my addr 172.21.2.112:0/2329511630 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:36:16.308 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.306 7f1eb1ede700 1 --2- 172.21.2.112:0/2329511630 >> v2:172.21.2.120:3300/0 conn(0x7f1eac0d1870 0x7f1eac1b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:16.308 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.306 7f1eb26df700 1 -- 172.21.2.112:0/2329511630 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1eac1b0ae0 msgr2=0x7f1eac108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:16.308 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.306 7f1eb16dd700 1 --2- 172.21.2.112:0/2329511630 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1eac1b0ae0 0x7f1eac108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:16.308 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.306 7f1eb26df700 1 --2- 172.21.2.112:0/2329511630 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1eac1b0ae0 0x7f1eac108940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:16.308 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.306 7f1eb26df700 1 -- 172.21.2.112:0/2329511630 >> v2:172.21.2.120:3300/0 conn(0x7f1eac0d1870 msgr2=0x7f1eac1b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:16.308 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.306 7f1eb26df700 1 --2- 172.21.2.112:0/2329511630 >> v2:172.21.2.120:3300/0 conn(0x7f1eac0d1870 0x7f1eac1b04c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:16.308 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.306 7f1eb26df700 1 -- 172.21.2.112:0/2329511630 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1eac109580 con 0x7f1eac0d1c70 2021-02-10T11:36:16.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.306 7f1eb26df700 1 --2- 172.21.2.112:0/2329511630 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1eac0d1c70 0x7f1eac108f60 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 rx=0x7f1ea4006f00 tx=0x7f1ea4009f10).ready entity=mon.0 client_cookie=d67923b8fc4f6af6 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:16.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.306 7f1eb0edc700 1 -- 172.21.2.112:0/2329511630 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1ea4003030 con 0x7f1eac0d1c70 2021-02-10T11:36:16.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.306 7f1eb0edc700 1 -- 172.21.2.112:0/2329511630 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ea400b3b0 con 0x7f1eac0d1c70 2021-02-10T11:36:16.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.306 7f1eb0edc700 1 -- 172.21.2.112:0/2329511630 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1ea400b8d0 con 0x7f1eac0d1c70 2021-02-10T11:36:16.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.306 7f1eb40c2700 1 -- 172.21.2.112:0/2329511630 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1eac0d1c70 msgr2=0x7f1eac108f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:16.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.306 7f1eb40c2700 1 --2- 172.21.2.112:0/2329511630 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1eac0d1c70 0x7f1eac108f60 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 rx=0x7f1ea4006f00 tx=0x7f1ea4009f10).stop 2021-02-10T11:36:16.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.306 7f1eb40c2700 1 -- 172.21.2.112:0/2329511630 shutdown_connections 2021-02-10T11:36:16.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.306 7f1eb40c2700 1 --2- 172.21.2.112:0/2329511630 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1eac0d1c70 0x7f1eac108f60 unknown :-1 s=CLOSED pgs=1597 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:16.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.306 7f1eb40c2700 1 --2- 172.21.2.112:0/2329511630 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1eac1b0ae0 0x7f1eac108940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:16.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.306 7f1eb40c2700 1 --2- 172.21.2.112:0/2329511630 >> v2:172.21.2.120:3300/0 conn(0x7f1eac0d1870 0x7f1eac1b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:16.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.306 7f1eb40c2700 1 -- 172.21.2.112:0/2329511630 >> 172.21.2.112:0/2329511630 conn(0x7f1eac0ce420 msgr2=0x7f1eac0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:16.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.307 7f1eb40c2700 1 -- 172.21.2.112:0/2329511630 shutdown_connections 2021-02-10T11:36:16.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.307 7f1eb40c2700 1 -- 172.21.2.112:0/2329511630 wait complete. 2021-02-10T11:36:16.311 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.321 7f1eb40c2700 1 Processor -- start 2021-02-10T11:36:16.311 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.321 7f1eb40c2700 1 -- start start 2021-02-10T11:36:16.311 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.321 7f1eb40c2700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1eac1a7b70 0x7f1eac1a7f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:16.311 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.321 7f1eb40c2700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1eac0ae250 0x7f1eac0ae650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:16.311 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.321 7f1eb40c2700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1eac0aec70 0x7f1eac0af070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:16.311 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.321 7f1eb40c2700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1eac0adcd0 con 0x7f1eac0ae250 2021-02-10T11:36:16.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.321 7f1eb40c2700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1eac0b7e00 con 0x7f1eac0aec70 2021-02-10T11:36:16.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.321 7f1eb40c2700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1eac0ac4b0 con 0x7f1eac1a7b70 2021-02-10T11:36:16.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.321 7f1eb16dd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1eac0ae250 0x7f1eac0ae650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:16.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.321 7f1eb16dd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1eac0ae250 0x7f1eac0ae650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:51908/0 (socket says 172.21.2.112:51908) 2021-02-10T11:36:16.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.321 7f1eb16dd700 1 -- 172.21.2.112:0/1547918558 learned_addr learned my addr 172.21.2.112:0/1547918558 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:36:16.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.322 7f1eb1ede700 1 --2- 172.21.2.112:0/1547918558 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1eac1a7b70 0x7f1eac1a7f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:16.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.322 7f1eb26df700 1 --2- 172.21.2.112:0/1547918558 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1eac0aec70 0x7f1eac0af070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:16.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.322 7f1eb16dd700 1 -- 172.21.2.112:0/1547918558 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1eac1a7b70 msgr2=0x7f1eac1a7f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:16.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.322 7f1eb16dd700 1 --2- 172.21.2.112:0/1547918558 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1eac1a7b70 0x7f1eac1a7f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:16.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.322 7f1eb16dd700 1 -- 172.21.2.112:0/1547918558 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1eac0aec70 msgr2=0x7f1eac0af070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:16.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.322 7f1eb16dd700 1 --2- 172.21.2.112:0/1547918558 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1eac0aec70 0x7f1eac0af070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:16.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.322 7f1eb16dd700 1 -- 172.21.2.112:0/1547918558 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1eac0ac6d0 con 0x7f1eac0ae250 2021-02-10T11:36:16.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.322 7f1eb26df700 1 --2- 172.21.2.112:0/1547918558 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1eac0aec70 0x7f1eac0af070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:36:16.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.322 7f1eb16dd700 1 --2- 172.21.2.112:0/1547918558 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1eac0ae250 0x7f1eac0ae650 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 rx=0x7f1ea0006f10 tx=0x7f1ea000b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:16.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.322 7f1e9e7fc700 1 -- 172.21.2.112:0/1547918558 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1ea000bd80 con 0x7f1eac0ae250 2021-02-10T11:36:16.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.322 7f1e9e7fc700 1 -- 172.21.2.112:0/1547918558 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ea0000c90 con 0x7f1eac0ae250 2021-02-10T11:36:16.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.322 7f1e9e7fc700 1 -- 172.21.2.112:0/1547918558 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1ea0002220 con 0x7f1eac0ae250 2021-02-10T11:36:16.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.322 7f1eb40c2700 1 -- 172.21.2.112:0/1547918558 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1eac0ac6d0 con 0x7f1eac0ae250 2021-02-10T11:36:16.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.323 7f1eb40c2700 1 -- 172.21.2.112:0/1547918558 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1eac0ac6d0 con 0x7f1eac0ae250 2021-02-10T11:36:16.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.323 7f1e9e7fc700 1 -- 172.21.2.112:0/1547918558 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1ea00052e0 con 0x7f1eac0ae250 2021-02-10T11:36:16.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.323 7f1e9e7fc700 1 --2- 172.21.2.112:0/1547918558 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1e940186c0 0x7f1e9401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:16.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.323 7f1e9e7fc700 1 -- 172.21.2.112:0/1547918558 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f1ea00323d0 con 0x7f1eac0ae250 2021-02-10T11:36:16.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.323 7f1eb1ede700 1 --2- 172.21.2.112:0/1547918558 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1e940186c0 0x7f1e9401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:16.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.324 7f1eb1ede700 1 --2- 172.21.2.112:0/1547918558 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1e940186c0 0x7f1e9401abc0 crc :-1 s=READY pgs=768 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:16.342 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.355 7f1eb40c2700 1 -- 172.21.2.112:0/1547918558 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1eac082a70 con 0x7f1eac0ae250 2021-02-10T11:36:16.344 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.358 7f1e9e7fc700 1 -- 172.21.2.112:0/1547918558 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1ea0006ba0 con 0x7f1eac0ae250 2021-02-10T11:36:16.473 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.487 7f1eb40c2700 1 -- 172.21.2.112:0/1547918558 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f1eac082a70 con 0x7f1eac0ae250 2021-02-10T11:36:16.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.487 7f1e9e7fc700 1 -- 172.21.2.112:0/1547918558 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v73) v1 ==== 104+0+10 (secure 0 0 0) 0x7f1ea00052e0 con 0x7f1eac0ae250 2021-02-10T11:36:16.474 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:36:16.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.489 7f1eb40c2700 1 -- 172.21.2.112:0/1547918558 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1e940186c0 msgr2=0x7f1e9401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:16.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.489 7f1eb40c2700 1 --2- 172.21.2.112:0/1547918558 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1e940186c0 0x7f1e9401abc0 crc :-1 s=READY pgs=768 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:16.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.489 7f1eb40c2700 1 -- 172.21.2.112:0/1547918558 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1eac0ae250 msgr2=0x7f1eac0ae650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:16.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.489 7f1eb40c2700 1 --2- 172.21.2.112:0/1547918558 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1eac0ae250 0x7f1eac0ae650 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 rx=0x7f1ea0006f10 tx=0x7f1ea000b990).stop 2021-02-10T11:36:16.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.489 7f1eb40c2700 1 -- 172.21.2.112:0/1547918558 shutdown_connections 2021-02-10T11:36:16.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.489 7f1eb40c2700 1 --2- 172.21.2.112:0/1547918558 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1e940186c0 0x7f1e9401abc0 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:16.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.489 7f1eb40c2700 1 --2- 172.21.2.112:0/1547918558 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1eac0aec70 0x7f1eac0af070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:16.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.489 7f1eb40c2700 1 --2- 172.21.2.112:0/1547918558 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1eac0ae250 0x7f1eac0ae650 unknown :-1 s=CLOSED pgs=1598 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:16.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.489 7f1eb40c2700 1 --2- 172.21.2.112:0/1547918558 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1eac1a7b70 0x7f1eac1a7f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:16.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.489 7f1eb40c2700 1 -- 172.21.2.112:0/1547918558 >> 172.21.2.112:0/1547918558 conn(0x7f1eac0a7410 msgr2=0x7f1eac0946e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:16.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.490 7f1eb40c2700 1 -- 172.21.2.112:0/1547918558 shutdown_connections 2021-02-10T11:36:16.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.490 7f1eb40c2700 1 -- 172.21.2.112:0/1547918558 wait complete. 2021-02-10T11:36:16.484 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:36:16.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.912 7ffb7ab49700 1 Processor -- start 2021-02-10T11:36:16.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.912 7ffb7ab49700 1 -- start start 2021-02-10T11:36:16.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.912 7ffb7ab49700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ffb740d1870 0x7ffb741b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:16.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.912 7ffb7ab49700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7ffb741b0ae0 0x7ffb74108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:16.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.912 7ffb7ab49700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb74109a60 con 0x7ffb740d1c70 2021-02-10T11:36:16.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.912 7ffb7ab49700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb740e5ef0 con 0x7ffb741b0ae0 2021-02-10T11:36:16.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.912 7ffb7ab49700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb74109750 con 0x7ffb740d1870 2021-02-10T11:36:16.917 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.913 7ffb73fff700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7ffb741b0ae0 0x7ffb74108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:16.917 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.914 7ffb79166700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7ffb740d1c70 0x7ffb74108f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:44936/0 (socket says 172.21.2.112:44936) 2021-02-10T11:36:16.917 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.914 7ffb79166700 1 -- 172.21.2.112:0/1949254611 learned_addr learned my addr 172.21.2.112:0/1949254611 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:36:16.917 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.914 7ffb78965700 1 --2- 172.21.2.112:0/1949254611 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ffb740d1870 0x7ffb741b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:16.917 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.914 7ffb73fff700 1 -- 172.21.2.112:0/1949254611 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ffb740d1870 msgr2=0x7ffb741b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:16.917 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.914 7ffb73fff700 1 --2- 172.21.2.112:0/1949254611 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ffb740d1870 0x7ffb741b04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:16.917 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.914 7ffb73fff700 1 -- 172.21.2.112:0/1949254611 >> v1:172.21.2.120:6789/0 conn(0x7ffb740d1c70 legacy=0x7ffb74108f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:16.917 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.914 7ffb73fff700 1 -- 172.21.2.112:0/1949254611 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb74088070 con 0x7ffb741b0ae0 2021-02-10T11:36:16.917 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.914 7ffb78965700 1 --2- 172.21.2.112:0/1949254611 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ffb740d1870 0x7ffb741b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:36:16.917 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.914 7ffb73fff700 1 --2- 172.21.2.112:0/1949254611 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ffb741b0ae0 0x7ffb74108940 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 rx=0x7ffb64006650 tx=0x7ffb64009a80).ready entity=mon.1 client_cookie=c5f8c9a1891b48be server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:16.917 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.915 7ffb737fe700 1 -- 172.21.2.112:0/1949254611 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffb6400c0c0 con 0x7ffb741b0ae0 2021-02-10T11:36:16.917 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.915 7ffb737fe700 1 -- 172.21.2.112:0/1949254611 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb6400c340 con 0x7ffb741b0ae0 2021-02-10T11:36:16.917 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.915 7ffb737fe700 1 -- 172.21.2.112:0/1949254611 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffb6400c860 con 0x7ffb741b0ae0 2021-02-10T11:36:16.918 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.915 7ffb7ab49700 1 -- 172.21.2.112:0/1949254611 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ffb741b0ae0 msgr2=0x7ffb74108940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:16.918 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.915 7ffb7ab49700 1 --2- 172.21.2.112:0/1949254611 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ffb741b0ae0 0x7ffb74108940 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 rx=0x7ffb64006650 tx=0x7ffb64009a80).stop 2021-02-10T11:36:16.918 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.915 7ffb7ab49700 1 -- 172.21.2.112:0/1949254611 shutdown_connections 2021-02-10T11:36:16.918 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.915 7ffb7ab49700 1 --2- 172.21.2.112:0/1949254611 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ffb741b0ae0 0x7ffb74108940 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:16.918 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.915 7ffb7ab49700 1 --2- 172.21.2.112:0/1949254611 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ffb740d1870 0x7ffb741b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:16.918 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.915 7ffb7ab49700 1 -- 172.21.2.112:0/1949254611 >> 172.21.2.112:0/1949254611 conn(0x7ffb740ce420 msgr2=0x7ffb740d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:16.918 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.915 7ffb7ab49700 1 -- 172.21.2.112:0/1949254611 shutdown_connections 2021-02-10T11:36:16.918 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.915 7ffb7ab49700 1 -- 172.21.2.112:0/1949254611 wait complete. 2021-02-10T11:36:16.918 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.929 7ffb7ab49700 1 Processor -- start 2021-02-10T11:36:16.918 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.930 7ffb7ab49700 1 -- start start 2021-02-10T11:36:16.919 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.930 7ffb7ab49700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ffb740a58c0 0x7ffb740a4d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:16.919 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.930 7ffb7ab49700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ffb740b1ba0 0x7ffb740b1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:16.919 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.930 7ffb7ab49700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ffb740b25c0 0x7ffb740b3270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:16.919 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.930 7ffb7ab49700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb740b2a40 con 0x7ffb740b1ba0 2021-02-10T11:36:16.919 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.930 7ffb7ab49700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb740a5340 con 0x7ffb740b25c0 2021-02-10T11:36:16.919 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.930 7ffb7ab49700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb740a5560 con 0x7ffb740a58c0 2021-02-10T11:36:16.919 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.930 7ffb73fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ffb740b1ba0 0x7ffb740b1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:16.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.930 7ffb73fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ffb740b1ba0 0x7ffb740b1fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:51922/0 (socket says 172.21.2.112:51922) 2021-02-10T11:36:16.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.930 7ffb73fff700 1 -- 172.21.2.112:0/1110465630 learned_addr learned my addr 172.21.2.112:0/1110465630 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:36:16.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.931 7ffb73fff700 1 -- 172.21.2.112:0/1110465630 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ffb740a58c0 msgr2=0x7ffb740a4d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:16.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.931 7ffb73fff700 1 --2- 172.21.2.112:0/1110465630 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ffb740a58c0 0x7ffb740a4d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:16.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.931 7ffb73fff700 1 -- 172.21.2.112:0/1110465630 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ffb740b25c0 msgr2=0x7ffb740b3270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:16.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.931 7ffb73fff700 1 --2- 172.21.2.112:0/1110465630 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ffb740b25c0 0x7ffb740b3270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:16.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.931 7ffb73fff700 1 -- 172.21.2.112:0/1110465630 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb740b3890 con 0x7ffb740b1ba0 2021-02-10T11:36:16.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.931 7ffb73fff700 1 --2- 172.21.2.112:0/1110465630 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ffb740b1ba0 0x7ffb740b1fa0 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 rx=0x7ffb64009630 tx=0x7ffb6400b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:16.923 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.931 7ffb717fa700 1 -- 172.21.2.112:0/1110465630 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffb640093a0 con 0x7ffb740b1ba0 2021-02-10T11:36:16.924 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.931 7ffb717fa700 1 -- 172.21.2.112:0/1110465630 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb6400bd70 con 0x7ffb740b1ba0 2021-02-10T11:36:16.924 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.931 7ffb717fa700 1 -- 172.21.2.112:0/1110465630 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffb64005560 con 0x7ffb740b1ba0 2021-02-10T11:36:16.924 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.931 7ffb7ab49700 1 -- 172.21.2.112:0/1110465630 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb740b3890 con 0x7ffb740b1ba0 2021-02-10T11:36:16.924 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.931 7ffb7ab49700 1 -- 172.21.2.112:0/1110465630 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb740b3de0 con 0x7ffb740b1ba0 2021-02-10T11:36:16.924 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.932 7ffb717fa700 1 -- 172.21.2.112:0/1110465630 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ffb6400a000 con 0x7ffb740b1ba0 2021-02-10T11:36:16.924 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.932 7ffb717fa700 1 --2- 172.21.2.112:0/1110465630 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ffb580186c0 0x7ffb5801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:16.924 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.932 7ffb717fa700 1 -- 172.21.2.112:0/1110465630 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5573+0+0 (secure 0 0 0) 0x7ffb64032400 con 0x7ffb740b1ba0 2021-02-10T11:36:16.924 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.932 7ffb78965700 1 --2- 172.21.2.112:0/1110465630 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ffb580186c0 0x7ffb5801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:16.924 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.933 7ffb78965700 1 --2- 172.21.2.112:0/1110465630 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ffb580186c0 0x7ffb5801abc0 crc :-1 s=READY pgs=769 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:16.952 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.962 7ffb7ab49700 1 -- 172.21.2.112:0/1110465630 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb74082980 con 0x7ffb740b1ba0 2021-02-10T11:36:16.953 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:16.965 7ffb717fa700 1 -- 172.21.2.112:0/1110465630 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ffb64009340 con 0x7ffb740b1ba0 2021-02-10T11:36:17.079 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:17.092 7ffb7ab49700 1 -- 172.21.2.112:0/1110465630 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7ffb74109a60 con 0x7ffb740b1ba0 2021-02-10T11:36:17.079 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:17.092 7ffb717fa700 1 -- 172.21.2.112:0/1110465630 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v73) v1 ==== 100+0+10 (secure 0 0 0) 0x7ffb6402df40 con 0x7ffb740b1ba0 2021-02-10T11:36:17.080 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:36:17.082 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:17.094 7ffb7ab49700 1 -- 172.21.2.112:0/1110465630 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ffb580186c0 msgr2=0x7ffb5801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:17.082 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:17.094 7ffb7ab49700 1 --2- 172.21.2.112:0/1110465630 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ffb580186c0 0x7ffb5801abc0 crc :-1 s=READY pgs=769 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:17.082 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:17.094 7ffb7ab49700 1 -- 172.21.2.112:0/1110465630 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ffb740b1ba0 msgr2=0x7ffb740b1fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:17.082 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:17.094 7ffb7ab49700 1 --2- 172.21.2.112:0/1110465630 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ffb740b1ba0 0x7ffb740b1fa0 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 rx=0x7ffb64009630 tx=0x7ffb6400b980).stop 2021-02-10T11:36:17.082 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:17.094 7ffb7ab49700 1 -- 172.21.2.112:0/1110465630 shutdown_connections 2021-02-10T11:36:17.082 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:17.094 7ffb7ab49700 1 --2- 172.21.2.112:0/1110465630 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ffb580186c0 0x7ffb5801abc0 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:17.082 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:17.094 7ffb7ab49700 1 --2- 172.21.2.112:0/1110465630 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ffb740b25c0 0x7ffb740b3270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:17.082 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:17.095 7ffb7ab49700 1 --2- 172.21.2.112:0/1110465630 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ffb740b1ba0 0x7ffb740b1fa0 unknown :-1 s=CLOSED pgs=1599 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:17.082 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:17.095 7ffb7ab49700 1 --2- 172.21.2.112:0/1110465630 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ffb740a58c0 0x7ffb740a4d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:17.083 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:17.095 7ffb7ab49700 1 -- 172.21.2.112:0/1110465630 >> 172.21.2.112:0/1110465630 conn(0x7ffb740b5dc0 msgr2=0x7ffb74095960 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:17.083 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:17.095 7ffb7ab49700 1 -- 172.21.2.112:0/1110465630 shutdown_connections 2021-02-10T11:36:17.083 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:17.095 7ffb7ab49700 1 -- 172.21.2.112:0/1110465630 wait complete. 2021-02-10T11:36:17.090 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:36:17.090 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:36:17.519 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:17.515 7fe825d22700 1 Processor -- start 2021-02-10T11:36:17.520 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:17.515 7fe825d22700 1 -- start start 2021-02-10T11:36:17.520 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:17.516 7fe825d22700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fe8200d1870 0x7fe8201b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:17.520 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:17.516 7fe825d22700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe8201b0a10 0x7fe8200cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:17.520 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:17.516 7fe825d22700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8200f0b10 con 0x7fe8200d1c70 2021-02-10T11:36:17.520 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:17.516 7fe825d22700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8200e5ed0 con 0x7fe8200d1870 2021-02-10T11:36:17.520 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:17.516 7fe825d22700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8200f0650 con 0x7fe8201b0a10 2021-02-10T11:36:17.520 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:17.516 7fe81ffff700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fe8200d1c70 0x7fe8200cf950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:51390/0 (socket says 172.21.2.112:51390) 2021-02-10T11:36:17.520 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:17.516 7fe81ffff700 1 -- 172.21.2.112:0/2573622833 learned_addr learned my addr 172.21.2.112:0/2573622833 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:36:17.520 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:17.516 7fe81e7fc700 1 -- 172.21.2.112:0/2573622833 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2854908141 0 0) 0x7fe818001500 con 0x7fe8200d1c70 2021-02-10T11:36:17.520 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:17.517 7fe81e7fc700 1 -- 172.21.2.112:0/2573622833 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe80c002fe0 con 0x7fe8200d1c70 2021-02-10T11:36:17.520 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:17.517 7fe81f7fe700 1 --2- 172.21.2.112:0/2573622833 >> v2:172.21.2.120:3300/0 conn(0x7fe8200d1870 0x7fe8201b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:17.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:17.517 7fe81effd700 1 --2- 172.21.2.112:0/2573622833 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe8201b0a10 0x7fe8200cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:17.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:17.517 7fe81e7fc700 1 -- 172.21.2.112:0/2573622833 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 315396217 0 0) 0x7fe818001460 con 0x7fe8200d1c70 2021-02-10T11:36:17.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:17.517 7fe81e7fc700 1 -- 172.21.2.112:0/2573622833 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe8201b0a10 msgr2=0x7fe8200cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:17.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:17.517 7fe81e7fc700 1 --2- 172.21.2.112:0/2573622833 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe8201b0a10 0x7fe8200cf330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:17.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:17.517 7fe81e7fc700 1 -- 172.21.2.112:0/2573622833 >> v2:172.21.2.120:3300/0 conn(0x7fe8200d1870 msgr2=0x7fe8201b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:17.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:17.517 7fe81e7fc700 1 --2- 172.21.2.112:0/2573622833 >> v2:172.21.2.120:3300/0 conn(0x7fe8200d1870 0x7fe8201b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:17.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:17.517 7fe81e7fc700 1 -- 172.21.2.112:0/2573622833 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8200f0870 con 0x7fe8200d1c70 2021-02-10T11:36:17.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:17.517 7fe81e7fc700 1 -- 172.21.2.112:0/2573622833 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7fe818001980 con 0x7fe8200d1c70 2021-02-10T11:36:17.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:17.517 7fe81e7fc700 1 -- 172.21.2.112:0/2573622833 >> v1:172.21.2.112:6789/0 conn(0x7fe8200d1c70 legacy=0x7fe8200cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:17.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:17.517 7fe81e7fc700 1 --2- 172.21.2.112:0/2573622833 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe80c003a60 0x7fe80c005ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:17.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:17.517 7fe81e7fc700 1 --2- 172.21.2.112:0/2573622833 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe80c04a780 0x7fe80c04cc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:17.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:17.517 7fe81e7fc700 1 --2- 172.21.2.112:0/2573622833 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe80c04d510 0x7fe80c04fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:17.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:17.517 7fe81e7fc700 1 -- 172.21.2.112:0/2573622833 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe80c0502e0 con 0x7fe80c04d510 2021-02-10T11:36:17.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:17.517 7fe81e7fc700 1 -- 172.21.2.112:0/2573622833 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe80c050500 con 0x7fe80c003a60 2021-02-10T11:36:17.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:17.517 7fe81e7fc700 1 -- 172.21.2.112:0/2573622833 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe80c050720 con 0x7fe80c04a780 2021-02-10T11:36:17.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:17.517 7fe81ffff700 1 --2- 172.21.2.112:0/2573622833 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe80c04d510 0x7fe80c04fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:17.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:17.517 7fe81f7fe700 1 --2- 172.21.2.112:0/2573622833 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe80c003a60 0x7fe80c005ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:17.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:17.517 7fe81effd700 1 --2- 172.21.2.112:0/2573622833 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe80c04a780 0x7fe80c04cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:17.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:17.517 7fe81ffff700 1 -- 172.21.2.112:0/2573622833 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe80c04a780 msgr2=0x7fe80c04cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:17.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:17.517 7fe81ffff700 1 --2- 172.21.2.112:0/2573622833 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe80c04a780 0x7fe80c04cc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:17.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:17.517 7fe81ffff700 1 -- 172.21.2.112:0/2573622833 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe80c003a60 msgr2=0x7fe80c005ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:17.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:17.517 7fe81ffff700 1 --2- 172.21.2.112:0/2573622833 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe80c003a60 0x7fe80c005ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:17.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:17.517 7fe81ffff700 1 -- 172.21.2.112:0/2573622833 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe80c0509c0 con 0x7fe80c04d510 2021-02-10T11:36:17.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:17.518 7fe81ffff700 1 --2- 172.21.2.112:0/2573622833 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe80c04d510 0x7fe80c04fa30 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 rx=0x7fe818001140 tx=0x7fe81800a310).ready entity=mon.0 client_cookie=b8af22a7015e43e7 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:17.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:17.518 7fe81e7fc700 1 -- 172.21.2.112:0/2573622833 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe8180070c0 con 0x7fe80c04d510 2021-02-10T11:36:17.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:17.518 7fe81e7fc700 1 -- 172.21.2.112:0/2573622833 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe818007440 con 0x7fe80c04d510 2021-02-10T11:36:17.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:17.518 7fe81e7fc700 1 -- 172.21.2.112:0/2573622833 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe8180049a0 con 0x7fe80c04d510 2021-02-10T11:36:17.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:17.518 7fe825d22700 1 -- 172.21.2.112:0/2573622833 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe80c04d510 msgr2=0x7fe80c04fa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:17.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:17.518 7fe825d22700 1 --2- 172.21.2.112:0/2573622833 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe80c04d510 0x7fe80c04fa30 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 rx=0x7fe818001140 tx=0x7fe81800a310).stop 2021-02-10T11:36:17.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:17.518 7fe81ffff700 1 -- 172.21.2.112:0/2573622833 reap_dead start 2021-02-10T11:36:17.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:17.519 7fe825d22700 1 -- 172.21.2.112:0/2573622833 shutdown_connections 2021-02-10T11:36:17.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:17.519 7fe825d22700 1 -- 172.21.2.112:0/2573622833 >> 172.21.2.112:0/2573622833 conn(0x7fe8200ce450 msgr2=0x7fe8200d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:17.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:17.519 7fe825d22700 1 -- 172.21.2.112:0/2573622833 shutdown_connections 2021-02-10T11:36:17.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:17.519 7fe825d22700 1 -- 172.21.2.112:0/2573622833 wait complete. 2021-02-10T11:36:17.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:17.533 7fe825d22700 1 Processor -- start 2021-02-10T11:36:17.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:17.534 7fe825d22700 1 -- start start 2021-02-10T11:36:17.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:17.534 7fe825d22700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe8201a4a60 0x7fe8200c1c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:17.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:17.534 7fe825d22700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe8200c2290 0x7fe8200c2690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:17.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:17.534 7fe825d22700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe8200c2cb0 0x7fe8200bfd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:17.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:17.534 7fe825d22700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8200f05b0 con 0x7fe8201a4a60 2021-02-10T11:36:17.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:17.534 7fe825d22700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8200f07d0 con 0x7fe8200c2cb0 2021-02-10T11:36:17.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:17.534 7fe825d22700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8200c0340 con 0x7fe8200c2290 2021-02-10T11:36:17.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:17.534 7fe81f7fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe8201a4a60 0x7fe8200c1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:17.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:17.534 7fe81f7fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe8201a4a60 0x7fe8200c1c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:51944/0 (socket says 172.21.2.112:51944) 2021-02-10T11:36:17.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:17.534 7fe81f7fe700 1 -- 172.21.2.112:0/3443230958 learned_addr learned my addr 172.21.2.112:0/3443230958 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:36:17.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:17.534 7fe81f7fe700 1 -- 172.21.2.112:0/3443230958 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe8200c2290 msgr2=0x7fe8200c2690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:17.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:17.534 7fe81f7fe700 1 --2- 172.21.2.112:0/3443230958 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe8200c2290 0x7fe8200c2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:17.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:17.534 7fe81f7fe700 1 -- 172.21.2.112:0/3443230958 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe8200c2cb0 msgr2=0x7fe8200bfd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:17.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:17.534 7fe81f7fe700 1 --2- 172.21.2.112:0/3443230958 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe8200c2cb0 0x7fe8200bfd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:17.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:17.534 7fe81f7fe700 1 -- 172.21.2.112:0/3443230958 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8200c0560 con 0x7fe8201a4a60 2021-02-10T11:36:17.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:17.535 7fe81f7fe700 1 --2- 172.21.2.112:0/3443230958 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe8201a4a60 0x7fe8200c1c70 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 rx=0x7fe80800b560 tx=0x7fe80800b900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:17.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:17.535 7fe817fff700 1 -- 172.21.2.112:0/3443230958 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe80800bcf0 con 0x7fe8201a4a60 2021-02-10T11:36:17.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:17.535 7fe817fff700 1 -- 172.21.2.112:0/3443230958 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe808000c90 con 0x7fe8201a4a60 2021-02-10T11:36:17.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:17.535 7fe817fff700 1 -- 172.21.2.112:0/3443230958 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe8080022b0 con 0x7fe8201a4a60 2021-02-10T11:36:17.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:17.535 7fe825d22700 1 -- 172.21.2.112:0/3443230958 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8200c0560 con 0x7fe8201a4a60 2021-02-10T11:36:17.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:17.535 7fe825d22700 1 -- 172.21.2.112:0/3443230958 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8200c0560 con 0x7fe8201a4a60 2021-02-10T11:36:17.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:17.536 7fe817fff700 1 -- 172.21.2.112:0/3443230958 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe808006b00 con 0x7fe8201a4a60 2021-02-10T11:36:17.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:17.536 7fe817fff700 1 --2- 172.21.2.112:0/3443230958 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe8040186c0 0x7fe80401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:17.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:17.536 7fe817fff700 1 -- 172.21.2.112:0/3443230958 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fe808032590 con 0x7fe8201a4a60 2021-02-10T11:36:17.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:17.536 7fe81effd700 1 --2- 172.21.2.112:0/3443230958 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe8040186c0 0x7fe80401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:17.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:17.537 7fe81effd700 1 --2- 172.21.2.112:0/3443230958 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe8040186c0 0x7fe80401abc0 crc :-1 s=READY pgs=770 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:17.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:17.566 7fe825d22700 1 -- 172.21.2.112:0/3443230958 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8200f0af0 con 0x7fe8201a4a60 2021-02-10T11:36:17.555 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:17.568 7fe817fff700 1 -- 172.21.2.112:0/3443230958 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe808030d40 con 0x7fe8201a4a60 2021-02-10T11:36:17.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:17.696 7fe825d22700 1 -- 172.21.2.112:0/3443230958 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe8200f0af0 con 0x7fe8201a4a60 2021-02-10T11:36:17.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:17.696 7fe817fff700 1 -- 172.21.2.112:0/3443230958 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+2101 (secure 0 0 0) 0x7fe80800bcf0 con 0x7fe8201a4a60 2021-02-10T11:36:17.684 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 56 2021-02-10T11:36:17.684 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:36:17.684 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":56,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8820,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/3613517698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":3613517698},{"type":"v1","addr":"172.21.2.112:6833","nonce":3613517698}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":52}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:36:00.289087","modified":"2021-02-10 11:36:02.295150","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7087},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7087":{"gid":7087,"name":"b","rank":0,"incarnation":53,"state":"up:active","state_seq":3,"addr":"172.21.2.120:6835/1870802251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":1870802251},{"type":"v1","addr":"172.21.2.120:6835","nonce":1870802251}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-02-10T11:36:17.685 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:17.698 7fe825d22700 1 -- 172.21.2.112:0/3443230958 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe8040186c0 msgr2=0x7fe80401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:17.685 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:17.698 7fe825d22700 1 --2- 172.21.2.112:0/3443230958 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe8040186c0 0x7fe80401abc0 crc :-1 s=READY pgs=770 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:17.685 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:17.698 7fe825d22700 1 -- 172.21.2.112:0/3443230958 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe8201a4a60 msgr2=0x7fe8200c1c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:17.685 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:17.698 7fe825d22700 1 --2- 172.21.2.112:0/3443230958 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe8201a4a60 0x7fe8200c1c70 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 rx=0x7fe80800b560 tx=0x7fe80800b900).stop 2021-02-10T11:36:17.685 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:17.699 7fe825d22700 1 -- 172.21.2.112:0/3443230958 shutdown_connections 2021-02-10T11:36:17.685 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:17.699 7fe825d22700 1 --2- 172.21.2.112:0/3443230958 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe8040186c0 0x7fe80401abc0 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:17.685 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:17.699 7fe825d22700 1 --2- 172.21.2.112:0/3443230958 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe8200c2cb0 0x7fe8200bfd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:17.686 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:17.699 7fe825d22700 1 --2- 172.21.2.112:0/3443230958 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe8200c2290 0x7fe8200c2690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:17.686 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:17.699 7fe825d22700 1 --2- 172.21.2.112:0/3443230958 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe8201a4a60 0x7fe8200c1c70 unknown :-1 s=CLOSED pgs=1602 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:17.686 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:17.699 7fe825d22700 1 -- 172.21.2.112:0/3443230958 >> 172.21.2.112:0/3443230958 conn(0x7fe8200ebe30 msgr2=0x7fe8200bf200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:17.686 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:17.699 7fe825d22700 1 -- 172.21.2.112:0/3443230958 shutdown_connections 2021-02-10T11:36:17.686 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:17.699 7fe825d22700 1 -- 172.21.2.112:0/3443230958 wait complete. 2021-02-10T11:36:17.895 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 56, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:36:00.289087', 'modified': '2021-02-10 11:36:02.295150', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7087}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7087': {'gid': 7087, 'name': 'b', 'rank': 0, 'incarnation': 53, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.120:6835/1870802251', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.120:6834', 'nonce': 1870802251}, {'type': 'v1', 'addr': '172.21.2.120:6835', 'nonce': 1870802251}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:36:17.895 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:36:17.896 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:36:18.292 INFO:teuthology.orchestra.run.gibba020.stdout:{ 2021-02-10T11:36:18.292 INFO:teuthology.orchestra.run.gibba020.stdout: "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", 2021-02-10T11:36:18.292 INFO:teuthology.orchestra.run.gibba020.stdout: "whoami": 0, 2021-02-10T11:36:18.292 INFO:teuthology.orchestra.run.gibba020.stdout: "id": 7087, 2021-02-10T11:36:18.293 INFO:teuthology.orchestra.run.gibba020.stdout: "want_state": "up:active", 2021-02-10T11:36:18.293 INFO:teuthology.orchestra.run.gibba020.stdout: "state": "up:active", 2021-02-10T11:36:18.293 INFO:teuthology.orchestra.run.gibba020.stdout: "rank_uptime": 18.014332570000001, 2021-02-10T11:36:18.293 INFO:teuthology.orchestra.run.gibba020.stdout: "mdsmap_epoch": 56, 2021-02-10T11:36:18.293 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch": 73, 2021-02-10T11:36:18.293 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch_barrier": 73, 2021-02-10T11:36:18.293 INFO:teuthology.orchestra.run.gibba020.stdout: "uptime": 23.66682398 2021-02-10T11:36:18.293 INFO:teuthology.orchestra.run.gibba020.stdout:} 2021-02-10T11:36:18.296 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", "whoami": 0, "id": 7087, "want_state": "up:active", "state": "up:active", "rank_uptime": 18.014332570000001, "mdsmap_epoch": 56, "osdmap_epoch": 73, "osdmap_epoch_barrier": 73, "uptime": 23.66682398 } 2021-02-10T11:36:18.296 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:36:18.297 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2021-02-10T11:36:18.297 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.1 at ubuntu@gibba012.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2021-02-10T11:36:18.297 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:18.297 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:36:18.550 DEBUG:teuthology.orchestra.run.gibba012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:36:18.554 DEBUG:teuthology.orchestra.run.gibba020:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:36:18.631 DEBUG:teuthology.orchestra.run.gibba012:> sudo modprobe fuse 2021-02-10T11:36:18.896 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:36:18.896 DEBUG:teuthology.orchestra.run.gibba012:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:36:19.238 INFO:teuthology.orchestra.run.gibba012.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:36:19.239 INFO:teuthology.orchestra.run.gibba012.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:36:19.239 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:36:19.240 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:19.240 DEBUG:teuthology.orchestra.run.gibba012:> ls /sys/fs/fuse/connections 2021-02-10T11:36:19.574 INFO:teuthology.orchestra.run.gibba012.stdout:42 2021-02-10T11:36:19.575 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [42] 2021-02-10T11:36:19.575 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.1 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:36:19.579 DEBUG:teuthology.orchestra.run.gibba012:> sudo modprobe fuse 2021-02-10T11:36:19.922 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:36:19.922 DEBUG:teuthology.orchestra.run.gibba012:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:36:19.988 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba012.stderr:ceph-fuse[141806]: starting ceph client 2021-02-10T11:36:19.988 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba012.stderr:2021-02-10 11:36:20.001 7f8a3c836f80 -1 init, newargv = 0x56503df89220 newargc=9 2021-02-10T11:36:19.993 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba012.stderr:ceph-fuse[141806]: starting fuse 2021-02-10T11:36:20.264 INFO:teuthology.orchestra.run.gibba012.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:36:20.264 INFO:teuthology.orchestra.run.gibba012.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:36:20.265 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:36:20.265 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:20.265 DEBUG:teuthology.orchestra.run.gibba012:> ls /sys/fs/fuse/connections 2021-02-10T11:36:20.600 INFO:teuthology.orchestra.run.gibba012.stdout:42 2021-02-10T11:36:20.601 INFO:teuthology.orchestra.run.gibba012.stdout:43 2021-02-10T11:36:20.601 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42, 43] 2021-02-10T11:36:20.601 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:20.601 DEBUG:teuthology.orchestra.run.gibba012:> sudo python3 -c ' 2021-02-10T11:36:20.601 DEBUG:teuthology.orchestra.run.gibba012:> import glob 2021-02-10T11:36:20.601 DEBUG:teuthology.orchestra.run.gibba012:> import re 2021-02-10T11:36:20.601 DEBUG:teuthology.orchestra.run.gibba012:> import os 2021-02-10T11:36:20.602 DEBUG:teuthology.orchestra.run.gibba012:> import subprocess 2021-02-10T11:36:20.602 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:36:20.602 DEBUG:teuthology.orchestra.run.gibba012:> def find_socket(client_name): 2021-02-10T11:36:20.602 DEBUG:teuthology.orchestra.run.gibba012:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2021-02-10T11:36:20.602 DEBUG:teuthology.orchestra.run.gibba012:> files = glob.glob(asok_path) 2021-02-10T11:36:20.602 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:36:20.602 DEBUG:teuthology.orchestra.run.gibba012:> # Given a non-glob path, it better be there 2021-02-10T11:36:20.602 DEBUG:teuthology.orchestra.run.gibba012:> if "*" not in asok_path: 2021-02-10T11:36:20.602 DEBUG:teuthology.orchestra.run.gibba012:> assert(len(files) == 1) 2021-02-10T11:36:20.602 DEBUG:teuthology.orchestra.run.gibba012:> return files[0] 2021-02-10T11:36:20.602 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:36:20.602 DEBUG:teuthology.orchestra.run.gibba012:> for f in files: 2021-02-10T11:36:20.602 DEBUG:teuthology.orchestra.run.gibba012:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:36:20.603 DEBUG:teuthology.orchestra.run.gibba012:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:36:20.603 DEBUG:teuthology.orchestra.run.gibba012:> return f 2021-02-10T11:36:20.603 DEBUG:teuthology.orchestra.run.gibba012:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:36:20.603 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:36:20.603 DEBUG:teuthology.orchestra.run.gibba012:> print(find_socket("client.1")) 2021-02-10T11:36:20.603 DEBUG:teuthology.orchestra.run.gibba012:> ' 2021-02-10T11:36:20.963 INFO:teuthology.orchestra.run.gibba012.stdout:/var/run/ceph/ceph-client.1.141806.asok 2021-02-10T11:36:20.966 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.141806.asok 2021-02-10T11:36:20.966 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:20.966 DEBUG:teuthology.orchestra.run.gibba012:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.141806.asok status 2021-02-10T11:36:21.355 INFO:teuthology.orchestra.run.gibba012.stdout:{ 2021-02-10T11:36:21.355 INFO:teuthology.orchestra.run.gibba012.stdout: "metadata": { 2021-02-10T11:36:21.355 INFO:teuthology.orchestra.run.gibba012.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:36:21.355 INFO:teuthology.orchestra.run.gibba012.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:36:21.356 INFO:teuthology.orchestra.run.gibba012.stdout: "entity_id": "1", 2021-02-10T11:36:21.356 INFO:teuthology.orchestra.run.gibba012.stdout: "hostname": "gibba012", 2021-02-10T11:36:21.356 INFO:teuthology.orchestra.run.gibba012.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2021-02-10T11:36:21.356 INFO:teuthology.orchestra.run.gibba012.stdout: "pid": "141806", 2021-02-10T11:36:21.356 INFO:teuthology.orchestra.run.gibba012.stdout: "root": "/" 2021-02-10T11:36:21.356 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:36:21.356 INFO:teuthology.orchestra.run.gibba012.stdout: "dentry_count": 0, 2021-02-10T11:36:21.356 INFO:teuthology.orchestra.run.gibba012.stdout: "dentry_pinned_count": 0, 2021-02-10T11:36:21.356 INFO:teuthology.orchestra.run.gibba012.stdout: "id": 9024, 2021-02-10T11:36:21.356 INFO:teuthology.orchestra.run.gibba012.stdout: "inst": { 2021-02-10T11:36:21.356 INFO:teuthology.orchestra.run.gibba012.stdout: "name": { 2021-02-10T11:36:21.356 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "client", 2021-02-10T11:36:21.356 INFO:teuthology.orchestra.run.gibba012.stdout: "num": 9024 2021-02-10T11:36:21.356 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:36:21.357 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": { 2021-02-10T11:36:21.357 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:36:21.357 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:0", 2021-02-10T11:36:21.357 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 4191416827 2021-02-10T11:36:21.357 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:36:21.357 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:36:21.358 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": { 2021-02-10T11:36:21.358 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:36:21.358 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:0", 2021-02-10T11:36:21.358 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 4191416827 2021-02-10T11:36:21.358 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:36:21.358 INFO:teuthology.orchestra.run.gibba012.stdout: "inst_str": "client.9024 172.21.2.112:0/4191416827", 2021-02-10T11:36:21.359 INFO:teuthology.orchestra.run.gibba012.stdout: "addr_str": "172.21.2.112:0/4191416827", 2021-02-10T11:36:21.359 INFO:teuthology.orchestra.run.gibba012.stdout: "inode_count": 1, 2021-02-10T11:36:21.359 INFO:teuthology.orchestra.run.gibba012.stdout: "mds_epoch": 56, 2021-02-10T11:36:21.359 INFO:teuthology.orchestra.run.gibba012.stdout: "osd_epoch": 73, 2021-02-10T11:36:21.359 INFO:teuthology.orchestra.run.gibba012.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:36:21.359 INFO:teuthology.orchestra.run.gibba012.stdout: "blacklisted": false 2021-02-10T11:36:21.360 INFO:teuthology.orchestra.run.gibba012.stdout:} 2021-02-10T11:36:21.362 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:21.362 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:36:21.362 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:36:21.698 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:36:21.699 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:36:21.699 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:21.699 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:36:22.044 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2021-02-10T11:36:22.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:22.471 7f4db0aef700 1 Processor -- start 2021-02-10T11:36:22.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:22.471 7f4db0aef700 1 -- start start 2021-02-10T11:36:22.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:22.472 7f4db0aef700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f4dac0d1880 0x7f4dac1b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:22.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:22.472 7f4db0aef700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dac0fe1d0 con 0x7f4dac0cff80 2021-02-10T11:36:22.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:22.472 7f4db0aef700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dac109a70 con 0x7f4dac1b0af0 2021-02-10T11:36:22.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:22.472 7f4db0aef700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4dac088070 con 0x7f4dac0d1880 2021-02-10T11:36:22.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:22.472 7f4dab05f700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f4dac0cff80 0x7f4dac109140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:51424/0 (socket says 172.21.2.112:51424) 2021-02-10T11:36:22.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:22.472 7f4dab05f700 1 -- 172.21.2.112:0/905485523 learned_addr learned my addr 172.21.2.112:0/905485523 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:36:22.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:22.472 7f4da985c700 1 -- 172.21.2.112:0/905485523 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2481889389 0 0) 0x7f4da4001500 con 0x7f4dac0cff80 2021-02-10T11:36:22.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:22.472 7f4da985c700 1 -- 172.21.2.112:0/905485523 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d94002fe0 con 0x7f4dac0cff80 2021-02-10T11:36:22.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:22.473 7f4daa85e700 1 --2- 172.21.2.112:0/905485523 >> v2:172.21.2.120:3300/0 conn(0x7f4dac0d1880 0x7f4dac1b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:22.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:22.473 7f4da985c700 1 -- 172.21.2.112:0/905485523 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 939381284 0 0) 0x7f4da4001500 con 0x7f4dac0cff80 2021-02-10T11:36:22.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:22.473 7f4da985c700 1 -- 172.21.2.112:0/905485523 >> v2:172.21.2.120:3300/0 conn(0x7f4dac0d1880 msgr2=0x7f4dac1b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:22.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:22.473 7f4da985c700 1 --2- 172.21.2.112:0/905485523 >> v2:172.21.2.120:3300/0 conn(0x7f4dac0d1880 0x7f4dac1b04d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:22.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:22.473 7f4da985c700 1 -- 172.21.2.112:0/905485523 >> v1:172.21.2.120:6789/0 conn(0x7f4dac1b0af0 legacy=0x7f4dac108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:22.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:22.473 7f4da985c700 1 -- 172.21.2.112:0/905485523 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dac088260 con 0x7f4dac0cff80 2021-02-10T11:36:22.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:22.473 7f4da985c700 1 -- 172.21.2.112:0/905485523 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f4da4001a00 con 0x7f4dac0cff80 2021-02-10T11:36:22.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:22.473 7f4da985c700 1 -- 172.21.2.112:0/905485523 >> v1:172.21.2.112:6789/0 conn(0x7f4dac0cff80 legacy=0x7f4dac109140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:22.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:22.473 7f4da985c700 1 --2- 172.21.2.112:0/905485523 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4d94003aa0 0x7f4d94005f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:22.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:22.473 7f4da985c700 1 --2- 172.21.2.112:0/905485523 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4d94006780 0x7f4d94008ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:22.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:22.473 7f4da985c700 1 --2- 172.21.2.112:0/905485523 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4d94009580 0x7f4d9400baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:22.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:22.473 7f4da985c700 1 -- 172.21.2.112:0/905485523 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d9400c2f0 con 0x7f4d94003aa0 2021-02-10T11:36:22.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:22.473 7f4da985c700 1 -- 172.21.2.112:0/905485523 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d9400c510 con 0x7f4d94009580 2021-02-10T11:36:22.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:22.473 7f4da985c700 1 -- 172.21.2.112:0/905485523 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d9400c730 con 0x7f4d94006780 2021-02-10T11:36:22.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:22.473 7f4daa85e700 1 --2- 172.21.2.112:0/905485523 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4d94003aa0 0x7f4d94005f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:22.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:22.473 7f4daa05d700 1 --2- 172.21.2.112:0/905485523 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4d94006780 0x7f4d94008ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:22.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:22.473 7f4daa85e700 1 -- 172.21.2.112:0/905485523 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4d94006780 msgr2=0x7f4d94008ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:22.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:22.473 7f4daa85e700 1 --2- 172.21.2.112:0/905485523 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4d94006780 0x7f4d94008ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:22.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:22.473 7f4daa85e700 1 -- 172.21.2.112:0/905485523 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4d94009580 msgr2=0x7f4d9400baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:22.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:22.473 7f4dab05f700 1 --2- 172.21.2.112:0/905485523 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4d94009580 0x7f4d9400baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:22.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:22.473 7f4daa85e700 1 --2- 172.21.2.112:0/905485523 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4d94009580 0x7f4d9400baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:22.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:22.473 7f4daa85e700 1 -- 172.21.2.112:0/905485523 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d94050970 con 0x7f4d94003aa0 2021-02-10T11:36:22.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:22.473 7f4dab05f700 1 --2- 172.21.2.112:0/905485523 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4d94009580 0x7f4d9400baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:36:22.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:22.473 7f4dab05f700 1 -- 172.21.2.112:0/905485523 reap_dead start 2021-02-10T11:36:22.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:22.474 7f4daa85e700 1 --2- 172.21.2.112:0/905485523 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4d94003aa0 0x7f4d94005f60 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 rx=0x7f4d9c00c870 tx=0x7f4d9c00cc10).ready entity=mon.0 client_cookie=9f1a2013dbf9f086 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:22.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:22.474 7f4da985c700 1 -- 172.21.2.112:0/905485523 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4d9c0009b0 con 0x7f4d94003aa0 2021-02-10T11:36:22.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:22.474 7f4da985c700 1 -- 172.21.2.112:0/905485523 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d9c0090d0 con 0x7f4d94003aa0 2021-02-10T11:36:22.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:22.474 7f4da985c700 1 -- 172.21.2.112:0/905485523 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4d9c0095f0 con 0x7f4d94003aa0 2021-02-10T11:36:22.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:22.474 7f4db0aef700 1 -- 172.21.2.112:0/905485523 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4d94003aa0 msgr2=0x7f4d94005f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:22.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:22.474 7f4db0aef700 1 --2- 172.21.2.112:0/905485523 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4d94003aa0 0x7f4d94005f60 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 rx=0x7f4d9c00c870 tx=0x7f4d9c00cc10).stop 2021-02-10T11:36:22.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:22.474 7f4db0aef700 1 -- 172.21.2.112:0/905485523 shutdown_connections 2021-02-10T11:36:22.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:22.474 7f4db0aef700 1 --2- 172.21.2.112:0/905485523 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4d94003aa0 0x7f4d94005f60 unknown :-1 s=CLOSED pgs=1607 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:22.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:22.474 7f4db0aef700 1 -- 172.21.2.112:0/905485523 >> 172.21.2.112:0/905485523 conn(0x7f4dac0ce430 msgr2=0x7f4dac0d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:22.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:22.475 7f4db0aef700 1 -- 172.21.2.112:0/905485523 shutdown_connections 2021-02-10T11:36:22.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:22.475 7f4db0aef700 1 -- 172.21.2.112:0/905485523 wait complete. 2021-02-10T11:36:22.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:22.489 7f4db0aef700 1 Processor -- start 2021-02-10T11:36:22.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:22.489 7f4db0aef700 1 -- start start 2021-02-10T11:36:22.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:22.490 7f4db0aef700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4dac1a3570 0x7f4dac1a3970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:22.481 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:22.490 7f4db0aef700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4dac09c9f0 0x7f4dac09cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:22.481 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:22.490 7f4db0aef700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4dac09d410 0x7f4dac098170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:22.481 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:22.490 7f4db0aef700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dac09c470 con 0x7f4dac09d410 2021-02-10T11:36:22.481 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:22.490 7f4db0aef700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dac0b2840 con 0x7f4dac1a3570 2021-02-10T11:36:22.481 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:22.490 7f4db0aef700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dac0b2a20 con 0x7f4dac09c9f0 2021-02-10T11:36:22.481 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:22.490 7f4dab05f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4dac09d410 0x7f4dac098170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:22.481 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:22.490 7f4dab05f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4dac09d410 0x7f4dac098170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:51976/0 (socket says 172.21.2.112:51976) 2021-02-10T11:36:22.481 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:22.490 7f4dab05f700 1 -- 172.21.2.112:0/1799897377 learned_addr learned my addr 172.21.2.112:0/1799897377 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:36:22.481 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:22.490 7f4daa05d700 1 --2- 172.21.2.112:0/1799897377 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4dac09c9f0 0x7f4dac09cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:22.481 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:22.490 7f4daa85e700 1 --2- 172.21.2.112:0/1799897377 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4dac1a3570 0x7f4dac1a3970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:22.481 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:22.490 7f4dab05f700 1 -- 172.21.2.112:0/1799897377 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4dac09c9f0 msgr2=0x7f4dac09cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:22.481 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:22.490 7f4dab05f700 1 --2- 172.21.2.112:0/1799897377 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4dac09c9f0 0x7f4dac09cdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:22.481 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:22.490 7f4dab05f700 1 -- 172.21.2.112:0/1799897377 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4dac1a3570 msgr2=0x7f4dac1a3970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:22.482 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:22.490 7f4dab05f700 1 --2- 172.21.2.112:0/1799897377 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4dac1a3570 0x7f4dac1a3970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:22.482 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:22.490 7f4dab05f700 1 -- 172.21.2.112:0/1799897377 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dac09d880 con 0x7f4dac09d410 2021-02-10T11:36:22.482 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:22.490 7f4dab05f700 1 --2- 172.21.2.112:0/1799897377 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4dac09d410 0x7f4dac098170 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 rx=0x7f4da4009980 tx=0x7f4da400bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:22.482 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:22.490 7f4d9b7fe700 1 -- 172.21.2.112:0/1799897377 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4da4006030 con 0x7f4dac09d410 2021-02-10T11:36:22.482 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:22.491 7f4d9b7fe700 1 -- 172.21.2.112:0/1799897377 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4da40062b0 con 0x7f4dac09d410 2021-02-10T11:36:22.482 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:22.491 7f4d9b7fe700 1 -- 172.21.2.112:0/1799897377 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4da4004720 con 0x7f4dac09d410 2021-02-10T11:36:22.482 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:22.491 7f4db0aef700 1 -- 172.21.2.112:0/1799897377 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4dac09d880 con 0x7f4dac09d410 2021-02-10T11:36:22.482 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:22.491 7f4db0aef700 1 -- 172.21.2.112:0/1799897377 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4dac09d880 con 0x7f4dac09d410 2021-02-10T11:36:22.482 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:22.491 7f4d9b7fe700 1 -- 172.21.2.112:0/1799897377 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4da4006490 con 0x7f4dac09d410 2021-02-10T11:36:22.482 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:22.491 7f4d9b7fe700 1 --2- 172.21.2.112:0/1799897377 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4d900186c0 0x7f4d9001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:22.482 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:22.491 7f4d9b7fe700 1 -- 172.21.2.112:0/1799897377 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f4da4032b20 con 0x7f4dac09d410 2021-02-10T11:36:22.482 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:22.492 7f4daa85e700 1 --2- 172.21.2.112:0/1799897377 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4d900186c0 0x7f4d9001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:22.482 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:22.492 7f4daa85e700 1 --2- 172.21.2.112:0/1799897377 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4d900186c0 0x7f4d9001abc0 crc :-1 s=READY pgs=771 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:22.508 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:22.522 7f4db0aef700 1 -- 172.21.2.112:0/1799897377 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4dac082a70 con 0x7f4dac09d410 2021-02-10T11:36:22.510 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:22.524 7f4d9b7fe700 1 -- 172.21.2.112:0/1799897377 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4da400be10 con 0x7f4dac09d410 2021-02-10T11:36:22.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:22.660 7f4db0aef700 1 -- 172.21.2.112:0/1799897377 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f4dac082a70 con 0x7f4dac09d410 2021-02-10T11:36:23.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:23.116 7f4d9b7fe700 1 -- 172.21.2.112:0/1799897377 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v57) v1 ==== 136+0+0 (secure 0 0 0) 0x7f4da40062b0 con 0x7f4dac09d410 2021-02-10T11:36:23.116 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:23.129 7f4db0aef700 1 -- 172.21.2.112:0/1799897377 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4d900186c0 msgr2=0x7f4d9001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:23.117 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:23.129 7f4db0aef700 1 --2- 172.21.2.112:0/1799897377 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4d900186c0 0x7f4d9001abc0 crc :-1 s=READY pgs=771 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:23.117 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:23.130 7f4db0aef700 1 -- 172.21.2.112:0/1799897377 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4dac09d410 msgr2=0x7f4dac098170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:23.117 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:23.130 7f4db0aef700 1 --2- 172.21.2.112:0/1799897377 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4dac09d410 0x7f4dac098170 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 rx=0x7f4da4009980 tx=0x7f4da400bcd0).stop 2021-02-10T11:36:23.117 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:23.130 7f4db0aef700 1 -- 172.21.2.112:0/1799897377 shutdown_connections 2021-02-10T11:36:23.117 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:23.130 7f4db0aef700 1 --2- 172.21.2.112:0/1799897377 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4d900186c0 0x7f4d9001abc0 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:23.117 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:23.130 7f4db0aef700 1 --2- 172.21.2.112:0/1799897377 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4dac09d410 0x7f4dac098170 unknown :-1 s=CLOSED pgs=1608 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:23.117 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:23.130 7f4db0aef700 1 --2- 172.21.2.112:0/1799897377 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4dac09c9f0 0x7f4dac09cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:23.117 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:23.130 7f4db0aef700 1 --2- 172.21.2.112:0/1799897377 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4dac1a3570 0x7f4dac1a3970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:23.117 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:23.130 7f4db0aef700 1 -- 172.21.2.112:0/1799897377 >> 172.21.2.112:0/1799897377 conn(0x7f4dac0c2cd0 msgr2=0x7f4dac0b5950 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:23.117 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:23.130 7f4db0aef700 1 -- 172.21.2.112:0/1799897377 shutdown_connections 2021-02-10T11:36:23.117 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:23.130 7f4db0aef700 1 -- 172.21.2.112:0/1799897377 wait complete. 2021-02-10T11:36:23.125 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs volume ls 2021-02-10T11:36:23.550 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:23.547 7faf30095700 1 Processor -- start 2021-02-10T11:36:23.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:23.547 7faf30095700 1 -- start start 2021-02-10T11:36:23.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:23.548 7faf30095700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7faf280d1870 0x7faf281b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:23.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:23.548 7faf30095700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7faf281b0a10 0x7faf280cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:23.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:23.548 7faf30095700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7faf280d1c70 0x7faf280cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:23.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:23.548 7faf30095700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf280fe200 con 0x7faf280d1870 2021-02-10T11:36:23.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:23.548 7faf30095700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf280e5f10 con 0x7faf280d1c70 2021-02-10T11:36:23.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:23.548 7faf30095700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf280f0b50 con 0x7faf281b0a10 2021-02-10T11:36:23.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:23.548 7faf2deb1700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7faf280d1870 0x7faf281b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:23.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:23.548 7faf2deb1700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7faf280d1870 0x7faf281b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:51980/0 (socket says 172.21.2.112:51980) 2021-02-10T11:36:23.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:23.548 7faf2deb1700 1 -- 172.21.2.112:0/1930328795 learned_addr learned my addr 172.21.2.112:0/1930328795 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:36:23.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:23.548 7faf2d6b0700 1 --2- 172.21.2.112:0/1930328795 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7faf281b0a10 0x7faf280cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:23.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:23.548 7faf2e6b2700 1 --2- 172.21.2.112:0/1930328795 >> v2:172.21.2.120:3300/0 conn(0x7faf280d1c70 0x7faf280cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:23.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:23.548 7faf2deb1700 1 -- 172.21.2.112:0/1930328795 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7faf281b0a10 msgr2=0x7faf280cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:23.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:23.548 7faf2deb1700 1 --2- 172.21.2.112:0/1930328795 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7faf281b0a10 0x7faf280cf330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:23.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:23.548 7faf2deb1700 1 -- 172.21.2.112:0/1930328795 >> v2:172.21.2.120:3300/0 conn(0x7faf280d1c70 msgr2=0x7faf280cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:23.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:23.548 7faf2deb1700 1 --2- 172.21.2.112:0/1930328795 >> v2:172.21.2.120:3300/0 conn(0x7faf280d1c70 0x7faf280cf950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:23.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:23.548 7faf2deb1700 1 -- 172.21.2.112:0/1930328795 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf280cff70 con 0x7faf280d1870 2021-02-10T11:36:23.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:23.548 7faf2d6b0700 1 --2- 172.21.2.112:0/1930328795 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7faf281b0a10 0x7faf280cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:36:23.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:23.548 7faf2e6b2700 1 --2- 172.21.2.112:0/1930328795 >> v2:172.21.2.120:3300/0 conn(0x7faf280d1c70 0x7faf280cf950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:36:23.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:23.549 7faf2deb1700 1 --2- 172.21.2.112:0/1930328795 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faf280d1870 0x7faf281b03f0 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 rx=0x7faf24006700 tx=0x7faf24009b80).ready entity=mon.0 client_cookie=d11738f352de77da server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:23.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:23.549 7faf2ceaf700 1 -- 172.21.2.112:0/1930328795 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faf2400c0c0 con 0x7faf280d1870 2021-02-10T11:36:23.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:23.549 7faf2ceaf700 1 -- 172.21.2.112:0/1930328795 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf2400c400 con 0x7faf280d1870 2021-02-10T11:36:23.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:23.549 7faf2ceaf700 1 -- 172.21.2.112:0/1930328795 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faf2400c920 con 0x7faf280d1870 2021-02-10T11:36:23.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:23.549 7faf30095700 1 -- 172.21.2.112:0/1930328795 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faf280d1870 msgr2=0x7faf281b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:23.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:23.549 7faf30095700 1 --2- 172.21.2.112:0/1930328795 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faf280d1870 0x7faf281b03f0 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 rx=0x7faf24006700 tx=0x7faf24009b80).stop 2021-02-10T11:36:23.553 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:23.550 7faf30095700 1 -- 172.21.2.112:0/1930328795 shutdown_connections 2021-02-10T11:36:23.553 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:23.550 7faf30095700 1 --2- 172.21.2.112:0/1930328795 >> v2:172.21.2.120:3300/0 conn(0x7faf280d1c70 0x7faf280cf950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:23.553 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:23.550 7faf30095700 1 --2- 172.21.2.112:0/1930328795 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7faf281b0a10 0x7faf280cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:23.553 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:23.550 7faf30095700 1 --2- 172.21.2.112:0/1930328795 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faf280d1870 0x7faf281b03f0 unknown :-1 s=CLOSED pgs=1609 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:23.553 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:23.550 7faf30095700 1 -- 172.21.2.112:0/1930328795 >> 172.21.2.112:0/1930328795 conn(0x7faf280ce450 msgr2=0x7faf280d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:23.553 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:23.550 7faf30095700 1 -- 172.21.2.112:0/1930328795 shutdown_connections 2021-02-10T11:36:23.553 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:23.550 7faf30095700 1 -- 172.21.2.112:0/1930328795 wait complete. 2021-02-10T11:36:23.554 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:23.565 7faf30095700 1 Processor -- start 2021-02-10T11:36:23.554 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:23.565 7faf30095700 1 -- start start 2021-02-10T11:36:23.554 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:23.565 7faf30095700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faf281a06a0 0x7faf2809f380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:23.555 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:23.565 7faf30095700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7faf2809f9a0 0x7faf2808b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:23.555 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:23.565 7faf30095700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7faf2808bb80 0x7faf2808bf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:23.555 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:23.565 7faf30095700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf2808c780 con 0x7faf281a06a0 2021-02-10T11:36:23.555 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:23.565 7faf30095700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf2809fda0 con 0x7faf2808bb80 2021-02-10T11:36:23.555 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:23.565 7faf30095700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf2808cd00 con 0x7faf2809f9a0 2021-02-10T11:36:23.555 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:23.565 7faf2deb1700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faf281a06a0 0x7faf2809f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:23.555 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:23.565 7faf2deb1700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faf281a06a0 0x7faf2809f380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:51986/0 (socket says 172.21.2.112:51986) 2021-02-10T11:36:23.555 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:23.565 7faf2deb1700 1 -- 172.21.2.112:0/3338678470 learned_addr learned my addr 172.21.2.112:0/3338678470 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:36:23.555 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:23.566 7faf2deb1700 1 -- 172.21.2.112:0/3338678470 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7faf2809f9a0 msgr2=0x7faf2808b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:23.555 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:23.566 7faf2deb1700 1 --2- 172.21.2.112:0/3338678470 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7faf2809f9a0 0x7faf2808b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:23.555 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:23.566 7faf2deb1700 1 -- 172.21.2.112:0/3338678470 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7faf2808bb80 msgr2=0x7faf2808bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:23.555 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:23.566 7faf2deb1700 1 --2- 172.21.2.112:0/3338678470 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7faf2808bb80 0x7faf2808bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:23.555 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:23.566 7faf2deb1700 1 -- 172.21.2.112:0/3338678470 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf2808cf20 con 0x7faf281a06a0 2021-02-10T11:36:23.555 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:23.566 7faf2deb1700 1 --2- 172.21.2.112:0/3338678470 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faf281a06a0 0x7faf2809f380 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 rx=0x7faf240066a0 tx=0x7faf2400b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:23.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:23.566 7faf1e7fc700 1 -- 172.21.2.112:0/3338678470 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faf2400bd80 con 0x7faf281a06a0 2021-02-10T11:36:23.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:23.566 7faf1e7fc700 1 -- 172.21.2.112:0/3338678470 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf240052e0 con 0x7faf281a06a0 2021-02-10T11:36:23.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:23.566 7faf1e7fc700 1 -- 172.21.2.112:0/3338678470 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faf24005800 con 0x7faf281a06a0 2021-02-10T11:36:23.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:23.566 7faf30095700 1 -- 172.21.2.112:0/3338678470 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf2808cf20 con 0x7faf281a06a0 2021-02-10T11:36:23.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:23.567 7faf30095700 1 -- 172.21.2.112:0/3338678470 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf2808cf20 con 0x7faf281a06a0 2021-02-10T11:36:23.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:23.567 7faf1e7fc700 1 -- 172.21.2.112:0/3338678470 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7faf24006030 con 0x7faf281a06a0 2021-02-10T11:36:23.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:23.567 7faf1e7fc700 1 --2- 172.21.2.112:0/3338678470 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7faf10018730 0x7faf1001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:23.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:23.567 7faf1e7fc700 1 -- 172.21.2.112:0/3338678470 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5573+0+0 (secure 0 0 0) 0x7faf24032650 con 0x7faf281a06a0 2021-02-10T11:36:23.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:23.567 7faf2d6b0700 1 --2- 172.21.2.112:0/3338678470 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7faf10018730 0x7faf1001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:23.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:23.568 7faf2d6b0700 1 --2- 172.21.2.112:0/3338678470 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7faf10018730 0x7faf1001ac10 crc :-1 s=READY pgs=772 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:23.584 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:23.597 7faf30095700 1 -- 172.21.2.112:0/3338678470 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf280ebc50 con 0x7faf281a06a0 2021-02-10T11:36:23.586 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:23.600 7faf1e7fc700 1 -- 172.21.2.112:0/3338678470 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7faf2401f510 con 0x7faf281a06a0 2021-02-10T11:36:23.721 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:23.734 7faf30095700 1 -- 172.21.2.112:0/3338678470 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs volume ls", "target": ["mgr", ""]}) v1 -- 0x7faf280ebc50 con 0x7faf10018730 2021-02-10T11:36:23.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:23.736 7faf1e7fc700 1 -- 172.21.2.112:0/3338678470 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+41 (crc 0 0 0) 0x7faf180077a0 con 0x7faf10018730 2021-02-10T11:36:23.724 INFO:teuthology.orchestra.run.gibba012.stdout:[ 2021-02-10T11:36:23.724 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:36:23.724 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "cephfs" 2021-02-10T11:36:23.724 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:36:23.724 INFO:teuthology.orchestra.run.gibba012.stdout:] 2021-02-10T11:36:23.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:23.739 7faf30095700 1 -- 172.21.2.112:0/3338678470 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7faf10018730 msgr2=0x7faf1001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:23.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:23.739 7faf30095700 1 --2- 172.21.2.112:0/3338678470 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7faf10018730 0x7faf1001ac10 crc :-1 s=READY pgs=772 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:23.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:23.739 7faf30095700 1 -- 172.21.2.112:0/3338678470 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faf281a06a0 msgr2=0x7faf2809f380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:23.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:23.739 7faf30095700 1 --2- 172.21.2.112:0/3338678470 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faf281a06a0 0x7faf2809f380 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 rx=0x7faf240066a0 tx=0x7faf2400b990).stop 2021-02-10T11:36:23.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:23.739 7faf30095700 1 -- 172.21.2.112:0/3338678470 shutdown_connections 2021-02-10T11:36:23.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:23.739 7faf30095700 1 --2- 172.21.2.112:0/3338678470 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7faf10018730 0x7faf1001ac10 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:23.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:23.739 7faf30095700 1 --2- 172.21.2.112:0/3338678470 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7faf2808bb80 0x7faf2808bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:23.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:23.739 7faf30095700 1 --2- 172.21.2.112:0/3338678470 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7faf2809f9a0 0x7faf2808b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:23.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:23.739 7faf30095700 1 --2- 172.21.2.112:0/3338678470 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faf281a06a0 0x7faf2809f380 unknown :-1 s=CLOSED pgs=1610 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:23.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:23.739 7faf30095700 1 -- 172.21.2.112:0/3338678470 >> 172.21.2.112:0/3338678470 conn(0x7faf280b83b0 msgr2=0x7faf280a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:23.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:23.739 7faf30095700 1 -- 172.21.2.112:0/3338678470 shutdown_connections 2021-02-10T11:36:23.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:23.739 7faf30095700 1 -- 172.21.2.112:0/3338678470 wait complete. 2021-02-10T11:36:23.734 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:36:24.166 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.162 7fbff4343700 1 Processor -- start 2021-02-10T11:36:24.166 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.162 7fbff4343700 1 -- start start 2021-02-10T11:36:24.166 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.163 7fbff4343700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fbfec0d1870 0x7fbfec1b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:24.166 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.163 7fbff4343700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfec0fe1c0 con 0x7fbfec0cff70 2021-02-10T11:36:24.166 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.163 7fbff4343700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfec109a60 con 0x7fbfec1b0ae0 2021-02-10T11:36:24.166 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.163 7fbff4343700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfec088070 con 0x7fbfec0d1870 2021-02-10T11:36:24.166 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.163 7fbff215f700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fbfec0d1870 0x7fbfec1b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:24.166 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.163 7fbff2960700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fbfec0cff70 0x7fbfec109130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:51454/0 (socket says 172.21.2.112:51454) 2021-02-10T11:36:24.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.163 7fbff2960700 1 -- 172.21.2.112:0/658597028 learned_addr learned my addr 172.21.2.112:0/658597028 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:36:24.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.163 7fbff195e700 1 --1- 172.21.2.112:0/658597028 >> v1:172.21.2.120:6789/0 conn(0x7fbfec1b0ae0 0x7fbfec108940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:45012/0 (socket says 172.21.2.112:45012) 2021-02-10T11:36:24.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.163 7fbff115d700 1 -- 172.21.2.112:0/658597028 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 705371779 0 0) 0x7fbfe4000d30 con 0x7fbfec0cff70 2021-02-10T11:36:24.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.163 7fbff115d700 1 -- 172.21.2.112:0/658597028 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbfd8002fe0 con 0x7fbfec0cff70 2021-02-10T11:36:24.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.163 7fbff115d700 1 -- 172.21.2.112:0/658597028 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3958469794 0 0) 0x7fbfdc000c60 con 0x7fbfec1b0ae0 2021-02-10T11:36:24.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.163 7fbff215f700 1 -- 172.21.2.112:0/658597028 >> v1:172.21.2.120:6789/0 conn(0x7fbfec1b0ae0 legacy=0x7fbfec108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:24.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.163 7fbff215f700 1 -- 172.21.2.112:0/658597028 >> v1:172.21.2.112:6789/0 conn(0x7fbfec0cff70 legacy=0x7fbfec109130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:24.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.163 7fbff215f700 1 -- 172.21.2.112:0/658597028 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfec088260 con 0x7fbfec0d1870 2021-02-10T11:36:24.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.164 7fbff215f700 1 --2- 172.21.2.112:0/658597028 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbfec0d1870 0x7fbfec1b04c0 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 rx=0x7fbfe8006650 tx=0x7fbfe8009a80).ready entity=mon.0 client_cookie=bab5e5910d11e59 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:24.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.164 7fbff115d700 1 -- 172.21.2.112:0/658597028 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbfe800c0c0 con 0x7fbfec0d1870 2021-02-10T11:36:24.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.164 7fbff115d700 1 -- 172.21.2.112:0/658597028 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfe800c340 con 0x7fbfec0d1870 2021-02-10T11:36:24.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.164 7fbff115d700 1 -- 172.21.2.112:0/658597028 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbfe800c860 con 0x7fbfec0d1870 2021-02-10T11:36:24.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.164 7fbff4343700 1 -- 172.21.2.112:0/658597028 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbfec0d1870 msgr2=0x7fbfec1b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:24.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.164 7fbff4343700 1 --2- 172.21.2.112:0/658597028 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbfec0d1870 0x7fbfec1b04c0 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 rx=0x7fbfe8006650 tx=0x7fbfe8009a80).stop 2021-02-10T11:36:24.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.165 7fbff4343700 1 -- 172.21.2.112:0/658597028 shutdown_connections 2021-02-10T11:36:24.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.165 7fbff4343700 1 --2- 172.21.2.112:0/658597028 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbfec0d1870 0x7fbfec1b04c0 unknown :-1 s=CLOSED pgs=1612 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:24.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.165 7fbff4343700 1 -- 172.21.2.112:0/658597028 >> 172.21.2.112:0/658597028 conn(0x7fbfec0ce420 msgr2=0x7fbfec0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:24.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.165 7fbff4343700 1 -- 172.21.2.112:0/658597028 shutdown_connections 2021-02-10T11:36:24.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.165 7fbff4343700 1 -- 172.21.2.112:0/658597028 wait complete. 2021-02-10T11:36:24.169 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.180 7fbff4343700 1 Processor -- start 2021-02-10T11:36:24.169 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.180 7fbff4343700 1 -- start start 2021-02-10T11:36:24.169 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.180 7fbff4343700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbfec1a7b70 0x7fbfec1a7f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:24.169 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.180 7fbff4343700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fbfec0ae250 0x7fbfec0ae650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:24.170 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.180 7fbff4343700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbfec0aec70 0x7fbfec0af070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:24.170 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.180 7fbff4343700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfec0adcd0 con 0x7fbfec0aec70 2021-02-10T11:36:24.170 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.180 7fbff4343700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfec0b7e00 con 0x7fbfec0ae250 2021-02-10T11:36:24.170 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.180 7fbff4343700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfec0ac4b0 con 0x7fbfec1a7b70 2021-02-10T11:36:24.170 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.180 7fbff2960700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbfec0aec70 0x7fbfec0af070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:24.170 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.180 7fbff2960700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbfec0aec70 0x7fbfec0af070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:51994/0 (socket says 172.21.2.112:51994) 2021-02-10T11:36:24.170 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.180 7fbff2960700 1 -- 172.21.2.112:0/2110070404 learned_addr learned my addr 172.21.2.112:0/2110070404 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:36:24.170 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.180 7fbff195e700 1 --2- 172.21.2.112:0/2110070404 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fbfec0ae250 0x7fbfec0ae650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:24.170 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.180 7fbff215f700 1 --2- 172.21.2.112:0/2110070404 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbfec1a7b70 0x7fbfec1a7f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:24.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.181 7fbff2960700 1 -- 172.21.2.112:0/2110070404 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbfec1a7b70 msgr2=0x7fbfec1a7f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:24.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.181 7fbff2960700 1 --2- 172.21.2.112:0/2110070404 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbfec1a7b70 0x7fbfec1a7f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:24.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.181 7fbff2960700 1 -- 172.21.2.112:0/2110070404 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fbfec0ae250 msgr2=0x7fbfec0ae650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:24.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.181 7fbff2960700 1 --2- 172.21.2.112:0/2110070404 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fbfec0ae250 0x7fbfec0ae650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:24.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.181 7fbff2960700 1 -- 172.21.2.112:0/2110070404 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfec0ac6d0 con 0x7fbfec0aec70 2021-02-10T11:36:24.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.181 7fbff2960700 1 --2- 172.21.2.112:0/2110070404 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbfec0aec70 0x7fbfec0af070 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 rx=0x7fbfe4009980 tx=0x7fbfe400bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:24.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.181 7fbfe2ffd700 1 -- 172.21.2.112:0/2110070404 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbfe400be10 con 0x7fbfec0aec70 2021-02-10T11:36:24.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.181 7fbfe2ffd700 1 -- 172.21.2.112:0/2110070404 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfe4006080 con 0x7fbfec0aec70 2021-02-10T11:36:24.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.181 7fbfe2ffd700 1 -- 172.21.2.112:0/2110070404 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbfe40064e0 con 0x7fbfec0aec70 2021-02-10T11:36:24.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.181 7fbff4343700 1 -- 172.21.2.112:0/2110070404 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfec0ac6d0 con 0x7fbfec0aec70 2021-02-10T11:36:24.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.181 7fbff4343700 1 -- 172.21.2.112:0/2110070404 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfec0ac6d0 con 0x7fbfec0aec70 2021-02-10T11:36:24.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.182 7fbfe2ffd700 1 -- 172.21.2.112:0/2110070404 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fbfe4006260 con 0x7fbfec0aec70 2021-02-10T11:36:24.172 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.182 7fbfe2ffd700 1 --2- 172.21.2.112:0/2110070404 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fbfd40186c0 0x7fbfd401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:24.172 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.182 7fbfe2ffd700 1 -- 172.21.2.112:0/2110070404 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fbfe4032780 con 0x7fbfec0aec70 2021-02-10T11:36:24.172 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.182 7fbff215f700 1 --2- 172.21.2.112:0/2110070404 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fbfd40186c0 0x7fbfd401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:24.172 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.182 7fbff215f700 1 --2- 172.21.2.112:0/2110070404 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fbfd40186c0 0x7fbfd401abc0 crc :-1 s=READY pgs=773 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:24.199 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.212 7fbff4343700 1 -- 172.21.2.112:0/2110070404 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfec082a70 con 0x7fbfec0aec70 2021-02-10T11:36:24.201 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.215 7fbfe2ffd700 1 -- 172.21.2.112:0/2110070404 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fbfe4000f90 con 0x7fbfec0aec70 2021-02-10T11:36:24.351 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.364 7fbff4343700 1 -- 172.21.2.112:0/2110070404 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fbfec082a70 con 0x7fbfec0aec70 2021-02-10T11:36:24.351 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.364 7fbfe2ffd700 1 -- 172.21.2.112:0/2110070404 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v20)=0 v20) v1 ==== 122+0+0 (secure 0 0 0) 0x7fbfe4006260 con 0x7fbfec0aec70 2021-02-10T11:36:24.353 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.366 7fbff4343700 1 -- 172.21.2.112:0/2110070404 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fbfd40186c0 msgr2=0x7fbfd401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:24.353 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.366 7fbff4343700 1 --2- 172.21.2.112:0/2110070404 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fbfd40186c0 0x7fbfd401abc0 crc :-1 s=READY pgs=773 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:24.353 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.366 7fbff4343700 1 -- 172.21.2.112:0/2110070404 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbfec0aec70 msgr2=0x7fbfec0af070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:24.353 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.366 7fbff4343700 1 --2- 172.21.2.112:0/2110070404 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbfec0aec70 0x7fbfec0af070 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 rx=0x7fbfe4009980 tx=0x7fbfe400bcd0).stop 2021-02-10T11:36:24.353 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.366 7fbff4343700 1 -- 172.21.2.112:0/2110070404 shutdown_connections 2021-02-10T11:36:24.354 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.366 7fbff4343700 1 --2- 172.21.2.112:0/2110070404 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fbfd40186c0 0x7fbfd401abc0 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:24.354 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.366 7fbff4343700 1 --2- 172.21.2.112:0/2110070404 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fbfec0ae250 0x7fbfec0ae650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:24.354 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.366 7fbff4343700 1 --2- 172.21.2.112:0/2110070404 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbfec0aec70 0x7fbfec0af070 unknown :-1 s=CLOSED pgs=1613 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:24.354 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.366 7fbff4343700 1 --2- 172.21.2.112:0/2110070404 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbfec1a7b70 0x7fbfec1a7f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:24.354 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.367 7fbff4343700 1 -- 172.21.2.112:0/2110070404 >> 172.21.2.112:0/2110070404 conn(0x7fbfec0a7410 msgr2=0x7fbfec0946e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:24.354 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.367 7fbff4343700 1 -- 172.21.2.112:0/2110070404 shutdown_connections 2021-02-10T11:36:24.354 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.367 7fbff4343700 1 -- 172.21.2.112:0/2110070404 wait complete. 2021-02-10T11:36:24.362 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000232725 2021-02-10T11:36:24.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.790 7f069ce60700 1 Processor -- start 2021-02-10T11:36:24.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.790 7f069ce60700 1 -- start start 2021-02-10T11:36:24.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.790 7f069ce60700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f06980d1870 0x7f06981b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:24.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.790 7f069ce60700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06980fe1c0 con 0x7f06981b0ae0 2021-02-10T11:36:24.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.790 7f069ce60700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0698109a60 con 0x7f06980cff70 2021-02-10T11:36:24.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.790 7f069ce60700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0698088070 con 0x7f06980d1870 2021-02-10T11:36:24.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.791 7f0696312700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f06981b0ae0 0x7f0698108940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:51466/0 (socket says 172.21.2.112:51466) 2021-02-10T11:36:24.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.791 7f0696312700 1 -- 172.21.2.112:0/4088196583 learned_addr learned my addr 172.21.2.112:0/4088196583 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:36:24.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.791 7f0696b13700 1 --2- 172.21.2.112:0/4088196583 >> v2:172.21.2.120:3300/0 conn(0x7f06980d1870 0x7f06981b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:24.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.791 7f0695b11700 1 -- 172.21.2.112:0/4088196583 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1591672579 0 0) 0x7f06800009b0 con 0x7f06981b0ae0 2021-02-10T11:36:24.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.791 7f0695b11700 1 -- 172.21.2.112:0/4088196583 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0684002fe0 con 0x7f06981b0ae0 2021-02-10T11:36:24.800 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.791 7f0695b11700 1 -- 172.21.2.112:0/4088196583 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3069157216 0 0) 0x7f0680001010 con 0x7f06981b0ae0 2021-02-10T11:36:24.800 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.791 7f0695b11700 1 -- 172.21.2.112:0/4088196583 >> v2:172.21.2.120:3300/0 conn(0x7f06980d1870 msgr2=0x7f06981b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:24.800 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.791 7f0695b11700 1 --2- 172.21.2.112:0/4088196583 >> v2:172.21.2.120:3300/0 conn(0x7f06980d1870 0x7f06981b04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:24.800 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.791 7f0695b11700 1 -- 172.21.2.112:0/4088196583 >> v1:172.21.2.120:6789/0 conn(0x7f06980cff70 legacy=0x7f0698109130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:24.800 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.791 7f0695b11700 1 -- 172.21.2.112:0/4088196583 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0698088260 con 0x7f06981b0ae0 2021-02-10T11:36:24.800 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.791 7f0695b11700 1 -- 172.21.2.112:0/4088196583 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f0680001590 con 0x7f06981b0ae0 2021-02-10T11:36:24.801 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.791 7f0695b11700 1 -- 172.21.2.112:0/4088196583 >> v1:172.21.2.112:6789/0 conn(0x7f06981b0ae0 legacy=0x7f0698108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:24.801 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.791 7f0695b11700 1 --2- 172.21.2.112:0/4088196583 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0684003a70 0x7f0684005f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:24.801 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.791 7f0695b11700 1 --2- 172.21.2.112:0/4088196583 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f06840067a0 0x7f0684008ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:24.801 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.792 7f0695b11700 1 --2- 172.21.2.112:0/4088196583 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f06840095a0 0x7f068400bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:24.801 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.792 7f0695b11700 1 -- 172.21.2.112:0/4088196583 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f068400c310 con 0x7f0684003a70 2021-02-10T11:36:24.801 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.792 7f0695b11700 1 -- 172.21.2.112:0/4088196583 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f068400c530 con 0x7f06840095a0 2021-02-10T11:36:24.802 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.792 7f0695b11700 1 -- 172.21.2.112:0/4088196583 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f068400c750 con 0x7f06840067a0 2021-02-10T11:36:24.802 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.792 7f0696b13700 1 --2- 172.21.2.112:0/4088196583 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0684003a70 0x7f0684005f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:24.802 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.792 7f0696312700 1 --2- 172.21.2.112:0/4088196583 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f06840067a0 0x7f0684008ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:24.802 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.792 7f0696b13700 1 -- 172.21.2.112:0/4088196583 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f06840067a0 msgr2=0x7f0684008ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:24.802 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.792 7f0696b13700 1 --2- 172.21.2.112:0/4088196583 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f06840067a0 0x7f0684008ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:24.802 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.792 7f0697314700 1 --2- 172.21.2.112:0/4088196583 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f06840095a0 0x7f068400bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:24.802 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.792 7f0696b13700 1 -- 172.21.2.112:0/4088196583 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f06840095a0 msgr2=0x7f068400bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:24.803 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.792 7f0696b13700 1 --2- 172.21.2.112:0/4088196583 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f06840095a0 0x7f068400bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:24.803 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.792 7f0696b13700 1 -- 172.21.2.112:0/4088196583 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06840509c0 con 0x7f0684003a70 2021-02-10T11:36:24.803 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.792 7f0697314700 1 --2- 172.21.2.112:0/4088196583 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f06840095a0 0x7f068400bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:36:24.803 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.792 7f0697314700 1 -- 172.21.2.112:0/4088196583 reap_dead start 2021-02-10T11:36:24.803 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.792 7f0696b13700 1 --2- 172.21.2.112:0/4088196583 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0684003a70 0x7f0684005f00 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 rx=0x7f068800c870 tx=0x7f068800cc10).ready entity=mon.0 client_cookie=b355fd84d7991c97 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:24.803 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.792 7f0695b11700 1 -- 172.21.2.112:0/4088196583 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f06880009b0 con 0x7f0684003a70 2021-02-10T11:36:24.804 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.792 7f0695b11700 1 -- 172.21.2.112:0/4088196583 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06880090d0 con 0x7f0684003a70 2021-02-10T11:36:24.804 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.792 7f0695b11700 1 -- 172.21.2.112:0/4088196583 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0688009520 con 0x7f0684003a70 2021-02-10T11:36:24.804 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.792 7f069ce60700 1 -- 172.21.2.112:0/4088196583 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0684003a70 msgr2=0x7f0684005f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:24.804 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.792 7f069ce60700 1 --2- 172.21.2.112:0/4088196583 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0684003a70 0x7f0684005f00 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 rx=0x7f068800c870 tx=0x7f068800cc10).stop 2021-02-10T11:36:24.804 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.793 7f069ce60700 1 -- 172.21.2.112:0/4088196583 shutdown_connections 2021-02-10T11:36:24.804 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.793 7f069ce60700 1 --2- 172.21.2.112:0/4088196583 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0684003a70 0x7f0684005f00 unknown :-1 s=CLOSED pgs=1615 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:24.805 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.793 7f069ce60700 1 -- 172.21.2.112:0/4088196583 >> 172.21.2.112:0/4088196583 conn(0x7f06980ce420 msgr2=0x7f06980d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:24.805 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.793 7f069ce60700 1 -- 172.21.2.112:0/4088196583 shutdown_connections 2021-02-10T11:36:24.805 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.793 7f069ce60700 1 -- 172.21.2.112:0/4088196583 wait complete. 2021-02-10T11:36:24.805 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.807 7f069ce60700 1 Processor -- start 2021-02-10T11:36:24.805 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.808 7f069ce60700 1 -- start start 2021-02-10T11:36:24.806 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.808 7f069ce60700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f069809bcc0 0x7f069809c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:24.806 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.808 7f069ce60700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f06980a0820 0x7f06980a0c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:24.806 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.808 7f069ce60700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f06980a1240 0x7f06980858d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:24.806 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.808 7f069ce60700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06980a66b0 con 0x7f06980a1240 2021-02-10T11:36:24.806 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.808 7f069ce60700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06981a49b0 con 0x7f069809bcc0 2021-02-10T11:36:24.806 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.808 7f069ce60700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06981a4b90 con 0x7f06980a0820 2021-02-10T11:36:24.807 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.808 7f0697314700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f06980a1240 0x7f06980858d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:24.807 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.808 7f0697314700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f06980a1240 0x7f06980858d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:52016/0 (socket says 172.21.2.112:52016) 2021-02-10T11:36:24.807 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.808 7f0697314700 1 -- 172.21.2.112:0/2683294394 learned_addr learned my addr 172.21.2.112:0/2683294394 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:36:24.807 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.808 7f0696312700 1 --2- 172.21.2.112:0/2683294394 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f06980a0820 0x7f06980a0c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:24.807 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.808 7f0696b13700 1 --2- 172.21.2.112:0/2683294394 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f069809bcc0 0x7f069809c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:24.807 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.809 7f0697314700 1 -- 172.21.2.112:0/2683294394 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f06980a0820 msgr2=0x7f06980a0c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:24.808 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.809 7f0697314700 1 --2- 172.21.2.112:0/2683294394 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f06980a0820 0x7f06980a0c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:24.808 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.809 7f0697314700 1 -- 172.21.2.112:0/2683294394 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f069809bcc0 msgr2=0x7f069809c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:24.808 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.809 7f0697314700 1 --2- 172.21.2.112:0/2683294394 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f069809bcc0 0x7f069809c0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:24.808 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.809 7f0697314700 1 -- 172.21.2.112:0/2683294394 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06980a1640 con 0x7f06980a1240 2021-02-10T11:36:24.808 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.809 7f0697314700 1 --2- 172.21.2.112:0/2683294394 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f06980a1240 0x7f06980858d0 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 rx=0x7f06900099a0 tx=0x7f069000bcc0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:24.809 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.809 7f068f7fe700 1 -- 172.21.2.112:0/2683294394 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0690006030 con 0x7f06980a1240 2021-02-10T11:36:24.809 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.809 7f068f7fe700 1 -- 172.21.2.112:0/2683294394 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0690003020 con 0x7f06980a1240 2021-02-10T11:36:24.809 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.809 7f069ce60700 1 -- 172.21.2.112:0/2683294394 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06980a1640 con 0x7f06980a1240 2021-02-10T11:36:24.809 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.809 7f069ce60700 1 -- 172.21.2.112:0/2683294394 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06980861e0 con 0x7f06980a1240 2021-02-10T11:36:24.810 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.809 7f068f7fe700 1 -- 172.21.2.112:0/2683294394 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0690006520 con 0x7f06980a1240 2021-02-10T11:36:24.810 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.810 7f068f7fe700 1 -- 172.21.2.112:0/2683294394 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0690006030 con 0x7f06980a1240 2021-02-10T11:36:24.810 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.810 7f068f7fe700 1 --2- 172.21.2.112:0/2683294394 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f067c018730 0x7f067c01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:24.810 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.810 7f068f7fe700 1 -- 172.21.2.112:0/2683294394 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f0690032b30 con 0x7f06980a1240 2021-02-10T11:36:24.810 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.810 7f0696b13700 1 --2- 172.21.2.112:0/2683294394 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f067c018730 0x7f067c01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:24.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.811 7f0696b13700 1 --2- 172.21.2.112:0/2683294394 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f067c018730 0x7f067c01ac10 crc :-1 s=READY pgs=774 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:24.827 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.840 7f069ce60700 1 -- 172.21.2.112:0/2683294394 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06980fe1c0 con 0x7f06980a1240 2021-02-10T11:36:24.831 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.843 7f068f7fe700 1 -- 172.21.2.112:0/2683294394 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f069000be00 con 0x7f06980a1240 2021-02-10T11:36:24.954 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:24.968 7f069ce60700 1 -- 172.21.2.112:0/2683294394 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000232725", "target": ["mgr", ""]}) v1 -- 0x7f06980861e0 con 0x7f067c018730 2021-02-10T11:36:24.959 INFO:tasks.ceph.mgr.x.gibba020.stderr:2021-02-10 11:36:24.969 7f6382eee700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:36:24.959 INFO:tasks.ceph.mgr.x.gibba020.stderr:2021-02-10 11:36:24.969 7f6382eee700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:36:24.959 INFO:tasks.ceph.mgr.x.gibba020.stderr:2021-02-10 11:36:24.969 7f6382eee700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:36:24.959 INFO:tasks.ceph.mgr.x.gibba020.stderr:2021-02-10 11:36:24.969 7f6382eee700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:36:24.959 INFO:tasks.ceph.mgr.x.gibba020.stderr:2021-02-10 11:36:24.969 7f6382eee700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:36:24.990 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:25.003 7f068f7fe700 1 -- 172.21.2.112:0/2683294394 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f0688006440 con 0x7f067c018730 2021-02-10T11:36:25.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:25.034 7f069ce60700 1 -- 172.21.2.112:0/2683294394 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f067c018730 msgr2=0x7f067c01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:25.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:25.034 7f069ce60700 1 --2- 172.21.2.112:0/2683294394 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f067c018730 0x7f067c01ac10 crc :-1 s=READY pgs=774 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:25.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:25.034 7f069ce60700 1 -- 172.21.2.112:0/2683294394 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f06980a1240 msgr2=0x7f06980858d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:25.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:25.034 7f069ce60700 1 --2- 172.21.2.112:0/2683294394 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f06980a1240 0x7f06980858d0 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 rx=0x7f06900099a0 tx=0x7f069000bcc0).stop 2021-02-10T11:36:25.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:25.034 7f069ce60700 1 -- 172.21.2.112:0/2683294394 shutdown_connections 2021-02-10T11:36:25.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:25.034 7f069ce60700 1 --2- 172.21.2.112:0/2683294394 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f067c018730 0x7f067c01ac10 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:25.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:25.034 7f069ce60700 1 --2- 172.21.2.112:0/2683294394 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f06980a1240 0x7f06980858d0 unknown :-1 s=CLOSED pgs=1616 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:25.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:25.034 7f069ce60700 1 --2- 172.21.2.112:0/2683294394 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f06980a0820 0x7f06980a0c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:25.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:25.034 7f069ce60700 1 --2- 172.21.2.112:0/2683294394 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f069809bcc0 0x7f069809c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:25.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:25.034 7f069ce60700 1 -- 172.21.2.112:0/2683294394 >> 172.21.2.112:0/2683294394 conn(0x7f06980bef00 msgr2=0x7f069809d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:25.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:25.034 7f069ce60700 1 -- 172.21.2.112:0/2683294394 shutdown_connections 2021-02-10T11:36:25.024 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:25.034 7f069ce60700 1 -- 172.21.2.112:0/2683294394 wait complete. 2021-02-10T11:36:25.030 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000232725 2021-02-10T11:36:25.464 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:25.459 7f87f8f41700 1 Processor -- start 2021-02-10T11:36:25.464 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:25.459 7f87f8f41700 1 -- start start 2021-02-10T11:36:25.464 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:25.459 7f87f8f41700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f87f40d1870 0x7f87f41b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:25.464 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:25.459 7f87f8f41700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f87f41b0ae0 0x7f87f4108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:25.464 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:25.459 7f87f8f41700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87f40fe1c0 con 0x7f87f40d1c70 2021-02-10T11:36:25.464 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:25.459 7f87f8f41700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87f4109a60 con 0x7f87f40d1870 2021-02-10T11:36:25.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:25.459 7f87f8f41700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87f4109750 con 0x7f87f41b0ae0 2021-02-10T11:36:25.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:25.460 7f87f3314700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f87f40d1c70 0x7f87f4108f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:51484/0 (socket says 172.21.2.112:51484) 2021-02-10T11:36:25.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:25.460 7f87f3314700 1 -- 172.21.2.112:0/570538980 learned_addr learned my addr 172.21.2.112:0/570538980 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:36:25.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:25.460 7f87f1b11700 1 -- 172.21.2.112:0/570538980 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3742038613 0 0) 0x7f87ec001500 con 0x7f87f40d1c70 2021-02-10T11:36:25.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:25.460 7f87f1b11700 1 -- 172.21.2.112:0/570538980 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87dc002fe0 con 0x7f87f40d1c70 2021-02-10T11:36:25.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:25.460 7f87f2b13700 1 --2- 172.21.2.112:0/570538980 >> v2:172.21.2.120:3300/0 conn(0x7f87f40d1870 0x7f87f41b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:25.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:25.460 7f87f2312700 1 --2- 172.21.2.112:0/570538980 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f87f41b0ae0 0x7f87f4108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:25.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:25.461 7f87f1b11700 1 -- 172.21.2.112:0/570538980 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 710874389 0 0) 0x7f87ec001500 con 0x7f87f40d1c70 2021-02-10T11:36:25.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:25.461 7f87f1b11700 1 -- 172.21.2.112:0/570538980 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f87f41b0ae0 msgr2=0x7f87f4108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:25.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:25.461 7f87f1b11700 1 --2- 172.21.2.112:0/570538980 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f87f41b0ae0 0x7f87f4108940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:25.466 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:25.461 7f87f1b11700 1 -- 172.21.2.112:0/570538980 >> v2:172.21.2.120:3300/0 conn(0x7f87f40d1870 msgr2=0x7f87f41b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:25.466 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:25.461 7f87f1b11700 1 --2- 172.21.2.112:0/570538980 >> v2:172.21.2.120:3300/0 conn(0x7f87f40d1870 0x7f87f41b04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:25.466 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:25.461 7f87f1b11700 1 -- 172.21.2.112:0/570538980 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87f4088070 con 0x7f87f40d1c70 2021-02-10T11:36:25.466 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:25.461 7f87f1b11700 1 -- 172.21.2.112:0/570538980 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f87ec001a00 con 0x7f87f40d1c70 2021-02-10T11:36:25.466 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:25.461 7f87f2b13700 1 --2- 172.21.2.112:0/570538980 >> v2:172.21.2.120:3300/0 conn(0x7f87f40d1870 0x7f87f41b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:36:25.466 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:25.461 7f87f1b11700 1 -- 172.21.2.112:0/570538980 >> v1:172.21.2.112:6789/0 conn(0x7f87f40d1c70 legacy=0x7f87f4108f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:25.466 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:25.461 7f87f1b11700 1 --2- 172.21.2.112:0/570538980 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f87dc047aa0 0x7f87dc049f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:25.466 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:25.461 7f87f1b11700 1 --2- 172.21.2.112:0/570538980 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f87dc04a790 0x7f87dc04cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:25.466 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:25.461 7f87f1b11700 1 --2- 172.21.2.112:0/570538980 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f87dc04d500 0x7f87dc04fa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:25.466 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:25.461 7f87f1b11700 1 -- 172.21.2.112:0/570538980 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87dc0502d0 con 0x7f87dc047aa0 2021-02-10T11:36:25.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:25.461 7f87f1b11700 1 -- 172.21.2.112:0/570538980 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87dc0504f0 con 0x7f87dc04d500 2021-02-10T11:36:25.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:25.461 7f87f1b11700 1 -- 172.21.2.112:0/570538980 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87dc050710 con 0x7f87dc04a790 2021-02-10T11:36:25.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:25.461 7f87f2b13700 1 --2- 172.21.2.112:0/570538980 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f87dc047aa0 0x7f87dc049f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:25.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:25.461 7f87f2312700 1 --2- 172.21.2.112:0/570538980 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f87dc04a790 0x7f87dc04cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:25.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:25.461 7f87f3314700 1 --2- 172.21.2.112:0/570538980 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f87dc04d500 0x7f87dc04fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:25.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:25.461 7f87f2b13700 1 -- 172.21.2.112:0/570538980 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f87dc04a790 msgr2=0x7f87dc04cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:25.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:25.461 7f87f2b13700 1 --2- 172.21.2.112:0/570538980 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f87dc04a790 0x7f87dc04cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:25.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:25.461 7f87f2b13700 1 -- 172.21.2.112:0/570538980 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f87dc04d500 msgr2=0x7f87dc04fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:25.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:25.461 7f87f2b13700 1 --2- 172.21.2.112:0/570538980 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f87dc04d500 0x7f87dc04fa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:25.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:25.461 7f87f2b13700 1 -- 172.21.2.112:0/570538980 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87dc0509b0 con 0x7f87dc047aa0 2021-02-10T11:36:25.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:25.462 7f87f2b13700 1 --2- 172.21.2.112:0/570538980 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f87dc047aa0 0x7f87dc049f70 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 rx=0x7f87e4006b00 tx=0x7f87e400ce20).ready entity=mon.0 client_cookie=c9e3e7a7fb6f4e24 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:25.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:25.462 7f87f1b11700 1 -- 172.21.2.112:0/570538980 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f87e4009310 con 0x7f87dc047aa0 2021-02-10T11:36:25.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:25.462 7f87f1b11700 1 -- 172.21.2.112:0/570538980 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87e40068d0 con 0x7f87dc047aa0 2021-02-10T11:36:25.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:25.462 7f87f1b11700 1 -- 172.21.2.112:0/570538980 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f87e4009890 con 0x7f87dc047aa0 2021-02-10T11:36:25.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:25.462 7f87f8f41700 1 -- 172.21.2.112:0/570538980 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f87dc047aa0 msgr2=0x7f87dc049f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:25.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:25.462 7f87f8f41700 1 --2- 172.21.2.112:0/570538980 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f87dc047aa0 0x7f87dc049f70 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 rx=0x7f87e4006b00 tx=0x7f87e400ce20).stop 2021-02-10T11:36:25.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:25.462 7f87f3314700 1 -- 172.21.2.112:0/570538980 reap_dead start 2021-02-10T11:36:25.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:25.463 7f87f8f41700 1 -- 172.21.2.112:0/570538980 shutdown_connections 2021-02-10T11:36:25.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:25.463 7f87f8f41700 1 -- 172.21.2.112:0/570538980 >> 172.21.2.112:0/570538980 conn(0x7f87f40ce420 msgr2=0x7f87f40d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:25.469 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:25.463 7f87f8f41700 1 -- 172.21.2.112:0/570538980 shutdown_connections 2021-02-10T11:36:25.469 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:25.463 7f87f8f41700 1 -- 172.21.2.112:0/570538980 wait complete. 2021-02-10T11:36:25.469 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:25.477 7f87f8f41700 1 Processor -- start 2021-02-10T11:36:25.469 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:25.477 7f87f8f41700 1 -- start start 2021-02-10T11:36:25.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:25.478 7f87f8f41700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f87f4085bc0 0x7f87f40863e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:25.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:25.478 7f87f8f41700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f87f40954c0 0x7f87f4096300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:25.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:25.478 7f87f8f41700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f87f40958c0 0x7f87f4096920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:25.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:25.478 7f87f8f41700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87f4095d80 con 0x7f87f40954c0 2021-02-10T11:36:25.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:25.478 7f87f8f41700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87f40bfa60 con 0x7f87f40958c0 2021-02-10T11:36:25.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:25.478 7f87f8f41700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87f4085fc0 con 0x7f87f4085bc0 2021-02-10T11:36:25.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:25.478 7f87f2312700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f87f40954c0 0x7f87f4096300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:25.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:25.478 7f87f2312700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f87f40954c0 0x7f87f4096300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:52036/0 (socket says 172.21.2.112:52036) 2021-02-10T11:36:25.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:25.478 7f87f2312700 1 -- 172.21.2.112:0/111927489 learned_addr learned my addr 172.21.2.112:0/111927489 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:36:25.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:25.478 7f87f2b13700 1 --2- 172.21.2.112:0/111927489 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f87f4085bc0 0x7f87f40863e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:25.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:25.478 7f87f3314700 1 --2- 172.21.2.112:0/111927489 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f87f40958c0 0x7f87f4096920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:25.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:25.478 7f87f2b13700 1 -- 172.21.2.112:0/111927489 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f87f40958c0 msgr2=0x7f87f4096920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:25.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:25.478 7f87f2b13700 1 --2- 172.21.2.112:0/111927489 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f87f40958c0 0x7f87f4096920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:25.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:25.478 7f87f2b13700 1 -- 172.21.2.112:0/111927489 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f87f40954c0 msgr2=0x7f87f4096300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:25.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:25.478 7f87f2b13700 1 --2- 172.21.2.112:0/111927489 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f87f40954c0 0x7f87f4096300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:25.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:25.479 7f87f2b13700 1 -- 172.21.2.112:0/111927489 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87f4086c10 con 0x7f87f4085bc0 2021-02-10T11:36:25.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:25.479 7f87f2312700 1 --2- 172.21.2.112:0/111927489 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f87f40954c0 0x7f87f4096300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:36:25.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:25.479 7f87f2b13700 1 --2- 172.21.2.112:0/111927489 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f87f4085bc0 0x7f87f40863e0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f87e40099f0 tx=0x7f87e4009d90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:25.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:25.479 7f87e37fe700 1 -- 172.21.2.112:0/111927489 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f87e40009b0 con 0x7f87f4085bc0 2021-02-10T11:36:25.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:25.479 7f87e37fe700 1 -- 172.21.2.112:0/111927489 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87e400ca90 con 0x7f87f4085bc0 2021-02-10T11:36:25.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:25.479 7f87e37fe700 1 -- 172.21.2.112:0/111927489 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f87e4002030 con 0x7f87f4085bc0 2021-02-10T11:36:25.473 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:25.479 7f87f8f41700 1 -- 172.21.2.112:0/111927489 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87f4086c10 con 0x7f87f4085bc0 2021-02-10T11:36:25.473 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:25.479 7f87f8f41700 1 -- 172.21.2.112:0/111927489 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87f4086c10 con 0x7f87f4085bc0 2021-02-10T11:36:25.473 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:25.480 7f87e37fe700 1 -- 172.21.2.112:0/111927489 <== mon.2 v2:172.21.2.120:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f87e400b300 con 0x7f87f4085bc0 2021-02-10T11:36:25.473 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:25.480 7f87e37fe700 1 --2- 172.21.2.112:0/111927489 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f87d80186c0 0x7f87d801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:25.473 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:25.480 7f87e37fe700 1 -- 172.21.2.112:0/111927489 <== mon.2 v2:172.21.2.120:3301/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f87e4032470 con 0x7f87f4085bc0 2021-02-10T11:36:25.473 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:25.480 7f87f2312700 1 --2- 172.21.2.112:0/111927489 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f87d80186c0 0x7f87d801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:25.473 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:25.480 7f87f2312700 1 --2- 172.21.2.112:0/111927489 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f87d80186c0 0x7f87d801abc0 crc :-1 s=READY pgs=775 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:25.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:25.509 7f87f8f41700 1 -- 172.21.2.112:0/111927489 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87f40fe1c0 con 0x7f87f4085bc0 2021-02-10T11:36:25.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:25.513 7f87e37fe700 1 -- 172.21.2.112:0/111927489 <== mon.2 v2:172.21.2.120:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f87e4000c90 con 0x7f87f4085bc0 2021-02-10T11:36:25.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:25.645 7f87f8f41700 1 -- 172.21.2.112:0/111927489 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000232725", "target": ["mgr", ""]}) v1 -- 0x7f87f40fe1c0 con 0x7f87d80186c0 2021-02-10T11:36:25.635 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:25.648 7f87e37fe700 1 -- 172.21.2.112:0/111927489 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f87e8000940 con 0x7f87d80186c0 2021-02-10T11:36:25.639 INFO:teuthology.orchestra.run.gibba012.stdout:/volumes/_nogroup/subvolume_0000000000232725/934c1bce-fa93-4119-a506-af6a0edaf169 2021-02-10T11:36:25.640 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:25.654 7f87f8f41700 1 -- 172.21.2.112:0/111927489 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f87d80186c0 msgr2=0x7f87d801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:25.640 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:25.654 7f87f8f41700 1 --2- 172.21.2.112:0/111927489 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f87d80186c0 0x7f87d801abc0 crc :-1 s=READY pgs=775 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:25.640 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:25.654 7f87f8f41700 1 -- 172.21.2.112:0/111927489 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f87f4085bc0 msgr2=0x7f87f40863e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:25.641 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:25.654 7f87f8f41700 1 --2- 172.21.2.112:0/111927489 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f87f4085bc0 0x7f87f40863e0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f87e40099f0 tx=0x7f87e4009d90).stop 2021-02-10T11:36:25.641 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:25.654 7f87f8f41700 1 -- 172.21.2.112:0/111927489 shutdown_connections 2021-02-10T11:36:25.641 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:25.654 7f87f8f41700 1 --2- 172.21.2.112:0/111927489 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f87d80186c0 0x7f87d801abc0 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:25.641 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:25.654 7f87f8f41700 1 --2- 172.21.2.112:0/111927489 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f87f40958c0 0x7f87f4096920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:25.641 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:25.654 7f87f8f41700 1 --2- 172.21.2.112:0/111927489 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f87f40954c0 0x7f87f4096300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:25.641 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:25.654 7f87f8f41700 1 --2- 172.21.2.112:0/111927489 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f87f4085bc0 0x7f87f40863e0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:25.641 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:25.654 7f87f8f41700 1 -- 172.21.2.112:0/111927489 >> 172.21.2.112:0/111927489 conn(0x7f87f40bef00 msgr2=0x7f87f409d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:25.641 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:25.654 7f87f8f41700 1 -- 172.21.2.112:0/111927489 shutdown_connections 2021-02-10T11:36:25.641 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:25.654 7f87f8f41700 1 -- 172.21.2.112:0/111927489 wait complete. 2021-02-10T11:36:25.649 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000232725 with 1 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000232725/934c1bce-fa93-4119-a506-af6a0edaf169 2021-02-10T11:36:25.649 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:25.649 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:36:25.649 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:36:25.984 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:36:25.984 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:36:25.985 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000232725/934c1bce-fa93-4119-a506-af6a0edaf169/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:36:26.370 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:36:26.370 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:36:26.370 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0278882 s, 37.6 MB/s 2021-02-10T11:36:26.371 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume snapshot create cephfs subvolume_0000000000232725 snapshot_0000000000664158 2021-02-10T11:36:26.801 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:26.797 7f9c54647700 1 Processor -- start 2021-02-10T11:36:26.801 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:26.797 7f9c54647700 1 -- start start 2021-02-10T11:36:26.801 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:26.798 7f9c54647700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f9c4c0d1880 0x7f9c4c1b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:26.801 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:26.798 7f9c54647700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9c4c0cff80 0x7f9c4c109140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:26.801 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:26.798 7f9c54647700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c4c109a70 con 0x7f9c4c1b0af0 2021-02-10T11:36:26.801 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:26.798 7f9c54647700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c4c109760 con 0x7f9c4c0d1880 2021-02-10T11:36:26.801 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:26.798 7f9c54647700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c4c088070 con 0x7f9c4c0cff80 2021-02-10T11:36:26.801 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:26.798 7f9c51c62700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f9c4c1b0af0 0x7f9c4c108950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:51506/0 (socket says 172.21.2.112:51506) 2021-02-10T11:36:26.802 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:26.798 7f9c51c62700 1 -- 172.21.2.112:0/1548985068 learned_addr learned my addr 172.21.2.112:0/1548985068 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:36:26.802 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:26.798 7f9c52c64700 1 --2- 172.21.2.112:0/1548985068 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9c4c0cff80 0x7f9c4c109140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:26.802 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:26.798 7f9c52463700 1 --2- 172.21.2.112:0/1548985068 >> v2:172.21.2.120:3300/0 conn(0x7f9c4c0d1880 0x7f9c4c1b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:26.802 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:26.799 7f9c51461700 1 -- 172.21.2.112:0/1548985068 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2219775080 0 0) 0x7f9c480009b0 con 0x7f9c4c1b0af0 2021-02-10T11:36:26.802 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:26.799 7f9c52463700 1 -- 172.21.2.112:0/1548985068 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9c4c0cff80 msgr2=0x7f9c4c109140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:26.802 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:26.799 7f9c52463700 1 --2- 172.21.2.112:0/1548985068 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9c4c0cff80 0x7f9c4c109140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:26.802 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:26.799 7f9c52463700 1 -- 172.21.2.112:0/1548985068 >> v1:172.21.2.112:6789/0 conn(0x7f9c4c1b0af0 legacy=0x7f9c4c108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:26.802 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:26.799 7f9c52463700 1 -- 172.21.2.112:0/1548985068 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c4c0d1c80 con 0x7f9c4c0d1880 2021-02-10T11:36:26.802 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:26.799 7f9c52463700 1 --2- 172.21.2.112:0/1548985068 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9c4c0d1880 0x7f9c4c1b04d0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 rx=0x7f9c40006650 tx=0x7f9c40009a80).ready entity=mon.1 client_cookie=bf2bccf52c8db0c0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:26.802 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:26.799 7f9c51461700 1 -- 172.21.2.112:0/1548985068 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9c400009b0 con 0x7f9c4c0d1880 2021-02-10T11:36:26.802 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:26.799 7f9c51461700 1 -- 172.21.2.112:0/1548985068 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c4000c0c0 con 0x7f9c4c0d1880 2021-02-10T11:36:26.802 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:26.799 7f9c51461700 1 -- 172.21.2.112:0/1548985068 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9c4000c550 con 0x7f9c4c0d1880 2021-02-10T11:36:26.803 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:26.799 7f9c54647700 1 -- 172.21.2.112:0/1548985068 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9c4c0d1880 msgr2=0x7f9c4c1b04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:26.803 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:26.799 7f9c54647700 1 --2- 172.21.2.112:0/1548985068 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9c4c0d1880 0x7f9c4c1b04d0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 rx=0x7f9c40006650 tx=0x7f9c40009a80).stop 2021-02-10T11:36:26.803 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:26.800 7f9c54647700 1 -- 172.21.2.112:0/1548985068 shutdown_connections 2021-02-10T11:36:26.803 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:26.800 7f9c54647700 1 --2- 172.21.2.112:0/1548985068 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9c4c0cff80 0x7f9c4c109140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:26.803 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:26.800 7f9c54647700 1 --2- 172.21.2.112:0/1548985068 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9c4c0d1880 0x7f9c4c1b04d0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:26.803 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:26.800 7f9c54647700 1 -- 172.21.2.112:0/1548985068 >> 172.21.2.112:0/1548985068 conn(0x7f9c4c0ce430 msgr2=0x7f9c4c0d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:26.803 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:26.800 7f9c54647700 1 -- 172.21.2.112:0/1548985068 shutdown_connections 2021-02-10T11:36:26.803 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:26.800 7f9c54647700 1 -- 172.21.2.112:0/1548985068 wait complete. 2021-02-10T11:36:26.805 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:26.815 7f9c54647700 1 Processor -- start 2021-02-10T11:36:26.805 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:26.815 7f9c54647700 1 -- start start 2021-02-10T11:36:26.805 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:26.815 7f9c54647700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9c4c095710 0x7f9c4c19dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:26.805 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:26.815 7f9c54647700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9c4c0aeeb0 0x7f9c4c19e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:26.805 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:26.815 7f9c54647700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9c4c19e900 0x7f9c4c1b7990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:26.805 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:26.815 7f9c54647700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c4c19ed00 con 0x7f9c4c095710 2021-02-10T11:36:26.805 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:26.815 7f9c54647700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c4c094050 con 0x7f9c4c0aeeb0 2021-02-10T11:36:26.805 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:26.815 7f9c54647700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c4c1b7fb0 con 0x7f9c4c19e900 2021-02-10T11:36:26.806 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:26.815 7f9c52463700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9c4c095710 0x7f9c4c19dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:26.806 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:26.815 7f9c52463700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9c4c095710 0x7f9c4c19dcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:52046/0 (socket says 172.21.2.112:52046) 2021-02-10T11:36:26.806 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:26.815 7f9c52463700 1 -- 172.21.2.112:0/3919978498 learned_addr learned my addr 172.21.2.112:0/3919978498 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:36:26.806 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:26.816 7f9c51c62700 1 --2- 172.21.2.112:0/3919978498 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9c4c0aeeb0 0x7f9c4c19e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:26.806 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:26.816 7f9c52c64700 1 --2- 172.21.2.112:0/3919978498 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9c4c19e900 0x7f9c4c1b7990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:26.806 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:26.816 7f9c52463700 1 -- 172.21.2.112:0/3919978498 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9c4c19e900 msgr2=0x7f9c4c1b7990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:26.806 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:26.816 7f9c52463700 1 --2- 172.21.2.112:0/3919978498 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9c4c19e900 0x7f9c4c1b7990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:26.806 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:26.816 7f9c52463700 1 -- 172.21.2.112:0/3919978498 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9c4c0aeeb0 msgr2=0x7f9c4c19e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:26.806 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:26.816 7f9c52463700 1 --2- 172.21.2.112:0/3919978498 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9c4c0aeeb0 0x7f9c4c19e2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:26.806 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:26.816 7f9c52463700 1 -- 172.21.2.112:0/3919978498 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c4c1b81d0 con 0x7f9c4c095710 2021-02-10T11:36:26.806 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:26.816 7f9c52c64700 1 --2- 172.21.2.112:0/3919978498 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9c4c19e900 0x7f9c4c1b7990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:36:26.806 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:26.816 7f9c52463700 1 --2- 172.21.2.112:0/3919978498 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9c4c095710 0x7f9c4c19dcc0 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 rx=0x7f9c4000b5e0 tx=0x7f9c4000b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:26.807 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:26.816 7f9c3effd700 1 -- 172.21.2.112:0/3919978498 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9c4000bd70 con 0x7f9c4c095710 2021-02-10T11:36:26.807 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:26.816 7f9c3effd700 1 -- 172.21.2.112:0/3919978498 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c40009290 con 0x7f9c4c095710 2021-02-10T11:36:26.807 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:26.816 7f9c54647700 1 -- 172.21.2.112:0/3919978498 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c4c1b81d0 con 0x7f9c4c095710 2021-02-10T11:36:26.807 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:26.816 7f9c3effd700 1 -- 172.21.2.112:0/3919978498 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9c40002300 con 0x7f9c4c095710 2021-02-10T11:36:26.807 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:26.816 7f9c54647700 1 -- 172.21.2.112:0/3919978498 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c4c1b81d0 con 0x7f9c4c095710 2021-02-10T11:36:26.807 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:26.817 7f9c3effd700 1 -- 172.21.2.112:0/3919978498 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9c40000c90 con 0x7f9c4c095710 2021-02-10T11:36:26.807 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:26.817 7f9c3effd700 1 --2- 172.21.2.112:0/3919978498 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9c34018730 0x7f9c3401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:26.807 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:26.817 7f9c3effd700 1 -- 172.21.2.112:0/3919978498 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f9c40032650 con 0x7f9c4c095710 2021-02-10T11:36:26.807 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:26.817 7f9c51c62700 1 --2- 172.21.2.112:0/3919978498 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9c34018730 0x7f9c3401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:26.807 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:26.818 7f9c51c62700 1 --2- 172.21.2.112:0/3919978498 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9c34018730 0x7f9c3401ac10 crc :-1 s=READY pgs=776 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:26.834 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:26.848 7f9c54647700 1 -- 172.21.2.112:0/3919978498 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c4c082aa0 con 0x7f9c4c095710 2021-02-10T11:36:26.836 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:26.850 7f9c3effd700 1 -- 172.21.2.112:0/3919978498 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9c40009290 con 0x7f9c4c095710 2021-02-10T11:36:26.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:26.978 7f9c54647700 1 -- 172.21.2.112:0/3919978498 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume snapshot create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000232725", "target": ["mgr", ""], "snap_name": "snapshot_0000000000664158"}) v1 -- 0x7f9c4c19ed00 con 0x7f9c34018730 2021-02-10T11:36:26.975 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:26.988 7f9c3effd700 1 -- 172.21.2.112:0/3919978498 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f9c480009b0 con 0x7f9c34018730 2021-02-10T11:36:26.982 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:26.996 7f9c54647700 1 -- 172.21.2.112:0/3919978498 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9c34018730 msgr2=0x7f9c3401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:26.982 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:26.996 7f9c54647700 1 --2- 172.21.2.112:0/3919978498 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9c34018730 0x7f9c3401ac10 crc :-1 s=READY pgs=776 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:26.982 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:26.996 7f9c54647700 1 -- 172.21.2.112:0/3919978498 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9c4c095710 msgr2=0x7f9c4c19dcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:26.983 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:26.996 7f9c54647700 1 --2- 172.21.2.112:0/3919978498 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9c4c095710 0x7f9c4c19dcc0 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 rx=0x7f9c4000b5e0 tx=0x7f9c4000b980).stop 2021-02-10T11:36:26.983 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:26.996 7f9c54647700 1 -- 172.21.2.112:0/3919978498 shutdown_connections 2021-02-10T11:36:26.983 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:26.996 7f9c54647700 1 --2- 172.21.2.112:0/3919978498 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9c34018730 0x7f9c3401ac10 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:26.983 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:26.996 7f9c54647700 1 --2- 172.21.2.112:0/3919978498 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9c4c0aeeb0 0x7f9c4c19e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:26.983 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:26.996 7f9c54647700 1 --2- 172.21.2.112:0/3919978498 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9c4c19e900 0x7f9c4c1b7990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:26.983 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:26.996 7f9c54647700 1 --2- 172.21.2.112:0/3919978498 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9c4c095710 0x7f9c4c19dcc0 unknown :-1 s=CLOSED pgs=1620 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:26.983 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:26.996 7f9c54647700 1 -- 172.21.2.112:0/3919978498 >> 172.21.2.112:0/3919978498 conn(0x7f9c4c0846b0 msgr2=0x7f9c4c094950 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:26.983 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:26.996 7f9c54647700 1 -- 172.21.2.112:0/3919978498 shutdown_connections 2021-02-10T11:36:26.984 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:26.996 7f9c54647700 1 -- 172.21.2.112:0/3919978498 wait complete. 2021-02-10T11:36:26.991 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume snapshot clone cephfs subvolume_0000000000232725 snapshot_0000000000664158 clone_0000000000397074 2021-02-10T11:36:27.420 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:27.418 7fdc1dc83700 1 Processor -- start 2021-02-10T11:36:27.421 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:27.418 7fdc1dc83700 1 -- start start 2021-02-10T11:36:27.421 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:27.418 7fdc1dc83700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdc180dce80 0x7fdc180d14f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:27.421 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:27.418 7fdc1dc83700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fdc180eb220 0x7fdc180eb620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:27.421 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:27.418 7fdc1dc83700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc1810a550 con 0x7fdc1810aad0 2021-02-10T11:36:27.421 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:27.418 7fdc1dc83700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc180d1fa0 con 0x7fdc180eb220 2021-02-10T11:36:27.421 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:27.418 7fdc1dc83700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc180881d0 con 0x7fdc180dce80 2021-02-10T11:36:27.421 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:27.418 7fdc16ffd700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fdc180eb220 0x7fdc180eb620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:27.421 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:27.418 7fdc16ffd700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fdc180eb220 0x7fdc180eb620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:52054/0 (socket says 172.21.2.112:52054) 2021-02-10T11:36:27.422 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:27.418 7fdc16ffd700 1 -- 172.21.2.112:0/1853636555 learned_addr learned my addr 172.21.2.112:0/1853636555 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:36:27.422 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:27.419 7fdc16ffd700 1 -- 172.21.2.112:0/1853636555 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdc180dce80 msgr2=0x7fdc180d14f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:27.422 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:27.419 7fdc16ffd700 1 --2- 172.21.2.112:0/1853636555 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdc180dce80 0x7fdc180d14f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:27.422 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:27.419 7fdc16ffd700 1 -- 172.21.2.112:0/1853636555 >> v1:172.21.2.120:6789/0 conn(0x7fdc1810aad0 legacy=0x7fdc1810aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:27.422 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:27.419 7fdc16ffd700 1 -- 172.21.2.112:0/1853636555 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc180dd280 con 0x7fdc180eb220 2021-02-10T11:36:27.422 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:27.419 7fdc16ffd700 1 --2- 172.21.2.112:0/1853636555 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdc180eb220 0x7fdc180eb620 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 rx=0x7fdc08006700 tx=0x7fdc08009b80).ready entity=mon.0 client_cookie=89197f8dada69f93 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:27.422 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:27.419 7fdc167fc700 1 -- 172.21.2.112:0/1853636555 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdc0800c0c0 con 0x7fdc180eb220 2021-02-10T11:36:27.422 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:27.419 7fdc167fc700 1 -- 172.21.2.112:0/1853636555 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc0800c440 con 0x7fdc180eb220 2021-02-10T11:36:27.422 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:27.419 7fdc167fc700 1 -- 172.21.2.112:0/1853636555 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdc0800c960 con 0x7fdc180eb220 2021-02-10T11:36:27.422 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:27.419 7fdc1dc83700 1 -- 172.21.2.112:0/1853636555 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdc180eb220 msgr2=0x7fdc180eb620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:27.422 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:27.419 7fdc1dc83700 1 --2- 172.21.2.112:0/1853636555 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdc180eb220 0x7fdc180eb620 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 rx=0x7fdc08006700 tx=0x7fdc08009b80).stop 2021-02-10T11:36:27.423 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:27.420 7fdc1dc83700 1 -- 172.21.2.112:0/1853636555 shutdown_connections 2021-02-10T11:36:27.423 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:27.420 7fdc1dc83700 1 --2- 172.21.2.112:0/1853636555 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdc180eb220 0x7fdc180eb620 unknown :-1 s=CLOSED pgs=1621 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:27.423 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:27.420 7fdc1dc83700 1 --2- 172.21.2.112:0/1853636555 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdc180dce80 0x7fdc180d14f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:27.423 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:27.420 7fdc1dc83700 1 -- 172.21.2.112:0/1853636555 >> 172.21.2.112:0/1853636555 conn(0x7fdc180ea630 msgr2=0x7fdc180d0800 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:27.423 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:27.420 7fdc1dc83700 1 -- 172.21.2.112:0/1853636555 shutdown_connections 2021-02-10T11:36:27.423 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:27.420 7fdc1dc83700 1 -- 172.21.2.112:0/1853636555 wait complete. 2021-02-10T11:36:27.424 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:27.434 7fdc1dc83700 1 Processor -- start 2021-02-10T11:36:27.425 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:27.435 7fdc1dc83700 1 -- start start 2021-02-10T11:36:27.425 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:27.435 7fdc1dc83700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdc18090900 0x7fdc180b0870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:27.425 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:27.435 7fdc1dc83700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdc180b0e90 0x7fdc181074c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:27.425 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:27.435 7fdc1dc83700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fdc1819a5b0 0x7fdc18107ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:27.425 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:27.435 7fdc1dc83700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc1808c5b0 con 0x7fdc180b0e90 2021-02-10T11:36:27.425 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:27.435 7fdc1dc83700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc18108190 con 0x7fdc1819a5b0 2021-02-10T11:36:27.425 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:27.435 7fdc1dc83700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc1808cb30 con 0x7fdc18090900 2021-02-10T11:36:27.425 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:27.435 7fdc16ffd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdc180b0e90 0x7fdc181074c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:27.425 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:27.435 7fdc16ffd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdc180b0e90 0x7fdc181074c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:52062/0 (socket says 172.21.2.112:52062) 2021-02-10T11:36:27.425 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:27.435 7fdc16ffd700 1 -- 172.21.2.112:0/1706802658 learned_addr learned my addr 172.21.2.112:0/1706802658 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:36:27.425 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:27.436 7fdc16ffd700 1 -- 172.21.2.112:0/1706802658 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdc18090900 msgr2=0x7fdc180b0870 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:36:27.425 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:27.436 7fdc16ffd700 1 --2- 172.21.2.112:0/1706802658 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdc18090900 0x7fdc180b0870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:27.426 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:27.436 7fdc16ffd700 1 -- 172.21.2.112:0/1706802658 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fdc1819a5b0 msgr2=0x7fdc18107ae0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:36:27.426 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:27.436 7fdc16ffd700 1 --2- 172.21.2.112:0/1706802658 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fdc1819a5b0 0x7fdc18107ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:27.426 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:27.436 7fdc16ffd700 1 -- 172.21.2.112:0/1706802658 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc1808cd50 con 0x7fdc180b0e90 2021-02-10T11:36:27.426 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:27.436 7fdc16ffd700 1 --2- 172.21.2.112:0/1706802658 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdc180b0e90 0x7fdc181074c0 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 rx=0x7fdc0800b5e0 tx=0x7fdc0800b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:27.426 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:27.436 7fdc0ffff700 1 -- 172.21.2.112:0/1706802658 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdc0800bd70 con 0x7fdc180b0e90 2021-02-10T11:36:27.426 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:27.436 7fdc0ffff700 1 -- 172.21.2.112:0/1706802658 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc08002030 con 0x7fdc180b0e90 2021-02-10T11:36:27.426 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:27.436 7fdc1dc83700 1 -- 172.21.2.112:0/1706802658 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc1808cd50 con 0x7fdc180b0e90 2021-02-10T11:36:27.426 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:27.436 7fdc0ffff700 1 -- 172.21.2.112:0/1706802658 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdc08002550 con 0x7fdc180b0e90 2021-02-10T11:36:27.426 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:27.436 7fdc1dc83700 1 -- 172.21.2.112:0/1706802658 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc1808cd50 con 0x7fdc180b0e90 2021-02-10T11:36:27.426 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:27.437 7fdc0ffff700 1 -- 172.21.2.112:0/1706802658 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fdc08002210 con 0x7fdc180b0e90 2021-02-10T11:36:27.426 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:27.437 7fdc0ffff700 1 --2- 172.21.2.112:0/1706802658 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fdbfc0186c0 0x7fdbfc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:27.426 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:27.437 7fdc0ffff700 1 -- 172.21.2.112:0/1706802658 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fdc080327d0 con 0x7fdc180b0e90 2021-02-10T11:36:27.427 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:27.437 7fdc177fe700 1 --2- 172.21.2.112:0/1706802658 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fdbfc0186c0 0x7fdbfc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:27.427 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:27.437 7fdc177fe700 1 --2- 172.21.2.112:0/1706802658 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fdbfc0186c0 0x7fdbfc01abc0 crc :-1 s=READY pgs=777 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:27.454 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:27.468 7fdc1dc83700 1 -- 172.21.2.112:0/1706802658 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc180f9360 con 0x7fdc180b0e90 2021-02-10T11:36:27.457 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:27.471 7fdc0ffff700 1 -- 172.21.2.112:0/1706802658 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fdc08002240 con 0x7fdc180b0e90 2021-02-10T11:36:27.587 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:27.600 7fdc1dc83700 1 -- 172.21.2.112:0/1706802658 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "subvolume_0000000000232725", "target": ["mgr", ""], "snap_name": "snapshot_0000000000664158", "target_sub_name": "clone_0000000000397074"}) v1 -- 0x7fdc1808c5b0 con 0x7fdbfc0186c0 2021-02-10T11:36:27.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:27.642 7fdc0ffff700 1 -- 172.21.2.112:0/1706802658 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fdc000009b0 con 0x7fdbfc0186c0 2021-02-10T11:36:27.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:27.666 7fdc1dc83700 1 -- 172.21.2.112:0/1706802658 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fdbfc0186c0 msgr2=0x7fdbfc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:27.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:27.667 7fdc1dc83700 1 --2- 172.21.2.112:0/1706802658 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fdbfc0186c0 0x7fdbfc01abc0 crc :-1 s=READY pgs=777 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:27.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:27.667 7fdc1dc83700 1 -- 172.21.2.112:0/1706802658 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdc180b0e90 msgr2=0x7fdc181074c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:27.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:27.667 7fdc1dc83700 1 --2- 172.21.2.112:0/1706802658 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdc180b0e90 0x7fdc181074c0 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 rx=0x7fdc0800b5e0 tx=0x7fdc0800b980).stop 2021-02-10T11:36:27.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:27.667 7fdc1dc83700 1 -- 172.21.2.112:0/1706802658 shutdown_connections 2021-02-10T11:36:27.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:27.667 7fdc1dc83700 1 --2- 172.21.2.112:0/1706802658 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fdbfc0186c0 0x7fdbfc01abc0 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:27.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:27.667 7fdc1dc83700 1 --2- 172.21.2.112:0/1706802658 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fdc1819a5b0 0x7fdc18107ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:27.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:27.667 7fdc1dc83700 1 --2- 172.21.2.112:0/1706802658 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdc180b0e90 0x7fdc181074c0 unknown :-1 s=CLOSED pgs=1622 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:27.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:27.667 7fdc1dc83700 1 --2- 172.21.2.112:0/1706802658 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdc18090900 0x7fdc180b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:27.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:27.667 7fdc1dc83700 1 -- 172.21.2.112:0/1706802658 >> 172.21.2.112:0/1706802658 conn(0x7fdc181a8260 msgr2=0x7fdc181a7230 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:27.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:27.667 7fdc1dc83700 1 -- 172.21.2.112:0/1706802658 shutdown_connections 2021-02-10T11:36:27.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:27.667 7fdc1dc83700 1 -- 172.21.2.112:0/1706802658 wait complete. 2021-02-10T11:36:27.662 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs clone status cephfs clone_0000000000397074 2021-02-10T11:36:28.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.090 7f4ab97a1700 1 Processor -- start 2021-02-10T11:36:28.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.090 7f4ab97a1700 1 -- start start 2021-02-10T11:36:28.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.090 7f4ab97a1700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f4ab40d1870 0x7f4ab41b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:28.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.090 7f4ab97a1700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f4ab41b0ae0 0x7f4ab4108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:28.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.090 7f4ab97a1700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ab40fe1c0 con 0x7f4ab40d1c70 2021-02-10T11:36:28.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.090 7f4ab97a1700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ab4109a60 con 0x7f4ab41b0ae0 2021-02-10T11:36:28.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.090 7f4ab97a1700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ab4109750 con 0x7f4ab40d1870 2021-02-10T11:36:28.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.090 7f4ab2d73700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f4ab41b0ae0 0x7f4ab4108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:28.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.091 7f4ab3d75700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f4ab40d1c70 0x7f4ab4108f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:51528/0 (socket says 172.21.2.112:51528) 2021-02-10T11:36:28.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.091 7f4ab2d73700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f4ab41b0ae0 0x7f4ab4108940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:52066/0 (socket says 172.21.2.112:52066) 2021-02-10T11:36:28.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.091 7f4ab3d75700 1 -- 172.21.2.112:0/3906172660 learned_addr learned my addr 172.21.2.112:0/3906172660 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:36:28.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.091 7f4ab2d73700 1 -- 172.21.2.112:0/3906172660 >> v2:172.21.2.120:3300/0 conn(0x7f4ab40d1870 msgr2=0x7f4ab41b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:28.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.091 7f4ab2d73700 1 --2- 172.21.2.112:0/3906172660 >> v2:172.21.2.120:3300/0 conn(0x7f4ab40d1870 0x7f4ab41b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:28.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.091 7f4ab2d73700 1 -- 172.21.2.112:0/3906172660 >> v1:172.21.2.112:6789/0 conn(0x7f4ab40d1c70 legacy=0x7f4ab4108f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:28.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.091 7f4ab2d73700 1 -- 172.21.2.112:0/3906172660 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ab4088070 con 0x7f4ab41b0ae0 2021-02-10T11:36:28.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.091 7f4ab2d73700 1 --2- 172.21.2.112:0/3906172660 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4ab41b0ae0 0x7f4ab4108940 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 rx=0x7f4aa4006650 tx=0x7f4aa4009a80).ready entity=mon.0 client_cookie=c4e3b7e770f1f9de server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:28.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.091 7f4ab2572700 1 -- 172.21.2.112:0/3906172660 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4aa400c0c0 con 0x7f4ab41b0ae0 2021-02-10T11:36:28.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.091 7f4ab2572700 1 -- 172.21.2.112:0/3906172660 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4aa400c340 con 0x7f4ab41b0ae0 2021-02-10T11:36:28.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.091 7f4ab2572700 1 -- 172.21.2.112:0/3906172660 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4aa400c860 con 0x7f4ab41b0ae0 2021-02-10T11:36:28.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.091 7f4ab97a1700 1 -- 172.21.2.112:0/3906172660 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4ab41b0ae0 msgr2=0x7f4ab4108940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:28.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.091 7f4ab97a1700 1 --2- 172.21.2.112:0/3906172660 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4ab41b0ae0 0x7f4ab4108940 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 rx=0x7f4aa4006650 tx=0x7f4aa4009a80).stop 2021-02-10T11:36:28.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.092 7f4ab97a1700 1 -- 172.21.2.112:0/3906172660 shutdown_connections 2021-02-10T11:36:28.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.092 7f4ab97a1700 1 --2- 172.21.2.112:0/3906172660 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4ab41b0ae0 0x7f4ab4108940 unknown :-1 s=CLOSED pgs=1624 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:28.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.092 7f4ab97a1700 1 --2- 172.21.2.112:0/3906172660 >> v2:172.21.2.120:3300/0 conn(0x7f4ab40d1870 0x7f4ab41b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:28.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.092 7f4ab97a1700 1 -- 172.21.2.112:0/3906172660 >> 172.21.2.112:0/3906172660 conn(0x7f4ab40ce420 msgr2=0x7f4ab40d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:28.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.092 7f4ab97a1700 1 -- 172.21.2.112:0/3906172660 shutdown_connections 2021-02-10T11:36:28.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.092 7f4ab97a1700 1 -- 172.21.2.112:0/3906172660 wait complete. 2021-02-10T11:36:28.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.107 7f4ab97a1700 1 Processor -- start 2021-02-10T11:36:28.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.107 7f4ab97a1700 1 -- start start 2021-02-10T11:36:28.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.107 7f4ab97a1700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4ab40c3820 0x7f4ab40a7950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:28.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.107 7f4ab97a1700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4ab41a44c0 0x7f4ab41a48c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:28.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.107 7f4ab97a1700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4ab40863d0 0x7f4ab40867d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:28.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.107 7f4ab97a1700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ab4085ed0 con 0x7f4ab41a44c0 2021-02-10T11:36:28.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.107 7f4ab97a1700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ab40c0740 con 0x7f4ab40863d0 2021-02-10T11:36:28.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.107 7f4ab97a1700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ab4086df0 con 0x7f4ab40c3820 2021-02-10T11:36:28.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.107 7f4ab2d73700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4ab41a44c0 0x7f4ab41a48c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:28.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.107 7f4ab2d73700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4ab41a44c0 0x7f4ab41a48c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:52076/0 (socket says 172.21.2.112:52076) 2021-02-10T11:36:28.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.107 7f4ab2d73700 1 -- 172.21.2.112:0/4288678266 learned_addr learned my addr 172.21.2.112:0/4288678266 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:36:28.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.108 7f4ab2d73700 1 -- 172.21.2.112:0/4288678266 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4ab40c3820 msgr2=0x7f4ab40a7950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:28.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.108 7f4ab2d73700 1 --2- 172.21.2.112:0/4288678266 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4ab40c3820 0x7f4ab40a7950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:28.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.108 7f4ab2d73700 1 -- 172.21.2.112:0/4288678266 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4ab40863d0 msgr2=0x7f4ab40867d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:28.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.108 7f4ab2d73700 1 --2- 172.21.2.112:0/4288678266 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4ab40863d0 0x7f4ab40867d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:28.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.108 7f4ab2d73700 1 -- 172.21.2.112:0/4288678266 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ab4087010 con 0x7f4ab41a44c0 2021-02-10T11:36:28.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.108 7f4ab2d73700 1 --2- 172.21.2.112:0/4288678266 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4ab41a44c0 0x7f4ab41a48c0 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 rx=0x7f4aa4009630 tx=0x7f4aa400b900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:28.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.108 7f4aabfff700 1 -- 172.21.2.112:0/4288678266 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4aa40093a0 con 0x7f4ab41a44c0 2021-02-10T11:36:28.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.108 7f4aabfff700 1 -- 172.21.2.112:0/4288678266 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4aa400bcf0 con 0x7f4ab41a44c0 2021-02-10T11:36:28.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.108 7f4aabfff700 1 -- 172.21.2.112:0/4288678266 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4aa4005560 con 0x7f4ab41a44c0 2021-02-10T11:36:28.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.108 7f4ab97a1700 1 -- 172.21.2.112:0/4288678266 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ab4087010 con 0x7f4ab41a44c0 2021-02-10T11:36:28.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.108 7f4ab97a1700 1 -- 172.21.2.112:0/4288678266 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ab4083000 con 0x7f4ab41a44c0 2021-02-10T11:36:28.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.109 7f4aabfff700 1 -- 172.21.2.112:0/4288678266 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4aa400a000 con 0x7f4ab41a44c0 2021-02-10T11:36:28.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.109 7f4aabfff700 1 --2- 172.21.2.112:0/4288678266 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4a980186c0 0x7f4a9801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:28.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.109 7f4aabfff700 1 -- 172.21.2.112:0/4288678266 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f4aa40327b0 con 0x7f4ab41a44c0 2021-02-10T11:36:28.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.109 7f4ab3574700 1 --2- 172.21.2.112:0/4288678266 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4a980186c0 0x7f4a9801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:28.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.109 7f4ab3574700 1 --2- 172.21.2.112:0/4288678266 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4a980186c0 0x7f4a9801abc0 crc :-1 s=READY pgs=778 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:28.127 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.140 7f4ab97a1700 1 -- 172.21.2.112:0/4288678266 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ab40fe1c0 con 0x7f4ab41a44c0 2021-02-10T11:36:28.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.143 7f4aabfff700 1 -- 172.21.2.112:0/4288678266 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4aa4009340 con 0x7f4ab41a44c0 2021-02-10T11:36:28.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.277 7f4ab97a1700 1 -- 172.21.2.112:0/4288678266 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "clone_0000000000397074", "target": ["mgr", ""]}) v1 -- 0x7f4ab4083000 con 0x7f4a980186c0 2021-02-10T11:36:28.266 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.280 7f4aabfff700 1 -- 172.21.2.112:0/4288678266 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+46 (crc 0 0 0) 0x7f4a9c006a10 con 0x7f4a980186c0 2021-02-10T11:36:28.266 INFO:teuthology.orchestra.run.gibba012.stdout:{ 2021-02-10T11:36:28.267 INFO:teuthology.orchestra.run.gibba012.stdout: "status": { 2021-02-10T11:36:28.267 INFO:teuthology.orchestra.run.gibba012.stdout: "state": "complete" 2021-02-10T11:36:28.267 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:36:28.267 INFO:teuthology.orchestra.run.gibba012.stdout:} 2021-02-10T11:36:28.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.282 7f4ab97a1700 1 -- 172.21.2.112:0/4288678266 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4a980186c0 msgr2=0x7f4a9801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:28.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.282 7f4ab97a1700 1 --2- 172.21.2.112:0/4288678266 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4a980186c0 0x7f4a9801abc0 crc :-1 s=READY pgs=778 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:28.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.282 7f4ab97a1700 1 -- 172.21.2.112:0/4288678266 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4ab41a44c0 msgr2=0x7f4ab41a48c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:28.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.282 7f4ab97a1700 1 --2- 172.21.2.112:0/4288678266 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4ab41a44c0 0x7f4ab41a48c0 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 rx=0x7f4aa4009630 tx=0x7f4aa400b900).stop 2021-02-10T11:36:28.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.282 7f4ab97a1700 1 -- 172.21.2.112:0/4288678266 shutdown_connections 2021-02-10T11:36:28.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.282 7f4ab97a1700 1 --2- 172.21.2.112:0/4288678266 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4a980186c0 0x7f4a9801abc0 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:28.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.282 7f4ab97a1700 1 --2- 172.21.2.112:0/4288678266 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4ab40863d0 0x7f4ab40867d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:28.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.282 7f4ab97a1700 1 --2- 172.21.2.112:0/4288678266 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4ab41a44c0 0x7f4ab41a48c0 unknown :-1 s=CLOSED pgs=1625 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:28.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.282 7f4ab97a1700 1 --2- 172.21.2.112:0/4288678266 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4ab40c3820 0x7f4ab40a7950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:28.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.282 7f4ab97a1700 1 -- 172.21.2.112:0/4288678266 >> 172.21.2.112:0/4288678266 conn(0x7f4ab40872c0 msgr2=0x7f4ab4095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:28.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.282 7f4ab97a1700 1 -- 172.21.2.112:0/4288678266 shutdown_connections 2021-02-10T11:36:28.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.282 7f4ab97a1700 1 -- 172.21.2.112:0/4288678266 wait complete. 2021-02-10T11:36:28.277 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume snapshot rm cephfs subvolume_0000000000232725 snapshot_0000000000664158 2021-02-10T11:36:28.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.706 7f286e42b700 1 Processor -- start 2021-02-10T11:36:28.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.706 7f286e42b700 1 -- start start 2021-02-10T11:36:28.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.706 7f286e42b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f28680d1880 0x7f28681b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:28.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.706 7f286e42b700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f28681b0af0 0x7f2868108950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:28.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.706 7f286e42b700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f28680d1c80 0x7f2868108f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:28.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.706 7f286e42b700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28680e5f00 con 0x7f28681b0af0 2021-02-10T11:36:28.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.706 7f286e42b700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2868109a70 con 0x7f28680d1c80 2021-02-10T11:36:28.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.706 7f286e42b700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2868088070 con 0x7f28680d1880 2021-02-10T11:36:28.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.707 7f28677fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f28681b0af0 0x7f2868108950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:28.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.707 7f28677fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f28681b0af0 0x7f2868108950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:52080/0 (socket says 172.21.2.112:52080) 2021-02-10T11:36:28.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.707 7f28677fe700 1 -- 172.21.2.112:0/4156721173 learned_addr learned my addr 172.21.2.112:0/4156721173 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:36:28.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.707 7f286ca48700 1 --2- 172.21.2.112:0/4156721173 >> v2:172.21.2.120:3300/0 conn(0x7f28680d1c80 0x7f2868108f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:28.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.707 7f28677fe700 1 -- 172.21.2.112:0/4156721173 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f28680d1880 msgr2=0x7f28681b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:28.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.707 7f28677fe700 1 --2- 172.21.2.112:0/4156721173 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f28680d1880 0x7f28681b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:28.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.707 7f28677fe700 1 -- 172.21.2.112:0/4156721173 >> v2:172.21.2.120:3300/0 conn(0x7f28680d1c80 msgr2=0x7f2868108f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:28.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.707 7f28677fe700 1 --2- 172.21.2.112:0/4156721173 >> v2:172.21.2.120:3300/0 conn(0x7f28680d1c80 0x7f2868108f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:28.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.707 7f28677fe700 1 -- 172.21.2.112:0/4156721173 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2868109590 con 0x7f28681b0af0 2021-02-10T11:36:28.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.707 7f28677fe700 1 --2- 172.21.2.112:0/4156721173 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f28681b0af0 0x7f2868108950 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 rx=0x7f2850006700 tx=0x7f2850009b80).ready entity=mon.0 client_cookie=bf63e46df0c7234e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:28.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.707 7f2866ffd700 1 -- 172.21.2.112:0/4156721173 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f285000c0c0 con 0x7f28681b0af0 2021-02-10T11:36:28.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.707 7f2866ffd700 1 -- 172.21.2.112:0/4156721173 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f285000c340 con 0x7f28681b0af0 2021-02-10T11:36:28.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.707 7f2866ffd700 1 -- 172.21.2.112:0/4156721173 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f285000c860 con 0x7f28681b0af0 2021-02-10T11:36:28.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.707 7f286e42b700 1 -- 172.21.2.112:0/4156721173 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f28681b0af0 msgr2=0x7f2868108950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:28.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.707 7f286e42b700 1 --2- 172.21.2.112:0/4156721173 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f28681b0af0 0x7f2868108950 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 rx=0x7f2850006700 tx=0x7f2850009b80).stop 2021-02-10T11:36:28.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.708 7f286e42b700 1 -- 172.21.2.112:0/4156721173 shutdown_connections 2021-02-10T11:36:28.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.708 7f286e42b700 1 --2- 172.21.2.112:0/4156721173 >> v2:172.21.2.120:3300/0 conn(0x7f28680d1c80 0x7f2868108f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:28.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.708 7f286e42b700 1 --2- 172.21.2.112:0/4156721173 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f28681b0af0 0x7f2868108950 unknown :-1 s=CLOSED pgs=1626 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:28.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.708 7f286e42b700 1 --2- 172.21.2.112:0/4156721173 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f28680d1880 0x7f28681b04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:28.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.708 7f286e42b700 1 -- 172.21.2.112:0/4156721173 >> 172.21.2.112:0/4156721173 conn(0x7f28680ce430 msgr2=0x7f28680d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:28.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.708 7f286e42b700 1 -- 172.21.2.112:0/4156721173 shutdown_connections 2021-02-10T11:36:28.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.708 7f286e42b700 1 -- 172.21.2.112:0/4156721173 wait complete. 2021-02-10T11:36:28.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.723 7f286e42b700 1 Processor -- start 2021-02-10T11:36:28.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.723 7f286e42b700 1 -- start start 2021-02-10T11:36:28.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.723 7f286e42b700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f28680b1f10 0x7f28680b2310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:28.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.723 7f286e42b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2868083160 0x7f2868083560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:28.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.723 7f286e42b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2868083b80 0x7f2868083f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:28.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.723 7f286e42b700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f286809a820 con 0x7f2868083b80 2021-02-10T11:36:28.714 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.723 7f286e42b700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f286809acc0 con 0x7f28680b1f10 2021-02-10T11:36:28.714 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.723 7f286e42b700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28681af2d0 con 0x7f2868083160 2021-02-10T11:36:28.714 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.724 7f286ca48700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2868083b80 0x7f2868083f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:28.714 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.724 7f286ca48700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2868083b80 0x7f2868083f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:52088/0 (socket says 172.21.2.112:52088) 2021-02-10T11:36:28.714 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.724 7f286ca48700 1 -- 172.21.2.112:0/2003098638 learned_addr learned my addr 172.21.2.112:0/2003098638 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:36:28.714 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.724 7f2867fff700 1 --2- 172.21.2.112:0/2003098638 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f28680b1f10 0x7f28680b2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:28.714 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.724 7f28677fe700 1 --2- 172.21.2.112:0/2003098638 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2868083160 0x7f2868083560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:28.714 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.724 7f286ca48700 1 -- 172.21.2.112:0/2003098638 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2868083160 msgr2=0x7f2868083560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:28.714 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.724 7f286ca48700 1 --2- 172.21.2.112:0/2003098638 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2868083160 0x7f2868083560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:28.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.724 7f286ca48700 1 -- 172.21.2.112:0/2003098638 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f28680b1f10 msgr2=0x7f28680b2310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:28.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.724 7f286ca48700 1 --2- 172.21.2.112:0/2003098638 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f28680b1f10 0x7f28680b2310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:28.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.724 7f286ca48700 1 -- 172.21.2.112:0/2003098638 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28681af4f0 con 0x7f2868083b80 2021-02-10T11:36:28.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.724 7f286ca48700 1 --2- 172.21.2.112:0/2003098638 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2868083b80 0x7f2868083f80 secure :-1 s=READY pgs=1627 cs=0 l=1 rev1=1 rx=0x7f2860006f20 tx=0x7f286000b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:28.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.724 7f2864ff9700 1 -- 172.21.2.112:0/2003098638 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2860006bb0 con 0x7f2868083b80 2021-02-10T11:36:28.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.724 7f2864ff9700 1 -- 172.21.2.112:0/2003098638 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f286000bd80 con 0x7f2868083b80 2021-02-10T11:36:28.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.724 7f2864ff9700 1 -- 172.21.2.112:0/2003098638 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f28600022b0 con 0x7f2868083b80 2021-02-10T11:36:28.716 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.724 7f286e42b700 1 -- 172.21.2.112:0/2003098638 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28681af4f0 con 0x7f2868083b80 2021-02-10T11:36:28.716 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.724 7f286e42b700 1 -- 172.21.2.112:0/2003098638 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28681af4f0 con 0x7f2868083b80 2021-02-10T11:36:28.716 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.725 7f2864ff9700 1 -- 172.21.2.112:0/2003098638 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2860005330 con 0x7f2868083b80 2021-02-10T11:36:28.716 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.725 7f2864ff9700 1 --2- 172.21.2.112:0/2003098638 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f284c018730 0x7f284c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:28.716 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.725 7f2864ff9700 1 -- 172.21.2.112:0/2003098638 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f2860032660 con 0x7f2868083b80 2021-02-10T11:36:28.716 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.725 7f2867fff700 1 --2- 172.21.2.112:0/2003098638 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f284c018730 0x7f284c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:28.716 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.726 7f2867fff700 1 --2- 172.21.2.112:0/2003098638 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f284c018730 0x7f284c01abc0 crc :-1 s=READY pgs=779 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:28.742 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.756 7f286e42b700 1 -- 172.21.2.112:0/2003098638 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2868109a70 con 0x7f2868083b80 2021-02-10T11:36:28.745 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.758 7f2864ff9700 1 -- 172.21.2.112:0/2003098638 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2860006370 con 0x7f2868083b80 2021-02-10T11:36:28.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.887 7f286e42b700 1 -- 172.21.2.112:0/2003098638 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume snapshot rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000232725", "target": ["mgr", ""], "snap_name": "snapshot_0000000000664158"}) v1 -- 0x7f2868109a70 con 0x7f284c018730 2021-02-10T11:36:28.882 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.896 7f2864ff9700 1 -- 172.21.2.112:0/2003098638 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f2858000d30 con 0x7f284c018730 2021-02-10T11:36:28.890 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.903 7f286e42b700 1 -- 172.21.2.112:0/2003098638 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f284c018730 msgr2=0x7f284c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:28.890 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.903 7f286e42b700 1 --2- 172.21.2.112:0/2003098638 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f284c018730 0x7f284c01abc0 crc :-1 s=READY pgs=779 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:28.890 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.903 7f286e42b700 1 -- 172.21.2.112:0/2003098638 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2868083b80 msgr2=0x7f2868083f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:28.890 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.903 7f286e42b700 1 --2- 172.21.2.112:0/2003098638 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2868083b80 0x7f2868083f80 secure :-1 s=READY pgs=1627 cs=0 l=1 rev1=1 rx=0x7f2860006f20 tx=0x7f286000b990).stop 2021-02-10T11:36:28.890 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.903 7f286e42b700 1 -- 172.21.2.112:0/2003098638 shutdown_connections 2021-02-10T11:36:28.891 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.903 7f286e42b700 1 --2- 172.21.2.112:0/2003098638 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f284c018730 0x7f284c01abc0 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:28.891 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.903 7f286e42b700 1 --2- 172.21.2.112:0/2003098638 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2868083b80 0x7f2868083f80 unknown :-1 s=CLOSED pgs=1627 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:28.891 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.903 7f286e42b700 1 --2- 172.21.2.112:0/2003098638 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2868083160 0x7f2868083560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:28.891 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.903 7f286e42b700 1 --2- 172.21.2.112:0/2003098638 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f28680b1f10 0x7f28680b2310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:28.891 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.903 7f286e42b700 1 -- 172.21.2.112:0/2003098638 >> 172.21.2.112:0/2003098638 conn(0x7f2868087390 msgr2=0x7f2868090c90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:28.891 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.904 7f286e42b700 1 -- 172.21.2.112:0/2003098638 shutdown_connections 2021-02-10T11:36:28.891 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:28.904 7f286e42b700 1 -- 172.21.2.112:0/2003098638 wait complete. 2021-02-10T11:36:28.898 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume info cephfs clone_0000000000397074 2021-02-10T11:36:29.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.327 7fd3af8a4700 1 Processor -- start 2021-02-10T11:36:29.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.327 7fd3af8a4700 1 -- start start 2021-02-10T11:36:29.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.327 7fd3af8a4700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd3a80d1860 0x7fd3a81b04b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:29.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.327 7fd3af8a4700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fd3a81b0ad0 0x7fd3a8108930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:29.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.327 7fd3af8a4700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3a8109a50 con 0x7fd3a80d1c60 2021-02-10T11:36:29.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.327 7fd3af8a4700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3a80e5ee0 con 0x7fd3a81b0ad0 2021-02-10T11:36:29.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.327 7fd3af8a4700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3a8109740 con 0x7fd3a80d1860 2021-02-10T11:36:29.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.328 7fd3acebf700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fd3a81b0ad0 0x7fd3a8108930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:29.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.328 7fd3adec1700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fd3a80d1c60 0x7fd3a8108f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:51554/0 (socket says 172.21.2.112:51554) 2021-02-10T11:36:29.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.328 7fd3adec1700 1 -- 172.21.2.112:0/2551139603 learned_addr learned my addr 172.21.2.112:0/2551139603 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:36:29.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.328 7fd3ad6c0700 1 --2- 172.21.2.112:0/2551139603 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd3a80d1860 0x7fd3a81b04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:29.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.328 7fd3acebf700 1 -- 172.21.2.112:0/2551139603 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd3a80d1860 msgr2=0x7fd3a81b04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:29.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.328 7fd3acebf700 1 --2- 172.21.2.112:0/2551139603 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd3a80d1860 0x7fd3a81b04b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:29.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.328 7fd3acebf700 1 -- 172.21.2.112:0/2551139603 >> v1:172.21.2.112:6789/0 conn(0x7fd3a80d1c60 legacy=0x7fd3a8108f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:29.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.328 7fd39bfff700 1 -- 172.21.2.112:0/2551139603 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3554976872 0 0) 0x7fd3a0001500 con 0x7fd3a80d1c60 2021-02-10T11:36:29.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.328 7fd3acebf700 1 -- 172.21.2.112:0/2551139603 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3a8088070 con 0x7fd3a81b0ad0 2021-02-10T11:36:29.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.328 7fd3ad6c0700 1 --2- 172.21.2.112:0/2551139603 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd3a80d1860 0x7fd3a81b04b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:36:29.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.328 7fd3acebf700 1 --2- 172.21.2.112:0/2551139603 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd3a81b0ad0 0x7fd3a8108930 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 rx=0x7fd39c006650 tx=0x7fd39c009a80).ready entity=mon.0 client_cookie=6da3c7268041fd56 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:29.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.328 7fd39bfff700 1 -- 172.21.2.112:0/2551139603 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd39c0009b0 con 0x7fd3a81b0ad0 2021-02-10T11:36:29.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.329 7fd39bfff700 1 -- 172.21.2.112:0/2551139603 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd39c00c0c0 con 0x7fd3a81b0ad0 2021-02-10T11:36:29.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.329 7fd39bfff700 1 -- 172.21.2.112:0/2551139603 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd39c00c510 con 0x7fd3a81b0ad0 2021-02-10T11:36:29.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.329 7fd3af8a4700 1 -- 172.21.2.112:0/2551139603 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd3a81b0ad0 msgr2=0x7fd3a8108930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:29.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.329 7fd3af8a4700 1 --2- 172.21.2.112:0/2551139603 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd3a81b0ad0 0x7fd3a8108930 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 rx=0x7fd39c006650 tx=0x7fd39c009a80).stop 2021-02-10T11:36:29.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.329 7fd3af8a4700 1 -- 172.21.2.112:0/2551139603 shutdown_connections 2021-02-10T11:36:29.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.329 7fd3af8a4700 1 --2- 172.21.2.112:0/2551139603 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd3a81b0ad0 0x7fd3a8108930 unknown :-1 s=CLOSED pgs=1629 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:29.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.329 7fd3af8a4700 1 --2- 172.21.2.112:0/2551139603 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd3a80d1860 0x7fd3a81b04b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:29.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.329 7fd3af8a4700 1 -- 172.21.2.112:0/2551139603 >> 172.21.2.112:0/2551139603 conn(0x7fd3a80ce410 msgr2=0x7fd3a80d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:29.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.329 7fd3af8a4700 1 -- 172.21.2.112:0/2551139603 shutdown_connections 2021-02-10T11:36:29.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.329 7fd3af8a4700 1 -- 172.21.2.112:0/2551139603 wait complete. 2021-02-10T11:36:29.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.344 7fd3af8a4700 1 Processor -- start 2021-02-10T11:36:29.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.344 7fd3af8a4700 1 -- start start 2021-02-10T11:36:29.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.344 7fd3af8a4700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd3a80b6da0 0x7fd3a80b71a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:29.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.344 7fd3af8a4700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd3a80a70d0 0x7fd3a80b77c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:29.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.344 7fd3af8a4700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd3a80b8780 0x7fd3a80b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:29.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.344 7fd3af8a4700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3a80a7610 con 0x7fd3a80b8780 2021-02-10T11:36:29.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.344 7fd3af8a4700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3a80b7de0 con 0x7fd3a80a70d0 2021-02-10T11:36:29.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.344 7fd3af8a4700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3a80b8000 con 0x7fd3a80b6da0 2021-02-10T11:36:29.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.345 7fd3adec1700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd3a80b8780 0x7fd3a80b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:29.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.345 7fd3adec1700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd3a80b8780 0x7fd3a80b8b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:52100/0 (socket says 172.21.2.112:52100) 2021-02-10T11:36:29.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.345 7fd3adec1700 1 -- 172.21.2.112:0/1025538686 learned_addr learned my addr 172.21.2.112:0/1025538686 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:36:29.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.345 7fd3adec1700 1 -- 172.21.2.112:0/1025538686 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd3a80b6da0 msgr2=0x7fd3a80b71a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:29.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.345 7fd3adec1700 1 --2- 172.21.2.112:0/1025538686 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd3a80b6da0 0x7fd3a80b71a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:29.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.345 7fd3adec1700 1 -- 172.21.2.112:0/1025538686 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd3a80a70d0 msgr2=0x7fd3a80b77c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:29.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.345 7fd3adec1700 1 --2- 172.21.2.112:0/1025538686 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd3a80a70d0 0x7fd3a80b77c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:29.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.345 7fd3adec1700 1 -- 172.21.2.112:0/1025538686 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3a80b91a0 con 0x7fd3a80b8780 2021-02-10T11:36:29.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.345 7fd3adec1700 1 --2- 172.21.2.112:0/1025538686 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd3a80b8780 0x7fd3a80b8b80 secure :-1 s=READY pgs=1630 cs=0 l=1 rev1=1 rx=0x7fd3a0001c10 tx=0x7fd3a000ac60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:29.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.345 7fd399ffb700 1 -- 172.21.2.112:0/1025538686 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd3a00070d0 con 0x7fd3a80b8780 2021-02-10T11:36:29.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.345 7fd399ffb700 1 -- 172.21.2.112:0/1025538686 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3a0007350 con 0x7fd3a80b8780 2021-02-10T11:36:29.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.345 7fd399ffb700 1 -- 172.21.2.112:0/1025538686 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd3a00049a0 con 0x7fd3a80b8780 2021-02-10T11:36:29.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.345 7fd3af8a4700 1 -- 172.21.2.112:0/1025538686 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3a80b91a0 con 0x7fd3a80b8780 2021-02-10T11:36:29.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.346 7fd3af8a4700 1 -- 172.21.2.112:0/1025538686 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3a80b91a0 con 0x7fd3a80b8780 2021-02-10T11:36:29.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.346 7fd399ffb700 1 -- 172.21.2.112:0/1025538686 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd3a0004720 con 0x7fd3a80b8780 2021-02-10T11:36:29.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.346 7fd399ffb700 1 --2- 172.21.2.112:0/1025538686 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd388018700 0x7fd38801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:29.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.346 7fd399ffb700 1 -- 172.21.2.112:0/1025538686 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fd3a00329a0 con 0x7fd3a80b8780 2021-02-10T11:36:29.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.346 7fd3ad6c0700 1 --2- 172.21.2.112:0/1025538686 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd388018700 0x7fd38801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:29.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.347 7fd3ad6c0700 1 --2- 172.21.2.112:0/1025538686 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd388018700 0x7fd38801abc0 crc :-1 s=READY pgs=780 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:29.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.377 7fd3af8a4700 1 -- 172.21.2.112:0/1025538686 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3a8082920 con 0x7fd3a80b8780 2021-02-10T11:36:29.366 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.380 7fd399ffb700 1 -- 172.21.2.112:0/1025538686 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd3a0007330 con 0x7fd3a80b8780 2021-02-10T11:36:29.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.510 7fd3af8a4700 1 -- 172.21.2.112:0/1025538686 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume info", "vol_name": "cephfs", "sub_name": "clone_0000000000397074", "target": ["mgr", ""]}) v1 -- 0x7fd3a8082920 con 0x7fd388018700 2021-02-10T11:36:29.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.514 7fd399ffb700 1 -- 172.21.2.112:0/1025538686 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+724 (crc 0 0 0) 0x7fd3a40077a0 con 0x7fd388018700 2021-02-10T11:36:29.503 INFO:teuthology.orchestra.run.gibba012.stdout:{ 2021-02-10T11:36:29.503 INFO:teuthology.orchestra.run.gibba012.stdout: "atime": "2021-02-10 11:36:24", 2021-02-10T11:36:29.503 INFO:teuthology.orchestra.run.gibba012.stdout: "bytes_pcent": "undefined", 2021-02-10T11:36:29.504 INFO:teuthology.orchestra.run.gibba012.stdout: "bytes_quota": "infinite", 2021-02-10T11:36:29.504 INFO:teuthology.orchestra.run.gibba012.stdout: "bytes_used": 0, 2021-02-10T11:36:29.504 INFO:teuthology.orchestra.run.gibba012.stdout: "created_at": "2021-02-10 11:36:27", 2021-02-10T11:36:29.504 INFO:teuthology.orchestra.run.gibba012.stdout: "ctime": "2021-02-10 11:36:27", 2021-02-10T11:36:29.504 INFO:teuthology.orchestra.run.gibba012.stdout: "data_pool": "cephfs_data", 2021-02-10T11:36:29.504 INFO:teuthology.orchestra.run.gibba012.stdout: "features": [ 2021-02-10T11:36:29.504 INFO:teuthology.orchestra.run.gibba012.stdout: "snapshot-clone", 2021-02-10T11:36:29.504 INFO:teuthology.orchestra.run.gibba012.stdout: "snapshot-autoprotect", 2021-02-10T11:36:29.504 INFO:teuthology.orchestra.run.gibba012.stdout: "snapshot-retention" 2021-02-10T11:36:29.504 INFO:teuthology.orchestra.run.gibba012.stdout: ], 2021-02-10T11:36:29.504 INFO:teuthology.orchestra.run.gibba012.stdout: "gid": 0, 2021-02-10T11:36:29.504 INFO:teuthology.orchestra.run.gibba012.stdout: "mode": 16877, 2021-02-10T11:36:29.504 INFO:teuthology.orchestra.run.gibba012.stdout: "mon_addrs": [ 2021-02-10T11:36:29.505 INFO:teuthology.orchestra.run.gibba012.stdout: "172.21.2.112:6789", 2021-02-10T11:36:29.505 INFO:teuthology.orchestra.run.gibba012.stdout: "172.21.2.120:6789", 2021-02-10T11:36:29.505 INFO:teuthology.orchestra.run.gibba012.stdout: "172.21.2.120:6790" 2021-02-10T11:36:29.505 INFO:teuthology.orchestra.run.gibba012.stdout: ], 2021-02-10T11:36:29.505 INFO:teuthology.orchestra.run.gibba012.stdout: "mtime": "2021-02-10 11:36:26", 2021-02-10T11:36:29.505 INFO:teuthology.orchestra.run.gibba012.stdout: "path": "/volumes/_nogroup/clone_0000000000397074/344d1578-9403-413f-922e-d039f3c8e6bb", 2021-02-10T11:36:29.505 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_namespace": "", 2021-02-10T11:36:29.505 INFO:teuthology.orchestra.run.gibba012.stdout: "state": "complete", 2021-02-10T11:36:29.505 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "clone", 2021-02-10T11:36:29.505 INFO:teuthology.orchestra.run.gibba012.stdout: "uid": 0 2021-02-10T11:36:29.505 INFO:teuthology.orchestra.run.gibba012.stdout:} 2021-02-10T11:36:29.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.518 7fd3af8a4700 1 -- 172.21.2.112:0/1025538686 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd388018700 msgr2=0x7fd38801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:29.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.518 7fd3af8a4700 1 --2- 172.21.2.112:0/1025538686 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd388018700 0x7fd38801abc0 crc :-1 s=READY pgs=780 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:29.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.518 7fd3af8a4700 1 -- 172.21.2.112:0/1025538686 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd3a80b8780 msgr2=0x7fd3a80b8b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:29.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.518 7fd3af8a4700 1 --2- 172.21.2.112:0/1025538686 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd3a80b8780 0x7fd3a80b8b80 secure :-1 s=READY pgs=1630 cs=0 l=1 rev1=1 rx=0x7fd3a0001c10 tx=0x7fd3a000ac60).stop 2021-02-10T11:36:29.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.519 7fd3af8a4700 1 -- 172.21.2.112:0/1025538686 shutdown_connections 2021-02-10T11:36:29.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.519 7fd3af8a4700 1 --2- 172.21.2.112:0/1025538686 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd388018700 0x7fd38801abc0 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:29.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.519 7fd3af8a4700 1 --2- 172.21.2.112:0/1025538686 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd3a80a70d0 0x7fd3a80b77c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:29.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.519 7fd3af8a4700 1 --2- 172.21.2.112:0/1025538686 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd3a80b8780 0x7fd3a80b8b80 unknown :-1 s=CLOSED pgs=1630 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:29.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.519 7fd3af8a4700 1 --2- 172.21.2.112:0/1025538686 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd3a80b6da0 0x7fd3a80b71a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:29.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.519 7fd3af8a4700 1 -- 172.21.2.112:0/1025538686 >> 172.21.2.112:0/1025538686 conn(0x7fd3a8092c70 msgr2=0x7fd3a8093530 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:29.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.519 7fd3af8a4700 1 -- 172.21.2.112:0/1025538686 shutdown_connections 2021-02-10T11:36:29.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.519 7fd3af8a4700 1 -- 172.21.2.112:0/1025538686 wait complete. 2021-02-10T11:36:29.513 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000232725 2021-02-10T11:36:29.943 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.940 7f0d56662700 1 Processor -- start 2021-02-10T11:36:29.944 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.940 7f0d56662700 1 -- start start 2021-02-10T11:36:29.944 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.941 7f0d56662700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f0d500d1870 0x7f0d501b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:29.944 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.941 7f0d56662700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0d501b0ae0 0x7f0d50108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:29.944 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.941 7f0d56662700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f0d500d1c70 0x7f0d50108f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:29.944 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.941 7f0d56662700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d500fe1c0 con 0x7f0d500d1870 2021-02-10T11:36:29.944 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.941 7f0d56662700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d50109a60 con 0x7f0d500d1c70 2021-02-10T11:36:29.944 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.941 7f0d56662700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d50088070 con 0x7f0d501b0ae0 2021-02-10T11:36:29.944 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.941 7f0d4ffff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f0d500d1870 0x7f0d501b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:29.944 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.941 7f0d4ffff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f0d500d1870 0x7f0d501b04c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:52104/0 (socket says 172.21.2.112:52104) 2021-02-10T11:36:29.944 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.941 7f0d4ffff700 1 -- 172.21.2.112:0/1250654825 learned_addr learned my addr 172.21.2.112:0/1250654825 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:36:29.945 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.941 7f0d54c7f700 1 --2- 172.21.2.112:0/1250654825 >> v2:172.21.2.120:3300/0 conn(0x7f0d500d1c70 0x7f0d50108f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:29.945 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.941 7f0d4f7fe700 1 --2- 172.21.2.112:0/1250654825 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0d501b0ae0 0x7f0d50108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:29.945 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.941 7f0d4ffff700 1 -- 172.21.2.112:0/1250654825 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0d501b0ae0 msgr2=0x7f0d50108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:29.945 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.941 7f0d4ffff700 1 --2- 172.21.2.112:0/1250654825 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0d501b0ae0 0x7f0d50108940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:29.945 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.941 7f0d4ffff700 1 -- 172.21.2.112:0/1250654825 >> v2:172.21.2.120:3300/0 conn(0x7f0d500d1c70 msgr2=0x7f0d50108f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:29.945 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.941 7f0d4ffff700 1 --2- 172.21.2.112:0/1250654825 >> v2:172.21.2.120:3300/0 conn(0x7f0d500d1c70 0x7f0d50108f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:29.945 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.941 7f0d4ffff700 1 -- 172.21.2.112:0/1250654825 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d50109580 con 0x7f0d500d1870 2021-02-10T11:36:29.945 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.941 7f0d4ffff700 1 --2- 172.21.2.112:0/1250654825 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0d500d1870 0x7f0d501b04c0 secure :-1 s=READY pgs=1631 cs=0 l=1 rev1=1 rx=0x7f0d40006a80 tx=0x7f0d40009f00).ready entity=mon.0 client_cookie=2961c1eafc7dfcad server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:29.945 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.942 7f0d4effd700 1 -- 172.21.2.112:0/1250654825 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0d4000b5c0 con 0x7f0d500d1870 2021-02-10T11:36:29.945 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.942 7f0d4effd700 1 -- 172.21.2.112:0/1250654825 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d4000b8c0 con 0x7f0d500d1870 2021-02-10T11:36:29.945 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.942 7f0d4effd700 1 -- 172.21.2.112:0/1250654825 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0d4000bde0 con 0x7f0d500d1870 2021-02-10T11:36:29.945 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.942 7f0d56662700 1 -- 172.21.2.112:0/1250654825 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0d500d1870 msgr2=0x7f0d501b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:29.945 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.942 7f0d56662700 1 --2- 172.21.2.112:0/1250654825 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0d500d1870 0x7f0d501b04c0 secure :-1 s=READY pgs=1631 cs=0 l=1 rev1=1 rx=0x7f0d40006a80 tx=0x7f0d40009f00).stop 2021-02-10T11:36:29.945 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.942 7f0d56662700 1 -- 172.21.2.112:0/1250654825 shutdown_connections 2021-02-10T11:36:29.946 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.942 7f0d56662700 1 --2- 172.21.2.112:0/1250654825 >> v2:172.21.2.120:3300/0 conn(0x7f0d500d1c70 0x7f0d50108f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:29.946 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.942 7f0d56662700 1 --2- 172.21.2.112:0/1250654825 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0d501b0ae0 0x7f0d50108940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:29.946 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.942 7f0d56662700 1 --2- 172.21.2.112:0/1250654825 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0d500d1870 0x7f0d501b04c0 unknown :-1 s=CLOSED pgs=1631 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:29.946 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.942 7f0d56662700 1 -- 172.21.2.112:0/1250654825 >> 172.21.2.112:0/1250654825 conn(0x7f0d500ce420 msgr2=0x7f0d500d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:29.946 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.942 7f0d56662700 1 -- 172.21.2.112:0/1250654825 shutdown_connections 2021-02-10T11:36:29.946 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.942 7f0d56662700 1 -- 172.21.2.112:0/1250654825 wait complete. 2021-02-10T11:36:29.946 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.957 7f0d56662700 1 Processor -- start 2021-02-10T11:36:29.946 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.957 7f0d56662700 1 -- start start 2021-02-10T11:36:29.947 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.957 7f0d56662700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0d50085bc0 0x7f0d500863e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:29.947 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.957 7f0d56662700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0d500954c0 0x7f0d50096300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:29.948 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.957 7f0d56662700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0d500958c0 0x7f0d50096920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:29.948 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.957 7f0d56662700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d50095d80 con 0x7f0d500958c0 2021-02-10T11:36:29.948 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.957 7f0d56662700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d500bfa60 con 0x7f0d50085bc0 2021-02-10T11:36:29.948 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.957 7f0d56662700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d50085fc0 con 0x7f0d500954c0 2021-02-10T11:36:29.948 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.958 7f0d54c7f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0d500958c0 0x7f0d50096920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:29.948 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.958 7f0d54c7f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0d500958c0 0x7f0d50096920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:52114/0 (socket says 172.21.2.112:52114) 2021-02-10T11:36:29.948 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.958 7f0d54c7f700 1 -- 172.21.2.112:0/1878718609 learned_addr learned my addr 172.21.2.112:0/1878718609 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:36:29.948 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.958 7f0d4ffff700 1 --2- 172.21.2.112:0/1878718609 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0d50085bc0 0x7f0d500863e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:29.948 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.958 7f0d4f7fe700 1 --2- 172.21.2.112:0/1878718609 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0d500954c0 0x7f0d50096300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:29.948 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.958 7f0d54c7f700 1 -- 172.21.2.112:0/1878718609 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0d500954c0 msgr2=0x7f0d50096300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:29.948 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.958 7f0d54c7f700 1 --2- 172.21.2.112:0/1878718609 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0d500954c0 0x7f0d50096300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:29.948 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.958 7f0d54c7f700 1 -- 172.21.2.112:0/1878718609 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0d50085bc0 msgr2=0x7f0d500863e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:29.949 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.958 7f0d54c7f700 1 --2- 172.21.2.112:0/1878718609 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0d50085bc0 0x7f0d500863e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:29.949 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.958 7f0d54c7f700 1 -- 172.21.2.112:0/1878718609 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d50086c10 con 0x7f0d500958c0 2021-02-10T11:36:29.949 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.958 7f0d54c7f700 1 --2- 172.21.2.112:0/1878718609 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0d500958c0 0x7f0d50096920 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 rx=0x7f0d48006b90 tx=0x7f0d4800b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:29.949 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.959 7f0d4cff9700 1 -- 172.21.2.112:0/1878718609 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0d48006c60 con 0x7f0d500958c0 2021-02-10T11:36:29.949 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.959 7f0d4cff9700 1 -- 172.21.2.112:0/1878718609 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d48002080 con 0x7f0d500958c0 2021-02-10T11:36:29.949 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.959 7f0d4cff9700 1 -- 172.21.2.112:0/1878718609 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0d480024d0 con 0x7f0d500958c0 2021-02-10T11:36:29.949 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.959 7f0d56662700 1 -- 172.21.2.112:0/1878718609 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d50086c10 con 0x7f0d500958c0 2021-02-10T11:36:29.949 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.959 7f0d56662700 1 -- 172.21.2.112:0/1878718609 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d50097230 con 0x7f0d500958c0 2021-02-10T11:36:29.949 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.960 7f0d4cff9700 1 -- 172.21.2.112:0/1878718609 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0d480067a0 con 0x7f0d500958c0 2021-02-10T11:36:29.949 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.960 7f0d4cff9700 1 --2- 172.21.2.112:0/1878718609 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0d340186c0 0x7f0d3401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:29.949 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.960 7f0d4cff9700 1 -- 172.21.2.112:0/1878718609 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f0d480325c0 con 0x7f0d500958c0 2021-02-10T11:36:29.949 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.960 7f0d4ffff700 1 --2- 172.21.2.112:0/1878718609 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0d340186c0 0x7f0d3401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:29.949 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.960 7f0d4ffff700 1 --2- 172.21.2.112:0/1878718609 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0d340186c0 0x7f0d3401abc0 crc :-1 s=READY pgs=781 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:29.977 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.990 7f0d56662700 1 -- 172.21.2.112:0/1878718609 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d500fe1c0 con 0x7f0d500958c0 2021-02-10T11:36:29.979 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:29.993 7f0d4cff9700 1 -- 172.21.2.112:0/1878718609 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0d48030d70 con 0x7f0d500958c0 2021-02-10T11:36:30.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:30.120 7f0d56662700 1 -- 172.21.2.112:0/1878718609 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000232725", "target": ["mgr", ""]}) v1 -- 0x7f0d50097230 con 0x7f0d340186c0 2021-02-10T11:36:30.117 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:30.131 7f0d4cff9700 1 -- 172.21.2.112:0/1878718609 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f0d40000d30 con 0x7f0d340186c0 2021-02-10T11:36:30.124 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:30.137 7f0d56662700 1 -- 172.21.2.112:0/1878718609 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0d340186c0 msgr2=0x7f0d3401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:30.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:30.137 7f0d56662700 1 --2- 172.21.2.112:0/1878718609 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0d340186c0 0x7f0d3401abc0 crc :-1 s=READY pgs=781 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:30.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:30.137 7f0d56662700 1 -- 172.21.2.112:0/1878718609 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0d500958c0 msgr2=0x7f0d50096920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:30.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:30.137 7f0d56662700 1 --2- 172.21.2.112:0/1878718609 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0d500958c0 0x7f0d50096920 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 rx=0x7f0d48006b90 tx=0x7f0d4800b990).stop 2021-02-10T11:36:30.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:30.138 7f0d56662700 1 -- 172.21.2.112:0/1878718609 shutdown_connections 2021-02-10T11:36:30.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:30.138 7f0d56662700 1 --2- 172.21.2.112:0/1878718609 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0d340186c0 0x7f0d3401abc0 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:30.126 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:30.138 7f0d56662700 1 --2- 172.21.2.112:0/1878718609 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0d500958c0 0x7f0d50096920 unknown :-1 s=CLOSED pgs=1632 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:30.126 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:30.138 7f0d56662700 1 --2- 172.21.2.112:0/1878718609 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0d500954c0 0x7f0d50096300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:30.126 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:30.138 7f0d56662700 1 --2- 172.21.2.112:0/1878718609 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0d50085bc0 0x7f0d500863e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:30.126 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:30.138 7f0d56662700 1 -- 172.21.2.112:0/1878718609 >> 172.21.2.112:0/1878718609 conn(0x7f0d500bef00 msgr2=0x7f0d5009d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:30.126 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:30.138 7f0d56662700 1 -- 172.21.2.112:0/1878718609 shutdown_connections 2021-02-10T11:36:30.127 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:30.138 7f0d56662700 1 -- 172.21.2.112:0/1878718609 wait complete. 2021-02-10T11:36:30.133 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs clone_0000000000397074 2021-02-10T11:36:30.566 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:30.562 7fd046d4a700 1 Processor -- start 2021-02-10T11:36:30.566 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:30.562 7fd046d4a700 1 -- start start 2021-02-10T11:36:30.567 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:30.563 7fd046d4a700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fd04010aff0 0x7fd0400cf750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:30.567 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:30.563 7fd046d4a700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0400fe1c0 con 0x7fd0400d1c70 2021-02-10T11:36:30.567 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:30.563 7fd046d4a700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0401b0cb0 con 0x7fd0400d1870 2021-02-10T11:36:30.567 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:30.563 7fd046d4a700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd040109a60 con 0x7fd04010aff0 2021-02-10T11:36:30.567 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:30.563 7fd03ffff700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fd0400d1c70 0x7fd0400cef60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:51580/0 (socket says 172.21.2.112:51580) 2021-02-10T11:36:30.567 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:30.563 7fd03ffff700 1 -- 172.21.2.112:0/2814218740 learned_addr learned my addr 172.21.2.112:0/2814218740 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:36:30.567 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:30.563 7fd03f7fe700 1 -- 172.21.2.112:0/2814218740 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1531567773 0 0) 0x7fd030001430 con 0x7fd0400d1c70 2021-02-10T11:36:30.567 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:30.563 7fd03f7fe700 1 -- 172.21.2.112:0/2814218740 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd02c002fe0 con 0x7fd0400d1c70 2021-02-10T11:36:30.567 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:30.563 7fd045367700 1 --2- 172.21.2.112:0/2814218740 >> v2:172.21.2.120:3300/0 conn(0x7fd04010aff0 0x7fd0400cf750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:30.567 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:30.564 7fd03f7fe700 1 -- 172.21.2.112:0/2814218740 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4204443296 0 0) 0x7fd030001390 con 0x7fd0400d1c70 2021-02-10T11:36:30.567 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:30.564 7fd03f7fe700 1 -- 172.21.2.112:0/2814218740 >> v2:172.21.2.120:3300/0 conn(0x7fd04010aff0 msgr2=0x7fd0400cf750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:30.567 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:30.564 7fd03f7fe700 1 --2- 172.21.2.112:0/2814218740 >> v2:172.21.2.120:3300/0 conn(0x7fd04010aff0 0x7fd0400cf750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:30.567 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:30.564 7fd03f7fe700 1 -- 172.21.2.112:0/2814218740 >> v1:172.21.2.120:6789/0 conn(0x7fd0400d1870 legacy=0x7fd0401b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:30.568 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:30.564 7fd03f7fe700 1 -- 172.21.2.112:0/2814218740 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd04010b3f0 con 0x7fd0400d1c70 2021-02-10T11:36:30.568 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:30.564 7fd03f7fe700 1 -- 172.21.2.112:0/2814218740 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7fd0300018b0 con 0x7fd0400d1c70 2021-02-10T11:36:30.568 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:30.564 7fd03f7fe700 1 -- 172.21.2.112:0/2814218740 >> v1:172.21.2.112:6789/0 conn(0x7fd0400d1c70 legacy=0x7fd0400cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:30.568 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:30.564 7fd03f7fe700 1 --2- 172.21.2.112:0/2814218740 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd02c003a60 0x7fd02c005f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:30.568 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:30.564 7fd03f7fe700 1 --2- 172.21.2.112:0/2814218740 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd02c006780 0x7fd02c008ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:30.568 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:30.564 7fd03f7fe700 1 --2- 172.21.2.112:0/2814218740 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd02c0095a0 0x7fd02c00baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:30.568 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:30.564 7fd03f7fe700 1 -- 172.21.2.112:0/2814218740 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd02c00c2f0 con 0x7fd02c003a60 2021-02-10T11:36:30.568 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:30.564 7fd03f7fe700 1 -- 172.21.2.112:0/2814218740 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd02c00c510 con 0x7fd02c0095a0 2021-02-10T11:36:30.568 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:30.564 7fd03f7fe700 1 -- 172.21.2.112:0/2814218740 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd02c00c730 con 0x7fd02c006780 2021-02-10T11:36:30.568 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:30.564 7fd044b66700 1 --2- 172.21.2.112:0/2814218740 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd02c003a60 0x7fd02c005f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:30.568 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:30.564 7fd045367700 1 --2- 172.21.2.112:0/2814218740 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd02c0095a0 0x7fd02c00baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:30.568 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:30.564 7fd03ffff700 1 --2- 172.21.2.112:0/2814218740 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd02c006780 0x7fd02c008ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:30.568 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:30.564 7fd044b66700 1 -- 172.21.2.112:0/2814218740 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd02c006780 msgr2=0x7fd02c008ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:30.568 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:30.564 7fd044b66700 1 --2- 172.21.2.112:0/2814218740 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd02c006780 0x7fd02c008ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:30.569 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:30.564 7fd044b66700 1 -- 172.21.2.112:0/2814218740 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd02c0095a0 msgr2=0x7fd02c00baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:30.569 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:30.564 7fd044b66700 1 --2- 172.21.2.112:0/2814218740 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd02c0095a0 0x7fd02c00baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:30.569 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:30.564 7fd044b66700 1 -- 172.21.2.112:0/2814218740 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd02c0509a0 con 0x7fd02c003a60 2021-02-10T11:36:30.569 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:30.564 7fd045367700 1 --2- 172.21.2.112:0/2814218740 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd02c0095a0 0x7fd02c00baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:36:30.569 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:30.564 7fd045367700 1 -- 172.21.2.112:0/2814218740 reap_dead start 2021-02-10T11:36:30.569 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:30.564 7fd044b66700 1 --2- 172.21.2.112:0/2814218740 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd02c003a60 0x7fd02c005f60 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 rx=0x7fd028006ac0 tx=0x7fd028009af0).ready entity=mon.0 client_cookie=2df3fbfca279beec server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:30.569 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:30.565 7fd03f7fe700 1 -- 172.21.2.112:0/2814218740 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd028007280 con 0x7fd02c003a60 2021-02-10T11:36:30.569 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:30.565 7fd03f7fe700 1 -- 172.21.2.112:0/2814218740 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd028007500 con 0x7fd02c003a60 2021-02-10T11:36:30.569 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:30.565 7fd03f7fe700 1 -- 172.21.2.112:0/2814218740 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd028007a20 con 0x7fd02c003a60 2021-02-10T11:36:30.569 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:30.565 7fd046d4a700 1 -- 172.21.2.112:0/2814218740 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd02c003a60 msgr2=0x7fd02c005f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:30.569 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:30.565 7fd046d4a700 1 --2- 172.21.2.112:0/2814218740 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd02c003a60 0x7fd02c005f60 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 rx=0x7fd028006ac0 tx=0x7fd028009af0).stop 2021-02-10T11:36:30.569 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:30.565 7fd046d4a700 1 -- 172.21.2.112:0/2814218740 shutdown_connections 2021-02-10T11:36:30.569 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:30.565 7fd046d4a700 1 --2- 172.21.2.112:0/2814218740 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd02c003a60 0x7fd02c005f60 unknown :-1 s=CLOSED pgs=1634 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:30.570 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:30.565 7fd046d4a700 1 -- 172.21.2.112:0/2814218740 >> 172.21.2.112:0/2814218740 conn(0x7fd0400ce420 msgr2=0x7fd0400d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:30.570 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:30.565 7fd046d4a700 1 -- 172.21.2.112:0/2814218740 shutdown_connections 2021-02-10T11:36:30.570 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:30.565 7fd046d4a700 1 -- 172.21.2.112:0/2814218740 wait complete. 2021-02-10T11:36:30.570 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:30.580 7fd046d4a700 1 Processor -- start 2021-02-10T11:36:30.570 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:30.580 7fd046d4a700 1 -- start start 2021-02-10T11:36:30.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:30.581 7fd046d4a700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd0401a79a0 0x7fd0401a7da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:30.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:30.581 7fd046d4a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd0400bdb40 0x7fd0400bdf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:30.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:30.581 7fd046d4a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd040082ca0 0x7fd0400830a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:30.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:30.581 7fd046d4a700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd04019d990 con 0x7fd0400bdb40 2021-02-10T11:36:30.572 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:30.581 7fd046d4a700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd04019dbb0 con 0x7fd0401a79a0 2021-02-10T11:36:30.572 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:30.581 7fd046d4a700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0400836c0 con 0x7fd040082ca0 2021-02-10T11:36:30.572 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:30.581 7fd03ffff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd0400bdb40 0x7fd0400bdf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:30.572 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:30.581 7fd03ffff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd0400bdb40 0x7fd0400bdf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:52126/0 (socket says 172.21.2.112:52126) 2021-02-10T11:36:30.572 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:30.581 7fd03ffff700 1 -- 172.21.2.112:0/267607394 learned_addr learned my addr 172.21.2.112:0/267607394 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:36:30.572 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:30.581 7fd03ffff700 1 -- 172.21.2.112:0/267607394 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd040082ca0 msgr2=0x7fd0400830a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:36:30.572 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:30.581 7fd03ffff700 1 --2- 172.21.2.112:0/267607394 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd040082ca0 0x7fd0400830a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:30.572 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:30.581 7fd03ffff700 1 -- 172.21.2.112:0/267607394 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd0401a79a0 msgr2=0x7fd0401a7da0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:36:30.572 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:30.581 7fd03ffff700 1 --2- 172.21.2.112:0/267607394 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd0401a79a0 0x7fd0401a7da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:30.572 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:30.581 7fd03ffff700 1 -- 172.21.2.112:0/267607394 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0400838e0 con 0x7fd0400bdb40 2021-02-10T11:36:30.572 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:30.581 7fd03ffff700 1 --2- 172.21.2.112:0/267607394 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd0400bdb40 0x7fd0400bdf40 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 rx=0x7fd0300079d0 tx=0x7fd03000aca0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:30.572 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:30.581 7fd03d7fa700 1 -- 172.21.2.112:0/267607394 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd0300070d0 con 0x7fd0400bdb40 2021-02-10T11:36:30.572 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:30.581 7fd03d7fa700 1 -- 172.21.2.112:0/267607394 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0300044c0 con 0x7fd0400bdb40 2021-02-10T11:36:30.572 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:30.581 7fd03d7fa700 1 -- 172.21.2.112:0/267607394 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd0300049e0 con 0x7fd0400bdb40 2021-02-10T11:36:30.573 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:30.581 7fd046d4a700 1 -- 172.21.2.112:0/267607394 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0400838e0 con 0x7fd0400bdb40 2021-02-10T11:36:30.573 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:30.581 7fd046d4a700 1 -- 172.21.2.112:0/267607394 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd040084420 con 0x7fd0400bdb40 2021-02-10T11:36:30.573 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:30.582 7fd03d7fa700 1 -- 172.21.2.112:0/267607394 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd0300046a0 con 0x7fd0400bdb40 2021-02-10T11:36:30.573 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:30.582 7fd03d7fa700 1 --2- 172.21.2.112:0/267607394 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd0240186c0 0x7fd02401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:30.573 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:30.582 7fd03d7fa700 1 -- 172.21.2.112:0/267607394 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fd030032b50 con 0x7fd0400bdb40 2021-02-10T11:36:30.573 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:30.582 7fd044b66700 1 --2- 172.21.2.112:0/267607394 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd0240186c0 0x7fd02401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:30.573 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:30.583 7fd044b66700 1 --2- 172.21.2.112:0/267607394 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd0240186c0 0x7fd02401abc0 crc :-1 s=READY pgs=782 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:30.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:30.612 7fd046d4a700 1 -- 172.21.2.112:0/267607394 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd040082a70 con 0x7fd0400bdb40 2021-02-10T11:36:30.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:30.615 7fd03d7fa700 1 -- 172.21.2.112:0/267607394 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd0300046d0 con 0x7fd0400bdb40 2021-02-10T11:36:30.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:30.743 7fd046d4a700 1 -- 172.21.2.112:0/267607394 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "clone_0000000000397074", "target": ["mgr", ""]}) v1 -- 0x7fd040084420 con 0x7fd0240186c0 2021-02-10T11:36:30.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:30.752 7fd03d7fa700 1 -- 172.21.2.112:0/267607394 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fd0280009b0 con 0x7fd0240186c0 2021-02-10T11:36:30.748 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:30.761 7fd046d4a700 1 -- 172.21.2.112:0/267607394 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd0240186c0 msgr2=0x7fd02401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:30.748 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:30.761 7fd046d4a700 1 --2- 172.21.2.112:0/267607394 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd0240186c0 0x7fd02401abc0 crc :-1 s=READY pgs=782 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:30.748 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:30.761 7fd046d4a700 1 -- 172.21.2.112:0/267607394 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd0400bdb40 msgr2=0x7fd0400bdf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:30.748 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:30.761 7fd046d4a700 1 --2- 172.21.2.112:0/267607394 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd0400bdb40 0x7fd0400bdf40 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 rx=0x7fd0300079d0 tx=0x7fd03000aca0).stop 2021-02-10T11:36:30.749 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:30.762 7fd046d4a700 1 -- 172.21.2.112:0/267607394 shutdown_connections 2021-02-10T11:36:30.749 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:30.762 7fd046d4a700 1 --2- 172.21.2.112:0/267607394 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd0240186c0 0x7fd02401abc0 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:30.749 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:30.762 7fd046d4a700 1 --2- 172.21.2.112:0/267607394 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd040082ca0 0x7fd0400830a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:30.749 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:30.762 7fd046d4a700 1 --2- 172.21.2.112:0/267607394 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd0400bdb40 0x7fd0400bdf40 unknown :-1 s=CLOSED pgs=1635 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:30.749 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:30.762 7fd046d4a700 1 --2- 172.21.2.112:0/267607394 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd0401a79a0 0x7fd0401a7da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:30.749 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:30.762 7fd046d4a700 1 -- 172.21.2.112:0/267607394 >> 172.21.2.112:0/267607394 conn(0x7fd0400bef00 msgr2=0x7fd04009f200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:30.749 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:30.762 7fd046d4a700 1 -- 172.21.2.112:0/267607394 shutdown_connections 2021-02-10T11:36:30.749 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:30.762 7fd046d4a700 1 -- 172.21.2.112:0/267607394 wait complete. 2021-02-10T11:36:30.756 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo getfattr --only-values -n ceph.dir.entries /home/ubuntu/cephtest/mnt.0/./volumes/_deleting 2021-02-10T11:36:31.108 INFO:teuthology.orchestra.run.gibba012.stderr:getfattr: Removing leading '/' from absolute path names 2021-02-10T11:36:31.109 INFO:teuthology.orchestra.run.gibba012.stdout:0 2021-02-10T11:36:31.109 DEBUG:tasks.cephfs.mount:Directory ./volumes/_deleting seen empty from 0 after 0s 2021-02-10T11:36:31.109 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:36:31.109 DEBUG:teuthology.orchestra.run.gibba012:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:36:31.454 DEBUG:teuthology.orchestra.run.gibba020:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:36:31.799 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:36:31.800 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:31.800 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:36:31.800 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:36:32.135 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:36:32.135 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:36:32.136 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba012.front.sepia.ceph.com... 2021-02-10T11:36:32.136 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:36:32.136 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:36:32.492 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba012.stderr:ceph-fuse[141138]: fuse finished with error 0 and tester_r 0 2021-02-10T11:36:32.493 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:32.493 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:36:32.494 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:36:32.834 INFO:teuthology.orchestra.run.gibba012.stdout:ext2/ext3 2021-02-10T11:36:32.835 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:36:32.835 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:36:32.835 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:36:33.171 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:33.171 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:36:33.172 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:36:33.506 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:36:33.506 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:36:33.506 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba012.front.sepia.ceph.com... 2021-02-10T11:36:33.506 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:36:33.507 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:36:33.859 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba012.stderr:ceph-fuse[141806]: fuse finished with error 0 and tester_r 0 2021-02-10T11:36:33.860 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:33.861 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:36:33.861 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:36:34.194 INFO:teuthology.orchestra.run.gibba012.stdout:ext2/ext3 2021-02-10T11:36:34.194 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:36:34.195 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:36:34.195 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:36:34.531 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:34.531 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:36:34.531 DEBUG:teuthology.orchestra.run.gibba020:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:36:34.864 INFO:teuthology.orchestra.run.gibba020.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:36:34.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:36:34.865 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:36:34.865 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:36:34.865 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:36:35.109 INFO:tasks.ceph.osd.6.gibba020.stderr:2021-02-10 11:36:35.119 7fa2a6523700 -1 osd.6 pg_epoch: 74 pg[15.5( v 73'8 (0'0,73'8] local-lis/les=71/72 n=2 ec=71/71 lis/c 71/71 les/c/f 72/72/0 71/71/71) [6,3] r=0 lpr=71 crt=73'8 lcod 73'7 mlcod 73'7 active+clean+snaptrim trimq=[2~2]] removing snap head 2021-02-10T11:36:35.199 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:35.200 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:36:35.200 DEBUG:teuthology.orchestra.run.gibba020:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:36:35.532 INFO:teuthology.orchestra.run.gibba020.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:36:35.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:36:35.533 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:36:35.533 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:36:35.533 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:36:35.867 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config rm mon mon_allow_pool_delete 2021-02-10T11:36:36.297 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.294 7feabd177700 1 Processor -- start 2021-02-10T11:36:36.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.294 7feabd177700 1 -- start start 2021-02-10T11:36:36.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.294 7feabd177700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7feab80d1870 0x7feab81b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:36.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.294 7feabd177700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feab80f0b10 con 0x7feab81b0a10 2021-02-10T11:36:36.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.294 7feabd177700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feab80d0310 con 0x7feab80f0650 2021-02-10T11:36:36.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.294 7feabd177700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feab80e5ed0 con 0x7feab80d1870 2021-02-10T11:36:36.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.294 7feab659c700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7feab81b0a10 0x7feab80cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:51598/0 (socket says 172.21.2.112:51598) 2021-02-10T11:36:36.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.295 7feab659c700 1 -- 172.21.2.112:0/1255495486 learned_addr learned my addr 172.21.2.112:0/1255495486 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:36:36.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.295 7feab6d9d700 1 --2- 172.21.2.112:0/1255495486 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7feab80d1870 0x7feab81b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:36.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.295 7feab5d9b700 1 -- 172.21.2.112:0/1255495486 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2939763620 0 0) 0x7feaa80009b0 con 0x7feab81b0a10 2021-02-10T11:36:36.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.295 7feab5d9b700 1 -- 172.21.2.112:0/1255495486 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feaa4002fe0 con 0x7feab81b0a10 2021-02-10T11:36:36.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.295 7feab5d9b700 1 -- 172.21.2.112:0/1255495486 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2178862816 0 0) 0x7feab0001400 con 0x7feab80f0650 2021-02-10T11:36:36.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.295 7feab5d9b700 1 -- 172.21.2.112:0/1255495486 --> v1:172.21.2.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feaa4003210 con 0x7feab80f0650 2021-02-10T11:36:36.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.295 7feab6d9d700 1 -- 172.21.2.112:0/1255495486 >> v1:172.21.2.120:6789/0 conn(0x7feab80f0650 legacy=0x7feab80cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:36.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.295 7feab6d9d700 1 -- 172.21.2.112:0/1255495486 >> v1:172.21.2.112:6789/0 conn(0x7feab81b0a10 legacy=0x7feab80cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:36.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.295 7feab6d9d700 1 -- 172.21.2.112:0/1255495486 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feab80d0530 con 0x7feab80d1870 2021-02-10T11:36:36.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.295 7feab6d9d700 1 --2- 172.21.2.112:0/1255495486 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7feab80d1870 0x7feab81b03f0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7feaa0006650 tx=0x7feaa0009a00).ready entity=mon.2 client_cookie=6e66fca69d138c4 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:36.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.296 7feab5d9b700 1 -- 172.21.2.112:0/1255495486 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feaa00009b0 con 0x7feab80d1870 2021-02-10T11:36:36.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.296 7feab5d9b700 1 -- 172.21.2.112:0/1255495486 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feaa000c0c0 con 0x7feab80d1870 2021-02-10T11:36:36.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.296 7feab5d9b700 1 -- 172.21.2.112:0/1255495486 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feaa000c5a0 con 0x7feab80d1870 2021-02-10T11:36:36.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.296 7feabd177700 1 -- 172.21.2.112:0/1255495486 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7feab80d1870 msgr2=0x7feab81b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:36.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.296 7feabd177700 1 --2- 172.21.2.112:0/1255495486 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7feab80d1870 0x7feab81b03f0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7feaa0006650 tx=0x7feaa0009a00).stop 2021-02-10T11:36:36.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.297 7feabd177700 1 -- 172.21.2.112:0/1255495486 shutdown_connections 2021-02-10T11:36:36.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.297 7feabd177700 1 --2- 172.21.2.112:0/1255495486 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7feab80d1870 0x7feab81b03f0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:36.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.297 7feabd177700 1 -- 172.21.2.112:0/1255495486 >> 172.21.2.112:0/1255495486 conn(0x7feab80ce450 msgr2=0x7feab80d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:36.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.297 7feabd177700 1 -- 172.21.2.112:0/1255495486 shutdown_connections 2021-02-10T11:36:36.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.297 7feabd177700 1 -- 172.21.2.112:0/1255495486 wait complete. 2021-02-10T11:36:36.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.311 7feabd177700 1 Processor -- start 2021-02-10T11:36:36.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.311 7feabd177700 1 -- start start 2021-02-10T11:36:36.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.312 7feabd177700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7feab81a4a80 0x7feab80906b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:36.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.312 7feabd177700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7feab80c30f0 0x7feab80c34f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:36.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.312 7feabd177700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7feab80ef5a0 0x7feab80ef9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:36.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.312 7feabd177700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feab80c3b10 con 0x7feab80c30f0 2021-02-10T11:36:36.302 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.312 7feabd177700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feab80ef020 con 0x7feab80ef5a0 2021-02-10T11:36:36.302 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.312 7feabd177700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feab80effc0 con 0x7feab81a4a80 2021-02-10T11:36:36.302 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.312 7feab659c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7feab80c30f0 0x7feab80c34f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:36.302 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.312 7feab659c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7feab80c30f0 0x7feab80c34f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:52144/0 (socket says 172.21.2.112:52144) 2021-02-10T11:36:36.302 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.312 7feab659c700 1 -- 172.21.2.112:0/431488428 learned_addr learned my addr 172.21.2.112:0/431488428 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:36:36.302 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.312 7feab6d9d700 1 --2- 172.21.2.112:0/431488428 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7feab81a4a80 0x7feab80906b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:36.302 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.312 7feab759e700 1 --2- 172.21.2.112:0/431488428 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7feab80ef5a0 0x7feab80ef9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:36.302 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.312 7feab659c700 1 -- 172.21.2.112:0/431488428 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7feab81a4a80 msgr2=0x7feab80906b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:36.302 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.312 7feab659c700 1 --2- 172.21.2.112:0/431488428 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7feab81a4a80 0x7feab80906b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:36.302 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.312 7feab659c700 1 -- 172.21.2.112:0/431488428 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7feab80ef5a0 msgr2=0x7feab80ef9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:36.302 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.312 7feab659c700 1 --2- 172.21.2.112:0/431488428 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7feab80ef5a0 0x7feab80ef9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:36.302 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.312 7feab659c700 1 -- 172.21.2.112:0/431488428 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feab80ebe70 con 0x7feab80c30f0 2021-02-10T11:36:36.302 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.312 7feab659c700 1 --2- 172.21.2.112:0/431488428 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7feab80c30f0 0x7feab80c34f0 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 rx=0x7feaa8001c60 tx=0x7feaa800a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:36.303 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.313 7feaaf7fe700 1 -- 172.21.2.112:0/431488428 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feaa800acc0 con 0x7feab80c30f0 2021-02-10T11:36:36.303 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.313 7feaaf7fe700 1 -- 172.21.2.112:0/431488428 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feaa8004140 con 0x7feab80c30f0 2021-02-10T11:36:36.303 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.313 7feabd177700 1 -- 172.21.2.112:0/431488428 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feab80ebe70 con 0x7feab80c30f0 2021-02-10T11:36:36.303 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.313 7feabd177700 1 -- 172.21.2.112:0/431488428 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feab80ec400 con 0x7feab80c30f0 2021-02-10T11:36:36.303 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.313 7feaaf7fe700 1 -- 172.21.2.112:0/431488428 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feaa8004620 con 0x7feab80c30f0 2021-02-10T11:36:36.303 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.313 7feaaf7fe700 1 -- 172.21.2.112:0/431488428 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7feaa8004320 con 0x7feab80c30f0 2021-02-10T11:36:36.303 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.313 7feaaf7fe700 1 --2- 172.21.2.112:0/431488428 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fea9c018730 0x7fea9c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:36.303 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.313 7feaaf7fe700 1 -- 172.21.2.112:0/431488428 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5645+0+0 (secure 0 0 0) 0x7feaa80327b0 con 0x7feab80c30f0 2021-02-10T11:36:36.303 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.314 7feab6d9d700 1 --2- 172.21.2.112:0/431488428 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fea9c018730 0x7fea9c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:36.303 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.314 7feab6d9d700 1 --2- 172.21.2.112:0/431488428 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fea9c018730 0x7fea9c01abc0 crc :-1 s=READY pgs=783 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:36.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.344 7feabd177700 1 -- 172.21.2.112:0/431488428 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feab80f0b10 con 0x7feab80c30f0 2021-02-10T11:36:36.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.347 7feaaf7fe700 1 -- 172.21.2.112:0/431488428 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7feaa8000960 con 0x7feab80c30f0 2021-02-10T11:36:36.483 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.496 7feabd177700 1 -- 172.21.2.112:0/431488428 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7feab80c3b10 con 0x7feab80c30f0 2021-02-10T11:36:36.486 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.499 7feaaf7fe700 1 -- 172.21.2.112:0/431488428 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v21) v1 ==== 104+0+0 (secure 0 0 0) 0x7feaa8004140 con 0x7feab80c30f0 2021-02-10T11:36:36.487 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.501 7feabd177700 1 -- 172.21.2.112:0/431488428 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fea9c018730 msgr2=0x7fea9c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:36.488 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.501 7feabd177700 1 --2- 172.21.2.112:0/431488428 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fea9c018730 0x7fea9c01abc0 crc :-1 s=READY pgs=783 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:36.488 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.501 7feabd177700 1 -- 172.21.2.112:0/431488428 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7feab80c30f0 msgr2=0x7feab80c34f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:36.488 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.501 7feabd177700 1 --2- 172.21.2.112:0/431488428 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7feab80c30f0 0x7feab80c34f0 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 rx=0x7feaa8001c60 tx=0x7feaa800a8d0).stop 2021-02-10T11:36:36.488 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.501 7feabd177700 1 -- 172.21.2.112:0/431488428 shutdown_connections 2021-02-10T11:36:36.488 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.501 7feabd177700 1 --2- 172.21.2.112:0/431488428 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fea9c018730 0x7fea9c01abc0 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:36.488 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.501 7feabd177700 1 --2- 172.21.2.112:0/431488428 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7feab80ef5a0 0x7feab80ef9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:36.488 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.501 7feabd177700 1 --2- 172.21.2.112:0/431488428 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7feab80c30f0 0x7feab80c34f0 unknown :-1 s=CLOSED pgs=1637 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:36.488 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.501 7feabd177700 1 --2- 172.21.2.112:0/431488428 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7feab81a4a80 0x7feab80906b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:36.488 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.501 7feabd177700 1 -- 172.21.2.112:0/431488428 >> 172.21.2.112:0/431488428 conn(0x7feab80a6bb0 msgr2=0x7feab8095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:36.489 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.501 7feabd177700 1 -- 172.21.2.112:0/431488428 shutdown_connections 2021-02-10T11:36:36.489 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.501 7feabd177700 1 -- 172.21.2.112:0/431488428 wait complete. 2021-02-10T11:36:36.496 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestVolumes.test_clone_subvolume_info' 2021-02-10T11:36:36.925 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.921 7f0dcd99c700 1 Processor -- start 2021-02-10T11:36:36.925 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.921 7f0dcd99c700 1 -- start start 2021-02-10T11:36:36.925 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.921 7f0dcd99c700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f0dc80d1870 0x7f0dc81b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:36.925 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.921 7f0dcd99c700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0dc80f0650 0x7f0dc80cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:36.925 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.921 7f0dcd99c700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dc80f0b10 con 0x7f0dc81b0a10 2021-02-10T11:36:36.926 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.921 7f0dcd99c700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0dc80e5ed0 con 0x7f0dc80d1870 2021-02-10T11:36:36.926 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.921 7f0dcd99c700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0dc80d1c70 con 0x7f0dc80f0650 2021-02-10T11:36:36.926 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.921 7f0dc6d73700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f0dc81b0a10 0x7f0dc80cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:51610/0 (socket says 172.21.2.112:51610) 2021-02-10T11:36:36.926 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.921 7f0dc6d73700 1 -- 172.21.2.112:0/392950903 learned_addr learned my addr 172.21.2.112:0/392950903 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:36:36.926 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.922 7f0dc6572700 1 -- 172.21.2.112:0/392950903 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2358467133 0 0) 0x7f0dbc0010b0 con 0x7f0dc81b0a10 2021-02-10T11:36:36.926 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.922 7f0dc6572700 1 -- 172.21.2.112:0/392950903 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0db0002fe0 con 0x7f0dc81b0a10 2021-02-10T11:36:36.926 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.922 7f0dc7d75700 1 --2- 172.21.2.112:0/392950903 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0dc80f0650 0x7f0dc80cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:36.927 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.922 7f0dc7574700 1 --2- 172.21.2.112:0/392950903 >> v2:172.21.2.120:3300/0 conn(0x7f0dc80d1870 0x7f0dc81b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:36.927 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.922 7f0dc6572700 1 -- 172.21.2.112:0/392950903 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 430010783 0 0) 0x7f0dbc0010b0 con 0x7f0dc81b0a10 2021-02-10T11:36:36.927 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.922 7f0dc6572700 1 -- 172.21.2.112:0/392950903 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0dc80f0650 msgr2=0x7f0dc80cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:36.927 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.922 7f0dc6572700 1 --2- 172.21.2.112:0/392950903 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0dc80f0650 0x7f0dc80cfb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:36.927 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.922 7f0dc6572700 1 -- 172.21.2.112:0/392950903 >> v2:172.21.2.120:3300/0 conn(0x7f0dc80d1870 msgr2=0x7f0dc81b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:36.927 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.922 7f0dc6572700 1 --2- 172.21.2.112:0/392950903 >> v2:172.21.2.120:3300/0 conn(0x7f0dc80d1870 0x7f0dc81b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:36.927 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.922 7f0dc6572700 1 -- 172.21.2.112:0/392950903 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dc80d1e90 con 0x7f0dc81b0a10 2021-02-10T11:36:36.927 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.922 7f0dc6572700 1 -- 172.21.2.112:0/392950903 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f0dbc0015b0 con 0x7f0dc81b0a10 2021-02-10T11:36:36.928 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.922 7f0dc6572700 1 -- 172.21.2.112:0/392950903 >> v1:172.21.2.112:6789/0 conn(0x7f0dc81b0a10 legacy=0x7f0dc80cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:36.928 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.922 7f0dc6572700 1 --2- 172.21.2.112:0/392950903 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0db0003b10 0x7f0db0005fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:36.928 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.922 7f0dc6572700 1 --2- 172.21.2.112:0/392950903 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0db004a780 0x7f0db004cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:36.928 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.922 7f0dc6572700 1 --2- 172.21.2.112:0/392950903 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0db004d500 0x7f0db004fa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:36.928 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.922 7f0dc6572700 1 -- 172.21.2.112:0/392950903 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0db00502d0 con 0x7f0db004d500 2021-02-10T11:36:36.929 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.922 7f0dc6572700 1 -- 172.21.2.112:0/392950903 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0db00504f0 con 0x7f0db004a780 2021-02-10T11:36:36.929 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.922 7f0dc6572700 1 -- 172.21.2.112:0/392950903 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0db0050710 con 0x7f0db0003b10 2021-02-10T11:36:36.929 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.922 7f0dc7d75700 1 --2- 172.21.2.112:0/392950903 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0db004d500 0x7f0db004fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:36.929 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.922 7f0dc7574700 1 --2- 172.21.2.112:0/392950903 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0db0003b10 0x7f0db0005fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:36.929 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.922 7f0dc6d73700 1 --2- 172.21.2.112:0/392950903 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0db004a780 0x7f0db004cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:36.929 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.923 7f0dc7d75700 1 -- 172.21.2.112:0/392950903 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0db0003b10 msgr2=0x7f0db0005fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:36.929 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.923 7f0dc7d75700 1 --2- 172.21.2.112:0/392950903 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0db0003b10 0x7f0db0005fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:36.929 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.923 7f0dc7d75700 1 -- 172.21.2.112:0/392950903 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0db004a780 msgr2=0x7f0db004cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:36.929 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.923 7f0dc7d75700 1 --2- 172.21.2.112:0/392950903 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0db004a780 0x7f0db004cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:36.929 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.923 7f0dc7d75700 1 -- 172.21.2.112:0/392950903 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0db00509b0 con 0x7f0db004d500 2021-02-10T11:36:36.929 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.923 7f0dc7d75700 1 --2- 172.21.2.112:0/392950903 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0db004d500 0x7f0db004fa20 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 rx=0x7f0dc0009b10 tx=0x7f0dc0009eb0).ready entity=mon.0 client_cookie=895ab88ca0ab968c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:36.929 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.923 7f0dc6572700 1 -- 172.21.2.112:0/392950903 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0dc000b510 con 0x7f0db004d500 2021-02-10T11:36:36.930 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.923 7f0dc6572700 1 -- 172.21.2.112:0/392950903 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dc0003030 con 0x7f0db004d500 2021-02-10T11:36:36.930 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.923 7f0dc6572700 1 -- 172.21.2.112:0/392950903 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0dc000b790 con 0x7f0db004d500 2021-02-10T11:36:36.930 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.923 7f0dcd99c700 1 -- 172.21.2.112:0/392950903 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0db004d500 msgr2=0x7f0db004fa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:36.930 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.923 7f0dcd99c700 1 --2- 172.21.2.112:0/392950903 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0db004d500 0x7f0db004fa20 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 rx=0x7f0dc0009b10 tx=0x7f0dc0009eb0).stop 2021-02-10T11:36:36.930 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.923 7f0dc7d75700 1 -- 172.21.2.112:0/392950903 reap_dead start 2021-02-10T11:36:36.930 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.924 7f0dcd99c700 1 -- 172.21.2.112:0/392950903 shutdown_connections 2021-02-10T11:36:36.930 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.924 7f0dcd99c700 1 -- 172.21.2.112:0/392950903 >> 172.21.2.112:0/392950903 conn(0x7f0dc80ce450 msgr2=0x7f0dc80d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:36.930 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.924 7f0dcd99c700 1 -- 172.21.2.112:0/392950903 shutdown_connections 2021-02-10T11:36:36.930 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.924 7f0dcd99c700 1 -- 172.21.2.112:0/392950903 wait complete. 2021-02-10T11:36:36.930 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.938 7f0dcd99c700 1 Processor -- start 2021-02-10T11:36:36.930 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.939 7f0dcd99c700 1 -- start start 2021-02-10T11:36:36.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.939 7f0dcd99c700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0dc81a4a60 0x7f0dc80c1c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:36.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.939 7f0dcd99c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0dc80c2290 0x7f0dc80c2690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:36.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.939 7f0dcd99c700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0dc80c2cb0 0x7f0dc80bfd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:36.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.939 7f0dcd99c700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dc80f05b0 con 0x7f0dc80c2290 2021-02-10T11:36:36.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.939 7f0dcd99c700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dc80f07d0 con 0x7f0dc81a4a60 2021-02-10T11:36:36.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.939 7f0dcd99c700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0dc80c0340 con 0x7f0dc80c2cb0 2021-02-10T11:36:36.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.939 7f0dc6d73700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0dc80c2290 0x7f0dc80c2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:36.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.939 7f0dc6d73700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0dc80c2290 0x7f0dc80c2690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:52158/0 (socket says 172.21.2.112:52158) 2021-02-10T11:36:36.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.939 7f0dc6d73700 1 -- 172.21.2.112:0/2961180231 learned_addr learned my addr 172.21.2.112:0/2961180231 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:36:36.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.939 7f0dc7d75700 1 --2- 172.21.2.112:0/2961180231 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0dc80c2cb0 0x7f0dc80bfd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:36.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.939 7f0dc7574700 1 --2- 172.21.2.112:0/2961180231 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0dc81a4a60 0x7f0dc80c1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:36.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.940 7f0dc6d73700 1 -- 172.21.2.112:0/2961180231 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0dc80c2cb0 msgr2=0x7f0dc80bfd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:36.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.940 7f0dc6d73700 1 --2- 172.21.2.112:0/2961180231 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0dc80c2cb0 0x7f0dc80bfd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:36.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.940 7f0dc6d73700 1 -- 172.21.2.112:0/2961180231 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0dc81a4a60 msgr2=0x7f0dc80c1c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:36.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.940 7f0dc6d73700 1 --2- 172.21.2.112:0/2961180231 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0dc81a4a60 0x7f0dc80c1c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:36.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.940 7f0dc6d73700 1 -- 172.21.2.112:0/2961180231 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dc80c0560 con 0x7f0dc80c2290 2021-02-10T11:36:36.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.940 7f0dc6d73700 1 --2- 172.21.2.112:0/2961180231 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0dc80c2290 0x7f0dc80c2690 secure :-1 s=READY pgs=1640 cs=0 l=1 rev1=1 rx=0x7f0dbc001c00 tx=0x7f0dbc00a8e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:36.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.940 7f0daffff700 1 -- 172.21.2.112:0/2961180231 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0dbc0009b0 con 0x7f0dc80c2290 2021-02-10T11:36:36.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.940 7f0daffff700 1 -- 172.21.2.112:0/2961180231 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dbc00acd0 con 0x7f0dc80c2290 2021-02-10T11:36:36.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.940 7f0daffff700 1 -- 172.21.2.112:0/2961180231 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0dbc004540 con 0x7f0dc80c2290 2021-02-10T11:36:36.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.940 7f0dcd99c700 1 -- 172.21.2.112:0/2961180231 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0dc80c0560 con 0x7f0dc80c2290 2021-02-10T11:36:36.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.940 7f0dcd99c700 1 -- 172.21.2.112:0/2961180231 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0dc80c0560 con 0x7f0dc80c2290 2021-02-10T11:36:36.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.941 7f0daffff700 1 -- 172.21.2.112:0/2961180231 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0dbc004350 con 0x7f0dc80c2290 2021-02-10T11:36:36.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.941 7f0daffff700 1 --2- 172.21.2.112:0/2961180231 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0da8018730 0x7f0da801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:36.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.941 7f0daffff700 1 -- 172.21.2.112:0/2961180231 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5645+0+0 (secure 0 0 0) 0x7f0dbc0327c0 con 0x7f0dc80c2290 2021-02-10T11:36:36.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.941 7f0dc7574700 1 --2- 172.21.2.112:0/2961180231 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0da8018730 0x7f0da801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:36.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.942 7f0dc7574700 1 --2- 172.21.2.112:0/2961180231 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0da8018730 0x7f0da801ac10 crc :-1 s=READY pgs=784 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:36.957 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.971 7f0dcd99c700 1 -- 172.21.2.112:0/2961180231 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0dc80f0af0 con 0x7f0dc80c2290 2021-02-10T11:36:36.960 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:36.973 7f0daffff700 1 -- 172.21.2.112:0/2961180231 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0dbc004140 con 0x7f0dc80c2290 2021-02-10T11:36:37.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:37.106 7f0dcd99c700 1 -- 172.21.2.112:0/2961180231 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_clone_subvolume_info"]} v 0) v1 -- 0x7f0dc80f0af0 con 0x7f0dc80c2290 2021-02-10T11:36:37.112 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:37.126 7f0daffff700 1 -- 172.21.2.112:0/2961180231 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_clone_subvolume_info"]}]=0 v413) v1 ==== 142+0+0 (secure 0 0 0) 0x7f0dbc004350 con 0x7f0dc80c2290 2021-02-10T11:36:37.126 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:37.139 7f0dcd99c700 1 -- 172.21.2.112:0/2961180231 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0da8018730 msgr2=0x7f0da801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:37.126 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:37.139 7f0dcd99c700 1 --2- 172.21.2.112:0/2961180231 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0da8018730 0x7f0da801ac10 crc :-1 s=READY pgs=784 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:37.126 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:37.139 7f0dcd99c700 1 -- 172.21.2.112:0/2961180231 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0dc80c2290 msgr2=0x7f0dc80c2690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:37.126 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:37.139 7f0dcd99c700 1 --2- 172.21.2.112:0/2961180231 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0dc80c2290 0x7f0dc80c2690 secure :-1 s=READY pgs=1640 cs=0 l=1 rev1=1 rx=0x7f0dbc001c00 tx=0x7f0dbc00a8e0).stop 2021-02-10T11:36:37.126 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:37.140 7f0dcd99c700 1 -- 172.21.2.112:0/2961180231 shutdown_connections 2021-02-10T11:36:37.126 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:37.140 7f0dcd99c700 1 --2- 172.21.2.112:0/2961180231 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0da8018730 0x7f0da801ac10 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:37.127 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:37.140 7f0dcd99c700 1 --2- 172.21.2.112:0/2961180231 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0dc80c2cb0 0x7f0dc80bfd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:37.127 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:37.140 7f0dcd99c700 1 --2- 172.21.2.112:0/2961180231 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0dc80c2290 0x7f0dc80c2690 unknown :-1 s=CLOSED pgs=1640 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:37.127 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:37.140 7f0dcd99c700 1 --2- 172.21.2.112:0/2961180231 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0dc81a4a60 0x7f0dc80c1c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:37.127 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:37.140 7f0dcd99c700 1 -- 172.21.2.112:0/2961180231 >> 172.21.2.112:0/2961180231 conn(0x7f0dc80ebe30 msgr2=0x7f0dc80bf200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:37.127 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:37.140 7f0dcd99c700 1 -- 172.21.2.112:0/2961180231 shutdown_connections 2021-02-10T11:36:37.127 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:37.140 7f0dcd99c700 1 -- 172.21.2.112:0/2961180231 wait complete. 2021-02-10T11:36:37.134 INFO:tasks.cephfs_test_runner:test_clone_subvolume_info (tasks.cephfs.test_volumes.TestVolumes) ... ok 2021-02-10T11:36:37.135 INFO:tasks.cephfs_test_runner:Starting test: test_connection_expiration (tasks.cephfs.test_volumes.TestVolumes) 2021-02-10T11:36:37.135 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestVolumes.test_connection_expiration' 2021-02-10T11:36:37.564 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:37.558 7f8df04de700 1 Processor -- start 2021-02-10T11:36:37.564 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:37.559 7f8df04de700 1 -- start start 2021-02-10T11:36:37.565 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:37.559 7f8df04de700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f8de80d0340 0x7f8de80ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:37.565 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:37.559 7f8df04de700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8de80f0b10 con 0x7f8de80d1c70 2021-02-10T11:36:37.565 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:37.559 7f8df04de700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8de81b0be0 con 0x7f8de80d1870 2021-02-10T11:36:37.565 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:37.559 7f8df04de700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8de80e5ed0 con 0x7f8de80d0340 2021-02-10T11:36:37.565 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:37.559 7f8deeafb700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f8de80d1c70 0x7f8de80efc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:51630/0 (socket says 172.21.2.112:51630) 2021-02-10T11:36:37.565 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:37.559 7f8deeafb700 1 -- 172.21.2.112:0/135906055 learned_addr learned my addr 172.21.2.112:0/135906055 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:36:37.565 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:37.560 7f8ded2f8700 1 -- 172.21.2.112:0/135906055 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1899992927 0 0) 0x7f8de0000f90 con 0x7f8de80d1c70 2021-02-10T11:36:37.565 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:37.560 7f8ded2f8700 1 -- 172.21.2.112:0/135906055 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8dd4002fe0 con 0x7f8de80d1c70 2021-02-10T11:36:37.565 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:37.560 7f8dedaf9700 1 --2- 172.21.2.112:0/135906055 >> v2:172.21.2.120:3300/0 conn(0x7f8de80d0340 0x7f8de80ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:37.565 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:37.560 7f8ded2f8700 1 -- 172.21.2.112:0/135906055 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1629156668 0 0) 0x7f8de0001460 con 0x7f8de80d1c70 2021-02-10T11:36:37.565 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:37.560 7f8ded2f8700 1 -- 172.21.2.112:0/135906055 >> v2:172.21.2.120:3300/0 conn(0x7f8de80d0340 msgr2=0x7f8de80ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:37.565 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:37.560 7f8ded2f8700 1 --2- 172.21.2.112:0/135906055 >> v2:172.21.2.120:3300/0 conn(0x7f8de80d0340 0x7f8de80ef640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:37.566 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:37.560 7f8ded2f8700 1 -- 172.21.2.112:0/135906055 >> v1:172.21.2.120:6789/0 conn(0x7f8de80d1870 legacy=0x7f8de81b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:37.566 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:37.560 7f8ded2f8700 1 -- 172.21.2.112:0/135906055 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8de80f0450 con 0x7f8de80d1c70 2021-02-10T11:36:37.566 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:37.560 7f8ded2f8700 1 -- 172.21.2.112:0/135906055 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f8de0001980 con 0x7f8de80d1c70 2021-02-10T11:36:37.566 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:37.560 7f8ded2f8700 1 -- 172.21.2.112:0/135906055 >> v1:172.21.2.112:6789/0 conn(0x7f8de80d1c70 legacy=0x7f8de80efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:37.566 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:37.560 7f8ded2f8700 1 --2- 172.21.2.112:0/135906055 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8dd4003ac0 0x7f8dd4005f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:37.566 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:37.560 7f8ded2f8700 1 --2- 172.21.2.112:0/135906055 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8dd4006780 0x7f8dd4008ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:37.566 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:37.561 7f8ded2f8700 1 --2- 172.21.2.112:0/135906055 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8dd40095a0 0x7f8dd400baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:37.566 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:37.561 7f8ded2f8700 1 -- 172.21.2.112:0/135906055 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dd400c2f0 con 0x7f8dd4003ac0 2021-02-10T11:36:37.566 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:37.561 7f8ded2f8700 1 -- 172.21.2.112:0/135906055 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dd400c510 con 0x7f8dd40095a0 2021-02-10T11:36:37.566 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:37.561 7f8ded2f8700 1 -- 172.21.2.112:0/135906055 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8dd400c730 con 0x7f8dd4006780 2021-02-10T11:36:37.566 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:37.561 7f8dee2fa700 1 --2- 172.21.2.112:0/135906055 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8dd4003ac0 0x7f8dd4005f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:37.566 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:37.561 7f8dedaf9700 1 --2- 172.21.2.112:0/135906055 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8dd4006780 0x7f8dd4008ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:37.566 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:37.561 7f8deeafb700 1 --2- 172.21.2.112:0/135906055 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8dd40095a0 0x7f8dd400baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:37.567 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:37.561 7f8dee2fa700 1 -- 172.21.2.112:0/135906055 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8dd4006780 msgr2=0x7f8dd4008ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:37.567 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:37.561 7f8dee2fa700 1 --2- 172.21.2.112:0/135906055 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8dd4006780 0x7f8dd4008ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:37.567 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:37.561 7f8dee2fa700 1 -- 172.21.2.112:0/135906055 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8dd40095a0 msgr2=0x7f8dd400baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:37.567 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:37.561 7f8dee2fa700 1 --2- 172.21.2.112:0/135906055 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8dd40095a0 0x7f8dd400baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:37.567 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:37.561 7f8dee2fa700 1 -- 172.21.2.112:0/135906055 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dd40509a0 con 0x7f8dd4003ac0 2021-02-10T11:36:37.567 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:37.561 7f8deeafb700 1 -- 172.21.2.112:0/135906055 reap_dead start 2021-02-10T11:36:37.567 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:37.561 7f8dee2fa700 1 --2- 172.21.2.112:0/135906055 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8dd4003ac0 0x7f8dd4005f60 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 rx=0x7f8de4006ab0 tx=0x7f8de4009b40).ready entity=mon.0 client_cookie=6c19ab4662124d02 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:37.567 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:37.561 7f8ded2f8700 1 -- 172.21.2.112:0/135906055 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8de400b2e0 con 0x7f8dd4003ac0 2021-02-10T11:36:37.567 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:37.561 7f8ded2f8700 1 -- 172.21.2.112:0/135906055 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8de400b660 con 0x7f8dd4003ac0 2021-02-10T11:36:37.567 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:37.561 7f8ded2f8700 1 -- 172.21.2.112:0/135906055 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8de400bac0 con 0x7f8dd4003ac0 2021-02-10T11:36:37.567 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:37.561 7f8df04de700 1 -- 172.21.2.112:0/135906055 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8dd4003ac0 msgr2=0x7f8dd4005f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:37.567 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:37.561 7f8df04de700 1 --2- 172.21.2.112:0/135906055 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8dd4003ac0 0x7f8dd4005f60 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 rx=0x7f8de4006ab0 tx=0x7f8de4009b40).stop 2021-02-10T11:36:37.567 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:37.562 7f8df04de700 1 -- 172.21.2.112:0/135906055 shutdown_connections 2021-02-10T11:36:37.568 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:37.562 7f8df04de700 1 --2- 172.21.2.112:0/135906055 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8dd4003ac0 0x7f8dd4005f60 unknown :-1 s=CLOSED pgs=1642 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:37.568 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:37.562 7f8df04de700 1 -- 172.21.2.112:0/135906055 >> 172.21.2.112:0/135906055 conn(0x7f8de80ce450 msgr2=0x7f8de80d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:37.568 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:37.562 7f8df04de700 1 -- 172.21.2.112:0/135906055 shutdown_connections 2021-02-10T11:36:37.568 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:37.562 7f8df04de700 1 -- 172.21.2.112:0/135906055 wait complete. 2021-02-10T11:36:37.568 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:37.577 7f8df04de700 1 Processor -- start 2021-02-10T11:36:37.568 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:37.577 7f8df04de700 1 -- start start 2021-02-10T11:36:37.568 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:37.577 7f8df04de700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8de8195650 0x7f8de80b5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:37.568 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:37.577 7f8df04de700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8de80b5b80 0x7f8de81a2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:37.568 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:37.577 7f8df04de700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8de80acfa0 0x7f8de81a3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:37.568 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:37.577 7f8df04de700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8de80b6010 con 0x7f8de8195650 2021-02-10T11:36:37.568 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:37.577 7f8df04de700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8de81a3a60 con 0x7f8de80b5b80 2021-02-10T11:36:37.568 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:37.577 7f8df04de700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8de81a3c40 con 0x7f8de80acfa0 2021-02-10T11:36:37.568 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:37.578 7f8dee2fa700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8de8195650 0x7f8de80b5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:37.569 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:37.578 7f8dee2fa700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8de8195650 0x7f8de80b5560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:52176/0 (socket says 172.21.2.112:52176) 2021-02-10T11:36:37.569 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:37.578 7f8dee2fa700 1 -- 172.21.2.112:0/1449618453 learned_addr learned my addr 172.21.2.112:0/1449618453 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:36:37.569 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:37.578 7f8dee2fa700 1 -- 172.21.2.112:0/1449618453 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8de80acfa0 msgr2=0x7f8de81a3440 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:36:37.569 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:37.578 7f8dee2fa700 1 --2- 172.21.2.112:0/1449618453 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8de80acfa0 0x7f8de81a3440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:37.569 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:37.578 7f8dee2fa700 1 -- 172.21.2.112:0/1449618453 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8de80b5b80 msgr2=0x7f8de81a2e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:37.569 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:37.578 7f8dee2fa700 1 --2- 172.21.2.112:0/1449618453 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8de80b5b80 0x7f8de81a2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:37.569 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:37.578 7f8dee2fa700 1 -- 172.21.2.112:0/1449618453 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8de80bcae0 con 0x7f8de8195650 2021-02-10T11:36:37.569 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:37.578 7f8dee2fa700 1 --2- 172.21.2.112:0/1449618453 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8de8195650 0x7f8de80b5560 secure :-1 s=READY pgs=1643 cs=0 l=1 rev1=1 rx=0x7f8de40093a0 tx=0x7f8de400b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:37.570 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:37.578 7f8ddeffd700 1 -- 172.21.2.112:0/1449618453 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8de400bd30 con 0x7f8de8195650 2021-02-10T11:36:37.570 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:37.578 7f8ddeffd700 1 -- 172.21.2.112:0/1449618453 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8de4006240 con 0x7f8de8195650 2021-02-10T11:36:37.570 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:37.578 7f8ddeffd700 1 -- 172.21.2.112:0/1449618453 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8de4005720 con 0x7f8de8195650 2021-02-10T11:36:37.570 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:37.578 7f8df04de700 1 -- 172.21.2.112:0/1449618453 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8de80bcae0 con 0x7f8de8195650 2021-02-10T11:36:37.570 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:37.578 7f8df04de700 1 -- 172.21.2.112:0/1449618453 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8de80bd070 con 0x7f8de8195650 2021-02-10T11:36:37.570 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:37.579 7f8ddeffd700 1 -- 172.21.2.112:0/1449618453 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8de4006750 con 0x7f8de8195650 2021-02-10T11:36:37.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:37.579 7f8ddeffd700 1 --2- 172.21.2.112:0/1449618453 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8dd00186c0 0x7f8dd001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:37.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:37.579 7f8ddeffd700 1 -- 172.21.2.112:0/1449618453 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5645+0+0 (secure 0 0 0) 0x7f8de4032600 con 0x7f8de8195650 2021-02-10T11:36:37.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:37.580 7f8dedaf9700 1 --2- 172.21.2.112:0/1449618453 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8dd00186c0 0x7f8dd001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:37.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:37.580 7f8dedaf9700 1 --2- 172.21.2.112:0/1449618453 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8dd00186c0 0x7f8dd001abc0 crc :-1 s=READY pgs=785 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:37.596 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:37.609 7f8df04de700 1 -- 172.21.2.112:0/1449618453 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8de80bcaa0 con 0x7f8de8195650 2021-02-10T11:36:37.598 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:37.612 7f8ddeffd700 1 -- 172.21.2.112:0/1449618453 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8de4030d70 con 0x7f8de8195650 2021-02-10T11:36:37.732 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:37.746 7f8df04de700 1 -- 172.21.2.112:0/1449618453 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_connection_expiration"]} v 0) v1 -- 0x7f8de80bcaa0 con 0x7f8de8195650 2021-02-10T11:36:38.118 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:38.132 7f8ddeffd700 1 -- 172.21.2.112:0/1449618453 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_connection_expiration"]}]=0 v414) v1 ==== 146+0+0 (secure 0 0 0) 0x7f8de4006750 con 0x7f8de8195650 2021-02-10T11:36:38.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:38.165 7f8df04de700 1 -- 172.21.2.112:0/1449618453 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8dd00186c0 msgr2=0x7f8dd001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:38.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:38.165 7f8df04de700 1 --2- 172.21.2.112:0/1449618453 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8dd00186c0 0x7f8dd001abc0 crc :-1 s=READY pgs=785 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:38.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:38.165 7f8df04de700 1 -- 172.21.2.112:0/1449618453 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8de8195650 msgr2=0x7f8de80b5560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:38.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:38.165 7f8df04de700 1 --2- 172.21.2.112:0/1449618453 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8de8195650 0x7f8de80b5560 secure :-1 s=READY pgs=1643 cs=0 l=1 rev1=1 rx=0x7f8de40093a0 tx=0x7f8de400b940).stop 2021-02-10T11:36:38.152 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:38.165 7f8df04de700 1 -- 172.21.2.112:0/1449618453 shutdown_connections 2021-02-10T11:36:38.152 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:38.165 7f8df04de700 1 --2- 172.21.2.112:0/1449618453 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8dd00186c0 0x7f8dd001abc0 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:38.152 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:38.165 7f8df04de700 1 --2- 172.21.2.112:0/1449618453 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8de80b5b80 0x7f8de81a2e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:38.152 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:38.165 7f8df04de700 1 --2- 172.21.2.112:0/1449618453 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8de80acfa0 0x7f8de81a3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:38.152 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:38.165 7f8df04de700 1 --2- 172.21.2.112:0/1449618453 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8de8195650 0x7f8de80b5560 unknown :-1 s=CLOSED pgs=1643 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:38.152 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:38.165 7f8df04de700 1 -- 172.21.2.112:0/1449618453 >> 172.21.2.112:0/1449618453 conn(0x7f8de80b3040 msgr2=0x7f8de80c3250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:38.152 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:38.165 7f8df04de700 1 -- 172.21.2.112:0/1449618453 shutdown_connections 2021-02-10T11:36:38.152 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:38.165 7f8df04de700 1 -- 172.21.2.112:0/1449618453 wait complete. 2021-02-10T11:36:38.160 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:36:38.590 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:38.587 7fa435367700 1 Processor -- start 2021-02-10T11:36:38.591 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:38.587 7fa435367700 1 -- start start 2021-02-10T11:36:38.591 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:38.588 7fa435367700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa4300d1870 0x7fa4301b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:38.591 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:38.588 7fa435367700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fa4301b0ae0 0x7fa430108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:38.591 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:38.588 7fa435367700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fa4300d1c70 0x7fa430108f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:38.591 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:38.588 7fa435367700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4300e5ef0 con 0x7fa4301b0ae0 2021-02-10T11:36:38.591 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:38.588 7fa435367700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa430109a60 con 0x7fa4300d1c70 2021-02-10T11:36:38.591 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:38.588 7fa435367700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa430088070 con 0x7fa4300d1870 2021-02-10T11:36:38.591 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:38.588 7fa42687c700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fa4301b0ae0 0x7fa430108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:38.591 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:38.588 7fa42f07d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa4300d1870 0x7fa4301b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:38.591 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:38.588 7fa42687c700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fa4301b0ae0 0x7fa430108940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:52186/0 (socket says 172.21.2.112:52186) 2021-02-10T11:36:38.591 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:38.588 7fa42f07d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa4300d1870 0x7fa4301b04c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3301/0 says I am v2:172.21.2.112:51866/0 (socket says 172.21.2.112:51866) 2021-02-10T11:36:38.592 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:38.588 7fa42687c700 1 -- 172.21.2.112:0/1050873018 learned_addr learned my addr 172.21.2.112:0/1050873018 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:36:38.592 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:38.588 7fa42f87e700 1 --2- 172.21.2.112:0/1050873018 >> v2:172.21.2.120:3300/0 conn(0x7fa4300d1c70 0x7fa430108f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:38.592 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:38.588 7fa42687c700 1 -- 172.21.2.112:0/1050873018 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa4300d1870 msgr2=0x7fa4301b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:38.592 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:38.588 7fa42687c700 1 --2- 172.21.2.112:0/1050873018 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa4300d1870 0x7fa4301b04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:38.592 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:38.588 7fa42687c700 1 -- 172.21.2.112:0/1050873018 >> v2:172.21.2.120:3300/0 conn(0x7fa4300d1c70 msgr2=0x7fa430108f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:38.592 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:38.588 7fa42687c700 1 --2- 172.21.2.112:0/1050873018 >> v2:172.21.2.120:3300/0 conn(0x7fa4300d1c70 0x7fa430108f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:38.592 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:38.588 7fa42687c700 1 -- 172.21.2.112:0/1050873018 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa430109580 con 0x7fa4301b0ae0 2021-02-10T11:36:38.592 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:38.588 7fa42f07d700 1 --2- 172.21.2.112:0/1050873018 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa4300d1870 0x7fa4301b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:36:38.592 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:38.589 7fa42687c700 1 --2- 172.21.2.112:0/1050873018 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa4301b0ae0 0x7fa430108940 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 rx=0x7fa418006700 tx=0x7fa418009b80).ready entity=mon.0 client_cookie=bac8bd6ab6f7d21 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:38.592 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:38.589 7fa42e87c700 1 -- 172.21.2.112:0/1050873018 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa41800c0c0 con 0x7fa4301b0ae0 2021-02-10T11:36:38.592 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:38.589 7fa42e87c700 1 -- 172.21.2.112:0/1050873018 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa41800c400 con 0x7fa4301b0ae0 2021-02-10T11:36:38.592 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:38.589 7fa42e87c700 1 -- 172.21.2.112:0/1050873018 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa41800c8e0 con 0x7fa4301b0ae0 2021-02-10T11:36:38.592 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:38.589 7fa435367700 1 -- 172.21.2.112:0/1050873018 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa4301b0ae0 msgr2=0x7fa430108940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:38.592 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:38.589 7fa435367700 1 --2- 172.21.2.112:0/1050873018 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa4301b0ae0 0x7fa430108940 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 rx=0x7fa418006700 tx=0x7fa418009b80).stop 2021-02-10T11:36:38.593 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:38.590 7fa435367700 1 -- 172.21.2.112:0/1050873018 shutdown_connections 2021-02-10T11:36:38.593 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:38.590 7fa435367700 1 --2- 172.21.2.112:0/1050873018 >> v2:172.21.2.120:3300/0 conn(0x7fa4300d1c70 0x7fa430108f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:38.593 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:38.590 7fa435367700 1 --2- 172.21.2.112:0/1050873018 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa4301b0ae0 0x7fa430108940 unknown :-1 s=CLOSED pgs=1644 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:38.593 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:38.590 7fa435367700 1 --2- 172.21.2.112:0/1050873018 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa4300d1870 0x7fa4301b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:38.593 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:38.590 7fa435367700 1 -- 172.21.2.112:0/1050873018 >> 172.21.2.112:0/1050873018 conn(0x7fa4300ce420 msgr2=0x7fa4300d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:38.593 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:38.590 7fa435367700 1 -- 172.21.2.112:0/1050873018 shutdown_connections 2021-02-10T11:36:38.593 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:38.590 7fa435367700 1 -- 172.21.2.112:0/1050873018 wait complete. 2021-02-10T11:36:38.595 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:38.604 7fa435367700 1 Processor -- start 2021-02-10T11:36:38.595 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:38.605 7fa435367700 1 -- start start 2021-02-10T11:36:38.595 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:38.605 7fa435367700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa4300a2580 0x7fa4300a1cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:38.595 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:38.605 7fa435367700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa430094c40 0x7fa4301a75e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:38.595 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:38.605 7fa435367700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa430095040 0x7fa4301a7c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:38.595 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:38.605 7fa435367700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa430188180 con 0x7fa430095040 2021-02-10T11:36:38.595 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:38.605 7fa435367700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa430187e70 con 0x7fa430094c40 2021-02-10T11:36:38.595 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:38.605 7fa435367700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4301a8220 con 0x7fa4300a2580 2021-02-10T11:36:38.595 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:38.605 7fa42687c700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa430094c40 0x7fa4301a75e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:38.595 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:38.605 7fa42f87e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa430095040 0x7fa4301a7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:38.595 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:38.605 7fa42687c700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa430094c40 0x7fa4301a75e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3300/0 says I am v2:172.21.2.112:54470/0 (socket says 172.21.2.112:54470) 2021-02-10T11:36:38.596 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:38.605 7fa42687c700 1 -- 172.21.2.112:0/3283690970 learned_addr learned my addr 172.21.2.112:0/3283690970 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:36:38.596 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:38.605 7fa42f07d700 1 --2- 172.21.2.112:0/3283690970 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa4300a2580 0x7fa4300a1cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:38.596 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:38.606 7fa42687c700 1 -- 172.21.2.112:0/3283690970 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa4300a2580 msgr2=0x7fa4300a1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:38.596 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:38.606 7fa42687c700 1 --2- 172.21.2.112:0/3283690970 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa4300a2580 0x7fa4300a1cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:38.596 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:38.606 7fa42687c700 1 -- 172.21.2.112:0/3283690970 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa430095040 msgr2=0x7fa4301a7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:38.596 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:38.606 7fa42687c700 1 --2- 172.21.2.112:0/3283690970 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa430095040 0x7fa4301a7c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:38.596 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:38.606 7fa42687c700 1 -- 172.21.2.112:0/3283690970 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa430188700 con 0x7fa430094c40 2021-02-10T11:36:38.596 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:38.606 7fa42f87e700 1 --2- 172.21.2.112:0/3283690970 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa430095040 0x7fa4301a7c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:36:38.596 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:38.606 7fa42687c700 1 --2- 172.21.2.112:0/3283690970 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa430094c40 0x7fa4301a75e0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 rx=0x7fa41800b550 tx=0x7fa41800b8f0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:38.597 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:38.606 7fa42c878700 1 -- 172.21.2.112:0/3283690970 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa41800bce0 con 0x7fa430094c40 2021-02-10T11:36:38.597 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:38.606 7fa42c878700 1 -- 172.21.2.112:0/3283690970 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa418002030 con 0x7fa430094c40 2021-02-10T11:36:38.597 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:38.606 7fa42c878700 1 -- 172.21.2.112:0/3283690970 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa418002480 con 0x7fa430094c40 2021-02-10T11:36:38.597 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:38.606 7fa435367700 1 -- 172.21.2.112:0/3283690970 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa430188700 con 0x7fa430094c40 2021-02-10T11:36:38.597 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:38.606 7fa435367700 1 -- 172.21.2.112:0/3283690970 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa430188c90 con 0x7fa430094c40 2021-02-10T11:36:38.597 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:38.607 7fa42c878700 1 -- 172.21.2.112:0/3283690970 <== mon.1 v2:172.21.2.120:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa418000c90 con 0x7fa430094c40 2021-02-10T11:36:38.597 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:38.607 7fa42c878700 1 --2- 172.21.2.112:0/3283690970 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa4140186c0 0x7fa41401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:38.597 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:38.607 7fa42c878700 1 -- 172.21.2.112:0/3283690970 <== mon.1 v2:172.21.2.120:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5645+0+0 (secure 0 0 0) 0x7fa4180325e0 con 0x7fa430094c40 2021-02-10T11:36:38.597 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:38.607 7fa42f07d700 1 --2- 172.21.2.112:0/3283690970 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa4140186c0 0x7fa41401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:38.597 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:38.608 7fa42f07d700 1 --2- 172.21.2.112:0/3283690970 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa4140186c0 0x7fa41401abc0 crc :-1 s=READY pgs=786 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:38.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:38.637 7fa435367700 1 -- 172.21.2.112:0/3283690970 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa430082980 con 0x7fa430094c40 2021-02-10T11:36:38.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:38.640 7fa42c878700 1 -- 172.21.2.112:0/3283690970 <== mon.1 v2:172.21.2.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa4180093e0 con 0x7fa430094c40 2021-02-10T11:36:38.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:38.794 7fa435367700 1 -- 172.21.2.112:0/3283690970 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fa430109a60 con 0x7fa430094c40 2021-02-10T11:36:38.787 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:38.801 7fa42c878700 1 -- 172.21.2.112:0/3283690970 <== mon.1 v2:172.21.2.120:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v22)=0 v22) v1 ==== 122+0+0 (secure 0 0 0) 0x7fa418000c90 con 0x7fa430094c40 2021-02-10T11:36:38.790 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:38.803 7fa435367700 1 -- 172.21.2.112:0/3283690970 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa4140186c0 msgr2=0x7fa41401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:38.790 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:38.803 7fa435367700 1 --2- 172.21.2.112:0/3283690970 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa4140186c0 0x7fa41401abc0 crc :-1 s=READY pgs=786 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:38.790 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:38.803 7fa435367700 1 -- 172.21.2.112:0/3283690970 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa430094c40 msgr2=0x7fa4301a75e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:38.790 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:38.803 7fa435367700 1 --2- 172.21.2.112:0/3283690970 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa430094c40 0x7fa4301a75e0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 rx=0x7fa41800b550 tx=0x7fa41800b8f0).stop 2021-02-10T11:36:38.790 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:38.804 7fa435367700 1 -- 172.21.2.112:0/3283690970 shutdown_connections 2021-02-10T11:36:38.790 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:38.804 7fa435367700 1 --2- 172.21.2.112:0/3283690970 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa4140186c0 0x7fa41401abc0 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:38.791 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:38.804 7fa435367700 1 --2- 172.21.2.112:0/3283690970 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa430094c40 0x7fa4301a75e0 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:38.791 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:38.804 7fa435367700 1 --2- 172.21.2.112:0/3283690970 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa430095040 0x7fa4301a7c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:38.791 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:38.804 7fa435367700 1 --2- 172.21.2.112:0/3283690970 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa4300a2580 0x7fa4300a1cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:38.791 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:38.804 7fa435367700 1 -- 172.21.2.112:0/3283690970 >> 172.21.2.112:0/3283690970 conn(0x7fa4300a2b30 msgr2=0x7fa43009ce60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:38.791 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:38.804 7fa435367700 1 -- 172.21.2.112:0/3283690970 shutdown_connections 2021-02-10T11:36:38.791 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:38.804 7fa435367700 1 -- 172.21.2.112:0/3283690970 wait complete. 2021-02-10T11:36:38.799 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:36:38.799 DEBUG:teuthology.orchestra.run.gibba012:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:36:39.144 DEBUG:teuthology.orchestra.run.gibba020:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:36:39.488 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:36:39.488 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:39.489 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:36:39.489 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:36:39.824 INFO:teuthology.orchestra.run.gibba012.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.0’: No such file or directory 2021-02-10T11:36:39.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:36:39.824 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2021-02-10T11:36:39.824 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:39.825 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:36:39.825 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:36:40.160 INFO:teuthology.orchestra.run.gibba012.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: No such file or directory 2021-02-10T11:36:40.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:36:40.160 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2021-02-10T11:36:40.161 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:40.161 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:36:40.161 DEBUG:teuthology.orchestra.run.gibba020:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:36:40.493 INFO:teuthology.orchestra.run.gibba020.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:36:40.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:36:40.494 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:36:40.494 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:40.494 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:36:40.494 DEBUG:teuthology.orchestra.run.gibba020:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:36:40.829 INFO:teuthology.orchestra.run.gibba020.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:36:40.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:36:40.830 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:36:40.830 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json-pretty 2021-02-10T11:36:41.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.255 7f8ebbbe3700 1 Processor -- start 2021-02-10T11:36:41.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.255 7f8ebbbe3700 1 -- start start 2021-02-10T11:36:41.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.256 7f8ebbbe3700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f8eb41b0470 0x7f8eb41b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:41.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.256 7f8ebbbe3700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8eb40f0b50 con 0x7f8eb40f20e0 2021-02-10T11:36:41.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.256 7f8ebbbe3700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8eb40f24e0 con 0x7f8eb40ef610 2021-02-10T11:36:41.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.256 7f8ebbbe3700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8eb40e5ea0 con 0x7f8eb41b0470 2021-02-10T11:36:41.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.256 7f8eb99ff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f8eb41b0470 0x7f8eb41b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:41.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.256 7f8eba200700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f8eb40f20e0 0x7f8eb40f0200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:51660/0 (socket says 172.21.2.112:51660) 2021-02-10T11:36:41.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.256 7f8eb99ff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f8eb41b0470 0x7f8eb41b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:52198/0 (socket says 172.21.2.112:52198) 2021-02-10T11:36:41.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.256 7f8eba200700 1 -- 172.21.2.112:0/1070025137 learned_addr learned my addr 172.21.2.112:0/1070025137 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:36:41.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.257 7f8eb99ff700 1 -- 172.21.2.112:0/1070025137 >> v1:172.21.2.120:6789/0 conn(0x7f8eb40ef610 legacy=0x7f8eb40efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:41.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.257 7f8eb99ff700 1 -- 172.21.2.112:0/1070025137 >> v1:172.21.2.112:6789/0 conn(0x7f8eb40f20e0 legacy=0x7f8eb40f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:41.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.257 7f8eb89fd700 1 -- 172.21.2.112:0/1070025137 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1789665927 0 0) 0x7f8eac001500 con 0x7f8eb40f20e0 2021-02-10T11:36:41.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.257 7f8eb99ff700 1 -- 172.21.2.112:0/1070025137 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8eb40e52e0 con 0x7f8eb41b0470 2021-02-10T11:36:41.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.257 7f8eb99ff700 1 --2- 172.21.2.112:0/1070025137 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8eb41b0470 0x7f8eb41b0870 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 rx=0x7f8eb0006650 tx=0x7f8eb0009a80).ready entity=mon.0 client_cookie=405d951d7e5c3a00 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:41.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.257 7f8eb89fd700 1 -- 172.21.2.112:0/1070025137 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8eb000c0c0 con 0x7f8eb41b0470 2021-02-10T11:36:41.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.257 7f8eb89fd700 1 -- 172.21.2.112:0/1070025137 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8eb000c340 con 0x7f8eb41b0470 2021-02-10T11:36:41.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.257 7f8eb89fd700 1 -- 172.21.2.112:0/1070025137 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8eb000c860 con 0x7f8eb41b0470 2021-02-10T11:36:41.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.257 7f8ebbbe3700 1 -- 172.21.2.112:0/1070025137 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8eb41b0470 msgr2=0x7f8eb41b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:41.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.257 7f8ebbbe3700 1 --2- 172.21.2.112:0/1070025137 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8eb41b0470 0x7f8eb41b0870 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 rx=0x7f8eb0006650 tx=0x7f8eb0009a80).stop 2021-02-10T11:36:41.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.258 7f8ebbbe3700 1 -- 172.21.2.112:0/1070025137 shutdown_connections 2021-02-10T11:36:41.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.258 7f8ebbbe3700 1 --2- 172.21.2.112:0/1070025137 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8eb41b0470 0x7f8eb41b0870 unknown :-1 s=CLOSED pgs=1646 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:41.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.258 7f8ebbbe3700 1 -- 172.21.2.112:0/1070025137 >> 172.21.2.112:0/1070025137 conn(0x7f8eb40ce3e0 msgr2=0x7f8eb40e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:41.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.258 7f8ebbbe3700 1 -- 172.21.2.112:0/1070025137 shutdown_connections 2021-02-10T11:36:41.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.258 7f8ebbbe3700 1 -- 172.21.2.112:0/1070025137 wait complete. 2021-02-10T11:36:41.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.273 7f8ebbbe3700 1 Processor -- start 2021-02-10T11:36:41.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.273 7f8ebbbe3700 1 -- start start 2021-02-10T11:36:41.262 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.273 7f8ebbbe3700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8eb4189270 0x7f8eb40a22d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:41.262 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.273 7f8ebbbe3700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8eb40ee5c0 0x7f8eb40ee9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:41.262 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.273 7f8ebbbe3700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8eb40b11b0 0x7f8eb40b15b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:41.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.273 7f8ebbbe3700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8eb41a8960 con 0x7f8eb40ee5c0 2021-02-10T11:36:41.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.273 7f8ebbbe3700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8eb40a28f0 con 0x7f8eb40b11b0 2021-02-10T11:36:41.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.273 7f8ebbbe3700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8eb40b1bd0 con 0x7f8eb4189270 2021-02-10T11:36:41.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.273 7f8eb91fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8eb40ee5c0 0x7f8eb40ee9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:41.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.273 7f8eb91fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8eb40ee5c0 0x7f8eb40ee9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:52202/0 (socket says 172.21.2.112:52202) 2021-02-10T11:36:41.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.273 7f8eb91fe700 1 -- 172.21.2.112:0/4007285324 learned_addr learned my addr 172.21.2.112:0/4007285324 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:36:41.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.274 7f8eba200700 1 --2- 172.21.2.112:0/4007285324 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8eb40b11b0 0x7f8eb40b15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:41.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.274 7f8eb91fe700 1 -- 172.21.2.112:0/4007285324 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8eb4189270 msgr2=0x7f8eb40a22d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:41.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.274 7f8eb99ff700 1 --2- 172.21.2.112:0/4007285324 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8eb4189270 0x7f8eb40a22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:41.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.274 7f8eb91fe700 1 --2- 172.21.2.112:0/4007285324 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8eb4189270 0x7f8eb40a22d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:41.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.274 7f8eb91fe700 1 -- 172.21.2.112:0/4007285324 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8eb40b11b0 msgr2=0x7f8eb40b15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:41.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.274 7f8eb91fe700 1 --2- 172.21.2.112:0/4007285324 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8eb40b11b0 0x7f8eb40b15b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:41.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.274 7f8eb91fe700 1 -- 172.21.2.112:0/4007285324 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8eb40b1df0 con 0x7f8eb40ee5c0 2021-02-10T11:36:41.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.274 7f8eba200700 1 --2- 172.21.2.112:0/4007285324 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8eb40b11b0 0x7f8eb40b15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:36:41.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.274 7f8eb99ff700 1 --2- 172.21.2.112:0/4007285324 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8eb4189270 0x7f8eb40a22d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:36:41.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.274 7f8eb91fe700 1 --2- 172.21.2.112:0/4007285324 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8eb40ee5c0 0x7f8eb40ee9c0 secure :-1 s=READY pgs=1647 cs=0 l=1 rev1=1 rx=0x7f8ea400b5b0 tx=0x7f8ea400b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:41.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.274 7f8eaa7fc700 1 -- 172.21.2.112:0/4007285324 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8ea400bd40 con 0x7f8eb40ee5c0 2021-02-10T11:36:41.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.274 7f8eaa7fc700 1 -- 172.21.2.112:0/4007285324 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ea4000ce0 con 0x7f8eb40ee5c0 2021-02-10T11:36:41.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.274 7f8eaa7fc700 1 -- 172.21.2.112:0/4007285324 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8ea40054a0 con 0x7f8eb40ee5c0 2021-02-10T11:36:41.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.274 7f8ebbbe3700 1 -- 172.21.2.112:0/4007285324 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8eb40b1df0 con 0x7f8eb40ee5c0 2021-02-10T11:36:41.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.275 7f8ebbbe3700 1 -- 172.21.2.112:0/4007285324 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8eb40b1df0 con 0x7f8eb40ee5c0 2021-02-10T11:36:41.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.275 7f8eaa7fc700 1 -- 172.21.2.112:0/4007285324 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8ea4006030 con 0x7f8eb40ee5c0 2021-02-10T11:36:41.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.275 7f8eaa7fc700 1 --2- 172.21.2.112:0/4007285324 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8e9c018730 0x7f8e9c01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:41.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.275 7f8eaa7fc700 1 -- 172.21.2.112:0/4007285324 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5645+0+0 (secure 0 0 0) 0x7f8ea4032880 con 0x7f8eb40ee5c0 2021-02-10T11:36:41.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.276 7f8eb99ff700 1 --2- 172.21.2.112:0/4007285324 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8e9c018730 0x7f8e9c01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:41.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.276 7f8eb99ff700 1 --2- 172.21.2.112:0/4007285324 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8e9c018730 0x7f8e9c01ac10 crc :-1 s=READY pgs=787 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:41.292 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.306 7f8ebbbe3700 1 -- 172.21.2.112:0/4007285324 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8eb40edac0 con 0x7f8eb40ee5c0 2021-02-10T11:36:41.295 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.308 7f8eaa7fc700 1 -- 172.21.2.112:0/4007285324 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8ea400bd40 con 0x7f8eb40ee5c0 2021-02-10T11:36:41.419 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.433 7f8ebbbe3700 1 -- 172.21.2.112:0/4007285324 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f8eb40eddb0 con 0x7f8eb40ee5c0 2021-02-10T11:36:41.420 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.434 7f8eaa7fc700 1 -- 172.21.2.112:0/4007285324 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v75) v1 ==== 81+0+30574 (secure 0 0 0) 0x7f8ea4000ce0 con 0x7f8eb40ee5c0 2021-02-10T11:36:41.421 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:36:41.421 INFO:teuthology.orchestra.run.gibba012.stdout:{ 2021-02-10T11:36:41.421 INFO:teuthology.orchestra.run.gibba012.stdout: "epoch": 75, 2021-02-10T11:36:41.421 INFO:teuthology.orchestra.run.gibba012.stdout: "fsid": "f14f004e-0699-4584-ad81-3754258493de", 2021-02-10T11:36:41.421 INFO:teuthology.orchestra.run.gibba012.stdout: "created": "2021-02-10 11:13:26.501424", 2021-02-10T11:36:41.421 INFO:teuthology.orchestra.run.gibba012.stdout: "modified": "2021-02-10 11:36:41.427653", 2021-02-10T11:36:41.421 INFO:teuthology.orchestra.run.gibba012.stdout: "last_up_change": "2021-02-10 11:13:52.780316", 2021-02-10T11:36:41.421 INFO:teuthology.orchestra.run.gibba012.stdout: "last_in_change": "2021-02-10 11:13:52.780316", 2021-02-10T11:36:41.421 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-02-10T11:36:41.422 INFO:teuthology.orchestra.run.gibba012.stdout: "flags_num": 5799936, 2021-02-10T11:36:41.422 INFO:teuthology.orchestra.run.gibba012.stdout: "flags_set": [ 2021-02-10T11:36:41.422 INFO:teuthology.orchestra.run.gibba012.stdout: "pglog_hardlimit", 2021-02-10T11:36:41.422 INFO:teuthology.orchestra.run.gibba012.stdout: "purged_snapdirs", 2021-02-10T11:36:41.422 INFO:teuthology.orchestra.run.gibba012.stdout: "recovery_deletes", 2021-02-10T11:36:41.422 INFO:teuthology.orchestra.run.gibba012.stdout: "sortbitwise" 2021-02-10T11:36:41.422 INFO:teuthology.orchestra.run.gibba012.stdout: ], 2021-02-10T11:36:41.422 INFO:teuthology.orchestra.run.gibba012.stdout: "crush_version": 4, 2021-02-10T11:36:41.422 INFO:teuthology.orchestra.run.gibba012.stdout: "full_ratio": 0.94999998807907104, 2021-02-10T11:36:41.422 INFO:teuthology.orchestra.run.gibba012.stdout: "backfillfull_ratio": 0.89999997615814209, 2021-02-10T11:36:41.422 INFO:teuthology.orchestra.run.gibba012.stdout: "nearfull_ratio": 0.85000002384185791, 2021-02-10T11:36:41.423 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_snapshot": "", 2021-02-10T11:36:41.423 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_max": 15, 2021-02-10T11:36:41.423 INFO:teuthology.orchestra.run.gibba012.stdout: "max_osd": 8, 2021-02-10T11:36:41.423 INFO:teuthology.orchestra.run.gibba012.stdout: "require_min_compat_client": "jewel", 2021-02-10T11:36:41.423 INFO:teuthology.orchestra.run.gibba012.stdout: "min_compat_client": "jewel", 2021-02-10T11:36:41.423 INFO:teuthology.orchestra.run.gibba012.stdout: "require_osd_release": "nautilus", 2021-02-10T11:36:41.423 INFO:teuthology.orchestra.run.gibba012.stdout: "pools": [ 2021-02-10T11:36:41.423 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:36:41.423 INFO:teuthology.orchestra.run.gibba012.stdout: "pool": 1, 2021-02-10T11:36:41.423 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_name": "rbd", 2021-02-10T11:36:41.424 INFO:teuthology.orchestra.run.gibba012.stdout: "create_time": "2021-02-10 11:13:57.508758", 2021-02-10T11:36:41.424 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:36:41.424 INFO:teuthology.orchestra.run.gibba012.stdout: "flags_names": "hashpspool", 2021-02-10T11:36:41.424 INFO:teuthology.orchestra.run.gibba012.stdout: "type": 1, 2021-02-10T11:36:41.424 INFO:teuthology.orchestra.run.gibba012.stdout: "size": 2, 2021-02-10T11:36:41.424 INFO:teuthology.orchestra.run.gibba012.stdout: "min_size": 1, 2021-02-10T11:36:41.424 INFO:teuthology.orchestra.run.gibba012.stdout: "crush_rule": 0, 2021-02-10T11:36:41.424 INFO:teuthology.orchestra.run.gibba012.stdout: "object_hash": 2, 2021-02-10T11:36:41.424 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:36:41.424 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num": 8, 2021-02-10T11:36:41.424 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num": 8, 2021-02-10T11:36:41.424 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num_target": 8, 2021-02-10T11:36:41.424 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_target": 8, 2021-02-10T11:36:41.425 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_pending": 8, 2021-02-10T11:36:41.425 INFO:teuthology.orchestra.run.gibba012.stdout: "last_pg_merge_meta": { 2021-02-10T11:36:41.425 INFO:teuthology.orchestra.run.gibba012.stdout: "source_pgid": "0.0", 2021-02-10T11:36:41.425 INFO:teuthology.orchestra.run.gibba012.stdout: "ready_epoch": 0, 2021-02-10T11:36:41.425 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_started": 0, 2021-02-10T11:36:41.425 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_clean": 0, 2021-02-10T11:36:41.425 INFO:teuthology.orchestra.run.gibba012.stdout: "source_version": "0'0", 2021-02-10T11:36:41.425 INFO:teuthology.orchestra.run.gibba012.stdout: "target_version": "0'0" 2021-02-10T11:36:41.425 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:36:41.425 INFO:teuthology.orchestra.run.gibba012.stdout: "last_change": "16", 2021-02-10T11:36:41.425 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend": "0", 2021-02-10T11:36:41.425 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:36:41.425 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:36:41.426 INFO:teuthology.orchestra.run.gibba012.stdout: "auid": 0, 2021-02-10T11:36:41.426 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:36:41.426 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_seq": 0, 2021-02-10T11:36:41.426 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_epoch": 0, 2021-02-10T11:36:41.426 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_snaps": [], 2021-02-10T11:36:41.426 INFO:teuthology.orchestra.run.gibba012.stdout: "removed_snaps": "[]", 2021-02-10T11:36:41.426 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_bytes": 0, 2021-02-10T11:36:41.426 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_objects": 0, 2021-02-10T11:36:41.426 INFO:teuthology.orchestra.run.gibba012.stdout: "tiers": [], 2021-02-10T11:36:41.426 INFO:teuthology.orchestra.run.gibba012.stdout: "tier_of": -1, 2021-02-10T11:36:41.426 INFO:teuthology.orchestra.run.gibba012.stdout: "read_tier": -1, 2021-02-10T11:36:41.427 INFO:teuthology.orchestra.run.gibba012.stdout: "write_tier": -1, 2021-02-10T11:36:41.427 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_mode": "none", 2021-02-10T11:36:41.427 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_bytes": 0, 2021-02-10T11:36:41.427 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_objects": 0, 2021-02-10T11:36:41.427 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:36:41.427 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:36:41.427 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:36:41.427 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_flush_age": 0, 2021-02-10T11:36:41.428 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_evict_age": 0, 2021-02-10T11:36:41.428 INFO:teuthology.orchestra.run.gibba012.stdout: "erasure_code_profile": "", 2021-02-10T11:36:41.428 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_params": { 2021-02-10T11:36:41.428 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "none" 2021-02-10T11:36:41.428 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:36:41.428 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_period": 0, 2021-02-10T11:36:41.428 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_count": 0, 2021-02-10T11:36:41.428 INFO:teuthology.orchestra.run.gibba012.stdout: "use_gmt_hitset": true, 2021-02-10T11:36:41.429 INFO:teuthology.orchestra.run.gibba012.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:36:41.429 INFO:teuthology.orchestra.run.gibba012.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:36:41.429 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:36:41.429 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:36:41.429 INFO:teuthology.orchestra.run.gibba012.stdout: "grade_table": [], 2021-02-10T11:36:41.429 INFO:teuthology.orchestra.run.gibba012.stdout: "stripe_width": 0, 2021-02-10T11:36:41.429 INFO:teuthology.orchestra.run.gibba012.stdout: "expected_num_objects": 0, 2021-02-10T11:36:41.429 INFO:teuthology.orchestra.run.gibba012.stdout: "fast_read": false, 2021-02-10T11:36:41.429 INFO:teuthology.orchestra.run.gibba012.stdout: "options": {}, 2021-02-10T11:36:41.429 INFO:teuthology.orchestra.run.gibba012.stdout: "application_metadata": { 2021-02-10T11:36:41.429 INFO:teuthology.orchestra.run.gibba012.stdout: "rbd": {} 2021-02-10T11:36:41.429 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:36:41.429 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:36:41.429 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:36:41.430 INFO:teuthology.orchestra.run.gibba012.stdout: "pool": 14, 2021-02-10T11:36:41.430 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_name": "cephfs_metadata", 2021-02-10T11:36:41.430 INFO:teuthology.orchestra.run.gibba012.stdout: "create_time": "2021-02-10 11:35:58.263396", 2021-02-10T11:36:41.430 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:36:41.430 INFO:teuthology.orchestra.run.gibba012.stdout: "flags_names": "hashpspool", 2021-02-10T11:36:41.430 INFO:teuthology.orchestra.run.gibba012.stdout: "type": 1, 2021-02-10T11:36:41.430 INFO:teuthology.orchestra.run.gibba012.stdout: "size": 2, 2021-02-10T11:36:41.430 INFO:teuthology.orchestra.run.gibba012.stdout: "min_size": 1, 2021-02-10T11:36:41.430 INFO:teuthology.orchestra.run.gibba012.stdout: "crush_rule": 0, 2021-02-10T11:36:41.430 INFO:teuthology.orchestra.run.gibba012.stdout: "object_hash": 2, 2021-02-10T11:36:41.430 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:36:41.430 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num": 8, 2021-02-10T11:36:41.430 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num": 8, 2021-02-10T11:36:41.431 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num_target": 8, 2021-02-10T11:36:41.431 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_target": 8, 2021-02-10T11:36:41.431 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_pending": 8, 2021-02-10T11:36:41.431 INFO:teuthology.orchestra.run.gibba012.stdout: "last_pg_merge_meta": { 2021-02-10T11:36:41.431 INFO:teuthology.orchestra.run.gibba012.stdout: "source_pgid": "0.0", 2021-02-10T11:36:41.431 INFO:teuthology.orchestra.run.gibba012.stdout: "ready_epoch": 0, 2021-02-10T11:36:41.431 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_started": 0, 2021-02-10T11:36:41.431 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_clean": 0, 2021-02-10T11:36:41.431 INFO:teuthology.orchestra.run.gibba012.stdout: "source_version": "0'0", 2021-02-10T11:36:41.431 INFO:teuthology.orchestra.run.gibba012.stdout: "target_version": "0'0" 2021-02-10T11:36:41.431 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:36:41.431 INFO:teuthology.orchestra.run.gibba012.stdout: "last_change": "72", 2021-02-10T11:36:41.431 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend": "0", 2021-02-10T11:36:41.432 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:36:41.432 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:36:41.432 INFO:teuthology.orchestra.run.gibba012.stdout: "auid": 0, 2021-02-10T11:36:41.432 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:36:41.432 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_seq": 0, 2021-02-10T11:36:41.432 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_epoch": 0, 2021-02-10T11:36:41.432 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_snaps": [], 2021-02-10T11:36:41.432 INFO:teuthology.orchestra.run.gibba012.stdout: "removed_snaps": "[]", 2021-02-10T11:36:41.432 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_bytes": 0, 2021-02-10T11:36:41.432 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_objects": 0, 2021-02-10T11:36:41.432 INFO:teuthology.orchestra.run.gibba012.stdout: "tiers": [], 2021-02-10T11:36:41.432 INFO:teuthology.orchestra.run.gibba012.stdout: "tier_of": -1, 2021-02-10T11:36:41.432 INFO:teuthology.orchestra.run.gibba012.stdout: "read_tier": -1, 2021-02-10T11:36:41.433 INFO:teuthology.orchestra.run.gibba012.stdout: "write_tier": -1, 2021-02-10T11:36:41.433 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_mode": "none", 2021-02-10T11:36:41.433 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_bytes": 0, 2021-02-10T11:36:41.433 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_objects": 0, 2021-02-10T11:36:41.433 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:36:41.433 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:36:41.433 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:36:41.433 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_flush_age": 0, 2021-02-10T11:36:41.433 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_evict_age": 0, 2021-02-10T11:36:41.433 INFO:teuthology.orchestra.run.gibba012.stdout: "erasure_code_profile": "", 2021-02-10T11:36:41.433 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_params": { 2021-02-10T11:36:41.433 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "none" 2021-02-10T11:36:41.433 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:36:41.433 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_period": 0, 2021-02-10T11:36:41.434 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_count": 0, 2021-02-10T11:36:41.434 INFO:teuthology.orchestra.run.gibba012.stdout: "use_gmt_hitset": true, 2021-02-10T11:36:41.434 INFO:teuthology.orchestra.run.gibba012.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:36:41.434 INFO:teuthology.orchestra.run.gibba012.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:36:41.434 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:36:41.434 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:36:41.434 INFO:teuthology.orchestra.run.gibba012.stdout: "grade_table": [], 2021-02-10T11:36:41.434 INFO:teuthology.orchestra.run.gibba012.stdout: "stripe_width": 0, 2021-02-10T11:36:41.434 INFO:teuthology.orchestra.run.gibba012.stdout: "expected_num_objects": 0, 2021-02-10T11:36:41.434 INFO:teuthology.orchestra.run.gibba012.stdout: "fast_read": false, 2021-02-10T11:36:41.434 INFO:teuthology.orchestra.run.gibba012.stdout: "options": { 2021-02-10T11:36:41.434 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_autoscale_bias": 4, 2021-02-10T11:36:41.435 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_min": 16, 2021-02-10T11:36:41.435 INFO:teuthology.orchestra.run.gibba012.stdout: "recovery_priority": 5 2021-02-10T11:36:41.435 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:36:41.435 INFO:teuthology.orchestra.run.gibba012.stdout: "application_metadata": { 2021-02-10T11:36:41.435 INFO:teuthology.orchestra.run.gibba012.stdout: "cephfs": { 2021-02-10T11:36:41.435 INFO:teuthology.orchestra.run.gibba012.stdout: "metadata": "cephfs" 2021-02-10T11:36:41.435 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:36:41.435 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:36:41.435 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:36:41.435 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:36:41.435 INFO:teuthology.orchestra.run.gibba012.stdout: "pool": 15, 2021-02-10T11:36:41.435 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_name": "cephfs_data", 2021-02-10T11:36:41.435 INFO:teuthology.orchestra.run.gibba012.stdout: "create_time": "2021-02-10 11:35:59.285157", 2021-02-10T11:36:41.435 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 8193, 2021-02-10T11:36:41.436 INFO:teuthology.orchestra.run.gibba012.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2021-02-10T11:36:41.436 INFO:teuthology.orchestra.run.gibba012.stdout: "type": 1, 2021-02-10T11:36:41.436 INFO:teuthology.orchestra.run.gibba012.stdout: "size": 2, 2021-02-10T11:36:41.436 INFO:teuthology.orchestra.run.gibba012.stdout: "min_size": 1, 2021-02-10T11:36:41.436 INFO:teuthology.orchestra.run.gibba012.stdout: "crush_rule": 0, 2021-02-10T11:36:41.436 INFO:teuthology.orchestra.run.gibba012.stdout: "object_hash": 2, 2021-02-10T11:36:41.436 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:36:41.436 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num": 8, 2021-02-10T11:36:41.436 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num": 8, 2021-02-10T11:36:41.436 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num_target": 8, 2021-02-10T11:36:41.436 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_target": 8, 2021-02-10T11:36:41.436 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_pending": 8, 2021-02-10T11:36:41.436 INFO:teuthology.orchestra.run.gibba012.stdout: "last_pg_merge_meta": { 2021-02-10T11:36:41.437 INFO:teuthology.orchestra.run.gibba012.stdout: "source_pgid": "0.0", 2021-02-10T11:36:41.437 INFO:teuthology.orchestra.run.gibba012.stdout: "ready_epoch": 0, 2021-02-10T11:36:41.437 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_started": 0, 2021-02-10T11:36:41.437 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_clean": 0, 2021-02-10T11:36:41.437 INFO:teuthology.orchestra.run.gibba012.stdout: "source_version": "0'0", 2021-02-10T11:36:41.437 INFO:teuthology.orchestra.run.gibba012.stdout: "target_version": "0'0" 2021-02-10T11:36:41.437 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:36:41.437 INFO:teuthology.orchestra.run.gibba012.stdout: "last_change": "74", 2021-02-10T11:36:41.437 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend": "0", 2021-02-10T11:36:41.437 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:36:41.437 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:36:41.437 INFO:teuthology.orchestra.run.gibba012.stdout: "auid": 0, 2021-02-10T11:36:41.437 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:36:41.437 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_seq": 3, 2021-02-10T11:36:41.438 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_epoch": 74, 2021-02-10T11:36:41.438 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_snaps": [], 2021-02-10T11:36:41.438 INFO:teuthology.orchestra.run.gibba012.stdout: "removed_snaps": "[2~2]", 2021-02-10T11:36:41.438 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_bytes": 0, 2021-02-10T11:36:41.438 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_objects": 0, 2021-02-10T11:36:41.438 INFO:teuthology.orchestra.run.gibba012.stdout: "tiers": [], 2021-02-10T11:36:41.438 INFO:teuthology.orchestra.run.gibba012.stdout: "tier_of": -1, 2021-02-10T11:36:41.438 INFO:teuthology.orchestra.run.gibba012.stdout: "read_tier": -1, 2021-02-10T11:36:41.438 INFO:teuthology.orchestra.run.gibba012.stdout: "write_tier": -1, 2021-02-10T11:36:41.438 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_mode": "none", 2021-02-10T11:36:41.438 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_bytes": 0, 2021-02-10T11:36:41.438 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_objects": 0, 2021-02-10T11:36:41.439 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:36:41.439 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:36:41.439 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:36:41.439 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_flush_age": 0, 2021-02-10T11:36:41.439 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_evict_age": 0, 2021-02-10T11:36:41.439 INFO:teuthology.orchestra.run.gibba012.stdout: "erasure_code_profile": "", 2021-02-10T11:36:41.439 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_params": { 2021-02-10T11:36:41.439 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "none" 2021-02-10T11:36:41.439 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:36:41.439 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_period": 0, 2021-02-10T11:36:41.439 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_count": 0, 2021-02-10T11:36:41.439 INFO:teuthology.orchestra.run.gibba012.stdout: "use_gmt_hitset": true, 2021-02-10T11:36:41.439 INFO:teuthology.orchestra.run.gibba012.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:36:41.440 INFO:teuthology.orchestra.run.gibba012.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:36:41.440 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:36:41.440 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:36:41.440 INFO:teuthology.orchestra.run.gibba012.stdout: "grade_table": [], 2021-02-10T11:36:41.440 INFO:teuthology.orchestra.run.gibba012.stdout: "stripe_width": 0, 2021-02-10T11:36:41.440 INFO:teuthology.orchestra.run.gibba012.stdout: "expected_num_objects": 0, 2021-02-10T11:36:41.440 INFO:teuthology.orchestra.run.gibba012.stdout: "fast_read": false, 2021-02-10T11:36:41.440 INFO:teuthology.orchestra.run.gibba012.stdout: "options": {}, 2021-02-10T11:36:41.440 INFO:teuthology.orchestra.run.gibba012.stdout: "application_metadata": { 2021-02-10T11:36:41.440 INFO:teuthology.orchestra.run.gibba012.stdout: "cephfs": { 2021-02-10T11:36:41.440 INFO:teuthology.orchestra.run.gibba012.stdout: "data": "cephfs" 2021-02-10T11:36:41.440 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:36:41.440 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:36:41.441 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:36:41.441 INFO:teuthology.orchestra.run.gibba012.stdout: ], 2021-02-10T11:36:41.441 INFO:teuthology.orchestra.run.gibba012.stdout: "osds": [ 2021-02-10T11:36:41.441 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:36:41.441 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 0, 2021-02-10T11:36:41.441 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "95444b8f-6370-4a1d-ae5f-df62651b77da", 2021-02-10T11:36:41.441 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:36:41.441 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:36:41.441 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:36:41.441 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:36:41.441 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:36:41.441 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:36:41.441 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:36:41.442 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 71, 2021-02-10T11:36:41.442 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:36:41.442 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:36:41.442 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:36:41.442 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:36:41.442 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:36:41.442 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:36:41.442 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6817", 2021-02-10T11:36:41.442 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:36:41.442 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:36:41.442 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:36:41.442 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:36:41.442 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6818", 2021-02-10T11:36:41.443 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:36:41.443 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:36:41.443 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:36:41.443 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:36:41.443 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:36:41.443 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:36:41.443 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:36:41.443 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:36:41.443 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6820", 2021-02-10T11:36:41.443 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:36:41.443 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:36:41.443 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:36:41.443 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:36:41.444 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6822", 2021-02-10T11:36:41.444 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:36:41.444 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:36:41.444 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:36:41.444 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:36:41.444 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:36:41.444 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:36:41.444 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:36:41.444 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:36:41.444 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6828", 2021-02-10T11:36:41.444 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:36:41.444 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:36:41.444 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:36:41.445 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:36:41.445 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6830", 2021-02-10T11:36:41.445 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:36:41.445 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:36:41.445 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:36:41.445 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:36:41.445 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:36:41.445 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:36:41.445 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:36:41.445 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:36:41.445 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6824", 2021-02-10T11:36:41.445 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:36:41.445 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:36:41.446 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:36:41.446 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:36:41.446 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6826", 2021-02-10T11:36:41.446 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:36:41.446 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:36:41.446 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:36:41.446 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:36:41.446 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.112:6818/36030", 2021-02-10T11:36:41.446 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.112:6822/36030", 2021-02-10T11:36:41.446 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.112:6830/36030", 2021-02-10T11:36:41.446 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.112:6826/36030", 2021-02-10T11:36:41.446 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:36:41.446 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:36:41.447 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:36:41.447 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:36:41.447 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:36:41.447 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:36:41.447 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 1, 2021-02-10T11:36:41.447 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "a23e217e-0d8e-486e-be07-712adfabd262", 2021-02-10T11:36:41.447 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:36:41.447 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:36:41.447 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:36:41.447 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:36:41.447 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:36:41.448 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:36:41.448 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:36:41.448 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 71, 2021-02-10T11:36:41.448 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:36:41.448 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:36:41.448 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:36:41.448 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:36:41.448 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:36:41.448 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:36:41.448 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6800", 2021-02-10T11:36:41.449 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:36:41.449 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:36:41.449 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:36:41.449 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:36:41.449 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6802", 2021-02-10T11:36:41.449 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:36:41.449 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:36:41.449 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:36:41.449 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:36:41.449 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:36:41.449 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:36:41.449 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:36:41.450 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:36:41.450 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6804", 2021-02-10T11:36:41.450 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:36:41.450 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:36:41.450 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:36:41.450 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:36:41.450 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6806", 2021-02-10T11:36:41.450 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:36:41.450 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:36:41.450 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:36:41.450 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:36:41.450 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:36:41.450 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:36:41.451 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:36:41.451 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:36:41.451 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6812", 2021-02-10T11:36:41.451 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:36:41.451 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:36:41.451 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:36:41.451 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:36:41.451 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6814", 2021-02-10T11:36:41.451 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:36:41.451 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:36:41.451 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:36:41.451 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:36:41.451 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:36:41.452 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:36:41.452 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:36:41.452 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:36:41.452 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6808", 2021-02-10T11:36:41.452 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:36:41.452 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:36:41.452 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:36:41.452 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:36:41.452 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6810", 2021-02-10T11:36:41.452 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:36:41.452 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:36:41.452 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:36:41.452 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:36:41.453 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.112:6802/36029", 2021-02-10T11:36:41.453 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.112:6806/36029", 2021-02-10T11:36:41.453 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.112:6814/36029", 2021-02-10T11:36:41.453 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.112:6810/36029", 2021-02-10T11:36:41.453 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:36:41.453 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:36:41.453 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:36:41.454 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:36:41.454 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:36:41.454 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:36:41.454 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 2, 2021-02-10T11:36:41.454 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "026afa8a-c0f1-4b35-9a76-53c00fe0c2fc", 2021-02-10T11:36:41.454 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:36:41.454 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:36:41.454 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:36:41.454 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:36:41.454 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:36:41.455 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:36:41.455 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:36:41.455 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 71, 2021-02-10T11:36:41.455 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:36:41.455 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:36:41.455 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:36:41.455 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:36:41.456 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:36:41.456 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:36:41.456 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6801", 2021-02-10T11:36:41.456 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:36:41.456 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:36:41.456 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:36:41.456 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:36:41.457 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6803", 2021-02-10T11:36:41.457 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:36:41.457 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:36:41.457 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:36:41.457 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:36:41.457 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:36:41.457 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:36:41.457 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:36:41.457 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:36:41.457 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6805", 2021-02-10T11:36:41.457 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:36:41.457 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:36:41.457 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:36:41.458 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:36:41.458 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6807", 2021-02-10T11:36:41.458 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:36:41.458 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:36:41.458 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:36:41.458 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:36:41.458 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:36:41.458 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:36:41.458 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:36:41.458 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:36:41.458 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6813", 2021-02-10T11:36:41.458 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:36:41.458 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:36:41.459 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:36:41.459 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:36:41.459 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6815", 2021-02-10T11:36:41.459 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:36:41.459 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:36:41.459 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:36:41.459 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:36:41.459 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:36:41.459 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:36:41.460 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:36:41.460 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:36:41.460 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6809", 2021-02-10T11:36:41.460 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:36:41.460 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:36:41.460 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:36:41.460 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:36:41.460 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6811", 2021-02-10T11:36:41.460 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:36:41.460 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:36:41.460 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:36:41.461 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:36:41.461 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.112:6803/36031", 2021-02-10T11:36:41.461 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.112:6807/36031", 2021-02-10T11:36:41.461 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.112:6815/36031", 2021-02-10T11:36:41.461 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.112:6811/36031", 2021-02-10T11:36:41.461 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:36:41.461 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:36:41.461 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:36:41.461 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:36:41.461 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:36:41.461 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:36:41.461 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 3, 2021-02-10T11:36:41.461 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "58795682-e00d-4173-bb84-f7075ca89a11", 2021-02-10T11:36:41.462 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:36:41.462 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:36:41.462 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:36:41.462 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:36:41.462 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:36:41.462 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:36:41.462 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:36:41.462 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 69, 2021-02-10T11:36:41.462 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:36:41.463 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:36:41.463 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:36:41.463 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:36:41.463 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:36:41.463 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:36:41.463 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6816", 2021-02-10T11:36:41.463 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:36:41.464 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:36:41.464 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:36:41.464 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:36:41.464 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6819", 2021-02-10T11:36:41.464 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:36:41.464 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:36:41.464 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:36:41.465 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:36:41.465 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:36:41.465 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:36:41.465 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:36:41.465 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:36:41.465 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6821", 2021-02-10T11:36:41.465 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:36:41.465 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:36:41.465 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:36:41.465 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:36:41.465 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6823", 2021-02-10T11:36:41.465 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:36:41.466 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:36:41.466 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:36:41.466 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:36:41.466 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:36:41.466 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:36:41.466 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:36:41.466 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:36:41.466 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6829", 2021-02-10T11:36:41.466 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:36:41.466 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:36:41.466 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:36:41.466 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:36:41.466 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6831", 2021-02-10T11:36:41.467 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:36:41.467 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:36:41.467 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:36:41.467 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:36:41.467 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:36:41.467 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:36:41.467 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:36:41.467 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:36:41.467 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6825", 2021-02-10T11:36:41.467 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:36:41.467 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:36:41.467 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:36:41.467 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:36:41.468 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6827", 2021-02-10T11:36:41.468 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:36:41.468 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:36:41.468 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:36:41.468 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:36:41.468 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.112:6819/36028", 2021-02-10T11:36:41.468 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.112:6823/36028", 2021-02-10T11:36:41.468 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.112:6831/36028", 2021-02-10T11:36:41.468 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.112:6827/36028", 2021-02-10T11:36:41.468 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:36:41.468 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:36:41.468 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:36:41.468 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:36:41.469 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:36:41.469 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:36:41.469 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 4, 2021-02-10T11:36:41.469 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "6763f8d6-53dc-4e45-bdf6-a6eabcf81e39", 2021-02-10T11:36:41.469 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:36:41.469 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:36:41.469 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:36:41.469 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:36:41.469 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:36:41.469 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:36:41.469 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:36:41.469 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 71, 2021-02-10T11:36:41.469 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:36:41.470 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:36:41.470 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:36:41.470 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:36:41.470 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:36:41.470 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:36:41.470 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6810", 2021-02-10T11:36:41.470 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:36:41.470 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:36:41.470 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:36:41.470 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:36:41.470 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6813", 2021-02-10T11:36:41.470 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:36:41.471 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:36:41.471 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:36:41.471 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:36:41.471 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:36:41.471 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:36:41.471 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:36:41.471 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:36:41.471 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6816", 2021-02-10T11:36:41.471 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:36:41.471 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:36:41.471 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:36:41.471 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:36:41.471 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6819", 2021-02-10T11:36:41.472 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:36:41.472 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:36:41.472 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:36:41.472 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:36:41.472 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:36:41.472 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:36:41.472 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:36:41.472 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:36:41.472 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6825", 2021-02-10T11:36:41.472 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:36:41.472 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:36:41.472 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:36:41.472 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:36:41.473 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6827", 2021-02-10T11:36:41.473 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:36:41.473 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:36:41.473 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:36:41.473 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:36:41.473 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:36:41.473 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:36:41.473 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:36:41.473 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:36:41.473 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6821", 2021-02-10T11:36:41.473 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:36:41.474 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:36:41.474 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:36:41.474 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:36:41.474 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6823", 2021-02-10T11:36:41.474 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:36:41.474 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:36:41.474 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:36:41.474 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:36:41.474 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.120:6813/36388", 2021-02-10T11:36:41.474 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.120:6819/36388", 2021-02-10T11:36:41.474 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.120:6827/36388", 2021-02-10T11:36:41.475 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.120:6823/36388", 2021-02-10T11:36:41.475 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:36:41.475 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:36:41.475 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:36:41.475 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:36:41.475 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:36:41.475 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:36:41.475 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 5, 2021-02-10T11:36:41.475 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "5d0e4dd6-0cb0-4bf8-8314-626acaa1b544", 2021-02-10T11:36:41.475 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:36:41.475 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:36:41.476 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:36:41.476 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:36:41.476 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:36:41.476 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:36:41.476 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:36:41.476 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 69, 2021-02-10T11:36:41.476 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:36:41.476 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:36:41.476 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:36:41.476 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:36:41.476 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:36:41.476 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:36:41.477 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6804", 2021-02-10T11:36:41.477 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:36:41.477 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:36:41.477 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:36:41.477 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:36:41.477 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6806", 2021-02-10T11:36:41.477 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:36:41.477 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:36:41.477 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:36:41.477 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:36:41.477 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:36:41.478 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:36:41.478 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:36:41.478 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:36:41.478 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6809", 2021-02-10T11:36:41.478 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:36:41.478 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:36:41.478 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:36:41.478 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:36:41.478 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6812", 2021-02-10T11:36:41.478 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:36:41.478 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:36:41.478 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:36:41.478 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:36:41.479 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:36:41.479 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:36:41.479 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:36:41.479 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:36:41.479 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6820", 2021-02-10T11:36:41.479 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:36:41.479 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:36:41.479 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:36:41.479 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:36:41.479 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6822", 2021-02-10T11:36:41.479 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:36:41.479 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:36:41.479 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:36:41.480 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:36:41.480 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:36:41.480 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:36:41.480 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:36:41.480 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:36:41.480 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6815", 2021-02-10T11:36:41.480 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:36:41.480 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:36:41.480 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:36:41.480 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:36:41.480 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6818", 2021-02-10T11:36:41.480 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:36:41.480 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:36:41.480 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:36:41.481 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:36:41.481 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.120:6806/36387", 2021-02-10T11:36:41.481 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.120:6812/36387", 2021-02-10T11:36:41.481 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.120:6822/36387", 2021-02-10T11:36:41.481 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.120:6818/36387", 2021-02-10T11:36:41.481 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:36:41.481 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:36:41.481 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:36:41.481 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:36:41.481 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:36:41.481 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:36:41.481 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 6, 2021-02-10T11:36:41.481 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "b010e187-4541-4b00-80dd-f948f6ab8f15", 2021-02-10T11:36:41.482 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:36:41.482 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:36:41.482 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:36:41.482 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:36:41.482 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:36:41.482 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:36:41.482 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:36:41.482 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 71, 2021-02-10T11:36:41.482 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:36:41.482 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:36:41.482 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:36:41.482 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:36:41.482 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:36:41.483 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:36:41.483 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6824", 2021-02-10T11:36:41.483 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:36:41.483 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:36:41.483 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:36:41.483 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:36:41.483 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6826", 2021-02-10T11:36:41.484 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:36:41.484 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:36:41.484 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:36:41.484 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:36:41.484 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:36:41.484 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:36:41.484 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:36:41.484 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:36:41.484 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6828", 2021-02-10T11:36:41.484 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:36:41.484 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:36:41.484 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:36:41.484 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:36:41.485 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6829", 2021-02-10T11:36:41.485 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:36:41.485 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:36:41.485 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:36:41.485 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:36:41.485 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:36:41.485 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:36:41.485 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:36:41.485 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:36:41.485 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6832", 2021-02-10T11:36:41.485 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:36:41.485 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:36:41.486 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:36:41.486 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:36:41.486 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6833", 2021-02-10T11:36:41.486 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:36:41.486 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:36:41.486 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:36:41.486 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:36:41.486 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:36:41.486 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:36:41.487 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:36:41.487 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:36:41.487 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6830", 2021-02-10T11:36:41.487 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:36:41.487 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:36:41.487 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:36:41.487 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:36:41.488 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6831", 2021-02-10T11:36:41.488 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:36:41.488 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:36:41.488 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:36:41.488 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:36:41.488 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.120:6826/36385", 2021-02-10T11:36:41.488 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.120:6829/36385", 2021-02-10T11:36:41.488 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.120:6833/36385", 2021-02-10T11:36:41.488 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.120:6831/36385", 2021-02-10T11:36:41.488 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:36:41.488 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:36:41.488 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:36:41.489 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:36:41.489 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:36:41.489 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:36:41.489 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 7, 2021-02-10T11:36:41.489 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "671e2772-33b7-497e-8e44-2178bd850bcf", 2021-02-10T11:36:41.489 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:36:41.489 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:36:41.489 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:36:41.489 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:36:41.489 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:36:41.489 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:36:41.489 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:36:41.489 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 62, 2021-02-10T11:36:41.490 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:36:41.490 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:36:41.490 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:36:41.490 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:36:41.490 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:36:41.490 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:36:41.490 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6802", 2021-02-10T11:36:41.490 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:36:41.490 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:36:41.490 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:36:41.490 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:36:41.490 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6803", 2021-02-10T11:36:41.490 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:36:41.491 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:36:41.491 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:36:41.491 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:36:41.491 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:36:41.491 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:36:41.491 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:36:41.491 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:36:41.491 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6805", 2021-02-10T11:36:41.491 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:36:41.491 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:36:41.491 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:36:41.491 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:36:41.491 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6807", 2021-02-10T11:36:41.492 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:36:41.492 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:36:41.492 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:36:41.492 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:36:41.492 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:36:41.492 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:36:41.492 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:36:41.492 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:36:41.492 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6814", 2021-02-10T11:36:41.492 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:36:41.492 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:36:41.492 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:36:41.492 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:36:41.493 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6817", 2021-02-10T11:36:41.493 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:36:41.493 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:36:41.493 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:36:41.493 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:36:41.493 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:36:41.493 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:36:41.493 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:36:41.493 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:36:41.493 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6808", 2021-02-10T11:36:41.493 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:36:41.493 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:36:41.493 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:36:41.494 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:36:41.494 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6811", 2021-02-10T11:36:41.494 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:36:41.494 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:36:41.494 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:36:41.494 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:36:41.494 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.120:6803/36386", 2021-02-10T11:36:41.494 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.120:6807/36386", 2021-02-10T11:36:41.494 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.120:6817/36386", 2021-02-10T11:36:41.494 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.120:6811/36386", 2021-02-10T11:36:41.494 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:36:41.494 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:36:41.494 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:36:41.495 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:36:41.495 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:36:41.495 INFO:teuthology.orchestra.run.gibba012.stdout: ], 2021-02-10T11:36:41.495 INFO:teuthology.orchestra.run.gibba012.stdout: "osd_xinfo": [ 2021-02-10T11:36:41.495 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:36:41.495 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 0, 2021-02-10T11:36:41.495 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:36:41.495 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:36:41.495 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:36:41.495 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:36:41.495 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:36:41.495 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:36:41.495 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:36:41.496 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 1, 2021-02-10T11:36:41.496 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:36:41.496 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:36:41.496 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:36:41.496 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:36:41.496 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:36:41.496 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:36:41.496 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:36:41.496 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 2, 2021-02-10T11:36:41.496 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:36:41.496 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:36:41.496 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:36:41.496 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:36:41.497 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:36:41.497 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:36:41.497 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:36:41.497 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 3, 2021-02-10T11:36:41.497 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:36:41.497 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:36:41.497 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:36:41.497 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:36:41.497 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:36:41.497 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:36:41.497 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:36:41.497 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 4, 2021-02-10T11:36:41.497 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:36:41.498 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:36:41.498 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:36:41.498 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:36:41.498 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:36:41.498 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:36:41.498 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:36:41.498 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 5, 2021-02-10T11:36:41.499 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:36:41.499 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:36:41.499 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:36:41.499 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:36:41.499 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:36:41.499 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:36:41.499 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:36:41.499 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 6, 2021-02-10T11:36:41.500 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:36:41.500 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:36:41.500 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:36:41.500 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:36:41.500 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:36:41.500 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:36:41.500 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:36:41.500 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 7, 2021-02-10T11:36:41.500 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:36:41.500 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:36:41.501 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:36:41.501 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:36:41.501 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:36:41.501 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:36:41.501 INFO:teuthology.orchestra.run.gibba012.stdout: ], 2021-02-10T11:36:41.501 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_upmap": [], 2021-02-10T11:36:41.501 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_upmap_items": [], 2021-02-10T11:36:41.501 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_temp": [], 2021-02-10T11:36:41.501 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_temp": [], 2021-02-10T11:36:41.502 INFO:teuthology.orchestra.run.gibba012.stdout: "blacklist": {}, 2021-02-10T11:36:41.502 INFO:teuthology.orchestra.run.gibba012.stdout: "erasure_code_profiles": { 2021-02-10T11:36:41.502 INFO:teuthology.orchestra.run.gibba012.stdout: "default": { 2021-02-10T11:36:41.502 INFO:teuthology.orchestra.run.gibba012.stdout: "crush-failure-domain": "osd", 2021-02-10T11:36:41.502 INFO:teuthology.orchestra.run.gibba012.stdout: "k": "2", 2021-02-10T11:36:41.502 INFO:teuthology.orchestra.run.gibba012.stdout: "m": "1", 2021-02-10T11:36:41.502 INFO:teuthology.orchestra.run.gibba012.stdout: "plugin": "jerasure", 2021-02-10T11:36:41.502 INFO:teuthology.orchestra.run.gibba012.stdout: "ruleset-failure-domain": "osd", 2021-02-10T11:36:41.502 INFO:teuthology.orchestra.run.gibba012.stdout: "technique": "reed_sol_van" 2021-02-10T11:36:41.502 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:36:41.502 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:36:41.503 INFO:teuthology.orchestra.run.gibba012.stdout: "removed_snaps_queue": [], 2021-02-10T11:36:41.503 INFO:teuthology.orchestra.run.gibba012.stdout: "new_removed_snaps": [], 2021-02-10T11:36:41.503 INFO:teuthology.orchestra.run.gibba012.stdout: "new_purged_snaps": [ 2021-02-10T11:36:41.503 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:36:41.503 INFO:teuthology.orchestra.run.gibba012.stdout: "pool": 15, 2021-02-10T11:36:41.503 INFO:teuthology.orchestra.run.gibba012.stdout: "snaps": [ 2021-02-10T11:36:41.503 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:36:41.503 INFO:teuthology.orchestra.run.gibba012.stdout: "begin": 2, 2021-02-10T11:36:41.503 INFO:teuthology.orchestra.run.gibba012.stdout: "length": 2 2021-02-10T11:36:41.503 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:36:41.503 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:36:41.503 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:36:41.503 INFO:teuthology.orchestra.run.gibba012.stdout: ], 2021-02-10T11:36:41.504 INFO:teuthology.orchestra.run.gibba012.stdout: "crush_node_flags": {}, 2021-02-10T11:36:41.504 INFO:teuthology.orchestra.run.gibba012.stdout: "device_class_flags": {} 2021-02-10T11:36:41.504 INFO:teuthology.orchestra.run.gibba012.stdout:} 2021-02-10T11:36:41.505 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.436 7f8ebbbe3700 1 -- 172.21.2.112:0/4007285324 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8e9c018730 msgr2=0x7f8e9c01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:41.505 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.436 7f8ebbbe3700 1 --2- 172.21.2.112:0/4007285324 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8e9c018730 0x7f8e9c01ac10 crc :-1 s=READY pgs=787 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:41.505 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.436 7f8ebbbe3700 1 -- 172.21.2.112:0/4007285324 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8eb40ee5c0 msgr2=0x7f8eb40ee9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:41.505 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.436 7f8ebbbe3700 1 --2- 172.21.2.112:0/4007285324 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8eb40ee5c0 0x7f8eb40ee9c0 secure :-1 s=READY pgs=1647 cs=0 l=1 rev1=1 rx=0x7f8ea400b5b0 tx=0x7f8ea400b950).stop 2021-02-10T11:36:41.505 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.436 7f8ebbbe3700 1 -- 172.21.2.112:0/4007285324 shutdown_connections 2021-02-10T11:36:41.505 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.436 7f8ebbbe3700 1 --2- 172.21.2.112:0/4007285324 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8e9c018730 0x7f8e9c01ac10 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:41.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.436 7f8ebbbe3700 1 --2- 172.21.2.112:0/4007285324 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8eb40b11b0 0x7f8eb40b15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:41.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.436 7f8ebbbe3700 1 --2- 172.21.2.112:0/4007285324 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8eb40ee5c0 0x7f8eb40ee9c0 unknown :-1 s=CLOSED pgs=1647 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:41.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.436 7f8ebbbe3700 1 --2- 172.21.2.112:0/4007285324 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8eb4189270 0x7f8eb40a22d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:41.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.436 7f8ebbbe3700 1 -- 172.21.2.112:0/4007285324 >> 172.21.2.112:0/4007285324 conn(0x7f8eb418b220 msgr2=0x7f8eb41b1af0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:41.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.436 7f8ebbbe3700 1 -- 172.21.2.112:0/4007285324 shutdown_connections 2021-02-10T11:36:41.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.436 7f8ebbbe3700 1 -- 172.21.2.112:0/4007285324 wait complete. 2021-02-10T11:36:41.507 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:36:41.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.930 7f2d328fa700 1 Processor -- start 2021-02-10T11:36:41.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.930 7f2d328fa700 1 -- start start 2021-02-10T11:36:41.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.931 7f2d328fa700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f2d2c1b0870 0x7f2d2c0f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:41.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.931 7f2d328fa700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2d2c0efe00 0x7f2d2c0f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:41.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.931 7f2d328fa700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d2c1b0c70 con 0x7f2d2c1b0470 2021-02-10T11:36:41.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.931 7f2d328fa700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d2c0f0b50 con 0x7f2d2c1b0870 2021-02-10T11:36:41.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.931 7f2d328fa700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d2c0f0820 con 0x7f2d2c0efe00 2021-02-10T11:36:41.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.931 7f2d2b7fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f2d2c1b0870 0x7f2d2c0f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:41.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.931 7f2d2b7fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f2d2c1b0870 0x7f2d2c0f20e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:52212/0 (socket says 172.21.2.112:52212) 2021-02-10T11:36:41.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.931 7f2d2b7fe700 1 -- 172.21.2.112:0/492066775 learned_addr learned my addr 172.21.2.112:0/492066775 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:36:41.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.931 7f2d30f17700 1 --2- 172.21.2.112:0/492066775 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2d2c0efe00 0x7f2d2c0f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:41.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.931 7f2d2b7fe700 1 -- 172.21.2.112:0/492066775 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2d2c0efe00 msgr2=0x7f2d2c0f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:41.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.931 7f2d2b7fe700 1 --2- 172.21.2.112:0/492066775 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2d2c0efe00 0x7f2d2c0f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:41.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.931 7f2d2b7fe700 1 -- 172.21.2.112:0/492066775 >> v1:172.21.2.120:6789/0 conn(0x7f2d2c1b0470 legacy=0x7f2d2c0ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:41.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.931 7f2d2b7fe700 1 -- 172.21.2.112:0/492066775 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d2c0e52e0 con 0x7f2d2c1b0870 2021-02-10T11:36:41.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.932 7f2d2b7fe700 1 --2- 172.21.2.112:0/492066775 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2d2c1b0870 0x7f2d2c0f20e0 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 rx=0x7f2d20006700 tx=0x7f2d20009b80).ready entity=mon.0 client_cookie=4645fe28214b3e3e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:41.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.932 7f2d2affd700 1 -- 172.21.2.112:0/492066775 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2d2000c0c0 con 0x7f2d2c1b0870 2021-02-10T11:36:41.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.932 7f2d2affd700 1 -- 172.21.2.112:0/492066775 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d2000c440 con 0x7f2d2c1b0870 2021-02-10T11:36:41.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.932 7f2d2affd700 1 -- 172.21.2.112:0/492066775 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2d2000c920 con 0x7f2d2c1b0870 2021-02-10T11:36:41.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.932 7f2d328fa700 1 -- 172.21.2.112:0/492066775 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2d2c1b0870 msgr2=0x7f2d2c0f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:41.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.932 7f2d328fa700 1 --2- 172.21.2.112:0/492066775 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2d2c1b0870 0x7f2d2c0f20e0 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 rx=0x7f2d20006700 tx=0x7f2d20009b80).stop 2021-02-10T11:36:41.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.932 7f2d328fa700 1 -- 172.21.2.112:0/492066775 shutdown_connections 2021-02-10T11:36:41.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.932 7f2d328fa700 1 --2- 172.21.2.112:0/492066775 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2d2c1b0870 0x7f2d2c0f20e0 unknown :-1 s=CLOSED pgs=1648 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:41.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.932 7f2d328fa700 1 --2- 172.21.2.112:0/492066775 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2d2c0efe00 0x7f2d2c0f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:41.935 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.932 7f2d328fa700 1 -- 172.21.2.112:0/492066775 >> 172.21.2.112:0/492066775 conn(0x7f2d2c0ce3e0 msgr2=0x7f2d2c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:41.935 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.932 7f2d328fa700 1 -- 172.21.2.112:0/492066775 shutdown_connections 2021-02-10T11:36:41.935 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.932 7f2d328fa700 1 -- 172.21.2.112:0/492066775 wait complete. 2021-02-10T11:36:41.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.947 7f2d328fa700 1 Processor -- start 2021-02-10T11:36:41.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.947 7f2d328fa700 1 -- start start 2021-02-10T11:36:41.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.947 7f2d328fa700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2d2c19fec0 0x7f2d2c1acfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:41.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.947 7f2d328fa700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2d2c1aea70 0x7f2d2c1aee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:41.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.947 7f2d328fa700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2d2c1af490 0x7f2d2c1a0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:41.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.947 7f2d328fa700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d2c1a02c0 con 0x7f2d2c1aea70 2021-02-10T11:36:41.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.947 7f2d328fa700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d2c1af890 con 0x7f2d2c1af490 2021-02-10T11:36:41.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.947 7f2d328fa700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d2c1afab0 con 0x7f2d2c19fec0 2021-02-10T11:36:41.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.947 7f2d2b7fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2d2c1aea70 0x7f2d2c1aee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:41.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.947 7f2d2b7fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2d2c1aea70 0x7f2d2c1aee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:52214/0 (socket says 172.21.2.112:52214) 2021-02-10T11:36:41.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.947 7f2d2b7fe700 1 -- 172.21.2.112:0/1141847101 learned_addr learned my addr 172.21.2.112:0/1141847101 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:36:41.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.948 7f2d2b7fe700 1 -- 172.21.2.112:0/1141847101 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2d2c19fec0 msgr2=0x7f2d2c1acfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:41.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.948 7f2d2b7fe700 1 --2- 172.21.2.112:0/1141847101 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2d2c19fec0 0x7f2d2c1acfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:41.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.948 7f2d2b7fe700 1 -- 172.21.2.112:0/1141847101 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2d2c1af490 msgr2=0x7f2d2c1a0ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:41.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.948 7f2d2b7fe700 1 --2- 172.21.2.112:0/1141847101 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2d2c1af490 0x7f2d2c1a0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:41.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.948 7f2d2b7fe700 1 -- 172.21.2.112:0/1141847101 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d2c1a1100 con 0x7f2d2c1aea70 2021-02-10T11:36:41.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.948 7f2d2b7fe700 1 --2- 172.21.2.112:0/1141847101 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2d2c1aea70 0x7f2d2c1aee70 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 rx=0x7f2d2000b650 tx=0x7f2d2000b9f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:41.938 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.948 7f2d28ff9700 1 -- 172.21.2.112:0/1141847101 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2d2000bde0 con 0x7f2d2c1aea70 2021-02-10T11:36:41.938 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.948 7f2d28ff9700 1 -- 172.21.2.112:0/1141847101 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d20006570 con 0x7f2d2c1aea70 2021-02-10T11:36:41.938 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.948 7f2d28ff9700 1 -- 172.21.2.112:0/1141847101 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2d200022b0 con 0x7f2d2c1aea70 2021-02-10T11:36:41.938 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.948 7f2d328fa700 1 -- 172.21.2.112:0/1141847101 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d2c1a1100 con 0x7f2d2c1aea70 2021-02-10T11:36:41.938 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.948 7f2d328fa700 1 -- 172.21.2.112:0/1141847101 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d2c1a1100 con 0x7f2d2c1aea70 2021-02-10T11:36:41.938 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.949 7f2d28ff9700 1 -- 172.21.2.112:0/1141847101 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2d20009f90 con 0x7f2d2c1aea70 2021-02-10T11:36:41.938 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.949 7f2d28ff9700 1 --2- 172.21.2.112:0/1141847101 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2d0c0186c0 0x7f2d0c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:41.938 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.949 7f2d28ff9700 1 -- 172.21.2.112:0/1141847101 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5617+0+0 (secure 0 0 0) 0x7f2d20032800 con 0x7f2d2c1aea70 2021-02-10T11:36:41.938 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.950 7f2d2bfff700 1 --2- 172.21.2.112:0/1141847101 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2d0c0186c0 0x7f2d0c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:41.938 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.950 7f2d2bfff700 1 --2- 172.21.2.112:0/1141847101 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2d0c0186c0 0x7f2d0c01abc0 crc :-1 s=READY pgs=788 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:41.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.981 7f2d328fa700 1 -- 172.21.2.112:0/1141847101 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d2c1a05b0 con 0x7f2d2c1aea70 2021-02-10T11:36:41.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:41.983 7f2d28ff9700 1 -- 172.21.2.112:0/1141847101 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2d20009230 con 0x7f2d2c1aea70 2021-02-10T11:36:42.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:42.113 7f2d328fa700 1 -- 172.21.2.112:0/1141847101 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2d2c1a05b0 con 0x7f2d2c1aea70 2021-02-10T11:36:42.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:42.113 7f2d28ff9700 1 -- 172.21.2.112:0/1141847101 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) v1 ==== 94+0+2101 (secure 0 0 0) 0x7f2d2000bde0 con 0x7f2d2c1aea70 2021-02-10T11:36:42.100 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 57 2021-02-10T11:36:42.100 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:36:42.100 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":57,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8820,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/3613517698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":3613517698},{"type":"v1","addr":"172.21.2.112:6833","nonce":3613517698}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":52}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:36:00.289087","modified":"2021-02-10 11:36:02.295150","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7087},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7087":{"gid":7087,"name":"b","rank":0,"incarnation":53,"state":"up:active","state_seq":3,"addr":"172.21.2.120:6835/1870802251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":1870802251},{"type":"v1","addr":"172.21.2.120:6835","nonce":1870802251}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-02-10T11:36:42.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:42.115 7f2d328fa700 1 -- 172.21.2.112:0/1141847101 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2d0c0186c0 msgr2=0x7f2d0c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:42.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:42.115 7f2d328fa700 1 --2- 172.21.2.112:0/1141847101 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2d0c0186c0 0x7f2d0c01abc0 crc :-1 s=READY pgs=788 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:42.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:42.115 7f2d328fa700 1 -- 172.21.2.112:0/1141847101 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2d2c1aea70 msgr2=0x7f2d2c1aee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:42.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:42.115 7f2d328fa700 1 --2- 172.21.2.112:0/1141847101 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2d2c1aea70 0x7f2d2c1aee70 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 rx=0x7f2d2000b650 tx=0x7f2d2000b9f0).stop 2021-02-10T11:36:42.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:42.115 7f2d328fa700 1 -- 172.21.2.112:0/1141847101 shutdown_connections 2021-02-10T11:36:42.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:42.115 7f2d328fa700 1 --2- 172.21.2.112:0/1141847101 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2d0c0186c0 0x7f2d0c01abc0 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:42.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:42.115 7f2d328fa700 1 --2- 172.21.2.112:0/1141847101 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2d2c1af490 0x7f2d2c1a0ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:42.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:42.115 7f2d328fa700 1 --2- 172.21.2.112:0/1141847101 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2d2c1aea70 0x7f2d2c1aee70 unknown :-1 s=CLOSED pgs=1649 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:42.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:42.115 7f2d328fa700 1 --2- 172.21.2.112:0/1141847101 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2d2c19fec0 0x7f2d2c1acfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:42.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:42.115 7f2d328fa700 1 -- 172.21.2.112:0/1141847101 >> 172.21.2.112:0/1141847101 conn(0x7f2d2c1a37a0 msgr2=0x7f2d2c18c230 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:42.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:42.115 7f2d328fa700 1 -- 172.21.2.112:0/1141847101 shutdown_connections 2021-02-10T11:36:42.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:42.115 7f2d328fa700 1 -- 172.21.2.112:0/1141847101 wait complete. 2021-02-10T11:36:42.111 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs fail cephfs 2021-02-10T11:36:42.537 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:42.534 7f10ccb79700 1 Processor -- start 2021-02-10T11:36:42.538 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:42.534 7f10ccb79700 1 -- start start 2021-02-10T11:36:42.538 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:42.534 7f10ccb79700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f10c80d0340 0x7f10c80ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:42.538 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:42.534 7f10ccb79700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10c80f0b10 con 0x7f10c80d1c70 2021-02-10T11:36:42.538 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:42.534 7f10ccb79700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10c81b0be0 con 0x7f10c80d1870 2021-02-10T11:36:42.538 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:42.534 7f10ccb79700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10c80e5ed0 con 0x7f10c80d0340 2021-02-10T11:36:42.538 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:42.535 7f10c705f700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f10c80d1c70 0x7f10c80efc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:51688/0 (socket says 172.21.2.112:51688) 2021-02-10T11:36:42.538 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:42.535 7f10c705f700 1 -- 172.21.2.112:0/3056057365 learned_addr learned my addr 172.21.2.112:0/3056057365 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:36:42.538 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:42.535 7f10c605d700 1 --2- 172.21.2.112:0/3056057365 >> v2:172.21.2.120:3300/0 conn(0x7f10c80d0340 0x7f10c80ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:42.538 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:42.535 7f10c585c700 1 -- 172.21.2.112:0/3056057365 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2287638978 0 0) 0x7f10c0001500 con 0x7f10c80d1c70 2021-02-10T11:36:42.538 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:42.535 7f10c585c700 1 -- 172.21.2.112:0/3056057365 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10b4002fe0 con 0x7f10c80d1c70 2021-02-10T11:36:42.539 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:42.535 7f10c585c700 1 -- 172.21.2.112:0/3056057365 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4275424641 0 0) 0x7f10c0001500 con 0x7f10c80d1c70 2021-02-10T11:36:42.539 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:42.535 7f10c605d700 1 -- 172.21.2.112:0/3056057365 >> v1:172.21.2.120:6789/0 conn(0x7f10c80d1870 legacy=0x7f10c81b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:42.539 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:42.535 7f10c605d700 1 -- 172.21.2.112:0/3056057365 >> v1:172.21.2.112:6789/0 conn(0x7f10c80d1c70 legacy=0x7f10c80efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:42.539 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:42.535 7f10c605d700 1 -- 172.21.2.112:0/3056057365 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10c80f0450 con 0x7f10c80d0340 2021-02-10T11:36:42.539 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:42.536 7f10c605d700 1 --2- 172.21.2.112:0/3056057365 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f10c80d0340 0x7f10c80ef640 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 rx=0x7f10b8006650 tx=0x7f10b8009a80).ready entity=mon.1 client_cookie=876ae3d957803e2c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:42.539 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:42.536 7f10c585c700 1 -- 172.21.2.112:0/3056057365 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f10b800c0c0 con 0x7f10c80d0340 2021-02-10T11:36:42.539 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:42.536 7f10c585c700 1 -- 172.21.2.112:0/3056057365 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10b800c340 con 0x7f10c80d0340 2021-02-10T11:36:42.539 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:42.536 7f10c585c700 1 -- 172.21.2.112:0/3056057365 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f10b800c860 con 0x7f10c80d0340 2021-02-10T11:36:42.539 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:42.536 7f10ccb79700 1 -- 172.21.2.112:0/3056057365 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f10c80d0340 msgr2=0x7f10c80ef640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:42.539 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:42.536 7f10ccb79700 1 --2- 172.21.2.112:0/3056057365 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f10c80d0340 0x7f10c80ef640 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 rx=0x7f10b8006650 tx=0x7f10b8009a80).stop 2021-02-10T11:36:42.539 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:42.537 7f10ccb79700 1 -- 172.21.2.112:0/3056057365 shutdown_connections 2021-02-10T11:36:42.539 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:42.537 7f10ccb79700 1 --2- 172.21.2.112:0/3056057365 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f10c80d0340 0x7f10c80ef640 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:42.539 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:42.537 7f10ccb79700 1 -- 172.21.2.112:0/3056057365 >> 172.21.2.112:0/3056057365 conn(0x7f10c80ce450 msgr2=0x7f10c80d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:42.539 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:42.537 7f10ccb79700 1 -- 172.21.2.112:0/3056057365 shutdown_connections 2021-02-10T11:36:42.540 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:42.537 7f10ccb79700 1 -- 172.21.2.112:0/3056057365 wait complete. 2021-02-10T11:36:42.541 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:42.551 7f10ccb79700 1 Processor -- start 2021-02-10T11:36:42.541 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:42.552 7f10ccb79700 1 -- start start 2021-02-10T11:36:42.541 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:42.552 7f10ccb79700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f10c80a2fb0 0x7f10c80a33b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:42.541 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:42.552 7f10ccb79700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f10c80a39d0 0x7f10c80a4860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:42.541 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:42.552 7f10ccb79700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f10c80a3dd0 0x7f10c80a4e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:42.541 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:42.552 7f10ccb79700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10c80a2620 con 0x7f10c80a2fb0 2021-02-10T11:36:42.541 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:42.552 7f10ccb79700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10c80a2840 con 0x7f10c80a39d0 2021-02-10T11:36:42.541 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:42.552 7f10ccb79700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10c80a2b80 con 0x7f10c80a3dd0 2021-02-10T11:36:42.541 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:42.552 7f10c685e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f10c80a2fb0 0x7f10c80a33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:42.541 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:42.552 7f10c685e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f10c80a2fb0 0x7f10c80a33b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:52230/0 (socket says 172.21.2.112:52230) 2021-02-10T11:36:42.542 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:42.552 7f10c685e700 1 -- 172.21.2.112:0/749159880 learned_addr learned my addr 172.21.2.112:0/749159880 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:36:42.542 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:42.552 7f10c685e700 1 -- 172.21.2.112:0/749159880 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f10c80a3dd0 msgr2=0x7f10c80a4e80 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:36:42.542 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:42.552 7f10c685e700 1 --2- 172.21.2.112:0/749159880 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f10c80a3dd0 0x7f10c80a4e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:42.542 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:42.552 7f10c685e700 1 -- 172.21.2.112:0/749159880 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f10c80a39d0 msgr2=0x7f10c80a4860 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:36:42.542 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:42.552 7f10c685e700 1 --2- 172.21.2.112:0/749159880 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f10c80a39d0 0x7f10c80a4860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:42.542 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:42.552 7f10c685e700 1 -- 172.21.2.112:0/749159880 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10c80a54a0 con 0x7f10c80a2fb0 2021-02-10T11:36:42.542 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:42.552 7f10c685e700 1 --2- 172.21.2.112:0/749159880 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f10c80a2fb0 0x7f10c80a33b0 secure :-1 s=READY pgs=1651 cs=0 l=1 rev1=1 rx=0x7f10b000b5b0 tx=0x7f10b000b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:42.542 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:42.553 7f10bf7fe700 1 -- 172.21.2.112:0/749159880 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f10b000bd40 con 0x7f10c80a2fb0 2021-02-10T11:36:42.542 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:42.553 7f10bf7fe700 1 -- 172.21.2.112:0/749159880 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10b0002080 con 0x7f10c80a2fb0 2021-02-10T11:36:42.542 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:42.553 7f10bf7fe700 1 -- 172.21.2.112:0/749159880 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f10b00024e0 con 0x7f10c80a2fb0 2021-02-10T11:36:42.542 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:42.553 7f10ccb79700 1 -- 172.21.2.112:0/749159880 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10c80a54a0 con 0x7f10c80a2fb0 2021-02-10T11:36:42.542 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:42.553 7f10ccb79700 1 -- 172.21.2.112:0/749159880 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10c80a5a30 con 0x7f10c80a2fb0 2021-02-10T11:36:42.542 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:42.553 7f10bf7fe700 1 -- 172.21.2.112:0/749159880 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f10b0002260 con 0x7f10c80a2fb0 2021-02-10T11:36:42.542 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:42.553 7f10bf7fe700 1 --2- 172.21.2.112:0/749159880 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f10ac0186c0 0x7f10ac01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:42.543 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:42.553 7f10bf7fe700 1 -- 172.21.2.112:0/749159880 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5617+0+0 (secure 0 0 0) 0x7f10b00325b0 con 0x7f10c80a2fb0 2021-02-10T11:36:42.543 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:42.554 7f10c605d700 1 --2- 172.21.2.112:0/749159880 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f10ac0186c0 0x7f10ac01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:42.543 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:42.556 7f10c605d700 1 --2- 172.21.2.112:0/749159880 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f10ac0186c0 0x7f10ac01abc0 crc :-1 s=READY pgs=789 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:42.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:42.584 7f10ccb79700 1 -- 172.21.2.112:0/749159880 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10c80a4330 con 0x7f10c80a2fb0 2021-02-10T11:36:42.573 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:42.587 7f10bf7fe700 1 -- 172.21.2.112:0/749159880 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f10b0000940 con 0x7f10c80a2fb0 2021-02-10T11:36:42.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:42.713 7f10ccb79700 1 -- 172.21.2.112:0/749159880 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f10c80a4330 con 0x7f10c80a2fb0 2021-02-10T11:36:43.421 INFO:tasks.ceph.mon.a.gibba012.stderr:2021-02-10 11:36:43.434 7fc3fe6e2700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-02-10T11:36:43.427 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:43.440 7f10bf7fe700 1 -- 172.21.2.112:0/749159880 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v58) v1 ==== 161+0+0 (secure 0 0 0) 0x7f10b001f320 con 0x7f10c80a2fb0 2021-02-10T11:36:43.467 INFO:teuthology.orchestra.run.gibba012.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2021-02-10T11:36:43.469 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:43.483 7f10ccb79700 1 -- 172.21.2.112:0/749159880 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f10ac0186c0 msgr2=0x7f10ac01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:43.469 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:43.483 7f10ccb79700 1 --2- 172.21.2.112:0/749159880 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f10ac0186c0 0x7f10ac01abc0 crc :-1 s=READY pgs=789 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:43.470 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:43.483 7f10ccb79700 1 -- 172.21.2.112:0/749159880 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f10c80a2fb0 msgr2=0x7f10c80a33b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:43.470 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:43.483 7f10ccb79700 1 --2- 172.21.2.112:0/749159880 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f10c80a2fb0 0x7f10c80a33b0 secure :-1 s=READY pgs=1651 cs=0 l=1 rev1=1 rx=0x7f10b000b5b0 tx=0x7f10b000b950).stop 2021-02-10T11:36:43.470 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:43.483 7f10ccb79700 1 -- 172.21.2.112:0/749159880 shutdown_connections 2021-02-10T11:36:43.470 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:43.483 7f10ccb79700 1 --2- 172.21.2.112:0/749159880 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f10ac0186c0 0x7f10ac01abc0 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:43.470 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:43.483 7f10ccb79700 1 --2- 172.21.2.112:0/749159880 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f10c80a39d0 0x7f10c80a4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:43.470 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:43.483 7f10ccb79700 1 --2- 172.21.2.112:0/749159880 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f10c80a3dd0 0x7f10c80a4e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:43.470 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:43.483 7f10ccb79700 1 --2- 172.21.2.112:0/749159880 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f10c80a2fb0 0x7f10c80a33b0 unknown :-1 s=CLOSED pgs=1651 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:43.470 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:43.483 7f10ccb79700 1 -- 172.21.2.112:0/749159880 >> 172.21.2.112:0/749159880 conn(0x7f10c808f030 msgr2=0x7f10c80b5e80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:43.470 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:43.483 7f10ccb79700 1 -- 172.21.2.112:0/749159880 shutdown_connections 2021-02-10T11:36:43.470 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:43.483 7f10ccb79700 1 -- 172.21.2.112:0/749159880 wait complete. 2021-02-10T11:36:43.478 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:36:43.479 INFO:tasks.ceph.mds.b.gibba020.stdout:starting mds.b at 2021-02-10T11:36:43.907 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:43.905 7f931f09f700 1 Processor -- start 2021-02-10T11:36:43.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:43.905 7f931f09f700 1 -- start start 2021-02-10T11:36:43.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:43.905 7f931f09f700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f93180d1c70 0x7f93180f20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:43.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:43.905 7f931f09f700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93180f0b10 con 0x7f93180d1870 2021-02-10T11:36:43.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:43.905 7f931f09f700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93181b0be0 con 0x7f93180d0340 2021-02-10T11:36:43.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:43.905 7f931f09f700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93180e5ed0 con 0x7f93180d1c70 2021-02-10T11:36:43.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:43.905 7f931cebb700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f93180d1870 0x7f93181b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:51698/0 (socket says 172.21.2.112:51698) 2021-02-10T11:36:43.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:43.905 7f931cebb700 1 -- 172.21.2.112:0/990155720 learned_addr learned my addr 172.21.2.112:0/990155720 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:36:43.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:43.905 7f931d6bc700 1 --2- 172.21.2.112:0/990155720 >> v2:172.21.2.120:3300/0 conn(0x7f93180d1c70 0x7f93180f20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:43.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:43.906 7f93177fe700 1 -- 172.21.2.112:0/990155720 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1039707842 0 0) 0x7f9300001180 con 0x7f93180d1870 2021-02-10T11:36:43.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:43.906 7f93177fe700 1 -- 172.21.2.112:0/990155720 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9304002fe0 con 0x7f93180d1870 2021-02-10T11:36:43.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:43.906 7f931d6bc700 1 -- 172.21.2.112:0/990155720 >> v1:172.21.2.120:6789/0 conn(0x7f93180d0340 legacy=0x7f93180ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:43.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:43.906 7f931d6bc700 1 -- 172.21.2.112:0/990155720 >> v1:172.21.2.112:6789/0 conn(0x7f93180d1870 legacy=0x7f93181b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:43.909 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:43.906 7f931d6bc700 1 -- 172.21.2.112:0/990155720 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93180efe30 con 0x7f93180d1c70 2021-02-10T11:36:43.909 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:43.906 7f931d6bc700 1 --2- 172.21.2.112:0/990155720 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f93180d1c70 0x7f93180f20a0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 rx=0x7f9310006ad0 tx=0x7f9310009ad0).ready entity=mon.1 client_cookie=e715367f27087ce1 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:43.909 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:43.906 7f93177fe700 1 -- 172.21.2.112:0/990155720 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f93100022e0 con 0x7f93180d1c70 2021-02-10T11:36:43.909 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:43.906 7f93177fe700 1 -- 172.21.2.112:0/990155720 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9310006410 con 0x7f93180d1c70 2021-02-10T11:36:43.909 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:43.906 7f93177fe700 1 -- 172.21.2.112:0/990155720 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9310002560 con 0x7f93180d1c70 2021-02-10T11:36:43.909 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:43.906 7f931f09f700 1 -- 172.21.2.112:0/990155720 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f93180d1c70 msgr2=0x7f93180f20a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:43.909 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:43.906 7f931f09f700 1 --2- 172.21.2.112:0/990155720 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f93180d1c70 0x7f93180f20a0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 rx=0x7f9310006ad0 tx=0x7f9310009ad0).stop 2021-02-10T11:36:43.909 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:43.907 7f931f09f700 1 -- 172.21.2.112:0/990155720 shutdown_connections 2021-02-10T11:36:43.909 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:43.907 7f931f09f700 1 --2- 172.21.2.112:0/990155720 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f93180d1c70 0x7f93180f20a0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:43.909 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:43.907 7f931f09f700 1 -- 172.21.2.112:0/990155720 >> 172.21.2.112:0/990155720 conn(0x7f93180ce450 msgr2=0x7f93180d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:43.909 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:43.907 7f931f09f700 1 -- 172.21.2.112:0/990155720 shutdown_connections 2021-02-10T11:36:43.909 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:43.907 7f931f09f700 1 -- 172.21.2.112:0/990155720 wait complete. 2021-02-10T11:36:43.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:43.921 7f931f09f700 1 Processor -- start 2021-02-10T11:36:43.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:43.921 7f931f09f700 1 -- start start 2021-02-10T11:36:43.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:43.922 7f931f09f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f93180b4e00 0x7f93180a0fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:43.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:43.922 7f931f09f700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f93180a3a60 0x7f93180a3e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:43.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:43.922 7f931f09f700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9318094750 0x7f9318094b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:43.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:43.922 7f931f09f700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93180a4550 con 0x7f93180b4e00 2021-02-10T11:36:43.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:43.922 7f931f09f700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f931818bf30 con 0x7f9318094750 2021-02-10T11:36:43.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:43.922 7f931f09f700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f931818ba60 con 0x7f93180a3a60 2021-02-10T11:36:43.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:43.922 7f931cebb700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f93180b4e00 0x7f93180a0fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:43.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:43.922 7f931cebb700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f93180b4e00 0x7f93180a0fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:52242/0 (socket says 172.21.2.112:52242) 2021-02-10T11:36:43.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:43.922 7f931cebb700 1 -- 172.21.2.112:0/2219225085 learned_addr learned my addr 172.21.2.112:0/2219225085 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:36:43.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:43.922 7f9317fff700 1 --2- 172.21.2.112:0/2219225085 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f93180a3a60 0x7f93180a3e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:43.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:43.922 7f931d6bc700 1 --2- 172.21.2.112:0/2219225085 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9318094750 0x7f9318094b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:43.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:43.922 7f931cebb700 1 -- 172.21.2.112:0/2219225085 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f93180a3a60 msgr2=0x7f93180a3e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:43.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:43.922 7f931cebb700 1 --2- 172.21.2.112:0/2219225085 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f93180a3a60 0x7f93180a3e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:43.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:43.922 7f931cebb700 1 -- 172.21.2.112:0/2219225085 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9318094750 msgr2=0x7f9318094b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:43.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:43.922 7f931cebb700 1 --2- 172.21.2.112:0/2219225085 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9318094750 0x7f9318094b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:43.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:43.922 7f931cebb700 1 -- 172.21.2.112:0/2219225085 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93180a15c0 con 0x7f93180b4e00 2021-02-10T11:36:43.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:43.923 7f931cebb700 1 --2- 172.21.2.112:0/2219225085 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f93180b4e00 0x7f93180a0fa0 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 rx=0x7f930000a5d0 tx=0x7f930000a970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:43.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:43.923 7f93157fa700 1 -- 172.21.2.112:0/2219225085 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9300000ce0 con 0x7f93180b4e00 2021-02-10T11:36:43.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:43.923 7f93157fa700 1 -- 172.21.2.112:0/2219225085 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93000043a0 con 0x7f93180b4e00 2021-02-10T11:36:43.913 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:43.923 7f93157fa700 1 -- 172.21.2.112:0/2219225085 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f93000047f0 con 0x7f93180b4e00 2021-02-10T11:36:43.913 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:43.923 7f931f09f700 1 -- 172.21.2.112:0/2219225085 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93180a15c0 con 0x7f93180b4e00 2021-02-10T11:36:43.913 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:43.923 7f931f09f700 1 -- 172.21.2.112:0/2219225085 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9318095460 con 0x7f93180b4e00 2021-02-10T11:36:43.913 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:43.923 7f93157fa700 1 -- 172.21.2.112:0/2219225085 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9300004790 con 0x7f93180b4e00 2021-02-10T11:36:43.913 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:43.923 7f93157fa700 1 --2- 172.21.2.112:0/2219225085 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f92fc0186d0 0x7f92fc01abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:43.913 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:43.923 7f93157fa700 1 -- 172.21.2.112:0/2219225085 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 5675+0+0 (secure 0 0 0) 0x7f9300032660 con 0x7f93180b4e00 2021-02-10T11:36:43.913 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:43.924 7f9317fff700 1 --2- 172.21.2.112:0/2219225085 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f92fc0186d0 0x7f92fc01abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:43.913 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:43.924 7f9317fff700 1 --2- 172.21.2.112:0/2219225085 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f92fc0186d0 0x7f92fc01abd0 crc :-1 s=READY pgs=790 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:43.941 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:43.955 7f931f09f700 1 -- 172.21.2.112:0/2219225085 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93180ee0e0 con 0x7f93180b4e00 2021-02-10T11:36:43.944 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:43.957 7f93157fa700 1 -- 172.21.2.112:0/2219225085 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f93000043a0 con 0x7f93180b4e00 2021-02-10T11:36:44.074 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:44.087 7f931f09f700 1 -- 172.21.2.112:0/2219225085 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f93180ce450 con 0x7f93180b4e00 2021-02-10T11:36:44.074 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:44.088 7f93157fa700 1 -- 172.21.2.112:0/2219225085 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+1756 (secure 0 0 0) 0x7f930000ad60 con 0x7f93180b4e00 2021-02-10T11:36:44.075 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 58 2021-02-10T11:36:44.075 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:36:44.075 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":58,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8820,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/3613517698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":3613517698},{"type":"v1","addr":"172.21.2.112:6833","nonce":3613517698}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":52}],"filesystems":[{"mdsmap":{"epoch":58,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:36:00.289087","modified":"2021-02-10 11:36:43.435585","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-02-10T11:36:44.076 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:44.089 7f931f09f700 1 -- 172.21.2.112:0/2219225085 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f92fc0186d0 msgr2=0x7f92fc01abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:44.076 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:44.089 7f931f09f700 1 --2- 172.21.2.112:0/2219225085 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f92fc0186d0 0x7f92fc01abd0 crc :-1 s=READY pgs=790 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:44.076 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:44.090 7f931f09f700 1 -- 172.21.2.112:0/2219225085 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f93180b4e00 msgr2=0x7f93180a0fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:44.076 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:44.090 7f931f09f700 1 --2- 172.21.2.112:0/2219225085 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f93180b4e00 0x7f93180a0fa0 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 rx=0x7f930000a5d0 tx=0x7f930000a970).stop 2021-02-10T11:36:44.077 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:44.090 7f931f09f700 1 -- 172.21.2.112:0/2219225085 shutdown_connections 2021-02-10T11:36:44.077 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:44.090 7f931f09f700 1 --2- 172.21.2.112:0/2219225085 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f92fc0186d0 0x7f92fc01abd0 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:44.077 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:44.090 7f931f09f700 1 --2- 172.21.2.112:0/2219225085 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9318094750 0x7f9318094b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:44.077 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:44.090 7f931f09f700 1 --2- 172.21.2.112:0/2219225085 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f93180a3a60 0x7f93180a3e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:44.077 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:44.090 7f931f09f700 1 --2- 172.21.2.112:0/2219225085 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f93180b4e00 0x7f93180a0fa0 unknown :-1 s=CLOSED pgs=1654 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:44.077 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:44.090 7f931f09f700 1 -- 172.21.2.112:0/2219225085 >> 172.21.2.112:0/2219225085 conn(0x7f931818cab0 msgr2=0x7f9318092c60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:44.077 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:44.090 7f931f09f700 1 -- 172.21.2.112:0/2219225085 shutdown_connections 2021-02-10T11:36:44.077 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:44.090 7f931f09f700 1 -- 172.21.2.112:0/2219225085 wait complete. 2021-02-10T11:36:44.085 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2021-02-10T11:36:44.518 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:44.515 7f9259ef4700 1 Processor -- start 2021-02-10T11:36:44.519 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:44.516 7f9259ef4700 1 -- start start 2021-02-10T11:36:44.519 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:44.516 7f9259ef4700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f92540d0330 0x7f92540ef630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:44.519 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:44.516 7f9259ef4700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f92540d1c60 0x7f92540efc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:44.519 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:44.516 7f9259ef4700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92540f0b00 con 0x7f92540d1860 2021-02-10T11:36:44.519 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:44.516 7f9259ef4700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92540e5ec0 con 0x7f92540d1c60 2021-02-10T11:36:44.519 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:44.516 7f9259ef4700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92541b0bd0 con 0x7f92540d0330 2021-02-10T11:36:44.519 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:44.516 7f92537fe700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f92540d1860 0x7f92541b03e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:51712/0 (socket says 172.21.2.112:51712) 2021-02-10T11:36:44.519 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:44.516 7f92537fe700 1 -- 172.21.2.112:0/2155083025 learned_addr learned my addr 172.21.2.112:0/2155083025 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:36:44.519 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:44.516 7f9253fff700 1 --2- 172.21.2.112:0/2155083025 >> v2:172.21.2.112:3300/0 conn(0x7f92540d1c60 0x7f92540efc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:44.519 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:44.517 7f9253fff700 1 -- 172.21.2.112:0/2155083025 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f92540d0330 msgr2=0x7f92540ef630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:44.519 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:44.517 7f9253fff700 1 --2- 172.21.2.112:0/2155083025 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f92540d0330 0x7f92540ef630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:44.519 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:44.517 7f9253fff700 1 -- 172.21.2.112:0/2155083025 >> v1:172.21.2.112:6789/0 conn(0x7f92540d1860 legacy=0x7f92541b03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:44.520 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:44.517 7f9253fff700 1 -- 172.21.2.112:0/2155083025 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92540f2090 con 0x7f92540d1c60 2021-02-10T11:36:44.520 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:44.517 7f9253fff700 1 --2- 172.21.2.112:0/2155083025 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f92540d1c60 0x7f92540efc50 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 rx=0x7f924c006e30 tx=0x7f924c009e60).ready entity=mon.0 client_cookie=9d9542e6ee63f8f5 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:44.520 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:44.517 7f92527fc700 1 -- 172.21.2.112:0/2155083025 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f924c006a50 con 0x7f92540d1c60 2021-02-10T11:36:44.520 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:44.517 7f92527fc700 1 -- 172.21.2.112:0/2155083025 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f924c00b300 con 0x7f92540d1c60 2021-02-10T11:36:44.520 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:44.517 7f92527fc700 1 -- 172.21.2.112:0/2155083025 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f924c00b7e0 con 0x7f92540d1c60 2021-02-10T11:36:44.520 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:44.517 7f9259ef4700 1 -- 172.21.2.112:0/2155083025 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f92540d1c60 msgr2=0x7f92540efc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:44.520 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:44.517 7f9259ef4700 1 --2- 172.21.2.112:0/2155083025 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f92540d1c60 0x7f92540efc50 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 rx=0x7f924c006e30 tx=0x7f924c009e60).stop 2021-02-10T11:36:44.520 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:44.518 7f9259ef4700 1 -- 172.21.2.112:0/2155083025 shutdown_connections 2021-02-10T11:36:44.520 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:44.518 7f9259ef4700 1 --2- 172.21.2.112:0/2155083025 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f92540d1c60 0x7f92540efc50 unknown :-1 s=CLOSED pgs=1656 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:44.520 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:44.518 7f9259ef4700 1 --2- 172.21.2.112:0/2155083025 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f92540d0330 0x7f92540ef630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:44.520 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:44.518 7f9259ef4700 1 -- 172.21.2.112:0/2155083025 >> 172.21.2.112:0/2155083025 conn(0x7f92540ce440 msgr2=0x7f92540d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:44.520 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:44.518 7f9259ef4700 1 -- 172.21.2.112:0/2155083025 shutdown_connections 2021-02-10T11:36:44.520 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:44.518 7f9259ef4700 1 -- 172.21.2.112:0/2155083025 wait complete. 2021-02-10T11:36:44.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:44.532 7f9259ef4700 1 Processor -- start 2021-02-10T11:36:44.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:44.533 7f9259ef4700 1 -- start start 2021-02-10T11:36:44.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:44.533 7f9259ef4700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f92540b7fd0 0x7f925418e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:44.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:44.533 7f9259ef4700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f92541a0620 0x7f92540b7190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:44.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:44.533 7f9259ef4700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f925418ecc0 0x7f92540b77b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:44.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:44.533 7f9259ef4700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f925409c9a0 con 0x7f92540b7fd0 2021-02-10T11:36:44.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:44.533 7f9259ef4700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f925409cef0 con 0x7f925418ecc0 2021-02-10T11:36:44.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:44.533 7f9259ef4700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f925409d110 con 0x7f92541a0620 2021-02-10T11:36:44.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:44.533 7f92537fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f92540b7fd0 0x7f925418e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:44.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:44.533 7f92537fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f92540b7fd0 0x7f925418e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:52256/0 (socket says 172.21.2.112:52256) 2021-02-10T11:36:44.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:44.533 7f92537fe700 1 -- 172.21.2.112:0/3344869735 learned_addr learned my addr 172.21.2.112:0/3344869735 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:36:44.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:44.533 7f92537fe700 1 -- 172.21.2.112:0/3344869735 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f92541a0620 msgr2=0x7f92540b7190 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:36:44.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:44.533 7f92537fe700 1 --2- 172.21.2.112:0/3344869735 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f92541a0620 0x7f92540b7190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:44.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:44.533 7f92537fe700 1 -- 172.21.2.112:0/3344869735 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f925418ecc0 msgr2=0x7f92540b77b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:36:44.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:44.533 7f92537fe700 1 --2- 172.21.2.112:0/3344869735 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f925418ecc0 0x7f92540b77b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:44.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:44.534 7f92537fe700 1 -- 172.21.2.112:0/3344869735 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f925409d330 con 0x7f92540b7fd0 2021-02-10T11:36:44.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:44.534 7f92537fe700 1 --2- 172.21.2.112:0/3344869735 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f92540b7fd0 0x7f925418e6a0 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 rx=0x7f924400a4f0 tx=0x7f924400a890).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:44.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:44.534 7f924bfff700 1 -- 172.21.2.112:0/3344869735 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f924400ac80 con 0x7f92540b7fd0 2021-02-10T11:36:44.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:44.534 7f924bfff700 1 -- 172.21.2.112:0/3344869735 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92440009b0 con 0x7f92540b7fd0 2021-02-10T11:36:44.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:44.534 7f9259ef4700 1 -- 172.21.2.112:0/3344869735 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f925409d330 con 0x7f92540b7fd0 2021-02-10T11:36:44.524 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:44.534 7f9259ef4700 1 -- 172.21.2.112:0/3344869735 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f925409d8c0 con 0x7f92540b7fd0 2021-02-10T11:36:44.524 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:44.534 7f924bfff700 1 -- 172.21.2.112:0/3344869735 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9244004590 con 0x7f92540b7fd0 2021-02-10T11:36:44.524 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:44.534 7f924bfff700 1 -- 172.21.2.112:0/3344869735 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f92440043a0 con 0x7f92540b7fd0 2021-02-10T11:36:44.524 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:44.535 7f924bfff700 1 --2- 172.21.2.112:0/3344869735 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f92380186d0 0x7f923801abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:44.524 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:44.535 7f924bfff700 1 -- 172.21.2.112:0/3344869735 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 5675+0+0 (secure 0 0 0) 0x7f9244032820 con 0x7f92540b7fd0 2021-02-10T11:36:44.524 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:44.535 7f9252ffd700 1 --2- 172.21.2.112:0/3344869735 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f92380186d0 0x7f923801abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:44.524 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:44.535 7f9252ffd700 1 --2- 172.21.2.112:0/3344869735 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f92380186d0 0x7f923801abd0 crc :-1 s=READY pgs=791 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:44.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:44.566 7f9259ef4700 1 -- 172.21.2.112:0/3344869735 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92540ebba0 con 0x7f92540b7fd0 2021-02-10T11:36:44.555 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:44.568 7f924bfff700 1 -- 172.21.2.112:0/3344869735 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9244000ed0 con 0x7f92540b7fd0 2021-02-10T11:36:44.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:44.696 7f9259ef4700 1 -- 172.21.2.112:0/3344869735 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f92540f0b00 con 0x7f92540b7fd0 2021-02-10T11:36:45.434 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:45.447 7f924bfff700 1 -- 172.21.2.112:0/3344869735 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v59) v1 ==== 104+0+0 (secure 0 0 0) 0x7f924401f500 con 0x7f92540b7fd0 2021-02-10T11:36:45.455 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:45.468 7f9259ef4700 1 -- 172.21.2.112:0/3344869735 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f92380186d0 msgr2=0x7f923801abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:45.455 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:45.468 7f9259ef4700 1 --2- 172.21.2.112:0/3344869735 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f92380186d0 0x7f923801abd0 crc :-1 s=READY pgs=791 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:45.455 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:45.468 7f9259ef4700 1 -- 172.21.2.112:0/3344869735 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f92540b7fd0 msgr2=0x7f925418e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:45.455 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:45.468 7f9259ef4700 1 --2- 172.21.2.112:0/3344869735 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f92540b7fd0 0x7f925418e6a0 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 rx=0x7f924400a4f0 tx=0x7f924400a890).stop 2021-02-10T11:36:45.455 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:45.469 7f9259ef4700 1 -- 172.21.2.112:0/3344869735 shutdown_connections 2021-02-10T11:36:45.455 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:45.469 7f9259ef4700 1 --2- 172.21.2.112:0/3344869735 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f92380186d0 0x7f923801abd0 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:45.455 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:45.469 7f9259ef4700 1 --2- 172.21.2.112:0/3344869735 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f925418ecc0 0x7f92540b77b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:45.456 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:45.469 7f9259ef4700 1 --2- 172.21.2.112:0/3344869735 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f92541a0620 0x7f92540b7190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:45.456 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:45.469 7f9259ef4700 1 --2- 172.21.2.112:0/3344869735 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f92540b7fd0 0x7f925418e6a0 unknown :-1 s=CLOSED pgs=1657 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:45.456 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:45.469 7f9259ef4700 1 -- 172.21.2.112:0/3344869735 >> 172.21.2.112:0/3344869735 conn(0x7f925409fd30 msgr2=0x7f92540ebdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:45.456 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:45.469 7f9259ef4700 1 -- 172.21.2.112:0/3344869735 shutdown_connections 2021-02-10T11:36:45.456 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:45.469 7f9259ef4700 1 -- 172.21.2.112:0/3344869735 wait complete. 2021-02-10T11:36:45.464 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2021-02-10T11:36:45.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:45.892 7f8b0c56c700 1 Processor -- start 2021-02-10T11:36:45.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:45.892 7f8b0c56c700 1 -- start start 2021-02-10T11:36:45.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:45.893 7f8b0c56c700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f8b0410b000 0x7f8b040cf760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:45.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:45.893 7f8b0c56c700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b040fe1d0 con 0x7f8b040d1880 2021-02-10T11:36:45.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:45.893 7f8b0c56c700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b041b0cc0 con 0x7f8b040d1c80 2021-02-10T11:36:45.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:45.893 7f8b0c56c700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b04109a70 con 0x7f8b0410b000 2021-02-10T11:36:45.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:45.893 7f8b0ab89700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f8b0410b000 0x7f8b040cf760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:45.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:45.893 7f8b0a388700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f8b040d1880 0x7f8b041b04d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:51724/0 (socket says 172.21.2.112:51724) 2021-02-10T11:36:45.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:45.893 7f8b0a388700 1 -- 172.21.2.112:0/2050652263 learned_addr learned my addr 172.21.2.112:0/2050652263 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:36:45.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:45.893 7f8b0ab89700 1 -- 172.21.2.112:0/2050652263 >> v1:172.21.2.120:6789/0 conn(0x7f8b040d1c80 legacy=0x7f8b040cef70 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:36:45.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:45.893 7f8b0ab89700 1 -- 172.21.2.112:0/2050652263 >> v1:172.21.2.112:6789/0 conn(0x7f8b040d1880 legacy=0x7f8b041b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:45.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:45.893 7f8b0ab89700 1 -- 172.21.2.112:0/2050652263 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b0410b400 con 0x7f8b0410b000 2021-02-10T11:36:45.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:45.893 7f8b0ab89700 1 --2- 172.21.2.112:0/2050652263 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8b0410b000 0x7f8b040cf760 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 rx=0x7f8afc006e30 tx=0x7f8afc009e60).ready entity=mon.0 client_cookie=8c783f5ea3320872 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:45.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:45.894 7f8b09386700 1 -- 172.21.2.112:0/2050652263 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8afc006a50 con 0x7f8b0410b000 2021-02-10T11:36:45.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:45.894 7f8b09386700 1 -- 172.21.2.112:0/2050652263 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8afc000f90 con 0x7f8b0410b000 2021-02-10T11:36:45.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:45.894 7f8b09386700 1 -- 172.21.2.112:0/2050652263 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8afc00b300 con 0x7f8b0410b000 2021-02-10T11:36:45.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:45.894 7f8b0c56c700 1 -- 172.21.2.112:0/2050652263 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8b0410b000 msgr2=0x7f8b040cf760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:45.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:45.894 7f8b0c56c700 1 --2- 172.21.2.112:0/2050652263 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8b0410b000 0x7f8b040cf760 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 rx=0x7f8afc006e30 tx=0x7f8afc009e60).stop 2021-02-10T11:36:45.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:45.894 7f8b0c56c700 1 -- 172.21.2.112:0/2050652263 shutdown_connections 2021-02-10T11:36:45.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:45.894 7f8b0c56c700 1 --2- 172.21.2.112:0/2050652263 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8b0410b000 0x7f8b040cf760 unknown :-1 s=CLOSED pgs=1659 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:45.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:45.894 7f8b0c56c700 1 -- 172.21.2.112:0/2050652263 >> 172.21.2.112:0/2050652263 conn(0x7f8b040ce430 msgr2=0x7f8b040d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:45.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:45.894 7f8b0c56c700 1 -- 172.21.2.112:0/2050652263 shutdown_connections 2021-02-10T11:36:45.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:45.894 7f8b0c56c700 1 -- 172.21.2.112:0/2050652263 wait complete. 2021-02-10T11:36:45.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:45.909 7f8b0c56c700 1 Processor -- start 2021-02-10T11:36:45.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:45.909 7f8b0c56c700 1 -- start start 2021-02-10T11:36:45.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:45.909 7f8b0c56c700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8b041a5940 0x7f8b04107730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:45.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:45.909 7f8b0c56c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8b041a5d40 0x7f8b04107d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:45.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:45.909 7f8b0c56c700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8b04108370 0x7f8b041a6840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:45.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:45.909 7f8b0c56c700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b041a6140 con 0x7f8b041a5d40 2021-02-10T11:36:45.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:45.909 7f8b0c56c700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b041a6e60 con 0x7f8b04108370 2021-02-10T11:36:45.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:45.909 7f8b0c56c700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b041a7080 con 0x7f8b041a5940 2021-02-10T11:36:45.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:45.909 7f8b09b87700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8b041a5d40 0x7f8b04107d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:45.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:45.909 7f8b09b87700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8b041a5d40 0x7f8b04107d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:52268/0 (socket says 172.21.2.112:52268) 2021-02-10T11:36:45.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:45.909 7f8b09b87700 1 -- 172.21.2.112:0/2921559338 learned_addr learned my addr 172.21.2.112:0/2921559338 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:36:45.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:45.910 7f8b09b87700 1 -- 172.21.2.112:0/2921559338 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8b041a5940 msgr2=0x7f8b04107730 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:36:45.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:45.910 7f8b09b87700 1 --2- 172.21.2.112:0/2921559338 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8b041a5940 0x7f8b04107730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:45.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:45.910 7f8b09b87700 1 -- 172.21.2.112:0/2921559338 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8b04108370 msgr2=0x7f8b041a6840 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:36:45.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:45.910 7f8b09b87700 1 --2- 172.21.2.112:0/2921559338 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8b04108370 0x7f8b041a6840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:45.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:45.910 7f8b09b87700 1 -- 172.21.2.112:0/2921559338 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b041a72a0 con 0x7f8b041a5d40 2021-02-10T11:36:45.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:45.910 7f8b09b87700 1 --2- 172.21.2.112:0/2921559338 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8b041a5d40 0x7f8b04107d50 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 rx=0x7f8af400b530 tx=0x7f8af400b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:45.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:45.910 7f8afaffd700 1 -- 172.21.2.112:0/2921559338 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8af400bcc0 con 0x7f8b041a5d40 2021-02-10T11:36:45.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:45.910 7f8afaffd700 1 -- 172.21.2.112:0/2921559338 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8af40009b0 con 0x7f8b041a5d40 2021-02-10T11:36:45.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:45.910 7f8b0c56c700 1 -- 172.21.2.112:0/2921559338 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b041a72a0 con 0x7f8b041a5d40 2021-02-10T11:36:45.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:45.910 7f8afaffd700 1 -- 172.21.2.112:0/2921559338 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8af40062a0 con 0x7f8b041a5d40 2021-02-10T11:36:45.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:45.910 7f8b0c56c700 1 -- 172.21.2.112:0/2921559338 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b041a7830 con 0x7f8b041a5d40 2021-02-10T11:36:45.900 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:45.911 7f8afaffd700 1 -- 172.21.2.112:0/2921559338 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8af4006020 con 0x7f8b041a5d40 2021-02-10T11:36:45.900 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:45.911 7f8afaffd700 1 --2- 172.21.2.112:0/2921559338 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8aec018730 0x7f8aec01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:45.900 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:45.911 7f8afaffd700 1 -- 172.21.2.112:0/2921559338 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 5675+0+0 (secure 0 0 0) 0x7f8af40327d0 con 0x7f8b041a5d40 2021-02-10T11:36:45.900 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:45.911 7f8b0a388700 1 --2- 172.21.2.112:0/2921559338 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8aec018730 0x7f8aec01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:45.900 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:45.912 7f8b0a388700 1 --2- 172.21.2.112:0/2921559338 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8aec018730 0x7f8aec01ac10 crc :-1 s=READY pgs=792 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:45.929 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:45.942 7f8b0c56c700 1 -- 172.21.2.112:0/2921559338 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b04080340 con 0x7f8b041a5d40 2021-02-10T11:36:45.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:45.945 7f8afaffd700 1 -- 172.21.2.112:0/2921559338 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8af4000c60 con 0x7f8b041a5d40 2021-02-10T11:36:46.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:46.074 7f8b0c56c700 1 -- 172.21.2.112:0/2921559338 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f8b04088070 con 0x7f8b041a5d40 2021-02-10T11:36:46.437 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:46.450 7f8afaffd700 1 -- 172.21.2.112:0/2921559338 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v77) v1 ==== 185+0+0 (secure 0 0 0) 0x7f8af4006020 con 0x7f8b041a5d40 2021-02-10T11:36:46.477 INFO:teuthology.orchestra.run.gibba012.stderr:pool 'cephfs_metadata' removed 2021-02-10T11:36:46.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:46.492 7f8b0c56c700 1 -- 172.21.2.112:0/2921559338 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8aec018730 msgr2=0x7f8aec01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:46.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:46.493 7f8b0c56c700 1 --2- 172.21.2.112:0/2921559338 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8aec018730 0x7f8aec01ac10 crc :-1 s=READY pgs=792 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:46.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:46.493 7f8b0c56c700 1 -- 172.21.2.112:0/2921559338 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8b041a5d40 msgr2=0x7f8b04107d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:46.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:46.493 7f8b0c56c700 1 --2- 172.21.2.112:0/2921559338 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8b041a5d40 0x7f8b04107d50 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 rx=0x7f8af400b530 tx=0x7f8af400b8d0).stop 2021-02-10T11:36:46.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:46.493 7f8b0c56c700 1 -- 172.21.2.112:0/2921559338 shutdown_connections 2021-02-10T11:36:46.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:46.493 7f8b0c56c700 1 --2- 172.21.2.112:0/2921559338 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8aec018730 0x7f8aec01ac10 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:46.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:46.493 7f8b0c56c700 1 --2- 172.21.2.112:0/2921559338 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8b04108370 0x7f8b041a6840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:46.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:46.493 7f8b0c56c700 1 --2- 172.21.2.112:0/2921559338 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8b041a5d40 0x7f8b04107d50 unknown :-1 s=CLOSED pgs=1660 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:46.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:46.493 7f8b0c56c700 1 --2- 172.21.2.112:0/2921559338 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8b041a5940 0x7f8b04107730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:46.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:46.493 7f8b0c56c700 1 -- 172.21.2.112:0/2921559338 >> 172.21.2.112:0/2921559338 conn(0x7f8b0419afc0 msgr2=0x7f8b040b3b70 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:46.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:46.493 7f8b0c56c700 1 -- 172.21.2.112:0/2921559338 shutdown_connections 2021-02-10T11:36:46.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:46.493 7f8b0c56c700 1 -- 172.21.2.112:0/2921559338 wait complete. 2021-02-10T11:36:46.488 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2021-02-10T11:36:46.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:46.917 7f2637136700 1 Processor -- start 2021-02-10T11:36:46.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:46.917 7f2637136700 1 -- start start 2021-02-10T11:36:46.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:46.918 7f2637136700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f263010b000 0x7f26300cf760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:46.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:46.918 7f2637136700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26300fe1d0 con 0x7f26300d1c80 2021-02-10T11:36:46.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:46.918 7f2637136700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26301b0cc0 con 0x7f26300d1880 2021-02-10T11:36:46.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:46.918 7f2637136700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2630109a70 con 0x7f263010b000 2021-02-10T11:36:46.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:46.918 7f2635753700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f263010b000 0x7f26300cf760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:46.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:46.918 7f2635753700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f263010b000 0x7f26300cf760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:52276/0 (socket says 172.21.2.112:52276) 2021-02-10T11:36:46.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:46.918 7f262ffff700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f26300d1c80 0x7f26300cef70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:51738/0 (socket says 172.21.2.112:51738) 2021-02-10T11:36:46.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:46.918 7f2635753700 1 -- 172.21.2.112:0/2655239379 learned_addr learned my addr 172.21.2.112:0/2655239379 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:36:46.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:46.918 7f2635753700 1 -- 172.21.2.112:0/2655239379 >> v1:172.21.2.120:6789/0 conn(0x7f26300d1880 legacy=0x7f26301b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:46.921 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:46.918 7f2635753700 1 -- 172.21.2.112:0/2655239379 >> v1:172.21.2.112:6789/0 conn(0x7f26300d1c80 legacy=0x7f26300cef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:46.921 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:46.918 7f2635753700 1 -- 172.21.2.112:0/2655239379 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f263010b400 con 0x7f263010b000 2021-02-10T11:36:46.921 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:46.918 7f2635753700 1 --2- 172.21.2.112:0/2655239379 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f263010b000 0x7f26300cf760 secure :-1 s=READY pgs=1662 cs=0 l=1 rev1=1 rx=0x7f2628006f00 tx=0x7f2628009f10).ready entity=mon.0 client_cookie=a683296f40207f13 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:46.921 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:46.918 7f262f7fe700 1 -- 172.21.2.112:0/2655239379 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2628003030 con 0x7f263010b000 2021-02-10T11:36:46.921 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:46.918 7f262f7fe700 1 -- 172.21.2.112:0/2655239379 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f262800b3b0 con 0x7f263010b000 2021-02-10T11:36:46.921 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:46.918 7f262f7fe700 1 -- 172.21.2.112:0/2655239379 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f262800b800 con 0x7f263010b000 2021-02-10T11:36:46.921 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:46.918 7f2637136700 1 -- 172.21.2.112:0/2655239379 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f263010b000 msgr2=0x7f26300cf760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:46.921 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:46.918 7f2637136700 1 --2- 172.21.2.112:0/2655239379 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f263010b000 0x7f26300cf760 secure :-1 s=READY pgs=1662 cs=0 l=1 rev1=1 rx=0x7f2628006f00 tx=0x7f2628009f10).stop 2021-02-10T11:36:46.921 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:46.919 7f2637136700 1 -- 172.21.2.112:0/2655239379 shutdown_connections 2021-02-10T11:36:46.921 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:46.919 7f2637136700 1 --2- 172.21.2.112:0/2655239379 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f263010b000 0x7f26300cf760 unknown :-1 s=CLOSED pgs=1662 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:46.921 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:46.919 7f2637136700 1 -- 172.21.2.112:0/2655239379 >> 172.21.2.112:0/2655239379 conn(0x7f26300ce430 msgr2=0x7f26300d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:46.921 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:46.919 7f2637136700 1 -- 172.21.2.112:0/2655239379 shutdown_connections 2021-02-10T11:36:46.921 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:46.919 7f2637136700 1 -- 172.21.2.112:0/2655239379 wait complete. 2021-02-10T11:36:46.921 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:46.933 7f2637136700 1 Processor -- start 2021-02-10T11:36:46.922 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:46.933 7f2637136700 1 -- start start 2021-02-10T11:36:46.923 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:46.934 7f2637136700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f26300956a0 0x7f2630094fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:46.924 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:46.934 7f2637136700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f26301b0200 0x7f26301b0600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:46.924 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:46.934 7f2637136700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2630106af0 0x7f2630106ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:46.924 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:46.934 7f2637136700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26301b0cb0 con 0x7f26301b0200 2021-02-10T11:36:46.924 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:46.934 7f2637136700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2630108400 con 0x7f2630106af0 2021-02-10T11:36:46.924 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:46.934 7f2637136700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2630107510 con 0x7f26300956a0 2021-02-10T11:36:46.924 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:46.934 7f262ffff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f26301b0200 0x7f26301b0600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:46.924 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:46.934 7f262ffff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f26301b0200 0x7f26301b0600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:52280/0 (socket says 172.21.2.112:52280) 2021-02-10T11:36:46.924 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:46.934 7f262ffff700 1 -- 172.21.2.112:0/300980808 learned_addr learned my addr 172.21.2.112:0/300980808 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:36:46.924 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:46.934 7f2635753700 1 --2- 172.21.2.112:0/300980808 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2630106af0 0x7f2630106ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:46.925 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:46.934 7f2634f52700 1 --2- 172.21.2.112:0/300980808 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f26300956a0 0x7f2630094fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:46.925 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:46.934 7f262ffff700 1 -- 172.21.2.112:0/300980808 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f26300956a0 msgr2=0x7f2630094fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:46.925 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:46.934 7f262ffff700 1 --2- 172.21.2.112:0/300980808 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f26300956a0 0x7f2630094fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:46.925 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:46.934 7f262ffff700 1 -- 172.21.2.112:0/300980808 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2630106af0 msgr2=0x7f2630106ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:46.925 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:46.934 7f262ffff700 1 --2- 172.21.2.112:0/300980808 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2630106af0 0x7f2630106ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:46.925 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:46.934 7f262ffff700 1 -- 172.21.2.112:0/300980808 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2630107730 con 0x7f26301b0200 2021-02-10T11:36:46.925 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:46.935 7f262ffff700 1 --2- 172.21.2.112:0/300980808 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f26301b0200 0x7f26301b0600 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 rx=0x7f26200014d0 tx=0x7f262000a8a0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:46.925 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:46.935 7f262d7fa700 1 -- 172.21.2.112:0/300980808 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2620000ef0 con 0x7f26301b0200 2021-02-10T11:36:46.925 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:46.935 7f262d7fa700 1 -- 172.21.2.112:0/300980808 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f262000ac90 con 0x7f26301b0200 2021-02-10T11:36:46.925 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:46.935 7f262d7fa700 1 -- 172.21.2.112:0/300980808 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2620004590 con 0x7f26301b0200 2021-02-10T11:36:46.925 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:46.935 7f2637136700 1 -- 172.21.2.112:0/300980808 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2630107730 con 0x7f26301b0200 2021-02-10T11:36:46.925 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:46.935 7f2637136700 1 -- 172.21.2.112:0/300980808 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26301b4760 con 0x7f26301b0200 2021-02-10T11:36:46.925 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:46.935 7f262d7fa700 1 -- 172.21.2.112:0/300980808 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f26200043a0 con 0x7f26301b0200 2021-02-10T11:36:46.925 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:46.935 7f262d7fa700 1 --2- 172.21.2.112:0/300980808 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2614018730 0x7f261401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:46.926 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:46.935 7f262d7fa700 1 -- 172.21.2.112:0/300980808 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5237+0+0 (secure 0 0 0) 0x7f2620032580 con 0x7f26301b0200 2021-02-10T11:36:46.926 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:46.936 7f2634f52700 1 --2- 172.21.2.112:0/300980808 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2614018730 0x7f261401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:46.926 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:46.936 7f2634f52700 1 --2- 172.21.2.112:0/300980808 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2614018730 0x7f261401abc0 crc :-1 s=READY pgs=793 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:46.953 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:46.967 7f2637136700 1 -- 172.21.2.112:0/300980808 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2630080340 con 0x7f26301b0200 2021-02-10T11:36:46.956 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:46.969 7f262d7fa700 1 -- 172.21.2.112:0/300980808 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2620000ef0 con 0x7f26301b0200 2021-02-10T11:36:47.084 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:47.097 7f2637136700 1 -- 172.21.2.112:0/300980808 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f2630088070 con 0x7f26301b0200 2021-02-10T11:36:47.441 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:47.455 7f262d7fa700 1 -- 172.21.2.112:0/300980808 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v78) v1 ==== 173+0+0 (secure 0 0 0) 0x7f26200043a0 con 0x7f26301b0200 2021-02-10T11:36:47.450 INFO:teuthology.orchestra.run.gibba012.stderr:pool 'cephfs_data' removed 2021-02-10T11:36:47.451 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:47.465 7f2637136700 1 -- 172.21.2.112:0/300980808 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2614018730 msgr2=0x7f261401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:47.452 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:47.465 7f2637136700 1 --2- 172.21.2.112:0/300980808 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2614018730 0x7f261401abc0 crc :-1 s=READY pgs=793 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:47.452 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:47.465 7f2637136700 1 -- 172.21.2.112:0/300980808 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f26301b0200 msgr2=0x7f26301b0600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:47.452 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:47.465 7f2637136700 1 --2- 172.21.2.112:0/300980808 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f26301b0200 0x7f26301b0600 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 rx=0x7f26200014d0 tx=0x7f262000a8a0).stop 2021-02-10T11:36:47.452 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:47.466 7f2637136700 1 -- 172.21.2.112:0/300980808 shutdown_connections 2021-02-10T11:36:47.452 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:47.466 7f2637136700 1 --2- 172.21.2.112:0/300980808 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2614018730 0x7f261401abc0 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:47.452 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:47.466 7f2637136700 1 --2- 172.21.2.112:0/300980808 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2630106af0 0x7f2630106ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:47.453 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:47.466 7f2637136700 1 --2- 172.21.2.112:0/300980808 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f26301b0200 0x7f26301b0600 unknown :-1 s=CLOSED pgs=1663 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:47.453 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:47.466 7f2637136700 1 --2- 172.21.2.112:0/300980808 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f26300956a0 0x7f2630094fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:47.453 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:47.466 7f2637136700 1 -- 172.21.2.112:0/300980808 >> 172.21.2.112:0/300980808 conn(0x7f263009fb30 msgr2=0x7f26301b5a60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:47.453 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:47.466 7f2637136700 1 -- 172.21.2.112:0/300980808 shutdown_connections 2021-02-10T11:36:47.453 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:47.466 7f2637136700 1 -- 172.21.2.112:0/300980808 wait complete. 2021-02-10T11:36:47.461 INFO:tasks.ceph.mds.a:Restarting daemon 2021-02-10T11:36:47.461 INFO:tasks.ceph.mds.a:Stopping old one... 2021-02-10T11:36:47.461 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2021-02-10T11:36:47.461 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:36:47.461 INFO:tasks.ceph.mds.b:Restarting daemon 2021-02-10T11:36:47.461 INFO:tasks.ceph.mds.b:Stopping old one... 2021-02-10T11:36:47.462 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2021-02-10T11:36:47.462 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:36:47.467 INFO:tasks.ceph.mds.a:Stopped 2021-02-10T11:36:47.467 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2021-02-10T11:36:47.468 INFO:tasks.ceph.mds.b:Stopped 2021-02-10T11:36:47.468 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2021-02-10T11:36:47.510 INFO:tasks.ceph.mds.a:Started 2021-02-10T11:36:47.510 DEBUG:teuthology.parallel:result is None 2021-02-10T11:36:47.511 INFO:tasks.ceph.mds.b:Started 2021-02-10T11:36:47.511 DEBUG:teuthology.parallel:result is None 2021-02-10T11:36:47.511 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd blacklist clear 2021-02-10T11:36:47.913 INFO:tasks.ceph.mds.b.gibba020.stdout:starting mds.b at 2021-02-10T11:36:47.915 INFO:tasks.ceph.mds.a.gibba012.stdout:starting mds.a at 2021-02-10T11:36:47.935 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:47.932 7f280d044700 1 Processor -- start 2021-02-10T11:36:47.935 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:47.932 7f280d044700 1 -- start start 2021-02-10T11:36:47.935 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:47.932 7f280d044700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f28080d0340 0x7f28080ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:47.935 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:47.932 7f280d044700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f28080d1c70 0x7f28080efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:47.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:47.932 7f280d044700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28080f0b10 con 0x7f28080d1870 2021-02-10T11:36:47.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:47.932 7f280d044700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28080e5ed0 con 0x7f28080d1c70 2021-02-10T11:36:47.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:47.932 7f280d044700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28081b0be0 con 0x7f28080d0340 2021-02-10T11:36:47.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:47.933 7f280759e700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f28080d1c70 0x7f28080efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:47.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:47.933 7f280759e700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f28080d1c70 0x7f28080efc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:52302/0 (socket says 172.21.2.112:52302) 2021-02-10T11:36:47.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:47.933 7f280759e700 1 -- 172.21.2.112:0/3485785252 learned_addr learned my addr 172.21.2.112:0/3485785252 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:36:47.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:47.933 7f280659c700 1 --2- 172.21.2.112:0/3485785252 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f28080d0340 0x7f28080ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:47.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:47.933 7f280759e700 1 -- 172.21.2.112:0/3485785252 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f28080d0340 msgr2=0x7f28080ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:47.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:47.933 7f280759e700 1 --2- 172.21.2.112:0/3485785252 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f28080d0340 0x7f28080ef640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:47.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:47.933 7f280759e700 1 -- 172.21.2.112:0/3485785252 >> v1:172.21.2.112:6789/0 conn(0x7f28080d1870 legacy=0x7f28081b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:47.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:47.933 7f280759e700 1 -- 172.21.2.112:0/3485785252 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28080f20a0 con 0x7f28080d1c70 2021-02-10T11:36:47.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:47.933 7f280659c700 1 --2- 172.21.2.112:0/3485785252 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f28080d0340 0x7f28080ef640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:36:47.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:47.933 7f280759e700 1 --2- 172.21.2.112:0/3485785252 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f28080d1c70 0x7f28080efc60 secure :-1 s=READY pgs=1669 cs=0 l=1 rev1=1 rx=0x7f27f8006b80 tx=0x7f27f8009b90).ready entity=mon.0 client_cookie=be4b00f1632caab1 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:47.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:47.933 7f2805d9b700 1 -- 172.21.2.112:0/3485785252 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f27f800c0c0 con 0x7f28080d1c70 2021-02-10T11:36:47.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:47.933 7f2805d9b700 1 -- 172.21.2.112:0/3485785252 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27f800c380 con 0x7f28080d1c70 2021-02-10T11:36:47.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:47.933 7f2805d9b700 1 -- 172.21.2.112:0/3485785252 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f27f800c8a0 con 0x7f28080d1c70 2021-02-10T11:36:47.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:47.933 7f280d044700 1 -- 172.21.2.112:0/3485785252 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f28080d1c70 msgr2=0x7f28080efc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:47.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:47.933 7f280d044700 1 --2- 172.21.2.112:0/3485785252 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f28080d1c70 0x7f28080efc60 secure :-1 s=READY pgs=1669 cs=0 l=1 rev1=1 rx=0x7f27f8006b80 tx=0x7f27f8009b90).stop 2021-02-10T11:36:47.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:47.934 7f280d044700 1 -- 172.21.2.112:0/3485785252 shutdown_connections 2021-02-10T11:36:47.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:47.934 7f280d044700 1 --2- 172.21.2.112:0/3485785252 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f28080d1c70 0x7f28080efc60 unknown :-1 s=CLOSED pgs=1669 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:47.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:47.934 7f280d044700 1 --2- 172.21.2.112:0/3485785252 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f28080d0340 0x7f28080ef640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:47.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:47.934 7f280d044700 1 -- 172.21.2.112:0/3485785252 >> 172.21.2.112:0/3485785252 conn(0x7f28080ce450 msgr2=0x7f28080d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:47.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:47.934 7f280d044700 1 -- 172.21.2.112:0/3485785252 shutdown_connections 2021-02-10T11:36:47.938 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:47.934 7f280d044700 1 -- 172.21.2.112:0/3485785252 wait complete. 2021-02-10T11:36:47.939 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:47.949 7f280d044700 1 Processor -- start 2021-02-10T11:36:47.939 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:47.949 7f280d044700 1 -- start start 2021-02-10T11:36:47.939 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:47.949 7f280d044700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f28080948c0 0x7f2808094d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:47.939 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:47.949 7f280d044700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2808093d40 0x7f2808094140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:47.939 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:47.949 7f280d044700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f28080bfc70 0x7f28080c0070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:47.940 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:47.949 7f280d044700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f280808ded0 con 0x7f2808093d40 2021-02-10T11:36:47.940 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:47.949 7f280d044700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28080953b0 con 0x7f28080bfc70 2021-02-10T11:36:47.940 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:47.949 7f280d044700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28080c0690 con 0x7f28080948c0 2021-02-10T11:36:47.940 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:47.949 7f280659c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2808093d40 0x7f2808094140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:47.940 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:47.949 7f280659c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2808093d40 0x7f2808094140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:52306/0 (socket says 172.21.2.112:52306) 2021-02-10T11:36:47.940 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:47.949 7f280659c700 1 -- 172.21.2.112:0/3997928617 learned_addr learned my addr 172.21.2.112:0/3997928617 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:36:47.940 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:47.950 7f280659c700 1 -- 172.21.2.112:0/3997928617 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f28080948c0 msgr2=0x7f2808094d90 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:36:47.940 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:47.950 7f280659c700 1 --2- 172.21.2.112:0/3997928617 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f28080948c0 0x7f2808094d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:47.940 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:47.950 7f280659c700 1 -- 172.21.2.112:0/3997928617 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f28080bfc70 msgr2=0x7f28080c0070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:47.940 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:47.950 7f280659c700 1 --2- 172.21.2.112:0/3997928617 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f28080bfc70 0x7f28080c0070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:47.940 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:47.950 7f280659c700 1 -- 172.21.2.112:0/3997928617 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28080c08b0 con 0x7f2808093d40 2021-02-10T11:36:47.940 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:47.950 7f280659c700 1 --2- 172.21.2.112:0/3997928617 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2808093d40 0x7f2808094140 secure :-1 s=READY pgs=1670 cs=0 l=1 rev1=1 rx=0x7f28000099a0 tx=0x7f280000bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:47.940 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:47.950 7f27f77fe700 1 -- 172.21.2.112:0/3997928617 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f280000c2f0 con 0x7f2808093d40 2021-02-10T11:36:47.941 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:47.950 7f27f77fe700 1 -- 172.21.2.112:0/3997928617 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f280000c630 con 0x7f2808093d40 2021-02-10T11:36:47.941 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:47.950 7f27f77fe700 1 -- 172.21.2.112:0/3997928617 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f280000cb10 con 0x7f2808093d40 2021-02-10T11:36:47.941 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:47.950 7f280d044700 1 -- 172.21.2.112:0/3997928617 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28080c08b0 con 0x7f2808093d40 2021-02-10T11:36:47.941 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:47.950 7f280d044700 1 -- 172.21.2.112:0/3997928617 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28080c08b0 con 0x7f2808093d40 2021-02-10T11:36:47.941 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:47.950 7f27f77fe700 1 -- 172.21.2.112:0/3997928617 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f280000a000 con 0x7f2808093d40 2021-02-10T11:36:47.941 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:47.950 7f27f77fe700 1 --2- 172.21.2.112:0/3997928617 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f27ec018730 0x7f27ec01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:47.941 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:47.950 7f27f77fe700 1 -- 172.21.2.112:0/3997928617 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 4839+0+0 (secure 0 0 0) 0x7f28000326b0 con 0x7f2808093d40 2021-02-10T11:36:47.941 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:47.951 7f2806d9d700 1 --2- 172.21.2.112:0/3997928617 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f27ec018730 0x7f27ec01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:47.941 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:47.951 7f2806d9d700 1 --2- 172.21.2.112:0/3997928617 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f27ec018730 0x7f27ec01abc0 crc :-1 s=READY pgs=794 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:47.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:47.982 7f280d044700 1 -- 172.21.2.112:0/3997928617 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28080ebc90 con 0x7f2808093d40 2021-02-10T11:36:47.971 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:47.984 7f27f77fe700 1 -- 172.21.2.112:0/3997928617 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f280000be30 con 0x7f2808093d40 2021-02-10T11:36:48.111 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:48.125 7f280d044700 1 -- 172.21.2.112:0/3997928617 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7f28080f0b10 con 0x7f2808093d40 2021-02-10T11:36:48.447 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:48.460 7f27f77fe700 1 -- 172.21.2.112:0/3997928617 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v79) v1 ==== 97+0+0 (secure 0 0 0) 0x7f280000a000 con 0x7f2808093d40 2021-02-10T11:36:48.478 INFO:teuthology.orchestra.run.gibba012.stderr: removed all blacklist entries 2021-02-10T11:36:48.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:48.493 7f280d044700 1 -- 172.21.2.112:0/3997928617 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f27ec018730 msgr2=0x7f27ec01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:48.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:48.493 7f280d044700 1 --2- 172.21.2.112:0/3997928617 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f27ec018730 0x7f27ec01abc0 crc :-1 s=READY pgs=794 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:48.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:48.493 7f280d044700 1 -- 172.21.2.112:0/3997928617 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2808093d40 msgr2=0x7f2808094140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:48.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:48.493 7f280d044700 1 --2- 172.21.2.112:0/3997928617 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2808093d40 0x7f2808094140 secure :-1 s=READY pgs=1670 cs=0 l=1 rev1=1 rx=0x7f28000099a0 tx=0x7f280000bcf0).stop 2021-02-10T11:36:48.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:48.493 7f280d044700 1 -- 172.21.2.112:0/3997928617 shutdown_connections 2021-02-10T11:36:48.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:48.493 7f280d044700 1 --2- 172.21.2.112:0/3997928617 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f27ec018730 0x7f27ec01abc0 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:48.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:48.493 7f280d044700 1 --2- 172.21.2.112:0/3997928617 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f28080bfc70 0x7f28080c0070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:48.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:48.493 7f280d044700 1 --2- 172.21.2.112:0/3997928617 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2808093d40 0x7f2808094140 unknown :-1 s=CLOSED pgs=1670 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:48.481 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:48.493 7f280d044700 1 --2- 172.21.2.112:0/3997928617 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f28080948c0 0x7f2808094d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:48.481 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:48.493 7f280d044700 1 -- 172.21.2.112:0/3997928617 >> 172.21.2.112:0/3997928617 conn(0x7f28080bf640 msgr2=0x7f280809f200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:48.481 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:48.493 7f280d044700 1 -- 172.21.2.112:0/3997928617 shutdown_connections 2021-02-10T11:36:48.481 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:48.493 7f280d044700 1 -- 172.21.2.112:0/3997928617 wait complete. 2021-02-10T11:36:48.488 INFO:tasks.cephfs.cephfs_test_case:['0', '1', '2', '3'] 2021-02-10T11:36:48.489 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth ls --format=json-pretty 2021-02-10T11:36:48.914 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:48.911 7ff9f3635700 1 Processor -- start 2021-02-10T11:36:48.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:48.911 7ff9f3635700 1 -- start start 2021-02-10T11:36:48.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:48.912 7ff9f3635700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff9ec0d1870 0x7ff9ec1b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:48.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:48.912 7ff9f3635700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7ff9ec0f0650 0x7ff9ec0cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:48.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:48.912 7ff9f3635700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9ec0f0b10 con 0x7ff9ec1b0a10 2021-02-10T11:36:48.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:48.912 7ff9f3635700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9ec0fe230 con 0x7ff9ec0f0650 2021-02-10T11:36:48.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:48.912 7ff9f3635700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9ec0e5ed0 con 0x7ff9ec0d1870 2021-02-10T11:36:48.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:48.912 7ff9f1c52700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7ff9ec0f0650 0x7ff9ec0cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:48.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:48.912 7ff9f1c52700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7ff9ec0f0650 0x7ff9ec0cfb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:52314/0 (socket says 172.21.2.112:52314) 2021-02-10T11:36:48.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:48.912 7ff9f1c52700 1 -- 172.21.2.112:0/1744832000 learned_addr learned my addr 172.21.2.112:0/1744832000 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:36:48.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:48.912 7ff9f1c52700 1 -- 172.21.2.112:0/1744832000 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff9ec0d1870 msgr2=0x7ff9ec1b03f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:36:48.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:48.912 7ff9f1c52700 1 --2- 172.21.2.112:0/1744832000 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff9ec0d1870 0x7ff9ec1b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:48.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:48.912 7ff9f1c52700 1 -- 172.21.2.112:0/1744832000 >> v1:172.21.2.120:6789/0 conn(0x7ff9ec1b0a10 legacy=0x7ff9ec0cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:48.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:48.912 7ff9f1c52700 1 -- 172.21.2.112:0/1744832000 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9ec0f20a0 con 0x7ff9ec0f0650 2021-02-10T11:36:48.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:48.912 7ff9f1c52700 1 --2- 172.21.2.112:0/1744832000 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff9ec0f0650 0x7ff9ec0cfb20 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 rx=0x7ff9e4006f00 tx=0x7ff9e4009e90).ready entity=mon.0 client_cookie=16f7c3b7ecc61466 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:48.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:48.913 7ff9dffff700 1 -- 172.21.2.112:0/1744832000 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff9e400b300 con 0x7ff9ec0f0650 2021-02-10T11:36:48.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:48.913 7ff9dffff700 1 -- 172.21.2.112:0/1744832000 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9e400b580 con 0x7ff9ec0f0650 2021-02-10T11:36:48.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:48.913 7ff9dffff700 1 -- 172.21.2.112:0/1744832000 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff9e400b9d0 con 0x7ff9ec0f0650 2021-02-10T11:36:48.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:48.913 7ff9f3635700 1 -- 172.21.2.112:0/1744832000 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff9ec0f0650 msgr2=0x7ff9ec0cfb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:48.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:48.913 7ff9f3635700 1 --2- 172.21.2.112:0/1744832000 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff9ec0f0650 0x7ff9ec0cfb20 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 rx=0x7ff9e4006f00 tx=0x7ff9e4009e90).stop 2021-02-10T11:36:48.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:48.913 7ff9f3635700 1 -- 172.21.2.112:0/1744832000 shutdown_connections 2021-02-10T11:36:48.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:48.913 7ff9f3635700 1 --2- 172.21.2.112:0/1744832000 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff9ec0f0650 0x7ff9ec0cfb20 unknown :-1 s=CLOSED pgs=1671 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:48.917 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:48.913 7ff9f3635700 1 --2- 172.21.2.112:0/1744832000 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff9ec0d1870 0x7ff9ec1b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:48.917 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:48.913 7ff9f3635700 1 -- 172.21.2.112:0/1744832000 >> 172.21.2.112:0/1744832000 conn(0x7ff9ec0ce450 msgr2=0x7ff9ec0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:48.917 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:48.914 7ff9f3635700 1 -- 172.21.2.112:0/1744832000 shutdown_connections 2021-02-10T11:36:48.917 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:48.914 7ff9f3635700 1 -- 172.21.2.112:0/1744832000 wait complete. 2021-02-10T11:36:48.917 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:48.928 7ff9f3635700 1 Processor -- start 2021-02-10T11:36:48.917 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:48.928 7ff9f3635700 1 -- start start 2021-02-10T11:36:48.918 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:48.928 7ff9f3635700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff9ec099d50 0x7ff9ec1b31b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:48.918 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:48.928 7ff9f3635700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff9ec187e50 0x7ff9ec188250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:48.918 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:48.928 7ff9f3635700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff9ec1b12a0 0x7ff9ec1b16a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:48.918 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:48.928 7ff9f3635700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9ec188870 con 0x7ff9ec187e50 2021-02-10T11:36:48.918 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:48.928 7ff9f3635700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9ec1b37d0 con 0x7ff9ec099d50 2021-02-10T11:36:48.919 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:48.928 7ff9f3635700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9ec1b1cc0 con 0x7ff9ec1b12a0 2021-02-10T11:36:48.919 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:48.929 7ff9f0c50700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff9ec187e50 0x7ff9ec188250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:48.919 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:48.929 7ff9f0c50700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff9ec187e50 0x7ff9ec188250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:52318/0 (socket says 172.21.2.112:52318) 2021-02-10T11:36:48.919 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:48.929 7ff9f0c50700 1 -- 172.21.2.112:0/2056708895 learned_addr learned my addr 172.21.2.112:0/2056708895 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:36:48.919 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:48.929 7ff9f0c50700 1 -- 172.21.2.112:0/2056708895 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff9ec1b12a0 msgr2=0x7ff9ec1b16a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:36:48.919 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:48.929 7ff9f0c50700 1 --2- 172.21.2.112:0/2056708895 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff9ec1b12a0 0x7ff9ec1b16a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:48.919 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:48.929 7ff9f0c50700 1 -- 172.21.2.112:0/2056708895 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff9ec099d50 msgr2=0x7ff9ec1b31b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:36:48.919 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:48.929 7ff9f0c50700 1 --2- 172.21.2.112:0/2056708895 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff9ec099d50 0x7ff9ec1b31b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:48.919 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:48.929 7ff9f0c50700 1 -- 172.21.2.112:0/2056708895 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9ec1b1ee0 con 0x7ff9ec187e50 2021-02-10T11:36:48.919 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:48.929 7ff9f0c50700 1 --2- 172.21.2.112:0/2056708895 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff9ec187e50 0x7ff9ec188250 secure :-1 s=READY pgs=1672 cs=0 l=1 rev1=1 rx=0x7ff9e800b5b0 tx=0x7ff9e800b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:48.919 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:48.929 7ff9ddffb700 1 -- 172.21.2.112:0/2056708895 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff9e80009b0 con 0x7ff9ec187e50 2021-02-10T11:36:48.919 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:48.929 7ff9ddffb700 1 -- 172.21.2.112:0/2056708895 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9e800bd40 con 0x7ff9ec187e50 2021-02-10T11:36:48.919 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:48.929 7ff9ddffb700 1 -- 172.21.2.112:0/2056708895 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff9e80062a0 con 0x7ff9ec187e50 2021-02-10T11:36:48.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:48.929 7ff9f3635700 1 -- 172.21.2.112:0/2056708895 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9ec1b1ee0 con 0x7ff9ec187e50 2021-02-10T11:36:48.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:48.929 7ff9f3635700 1 -- 172.21.2.112:0/2056708895 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9ec1b2470 con 0x7ff9ec187e50 2021-02-10T11:36:48.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:48.930 7ff9ddffb700 1 -- 172.21.2.112:0/2056708895 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff9e8006020 con 0x7ff9ec187e50 2021-02-10T11:36:48.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:48.930 7ff9ddffb700 1 --2- 172.21.2.112:0/2056708895 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff9cc0186c0 0x7ff9cc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:48.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:48.930 7ff9ddffb700 1 -- 172.21.2.112:0/2056708895 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 4753+0+0 (secure 0 0 0) 0x7ff9e8032240 con 0x7ff9ec187e50 2021-02-10T11:36:48.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:48.930 7ff9f1451700 1 --2- 172.21.2.112:0/2056708895 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff9cc0186c0 0x7ff9cc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:48.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:48.931 7ff9f1451700 1 --2- 172.21.2.112:0/2056708895 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff9cc0186c0 0x7ff9cc01abc0 crc :-1 s=READY pgs=795 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:48.948 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:48.962 7ff9f3635700 1 -- 172.21.2.112:0/2056708895 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9ec09a1e0 con 0x7ff9ec187e50 2021-02-10T11:36:48.950 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:48.964 7ff9ddffb700 1 -- 172.21.2.112:0/2056708895 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff9e80009b0 con 0x7ff9ec187e50 2021-02-10T11:36:48.967 DEBUG:teuthology.orchestra.run.gibba012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:36:48.970 DEBUG:teuthology.orchestra.run.gibba020:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:36:49.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:49.099 7ff9f3635700 1 -- 172.21.2.112:0/2056708895 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7ff9ec09a1e0 con 0x7ff9ec187e50 2021-02-10T11:36:49.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:49.100 7ff9ddffb700 1 -- 172.21.2.112:0/2056708895 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v48) v1 ==== 80+0+4513 (secure 0 0 0) 0x7ff9e8006000 con 0x7ff9ec187e50 2021-02-10T11:36:49.087 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:36:49.087 INFO:teuthology.orchestra.run.gibba012.stdout:{ 2021-02-10T11:36:49.087 INFO:teuthology.orchestra.run.gibba012.stdout: "auth_dump": [ 2021-02-10T11:36:49.087 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:36:49.087 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "mds.a", 2021-02-10T11:36:49.087 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCKvyNgPkV4AxAAEmVNaghdccP6lWd8hvQnxw==", 2021-02-10T11:36:49.087 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:36:49.087 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:36:49.087 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:36:49.087 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:36:49.087 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:36:49.088 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:36:49.088 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:36:49.088 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:36:49.088 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "mds.b", 2021-02-10T11:36:49.088 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCKvyNguxONLhAAHIpry+R89MAwdhFJyEhVCw==", 2021-02-10T11:36:49.088 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:36:49.088 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:36:49.088 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:36:49.088 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:36:49.088 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:36:49.088 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:36:49.088 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:36:49.088 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:36:49.089 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.0", 2021-02-10T11:36:49.089 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCSvyNgXcKPLhAAP/Bj2F+06DCtPCzQDNq+cQ==", 2021-02-10T11:36:49.089 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:36:49.089 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:36:49.089 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:36:49.089 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:36:49.089 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:36:49.089 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:36:49.089 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:36:49.089 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.1", 2021-02-10T11:36:49.089 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCVvyNghfbcLhAA66PrRwR2M7o6VTF5UhU/Fg==", 2021-02-10T11:36:49.089 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:36:49.090 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:36:49.090 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:36:49.090 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:36:49.090 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:36:49.090 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:36:49.090 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:36:49.090 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.2", 2021-02-10T11:36:49.090 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCYvyNgwiPqLhAA66BVkRsgaUlGpAsQ15Nf8Q==", 2021-02-10T11:36:49.090 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:36:49.090 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:36:49.090 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:36:49.090 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:36:49.091 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:36:49.091 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:36:49.091 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:36:49.091 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.3", 2021-02-10T11:36:49.091 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCbvyNgqBZrLxAAERPe/1L3SXAEzI3d9ReHHg==", 2021-02-10T11:36:49.091 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:36:49.091 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:36:49.091 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:36:49.091 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:36:49.091 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:36:49.091 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:36:49.091 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:36:49.092 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.4", 2021-02-10T11:36:49.092 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCkvyNgDQsxIhAAHmVjVYnx20N4SDmKqvbcuA==", 2021-02-10T11:36:49.092 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:36:49.092 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:36:49.092 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:36:49.092 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:36:49.092 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:36:49.092 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:36:49.092 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:36:49.092 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.5", 2021-02-10T11:36:49.092 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCnvyNg8ITFIhAADd4aKybAV7btoGPx3kCg3A==", 2021-02-10T11:36:49.093 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:36:49.093 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:36:49.093 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:36:49.093 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:36:49.093 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:36:49.093 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:36:49.093 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:36:49.093 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.6", 2021-02-10T11:36:49.093 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCqvyNgxOhDIxAAzXixEa/nlDc1xXrg0aeLwg==", 2021-02-10T11:36:49.093 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:36:49.093 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:36:49.094 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:36:49.094 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:36:49.094 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:36:49.094 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:36:49.094 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:36:49.094 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.7", 2021-02-10T11:36:49.094 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCtvyNgxSqGIxAACxJQlwP7DpiVVCZHA3zn1A==", 2021-02-10T11:36:49.095 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:36:49.095 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:36:49.095 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:36:49.095 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:36:49.095 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:36:49.095 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:36:49.095 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:36:49.095 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.0", 2021-02-10T11:36:49.095 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCLvyNgDkNiHRAAygU8Za4PBusTqCgwAYWTiQ==", 2021-02-10T11:36:49.095 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:36:49.095 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:36:49.095 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow r", 2021-02-10T11:36:49.095 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:36:49.096 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:36:49.096 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:36:49.096 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:36:49.096 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.1", 2021-02-10T11:36:49.096 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCLvyNgxmLqMxAA1MlSDJ9LH9u4f2sL7mm8/Q==", 2021-02-10T11:36:49.097 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:36:49.097 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:36:49.097 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow r", 2021-02-10T11:36:49.097 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:36:49.097 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:36:49.097 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:36:49.097 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:36:49.097 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.2", 2021-02-10T11:36:49.097 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCMvyNgp8veDhAAniSVHYPtwwTog/pjbaRQmw==", 2021-02-10T11:36:49.097 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:36:49.097 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:36:49.097 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow r", 2021-02-10T11:36:49.098 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:36:49.098 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:36:49.098 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:36:49.098 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:36:49.098 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.3", 2021-02-10T11:36:49.098 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCMvyNgQEVoJRAApHcWdm1sldIxu+heyD4HdQ==", 2021-02-10T11:36:49.098 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:36:49.098 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:36:49.098 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow r", 2021-02-10T11:36:49.099 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:36:49.099 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:36:49.099 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:36:49.099 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:36:49.099 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.admin", 2021-02-10T11:36:49.099 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCGvyNg7jjJBRAAtot+ypUZOiTo1GbY+oVhVw==", 2021-02-10T11:36:49.099 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:36:49.099 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow *", 2021-02-10T11:36:49.099 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:36:49.099 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:36:49.100 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:36:49.100 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:36:49.100 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:36:49.100 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:36:49.100 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "mgr.x", 2021-02-10T11:36:49.100 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCJvyNgTPVwKBAAjduiBE8+RSXBv82tdPecWg==", 2021-02-10T11:36:49.100 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:36:49.100 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow *", 2021-02-10T11:36:49.100 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow profile mgr", 2021-02-10T11:36:49.100 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:36:49.100 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:36:49.100 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:36:49.101 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:36:49.101 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "mgr.y", 2021-02-10T11:36:49.101 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCJvyNgQ1tTDhAASwBssDy+Rz31ujw8fNlWow==", 2021-02-10T11:36:49.101 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:36:49.101 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow *", 2021-02-10T11:36:49.101 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow profile mgr", 2021-02-10T11:36:49.101 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:36:49.101 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:36:49.101 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:36:49.101 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:36:49.101 INFO:teuthology.orchestra.run.gibba012.stdout:} 2021-02-10T11:36:49.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:49.101 7ff9f3635700 1 -- 172.21.2.112:0/2056708895 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff9cc0186c0 msgr2=0x7ff9cc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:49.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:49.101 7ff9f3635700 1 --2- 172.21.2.112:0/2056708895 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff9cc0186c0 0x7ff9cc01abc0 crc :-1 s=READY pgs=795 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:49.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:49.101 7ff9f3635700 1 -- 172.21.2.112:0/2056708895 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff9ec187e50 msgr2=0x7ff9ec188250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:49.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:49.102 7ff9f3635700 1 --2- 172.21.2.112:0/2056708895 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff9ec187e50 0x7ff9ec188250 secure :-1 s=READY pgs=1672 cs=0 l=1 rev1=1 rx=0x7ff9e800b5b0 tx=0x7ff9e800b950).stop 2021-02-10T11:36:49.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:49.102 7ff9f3635700 1 -- 172.21.2.112:0/2056708895 shutdown_connections 2021-02-10T11:36:49.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:49.102 7ff9f3635700 1 --2- 172.21.2.112:0/2056708895 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff9cc0186c0 0x7ff9cc01abc0 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:49.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:49.102 7ff9f3635700 1 --2- 172.21.2.112:0/2056708895 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff9ec1b12a0 0x7ff9ec1b16a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:49.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:49.102 7ff9f3635700 1 --2- 172.21.2.112:0/2056708895 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff9ec187e50 0x7ff9ec188250 unknown :-1 s=CLOSED pgs=1672 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:49.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:49.102 7ff9f3635700 1 --2- 172.21.2.112:0/2056708895 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff9ec099d50 0x7ff9ec1b31b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:49.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:49.102 7ff9f3635700 1 -- 172.21.2.112:0/2056708895 >> 172.21.2.112:0/2056708895 conn(0x7ff9ec19f8c0 msgr2=0x7ff9ec1a5fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:49.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:49.102 7ff9f3635700 1 -- 172.21.2.112:0/2056708895 shutdown_connections 2021-02-10T11:36:49.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:49.102 7ff9f3635700 1 -- 172.21.2.112:0/2056708895 wait complete. 2021-02-10T11:36:49.105 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:36:49.531 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:49.529 7fb99d9a2700 1 Processor -- start 2021-02-10T11:36:49.532 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:49.529 7fb99d9a2700 1 -- start start 2021-02-10T11:36:49.532 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:49.529 7fb99d9a2700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb9980d0340 0x7fb9980ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:49.532 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:49.529 7fb99d9a2700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9980f0b10 con 0x7fb9980d1c70 2021-02-10T11:36:49.532 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:49.529 7fb99d9a2700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9981b0be0 con 0x7fb9980d1870 2021-02-10T11:36:49.532 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:49.529 7fb99d9a2700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9980e5ed0 con 0x7fb9980d0340 2021-02-10T11:36:49.532 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:49.529 7fb997d75700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fb9980d1c70 0x7fb9980efc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:51792/0 (socket says 172.21.2.112:51792) 2021-02-10T11:36:49.532 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:49.529 7fb997d75700 1 -- 172.21.2.112:0/1615335900 learned_addr learned my addr 172.21.2.112:0/1615335900 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:36:49.532 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:49.529 7fb996d73700 1 --2- 172.21.2.112:0/1615335900 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb9980d0340 0x7fb9980ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:49.532 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:49.530 7fb996572700 1 -- 172.21.2.112:0/1615335900 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1838504996 0 0) 0x7fb990001500 con 0x7fb9980d1c70 2021-02-10T11:36:49.532 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:49.530 7fb996572700 1 -- 172.21.2.112:0/1615335900 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb984002fe0 con 0x7fb9980d1c70 2021-02-10T11:36:49.532 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:49.530 7fb996572700 1 -- 172.21.2.112:0/1615335900 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3877712903 0 0) 0x7fb980001080 con 0x7fb9980d1870 2021-02-10T11:36:49.533 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:49.530 7fb996572700 1 -- 172.21.2.112:0/1615335900 --> v1:172.21.2.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9840042c0 con 0x7fb9980d1870 2021-02-10T11:36:49.533 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:49.530 7fb996d73700 1 -- 172.21.2.112:0/1615335900 >> v1:172.21.2.120:6789/0 conn(0x7fb9980d1870 legacy=0x7fb9981b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:49.533 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:49.530 7fb996d73700 1 -- 172.21.2.112:0/1615335900 >> v1:172.21.2.112:6789/0 conn(0x7fb9980d1c70 legacy=0x7fb9980efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:49.533 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:49.530 7fb996d73700 1 -- 172.21.2.112:0/1615335900 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9980f0450 con 0x7fb9980d0340 2021-02-10T11:36:49.533 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:49.530 7fb996d73700 1 --2- 172.21.2.112:0/1615335900 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb9980d0340 0x7fb9980ef640 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7fb988006650 tx=0x7fb988009a80).ready entity=mon.2 client_cookie=abd42cdf3e4dd2e7 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:49.533 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:49.530 7fb996572700 1 -- 172.21.2.112:0/1615335900 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb98800c0c0 con 0x7fb9980d0340 2021-02-10T11:36:49.533 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:49.530 7fb996572700 1 -- 172.21.2.112:0/1615335900 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb98800c340 con 0x7fb9980d0340 2021-02-10T11:36:49.533 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:49.530 7fb996572700 1 -- 172.21.2.112:0/1615335900 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb98800c7a0 con 0x7fb9980d0340 2021-02-10T11:36:49.533 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:49.530 7fb99d9a2700 1 -- 172.21.2.112:0/1615335900 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb9980d0340 msgr2=0x7fb9980ef640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:49.533 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:49.530 7fb99d9a2700 1 --2- 172.21.2.112:0/1615335900 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb9980d0340 0x7fb9980ef640 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7fb988006650 tx=0x7fb988009a80).stop 2021-02-10T11:36:49.533 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:49.531 7fb99d9a2700 1 -- 172.21.2.112:0/1615335900 shutdown_connections 2021-02-10T11:36:49.533 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:49.531 7fb99d9a2700 1 --2- 172.21.2.112:0/1615335900 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb9980d0340 0x7fb9980ef640 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:49.533 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:49.531 7fb99d9a2700 1 -- 172.21.2.112:0/1615335900 >> 172.21.2.112:0/1615335900 conn(0x7fb9980ce450 msgr2=0x7fb9980d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:49.534 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:49.531 7fb99d9a2700 1 -- 172.21.2.112:0/1615335900 shutdown_connections 2021-02-10T11:36:49.534 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:49.531 7fb99d9a2700 1 -- 172.21.2.112:0/1615335900 wait complete. 2021-02-10T11:36:49.535 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:49.545 7fb99d9a2700 1 Processor -- start 2021-02-10T11:36:49.535 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:49.546 7fb99d9a2700 1 -- start start 2021-02-10T11:36:49.535 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:49.546 7fb99d9a2700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb9980ae880 0x7fb9981ae640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:49.535 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:49.546 7fb99d9a2700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb9980aec80 0x7fb9980ed0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:49.535 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:49.546 7fb99d9a2700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb9980ed710 0x7fb9981af450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:49.536 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:49.546 7fb99d9a2700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9981aeed0 con 0x7fb9980ae880 2021-02-10T11:36:49.536 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:49.546 7fb99d9a2700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9981ac200 con 0x7fb9980ed710 2021-02-10T11:36:49.536 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:49.546 7fb99d9a2700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9981b08a0 con 0x7fb9980aec80 2021-02-10T11:36:49.536 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:49.546 7fb997574700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb9980ae880 0x7fb9981ae640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:49.536 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:49.546 7fb997574700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb9980ae880 0x7fb9981ae640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:52334/0 (socket says 172.21.2.112:52334) 2021-02-10T11:36:49.536 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:49.546 7fb997574700 1 -- 172.21.2.112:0/1372396770 learned_addr learned my addr 172.21.2.112:0/1372396770 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:36:49.536 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:49.546 7fb996d73700 1 --2- 172.21.2.112:0/1372396770 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb9980aec80 0x7fb9980ed0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:49.536 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:49.546 7fb997d75700 1 --2- 172.21.2.112:0/1372396770 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb9980ed710 0x7fb9981af450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:49.536 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:49.547 7fb997574700 1 -- 172.21.2.112:0/1372396770 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb9980aec80 msgr2=0x7fb9980ed0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:49.536 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:49.547 7fb997574700 1 --2- 172.21.2.112:0/1372396770 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb9980aec80 0x7fb9980ed0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:49.536 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:49.547 7fb997574700 1 -- 172.21.2.112:0/1372396770 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb9980ed710 msgr2=0x7fb9981af450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:49.536 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:49.547 7fb997574700 1 --2- 172.21.2.112:0/1372396770 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb9980ed710 0x7fb9981af450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:49.536 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:49.547 7fb997574700 1 -- 172.21.2.112:0/1372396770 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9981afa70 con 0x7fb9980ae880 2021-02-10T11:36:49.536 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:49.547 7fb996d73700 1 --2- 172.21.2.112:0/1372396770 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb9980aec80 0x7fb9980ed0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:36:49.537 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:49.547 7fb997574700 1 --2- 172.21.2.112:0/1372396770 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb9980ae880 0x7fb9981ae640 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 rx=0x7fb980006f00 tx=0x7fb98000b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:49.537 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:49.547 7fb98ffff700 1 -- 172.21.2.112:0/1372396770 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb98000bd30 con 0x7fb9980ae880 2021-02-10T11:36:49.537 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:49.547 7fb98ffff700 1 -- 172.21.2.112:0/1372396770 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb980000e20 con 0x7fb9980ae880 2021-02-10T11:36:49.537 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:49.547 7fb98ffff700 1 -- 172.21.2.112:0/1372396770 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb9800054f0 con 0x7fb9980ae880 2021-02-10T11:36:49.537 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:49.547 7fb99d9a2700 1 -- 172.21.2.112:0/1372396770 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9981afa70 con 0x7fb9980ae880 2021-02-10T11:36:49.537 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:49.547 7fb99d9a2700 1 -- 172.21.2.112:0/1372396770 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9981b0000 con 0x7fb9980ae880 2021-02-10T11:36:49.537 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:49.547 7fb98ffff700 1 -- 172.21.2.112:0/1372396770 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb980006030 con 0x7fb9980ae880 2021-02-10T11:36:49.537 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:49.547 7fb98ffff700 1 --2- 172.21.2.112:0/1372396770 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb97c0186c0 0x7fb97c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:49.537 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:49.548 7fb98ffff700 1 -- 172.21.2.112:0/1372396770 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 4753+0+0 (secure 0 0 0) 0x7fb9800324a0 con 0x7fb9980ae880 2021-02-10T11:36:49.537 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:49.548 7fb996d73700 1 --2- 172.21.2.112:0/1372396770 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb97c0186c0 0x7fb97c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:49.537 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:49.548 7fb996d73700 1 --2- 172.21.2.112:0/1372396770 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb97c0186c0 0x7fb97c01abc0 crc :-1 s=READY pgs=796 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:49.566 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:49.579 7fb99d9a2700 1 -- 172.21.2.112:0/1372396770 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9980ebc00 con 0x7fb9980ae880 2021-02-10T11:36:49.568 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:49.582 7fb98ffff700 1 -- 172.21.2.112:0/1372396770 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb980000bc0 con 0x7fb9980ae880 2021-02-10T11:36:49.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:49.705 7fb99d9a2700 1 -- 172.21.2.112:0/1372396770 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb9980f0b10 con 0x7fb9980ae880 2021-02-10T11:36:49.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:49.706 7fb98ffff700 1 -- 172.21.2.112:0/1372396770 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v79) v1 ==== 74+0+11145 (secure 0 0 0) 0x7fb980000e20 con 0x7fb9980ae880 2021-02-10T11:36:49.693 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:36:49.693 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":79,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:36:48.455316","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:36:49.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:49.708 7fb99d9a2700 1 -- 172.21.2.112:0/1372396770 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb97c0186c0 msgr2=0x7fb97c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:49.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:49.708 7fb99d9a2700 1 --2- 172.21.2.112:0/1372396770 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb97c0186c0 0x7fb97c01abc0 crc :-1 s=READY pgs=796 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:49.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:49.708 7fb99d9a2700 1 -- 172.21.2.112:0/1372396770 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb9980ae880 msgr2=0x7fb9981ae640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:49.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:49.708 7fb99d9a2700 1 --2- 172.21.2.112:0/1372396770 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb9980ae880 0x7fb9981ae640 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 rx=0x7fb980006f00 tx=0x7fb98000b940).stop 2021-02-10T11:36:49.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:49.708 7fb99d9a2700 1 -- 172.21.2.112:0/1372396770 shutdown_connections 2021-02-10T11:36:49.695 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:49.708 7fb99d9a2700 1 --2- 172.21.2.112:0/1372396770 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb97c0186c0 0x7fb97c01abc0 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:49.695 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:49.708 7fb99d9a2700 1 --2- 172.21.2.112:0/1372396770 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb9980ed710 0x7fb9981af450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:49.695 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:49.708 7fb99d9a2700 1 --2- 172.21.2.112:0/1372396770 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb9980aec80 0x7fb9980ed0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:49.695 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:49.708 7fb99d9a2700 1 --2- 172.21.2.112:0/1372396770 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb9980ae880 0x7fb9981ae640 unknown :-1 s=CLOSED pgs=1674 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:49.695 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:49.708 7fb99d9a2700 1 -- 172.21.2.112:0/1372396770 >> 172.21.2.112:0/1372396770 conn(0x7fb99819c560 msgr2=0x7fb9980eedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:49.695 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:49.708 7fb99d9a2700 1 -- 172.21.2.112:0/1372396770 shutdown_connections 2021-02-10T11:36:49.695 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:49.708 7fb99d9a2700 1 -- 172.21.2.112:0/1372396770 wait complete. 2021-02-10T11:36:49.703 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:13:57.508758', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2021-02-10T11:36:49.704 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:36:50.135 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.132 7f0b51d31700 1 Processor -- start 2021-02-10T11:36:50.135 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.132 7f0b51d31700 1 -- start start 2021-02-10T11:36:50.135 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.132 7f0b51d31700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f0b4c0d1870 0x7f0b4c1b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:50.135 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.132 7f0b51d31700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f0b4c1b0ae0 0x7f0b4c108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:50.135 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.132 7f0b51d31700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b4c0fe1c0 con 0x7f0b4c0d1c70 2021-02-10T11:36:50.135 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.132 7f0b51d31700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b4c109a60 con 0x7f0b4c0d1870 2021-02-10T11:36:50.135 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.132 7f0b51d31700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b4c109750 con 0x7f0b4c1b0ae0 2021-02-10T11:36:50.135 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.132 7f0b4b7fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f0b4c0d1870 0x7f0b4c1b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:50.136 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.132 7f0b4bfff700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f0b4c0d1c70 0x7f0b4c108f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:51802/0 (socket says 172.21.2.112:51802) 2021-02-10T11:36:50.136 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.132 7f0b4bfff700 1 -- 172.21.2.112:0/561978727 learned_addr learned my addr 172.21.2.112:0/561978727 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:36:50.136 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.133 7f0b4b7fe700 1 -- 172.21.2.112:0/561978727 >> v2:172.21.2.120:3300/0 conn(0x7f0b4c1b0ae0 msgr2=0x7f0b4c108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:50.136 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.133 7f0b4b7fe700 1 --2- 172.21.2.112:0/561978727 >> v2:172.21.2.120:3300/0 conn(0x7f0b4c1b0ae0 0x7f0b4c108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:50.136 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.133 7f0b4b7fe700 1 -- 172.21.2.112:0/561978727 >> v1:172.21.2.112:6789/0 conn(0x7f0b4c0d1c70 legacy=0x7f0b4c108f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:50.136 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.133 7f0b4a7fc700 1 -- 172.21.2.112:0/561978727 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2552873719 0 0) 0x7f0b44000c10 con 0x7f0b4c0d1c70 2021-02-10T11:36:50.136 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.133 7f0b4b7fe700 1 -- 172.21.2.112:0/561978727 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b4c088070 con 0x7f0b4c0d1870 2021-02-10T11:36:50.136 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.133 7f0b4b7fe700 1 --2- 172.21.2.112:0/561978727 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0b4c0d1870 0x7f0b4c1b04c0 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 rx=0x7f0b3c006650 tx=0x7f0b3c009a00).ready entity=mon.0 client_cookie=b6c1348531bc47be server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:50.136 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.133 7f0b4a7fc700 1 -- 172.21.2.112:0/561978727 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0b3c0009b0 con 0x7f0b4c0d1870 2021-02-10T11:36:50.136 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.133 7f0b4a7fc700 1 -- 172.21.2.112:0/561978727 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b3c00c0c0 con 0x7f0b4c0d1870 2021-02-10T11:36:50.136 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.133 7f0b4a7fc700 1 -- 172.21.2.112:0/561978727 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0b3c00c5a0 con 0x7f0b4c0d1870 2021-02-10T11:36:50.136 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.133 7f0b51d31700 1 -- 172.21.2.112:0/561978727 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0b4c0d1870 msgr2=0x7f0b4c1b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:50.136 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.133 7f0b51d31700 1 --2- 172.21.2.112:0/561978727 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0b4c0d1870 0x7f0b4c1b04c0 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 rx=0x7f0b3c006650 tx=0x7f0b3c009a00).stop 2021-02-10T11:36:50.136 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.134 7f0b51d31700 1 -- 172.21.2.112:0/561978727 shutdown_connections 2021-02-10T11:36:50.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.134 7f0b51d31700 1 --2- 172.21.2.112:0/561978727 >> v2:172.21.2.120:3300/0 conn(0x7f0b4c1b0ae0 0x7f0b4c108940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:50.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.134 7f0b51d31700 1 --2- 172.21.2.112:0/561978727 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0b4c0d1870 0x7f0b4c1b04c0 unknown :-1 s=CLOSED pgs=1676 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:50.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.134 7f0b51d31700 1 -- 172.21.2.112:0/561978727 >> 172.21.2.112:0/561978727 conn(0x7f0b4c0ce420 msgr2=0x7f0b4c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:50.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.134 7f0b51d31700 1 -- 172.21.2.112:0/561978727 shutdown_connections 2021-02-10T11:36:50.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.134 7f0b51d31700 1 -- 172.21.2.112:0/561978727 wait complete. 2021-02-10T11:36:50.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.148 7f0b51d31700 1 Processor -- start 2021-02-10T11:36:50.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.149 7f0b51d31700 1 -- start start 2021-02-10T11:36:50.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.149 7f0b51d31700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0b4c0c3820 0x7f0b4c0a7950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:50.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.149 7f0b51d31700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0b4c1a44c0 0x7f0b4c1a48c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:50.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.149 7f0b51d31700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0b4c0863d0 0x7f0b4c0867d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:50.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.149 7f0b51d31700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b4c085ed0 con 0x7f0b4c1a44c0 2021-02-10T11:36:50.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.149 7f0b51d31700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b4c0c0740 con 0x7f0b4c0c3820 2021-02-10T11:36:50.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.149 7f0b51d31700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b4c086df0 con 0x7f0b4c0863d0 2021-02-10T11:36:50.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.149 7f0b4affd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0b4c1a44c0 0x7f0b4c1a48c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:50.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.149 7f0b4affd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0b4c1a44c0 0x7f0b4c1a48c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:52346/0 (socket says 172.21.2.112:52346) 2021-02-10T11:36:50.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.149 7f0b4affd700 1 -- 172.21.2.112:0/3372363170 learned_addr learned my addr 172.21.2.112:0/3372363170 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:36:50.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.150 7f0b4affd700 1 -- 172.21.2.112:0/3372363170 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0b4c0863d0 msgr2=0x7f0b4c0867d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:50.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.150 7f0b4affd700 1 --2- 172.21.2.112:0/3372363170 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0b4c0863d0 0x7f0b4c0867d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:50.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.150 7f0b4affd700 1 -- 172.21.2.112:0/3372363170 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0b4c0c3820 msgr2=0x7f0b4c0a7950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:50.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.150 7f0b4affd700 1 --2- 172.21.2.112:0/3372363170 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0b4c0c3820 0x7f0b4c0a7950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:50.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.150 7f0b4affd700 1 -- 172.21.2.112:0/3372363170 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b4c087010 con 0x7f0b4c1a44c0 2021-02-10T11:36:50.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.150 7f0b4affd700 1 --2- 172.21.2.112:0/3372363170 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0b4c1a44c0 0x7f0b4c1a48c0 secure :-1 s=READY pgs=1677 cs=0 l=1 rev1=1 rx=0x7f0b34009980 tx=0x7f0b3400bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:50.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.150 7f0b43fff700 1 -- 172.21.2.112:0/3372363170 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0b3400c2f0 con 0x7f0b4c1a44c0 2021-02-10T11:36:50.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.150 7f0b43fff700 1 -- 172.21.2.112:0/3372363170 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b3400c570 con 0x7f0b4c1a44c0 2021-02-10T11:36:50.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.150 7f0b43fff700 1 -- 172.21.2.112:0/3372363170 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0b3400c9d0 con 0x7f0b4c1a44c0 2021-02-10T11:36:50.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.150 7f0b51d31700 1 -- 172.21.2.112:0/3372363170 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b4c087010 con 0x7f0b4c1a44c0 2021-02-10T11:36:50.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.150 7f0b51d31700 1 -- 172.21.2.112:0/3372363170 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b4c087010 con 0x7f0b4c1a44c0 2021-02-10T11:36:50.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.151 7f0b43fff700 1 -- 172.21.2.112:0/3372363170 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0b34001010 con 0x7f0b4c1a44c0 2021-02-10T11:36:50.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.151 7f0b43fff700 1 --2- 172.21.2.112:0/3372363170 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0b30018730 0x7f0b3001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:50.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.151 7f0b43fff700 1 -- 172.21.2.112:0/3372363170 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f0b340326a0 con 0x7f0b4c1a44c0 2021-02-10T11:36:50.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.151 7f0b4b7fe700 1 --2- 172.21.2.112:0/3372363170 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0b30018730 0x7f0b3001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:50.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.152 7f0b4b7fe700 1 --2- 172.21.2.112:0/3372363170 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0b30018730 0x7f0b3001ac10 crc :-1 s=READY pgs=797 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:50.169 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.182 7f0b51d31700 1 -- 172.21.2.112:0/3372363170 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b4c0fe1c0 con 0x7f0b4c1a44c0 2021-02-10T11:36:50.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.185 7f0b43fff700 1 -- 172.21.2.112:0/3372363170 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0b3400be10 con 0x7f0b4c1a44c0 2021-02-10T11:36:50.297 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.311 7f0b51d31700 1 -- 172.21.2.112:0/3372363170 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f0b4c0fe1c0 con 0x7f0b4c1a44c0 2021-02-10T11:36:50.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.311 7f0b43fff700 1 -- 172.21.2.112:0/3372363170 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v79) v1 ==== 92+0+10 (secure 0 0 0) 0x7f0b34001010 con 0x7f0b4c1a44c0 2021-02-10T11:36:50.298 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:36:50.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.313 7f0b51d31700 1 -- 172.21.2.112:0/3372363170 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0b30018730 msgr2=0x7f0b3001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:50.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.313 7f0b51d31700 1 --2- 172.21.2.112:0/3372363170 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0b30018730 0x7f0b3001ac10 crc :-1 s=READY pgs=797 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:50.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.313 7f0b51d31700 1 -- 172.21.2.112:0/3372363170 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0b4c1a44c0 msgr2=0x7f0b4c1a48c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:50.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.313 7f0b51d31700 1 --2- 172.21.2.112:0/3372363170 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0b4c1a44c0 0x7f0b4c1a48c0 secure :-1 s=READY pgs=1677 cs=0 l=1 rev1=1 rx=0x7f0b34009980 tx=0x7f0b3400bcd0).stop 2021-02-10T11:36:50.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.313 7f0b51d31700 1 -- 172.21.2.112:0/3372363170 shutdown_connections 2021-02-10T11:36:50.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.313 7f0b51d31700 1 --2- 172.21.2.112:0/3372363170 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0b30018730 0x7f0b3001ac10 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:50.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.313 7f0b51d31700 1 --2- 172.21.2.112:0/3372363170 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0b4c0863d0 0x7f0b4c0867d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:50.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.313 7f0b51d31700 1 --2- 172.21.2.112:0/3372363170 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0b4c1a44c0 0x7f0b4c1a48c0 unknown :-1 s=CLOSED pgs=1677 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:50.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.313 7f0b51d31700 1 --2- 172.21.2.112:0/3372363170 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0b4c0c3820 0x7f0b4c0a7950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:50.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.313 7f0b51d31700 1 -- 172.21.2.112:0/3372363170 >> 172.21.2.112:0/3372363170 conn(0x7f0b4c0872c0 msgr2=0x7f0b4c095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:50.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.314 7f0b51d31700 1 -- 172.21.2.112:0/3372363170 shutdown_connections 2021-02-10T11:36:50.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.314 7f0b51d31700 1 -- 172.21.2.112:0/3372363170 wait complete. 2021-02-10T11:36:50.308 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --format=json-pretty osd lspools 2021-02-10T11:36:50.735 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.732 7f10465d3700 1 Processor -- start 2021-02-10T11:36:50.736 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.732 7f10465d3700 1 -- start start 2021-02-10T11:36:50.736 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.732 7f10465d3700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f10400d0340 0x7f10400ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:50.736 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.732 7f10465d3700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f10400d1c70 0x7f10400efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:50.736 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.732 7f10465d3700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10400f0b10 con 0x7f10400d1870 2021-02-10T11:36:50.736 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.732 7f10465d3700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10400e5ed0 con 0x7f10400d0340 2021-02-10T11:36:50.736 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.732 7f10465d3700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10401b0be0 con 0x7f10400d1c70 2021-02-10T11:36:50.736 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.732 7f103ffff700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f10400d1870 0x7f10401b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:51812/0 (socket says 172.21.2.112:51812) 2021-02-10T11:36:50.736 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.732 7f103ffff700 1 -- 172.21.2.112:0/1941931339 learned_addr learned my addr 172.21.2.112:0/1941931339 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:36:50.736 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.733 7f103f7fe700 1 --2- 172.21.2.112:0/1941931339 >> v2:172.21.2.120:3300/0 conn(0x7f10400d0340 0x7f10400ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:50.736 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.733 7f1044bf0700 1 --2- 172.21.2.112:0/1941931339 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f10400d1c70 0x7f10400efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:50.736 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.733 7f103effd700 1 -- 172.21.2.112:0/1941931339 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 659918035 0 0) 0x7f1030000ce0 con 0x7f10400d1870 2021-02-10T11:36:50.737 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.733 7f103effd700 1 -- 172.21.2.112:0/1941931339 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1028002fe0 con 0x7f10400d1870 2021-02-10T11:36:50.737 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.733 7f103f7fe700 1 -- 172.21.2.112:0/1941931339 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f10400d1c70 msgr2=0x7f10400efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:50.737 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.733 7f103effd700 1 -- 172.21.2.112:0/1941931339 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 250079471 0 0) 0x7f1030001180 con 0x7f10400d1870 2021-02-10T11:36:50.737 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.733 7f103f7fe700 1 --2- 172.21.2.112:0/1941931339 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f10400d1c70 0x7f10400efc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:50.737 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.733 7f103f7fe700 1 -- 172.21.2.112:0/1941931339 >> v1:172.21.2.112:6789/0 conn(0x7f10400d1870 legacy=0x7f10401b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:50.737 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.733 7f103f7fe700 1 -- 172.21.2.112:0/1941931339 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10400f20a0 con 0x7f10400d0340 2021-02-10T11:36:50.737 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.733 7f1044bf0700 1 --2- 172.21.2.112:0/1941931339 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f10400d1c70 0x7f10400efc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:36:50.737 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.734 7f103f7fe700 1 --2- 172.21.2.112:0/1941931339 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f10400d0340 0x7f10400ef640 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 rx=0x7f1034006650 tx=0x7f1034009a80).ready entity=mon.1 client_cookie=335e4fa51c052caf server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:50.737 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.734 7f103effd700 1 -- 172.21.2.112:0/1941931339 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f103400c0c0 con 0x7f10400d0340 2021-02-10T11:36:50.737 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.734 7f103effd700 1 -- 172.21.2.112:0/1941931339 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f103400c340 con 0x7f10400d0340 2021-02-10T11:36:50.737 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.734 7f103effd700 1 -- 172.21.2.112:0/1941931339 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f103400c860 con 0x7f10400d0340 2021-02-10T11:36:50.737 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.734 7f10465d3700 1 -- 172.21.2.112:0/1941931339 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f10400d0340 msgr2=0x7f10400ef640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:50.737 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.734 7f10465d3700 1 --2- 172.21.2.112:0/1941931339 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f10400d0340 0x7f10400ef640 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 rx=0x7f1034006650 tx=0x7f1034009a80).stop 2021-02-10T11:36:50.737 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.735 7f10465d3700 1 -- 172.21.2.112:0/1941931339 shutdown_connections 2021-02-10T11:36:50.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.735 7f10465d3700 1 --2- 172.21.2.112:0/1941931339 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f10400d1c70 0x7f10400efc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:50.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.735 7f10465d3700 1 --2- 172.21.2.112:0/1941931339 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f10400d0340 0x7f10400ef640 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:50.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.735 7f10465d3700 1 -- 172.21.2.112:0/1941931339 >> 172.21.2.112:0/1941931339 conn(0x7f10400ce450 msgr2=0x7f10400d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:50.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.735 7f10465d3700 1 -- 172.21.2.112:0/1941931339 shutdown_connections 2021-02-10T11:36:50.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.735 7f10465d3700 1 -- 172.21.2.112:0/1941931339 wait complete. 2021-02-10T11:36:50.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.749 7f10465d3700 1 Processor -- start 2021-02-10T11:36:50.741 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.749 7f10465d3700 1 -- start start 2021-02-10T11:36:50.741 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.750 7f10465d3700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1040195650 0x7f10400b5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:50.741 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.750 7f10465d3700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f10400b5b80 0x7f10401a2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:50.741 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.750 7f10465d3700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f10400acfa0 0x7f10401a3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:50.741 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.750 7f10465d3700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10400b6010 con 0x7f10400b5b80 2021-02-10T11:36:50.741 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.750 7f10465d3700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10401a3a60 con 0x7f10400acfa0 2021-02-10T11:36:50.741 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.750 7f10465d3700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10401a3c40 con 0x7f1040195650 2021-02-10T11:36:50.741 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.750 7f103f7fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f10400b5b80 0x7f10401a2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:50.741 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.750 7f103f7fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f10400b5b80 0x7f10401a2e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:52356/0 (socket says 172.21.2.112:52356) 2021-02-10T11:36:50.741 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.750 7f103f7fe700 1 -- 172.21.2.112:0/3906082345 learned_addr learned my addr 172.21.2.112:0/3906082345 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:36:50.741 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.750 7f103f7fe700 1 -- 172.21.2.112:0/3906082345 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1040195650 msgr2=0x7f10400b5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:50.742 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.750 7f103f7fe700 1 --2- 172.21.2.112:0/3906082345 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1040195650 0x7f10400b5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:50.742 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.750 7f103f7fe700 1 -- 172.21.2.112:0/3906082345 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f10400acfa0 msgr2=0x7f10401a3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:50.742 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.750 7f103f7fe700 1 --2- 172.21.2.112:0/3906082345 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f10400acfa0 0x7f10401a3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:50.742 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.750 7f103f7fe700 1 -- 172.21.2.112:0/3906082345 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10400bcae0 con 0x7f10400b5b80 2021-02-10T11:36:50.742 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.751 7f103f7fe700 1 --2- 172.21.2.112:0/3906082345 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f10400b5b80 0x7f10401a2e20 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 rx=0x7f1034009630 tx=0x7f103400b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:50.742 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.751 7f103cff9700 1 -- 172.21.2.112:0/3906082345 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f10340093a0 con 0x7f10400b5b80 2021-02-10T11:36:50.742 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.751 7f103cff9700 1 -- 172.21.2.112:0/3906082345 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f103400bd70 con 0x7f10400b5b80 2021-02-10T11:36:50.742 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.751 7f10465d3700 1 -- 172.21.2.112:0/3906082345 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10400bcae0 con 0x7f10400b5b80 2021-02-10T11:36:50.742 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.751 7f10465d3700 1 -- 172.21.2.112:0/3906082345 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10400bd070 con 0x7f10400b5b80 2021-02-10T11:36:50.742 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.751 7f103cff9700 1 -- 172.21.2.112:0/3906082345 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1034005560 con 0x7f10400b5b80 2021-02-10T11:36:50.742 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.751 7f103cff9700 1 -- 172.21.2.112:0/3906082345 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f103400a000 con 0x7f10400b5b80 2021-02-10T11:36:50.742 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.752 7f103cff9700 1 --2- 172.21.2.112:0/3906082345 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f10200186d0 0x7f102001abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:50.742 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.752 7f103cff9700 1 -- 172.21.2.112:0/3906082345 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f10340324a0 con 0x7f10400b5b80 2021-02-10T11:36:50.742 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.752 7f103ffff700 1 --2- 172.21.2.112:0/3906082345 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f10200186d0 0x7f102001abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:50.743 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.753 7f103ffff700 1 --2- 172.21.2.112:0/3906082345 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f10200186d0 0x7f102001abd0 crc :-1 s=READY pgs=798 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:50.769 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.783 7f10465d3700 1 -- 172.21.2.112:0/3906082345 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10400f0ab0 con 0x7f10400b5b80 2021-02-10T11:36:50.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.785 7f103cff9700 1 -- 172.21.2.112:0/3906082345 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1034009340 con 0x7f10400b5b80 2021-02-10T11:36:50.900 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.913 7f10465d3700 1 -- 172.21.2.112:0/3906082345 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f10400f0ab0 con 0x7f10400b5b80 2021-02-10T11:36:50.900 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.914 7f103cff9700 1 -- 172.21.2.112:0/3906082345 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v79) v1 ==== 84+0+64 (secure 0 0 0) 0x7f103400bd70 con 0x7f10400b5b80 2021-02-10T11:36:50.901 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:36:50.901 INFO:teuthology.orchestra.run.gibba012.stdout:[ 2021-02-10T11:36:50.901 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:36:50.901 INFO:teuthology.orchestra.run.gibba012.stdout: "poolnum": 1, 2021-02-10T11:36:50.901 INFO:teuthology.orchestra.run.gibba012.stdout: "poolname": "rbd" 2021-02-10T11:36:50.901 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:36:50.901 INFO:teuthology.orchestra.run.gibba012.stdout:] 2021-02-10T11:36:50.902 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.916 7f10465d3700 1 -- 172.21.2.112:0/3906082345 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f10200186d0 msgr2=0x7f102001abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:50.902 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.916 7f10465d3700 1 --2- 172.21.2.112:0/3906082345 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f10200186d0 0x7f102001abd0 crc :-1 s=READY pgs=798 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:50.902 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.916 7f10465d3700 1 -- 172.21.2.112:0/3906082345 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f10400b5b80 msgr2=0x7f10401a2e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:50.902 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.916 7f10465d3700 1 --2- 172.21.2.112:0/3906082345 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f10400b5b80 0x7f10401a2e20 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 rx=0x7f1034009630 tx=0x7f103400b980).stop 2021-02-10T11:36:50.903 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.916 7f10465d3700 1 -- 172.21.2.112:0/3906082345 shutdown_connections 2021-02-10T11:36:50.903 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.916 7f10465d3700 1 --2- 172.21.2.112:0/3906082345 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f10200186d0 0x7f102001abd0 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:50.903 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.916 7f10465d3700 1 --2- 172.21.2.112:0/3906082345 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f10400acfa0 0x7f10401a3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:50.903 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.916 7f10465d3700 1 --2- 172.21.2.112:0/3906082345 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f10400b5b80 0x7f10401a2e20 unknown :-1 s=CLOSED pgs=1679 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:50.903 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.916 7f10465d3700 1 --2- 172.21.2.112:0/3906082345 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1040195650 0x7f10400b5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:50.903 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.916 7f10465d3700 1 -- 172.21.2.112:0/3906082345 >> 172.21.2.112:0/3906082345 conn(0x7f10400b3040 msgr2=0x7f10400c3250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:50.903 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.916 7f10465d3700 1 -- 172.21.2.112:0/3906082345 shutdown_connections 2021-02-10T11:36:50.903 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:50.916 7f10465d3700 1 -- 172.21.2.112:0/3906082345 wait complete. 2021-02-10T11:36:50.911 INFO:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2021-02-10T11:36:50.911 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_metadata 8 2021-02-10T11:36:51.342 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:51.338 7febe58ff700 1 Processor -- start 2021-02-10T11:36:51.342 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:51.338 7febe58ff700 1 -- start start 2021-02-10T11:36:51.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:51.339 7febe58ff700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7febe00d1c70 0x7febe00cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:51.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:51.339 7febe58ff700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febe00d0170 con 0x7febe00d1870 2021-02-10T11:36:51.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:51.339 7febe58ff700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febe00fe1c0 con 0x7febe00cf580 2021-02-10T11:36:51.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:51.339 7febe58ff700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febe01b0cb0 con 0x7febe00d1c70 2021-02-10T11:36:51.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:51.339 7febdf574700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7febe00d1870 0x7febe01b04c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:51824/0 (socket says 172.21.2.112:51824) 2021-02-10T11:36:51.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:51.339 7febdf574700 1 -- 172.21.2.112:0/2051995550 learned_addr learned my addr 172.21.2.112:0/2051995550 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:36:51.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:51.339 7febde572700 1 -- 172.21.2.112:0/2051995550 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3176820264 0 0) 0x7febd0000ce0 con 0x7febe00d1870 2021-02-10T11:36:51.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:51.339 7febde572700 1 -- 172.21.2.112:0/2051995550 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febcc002fe0 con 0x7febe00d1870 2021-02-10T11:36:51.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:51.340 7febded73700 1 --2- 172.21.2.112:0/2051995550 >> v2:172.21.2.120:3300/0 conn(0x7febe00d1c70 0x7febe00cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:51.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:51.340 7febde572700 1 -- 172.21.2.112:0/2051995550 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3494557822 0 0) 0x7febd0001180 con 0x7febe00d1870 2021-02-10T11:36:51.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:51.340 7febde572700 1 -- 172.21.2.112:0/2051995550 >> v2:172.21.2.120:3300/0 conn(0x7febe00d1c70 msgr2=0x7febe00cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:51.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:51.340 7febde572700 1 --2- 172.21.2.112:0/2051995550 >> v2:172.21.2.120:3300/0 conn(0x7febe00d1c70 0x7febe00cef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:51.344 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:51.340 7febde572700 1 -- 172.21.2.112:0/2051995550 >> v1:172.21.2.120:6789/0 conn(0x7febe00cf580 legacy=0x7febe00cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:51.344 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:51.340 7febde572700 1 -- 172.21.2.112:0/2051995550 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febe0109a60 con 0x7febe00d1870 2021-02-10T11:36:51.344 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:51.340 7febde572700 1 -- 172.21.2.112:0/2051995550 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7febd0001680 con 0x7febe00d1870 2021-02-10T11:36:51.344 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:51.340 7febde572700 1 -- 172.21.2.112:0/2051995550 >> v1:172.21.2.112:6789/0 conn(0x7febe00d1870 legacy=0x7febe01b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:51.344 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:51.340 7febde572700 1 --2- 172.21.2.112:0/2051995550 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7febcc003b10 0x7febcc005f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:51.344 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:51.340 7febde572700 1 --2- 172.21.2.112:0/2051995550 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7febcc006820 0x7febcc008d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:51.344 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:51.340 7febde572700 1 --2- 172.21.2.112:0/2051995550 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7febcc009620 0x7febcc00bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:51.344 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:51.340 7febde572700 1 -- 172.21.2.112:0/2051995550 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febcc00c390 con 0x7febcc006820 2021-02-10T11:36:51.344 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:51.340 7febde572700 1 -- 172.21.2.112:0/2051995550 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febcc00c5b0 con 0x7febcc003b10 2021-02-10T11:36:51.344 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:51.340 7febde572700 1 -- 172.21.2.112:0/2051995550 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febcc00c7d0 con 0x7febcc009620 2021-02-10T11:36:51.344 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:51.340 7febded73700 1 --2- 172.21.2.112:0/2051995550 >> v2:172.21.2.120:3300/0 conn(0x7febe00d1c70 0x7febe00cef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:36:51.344 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:51.340 7febded73700 1 --2- 172.21.2.112:0/2051995550 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7febcc006820 0x7febcc008d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:51.344 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:51.340 7febdf574700 1 --2- 172.21.2.112:0/2051995550 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7febcc003b10 0x7febcc005f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:51.345 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:51.340 7febdfd75700 1 --2- 172.21.2.112:0/2051995550 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7febcc009620 0x7febcc00bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:51.345 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:51.340 7febded73700 1 -- 172.21.2.112:0/2051995550 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7febcc009620 msgr2=0x7febcc00bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:51.345 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:51.340 7febded73700 1 --2- 172.21.2.112:0/2051995550 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7febcc009620 0x7febcc00bb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:51.345 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:51.340 7febded73700 1 -- 172.21.2.112:0/2051995550 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7febcc003b10 msgr2=0x7febcc005f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:51.345 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:51.340 7febded73700 1 --2- 172.21.2.112:0/2051995550 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7febcc003b10 0x7febcc005f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:51.345 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:51.340 7febded73700 1 -- 172.21.2.112:0/2051995550 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febcc00ca70 con 0x7febcc006820 2021-02-10T11:36:51.345 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:51.340 7febdfd75700 1 -- 172.21.2.112:0/2051995550 reap_dead start 2021-02-10T11:36:51.345 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:51.340 7febded73700 1 --2- 172.21.2.112:0/2051995550 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7febcc006820 0x7febcc008d20 secure :-1 s=READY pgs=1681 cs=0 l=1 rev1=1 rx=0x7febc8006cf0 tx=0x7febc804dd20).ready entity=mon.0 client_cookie=96df2febd20c0485 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:51.345 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:51.341 7febde572700 1 -- 172.21.2.112:0/2051995550 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7febc8006d30 con 0x7febcc006820 2021-02-10T11:36:51.345 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:51.341 7febde572700 1 -- 172.21.2.112:0/2051995550 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febc8006910 con 0x7febcc006820 2021-02-10T11:36:51.345 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:51.341 7febde572700 1 -- 172.21.2.112:0/2051995550 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7febc80020d0 con 0x7febcc006820 2021-02-10T11:36:51.345 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:51.341 7febe58ff700 1 -- 172.21.2.112:0/2051995550 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7febcc006820 msgr2=0x7febcc008d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:51.345 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:51.341 7febe58ff700 1 --2- 172.21.2.112:0/2051995550 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7febcc006820 0x7febcc008d20 secure :-1 s=READY pgs=1681 cs=0 l=1 rev1=1 rx=0x7febc8006cf0 tx=0x7febc804dd20).stop 2021-02-10T11:36:51.345 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:51.341 7febe58ff700 1 -- 172.21.2.112:0/2051995550 shutdown_connections 2021-02-10T11:36:51.346 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:51.341 7febe58ff700 1 --2- 172.21.2.112:0/2051995550 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7febcc006820 0x7febcc008d20 unknown :-1 s=CLOSED pgs=1681 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:51.346 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:51.341 7febe58ff700 1 -- 172.21.2.112:0/2051995550 >> 172.21.2.112:0/2051995550 conn(0x7febe00ce420 msgr2=0x7febe00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:51.346 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:51.341 7febe58ff700 1 -- 172.21.2.112:0/2051995550 shutdown_connections 2021-02-10T11:36:51.346 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:51.341 7febe58ff700 1 -- 172.21.2.112:0/2051995550 wait complete. 2021-02-10T11:36:51.346 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:51.356 7febe58ff700 1 Processor -- start 2021-02-10T11:36:51.346 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:51.356 7febe58ff700 1 -- start start 2021-02-10T11:36:51.347 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:51.357 7febe58ff700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7febe0085bc0 0x7febe00863e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:51.347 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:51.357 7febe58ff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7febe00954c0 0x7febe0096300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:51.347 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:51.357 7febe58ff700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7febe00958c0 0x7febe0096920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:51.347 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:51.357 7febe58ff700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febe0095d80 con 0x7febe00954c0 2021-02-10T11:36:51.347 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:51.357 7febe58ff700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febe00bfa60 con 0x7febe0085bc0 2021-02-10T11:36:51.348 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:51.357 7febe58ff700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febe0085fc0 con 0x7febe00958c0 2021-02-10T11:36:51.348 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:51.357 7febded73700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7febe00954c0 0x7febe0096300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:51.348 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:51.357 7febded73700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7febe00954c0 0x7febe0096300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:52376/0 (socket says 172.21.2.112:52376) 2021-02-10T11:36:51.348 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:51.357 7febded73700 1 -- 172.21.2.112:0/2554031874 learned_addr learned my addr 172.21.2.112:0/2554031874 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:36:51.348 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:51.357 7febded73700 1 -- 172.21.2.112:0/2554031874 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7febe00958c0 msgr2=0x7febe0096920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:51.348 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:51.357 7febded73700 1 --2- 172.21.2.112:0/2554031874 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7febe00958c0 0x7febe0096920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:51.348 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:51.357 7febded73700 1 -- 172.21.2.112:0/2554031874 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7febe0085bc0 msgr2=0x7febe00863e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:51.348 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:51.357 7febded73700 1 --2- 172.21.2.112:0/2554031874 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7febe0085bc0 0x7febe00863e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:51.348 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:51.357 7febded73700 1 -- 172.21.2.112:0/2554031874 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febe0086c10 con 0x7febe00954c0 2021-02-10T11:36:51.348 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:51.358 7febded73700 1 --2- 172.21.2.112:0/2554031874 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7febe00954c0 0x7febe0096300 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 rx=0x7febc804d860 tx=0x7febc8050970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:51.348 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:51.358 7febd7fff700 1 -- 172.21.2.112:0/2554031874 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7febc8050d60 con 0x7febe00954c0 2021-02-10T11:36:51.348 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:51.358 7febd7fff700 1 -- 172.21.2.112:0/2554031874 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febc804d3d0 con 0x7febe00954c0 2021-02-10T11:36:51.348 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:51.358 7febd7fff700 1 -- 172.21.2.112:0/2554031874 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7febc8002030 con 0x7febe00954c0 2021-02-10T11:36:51.349 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:51.358 7febe58ff700 1 -- 172.21.2.112:0/2554031874 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febe0086c10 con 0x7febe00954c0 2021-02-10T11:36:51.349 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:51.358 7febe58ff700 1 -- 172.21.2.112:0/2554031874 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febe0086c10 con 0x7febe00954c0 2021-02-10T11:36:51.349 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:51.358 7febd7fff700 1 -- 172.21.2.112:0/2554031874 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7febc8000c90 con 0x7febe00954c0 2021-02-10T11:36:51.349 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:51.359 7febd7fff700 1 --2- 172.21.2.112:0/2554031874 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7febc40186c0 0x7febc401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:51.349 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:51.359 7febd7fff700 1 -- 172.21.2.112:0/2554031874 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 4753+0+0 (secure 0 0 0) 0x7febc8076300 con 0x7febe00954c0 2021-02-10T11:36:51.349 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:51.359 7febdf574700 1 --2- 172.21.2.112:0/2554031874 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7febc40186c0 0x7febc401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:51.349 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:51.360 7febdf574700 1 --2- 172.21.2.112:0/2554031874 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7febc40186c0 0x7febc401abc0 crc :-1 s=READY pgs=799 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:51.375 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:51.389 7febe58ff700 1 -- 172.21.2.112:0/2554031874 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febe00fe1c0 con 0x7febe00954c0 2021-02-10T11:36:51.377 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:51.391 7febd7fff700 1 -- 172.21.2.112:0/2554031874 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7febc8006030 con 0x7febe00954c0 2021-02-10T11:36:51.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:51.520 7febe58ff700 1 -- 172.21.2.112:0/2554031874 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"} v 0) v1 -- 0x7febe00fe1c0 con 0x7febe00954c0 2021-02-10T11:36:51.923 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:51.937 7febd7fff700 1 -- 172.21.2.112:0/2554031874 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"}]=0 pool 'cephfs_metadata' created v80) v1 ==== 133+0+0 (secure 0 0 0) 0x7febc8000c90 con 0x7febe00954c0 2021-02-10T11:36:51.972 INFO:teuthology.orchestra.run.gibba012.stderr:pool 'cephfs_metadata' created 2021-02-10T11:36:51.974 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:51.988 7febe58ff700 1 -- 172.21.2.112:0/2554031874 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7febc40186c0 msgr2=0x7febc401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:51.974 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:51.988 7febe58ff700 1 --2- 172.21.2.112:0/2554031874 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7febc40186c0 0x7febc401abc0 crc :-1 s=READY pgs=799 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:51.974 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:51.988 7febe58ff700 1 -- 172.21.2.112:0/2554031874 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7febe00954c0 msgr2=0x7febe0096300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:51.974 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:51.988 7febe58ff700 1 --2- 172.21.2.112:0/2554031874 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7febe00954c0 0x7febe0096300 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 rx=0x7febc804d860 tx=0x7febc8050970).stop 2021-02-10T11:36:51.975 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:51.988 7febe58ff700 1 -- 172.21.2.112:0/2554031874 shutdown_connections 2021-02-10T11:36:51.975 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:51.988 7febe58ff700 1 --2- 172.21.2.112:0/2554031874 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7febc40186c0 0x7febc401abc0 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:51.975 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:51.988 7febe58ff700 1 --2- 172.21.2.112:0/2554031874 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7febe00958c0 0x7febe0096920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:51.975 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:51.988 7febe58ff700 1 --2- 172.21.2.112:0/2554031874 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7febe00954c0 0x7febe0096300 unknown :-1 s=CLOSED pgs=1682 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:51.975 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:51.988 7febe58ff700 1 --2- 172.21.2.112:0/2554031874 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7febe0085bc0 0x7febe00863e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:51.975 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:51.988 7febe58ff700 1 -- 172.21.2.112:0/2554031874 >> 172.21.2.112:0/2554031874 conn(0x7febe00bef00 msgr2=0x7febe009d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:51.975 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:51.988 7febe58ff700 1 -- 172.21.2.112:0/2554031874 shutdown_connections 2021-02-10T11:36:51.975 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:51.988 7febe58ff700 1 -- 172.21.2.112:0/2554031874 wait complete. 2021-02-10T11:36:51.982 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_data 8 2021-02-10T11:36:52.413 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:52.409 7f9345531700 1 Processor -- start 2021-02-10T11:36:52.413 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:52.409 7f9345531700 1 -- start start 2021-02-10T11:36:52.413 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:52.410 7f9345531700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f93400d1870 0x7f93401b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:52.413 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:52.410 7f9345531700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9340109a60 con 0x7f93400cff70 2021-02-10T11:36:52.413 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:52.410 7f9345531700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9340088070 con 0x7f93401b0ae0 2021-02-10T11:36:52.413 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:52.410 7f9345531700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93400e5ef0 con 0x7f93400d1870 2021-02-10T11:36:52.413 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:52.410 7f933fac0700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f93400cff70 0x7f9340109130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:51844/0 (socket says 172.21.2.112:51844) 2021-02-10T11:36:52.413 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:52.410 7f933fac0700 1 -- 172.21.2.112:0/1677875032 learned_addr learned my addr 172.21.2.112:0/1677875032 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:36:52.414 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:52.410 7f933e2bd700 1 -- 172.21.2.112:0/1677875032 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 752129500 0 0) 0x7f9338001500 con 0x7f93400cff70 2021-02-10T11:36:52.414 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:52.410 7f933e2bd700 1 -- 172.21.2.112:0/1677875032 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f932c002fe0 con 0x7f93400cff70 2021-02-10T11:36:52.414 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:52.411 7f933e2bd700 1 -- 172.21.2.112:0/1677875032 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2714505211 0 0) 0x7f9338001460 con 0x7f93400cff70 2021-02-10T11:36:52.414 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:52.411 7f933e2bd700 1 -- 172.21.2.112:0/1677875032 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f93400d1870 msgr2=0x7f93401b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:52.414 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:52.411 7f933e2bd700 1 --2- 172.21.2.112:0/1677875032 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f93400d1870 0x7f93401b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:52.414 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:52.411 7f933e2bd700 1 -- 172.21.2.112:0/1677875032 >> v1:172.21.2.120:6789/0 conn(0x7f93401b0ae0 legacy=0x7f9340108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:52.414 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:52.411 7f933e2bd700 1 -- 172.21.2.112:0/1677875032 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93400d1c70 con 0x7f93400cff70 2021-02-10T11:36:52.414 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:52.411 7f933e2bd700 1 -- 172.21.2.112:0/1677875032 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f9338001980 con 0x7f93400cff70 2021-02-10T11:36:52.414 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:52.411 7f933e2bd700 1 -- 172.21.2.112:0/1677875032 >> v1:172.21.2.112:6789/0 conn(0x7f93400cff70 legacy=0x7f9340109130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:52.414 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:52.411 7f933e2bd700 1 --2- 172.21.2.112:0/1677875032 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f932c003a70 0x7f932c005f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:52.414 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:52.411 7f933e2bd700 1 --2- 172.21.2.112:0/1677875032 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f932c006780 0x7f932c008ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:52.414 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:52.411 7f933e2bd700 1 --2- 172.21.2.112:0/1677875032 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f932c0095a0 0x7f932c00ba90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:52.415 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:52.411 7f933e2bd700 1 -- 172.21.2.112:0/1677875032 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f932c00c2e0 con 0x7f932c003a70 2021-02-10T11:36:52.415 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:52.411 7f933e2bd700 1 -- 172.21.2.112:0/1677875032 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f932c00c500 con 0x7f932c006780 2021-02-10T11:36:52.415 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:52.411 7f933e2bd700 1 -- 172.21.2.112:0/1677875032 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f932c00c720 con 0x7f932c0095a0 2021-02-10T11:36:52.415 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:52.411 7f933f2bf700 1 --2- 172.21.2.112:0/1677875032 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f932c003a70 0x7f932c005f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:52.415 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:52.411 7f933f2bf700 1 -- 172.21.2.112:0/1677875032 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f932c0095a0 msgr2=0x7f932c00ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:52.415 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:52.411 7f933fac0700 1 --2- 172.21.2.112:0/1677875032 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f932c0095a0 0x7f932c00ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:52.415 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:52.411 7f933eabe700 1 --2- 172.21.2.112:0/1677875032 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f932c006780 0x7f932c008ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:52.415 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:52.411 7f933f2bf700 1 --2- 172.21.2.112:0/1677875032 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f932c0095a0 0x7f932c00ba90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:52.415 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:52.411 7f933f2bf700 1 -- 172.21.2.112:0/1677875032 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f932c006780 msgr2=0x7f932c008ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:52.415 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:52.411 7f933f2bf700 1 --2- 172.21.2.112:0/1677875032 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f932c006780 0x7f932c008ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:52.415 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:52.411 7f933f2bf700 1 -- 172.21.2.112:0/1677875032 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f932c00c9c0 con 0x7f932c003a70 2021-02-10T11:36:52.415 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:52.411 7f933fac0700 1 --2- 172.21.2.112:0/1677875032 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f932c0095a0 0x7f932c00ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:36:52.416 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:52.411 7f933eabe700 1 --2- 172.21.2.112:0/1677875032 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f932c006780 0x7f932c008ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:36:52.416 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:52.411 7f933f2bf700 1 --2- 172.21.2.112:0/1677875032 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f932c003a70 0x7f932c005f00 secure :-1 s=READY pgs=1684 cs=0 l=1 rev1=1 rx=0x7f9328006ab0 tx=0x7f932804fbc0).ready entity=mon.0 client_cookie=ffb097a2c6a81687 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:52.416 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:52.411 7f933e2bd700 1 -- 172.21.2.112:0/1677875032 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9328002080 con 0x7f932c003a70 2021-02-10T11:36:52.416 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:52.411 7f933e2bd700 1 -- 172.21.2.112:0/1677875032 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93280063f0 con 0x7f932c003a70 2021-02-10T11:36:52.416 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:52.411 7f933e2bd700 1 -- 172.21.2.112:0/1677875032 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9328002300 con 0x7f932c003a70 2021-02-10T11:36:52.416 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:52.411 7f9345531700 1 -- 172.21.2.112:0/1677875032 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f932c003a70 msgr2=0x7f932c005f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:52.416 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:52.411 7f9345531700 1 --2- 172.21.2.112:0/1677875032 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f932c003a70 0x7f932c005f00 secure :-1 s=READY pgs=1684 cs=0 l=1 rev1=1 rx=0x7f9328006ab0 tx=0x7f932804fbc0).stop 2021-02-10T11:36:52.416 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:52.411 7f933fac0700 1 -- 172.21.2.112:0/1677875032 reap_dead start 2021-02-10T11:36:52.416 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:52.412 7f9345531700 1 -- 172.21.2.112:0/1677875032 shutdown_connections 2021-02-10T11:36:52.416 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:52.412 7f9345531700 1 -- 172.21.2.112:0/1677875032 >> 172.21.2.112:0/1677875032 conn(0x7f93400ce420 msgr2=0x7f93400d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:52.416 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:52.412 7f9345531700 1 -- 172.21.2.112:0/1677875032 shutdown_connections 2021-02-10T11:36:52.416 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:52.412 7f9345531700 1 -- 172.21.2.112:0/1677875032 wait complete. 2021-02-10T11:36:52.416 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:52.426 7f9345531700 1 Processor -- start 2021-02-10T11:36:52.417 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:52.426 7f9345531700 1 -- start start 2021-02-10T11:36:52.418 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:52.427 7f9345531700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f934009bcc0 0x7f934009c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:52.418 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:52.427 7f9345531700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f93400a0820 0x7f93400a0c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:52.418 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:52.427 7f9345531700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f93400a1240 0x7f93400858d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:52.418 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:52.427 7f9345531700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93400a66b0 con 0x7f93400a0820 2021-02-10T11:36:52.418 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:52.427 7f9345531700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93401a49b0 con 0x7f93400a1240 2021-02-10T11:36:52.418 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:52.427 7f9345531700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93401a4b90 con 0x7f934009bcc0 2021-02-10T11:36:52.418 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:52.427 7f933eabe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f93400a0820 0x7f93400a0c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:52.418 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:52.427 7f933eabe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f93400a0820 0x7f93400a0c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:52396/0 (socket says 172.21.2.112:52396) 2021-02-10T11:36:52.419 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:52.427 7f933eabe700 1 -- 172.21.2.112:0/1912432377 learned_addr learned my addr 172.21.2.112:0/1912432377 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:36:52.419 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:52.427 7f933eabe700 1 -- 172.21.2.112:0/1912432377 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f934009bcc0 msgr2=0x7f934009c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:52.419 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:52.427 7f933eabe700 1 --2- 172.21.2.112:0/1912432377 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f934009bcc0 0x7f934009c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:52.419 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:52.427 7f933eabe700 1 -- 172.21.2.112:0/1912432377 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f93400a1240 msgr2=0x7f93400858d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:52.419 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:52.427 7f933eabe700 1 --2- 172.21.2.112:0/1912432377 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f93400a1240 0x7f93400858d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:52.419 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:52.427 7f933eabe700 1 -- 172.21.2.112:0/1912432377 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93400a1640 con 0x7f93400a0820 2021-02-10T11:36:52.419 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:52.427 7f933eabe700 1 --2- 172.21.2.112:0/1912432377 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f93400a0820 0x7f93400a0c20 secure :-1 s=READY pgs=1685 cs=0 l=1 rev1=1 rx=0x7f9330006f10 tx=0x7f933000b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:52.419 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:52.428 7f9337fff700 1 -- 172.21.2.112:0/1912432377 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f933000bd80 con 0x7f93400a0820 2021-02-10T11:36:52.419 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:52.428 7f9337fff700 1 -- 172.21.2.112:0/1912432377 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93300063e0 con 0x7f93400a0820 2021-02-10T11:36:52.419 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:52.428 7f9337fff700 1 -- 172.21.2.112:0/1912432377 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9330002030 con 0x7f93400a0820 2021-02-10T11:36:52.419 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:52.428 7f9345531700 1 -- 172.21.2.112:0/1912432377 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93400a1640 con 0x7f93400a0820 2021-02-10T11:36:52.419 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:52.428 7f9345531700 1 -- 172.21.2.112:0/1912432377 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93400861e0 con 0x7f93400a0820 2021-02-10T11:36:52.420 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:52.428 7f9337fff700 1 -- 172.21.2.112:0/1912432377 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9330000c50 con 0x7f93400a0820 2021-02-10T11:36:52.420 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:52.429 7f9337fff700 1 --2- 172.21.2.112:0/1912432377 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f93240186c0 0x7f932401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:52.420 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:52.429 7f9337fff700 1 -- 172.21.2.112:0/1912432377 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5107+0+0 (secure 0 0 0) 0x7f93300323a0 con 0x7f93400a0820 2021-02-10T11:36:52.420 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:52.429 7f933f2bf700 1 --2- 172.21.2.112:0/1912432377 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f93240186c0 0x7f932401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:52.420 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:52.429 7f933f2bf700 1 --2- 172.21.2.112:0/1912432377 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f93240186c0 0x7f932401abc0 crc :-1 s=READY pgs=800 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:52.446 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:52.460 7f9345531700 1 -- 172.21.2.112:0/1912432377 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9340086a70 con 0x7f93400a0820 2021-02-10T11:36:52.449 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:52.462 7f9337fff700 1 -- 172.21.2.112:0/1912432377 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9330000940 con 0x7f93400a0820 2021-02-10T11:36:52.577 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:52.591 7f9345531700 1 -- 172.21.2.112:0/1912432377 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"} v 0) v1 -- 0x7f9340086a70 con 0x7f93400a0820 2021-02-10T11:36:52.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:52.946 7f9337fff700 1 -- 172.21.2.112:0/1912432377 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"}]=0 pool 'cephfs_data' created v81) v1 ==== 125+0+0 (secure 0 0 0) 0x7f9330000c50 con 0x7f93400a0820 2021-02-10T11:36:52.943 INFO:teuthology.orchestra.run.gibba012.stderr:pool 'cephfs_data' created 2021-02-10T11:36:52.945 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:52.958 7f9345531700 1 -- 172.21.2.112:0/1912432377 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f93240186c0 msgr2=0x7f932401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:52.945 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:52.958 7f9345531700 1 --2- 172.21.2.112:0/1912432377 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f93240186c0 0x7f932401abc0 crc :-1 s=READY pgs=800 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:52.945 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:52.958 7f9345531700 1 -- 172.21.2.112:0/1912432377 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f93400a0820 msgr2=0x7f93400a0c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:52.945 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:52.958 7f9345531700 1 --2- 172.21.2.112:0/1912432377 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f93400a0820 0x7f93400a0c20 secure :-1 s=READY pgs=1685 cs=0 l=1 rev1=1 rx=0x7f9330006f10 tx=0x7f933000b990).stop 2021-02-10T11:36:52.945 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:52.958 7f9345531700 1 -- 172.21.2.112:0/1912432377 shutdown_connections 2021-02-10T11:36:52.945 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:52.958 7f9345531700 1 --2- 172.21.2.112:0/1912432377 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f93240186c0 0x7f932401abc0 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:52.945 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:52.958 7f9345531700 1 --2- 172.21.2.112:0/1912432377 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f93400a1240 0x7f93400858d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:52.946 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:52.958 7f9345531700 1 --2- 172.21.2.112:0/1912432377 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f93400a0820 0x7f93400a0c20 unknown :-1 s=CLOSED pgs=1685 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:52.946 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:52.958 7f9345531700 1 --2- 172.21.2.112:0/1912432377 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f934009bcc0 0x7f934009c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:52.946 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:52.958 7f9345531700 1 -- 172.21.2.112:0/1912432377 >> 172.21.2.112:0/1912432377 conn(0x7f93400bef00 msgr2=0x7f934009d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:52.946 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:52.958 7f9345531700 1 -- 172.21.2.112:0/1912432377 shutdown_connections 2021-02-10T11:36:52.946 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:52.958 7f9345531700 1 -- 172.21.2.112:0/1912432377 wait complete. 2021-02-10T11:36:52.954 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2021-02-10T11:36:53.383 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:53.380 7fd8f186d700 1 Processor -- start 2021-02-10T11:36:53.383 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:53.380 7fd8f186d700 1 -- start start 2021-02-10T11:36:53.383 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:53.381 7fd8f186d700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fd8ec0d1880 0x7fd8ec1b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:53.383 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:53.381 7fd8f186d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fd8ec1b0af0 0x7fd8ec108950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:53.383 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:53.381 7fd8f186d700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8ec0fe1d0 con 0x7fd8ec0d1c80 2021-02-10T11:36:53.383 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:53.381 7fd8f186d700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8ec109a70 con 0x7fd8ec1b0af0 2021-02-10T11:36:53.383 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:53.381 7fd8f186d700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8ec109760 con 0x7fd8ec0d1880 2021-02-10T11:36:53.383 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:53.381 7fd8ead73700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fd8ec1b0af0 0x7fd8ec108950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:53.384 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:53.381 7fd8ead73700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fd8ec1b0af0 0x7fd8ec108950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:52402/0 (socket says 172.21.2.112:52402) 2021-02-10T11:36:53.384 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:53.381 7fd8ead73700 1 -- 172.21.2.112:0/1076731614 learned_addr learned my addr 172.21.2.112:0/1076731614 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:36:53.384 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:53.381 7fd8ead73700 1 -- 172.21.2.112:0/1076731614 >> v2:172.21.2.120:3300/0 conn(0x7fd8ec0d1880 msgr2=0x7fd8ec1b04d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:36:53.384 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:53.381 7fd8ead73700 1 --2- 172.21.2.112:0/1076731614 >> v2:172.21.2.120:3300/0 conn(0x7fd8ec0d1880 0x7fd8ec1b04d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:53.384 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:53.381 7fd8ead73700 1 -- 172.21.2.112:0/1076731614 >> v1:172.21.2.120:6789/0 conn(0x7fd8ec0d1c80 legacy=0x7fd8ec108f70 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:36:53.384 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:53.381 7fd8ead73700 1 -- 172.21.2.112:0/1076731614 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8ec088070 con 0x7fd8ec1b0af0 2021-02-10T11:36:53.384 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:53.381 7fd8ead73700 1 --2- 172.21.2.112:0/1076731614 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd8ec1b0af0 0x7fd8ec108950 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 rx=0x7fd8dc006700 tx=0x7fd8dc009b00).ready entity=mon.0 client_cookie=ea5b09d8d45e649b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:53.384 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:53.382 7fd8ea572700 1 -- 172.21.2.112:0/1076731614 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd8dc00c0c0 con 0x7fd8ec1b0af0 2021-02-10T11:36:53.384 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:53.382 7fd8ea572700 1 -- 172.21.2.112:0/1076731614 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8dc00c400 con 0x7fd8ec1b0af0 2021-02-10T11:36:53.384 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:53.382 7fd8ea572700 1 -- 172.21.2.112:0/1076731614 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd8dc00c920 con 0x7fd8ec1b0af0 2021-02-10T11:36:53.384 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:53.382 7fd8f186d700 1 -- 172.21.2.112:0/1076731614 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd8ec1b0af0 msgr2=0x7fd8ec108950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:53.384 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:53.382 7fd8f186d700 1 --2- 172.21.2.112:0/1076731614 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd8ec1b0af0 0x7fd8ec108950 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 rx=0x7fd8dc006700 tx=0x7fd8dc009b00).stop 2021-02-10T11:36:53.385 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:53.382 7fd8f186d700 1 -- 172.21.2.112:0/1076731614 shutdown_connections 2021-02-10T11:36:53.385 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:53.382 7fd8f186d700 1 --2- 172.21.2.112:0/1076731614 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd8ec1b0af0 0x7fd8ec108950 unknown :-1 s=CLOSED pgs=1686 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:53.385 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:53.382 7fd8f186d700 1 --2- 172.21.2.112:0/1076731614 >> v2:172.21.2.120:3300/0 conn(0x7fd8ec0d1880 0x7fd8ec1b04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:53.385 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:53.382 7fd8f186d700 1 -- 172.21.2.112:0/1076731614 >> 172.21.2.112:0/1076731614 conn(0x7fd8ec0ce430 msgr2=0x7fd8ec0d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:53.385 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:53.383 7fd8f186d700 1 -- 172.21.2.112:0/1076731614 shutdown_connections 2021-02-10T11:36:53.385 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:53.383 7fd8f186d700 1 -- 172.21.2.112:0/1076731614 wait complete. 2021-02-10T11:36:53.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:53.397 7fd8f186d700 1 Processor -- start 2021-02-10T11:36:53.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:53.397 7fd8f186d700 1 -- start start 2021-02-10T11:36:53.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:53.397 7fd8f186d700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd8ec1add90 0x7fd8ec1b0630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:53.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:53.397 7fd8f186d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd8ec19b180 0x7fd8ec0b7aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:53.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:53.397 7fd8f186d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd8ec19b580 0x7fd8ec0b80c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:53.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:53.397 7fd8f186d700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8ec1ae190 con 0x7fd8ec19b180 2021-02-10T11:36:53.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:53.397 7fd8f186d700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8ec0b86e0 con 0x7fd8ec1add90 2021-02-10T11:36:53.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:53.397 7fd8f186d700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8ec0b8900 con 0x7fd8ec19b580 2021-02-10T11:36:53.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:53.398 7fd8ead73700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd8ec19b180 0x7fd8ec0b7aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:53.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:53.398 7fd8ead73700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd8ec19b180 0x7fd8ec0b7aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:52408/0 (socket says 172.21.2.112:52408) 2021-02-10T11:36:53.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:53.398 7fd8ead73700 1 -- 172.21.2.112:0/721600760 learned_addr learned my addr 172.21.2.112:0/721600760 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:36:53.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:53.398 7fd8eb574700 1 --2- 172.21.2.112:0/721600760 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd8ec1add90 0x7fd8ec1b0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:53.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:53.398 7fd8ebd75700 1 --2- 172.21.2.112:0/721600760 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd8ec19b580 0x7fd8ec0b80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:53.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:53.398 7fd8ead73700 1 -- 172.21.2.112:0/721600760 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd8ec19b580 msgr2=0x7fd8ec0b80c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:53.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:53.398 7fd8ead73700 1 --2- 172.21.2.112:0/721600760 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd8ec19b580 0x7fd8ec0b80c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:53.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:53.398 7fd8ead73700 1 -- 172.21.2.112:0/721600760 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd8ec1add90 msgr2=0x7fd8ec1b0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:53.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:53.398 7fd8ead73700 1 --2- 172.21.2.112:0/721600760 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd8ec1add90 0x7fd8ec1b0630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:53.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:53.398 7fd8ead73700 1 -- 172.21.2.112:0/721600760 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8ec19c040 con 0x7fd8ec19b180 2021-02-10T11:36:53.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:53.398 7fd8ead73700 1 --2- 172.21.2.112:0/721600760 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd8ec19b180 0x7fd8ec0b7aa0 secure :-1 s=READY pgs=1687 cs=0 l=1 rev1=1 rx=0x7fd8dc00b660 tx=0x7fd8dc00ba00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:53.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:53.399 7fd8e3fff700 1 -- 172.21.2.112:0/721600760 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd8dc009440 con 0x7fd8ec19b180 2021-02-10T11:36:53.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:53.399 7fd8e3fff700 1 -- 172.21.2.112:0/721600760 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8dc000c90 con 0x7fd8ec19b180 2021-02-10T11:36:53.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:53.399 7fd8e3fff700 1 -- 172.21.2.112:0/721600760 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd8dc002030 con 0x7fd8ec19b180 2021-02-10T11:36:53.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:53.399 7fd8f186d700 1 -- 172.21.2.112:0/721600760 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8ec19c040 con 0x7fd8ec19b180 2021-02-10T11:36:53.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:53.399 7fd8f186d700 1 -- 172.21.2.112:0/721600760 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8ec19c5d0 con 0x7fd8ec19b180 2021-02-10T11:36:53.389 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:53.399 7fd8e3fff700 1 -- 172.21.2.112:0/721600760 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd8dc0093e0 con 0x7fd8ec19b180 2021-02-10T11:36:53.389 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:53.400 7fd8e3fff700 1 --2- 172.21.2.112:0/721600760 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd8d00186c0 0x7fd8d001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:53.389 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:53.400 7fd8e3fff700 1 -- 172.21.2.112:0/721600760 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5457+0+0 (secure 0 0 0) 0x7fd8dc032380 con 0x7fd8ec19b180 2021-02-10T11:36:53.389 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:53.400 7fd8eb574700 1 --2- 172.21.2.112:0/721600760 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd8d00186c0 0x7fd8d001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:53.389 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:53.400 7fd8eb574700 1 --2- 172.21.2.112:0/721600760 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd8d00186c0 0x7fd8d001abc0 crc :-1 s=READY pgs=803 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:53.418 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:53.431 7fd8f186d700 1 -- 172.21.2.112:0/721600760 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8ec0fe1d0 con 0x7fd8ec19b180 2021-02-10T11:36:53.420 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:53.434 7fd8e3fff700 1 -- 172.21.2.112:0/721600760 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd8dc030d20 con 0x7fd8ec19b180 2021-02-10T11:36:53.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:53.562 7fd8f186d700 1 -- 172.21.2.112:0/721600760 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"} v 0) v1 -- 0x7fd8ec0fe1d0 con 0x7fd8ec19b180 2021-02-10T11:36:53.553 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:53.566 7fd8e3fff700 1 -- 172.21.2.112:0/721600760 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"}]=0 new fs with metadata pool 16 and data pool 17 v61) v1 ==== 189+0+0 (secure 0 0 0) 0x7fd8dc0093e0 con 0x7fd8ec19b180 2021-02-10T11:36:53.556 INFO:teuthology.orchestra.run.gibba012.stderr:new fs with metadata pool 16 and data pool 17 2021-02-10T11:36:53.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:53.573 7fd8f186d700 1 -- 172.21.2.112:0/721600760 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd8d00186c0 msgr2=0x7fd8d001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:53.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:53.573 7fd8f186d700 1 --2- 172.21.2.112:0/721600760 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd8d00186c0 0x7fd8d001abc0 crc :-1 s=READY pgs=803 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:53.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:53.573 7fd8f186d700 1 -- 172.21.2.112:0/721600760 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd8ec19b180 msgr2=0x7fd8ec0b7aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:53.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:53.573 7fd8f186d700 1 --2- 172.21.2.112:0/721600760 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd8ec19b180 0x7fd8ec0b7aa0 secure :-1 s=READY pgs=1687 cs=0 l=1 rev1=1 rx=0x7fd8dc00b660 tx=0x7fd8dc00ba00).stop 2021-02-10T11:36:53.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:53.573 7fd8f186d700 1 -- 172.21.2.112:0/721600760 shutdown_connections 2021-02-10T11:36:53.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:53.573 7fd8f186d700 1 --2- 172.21.2.112:0/721600760 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd8d00186c0 0x7fd8d001abc0 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:53.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:53.573 7fd8f186d700 1 --2- 172.21.2.112:0/721600760 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd8ec19b580 0x7fd8ec0b80c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:53.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:53.573 7fd8f186d700 1 --2- 172.21.2.112:0/721600760 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd8ec19b180 0x7fd8ec0b7aa0 unknown :-1 s=CLOSED pgs=1687 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:53.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:53.573 7fd8f186d700 1 --2- 172.21.2.112:0/721600760 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd8ec1add90 0x7fd8ec1b0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:53.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:53.573 7fd8f186d700 1 -- 172.21.2.112:0/721600760 >> 172.21.2.112:0/721600760 conn(0x7fd8ec19f940 msgr2=0x7fd8ec1ab890 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:53.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:53.573 7fd8f186d700 1 -- 172.21.2.112:0/721600760 shutdown_connections 2021-02-10T11:36:53.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:53.573 7fd8f186d700 1 -- 172.21.2.112:0/721600760 wait complete. 2021-02-10T11:36:53.567 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:36:53.992 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:53.990 7f25c4a6a700 1 Processor -- start 2021-02-10T11:36:53.993 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:53.990 7f25c4a6a700 1 -- start start 2021-02-10T11:36:53.993 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:53.990 7f25c4a6a700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f25c00d0340 0x7f25c00ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:53.993 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:53.990 7f25c4a6a700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25c00f0b10 con 0x7f25c00d1c70 2021-02-10T11:36:53.993 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:53.990 7f25c4a6a700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25c01b0be0 con 0x7f25c00d1870 2021-02-10T11:36:53.993 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:53.990 7f25c4a6a700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25c00e5ed0 con 0x7f25c00d0340 2021-02-10T11:36:53.993 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:53.990 7f25be05d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f25c00d0340 0x7f25c00ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:53.993 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:53.990 7f25bf05f700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f25c00d1c70 0x7f25c00efc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:51880/0 (socket says 172.21.2.112:51880) 2021-02-10T11:36:53.993 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:53.990 7f25bf05f700 1 -- 172.21.2.112:0/2974056290 learned_addr learned my addr 172.21.2.112:0/2974056290 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:36:53.993 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:53.991 7f25be05d700 1 -- 172.21.2.112:0/2974056290 >> v1:172.21.2.120:6789/0 conn(0x7f25c00d1870 legacy=0x7f25c01b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:53.993 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:53.991 7f25be05d700 1 -- 172.21.2.112:0/2974056290 >> v1:172.21.2.112:6789/0 conn(0x7f25c00d1c70 legacy=0x7f25c00efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:53.993 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:53.991 7f25be05d700 1 -- 172.21.2.112:0/2974056290 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25c00f0450 con 0x7f25c00d0340 2021-02-10T11:36:53.994 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:53.991 7f25be05d700 1 --2- 172.21.2.112:0/2974056290 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f25c00d0340 0x7f25c00ef640 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 rx=0x7f25b0006650 tx=0x7f25b0009a00).ready entity=mon.0 client_cookie=78c085c0ec1a30e5 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:53.994 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:53.991 7f25bd85c700 1 -- 172.21.2.112:0/2974056290 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f25b0009df0 con 0x7f25c00d0340 2021-02-10T11:36:53.994 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:53.991 7f25bd85c700 1 -- 172.21.2.112:0/2974056290 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25b00009b0 con 0x7f25c00d0340 2021-02-10T11:36:53.994 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:53.991 7f25bd85c700 1 -- 172.21.2.112:0/2974056290 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f25b000c2b0 con 0x7f25c00d0340 2021-02-10T11:36:53.994 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:53.991 7f25c4a6a700 1 -- 172.21.2.112:0/2974056290 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f25c00d0340 msgr2=0x7f25c00ef640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:53.994 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:53.991 7f25c4a6a700 1 --2- 172.21.2.112:0/2974056290 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f25c00d0340 0x7f25c00ef640 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 rx=0x7f25b0006650 tx=0x7f25b0009a00).stop 2021-02-10T11:36:53.994 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:53.992 7f25c4a6a700 1 -- 172.21.2.112:0/2974056290 shutdown_connections 2021-02-10T11:36:53.994 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:53.992 7f25c4a6a700 1 --2- 172.21.2.112:0/2974056290 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f25c00d0340 0x7f25c00ef640 unknown :-1 s=CLOSED pgs=1689 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:53.994 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:53.992 7f25c4a6a700 1 -- 172.21.2.112:0/2974056290 >> 172.21.2.112:0/2974056290 conn(0x7f25c00ce450 msgr2=0x7f25c00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:53.994 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:53.992 7f25c4a6a700 1 -- 172.21.2.112:0/2974056290 shutdown_connections 2021-02-10T11:36:53.994 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:53.992 7f25c4a6a700 1 -- 172.21.2.112:0/2974056290 wait complete. 2021-02-10T11:36:53.996 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:54.007 7f25c4a6a700 1 Processor -- start 2021-02-10T11:36:53.996 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:54.007 7f25c4a6a700 1 -- start start 2021-02-10T11:36:53.996 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:54.007 7f25c4a6a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f25c00b32d0 0x7f25c00b11e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:53.996 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:54.007 7f25c4a6a700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f25c00b1800 0x7f25c00aaa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:53.996 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:54.007 7f25c4a6a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f25c00ab090 0x7f25c00ab490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:53.996 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:54.007 7f25c4a6a700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25c00b1d10 con 0x7f25c00b32d0 2021-02-10T11:36:53.996 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:54.007 7f25c4a6a700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25c00b1f30 con 0x7f25c00b1800 2021-02-10T11:36:53.996 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:54.007 7f25c4a6a700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25c00b2150 con 0x7f25c00ab090 2021-02-10T11:36:53.997 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:54.007 7f25be85e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f25c00b32d0 0x7f25c00b11e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:53.997 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:54.007 7f25be85e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f25c00b32d0 0x7f25c00b11e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:52422/0 (socket says 172.21.2.112:52422) 2021-02-10T11:36:53.997 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:54.007 7f25be85e700 1 -- 172.21.2.112:0/416990525 learned_addr learned my addr 172.21.2.112:0/416990525 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:36:53.997 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:54.008 7f25be85e700 1 -- 172.21.2.112:0/416990525 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f25c00ab090 msgr2=0x7f25c00ab490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:53.997 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:54.008 7f25be85e700 1 --2- 172.21.2.112:0/416990525 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f25c00ab090 0x7f25c00ab490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:53.997 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:54.008 7f25be85e700 1 -- 172.21.2.112:0/416990525 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f25c00b1800 msgr2=0x7f25c00aaa70 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:36:53.997 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:54.008 7f25be85e700 1 --2- 172.21.2.112:0/416990525 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f25c00b1800 0x7f25c00aaa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:53.997 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:54.008 7f25be85e700 1 -- 172.21.2.112:0/416990525 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25c00b2370 con 0x7f25c00b32d0 2021-02-10T11:36:53.997 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:54.008 7f25be85e700 1 --2- 172.21.2.112:0/416990525 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f25c00b32d0 0x7f25c00b11e0 secure :-1 s=READY pgs=1690 cs=0 l=1 rev1=1 rx=0x7f25a800b530 tx=0x7f25a800b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:53.997 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:54.008 7f25b77fe700 1 -- 172.21.2.112:0/416990525 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f25a800bcc0 con 0x7f25c00b32d0 2021-02-10T11:36:53.997 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:54.008 7f25b77fe700 1 -- 172.21.2.112:0/416990525 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25a8006020 con 0x7f25c00b32d0 2021-02-10T11:36:53.997 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:54.008 7f25b77fe700 1 -- 172.21.2.112:0/416990525 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f25a80054a0 con 0x7f25c00b32d0 2021-02-10T11:36:53.997 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:54.008 7f25c4a6a700 1 -- 172.21.2.112:0/416990525 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25c00b2370 con 0x7f25c00b32d0 2021-02-10T11:36:53.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:54.008 7f25c4a6a700 1 -- 172.21.2.112:0/416990525 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25c00b2370 con 0x7f25c00b32d0 2021-02-10T11:36:53.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:54.009 7f25b77fe700 1 -- 172.21.2.112:0/416990525 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f25a8006200 con 0x7f25c00b32d0 2021-02-10T11:36:53.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:54.009 7f25b77fe700 1 --2- 172.21.2.112:0/416990525 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f25a40186c0 0x7f25a401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:53.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:54.009 7f25b77fe700 1 -- 172.21.2.112:0/416990525 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f25a8032590 con 0x7f25c00b32d0 2021-02-10T11:36:53.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:54.009 7f25be05d700 1 --2- 172.21.2.112:0/416990525 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f25a40186c0 0x7f25a401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:53.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:54.010 7f25be05d700 1 --2- 172.21.2.112:0/416990525 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f25a40186c0 0x7f25a401abc0 crc :-1 s=READY pgs=804 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:54.027 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:54.040 7f25c4a6a700 1 -- 172.21.2.112:0/416990525 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25c00ffb10 con 0x7f25c00b32d0 2021-02-10T11:36:54.029 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:54.043 7f25b77fe700 1 -- 172.21.2.112:0/416990525 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f25a802def0 con 0x7f25c00b32d0 2021-02-10T11:36:54.154 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:54.168 7f25c4a6a700 1 -- 172.21.2.112:0/416990525 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f25c00ffb10 con 0x7f25c00b32d0 2021-02-10T11:36:54.155 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:54.169 7f25b77fe700 1 -- 172.21.2.112:0/416990525 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+14009 (secure 0 0 0) 0x7f25a802def0 con 0x7f25c00b32d0 2021-02-10T11:36:54.155 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:36:54.155 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":82,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:36:53.563650","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:36:51.538038","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":17,"pool_name":"cephfs_data","create_time":"2021-02-10 11:36:52.609006","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:36:54.157 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:54.170 7f25c4a6a700 1 -- 172.21.2.112:0/416990525 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f25a40186c0 msgr2=0x7f25a401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:54.157 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:54.170 7f25c4a6a700 1 --2- 172.21.2.112:0/416990525 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f25a40186c0 0x7f25a401abc0 crc :-1 s=READY pgs=804 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:54.157 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:54.170 7f25c4a6a700 1 -- 172.21.2.112:0/416990525 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f25c00b32d0 msgr2=0x7f25c00b11e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:54.157 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:54.170 7f25c4a6a700 1 --2- 172.21.2.112:0/416990525 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f25c00b32d0 0x7f25c00b11e0 secure :-1 s=READY pgs=1690 cs=0 l=1 rev1=1 rx=0x7f25a800b530 tx=0x7f25a800b8d0).stop 2021-02-10T11:36:54.157 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:54.170 7f25c4a6a700 1 -- 172.21.2.112:0/416990525 shutdown_connections 2021-02-10T11:36:54.157 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:54.170 7f25c4a6a700 1 --2- 172.21.2.112:0/416990525 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f25a40186c0 0x7f25a401abc0 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:54.157 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:54.170 7f25c4a6a700 1 --2- 172.21.2.112:0/416990525 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f25c00b1800 0x7f25c00aaa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:54.157 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:54.170 7f25c4a6a700 1 --2- 172.21.2.112:0/416990525 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f25c00ab090 0x7f25c00ab490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:54.158 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:54.170 7f25c4a6a700 1 --2- 172.21.2.112:0/416990525 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f25c00b32d0 0x7f25c00b11e0 unknown :-1 s=CLOSED pgs=1690 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:54.158 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:54.170 7f25c4a6a700 1 -- 172.21.2.112:0/416990525 >> 172.21.2.112:0/416990525 conn(0x7f25c010a820 msgr2=0x7f25c0101c00 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:54.158 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:54.170 7f25c4a6a700 1 -- 172.21.2.112:0/416990525 shutdown_connections 2021-02-10T11:36:54.158 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:54.171 7f25c4a6a700 1 -- 172.21.2.112:0/416990525 wait complete. 2021-02-10T11:36:54.165 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:36:54.594 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:54.591 7f6fae2bb700 1 Processor -- start 2021-02-10T11:36:54.594 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:54.591 7f6fae2bb700 1 -- start start 2021-02-10T11:36:54.594 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:54.591 7f6fae2bb700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f6fa80d0340 0x7f6fa80ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:54.594 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:54.591 7f6fae2bb700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f6fa80d1c70 0x7f6fa80efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:54.594 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:54.591 7f6fae2bb700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fa80f0b10 con 0x7f6fa80d1870 2021-02-10T11:36:54.594 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:54.591 7f6fae2bb700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fa80e5ed0 con 0x7f6fa80d0340 2021-02-10T11:36:54.594 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:54.591 7f6fae2bb700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fa81b0be0 con 0x7f6fa80d1c70 2021-02-10T11:36:54.594 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:54.591 7f6fa77fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f6fa80d0340 0x7f6fa80ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:54.594 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:54.591 7f6fa77fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f6fa80d0340 0x7f6fa80ef640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:52428/0 (socket says 172.21.2.112:52428) 2021-02-10T11:36:54.594 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:54.591 7f6fa7fff700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f6fa80d1870 0x7f6fa81b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:51890/0 (socket says 172.21.2.112:51890) 2021-02-10T11:36:54.595 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:54.591 7f6fa77fe700 1 -- 172.21.2.112:0/2798789051 learned_addr learned my addr 172.21.2.112:0/2798789051 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:36:54.595 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:54.591 7f6fac8d8700 1 --2- 172.21.2.112:0/2798789051 >> v2:172.21.2.120:3300/0 conn(0x7f6fa80d1c70 0x7f6fa80efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:54.595 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:54.592 7f6fa77fe700 1 -- 172.21.2.112:0/2798789051 >> v2:172.21.2.120:3300/0 conn(0x7f6fa80d1c70 msgr2=0x7f6fa80efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:54.595 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:54.592 7f6fa77fe700 1 --2- 172.21.2.112:0/2798789051 >> v2:172.21.2.120:3300/0 conn(0x7f6fa80d1c70 0x7f6fa80efc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:54.595 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:54.592 7f6fa77fe700 1 -- 172.21.2.112:0/2798789051 >> v1:172.21.2.112:6789/0 conn(0x7f6fa80d1870 legacy=0x7f6fa81b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:54.595 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:54.592 7f6fa77fe700 1 -- 172.21.2.112:0/2798789051 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fa80f20a0 con 0x7f6fa80d0340 2021-02-10T11:36:54.595 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:54.592 7f6fa77fe700 1 --2- 172.21.2.112:0/2798789051 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6fa80d0340 0x7f6fa80ef640 secure :-1 s=READY pgs=1692 cs=0 l=1 rev1=1 rx=0x7f6f9c006700 tx=0x7f6f9c009b00).ready entity=mon.0 client_cookie=62ba9d03084b9962 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:54.595 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:54.592 7f6fa6ffd700 1 -- 172.21.2.112:0/2798789051 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6f9c00c0c0 con 0x7f6fa80d0340 2021-02-10T11:36:54.595 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:54.592 7f6fa6ffd700 1 -- 172.21.2.112:0/2798789051 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f9c00c400 con 0x7f6fa80d0340 2021-02-10T11:36:54.595 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:54.592 7f6fa6ffd700 1 -- 172.21.2.112:0/2798789051 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6f9c00c920 con 0x7f6fa80d0340 2021-02-10T11:36:54.595 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:54.592 7f6fae2bb700 1 -- 172.21.2.112:0/2798789051 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6fa80d0340 msgr2=0x7f6fa80ef640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:54.595 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:54.592 7f6fae2bb700 1 --2- 172.21.2.112:0/2798789051 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6fa80d0340 0x7f6fa80ef640 secure :-1 s=READY pgs=1692 cs=0 l=1 rev1=1 rx=0x7f6f9c006700 tx=0x7f6f9c009b00).stop 2021-02-10T11:36:54.595 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:54.593 7f6fae2bb700 1 -- 172.21.2.112:0/2798789051 shutdown_connections 2021-02-10T11:36:54.596 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:54.593 7f6fae2bb700 1 --2- 172.21.2.112:0/2798789051 >> v2:172.21.2.120:3300/0 conn(0x7f6fa80d1c70 0x7f6fa80efc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:54.596 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:54.593 7f6fae2bb700 1 --2- 172.21.2.112:0/2798789051 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6fa80d0340 0x7f6fa80ef640 unknown :-1 s=CLOSED pgs=1692 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:54.596 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:54.593 7f6fae2bb700 1 -- 172.21.2.112:0/2798789051 >> 172.21.2.112:0/2798789051 conn(0x7f6fa80ce450 msgr2=0x7f6fa80d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:54.596 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:54.593 7f6fae2bb700 1 -- 172.21.2.112:0/2798789051 shutdown_connections 2021-02-10T11:36:54.596 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:54.593 7f6fae2bb700 1 -- 172.21.2.112:0/2798789051 wait complete. 2021-02-10T11:36:54.597 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:54.608 7f6fae2bb700 1 Processor -- start 2021-02-10T11:36:54.597 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:54.608 7f6fae2bb700 1 -- start start 2021-02-10T11:36:54.597 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:54.608 7f6fae2bb700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6fa80ada90 0x7f6fa80ade90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:54.597 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:54.608 7f6fae2bb700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6fa8099d20 0x7f6fa809a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:54.598 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:54.608 7f6fae2bb700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6fa809a740 0x7f6fa8097bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:54.598 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:54.608 7f6fae2bb700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fa80997a0 con 0x7f6fa809a740 2021-02-10T11:36:54.598 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:54.608 7f6fae2bb700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fa818f0c0 con 0x7f6fa80ada90 2021-02-10T11:36:54.598 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:54.608 7f6fae2bb700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fa809ab40 con 0x7f6fa8099d20 2021-02-10T11:36:54.598 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:54.608 7f6fac8d8700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6fa809a740 0x7f6fa8097bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:54.598 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:54.608 7f6fac8d8700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6fa809a740 0x7f6fa8097bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:52438/0 (socket says 172.21.2.112:52438) 2021-02-10T11:36:54.598 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:54.608 7f6fac8d8700 1 -- 172.21.2.112:0/3322374383 learned_addr learned my addr 172.21.2.112:0/3322374383 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:36:54.598 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:54.608 7f6fa7fff700 1 --2- 172.21.2.112:0/3322374383 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6fa80ada90 0x7f6fa80ade90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:54.598 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:54.608 7f6fa77fe700 1 --2- 172.21.2.112:0/3322374383 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6fa8099d20 0x7f6fa809a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:54.598 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:54.608 7f6fac8d8700 1 -- 172.21.2.112:0/3322374383 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6fa8099d20 msgr2=0x7f6fa809a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:54.598 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:54.608 7f6fac8d8700 1 --2- 172.21.2.112:0/3322374383 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6fa8099d20 0x7f6fa809a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:54.598 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:54.608 7f6fac8d8700 1 -- 172.21.2.112:0/3322374383 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6fa80ada90 msgr2=0x7f6fa80ade90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:54.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:54.608 7f6fac8d8700 1 --2- 172.21.2.112:0/3322374383 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6fa80ada90 0x7f6fa80ade90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:54.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:54.608 7f6fac8d8700 1 -- 172.21.2.112:0/3322374383 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fa809ad60 con 0x7f6fa809a740 2021-02-10T11:36:54.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:54.609 7f6fac8d8700 1 --2- 172.21.2.112:0/3322374383 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6fa809a740 0x7f6fa8097bc0 secure :-1 s=READY pgs=1693 cs=0 l=1 rev1=1 rx=0x7f6fa000b580 tx=0x7f6fa000b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:54.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:54.609 7f6fa4ff9700 1 -- 172.21.2.112:0/3322374383 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6fa000bd10 con 0x7f6fa809a740 2021-02-10T11:36:54.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:54.609 7f6fa4ff9700 1 -- 172.21.2.112:0/3322374383 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fa0006720 con 0x7f6fa809a740 2021-02-10T11:36:54.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:54.609 7f6fae2bb700 1 -- 172.21.2.112:0/3322374383 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fa809ad60 con 0x7f6fa809a740 2021-02-10T11:36:54.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:54.609 7f6fa4ff9700 1 -- 172.21.2.112:0/3322374383 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6fa0002220 con 0x7f6fa809a740 2021-02-10T11:36:54.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:54.609 7f6fae2bb700 1 -- 172.21.2.112:0/3322374383 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fa809ad60 con 0x7f6fa809a740 2021-02-10T11:36:54.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:54.609 7f6fa4ff9700 1 -- 172.21.2.112:0/3322374383 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6fa0005330 con 0x7f6fa809a740 2021-02-10T11:36:54.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:54.609 7f6fa4ff9700 1 --2- 172.21.2.112:0/3322374383 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6f88018730 0x7f6f8801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:54.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:54.609 7f6fa4ff9700 1 -- 172.21.2.112:0/3322374383 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f6fa00326a0 con 0x7f6fa809a740 2021-02-10T11:36:54.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:54.609 7f6fa7fff700 1 --2- 172.21.2.112:0/3322374383 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6f88018730 0x7f6f8801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:54.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:54.610 7f6fa7fff700 1 --2- 172.21.2.112:0/3322374383 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6f88018730 0x7f6f8801ac10 crc :-1 s=READY pgs=805 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:54.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:54.640 7f6fae2bb700 1 -- 172.21.2.112:0/3322374383 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fa80ebc00 con 0x7f6fa809a740 2021-02-10T11:36:54.629 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:54.643 7f6fa4ff9700 1 -- 172.21.2.112:0/3322374383 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6fa0006370 con 0x7f6fa809a740 2021-02-10T11:36:54.757 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:54.770 7f6fae2bb700 1 -- 172.21.2.112:0/3322374383 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6fa80ebc00 con 0x7f6fa809a740 2021-02-10T11:36:54.757 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:54.771 7f6fa4ff9700 1 -- 172.21.2.112:0/3322374383 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) v1 ==== 74+0+13996 (secure 0 0 0) 0x7f6fa0006720 con 0x7f6fa809a740 2021-02-10T11:36:54.758 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:36:54.758 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":83,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:36:54.566363","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:36:51.538038","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":17,"pool_name":"cephfs_data","create_time":"2021-02-10 11:36:52.609006","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:36:54.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:54.772 7f6fae2bb700 1 -- 172.21.2.112:0/3322374383 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6f88018730 msgr2=0x7f6f8801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:54.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:54.772 7f6fae2bb700 1 --2- 172.21.2.112:0/3322374383 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6f88018730 0x7f6f8801ac10 crc :-1 s=READY pgs=805 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:54.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:54.772 7f6fae2bb700 1 -- 172.21.2.112:0/3322374383 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6fa809a740 msgr2=0x7f6fa8097bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:54.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:54.772 7f6fae2bb700 1 --2- 172.21.2.112:0/3322374383 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6fa809a740 0x7f6fa8097bc0 secure :-1 s=READY pgs=1693 cs=0 l=1 rev1=1 rx=0x7f6fa000b580 tx=0x7f6fa000b920).stop 2021-02-10T11:36:54.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:54.773 7f6fae2bb700 1 -- 172.21.2.112:0/3322374383 shutdown_connections 2021-02-10T11:36:54.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:54.773 7f6fae2bb700 1 --2- 172.21.2.112:0/3322374383 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6f88018730 0x7f6f8801ac10 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:54.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:54.773 7f6fae2bb700 1 --2- 172.21.2.112:0/3322374383 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6fa809a740 0x7f6fa8097bc0 unknown :-1 s=CLOSED pgs=1693 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:54.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:54.773 7f6fae2bb700 1 --2- 172.21.2.112:0/3322374383 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6fa8099d20 0x7f6fa809a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:54.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:54.773 7f6fae2bb700 1 --2- 172.21.2.112:0/3322374383 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6fa80ada90 0x7f6fa80ade90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:54.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:54.773 7f6fae2bb700 1 -- 172.21.2.112:0/3322374383 >> 172.21.2.112:0/3322374383 conn(0x7f6fa80f0550 msgr2=0x7f6fa8191730 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:54.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:54.773 7f6fae2bb700 1 -- 172.21.2.112:0/3322374383 shutdown_connections 2021-02-10T11:36:54.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:54.773 7f6fae2bb700 1 -- 172.21.2.112:0/3322374383 wait complete. 2021-02-10T11:36:54.768 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2021-02-10T11:36:55.199 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:55.196 7f4807cd2700 1 Processor -- start 2021-02-10T11:36:55.199 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:55.196 7f4807cd2700 1 -- start start 2021-02-10T11:36:55.200 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:55.197 7f4807cd2700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f48000d1860 0x7f48001b04b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:55.200 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:55.197 7f4807cd2700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f48001b0ad0 0x7f4800108930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:55.200 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:55.197 7f4807cd2700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48000fe1b0 con 0x7f48000d1c60 2021-02-10T11:36:55.200 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:55.197 7f4807cd2700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4800109a50 con 0x7f48000d1860 2021-02-10T11:36:55.200 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:55.197 7f4807cd2700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4800109740 con 0x7f48001b0ad0 2021-02-10T11:36:55.200 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:55.197 7f4805aee700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f48000d1860 0x7f48001b04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:55.200 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:55.197 7f4805aee700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f48000d1860 0x7f48001b04b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:52442/0 (socket says 172.21.2.112:52442) 2021-02-10T11:36:55.200 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:55.197 7f4805aee700 1 -- 172.21.2.112:0/3042295747 learned_addr learned my addr 172.21.2.112:0/3042295747 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:36:55.200 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:55.197 7f48052ed700 1 --2- 172.21.2.112:0/3042295747 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f48001b0ad0 0x7f4800108930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:55.200 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:55.197 7f4805aee700 1 -- 172.21.2.112:0/3042295747 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f48001b0ad0 msgr2=0x7f4800108930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:55.200 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:55.197 7f4805aee700 1 --2- 172.21.2.112:0/3042295747 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f48001b0ad0 0x7f4800108930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:55.200 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:55.197 7f4805aee700 1 -- 172.21.2.112:0/3042295747 >> v1:172.21.2.112:6789/0 conn(0x7f48000d1c60 legacy=0x7f4800108f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:55.201 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:55.197 7f4805aee700 1 -- 172.21.2.112:0/3042295747 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4800088070 con 0x7f48000d1860 2021-02-10T11:36:55.201 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:55.197 7f4805aee700 1 --2- 172.21.2.112:0/3042295747 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f48000d1860 0x7f48001b04b0 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 rx=0x7f47f0006700 tx=0x7f47f0009b80).ready entity=mon.0 client_cookie=a617bcf428903217 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:55.201 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:55.198 7f4804aec700 1 -- 172.21.2.112:0/3042295747 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f47f000c0c0 con 0x7f48000d1860 2021-02-10T11:36:55.201 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:55.198 7f4804aec700 1 -- 172.21.2.112:0/3042295747 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47f000c440 con 0x7f48000d1860 2021-02-10T11:36:55.201 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:55.198 7f4804aec700 1 -- 172.21.2.112:0/3042295747 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f47f000c960 con 0x7f48000d1860 2021-02-10T11:36:55.201 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:55.198 7f4807cd2700 1 -- 172.21.2.112:0/3042295747 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f48000d1860 msgr2=0x7f48001b04b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:55.201 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:55.198 7f4807cd2700 1 --2- 172.21.2.112:0/3042295747 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f48000d1860 0x7f48001b04b0 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 rx=0x7f47f0006700 tx=0x7f47f0009b80).stop 2021-02-10T11:36:55.201 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:55.198 7f4807cd2700 1 -- 172.21.2.112:0/3042295747 shutdown_connections 2021-02-10T11:36:55.201 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:55.198 7f4807cd2700 1 --2- 172.21.2.112:0/3042295747 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f48001b0ad0 0x7f4800108930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:55.201 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:55.198 7f4807cd2700 1 --2- 172.21.2.112:0/3042295747 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f48000d1860 0x7f48001b04b0 unknown :-1 s=CLOSED pgs=1695 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:55.201 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:55.198 7f4807cd2700 1 -- 172.21.2.112:0/3042295747 >> 172.21.2.112:0/3042295747 conn(0x7f48000ce410 msgr2=0x7f48000d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:55.202 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:55.199 7f4807cd2700 1 -- 172.21.2.112:0/3042295747 shutdown_connections 2021-02-10T11:36:55.202 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:55.199 7f4807cd2700 1 -- 172.21.2.112:0/3042295747 wait complete. 2021-02-10T11:36:55.203 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:55.213 7f4807cd2700 1 Processor -- start 2021-02-10T11:36:55.203 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:55.213 7f4807cd2700 1 -- start start 2021-02-10T11:36:55.203 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:55.214 7f4807cd2700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f48000b50e0 0x7f48000b4540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:55.203 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:55.214 7f4807cd2700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f48000b64a0 0x7f48000b68a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:55.203 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:55.214 7f4807cd2700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f48000c2fc0 0x7f48000c33c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:55.203 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:55.214 7f4807cd2700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48000b4b60 con 0x7f48000b64a0 2021-02-10T11:36:55.204 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:55.214 7f4807cd2700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48000b4d80 con 0x7f48000b50e0 2021-02-10T11:36:55.204 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:55.214 7f4807cd2700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48000b5590 con 0x7f48000c2fc0 2021-02-10T11:36:55.204 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:55.214 7f48052ed700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f48000b64a0 0x7f48000b68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:55.204 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:55.214 7f48052ed700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f48000b64a0 0x7f48000b68a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:52450/0 (socket says 172.21.2.112:52450) 2021-02-10T11:36:55.204 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:55.214 7f48052ed700 1 -- 172.21.2.112:0/3893078084 learned_addr learned my addr 172.21.2.112:0/3893078084 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:36:55.204 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:55.214 7f4805aee700 1 --2- 172.21.2.112:0/3893078084 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f48000b50e0 0x7f48000b4540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:55.204 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:55.214 7f48062ef700 1 --2- 172.21.2.112:0/3893078084 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f48000c2fc0 0x7f48000c33c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:55.204 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:55.214 7f48052ed700 1 -- 172.21.2.112:0/3893078084 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f48000c2fc0 msgr2=0x7f48000c33c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:55.204 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:55.214 7f48052ed700 1 --2- 172.21.2.112:0/3893078084 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f48000c2fc0 0x7f48000c33c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:55.204 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:55.214 7f48052ed700 1 -- 172.21.2.112:0/3893078084 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f48000b50e0 msgr2=0x7f48000b4540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:55.204 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:55.214 7f48052ed700 1 --2- 172.21.2.112:0/3893078084 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f48000b50e0 0x7f48000b4540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:55.204 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:55.214 7f48052ed700 1 -- 172.21.2.112:0/3893078084 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48000b57b0 con 0x7f48000b64a0 2021-02-10T11:36:55.204 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:55.214 7f48052ed700 1 --2- 172.21.2.112:0/3893078084 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f48000b64a0 0x7f48000b68a0 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 rx=0x7f47f80099e0 tx=0x7f47f800bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:55.205 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:55.215 7f47f67fc700 1 -- 172.21.2.112:0/3893078084 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f47f800c2f0 con 0x7f48000b64a0 2021-02-10T11:36:55.205 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:55.215 7f47f67fc700 1 -- 172.21.2.112:0/3893078084 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47f800c670 con 0x7f48000b64a0 2021-02-10T11:36:55.205 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:55.215 7f47f67fc700 1 -- 172.21.2.112:0/3893078084 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f47f800cb90 con 0x7f48000b64a0 2021-02-10T11:36:55.205 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:55.215 7f4807cd2700 1 -- 172.21.2.112:0/3893078084 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48000b57b0 con 0x7f48000b64a0 2021-02-10T11:36:55.205 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:55.215 7f4807cd2700 1 -- 172.21.2.112:0/3893078084 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48000b57b0 con 0x7f48000b64a0 2021-02-10T11:36:55.205 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:55.215 7f47f67fc700 1 -- 172.21.2.112:0/3893078084 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f47f800a020 con 0x7f48000b64a0 2021-02-10T11:36:55.205 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:55.215 7f47f67fc700 1 --2- 172.21.2.112:0/3893078084 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f47e80186c0 0x7f47e801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:55.205 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:55.215 7f47f67fc700 1 -- 172.21.2.112:0/3893078084 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f47f8032b40 con 0x7f48000b64a0 2021-02-10T11:36:55.205 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:55.216 7f4805aee700 1 --2- 172.21.2.112:0/3893078084 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f47e80186c0 0x7f47e801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:55.205 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:55.216 7f4805aee700 1 --2- 172.21.2.112:0/3893078084 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f47e80186c0 0x7f47e801abc0 crc :-1 s=READY pgs=806 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:55.233 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:55.247 7f4807cd2700 1 -- 172.21.2.112:0/3893078084 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4800082a20 con 0x7f48000b64a0 2021-02-10T11:36:55.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:55.249 7f47f67fc700 1 -- 172.21.2.112:0/3893078084 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f47f800c2f0 con 0x7f48000b64a0 2021-02-10T11:36:55.374 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:55.387 7f4807cd2700 1 -- 172.21.2.112:0/3893078084 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"} v 0) v1 -- 0x7f4800082a20 con 0x7f48000b64a0 2021-02-10T11:36:55.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:55.575 7f47f67fc700 1 -- 172.21.2.112:0/3893078084 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"}]=0 v64) v1 ==== 118+0+0 (secure 0 0 0) 0x7f47f8009fe0 con 0x7f48000b64a0 2021-02-10T11:36:55.590 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:55.604 7f4807cd2700 1 -- 172.21.2.112:0/3893078084 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f47e80186c0 msgr2=0x7f47e801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:55.591 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:55.604 7f4807cd2700 1 --2- 172.21.2.112:0/3893078084 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f47e80186c0 0x7f47e801abc0 crc :-1 s=READY pgs=806 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:55.591 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:55.604 7f4807cd2700 1 -- 172.21.2.112:0/3893078084 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f48000b64a0 msgr2=0x7f48000b68a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:55.591 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:55.604 7f4807cd2700 1 --2- 172.21.2.112:0/3893078084 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f48000b64a0 0x7f48000b68a0 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 rx=0x7f47f80099e0 tx=0x7f47f800bd40).stop 2021-02-10T11:36:55.591 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:55.604 7f4807cd2700 1 -- 172.21.2.112:0/3893078084 shutdown_connections 2021-02-10T11:36:55.591 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:55.604 7f4807cd2700 1 --2- 172.21.2.112:0/3893078084 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f47e80186c0 0x7f47e801abc0 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:55.591 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:55.604 7f4807cd2700 1 --2- 172.21.2.112:0/3893078084 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f48000c2fc0 0x7f48000c33c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:55.591 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:55.604 7f4807cd2700 1 --2- 172.21.2.112:0/3893078084 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f48000b64a0 0x7f48000b68a0 unknown :-1 s=CLOSED pgs=1696 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:55.592 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:55.604 7f4807cd2700 1 --2- 172.21.2.112:0/3893078084 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f48000b50e0 0x7f48000b4540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:55.592 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:55.604 7f4807cd2700 1 -- 172.21.2.112:0/3893078084 >> 172.21.2.112:0/3893078084 conn(0x7f48000bee90 msgr2=0x7f4800190670 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:55.592 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:55.605 7f4807cd2700 1 -- 172.21.2.112:0/3893078084 shutdown_connections 2021-02-10T11:36:55.592 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:55.605 7f4807cd2700 1 -- 172.21.2.112:0/3893078084 wait complete. 2021-02-10T11:36:55.600 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:36:56.026 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.023 7fbd4f78d700 1 Processor -- start 2021-02-10T11:36:56.027 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.023 7fbd4f78d700 1 -- start start 2021-02-10T11:36:56.027 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.023 7fbd4f78d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbd480d1c70 0x7fbd480f20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:56.027 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.023 7fbd4f78d700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd480f0b10 con 0x7fbd480d0340 2021-02-10T11:36:56.027 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.023 7fbd4f78d700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd481b0be0 con 0x7fbd480d1870 2021-02-10T11:36:56.027 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.023 7fbd4f78d700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd480e5ed0 con 0x7fbd480d1c70 2021-02-10T11:36:56.027 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.023 7fbd4cda8700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fbd480d0340 0x7fbd480ef640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:51920/0 (socket says 172.21.2.112:51920) 2021-02-10T11:36:56.027 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.023 7fbd4cda8700 1 -- 172.21.2.112:0/4226706302 learned_addr learned my addr 172.21.2.112:0/4226706302 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:36:56.027 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.024 7fbd3ffff700 1 -- 172.21.2.112:0/4226706302 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2800436504 0 0) 0x7fbd380010b0 con 0x7fbd480d0340 2021-02-10T11:36:56.027 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.024 7fbd3ffff700 1 -- 172.21.2.112:0/4226706302 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd34002fe0 con 0x7fbd480d0340 2021-02-10T11:36:56.027 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.024 7fbd4ddaa700 1 --2- 172.21.2.112:0/4226706302 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbd480d1c70 0x7fbd480f20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:56.028 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.024 7fbd3ffff700 1 -- 172.21.2.112:0/4226706302 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3890978832 0 0) 0x7fbd380010b0 con 0x7fbd480d0340 2021-02-10T11:36:56.028 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.024 7fbd3ffff700 1 -- 172.21.2.112:0/4226706302 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbd480d1c70 msgr2=0x7fbd480f20a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:56.028 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.024 7fbd3ffff700 1 --2- 172.21.2.112:0/4226706302 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbd480d1c70 0x7fbd480f20a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:56.028 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.024 7fbd3ffff700 1 -- 172.21.2.112:0/4226706302 >> v1:172.21.2.120:6789/0 conn(0x7fbd480d1870 legacy=0x7fbd481b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:56.028 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.024 7fbd3ffff700 1 -- 172.21.2.112:0/4226706302 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd480efe30 con 0x7fbd480d0340 2021-02-10T11:36:56.028 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.024 7fbd3ffff700 1 -- 172.21.2.112:0/4226706302 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7fbd380015b0 con 0x7fbd480d0340 2021-02-10T11:36:56.028 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.024 7fbd3ffff700 1 -- 172.21.2.112:0/4226706302 >> v1:172.21.2.112:6789/0 conn(0x7fbd480d0340 legacy=0x7fbd480ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:56.028 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.024 7fbd3ffff700 1 --2- 172.21.2.112:0/4226706302 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbd34003b10 0x7fbd34005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:56.028 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.024 7fbd3ffff700 1 --2- 172.21.2.112:0/4226706302 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fbd340067b0 0x7fbd34008cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:56.028 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.024 7fbd3ffff700 1 --2- 172.21.2.112:0/4226706302 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbd340095b0 0x7fbd3400ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:56.028 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.024 7fbd3ffff700 1 -- 172.21.2.112:0/4226706302 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd3400c2d0 con 0x7fbd34003b10 2021-02-10T11:36:56.028 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.024 7fbd3ffff700 1 -- 172.21.2.112:0/4226706302 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd3400c4f0 con 0x7fbd340067b0 2021-02-10T11:36:56.028 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.024 7fbd3ffff700 1 -- 172.21.2.112:0/4226706302 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd3400c710 con 0x7fbd340095b0 2021-02-10T11:36:56.028 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.024 7fbd4d5a9700 1 --2- 172.21.2.112:0/4226706302 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbd34003b10 0x7fbd34005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:56.029 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.024 7fbd4ddaa700 1 --2- 172.21.2.112:0/4226706302 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbd340095b0 0x7fbd3400ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:56.029 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.025 7fbd4cda8700 1 --2- 172.21.2.112:0/4226706302 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fbd340067b0 0x7fbd34008cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:56.029 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.025 7fbd4d5a9700 1 -- 172.21.2.112:0/4226706302 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbd340095b0 msgr2=0x7fbd3400ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:56.029 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.025 7fbd4d5a9700 1 --2- 172.21.2.112:0/4226706302 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbd340095b0 0x7fbd3400ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:56.029 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.025 7fbd4d5a9700 1 -- 172.21.2.112:0/4226706302 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fbd340067b0 msgr2=0x7fbd34008cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:56.029 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.025 7fbd4d5a9700 1 --2- 172.21.2.112:0/4226706302 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fbd340067b0 0x7fbd34008cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:56.029 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.025 7fbd4d5a9700 1 -- 172.21.2.112:0/4226706302 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd34050980 con 0x7fbd34003b10 2021-02-10T11:36:56.029 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.025 7fbd4d5a9700 1 --2- 172.21.2.112:0/4226706302 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbd34003b10 0x7fbd34005fa0 secure :-1 s=READY pgs=1698 cs=0 l=1 rev1=1 rx=0x7fbd44006ac0 tx=0x7fbd44009a70).ready entity=mon.0 client_cookie=cebf9a1acf41c15e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:56.029 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.025 7fbd3ffff700 1 -- 172.21.2.112:0/4226706302 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbd44007280 con 0x7fbd34003b10 2021-02-10T11:36:56.029 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.025 7fbd3ffff700 1 -- 172.21.2.112:0/4226706302 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd44007580 con 0x7fbd34003b10 2021-02-10T11:36:56.029 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.025 7fbd3ffff700 1 -- 172.21.2.112:0/4226706302 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbd44007a60 con 0x7fbd34003b10 2021-02-10T11:36:56.029 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.025 7fbd4f78d700 1 -- 172.21.2.112:0/4226706302 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbd34003b10 msgr2=0x7fbd34005fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:56.029 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.025 7fbd4f78d700 1 --2- 172.21.2.112:0/4226706302 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbd34003b10 0x7fbd34005fa0 secure :-1 s=READY pgs=1698 cs=0 l=1 rev1=1 rx=0x7fbd44006ac0 tx=0x7fbd44009a70).stop 2021-02-10T11:36:56.029 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.025 7fbd4ddaa700 1 -- 172.21.2.112:0/4226706302 reap_dead start 2021-02-10T11:36:56.030 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.025 7fbd4f78d700 1 -- 172.21.2.112:0/4226706302 shutdown_connections 2021-02-10T11:36:56.030 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.025 7fbd4f78d700 1 -- 172.21.2.112:0/4226706302 >> 172.21.2.112:0/4226706302 conn(0x7fbd480ce450 msgr2=0x7fbd480d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:56.030 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.026 7fbd4f78d700 1 -- 172.21.2.112:0/4226706302 shutdown_connections 2021-02-10T11:36:56.030 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.026 7fbd4f78d700 1 -- 172.21.2.112:0/4226706302 wait complete. 2021-02-10T11:36:56.030 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.040 7fbd4f78d700 1 Processor -- start 2021-02-10T11:36:56.030 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.040 7fbd4f78d700 1 -- start start 2021-02-10T11:36:56.031 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.041 7fbd4f78d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbd48195650 0x7fbd480b5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:56.031 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.041 7fbd4f78d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbd480b5b80 0x7fbd481a2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:56.031 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.041 7fbd4f78d700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fbd480acfa0 0x7fbd481a3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:56.031 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.041 7fbd4f78d700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd480b6010 con 0x7fbd480b5b80 2021-02-10T11:36:56.032 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.041 7fbd4f78d700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd481a3a60 con 0x7fbd480acfa0 2021-02-10T11:36:56.032 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.041 7fbd4f78d700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd481a3c40 con 0x7fbd48195650 2021-02-10T11:36:56.032 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.041 7fbd4cda8700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbd480b5b80 0x7fbd481a2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:56.032 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.041 7fbd4cda8700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbd480b5b80 0x7fbd481a2e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:52466/0 (socket says 172.21.2.112:52466) 2021-02-10T11:36:56.032 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.041 7fbd4cda8700 1 -- 172.21.2.112:0/581633294 learned_addr learned my addr 172.21.2.112:0/581633294 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:36:56.032 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.041 7fbd4cda8700 1 -- 172.21.2.112:0/581633294 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbd48195650 msgr2=0x7fbd480b5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:56.032 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.041 7fbd4cda8700 1 --2- 172.21.2.112:0/581633294 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbd48195650 0x7fbd480b5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:56.032 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.041 7fbd4cda8700 1 -- 172.21.2.112:0/581633294 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fbd480acfa0 msgr2=0x7fbd481a3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:56.032 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.041 7fbd4cda8700 1 --2- 172.21.2.112:0/581633294 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fbd480acfa0 0x7fbd481a3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:56.032 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.041 7fbd4cda8700 1 -- 172.21.2.112:0/581633294 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd480bcae0 con 0x7fbd480b5b80 2021-02-10T11:36:56.032 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.041 7fbd4cda8700 1 --2- 172.21.2.112:0/581633294 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbd480b5b80 0x7fbd481a2e20 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 rx=0x7fbd38007650 tx=0x7fbd3800a880).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:56.032 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.041 7fbd3dffb700 1 -- 172.21.2.112:0/581633294 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbd3800ac70 con 0x7fbd480b5b80 2021-02-10T11:36:56.032 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.041 7fbd3dffb700 1 -- 172.21.2.112:0/581633294 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd38004350 con 0x7fbd480b5b80 2021-02-10T11:36:56.033 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.041 7fbd3dffb700 1 -- 172.21.2.112:0/581633294 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbd380047b0 con 0x7fbd480b5b80 2021-02-10T11:36:56.033 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.041 7fbd4f78d700 1 -- 172.21.2.112:0/581633294 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd480bcae0 con 0x7fbd480b5b80 2021-02-10T11:36:56.033 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.042 7fbd4f78d700 1 -- 172.21.2.112:0/581633294 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd480bd070 con 0x7fbd480b5b80 2021-02-10T11:36:56.033 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.042 7fbd3dffb700 1 -- 172.21.2.112:0/581633294 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fbd3800ac70 con 0x7fbd480b5b80 2021-02-10T11:36:56.033 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.042 7fbd3dffb700 1 --2- 172.21.2.112:0/581633294 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fbd280186c0 0x7fbd2801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:56.033 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.042 7fbd3dffb700 1 -- 172.21.2.112:0/581633294 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fbd380325b0 con 0x7fbd480b5b80 2021-02-10T11:36:56.033 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.043 7fbd4d5a9700 1 --2- 172.21.2.112:0/581633294 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fbd280186c0 0x7fbd2801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:56.033 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.043 7fbd4d5a9700 1 --2- 172.21.2.112:0/581633294 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fbd280186c0 0x7fbd2801abc0 crc :-1 s=READY pgs=807 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:56.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.073 7fbd4f78d700 1 -- 172.21.2.112:0/581633294 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd480b6010 con 0x7fbd480b5b80 2021-02-10T11:36:56.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.076 7fbd3dffb700 1 -- 172.21.2.112:0/581633294 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fbd38001870 con 0x7fbd480b5b80 2021-02-10T11:36:56.191 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.204 7fbd4f78d700 1 -- 172.21.2.112:0/581633294 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbd480b6010 con 0x7fbd480b5b80 2021-02-10T11:36:56.191 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.204 7fbd3dffb700 1 -- 172.21.2.112:0/581633294 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) v1 ==== 94+0+2101 (secure 0 0 0) 0x7fbd3800ac70 con 0x7fbd480b5b80 2021-02-10T11:36:56.191 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:36:56.192 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":64,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9204,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/1512510827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":1512510827},{"type":"v1","addr":"172.21.2.112:6833","nonce":1512510827}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":60}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:36:53.563735","modified":"2021-02-10 11:36:55.573740","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9201},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9201":{"gid":9201,"name":"b","rank":0,"incarnation":61,"state":"up:active","state_seq":3,"addr":"172.21.2.120:6835/1232001750","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":1232001750},{"type":"v1","addr":"172.21.2.120:6835","nonce":1232001750}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2021-02-10T11:36:56.192 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 64 2021-02-10T11:36:56.193 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.206 7fbd4f78d700 1 -- 172.21.2.112:0/581633294 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fbd280186c0 msgr2=0x7fbd2801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:56.193 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.206 7fbd4f78d700 1 --2- 172.21.2.112:0/581633294 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fbd280186c0 0x7fbd2801abc0 crc :-1 s=READY pgs=807 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:56.193 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.206 7fbd4f78d700 1 -- 172.21.2.112:0/581633294 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbd480b5b80 msgr2=0x7fbd481a2e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:56.193 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.206 7fbd4f78d700 1 --2- 172.21.2.112:0/581633294 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbd480b5b80 0x7fbd481a2e20 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 rx=0x7fbd38007650 tx=0x7fbd3800a880).stop 2021-02-10T11:36:56.194 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.207 7fbd4f78d700 1 -- 172.21.2.112:0/581633294 shutdown_connections 2021-02-10T11:36:56.194 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.207 7fbd4f78d700 1 --2- 172.21.2.112:0/581633294 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fbd280186c0 0x7fbd2801abc0 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:56.194 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.207 7fbd4f78d700 1 --2- 172.21.2.112:0/581633294 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fbd480acfa0 0x7fbd481a3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:56.194 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.207 7fbd4f78d700 1 --2- 172.21.2.112:0/581633294 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbd480b5b80 0x7fbd481a2e20 unknown :-1 s=CLOSED pgs=1699 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:56.194 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.207 7fbd4f78d700 1 --2- 172.21.2.112:0/581633294 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbd48195650 0x7fbd480b5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:56.195 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.207 7fbd4f78d700 1 -- 172.21.2.112:0/581633294 >> 172.21.2.112:0/581633294 conn(0x7fbd480b3040 msgr2=0x7fbd480c3250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:56.195 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.207 7fbd4f78d700 1 -- 172.21.2.112:0/581633294 shutdown_connections 2021-02-10T11:36:56.195 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.207 7fbd4f78d700 1 -- 172.21.2.112:0/581633294 wait complete. 2021-02-10T11:36:56.202 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:36:56.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.627 7f4362d35700 1 Processor -- start 2021-02-10T11:36:56.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.627 7f4362d35700 1 -- start start 2021-02-10T11:36:56.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.628 7f4362d35700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f435c0d0340 0x7f435c0ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:56.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.628 7f4362d35700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f435c0d1c70 0x7f435c0efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:56.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.628 7f4362d35700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f435c0f0b10 con 0x7f435c0d1870 2021-02-10T11:36:56.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.628 7f4362d35700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f435c0e5ed0 con 0x7f435c0d1c70 2021-02-10T11:36:56.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.628 7f4362d35700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f435c1b0be0 con 0x7f435c0d0340 2021-02-10T11:36:56.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.628 7f4361352700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f435c0d1c70 0x7f435c0efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:56.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.628 7f4361352700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f435c0d1c70 0x7f435c0efc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:52478/0 (socket says 172.21.2.112:52478) 2021-02-10T11:36:56.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.628 7f4361352700 1 -- 172.21.2.112:0/3432368258 learned_addr learned my addr 172.21.2.112:0/3432368258 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:36:56.632 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.628 7f4361352700 1 -- 172.21.2.112:0/3432368258 >> v2:172.21.2.120:3300/0 conn(0x7f435c0d0340 msgr2=0x7f435c0ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:56.632 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.628 7f4361352700 1 --2- 172.21.2.112:0/3432368258 >> v2:172.21.2.120:3300/0 conn(0x7f435c0d0340 0x7f435c0ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:56.632 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.628 7f4361352700 1 -- 172.21.2.112:0/3432368258 >> v1:172.21.2.120:6789/0 conn(0x7f435c0d1870 legacy=0x7f435c1b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:56.632 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.628 7f4361352700 1 -- 172.21.2.112:0/3432368258 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f435c0f20a0 con 0x7f435c0d1c70 2021-02-10T11:36:56.632 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.628 7f4361352700 1 --2- 172.21.2.112:0/3432368258 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f435c0d1c70 0x7f435c0efc60 secure :-1 s=READY pgs=1700 cs=0 l=1 rev1=1 rx=0x7f4354006b80 tx=0x7f4354009b10).ready entity=mon.0 client_cookie=9769a10ed1f5a231 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:56.632 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.629 7f435b7fe700 1 -- 172.21.2.112:0/3432368258 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f43540022e0 con 0x7f435c0d1c70 2021-02-10T11:36:56.632 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.629 7f435b7fe700 1 -- 172.21.2.112:0/3432368258 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4354002620 con 0x7f435c0d1c70 2021-02-10T11:36:56.632 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.629 7f435b7fe700 1 -- 172.21.2.112:0/3432368258 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4354000c10 con 0x7f435c0d1c70 2021-02-10T11:36:56.632 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.629 7f4362d35700 1 -- 172.21.2.112:0/3432368258 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f435c0d1c70 msgr2=0x7f435c0efc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:56.632 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.629 7f4362d35700 1 --2- 172.21.2.112:0/3432368258 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f435c0d1c70 0x7f435c0efc60 secure :-1 s=READY pgs=1700 cs=0 l=1 rev1=1 rx=0x7f4354006b80 tx=0x7f4354009b10).stop 2021-02-10T11:36:56.632 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.629 7f4362d35700 1 -- 172.21.2.112:0/3432368258 shutdown_connections 2021-02-10T11:36:56.632 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.629 7f4362d35700 1 --2- 172.21.2.112:0/3432368258 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f435c0d1c70 0x7f435c0efc60 unknown :-1 s=CLOSED pgs=1700 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:56.632 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.629 7f4362d35700 1 --2- 172.21.2.112:0/3432368258 >> v2:172.21.2.120:3300/0 conn(0x7f435c0d0340 0x7f435c0ef640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:56.633 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.629 7f4362d35700 1 -- 172.21.2.112:0/3432368258 >> 172.21.2.112:0/3432368258 conn(0x7f435c0ce450 msgr2=0x7f435c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:56.633 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.629 7f4362d35700 1 -- 172.21.2.112:0/3432368258 shutdown_connections 2021-02-10T11:36:56.633 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.629 7f4362d35700 1 -- 172.21.2.112:0/3432368258 wait complete. 2021-02-10T11:36:56.633 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.644 7f4362d35700 1 Processor -- start 2021-02-10T11:36:56.633 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.644 7f4362d35700 1 -- start start 2021-02-10T11:36:56.634 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.645 7f4362d35700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f435c0eef50 0x7f435c0ef350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:56.634 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.645 7f4362d35700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f435c0f1ae0 0x7f435c0f1ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:56.634 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.645 7f4362d35700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f435c0f2500 0x7f435c18f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:56.634 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.645 7f4362d35700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f435c0f2900 con 0x7f435c0eef50 2021-02-10T11:36:56.634 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.645 7f4362d35700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f435c18fd50 con 0x7f435c0f2500 2021-02-10T11:36:56.634 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.645 7f4362d35700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f435c18ff70 con 0x7f435c0f1ae0 2021-02-10T11:36:56.635 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.645 7f4360b51700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f435c0eef50 0x7f435c0ef350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:56.635 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.645 7f4360b51700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f435c0eef50 0x7f435c0ef350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:52480/0 (socket says 172.21.2.112:52480) 2021-02-10T11:36:56.635 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.645 7f4360b51700 1 -- 172.21.2.112:0/716705052 learned_addr learned my addr 172.21.2.112:0/716705052 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:36:56.635 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.645 7f4360b51700 1 -- 172.21.2.112:0/716705052 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f435c0f1ae0 msgr2=0x7f435c0f1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:56.635 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.645 7f4360b51700 1 --2- 172.21.2.112:0/716705052 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f435c0f1ae0 0x7f435c0f1ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:56.635 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.645 7f4360b51700 1 -- 172.21.2.112:0/716705052 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f435c0f2500 msgr2=0x7f435c18f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:56.635 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.645 7f4360b51700 1 --2- 172.21.2.112:0/716705052 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f435c0f2500 0x7f435c18f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:56.635 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.645 7f4360b51700 1 -- 172.21.2.112:0/716705052 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f435c190190 con 0x7f435c0eef50 2021-02-10T11:36:56.635 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.645 7f4360b51700 1 --2- 172.21.2.112:0/716705052 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f435c0eef50 0x7f435c0ef350 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 rx=0x7f434400b5b0 tx=0x7f434400b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:56.635 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.645 7f43597fa700 1 -- 172.21.2.112:0/716705052 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f434400bd40 con 0x7f435c0eef50 2021-02-10T11:36:56.635 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.645 7f43597fa700 1 -- 172.21.2.112:0/716705052 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4344000e20 con 0x7f435c0eef50 2021-02-10T11:36:56.635 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.645 7f43597fa700 1 -- 172.21.2.112:0/716705052 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f43440054a0 con 0x7f435c0eef50 2021-02-10T11:36:56.635 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.646 7f4362d35700 1 -- 172.21.2.112:0/716705052 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f435c190190 con 0x7f435c0eef50 2021-02-10T11:36:56.636 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.646 7f4362d35700 1 -- 172.21.2.112:0/716705052 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f435c190720 con 0x7f435c0eef50 2021-02-10T11:36:56.636 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.646 7f43597fa700 1 -- 172.21.2.112:0/716705052 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4344006030 con 0x7f435c0eef50 2021-02-10T11:36:56.636 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.646 7f43597fa700 1 --2- 172.21.2.112:0/716705052 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f43400186c0 0x7f434001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:56.636 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.646 7f43597fa700 1 -- 172.21.2.112:0/716705052 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f43440323e0 con 0x7f435c0eef50 2021-02-10T11:36:56.636 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.647 7f435bfff700 1 --2- 172.21.2.112:0/716705052 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f43400186c0 0x7f434001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:56.636 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.647 7f435bfff700 1 --2- 172.21.2.112:0/716705052 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f43400186c0 0x7f434001abc0 crc :-1 s=READY pgs=808 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:56.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.677 7f4362d35700 1 -- 172.21.2.112:0/716705052 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f435c0ee080 con 0x7f435c0eef50 2021-02-10T11:36:56.666 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.680 7f43597fa700 1 -- 172.21.2.112:0/716705052 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f434401d720 con 0x7f435c0eef50 2021-02-10T11:36:56.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.806 7f4362d35700 1 -- 172.21.2.112:0/716705052 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f435c0ce450 con 0x7f435c0eef50 2021-02-10T11:36:56.793 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.807 7f43597fa700 1 -- 172.21.2.112:0/716705052 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) v1 ==== 74+0+13996 (secure 0 0 0) 0x7f434401d720 con 0x7f435c0eef50 2021-02-10T11:36:56.793 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:36:56.794 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":83,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:36:54.566363","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:36:51.538038","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":17,"pool_name":"cephfs_data","create_time":"2021-02-10 11:36:52.609006","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:36:56.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.808 7f4362d35700 1 -- 172.21.2.112:0/716705052 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f43400186c0 msgr2=0x7f434001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:56.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.808 7f4362d35700 1 --2- 172.21.2.112:0/716705052 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f43400186c0 0x7f434001abc0 crc :-1 s=READY pgs=808 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:56.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.808 7f4362d35700 1 -- 172.21.2.112:0/716705052 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f435c0eef50 msgr2=0x7f435c0ef350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:56.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.808 7f4362d35700 1 --2- 172.21.2.112:0/716705052 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f435c0eef50 0x7f435c0ef350 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 rx=0x7f434400b5b0 tx=0x7f434400b950).stop 2021-02-10T11:36:56.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.808 7f4362d35700 1 -- 172.21.2.112:0/716705052 shutdown_connections 2021-02-10T11:36:56.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.808 7f4362d35700 1 --2- 172.21.2.112:0/716705052 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f43400186c0 0x7f434001abc0 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:56.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.808 7f4362d35700 1 --2- 172.21.2.112:0/716705052 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f435c0f2500 0x7f435c18f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:56.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.808 7f4362d35700 1 --2- 172.21.2.112:0/716705052 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f435c0f1ae0 0x7f435c0f1ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:56.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.808 7f4362d35700 1 --2- 172.21.2.112:0/716705052 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f435c0eef50 0x7f435c0ef350 unknown :-1 s=CLOSED pgs=1701 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:56.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.808 7f4362d35700 1 -- 172.21.2.112:0/716705052 >> 172.21.2.112:0/716705052 conn(0x7f435c19ed80 msgr2=0x7f435c092c60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:56.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.808 7f4362d35700 1 -- 172.21.2.112:0/716705052 shutdown_connections 2021-02-10T11:36:56.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:56.809 7f4362d35700 1 -- 172.21.2.112:0/716705052 wait complete. 2021-02-10T11:36:56.804 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:36:57.234 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.231 7f892c097700 1 Processor -- start 2021-02-10T11:36:57.234 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.231 7f892c097700 1 -- start start 2021-02-10T11:36:57.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.231 7f892c097700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f89240d1c80 0x7f89240cef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:57.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.231 7f892c097700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f89240cf590 0x7f89240cf990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:57.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.231 7f892c097700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89240fe1d0 con 0x7f89240d1880 2021-02-10T11:36:57.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.231 7f892c097700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89241b0cc0 con 0x7f89240cf590 2021-02-10T11:36:57.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.231 7f892c097700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8924109a70 con 0x7f89240d1c80 2021-02-10T11:36:57.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.232 7f892a6b4700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f89240cf590 0x7f89240cf990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:57.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.232 7f892a6b4700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f89240cf590 0x7f89240cf990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:52492/0 (socket says 172.21.2.112:52492) 2021-02-10T11:36:57.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.232 7f892a6b4700 1 -- 172.21.2.112:0/2063402191 learned_addr learned my addr 172.21.2.112:0/2063402191 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:36:57.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.232 7f8929eb3700 1 --1- 172.21.2.112:0/2063402191 >> v1:172.21.2.112:6789/0 conn(0x7f89240d1880 0x7f89241b04d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:51950/0 (socket says 172.21.2.112:51950) 2021-02-10T11:36:57.236 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.232 7f892a6b4700 1 -- 172.21.2.112:0/2063402191 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f89240d1c80 msgr2=0x7f89240cef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:57.236 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.232 7f892a6b4700 1 --2- 172.21.2.112:0/2063402191 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f89240d1c80 0x7f89240cef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:57.236 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.232 7f892a6b4700 1 -- 172.21.2.112:0/2063402191 >> v1:172.21.2.112:6789/0 conn(0x7f89240d1880 legacy=0x7f89241b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:57.236 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.232 7f892a6b4700 1 -- 172.21.2.112:0/2063402191 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8924088070 con 0x7f89240cf590 2021-02-10T11:36:57.236 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.232 7f892a6b4700 1 --2- 172.21.2.112:0/2063402191 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f89240cf590 0x7f89240cf990 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 rx=0x7f891c006f00 tx=0x7f891c009f10).ready entity=mon.0 client_cookie=173057ec70f7c94d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:57.236 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.232 7f8928eb1700 1 -- 172.21.2.112:0/2063402191 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f891c00b3b0 con 0x7f89240cf590 2021-02-10T11:36:57.236 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.232 7f8928eb1700 1 -- 172.21.2.112:0/2063402191 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f891c00b730 con 0x7f89240cf590 2021-02-10T11:36:57.236 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.232 7f8928eb1700 1 -- 172.21.2.112:0/2063402191 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f891c00bc10 con 0x7f89240cf590 2021-02-10T11:36:57.236 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.232 7f892c097700 1 -- 172.21.2.112:0/2063402191 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f89240cf590 msgr2=0x7f89240cf990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:57.236 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.232 7f892c097700 1 --2- 172.21.2.112:0/2063402191 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f89240cf590 0x7f89240cf990 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 rx=0x7f891c006f00 tx=0x7f891c009f10).stop 2021-02-10T11:36:57.236 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.233 7f892c097700 1 -- 172.21.2.112:0/2063402191 shutdown_connections 2021-02-10T11:36:57.236 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.233 7f892c097700 1 --2- 172.21.2.112:0/2063402191 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f89240cf590 0x7f89240cf990 unknown :-1 s=CLOSED pgs=1703 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:57.237 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.233 7f892c097700 1 --2- 172.21.2.112:0/2063402191 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f89240d1c80 0x7f89240cef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:57.237 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.233 7f892c097700 1 -- 172.21.2.112:0/2063402191 >> 172.21.2.112:0/2063402191 conn(0x7f89240ce430 msgr2=0x7f89240d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:57.237 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.233 7f892c097700 1 -- 172.21.2.112:0/2063402191 shutdown_connections 2021-02-10T11:36:57.237 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.233 7f892c097700 1 -- 172.21.2.112:0/2063402191 wait complete. 2021-02-10T11:36:57.237 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.248 7f892c097700 1 Processor -- start 2021-02-10T11:36:57.237 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.248 7f892c097700 1 -- start start 2021-02-10T11:36:57.238 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.248 7f892c097700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f89241b9280 0x7f892418f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:57.239 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.248 7f892c097700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f89241b9680 0x7f892418fd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:57.239 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.248 7f892c097700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f89241acd60 0x7f89241ad160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:57.239 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.248 7f892c097700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8924190390 con 0x7f89241b9280 2021-02-10T11:36:57.239 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.248 7f892c097700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89241907c0 con 0x7f89241acd60 2021-02-10T11:36:57.239 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.248 7f892c097700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89241ac3d0 con 0x7f89241b9680 2021-02-10T11:36:57.239 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.249 7f8929eb3700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f89241b9280 0x7f892418f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:57.240 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.249 7f8929eb3700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f89241b9280 0x7f892418f750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:52494/0 (socket says 172.21.2.112:52494) 2021-02-10T11:36:57.240 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.249 7f8929eb3700 1 -- 172.21.2.112:0/3801084672 learned_addr learned my addr 172.21.2.112:0/3801084672 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:36:57.240 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.249 7f8929eb3700 1 -- 172.21.2.112:0/3801084672 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f89241b9680 msgr2=0x7f892418fd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:57.240 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.249 7f8929eb3700 1 --2- 172.21.2.112:0/3801084672 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f89241b9680 0x7f892418fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:57.240 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.249 7f8929eb3700 1 -- 172.21.2.112:0/3801084672 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f89241acd60 msgr2=0x7f89241ad160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:57.240 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.249 7f8929eb3700 1 --2- 172.21.2.112:0/3801084672 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f89241acd60 0x7f89241ad160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:57.240 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.249 7f8929eb3700 1 -- 172.21.2.112:0/3801084672 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89241ad780 con 0x7f89241b9280 2021-02-10T11:36:57.240 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.249 7f8929eb3700 1 --2- 172.21.2.112:0/3801084672 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f89241b9280 0x7f892418f750 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 rx=0x7f891800a590 tx=0x7f891800a930).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:57.240 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.249 7f89167fc700 1 -- 172.21.2.112:0/3801084672 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8918000ef0 con 0x7f89241b9280 2021-02-10T11:36:57.240 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.249 7f89167fc700 1 -- 172.21.2.112:0/3801084672 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f891800ad20 con 0x7f89241b9280 2021-02-10T11:36:57.240 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.249 7f89167fc700 1 -- 172.21.2.112:0/3801084672 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8918004590 con 0x7f89241b9280 2021-02-10T11:36:57.240 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.249 7f892c097700 1 -- 172.21.2.112:0/3801084672 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89241ad780 con 0x7f89241b9280 2021-02-10T11:36:57.241 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.250 7f892c097700 1 -- 172.21.2.112:0/3801084672 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89241ad780 con 0x7f89241b9280 2021-02-10T11:36:57.241 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.250 7f89167fc700 1 -- 172.21.2.112:0/3801084672 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f89180045b0 con 0x7f89241b9280 2021-02-10T11:36:57.241 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.250 7f89167fc700 1 --2- 172.21.2.112:0/3801084672 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f890c0186d0 0x7f890c01abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:57.241 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.250 7f89167fc700 1 -- 172.21.2.112:0/3801084672 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f8918032820 con 0x7f89241b9280 2021-02-10T11:36:57.241 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.251 7f89296b2700 1 --2- 172.21.2.112:0/3801084672 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f890c0186d0 0x7f890c01abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:57.241 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.251 7f89296b2700 1 --2- 172.21.2.112:0/3801084672 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f890c0186d0 0x7f890c01abd0 crc :-1 s=READY pgs=809 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:57.268 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.281 7f892c097700 1 -- 172.21.2.112:0/3801084672 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89240829a0 con 0x7f89241b9280 2021-02-10T11:36:57.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.284 7f89167fc700 1 -- 172.21.2.112:0/3801084672 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f89180043a0 con 0x7f89241b9280 2021-02-10T11:36:57.403 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.416 7f892c097700 1 -- 172.21.2.112:0/3801084672 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f89240829a0 con 0x7f89241b9280 2021-02-10T11:36:57.406 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.420 7f89167fc700 1 -- 172.21.2.112:0/3801084672 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v49) v1 ==== 184+0+0 (secure 0 0 0) 0x7f8918008000 con 0x7f89241b9280 2021-02-10T11:36:57.410 INFO:teuthology.orchestra.run.gibba012.stderr:updated caps for client.0 2021-02-10T11:36:57.411 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.425 7f892c097700 1 -- 172.21.2.112:0/3801084672 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f890c0186d0 msgr2=0x7f890c01abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:57.411 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.425 7f892c097700 1 --2- 172.21.2.112:0/3801084672 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f890c0186d0 0x7f890c01abd0 crc :-1 s=READY pgs=809 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:57.412 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.425 7f892c097700 1 -- 172.21.2.112:0/3801084672 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f89241b9280 msgr2=0x7f892418f750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:57.412 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.425 7f892c097700 1 --2- 172.21.2.112:0/3801084672 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f89241b9280 0x7f892418f750 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 rx=0x7f891800a590 tx=0x7f891800a930).stop 2021-02-10T11:36:57.412 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.425 7f892c097700 1 -- 172.21.2.112:0/3801084672 shutdown_connections 2021-02-10T11:36:57.412 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.425 7f892c097700 1 --2- 172.21.2.112:0/3801084672 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f890c0186d0 0x7f890c01abd0 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:57.412 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.425 7f892c097700 1 --2- 172.21.2.112:0/3801084672 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f89241acd60 0x7f89241ad160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:57.412 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.425 7f892c097700 1 --2- 172.21.2.112:0/3801084672 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f89241b9680 0x7f892418fd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:57.412 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.425 7f892c097700 1 --2- 172.21.2.112:0/3801084672 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f89241b9280 0x7f892418f750 unknown :-1 s=CLOSED pgs=1704 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:57.412 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.425 7f892c097700 1 -- 172.21.2.112:0/3801084672 >> 172.21.2.112:0/3801084672 conn(0x7f89240845b0 msgr2=0x7f892418c290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:57.413 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.425 7f892c097700 1 -- 172.21.2.112:0/3801084672 shutdown_connections 2021-02-10T11:36:57.413 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.425 7f892c097700 1 -- 172.21.2.112:0/3801084672 wait complete. 2021-02-10T11:36:57.420 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:36:57.850 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.846 7f7bf9f6d700 1 Processor -- start 2021-02-10T11:36:57.851 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.846 7f7bf9f6d700 1 -- start start 2021-02-10T11:36:57.851 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.846 7f7bf9f6d700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f7bf40d1860 0x7f7bf41b04b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:57.851 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.846 7f7bf9f6d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7bf40cff60 0x7f7bf4109120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:57.851 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.846 7f7bf9f6d700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bf4109a50 con 0x7f7bf41b0ad0 2021-02-10T11:36:57.851 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.846 7f7bf9f6d700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7bf4109740 con 0x7f7bf40d1860 2021-02-10T11:36:57.851 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.846 7f7bf9f6d700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bf4088070 con 0x7f7bf40cff60 2021-02-10T11:36:57.851 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.846 7f7bf2ffd700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f7bf41b0ad0 0x7f7bf4108930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:51964/0 (socket says 172.21.2.112:51964) 2021-02-10T11:36:57.851 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.846 7f7bf2ffd700 1 -- 172.21.2.112:0/2979162265 learned_addr learned my addr 172.21.2.112:0/2979162265 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:36:57.851 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.847 7f7bf27fc700 1 -- 172.21.2.112:0/2979162265 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1675651810 0 0) 0x7f7be80010b0 con 0x7f7bf41b0ad0 2021-02-10T11:36:57.851 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.847 7f7bf27fc700 1 -- 172.21.2.112:0/2979162265 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7bdc002fe0 con 0x7f7bf41b0ad0 2021-02-10T11:36:57.852 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.847 7f7bf37fe700 1 --2- 172.21.2.112:0/2979162265 >> v2:172.21.2.120:3300/0 conn(0x7f7bf40d1860 0x7f7bf41b04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:57.852 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.847 7f7bf3fff700 1 --2- 172.21.2.112:0/2979162265 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7bf40cff60 0x7f7bf4109120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:57.852 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.847 7f7bf27fc700 1 -- 172.21.2.112:0/2979162265 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1761591564 0 0) 0x7f7be80010b0 con 0x7f7bf41b0ad0 2021-02-10T11:36:57.852 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.847 7f7bf27fc700 1 -- 172.21.2.112:0/2979162265 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7bf40cff60 msgr2=0x7f7bf4109120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:57.852 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.847 7f7bf27fc700 1 --2- 172.21.2.112:0/2979162265 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7bf40cff60 0x7f7bf4109120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:57.852 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.847 7f7bf27fc700 1 -- 172.21.2.112:0/2979162265 >> v2:172.21.2.120:3300/0 conn(0x7f7bf40d1860 msgr2=0x7f7bf41b04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:57.852 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.847 7f7bf27fc700 1 --2- 172.21.2.112:0/2979162265 >> v2:172.21.2.120:3300/0 conn(0x7f7bf40d1860 0x7f7bf41b04b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:57.852 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.847 7f7bf27fc700 1 -- 172.21.2.112:0/2979162265 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bf40d1c60 con 0x7f7bf41b0ad0 2021-02-10T11:36:57.852 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.847 7f7bf27fc700 1 -- 172.21.2.112:0/2979162265 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f7be80015b0 con 0x7f7bf41b0ad0 2021-02-10T11:36:57.852 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.847 7f7bf27fc700 1 -- 172.21.2.112:0/2979162265 >> v1:172.21.2.112:6789/0 conn(0x7f7bf41b0ad0 legacy=0x7f7bf4108930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:57.852 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.847 7f7bf27fc700 1 --2- 172.21.2.112:0/2979162265 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7bdc003ac0 0x7f7bdc005f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:57.852 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.847 7f7bf27fc700 1 --2- 172.21.2.112:0/2979162265 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7bdc006780 0x7f7bdc008ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:57.853 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.848 7f7bf27fc700 1 --2- 172.21.2.112:0/2979162265 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7bdc04d5b0 0x7f7bdc04fa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:57.853 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.848 7f7bf27fc700 1 -- 172.21.2.112:0/2979162265 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bdc0502d0 con 0x7f7bdc006780 2021-02-10T11:36:57.853 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.848 7f7bf27fc700 1 -- 172.21.2.112:0/2979162265 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bdc0504f0 con 0x7f7bdc003ac0 2021-02-10T11:36:57.853 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.848 7f7bf27fc700 1 -- 172.21.2.112:0/2979162265 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bdc050710 con 0x7f7bdc04d5b0 2021-02-10T11:36:57.853 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.848 7f7bf2ffd700 1 --2- 172.21.2.112:0/2979162265 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7bdc006780 0x7f7bdc008ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:57.853 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.848 7f7bf37fe700 1 --2- 172.21.2.112:0/2979162265 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7bdc003ac0 0x7f7bdc005f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:57.853 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.848 7f7bf2ffd700 1 -- 172.21.2.112:0/2979162265 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7bdc04d5b0 msgr2=0x7f7bdc04fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:57.853 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.848 7f7bf2ffd700 1 --2- 172.21.2.112:0/2979162265 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7bdc04d5b0 0x7f7bdc04fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:57.853 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.848 7f7bf2ffd700 1 -- 172.21.2.112:0/2979162265 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7bdc003ac0 msgr2=0x7f7bdc005f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:57.853 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.848 7f7bf2ffd700 1 --2- 172.21.2.112:0/2979162265 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7bdc003ac0 0x7f7bdc005f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:57.853 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.848 7f7bf2ffd700 1 -- 172.21.2.112:0/2979162265 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bdc0509b0 con 0x7f7bdc006780 2021-02-10T11:36:57.853 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.848 7f7bf2ffd700 1 --2- 172.21.2.112:0/2979162265 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7bdc006780 0x7f7bdc008ca0 secure :-1 s=READY pgs=1706 cs=0 l=1 rev1=1 rx=0x7f7be8001380 tx=0x7f7be8007d60).ready entity=mon.0 client_cookie=4372e342c9820481 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:57.854 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.848 7f7bf27fc700 1 -- 172.21.2.112:0/2979162265 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7be800a520 con 0x7f7bdc006780 2021-02-10T11:36:57.854 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.848 7f7bf27fc700 1 -- 172.21.2.112:0/2979162265 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7be800a8a0 con 0x7f7bdc006780 2021-02-10T11:36:57.854 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.848 7f7bf27fc700 1 -- 172.21.2.112:0/2979162265 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7be800ad80 con 0x7f7bdc006780 2021-02-10T11:36:57.854 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.848 7f7bf9f6d700 1 -- 172.21.2.112:0/2979162265 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7bdc006780 msgr2=0x7f7bdc008ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:57.854 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.848 7f7bf9f6d700 1 --2- 172.21.2.112:0/2979162265 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7bdc006780 0x7f7bdc008ca0 secure :-1 s=READY pgs=1706 cs=0 l=1 rev1=1 rx=0x7f7be8001380 tx=0x7f7be8007d60).stop 2021-02-10T11:36:57.854 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.848 7f7bf3fff700 1 -- 172.21.2.112:0/2979162265 reap_dead start 2021-02-10T11:36:57.854 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.849 7f7bf9f6d700 1 -- 172.21.2.112:0/2979162265 shutdown_connections 2021-02-10T11:36:57.854 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.849 7f7bf9f6d700 1 -- 172.21.2.112:0/2979162265 >> 172.21.2.112:0/2979162265 conn(0x7f7bf40ce410 msgr2=0x7f7bf40d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:57.855 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.849 7f7bf9f6d700 1 -- 172.21.2.112:0/2979162265 shutdown_connections 2021-02-10T11:36:57.855 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.849 7f7bf9f6d700 1 -- 172.21.2.112:0/2979162265 wait complete. 2021-02-10T11:36:57.856 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.863 7f7bf9f6d700 1 Processor -- start 2021-02-10T11:36:57.856 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.864 7f7bf9f6d700 1 -- start start 2021-02-10T11:36:57.856 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.864 7f7bf9f6d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7bf419c1c0 0x7f7bf408c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:57.856 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.864 7f7bf9f6d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7bf408cee0 0x7f7bf4090500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:57.856 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.864 7f7bf9f6d700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7bf4090b20 0x7f7bf40b6670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:57.857 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.864 7f7bf9f6d700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bf408d360 con 0x7f7bf408cee0 2021-02-10T11:36:57.857 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.864 7f7bf9f6d700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bf4090f20 con 0x7f7bf4090b20 2021-02-10T11:36:57.857 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.864 7f7bf9f6d700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bf4091140 con 0x7f7bf419c1c0 2021-02-10T11:36:57.857 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.864 7f7bf2ffd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7bf408cee0 0x7f7bf4090500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:57.857 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.864 7f7bf2ffd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7bf408cee0 0x7f7bf4090500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:52518/0 (socket says 172.21.2.112:52518) 2021-02-10T11:36:57.857 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.864 7f7bf2ffd700 1 -- 172.21.2.112:0/3972402357 learned_addr learned my addr 172.21.2.112:0/3972402357 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:36:57.857 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.864 7f7bf2ffd700 1 -- 172.21.2.112:0/3972402357 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7bf419c1c0 msgr2=0x7f7bf408c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:57.857 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.864 7f7bf2ffd700 1 --2- 172.21.2.112:0/3972402357 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7bf419c1c0 0x7f7bf408c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:57.857 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.865 7f7bf2ffd700 1 -- 172.21.2.112:0/3972402357 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7bf4090b20 msgr2=0x7f7bf40b6670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:57.857 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.865 7f7bf2ffd700 1 --2- 172.21.2.112:0/3972402357 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7bf4090b20 0x7f7bf40b6670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:57.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.865 7f7bf2ffd700 1 -- 172.21.2.112:0/3972402357 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bf40b6c90 con 0x7f7bf408cee0 2021-02-10T11:36:57.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.865 7f7bf2ffd700 1 --2- 172.21.2.112:0/3972402357 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7bf408cee0 0x7f7bf4090500 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 rx=0x7f7be8001bf0 tx=0x7f7be800a900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:57.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.865 7f7bdbfff700 1 -- 172.21.2.112:0/3972402357 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7be800acf0 con 0x7f7bf408cee0 2021-02-10T11:36:57.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.865 7f7bdbfff700 1 -- 172.21.2.112:0/3972402357 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7be8004140 con 0x7f7bf408cee0 2021-02-10T11:36:57.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.865 7f7bdbfff700 1 -- 172.21.2.112:0/3972402357 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7be8004660 con 0x7f7bf408cee0 2021-02-10T11:36:57.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.865 7f7bf9f6d700 1 -- 172.21.2.112:0/3972402357 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bf40b6c90 con 0x7f7bf408cee0 2021-02-10T11:36:57.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.865 7f7bf9f6d700 1 -- 172.21.2.112:0/3972402357 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bf40b6c90 con 0x7f7bf408cee0 2021-02-10T11:36:57.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.866 7f7bdbfff700 1 -- 172.21.2.112:0/3972402357 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7be8004320 con 0x7f7bf408cee0 2021-02-10T11:36:57.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.866 7f7bdbfff700 1 --2- 172.21.2.112:0/3972402357 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7bd40186c0 0x7f7bd401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:57.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.866 7f7bdbfff700 1 -- 172.21.2.112:0/3972402357 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f7be80325a0 con 0x7f7bf408cee0 2021-02-10T11:36:57.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.866 7f7bf37fe700 1 --2- 172.21.2.112:0/3972402357 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7bd40186c0 0x7f7bd401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:57.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.868 7f7bf37fe700 1 --2- 172.21.2.112:0/3972402357 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7bd40186c0 0x7f7bd401abc0 crc :-1 s=READY pgs=810 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:57.884 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.896 7f7bf9f6d700 1 -- 172.21.2.112:0/3972402357 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bf40b6c90 con 0x7f7bf408cee0 2021-02-10T11:36:57.886 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:57.899 7f7bdbfff700 1 -- 172.21.2.112:0/3972402357 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7be8030dd0 con 0x7f7bf408cee0 2021-02-10T11:36:58.017 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:58.030 7f7bf9f6d700 1 -- 172.21.2.112:0/3972402357 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f7bf40b6c90 con 0x7f7bf408cee0 2021-02-10T11:36:58.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:58.033 7f7bdbfff700 1 -- 172.21.2.112:0/3972402357 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v50) v1 ==== 184+0+0 (secure 0 0 0) 0x7f7be801f520 con 0x7f7bf408cee0 2021-02-10T11:36:58.020 INFO:teuthology.orchestra.run.gibba012.stderr:updated caps for client.1 2021-02-10T11:36:58.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:58.034 7f7bf9f6d700 1 -- 172.21.2.112:0/3972402357 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7bd40186c0 msgr2=0x7f7bd401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:58.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:58.034 7f7bf9f6d700 1 --2- 172.21.2.112:0/3972402357 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7bd40186c0 0x7f7bd401abc0 crc :-1 s=READY pgs=810 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:58.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:58.034 7f7bf9f6d700 1 -- 172.21.2.112:0/3972402357 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7bf408cee0 msgr2=0x7f7bf4090500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:58.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:58.034 7f7bf9f6d700 1 --2- 172.21.2.112:0/3972402357 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7bf408cee0 0x7f7bf4090500 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 rx=0x7f7be8001bf0 tx=0x7f7be800a900).stop 2021-02-10T11:36:58.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:58.034 7f7bf9f6d700 1 -- 172.21.2.112:0/3972402357 shutdown_connections 2021-02-10T11:36:58.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:58.034 7f7bf9f6d700 1 --2- 172.21.2.112:0/3972402357 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7bd40186c0 0x7f7bd401abc0 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:58.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:58.034 7f7bf9f6d700 1 --2- 172.21.2.112:0/3972402357 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7bf4090b20 0x7f7bf40b6670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:58.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:58.034 7f7bf9f6d700 1 --2- 172.21.2.112:0/3972402357 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7bf408cee0 0x7f7bf4090500 unknown :-1 s=CLOSED pgs=1707 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:58.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:58.034 7f7bf9f6d700 1 --2- 172.21.2.112:0/3972402357 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7bf419c1c0 0x7f7bf408c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:58.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:58.034 7f7bf9f6d700 1 -- 172.21.2.112:0/3972402357 >> 172.21.2.112:0/3972402357 conn(0x7f7bf409da90 msgr2=0x7f7bf409d020 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:58.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:58.035 7f7bf9f6d700 1 -- 172.21.2.112:0/3972402357 shutdown_connections 2021-02-10T11:36:58.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:58.035 7f7bf9f6d700 1 -- 172.21.2.112:0/3972402357 wait complete. 2021-02-10T11:36:58.030 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.2 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:36:58.469 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:58.465 7f4055f7b700 1 Processor -- start 2021-02-10T11:36:58.469 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:58.465 7f4055f7b700 1 -- start start 2021-02-10T11:36:58.470 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:58.465 7f4055f7b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f40500d1c60 0x7f40500cef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:58.470 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:58.465 7f4055f7b700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f40500cf570 0x7f40500cf970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:58.470 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:58.465 7f4055f7b700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40500fe1b0 con 0x7f40500d1860 2021-02-10T11:36:58.470 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:58.465 7f4055f7b700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40501b0ca0 con 0x7f40500cf570 2021-02-10T11:36:58.470 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:58.465 7f4055f7b700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4050109a50 con 0x7f40500d1c60 2021-02-10T11:36:58.470 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:58.465 7f404f7fe700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f40500d1860 0x7f40501b04b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:51984/0 (socket says 172.21.2.112:51984) 2021-02-10T11:36:58.470 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:58.465 7f404f7fe700 1 -- 172.21.2.112:0/530706420 learned_addr learned my addr 172.21.2.112:0/530706420 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:36:58.470 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:58.466 7f404effd700 1 --2- 172.21.2.112:0/530706420 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f40500d1c60 0x7f40500cef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:58.470 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:58.466 7f404ffff700 1 --2- 172.21.2.112:0/530706420 >> v2:172.21.2.120:3300/0 conn(0x7f40500cf570 0x7f40500cf970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:58.470 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:58.466 7f404e7fc700 1 -- 172.21.2.112:0/530706420 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 894988981 0 0) 0x7f4038000ce0 con 0x7f40500d1860 2021-02-10T11:36:58.470 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:58.466 7f404e7fc700 1 -- 172.21.2.112:0/530706420 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f403c002fe0 con 0x7f40500d1860 2021-02-10T11:36:58.470 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:58.466 7f404ffff700 1 -- 172.21.2.112:0/530706420 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f40500d1c60 msgr2=0x7f40500cef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:58.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:58.466 7f404ffff700 1 --2- 172.21.2.112:0/530706420 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f40500d1c60 0x7f40500cef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:58.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:58.466 7f404ffff700 1 -- 172.21.2.112:0/530706420 >> v1:172.21.2.112:6789/0 conn(0x7f40500d1860 legacy=0x7f40501b04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:58.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:58.466 7f404ffff700 1 -- 172.21.2.112:0/530706420 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4050088070 con 0x7f40500cf570 2021-02-10T11:36:58.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:58.466 7f404effd700 1 --2- 172.21.2.112:0/530706420 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f40500d1c60 0x7f40500cef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:36:58.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:58.466 7f404ffff700 1 --2- 172.21.2.112:0/530706420 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f40500cf570 0x7f40500cf970 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 rx=0x7f4048006e50 tx=0x7f4048009e70).ready entity=mon.1 client_cookie=eb10a84bf8f34d39 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:58.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:58.466 7f404e7fc700 1 -- 172.21.2.112:0/530706420 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f404800b510 con 0x7f40500cf570 2021-02-10T11:36:58.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:58.466 7f404e7fc700 1 -- 172.21.2.112:0/530706420 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4048006790 con 0x7f40500cf570 2021-02-10T11:36:58.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:58.466 7f404e7fc700 1 -- 172.21.2.112:0/530706420 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f404800b940 con 0x7f40500cf570 2021-02-10T11:36:58.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:58.467 7f4055f7b700 1 -- 172.21.2.112:0/530706420 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f40500cf570 msgr2=0x7f40500cf970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:58.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:58.467 7f4055f7b700 1 --2- 172.21.2.112:0/530706420 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f40500cf570 0x7f40500cf970 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 rx=0x7f4048006e50 tx=0x7f4048009e70).stop 2021-02-10T11:36:58.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:58.467 7f4055f7b700 1 -- 172.21.2.112:0/530706420 shutdown_connections 2021-02-10T11:36:58.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:58.467 7f4055f7b700 1 --2- 172.21.2.112:0/530706420 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f40500cf570 0x7f40500cf970 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:58.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:58.467 7f4055f7b700 1 --2- 172.21.2.112:0/530706420 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f40500d1c60 0x7f40500cef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:58.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:58.467 7f4055f7b700 1 -- 172.21.2.112:0/530706420 >> 172.21.2.112:0/530706420 conn(0x7f40500ce410 msgr2=0x7f40500d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:58.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:58.467 7f4055f7b700 1 -- 172.21.2.112:0/530706420 shutdown_connections 2021-02-10T11:36:58.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:58.467 7f4055f7b700 1 -- 172.21.2.112:0/530706420 wait complete. 2021-02-10T11:36:58.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:58.482 7f4055f7b700 1 Processor -- start 2021-02-10T11:36:58.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:58.482 7f4055f7b700 1 -- start start 2021-02-10T11:36:58.473 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:58.482 7f4055f7b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f40500b5fb0 0x7f40500a81f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:58.473 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:58.482 7f4055f7b700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f40500a8810 0x7f40500a2540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:58.473 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:58.482 7f4055f7b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f40500a8c10 0x7f40500a2b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:58.473 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:58.482 7f4055f7b700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40500b63b0 con 0x7f40500a8c10 2021-02-10T11:36:58.473 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:58.482 7f4055f7b700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40500a9210 con 0x7f40500a8810 2021-02-10T11:36:58.473 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:58.482 7f4055f7b700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40500b2650 con 0x7f40500b5fb0 2021-02-10T11:36:58.473 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:58.482 7f404ffff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f40500a8c10 0x7f40500a2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:58.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:58.482 7f404ffff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f40500a8c10 0x7f40500a2b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:52532/0 (socket says 172.21.2.112:52532) 2021-02-10T11:36:58.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:58.482 7f404ffff700 1 -- 172.21.2.112:0/2069877858 learned_addr learned my addr 172.21.2.112:0/2069877858 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:36:58.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:58.483 7f404ffff700 1 -- 172.21.2.112:0/2069877858 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f40500b5fb0 msgr2=0x7f40500a81f0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:36:58.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:58.483 7f404ffff700 1 --2- 172.21.2.112:0/2069877858 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f40500b5fb0 0x7f40500a81f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:58.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:58.483 7f404ffff700 1 -- 172.21.2.112:0/2069877858 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f40500a8810 msgr2=0x7f40500a2540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:58.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:58.483 7f404ffff700 1 --2- 172.21.2.112:0/2069877858 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f40500a8810 0x7f40500a2540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:58.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:58.483 7f404ffff700 1 -- 172.21.2.112:0/2069877858 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40500a3180 con 0x7f40500a8c10 2021-02-10T11:36:58.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:58.483 7f404ffff700 1 --2- 172.21.2.112:0/2069877858 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f40500a8c10 0x7f40500a2b60 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 rx=0x7f40480099f0 tx=0x7f404800bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:58.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:58.483 7f4047fff700 1 -- 172.21.2.112:0/2069877858 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f40480093d0 con 0x7f40500a8c10 2021-02-10T11:36:58.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:58.483 7f4047fff700 1 -- 172.21.2.112:0/2069877858 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4048003030 con 0x7f40500a8c10 2021-02-10T11:36:58.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:58.483 7f4047fff700 1 -- 172.21.2.112:0/2069877858 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f40480015a0 con 0x7f40500a8c10 2021-02-10T11:36:58.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:58.483 7f4055f7b700 1 -- 172.21.2.112:0/2069877858 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40500a3180 con 0x7f40500a8c10 2021-02-10T11:36:58.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:58.483 7f4055f7b700 1 -- 172.21.2.112:0/2069877858 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40500a3710 con 0x7f40500a8c10 2021-02-10T11:36:58.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:58.484 7f4047fff700 1 -- 172.21.2.112:0/2069877858 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f40480011a0 con 0x7f40500a8c10 2021-02-10T11:36:58.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:58.484 7f4047fff700 1 --2- 172.21.2.112:0/2069877858 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f40340186c0 0x7f403401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:58.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:58.484 7f4047fff700 1 -- 172.21.2.112:0/2069877858 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f4048032910 con 0x7f40500a8c10 2021-02-10T11:36:58.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:58.484 7f404f7fe700 1 --2- 172.21.2.112:0/2069877858 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f40340186c0 0x7f403401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:58.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:58.484 7f404f7fe700 1 --2- 172.21.2.112:0/2069877858 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f40340186c0 0x7f403401abc0 crc :-1 s=READY pgs=811 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:58.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:58.515 7f4055f7b700 1 -- 172.21.2.112:0/2069877858 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40500fe1b0 con 0x7f40500a8c10 2021-02-10T11:36:58.505 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:58.518 7f4047fff700 1 -- 172.21.2.112:0/2069877858 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f40480093d0 con 0x7f40500a8c10 2021-02-10T11:36:58.636 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:58.648 7f4055f7b700 1 -- 172.21.2.112:0/2069877858 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f40500fe1b0 con 0x7f40500a8c10 2021-02-10T11:36:58.639 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:58.652 7f4047fff700 1 -- 172.21.2.112:0/2069877858 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.2 v51) v1 ==== 184+0+0 (secure 0 0 0) 0x7f4048001580 con 0x7f40500a8c10 2021-02-10T11:36:58.643 INFO:teuthology.orchestra.run.gibba012.stderr:updated caps for client.2 2021-02-10T11:36:58.645 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:58.657 7f4055f7b700 1 -- 172.21.2.112:0/2069877858 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f40340186c0 msgr2=0x7f403401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:58.646 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:58.657 7f4055f7b700 1 --2- 172.21.2.112:0/2069877858 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f40340186c0 0x7f403401abc0 crc :-1 s=READY pgs=811 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:58.646 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:58.657 7f4055f7b700 1 -- 172.21.2.112:0/2069877858 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f40500a8c10 msgr2=0x7f40500a2b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:58.646 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:58.657 7f4055f7b700 1 --2- 172.21.2.112:0/2069877858 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f40500a8c10 0x7f40500a2b60 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 rx=0x7f40480099f0 tx=0x7f404800bd10).stop 2021-02-10T11:36:58.646 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:58.658 7f4055f7b700 1 -- 172.21.2.112:0/2069877858 shutdown_connections 2021-02-10T11:36:58.646 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:58.658 7f4055f7b700 1 --2- 172.21.2.112:0/2069877858 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f40340186c0 0x7f403401abc0 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:58.646 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:58.658 7f4055f7b700 1 --2- 172.21.2.112:0/2069877858 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f40500a8810 0x7f40500a2540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:58.646 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:58.658 7f4055f7b700 1 --2- 172.21.2.112:0/2069877858 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f40500a8c10 0x7f40500a2b60 unknown :-1 s=CLOSED pgs=1709 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:58.646 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:58.658 7f4055f7b700 1 --2- 172.21.2.112:0/2069877858 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f40500b5fb0 0x7f40500a81f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:58.646 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:58.658 7f4055f7b700 1 -- 172.21.2.112:0/2069877858 >> 172.21.2.112:0/2069877858 conn(0x7f405009e660 msgr2=0x7f40500a9550 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:58.646 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:58.658 7f4055f7b700 1 -- 172.21.2.112:0/2069877858 shutdown_connections 2021-02-10T11:36:58.646 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:58.658 7f4055f7b700 1 -- 172.21.2.112:0/2069877858 wait complete. 2021-02-10T11:36:58.654 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.3 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:36:59.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.090 7f78f58e5700 1 Processor -- start 2021-02-10T11:36:59.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.090 7f78f58e5700 1 -- start start 2021-02-10T11:36:59.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.090 7f78f58e5700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f78f00d1c80 0x7f78f00cef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:59.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.090 7f78f58e5700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f78f00cf590 0x7f78f00cf990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:59.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.090 7f78f58e5700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78f00fe1d0 con 0x7f78f00d1880 2021-02-10T11:36:59.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.090 7f78f58e5700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78f01b0cc0 con 0x7f78f00cf590 2021-02-10T11:36:59.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.090 7f78f58e5700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78f0109a70 con 0x7f78f00d1c80 2021-02-10T11:36:59.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.090 7f78ef574700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f78f00d1880 0x7f78f01b04d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:51998/0 (socket says 172.21.2.112:51998) 2021-02-10T11:36:59.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.091 7f78ef574700 1 -- 172.21.2.112:0/3014129771 learned_addr learned my addr 172.21.2.112:0/3014129771 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:36:59.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.091 7f78ee572700 1 -- 172.21.2.112:0/3014129771 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 140973921 0 0) 0x7f78e0001180 con 0x7f78f00d1880 2021-02-10T11:36:59.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.091 7f78ee572700 1 -- 172.21.2.112:0/3014129771 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78d8002fe0 con 0x7f78f00d1880 2021-02-10T11:36:59.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.091 7f78efd75700 1 --2- 172.21.2.112:0/3014129771 >> v2:172.21.2.120:3300/0 conn(0x7f78f00cf590 0x7f78f00cf990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:59.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.091 7f78eed73700 1 --2- 172.21.2.112:0/3014129771 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f78f00d1c80 0x7f78f00cef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:59.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.091 7f78ee572700 1 -- 172.21.2.112:0/3014129771 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1078813668 0 0) 0x7f78e0001180 con 0x7f78f00d1880 2021-02-10T11:36:59.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.091 7f78ee572700 1 -- 172.21.2.112:0/3014129771 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f78f00d1c80 msgr2=0x7f78f00cef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:59.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.091 7f78ee572700 1 --2- 172.21.2.112:0/3014129771 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f78f00d1c80 0x7f78f00cef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:59.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.091 7f78ee572700 1 -- 172.21.2.112:0/3014129771 >> v2:172.21.2.120:3300/0 conn(0x7f78f00cf590 msgr2=0x7f78f00cf990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:59.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.091 7f78ee572700 1 --2- 172.21.2.112:0/3014129771 >> v2:172.21.2.120:3300/0 conn(0x7f78f00cf590 0x7f78f00cf990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:59.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.091 7f78ee572700 1 -- 172.21.2.112:0/3014129771 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78f0088070 con 0x7f78f00d1880 2021-02-10T11:36:59.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.091 7f78ee572700 1 -- 172.21.2.112:0/3014129771 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f78e0001680 con 0x7f78f00d1880 2021-02-10T11:36:59.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.091 7f78ee572700 1 -- 172.21.2.112:0/3014129771 >> v1:172.21.2.112:6789/0 conn(0x7f78f00d1880 legacy=0x7f78f01b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:59.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.091 7f78ee572700 1 --2- 172.21.2.112:0/3014129771 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f78d8003a60 0x7f78d8005f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:59.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.091 7f78ee572700 1 --2- 172.21.2.112:0/3014129771 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f78d8006780 0x7f78d8008ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:59.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.091 7f78ee572700 1 --2- 172.21.2.112:0/3014129771 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f78d80095a0 0x7f78d800ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:59.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.091 7f78ee572700 1 -- 172.21.2.112:0/3014129771 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78d800c2c0 con 0x7f78d8006780 2021-02-10T11:36:59.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.091 7f78ee572700 1 -- 172.21.2.112:0/3014129771 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78d80504b0 con 0x7f78d8003a60 2021-02-10T11:36:59.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.091 7f78ee572700 1 -- 172.21.2.112:0/3014129771 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78d80506d0 con 0x7f78d80095a0 2021-02-10T11:36:59.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.091 7f78eed73700 1 --2- 172.21.2.112:0/3014129771 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f78d8006780 0x7f78d8008ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:59.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.092 7f78efd75700 1 --2- 172.21.2.112:0/3014129771 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f78d80095a0 0x7f78d800ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:59.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.092 7f78ef574700 1 --2- 172.21.2.112:0/3014129771 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f78d8003a60 0x7f78d8005f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:59.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.092 7f78eed73700 1 -- 172.21.2.112:0/3014129771 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f78d80095a0 msgr2=0x7f78d800ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:59.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.092 7f78eed73700 1 --2- 172.21.2.112:0/3014129771 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f78d80095a0 0x7f78d800ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:59.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.092 7f78eed73700 1 -- 172.21.2.112:0/3014129771 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f78d8003a60 msgr2=0x7f78d8005f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:59.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.092 7f78eed73700 1 --2- 172.21.2.112:0/3014129771 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f78d8003a60 0x7f78d8005f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:59.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.092 7f78eed73700 1 -- 172.21.2.112:0/3014129771 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78d8050970 con 0x7f78d8006780 2021-02-10T11:36:59.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.092 7f78eed73700 1 --2- 172.21.2.112:0/3014129771 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f78d8006780 0x7f78d8008ca0 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 rx=0x7f78e4006ad0 tx=0x7f78e4009b40).ready entity=mon.0 client_cookie=33635a6ab1fd5a4b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:59.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.092 7f78ee572700 1 -- 172.21.2.112:0/3014129771 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f78e40066d0 con 0x7f78d8006780 2021-02-10T11:36:59.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.092 7f78ee572700 1 -- 172.21.2.112:0/3014129771 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78e400c0c0 con 0x7f78d8006780 2021-02-10T11:36:59.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.092 7f78ee572700 1 -- 172.21.2.112:0/3014129771 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f78e400c5e0 con 0x7f78d8006780 2021-02-10T11:36:59.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.092 7f78f58e5700 1 -- 172.21.2.112:0/3014129771 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f78d8006780 msgr2=0x7f78d8008ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:59.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.092 7f78f58e5700 1 --2- 172.21.2.112:0/3014129771 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f78d8006780 0x7f78d8008ca0 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 rx=0x7f78e4006ad0 tx=0x7f78e4009b40).stop 2021-02-10T11:36:59.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.092 7f78efd75700 1 -- 172.21.2.112:0/3014129771 reap_dead start 2021-02-10T11:36:59.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.093 7f78f58e5700 1 -- 172.21.2.112:0/3014129771 shutdown_connections 2021-02-10T11:36:59.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.093 7f78f58e5700 1 -- 172.21.2.112:0/3014129771 >> 172.21.2.112:0/3014129771 conn(0x7f78f00ce430 msgr2=0x7f78f00d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:59.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.093 7f78f58e5700 1 -- 172.21.2.112:0/3014129771 shutdown_connections 2021-02-10T11:36:59.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.093 7f78f58e5700 1 -- 172.21.2.112:0/3014129771 wait complete. 2021-02-10T11:36:59.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.108 7f78f58e5700 1 Processor -- start 2021-02-10T11:36:59.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.108 7f78f58e5700 1 -- start start 2021-02-10T11:36:59.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.108 7f78f58e5700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f78f01a0580 0x7f78f0086fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:59.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.108 7f78f58e5700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f78f00b66b0 0x7f78f00b6ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:59.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.108 7f78f58e5700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f78f0082bb0 0x7f78f0082fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:59.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.108 7f78f58e5700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78f00875d0 con 0x7f78f0082bb0 2021-02-10T11:36:59.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.108 7f78f58e5700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78f00835d0 con 0x7f78f01a0580 2021-02-10T11:36:59.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.108 7f78f58e5700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78f00837f0 con 0x7f78f00b66b0 2021-02-10T11:36:59.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.108 7f78efd75700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f78f0082bb0 0x7f78f0082fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:59.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.109 7f78efd75700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f78f0082bb0 0x7f78f0082fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:52550/0 (socket says 172.21.2.112:52550) 2021-02-10T11:36:59.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.109 7f78efd75700 1 -- 172.21.2.112:0/2669077521 learned_addr learned my addr 172.21.2.112:0/2669077521 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:36:59.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.109 7f78efd75700 1 -- 172.21.2.112:0/2669077521 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f78f00b66b0 msgr2=0x7f78f00b6ab0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:36:59.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.109 7f78efd75700 1 --2- 172.21.2.112:0/2669077521 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f78f00b66b0 0x7f78f00b6ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:59.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.109 7f78efd75700 1 -- 172.21.2.112:0/2669077521 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f78f01a0580 msgr2=0x7f78f0086fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:59.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.109 7f78efd75700 1 --2- 172.21.2.112:0/2669077521 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f78f01a0580 0x7f78f0086fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:59.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.109 7f78efd75700 1 -- 172.21.2.112:0/2669077521 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78f0083a10 con 0x7f78f0082bb0 2021-02-10T11:36:59.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.109 7f78efd75700 1 --2- 172.21.2.112:0/2669077521 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f78f0082bb0 0x7f78f0082fb0 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 rx=0x7f78e800b8b0 tx=0x7f78e800bc50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:59.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.109 7f78d7fff700 1 -- 172.21.2.112:0/2669077521 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f78e80090c0 con 0x7f78f0082bb0 2021-02-10T11:36:59.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.109 7f78d7fff700 1 -- 172.21.2.112:0/2669077521 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78e80011f0 con 0x7f78f0082bb0 2021-02-10T11:36:59.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.109 7f78f58e5700 1 -- 172.21.2.112:0/2669077521 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78f0083a10 con 0x7f78f0082bb0 2021-02-10T11:36:59.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.109 7f78d7fff700 1 -- 172.21.2.112:0/2669077521 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f78e8001710 con 0x7f78f0082bb0 2021-02-10T11:36:59.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.109 7f78f58e5700 1 -- 172.21.2.112:0/2669077521 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78f0083fa0 con 0x7f78f0082bb0 2021-02-10T11:36:59.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.110 7f78d7fff700 1 -- 172.21.2.112:0/2669077521 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f78e8000df0 con 0x7f78f0082bb0 2021-02-10T11:36:59.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.110 7f78d7fff700 1 --2- 172.21.2.112:0/2669077521 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f78d0018730 0x7f78d001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:59.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.110 7f78d7fff700 1 -- 172.21.2.112:0/2669077521 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f78e8032990 con 0x7f78f0082bb0 2021-02-10T11:36:59.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.110 7f78ef574700 1 --2- 172.21.2.112:0/2669077521 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f78d0018730 0x7f78d001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:59.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.111 7f78ef574700 1 --2- 172.21.2.112:0/2669077521 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f78d0018730 0x7f78d001ac10 crc :-1 s=READY pgs=812 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:59.127 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.140 7f78f58e5700 1 -- 172.21.2.112:0/2669077521 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78f0082980 con 0x7f78f0082bb0 2021-02-10T11:36:59.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.143 7f78d7fff700 1 -- 172.21.2.112:0/2669077521 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f78e800bd90 con 0x7f78f0082bb0 2021-02-10T11:36:59.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.273 7f78f58e5700 1 -- 172.21.2.112:0/2669077521 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f78f0082980 con 0x7f78f0082bb0 2021-02-10T11:36:59.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.276 7f78d7fff700 1 -- 172.21.2.112:0/2669077521 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.3 v52) v1 ==== 184+0+0 (secure 0 0 0) 0x7f78e801f850 con 0x7f78f0082bb0 2021-02-10T11:36:59.268 INFO:teuthology.orchestra.run.gibba012.stderr:updated caps for client.3 2021-02-10T11:36:59.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.282 7f78f58e5700 1 -- 172.21.2.112:0/2669077521 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f78d0018730 msgr2=0x7f78d001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:59.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.282 7f78f58e5700 1 --2- 172.21.2.112:0/2669077521 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f78d0018730 0x7f78d001ac10 crc :-1 s=READY pgs=812 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:59.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.282 7f78f58e5700 1 -- 172.21.2.112:0/2669077521 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f78f0082bb0 msgr2=0x7f78f0082fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:59.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.282 7f78f58e5700 1 --2- 172.21.2.112:0/2669077521 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f78f0082bb0 0x7f78f0082fb0 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 rx=0x7f78e800b8b0 tx=0x7f78e800bc50).stop 2021-02-10T11:36:59.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.282 7f78f58e5700 1 -- 172.21.2.112:0/2669077521 shutdown_connections 2021-02-10T11:36:59.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.282 7f78f58e5700 1 --2- 172.21.2.112:0/2669077521 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f78d0018730 0x7f78d001ac10 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:59.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.282 7f78f58e5700 1 --2- 172.21.2.112:0/2669077521 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f78f0082bb0 0x7f78f0082fb0 unknown :-1 s=CLOSED pgs=1712 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:59.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.282 7f78f58e5700 1 --2- 172.21.2.112:0/2669077521 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f78f00b66b0 0x7f78f00b6ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:59.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.282 7f78f58e5700 1 --2- 172.21.2.112:0/2669077521 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f78f01a0580 0x7f78f0086fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:59.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.282 7f78f58e5700 1 -- 172.21.2.112:0/2669077521 >> 172.21.2.112:0/2669077521 conn(0x7f78f00afc60 msgr2=0x7f78f00b50d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:59.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.282 7f78f58e5700 1 -- 172.21.2.112:0/2669077521 shutdown_connections 2021-02-10T11:36:59.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.282 7f78f58e5700 1 -- 172.21.2.112:0/2669077521 wait complete. 2021-02-10T11:36:59.278 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:36:59.707 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.703 7fb56742a700 1 Processor -- start 2021-02-10T11:36:59.708 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.703 7fb56742a700 1 -- start start 2021-02-10T11:36:59.708 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.704 7fb56742a700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fb5600d1870 0x7fb5601b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:59.708 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.704 7fb56742a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb5600f0650 0x7fb5600cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:59.708 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.704 7fb56742a700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5600f0b10 con 0x7fb5601b0a10 2021-02-10T11:36:59.708 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.704 7fb56742a700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5600e5ed0 con 0x7fb5600d1870 2021-02-10T11:36:59.708 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.704 7fb56742a700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5600d1c70 con 0x7fb5600f0650 2021-02-10T11:36:59.708 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.704 7fb564a45700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fb5601b0a10 0x7fb5600cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:52018/0 (socket says 172.21.2.112:52018) 2021-02-10T11:36:59.708 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.704 7fb564a45700 1 -- 172.21.2.112:0/2766905957 learned_addr learned my addr 172.21.2.112:0/2766905957 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:36:59.708 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.704 7fb553fff700 1 -- 172.21.2.112:0/2766905957 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3545046587 0 0) 0x7fb5540010b0 con 0x7fb5601b0a10 2021-02-10T11:36:59.708 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.704 7fb553fff700 1 -- 172.21.2.112:0/2766905957 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb54c002fe0 con 0x7fb5601b0a10 2021-02-10T11:36:59.708 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.704 7fb553fff700 1 -- 172.21.2.112:0/2766905957 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1045992423 0 0) 0x7fb5540010b0 con 0x7fb5601b0a10 2021-02-10T11:36:59.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.704 7fb565a47700 1 --2- 172.21.2.112:0/2766905957 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb5600f0650 0x7fb5600cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:59.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.704 7fb565246700 1 --2- 172.21.2.112:0/2766905957 >> v2:172.21.2.120:3300/0 conn(0x7fb5600d1870 0x7fb5601b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:59.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.704 7fb553fff700 1 -- 172.21.2.112:0/2766905957 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb5600f0650 msgr2=0x7fb5600cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:59.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.704 7fb553fff700 1 --2- 172.21.2.112:0/2766905957 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb5600f0650 0x7fb5600cfb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:59.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.704 7fb553fff700 1 -- 172.21.2.112:0/2766905957 >> v2:172.21.2.120:3300/0 conn(0x7fb5600d1870 msgr2=0x7fb5601b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:59.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.704 7fb553fff700 1 --2- 172.21.2.112:0/2766905957 >> v2:172.21.2.120:3300/0 conn(0x7fb5600d1870 0x7fb5601b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:59.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.704 7fb553fff700 1 -- 172.21.2.112:0/2766905957 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5600d1e90 con 0x7fb5601b0a10 2021-02-10T11:36:59.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.704 7fb553fff700 1 -- 172.21.2.112:0/2766905957 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7fb5540015b0 con 0x7fb5601b0a10 2021-02-10T11:36:59.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.705 7fb553fff700 1 -- 172.21.2.112:0/2766905957 >> v1:172.21.2.112:6789/0 conn(0x7fb5601b0a10 legacy=0x7fb5600cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:59.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.705 7fb553fff700 1 --2- 172.21.2.112:0/2766905957 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb54c003b90 0x7fb54c006040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:59.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.705 7fb553fff700 1 --2- 172.21.2.112:0/2766905957 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb54c0068e0 0x7fb54c008dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:59.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.705 7fb553fff700 1 --2- 172.21.2.112:0/2766905957 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb54c009670 0x7fb54c00bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:59.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.705 7fb553fff700 1 -- 172.21.2.112:0/2766905957 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb54c00c420 con 0x7fb54c003b90 2021-02-10T11:36:59.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.705 7fb553fff700 1 -- 172.21.2.112:0/2766905957 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb54c00c640 con 0x7fb54c009670 2021-02-10T11:36:59.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.705 7fb553fff700 1 -- 172.21.2.112:0/2766905957 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb54c00c860 con 0x7fb54c0068e0 2021-02-10T11:36:59.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.705 7fb565246700 1 --2- 172.21.2.112:0/2766905957 >> v2:172.21.2.120:3300/0 conn(0x7fb5600d1870 0x7fb5601b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:36:59.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.705 7fb565a47700 1 --2- 172.21.2.112:0/2766905957 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb5600f0650 0x7fb5600cfb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:36:59.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.705 7fb565246700 1 --2- 172.21.2.112:0/2766905957 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb54c003b90 0x7fb54c006040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:59.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.705 7fb564a45700 1 --2- 172.21.2.112:0/2766905957 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb54c0068e0 0x7fb54c008dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:59.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.705 7fb565a47700 1 --2- 172.21.2.112:0/2766905957 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb54c009670 0x7fb54c00bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:59.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.705 7fb565246700 1 -- 172.21.2.112:0/2766905957 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb54c0068e0 msgr2=0x7fb54c008dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:59.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.705 7fb565246700 1 --2- 172.21.2.112:0/2766905957 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb54c0068e0 0x7fb54c008dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:59.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.705 7fb565246700 1 -- 172.21.2.112:0/2766905957 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb54c009670 msgr2=0x7fb54c00bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:59.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.705 7fb565246700 1 --2- 172.21.2.112:0/2766905957 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb54c009670 0x7fb54c00bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:59.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.705 7fb565246700 1 -- 172.21.2.112:0/2766905957 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb54c050b10 con 0x7fb54c003b90 2021-02-10T11:36:59.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.705 7fb565a47700 1 --2- 172.21.2.112:0/2766905957 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb54c009670 0x7fb54c00bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:36:59.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.705 7fb565246700 1 --2- 172.21.2.112:0/2766905957 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb54c003b90 0x7fb54c006040 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 rx=0x7fb55c0098a0 tx=0x7fb55c009c40).ready entity=mon.0 client_cookie=f6829ce5980e99d5 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:59.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.705 7fb553fff700 1 -- 172.21.2.112:0/2766905957 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb55c0065d0 con 0x7fb54c003b90 2021-02-10T11:36:59.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.705 7fb553fff700 1 -- 172.21.2.112:0/2766905957 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb55c000cd0 con 0x7fb54c003b90 2021-02-10T11:36:59.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.705 7fb553fff700 1 -- 172.21.2.112:0/2766905957 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb55c002220 con 0x7fb54c003b90 2021-02-10T11:36:59.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.705 7fb56742a700 1 -- 172.21.2.112:0/2766905957 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb54c003b90 msgr2=0x7fb54c006040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:59.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.705 7fb56742a700 1 --2- 172.21.2.112:0/2766905957 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb54c003b90 0x7fb54c006040 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 rx=0x7fb55c0098a0 tx=0x7fb55c009c40).stop 2021-02-10T11:36:59.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.705 7fb565a47700 1 -- 172.21.2.112:0/2766905957 reap_dead start 2021-02-10T11:36:59.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.706 7fb56742a700 1 -- 172.21.2.112:0/2766905957 shutdown_connections 2021-02-10T11:36:59.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.706 7fb56742a700 1 -- 172.21.2.112:0/2766905957 >> 172.21.2.112:0/2766905957 conn(0x7fb5600ce450 msgr2=0x7fb5600d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:59.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.706 7fb56742a700 1 -- 172.21.2.112:0/2766905957 shutdown_connections 2021-02-10T11:36:59.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.706 7fb56742a700 1 -- 172.21.2.112:0/2766905957 wait complete. 2021-02-10T11:36:59.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.721 7fb56742a700 1 Processor -- start 2021-02-10T11:36:59.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.721 7fb56742a700 1 -- start start 2021-02-10T11:36:59.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.721 7fb56742a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb560195650 0x7fb5600b5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:59.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.721 7fb56742a700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb5600b5b80 0x7fb5601a2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:59.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.721 7fb56742a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb5600acfa0 0x7fb5601a3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:59.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.721 7fb56742a700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5600b6010 con 0x7fb560195650 2021-02-10T11:36:59.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.721 7fb56742a700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5601a3a60 con 0x7fb5600b5b80 2021-02-10T11:36:59.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.721 7fb56742a700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5601a3c40 con 0x7fb5600acfa0 2021-02-10T11:36:59.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.722 7fb565246700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb560195650 0x7fb5600b5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:59.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.722 7fb565246700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb560195650 0x7fb5600b5560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:52566/0 (socket says 172.21.2.112:52566) 2021-02-10T11:36:59.714 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.722 7fb565246700 1 -- 172.21.2.112:0/3888412594 learned_addr learned my addr 172.21.2.112:0/3888412594 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:36:59.714 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.722 7fb565246700 1 -- 172.21.2.112:0/3888412594 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb5600acfa0 msgr2=0x7fb5601a3440 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:36:59.714 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.722 7fb565246700 1 --2- 172.21.2.112:0/3888412594 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb5600acfa0 0x7fb5601a3440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:59.714 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.722 7fb565246700 1 -- 172.21.2.112:0/3888412594 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb5600b5b80 msgr2=0x7fb5601a2e20 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:36:59.714 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.722 7fb565246700 1 --2- 172.21.2.112:0/3888412594 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb5600b5b80 0x7fb5601a2e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:59.714 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.722 7fb565246700 1 -- 172.21.2.112:0/3888412594 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5600bcae0 con 0x7fb560195650 2021-02-10T11:36:59.714 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.722 7fb565246700 1 --2- 172.21.2.112:0/3888412594 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb560195650 0x7fb5600b5560 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 rx=0x7fb55c006f40 tx=0x7fb55c00b9e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:59.714 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.722 7fb551ffb700 1 -- 172.21.2.112:0/3888412594 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb55c00bdd0 con 0x7fb560195650 2021-02-10T11:36:59.714 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.722 7fb551ffb700 1 -- 172.21.2.112:0/3888412594 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb55c006030 con 0x7fb560195650 2021-02-10T11:36:59.714 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.722 7fb56742a700 1 -- 172.21.2.112:0/3888412594 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5600bcae0 con 0x7fb560195650 2021-02-10T11:36:59.714 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.722 7fb551ffb700 1 -- 172.21.2.112:0/3888412594 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb55c0054d0 con 0x7fb560195650 2021-02-10T11:36:59.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.722 7fb56742a700 1 -- 172.21.2.112:0/3888412594 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5600bd070 con 0x7fb560195650 2021-02-10T11:36:59.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.723 7fb551ffb700 1 -- 172.21.2.112:0/3888412594 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb55c000940 con 0x7fb560195650 2021-02-10T11:36:59.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.723 7fb551ffb700 1 --2- 172.21.2.112:0/3888412594 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb540018730 0x7fb54001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:59.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.723 7fb551ffb700 1 -- 172.21.2.112:0/3888412594 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fb55c032810 con 0x7fb560195650 2021-02-10T11:36:59.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.724 7fb564a45700 1 --2- 172.21.2.112:0/3888412594 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb540018730 0x7fb54001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:59.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.725 7fb564a45700 1 --2- 172.21.2.112:0/3888412594 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb540018730 0x7fb54001ac10 crc :-1 s=READY pgs=813 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:59.740 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.754 7fb56742a700 1 -- 172.21.2.112:0/3888412594 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5600f0ab0 con 0x7fb560195650 2021-02-10T11:36:59.743 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.756 7fb551ffb700 1 -- 172.21.2.112:0/3888412594 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb55c0065d0 con 0x7fb560195650 2021-02-10T11:36:59.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.885 7fb56742a700 1 -- 172.21.2.112:0/3888412594 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb5600f0ab0 con 0x7fb560195650 2021-02-10T11:36:59.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.886 7fb551ffb700 1 -- 172.21.2.112:0/3888412594 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) v1 ==== 94+0+2101 (secure 0 0 0) 0x7fb55c00bdd0 con 0x7fb560195650 2021-02-10T11:36:59.873 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:36:59.873 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":64,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9204,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/1512510827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":1512510827},{"type":"v1","addr":"172.21.2.112:6833","nonce":1512510827}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":60}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:36:53.563735","modified":"2021-02-10 11:36:55.573740","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9201},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9201":{"gid":9201,"name":"b","rank":0,"incarnation":61,"state":"up:active","state_seq":3,"addr":"172.21.2.120:6835/1232001750","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":1232001750},{"type":"v1","addr":"172.21.2.120:6835","nonce":1232001750}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2021-02-10T11:36:59.873 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 64 2021-02-10T11:36:59.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.888 7fb56742a700 1 -- 172.21.2.112:0/3888412594 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb540018730 msgr2=0x7fb54001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:59.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.888 7fb56742a700 1 --2- 172.21.2.112:0/3888412594 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb540018730 0x7fb54001ac10 crc :-1 s=READY pgs=813 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:59.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.888 7fb56742a700 1 -- 172.21.2.112:0/3888412594 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb560195650 msgr2=0x7fb5600b5560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:59.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.888 7fb56742a700 1 --2- 172.21.2.112:0/3888412594 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb560195650 0x7fb5600b5560 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 rx=0x7fb55c006f40 tx=0x7fb55c00b9e0).stop 2021-02-10T11:36:59.875 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.888 7fb56742a700 1 -- 172.21.2.112:0/3888412594 shutdown_connections 2021-02-10T11:36:59.875 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.888 7fb56742a700 1 --2- 172.21.2.112:0/3888412594 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb540018730 0x7fb54001ac10 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:59.875 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.888 7fb56742a700 1 --2- 172.21.2.112:0/3888412594 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb5600b5b80 0x7fb5601a2e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:59.875 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.888 7fb56742a700 1 --2- 172.21.2.112:0/3888412594 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb5600acfa0 0x7fb5601a3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:59.875 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.888 7fb56742a700 1 --2- 172.21.2.112:0/3888412594 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb560195650 0x7fb5600b5560 unknown :-1 s=CLOSED pgs=1715 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:59.875 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.888 7fb56742a700 1 -- 172.21.2.112:0/3888412594 >> 172.21.2.112:0/3888412594 conn(0x7fb5600b3040 msgr2=0x7fb5600c3250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:59.876 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.888 7fb56742a700 1 -- 172.21.2.112:0/3888412594 shutdown_connections 2021-02-10T11:36:59.876 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:36:59.888 7fb56742a700 1 -- 172.21.2.112:0/3888412594 wait complete. 2021-02-10T11:37:00.085 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 64, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:36:53.563735', 'modified': '2021-02-10 11:36:55.573740', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9201}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9201': {'gid': 9201, 'name': 'b', 'rank': 0, 'incarnation': 61, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.120:6835/1232001750', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.120:6834', 'nonce': 1232001750}, {'type': 'v1', 'addr': '172.21.2.120:6835', 'nonce': 1232001750}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:37:00.085 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:37:00.085 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:37:00.483 INFO:teuthology.orchestra.run.gibba020.stdout:{ 2021-02-10T11:37:00.483 INFO:teuthology.orchestra.run.gibba020.stdout: "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", 2021-02-10T11:37:00.483 INFO:teuthology.orchestra.run.gibba020.stdout: "whoami": 0, 2021-02-10T11:37:00.483 INFO:teuthology.orchestra.run.gibba020.stdout: "id": 9201, 2021-02-10T11:37:00.483 INFO:teuthology.orchestra.run.gibba020.stdout: "want_state": "up:active", 2021-02-10T11:37:00.483 INFO:teuthology.orchestra.run.gibba020.stdout: "state": "up:active", 2021-02-10T11:37:00.483 INFO:teuthology.orchestra.run.gibba020.stdout: "rank_uptime": 6.9305039749999997, 2021-02-10T11:37:00.483 INFO:teuthology.orchestra.run.gibba020.stdout: "mdsmap_epoch": 64, 2021-02-10T11:37:00.483 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch": 83, 2021-02-10T11:37:00.483 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch_barrier": 83, 2021-02-10T11:37:00.483 INFO:teuthology.orchestra.run.gibba020.stdout: "uptime": 12.568272713000001 2021-02-10T11:37:00.484 INFO:teuthology.orchestra.run.gibba020.stdout:} 2021-02-10T11:37:00.487 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", "whoami": 0, "id": 9201, "want_state": "up:active", "state": "up:active", "rank_uptime": 6.9305039749999997, "mdsmap_epoch": 64, "osdmap_epoch": 83, "osdmap_epoch_barrier": 83, "uptime": 12.568272713000001 } 2021-02-10T11:37:00.487 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:37:00.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:00.912 7f94c8be5700 1 Processor -- start 2021-02-10T11:37:00.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:00.912 7f94c8be5700 1 -- start start 2021-02-10T11:37:00.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:00.912 7f94c8be5700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f94c41b0470 0x7f94c41b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:00.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:00.912 7f94c8be5700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f94c40ef610 0x7f94c40efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:00.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:00.912 7f94c8be5700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94c40f0b50 con 0x7f94c40f0030 2021-02-10T11:37:00.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:00.912 7f94c8be5700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94c40e52e0 con 0x7f94c41b0470 2021-02-10T11:37:00.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:00.912 7f94c8be5700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94c40e5490 con 0x7f94c40ef610 2021-02-10T11:37:00.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:00.913 7f94c285e700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f94c41b0470 0x7f94c41b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:00.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:00.913 7f94c285e700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f94c41b0470 0x7f94c41b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:52574/0 (socket says 172.21.2.112:52574) 2021-02-10T11:37:00.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:00.913 7f94c285e700 1 -- 172.21.2.112:0/1853545732 learned_addr learned my addr 172.21.2.112:0/1853545732 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:37:00.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:00.913 7f94c285e700 1 -- 172.21.2.112:0/1853545732 >> v2:172.21.2.120:3300/0 conn(0x7f94c40ef610 msgr2=0x7f94c40efa10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:37:00.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:00.913 7f94c285e700 1 --2- 172.21.2.112:0/1853545732 >> v2:172.21.2.120:3300/0 conn(0x7f94c40ef610 0x7f94c40efa10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:00.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:00.913 7f94c285e700 1 -- 172.21.2.112:0/1853545732 >> v1:172.21.2.120:6789/0 conn(0x7f94c40f0030 legacy=0x7f94c40f10d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:37:00.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:00.913 7f94c285e700 1 -- 172.21.2.112:0/1853545732 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94c40f0430 con 0x7f94c41b0470 2021-02-10T11:37:00.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:00.913 7f94c285e700 1 --2- 172.21.2.112:0/1853545732 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f94c41b0470 0x7f94c41b0870 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 rx=0x7f94ac006700 tx=0x7f94ac009b00).ready entity=mon.0 client_cookie=355303d5e4f657db server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:00.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:00.913 7f94c185c700 1 -- 172.21.2.112:0/1853545732 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f94ac00c0c0 con 0x7f94c41b0470 2021-02-10T11:37:00.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:00.913 7f94c185c700 1 -- 172.21.2.112:0/1853545732 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94ac00c400 con 0x7f94c41b0470 2021-02-10T11:37:00.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:00.913 7f94c185c700 1 -- 172.21.2.112:0/1853545732 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f94ac00c920 con 0x7f94c41b0470 2021-02-10T11:37:00.917 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:00.913 7f94c8be5700 1 -- 172.21.2.112:0/1853545732 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f94c41b0470 msgr2=0x7f94c41b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:00.917 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:00.913 7f94c8be5700 1 --2- 172.21.2.112:0/1853545732 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f94c41b0470 0x7f94c41b0870 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 rx=0x7f94ac006700 tx=0x7f94ac009b00).stop 2021-02-10T11:37:00.917 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:00.914 7f94c8be5700 1 -- 172.21.2.112:0/1853545732 shutdown_connections 2021-02-10T11:37:00.917 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:00.914 7f94c8be5700 1 --2- 172.21.2.112:0/1853545732 >> v2:172.21.2.120:3300/0 conn(0x7f94c40ef610 0x7f94c40efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:00.917 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:00.914 7f94c8be5700 1 --2- 172.21.2.112:0/1853545732 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f94c41b0470 0x7f94c41b0870 unknown :-1 s=CLOSED pgs=1716 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:00.917 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:00.914 7f94c8be5700 1 -- 172.21.2.112:0/1853545732 >> 172.21.2.112:0/1853545732 conn(0x7f94c40ce3e0 msgr2=0x7f94c40e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:00.917 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:00.914 7f94c8be5700 1 -- 172.21.2.112:0/1853545732 shutdown_connections 2021-02-10T11:37:00.917 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:00.914 7f94c8be5700 1 -- 172.21.2.112:0/1853545732 wait complete. 2021-02-10T11:37:00.918 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:00.928 7f94c8be5700 1 Processor -- start 2021-02-10T11:37:00.919 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:00.929 7f94c8be5700 1 -- start start 2021-02-10T11:37:00.919 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:00.929 7f94c8be5700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f94c40b0970 0x7f94c4092ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:00.919 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:00.929 7f94c8be5700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f94c40934f0 0x7f94c4094dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:00.919 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:00.929 7f94c8be5700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f94c40938f0 0x7f94c40953e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:00.919 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:00.929 7f94c8be5700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94c4195ad0 con 0x7f94c40934f0 2021-02-10T11:37:00.919 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:00.929 7f94c8be5700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94c4095f40 con 0x7f94c40938f0 2021-02-10T11:37:00.919 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:00.929 7f94c8be5700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94c4095a30 con 0x7f94c40b0970 2021-02-10T11:37:00.919 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:00.929 7f94c205d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f94c40934f0 0x7f94c4094dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:00.919 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:00.929 7f94c205d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f94c40934f0 0x7f94c4094dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:52582/0 (socket says 172.21.2.112:52582) 2021-02-10T11:37:00.919 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:00.929 7f94c205d700 1 -- 172.21.2.112:0/3990354394 learned_addr learned my addr 172.21.2.112:0/3990354394 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:37:00.919 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:00.929 7f94c205d700 1 -- 172.21.2.112:0/3990354394 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f94c40b0970 msgr2=0x7f94c4092ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:00.919 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:00.929 7f94c205d700 1 --2- 172.21.2.112:0/3990354394 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f94c40b0970 0x7f94c4092ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:00.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:00.929 7f94c205d700 1 -- 172.21.2.112:0/3990354394 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f94c40938f0 msgr2=0x7f94c40953e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:00.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:00.930 7f94c205d700 1 --2- 172.21.2.112:0/3990354394 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f94c40938f0 0x7f94c40953e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:00.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:00.930 7f94c205d700 1 -- 172.21.2.112:0/3990354394 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94c4095c50 con 0x7f94c40934f0 2021-02-10T11:37:00.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:00.930 7f94c205d700 1 --2- 172.21.2.112:0/3990354394 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f94c40934f0 0x7f94c4094dc0 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 rx=0x7f94b400b560 tx=0x7f94b400b900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:00.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:00.930 7f94bb7fe700 1 -- 172.21.2.112:0/3990354394 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f94b400bcf0 con 0x7f94c40934f0 2021-02-10T11:37:00.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:00.930 7f94bb7fe700 1 -- 172.21.2.112:0/3990354394 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94b4006030 con 0x7f94c40934f0 2021-02-10T11:37:00.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:00.930 7f94bb7fe700 1 -- 172.21.2.112:0/3990354394 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f94b40054d0 con 0x7f94c40934f0 2021-02-10T11:37:00.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:00.930 7f94c8be5700 1 -- 172.21.2.112:0/3990354394 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94c4095c50 con 0x7f94c40934f0 2021-02-10T11:37:00.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:00.930 7f94c8be5700 1 -- 172.21.2.112:0/3990354394 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94c40967b0 con 0x7f94c40934f0 2021-02-10T11:37:00.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:00.931 7f94bb7fe700 1 -- 172.21.2.112:0/3990354394 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f94b4006210 con 0x7f94c40934f0 2021-02-10T11:37:00.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:00.931 7f94bb7fe700 1 --2- 172.21.2.112:0/3990354394 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f94a80186c0 0x7f94a801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:00.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:00.931 7f94bb7fe700 1 -- 172.21.2.112:0/3990354394 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f94b40325a0 con 0x7f94c40934f0 2021-02-10T11:37:00.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:00.931 7f94c285e700 1 --2- 172.21.2.112:0/3990354394 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f94a80186c0 0x7f94a801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:00.921 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:00.932 7f94c285e700 1 --2- 172.21.2.112:0/3990354394 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f94a80186c0 0x7f94a801abc0 crc :-1 s=READY pgs=814 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:00.949 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:00.963 7f94c8be5700 1 -- 172.21.2.112:0/3990354394 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94c40e52e0 con 0x7f94c40934f0 2021-02-10T11:37:00.952 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:00.965 7f94bb7fe700 1 -- 172.21.2.112:0/3990354394 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f94b4006b80 con 0x7f94c40934f0 2021-02-10T11:37:01.077 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:01.090 7f94c8be5700 1 -- 172.21.2.112:0/3990354394 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f94c40e52e0 con 0x7f94c40934f0 2021-02-10T11:37:01.077 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:01.091 7f94bb7fe700 1 -- 172.21.2.112:0/3990354394 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) v1 ==== 74+0+13996 (secure 0 0 0) 0x7f94b4006b80 con 0x7f94c40934f0 2021-02-10T11:37:01.078 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:37:01.078 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":83,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:36:54.566363","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:36:51.538038","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":17,"pool_name":"cephfs_data","create_time":"2021-02-10 11:36:52.609006","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:37:01.079 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:01.092 7f94c8be5700 1 -- 172.21.2.112:0/3990354394 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f94a80186c0 msgr2=0x7f94a801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:01.080 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:01.092 7f94c8be5700 1 --2- 172.21.2.112:0/3990354394 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f94a80186c0 0x7f94a801abc0 crc :-1 s=READY pgs=814 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:01.080 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:01.092 7f94c8be5700 1 -- 172.21.2.112:0/3990354394 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f94c40934f0 msgr2=0x7f94c4094dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:01.080 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:01.092 7f94c8be5700 1 --2- 172.21.2.112:0/3990354394 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f94c40934f0 0x7f94c4094dc0 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 rx=0x7f94b400b560 tx=0x7f94b400b900).stop 2021-02-10T11:37:01.080 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:01.092 7f94c8be5700 1 -- 172.21.2.112:0/3990354394 shutdown_connections 2021-02-10T11:37:01.080 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:01.092 7f94c8be5700 1 --2- 172.21.2.112:0/3990354394 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f94a80186c0 0x7f94a801abc0 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:01.080 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:01.092 7f94c8be5700 1 --2- 172.21.2.112:0/3990354394 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f94c40938f0 0x7f94c40953e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:01.080 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:01.092 7f94c8be5700 1 --2- 172.21.2.112:0/3990354394 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f94c40934f0 0x7f94c4094dc0 unknown :-1 s=CLOSED pgs=1717 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:01.080 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:01.092 7f94c8be5700 1 --2- 172.21.2.112:0/3990354394 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f94c40b0970 0x7f94c4092ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:01.080 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:01.092 7f94c8be5700 1 -- 172.21.2.112:0/3990354394 >> 172.21.2.112:0/3990354394 conn(0x7f94c419f7a0 msgr2=0x7f94c40940e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:01.080 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:01.093 7f94c8be5700 1 -- 172.21.2.112:0/3990354394 shutdown_connections 2021-02-10T11:37:01.081 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:01.093 7f94c8be5700 1 -- 172.21.2.112:0/3990354394 wait complete. 2021-02-10T11:37:01.088 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:13:57.508758', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:36:51.538038', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:36:52.609006', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '83', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:37:01.088 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:37:01.520 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:01.516 7f276043d700 1 Processor -- start 2021-02-10T11:37:01.520 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:01.516 7f276043d700 1 -- start start 2021-02-10T11:37:01.520 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:01.517 7f276043d700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f27580d1c60 0x7f27580cef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:01.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:01.517 7f276043d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f27580cf570 0x7f27580cf970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:01.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:01.517 7f276043d700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27580fe1b0 con 0x7f27580d1860 2021-02-10T11:37:01.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:01.517 7f276043d700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27581b0ca0 con 0x7f27580d1c60 2021-02-10T11:37:01.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:01.517 7f276043d700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2758109a50 con 0x7f27580cf570 2021-02-10T11:37:01.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:01.517 7f275ea5a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f27580cf570 0x7f27580cf970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:01.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:01.517 7f275da58700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f27580d1c60 0x7f27580cef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:01.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:01.517 7f275e259700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7f27580d1860 0x7f27581b04b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:45606/0 (socket says 172.21.2.112:45606) 2021-02-10T11:37:01.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:01.517 7f275ea5a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f27580cf570 0x7f27580cf970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3301/0 says I am v2:172.21.2.112:52272/0 (socket says 172.21.2.112:52272) 2021-02-10T11:37:01.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:01.517 7f275e259700 1 -- 172.21.2.112:0/3265998912 learned_addr learned my addr 172.21.2.112:0/3265998912 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:37:01.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:01.518 7f275d257700 1 -- 172.21.2.112:0/3265998912 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 292569957 0 0) 0x7f2750001440 con 0x7f27580d1860 2021-02-10T11:37:01.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:01.518 7f275ea5a700 1 -- 172.21.2.112:0/3265998912 >> v2:172.21.2.120:3300/0 conn(0x7f27580d1c60 msgr2=0x7f27580cef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:37:01.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:01.518 7f275ea5a700 1 --2- 172.21.2.112:0/3265998912 >> v2:172.21.2.120:3300/0 conn(0x7f27580d1c60 0x7f27580cef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:01.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:01.518 7f275ea5a700 1 -- 172.21.2.112:0/3265998912 >> v1:172.21.2.120:6789/0 conn(0x7f27580d1860 legacy=0x7f27581b04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:01.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:01.518 7f275ea5a700 1 -- 172.21.2.112:0/3265998912 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27580d2fd0 con 0x7f27580cf570 2021-02-10T11:37:01.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:01.518 7f275ea5a700 1 --2- 172.21.2.112:0/3265998912 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f27580cf570 0x7f27580cf970 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f2754006ad0 tx=0x7f2754009ad0).ready entity=mon.2 client_cookie=81d5cf8df10534e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:01.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:01.518 7f275d257700 1 -- 172.21.2.112:0/3265998912 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f27540066a0 con 0x7f27580cf570 2021-02-10T11:37:01.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:01.518 7f275d257700 1 -- 172.21.2.112:0/3265998912 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2754006410 con 0x7f27580cf570 2021-02-10T11:37:01.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:01.518 7f275d257700 1 -- 172.21.2.112:0/3265998912 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f27540024d0 con 0x7f27580cf570 2021-02-10T11:37:01.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:01.518 7f276043d700 1 -- 172.21.2.112:0/3265998912 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f27580cf570 msgr2=0x7f27580cf970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:01.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:01.518 7f276043d700 1 --2- 172.21.2.112:0/3265998912 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f27580cf570 0x7f27580cf970 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f2754006ad0 tx=0x7f2754009ad0).stop 2021-02-10T11:37:01.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:01.519 7f276043d700 1 -- 172.21.2.112:0/3265998912 shutdown_connections 2021-02-10T11:37:01.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:01.519 7f276043d700 1 --2- 172.21.2.112:0/3265998912 >> v2:172.21.2.120:3300/0 conn(0x7f27580d1c60 0x7f27580cef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:01.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:01.519 7f276043d700 1 --2- 172.21.2.112:0/3265998912 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f27580cf570 0x7f27580cf970 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:01.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:01.519 7f276043d700 1 -- 172.21.2.112:0/3265998912 >> 172.21.2.112:0/3265998912 conn(0x7f27580ce410 msgr2=0x7f27580d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:01.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:01.519 7f276043d700 1 -- 172.21.2.112:0/3265998912 shutdown_connections 2021-02-10T11:37:01.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:01.519 7f276043d700 1 -- 172.21.2.112:0/3265998912 wait complete. 2021-02-10T11:37:01.524 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:01.534 7f276043d700 1 Processor -- start 2021-02-10T11:37:01.524 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:01.534 7f276043d700 1 -- start start 2021-02-10T11:37:01.524 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:01.534 7f276043d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f27580adf50 0x7f27580ae350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:01.524 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:01.534 7f276043d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f27580ae970 0x7f27580aed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:01.524 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:01.534 7f276043d700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f27580982c0 0x7f27580986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:01.524 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:01.534 7f276043d700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f275809f450 con 0x7f27580adf50 2021-02-10T11:37:01.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:01.534 7f276043d700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2758098ce0 con 0x7f27580982c0 2021-02-10T11:37:01.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:01.534 7f276043d700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2758098ed0 con 0x7f27580ae970 2021-02-10T11:37:01.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:01.534 7f275e259700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f27580adf50 0x7f27580ae350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:01.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:01.534 7f275e259700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f27580adf50 0x7f27580ae350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:52592/0 (socket says 172.21.2.112:52592) 2021-02-10T11:37:01.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:01.534 7f275e259700 1 -- 172.21.2.112:0/3509707165 learned_addr learned my addr 172.21.2.112:0/3509707165 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:37:01.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:01.535 7f275e259700 1 -- 172.21.2.112:0/3509707165 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f27580ae970 msgr2=0x7f27580aed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:01.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:01.535 7f275e259700 1 --2- 172.21.2.112:0/3509707165 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f27580ae970 0x7f27580aed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:01.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:01.535 7f275ea5a700 1 --2- 172.21.2.112:0/3509707165 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f27580982c0 0x7f27580986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:01.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:01.535 7f275e259700 1 -- 172.21.2.112:0/3509707165 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f27580982c0 msgr2=0x7f27580986c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:01.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:01.535 7f275e259700 1 --2- 172.21.2.112:0/3509707165 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f27580982c0 0x7f27580986c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:01.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:01.535 7f275e259700 1 -- 172.21.2.112:0/3509707165 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27580990f0 con 0x7f27580adf50 2021-02-10T11:37:01.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:01.535 7f275e259700 1 --2- 172.21.2.112:0/3509707165 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f27580adf50 0x7f27580ae350 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 rx=0x7f275000a900 tx=0x7f275000aca0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:01.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:01.535 7f274effd700 1 -- 172.21.2.112:0/3509707165 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f27500048b0 con 0x7f27580adf50 2021-02-10T11:37:01.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:01.535 7f274effd700 1 -- 172.21.2.112:0/3509707165 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2750004b70 con 0x7f27580adf50 2021-02-10T11:37:01.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:01.535 7f274effd700 1 -- 172.21.2.112:0/3509707165 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f27500070d0 con 0x7f27580adf50 2021-02-10T11:37:01.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:01.535 7f276043d700 1 -- 172.21.2.112:0/3509707165 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27580990f0 con 0x7f27580adf50 2021-02-10T11:37:01.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:01.535 7f276043d700 1 -- 172.21.2.112:0/3509707165 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2758099680 con 0x7f27580adf50 2021-02-10T11:37:01.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:01.536 7f274effd700 1 -- 172.21.2.112:0/3509707165 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2750007560 con 0x7f27580adf50 2021-02-10T11:37:01.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:01.536 7f274effd700 1 --2- 172.21.2.112:0/3509707165 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f27400186c0 0x7f274001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:01.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:01.536 7f274effd700 1 -- 172.21.2.112:0/3509707165 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f2750032860 con 0x7f27580adf50 2021-02-10T11:37:01.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:01.536 7f275da58700 1 --2- 172.21.2.112:0/3509707165 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f27400186c0 0x7f274001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:01.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:01.537 7f275da58700 1 --2- 172.21.2.112:0/3509707165 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f27400186c0 0x7f274001abc0 crc :-1 s=READY pgs=815 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:01.553 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:01.566 7f276043d700 1 -- 172.21.2.112:0/3509707165 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2758082a20 con 0x7f27580adf50 2021-02-10T11:37:01.555 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:01.569 7f274effd700 1 -- 172.21.2.112:0/3509707165 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f275002dfd0 con 0x7f27580adf50 2021-02-10T11:37:01.686 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:01.698 7f276043d700 1 -- 172.21.2.112:0/3509707165 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f2758082a20 con 0x7f27580adf50 2021-02-10T11:37:01.686 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:01.699 7f274effd700 1 -- 172.21.2.112:0/3509707165 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v83) v1 ==== 92+0+10 (secure 0 0 0) 0x7f275002dfd0 con 0x7f27580adf50 2021-02-10T11:37:01.686 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:37:01.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:01.701 7f276043d700 1 -- 172.21.2.112:0/3509707165 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f27400186c0 msgr2=0x7f274001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:01.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:01.701 7f276043d700 1 --2- 172.21.2.112:0/3509707165 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f27400186c0 0x7f274001abc0 crc :-1 s=READY pgs=815 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:01.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:01.701 7f276043d700 1 -- 172.21.2.112:0/3509707165 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f27580adf50 msgr2=0x7f27580ae350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:01.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:01.701 7f276043d700 1 --2- 172.21.2.112:0/3509707165 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f27580adf50 0x7f27580ae350 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 rx=0x7f275000a900 tx=0x7f275000aca0).stop 2021-02-10T11:37:01.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:01.701 7f276043d700 1 -- 172.21.2.112:0/3509707165 shutdown_connections 2021-02-10T11:37:01.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:01.701 7f276043d700 1 --2- 172.21.2.112:0/3509707165 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f27400186c0 0x7f274001abc0 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:01.689 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:01.701 7f276043d700 1 --2- 172.21.2.112:0/3509707165 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f27580982c0 0x7f27580986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:01.689 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:01.701 7f276043d700 1 --2- 172.21.2.112:0/3509707165 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f27580ae970 0x7f27580aed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:01.689 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:01.701 7f276043d700 1 --2- 172.21.2.112:0/3509707165 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f27580adf50 0x7f27580ae350 unknown :-1 s=CLOSED pgs=1718 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:01.689 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:01.701 7f276043d700 1 -- 172.21.2.112:0/3509707165 >> 172.21.2.112:0/3509707165 conn(0x7f27580bee90 msgr2=0x7f27580bb880 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:01.689 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:01.701 7f276043d700 1 -- 172.21.2.112:0/3509707165 shutdown_connections 2021-02-10T11:37:01.689 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:01.701 7f276043d700 1 -- 172.21.2.112:0/3509707165 wait complete. 2021-02-10T11:37:01.696 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:37:02.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.124 7f2efe647700 1 Processor -- start 2021-02-10T11:37:02.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.124 7f2efe647700 1 -- start start 2021-02-10T11:37:02.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.124 7f2efe647700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f2ef80d1870 0x7f2ef81b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:02.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.125 7f2efe647700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ef80fe1c0 con 0x7f2ef80cff70 2021-02-10T11:37:02.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.125 7f2efe647700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ef8109a60 con 0x7f2ef81b0ae0 2021-02-10T11:37:02.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.125 7f2efe647700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ef8088070 con 0x7f2ef80d1870 2021-02-10T11:37:02.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.125 7f2efcc64700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f2ef80cff70 0x7f2ef8109130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:52062/0 (socket says 172.21.2.112:52062) 2021-02-10T11:37:02.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.125 7f2efcc64700 1 -- 172.21.2.112:0/3214201752 learned_addr learned my addr 172.21.2.112:0/3214201752 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:37:02.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.125 7f2ef6ffd700 1 -- 172.21.2.112:0/3214201752 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1172424225 0 0) 0x7f2ef0000d30 con 0x7f2ef80cff70 2021-02-10T11:37:02.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.125 7f2ef6ffd700 1 -- 172.21.2.112:0/3214201752 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ee0002fe0 con 0x7f2ef80cff70 2021-02-10T11:37:02.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.125 7f2ef7fff700 1 --2- 172.21.2.112:0/3214201752 >> v2:172.21.2.120:3300/0 conn(0x7f2ef80d1870 0x7f2ef81b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:02.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.125 7f2ef6ffd700 1 -- 172.21.2.112:0/3214201752 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3274826019 0 0) 0x7f2ef0001270 con 0x7f2ef80cff70 2021-02-10T11:37:02.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.125 7f2ef6ffd700 1 -- 172.21.2.112:0/3214201752 >> v2:172.21.2.120:3300/0 conn(0x7f2ef80d1870 msgr2=0x7f2ef81b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:37:02.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.125 7f2ef6ffd700 1 --2- 172.21.2.112:0/3214201752 >> v2:172.21.2.120:3300/0 conn(0x7f2ef80d1870 0x7f2ef81b04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:02.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.125 7f2ef6ffd700 1 -- 172.21.2.112:0/3214201752 >> v1:172.21.2.120:6789/0 conn(0x7f2ef81b0ae0 legacy=0x7f2ef8108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:02.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.125 7f2ef6ffd700 1 -- 172.21.2.112:0/3214201752 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ef8088260 con 0x7f2ef80cff70 2021-02-10T11:37:02.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.125 7f2ef6ffd700 1 -- 172.21.2.112:0/3214201752 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f2ef0001a00 con 0x7f2ef80cff70 2021-02-10T11:37:02.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.125 7f2ef7fff700 1 --2- 172.21.2.112:0/3214201752 >> v2:172.21.2.120:3300/0 conn(0x7f2ef80d1870 0x7f2ef81b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:37:02.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.125 7f2ef6ffd700 1 -- 172.21.2.112:0/3214201752 >> v1:172.21.2.112:6789/0 conn(0x7f2ef80cff70 legacy=0x7f2ef8109130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:02.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.125 7f2ef6ffd700 1 --2- 172.21.2.112:0/3214201752 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2ee0003a70 0x7f2ee0005f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:02.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.125 7f2ef6ffd700 1 --2- 172.21.2.112:0/3214201752 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2ee00067a0 0x7f2ee0008ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:02.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.125 7f2ef6ffd700 1 --2- 172.21.2.112:0/3214201752 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2ee00095a0 0x7f2ee000bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:02.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.125 7f2ef6ffd700 1 -- 172.21.2.112:0/3214201752 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ee000c310 con 0x7f2ee00095a0 2021-02-10T11:37:02.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.125 7f2ef6ffd700 1 -- 172.21.2.112:0/3214201752 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ee000c530 con 0x7f2ee0003a70 2021-02-10T11:37:02.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.125 7f2ef6ffd700 1 -- 172.21.2.112:0/3214201752 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ee000c750 con 0x7f2ee00067a0 2021-02-10T11:37:02.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.126 7f2efcc64700 1 --2- 172.21.2.112:0/3214201752 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2ee00095a0 0x7f2ee000bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:02.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.126 7f2ef7fff700 1 --2- 172.21.2.112:0/3214201752 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2ee0003a70 0x7f2ee0005f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:02.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.126 7f2ef77fe700 1 --2- 172.21.2.112:0/3214201752 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2ee00067a0 0x7f2ee0008ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:02.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.126 7f2efcc64700 1 -- 172.21.2.112:0/3214201752 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2ee00067a0 msgr2=0x7f2ee0008ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:37:02.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.126 7f2efcc64700 1 --2- 172.21.2.112:0/3214201752 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2ee00067a0 0x7f2ee0008ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:02.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.126 7f2efcc64700 1 -- 172.21.2.112:0/3214201752 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2ee0003a70 msgr2=0x7f2ee0005f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:37:02.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.126 7f2efcc64700 1 --2- 172.21.2.112:0/3214201752 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2ee0003a70 0x7f2ee0005f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:02.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.126 7f2efcc64700 1 -- 172.21.2.112:0/3214201752 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ee0050a00 con 0x7f2ee00095a0 2021-02-10T11:37:02.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.126 7f2efcc64700 1 -- 172.21.2.112:0/3214201752 reap_dead start 2021-02-10T11:37:02.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.126 7f2efcc64700 1 --2- 172.21.2.112:0/3214201752 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2ee00095a0 0x7f2ee000bac0 secure :-1 s=READY pgs=1720 cs=0 l=1 rev1=1 rx=0x7f2ef0001840 tx=0x7f2ef000a310).ready entity=mon.0 client_cookie=12c6803500fc833b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:02.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.126 7f2ef6ffd700 1 -- 172.21.2.112:0/3214201752 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2ef0007340 con 0x7f2ee00095a0 2021-02-10T11:37:02.131 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.126 7f2ef6ffd700 1 -- 172.21.2.112:0/3214201752 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ef0004720 con 0x7f2ee00095a0 2021-02-10T11:37:02.131 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.126 7f2ef6ffd700 1 -- 172.21.2.112:0/3214201752 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2ef0004b70 con 0x7f2ee00095a0 2021-02-10T11:37:02.131 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.126 7f2efe647700 1 -- 172.21.2.112:0/3214201752 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2ee00095a0 msgr2=0x7f2ee000bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:02.131 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.126 7f2efe647700 1 --2- 172.21.2.112:0/3214201752 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2ee00095a0 0x7f2ee000bac0 secure :-1 s=READY pgs=1720 cs=0 l=1 rev1=1 rx=0x7f2ef0001840 tx=0x7f2ef000a310).stop 2021-02-10T11:37:02.131 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.127 7f2efe647700 1 -- 172.21.2.112:0/3214201752 shutdown_connections 2021-02-10T11:37:02.131 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.127 7f2efe647700 1 --2- 172.21.2.112:0/3214201752 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2ee00095a0 0x7f2ee000bac0 unknown :-1 s=CLOSED pgs=1720 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:02.131 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.127 7f2efe647700 1 -- 172.21.2.112:0/3214201752 >> 172.21.2.112:0/3214201752 conn(0x7f2ef80ce420 msgr2=0x7f2ef80d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:02.131 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.127 7f2efe647700 1 -- 172.21.2.112:0/3214201752 shutdown_connections 2021-02-10T11:37:02.131 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.127 7f2efe647700 1 -- 172.21.2.112:0/3214201752 wait complete. 2021-02-10T11:37:02.133 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.141 7f2efe647700 1 Processor -- start 2021-02-10T11:37:02.133 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.141 7f2efe647700 1 -- start start 2021-02-10T11:37:02.133 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.142 7f2efe647700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2ef8085bc0 0x7f2ef80863e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:02.133 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.142 7f2efe647700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2ef80954c0 0x7f2ef8096300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:02.133 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.142 7f2efe647700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2ef80958c0 0x7f2ef8096920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:02.133 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.142 7f2efe647700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ef8095d80 con 0x7f2ef8085bc0 2021-02-10T11:37:02.133 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.142 7f2efe647700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ef80bfa60 con 0x7f2ef80954c0 2021-02-10T11:37:02.133 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.142 7f2efe647700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ef8085fc0 con 0x7f2ef80958c0 2021-02-10T11:37:02.133 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.142 7f2ef7fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2ef8085bc0 0x7f2ef80863e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:02.133 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.142 7f2ef7fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2ef8085bc0 0x7f2ef80863e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:52614/0 (socket says 172.21.2.112:52614) 2021-02-10T11:37:02.133 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.142 7f2ef7fff700 1 -- 172.21.2.112:0/1854882698 learned_addr learned my addr 172.21.2.112:0/1854882698 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:37:02.134 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.142 7f2ef77fe700 1 --2- 172.21.2.112:0/1854882698 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2ef80954c0 0x7f2ef8096300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:02.134 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.142 7f2efcc64700 1 --2- 172.21.2.112:0/1854882698 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2ef80958c0 0x7f2ef8096920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:02.134 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.142 7f2ef7fff700 1 -- 172.21.2.112:0/1854882698 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2ef80958c0 msgr2=0x7f2ef8096920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:02.134 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.142 7f2ef7fff700 1 --2- 172.21.2.112:0/1854882698 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2ef80958c0 0x7f2ef8096920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:02.134 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.142 7f2ef7fff700 1 -- 172.21.2.112:0/1854882698 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2ef80954c0 msgr2=0x7f2ef8096300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:02.134 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.142 7f2ef7fff700 1 --2- 172.21.2.112:0/1854882698 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2ef80954c0 0x7f2ef8096300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:02.134 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.142 7f2ef7fff700 1 -- 172.21.2.112:0/1854882698 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ef8086c10 con 0x7f2ef8085bc0 2021-02-10T11:37:02.134 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.142 7f2efcc64700 1 --2- 172.21.2.112:0/1854882698 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2ef80958c0 0x7f2ef8096920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:37:02.134 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.143 7f2ef7fff700 1 --2- 172.21.2.112:0/1854882698 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2ef8085bc0 0x7f2ef80863e0 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 rx=0x7f2ee8006f20 tx=0x7f2ee800b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:02.134 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.143 7f2ef4ff9700 1 -- 172.21.2.112:0/1854882698 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2ee800bd80 con 0x7f2ef8085bc0 2021-02-10T11:37:02.134 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.143 7f2ef4ff9700 1 -- 172.21.2.112:0/1854882698 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ee8000c90 con 0x7f2ef8085bc0 2021-02-10T11:37:02.134 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.143 7f2ef4ff9700 1 -- 172.21.2.112:0/1854882698 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2ee80022b0 con 0x7f2ef8085bc0 2021-02-10T11:37:02.134 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.143 7f2efe647700 1 -- 172.21.2.112:0/1854882698 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ef8086c10 con 0x7f2ef8085bc0 2021-02-10T11:37:02.135 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.143 7f2efe647700 1 -- 172.21.2.112:0/1854882698 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ef8097230 con 0x7f2ef8085bc0 2021-02-10T11:37:02.135 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.143 7f2ef4ff9700 1 -- 172.21.2.112:0/1854882698 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2ee800a000 con 0x7f2ef8085bc0 2021-02-10T11:37:02.135 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.144 7f2ef4ff9700 1 --2- 172.21.2.112:0/1854882698 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2ed80186c0 0x7f2ed801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:02.135 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.144 7f2ef4ff9700 1 -- 172.21.2.112:0/1854882698 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f2ee8032530 con 0x7f2ef8085bc0 2021-02-10T11:37:02.135 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.144 7f2ef77fe700 1 --2- 172.21.2.112:0/1854882698 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2ed80186c0 0x7f2ed801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:02.135 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.144 7f2ef77fe700 1 --2- 172.21.2.112:0/1854882698 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2ed80186c0 0x7f2ed801abc0 crc :-1 s=READY pgs=816 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:02.161 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.174 7f2efe647700 1 -- 172.21.2.112:0/1854882698 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ef80fe1c0 con 0x7f2ef8085bc0 2021-02-10T11:37:02.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.177 7f2ef4ff9700 1 -- 172.21.2.112:0/1854882698 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2ee80009b0 con 0x7f2ef8085bc0 2021-02-10T11:37:02.291 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.304 7f2efe647700 1 -- 172.21.2.112:0/1854882698 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f2ef80fe1c0 con 0x7f2ef8085bc0 2021-02-10T11:37:02.292 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.305 7f2ef4ff9700 1 -- 172.21.2.112:0/1854882698 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v83) v1 ==== 104+0+10 (secure 0 0 0) 0x7f2ee800a000 con 0x7f2ef8085bc0 2021-02-10T11:37:02.292 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:37:02.293 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.306 7f2efe647700 1 -- 172.21.2.112:0/1854882698 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2ed80186c0 msgr2=0x7f2ed801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:02.293 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.306 7f2efe647700 1 --2- 172.21.2.112:0/1854882698 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2ed80186c0 0x7f2ed801abc0 crc :-1 s=READY pgs=816 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:02.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.306 7f2efe647700 1 -- 172.21.2.112:0/1854882698 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2ef8085bc0 msgr2=0x7f2ef80863e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:02.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.306 7f2efe647700 1 --2- 172.21.2.112:0/1854882698 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2ef8085bc0 0x7f2ef80863e0 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 rx=0x7f2ee8006f20 tx=0x7f2ee800b990).stop 2021-02-10T11:37:02.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.307 7f2efe647700 1 -- 172.21.2.112:0/1854882698 shutdown_connections 2021-02-10T11:37:02.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.307 7f2efe647700 1 --2- 172.21.2.112:0/1854882698 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2ed80186c0 0x7f2ed801abc0 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:02.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.307 7f2efe647700 1 --2- 172.21.2.112:0/1854882698 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2ef80954c0 0x7f2ef8096300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:02.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.307 7f2efe647700 1 --2- 172.21.2.112:0/1854882698 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2ef80958c0 0x7f2ef8096920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:02.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.307 7f2efe647700 1 --2- 172.21.2.112:0/1854882698 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2ef8085bc0 0x7f2ef80863e0 unknown :-1 s=CLOSED pgs=1721 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:02.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.307 7f2efe647700 1 -- 172.21.2.112:0/1854882698 >> 172.21.2.112:0/1854882698 conn(0x7f2ef80bef00 msgr2=0x7f2ef809d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:02.295 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.307 7f2efe647700 1 -- 172.21.2.112:0/1854882698 shutdown_connections 2021-02-10T11:37:02.295 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.307 7f2efe647700 1 -- 172.21.2.112:0/1854882698 wait complete. 2021-02-10T11:37:02.302 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:37:02.733 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.729 7ffacfa17700 1 Processor -- start 2021-02-10T11:37:02.733 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.729 7ffacfa17700 1 -- start start 2021-02-10T11:37:02.733 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.730 7ffacfa17700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7ffac80d1c70 0x7ffac80cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:02.733 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.730 7ffacfa17700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ffac80cf580 0x7ffac80cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:02.734 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.730 7ffacfa17700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffac80fe1c0 con 0x7ffac80d1870 2021-02-10T11:37:02.734 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.730 7ffacfa17700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffac81b0cb0 con 0x7ffac80d1c70 2021-02-10T11:37:02.734 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.730 7ffacfa17700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffac8109a60 con 0x7ffac80cf580 2021-02-10T11:37:02.734 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.730 7ffacd032700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7ffac80d1c70 0x7ffac80cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:02.734 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.730 7ffacd032700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7ffac80d1c70 0x7ffac80cef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:52620/0 (socket says 172.21.2.112:52620) 2021-02-10T11:37:02.734 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.730 7ffacd032700 1 -- 172.21.2.112:0/3608404447 learned_addr learned my addr 172.21.2.112:0/3608404447 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:37:02.734 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.730 7ffacd032700 1 -- 172.21.2.112:0/3608404447 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ffac80cf580 msgr2=0x7ffac80cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:37:02.734 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.730 7ffacd032700 1 --2- 172.21.2.112:0/3608404447 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ffac80cf580 0x7ffac80cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:02.734 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.730 7ffacd032700 1 -- 172.21.2.112:0/3608404447 >> v1:172.21.2.112:6789/0 conn(0x7ffac80d1870 legacy=0x7ffac81b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:02.734 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.730 7ffacd032700 1 -- 172.21.2.112:0/3608404447 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffac8088070 con 0x7ffac80d1c70 2021-02-10T11:37:02.735 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.731 7ffacd032700 1 --2- 172.21.2.112:0/3608404447 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ffac80d1c70 0x7ffac80cef60 secure :-1 s=READY pgs=1723 cs=0 l=1 rev1=1 rx=0x7ffabc006700 tx=0x7ffabc009b80).ready entity=mon.0 client_cookie=9c0a329ad2da5d68 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:02.735 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.731 7ffacc831700 1 -- 172.21.2.112:0/3608404447 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffabc00c0c0 con 0x7ffac80d1c70 2021-02-10T11:37:02.735 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.731 7ffacc831700 1 -- 172.21.2.112:0/3608404447 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffabc00c440 con 0x7ffac80d1c70 2021-02-10T11:37:02.735 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.731 7ffacc831700 1 -- 172.21.2.112:0/3608404447 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffabc00c960 con 0x7ffac80d1c70 2021-02-10T11:37:02.735 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.731 7ffacfa17700 1 -- 172.21.2.112:0/3608404447 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ffac80d1c70 msgr2=0x7ffac80cef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:02.735 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.731 7ffacfa17700 1 --2- 172.21.2.112:0/3608404447 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ffac80d1c70 0x7ffac80cef60 secure :-1 s=READY pgs=1723 cs=0 l=1 rev1=1 rx=0x7ffabc006700 tx=0x7ffabc009b80).stop 2021-02-10T11:37:02.735 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.731 7ffacfa17700 1 -- 172.21.2.112:0/3608404447 shutdown_connections 2021-02-10T11:37:02.735 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.731 7ffacfa17700 1 --2- 172.21.2.112:0/3608404447 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ffac80cf580 0x7ffac80cf980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:02.735 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.731 7ffacfa17700 1 --2- 172.21.2.112:0/3608404447 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ffac80d1c70 0x7ffac80cef60 unknown :-1 s=CLOSED pgs=1723 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:02.735 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.731 7ffacfa17700 1 -- 172.21.2.112:0/3608404447 >> 172.21.2.112:0/3608404447 conn(0x7ffac80ce420 msgr2=0x7ffac80d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:02.735 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.732 7ffacfa17700 1 -- 172.21.2.112:0/3608404447 shutdown_connections 2021-02-10T11:37:02.735 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.732 7ffacfa17700 1 -- 172.21.2.112:0/3608404447 wait complete. 2021-02-10T11:37:02.735 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.746 7ffacfa17700 1 Processor -- start 2021-02-10T11:37:02.736 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.746 7ffacfa17700 1 -- start start 2021-02-10T11:37:02.737 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.747 7ffacfa17700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ffac80c3820 0x7ffac8084cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:02.737 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.747 7ffacfa17700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ffac80a0290 0x7ffac80852d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:02.737 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.747 7ffacfa17700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ffac8086270 0x7ffac8086670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:02.737 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.747 7ffacfa17700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffac80c0700 con 0x7ffac8086270 2021-02-10T11:37:02.737 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.747 7ffacfa17700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffac80858f0 con 0x7ffac80a0290 2021-02-10T11:37:02.737 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.747 7ffacfa17700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffac8085b10 con 0x7ffac80c3820 2021-02-10T11:37:02.737 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.747 7fface034700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ffac8086270 0x7ffac8086670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:02.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.747 7fface034700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ffac8086270 0x7ffac8086670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:52624/0 (socket says 172.21.2.112:52624) 2021-02-10T11:37:02.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.747 7fface034700 1 -- 172.21.2.112:0/2591907945 learned_addr learned my addr 172.21.2.112:0/2591907945 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:37:02.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.747 7ffacd032700 1 --2- 172.21.2.112:0/2591907945 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ffac80a0290 0x7ffac80852d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:02.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.747 7ffacd833700 1 --2- 172.21.2.112:0/2591907945 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ffac80c3820 0x7ffac8084cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:02.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.747 7fface034700 1 -- 172.21.2.112:0/2591907945 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ffac80c3820 msgr2=0x7ffac8084cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:02.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.747 7fface034700 1 --2- 172.21.2.112:0/2591907945 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ffac80c3820 0x7ffac8084cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:02.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.747 7fface034700 1 -- 172.21.2.112:0/2591907945 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ffac80a0290 msgr2=0x7ffac80852d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:02.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.747 7fface034700 1 --2- 172.21.2.112:0/2591907945 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ffac80a0290 0x7ffac80852d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:02.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.747 7fface034700 1 -- 172.21.2.112:0/2591907945 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffac8086c90 con 0x7ffac8086270 2021-02-10T11:37:02.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.747 7fface034700 1 --2- 172.21.2.112:0/2591907945 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ffac8086270 0x7ffac8086670 secure :-1 s=READY pgs=1724 cs=0 l=1 rev1=1 rx=0x7ffac0009980 tx=0x7ffac000bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:02.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.748 7ffaba7fc700 1 -- 172.21.2.112:0/2591907945 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffac0005870 con 0x7ffac8086270 2021-02-10T11:37:02.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.748 7ffaba7fc700 1 -- 172.21.2.112:0/2591907945 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffac0005af0 con 0x7ffac8086270 2021-02-10T11:37:02.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.748 7ffaba7fc700 1 -- 172.21.2.112:0/2591907945 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffac0006030 con 0x7ffac8086270 2021-02-10T11:37:02.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.748 7ffacfa17700 1 -- 172.21.2.112:0/2591907945 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffac8086c90 con 0x7ffac8086270 2021-02-10T11:37:02.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.748 7ffacfa17700 1 -- 172.21.2.112:0/2591907945 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffac8086c90 con 0x7ffac8086270 2021-02-10T11:37:02.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.748 7ffaba7fc700 1 -- 172.21.2.112:0/2591907945 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ffac0005cd0 con 0x7ffac8086270 2021-02-10T11:37:02.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.748 7ffaba7fc700 1 --2- 172.21.2.112:0/2591907945 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ffab00186c0 0x7ffab001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:02.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.748 7ffaba7fc700 1 -- 172.21.2.112:0/2591907945 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5573+0+0 (secure 0 0 0) 0x7ffac0032920 con 0x7ffac8086270 2021-02-10T11:37:02.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.749 7ffacd833700 1 --2- 172.21.2.112:0/2591907945 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ffab00186c0 0x7ffab001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:02.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.749 7ffacd833700 1 --2- 172.21.2.112:0/2591907945 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ffab00186c0 0x7ffab001abc0 crc :-1 s=READY pgs=817 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:02.767 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.780 7ffacfa17700 1 -- 172.21.2.112:0/2591907945 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffac8082a70 con 0x7ffac8086270 2021-02-10T11:37:02.769 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.783 7ffaba7fc700 1 -- 172.21.2.112:0/2591907945 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ffac000be10 con 0x7ffac8086270 2021-02-10T11:37:02.900 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.913 7ffacfa17700 1 -- 172.21.2.112:0/2591907945 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7ffac8082a70 con 0x7ffac8086270 2021-02-10T11:37:02.900 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.913 7ffaba7fc700 1 -- 172.21.2.112:0/2591907945 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v83) v1 ==== 100+0+10 (secure 0 0 0) 0x7ffac0006f20 con 0x7ffac8086270 2021-02-10T11:37:02.900 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:37:02.902 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.915 7ffacfa17700 1 -- 172.21.2.112:0/2591907945 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ffab00186c0 msgr2=0x7ffab001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:02.902 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.915 7ffacfa17700 1 --2- 172.21.2.112:0/2591907945 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ffab00186c0 0x7ffab001abc0 crc :-1 s=READY pgs=817 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:02.902 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.915 7ffacfa17700 1 -- 172.21.2.112:0/2591907945 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ffac8086270 msgr2=0x7ffac8086670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:02.902 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.915 7ffacfa17700 1 --2- 172.21.2.112:0/2591907945 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ffac8086270 0x7ffac8086670 secure :-1 s=READY pgs=1724 cs=0 l=1 rev1=1 rx=0x7ffac0009980 tx=0x7ffac000bcd0).stop 2021-02-10T11:37:02.902 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.915 7ffacfa17700 1 -- 172.21.2.112:0/2591907945 shutdown_connections 2021-02-10T11:37:02.902 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.915 7ffacfa17700 1 --2- 172.21.2.112:0/2591907945 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ffab00186c0 0x7ffab001abc0 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:02.902 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.915 7ffacfa17700 1 --2- 172.21.2.112:0/2591907945 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ffac80a0290 0x7ffac80852d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:02.903 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.915 7ffacfa17700 1 --2- 172.21.2.112:0/2591907945 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ffac8086270 0x7ffac8086670 unknown :-1 s=CLOSED pgs=1724 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:02.903 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.915 7ffacfa17700 1 --2- 172.21.2.112:0/2591907945 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ffac80c3820 0x7ffac8084cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:02.903 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.915 7ffacfa17700 1 -- 172.21.2.112:0/2591907945 >> 172.21.2.112:0/2591907945 conn(0x7ffac80872c0 msgr2=0x7ffac8095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:02.903 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.915 7ffacfa17700 1 -- 172.21.2.112:0/2591907945 shutdown_connections 2021-02-10T11:37:02.903 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:02.915 7ffacfa17700 1 -- 172.21.2.112:0/2591907945 wait complete. 2021-02-10T11:37:02.911 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:37:02.911 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:37:03.339 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:03.335 7f2381f9e700 1 Processor -- start 2021-02-10T11:37:03.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:03.335 7f2381f9e700 1 -- start start 2021-02-10T11:37:03.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:03.335 7f2381f9e700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f237c0d1870 0x7f237c1b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:03.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:03.335 7f2381f9e700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f237c1b0a10 0x7f237c0cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:03.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:03.335 7f2381f9e700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f237c0d1c70 0x7f237c0cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:03.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:03.335 7f2381f9e700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f237c0fe230 con 0x7f237c1b0a10 2021-02-10T11:37:03.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:03.335 7f2381f9e700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f237c0e5ed0 con 0x7f237c0d1c70 2021-02-10T11:37:03.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:03.335 7f2381f9e700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f237c0f0b10 con 0x7f237c0d1870 2021-02-10T11:37:03.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:03.335 7f237affd700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f237c1b0a10 0x7f237c0cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:03.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:03.335 7f237affd700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f237c1b0a10 0x7f237c0cf330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:52632/0 (socket says 172.21.2.112:52632) 2021-02-10T11:37:03.342 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:03.335 7f237affd700 1 -- 172.21.2.112:0/1595679280 learned_addr learned my addr 172.21.2.112:0/1595679280 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:37:03.342 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:03.335 7f237bfff700 1 --2- 172.21.2.112:0/1595679280 >> v2:172.21.2.120:3300/0 conn(0x7f237c0d1c70 0x7f237c0cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:03.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:03.335 7f237b7fe700 1 --2- 172.21.2.112:0/1595679280 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f237c0d1870 0x7f237c1b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:03.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:03.336 7f237affd700 1 -- 172.21.2.112:0/1595679280 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f237c0d1870 msgr2=0x7f237c1b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:37:03.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:03.336 7f237affd700 1 --2- 172.21.2.112:0/1595679280 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f237c0d1870 0x7f237c1b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:03.344 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:03.336 7f237affd700 1 -- 172.21.2.112:0/1595679280 >> v2:172.21.2.120:3300/0 conn(0x7f237c0d1c70 msgr2=0x7f237c0cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:37:03.346 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:03.336 7f237affd700 1 --2- 172.21.2.112:0/1595679280 >> v2:172.21.2.120:3300/0 conn(0x7f237c0d1c70 0x7f237c0cf950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:03.346 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:03.336 7f237affd700 1 -- 172.21.2.112:0/1595679280 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f237c0cff70 con 0x7f237c1b0a10 2021-02-10T11:37:03.346 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:03.336 7f237b7fe700 1 --2- 172.21.2.112:0/1595679280 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f237c0d1870 0x7f237c1b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:37:03.347 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:03.336 7f237affd700 1 --2- 172.21.2.112:0/1595679280 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f237c1b0a10 0x7f237c0cf330 secure :-1 s=READY pgs=1725 cs=0 l=1 rev1=1 rx=0x7f236c006700 tx=0x7f236c009b80).ready entity=mon.0 client_cookie=63100bd6d8b3dd5c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:03.347 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:03.336 7f237a7fc700 1 -- 172.21.2.112:0/1595679280 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f236c00c0c0 con 0x7f237c1b0a10 2021-02-10T11:37:03.347 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:03.336 7f237a7fc700 1 -- 172.21.2.112:0/1595679280 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f236c00c440 con 0x7f237c1b0a10 2021-02-10T11:37:03.348 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:03.336 7f237a7fc700 1 -- 172.21.2.112:0/1595679280 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f236c00c960 con 0x7f237c1b0a10 2021-02-10T11:37:03.349 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:03.336 7f2381f9e700 1 -- 172.21.2.112:0/1595679280 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f237c1b0a10 msgr2=0x7f237c0cf330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:03.349 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:03.336 7f2381f9e700 1 --2- 172.21.2.112:0/1595679280 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f237c1b0a10 0x7f237c0cf330 secure :-1 s=READY pgs=1725 cs=0 l=1 rev1=1 rx=0x7f236c006700 tx=0x7f236c009b80).stop 2021-02-10T11:37:03.349 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:03.337 7f2381f9e700 1 -- 172.21.2.112:0/1595679280 shutdown_connections 2021-02-10T11:37:03.349 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:03.337 7f2381f9e700 1 --2- 172.21.2.112:0/1595679280 >> v2:172.21.2.120:3300/0 conn(0x7f237c0d1c70 0x7f237c0cf950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:03.350 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:03.337 7f2381f9e700 1 --2- 172.21.2.112:0/1595679280 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f237c1b0a10 0x7f237c0cf330 unknown :-1 s=CLOSED pgs=1725 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:03.350 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:03.337 7f2381f9e700 1 --2- 172.21.2.112:0/1595679280 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f237c0d1870 0x7f237c1b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:03.350 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:03.337 7f2381f9e700 1 -- 172.21.2.112:0/1595679280 >> 172.21.2.112:0/1595679280 conn(0x7f237c0ce450 msgr2=0x7f237c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:03.350 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:03.337 7f2381f9e700 1 -- 172.21.2.112:0/1595679280 shutdown_connections 2021-02-10T11:37:03.350 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:03.337 7f2381f9e700 1 -- 172.21.2.112:0/1595679280 wait complete. 2021-02-10T11:37:03.356 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:03.351 7f2381f9e700 1 Processor -- start 2021-02-10T11:37:03.356 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:03.351 7f2381f9e700 1 -- start start 2021-02-10T11:37:03.357 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:03.352 7f2381f9e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f237c1942b0 0x7f237c0b3550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:03.358 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:03.352 7f2381f9e700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f237c195070 0x7f237c195470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:03.358 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:03.352 7f2381f9e700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f237c0bb8e0 0x7f237c0bbce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:03.358 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:03.352 7f2381f9e700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f237c1946b0 con 0x7f237c1942b0 2021-02-10T11:37:03.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:03.352 7f2381f9e700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f237c0b3b70 con 0x7f237c0bb8e0 2021-02-10T11:37:03.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:03.352 7f2381f9e700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f237c0b3d90 con 0x7f237c195070 2021-02-10T11:37:03.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:03.352 7f237b7fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f237c1942b0 0x7f237c0b3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:03.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:03.352 7f237b7fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f237c1942b0 0x7f237c0b3550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:52636/0 (socket says 172.21.2.112:52636) 2021-02-10T11:37:03.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:03.352 7f237b7fe700 1 -- 172.21.2.112:0/418977033 learned_addr learned my addr 172.21.2.112:0/418977033 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:37:03.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:03.352 7f237affd700 1 --2- 172.21.2.112:0/418977033 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f237c195070 0x7f237c195470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:03.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:03.353 7f237bfff700 1 --2- 172.21.2.112:0/418977033 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f237c0bb8e0 0x7f237c0bbce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:03.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:03.353 7f237b7fe700 1 -- 172.21.2.112:0/418977033 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f237c195070 msgr2=0x7f237c195470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:03.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:03.353 7f237b7fe700 1 --2- 172.21.2.112:0/418977033 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f237c195070 0x7f237c195470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:03.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:03.353 7f237b7fe700 1 -- 172.21.2.112:0/418977033 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f237c0bb8e0 msgr2=0x7f237c0bbce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:03.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:03.353 7f237b7fe700 1 --2- 172.21.2.112:0/418977033 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f237c0bb8e0 0x7f237c0bbce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:03.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:03.353 7f237b7fe700 1 -- 172.21.2.112:0/418977033 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f237c1a49e0 con 0x7f237c1942b0 2021-02-10T11:37:03.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:03.353 7f237b7fe700 1 --2- 172.21.2.112:0/418977033 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f237c1942b0 0x7f237c0b3550 secure :-1 s=READY pgs=1726 cs=0 l=1 rev1=1 rx=0x7f2364006f20 tx=0x7f236400b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:03.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:03.353 7f2373fff700 1 -- 172.21.2.112:0/418977033 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f236400bd80 con 0x7f237c1942b0 2021-02-10T11:37:03.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:03.353 7f2373fff700 1 -- 172.21.2.112:0/418977033 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2364000c90 con 0x7f237c1942b0 2021-02-10T11:37:03.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:03.353 7f2373fff700 1 -- 172.21.2.112:0/418977033 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f23640022b0 con 0x7f237c1942b0 2021-02-10T11:37:03.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:03.353 7f2381f9e700 1 -- 172.21.2.112:0/418977033 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f237c1a49e0 con 0x7f237c1942b0 2021-02-10T11:37:03.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:03.353 7f2381f9e700 1 -- 172.21.2.112:0/418977033 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f237c0bc5f0 con 0x7f237c1942b0 2021-02-10T11:37:03.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:03.354 7f2373fff700 1 -- 172.21.2.112:0/418977033 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f236400a000 con 0x7f237c1942b0 2021-02-10T11:37:03.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:03.354 7f2373fff700 1 --2- 172.21.2.112:0/418977033 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f23600186c0 0x7f236001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:03.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:03.354 7f2373fff700 1 -- 172.21.2.112:0/418977033 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f23640323f0 con 0x7f237c1942b0 2021-02-10T11:37:03.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:03.354 7f237affd700 1 --2- 172.21.2.112:0/418977033 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f23600186c0 0x7f236001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:03.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:03.355 7f237affd700 1 --2- 172.21.2.112:0/418977033 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f23600186c0 0x7f236001abc0 crc :-1 s=READY pgs=818 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:03.372 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:03.385 7f2381f9e700 1 -- 172.21.2.112:0/418977033 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f237c0bc5f0 con 0x7f237c1942b0 2021-02-10T11:37:03.376 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:03.388 7f2373fff700 1 -- 172.21.2.112:0/418977033 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f236402c360 con 0x7f237c1942b0 2021-02-10T11:37:03.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:03.515 7f2381f9e700 1 -- 172.21.2.112:0/418977033 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f237c0bc5f0 con 0x7f237c1942b0 2021-02-10T11:37:03.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:03.515 7f2373fff700 1 -- 172.21.2.112:0/418977033 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) v1 ==== 94+0+2101 (secure 0 0 0) 0x7f236402cef0 con 0x7f237c1942b0 2021-02-10T11:37:03.503 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 64 2021-02-10T11:37:03.503 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:37:03.503 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":64,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9204,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/1512510827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":1512510827},{"type":"v1","addr":"172.21.2.112:6833","nonce":1512510827}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":60}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:36:53.563735","modified":"2021-02-10 11:36:55.573740","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9201},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9201":{"gid":9201,"name":"b","rank":0,"incarnation":61,"state":"up:active","state_seq":3,"addr":"172.21.2.120:6835/1232001750","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":1232001750},{"type":"v1","addr":"172.21.2.120:6835","nonce":1232001750}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2021-02-10T11:37:03.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:03.517 7f2381f9e700 1 -- 172.21.2.112:0/418977033 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f23600186c0 msgr2=0x7f236001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:03.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:03.517 7f2381f9e700 1 --2- 172.21.2.112:0/418977033 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f23600186c0 0x7f236001abc0 crc :-1 s=READY pgs=818 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:03.505 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:03.517 7f2381f9e700 1 -- 172.21.2.112:0/418977033 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f237c1942b0 msgr2=0x7f237c0b3550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:03.505 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:03.517 7f2381f9e700 1 --2- 172.21.2.112:0/418977033 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f237c1942b0 0x7f237c0b3550 secure :-1 s=READY pgs=1726 cs=0 l=1 rev1=1 rx=0x7f2364006f20 tx=0x7f236400b990).stop 2021-02-10T11:37:03.505 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:03.518 7f2381f9e700 1 -- 172.21.2.112:0/418977033 shutdown_connections 2021-02-10T11:37:03.505 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:03.518 7f2381f9e700 1 --2- 172.21.2.112:0/418977033 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f23600186c0 0x7f236001abc0 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:03.505 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:03.518 7f2381f9e700 1 --2- 172.21.2.112:0/418977033 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f237c0bb8e0 0x7f237c0bbce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:03.505 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:03.518 7f2381f9e700 1 --2- 172.21.2.112:0/418977033 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f237c195070 0x7f237c195470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:03.505 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:03.518 7f2381f9e700 1 --2- 172.21.2.112:0/418977033 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f237c1942b0 0x7f237c0b3550 unknown :-1 s=CLOSED pgs=1726 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:03.505 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:03.518 7f2381f9e700 1 -- 172.21.2.112:0/418977033 >> 172.21.2.112:0/418977033 conn(0x7f237c0b3040 msgr2=0x7f237c0c3250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:03.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:03.518 7f2381f9e700 1 -- 172.21.2.112:0/418977033 shutdown_connections 2021-02-10T11:37:03.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:03.518 7f2381f9e700 1 -- 172.21.2.112:0/418977033 wait complete. 2021-02-10T11:37:03.715 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 64, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:36:53.563735', 'modified': '2021-02-10 11:36:55.573740', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9201}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9201': {'gid': 9201, 'name': 'b', 'rank': 0, 'incarnation': 61, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.120:6835/1232001750', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.120:6834', 'nonce': 1232001750}, {'type': 'v1', 'addr': '172.21.2.120:6835', 'nonce': 1232001750}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:37:03.715 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:37:03.715 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:37:04.117 INFO:teuthology.orchestra.run.gibba020.stdout:{ 2021-02-10T11:37:04.118 INFO:teuthology.orchestra.run.gibba020.stdout: "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", 2021-02-10T11:37:04.118 INFO:teuthology.orchestra.run.gibba020.stdout: "whoami": 0, 2021-02-10T11:37:04.118 INFO:teuthology.orchestra.run.gibba020.stdout: "id": 9201, 2021-02-10T11:37:04.118 INFO:teuthology.orchestra.run.gibba020.stdout: "want_state": "up:active", 2021-02-10T11:37:04.118 INFO:teuthology.orchestra.run.gibba020.stdout: "state": "up:active", 2021-02-10T11:37:04.118 INFO:teuthology.orchestra.run.gibba020.stdout: "rank_uptime": 10.565305788, 2021-02-10T11:37:04.118 INFO:teuthology.orchestra.run.gibba020.stdout: "mdsmap_epoch": 64, 2021-02-10T11:37:04.118 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch": 83, 2021-02-10T11:37:04.118 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch_barrier": 83, 2021-02-10T11:37:04.118 INFO:teuthology.orchestra.run.gibba020.stdout: "uptime": 16.203075739999999 2021-02-10T11:37:04.118 INFO:teuthology.orchestra.run.gibba020.stdout:} 2021-02-10T11:37:04.121 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", "whoami": 0, "id": 9201, "want_state": "up:active", "state": "up:active", "rank_uptime": 10.565305788, "mdsmap_epoch": 64, "osdmap_epoch": 83, "osdmap_epoch_barrier": 83, "uptime": 16.203075739999999 } 2021-02-10T11:37:04.122 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:37:04.122 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2021-02-10T11:37:04.122 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@gibba012.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2021-02-10T11:37:04.122 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:04.122 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:37:04.457 DEBUG:teuthology.orchestra.run.gibba012:> sudo modprobe fuse 2021-02-10T11:37:04.800 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:37:04.801 DEBUG:teuthology.orchestra.run.gibba012:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:37:05.144 INFO:teuthology.orchestra.run.gibba012.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:37:05.144 INFO:teuthology.orchestra.run.gibba012.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:37:05.145 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:37:05.145 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:05.145 DEBUG:teuthology.orchestra.run.gibba012:> ls /sys/fs/fuse/connections 2021-02-10T11:37:05.480 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2021-02-10T11:37:05.480 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:37:05.483 DEBUG:teuthology.orchestra.run.gibba012:> sudo modprobe fuse 2021-02-10T11:37:05.828 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:37:05.828 DEBUG:teuthology.orchestra.run.gibba012:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:37:05.894 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba012.stderr:ceph-fuse[145270]: starting ceph client 2021-02-10T11:37:05.894 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba012.stderr:2021-02-10 11:37:05.907 7fab356b5f80 -1 init, newargv = 0x5609d36c8170 newargc=9 2021-02-10T11:37:05.900 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba012.stderr:ceph-fuse[145270]: starting fuse 2021-02-10T11:37:06.172 INFO:teuthology.orchestra.run.gibba012.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:37:06.172 INFO:teuthology.orchestra.run.gibba012.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:37:06.173 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:37:06.173 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:06.173 DEBUG:teuthology.orchestra.run.gibba012:> ls /sys/fs/fuse/connections 2021-02-10T11:37:06.507 INFO:teuthology.orchestra.run.gibba012.stdout:42 2021-02-10T11:37:06.507 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2021-02-10T11:37:06.508 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:06.508 DEBUG:teuthology.orchestra.run.gibba012:> sudo python3 -c ' 2021-02-10T11:37:06.508 DEBUG:teuthology.orchestra.run.gibba012:> import glob 2021-02-10T11:37:06.508 DEBUG:teuthology.orchestra.run.gibba012:> import re 2021-02-10T11:37:06.508 DEBUG:teuthology.orchestra.run.gibba012:> import os 2021-02-10T11:37:06.508 DEBUG:teuthology.orchestra.run.gibba012:> import subprocess 2021-02-10T11:37:06.508 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:37:06.509 DEBUG:teuthology.orchestra.run.gibba012:> def find_socket(client_name): 2021-02-10T11:37:06.509 DEBUG:teuthology.orchestra.run.gibba012:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2021-02-10T11:37:06.509 DEBUG:teuthology.orchestra.run.gibba012:> files = glob.glob(asok_path) 2021-02-10T11:37:06.509 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:37:06.509 DEBUG:teuthology.orchestra.run.gibba012:> # Given a non-glob path, it better be there 2021-02-10T11:37:06.509 DEBUG:teuthology.orchestra.run.gibba012:> if "*" not in asok_path: 2021-02-10T11:37:06.509 DEBUG:teuthology.orchestra.run.gibba012:> assert(len(files) == 1) 2021-02-10T11:37:06.509 DEBUG:teuthology.orchestra.run.gibba012:> return files[0] 2021-02-10T11:37:06.509 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:37:06.509 DEBUG:teuthology.orchestra.run.gibba012:> for f in files: 2021-02-10T11:37:06.509 DEBUG:teuthology.orchestra.run.gibba012:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:37:06.509 DEBUG:teuthology.orchestra.run.gibba012:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:37:06.509 DEBUG:teuthology.orchestra.run.gibba012:> return f 2021-02-10T11:37:06.509 DEBUG:teuthology.orchestra.run.gibba012:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:37:06.509 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:37:06.510 DEBUG:teuthology.orchestra.run.gibba012:> print(find_socket("client.0")) 2021-02-10T11:37:06.510 DEBUG:teuthology.orchestra.run.gibba012:> ' 2021-02-10T11:37:06.871 INFO:teuthology.orchestra.run.gibba012.stdout:/var/run/ceph/ceph-client.0.145270.asok 2021-02-10T11:37:06.874 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.145270.asok 2021-02-10T11:37:06.874 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:06.874 DEBUG:teuthology.orchestra.run.gibba012:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.145270.asok status 2021-02-10T11:37:07.263 INFO:teuthology.orchestra.run.gibba012.stdout:{ 2021-02-10T11:37:07.263 INFO:teuthology.orchestra.run.gibba012.stdout: "metadata": { 2021-02-10T11:37:07.263 INFO:teuthology.orchestra.run.gibba012.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:37:07.263 INFO:teuthology.orchestra.run.gibba012.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:37:07.263 INFO:teuthology.orchestra.run.gibba012.stdout: "entity_id": "0", 2021-02-10T11:37:07.263 INFO:teuthology.orchestra.run.gibba012.stdout: "hostname": "gibba012", 2021-02-10T11:37:07.263 INFO:teuthology.orchestra.run.gibba012.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2021-02-10T11:37:07.264 INFO:teuthology.orchestra.run.gibba012.stdout: "pid": "145270", 2021-02-10T11:37:07.264 INFO:teuthology.orchestra.run.gibba012.stdout: "root": "/" 2021-02-10T11:37:07.264 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:37:07.264 INFO:teuthology.orchestra.run.gibba012.stdout: "dentry_count": 0, 2021-02-10T11:37:07.264 INFO:teuthology.orchestra.run.gibba012.stdout: "dentry_pinned_count": 0, 2021-02-10T11:37:07.264 INFO:teuthology.orchestra.run.gibba012.stdout: "id": 9366, 2021-02-10T11:37:07.264 INFO:teuthology.orchestra.run.gibba012.stdout: "inst": { 2021-02-10T11:37:07.264 INFO:teuthology.orchestra.run.gibba012.stdout: "name": { 2021-02-10T11:37:07.264 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "client", 2021-02-10T11:37:07.264 INFO:teuthology.orchestra.run.gibba012.stdout: "num": 9366 2021-02-10T11:37:07.264 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:37:07.265 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": { 2021-02-10T11:37:07.265 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:37:07.265 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:0", 2021-02-10T11:37:07.265 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 2485950326 2021-02-10T11:37:07.265 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:37:07.265 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:37:07.265 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": { 2021-02-10T11:37:07.265 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:37:07.265 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:0", 2021-02-10T11:37:07.265 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 2485950326 2021-02-10T11:37:07.266 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:37:07.266 INFO:teuthology.orchestra.run.gibba012.stdout: "inst_str": "client.9366 172.21.2.112:0/2485950326", 2021-02-10T11:37:07.266 INFO:teuthology.orchestra.run.gibba012.stdout: "addr_str": "172.21.2.112:0/2485950326", 2021-02-10T11:37:07.266 INFO:teuthology.orchestra.run.gibba012.stdout: "inode_count": 1, 2021-02-10T11:37:07.266 INFO:teuthology.orchestra.run.gibba012.stdout: "mds_epoch": 64, 2021-02-10T11:37:07.266 INFO:teuthology.orchestra.run.gibba012.stdout: "osd_epoch": 83, 2021-02-10T11:37:07.266 INFO:teuthology.orchestra.run.gibba012.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:37:07.266 INFO:teuthology.orchestra.run.gibba012.stdout: "blacklisted": false 2021-02-10T11:37:07.266 INFO:teuthology.orchestra.run.gibba012.stdout:} 2021-02-10T11:37:07.267 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:07.267 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:37:07.267 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:37:07.603 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:37:07.603 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:37:07.603 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:07.604 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:37:07.949 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:37:08.375 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.370 7f5197389700 1 Processor -- start 2021-02-10T11:37:08.375 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.370 7f5197389700 1 -- start start 2021-02-10T11:37:08.375 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.371 7f5197389700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f51900f05f0 0x7f51900f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:08.375 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.371 7f5197389700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51901b0c70 con 0x7f51901b0870 2021-02-10T11:37:08.375 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.371 7f5197389700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51900f0b50 con 0x7f51901b0470 2021-02-10T11:37:08.376 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.371 7f5197389700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51900e5ea0 con 0x7f51900f05f0 2021-02-10T11:37:08.376 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.371 7f51949a4700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f51901b0870 0x7f51900efe00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:52120/0 (socket says 172.21.2.112:52120) 2021-02-10T11:37:08.376 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.371 7f51949a4700 1 -- 172.21.2.112:0/1233978697 learned_addr learned my addr 172.21.2.112:0/1233978697 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:37:08.376 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.371 7f5187fff700 1 -- 172.21.2.112:0/1233978697 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1673055649 0 0) 0x7f51800010b0 con 0x7f51901b0870 2021-02-10T11:37:08.376 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.371 7f5187fff700 1 -- 172.21.2.112:0/1233978697 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f517c002fe0 con 0x7f51901b0870 2021-02-10T11:37:08.376 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.372 7f51959a6700 1 --2- 172.21.2.112:0/1233978697 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f51900f05f0 0x7f51900f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:08.376 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.372 7f5187fff700 1 -- 172.21.2.112:0/1233978697 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 358295437 0 0) 0x7f51800010b0 con 0x7f51901b0870 2021-02-10T11:37:08.376 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.372 7f5187fff700 1 -- 172.21.2.112:0/1233978697 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f51900f05f0 msgr2=0x7f51900f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:37:08.376 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.372 7f5187fff700 1 --2- 172.21.2.112:0/1233978697 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f51900f05f0 0x7f51900f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:08.376 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.372 7f5187fff700 1 -- 172.21.2.112:0/1233978697 >> v1:172.21.2.120:6789/0 conn(0x7f51901b0470 legacy=0x7f51900ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:08.376 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.372 7f5187fff700 1 -- 172.21.2.112:0/1233978697 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51900e52e0 con 0x7f51901b0870 2021-02-10T11:37:08.376 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.372 7f5187fff700 1 -- 172.21.2.112:0/1233978697 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f51800015b0 con 0x7f51901b0870 2021-02-10T11:37:08.376 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.372 7f5187fff700 1 -- 172.21.2.112:0/1233978697 >> v1:172.21.2.112:6789/0 conn(0x7f51901b0870 legacy=0x7f51900efe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:08.377 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.372 7f5187fff700 1 --2- 172.21.2.112:0/1233978697 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f517c003a70 0x7f517c005ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:08.377 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.372 7f5187fff700 1 --2- 172.21.2.112:0/1233978697 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f517c006760 0x7f517c008c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:08.377 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.372 7f5187fff700 1 --2- 172.21.2.112:0/1233978697 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f517c009500 0x7f517c00ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:08.377 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.372 7f5187fff700 1 -- 172.21.2.112:0/1233978697 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f517c00c2d0 con 0x7f517c006760 2021-02-10T11:37:08.377 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.372 7f5187fff700 1 -- 172.21.2.112:0/1233978697 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f517c00c4f0 con 0x7f517c009500 2021-02-10T11:37:08.377 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.372 7f5187fff700 1 -- 172.21.2.112:0/1233978697 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f517c00c710 con 0x7f517c003a70 2021-02-10T11:37:08.377 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.372 7f51949a4700 1 --2- 172.21.2.112:0/1233978697 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f517c006760 0x7f517c008c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:08.377 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.372 7f51951a5700 1 --2- 172.21.2.112:0/1233978697 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f517c003a70 0x7f517c005ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:08.377 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.372 7f51959a6700 1 --2- 172.21.2.112:0/1233978697 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f517c009500 0x7f517c00ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:08.377 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.372 7f51949a4700 1 -- 172.21.2.112:0/1233978697 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f517c003a70 msgr2=0x7f517c005ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:37:08.377 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.372 7f51949a4700 1 --2- 172.21.2.112:0/1233978697 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f517c003a70 0x7f517c005ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:08.377 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.372 7f51949a4700 1 -- 172.21.2.112:0/1233978697 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f517c009500 msgr2=0x7f517c00ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:37:08.377 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.372 7f51949a4700 1 --2- 172.21.2.112:0/1233978697 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f517c009500 0x7f517c00ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:08.377 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.372 7f51949a4700 1 -- 172.21.2.112:0/1233978697 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f517c00c9b0 con 0x7f517c006760 2021-02-10T11:37:08.378 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.372 7f51959a6700 1 --2- 172.21.2.112:0/1233978697 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f517c009500 0x7f517c00ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:37:08.378 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.372 7f51949a4700 1 --2- 172.21.2.112:0/1233978697 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f517c006760 0x7f517c008c60 secure :-1 s=READY pgs=1730 cs=0 l=1 rev1=1 rx=0x7f5180007560 tx=0x7f5180007c90).ready entity=mon.0 client_cookie=e6dfa269acc4adf3 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:08.378 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.372 7f5187fff700 1 -- 172.21.2.112:0/1233978697 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5180001010 con 0x7f517c006760 2021-02-10T11:37:08.378 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.372 7f5187fff700 1 -- 172.21.2.112:0/1233978697 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51800013f0 con 0x7f517c006760 2021-02-10T11:37:08.378 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.372 7f5187fff700 1 -- 172.21.2.112:0/1233978697 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f51800070d0 con 0x7f517c006760 2021-02-10T11:37:08.378 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.372 7f5197389700 1 -- 172.21.2.112:0/1233978697 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f517c006760 msgr2=0x7f517c008c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:08.378 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.372 7f5197389700 1 --2- 172.21.2.112:0/1233978697 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f517c006760 0x7f517c008c60 secure :-1 s=READY pgs=1730 cs=0 l=1 rev1=1 rx=0x7f5180007560 tx=0x7f5180007c90).stop 2021-02-10T11:37:08.378 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.372 7f51959a6700 1 -- 172.21.2.112:0/1233978697 reap_dead start 2021-02-10T11:37:08.378 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.373 7f5197389700 1 -- 172.21.2.112:0/1233978697 shutdown_connections 2021-02-10T11:37:08.378 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.373 7f5197389700 1 -- 172.21.2.112:0/1233978697 >> 172.21.2.112:0/1233978697 conn(0x7f51900ce3e0 msgr2=0x7f51900e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:08.378 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.373 7f5197389700 1 -- 172.21.2.112:0/1233978697 shutdown_connections 2021-02-10T11:37:08.379 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.373 7f5197389700 1 -- 172.21.2.112:0/1233978697 wait complete. 2021-02-10T11:37:08.379 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.388 7f5197389700 1 Processor -- start 2021-02-10T11:37:08.379 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.388 7f5197389700 1 -- start start 2021-02-10T11:37:08.380 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.388 7f5197389700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f51900b11f0 0x7f5190189d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:08.380 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.388 7f5197389700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f519018a390 0x7f519018a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:08.380 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.388 7f5197389700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f519018adb0 0x7f5190187e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:08.380 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.388 7f5197389700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51900b15f0 con 0x7f51900b11f0 2021-02-10T11:37:08.380 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.388 7f5197389700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5190188440 con 0x7f519018a390 2021-02-10T11:37:08.380 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.389 7f5197389700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5190188660 con 0x7f519018adb0 2021-02-10T11:37:08.380 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.389 7f51951a5700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f51900b11f0 0x7f5190189d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:08.381 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.389 7f51951a5700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f51900b11f0 0x7f5190189d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:52670/0 (socket says 172.21.2.112:52670) 2021-02-10T11:37:08.381 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.389 7f51951a5700 1 -- 172.21.2.112:0/751657449 learned_addr learned my addr 172.21.2.112:0/751657449 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:37:08.381 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.389 7f51951a5700 1 -- 172.21.2.112:0/751657449 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f519018adb0 msgr2=0x7f5190187e20 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:37:08.381 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.389 7f51951a5700 1 --2- 172.21.2.112:0/751657449 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f519018adb0 0x7f5190187e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:08.381 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.389 7f51951a5700 1 -- 172.21.2.112:0/751657449 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f519018a390 msgr2=0x7f519018a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:08.381 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.389 7f51951a5700 1 --2- 172.21.2.112:0/751657449 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f519018a390 0x7f519018a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:08.381 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.389 7f51951a5700 1 -- 172.21.2.112:0/751657449 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5190188880 con 0x7f51900b11f0 2021-02-10T11:37:08.381 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.389 7f51951a5700 1 --2- 172.21.2.112:0/751657449 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f51900b11f0 0x7f5190189d70 secure :-1 s=READY pgs=1731 cs=0 l=1 rev1=1 rx=0x7f518c006a30 tx=0x7f518c00b8e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:08.381 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.389 7f5185ffb700 1 -- 172.21.2.112:0/751657449 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f518c006b60 con 0x7f51900b11f0 2021-02-10T11:37:08.381 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.389 7f5185ffb700 1 -- 172.21.2.112:0/751657449 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f518c000e20 con 0x7f51900b11f0 2021-02-10T11:37:08.381 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.389 7f5185ffb700 1 -- 172.21.2.112:0/751657449 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f518c002080 con 0x7f51900b11f0 2021-02-10T11:37:08.381 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.389 7f5197389700 1 -- 172.21.2.112:0/751657449 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5190188880 con 0x7f51900b11f0 2021-02-10T11:37:08.381 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.389 7f5197389700 1 -- 172.21.2.112:0/751657449 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5190188880 con 0x7f51900b11f0 2021-02-10T11:37:08.381 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.390 7f5185ffb700 1 -- 172.21.2.112:0/751657449 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f518c006b60 con 0x7f51900b11f0 2021-02-10T11:37:08.381 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.390 7f5185ffb700 1 --2- 172.21.2.112:0/751657449 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5170018730 0x7f517001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:08.382 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.390 7f5185ffb700 1 -- 172.21.2.112:0/751657449 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f518c032790 con 0x7f51900b11f0 2021-02-10T11:37:08.382 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.390 7f51949a4700 1 --2- 172.21.2.112:0/751657449 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5170018730 0x7f517001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:08.382 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.391 7f51949a4700 1 --2- 172.21.2.112:0/751657449 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5170018730 0x7f517001abc0 crc :-1 s=READY pgs=819 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:08.408 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.421 7f5197389700 1 -- 172.21.2.112:0/751657449 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5190189840 con 0x7f51900b11f0 2021-02-10T11:37:08.410 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.423 7f5185ffb700 1 -- 172.21.2.112:0/751657449 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f518c006b60 con 0x7f51900b11f0 2021-02-10T11:37:08.534 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.547 7f5197389700 1 -- 172.21.2.112:0/751657449 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5190189840 con 0x7f51900b11f0 2021-02-10T11:37:08.535 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.548 7f5185ffb700 1 -- 172.21.2.112:0/751657449 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) v1 ==== 74+0+13996 (secure 0 0 0) 0x7f518c00bcd0 con 0x7f51900b11f0 2021-02-10T11:37:08.535 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:37:08.535 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":83,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:36:54.566363","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:36:51.538038","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":17,"pool_name":"cephfs_data","create_time":"2021-02-10 11:36:52.609006","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:37:08.536 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.549 7f5197389700 1 -- 172.21.2.112:0/751657449 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5170018730 msgr2=0x7f517001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:08.536 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.549 7f5197389700 1 --2- 172.21.2.112:0/751657449 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5170018730 0x7f517001abc0 crc :-1 s=READY pgs=819 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:08.536 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.549 7f5197389700 1 -- 172.21.2.112:0/751657449 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f51900b11f0 msgr2=0x7f5190189d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:08.536 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.549 7f5197389700 1 --2- 172.21.2.112:0/751657449 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f51900b11f0 0x7f5190189d70 secure :-1 s=READY pgs=1731 cs=0 l=1 rev1=1 rx=0x7f518c006a30 tx=0x7f518c00b8e0).stop 2021-02-10T11:37:08.537 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.549 7f5197389700 1 -- 172.21.2.112:0/751657449 shutdown_connections 2021-02-10T11:37:08.537 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.549 7f5197389700 1 --2- 172.21.2.112:0/751657449 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5170018730 0x7f517001abc0 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:08.537 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.549 7f5197389700 1 --2- 172.21.2.112:0/751657449 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f519018a390 0x7f519018a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:08.537 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.549 7f5197389700 1 --2- 172.21.2.112:0/751657449 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f519018adb0 0x7f5190187e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:08.537 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.549 7f5197389700 1 --2- 172.21.2.112:0/751657449 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f51900b11f0 0x7f5190189d70 unknown :-1 s=CLOSED pgs=1731 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:08.537 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.549 7f5197389700 1 -- 172.21.2.112:0/751657449 >> 172.21.2.112:0/751657449 conn(0x7f51900a78c0 msgr2=0x7f51900b1f70 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:08.538 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.550 7f5197389700 1 -- 172.21.2.112:0/751657449 shutdown_connections 2021-02-10T11:37:08.538 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.550 7f5197389700 1 -- 172.21.2.112:0/751657449 wait complete. 2021-02-10T11:37:08.545 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:13:57.508758', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:36:51.538038', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:36:52.609006', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '83', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:37:08.545 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:37:08.976 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.973 7f8f9acae700 1 Processor -- start 2021-02-10T11:37:08.976 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.973 7f8f9acae700 1 -- start start 2021-02-10T11:37:08.976 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.973 7f8f9acae700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8f940d1870 0x7f8f941b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:08.976 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.974 7f8f9acae700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f8f940cff70 0x7f8f94109130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:08.977 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.974 7f8f9acae700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f94109a60 con 0x7f8f941b0ae0 2021-02-10T11:37:08.977 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.974 7f8f9acae700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f94109750 con 0x7f8f940cff70 2021-02-10T11:37:08.977 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.974 7f8f9acae700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f94088070 con 0x7f8f940d1870 2021-02-10T11:37:08.977 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.974 7f8f992cb700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f8f940cff70 0x7f8f94109130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:08.977 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.974 7f8f992cb700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f8f940cff70 0x7f8f94109130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:52678/0 (socket says 172.21.2.112:52678) 2021-02-10T11:37:08.977 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.974 7f8f992cb700 1 -- 172.21.2.112:0/748496751 learned_addr learned my addr 172.21.2.112:0/748496751 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:37:08.977 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.974 7f8f992cb700 1 -- 172.21.2.112:0/748496751 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8f940d1870 msgr2=0x7f8f941b04c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:37:08.977 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.974 7f8f992cb700 1 --2- 172.21.2.112:0/748496751 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8f940d1870 0x7f8f941b04c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:08.977 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.974 7f8f992cb700 1 -- 172.21.2.112:0/748496751 >> v1:172.21.2.112:6789/0 conn(0x7f8f941b0ae0 legacy=0x7f8f94108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:08.977 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.974 7f8f992cb700 1 -- 172.21.2.112:0/748496751 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f94088260 con 0x7f8f940cff70 2021-02-10T11:37:08.977 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.974 7f8f992cb700 1 --2- 172.21.2.112:0/748496751 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8f940cff70 0x7f8f94109130 secure :-1 s=READY pgs=1733 cs=0 l=1 rev1=1 rx=0x7f8f8c006f00 tx=0x7f8f8c009f10).ready entity=mon.0 client_cookie=b768ad3894e8c527 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:08.977 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.974 7f8f937fe700 1 -- 172.21.2.112:0/748496751 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8f8c00b3b0 con 0x7f8f940cff70 2021-02-10T11:37:08.977 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.974 7f8f937fe700 1 -- 172.21.2.112:0/748496751 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f8c00b630 con 0x7f8f940cff70 2021-02-10T11:37:08.978 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.974 7f8f937fe700 1 -- 172.21.2.112:0/748496751 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8f8c00bb50 con 0x7f8f940cff70 2021-02-10T11:37:08.978 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.974 7f8f9acae700 1 -- 172.21.2.112:0/748496751 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8f940cff70 msgr2=0x7f8f94109130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:08.978 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.974 7f8f9acae700 1 --2- 172.21.2.112:0/748496751 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8f940cff70 0x7f8f94109130 secure :-1 s=READY pgs=1733 cs=0 l=1 rev1=1 rx=0x7f8f8c006f00 tx=0x7f8f8c009f10).stop 2021-02-10T11:37:08.978 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.975 7f8f9acae700 1 -- 172.21.2.112:0/748496751 shutdown_connections 2021-02-10T11:37:08.978 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.975 7f8f9acae700 1 --2- 172.21.2.112:0/748496751 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8f940cff70 0x7f8f94109130 unknown :-1 s=CLOSED pgs=1733 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:08.978 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.975 7f8f9acae700 1 --2- 172.21.2.112:0/748496751 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8f940d1870 0x7f8f941b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:08.978 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.975 7f8f9acae700 1 -- 172.21.2.112:0/748496751 >> 172.21.2.112:0/748496751 conn(0x7f8f940ce420 msgr2=0x7f8f940d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:08.978 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.975 7f8f9acae700 1 -- 172.21.2.112:0/748496751 shutdown_connections 2021-02-10T11:37:08.978 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.975 7f8f9acae700 1 -- 172.21.2.112:0/748496751 wait complete. 2021-02-10T11:37:08.979 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.989 7f8f9acae700 1 Processor -- start 2021-02-10T11:37:08.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.989 7f8f9acae700 1 -- start start 2021-02-10T11:37:08.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.990 7f8f9acae700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8f940c2370 0x7f8f9418f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:08.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.990 7f8f9acae700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8f94084cb0 0x7f8f941a7470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:08.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.990 7f8f9acae700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8f941a7a90 0x7f8f94085700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:08.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.990 7f8f9acae700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f940be840 con 0x7f8f940c2370 2021-02-10T11:37:08.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.990 7f8f9acae700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f940c29a0 con 0x7f8f94084cb0 2021-02-10T11:37:08.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.990 7f8f9acae700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f941a7e90 con 0x7f8f941a7a90 2021-02-10T11:37:08.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.990 7f8f98aca700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8f940c2370 0x7f8f9418f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:08.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.990 7f8f98aca700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8f940c2370 0x7f8f9418f870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:52684/0 (socket says 172.21.2.112:52684) 2021-02-10T11:37:08.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.990 7f8f98aca700 1 -- 172.21.2.112:0/4033879191 learned_addr learned my addr 172.21.2.112:0/4033879191 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:37:08.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.990 7f8f98aca700 1 -- 172.21.2.112:0/4033879191 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8f941a7a90 msgr2=0x7f8f94085700 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:37:08.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.990 7f8f98aca700 1 --2- 172.21.2.112:0/4033879191 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8f941a7a90 0x7f8f94085700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:08.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.990 7f8f98aca700 1 -- 172.21.2.112:0/4033879191 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8f94084cb0 msgr2=0x7f8f941a7470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:08.981 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.990 7f8f98aca700 1 --2- 172.21.2.112:0/4033879191 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8f94084cb0 0x7f8f941a7470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:08.981 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.990 7f8f98aca700 1 -- 172.21.2.112:0/4033879191 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f941a80b0 con 0x7f8f940c2370 2021-02-10T11:37:08.981 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.991 7f8f98aca700 1 --2- 172.21.2.112:0/4033879191 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8f940c2370 0x7f8f9418f870 secure :-1 s=READY pgs=1734 cs=0 l=1 rev1=1 rx=0x7f8f7c00b5c0 tx=0x7f8f7c00b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:08.981 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.991 7f8f917fa700 1 -- 172.21.2.112:0/4033879191 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8f7c00bd50 con 0x7f8f940c2370 2021-02-10T11:37:08.981 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.991 7f8f917fa700 1 -- 172.21.2.112:0/4033879191 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f7c006030 con 0x7f8f940c2370 2021-02-10T11:37:08.981 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.991 7f8f917fa700 1 -- 172.21.2.112:0/4033879191 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8f7c0054d0 con 0x7f8f940c2370 2021-02-10T11:37:08.981 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.991 7f8f9acae700 1 -- 172.21.2.112:0/4033879191 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f941a80b0 con 0x7f8f940c2370 2021-02-10T11:37:08.981 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.991 7f8f9acae700 1 -- 172.21.2.112:0/4033879191 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f94086010 con 0x7f8f940c2370 2021-02-10T11:37:08.981 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.992 7f8f917fa700 1 -- 172.21.2.112:0/4033879191 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8f7c006210 con 0x7f8f940c2370 2021-02-10T11:37:08.981 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.992 7f8f917fa700 1 --2- 172.21.2.112:0/4033879191 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8f780186c0 0x7f8f7801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:08.981 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.992 7f8f917fa700 1 -- 172.21.2.112:0/4033879191 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f8f7c032420 con 0x7f8f940c2370 2021-02-10T11:37:08.981 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.992 7f8f93fff700 1 --2- 172.21.2.112:0/4033879191 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8f780186c0 0x7f8f7801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:08.981 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:08.992 7f8f93fff700 1 --2- 172.21.2.112:0/4033879191 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8f780186c0 0x7f8f7801abc0 crc :-1 s=READY pgs=820 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:09.011 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:09.024 7f8f9acae700 1 -- 172.21.2.112:0/4033879191 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f94082a70 con 0x7f8f940c2370 2021-02-10T11:37:09.013 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:09.026 7f8f917fa700 1 -- 172.21.2.112:0/4033879191 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8f7c000940 con 0x7f8f940c2370 2021-02-10T11:37:09.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:09.154 7f8f9acae700 1 -- 172.21.2.112:0/4033879191 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f8f94082a70 con 0x7f8f940c2370 2021-02-10T11:37:09.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:09.154 7f8f917fa700 1 -- 172.21.2.112:0/4033879191 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v83) v1 ==== 92+0+10 (secure 0 0 0) 0x7f8f7c00bd50 con 0x7f8f940c2370 2021-02-10T11:37:09.142 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:37:09.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:09.156 7f8f9acae700 1 -- 172.21.2.112:0/4033879191 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8f780186c0 msgr2=0x7f8f7801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:09.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:09.156 7f8f9acae700 1 --2- 172.21.2.112:0/4033879191 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8f780186c0 0x7f8f7801abc0 crc :-1 s=READY pgs=820 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:09.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:09.156 7f8f9acae700 1 -- 172.21.2.112:0/4033879191 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8f940c2370 msgr2=0x7f8f9418f870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:09.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:09.156 7f8f9acae700 1 --2- 172.21.2.112:0/4033879191 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8f940c2370 0x7f8f9418f870 secure :-1 s=READY pgs=1734 cs=0 l=1 rev1=1 rx=0x7f8f7c00b5c0 tx=0x7f8f7c00b960).stop 2021-02-10T11:37:09.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:09.156 7f8f9acae700 1 -- 172.21.2.112:0/4033879191 shutdown_connections 2021-02-10T11:37:09.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:09.156 7f8f9acae700 1 --2- 172.21.2.112:0/4033879191 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8f780186c0 0x7f8f7801abc0 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:09.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:09.156 7f8f9acae700 1 --2- 172.21.2.112:0/4033879191 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8f94084cb0 0x7f8f941a7470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:09.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:09.156 7f8f9acae700 1 --2- 172.21.2.112:0/4033879191 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8f941a7a90 0x7f8f94085700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:09.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:09.156 7f8f9acae700 1 --2- 172.21.2.112:0/4033879191 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8f940c2370 0x7f8f9418f870 unknown :-1 s=CLOSED pgs=1734 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:09.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:09.156 7f8f9acae700 1 -- 172.21.2.112:0/4033879191 >> 172.21.2.112:0/4033879191 conn(0x7f8f940b5dc0 msgr2=0x7f8f940c2d90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:09.145 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:09.156 7f8f9acae700 1 -- 172.21.2.112:0/4033879191 shutdown_connections 2021-02-10T11:37:09.145 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:09.156 7f8f9acae700 1 -- 172.21.2.112:0/4033879191 wait complete. 2021-02-10T11:37:09.152 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:37:09.583 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:09.579 7ff77d2cd700 1 Processor -- start 2021-02-10T11:37:09.583 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:09.579 7ff77d2cd700 1 -- start start 2021-02-10T11:37:09.583 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:09.579 7ff77d2cd700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7ff7780d1c70 0x7ff7780cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:09.583 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:09.579 7ff77d2cd700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7ff7780cf580 0x7ff7780cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:09.583 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:09.579 7ff77d2cd700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7780fe1c0 con 0x7ff7780d1870 2021-02-10T11:37:09.583 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:09.579 7ff77d2cd700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7781b0cb0 con 0x7ff7780d1c70 2021-02-10T11:37:09.583 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:09.579 7ff77d2cd700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff778109a60 con 0x7ff7780cf580 2021-02-10T11:37:09.583 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:09.579 7ff77687c700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7ff7780d1c70 0x7ff7780cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:09.583 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:09.580 7ff77687c700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7ff7780d1c70 0x7ff7780cef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:52692/0 (socket says 172.21.2.112:52692) 2021-02-10T11:37:09.584 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:09.580 7ff77687c700 1 -- 172.21.2.112:0/2283604817 learned_addr learned my addr 172.21.2.112:0/2283604817 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:37:09.584 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:09.580 7ff77787e700 1 --2- 172.21.2.112:0/2283604817 >> v2:172.21.2.120:3300/0 conn(0x7ff7780cf580 0x7ff7780cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:09.584 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:09.580 7ff77687c700 1 -- 172.21.2.112:0/2283604817 >> v2:172.21.2.120:3300/0 conn(0x7ff7780cf580 msgr2=0x7ff7780cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:37:09.584 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:09.580 7ff77687c700 1 --2- 172.21.2.112:0/2283604817 >> v2:172.21.2.120:3300/0 conn(0x7ff7780cf580 0x7ff7780cf980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:09.584 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:09.580 7ff77687c700 1 -- 172.21.2.112:0/2283604817 >> v1:172.21.2.120:6789/0 conn(0x7ff7780d1870 legacy=0x7ff7781b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:09.584 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:09.580 7ff77687c700 1 -- 172.21.2.112:0/2283604817 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff778088070 con 0x7ff7780d1c70 2021-02-10T11:37:09.584 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:09.580 7ff77687c700 1 --2- 172.21.2.112:0/2283604817 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff7780d1c70 0x7ff7780cef60 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 rx=0x7ff768006700 tx=0x7ff768009b80).ready entity=mon.0 client_cookie=99c2d892a7283cc2 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:09.584 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:09.580 7ff77607b700 1 -- 172.21.2.112:0/2283604817 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff76800c0c0 con 0x7ff7780d1c70 2021-02-10T11:37:09.584 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:09.580 7ff77607b700 1 -- 172.21.2.112:0/2283604817 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff76800c400 con 0x7ff7780d1c70 2021-02-10T11:37:09.584 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:09.580 7ff77607b700 1 -- 172.21.2.112:0/2283604817 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff76800c8e0 con 0x7ff7780d1c70 2021-02-10T11:37:09.584 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:09.580 7ff77d2cd700 1 -- 172.21.2.112:0/2283604817 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff7780d1c70 msgr2=0x7ff7780cef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:09.584 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:09.580 7ff77d2cd700 1 --2- 172.21.2.112:0/2283604817 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff7780d1c70 0x7ff7780cef60 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 rx=0x7ff768006700 tx=0x7ff768009b80).stop 2021-02-10T11:37:09.584 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:09.581 7ff77d2cd700 1 -- 172.21.2.112:0/2283604817 shutdown_connections 2021-02-10T11:37:09.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:09.581 7ff77d2cd700 1 --2- 172.21.2.112:0/2283604817 >> v2:172.21.2.120:3300/0 conn(0x7ff7780cf580 0x7ff7780cf980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:09.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:09.581 7ff77d2cd700 1 --2- 172.21.2.112:0/2283604817 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff7780d1c70 0x7ff7780cef60 unknown :-1 s=CLOSED pgs=1735 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:09.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:09.581 7ff77d2cd700 1 -- 172.21.2.112:0/2283604817 >> 172.21.2.112:0/2283604817 conn(0x7ff7780ce420 msgr2=0x7ff7780d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:09.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:09.581 7ff77d2cd700 1 -- 172.21.2.112:0/2283604817 shutdown_connections 2021-02-10T11:37:09.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:09.581 7ff77d2cd700 1 -- 172.21.2.112:0/2283604817 wait complete. 2021-02-10T11:37:09.587 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:09.596 7ff77d2cd700 1 Processor -- start 2021-02-10T11:37:09.587 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:09.596 7ff77d2cd700 1 -- start start 2021-02-10T11:37:09.587 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:09.596 7ff77d2cd700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff778085bc0 0x7ff7780863e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:09.587 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:09.596 7ff77d2cd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff7780954c0 0x7ff778096300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:09.587 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:09.596 7ff77d2cd700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff7780958c0 0x7ff778096920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:09.587 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:09.596 7ff77d2cd700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff778095d80 con 0x7ff7780954c0 2021-02-10T11:37:09.587 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:09.596 7ff77d2cd700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7780bfa60 con 0x7ff7780958c0 2021-02-10T11:37:09.588 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:09.596 7ff77d2cd700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff778085fc0 con 0x7ff778085bc0 2021-02-10T11:37:09.588 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:09.597 7ff77687c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff7780954c0 0x7ff778096300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:09.588 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:09.597 7ff77687c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff7780954c0 0x7ff778096300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:52696/0 (socket says 172.21.2.112:52696) 2021-02-10T11:37:09.588 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:09.597 7ff77687c700 1 -- 172.21.2.112:0/485712777 learned_addr learned my addr 172.21.2.112:0/485712777 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:37:09.588 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:09.597 7ff77787e700 1 --2- 172.21.2.112:0/485712777 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff7780958c0 0x7ff778096920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:09.588 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:09.597 7ff77707d700 1 --2- 172.21.2.112:0/485712777 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff778085bc0 0x7ff7780863e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:09.588 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:09.597 7ff77687c700 1 -- 172.21.2.112:0/485712777 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff778085bc0 msgr2=0x7ff7780863e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:09.588 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:09.597 7ff77687c700 1 --2- 172.21.2.112:0/485712777 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff778085bc0 0x7ff7780863e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:09.588 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:09.597 7ff77687c700 1 -- 172.21.2.112:0/485712777 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff7780958c0 msgr2=0x7ff778096920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:09.588 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:09.597 7ff77687c700 1 --2- 172.21.2.112:0/485712777 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff7780958c0 0x7ff778096920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:09.588 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:09.597 7ff77687c700 1 -- 172.21.2.112:0/485712777 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff778086c10 con 0x7ff7780954c0 2021-02-10T11:37:09.588 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:09.597 7ff77687c700 1 --2- 172.21.2.112:0/485712777 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff7780954c0 0x7ff778096300 secure :-1 s=READY pgs=1736 cs=0 l=1 rev1=1 rx=0x7ff7680066a0 tx=0x7ff76800b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:09.589 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:09.597 7ff76ffff700 1 -- 172.21.2.112:0/485712777 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff76800bd80 con 0x7ff7780954c0 2021-02-10T11:37:09.589 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:09.597 7ff76ffff700 1 -- 172.21.2.112:0/485712777 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7680052e0 con 0x7ff7780954c0 2021-02-10T11:37:09.589 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:09.597 7ff77d2cd700 1 -- 172.21.2.112:0/485712777 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff778086c10 con 0x7ff7780954c0 2021-02-10T11:37:09.589 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:09.597 7ff76ffff700 1 -- 172.21.2.112:0/485712777 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff768005730 con 0x7ff7780954c0 2021-02-10T11:37:09.589 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:09.598 7ff77d2cd700 1 -- 172.21.2.112:0/485712777 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff778086c10 con 0x7ff7780954c0 2021-02-10T11:37:09.589 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:09.598 7ff76ffff700 1 -- 172.21.2.112:0/485712777 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff768006240 con 0x7ff7780954c0 2021-02-10T11:37:09.589 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:09.598 7ff76ffff700 1 --2- 172.21.2.112:0/485712777 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff75c018730 0x7ff75c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:09.589 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:09.598 7ff76ffff700 1 -- 172.21.2.112:0/485712777 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5573+0+0 (secure 0 0 0) 0x7ff768032660 con 0x7ff7780954c0 2021-02-10T11:37:09.589 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:09.598 7ff77707d700 1 --2- 172.21.2.112:0/485712777 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff75c018730 0x7ff75c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:09.589 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:09.599 7ff77707d700 1 --2- 172.21.2.112:0/485712777 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff75c018730 0x7ff75c01abc0 crc :-1 s=READY pgs=821 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:09.616 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:09.629 7ff77d2cd700 1 -- 172.21.2.112:0/485712777 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7780fe1c0 con 0x7ff7780954c0 2021-02-10T11:37:09.619 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:09.632 7ff76ffff700 1 -- 172.21.2.112:0/485712777 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff768006030 con 0x7ff7780954c0 2021-02-10T11:37:09.749 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:09.761 7ff77d2cd700 1 -- 172.21.2.112:0/485712777 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7ff7780fe1c0 con 0x7ff7780954c0 2021-02-10T11:37:09.749 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:09.762 7ff76ffff700 1 -- 172.21.2.112:0/485712777 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v83) v1 ==== 104+0+10 (secure 0 0 0) 0x7ff768006240 con 0x7ff7780954c0 2021-02-10T11:37:09.750 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:37:09.751 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:09.764 7ff77d2cd700 1 -- 172.21.2.112:0/485712777 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff75c018730 msgr2=0x7ff75c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:09.751 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:09.764 7ff77d2cd700 1 --2- 172.21.2.112:0/485712777 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff75c018730 0x7ff75c01abc0 crc :-1 s=READY pgs=821 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:09.751 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:09.764 7ff77d2cd700 1 -- 172.21.2.112:0/485712777 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff7780954c0 msgr2=0x7ff778096300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:09.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:09.764 7ff77d2cd700 1 --2- 172.21.2.112:0/485712777 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff7780954c0 0x7ff778096300 secure :-1 s=READY pgs=1736 cs=0 l=1 rev1=1 rx=0x7ff7680066a0 tx=0x7ff76800b990).stop 2021-02-10T11:37:09.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:09.764 7ff77d2cd700 1 -- 172.21.2.112:0/485712777 shutdown_connections 2021-02-10T11:37:09.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:09.764 7ff77d2cd700 1 --2- 172.21.2.112:0/485712777 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff75c018730 0x7ff75c01abc0 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:09.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:09.764 7ff77d2cd700 1 --2- 172.21.2.112:0/485712777 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff7780958c0 0x7ff778096920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:09.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:09.764 7ff77d2cd700 1 --2- 172.21.2.112:0/485712777 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff7780954c0 0x7ff778096300 unknown :-1 s=CLOSED pgs=1736 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:09.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:09.764 7ff77d2cd700 1 --2- 172.21.2.112:0/485712777 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff778085bc0 0x7ff7780863e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:09.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:09.764 7ff77d2cd700 1 -- 172.21.2.112:0/485712777 >> 172.21.2.112:0/485712777 conn(0x7ff7780bef00 msgr2=0x7ff77809d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:09.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:09.764 7ff77d2cd700 1 -- 172.21.2.112:0/485712777 shutdown_connections 2021-02-10T11:37:09.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:09.764 7ff77d2cd700 1 -- 172.21.2.112:0/485712777 wait complete. 2021-02-10T11:37:09.760 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:37:10.193 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.188 7f97fcb85700 1 Processor -- start 2021-02-10T11:37:10.193 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.189 7f97fcb85700 1 -- start start 2021-02-10T11:37:10.193 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.189 7f97fcb85700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f97f80d1860 0x7f97f81b04b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:10.193 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.189 7f97fcb85700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f97f81b0ad0 0x7f97f8108930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:10.193 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.189 7f97fcb85700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97f8109a50 con 0x7f97f80d1c60 2021-02-10T11:37:10.193 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.189 7f97fcb85700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97f80e5ee0 con 0x7f97f81b0ad0 2021-02-10T11:37:10.194 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.189 7f97fcb85700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97f8109740 con 0x7f97f80d1860 2021-02-10T11:37:10.194 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.189 7f97f705f700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f97f80d1c60 0x7f97f8108f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:52170/0 (socket says 172.21.2.112:52170) 2021-02-10T11:37:10.194 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.189 7f97f705f700 1 -- 172.21.2.112:0/3867944157 learned_addr learned my addr 172.21.2.112:0/3867944157 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:37:10.194 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.190 7f97f685e700 1 --2- 172.21.2.112:0/3867944157 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f97f80d1860 0x7f97f81b04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:10.194 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.190 7f97f605d700 1 --2- 172.21.2.112:0/3867944157 >> v2:172.21.2.120:3300/0 conn(0x7f97f81b0ad0 0x7f97f8108930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:10.194 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.190 7f97f585c700 1 -- 172.21.2.112:0/3867944157 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 473818881 0 0) 0x7f97f0001500 con 0x7f97f80d1c60 2021-02-10T11:37:10.194 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.190 7f97f585c700 1 -- 172.21.2.112:0/3867944157 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97e4002fe0 con 0x7f97f80d1c60 2021-02-10T11:37:10.194 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.190 7f97f685e700 1 -- 172.21.2.112:0/3867944157 >> v2:172.21.2.120:3300/0 conn(0x7f97f81b0ad0 msgr2=0x7f97f8108930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:37:10.194 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.190 7f97f685e700 1 --2- 172.21.2.112:0/3867944157 >> v2:172.21.2.120:3300/0 conn(0x7f97f81b0ad0 0x7f97f8108930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:10.194 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.190 7f97f685e700 1 -- 172.21.2.112:0/3867944157 >> v1:172.21.2.112:6789/0 conn(0x7f97f80d1c60 legacy=0x7f97f8108f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:10.194 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.190 7f97f685e700 1 -- 172.21.2.112:0/3867944157 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97f8088070 con 0x7f97f80d1860 2021-02-10T11:37:10.195 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.190 7f97f605d700 1 --2- 172.21.2.112:0/3867944157 >> v2:172.21.2.120:3300/0 conn(0x7f97f81b0ad0 0x7f97f8108930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:37:10.195 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.190 7f97f685e700 1 --2- 172.21.2.112:0/3867944157 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f97f80d1860 0x7f97f81b04b0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f97e0006650 tx=0x7f97e0009a80).ready entity=mon.2 client_cookie=15320304b800c5d7 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:10.195 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.191 7f97f585c700 1 -- 172.21.2.112:0/3867944157 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f97e00009b0 con 0x7f97f80d1860 2021-02-10T11:37:10.195 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.191 7f97f585c700 1 -- 172.21.2.112:0/3867944157 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97e000c0c0 con 0x7f97f80d1860 2021-02-10T11:37:10.195 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.191 7f97f585c700 1 -- 172.21.2.112:0/3867944157 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f97e000c5e0 con 0x7f97f80d1860 2021-02-10T11:37:10.195 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.191 7f97fcb85700 1 -- 172.21.2.112:0/3867944157 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f97f80d1860 msgr2=0x7f97f81b04b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:10.195 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.191 7f97fcb85700 1 --2- 172.21.2.112:0/3867944157 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f97f80d1860 0x7f97f81b04b0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f97e0006650 tx=0x7f97e0009a80).stop 2021-02-10T11:37:10.195 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.191 7f97fcb85700 1 -- 172.21.2.112:0/3867944157 shutdown_connections 2021-02-10T11:37:10.195 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.191 7f97fcb85700 1 --2- 172.21.2.112:0/3867944157 >> v2:172.21.2.120:3300/0 conn(0x7f97f81b0ad0 0x7f97f8108930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:10.195 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.191 7f97fcb85700 1 --2- 172.21.2.112:0/3867944157 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f97f80d1860 0x7f97f81b04b0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:10.195 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.191 7f97fcb85700 1 -- 172.21.2.112:0/3867944157 >> 172.21.2.112:0/3867944157 conn(0x7f97f80ce410 msgr2=0x7f97f80d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:10.195 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.192 7f97fcb85700 1 -- 172.21.2.112:0/3867944157 shutdown_connections 2021-02-10T11:37:10.195 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.192 7f97fcb85700 1 -- 172.21.2.112:0/3867944157 wait complete. 2021-02-10T11:37:10.197 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.206 7f97fcb85700 1 Processor -- start 2021-02-10T11:37:10.197 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.206 7f97fcb85700 1 -- start start 2021-02-10T11:37:10.197 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.206 7f97fcb85700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f97f8094d20 0x7f97f80a3e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:10.197 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.206 7f97fcb85700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f97f80a4460 0x7f97f80a4860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:10.197 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.206 7f97fcb85700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f97f80bf9e0 0x7f97f80bfde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:10.197 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.206 7f97fcb85700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97f80a7560 con 0x7f97f80a4460 2021-02-10T11:37:10.197 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.206 7f97fcb85700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97f80b0bc0 con 0x7f97f80bf9e0 2021-02-10T11:37:10.198 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.206 7f97fcb85700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97f8095120 con 0x7f97f8094d20 2021-02-10T11:37:10.198 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.207 7f97f605d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f97f80a4460 0x7f97f80a4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:10.198 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.207 7f97f605d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f97f80a4460 0x7f97f80a4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:52710/0 (socket says 172.21.2.112:52710) 2021-02-10T11:37:10.198 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.207 7f97f605d700 1 -- 172.21.2.112:0/277008482 learned_addr learned my addr 172.21.2.112:0/277008482 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:37:10.198 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.207 7f97f605d700 1 -- 172.21.2.112:0/277008482 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f97f8094d20 msgr2=0x7f97f80a3e40 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:37:10.198 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.207 7f97f605d700 1 --2- 172.21.2.112:0/277008482 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f97f8094d20 0x7f97f80a3e40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:10.198 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.207 7f97f605d700 1 -- 172.21.2.112:0/277008482 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f97f80bf9e0 msgr2=0x7f97f80bfde0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:37:10.198 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.207 7f97f605d700 1 --2- 172.21.2.112:0/277008482 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f97f80bf9e0 0x7f97f80bfde0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:10.198 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.207 7f97f605d700 1 -- 172.21.2.112:0/277008482 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97f80c0400 con 0x7f97f80a4460 2021-02-10T11:37:10.198 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.207 7f97f605d700 1 --2- 172.21.2.112:0/277008482 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f97f80a4460 0x7f97f80a4860 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 rx=0x7f97e8006f40 tx=0x7f97e800b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:10.198 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.207 7f97ef7fe700 1 -- 172.21.2.112:0/277008482 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f97e80009b0 con 0x7f97f80a4460 2021-02-10T11:37:10.198 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.207 7f97ef7fe700 1 -- 172.21.2.112:0/277008482 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97e800bd80 con 0x7f97f80a4460 2021-02-10T11:37:10.199 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.207 7f97fcb85700 1 -- 172.21.2.112:0/277008482 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97f80c0400 con 0x7f97f80a4460 2021-02-10T11:37:10.199 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.207 7f97fcb85700 1 -- 172.21.2.112:0/277008482 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97f80c0990 con 0x7f97f80a4460 2021-02-10T11:37:10.199 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.207 7f97ef7fe700 1 -- 172.21.2.112:0/277008482 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f97e8002220 con 0x7f97f80a4460 2021-02-10T11:37:10.199 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.208 7f97ef7fe700 1 -- 172.21.2.112:0/277008482 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f97e8000c90 con 0x7f97f80a4460 2021-02-10T11:37:10.199 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.208 7f97ef7fe700 1 --2- 172.21.2.112:0/277008482 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f97dc018730 0x7f97dc01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:10.199 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.208 7f97ef7fe700 1 -- 172.21.2.112:0/277008482 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f97e80327e0 con 0x7f97f80a4460 2021-02-10T11:37:10.199 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.208 7f97f685e700 1 --2- 172.21.2.112:0/277008482 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f97dc018730 0x7f97dc01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:10.199 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.209 7f97f685e700 1 --2- 172.21.2.112:0/277008482 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f97dc018730 0x7f97dc01ac10 crc :-1 s=READY pgs=822 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:10.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.239 7f97fcb85700 1 -- 172.21.2.112:0/277008482 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97f8082920 con 0x7f97f80a4460 2021-02-10T11:37:10.228 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.241 7f97ef7fe700 1 -- 172.21.2.112:0/277008482 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f97e8000940 con 0x7f97f80a4460 2021-02-10T11:37:10.357 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.369 7f97fcb85700 1 -- 172.21.2.112:0/277008482 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f97f8109a50 con 0x7f97f80a4460 2021-02-10T11:37:10.357 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.370 7f97ef7fe700 1 -- 172.21.2.112:0/277008482 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v83) v1 ==== 100+0+10 (secure 0 0 0) 0x7f97e8000c90 con 0x7f97f80a4460 2021-02-10T11:37:10.357 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:37:10.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.372 7f97fcb85700 1 -- 172.21.2.112:0/277008482 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f97dc018730 msgr2=0x7f97dc01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:10.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.372 7f97fcb85700 1 --2- 172.21.2.112:0/277008482 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f97dc018730 0x7f97dc01ac10 crc :-1 s=READY pgs=822 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:10.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.372 7f97fcb85700 1 -- 172.21.2.112:0/277008482 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f97f80a4460 msgr2=0x7f97f80a4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:10.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.372 7f97fcb85700 1 --2- 172.21.2.112:0/277008482 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f97f80a4460 0x7f97f80a4860 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 rx=0x7f97e8006f40 tx=0x7f97e800b990).stop 2021-02-10T11:37:10.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.372 7f97fcb85700 1 -- 172.21.2.112:0/277008482 shutdown_connections 2021-02-10T11:37:10.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.372 7f97fcb85700 1 --2- 172.21.2.112:0/277008482 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f97dc018730 0x7f97dc01ac10 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:10.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.372 7f97fcb85700 1 --2- 172.21.2.112:0/277008482 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f97f80bf9e0 0x7f97f80bfde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:10.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.372 7f97fcb85700 1 --2- 172.21.2.112:0/277008482 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f97f80a4460 0x7f97f80a4860 unknown :-1 s=CLOSED pgs=1738 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:10.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.372 7f97fcb85700 1 --2- 172.21.2.112:0/277008482 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f97f8094d20 0x7f97f80a3e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:10.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.372 7f97fcb85700 1 -- 172.21.2.112:0/277008482 >> 172.21.2.112:0/277008482 conn(0x7f97f80c3e40 msgr2=0x7f97f80916a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:10.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.372 7f97fcb85700 1 -- 172.21.2.112:0/277008482 shutdown_connections 2021-02-10T11:37:10.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.372 7f97fcb85700 1 -- 172.21.2.112:0/277008482 wait complete. 2021-02-10T11:37:10.368 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:37:10.368 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:37:10.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.792 7fec5cf7b700 1 Processor -- start 2021-02-10T11:37:10.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.792 7fec5cf7b700 1 -- start start 2021-02-10T11:37:10.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.793 7fec5cf7b700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fec580d1870 0x7fec581b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:10.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.793 7fec5cf7b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fec581b0a10 0x7fec580cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:10.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.793 7fec5cf7b700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec580f0b10 con 0x7fec580d1c70 2021-02-10T11:37:10.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.793 7fec5cf7b700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec580e5ed0 con 0x7fec580d1870 2021-02-10T11:37:10.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.793 7fec5cf7b700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec580f0650 con 0x7fec581b0a10 2021-02-10T11:37:10.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.794 7fec57314700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7fec580d1c70 0x7fec580cf950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:45742/0 (socket says 172.21.2.112:45742) 2021-02-10T11:37:10.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.794 7fec56312700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fec581b0a10 0x7fec580cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:10.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.794 7fec56b13700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fec580d1870 0x7fec581b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:10.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.794 7fec57314700 1 -- 172.21.2.112:0/3974230620 learned_addr learned my addr 172.21.2.112:0/3974230620 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:37:10.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.794 7fec56b13700 1 -- 172.21.2.112:0/3974230620 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fec581b0a10 msgr2=0x7fec580cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:37:10.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.794 7fec56b13700 1 --2- 172.21.2.112:0/3974230620 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fec581b0a10 0x7fec580cf330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:10.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.794 7fec56b13700 1 -- 172.21.2.112:0/3974230620 >> v1:172.21.2.120:6789/0 conn(0x7fec580d1c70 legacy=0x7fec580cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:10.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.794 7fec56b13700 1 -- 172.21.2.112:0/3974230620 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec580f0870 con 0x7fec580d1870 2021-02-10T11:37:10.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.794 7fec56312700 1 --2- 172.21.2.112:0/3974230620 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fec581b0a10 0x7fec580cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:37:10.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.795 7fec56b13700 1 --2- 172.21.2.112:0/3974230620 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fec580d1870 0x7fec581b03f0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 rx=0x7fec48006650 tx=0x7fec48009a80).ready entity=mon.1 client_cookie=ce81bce8152eb760 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:10.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.795 7fec55b11700 1 -- 172.21.2.112:0/3974230620 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fec480009b0 con 0x7fec580d1870 2021-02-10T11:37:10.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.795 7fec55b11700 1 -- 172.21.2.112:0/3974230620 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec4800c0c0 con 0x7fec580d1870 2021-02-10T11:37:10.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.795 7fec55b11700 1 -- 172.21.2.112:0/3974230620 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fec4800c5e0 con 0x7fec580d1870 2021-02-10T11:37:10.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.795 7fec5cf7b700 1 -- 172.21.2.112:0/3974230620 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fec580d1870 msgr2=0x7fec581b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:10.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.795 7fec5cf7b700 1 --2- 172.21.2.112:0/3974230620 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fec580d1870 0x7fec581b03f0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 rx=0x7fec48006650 tx=0x7fec48009a80).stop 2021-02-10T11:37:10.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.796 7fec5cf7b700 1 -- 172.21.2.112:0/3974230620 shutdown_connections 2021-02-10T11:37:10.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.796 7fec5cf7b700 1 --2- 172.21.2.112:0/3974230620 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fec581b0a10 0x7fec580cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:10.800 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.796 7fec5cf7b700 1 --2- 172.21.2.112:0/3974230620 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fec580d1870 0x7fec581b03f0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:10.800 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.796 7fec5cf7b700 1 -- 172.21.2.112:0/3974230620 >> 172.21.2.112:0/3974230620 conn(0x7fec580ce450 msgr2=0x7fec580d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:10.800 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.796 7fec5cf7b700 1 -- 172.21.2.112:0/3974230620 shutdown_connections 2021-02-10T11:37:10.800 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.796 7fec5cf7b700 1 -- 172.21.2.112:0/3974230620 wait complete. 2021-02-10T11:37:10.801 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.811 7fec5cf7b700 1 Processor -- start 2021-02-10T11:37:10.801 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.811 7fec5cf7b700 1 -- start start 2021-02-10T11:37:10.801 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.811 7fec5cf7b700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fec580ada90 0x7fec580ade90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:10.802 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.811 7fec5cf7b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fec58099d20 0x7fec5809a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:10.802 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.811 7fec5cf7b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fec5809a740 0x7fec58097bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:10.802 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.811 7fec5cf7b700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec580997a0 con 0x7fec58099d20 2021-02-10T11:37:10.802 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.811 7fec5cf7b700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec5818f0c0 con 0x7fec580ada90 2021-02-10T11:37:10.802 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.811 7fec5cf7b700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec5809ab40 con 0x7fec5809a740 2021-02-10T11:37:10.802 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.811 7fec56312700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fec58099d20 0x7fec5809a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:10.802 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.811 7fec56312700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fec58099d20 0x7fec5809a120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:52726/0 (socket says 172.21.2.112:52726) 2021-02-10T11:37:10.802 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.811 7fec56312700 1 -- 172.21.2.112:0/3560422162 learned_addr learned my addr 172.21.2.112:0/3560422162 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:37:10.802 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.811 7fec56b13700 1 --2- 172.21.2.112:0/3560422162 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fec580ada90 0x7fec580ade90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:10.802 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.811 7fec57314700 1 --2- 172.21.2.112:0/3560422162 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fec5809a740 0x7fec58097bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:10.802 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.812 7fec56b13700 1 -- 172.21.2.112:0/3560422162 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fec5809a740 msgr2=0x7fec58097bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:10.802 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.812 7fec56b13700 1 --2- 172.21.2.112:0/3560422162 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fec5809a740 0x7fec58097bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:10.803 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.812 7fec56b13700 1 -- 172.21.2.112:0/3560422162 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fec58099d20 msgr2=0x7fec5809a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:10.803 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.812 7fec56b13700 1 --2- 172.21.2.112:0/3560422162 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fec58099d20 0x7fec5809a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:10.803 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.812 7fec56b13700 1 -- 172.21.2.112:0/3560422162 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec5809ad60 con 0x7fec580ada90 2021-02-10T11:37:10.803 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.812 7fec56b13700 1 --2- 172.21.2.112:0/3560422162 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fec580ada90 0x7fec580ade90 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 rx=0x7fec4800b530 tx=0x7fec4800b8d0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:10.803 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.812 7fec4f7fe700 1 -- 172.21.2.112:0/3560422162 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fec4800bcc0 con 0x7fec580ada90 2021-02-10T11:37:10.803 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.812 7fec4f7fe700 1 -- 172.21.2.112:0/3560422162 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec480091c0 con 0x7fec580ada90 2021-02-10T11:37:10.803 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.812 7fec5cf7b700 1 -- 172.21.2.112:0/3560422162 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec5809ad60 con 0x7fec580ada90 2021-02-10T11:37:10.803 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.812 7fec4f7fe700 1 -- 172.21.2.112:0/3560422162 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fec480052e0 con 0x7fec580ada90 2021-02-10T11:37:10.803 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.812 7fec5cf7b700 1 -- 172.21.2.112:0/3560422162 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec5809ad60 con 0x7fec580ada90 2021-02-10T11:37:10.803 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.813 7fec4f7fe700 1 -- 172.21.2.112:0/3560422162 <== mon.1 v2:172.21.2.120:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fec480091c0 con 0x7fec580ada90 2021-02-10T11:37:10.803 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.813 7fec4f7fe700 1 --2- 172.21.2.112:0/3560422162 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fec3c0186c0 0x7fec3c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:10.803 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.813 7fec4f7fe700 1 -- 172.21.2.112:0/3560422162 <== mon.1 v2:172.21.2.120:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fec480322a0 con 0x7fec580ada90 2021-02-10T11:37:10.803 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.814 7fec56312700 1 --2- 172.21.2.112:0/3560422162 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fec3c0186c0 0x7fec3c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:10.803 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.814 7fec56312700 1 --2- 172.21.2.112:0/3560422162 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fec3c0186c0 0x7fec3c01abc0 crc :-1 s=READY pgs=823 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:10.830 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.843 7fec5cf7b700 1 -- 172.21.2.112:0/3560422162 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec580ebc00 con 0x7fec580ada90 2021-02-10T11:37:10.832 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.845 7fec4f7fe700 1 -- 172.21.2.112:0/3560422162 <== mon.1 v2:172.21.2.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fec480091c0 con 0x7fec580ada90 2021-02-10T11:37:10.960 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.973 7fec5cf7b700 1 -- 172.21.2.112:0/3560422162 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fec580ebc00 con 0x7fec580ada90 2021-02-10T11:37:10.961 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.973 7fec4f7fe700 1 -- 172.21.2.112:0/3560422162 <== mon.1 v2:172.21.2.120:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) v1 ==== 94+0+2101 (secure 0 0 0) 0x7fec4800bcc0 con 0x7fec580ada90 2021-02-10T11:37:10.961 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:37:10.962 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":64,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9204,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/1512510827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":1512510827},{"type":"v1","addr":"172.21.2.112:6833","nonce":1512510827}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":60}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:36:53.563735","modified":"2021-02-10 11:36:55.573740","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9201},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9201":{"gid":9201,"name":"b","rank":0,"incarnation":61,"state":"up:active","state_seq":3,"addr":"172.21.2.120:6835/1232001750","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":1232001750},{"type":"v1","addr":"172.21.2.120:6835","nonce":1232001750}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2021-02-10T11:37:10.962 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 64 2021-02-10T11:37:10.963 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.975 7fec5cf7b700 1 -- 172.21.2.112:0/3560422162 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fec3c0186c0 msgr2=0x7fec3c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:10.963 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.975 7fec5cf7b700 1 --2- 172.21.2.112:0/3560422162 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fec3c0186c0 0x7fec3c01abc0 crc :-1 s=READY pgs=823 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:10.963 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.975 7fec5cf7b700 1 -- 172.21.2.112:0/3560422162 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fec580ada90 msgr2=0x7fec580ade90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:10.963 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.975 7fec5cf7b700 1 --2- 172.21.2.112:0/3560422162 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fec580ada90 0x7fec580ade90 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 rx=0x7fec4800b530 tx=0x7fec4800b8d0).stop 2021-02-10T11:37:10.963 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.976 7fec5cf7b700 1 -- 172.21.2.112:0/3560422162 shutdown_connections 2021-02-10T11:37:10.963 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.976 7fec5cf7b700 1 --2- 172.21.2.112:0/3560422162 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fec3c0186c0 0x7fec3c01abc0 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:10.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.976 7fec5cf7b700 1 --2- 172.21.2.112:0/3560422162 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fec5809a740 0x7fec58097bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:10.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.976 7fec5cf7b700 1 --2- 172.21.2.112:0/3560422162 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fec58099d20 0x7fec5809a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:10.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.976 7fec5cf7b700 1 --2- 172.21.2.112:0/3560422162 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fec580ada90 0x7fec580ade90 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:10.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.976 7fec5cf7b700 1 -- 172.21.2.112:0/3560422162 >> 172.21.2.112:0/3560422162 conn(0x7fec580f0550 msgr2=0x7fec58191730 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:10.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.976 7fec5cf7b700 1 -- 172.21.2.112:0/3560422162 shutdown_connections 2021-02-10T11:37:10.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:10.976 7fec5cf7b700 1 -- 172.21.2.112:0/3560422162 wait complete. 2021-02-10T11:37:11.173 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 64, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:36:53.563735', 'modified': '2021-02-10 11:36:55.573740', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9201}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9201': {'gid': 9201, 'name': 'b', 'rank': 0, 'incarnation': 61, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.120:6835/1232001750', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.120:6834', 'nonce': 1232001750}, {'type': 'v1', 'addr': '172.21.2.120:6835', 'nonce': 1232001750}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:37:11.173 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:37:11.173 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:37:11.570 INFO:teuthology.orchestra.run.gibba020.stdout:{ 2021-02-10T11:37:11.570 INFO:teuthology.orchestra.run.gibba020.stdout: "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", 2021-02-10T11:37:11.570 INFO:teuthology.orchestra.run.gibba020.stdout: "whoami": 0, 2021-02-10T11:37:11.571 INFO:teuthology.orchestra.run.gibba020.stdout: "id": 9201, 2021-02-10T11:37:11.571 INFO:teuthology.orchestra.run.gibba020.stdout: "want_state": "up:active", 2021-02-10T11:37:11.571 INFO:teuthology.orchestra.run.gibba020.stdout: "state": "up:active", 2021-02-10T11:37:11.571 INFO:teuthology.orchestra.run.gibba020.stdout: "rank_uptime": 18.017592654000001, 2021-02-10T11:37:11.571 INFO:teuthology.orchestra.run.gibba020.stdout: "mdsmap_epoch": 64, 2021-02-10T11:37:11.571 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch": 83, 2021-02-10T11:37:11.571 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch_barrier": 83, 2021-02-10T11:37:11.571 INFO:teuthology.orchestra.run.gibba020.stdout: "uptime": 23.655360030000001 2021-02-10T11:37:11.571 INFO:teuthology.orchestra.run.gibba020.stdout:} 2021-02-10T11:37:11.574 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", "whoami": 0, "id": 9201, "want_state": "up:active", "state": "up:active", "rank_uptime": 18.017592654000001, "mdsmap_epoch": 64, "osdmap_epoch": 83, "osdmap_epoch_barrier": 83, "uptime": 23.655360030000001 } 2021-02-10T11:37:11.574 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:37:11.574 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2021-02-10T11:37:11.575 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.1 at ubuntu@gibba012.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2021-02-10T11:37:11.575 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:11.575 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:37:11.910 DEBUG:teuthology.orchestra.run.gibba012:> sudo modprobe fuse 2021-02-10T11:37:12.255 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:37:12.255 DEBUG:teuthology.orchestra.run.gibba012:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:37:12.597 INFO:teuthology.orchestra.run.gibba012.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:37:12.597 INFO:teuthology.orchestra.run.gibba012.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:37:12.598 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:37:12.598 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:12.598 DEBUG:teuthology.orchestra.run.gibba012:> ls /sys/fs/fuse/connections 2021-02-10T11:37:12.933 INFO:teuthology.orchestra.run.gibba012.stdout:42 2021-02-10T11:37:12.933 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [42] 2021-02-10T11:37:12.934 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.1 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:37:12.936 DEBUG:teuthology.orchestra.run.gibba012:> sudo modprobe fuse 2021-02-10T11:37:13.280 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:37:13.280 DEBUG:teuthology.orchestra.run.gibba012:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:37:13.346 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba012.stderr:ceph-fuse[145910]: starting ceph client 2021-02-10T11:37:13.346 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba012.stderr:2021-02-10 11:37:13.358 7f81e0787f80 -1 init, newargv = 0x561fcc64bd70 newargc=9 2021-02-10T11:37:13.352 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba012.stderr:ceph-fuse[145910]: starting fuse 2021-02-10T11:37:13.622 INFO:teuthology.orchestra.run.gibba012.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:37:13.622 INFO:teuthology.orchestra.run.gibba012.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:37:13.623 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:37:13.623 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:13.623 DEBUG:teuthology.orchestra.run.gibba012:> ls /sys/fs/fuse/connections 2021-02-10T11:37:13.959 INFO:teuthology.orchestra.run.gibba012.stdout:42 2021-02-10T11:37:13.959 INFO:teuthology.orchestra.run.gibba012.stdout:43 2021-02-10T11:37:13.959 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42, 43] 2021-02-10T11:37:13.959 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:13.959 DEBUG:teuthology.orchestra.run.gibba012:> sudo python3 -c ' 2021-02-10T11:37:13.960 DEBUG:teuthology.orchestra.run.gibba012:> import glob 2021-02-10T11:37:13.960 DEBUG:teuthology.orchestra.run.gibba012:> import re 2021-02-10T11:37:13.960 DEBUG:teuthology.orchestra.run.gibba012:> import os 2021-02-10T11:37:13.960 DEBUG:teuthology.orchestra.run.gibba012:> import subprocess 2021-02-10T11:37:13.960 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:37:13.960 DEBUG:teuthology.orchestra.run.gibba012:> def find_socket(client_name): 2021-02-10T11:37:13.960 DEBUG:teuthology.orchestra.run.gibba012:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2021-02-10T11:37:13.960 DEBUG:teuthology.orchestra.run.gibba012:> files = glob.glob(asok_path) 2021-02-10T11:37:13.960 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:37:13.960 DEBUG:teuthology.orchestra.run.gibba012:> # Given a non-glob path, it better be there 2021-02-10T11:37:13.960 DEBUG:teuthology.orchestra.run.gibba012:> if "*" not in asok_path: 2021-02-10T11:37:13.960 DEBUG:teuthology.orchestra.run.gibba012:> assert(len(files) == 1) 2021-02-10T11:37:13.960 DEBUG:teuthology.orchestra.run.gibba012:> return files[0] 2021-02-10T11:37:13.960 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:37:13.961 DEBUG:teuthology.orchestra.run.gibba012:> for f in files: 2021-02-10T11:37:13.961 DEBUG:teuthology.orchestra.run.gibba012:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:37:13.961 DEBUG:teuthology.orchestra.run.gibba012:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:37:13.961 DEBUG:teuthology.orchestra.run.gibba012:> return f 2021-02-10T11:37:13.961 DEBUG:teuthology.orchestra.run.gibba012:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:37:13.961 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:37:13.961 DEBUG:teuthology.orchestra.run.gibba012:> print(find_socket("client.1")) 2021-02-10T11:37:13.961 DEBUG:teuthology.orchestra.run.gibba012:> ' 2021-02-10T11:37:14.322 INFO:teuthology.orchestra.run.gibba012.stdout:/var/run/ceph/ceph-client.1.145910.asok 2021-02-10T11:37:14.325 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.145910.asok 2021-02-10T11:37:14.325 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:14.325 DEBUG:teuthology.orchestra.run.gibba012:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.145910.asok status 2021-02-10T11:37:14.713 INFO:teuthology.orchestra.run.gibba012.stdout:{ 2021-02-10T11:37:14.713 INFO:teuthology.orchestra.run.gibba012.stdout: "metadata": { 2021-02-10T11:37:14.714 INFO:teuthology.orchestra.run.gibba012.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:37:14.714 INFO:teuthology.orchestra.run.gibba012.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:37:14.714 INFO:teuthology.orchestra.run.gibba012.stdout: "entity_id": "1", 2021-02-10T11:37:14.714 INFO:teuthology.orchestra.run.gibba012.stdout: "hostname": "gibba012", 2021-02-10T11:37:14.714 INFO:teuthology.orchestra.run.gibba012.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2021-02-10T11:37:14.714 INFO:teuthology.orchestra.run.gibba012.stdout: "pid": "145910", 2021-02-10T11:37:14.714 INFO:teuthology.orchestra.run.gibba012.stdout: "root": "/" 2021-02-10T11:37:14.714 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:37:14.714 INFO:teuthology.orchestra.run.gibba012.stdout: "dentry_count": 0, 2021-02-10T11:37:14.714 INFO:teuthology.orchestra.run.gibba012.stdout: "dentry_pinned_count": 0, 2021-02-10T11:37:14.714 INFO:teuthology.orchestra.run.gibba012.stdout: "id": 9402, 2021-02-10T11:37:14.715 INFO:teuthology.orchestra.run.gibba012.stdout: "inst": { 2021-02-10T11:37:14.715 INFO:teuthology.orchestra.run.gibba012.stdout: "name": { 2021-02-10T11:37:14.715 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "client", 2021-02-10T11:37:14.715 INFO:teuthology.orchestra.run.gibba012.stdout: "num": 9402 2021-02-10T11:37:14.715 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:37:14.715 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": { 2021-02-10T11:37:14.715 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:37:14.715 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:0", 2021-02-10T11:37:14.715 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 1832264619 2021-02-10T11:37:14.715 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:37:14.715 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:37:14.715 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": { 2021-02-10T11:37:14.715 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:37:14.716 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:0", 2021-02-10T11:37:14.716 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 1832264619 2021-02-10T11:37:14.716 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:37:14.716 INFO:teuthology.orchestra.run.gibba012.stdout: "inst_str": "client.9402 172.21.2.112:0/1832264619", 2021-02-10T11:37:14.716 INFO:teuthology.orchestra.run.gibba012.stdout: "addr_str": "172.21.2.112:0/1832264619", 2021-02-10T11:37:14.716 INFO:teuthology.orchestra.run.gibba012.stdout: "inode_count": 1, 2021-02-10T11:37:14.716 INFO:teuthology.orchestra.run.gibba012.stdout: "mds_epoch": 64, 2021-02-10T11:37:14.716 INFO:teuthology.orchestra.run.gibba012.stdout: "osd_epoch": 83, 2021-02-10T11:37:14.716 INFO:teuthology.orchestra.run.gibba012.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:37:14.716 INFO:teuthology.orchestra.run.gibba012.stdout: "blacklisted": false 2021-02-10T11:37:14.716 INFO:teuthology.orchestra.run.gibba012.stdout:} 2021-02-10T11:37:14.717 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:14.718 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:37:14.718 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:37:15.052 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:37:15.053 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:37:15.053 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:15.053 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:37:15.399 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2021-02-10T11:37:15.831 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:15.827 7fa35e3cc700 1 Processor -- start 2021-02-10T11:37:15.831 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:15.827 7fa35e3cc700 1 -- start start 2021-02-10T11:37:15.831 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:15.827 7fa35e3cc700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fa3580d1870 0x7fa3581b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:15.832 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:15.827 7fa35e3cc700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fa3580cff70 0x7fa358109130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:15.832 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:15.827 7fa35e3cc700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3580fe1c0 con 0x7fa3581b0ae0 2021-02-10T11:37:15.832 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:15.828 7fa35e3cc700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa358109ad0 con 0x7fa3580cff70 2021-02-10T11:37:15.832 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:15.828 7fa35e3cc700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa358109750 con 0x7fa3580d1870 2021-02-10T11:37:15.832 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:15.828 7fa35c9e9700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fa3580cff70 0x7fa358109130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:15.832 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:15.828 7fa35c9e9700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fa3580cff70 0x7fa358109130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:52744/0 (socket says 172.21.2.112:52744) 2021-02-10T11:37:15.832 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:15.828 7fa35c9e9700 1 -- 172.21.2.112:0/766674730 learned_addr learned my addr 172.21.2.112:0/766674730 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:37:15.832 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:15.828 7fa357fff700 1 --2- 172.21.2.112:0/766674730 >> v2:172.21.2.120:3300/0 conn(0x7fa3580d1870 0x7fa3581b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:15.832 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:15.828 7fa35c9e9700 1 -- 172.21.2.112:0/766674730 >> v2:172.21.2.120:3300/0 conn(0x7fa3580d1870 msgr2=0x7fa3581b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:37:15.832 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:15.828 7fa35c9e9700 1 --2- 172.21.2.112:0/766674730 >> v2:172.21.2.120:3300/0 conn(0x7fa3580d1870 0x7fa3581b04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:15.832 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:15.828 7fa35c9e9700 1 -- 172.21.2.112:0/766674730 >> v1:172.21.2.120:6789/0 conn(0x7fa3581b0ae0 legacy=0x7fa358108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:15.832 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:15.828 7fa35c9e9700 1 -- 172.21.2.112:0/766674730 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa358088070 con 0x7fa3580cff70 2021-02-10T11:37:15.832 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:15.828 7fa35c9e9700 1 --2- 172.21.2.112:0/766674730 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa3580cff70 0x7fa358109130 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 rx=0x7fa350006f00 tx=0x7fa350009f10).ready entity=mon.0 client_cookie=11aee9c99e35186 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:15.833 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:15.829 7fa356ffd700 1 -- 172.21.2.112:0/766674730 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa35000b3b0 con 0x7fa3580cff70 2021-02-10T11:37:15.833 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:15.829 7fa356ffd700 1 -- 172.21.2.112:0/766674730 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa35000b730 con 0x7fa3580cff70 2021-02-10T11:37:15.833 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:15.829 7fa356ffd700 1 -- 172.21.2.112:0/766674730 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa35000bc50 con 0x7fa3580cff70 2021-02-10T11:37:15.833 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:15.829 7fa35e3cc700 1 -- 172.21.2.112:0/766674730 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa3580cff70 msgr2=0x7fa358109130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:15.833 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:15.829 7fa35e3cc700 1 --2- 172.21.2.112:0/766674730 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa3580cff70 0x7fa358109130 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 rx=0x7fa350006f00 tx=0x7fa350009f10).stop 2021-02-10T11:37:15.833 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:15.829 7fa35e3cc700 1 -- 172.21.2.112:0/766674730 shutdown_connections 2021-02-10T11:37:15.833 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:15.829 7fa35e3cc700 1 --2- 172.21.2.112:0/766674730 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa3580cff70 0x7fa358109130 unknown :-1 s=CLOSED pgs=1741 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:15.833 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:15.829 7fa35e3cc700 1 --2- 172.21.2.112:0/766674730 >> v2:172.21.2.120:3300/0 conn(0x7fa3580d1870 0x7fa3581b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:15.833 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:15.829 7fa35e3cc700 1 -- 172.21.2.112:0/766674730 >> 172.21.2.112:0/766674730 conn(0x7fa3580ce420 msgr2=0x7fa3580d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:15.833 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:15.829 7fa35e3cc700 1 -- 172.21.2.112:0/766674730 shutdown_connections 2021-02-10T11:37:15.833 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:15.829 7fa35e3cc700 1 -- 172.21.2.112:0/766674730 wait complete. 2021-02-10T11:37:15.833 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:15.844 7fa35e3cc700 1 Processor -- start 2021-02-10T11:37:15.833 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:15.844 7fa35e3cc700 1 -- start start 2021-02-10T11:37:15.835 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:15.844 7fa35e3cc700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa3581a05f0 0x7fa3580a8f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:15.835 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:15.844 7fa35e3cc700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa3580aba50 0x7fa3580b3090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:15.835 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:15.844 7fa35e3cc700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa3580b36b0 0x7fa3580b42a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:15.836 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:15.844 7fa35e3cc700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3580b3d20 con 0x7fa3580aba50 2021-02-10T11:37:15.836 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:15.844 7fa35e3cc700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3580abe50 con 0x7fa3580b36b0 2021-02-10T11:37:15.836 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:15.844 7fa35e3cc700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3580b23c0 con 0x7fa3581a05f0 2021-02-10T11:37:15.836 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:15.844 7fa3577fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa3580aba50 0x7fa3580b3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:15.836 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:15.844 7fa3577fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa3580aba50 0x7fa3580b3090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:52750/0 (socket says 172.21.2.112:52750) 2021-02-10T11:37:15.836 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:15.845 7fa3577fe700 1 -- 172.21.2.112:0/3367572966 learned_addr learned my addr 172.21.2.112:0/3367572966 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:37:15.836 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:15.845 7fa35c9e9700 1 --2- 172.21.2.112:0/3367572966 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa3580b36b0 0x7fa3580b42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:15.836 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:15.845 7fa357fff700 1 --2- 172.21.2.112:0/3367572966 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa3581a05f0 0x7fa3580a8f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:15.836 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:15.845 7fa3577fe700 1 -- 172.21.2.112:0/3367572966 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa3581a05f0 msgr2=0x7fa3580a8f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:15.836 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:15.845 7fa3577fe700 1 --2- 172.21.2.112:0/3367572966 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa3581a05f0 0x7fa3580a8f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:15.836 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:15.845 7fa3577fe700 1 -- 172.21.2.112:0/3367572966 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa3580b36b0 msgr2=0x7fa3580b42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:15.836 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:15.845 7fa3577fe700 1 --2- 172.21.2.112:0/3367572966 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa3580b36b0 0x7fa3580b42a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:15.837 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:15.845 7fa3577fe700 1 -- 172.21.2.112:0/3367572966 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3580b48c0 con 0x7fa3580aba50 2021-02-10T11:37:15.837 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:15.845 7fa35c9e9700 1 --2- 172.21.2.112:0/3367572966 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa3580b36b0 0x7fa3580b42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:37:15.837 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:15.845 7fa3577fe700 1 --2- 172.21.2.112:0/3367572966 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa3580aba50 0x7fa3580b3090 secure :-1 s=READY pgs=1742 cs=0 l=1 rev1=1 rx=0x7fa34c006ed0 tx=0x7fa34c00b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:15.837 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:15.845 7fa354ff9700 1 -- 172.21.2.112:0/3367572966 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa34c006240 con 0x7fa3580aba50 2021-02-10T11:37:15.837 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:15.845 7fa354ff9700 1 -- 172.21.2.112:0/3367572966 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa34c0054b0 con 0x7fa3580aba50 2021-02-10T11:37:15.837 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:15.845 7fa354ff9700 1 -- 172.21.2.112:0/3367572966 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa34c005950 con 0x7fa3580aba50 2021-02-10T11:37:15.837 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:15.845 7fa35e3cc700 1 -- 172.21.2.112:0/3367572966 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3580b48c0 con 0x7fa3580aba50 2021-02-10T11:37:15.837 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:15.845 7fa35e3cc700 1 -- 172.21.2.112:0/3367572966 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3580b48c0 con 0x7fa3580aba50 2021-02-10T11:37:15.837 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:15.846 7fa354ff9700 1 -- 172.21.2.112:0/3367572966 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa34c006030 con 0x7fa3580aba50 2021-02-10T11:37:15.837 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:15.846 7fa354ff9700 1 --2- 172.21.2.112:0/3367572966 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa3380186c0 0x7fa33801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:15.837 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:15.846 7fa354ff9700 1 -- 172.21.2.112:0/3367572966 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fa34c032790 con 0x7fa3580aba50 2021-02-10T11:37:15.837 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:15.846 7fa357fff700 1 --2- 172.21.2.112:0/3367572966 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa3380186c0 0x7fa33801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:15.837 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:15.847 7fa357fff700 1 --2- 172.21.2.112:0/3367572966 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa3380186c0 0x7fa33801abc0 crc :-1 s=READY pgs=824 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:15.865 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:15.877 7fa35e3cc700 1 -- 172.21.2.112:0/3367572966 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3580825d0 con 0x7fa3580aba50 2021-02-10T11:37:15.867 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:15.880 7fa354ff9700 1 -- 172.21.2.112:0/3367572966 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa34c0009b0 con 0x7fa3580aba50 2021-02-10T11:37:16.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:16.016 7fa35e3cc700 1 -- 172.21.2.112:0/3367572966 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fa3580825d0 con 0x7fa3580aba50 2021-02-10T11:37:16.058 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:16.070 7fa354ff9700 1 -- 172.21.2.112:0/3367572966 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v65) v1 ==== 136+0+0 (secure 0 0 0) 0x7fa34c006030 con 0x7fa3580aba50 2021-02-10T11:37:16.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:16.082 7fa35e3cc700 1 -- 172.21.2.112:0/3367572966 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa3380186c0 msgr2=0x7fa33801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:16.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:16.082 7fa35e3cc700 1 --2- 172.21.2.112:0/3367572966 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa3380186c0 0x7fa33801abc0 crc :-1 s=READY pgs=824 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:16.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:16.082 7fa35e3cc700 1 -- 172.21.2.112:0/3367572966 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa3580aba50 msgr2=0x7fa3580b3090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:16.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:16.082 7fa35e3cc700 1 --2- 172.21.2.112:0/3367572966 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa3580aba50 0x7fa3580b3090 secure :-1 s=READY pgs=1742 cs=0 l=1 rev1=1 rx=0x7fa34c006ed0 tx=0x7fa34c00b960).stop 2021-02-10T11:37:16.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:16.083 7fa35e3cc700 1 -- 172.21.2.112:0/3367572966 shutdown_connections 2021-02-10T11:37:16.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:16.083 7fa35e3cc700 1 --2- 172.21.2.112:0/3367572966 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa3380186c0 0x7fa33801abc0 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:16.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:16.083 7fa35e3cc700 1 --2- 172.21.2.112:0/3367572966 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa3580b36b0 0x7fa3580b42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:16.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:16.083 7fa35e3cc700 1 --2- 172.21.2.112:0/3367572966 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa3580aba50 0x7fa3580b3090 unknown :-1 s=CLOSED pgs=1742 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:16.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:16.083 7fa35e3cc700 1 --2- 172.21.2.112:0/3367572966 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa3581a05f0 0x7fa3580a8f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:16.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:16.083 7fa35e3cc700 1 -- 172.21.2.112:0/3367572966 >> 172.21.2.112:0/3367572966 conn(0x7fa3580bc070 msgr2=0x7fa3580ba770 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:16.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:16.083 7fa35e3cc700 1 -- 172.21.2.112:0/3367572966 shutdown_connections 2021-02-10T11:37:16.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:16.083 7fa35e3cc700 1 -- 172.21.2.112:0/3367572966 wait complete. 2021-02-10T11:37:16.079 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs volume ls 2021-02-10T11:37:16.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:16.502 7f60e9faf700 1 Processor -- start 2021-02-10T11:37:16.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:16.502 7f60e9faf700 1 -- start start 2021-02-10T11:37:16.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:16.502 7f60e9faf700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f60e40d1870 0x7f60e41b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:16.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:16.502 7f60e9faf700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f60e41b0a10 0x7f60e40cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:16.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:16.502 7f60e9faf700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60e40f0b10 con 0x7f60e40d1c70 2021-02-10T11:37:16.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:16.502 7f60e9faf700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60e40fe230 con 0x7f60e41b0a10 2021-02-10T11:37:16.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:16.502 7f60e9faf700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60e40e5ed0 con 0x7f60e40d1870 2021-02-10T11:37:16.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:16.503 7f60e2ffd700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f60e41b0a10 0x7f60e40cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:16.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:16.503 7f60e37fe700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f60e40d1870 0x7f60e41b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:16.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:16.503 7f60e3fff700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7f60e40d1c70 0x7f60e40cf950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:45778/0 (socket says 172.21.2.112:45778) 2021-02-10T11:37:16.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:16.503 7f60e3fff700 1 -- 172.21.2.112:0/3087652065 learned_addr learned my addr 172.21.2.112:0/3087652065 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:37:16.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:16.503 7f60e2ffd700 1 -- 172.21.2.112:0/3087652065 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f60e40d1870 msgr2=0x7f60e41b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:37:16.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:16.503 7f60e2ffd700 1 --2- 172.21.2.112:0/3087652065 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f60e40d1870 0x7f60e41b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:16.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:16.503 7f60e2ffd700 1 -- 172.21.2.112:0/3087652065 >> v1:172.21.2.120:6789/0 conn(0x7f60e40d1c70 legacy=0x7f60e40cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:16.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:16.503 7f60e2ffd700 1 -- 172.21.2.112:0/3087652065 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60e40f0650 con 0x7f60e41b0a10 2021-02-10T11:37:16.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:16.503 7f60e37fe700 1 --2- 172.21.2.112:0/3087652065 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f60e40d1870 0x7f60e41b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:37:16.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:16.503 7f60e2ffd700 1 --2- 172.21.2.112:0/3087652065 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f60e41b0a10 0x7f60e40cf330 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 rx=0x7f60d4006650 tx=0x7f60d4009a80).ready entity=mon.1 client_cookie=88565624970beb71 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:16.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:16.504 7f60e27fc700 1 -- 172.21.2.112:0/3087652065 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f60d400c0c0 con 0x7f60e41b0a10 2021-02-10T11:37:16.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:16.504 7f60e27fc700 1 -- 172.21.2.112:0/3087652065 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60d400c340 con 0x7f60e41b0a10 2021-02-10T11:37:16.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:16.504 7f60e27fc700 1 -- 172.21.2.112:0/3087652065 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f60d400c860 con 0x7f60e41b0a10 2021-02-10T11:37:16.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:16.504 7f60e9faf700 1 -- 172.21.2.112:0/3087652065 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f60e41b0a10 msgr2=0x7f60e40cf330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:16.508 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:16.504 7f60e9faf700 1 --2- 172.21.2.112:0/3087652065 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f60e41b0a10 0x7f60e40cf330 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 rx=0x7f60d4006650 tx=0x7f60d4009a80).stop 2021-02-10T11:37:16.508 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:16.504 7f60e9faf700 1 -- 172.21.2.112:0/3087652065 shutdown_connections 2021-02-10T11:37:16.508 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:16.504 7f60e9faf700 1 --2- 172.21.2.112:0/3087652065 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f60e41b0a10 0x7f60e40cf330 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:16.508 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:16.504 7f60e9faf700 1 --2- 172.21.2.112:0/3087652065 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f60e40d1870 0x7f60e41b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:16.508 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:16.504 7f60e9faf700 1 -- 172.21.2.112:0/3087652065 >> 172.21.2.112:0/3087652065 conn(0x7f60e40ce450 msgr2=0x7f60e40d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:16.508 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:16.504 7f60e9faf700 1 -- 172.21.2.112:0/3087652065 shutdown_connections 2021-02-10T11:37:16.508 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:16.504 7f60e9faf700 1 -- 172.21.2.112:0/3087652065 wait complete. 2021-02-10T11:37:16.510 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:16.519 7f60e9faf700 1 Processor -- start 2021-02-10T11:37:16.510 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:16.519 7f60e9faf700 1 -- start start 2021-02-10T11:37:16.510 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:16.519 7f60e9faf700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f60e4189d00 0x7f60e418a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:16.510 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:16.519 7f60e9faf700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f60e418a720 0x7f60e418ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:16.510 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:16.519 7f60e9faf700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f60e4187e50 0x7f60e4188250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:16.510 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:16.519 7f60e9faf700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60e40c37e0 con 0x7f60e418a720 2021-02-10T11:37:16.510 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:16.519 7f60e9faf700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60e40c39d0 con 0x7f60e4189d00 2021-02-10T11:37:16.510 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:16.519 7f60e9faf700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60e4188870 con 0x7f60e4187e50 2021-02-10T11:37:16.510 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:16.519 7f60e2ffd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f60e418a720 0x7f60e418ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:16.511 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:16.519 7f60e2ffd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f60e418a720 0x7f60e418ab20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:52760/0 (socket says 172.21.2.112:52760) 2021-02-10T11:37:16.511 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:16.519 7f60e2ffd700 1 -- 172.21.2.112:0/2434535027 learned_addr learned my addr 172.21.2.112:0/2434535027 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:37:16.511 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:16.519 7f60e3fff700 1 --2- 172.21.2.112:0/2434535027 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f60e4187e50 0x7f60e4188250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:16.511 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:16.519 7f60e37fe700 1 --2- 172.21.2.112:0/2434535027 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f60e4189d00 0x7f60e418a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:16.511 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:16.520 7f60e2ffd700 1 -- 172.21.2.112:0/2434535027 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f60e4187e50 msgr2=0x7f60e4188250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:16.511 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:16.520 7f60e2ffd700 1 --2- 172.21.2.112:0/2434535027 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f60e4187e50 0x7f60e4188250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:16.511 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:16.520 7f60e2ffd700 1 -- 172.21.2.112:0/2434535027 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f60e4189d00 msgr2=0x7f60e418a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:16.511 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:16.520 7f60e2ffd700 1 --2- 172.21.2.112:0/2434535027 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f60e4189d00 0x7f60e418a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:16.511 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:16.520 7f60e2ffd700 1 -- 172.21.2.112:0/2434535027 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60e4188a90 con 0x7f60e418a720 2021-02-10T11:37:16.511 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:16.520 7f60e2ffd700 1 --2- 172.21.2.112:0/2434535027 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f60e418a720 0x7f60e418ab20 secure :-1 s=READY pgs=1743 cs=0 l=1 rev1=1 rx=0x7f60d4009630 tx=0x7f60d400b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:16.511 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:16.520 7f60dbfff700 1 -- 172.21.2.112:0/2434535027 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f60d40093a0 con 0x7f60e418a720 2021-02-10T11:37:16.512 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:16.520 7f60dbfff700 1 -- 172.21.2.112:0/2434535027 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60d400bd70 con 0x7f60e418a720 2021-02-10T11:37:16.512 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:16.520 7f60dbfff700 1 -- 172.21.2.112:0/2434535027 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f60d4005560 con 0x7f60e418a720 2021-02-10T11:37:16.512 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:16.520 7f60e9faf700 1 -- 172.21.2.112:0/2434535027 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60e4188a90 con 0x7f60e418a720 2021-02-10T11:37:16.512 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:16.520 7f60e9faf700 1 -- 172.21.2.112:0/2434535027 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60e4189020 con 0x7f60e418a720 2021-02-10T11:37:16.512 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:16.521 7f60dbfff700 1 -- 172.21.2.112:0/2434535027 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f60d400a000 con 0x7f60e418a720 2021-02-10T11:37:16.512 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:16.521 7f60dbfff700 1 --2- 172.21.2.112:0/2434535027 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f60c80186c0 0x7f60c801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:16.512 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:16.521 7f60dbfff700 1 -- 172.21.2.112:0/2434535027 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f60d40325e0 con 0x7f60e418a720 2021-02-10T11:37:16.512 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:16.521 7f60e37fe700 1 --2- 172.21.2.112:0/2434535027 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f60c80186c0 0x7f60c801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:16.512 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:16.522 7f60e37fe700 1 --2- 172.21.2.112:0/2434535027 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f60c80186c0 0x7f60c801abc0 crc :-1 s=READY pgs=825 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:16.539 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:16.552 7f60e9faf700 1 -- 172.21.2.112:0/2434535027 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60e40ebc50 con 0x7f60e418a720 2021-02-10T11:37:16.542 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:16.554 7f60dbfff700 1 -- 172.21.2.112:0/2434535027 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f60d4009340 con 0x7f60e418a720 2021-02-10T11:37:16.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:16.688 7f60e9faf700 1 -- 172.21.2.112:0/2434535027 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs volume ls", "target": ["mgr", ""]}) v1 -- 0x7f60e4189020 con 0x7f60c80186c0 2021-02-10T11:37:16.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:16.690 7f60dbfff700 1 -- 172.21.2.112:0/2434535027 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+41 (crc 0 0 0) 0x7f60cc006a30 con 0x7f60c80186c0 2021-02-10T11:37:16.679 INFO:teuthology.orchestra.run.gibba012.stdout:[ 2021-02-10T11:37:16.679 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:37:16.679 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "cephfs" 2021-02-10T11:37:16.679 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:37:16.680 INFO:teuthology.orchestra.run.gibba012.stdout:] 2021-02-10T11:37:16.681 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:16.693 7f60e9faf700 1 -- 172.21.2.112:0/2434535027 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f60c80186c0 msgr2=0x7f60c801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:16.681 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:16.693 7f60e9faf700 1 --2- 172.21.2.112:0/2434535027 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f60c80186c0 0x7f60c801abc0 crc :-1 s=READY pgs=825 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:16.681 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:16.693 7f60e9faf700 1 -- 172.21.2.112:0/2434535027 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f60e418a720 msgr2=0x7f60e418ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:16.681 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:16.693 7f60e9faf700 1 --2- 172.21.2.112:0/2434535027 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f60e418a720 0x7f60e418ab20 secure :-1 s=READY pgs=1743 cs=0 l=1 rev1=1 rx=0x7f60d4009630 tx=0x7f60d400b980).stop 2021-02-10T11:37:16.681 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:16.693 7f60e9faf700 1 -- 172.21.2.112:0/2434535027 shutdown_connections 2021-02-10T11:37:16.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:16.693 7f60e9faf700 1 --2- 172.21.2.112:0/2434535027 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f60c80186c0 0x7f60c801abc0 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:16.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:16.693 7f60e9faf700 1 --2- 172.21.2.112:0/2434535027 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f60e4187e50 0x7f60e4188250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:16.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:16.693 7f60e9faf700 1 --2- 172.21.2.112:0/2434535027 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f60e418a720 0x7f60e418ab20 unknown :-1 s=CLOSED pgs=1743 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:16.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:16.693 7f60e9faf700 1 --2- 172.21.2.112:0/2434535027 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f60e4189d00 0x7f60e418a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:16.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:16.693 7f60e9faf700 1 -- 172.21.2.112:0/2434535027 >> 172.21.2.112:0/2434535027 conn(0x7f60e40aed50 msgr2=0x7f60e409c9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:16.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:16.694 7f60e9faf700 1 -- 172.21.2.112:0/2434535027 shutdown_connections 2021-02-10T11:37:16.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:16.694 7f60e9faf700 1 -- 172.21.2.112:0/2434535027 wait complete. 2021-02-10T11:37:16.690 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:37:17.120 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:17.116 7f41157ff700 1 Processor -- start 2021-02-10T11:37:17.120 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:17.116 7f41157ff700 1 -- start start 2021-02-10T11:37:17.120 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:17.117 7f41157ff700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f41100d1c70 0x7f41100cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:17.121 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:17.117 7f41157ff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f41100cf580 0x7f41100cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:17.121 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:17.117 7f41157ff700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41100fe1c0 con 0x7f41100d1870 2021-02-10T11:37:17.121 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:17.117 7f41157ff700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41101b0cb0 con 0x7f41100cf580 2021-02-10T11:37:17.121 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:17.117 7f41157ff700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4110109a60 con 0x7f41100d1c70 2021-02-10T11:37:17.121 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:17.117 7f410fd75700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f41100cf580 0x7f41100cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:17.121 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:17.117 7f410fd75700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f41100cf580 0x7f41100cf980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:52770/0 (socket says 172.21.2.112:52770) 2021-02-10T11:37:17.121 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:17.117 7f410fd75700 1 -- 172.21.2.112:0/3449763007 learned_addr learned my addr 172.21.2.112:0/3449763007 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:37:17.121 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:17.117 7f410fd75700 1 -- 172.21.2.112:0/3449763007 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f41100d1c70 msgr2=0x7f41100cef60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:37:17.121 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:17.117 7f410fd75700 1 --2- 172.21.2.112:0/3449763007 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f41100d1c70 0x7f41100cef60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:17.121 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:17.117 7f410fd75700 1 -- 172.21.2.112:0/3449763007 >> v1:172.21.2.120:6789/0 conn(0x7f41100d1870 legacy=0x7f41101b04c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:37:17.121 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:17.117 7f410fd75700 1 -- 172.21.2.112:0/3449763007 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4110088070 con 0x7f41100cf580 2021-02-10T11:37:17.121 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:17.118 7f410fd75700 1 --2- 172.21.2.112:0/3449763007 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f41100cf580 0x7f41100cf980 secure :-1 s=READY pgs=1744 cs=0 l=1 rev1=1 rx=0x7f4108006f00 tx=0x7f4108009f10).ready entity=mon.0 client_cookie=8f7ad27dd339b124 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:17.121 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:17.118 7f410e572700 1 -- 172.21.2.112:0/3449763007 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f410800b3b0 con 0x7f41100cf580 2021-02-10T11:37:17.122 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:17.118 7f410e572700 1 -- 172.21.2.112:0/3449763007 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f410800b630 con 0x7f41100cf580 2021-02-10T11:37:17.122 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:17.118 7f410e572700 1 -- 172.21.2.112:0/3449763007 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f410800bb50 con 0x7f41100cf580 2021-02-10T11:37:17.122 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:17.118 7f41157ff700 1 -- 172.21.2.112:0/3449763007 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f41100cf580 msgr2=0x7f41100cf980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:17.122 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:17.118 7f41157ff700 1 --2- 172.21.2.112:0/3449763007 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f41100cf580 0x7f41100cf980 secure :-1 s=READY pgs=1744 cs=0 l=1 rev1=1 rx=0x7f4108006f00 tx=0x7f4108009f10).stop 2021-02-10T11:37:17.122 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:17.119 7f41157ff700 1 -- 172.21.2.112:0/3449763007 shutdown_connections 2021-02-10T11:37:17.122 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:17.119 7f41157ff700 1 --2- 172.21.2.112:0/3449763007 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f41100cf580 0x7f41100cf980 unknown :-1 s=CLOSED pgs=1744 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:17.122 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:17.119 7f41157ff700 1 --2- 172.21.2.112:0/3449763007 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f41100d1c70 0x7f41100cef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:17.122 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:17.119 7f41157ff700 1 -- 172.21.2.112:0/3449763007 >> 172.21.2.112:0/3449763007 conn(0x7f41100ce420 msgr2=0x7f41100d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:17.122 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:17.119 7f41157ff700 1 -- 172.21.2.112:0/3449763007 shutdown_connections 2021-02-10T11:37:17.122 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:17.119 7f41157ff700 1 -- 172.21.2.112:0/3449763007 wait complete. 2021-02-10T11:37:17.124 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:17.133 7f41157ff700 1 Processor -- start 2021-02-10T11:37:17.124 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:17.133 7f41157ff700 1 -- start start 2021-02-10T11:37:17.124 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:17.134 7f41157ff700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f41100c2370 0x7f411018f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:17.124 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:17.134 7f41157ff700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4110084cb0 0x7f41101a7470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:17.124 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:17.134 7f41157ff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f41101a7a90 0x7f4110085700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:17.124 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:17.134 7f41157ff700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41100be840 con 0x7f41101a7a90 2021-02-10T11:37:17.124 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:17.134 7f41157ff700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41100c29a0 con 0x7f4110084cb0 2021-02-10T11:37:17.124 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:17.134 7f41157ff700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41101a7e90 con 0x7f41100c2370 2021-02-10T11:37:17.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:17.134 7f410fd75700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f41101a7a90 0x7f4110085700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:17.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:17.134 7f410fd75700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f41101a7a90 0x7f4110085700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:52774/0 (socket says 172.21.2.112:52774) 2021-02-10T11:37:17.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:17.134 7f410fd75700 1 -- 172.21.2.112:0/764287486 learned_addr learned my addr 172.21.2.112:0/764287486 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:37:17.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:17.134 7f410fd75700 1 -- 172.21.2.112:0/764287486 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f41100c2370 msgr2=0x7f411018f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:17.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:17.134 7f410fd75700 1 --2- 172.21.2.112:0/764287486 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f41100c2370 0x7f411018f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:17.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:17.134 7f410fd75700 1 -- 172.21.2.112:0/764287486 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4110084cb0 msgr2=0x7f41101a7470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:17.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:17.134 7f410fd75700 1 --2- 172.21.2.112:0/764287486 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4110084cb0 0x7f41101a7470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:17.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:17.134 7f410fd75700 1 -- 172.21.2.112:0/764287486 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41101a80b0 con 0x7f41101a7a90 2021-02-10T11:37:17.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:17.134 7f410fd75700 1 --2- 172.21.2.112:0/764287486 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f41101a7a90 0x7f4110085700 secure :-1 s=READY pgs=1745 cs=0 l=1 rev1=1 rx=0x7f4108009390 tx=0x7f410800bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:17.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:17.135 7f4107fff700 1 -- 172.21.2.112:0/764287486 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4108003030 con 0x7f41101a7a90 2021-02-10T11:37:17.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:17.135 7f4107fff700 1 -- 172.21.2.112:0/764287486 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41080068e0 con 0x7f41101a7a90 2021-02-10T11:37:17.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:17.135 7f4107fff700 1 -- 172.21.2.112:0/764287486 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f410800c570 con 0x7f41101a7a90 2021-02-10T11:37:17.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:17.135 7f41157ff700 1 -- 172.21.2.112:0/764287486 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41101a80b0 con 0x7f41101a7a90 2021-02-10T11:37:17.126 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:17.135 7f41157ff700 1 -- 172.21.2.112:0/764287486 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4110086010 con 0x7f41101a7a90 2021-02-10T11:37:17.126 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:17.135 7f4107fff700 1 -- 172.21.2.112:0/764287486 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4108005820 con 0x7f41101a7a90 2021-02-10T11:37:17.126 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:17.135 7f4107fff700 1 --2- 172.21.2.112:0/764287486 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f40f40186c0 0x7f40f401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:17.126 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:17.135 7f4107fff700 1 -- 172.21.2.112:0/764287486 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f41080328e0 con 0x7f41101a7a90 2021-02-10T11:37:17.126 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:17.136 7f410f574700 1 --2- 172.21.2.112:0/764287486 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f40f40186c0 0x7f40f401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:17.126 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:17.136 7f410f574700 1 --2- 172.21.2.112:0/764287486 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f40f40186c0 0x7f40f401abc0 crc :-1 s=READY pgs=826 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:17.154 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:17.167 7f41157ff700 1 -- 172.21.2.112:0/764287486 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4110082a70 con 0x7f41101a7a90 2021-02-10T11:37:17.157 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:17.169 7f4107fff700 1 -- 172.21.2.112:0/764287486 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4108009450 con 0x7f41101a7a90 2021-02-10T11:37:17.308 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:17.320 7f41157ff700 1 -- 172.21.2.112:0/764287486 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f4110082a70 con 0x7f41101a7a90 2021-02-10T11:37:17.308 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:17.320 7f4107fff700 1 -- 172.21.2.112:0/764287486 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v22)=0 v22) v1 ==== 122+0+0 (secure 0 0 0) 0x7f4108005820 con 0x7f41101a7a90 2021-02-10T11:37:17.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:17.322 7f41157ff700 1 -- 172.21.2.112:0/764287486 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f40f40186c0 msgr2=0x7f40f401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:17.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:17.322 7f41157ff700 1 --2- 172.21.2.112:0/764287486 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f40f40186c0 0x7f40f401abc0 crc :-1 s=READY pgs=826 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:17.311 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:17.322 7f41157ff700 1 -- 172.21.2.112:0/764287486 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f41101a7a90 msgr2=0x7f4110085700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:17.311 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:17.322 7f41157ff700 1 --2- 172.21.2.112:0/764287486 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f41101a7a90 0x7f4110085700 secure :-1 s=READY pgs=1745 cs=0 l=1 rev1=1 rx=0x7f4108009390 tx=0x7f410800bd40).stop 2021-02-10T11:37:17.311 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:17.323 7f41157ff700 1 -- 172.21.2.112:0/764287486 shutdown_connections 2021-02-10T11:37:17.311 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:17.323 7f41157ff700 1 --2- 172.21.2.112:0/764287486 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f40f40186c0 0x7f40f401abc0 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:17.311 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:17.323 7f41157ff700 1 --2- 172.21.2.112:0/764287486 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4110084cb0 0x7f41101a7470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:17.311 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:17.323 7f41157ff700 1 --2- 172.21.2.112:0/764287486 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f41101a7a90 0x7f4110085700 unknown :-1 s=CLOSED pgs=1745 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:17.311 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:17.323 7f41157ff700 1 --2- 172.21.2.112:0/764287486 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f41100c2370 0x7f411018f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:17.311 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:17.323 7f41157ff700 1 -- 172.21.2.112:0/764287486 >> 172.21.2.112:0/764287486 conn(0x7f41100b5dc0 msgr2=0x7f41100c2d90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:17.311 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:17.323 7f41157ff700 1 -- 172.21.2.112:0/764287486 shutdown_connections 2021-02-10T11:37:17.311 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:17.323 7f41157ff700 1 -- 172.21.2.112:0/764287486 wait complete. 2021-02-10T11:37:17.319 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:17.319 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:37:17.319 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:37:17.654 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:37:17.655 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:37:17.655 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba012.front.sepia.ceph.com... 2021-02-10T11:37:17.655 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:37:17.655 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:37:18.007 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba012.stderr:ceph-fuse[145270]: fuse finished with error 0 and tester_r 0 2021-02-10T11:37:18.008 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:18.008 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:37:18.008 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:37:18.342 INFO:teuthology.orchestra.run.gibba012.stdout:ext2/ext3 2021-02-10T11:37:18.342 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:37:18.342 INFO:teuthology.orchestra.run:waiting for 900 2021-02-10T11:37:18.343 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:37:18.343 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:37:18.678 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:18.678 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:37:18.679 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:37:19.013 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:37:19.014 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:37:19.014 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba012.front.sepia.ceph.com... 2021-02-10T11:37:19.014 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:37:19.014 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:37:19.368 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba012.stderr:ceph-fuse[145910]: fuse finished with error 0 and tester_r 0 2021-02-10T11:37:19.369 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:19.369 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:37:19.369 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:37:19.379 DEBUG:teuthology.orchestra.run.gibba012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:37:19.414 DEBUG:teuthology.orchestra.run.gibba020:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:37:19.703 INFO:teuthology.orchestra.run.gibba012.stdout:ext2/ext3 2021-02-10T11:37:19.703 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:37:19.703 INFO:teuthology.orchestra.run:waiting for 900 2021-02-10T11:37:19.704 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:37:19.704 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:37:20.039 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:37:20.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:20.463 7f07d61be700 1 Processor -- start 2021-02-10T11:37:20.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:20.463 7f07d61be700 1 -- start start 2021-02-10T11:37:20.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:20.463 7f07d61be700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f07d00d1870 0x7f07d01b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:20.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:20.463 7f07d61be700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07d00f0b10 con 0x7f07d01b0a10 2021-02-10T11:37:20.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:20.463 7f07d61be700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07d00d0310 con 0x7f07d00f0650 2021-02-10T11:37:20.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:20.463 7f07d61be700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07d00d0530 con 0x7f07d00d1870 2021-02-10T11:37:20.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:20.463 7f07cf7fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f07d00d1870 0x7f07d01b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:20.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:20.463 7f07ceffd700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f07d01b0a10 0x7f07d00cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:52242/0 (socket says 172.21.2.112:52242) 2021-02-10T11:37:20.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:20.463 7f07ceffd700 1 -- 172.21.2.112:0/1846648348 learned_addr learned my addr 172.21.2.112:0/1846648348 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:37:20.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:20.464 7f07cf7fe700 1 -- 172.21.2.112:0/1846648348 >> v1:172.21.2.120:6789/0 conn(0x7f07d00f0650 legacy=0x7f07d00cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:37:20.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:20.464 7f07cf7fe700 1 -- 172.21.2.112:0/1846648348 >> v1:172.21.2.112:6789/0 conn(0x7f07d01b0a10 legacy=0x7f07d00cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:20.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:20.464 7f07cf7fe700 1 -- 172.21.2.112:0/1846648348 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07d00d1c70 con 0x7f07d00d1870 2021-02-10T11:37:20.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:20.464 7f07cf7fe700 1 --2- 172.21.2.112:0/1846648348 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f07d00d1870 0x7f07d01b03f0 secure :-1 s=READY pgs=1747 cs=0 l=1 rev1=1 rx=0x7f07c0006650 tx=0x7f07c0009a80).ready entity=mon.0 client_cookie=15d731296e5cca8d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:20.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:20.465 7f07ce7fc700 1 -- 172.21.2.112:0/1846648348 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f07c000c0c0 con 0x7f07d00d1870 2021-02-10T11:37:20.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:20.465 7f07ce7fc700 1 -- 172.21.2.112:0/1846648348 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07c000c340 con 0x7f07d00d1870 2021-02-10T11:37:20.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:20.465 7f07ce7fc700 1 -- 172.21.2.112:0/1846648348 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f07c000c860 con 0x7f07d00d1870 2021-02-10T11:37:20.469 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:20.465 7f07d61be700 1 -- 172.21.2.112:0/1846648348 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f07d00d1870 msgr2=0x7f07d01b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:20.469 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:20.465 7f07d61be700 1 --2- 172.21.2.112:0/1846648348 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f07d00d1870 0x7f07d01b03f0 secure :-1 s=READY pgs=1747 cs=0 l=1 rev1=1 rx=0x7f07c0006650 tx=0x7f07c0009a80).stop 2021-02-10T11:37:20.469 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:20.465 7f07d61be700 1 -- 172.21.2.112:0/1846648348 shutdown_connections 2021-02-10T11:37:20.469 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:20.465 7f07d61be700 1 --2- 172.21.2.112:0/1846648348 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f07d00d1870 0x7f07d01b03f0 unknown :-1 s=CLOSED pgs=1747 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:20.469 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:20.465 7f07d61be700 1 -- 172.21.2.112:0/1846648348 >> 172.21.2.112:0/1846648348 conn(0x7f07d00ce450 msgr2=0x7f07d00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:20.469 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:20.465 7f07d61be700 1 -- 172.21.2.112:0/1846648348 shutdown_connections 2021-02-10T11:37:20.469 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:20.465 7f07d61be700 1 -- 172.21.2.112:0/1846648348 wait complete. 2021-02-10T11:37:20.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:20.480 7f07d61be700 1 Processor -- start 2021-02-10T11:37:20.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:20.480 7f07d61be700 1 -- start start 2021-02-10T11:37:20.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:20.480 7f07d61be700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f07d00ae6b0 0x7f07d00a8bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:20.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:20.480 7f07d61be700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f07d00adbd0 0x7f07d00a9a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:20.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:20.480 7f07d61be700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f07d00adfd0 0x7f07d00aa090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:20.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:20.480 7f07d61be700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07d01bf350 con 0x7f07d00adbd0 2021-02-10T11:37:20.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:20.480 7f07d61be700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07d00a9560 con 0x7f07d00ae6b0 2021-02-10T11:37:20.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:20.480 7f07d61be700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07d00a9210 con 0x7f07d00adfd0 2021-02-10T11:37:20.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:20.481 7f07ceffd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f07d00adbd0 0x7f07d00a9a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:20.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:20.481 7f07ceffd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f07d00adbd0 0x7f07d00a9a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:52790/0 (socket says 172.21.2.112:52790) 2021-02-10T11:37:20.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:20.481 7f07ceffd700 1 -- 172.21.2.112:0/892091819 learned_addr learned my addr 172.21.2.112:0/892091819 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:37:20.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:20.481 7f07ceffd700 1 -- 172.21.2.112:0/892091819 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f07d00adfd0 msgr2=0x7f07d00aa090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:20.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:20.481 7f07ceffd700 1 --2- 172.21.2.112:0/892091819 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f07d00adfd0 0x7f07d00aa090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:20.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:20.481 7f07ceffd700 1 -- 172.21.2.112:0/892091819 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f07d00ae6b0 msgr2=0x7f07d00a8bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:20.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:20.481 7f07ceffd700 1 --2- 172.21.2.112:0/892091819 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f07d00ae6b0 0x7f07d00a8bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:20.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:20.481 7f07ceffd700 1 -- 172.21.2.112:0/892091819 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07d00aeab0 con 0x7f07d00adbd0 2021-02-10T11:37:20.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:20.481 7f07ceffd700 1 --2- 172.21.2.112:0/892091819 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f07d00adbd0 0x7f07d00a9a70 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 rx=0x7f07c4001c60 tx=0x7f07c400a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:20.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:20.481 7f07b7fff700 1 -- 172.21.2.112:0/892091819 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f07c40010b0 con 0x7f07d00adbd0 2021-02-10T11:37:20.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:20.481 7f07b7fff700 1 -- 172.21.2.112:0/892091819 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07c4004140 con 0x7f07d00adbd0 2021-02-10T11:37:20.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:20.481 7f07b7fff700 1 -- 172.21.2.112:0/892091819 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f07c40045e0 con 0x7f07d00adbd0 2021-02-10T11:37:20.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:20.481 7f07d61be700 1 -- 172.21.2.112:0/892091819 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07d00aeab0 con 0x7f07d00adbd0 2021-02-10T11:37:20.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:20.481 7f07d61be700 1 -- 172.21.2.112:0/892091819 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07d00aeab0 con 0x7f07d00adbd0 2021-02-10T11:37:20.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:20.482 7f07b7fff700 1 -- 172.21.2.112:0/892091819 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f07c40010b0 con 0x7f07d00adbd0 2021-02-10T11:37:20.473 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:20.482 7f07b7fff700 1 --2- 172.21.2.112:0/892091819 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f07b0018730 0x7f07b001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:20.473 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:20.482 7f07b7fff700 1 -- 172.21.2.112:0/892091819 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f07c40325f0 con 0x7f07d00adbd0 2021-02-10T11:37:20.473 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:20.482 7f07cf7fe700 1 --2- 172.21.2.112:0/892091819 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f07b0018730 0x7f07b001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:20.473 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:20.483 7f07cf7fe700 1 --2- 172.21.2.112:0/892091819 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f07b0018730 0x7f07b001ac10 crc :-1 s=READY pgs=827 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:20.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:20.513 7f07d61be700 1 -- 172.21.2.112:0/892091819 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07d00e7de0 con 0x7f07d00adbd0 2021-02-10T11:37:20.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:20.516 7f07b7fff700 1 -- 172.21.2.112:0/892091819 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f07c40009b0 con 0x7f07d00adbd0 2021-02-10T11:37:20.634 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:20.646 7f07d61be700 1 -- 172.21.2.112:0/892091819 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f07d00e7de0 con 0x7f07d00adbd0 2021-02-10T11:37:20.634 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:20.646 7f07b7fff700 1 -- 172.21.2.112:0/892091819 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) v1 ==== 94+0+2101 (secure 0 0 0) 0x7f07c40010b0 con 0x7f07d00adbd0 2021-02-10T11:37:20.634 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:37:20.635 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":65,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9204,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/1512510827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":1512510827},{"type":"v1","addr":"172.21.2.112:6833","nonce":1512510827}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":60}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:36:53.563735","modified":"2021-02-10 11:36:55.573740","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9201},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9201":{"gid":9201,"name":"b","rank":0,"incarnation":61,"state":"up:active","state_seq":3,"addr":"172.21.2.120:6835/1232001750","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":1232001750},{"type":"v1","addr":"172.21.2.120:6835","nonce":1232001750}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2021-02-10T11:37:20.635 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 65 2021-02-10T11:37:20.636 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:20.648 7f07d61be700 1 -- 172.21.2.112:0/892091819 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f07b0018730 msgr2=0x7f07b001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:20.636 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:20.648 7f07d61be700 1 --2- 172.21.2.112:0/892091819 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f07b0018730 0x7f07b001ac10 crc :-1 s=READY pgs=827 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:20.636 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:20.648 7f07d61be700 1 -- 172.21.2.112:0/892091819 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f07d00adbd0 msgr2=0x7f07d00a9a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:20.636 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:20.648 7f07d61be700 1 --2- 172.21.2.112:0/892091819 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f07d00adbd0 0x7f07d00a9a70 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 rx=0x7f07c4001c60 tx=0x7f07c400a8d0).stop 2021-02-10T11:37:20.636 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:20.648 7f07d61be700 1 -- 172.21.2.112:0/892091819 shutdown_connections 2021-02-10T11:37:20.636 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:20.648 7f07d61be700 1 --2- 172.21.2.112:0/892091819 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f07b0018730 0x7f07b001ac10 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:20.636 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:20.648 7f07d61be700 1 --2- 172.21.2.112:0/892091819 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f07d00adfd0 0x7f07d00aa090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:20.636 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:20.648 7f07d61be700 1 --2- 172.21.2.112:0/892091819 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f07d00adbd0 0x7f07d00a9a70 unknown :-1 s=CLOSED pgs=1748 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:20.636 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:20.648 7f07d61be700 1 --2- 172.21.2.112:0/892091819 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f07d00ae6b0 0x7f07d00a8bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:20.637 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:20.648 7f07d61be700 1 -- 172.21.2.112:0/892091819 >> 172.21.2.112:0/892091819 conn(0x7f07d00a0030 msgr2=0x7f07d00eb1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:20.637 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:20.648 7f07d61be700 1 -- 172.21.2.112:0/892091819 shutdown_connections 2021-02-10T11:37:20.637 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:20.648 7f07d61be700 1 -- 172.21.2.112:0/892091819 wait complete. 2021-02-10T11:37:20.645 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok session ls 2021-02-10T11:37:21.041 INFO:teuthology.orchestra.run.gibba020.stdout:[] 2021-02-10T11:37:21.045 INFO:tasks.cephfs.filesystem:_json_asok output: [] 2021-02-10T11:37:21.046 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000852595 2021-02-10T11:37:21.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:21.473 7f0882416700 1 Processor -- start 2021-02-10T11:37:21.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:21.473 7f0882416700 1 -- start start 2021-02-10T11:37:21.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:21.473 7f0882416700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f087c0d1c60 0x7f087c0cef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:21.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:21.473 7f0882416700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f087c0d0160 con 0x7f087c0cf570 2021-02-10T11:37:21.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:21.473 7f0882416700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f087c0fe1b0 con 0x7f087c0d1860 2021-02-10T11:37:21.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:21.473 7f0882416700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f087c1b0ca0 con 0x7f087c0d1c60 2021-02-10T11:37:21.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:21.473 7f0880a33700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f087c0cf570 0x7f087c0cf970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:52258/0 (socket says 172.21.2.112:52258) 2021-02-10T11:37:21.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:21.473 7f0880a33700 1 -- 172.21.2.112:0/802535162 learned_addr learned my addr 172.21.2.112:0/802535162 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:37:21.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:21.474 7f087affd700 1 -- 172.21.2.112:0/802535162 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3477368052 0 0) 0x7f0874001500 con 0x7f087c0cf570 2021-02-10T11:37:21.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:21.474 7f087affd700 1 -- 172.21.2.112:0/802535162 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0864002fe0 con 0x7f087c0cf570 2021-02-10T11:37:21.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:21.474 7f087affd700 1 -- 172.21.2.112:0/802535162 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2637327300 0 0) 0x7f0874001460 con 0x7f087c0cf570 2021-02-10T11:37:21.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:21.474 7f087affd700 1 -- 172.21.2.112:0/802535162 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f087c0d1c60 msgr2=0x7f087c0cef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:37:21.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:21.474 7f087affd700 1 --2- 172.21.2.112:0/802535162 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f087c0d1c60 0x7f087c0cef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:21.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:21.474 7f087affd700 1 -- 172.21.2.112:0/802535162 >> v1:172.21.2.120:6789/0 conn(0x7f087c0d1860 legacy=0x7f087c1b04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:37:21.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:21.474 7f087affd700 1 -- 172.21.2.112:0/802535162 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f087c109a50 con 0x7f087c0cf570 2021-02-10T11:37:21.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:21.474 7f087affd700 1 -- 172.21.2.112:0/802535162 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f0874001980 con 0x7f087c0cf570 2021-02-10T11:37:21.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:21.474 7f087affd700 1 -- 172.21.2.112:0/802535162 >> v1:172.21.2.112:6789/0 conn(0x7f087c0cf570 legacy=0x7f087c0cf970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:21.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:21.474 7f087affd700 1 --2- 172.21.2.112:0/802535162 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0864003b90 0x7f0864005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:21.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:21.474 7f087affd700 1 --2- 172.21.2.112:0/802535162 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f08640067f0 0x7f0864008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:21.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:21.474 7f087affd700 1 --2- 172.21.2.112:0/802535162 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0864009590 0x7f086400ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:21.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:21.474 7f087affd700 1 -- 172.21.2.112:0/802535162 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f086400c2d0 con 0x7f0864003b90 2021-02-10T11:37:21.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:21.474 7f087affd700 1 -- 172.21.2.112:0/802535162 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f086400c4f0 con 0x7f08640067f0 2021-02-10T11:37:21.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:21.474 7f087affd700 1 -- 172.21.2.112:0/802535162 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f086400c710 con 0x7f0864009590 2021-02-10T11:37:21.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:21.474 7f087bfff700 1 --2- 172.21.2.112:0/802535162 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0864003b90 0x7f0864005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:21.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:21.474 7f0880a33700 1 --2- 172.21.2.112:0/802535162 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0864009590 0x7f086400ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:21.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:21.474 7f087b7fe700 1 --2- 172.21.2.112:0/802535162 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f08640067f0 0x7f0864008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:21.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:21.474 7f087bfff700 1 -- 172.21.2.112:0/802535162 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0864009590 msgr2=0x7f086400ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:37:21.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:21.474 7f087bfff700 1 --2- 172.21.2.112:0/802535162 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0864009590 0x7f086400ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:21.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:21.474 7f087bfff700 1 -- 172.21.2.112:0/802535162 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f08640067f0 msgr2=0x7f0864008c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:37:21.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:21.474 7f087bfff700 1 --2- 172.21.2.112:0/802535162 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f08640067f0 0x7f0864008c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:21.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:21.474 7f087bfff700 1 -- 172.21.2.112:0/802535162 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f086400c9b0 con 0x7f0864003b90 2021-02-10T11:37:21.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:21.475 7f0880a33700 1 --2- 172.21.2.112:0/802535162 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0864009590 0x7f086400ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:37:21.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:21.475 7f087bfff700 1 --2- 172.21.2.112:0/802535162 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0864003b90 0x7f0864005fa0 secure :-1 s=READY pgs=1750 cs=0 l=1 rev1=1 rx=0x7f086c006ab0 tx=0x7f086c04fb10).ready entity=mon.0 client_cookie=ed384431dd2d11eb server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:21.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:21.475 7f087affd700 1 -- 172.21.2.112:0/802535162 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f086c001290 con 0x7f0864003b90 2021-02-10T11:37:21.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:21.475 7f087affd700 1 -- 172.21.2.112:0/802535162 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f086c0066f0 con 0x7f0864003b90 2021-02-10T11:37:21.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:21.475 7f087affd700 1 -- 172.21.2.112:0/802535162 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f086c001850 con 0x7f0864003b90 2021-02-10T11:37:21.481 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:21.475 7f0882416700 1 -- 172.21.2.112:0/802535162 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0864003b90 msgr2=0x7f0864005fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:21.481 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:21.475 7f0882416700 1 --2- 172.21.2.112:0/802535162 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0864003b90 0x7f0864005fa0 secure :-1 s=READY pgs=1750 cs=0 l=1 rev1=1 rx=0x7f086c006ab0 tx=0x7f086c04fb10).stop 2021-02-10T11:37:21.481 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:21.475 7f0880a33700 1 -- 172.21.2.112:0/802535162 reap_dead start 2021-02-10T11:37:21.481 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:21.475 7f0882416700 1 -- 172.21.2.112:0/802535162 shutdown_connections 2021-02-10T11:37:21.481 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:21.475 7f0882416700 1 -- 172.21.2.112:0/802535162 >> 172.21.2.112:0/802535162 conn(0x7f087c0ce410 msgr2=0x7f087c0d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:21.481 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:21.475 7f0882416700 1 -- 172.21.2.112:0/802535162 shutdown_connections 2021-02-10T11:37:21.481 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:21.476 7f0882416700 1 -- 172.21.2.112:0/802535162 wait complete. 2021-02-10T11:37:21.481 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:21.490 7f0882416700 1 Processor -- start 2021-02-10T11:37:21.483 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:21.490 7f0882416700 1 -- start start 2021-02-10T11:37:21.483 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:21.491 7f0882416700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f087c0badd0 0x7f087c09eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:21.483 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:21.491 7f0882416700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f087c0adf50 0x7f087c09f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:21.483 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:21.491 7f0882416700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f087c1a7470 0x7f087c1a7870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:21.483 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:21.491 7f0882416700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f087c09e5a0 con 0x7f087c0badd0 2021-02-10T11:37:21.483 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:21.491 7f0882416700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f087c09f6e0 con 0x7f087c1a7470 2021-02-10T11:37:21.483 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:21.491 7f0882416700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f087c0bb220 con 0x7f087c0adf50 2021-02-10T11:37:21.483 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:21.491 7f087bfff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f087c0badd0 0x7f087c09eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:21.484 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:21.491 7f087bfff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f087c0badd0 0x7f087c09eaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:52808/0 (socket says 172.21.2.112:52808) 2021-02-10T11:37:21.484 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:21.491 7f087bfff700 1 -- 172.21.2.112:0/835451079 learned_addr learned my addr 172.21.2.112:0/835451079 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:37:21.484 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:21.491 7f0880a33700 1 --2- 172.21.2.112:0/835451079 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f087c1a7470 0x7f087c1a7870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:21.484 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:21.491 7f087b7fe700 1 --2- 172.21.2.112:0/835451079 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f087c0adf50 0x7f087c09f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:21.484 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:21.491 7f087bfff700 1 -- 172.21.2.112:0/835451079 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f087c0adf50 msgr2=0x7f087c09f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:21.484 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:21.491 7f087bfff700 1 --2- 172.21.2.112:0/835451079 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f087c0adf50 0x7f087c09f0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:21.484 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:21.491 7f087bfff700 1 -- 172.21.2.112:0/835451079 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f087c1a7470 msgr2=0x7f087c1a7870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:21.484 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:21.491 7f087bfff700 1 --2- 172.21.2.112:0/835451079 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f087c1a7470 0x7f087c1a7870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:21.484 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:21.491 7f087bfff700 1 -- 172.21.2.112:0/835451079 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f087c1a7e90 con 0x7f087c0badd0 2021-02-10T11:37:21.485 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:21.491 7f087bfff700 1 --2- 172.21.2.112:0/835451079 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f087c0badd0 0x7f087c09eaa0 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 rx=0x7f086c009650 tx=0x7f086c0099f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:21.485 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:21.492 7f0878ff9700 1 -- 172.21.2.112:0/835451079 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f086c006030 con 0x7f087c0badd0 2021-02-10T11:37:21.485 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:21.492 7f0878ff9700 1 -- 172.21.2.112:0/835451079 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f086c006370 con 0x7f087c0badd0 2021-02-10T11:37:21.485 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:21.492 7f0878ff9700 1 -- 172.21.2.112:0/835451079 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f086c0057c0 con 0x7f087c0badd0 2021-02-10T11:37:21.485 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:21.492 7f0882416700 1 -- 172.21.2.112:0/835451079 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f087c1a7e90 con 0x7f087c0badd0 2021-02-10T11:37:21.485 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:21.492 7f0882416700 1 -- 172.21.2.112:0/835451079 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f087c09d1f0 con 0x7f087c0badd0 2021-02-10T11:37:21.485 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:21.492 7f0878ff9700 1 -- 172.21.2.112:0/835451079 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f086c006030 con 0x7f087c0badd0 2021-02-10T11:37:21.486 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:21.492 7f0878ff9700 1 --2- 172.21.2.112:0/835451079 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f085c0186c0 0x7f085c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:21.486 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:21.492 7f0878ff9700 1 -- 172.21.2.112:0/835451079 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f086c0765a0 con 0x7f087c0badd0 2021-02-10T11:37:21.486 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:21.493 7f087b7fe700 1 --2- 172.21.2.112:0/835451079 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f085c0186c0 0x7f085c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:21.486 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:21.493 7f087b7fe700 1 --2- 172.21.2.112:0/835451079 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f085c0186c0 0x7f085c01abc0 crc :-1 s=READY pgs=828 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:21.511 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:21.523 7f0882416700 1 -- 172.21.2.112:0/835451079 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f087c082a20 con 0x7f087c0badd0 2021-02-10T11:37:21.514 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:21.526 7f0878ff9700 1 -- 172.21.2.112:0/835451079 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f086c000ac0 con 0x7f087c0badd0 2021-02-10T11:37:21.643 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:21.655 7f0882416700 1 -- 172.21.2.112:0/835451079 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000852595", "target": ["mgr", ""]}) v1 -- 0x7f087c09d1f0 con 0x7f085c0186c0 2021-02-10T11:37:21.647 INFO:tasks.ceph.mgr.x.gibba020.stderr:2021-02-10 11:37:21.655 7f6382eee700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:37:21.647 INFO:tasks.ceph.mgr.x.gibba020.stderr:2021-02-10 11:37:21.655 7f6382eee700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:37:21.647 INFO:tasks.ceph.mgr.x.gibba020.stderr:2021-02-10 11:37:21.655 7f6382eee700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:37:21.647 INFO:tasks.ceph.mgr.x.gibba020.stderr:2021-02-10 11:37:21.655 7f6382eee700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:37:21.647 INFO:tasks.ceph.mgr.x.gibba020.stderr:2021-02-10 11:37:21.655 7f6382eee700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:37:21.679 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:21.691 7f0878ff9700 1 -- 172.21.2.112:0/835451079 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f0870006e70 con 0x7f085c0186c0 2021-02-10T11:37:21.708 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:21.720 7f0882416700 1 -- 172.21.2.112:0/835451079 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f085c0186c0 msgr2=0x7f085c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:21.708 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:21.720 7f0882416700 1 --2- 172.21.2.112:0/835451079 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f085c0186c0 0x7f085c01abc0 crc :-1 s=READY pgs=828 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:21.708 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:21.720 7f0882416700 1 -- 172.21.2.112:0/835451079 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f087c0badd0 msgr2=0x7f087c09eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:21.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:21.720 7f0882416700 1 --2- 172.21.2.112:0/835451079 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f087c0badd0 0x7f087c09eaa0 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 rx=0x7f086c009650 tx=0x7f086c0099f0).stop 2021-02-10T11:37:21.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:21.721 7f0882416700 1 -- 172.21.2.112:0/835451079 shutdown_connections 2021-02-10T11:37:21.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:21.721 7f0882416700 1 --2- 172.21.2.112:0/835451079 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f085c0186c0 0x7f085c01abc0 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:21.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:21.721 7f0882416700 1 --2- 172.21.2.112:0/835451079 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f087c1a7470 0x7f087c1a7870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:21.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:21.721 7f0882416700 1 --2- 172.21.2.112:0/835451079 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f087c0adf50 0x7f087c09f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:21.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:21.721 7f0882416700 1 --2- 172.21.2.112:0/835451079 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f087c0badd0 0x7f087c09eaa0 unknown :-1 s=CLOSED pgs=1751 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:21.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:21.721 7f0882416700 1 -- 172.21.2.112:0/835451079 >> 172.21.2.112:0/835451079 conn(0x7f087c0bee90 msgr2=0x7f087c0bb880 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:21.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:21.721 7f0882416700 1 -- 172.21.2.112:0/835451079 shutdown_connections 2021-02-10T11:37:21.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:21.721 7f0882416700 1 -- 172.21.2.112:0/835451079 wait complete. 2021-02-10T11:37:21.717 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:37:22.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:22.139 7fd52de1c700 1 Processor -- start 2021-02-10T11:37:22.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:22.139 7fd52de1c700 1 -- start start 2021-02-10T11:37:22.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:22.139 7fd52de1c700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fd5280d1870 0x7fd5281b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:22.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:22.139 7fd52de1c700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fd5280f0650 0x7fd5280cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:22.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:22.139 7fd52de1c700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5280f0b10 con 0x7fd5281b0a10 2021-02-10T11:37:22.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:22.139 7fd52de1c700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5280e5ed0 con 0x7fd5280d1870 2021-02-10T11:37:22.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:22.139 7fd52de1c700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5280d1c70 con 0x7fd5280f0650 2021-02-10T11:37:22.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:22.139 7fd5277fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fd5280d1870 0x7fd5281b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:22.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:22.139 7fd5277fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fd5280d1870 0x7fd5281b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:52814/0 (socket says 172.21.2.112:52814) 2021-02-10T11:37:22.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:22.139 7fd5277fe700 1 -- 172.21.2.112:0/1281461384 learned_addr learned my addr 172.21.2.112:0/1281461384 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:37:22.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:22.140 7fd5277fe700 1 -- 172.21.2.112:0/1281461384 >> v2:172.21.2.120:3300/0 conn(0x7fd5280f0650 msgr2=0x7fd5280cfb20 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:37:22.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:22.140 7fd5277fe700 1 --2- 172.21.2.112:0/1281461384 >> v2:172.21.2.120:3300/0 conn(0x7fd5280f0650 0x7fd5280cfb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:22.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:22.140 7fd5277fe700 1 -- 172.21.2.112:0/1281461384 >> v1:172.21.2.120:6789/0 conn(0x7fd5281b0a10 legacy=0x7fd5280cf330 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:37:22.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:22.140 7fd5277fe700 1 -- 172.21.2.112:0/1281461384 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5280d1e90 con 0x7fd5280d1870 2021-02-10T11:37:22.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:22.140 7fd5277fe700 1 --2- 172.21.2.112:0/1281461384 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd5280d1870 0x7fd5281b03f0 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 rx=0x7fd518006700 tx=0x7fd518009b80).ready entity=mon.0 client_cookie=1033f2df444ce561 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:22.145 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:22.140 7fd5267fc700 1 -- 172.21.2.112:0/1281461384 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd51800c0c0 con 0x7fd5280d1870 2021-02-10T11:37:22.145 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:22.140 7fd5267fc700 1 -- 172.21.2.112:0/1281461384 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd51800c440 con 0x7fd5280d1870 2021-02-10T11:37:22.145 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:22.140 7fd5267fc700 1 -- 172.21.2.112:0/1281461384 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd51800c960 con 0x7fd5280d1870 2021-02-10T11:37:22.145 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:22.140 7fd52de1c700 1 -- 172.21.2.112:0/1281461384 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd5280d1870 msgr2=0x7fd5281b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:22.145 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:22.140 7fd52de1c700 1 --2- 172.21.2.112:0/1281461384 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd5280d1870 0x7fd5281b03f0 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 rx=0x7fd518006700 tx=0x7fd518009b80).stop 2021-02-10T11:37:22.145 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:22.141 7fd52de1c700 1 -- 172.21.2.112:0/1281461384 shutdown_connections 2021-02-10T11:37:22.145 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:22.141 7fd52de1c700 1 --2- 172.21.2.112:0/1281461384 >> v2:172.21.2.120:3300/0 conn(0x7fd5280f0650 0x7fd5280cfb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:22.145 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:22.141 7fd52de1c700 1 --2- 172.21.2.112:0/1281461384 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd5280d1870 0x7fd5281b03f0 unknown :-1 s=CLOSED pgs=1752 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:22.145 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:22.141 7fd52de1c700 1 -- 172.21.2.112:0/1281461384 >> 172.21.2.112:0/1281461384 conn(0x7fd5280ce450 msgr2=0x7fd5280d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:22.145 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:22.141 7fd52de1c700 1 -- 172.21.2.112:0/1281461384 shutdown_connections 2021-02-10T11:37:22.145 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:22.141 7fd52de1c700 1 -- 172.21.2.112:0/1281461384 wait complete. 2021-02-10T11:37:22.147 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:22.155 7fd52de1c700 1 Processor -- start 2021-02-10T11:37:22.148 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:22.156 7fd52de1c700 1 -- start start 2021-02-10T11:37:22.148 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:22.156 7fd52de1c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd5280b0ad0 0x7fd528091f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:22.148 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:22.156 7fd52de1c700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd5281a8850 0x7fd5281a8c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:22.148 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:22.156 7fd52de1c700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd5281a6fa0 0x7fd5281a73a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:22.148 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:22.156 7fd52de1c700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5281abfe0 con 0x7fd5280b0ad0 2021-02-10T11:37:22.148 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:22.156 7fd52de1c700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5281a9c40 con 0x7fd5281a6fa0 2021-02-10T11:37:22.148 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:22.156 7fd52de1c700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5281a9730 con 0x7fd5281a8850 2021-02-10T11:37:22.148 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:22.156 7fd5277fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd5280b0ad0 0x7fd528091f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:22.148 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:22.156 7fd5277fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd5280b0ad0 0x7fd528091f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:52816/0 (socket says 172.21.2.112:52816) 2021-02-10T11:37:22.148 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:22.156 7fd5277fe700 1 -- 172.21.2.112:0/2967306955 learned_addr learned my addr 172.21.2.112:0/2967306955 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:37:22.148 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:22.157 7fd526ffd700 1 --2- 172.21.2.112:0/2967306955 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd5281a8850 0x7fd5281a8c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:22.149 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:22.157 7fd5277fe700 1 -- 172.21.2.112:0/2967306955 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd5281a8850 msgr2=0x7fd5281a8c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:22.149 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:22.157 7fd527fff700 1 --2- 172.21.2.112:0/2967306955 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd5281a6fa0 0x7fd5281a73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:22.149 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:22.157 7fd5277fe700 1 --2- 172.21.2.112:0/2967306955 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd5281a8850 0x7fd5281a8c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:22.149 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:22.157 7fd5277fe700 1 -- 172.21.2.112:0/2967306955 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd5281a6fa0 msgr2=0x7fd5281a73a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:22.149 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:22.157 7fd5277fe700 1 --2- 172.21.2.112:0/2967306955 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd5281a6fa0 0x7fd5281a73a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:22.149 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:22.157 7fd5277fe700 1 -- 172.21.2.112:0/2967306955 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5281a9270 con 0x7fd5280b0ad0 2021-02-10T11:37:22.149 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:22.157 7fd526ffd700 1 --2- 172.21.2.112:0/2967306955 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd5281a8850 0x7fd5281a8c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:37:22.149 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:22.157 7fd527fff700 1 --2- 172.21.2.112:0/2967306955 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd5281a6fa0 0x7fd5281a73a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:37:22.149 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:22.157 7fd5277fe700 1 --2- 172.21.2.112:0/2967306955 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd5280b0ad0 0x7fd528091f30 secure :-1 s=READY pgs=1753 cs=0 l=1 rev1=1 rx=0x7fd51800b5e0 tx=0x7fd51800b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:22.149 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:22.157 7fd51ffff700 1 -- 172.21.2.112:0/2967306955 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd51800bd70 con 0x7fd5280b0ad0 2021-02-10T11:37:22.149 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:22.158 7fd51ffff700 1 -- 172.21.2.112:0/2967306955 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd518002030 con 0x7fd5280b0ad0 2021-02-10T11:37:22.150 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:22.158 7fd52de1c700 1 -- 172.21.2.112:0/2967306955 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5281a9270 con 0x7fd5280b0ad0 2021-02-10T11:37:22.150 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:22.158 7fd51ffff700 1 -- 172.21.2.112:0/2967306955 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd518002550 con 0x7fd5280b0ad0 2021-02-10T11:37:22.150 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:22.158 7fd52de1c700 1 -- 172.21.2.112:0/2967306955 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5281a7cb0 con 0x7fd5280b0ad0 2021-02-10T11:37:22.150 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:22.158 7fd51ffff700 1 -- 172.21.2.112:0/2967306955 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd518000c90 con 0x7fd5280b0ad0 2021-02-10T11:37:22.150 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:22.158 7fd51ffff700 1 --2- 172.21.2.112:0/2967306955 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd50c0186c0 0x7fd50c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:22.150 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:22.158 7fd51ffff700 1 -- 172.21.2.112:0/2967306955 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fd518032580 con 0x7fd5280b0ad0 2021-02-10T11:37:22.150 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:22.159 7fd526ffd700 1 --2- 172.21.2.112:0/2967306955 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd50c0186c0 0x7fd50c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:22.150 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:22.159 7fd526ffd700 1 --2- 172.21.2.112:0/2967306955 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd50c0186c0 0x7fd50c01abc0 crc :-1 s=READY pgs=829 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:22.177 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:22.189 7fd52de1c700 1 -- 172.21.2.112:0/2967306955 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5280ce450 con 0x7fd5280b0ad0 2021-02-10T11:37:22.180 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:22.192 7fd51ffff700 1 -- 172.21.2.112:0/2967306955 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd518000940 con 0x7fd5280b0ad0 2021-02-10T11:37:22.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:22.317 7fd52de1c700 1 -- 172.21.2.112:0/2967306955 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd5280ce450 con 0x7fd5280b0ad0 2021-02-10T11:37:22.306 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:22.318 7fd51ffff700 1 -- 172.21.2.112:0/2967306955 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) v1 ==== 94+0+2101 (secure 0 0 0) 0x7fd518000c90 con 0x7fd5280b0ad0 2021-02-10T11:37:22.307 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 65 2021-02-10T11:37:22.307 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:37:22.307 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":65,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9204,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/1512510827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":1512510827},{"type":"v1","addr":"172.21.2.112:6833","nonce":1512510827}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":60}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:36:53.563735","modified":"2021-02-10 11:36:55.573740","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9201},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9201":{"gid":9201,"name":"b","rank":0,"incarnation":61,"state":"up:active","state_seq":3,"addr":"172.21.2.120:6835/1232001750","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":1232001750},{"type":"v1","addr":"172.21.2.120:6835","nonce":1232001750}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2021-02-10T11:37:22.308 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:22.320 7fd52de1c700 1 -- 172.21.2.112:0/2967306955 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd50c0186c0 msgr2=0x7fd50c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:22.308 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:22.320 7fd52de1c700 1 --2- 172.21.2.112:0/2967306955 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd50c0186c0 0x7fd50c01abc0 crc :-1 s=READY pgs=829 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:22.308 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:22.320 7fd52de1c700 1 -- 172.21.2.112:0/2967306955 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd5280b0ad0 msgr2=0x7fd528091f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:22.308 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:22.320 7fd52de1c700 1 --2- 172.21.2.112:0/2967306955 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd5280b0ad0 0x7fd528091f30 secure :-1 s=READY pgs=1753 cs=0 l=1 rev1=1 rx=0x7fd51800b5e0 tx=0x7fd51800b980).stop 2021-02-10T11:37:22.308 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:22.320 7fd52de1c700 1 -- 172.21.2.112:0/2967306955 shutdown_connections 2021-02-10T11:37:22.308 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:22.320 7fd52de1c700 1 --2- 172.21.2.112:0/2967306955 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd50c0186c0 0x7fd50c01abc0 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:22.308 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:22.320 7fd52de1c700 1 --2- 172.21.2.112:0/2967306955 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd5281a6fa0 0x7fd5281a73a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:22.308 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:22.320 7fd52de1c700 1 --2- 172.21.2.112:0/2967306955 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd5281a8850 0x7fd5281a8c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:22.308 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:22.320 7fd52de1c700 1 --2- 172.21.2.112:0/2967306955 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd5280b0ad0 0x7fd528091f30 unknown :-1 s=CLOSED pgs=1753 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:22.308 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:22.320 7fd52de1c700 1 -- 172.21.2.112:0/2967306955 >> 172.21.2.112:0/2967306955 conn(0x7fd52819f8f0 msgr2=0x7fd5281a5f80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:22.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:22.320 7fd52de1c700 1 -- 172.21.2.112:0/2967306955 shutdown_connections 2021-02-10T11:37:22.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:22.320 7fd52de1c700 1 -- 172.21.2.112:0/2967306955 wait complete. 2021-02-10T11:37:22.317 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok session ls 2021-02-10T11:37:22.714 INFO:teuthology.orchestra.run.gibba020.stdout:[ 2021-02-10T11:37:22.714 INFO:teuthology.orchestra.run.gibba020.stdout: { 2021-02-10T11:37:22.714 INFO:teuthology.orchestra.run.gibba020.stdout: "id": 7390, 2021-02-10T11:37:22.714 INFO:teuthology.orchestra.run.gibba020.stdout: "entity": { 2021-02-10T11:37:22.715 INFO:teuthology.orchestra.run.gibba020.stdout: "name": { 2021-02-10T11:37:22.715 INFO:teuthology.orchestra.run.gibba020.stdout: "type": "client", 2021-02-10T11:37:22.715 INFO:teuthology.orchestra.run.gibba020.stdout: "num": 7390 2021-02-10T11:37:22.715 INFO:teuthology.orchestra.run.gibba020.stdout: }, 2021-02-10T11:37:22.715 INFO:teuthology.orchestra.run.gibba020.stdout: "addr": { 2021-02-10T11:37:22.715 INFO:teuthology.orchestra.run.gibba020.stdout: "type": "any", 2021-02-10T11:37:22.715 INFO:teuthology.orchestra.run.gibba020.stdout: "addr": "172.21.2.120:0", 2021-02-10T11:37:22.715 INFO:teuthology.orchestra.run.gibba020.stdout: "nonce": 1069935987 2021-02-10T11:37:22.715 INFO:teuthology.orchestra.run.gibba020.stdout: } 2021-02-10T11:37:22.715 INFO:teuthology.orchestra.run.gibba020.stdout: }, 2021-02-10T11:37:22.715 INFO:teuthology.orchestra.run.gibba020.stdout: "state": "open", 2021-02-10T11:37:22.715 INFO:teuthology.orchestra.run.gibba020.stdout: "num_leases": 1, 2021-02-10T11:37:22.715 INFO:teuthology.orchestra.run.gibba020.stdout: "num_caps": 7, 2021-02-10T11:37:22.716 INFO:teuthology.orchestra.run.gibba020.stdout: "request_load_avg": 13, 2021-02-10T11:37:22.716 INFO:teuthology.orchestra.run.gibba020.stdout: "uptime": 1.0649426420000001, 2021-02-10T11:37:22.716 INFO:teuthology.orchestra.run.gibba020.stdout: "requests_in_flight": 1, 2021-02-10T11:37:22.716 INFO:teuthology.orchestra.run.gibba020.stdout: "completed_requests": 0, 2021-02-10T11:37:22.716 INFO:teuthology.orchestra.run.gibba020.stdout: "reconnecting": false, 2021-02-10T11:37:22.716 INFO:teuthology.orchestra.run.gibba020.stdout: "recall_caps": { 2021-02-10T11:37:22.716 INFO:teuthology.orchestra.run.gibba020.stdout: "value": 0, 2021-02-10T11:37:22.716 INFO:teuthology.orchestra.run.gibba020.stdout: "halflife": 60 2021-02-10T11:37:22.716 INFO:teuthology.orchestra.run.gibba020.stdout: }, 2021-02-10T11:37:22.716 INFO:teuthology.orchestra.run.gibba020.stdout: "release_caps": { 2021-02-10T11:37:22.716 INFO:teuthology.orchestra.run.gibba020.stdout: "value": 0, 2021-02-10T11:37:22.716 INFO:teuthology.orchestra.run.gibba020.stdout: "halflife": 60 2021-02-10T11:37:22.716 INFO:teuthology.orchestra.run.gibba020.stdout: }, 2021-02-10T11:37:22.717 INFO:teuthology.orchestra.run.gibba020.stdout: "recall_caps_throttle": { 2021-02-10T11:37:22.717 INFO:teuthology.orchestra.run.gibba020.stdout: "value": 0, 2021-02-10T11:37:22.717 INFO:teuthology.orchestra.run.gibba020.stdout: "halflife": 2.5 2021-02-10T11:37:22.717 INFO:teuthology.orchestra.run.gibba020.stdout: }, 2021-02-10T11:37:22.717 INFO:teuthology.orchestra.run.gibba020.stdout: "recall_caps_throttle2o": { 2021-02-10T11:37:22.717 INFO:teuthology.orchestra.run.gibba020.stdout: "value": 0, 2021-02-10T11:37:22.717 INFO:teuthology.orchestra.run.gibba020.stdout: "halflife": 0.5 2021-02-10T11:37:22.717 INFO:teuthology.orchestra.run.gibba020.stdout: }, 2021-02-10T11:37:22.717 INFO:teuthology.orchestra.run.gibba020.stdout: "session_cache_liveness": { 2021-02-10T11:37:22.717 INFO:teuthology.orchestra.run.gibba020.stdout: "value": 19.951416658234642, 2021-02-10T11:37:22.717 INFO:teuthology.orchestra.run.gibba020.stdout: "halflife": 300 2021-02-10T11:37:22.718 INFO:teuthology.orchestra.run.gibba020.stdout: }, 2021-02-10T11:37:22.718 INFO:teuthology.orchestra.run.gibba020.stdout: "cap_acquisition": { 2021-02-10T11:37:22.718 INFO:teuthology.orchestra.run.gibba020.stdout: "value": 0, 2021-02-10T11:37:22.718 INFO:teuthology.orchestra.run.gibba020.stdout: "halflife": 10 2021-02-10T11:37:22.718 INFO:teuthology.orchestra.run.gibba020.stdout: }, 2021-02-10T11:37:22.718 INFO:teuthology.orchestra.run.gibba020.stdout: "inst": "client.7390 172.21.2.120:0/1069935987", 2021-02-10T11:37:22.718 INFO:teuthology.orchestra.run.gibba020.stdout: "completed_requests": [], 2021-02-10T11:37:22.718 INFO:teuthology.orchestra.run.gibba020.stdout: "prealloc_inos": [ 2021-02-10T11:37:22.718 INFO:teuthology.orchestra.run.gibba020.stdout: { 2021-02-10T11:37:22.718 INFO:teuthology.orchestra.run.gibba020.stdout: "start": 1099511627782, 2021-02-10T11:37:22.718 INFO:teuthology.orchestra.run.gibba020.stdout: "length": 995 2021-02-10T11:37:22.718 INFO:teuthology.orchestra.run.gibba020.stdout: } 2021-02-10T11:37:22.718 INFO:teuthology.orchestra.run.gibba020.stdout: ], 2021-02-10T11:37:22.719 INFO:teuthology.orchestra.run.gibba020.stdout: "used_inos": [ 2021-02-10T11:37:22.719 INFO:teuthology.orchestra.run.gibba020.stdout: { 2021-02-10T11:37:22.719 INFO:teuthology.orchestra.run.gibba020.stdout: "start": 1099511627781, 2021-02-10T11:37:22.719 INFO:teuthology.orchestra.run.gibba020.stdout: "length": 1 2021-02-10T11:37:22.719 INFO:teuthology.orchestra.run.gibba020.stdout: } 2021-02-10T11:37:22.719 INFO:teuthology.orchestra.run.gibba020.stdout: ], 2021-02-10T11:37:22.719 INFO:teuthology.orchestra.run.gibba020.stdout: "client_metadata": { 2021-02-10T11:37:22.719 INFO:teuthology.orchestra.run.gibba020.stdout: "features": "0x0000000000001fff", 2021-02-10T11:37:22.719 INFO:teuthology.orchestra.run.gibba020.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:37:22.719 INFO:teuthology.orchestra.run.gibba020.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:37:22.719 INFO:teuthology.orchestra.run.gibba020.stdout: "entity_id": "x", 2021-02-10T11:37:22.719 INFO:teuthology.orchestra.run.gibba020.stdout: "hostname": "gibba020", 2021-02-10T11:37:22.720 INFO:teuthology.orchestra.run.gibba020.stdout: "pid": "34990", 2021-02-10T11:37:22.720 INFO:teuthology.orchestra.run.gibba020.stdout: "root": "/" 2021-02-10T11:37:22.720 INFO:teuthology.orchestra.run.gibba020.stdout: } 2021-02-10T11:37:22.720 INFO:teuthology.orchestra.run.gibba020.stdout: } 2021-02-10T11:37:22.720 INFO:teuthology.orchestra.run.gibba020.stdout:] 2021-02-10T11:37:22.721 INFO:tasks.cephfs.filesystem:_json_asok output: [ { "id": 7390, "entity": { "name": { "type": "client", "num": 7390 }, "addr": { "type": "any", "addr": "172.21.2.120:0", "nonce": 1069935987 } }, "state": "open", "num_leases": 1, "num_caps": 7, "request_load_avg": 13, "uptime": 1.0649426420000001, "requests_in_flight": 1, "completed_requests": 0, "reconnecting": false, "recall_caps": { "value": 0, "halflife": 60 }, "release_caps": { "value": 0, "halflife": 60 }, "recall_caps_throttle": { "value": 0, "halflife": 2.5 }, "recall_caps_throttle2o": { "value": 0, "halflife": 0.5 }, "session_cache_liveness": { "value": 19.951416658234642, "halflife": 300 }, "cap_acquisition": { "value": 0, "halflife": 10 }, "inst": "client.7390 172.21.2.120:0/1069935987", "completed_requests": [], "prealloc_inos": [ { "start": 1099511627782, "length": 995 } ], "used_inos": [ { "start": 1099511627781, "length": 1 } ], "client_metadata": { "features": "0x0000000000001fff", "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", "entity_id": "x", "hostname": "gibba020", "pid": "34990", "root": "/" } } ] 2021-02-10T11:37:22.721 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:37:23.147 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:23.142 7ff842625700 1 Processor -- start 2021-02-10T11:37:23.147 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:23.142 7ff842625700 1 -- start start 2021-02-10T11:37:23.147 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:23.143 7ff842625700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7ff83c0d1870 0x7ff83c1b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:23.147 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:23.143 7ff842625700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7ff83c0f0650 0x7ff83c0cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:23.147 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:23.143 7ff842625700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff83c0f0b10 con 0x7ff83c1b0a10 2021-02-10T11:37:23.147 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:23.143 7ff842625700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff83c0e5ed0 con 0x7ff83c0d1870 2021-02-10T11:37:23.148 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:23.143 7ff842625700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff83c0d1c70 con 0x7ff83c0f0650 2021-02-10T11:37:23.148 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:23.143 7ff83bfff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7ff83c0d1870 0x7ff83c1b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:23.148 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:23.143 7ff83bfff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7ff83c0d1870 0x7ff83c1b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:52824/0 (socket says 172.21.2.112:52824) 2021-02-10T11:37:23.148 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:23.143 7ff83bfff700 1 -- 172.21.2.112:0/4103263938 learned_addr learned my addr 172.21.2.112:0/4103263938 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:37:23.148 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:23.143 7ff83b7fe700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7ff83c1b0a10 0x7ff83c0cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:52286/0 (socket says 172.21.2.112:52286) 2021-02-10T11:37:23.148 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:23.143 7ff840c42700 1 --2- 172.21.2.112:0/4103263938 >> v2:172.21.2.120:3300/0 conn(0x7ff83c0f0650 0x7ff83c0cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:23.148 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:23.143 7ff83bfff700 1 -- 172.21.2.112:0/4103263938 >> v2:172.21.2.120:3300/0 conn(0x7ff83c0f0650 msgr2=0x7ff83c0cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:37:23.148 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:23.143 7ff83bfff700 1 --2- 172.21.2.112:0/4103263938 >> v2:172.21.2.120:3300/0 conn(0x7ff83c0f0650 0x7ff83c0cfb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:23.148 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:23.143 7ff83bfff700 1 -- 172.21.2.112:0/4103263938 >> v1:172.21.2.112:6789/0 conn(0x7ff83c1b0a10 legacy=0x7ff83c0cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:23.148 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:23.143 7ff83bfff700 1 -- 172.21.2.112:0/4103263938 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff83c0d1e90 con 0x7ff83c0d1870 2021-02-10T11:37:23.148 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:23.144 7ff840c42700 1 --2- 172.21.2.112:0/4103263938 >> v2:172.21.2.120:3300/0 conn(0x7ff83c0f0650 0x7ff83c0cfb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:37:23.148 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:23.144 7ff83bfff700 1 --2- 172.21.2.112:0/4103263938 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff83c0d1870 0x7ff83c1b03f0 secure :-1 s=READY pgs=1755 cs=0 l=1 rev1=1 rx=0x7ff82c006700 tx=0x7ff82c009b80).ready entity=mon.0 client_cookie=6c6d05b4a170bb23 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:23.149 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:23.144 7ff83affd700 1 -- 172.21.2.112:0/4103263938 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff82c00c0c0 con 0x7ff83c0d1870 2021-02-10T11:37:23.149 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:23.144 7ff83affd700 1 -- 172.21.2.112:0/4103263938 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff82c00c340 con 0x7ff83c0d1870 2021-02-10T11:37:23.149 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:23.144 7ff83affd700 1 -- 172.21.2.112:0/4103263938 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff82c00c860 con 0x7ff83c0d1870 2021-02-10T11:37:23.149 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:23.144 7ff842625700 1 -- 172.21.2.112:0/4103263938 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff83c0d1870 msgr2=0x7ff83c1b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:23.149 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:23.144 7ff842625700 1 --2- 172.21.2.112:0/4103263938 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff83c0d1870 0x7ff83c1b03f0 secure :-1 s=READY pgs=1755 cs=0 l=1 rev1=1 rx=0x7ff82c006700 tx=0x7ff82c009b80).stop 2021-02-10T11:37:23.149 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:23.145 7ff842625700 1 -- 172.21.2.112:0/4103263938 shutdown_connections 2021-02-10T11:37:23.149 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:23.145 7ff842625700 1 --2- 172.21.2.112:0/4103263938 >> v2:172.21.2.120:3300/0 conn(0x7ff83c0f0650 0x7ff83c0cfb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:23.149 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:23.145 7ff842625700 1 --2- 172.21.2.112:0/4103263938 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff83c0d1870 0x7ff83c1b03f0 unknown :-1 s=CLOSED pgs=1755 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:23.149 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:23.145 7ff842625700 1 -- 172.21.2.112:0/4103263938 >> 172.21.2.112:0/4103263938 conn(0x7ff83c0ce450 msgr2=0x7ff83c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:23.149 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:23.145 7ff842625700 1 -- 172.21.2.112:0/4103263938 shutdown_connections 2021-02-10T11:37:23.149 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:23.145 7ff842625700 1 -- 172.21.2.112:0/4103263938 wait complete. 2021-02-10T11:37:23.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:23.159 7ff842625700 1 Processor -- start 2021-02-10T11:37:23.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:23.160 7ff842625700 1 -- start start 2021-02-10T11:37:23.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:23.160 7ff842625700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff83c1a4a60 0x7ff83c0c1c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:23.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:23.160 7ff842625700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff83c0c2290 0x7ff83c0c2690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:23.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:23.160 7ff842625700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff83c0c2cb0 0x7ff83c0bfd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:23.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:23.160 7ff842625700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff83c0f05b0 con 0x7ff83c0c2cb0 2021-02-10T11:37:23.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:23.160 7ff842625700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff83c0f07d0 con 0x7ff83c1a4a60 2021-02-10T11:37:23.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:23.160 7ff842625700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff83c0c0340 con 0x7ff83c0c2290 2021-02-10T11:37:23.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:23.160 7ff840c42700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff83c0c2cb0 0x7ff83c0bfd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:23.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:23.160 7ff840c42700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff83c0c2cb0 0x7ff83c0bfd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:52826/0 (socket says 172.21.2.112:52826) 2021-02-10T11:37:23.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:23.160 7ff840c42700 1 -- 172.21.2.112:0/3200114544 learned_addr learned my addr 172.21.2.112:0/3200114544 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:37:23.152 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:23.161 7ff840c42700 1 -- 172.21.2.112:0/3200114544 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff83c0c2290 msgr2=0x7ff83c0c2690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:23.152 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:23.161 7ff840c42700 1 --2- 172.21.2.112:0/3200114544 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff83c0c2290 0x7ff83c0c2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:23.152 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:23.161 7ff840c42700 1 -- 172.21.2.112:0/3200114544 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff83c1a4a60 msgr2=0x7ff83c0c1c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:23.152 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:23.161 7ff840c42700 1 --2- 172.21.2.112:0/3200114544 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff83c1a4a60 0x7ff83c0c1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:23.152 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:23.161 7ff840c42700 1 -- 172.21.2.112:0/3200114544 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff83c0c0560 con 0x7ff83c0c2cb0 2021-02-10T11:37:23.152 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:23.161 7ff840c42700 1 --2- 172.21.2.112:0/3200114544 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff83c0c2cb0 0x7ff83c0bfd20 secure :-1 s=READY pgs=1756 cs=0 l=1 rev1=1 rx=0x7ff834009980 tx=0x7ff83400bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:23.152 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:23.161 7ff838ff9700 1 -- 172.21.2.112:0/3200114544 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff834006030 con 0x7ff83c0c2cb0 2021-02-10T11:37:23.152 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:23.161 7ff838ff9700 1 -- 172.21.2.112:0/3200114544 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff834006790 con 0x7ff83c0c2cb0 2021-02-10T11:37:23.152 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:23.161 7ff838ff9700 1 -- 172.21.2.112:0/3200114544 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff8340056b0 con 0x7ff83c0c2cb0 2021-02-10T11:37:23.152 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:23.161 7ff842625700 1 -- 172.21.2.112:0/3200114544 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff83c0c0560 con 0x7ff83c0c2cb0 2021-02-10T11:37:23.152 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:23.161 7ff842625700 1 -- 172.21.2.112:0/3200114544 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff83c0c0af0 con 0x7ff83c0c2cb0 2021-02-10T11:37:23.152 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:23.162 7ff838ff9700 1 -- 172.21.2.112:0/3200114544 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff834006030 con 0x7ff83c0c2cb0 2021-02-10T11:37:23.152 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:23.162 7ff838ff9700 1 --2- 172.21.2.112:0/3200114544 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff81c0186c0 0x7ff81c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:23.153 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:23.162 7ff838ff9700 1 -- 172.21.2.112:0/3200114544 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5573+0+0 (secure 0 0 0) 0x7ff8340327d0 con 0x7ff83c0c2cb0 2021-02-10T11:37:23.153 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:23.162 7ff83bfff700 1 --2- 172.21.2.112:0/3200114544 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff81c0186c0 0x7ff81c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:23.153 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:23.163 7ff83bfff700 1 --2- 172.21.2.112:0/3200114544 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff81c0186c0 0x7ff81c01abc0 crc :-1 s=READY pgs=830 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:23.181 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:23.193 7ff842625700 1 -- 172.21.2.112:0/3200114544 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff83c0f0b10 con 0x7ff83c0c2cb0 2021-02-10T11:37:23.183 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:23.196 7ff838ff9700 1 -- 172.21.2.112:0/3200114544 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff83402dfc0 con 0x7ff83c0c2cb0 2021-02-10T11:37:23.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:23.322 7ff842625700 1 -- 172.21.2.112:0/3200114544 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff83c0f0b10 con 0x7ff83c0c2cb0 2021-02-10T11:37:23.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:23.322 7ff838ff9700 1 -- 172.21.2.112:0/3200114544 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) v1 ==== 94+0+2101 (secure 0 0 0) 0x7ff83402dfc0 con 0x7ff83c0c2cb0 2021-02-10T11:37:23.311 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 65 2021-02-10T11:37:23.311 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:37:23.311 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":65,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9204,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/1512510827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":1512510827},{"type":"v1","addr":"172.21.2.112:6833","nonce":1512510827}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":60}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:36:53.563735","modified":"2021-02-10 11:36:55.573740","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9201},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9201":{"gid":9201,"name":"b","rank":0,"incarnation":61,"state":"up:active","state_seq":3,"addr":"172.21.2.120:6835/1232001750","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":1232001750},{"type":"v1","addr":"172.21.2.120:6835","nonce":1232001750}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2021-02-10T11:37:23.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:23.324 7ff842625700 1 -- 172.21.2.112:0/3200114544 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff81c0186c0 msgr2=0x7ff81c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:23.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:23.324 7ff842625700 1 --2- 172.21.2.112:0/3200114544 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff81c0186c0 0x7ff81c01abc0 crc :-1 s=READY pgs=830 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:23.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:23.324 7ff842625700 1 -- 172.21.2.112:0/3200114544 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff83c0c2cb0 msgr2=0x7ff83c0bfd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:23.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:23.324 7ff842625700 1 --2- 172.21.2.112:0/3200114544 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff83c0c2cb0 0x7ff83c0bfd20 secure :-1 s=READY pgs=1756 cs=0 l=1 rev1=1 rx=0x7ff834009980 tx=0x7ff83400bcd0).stop 2021-02-10T11:37:23.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:23.324 7ff842625700 1 -- 172.21.2.112:0/3200114544 shutdown_connections 2021-02-10T11:37:23.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:23.324 7ff842625700 1 --2- 172.21.2.112:0/3200114544 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff81c0186c0 0x7ff81c01abc0 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:23.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:23.324 7ff842625700 1 --2- 172.21.2.112:0/3200114544 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff83c0c2cb0 0x7ff83c0bfd20 unknown :-1 s=CLOSED pgs=1756 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:23.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:23.324 7ff842625700 1 --2- 172.21.2.112:0/3200114544 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff83c0c2290 0x7ff83c0c2690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:23.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:23.324 7ff842625700 1 --2- 172.21.2.112:0/3200114544 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff83c1a4a60 0x7ff83c0c1c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:23.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:23.324 7ff842625700 1 -- 172.21.2.112:0/3200114544 >> 172.21.2.112:0/3200114544 conn(0x7ff83c0ebe30 msgr2=0x7ff83c0bf200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:23.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:23.325 7ff842625700 1 -- 172.21.2.112:0/3200114544 shutdown_connections 2021-02-10T11:37:23.314 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:23.325 7ff842625700 1 -- 172.21.2.112:0/3200114544 wait complete. 2021-02-10T11:37:23.321 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok session ls 2021-02-10T11:37:23.718 INFO:teuthology.orchestra.run.gibba020.stdout:[ 2021-02-10T11:37:23.718 INFO:teuthology.orchestra.run.gibba020.stdout: { 2021-02-10T11:37:23.718 INFO:teuthology.orchestra.run.gibba020.stdout: "id": 7390, 2021-02-10T11:37:23.718 INFO:teuthology.orchestra.run.gibba020.stdout: "entity": { 2021-02-10T11:37:23.718 INFO:teuthology.orchestra.run.gibba020.stdout: "name": { 2021-02-10T11:37:23.718 INFO:teuthology.orchestra.run.gibba020.stdout: "type": "client", 2021-02-10T11:37:23.718 INFO:teuthology.orchestra.run.gibba020.stdout: "num": 7390 2021-02-10T11:37:23.718 INFO:teuthology.orchestra.run.gibba020.stdout: }, 2021-02-10T11:37:23.719 INFO:teuthology.orchestra.run.gibba020.stdout: "addr": { 2021-02-10T11:37:23.719 INFO:teuthology.orchestra.run.gibba020.stdout: "type": "any", 2021-02-10T11:37:23.719 INFO:teuthology.orchestra.run.gibba020.stdout: "addr": "172.21.2.120:0", 2021-02-10T11:37:23.719 INFO:teuthology.orchestra.run.gibba020.stdout: "nonce": 1069935987 2021-02-10T11:37:23.719 INFO:teuthology.orchestra.run.gibba020.stdout: } 2021-02-10T11:37:23.719 INFO:teuthology.orchestra.run.gibba020.stdout: }, 2021-02-10T11:37:23.719 INFO:teuthology.orchestra.run.gibba020.stdout: "state": "open", 2021-02-10T11:37:23.719 INFO:teuthology.orchestra.run.gibba020.stdout: "num_leases": 1, 2021-02-10T11:37:23.719 INFO:teuthology.orchestra.run.gibba020.stdout: "num_caps": 7, 2021-02-10T11:37:23.719 INFO:teuthology.orchestra.run.gibba020.stdout: "request_load_avg": 13, 2021-02-10T11:37:23.719 INFO:teuthology.orchestra.run.gibba020.stdout: "uptime": 2.0688887309999999, 2021-02-10T11:37:23.719 INFO:teuthology.orchestra.run.gibba020.stdout: "requests_in_flight": 0, 2021-02-10T11:37:23.719 INFO:teuthology.orchestra.run.gibba020.stdout: "completed_requests": 1, 2021-02-10T11:37:23.720 INFO:teuthology.orchestra.run.gibba020.stdout: "reconnecting": false, 2021-02-10T11:37:23.720 INFO:teuthology.orchestra.run.gibba020.stdout: "recall_caps": { 2021-02-10T11:37:23.720 INFO:teuthology.orchestra.run.gibba020.stdout: "value": 0, 2021-02-10T11:37:23.720 INFO:teuthology.orchestra.run.gibba020.stdout: "halflife": 60 2021-02-10T11:37:23.720 INFO:teuthology.orchestra.run.gibba020.stdout: }, 2021-02-10T11:37:23.720 INFO:teuthology.orchestra.run.gibba020.stdout: "release_caps": { 2021-02-10T11:37:23.720 INFO:teuthology.orchestra.run.gibba020.stdout: "value": 0, 2021-02-10T11:37:23.720 INFO:teuthology.orchestra.run.gibba020.stdout: "halflife": 60 2021-02-10T11:37:23.720 INFO:teuthology.orchestra.run.gibba020.stdout: }, 2021-02-10T11:37:23.720 INFO:teuthology.orchestra.run.gibba020.stdout: "recall_caps_throttle": { 2021-02-10T11:37:23.720 INFO:teuthology.orchestra.run.gibba020.stdout: "value": 0, 2021-02-10T11:37:23.720 INFO:teuthology.orchestra.run.gibba020.stdout: "halflife": 2.5 2021-02-10T11:37:23.720 INFO:teuthology.orchestra.run.gibba020.stdout: }, 2021-02-10T11:37:23.721 INFO:teuthology.orchestra.run.gibba020.stdout: "recall_caps_throttle2o": { 2021-02-10T11:37:23.721 INFO:teuthology.orchestra.run.gibba020.stdout: "value": 0, 2021-02-10T11:37:23.721 INFO:teuthology.orchestra.run.gibba020.stdout: "halflife": 0.5 2021-02-10T11:37:23.721 INFO:teuthology.orchestra.run.gibba020.stdout: }, 2021-02-10T11:37:23.721 INFO:teuthology.orchestra.run.gibba020.stdout: "session_cache_liveness": { 2021-02-10T11:37:23.721 INFO:teuthology.orchestra.run.gibba020.stdout: "value": 19.905190826106637, 2021-02-10T11:37:23.721 INFO:teuthology.orchestra.run.gibba020.stdout: "halflife": 300 2021-02-10T11:37:23.721 INFO:teuthology.orchestra.run.gibba020.stdout: }, 2021-02-10T11:37:23.721 INFO:teuthology.orchestra.run.gibba020.stdout: "cap_acquisition": { 2021-02-10T11:37:23.721 INFO:teuthology.orchestra.run.gibba020.stdout: "value": 0, 2021-02-10T11:37:23.721 INFO:teuthology.orchestra.run.gibba020.stdout: "halflife": 10 2021-02-10T11:37:23.721 INFO:teuthology.orchestra.run.gibba020.stdout: }, 2021-02-10T11:37:23.721 INFO:teuthology.orchestra.run.gibba020.stdout: "inst": "client.7390 172.21.2.120:0/1069935987", 2021-02-10T11:37:23.721 INFO:teuthology.orchestra.run.gibba020.stdout: "completed_requests": [ 2021-02-10T11:37:23.722 INFO:teuthology.orchestra.run.gibba020.stdout: { 2021-02-10T11:37:23.722 INFO:teuthology.orchestra.run.gibba020.stdout: "tid": 14, 2021-02-10T11:37:23.722 INFO:teuthology.orchestra.run.gibba020.stdout: "created_ino": "0x10000000005" 2021-02-10T11:37:23.722 INFO:teuthology.orchestra.run.gibba020.stdout: } 2021-02-10T11:37:23.722 INFO:teuthology.orchestra.run.gibba020.stdout: ], 2021-02-10T11:37:23.722 INFO:teuthology.orchestra.run.gibba020.stdout: "prealloc_inos": [ 2021-02-10T11:37:23.722 INFO:teuthology.orchestra.run.gibba020.stdout: { 2021-02-10T11:37:23.722 INFO:teuthology.orchestra.run.gibba020.stdout: "start": 1099511627782, 2021-02-10T11:37:23.722 INFO:teuthology.orchestra.run.gibba020.stdout: "length": 995 2021-02-10T11:37:23.722 INFO:teuthology.orchestra.run.gibba020.stdout: } 2021-02-10T11:37:23.722 INFO:teuthology.orchestra.run.gibba020.stdout: ], 2021-02-10T11:37:23.722 INFO:teuthology.orchestra.run.gibba020.stdout: "used_inos": [], 2021-02-10T11:37:23.723 INFO:teuthology.orchestra.run.gibba020.stdout: "client_metadata": { 2021-02-10T11:37:23.723 INFO:teuthology.orchestra.run.gibba020.stdout: "features": "0x0000000000001fff", 2021-02-10T11:37:23.723 INFO:teuthology.orchestra.run.gibba020.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:37:23.723 INFO:teuthology.orchestra.run.gibba020.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:37:23.723 INFO:teuthology.orchestra.run.gibba020.stdout: "entity_id": "x", 2021-02-10T11:37:23.723 INFO:teuthology.orchestra.run.gibba020.stdout: "hostname": "gibba020", 2021-02-10T11:37:23.723 INFO:teuthology.orchestra.run.gibba020.stdout: "pid": "34990", 2021-02-10T11:37:23.723 INFO:teuthology.orchestra.run.gibba020.stdout: "root": "/" 2021-02-10T11:37:23.723 INFO:teuthology.orchestra.run.gibba020.stdout: } 2021-02-10T11:37:23.723 INFO:teuthology.orchestra.run.gibba020.stdout: } 2021-02-10T11:37:23.723 INFO:teuthology.orchestra.run.gibba020.stdout:] 2021-02-10T11:37:23.724 INFO:tasks.cephfs.filesystem:_json_asok output: [ { "id": 7390, "entity": { "name": { "type": "client", "num": 7390 }, "addr": { "type": "any", "addr": "172.21.2.120:0", "nonce": 1069935987 } }, "state": "open", "num_leases": 1, "num_caps": 7, "request_load_avg": 13, "uptime": 2.0688887309999999, "requests_in_flight": 0, "completed_requests": 1, "reconnecting": false, "recall_caps": { "value": 0, "halflife": 60 }, "release_caps": { "value": 0, "halflife": 60 }, "recall_caps_throttle": { "value": 0, "halflife": 2.5 }, "recall_caps_throttle2o": { "value": 0, "halflife": 0.5 }, "session_cache_liveness": { "value": 19.905190826106637, "halflife": 300 }, "cap_acquisition": { "value": 0, "halflife": 10 }, "inst": "client.7390 172.21.2.120:0/1069935987", "completed_requests": [ { "tid": 14, "created_ino": "0x10000000005" } ], "prealloc_inos": [ { "start": 1099511627782, "length": 995 } ], "used_inos": [], "client_metadata": { "features": "0x0000000000001fff", "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", "entity_id": "x", "hostname": "gibba020", "pid": "34990", "root": "/" } } ] 2021-02-10T11:37:23.725 DEBUG:tasks.ceph_test_case:wait_until_true: waiting... 2021-02-10T11:37:28.725 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:37:29.154 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:29.148 7f880f4ab700 1 Processor -- start 2021-02-10T11:37:29.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:29.148 7f880f4ab700 1 -- start start 2021-02-10T11:37:29.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:29.149 7f880f4ab700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f88080d1c70 0x7f88080f20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:29.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:29.149 7f880f4ab700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88080f0b10 con 0x7f88080d1870 2021-02-10T11:37:29.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:29.149 7f880f4ab700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88081b0be0 con 0x7f88080d0340 2021-02-10T11:37:29.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:29.149 7f880f4ab700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88080e5ed0 con 0x7f88080d1c70 2021-02-10T11:37:29.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:29.149 7f880d2c7700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f88080d1870 0x7f88081b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:52296/0 (socket says 172.21.2.112:52296) 2021-02-10T11:37:29.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:29.149 7f880d2c7700 1 -- 172.21.2.112:0/1361767714 learned_addr learned my addr 172.21.2.112:0/1361767714 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:37:29.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:29.149 7f880dac8700 1 --2- 172.21.2.112:0/1361767714 >> v2:172.21.2.120:3300/0 conn(0x7f88080d1c70 0x7f88080f20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:29.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:29.149 7f87fffff700 1 -- 172.21.2.112:0/1361767714 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2907140470 0 0) 0x7f88040010b0 con 0x7f88080d1870 2021-02-10T11:37:29.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:29.149 7f87fffff700 1 -- 172.21.2.112:0/1361767714 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87f4002fe0 con 0x7f88080d1870 2021-02-10T11:37:29.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:29.150 7f87fffff700 1 -- 172.21.2.112:0/1361767714 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1712819858 0 0) 0x7f87f8000ff0 con 0x7f88080d0340 2021-02-10T11:37:29.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:29.150 7f87fffff700 1 -- 172.21.2.112:0/1361767714 --> v1:172.21.2.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87f4003210 con 0x7f88080d0340 2021-02-10T11:37:29.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:29.150 7f880dac8700 1 -- 172.21.2.112:0/1361767714 >> v1:172.21.2.120:6789/0 conn(0x7f88080d0340 legacy=0x7f88080ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:29.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:29.150 7f880dac8700 1 -- 172.21.2.112:0/1361767714 >> v1:172.21.2.112:6789/0 conn(0x7f88080d1870 legacy=0x7f88081b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:29.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:29.150 7f87fffff700 1 -- 172.21.2.112:0/1361767714 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2605132037 0 0) 0x7f88040010b0 con 0x7f88080d1870 2021-02-10T11:37:29.169 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:29.150 7f880dac8700 1 -- 172.21.2.112:0/1361767714 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88080efe30 con 0x7f88080d1c70 2021-02-10T11:37:29.169 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:29.150 7f880dac8700 1 --2- 172.21.2.112:0/1361767714 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f88080d1c70 0x7f88080f20a0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 rx=0x7f8800006e50 tx=0x7f8800009e70).ready entity=mon.1 client_cookie=775e3922b87b2fce server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:29.169 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:29.150 7f87fffff700 1 -- 172.21.2.112:0/1361767714 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f880000b510 con 0x7f88080d1c70 2021-02-10T11:37:29.169 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:29.150 7f87fffff700 1 -- 172.21.2.112:0/1361767714 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8800006790 con 0x7f88080d1c70 2021-02-10T11:37:29.169 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:29.150 7f87fffff700 1 -- 172.21.2.112:0/1361767714 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f880000b940 con 0x7f88080d1c70 2021-02-10T11:37:29.169 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:29.150 7f880f4ab700 1 -- 172.21.2.112:0/1361767714 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f88080d1c70 msgr2=0x7f88080f20a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:29.169 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:29.150 7f880f4ab700 1 --2- 172.21.2.112:0/1361767714 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f88080d1c70 0x7f88080f20a0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 rx=0x7f8800006e50 tx=0x7f8800009e70).stop 2021-02-10T11:37:29.169 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:29.151 7f880f4ab700 1 -- 172.21.2.112:0/1361767714 shutdown_connections 2021-02-10T11:37:29.169 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:29.151 7f880f4ab700 1 --2- 172.21.2.112:0/1361767714 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f88080d1c70 0x7f88080f20a0 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:29.169 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:29.151 7f880f4ab700 1 -- 172.21.2.112:0/1361767714 >> 172.21.2.112:0/1361767714 conn(0x7f88080ce450 msgr2=0x7f88080d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:29.169 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:29.151 7f880f4ab700 1 -- 172.21.2.112:0/1361767714 shutdown_connections 2021-02-10T11:37:29.169 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:29.151 7f880f4ab700 1 -- 172.21.2.112:0/1361767714 wait complete. 2021-02-10T11:37:29.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:29.166 7f880f4ab700 1 Processor -- start 2021-02-10T11:37:29.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:29.166 7f880f4ab700 1 -- start start 2021-02-10T11:37:29.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:29.166 7f880f4ab700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f88080ada90 0x7f88080ade90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:29.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:29.166 7f880f4ab700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8808099d20 0x7f880809a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:29.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:29.166 7f880f4ab700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f880809a740 0x7f8808097bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:29.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:29.166 7f880f4ab700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88080997a0 con 0x7f8808099d20 2021-02-10T11:37:29.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:29.166 7f880f4ab700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f880818f0c0 con 0x7f88080ada90 2021-02-10T11:37:29.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:29.166 7f880f4ab700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f880809ab40 con 0x7f880809a740 2021-02-10T11:37:29.172 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:29.166 7f880cac6700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8808099d20 0x7f880809a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:29.172 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:29.166 7f880cac6700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8808099d20 0x7f880809a120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:52840/0 (socket says 172.21.2.112:52840) 2021-02-10T11:37:29.172 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:29.167 7f880cac6700 1 -- 172.21.2.112:0/3679260876 learned_addr learned my addr 172.21.2.112:0/3679260876 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:37:29.172 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:29.167 7f880cac6700 1 -- 172.21.2.112:0/3679260876 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f880809a740 msgr2=0x7f8808097bc0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:37:29.172 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:29.167 7f880cac6700 1 --2- 172.21.2.112:0/3679260876 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f880809a740 0x7f8808097bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:29.172 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:29.167 7f880cac6700 1 -- 172.21.2.112:0/3679260876 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f88080ada90 msgr2=0x7f88080ade90 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:37:29.172 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:29.167 7f880cac6700 1 --2- 172.21.2.112:0/3679260876 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f88080ada90 0x7f88080ade90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:29.172 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:29.167 7f880cac6700 1 -- 172.21.2.112:0/3679260876 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f880809ad60 con 0x7f8808099d20 2021-02-10T11:37:29.172 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:29.167 7f880cac6700 1 --2- 172.21.2.112:0/3679260876 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8808099d20 0x7f880809a120 secure :-1 s=READY pgs=1758 cs=0 l=1 rev1=1 rx=0x7f87f8006f20 tx=0x7f87f800b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:29.172 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:29.167 7f87fdffb700 1 -- 172.21.2.112:0/3679260876 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f87f800bd50 con 0x7f8808099d20 2021-02-10T11:37:29.172 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:29.167 7f87fdffb700 1 -- 172.21.2.112:0/3679260876 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87f80067e0 con 0x7f8808099d20 2021-02-10T11:37:29.172 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:29.167 7f880f4ab700 1 -- 172.21.2.112:0/3679260876 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f880809ad60 con 0x7f8808099d20 2021-02-10T11:37:29.172 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:29.167 7f880f4ab700 1 -- 172.21.2.112:0/3679260876 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f880809ad60 con 0x7f8808099d20 2021-02-10T11:37:29.172 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:29.167 7f87fdffb700 1 -- 172.21.2.112:0/3679260876 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f87f80012e0 con 0x7f8808099d20 2021-02-10T11:37:29.173 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:29.168 7f87fdffb700 1 -- 172.21.2.112:0/3679260876 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f87f800a000 con 0x7f8808099d20 2021-02-10T11:37:29.173 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:29.168 7f87fdffb700 1 --2- 172.21.2.112:0/3679260876 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f87e80186d0 0x7f87e801abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:29.173 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:29.168 7f87fdffb700 1 -- 172.21.2.112:0/3679260876 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f87f8032760 con 0x7f8808099d20 2021-02-10T11:37:29.173 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:29.169 7f880d2c7700 1 --2- 172.21.2.112:0/3679260876 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f87e80186d0 0x7f87e801abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:29.173 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:29.169 7f880d2c7700 1 --2- 172.21.2.112:0/3679260876 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f87e80186d0 0x7f87e801abd0 crc :-1 s=READY pgs=831 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:29.187 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:29.199 7f880f4ab700 1 -- 172.21.2.112:0/3679260876 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87e0002fd0 con 0x7f8808099d20 2021-02-10T11:37:29.190 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:29.202 7f87fdffb700 1 -- 172.21.2.112:0/3679260876 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f87f8006c10 con 0x7f8808099d20 2021-02-10T11:37:29.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:29.328 7f880f4ab700 1 -- 172.21.2.112:0/3679260876 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f87e0002de0 con 0x7f8808099d20 2021-02-10T11:37:29.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:29.329 7f87fdffb700 1 -- 172.21.2.112:0/3679260876 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) v1 ==== 94+0+2101 (secure 0 0 0) 0x7f87f800bd50 con 0x7f8808099d20 2021-02-10T11:37:29.318 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 65 2021-02-10T11:37:29.318 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:37:29.318 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":65,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9204,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/1512510827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":1512510827},{"type":"v1","addr":"172.21.2.112:6833","nonce":1512510827}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":60}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:36:53.563735","modified":"2021-02-10 11:36:55.573740","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9201},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9201":{"gid":9201,"name":"b","rank":0,"incarnation":61,"state":"up:active","state_seq":3,"addr":"172.21.2.120:6835/1232001750","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":1232001750},{"type":"v1","addr":"172.21.2.120:6835","nonce":1232001750}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2021-02-10T11:37:29.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:29.330 7f880f4ab700 1 -- 172.21.2.112:0/3679260876 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f87e80186d0 msgr2=0x7f87e801abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:29.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:29.330 7f880f4ab700 1 --2- 172.21.2.112:0/3679260876 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f87e80186d0 0x7f87e801abd0 crc :-1 s=READY pgs=831 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:29.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:29.330 7f880f4ab700 1 -- 172.21.2.112:0/3679260876 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8808099d20 msgr2=0x7f880809a120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:29.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:29.330 7f880f4ab700 1 --2- 172.21.2.112:0/3679260876 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8808099d20 0x7f880809a120 secure :-1 s=READY pgs=1758 cs=0 l=1 rev1=1 rx=0x7f87f8006f20 tx=0x7f87f800b960).stop 2021-02-10T11:37:29.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:29.331 7f880f4ab700 1 -- 172.21.2.112:0/3679260876 shutdown_connections 2021-02-10T11:37:29.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:29.331 7f880f4ab700 1 --2- 172.21.2.112:0/3679260876 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f87e80186d0 0x7f87e801abd0 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:29.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:29.331 7f880f4ab700 1 --2- 172.21.2.112:0/3679260876 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f880809a740 0x7f8808097bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:29.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:29.331 7f880f4ab700 1 --2- 172.21.2.112:0/3679260876 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8808099d20 0x7f880809a120 unknown :-1 s=CLOSED pgs=1758 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:29.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:29.331 7f880f4ab700 1 --2- 172.21.2.112:0/3679260876 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f88080ada90 0x7f88080ade90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:29.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:29.331 7f880f4ab700 1 -- 172.21.2.112:0/3679260876 >> 172.21.2.112:0/3679260876 conn(0x7f88080f0550 msgr2=0x7f8808191730 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:29.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:29.331 7f880f4ab700 1 -- 172.21.2.112:0/3679260876 shutdown_connections 2021-02-10T11:37:29.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:29.331 7f880f4ab700 1 -- 172.21.2.112:0/3679260876 wait complete. 2021-02-10T11:37:29.328 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok session ls 2021-02-10T11:37:29.724 INFO:teuthology.orchestra.run.gibba020.stdout:[ 2021-02-10T11:37:29.724 INFO:teuthology.orchestra.run.gibba020.stdout: { 2021-02-10T11:37:29.724 INFO:teuthology.orchestra.run.gibba020.stdout: "id": 7390, 2021-02-10T11:37:29.724 INFO:teuthology.orchestra.run.gibba020.stdout: "entity": { 2021-02-10T11:37:29.724 INFO:teuthology.orchestra.run.gibba020.stdout: "name": { 2021-02-10T11:37:29.724 INFO:teuthology.orchestra.run.gibba020.stdout: "type": "client", 2021-02-10T11:37:29.724 INFO:teuthology.orchestra.run.gibba020.stdout: "num": 7390 2021-02-10T11:37:29.724 INFO:teuthology.orchestra.run.gibba020.stdout: }, 2021-02-10T11:37:29.724 INFO:teuthology.orchestra.run.gibba020.stdout: "addr": { 2021-02-10T11:37:29.724 INFO:teuthology.orchestra.run.gibba020.stdout: "type": "any", 2021-02-10T11:37:29.725 INFO:teuthology.orchestra.run.gibba020.stdout: "addr": "172.21.2.120:0", 2021-02-10T11:37:29.725 INFO:teuthology.orchestra.run.gibba020.stdout: "nonce": 1069935987 2021-02-10T11:37:29.725 INFO:teuthology.orchestra.run.gibba020.stdout: } 2021-02-10T11:37:29.725 INFO:teuthology.orchestra.run.gibba020.stdout: }, 2021-02-10T11:37:29.725 INFO:teuthology.orchestra.run.gibba020.stdout: "state": "open", 2021-02-10T11:37:29.725 INFO:teuthology.orchestra.run.gibba020.stdout: "num_leases": 1, 2021-02-10T11:37:29.725 INFO:teuthology.orchestra.run.gibba020.stdout: "num_caps": 7, 2021-02-10T11:37:29.725 INFO:teuthology.orchestra.run.gibba020.stdout: "request_load_avg": 12, 2021-02-10T11:37:29.725 INFO:teuthology.orchestra.run.gibba020.stdout: "uptime": 8.0755694039999995, 2021-02-10T11:37:29.725 INFO:teuthology.orchestra.run.gibba020.stdout: "requests_in_flight": 0, 2021-02-10T11:37:29.725 INFO:teuthology.orchestra.run.gibba020.stdout: "completed_requests": 1, 2021-02-10T11:37:29.725 INFO:teuthology.orchestra.run.gibba020.stdout: "reconnecting": false, 2021-02-10T11:37:29.726 INFO:teuthology.orchestra.run.gibba020.stdout: "recall_caps": { 2021-02-10T11:37:29.726 INFO:teuthology.orchestra.run.gibba020.stdout: "value": 0, 2021-02-10T11:37:29.726 INFO:teuthology.orchestra.run.gibba020.stdout: "halflife": 60 2021-02-10T11:37:29.726 INFO:teuthology.orchestra.run.gibba020.stdout: }, 2021-02-10T11:37:29.726 INFO:teuthology.orchestra.run.gibba020.stdout: "release_caps": { 2021-02-10T11:37:29.726 INFO:teuthology.orchestra.run.gibba020.stdout: "value": 0, 2021-02-10T11:37:29.726 INFO:teuthology.orchestra.run.gibba020.stdout: "halflife": 60 2021-02-10T11:37:29.726 INFO:teuthology.orchestra.run.gibba020.stdout: }, 2021-02-10T11:37:29.726 INFO:teuthology.orchestra.run.gibba020.stdout: "recall_caps_throttle": { 2021-02-10T11:37:29.726 INFO:teuthology.orchestra.run.gibba020.stdout: "value": 0, 2021-02-10T11:37:29.726 INFO:teuthology.orchestra.run.gibba020.stdout: "halflife": 2.5 2021-02-10T11:37:29.726 INFO:teuthology.orchestra.run.gibba020.stdout: }, 2021-02-10T11:37:29.727 INFO:teuthology.orchestra.run.gibba020.stdout: "recall_caps_throttle2o": { 2021-02-10T11:37:29.727 INFO:teuthology.orchestra.run.gibba020.stdout: "value": 0, 2021-02-10T11:37:29.727 INFO:teuthology.orchestra.run.gibba020.stdout: "halflife": 0.5 2021-02-10T11:37:29.727 INFO:teuthology.orchestra.run.gibba020.stdout: }, 2021-02-10T11:37:29.727 INFO:teuthology.orchestra.run.gibba020.stdout: "session_cache_liveness": { 2021-02-10T11:37:29.727 INFO:teuthology.orchestra.run.gibba020.stdout: "value": 20.626066922710365, 2021-02-10T11:37:29.727 INFO:teuthology.orchestra.run.gibba020.stdout: "halflife": 300 2021-02-10T11:37:29.727 INFO:teuthology.orchestra.run.gibba020.stdout: }, 2021-02-10T11:37:29.727 INFO:teuthology.orchestra.run.gibba020.stdout: "cap_acquisition": { 2021-02-10T11:37:29.727 INFO:teuthology.orchestra.run.gibba020.stdout: "value": 0, 2021-02-10T11:37:29.727 INFO:teuthology.orchestra.run.gibba020.stdout: "halflife": 10 2021-02-10T11:37:29.727 INFO:teuthology.orchestra.run.gibba020.stdout: }, 2021-02-10T11:37:29.727 INFO:teuthology.orchestra.run.gibba020.stdout: "inst": "client.7390 172.21.2.120:0/1069935987", 2021-02-10T11:37:29.727 INFO:teuthology.orchestra.run.gibba020.stdout: "completed_requests": [ 2021-02-10T11:37:29.728 INFO:teuthology.orchestra.run.gibba020.stdout: { 2021-02-10T11:37:29.728 INFO:teuthology.orchestra.run.gibba020.stdout: "tid": 14, 2021-02-10T11:37:29.728 INFO:teuthology.orchestra.run.gibba020.stdout: "created_ino": "0x10000000005" 2021-02-10T11:37:29.728 INFO:teuthology.orchestra.run.gibba020.stdout: } 2021-02-10T11:37:29.728 INFO:teuthology.orchestra.run.gibba020.stdout: ], 2021-02-10T11:37:29.728 INFO:teuthology.orchestra.run.gibba020.stdout: "prealloc_inos": [ 2021-02-10T11:37:29.728 INFO:teuthology.orchestra.run.gibba020.stdout: { 2021-02-10T11:37:29.728 INFO:teuthology.orchestra.run.gibba020.stdout: "start": 1099511627782, 2021-02-10T11:37:29.728 INFO:teuthology.orchestra.run.gibba020.stdout: "length": 995 2021-02-10T11:37:29.728 INFO:teuthology.orchestra.run.gibba020.stdout: } 2021-02-10T11:37:29.728 INFO:teuthology.orchestra.run.gibba020.stdout: ], 2021-02-10T11:37:29.728 INFO:teuthology.orchestra.run.gibba020.stdout: "used_inos": [], 2021-02-10T11:37:29.728 INFO:teuthology.orchestra.run.gibba020.stdout: "client_metadata": { 2021-02-10T11:37:29.729 INFO:teuthology.orchestra.run.gibba020.stdout: "features": "0x0000000000001fff", 2021-02-10T11:37:29.729 INFO:teuthology.orchestra.run.gibba020.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:37:29.729 INFO:teuthology.orchestra.run.gibba020.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:37:29.729 INFO:teuthology.orchestra.run.gibba020.stdout: "entity_id": "x", 2021-02-10T11:37:29.729 INFO:teuthology.orchestra.run.gibba020.stdout: "hostname": "gibba020", 2021-02-10T11:37:29.729 INFO:teuthology.orchestra.run.gibba020.stdout: "pid": "34990", 2021-02-10T11:37:29.729 INFO:teuthology.orchestra.run.gibba020.stdout: "root": "/" 2021-02-10T11:37:29.729 INFO:teuthology.orchestra.run.gibba020.stdout: } 2021-02-10T11:37:29.729 INFO:teuthology.orchestra.run.gibba020.stdout: } 2021-02-10T11:37:29.729 INFO:teuthology.orchestra.run.gibba020.stdout:] 2021-02-10T11:37:29.730 INFO:tasks.cephfs.filesystem:_json_asok output: [ { "id": 7390, "entity": { "name": { "type": "client", "num": 7390 }, "addr": { "type": "any", "addr": "172.21.2.120:0", "nonce": 1069935987 } }, "state": "open", "num_leases": 1, "num_caps": 7, "request_load_avg": 12, "uptime": 8.0755694039999995, "requests_in_flight": 0, "completed_requests": 1, "reconnecting": false, "recall_caps": { "value": 0, "halflife": 60 }, "release_caps": { "value": 0, "halflife": 60 }, "recall_caps_throttle": { "value": 0, "halflife": 2.5 }, "recall_caps_throttle2o": { "value": 0, "halflife": 0.5 }, "session_cache_liveness": { "value": 20.626066922710365, "halflife": 300 }, "cap_acquisition": { "value": 0, "halflife": 10 }, "inst": "client.7390 172.21.2.120:0/1069935987", "completed_requests": [ { "tid": 14, "created_ino": "0x10000000005" } ], "prealloc_inos": [ { "start": 1099511627782, "length": 995 } ], "used_inos": [], "client_metadata": { "features": "0x0000000000001fff", "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", "entity_id": "x", "hostname": "gibba020", "pid": "34990", "root": "/" } } ] 2021-02-10T11:37:29.730 DEBUG:tasks.ceph_test_case:wait_until_true: waiting... 2021-02-10T11:37:34.731 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:37:35.161 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:35.155 7eff60889700 1 Processor -- start 2021-02-10T11:37:35.161 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:35.155 7eff60889700 1 -- start start 2021-02-10T11:37:35.161 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:35.156 7eff60889700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7eff5c0d0340 0x7eff5c0ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:35.161 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:35.156 7eff60889700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff5c0f0b10 con 0x7eff5c0d1870 2021-02-10T11:37:35.161 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:35.156 7eff60889700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff5c1b0be0 con 0x7eff5c0d1c70 2021-02-10T11:37:35.161 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:35.156 7eff60889700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff5c0e5ed0 con 0x7eff5c0d0340 2021-02-10T11:37:35.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:35.156 7eff5a61c700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7eff5c0d1870 0x7eff5c1b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:52308/0 (socket says 172.21.2.112:52308) 2021-02-10T11:37:35.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:35.156 7eff5a61c700 1 -- 172.21.2.112:0/3707382235 learned_addr learned my addr 172.21.2.112:0/3707382235 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:37:35.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:35.156 7eff59e1b700 1 --2- 172.21.2.112:0/3707382235 >> v2:172.21.2.120:3300/0 conn(0x7eff5c0d0340 0x7eff5c0ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:35.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:35.157 7eff5961a700 1 -- 172.21.2.112:0/3707382235 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1621649782 0 0) 0x7eff4c001180 con 0x7eff5c0d1870 2021-02-10T11:37:35.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:35.157 7eff5961a700 1 -- 172.21.2.112:0/3707382235 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff44002fe0 con 0x7eff5c0d1870 2021-02-10T11:37:35.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:35.157 7eff59e1b700 1 -- 172.21.2.112:0/3707382235 >> v1:172.21.2.120:6789/0 conn(0x7eff5c0d1c70 legacy=0x7eff5c0efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:35.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:35.157 7eff59e1b700 1 -- 172.21.2.112:0/3707382235 >> v1:172.21.2.112:6789/0 conn(0x7eff5c0d1870 legacy=0x7eff5c1b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:35.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:35.157 7eff59e1b700 1 -- 172.21.2.112:0/3707382235 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff5c0f0450 con 0x7eff5c0d0340 2021-02-10T11:37:35.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:35.157 7eff59e1b700 1 --2- 172.21.2.112:0/3707382235 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7eff5c0d0340 0x7eff5c0ef640 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 rx=0x7eff50006650 tx=0x7eff50009a80).ready entity=mon.1 client_cookie=315d532486866bfc server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:35.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:35.157 7eff5961a700 1 -- 172.21.2.112:0/3707382235 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7eff500009b0 con 0x7eff5c0d0340 2021-02-10T11:37:35.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:35.157 7eff5961a700 1 -- 172.21.2.112:0/3707382235 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff5000c0c0 con 0x7eff5c0d0340 2021-02-10T11:37:35.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:35.157 7eff5961a700 1 -- 172.21.2.112:0/3707382235 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7eff5000c5e0 con 0x7eff5c0d0340 2021-02-10T11:37:35.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:35.157 7eff60889700 1 -- 172.21.2.112:0/3707382235 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7eff5c0d0340 msgr2=0x7eff5c0ef640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:35.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:35.157 7eff60889700 1 --2- 172.21.2.112:0/3707382235 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7eff5c0d0340 0x7eff5c0ef640 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 rx=0x7eff50006650 tx=0x7eff50009a80).stop 2021-02-10T11:37:35.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:35.158 7eff60889700 1 -- 172.21.2.112:0/3707382235 shutdown_connections 2021-02-10T11:37:35.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:35.158 7eff60889700 1 --2- 172.21.2.112:0/3707382235 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7eff5c0d0340 0x7eff5c0ef640 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:35.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:35.158 7eff60889700 1 -- 172.21.2.112:0/3707382235 >> 172.21.2.112:0/3707382235 conn(0x7eff5c0ce450 msgr2=0x7eff5c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:35.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:35.158 7eff60889700 1 -- 172.21.2.112:0/3707382235 shutdown_connections 2021-02-10T11:37:35.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:35.158 7eff60889700 1 -- 172.21.2.112:0/3707382235 wait complete. 2021-02-10T11:37:35.165 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:35.173 7eff60889700 1 Processor -- start 2021-02-10T11:37:35.165 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:35.173 7eff60889700 1 -- start start 2021-02-10T11:37:35.165 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:35.173 7eff60889700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7eff5c18f0a0 0x7eff5c196ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:35.165 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:35.173 7eff60889700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7eff5c1970d0 0x7eff5c1974d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:35.165 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:35.173 7eff60889700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7eff5c094650 0x7eff5c094a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:35.165 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:35.173 7eff60889700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff5c095bd0 con 0x7eff5c18f0a0 2021-02-10T11:37:35.166 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:35.173 7eff60889700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff5c0ad7c0 con 0x7eff5c1970d0 2021-02-10T11:37:35.166 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:35.173 7eff60889700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff5c18ec70 con 0x7eff5c094650 2021-02-10T11:37:35.166 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:35.173 7eff5a61c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7eff5c18f0a0 0x7eff5c196ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:35.166 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:35.173 7eff5a61c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7eff5c18f0a0 0x7eff5c196ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:52848/0 (socket says 172.21.2.112:52848) 2021-02-10T11:37:35.166 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:35.173 7eff5a61c700 1 -- 172.21.2.112:0/2383821804 learned_addr learned my addr 172.21.2.112:0/2383821804 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:37:35.166 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:35.174 7eff59e1b700 1 --2- 172.21.2.112:0/2383821804 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7eff5c1970d0 0x7eff5c1974d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:35.166 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:35.174 7eff5ae1d700 1 --2- 172.21.2.112:0/2383821804 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7eff5c094650 0x7eff5c094a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:35.166 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:35.174 7eff5a61c700 1 -- 172.21.2.112:0/2383821804 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7eff5c094650 msgr2=0x7eff5c094a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:35.166 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:35.174 7eff5a61c700 1 --2- 172.21.2.112:0/2383821804 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7eff5c094650 0x7eff5c094a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:35.166 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:35.174 7eff5a61c700 1 -- 172.21.2.112:0/2383821804 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7eff5c1970d0 msgr2=0x7eff5c1974d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:35.166 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:35.174 7eff5a61c700 1 --2- 172.21.2.112:0/2383821804 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7eff5c1970d0 0x7eff5c1974d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:35.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:35.174 7eff5a61c700 1 -- 172.21.2.112:0/2383821804 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff5c095070 con 0x7eff5c18f0a0 2021-02-10T11:37:35.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:35.174 7eff5ae1d700 1 --2- 172.21.2.112:0/2383821804 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7eff5c094650 0x7eff5c094a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:37:35.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:35.174 7eff5a61c700 1 --2- 172.21.2.112:0/2383821804 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7eff5c18f0a0 0x7eff5c196ab0 secure :-1 s=READY pgs=1760 cs=0 l=1 rev1=1 rx=0x7eff4c00a5d0 tx=0x7eff4c00a970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:35.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:35.174 7eff4affd700 1 -- 172.21.2.112:0/2383821804 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7eff4c000ce0 con 0x7eff5c18f0a0 2021-02-10T11:37:35.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:35.174 7eff4affd700 1 -- 172.21.2.112:0/2383821804 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff4c0043a0 con 0x7eff5c18f0a0 2021-02-10T11:37:35.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:35.174 7eff4affd700 1 -- 172.21.2.112:0/2383821804 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7eff4c004800 con 0x7eff5c18f0a0 2021-02-10T11:37:35.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:35.174 7eff60889700 1 -- 172.21.2.112:0/2383821804 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff5c095070 con 0x7eff5c18f0a0 2021-02-10T11:37:35.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:35.174 7eff60889700 1 -- 172.21.2.112:0/2383821804 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff5c095070 con 0x7eff5c18f0a0 2021-02-10T11:37:35.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:35.175 7eff4affd700 1 -- 172.21.2.112:0/2383821804 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7eff4c004580 con 0x7eff5c18f0a0 2021-02-10T11:37:35.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:35.175 7eff4affd700 1 --2- 172.21.2.112:0/2383821804 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7eff40018730 0x7eff4001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:35.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:35.175 7eff4affd700 1 -- 172.21.2.112:0/2383821804 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5573+0+0 (secure 0 0 0) 0x7eff4c0327c0 con 0x7eff5c18f0a0 2021-02-10T11:37:35.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:35.175 7eff59e1b700 1 --2- 172.21.2.112:0/2383821804 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7eff40018730 0x7eff4001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:35.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:35.176 7eff59e1b700 1 --2- 172.21.2.112:0/2383821804 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7eff40018730 0x7eff4001ac10 crc :-1 s=READY pgs=832 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:35.194 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:35.206 7eff60889700 1 -- 172.21.2.112:0/2383821804 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff5c0ad790 con 0x7eff5c18f0a0 2021-02-10T11:37:35.197 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:35.208 7eff4affd700 1 -- 172.21.2.112:0/2383821804 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7eff4c0045b0 con 0x7eff5c18f0a0 2021-02-10T11:37:35.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:35.336 7eff60889700 1 -- 172.21.2.112:0/2383821804 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7eff5c0ad790 con 0x7eff5c18f0a0 2021-02-10T11:37:35.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:35.337 7eff4affd700 1 -- 172.21.2.112:0/2383821804 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) v1 ==== 94+0+2101 (secure 0 0 0) 0x7eff4c0043a0 con 0x7eff5c18f0a0 2021-02-10T11:37:35.326 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:37:35.326 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":65,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9204,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/1512510827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":1512510827},{"type":"v1","addr":"172.21.2.112:6833","nonce":1512510827}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":60}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:36:53.563735","modified":"2021-02-10 11:36:55.573740","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9201},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9201":{"gid":9201,"name":"b","rank":0,"incarnation":61,"state":"up:active","state_seq":3,"addr":"172.21.2.120:6835/1232001750","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":1232001750},{"type":"v1","addr":"172.21.2.120:6835","nonce":1232001750}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2021-02-10T11:37:35.326 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 65 2021-02-10T11:37:35.327 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:35.339 7eff60889700 1 -- 172.21.2.112:0/2383821804 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7eff40018730 msgr2=0x7eff4001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:35.327 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:35.339 7eff60889700 1 --2- 172.21.2.112:0/2383821804 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7eff40018730 0x7eff4001ac10 crc :-1 s=READY pgs=832 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:35.327 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:35.339 7eff60889700 1 -- 172.21.2.112:0/2383821804 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7eff5c18f0a0 msgr2=0x7eff5c196ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:35.327 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:35.339 7eff60889700 1 --2- 172.21.2.112:0/2383821804 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7eff5c18f0a0 0x7eff5c196ab0 secure :-1 s=READY pgs=1760 cs=0 l=1 rev1=1 rx=0x7eff4c00a5d0 tx=0x7eff4c00a970).stop 2021-02-10T11:37:35.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:35.339 7eff60889700 1 -- 172.21.2.112:0/2383821804 shutdown_connections 2021-02-10T11:37:35.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:35.339 7eff60889700 1 --2- 172.21.2.112:0/2383821804 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7eff40018730 0x7eff4001ac10 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:35.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:35.339 7eff60889700 1 --2- 172.21.2.112:0/2383821804 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7eff5c1970d0 0x7eff5c1974d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:35.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:35.339 7eff60889700 1 --2- 172.21.2.112:0/2383821804 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7eff5c094650 0x7eff5c094a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:35.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:35.339 7eff60889700 1 --2- 172.21.2.112:0/2383821804 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7eff5c18f0a0 0x7eff5c196ab0 unknown :-1 s=CLOSED pgs=1760 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:35.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:35.339 7eff60889700 1 -- 172.21.2.112:0/2383821804 >> 172.21.2.112:0/2383821804 conn(0x7eff5c1a38c0 msgr2=0x7eff5c1a2f00 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:35.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:35.339 7eff60889700 1 -- 172.21.2.112:0/2383821804 shutdown_connections 2021-02-10T11:37:35.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:35.339 7eff60889700 1 -- 172.21.2.112:0/2383821804 wait complete. 2021-02-10T11:37:35.336 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok session ls 2021-02-10T11:37:35.731 INFO:teuthology.orchestra.run.gibba020.stdout:[ 2021-02-10T11:37:35.732 INFO:teuthology.orchestra.run.gibba020.stdout: { 2021-02-10T11:37:35.732 INFO:teuthology.orchestra.run.gibba020.stdout: "id": 7390, 2021-02-10T11:37:35.732 INFO:teuthology.orchestra.run.gibba020.stdout: "entity": { 2021-02-10T11:37:35.732 INFO:teuthology.orchestra.run.gibba020.stdout: "name": { 2021-02-10T11:37:35.732 INFO:teuthology.orchestra.run.gibba020.stdout: "type": "client", 2021-02-10T11:37:35.732 INFO:teuthology.orchestra.run.gibba020.stdout: "num": 7390 2021-02-10T11:37:35.732 INFO:teuthology.orchestra.run.gibba020.stdout: }, 2021-02-10T11:37:35.732 INFO:teuthology.orchestra.run.gibba020.stdout: "addr": { 2021-02-10T11:37:35.732 INFO:teuthology.orchestra.run.gibba020.stdout: "type": "any", 2021-02-10T11:37:35.732 INFO:teuthology.orchestra.run.gibba020.stdout: "addr": "172.21.2.120:0", 2021-02-10T11:37:35.732 INFO:teuthology.orchestra.run.gibba020.stdout: "nonce": 1069935987 2021-02-10T11:37:35.733 INFO:teuthology.orchestra.run.gibba020.stdout: } 2021-02-10T11:37:35.733 INFO:teuthology.orchestra.run.gibba020.stdout: }, 2021-02-10T11:37:35.733 INFO:teuthology.orchestra.run.gibba020.stdout: "state": "open", 2021-02-10T11:37:35.733 INFO:teuthology.orchestra.run.gibba020.stdout: "num_leases": 1, 2021-02-10T11:37:35.733 INFO:teuthology.orchestra.run.gibba020.stdout: "num_caps": 7, 2021-02-10T11:37:35.733 INFO:teuthology.orchestra.run.gibba020.stdout: "request_load_avg": 11, 2021-02-10T11:37:35.733 INFO:teuthology.orchestra.run.gibba020.stdout: "uptime": 14.082255519, 2021-02-10T11:37:35.733 INFO:teuthology.orchestra.run.gibba020.stdout: "requests_in_flight": 0, 2021-02-10T11:37:35.733 INFO:teuthology.orchestra.run.gibba020.stdout: "completed_requests": 1, 2021-02-10T11:37:35.733 INFO:teuthology.orchestra.run.gibba020.stdout: "reconnecting": false, 2021-02-10T11:37:35.733 INFO:teuthology.orchestra.run.gibba020.stdout: "recall_caps": { 2021-02-10T11:37:35.733 INFO:teuthology.orchestra.run.gibba020.stdout: "value": 0, 2021-02-10T11:37:35.733 INFO:teuthology.orchestra.run.gibba020.stdout: "halflife": 60 2021-02-10T11:37:35.734 INFO:teuthology.orchestra.run.gibba020.stdout: }, 2021-02-10T11:37:35.734 INFO:teuthology.orchestra.run.gibba020.stdout: "release_caps": { 2021-02-10T11:37:35.734 INFO:teuthology.orchestra.run.gibba020.stdout: "value": 0, 2021-02-10T11:37:35.734 INFO:teuthology.orchestra.run.gibba020.stdout: "halflife": 60 2021-02-10T11:37:35.734 INFO:teuthology.orchestra.run.gibba020.stdout: }, 2021-02-10T11:37:35.734 INFO:teuthology.orchestra.run.gibba020.stdout: "recall_caps_throttle": { 2021-02-10T11:37:35.734 INFO:teuthology.orchestra.run.gibba020.stdout: "value": 0, 2021-02-10T11:37:35.734 INFO:teuthology.orchestra.run.gibba020.stdout: "halflife": 2.5 2021-02-10T11:37:35.734 INFO:teuthology.orchestra.run.gibba020.stdout: }, 2021-02-10T11:37:35.734 INFO:teuthology.orchestra.run.gibba020.stdout: "recall_caps_throttle2o": { 2021-02-10T11:37:35.734 INFO:teuthology.orchestra.run.gibba020.stdout: "value": 0, 2021-02-10T11:37:35.734 INFO:teuthology.orchestra.run.gibba020.stdout: "halflife": 0.5 2021-02-10T11:37:35.735 INFO:teuthology.orchestra.run.gibba020.stdout: }, 2021-02-10T11:37:35.735 INFO:teuthology.orchestra.run.gibba020.stdout: "session_cache_liveness": { 2021-02-10T11:37:35.735 INFO:teuthology.orchestra.run.gibba020.stdout: "value": 20.341787517697046, 2021-02-10T11:37:35.735 INFO:teuthology.orchestra.run.gibba020.stdout: "halflife": 300 2021-02-10T11:37:35.735 INFO:teuthology.orchestra.run.gibba020.stdout: }, 2021-02-10T11:37:35.735 INFO:teuthology.orchestra.run.gibba020.stdout: "cap_acquisition": { 2021-02-10T11:37:35.735 INFO:teuthology.orchestra.run.gibba020.stdout: "value": 0, 2021-02-10T11:37:35.735 INFO:teuthology.orchestra.run.gibba020.stdout: "halflife": 10 2021-02-10T11:37:35.735 INFO:teuthology.orchestra.run.gibba020.stdout: }, 2021-02-10T11:37:35.735 INFO:teuthology.orchestra.run.gibba020.stdout: "inst": "client.7390 172.21.2.120:0/1069935987", 2021-02-10T11:37:35.735 INFO:teuthology.orchestra.run.gibba020.stdout: "completed_requests": [ 2021-02-10T11:37:35.735 INFO:teuthology.orchestra.run.gibba020.stdout: { 2021-02-10T11:37:35.735 INFO:teuthology.orchestra.run.gibba020.stdout: "tid": 14, 2021-02-10T11:37:35.735 INFO:teuthology.orchestra.run.gibba020.stdout: "created_ino": "0x10000000005" 2021-02-10T11:37:35.736 INFO:teuthology.orchestra.run.gibba020.stdout: } 2021-02-10T11:37:35.736 INFO:teuthology.orchestra.run.gibba020.stdout: ], 2021-02-10T11:37:35.736 INFO:teuthology.orchestra.run.gibba020.stdout: "prealloc_inos": [ 2021-02-10T11:37:35.736 INFO:teuthology.orchestra.run.gibba020.stdout: { 2021-02-10T11:37:35.736 INFO:teuthology.orchestra.run.gibba020.stdout: "start": 1099511627782, 2021-02-10T11:37:35.736 INFO:teuthology.orchestra.run.gibba020.stdout: "length": 995 2021-02-10T11:37:35.736 INFO:teuthology.orchestra.run.gibba020.stdout: } 2021-02-10T11:37:35.736 INFO:teuthology.orchestra.run.gibba020.stdout: ], 2021-02-10T11:37:35.736 INFO:teuthology.orchestra.run.gibba020.stdout: "used_inos": [], 2021-02-10T11:37:35.736 INFO:teuthology.orchestra.run.gibba020.stdout: "client_metadata": { 2021-02-10T11:37:35.736 INFO:teuthology.orchestra.run.gibba020.stdout: "features": "0x0000000000001fff", 2021-02-10T11:37:35.736 INFO:teuthology.orchestra.run.gibba020.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:37:35.736 INFO:teuthology.orchestra.run.gibba020.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:37:35.737 INFO:teuthology.orchestra.run.gibba020.stdout: "entity_id": "x", 2021-02-10T11:37:35.737 INFO:teuthology.orchestra.run.gibba020.stdout: "hostname": "gibba020", 2021-02-10T11:37:35.737 INFO:teuthology.orchestra.run.gibba020.stdout: "pid": "34990", 2021-02-10T11:37:35.737 INFO:teuthology.orchestra.run.gibba020.stdout: "root": "/" 2021-02-10T11:37:35.737 INFO:teuthology.orchestra.run.gibba020.stdout: } 2021-02-10T11:37:35.737 INFO:teuthology.orchestra.run.gibba020.stdout: } 2021-02-10T11:37:35.737 INFO:teuthology.orchestra.run.gibba020.stdout:] 2021-02-10T11:37:35.738 INFO:tasks.cephfs.filesystem:_json_asok output: [ { "id": 7390, "entity": { "name": { "type": "client", "num": 7390 }, "addr": { "type": "any", "addr": "172.21.2.120:0", "nonce": 1069935987 } }, "state": "open", "num_leases": 1, "num_caps": 7, "request_load_avg": 11, "uptime": 14.082255519, "requests_in_flight": 0, "completed_requests": 1, "reconnecting": false, "recall_caps": { "value": 0, "halflife": 60 }, "release_caps": { "value": 0, "halflife": 60 }, "recall_caps_throttle": { "value": 0, "halflife": 2.5 }, "recall_caps_throttle2o": { "value": 0, "halflife": 0.5 }, "session_cache_liveness": { "value": 20.341787517697046, "halflife": 300 }, "cap_acquisition": { "value": 0, "halflife": 10 }, "inst": "client.7390 172.21.2.120:0/1069935987", "completed_requests": [ { "tid": 14, "created_ino": "0x10000000005" } ], "prealloc_inos": [ { "start": 1099511627782, "length": 995 } ], "used_inos": [], "client_metadata": { "features": "0x0000000000001fff", "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", "entity_id": "x", "hostname": "gibba020", "pid": "34990", "root": "/" } } ] 2021-02-10T11:37:35.738 DEBUG:tasks.ceph_test_case:wait_until_true: waiting... 2021-02-10T11:37:40.739 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:37:41.166 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:41.161 7f151f34c700 1 Processor -- start 2021-02-10T11:37:41.166 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:41.161 7f151f34c700 1 -- start start 2021-02-10T11:37:41.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:41.161 7f151f34c700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f15180d1870 0x7f15181b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:41.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:41.161 7f151f34c700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f15180f0650 0x7f15180cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:41.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:41.161 7f151f34c700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15180f0b10 con 0x7f15181b0a10 2021-02-10T11:37:41.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:41.161 7f151f34c700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15180e5ed0 con 0x7f15180d1870 2021-02-10T11:37:41.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:41.161 7f151f34c700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15180d1c70 con 0x7f15180f0650 2021-02-10T11:37:41.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:41.162 7f151d168700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f15180d1870 0x7f15181b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:41.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:41.162 7f151c967700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f15181b0a10 0x7f15180cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:52320/0 (socket says 172.21.2.112:52320) 2021-02-10T11:37:41.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:41.162 7f151c967700 1 -- 172.21.2.112:0/3357671054 learned_addr learned my addr 172.21.2.112:0/3357671054 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:37:41.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:41.162 7f151d168700 1 -- 172.21.2.112:0/3357671054 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f15180f0650 msgr2=0x7f15180cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:37:41.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:41.162 7f151d168700 1 --2- 172.21.2.112:0/3357671054 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f15180f0650 0x7f15180cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:41.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:41.162 7f151d168700 1 -- 172.21.2.112:0/3357671054 >> v1:172.21.2.112:6789/0 conn(0x7f15181b0a10 legacy=0x7f15180cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:41.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:41.162 7f150ffff700 1 -- 172.21.2.112:0/3357671054 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1598864145 0 0) 0x7f15080009b0 con 0x7f15181b0a10 2021-02-10T11:37:41.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:41.162 7f151d168700 1 -- 172.21.2.112:0/3357671054 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15180d1e90 con 0x7f15180d1870 2021-02-10T11:37:41.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:41.162 7f151d168700 1 --2- 172.21.2.112:0/3357671054 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f15180d1870 0x7f15181b03f0 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 rx=0x7f1514006650 tx=0x7f1514009a00).ready entity=mon.0 client_cookie=166f6df9817ccf49 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:41.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:41.162 7f150ffff700 1 -- 172.21.2.112:0/3357671054 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f15140009b0 con 0x7f15180d1870 2021-02-10T11:37:41.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:41.163 7f150ffff700 1 -- 172.21.2.112:0/3357671054 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f151400c0c0 con 0x7f15180d1870 2021-02-10T11:37:41.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:41.163 7f150ffff700 1 -- 172.21.2.112:0/3357671054 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f151400c510 con 0x7f15180d1870 2021-02-10T11:37:41.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:41.163 7f151f34c700 1 -- 172.21.2.112:0/3357671054 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f15180d1870 msgr2=0x7f15181b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:41.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:41.163 7f151f34c700 1 --2- 172.21.2.112:0/3357671054 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f15180d1870 0x7f15181b03f0 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 rx=0x7f1514006650 tx=0x7f1514009a00).stop 2021-02-10T11:37:41.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:41.163 7f151f34c700 1 -- 172.21.2.112:0/3357671054 shutdown_connections 2021-02-10T11:37:41.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:41.163 7f151f34c700 1 --2- 172.21.2.112:0/3357671054 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f15180f0650 0x7f15180cfb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:41.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:41.163 7f151f34c700 1 --2- 172.21.2.112:0/3357671054 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f15180d1870 0x7f15181b03f0 unknown :-1 s=CLOSED pgs=1762 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:41.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:41.163 7f151f34c700 1 -- 172.21.2.112:0/3357671054 >> 172.21.2.112:0/3357671054 conn(0x7f15180ce450 msgr2=0x7f15180d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:41.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:41.164 7f151f34c700 1 -- 172.21.2.112:0/3357671054 shutdown_connections 2021-02-10T11:37:41.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:41.164 7f151f34c700 1 -- 172.21.2.112:0/3357671054 wait complete. 2021-02-10T11:37:41.170 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:41.178 7f151f34c700 1 Processor -- start 2021-02-10T11:37:41.170 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:41.178 7f151f34c700 1 -- start start 2021-02-10T11:37:41.170 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:41.178 7f151f34c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f15180b0ad0 0x7f1518091f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:41.170 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:41.178 7f151f34c700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f15181a8850 0x7f15181a8c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:41.170 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:41.178 7f151f34c700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f15181a6fa0 0x7f15181a73a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:41.170 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:41.178 7f151f34c700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15181abfe0 con 0x7f15180b0ad0 2021-02-10T11:37:41.170 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:41.178 7f151f34c700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15181a9c40 con 0x7f15181a6fa0 2021-02-10T11:37:41.170 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:41.178 7f151f34c700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15181a9730 con 0x7f15181a8850 2021-02-10T11:37:41.170 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:41.179 7f151d168700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f15180b0ad0 0x7f1518091f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:41.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:41.179 7f151d168700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f15180b0ad0 0x7f1518091f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:52862/0 (socket says 172.21.2.112:52862) 2021-02-10T11:37:41.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:41.179 7f151d168700 1 -- 172.21.2.112:0/678155676 learned_addr learned my addr 172.21.2.112:0/678155676 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:37:41.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:41.179 7f151d168700 1 -- 172.21.2.112:0/678155676 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f15181a8850 msgr2=0x7f15181a8c50 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:37:41.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:41.179 7f151d168700 1 --2- 172.21.2.112:0/678155676 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f15181a8850 0x7f15181a8c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:41.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:41.179 7f151d168700 1 -- 172.21.2.112:0/678155676 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f15181a6fa0 msgr2=0x7f15181a73a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:41.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:41.179 7f151d168700 1 --2- 172.21.2.112:0/678155676 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f15181a6fa0 0x7f15181a73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:41.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:41.179 7f151d168700 1 -- 172.21.2.112:0/678155676 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15181a9270 con 0x7f15180b0ad0 2021-02-10T11:37:41.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:41.179 7f151d168700 1 --2- 172.21.2.112:0/678155676 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f15180b0ad0 0x7f1518091f30 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 rx=0x7f151400b640 tx=0x7f151400b9e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:41.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:41.179 7f150dffb700 1 -- 172.21.2.112:0/678155676 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1514006ae0 con 0x7f15180b0ad0 2021-02-10T11:37:41.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:41.179 7f150dffb700 1 -- 172.21.2.112:0/678155676 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f151400bdd0 con 0x7f15180b0ad0 2021-02-10T11:37:41.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:41.179 7f151f34c700 1 -- 172.21.2.112:0/678155676 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15181a9270 con 0x7f15180b0ad0 2021-02-10T11:37:41.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:41.179 7f150dffb700 1 -- 172.21.2.112:0/678155676 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f15140022b0 con 0x7f15180b0ad0 2021-02-10T11:37:41.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:41.179 7f151f34c700 1 -- 172.21.2.112:0/678155676 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15181a7cb0 con 0x7f15180b0ad0 2021-02-10T11:37:41.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:41.180 7f150dffb700 1 -- 172.21.2.112:0/678155676 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1514000c90 con 0x7f15180b0ad0 2021-02-10T11:37:41.172 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:41.180 7f150dffb700 1 --2- 172.21.2.112:0/678155676 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f14f80186c0 0x7f14f801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:41.172 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:41.180 7f150dffb700 1 -- 172.21.2.112:0/678155676 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f1514032400 con 0x7f15180b0ad0 2021-02-10T11:37:41.172 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:41.181 7f151c967700 1 --2- 172.21.2.112:0/678155676 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f14f80186c0 0x7f14f801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:41.172 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:41.181 7f151c967700 1 --2- 172.21.2.112:0/678155676 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f14f80186c0 0x7f14f801abc0 crc :-1 s=READY pgs=833 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:41.200 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:41.211 7f151f34c700 1 -- 172.21.2.112:0/678155676 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15180ce450 con 0x7f15180b0ad0 2021-02-10T11:37:41.203 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:41.214 7f150dffb700 1 -- 172.21.2.112:0/678155676 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f151402e3b0 con 0x7f15180b0ad0 2021-02-10T11:37:41.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:41.343 7f151f34c700 1 -- 172.21.2.112:0/678155676 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f15180ce450 con 0x7f15180b0ad0 2021-02-10T11:37:41.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:41.343 7f150dffb700 1 -- 172.21.2.112:0/678155676 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) v1 ==== 94+0+2101 (secure 0 0 0) 0x7f151402e3b0 con 0x7f15180b0ad0 2021-02-10T11:37:41.333 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:37:41.333 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":65,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9204,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/1512510827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":1512510827},{"type":"v1","addr":"172.21.2.112:6833","nonce":1512510827}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":60}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:36:53.563735","modified":"2021-02-10 11:36:55.573740","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9201},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9201":{"gid":9201,"name":"b","rank":0,"incarnation":61,"state":"up:active","state_seq":3,"addr":"172.21.2.120:6835/1232001750","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":1232001750},{"type":"v1","addr":"172.21.2.120:6835","nonce":1232001750}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2021-02-10T11:37:41.333 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 65 2021-02-10T11:37:41.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:41.345 7f151f34c700 1 -- 172.21.2.112:0/678155676 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f14f80186c0 msgr2=0x7f14f801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:41.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:41.345 7f151f34c700 1 --2- 172.21.2.112:0/678155676 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f14f80186c0 0x7f14f801abc0 crc :-1 s=READY pgs=833 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:41.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:41.345 7f151f34c700 1 -- 172.21.2.112:0/678155676 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f15180b0ad0 msgr2=0x7f1518091f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:41.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:41.345 7f151f34c700 1 --2- 172.21.2.112:0/678155676 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f15180b0ad0 0x7f1518091f30 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 rx=0x7f151400b640 tx=0x7f151400b9e0).stop 2021-02-10T11:37:41.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:41.345 7f151f34c700 1 -- 172.21.2.112:0/678155676 shutdown_connections 2021-02-10T11:37:41.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:41.345 7f151f34c700 1 --2- 172.21.2.112:0/678155676 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f14f80186c0 0x7f14f801abc0 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:41.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:41.345 7f151f34c700 1 --2- 172.21.2.112:0/678155676 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f15181a6fa0 0x7f15181a73a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:41.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:41.345 7f151f34c700 1 --2- 172.21.2.112:0/678155676 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f15181a8850 0x7f15181a8c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:41.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:41.345 7f151f34c700 1 --2- 172.21.2.112:0/678155676 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f15180b0ad0 0x7f1518091f30 unknown :-1 s=CLOSED pgs=1763 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:41.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:41.345 7f151f34c700 1 -- 172.21.2.112:0/678155676 >> 172.21.2.112:0/678155676 conn(0x7f151819f8f0 msgr2=0x7f15181a5f80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:41.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:41.345 7f151f34c700 1 -- 172.21.2.112:0/678155676 shutdown_connections 2021-02-10T11:37:41.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:41.345 7f151f34c700 1 -- 172.21.2.112:0/678155676 wait complete. 2021-02-10T11:37:41.343 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok session ls 2021-02-10T11:37:41.739 INFO:teuthology.orchestra.run.gibba020.stdout:[ 2021-02-10T11:37:41.739 INFO:teuthology.orchestra.run.gibba020.stdout: { 2021-02-10T11:37:41.740 INFO:teuthology.orchestra.run.gibba020.stdout: "id": 7390, 2021-02-10T11:37:41.740 INFO:teuthology.orchestra.run.gibba020.stdout: "entity": { 2021-02-10T11:37:41.740 INFO:teuthology.orchestra.run.gibba020.stdout: "name": { 2021-02-10T11:37:41.740 INFO:teuthology.orchestra.run.gibba020.stdout: "type": "client", 2021-02-10T11:37:41.740 INFO:teuthology.orchestra.run.gibba020.stdout: "num": 7390 2021-02-10T11:37:41.740 INFO:teuthology.orchestra.run.gibba020.stdout: }, 2021-02-10T11:37:41.740 INFO:teuthology.orchestra.run.gibba020.stdout: "addr": { 2021-02-10T11:37:41.740 INFO:teuthology.orchestra.run.gibba020.stdout: "type": "any", 2021-02-10T11:37:41.740 INFO:teuthology.orchestra.run.gibba020.stdout: "addr": "172.21.2.120:0", 2021-02-10T11:37:41.740 INFO:teuthology.orchestra.run.gibba020.stdout: "nonce": 1069935987 2021-02-10T11:37:41.740 INFO:teuthology.orchestra.run.gibba020.stdout: } 2021-02-10T11:37:41.740 INFO:teuthology.orchestra.run.gibba020.stdout: }, 2021-02-10T11:37:41.741 INFO:teuthology.orchestra.run.gibba020.stdout: "state": "open", 2021-02-10T11:37:41.741 INFO:teuthology.orchestra.run.gibba020.stdout: "num_leases": 1, 2021-02-10T11:37:41.741 INFO:teuthology.orchestra.run.gibba020.stdout: "num_caps": 7, 2021-02-10T11:37:41.741 INFO:teuthology.orchestra.run.gibba020.stdout: "request_load_avg": 11, 2021-02-10T11:37:41.741 INFO:teuthology.orchestra.run.gibba020.stdout: "uptime": 20.089946896000001, 2021-02-10T11:37:41.741 INFO:teuthology.orchestra.run.gibba020.stdout: "requests_in_flight": 0, 2021-02-10T11:37:41.741 INFO:teuthology.orchestra.run.gibba020.stdout: "completed_requests": 1, 2021-02-10T11:37:41.741 INFO:teuthology.orchestra.run.gibba020.stdout: "reconnecting": false, 2021-02-10T11:37:41.741 INFO:teuthology.orchestra.run.gibba020.stdout: "recall_caps": { 2021-02-10T11:37:41.741 INFO:teuthology.orchestra.run.gibba020.stdout: "value": 0, 2021-02-10T11:37:41.741 INFO:teuthology.orchestra.run.gibba020.stdout: "halflife": 60 2021-02-10T11:37:41.741 INFO:teuthology.orchestra.run.gibba020.stdout: }, 2021-02-10T11:37:41.742 INFO:teuthology.orchestra.run.gibba020.stdout: "release_caps": { 2021-02-10T11:37:41.742 INFO:teuthology.orchestra.run.gibba020.stdout: "value": 0, 2021-02-10T11:37:41.742 INFO:teuthology.orchestra.run.gibba020.stdout: "halflife": 60 2021-02-10T11:37:41.742 INFO:teuthology.orchestra.run.gibba020.stdout: }, 2021-02-10T11:37:41.742 INFO:teuthology.orchestra.run.gibba020.stdout: "recall_caps_throttle": { 2021-02-10T11:37:41.742 INFO:teuthology.orchestra.run.gibba020.stdout: "value": 0, 2021-02-10T11:37:41.742 INFO:teuthology.orchestra.run.gibba020.stdout: "halflife": 2.5 2021-02-10T11:37:41.742 INFO:teuthology.orchestra.run.gibba020.stdout: }, 2021-02-10T11:37:41.742 INFO:teuthology.orchestra.run.gibba020.stdout: "recall_caps_throttle2o": { 2021-02-10T11:37:41.742 INFO:teuthology.orchestra.run.gibba020.stdout: "value": 0, 2021-02-10T11:37:41.743 INFO:teuthology.orchestra.run.gibba020.stdout: "halflife": 0.5 2021-02-10T11:37:41.743 INFO:teuthology.orchestra.run.gibba020.stdout: }, 2021-02-10T11:37:41.743 INFO:teuthology.orchestra.run.gibba020.stdout: "session_cache_liveness": { 2021-02-10T11:37:41.743 INFO:teuthology.orchestra.run.gibba020.stdout: "value": 20.061379606786346, 2021-02-10T11:37:41.743 INFO:teuthology.orchestra.run.gibba020.stdout: "halflife": 300 2021-02-10T11:37:41.743 INFO:teuthology.orchestra.run.gibba020.stdout: }, 2021-02-10T11:37:41.743 INFO:teuthology.orchestra.run.gibba020.stdout: "cap_acquisition": { 2021-02-10T11:37:41.743 INFO:teuthology.orchestra.run.gibba020.stdout: "value": 0, 2021-02-10T11:37:41.743 INFO:teuthology.orchestra.run.gibba020.stdout: "halflife": 10 2021-02-10T11:37:41.743 INFO:teuthology.orchestra.run.gibba020.stdout: }, 2021-02-10T11:37:41.743 INFO:teuthology.orchestra.run.gibba020.stdout: "inst": "client.7390 172.21.2.120:0/1069935987", 2021-02-10T11:37:41.743 INFO:teuthology.orchestra.run.gibba020.stdout: "completed_requests": [ 2021-02-10T11:37:41.743 INFO:teuthology.orchestra.run.gibba020.stdout: { 2021-02-10T11:37:41.744 INFO:teuthology.orchestra.run.gibba020.stdout: "tid": 14, 2021-02-10T11:37:41.744 INFO:teuthology.orchestra.run.gibba020.stdout: "created_ino": "0x10000000005" 2021-02-10T11:37:41.744 INFO:teuthology.orchestra.run.gibba020.stdout: } 2021-02-10T11:37:41.744 INFO:teuthology.orchestra.run.gibba020.stdout: ], 2021-02-10T11:37:41.744 INFO:teuthology.orchestra.run.gibba020.stdout: "prealloc_inos": [ 2021-02-10T11:37:41.744 INFO:teuthology.orchestra.run.gibba020.stdout: { 2021-02-10T11:37:41.744 INFO:teuthology.orchestra.run.gibba020.stdout: "start": 1099511627782, 2021-02-10T11:37:41.744 INFO:teuthology.orchestra.run.gibba020.stdout: "length": 995 2021-02-10T11:37:41.744 INFO:teuthology.orchestra.run.gibba020.stdout: } 2021-02-10T11:37:41.744 INFO:teuthology.orchestra.run.gibba020.stdout: ], 2021-02-10T11:37:41.744 INFO:teuthology.orchestra.run.gibba020.stdout: "used_inos": [], 2021-02-10T11:37:41.745 INFO:teuthology.orchestra.run.gibba020.stdout: "client_metadata": { 2021-02-10T11:37:41.745 INFO:teuthology.orchestra.run.gibba020.stdout: "features": "0x0000000000001fff", 2021-02-10T11:37:41.745 INFO:teuthology.orchestra.run.gibba020.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:37:41.745 INFO:teuthology.orchestra.run.gibba020.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:37:41.745 INFO:teuthology.orchestra.run.gibba020.stdout: "entity_id": "x", 2021-02-10T11:37:41.745 INFO:teuthology.orchestra.run.gibba020.stdout: "hostname": "gibba020", 2021-02-10T11:37:41.745 INFO:teuthology.orchestra.run.gibba020.stdout: "pid": "34990", 2021-02-10T11:37:41.745 INFO:teuthology.orchestra.run.gibba020.stdout: "root": "/" 2021-02-10T11:37:41.745 INFO:teuthology.orchestra.run.gibba020.stdout: } 2021-02-10T11:37:41.745 INFO:teuthology.orchestra.run.gibba020.stdout: } 2021-02-10T11:37:41.745 INFO:teuthology.orchestra.run.gibba020.stdout:] 2021-02-10T11:37:41.746 INFO:tasks.cephfs.filesystem:_json_asok output: [ { "id": 7390, "entity": { "name": { "type": "client", "num": 7390 }, "addr": { "type": "any", "addr": "172.21.2.120:0", "nonce": 1069935987 } }, "state": "open", "num_leases": 1, "num_caps": 7, "request_load_avg": 11, "uptime": 20.089946896000001, "requests_in_flight": 0, "completed_requests": 1, "reconnecting": false, "recall_caps": { "value": 0, "halflife": 60 }, "release_caps": { "value": 0, "halflife": 60 }, "recall_caps_throttle": { "value": 0, "halflife": 2.5 }, "recall_caps_throttle2o": { "value": 0, "halflife": 0.5 }, "session_cache_liveness": { "value": 20.061379606786346, "halflife": 300 }, "cap_acquisition": { "value": 0, "halflife": 10 }, "inst": "client.7390 172.21.2.120:0/1069935987", "completed_requests": [ { "tid": 14, "created_ino": "0x10000000005" } ], "prealloc_inos": [ { "start": 1099511627782, "length": 995 } ], "used_inos": [], "client_metadata": { "features": "0x0000000000001fff", "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", "entity_id": "x", "hostname": "gibba020", "pid": "34990", "root": "/" } } ] 2021-02-10T11:37:41.746 DEBUG:tasks.ceph_test_case:wait_until_true: waiting... 2021-02-10T11:37:46.747 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:37:47.175 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:47.169 7f25b7233700 1 Processor -- start 2021-02-10T11:37:47.175 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:47.169 7f25b7233700 1 -- start start 2021-02-10T11:37:47.175 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:47.170 7f25b7233700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f25b00d1870 0x7f25b01b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:47.175 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:47.170 7f25b7233700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f25b00f0650 0x7f25b00cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:47.175 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:47.170 7f25b7233700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25b00f0b10 con 0x7f25b01b0a10 2021-02-10T11:37:47.175 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:47.170 7f25b7233700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25b00e5ed0 con 0x7f25b00d1870 2021-02-10T11:37:47.175 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:47.170 7f25b7233700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25b00d1c70 con 0x7f25b00f0650 2021-02-10T11:37:47.175 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:47.170 7f25b504f700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f25b00d1870 0x7f25b01b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:47.176 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:47.170 7f25b484e700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f25b01b0a10 0x7f25b00cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:52332/0 (socket says 172.21.2.112:52332) 2021-02-10T11:37:47.176 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:47.170 7f25b484e700 1 -- 172.21.2.112:0/886036289 learned_addr learned my addr 172.21.2.112:0/886036289 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:37:47.176 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:47.170 7f25b504f700 1 -- 172.21.2.112:0/886036289 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f25b00f0650 msgr2=0x7f25b00cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:37:47.176 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:47.170 7f25b504f700 1 --2- 172.21.2.112:0/886036289 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f25b00f0650 0x7f25b00cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:47.176 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:47.170 7f25b504f700 1 -- 172.21.2.112:0/886036289 >> v1:172.21.2.112:6789/0 conn(0x7f25b01b0a10 legacy=0x7f25b00cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:47.176 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:47.170 7f25b504f700 1 -- 172.21.2.112:0/886036289 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25b00d1e90 con 0x7f25b00d1870 2021-02-10T11:37:47.176 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:47.171 7f25b504f700 1 --2- 172.21.2.112:0/886036289 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f25b00d1870 0x7f25b01b03f0 secure :-1 s=READY pgs=1765 cs=0 l=1 rev1=1 rx=0x7f25ac006650 tx=0x7f25ac009a00).ready entity=mon.0 client_cookie=73da1981981a6303 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:47.176 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:47.171 7f25a7fff700 1 -- 172.21.2.112:0/886036289 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f25ac0009b0 con 0x7f25b00d1870 2021-02-10T11:37:47.176 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:47.171 7f25a7fff700 1 -- 172.21.2.112:0/886036289 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25ac00c0c0 con 0x7f25b00d1870 2021-02-10T11:37:47.176 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:47.171 7f25a7fff700 1 -- 172.21.2.112:0/886036289 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f25ac00c5a0 con 0x7f25b00d1870 2021-02-10T11:37:47.176 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:47.171 7f25b7233700 1 -- 172.21.2.112:0/886036289 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f25b00d1870 msgr2=0x7f25b01b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:47.176 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:47.171 7f25b7233700 1 --2- 172.21.2.112:0/886036289 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f25b00d1870 0x7f25b01b03f0 secure :-1 s=READY pgs=1765 cs=0 l=1 rev1=1 rx=0x7f25ac006650 tx=0x7f25ac009a00).stop 2021-02-10T11:37:47.176 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:47.171 7f25b7233700 1 -- 172.21.2.112:0/886036289 shutdown_connections 2021-02-10T11:37:47.176 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:47.171 7f25b7233700 1 --2- 172.21.2.112:0/886036289 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f25b00f0650 0x7f25b00cfb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:47.177 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:47.171 7f25b7233700 1 --2- 172.21.2.112:0/886036289 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f25b00d1870 0x7f25b01b03f0 unknown :-1 s=CLOSED pgs=1765 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:47.177 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:47.171 7f25b7233700 1 -- 172.21.2.112:0/886036289 >> 172.21.2.112:0/886036289 conn(0x7f25b00ce450 msgr2=0x7f25b00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:47.177 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:47.172 7f25b7233700 1 -- 172.21.2.112:0/886036289 shutdown_connections 2021-02-10T11:37:47.177 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:47.172 7f25b7233700 1 -- 172.21.2.112:0/886036289 wait complete. 2021-02-10T11:37:47.178 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:47.186 7f25b7233700 1 Processor -- start 2021-02-10T11:37:47.178 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:47.186 7f25b7233700 1 -- start start 2021-02-10T11:37:47.178 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:47.186 7f25b7233700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f25b019af00 0x7f25b019b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:47.178 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:47.186 7f25b7233700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f25b019ba80 0x7f25b019be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:47.178 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:47.186 7f25b7233700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f25b019c4a0 0x7f25b008a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:47.179 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:47.186 7f25b7233700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25b008fb50 con 0x7f25b019af00 2021-02-10T11:37:47.179 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:47.186 7f25b7233700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25b008afb0 con 0x7f25b019ba80 2021-02-10T11:37:47.179 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:47.186 7f25b7233700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25b008b1a0 con 0x7f25b019c4a0 2021-02-10T11:37:47.179 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:47.187 7f25b504f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f25b019af00 0x7f25b019b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:47.179 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:47.187 7f25b504f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f25b019af00 0x7f25b019b300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:52878/0 (socket says 172.21.2.112:52878) 2021-02-10T11:37:47.179 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:47.187 7f25b504f700 1 -- 172.21.2.112:0/4019930277 learned_addr learned my addr 172.21.2.112:0/4019930277 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:37:47.179 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:47.187 7f25b504f700 1 -- 172.21.2.112:0/4019930277 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f25b019c4a0 msgr2=0x7f25b008a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:47.179 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:47.187 7f25b504f700 1 --2- 172.21.2.112:0/4019930277 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f25b019c4a0 0x7f25b008a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:47.179 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:47.187 7f25b504f700 1 -- 172.21.2.112:0/4019930277 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f25b019ba80 msgr2=0x7f25b019be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:47.179 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:47.187 7f25b504f700 1 --2- 172.21.2.112:0/4019930277 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f25b019ba80 0x7f25b019be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:47.179 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:47.187 7f25b504f700 1 -- 172.21.2.112:0/4019930277 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25b008b3c0 con 0x7f25b019af00 2021-02-10T11:37:47.179 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:47.187 7f25b504f700 1 --2- 172.21.2.112:0/4019930277 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f25b019af00 0x7f25b019b300 secure :-1 s=READY pgs=1766 cs=0 l=1 rev1=1 rx=0x7f25ac00b640 tx=0x7f25ac00b9e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:47.179 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:47.187 7f25a5ffb700 1 -- 172.21.2.112:0/4019930277 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f25ac00bdd0 con 0x7f25b019af00 2021-02-10T11:37:47.179 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:47.187 7f25a5ffb700 1 -- 172.21.2.112:0/4019930277 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25ac002030 con 0x7f25b019af00 2021-02-10T11:37:47.180 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:47.187 7f25a5ffb700 1 -- 172.21.2.112:0/4019930277 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f25ac002550 con 0x7f25b019af00 2021-02-10T11:37:47.180 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:47.187 7f25b7233700 1 -- 172.21.2.112:0/4019930277 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25b008b3c0 con 0x7f25b019af00 2021-02-10T11:37:47.180 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:47.187 7f25b7233700 1 -- 172.21.2.112:0/4019930277 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25b008b950 con 0x7f25b019af00 2021-02-10T11:37:47.180 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:47.188 7f25a5ffb700 1 -- 172.21.2.112:0/4019930277 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f25ac006a10 con 0x7f25b019af00 2021-02-10T11:37:47.180 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:47.188 7f25a5ffb700 1 --2- 172.21.2.112:0/4019930277 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f25900186c0 0x7f259001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:47.180 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:47.188 7f25a5ffb700 1 -- 172.21.2.112:0/4019930277 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f25ac0325b0 con 0x7f25b019af00 2021-02-10T11:37:47.180 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:47.189 7f25b484e700 1 --2- 172.21.2.112:0/4019930277 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f25900186c0 0x7f259001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:47.180 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:47.189 7f25b484e700 1 --2- 172.21.2.112:0/4019930277 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f25900186c0 0x7f259001abc0 crc :-1 s=READY pgs=834 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:47.209 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:47.220 7f25b7233700 1 -- 172.21.2.112:0/4019930277 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25b00ee030 con 0x7f25b019af00 2021-02-10T11:37:47.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:47.222 7f25a5ffb700 1 -- 172.21.2.112:0/4019930277 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f25ac029ad0 con 0x7f25b019af00 2021-02-10T11:37:47.342 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:47.353 7f25b7233700 1 -- 172.21.2.112:0/4019930277 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f25b00ce450 con 0x7f25b019af00 2021-02-10T11:37:47.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:47.353 7f25a5ffb700 1 -- 172.21.2.112:0/4019930277 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) v1 ==== 94+0+2101 (secure 0 0 0) 0x7f25ac029ad0 con 0x7f25b019af00 2021-02-10T11:37:47.343 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:37:47.343 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":65,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9204,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/1512510827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":1512510827},{"type":"v1","addr":"172.21.2.112:6833","nonce":1512510827}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":60}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:36:53.563735","modified":"2021-02-10 11:36:55.573740","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9201},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9201":{"gid":9201,"name":"b","rank":0,"incarnation":61,"state":"up:active","state_seq":3,"addr":"172.21.2.120:6835/1232001750","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":1232001750},{"type":"v1","addr":"172.21.2.120:6835","nonce":1232001750}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2021-02-10T11:37:47.344 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 65 2021-02-10T11:37:47.344 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:47.355 7f25b7233700 1 -- 172.21.2.112:0/4019930277 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f25900186c0 msgr2=0x7f259001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:47.345 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:47.355 7f25b7233700 1 --2- 172.21.2.112:0/4019930277 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f25900186c0 0x7f259001abc0 crc :-1 s=READY pgs=834 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:47.345 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:47.355 7f25b7233700 1 -- 172.21.2.112:0/4019930277 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f25b019af00 msgr2=0x7f25b019b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:47.345 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:47.355 7f25b7233700 1 --2- 172.21.2.112:0/4019930277 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f25b019af00 0x7f25b019b300 secure :-1 s=READY pgs=1766 cs=0 l=1 rev1=1 rx=0x7f25ac00b640 tx=0x7f25ac00b9e0).stop 2021-02-10T11:37:47.345 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:47.356 7f25b7233700 1 -- 172.21.2.112:0/4019930277 shutdown_connections 2021-02-10T11:37:47.345 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:47.356 7f25b7233700 1 --2- 172.21.2.112:0/4019930277 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f25900186c0 0x7f259001abc0 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:47.345 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:47.356 7f25b7233700 1 --2- 172.21.2.112:0/4019930277 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f25b019ba80 0x7f25b019be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:47.346 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:47.356 7f25b7233700 1 --2- 172.21.2.112:0/4019930277 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f25b019c4a0 0x7f25b008a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:47.346 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:47.356 7f25b7233700 1 --2- 172.21.2.112:0/4019930277 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f25b019af00 0x7f25b019b300 unknown :-1 s=CLOSED pgs=1766 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:47.346 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:47.356 7f25b7233700 1 -- 172.21.2.112:0/4019930277 >> 172.21.2.112:0/4019930277 conn(0x7f25b019f8c0 msgr2=0x7f25b01a5fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:47.346 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:47.356 7f25b7233700 1 -- 172.21.2.112:0/4019930277 shutdown_connections 2021-02-10T11:37:47.346 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:47.356 7f25b7233700 1 -- 172.21.2.112:0/4019930277 wait complete. 2021-02-10T11:37:47.354 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok session ls 2021-02-10T11:37:47.750 INFO:teuthology.orchestra.run.gibba020.stdout:[ 2021-02-10T11:37:47.750 INFO:teuthology.orchestra.run.gibba020.stdout: { 2021-02-10T11:37:47.750 INFO:teuthology.orchestra.run.gibba020.stdout: "id": 7390, 2021-02-10T11:37:47.750 INFO:teuthology.orchestra.run.gibba020.stdout: "entity": { 2021-02-10T11:37:47.750 INFO:teuthology.orchestra.run.gibba020.stdout: "name": { 2021-02-10T11:37:47.750 INFO:teuthology.orchestra.run.gibba020.stdout: "type": "client", 2021-02-10T11:37:47.750 INFO:teuthology.orchestra.run.gibba020.stdout: "num": 7390 2021-02-10T11:37:47.750 INFO:teuthology.orchestra.run.gibba020.stdout: }, 2021-02-10T11:37:47.751 INFO:teuthology.orchestra.run.gibba020.stdout: "addr": { 2021-02-10T11:37:47.751 INFO:teuthology.orchestra.run.gibba020.stdout: "type": "any", 2021-02-10T11:37:47.751 INFO:teuthology.orchestra.run.gibba020.stdout: "addr": "172.21.2.120:0", 2021-02-10T11:37:47.751 INFO:teuthology.orchestra.run.gibba020.stdout: "nonce": 1069935987 2021-02-10T11:37:47.751 INFO:teuthology.orchestra.run.gibba020.stdout: } 2021-02-10T11:37:47.751 INFO:teuthology.orchestra.run.gibba020.stdout: }, 2021-02-10T11:37:47.751 INFO:teuthology.orchestra.run.gibba020.stdout: "state": "open", 2021-02-10T11:37:47.751 INFO:teuthology.orchestra.run.gibba020.stdout: "num_leases": 1, 2021-02-10T11:37:47.751 INFO:teuthology.orchestra.run.gibba020.stdout: "num_caps": 7, 2021-02-10T11:37:47.751 INFO:teuthology.orchestra.run.gibba020.stdout: "request_load_avg": 10, 2021-02-10T11:37:47.751 INFO:teuthology.orchestra.run.gibba020.stdout: "uptime": 26.100643313999999, 2021-02-10T11:37:47.751 INFO:teuthology.orchestra.run.gibba020.stdout: "requests_in_flight": 0, 2021-02-10T11:37:47.751 INFO:teuthology.orchestra.run.gibba020.stdout: "completed_requests": 1, 2021-02-10T11:37:47.752 INFO:teuthology.orchestra.run.gibba020.stdout: "reconnecting": false, 2021-02-10T11:37:47.752 INFO:teuthology.orchestra.run.gibba020.stdout: "recall_caps": { 2021-02-10T11:37:47.752 INFO:teuthology.orchestra.run.gibba020.stdout: "value": 0, 2021-02-10T11:37:47.752 INFO:teuthology.orchestra.run.gibba020.stdout: "halflife": 60 2021-02-10T11:37:47.752 INFO:teuthology.orchestra.run.gibba020.stdout: }, 2021-02-10T11:37:47.752 INFO:teuthology.orchestra.run.gibba020.stdout: "release_caps": { 2021-02-10T11:37:47.752 INFO:teuthology.orchestra.run.gibba020.stdout: "value": 0, 2021-02-10T11:37:47.752 INFO:teuthology.orchestra.run.gibba020.stdout: "halflife": 60 2021-02-10T11:37:47.752 INFO:teuthology.orchestra.run.gibba020.stdout: }, 2021-02-10T11:37:47.752 INFO:teuthology.orchestra.run.gibba020.stdout: "recall_caps_throttle": { 2021-02-10T11:37:47.752 INFO:teuthology.orchestra.run.gibba020.stdout: "value": 0, 2021-02-10T11:37:47.752 INFO:teuthology.orchestra.run.gibba020.stdout: "halflife": 2.5 2021-02-10T11:37:47.752 INFO:teuthology.orchestra.run.gibba020.stdout: }, 2021-02-10T11:37:47.753 INFO:teuthology.orchestra.run.gibba020.stdout: "recall_caps_throttle2o": { 2021-02-10T11:37:47.753 INFO:teuthology.orchestra.run.gibba020.stdout: "value": 0, 2021-02-10T11:37:47.753 INFO:teuthology.orchestra.run.gibba020.stdout: "halflife": 0.5 2021-02-10T11:37:47.753 INFO:teuthology.orchestra.run.gibba020.stdout: }, 2021-02-10T11:37:47.753 INFO:teuthology.orchestra.run.gibba020.stdout: "session_cache_liveness": { 2021-02-10T11:37:47.753 INFO:teuthology.orchestra.run.gibba020.stdout: "value": 19.784699700887305, 2021-02-10T11:37:47.753 INFO:teuthology.orchestra.run.gibba020.stdout: "halflife": 300 2021-02-10T11:37:47.753 INFO:teuthology.orchestra.run.gibba020.stdout: }, 2021-02-10T11:37:47.753 INFO:teuthology.orchestra.run.gibba020.stdout: "cap_acquisition": { 2021-02-10T11:37:47.753 INFO:teuthology.orchestra.run.gibba020.stdout: "value": 0, 2021-02-10T11:37:47.753 INFO:teuthology.orchestra.run.gibba020.stdout: "halflife": 10 2021-02-10T11:37:47.753 INFO:teuthology.orchestra.run.gibba020.stdout: }, 2021-02-10T11:37:47.753 INFO:teuthology.orchestra.run.gibba020.stdout: "inst": "client.7390 172.21.2.120:0/1069935987", 2021-02-10T11:37:47.754 INFO:teuthology.orchestra.run.gibba020.stdout: "completed_requests": [ 2021-02-10T11:37:47.754 INFO:teuthology.orchestra.run.gibba020.stdout: { 2021-02-10T11:37:47.754 INFO:teuthology.orchestra.run.gibba020.stdout: "tid": 14, 2021-02-10T11:37:47.754 INFO:teuthology.orchestra.run.gibba020.stdout: "created_ino": "0x10000000005" 2021-02-10T11:37:47.754 INFO:teuthology.orchestra.run.gibba020.stdout: } 2021-02-10T11:37:47.754 INFO:teuthology.orchestra.run.gibba020.stdout: ], 2021-02-10T11:37:47.754 INFO:teuthology.orchestra.run.gibba020.stdout: "prealloc_inos": [ 2021-02-10T11:37:47.754 INFO:teuthology.orchestra.run.gibba020.stdout: { 2021-02-10T11:37:47.754 INFO:teuthology.orchestra.run.gibba020.stdout: "start": 1099511627782, 2021-02-10T11:37:47.754 INFO:teuthology.orchestra.run.gibba020.stdout: "length": 995 2021-02-10T11:37:47.754 INFO:teuthology.orchestra.run.gibba020.stdout: } 2021-02-10T11:37:47.754 INFO:teuthology.orchestra.run.gibba020.stdout: ], 2021-02-10T11:37:47.754 INFO:teuthology.orchestra.run.gibba020.stdout: "used_inos": [], 2021-02-10T11:37:47.755 INFO:teuthology.orchestra.run.gibba020.stdout: "client_metadata": { 2021-02-10T11:37:47.755 INFO:teuthology.orchestra.run.gibba020.stdout: "features": "0x0000000000001fff", 2021-02-10T11:37:47.755 INFO:teuthology.orchestra.run.gibba020.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:37:47.755 INFO:teuthology.orchestra.run.gibba020.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:37:47.755 INFO:teuthology.orchestra.run.gibba020.stdout: "entity_id": "x", 2021-02-10T11:37:47.755 INFO:teuthology.orchestra.run.gibba020.stdout: "hostname": "gibba020", 2021-02-10T11:37:47.755 INFO:teuthology.orchestra.run.gibba020.stdout: "pid": "34990", 2021-02-10T11:37:47.755 INFO:teuthology.orchestra.run.gibba020.stdout: "root": "/" 2021-02-10T11:37:47.755 INFO:teuthology.orchestra.run.gibba020.stdout: } 2021-02-10T11:37:47.755 INFO:teuthology.orchestra.run.gibba020.stdout: } 2021-02-10T11:37:47.755 INFO:teuthology.orchestra.run.gibba020.stdout:] 2021-02-10T11:37:47.756 INFO:tasks.cephfs.filesystem:_json_asok output: [ { "id": 7390, "entity": { "name": { "type": "client", "num": 7390 }, "addr": { "type": "any", "addr": "172.21.2.120:0", "nonce": 1069935987 } }, "state": "open", "num_leases": 1, "num_caps": 7, "request_load_avg": 10, "uptime": 26.100643313999999, "requests_in_flight": 0, "completed_requests": 1, "reconnecting": false, "recall_caps": { "value": 0, "halflife": 60 }, "release_caps": { "value": 0, "halflife": 60 }, "recall_caps_throttle": { "value": 0, "halflife": 2.5 }, "recall_caps_throttle2o": { "value": 0, "halflife": 0.5 }, "session_cache_liveness": { "value": 19.784699700887305, "halflife": 300 }, "cap_acquisition": { "value": 0, "halflife": 10 }, "inst": "client.7390 172.21.2.120:0/1069935987", "completed_requests": [ { "tid": 14, "created_ino": "0x10000000005" } ], "prealloc_inos": [ { "start": 1099511627782, "length": 995 } ], "used_inos": [], "client_metadata": { "features": "0x0000000000001fff", "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", "entity_id": "x", "hostname": "gibba020", "pid": "34990", "root": "/" } } ] 2021-02-10T11:37:47.756 DEBUG:tasks.ceph_test_case:wait_until_true: waiting... 2021-02-10T11:37:49.783 DEBUG:teuthology.orchestra.run.gibba012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:37:49.787 DEBUG:teuthology.orchestra.run.gibba020:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:37:52.758 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:37:53.186 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:53.179 7f6b92ac2700 1 Processor -- start 2021-02-10T11:37:53.186 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:53.179 7f6b92ac2700 1 -- start start 2021-02-10T11:37:53.186 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:53.180 7f6b92ac2700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6b8c1b0470 0x7f6b8c1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:53.186 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:53.180 7f6b92ac2700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b8c0f0b50 con 0x7f6b8c0ef610 2021-02-10T11:37:53.187 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:53.180 7f6b92ac2700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b8c0f24e0 con 0x7f6b8c0f20e0 2021-02-10T11:37:53.187 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:53.180 7f6b92ac2700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b8c0e5ea0 con 0x7f6b8c1b0470 2021-02-10T11:37:53.187 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:53.180 7f6b8bfff700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f6b8c0ef610 0x7f6b8c0efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:52346/0 (socket says 172.21.2.112:52346) 2021-02-10T11:37:53.187 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:53.180 7f6b8bfff700 1 -- 172.21.2.112:0/996319951 learned_addr learned my addr 172.21.2.112:0/996319951 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:37:53.187 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:53.180 7f6b908de700 1 --2- 172.21.2.112:0/996319951 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6b8c1b0470 0x7f6b8c1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:53.187 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:53.180 7f6b8b7fe700 1 -- 172.21.2.112:0/996319951 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1970748068 0 0) 0x7f6b7c000ce0 con 0x7f6b8c0ef610 2021-02-10T11:37:53.187 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:53.180 7f6b8b7fe700 1 -- 172.21.2.112:0/996319951 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b78002fe0 con 0x7f6b8c0ef610 2021-02-10T11:37:53.187 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:53.181 7f6b8b7fe700 1 -- 172.21.2.112:0/996319951 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2326634171 0 0) 0x7f6b7c0010e0 con 0x7f6b8c0ef610 2021-02-10T11:37:53.187 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:53.181 7f6b8b7fe700 1 -- 172.21.2.112:0/996319951 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6b8c1b0470 msgr2=0x7f6b8c1b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:37:53.187 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:53.181 7f6b8b7fe700 1 --2- 172.21.2.112:0/996319951 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6b8c1b0470 0x7f6b8c1b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:53.187 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:53.181 7f6b8b7fe700 1 -- 172.21.2.112:0/996319951 >> v1:172.21.2.120:6789/0 conn(0x7f6b8c0f20e0 legacy=0x7f6b8c0f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:53.187 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:53.181 7f6b8b7fe700 1 -- 172.21.2.112:0/996319951 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b8c0e52e0 con 0x7f6b8c0ef610 2021-02-10T11:37:53.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:53.181 7f6b8b7fe700 1 -- 172.21.2.112:0/996319951 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f6b7c001660 con 0x7f6b8c0ef610 2021-02-10T11:37:53.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:53.181 7f6b8b7fe700 1 -- 172.21.2.112:0/996319951 >> v1:172.21.2.112:6789/0 conn(0x7f6b8c0ef610 legacy=0x7f6b8c0efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:53.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:53.181 7f6b8b7fe700 1 --2- 172.21.2.112:0/996319951 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6b78003ac0 0x7f6b78005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:53.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:53.181 7f6b8b7fe700 1 --2- 172.21.2.112:0/996319951 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6b78006790 0x7f6b78008c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:53.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:53.181 7f6b8b7fe700 1 --2- 172.21.2.112:0/996319951 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6b78009500 0x7f6b7800ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:53.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:53.181 7f6b8b7fe700 1 -- 172.21.2.112:0/996319951 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b7800c2d0 con 0x7f6b78003ac0 2021-02-10T11:37:53.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:53.181 7f6b8b7fe700 1 -- 172.21.2.112:0/996319951 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b7800c4f0 con 0x7f6b78009500 2021-02-10T11:37:53.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:53.181 7f6b8b7fe700 1 -- 172.21.2.112:0/996319951 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b7800c710 con 0x7f6b78006790 2021-02-10T11:37:53.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:53.181 7f6b908de700 1 --2- 172.21.2.112:0/996319951 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6b78003ac0 0x7f6b78005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:53.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:53.181 7f6b8bfff700 1 --2- 172.21.2.112:0/996319951 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6b78006790 0x7f6b78008c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:53.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:53.181 7f6b908de700 1 -- 172.21.2.112:0/996319951 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6b78006790 msgr2=0x7f6b78008c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:37:53.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:53.181 7f6b908de700 1 --2- 172.21.2.112:0/996319951 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6b78006790 0x7f6b78008c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:53.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:53.181 7f6b908de700 1 -- 172.21.2.112:0/996319951 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6b78009500 msgr2=0x7f6b7800ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:37:53.189 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:53.181 7f6b908de700 1 --2- 172.21.2.112:0/996319951 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6b78009500 0x7f6b7800ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:53.189 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:53.181 7f6b908de700 1 -- 172.21.2.112:0/996319951 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b78050980 con 0x7f6b78003ac0 2021-02-10T11:37:53.189 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:53.181 7f6b908de700 1 --2- 172.21.2.112:0/996319951 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6b78003ac0 0x7f6b78005ef0 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 rx=0x7f6b74006ac0 tx=0x7f6b74009a30).ready entity=mon.0 client_cookie=6de054b78e341c7e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:53.189 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:53.181 7f6b8b7fe700 1 -- 172.21.2.112:0/996319951 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6b7400c0c0 con 0x7f6b78003ac0 2021-02-10T11:37:53.189 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:53.181 7f6b8b7fe700 1 -- 172.21.2.112:0/996319951 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b74009e20 con 0x7f6b78003ac0 2021-02-10T11:37:53.189 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:53.181 7f6b8b7fe700 1 -- 172.21.2.112:0/996319951 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6b7400c5f0 con 0x7f6b78003ac0 2021-02-10T11:37:53.189 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:53.181 7f6b92ac2700 1 -- 172.21.2.112:0/996319951 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6b78003ac0 msgr2=0x7f6b78005ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:53.189 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:53.181 7f6b92ac2700 1 --2- 172.21.2.112:0/996319951 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6b78003ac0 0x7f6b78005ef0 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 rx=0x7f6b74006ac0 tx=0x7f6b74009a30).stop 2021-02-10T11:37:53.189 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:53.182 7f6b910df700 1 -- 172.21.2.112:0/996319951 reap_dead start 2021-02-10T11:37:53.189 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:53.182 7f6b92ac2700 1 -- 172.21.2.112:0/996319951 shutdown_connections 2021-02-10T11:37:53.189 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:53.182 7f6b92ac2700 1 -- 172.21.2.112:0/996319951 >> 172.21.2.112:0/996319951 conn(0x7f6b8c0ce3e0 msgr2=0x7f6b8c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:53.189 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:53.182 7f6b92ac2700 1 -- 172.21.2.112:0/996319951 shutdown_connections 2021-02-10T11:37:53.189 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:53.182 7f6b92ac2700 1 -- 172.21.2.112:0/996319951 wait complete. 2021-02-10T11:37:53.190 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:53.197 7f6b92ac2700 1 Processor -- start 2021-02-10T11:37:53.190 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:53.197 7f6b92ac2700 1 -- start start 2021-02-10T11:37:53.191 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:53.197 7f6b92ac2700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6b8c108ce0 0x7f6b8c09da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:53.191 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:53.197 7f6b92ac2700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6b8c1090e0 0x7f6b8c09e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:53.191 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:53.197 7f6b92ac2700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6b8c09eee0 0x7f6b8c09f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:53.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:53.197 7f6b92ac2700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b8c1094e0 con 0x7f6b8c108ce0 2021-02-10T11:37:53.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:53.197 7f6b92ac2700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b8c09e960 con 0x7f6b8c09eee0 2021-02-10T11:37:53.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:53.197 7f6b92ac2700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b8c09e690 con 0x7f6b8c1090e0 2021-02-10T11:37:53.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:53.197 7f6b908de700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6b8c108ce0 0x7f6b8c09da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:53.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:53.197 7f6b908de700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6b8c108ce0 0x7f6b8c09da50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:52896/0 (socket says 172.21.2.112:52896) 2021-02-10T11:37:53.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:53.197 7f6b908de700 1 -- 172.21.2.112:0/3927939576 learned_addr learned my addr 172.21.2.112:0/3927939576 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:37:53.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:53.198 7f6b910df700 1 --2- 172.21.2.112:0/3927939576 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6b8c09eee0 0x7f6b8c09f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:53.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:53.198 7f6b8bfff700 1 --2- 172.21.2.112:0/3927939576 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6b8c1090e0 0x7f6b8c09e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:53.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:53.198 7f6b908de700 1 -- 172.21.2.112:0/3927939576 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6b8c1090e0 msgr2=0x7f6b8c09e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:53.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:53.198 7f6b908de700 1 --2- 172.21.2.112:0/3927939576 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6b8c1090e0 0x7f6b8c09e070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:53.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:53.198 7f6b908de700 1 -- 172.21.2.112:0/3927939576 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6b8c09eee0 msgr2=0x7f6b8c09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:53.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:53.198 7f6b908de700 1 --2- 172.21.2.112:0/3927939576 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6b8c09eee0 0x7f6b8c09f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:53.193 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:53.198 7f6b908de700 1 -- 172.21.2.112:0/3927939576 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b8c09f900 con 0x7f6b8c108ce0 2021-02-10T11:37:53.193 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:53.198 7f6b910df700 1 --2- 172.21.2.112:0/3927939576 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6b8c09eee0 0x7f6b8c09f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:37:53.193 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:53.198 7f6b908de700 1 --2- 172.21.2.112:0/3927939576 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6b8c108ce0 0x7f6b8c09da50 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 rx=0x7f6b74006f30 tx=0x7f6b7400b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:53.193 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:53.198 7f6b897fa700 1 -- 172.21.2.112:0/3927939576 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6b740009b0 con 0x7f6b8c108ce0 2021-02-10T11:37:53.193 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:53.198 7f6b897fa700 1 -- 172.21.2.112:0/3927939576 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b7400bd80 con 0x7f6b8c108ce0 2021-02-10T11:37:53.193 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:53.198 7f6b92ac2700 1 -- 172.21.2.112:0/3927939576 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b8c09f900 con 0x7f6b8c108ce0 2021-02-10T11:37:53.193 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:53.198 7f6b92ac2700 1 -- 172.21.2.112:0/3927939576 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b8c09f900 con 0x7f6b8c108ce0 2021-02-10T11:37:53.193 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:53.198 7f6b897fa700 1 -- 172.21.2.112:0/3927939576 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6b740022b0 con 0x7f6b8c108ce0 2021-02-10T11:37:53.193 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:53.199 7f6b897fa700 1 -- 172.21.2.112:0/3927939576 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6b740052e0 con 0x7f6b8c108ce0 2021-02-10T11:37:53.193 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:53.199 7f6b897fa700 1 --2- 172.21.2.112:0/3927939576 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6b70018730 0x7f6b7001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:53.193 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:53.199 7f6b897fa700 1 -- 172.21.2.112:0/3927939576 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f6b740326c0 con 0x7f6b8c108ce0 2021-02-10T11:37:53.193 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:53.199 7f6b8bfff700 1 --2- 172.21.2.112:0/3927939576 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6b70018730 0x7f6b7001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:53.193 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:53.200 7f6b8bfff700 1 --2- 172.21.2.112:0/3927939576 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6b70018730 0x7f6b7001ac10 crc :-1 s=READY pgs=835 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:53.219 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:53.229 7f6b92ac2700 1 -- 172.21.2.112:0/3927939576 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b8c0f0b50 con 0x7f6b8c108ce0 2021-02-10T11:37:53.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:53.232 7f6b897fa700 1 -- 172.21.2.112:0/3927939576 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6b74000c90 con 0x7f6b8c108ce0 2021-02-10T11:37:53.349 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:53.359 7f6b92ac2700 1 -- 172.21.2.112:0/3927939576 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6b8c0f0b50 con 0x7f6b8c108ce0 2021-02-10T11:37:53.349 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:53.360 7f6b897fa700 1 -- 172.21.2.112:0/3927939576 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) v1 ==== 94+0+2101 (secure 0 0 0) 0x7f6b7400bd80 con 0x7f6b8c108ce0 2021-02-10T11:37:53.350 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:37:53.350 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":65,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9204,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/1512510827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":1512510827},{"type":"v1","addr":"172.21.2.112:6833","nonce":1512510827}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":60}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:36:53.563735","modified":"2021-02-10 11:36:55.573740","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9201},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9201":{"gid":9201,"name":"b","rank":0,"incarnation":61,"state":"up:active","state_seq":3,"addr":"172.21.2.120:6835/1232001750","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":1232001750},{"type":"v1","addr":"172.21.2.120:6835","nonce":1232001750}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2021-02-10T11:37:53.350 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 65 2021-02-10T11:37:53.351 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:53.362 7f6b92ac2700 1 -- 172.21.2.112:0/3927939576 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6b70018730 msgr2=0x7f6b7001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:53.351 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:53.362 7f6b92ac2700 1 --2- 172.21.2.112:0/3927939576 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6b70018730 0x7f6b7001ac10 crc :-1 s=READY pgs=835 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:53.351 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:53.362 7f6b92ac2700 1 -- 172.21.2.112:0/3927939576 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6b8c108ce0 msgr2=0x7f6b8c09da50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:53.351 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:53.362 7f6b92ac2700 1 --2- 172.21.2.112:0/3927939576 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6b8c108ce0 0x7f6b8c09da50 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 rx=0x7f6b74006f30 tx=0x7f6b7400b990).stop 2021-02-10T11:37:53.352 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:53.362 7f6b92ac2700 1 -- 172.21.2.112:0/3927939576 shutdown_connections 2021-02-10T11:37:53.352 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:53.362 7f6b92ac2700 1 --2- 172.21.2.112:0/3927939576 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6b70018730 0x7f6b7001ac10 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:53.352 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:53.362 7f6b92ac2700 1 --2- 172.21.2.112:0/3927939576 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6b8c09eee0 0x7f6b8c09f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:53.352 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:53.362 7f6b92ac2700 1 --2- 172.21.2.112:0/3927939576 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6b8c1090e0 0x7f6b8c09e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:53.352 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:53.362 7f6b92ac2700 1 --2- 172.21.2.112:0/3927939576 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6b8c108ce0 0x7f6b8c09da50 unknown :-1 s=CLOSED pgs=1769 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:53.353 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:53.362 7f6b92ac2700 1 -- 172.21.2.112:0/3927939576 >> 172.21.2.112:0/3927939576 conn(0x7f6b8c0bc6c0 msgr2=0x7f6b8c0b1f70 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:53.353 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:53.362 7f6b92ac2700 1 -- 172.21.2.112:0/3927939576 shutdown_connections 2021-02-10T11:37:53.353 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:53.362 7f6b92ac2700 1 -- 172.21.2.112:0/3927939576 wait complete. 2021-02-10T11:37:53.360 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok session ls 2021-02-10T11:37:53.758 INFO:teuthology.orchestra.run.gibba020.stdout:[ 2021-02-10T11:37:53.758 INFO:teuthology.orchestra.run.gibba020.stdout: { 2021-02-10T11:37:53.758 INFO:teuthology.orchestra.run.gibba020.stdout: "id": 7390, 2021-02-10T11:37:53.758 INFO:teuthology.orchestra.run.gibba020.stdout: "entity": { 2021-02-10T11:37:53.758 INFO:teuthology.orchestra.run.gibba020.stdout: "name": { 2021-02-10T11:37:53.758 INFO:teuthology.orchestra.run.gibba020.stdout: "type": "client", 2021-02-10T11:37:53.758 INFO:teuthology.orchestra.run.gibba020.stdout: "num": 7390 2021-02-10T11:37:53.758 INFO:teuthology.orchestra.run.gibba020.stdout: }, 2021-02-10T11:37:53.758 INFO:teuthology.orchestra.run.gibba020.stdout: "addr": { 2021-02-10T11:37:53.758 INFO:teuthology.orchestra.run.gibba020.stdout: "type": "any", 2021-02-10T11:37:53.759 INFO:teuthology.orchestra.run.gibba020.stdout: "addr": "172.21.2.120:0", 2021-02-10T11:37:53.759 INFO:teuthology.orchestra.run.gibba020.stdout: "nonce": 1069935987 2021-02-10T11:37:53.759 INFO:teuthology.orchestra.run.gibba020.stdout: } 2021-02-10T11:37:53.759 INFO:teuthology.orchestra.run.gibba020.stdout: }, 2021-02-10T11:37:53.759 INFO:teuthology.orchestra.run.gibba020.stdout: "state": "open", 2021-02-10T11:37:53.759 INFO:teuthology.orchestra.run.gibba020.stdout: "num_leases": 0, 2021-02-10T11:37:53.759 INFO:teuthology.orchestra.run.gibba020.stdout: "num_caps": 7, 2021-02-10T11:37:53.759 INFO:teuthology.orchestra.run.gibba020.stdout: "request_load_avg": 9, 2021-02-10T11:37:53.759 INFO:teuthology.orchestra.run.gibba020.stdout: "uptime": 32.108344957, 2021-02-10T11:37:53.759 INFO:teuthology.orchestra.run.gibba020.stdout: "requests_in_flight": 0, 2021-02-10T11:37:53.759 INFO:teuthology.orchestra.run.gibba020.stdout: "completed_requests": 1, 2021-02-10T11:37:53.759 INFO:teuthology.orchestra.run.gibba020.stdout: "reconnecting": false, 2021-02-10T11:37:53.759 INFO:teuthology.orchestra.run.gibba020.stdout: "recall_caps": { 2021-02-10T11:37:53.759 INFO:teuthology.orchestra.run.gibba020.stdout: "value": 0, 2021-02-10T11:37:53.760 INFO:teuthology.orchestra.run.gibba020.stdout: "halflife": 60 2021-02-10T11:37:53.760 INFO:teuthology.orchestra.run.gibba020.stdout: }, 2021-02-10T11:37:53.760 INFO:teuthology.orchestra.run.gibba020.stdout: "release_caps": { 2021-02-10T11:37:53.760 INFO:teuthology.orchestra.run.gibba020.stdout: "value": 0, 2021-02-10T11:37:53.760 INFO:teuthology.orchestra.run.gibba020.stdout: "halflife": 60 2021-02-10T11:37:53.760 INFO:teuthology.orchestra.run.gibba020.stdout: }, 2021-02-10T11:37:53.760 INFO:teuthology.orchestra.run.gibba020.stdout: "recall_caps_throttle": { 2021-02-10T11:37:53.760 INFO:teuthology.orchestra.run.gibba020.stdout: "value": 0, 2021-02-10T11:37:53.760 INFO:teuthology.orchestra.run.gibba020.stdout: "halflife": 2.5 2021-02-10T11:37:53.760 INFO:teuthology.orchestra.run.gibba020.stdout: }, 2021-02-10T11:37:53.760 INFO:teuthology.orchestra.run.gibba020.stdout: "recall_caps_throttle2o": { 2021-02-10T11:37:53.760 INFO:teuthology.orchestra.run.gibba020.stdout: "value": 0, 2021-02-10T11:37:53.760 INFO:teuthology.orchestra.run.gibba020.stdout: "halflife": 0.5 2021-02-10T11:37:53.760 INFO:teuthology.orchestra.run.gibba020.stdout: }, 2021-02-10T11:37:53.761 INFO:teuthology.orchestra.run.gibba020.stdout: "session_cache_liveness": { 2021-02-10T11:37:53.761 INFO:teuthology.orchestra.run.gibba020.stdout: "value": 19.511970683304092, 2021-02-10T11:37:53.761 INFO:teuthology.orchestra.run.gibba020.stdout: "halflife": 300 2021-02-10T11:37:53.761 INFO:teuthology.orchestra.run.gibba020.stdout: }, 2021-02-10T11:37:53.761 INFO:teuthology.orchestra.run.gibba020.stdout: "cap_acquisition": { 2021-02-10T11:37:53.761 INFO:teuthology.orchestra.run.gibba020.stdout: "value": 0, 2021-02-10T11:37:53.761 INFO:teuthology.orchestra.run.gibba020.stdout: "halflife": 10 2021-02-10T11:37:53.761 INFO:teuthology.orchestra.run.gibba020.stdout: }, 2021-02-10T11:37:53.761 INFO:teuthology.orchestra.run.gibba020.stdout: "inst": "client.7390 172.21.2.120:0/1069935987", 2021-02-10T11:37:53.761 INFO:teuthology.orchestra.run.gibba020.stdout: "completed_requests": [ 2021-02-10T11:37:53.761 INFO:teuthology.orchestra.run.gibba020.stdout: { 2021-02-10T11:37:53.761 INFO:teuthology.orchestra.run.gibba020.stdout: "tid": 14, 2021-02-10T11:37:53.761 INFO:teuthology.orchestra.run.gibba020.stdout: "created_ino": "0x10000000005" 2021-02-10T11:37:53.762 INFO:teuthology.orchestra.run.gibba020.stdout: } 2021-02-10T11:37:53.762 INFO:teuthology.orchestra.run.gibba020.stdout: ], 2021-02-10T11:37:53.762 INFO:teuthology.orchestra.run.gibba020.stdout: "prealloc_inos": [ 2021-02-10T11:37:53.762 INFO:teuthology.orchestra.run.gibba020.stdout: { 2021-02-10T11:37:53.762 INFO:teuthology.orchestra.run.gibba020.stdout: "start": 1099511627782, 2021-02-10T11:37:53.762 INFO:teuthology.orchestra.run.gibba020.stdout: "length": 995 2021-02-10T11:37:53.762 INFO:teuthology.orchestra.run.gibba020.stdout: } 2021-02-10T11:37:53.762 INFO:teuthology.orchestra.run.gibba020.stdout: ], 2021-02-10T11:37:53.762 INFO:teuthology.orchestra.run.gibba020.stdout: "used_inos": [], 2021-02-10T11:37:53.762 INFO:teuthology.orchestra.run.gibba020.stdout: "client_metadata": { 2021-02-10T11:37:53.762 INFO:teuthology.orchestra.run.gibba020.stdout: "features": "0x0000000000001fff", 2021-02-10T11:37:53.762 INFO:teuthology.orchestra.run.gibba020.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:37:53.762 INFO:teuthology.orchestra.run.gibba020.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:37:53.762 INFO:teuthology.orchestra.run.gibba020.stdout: "entity_id": "x", 2021-02-10T11:37:53.763 INFO:teuthology.orchestra.run.gibba020.stdout: "hostname": "gibba020", 2021-02-10T11:37:53.763 INFO:teuthology.orchestra.run.gibba020.stdout: "pid": "34990", 2021-02-10T11:37:53.763 INFO:teuthology.orchestra.run.gibba020.stdout: "root": "/" 2021-02-10T11:37:53.763 INFO:teuthology.orchestra.run.gibba020.stdout: } 2021-02-10T11:37:53.763 INFO:teuthology.orchestra.run.gibba020.stdout: } 2021-02-10T11:37:53.763 INFO:teuthology.orchestra.run.gibba020.stdout:] 2021-02-10T11:37:53.764 INFO:tasks.cephfs.filesystem:_json_asok output: [ { "id": 7390, "entity": { "name": { "type": "client", "num": 7390 }, "addr": { "type": "any", "addr": "172.21.2.120:0", "nonce": 1069935987 } }, "state": "open", "num_leases": 0, "num_caps": 7, "request_load_avg": 9, "uptime": 32.108344957, "requests_in_flight": 0, "completed_requests": 1, "reconnecting": false, "recall_caps": { "value": 0, "halflife": 60 }, "release_caps": { "value": 0, "halflife": 60 }, "recall_caps_throttle": { "value": 0, "halflife": 2.5 }, "recall_caps_throttle2o": { "value": 0, "halflife": 0.5 }, "session_cache_liveness": { "value": 19.511970683304092, "halflife": 300 }, "cap_acquisition": { "value": 0, "halflife": 10 }, "inst": "client.7390 172.21.2.120:0/1069935987", "completed_requests": [ { "tid": 14, "created_ino": "0x10000000005" } ], "prealloc_inos": [ { "start": 1099511627782, "length": 995 } ], "used_inos": [], "client_metadata": { "features": "0x0000000000001fff", "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", "entity_id": "x", "hostname": "gibba020", "pid": "34990", "root": "/" } } ] 2021-02-10T11:37:53.764 DEBUG:tasks.ceph_test_case:wait_until_true: waiting... 2021-02-10T11:37:58.765 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:37:59.193 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:59.186 7f04a2ccb700 1 Processor -- start 2021-02-10T11:37:59.193 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:59.186 7f04a2ccb700 1 -- start start 2021-02-10T11:37:59.193 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:59.187 7f04a2ccb700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f049c0d1870 0x7f049c1b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:59.193 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:59.187 7f04a2ccb700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f049c1b0a10 0x7f049c0cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:59.194 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:59.187 7f04a2ccb700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f049c0d1c70 0x7f049c0cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:59.194 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:59.187 7f04a2ccb700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f049c0fe230 con 0x7f049c0d1c70 2021-02-10T11:37:59.194 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:59.187 7f04a2ccb700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f049c0e5ed0 con 0x7f049c1b0a10 2021-02-10T11:37:59.194 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:59.187 7f04a2ccb700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f049c0f0b10 con 0x7f049c0d1870 2021-02-10T11:37:59.194 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:59.187 7f04a12e8700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f049c0d1c70 0x7f049c0cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:59.194 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:59.187 7f04a12e8700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f049c0d1c70 0x7f049c0cf950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:52904/0 (socket says 172.21.2.112:52904) 2021-02-10T11:37:59.194 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:59.187 7f04a12e8700 1 -- 172.21.2.112:0/4264938552 learned_addr learned my addr 172.21.2.112:0/4264938552 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:37:59.194 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:59.187 7f04a0ae7700 1 --2- 172.21.2.112:0/4264938552 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f049c0d1870 0x7f049c1b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:59.194 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:59.187 7f049bfff700 1 --2- 172.21.2.112:0/4264938552 >> v2:172.21.2.120:3300/0 conn(0x7f049c1b0a10 0x7f049c0cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:59.194 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:59.187 7f04a12e8700 1 -- 172.21.2.112:0/4264938552 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f049c0d1870 msgr2=0x7f049c1b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:37:59.194 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:59.187 7f04a12e8700 1 --2- 172.21.2.112:0/4264938552 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f049c0d1870 0x7f049c1b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:59.194 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:59.187 7f04a12e8700 1 -- 172.21.2.112:0/4264938552 >> v2:172.21.2.120:3300/0 conn(0x7f049c1b0a10 msgr2=0x7f049c0cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:37:59.195 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:59.187 7f04a12e8700 1 --2- 172.21.2.112:0/4264938552 >> v2:172.21.2.120:3300/0 conn(0x7f049c1b0a10 0x7f049c0cf330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:59.195 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:59.187 7f04a12e8700 1 -- 172.21.2.112:0/4264938552 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f049c0cff70 con 0x7f049c0d1c70 2021-02-10T11:37:59.195 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:59.187 7f04a0ae7700 1 --2- 172.21.2.112:0/4264938552 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f049c0d1870 0x7f049c1b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:37:59.195 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:59.187 7f04a12e8700 1 --2- 172.21.2.112:0/4264938552 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f049c0d1c70 0x7f049c0cf950 secure :-1 s=READY pgs=1770 cs=0 l=1 rev1=1 rx=0x7f0494006f00 tx=0x7f0494009f10).ready entity=mon.0 client_cookie=eb63d82aa27dff5f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:59.195 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:59.187 7f049b7fe700 1 -- 172.21.2.112:0/4264938552 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f049400b3b0 con 0x7f049c0d1c70 2021-02-10T11:37:59.195 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:59.187 7f049b7fe700 1 -- 172.21.2.112:0/4264938552 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f049400b730 con 0x7f049c0d1c70 2021-02-10T11:37:59.195 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:59.188 7f04a2ccb700 1 -- 172.21.2.112:0/4264938552 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f049c0d1c70 msgr2=0x7f049c0cf950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:59.195 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:59.188 7f04a2ccb700 1 --2- 172.21.2.112:0/4264938552 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f049c0d1c70 0x7f049c0cf950 secure :-1 s=READY pgs=1770 cs=0 l=1 rev1=1 rx=0x7f0494006f00 tx=0x7f0494009f10).stop 2021-02-10T11:37:59.195 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:59.187 7f049b7fe700 1 -- 172.21.2.112:0/4264938552 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f049400bc10 con 0x7f049c0d1c70 2021-02-10T11:37:59.195 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:59.188 7f04a2ccb700 1 -- 172.21.2.112:0/4264938552 shutdown_connections 2021-02-10T11:37:59.195 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:59.188 7f04a2ccb700 1 --2- 172.21.2.112:0/4264938552 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f049c0d1c70 0x7f049c0cf950 unknown :-1 s=CLOSED pgs=1770 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:59.195 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:59.188 7f04a2ccb700 1 --2- 172.21.2.112:0/4264938552 >> v2:172.21.2.120:3300/0 conn(0x7f049c1b0a10 0x7f049c0cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:59.195 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:59.188 7f04a2ccb700 1 --2- 172.21.2.112:0/4264938552 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f049c0d1870 0x7f049c1b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:59.196 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:59.188 7f04a2ccb700 1 -- 172.21.2.112:0/4264938552 >> 172.21.2.112:0/4264938552 conn(0x7f049c0ce450 msgr2=0x7f049c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:59.196 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:59.188 7f04a2ccb700 1 -- 172.21.2.112:0/4264938552 shutdown_connections 2021-02-10T11:37:59.196 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:59.188 7f04a2ccb700 1 -- 172.21.2.112:0/4264938552 wait complete. 2021-02-10T11:37:59.197 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:59.203 7f04a2ccb700 1 Processor -- start 2021-02-10T11:37:59.197 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:59.203 7f04a2ccb700 1 -- start start 2021-02-10T11:37:59.197 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:59.203 7f04a2ccb700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f049c0aab30 0x7f049c08ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:59.197 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:59.203 7f04a2ccb700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f049c08f2a0 0x7f049c08f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:59.198 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:59.203 7f04a2ccb700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f049c08fcc0 0x7f049c19a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:59.198 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:59.203 7f04a2ccb700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f049c197820 con 0x7f049c08f2a0 2021-02-10T11:37:59.198 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:59.203 7f04a2ccb700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f049c1ac2a0 con 0x7f049c08fcc0 2021-02-10T11:37:59.198 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:59.203 7f04a2ccb700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f049c1abde0 con 0x7f049c0aab30 2021-02-10T11:37:59.198 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:59.204 7f049bfff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f049c08f2a0 0x7f049c08f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:59.198 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:59.204 7f049bfff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f049c08f2a0 0x7f049c08f6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:52910/0 (socket says 172.21.2.112:52910) 2021-02-10T11:37:59.198 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:59.204 7f049bfff700 1 -- 172.21.2.112:0/3658361203 learned_addr learned my addr 172.21.2.112:0/3658361203 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:37:59.198 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:59.204 7f04a12e8700 1 --2- 172.21.2.112:0/3658361203 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f049c08fcc0 0x7f049c19a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:59.198 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:59.204 7f04a0ae7700 1 --2- 172.21.2.112:0/3658361203 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f049c0aab30 0x7f049c08ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:59.198 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:59.204 7f049bfff700 1 -- 172.21.2.112:0/3658361203 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f049c0aab30 msgr2=0x7f049c08ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:59.198 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:59.204 7f049bfff700 1 --2- 172.21.2.112:0/3658361203 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f049c0aab30 0x7f049c08ec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:59.198 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:59.204 7f049bfff700 1 -- 172.21.2.112:0/3658361203 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f049c08fcc0 msgr2=0x7f049c19a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:59.198 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:59.204 7f049bfff700 1 --2- 172.21.2.112:0/3658361203 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f049c08fcc0 0x7f049c19a2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:59.199 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:59.204 7f049bfff700 1 -- 172.21.2.112:0/3658361203 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f049c19a8f0 con 0x7f049c08f2a0 2021-02-10T11:37:59.199 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:59.204 7f04a0ae7700 1 --2- 172.21.2.112:0/3658361203 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f049c0aab30 0x7f049c08ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:37:59.199 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:59.204 7f049bfff700 1 --2- 172.21.2.112:0/3658361203 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f049c08f2a0 0x7f049c08f6a0 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 rx=0x7f048c006f00 tx=0x7f048c00b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:59.199 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:59.204 7f04997fa700 1 -- 172.21.2.112:0/3658361203 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f048c00bd30 con 0x7f049c08f2a0 2021-02-10T11:37:59.199 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:59.204 7f04997fa700 1 -- 172.21.2.112:0/3658361203 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f048c002030 con 0x7f049c08f2a0 2021-02-10T11:37:59.199 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:59.204 7f04997fa700 1 -- 172.21.2.112:0/3658361203 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f048c002550 con 0x7f049c08f2a0 2021-02-10T11:37:59.199 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:59.204 7f04a2ccb700 1 -- 172.21.2.112:0/3658361203 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f049c19a8f0 con 0x7f049c08f2a0 2021-02-10T11:37:59.199 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:59.204 7f04a2ccb700 1 -- 172.21.2.112:0/3658361203 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f049c19a8f0 con 0x7f049c08f2a0 2021-02-10T11:37:59.199 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:59.205 7f04997fa700 1 -- 172.21.2.112:0/3658361203 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f048c000c90 con 0x7f049c08f2a0 2021-02-10T11:37:59.199 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:59.205 7f04997fa700 1 --2- 172.21.2.112:0/3658361203 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0480018730 0x7f048001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:59.199 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:59.205 7f04997fa700 1 -- 172.21.2.112:0/3658361203 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f048c032680 con 0x7f049c08f2a0 2021-02-10T11:37:59.199 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:59.205 7f04a0ae7700 1 --2- 172.21.2.112:0/3658361203 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0480018730 0x7f048001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:59.199 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:59.205 7f04a0ae7700 1 --2- 172.21.2.112:0/3658361203 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0480018730 0x7f048001ac10 crc :-1 s=READY pgs=836 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:59.227 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:59.236 7f04a2ccb700 1 -- 172.21.2.112:0/3658361203 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f049c0ebc00 con 0x7f049c08f2a0 2021-02-10T11:37:59.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:59.240 7f04997fa700 1 -- 172.21.2.112:0/3658361203 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f048c00bd30 con 0x7f049c08f2a0 2021-02-10T11:37:59.358 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:59.367 7f04a2ccb700 1 -- 172.21.2.112:0/3658361203 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f049c0ebc00 con 0x7f049c08f2a0 2021-02-10T11:37:59.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:59.369 7f04997fa700 1 -- 172.21.2.112:0/3658361203 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) v1 ==== 94+0+2101 (secure 0 0 0) 0x7f048c000c90 con 0x7f049c08f2a0 2021-02-10T11:37:59.359 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:37:59.359 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":65,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9204,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/1512510827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":1512510827},{"type":"v1","addr":"172.21.2.112:6833","nonce":1512510827}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":60}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:36:53.563735","modified":"2021-02-10 11:36:55.573740","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9201},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9201":{"gid":9201,"name":"b","rank":0,"incarnation":61,"state":"up:active","state_seq":3,"addr":"172.21.2.120:6835/1232001750","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":1232001750},{"type":"v1","addr":"172.21.2.120:6835","nonce":1232001750}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2021-02-10T11:37:59.359 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 65 2021-02-10T11:37:59.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:59.369 7f04a2ccb700 1 -- 172.21.2.112:0/3658361203 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0480018730 msgr2=0x7f048001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:59.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:59.369 7f04a2ccb700 1 --2- 172.21.2.112:0/3658361203 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0480018730 0x7f048001ac10 crc :-1 s=READY pgs=836 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:59.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:59.369 7f04a2ccb700 1 -- 172.21.2.112:0/3658361203 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f049c08f2a0 msgr2=0x7f049c08f6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:59.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:59.369 7f04a2ccb700 1 --2- 172.21.2.112:0/3658361203 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f049c08f2a0 0x7f049c08f6a0 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 rx=0x7f048c006f00 tx=0x7f048c00b940).stop 2021-02-10T11:37:59.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:59.371 7f04a2ccb700 1 -- 172.21.2.112:0/3658361203 shutdown_connections 2021-02-10T11:37:59.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:59.371 7f04a2ccb700 1 --2- 172.21.2.112:0/3658361203 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0480018730 0x7f048001ac10 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:59.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:59.371 7f04a2ccb700 1 --2- 172.21.2.112:0/3658361203 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f049c08fcc0 0x7f049c19a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:59.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:59.371 7f04a2ccb700 1 --2- 172.21.2.112:0/3658361203 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f049c08f2a0 0x7f049c08f6a0 unknown :-1 s=CLOSED pgs=1771 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:59.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:59.371 7f04a2ccb700 1 --2- 172.21.2.112:0/3658361203 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f049c0aab30 0x7f049c08ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:59.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:59.371 7f04a2ccb700 1 -- 172.21.2.112:0/3658361203 >> 172.21.2.112:0/3658361203 conn(0x7f049c19c560 msgr2=0x7f049c0ebe30 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:59.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:59.371 7f04a2ccb700 1 -- 172.21.2.112:0/3658361203 shutdown_connections 2021-02-10T11:37:59.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:37:59.371 7f04a2ccb700 1 -- 172.21.2.112:0/3658361203 wait complete. 2021-02-10T11:37:59.369 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok session ls 2021-02-10T11:37:59.767 INFO:teuthology.orchestra.run.gibba020.stdout:[ 2021-02-10T11:37:59.767 INFO:teuthology.orchestra.run.gibba020.stdout: { 2021-02-10T11:37:59.768 INFO:teuthology.orchestra.run.gibba020.stdout: "id": 7390, 2021-02-10T11:37:59.768 INFO:teuthology.orchestra.run.gibba020.stdout: "entity": { 2021-02-10T11:37:59.768 INFO:teuthology.orchestra.run.gibba020.stdout: "name": { 2021-02-10T11:37:59.768 INFO:teuthology.orchestra.run.gibba020.stdout: "type": "client", 2021-02-10T11:37:59.768 INFO:teuthology.orchestra.run.gibba020.stdout: "num": 7390 2021-02-10T11:37:59.768 INFO:teuthology.orchestra.run.gibba020.stdout: }, 2021-02-10T11:37:59.768 INFO:teuthology.orchestra.run.gibba020.stdout: "addr": { 2021-02-10T11:37:59.768 INFO:teuthology.orchestra.run.gibba020.stdout: "type": "any", 2021-02-10T11:37:59.768 INFO:teuthology.orchestra.run.gibba020.stdout: "addr": "172.21.2.120:0", 2021-02-10T11:37:59.768 INFO:teuthology.orchestra.run.gibba020.stdout: "nonce": 1069935987 2021-02-10T11:37:59.768 INFO:teuthology.orchestra.run.gibba020.stdout: } 2021-02-10T11:37:59.768 INFO:teuthology.orchestra.run.gibba020.stdout: }, 2021-02-10T11:37:59.768 INFO:teuthology.orchestra.run.gibba020.stdout: "state": "open", 2021-02-10T11:37:59.769 INFO:teuthology.orchestra.run.gibba020.stdout: "num_leases": 0, 2021-02-10T11:37:59.769 INFO:teuthology.orchestra.run.gibba020.stdout: "num_caps": 7, 2021-02-10T11:37:59.769 INFO:teuthology.orchestra.run.gibba020.stdout: "request_load_avg": 9, 2021-02-10T11:37:59.769 INFO:teuthology.orchestra.run.gibba020.stdout: "uptime": 38.117051502999999, 2021-02-10T11:37:59.769 INFO:teuthology.orchestra.run.gibba020.stdout: "requests_in_flight": 0, 2021-02-10T11:37:59.769 INFO:teuthology.orchestra.run.gibba020.stdout: "completed_requests": 1, 2021-02-10T11:37:59.769 INFO:teuthology.orchestra.run.gibba020.stdout: "reconnecting": false, 2021-02-10T11:37:59.769 INFO:teuthology.orchestra.run.gibba020.stdout: "recall_caps": { 2021-02-10T11:37:59.769 INFO:teuthology.orchestra.run.gibba020.stdout: "value": 0, 2021-02-10T11:37:59.769 INFO:teuthology.orchestra.run.gibba020.stdout: "halflife": 60 2021-02-10T11:37:59.769 INFO:teuthology.orchestra.run.gibba020.stdout: }, 2021-02-10T11:37:59.769 INFO:teuthology.orchestra.run.gibba020.stdout: "release_caps": { 2021-02-10T11:37:59.769 INFO:teuthology.orchestra.run.gibba020.stdout: "value": 0, 2021-02-10T11:37:59.769 INFO:teuthology.orchestra.run.gibba020.stdout: "halflife": 60 2021-02-10T11:37:59.770 INFO:teuthology.orchestra.run.gibba020.stdout: }, 2021-02-10T11:37:59.770 INFO:teuthology.orchestra.run.gibba020.stdout: "recall_caps_throttle": { 2021-02-10T11:37:59.770 INFO:teuthology.orchestra.run.gibba020.stdout: "value": 0, 2021-02-10T11:37:59.770 INFO:teuthology.orchestra.run.gibba020.stdout: "halflife": 2.5 2021-02-10T11:37:59.770 INFO:teuthology.orchestra.run.gibba020.stdout: }, 2021-02-10T11:37:59.770 INFO:teuthology.orchestra.run.gibba020.stdout: "recall_caps_throttle2o": { 2021-02-10T11:37:59.770 INFO:teuthology.orchestra.run.gibba020.stdout: "value": 0, 2021-02-10T11:37:59.770 INFO:teuthology.orchestra.run.gibba020.stdout: "halflife": 0.5 2021-02-10T11:37:59.770 INFO:teuthology.orchestra.run.gibba020.stdout: }, 2021-02-10T11:37:59.770 INFO:teuthology.orchestra.run.gibba020.stdout: "session_cache_liveness": { 2021-02-10T11:37:59.770 INFO:teuthology.orchestra.run.gibba020.stdout: "value": 19.242956514226488, 2021-02-10T11:37:59.770 INFO:teuthology.orchestra.run.gibba020.stdout: "halflife": 300 2021-02-10T11:37:59.771 INFO:teuthology.orchestra.run.gibba020.stdout: }, 2021-02-10T11:37:59.771 INFO:teuthology.orchestra.run.gibba020.stdout: "cap_acquisition": { 2021-02-10T11:37:59.771 INFO:teuthology.orchestra.run.gibba020.stdout: "value": 0, 2021-02-10T11:37:59.771 INFO:teuthology.orchestra.run.gibba020.stdout: "halflife": 10 2021-02-10T11:37:59.771 INFO:teuthology.orchestra.run.gibba020.stdout: }, 2021-02-10T11:37:59.771 INFO:teuthology.orchestra.run.gibba020.stdout: "inst": "client.7390 172.21.2.120:0/1069935987", 2021-02-10T11:37:59.771 INFO:teuthology.orchestra.run.gibba020.stdout: "completed_requests": [ 2021-02-10T11:37:59.771 INFO:teuthology.orchestra.run.gibba020.stdout: { 2021-02-10T11:37:59.771 INFO:teuthology.orchestra.run.gibba020.stdout: "tid": 14, 2021-02-10T11:37:59.771 INFO:teuthology.orchestra.run.gibba020.stdout: "created_ino": "0x10000000005" 2021-02-10T11:37:59.771 INFO:teuthology.orchestra.run.gibba020.stdout: } 2021-02-10T11:37:59.771 INFO:teuthology.orchestra.run.gibba020.stdout: ], 2021-02-10T11:37:59.772 INFO:teuthology.orchestra.run.gibba020.stdout: "prealloc_inos": [ 2021-02-10T11:37:59.772 INFO:teuthology.orchestra.run.gibba020.stdout: { 2021-02-10T11:37:59.772 INFO:teuthology.orchestra.run.gibba020.stdout: "start": 1099511627782, 2021-02-10T11:37:59.772 INFO:teuthology.orchestra.run.gibba020.stdout: "length": 995 2021-02-10T11:37:59.772 INFO:teuthology.orchestra.run.gibba020.stdout: } 2021-02-10T11:37:59.772 INFO:teuthology.orchestra.run.gibba020.stdout: ], 2021-02-10T11:37:59.772 INFO:teuthology.orchestra.run.gibba020.stdout: "used_inos": [], 2021-02-10T11:37:59.772 INFO:teuthology.orchestra.run.gibba020.stdout: "client_metadata": { 2021-02-10T11:37:59.772 INFO:teuthology.orchestra.run.gibba020.stdout: "features": "0x0000000000001fff", 2021-02-10T11:37:59.772 INFO:teuthology.orchestra.run.gibba020.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:37:59.772 INFO:teuthology.orchestra.run.gibba020.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:37:59.772 INFO:teuthology.orchestra.run.gibba020.stdout: "entity_id": "x", 2021-02-10T11:37:59.773 INFO:teuthology.orchestra.run.gibba020.stdout: "hostname": "gibba020", 2021-02-10T11:37:59.773 INFO:teuthology.orchestra.run.gibba020.stdout: "pid": "34990", 2021-02-10T11:37:59.773 INFO:teuthology.orchestra.run.gibba020.stdout: "root": "/" 2021-02-10T11:37:59.773 INFO:teuthology.orchestra.run.gibba020.stdout: } 2021-02-10T11:37:59.773 INFO:teuthology.orchestra.run.gibba020.stdout: } 2021-02-10T11:37:59.773 INFO:teuthology.orchestra.run.gibba020.stdout:] 2021-02-10T11:37:59.774 INFO:tasks.cephfs.filesystem:_json_asok output: [ { "id": 7390, "entity": { "name": { "type": "client", "num": 7390 }, "addr": { "type": "any", "addr": "172.21.2.120:0", "nonce": 1069935987 } }, "state": "open", "num_leases": 0, "num_caps": 7, "request_load_avg": 9, "uptime": 38.117051502999999, "requests_in_flight": 0, "completed_requests": 1, "reconnecting": false, "recall_caps": { "value": 0, "halflife": 60 }, "release_caps": { "value": 0, "halflife": 60 }, "recall_caps_throttle": { "value": 0, "halflife": 2.5 }, "recall_caps_throttle2o": { "value": 0, "halflife": 0.5 }, "session_cache_liveness": { "value": 19.242956514226488, "halflife": 300 }, "cap_acquisition": { "value": 0, "halflife": 10 }, "inst": "client.7390 172.21.2.120:0/1069935987", "completed_requests": [ { "tid": 14, "created_ino": "0x10000000005" } ], "prealloc_inos": [ { "start": 1099511627782, "length": 995 } ], "used_inos": [], "client_metadata": { "features": "0x0000000000001fff", "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", "entity_id": "x", "hostname": "gibba020", "pid": "34990", "root": "/" } } ] 2021-02-10T11:37:59.774 DEBUG:tasks.ceph_test_case:wait_until_true: waiting... 2021-02-10T11:38:04.775 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:38:05.202 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:05.195 7f2cebc94700 1 Processor -- start 2021-02-10T11:38:05.203 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:05.195 7f2cebc94700 1 -- start start 2021-02-10T11:38:05.203 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:05.196 7f2cebc94700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2ce40d1870 0x7f2ce41b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:05.203 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:05.196 7f2cebc94700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f2ce41b0a10 0x7f2ce40cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:05.203 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:05.196 7f2cebc94700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ce40f0b10 con 0x7f2ce40d1c70 2021-02-10T11:38:05.203 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:05.196 7f2cebc94700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ce40fe230 con 0x7f2ce41b0a10 2021-02-10T11:38:05.203 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:05.196 7f2cebc94700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ce40e5ed0 con 0x7f2ce40d1870 2021-02-10T11:38:05.203 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:05.196 7f2ce92af700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f2ce41b0a10 0x7f2ce40cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:05.203 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:05.196 7f2ce92af700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f2ce41b0a10 0x7f2ce40cf330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:52916/0 (socket says 172.21.2.112:52916) 2021-02-10T11:38:05.203 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:05.196 7f2ce92af700 1 -- 172.21.2.112:0/1567687563 learned_addr learned my addr 172.21.2.112:0/1567687563 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:38:05.203 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:05.196 7f2ce92af700 1 -- 172.21.2.112:0/1567687563 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2ce40d1870 msgr2=0x7f2ce41b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:05.203 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:05.196 7f2ce92af700 1 --2- 172.21.2.112:0/1567687563 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2ce40d1870 0x7f2ce41b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:05.203 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:05.196 7f2ce92af700 1 -- 172.21.2.112:0/1567687563 >> v1:172.21.2.112:6789/0 conn(0x7f2ce40d1c70 legacy=0x7f2ce40cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:05.204 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:05.196 7f2ce92af700 1 -- 172.21.2.112:0/1567687563 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ce40f0650 con 0x7f2ce41b0a10 2021-02-10T11:38:05.204 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:05.196 7f2ce8aae700 1 -- 172.21.2.112:0/1567687563 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 638449545 0 0) 0x7f2cdc000f90 con 0x7f2ce40d1c70 2021-02-10T11:38:05.204 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:05.197 7f2ce92af700 1 --2- 172.21.2.112:0/1567687563 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2ce41b0a10 0x7f2ce40cf330 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 rx=0x7f2cd8006700 tx=0x7f2cd8009b80).ready entity=mon.0 client_cookie=98c4e314e91fcb51 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:05.204 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:05.197 7f2ce8aae700 1 -- 172.21.2.112:0/1567687563 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2cd800c0c0 con 0x7f2ce41b0a10 2021-02-10T11:38:05.204 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:05.197 7f2ce8aae700 1 -- 172.21.2.112:0/1567687563 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cd800c340 con 0x7f2ce41b0a10 2021-02-10T11:38:05.204 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:05.197 7f2ce8aae700 1 -- 172.21.2.112:0/1567687563 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2cd800c860 con 0x7f2ce41b0a10 2021-02-10T11:38:05.204 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:05.197 7f2cebc94700 1 -- 172.21.2.112:0/1567687563 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2ce41b0a10 msgr2=0x7f2ce40cf330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:05.204 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:05.197 7f2cebc94700 1 --2- 172.21.2.112:0/1567687563 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2ce41b0a10 0x7f2ce40cf330 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 rx=0x7f2cd8006700 tx=0x7f2cd8009b80).stop 2021-02-10T11:38:05.204 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:05.198 7f2cebc94700 1 -- 172.21.2.112:0/1567687563 shutdown_connections 2021-02-10T11:38:05.204 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:05.198 7f2cebc94700 1 --2- 172.21.2.112:0/1567687563 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2ce41b0a10 0x7f2ce40cf330 unknown :-1 s=CLOSED pgs=1773 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:05.204 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:05.198 7f2cebc94700 1 --2- 172.21.2.112:0/1567687563 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2ce40d1870 0x7f2ce41b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:05.204 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:05.198 7f2cebc94700 1 -- 172.21.2.112:0/1567687563 >> 172.21.2.112:0/1567687563 conn(0x7f2ce40ce450 msgr2=0x7f2ce40d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:05.204 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:05.198 7f2cebc94700 1 -- 172.21.2.112:0/1567687563 shutdown_connections 2021-02-10T11:38:05.205 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:05.198 7f2cebc94700 1 -- 172.21.2.112:0/1567687563 wait complete. 2021-02-10T11:38:05.206 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:05.213 7f2cebc94700 1 Processor -- start 2021-02-10T11:38:05.206 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:05.213 7f2cebc94700 1 -- start start 2021-02-10T11:38:05.206 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:05.213 7f2cebc94700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2ce40ed0f0 0x7f2ce41b6490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:05.206 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:05.213 7f2cebc94700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2ce41b6ab0 0x7f2ce40ad450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:05.206 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:05.213 7f2cebc94700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2ce40ada70 0x7f2ce41b7840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:05.206 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:05.213 7f2cebc94700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ce40ed930 con 0x7f2ce41b6ab0 2021-02-10T11:38:05.206 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:05.213 7f2cebc94700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ce40ed4f0 con 0x7f2ce40ada70 2021-02-10T11:38:05.206 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:05.213 7f2cebc94700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ce41b6eb0 con 0x7f2ce40ed0f0 2021-02-10T11:38:05.206 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:05.213 7f2ce92af700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2ce41b6ab0 0x7f2ce40ad450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:05.207 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:05.213 7f2ce92af700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2ce41b6ab0 0x7f2ce40ad450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:52924/0 (socket says 172.21.2.112:52924) 2021-02-10T11:38:05.207 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:05.213 7f2ce92af700 1 -- 172.21.2.112:0/1257798849 learned_addr learned my addr 172.21.2.112:0/1257798849 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:38:05.207 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:05.214 7f2ce92af700 1 -- 172.21.2.112:0/1257798849 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2ce40ed0f0 msgr2=0x7f2ce41b6490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:05.207 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:05.214 7f2ce92af700 1 --2- 172.21.2.112:0/1257798849 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2ce40ed0f0 0x7f2ce41b6490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:05.207 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:05.214 7f2ce92af700 1 -- 172.21.2.112:0/1257798849 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2ce40ada70 msgr2=0x7f2ce41b7840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:05.208 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:05.214 7f2ce92af700 1 --2- 172.21.2.112:0/1257798849 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2ce40ada70 0x7f2ce41b7840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:05.208 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:05.214 7f2ce92af700 1 -- 172.21.2.112:0/1257798849 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ce40ade70 con 0x7f2ce41b6ab0 2021-02-10T11:38:05.208 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:05.214 7f2ce92af700 1 --2- 172.21.2.112:0/1257798849 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2ce41b6ab0 0x7f2ce40ad450 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 rx=0x7f2cd800b5e0 tx=0x7f2cd800b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:05.208 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:05.214 7f2cd67fc700 1 -- 172.21.2.112:0/1257798849 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2cd800bd70 con 0x7f2ce41b6ab0 2021-02-10T11:38:05.209 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:05.214 7f2cd67fc700 1 -- 172.21.2.112:0/1257798849 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cd8002030 con 0x7f2ce41b6ab0 2021-02-10T11:38:05.209 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:05.214 7f2cd67fc700 1 -- 172.21.2.112:0/1257798849 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2cd80024d0 con 0x7f2ce41b6ab0 2021-02-10T11:38:05.209 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:05.214 7f2cebc94700 1 -- 172.21.2.112:0/1257798849 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ce40ade70 con 0x7f2ce41b6ab0 2021-02-10T11:38:05.209 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:05.214 7f2cebc94700 1 -- 172.21.2.112:0/1257798849 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ce40ade70 con 0x7f2ce41b6ab0 2021-02-10T11:38:05.209 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:05.215 7f2cd67fc700 1 -- 172.21.2.112:0/1257798849 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2cd8000c90 con 0x7f2ce41b6ab0 2021-02-10T11:38:05.209 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:05.215 7f2cd67fc700 1 --2- 172.21.2.112:0/1257798849 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2ccc0186d0 0x7f2ccc01abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:05.210 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:05.215 7f2cd67fc700 1 -- 172.21.2.112:0/1257798849 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f2cd8032650 con 0x7f2ce41b6ab0 2021-02-10T11:38:05.210 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:05.215 7f2ce9ab0700 1 --2- 172.21.2.112:0/1257798849 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2ccc0186d0 0x7f2ccc01abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:05.210 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:05.216 7f2ce9ab0700 1 --2- 172.21.2.112:0/1257798849 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2ccc0186d0 0x7f2ccc01abd0 crc :-1 s=READY pgs=837 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:05.236 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:05.246 7f2cebc94700 1 -- 172.21.2.112:0/1257798849 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ce40ebc00 con 0x7f2ce41b6ab0 2021-02-10T11:38:05.238 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:05.248 7f2cd67fc700 1 -- 172.21.2.112:0/1257798849 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2cd801f510 con 0x7f2ce41b6ab0 2021-02-10T11:38:05.368 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:05.378 7f2cebc94700 1 -- 172.21.2.112:0/1257798849 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2ce40f0b10 con 0x7f2ce41b6ab0 2021-02-10T11:38:05.368 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:05.378 7f2cd67fc700 1 -- 172.21.2.112:0/1257798849 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) v1 ==== 94+0+2101 (secure 0 0 0) 0x7f2cd801f510 con 0x7f2ce41b6ab0 2021-02-10T11:38:05.369 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 65 2021-02-10T11:38:05.369 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:38:05.369 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":65,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9204,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/1512510827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":1512510827},{"type":"v1","addr":"172.21.2.112:6833","nonce":1512510827}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":60}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:36:53.563735","modified":"2021-02-10 11:36:55.573740","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9201},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9201":{"gid":9201,"name":"b","rank":0,"incarnation":61,"state":"up:active","state_seq":3,"addr":"172.21.2.120:6835/1232001750","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":1232001750},{"type":"v1","addr":"172.21.2.120:6835","nonce":1232001750}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2021-02-10T11:38:05.370 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:05.380 7f2cebc94700 1 -- 172.21.2.112:0/1257798849 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2ccc0186d0 msgr2=0x7f2ccc01abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:05.370 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:05.380 7f2cebc94700 1 --2- 172.21.2.112:0/1257798849 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2ccc0186d0 0x7f2ccc01abd0 crc :-1 s=READY pgs=837 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:05.371 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:05.380 7f2cebc94700 1 -- 172.21.2.112:0/1257798849 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2ce41b6ab0 msgr2=0x7f2ce40ad450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:05.371 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:05.380 7f2cebc94700 1 --2- 172.21.2.112:0/1257798849 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2ce41b6ab0 0x7f2ce40ad450 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 rx=0x7f2cd800b5e0 tx=0x7f2cd800b980).stop 2021-02-10T11:38:05.371 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:05.380 7f2cebc94700 1 -- 172.21.2.112:0/1257798849 shutdown_connections 2021-02-10T11:38:05.371 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:05.380 7f2cebc94700 1 --2- 172.21.2.112:0/1257798849 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2ccc0186d0 0x7f2ccc01abd0 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:05.371 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:05.380 7f2cebc94700 1 --2- 172.21.2.112:0/1257798849 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2ce40ada70 0x7f2ce41b7840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:05.371 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:05.380 7f2cebc94700 1 --2- 172.21.2.112:0/1257798849 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2ce41b6ab0 0x7f2ce40ad450 unknown :-1 s=CLOSED pgs=1774 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:05.371 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:05.380 7f2cebc94700 1 --2- 172.21.2.112:0/1257798849 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2ce40ed0f0 0x7f2ce41b6490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:05.371 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:05.380 7f2cebc94700 1 -- 172.21.2.112:0/1257798849 >> 172.21.2.112:0/1257798849 conn(0x7f2ce419c580 msgr2=0x7f2ce40916f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:05.371 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:05.380 7f2cebc94700 1 -- 172.21.2.112:0/1257798849 shutdown_connections 2021-02-10T11:38:05.371 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:05.380 7f2cebc94700 1 -- 172.21.2.112:0/1257798849 wait complete. 2021-02-10T11:38:05.378 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok session ls 2021-02-10T11:38:05.775 INFO:teuthology.orchestra.run.gibba020.stdout:[ 2021-02-10T11:38:05.776 INFO:teuthology.orchestra.run.gibba020.stdout: { 2021-02-10T11:38:05.776 INFO:teuthology.orchestra.run.gibba020.stdout: "id": 7390, 2021-02-10T11:38:05.776 INFO:teuthology.orchestra.run.gibba020.stdout: "entity": { 2021-02-10T11:38:05.776 INFO:teuthology.orchestra.run.gibba020.stdout: "name": { 2021-02-10T11:38:05.776 INFO:teuthology.orchestra.run.gibba020.stdout: "type": "client", 2021-02-10T11:38:05.777 INFO:teuthology.orchestra.run.gibba020.stdout: "num": 7390 2021-02-10T11:38:05.777 INFO:teuthology.orchestra.run.gibba020.stdout: }, 2021-02-10T11:38:05.777 INFO:teuthology.orchestra.run.gibba020.stdout: "addr": { 2021-02-10T11:38:05.777 INFO:teuthology.orchestra.run.gibba020.stdout: "type": "any", 2021-02-10T11:38:05.777 INFO:teuthology.orchestra.run.gibba020.stdout: "addr": "172.21.2.120:0", 2021-02-10T11:38:05.778 INFO:teuthology.orchestra.run.gibba020.stdout: "nonce": 1069935987 2021-02-10T11:38:05.778 INFO:teuthology.orchestra.run.gibba020.stdout: } 2021-02-10T11:38:05.778 INFO:teuthology.orchestra.run.gibba020.stdout: }, 2021-02-10T11:38:05.778 INFO:teuthology.orchestra.run.gibba020.stdout: "state": "open", 2021-02-10T11:38:05.778 INFO:teuthology.orchestra.run.gibba020.stdout: "num_leases": 0, 2021-02-10T11:38:05.779 INFO:teuthology.orchestra.run.gibba020.stdout: "num_caps": 7, 2021-02-10T11:38:05.779 INFO:teuthology.orchestra.run.gibba020.stdout: "request_load_avg": 8, 2021-02-10T11:38:05.779 INFO:teuthology.orchestra.run.gibba020.stdout: "uptime": 44.125762893000001, 2021-02-10T11:38:05.779 INFO:teuthology.orchestra.run.gibba020.stdout: "requests_in_flight": 0, 2021-02-10T11:38:05.779 INFO:teuthology.orchestra.run.gibba020.stdout: "completed_requests": 1, 2021-02-10T11:38:05.779 INFO:teuthology.orchestra.run.gibba020.stdout: "reconnecting": false, 2021-02-10T11:38:05.780 INFO:teuthology.orchestra.run.gibba020.stdout: "recall_caps": { 2021-02-10T11:38:05.780 INFO:teuthology.orchestra.run.gibba020.stdout: "value": 0, 2021-02-10T11:38:05.780 INFO:teuthology.orchestra.run.gibba020.stdout: "halflife": 60 2021-02-10T11:38:05.780 INFO:teuthology.orchestra.run.gibba020.stdout: }, 2021-02-10T11:38:05.780 INFO:teuthology.orchestra.run.gibba020.stdout: "release_caps": { 2021-02-10T11:38:05.781 INFO:teuthology.orchestra.run.gibba020.stdout: "value": 0, 2021-02-10T11:38:05.781 INFO:teuthology.orchestra.run.gibba020.stdout: "halflife": 60 2021-02-10T11:38:05.781 INFO:teuthology.orchestra.run.gibba020.stdout: }, 2021-02-10T11:38:05.781 INFO:teuthology.orchestra.run.gibba020.stdout: "recall_caps_throttle": { 2021-02-10T11:38:05.781 INFO:teuthology.orchestra.run.gibba020.stdout: "value": 0, 2021-02-10T11:38:05.781 INFO:teuthology.orchestra.run.gibba020.stdout: "halflife": 2.5 2021-02-10T11:38:05.782 INFO:teuthology.orchestra.run.gibba020.stdout: }, 2021-02-10T11:38:05.782 INFO:teuthology.orchestra.run.gibba020.stdout: "recall_caps_throttle2o": { 2021-02-10T11:38:05.782 INFO:teuthology.orchestra.run.gibba020.stdout: "value": 0, 2021-02-10T11:38:05.782 INFO:teuthology.orchestra.run.gibba020.stdout: "halflife": 0.5 2021-02-10T11:38:05.782 INFO:teuthology.orchestra.run.gibba020.stdout: }, 2021-02-10T11:38:05.783 INFO:teuthology.orchestra.run.gibba020.stdout: "session_cache_liveness": { 2021-02-10T11:38:05.783 INFO:teuthology.orchestra.run.gibba020.stdout: "value": 18.977651067349836, 2021-02-10T11:38:05.783 INFO:teuthology.orchestra.run.gibba020.stdout: "halflife": 300 2021-02-10T11:38:05.783 INFO:teuthology.orchestra.run.gibba020.stdout: }, 2021-02-10T11:38:05.783 INFO:teuthology.orchestra.run.gibba020.stdout: "cap_acquisition": { 2021-02-10T11:38:05.784 INFO:teuthology.orchestra.run.gibba020.stdout: "value": 0, 2021-02-10T11:38:05.784 INFO:teuthology.orchestra.run.gibba020.stdout: "halflife": 10 2021-02-10T11:38:05.784 INFO:teuthology.orchestra.run.gibba020.stdout: }, 2021-02-10T11:38:05.784 INFO:teuthology.orchestra.run.gibba020.stdout: "inst": "client.7390 172.21.2.120:0/1069935987", 2021-02-10T11:38:05.784 INFO:teuthology.orchestra.run.gibba020.stdout: "completed_requests": [ 2021-02-10T11:38:05.785 INFO:teuthology.orchestra.run.gibba020.stdout: { 2021-02-10T11:38:05.785 INFO:teuthology.orchestra.run.gibba020.stdout: "tid": 14, 2021-02-10T11:38:05.785 INFO:teuthology.orchestra.run.gibba020.stdout: "created_ino": "0x10000000005" 2021-02-10T11:38:05.785 INFO:teuthology.orchestra.run.gibba020.stdout: } 2021-02-10T11:38:05.785 INFO:teuthology.orchestra.run.gibba020.stdout: ], 2021-02-10T11:38:05.785 INFO:teuthology.orchestra.run.gibba020.stdout: "prealloc_inos": [ 2021-02-10T11:38:05.786 INFO:teuthology.orchestra.run.gibba020.stdout: { 2021-02-10T11:38:05.786 INFO:teuthology.orchestra.run.gibba020.stdout: "start": 1099511627782, 2021-02-10T11:38:05.786 INFO:teuthology.orchestra.run.gibba020.stdout: "length": 995 2021-02-10T11:38:05.786 INFO:teuthology.orchestra.run.gibba020.stdout: } 2021-02-10T11:38:05.786 INFO:teuthology.orchestra.run.gibba020.stdout: ], 2021-02-10T11:38:05.787 INFO:teuthology.orchestra.run.gibba020.stdout: "used_inos": [], 2021-02-10T11:38:05.787 INFO:teuthology.orchestra.run.gibba020.stdout: "client_metadata": { 2021-02-10T11:38:05.787 INFO:teuthology.orchestra.run.gibba020.stdout: "features": "0x0000000000001fff", 2021-02-10T11:38:05.787 INFO:teuthology.orchestra.run.gibba020.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:38:05.787 INFO:teuthology.orchestra.run.gibba020.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:38:05.788 INFO:teuthology.orchestra.run.gibba020.stdout: "entity_id": "x", 2021-02-10T11:38:05.788 INFO:teuthology.orchestra.run.gibba020.stdout: "hostname": "gibba020", 2021-02-10T11:38:05.788 INFO:teuthology.orchestra.run.gibba020.stdout: "pid": "34990", 2021-02-10T11:38:05.788 INFO:teuthology.orchestra.run.gibba020.stdout: "root": "/" 2021-02-10T11:38:05.788 INFO:teuthology.orchestra.run.gibba020.stdout: } 2021-02-10T11:38:05.788 INFO:teuthology.orchestra.run.gibba020.stdout: } 2021-02-10T11:38:05.789 INFO:teuthology.orchestra.run.gibba020.stdout:] 2021-02-10T11:38:05.790 INFO:tasks.cephfs.filesystem:_json_asok output: [ { "id": 7390, "entity": { "name": { "type": "client", "num": 7390 }, "addr": { "type": "any", "addr": "172.21.2.120:0", "nonce": 1069935987 } }, "state": "open", "num_leases": 0, "num_caps": 7, "request_load_avg": 8, "uptime": 44.125762893000001, "requests_in_flight": 0, "completed_requests": 1, "reconnecting": false, "recall_caps": { "value": 0, "halflife": 60 }, "release_caps": { "value": 0, "halflife": 60 }, "recall_caps_throttle": { "value": 0, "halflife": 2.5 }, "recall_caps_throttle2o": { "value": 0, "halflife": 0.5 }, "session_cache_liveness": { "value": 18.977651067349836, "halflife": 300 }, "cap_acquisition": { "value": 0, "halflife": 10 }, "inst": "client.7390 172.21.2.120:0/1069935987", "completed_requests": [ { "tid": 14, "created_ino": "0x10000000005" } ], "prealloc_inos": [ { "start": 1099511627782, "length": 995 } ], "used_inos": [], "client_metadata": { "features": "0x0000000000001fff", "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", "entity_id": "x", "hostname": "gibba020", "pid": "34990", "root": "/" } } ] 2021-02-10T11:38:05.790 DEBUG:tasks.ceph_test_case:wait_until_true: waiting... 2021-02-10T11:38:10.791 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:38:11.219 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:11.213 7f4866280700 1 Processor -- start 2021-02-10T11:38:11.220 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:11.213 7f4866280700 1 -- start start 2021-02-10T11:38:11.220 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:11.213 7f4866280700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f48600d1c70 0x7f48600f20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:11.220 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:11.213 7f4866280700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48600f0b10 con 0x7f48600d0340 2021-02-10T11:38:11.220 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:11.213 7f4866280700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48601b0be0 con 0x7f48600d1870 2021-02-10T11:38:11.220 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:11.213 7f4866280700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48600e5ed0 con 0x7f48600d1c70 2021-02-10T11:38:11.220 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:11.214 7f486489d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f48600d1c70 0x7f48600f20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:11.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:11.214 7f486489d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f48600d1c70 0x7f48600f20a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:52934/0 (socket says 172.21.2.112:52934) 2021-02-10T11:38:11.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:11.214 7f486489d700 1 -- 172.21.2.112:0/3197058651 learned_addr learned my addr 172.21.2.112:0/3197058651 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:38:11.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:11.214 7f486489d700 1 -- 172.21.2.112:0/3197058651 >> v1:172.21.2.120:6789/0 conn(0x7f48600d1870 legacy=0x7f48601b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:11.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:11.214 7f486489d700 1 -- 172.21.2.112:0/3197058651 >> v1:172.21.2.112:6789/0 conn(0x7f48600d0340 legacy=0x7f48600ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:11.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:11.214 7f486489d700 1 -- 172.21.2.112:0/3197058651 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48600efe30 con 0x7f48600d1c70 2021-02-10T11:38:11.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:11.214 7f486489d700 1 --2- 172.21.2.112:0/3197058651 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f48600d1c70 0x7f48600f20a0 secure :-1 s=READY pgs=1776 cs=0 l=1 rev1=1 rx=0x7f4858006f00 tx=0x7f4858009f10).ready entity=mon.0 client_cookie=e53ec2d861dc97b8 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:11.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:11.214 7f485effd700 1 -- 172.21.2.112:0/3197058651 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f485800b3b0 con 0x7f48600d1c70 2021-02-10T11:38:11.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:11.214 7f485effd700 1 -- 172.21.2.112:0/3197058651 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f485800b730 con 0x7f48600d1c70 2021-02-10T11:38:11.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:11.214 7f485effd700 1 -- 172.21.2.112:0/3197058651 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f485800bc50 con 0x7f48600d1c70 2021-02-10T11:38:11.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:11.214 7f4866280700 1 -- 172.21.2.112:0/3197058651 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f48600d1c70 msgr2=0x7f48600f20a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:11.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:11.214 7f4866280700 1 --2- 172.21.2.112:0/3197058651 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f48600d1c70 0x7f48600f20a0 secure :-1 s=READY pgs=1776 cs=0 l=1 rev1=1 rx=0x7f4858006f00 tx=0x7f4858009f10).stop 2021-02-10T11:38:11.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:11.215 7f4866280700 1 -- 172.21.2.112:0/3197058651 shutdown_connections 2021-02-10T11:38:11.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:11.215 7f4866280700 1 --2- 172.21.2.112:0/3197058651 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f48600d1c70 0x7f48600f20a0 unknown :-1 s=CLOSED pgs=1776 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:11.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:11.215 7f4866280700 1 -- 172.21.2.112:0/3197058651 >> 172.21.2.112:0/3197058651 conn(0x7f48600ce450 msgr2=0x7f48600d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:11.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:11.215 7f4866280700 1 -- 172.21.2.112:0/3197058651 shutdown_connections 2021-02-10T11:38:11.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:11.215 7f4866280700 1 -- 172.21.2.112:0/3197058651 wait complete. 2021-02-10T11:38:11.224 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:11.230 7f4866280700 1 Processor -- start 2021-02-10T11:38:11.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:11.230 7f4866280700 1 -- start start 2021-02-10T11:38:11.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:11.230 7f4866280700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f48600c0d20 0x7f48600c1120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:11.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:11.230 7f4866280700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f48600c1740 0x7f48600c1b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:11.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:11.230 7f4866280700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f48600ab660 0x7f48600aba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:11.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:11.230 7f4866280700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48600b9860 con 0x7f48600ab660 2021-02-10T11:38:11.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:11.230 7f4866280700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48600ac080 con 0x7f48600c0d20 2021-02-10T11:38:11.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:11.230 7f4866280700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48600ac2a0 con 0x7f48600c1740 2021-02-10T11:38:11.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:11.230 7f486489d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f48600ab660 0x7f48600aba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:11.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:11.230 7f486489d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f48600ab660 0x7f48600aba60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:52936/0 (socket says 172.21.2.112:52936) 2021-02-10T11:38:11.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:11.230 7f486489d700 1 -- 172.21.2.112:0/3862157704 learned_addr learned my addr 172.21.2.112:0/3862157704 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:38:11.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:11.231 7f486489d700 1 -- 172.21.2.112:0/3862157704 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f48600c1740 msgr2=0x7f48600c1b40 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:38:11.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:11.231 7f486489d700 1 --2- 172.21.2.112:0/3862157704 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f48600c1740 0x7f48600c1b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:11.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:11.231 7f486489d700 1 -- 172.21.2.112:0/3862157704 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f48600c0d20 msgr2=0x7f48600c1120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:11.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:11.231 7f486489d700 1 --2- 172.21.2.112:0/3862157704 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f48600c0d20 0x7f48600c1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:11.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:11.231 7f486489d700 1 -- 172.21.2.112:0/3862157704 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48600ac4c0 con 0x7f48600ab660 2021-02-10T11:38:11.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:11.231 7f486489d700 1 --2- 172.21.2.112:0/3862157704 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f48600ab660 0x7f48600aba60 secure :-1 s=READY pgs=1777 cs=0 l=1 rev1=1 rx=0x7f485800b9b0 tx=0x7f485800bd50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:11.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:11.231 7f485cff9700 1 -- 172.21.2.112:0/3862157704 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4858009450 con 0x7f48600ab660 2021-02-10T11:38:11.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:11.231 7f485cff9700 1 -- 172.21.2.112:0/3862157704 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f485800c2f0 con 0x7f48600ab660 2021-02-10T11:38:11.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:11.231 7f4866280700 1 -- 172.21.2.112:0/3862157704 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48600ac4c0 con 0x7f48600ab660 2021-02-10T11:38:11.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:11.231 7f485cff9700 1 -- 172.21.2.112:0/3862157704 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f485800c810 con 0x7f48600ab660 2021-02-10T11:38:11.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:11.231 7f4866280700 1 -- 172.21.2.112:0/3862157704 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48600aca50 con 0x7f48600ab660 2021-02-10T11:38:11.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:11.232 7f485cff9700 1 -- 172.21.2.112:0/3862157704 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4858001270 con 0x7f48600ab660 2021-02-10T11:38:11.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:11.232 7f485cff9700 1 --2- 172.21.2.112:0/3862157704 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f483c0186c0 0x7f483c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:11.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:11.232 7f485cff9700 1 -- 172.21.2.112:0/3862157704 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f48580327c0 con 0x7f48600ab660 2021-02-10T11:38:11.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:11.232 7f485ffff700 1 --2- 172.21.2.112:0/3862157704 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f483c0186c0 0x7f483c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:11.227 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:11.232 7f485ffff700 1 --2- 172.21.2.112:0/3862157704 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f483c0186c0 0x7f483c01abc0 crc :-1 s=READY pgs=838 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:11.254 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:11.263 7f4866280700 1 -- 172.21.2.112:0/3862157704 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48600e7df0 con 0x7f48600ab660 2021-02-10T11:38:11.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:11.266 7f485cff9700 1 -- 172.21.2.112:0/3862157704 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f485802dfe0 con 0x7f48600ab660 2021-02-10T11:38:11.382 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:11.392 7f4866280700 1 -- 172.21.2.112:0/3862157704 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f48600ce450 con 0x7f48600ab660 2021-02-10T11:38:11.383 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:11.392 7f485cff9700 1 -- 172.21.2.112:0/3862157704 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) v1 ==== 94+0+2101 (secure 0 0 0) 0x7f485802dfe0 con 0x7f48600ab660 2021-02-10T11:38:11.383 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:38:11.383 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":65,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9204,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/1512510827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":1512510827},{"type":"v1","addr":"172.21.2.112:6833","nonce":1512510827}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":60}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:36:53.563735","modified":"2021-02-10 11:36:55.573740","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9201},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9201":{"gid":9201,"name":"b","rank":0,"incarnation":61,"state":"up:active","state_seq":3,"addr":"172.21.2.120:6835/1232001750","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":1232001750},{"type":"v1","addr":"172.21.2.120:6835","nonce":1232001750}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2021-02-10T11:38:11.384 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 65 2021-02-10T11:38:11.385 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:11.394 7f4866280700 1 -- 172.21.2.112:0/3862157704 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f483c0186c0 msgr2=0x7f483c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:11.385 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:11.394 7f4866280700 1 --2- 172.21.2.112:0/3862157704 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f483c0186c0 0x7f483c01abc0 crc :-1 s=READY pgs=838 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:11.385 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:11.394 7f4866280700 1 -- 172.21.2.112:0/3862157704 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f48600ab660 msgr2=0x7f48600aba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:11.385 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:11.394 7f4866280700 1 --2- 172.21.2.112:0/3862157704 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f48600ab660 0x7f48600aba60 secure :-1 s=READY pgs=1777 cs=0 l=1 rev1=1 rx=0x7f485800b9b0 tx=0x7f485800bd50).stop 2021-02-10T11:38:11.385 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:11.394 7f4866280700 1 -- 172.21.2.112:0/3862157704 shutdown_connections 2021-02-10T11:38:11.385 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:11.394 7f4866280700 1 --2- 172.21.2.112:0/3862157704 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f483c0186c0 0x7f483c01abc0 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:11.385 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:11.394 7f4866280700 1 --2- 172.21.2.112:0/3862157704 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f48600ab660 0x7f48600aba60 unknown :-1 s=CLOSED pgs=1777 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:11.385 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:11.394 7f4866280700 1 --2- 172.21.2.112:0/3862157704 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f48600c1740 0x7f48600c1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:11.385 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:11.394 7f4866280700 1 --2- 172.21.2.112:0/3862157704 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f48600c0d20 0x7f48600c1120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:11.385 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:11.394 7f4866280700 1 -- 172.21.2.112:0/3862157704 >> 172.21.2.112:0/3862157704 conn(0x7f48600b2470 msgr2=0x7f48600b6970 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:11.386 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:11.395 7f4866280700 1 -- 172.21.2.112:0/3862157704 shutdown_connections 2021-02-10T11:38:11.386 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:11.395 7f4866280700 1 -- 172.21.2.112:0/3862157704 wait complete. 2021-02-10T11:38:11.394 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok session ls 2021-02-10T11:38:11.791 INFO:teuthology.orchestra.run.gibba020.stdout:[ 2021-02-10T11:38:11.791 INFO:teuthology.orchestra.run.gibba020.stdout: { 2021-02-10T11:38:11.791 INFO:teuthology.orchestra.run.gibba020.stdout: "id": 7390, 2021-02-10T11:38:11.791 INFO:teuthology.orchestra.run.gibba020.stdout: "entity": { 2021-02-10T11:38:11.792 INFO:teuthology.orchestra.run.gibba020.stdout: "name": { 2021-02-10T11:38:11.792 INFO:teuthology.orchestra.run.gibba020.stdout: "type": "client", 2021-02-10T11:38:11.792 INFO:teuthology.orchestra.run.gibba020.stdout: "num": 7390 2021-02-10T11:38:11.792 INFO:teuthology.orchestra.run.gibba020.stdout: }, 2021-02-10T11:38:11.792 INFO:teuthology.orchestra.run.gibba020.stdout: "addr": { 2021-02-10T11:38:11.792 INFO:teuthology.orchestra.run.gibba020.stdout: "type": "any", 2021-02-10T11:38:11.792 INFO:teuthology.orchestra.run.gibba020.stdout: "addr": "172.21.2.120:0", 2021-02-10T11:38:11.792 INFO:teuthology.orchestra.run.gibba020.stdout: "nonce": 1069935987 2021-02-10T11:38:11.792 INFO:teuthology.orchestra.run.gibba020.stdout: } 2021-02-10T11:38:11.792 INFO:teuthology.orchestra.run.gibba020.stdout: }, 2021-02-10T11:38:11.792 INFO:teuthology.orchestra.run.gibba020.stdout: "state": "open", 2021-02-10T11:38:11.792 INFO:teuthology.orchestra.run.gibba020.stdout: "num_leases": 0, 2021-02-10T11:38:11.793 INFO:teuthology.orchestra.run.gibba020.stdout: "num_caps": 7, 2021-02-10T11:38:11.793 INFO:teuthology.orchestra.run.gibba020.stdout: "request_load_avg": 7, 2021-02-10T11:38:11.793 INFO:teuthology.orchestra.run.gibba020.stdout: "uptime": 50.141478679999999, 2021-02-10T11:38:11.793 INFO:teuthology.orchestra.run.gibba020.stdout: "requests_in_flight": 0, 2021-02-10T11:38:11.793 INFO:teuthology.orchestra.run.gibba020.stdout: "completed_requests": 1, 2021-02-10T11:38:11.793 INFO:teuthology.orchestra.run.gibba020.stdout: "reconnecting": false, 2021-02-10T11:38:11.793 INFO:teuthology.orchestra.run.gibba020.stdout: "recall_caps": { 2021-02-10T11:38:11.793 INFO:teuthology.orchestra.run.gibba020.stdout: "value": 0, 2021-02-10T11:38:11.793 INFO:teuthology.orchestra.run.gibba020.stdout: "halflife": 60 2021-02-10T11:38:11.793 INFO:teuthology.orchestra.run.gibba020.stdout: }, 2021-02-10T11:38:11.793 INFO:teuthology.orchestra.run.gibba020.stdout: "release_caps": { 2021-02-10T11:38:11.793 INFO:teuthology.orchestra.run.gibba020.stdout: "value": 0, 2021-02-10T11:38:11.794 INFO:teuthology.orchestra.run.gibba020.stdout: "halflife": 60 2021-02-10T11:38:11.794 INFO:teuthology.orchestra.run.gibba020.stdout: }, 2021-02-10T11:38:11.794 INFO:teuthology.orchestra.run.gibba020.stdout: "recall_caps_throttle": { 2021-02-10T11:38:11.794 INFO:teuthology.orchestra.run.gibba020.stdout: "value": 0, 2021-02-10T11:38:11.794 INFO:teuthology.orchestra.run.gibba020.stdout: "halflife": 2.5 2021-02-10T11:38:11.794 INFO:teuthology.orchestra.run.gibba020.stdout: }, 2021-02-10T11:38:11.794 INFO:teuthology.orchestra.run.gibba020.stdout: "recall_caps_throttle2o": { 2021-02-10T11:38:11.794 INFO:teuthology.orchestra.run.gibba020.stdout: "value": 0, 2021-02-10T11:38:11.794 INFO:teuthology.orchestra.run.gibba020.stdout: "halflife": 0.5 2021-02-10T11:38:11.794 INFO:teuthology.orchestra.run.gibba020.stdout: }, 2021-02-10T11:38:11.795 INFO:teuthology.orchestra.run.gibba020.stdout: "session_cache_liveness": { 2021-02-10T11:38:11.795 INFO:teuthology.orchestra.run.gibba020.stdout: "value": 18.715700535608597, 2021-02-10T11:38:11.795 INFO:teuthology.orchestra.run.gibba020.stdout: "halflife": 300 2021-02-10T11:38:11.795 INFO:teuthology.orchestra.run.gibba020.stdout: }, 2021-02-10T11:38:11.795 INFO:teuthology.orchestra.run.gibba020.stdout: "cap_acquisition": { 2021-02-10T11:38:11.795 INFO:teuthology.orchestra.run.gibba020.stdout: "value": 0, 2021-02-10T11:38:11.796 INFO:teuthology.orchestra.run.gibba020.stdout: "halflife": 10 2021-02-10T11:38:11.796 INFO:teuthology.orchestra.run.gibba020.stdout: }, 2021-02-10T11:38:11.796 INFO:teuthology.orchestra.run.gibba020.stdout: "inst": "client.7390 172.21.2.120:0/1069935987", 2021-02-10T11:38:11.796 INFO:teuthology.orchestra.run.gibba020.stdout: "completed_requests": [ 2021-02-10T11:38:11.796 INFO:teuthology.orchestra.run.gibba020.stdout: { 2021-02-10T11:38:11.797 INFO:teuthology.orchestra.run.gibba020.stdout: "tid": 14, 2021-02-10T11:38:11.797 INFO:teuthology.orchestra.run.gibba020.stdout: "created_ino": "0x10000000005" 2021-02-10T11:38:11.797 INFO:teuthology.orchestra.run.gibba020.stdout: } 2021-02-10T11:38:11.797 INFO:teuthology.orchestra.run.gibba020.stdout: ], 2021-02-10T11:38:11.797 INFO:teuthology.orchestra.run.gibba020.stdout: "prealloc_inos": [ 2021-02-10T11:38:11.797 INFO:teuthology.orchestra.run.gibba020.stdout: { 2021-02-10T11:38:11.797 INFO:teuthology.orchestra.run.gibba020.stdout: "start": 1099511627782, 2021-02-10T11:38:11.798 INFO:teuthology.orchestra.run.gibba020.stdout: "length": 995 2021-02-10T11:38:11.798 INFO:teuthology.orchestra.run.gibba020.stdout: } 2021-02-10T11:38:11.798 INFO:teuthology.orchestra.run.gibba020.stdout: ], 2021-02-10T11:38:11.798 INFO:teuthology.orchestra.run.gibba020.stdout: "used_inos": [], 2021-02-10T11:38:11.798 INFO:teuthology.orchestra.run.gibba020.stdout: "client_metadata": { 2021-02-10T11:38:11.798 INFO:teuthology.orchestra.run.gibba020.stdout: "features": "0x0000000000001fff", 2021-02-10T11:38:11.798 INFO:teuthology.orchestra.run.gibba020.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:38:11.798 INFO:teuthology.orchestra.run.gibba020.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:38:11.798 INFO:teuthology.orchestra.run.gibba020.stdout: "entity_id": "x", 2021-02-10T11:38:11.798 INFO:teuthology.orchestra.run.gibba020.stdout: "hostname": "gibba020", 2021-02-10T11:38:11.798 INFO:teuthology.orchestra.run.gibba020.stdout: "pid": "34990", 2021-02-10T11:38:11.798 INFO:teuthology.orchestra.run.gibba020.stdout: "root": "/" 2021-02-10T11:38:11.798 INFO:teuthology.orchestra.run.gibba020.stdout: } 2021-02-10T11:38:11.799 INFO:teuthology.orchestra.run.gibba020.stdout: } 2021-02-10T11:38:11.799 INFO:teuthology.orchestra.run.gibba020.stdout:] 2021-02-10T11:38:11.800 INFO:tasks.cephfs.filesystem:_json_asok output: [ { "id": 7390, "entity": { "name": { "type": "client", "num": 7390 }, "addr": { "type": "any", "addr": "172.21.2.120:0", "nonce": 1069935987 } }, "state": "open", "num_leases": 0, "num_caps": 7, "request_load_avg": 7, "uptime": 50.141478679999999, "requests_in_flight": 0, "completed_requests": 1, "reconnecting": false, "recall_caps": { "value": 0, "halflife": 60 }, "release_caps": { "value": 0, "halflife": 60 }, "recall_caps_throttle": { "value": 0, "halflife": 2.5 }, "recall_caps_throttle2o": { "value": 0, "halflife": 0.5 }, "session_cache_liveness": { "value": 18.715700535608597, "halflife": 300 }, "cap_acquisition": { "value": 0, "halflife": 10 }, "inst": "client.7390 172.21.2.120:0/1069935987", "completed_requests": [ { "tid": 14, "created_ino": "0x10000000005" } ], "prealloc_inos": [ { "start": 1099511627782, "length": 995 } ], "used_inos": [], "client_metadata": { "features": "0x0000000000001fff", "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", "entity_id": "x", "hostname": "gibba020", "pid": "34990", "root": "/" } } ] 2021-02-10T11:38:11.800 DEBUG:tasks.ceph_test_case:wait_until_true: waiting... 2021-02-10T11:38:16.801 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:38:17.231 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:17.221 7fa593b82700 1 Processor -- start 2021-02-10T11:38:17.231 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:17.221 7fa593b82700 1 -- start start 2021-02-10T11:38:17.231 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:17.222 7fa593b82700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fa58c0d0340 0x7fa58c0ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:17.231 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:17.222 7fa593b82700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa58c0d1c70 0x7fa58c0efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:17.231 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:17.222 7fa593b82700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa58c0f0b10 con 0x7fa58c0d1870 2021-02-10T11:38:17.231 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:17.222 7fa593b82700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa58c0e5ed0 con 0x7fa58c0d0340 2021-02-10T11:38:17.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:17.222 7fa593b82700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa58c1b0be0 con 0x7fa58c0d1c70 2021-02-10T11:38:17.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:17.222 7fa59199e700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fa58c0d1870 0x7fa58c1b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:52406/0 (socket says 172.21.2.112:52406) 2021-02-10T11:38:17.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:17.222 7fa59199e700 1 -- 172.21.2.112:0/3314361474 learned_addr learned my addr 172.21.2.112:0/3314361474 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:38:17.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:17.222 7fa59099c700 1 -- 172.21.2.112:0/3314361474 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1246587694 0 0) 0x7fa5880010b0 con 0x7fa58c0d1870 2021-02-10T11:38:17.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:17.223 7fa59099c700 1 -- 172.21.2.112:0/3314361474 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa578002fe0 con 0x7fa58c0d1870 2021-02-10T11:38:17.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:17.223 7fa59119d700 1 --2- 172.21.2.112:0/3314361474 >> v2:172.21.2.120:3300/0 conn(0x7fa58c0d0340 0x7fa58c0ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:17.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:17.223 7fa59219f700 1 --2- 172.21.2.112:0/3314361474 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa58c0d1c70 0x7fa58c0efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:17.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:17.223 7fa59099c700 1 -- 172.21.2.112:0/3314361474 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3286955053 0 0) 0x7fa5880010b0 con 0x7fa58c0d1870 2021-02-10T11:38:17.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:17.223 7fa59099c700 1 -- 172.21.2.112:0/3314361474 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa58c0d1c70 msgr2=0x7fa58c0efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:17.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:17.223 7fa59099c700 1 --2- 172.21.2.112:0/3314361474 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa58c0d1c70 0x7fa58c0efc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:17.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:17.223 7fa59099c700 1 -- 172.21.2.112:0/3314361474 >> v2:172.21.2.120:3300/0 conn(0x7fa58c0d0340 msgr2=0x7fa58c0ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:17.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:17.223 7fa59099c700 1 --2- 172.21.2.112:0/3314361474 >> v2:172.21.2.120:3300/0 conn(0x7fa58c0d0340 0x7fa58c0ef640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:17.233 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:17.223 7fa59099c700 1 -- 172.21.2.112:0/3314361474 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa58c0f20a0 con 0x7fa58c0d1870 2021-02-10T11:38:17.233 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:17.223 7fa59099c700 1 -- 172.21.2.112:0/3314361474 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7fa5880015b0 con 0x7fa58c0d1870 2021-02-10T11:38:17.233 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:17.223 7fa59099c700 1 -- 172.21.2.112:0/3314361474 >> v1:172.21.2.112:6789/0 conn(0x7fa58c0d1870 legacy=0x7fa58c1b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:17.233 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:17.223 7fa59099c700 1 --2- 172.21.2.112:0/3314361474 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa578003b10 0x7fa578006010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:17.233 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:17.223 7fa59099c700 1 --2- 172.21.2.112:0/3314361474 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa5780068b0 0x7fa578008dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:17.233 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:17.223 7fa59099c700 1 --2- 172.21.2.112:0/3314361474 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa5780096d0 0x7fa57800bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:17.233 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:17.223 7fa59099c700 1 -- 172.21.2.112:0/3314361474 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa578050390 con 0x7fa578003b10 2021-02-10T11:38:17.233 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:17.223 7fa59099c700 1 -- 172.21.2.112:0/3314361474 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5780505b0 con 0x7fa5780068b0 2021-02-10T11:38:17.233 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:17.223 7fa59099c700 1 -- 172.21.2.112:0/3314361474 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5780507d0 con 0x7fa5780096d0 2021-02-10T11:38:17.233 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:17.223 7fa59219f700 1 --2- 172.21.2.112:0/3314361474 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa58c0d1c70 0x7fa58c0efc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:38:17.233 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:17.223 7fa59199e700 1 --2- 172.21.2.112:0/3314361474 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa578003b10 0x7fa578006010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:17.234 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:17.223 7fa59119d700 1 --2- 172.21.2.112:0/3314361474 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa5780068b0 0x7fa578008dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:17.234 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:17.223 7fa59199e700 1 -- 172.21.2.112:0/3314361474 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa5780096d0 msgr2=0x7fa57800bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:17.234 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:17.223 7fa59199e700 1 --2- 172.21.2.112:0/3314361474 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa5780096d0 0x7fa57800bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:17.234 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:17.223 7fa59199e700 1 -- 172.21.2.112:0/3314361474 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa5780068b0 msgr2=0x7fa578008dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:17.234 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:17.223 7fa59199e700 1 --2- 172.21.2.112:0/3314361474 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa5780068b0 0x7fa578008dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:17.234 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:17.223 7fa59199e700 1 -- 172.21.2.112:0/3314361474 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa578050a70 con 0x7fa578003b10 2021-02-10T11:38:17.234 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:17.223 7fa59119d700 1 --2- 172.21.2.112:0/3314361474 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa5780068b0 0x7fa578008dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:38:17.234 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:17.224 7fa59199e700 1 --2- 172.21.2.112:0/3314361474 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa578003b10 0x7fa578006010 secure :-1 s=READY pgs=1779 cs=0 l=1 rev1=1 rx=0x7fa588001360 tx=0x7fa588007d60).ready entity=mon.0 client_cookie=6a085f4e6f326d33 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:17.234 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:17.224 7fa59099c700 1 -- 172.21.2.112:0/3314361474 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa58800a520 con 0x7fa578003b10 2021-02-10T11:38:17.234 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:17.224 7fa59099c700 1 -- 172.21.2.112:0/3314361474 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa588001450 con 0x7fa578003b10 2021-02-10T11:38:17.234 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:17.224 7fa59099c700 1 -- 172.21.2.112:0/3314361474 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa58800ab20 con 0x7fa578003b10 2021-02-10T11:38:17.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:17.224 7fa593b82700 1 -- 172.21.2.112:0/3314361474 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa578003b10 msgr2=0x7fa578006010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:17.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:17.224 7fa593b82700 1 --2- 172.21.2.112:0/3314361474 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa578003b10 0x7fa578006010 secure :-1 s=READY pgs=1779 cs=0 l=1 rev1=1 rx=0x7fa588001360 tx=0x7fa588007d60).stop 2021-02-10T11:38:17.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:17.224 7fa59219f700 1 -- 172.21.2.112:0/3314361474 reap_dead start 2021-02-10T11:38:17.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:17.224 7fa593b82700 1 -- 172.21.2.112:0/3314361474 shutdown_connections 2021-02-10T11:38:17.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:17.224 7fa593b82700 1 -- 172.21.2.112:0/3314361474 >> 172.21.2.112:0/3314361474 conn(0x7fa58c0ce450 msgr2=0x7fa58c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:17.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:17.225 7fa593b82700 1 -- 172.21.2.112:0/3314361474 shutdown_connections 2021-02-10T11:38:17.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:17.225 7fa593b82700 1 -- 172.21.2.112:0/3314361474 wait complete. 2021-02-10T11:38:17.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:17.239 7fa593b82700 1 Processor -- start 2021-02-10T11:38:17.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:17.239 7fa593b82700 1 -- start start 2021-02-10T11:38:17.236 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:17.240 7fa593b82700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa58c1942b0 0x7fa58c0b3550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:17.236 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:17.240 7fa593b82700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa58c195070 0x7fa58c195470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:17.236 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:17.240 7fa593b82700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa58c0bb8e0 0x7fa58c0bbce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:17.236 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:17.240 7fa593b82700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa58c1946b0 con 0x7fa58c1942b0 2021-02-10T11:38:17.236 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:17.240 7fa593b82700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa58c0b3b70 con 0x7fa58c195070 2021-02-10T11:38:17.236 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:17.240 7fa593b82700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa58c0b3d90 con 0x7fa58c0bb8e0 2021-02-10T11:38:17.236 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:17.240 7fa59199e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa58c1942b0 0x7fa58c0b3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:17.236 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:17.240 7fa59199e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa58c1942b0 0x7fa58c0b3550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:52956/0 (socket says 172.21.2.112:52956) 2021-02-10T11:38:17.236 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:17.240 7fa59199e700 1 -- 172.21.2.112:0/2359000943 learned_addr learned my addr 172.21.2.112:0/2359000943 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:38:17.236 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:17.240 7fa59219f700 1 --2- 172.21.2.112:0/2359000943 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa58c0bb8e0 0x7fa58c0bbce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:17.236 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:17.240 7fa59119d700 1 --2- 172.21.2.112:0/2359000943 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa58c195070 0x7fa58c195470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:17.236 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:17.240 7fa59199e700 1 -- 172.21.2.112:0/2359000943 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa58c0bb8e0 msgr2=0x7fa58c0bbce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:17.236 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:17.240 7fa59199e700 1 --2- 172.21.2.112:0/2359000943 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa58c0bb8e0 0x7fa58c0bbce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:17.237 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:17.240 7fa59199e700 1 -- 172.21.2.112:0/2359000943 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa58c195070 msgr2=0x7fa58c195470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:17.237 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:17.240 7fa59199e700 1 --2- 172.21.2.112:0/2359000943 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa58c195070 0x7fa58c195470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:17.237 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:17.240 7fa59199e700 1 -- 172.21.2.112:0/2359000943 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa58c1a49e0 con 0x7fa58c1942b0 2021-02-10T11:38:17.237 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:17.240 7fa59219f700 1 --2- 172.21.2.112:0/2359000943 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa58c0bb8e0 0x7fa58c0bbce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:38:17.237 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:17.241 7fa59199e700 1 --2- 172.21.2.112:0/2359000943 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa58c1942b0 0x7fa58c0b3550 secure :-1 s=READY pgs=1780 cs=0 l=1 rev1=1 rx=0x7fa588001c10 tx=0x7fa58800a940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:17.238 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:17.241 7fa57e7fc700 1 -- 172.21.2.112:0/2359000943 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa58800ad30 con 0x7fa58c1942b0 2021-02-10T11:38:17.238 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:17.241 7fa57e7fc700 1 -- 172.21.2.112:0/2359000943 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa588004140 con 0x7fa58c1942b0 2021-02-10T11:38:17.238 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:17.241 7fa57e7fc700 1 -- 172.21.2.112:0/2359000943 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa588004660 con 0x7fa58c1942b0 2021-02-10T11:38:17.238 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:17.241 7fa593b82700 1 -- 172.21.2.112:0/2359000943 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa58c1a49e0 con 0x7fa58c1942b0 2021-02-10T11:38:17.238 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:17.241 7fa593b82700 1 -- 172.21.2.112:0/2359000943 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa58c1a49e0 con 0x7fa58c1942b0 2021-02-10T11:38:17.238 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:17.241 7fa57e7fc700 1 -- 172.21.2.112:0/2359000943 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa588000c90 con 0x7fa58c1942b0 2021-02-10T11:38:17.238 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:17.242 7fa57e7fc700 1 --2- 172.21.2.112:0/2359000943 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa574018730 0x7fa57401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:17.238 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:17.242 7fa57e7fc700 1 -- 172.21.2.112:0/2359000943 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fa5880327a0 con 0x7fa58c1942b0 2021-02-10T11:38:17.238 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:17.242 7fa59119d700 1 --2- 172.21.2.112:0/2359000943 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa574018730 0x7fa57401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:17.238 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:17.242 7fa59119d700 1 --2- 172.21.2.112:0/2359000943 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa574018730 0x7fa57401ac10 crc :-1 s=READY pgs=839 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:17.262 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:17.272 7fa593b82700 1 -- 172.21.2.112:0/2359000943 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa58c1a49e0 con 0x7fa58c1942b0 2021-02-10T11:38:17.265 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:17.274 7fa57e7fc700 1 -- 172.21.2.112:0/2359000943 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa58801f4e0 con 0x7fa58c1942b0 2021-02-10T11:38:17.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:17.402 7fa593b82700 1 -- 172.21.2.112:0/2359000943 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa58c1a49e0 con 0x7fa58c1942b0 2021-02-10T11:38:17.393 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:17.402 7fa57e7fc700 1 -- 172.21.2.112:0/2359000943 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) v1 ==== 94+0+2101 (secure 0 0 0) 0x7fa58800ad30 con 0x7fa58c1942b0 2021-02-10T11:38:17.393 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:38:17.393 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":65,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9204,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/1512510827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":1512510827},{"type":"v1","addr":"172.21.2.112:6833","nonce":1512510827}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":60}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:36:53.563735","modified":"2021-02-10 11:36:55.573740","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9201},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9201":{"gid":9201,"name":"b","rank":0,"incarnation":61,"state":"up:active","state_seq":3,"addr":"172.21.2.120:6835/1232001750","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":1232001750},{"type":"v1","addr":"172.21.2.120:6835","nonce":1232001750}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2021-02-10T11:38:17.394 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 65 2021-02-10T11:38:17.394 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:17.404 7fa593b82700 1 -- 172.21.2.112:0/2359000943 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa574018730 msgr2=0x7fa57401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:17.395 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:17.404 7fa593b82700 1 --2- 172.21.2.112:0/2359000943 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa574018730 0x7fa57401ac10 crc :-1 s=READY pgs=839 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:17.395 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:17.404 7fa593b82700 1 -- 172.21.2.112:0/2359000943 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa58c1942b0 msgr2=0x7fa58c0b3550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:17.395 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:17.404 7fa593b82700 1 --2- 172.21.2.112:0/2359000943 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa58c1942b0 0x7fa58c0b3550 secure :-1 s=READY pgs=1780 cs=0 l=1 rev1=1 rx=0x7fa588001c10 tx=0x7fa58800a940).stop 2021-02-10T11:38:17.395 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:17.404 7fa593b82700 1 -- 172.21.2.112:0/2359000943 shutdown_connections 2021-02-10T11:38:17.395 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:17.404 7fa593b82700 1 --2- 172.21.2.112:0/2359000943 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa574018730 0x7fa57401ac10 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:17.395 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:17.404 7fa593b82700 1 --2- 172.21.2.112:0/2359000943 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa58c195070 0x7fa58c195470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:17.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:17.404 7fa593b82700 1 --2- 172.21.2.112:0/2359000943 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa58c0bb8e0 0x7fa58c0bbce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:17.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:17.404 7fa593b82700 1 --2- 172.21.2.112:0/2359000943 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa58c1942b0 0x7fa58c0b3550 unknown :-1 s=CLOSED pgs=1780 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:17.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:17.404 7fa593b82700 1 -- 172.21.2.112:0/2359000943 >> 172.21.2.112:0/2359000943 conn(0x7fa58c0b3040 msgr2=0x7fa58c0c3250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:17.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:17.404 7fa593b82700 1 -- 172.21.2.112:0/2359000943 shutdown_connections 2021-02-10T11:38:17.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:17.404 7fa593b82700 1 -- 172.21.2.112:0/2359000943 wait complete. 2021-02-10T11:38:17.403 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok session ls 2021-02-10T11:38:17.799 INFO:teuthology.orchestra.run.gibba020.stdout:[ 2021-02-10T11:38:17.800 INFO:teuthology.orchestra.run.gibba020.stdout: { 2021-02-10T11:38:17.800 INFO:teuthology.orchestra.run.gibba020.stdout: "id": 7390, 2021-02-10T11:38:17.800 INFO:teuthology.orchestra.run.gibba020.stdout: "entity": { 2021-02-10T11:38:17.800 INFO:teuthology.orchestra.run.gibba020.stdout: "name": { 2021-02-10T11:38:17.800 INFO:teuthology.orchestra.run.gibba020.stdout: "type": "client", 2021-02-10T11:38:17.800 INFO:teuthology.orchestra.run.gibba020.stdout: "num": 7390 2021-02-10T11:38:17.800 INFO:teuthology.orchestra.run.gibba020.stdout: }, 2021-02-10T11:38:17.800 INFO:teuthology.orchestra.run.gibba020.stdout: "addr": { 2021-02-10T11:38:17.800 INFO:teuthology.orchestra.run.gibba020.stdout: "type": "any", 2021-02-10T11:38:17.800 INFO:teuthology.orchestra.run.gibba020.stdout: "addr": "172.21.2.120:0", 2021-02-10T11:38:17.800 INFO:teuthology.orchestra.run.gibba020.stdout: "nonce": 1069935987 2021-02-10T11:38:17.800 INFO:teuthology.orchestra.run.gibba020.stdout: } 2021-02-10T11:38:17.800 INFO:teuthology.orchestra.run.gibba020.stdout: }, 2021-02-10T11:38:17.801 INFO:teuthology.orchestra.run.gibba020.stdout: "state": "open", 2021-02-10T11:38:17.801 INFO:teuthology.orchestra.run.gibba020.stdout: "num_leases": 0, 2021-02-10T11:38:17.801 INFO:teuthology.orchestra.run.gibba020.stdout: "num_caps": 7, 2021-02-10T11:38:17.801 INFO:teuthology.orchestra.run.gibba020.stdout: "request_load_avg": 7, 2021-02-10T11:38:17.801 INFO:teuthology.orchestra.run.gibba020.stdout: "uptime": 56.149199467999999, 2021-02-10T11:38:17.801 INFO:teuthology.orchestra.run.gibba020.stdout: "requests_in_flight": 0, 2021-02-10T11:38:17.801 INFO:teuthology.orchestra.run.gibba020.stdout: "completed_requests": 1, 2021-02-10T11:38:17.801 INFO:teuthology.orchestra.run.gibba020.stdout: "reconnecting": false, 2021-02-10T11:38:17.801 INFO:teuthology.orchestra.run.gibba020.stdout: "recall_caps": { 2021-02-10T11:38:17.801 INFO:teuthology.orchestra.run.gibba020.stdout: "value": 0, 2021-02-10T11:38:17.801 INFO:teuthology.orchestra.run.gibba020.stdout: "halflife": 60 2021-02-10T11:38:17.801 INFO:teuthology.orchestra.run.gibba020.stdout: }, 2021-02-10T11:38:17.801 INFO:teuthology.orchestra.run.gibba020.stdout: "release_caps": { 2021-02-10T11:38:17.802 INFO:teuthology.orchestra.run.gibba020.stdout: "value": 0, 2021-02-10T11:38:17.802 INFO:teuthology.orchestra.run.gibba020.stdout: "halflife": 60 2021-02-10T11:38:17.802 INFO:teuthology.orchestra.run.gibba020.stdout: }, 2021-02-10T11:38:17.802 INFO:teuthology.orchestra.run.gibba020.stdout: "recall_caps_throttle": { 2021-02-10T11:38:17.802 INFO:teuthology.orchestra.run.gibba020.stdout: "value": 0, 2021-02-10T11:38:17.802 INFO:teuthology.orchestra.run.gibba020.stdout: "halflife": 2.5 2021-02-10T11:38:17.802 INFO:teuthology.orchestra.run.gibba020.stdout: }, 2021-02-10T11:38:17.802 INFO:teuthology.orchestra.run.gibba020.stdout: "recall_caps_throttle2o": { 2021-02-10T11:38:17.802 INFO:teuthology.orchestra.run.gibba020.stdout: "value": 0, 2021-02-10T11:38:17.802 INFO:teuthology.orchestra.run.gibba020.stdout: "halflife": 0.5 2021-02-10T11:38:17.802 INFO:teuthology.orchestra.run.gibba020.stdout: }, 2021-02-10T11:38:17.802 INFO:teuthology.orchestra.run.gibba020.stdout: "session_cache_liveness": { 2021-02-10T11:38:17.802 INFO:teuthology.orchestra.run.gibba020.stdout: "value": 18.457706689296391, 2021-02-10T11:38:17.803 INFO:teuthology.orchestra.run.gibba020.stdout: "halflife": 300 2021-02-10T11:38:17.803 INFO:teuthology.orchestra.run.gibba020.stdout: }, 2021-02-10T11:38:17.803 INFO:teuthology.orchestra.run.gibba020.stdout: "cap_acquisition": { 2021-02-10T11:38:17.803 INFO:teuthology.orchestra.run.gibba020.stdout: "value": 0, 2021-02-10T11:38:17.803 INFO:teuthology.orchestra.run.gibba020.stdout: "halflife": 10 2021-02-10T11:38:17.803 INFO:teuthology.orchestra.run.gibba020.stdout: }, 2021-02-10T11:38:17.803 INFO:teuthology.orchestra.run.gibba020.stdout: "inst": "client.7390 172.21.2.120:0/1069935987", 2021-02-10T11:38:17.803 INFO:teuthology.orchestra.run.gibba020.stdout: "completed_requests": [ 2021-02-10T11:38:17.803 INFO:teuthology.orchestra.run.gibba020.stdout: { 2021-02-10T11:38:17.803 INFO:teuthology.orchestra.run.gibba020.stdout: "tid": 14, 2021-02-10T11:38:17.803 INFO:teuthology.orchestra.run.gibba020.stdout: "created_ino": "0x10000000005" 2021-02-10T11:38:17.803 INFO:teuthology.orchestra.run.gibba020.stdout: } 2021-02-10T11:38:17.803 INFO:teuthology.orchestra.run.gibba020.stdout: ], 2021-02-10T11:38:17.804 INFO:teuthology.orchestra.run.gibba020.stdout: "prealloc_inos": [ 2021-02-10T11:38:17.804 INFO:teuthology.orchestra.run.gibba020.stdout: { 2021-02-10T11:38:17.804 INFO:teuthology.orchestra.run.gibba020.stdout: "start": 1099511627782, 2021-02-10T11:38:17.804 INFO:teuthology.orchestra.run.gibba020.stdout: "length": 995 2021-02-10T11:38:17.804 INFO:teuthology.orchestra.run.gibba020.stdout: } 2021-02-10T11:38:17.804 INFO:teuthology.orchestra.run.gibba020.stdout: ], 2021-02-10T11:38:17.804 INFO:teuthology.orchestra.run.gibba020.stdout: "used_inos": [], 2021-02-10T11:38:17.804 INFO:teuthology.orchestra.run.gibba020.stdout: "client_metadata": { 2021-02-10T11:38:17.804 INFO:teuthology.orchestra.run.gibba020.stdout: "features": "0x0000000000001fff", 2021-02-10T11:38:17.804 INFO:teuthology.orchestra.run.gibba020.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:38:17.804 INFO:teuthology.orchestra.run.gibba020.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:38:17.804 INFO:teuthology.orchestra.run.gibba020.stdout: "entity_id": "x", 2021-02-10T11:38:17.804 INFO:teuthology.orchestra.run.gibba020.stdout: "hostname": "gibba020", 2021-02-10T11:38:17.805 INFO:teuthology.orchestra.run.gibba020.stdout: "pid": "34990", 2021-02-10T11:38:17.805 INFO:teuthology.orchestra.run.gibba020.stdout: "root": "/" 2021-02-10T11:38:17.805 INFO:teuthology.orchestra.run.gibba020.stdout: } 2021-02-10T11:38:17.805 INFO:teuthology.orchestra.run.gibba020.stdout: } 2021-02-10T11:38:17.805 INFO:teuthology.orchestra.run.gibba020.stdout:] 2021-02-10T11:38:17.806 INFO:tasks.cephfs.filesystem:_json_asok output: [ { "id": 7390, "entity": { "name": { "type": "client", "num": 7390 }, "addr": { "type": "any", "addr": "172.21.2.120:0", "nonce": 1069935987 } }, "state": "open", "num_leases": 0, "num_caps": 7, "request_load_avg": 7, "uptime": 56.149199467999999, "requests_in_flight": 0, "completed_requests": 1, "reconnecting": false, "recall_caps": { "value": 0, "halflife": 60 }, "release_caps": { "value": 0, "halflife": 60 }, "recall_caps_throttle": { "value": 0, "halflife": 2.5 }, "recall_caps_throttle2o": { "value": 0, "halflife": 0.5 }, "session_cache_liveness": { "value": 18.457706689296391, "halflife": 300 }, "cap_acquisition": { "value": 0, "halflife": 10 }, "inst": "client.7390 172.21.2.120:0/1069935987", "completed_requests": [ { "tid": 14, "created_ino": "0x10000000005" } ], "prealloc_inos": [ { "start": 1099511627782, "length": 995 } ], "used_inos": [], "client_metadata": { "features": "0x0000000000001fff", "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", "entity_id": "x", "hostname": "gibba020", "pid": "34990", "root": "/" } } ] 2021-02-10T11:38:17.806 DEBUG:tasks.ceph_test_case:wait_until_true: waiting... 2021-02-10T11:38:20.178 DEBUG:teuthology.orchestra.run.gibba012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:38:20.182 DEBUG:teuthology.orchestra.run.gibba020:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:38:22.807 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:38:23.234 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:23.226 7f5e196fc700 1 Processor -- start 2021-02-10T11:38:23.234 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:23.226 7f5e196fc700 1 -- start start 2021-02-10T11:38:23.234 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:23.227 7f5e196fc700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f5e140d1870 0x7f5e141b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:23.234 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:23.227 7f5e196fc700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5e141b0a10 0x7f5e140cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:23.234 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:23.227 7f5e196fc700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e140f0b10 con 0x7f5e140d1c70 2021-02-10T11:38:23.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:23.227 7f5e196fc700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e140e5ed0 con 0x7f5e140d1870 2021-02-10T11:38:23.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:23.227 7f5e196fc700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e140f0650 con 0x7f5e141b0a10 2021-02-10T11:38:23.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:23.227 7f5e132bf700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f5e140d1870 0x7f5e141b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:23.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:23.227 7f5e13ac0700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f5e140d1c70 0x7f5e140cf950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:52424/0 (socket says 172.21.2.112:52424) 2021-02-10T11:38:23.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:23.227 7f5e13ac0700 1 -- 172.21.2.112:0/408009460 learned_addr learned my addr 172.21.2.112:0/408009460 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:38:23.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:23.227 7f5e0aabe700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5e141b0a10 0x7f5e140cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:23.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:23.228 7f5e12abe700 1 -- 172.21.2.112:0/408009460 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3429691631 0 0) 0x7f5e0c001500 con 0x7f5e140d1c70 2021-02-10T11:38:23.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:23.228 7f5e132bf700 1 -- 172.21.2.112:0/408009460 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5e141b0a10 msgr2=0x7f5e140cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:23.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:23.228 7f5e132bf700 1 --2- 172.21.2.112:0/408009460 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5e141b0a10 0x7f5e140cf330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:23.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:23.228 7f5e132bf700 1 -- 172.21.2.112:0/408009460 >> v1:172.21.2.112:6789/0 conn(0x7f5e140d1c70 legacy=0x7f5e140cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:23.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:23.228 7f5e132bf700 1 -- 172.21.2.112:0/408009460 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e140f0870 con 0x7f5e140d1870 2021-02-10T11:38:23.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:23.228 7f5e0aabe700 1 --2- 172.21.2.112:0/408009460 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5e141b0a10 0x7f5e140cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:38:23.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:23.228 7f5e132bf700 1 --2- 172.21.2.112:0/408009460 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5e140d1870 0x7f5e141b03f0 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 rx=0x7f5e04006650 tx=0x7f5e04009a80).ready entity=mon.0 client_cookie=4066d6a13d8fb632 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:23.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:23.228 7f5e12abe700 1 -- 172.21.2.112:0/408009460 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5e0400c0c0 con 0x7f5e140d1870 2021-02-10T11:38:23.236 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:23.228 7f5e12abe700 1 -- 172.21.2.112:0/408009460 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e0400c340 con 0x7f5e140d1870 2021-02-10T11:38:23.236 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:23.228 7f5e12abe700 1 -- 172.21.2.112:0/408009460 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5e0400c860 con 0x7f5e140d1870 2021-02-10T11:38:23.236 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:23.228 7f5e196fc700 1 -- 172.21.2.112:0/408009460 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5e140d1870 msgr2=0x7f5e141b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:23.236 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:23.228 7f5e196fc700 1 --2- 172.21.2.112:0/408009460 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5e140d1870 0x7f5e141b03f0 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 rx=0x7f5e04006650 tx=0x7f5e04009a80).stop 2021-02-10T11:38:23.236 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:23.229 7f5e196fc700 1 -- 172.21.2.112:0/408009460 shutdown_connections 2021-02-10T11:38:23.236 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:23.229 7f5e196fc700 1 --2- 172.21.2.112:0/408009460 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5e141b0a10 0x7f5e140cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:23.236 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:23.229 7f5e196fc700 1 --2- 172.21.2.112:0/408009460 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5e140d1870 0x7f5e141b03f0 unknown :-1 s=CLOSED pgs=1782 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:23.236 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:23.229 7f5e196fc700 1 -- 172.21.2.112:0/408009460 >> 172.21.2.112:0/408009460 conn(0x7f5e140ce450 msgr2=0x7f5e140d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:23.236 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:23.229 7f5e196fc700 1 -- 172.21.2.112:0/408009460 shutdown_connections 2021-02-10T11:38:23.236 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:23.229 7f5e196fc700 1 -- 172.21.2.112:0/408009460 wait complete. 2021-02-10T11:38:23.238 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:23.244 7f5e196fc700 1 Processor -- start 2021-02-10T11:38:23.238 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:23.244 7f5e196fc700 1 -- start start 2021-02-10T11:38:23.238 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:23.244 7f5e196fc700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5e14195650 0x7f5e140b5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:23.238 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:23.244 7f5e196fc700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5e140b5b80 0x7f5e141a2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:23.238 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:23.244 7f5e196fc700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5e140acfa0 0x7f5e141a3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:23.238 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:23.244 7f5e196fc700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e140b6010 con 0x7f5e140b5b80 2021-02-10T11:38:23.238 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:23.244 7f5e196fc700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e141a3a60 con 0x7f5e140acfa0 2021-02-10T11:38:23.239 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:23.244 7f5e196fc700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e141a3c40 con 0x7f5e14195650 2021-02-10T11:38:23.239 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:23.244 7f5e0aabe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5e140b5b80 0x7f5e141a2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:23.239 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:23.244 7f5e0aabe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5e140b5b80 0x7f5e141a2e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:52970/0 (socket says 172.21.2.112:52970) 2021-02-10T11:38:23.239 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:23.244 7f5e0aabe700 1 -- 172.21.2.112:0/4135770684 learned_addr learned my addr 172.21.2.112:0/4135770684 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:38:23.239 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:23.244 7f5e132bf700 1 --2- 172.21.2.112:0/4135770684 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5e14195650 0x7f5e140b5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:23.239 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:23.245 7f5e13ac0700 1 --2- 172.21.2.112:0/4135770684 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5e140acfa0 0x7f5e141a3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:23.239 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:23.245 7f5e0aabe700 1 -- 172.21.2.112:0/4135770684 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5e14195650 msgr2=0x7f5e140b5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:23.239 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:23.245 7f5e0aabe700 1 --2- 172.21.2.112:0/4135770684 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5e14195650 0x7f5e140b5560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:23.239 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:23.245 7f5e0aabe700 1 -- 172.21.2.112:0/4135770684 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5e140acfa0 msgr2=0x7f5e141a3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:23.239 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:23.245 7f5e0aabe700 1 --2- 172.21.2.112:0/4135770684 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5e140acfa0 0x7f5e141a3440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:23.239 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:23.245 7f5e0aabe700 1 -- 172.21.2.112:0/4135770684 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e140bcae0 con 0x7f5e140b5b80 2021-02-10T11:38:23.239 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:23.245 7f5e132bf700 1 --2- 172.21.2.112:0/4135770684 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5e14195650 0x7f5e140b5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:38:23.239 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:23.245 7f5e13ac0700 1 --2- 172.21.2.112:0/4135770684 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5e140acfa0 0x7f5e141a3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:38:23.239 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:23.245 7f5e0aabe700 1 --2- 172.21.2.112:0/4135770684 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5e140b5b80 0x7f5e141a2e20 secure :-1 s=READY pgs=1783 cs=0 l=1 rev1=1 rx=0x7f5dfc006f40 tx=0x7f5dfc00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:23.240 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:23.245 7f5e10aba700 1 -- 172.21.2.112:0/4135770684 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5dfc0009b0 con 0x7f5e140b5b80 2021-02-10T11:38:23.240 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:23.245 7f5e10aba700 1 -- 172.21.2.112:0/4135770684 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5dfc00bd80 con 0x7f5e140b5b80 2021-02-10T11:38:23.240 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:23.245 7f5e10aba700 1 -- 172.21.2.112:0/4135770684 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5dfc002220 con 0x7f5e140b5b80 2021-02-10T11:38:23.240 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:23.245 7f5e196fc700 1 -- 172.21.2.112:0/4135770684 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e140bcae0 con 0x7f5e140b5b80 2021-02-10T11:38:23.240 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:23.246 7f5e196fc700 1 -- 172.21.2.112:0/4135770684 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e140bd070 con 0x7f5e140b5b80 2021-02-10T11:38:23.240 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:23.246 7f5e10aba700 1 -- 172.21.2.112:0/4135770684 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5dfc006b00 con 0x7f5e140b5b80 2021-02-10T11:38:23.240 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:23.246 7f5e10aba700 1 --2- 172.21.2.112:0/4135770684 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5df80186c0 0x7f5df801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:23.240 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:23.246 7f5e10aba700 1 -- 172.21.2.112:0/4135770684 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f5dfc032560 con 0x7f5e140b5b80 2021-02-10T11:38:23.240 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:23.247 7f5e132bf700 1 --2- 172.21.2.112:0/4135770684 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5df80186c0 0x7f5df801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:23.240 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:23.247 7f5e132bf700 1 --2- 172.21.2.112:0/4135770684 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5df80186c0 0x7f5df801abc0 crc :-1 s=READY pgs=840 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:23.268 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:23.277 7f5e196fc700 1 -- 172.21.2.112:0/4135770684 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e140f0ab0 con 0x7f5e140b5b80 2021-02-10T11:38:23.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:23.279 7f5e10aba700 1 -- 172.21.2.112:0/4135770684 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5dfc000940 con 0x7f5e140b5b80 2021-02-10T11:38:23.398 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:23.408 7f5e196fc700 1 -- 172.21.2.112:0/4135770684 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5e140f0ab0 con 0x7f5e140b5b80 2021-02-10T11:38:23.399 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:23.408 7f5e10aba700 1 -- 172.21.2.112:0/4135770684 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) v1 ==== 94+0+2101 (secure 0 0 0) 0x7f5dfc006b00 con 0x7f5e140b5b80 2021-02-10T11:38:23.399 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:38:23.400 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":65,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9204,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/1512510827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":1512510827},{"type":"v1","addr":"172.21.2.112:6833","nonce":1512510827}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":60}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:36:53.563735","modified":"2021-02-10 11:36:55.573740","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9201},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9201":{"gid":9201,"name":"b","rank":0,"incarnation":61,"state":"up:active","state_seq":3,"addr":"172.21.2.120:6835/1232001750","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":1232001750},{"type":"v1","addr":"172.21.2.120:6835","nonce":1232001750}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2021-02-10T11:38:23.400 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 65 2021-02-10T11:38:23.401 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:23.410 7f5e196fc700 1 -- 172.21.2.112:0/4135770684 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5df80186c0 msgr2=0x7f5df801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:23.401 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:23.410 7f5e196fc700 1 --2- 172.21.2.112:0/4135770684 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5df80186c0 0x7f5df801abc0 crc :-1 s=READY pgs=840 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:23.401 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:23.410 7f5e196fc700 1 -- 172.21.2.112:0/4135770684 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5e140b5b80 msgr2=0x7f5e141a2e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:23.401 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:23.410 7f5e196fc700 1 --2- 172.21.2.112:0/4135770684 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5e140b5b80 0x7f5e141a2e20 secure :-1 s=READY pgs=1783 cs=0 l=1 rev1=1 rx=0x7f5dfc006f40 tx=0x7f5dfc00b990).stop 2021-02-10T11:38:23.401 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:23.410 7f5e196fc700 1 -- 172.21.2.112:0/4135770684 shutdown_connections 2021-02-10T11:38:23.401 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:23.410 7f5e196fc700 1 --2- 172.21.2.112:0/4135770684 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5df80186c0 0x7f5df801abc0 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:23.401 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:23.410 7f5e196fc700 1 --2- 172.21.2.112:0/4135770684 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5e140acfa0 0x7f5e141a3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:23.401 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:23.410 7f5e196fc700 1 --2- 172.21.2.112:0/4135770684 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5e140b5b80 0x7f5e141a2e20 unknown :-1 s=CLOSED pgs=1783 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:23.401 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:23.410 7f5e196fc700 1 --2- 172.21.2.112:0/4135770684 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5e14195650 0x7f5e140b5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:23.401 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:23.410 7f5e196fc700 1 -- 172.21.2.112:0/4135770684 >> 172.21.2.112:0/4135770684 conn(0x7f5e140b3040 msgr2=0x7f5e140c3250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:23.402 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:23.410 7f5e196fc700 1 -- 172.21.2.112:0/4135770684 shutdown_connections 2021-02-10T11:38:23.402 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:23.410 7f5e196fc700 1 -- 172.21.2.112:0/4135770684 wait complete. 2021-02-10T11:38:23.409 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok session ls 2021-02-10T11:38:23.806 INFO:teuthology.orchestra.run.gibba020.stdout:[ 2021-02-10T11:38:23.806 INFO:teuthology.orchestra.run.gibba020.stdout: { 2021-02-10T11:38:23.806 INFO:teuthology.orchestra.run.gibba020.stdout: "id": 7390, 2021-02-10T11:38:23.806 INFO:teuthology.orchestra.run.gibba020.stdout: "entity": { 2021-02-10T11:38:23.807 INFO:teuthology.orchestra.run.gibba020.stdout: "name": { 2021-02-10T11:38:23.807 INFO:teuthology.orchestra.run.gibba020.stdout: "type": "client", 2021-02-10T11:38:23.807 INFO:teuthology.orchestra.run.gibba020.stdout: "num": 7390 2021-02-10T11:38:23.807 INFO:teuthology.orchestra.run.gibba020.stdout: }, 2021-02-10T11:38:23.807 INFO:teuthology.orchestra.run.gibba020.stdout: "addr": { 2021-02-10T11:38:23.807 INFO:teuthology.orchestra.run.gibba020.stdout: "type": "any", 2021-02-10T11:38:23.807 INFO:teuthology.orchestra.run.gibba020.stdout: "addr": "172.21.2.120:0", 2021-02-10T11:38:23.807 INFO:teuthology.orchestra.run.gibba020.stdout: "nonce": 1069935987 2021-02-10T11:38:23.807 INFO:teuthology.orchestra.run.gibba020.stdout: } 2021-02-10T11:38:23.807 INFO:teuthology.orchestra.run.gibba020.stdout: }, 2021-02-10T11:38:23.807 INFO:teuthology.orchestra.run.gibba020.stdout: "state": "open", 2021-02-10T11:38:23.807 INFO:teuthology.orchestra.run.gibba020.stdout: "num_leases": 0, 2021-02-10T11:38:23.807 INFO:teuthology.orchestra.run.gibba020.stdout: "num_caps": 7, 2021-02-10T11:38:23.808 INFO:teuthology.orchestra.run.gibba020.stdout: "request_load_avg": 6, 2021-02-10T11:38:23.808 INFO:teuthology.orchestra.run.gibba020.stdout: "uptime": 62.154924856999997, 2021-02-10T11:38:23.808 INFO:teuthology.orchestra.run.gibba020.stdout: "requests_in_flight": 0, 2021-02-10T11:38:23.808 INFO:teuthology.orchestra.run.gibba020.stdout: "completed_requests": 1, 2021-02-10T11:38:23.808 INFO:teuthology.orchestra.run.gibba020.stdout: "reconnecting": false, 2021-02-10T11:38:23.808 INFO:teuthology.orchestra.run.gibba020.stdout: "recall_caps": { 2021-02-10T11:38:23.808 INFO:teuthology.orchestra.run.gibba020.stdout: "value": 0, 2021-02-10T11:38:23.808 INFO:teuthology.orchestra.run.gibba020.stdout: "halflife": 60 2021-02-10T11:38:23.808 INFO:teuthology.orchestra.run.gibba020.stdout: }, 2021-02-10T11:38:23.808 INFO:teuthology.orchestra.run.gibba020.stdout: "release_caps": { 2021-02-10T11:38:23.808 INFO:teuthology.orchestra.run.gibba020.stdout: "value": 0, 2021-02-10T11:38:23.808 INFO:teuthology.orchestra.run.gibba020.stdout: "halflife": 60 2021-02-10T11:38:23.808 INFO:teuthology.orchestra.run.gibba020.stdout: }, 2021-02-10T11:38:23.808 INFO:teuthology.orchestra.run.gibba020.stdout: "recall_caps_throttle": { 2021-02-10T11:38:23.809 INFO:teuthology.orchestra.run.gibba020.stdout: "value": 0, 2021-02-10T11:38:23.809 INFO:teuthology.orchestra.run.gibba020.stdout: "halflife": 2.5 2021-02-10T11:38:23.809 INFO:teuthology.orchestra.run.gibba020.stdout: }, 2021-02-10T11:38:23.809 INFO:teuthology.orchestra.run.gibba020.stdout: "recall_caps_throttle2o": { 2021-02-10T11:38:23.809 INFO:teuthology.orchestra.run.gibba020.stdout: "value": 0, 2021-02-10T11:38:23.809 INFO:teuthology.orchestra.run.gibba020.stdout: "halflife": 0.5 2021-02-10T11:38:23.809 INFO:teuthology.orchestra.run.gibba020.stdout: }, 2021-02-10T11:38:23.809 INFO:teuthology.orchestra.run.gibba020.stdout: "session_cache_liveness": { 2021-02-10T11:38:23.809 INFO:teuthology.orchestra.run.gibba020.stdout: "value": 18.203353183053707, 2021-02-10T11:38:23.809 INFO:teuthology.orchestra.run.gibba020.stdout: "halflife": 300 2021-02-10T11:38:23.809 INFO:teuthology.orchestra.run.gibba020.stdout: }, 2021-02-10T11:38:23.809 INFO:teuthology.orchestra.run.gibba020.stdout: "cap_acquisition": { 2021-02-10T11:38:23.809 INFO:teuthology.orchestra.run.gibba020.stdout: "value": 0, 2021-02-10T11:38:23.810 INFO:teuthology.orchestra.run.gibba020.stdout: "halflife": 10 2021-02-10T11:38:23.810 INFO:teuthology.orchestra.run.gibba020.stdout: }, 2021-02-10T11:38:23.810 INFO:teuthology.orchestra.run.gibba020.stdout: "inst": "client.7390 172.21.2.120:0/1069935987", 2021-02-10T11:38:23.810 INFO:teuthology.orchestra.run.gibba020.stdout: "completed_requests": [ 2021-02-10T11:38:23.810 INFO:teuthology.orchestra.run.gibba020.stdout: { 2021-02-10T11:38:23.810 INFO:teuthology.orchestra.run.gibba020.stdout: "tid": 14, 2021-02-10T11:38:23.810 INFO:teuthology.orchestra.run.gibba020.stdout: "created_ino": "0x10000000005" 2021-02-10T11:38:23.810 INFO:teuthology.orchestra.run.gibba020.stdout: } 2021-02-10T11:38:23.810 INFO:teuthology.orchestra.run.gibba020.stdout: ], 2021-02-10T11:38:23.810 INFO:teuthology.orchestra.run.gibba020.stdout: "prealloc_inos": [ 2021-02-10T11:38:23.810 INFO:teuthology.orchestra.run.gibba020.stdout: { 2021-02-10T11:38:23.810 INFO:teuthology.orchestra.run.gibba020.stdout: "start": 1099511627782, 2021-02-10T11:38:23.810 INFO:teuthology.orchestra.run.gibba020.stdout: "length": 995 2021-02-10T11:38:23.811 INFO:teuthology.orchestra.run.gibba020.stdout: } 2021-02-10T11:38:23.811 INFO:teuthology.orchestra.run.gibba020.stdout: ], 2021-02-10T11:38:23.811 INFO:teuthology.orchestra.run.gibba020.stdout: "used_inos": [], 2021-02-10T11:38:23.811 INFO:teuthology.orchestra.run.gibba020.stdout: "client_metadata": { 2021-02-10T11:38:23.811 INFO:teuthology.orchestra.run.gibba020.stdout: "features": "0x0000000000001fff", 2021-02-10T11:38:23.811 INFO:teuthology.orchestra.run.gibba020.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:38:23.811 INFO:teuthology.orchestra.run.gibba020.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:38:23.811 INFO:teuthology.orchestra.run.gibba020.stdout: "entity_id": "x", 2021-02-10T11:38:23.811 INFO:teuthology.orchestra.run.gibba020.stdout: "hostname": "gibba020", 2021-02-10T11:38:23.811 INFO:teuthology.orchestra.run.gibba020.stdout: "pid": "34990", 2021-02-10T11:38:23.811 INFO:teuthology.orchestra.run.gibba020.stdout: "root": "/" 2021-02-10T11:38:23.811 INFO:teuthology.orchestra.run.gibba020.stdout: } 2021-02-10T11:38:23.811 INFO:teuthology.orchestra.run.gibba020.stdout: } 2021-02-10T11:38:23.812 INFO:teuthology.orchestra.run.gibba020.stdout:] 2021-02-10T11:38:23.812 INFO:tasks.cephfs.filesystem:_json_asok output: [ { "id": 7390, "entity": { "name": { "type": "client", "num": 7390 }, "addr": { "type": "any", "addr": "172.21.2.120:0", "nonce": 1069935987 } }, "state": "open", "num_leases": 0, "num_caps": 7, "request_load_avg": 6, "uptime": 62.154924856999997, "requests_in_flight": 0, "completed_requests": 1, "reconnecting": false, "recall_caps": { "value": 0, "halflife": 60 }, "release_caps": { "value": 0, "halflife": 60 }, "recall_caps_throttle": { "value": 0, "halflife": 2.5 }, "recall_caps_throttle2o": { "value": 0, "halflife": 0.5 }, "session_cache_liveness": { "value": 18.203353183053707, "halflife": 300 }, "cap_acquisition": { "value": 0, "halflife": 10 }, "inst": "client.7390 172.21.2.120:0/1069935987", "completed_requests": [ { "tid": 14, "created_ino": "0x10000000005" } ], "prealloc_inos": [ { "start": 1099511627782, "length": 995 } ], "used_inos": [], "client_metadata": { "features": "0x0000000000001fff", "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", "entity_id": "x", "hostname": "gibba020", "pid": "34990", "root": "/" } } ] 2021-02-10T11:38:23.813 DEBUG:tasks.ceph_test_case:wait_until_true: waiting... 2021-02-10T11:38:28.814 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:38:29.239 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:29.232 7fb0069b3700 1 Processor -- start 2021-02-10T11:38:29.239 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:29.232 7fb0069b3700 1 -- start start 2021-02-10T11:38:29.239 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:29.232 7fb0069b3700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fb0001b0470 0x7fb0001b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:29.239 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:29.232 7fb0069b3700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fb0000f20e0 0x7fb0000f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:29.239 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:29.232 7fb0069b3700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0000f0b50 con 0x7fb0000ef610 2021-02-10T11:38:29.239 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:29.232 7fb0069b3700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0000f0820 con 0x7fb0001b0470 2021-02-10T11:38:29.240 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:29.232 7fb0069b3700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0000f24e0 con 0x7fb0000f20e0 2021-02-10T11:38:29.240 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:29.233 7faffffff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fb0001b0470 0x7fb0001b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:29.240 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:29.233 7faffffff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fb0001b0470 0x7fb0001b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:52974/0 (socket says 172.21.2.112:52974) 2021-02-10T11:38:29.240 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:29.233 7faffffff700 1 -- 172.21.2.112:0/3898004425 learned_addr learned my addr 172.21.2.112:0/3898004425 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:38:29.240 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:29.233 7faffffff700 1 -- 172.21.2.112:0/3898004425 >> v2:172.21.2.120:3300/0 conn(0x7fb0000f20e0 msgr2=0x7fb0000f0200 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:38:29.240 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:29.233 7faffffff700 1 --2- 172.21.2.112:0/3898004425 >> v2:172.21.2.120:3300/0 conn(0x7fb0000f20e0 0x7fb0000f0200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:29.240 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:29.233 7faffffff700 1 -- 172.21.2.112:0/3898004425 >> v1:172.21.2.120:6789/0 conn(0x7fb0000ef610 legacy=0x7fb0000efa10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:38:29.240 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:29.233 7faffffff700 1 -- 172.21.2.112:0/3898004425 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0000e52e0 con 0x7fb0001b0470 2021-02-10T11:38:29.240 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:29.233 7faffffff700 1 --2- 172.21.2.112:0/3898004425 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb0001b0470 0x7fb0001b0870 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 rx=0x7faff0006700 tx=0x7faff0009b80).ready entity=mon.0 client_cookie=36dd54d99e668ef2 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:29.240 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:29.233 7faffeffd700 1 -- 172.21.2.112:0/3898004425 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faff000c0c0 con 0x7fb0001b0470 2021-02-10T11:38:29.240 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:29.233 7faffeffd700 1 -- 172.21.2.112:0/3898004425 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faff000c340 con 0x7fb0001b0470 2021-02-10T11:38:29.240 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:29.233 7faffeffd700 1 -- 172.21.2.112:0/3898004425 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faff000c820 con 0x7fb0001b0470 2021-02-10T11:38:29.240 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:29.233 7fb0069b3700 1 -- 172.21.2.112:0/3898004425 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb0001b0470 msgr2=0x7fb0001b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:29.241 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:29.233 7fb0069b3700 1 --2- 172.21.2.112:0/3898004425 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb0001b0470 0x7fb0001b0870 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 rx=0x7faff0006700 tx=0x7faff0009b80).stop 2021-02-10T11:38:29.241 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:29.234 7fb0069b3700 1 -- 172.21.2.112:0/3898004425 shutdown_connections 2021-02-10T11:38:29.241 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:29.234 7fb0069b3700 1 --2- 172.21.2.112:0/3898004425 >> v2:172.21.2.120:3300/0 conn(0x7fb0000f20e0 0x7fb0000f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:29.241 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:29.234 7fb0069b3700 1 --2- 172.21.2.112:0/3898004425 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb0001b0470 0x7fb0001b0870 unknown :-1 s=CLOSED pgs=1784 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:29.241 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:29.234 7fb0069b3700 1 -- 172.21.2.112:0/3898004425 >> 172.21.2.112:0/3898004425 conn(0x7fb0000ce3e0 msgr2=0x7fb0000e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:29.241 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:29.234 7fb0069b3700 1 -- 172.21.2.112:0/3898004425 shutdown_connections 2021-02-10T11:38:29.241 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:29.234 7fb0069b3700 1 -- 172.21.2.112:0/3898004425 wait complete. 2021-02-10T11:38:29.242 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:29.248 7fb0069b3700 1 Processor -- start 2021-02-10T11:38:29.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:29.248 7fb0069b3700 1 -- start start 2021-02-10T11:38:29.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:29.249 7fb0069b3700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb0001aca20 0x7fb0000b49c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:29.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:29.249 7fb0069b3700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb0000b4fe0 0x7fb00019ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:29.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:29.249 7fb0069b3700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb00019c470 0x7fb00019d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:29.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:29.249 7fb0069b3700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0001a2d60 con 0x7fb00019c470 2021-02-10T11:38:29.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:29.249 7fb0069b3700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb00009e290 con 0x7fb0001aca20 2021-02-10T11:38:29.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:29.249 7fb0069b3700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0001a28f0 con 0x7fb0000b4fe0 2021-02-10T11:38:29.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:29.249 7fb004fd0700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb00019c470 0x7fb00019d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:29.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:29.249 7fb004fd0700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb00019c470 0x7fb00019d4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:52980/0 (socket says 172.21.2.112:52980) 2021-02-10T11:38:29.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:29.249 7fb004fd0700 1 -- 172.21.2.112:0/2487068876 learned_addr learned my addr 172.21.2.112:0/2487068876 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:38:29.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:29.249 7fb004fd0700 1 -- 172.21.2.112:0/2487068876 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb0000b4fe0 msgr2=0x7fb00019ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:29.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:29.249 7fb004fd0700 1 --2- 172.21.2.112:0/2487068876 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb0000b4fe0 0x7fb00019ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:29.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:29.249 7fb004fd0700 1 -- 172.21.2.112:0/2487068876 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb0001aca20 msgr2=0x7fb0000b49c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:29.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:29.249 7fb004fd0700 1 --2- 172.21.2.112:0/2487068876 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb0001aca20 0x7fb0000b49c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:29.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:29.249 7fb004fd0700 1 -- 172.21.2.112:0/2487068876 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0001ace20 con 0x7fb00019c470 2021-02-10T11:38:29.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:29.249 7fb004fd0700 1 --2- 172.21.2.112:0/2487068876 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb00019c470 0x7fb00019d4b0 secure :-1 s=READY pgs=1785 cs=0 l=1 rev1=1 rx=0x7faff8009960 tx=0x7faff800bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:29.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:29.250 7faffcff9700 1 -- 172.21.2.112:0/2487068876 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faff8003030 con 0x7fb00019c470 2021-02-10T11:38:29.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:29.250 7faffcff9700 1 -- 172.21.2.112:0/2487068876 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faff8001310 con 0x7fb00019c470 2021-02-10T11:38:29.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:29.250 7faffcff9700 1 -- 172.21.2.112:0/2487068876 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faff8001830 con 0x7fb00019c470 2021-02-10T11:38:29.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:29.250 7fb0069b3700 1 -- 172.21.2.112:0/2487068876 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0001ace20 con 0x7fb00019c470 2021-02-10T11:38:29.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:29.250 7fb0069b3700 1 -- 172.21.2.112:0/2487068876 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb00019ddc0 con 0x7fb00019c470 2021-02-10T11:38:29.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:29.250 7faffcff9700 1 -- 172.21.2.112:0/2487068876 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7faff80014f0 con 0x7fb00019c470 2021-02-10T11:38:29.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:29.250 7faffcff9700 1 --2- 172.21.2.112:0/2487068876 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fafe00186c0 0x7fafe001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:29.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:29.250 7faffcff9700 1 -- 172.21.2.112:0/2487068876 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5573+0+0 (secure 0 0 0) 0x7faff8032ba0 con 0x7fb00019c470 2021-02-10T11:38:29.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:29.251 7faffffff700 1 --2- 172.21.2.112:0/2487068876 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fafe00186c0 0x7fafe001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:29.245 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:29.251 7faffffff700 1 --2- 172.21.2.112:0/2487068876 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fafe00186c0 0x7fafe001abc0 crc :-1 s=READY pgs=841 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:29.274 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:29.283 7fb0069b3700 1 -- 172.21.2.112:0/2487068876 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb00019e390 con 0x7fb00019c470 2021-02-10T11:38:29.276 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:29.285 7faffcff9700 1 -- 172.21.2.112:0/2487068876 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7faff8000f40 con 0x7fb00019c470 2021-02-10T11:38:29.407 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:29.416 7fb0069b3700 1 -- 172.21.2.112:0/2487068876 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb00019e390 con 0x7fb00019c470 2021-02-10T11:38:29.407 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:29.416 7faffcff9700 1 -- 172.21.2.112:0/2487068876 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) v1 ==== 94+0+2101 (secure 0 0 0) 0x7faff8001310 con 0x7fb00019c470 2021-02-10T11:38:29.408 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:38:29.408 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":65,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9204,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/1512510827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":1512510827},{"type":"v1","addr":"172.21.2.112:6833","nonce":1512510827}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":60}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:36:53.563735","modified":"2021-02-10 11:36:55.573740","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9201},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9201":{"gid":9201,"name":"b","rank":0,"incarnation":61,"state":"up:active","state_seq":3,"addr":"172.21.2.120:6835/1232001750","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":1232001750},{"type":"v1","addr":"172.21.2.120:6835","nonce":1232001750}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2021-02-10T11:38:29.408 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 65 2021-02-10T11:38:29.409 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:29.418 7fb0069b3700 1 -- 172.21.2.112:0/2487068876 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fafe00186c0 msgr2=0x7fafe001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:29.409 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:29.418 7fb0069b3700 1 --2- 172.21.2.112:0/2487068876 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fafe00186c0 0x7fafe001abc0 crc :-1 s=READY pgs=841 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:29.409 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:29.418 7fb0069b3700 1 -- 172.21.2.112:0/2487068876 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb00019c470 msgr2=0x7fb00019d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:29.410 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:29.418 7fb0069b3700 1 --2- 172.21.2.112:0/2487068876 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb00019c470 0x7fb00019d4b0 secure :-1 s=READY pgs=1785 cs=0 l=1 rev1=1 rx=0x7faff8009960 tx=0x7faff800bcf0).stop 2021-02-10T11:38:29.410 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:29.418 7fb0069b3700 1 -- 172.21.2.112:0/2487068876 shutdown_connections 2021-02-10T11:38:29.410 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:29.418 7fb0069b3700 1 --2- 172.21.2.112:0/2487068876 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fafe00186c0 0x7fafe001abc0 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:29.410 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:29.418 7fb0069b3700 1 --2- 172.21.2.112:0/2487068876 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb00019c470 0x7fb00019d4b0 unknown :-1 s=CLOSED pgs=1785 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:29.410 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:29.418 7fb0069b3700 1 --2- 172.21.2.112:0/2487068876 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb0000b4fe0 0x7fb00019ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:29.410 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:29.418 7fb0069b3700 1 --2- 172.21.2.112:0/2487068876 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb0001aca20 0x7fb0000b49c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:29.410 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:29.418 7fb0069b3700 1 -- 172.21.2.112:0/2487068876 >> 172.21.2.112:0/2487068876 conn(0x7fb00009e730 msgr2=0x7fb00019eef0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:29.410 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:29.418 7fb0069b3700 1 -- 172.21.2.112:0/2487068876 shutdown_connections 2021-02-10T11:38:29.410 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:29.418 7fb0069b3700 1 -- 172.21.2.112:0/2487068876 wait complete. 2021-02-10T11:38:29.418 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok session ls 2021-02-10T11:38:29.812 INFO:teuthology.orchestra.run.gibba020.stdout:[] 2021-02-10T11:38:29.816 INFO:tasks.cephfs.filesystem:_json_asok output: [] 2021-02-10T11:38:29.817 DEBUG:tasks.ceph_test_case:wait_until_true: success in 55s 2021-02-10T11:38:29.817 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:38:29.817 DEBUG:teuthology.orchestra.run.gibba012:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:38:30.162 DEBUG:teuthology.orchestra.run.gibba020:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:38:30.506 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:38:30.506 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:38:30.506 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:38:30.506 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:38:30.843 INFO:teuthology.orchestra.run.gibba012.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.0’: No such file or directory 2021-02-10T11:38:30.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:38:30.843 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2021-02-10T11:38:30.843 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:38:30.844 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:38:31.178 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:38:31.178 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:38:31.179 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:38:31.513 INFO:teuthology.orchestra.run.gibba012.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: No such file or directory 2021-02-10T11:38:31.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:38:31.514 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2021-02-10T11:38:31.514 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:38:31.514 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:38:31.848 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:38:31.849 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:38:31.849 DEBUG:teuthology.orchestra.run.gibba020:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:38:32.183 INFO:teuthology.orchestra.run.gibba020.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:38:32.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:38:32.184 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:38:32.184 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:38:32.184 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:38:32.518 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:38:32.518 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:38:32.519 DEBUG:teuthology.orchestra.run.gibba020:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:38:32.854 INFO:teuthology.orchestra.run.gibba020.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:38:32.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:38:32.855 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:38:32.855 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:38:32.855 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:38:33.192 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config rm mon mon_allow_pool_delete 2021-02-10T11:38:33.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:33.618 7fcaba356700 1 Processor -- start 2021-02-10T11:38:33.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:33.618 7fcaba356700 1 -- start start 2021-02-10T11:38:33.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:33.618 7fcaba356700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcab40d1870 0x7fcab41b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:33.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:33.618 7fcaba356700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fcab41b0a10 0x7fcab40cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:33.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:33.618 7fcaba356700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcab40f0b10 con 0x7fcab40d1c70 2021-02-10T11:38:33.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:33.618 7fcaba356700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcab40fe230 con 0x7fcab41b0a10 2021-02-10T11:38:33.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:33.618 7fcaba356700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcab40e5ed0 con 0x7fcab40d1870 2021-02-10T11:38:33.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:33.619 7fcab8973700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fcab40d1c70 0x7fcab40cf950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:52448/0 (socket says 172.21.2.112:52448) 2021-02-10T11:38:33.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:33.619 7fcab8973700 1 -- 172.21.2.112:0/510462488 learned_addr learned my addr 172.21.2.112:0/510462488 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:38:33.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:33.619 7fcab2ffd700 1 -- 172.21.2.112:0/510462488 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 459673266 0 0) 0x7fcaac001500 con 0x7fcab40d1c70 2021-02-10T11:38:33.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:33.619 7fcab2ffd700 1 -- 172.21.2.112:0/510462488 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca9c002fe0 con 0x7fcab40d1c70 2021-02-10T11:38:33.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:33.619 7fcab2ffd700 1 -- 172.21.2.112:0/510462488 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3948030345 0 0) 0x7fcaac001460 con 0x7fcab40d1c70 2021-02-10T11:38:33.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:33.619 7fcab2ffd700 1 -- 172.21.2.112:0/510462488 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcab40d1870 msgr2=0x7fcab41b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:33.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:33.619 7fcab2ffd700 1 --2- 172.21.2.112:0/510462488 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcab40d1870 0x7fcab41b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:33.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:33.619 7fcab2ffd700 1 -- 172.21.2.112:0/510462488 >> v2:172.21.2.120:3300/0 conn(0x7fcab41b0a10 msgr2=0x7fcab40cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:33.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:33.619 7fcab2ffd700 1 --2- 172.21.2.112:0/510462488 >> v2:172.21.2.120:3300/0 conn(0x7fcab41b0a10 0x7fcab40cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:33.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:33.619 7fcab2ffd700 1 -- 172.21.2.112:0/510462488 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcab40f0650 con 0x7fcab40d1c70 2021-02-10T11:38:33.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:33.619 7fcab2ffd700 1 -- 172.21.2.112:0/510462488 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7fcaac001980 con 0x7fcab40d1c70 2021-02-10T11:38:33.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:33.619 7fcab2ffd700 1 -- 172.21.2.112:0/510462488 >> v1:172.21.2.112:6789/0 conn(0x7fcab40d1c70 legacy=0x7fcab40cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:33.629 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:33.619 7fcab2ffd700 1 --2- 172.21.2.112:0/510462488 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fca9c003af0 0x7fca9c005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:33.629 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:33.619 7fcab2ffd700 1 --2- 172.21.2.112:0/510462488 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fca9c0067b0 0x7fca9c008c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:33.629 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:33.619 7fcab2ffd700 1 --2- 172.21.2.112:0/510462488 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fca9c009520 0x7fca9c00ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:33.629 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:33.619 7fcab2ffd700 1 -- 172.21.2.112:0/510462488 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca9c00c2d0 con 0x7fca9c003af0 2021-02-10T11:38:33.629 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:33.619 7fcab2ffd700 1 -- 172.21.2.112:0/510462488 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca9c00c4f0 con 0x7fca9c009520 2021-02-10T11:38:33.629 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:33.619 7fcab2ffd700 1 -- 172.21.2.112:0/510462488 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca9c00c710 con 0x7fca9c0067b0 2021-02-10T11:38:33.629 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:33.619 7fcab3fff700 1 --2- 172.21.2.112:0/510462488 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fca9c003af0 0x7fca9c005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:33.629 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:33.620 7fcab37fe700 1 --2- 172.21.2.112:0/510462488 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fca9c0067b0 0x7fca9c008c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:33.629 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:33.620 7fcab8973700 1 --2- 172.21.2.112:0/510462488 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fca9c009520 0x7fca9c00ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:33.629 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:33.620 7fcab3fff700 1 -- 172.21.2.112:0/510462488 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fca9c0067b0 msgr2=0x7fca9c008c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:33.629 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:33.620 7fcab3fff700 1 --2- 172.21.2.112:0/510462488 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fca9c0067b0 0x7fca9c008c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:33.629 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:33.620 7fcab3fff700 1 -- 172.21.2.112:0/510462488 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fca9c009520 msgr2=0x7fca9c00ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:33.629 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:33.620 7fcab3fff700 1 --2- 172.21.2.112:0/510462488 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fca9c009520 0x7fca9c00ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:33.630 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:33.620 7fcab3fff700 1 -- 172.21.2.112:0/510462488 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca9c00c9b0 con 0x7fca9c003af0 2021-02-10T11:38:33.630 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:33.620 7fcab3fff700 1 --2- 172.21.2.112:0/510462488 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fca9c003af0 0x7fca9c005fa0 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 rx=0x7fcaa404d720 tx=0x7fcaa404dac0).ready entity=mon.0 client_cookie=b43459f6612deaca server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:33.630 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:33.620 7fcab2ffd700 1 -- 172.21.2.112:0/510462488 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcaa40066d0 con 0x7fca9c003af0 2021-02-10T11:38:33.630 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:33.620 7fcab2ffd700 1 -- 172.21.2.112:0/510462488 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcaa4002030 con 0x7fca9c003af0 2021-02-10T11:38:33.630 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:33.620 7fcab2ffd700 1 -- 172.21.2.112:0/510462488 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcaa4002500 con 0x7fca9c003af0 2021-02-10T11:38:33.630 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:33.620 7fcaba356700 1 -- 172.21.2.112:0/510462488 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fca9c003af0 msgr2=0x7fca9c005fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:33.630 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:33.620 7fcaba356700 1 --2- 172.21.2.112:0/510462488 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fca9c003af0 0x7fca9c005fa0 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 rx=0x7fcaa404d720 tx=0x7fcaa404dac0).stop 2021-02-10T11:38:33.630 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:33.620 7fcab8973700 1 -- 172.21.2.112:0/510462488 reap_dead start 2021-02-10T11:38:33.630 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:33.621 7fcaba356700 1 -- 172.21.2.112:0/510462488 shutdown_connections 2021-02-10T11:38:33.630 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:33.621 7fcaba356700 1 -- 172.21.2.112:0/510462488 >> 172.21.2.112:0/510462488 conn(0x7fcab40ce450 msgr2=0x7fcab40d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:33.630 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:33.621 7fcaba356700 1 -- 172.21.2.112:0/510462488 shutdown_connections 2021-02-10T11:38:33.630 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:33.621 7fcaba356700 1 -- 172.21.2.112:0/510462488 wait complete. 2021-02-10T11:38:33.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:33.636 7fcaba356700 1 Processor -- start 2021-02-10T11:38:33.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:33.636 7fcaba356700 1 -- start start 2021-02-10T11:38:33.632 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:33.636 7fcaba356700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcab40bdcd0 0x7fcab40a6d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:33.632 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:33.636 7fcaba356700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcab40a7390 0x7fcab41a7270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:33.632 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:33.636 7fcaba356700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcab41a7890 0x7fcab41a7c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:33.632 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:33.636 7fcaba356700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcab409b8d0 con 0x7fcab40bdcd0 2021-02-10T11:38:33.632 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:33.636 7fcaba356700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcab409b420 con 0x7fcab40a7390 2021-02-10T11:38:33.632 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:33.636 7fcaba356700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcab409b640 con 0x7fcab41a7890 2021-02-10T11:38:33.632 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:33.636 7fcab3fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcab40bdcd0 0x7fcab40a6d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:33.632 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:33.636 7fcab3fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcab40bdcd0 0x7fcab40a6d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:52996/0 (socket says 172.21.2.112:52996) 2021-02-10T11:38:33.632 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:33.636 7fcab3fff700 1 -- 172.21.2.112:0/4060567520 learned_addr learned my addr 172.21.2.112:0/4060567520 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:38:33.633 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:33.636 7fcab3fff700 1 -- 172.21.2.112:0/4060567520 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcab41a7890 msgr2=0x7fcab41a7c90 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:38:33.633 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:33.636 7fcab3fff700 1 --2- 172.21.2.112:0/4060567520 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcab41a7890 0x7fcab41a7c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:33.633 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:33.636 7fcab3fff700 1 -- 172.21.2.112:0/4060567520 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcab40a7390 msgr2=0x7fcab41a7270 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:38:33.633 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:33.636 7fcab3fff700 1 --2- 172.21.2.112:0/4060567520 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcab40a7390 0x7fcab41a7270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:33.633 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:33.636 7fcab3fff700 1 -- 172.21.2.112:0/4060567520 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcab40c21d0 con 0x7fcab40bdcd0 2021-02-10T11:38:33.633 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:33.637 7fcab3fff700 1 --2- 172.21.2.112:0/4060567520 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcab40bdcd0 0x7fcab40a6d70 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 rx=0x7fcaa4006f80 tx=0x7fcaa4050970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:33.633 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:33.637 7fcab0ff9700 1 -- 172.21.2.112:0/4060567520 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcaa404d080 con 0x7fcab40bdcd0 2021-02-10T11:38:33.633 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:33.637 7fcab0ff9700 1 -- 172.21.2.112:0/4060567520 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcaa4050d60 con 0x7fcab40bdcd0 2021-02-10T11:38:33.633 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:33.637 7fcab0ff9700 1 -- 172.21.2.112:0/4060567520 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcaa4002350 con 0x7fcab40bdcd0 2021-02-10T11:38:33.633 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:33.637 7fcaba356700 1 -- 172.21.2.112:0/4060567520 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcab40c21d0 con 0x7fcab40bdcd0 2021-02-10T11:38:33.633 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:33.637 7fcaba356700 1 -- 172.21.2.112:0/4060567520 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcab40c21d0 con 0x7fcab40bdcd0 2021-02-10T11:38:33.633 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:33.638 7fcab0ff9700 1 -- 172.21.2.112:0/4060567520 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fcaa4005310 con 0x7fcab40bdcd0 2021-02-10T11:38:33.633 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:33.638 7fcab0ff9700 1 --2- 172.21.2.112:0/4060567520 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fca940186c0 0x7fca9401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:33.634 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:33.638 7fcab0ff9700 1 -- 172.21.2.112:0/4060567520 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fcaa40765c0 con 0x7fcab40bdcd0 2021-02-10T11:38:33.634 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:33.639 7fcab37fe700 1 --2- 172.21.2.112:0/4060567520 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fca940186c0 0x7fca9401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:33.634 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:33.639 7fcab37fe700 1 --2- 172.21.2.112:0/4060567520 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fca940186c0 0x7fca9401abc0 crc :-1 s=READY pgs=842 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:33.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:33.668 7fcaba356700 1 -- 172.21.2.112:0/4060567520 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcab40ebcb0 con 0x7fcab40bdcd0 2021-02-10T11:38:33.662 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:33.671 7fcab0ff9700 1 -- 172.21.2.112:0/4060567520 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fcaa4074d70 con 0x7fcab40bdcd0 2021-02-10T11:38:33.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:33.819 7fcaba356700 1 -- 172.21.2.112:0/4060567520 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fcab40f0b10 con 0x7fcab40bdcd0 2021-02-10T11:38:33.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:33.823 7fcab0ff9700 1 -- 172.21.2.112:0/4060567520 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v23) v1 ==== 104+0+0 (secure 0 0 0) 0x7fcaa4005310 con 0x7fcab40bdcd0 2021-02-10T11:38:33.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:33.829 7fcaba356700 1 -- 172.21.2.112:0/4060567520 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fca940186c0 msgr2=0x7fca9401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:33.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:33.829 7fcaba356700 1 --2- 172.21.2.112:0/4060567520 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fca940186c0 0x7fca9401abc0 crc :-1 s=READY pgs=842 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:33.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:33.829 7fcaba356700 1 -- 172.21.2.112:0/4060567520 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcab40bdcd0 msgr2=0x7fcab40a6d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:33.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:33.829 7fcaba356700 1 --2- 172.21.2.112:0/4060567520 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcab40bdcd0 0x7fcab40a6d70 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 rx=0x7fcaa4006f80 tx=0x7fcaa4050970).stop 2021-02-10T11:38:33.821 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:33.829 7fcaba356700 1 -- 172.21.2.112:0/4060567520 shutdown_connections 2021-02-10T11:38:33.821 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:33.829 7fcaba356700 1 --2- 172.21.2.112:0/4060567520 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fca940186c0 0x7fca9401abc0 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:33.821 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:33.829 7fcaba356700 1 --2- 172.21.2.112:0/4060567520 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcab40a7390 0x7fcab41a7270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:33.821 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:33.829 7fcaba356700 1 --2- 172.21.2.112:0/4060567520 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcab41a7890 0x7fcab41a7c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:33.821 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:33.829 7fcaba356700 1 --2- 172.21.2.112:0/4060567520 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcab40bdcd0 0x7fcab40a6d70 unknown :-1 s=CLOSED pgs=1788 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:33.821 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:33.829 7fcaba356700 1 -- 172.21.2.112:0/4060567520 >> 172.21.2.112:0/4060567520 conn(0x7fcab40aed10 msgr2=0x7fcab40b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:33.821 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:33.829 7fcaba356700 1 -- 172.21.2.112:0/4060567520 shutdown_connections 2021-02-10T11:38:33.821 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:33.829 7fcaba356700 1 -- 172.21.2.112:0/4060567520 wait complete. 2021-02-10T11:38:33.828 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestVolumes.test_connection_expiration' 2021-02-10T11:38:34.255 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:34.247 7f33714cc700 1 Processor -- start 2021-02-10T11:38:34.255 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:34.247 7f33714cc700 1 -- start start 2021-02-10T11:38:34.255 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:34.248 7f33714cc700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f336c0d1870 0x7f336c1b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:34.255 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:34.248 7f33714cc700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f336c1b0a10 0x7f336c0cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:34.255 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:34.248 7f33714cc700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f336c0f0b10 con 0x7f336c0d1c70 2021-02-10T11:38:34.255 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:34.248 7f33714cc700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f336c0e5ed0 con 0x7f336c0d1870 2021-02-10T11:38:34.255 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:34.248 7f33714cc700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f336c0f0650 con 0x7f336c1b0a10 2021-02-10T11:38:34.255 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:34.248 7f336b2bf700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f336c0d1870 0x7f336c1b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:34.255 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:34.248 7f336b2bf700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f336c0d1870 0x7f336c1b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:53006/0 (socket says 172.21.2.112:53006) 2021-02-10T11:38:34.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:34.248 7f336b2bf700 1 -- 172.21.2.112:0/3164386192 learned_addr learned my addr 172.21.2.112:0/3164386192 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:38:34.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:34.248 7f336bac0700 1 --1- 172.21.2.112:0/3164386192 >> v1:172.21.2.112:6789/0 conn(0x7f336c0d1c70 0x7f336c0cf950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:52466/0 (socket says 172.21.2.112:52466) 2021-02-10T11:38:34.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:34.249 7f336b2bf700 1 -- 172.21.2.112:0/3164386192 >> v2:172.21.2.120:3300/0 conn(0x7f336c1b0a10 msgr2=0x7f336c0cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:34.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:34.249 7f336b2bf700 1 --2- 172.21.2.112:0/3164386192 >> v2:172.21.2.120:3300/0 conn(0x7f336c1b0a10 0x7f336c0cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:34.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:34.249 7f336b2bf700 1 -- 172.21.2.112:0/3164386192 >> v1:172.21.2.112:6789/0 conn(0x7f336c0d1c70 legacy=0x7f336c0cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:34.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:34.249 7f336a2bd700 1 -- 172.21.2.112:0/3164386192 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2920276480 0 0) 0x7f3364001480 con 0x7f336c0d1c70 2021-02-10T11:38:34.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:34.249 7f336b2bf700 1 -- 172.21.2.112:0/3164386192 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f336c0f0840 con 0x7f336c0d1870 2021-02-10T11:38:34.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:34.249 7f336b2bf700 1 --2- 172.21.2.112:0/3164386192 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f336c0d1870 0x7f336c1b03f0 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 rx=0x7f335c006700 tx=0x7f335c009b00).ready entity=mon.0 client_cookie=c7b7a67e354bf7c5 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:34.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:34.249 7f336a2bd700 1 -- 172.21.2.112:0/3164386192 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f335c00c0c0 con 0x7f336c0d1870 2021-02-10T11:38:34.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:34.249 7f336a2bd700 1 -- 172.21.2.112:0/3164386192 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f335c00c400 con 0x7f336c0d1870 2021-02-10T11:38:34.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:34.249 7f336a2bd700 1 -- 172.21.2.112:0/3164386192 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f335c00c920 con 0x7f336c0d1870 2021-02-10T11:38:34.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:34.249 7f33714cc700 1 -- 172.21.2.112:0/3164386192 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f336c0d1870 msgr2=0x7f336c1b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:34.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:34.249 7f33714cc700 1 --2- 172.21.2.112:0/3164386192 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f336c0d1870 0x7f336c1b03f0 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 rx=0x7f335c006700 tx=0x7f335c009b00).stop 2021-02-10T11:38:34.257 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:34.250 7f33714cc700 1 -- 172.21.2.112:0/3164386192 shutdown_connections 2021-02-10T11:38:34.257 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:34.250 7f33714cc700 1 --2- 172.21.2.112:0/3164386192 >> v2:172.21.2.120:3300/0 conn(0x7f336c1b0a10 0x7f336c0cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:34.257 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:34.250 7f33714cc700 1 --2- 172.21.2.112:0/3164386192 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f336c0d1870 0x7f336c1b03f0 unknown :-1 s=CLOSED pgs=1790 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:34.257 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:34.250 7f33714cc700 1 -- 172.21.2.112:0/3164386192 >> 172.21.2.112:0/3164386192 conn(0x7f336c0ce450 msgr2=0x7f336c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:34.257 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:34.250 7f33714cc700 1 -- 172.21.2.112:0/3164386192 shutdown_connections 2021-02-10T11:38:34.257 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:34.250 7f33714cc700 1 -- 172.21.2.112:0/3164386192 wait complete. 2021-02-10T11:38:34.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:34.264 7f33714cc700 1 Processor -- start 2021-02-10T11:38:34.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:34.264 7f33714cc700 1 -- start start 2021-02-10T11:38:34.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:34.265 7f33714cc700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f336c1b7cf0 0x7f336c19cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:34.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:34.265 7f33714cc700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f336c1b80f0 0x7f336c1b84f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:34.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:34.265 7f33714cc700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f336c19da20 0x7f336c19de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:34.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:34.265 7f33714cc700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f336c0a77c0 con 0x7f336c19da20 2021-02-10T11:38:34.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:34.265 7f33714cc700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f336c19d4a0 con 0x7f336c1b80f0 2021-02-10T11:38:34.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:34.265 7f33714cc700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f336c0954f0 con 0x7f336c1b7cf0 2021-02-10T11:38:34.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:34.265 7f336bac0700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f336c19da20 0x7f336c19de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:34.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:34.265 7f336bac0700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f336c19da20 0x7f336c19de20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:53010/0 (socket says 172.21.2.112:53010) 2021-02-10T11:38:34.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:34.265 7f336bac0700 1 -- 172.21.2.112:0/3387010709 learned_addr learned my addr 172.21.2.112:0/3387010709 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:38:34.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:34.265 7f336b2bf700 1 --2- 172.21.2.112:0/3387010709 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f336c1b7cf0 0x7f336c19cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:34.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:34.265 7f336aabe700 1 --2- 172.21.2.112:0/3387010709 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f336c1b80f0 0x7f336c1b84f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:34.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:34.265 7f336bac0700 1 -- 172.21.2.112:0/3387010709 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f336c1b7cf0 msgr2=0x7f336c19cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:34.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:34.265 7f336bac0700 1 --2- 172.21.2.112:0/3387010709 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f336c1b7cf0 0x7f336c19cdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:34.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:34.265 7f336bac0700 1 -- 172.21.2.112:0/3387010709 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f336c1b80f0 msgr2=0x7f336c1b84f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:34.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:34.265 7f336bac0700 1 --2- 172.21.2.112:0/3387010709 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f336c1b80f0 0x7f336c1b84f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:34.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:34.265 7f336bac0700 1 -- 172.21.2.112:0/3387010709 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f336c1b8d60 con 0x7f336c19da20 2021-02-10T11:38:34.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:34.266 7f336bac0700 1 --2- 172.21.2.112:0/3387010709 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f336c19da20 0x7f336c19de20 secure :-1 s=READY pgs=1791 cs=0 l=1 rev1=1 rx=0x7f336400a870 tx=0x7f336400ac10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:34.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:34.266 7f3353fff700 1 -- 172.21.2.112:0/3387010709 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3364007d50 con 0x7f336c19da20 2021-02-10T11:38:34.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:34.266 7f3353fff700 1 -- 172.21.2.112:0/3387010709 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3364007280 con 0x7f336c19da20 2021-02-10T11:38:34.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:34.266 7f3353fff700 1 -- 172.21.2.112:0/3387010709 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3364004720 con 0x7f336c19da20 2021-02-10T11:38:34.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:34.266 7f33714cc700 1 -- 172.21.2.112:0/3387010709 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f336c1b8d60 con 0x7f336c19da20 2021-02-10T11:38:34.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:34.266 7f33714cc700 1 -- 172.21.2.112:0/3387010709 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f336c1b8d60 con 0x7f336c19da20 2021-02-10T11:38:34.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:34.267 7f3353fff700 1 -- 172.21.2.112:0/3387010709 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3364007460 con 0x7f336c19da20 2021-02-10T11:38:34.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:34.267 7f3353fff700 1 --2- 172.21.2.112:0/3387010709 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f334c0186c0 0x7f334c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:34.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:34.267 7f3353fff700 1 -- 172.21.2.112:0/3387010709 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f3364032790 con 0x7f336c19da20 2021-02-10T11:38:34.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:34.267 7f336b2bf700 1 --2- 172.21.2.112:0/3387010709 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f334c0186c0 0x7f334c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:34.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:34.267 7f336b2bf700 1 --2- 172.21.2.112:0/3387010709 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f334c0186c0 0x7f334c01abc0 crc :-1 s=READY pgs=843 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:34.289 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:34.298 7f33714cc700 1 -- 172.21.2.112:0/3387010709 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f336c0ebc00 con 0x7f336c19da20 2021-02-10T11:38:34.292 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:34.300 7f3353fff700 1 -- 172.21.2.112:0/3387010709 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3364007070 con 0x7f336c19da20 2021-02-10T11:38:34.426 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:34.435 7f33714cc700 1 -- 172.21.2.112:0/3387010709 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_connection_expiration"]} v 0) v1 -- 0x7f336c0f0b10 con 0x7f336c19da20 2021-02-10T11:38:34.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:34.880 7f3353fff700 1 -- 172.21.2.112:0/3387010709 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_connection_expiration"]}]=0 v467) v1 ==== 143+0+0 (secure 0 0 0) 0x7f3364007360 con 0x7f336c19da20 2021-02-10T11:38:34.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:34.904 7f33714cc700 1 -- 172.21.2.112:0/3387010709 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f334c0186c0 msgr2=0x7f334c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:34.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:34.904 7f33714cc700 1 --2- 172.21.2.112:0/3387010709 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f334c0186c0 0x7f334c01abc0 crc :-1 s=READY pgs=843 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:34.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:34.904 7f33714cc700 1 -- 172.21.2.112:0/3387010709 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f336c19da20 msgr2=0x7f336c19de20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:34.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:34.904 7f33714cc700 1 --2- 172.21.2.112:0/3387010709 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f336c19da20 0x7f336c19de20 secure :-1 s=READY pgs=1791 cs=0 l=1 rev1=1 rx=0x7f336400a870 tx=0x7f336400ac10).stop 2021-02-10T11:38:34.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:34.904 7f33714cc700 1 -- 172.21.2.112:0/3387010709 shutdown_connections 2021-02-10T11:38:34.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:34.904 7f33714cc700 1 --2- 172.21.2.112:0/3387010709 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f334c0186c0 0x7f334c01abc0 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:34.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:34.904 7f33714cc700 1 --2- 172.21.2.112:0/3387010709 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f336c1b80f0 0x7f336c1b84f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:34.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:34.904 7f33714cc700 1 --2- 172.21.2.112:0/3387010709 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f336c19da20 0x7f336c19de20 unknown :-1 s=CLOSED pgs=1791 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:34.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:34.904 7f33714cc700 1 --2- 172.21.2.112:0/3387010709 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f336c1b7cf0 0x7f336c19cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:34.897 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:34.904 7f33714cc700 1 -- 172.21.2.112:0/3387010709 >> 172.21.2.112:0/3387010709 conn(0x7f336c0ed810 msgr2=0x7f336c094980 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:34.897 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:34.905 7f33714cc700 1 -- 172.21.2.112:0/3387010709 shutdown_connections 2021-02-10T11:38:34.897 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:34.905 7f33714cc700 1 -- 172.21.2.112:0/3387010709 wait complete. 2021-02-10T11:38:34.904 INFO:tasks.cephfs_test_runner:test_connection_expiration (tasks.cephfs.test_volumes.TestVolumes) ... ok 2021-02-10T11:38:34.904 INFO:tasks.cephfs_test_runner:Starting test: test_deauthorize_auth_id_after_out_of_band_update (tasks.cephfs.test_volumes.TestVolumes) 2021-02-10T11:38:34.904 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestVolumes.test_deauthorize_auth_id_after_out_of_band_update' 2021-02-10T11:38:35.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:35.323 7f35a3608700 1 Processor -- start 2021-02-10T11:38:35.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:35.323 7f35a3608700 1 -- start start 2021-02-10T11:38:35.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:35.324 7f35a3608700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f359c1b0870 0x7f359c0f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:35.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:35.324 7f35a3608700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f359c0efe00 0x7f359c0f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:35.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:35.324 7f35a3608700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f359c1b0c70 con 0x7f359c1b0470 2021-02-10T11:38:35.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:35.324 7f35a3608700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f359c0e5ea0 con 0x7f359c1b0870 2021-02-10T11:38:35.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:35.324 7f35a3608700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f359c0f0b50 con 0x7f359c0efe00 2021-02-10T11:38:35.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:35.324 7f35a0c23700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f359c1b0870 0x7f359c0f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:35.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:35.324 7f35a1424700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f359c1b0470 0x7f359c0ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:52480/0 (socket says 172.21.2.112:52480) 2021-02-10T11:38:35.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:35.324 7f35a0c23700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f359c1b0870 0x7f359c0f20e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:53018/0 (socket says 172.21.2.112:53018) 2021-02-10T11:38:35.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:35.324 7f35a1424700 1 -- 172.21.2.112:0/3978635935 learned_addr learned my addr 172.21.2.112:0/3978635935 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:38:35.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:35.325 7f35a0c23700 1 -- 172.21.2.112:0/3978635935 >> v2:172.21.2.120:3300/0 conn(0x7f359c0efe00 msgr2=0x7f359c0f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:35.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:35.325 7f35a0c23700 1 --2- 172.21.2.112:0/3978635935 >> v2:172.21.2.120:3300/0 conn(0x7f359c0efe00 0x7f359c0f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:35.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:35.325 7f35a0c23700 1 -- 172.21.2.112:0/3978635935 >> v1:172.21.2.112:6789/0 conn(0x7f359c1b0470 legacy=0x7f359c0ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:35.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:35.325 7f35a0c23700 1 -- 172.21.2.112:0/3978635935 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f359c0f0820 con 0x7f359c1b0870 2021-02-10T11:38:35.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:35.325 7f35a0c23700 1 --2- 172.21.2.112:0/3978635935 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f359c1b0870 0x7f359c0f20e0 secure :-1 s=READY pgs=1793 cs=0 l=1 rev1=1 rx=0x7f3598006650 tx=0x7f3598009a00).ready entity=mon.0 client_cookie=f8ce18d1e24a740d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:35.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:35.325 7f358ffff700 1 -- 172.21.2.112:0/3978635935 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f35980009b0 con 0x7f359c1b0870 2021-02-10T11:38:35.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:35.325 7f358ffff700 1 -- 172.21.2.112:0/3978635935 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f359800c0c0 con 0x7f359c1b0870 2021-02-10T11:38:35.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:35.325 7f358ffff700 1 -- 172.21.2.112:0/3978635935 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f359800c5a0 con 0x7f359c1b0870 2021-02-10T11:38:35.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:35.325 7f35a3608700 1 -- 172.21.2.112:0/3978635935 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f359c1b0870 msgr2=0x7f359c0f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:35.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:35.325 7f35a3608700 1 --2- 172.21.2.112:0/3978635935 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f359c1b0870 0x7f359c0f20e0 secure :-1 s=READY pgs=1793 cs=0 l=1 rev1=1 rx=0x7f3598006650 tx=0x7f3598009a00).stop 2021-02-10T11:38:35.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:35.326 7f35a3608700 1 -- 172.21.2.112:0/3978635935 shutdown_connections 2021-02-10T11:38:35.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:35.326 7f35a3608700 1 --2- 172.21.2.112:0/3978635935 >> v2:172.21.2.120:3300/0 conn(0x7f359c0efe00 0x7f359c0f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:35.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:35.326 7f35a3608700 1 --2- 172.21.2.112:0/3978635935 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f359c1b0870 0x7f359c0f20e0 unknown :-1 s=CLOSED pgs=1793 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:35.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:35.326 7f35a3608700 1 -- 172.21.2.112:0/3978635935 >> 172.21.2.112:0/3978635935 conn(0x7f359c0ce3e0 msgr2=0x7f359c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:35.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:35.326 7f35a3608700 1 -- 172.21.2.112:0/3978635935 shutdown_connections 2021-02-10T11:38:35.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:35.326 7f35a3608700 1 -- 172.21.2.112:0/3978635935 wait complete. 2021-02-10T11:38:35.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:35.341 7f35a3608700 1 Processor -- start 2021-02-10T11:38:35.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:35.341 7f35a3608700 1 -- start start 2021-02-10T11:38:35.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:35.341 7f35a3608700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f359c1095b0 0x7f359c19e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:35.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:35.341 7f35a3608700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f359c1099b0 0x7f359c0933c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:35.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:35.341 7f35a3608700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f359c0939e0 0x7f359c1ace00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:35.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:35.341 7f35a3608700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f359c093de0 con 0x7f359c0939e0 2021-02-10T11:38:35.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:35.341 7f35a3608700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f359c0aed40 con 0x7f359c1099b0 2021-02-10T11:38:35.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:35.341 7f35a3608700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f359c1ad420 con 0x7f359c1095b0 2021-02-10T11:38:35.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:35.341 7f35a1c25700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f359c0939e0 0x7f359c1ace00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:35.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:35.341 7f35a1c25700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f359c0939e0 0x7f359c1ace00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:53028/0 (socket says 172.21.2.112:53028) 2021-02-10T11:38:35.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:35.341 7f35a1c25700 1 -- 172.21.2.112:0/1647601661 learned_addr learned my addr 172.21.2.112:0/1647601661 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:38:35.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:35.342 7f35a1c25700 1 -- 172.21.2.112:0/1647601661 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f359c1095b0 msgr2=0x7f359c19e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:35.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:35.342 7f35a1c25700 1 --2- 172.21.2.112:0/1647601661 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f359c1095b0 0x7f359c19e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:35.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:35.342 7f35a1c25700 1 -- 172.21.2.112:0/1647601661 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f359c1099b0 msgr2=0x7f359c0933c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:35.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:35.342 7f35a1c25700 1 --2- 172.21.2.112:0/1647601661 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f359c1099b0 0x7f359c0933c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:35.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:35.342 7f35a1c25700 1 -- 172.21.2.112:0/1647601661 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f359c1ad640 con 0x7f359c0939e0 2021-02-10T11:38:35.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:35.342 7f35a1c25700 1 --2- 172.21.2.112:0/1647601661 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f359c0939e0 0x7f359c1ace00 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 rx=0x7f359400b8c0 tx=0x7f359400bc60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:35.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:35.342 7f358dffb700 1 -- 172.21.2.112:0/1647601661 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3594005af0 con 0x7f359c0939e0 2021-02-10T11:38:35.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:35.342 7f358dffb700 1 -- 172.21.2.112:0/1647601661 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3594005d30 con 0x7f359c0939e0 2021-02-10T11:38:35.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:35.342 7f35a3608700 1 -- 172.21.2.112:0/1647601661 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f359c1ad640 con 0x7f359c0939e0 2021-02-10T11:38:35.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:35.342 7f35a3608700 1 -- 172.21.2.112:0/1647601661 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f359c1adbd0 con 0x7f359c0939e0 2021-02-10T11:38:35.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:35.342 7f358dffb700 1 -- 172.21.2.112:0/1647601661 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3594006220 con 0x7f359c0939e0 2021-02-10T11:38:35.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:35.343 7f358dffb700 1 -- 172.21.2.112:0/1647601661 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3594006030 con 0x7f359c0939e0 2021-02-10T11:38:35.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:35.343 7f358dffb700 1 --2- 172.21.2.112:0/1647601661 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f357c018730 0x7f357c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:35.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:35.343 7f358dffb700 1 -- 172.21.2.112:0/1647601661 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f3594032ba0 con 0x7f359c0939e0 2021-02-10T11:38:35.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:35.343 7f35a1424700 1 --2- 172.21.2.112:0/1647601661 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f357c018730 0x7f357c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:35.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:35.344 7f35a1424700 1 --2- 172.21.2.112:0/1647601661 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f357c018730 0x7f357c01abc0 crc :-1 s=READY pgs=844 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:35.365 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:35.374 7f35a3608700 1 -- 172.21.2.112:0/1647601661 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f359c0f0b50 con 0x7f359c0939e0 2021-02-10T11:38:35.368 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:35.377 7f358dffb700 1 -- 172.21.2.112:0/1647601661 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f35940064c0 con 0x7f359c0939e0 2021-02-10T11:38:35.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:35.507 7f35a3608700 1 -- 172.21.2.112:0/1647601661 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_deauthorize_auth_id_after_out_of_band_update"]} v 0) v1 -- 0x7f359c0f0b50 con 0x7f359c0939e0 2021-02-10T11:38:35.875 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:35.884 7f358dffb700 1 -- 172.21.2.112:0/1647601661 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_deauthorize_auth_id_after_out_of_band_update"]}]=0 v468) v1 ==== 169+0+0 (secure 0 0 0) 0x7f35940064c0 con 0x7f359c0939e0 2021-02-10T11:38:35.917 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:35.925 7f35a3608700 1 -- 172.21.2.112:0/1647601661 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f357c018730 msgr2=0x7f357c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:35.917 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:35.925 7f35a3608700 1 --2- 172.21.2.112:0/1647601661 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f357c018730 0x7f357c01abc0 crc :-1 s=READY pgs=844 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:35.917 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:35.925 7f35a3608700 1 -- 172.21.2.112:0/1647601661 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f359c0939e0 msgr2=0x7f359c1ace00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:35.917 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:35.925 7f35a3608700 1 --2- 172.21.2.112:0/1647601661 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f359c0939e0 0x7f359c1ace00 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 rx=0x7f359400b8c0 tx=0x7f359400bc60).stop 2021-02-10T11:38:35.917 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:35.926 7f35a3608700 1 -- 172.21.2.112:0/1647601661 shutdown_connections 2021-02-10T11:38:35.918 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:35.926 7f35a3608700 1 --2- 172.21.2.112:0/1647601661 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f357c018730 0x7f357c01abc0 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:35.918 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:35.926 7f35a3608700 1 --2- 172.21.2.112:0/1647601661 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f359c1099b0 0x7f359c0933c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:35.918 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:35.926 7f35a3608700 1 --2- 172.21.2.112:0/1647601661 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f359c0939e0 0x7f359c1ace00 unknown :-1 s=CLOSED pgs=1794 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:35.918 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:35.926 7f35a3608700 1 --2- 172.21.2.112:0/1647601661 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f359c1095b0 0x7f359c19e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:35.918 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:35.926 7f35a3608700 1 -- 172.21.2.112:0/1647601661 >> 172.21.2.112:0/1647601661 conn(0x7f359c19f7b0 msgr2=0x7f359c18f380 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:35.918 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:35.926 7f35a3608700 1 -- 172.21.2.112:0/1647601661 shutdown_connections 2021-02-10T11:38:35.918 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:35.926 7f35a3608700 1 -- 172.21.2.112:0/1647601661 wait complete. 2021-02-10T11:38:35.925 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:38:36.355 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:36.347 7f97ad7e5700 1 Processor -- start 2021-02-10T11:38:36.355 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:36.347 7f97ad7e5700 1 -- start start 2021-02-10T11:38:36.355 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:36.348 7f97ad7e5700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f97a80d1870 0x7f97a81b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:36.355 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:36.348 7f97ad7e5700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f97a81b0ae0 0x7f97a8108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:36.355 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:36.348 7f97ad7e5700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f97a80d1c70 0x7f97a8108f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:36.355 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:36.348 7f97ad7e5700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97a80e5ef0 con 0x7f97a80d1c70 2021-02-10T11:38:36.355 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:36.348 7f97ad7e5700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97a8109a60 con 0x7f97a81b0ae0 2021-02-10T11:38:36.356 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:36.348 7f97ad7e5700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97a8088070 con 0x7f97a80d1870 2021-02-10T11:38:36.356 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:36.348 7f97a7d75700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f97a80d1c70 0x7f97a8108f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:36.356 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:36.348 7f97a7d75700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f97a80d1c70 0x7f97a8108f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:53032/0 (socket says 172.21.2.112:53032) 2021-02-10T11:38:36.356 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:36.348 7f97a7d75700 1 -- 172.21.2.112:0/227110183 learned_addr learned my addr 172.21.2.112:0/227110183 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:38:36.356 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:36.348 7f97a7d75700 1 -- 172.21.2.112:0/227110183 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f97a80d1870 msgr2=0x7f97a81b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:36.356 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:36.348 7f97a7d75700 1 --2- 172.21.2.112:0/227110183 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f97a80d1870 0x7f97a81b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:36.356 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:36.348 7f97a7d75700 1 -- 172.21.2.112:0/227110183 >> v2:172.21.2.120:3300/0 conn(0x7f97a81b0ae0 msgr2=0x7f97a8108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:36.356 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:36.348 7f97a7d75700 1 --2- 172.21.2.112:0/227110183 >> v2:172.21.2.120:3300/0 conn(0x7f97a81b0ae0 0x7f97a8108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:36.356 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:36.348 7f97a7d75700 1 -- 172.21.2.112:0/227110183 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97a8109580 con 0x7f97a80d1c70 2021-02-10T11:38:36.356 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:36.348 7f97a7d75700 1 --2- 172.21.2.112:0/227110183 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f97a80d1c70 0x7f97a8108f60 secure :-1 s=READY pgs=1795 cs=0 l=1 rev1=1 rx=0x7f97a0006f00 tx=0x7f97a0009f10).ready entity=mon.0 client_cookie=6c9b425cd7d87d88 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:36.356 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:36.349 7f97a6572700 1 -- 172.21.2.112:0/227110183 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f97a000b3b0 con 0x7f97a80d1c70 2021-02-10T11:38:36.356 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:36.349 7f97a6572700 1 -- 172.21.2.112:0/227110183 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97a000b730 con 0x7f97a80d1c70 2021-02-10T11:38:36.356 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:36.349 7f97a6572700 1 -- 172.21.2.112:0/227110183 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f97a000bc50 con 0x7f97a80d1c70 2021-02-10T11:38:36.357 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:36.349 7f97ad7e5700 1 -- 172.21.2.112:0/227110183 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f97a80d1c70 msgr2=0x7f97a8108f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:36.357 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:36.349 7f97ad7e5700 1 --2- 172.21.2.112:0/227110183 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f97a80d1c70 0x7f97a8108f60 secure :-1 s=READY pgs=1795 cs=0 l=1 rev1=1 rx=0x7f97a0006f00 tx=0x7f97a0009f10).stop 2021-02-10T11:38:36.357 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:36.349 7f97ad7e5700 1 -- 172.21.2.112:0/227110183 shutdown_connections 2021-02-10T11:38:36.357 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:36.349 7f97ad7e5700 1 --2- 172.21.2.112:0/227110183 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f97a80d1c70 0x7f97a8108f60 unknown :-1 s=CLOSED pgs=1795 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:36.357 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:36.350 7f97ad7e5700 1 --2- 172.21.2.112:0/227110183 >> v2:172.21.2.120:3300/0 conn(0x7f97a81b0ae0 0x7f97a8108940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:36.357 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:36.350 7f97ad7e5700 1 --2- 172.21.2.112:0/227110183 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f97a80d1870 0x7f97a81b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:36.357 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:36.350 7f97ad7e5700 1 -- 172.21.2.112:0/227110183 >> 172.21.2.112:0/227110183 conn(0x7f97a80ce420 msgr2=0x7f97a80d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:36.357 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:36.350 7f97ad7e5700 1 -- 172.21.2.112:0/227110183 shutdown_connections 2021-02-10T11:38:36.357 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:36.350 7f97ad7e5700 1 -- 172.21.2.112:0/227110183 wait complete. 2021-02-10T11:38:36.358 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:36.364 7f97ad7e5700 1 Processor -- start 2021-02-10T11:38:36.358 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:36.364 7f97ad7e5700 1 -- start start 2021-02-10T11:38:36.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:36.365 7f97ad7e5700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f97a80a73c0 0x7f97a80a6b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:36.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:36.365 7f97ad7e5700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f97a80add40 0x7f97a80ae140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:36.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:36.365 7f97ad7e5700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f97a80ae760 0x7f97a80af180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:36.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:36.365 7f97ad7e5700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97a80aec00 con 0x7f97a80a73c0 2021-02-10T11:38:36.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:36.365 7f97ad7e5700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97a80875d0 con 0x7f97a80ae760 2021-02-10T11:38:36.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:36.365 7f97ad7e5700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97a80af7a0 con 0x7f97a80add40 2021-02-10T11:38:36.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:36.365 7f97a7574700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f97a80a73c0 0x7f97a80a6b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:36.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:36.365 7f97a7574700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f97a80a73c0 0x7f97a80a6b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:53036/0 (socket says 172.21.2.112:53036) 2021-02-10T11:38:36.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:36.365 7f97a7574700 1 -- 172.21.2.112:0/982280131 learned_addr learned my addr 172.21.2.112:0/982280131 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:38:36.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:36.365 7f97a7574700 1 -- 172.21.2.112:0/982280131 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f97a80add40 msgr2=0x7f97a80ae140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:36.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:36.365 7f97a7574700 1 --2- 172.21.2.112:0/982280131 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f97a80add40 0x7f97a80ae140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:36.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:36.365 7f97a7574700 1 -- 172.21.2.112:0/982280131 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f97a80ae760 msgr2=0x7f97a80af180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:36.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:36.365 7f97a7574700 1 --2- 172.21.2.112:0/982280131 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f97a80ae760 0x7f97a80af180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:36.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:36.365 7f97a7574700 1 -- 172.21.2.112:0/982280131 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97a80af9c0 con 0x7f97a80a73c0 2021-02-10T11:38:36.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:36.366 7f97a7574700 1 --2- 172.21.2.112:0/982280131 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f97a80a73c0 0x7f97a80a6b50 secure :-1 s=READY pgs=1796 cs=0 l=1 rev1=1 rx=0x7f9798006f00 tx=0x7f979800b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:36.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:36.366 7f978ffff700 1 -- 172.21.2.112:0/982280131 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f979800bd30 con 0x7f97a80a73c0 2021-02-10T11:38:36.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:36.366 7f978ffff700 1 -- 172.21.2.112:0/982280131 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9798000c90 con 0x7f97a80a73c0 2021-02-10T11:38:36.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:36.366 7f978ffff700 1 -- 172.21.2.112:0/982280131 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f97980022a0 con 0x7f97a80a73c0 2021-02-10T11:38:36.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:36.366 7f97ad7e5700 1 -- 172.21.2.112:0/982280131 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97a80af9c0 con 0x7f97a80a73c0 2021-02-10T11:38:36.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:36.366 7f97ad7e5700 1 -- 172.21.2.112:0/982280131 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97a80aff50 con 0x7f97a80a73c0 2021-02-10T11:38:36.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:36.367 7f978ffff700 1 -- 172.21.2.112:0/982280131 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9798005310 con 0x7f97a80a73c0 2021-02-10T11:38:36.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:36.367 7f978ffff700 1 --2- 172.21.2.112:0/982280131 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f97880186c0 0x7f978801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:36.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:36.367 7f978ffff700 1 -- 172.21.2.112:0/982280131 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f9798032520 con 0x7f97a80a73c0 2021-02-10T11:38:36.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:36.367 7f97a6d73700 1 --2- 172.21.2.112:0/982280131 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f97880186c0 0x7f978801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:36.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:36.369 7f97a6d73700 1 --2- 172.21.2.112:0/982280131 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f97880186c0 0x7f978801abc0 crc :-1 s=READY pgs=845 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:36.389 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:36.398 7f97ad7e5700 1 -- 172.21.2.112:0/982280131 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97a80b04f0 con 0x7f97a80a73c0 2021-02-10T11:38:36.391 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:36.400 7f978ffff700 1 -- 172.21.2.112:0/982280131 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9798025fc0 con 0x7f97a80a73c0 2021-02-10T11:38:36.540 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:36.548 7f97ad7e5700 1 -- 172.21.2.112:0/982280131 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f97a80b04f0 con 0x7f97a80a73c0 2021-02-10T11:38:36.544 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:36.552 7f978ffff700 1 -- 172.21.2.112:0/982280131 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v24)=0 v24) v1 ==== 122+0+0 (secure 0 0 0) 0x7f9798025fc0 con 0x7f97a80a73c0 2021-02-10T11:38:36.549 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:36.557 7f97ad7e5700 1 -- 172.21.2.112:0/982280131 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f97880186c0 msgr2=0x7f978801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:36.549 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:36.557 7f97ad7e5700 1 --2- 172.21.2.112:0/982280131 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f97880186c0 0x7f978801abc0 crc :-1 s=READY pgs=845 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:36.549 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:36.557 7f97ad7e5700 1 -- 172.21.2.112:0/982280131 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f97a80a73c0 msgr2=0x7f97a80a6b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:36.549 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:36.557 7f97ad7e5700 1 --2- 172.21.2.112:0/982280131 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f97a80a73c0 0x7f97a80a6b50 secure :-1 s=READY pgs=1796 cs=0 l=1 rev1=1 rx=0x7f9798006f00 tx=0x7f979800b940).stop 2021-02-10T11:38:36.549 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:36.558 7f97ad7e5700 1 -- 172.21.2.112:0/982280131 shutdown_connections 2021-02-10T11:38:36.549 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:36.558 7f97ad7e5700 1 --2- 172.21.2.112:0/982280131 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f97880186c0 0x7f978801abc0 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:36.549 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:36.558 7f97ad7e5700 1 --2- 172.21.2.112:0/982280131 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f97a80ae760 0x7f97a80af180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:36.549 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:36.558 7f97ad7e5700 1 --2- 172.21.2.112:0/982280131 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f97a80add40 0x7f97a80ae140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:36.550 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:36.558 7f97ad7e5700 1 --2- 172.21.2.112:0/982280131 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f97a80a73c0 0x7f97a80a6b50 unknown :-1 s=CLOSED pgs=1796 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:36.550 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:36.558 7f97ad7e5700 1 -- 172.21.2.112:0/982280131 >> 172.21.2.112:0/982280131 conn(0x7f97a80be980 msgr2=0x7f97a8092cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:36.550 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:36.558 7f97ad7e5700 1 -- 172.21.2.112:0/982280131 shutdown_connections 2021-02-10T11:38:36.550 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:36.558 7f97ad7e5700 1 -- 172.21.2.112:0/982280131 wait complete. 2021-02-10T11:38:36.557 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:38:36.557 DEBUG:teuthology.orchestra.run.gibba012:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:38:36.903 DEBUG:teuthology.orchestra.run.gibba020:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:38:37.246 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:38:37.246 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:38:37.247 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:38:37.247 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:38:37.580 INFO:teuthology.orchestra.run.gibba012.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.0’: No such file or directory 2021-02-10T11:38:37.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:38:37.581 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2021-02-10T11:38:37.581 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:38:37.581 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:38:37.581 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:38:37.916 INFO:teuthology.orchestra.run.gibba012.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: No such file or directory 2021-02-10T11:38:37.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:38:37.917 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2021-02-10T11:38:37.917 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:38:37.917 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:38:37.917 DEBUG:teuthology.orchestra.run.gibba020:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:38:38.250 INFO:teuthology.orchestra.run.gibba020.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:38:38.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:38:38.251 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:38:38.251 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:38:38.251 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:38:38.251 DEBUG:teuthology.orchestra.run.gibba020:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:38:38.590 INFO:teuthology.orchestra.run.gibba020.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:38:38.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:38:38.591 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:38:38.591 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json-pretty 2021-02-10T11:38:39.019 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.011 7f3e7c8f8700 1 Processor -- start 2021-02-10T11:38:39.019 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.011 7f3e7c8f8700 1 -- start start 2021-02-10T11:38:39.019 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.011 7f3e7c8f8700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f3e780d1870 0x7f3e781b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:39.019 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.011 7f3e7c8f8700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f3e780f0650 0x7f3e780cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:39.019 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.011 7f3e7c8f8700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e780f0b10 con 0x7f3e781b0a10 2021-02-10T11:38:39.019 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.011 7f3e7c8f8700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e780e5ed0 con 0x7f3e780f0650 2021-02-10T11:38:39.019 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.011 7f3e7c8f8700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e780d1c70 con 0x7f3e780d1870 2021-02-10T11:38:39.019 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.012 7f3e76e1d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f3e780f0650 0x7f3e780cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:39.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.012 7f3e76e1d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f3e780f0650 0x7f3e780cfb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:53044/0 (socket says 172.21.2.112:53044) 2021-02-10T11:38:39.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.012 7f3e76e1d700 1 -- 172.21.2.112:0/965853939 learned_addr learned my addr 172.21.2.112:0/965853939 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:38:39.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.012 7f3e76e1d700 1 -- 172.21.2.112:0/965853939 >> v2:172.21.2.120:3300/0 conn(0x7f3e780d1870 msgr2=0x7f3e781b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:39.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.012 7f3e76e1d700 1 --2- 172.21.2.112:0/965853939 >> v2:172.21.2.120:3300/0 conn(0x7f3e780d1870 0x7f3e781b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:39.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.012 7f3e76e1d700 1 -- 172.21.2.112:0/965853939 >> v1:172.21.2.120:6789/0 conn(0x7f3e781b0a10 legacy=0x7f3e780cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:39.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.012 7f3e76e1d700 1 -- 172.21.2.112:0/965853939 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e780d1e90 con 0x7f3e780f0650 2021-02-10T11:38:39.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.012 7f3e76e1d700 1 --2- 172.21.2.112:0/965853939 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3e780f0650 0x7f3e780cfb20 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 rx=0x7f3e70006f00 tx=0x7f3e70009f10).ready entity=mon.0 client_cookie=49ce705cf92d6430 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:39.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.012 7f3e7561a700 1 -- 172.21.2.112:0/965853939 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3e7000b3b0 con 0x7f3e780f0650 2021-02-10T11:38:39.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.012 7f3e7561a700 1 -- 172.21.2.112:0/965853939 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e7000b630 con 0x7f3e780f0650 2021-02-10T11:38:39.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.012 7f3e7561a700 1 -- 172.21.2.112:0/965853939 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3e7000bb10 con 0x7f3e780f0650 2021-02-10T11:38:39.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.012 7f3e7c8f8700 1 -- 172.21.2.112:0/965853939 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3e780f0650 msgr2=0x7f3e780cfb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:39.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.012 7f3e7c8f8700 1 --2- 172.21.2.112:0/965853939 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3e780f0650 0x7f3e780cfb20 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 rx=0x7f3e70006f00 tx=0x7f3e70009f10).stop 2021-02-10T11:38:39.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.013 7f3e7c8f8700 1 -- 172.21.2.112:0/965853939 shutdown_connections 2021-02-10T11:38:39.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.013 7f3e7c8f8700 1 --2- 172.21.2.112:0/965853939 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3e780f0650 0x7f3e780cfb20 unknown :-1 s=CLOSED pgs=1797 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:39.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.013 7f3e7c8f8700 1 --2- 172.21.2.112:0/965853939 >> v2:172.21.2.120:3300/0 conn(0x7f3e780d1870 0x7f3e781b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:39.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.013 7f3e7c8f8700 1 -- 172.21.2.112:0/965853939 >> 172.21.2.112:0/965853939 conn(0x7f3e780ce450 msgr2=0x7f3e780d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:39.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.013 7f3e7c8f8700 1 -- 172.21.2.112:0/965853939 shutdown_connections 2021-02-10T11:38:39.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.013 7f3e7c8f8700 1 -- 172.21.2.112:0/965853939 wait complete. 2021-02-10T11:38:39.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.028 7f3e7c8f8700 1 Processor -- start 2021-02-10T11:38:39.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.028 7f3e7c8f8700 1 -- start start 2021-02-10T11:38:39.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.028 7f3e7c8f8700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3e780b0ad0 0x7f3e78091f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:39.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.028 7f3e7c8f8700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3e781a8850 0x7f3e781a8c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:39.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.028 7f3e7c8f8700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3e781a6fa0 0x7f3e781a73a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:39.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.028 7f3e7c8f8700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e781abfe0 con 0x7f3e781a6fa0 2021-02-10T11:38:39.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.028 7f3e7c8f8700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e781a9c40 con 0x7f3e781a8850 2021-02-10T11:38:39.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.028 7f3e7c8f8700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e781a9730 con 0x7f3e780b0ad0 2021-02-10T11:38:39.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.028 7f3e76e1d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3e781a6fa0 0x7f3e781a73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:39.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.028 7f3e76e1d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3e781a6fa0 0x7f3e781a73a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:53052/0 (socket says 172.21.2.112:53052) 2021-02-10T11:38:39.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.028 7f3e76e1d700 1 -- 172.21.2.112:0/4280816153 learned_addr learned my addr 172.21.2.112:0/4280816153 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:38:39.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.029 7f3e76e1d700 1 -- 172.21.2.112:0/4280816153 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3e780b0ad0 msgr2=0x7f3e78091f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:39.024 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.029 7f3e76e1d700 1 --2- 172.21.2.112:0/4280816153 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3e780b0ad0 0x7f3e78091f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:39.024 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.029 7f3e76e1d700 1 -- 172.21.2.112:0/4280816153 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3e781a8850 msgr2=0x7f3e781a8c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:39.024 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.029 7f3e76e1d700 1 --2- 172.21.2.112:0/4280816153 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3e781a8850 0x7f3e781a8c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:39.024 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.029 7f3e76e1d700 1 -- 172.21.2.112:0/4280816153 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e781a9270 con 0x7f3e781a6fa0 2021-02-10T11:38:39.024 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.029 7f3e76e1d700 1 --2- 172.21.2.112:0/4280816153 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3e781a6fa0 0x7f3e781a73a0 secure :-1 s=READY pgs=1798 cs=0 l=1 rev1=1 rx=0x7f3e70009390 tx=0x7f3e7000bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:39.024 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.029 7f3e6effd700 1 -- 172.21.2.112:0/4280816153 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3e70009450 con 0x7f3e781a6fa0 2021-02-10T11:38:39.024 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.029 7f3e6effd700 1 -- 172.21.2.112:0/4280816153 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e700068e0 con 0x7f3e781a6fa0 2021-02-10T11:38:39.024 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.029 7f3e6effd700 1 -- 172.21.2.112:0/4280816153 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3e7000c4e0 con 0x7f3e781a6fa0 2021-02-10T11:38:39.024 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.029 7f3e7c8f8700 1 -- 172.21.2.112:0/4280816153 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e781a9270 con 0x7f3e781a6fa0 2021-02-10T11:38:39.024 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.029 7f3e7c8f8700 1 -- 172.21.2.112:0/4280816153 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e781a7cb0 con 0x7f3e781a6fa0 2021-02-10T11:38:39.024 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.030 7f3e6effd700 1 -- 172.21.2.112:0/4280816153 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3e70001270 con 0x7f3e781a6fa0 2021-02-10T11:38:39.025 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.030 7f3e6effd700 1 --2- 172.21.2.112:0/4280816153 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3e5c0186c0 0x7f3e5c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:39.025 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.030 7f3e6effd700 1 -- 172.21.2.112:0/4280816153 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f3e70032910 con 0x7f3e781a6fa0 2021-02-10T11:38:39.025 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.030 7f3e7661c700 1 --2- 172.21.2.112:0/4280816153 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3e5c0186c0 0x7f3e5c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:39.025 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.031 7f3e7661c700 1 --2- 172.21.2.112:0/4280816153 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3e5c0186c0 0x7f3e5c01abc0 crc :-1 s=READY pgs=846 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:39.053 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.061 7f3e7c8f8700 1 -- 172.21.2.112:0/4280816153 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e780ce450 con 0x7f3e781a6fa0 2021-02-10T11:38:39.056 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.064 7f3e6effd700 1 -- 172.21.2.112:0/4280816153 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3e70003000 con 0x7f3e781a6fa0 2021-02-10T11:38:39.181 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.189 7f3e7c8f8700 1 -- 172.21.2.112:0/4280816153 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f3e780ce450 con 0x7f3e781a6fa0 2021-02-10T11:38:39.182 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.190 7f3e6effd700 1 -- 172.21.2.112:0/4280816153 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v83) v1 ==== 81+0+30363 (secure 0 0 0) 0x7f3e70001270 con 0x7f3e781a6fa0 2021-02-10T11:38:39.182 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:38:39.183 INFO:teuthology.orchestra.run.gibba012.stdout:{ 2021-02-10T11:38:39.183 INFO:teuthology.orchestra.run.gibba012.stdout: "epoch": 83, 2021-02-10T11:38:39.183 INFO:teuthology.orchestra.run.gibba012.stdout: "fsid": "f14f004e-0699-4584-ad81-3754258493de", 2021-02-10T11:38:39.183 INFO:teuthology.orchestra.run.gibba012.stdout: "created": "2021-02-10 11:13:26.501424", 2021-02-10T11:38:39.183 INFO:teuthology.orchestra.run.gibba012.stdout: "modified": "2021-02-10 11:36:54.566363", 2021-02-10T11:38:39.183 INFO:teuthology.orchestra.run.gibba012.stdout: "last_up_change": "2021-02-10 11:13:52.780316", 2021-02-10T11:38:39.183 INFO:teuthology.orchestra.run.gibba012.stdout: "last_in_change": "2021-02-10 11:13:52.780316", 2021-02-10T11:38:39.183 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-02-10T11:38:39.183 INFO:teuthology.orchestra.run.gibba012.stdout: "flags_num": 5799936, 2021-02-10T11:38:39.183 INFO:teuthology.orchestra.run.gibba012.stdout: "flags_set": [ 2021-02-10T11:38:39.183 INFO:teuthology.orchestra.run.gibba012.stdout: "pglog_hardlimit", 2021-02-10T11:38:39.184 INFO:teuthology.orchestra.run.gibba012.stdout: "purged_snapdirs", 2021-02-10T11:38:39.184 INFO:teuthology.orchestra.run.gibba012.stdout: "recovery_deletes", 2021-02-10T11:38:39.184 INFO:teuthology.orchestra.run.gibba012.stdout: "sortbitwise" 2021-02-10T11:38:39.184 INFO:teuthology.orchestra.run.gibba012.stdout: ], 2021-02-10T11:38:39.184 INFO:teuthology.orchestra.run.gibba012.stdout: "crush_version": 4, 2021-02-10T11:38:39.184 INFO:teuthology.orchestra.run.gibba012.stdout: "full_ratio": 0.94999998807907104, 2021-02-10T11:38:39.184 INFO:teuthology.orchestra.run.gibba012.stdout: "backfillfull_ratio": 0.89999997615814209, 2021-02-10T11:38:39.185 INFO:teuthology.orchestra.run.gibba012.stdout: "nearfull_ratio": 0.85000002384185791, 2021-02-10T11:38:39.185 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_snapshot": "", 2021-02-10T11:38:39.185 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_max": 17, 2021-02-10T11:38:39.185 INFO:teuthology.orchestra.run.gibba012.stdout: "max_osd": 8, 2021-02-10T11:38:39.185 INFO:teuthology.orchestra.run.gibba012.stdout: "require_min_compat_client": "jewel", 2021-02-10T11:38:39.185 INFO:teuthology.orchestra.run.gibba012.stdout: "min_compat_client": "jewel", 2021-02-10T11:38:39.185 INFO:teuthology.orchestra.run.gibba012.stdout: "require_osd_release": "nautilus", 2021-02-10T11:38:39.185 INFO:teuthology.orchestra.run.gibba012.stdout: "pools": [ 2021-02-10T11:38:39.185 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:38:39.185 INFO:teuthology.orchestra.run.gibba012.stdout: "pool": 1, 2021-02-10T11:38:39.185 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_name": "rbd", 2021-02-10T11:38:39.185 INFO:teuthology.orchestra.run.gibba012.stdout: "create_time": "2021-02-10 11:13:57.508758", 2021-02-10T11:38:39.186 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:38:39.186 INFO:teuthology.orchestra.run.gibba012.stdout: "flags_names": "hashpspool", 2021-02-10T11:38:39.186 INFO:teuthology.orchestra.run.gibba012.stdout: "type": 1, 2021-02-10T11:38:39.186 INFO:teuthology.orchestra.run.gibba012.stdout: "size": 2, 2021-02-10T11:38:39.186 INFO:teuthology.orchestra.run.gibba012.stdout: "min_size": 1, 2021-02-10T11:38:39.186 INFO:teuthology.orchestra.run.gibba012.stdout: "crush_rule": 0, 2021-02-10T11:38:39.186 INFO:teuthology.orchestra.run.gibba012.stdout: "object_hash": 2, 2021-02-10T11:38:39.186 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:38:39.186 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num": 8, 2021-02-10T11:38:39.186 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num": 8, 2021-02-10T11:38:39.186 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num_target": 8, 2021-02-10T11:38:39.186 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_target": 8, 2021-02-10T11:38:39.187 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_pending": 8, 2021-02-10T11:38:39.187 INFO:teuthology.orchestra.run.gibba012.stdout: "last_pg_merge_meta": { 2021-02-10T11:38:39.187 INFO:teuthology.orchestra.run.gibba012.stdout: "source_pgid": "0.0", 2021-02-10T11:38:39.187 INFO:teuthology.orchestra.run.gibba012.stdout: "ready_epoch": 0, 2021-02-10T11:38:39.187 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_started": 0, 2021-02-10T11:38:39.187 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_clean": 0, 2021-02-10T11:38:39.187 INFO:teuthology.orchestra.run.gibba012.stdout: "source_version": "0'0", 2021-02-10T11:38:39.187 INFO:teuthology.orchestra.run.gibba012.stdout: "target_version": "0'0" 2021-02-10T11:38:39.187 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:38:39.187 INFO:teuthology.orchestra.run.gibba012.stdout: "last_change": "16", 2021-02-10T11:38:39.187 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend": "0", 2021-02-10T11:38:39.187 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:38:39.187 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:38:39.188 INFO:teuthology.orchestra.run.gibba012.stdout: "auid": 0, 2021-02-10T11:38:39.188 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:38:39.188 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_seq": 0, 2021-02-10T11:38:39.188 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_epoch": 0, 2021-02-10T11:38:39.188 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_snaps": [], 2021-02-10T11:38:39.188 INFO:teuthology.orchestra.run.gibba012.stdout: "removed_snaps": "[]", 2021-02-10T11:38:39.188 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_bytes": 0, 2021-02-10T11:38:39.188 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_objects": 0, 2021-02-10T11:38:39.188 INFO:teuthology.orchestra.run.gibba012.stdout: "tiers": [], 2021-02-10T11:38:39.188 INFO:teuthology.orchestra.run.gibba012.stdout: "tier_of": -1, 2021-02-10T11:38:39.188 INFO:teuthology.orchestra.run.gibba012.stdout: "read_tier": -1, 2021-02-10T11:38:39.188 INFO:teuthology.orchestra.run.gibba012.stdout: "write_tier": -1, 2021-02-10T11:38:39.189 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_mode": "none", 2021-02-10T11:38:39.189 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_bytes": 0, 2021-02-10T11:38:39.189 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_objects": 0, 2021-02-10T11:38:39.189 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:38:39.189 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:38:39.189 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:38:39.189 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_flush_age": 0, 2021-02-10T11:38:39.189 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_evict_age": 0, 2021-02-10T11:38:39.189 INFO:teuthology.orchestra.run.gibba012.stdout: "erasure_code_profile": "", 2021-02-10T11:38:39.189 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_params": { 2021-02-10T11:38:39.190 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "none" 2021-02-10T11:38:39.190 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:38:39.190 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_period": 0, 2021-02-10T11:38:39.190 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_count": 0, 2021-02-10T11:38:39.190 INFO:teuthology.orchestra.run.gibba012.stdout: "use_gmt_hitset": true, 2021-02-10T11:38:39.190 INFO:teuthology.orchestra.run.gibba012.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:38:39.190 INFO:teuthology.orchestra.run.gibba012.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:38:39.190 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:38:39.190 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:38:39.190 INFO:teuthology.orchestra.run.gibba012.stdout: "grade_table": [], 2021-02-10T11:38:39.190 INFO:teuthology.orchestra.run.gibba012.stdout: "stripe_width": 0, 2021-02-10T11:38:39.191 INFO:teuthology.orchestra.run.gibba012.stdout: "expected_num_objects": 0, 2021-02-10T11:38:39.191 INFO:teuthology.orchestra.run.gibba012.stdout: "fast_read": false, 2021-02-10T11:38:39.191 INFO:teuthology.orchestra.run.gibba012.stdout: "options": {}, 2021-02-10T11:38:39.191 INFO:teuthology.orchestra.run.gibba012.stdout: "application_metadata": { 2021-02-10T11:38:39.191 INFO:teuthology.orchestra.run.gibba012.stdout: "rbd": {} 2021-02-10T11:38:39.191 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:38:39.191 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:38:39.191 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:38:39.191 INFO:teuthology.orchestra.run.gibba012.stdout: "pool": 16, 2021-02-10T11:38:39.191 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_name": "cephfs_metadata", 2021-02-10T11:38:39.191 INFO:teuthology.orchestra.run.gibba012.stdout: "create_time": "2021-02-10 11:36:51.538038", 2021-02-10T11:38:39.191 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:38:39.191 INFO:teuthology.orchestra.run.gibba012.stdout: "flags_names": "hashpspool", 2021-02-10T11:38:39.192 INFO:teuthology.orchestra.run.gibba012.stdout: "type": 1, 2021-02-10T11:38:39.192 INFO:teuthology.orchestra.run.gibba012.stdout: "size": 2, 2021-02-10T11:38:39.192 INFO:teuthology.orchestra.run.gibba012.stdout: "min_size": 1, 2021-02-10T11:38:39.192 INFO:teuthology.orchestra.run.gibba012.stdout: "crush_rule": 0, 2021-02-10T11:38:39.192 INFO:teuthology.orchestra.run.gibba012.stdout: "object_hash": 2, 2021-02-10T11:38:39.192 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:38:39.192 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num": 8, 2021-02-10T11:38:39.192 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num": 8, 2021-02-10T11:38:39.192 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num_target": 8, 2021-02-10T11:38:39.193 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_target": 8, 2021-02-10T11:38:39.193 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_pending": 8, 2021-02-10T11:38:39.193 INFO:teuthology.orchestra.run.gibba012.stdout: "last_pg_merge_meta": { 2021-02-10T11:38:39.193 INFO:teuthology.orchestra.run.gibba012.stdout: "source_pgid": "0.0", 2021-02-10T11:38:39.193 INFO:teuthology.orchestra.run.gibba012.stdout: "ready_epoch": 0, 2021-02-10T11:38:39.193 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_started": 0, 2021-02-10T11:38:39.193 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_clean": 0, 2021-02-10T11:38:39.193 INFO:teuthology.orchestra.run.gibba012.stdout: "source_version": "0'0", 2021-02-10T11:38:39.193 INFO:teuthology.orchestra.run.gibba012.stdout: "target_version": "0'0" 2021-02-10T11:38:39.193 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:38:39.193 INFO:teuthology.orchestra.run.gibba012.stdout: "last_change": "82", 2021-02-10T11:38:39.194 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend": "0", 2021-02-10T11:38:39.194 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:38:39.194 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:38:39.194 INFO:teuthology.orchestra.run.gibba012.stdout: "auid": 0, 2021-02-10T11:38:39.194 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:38:39.194 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_seq": 0, 2021-02-10T11:38:39.194 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_epoch": 0, 2021-02-10T11:38:39.194 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_snaps": [], 2021-02-10T11:38:39.194 INFO:teuthology.orchestra.run.gibba012.stdout: "removed_snaps": "[]", 2021-02-10T11:38:39.194 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_bytes": 0, 2021-02-10T11:38:39.194 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_objects": 0, 2021-02-10T11:38:39.194 INFO:teuthology.orchestra.run.gibba012.stdout: "tiers": [], 2021-02-10T11:38:39.195 INFO:teuthology.orchestra.run.gibba012.stdout: "tier_of": -1, 2021-02-10T11:38:39.195 INFO:teuthology.orchestra.run.gibba012.stdout: "read_tier": -1, 2021-02-10T11:38:39.195 INFO:teuthology.orchestra.run.gibba012.stdout: "write_tier": -1, 2021-02-10T11:38:39.195 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_mode": "none", 2021-02-10T11:38:39.195 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_bytes": 0, 2021-02-10T11:38:39.195 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_objects": 0, 2021-02-10T11:38:39.195 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:38:39.195 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:38:39.195 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:38:39.195 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_flush_age": 0, 2021-02-10T11:38:39.195 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_evict_age": 0, 2021-02-10T11:38:39.195 INFO:teuthology.orchestra.run.gibba012.stdout: "erasure_code_profile": "", 2021-02-10T11:38:39.196 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_params": { 2021-02-10T11:38:39.196 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "none" 2021-02-10T11:38:39.196 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:38:39.196 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_period": 0, 2021-02-10T11:38:39.196 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_count": 0, 2021-02-10T11:38:39.196 INFO:teuthology.orchestra.run.gibba012.stdout: "use_gmt_hitset": true, 2021-02-10T11:38:39.196 INFO:teuthology.orchestra.run.gibba012.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:38:39.196 INFO:teuthology.orchestra.run.gibba012.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:38:39.196 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:38:39.197 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:38:39.197 INFO:teuthology.orchestra.run.gibba012.stdout: "grade_table": [], 2021-02-10T11:38:39.197 INFO:teuthology.orchestra.run.gibba012.stdout: "stripe_width": 0, 2021-02-10T11:38:39.197 INFO:teuthology.orchestra.run.gibba012.stdout: "expected_num_objects": 0, 2021-02-10T11:38:39.197 INFO:teuthology.orchestra.run.gibba012.stdout: "fast_read": false, 2021-02-10T11:38:39.197 INFO:teuthology.orchestra.run.gibba012.stdout: "options": { 2021-02-10T11:38:39.197 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_autoscale_bias": 4, 2021-02-10T11:38:39.197 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_min": 16, 2021-02-10T11:38:39.197 INFO:teuthology.orchestra.run.gibba012.stdout: "recovery_priority": 5 2021-02-10T11:38:39.197 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:38:39.197 INFO:teuthology.orchestra.run.gibba012.stdout: "application_metadata": { 2021-02-10T11:38:39.197 INFO:teuthology.orchestra.run.gibba012.stdout: "cephfs": { 2021-02-10T11:38:39.197 INFO:teuthology.orchestra.run.gibba012.stdout: "metadata": "cephfs" 2021-02-10T11:38:39.198 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:38:39.198 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:38:39.198 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:38:39.198 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:38:39.198 INFO:teuthology.orchestra.run.gibba012.stdout: "pool": 17, 2021-02-10T11:38:39.198 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_name": "cephfs_data", 2021-02-10T11:38:39.198 INFO:teuthology.orchestra.run.gibba012.stdout: "create_time": "2021-02-10 11:36:52.609006", 2021-02-10T11:38:39.198 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:38:39.198 INFO:teuthology.orchestra.run.gibba012.stdout: "flags_names": "hashpspool", 2021-02-10T11:38:39.198 INFO:teuthology.orchestra.run.gibba012.stdout: "type": 1, 2021-02-10T11:38:39.198 INFO:teuthology.orchestra.run.gibba012.stdout: "size": 2, 2021-02-10T11:38:39.198 INFO:teuthology.orchestra.run.gibba012.stdout: "min_size": 1, 2021-02-10T11:38:39.198 INFO:teuthology.orchestra.run.gibba012.stdout: "crush_rule": 0, 2021-02-10T11:38:39.199 INFO:teuthology.orchestra.run.gibba012.stdout: "object_hash": 2, 2021-02-10T11:38:39.199 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:38:39.199 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num": 8, 2021-02-10T11:38:39.199 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num": 8, 2021-02-10T11:38:39.199 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num_target": 8, 2021-02-10T11:38:39.199 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_target": 8, 2021-02-10T11:38:39.199 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_pending": 8, 2021-02-10T11:38:39.199 INFO:teuthology.orchestra.run.gibba012.stdout: "last_pg_merge_meta": { 2021-02-10T11:38:39.199 INFO:teuthology.orchestra.run.gibba012.stdout: "source_pgid": "0.0", 2021-02-10T11:38:39.199 INFO:teuthology.orchestra.run.gibba012.stdout: "ready_epoch": 0, 2021-02-10T11:38:39.199 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_started": 0, 2021-02-10T11:38:39.199 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_clean": 0, 2021-02-10T11:38:39.199 INFO:teuthology.orchestra.run.gibba012.stdout: "source_version": "0'0", 2021-02-10T11:38:39.200 INFO:teuthology.orchestra.run.gibba012.stdout: "target_version": "0'0" 2021-02-10T11:38:39.200 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:38:39.200 INFO:teuthology.orchestra.run.gibba012.stdout: "last_change": "83", 2021-02-10T11:38:39.200 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend": "0", 2021-02-10T11:38:39.200 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:38:39.200 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:38:39.200 INFO:teuthology.orchestra.run.gibba012.stdout: "auid": 0, 2021-02-10T11:38:39.200 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:38:39.200 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_seq": 0, 2021-02-10T11:38:39.200 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_epoch": 0, 2021-02-10T11:38:39.200 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_snaps": [], 2021-02-10T11:38:39.200 INFO:teuthology.orchestra.run.gibba012.stdout: "removed_snaps": "[]", 2021-02-10T11:38:39.200 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_bytes": 0, 2021-02-10T11:38:39.201 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_objects": 0, 2021-02-10T11:38:39.201 INFO:teuthology.orchestra.run.gibba012.stdout: "tiers": [], 2021-02-10T11:38:39.201 INFO:teuthology.orchestra.run.gibba012.stdout: "tier_of": -1, 2021-02-10T11:38:39.201 INFO:teuthology.orchestra.run.gibba012.stdout: "read_tier": -1, 2021-02-10T11:38:39.201 INFO:teuthology.orchestra.run.gibba012.stdout: "write_tier": -1, 2021-02-10T11:38:39.201 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_mode": "none", 2021-02-10T11:38:39.201 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_bytes": 0, 2021-02-10T11:38:39.201 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_objects": 0, 2021-02-10T11:38:39.202 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:38:39.202 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:38:39.202 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:38:39.202 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_flush_age": 0, 2021-02-10T11:38:39.202 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_evict_age": 0, 2021-02-10T11:38:39.202 INFO:teuthology.orchestra.run.gibba012.stdout: "erasure_code_profile": "", 2021-02-10T11:38:39.202 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_params": { 2021-02-10T11:38:39.202 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "none" 2021-02-10T11:38:39.202 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:38:39.202 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_period": 0, 2021-02-10T11:38:39.202 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_count": 0, 2021-02-10T11:38:39.203 INFO:teuthology.orchestra.run.gibba012.stdout: "use_gmt_hitset": true, 2021-02-10T11:38:39.203 INFO:teuthology.orchestra.run.gibba012.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:38:39.203 INFO:teuthology.orchestra.run.gibba012.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:38:39.203 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:38:39.203 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:38:39.203 INFO:teuthology.orchestra.run.gibba012.stdout: "grade_table": [], 2021-02-10T11:38:39.203 INFO:teuthology.orchestra.run.gibba012.stdout: "stripe_width": 0, 2021-02-10T11:38:39.203 INFO:teuthology.orchestra.run.gibba012.stdout: "expected_num_objects": 0, 2021-02-10T11:38:39.203 INFO:teuthology.orchestra.run.gibba012.stdout: "fast_read": false, 2021-02-10T11:38:39.203 INFO:teuthology.orchestra.run.gibba012.stdout: "options": {}, 2021-02-10T11:38:39.203 INFO:teuthology.orchestra.run.gibba012.stdout: "application_metadata": { 2021-02-10T11:38:39.203 INFO:teuthology.orchestra.run.gibba012.stdout: "cephfs": { 2021-02-10T11:38:39.203 INFO:teuthology.orchestra.run.gibba012.stdout: "data": "cephfs" 2021-02-10T11:38:39.204 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:38:39.204 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:38:39.204 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:38:39.204 INFO:teuthology.orchestra.run.gibba012.stdout: ], 2021-02-10T11:38:39.204 INFO:teuthology.orchestra.run.gibba012.stdout: "osds": [ 2021-02-10T11:38:39.204 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:38:39.204 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 0, 2021-02-10T11:38:39.204 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "95444b8f-6370-4a1d-ae5f-df62651b77da", 2021-02-10T11:38:39.204 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:38:39.204 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:38:39.204 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:38:39.204 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:38:39.205 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:38:39.205 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:38:39.205 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:38:39.205 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 81, 2021-02-10T11:38:39.205 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:38:39.205 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:38:39.205 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:38:39.205 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:38:39.205 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:38:39.205 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:38:39.205 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6817", 2021-02-10T11:38:39.205 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:38:39.205 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:38:39.205 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:38:39.206 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:38:39.206 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6818", 2021-02-10T11:38:39.206 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:38:39.206 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:38:39.206 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:38:39.206 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:38:39.206 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:38:39.206 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:38:39.206 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:38:39.206 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:38:39.206 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6820", 2021-02-10T11:38:39.206 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:38:39.207 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:38:39.207 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:38:39.207 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:38:39.207 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6822", 2021-02-10T11:38:39.207 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:38:39.207 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:38:39.207 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:38:39.207 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:38:39.207 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:38:39.207 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:38:39.207 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:38:39.207 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:38:39.207 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6828", 2021-02-10T11:38:39.208 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:38:39.208 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:38:39.208 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:38:39.208 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:38:39.208 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6830", 2021-02-10T11:38:39.208 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:38:39.208 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:38:39.208 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:38:39.208 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:38:39.208 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:38:39.208 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:38:39.208 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:38:39.209 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:38:39.209 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6824", 2021-02-10T11:38:39.209 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:38:39.209 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:38:39.209 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:38:39.209 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:38:39.209 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6826", 2021-02-10T11:38:39.209 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:38:39.209 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:38:39.210 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:38:39.210 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:38:39.210 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.112:6818/36030", 2021-02-10T11:38:39.210 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.112:6822/36030", 2021-02-10T11:38:39.210 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.112:6830/36030", 2021-02-10T11:38:39.210 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.112:6826/36030", 2021-02-10T11:38:39.210 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:38:39.210 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:38:39.210 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:38:39.210 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:38:39.210 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:38:39.211 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:38:39.211 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 1, 2021-02-10T11:38:39.211 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "a23e217e-0d8e-486e-be07-712adfabd262", 2021-02-10T11:38:39.211 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:38:39.211 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:38:39.211 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:38:39.211 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:38:39.211 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:38:39.211 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:38:39.211 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:38:39.211 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 71, 2021-02-10T11:38:39.211 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:38:39.212 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:38:39.212 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:38:39.212 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:38:39.212 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:38:39.212 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:38:39.212 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6800", 2021-02-10T11:38:39.212 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:38:39.212 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:38:39.212 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:38:39.212 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:38:39.212 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6802", 2021-02-10T11:38:39.212 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:38:39.212 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:38:39.212 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:38:39.213 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:38:39.213 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:38:39.213 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:38:39.213 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:38:39.213 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:38:39.213 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6804", 2021-02-10T11:38:39.213 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:38:39.213 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:38:39.213 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:38:39.213 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:38:39.213 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6806", 2021-02-10T11:38:39.213 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:38:39.214 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:38:39.214 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:38:39.214 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:38:39.214 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:38:39.214 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:38:39.214 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:38:39.214 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:38:39.214 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6812", 2021-02-10T11:38:39.214 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:38:39.214 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:38:39.214 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:38:39.214 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:38:39.215 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6814", 2021-02-10T11:38:39.215 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:38:39.215 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:38:39.215 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:38:39.215 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:38:39.215 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:38:39.215 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:38:39.215 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:38:39.215 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:38:39.215 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6808", 2021-02-10T11:38:39.215 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:38:39.215 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:38:39.215 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:38:39.216 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:38:39.216 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6810", 2021-02-10T11:38:39.216 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:38:39.216 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:38:39.216 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:38:39.216 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:38:39.216 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.112:6802/36029", 2021-02-10T11:38:39.216 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.112:6806/36029", 2021-02-10T11:38:39.216 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.112:6814/36029", 2021-02-10T11:38:39.216 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.112:6810/36029", 2021-02-10T11:38:39.216 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:38:39.217 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:38:39.217 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:38:39.217 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:38:39.217 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:38:39.217 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:38:39.217 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 2, 2021-02-10T11:38:39.217 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "026afa8a-c0f1-4b35-9a76-53c00fe0c2fc", 2021-02-10T11:38:39.217 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:38:39.217 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:38:39.217 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:38:39.217 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:38:39.217 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:38:39.217 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:38:39.218 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:38:39.218 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 80, 2021-02-10T11:38:39.218 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:38:39.218 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:38:39.218 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:38:39.218 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:38:39.218 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:38:39.218 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:38:39.218 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6801", 2021-02-10T11:38:39.218 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:38:39.218 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:38:39.218 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:38:39.219 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:38:39.219 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6803", 2021-02-10T11:38:39.219 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:38:39.219 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:38:39.219 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:38:39.219 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:38:39.219 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:38:39.219 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:38:39.219 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:38:39.219 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:38:39.219 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6805", 2021-02-10T11:38:39.219 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:38:39.220 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:38:39.220 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:38:39.220 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:38:39.220 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6807", 2021-02-10T11:38:39.220 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:38:39.220 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:38:39.220 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:38:39.220 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:38:39.220 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:38:39.220 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:38:39.220 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:38:39.220 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:38:39.220 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6813", 2021-02-10T11:38:39.221 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:38:39.221 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:38:39.221 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:38:39.221 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:38:39.221 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6815", 2021-02-10T11:38:39.221 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:38:39.221 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:38:39.221 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:38:39.221 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:38:39.221 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:38:39.221 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:38:39.221 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:38:39.221 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:38:39.222 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6809", 2021-02-10T11:38:39.222 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:38:39.222 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:38:39.222 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:38:39.222 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:38:39.222 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6811", 2021-02-10T11:38:39.222 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:38:39.222 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:38:39.222 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:38:39.222 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:38:39.222 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.112:6803/36031", 2021-02-10T11:38:39.222 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.112:6807/36031", 2021-02-10T11:38:39.223 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.112:6815/36031", 2021-02-10T11:38:39.223 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.112:6811/36031", 2021-02-10T11:38:39.223 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:38:39.223 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:38:39.223 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:38:39.223 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:38:39.223 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:38:39.223 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:38:39.223 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 3, 2021-02-10T11:38:39.223 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "58795682-e00d-4173-bb84-f7075ca89a11", 2021-02-10T11:38:39.223 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:38:39.223 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:38:39.224 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:38:39.224 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:38:39.224 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:38:39.224 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:38:39.224 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:38:39.224 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 80, 2021-02-10T11:38:39.224 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:38:39.224 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:38:39.224 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:38:39.224 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:38:39.224 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:38:39.224 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:38:39.224 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6816", 2021-02-10T11:38:39.224 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:38:39.225 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:38:39.225 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:38:39.225 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:38:39.225 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6819", 2021-02-10T11:38:39.225 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:38:39.225 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:38:39.225 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:38:39.225 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:38:39.225 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:38:39.225 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:38:39.225 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:38:39.225 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:38:39.225 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6821", 2021-02-10T11:38:39.226 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:38:39.226 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:38:39.226 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:38:39.226 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:38:39.226 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6823", 2021-02-10T11:38:39.226 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:38:39.226 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:38:39.226 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:38:39.226 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:38:39.226 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:38:39.226 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:38:39.226 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:38:39.227 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:38:39.227 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6829", 2021-02-10T11:38:39.227 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:38:39.227 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:38:39.227 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:38:39.227 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:38:39.227 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6831", 2021-02-10T11:38:39.227 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:38:39.227 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:38:39.227 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:38:39.227 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:38:39.227 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:38:39.227 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:38:39.228 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:38:39.228 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:38:39.228 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6825", 2021-02-10T11:38:39.228 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:38:39.228 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:38:39.228 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:38:39.228 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:38:39.228 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6827", 2021-02-10T11:38:39.228 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:38:39.228 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:38:39.228 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:38:39.228 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:38:39.229 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.112:6819/36028", 2021-02-10T11:38:39.229 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.112:6823/36028", 2021-02-10T11:38:39.229 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.112:6831/36028", 2021-02-10T11:38:39.229 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.112:6827/36028", 2021-02-10T11:38:39.229 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:38:39.229 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:38:39.229 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:38:39.229 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:38:39.229 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:38:39.229 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:38:39.229 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 4, 2021-02-10T11:38:39.229 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "6763f8d6-53dc-4e45-bdf6-a6eabcf81e39", 2021-02-10T11:38:39.229 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:38:39.230 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:38:39.230 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:38:39.230 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:38:39.230 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:38:39.230 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:38:39.230 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:38:39.230 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 81, 2021-02-10T11:38:39.230 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:38:39.230 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:38:39.230 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:38:39.230 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:38:39.231 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:38:39.231 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:38:39.231 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6810", 2021-02-10T11:38:39.231 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:38:39.231 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:38:39.231 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:38:39.231 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:38:39.231 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6813", 2021-02-10T11:38:39.231 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:38:39.231 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:38:39.231 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:38:39.231 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:38:39.231 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:38:39.231 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:38:39.232 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:38:39.232 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:38:39.232 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6816", 2021-02-10T11:38:39.232 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:38:39.232 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:38:39.232 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:38:39.232 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:38:39.232 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6819", 2021-02-10T11:38:39.232 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:38:39.232 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:38:39.232 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:38:39.232 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:38:39.232 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:38:39.233 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:38:39.233 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:38:39.233 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:38:39.233 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6825", 2021-02-10T11:38:39.233 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:38:39.233 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:38:39.233 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:38:39.233 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:38:39.233 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6827", 2021-02-10T11:38:39.233 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:38:39.233 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:38:39.233 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:38:39.233 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:38:39.234 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:38:39.234 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:38:39.234 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:38:39.234 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:38:39.234 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6821", 2021-02-10T11:38:39.234 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:38:39.234 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:38:39.234 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:38:39.234 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:38:39.234 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6823", 2021-02-10T11:38:39.234 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:38:39.234 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:38:39.235 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:38:39.235 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:38:39.235 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.120:6813/36388", 2021-02-10T11:38:39.235 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.120:6819/36388", 2021-02-10T11:38:39.235 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.120:6827/36388", 2021-02-10T11:38:39.235 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.120:6823/36388", 2021-02-10T11:38:39.235 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:38:39.235 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:38:39.235 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:38:39.235 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:38:39.235 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:38:39.235 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:38:39.235 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 5, 2021-02-10T11:38:39.238 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "5d0e4dd6-0cb0-4bf8-8314-626acaa1b544", 2021-02-10T11:38:39.238 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:38:39.238 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:38:39.238 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:38:39.239 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:38:39.239 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:38:39.239 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:38:39.239 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:38:39.239 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 80, 2021-02-10T11:38:39.239 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:38:39.239 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:38:39.239 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:38:39.239 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:38:39.239 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:38:39.239 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:38:39.239 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6804", 2021-02-10T11:38:39.239 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:38:39.240 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:38:39.240 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:38:39.240 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:38:39.240 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6806", 2021-02-10T11:38:39.240 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:38:39.240 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:38:39.240 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:38:39.240 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:38:39.240 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:38:39.240 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:38:39.240 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:38:39.240 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:38:39.240 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6809", 2021-02-10T11:38:39.241 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:38:39.241 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:38:39.241 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:38:39.241 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:38:39.241 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6812", 2021-02-10T11:38:39.241 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:38:39.241 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:38:39.241 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:38:39.241 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:38:39.241 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:38:39.241 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:38:39.241 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:38:39.241 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:38:39.242 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6820", 2021-02-10T11:38:39.242 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:38:39.242 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:38:39.242 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:38:39.242 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:38:39.242 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6822", 2021-02-10T11:38:39.242 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:38:39.242 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:38:39.242 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:38:39.242 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:38:39.242 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:38:39.242 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:38:39.243 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:38:39.243 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:38:39.243 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6815", 2021-02-10T11:38:39.243 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:38:39.243 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:38:39.243 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:38:39.243 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:38:39.243 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6818", 2021-02-10T11:38:39.243 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:38:39.243 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:38:39.243 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:38:39.243 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:38:39.243 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.120:6806/36387", 2021-02-10T11:38:39.243 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.120:6812/36387", 2021-02-10T11:38:39.244 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.120:6822/36387", 2021-02-10T11:38:39.244 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.120:6818/36387", 2021-02-10T11:38:39.244 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:38:39.244 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:38:39.244 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:38:39.244 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:38:39.244 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:38:39.244 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:38:39.244 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 6, 2021-02-10T11:38:39.244 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "b010e187-4541-4b00-80dd-f948f6ab8f15", 2021-02-10T11:38:39.244 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:38:39.244 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:38:39.244 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:38:39.245 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:38:39.245 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:38:39.245 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:38:39.245 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:38:39.245 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 81, 2021-02-10T11:38:39.245 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:38:39.245 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:38:39.245 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:38:39.245 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:38:39.245 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:38:39.245 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:38:39.245 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6824", 2021-02-10T11:38:39.245 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:38:39.245 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:38:39.246 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:38:39.246 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:38:39.246 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6826", 2021-02-10T11:38:39.246 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:38:39.246 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:38:39.246 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:38:39.246 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:38:39.246 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:38:39.246 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:38:39.246 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:38:39.246 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:38:39.246 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6828", 2021-02-10T11:38:39.247 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:38:39.247 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:38:39.247 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:38:39.247 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:38:39.247 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6829", 2021-02-10T11:38:39.247 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:38:39.247 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:38:39.247 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:38:39.247 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:38:39.247 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:38:39.247 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:38:39.247 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:38:39.247 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:38:39.248 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6832", 2021-02-10T11:38:39.248 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:38:39.248 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:38:39.248 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:38:39.248 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:38:39.248 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6833", 2021-02-10T11:38:39.248 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:38:39.248 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:38:39.248 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:38:39.248 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:38:39.248 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:38:39.248 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:38:39.248 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:38:39.249 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:38:39.249 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6830", 2021-02-10T11:38:39.249 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:38:39.249 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:38:39.249 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:38:39.249 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:38:39.249 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6831", 2021-02-10T11:38:39.249 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:38:39.249 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:38:39.249 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:38:39.249 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:38:39.249 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.120:6826/36385", 2021-02-10T11:38:39.249 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.120:6829/36385", 2021-02-10T11:38:39.250 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.120:6833/36385", 2021-02-10T11:38:39.250 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.120:6831/36385", 2021-02-10T11:38:39.250 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:38:39.250 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:38:39.250 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:38:39.250 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:38:39.250 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:38:39.250 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:38:39.250 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 7, 2021-02-10T11:38:39.250 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "671e2772-33b7-497e-8e44-2178bd850bcf", 2021-02-10T11:38:39.250 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:38:39.250 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:38:39.250 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:38:39.251 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:38:39.251 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:38:39.251 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:38:39.251 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:38:39.251 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 81, 2021-02-10T11:38:39.251 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:38:39.251 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:38:39.251 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:38:39.251 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:38:39.251 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:38:39.251 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:38:39.251 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6802", 2021-02-10T11:38:39.252 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:38:39.252 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:38:39.252 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:38:39.252 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:38:39.252 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6803", 2021-02-10T11:38:39.252 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:38:39.252 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:38:39.252 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:38:39.252 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:38:39.252 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:38:39.252 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:38:39.252 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:38:39.253 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:38:39.253 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6805", 2021-02-10T11:38:39.253 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:38:39.253 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:38:39.253 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:38:39.253 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:38:39.253 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6807", 2021-02-10T11:38:39.253 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:38:39.253 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:38:39.253 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:38:39.253 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:38:39.253 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:38:39.253 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:38:39.254 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:38:39.254 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:38:39.254 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6814", 2021-02-10T11:38:39.254 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:38:39.254 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:38:39.254 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:38:39.254 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:38:39.254 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6817", 2021-02-10T11:38:39.254 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:38:39.254 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:38:39.254 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:38:39.254 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:38:39.255 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:38:39.255 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:38:39.255 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:38:39.255 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:38:39.255 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6808", 2021-02-10T11:38:39.255 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:38:39.255 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:38:39.255 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:38:39.255 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:38:39.255 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6811", 2021-02-10T11:38:39.255 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:38:39.255 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:38:39.255 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:38:39.256 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:38:39.256 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.120:6803/36386", 2021-02-10T11:38:39.256 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.120:6807/36386", 2021-02-10T11:38:39.256 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.120:6817/36386", 2021-02-10T11:38:39.256 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.120:6811/36386", 2021-02-10T11:38:39.256 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:38:39.256 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:38:39.256 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:38:39.256 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:38:39.256 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:38:39.256 INFO:teuthology.orchestra.run.gibba012.stdout: ], 2021-02-10T11:38:39.256 INFO:teuthology.orchestra.run.gibba012.stdout: "osd_xinfo": [ 2021-02-10T11:38:39.256 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:38:39.257 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 0, 2021-02-10T11:38:39.257 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:38:39.257 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:38:39.257 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:38:39.257 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:38:39.257 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:38:39.257 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:38:39.257 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:38:39.257 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 1, 2021-02-10T11:38:39.257 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:38:39.257 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:38:39.257 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:38:39.258 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:38:39.258 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:38:39.258 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:38:39.258 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:38:39.258 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 2, 2021-02-10T11:38:39.258 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:38:39.258 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:38:39.258 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:38:39.258 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:38:39.258 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:38:39.258 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:38:39.258 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:38:39.258 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 3, 2021-02-10T11:38:39.259 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:38:39.259 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:38:39.259 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:38:39.259 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:38:39.259 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:38:39.259 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:38:39.259 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:38:39.259 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 4, 2021-02-10T11:38:39.259 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:38:39.259 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:38:39.259 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:38:39.259 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:38:39.260 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:38:39.260 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:38:39.260 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:38:39.260 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 5, 2021-02-10T11:38:39.260 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:38:39.260 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:38:39.260 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:38:39.260 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:38:39.260 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:38:39.260 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:38:39.260 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:38:39.260 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 6, 2021-02-10T11:38:39.260 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:38:39.261 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:38:39.261 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:38:39.261 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:38:39.261 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:38:39.261 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:38:39.261 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:38:39.261 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 7, 2021-02-10T11:38:39.261 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:38:39.261 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:38:39.261 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:38:39.261 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:38:39.261 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:38:39.261 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:38:39.262 INFO:teuthology.orchestra.run.gibba012.stdout: ], 2021-02-10T11:38:39.262 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_upmap": [], 2021-02-10T11:38:39.262 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_upmap_items": [], 2021-02-10T11:38:39.262 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_temp": [], 2021-02-10T11:38:39.262 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_temp": [], 2021-02-10T11:38:39.262 INFO:teuthology.orchestra.run.gibba012.stdout: "blacklist": {}, 2021-02-10T11:38:39.262 INFO:teuthology.orchestra.run.gibba012.stdout: "erasure_code_profiles": { 2021-02-10T11:38:39.262 INFO:teuthology.orchestra.run.gibba012.stdout: "default": { 2021-02-10T11:38:39.262 INFO:teuthology.orchestra.run.gibba012.stdout: "crush-failure-domain": "osd", 2021-02-10T11:38:39.262 INFO:teuthology.orchestra.run.gibba012.stdout: "k": "2", 2021-02-10T11:38:39.262 INFO:teuthology.orchestra.run.gibba012.stdout: "m": "1", 2021-02-10T11:38:39.262 INFO:teuthology.orchestra.run.gibba012.stdout: "plugin": "jerasure", 2021-02-10T11:38:39.263 INFO:teuthology.orchestra.run.gibba012.stdout: "ruleset-failure-domain": "osd", 2021-02-10T11:38:39.263 INFO:teuthology.orchestra.run.gibba012.stdout: "technique": "reed_sol_van" 2021-02-10T11:38:39.263 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:38:39.263 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:38:39.263 INFO:teuthology.orchestra.run.gibba012.stdout: "removed_snaps_queue": [], 2021-02-10T11:38:39.263 INFO:teuthology.orchestra.run.gibba012.stdout: "new_removed_snaps": [], 2021-02-10T11:38:39.263 INFO:teuthology.orchestra.run.gibba012.stdout: "new_purged_snaps": [], 2021-02-10T11:38:39.263 INFO:teuthology.orchestra.run.gibba012.stdout: "crush_node_flags": {}, 2021-02-10T11:38:39.263 INFO:teuthology.orchestra.run.gibba012.stdout: "device_class_flags": {} 2021-02-10T11:38:39.263 INFO:teuthology.orchestra.run.gibba012.stdout:} 2021-02-10T11:38:39.265 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.192 7f3e7c8f8700 1 -- 172.21.2.112:0/4280816153 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3e5c0186c0 msgr2=0x7f3e5c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:39.265 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.192 7f3e7c8f8700 1 --2- 172.21.2.112:0/4280816153 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3e5c0186c0 0x7f3e5c01abc0 crc :-1 s=READY pgs=846 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:39.265 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.192 7f3e7c8f8700 1 -- 172.21.2.112:0/4280816153 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3e781a6fa0 msgr2=0x7f3e781a73a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:39.265 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.192 7f3e7c8f8700 1 --2- 172.21.2.112:0/4280816153 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3e781a6fa0 0x7f3e781a73a0 secure :-1 s=READY pgs=1798 cs=0 l=1 rev1=1 rx=0x7f3e70009390 tx=0x7f3e7000bd40).stop 2021-02-10T11:38:39.265 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.192 7f3e7c8f8700 1 -- 172.21.2.112:0/4280816153 shutdown_connections 2021-02-10T11:38:39.265 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.192 7f3e7c8f8700 1 --2- 172.21.2.112:0/4280816153 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3e5c0186c0 0x7f3e5c01abc0 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:39.265 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.192 7f3e7c8f8700 1 --2- 172.21.2.112:0/4280816153 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3e781a8850 0x7f3e781a8c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:39.265 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.192 7f3e7c8f8700 1 --2- 172.21.2.112:0/4280816153 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3e781a6fa0 0x7f3e781a73a0 unknown :-1 s=CLOSED pgs=1798 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:39.266 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.192 7f3e7c8f8700 1 --2- 172.21.2.112:0/4280816153 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3e780b0ad0 0x7f3e78091f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:39.266 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.192 7f3e7c8f8700 1 -- 172.21.2.112:0/4280816153 >> 172.21.2.112:0/4280816153 conn(0x7f3e7819f8f0 msgr2=0x7f3e781a5f80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:39.266 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.192 7f3e7c8f8700 1 -- 172.21.2.112:0/4280816153 shutdown_connections 2021-02-10T11:38:39.266 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.192 7f3e7c8f8700 1 -- 172.21.2.112:0/4280816153 wait complete. 2021-02-10T11:38:39.267 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:38:39.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.685 7f1e664cf700 1 Processor -- start 2021-02-10T11:38:39.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.685 7f1e664cf700 1 -- start start 2021-02-10T11:38:39.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.685 7f1e664cf700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f1e601b0470 0x7f1e601b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:39.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.685 7f1e664cf700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1e600f20e0 0x7f1e600f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:39.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.685 7f1e664cf700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e600f0b50 con 0x7f1e600ef610 2021-02-10T11:38:39.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.685 7f1e664cf700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e600f0820 con 0x7f1e601b0470 2021-02-10T11:38:39.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.685 7f1e664cf700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e600f24e0 con 0x7f1e600f20e0 2021-02-10T11:38:39.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.686 7f1e64aec700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1e600f20e0 0x7f1e600f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:39.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.686 7f1e64aec700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1e600f20e0 0x7f1e600f0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3301/0 says I am v2:172.21.2.112:52738/0 (socket says 172.21.2.112:52738) 2021-02-10T11:38:39.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.686 7f1e5f7fe700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7f1e600ef610 0x7f1e600efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:46076/0 (socket says 172.21.2.112:46076) 2021-02-10T11:38:39.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.686 7f1e64aec700 1 -- 172.21.2.112:0/1459166517 learned_addr learned my addr 172.21.2.112:0/1459166517 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:38:39.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.686 7f1e5ffff700 1 --2- 172.21.2.112:0/1459166517 >> v2:172.21.2.120:3300/0 conn(0x7f1e601b0470 0x7f1e601b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:39.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.686 7f1e64aec700 1 -- 172.21.2.112:0/1459166517 >> v2:172.21.2.120:3300/0 conn(0x7f1e601b0470 msgr2=0x7f1e601b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:39.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.686 7f1e64aec700 1 --2- 172.21.2.112:0/1459166517 >> v2:172.21.2.120:3300/0 conn(0x7f1e601b0470 0x7f1e601b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:39.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.686 7f1e64aec700 1 -- 172.21.2.112:0/1459166517 >> v1:172.21.2.120:6789/0 conn(0x7f1e600ef610 legacy=0x7f1e600efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:39.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.686 7f1e64aec700 1 -- 172.21.2.112:0/1459166517 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e600e52e0 con 0x7f1e600f20e0 2021-02-10T11:38:39.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.686 7f1e64aec700 1 --2- 172.21.2.112:0/1459166517 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1e600f20e0 0x7f1e600f0200 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f1e58006aa0 tx=0x7f1e58009f20).ready entity=mon.2 client_cookie=be59824053897896 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:39.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.687 7f1e5effd700 1 -- 172.21.2.112:0/1459166517 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1e5800b5c0 con 0x7f1e600f20e0 2021-02-10T11:38:39.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.687 7f1e5effd700 1 -- 172.21.2.112:0/1459166517 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e58003030 con 0x7f1e600f20e0 2021-02-10T11:38:39.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.687 7f1e5effd700 1 -- 172.21.2.112:0/1459166517 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1e5800bab0 con 0x7f1e600f20e0 2021-02-10T11:38:39.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.687 7f1e664cf700 1 -- 172.21.2.112:0/1459166517 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1e600f20e0 msgr2=0x7f1e600f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:39.695 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.687 7f1e664cf700 1 --2- 172.21.2.112:0/1459166517 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1e600f20e0 0x7f1e600f0200 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f1e58006aa0 tx=0x7f1e58009f20).stop 2021-02-10T11:38:39.695 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.687 7f1e664cf700 1 -- 172.21.2.112:0/1459166517 shutdown_connections 2021-02-10T11:38:39.695 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.687 7f1e664cf700 1 --2- 172.21.2.112:0/1459166517 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1e600f20e0 0x7f1e600f0200 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:39.695 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.687 7f1e664cf700 1 --2- 172.21.2.112:0/1459166517 >> v2:172.21.2.120:3300/0 conn(0x7f1e601b0470 0x7f1e601b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:39.695 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.687 7f1e664cf700 1 -- 172.21.2.112:0/1459166517 >> 172.21.2.112:0/1459166517 conn(0x7f1e600ce3e0 msgr2=0x7f1e600e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:39.695 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.687 7f1e664cf700 1 -- 172.21.2.112:0/1459166517 shutdown_connections 2021-02-10T11:38:39.695 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.687 7f1e664cf700 1 -- 172.21.2.112:0/1459166517 wait complete. 2021-02-10T11:38:39.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.702 7f1e664cf700 1 Processor -- start 2021-02-10T11:38:39.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.702 7f1e664cf700 1 -- start start 2021-02-10T11:38:39.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.702 7f1e664cf700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1e6008ebb0 0x7f1e6008efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:39.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.702 7f1e664cf700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1e600b65c0 0x7f1e600b69c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:39.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.702 7f1e664cf700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1e600b6fe0 0x7f1e600b7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:39.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.702 7f1e664cf700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e600b76d0 con 0x7f1e600b6fe0 2021-02-10T11:38:39.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.702 7f1e664cf700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e6008f5d0 con 0x7f1e6008ebb0 2021-02-10T11:38:39.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.702 7f1e664cf700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e601a04d0 con 0x7f1e600b65c0 2021-02-10T11:38:39.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.702 7f1e64aec700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1e600b6fe0 0x7f1e600b7bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:39.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.702 7f1e64aec700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1e600b6fe0 0x7f1e600b7bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:53060/0 (socket says 172.21.2.112:53060) 2021-02-10T11:38:39.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.702 7f1e64aec700 1 -- 172.21.2.112:0/3512241544 learned_addr learned my addr 172.21.2.112:0/3512241544 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:38:39.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.702 7f1e64aec700 1 -- 172.21.2.112:0/3512241544 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1e600b65c0 msgr2=0x7f1e600b69c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:38:39.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.702 7f1e64aec700 1 --2- 172.21.2.112:0/3512241544 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1e600b65c0 0x7f1e600b69c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:39.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.703 7f1e64aec700 1 -- 172.21.2.112:0/3512241544 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1e6008ebb0 msgr2=0x7f1e6008efb0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:38:39.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.703 7f1e64aec700 1 --2- 172.21.2.112:0/3512241544 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1e6008ebb0 0x7f1e6008efb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:39.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.703 7f1e64aec700 1 -- 172.21.2.112:0/3512241544 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e600b81f0 con 0x7f1e600b6fe0 2021-02-10T11:38:39.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.703 7f1e64aec700 1 --2- 172.21.2.112:0/3512241544 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1e600b6fe0 0x7f1e600b7bd0 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 rx=0x7f1e5800b970 tx=0x7f1e5800bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:39.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.703 7f1e5cff9700 1 -- 172.21.2.112:0/3512241544 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1e58009450 con 0x7f1e600b6fe0 2021-02-10T11:38:39.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.703 7f1e5cff9700 1 -- 172.21.2.112:0/3512241544 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e5800c2f0 con 0x7f1e600b6fe0 2021-02-10T11:38:39.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.703 7f1e5cff9700 1 -- 172.21.2.112:0/3512241544 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1e5800c810 con 0x7f1e600b6fe0 2021-02-10T11:38:39.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.703 7f1e664cf700 1 -- 172.21.2.112:0/3512241544 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e600b81f0 con 0x7f1e600b6fe0 2021-02-10T11:38:39.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.703 7f1e664cf700 1 -- 172.21.2.112:0/3512241544 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e600b81f0 con 0x7f1e600b6fe0 2021-02-10T11:38:39.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.704 7f1e5cff9700 1 -- 172.21.2.112:0/3512241544 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1e58009fb0 con 0x7f1e600b6fe0 2021-02-10T11:38:39.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.704 7f1e5cff9700 1 --2- 172.21.2.112:0/3512241544 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1e400186c0 0x7f1e4001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:39.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.704 7f1e5cff9700 1 -- 172.21.2.112:0/3512241544 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f1e580328e0 con 0x7f1e600b6fe0 2021-02-10T11:38:39.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.704 7f1e5ffff700 1 --2- 172.21.2.112:0/3512241544 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1e400186c0 0x7f1e4001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:39.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.705 7f1e5ffff700 1 --2- 172.21.2.112:0/3512241544 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1e400186c0 0x7f1e4001abc0 crc :-1 s=READY pgs=847 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:39.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.735 7f1e664cf700 1 -- 172.21.2.112:0/3512241544 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e60105a20 con 0x7f1e600b6fe0 2021-02-10T11:38:39.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.738 7f1e5cff9700 1 -- 172.21.2.112:0/3512241544 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1e58000df0 con 0x7f1e600b6fe0 2021-02-10T11:38:39.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.868 7f1e664cf700 1 -- 172.21.2.112:0/3512241544 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1e600f0b50 con 0x7f1e600b6fe0 2021-02-10T11:38:39.860 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.868 7f1e5cff9700 1 -- 172.21.2.112:0/3512241544 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) v1 ==== 94+0+2101 (secure 0 0 0) 0x7f1e58009fb0 con 0x7f1e600b6fe0 2021-02-10T11:38:39.860 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 65 2021-02-10T11:38:39.861 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:38:39.861 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":65,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9204,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/1512510827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":1512510827},{"type":"v1","addr":"172.21.2.112:6833","nonce":1512510827}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":60}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:36:53.563735","modified":"2021-02-10 11:36:55.573740","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9201},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9201":{"gid":9201,"name":"b","rank":0,"incarnation":61,"state":"up:active","state_seq":3,"addr":"172.21.2.120:6835/1232001750","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":1232001750},{"type":"v1","addr":"172.21.2.120:6835","nonce":1232001750}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2021-02-10T11:38:39.861 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.870 7f1e664cf700 1 -- 172.21.2.112:0/3512241544 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1e400186c0 msgr2=0x7f1e4001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:39.861 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.870 7f1e664cf700 1 --2- 172.21.2.112:0/3512241544 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1e400186c0 0x7f1e4001abc0 crc :-1 s=READY pgs=847 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:39.861 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.870 7f1e664cf700 1 -- 172.21.2.112:0/3512241544 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1e600b6fe0 msgr2=0x7f1e600b7bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:39.862 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.870 7f1e664cf700 1 --2- 172.21.2.112:0/3512241544 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1e600b6fe0 0x7f1e600b7bd0 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 rx=0x7f1e5800b970 tx=0x7f1e5800bd10).stop 2021-02-10T11:38:39.862 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.870 7f1e664cf700 1 -- 172.21.2.112:0/3512241544 shutdown_connections 2021-02-10T11:38:39.862 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.870 7f1e664cf700 1 --2- 172.21.2.112:0/3512241544 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1e400186c0 0x7f1e4001abc0 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:39.862 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.870 7f1e664cf700 1 --2- 172.21.2.112:0/3512241544 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1e600b6fe0 0x7f1e600b7bd0 unknown :-1 s=CLOSED pgs=1799 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:39.862 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.870 7f1e664cf700 1 --2- 172.21.2.112:0/3512241544 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1e600b65c0 0x7f1e600b69c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:39.862 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.870 7f1e664cf700 1 --2- 172.21.2.112:0/3512241544 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1e6008ebb0 0x7f1e6008efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:39.862 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.870 7f1e664cf700 1 -- 172.21.2.112:0/3512241544 >> 172.21.2.112:0/3512241544 conn(0x7f1e6009cb20 msgr2=0x7f1e600c3c80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:39.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.870 7f1e664cf700 1 -- 172.21.2.112:0/3512241544 shutdown_connections 2021-02-10T11:38:39.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:39.870 7f1e664cf700 1 -- 172.21.2.112:0/3512241544 wait complete. 2021-02-10T11:38:39.870 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs fail cephfs 2021-02-10T11:38:40.297 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:40.290 7f3fcf877700 1 Processor -- start 2021-02-10T11:38:40.297 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:40.290 7f3fcf877700 1 -- start start 2021-02-10T11:38:40.297 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:40.290 7f3fcf877700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f3fc80d0340 0x7f3fc80ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:40.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:40.290 7f3fcf877700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3fc80d1c70 0x7f3fc80efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:40.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:40.290 7f3fcf877700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fc80f0b10 con 0x7f3fc80d1870 2021-02-10T11:38:40.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:40.290 7f3fcf877700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fc80e5ed0 con 0x7f3fc80d0340 2021-02-10T11:38:40.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:40.290 7f3fcf877700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fc81b0be0 con 0x7f3fc80d1c70 2021-02-10T11:38:40.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:40.291 7f3fcce92700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f3fc80d0340 0x7f3fc80ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:40.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:40.291 7f3fcce92700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f3fc80d0340 0x7f3fc80ef640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:53068/0 (socket says 172.21.2.112:53068) 2021-02-10T11:38:40.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:40.291 7f3fcce92700 1 -- 172.21.2.112:0/2042763539 learned_addr learned my addr 172.21.2.112:0/2042763539 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:38:40.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:40.291 7f3fcce92700 1 -- 172.21.2.112:0/2042763539 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3fc80d1c70 msgr2=0x7f3fc80efc60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:38:40.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:40.291 7f3fcce92700 1 --2- 172.21.2.112:0/2042763539 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3fc80d1c70 0x7f3fc80efc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:40.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:40.291 7f3fcce92700 1 -- 172.21.2.112:0/2042763539 >> v1:172.21.2.120:6789/0 conn(0x7f3fc80d1870 legacy=0x7f3fc81b03f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:38:40.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:40.291 7f3fcce92700 1 -- 172.21.2.112:0/2042763539 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fc80f20a0 con 0x7f3fc80d0340 2021-02-10T11:38:40.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:40.291 7f3fcce92700 1 --2- 172.21.2.112:0/2042763539 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3fc80d0340 0x7f3fc80ef640 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 rx=0x7f3fbc006700 tx=0x7f3fbc009b80).ready entity=mon.0 client_cookie=db3d3893b3a98828 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:40.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:40.291 7f3fbbfff700 1 -- 172.21.2.112:0/2042763539 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3fbc00c0c0 con 0x7f3fc80d0340 2021-02-10T11:38:40.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:40.291 7f3fbbfff700 1 -- 172.21.2.112:0/2042763539 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fbc00c340 con 0x7f3fc80d0340 2021-02-10T11:38:40.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:40.291 7f3fbbfff700 1 -- 172.21.2.112:0/2042763539 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3fbc00c860 con 0x7f3fc80d0340 2021-02-10T11:38:40.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:40.291 7f3fcf877700 1 -- 172.21.2.112:0/2042763539 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3fc80d0340 msgr2=0x7f3fc80ef640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:40.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:40.291 7f3fcf877700 1 --2- 172.21.2.112:0/2042763539 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3fc80d0340 0x7f3fc80ef640 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 rx=0x7f3fbc006700 tx=0x7f3fbc009b80).stop 2021-02-10T11:38:40.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:40.292 7f3fcf877700 1 -- 172.21.2.112:0/2042763539 shutdown_connections 2021-02-10T11:38:40.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:40.292 7f3fcf877700 1 --2- 172.21.2.112:0/2042763539 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3fc80d0340 0x7f3fc80ef640 unknown :-1 s=CLOSED pgs=1800 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:40.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:40.292 7f3fcf877700 1 --2- 172.21.2.112:0/2042763539 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3fc80d1c70 0x7f3fc80efc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:40.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:40.292 7f3fcf877700 1 -- 172.21.2.112:0/2042763539 >> 172.21.2.112:0/2042763539 conn(0x7f3fc80ce450 msgr2=0x7f3fc80d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:40.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:40.292 7f3fcf877700 1 -- 172.21.2.112:0/2042763539 shutdown_connections 2021-02-10T11:38:40.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:40.292 7f3fcf877700 1 -- 172.21.2.112:0/2042763539 wait complete. 2021-02-10T11:38:40.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:40.306 7f3fcf877700 1 Processor -- start 2021-02-10T11:38:40.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:40.306 7f3fcf877700 1 -- start start 2021-02-10T11:38:40.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:40.307 7f3fcf877700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3fc80a5ef0 0x7f3fc8188f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:40.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:40.307 7f3fcf877700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3fc80a62f0 0x7f3fc8189570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:40.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:40.307 7f3fcf877700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3fc8189b90 0x7f3fc808cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:40.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:40.307 7f3fcf877700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fc808c360 con 0x7f3fc80a5ef0 2021-02-10T11:38:40.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:40.307 7f3fcf877700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fc8189f90 con 0x7f3fc8189b90 2021-02-10T11:38:40.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:40.307 7f3fcf877700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fc808d1b0 con 0x7f3fc80a62f0 2021-02-10T11:38:40.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:40.307 7f3fcd693700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3fc80a5ef0 0x7f3fc8188f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:40.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:40.307 7f3fcd693700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3fc80a5ef0 0x7f3fc8188f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:53078/0 (socket says 172.21.2.112:53078) 2021-02-10T11:38:40.302 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:40.307 7f3fcd693700 1 -- 172.21.2.112:0/1118257863 learned_addr learned my addr 172.21.2.112:0/1118257863 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:38:40.302 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:40.307 7f3fcd693700 1 -- 172.21.2.112:0/1118257863 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3fc80a62f0 msgr2=0x7f3fc8189570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:40.302 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:40.307 7f3fcd693700 1 --2- 172.21.2.112:0/1118257863 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3fc80a62f0 0x7f3fc8189570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:40.302 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:40.307 7f3fcd693700 1 -- 172.21.2.112:0/1118257863 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3fc8189b90 msgr2=0x7f3fc808cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:40.302 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:40.307 7f3fcd693700 1 --2- 172.21.2.112:0/1118257863 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3fc8189b90 0x7f3fc808cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:40.302 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:40.307 7f3fcd693700 1 -- 172.21.2.112:0/1118257863 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fc808d3d0 con 0x7f3fc80a5ef0 2021-02-10T11:38:40.302 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:40.307 7f3fcd693700 1 --2- 172.21.2.112:0/1118257863 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3fc80a5ef0 0x7f3fc8188f50 secure :-1 s=READY pgs=1801 cs=0 l=1 rev1=1 rx=0x7f3fc400b5b0 tx=0x7f3fc400b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:40.302 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:40.308 7f3fb9ffb700 1 -- 172.21.2.112:0/1118257863 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3fc400bd40 con 0x7f3fc80a5ef0 2021-02-10T11:38:40.302 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:40.308 7f3fb9ffb700 1 -- 172.21.2.112:0/1118257863 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fc4006240 con 0x7f3fc80a5ef0 2021-02-10T11:38:40.302 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:40.308 7f3fcf877700 1 -- 172.21.2.112:0/1118257863 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fc808d3d0 con 0x7f3fc80a5ef0 2021-02-10T11:38:40.302 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:40.308 7f3fcf877700 1 -- 172.21.2.112:0/1118257863 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fc808d960 con 0x7f3fc80a5ef0 2021-02-10T11:38:40.302 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:40.308 7f3fb9ffb700 1 -- 172.21.2.112:0/1118257863 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3fc4005720 con 0x7f3fc80a5ef0 2021-02-10T11:38:40.302 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:40.308 7f3fb9ffb700 1 -- 172.21.2.112:0/1118257863 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3fc40054a0 con 0x7f3fc80a5ef0 2021-02-10T11:38:40.303 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:40.308 7f3fb9ffb700 1 --2- 172.21.2.112:0/1118257863 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3fa80186c0 0x7f3fa801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:40.303 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:40.308 7f3fb9ffb700 1 -- 172.21.2.112:0/1118257863 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f3fc40325c0 con 0x7f3fc80a5ef0 2021-02-10T11:38:40.303 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:40.309 7f3fcce92700 1 --2- 172.21.2.112:0/1118257863 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3fa80186c0 0x7f3fa801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:40.303 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:40.310 7f3fcce92700 1 --2- 172.21.2.112:0/1118257863 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3fa80186c0 0x7f3fa801abc0 crc :-1 s=READY pgs=848 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:40.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:40.341 7f3fcf877700 1 -- 172.21.2.112:0/1118257863 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fc80ee0b0 con 0x7f3fc80a5ef0 2021-02-10T11:38:40.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:40.343 7f3fb9ffb700 1 -- 172.21.2.112:0/1118257863 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3fc401b5b0 con 0x7f3fc80a5ef0 2021-02-10T11:38:40.461 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:40.470 7f3fcf877700 1 -- 172.21.2.112:0/1118257863 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f3fc80ce450 con 0x7f3fc80a5ef0 2021-02-10T11:38:41.241 INFO:tasks.ceph.mon.a.gibba012.stderr:2021-02-10 11:38:41.250 7fc3fe6e2700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-02-10T11:38:41.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:41.255 7f3fb9ffb700 1 -- 172.21.2.112:0/1118257863 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v66) v1 ==== 161+0+0 (secure 0 0 0) 0x7f3fc401b5b0 con 0x7f3fc80a5ef0 2021-02-10T11:38:41.282 INFO:teuthology.orchestra.run.gibba012.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2021-02-10T11:38:41.283 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:41.292 7f3fcf877700 1 -- 172.21.2.112:0/1118257863 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3fa80186c0 msgr2=0x7f3fa801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:41.283 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:41.292 7f3fcf877700 1 --2- 172.21.2.112:0/1118257863 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3fa80186c0 0x7f3fa801abc0 crc :-1 s=READY pgs=848 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:41.283 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:41.292 7f3fcf877700 1 -- 172.21.2.112:0/1118257863 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3fc80a5ef0 msgr2=0x7f3fc8188f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:41.283 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:41.292 7f3fcf877700 1 --2- 172.21.2.112:0/1118257863 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3fc80a5ef0 0x7f3fc8188f50 secure :-1 s=READY pgs=1801 cs=0 l=1 rev1=1 rx=0x7f3fc400b5b0 tx=0x7f3fc400b950).stop 2021-02-10T11:38:41.284 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:41.292 7f3fcf877700 1 -- 172.21.2.112:0/1118257863 shutdown_connections 2021-02-10T11:38:41.284 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:41.292 7f3fcf877700 1 --2- 172.21.2.112:0/1118257863 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3fa80186c0 0x7f3fa801abc0 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:41.284 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:41.292 7f3fcf877700 1 --2- 172.21.2.112:0/1118257863 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3fc8189b90 0x7f3fc808cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:41.284 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:41.292 7f3fcf877700 1 --2- 172.21.2.112:0/1118257863 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3fc80a62f0 0x7f3fc8189570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:41.284 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:41.292 7f3fcf877700 1 --2- 172.21.2.112:0/1118257863 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3fc80a5ef0 0x7f3fc8188f50 unknown :-1 s=CLOSED pgs=1801 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:41.284 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:41.292 7f3fcf877700 1 -- 172.21.2.112:0/1118257863 >> 172.21.2.112:0/1118257863 conn(0x7f3fc80a6eb0 msgr2=0x7f3fc8098ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:41.284 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:41.292 7f3fcf877700 1 -- 172.21.2.112:0/1118257863 shutdown_connections 2021-02-10T11:38:41.284 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:41.292 7f3fcf877700 1 -- 172.21.2.112:0/1118257863 wait complete. 2021-02-10T11:38:41.292 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:38:41.294 INFO:tasks.ceph.mds.b.gibba020.stdout:starting mds.b at 2021-02-10T11:38:41.718 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:41.710 7f61ef215700 1 Processor -- start 2021-02-10T11:38:41.718 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:41.710 7f61ef215700 1 -- start start 2021-02-10T11:38:41.718 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:41.710 7f61ef215700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f61e81b0870 0x7f61e80f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:41.718 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:41.710 7f61ef215700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61e81b0c70 con 0x7f61e80efe00 2021-02-10T11:38:41.718 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:41.710 7f61ef215700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61e80f0b50 con 0x7f61e81b0470 2021-02-10T11:38:41.718 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:41.710 7f61ef215700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61e80e5ea0 con 0x7f61e81b0870 2021-02-10T11:38:41.718 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:41.711 7f61ec830700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f61e81b0870 0x7f61e80f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:41.718 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:41.711 7f61ed832700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f61e80efe00 0x7f61e80f0200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:52546/0 (socket says 172.21.2.112:52546) 2021-02-10T11:38:41.718 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:41.711 7f61ec830700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f61e81b0870 0x7f61e80f20e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:53082/0 (socket says 172.21.2.112:53082) 2021-02-10T11:38:41.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:41.711 7f61ed832700 1 -- 172.21.2.112:0/2730755441 learned_addr learned my addr 172.21.2.112:0/2730755441 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:38:41.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:41.711 7f61ec830700 1 -- 172.21.2.112:0/2730755441 >> v1:172.21.2.120:6789/0 conn(0x7f61e81b0470 legacy=0x7f61e80ef610 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:38:41.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:41.711 7f61ec830700 1 -- 172.21.2.112:0/2730755441 >> v1:172.21.2.112:6789/0 conn(0x7f61e80efe00 legacy=0x7f61e80f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:41.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:41.711 7f61ec830700 1 -- 172.21.2.112:0/2730755441 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61e80e52e0 con 0x7f61e81b0870 2021-02-10T11:38:41.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:41.711 7f61ec830700 1 --2- 172.21.2.112:0/2730755441 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f61e81b0870 0x7f61e80f20e0 secure :-1 s=READY pgs=1803 cs=0 l=1 rev1=1 rx=0x7f61e4006650 tx=0x7f61e4009a80).ready entity=mon.0 client_cookie=31063faa5e635c5f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:41.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:41.711 7f61dbfff700 1 -- 172.21.2.112:0/2730755441 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f61e400c0c0 con 0x7f61e81b0870 2021-02-10T11:38:41.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:41.711 7f61dbfff700 1 -- 172.21.2.112:0/2730755441 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61e400c340 con 0x7f61e81b0870 2021-02-10T11:38:41.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:41.711 7f61dbfff700 1 -- 172.21.2.112:0/2730755441 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f61e400c820 con 0x7f61e81b0870 2021-02-10T11:38:41.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:41.711 7f61ef215700 1 -- 172.21.2.112:0/2730755441 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f61e81b0870 msgr2=0x7f61e80f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:41.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:41.711 7f61ef215700 1 --2- 172.21.2.112:0/2730755441 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f61e81b0870 0x7f61e80f20e0 secure :-1 s=READY pgs=1803 cs=0 l=1 rev1=1 rx=0x7f61e4006650 tx=0x7f61e4009a80).stop 2021-02-10T11:38:41.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:41.712 7f61ef215700 1 -- 172.21.2.112:0/2730755441 shutdown_connections 2021-02-10T11:38:41.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:41.712 7f61ef215700 1 --2- 172.21.2.112:0/2730755441 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f61e81b0870 0x7f61e80f20e0 unknown :-1 s=CLOSED pgs=1803 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:41.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:41.712 7f61ef215700 1 -- 172.21.2.112:0/2730755441 >> 172.21.2.112:0/2730755441 conn(0x7f61e80ce3e0 msgr2=0x7f61e80e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:41.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:41.712 7f61ef215700 1 -- 172.21.2.112:0/2730755441 shutdown_connections 2021-02-10T11:38:41.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:41.712 7f61ef215700 1 -- 172.21.2.112:0/2730755441 wait complete. 2021-02-10T11:38:41.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:41.726 7f61ef215700 1 Processor -- start 2021-02-10T11:38:41.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:41.727 7f61ef215700 1 -- start start 2021-02-10T11:38:41.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:41.727 7f61ef215700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f61e80ba100 0x7f61e80b24f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:41.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:41.727 7f61ef215700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f61e8088080 0x7f61e8088480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:41.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:41.727 7f61ef215700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f61e80bae10 0x7f61e80bb210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:41.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:41.727 7f61ef215700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61e80ba890 con 0x7f61e80ba100 2021-02-10T11:38:41.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:41.727 7f61ef215700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61e80b9740 con 0x7f61e8088080 2021-02-10T11:38:41.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:41.727 7f61ef215700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61e80ba500 con 0x7f61e80bae10 2021-02-10T11:38:41.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:41.727 7f61ed031700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f61e80ba100 0x7f61e80b24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:41.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:41.727 7f61ed031700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f61e80ba100 0x7f61e80b24f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:53086/0 (socket says 172.21.2.112:53086) 2021-02-10T11:38:41.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:41.727 7f61ed031700 1 -- 172.21.2.112:0/47561366 learned_addr learned my addr 172.21.2.112:0/47561366 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:38:41.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:41.728 7f61ed832700 1 --2- 172.21.2.112:0/47561366 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f61e80bae10 0x7f61e80bb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:41.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:41.728 7f61ec830700 1 --2- 172.21.2.112:0/47561366 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f61e8088080 0x7f61e8088480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:41.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:41.728 7f61ed031700 1 -- 172.21.2.112:0/47561366 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f61e80bae10 msgr2=0x7f61e80bb210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:41.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:41.728 7f61ed031700 1 --2- 172.21.2.112:0/47561366 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f61e80bae10 0x7f61e80bb210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:41.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:41.728 7f61ed031700 1 -- 172.21.2.112:0/47561366 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f61e8088080 msgr2=0x7f61e8088480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:41.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:41.728 7f61ed031700 1 --2- 172.21.2.112:0/47561366 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f61e8088080 0x7f61e8088480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:41.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:41.728 7f61ed031700 1 -- 172.21.2.112:0/47561366 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61e80bb830 con 0x7f61e80ba100 2021-02-10T11:38:41.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:41.728 7f61ed031700 1 --2- 172.21.2.112:0/47561366 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f61e80ba100 0x7f61e80b24f0 secure :-1 s=READY pgs=1804 cs=0 l=1 rev1=1 rx=0x7f61dc00b5b0 tx=0x7f61dc00b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:41.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:41.728 7f61d9ffb700 1 -- 172.21.2.112:0/47561366 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f61dc00bd40 con 0x7f61e80ba100 2021-02-10T11:38:41.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:41.728 7f61d9ffb700 1 -- 172.21.2.112:0/47561366 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61dc006020 con 0x7f61e80ba100 2021-02-10T11:38:41.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:41.728 7f61d9ffb700 1 -- 172.21.2.112:0/47561366 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f61dc0054a0 con 0x7f61e80ba100 2021-02-10T11:38:41.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:41.728 7f61ef215700 1 -- 172.21.2.112:0/47561366 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61e80bb830 con 0x7f61e80ba100 2021-02-10T11:38:41.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:41.729 7f61ef215700 1 -- 172.21.2.112:0/47561366 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61e80bb830 con 0x7f61e80ba100 2021-02-10T11:38:41.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:41.729 7f61d9ffb700 1 -- 172.21.2.112:0/47561366 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f61dc006200 con 0x7f61e80ba100 2021-02-10T11:38:41.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:41.729 7f61d9ffb700 1 --2- 172.21.2.112:0/47561366 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f61c80186c0 0x7f61c801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:41.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:41.729 7f61d9ffb700 1 -- 172.21.2.112:0/47561366 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5659+0+0 (secure 0 0 0) 0x7f61dc0325d0 con 0x7f61e80ba100 2021-02-10T11:38:41.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:41.730 7f61ec830700 1 --2- 172.21.2.112:0/47561366 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f61c80186c0 0x7f61c801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:41.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:41.730 7f61ec830700 1 --2- 172.21.2.112:0/47561366 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f61c80186c0 0x7f61c801abc0 crc :-1 s=READY pgs=849 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:41.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:41.760 7f61ef215700 1 -- 172.21.2.112:0/47561366 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61e807ed60 con 0x7f61e80ba100 2021-02-10T11:38:41.754 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:41.763 7f61d9ffb700 1 -- 172.21.2.112:0/47561366 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f61dc030d30 con 0x7f61e80ba100 2021-02-10T11:38:41.885 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:41.894 7f61ef215700 1 -- 172.21.2.112:0/47561366 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f61e807ed60 con 0x7f61e80ba100 2021-02-10T11:38:41.886 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:41.894 7f61d9ffb700 1 -- 172.21.2.112:0/47561366 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+1756 (secure 0 0 0) 0x7f61dc00bd40 con 0x7f61e80ba100 2021-02-10T11:38:41.886 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 66 2021-02-10T11:38:41.886 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:38:41.887 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":66,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9204,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/1512510827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":1512510827},{"type":"v1","addr":"172.21.2.112:6833","nonce":1512510827}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":60}],"filesystems":[{"mdsmap":{"epoch":66,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:36:53.563735","modified":"2021-02-10 11:38:41.250851","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2021-02-10T11:38:41.888 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:41.896 7f61ef215700 1 -- 172.21.2.112:0/47561366 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f61c80186c0 msgr2=0x7f61c801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:41.888 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:41.896 7f61ef215700 1 --2- 172.21.2.112:0/47561366 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f61c80186c0 0x7f61c801abc0 crc :-1 s=READY pgs=849 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:41.888 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:41.896 7f61ef215700 1 -- 172.21.2.112:0/47561366 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f61e80ba100 msgr2=0x7f61e80b24f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:41.888 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:41.896 7f61ef215700 1 --2- 172.21.2.112:0/47561366 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f61e80ba100 0x7f61e80b24f0 secure :-1 s=READY pgs=1804 cs=0 l=1 rev1=1 rx=0x7f61dc00b5b0 tx=0x7f61dc00b950).stop 2021-02-10T11:38:41.888 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:41.896 7f61ef215700 1 -- 172.21.2.112:0/47561366 shutdown_connections 2021-02-10T11:38:41.888 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:41.896 7f61ef215700 1 --2- 172.21.2.112:0/47561366 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f61c80186c0 0x7f61c801abc0 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:41.889 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:41.896 7f61ef215700 1 --2- 172.21.2.112:0/47561366 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f61e8088080 0x7f61e8088480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:41.889 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:41.896 7f61ef215700 1 --2- 172.21.2.112:0/47561366 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f61e80bae10 0x7f61e80bb210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:41.889 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:41.896 7f61ef215700 1 --2- 172.21.2.112:0/47561366 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f61e80ba100 0x7f61e80b24f0 unknown :-1 s=CLOSED pgs=1804 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:41.889 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:41.896 7f61ef215700 1 -- 172.21.2.112:0/47561366 >> 172.21.2.112:0/47561366 conn(0x7f61e8080c00 msgr2=0x7f61e8082520 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:41.889 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:41.897 7f61ef215700 1 -- 172.21.2.112:0/47561366 shutdown_connections 2021-02-10T11:38:41.889 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:41.897 7f61ef215700 1 -- 172.21.2.112:0/47561366 wait complete. 2021-02-10T11:38:41.897 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2021-02-10T11:38:42.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:42.321 7fd494d48700 1 Processor -- start 2021-02-10T11:38:42.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:42.321 7fd494d48700 1 -- start start 2021-02-10T11:38:42.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:42.322 7fd494d48700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fd4900d1870 0x7fd4901b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:42.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:42.322 7fd494d48700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd4901b0a10 0x7fd4900cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:42.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:42.322 7fd494d48700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4900f0b10 con 0x7fd4900d1c70 2021-02-10T11:38:42.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:42.322 7fd494d48700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4900e5ed0 con 0x7fd4900d1870 2021-02-10T11:38:42.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:42.322 7fd494d48700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4900f0650 con 0x7fd4901b0a10 2021-02-10T11:38:42.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:42.322 7fd48eb13700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fd4900d1870 0x7fd4901b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:42.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:42.322 7fd48eb13700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fd4900d1870 0x7fd4901b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:53094/0 (socket says 172.21.2.112:53094) 2021-02-10T11:38:42.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:42.322 7fd48eb13700 1 -- 172.21.2.112:0/679258361 learned_addr learned my addr 172.21.2.112:0/679258361 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:38:42.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:42.322 7fd48eb13700 1 -- 172.21.2.112:0/679258361 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd4901b0a10 msgr2=0x7fd4900cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:42.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:42.322 7fd48eb13700 1 --2- 172.21.2.112:0/679258361 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd4901b0a10 0x7fd4900cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:42.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:42.322 7fd48eb13700 1 -- 172.21.2.112:0/679258361 >> v1:172.21.2.120:6789/0 conn(0x7fd4900d1c70 legacy=0x7fd4900cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:42.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:42.322 7fd48eb13700 1 -- 172.21.2.112:0/679258361 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4900f0870 con 0x7fd4900d1870 2021-02-10T11:38:42.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:42.322 7fd48eb13700 1 --2- 172.21.2.112:0/679258361 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd4900d1870 0x7fd4901b03f0 secure :-1 s=READY pgs=1805 cs=0 l=1 rev1=1 rx=0x7fd480006700 tx=0x7fd480009b80).ready entity=mon.0 client_cookie=e14994f346812077 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:42.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:42.322 7fd48db11700 1 -- 172.21.2.112:0/679258361 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd48000c0c0 con 0x7fd4900d1870 2021-02-10T11:38:42.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:42.323 7fd48db11700 1 -- 172.21.2.112:0/679258361 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd48000c440 con 0x7fd4900d1870 2021-02-10T11:38:42.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:42.323 7fd48db11700 1 -- 172.21.2.112:0/679258361 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd48000c8a0 con 0x7fd4900d1870 2021-02-10T11:38:42.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:42.323 7fd494d48700 1 -- 172.21.2.112:0/679258361 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd4900d1870 msgr2=0x7fd4901b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:42.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:42.323 7fd494d48700 1 --2- 172.21.2.112:0/679258361 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd4900d1870 0x7fd4901b03f0 secure :-1 s=READY pgs=1805 cs=0 l=1 rev1=1 rx=0x7fd480006700 tx=0x7fd480009b80).stop 2021-02-10T11:38:42.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:42.323 7fd494d48700 1 -- 172.21.2.112:0/679258361 shutdown_connections 2021-02-10T11:38:42.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:42.323 7fd494d48700 1 --2- 172.21.2.112:0/679258361 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd4901b0a10 0x7fd4900cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:42.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:42.323 7fd494d48700 1 --2- 172.21.2.112:0/679258361 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd4900d1870 0x7fd4901b03f0 unknown :-1 s=CLOSED pgs=1805 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:42.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:42.323 7fd494d48700 1 -- 172.21.2.112:0/679258361 >> 172.21.2.112:0/679258361 conn(0x7fd4900ce450 msgr2=0x7fd4900d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:42.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:42.323 7fd494d48700 1 -- 172.21.2.112:0/679258361 shutdown_connections 2021-02-10T11:38:42.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:42.323 7fd494d48700 1 -- 172.21.2.112:0/679258361 wait complete. 2021-02-10T11:38:42.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:42.338 7fd494d48700 1 Processor -- start 2021-02-10T11:38:42.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:42.338 7fd494d48700 1 -- start start 2021-02-10T11:38:42.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:42.338 7fd494d48700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd49018e0c0 0x7fd4900f2500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:42.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:42.338 7fd494d48700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd4901a2e20 0x7fd4901a3220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:42.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:42.338 7fd494d48700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd4901a3840 0x7fd49018ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:42.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:42.338 7fd494d48700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd49018e7c0 con 0x7fd49018e0c0 2021-02-10T11:38:42.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:42.338 7fd494d48700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4900f2ca0 con 0x7fd4901a3840 2021-02-10T11:38:42.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:42.338 7fd494d48700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd49018e4c0 con 0x7fd4901a2e20 2021-02-10T11:38:42.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:42.338 7fd48eb13700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd49018e0c0 0x7fd4900f2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:42.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:42.338 7fd48eb13700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd49018e0c0 0x7fd4900f2500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:53104/0 (socket says 172.21.2.112:53104) 2021-02-10T11:38:42.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:42.338 7fd48eb13700 1 -- 172.21.2.112:0/43694041 learned_addr learned my addr 172.21.2.112:0/43694041 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:38:42.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:42.339 7fd48eb13700 1 -- 172.21.2.112:0/43694041 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd4901a2e20 msgr2=0x7fd4901a3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:42.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:42.339 7fd48eb13700 1 --2- 172.21.2.112:0/43694041 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd4901a2e20 0x7fd4901a3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:42.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:42.339 7fd48eb13700 1 -- 172.21.2.112:0/43694041 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd4901a3840 msgr2=0x7fd49018ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:42.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:42.339 7fd48eb13700 1 --2- 172.21.2.112:0/43694041 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd4901a3840 0x7fd49018ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:42.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:42.339 7fd48eb13700 1 -- 172.21.2.112:0/43694041 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4901a3c40 con 0x7fd49018e0c0 2021-02-10T11:38:42.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:42.339 7fd48eb13700 1 --2- 172.21.2.112:0/43694041 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd49018e0c0 0x7fd4900f2500 secure :-1 s=READY pgs=1806 cs=0 l=1 rev1=1 rx=0x7fd48000b5e0 tx=0x7fd48000b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:42.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:42.339 7fd4877fe700 1 -- 172.21.2.112:0/43694041 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd48000bd70 con 0x7fd49018e0c0 2021-02-10T11:38:42.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:42.339 7fd4877fe700 1 -- 172.21.2.112:0/43694041 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd480002030 con 0x7fd49018e0c0 2021-02-10T11:38:42.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:42.339 7fd4877fe700 1 -- 172.21.2.112:0/43694041 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd4800024d0 con 0x7fd49018e0c0 2021-02-10T11:38:42.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:42.339 7fd494d48700 1 -- 172.21.2.112:0/43694041 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4901a3c40 con 0x7fd49018e0c0 2021-02-10T11:38:42.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:42.339 7fd494d48700 1 -- 172.21.2.112:0/43694041 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd49018f360 con 0x7fd49018e0c0 2021-02-10T11:38:42.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:42.340 7fd4877fe700 1 -- 172.21.2.112:0/43694041 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd480000c90 con 0x7fd49018e0c0 2021-02-10T11:38:42.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:42.340 7fd4877fe700 1 --2- 172.21.2.112:0/43694041 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd4740186c0 0x7fd47401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:42.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:42.340 7fd4877fe700 1 -- 172.21.2.112:0/43694041 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5659+0+0 (secure 0 0 0) 0x7fd480032630 con 0x7fd49018e0c0 2021-02-10T11:38:42.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:42.340 7fd48e312700 1 --2- 172.21.2.112:0/43694041 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd4740186c0 0x7fd47401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:42.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:42.341 7fd48e312700 1 --2- 172.21.2.112:0/43694041 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd4740186c0 0x7fd47401abc0 crc :-1 s=READY pgs=850 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:42.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:42.371 7fd494d48700 1 -- 172.21.2.112:0/43694041 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4900ee060 con 0x7fd49018e0c0 2021-02-10T11:38:42.365 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:42.374 7fd4877fe700 1 -- 172.21.2.112:0/43694041 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd4800009b0 con 0x7fd49018e0c0 2021-02-10T11:38:42.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:42.503 7fd494d48700 1 -- 172.21.2.112:0/43694041 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fd4900ce450 con 0x7fd49018e0c0 2021-02-10T11:38:43.253 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:43.262 7fd4877fe700 1 -- 172.21.2.112:0/43694041 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v67) v1 ==== 104+0+0 (secure 0 0 0) 0x7fd480000c90 con 0x7fd49018e0c0 2021-02-10T11:38:43.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:43.272 7fd494d48700 1 -- 172.21.2.112:0/43694041 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd4740186c0 msgr2=0x7fd47401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:43.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:43.272 7fd494d48700 1 --2- 172.21.2.112:0/43694041 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd4740186c0 0x7fd47401abc0 crc :-1 s=READY pgs=850 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:43.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:43.272 7fd494d48700 1 -- 172.21.2.112:0/43694041 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd49018e0c0 msgr2=0x7fd4900f2500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:43.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:43.272 7fd494d48700 1 --2- 172.21.2.112:0/43694041 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd49018e0c0 0x7fd4900f2500 secure :-1 s=READY pgs=1806 cs=0 l=1 rev1=1 rx=0x7fd48000b5e0 tx=0x7fd48000b980).stop 2021-02-10T11:38:43.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:43.272 7fd494d48700 1 -- 172.21.2.112:0/43694041 shutdown_connections 2021-02-10T11:38:43.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:43.272 7fd494d48700 1 --2- 172.21.2.112:0/43694041 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd4740186c0 0x7fd47401abc0 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:43.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:43.272 7fd494d48700 1 --2- 172.21.2.112:0/43694041 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd4901a3840 0x7fd49018ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:43.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:43.272 7fd494d48700 1 --2- 172.21.2.112:0/43694041 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd4901a2e20 0x7fd4901a3220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:43.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:43.272 7fd494d48700 1 --2- 172.21.2.112:0/43694041 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd49018e0c0 0x7fd4900f2500 unknown :-1 s=CLOSED pgs=1806 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:43.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:43.272 7fd494d48700 1 -- 172.21.2.112:0/43694041 >> 172.21.2.112:0/43694041 conn(0x7fd4900bc0b0 msgr2=0x7fd4900ee290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:43.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:43.272 7fd494d48700 1 -- 172.21.2.112:0/43694041 shutdown_connections 2021-02-10T11:38:43.265 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:43.272 7fd494d48700 1 -- 172.21.2.112:0/43694041 wait complete. 2021-02-10T11:38:43.272 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2021-02-10T11:38:43.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:43.695 7f235e3bc700 1 Processor -- start 2021-02-10T11:38:43.704 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:43.695 7f235e3bc700 1 -- start start 2021-02-10T11:38:43.704 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:43.695 7f235e3bc700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f23580d1c80 0x7f23580cef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:43.704 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:43.695 7f235e3bc700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f23580cf590 0x7f23580cf990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:43.704 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:43.695 7f235e3bc700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23580fe1d0 con 0x7f23580d1880 2021-02-10T11:38:43.704 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:43.695 7f235e3bc700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23581b0cc0 con 0x7f23580d1c80 2021-02-10T11:38:43.704 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:43.695 7f235e3bc700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2358109a70 con 0x7f23580cf590 2021-02-10T11:38:43.704 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:43.696 7f235c9d9700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f23580cf590 0x7f23580cf990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:43.704 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:43.696 7f235c9d9700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f23580cf590 0x7f23580cf990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3301/0 says I am v2:172.21.2.112:52790/0 (socket says 172.21.2.112:52790) 2021-02-10T11:38:43.704 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:43.696 7f235c9d9700 1 -- 172.21.2.112:0/1244273599 learned_addr learned my addr 172.21.2.112:0/1244273599 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:38:43.704 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:43.696 7f23577fe700 1 --2- 172.21.2.112:0/1244273599 >> v2:172.21.2.120:3300/0 conn(0x7f23580d1c80 0x7f23580cef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:43.704 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:43.696 7f235c9d9700 1 -- 172.21.2.112:0/1244273599 >> v2:172.21.2.120:3300/0 conn(0x7f23580d1c80 msgr2=0x7f23580cef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:43.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:43.696 7f235c9d9700 1 --2- 172.21.2.112:0/1244273599 >> v2:172.21.2.120:3300/0 conn(0x7f23580d1c80 0x7f23580cef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:43.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:43.696 7f235c9d9700 1 -- 172.21.2.112:0/1244273599 >> v1:172.21.2.120:6789/0 conn(0x7f23580d1880 legacy=0x7f23581b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:43.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:43.696 7f235c9d9700 1 -- 172.21.2.112:0/1244273599 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2358088070 con 0x7f23580cf590 2021-02-10T11:38:43.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:43.696 7f23577fe700 1 --2- 172.21.2.112:0/1244273599 >> v2:172.21.2.120:3300/0 conn(0x7f23580d1c80 0x7f23580cef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:38:43.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:43.696 7f235c9d9700 1 --2- 172.21.2.112:0/1244273599 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f23580cf590 0x7f23580cf990 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f2350006aa0 tx=0x7f2350009f20).ready entity=mon.2 client_cookie=9a9fdd00004f4c26 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:43.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:43.697 7f2356ffd700 1 -- 172.21.2.112:0/1244273599 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2350000f90 con 0x7f23580cf590 2021-02-10T11:38:43.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:43.697 7f2356ffd700 1 -- 172.21.2.112:0/1244273599 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2350003030 con 0x7f23580cf590 2021-02-10T11:38:43.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:43.697 7f2356ffd700 1 -- 172.21.2.112:0/1244273599 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f235000b7b0 con 0x7f23580cf590 2021-02-10T11:38:43.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:43.697 7f235e3bc700 1 -- 172.21.2.112:0/1244273599 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f23580cf590 msgr2=0x7f23580cf990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:43.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:43.697 7f235e3bc700 1 --2- 172.21.2.112:0/1244273599 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f23580cf590 0x7f23580cf990 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f2350006aa0 tx=0x7f2350009f20).stop 2021-02-10T11:38:43.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:43.698 7f235e3bc700 1 -- 172.21.2.112:0/1244273599 shutdown_connections 2021-02-10T11:38:43.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:43.698 7f235e3bc700 1 --2- 172.21.2.112:0/1244273599 >> v2:172.21.2.120:3300/0 conn(0x7f23580d1c80 0x7f23580cef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:43.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:43.698 7f235e3bc700 1 --2- 172.21.2.112:0/1244273599 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f23580cf590 0x7f23580cf990 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:43.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:43.698 7f235e3bc700 1 -- 172.21.2.112:0/1244273599 >> 172.21.2.112:0/1244273599 conn(0x7f23580ce430 msgr2=0x7f23580d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:43.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:43.698 7f235e3bc700 1 -- 172.21.2.112:0/1244273599 shutdown_connections 2021-02-10T11:38:43.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:43.698 7f235e3bc700 1 -- 172.21.2.112:0/1244273599 wait complete. 2021-02-10T11:38:43.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:43.712 7f235e3bc700 1 Processor -- start 2021-02-10T11:38:43.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:43.712 7f235e3bc700 1 -- start start 2021-02-10T11:38:43.707 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:43.712 7f235e3bc700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f235808a8f0 0x7f235808acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:43.708 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:43.712 7f235e3bc700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f235808b310 0x7f235808b710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:43.708 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:43.712 7f235e3bc700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f23581a5710 0x7f23581a5b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:43.708 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:43.712 7f235e3bc700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f235808a420 con 0x7f23581a5710 2021-02-10T11:38:43.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:43.712 7f235e3bc700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23581a6160 con 0x7f235808a8f0 2021-02-10T11:38:43.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:43.712 7f235e3bc700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23581a6380 con 0x7f235808b310 2021-02-10T11:38:43.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:43.713 7f235c9d9700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f23581a5710 0x7f23581a5b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:43.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:43.713 7f235c9d9700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f23581a5710 0x7f23581a5b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:53114/0 (socket says 172.21.2.112:53114) 2021-02-10T11:38:43.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:43.713 7f235c9d9700 1 -- 172.21.2.112:0/343507050 learned_addr learned my addr 172.21.2.112:0/343507050 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:38:43.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:43.713 7f235c9d9700 1 -- 172.21.2.112:0/343507050 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f235808b310 msgr2=0x7f235808b710 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:38:43.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:43.713 7f235c9d9700 1 --2- 172.21.2.112:0/343507050 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f235808b310 0x7f235808b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:43.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:43.713 7f235c9d9700 1 -- 172.21.2.112:0/343507050 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f235808a8f0 msgr2=0x7f235808acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:43.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:43.713 7f235c9d9700 1 --2- 172.21.2.112:0/343507050 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f235808a8f0 0x7f235808acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:43.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:43.713 7f235c9d9700 1 -- 172.21.2.112:0/343507050 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23581a65a0 con 0x7f23581a5710 2021-02-10T11:38:43.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:43.713 7f235c9d9700 1 --2- 172.21.2.112:0/343507050 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f23581a5710 0x7f23581a5b10 secure :-1 s=READY pgs=1807 cs=0 l=1 rev1=1 rx=0x7f235000b940 tx=0x7f235000bce0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:43.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:43.713 7f2354ff9700 1 -- 172.21.2.112:0/343507050 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f235000c4e0 con 0x7f23581a5710 2021-02-10T11:38:43.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:43.713 7f2354ff9700 1 -- 172.21.2.112:0/343507050 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f235000c7e0 con 0x7f23581a5710 2021-02-10T11:38:43.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:43.713 7f235e3bc700 1 -- 172.21.2.112:0/343507050 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23581a65a0 con 0x7f23581a5710 2021-02-10T11:38:43.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:43.713 7f2354ff9700 1 -- 172.21.2.112:0/343507050 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f235000cc30 con 0x7f23581a5710 2021-02-10T11:38:43.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:43.713 7f235e3bc700 1 -- 172.21.2.112:0/343507050 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23581a65a0 con 0x7f23581a5710 2021-02-10T11:38:43.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:43.714 7f2354ff9700 1 -- 172.21.2.112:0/343507050 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f235000c2f0 con 0x7f23581a5710 2021-02-10T11:38:43.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:43.714 7f2354ff9700 1 --2- 172.21.2.112:0/343507050 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2338018730 0x7f233801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:43.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:43.714 7f2354ff9700 1 -- 172.21.2.112:0/343507050 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5659+0+0 (secure 0 0 0) 0x7f23500329c0 con 0x7f23581a5710 2021-02-10T11:38:43.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:43.714 7f2357fff700 1 --2- 172.21.2.112:0/343507050 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2338018730 0x7f233801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:43.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:43.715 7f2357fff700 1 --2- 172.21.2.112:0/343507050 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2338018730 0x7f233801ac10 crc :-1 s=READY pgs=851 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:43.737 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:43.745 7f235e3bc700 1 -- 172.21.2.112:0/343507050 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2358082780 con 0x7f23581a5710 2021-02-10T11:38:43.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:43.748 7f2354ff9700 1 -- 172.21.2.112:0/343507050 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f235000be20 con 0x7f23581a5710 2021-02-10T11:38:43.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:43.878 7f235e3bc700 1 -- 172.21.2.112:0/343507050 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f2358082780 con 0x7f23581a5710 2021-02-10T11:38:44.258 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:44.266 7f2354ff9700 1 -- 172.21.2.112:0/343507050 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v85) v1 ==== 185+0+0 (secure 0 0 0) 0x7f235000c2f0 con 0x7f23581a5710 2021-02-10T11:38:44.286 INFO:teuthology.orchestra.run.gibba012.stderr:pool 'cephfs_metadata' removed 2021-02-10T11:38:44.288 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:44.297 7f235e3bc700 1 -- 172.21.2.112:0/343507050 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2338018730 msgr2=0x7f233801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:44.288 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:44.297 7f235e3bc700 1 --2- 172.21.2.112:0/343507050 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2338018730 0x7f233801ac10 crc :-1 s=READY pgs=851 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:44.288 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:44.297 7f235e3bc700 1 -- 172.21.2.112:0/343507050 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f23581a5710 msgr2=0x7f23581a5b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:44.288 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:44.297 7f235e3bc700 1 --2- 172.21.2.112:0/343507050 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f23581a5710 0x7f23581a5b10 secure :-1 s=READY pgs=1807 cs=0 l=1 rev1=1 rx=0x7f235000b940 tx=0x7f235000bce0).stop 2021-02-10T11:38:44.289 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:44.297 7f235e3bc700 1 -- 172.21.2.112:0/343507050 shutdown_connections 2021-02-10T11:38:44.289 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:44.297 7f235e3bc700 1 --2- 172.21.2.112:0/343507050 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2338018730 0x7f233801ac10 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:44.289 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:44.297 7f235e3bc700 1 --2- 172.21.2.112:0/343507050 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f23581a5710 0x7f23581a5b10 unknown :-1 s=CLOSED pgs=1807 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:44.289 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:44.297 7f235e3bc700 1 --2- 172.21.2.112:0/343507050 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f235808b310 0x7f235808b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:44.289 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:44.297 7f235e3bc700 1 --2- 172.21.2.112:0/343507050 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f235808a8f0 0x7f235808acf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:44.289 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:44.297 7f235e3bc700 1 -- 172.21.2.112:0/343507050 >> 172.21.2.112:0/343507050 conn(0x7f23581b9d60 msgr2=0x7f2358082fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:44.289 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:44.297 7f235e3bc700 1 -- 172.21.2.112:0/343507050 shutdown_connections 2021-02-10T11:38:44.289 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:44.297 7f235e3bc700 1 -- 172.21.2.112:0/343507050 wait complete. 2021-02-10T11:38:44.297 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2021-02-10T11:38:44.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:44.718 7f3ecfdfe700 1 Processor -- start 2021-02-10T11:38:44.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:44.718 7f3ecfdfe700 1 -- start start 2021-02-10T11:38:44.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:44.719 7f3ecfdfe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f3ec80d1880 0x7f3ec81b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:44.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:44.719 7f3ecfdfe700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f3ec81b0af0 0x7f3ec8108950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:44.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:44.719 7f3ecfdfe700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3ec80d1c80 0x7f3ec8108f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:44.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:44.719 7f3ecfdfe700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ec80fe1d0 con 0x7f3ec80d1880 2021-02-10T11:38:44.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:44.719 7f3ecfdfe700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ec8109a70 con 0x7f3ec81b0af0 2021-02-10T11:38:44.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:44.719 7f3ecfdfe700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ec8088070 con 0x7f3ec80d1c80 2021-02-10T11:38:44.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:44.719 7f3ecdc1a700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f3ec80d1880 0x7f3ec81b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:44.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:44.719 7f3ecdc1a700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f3ec80d1880 0x7f3ec81b04d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:53122/0 (socket says 172.21.2.112:53122) 2021-02-10T11:38:44.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:44.719 7f3ecdc1a700 1 -- 172.21.2.112:0/3159139890 learned_addr learned my addr 172.21.2.112:0/3159139890 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:38:44.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:44.719 7f3ecdc1a700 1 -- 172.21.2.112:0/3159139890 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3ec80d1c80 msgr2=0x7f3ec8108f70 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:38:44.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:44.719 7f3ecdc1a700 1 --2- 172.21.2.112:0/3159139890 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3ec80d1c80 0x7f3ec8108f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:44.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:44.719 7f3ecdc1a700 1 -- 172.21.2.112:0/3159139890 >> v2:172.21.2.120:3300/0 conn(0x7f3ec81b0af0 msgr2=0x7f3ec8108950 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:38:44.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:44.719 7f3ecdc1a700 1 --2- 172.21.2.112:0/3159139890 >> v2:172.21.2.120:3300/0 conn(0x7f3ec81b0af0 0x7f3ec8108950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:44.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:44.719 7f3ecdc1a700 1 -- 172.21.2.112:0/3159139890 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ec8109590 con 0x7f3ec80d1880 2021-02-10T11:38:44.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:44.720 7f3ecdc1a700 1 --2- 172.21.2.112:0/3159139890 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3ec80d1880 0x7f3ec81b04d0 secure :-1 s=READY pgs=1808 cs=0 l=1 rev1=1 rx=0x7f3ec4006700 tx=0x7f3ec4009b80).ready entity=mon.0 client_cookie=7a6d21b5bbcda66f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:44.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:44.720 7f3eccc18700 1 -- 172.21.2.112:0/3159139890 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3ec400c0c0 con 0x7f3ec80d1880 2021-02-10T11:38:44.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:44.720 7f3eccc18700 1 -- 172.21.2.112:0/3159139890 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ec400c440 con 0x7f3ec80d1880 2021-02-10T11:38:44.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:44.720 7f3eccc18700 1 -- 172.21.2.112:0/3159139890 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3ec400c960 con 0x7f3ec80d1880 2021-02-10T11:38:44.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:44.720 7f3ecfdfe700 1 -- 172.21.2.112:0/3159139890 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3ec80d1880 msgr2=0x7f3ec81b04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:44.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:44.720 7f3ecfdfe700 1 --2- 172.21.2.112:0/3159139890 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3ec80d1880 0x7f3ec81b04d0 secure :-1 s=READY pgs=1808 cs=0 l=1 rev1=1 rx=0x7f3ec4006700 tx=0x7f3ec4009b80).stop 2021-02-10T11:38:44.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:44.721 7f3ecfdfe700 1 -- 172.21.2.112:0/3159139890 shutdown_connections 2021-02-10T11:38:44.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:44.721 7f3ecfdfe700 1 --2- 172.21.2.112:0/3159139890 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3ec80d1c80 0x7f3ec8108f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:44.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:44.721 7f3ecfdfe700 1 --2- 172.21.2.112:0/3159139890 >> v2:172.21.2.120:3300/0 conn(0x7f3ec81b0af0 0x7f3ec8108950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:44.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:44.721 7f3ecfdfe700 1 --2- 172.21.2.112:0/3159139890 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3ec80d1880 0x7f3ec81b04d0 unknown :-1 s=CLOSED pgs=1808 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:44.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:44.721 7f3ecfdfe700 1 -- 172.21.2.112:0/3159139890 >> 172.21.2.112:0/3159139890 conn(0x7f3ec80ce430 msgr2=0x7f3ec80d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:44.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:44.721 7f3ecfdfe700 1 -- 172.21.2.112:0/3159139890 shutdown_connections 2021-02-10T11:38:44.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:44.721 7f3ecfdfe700 1 -- 172.21.2.112:0/3159139890 wait complete. 2021-02-10T11:38:44.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:44.735 7f3ecfdfe700 1 Processor -- start 2021-02-10T11:38:44.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:44.735 7f3ecfdfe700 1 -- start start 2021-02-10T11:38:44.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:44.736 7f3ecfdfe700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3ec8190df0 0x7f3ec81b7ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:44.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:44.736 7f3ecfdfe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3ec819ba20 0x7f3ec81b87c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:44.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:44.736 7f3ecfdfe700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3ec819be20 0x7f3ec818f830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:44.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:44.736 7f3ecfdfe700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ec81b8240 con 0x7f3ec819ba20 2021-02-10T11:38:44.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:44.736 7f3ecfdfe700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ec81911f0 con 0x7f3ec8190df0 2021-02-10T11:38:44.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:44.736 7f3ecfdfe700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ec81908a0 con 0x7f3ec819be20 2021-02-10T11:38:44.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:44.736 7f3ecd419700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3ec819ba20 0x7f3ec81b87c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:44.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:44.736 7f3ecd419700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3ec819ba20 0x7f3ec81b87c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:53124/0 (socket says 172.21.2.112:53124) 2021-02-10T11:38:44.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:44.736 7f3ecd419700 1 -- 172.21.2.112:0/4099546060 learned_addr learned my addr 172.21.2.112:0/4099546060 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:38:44.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:44.736 7f3ecd419700 1 -- 172.21.2.112:0/4099546060 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3ec819be20 msgr2=0x7f3ec818f830 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:38:44.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:44.736 7f3ecd419700 1 --2- 172.21.2.112:0/4099546060 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3ec819be20 0x7f3ec818f830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:44.732 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:44.736 7f3ecd419700 1 -- 172.21.2.112:0/4099546060 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3ec8190df0 msgr2=0x7f3ec81b7ac0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:38:44.732 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:44.736 7f3ecd419700 1 --2- 172.21.2.112:0/4099546060 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3ec8190df0 0x7f3ec81b7ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:44.732 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:44.736 7f3ecd419700 1 -- 172.21.2.112:0/4099546060 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ec81b8de0 con 0x7f3ec819ba20 2021-02-10T11:38:44.732 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:44.737 7f3ecd419700 1 --2- 172.21.2.112:0/4099546060 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3ec819ba20 0x7f3ec81b87c0 secure :-1 s=READY pgs=1809 cs=0 l=1 rev1=1 rx=0x7f3ebc00b5c0 tx=0x7f3ebc00b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:44.732 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:44.737 7f3eba7fc700 1 -- 172.21.2.112:0/4099546060 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3ebc00bd50 con 0x7f3ec819ba20 2021-02-10T11:38:44.732 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:44.737 7f3eba7fc700 1 -- 172.21.2.112:0/4099546060 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ebc006030 con 0x7f3ec819ba20 2021-02-10T11:38:44.732 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:44.737 7f3eba7fc700 1 -- 172.21.2.112:0/4099546060 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3ebc0054d0 con 0x7f3ec819ba20 2021-02-10T11:38:44.732 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:44.737 7f3ecfdfe700 1 -- 172.21.2.112:0/4099546060 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ec81b8de0 con 0x7f3ec819ba20 2021-02-10T11:38:44.732 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:44.737 7f3ecfdfe700 1 -- 172.21.2.112:0/4099546060 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ec81b8de0 con 0x7f3ec819ba20 2021-02-10T11:38:44.732 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:44.738 7f3eba7fc700 1 -- 172.21.2.112:0/4099546060 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3ebc006210 con 0x7f3ec819ba20 2021-02-10T11:38:44.732 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:44.738 7f3eba7fc700 1 --2- 172.21.2.112:0/4099546060 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3eb00186c0 0x7f3eb001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:44.732 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:44.738 7f3eba7fc700 1 -- 172.21.2.112:0/4099546060 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 5221+0+0 (secure 0 0 0) 0x7f3ebc032430 con 0x7f3ec819ba20 2021-02-10T11:38:44.732 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:44.738 7f3ecdc1a700 1 --2- 172.21.2.112:0/4099546060 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3eb00186c0 0x7f3eb001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:44.732 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:44.739 7f3ecdc1a700 1 --2- 172.21.2.112:0/4099546060 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3eb00186c0 0x7f3eb001abc0 crc :-1 s=READY pgs=852 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:44.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:44.769 7f3ecfdfe700 1 -- 172.21.2.112:0/4099546060 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ec8082aa0 con 0x7f3ec819ba20 2021-02-10T11:38:44.763 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:44.771 7f3eba7fc700 1 -- 172.21.2.112:0/4099546060 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3ebc006b80 con 0x7f3ec819ba20 2021-02-10T11:38:44.890 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:44.899 7f3ecfdfe700 1 -- 172.21.2.112:0/4099546060 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f3ec8082aa0 con 0x7f3ec819ba20 2021-02-10T11:38:45.262 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:45.270 7f3eba7fc700 1 -- 172.21.2.112:0/4099546060 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v86) v1 ==== 173+0+0 (secure 0 0 0) 0x7f3ebc006210 con 0x7f3ec819ba20 2021-02-10T11:38:45.307 INFO:teuthology.orchestra.run.gibba012.stderr:pool 'cephfs_data' removed 2021-02-10T11:38:45.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:45.318 7f3ecfdfe700 1 -- 172.21.2.112:0/4099546060 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3eb00186c0 msgr2=0x7f3eb001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:45.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:45.318 7f3ecfdfe700 1 --2- 172.21.2.112:0/4099546060 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3eb00186c0 0x7f3eb001abc0 crc :-1 s=READY pgs=852 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:45.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:45.318 7f3ecfdfe700 1 -- 172.21.2.112:0/4099546060 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3ec819ba20 msgr2=0x7f3ec81b87c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:45.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:45.318 7f3ecfdfe700 1 --2- 172.21.2.112:0/4099546060 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3ec819ba20 0x7f3ec81b87c0 secure :-1 s=READY pgs=1809 cs=0 l=1 rev1=1 rx=0x7f3ebc00b5c0 tx=0x7f3ebc00b960).stop 2021-02-10T11:38:45.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:45.318 7f3ecfdfe700 1 -- 172.21.2.112:0/4099546060 shutdown_connections 2021-02-10T11:38:45.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:45.318 7f3ecfdfe700 1 --2- 172.21.2.112:0/4099546060 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3eb00186c0 0x7f3eb001abc0 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:45.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:45.318 7f3ecfdfe700 1 --2- 172.21.2.112:0/4099546060 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3ec819be20 0x7f3ec818f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:45.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:45.318 7f3ecfdfe700 1 --2- 172.21.2.112:0/4099546060 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3ec819ba20 0x7f3ec81b87c0 unknown :-1 s=CLOSED pgs=1809 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:45.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:45.318 7f3ecfdfe700 1 --2- 172.21.2.112:0/4099546060 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3ec8190df0 0x7f3ec81b7ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:45.311 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:45.318 7f3ecfdfe700 1 -- 172.21.2.112:0/4099546060 >> 172.21.2.112:0/4099546060 conn(0x7f3ec819f940 msgr2=0x7f3ec818c290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:45.311 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:45.318 7f3ecfdfe700 1 -- 172.21.2.112:0/4099546060 shutdown_connections 2021-02-10T11:38:45.311 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:45.318 7f3ecfdfe700 1 -- 172.21.2.112:0/4099546060 wait complete. 2021-02-10T11:38:45.318 INFO:tasks.ceph.mds.a:Restarting daemon 2021-02-10T11:38:45.318 INFO:tasks.ceph.mds.a:Stopping old one... 2021-02-10T11:38:45.319 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2021-02-10T11:38:45.319 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:38:45.319 INFO:tasks.ceph.mds.b:Restarting daemon 2021-02-10T11:38:45.319 INFO:tasks.ceph.mds.b:Stopping old one... 2021-02-10T11:38:45.319 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2021-02-10T11:38:45.319 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:38:45.324 INFO:tasks.ceph.mds.a:Stopped 2021-02-10T11:38:45.325 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2021-02-10T11:38:45.326 INFO:tasks.ceph.mds.b:Stopped 2021-02-10T11:38:45.326 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2021-02-10T11:38:45.367 INFO:tasks.ceph.mds.a:Started 2021-02-10T11:38:45.367 DEBUG:teuthology.parallel:result is None 2021-02-10T11:38:45.368 INFO:tasks.ceph.mds.b:Started 2021-02-10T11:38:45.368 DEBUG:teuthology.parallel:result is None 2021-02-10T11:38:45.368 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd blacklist clear 2021-02-10T11:38:45.770 INFO:tasks.ceph.mds.b.gibba020.stdout:starting mds.b at 2021-02-10T11:38:45.773 INFO:tasks.ceph.mds.a.gibba012.stdout:starting mds.a at 2021-02-10T11:38:45.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:45.784 7fe0edc22700 1 Processor -- start 2021-02-10T11:38:45.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:45.784 7fe0edc22700 1 -- start start 2021-02-10T11:38:45.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:45.784 7fe0edc22700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fe0e80d1c60 0x7fe0e80f2090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:45.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:45.784 7fe0edc22700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0e80f0b00 con 0x7fe0e80d1860 2021-02-10T11:38:45.793 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:45.784 7fe0edc22700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0e81b0bd0 con 0x7fe0e80d0330 2021-02-10T11:38:45.793 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:45.784 7fe0edc22700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0e80e5ec0 con 0x7fe0e80d1c60 2021-02-10T11:38:45.793 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:45.785 7fe0e77fe700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fe0e80d1860 0x7fe0e81b03e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:52606/0 (socket says 172.21.2.112:52606) 2021-02-10T11:38:45.793 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:45.785 7fe0e77fe700 1 -- 172.21.2.112:0/2726771799 learned_addr learned my addr 172.21.2.112:0/2726771799 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:38:45.793 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:45.785 7fe0e7fff700 1 --2- 172.21.2.112:0/2726771799 >> v2:172.21.2.112:3300/0 conn(0x7fe0e80d1c60 0x7fe0e80f2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:45.793 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:45.785 7fe0e67fc700 1 -- 172.21.2.112:0/2726771799 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3170334233 0 0) 0x7fe0dc001180 con 0x7fe0e80d1860 2021-02-10T11:38:45.793 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:45.785 7fe0e67fc700 1 -- 172.21.2.112:0/2726771799 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0d0002fe0 con 0x7fe0e80d1860 2021-02-10T11:38:45.793 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:45.785 7fe0e7fff700 1 -- 172.21.2.112:0/2726771799 >> v1:172.21.2.120:6789/0 conn(0x7fe0e80d0330 legacy=0x7fe0e80ef630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:45.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:45.785 7fe0e7fff700 1 -- 172.21.2.112:0/2726771799 >> v1:172.21.2.112:6789/0 conn(0x7fe0e80d1860 legacy=0x7fe0e81b03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:45.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:45.785 7fe0e7fff700 1 -- 172.21.2.112:0/2726771799 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0e80d3350 con 0x7fe0e80d1c60 2021-02-10T11:38:45.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:45.785 7fe0e7fff700 1 --2- 172.21.2.112:0/2726771799 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe0e80d1c60 0x7fe0e80f2090 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 rx=0x7fe0e0006e30 tx=0x7fe0e0009e60).ready entity=mon.0 client_cookie=a94b697306c50f16 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:45.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:45.785 7fe0e67fc700 1 -- 172.21.2.112:0/2726771799 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe0e0006a50 con 0x7fe0e80d1c60 2021-02-10T11:38:45.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:45.785 7fe0e67fc700 1 -- 172.21.2.112:0/2726771799 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0e000b300 con 0x7fe0e80d1c60 2021-02-10T11:38:45.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:45.785 7fe0e67fc700 1 -- 172.21.2.112:0/2726771799 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe0e000b820 con 0x7fe0e80d1c60 2021-02-10T11:38:45.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:45.785 7fe0edc22700 1 -- 172.21.2.112:0/2726771799 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe0e80d1c60 msgr2=0x7fe0e80f2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:45.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:45.785 7fe0edc22700 1 --2- 172.21.2.112:0/2726771799 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe0e80d1c60 0x7fe0e80f2090 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 rx=0x7fe0e0006e30 tx=0x7fe0e0009e60).stop 2021-02-10T11:38:45.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:45.786 7fe0edc22700 1 -- 172.21.2.112:0/2726771799 shutdown_connections 2021-02-10T11:38:45.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:45.786 7fe0edc22700 1 --2- 172.21.2.112:0/2726771799 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe0e80d1c60 0x7fe0e80f2090 unknown :-1 s=CLOSED pgs=1815 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:45.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:45.786 7fe0edc22700 1 -- 172.21.2.112:0/2726771799 >> 172.21.2.112:0/2726771799 conn(0x7fe0e80ce440 msgr2=0x7fe0e80d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:45.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:45.786 7fe0edc22700 1 -- 172.21.2.112:0/2726771799 shutdown_connections 2021-02-10T11:38:45.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:45.786 7fe0edc22700 1 -- 172.21.2.112:0/2726771799 wait complete. 2021-02-10T11:38:45.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:45.801 7fe0edc22700 1 Processor -- start 2021-02-10T11:38:45.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:45.801 7fe0edc22700 1 -- start start 2021-02-10T11:38:45.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:45.801 7fe0edc22700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe0e81a78d0 0x7fe0e81a7cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:45.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:45.801 7fe0edc22700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe0e8193090 0x7fe0e8193490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:45.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:45.801 7fe0edc22700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe0e8193ab0 0x7fe0e80a80d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:45.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:45.801 7fe0edc22700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0e80a0790 con 0x7fe0e8193ab0 2021-02-10T11:38:45.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:45.801 7fe0edc22700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0e80a7b50 con 0x7fe0e81a78d0 2021-02-10T11:38:45.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:45.801 7fe0edc22700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0e80a02f0 con 0x7fe0e8193090 2021-02-10T11:38:45.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:45.801 7fe0e7fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe0e8193ab0 0x7fe0e80a80d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:45.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:45.801 7fe0e7fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe0e8193ab0 0x7fe0e80a80d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:53154/0 (socket says 172.21.2.112:53154) 2021-02-10T11:38:45.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:45.801 7fe0e7fff700 1 -- 172.21.2.112:0/2485177336 learned_addr learned my addr 172.21.2.112:0/2485177336 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:38:45.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:45.802 7fe0e6ffd700 1 --2- 172.21.2.112:0/2485177336 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe0e8193090 0x7fe0e8193490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:45.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:45.802 7fe0e77fe700 1 --2- 172.21.2.112:0/2485177336 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe0e81a78d0 0x7fe0e81a7cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:45.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:45.802 7fe0e7fff700 1 -- 172.21.2.112:0/2485177336 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe0e8193090 msgr2=0x7fe0e8193490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:45.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:45.802 7fe0e7fff700 1 --2- 172.21.2.112:0/2485177336 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe0e8193090 0x7fe0e8193490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:45.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:45.802 7fe0e7fff700 1 -- 172.21.2.112:0/2485177336 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe0e81a78d0 msgr2=0x7fe0e81a7cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:45.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:45.802 7fe0e7fff700 1 --2- 172.21.2.112:0/2485177336 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe0e81a78d0 0x7fe0e81a7cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:45.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:45.802 7fe0e7fff700 1 -- 172.21.2.112:0/2485177336 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0e8193eb0 con 0x7fe0e8193ab0 2021-02-10T11:38:45.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:45.802 7fe0e77fe700 1 --2- 172.21.2.112:0/2485177336 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe0e81a78d0 0x7fe0e81a7cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:38:45.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:45.802 7fe0e7fff700 1 --2- 172.21.2.112:0/2485177336 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe0e8193ab0 0x7fe0e80a80d0 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 rx=0x7fe0e000b910 tx=0x7fe0e000bcb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:45.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:45.802 7fe0cffff700 1 -- 172.21.2.112:0/2485177336 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe0e000bdf0 con 0x7fe0e8193ab0 2021-02-10T11:38:45.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:45.802 7fe0cffff700 1 -- 172.21.2.112:0/2485177336 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0e0005820 con 0x7fe0e8193ab0 2021-02-10T11:38:45.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:45.802 7fe0cffff700 1 -- 172.21.2.112:0/2485177336 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe0e0005c80 con 0x7fe0e8193ab0 2021-02-10T11:38:45.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:45.802 7fe0edc22700 1 -- 172.21.2.112:0/2485177336 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0e8193eb0 con 0x7fe0e8193ab0 2021-02-10T11:38:45.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:45.802 7fe0edc22700 1 -- 172.21.2.112:0/2485177336 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0e80a86f0 con 0x7fe0e8193ab0 2021-02-10T11:38:45.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:45.803 7fe0cffff700 1 -- 172.21.2.112:0/2485177336 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe0e0005a00 con 0x7fe0e8193ab0 2021-02-10T11:38:45.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:45.803 7fe0cffff700 1 --2- 172.21.2.112:0/2485177336 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe0c80186c0 0x7fe0c801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:45.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:45.803 7fe0cffff700 1 -- 172.21.2.112:0/2485177336 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 4839+0+0 (secure 0 0 0) 0x7fe0e00324f0 con 0x7fe0e8193ab0 2021-02-10T11:38:45.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:45.803 7fe0e77fe700 1 --2- 172.21.2.112:0/2485177336 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe0c80186c0 0x7fe0c801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:45.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:45.804 7fe0e77fe700 1 --2- 172.21.2.112:0/2485177336 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe0c80186c0 0x7fe0c801abc0 crc :-1 s=READY pgs=853 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:45.825 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:45.833 7fe0edc22700 1 -- 172.21.2.112:0/2485177336 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0e80ee0a0 con 0x7fe0e8193ab0 2021-02-10T11:38:45.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:45.836 7fe0cffff700 1 -- 172.21.2.112:0/2485177336 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe0e00093d0 con 0x7fe0e8193ab0 2021-02-10T11:38:45.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:45.975 7fe0edc22700 1 -- 172.21.2.112:0/2485177336 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7fe0e80ce440 con 0x7fe0e8193ab0 2021-02-10T11:38:46.306 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:46.314 7fe0cffff700 1 -- 172.21.2.112:0/2485177336 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v87) v1 ==== 97+0+0 (secure 0 0 0) 0x7fe0e00059e0 con 0x7fe0e8193ab0 2021-02-10T11:38:46.333 INFO:teuthology.orchestra.run.gibba012.stderr: removed all blacklist entries 2021-02-10T11:38:46.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:46.343 7fe0edc22700 1 -- 172.21.2.112:0/2485177336 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe0c80186c0 msgr2=0x7fe0c801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:46.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:46.343 7fe0edc22700 1 --2- 172.21.2.112:0/2485177336 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe0c80186c0 0x7fe0c801abc0 crc :-1 s=READY pgs=853 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:46.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:46.343 7fe0edc22700 1 -- 172.21.2.112:0/2485177336 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe0e8193ab0 msgr2=0x7fe0e80a80d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:46.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:46.343 7fe0edc22700 1 --2- 172.21.2.112:0/2485177336 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe0e8193ab0 0x7fe0e80a80d0 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 rx=0x7fe0e000b910 tx=0x7fe0e000bcb0).stop 2021-02-10T11:38:46.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:46.344 7fe0edc22700 1 -- 172.21.2.112:0/2485177336 shutdown_connections 2021-02-10T11:38:46.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:46.344 7fe0edc22700 1 --2- 172.21.2.112:0/2485177336 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe0c80186c0 0x7fe0c801abc0 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:46.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:46.344 7fe0edc22700 1 --2- 172.21.2.112:0/2485177336 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe0e8193ab0 0x7fe0e80a80d0 unknown :-1 s=CLOSED pgs=1816 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:46.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:46.344 7fe0edc22700 1 --2- 172.21.2.112:0/2485177336 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe0e8193090 0x7fe0e8193490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:46.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:46.344 7fe0edc22700 1 --2- 172.21.2.112:0/2485177336 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe0e81a78d0 0x7fe0e81a7cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:46.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:46.344 7fe0edc22700 1 -- 172.21.2.112:0/2485177336 >> 172.21.2.112:0/2485177336 conn(0x7fe0e8190ae0 msgr2=0x7fe0e80bf070 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:46.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:46.344 7fe0edc22700 1 -- 172.21.2.112:0/2485177336 shutdown_connections 2021-02-10T11:38:46.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:46.344 7fe0edc22700 1 -- 172.21.2.112:0/2485177336 wait complete. 2021-02-10T11:38:46.344 INFO:tasks.cephfs.cephfs_test_case:['0', '1', '2', '3'] 2021-02-10T11:38:46.344 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth ls --format=json-pretty 2021-02-10T11:38:46.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:46.762 7fb3bf504700 1 Processor -- start 2021-02-10T11:38:46.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:46.762 7fb3bf504700 1 -- start start 2021-02-10T11:38:46.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:46.763 7fb3bf504700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fb3b80d1870 0x7fb3b81b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:46.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:46.763 7fb3bf504700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fb3b80f0650 0x7fb3b80cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:46.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:46.763 7fb3bf504700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3b80f0b10 con 0x7fb3b81b0a10 2021-02-10T11:38:46.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:46.763 7fb3bf504700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3b80e5ed0 con 0x7fb3b80f0650 2021-02-10T11:38:46.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:46.763 7fb3bf504700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3b80d1c70 con 0x7fb3b80d1870 2021-02-10T11:38:46.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:46.763 7fb3bdb21700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fb3b80f0650 0x7fb3b80cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:46.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:46.763 7fb3bdb21700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fb3b80f0650 0x7fb3b80cfb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:53158/0 (socket says 172.21.2.112:53158) 2021-02-10T11:38:46.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:46.763 7fb3bdb21700 1 -- 172.21.2.112:0/2792979982 learned_addr learned my addr 172.21.2.112:0/2792979982 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:38:46.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:46.763 7fb3bdb21700 1 -- 172.21.2.112:0/2792979982 >> v2:172.21.2.120:3300/0 conn(0x7fb3b80d1870 msgr2=0x7fb3b81b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:46.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:46.763 7fb3bdb21700 1 --2- 172.21.2.112:0/2792979982 >> v2:172.21.2.120:3300/0 conn(0x7fb3b80d1870 0x7fb3b81b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:46.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:46.763 7fb3bdb21700 1 -- 172.21.2.112:0/2792979982 >> v1:172.21.2.120:6789/0 conn(0x7fb3b81b0a10 legacy=0x7fb3b80cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:46.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:46.763 7fb3bdb21700 1 -- 172.21.2.112:0/2792979982 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3b80d1e90 con 0x7fb3b80f0650 2021-02-10T11:38:46.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:46.764 7fb3bdb21700 1 --2- 172.21.2.112:0/2792979982 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb3b80f0650 0x7fb3b80cfb20 secure :-1 s=READY pgs=1817 cs=0 l=1 rev1=1 rx=0x7fb3b0006f00 tx=0x7fb3b0009e90).ready entity=mon.0 client_cookie=640c412d30e68ce0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:46.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:46.764 7fb3affff700 1 -- 172.21.2.112:0/2792979982 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb3b000b300 con 0x7fb3b80f0650 2021-02-10T11:38:46.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:46.764 7fb3affff700 1 -- 172.21.2.112:0/2792979982 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3b000b640 con 0x7fb3b80f0650 2021-02-10T11:38:46.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:46.764 7fb3affff700 1 -- 172.21.2.112:0/2792979982 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb3b000ba90 con 0x7fb3b80f0650 2021-02-10T11:38:46.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:46.764 7fb3bf504700 1 -- 172.21.2.112:0/2792979982 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb3b80f0650 msgr2=0x7fb3b80cfb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:46.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:46.764 7fb3bf504700 1 --2- 172.21.2.112:0/2792979982 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb3b80f0650 0x7fb3b80cfb20 secure :-1 s=READY pgs=1817 cs=0 l=1 rev1=1 rx=0x7fb3b0006f00 tx=0x7fb3b0009e90).stop 2021-02-10T11:38:46.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:46.765 7fb3bf504700 1 -- 172.21.2.112:0/2792979982 shutdown_connections 2021-02-10T11:38:46.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:46.765 7fb3bf504700 1 --2- 172.21.2.112:0/2792979982 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb3b80f0650 0x7fb3b80cfb20 unknown :-1 s=CLOSED pgs=1817 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:46.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:46.765 7fb3bf504700 1 --2- 172.21.2.112:0/2792979982 >> v2:172.21.2.120:3300/0 conn(0x7fb3b80d1870 0x7fb3b81b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:46.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:46.765 7fb3bf504700 1 -- 172.21.2.112:0/2792979982 >> 172.21.2.112:0/2792979982 conn(0x7fb3b80ce450 msgr2=0x7fb3b80d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:46.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:46.765 7fb3bf504700 1 -- 172.21.2.112:0/2792979982 shutdown_connections 2021-02-10T11:38:46.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:46.765 7fb3bf504700 1 -- 172.21.2.112:0/2792979982 wait complete. 2021-02-10T11:38:46.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:46.779 7fb3bf504700 1 Processor -- start 2021-02-10T11:38:46.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:46.779 7fb3bf504700 1 -- start start 2021-02-10T11:38:46.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:46.780 7fb3bf504700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb3b80b0ad0 0x7fb3b8091f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:46.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:46.780 7fb3bf504700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb3b81a8850 0x7fb3b81a8c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:46.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:46.780 7fb3bf504700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb3b81a6fa0 0x7fb3b81a73a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:46.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:46.780 7fb3bf504700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3b81abfe0 con 0x7fb3b81a6fa0 2021-02-10T11:38:46.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:46.780 7fb3bf504700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3b81a9c40 con 0x7fb3b80b0ad0 2021-02-10T11:38:46.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:46.780 7fb3bf504700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3b81a9730 con 0x7fb3b81a8850 2021-02-10T11:38:46.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:46.780 7fb3bdb21700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb3b81a6fa0 0x7fb3b81a73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:46.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:46.780 7fb3bdb21700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb3b81a6fa0 0x7fb3b81a73a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:53164/0 (socket says 172.21.2.112:53164) 2021-02-10T11:38:46.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:46.780 7fb3bdb21700 1 -- 172.21.2.112:0/3590973755 learned_addr learned my addr 172.21.2.112:0/3590973755 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:38:46.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:46.780 7fb3bdb21700 1 -- 172.21.2.112:0/3590973755 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb3b81a8850 msgr2=0x7fb3b81a8c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:46.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:46.780 7fb3bdb21700 1 --2- 172.21.2.112:0/3590973755 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb3b81a8850 0x7fb3b81a8c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:46.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:46.780 7fb3bdb21700 1 -- 172.21.2.112:0/3590973755 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb3b80b0ad0 msgr2=0x7fb3b8091f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:46.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:46.780 7fb3bdb21700 1 --2- 172.21.2.112:0/3590973755 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb3b80b0ad0 0x7fb3b8091f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:46.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:46.780 7fb3bdb21700 1 -- 172.21.2.112:0/3590973755 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3b81a9270 con 0x7fb3b81a6fa0 2021-02-10T11:38:46.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:46.781 7fb3bdb21700 1 --2- 172.21.2.112:0/3590973755 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb3b81a6fa0 0x7fb3b81a73a0 secure :-1 s=READY pgs=1818 cs=0 l=1 rev1=1 rx=0x7fb3b000b890 tx=0x7fb3b000bc30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:46.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:46.781 7fb3adffb700 1 -- 172.21.2.112:0/3590973755 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb3b000bd70 con 0x7fb3b81a6fa0 2021-02-10T11:38:46.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:46.781 7fb3adffb700 1 -- 172.21.2.112:0/3590973755 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3b0005820 con 0x7fb3b81a6fa0 2021-02-10T11:38:46.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:46.781 7fb3bf504700 1 -- 172.21.2.112:0/3590973755 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3b81a9270 con 0x7fb3b81a6fa0 2021-02-10T11:38:46.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:46.781 7fb3bf504700 1 -- 172.21.2.112:0/3590973755 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3b81a7cb0 con 0x7fb3b81a6fa0 2021-02-10T11:38:46.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:46.781 7fb3adffb700 1 -- 172.21.2.112:0/3590973755 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb3b0005cb0 con 0x7fb3b81a6fa0 2021-02-10T11:38:46.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:46.782 7fb3adffb700 1 -- 172.21.2.112:0/3590973755 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb3b000bd70 con 0x7fb3b81a6fa0 2021-02-10T11:38:46.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:46.782 7fb3adffb700 1 --2- 172.21.2.112:0/3590973755 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb3980186c0 0x7fb39801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:46.777 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:46.782 7fb3adffb700 1 -- 172.21.2.112:0/3590973755 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 4753+0+0 (secure 0 0 0) 0x7fb3b00325f0 con 0x7fb3b81a6fa0 2021-02-10T11:38:46.777 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:46.782 7fb3bd320700 1 --2- 172.21.2.112:0/3590973755 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb3980186c0 0x7fb39801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:46.777 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:46.783 7fb3bd320700 1 --2- 172.21.2.112:0/3590973755 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb3980186c0 0x7fb39801abc0 crc :-1 s=READY pgs=854 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:46.805 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:46.813 7fb3bf504700 1 -- 172.21.2.112:0/3590973755 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb390002fd0 con 0x7fb3b81a6fa0 2021-02-10T11:38:46.807 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:46.816 7fb3adffb700 1 -- 172.21.2.112:0/3590973755 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb3b0000f90 con 0x7fb3b81a6fa0 2021-02-10T11:38:46.941 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:46.950 7fb3bf504700 1 -- 172.21.2.112:0/3590973755 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fb390002de0 con 0x7fb3b81a6fa0 2021-02-10T11:38:46.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:46.950 7fb3adffb700 1 -- 172.21.2.112:0/3590973755 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v52) v1 ==== 80+0+4513 (secure 0 0 0) 0x7fb3b000bd70 con 0x7fb3b81a6fa0 2021-02-10T11:38:46.942 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:38:46.942 INFO:teuthology.orchestra.run.gibba012.stdout:{ 2021-02-10T11:38:46.943 INFO:teuthology.orchestra.run.gibba012.stdout: "auth_dump": [ 2021-02-10T11:38:46.943 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:38:46.943 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "mds.a", 2021-02-10T11:38:46.943 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCKvyNgPkV4AxAAEmVNaghdccP6lWd8hvQnxw==", 2021-02-10T11:38:46.943 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:38:46.943 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:38:46.943 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:38:46.943 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:38:46.943 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:38:46.943 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:38:46.943 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:38:46.944 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:38:46.944 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "mds.b", 2021-02-10T11:38:46.944 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCKvyNguxONLhAAHIpry+R89MAwdhFJyEhVCw==", 2021-02-10T11:38:46.944 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:38:46.944 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:38:46.944 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:38:46.944 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:38:46.944 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:38:46.944 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:38:46.944 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:38:46.944 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:38:46.944 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.0", 2021-02-10T11:38:46.945 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCSvyNgXcKPLhAAP/Bj2F+06DCtPCzQDNq+cQ==", 2021-02-10T11:38:46.945 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:38:46.945 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:38:46.945 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:38:46.945 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:38:46.945 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:38:46.945 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:38:46.945 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:38:46.945 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.1", 2021-02-10T11:38:46.945 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCVvyNghfbcLhAA66PrRwR2M7o6VTF5UhU/Fg==", 2021-02-10T11:38:46.945 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:38:46.945 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:38:46.946 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:38:46.946 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:38:46.946 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:38:46.946 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:38:46.946 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:38:46.946 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.2", 2021-02-10T11:38:46.946 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCYvyNgwiPqLhAA66BVkRsgaUlGpAsQ15Nf8Q==", 2021-02-10T11:38:46.946 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:38:46.946 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:38:46.946 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:38:46.946 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:38:46.946 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:38:46.947 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:38:46.947 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:38:46.947 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.3", 2021-02-10T11:38:46.947 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCbvyNgqBZrLxAAERPe/1L3SXAEzI3d9ReHHg==", 2021-02-10T11:38:46.947 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:38:46.947 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:38:46.947 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:38:46.947 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:38:46.947 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:38:46.947 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:38:46.947 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:38:46.947 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.4", 2021-02-10T11:38:46.947 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCkvyNgDQsxIhAAHmVjVYnx20N4SDmKqvbcuA==", 2021-02-10T11:38:46.948 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:38:46.948 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:38:46.948 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:38:46.948 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:38:46.948 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:38:46.948 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:38:46.948 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:38:46.948 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.5", 2021-02-10T11:38:46.948 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCnvyNg8ITFIhAADd4aKybAV7btoGPx3kCg3A==", 2021-02-10T11:38:46.948 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:38:46.948 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:38:46.948 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:38:46.948 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:38:46.948 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:38:46.949 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:38:46.949 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:38:46.949 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.6", 2021-02-10T11:38:46.949 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCqvyNgxOhDIxAAzXixEa/nlDc1xXrg0aeLwg==", 2021-02-10T11:38:46.949 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:38:46.949 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:38:46.949 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:38:46.949 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:38:46.949 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:38:46.949 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:38:46.949 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:38:46.949 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.7", 2021-02-10T11:38:46.949 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCtvyNgxSqGIxAACxJQlwP7DpiVVCZHA3zn1A==", 2021-02-10T11:38:46.949 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:38:46.950 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:38:46.950 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:38:46.950 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:38:46.950 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:38:46.950 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:38:46.950 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:38:46.950 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.0", 2021-02-10T11:38:46.950 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCLvyNgDkNiHRAAygU8Za4PBusTqCgwAYWTiQ==", 2021-02-10T11:38:46.950 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:38:46.950 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:38:46.950 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow r", 2021-02-10T11:38:46.950 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:38:46.951 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:38:46.951 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:38:46.951 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:38:46.951 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.1", 2021-02-10T11:38:46.951 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCLvyNgxmLqMxAA1MlSDJ9LH9u4f2sL7mm8/Q==", 2021-02-10T11:38:46.951 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:38:46.951 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:38:46.951 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow r", 2021-02-10T11:38:46.951 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:38:46.951 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:38:46.951 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:38:46.951 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:38:46.952 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.2", 2021-02-10T11:38:46.952 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCMvyNgp8veDhAAniSVHYPtwwTog/pjbaRQmw==", 2021-02-10T11:38:46.952 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:38:46.952 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:38:46.952 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow r", 2021-02-10T11:38:46.952 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:38:46.952 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:38:46.952 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:38:46.952 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:38:46.952 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.3", 2021-02-10T11:38:46.952 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCMvyNgQEVoJRAApHcWdm1sldIxu+heyD4HdQ==", 2021-02-10T11:38:46.952 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:38:46.952 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:38:46.952 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow r", 2021-02-10T11:38:46.953 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:38:46.953 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:38:46.953 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:38:46.953 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:38:46.953 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.admin", 2021-02-10T11:38:46.953 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCGvyNg7jjJBRAAtot+ypUZOiTo1GbY+oVhVw==", 2021-02-10T11:38:46.953 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:38:46.953 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow *", 2021-02-10T11:38:46.953 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:38:46.953 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:38:46.953 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:38:46.953 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:38:46.953 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:38:46.954 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:38:46.954 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "mgr.x", 2021-02-10T11:38:46.954 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCJvyNgTPVwKBAAjduiBE8+RSXBv82tdPecWg==", 2021-02-10T11:38:46.954 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:38:46.954 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow *", 2021-02-10T11:38:46.954 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow profile mgr", 2021-02-10T11:38:46.954 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:38:46.954 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:38:46.954 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:38:46.954 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:38:46.954 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "mgr.y", 2021-02-10T11:38:46.954 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCJvyNgQ1tTDhAASwBssDy+Rz31ujw8fNlWow==", 2021-02-10T11:38:46.954 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:38:46.955 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow *", 2021-02-10T11:38:46.955 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow profile mgr", 2021-02-10T11:38:46.955 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:38:46.955 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:38:46.955 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:38:46.955 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:38:46.955 INFO:teuthology.orchestra.run.gibba012.stdout:} 2021-02-10T11:38:46.956 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:46.952 7fb3bf504700 1 -- 172.21.2.112:0/3590973755 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb3980186c0 msgr2=0x7fb39801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:46.956 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:46.952 7fb3bf504700 1 --2- 172.21.2.112:0/3590973755 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb3980186c0 0x7fb39801abc0 crc :-1 s=READY pgs=854 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:46.956 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:46.952 7fb3bf504700 1 -- 172.21.2.112:0/3590973755 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb3b81a6fa0 msgr2=0x7fb3b81a73a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:46.956 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:46.952 7fb3bf504700 1 --2- 172.21.2.112:0/3590973755 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb3b81a6fa0 0x7fb3b81a73a0 secure :-1 s=READY pgs=1818 cs=0 l=1 rev1=1 rx=0x7fb3b000b890 tx=0x7fb3b000bc30).stop 2021-02-10T11:38:46.957 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:46.952 7fb3bf504700 1 -- 172.21.2.112:0/3590973755 shutdown_connections 2021-02-10T11:38:46.957 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:46.952 7fb3bf504700 1 --2- 172.21.2.112:0/3590973755 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb3980186c0 0x7fb39801abc0 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:46.957 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:46.952 7fb3bf504700 1 --2- 172.21.2.112:0/3590973755 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb3b81a6fa0 0x7fb3b81a73a0 unknown :-1 s=CLOSED pgs=1818 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:46.957 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:46.952 7fb3bf504700 1 --2- 172.21.2.112:0/3590973755 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb3b81a8850 0x7fb3b81a8c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:46.957 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:46.952 7fb3bf504700 1 --2- 172.21.2.112:0/3590973755 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb3b80b0ad0 0x7fb3b8091f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:46.957 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:46.952 7fb3bf504700 1 -- 172.21.2.112:0/3590973755 >> 172.21.2.112:0/3590973755 conn(0x7fb3b819f8f0 msgr2=0x7fb3b81a5f80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:46.957 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:46.952 7fb3bf504700 1 -- 172.21.2.112:0/3590973755 shutdown_connections 2021-02-10T11:38:46.957 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:46.952 7fb3bf504700 1 -- 172.21.2.112:0/3590973755 wait complete. 2021-02-10T11:38:46.958 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:38:47.384 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.376 7f9af613c700 1 Processor -- start 2021-02-10T11:38:47.384 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.376 7f9af613c700 1 -- start start 2021-02-10T11:38:47.384 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.376 7f9af613c700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f9af00d1870 0x7f9af01b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:47.384 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.376 7f9af613c700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9af00f0b10 con 0x7f9af01b0a10 2021-02-10T11:38:47.385 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.376 7f9af613c700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9af00d0310 con 0x7f9af00f0650 2021-02-10T11:38:47.385 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.376 7f9af613c700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9af00d0530 con 0x7f9af00d1870 2021-02-10T11:38:47.385 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.377 7f9aeeffd700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f9af01b0a10 0x7f9af00cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:52630/0 (socket says 172.21.2.112:52630) 2021-02-10T11:38:47.385 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.377 7f9aeeffd700 1 -- 172.21.2.112:0/4082525343 learned_addr learned my addr 172.21.2.112:0/4082525343 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:38:47.385 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.377 7f9aef7fe700 1 --2- 172.21.2.112:0/4082525343 >> v2:172.21.2.120:3300/0 conn(0x7f9af00d1870 0x7f9af01b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:47.385 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.377 7f9aee7fc700 1 -- 172.21.2.112:0/4082525343 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3093300481 0 0) 0x7f9ae8001400 con 0x7f9af00f0650 2021-02-10T11:38:47.385 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.377 7f9aef7fe700 1 -- 172.21.2.112:0/4082525343 >> v1:172.21.2.120:6789/0 conn(0x7f9af00f0650 legacy=0x7f9af00cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:47.385 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.377 7f9aef7fe700 1 -- 172.21.2.112:0/4082525343 >> v1:172.21.2.112:6789/0 conn(0x7f9af01b0a10 legacy=0x7f9af00cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:47.385 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.377 7f9aef7fe700 1 -- 172.21.2.112:0/4082525343 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9af00d1c70 con 0x7f9af00d1870 2021-02-10T11:38:47.385 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.377 7f9aef7fe700 1 --2- 172.21.2.112:0/4082525343 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9af00d1870 0x7f9af01b03f0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 rx=0x7f9ae0006650 tx=0x7f9ae0009a80).ready entity=mon.1 client_cookie=5765f2596e7d3a9e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:47.385 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.377 7f9aee7fc700 1 -- 172.21.2.112:0/4082525343 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9ae000c0c0 con 0x7f9af00d1870 2021-02-10T11:38:47.385 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.377 7f9aee7fc700 1 -- 172.21.2.112:0/4082525343 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ae000c340 con 0x7f9af00d1870 2021-02-10T11:38:47.386 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.378 7f9aee7fc700 1 -- 172.21.2.112:0/4082525343 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9ae000c820 con 0x7f9af00d1870 2021-02-10T11:38:47.386 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.378 7f9af613c700 1 -- 172.21.2.112:0/4082525343 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9af00d1870 msgr2=0x7f9af01b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:47.386 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.378 7f9af613c700 1 --2- 172.21.2.112:0/4082525343 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9af00d1870 0x7f9af01b03f0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 rx=0x7f9ae0006650 tx=0x7f9ae0009a80).stop 2021-02-10T11:38:47.386 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.378 7f9af613c700 1 -- 172.21.2.112:0/4082525343 shutdown_connections 2021-02-10T11:38:47.386 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.378 7f9af613c700 1 --2- 172.21.2.112:0/4082525343 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9af00d1870 0x7f9af01b03f0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:47.386 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.378 7f9af613c700 1 -- 172.21.2.112:0/4082525343 >> 172.21.2.112:0/4082525343 conn(0x7f9af00ce450 msgr2=0x7f9af00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:47.386 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.378 7f9af613c700 1 -- 172.21.2.112:0/4082525343 shutdown_connections 2021-02-10T11:38:47.386 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.378 7f9af613c700 1 -- 172.21.2.112:0/4082525343 wait complete. 2021-02-10T11:38:47.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.393 7f9af613c700 1 Processor -- start 2021-02-10T11:38:47.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.393 7f9af613c700 1 -- start start 2021-02-10T11:38:47.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.393 7f9af613c700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9af00b0ad0 0x7f9af0091f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:47.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.393 7f9af613c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9af01a8850 0x7f9af01a8c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:47.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.393 7f9af613c700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9af01a6fa0 0x7f9af01a73a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:47.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.393 7f9af613c700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9af01abfe0 con 0x7f9af01a8850 2021-02-10T11:38:47.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.393 7f9af613c700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9af01a9c40 con 0x7f9af00b0ad0 2021-02-10T11:38:47.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.393 7f9af613c700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9af01a9730 con 0x7f9af01a6fa0 2021-02-10T11:38:47.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.393 7f9aeeffd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9af01a8850 0x7f9af01a8c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:47.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.393 7f9aeeffd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9af01a8850 0x7f9af01a8c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:53172/0 (socket says 172.21.2.112:53172) 2021-02-10T11:38:47.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.393 7f9aeeffd700 1 -- 172.21.2.112:0/1311791073 learned_addr learned my addr 172.21.2.112:0/1311791073 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:38:47.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.394 7f9aeeffd700 1 -- 172.21.2.112:0/1311791073 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9af01a6fa0 msgr2=0x7f9af01a73a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:47.389 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.394 7f9aeeffd700 1 --2- 172.21.2.112:0/1311791073 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9af01a6fa0 0x7f9af01a73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:47.389 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.394 7f9aeeffd700 1 -- 172.21.2.112:0/1311791073 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9af00b0ad0 msgr2=0x7f9af0091f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:47.389 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.394 7f9aeeffd700 1 --2- 172.21.2.112:0/1311791073 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9af00b0ad0 0x7f9af0091f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:47.389 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.394 7f9aeeffd700 1 -- 172.21.2.112:0/1311791073 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9af01a9270 con 0x7f9af01a8850 2021-02-10T11:38:47.389 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.394 7f9aeeffd700 1 --2- 172.21.2.112:0/1311791073 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9af01a8850 0x7f9af01a8c50 secure :-1 s=READY pgs=1820 cs=0 l=1 rev1=1 rx=0x7f9ad8001be0 tx=0x7f9ad800a900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:47.389 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.394 7f9ae7fff700 1 -- 172.21.2.112:0/1311791073 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9ad800acf0 con 0x7f9af01a8850 2021-02-10T11:38:47.389 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.394 7f9ae7fff700 1 -- 172.21.2.112:0/1311791073 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ad8004140 con 0x7f9af01a8850 2021-02-10T11:38:47.389 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.394 7f9af613c700 1 -- 172.21.2.112:0/1311791073 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9af01a9270 con 0x7f9af01a8850 2021-02-10T11:38:47.389 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.394 7f9af613c700 1 -- 172.21.2.112:0/1311791073 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9af01a7cb0 con 0x7f9af01a8850 2021-02-10T11:38:47.389 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.394 7f9ae7fff700 1 -- 172.21.2.112:0/1311791073 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9ad8004620 con 0x7f9af01a8850 2021-02-10T11:38:47.389 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.395 7f9ae7fff700 1 -- 172.21.2.112:0/1311791073 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9ad8000cc0 con 0x7f9af01a8850 2021-02-10T11:38:47.389 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.395 7f9ae7fff700 1 --2- 172.21.2.112:0/1311791073 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9ad40186c0 0x7f9ad401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:47.389 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.395 7f9ae7fff700 1 -- 172.21.2.112:0/1311791073 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f9ad8032210 con 0x7f9af01a8850 2021-02-10T11:38:47.389 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.395 7f9aef7fe700 1 --2- 172.21.2.112:0/1311791073 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9ad40186c0 0x7f9ad401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:47.390 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.395 7f9aef7fe700 1 --2- 172.21.2.112:0/1311791073 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9ad40186c0 0x7f9ad401abc0 crc :-1 s=READY pgs=855 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:47.418 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.426 7f9af613c700 1 -- 172.21.2.112:0/1311791073 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9af00ce450 con 0x7f9af01a8850 2021-02-10T11:38:47.420 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.429 7f9ae7fff700 1 -- 172.21.2.112:0/1311791073 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9ad8000ff0 con 0x7f9af01a8850 2021-02-10T11:38:47.545 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.553 7f9af613c700 1 -- 172.21.2.112:0/1311791073 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9af00ce450 con 0x7f9af01a8850 2021-02-10T11:38:47.546 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.554 7f9ae7fff700 1 -- 172.21.2.112:0/1311791073 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+11145 (secure 0 0 0) 0x7f9ad8000cc0 con 0x7f9af01a8850 2021-02-10T11:38:47.546 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:38:47.547 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":87,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:38:46.309786","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:38:47.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.556 7f9af613c700 1 -- 172.21.2.112:0/1311791073 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9ad40186c0 msgr2=0x7f9ad401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:47.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.556 7f9af613c700 1 --2- 172.21.2.112:0/1311791073 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9ad40186c0 0x7f9ad401abc0 crc :-1 s=READY pgs=855 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:47.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.556 7f9af613c700 1 -- 172.21.2.112:0/1311791073 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9af01a8850 msgr2=0x7f9af01a8c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:47.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.556 7f9af613c700 1 --2- 172.21.2.112:0/1311791073 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9af01a8850 0x7f9af01a8c50 secure :-1 s=READY pgs=1820 cs=0 l=1 rev1=1 rx=0x7f9ad8001be0 tx=0x7f9ad800a900).stop 2021-02-10T11:38:47.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.556 7f9af613c700 1 -- 172.21.2.112:0/1311791073 shutdown_connections 2021-02-10T11:38:47.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.556 7f9af613c700 1 --2- 172.21.2.112:0/1311791073 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9ad40186c0 0x7f9ad401abc0 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:47.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.556 7f9af613c700 1 --2- 172.21.2.112:0/1311791073 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9af01a6fa0 0x7f9af01a73a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:47.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.556 7f9af613c700 1 --2- 172.21.2.112:0/1311791073 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9af01a8850 0x7f9af01a8c50 unknown :-1 s=CLOSED pgs=1820 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:47.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.556 7f9af613c700 1 --2- 172.21.2.112:0/1311791073 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9af00b0ad0 0x7f9af0091f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:47.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.556 7f9af613c700 1 -- 172.21.2.112:0/1311791073 >> 172.21.2.112:0/1311791073 conn(0x7f9af019f8f0 msgr2=0x7f9af01a5f80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:47.549 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.556 7f9af613c700 1 -- 172.21.2.112:0/1311791073 shutdown_connections 2021-02-10T11:38:47.549 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.556 7f9af613c700 1 -- 172.21.2.112:0/1311791073 wait complete. 2021-02-10T11:38:47.556 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:13:57.508758', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2021-02-10T11:38:47.557 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:38:47.988 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.980 7f72450a9700 1 Processor -- start 2021-02-10T11:38:47.988 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.980 7f72450a9700 1 -- start start 2021-02-10T11:38:47.989 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.980 7f72450a9700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f72400d1860 0x7f72401b04b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:47.989 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.980 7f72450a9700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f72401b0ad0 0x7f7240108930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:47.989 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.980 7f72450a9700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7240109a50 con 0x7f72400d1c60 2021-02-10T11:38:47.989 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.980 7f72450a9700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72400e5ee0 con 0x7f72401b0ad0 2021-02-10T11:38:47.989 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.980 7f72450a9700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7240109740 con 0x7f72400d1860 2021-02-10T11:38:47.989 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.981 7f723ed9d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f72400d1860 0x7f72401b04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:47.989 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.981 7f723e59c700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f72401b0ad0 0x7f7240108930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:47.989 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.981 7f723ed9d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f72400d1860 0x7f72401b04b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3301/0 says I am v2:172.21.2.112:52862/0 (socket says 172.21.2.112:52862) 2021-02-10T11:38:47.989 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.981 7f723e59c700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f72401b0ad0 0x7f7240108930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3300/0 says I am v2:172.21.2.112:55462/0 (socket says 172.21.2.112:55462) 2021-02-10T11:38:47.989 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.981 7f723ed9d700 1 -- 172.21.2.112:0/2868843253 learned_addr learned my addr 172.21.2.112:0/2868843253 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:38:47.989 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.981 7f723ed9d700 1 -- 172.21.2.112:0/2868843253 >> v2:172.21.2.120:3300/0 conn(0x7f72401b0ad0 msgr2=0x7f7240108930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:47.990 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.981 7f723ed9d700 1 --2- 172.21.2.112:0/2868843253 >> v2:172.21.2.120:3300/0 conn(0x7f72401b0ad0 0x7f7240108930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:47.990 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.981 7f723ed9d700 1 -- 172.21.2.112:0/2868843253 >> v1:172.21.2.120:6789/0 conn(0x7f72400d1c60 legacy=0x7f7240108f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:47.990 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.981 7f723ed9d700 1 -- 172.21.2.112:0/2868843253 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7240088070 con 0x7f72400d1860 2021-02-10T11:38:47.990 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.981 7f723ed9d700 1 --2- 172.21.2.112:0/2868843253 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f72400d1860 0x7f72401b04b0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f7230006700 tx=0x7f7230009b00).ready entity=mon.2 client_cookie=48c4b1b8afce1a12 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:47.990 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.981 7f723dd9b700 1 -- 172.21.2.112:0/2868843253 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f723000c0c0 con 0x7f72400d1860 2021-02-10T11:38:47.990 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.981 7f723dd9b700 1 -- 172.21.2.112:0/2868843253 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f723000c400 con 0x7f72400d1860 2021-02-10T11:38:47.990 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.982 7f723dd9b700 1 -- 172.21.2.112:0/2868843253 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f723000c860 con 0x7f72400d1860 2021-02-10T11:38:47.990 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.982 7f72450a9700 1 -- 172.21.2.112:0/2868843253 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f72400d1860 msgr2=0x7f72401b04b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:47.990 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.982 7f72450a9700 1 --2- 172.21.2.112:0/2868843253 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f72400d1860 0x7f72401b04b0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f7230006700 tx=0x7f7230009b00).stop 2021-02-10T11:38:47.990 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.982 7f72450a9700 1 -- 172.21.2.112:0/2868843253 shutdown_connections 2021-02-10T11:38:47.990 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.982 7f72450a9700 1 --2- 172.21.2.112:0/2868843253 >> v2:172.21.2.120:3300/0 conn(0x7f72401b0ad0 0x7f7240108930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:47.990 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.982 7f72450a9700 1 --2- 172.21.2.112:0/2868843253 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f72400d1860 0x7f72401b04b0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:47.991 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.982 7f72450a9700 1 -- 172.21.2.112:0/2868843253 >> 172.21.2.112:0/2868843253 conn(0x7f72400ce410 msgr2=0x7f72400d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:47.991 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.982 7f72450a9700 1 -- 172.21.2.112:0/2868843253 shutdown_connections 2021-02-10T11:38:47.991 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.982 7f72450a9700 1 -- 172.21.2.112:0/2868843253 wait complete. 2021-02-10T11:38:47.991 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.997 7f72450a9700 1 Processor -- start 2021-02-10T11:38:47.991 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.997 7f72450a9700 1 -- start start 2021-02-10T11:38:47.992 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.997 7f72450a9700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f72400b6da0 0x7f72400b71a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:47.992 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.997 7f72450a9700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f72400a70d0 0x7f72400b77c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:47.992 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.997 7f72450a9700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f72400b8780 0x7f72400b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:47.992 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.997 7f72450a9700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72400a7610 con 0x7f72400b8780 2021-02-10T11:38:47.993 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.997 7f72450a9700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72400b7de0 con 0x7f72400b6da0 2021-02-10T11:38:47.993 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.997 7f72450a9700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72400b8000 con 0x7f72400a70d0 2021-02-10T11:38:47.993 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.997 7f723ed9d700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f72400b6da0 0x7f72400b71a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:47.993 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.997 7f723ed9d700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f72400b6da0 0x7f72400b71a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3300/0 says I am v2:172.21.2.112:55466/0 (socket says 172.21.2.112:55466) 2021-02-10T11:38:47.993 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.997 7f723ed9d700 1 -- 172.21.2.112:0/2873938500 learned_addr learned my addr 172.21.2.112:0/2873938500 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:38:47.993 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.997 7f723e59c700 1 --2- 172.21.2.112:0/2873938500 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f72400a70d0 0x7f72400b77c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:47.993 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.997 7f723f59e700 1 --2- 172.21.2.112:0/2873938500 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f72400b8780 0x7f72400b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:47.993 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.998 7f723ed9d700 1 -- 172.21.2.112:0/2873938500 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f72400a70d0 msgr2=0x7f72400b77c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:47.993 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.998 7f723ed9d700 1 --2- 172.21.2.112:0/2873938500 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f72400a70d0 0x7f72400b77c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:47.993 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.998 7f723ed9d700 1 -- 172.21.2.112:0/2873938500 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f72400b8780 msgr2=0x7f72400b8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:47.993 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.998 7f723ed9d700 1 --2- 172.21.2.112:0/2873938500 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f72400b8780 0x7f72400b8b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:47.993 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.998 7f723ed9d700 1 -- 172.21.2.112:0/2873938500 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72400b91a0 con 0x7f72400b6da0 2021-02-10T11:38:47.993 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.998 7f723ed9d700 1 --2- 172.21.2.112:0/2873938500 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f72400b6da0 0x7f72400b71a0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 rx=0x7f7230006f20 tx=0x7f723000b960).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:47.994 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.998 7f722f7fe700 1 -- 172.21.2.112:0/2873938500 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f723000bd50 con 0x7f72400b6da0 2021-02-10T11:38:47.994 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.998 7f722f7fe700 1 -- 172.21.2.112:0/2873938500 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7230002080 con 0x7f72400b6da0 2021-02-10T11:38:47.994 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.998 7f72450a9700 1 -- 172.21.2.112:0/2873938500 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72400b91a0 con 0x7f72400b6da0 2021-02-10T11:38:47.994 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.998 7f722f7fe700 1 -- 172.21.2.112:0/2873938500 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7230009440 con 0x7f72400b6da0 2021-02-10T11:38:47.994 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.998 7f72450a9700 1 -- 172.21.2.112:0/2873938500 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72400b91a0 con 0x7f72400b6da0 2021-02-10T11:38:47.994 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.999 7f722f7fe700 1 -- 172.21.2.112:0/2873938500 <== mon.1 v2:172.21.2.120:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7230000c90 con 0x7f72400b6da0 2021-02-10T11:38:47.994 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.999 7f722f7fe700 1 --2- 172.21.2.112:0/2873938500 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f72240186c0 0x7f722401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:47.994 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.999 7f722f7fe700 1 -- 172.21.2.112:0/2873938500 <== mon.1 v2:172.21.2.120:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f7230032100 con 0x7f72400b6da0 2021-02-10T11:38:47.994 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:47.999 7f723e59c700 1 --2- 172.21.2.112:0/2873938500 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f72240186c0 0x7f722401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:47.994 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:48.000 7f723e59c700 1 --2- 172.21.2.112:0/2873938500 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f72240186c0 0x7f722401abc0 crc :-1 s=READY pgs=856 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:48.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:48.030 7f72450a9700 1 -- 172.21.2.112:0/2873938500 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7240082920 con 0x7f72400b6da0 2021-02-10T11:38:48.025 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:48.033 7f722f7fe700 1 -- 172.21.2.112:0/2873938500 <== mon.1 v2:172.21.2.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7230006390 con 0x7f72400b6da0 2021-02-10T11:38:48.153 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:48.161 7f72450a9700 1 -- 172.21.2.112:0/2873938500 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f7240109a50 con 0x7f72400b6da0 2021-02-10T11:38:48.153 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:48.162 7f722f7fe700 1 -- 172.21.2.112:0/2873938500 <== mon.1 v2:172.21.2.120:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v87) v1 ==== 92+0+10 (secure 0 0 0) 0x7f7230000c90 con 0x7f72400b6da0 2021-02-10T11:38:48.154 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:38:48.155 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:48.163 7f72450a9700 1 -- 172.21.2.112:0/2873938500 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f72240186c0 msgr2=0x7f722401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:48.155 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:48.163 7f72450a9700 1 --2- 172.21.2.112:0/2873938500 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f72240186c0 0x7f722401abc0 crc :-1 s=READY pgs=856 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:48.155 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:48.163 7f72450a9700 1 -- 172.21.2.112:0/2873938500 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f72400b6da0 msgr2=0x7f72400b71a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:48.155 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:48.163 7f72450a9700 1 --2- 172.21.2.112:0/2873938500 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f72400b6da0 0x7f72400b71a0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 rx=0x7f7230006f20 tx=0x7f723000b960).stop 2021-02-10T11:38:48.156 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:48.163 7f72450a9700 1 -- 172.21.2.112:0/2873938500 shutdown_connections 2021-02-10T11:38:48.156 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:48.163 7f72450a9700 1 --2- 172.21.2.112:0/2873938500 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f72240186c0 0x7f722401abc0 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:48.156 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:48.163 7f72450a9700 1 --2- 172.21.2.112:0/2873938500 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f72400b8780 0x7f72400b8b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:48.156 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:48.163 7f72450a9700 1 --2- 172.21.2.112:0/2873938500 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f72400a70d0 0x7f72400b77c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:48.156 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:48.163 7f72450a9700 1 --2- 172.21.2.112:0/2873938500 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f72400b6da0 0x7f72400b71a0 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:48.156 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:48.163 7f72450a9700 1 -- 172.21.2.112:0/2873938500 >> 172.21.2.112:0/2873938500 conn(0x7f7240092c70 msgr2=0x7f7240093530 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:48.156 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:48.164 7f72450a9700 1 -- 172.21.2.112:0/2873938500 shutdown_connections 2021-02-10T11:38:48.156 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:48.164 7f72450a9700 1 -- 172.21.2.112:0/2873938500 wait complete. 2021-02-10T11:38:48.164 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --format=json-pretty osd lspools 2021-02-10T11:38:48.590 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:48.582 7f82baca5700 1 Processor -- start 2021-02-10T11:38:48.590 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:48.582 7f82baca5700 1 -- start start 2021-02-10T11:38:48.590 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:48.583 7f82baca5700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f82b40d1870 0x7f82b41b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:48.590 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:48.583 7f82baca5700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82b40f0b10 con 0x7f82b41b0a10 2021-02-10T11:38:48.590 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:48.583 7f82baca5700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82b40d0310 con 0x7f82b40f0650 2021-02-10T11:38:48.590 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:48.583 7f82baca5700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82b40d0530 con 0x7f82b40d1870 2021-02-10T11:38:48.591 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:48.583 7f82b8ac1700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f82b40d1870 0x7f82b41b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:48.591 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:48.583 7f82b3fff700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f82b41b0a10 0x7f82b40cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:52656/0 (socket says 172.21.2.112:52656) 2021-02-10T11:38:48.591 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:48.583 7f82b3fff700 1 -- 172.21.2.112:0/3914713496 learned_addr learned my addr 172.21.2.112:0/3914713496 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:38:48.591 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:48.583 7f82b8ac1700 1 -- 172.21.2.112:0/3914713496 >> v1:172.21.2.120:6789/0 conn(0x7f82b40f0650 legacy=0x7f82b40cfb20 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:38:48.591 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:48.583 7f82b8ac1700 1 -- 172.21.2.112:0/3914713496 >> v1:172.21.2.112:6789/0 conn(0x7f82b41b0a10 legacy=0x7f82b40cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:48.591 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:48.583 7f82b8ac1700 1 -- 172.21.2.112:0/3914713496 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82b40d1c70 con 0x7f82b40d1870 2021-02-10T11:38:48.591 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:48.583 7f82b8ac1700 1 --2- 172.21.2.112:0/3914713496 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f82b40d1870 0x7f82b41b03f0 secure :-1 s=READY pgs=1822 cs=0 l=1 rev1=1 rx=0x7f829c006650 tx=0x7f829c009a00).ready entity=mon.0 client_cookie=fbdda0e1b767fbd1 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:48.591 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:48.583 7f82b37fe700 1 -- 172.21.2.112:0/3914713496 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f829c009df0 con 0x7f82b40d1870 2021-02-10T11:38:48.591 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:48.583 7f82b37fe700 1 -- 172.21.2.112:0/3914713496 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f829c0009b0 con 0x7f82b40d1870 2021-02-10T11:38:48.591 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:48.583 7f82b37fe700 1 -- 172.21.2.112:0/3914713496 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f829c00c2b0 con 0x7f82b40d1870 2021-02-10T11:38:48.591 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:48.583 7f82baca5700 1 -- 172.21.2.112:0/3914713496 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f82b40d1870 msgr2=0x7f82b41b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:48.591 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:48.583 7f82baca5700 1 --2- 172.21.2.112:0/3914713496 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f82b40d1870 0x7f82b41b03f0 secure :-1 s=READY pgs=1822 cs=0 l=1 rev1=1 rx=0x7f829c006650 tx=0x7f829c009a00).stop 2021-02-10T11:38:48.592 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:48.584 7f82baca5700 1 -- 172.21.2.112:0/3914713496 shutdown_connections 2021-02-10T11:38:48.592 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:48.584 7f82baca5700 1 --2- 172.21.2.112:0/3914713496 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f82b40d1870 0x7f82b41b03f0 unknown :-1 s=CLOSED pgs=1822 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:48.592 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:48.584 7f82baca5700 1 -- 172.21.2.112:0/3914713496 >> 172.21.2.112:0/3914713496 conn(0x7f82b40ce450 msgr2=0x7f82b40d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:48.592 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:48.584 7f82baca5700 1 -- 172.21.2.112:0/3914713496 shutdown_connections 2021-02-10T11:38:48.592 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:48.584 7f82baca5700 1 -- 172.21.2.112:0/3914713496 wait complete. 2021-02-10T11:38:48.593 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:48.598 7f82baca5700 1 Processor -- start 2021-02-10T11:38:48.593 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:48.599 7f82baca5700 1 -- start start 2021-02-10T11:38:48.593 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:48.599 7f82baca5700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f82b409bca0 0x7f82b409c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:48.594 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:48.599 7f82baca5700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f82b409c6c0 0x7f82b409cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:48.594 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:48.599 7f82baca5700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f82b408c1a0 0x7f82b408c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:48.594 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:48.599 7f82baca5700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82b409b720 con 0x7f82b409bca0 2021-02-10T11:38:48.594 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:48.599 7f82baca5700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82b40abcd0 con 0x7f82b408c1a0 2021-02-10T11:38:48.594 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:48.599 7f82baca5700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82b408cbc0 con 0x7f82b409c6c0 2021-02-10T11:38:48.594 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:48.599 7f82b8ac1700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f82b409bca0 0x7f82b409c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:48.594 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:48.599 7f82b8ac1700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f82b409bca0 0x7f82b409c0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:53202/0 (socket says 172.21.2.112:53202) 2021-02-10T11:38:48.594 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:48.599 7f82b8ac1700 1 -- 172.21.2.112:0/910326119 learned_addr learned my addr 172.21.2.112:0/910326119 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:38:48.594 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:48.599 7f82b8ac1700 1 -- 172.21.2.112:0/910326119 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f82b409c6c0 msgr2=0x7f82b409cac0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:38:48.594 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:48.599 7f82b8ac1700 1 --2- 172.21.2.112:0/910326119 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f82b409c6c0 0x7f82b409cac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:48.594 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:48.599 7f82b8ac1700 1 -- 172.21.2.112:0/910326119 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f82b408c1a0 msgr2=0x7f82b408c5a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:38:48.594 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:48.599 7f82b8ac1700 1 --2- 172.21.2.112:0/910326119 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f82b408c1a0 0x7f82b408c5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:48.595 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:48.599 7f82b8ac1700 1 -- 172.21.2.112:0/910326119 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82b408cde0 con 0x7f82b409bca0 2021-02-10T11:38:48.595 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:48.600 7f82b8ac1700 1 --2- 172.21.2.112:0/910326119 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f82b409bca0 0x7f82b409c0a0 secure :-1 s=READY pgs=1823 cs=0 l=1 rev1=1 rx=0x7f829c0096a0 tx=0x7f829c00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:48.595 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:48.600 7f82b17fa700 1 -- 172.21.2.112:0/910326119 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f829c00bd80 con 0x7f82b409bca0 2021-02-10T11:38:48.595 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:48.600 7f82b17fa700 1 -- 172.21.2.112:0/910326119 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f829c006030 con 0x7f82b409bca0 2021-02-10T11:38:48.595 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:48.600 7f82b17fa700 1 -- 172.21.2.112:0/910326119 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f829c005560 con 0x7f82b409bca0 2021-02-10T11:38:48.595 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:48.600 7f82baca5700 1 -- 172.21.2.112:0/910326119 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82b408cde0 con 0x7f82b409bca0 2021-02-10T11:38:48.595 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:48.600 7f82baca5700 1 -- 172.21.2.112:0/910326119 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82b408cde0 con 0x7f82b409bca0 2021-02-10T11:38:48.595 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:48.600 7f82b17fa700 1 -- 172.21.2.112:0/910326119 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f829c0052e0 con 0x7f82b409bca0 2021-02-10T11:38:48.595 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:48.601 7f82b17fa700 1 --2- 172.21.2.112:0/910326119 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f82980186c0 0x7f829801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:48.595 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:48.601 7f82b17fa700 1 -- 172.21.2.112:0/910326119 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f829c032340 con 0x7f82b409bca0 2021-02-10T11:38:48.595 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:48.601 7f82b3fff700 1 --2- 172.21.2.112:0/910326119 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f82980186c0 0x7f829801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:48.596 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:48.602 7f82b3fff700 1 --2- 172.21.2.112:0/910326119 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f82980186c0 0x7f829801abc0 crc :-1 s=READY pgs=857 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:48.624 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:48.632 7f82baca5700 1 -- 172.21.2.112:0/910326119 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82b40d1ee0 con 0x7f82b409bca0 2021-02-10T11:38:48.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:48.635 7f82b17fa700 1 -- 172.21.2.112:0/910326119 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f829c0009c0 con 0x7f82b409bca0 2021-02-10T11:38:48.750 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:48.758 7f82baca5700 1 -- 172.21.2.112:0/910326119 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f82b408d660 con 0x7f82b409bca0 2021-02-10T11:38:48.751 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:48.759 7f82b17fa700 1 -- 172.21.2.112:0/910326119 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v87) v1 ==== 84+0+64 (secure 0 0 0) 0x7f829c00bd80 con 0x7f82b409bca0 2021-02-10T11:38:48.751 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:38:48.751 INFO:teuthology.orchestra.run.gibba012.stdout:[ 2021-02-10T11:38:48.751 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:38:48.752 INFO:teuthology.orchestra.run.gibba012.stdout: "poolnum": 1, 2021-02-10T11:38:48.752 INFO:teuthology.orchestra.run.gibba012.stdout: "poolname": "rbd" 2021-02-10T11:38:48.752 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:38:48.752 INFO:teuthology.orchestra.run.gibba012.stdout:] 2021-02-10T11:38:48.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:48.761 7f82baca5700 1 -- 172.21.2.112:0/910326119 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f82980186c0 msgr2=0x7f829801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:48.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:48.761 7f82baca5700 1 --2- 172.21.2.112:0/910326119 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f82980186c0 0x7f829801abc0 crc :-1 s=READY pgs=857 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:48.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:48.761 7f82baca5700 1 -- 172.21.2.112:0/910326119 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f82b409bca0 msgr2=0x7f82b409c0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:48.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:48.761 7f82baca5700 1 --2- 172.21.2.112:0/910326119 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f82b409bca0 0x7f82b409c0a0 secure :-1 s=READY pgs=1823 cs=0 l=1 rev1=1 rx=0x7f829c0096a0 tx=0x7f829c00b990).stop 2021-02-10T11:38:48.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:48.761 7f82baca5700 1 -- 172.21.2.112:0/910326119 shutdown_connections 2021-02-10T11:38:48.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:48.761 7f82baca5700 1 --2- 172.21.2.112:0/910326119 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f82980186c0 0x7f829801abc0 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:48.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:48.761 7f82baca5700 1 --2- 172.21.2.112:0/910326119 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f82b408c1a0 0x7f82b408c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:48.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:48.761 7f82baca5700 1 --2- 172.21.2.112:0/910326119 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f82b409c6c0 0x7f82b409cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:48.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:48.761 7f82baca5700 1 --2- 172.21.2.112:0/910326119 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f82b409bca0 0x7f82b409c0a0 unknown :-1 s=CLOSED pgs=1823 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:48.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:48.761 7f82baca5700 1 -- 172.21.2.112:0/910326119 >> 172.21.2.112:0/910326119 conn(0x7f82b40a0450 msgr2=0x7f82b40f0460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:48.754 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:48.761 7f82baca5700 1 -- 172.21.2.112:0/910326119 shutdown_connections 2021-02-10T11:38:48.754 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:48.761 7f82baca5700 1 -- 172.21.2.112:0/910326119 wait complete. 2021-02-10T11:38:48.761 INFO:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2021-02-10T11:38:48.762 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_metadata 8 2021-02-10T11:38:49.193 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:49.182 7f64288fa700 1 Processor -- start 2021-02-10T11:38:49.193 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:49.182 7f64288fa700 1 -- start start 2021-02-10T11:38:49.193 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:49.183 7f64288fa700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f64240d1870 0x7f64241b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:49.193 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:49.183 7f64288fa700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f64241b0ae0 0x7f6424108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:49.193 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:49.183 7f64288fa700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6424109a60 con 0x7f64240d1c70 2021-02-10T11:38:49.193 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:49.183 7f64288fa700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64240e5ef0 con 0x7f64241b0ae0 2021-02-10T11:38:49.194 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:49.183 7f64288fa700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6424109750 con 0x7f64240d1870 2021-02-10T11:38:49.194 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:49.184 7f6421e1b700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f64241b0ae0 0x7f6424108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:49.194 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:49.184 7f642261c700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f64240d1870 0x7f64241b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:49.194 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:49.184 7f6422e1d700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7f64240d1c70 0x7f6424108f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:46228/0 (socket says 172.21.2.112:46228) 2021-02-10T11:38:49.194 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:49.184 7f6422e1d700 1 -- 172.21.2.112:0/3977599325 learned_addr learned my addr 172.21.2.112:0/3977599325 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:38:49.194 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:49.184 7f6421e1b700 1 -- 172.21.2.112:0/3977599325 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f64240d1870 msgr2=0x7f64241b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:49.194 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:49.184 7f6421e1b700 1 --2- 172.21.2.112:0/3977599325 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f64240d1870 0x7f64241b04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:49.194 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:49.184 7f6421e1b700 1 -- 172.21.2.112:0/3977599325 >> v1:172.21.2.120:6789/0 conn(0x7f64240d1c70 legacy=0x7f6424108f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:49.194 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:49.184 7f6421e1b700 1 -- 172.21.2.112:0/3977599325 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6424088070 con 0x7f64241b0ae0 2021-02-10T11:38:49.194 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:49.184 7f642261c700 1 --2- 172.21.2.112:0/3977599325 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f64240d1870 0x7f64241b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:38:49.194 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:49.184 7f6421e1b700 1 --2- 172.21.2.112:0/3977599325 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f64241b0ae0 0x7f6424108940 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 rx=0x7f6418006650 tx=0x7f6418009a80).ready entity=mon.1 client_cookie=5a2135400bda878c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:49.195 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:49.185 7f642161a700 1 -- 172.21.2.112:0/3977599325 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f641800c0c0 con 0x7f64241b0ae0 2021-02-10T11:38:49.195 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:49.185 7f642161a700 1 -- 172.21.2.112:0/3977599325 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f641800c340 con 0x7f64241b0ae0 2021-02-10T11:38:49.195 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:49.185 7f642161a700 1 -- 172.21.2.112:0/3977599325 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f641800c860 con 0x7f64241b0ae0 2021-02-10T11:38:49.195 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:49.185 7f64288fa700 1 -- 172.21.2.112:0/3977599325 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f64241b0ae0 msgr2=0x7f6424108940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:49.195 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:49.185 7f64288fa700 1 --2- 172.21.2.112:0/3977599325 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f64241b0ae0 0x7f6424108940 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 rx=0x7f6418006650 tx=0x7f6418009a80).stop 2021-02-10T11:38:49.195 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:49.186 7f64288fa700 1 -- 172.21.2.112:0/3977599325 shutdown_connections 2021-02-10T11:38:49.195 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:49.186 7f64288fa700 1 --2- 172.21.2.112:0/3977599325 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f64241b0ae0 0x7f6424108940 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:49.195 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:49.186 7f64288fa700 1 --2- 172.21.2.112:0/3977599325 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f64240d1870 0x7f64241b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:49.195 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:49.186 7f64288fa700 1 -- 172.21.2.112:0/3977599325 >> 172.21.2.112:0/3977599325 conn(0x7f64240ce420 msgr2=0x7f64240d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:49.195 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:49.186 7f64288fa700 1 -- 172.21.2.112:0/3977599325 shutdown_connections 2021-02-10T11:38:49.195 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:49.186 7f64288fa700 1 -- 172.21.2.112:0/3977599325 wait complete. 2021-02-10T11:38:49.197 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:49.200 7f64288fa700 1 Processor -- start 2021-02-10T11:38:49.197 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:49.200 7f64288fa700 1 -- start start 2021-02-10T11:38:49.197 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:49.201 7f64288fa700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f642408aa00 0x7f64241a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:49.197 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:49.201 7f64288fa700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f642409f580 0x7f642409f980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:49.197 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:49.201 7f64288fa700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f64240a0830 0x7f64240a0c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:49.197 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:49.201 7f64288fa700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64240a0070 con 0x7f642409f580 2021-02-10T11:38:49.197 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:49.201 7f64288fa700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f642408ae30 con 0x7f64240a0830 2021-02-10T11:38:49.197 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:49.201 7f64288fa700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f642408d2a0 con 0x7f642408aa00 2021-02-10T11:38:49.197 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:49.201 7f6421e1b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f642409f580 0x7f642409f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:49.198 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:49.201 7f6421e1b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f642409f580 0x7f642409f980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:53216/0 (socket says 172.21.2.112:53216) 2021-02-10T11:38:49.198 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:49.201 7f6421e1b700 1 -- 172.21.2.112:0/515684762 learned_addr learned my addr 172.21.2.112:0/515684762 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:38:49.198 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:49.201 7f6421e1b700 1 -- 172.21.2.112:0/515684762 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f642408aa00 msgr2=0x7f64241a43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:49.198 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:49.201 7f6421e1b700 1 --2- 172.21.2.112:0/515684762 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f642408aa00 0x7f64241a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:49.198 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:49.201 7f6421e1b700 1 -- 172.21.2.112:0/515684762 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f64240a0830 msgr2=0x7f64240a0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:49.198 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:49.201 7f6421e1b700 1 --2- 172.21.2.112:0/515684762 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f64240a0830 0x7f64240a0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:49.198 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:49.201 7f6421e1b700 1 -- 172.21.2.112:0/515684762 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64241a49d0 con 0x7f642409f580 2021-02-10T11:38:49.198 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:49.202 7f6421e1b700 1 --2- 172.21.2.112:0/515684762 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f642409f580 0x7f642409f980 secure :-1 s=READY pgs=1824 cs=0 l=1 rev1=1 rx=0x7f6418009630 tx=0x7f641800b900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:49.198 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:49.202 7f6412ffd700 1 -- 172.21.2.112:0/515684762 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f64180093a0 con 0x7f642409f580 2021-02-10T11:38:49.198 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:49.202 7f6412ffd700 1 -- 172.21.2.112:0/515684762 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f641800bcf0 con 0x7f642409f580 2021-02-10T11:38:49.198 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:49.202 7f6412ffd700 1 -- 172.21.2.112:0/515684762 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f64180054d0 con 0x7f642409f580 2021-02-10T11:38:49.198 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:49.202 7f64288fa700 1 -- 172.21.2.112:0/515684762 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64241a49d0 con 0x7f642409f580 2021-02-10T11:38:49.198 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:49.202 7f64288fa700 1 -- 172.21.2.112:0/515684762 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64241a49d0 con 0x7f642409f580 2021-02-10T11:38:49.199 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:49.202 7f6412ffd700 1 -- 172.21.2.112:0/515684762 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f64180052e0 con 0x7f642409f580 2021-02-10T11:38:49.199 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:49.202 7f6412ffd700 1 --2- 172.21.2.112:0/515684762 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f64080186c0 0x7f640801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:49.199 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:49.203 7f6412ffd700 1 -- 172.21.2.112:0/515684762 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f6418032230 con 0x7f642409f580 2021-02-10T11:38:49.199 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:49.203 7f642261c700 1 --2- 172.21.2.112:0/515684762 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f64080186c0 0x7f640801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:49.199 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:49.203 7f642261c700 1 --2- 172.21.2.112:0/515684762 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f64080186c0 0x7f640801abc0 crc :-1 s=READY pgs=858 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:49.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:49.233 7f64288fa700 1 -- 172.21.2.112:0/515684762 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f642408ae00 con 0x7f642409f580 2021-02-10T11:38:49.228 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:49.236 7f6412ffd700 1 -- 172.21.2.112:0/515684762 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f641802d810 con 0x7f642409f580 2021-02-10T11:38:49.357 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:49.364 7f64288fa700 1 -- 172.21.2.112:0/515684762 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"} v 0) v1 -- 0x7f642408ae00 con 0x7f642409f580 2021-02-10T11:38:49.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:49.785 7f6412ffd700 1 -- 172.21.2.112:0/515684762 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"}]=0 pool 'cephfs_metadata' created v88) v1 ==== 133+0+0 (secure 0 0 0) 0x7f641802d810 con 0x7f642409f580 2021-02-10T11:38:49.824 INFO:teuthology.orchestra.run.gibba012.stderr:pool 'cephfs_metadata' created 2021-02-10T11:38:49.826 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:49.833 7f64288fa700 1 -- 172.21.2.112:0/515684762 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f64080186c0 msgr2=0x7f640801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:49.826 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:49.833 7f64288fa700 1 --2- 172.21.2.112:0/515684762 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f64080186c0 0x7f640801abc0 crc :-1 s=READY pgs=858 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:49.826 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:49.833 7f64288fa700 1 -- 172.21.2.112:0/515684762 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f642409f580 msgr2=0x7f642409f980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:49.826 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:49.833 7f64288fa700 1 --2- 172.21.2.112:0/515684762 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f642409f580 0x7f642409f980 secure :-1 s=READY pgs=1824 cs=0 l=1 rev1=1 rx=0x7f6418009630 tx=0x7f641800b900).stop 2021-02-10T11:38:49.826 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:49.833 7f64288fa700 1 -- 172.21.2.112:0/515684762 shutdown_connections 2021-02-10T11:38:49.826 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:49.833 7f64288fa700 1 --2- 172.21.2.112:0/515684762 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f64080186c0 0x7f640801abc0 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:49.826 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:49.833 7f64288fa700 1 --2- 172.21.2.112:0/515684762 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f64240a0830 0x7f64240a0c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:49.826 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:49.833 7f64288fa700 1 --2- 172.21.2.112:0/515684762 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f642409f580 0x7f642409f980 unknown :-1 s=CLOSED pgs=1824 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:49.826 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:49.833 7f64288fa700 1 --2- 172.21.2.112:0/515684762 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f642408aa00 0x7f64241a43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:49.826 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:49.833 7f64288fa700 1 -- 172.21.2.112:0/515684762 >> 172.21.2.112:0/515684762 conn(0x7f64240bfb90 msgr2=0x7f64240a6a00 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:49.827 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:49.833 7f64288fa700 1 -- 172.21.2.112:0/515684762 shutdown_connections 2021-02-10T11:38:49.827 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:49.833 7f64288fa700 1 -- 172.21.2.112:0/515684762 wait complete. 2021-02-10T11:38:49.834 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_data 8 2021-02-10T11:38:50.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:50.254 7f7d5a7b5700 1 Processor -- start 2021-02-10T11:38:50.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:50.254 7f7d5a7b5700 1 -- start start 2021-02-10T11:38:50.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:50.254 7f7d5a7b5700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7d540d1860 0x7f7d541b04b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:50.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:50.254 7f7d5a7b5700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f7d540cff60 0x7f7d54109120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:50.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:50.254 7f7d5a7b5700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d54109a50 con 0x7f7d541b0ad0 2021-02-10T11:38:50.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:50.254 7f7d5a7b5700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d54109740 con 0x7f7d540cff60 2021-02-10T11:38:50.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:50.254 7f7d5a7b5700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d54088070 con 0x7f7d540d1860 2021-02-10T11:38:50.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:50.255 7f7d58dd2700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f7d540cff60 0x7f7d54109120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:50.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:50.255 7f7d58dd2700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f7d540cff60 0x7f7d54109120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:53222/0 (socket says 172.21.2.112:53222) 2021-02-10T11:38:50.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:50.255 7f7d58dd2700 1 -- 172.21.2.112:0/320426339 learned_addr learned my addr 172.21.2.112:0/320426339 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:38:50.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:50.255 7f7d58dd2700 1 -- 172.21.2.112:0/320426339 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7d540d1860 msgr2=0x7f7d541b04b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:38:50.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:50.255 7f7d58dd2700 1 --2- 172.21.2.112:0/320426339 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7d540d1860 0x7f7d541b04b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:50.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:50.255 7f7d58dd2700 1 -- 172.21.2.112:0/320426339 >> v1:172.21.2.120:6789/0 conn(0x7f7d541b0ad0 legacy=0x7f7d54108930 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:38:50.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:50.255 7f7d58dd2700 1 -- 172.21.2.112:0/320426339 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d54088260 con 0x7f7d540cff60 2021-02-10T11:38:50.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:50.255 7f7d58dd2700 1 --2- 172.21.2.112:0/320426339 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7d540cff60 0x7f7d54109120 secure :-1 s=READY pgs=1825 cs=0 l=1 rev1=1 rx=0x7f7d4c006f00 tx=0x7f7d4c009f10).ready entity=mon.0 client_cookie=48411e883573bc06 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:50.265 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:50.255 7f7d52ffd700 1 -- 172.21.2.112:0/320426339 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7d4c00b3b0 con 0x7f7d540cff60 2021-02-10T11:38:50.265 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:50.256 7f7d52ffd700 1 -- 172.21.2.112:0/320426339 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d4c00b670 con 0x7f7d540cff60 2021-02-10T11:38:50.265 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:50.256 7f7d52ffd700 1 -- 172.21.2.112:0/320426339 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7d4c00bb90 con 0x7f7d540cff60 2021-02-10T11:38:50.265 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:50.256 7f7d5a7b5700 1 -- 172.21.2.112:0/320426339 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7d540cff60 msgr2=0x7f7d54109120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:50.265 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:50.256 7f7d5a7b5700 1 --2- 172.21.2.112:0/320426339 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7d540cff60 0x7f7d54109120 secure :-1 s=READY pgs=1825 cs=0 l=1 rev1=1 rx=0x7f7d4c006f00 tx=0x7f7d4c009f10).stop 2021-02-10T11:38:50.265 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:50.256 7f7d5a7b5700 1 -- 172.21.2.112:0/320426339 shutdown_connections 2021-02-10T11:38:50.265 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:50.256 7f7d5a7b5700 1 --2- 172.21.2.112:0/320426339 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7d540cff60 0x7f7d54109120 unknown :-1 s=CLOSED pgs=1825 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:50.265 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:50.256 7f7d5a7b5700 1 --2- 172.21.2.112:0/320426339 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7d540d1860 0x7f7d541b04b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:50.265 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:50.256 7f7d5a7b5700 1 -- 172.21.2.112:0/320426339 >> 172.21.2.112:0/320426339 conn(0x7f7d540ce410 msgr2=0x7f7d540d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:50.265 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:50.256 7f7d5a7b5700 1 -- 172.21.2.112:0/320426339 shutdown_connections 2021-02-10T11:38:50.265 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:50.256 7f7d5a7b5700 1 -- 172.21.2.112:0/320426339 wait complete. 2021-02-10T11:38:50.267 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:50.271 7f7d5a7b5700 1 Processor -- start 2021-02-10T11:38:50.267 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:50.271 7f7d5a7b5700 1 -- start start 2021-02-10T11:38:50.267 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:50.271 7f7d5a7b5700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7d541a05e0 0x7f7d5418b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:50.267 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:50.271 7f7d5a7b5700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7d540b8780 0x7f7d540bfa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:50.267 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:50.271 7f7d5a7b5700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7d540c0030 0x7f7d54189e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:50.268 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:50.271 7f7d5a7b5700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d540b82a0 con 0x7f7d541a05e0 2021-02-10T11:38:50.268 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:50.271 7f7d5a7b5700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d540c0430 con 0x7f7d540b8780 2021-02-10T11:38:50.268 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:50.271 7f7d5a7b5700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d540c0650 con 0x7f7d540c0030 2021-02-10T11:38:50.268 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:50.271 7f7d53fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7d541a05e0 0x7f7d5418b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:50.268 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:50.271 7f7d53fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7d541a05e0 0x7f7d5418b970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:53228/0 (socket says 172.21.2.112:53228) 2021-02-10T11:38:50.268 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:50.271 7f7d53fff700 1 -- 172.21.2.112:0/1210787194 learned_addr learned my addr 172.21.2.112:0/1210787194 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:38:50.268 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:50.272 7f7d58dd2700 1 --2- 172.21.2.112:0/1210787194 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7d540c0030 0x7f7d54189e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:50.268 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:50.272 7f7d537fe700 1 --2- 172.21.2.112:0/1210787194 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7d540b8780 0x7f7d540bfa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:50.268 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:50.272 7f7d53fff700 1 -- 172.21.2.112:0/1210787194 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7d540c0030 msgr2=0x7f7d54189e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:50.268 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:50.272 7f7d53fff700 1 --2- 172.21.2.112:0/1210787194 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7d540c0030 0x7f7d54189e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:50.268 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:50.272 7f7d53fff700 1 -- 172.21.2.112:0/1210787194 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7d540b8780 msgr2=0x7f7d540bfa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:50.268 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:50.272 7f7d53fff700 1 --2- 172.21.2.112:0/1210787194 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7d540b8780 0x7f7d540bfa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:50.268 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:50.272 7f7d53fff700 1 -- 172.21.2.112:0/1210787194 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d5418a490 con 0x7f7d541a05e0 2021-02-10T11:38:50.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:50.272 7f7d537fe700 1 --2- 172.21.2.112:0/1210787194 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7d540b8780 0x7f7d540bfa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:38:50.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:50.272 7f7d53fff700 1 --2- 172.21.2.112:0/1210787194 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7d541a05e0 0x7f7d5418b970 secure :-1 s=READY pgs=1826 cs=0 l=1 rev1=1 rx=0x7f7d4400b5c0 tx=0x7f7d4400b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:50.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:50.272 7f7d50ff9700 1 -- 172.21.2.112:0/1210787194 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7d4400bd50 con 0x7f7d541a05e0 2021-02-10T11:38:50.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:50.272 7f7d50ff9700 1 -- 172.21.2.112:0/1210787194 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d44006030 con 0x7f7d541a05e0 2021-02-10T11:38:50.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:50.272 7f7d50ff9700 1 -- 172.21.2.112:0/1210787194 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7d440054d0 con 0x7f7d541a05e0 2021-02-10T11:38:50.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:50.272 7f7d5a7b5700 1 -- 172.21.2.112:0/1210787194 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d5418a490 con 0x7f7d541a05e0 2021-02-10T11:38:50.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:50.272 7f7d5a7b5700 1 -- 172.21.2.112:0/1210787194 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d5418aa20 con 0x7f7d541a05e0 2021-02-10T11:38:50.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:50.273 7f7d50ff9700 1 -- 172.21.2.112:0/1210787194 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7d44006210 con 0x7f7d541a05e0 2021-02-10T11:38:50.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:50.273 7f7d50ff9700 1 --2- 172.21.2.112:0/1210787194 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7d340186c0 0x7f7d3401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:50.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:50.273 7f7d50ff9700 1 -- 172.21.2.112:0/1210787194 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5107+0+0 (secure 0 0 0) 0x7f7d44032230 con 0x7f7d541a05e0 2021-02-10T11:38:50.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:50.273 7f7d537fe700 1 --2- 172.21.2.112:0/1210787194 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7d340186c0 0x7f7d3401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:50.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:50.274 7f7d537fe700 1 --2- 172.21.2.112:0/1210787194 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7d340186c0 0x7f7d3401abc0 crc :-1 s=READY pgs=859 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:50.297 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:50.304 7f7d5a7b5700 1 -- 172.21.2.112:0/1210787194 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d54082a20 con 0x7f7d541a05e0 2021-02-10T11:38:50.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:50.307 7f7d50ff9700 1 -- 172.21.2.112:0/1210787194 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7d44000960 con 0x7f7d541a05e0 2021-02-10T11:38:50.427 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:50.434 7f7d5a7b5700 1 -- 172.21.2.112:0/1210787194 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"} v 0) v1 -- 0x7f7d54082a20 con 0x7f7d541a05e0 2021-02-10T11:38:50.540 DEBUG:teuthology.orchestra.run.gibba012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:38:50.544 DEBUG:teuthology.orchestra.run.gibba020:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:38:50.787 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:50.794 7f7d50ff9700 1 -- 172.21.2.112:0/1210787194 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"}]=0 pool 'cephfs_data' created v90) v1 ==== 125+0+0 (secure 0 0 0) 0x7f7d44006210 con 0x7f7d541a05e0 2021-02-10T11:38:50.793 INFO:teuthology.orchestra.run.gibba012.stderr:pool 'cephfs_data' created 2021-02-10T11:38:50.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:50.801 7f7d5a7b5700 1 -- 172.21.2.112:0/1210787194 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7d340186c0 msgr2=0x7f7d3401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:50.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:50.801 7f7d5a7b5700 1 --2- 172.21.2.112:0/1210787194 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7d340186c0 0x7f7d3401abc0 crc :-1 s=READY pgs=859 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:50.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:50.801 7f7d5a7b5700 1 -- 172.21.2.112:0/1210787194 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7d541a05e0 msgr2=0x7f7d5418b970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:50.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:50.801 7f7d5a7b5700 1 --2- 172.21.2.112:0/1210787194 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7d541a05e0 0x7f7d5418b970 secure :-1 s=READY pgs=1826 cs=0 l=1 rev1=1 rx=0x7f7d4400b5c0 tx=0x7f7d4400b960).stop 2021-02-10T11:38:50.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:50.801 7f7d5a7b5700 1 -- 172.21.2.112:0/1210787194 shutdown_connections 2021-02-10T11:38:50.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:50.801 7f7d5a7b5700 1 --2- 172.21.2.112:0/1210787194 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7d340186c0 0x7f7d3401abc0 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:50.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:50.801 7f7d5a7b5700 1 --2- 172.21.2.112:0/1210787194 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7d540b8780 0x7f7d540bfa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:50.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:50.801 7f7d5a7b5700 1 --2- 172.21.2.112:0/1210787194 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7d540c0030 0x7f7d54189e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:50.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:50.801 7f7d5a7b5700 1 --2- 172.21.2.112:0/1210787194 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7d541a05e0 0x7f7d5418b970 unknown :-1 s=CLOSED pgs=1826 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:50.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:50.801 7f7d5a7b5700 1 -- 172.21.2.112:0/1210787194 >> 172.21.2.112:0/1210787194 conn(0x7f7d540a6a80 msgr2=0x7f7d540ae110 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:50.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:50.802 7f7d5a7b5700 1 -- 172.21.2.112:0/1210787194 shutdown_connections 2021-02-10T11:38:50.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:50.802 7f7d5a7b5700 1 -- 172.21.2.112:0/1210787194 wait complete. 2021-02-10T11:38:50.803 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2021-02-10T11:38:51.234 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.224 7fb0fadcf700 1 Processor -- start 2021-02-10T11:38:51.234 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.224 7fb0fadcf700 1 -- start start 2021-02-10T11:38:51.234 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.224 7fb0fadcf700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb0f40d1880 0x7fb0f41b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:51.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.224 7fb0fadcf700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0f4109a70 con 0x7fb0f40cff80 2021-02-10T11:38:51.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.224 7fb0fadcf700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0f4088070 con 0x7fb0f41b0af0 2021-02-10T11:38:51.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.224 7fb0fadcf700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0f40e5f00 con 0x7fb0f40d1880 2021-02-10T11:38:51.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.224 7fb0f93ec700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fb0f40cff80 0x7fb0f4109140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:52698/0 (socket says 172.21.2.112:52698) 2021-02-10T11:38:51.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.224 7fb0f93ec700 1 -- 172.21.2.112:0/2341628384 learned_addr learned my addr 172.21.2.112:0/2341628384 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:38:51.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.225 7fb0f8beb700 1 --2- 172.21.2.112:0/2341628384 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb0f40d1880 0x7fb0f41b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:51.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.225 7fb0f37fe700 1 -- 172.21.2.112:0/2341628384 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2809592950 0 0) 0x7fb0ec000f90 con 0x7fb0f40cff80 2021-02-10T11:38:51.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.225 7fb0f37fe700 1 -- 172.21.2.112:0/2341628384 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0e0002fe0 con 0x7fb0f40cff80 2021-02-10T11:38:51.236 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.225 7fb0f37fe700 1 -- 172.21.2.112:0/2341628384 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 702488472 0 0) 0x7fb0e4000ff0 con 0x7fb0f41b0af0 2021-02-10T11:38:51.236 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.225 7fb0f37fe700 1 -- 172.21.2.112:0/2341628384 --> v1:172.21.2.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0e0004370 con 0x7fb0f41b0af0 2021-02-10T11:38:51.236 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.225 7fb0f8beb700 1 -- 172.21.2.112:0/2341628384 >> v1:172.21.2.120:6789/0 conn(0x7fb0f41b0af0 legacy=0x7fb0f4108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:51.236 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.225 7fb0f8beb700 1 -- 172.21.2.112:0/2341628384 >> v1:172.21.2.112:6789/0 conn(0x7fb0f40cff80 legacy=0x7fb0f4109140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:51.236 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.225 7fb0f8beb700 1 -- 172.21.2.112:0/2341628384 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0f40d1c80 con 0x7fb0f40d1880 2021-02-10T11:38:51.236 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.225 7fb0f8beb700 1 --2- 172.21.2.112:0/2341628384 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb0f40d1880 0x7fb0f41b04d0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7fb0dc006650 tx=0x7fb0dc009a80).ready entity=mon.2 client_cookie=ff214001c3615e6a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:51.236 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.226 7fb0f37fe700 1 -- 172.21.2.112:0/2341628384 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb0dc0009b0 con 0x7fb0f40d1880 2021-02-10T11:38:51.237 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.226 7fb0f37fe700 1 -- 172.21.2.112:0/2341628384 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0dc00c0c0 con 0x7fb0f40d1880 2021-02-10T11:38:51.237 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.226 7fb0f37fe700 1 -- 172.21.2.112:0/2341628384 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb0dc00c510 con 0x7fb0f40d1880 2021-02-10T11:38:51.237 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.226 7fb0fadcf700 1 -- 172.21.2.112:0/2341628384 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb0f40d1880 msgr2=0x7fb0f41b04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:51.237 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.226 7fb0fadcf700 1 --2- 172.21.2.112:0/2341628384 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb0f40d1880 0x7fb0f41b04d0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7fb0dc006650 tx=0x7fb0dc009a80).stop 2021-02-10T11:38:51.237 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.226 7fb0fadcf700 1 -- 172.21.2.112:0/2341628384 shutdown_connections 2021-02-10T11:38:51.237 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.226 7fb0fadcf700 1 --2- 172.21.2.112:0/2341628384 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb0f40d1880 0x7fb0f41b04d0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:51.237 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.226 7fb0fadcf700 1 -- 172.21.2.112:0/2341628384 >> 172.21.2.112:0/2341628384 conn(0x7fb0f40ce430 msgr2=0x7fb0f40d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:51.238 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.226 7fb0fadcf700 1 -- 172.21.2.112:0/2341628384 shutdown_connections 2021-02-10T11:38:51.238 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.226 7fb0fadcf700 1 -- 172.21.2.112:0/2341628384 wait complete. 2021-02-10T11:38:51.238 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.241 7fb0fadcf700 1 Processor -- start 2021-02-10T11:38:51.238 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.241 7fb0fadcf700 1 -- start start 2021-02-10T11:38:51.239 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.241 7fb0fadcf700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb0f409a690 0x7fb0f409aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:51.240 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.241 7fb0fadcf700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb0f41b4160 0x7fb0f41b4560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:51.240 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.241 7fb0fadcf700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb0f41b7dd0 0x7fb0f41b81d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:51.240 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.241 7fb0fadcf700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0f41b4b80 con 0x7fb0f41b4160 2021-02-10T11:38:51.240 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.241 7fb0fadcf700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0f41b87f0 con 0x7fb0f41b7dd0 2021-02-10T11:38:51.240 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.241 7fb0fadcf700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0f41b8a10 con 0x7fb0f409a690 2021-02-10T11:38:51.240 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.242 7fb0f3fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb0f41b4160 0x7fb0f41b4560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:51.240 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.242 7fb0f8beb700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb0f409a690 0x7fb0f409aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:51.240 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.242 7fb0f3fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb0f41b4160 0x7fb0f41b4560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:53244/0 (socket says 172.21.2.112:53244) 2021-02-10T11:38:51.241 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.242 7fb0f3fff700 1 -- 172.21.2.112:0/2666522812 learned_addr learned my addr 172.21.2.112:0/2666522812 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:38:51.241 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.242 7fb0f93ec700 1 --2- 172.21.2.112:0/2666522812 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb0f41b7dd0 0x7fb0f41b81d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:51.241 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.242 7fb0f3fff700 1 -- 172.21.2.112:0/2666522812 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb0f409a690 msgr2=0x7fb0f409aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:51.241 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.242 7fb0f3fff700 1 --2- 172.21.2.112:0/2666522812 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb0f409a690 0x7fb0f409aa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:51.241 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.242 7fb0f3fff700 1 -- 172.21.2.112:0/2666522812 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb0f41b7dd0 msgr2=0x7fb0f41b81d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:51.241 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.242 7fb0f3fff700 1 --2- 172.21.2.112:0/2666522812 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb0f41b7dd0 0x7fb0f41b81d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:51.241 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.242 7fb0f3fff700 1 -- 172.21.2.112:0/2666522812 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0f41b8c30 con 0x7fb0f41b4160 2021-02-10T11:38:51.241 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.242 7fb0f8beb700 1 --2- 172.21.2.112:0/2666522812 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb0f409a690 0x7fb0f409aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:38:51.242 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.242 7fb0f3fff700 1 --2- 172.21.2.112:0/2666522812 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb0f41b4160 0x7fb0f41b4560 secure :-1 s=READY pgs=1828 cs=0 l=1 rev1=1 rx=0x7fb0e4006ed0 tx=0x7fb0e400b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:51.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.243 7fb0f17fa700 1 -- 172.21.2.112:0/2666522812 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb0e400bd50 con 0x7fb0f41b4160 2021-02-10T11:38:51.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.243 7fb0f17fa700 1 -- 172.21.2.112:0/2666522812 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0e4006240 con 0x7fb0f41b4160 2021-02-10T11:38:51.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.243 7fb0fadcf700 1 -- 172.21.2.112:0/2666522812 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0f41b8c30 con 0x7fb0f41b4160 2021-02-10T11:38:51.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.243 7fb0f17fa700 1 -- 172.21.2.112:0/2666522812 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb0e40056a0 con 0x7fb0f41b4160 2021-02-10T11:38:51.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.243 7fb0fadcf700 1 -- 172.21.2.112:0/2666522812 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0f41b8c30 con 0x7fb0f41b4160 2021-02-10T11:38:51.245 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.243 7fb0f17fa700 1 -- 172.21.2.112:0/2666522812 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb0e40054b0 con 0x7fb0f41b4160 2021-02-10T11:38:51.245 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.243 7fb0f17fa700 1 --2- 172.21.2.112:0/2666522812 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb0d80186c0 0x7fb0d801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:51.245 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.243 7fb0f17fa700 1 -- 172.21.2.112:0/2666522812 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 5457+0+0 (secure 0 0 0) 0x7fb0e40323b0 con 0x7fb0f41b4160 2021-02-10T11:38:51.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.245 7fb0f8beb700 1 --2- 172.21.2.112:0/2666522812 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb0d80186c0 0x7fb0d801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:51.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.245 7fb0f8beb700 1 --2- 172.21.2.112:0/2666522812 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb0d80186c0 0x7fb0d801abc0 crc :-1 s=READY pgs=862 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:51.267 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.274 7fb0fadcf700 1 -- 172.21.2.112:0/2666522812 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0f409a160 con 0x7fb0f41b4160 2021-02-10T11:38:51.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.278 7fb0f17fa700 1 -- 172.21.2.112:0/2666522812 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb0e4030d50 con 0x7fb0f41b4160 2021-02-10T11:38:51.401 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.408 7fb0fadcf700 1 -- 172.21.2.112:0/2666522812 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"} v 0) v1 -- 0x7fb0f409a160 con 0x7fb0f41b4160 2021-02-10T11:38:51.406 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.413 7fb0f17fa700 1 -- 172.21.2.112:0/2666522812 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"}]=0 new fs with metadata pool 18 and data pool 19 v70) v1 ==== 189+0+0 (secure 0 0 0) 0x7fb0e400a000 con 0x7fb0f41b4160 2021-02-10T11:38:51.408 INFO:teuthology.orchestra.run.gibba012.stderr:new fs with metadata pool 18 and data pool 19 2021-02-10T11:38:51.410 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.417 7fb0fadcf700 1 -- 172.21.2.112:0/2666522812 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb0d80186c0 msgr2=0x7fb0d801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:51.410 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.417 7fb0fadcf700 1 --2- 172.21.2.112:0/2666522812 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb0d80186c0 0x7fb0d801abc0 crc :-1 s=READY pgs=862 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:51.410 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.417 7fb0fadcf700 1 -- 172.21.2.112:0/2666522812 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb0f41b4160 msgr2=0x7fb0f41b4560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:51.410 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.417 7fb0fadcf700 1 --2- 172.21.2.112:0/2666522812 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb0f41b4160 0x7fb0f41b4560 secure :-1 s=READY pgs=1828 cs=0 l=1 rev1=1 rx=0x7fb0e4006ed0 tx=0x7fb0e400b960).stop 2021-02-10T11:38:51.411 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.417 7fb0fadcf700 1 -- 172.21.2.112:0/2666522812 shutdown_connections 2021-02-10T11:38:51.411 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.417 7fb0fadcf700 1 --2- 172.21.2.112:0/2666522812 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb0d80186c0 0x7fb0d801abc0 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:51.411 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.417 7fb0fadcf700 1 --2- 172.21.2.112:0/2666522812 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb0f41b7dd0 0x7fb0f41b81d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:51.411 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.417 7fb0fadcf700 1 --2- 172.21.2.112:0/2666522812 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb0f41b4160 0x7fb0f41b4560 unknown :-1 s=CLOSED pgs=1828 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:51.411 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.417 7fb0fadcf700 1 --2- 172.21.2.112:0/2666522812 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb0f409a690 0x7fb0f409aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:51.411 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.417 7fb0fadcf700 1 -- 172.21.2.112:0/2666522812 >> 172.21.2.112:0/2666522812 conn(0x7fb0f418e870 msgr2=0x7fb0f418ecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:51.411 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.417 7fb0fadcf700 1 -- 172.21.2.112:0/2666522812 shutdown_connections 2021-02-10T11:38:51.411 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.418 7fb0fadcf700 1 -- 172.21.2.112:0/2666522812 wait complete. 2021-02-10T11:38:51.419 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:38:51.848 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.840 7fa882874700 1 Processor -- start 2021-02-10T11:38:51.848 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.840 7fa882874700 1 -- start start 2021-02-10T11:38:51.848 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.840 7fa882874700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa87c1b0470 0x7fa87c1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:51.848 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.840 7fa882874700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fa87c0f20e0 0x7fa87c0f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:51.848 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.840 7fa882874700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa87c0f0b50 con 0x7fa87c0ef610 2021-02-10T11:38:51.848 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.840 7fa882874700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa87c0f0820 con 0x7fa87c0f20e0 2021-02-10T11:38:51.849 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.840 7fa882874700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa87c0f24e0 con 0x7fa87c1b0470 2021-02-10T11:38:51.849 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.841 7fa880e91700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fa87c0f20e0 0x7fa87c0f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:51.849 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.841 7fa87b7fe700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fa87c0ef610 0x7fa87c0efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:52714/0 (socket says 172.21.2.112:52714) 2021-02-10T11:38:51.849 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.841 7fa87bfff700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa87c1b0470 0x7fa87c1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:51.849 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.841 7fa87b7fe700 1 -- 172.21.2.112:0/1248181312 learned_addr learned my addr 172.21.2.112:0/1248181312 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:38:51.849 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.841 7fa880e91700 1 -- 172.21.2.112:0/1248181312 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa87c1b0470 msgr2=0x7fa87c1b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:51.849 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.841 7fa880e91700 1 --2- 172.21.2.112:0/1248181312 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa87c1b0470 0x7fa87c1b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:51.849 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.841 7fa880e91700 1 -- 172.21.2.112:0/1248181312 >> v1:172.21.2.112:6789/0 conn(0x7fa87c0ef610 legacy=0x7fa87c0efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:51.849 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.841 7fa880e91700 1 -- 172.21.2.112:0/1248181312 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa87c0e52e0 con 0x7fa87c0f20e0 2021-02-10T11:38:51.849 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.841 7fa880e91700 1 --2- 172.21.2.112:0/1248181312 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa87c0f20e0 0x7fa87c0f0200 secure :-1 s=READY pgs=1830 cs=0 l=1 rev1=1 rx=0x7fa874006ab0 tx=0x7fa874009b40).ready entity=mon.0 client_cookie=da84651eea078898 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:51.849 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.841 7fa87affd700 1 -- 172.21.2.112:0/1248181312 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa8740022e0 con 0x7fa87c0f20e0 2021-02-10T11:38:51.849 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.841 7fa87affd700 1 -- 172.21.2.112:0/1248181312 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa874002560 con 0x7fa87c0f20e0 2021-02-10T11:38:51.849 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.841 7fa87affd700 1 -- 172.21.2.112:0/1248181312 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa8740029f0 con 0x7fa87c0f20e0 2021-02-10T11:38:51.849 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.841 7fa882874700 1 -- 172.21.2.112:0/1248181312 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa87c0f20e0 msgr2=0x7fa87c0f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:51.850 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.841 7fa882874700 1 --2- 172.21.2.112:0/1248181312 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa87c0f20e0 0x7fa87c0f0200 secure :-1 s=READY pgs=1830 cs=0 l=1 rev1=1 rx=0x7fa874006ab0 tx=0x7fa874009b40).stop 2021-02-10T11:38:51.850 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.842 7fa882874700 1 -- 172.21.2.112:0/1248181312 shutdown_connections 2021-02-10T11:38:51.850 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.842 7fa882874700 1 --2- 172.21.2.112:0/1248181312 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa87c0f20e0 0x7fa87c0f0200 unknown :-1 s=CLOSED pgs=1830 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:51.850 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.842 7fa882874700 1 --2- 172.21.2.112:0/1248181312 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa87c1b0470 0x7fa87c1b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:51.850 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.842 7fa882874700 1 -- 172.21.2.112:0/1248181312 >> 172.21.2.112:0/1248181312 conn(0x7fa87c0ce3e0 msgr2=0x7fa87c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:51.850 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.842 7fa882874700 1 -- 172.21.2.112:0/1248181312 shutdown_connections 2021-02-10T11:38:51.850 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.842 7fa882874700 1 -- 172.21.2.112:0/1248181312 wait complete. 2021-02-10T11:38:51.852 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.856 7fa882874700 1 Processor -- start 2021-02-10T11:38:51.852 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.856 7fa882874700 1 -- start start 2021-02-10T11:38:51.852 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.857 7fa882874700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa87c0938f0 0x7fa87c1a4310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:51.852 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.857 7fa882874700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa87c093cf0 0x7fa87c092690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:51.852 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.857 7fa882874700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa87c092cb0 0x7fa87c0930b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:51.852 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.857 7fa882874700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa87c0b8300 con 0x7fa87c092cb0 2021-02-10T11:38:51.852 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.857 7fa882874700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa87c0b7f40 con 0x7fa87c0938f0 2021-02-10T11:38:51.852 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.857 7fa882874700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa87c1a4930 con 0x7fa87c093cf0 2021-02-10T11:38:51.852 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.857 7fa880e91700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa87c092cb0 0x7fa87c0930b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:51.852 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.857 7fa880e91700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa87c092cb0 0x7fa87c0930b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:53258/0 (socket says 172.21.2.112:53258) 2021-02-10T11:38:51.852 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.857 7fa87b7fe700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa87c093cf0 0x7fa87c092690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:51.853 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.857 7fa880e91700 1 -- 172.21.2.112:0/3984711907 learned_addr learned my addr 172.21.2.112:0/3984711907 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:38:51.853 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.857 7fa87bfff700 1 --2- 172.21.2.112:0/3984711907 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa87c0938f0 0x7fa87c1a4310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:51.853 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.858 7fa880e91700 1 -- 172.21.2.112:0/3984711907 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa87c093cf0 msgr2=0x7fa87c092690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:51.853 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.858 7fa880e91700 1 --2- 172.21.2.112:0/3984711907 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa87c093cf0 0x7fa87c092690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:51.853 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.858 7fa880e91700 1 -- 172.21.2.112:0/3984711907 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa87c0938f0 msgr2=0x7fa87c1a4310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:51.853 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.858 7fa880e91700 1 --2- 172.21.2.112:0/3984711907 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa87c0938f0 0x7fa87c1a4310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:51.853 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.858 7fa880e91700 1 -- 172.21.2.112:0/3984711907 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa87c0ad530 con 0x7fa87c092cb0 2021-02-10T11:38:51.853 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.858 7fa880e91700 1 --2- 172.21.2.112:0/3984711907 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa87c092cb0 0x7fa87c0930b0 secure :-1 s=READY pgs=1831 cs=0 l=1 rev1=1 rx=0x7fa87400b5e0 tx=0x7fa87400b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:51.853 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.858 7fa878ff9700 1 -- 172.21.2.112:0/3984711907 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa874009070 con 0x7fa87c092cb0 2021-02-10T11:38:51.853 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.858 7fa878ff9700 1 -- 172.21.2.112:0/3984711907 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa874002280 con 0x7fa87c092cb0 2021-02-10T11:38:51.853 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.858 7fa878ff9700 1 -- 172.21.2.112:0/3984711907 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa874002760 con 0x7fa87c092cb0 2021-02-10T11:38:51.853 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.858 7fa882874700 1 -- 172.21.2.112:0/3984711907 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa87c0ad530 con 0x7fa87c092cb0 2021-02-10T11:38:51.853 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.858 7fa882874700 1 -- 172.21.2.112:0/3984711907 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa87c0adac0 con 0x7fa87c092cb0 2021-02-10T11:38:51.853 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.859 7fa878ff9700 1 -- 172.21.2.112:0/3984711907 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa874002460 con 0x7fa87c092cb0 2021-02-10T11:38:51.854 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.859 7fa878ff9700 1 --2- 172.21.2.112:0/3984711907 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa8600186c0 0x7fa86001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:51.854 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.859 7fa878ff9700 1 -- 172.21.2.112:0/3984711907 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fa8740325d0 con 0x7fa87c092cb0 2021-02-10T11:38:51.854 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.859 7fa87bfff700 1 --2- 172.21.2.112:0/3984711907 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa8600186c0 0x7fa86001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:51.854 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.860 7fa87bfff700 1 --2- 172.21.2.112:0/3984711907 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa8600186c0 0x7fa86001abc0 crc :-1 s=READY pgs=863 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:51.882 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.890 7fa882874700 1 -- 172.21.2.112:0/3984711907 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa87c105a80 con 0x7fa87c092cb0 2021-02-10T11:38:51.884 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:51.892 7fa878ff9700 1 -- 172.21.2.112:0/3984711907 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa874002070 con 0x7fa87c092cb0 2021-02-10T11:38:52.009 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:52.017 7fa882874700 1 -- 172.21.2.112:0/3984711907 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa87c0f0b50 con 0x7fa87c092cb0 2021-02-10T11:38:52.009 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:52.017 7fa878ff9700 1 -- 172.21.2.112:0/3984711907 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) v1 ==== 74+0+14009 (secure 0 0 0) 0x7fa874000ef0 con 0x7fa87c092cb0 2021-02-10T11:38:52.010 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:38:52.010 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":91,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:38:51.410370","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:38:49.384013","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":19,"pool_name":"cephfs_data","create_time":"2021-02-10 11:38:50.454005","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:38:52.011 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:52.019 7fa882874700 1 -- 172.21.2.112:0/3984711907 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa8600186c0 msgr2=0x7fa86001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:52.011 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:52.019 7fa882874700 1 --2- 172.21.2.112:0/3984711907 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa8600186c0 0x7fa86001abc0 crc :-1 s=READY pgs=863 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:52.011 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:52.019 7fa882874700 1 -- 172.21.2.112:0/3984711907 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa87c092cb0 msgr2=0x7fa87c0930b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:52.011 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:52.019 7fa882874700 1 --2- 172.21.2.112:0/3984711907 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa87c092cb0 0x7fa87c0930b0 secure :-1 s=READY pgs=1831 cs=0 l=1 rev1=1 rx=0x7fa87400b5e0 tx=0x7fa87400b980).stop 2021-02-10T11:38:52.012 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:52.019 7fa882874700 1 -- 172.21.2.112:0/3984711907 shutdown_connections 2021-02-10T11:38:52.012 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:52.019 7fa882874700 1 --2- 172.21.2.112:0/3984711907 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa8600186c0 0x7fa86001abc0 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:52.012 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:52.019 7fa882874700 1 --2- 172.21.2.112:0/3984711907 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa87c092cb0 0x7fa87c0930b0 unknown :-1 s=CLOSED pgs=1831 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:52.012 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:52.019 7fa882874700 1 --2- 172.21.2.112:0/3984711907 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa87c093cf0 0x7fa87c092690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:52.012 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:52.019 7fa882874700 1 --2- 172.21.2.112:0/3984711907 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa87c0938f0 0x7fa87c1a4310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:52.012 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:52.019 7fa882874700 1 -- 172.21.2.112:0/3984711907 >> 172.21.2.112:0/3984711907 conn(0x7fa87c0a1dc0 msgr2=0x7fa87c08e860 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:52.012 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:52.020 7fa882874700 1 -- 172.21.2.112:0/3984711907 shutdown_connections 2021-02-10T11:38:52.012 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:52.020 7fa882874700 1 -- 172.21.2.112:0/3984711907 wait complete. 2021-02-10T11:38:52.021 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:38:52.448 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:52.439 7ff650410700 1 Processor -- start 2021-02-10T11:38:52.448 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:52.439 7ff650410700 1 -- start start 2021-02-10T11:38:52.448 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:52.439 7ff650410700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7ff6480d1870 0x7ff6481b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:52.448 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:52.439 7ff650410700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7ff6481b0a10 0x7ff6480cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:52.448 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:52.439 7ff650410700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6480f0b10 con 0x7ff6480d1c70 2021-02-10T11:38:52.448 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:52.439 7ff650410700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6480e5ed0 con 0x7ff6480d1870 2021-02-10T11:38:52.448 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:52.439 7ff650410700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6480f0650 con 0x7ff6481b0a10 2021-02-10T11:38:52.448 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:52.440 7ff64e22c700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7ff6480d1870 0x7ff6481b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:52.448 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:52.440 7ff64ea2d700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7ff6480d1c70 0x7ff6480cf950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:46286/0 (socket says 172.21.2.112:46286) 2021-02-10T11:38:52.448 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:52.440 7ff64da2b700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7ff6481b0a10 0x7ff6480cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:52.449 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:52.440 7ff64ea2d700 1 -- 172.21.2.112:0/1095627241 learned_addr learned my addr 172.21.2.112:0/1095627241 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:38:52.449 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:52.440 7ff64e22c700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7ff6480d1870 0x7ff6481b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:53262/0 (socket says 172.21.2.112:53262) 2021-02-10T11:38:52.449 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:52.440 7ff64e22c700 1 -- 172.21.2.112:0/1095627241 >> v2:172.21.2.120:3300/0 conn(0x7ff6481b0a10 msgr2=0x7ff6480cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:52.449 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:52.440 7ff64e22c700 1 --2- 172.21.2.112:0/1095627241 >> v2:172.21.2.120:3300/0 conn(0x7ff6481b0a10 0x7ff6480cf330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:52.449 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:52.440 7ff64e22c700 1 -- 172.21.2.112:0/1095627241 >> v1:172.21.2.120:6789/0 conn(0x7ff6480d1c70 legacy=0x7ff6480cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:52.449 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:52.440 7ff64d22a700 1 -- 172.21.2.112:0/1095627241 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1511456754 0 0) 0x7ff640001510 con 0x7ff6480d1c70 2021-02-10T11:38:52.449 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:52.440 7ff64e22c700 1 -- 172.21.2.112:0/1095627241 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6480f0840 con 0x7ff6480d1870 2021-02-10T11:38:52.449 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:52.440 7ff64da2b700 1 --2- 172.21.2.112:0/1095627241 >> v2:172.21.2.120:3300/0 conn(0x7ff6481b0a10 0x7ff6480cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:38:52.449 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:52.440 7ff64e22c700 1 --2- 172.21.2.112:0/1095627241 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff6480d1870 0x7ff6481b03f0 secure :-1 s=READY pgs=1832 cs=0 l=1 rev1=1 rx=0x7ff63c006650 tx=0x7ff63c009a80).ready entity=mon.0 client_cookie=a4b27ca9f165bd45 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:52.449 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:52.440 7ff64d22a700 1 -- 172.21.2.112:0/1095627241 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff63c0009b0 con 0x7ff6480d1870 2021-02-10T11:38:52.449 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:52.441 7ff64d22a700 1 -- 172.21.2.112:0/1095627241 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff63c00c0c0 con 0x7ff6480d1870 2021-02-10T11:38:52.449 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:52.441 7ff64d22a700 1 -- 172.21.2.112:0/1095627241 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff63c00c520 con 0x7ff6480d1870 2021-02-10T11:38:52.449 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:52.441 7ff650410700 1 -- 172.21.2.112:0/1095627241 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff6480d1870 msgr2=0x7ff6481b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:52.450 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:52.441 7ff650410700 1 --2- 172.21.2.112:0/1095627241 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff6480d1870 0x7ff6481b03f0 secure :-1 s=READY pgs=1832 cs=0 l=1 rev1=1 rx=0x7ff63c006650 tx=0x7ff63c009a80).stop 2021-02-10T11:38:52.450 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:52.441 7ff650410700 1 -- 172.21.2.112:0/1095627241 shutdown_connections 2021-02-10T11:38:52.450 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:52.441 7ff650410700 1 --2- 172.21.2.112:0/1095627241 >> v2:172.21.2.120:3300/0 conn(0x7ff6481b0a10 0x7ff6480cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:52.450 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:52.441 7ff650410700 1 --2- 172.21.2.112:0/1095627241 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff6480d1870 0x7ff6481b03f0 unknown :-1 s=CLOSED pgs=1832 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:52.450 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:52.441 7ff650410700 1 -- 172.21.2.112:0/1095627241 >> 172.21.2.112:0/1095627241 conn(0x7ff6480ce450 msgr2=0x7ff6480d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:52.450 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:52.441 7ff650410700 1 -- 172.21.2.112:0/1095627241 shutdown_connections 2021-02-10T11:38:52.450 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:52.441 7ff650410700 1 -- 172.21.2.112:0/1095627241 wait complete. 2021-02-10T11:38:52.452 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:52.456 7ff650410700 1 Processor -- start 2021-02-10T11:38:52.452 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:52.456 7ff650410700 1 -- start start 2021-02-10T11:38:52.452 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:52.457 7ff650410700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff648195650 0x7ff6480b5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:52.452 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:52.457 7ff650410700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff6480b5b80 0x7ff6481a2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:52.452 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:52.457 7ff650410700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff6480acfa0 0x7ff6481a3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:52.452 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:52.457 7ff650410700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6480b6010 con 0x7ff6480acfa0 2021-02-10T11:38:52.452 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:52.457 7ff650410700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6481a3a60 con 0x7ff648195650 2021-02-10T11:38:52.452 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:52.457 7ff650410700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6481a3c40 con 0x7ff6480b5b80 2021-02-10T11:38:52.452 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:52.457 7ff64ea2d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff6480acfa0 0x7ff6481a3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:52.452 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:52.457 7ff64ea2d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff6480acfa0 0x7ff6481a3440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:53270/0 (socket says 172.21.2.112:53270) 2021-02-10T11:38:52.453 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:52.457 7ff64ea2d700 1 -- 172.21.2.112:0/230597847 learned_addr learned my addr 172.21.2.112:0/230597847 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:38:52.453 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:52.457 7ff64da2b700 1 --2- 172.21.2.112:0/230597847 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff6480b5b80 0x7ff6481a2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:52.453 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:52.457 7ff64e22c700 1 --2- 172.21.2.112:0/230597847 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff648195650 0x7ff6480b5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:52.453 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:52.457 7ff64ea2d700 1 -- 172.21.2.112:0/230597847 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff6480b5b80 msgr2=0x7ff6481a2e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:52.453 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:52.457 7ff64ea2d700 1 --2- 172.21.2.112:0/230597847 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff6480b5b80 0x7ff6481a2e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:52.453 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:52.457 7ff64ea2d700 1 -- 172.21.2.112:0/230597847 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff648195650 msgr2=0x7ff6480b5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:52.453 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:52.457 7ff64ea2d700 1 --2- 172.21.2.112:0/230597847 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff648195650 0x7ff6480b5560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:52.453 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:52.457 7ff64ea2d700 1 -- 172.21.2.112:0/230597847 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6480bcae0 con 0x7ff6480acfa0 2021-02-10T11:38:52.453 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:52.457 7ff64da2b700 1 --2- 172.21.2.112:0/230597847 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff6480b5b80 0x7ff6481a2e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:38:52.453 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:52.457 7ff64ea2d700 1 --2- 172.21.2.112:0/230597847 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff6480acfa0 0x7ff6481a3440 secure :-1 s=READY pgs=1833 cs=0 l=1 rev1=1 rx=0x7ff64000a900 tx=0x7ff64000aca0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:52.453 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:52.458 7ff63affd700 1 -- 172.21.2.112:0/230597847 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff640000ee0 con 0x7ff6480acfa0 2021-02-10T11:38:52.453 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:52.458 7ff63affd700 1 -- 172.21.2.112:0/230597847 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff640007c90 con 0x7ff6480acfa0 2021-02-10T11:38:52.453 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:52.458 7ff650410700 1 -- 172.21.2.112:0/230597847 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6480bcae0 con 0x7ff6480acfa0 2021-02-10T11:38:52.454 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:52.458 7ff650410700 1 -- 172.21.2.112:0/230597847 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6480bcae0 con 0x7ff6480acfa0 2021-02-10T11:38:52.454 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:52.458 7ff63affd700 1 -- 172.21.2.112:0/230597847 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff640007210 con 0x7ff6480acfa0 2021-02-10T11:38:52.454 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:52.458 7ff63affd700 1 -- 172.21.2.112:0/230597847 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff640007020 con 0x7ff6480acfa0 2021-02-10T11:38:52.454 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:52.458 7ff63affd700 1 --2- 172.21.2.112:0/230597847 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff630018730 0x7ff63001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:52.454 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:52.458 7ff63affd700 1 -- 172.21.2.112:0/230597847 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5573+0+0 (secure 0 0 0) 0x7ff6400329d0 con 0x7ff6480acfa0 2021-02-10T11:38:52.454 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:52.459 7ff64e22c700 1 --2- 172.21.2.112:0/230597847 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff630018730 0x7ff63001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:52.454 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:52.459 7ff64e22c700 1 --2- 172.21.2.112:0/230597847 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff630018730 0x7ff63001ac10 crc :-1 s=READY pgs=864 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:52.481 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:52.489 7ff650410700 1 -- 172.21.2.112:0/230597847 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6480b6010 con 0x7ff6480acfa0 2021-02-10T11:38:52.484 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:52.492 7ff63affd700 1 -- 172.21.2.112:0/230597847 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff640007c70 con 0x7ff6480acfa0 2021-02-10T11:38:52.609 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:52.617 7ff650410700 1 -- 172.21.2.112:0/230597847 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff6480b6010 con 0x7ff6480acfa0 2021-02-10T11:38:52.609 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:52.617 7ff63affd700 1 -- 172.21.2.112:0/230597847 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) v1 ==== 74+0+13996 (secure 0 0 0) 0x7ff6400046a0 con 0x7ff6480acfa0 2021-02-10T11:38:52.610 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:38:52.610 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":92,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:38:52.412732","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:38:49.384013","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":19,"pool_name":"cephfs_data","create_time":"2021-02-10 11:38:50.454005","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:38:52.611 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:52.619 7ff650410700 1 -- 172.21.2.112:0/230597847 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff630018730 msgr2=0x7ff63001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:52.611 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:52.619 7ff650410700 1 --2- 172.21.2.112:0/230597847 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff630018730 0x7ff63001ac10 crc :-1 s=READY pgs=864 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:52.611 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:52.619 7ff650410700 1 -- 172.21.2.112:0/230597847 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff6480acfa0 msgr2=0x7ff6481a3440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:52.611 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:52.619 7ff650410700 1 --2- 172.21.2.112:0/230597847 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff6480acfa0 0x7ff6481a3440 secure :-1 s=READY pgs=1833 cs=0 l=1 rev1=1 rx=0x7ff64000a900 tx=0x7ff64000aca0).stop 2021-02-10T11:38:52.611 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:52.619 7ff650410700 1 -- 172.21.2.112:0/230597847 shutdown_connections 2021-02-10T11:38:52.611 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:52.619 7ff650410700 1 --2- 172.21.2.112:0/230597847 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff630018730 0x7ff63001ac10 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:52.611 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:52.619 7ff650410700 1 --2- 172.21.2.112:0/230597847 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff6480acfa0 0x7ff6481a3440 unknown :-1 s=CLOSED pgs=1833 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:52.612 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:52.619 7ff650410700 1 --2- 172.21.2.112:0/230597847 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff6480b5b80 0x7ff6481a2e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:52.612 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:52.619 7ff650410700 1 --2- 172.21.2.112:0/230597847 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff648195650 0x7ff6480b5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:52.612 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:52.619 7ff650410700 1 -- 172.21.2.112:0/230597847 >> 172.21.2.112:0/230597847 conn(0x7ff6480b3040 msgr2=0x7ff6480c3250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:52.612 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:52.619 7ff650410700 1 -- 172.21.2.112:0/230597847 shutdown_connections 2021-02-10T11:38:52.612 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:52.619 7ff650410700 1 -- 172.21.2.112:0/230597847 wait complete. 2021-02-10T11:38:52.620 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2021-02-10T11:38:53.051 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.041 7f396393c700 1 Processor -- start 2021-02-10T11:38:53.051 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.041 7f396393c700 1 -- start start 2021-02-10T11:38:53.052 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.042 7f396393c700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f395c0d1860 0x7f395c1b04b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:53.052 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.042 7f396393c700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f395c1b0ad0 0x7f395c108930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:53.052 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.042 7f396393c700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f395c0fe1b0 con 0x7f395c0d1c60 2021-02-10T11:38:53.052 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.042 7f396393c700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f395c109a50 con 0x7f395c0d1860 2021-02-10T11:38:53.052 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.042 7f396393c700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f395c109740 con 0x7f395c1b0ad0 2021-02-10T11:38:53.052 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.043 7f3960f57700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f395c1b0ad0 0x7f395c108930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:53.052 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.043 7f3961758700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f395c0d1860 0x7f395c1b04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:53.052 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.043 7f3961f59700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7f395c0d1c60 0x7f395c108f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:46296/0 (socket says 172.21.2.112:46296) 2021-02-10T11:38:53.052 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.043 7f3961f59700 1 -- 172.21.2.112:0/547271883 learned_addr learned my addr 172.21.2.112:0/547271883 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:38:53.052 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.044 7f3961758700 1 -- 172.21.2.112:0/547271883 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f395c1b0ad0 msgr2=0x7f395c108930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:53.052 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.044 7f3961758700 1 --2- 172.21.2.112:0/547271883 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f395c1b0ad0 0x7f395c108930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:53.052 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.044 7f3961758700 1 -- 172.21.2.112:0/547271883 >> v1:172.21.2.120:6789/0 conn(0x7f395c0d1c60 legacy=0x7f395c108f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:53.052 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.044 7f3961758700 1 -- 172.21.2.112:0/547271883 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f395c088070 con 0x7f395c0d1860 2021-02-10T11:38:53.053 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.044 7f3960f57700 1 --2- 172.21.2.112:0/547271883 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f395c1b0ad0 0x7f395c108930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:38:53.053 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.044 7f3961758700 1 --2- 172.21.2.112:0/547271883 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f395c0d1860 0x7f395c1b04b0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 rx=0x7f3958006650 tx=0x7f3958009a80).ready entity=mon.1 client_cookie=e981b9506157846d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:53.053 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.044 7f394ffff700 1 -- 172.21.2.112:0/547271883 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f39580009b0 con 0x7f395c0d1860 2021-02-10T11:38:53.053 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.044 7f394ffff700 1 -- 172.21.2.112:0/547271883 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f395800c0c0 con 0x7f395c0d1860 2021-02-10T11:38:53.053 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.044 7f394ffff700 1 -- 172.21.2.112:0/547271883 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f395800c550 con 0x7f395c0d1860 2021-02-10T11:38:53.053 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.044 7f396393c700 1 -- 172.21.2.112:0/547271883 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f395c0d1860 msgr2=0x7f395c1b04b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:53.053 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.044 7f396393c700 1 --2- 172.21.2.112:0/547271883 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f395c0d1860 0x7f395c1b04b0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 rx=0x7f3958006650 tx=0x7f3958009a80).stop 2021-02-10T11:38:53.053 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.045 7f396393c700 1 -- 172.21.2.112:0/547271883 shutdown_connections 2021-02-10T11:38:53.053 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.045 7f396393c700 1 --2- 172.21.2.112:0/547271883 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f395c1b0ad0 0x7f395c108930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:53.053 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.045 7f396393c700 1 --2- 172.21.2.112:0/547271883 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f395c0d1860 0x7f395c1b04b0 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:53.053 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.045 7f396393c700 1 -- 172.21.2.112:0/547271883 >> 172.21.2.112:0/547271883 conn(0x7f395c0ce410 msgr2=0x7f395c0d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:53.053 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.045 7f396393c700 1 -- 172.21.2.112:0/547271883 shutdown_connections 2021-02-10T11:38:53.053 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.045 7f396393c700 1 -- 172.21.2.112:0/547271883 wait complete. 2021-02-10T11:38:53.055 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.060 7f396393c700 1 Processor -- start 2021-02-10T11:38:53.055 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.060 7f396393c700 1 -- start start 2021-02-10T11:38:53.055 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.060 7f396393c700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f395c085270 0x7f395c085670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:53.055 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.060 7f396393c700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f395c085c90 0x7f395c086090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:53.055 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.060 7f396393c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f395c0c2740 0x7f395c0c2b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:53.055 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.060 7f396393c700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f395c084cf0 con 0x7f395c0c2740 2021-02-10T11:38:53.056 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.060 7f396393c700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f395c0be1b0 con 0x7f395c085270 2021-02-10T11:38:53.056 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.060 7f396393c700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f395c0be390 con 0x7f395c085c90 2021-02-10T11:38:53.056 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.061 7f3961f59700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f395c0c2740 0x7f395c0c2b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:53.056 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.061 7f3961758700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f395c085270 0x7f395c085670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:53.056 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.061 7f3961f59700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f395c0c2740 0x7f395c0c2b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:53284/0 (socket says 172.21.2.112:53284) 2021-02-10T11:38:53.056 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.061 7f3961f59700 1 -- 172.21.2.112:0/1512839775 learned_addr learned my addr 172.21.2.112:0/1512839775 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:38:53.056 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.061 7f3960f57700 1 --2- 172.21.2.112:0/1512839775 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f395c085c90 0x7f395c086090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:53.056 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.061 7f3961758700 1 -- 172.21.2.112:0/1512839775 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f395c085c90 msgr2=0x7f395c086090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:53.056 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.061 7f3961758700 1 --2- 172.21.2.112:0/1512839775 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f395c085c90 0x7f395c086090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:53.056 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.061 7f3961758700 1 -- 172.21.2.112:0/1512839775 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f395c0c2740 msgr2=0x7f395c0c2b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:53.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.061 7f3961758700 1 --2- 172.21.2.112:0/1512839775 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f395c0c2740 0x7f395c0c2b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:53.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.061 7f3961758700 1 -- 172.21.2.112:0/1512839775 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f395c0c3160 con 0x7f395c085270 2021-02-10T11:38:53.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.061 7f3961758700 1 --2- 172.21.2.112:0/1512839775 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f395c085270 0x7f395c085670 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 rx=0x7f395800b510 tx=0x7f395800b8b0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:53.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.061 7f394dffb700 1 -- 172.21.2.112:0/1512839775 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f395800bca0 con 0x7f395c085270 2021-02-10T11:38:53.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.061 7f394dffb700 1 -- 172.21.2.112:0/1512839775 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3958000c90 con 0x7f395c085270 2021-02-10T11:38:53.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.061 7f394dffb700 1 -- 172.21.2.112:0/1512839775 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f39580022b0 con 0x7f395c085270 2021-02-10T11:38:53.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.061 7f396393c700 1 -- 172.21.2.112:0/1512839775 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f395c0c3160 con 0x7f395c085270 2021-02-10T11:38:53.058 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.062 7f396393c700 1 -- 172.21.2.112:0/1512839775 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f395c0c3160 con 0x7f395c085270 2021-02-10T11:38:53.058 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.062 7f394dffb700 1 -- 172.21.2.112:0/1512839775 <== mon.1 v2:172.21.2.120:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f39580052e0 con 0x7f395c085270 2021-02-10T11:38:53.058 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.062 7f394dffb700 1 --2- 172.21.2.112:0/1512839775 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f393c018700 0x7f393c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:53.058 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.062 7f394dffb700 1 -- 172.21.2.112:0/1512839775 <== mon.1 v2:172.21.2.120:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f3958032720 con 0x7f395c085270 2021-02-10T11:38:53.058 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.063 7f3960f57700 1 --2- 172.21.2.112:0/1512839775 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f393c018700 0x7f393c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:53.058 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.063 7f3960f57700 1 --2- 172.21.2.112:0/1512839775 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f393c018700 0x7f393c01abc0 crc :-1 s=READY pgs=865 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:53.084 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.092 7f396393c700 1 -- 172.21.2.112:0/1512839775 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f395c082a20 con 0x7f395c085270 2021-02-10T11:38:53.088 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.096 7f394dffb700 1 -- 172.21.2.112:0/1512839775 <== mon.1 v2:172.21.2.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f39580009c0 con 0x7f395c085270 2021-02-10T11:38:53.227 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.235 7f396393c700 1 -- 172.21.2.112:0/1512839775 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_command({"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"} v 0) v1 -- 0x7f395c082a20 con 0x7f395c085270 2021-02-10T11:38:53.440 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.448 7f394dffb700 1 -- 172.21.2.112:0/1512839775 <== mon.1 v2:172.21.2.120:3300/0 7 ==== mon_command_ack([{"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"}]=0 v72) v1 ==== 118+0+0 (secure 0 0 0) 0x7f39580052e0 con 0x7f395c085270 2021-02-10T11:38:53.444 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.452 7f396393c700 1 -- 172.21.2.112:0/1512839775 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f393c018700 msgr2=0x7f393c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:53.444 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.452 7f396393c700 1 --2- 172.21.2.112:0/1512839775 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f393c018700 0x7f393c01abc0 crc :-1 s=READY pgs=865 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:53.444 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.452 7f396393c700 1 -- 172.21.2.112:0/1512839775 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f395c085270 msgr2=0x7f395c085670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:53.444 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.452 7f396393c700 1 --2- 172.21.2.112:0/1512839775 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f395c085270 0x7f395c085670 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 rx=0x7f395800b510 tx=0x7f395800b8b0).stop 2021-02-10T11:38:53.445 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.452 7f396393c700 1 -- 172.21.2.112:0/1512839775 shutdown_connections 2021-02-10T11:38:53.445 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.452 7f396393c700 1 --2- 172.21.2.112:0/1512839775 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f393c018700 0x7f393c01abc0 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:53.445 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.452 7f396393c700 1 --2- 172.21.2.112:0/1512839775 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f395c0c2740 0x7f395c0c2b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:53.445 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.452 7f396393c700 1 --2- 172.21.2.112:0/1512839775 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f395c085c90 0x7f395c086090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:53.445 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.452 7f396393c700 1 --2- 172.21.2.112:0/1512839775 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f395c085270 0x7f395c085670 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:53.445 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.453 7f396393c700 1 -- 172.21.2.112:0/1512839775 >> 172.21.2.112:0/1512839775 conn(0x7f395c0b5f10 msgr2=0x7f395c09f200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:53.445 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.453 7f396393c700 1 -- 172.21.2.112:0/1512839775 shutdown_connections 2021-02-10T11:38:53.445 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.453 7f396393c700 1 -- 172.21.2.112:0/1512839775 wait complete. 2021-02-10T11:38:53.453 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:38:53.882 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.873 7fe223b4e700 1 Processor -- start 2021-02-10T11:38:53.883 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.873 7fe223b4e700 1 -- start start 2021-02-10T11:38:53.883 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.874 7fe223b4e700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe21c1b0470 0x7fe21c1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:53.883 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.874 7fe223b4e700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fe21c0ef610 0x7fe21c0efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:53.883 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.874 7fe223b4e700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fe21c0f0030 0x7fe21c0f0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:53.883 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.874 7fe223b4e700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe21c0e5ea0 con 0x7fe21c0f0030 2021-02-10T11:38:53.883 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.874 7fe223b4e700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe21c0f0b50 con 0x7fe21c0ef610 2021-02-10T11:38:53.883 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.874 7fe223b4e700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe21c0e52e0 con 0x7fe21c1b0470 2021-02-10T11:38:53.883 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.874 7fe22216b700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fe21c0f0030 0x7fe21c0f0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:53.883 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.874 7fe22216b700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fe21c0f0030 0x7fe21c0f0430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:53292/0 (socket says 172.21.2.112:53292) 2021-02-10T11:38:53.883 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.874 7fe22216b700 1 -- 172.21.2.112:0/3425742596 learned_addr learned my addr 172.21.2.112:0/3425742596 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:38:53.884 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.874 7fe221169700 1 --2- 172.21.2.112:0/3425742596 >> v2:172.21.2.120:3300/0 conn(0x7fe21c0ef610 0x7fe21c0efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:53.884 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.874 7fe22196a700 1 --2- 172.21.2.112:0/3425742596 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe21c1b0470 0x7fe21c1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:53.884 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.874 7fe22216b700 1 -- 172.21.2.112:0/3425742596 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe21c1b0470 msgr2=0x7fe21c1b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:53.884 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.874 7fe22216b700 1 --2- 172.21.2.112:0/3425742596 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe21c1b0470 0x7fe21c1b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:53.884 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.874 7fe22216b700 1 -- 172.21.2.112:0/3425742596 >> v2:172.21.2.120:3300/0 conn(0x7fe21c0ef610 msgr2=0x7fe21c0efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:53.884 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.874 7fe22216b700 1 --2- 172.21.2.112:0/3425742596 >> v2:172.21.2.120:3300/0 conn(0x7fe21c0ef610 0x7fe21c0efa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:53.884 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.874 7fe22216b700 1 -- 172.21.2.112:0/3425742596 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe21c0f10d0 con 0x7fe21c0f0030 2021-02-10T11:38:53.884 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.875 7fe22216b700 1 --2- 172.21.2.112:0/3425742596 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe21c0f0030 0x7fe21c0f0430 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 rx=0x7fe214006f00 tx=0x7fe214009f10).ready entity=mon.0 client_cookie=2d42eecc558c5e57 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:53.884 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.875 7fe220968700 1 -- 172.21.2.112:0/3425742596 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe21400b3b0 con 0x7fe21c0f0030 2021-02-10T11:38:53.884 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.875 7fe220968700 1 -- 172.21.2.112:0/3425742596 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe214000d30 con 0x7fe21c0f0030 2021-02-10T11:38:53.884 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.875 7fe220968700 1 -- 172.21.2.112:0/3425742596 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe21400b820 con 0x7fe21c0f0030 2021-02-10T11:38:53.885 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.875 7fe223b4e700 1 -- 172.21.2.112:0/3425742596 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe21c0f0030 msgr2=0x7fe21c0f0430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:53.885 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.875 7fe223b4e700 1 --2- 172.21.2.112:0/3425742596 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe21c0f0030 0x7fe21c0f0430 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 rx=0x7fe214006f00 tx=0x7fe214009f10).stop 2021-02-10T11:38:53.885 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.876 7fe223b4e700 1 -- 172.21.2.112:0/3425742596 shutdown_connections 2021-02-10T11:38:53.885 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.876 7fe223b4e700 1 --2- 172.21.2.112:0/3425742596 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe21c0f0030 0x7fe21c0f0430 unknown :-1 s=CLOSED pgs=1834 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:53.885 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.876 7fe223b4e700 1 --2- 172.21.2.112:0/3425742596 >> v2:172.21.2.120:3300/0 conn(0x7fe21c0ef610 0x7fe21c0efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:53.885 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.876 7fe223b4e700 1 --2- 172.21.2.112:0/3425742596 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe21c1b0470 0x7fe21c1b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:53.885 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.876 7fe223b4e700 1 -- 172.21.2.112:0/3425742596 >> 172.21.2.112:0/3425742596 conn(0x7fe21c0ce3e0 msgr2=0x7fe21c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:53.885 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.876 7fe223b4e700 1 -- 172.21.2.112:0/3425742596 shutdown_connections 2021-02-10T11:38:53.885 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.876 7fe223b4e700 1 -- 172.21.2.112:0/3425742596 wait complete. 2021-02-10T11:38:53.885 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.890 7fe223b4e700 1 Processor -- start 2021-02-10T11:38:53.886 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.890 7fe223b4e700 1 -- start start 2021-02-10T11:38:53.887 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.891 7fe223b4e700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe21c0b11f0 0x7fe21c189d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:53.888 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.891 7fe223b4e700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe21c18a390 0x7fe21c18a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:53.888 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.891 7fe223b4e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe21c18adb0 0x7fe21c187e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:53.888 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.891 7fe223b4e700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe21c0b15f0 con 0x7fe21c18adb0 2021-02-10T11:38:53.888 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.891 7fe223b4e700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe21c188440 con 0x7fe21c18a390 2021-02-10T11:38:53.888 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.891 7fe223b4e700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe21c188660 con 0x7fe21c0b11f0 2021-02-10T11:38:53.888 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.891 7fe22216b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe21c18adb0 0x7fe21c187e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:53.888 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.891 7fe22216b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe21c18adb0 0x7fe21c187e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:53296/0 (socket says 172.21.2.112:53296) 2021-02-10T11:38:53.888 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.891 7fe22216b700 1 -- 172.21.2.112:0/4167333074 learned_addr learned my addr 172.21.2.112:0/4167333074 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:38:53.888 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.891 7fe22196a700 1 --2- 172.21.2.112:0/4167333074 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe21c0b11f0 0x7fe21c189d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:53.888 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.891 7fe221169700 1 --2- 172.21.2.112:0/4167333074 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe21c18a390 0x7fe21c18a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:53.888 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.891 7fe22216b700 1 -- 172.21.2.112:0/4167333074 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe21c0b11f0 msgr2=0x7fe21c189d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:53.888 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.891 7fe22216b700 1 --2- 172.21.2.112:0/4167333074 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe21c0b11f0 0x7fe21c189d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:53.888 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.891 7fe22216b700 1 -- 172.21.2.112:0/4167333074 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe21c18a390 msgr2=0x7fe21c18a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:53.889 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.891 7fe22216b700 1 --2- 172.21.2.112:0/4167333074 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe21c18a390 0x7fe21c18a790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:53.889 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.891 7fe22216b700 1 -- 172.21.2.112:0/4167333074 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe21c188880 con 0x7fe21c18adb0 2021-02-10T11:38:53.889 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.891 7fe22196a700 1 --2- 172.21.2.112:0/4167333074 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe21c0b11f0 0x7fe21c189d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:38:53.889 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.892 7fe221169700 1 --2- 172.21.2.112:0/4167333074 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe21c18a390 0x7fe21c18a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:38:53.889 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.892 7fe22216b700 1 --2- 172.21.2.112:0/4167333074 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe21c18adb0 0x7fe21c187e20 secure :-1 s=READY pgs=1835 cs=0 l=1 rev1=1 rx=0x7fe2140068c0 tx=0x7fe214002d70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:53.889 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.892 7fe20e7fc700 1 -- 172.21.2.112:0/4167333074 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe214000d30 con 0x7fe21c18adb0 2021-02-10T11:38:53.889 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.892 7fe20e7fc700 1 -- 172.21.2.112:0/4167333074 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe214005820 con 0x7fe21c18adb0 2021-02-10T11:38:53.889 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.892 7fe223b4e700 1 -- 172.21.2.112:0/4167333074 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe21c188880 con 0x7fe21c18adb0 2021-02-10T11:38:53.889 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.892 7fe20e7fc700 1 -- 172.21.2.112:0/4167333074 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe214005d40 con 0x7fe21c18adb0 2021-02-10T11:38:53.889 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.892 7fe223b4e700 1 -- 172.21.2.112:0/4167333074 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe21c188880 con 0x7fe21c18adb0 2021-02-10T11:38:53.889 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.893 7fe20e7fc700 1 -- 172.21.2.112:0/4167333074 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe214005a00 con 0x7fe21c18adb0 2021-02-10T11:38:53.889 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.893 7fe20e7fc700 1 --2- 172.21.2.112:0/4167333074 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe204018730 0x7fe20401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:53.889 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.893 7fe20e7fc700 1 -- 172.21.2.112:0/4167333074 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fe214032b20 con 0x7fe21c18adb0 2021-02-10T11:38:53.890 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.893 7fe22196a700 1 --2- 172.21.2.112:0/4167333074 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe204018730 0x7fe20401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:53.890 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.894 7fe22196a700 1 --2- 172.21.2.112:0/4167333074 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe204018730 0x7fe20401abc0 crc :-1 s=READY pgs=866 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:53.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.923 7fe223b4e700 1 -- 172.21.2.112:0/4167333074 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe21c189840 con 0x7fe21c18adb0 2021-02-10T11:38:53.918 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:53.926 7fe20e7fc700 1 -- 172.21.2.112:0/4167333074 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe214007c90 con 0x7fe21c18adb0 2021-02-10T11:38:54.051 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:54.059 7fe223b4e700 1 -- 172.21.2.112:0/4167333074 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe21c189840 con 0x7fe21c18adb0 2021-02-10T11:38:54.051 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:54.059 7fe20e7fc700 1 -- 172.21.2.112:0/4167333074 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+2101 (secure 0 0 0) 0x7fe214005820 con 0x7fe21c18adb0 2021-02-10T11:38:54.052 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:38:54.052 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":72,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9621,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/1174368781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":1174368781},{"type":"v1","addr":"172.21.2.112:6833","nonce":1174368781}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":69}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:38:51.410458","modified":"2021-02-10 11:38:53.445268","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6380},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6380":{"gid":6380,"name":"b","rank":0,"incarnation":70,"state":"up:active","state_seq":3,"addr":"172.21.2.120:6835/2402714407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":2402714407},{"type":"v1","addr":"172.21.2.120:6835","nonce":2402714407}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":9}]} 2021-02-10T11:38:54.052 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 72 2021-02-10T11:38:54.053 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:54.061 7fe223b4e700 1 -- 172.21.2.112:0/4167333074 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe204018730 msgr2=0x7fe20401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:54.053 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:54.061 7fe223b4e700 1 --2- 172.21.2.112:0/4167333074 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe204018730 0x7fe20401abc0 crc :-1 s=READY pgs=866 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:54.053 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:54.061 7fe223b4e700 1 -- 172.21.2.112:0/4167333074 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe21c18adb0 msgr2=0x7fe21c187e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:54.053 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:54.061 7fe223b4e700 1 --2- 172.21.2.112:0/4167333074 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe21c18adb0 0x7fe21c187e20 secure :-1 s=READY pgs=1835 cs=0 l=1 rev1=1 rx=0x7fe2140068c0 tx=0x7fe214002d70).stop 2021-02-10T11:38:54.053 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:54.061 7fe223b4e700 1 -- 172.21.2.112:0/4167333074 shutdown_connections 2021-02-10T11:38:54.054 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:54.061 7fe223b4e700 1 --2- 172.21.2.112:0/4167333074 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe204018730 0x7fe20401abc0 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:54.054 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:54.061 7fe223b4e700 1 --2- 172.21.2.112:0/4167333074 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe21c18a390 0x7fe21c18a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:54.054 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:54.061 7fe223b4e700 1 --2- 172.21.2.112:0/4167333074 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe21c18adb0 0x7fe21c187e20 unknown :-1 s=CLOSED pgs=1835 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:54.054 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:54.061 7fe223b4e700 1 --2- 172.21.2.112:0/4167333074 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe21c0b11f0 0x7fe21c189d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:54.054 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:54.061 7fe223b4e700 1 -- 172.21.2.112:0/4167333074 >> 172.21.2.112:0/4167333074 conn(0x7fe21c0a78c0 msgr2=0x7fe21c0b1f70 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:54.054 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:54.061 7fe223b4e700 1 -- 172.21.2.112:0/4167333074 shutdown_connections 2021-02-10T11:38:54.054 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:54.061 7fe223b4e700 1 -- 172.21.2.112:0/4167333074 wait complete. 2021-02-10T11:38:54.062 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:38:54.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:54.481 7f013d29b700 1 Processor -- start 2021-02-10T11:38:54.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:54.481 7f013d29b700 1 -- start start 2021-02-10T11:38:54.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:54.482 7f013d29b700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f01380d1c70 0x7f01380f20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:54.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:54.482 7f013d29b700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01380f0b10 con 0x7f01380d0340 2021-02-10T11:38:54.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:54.482 7f013d29b700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01381b0be0 con 0x7f01380d1870 2021-02-10T11:38:54.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:54.482 7f013d29b700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01380e5ed0 con 0x7f01380d1c70 2021-02-10T11:38:54.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:54.482 7f013687c700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f01380d0340 0x7f01380ef640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:52764/0 (socket says 172.21.2.112:52764) 2021-02-10T11:38:54.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:54.482 7f013687c700 1 -- 172.21.2.112:0/2495369719 learned_addr learned my addr 172.21.2.112:0/2495369719 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:38:54.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:54.482 7f013787e700 1 --2- 172.21.2.112:0/2495369719 >> v2:172.21.2.112:3300/0 conn(0x7f01380d1c70 0x7f01380f20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:54.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:54.482 7f013607b700 1 -- 172.21.2.112:0/2495369719 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2948388297 0 0) 0x7f01200009b0 con 0x7f01380d0340 2021-02-10T11:38:54.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:54.482 7f013607b700 1 -- 172.21.2.112:0/2495369719 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0124002fe0 con 0x7f01380d0340 2021-02-10T11:38:54.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:54.482 7f013787e700 1 -- 172.21.2.112:0/2495369719 >> v1:172.21.2.120:6789/0 conn(0x7f01380d1870 legacy=0x7f01381b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:54.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:54.482 7f013787e700 1 -- 172.21.2.112:0/2495369719 >> v1:172.21.2.112:6789/0 conn(0x7f01380d0340 legacy=0x7f01380ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:54.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:54.482 7f013787e700 1 -- 172.21.2.112:0/2495369719 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01380efe30 con 0x7f01380d1c70 2021-02-10T11:38:54.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:54.482 7f013787e700 1 --2- 172.21.2.112:0/2495369719 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f01380d1c70 0x7f01380f20a0 secure :-1 s=READY pgs=1837 cs=0 l=1 rev1=1 rx=0x7f0130006e30 tx=0x7f0130009e60).ready entity=mon.0 client_cookie=535741f5bdfb4f4 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:54.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:54.483 7f013607b700 1 -- 172.21.2.112:0/2495369719 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0130006a50 con 0x7f01380d1c70 2021-02-10T11:38:54.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:54.483 7f013607b700 1 -- 172.21.2.112:0/2495369719 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f013000b300 con 0x7f01380d1c70 2021-02-10T11:38:54.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:54.483 7f013607b700 1 -- 172.21.2.112:0/2495369719 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f013000b820 con 0x7f01380d1c70 2021-02-10T11:38:54.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:54.483 7f013d29b700 1 -- 172.21.2.112:0/2495369719 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f01380d1c70 msgr2=0x7f01380f20a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:54.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:54.483 7f013d29b700 1 --2- 172.21.2.112:0/2495369719 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f01380d1c70 0x7f01380f20a0 secure :-1 s=READY pgs=1837 cs=0 l=1 rev1=1 rx=0x7f0130006e30 tx=0x7f0130009e60).stop 2021-02-10T11:38:54.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:54.483 7f013d29b700 1 -- 172.21.2.112:0/2495369719 shutdown_connections 2021-02-10T11:38:54.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:54.483 7f013d29b700 1 --2- 172.21.2.112:0/2495369719 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f01380d1c70 0x7f01380f20a0 unknown :-1 s=CLOSED pgs=1837 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:54.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:54.483 7f013d29b700 1 -- 172.21.2.112:0/2495369719 >> 172.21.2.112:0/2495369719 conn(0x7f01380ce450 msgr2=0x7f01380d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:54.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:54.483 7f013d29b700 1 -- 172.21.2.112:0/2495369719 shutdown_connections 2021-02-10T11:38:54.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:54.483 7f013d29b700 1 -- 172.21.2.112:0/2495369719 wait complete. 2021-02-10T11:38:54.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:54.498 7f013d29b700 1 Processor -- start 2021-02-10T11:38:54.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:54.498 7f013d29b700 1 -- start start 2021-02-10T11:38:54.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:54.499 7f013d29b700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f013818f0a0 0x7f0138196ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:54.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:54.499 7f013d29b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f01381970d0 0x7f01381974d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:54.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:54.499 7f013d29b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0138094650 0x7f0138094a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:54.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:54.499 7f013d29b700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0138095bd0 con 0x7f0138094650 2021-02-10T11:38:54.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:54.499 7f013d29b700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01380ad7c0 con 0x7f013818f0a0 2021-02-10T11:38:54.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:54.499 7f013d29b700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f013818ec70 con 0x7f01381970d0 2021-02-10T11:38:54.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:54.499 7f013787e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0138094650 0x7f0138094a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:54.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:54.499 7f013787e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0138094650 0x7f0138094a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:53312/0 (socket says 172.21.2.112:53312) 2021-02-10T11:38:54.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:54.499 7f013787e700 1 -- 172.21.2.112:0/448965793 learned_addr learned my addr 172.21.2.112:0/448965793 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:38:54.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:54.499 7f013787e700 1 -- 172.21.2.112:0/448965793 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f01381970d0 msgr2=0x7f01381974d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:38:54.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:54.499 7f013787e700 1 --2- 172.21.2.112:0/448965793 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f01381970d0 0x7f01381974d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:54.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:54.499 7f013787e700 1 -- 172.21.2.112:0/448965793 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f013818f0a0 msgr2=0x7f0138196ab0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:38:54.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:54.499 7f013787e700 1 --2- 172.21.2.112:0/448965793 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f013818f0a0 0x7f0138196ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:54.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:54.499 7f013787e700 1 -- 172.21.2.112:0/448965793 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0138095070 con 0x7f0138094650 2021-02-10T11:38:54.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:54.499 7f013787e700 1 --2- 172.21.2.112:0/448965793 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0138094650 0x7f0138094a50 secure :-1 s=READY pgs=1838 cs=0 l=1 rev1=1 rx=0x7f013000b970 tx=0x7f013000bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:54.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:54.500 7f012ffff700 1 -- 172.21.2.112:0/448965793 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0130009380 con 0x7f0138094650 2021-02-10T11:38:54.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:54.500 7f012ffff700 1 -- 172.21.2.112:0/448965793 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0130005820 con 0x7f0138094650 2021-02-10T11:38:54.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:54.500 7f012ffff700 1 -- 172.21.2.112:0/448965793 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0130005d40 con 0x7f0138094650 2021-02-10T11:38:54.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:54.500 7f013d29b700 1 -- 172.21.2.112:0/448965793 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0138095070 con 0x7f0138094650 2021-02-10T11:38:54.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:54.500 7f013d29b700 1 -- 172.21.2.112:0/448965793 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0138095070 con 0x7f0138094650 2021-02-10T11:38:54.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:54.500 7f012ffff700 1 -- 172.21.2.112:0/448965793 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0130005a00 con 0x7f0138094650 2021-02-10T11:38:54.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:54.501 7f012ffff700 1 --2- 172.21.2.112:0/448965793 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f011c0186c0 0x7f011c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:54.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:54.501 7f012ffff700 1 -- 172.21.2.112:0/448965793 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f0130032950 con 0x7f0138094650 2021-02-10T11:38:54.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:54.501 7f013707d700 1 --2- 172.21.2.112:0/448965793 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f011c0186c0 0x7f011c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:54.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:54.501 7f013707d700 1 --2- 172.21.2.112:0/448965793 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f011c0186c0 0x7f011c01abc0 crc :-1 s=READY pgs=867 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:54.524 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:54.532 7f013d29b700 1 -- 172.21.2.112:0/448965793 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01380ad790 con 0x7f0138094650 2021-02-10T11:38:54.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:54.535 7f012ffff700 1 -- 172.21.2.112:0/448965793 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f013002d8e0 con 0x7f0138094650 2021-02-10T11:38:54.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:54.660 7f013d29b700 1 -- 172.21.2.112:0/448965793 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f01380ad790 con 0x7f0138094650 2021-02-10T11:38:54.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:54.661 7f012ffff700 1 -- 172.21.2.112:0/448965793 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) v1 ==== 74+0+13996 (secure 0 0 0) 0x7f013002d8e0 con 0x7f0138094650 2021-02-10T11:38:54.654 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:38:54.654 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":92,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:38:52.412732","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:38:49.384013","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":19,"pool_name":"cephfs_data","create_time":"2021-02-10 11:38:50.454005","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:38:54.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:54.663 7f013d29b700 1 -- 172.21.2.112:0/448965793 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f011c0186c0 msgr2=0x7f011c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:54.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:54.663 7f013d29b700 1 --2- 172.21.2.112:0/448965793 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f011c0186c0 0x7f011c01abc0 crc :-1 s=READY pgs=867 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:54.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:54.663 7f013d29b700 1 -- 172.21.2.112:0/448965793 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0138094650 msgr2=0x7f0138094a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:54.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:54.663 7f013d29b700 1 --2- 172.21.2.112:0/448965793 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0138094650 0x7f0138094a50 secure :-1 s=READY pgs=1838 cs=0 l=1 rev1=1 rx=0x7f013000b970 tx=0x7f013000bd10).stop 2021-02-10T11:38:54.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:54.663 7f013d29b700 1 -- 172.21.2.112:0/448965793 shutdown_connections 2021-02-10T11:38:54.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:54.663 7f013d29b700 1 --2- 172.21.2.112:0/448965793 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f011c0186c0 0x7f011c01abc0 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:54.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:54.663 7f013d29b700 1 --2- 172.21.2.112:0/448965793 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0138094650 0x7f0138094a50 unknown :-1 s=CLOSED pgs=1838 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:54.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:54.663 7f013d29b700 1 --2- 172.21.2.112:0/448965793 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f01381970d0 0x7f01381974d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:54.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:54.663 7f013d29b700 1 --2- 172.21.2.112:0/448965793 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f013818f0a0 0x7f0138196ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:54.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:54.663 7f013d29b700 1 -- 172.21.2.112:0/448965793 >> 172.21.2.112:0/448965793 conn(0x7f01381a38c0 msgr2=0x7f01381a2f00 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:54.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:54.663 7f013d29b700 1 -- 172.21.2.112:0/448965793 shutdown_connections 2021-02-10T11:38:54.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:54.663 7f013d29b700 1 -- 172.21.2.112:0/448965793 wait complete. 2021-02-10T11:38:54.664 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:38:55.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.093 7f734f6cc700 1 Processor -- start 2021-02-10T11:38:55.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.093 7f734f6cc700 1 -- start start 2021-02-10T11:38:55.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.093 7f734f6cc700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f73480d1880 0x7f73481b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:55.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.093 7f734f6cc700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73480fe1d0 con 0x7f73480cff80 2021-02-10T11:38:55.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.093 7f734f6cc700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7348109a70 con 0x7f73481b0af0 2021-02-10T11:38:55.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.093 7f734f6cc700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7348088070 con 0x7f73480d1880 2021-02-10T11:38:55.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.093 7f734dce9700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f73480cff80 0x7f7348109140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:52782/0 (socket says 172.21.2.112:52782) 2021-02-10T11:38:55.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.093 7f734dce9700 1 -- 172.21.2.112:0/3395690938 learned_addr learned my addr 172.21.2.112:0/3395690938 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:38:55.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.093 7f734d4e8700 1 --2- 172.21.2.112:0/3395690938 >> v2:172.21.2.112:3300/0 conn(0x7f73480d1880 0x7f73481b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:55.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.094 7f733bfff700 1 -- 172.21.2.112:0/3395690938 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4020674633 0 0) 0x7f7340001500 con 0x7f73480cff80 2021-02-10T11:38:55.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.094 7f733bfff700 1 -- 172.21.2.112:0/3395690938 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7334002fe0 con 0x7f73480cff80 2021-02-10T11:38:55.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.094 7f734d4e8700 1 -- 172.21.2.112:0/3395690938 >> v1:172.21.2.120:6789/0 conn(0x7f73481b0af0 legacy=0x7f7348108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:55.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.094 7f734d4e8700 1 -- 172.21.2.112:0/3395690938 >> v1:172.21.2.112:6789/0 conn(0x7f73480cff80 legacy=0x7f7348109140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:55.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.094 7f734d4e8700 1 -- 172.21.2.112:0/3395690938 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7348088260 con 0x7f73480d1880 2021-02-10T11:38:55.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.094 7f734d4e8700 1 --2- 172.21.2.112:0/3395690938 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f73480d1880 0x7f73481b04d0 secure :-1 s=READY pgs=1840 cs=0 l=1 rev1=1 rx=0x7f733c006650 tx=0x7f733c009a80).ready entity=mon.0 client_cookie=83288e242a77d47c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:55.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.094 7f733bfff700 1 -- 172.21.2.112:0/3395690938 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f733c0009b0 con 0x7f73480d1880 2021-02-10T11:38:55.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.094 7f733bfff700 1 -- 172.21.2.112:0/3395690938 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f733c00c0c0 con 0x7f73480d1880 2021-02-10T11:38:55.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.094 7f733bfff700 1 -- 172.21.2.112:0/3395690938 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f733c00c5e0 con 0x7f73480d1880 2021-02-10T11:38:55.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.094 7f734f6cc700 1 -- 172.21.2.112:0/3395690938 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f73480d1880 msgr2=0x7f73481b04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:55.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.094 7f734f6cc700 1 --2- 172.21.2.112:0/3395690938 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f73480d1880 0x7f73481b04d0 secure :-1 s=READY pgs=1840 cs=0 l=1 rev1=1 rx=0x7f733c006650 tx=0x7f733c009a80).stop 2021-02-10T11:38:55.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.095 7f734f6cc700 1 -- 172.21.2.112:0/3395690938 shutdown_connections 2021-02-10T11:38:55.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.095 7f734f6cc700 1 --2- 172.21.2.112:0/3395690938 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f73480d1880 0x7f73481b04d0 unknown :-1 s=CLOSED pgs=1840 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:55.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.095 7f734f6cc700 1 -- 172.21.2.112:0/3395690938 >> 172.21.2.112:0/3395690938 conn(0x7f73480ce430 msgr2=0x7f73480d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:55.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.095 7f734f6cc700 1 -- 172.21.2.112:0/3395690938 shutdown_connections 2021-02-10T11:38:55.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.095 7f734f6cc700 1 -- 172.21.2.112:0/3395690938 wait complete. 2021-02-10T11:38:55.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.109 7f734f6cc700 1 Processor -- start 2021-02-10T11:38:55.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.109 7f734f6cc700 1 -- start start 2021-02-10T11:38:55.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.110 7f734f6cc700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7348087210 0x7f7348098390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:55.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.110 7f734f6cc700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f73480ae250 0x7f73480ae650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:55.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.110 7f734f6cc700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f734819efb0 0x7f734819f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:55.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.110 7f734f6cc700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f734818bf50 con 0x7f734819efb0 2021-02-10T11:38:55.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.110 7f734f6cc700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7348098a20 con 0x7f7348087210 2021-02-10T11:38:55.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.110 7f734f6cc700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f734819f9d0 con 0x7f73480ae250 2021-02-10T11:38:55.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.110 7f734dce9700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f734819efb0 0x7f734819f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:55.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.110 7f734dce9700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f734819efb0 0x7f734819f3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:53322/0 (socket says 172.21.2.112:53322) 2021-02-10T11:38:55.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.110 7f734dce9700 1 -- 172.21.2.112:0/509816963 learned_addr learned my addr 172.21.2.112:0/509816963 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:38:55.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.110 7f734dce9700 1 -- 172.21.2.112:0/509816963 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f73480ae250 msgr2=0x7f73480ae650 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:38:55.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.110 7f734dce9700 1 --2- 172.21.2.112:0/509816963 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f73480ae250 0x7f73480ae650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:55.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.110 7f734dce9700 1 -- 172.21.2.112:0/509816963 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7348087210 msgr2=0x7f7348098390 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:38:55.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.110 7f734dce9700 1 --2- 172.21.2.112:0/509816963 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7348087210 0x7f7348098390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:55.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.110 7f734dce9700 1 -- 172.21.2.112:0/509816963 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73480838b0 con 0x7f734819efb0 2021-02-10T11:38:55.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.110 7f734dce9700 1 --2- 172.21.2.112:0/509816963 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f734819efb0 0x7f734819f3b0 secure :-1 s=READY pgs=1841 cs=0 l=1 rev1=1 rx=0x7f7340009980 tx=0x7f734000bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:55.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.111 7f7339ffb700 1 -- 172.21.2.112:0/509816963 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f734000be10 con 0x7f734819efb0 2021-02-10T11:38:55.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.111 7f7339ffb700 1 -- 172.21.2.112:0/509816963 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7340006030 con 0x7f734819efb0 2021-02-10T11:38:55.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.111 7f734f6cc700 1 -- 172.21.2.112:0/509816963 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73480838b0 con 0x7f734819efb0 2021-02-10T11:38:55.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.111 7f734f6cc700 1 -- 172.21.2.112:0/509816963 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73480838b0 con 0x7f734819efb0 2021-02-10T11:38:55.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.111 7f7339ffb700 1 -- 172.21.2.112:0/509816963 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7340004720 con 0x7f734819efb0 2021-02-10T11:38:55.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.111 7f7339ffb700 1 -- 172.21.2.112:0/509816963 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7340000d30 con 0x7f734819efb0 2021-02-10T11:38:55.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.111 7f7339ffb700 1 --2- 172.21.2.112:0/509816963 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f73280186c0 0x7f732801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:55.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.111 7f7339ffb700 1 -- 172.21.2.112:0/509816963 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f73400328f0 con 0x7f734819efb0 2021-02-10T11:38:55.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.112 7f734d4e8700 1 --2- 172.21.2.112:0/509816963 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f73280186c0 0x7f732801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:55.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.112 7f734d4e8700 1 --2- 172.21.2.112:0/509816963 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f73280186c0 0x7f732801abc0 crc :-1 s=READY pgs=868 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:55.135 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.143 7f734f6cc700 1 -- 172.21.2.112:0/509816963 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73480829a0 con 0x7f734819efb0 2021-02-10T11:38:55.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.146 7f7339ffb700 1 -- 172.21.2.112:0/509816963 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f734002b020 con 0x7f734819efb0 2021-02-10T11:38:55.267 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.275 7f734f6cc700 1 -- 172.21.2.112:0/509816963 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f73480829a0 con 0x7f734819efb0 2021-02-10T11:38:55.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.278 7f7339ffb700 1 -- 172.21.2.112:0/509816963 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v53) v1 ==== 184+0+0 (secure 0 0 0) 0x7f734002b020 con 0x7f734819efb0 2021-02-10T11:38:55.270 INFO:teuthology.orchestra.run.gibba012.stderr:updated caps for client.0 2021-02-10T11:38:55.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.279 7f734f6cc700 1 -- 172.21.2.112:0/509816963 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f73280186c0 msgr2=0x7f732801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:55.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.279 7f734f6cc700 1 --2- 172.21.2.112:0/509816963 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f73280186c0 0x7f732801abc0 crc :-1 s=READY pgs=868 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:55.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.279 7f734f6cc700 1 -- 172.21.2.112:0/509816963 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f734819efb0 msgr2=0x7f734819f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:55.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.279 7f734f6cc700 1 --2- 172.21.2.112:0/509816963 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f734819efb0 0x7f734819f3b0 secure :-1 s=READY pgs=1841 cs=0 l=1 rev1=1 rx=0x7f7340009980 tx=0x7f734000bcd0).stop 2021-02-10T11:38:55.272 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.280 7f734f6cc700 1 -- 172.21.2.112:0/509816963 shutdown_connections 2021-02-10T11:38:55.272 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.280 7f734f6cc700 1 --2- 172.21.2.112:0/509816963 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f73280186c0 0x7f732801abc0 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:55.272 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.280 7f734f6cc700 1 --2- 172.21.2.112:0/509816963 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f734819efb0 0x7f734819f3b0 unknown :-1 s=CLOSED pgs=1841 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:55.272 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.280 7f734f6cc700 1 --2- 172.21.2.112:0/509816963 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f73480ae250 0x7f73480ae650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:55.272 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.280 7f734f6cc700 1 --2- 172.21.2.112:0/509816963 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7348087210 0x7f7348098390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:55.272 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.280 7f734f6cc700 1 -- 172.21.2.112:0/509816963 >> 172.21.2.112:0/509816963 conn(0x7f734819c580 msgr2=0x7f7348082bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:55.273 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.280 7f734f6cc700 1 -- 172.21.2.112:0/509816963 shutdown_connections 2021-02-10T11:38:55.273 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.280 7f734f6cc700 1 -- 172.21.2.112:0/509816963 wait complete. 2021-02-10T11:38:55.280 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:38:55.718 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.709 7f237862d700 1 Processor -- start 2021-02-10T11:38:55.718 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.709 7f237862d700 1 -- start start 2021-02-10T11:38:55.718 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.709 7f237862d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f23700d1c80 0x7f23700cef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:55.718 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.709 7f237862d700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23700d0180 con 0x7f23700cf590 2021-02-10T11:38:55.718 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.709 7f237862d700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23700fe1d0 con 0x7f23700d1880 2021-02-10T11:38:55.718 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.709 7f237862d700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23701b0cc0 con 0x7f23700d1c80 2021-02-10T11:38:55.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.710 7f2376c4a700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f23700cf590 0x7f23700cf990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:52796/0 (socket says 172.21.2.112:52796) 2021-02-10T11:38:55.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.710 7f2376449700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7f23700d1880 0x7f23701b04d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:46350/0 (socket says 172.21.2.112:46350) 2021-02-10T11:38:55.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.710 7f2375c48700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f23700d1c80 0x7f23700cef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:55.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.710 7f2376c4a700 1 -- 172.21.2.112:0/3931092213 learned_addr learned my addr 172.21.2.112:0/3931092213 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:38:55.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.710 7f2375447700 1 -- 172.21.2.112:0/3931092213 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3918055342 0 0) 0x7f2368001500 con 0x7f23700cf590 2021-02-10T11:38:55.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.710 7f2375c48700 1 -- 172.21.2.112:0/3931092213 >> v1:172.21.2.120:6789/0 conn(0x7f23700d1880 legacy=0x7f23701b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:55.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.710 7f2375c48700 1 -- 172.21.2.112:0/3931092213 >> v1:172.21.2.112:6789/0 conn(0x7f23700cf590 legacy=0x7f23700cf990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:55.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.710 7f2375c48700 1 -- 172.21.2.112:0/3931092213 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2370109a70 con 0x7f23700d1c80 2021-02-10T11:38:55.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.711 7f2375c48700 1 --2- 172.21.2.112:0/3931092213 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f23700d1c80 0x7f23700cef70 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f236c006650 tx=0x7f236c009a80).ready entity=mon.2 client_cookie=e1b4b899b8b7a286 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:55.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.711 7f2375447700 1 -- 172.21.2.112:0/3931092213 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f236c0009b0 con 0x7f23700d1c80 2021-02-10T11:38:55.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.711 7f2375447700 1 -- 172.21.2.112:0/3931092213 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f236c00c0c0 con 0x7f23700d1c80 2021-02-10T11:38:55.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.711 7f2375447700 1 -- 172.21.2.112:0/3931092213 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f236c00c520 con 0x7f23700d1c80 2021-02-10T11:38:55.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.711 7f237862d700 1 -- 172.21.2.112:0/3931092213 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f23700d1c80 msgr2=0x7f23700cef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:55.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.711 7f237862d700 1 --2- 172.21.2.112:0/3931092213 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f23700d1c80 0x7f23700cef70 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f236c006650 tx=0x7f236c009a80).stop 2021-02-10T11:38:55.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.712 7f237862d700 1 -- 172.21.2.112:0/3931092213 shutdown_connections 2021-02-10T11:38:55.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.712 7f237862d700 1 --2- 172.21.2.112:0/3931092213 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f23700d1c80 0x7f23700cef70 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:55.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.712 7f237862d700 1 -- 172.21.2.112:0/3931092213 >> 172.21.2.112:0/3931092213 conn(0x7f23700ce430 msgr2=0x7f23700d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:55.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.712 7f237862d700 1 -- 172.21.2.112:0/3931092213 shutdown_connections 2021-02-10T11:38:55.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.712 7f237862d700 1 -- 172.21.2.112:0/3931092213 wait complete. 2021-02-10T11:38:55.721 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.726 7f237862d700 1 Processor -- start 2021-02-10T11:38:55.721 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.726 7f237862d700 1 -- start start 2021-02-10T11:38:55.721 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.727 7f237862d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2370086520 0x7f23701a1cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:55.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.727 7f237862d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2370086920 0x7f2370187e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:55.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.727 7f237862d700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f23701a22d0 0x7f23701884a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:55.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.727 7f237862d700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2370099600 con 0x7f2370086920 2021-02-10T11:38:55.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.727 7f237862d700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23700999f0 con 0x7f23701a22d0 2021-02-10T11:38:55.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.727 7f237862d700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2370188ac0 con 0x7f2370086520 2021-02-10T11:38:55.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.727 7f2375c48700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2370086920 0x7f2370187e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:55.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.727 7f2375c48700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2370086920 0x7f2370187e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:53338/0 (socket says 172.21.2.112:53338) 2021-02-10T11:38:55.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.727 7f2375c48700 1 -- 172.21.2.112:0/1917732817 learned_addr learned my addr 172.21.2.112:0/1917732817 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:38:55.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.727 7f2376c4a700 1 --2- 172.21.2.112:0/1917732817 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f23701a22d0 0x7f23701884a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:55.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.727 7f2376449700 1 --2- 172.21.2.112:0/1917732817 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2370086520 0x7f23701a1cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:55.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.727 7f2375c48700 1 -- 172.21.2.112:0/1917732817 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2370086520 msgr2=0x7f23701a1cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:55.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.727 7f2375c48700 1 --2- 172.21.2.112:0/1917732817 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2370086520 0x7f23701a1cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:55.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.727 7f2375c48700 1 -- 172.21.2.112:0/1917732817 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f23701a22d0 msgr2=0x7f23701884a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:55.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.727 7f2375c48700 1 --2- 172.21.2.112:0/1917732817 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f23701a22d0 0x7f23701884a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:55.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.727 7f2375c48700 1 -- 172.21.2.112:0/1917732817 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2370188ce0 con 0x7f2370086920 2021-02-10T11:38:55.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.728 7f2376449700 1 --2- 172.21.2.112:0/1917732817 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2370086520 0x7f23701a1cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:38:55.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.728 7f2375c48700 1 --2- 172.21.2.112:0/1917732817 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2370086920 0x7f2370187e80 secure :-1 s=READY pgs=1843 cs=0 l=1 rev1=1 rx=0x7f236c009630 tx=0x7f236c00b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:55.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.728 7f2362ffd700 1 -- 172.21.2.112:0/1917732817 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f236c0093a0 con 0x7f2370086920 2021-02-10T11:38:55.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.728 7f2362ffd700 1 -- 172.21.2.112:0/1917732817 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f236c00bd70 con 0x7f2370086920 2021-02-10T11:38:55.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.728 7f2362ffd700 1 -- 172.21.2.112:0/1917732817 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f236c0054d0 con 0x7f2370086920 2021-02-10T11:38:55.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.728 7f237862d700 1 -- 172.21.2.112:0/1917732817 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2370188ce0 con 0x7f2370086920 2021-02-10T11:38:55.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.728 7f237862d700 1 -- 172.21.2.112:0/1917732817 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2370189270 con 0x7f2370086920 2021-02-10T11:38:55.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.729 7f2362ffd700 1 -- 172.21.2.112:0/1917732817 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f236c007020 con 0x7f2370086920 2021-02-10T11:38:55.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.729 7f2362ffd700 1 --2- 172.21.2.112:0/1917732817 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f23580186c0 0x7f235801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:55.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.729 7f2362ffd700 1 -- 172.21.2.112:0/1917732817 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f236c032500 con 0x7f2370086920 2021-02-10T11:38:55.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.729 7f2376449700 1 --2- 172.21.2.112:0/1917732817 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f23580186c0 0x7f235801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:55.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.730 7f2376449700 1 --2- 172.21.2.112:0/1917732817 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f23580186c0 0x7f235801abc0 crc :-1 s=READY pgs=869 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:55.751 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.759 7f237862d700 1 -- 172.21.2.112:0/1917732817 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23700fe1d0 con 0x7f2370086920 2021-02-10T11:38:55.754 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.762 7f2362ffd700 1 -- 172.21.2.112:0/1917732817 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f236c009340 con 0x7f2370086920 2021-02-10T11:38:55.883 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.891 7f237862d700 1 -- 172.21.2.112:0/1917732817 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f23700999a0 con 0x7f2370086920 2021-02-10T11:38:55.887 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.895 7f2362ffd700 1 -- 172.21.2.112:0/1917732817 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v54) v1 ==== 184+0+0 (secure 0 0 0) 0x7f236c01f240 con 0x7f2370086920 2021-02-10T11:38:55.890 INFO:teuthology.orchestra.run.gibba012.stderr:updated caps for client.1 2021-02-10T11:38:55.892 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.900 7f237862d700 1 -- 172.21.2.112:0/1917732817 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f23580186c0 msgr2=0x7f235801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:55.892 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.900 7f237862d700 1 --2- 172.21.2.112:0/1917732817 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f23580186c0 0x7f235801abc0 crc :-1 s=READY pgs=869 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:55.893 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.900 7f237862d700 1 -- 172.21.2.112:0/1917732817 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2370086920 msgr2=0x7f2370187e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:55.893 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.900 7f237862d700 1 --2- 172.21.2.112:0/1917732817 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2370086920 0x7f2370187e80 secure :-1 s=READY pgs=1843 cs=0 l=1 rev1=1 rx=0x7f236c009630 tx=0x7f236c00b980).stop 2021-02-10T11:38:55.893 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.900 7f237862d700 1 -- 172.21.2.112:0/1917732817 shutdown_connections 2021-02-10T11:38:55.893 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.900 7f237862d700 1 --2- 172.21.2.112:0/1917732817 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f23580186c0 0x7f235801abc0 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:55.893 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.900 7f237862d700 1 --2- 172.21.2.112:0/1917732817 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f23701a22d0 0x7f23701884a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:55.893 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.900 7f237862d700 1 --2- 172.21.2.112:0/1917732817 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2370086920 0x7f2370187e80 unknown :-1 s=CLOSED pgs=1843 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:55.893 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.900 7f237862d700 1 --2- 172.21.2.112:0/1917732817 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2370086520 0x7f23701a1cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:55.893 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.900 7f237862d700 1 -- 172.21.2.112:0/1917732817 >> 172.21.2.112:0/1917732817 conn(0x7f237019f880 msgr2=0x7f237008f890 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:55.893 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.900 7f237862d700 1 -- 172.21.2.112:0/1917732817 shutdown_connections 2021-02-10T11:38:55.893 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:55.900 7f237862d700 1 -- 172.21.2.112:0/1917732817 wait complete. 2021-02-10T11:38:55.901 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.2 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:38:56.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.331 7f8f6231e700 1 Processor -- start 2021-02-10T11:38:56.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.331 7f8f6231e700 1 -- start start 2021-02-10T11:38:56.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.331 7f8f6231e700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8f5c0d1860 0x7f8f5c1b04b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:56.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.331 7f8f6231e700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f5c109a50 con 0x7f8f5c1b0ad0 2021-02-10T11:38:56.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.331 7f8f6231e700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f5c088070 con 0x7f8f5c0cff60 2021-02-10T11:38:56.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.331 7f8f6231e700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f5c0e5ee0 con 0x7f8f5c0d1860 2021-02-10T11:38:56.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.331 7f8f5b7fe700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f8f5c1b0ad0 0x7f8f5c108930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:52806/0 (socket says 172.21.2.112:52806) 2021-02-10T11:38:56.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.332 7f8f5b7fe700 1 -- 172.21.2.112:0/511046011 learned_addr learned my addr 172.21.2.112:0/511046011 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:38:56.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.332 7f8f5affd700 1 -- 172.21.2.112:0/511046011 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 529785953 0 0) 0x7f8f500009b0 con 0x7f8f5c1b0ad0 2021-02-10T11:38:56.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.332 7f8f5affd700 1 -- 172.21.2.112:0/511046011 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f44002fe0 con 0x7f8f5c1b0ad0 2021-02-10T11:38:56.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.332 7f8f5bfff700 1 --2- 172.21.2.112:0/511046011 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8f5c0d1860 0x7f8f5c1b04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:56.342 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.332 7f8f5affd700 1 -- 172.21.2.112:0/511046011 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4032262559 0 0) 0x7f8f50001010 con 0x7f8f5c1b0ad0 2021-02-10T11:38:56.342 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.332 7f8f5affd700 1 -- 172.21.2.112:0/511046011 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8f5c0d1860 msgr2=0x7f8f5c1b04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:56.342 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.332 7f8f5affd700 1 --2- 172.21.2.112:0/511046011 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8f5c0d1860 0x7f8f5c1b04b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:56.342 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.332 7f8f5affd700 1 -- 172.21.2.112:0/511046011 >> v1:172.21.2.120:6789/0 conn(0x7f8f5c0cff60 legacy=0x7f8f5c109120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:56.342 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.332 7f8f5affd700 1 -- 172.21.2.112:0/511046011 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f5c0d1c60 con 0x7f8f5c1b0ad0 2021-02-10T11:38:56.342 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.332 7f8f5affd700 1 -- 172.21.2.112:0/511046011 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f8f50001590 con 0x7f8f5c1b0ad0 2021-02-10T11:38:56.342 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.332 7f8f5affd700 1 -- 172.21.2.112:0/511046011 >> v1:172.21.2.112:6789/0 conn(0x7f8f5c1b0ad0 legacy=0x7f8f5c108930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:56.342 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.332 7f8f5affd700 1 --2- 172.21.2.112:0/511046011 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8f44003aa0 0x7f8f44005f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:56.342 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.332 7f8f5affd700 1 --2- 172.21.2.112:0/511046011 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8f44006750 0x7f8f44008c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:56.342 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.332 7f8f5affd700 1 --2- 172.21.2.112:0/511046011 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8f44009510 0x7f8f4400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:56.342 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.332 7f8f5affd700 1 -- 172.21.2.112:0/511046011 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f4400c2c0 con 0x7f8f44006750 2021-02-10T11:38:56.342 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.332 7f8f5affd700 1 -- 172.21.2.112:0/511046011 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f4400c4e0 con 0x7f8f44009510 2021-02-10T11:38:56.342 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.332 7f8f5b7fe700 1 --2- 172.21.2.112:0/511046011 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8f44006750 0x7f8f44008c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:56.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.333 7f8f5affd700 1 -- 172.21.2.112:0/511046011 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f44050710 con 0x7f8f44003aa0 2021-02-10T11:38:56.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.333 7f8f5bfff700 1 --2- 172.21.2.112:0/511046011 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8f44003aa0 0x7f8f44005f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:56.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.333 7f8f5b7fe700 1 -- 172.21.2.112:0/511046011 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8f44003aa0 msgr2=0x7f8f44005f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:56.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.333 7f8f6093b700 1 --2- 172.21.2.112:0/511046011 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8f44009510 0x7f8f4400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:56.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.333 7f8f5b7fe700 1 --2- 172.21.2.112:0/511046011 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8f44003aa0 0x7f8f44005f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:56.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.333 7f8f5b7fe700 1 -- 172.21.2.112:0/511046011 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8f44009510 msgr2=0x7f8f4400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:56.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.333 7f8f5b7fe700 1 --2- 172.21.2.112:0/511046011 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8f44009510 0x7f8f4400ba10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:56.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.333 7f8f5b7fe700 1 -- 172.21.2.112:0/511046011 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f44050940 con 0x7f8f44006750 2021-02-10T11:38:56.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.333 7f8f5b7fe700 1 --2- 172.21.2.112:0/511046011 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8f44006750 0x7f8f44008c70 secure :-1 s=READY pgs=1845 cs=0 l=1 rev1=1 rx=0x7f8f50001430 tx=0x7f8f50007d30).ready entity=mon.0 client_cookie=9731f93246fa4d6 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:56.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.333 7f8f5affd700 1 -- 172.21.2.112:0/511046011 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8f5000a520 con 0x7f8f44006750 2021-02-10T11:38:56.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.333 7f8f5affd700 1 -- 172.21.2.112:0/511046011 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f5000a8a0 con 0x7f8f44006750 2021-02-10T11:38:56.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.333 7f8f5affd700 1 -- 172.21.2.112:0/511046011 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8f5000a520 con 0x7f8f44006750 2021-02-10T11:38:56.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.333 7f8f6231e700 1 -- 172.21.2.112:0/511046011 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8f44006750 msgr2=0x7f8f44008c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:56.344 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.333 7f8f6231e700 1 --2- 172.21.2.112:0/511046011 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8f44006750 0x7f8f44008c70 secure :-1 s=READY pgs=1845 cs=0 l=1 rev1=1 rx=0x7f8f50001430 tx=0x7f8f50007d30).stop 2021-02-10T11:38:56.344 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.333 7f8f6093b700 1 -- 172.21.2.112:0/511046011 reap_dead start 2021-02-10T11:38:56.344 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.334 7f8f6231e700 1 -- 172.21.2.112:0/511046011 shutdown_connections 2021-02-10T11:38:56.344 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.334 7f8f6231e700 1 -- 172.21.2.112:0/511046011 >> 172.21.2.112:0/511046011 conn(0x7f8f5c0ce410 msgr2=0x7f8f5c0d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:56.344 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.334 7f8f6231e700 1 -- 172.21.2.112:0/511046011 shutdown_connections 2021-02-10T11:38:56.344 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.334 7f8f6231e700 1 -- 172.21.2.112:0/511046011 wait complete. 2021-02-10T11:38:56.344 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.348 7f8f6231e700 1 Processor -- start 2021-02-10T11:38:56.344 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.349 7f8f6231e700 1 -- start start 2021-02-10T11:38:56.346 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.349 7f8f6231e700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8f5c0bcee0 0x7f8f5c0b5de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:56.346 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.349 7f8f6231e700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8f5c09bce0 0x7f8f5c1a7620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:56.346 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.349 7f8f6231e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8f5c1a7c40 0x7f8f5c09c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:56.346 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.349 7f8f6231e700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f5c0b6400 con 0x7f8f5c1a7c40 2021-02-10T11:38:56.346 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.349 7f8f6231e700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f5c09c0e0 con 0x7f8f5c09bce0 2021-02-10T11:38:56.346 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.349 7f8f6231e700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f5c09c320 con 0x7f8f5c0bcee0 2021-02-10T11:38:56.346 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.349 7f8f6093b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8f5c1a7c40 0x7f8f5c09c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:56.346 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.349 7f8f6093b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8f5c1a7c40 0x7f8f5c09c8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:53358/0 (socket says 172.21.2.112:53358) 2021-02-10T11:38:56.346 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.349 7f8f6093b700 1 -- 172.21.2.112:0/2190680041 learned_addr learned my addr 172.21.2.112:0/2190680041 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:38:56.346 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.350 7f8f6093b700 1 -- 172.21.2.112:0/2190680041 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8f5c0bcee0 msgr2=0x7f8f5c0b5de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:56.347 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.350 7f8f6093b700 1 --2- 172.21.2.112:0/2190680041 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8f5c0bcee0 0x7f8f5c0b5de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:56.347 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.350 7f8f6093b700 1 -- 172.21.2.112:0/2190680041 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8f5c09bce0 msgr2=0x7f8f5c1a7620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:56.347 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.350 7f8f5b7fe700 1 --2- 172.21.2.112:0/2190680041 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8f5c09bce0 0x7f8f5c1a7620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:56.347 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.350 7f8f6093b700 1 --2- 172.21.2.112:0/2190680041 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8f5c09bce0 0x7f8f5c1a7620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:56.347 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.350 7f8f6093b700 1 -- 172.21.2.112:0/2190680041 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f5c0bd2e0 con 0x7f8f5c1a7c40 2021-02-10T11:38:56.347 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.350 7f8f5b7fe700 1 --2- 172.21.2.112:0/2190680041 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8f5c09bce0 0x7f8f5c1a7620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:38:56.347 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.350 7f8f6093b700 1 --2- 172.21.2.112:0/2190680041 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8f5c1a7c40 0x7f8f5c09c8a0 secure :-1 s=READY pgs=1846 cs=0 l=1 rev1=1 rx=0x7f8f5400b910 tx=0x7f8f5400bcb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:56.347 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.350 7f8f58ff9700 1 -- 172.21.2.112:0/2190680041 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8f54005820 con 0x7f8f5c1a7c40 2021-02-10T11:38:56.347 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.350 7f8f58ff9700 1 -- 172.21.2.112:0/2190680041 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f54005ae0 con 0x7f8f5c1a7c40 2021-02-10T11:38:56.348 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.350 7f8f58ff9700 1 -- 172.21.2.112:0/2190680041 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8f54006030 con 0x7f8f5c1a7c40 2021-02-10T11:38:56.348 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.350 7f8f6231e700 1 -- 172.21.2.112:0/2190680041 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f5c0bd2e0 con 0x7f8f5c1a7c40 2021-02-10T11:38:56.348 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.350 7f8f6231e700 1 -- 172.21.2.112:0/2190680041 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f5c1a8330 con 0x7f8f5c1a7c40 2021-02-10T11:38:56.348 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.351 7f8f58ff9700 1 -- 172.21.2.112:0/2190680041 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8f54005cc0 con 0x7f8f5c1a7c40 2021-02-10T11:38:56.348 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.351 7f8f58ff9700 1 --2- 172.21.2.112:0/2190680041 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8f3c0186c0 0x7f8f3c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:56.348 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.351 7f8f58ff9700 1 -- 172.21.2.112:0/2190680041 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f8f54032920 con 0x7f8f5c1a7c40 2021-02-10T11:38:56.349 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.351 7f8f5bfff700 1 --2- 172.21.2.112:0/2190680041 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8f3c0186c0 0x7f8f3c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:56.349 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.352 7f8f5bfff700 1 --2- 172.21.2.112:0/2190680041 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8f3c0186c0 0x7f8f3c01abc0 crc :-1 s=READY pgs=870 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:56.374 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.381 7f8f6231e700 1 -- 172.21.2.112:0/2190680041 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f5c09c0e0 con 0x7f8f5c1a7c40 2021-02-10T11:38:56.376 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.384 7f8f58ff9700 1 -- 172.21.2.112:0/2190680041 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8f5402dfc0 con 0x7f8f5c1a7c40 2021-02-10T11:38:56.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.514 7f8f6231e700 1 -- 172.21.2.112:0/2190680041 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f8f5c09c0e0 con 0x7f8f5c1a7c40 2021-02-10T11:38:56.511 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.519 7f8f58ff9700 1 -- 172.21.2.112:0/2190680041 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.2 v55) v1 ==== 184+0+0 (secure 0 0 0) 0x7f8f54005cc0 con 0x7f8f5c1a7c40 2021-02-10T11:38:56.514 INFO:teuthology.orchestra.run.gibba012.stderr:updated caps for client.2 2021-02-10T11:38:56.516 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.523 7f8f6231e700 1 -- 172.21.2.112:0/2190680041 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8f3c0186c0 msgr2=0x7f8f3c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:56.516 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.523 7f8f6231e700 1 --2- 172.21.2.112:0/2190680041 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8f3c0186c0 0x7f8f3c01abc0 crc :-1 s=READY pgs=870 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:56.516 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.523 7f8f6231e700 1 -- 172.21.2.112:0/2190680041 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8f5c1a7c40 msgr2=0x7f8f5c09c8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:56.516 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.523 7f8f6231e700 1 --2- 172.21.2.112:0/2190680041 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8f5c1a7c40 0x7f8f5c09c8a0 secure :-1 s=READY pgs=1846 cs=0 l=1 rev1=1 rx=0x7f8f5400b910 tx=0x7f8f5400bcb0).stop 2021-02-10T11:38:56.516 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.524 7f8f6231e700 1 -- 172.21.2.112:0/2190680041 shutdown_connections 2021-02-10T11:38:56.516 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.524 7f8f6231e700 1 --2- 172.21.2.112:0/2190680041 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8f3c0186c0 0x7f8f3c01abc0 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:56.516 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.524 7f8f6231e700 1 --2- 172.21.2.112:0/2190680041 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8f5c09bce0 0x7f8f5c1a7620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:56.517 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.524 7f8f6231e700 1 --2- 172.21.2.112:0/2190680041 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8f5c1a7c40 0x7f8f5c09c8a0 unknown :-1 s=CLOSED pgs=1846 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:56.517 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.524 7f8f6231e700 1 --2- 172.21.2.112:0/2190680041 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8f5c0bcee0 0x7f8f5c0b5de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:56.517 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.524 7f8f6231e700 1 -- 172.21.2.112:0/2190680041 >> 172.21.2.112:0/2190680041 conn(0x7f8f5c09e660 msgr2=0x7f8f5c0a9550 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:56.517 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.524 7f8f6231e700 1 -- 172.21.2.112:0/2190680041 shutdown_connections 2021-02-10T11:38:56.517 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.524 7f8f6231e700 1 -- 172.21.2.112:0/2190680041 wait complete. 2021-02-10T11:38:56.524 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.3 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:38:56.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.955 7fe52965c700 1 Processor -- start 2021-02-10T11:38:56.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.955 7fe52965c700 1 -- start start 2021-02-10T11:38:56.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.955 7fe52965c700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fe5240d1880 0x7fe5241b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:56.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.955 7fe52965c700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fe5240cff80 0x7fe524109140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:56.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.955 7fe52965c700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5240fe1d0 con 0x7fe5241b0af0 2021-02-10T11:38:56.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.955 7fe52965c700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe524109ae0 con 0x7fe5240d1880 2021-02-10T11:38:56.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.955 7fe52965c700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe524109760 con 0x7fe5240cff80 2021-02-10T11:38:56.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.956 7fe522abe700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fe5241b0af0 0x7fe524108950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:52824/0 (socket says 172.21.2.112:52824) 2021-02-10T11:38:56.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.956 7fe522abe700 1 -- 172.21.2.112:0/1062173732 learned_addr learned my addr 172.21.2.112:0/1062173732 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:38:56.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.956 7fe5232bf700 1 --2- 172.21.2.112:0/1062173732 >> v2:172.21.2.112:3300/0 conn(0x7fe5240d1880 0x7fe5241b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:56.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.956 7fe5232bf700 1 -- 172.21.2.112:0/1062173732 >> v2:172.21.2.120:3300/0 conn(0x7fe5240cff80 msgr2=0x7fe524109140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:56.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.956 7fe5232bf700 1 --2- 172.21.2.112:0/1062173732 >> v2:172.21.2.120:3300/0 conn(0x7fe5240cff80 0x7fe524109140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:56.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.956 7fe5232bf700 1 -- 172.21.2.112:0/1062173732 >> v1:172.21.2.112:6789/0 conn(0x7fe5241b0af0 legacy=0x7fe524108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:56.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.956 7fe5222bd700 1 -- 172.21.2.112:0/1062173732 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3841976978 0 0) 0x7fe5180010b0 con 0x7fe5241b0af0 2021-02-10T11:38:56.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.956 7fe5232bf700 1 -- 172.21.2.112:0/1062173732 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe524088070 con 0x7fe5240d1880 2021-02-10T11:38:56.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.957 7fe5232bf700 1 --2- 172.21.2.112:0/1062173732 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe5240d1880 0x7fe5241b04d0 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 rx=0x7fe514006650 tx=0x7fe514009a00).ready entity=mon.0 client_cookie=537026f9825e4cce server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:56.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.957 7fe5222bd700 1 -- 172.21.2.112:0/1062173732 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe5140009b0 con 0x7fe5240d1880 2021-02-10T11:38:56.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.957 7fe5222bd700 1 -- 172.21.2.112:0/1062173732 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe51400c0c0 con 0x7fe5240d1880 2021-02-10T11:38:56.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.957 7fe5222bd700 1 -- 172.21.2.112:0/1062173732 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe51400c5a0 con 0x7fe5240d1880 2021-02-10T11:38:56.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.957 7fe52965c700 1 -- 172.21.2.112:0/1062173732 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe5240d1880 msgr2=0x7fe5241b04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:56.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.957 7fe52965c700 1 --2- 172.21.2.112:0/1062173732 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe5240d1880 0x7fe5241b04d0 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 rx=0x7fe514006650 tx=0x7fe514009a00).stop 2021-02-10T11:38:56.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.957 7fe52965c700 1 -- 172.21.2.112:0/1062173732 shutdown_connections 2021-02-10T11:38:56.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.957 7fe52965c700 1 --2- 172.21.2.112:0/1062173732 >> v2:172.21.2.120:3300/0 conn(0x7fe5240cff80 0x7fe524109140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:56.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.957 7fe52965c700 1 --2- 172.21.2.112:0/1062173732 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe5240d1880 0x7fe5241b04d0 unknown :-1 s=CLOSED pgs=1848 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:56.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.957 7fe52965c700 1 -- 172.21.2.112:0/1062173732 >> 172.21.2.112:0/1062173732 conn(0x7fe5240ce430 msgr2=0x7fe5240d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:56.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.958 7fe52965c700 1 -- 172.21.2.112:0/1062173732 shutdown_connections 2021-02-10T11:38:56.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.958 7fe52965c700 1 -- 172.21.2.112:0/1062173732 wait complete. 2021-02-10T11:38:56.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.972 7fe52965c700 1 Processor -- start 2021-02-10T11:38:56.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.972 7fe52965c700 1 -- start start 2021-02-10T11:38:56.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.973 7fe52965c700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe5241b9280 0x7fe52418f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:56.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.973 7fe52965c700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe5241b9680 0x7fe52418fd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:56.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.973 7fe52965c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe5241acd60 0x7fe5241ad160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:56.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.973 7fe52965c700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe524190390 con 0x7fe5241acd60 2021-02-10T11:38:56.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.973 7fe52965c700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5241907c0 con 0x7fe5241b9680 2021-02-10T11:38:56.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.973 7fe52965c700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5241ac3d0 con 0x7fe5241b9280 2021-02-10T11:38:56.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.973 7fe523ac0700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe5241acd60 0x7fe5241ad160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:56.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.973 7fe523ac0700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe5241acd60 0x7fe5241ad160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:53370/0 (socket says 172.21.2.112:53370) 2021-02-10T11:38:56.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.973 7fe523ac0700 1 -- 172.21.2.112:0/3090030656 learned_addr learned my addr 172.21.2.112:0/3090030656 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:38:56.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.973 7fe523ac0700 1 -- 172.21.2.112:0/3090030656 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe5241b9280 msgr2=0x7fe52418f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:56.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.973 7fe523ac0700 1 --2- 172.21.2.112:0/3090030656 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe5241b9280 0x7fe52418f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:56.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.973 7fe523ac0700 1 -- 172.21.2.112:0/3090030656 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe5241b9680 msgr2=0x7fe52418fd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:56.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.973 7fe523ac0700 1 --2- 172.21.2.112:0/3090030656 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe5241b9680 0x7fe52418fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:56.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.973 7fe523ac0700 1 -- 172.21.2.112:0/3090030656 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5241ad780 con 0x7fe5241acd60 2021-02-10T11:38:56.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.973 7fe523ac0700 1 --2- 172.21.2.112:0/3090030656 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe5241acd60 0x7fe5241ad160 secure :-1 s=READY pgs=1849 cs=0 l=1 rev1=1 rx=0x7fe51c009980 tx=0x7fe51c00bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:56.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.974 7fe50bfff700 1 -- 172.21.2.112:0/3090030656 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe51c003020 con 0x7fe5241acd60 2021-02-10T11:38:56.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.974 7fe50bfff700 1 -- 172.21.2.112:0/3090030656 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe51c005a80 con 0x7fe5241acd60 2021-02-10T11:38:56.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.974 7fe52965c700 1 -- 172.21.2.112:0/3090030656 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5241ad780 con 0x7fe5241acd60 2021-02-10T11:38:56.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.974 7fe52965c700 1 -- 172.21.2.112:0/3090030656 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5241add10 con 0x7fe5241acd60 2021-02-10T11:38:56.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.974 7fe50bfff700 1 -- 172.21.2.112:0/3090030656 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe51c006030 con 0x7fe5241acd60 2021-02-10T11:38:56.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.974 7fe50bfff700 1 -- 172.21.2.112:0/3090030656 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe51c005c60 con 0x7fe5241acd60 2021-02-10T11:38:56.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.974 7fe50bfff700 1 --2- 172.21.2.112:0/3090030656 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe5040186d0 0x7fe50401abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:56.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.974 7fe50bfff700 1 -- 172.21.2.112:0/3090030656 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fe51c032b80 con 0x7fe5241acd60 2021-02-10T11:38:56.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.975 7fe5232bf700 1 --2- 172.21.2.112:0/3090030656 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe5040186d0 0x7fe50401abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:56.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:56.975 7fe5232bf700 1 --2- 172.21.2.112:0/3090030656 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe5040186d0 0x7fe50401abd0 crc :-1 s=READY pgs=871 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:56.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:57.005 7fe52965c700 1 -- 172.21.2.112:0/3090030656 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5240829a0 con 0x7fe5241acd60 2021-02-10T11:38:57.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:57.008 7fe50bfff700 1 -- 172.21.2.112:0/3090030656 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe51c005c90 con 0x7fe5241acd60 2021-02-10T11:38:57.131 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:57.139 7fe52965c700 1 -- 172.21.2.112:0/3090030656 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fe5240829a0 con 0x7fe5241acd60 2021-02-10T11:38:57.134 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:57.142 7fe50bfff700 1 -- 172.21.2.112:0/3090030656 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.3 v56) v1 ==== 184+0+0 (secure 0 0 0) 0x7fe51c005a80 con 0x7fe5241acd60 2021-02-10T11:38:57.138 INFO:teuthology.orchestra.run.gibba012.stderr:updated caps for client.3 2021-02-10T11:38:57.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:57.148 7fe52965c700 1 -- 172.21.2.112:0/3090030656 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe5040186d0 msgr2=0x7fe50401abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:57.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:57.148 7fe52965c700 1 --2- 172.21.2.112:0/3090030656 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe5040186d0 0x7fe50401abd0 crc :-1 s=READY pgs=871 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:57.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:57.148 7fe52965c700 1 -- 172.21.2.112:0/3090030656 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe5241acd60 msgr2=0x7fe5241ad160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:57.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:57.148 7fe52965c700 1 --2- 172.21.2.112:0/3090030656 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe5241acd60 0x7fe5241ad160 secure :-1 s=READY pgs=1849 cs=0 l=1 rev1=1 rx=0x7fe51c009980 tx=0x7fe51c00bcd0).stop 2021-02-10T11:38:57.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:57.148 7fe52965c700 1 -- 172.21.2.112:0/3090030656 shutdown_connections 2021-02-10T11:38:57.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:57.148 7fe52965c700 1 --2- 172.21.2.112:0/3090030656 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe5040186d0 0x7fe50401abd0 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:57.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:57.148 7fe52965c700 1 --2- 172.21.2.112:0/3090030656 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe5241b9680 0x7fe52418fd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:57.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:57.148 7fe52965c700 1 --2- 172.21.2.112:0/3090030656 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe5241acd60 0x7fe5241ad160 unknown :-1 s=CLOSED pgs=1849 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:57.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:57.148 7fe52965c700 1 --2- 172.21.2.112:0/3090030656 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe5241b9280 0x7fe52418f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:57.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:57.148 7fe52965c700 1 -- 172.21.2.112:0/3090030656 >> 172.21.2.112:0/3090030656 conn(0x7fe5240845b0 msgr2=0x7fe52418c290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:57.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:57.148 7fe52965c700 1 -- 172.21.2.112:0/3090030656 shutdown_connections 2021-02-10T11:38:57.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:57.148 7fe52965c700 1 -- 172.21.2.112:0/3090030656 wait complete. 2021-02-10T11:38:57.149 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:38:57.576 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:57.566 7efc825d5700 1 Processor -- start 2021-02-10T11:38:57.576 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:57.566 7efc825d5700 1 -- start start 2021-02-10T11:38:57.576 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:57.567 7efc825d5700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efc7c0d1870 0x7efc7c1b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:57.576 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:57.567 7efc825d5700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7efc7c1b0a10 0x7efc7c0cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:57.576 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:57.567 7efc825d5700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7efc7c0d1c70 0x7efc7c0cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:57.576 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:57.567 7efc825d5700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc7c0fe230 con 0x7efc7c1b0a10 2021-02-10T11:38:57.576 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:57.567 7efc825d5700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc7c0e5ed0 con 0x7efc7c0d1c70 2021-02-10T11:38:57.577 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:57.567 7efc825d5700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc7c0f0b10 con 0x7efc7c0d1870 2021-02-10T11:38:57.577 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:57.567 7efc7b7fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7efc7c1b0a10 0x7efc7c0cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:57.577 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:57.567 7efc80bf2700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7efc7c0d1c70 0x7efc7c0cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:57.577 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:57.567 7efc7b7fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7efc7c1b0a10 0x7efc7c0cf330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:53378/0 (socket says 172.21.2.112:53378) 2021-02-10T11:38:57.577 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:57.567 7efc80bf2700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7efc7c0d1c70 0x7efc7c0cf950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3300/0 says I am v2:172.21.2.112:55654/0 (socket says 172.21.2.112:55654) 2021-02-10T11:38:57.577 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:57.567 7efc7bfff700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efc7c0d1870 0x7efc7c1b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:57.577 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:57.567 7efc7b7fe700 1 -- 172.21.2.112:0/988518053 learned_addr learned my addr 172.21.2.112:0/988518053 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:38:57.577 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:57.568 7efc7bfff700 1 -- 172.21.2.112:0/988518053 >> v2:172.21.2.120:3300/0 conn(0x7efc7c0d1c70 msgr2=0x7efc7c0cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:57.577 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:57.568 7efc7bfff700 1 --2- 172.21.2.112:0/988518053 >> v2:172.21.2.120:3300/0 conn(0x7efc7c0d1c70 0x7efc7c0cf950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:57.577 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:57.568 7efc7bfff700 1 -- 172.21.2.112:0/988518053 >> v2:172.21.2.112:3300/0 conn(0x7efc7c1b0a10 msgr2=0x7efc7c0cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:57.577 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:57.568 7efc7bfff700 1 --2- 172.21.2.112:0/988518053 >> v2:172.21.2.112:3300/0 conn(0x7efc7c1b0a10 0x7efc7c0cf330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:57.577 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:57.568 7efc7bfff700 1 -- 172.21.2.112:0/988518053 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc7c0cff70 con 0x7efc7c0d1870 2021-02-10T11:38:57.577 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:57.568 7efc7bfff700 1 --2- 172.21.2.112:0/988518053 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efc7c0d1870 0x7efc7c1b03f0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7efc6c006650 tx=0x7efc6c009a80).ready entity=mon.2 client_cookie=cb36ed62595d66e9 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:57.577 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:57.568 7efc7affd700 1 -- 172.21.2.112:0/988518053 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efc6c0009b0 con 0x7efc7c0d1870 2021-02-10T11:38:57.578 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:57.568 7efc7affd700 1 -- 172.21.2.112:0/988518053 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc6c00c0c0 con 0x7efc7c0d1870 2021-02-10T11:38:57.578 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:57.568 7efc7affd700 1 -- 172.21.2.112:0/988518053 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efc6c00c5e0 con 0x7efc7c0d1870 2021-02-10T11:38:57.578 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:57.568 7efc825d5700 1 -- 172.21.2.112:0/988518053 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efc7c0d1870 msgr2=0x7efc7c1b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:57.578 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:57.568 7efc825d5700 1 --2- 172.21.2.112:0/988518053 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efc7c0d1870 0x7efc7c1b03f0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7efc6c006650 tx=0x7efc6c009a80).stop 2021-02-10T11:38:57.578 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:57.569 7efc825d5700 1 -- 172.21.2.112:0/988518053 shutdown_connections 2021-02-10T11:38:57.578 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:57.569 7efc825d5700 1 --2- 172.21.2.112:0/988518053 >> v2:172.21.2.120:3300/0 conn(0x7efc7c0d1c70 0x7efc7c0cf950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:57.578 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:57.569 7efc825d5700 1 --2- 172.21.2.112:0/988518053 >> v2:172.21.2.112:3300/0 conn(0x7efc7c1b0a10 0x7efc7c0cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:57.578 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:57.569 7efc825d5700 1 --2- 172.21.2.112:0/988518053 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efc7c0d1870 0x7efc7c1b03f0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:57.578 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:57.569 7efc825d5700 1 -- 172.21.2.112:0/988518053 >> 172.21.2.112:0/988518053 conn(0x7efc7c0ce450 msgr2=0x7efc7c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:57.578 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:57.569 7efc825d5700 1 -- 172.21.2.112:0/988518053 shutdown_connections 2021-02-10T11:38:57.578 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:57.569 7efc825d5700 1 -- 172.21.2.112:0/988518053 wait complete. 2021-02-10T11:38:57.578 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:57.584 7efc825d5700 1 Processor -- start 2021-02-10T11:38:57.579 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:57.584 7efc825d5700 1 -- start start 2021-02-10T11:38:57.580 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:57.584 7efc825d5700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7efc7c195650 0x7efc7c0b5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:57.580 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:57.584 7efc825d5700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efc7c0b5b80 0x7efc7c1a2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:57.580 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:57.584 7efc825d5700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efc7c0acfa0 0x7efc7c1a3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:57.580 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:57.584 7efc825d5700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc7c0b6010 con 0x7efc7c0acfa0 2021-02-10T11:38:57.580 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:57.584 7efc825d5700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc7c1a3a60 con 0x7efc7c195650 2021-02-10T11:38:57.581 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:57.584 7efc825d5700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc7c1a3c40 con 0x7efc7c0b5b80 2021-02-10T11:38:57.581 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:57.584 7efc7b7fe700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efc7c0b5b80 0x7efc7c1a2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:57.581 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:57.584 7efc7b7fe700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efc7c0b5b80 0x7efc7c1a2e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3301/0 says I am v2:172.21.2.112:53064/0 (socket says 172.21.2.112:53064) 2021-02-10T11:38:57.581 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:57.584 7efc7b7fe700 1 -- 172.21.2.112:0/3529249099 learned_addr learned my addr 172.21.2.112:0/3529249099 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:38:57.581 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:57.584 7efc80bf2700 1 --2- 172.21.2.112:0/3529249099 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efc7c0acfa0 0x7efc7c1a3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:57.581 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:57.584 7efc7bfff700 1 --2- 172.21.2.112:0/3529249099 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7efc7c195650 0x7efc7c0b5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:57.581 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:57.585 7efc7b7fe700 1 -- 172.21.2.112:0/3529249099 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7efc7c195650 msgr2=0x7efc7c0b5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:57.581 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:57.585 7efc7b7fe700 1 --2- 172.21.2.112:0/3529249099 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7efc7c195650 0x7efc7c0b5560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:57.581 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:57.585 7efc7b7fe700 1 -- 172.21.2.112:0/3529249099 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efc7c0acfa0 msgr2=0x7efc7c1a3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:57.581 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:57.585 7efc7b7fe700 1 --2- 172.21.2.112:0/3529249099 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efc7c0acfa0 0x7efc7c1a3440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:57.581 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:57.585 7efc7b7fe700 1 -- 172.21.2.112:0/3529249099 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc7c0bcae0 con 0x7efc7c0b5b80 2021-02-10T11:38:57.581 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:57.585 7efc7b7fe700 1 --2- 172.21.2.112:0/3529249099 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efc7c0b5b80 0x7efc7c1a2e20 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7efc7000b5e0 tx=0x7efc7000b980).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:57.581 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:57.585 7efc78ff9700 1 -- 172.21.2.112:0/3529249099 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efc700009b0 con 0x7efc7c0b5b80 2021-02-10T11:38:57.581 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:57.585 7efc825d5700 1 -- 172.21.2.112:0/3529249099 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc7c0bcae0 con 0x7efc7c0b5b80 2021-02-10T11:38:57.582 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:57.585 7efc825d5700 1 -- 172.21.2.112:0/3529249099 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc7c0bd070 con 0x7efc7c0b5b80 2021-02-10T11:38:57.582 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:57.585 7efc78ff9700 1 -- 172.21.2.112:0/3529249099 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc700063c0 con 0x7efc7c0b5b80 2021-02-10T11:38:57.582 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:57.585 7efc78ff9700 1 -- 172.21.2.112:0/3529249099 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efc7000bd70 con 0x7efc7c0b5b80 2021-02-10T11:38:57.582 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:57.586 7efc78ff9700 1 -- 172.21.2.112:0/3529249099 <== mon.2 v2:172.21.2.120:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7efc70002240 con 0x7efc7c0b5b80 2021-02-10T11:38:57.582 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:57.586 7efc78ff9700 1 --2- 172.21.2.112:0/3529249099 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7efc5c0186c0 0x7efc5c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:57.582 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:57.586 7efc78ff9700 1 -- 172.21.2.112:0/3529249099 <== mon.2 v2:172.21.2.120:3301/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5573+0+0 (secure 0 0 0) 0x7efc700325f0 con 0x7efc7c0b5b80 2021-02-10T11:38:57.582 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:57.586 7efc7bfff700 1 --2- 172.21.2.112:0/3529249099 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7efc5c0186c0 0x7efc5c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:57.582 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:57.586 7efc7bfff700 1 --2- 172.21.2.112:0/3529249099 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7efc5c0186c0 0x7efc5c01abc0 crc :-1 s=READY pgs=872 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:57.609 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:57.617 7efc825d5700 1 -- 172.21.2.112:0/3529249099 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc7c0b6010 con 0x7efc7c0b5b80 2021-02-10T11:38:57.612 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:57.619 7efc78ff9700 1 -- 172.21.2.112:0/3529249099 <== mon.2 v2:172.21.2.120:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7efc70002030 con 0x7efc7c0b5b80 2021-02-10T11:38:57.740 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:57.748 7efc825d5700 1 -- 172.21.2.112:0/3529249099 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efc7c0b6010 con 0x7efc7c0b5b80 2021-02-10T11:38:57.741 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:57.749 7efc78ff9700 1 -- 172.21.2.112:0/3529249099 <== mon.2 v2:172.21.2.120:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+2101 (secure 0 0 0) 0x7efc7000bd70 con 0x7efc7c0b5b80 2021-02-10T11:38:57.744 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:38:57.744 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":72,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9621,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/1174368781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":1174368781},{"type":"v1","addr":"172.21.2.112:6833","nonce":1174368781}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":69}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:38:51.410458","modified":"2021-02-10 11:38:53.445268","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6380},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6380":{"gid":6380,"name":"b","rank":0,"incarnation":70,"state":"up:active","state_seq":3,"addr":"172.21.2.120:6835/2402714407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":2402714407},{"type":"v1","addr":"172.21.2.120:6835","nonce":2402714407}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":9}]} 2021-02-10T11:38:57.744 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 72 2021-02-10T11:38:57.745 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:57.752 7efc825d5700 1 -- 172.21.2.112:0/3529249099 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7efc5c0186c0 msgr2=0x7efc5c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:57.745 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:57.752 7efc825d5700 1 --2- 172.21.2.112:0/3529249099 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7efc5c0186c0 0x7efc5c01abc0 crc :-1 s=READY pgs=872 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:57.745 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:57.752 7efc825d5700 1 -- 172.21.2.112:0/3529249099 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efc7c0b5b80 msgr2=0x7efc7c1a2e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:57.745 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:57.752 7efc825d5700 1 --2- 172.21.2.112:0/3529249099 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efc7c0b5b80 0x7efc7c1a2e20 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7efc7000b5e0 tx=0x7efc7000b980).stop 2021-02-10T11:38:57.745 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:57.753 7efc825d5700 1 -- 172.21.2.112:0/3529249099 shutdown_connections 2021-02-10T11:38:57.745 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:57.753 7efc825d5700 1 --2- 172.21.2.112:0/3529249099 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7efc5c0186c0 0x7efc5c01abc0 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:57.745 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:57.753 7efc825d5700 1 --2- 172.21.2.112:0/3529249099 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efc7c0acfa0 0x7efc7c1a3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:57.745 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:57.753 7efc825d5700 1 --2- 172.21.2.112:0/3529249099 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efc7c0b5b80 0x7efc7c1a2e20 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:57.745 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:57.753 7efc825d5700 1 --2- 172.21.2.112:0/3529249099 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7efc7c195650 0x7efc7c0b5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:57.745 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:57.753 7efc825d5700 1 -- 172.21.2.112:0/3529249099 >> 172.21.2.112:0/3529249099 conn(0x7efc7c0b3040 msgr2=0x7efc7c0c3250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:57.746 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:57.753 7efc825d5700 1 -- 172.21.2.112:0/3529249099 shutdown_connections 2021-02-10T11:38:57.746 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:57.753 7efc825d5700 1 -- 172.21.2.112:0/3529249099 wait complete. 2021-02-10T11:38:57.955 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 72, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:38:51.410458', 'modified': '2021-02-10 11:38:53.445268', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6380}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6380': {'gid': 6380, 'name': 'b', 'rank': 0, 'incarnation': 70, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.120:6835/2402714407', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.120:6834', 'nonce': 2402714407}, {'type': 'v1', 'addr': '172.21.2.120:6835', 'nonce': 2402714407}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:38:57.955 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:38:57.956 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:38:58.354 INFO:teuthology.orchestra.run.gibba020.stdout:{ 2021-02-10T11:38:58.354 INFO:teuthology.orchestra.run.gibba020.stdout: "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", 2021-02-10T11:38:58.355 INFO:teuthology.orchestra.run.gibba020.stdout: "whoami": 0, 2021-02-10T11:38:58.355 INFO:teuthology.orchestra.run.gibba020.stdout: "id": 6380, 2021-02-10T11:38:58.355 INFO:teuthology.orchestra.run.gibba020.stdout: "want_state": "up:active", 2021-02-10T11:38:58.355 INFO:teuthology.orchestra.run.gibba020.stdout: "state": "up:active", 2021-02-10T11:38:58.355 INFO:teuthology.orchestra.run.gibba020.stdout: "rank_uptime": 6.9491617249999997, 2021-02-10T11:38:58.355 INFO:teuthology.orchestra.run.gibba020.stdout: "mdsmap_epoch": 72, 2021-02-10T11:38:58.355 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch": 92, 2021-02-10T11:38:58.355 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch_barrier": 92, 2021-02-10T11:38:58.355 INFO:teuthology.orchestra.run.gibba020.stdout: "uptime": 12.582195671999999 2021-02-10T11:38:58.355 INFO:teuthology.orchestra.run.gibba020.stdout:} 2021-02-10T11:38:58.358 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", "whoami": 0, "id": 6380, "want_state": "up:active", "state": "up:active", "rank_uptime": 6.9491617249999997, "mdsmap_epoch": 72, "osdmap_epoch": 92, "osdmap_epoch_barrier": 92, "uptime": 12.582195671999999 } 2021-02-10T11:38:58.359 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:38:58.787 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:58.778 7f6015c55700 1 Processor -- start 2021-02-10T11:38:58.787 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:58.778 7f6015c55700 1 -- start start 2021-02-10T11:38:58.787 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:58.779 7f6015c55700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f60100d1870 0x7f60101b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:58.787 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:58.779 7f6015c55700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f60100f0650 0x7f60100cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:58.787 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:58.779 7f6015c55700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60100f0b10 con 0x7f60101b0a10 2021-02-10T11:38:58.787 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:58.779 7f6015c55700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60100e5ed0 con 0x7f60100f0650 2021-02-10T11:38:58.787 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:58.779 7f6015c55700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60100d1c70 con 0x7f60100d1870 2021-02-10T11:38:58.787 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:58.779 7f600effd700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f60101b0a10 0x7f60100cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:52850/0 (socket says 172.21.2.112:52850) 2021-02-10T11:38:58.788 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:58.779 7f600effd700 1 -- 172.21.2.112:0/2737676352 learned_addr learned my addr 172.21.2.112:0/2737676352 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:38:58.788 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:58.779 7f600ffff700 1 --2- 172.21.2.112:0/2737676352 >> v2:172.21.2.112:3300/0 conn(0x7f60100f0650 0x7f60100cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:58.788 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:58.779 7f600f7fe700 1 --2- 172.21.2.112:0/2737676352 >> v2:172.21.2.120:3300/0 conn(0x7f60100d1870 0x7f60101b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:58.788 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:58.779 7f600e7fc700 1 -- 172.21.2.112:0/2737676352 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 529268599 0 0) 0x7f5ff80010b0 con 0x7f60101b0a10 2021-02-10T11:38:58.788 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:58.779 7f600e7fc700 1 -- 172.21.2.112:0/2737676352 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ffc002fe0 con 0x7f60101b0a10 2021-02-10T11:38:58.788 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:58.779 7f600ffff700 1 -- 172.21.2.112:0/2737676352 >> v2:172.21.2.120:3300/0 conn(0x7f60100d1870 msgr2=0x7f60101b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:58.788 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:58.779 7f600ffff700 1 --2- 172.21.2.112:0/2737676352 >> v2:172.21.2.120:3300/0 conn(0x7f60100d1870 0x7f60101b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:58.788 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:58.779 7f600ffff700 1 -- 172.21.2.112:0/2737676352 >> v1:172.21.2.112:6789/0 conn(0x7f60101b0a10 legacy=0x7f60100cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:58.788 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:58.779 7f600ffff700 1 -- 172.21.2.112:0/2737676352 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60100d1e90 con 0x7f60100f0650 2021-02-10T11:38:58.788 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:58.779 7f600ffff700 1 --2- 172.21.2.112:0/2737676352 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f60100f0650 0x7f60100cfb20 secure :-1 s=READY pgs=1851 cs=0 l=1 rev1=1 rx=0x7f6008006e30 tx=0x7f6008009e60).ready entity=mon.0 client_cookie=1f26bdc85450d710 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:58.788 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:58.780 7f600e7fc700 1 -- 172.21.2.112:0/2737676352 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6008006a50 con 0x7f60100f0650 2021-02-10T11:38:58.788 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:58.780 7f600e7fc700 1 -- 172.21.2.112:0/2737676352 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f600800b300 con 0x7f60100f0650 2021-02-10T11:38:58.788 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:58.780 7f600e7fc700 1 -- 172.21.2.112:0/2737676352 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f600800b760 con 0x7f60100f0650 2021-02-10T11:38:58.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:58.780 7f6015c55700 1 -- 172.21.2.112:0/2737676352 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f60100f0650 msgr2=0x7f60100cfb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:58.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:58.780 7f6015c55700 1 --2- 172.21.2.112:0/2737676352 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f60100f0650 0x7f60100cfb20 secure :-1 s=READY pgs=1851 cs=0 l=1 rev1=1 rx=0x7f6008006e30 tx=0x7f6008009e60).stop 2021-02-10T11:38:58.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:58.780 7f6015c55700 1 -- 172.21.2.112:0/2737676352 shutdown_connections 2021-02-10T11:38:58.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:58.780 7f6015c55700 1 --2- 172.21.2.112:0/2737676352 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f60100f0650 0x7f60100cfb20 unknown :-1 s=CLOSED pgs=1851 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:58.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:58.780 7f6015c55700 1 --2- 172.21.2.112:0/2737676352 >> v2:172.21.2.120:3300/0 conn(0x7f60100d1870 0x7f60101b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:58.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:58.780 7f6015c55700 1 -- 172.21.2.112:0/2737676352 >> 172.21.2.112:0/2737676352 conn(0x7f60100ce450 msgr2=0x7f60100d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:58.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:58.780 7f6015c55700 1 -- 172.21.2.112:0/2737676352 shutdown_connections 2021-02-10T11:38:58.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:58.780 7f6015c55700 1 -- 172.21.2.112:0/2737676352 wait complete. 2021-02-10T11:38:58.791 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:58.795 7f6015c55700 1 Processor -- start 2021-02-10T11:38:58.791 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:58.795 7f6015c55700 1 -- start start 2021-02-10T11:38:58.791 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:58.796 7f6015c55700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6010195650 0x7f60100b5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:58.791 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:58.796 7f6015c55700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f60100b5b80 0x7f60101a2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:58.791 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:58.796 7f6015c55700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f60100acfa0 0x7f60101a3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:58.791 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:58.796 7f6015c55700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60100b6010 con 0x7f6010195650 2021-02-10T11:38:58.791 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:58.796 7f6015c55700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60101a3a60 con 0x7f60100b5b80 2021-02-10T11:38:58.791 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:58.796 7f6015c55700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60101a3c40 con 0x7f60100acfa0 2021-02-10T11:38:58.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:58.796 7f600f7fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6010195650 0x7f60100b5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:58.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:58.796 7f600f7fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6010195650 0x7f60100b5560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:53392/0 (socket says 172.21.2.112:53392) 2021-02-10T11:38:58.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:58.796 7f600f7fe700 1 -- 172.21.2.112:0/1470174542 learned_addr learned my addr 172.21.2.112:0/1470174542 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:38:58.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:58.796 7f600ffff700 1 --2- 172.21.2.112:0/1470174542 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f60100acfa0 0x7f60101a3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:58.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:58.796 7f600effd700 1 --2- 172.21.2.112:0/1470174542 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f60100b5b80 0x7f60101a2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:58.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:58.796 7f600f7fe700 1 -- 172.21.2.112:0/1470174542 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f60100acfa0 msgr2=0x7f60101a3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:58.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:58.796 7f600f7fe700 1 --2- 172.21.2.112:0/1470174542 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f60100acfa0 0x7f60101a3440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:58.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:58.796 7f600f7fe700 1 -- 172.21.2.112:0/1470174542 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f60100b5b80 msgr2=0x7f60101a2e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:58.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:58.796 7f600f7fe700 1 --2- 172.21.2.112:0/1470174542 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f60100b5b80 0x7f60101a2e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:58.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:58.796 7f600f7fe700 1 -- 172.21.2.112:0/1470174542 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60100bcae0 con 0x7f6010195650 2021-02-10T11:38:58.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:58.796 7f600f7fe700 1 --2- 172.21.2.112:0/1470174542 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6010195650 0x7f60100b5560 secure :-1 s=READY pgs=1852 cs=0 l=1 rev1=1 rx=0x7f600000b560 tx=0x7f600000b900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:58.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:58.796 7f6007fff700 1 -- 172.21.2.112:0/1470174542 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f60000009b0 con 0x7f6010195650 2021-02-10T11:38:58.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:58.796 7f6007fff700 1 -- 172.21.2.112:0/1470174542 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f600000bd30 con 0x7f6010195650 2021-02-10T11:38:58.793 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:58.796 7f6007fff700 1 -- 172.21.2.112:0/1470174542 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6000002030 con 0x7f6010195650 2021-02-10T11:38:58.793 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:58.796 7f6015c55700 1 -- 172.21.2.112:0/1470174542 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60100bcae0 con 0x7f6010195650 2021-02-10T11:38:58.793 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:58.797 7f6015c55700 1 -- 172.21.2.112:0/1470174542 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60100bcae0 con 0x7f6010195650 2021-02-10T11:38:58.793 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:58.797 7f6007fff700 1 -- 172.21.2.112:0/1470174542 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f60000052e0 con 0x7f6010195650 2021-02-10T11:38:58.793 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:58.797 7f6007fff700 1 --2- 172.21.2.112:0/1470174542 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5ff40186c0 0x7f5ff401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:58.793 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:58.797 7f6007fff700 1 -- 172.21.2.112:0/1470174542 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f60000323f0 con 0x7f6010195650 2021-02-10T11:38:58.793 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:58.797 7f600effd700 1 --2- 172.21.2.112:0/1470174542 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5ff40186c0 0x7f5ff401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:58.793 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:58.798 7f600effd700 1 --2- 172.21.2.112:0/1470174542 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5ff40186c0 0x7f5ff401abc0 crc :-1 s=READY pgs=873 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:58.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:58.828 7f6015c55700 1 -- 172.21.2.112:0/1470174542 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60100b6010 con 0x7f6010195650 2021-02-10T11:38:58.823 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:58.831 7f6007fff700 1 -- 172.21.2.112:0/1470174542 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f600002ded0 con 0x7f6010195650 2021-02-10T11:38:58.951 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:58.959 7f6015c55700 1 -- 172.21.2.112:0/1470174542 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f60100b6010 con 0x7f6010195650 2021-02-10T11:38:58.952 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:58.960 7f6007fff700 1 -- 172.21.2.112:0/1470174542 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) v1 ==== 74+0+13996 (secure 0 0 0) 0x7f600002d820 con 0x7f6010195650 2021-02-10T11:38:58.952 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:38:58.953 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":92,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:38:52.412732","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:38:49.384013","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":19,"pool_name":"cephfs_data","create_time":"2021-02-10 11:38:50.454005","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:38:58.954 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:58.961 7f6015c55700 1 -- 172.21.2.112:0/1470174542 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5ff40186c0 msgr2=0x7f5ff401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:58.954 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:58.961 7f6015c55700 1 --2- 172.21.2.112:0/1470174542 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5ff40186c0 0x7f5ff401abc0 crc :-1 s=READY pgs=873 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:58.954 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:58.961 7f6015c55700 1 -- 172.21.2.112:0/1470174542 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6010195650 msgr2=0x7f60100b5560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:58.954 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:58.961 7f6015c55700 1 --2- 172.21.2.112:0/1470174542 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6010195650 0x7f60100b5560 secure :-1 s=READY pgs=1852 cs=0 l=1 rev1=1 rx=0x7f600000b560 tx=0x7f600000b900).stop 2021-02-10T11:38:58.954 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:58.961 7f6015c55700 1 -- 172.21.2.112:0/1470174542 shutdown_connections 2021-02-10T11:38:58.954 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:58.961 7f6015c55700 1 --2- 172.21.2.112:0/1470174542 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5ff40186c0 0x7f5ff401abc0 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:58.954 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:58.961 7f6015c55700 1 --2- 172.21.2.112:0/1470174542 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f60100b5b80 0x7f60101a2e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:58.954 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:58.961 7f6015c55700 1 --2- 172.21.2.112:0/1470174542 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f60100acfa0 0x7f60101a3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:58.954 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:58.961 7f6015c55700 1 --2- 172.21.2.112:0/1470174542 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6010195650 0x7f60100b5560 unknown :-1 s=CLOSED pgs=1852 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:58.954 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:58.961 7f6015c55700 1 -- 172.21.2.112:0/1470174542 >> 172.21.2.112:0/1470174542 conn(0x7f60100b3040 msgr2=0x7f60100c3250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:58.955 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:58.962 7f6015c55700 1 -- 172.21.2.112:0/1470174542 shutdown_connections 2021-02-10T11:38:58.955 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:58.962 7f6015c55700 1 -- 172.21.2.112:0/1470174542 wait complete. 2021-02-10T11:38:58.963 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:13:57.508758', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 18, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:38:49.384013', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '91', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 19, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:38:50.454005', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '92', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:38:58.963 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:38:59.395 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:59.385 7f92c8077700 1 Processor -- start 2021-02-10T11:38:59.398 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:59.385 7f92c8077700 1 -- start start 2021-02-10T11:38:59.398 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:59.385 7f92c8077700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f92c00d1870 0x7f92c01b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:59.398 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:59.385 7f92c8077700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f92c01b0ae0 0x7f92c0108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:59.399 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:59.385 7f92c8077700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92c00fe1c0 con 0x7f92c00d1c70 2021-02-10T11:38:59.399 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:59.385 7f92c8077700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92c0109a60 con 0x7f92c00d1870 2021-02-10T11:38:59.399 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:59.385 7f92c8077700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92c0109750 con 0x7f92c01b0ae0 2021-02-10T11:38:59.399 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:59.386 7f92c5e93700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f92c00d1870 0x7f92c01b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:59.399 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:59.386 7f92c6694700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f92c00d1c70 0x7f92c0108f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:52860/0 (socket says 172.21.2.112:52860) 2021-02-10T11:38:59.399 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:59.386 7f92c6694700 1 -- 172.21.2.112:0/3742385091 learned_addr learned my addr 172.21.2.112:0/3742385091 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:38:59.399 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:59.386 7f92c5692700 1 --2- 172.21.2.112:0/3742385091 >> v2:172.21.2.120:3300/0 conn(0x7f92c01b0ae0 0x7f92c0108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:59.399 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:59.386 7f92c5e93700 1 -- 172.21.2.112:0/3742385091 >> v2:172.21.2.120:3300/0 conn(0x7f92c01b0ae0 msgr2=0x7f92c0108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:59.399 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:59.386 7f92c5e93700 1 --2- 172.21.2.112:0/3742385091 >> v2:172.21.2.120:3300/0 conn(0x7f92c01b0ae0 0x7f92c0108940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:59.399 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:59.386 7f92c5e93700 1 -- 172.21.2.112:0/3742385091 >> v1:172.21.2.112:6789/0 conn(0x7f92c00d1c70 legacy=0x7f92c0108f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:59.399 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:59.386 7f92c5e93700 1 -- 172.21.2.112:0/3742385091 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92c0088070 con 0x7f92c00d1870 2021-02-10T11:38:59.400 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:59.386 7f92c5e93700 1 --2- 172.21.2.112:0/3742385091 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f92c00d1870 0x7f92c01b04c0 secure :-1 s=READY pgs=1854 cs=0 l=1 rev1=1 rx=0x7f92bc006650 tx=0x7f92bc009a80).ready entity=mon.0 client_cookie=681deb6c682045e8 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:59.400 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:59.387 7f92c4e91700 1 -- 172.21.2.112:0/3742385091 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f92bc0009b0 con 0x7f92c00d1870 2021-02-10T11:38:59.400 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:59.387 7f92c4e91700 1 -- 172.21.2.112:0/3742385091 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92bc00c0c0 con 0x7f92c00d1870 2021-02-10T11:38:59.400 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:59.387 7f92c4e91700 1 -- 172.21.2.112:0/3742385091 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f92bc00c5e0 con 0x7f92c00d1870 2021-02-10T11:38:59.400 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:59.387 7f92c8077700 1 -- 172.21.2.112:0/3742385091 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f92c00d1870 msgr2=0x7f92c01b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:59.400 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:59.387 7f92c8077700 1 --2- 172.21.2.112:0/3742385091 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f92c00d1870 0x7f92c01b04c0 secure :-1 s=READY pgs=1854 cs=0 l=1 rev1=1 rx=0x7f92bc006650 tx=0x7f92bc009a80).stop 2021-02-10T11:38:59.400 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:59.388 7f92c8077700 1 -- 172.21.2.112:0/3742385091 shutdown_connections 2021-02-10T11:38:59.400 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:59.388 7f92c8077700 1 --2- 172.21.2.112:0/3742385091 >> v2:172.21.2.120:3300/0 conn(0x7f92c01b0ae0 0x7f92c0108940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:59.400 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:59.388 7f92c8077700 1 --2- 172.21.2.112:0/3742385091 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f92c00d1870 0x7f92c01b04c0 unknown :-1 s=CLOSED pgs=1854 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:59.400 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:59.388 7f92c8077700 1 -- 172.21.2.112:0/3742385091 >> 172.21.2.112:0/3742385091 conn(0x7f92c00ce420 msgr2=0x7f92c00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:59.400 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:59.388 7f92c8077700 1 -- 172.21.2.112:0/3742385091 shutdown_connections 2021-02-10T11:38:59.400 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:59.388 7f92c8077700 1 -- 172.21.2.112:0/3742385091 wait complete. 2021-02-10T11:38:59.400 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:59.403 7f92c8077700 1 Processor -- start 2021-02-10T11:38:59.401 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:59.403 7f92c8077700 1 -- start start 2021-02-10T11:38:59.402 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:59.403 7f92c8077700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f92c0191b10 0x7f92c00ae640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:59.402 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:59.403 7f92c8077700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f92c0191f10 0x7f92c019ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:59.402 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:59.403 7f92c8077700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f92c01a05f0 0x7f92c0082ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:59.402 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:59.403 7f92c8077700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92c0192440 con 0x7f92c0191b10 2021-02-10T11:38:59.402 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:59.403 7f92c8077700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92c01916e0 con 0x7f92c0191f10 2021-02-10T11:38:59.403 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:59.403 7f92c8077700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92c0191900 con 0x7f92c01a05f0 2021-02-10T11:38:59.403 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:59.403 7f92c5e93700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f92c0191b10 0x7f92c00ae640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:59.403 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:59.403 7f92c5e93700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f92c0191b10 0x7f92c00ae640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:53404/0 (socket says 172.21.2.112:53404) 2021-02-10T11:38:59.403 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:59.403 7f92c5e93700 1 -- 172.21.2.112:0/4095016219 learned_addr learned my addr 172.21.2.112:0/4095016219 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:38:59.403 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:59.403 7f92c6694700 1 --2- 172.21.2.112:0/4095016219 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f92c01a05f0 0x7f92c0082ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:59.403 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:59.403 7f92c5692700 1 --2- 172.21.2.112:0/4095016219 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f92c0191f10 0x7f92c019ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:59.403 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:59.404 7f92c5e93700 1 -- 172.21.2.112:0/4095016219 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f92c01a05f0 msgr2=0x7f92c0082ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:59.403 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:59.404 7f92c5e93700 1 --2- 172.21.2.112:0/4095016219 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f92c01a05f0 0x7f92c0082ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:59.403 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:59.404 7f92c5e93700 1 -- 172.21.2.112:0/4095016219 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f92c0191f10 msgr2=0x7f92c019ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:59.403 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:59.404 7f92c5e93700 1 --2- 172.21.2.112:0/4095016219 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f92c0191f10 0x7f92c019ffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:59.403 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:59.404 7f92c5e93700 1 -- 172.21.2.112:0/4095016219 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92c00834f0 con 0x7f92c0191b10 2021-02-10T11:38:59.403 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:59.404 7f92c5e93700 1 --2- 172.21.2.112:0/4095016219 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f92c0191b10 0x7f92c00ae640 secure :-1 s=READY pgs=1855 cs=0 l=1 rev1=1 rx=0x7f92bc009630 tx=0x7f92bc00b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:59.403 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:59.404 7f92b27fc700 1 -- 172.21.2.112:0/4095016219 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f92bc0093a0 con 0x7f92c0191b10 2021-02-10T11:38:59.404 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:59.404 7f92b27fc700 1 -- 172.21.2.112:0/4095016219 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92bc00bd70 con 0x7f92c0191b10 2021-02-10T11:38:59.404 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:59.404 7f92b27fc700 1 -- 172.21.2.112:0/4095016219 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f92bc0054d0 con 0x7f92c0191b10 2021-02-10T11:38:59.404 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:59.404 7f92c8077700 1 -- 172.21.2.112:0/4095016219 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92c00834f0 con 0x7f92c0191b10 2021-02-10T11:38:59.404 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:59.404 7f92c8077700 1 -- 172.21.2.112:0/4095016219 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92c00834f0 con 0x7f92c0191b10 2021-02-10T11:38:59.404 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:59.405 7f92b27fc700 1 -- 172.21.2.112:0/4095016219 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f92bc000c90 con 0x7f92c0191b10 2021-02-10T11:38:59.404 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:59.405 7f92b27fc700 1 --2- 172.21.2.112:0/4095016219 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f92a80186c0 0x7f92a801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:59.404 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:59.405 7f92b27fc700 1 -- 172.21.2.112:0/4095016219 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f92bc032560 con 0x7f92c0191b10 2021-02-10T11:38:59.404 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:59.405 7f92c5692700 1 --2- 172.21.2.112:0/4095016219 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f92a80186c0 0x7f92a801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:59.404 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:59.405 7f92c5692700 1 --2- 172.21.2.112:0/4095016219 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f92a80186c0 0x7f92a801abc0 crc :-1 s=READY pgs=874 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:59.427 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:59.435 7f92c8077700 1 -- 172.21.2.112:0/4095016219 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92c00fe1c0 con 0x7f92c0191b10 2021-02-10T11:38:59.430 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:59.438 7f92b27fc700 1 -- 172.21.2.112:0/4095016219 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f92bc018fc0 con 0x7f92c0191b10 2021-02-10T11:38:59.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:59.570 7f92c8077700 1 -- 172.21.2.112:0/4095016219 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f92c00fe1c0 con 0x7f92c0191b10 2021-02-10T11:38:59.563 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:59.570 7f92b27fc700 1 -- 172.21.2.112:0/4095016219 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v92) v1 ==== 92+0+10 (secure 0 0 0) 0x7f92bc018fc0 con 0x7f92c0191b10 2021-02-10T11:38:59.563 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:38:59.565 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:59.572 7f92c8077700 1 -- 172.21.2.112:0/4095016219 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f92a80186c0 msgr2=0x7f92a801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:59.565 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:59.572 7f92c8077700 1 --2- 172.21.2.112:0/4095016219 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f92a80186c0 0x7f92a801abc0 crc :-1 s=READY pgs=874 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:59.565 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:59.572 7f92c8077700 1 -- 172.21.2.112:0/4095016219 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f92c0191b10 msgr2=0x7f92c00ae640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:59.565 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:59.572 7f92c8077700 1 --2- 172.21.2.112:0/4095016219 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f92c0191b10 0x7f92c00ae640 secure :-1 s=READY pgs=1855 cs=0 l=1 rev1=1 rx=0x7f92bc009630 tx=0x7f92bc00b980).stop 2021-02-10T11:38:59.565 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:59.572 7f92c8077700 1 -- 172.21.2.112:0/4095016219 shutdown_connections 2021-02-10T11:38:59.565 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:59.572 7f92c8077700 1 --2- 172.21.2.112:0/4095016219 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f92a80186c0 0x7f92a801abc0 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:59.565 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:59.572 7f92c8077700 1 --2- 172.21.2.112:0/4095016219 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f92c0191f10 0x7f92c019ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:59.565 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:59.572 7f92c8077700 1 --2- 172.21.2.112:0/4095016219 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f92c01a05f0 0x7f92c0082ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:59.565 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:59.572 7f92c8077700 1 --2- 172.21.2.112:0/4095016219 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f92c0191b10 0x7f92c00ae640 unknown :-1 s=CLOSED pgs=1855 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:59.566 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:59.572 7f92c8077700 1 -- 172.21.2.112:0/4095016219 >> 172.21.2.112:0/4095016219 conn(0x7f92c00a02d0 msgr2=0x7f92c009f200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:59.566 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:59.573 7f92c8077700 1 -- 172.21.2.112:0/4095016219 shutdown_connections 2021-02-10T11:38:59.566 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:59.573 7f92c8077700 1 -- 172.21.2.112:0/4095016219 wait complete. 2021-02-10T11:38:59.573 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:39:00.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:59.995 7f259df58700 1 Processor -- start 2021-02-10T11:39:00.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:59.995 7f259df58700 1 -- start start 2021-02-10T11:39:00.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:59.995 7f259df58700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f25980d1860 0x7f25981b04b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:00.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:59.995 7f259df58700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f25980cff60 0x7f2598109120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:00.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:59.995 7f259df58700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25980fe1b0 con 0x7f25981b0ad0 2021-02-10T11:39:00.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:59.995 7f259df58700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2598109ac0 con 0x7f25980d1860 2021-02-10T11:39:00.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:59.995 7f259df58700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2598109740 con 0x7f25980cff60 2021-02-10T11:39:00.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:59.996 7f25977fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f25980d1860 0x7f25981b04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:00.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:59.996 7f25977fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f25980d1860 0x7f25981b04b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:53408/0 (socket says 172.21.2.112:53408) 2021-02-10T11:39:00.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:59.996 7f25977fe700 1 -- 172.21.2.112:0/139253877 learned_addr learned my addr 172.21.2.112:0/139253877 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:39:00.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:59.996 7f25977fe700 1 -- 172.21.2.112:0/139253877 >> v2:172.21.2.120:3300/0 conn(0x7f25980cff60 msgr2=0x7f2598109120 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:39:00.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:59.996 7f25977fe700 1 --2- 172.21.2.112:0/139253877 >> v2:172.21.2.120:3300/0 conn(0x7f25980cff60 0x7f2598109120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:00.005 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:59.996 7f25977fe700 1 -- 172.21.2.112:0/139253877 >> v1:172.21.2.120:6789/0 conn(0x7f25981b0ad0 legacy=0x7f2598108930 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:39:00.005 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:59.996 7f25977fe700 1 -- 172.21.2.112:0/139253877 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2598088070 con 0x7f25980d1860 2021-02-10T11:39:00.005 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:59.996 7f25977fe700 1 --2- 172.21.2.112:0/139253877 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f25980d1860 0x7f25981b04b0 secure :-1 s=READY pgs=1856 cs=0 l=1 rev1=1 rx=0x7f2580006700 tx=0x7f2580009b80).ready entity=mon.0 client_cookie=7bca5756d3308b5f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:00.005 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:59.996 7f25967fc700 1 -- 172.21.2.112:0/139253877 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f258000c0c0 con 0x7f25980d1860 2021-02-10T11:39:00.005 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:59.996 7f25967fc700 1 -- 172.21.2.112:0/139253877 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f258000c0c0 con 0x7f25980d1860 2021-02-10T11:39:00.005 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:59.996 7f25967fc700 1 -- 172.21.2.112:0/139253877 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f258000c340 con 0x7f25980d1860 2021-02-10T11:39:00.005 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:59.996 7f259df58700 1 -- 172.21.2.112:0/139253877 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f25980d1860 msgr2=0x7f25981b04b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:00.005 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:59.996 7f259df58700 1 --2- 172.21.2.112:0/139253877 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f25980d1860 0x7f25981b04b0 secure :-1 s=READY pgs=1856 cs=0 l=1 rev1=1 rx=0x7f2580006700 tx=0x7f2580009b80).stop 2021-02-10T11:39:00.005 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:59.997 7f259df58700 1 -- 172.21.2.112:0/139253877 shutdown_connections 2021-02-10T11:39:00.005 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:59.997 7f259df58700 1 --2- 172.21.2.112:0/139253877 >> v2:172.21.2.120:3300/0 conn(0x7f25980cff60 0x7f2598109120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:00.005 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:59.997 7f259df58700 1 --2- 172.21.2.112:0/139253877 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f25980d1860 0x7f25981b04b0 unknown :-1 s=CLOSED pgs=1856 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:00.005 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:59.997 7f259df58700 1 -- 172.21.2.112:0/139253877 >> 172.21.2.112:0/139253877 conn(0x7f25980ce410 msgr2=0x7f25980d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:00.005 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:59.997 7f259df58700 1 -- 172.21.2.112:0/139253877 shutdown_connections 2021-02-10T11:39:00.005 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:38:59.997 7f259df58700 1 -- 172.21.2.112:0/139253877 wait complete. 2021-02-10T11:39:00.007 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.012 7f259df58700 1 Processor -- start 2021-02-10T11:39:00.007 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.012 7f259df58700 1 -- start start 2021-02-10T11:39:00.007 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.012 7f259df58700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2598199ed0 0x7f25981a7c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:00.007 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.012 7f259df58700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f25981a8260 0x7f25980c1ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:00.007 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.012 7f259df58700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f25980c22c0 0x7f25980c26c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:00.007 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.012 7f259df58700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25980c17a0 con 0x7f25980c22c0 2021-02-10T11:39:00.008 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.012 7f259df58700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25980b8c70 con 0x7f25981a8260 2021-02-10T11:39:00.008 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.012 7f259df58700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25980c2ce0 con 0x7f2598199ed0 2021-02-10T11:39:00.008 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.012 7f2597fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f25980c22c0 0x7f25980c26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:00.008 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.012 7f2597fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f25980c22c0 0x7f25980c26c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:53418/0 (socket says 172.21.2.112:53418) 2021-02-10T11:39:00.008 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.012 7f2597fff700 1 -- 172.21.2.112:0/1807939016 learned_addr learned my addr 172.21.2.112:0/1807939016 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:39:00.008 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.013 7f2596ffd700 1 --2- 172.21.2.112:0/1807939016 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f25981a8260 0x7f25980c1ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:00.008 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.013 7f2597fff700 1 -- 172.21.2.112:0/1807939016 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2598199ed0 msgr2=0x7f25981a7c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:00.008 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.013 7f2597fff700 1 --2- 172.21.2.112:0/1807939016 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2598199ed0 0x7f25981a7c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:00.008 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.013 7f2597fff700 1 -- 172.21.2.112:0/1807939016 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f25981a8260 msgr2=0x7f25980c1ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:00.008 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.013 7f2597fff700 1 --2- 172.21.2.112:0/1807939016 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f25981a8260 0x7f25980c1ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:00.008 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.013 7f2597fff700 1 -- 172.21.2.112:0/1807939016 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25980b7200 con 0x7f25980c22c0 2021-02-10T11:39:00.008 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.013 7f2596ffd700 1 --2- 172.21.2.112:0/1807939016 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f25981a8260 0x7f25980c1ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:39:00.008 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.013 7f2597fff700 1 --2- 172.21.2.112:0/1807939016 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f25980c22c0 0x7f25980c26c0 secure :-1 s=READY pgs=1857 cs=0 l=1 rev1=1 rx=0x7f2590009960 tx=0x7f259000bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:00.008 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.013 7f258ffff700 1 -- 172.21.2.112:0/1807939016 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2590000d10 con 0x7f25980c22c0 2021-02-10T11:39:00.009 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.013 7f258ffff700 1 -- 172.21.2.112:0/1807939016 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25900030c0 con 0x7f25980c22c0 2021-02-10T11:39:00.009 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.013 7f258ffff700 1 -- 172.21.2.112:0/1807939016 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2590001500 con 0x7f25980c22c0 2021-02-10T11:39:00.009 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.013 7f259df58700 1 -- 172.21.2.112:0/1807939016 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25980b7200 con 0x7f25980c22c0 2021-02-10T11:39:00.009 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.013 7f259df58700 1 -- 172.21.2.112:0/1807939016 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25980b7790 con 0x7f25980c22c0 2021-02-10T11:39:00.009 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.014 7f258ffff700 1 -- 172.21.2.112:0/1807939016 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2590001310 con 0x7f25980c22c0 2021-02-10T11:39:00.009 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.014 7f258ffff700 1 --2- 172.21.2.112:0/1807939016 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f257c0186c0 0x7f257c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:00.009 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.014 7f258ffff700 1 -- 172.21.2.112:0/1807939016 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f2590032930 con 0x7f25980c22c0 2021-02-10T11:39:00.009 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.014 7f25977fe700 1 --2- 172.21.2.112:0/1807939016 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f257c0186c0 0x7f257c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:00.009 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.014 7f25977fe700 1 --2- 172.21.2.112:0/1807939016 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f257c0186c0 0x7f257c01abc0 crc :-1 s=READY pgs=875 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:00.037 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.045 7f259df58700 1 -- 172.21.2.112:0/1807939016 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25980c1720 con 0x7f25980c22c0 2021-02-10T11:39:00.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.048 7f258ffff700 1 -- 172.21.2.112:0/1807939016 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2590022bd0 con 0x7f25980c22c0 2021-02-10T11:39:00.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.175 7f259df58700 1 -- 172.21.2.112:0/1807939016 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f25980c1720 con 0x7f25980c22c0 2021-02-10T11:39:00.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.175 7f258ffff700 1 -- 172.21.2.112:0/1807939016 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v92) v1 ==== 104+0+10 (secure 0 0 0) 0x7f2590022bd0 con 0x7f25980c22c0 2021-02-10T11:39:00.169 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:39:00.170 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.177 7f259df58700 1 -- 172.21.2.112:0/1807939016 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f257c0186c0 msgr2=0x7f257c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:00.170 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.177 7f259df58700 1 --2- 172.21.2.112:0/1807939016 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f257c0186c0 0x7f257c01abc0 crc :-1 s=READY pgs=875 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:00.170 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.177 7f259df58700 1 -- 172.21.2.112:0/1807939016 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f25980c22c0 msgr2=0x7f25980c26c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:00.170 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.177 7f259df58700 1 --2- 172.21.2.112:0/1807939016 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f25980c22c0 0x7f25980c26c0 secure :-1 s=READY pgs=1857 cs=0 l=1 rev1=1 rx=0x7f2590009960 tx=0x7f259000bcf0).stop 2021-02-10T11:39:00.170 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.177 7f259df58700 1 -- 172.21.2.112:0/1807939016 shutdown_connections 2021-02-10T11:39:00.170 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.177 7f259df58700 1 --2- 172.21.2.112:0/1807939016 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f257c0186c0 0x7f257c01abc0 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:00.170 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.177 7f259df58700 1 --2- 172.21.2.112:0/1807939016 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f25981a8260 0x7f25980c1ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:00.170 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.177 7f259df58700 1 --2- 172.21.2.112:0/1807939016 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f25980c22c0 0x7f25980c26c0 unknown :-1 s=CLOSED pgs=1857 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:00.170 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.177 7f259df58700 1 --2- 172.21.2.112:0/1807939016 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2598199ed0 0x7f25981a7c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:00.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.177 7f259df58700 1 -- 172.21.2.112:0/1807939016 >> 172.21.2.112:0/1807939016 conn(0x7f259809c130 msgr2=0x7f2598095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:00.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.178 7f259df58700 1 -- 172.21.2.112:0/1807939016 shutdown_connections 2021-02-10T11:39:00.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.178 7f259df58700 1 -- 172.21.2.112:0/1807939016 wait complete. 2021-02-10T11:39:00.179 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:39:00.610 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.600 7f862f554700 1 Processor -- start 2021-02-10T11:39:00.610 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.601 7f862f554700 1 -- start start 2021-02-10T11:39:00.610 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.601 7f862f554700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f86280d1870 0x7f86281b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:00.610 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.601 7f862f554700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f86281b0ae0 0x7f8628108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:00.610 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.601 7f862f554700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f86280d1c70 0x7f8628108f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:00.610 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.601 7f862f554700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86280fe1c0 con 0x7f86281b0ae0 2021-02-10T11:39:00.610 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.601 7f862f554700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8628109a60 con 0x7f86280d1870 2021-02-10T11:39:00.610 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.601 7f862f554700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8628088070 con 0x7f86280d1c70 2021-02-10T11:39:00.610 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.601 7f862cb6f700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f86281b0ae0 0x7f8628108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:00.610 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.601 7f862db71700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f86280d1c70 0x7f8628108f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:00.611 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.601 7f862db71700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f86280d1c70 0x7f8628108f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3301/0 says I am v2:172.21.2.112:53104/0 (socket says 172.21.2.112:53104) 2021-02-10T11:39:00.611 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.601 7f862cb6f700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f86281b0ae0 0x7f8628108940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:53422/0 (socket says 172.21.2.112:53422) 2021-02-10T11:39:00.611 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.601 7f862db71700 1 -- 172.21.2.112:0/2397902391 learned_addr learned my addr 172.21.2.112:0/2397902391 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:39:00.611 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.601 7f862d370700 1 --2- 172.21.2.112:0/2397902391 >> v2:172.21.2.120:3300/0 conn(0x7f86280d1870 0x7f86281b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:00.611 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.602 7f862db71700 1 -- 172.21.2.112:0/2397902391 >> v2:172.21.2.120:3300/0 conn(0x7f86280d1870 msgr2=0x7f86281b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:00.611 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.602 7f862db71700 1 --2- 172.21.2.112:0/2397902391 >> v2:172.21.2.120:3300/0 conn(0x7f86280d1870 0x7f86281b04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:00.611 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.602 7f862db71700 1 -- 172.21.2.112:0/2397902391 >> v2:172.21.2.112:3300/0 conn(0x7f86281b0ae0 msgr2=0x7f8628108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:00.611 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.602 7f862db71700 1 --2- 172.21.2.112:0/2397902391 >> v2:172.21.2.112:3300/0 conn(0x7f86281b0ae0 0x7f8628108940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:00.611 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.602 7f862db71700 1 -- 172.21.2.112:0/2397902391 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8628109580 con 0x7f86280d1c70 2021-02-10T11:39:00.611 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.602 7f862d370700 1 --2- 172.21.2.112:0/2397902391 >> v2:172.21.2.120:3300/0 conn(0x7f86280d1870 0x7f86281b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:39:00.611 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.602 7f862cb6f700 1 --2- 172.21.2.112:0/2397902391 >> v2:172.21.2.112:3300/0 conn(0x7f86281b0ae0 0x7f8628108940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:39:00.611 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.602 7f862db71700 1 --2- 172.21.2.112:0/2397902391 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f86280d1c70 0x7f8628108f60 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f8620006aa0 tx=0x7f8620009f20).ready entity=mon.2 client_cookie=df58cf9501e8b5a8 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:00.612 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.602 7f861bfff700 1 -- 172.21.2.112:0/2397902391 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f86200066f0 con 0x7f86280d1c70 2021-02-10T11:39:00.612 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.602 7f861bfff700 1 -- 172.21.2.112:0/2397902391 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8620003030 con 0x7f86280d1c70 2021-02-10T11:39:00.612 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.602 7f861bfff700 1 -- 172.21.2.112:0/2397902391 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f862000b7b0 con 0x7f86280d1c70 2021-02-10T11:39:00.612 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.602 7f862f554700 1 -- 172.21.2.112:0/2397902391 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f86280d1c70 msgr2=0x7f8628108f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:00.612 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.602 7f862f554700 1 --2- 172.21.2.112:0/2397902391 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f86280d1c70 0x7f8628108f60 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f8620006aa0 tx=0x7f8620009f20).stop 2021-02-10T11:39:00.612 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.603 7f862f554700 1 -- 172.21.2.112:0/2397902391 shutdown_connections 2021-02-10T11:39:00.612 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.603 7f862f554700 1 --2- 172.21.2.112:0/2397902391 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f86280d1c70 0x7f8628108f60 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:00.612 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.603 7f862f554700 1 --2- 172.21.2.112:0/2397902391 >> v2:172.21.2.112:3300/0 conn(0x7f86281b0ae0 0x7f8628108940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:00.612 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.603 7f862f554700 1 --2- 172.21.2.112:0/2397902391 >> v2:172.21.2.120:3300/0 conn(0x7f86280d1870 0x7f86281b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:00.612 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.603 7f862f554700 1 -- 172.21.2.112:0/2397902391 >> 172.21.2.112:0/2397902391 conn(0x7f86280ce420 msgr2=0x7f86280d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:00.612 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.603 7f862f554700 1 -- 172.21.2.112:0/2397902391 shutdown_connections 2021-02-10T11:39:00.612 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.603 7f862f554700 1 -- 172.21.2.112:0/2397902391 wait complete. 2021-02-10T11:39:00.612 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.617 7f862f554700 1 Processor -- start 2021-02-10T11:39:00.612 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.617 7f862f554700 1 -- start start 2021-02-10T11:39:00.614 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.618 7f862f554700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f86281a79a0 0x7f86281a7da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:00.614 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.618 7f862f554700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f86280bdb40 0x7f86280bdf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:00.614 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.618 7f862f554700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8628082ca0 0x7f86280830a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:00.614 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.618 7f862f554700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f862819d990 con 0x7f86280bdb40 2021-02-10T11:39:00.614 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.618 7f862f554700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f862819dbb0 con 0x7f86281a79a0 2021-02-10T11:39:00.614 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.618 7f862f554700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86280836c0 con 0x7f8628082ca0 2021-02-10T11:39:00.614 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.618 7f862cb6f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f86280bdb40 0x7f86280bdf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:00.614 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.618 7f862cb6f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f86280bdb40 0x7f86280bdf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:53430/0 (socket says 172.21.2.112:53430) 2021-02-10T11:39:00.614 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.618 7f862cb6f700 1 -- 172.21.2.112:0/3075490797 learned_addr learned my addr 172.21.2.112:0/3075490797 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:39:00.614 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.618 7f862cb6f700 1 -- 172.21.2.112:0/3075490797 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8628082ca0 msgr2=0x7f86280830a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:39:00.615 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.618 7f862cb6f700 1 --2- 172.21.2.112:0/3075490797 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8628082ca0 0x7f86280830a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:00.615 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.618 7f862cb6f700 1 -- 172.21.2.112:0/3075490797 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f86281a79a0 msgr2=0x7f86281a7da0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:39:00.615 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.618 7f862cb6f700 1 --2- 172.21.2.112:0/3075490797 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f86281a79a0 0x7f86281a7da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:00.615 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.618 7f862cb6f700 1 -- 172.21.2.112:0/3075490797 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86280838e0 con 0x7f86280bdb40 2021-02-10T11:39:00.615 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.618 7f862cb6f700 1 --2- 172.21.2.112:0/3075490797 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f86280bdb40 0x7f86280bdf40 secure :-1 s=READY pgs=1858 cs=0 l=1 rev1=1 rx=0x7f8624006f40 tx=0x7f862400b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:00.615 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.619 7f8619ffb700 1 -- 172.21.2.112:0/3075490797 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f862400bd80 con 0x7f86280bdb40 2021-02-10T11:39:00.615 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.619 7f8619ffb700 1 -- 172.21.2.112:0/3075490797 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8624000c90 con 0x7f86280bdb40 2021-02-10T11:39:00.615 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.619 7f8619ffb700 1 -- 172.21.2.112:0/3075490797 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f86240022b0 con 0x7f86280bdb40 2021-02-10T11:39:00.615 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.619 7f862f554700 1 -- 172.21.2.112:0/3075490797 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86280838e0 con 0x7f86280bdb40 2021-02-10T11:39:00.615 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.619 7f862f554700 1 -- 172.21.2.112:0/3075490797 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8628084420 con 0x7f86280bdb40 2021-02-10T11:39:00.615 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.619 7f8619ffb700 1 -- 172.21.2.112:0/3075490797 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f862400a000 con 0x7f86280bdb40 2021-02-10T11:39:00.615 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.619 7f8619ffb700 1 --2- 172.21.2.112:0/3075490797 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f86080186c0 0x7f860801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:00.615 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.619 7f8619ffb700 1 -- 172.21.2.112:0/3075490797 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f8624032590 con 0x7f86280bdb40 2021-02-10T11:39:00.616 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.620 7f862d370700 1 --2- 172.21.2.112:0/3075490797 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f86080186c0 0x7f860801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:00.616 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.620 7f862d370700 1 --2- 172.21.2.112:0/3075490797 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f86080186c0 0x7f860801abc0 crc :-1 s=READY pgs=876 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:00.643 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.651 7f862f554700 1 -- 172.21.2.112:0/3075490797 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8628082a70 con 0x7f86280bdb40 2021-02-10T11:39:00.646 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.653 7f8619ffb700 1 -- 172.21.2.112:0/3075490797 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8624000990 con 0x7f86280bdb40 2021-02-10T11:39:00.774 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.782 7f862f554700 1 -- 172.21.2.112:0/3075490797 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f8628082a70 con 0x7f86280bdb40 2021-02-10T11:39:00.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.782 7f8619ffb700 1 -- 172.21.2.112:0/3075490797 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v92) v1 ==== 100+0+10 (secure 0 0 0) 0x7f862400bd80 con 0x7f86280bdb40 2021-02-10T11:39:00.775 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:39:00.777 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.784 7f862f554700 1 -- 172.21.2.112:0/3075490797 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f86080186c0 msgr2=0x7f860801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:00.777 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.784 7f862f554700 1 --2- 172.21.2.112:0/3075490797 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f86080186c0 0x7f860801abc0 crc :-1 s=READY pgs=876 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:00.777 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.784 7f862f554700 1 -- 172.21.2.112:0/3075490797 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f86280bdb40 msgr2=0x7f86280bdf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:00.777 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.784 7f862f554700 1 --2- 172.21.2.112:0/3075490797 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f86280bdb40 0x7f86280bdf40 secure :-1 s=READY pgs=1858 cs=0 l=1 rev1=1 rx=0x7f8624006f40 tx=0x7f862400b990).stop 2021-02-10T11:39:00.777 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.785 7f862f554700 1 -- 172.21.2.112:0/3075490797 shutdown_connections 2021-02-10T11:39:00.778 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.785 7f862f554700 1 --2- 172.21.2.112:0/3075490797 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f86080186c0 0x7f860801abc0 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:00.778 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.785 7f862f554700 1 --2- 172.21.2.112:0/3075490797 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8628082ca0 0x7f86280830a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:00.778 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.785 7f862f554700 1 --2- 172.21.2.112:0/3075490797 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f86280bdb40 0x7f86280bdf40 unknown :-1 s=CLOSED pgs=1858 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:00.778 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.785 7f862f554700 1 --2- 172.21.2.112:0/3075490797 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f86281a79a0 0x7f86281a7da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:00.778 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.785 7f862f554700 1 -- 172.21.2.112:0/3075490797 >> 172.21.2.112:0/3075490797 conn(0x7f86280bef00 msgr2=0x7f862809f200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:00.778 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.785 7f862f554700 1 -- 172.21.2.112:0/3075490797 shutdown_connections 2021-02-10T11:39:00.778 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:00.785 7f862f554700 1 -- 172.21.2.112:0/3075490797 wait complete. 2021-02-10T11:39:00.785 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:39:00.785 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:39:01.213 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:01.204 7f35142e9700 1 Processor -- start 2021-02-10T11:39:01.214 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:01.204 7f35142e9700 1 -- start start 2021-02-10T11:39:01.214 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:01.205 7f35142e9700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f350c1b0470 0x7f350c1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:01.214 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:01.205 7f35142e9700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f350c0ef610 0x7f350c0efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:01.214 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:01.205 7f35142e9700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f350c0f0b50 con 0x7f350c0f0030 2021-02-10T11:39:01.214 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:01.205 7f35142e9700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f350c0e52e0 con 0x7f350c1b0470 2021-02-10T11:39:01.214 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:01.205 7f35142e9700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f350c0e54d0 con 0x7f350c0ef610 2021-02-10T11:39:01.214 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:01.205 7f3512105700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f350c1b0470 0x7f350c1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:01.214 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:01.205 7f3512105700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f350c1b0470 0x7f350c1b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:53436/0 (socket says 172.21.2.112:53436) 2021-02-10T11:39:01.214 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:01.205 7f3512105700 1 -- 172.21.2.112:0/3934366754 learned_addr learned my addr 172.21.2.112:0/3934366754 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:39:01.214 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:01.205 7f3511904700 1 --2- 172.21.2.112:0/3934366754 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f350c0ef610 0x7f350c0efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:01.214 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:01.205 7f3512105700 1 -- 172.21.2.112:0/3934366754 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f350c0ef610 msgr2=0x7f350c0efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:01.214 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:01.205 7f3512105700 1 --2- 172.21.2.112:0/3934366754 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f350c0ef610 0x7f350c0efa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:01.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:01.205 7f3512105700 1 -- 172.21.2.112:0/3934366754 >> v1:172.21.2.120:6789/0 conn(0x7f350c0f0030 legacy=0x7f350c0f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:01.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:01.205 7f3512105700 1 -- 172.21.2.112:0/3934366754 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f350c0f0430 con 0x7f350c1b0470 2021-02-10T11:39:01.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:01.206 7f3512105700 1 --2- 172.21.2.112:0/3934366754 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f350c1b0470 0x7f350c1b0870 secure :-1 s=READY pgs=1859 cs=0 l=1 rev1=1 rx=0x7f3508006700 tx=0x7f3508009b80).ready entity=mon.0 client_cookie=5e3c74b4ad33c295 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:01.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:01.206 7f3511103700 1 -- 172.21.2.112:0/3934366754 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f350800c0c0 con 0x7f350c1b0470 2021-02-10T11:39:01.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:01.206 7f3511103700 1 -- 172.21.2.112:0/3934366754 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f350800c400 con 0x7f350c1b0470 2021-02-10T11:39:01.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:01.206 7f3511103700 1 -- 172.21.2.112:0/3934366754 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f350800c920 con 0x7f350c1b0470 2021-02-10T11:39:01.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:01.206 7f35142e9700 1 -- 172.21.2.112:0/3934366754 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f350c1b0470 msgr2=0x7f350c1b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:01.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:01.206 7f35142e9700 1 --2- 172.21.2.112:0/3934366754 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f350c1b0470 0x7f350c1b0870 secure :-1 s=READY pgs=1859 cs=0 l=1 rev1=1 rx=0x7f3508006700 tx=0x7f3508009b80).stop 2021-02-10T11:39:01.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:01.207 7f35142e9700 1 -- 172.21.2.112:0/3934366754 shutdown_connections 2021-02-10T11:39:01.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:01.207 7f35142e9700 1 --2- 172.21.2.112:0/3934366754 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f350c0ef610 0x7f350c0efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:01.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:01.207 7f35142e9700 1 --2- 172.21.2.112:0/3934366754 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f350c1b0470 0x7f350c1b0870 unknown :-1 s=CLOSED pgs=1859 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:01.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:01.207 7f35142e9700 1 -- 172.21.2.112:0/3934366754 >> 172.21.2.112:0/3934366754 conn(0x7f350c0ce3e0 msgr2=0x7f350c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:01.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:01.207 7f35142e9700 1 -- 172.21.2.112:0/3934366754 shutdown_connections 2021-02-10T11:39:01.216 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:01.207 7f35142e9700 1 -- 172.21.2.112:0/3934366754 wait complete. 2021-02-10T11:39:01.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:01.222 7f35142e9700 1 Processor -- start 2021-02-10T11:39:01.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:01.222 7f35142e9700 1 -- start start 2021-02-10T11:39:01.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:01.222 7f35142e9700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f350c1af8a0 0x7f350c198e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:01.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:01.222 7f35142e9700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f350c1af050 0x7f350c0aa3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:01.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:01.222 7f35142e9700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f350c197d00 0x7f350c198100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:01.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:01.222 7f35142e9700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f350c1988e0 con 0x7f350c1af8a0 2021-02-10T11:39:01.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:01.222 7f35142e9700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f350c1af450 con 0x7f350c197d00 2021-02-10T11:39:01.218 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:01.222 7f35142e9700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f350c0aaa10 con 0x7f350c1af050 2021-02-10T11:39:01.218 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:01.222 7f3512105700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f350c1af8a0 0x7f350c198e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:01.218 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:01.222 7f3512105700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f350c1af8a0 0x7f350c198e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:53440/0 (socket says 172.21.2.112:53440) 2021-02-10T11:39:01.218 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:01.222 7f3512105700 1 -- 172.21.2.112:0/3270147441 learned_addr learned my addr 172.21.2.112:0/3270147441 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:39:01.218 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:01.223 7f3512105700 1 -- 172.21.2.112:0/3270147441 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f350c1af050 msgr2=0x7f350c0aa3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:01.218 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:01.223 7f3512105700 1 --2- 172.21.2.112:0/3270147441 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f350c1af050 0x7f350c0aa3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:01.218 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:01.223 7f3512906700 1 --2- 172.21.2.112:0/3270147441 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f350c197d00 0x7f350c198100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:01.218 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:01.223 7f3512105700 1 -- 172.21.2.112:0/3270147441 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f350c197d00 msgr2=0x7f350c198100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:01.218 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:01.223 7f3512105700 1 --2- 172.21.2.112:0/3270147441 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f350c197d00 0x7f350c198100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:01.218 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:01.223 7f3512105700 1 -- 172.21.2.112:0/3270147441 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f350c19fec0 con 0x7f350c1af8a0 2021-02-10T11:39:01.218 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:01.223 7f3512906700 1 --2- 172.21.2.112:0/3270147441 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f350c197d00 0x7f350c198100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:39:01.218 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:01.223 7f3512105700 1 --2- 172.21.2.112:0/3270147441 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f350c1af8a0 0x7f350c198e60 secure :-1 s=READY pgs=1860 cs=0 l=1 rev1=1 rx=0x7f350800b580 tx=0x7f350800b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:01.219 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:01.223 7f34feffd700 1 -- 172.21.2.112:0/3270147441 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f350800bd10 con 0x7f350c1af8a0 2021-02-10T11:39:01.219 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:01.223 7f34feffd700 1 -- 172.21.2.112:0/3270147441 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35080052e0 con 0x7f350c1af8a0 2021-02-10T11:39:01.219 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:01.223 7f34feffd700 1 -- 172.21.2.112:0/3270147441 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3508005800 con 0x7f350c1af8a0 2021-02-10T11:39:01.219 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:01.223 7f35142e9700 1 -- 172.21.2.112:0/3270147441 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f350c19fec0 con 0x7f350c1af8a0 2021-02-10T11:39:01.219 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:01.223 7f35142e9700 1 -- 172.21.2.112:0/3270147441 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f350c1a0450 con 0x7f350c1af8a0 2021-02-10T11:39:01.219 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:01.224 7f34feffd700 1 -- 172.21.2.112:0/3270147441 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3508006030 con 0x7f350c1af8a0 2021-02-10T11:39:01.219 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:01.224 7f34feffd700 1 --2- 172.21.2.112:0/3270147441 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f34f40186c0 0x7f34f401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:01.219 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:01.224 7f34feffd700 1 -- 172.21.2.112:0/3270147441 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f3508032470 con 0x7f350c1af8a0 2021-02-10T11:39:01.219 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:01.224 7f3511904700 1 --2- 172.21.2.112:0/3270147441 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f34f40186c0 0x7f34f401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:01.219 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:01.225 7f3511904700 1 --2- 172.21.2.112:0/3270147441 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f34f40186c0 0x7f34f401abc0 crc :-1 s=READY pgs=877 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:01.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:01.255 7f35142e9700 1 -- 172.21.2.112:0/3270147441 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f350c105a20 con 0x7f350c1af8a0 2021-02-10T11:39:01.249 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:01.257 7f34feffd700 1 -- 172.21.2.112:0/3270147441 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3508030da0 con 0x7f350c1af8a0 2021-02-10T11:39:01.376 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:01.384 7f35142e9700 1 -- 172.21.2.112:0/3270147441 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f350c0f0b50 con 0x7f350c1af8a0 2021-02-10T11:39:01.377 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:01.384 7f34feffd700 1 -- 172.21.2.112:0/3270147441 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+2101 (secure 0 0 0) 0x7f350800bd10 con 0x7f350c1af8a0 2021-02-10T11:39:01.377 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 72 2021-02-10T11:39:01.377 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:39:01.378 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":72,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9621,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/1174368781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":1174368781},{"type":"v1","addr":"172.21.2.112:6833","nonce":1174368781}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":69}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:38:51.410458","modified":"2021-02-10 11:38:53.445268","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6380},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6380":{"gid":6380,"name":"b","rank":0,"incarnation":70,"state":"up:active","state_seq":3,"addr":"172.21.2.120:6835/2402714407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":2402714407},{"type":"v1","addr":"172.21.2.120:6835","nonce":2402714407}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":9}]} 2021-02-10T11:39:01.379 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:01.386 7f35142e9700 1 -- 172.21.2.112:0/3270147441 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f34f40186c0 msgr2=0x7f34f401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:01.379 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:01.386 7f35142e9700 1 --2- 172.21.2.112:0/3270147441 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f34f40186c0 0x7f34f401abc0 crc :-1 s=READY pgs=877 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:01.379 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:01.386 7f35142e9700 1 -- 172.21.2.112:0/3270147441 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f350c1af8a0 msgr2=0x7f350c198e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:01.379 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:01.386 7f35142e9700 1 --2- 172.21.2.112:0/3270147441 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f350c1af8a0 0x7f350c198e60 secure :-1 s=READY pgs=1860 cs=0 l=1 rev1=1 rx=0x7f350800b580 tx=0x7f350800b920).stop 2021-02-10T11:39:01.379 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:01.386 7f35142e9700 1 -- 172.21.2.112:0/3270147441 shutdown_connections 2021-02-10T11:39:01.379 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:01.386 7f35142e9700 1 --2- 172.21.2.112:0/3270147441 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f34f40186c0 0x7f34f401abc0 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:01.379 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:01.386 7f35142e9700 1 --2- 172.21.2.112:0/3270147441 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f350c197d00 0x7f350c198100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:01.380 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:01.386 7f35142e9700 1 --2- 172.21.2.112:0/3270147441 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f350c1af050 0x7f350c0aa3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:01.380 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:01.387 7f35142e9700 1 --2- 172.21.2.112:0/3270147441 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f350c1af8a0 0x7f350c198e60 unknown :-1 s=CLOSED pgs=1860 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:01.380 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:01.387 7f35142e9700 1 -- 172.21.2.112:0/3270147441 >> 172.21.2.112:0/3270147441 conn(0x7f350c1a37a0 msgr2=0x7f350c18c230 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:01.380 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:01.387 7f35142e9700 1 -- 172.21.2.112:0/3270147441 shutdown_connections 2021-02-10T11:39:01.380 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:01.387 7f35142e9700 1 -- 172.21.2.112:0/3270147441 wait complete. 2021-02-10T11:39:01.589 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 72, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:38:51.410458', 'modified': '2021-02-10 11:38:53.445268', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6380}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6380': {'gid': 6380, 'name': 'b', 'rank': 0, 'incarnation': 70, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.120:6835/2402714407', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.120:6834', 'nonce': 2402714407}, {'type': 'v1', 'addr': '172.21.2.120:6835', 'nonce': 2402714407}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:39:01.589 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:39:01.590 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:39:01.988 INFO:teuthology.orchestra.run.gibba020.stdout:{ 2021-02-10T11:39:01.988 INFO:teuthology.orchestra.run.gibba020.stdout: "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", 2021-02-10T11:39:01.988 INFO:teuthology.orchestra.run.gibba020.stdout: "whoami": 0, 2021-02-10T11:39:01.988 INFO:teuthology.orchestra.run.gibba020.stdout: "id": 6380, 2021-02-10T11:39:01.988 INFO:teuthology.orchestra.run.gibba020.stdout: "want_state": "up:active", 2021-02-10T11:39:01.989 INFO:teuthology.orchestra.run.gibba020.stdout: "state": "up:active", 2021-02-10T11:39:01.989 INFO:teuthology.orchestra.run.gibba020.stdout: "rank_uptime": 10.583036044, 2021-02-10T11:39:01.989 INFO:teuthology.orchestra.run.gibba020.stdout: "mdsmap_epoch": 72, 2021-02-10T11:39:01.989 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch": 92, 2021-02-10T11:39:01.989 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch_barrier": 92, 2021-02-10T11:39:01.989 INFO:teuthology.orchestra.run.gibba020.stdout: "uptime": 16.216071150000001 2021-02-10T11:39:01.989 INFO:teuthology.orchestra.run.gibba020.stdout:} 2021-02-10T11:39:01.993 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", "whoami": 0, "id": 6380, "want_state": "up:active", "state": "up:active", "rank_uptime": 10.583036044, "mdsmap_epoch": 72, "osdmap_epoch": 92, "osdmap_epoch_barrier": 92, "uptime": 16.216071150000001 } 2021-02-10T11:39:01.993 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:39:01.993 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2021-02-10T11:39:01.993 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@gibba012.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2021-02-10T11:39:01.993 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:39:01.993 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:39:02.328 DEBUG:teuthology.orchestra.run.gibba012:> sudo modprobe fuse 2021-02-10T11:39:02.673 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:39:02.673 DEBUG:teuthology.orchestra.run.gibba012:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:39:03.017 INFO:teuthology.orchestra.run.gibba012.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:39:03.017 INFO:teuthology.orchestra.run.gibba012.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:39:03.018 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:39:03.018 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:39:03.019 DEBUG:teuthology.orchestra.run.gibba012:> ls /sys/fs/fuse/connections 2021-02-10T11:39:03.352 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2021-02-10T11:39:03.353 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:39:03.355 DEBUG:teuthology.orchestra.run.gibba012:> sudo modprobe fuse 2021-02-10T11:39:03.700 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:39:03.700 DEBUG:teuthology.orchestra.run.gibba012:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:39:03.765 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba012.stderr:ceph-fuse[149823]: starting ceph client 2021-02-10T11:39:03.765 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba012.stderr:2021-02-10 11:39:03.772 7f774501df80 -1 init, newargv = 0x562a0ed701f0 newargc=9 2021-02-10T11:39:03.771 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba012.stderr:ceph-fuse[149823]: starting fuse 2021-02-10T11:39:04.042 INFO:teuthology.orchestra.run.gibba012.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:39:04.043 INFO:teuthology.orchestra.run.gibba012.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:39:04.043 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:39:04.044 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:39:04.044 DEBUG:teuthology.orchestra.run.gibba012:> ls /sys/fs/fuse/connections 2021-02-10T11:39:04.377 INFO:teuthology.orchestra.run.gibba012.stdout:42 2021-02-10T11:39:04.378 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2021-02-10T11:39:04.378 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:39:04.378 DEBUG:teuthology.orchestra.run.gibba012:> sudo python3 -c ' 2021-02-10T11:39:04.378 DEBUG:teuthology.orchestra.run.gibba012:> import glob 2021-02-10T11:39:04.378 DEBUG:teuthology.orchestra.run.gibba012:> import re 2021-02-10T11:39:04.378 DEBUG:teuthology.orchestra.run.gibba012:> import os 2021-02-10T11:39:04.378 DEBUG:teuthology.orchestra.run.gibba012:> import subprocess 2021-02-10T11:39:04.379 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:39:04.379 DEBUG:teuthology.orchestra.run.gibba012:> def find_socket(client_name): 2021-02-10T11:39:04.379 DEBUG:teuthology.orchestra.run.gibba012:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2021-02-10T11:39:04.379 DEBUG:teuthology.orchestra.run.gibba012:> files = glob.glob(asok_path) 2021-02-10T11:39:04.379 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:39:04.379 DEBUG:teuthology.orchestra.run.gibba012:> # Given a non-glob path, it better be there 2021-02-10T11:39:04.379 DEBUG:teuthology.orchestra.run.gibba012:> if "*" not in asok_path: 2021-02-10T11:39:04.379 DEBUG:teuthology.orchestra.run.gibba012:> assert(len(files) == 1) 2021-02-10T11:39:04.379 DEBUG:teuthology.orchestra.run.gibba012:> return files[0] 2021-02-10T11:39:04.379 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:39:04.379 DEBUG:teuthology.orchestra.run.gibba012:> for f in files: 2021-02-10T11:39:04.379 DEBUG:teuthology.orchestra.run.gibba012:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:39:04.379 DEBUG:teuthology.orchestra.run.gibba012:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:39:04.379 DEBUG:teuthology.orchestra.run.gibba012:> return f 2021-02-10T11:39:04.380 DEBUG:teuthology.orchestra.run.gibba012:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:39:04.380 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:39:04.380 DEBUG:teuthology.orchestra.run.gibba012:> print(find_socket("client.0")) 2021-02-10T11:39:04.380 DEBUG:teuthology.orchestra.run.gibba012:> ' 2021-02-10T11:39:04.740 INFO:teuthology.orchestra.run.gibba012.stdout:/var/run/ceph/ceph-client.0.149823.asok 2021-02-10T11:39:04.743 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.149823.asok 2021-02-10T11:39:04.743 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:39:04.743 DEBUG:teuthology.orchestra.run.gibba012:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.149823.asok status 2021-02-10T11:39:05.132 INFO:teuthology.orchestra.run.gibba012.stdout:{ 2021-02-10T11:39:05.133 INFO:teuthology.orchestra.run.gibba012.stdout: "metadata": { 2021-02-10T11:39:05.133 INFO:teuthology.orchestra.run.gibba012.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:39:05.133 INFO:teuthology.orchestra.run.gibba012.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:39:05.133 INFO:teuthology.orchestra.run.gibba012.stdout: "entity_id": "0", 2021-02-10T11:39:05.133 INFO:teuthology.orchestra.run.gibba012.stdout: "hostname": "gibba012", 2021-02-10T11:39:05.133 INFO:teuthology.orchestra.run.gibba012.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2021-02-10T11:39:05.133 INFO:teuthology.orchestra.run.gibba012.stdout: "pid": "149823", 2021-02-10T11:39:05.133 INFO:teuthology.orchestra.run.gibba012.stdout: "root": "/" 2021-02-10T11:39:05.133 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:39:05.133 INFO:teuthology.orchestra.run.gibba012.stdout: "dentry_count": 0, 2021-02-10T11:39:05.133 INFO:teuthology.orchestra.run.gibba012.stdout: "dentry_pinned_count": 0, 2021-02-10T11:39:05.134 INFO:teuthology.orchestra.run.gibba012.stdout: "id": 9783, 2021-02-10T11:39:05.134 INFO:teuthology.orchestra.run.gibba012.stdout: "inst": { 2021-02-10T11:39:05.134 INFO:teuthology.orchestra.run.gibba012.stdout: "name": { 2021-02-10T11:39:05.134 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "client", 2021-02-10T11:39:05.134 INFO:teuthology.orchestra.run.gibba012.stdout: "num": 9783 2021-02-10T11:39:05.134 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:39:05.134 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": { 2021-02-10T11:39:05.134 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:39:05.134 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:0", 2021-02-10T11:39:05.134 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 4049312459 2021-02-10T11:39:05.134 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:39:05.135 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:39:05.135 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": { 2021-02-10T11:39:05.135 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:39:05.135 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:0", 2021-02-10T11:39:05.135 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 4049312459 2021-02-10T11:39:05.135 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:39:05.135 INFO:teuthology.orchestra.run.gibba012.stdout: "inst_str": "client.9783 172.21.2.112:0/4049312459", 2021-02-10T11:39:05.135 INFO:teuthology.orchestra.run.gibba012.stdout: "addr_str": "172.21.2.112:0/4049312459", 2021-02-10T11:39:05.135 INFO:teuthology.orchestra.run.gibba012.stdout: "inode_count": 1, 2021-02-10T11:39:05.135 INFO:teuthology.orchestra.run.gibba012.stdout: "mds_epoch": 72, 2021-02-10T11:39:05.135 INFO:teuthology.orchestra.run.gibba012.stdout: "osd_epoch": 92, 2021-02-10T11:39:05.135 INFO:teuthology.orchestra.run.gibba012.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:39:05.136 INFO:teuthology.orchestra.run.gibba012.stdout: "blacklisted": false 2021-02-10T11:39:05.136 INFO:teuthology.orchestra.run.gibba012.stdout:} 2021-02-10T11:39:05.137 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:39:05.137 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:39:05.137 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:39:05.473 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:39:05.474 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:39:05.474 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:39:05.474 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:39:05.821 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:39:06.248 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.238 7f94d83d8700 1 Processor -- start 2021-02-10T11:39:06.248 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.239 7f94d83d8700 1 -- start start 2021-02-10T11:39:06.249 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.239 7f94d83d8700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f94d00d1870 0x7f94d01b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:06.249 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.239 7f94d83d8700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f94d00f0650 0x7f94d00cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:06.249 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.239 7f94d83d8700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94d00f0b10 con 0x7f94d01b0a10 2021-02-10T11:39:06.249 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.239 7f94d83d8700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94d00e5ed0 con 0x7f94d00f0650 2021-02-10T11:39:06.249 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.239 7f94d83d8700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94d00d1c70 con 0x7f94d00d1870 2021-02-10T11:39:06.249 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.239 7f94d69f5700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f94d00f0650 0x7f94d00cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:06.249 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.239 7f94d59f3700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f94d01b0a10 0x7f94d00cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:52922/0 (socket says 172.21.2.112:52922) 2021-02-10T11:39:06.249 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.239 7f94d59f3700 1 -- 172.21.2.112:0/3160789533 learned_addr learned my addr 172.21.2.112:0/3160789533 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:39:06.249 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.239 7f94d61f4700 1 --2- 172.21.2.112:0/3160789533 >> v2:172.21.2.120:3300/0 conn(0x7f94d00d1870 0x7f94d01b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:06.249 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.240 7f94d51f2700 1 -- 172.21.2.112:0/3160789533 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1879451809 0 0) 0x7f94c40010b0 con 0x7f94d01b0a10 2021-02-10T11:39:06.249 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.240 7f94d69f5700 1 -- 172.21.2.112:0/3160789533 >> v2:172.21.2.120:3300/0 conn(0x7f94d00d1870 msgr2=0x7f94d01b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:06.249 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.240 7f94d69f5700 1 --2- 172.21.2.112:0/3160789533 >> v2:172.21.2.120:3300/0 conn(0x7f94d00d1870 0x7f94d01b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:06.250 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.240 7f94d69f5700 1 -- 172.21.2.112:0/3160789533 >> v1:172.21.2.112:6789/0 conn(0x7f94d01b0a10 legacy=0x7f94d00cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:06.250 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.240 7f94d69f5700 1 -- 172.21.2.112:0/3160789533 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94d00d1e90 con 0x7f94d00f0650 2021-02-10T11:39:06.250 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.240 7f94d61f4700 1 --2- 172.21.2.112:0/3160789533 >> v2:172.21.2.120:3300/0 conn(0x7f94d00d1870 0x7f94d01b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:39:06.250 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.240 7f94d69f5700 1 --2- 172.21.2.112:0/3160789533 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f94d00f0650 0x7f94d00cfb20 secure :-1 s=READY pgs=1865 cs=0 l=1 rev1=1 rx=0x7f94c8006e30 tx=0x7f94c8009e60).ready entity=mon.0 client_cookie=7efbd76914090b7f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:06.250 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.240 7f94d51f2700 1 -- 172.21.2.112:0/3160789533 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f94c8000f90 con 0x7f94d00f0650 2021-02-10T11:39:06.250 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.240 7f94d51f2700 1 -- 172.21.2.112:0/3160789533 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94c800b300 con 0x7f94d00f0650 2021-02-10T11:39:06.250 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.240 7f94d51f2700 1 -- 172.21.2.112:0/3160789533 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f94c800b820 con 0x7f94d00f0650 2021-02-10T11:39:06.250 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.240 7f94d83d8700 1 -- 172.21.2.112:0/3160789533 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f94d00f0650 msgr2=0x7f94d00cfb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:06.250 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.240 7f94d83d8700 1 --2- 172.21.2.112:0/3160789533 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f94d00f0650 0x7f94d00cfb20 secure :-1 s=READY pgs=1865 cs=0 l=1 rev1=1 rx=0x7f94c8006e30 tx=0x7f94c8009e60).stop 2021-02-10T11:39:06.250 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.241 7f94d83d8700 1 -- 172.21.2.112:0/3160789533 shutdown_connections 2021-02-10T11:39:06.250 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.241 7f94d83d8700 1 --2- 172.21.2.112:0/3160789533 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f94d00f0650 0x7f94d00cfb20 unknown :-1 s=CLOSED pgs=1865 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:06.250 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.241 7f94d83d8700 1 --2- 172.21.2.112:0/3160789533 >> v2:172.21.2.120:3300/0 conn(0x7f94d00d1870 0x7f94d01b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:06.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.241 7f94d83d8700 1 -- 172.21.2.112:0/3160789533 >> 172.21.2.112:0/3160789533 conn(0x7f94d00ce450 msgr2=0x7f94d00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:06.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.241 7f94d83d8700 1 -- 172.21.2.112:0/3160789533 shutdown_connections 2021-02-10T11:39:06.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.241 7f94d83d8700 1 -- 172.21.2.112:0/3160789533 wait complete. 2021-02-10T11:39:06.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.256 7f94d83d8700 1 Processor -- start 2021-02-10T11:39:06.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.256 7f94d83d8700 1 -- start start 2021-02-10T11:39:06.252 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.256 7f94d83d8700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f94d01942b0 0x7f94d00b3550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:06.253 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.256 7f94d83d8700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f94d0195070 0x7f94d0195470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:06.253 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.256 7f94d83d8700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f94d00bb8e0 0x7f94d00bbce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:06.253 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.256 7f94d83d8700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94d01946b0 con 0x7f94d01942b0 2021-02-10T11:39:06.253 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.256 7f94d83d8700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94d00b3b70 con 0x7f94d0195070 2021-02-10T11:39:06.253 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.256 7f94d83d8700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94d00b3d90 con 0x7f94d00bb8e0 2021-02-10T11:39:06.253 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.257 7f94d61f4700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f94d01942b0 0x7f94d00b3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:06.253 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.257 7f94d61f4700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f94d01942b0 0x7f94d00b3550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:53464/0 (socket says 172.21.2.112:53464) 2021-02-10T11:39:06.253 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.257 7f94d61f4700 1 -- 172.21.2.112:0/1969458090 learned_addr learned my addr 172.21.2.112:0/1969458090 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:39:06.253 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.257 7f94d69f5700 1 --2- 172.21.2.112:0/1969458090 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f94d00bb8e0 0x7f94d00bbce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:06.253 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.257 7f94d59f3700 1 --2- 172.21.2.112:0/1969458090 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f94d0195070 0x7f94d0195470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:06.253 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.257 7f94d61f4700 1 -- 172.21.2.112:0/1969458090 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f94d00bb8e0 msgr2=0x7f94d00bbce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:06.253 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.257 7f94d61f4700 1 --2- 172.21.2.112:0/1969458090 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f94d00bb8e0 0x7f94d00bbce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:06.253 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.257 7f94d61f4700 1 -- 172.21.2.112:0/1969458090 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f94d0195070 msgr2=0x7f94d0195470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:06.254 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.257 7f94d61f4700 1 --2- 172.21.2.112:0/1969458090 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f94d0195070 0x7f94d0195470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:06.254 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.257 7f94d61f4700 1 -- 172.21.2.112:0/1969458090 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94d01a49e0 con 0x7f94d01942b0 2021-02-10T11:39:06.254 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.257 7f94d61f4700 1 --2- 172.21.2.112:0/1969458090 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f94d01942b0 0x7f94d00b3550 secure :-1 s=READY pgs=1866 cs=0 l=1 rev1=1 rx=0x7f94cc006f20 tx=0x7f94cc00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:06.254 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.257 7f94c2ffd700 1 -- 172.21.2.112:0/1969458090 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f94cc00bd80 con 0x7f94d01942b0 2021-02-10T11:39:06.254 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.257 7f94c2ffd700 1 -- 172.21.2.112:0/1969458090 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94cc000c90 con 0x7f94d01942b0 2021-02-10T11:39:06.254 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.257 7f94c2ffd700 1 -- 172.21.2.112:0/1969458090 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f94cc0022b0 con 0x7f94d01942b0 2021-02-10T11:39:06.254 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.257 7f94d83d8700 1 -- 172.21.2.112:0/1969458090 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94d01a49e0 con 0x7f94d01942b0 2021-02-10T11:39:06.254 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.258 7f94d83d8700 1 -- 172.21.2.112:0/1969458090 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94d01a49e0 con 0x7f94d01942b0 2021-02-10T11:39:06.254 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.258 7f94c2ffd700 1 -- 172.21.2.112:0/1969458090 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f94cc00a000 con 0x7f94d01942b0 2021-02-10T11:39:06.254 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.258 7f94c2ffd700 1 --2- 172.21.2.112:0/1969458090 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f94b80186d0 0x7f94b801abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:06.254 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.258 7f94c2ffd700 1 -- 172.21.2.112:0/1969458090 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f94cc032790 con 0x7f94d01942b0 2021-02-10T11:39:06.254 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.258 7f94d59f3700 1 --2- 172.21.2.112:0/1969458090 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f94b80186d0 0x7f94b801abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:06.254 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.259 7f94d59f3700 1 --2- 172.21.2.112:0/1969458090 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f94b80186d0 0x7f94b801abd0 crc :-1 s=READY pgs=878 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:06.282 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.289 7f94d83d8700 1 -- 172.21.2.112:0/1969458090 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94d00bc5f0 con 0x7f94d01942b0 2021-02-10T11:39:06.284 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.292 7f94c2ffd700 1 -- 172.21.2.112:0/1969458090 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f94cc006bb0 con 0x7f94d01942b0 2021-02-10T11:39:06.411 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.418 7f94d83d8700 1 -- 172.21.2.112:0/1969458090 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f94d00bc5f0 con 0x7f94d01942b0 2021-02-10T11:39:06.411 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.419 7f94c2ffd700 1 -- 172.21.2.112:0/1969458090 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) v1 ==== 74+0+13996 (secure 0 0 0) 0x7f94cc000c90 con 0x7f94d01942b0 2021-02-10T11:39:06.412 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:39:06.412 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":92,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:38:52.412732","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:38:49.384013","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":19,"pool_name":"cephfs_data","create_time":"2021-02-10 11:38:50.454005","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:39:06.413 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.421 7f94d83d8700 1 -- 172.21.2.112:0/1969458090 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f94b80186d0 msgr2=0x7f94b801abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:06.413 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.421 7f94d83d8700 1 --2- 172.21.2.112:0/1969458090 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f94b80186d0 0x7f94b801abd0 crc :-1 s=READY pgs=878 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:06.413 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.421 7f94d83d8700 1 -- 172.21.2.112:0/1969458090 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f94d01942b0 msgr2=0x7f94d00b3550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:06.414 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.421 7f94d83d8700 1 --2- 172.21.2.112:0/1969458090 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f94d01942b0 0x7f94d00b3550 secure :-1 s=READY pgs=1866 cs=0 l=1 rev1=1 rx=0x7f94cc006f20 tx=0x7f94cc00b990).stop 2021-02-10T11:39:06.414 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.421 7f94d83d8700 1 -- 172.21.2.112:0/1969458090 shutdown_connections 2021-02-10T11:39:06.414 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.421 7f94d83d8700 1 --2- 172.21.2.112:0/1969458090 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f94b80186d0 0x7f94b801abd0 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:06.414 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.421 7f94d83d8700 1 --2- 172.21.2.112:0/1969458090 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f94d0195070 0x7f94d0195470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:06.414 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.421 7f94d83d8700 1 --2- 172.21.2.112:0/1969458090 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f94d00bb8e0 0x7f94d00bbce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:06.414 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.421 7f94d83d8700 1 --2- 172.21.2.112:0/1969458090 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f94d01942b0 0x7f94d00b3550 unknown :-1 s=CLOSED pgs=1866 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:06.414 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.421 7f94d83d8700 1 -- 172.21.2.112:0/1969458090 >> 172.21.2.112:0/1969458090 conn(0x7f94d00b3040 msgr2=0x7f94d00c3250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:06.414 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.421 7f94d83d8700 1 -- 172.21.2.112:0/1969458090 shutdown_connections 2021-02-10T11:39:06.414 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.421 7f94d83d8700 1 -- 172.21.2.112:0/1969458090 wait complete. 2021-02-10T11:39:06.422 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:13:57.508758', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 18, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:38:49.384013', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '91', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 19, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:38:50.454005', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '92', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:39:06.422 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:39:06.853 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.844 7ff14e5ea700 1 Processor -- start 2021-02-10T11:39:06.857 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.845 7ff14e5ea700 1 -- start start 2021-02-10T11:39:06.857 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.845 7ff14e5ea700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7ff1480d1870 0x7ff1481b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:06.857 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.845 7ff14e5ea700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7ff1481b0ae0 0x7ff148108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:06.857 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.845 7ff14e5ea700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1480fe1c0 con 0x7ff1480d1c70 2021-02-10T11:39:06.857 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.845 7ff14e5ea700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff148109a60 con 0x7ff1480d1870 2021-02-10T11:39:06.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.845 7ff14e5ea700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff148109750 con 0x7ff1481b0ae0 2021-02-10T11:39:06.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.845 7ff147fff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7ff1480d1870 0x7ff1481b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:06.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.845 7ff14cc07700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7ff1480d1c70 0x7ff148108f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:52934/0 (socket says 172.21.2.112:52934) 2021-02-10T11:39:06.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.845 7ff14cc07700 1 -- 172.21.2.112:0/3306617560 learned_addr learned my addr 172.21.2.112:0/3306617560 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:39:06.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.846 7ff147fff700 1 -- 172.21.2.112:0/3306617560 >> v2:172.21.2.120:3300/0 conn(0x7ff1481b0ae0 msgr2=0x7ff148108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:06.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.846 7ff147fff700 1 --2- 172.21.2.112:0/3306617560 >> v2:172.21.2.120:3300/0 conn(0x7ff1481b0ae0 0x7ff148108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:06.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.846 7ff147fff700 1 -- 172.21.2.112:0/3306617560 >> v1:172.21.2.112:6789/0 conn(0x7ff1480d1c70 legacy=0x7ff148108f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:06.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.846 7ff146ffd700 1 -- 172.21.2.112:0/3306617560 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3631202891 0 0) 0x7ff140001500 con 0x7ff1480d1c70 2021-02-10T11:39:06.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.846 7ff147fff700 1 -- 172.21.2.112:0/3306617560 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff148088070 con 0x7ff1480d1870 2021-02-10T11:39:06.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.846 7ff147fff700 1 --2- 172.21.2.112:0/3306617560 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff1480d1870 0x7ff1481b04c0 secure :-1 s=READY pgs=1868 cs=0 l=1 rev1=1 rx=0x7ff138006650 tx=0x7ff138009a00).ready entity=mon.0 client_cookie=e22a37757a98e297 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:06.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.846 7ff146ffd700 1 -- 172.21.2.112:0/3306617560 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff138009df0 con 0x7ff1480d1870 2021-02-10T11:39:06.860 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.846 7ff146ffd700 1 -- 172.21.2.112:0/3306617560 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1380009b0 con 0x7ff1480d1870 2021-02-10T11:39:06.860 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.846 7ff146ffd700 1 -- 172.21.2.112:0/3306617560 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff13800c2b0 con 0x7ff1480d1870 2021-02-10T11:39:06.860 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.846 7ff14e5ea700 1 -- 172.21.2.112:0/3306617560 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff1480d1870 msgr2=0x7ff1481b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:06.860 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.846 7ff14e5ea700 1 --2- 172.21.2.112:0/3306617560 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff1480d1870 0x7ff1481b04c0 secure :-1 s=READY pgs=1868 cs=0 l=1 rev1=1 rx=0x7ff138006650 tx=0x7ff138009a00).stop 2021-02-10T11:39:06.860 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.847 7ff14e5ea700 1 -- 172.21.2.112:0/3306617560 shutdown_connections 2021-02-10T11:39:06.860 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.847 7ff14e5ea700 1 --2- 172.21.2.112:0/3306617560 >> v2:172.21.2.120:3300/0 conn(0x7ff1481b0ae0 0x7ff148108940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:06.860 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.847 7ff14e5ea700 1 --2- 172.21.2.112:0/3306617560 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff1480d1870 0x7ff1481b04c0 unknown :-1 s=CLOSED pgs=1868 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:06.860 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.847 7ff14e5ea700 1 -- 172.21.2.112:0/3306617560 >> 172.21.2.112:0/3306617560 conn(0x7ff1480ce420 msgr2=0x7ff1480d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:06.860 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.847 7ff14e5ea700 1 -- 172.21.2.112:0/3306617560 shutdown_connections 2021-02-10T11:39:06.860 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.847 7ff14e5ea700 1 -- 172.21.2.112:0/3306617560 wait complete. 2021-02-10T11:39:06.862 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.861 7ff14e5ea700 1 Processor -- start 2021-02-10T11:39:06.862 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.861 7ff14e5ea700 1 -- start start 2021-02-10T11:39:06.862 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.862 7ff14e5ea700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff1480c3820 0x7ff148084cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:06.862 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.862 7ff14e5ea700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff1480a0290 0x7ff1480852d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:06.862 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.862 7ff14e5ea700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff148086270 0x7ff148086670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:06.862 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.862 7ff14e5ea700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1480c0700 con 0x7ff148086270 2021-02-10T11:39:06.862 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.862 7ff14e5ea700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1480858f0 con 0x7ff1480a0290 2021-02-10T11:39:06.862 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.862 7ff14e5ea700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff148085b10 con 0x7ff1480c3820 2021-02-10T11:39:06.862 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.862 7ff14cc07700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff148086270 0x7ff148086670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:06.862 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.862 7ff14cc07700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff148086270 0x7ff148086670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:53478/0 (socket says 172.21.2.112:53478) 2021-02-10T11:39:06.862 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.862 7ff14cc07700 1 -- 172.21.2.112:0/3291531184 learned_addr learned my addr 172.21.2.112:0/3291531184 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:39:06.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.862 7ff14cc07700 1 -- 172.21.2.112:0/3291531184 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff1480c3820 msgr2=0x7ff148084cb0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:39:06.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.862 7ff14cc07700 1 --2- 172.21.2.112:0/3291531184 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff1480c3820 0x7ff148084cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:06.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.863 7ff14cc07700 1 -- 172.21.2.112:0/3291531184 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff1480a0290 msgr2=0x7ff1480852d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:06.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.863 7ff14cc07700 1 --2- 172.21.2.112:0/3291531184 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff1480a0290 0x7ff1480852d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:06.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.863 7ff14cc07700 1 -- 172.21.2.112:0/3291531184 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff148086c90 con 0x7ff148086270 2021-02-10T11:39:06.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.863 7ff14cc07700 1 --2- 172.21.2.112:0/3291531184 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff148086270 0x7ff148086670 secure :-1 s=READY pgs=1869 cs=0 l=1 rev1=1 rx=0x7ff140001c10 tx=0x7ff14000ac60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:06.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.863 7ff144ff9700 1 -- 172.21.2.112:0/3291531184 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff140007020 con 0x7ff148086270 2021-02-10T11:39:06.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.863 7ff144ff9700 1 -- 172.21.2.112:0/3291531184 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1400072a0 con 0x7ff148086270 2021-02-10T11:39:06.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.863 7ff144ff9700 1 -- 172.21.2.112:0/3291531184 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff140004720 con 0x7ff148086270 2021-02-10T11:39:06.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.863 7ff14e5ea700 1 -- 172.21.2.112:0/3291531184 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff148086c90 con 0x7ff148086270 2021-02-10T11:39:06.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.863 7ff14e5ea700 1 -- 172.21.2.112:0/3291531184 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff148086c90 con 0x7ff148086270 2021-02-10T11:39:06.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.864 7ff144ff9700 1 -- 172.21.2.112:0/3291531184 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff140007480 con 0x7ff148086270 2021-02-10T11:39:06.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.864 7ff144ff9700 1 --2- 172.21.2.112:0/3291531184 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff1280186c0 0x7ff12801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:06.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.864 7ff144ff9700 1 -- 172.21.2.112:0/3291531184 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5573+0+0 (secure 0 0 0) 0x7ff1400328b0 con 0x7ff148086270 2021-02-10T11:39:06.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.864 7ff147fff700 1 --2- 172.21.2.112:0/3291531184 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff1280186c0 0x7ff12801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:06.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.865 7ff147fff700 1 --2- 172.21.2.112:0/3291531184 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff1280186c0 0x7ff12801abc0 crc :-1 s=READY pgs=879 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:06.888 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.895 7ff14e5ea700 1 -- 172.21.2.112:0/3291531184 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff148082a70 con 0x7ff148086270 2021-02-10T11:39:06.890 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:06.898 7ff144ff9700 1 -- 172.21.2.112:0/3291531184 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff140000cc0 con 0x7ff148086270 2021-02-10T11:39:07.017 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:07.024 7ff14e5ea700 1 -- 172.21.2.112:0/3291531184 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7ff148082a70 con 0x7ff148086270 2021-02-10T11:39:07.018 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:07.025 7ff144ff9700 1 -- 172.21.2.112:0/3291531184 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v92) v1 ==== 92+0+10 (secure 0 0 0) 0x7ff140007480 con 0x7ff148086270 2021-02-10T11:39:07.018 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:39:07.019 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:07.027 7ff14e5ea700 1 -- 172.21.2.112:0/3291531184 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff1280186c0 msgr2=0x7ff12801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:07.019 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:07.027 7ff14e5ea700 1 --2- 172.21.2.112:0/3291531184 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff1280186c0 0x7ff12801abc0 crc :-1 s=READY pgs=879 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:07.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:07.027 7ff14e5ea700 1 -- 172.21.2.112:0/3291531184 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff148086270 msgr2=0x7ff148086670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:07.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:07.027 7ff14e5ea700 1 --2- 172.21.2.112:0/3291531184 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff148086270 0x7ff148086670 secure :-1 s=READY pgs=1869 cs=0 l=1 rev1=1 rx=0x7ff140001c10 tx=0x7ff14000ac60).stop 2021-02-10T11:39:07.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:07.027 7ff14e5ea700 1 -- 172.21.2.112:0/3291531184 shutdown_connections 2021-02-10T11:39:07.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:07.027 7ff14e5ea700 1 --2- 172.21.2.112:0/3291531184 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff1280186c0 0x7ff12801abc0 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:07.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:07.027 7ff14e5ea700 1 --2- 172.21.2.112:0/3291531184 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff1480a0290 0x7ff1480852d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:07.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:07.027 7ff14e5ea700 1 --2- 172.21.2.112:0/3291531184 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff148086270 0x7ff148086670 unknown :-1 s=CLOSED pgs=1869 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:07.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:07.027 7ff14e5ea700 1 --2- 172.21.2.112:0/3291531184 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff1480c3820 0x7ff148084cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:07.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:07.027 7ff14e5ea700 1 -- 172.21.2.112:0/3291531184 >> 172.21.2.112:0/3291531184 conn(0x7ff1480872c0 msgr2=0x7ff148095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:07.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:07.027 7ff14e5ea700 1 -- 172.21.2.112:0/3291531184 shutdown_connections 2021-02-10T11:39:07.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:07.027 7ff14e5ea700 1 -- 172.21.2.112:0/3291531184 wait complete. 2021-02-10T11:39:07.028 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:39:07.457 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:07.447 7fd0cc378700 1 Processor -- start 2021-02-10T11:39:07.457 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:07.447 7fd0cc378700 1 -- start start 2021-02-10T11:39:07.457 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:07.448 7fd0cc378700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fd0c410afe0 0x7fd0c40cf740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:07.457 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:07.448 7fd0cc378700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0c40fe1b0 con 0x7fd0c40d1c60 2021-02-10T11:39:07.457 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:07.448 7fd0cc378700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0c41b0ca0 con 0x7fd0c40d1860 2021-02-10T11:39:07.457 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:07.448 7fd0cc378700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0c4109a50 con 0x7fd0c410afe0 2021-02-10T11:39:07.458 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:07.448 7fd0c9993700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fd0c40d1c60 0x7fd0c40cef50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:52946/0 (socket says 172.21.2.112:52946) 2021-02-10T11:39:07.458 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:07.448 7fd0c9993700 1 -- 172.21.2.112:0/2172839263 learned_addr learned my addr 172.21.2.112:0/2172839263 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:39:07.458 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:07.448 7fd0c9192700 1 -- 172.21.2.112:0/2172839263 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1059502456 0 0) 0x7fd0b40009b0 con 0x7fd0c40d1c60 2021-02-10T11:39:07.458 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:07.448 7fd0c9192700 1 -- 172.21.2.112:0/2172839263 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd0b0002fe0 con 0x7fd0c40d1c60 2021-02-10T11:39:07.458 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:07.449 7fd0ca995700 1 --2- 172.21.2.112:0/2172839263 >> v2:172.21.2.120:3300/0 conn(0x7fd0c410afe0 0x7fd0c40cf740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:07.458 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:07.449 7fd0c9192700 1 -- 172.21.2.112:0/2172839263 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3098565318 0 0) 0x7fd0b40010b0 con 0x7fd0c40d1c60 2021-02-10T11:39:07.458 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:07.449 7fd0c9192700 1 -- 172.21.2.112:0/2172839263 >> v2:172.21.2.120:3300/0 conn(0x7fd0c410afe0 msgr2=0x7fd0c40cf740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:07.458 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:07.449 7fd0c9192700 1 --2- 172.21.2.112:0/2172839263 >> v2:172.21.2.120:3300/0 conn(0x7fd0c410afe0 0x7fd0c40cf740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:07.458 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:07.449 7fd0c9192700 1 -- 172.21.2.112:0/2172839263 >> v1:172.21.2.120:6789/0 conn(0x7fd0c40d1860 legacy=0x7fd0c41b04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:07.458 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:07.449 7fd0c9192700 1 -- 172.21.2.112:0/2172839263 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0c410b3e0 con 0x7fd0c40d1c60 2021-02-10T11:39:07.458 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:07.449 7fd0c9192700 1 -- 172.21.2.112:0/2172839263 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7fd0b40015b0 con 0x7fd0c40d1c60 2021-02-10T11:39:07.458 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:07.449 7fd0c9192700 1 -- 172.21.2.112:0/2172839263 >> v1:172.21.2.112:6789/0 conn(0x7fd0c40d1c60 legacy=0x7fd0c40cef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:07.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:07.449 7fd0c9192700 1 --2- 172.21.2.112:0/2172839263 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd0b0003ac0 0x7fd0b0005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:07.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:07.449 7fd0c9192700 1 --2- 172.21.2.112:0/2172839263 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd0b0006790 0x7fd0b0008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:07.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:07.449 7fd0c9192700 1 --2- 172.21.2.112:0/2172839263 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd0b0009590 0x7fd0b000bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:07.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:07.449 7fd0c9192700 1 -- 172.21.2.112:0/2172839263 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0b000c300 con 0x7fd0b0006790 2021-02-10T11:39:07.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:07.449 7fd0c9192700 1 -- 172.21.2.112:0/2172839263 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0b000c520 con 0x7fd0b0009590 2021-02-10T11:39:07.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:07.449 7fd0c9192700 1 -- 172.21.2.112:0/2172839263 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0b0050750 con 0x7fd0b0003ac0 2021-02-10T11:39:07.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:07.449 7fd0c9993700 1 --2- 172.21.2.112:0/2172839263 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd0b0006790 0x7fd0b0008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:07.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:07.449 7fd0ca995700 1 --2- 172.21.2.112:0/2172839263 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd0b0009590 0x7fd0b000bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:07.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:07.449 7fd0c9993700 1 -- 172.21.2.112:0/2172839263 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd0b0003ac0 msgr2=0x7fd0b0005ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:07.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:07.449 7fd0c9993700 1 --2- 172.21.2.112:0/2172839263 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd0b0003ac0 0x7fd0b0005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:07.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:07.449 7fd0c9993700 1 -- 172.21.2.112:0/2172839263 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd0b0009590 msgr2=0x7fd0b000bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:07.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:07.449 7fd0c9993700 1 --2- 172.21.2.112:0/2172839263 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd0b0009590 0x7fd0b000bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:07.460 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:07.449 7fd0ca995700 1 -- 172.21.2.112:0/2172839263 reap_dead start 2021-02-10T11:39:07.460 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:07.449 7fd0c9993700 1 -- 172.21.2.112:0/2172839263 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0b00509b0 con 0x7fd0b0006790 2021-02-10T11:39:07.460 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:07.449 7fd0c9993700 1 --2- 172.21.2.112:0/2172839263 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd0b0006790 0x7fd0b0008c90 secure :-1 s=READY pgs=1871 cs=0 l=1 rev1=1 rx=0x7fd0b4000b60 tx=0x7fd0b4007d40).ready entity=mon.0 client_cookie=e347160812d63194 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:07.460 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:07.449 7fd0c9192700 1 -- 172.21.2.112:0/2172839263 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd0b4001010 con 0x7fd0b0006790 2021-02-10T11:39:07.460 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:07.450 7fd0c9192700 1 -- 172.21.2.112:0/2172839263 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0b4000980 con 0x7fd0b0006790 2021-02-10T11:39:07.460 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:07.450 7fd0c9192700 1 -- 172.21.2.112:0/2172839263 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd0b400a520 con 0x7fd0b0006790 2021-02-10T11:39:07.460 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:07.450 7fd0cc378700 1 -- 172.21.2.112:0/2172839263 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd0b0006790 msgr2=0x7fd0b0008c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:07.460 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:07.450 7fd0cc378700 1 --2- 172.21.2.112:0/2172839263 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd0b0006790 0x7fd0b0008c90 secure :-1 s=READY pgs=1871 cs=0 l=1 rev1=1 rx=0x7fd0b4000b60 tx=0x7fd0b4007d40).stop 2021-02-10T11:39:07.460 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:07.450 7fd0cc378700 1 -- 172.21.2.112:0/2172839263 shutdown_connections 2021-02-10T11:39:07.460 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:07.450 7fd0cc378700 1 --2- 172.21.2.112:0/2172839263 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd0b0006790 0x7fd0b0008c90 unknown :-1 s=CLOSED pgs=1871 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:07.460 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:07.450 7fd0cc378700 1 -- 172.21.2.112:0/2172839263 >> 172.21.2.112:0/2172839263 conn(0x7fd0c40ce410 msgr2=0x7fd0c40d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:07.460 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:07.450 7fd0cc378700 1 -- 172.21.2.112:0/2172839263 shutdown_connections 2021-02-10T11:39:07.460 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:07.450 7fd0cc378700 1 -- 172.21.2.112:0/2172839263 wait complete. 2021-02-10T11:39:07.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:07.465 7fd0cc378700 1 Processor -- start 2021-02-10T11:39:07.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:07.465 7fd0cc378700 1 -- start start 2021-02-10T11:39:07.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:07.465 7fd0cc378700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd0c40badd0 0x7fd0c409eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:07.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:07.465 7fd0cc378700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd0c40adf50 0x7fd0c409f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:07.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:07.465 7fd0cc378700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd0c41a7470 0x7fd0c41a7870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:07.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:07.465 7fd0cc378700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0c409e5a0 con 0x7fd0c41a7470 2021-02-10T11:39:07.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:07.465 7fd0cc378700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0c409f6e0 con 0x7fd0c40adf50 2021-02-10T11:39:07.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:07.465 7fd0cc378700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0c40bb220 con 0x7fd0c40badd0 2021-02-10T11:39:07.466 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:07.465 7fd0ca995700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd0c41a7470 0x7fd0c41a7870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:07.466 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:07.466 7fd0ca995700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd0c41a7470 0x7fd0c41a7870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:53496/0 (socket says 172.21.2.112:53496) 2021-02-10T11:39:07.466 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:07.466 7fd0ca995700 1 -- 172.21.2.112:0/183556169 learned_addr learned my addr 172.21.2.112:0/183556169 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:39:07.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:07.466 7fd0ca995700 1 -- 172.21.2.112:0/183556169 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd0c40badd0 msgr2=0x7fd0c409eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:07.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:07.466 7fd0ca995700 1 --2- 172.21.2.112:0/183556169 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd0c40badd0 0x7fd0c409eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:07.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:07.466 7fd0ca995700 1 -- 172.21.2.112:0/183556169 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd0c40adf50 msgr2=0x7fd0c409f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:07.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:07.466 7fd0ca995700 1 --2- 172.21.2.112:0/183556169 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd0c40adf50 0x7fd0c409f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:07.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:07.466 7fd0ca995700 1 -- 172.21.2.112:0/183556169 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0c41a7e90 con 0x7fd0c41a7470 2021-02-10T11:39:07.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:07.466 7fd0ca995700 1 --2- 172.21.2.112:0/183556169 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd0c41a7470 0x7fd0c41a7870 secure :-1 s=READY pgs=1872 cs=0 l=1 rev1=1 rx=0x7fd0bc00b970 tx=0x7fd0bc00bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:07.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:07.466 7fd0baffd700 1 -- 172.21.2.112:0/183556169 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd0bc000f90 con 0x7fd0c41a7470 2021-02-10T11:39:07.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:07.466 7fd0baffd700 1 -- 172.21.2.112:0/183556169 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0bc0011f0 con 0x7fd0c41a7470 2021-02-10T11:39:07.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:07.466 7fd0baffd700 1 -- 172.21.2.112:0/183556169 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd0bc001710 con 0x7fd0c41a7470 2021-02-10T11:39:07.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:07.466 7fd0cc378700 1 -- 172.21.2.112:0/183556169 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0c41a7e90 con 0x7fd0c41a7470 2021-02-10T11:39:07.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:07.466 7fd0cc378700 1 -- 172.21.2.112:0/183556169 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0c409d1f0 con 0x7fd0c41a7470 2021-02-10T11:39:07.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:07.467 7fd0baffd700 1 -- 172.21.2.112:0/183556169 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd0bc000e80 con 0x7fd0c41a7470 2021-02-10T11:39:07.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:07.467 7fd0baffd700 1 --2- 172.21.2.112:0/183556169 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd0ac0186c0 0x7fd0ac01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:07.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:07.467 7fd0baffd700 1 -- 172.21.2.112:0/183556169 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fd0bc032780 con 0x7fd0c41a7470 2021-02-10T11:39:07.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:07.467 7fd0ca194700 1 --2- 172.21.2.112:0/183556169 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd0ac0186c0 0x7fd0ac01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:07.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:07.468 7fd0ca194700 1 --2- 172.21.2.112:0/183556169 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd0ac0186c0 0x7fd0ac01abc0 crc :-1 s=READY pgs=880 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:07.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:07.498 7fd0cc378700 1 -- 172.21.2.112:0/183556169 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0c409f6e0 con 0x7fd0c41a7470 2021-02-10T11:39:07.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:07.500 7fd0baffd700 1 -- 172.21.2.112:0/183556169 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd0bc006720 con 0x7fd0c41a7470 2021-02-10T11:39:07.620 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:07.627 7fd0cc378700 1 -- 172.21.2.112:0/183556169 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7fd0c409f6e0 con 0x7fd0c41a7470 2021-02-10T11:39:07.620 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:07.628 7fd0baffd700 1 -- 172.21.2.112:0/183556169 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v92) v1 ==== 104+0+10 (secure 0 0 0) 0x7fd0bc000e80 con 0x7fd0c41a7470 2021-02-10T11:39:07.621 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:39:07.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:07.630 7fd0cc378700 1 -- 172.21.2.112:0/183556169 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd0ac0186c0 msgr2=0x7fd0ac01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:07.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:07.630 7fd0cc378700 1 --2- 172.21.2.112:0/183556169 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd0ac0186c0 0x7fd0ac01abc0 crc :-1 s=READY pgs=880 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:07.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:07.630 7fd0cc378700 1 -- 172.21.2.112:0/183556169 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd0c41a7470 msgr2=0x7fd0c41a7870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:07.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:07.630 7fd0cc378700 1 --2- 172.21.2.112:0/183556169 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd0c41a7470 0x7fd0c41a7870 secure :-1 s=READY pgs=1872 cs=0 l=1 rev1=1 rx=0x7fd0bc00b970 tx=0x7fd0bc00bd10).stop 2021-02-10T11:39:07.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:07.630 7fd0cc378700 1 -- 172.21.2.112:0/183556169 shutdown_connections 2021-02-10T11:39:07.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:07.630 7fd0cc378700 1 --2- 172.21.2.112:0/183556169 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd0ac0186c0 0x7fd0ac01abc0 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:07.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:07.630 7fd0cc378700 1 --2- 172.21.2.112:0/183556169 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd0c40adf50 0x7fd0c409f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:07.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:07.630 7fd0cc378700 1 --2- 172.21.2.112:0/183556169 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd0c41a7470 0x7fd0c41a7870 unknown :-1 s=CLOSED pgs=1872 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:07.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:07.630 7fd0cc378700 1 --2- 172.21.2.112:0/183556169 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd0c40badd0 0x7fd0c409eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:07.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:07.630 7fd0cc378700 1 -- 172.21.2.112:0/183556169 >> 172.21.2.112:0/183556169 conn(0x7fd0c40bee90 msgr2=0x7fd0c40bb880 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:07.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:07.630 7fd0cc378700 1 -- 172.21.2.112:0/183556169 shutdown_connections 2021-02-10T11:39:07.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:07.630 7fd0cc378700 1 -- 172.21.2.112:0/183556169 wait complete. 2021-02-10T11:39:07.631 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:39:08.063 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.051 7f441c858700 1 Processor -- start 2021-02-10T11:39:08.063 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.051 7f441c858700 1 -- start start 2021-02-10T11:39:08.063 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.052 7f441c858700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f44180d1c70 0x7f44180cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:08.063 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.052 7f441c858700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f44180cf580 0x7f44180cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:08.063 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.052 7f441c858700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44180fe1c0 con 0x7f44180d1870 2021-02-10T11:39:08.063 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.052 7f441c858700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44181b0cb0 con 0x7f44180d1c70 2021-02-10T11:39:08.063 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.052 7f441c858700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4418109a60 con 0x7f44180cf580 2021-02-10T11:39:08.063 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.053 7f4416e1d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f44180cf580 0x7f44180cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:08.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.053 7f4415e1b700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f44180d1c70 0x7f44180cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:08.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.053 7f441661c700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7f44180d1870 0x7f44181b04c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:46524/0 (socket says 172.21.2.112:46524) 2021-02-10T11:39:08.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.053 7f441661c700 1 -- 172.21.2.112:0/3990084277 learned_addr learned my addr 172.21.2.112:0/3990084277 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:39:08.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.054 7f4416e1d700 1 -- 172.21.2.112:0/3990084277 >> v2:172.21.2.120:3300/0 conn(0x7f44180d1c70 msgr2=0x7f44180cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:08.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.054 7f4416e1d700 1 --2- 172.21.2.112:0/3990084277 >> v2:172.21.2.120:3300/0 conn(0x7f44180d1c70 0x7f44180cef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:08.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.054 7f4416e1d700 1 -- 172.21.2.112:0/3990084277 >> v1:172.21.2.120:6789/0 conn(0x7f44180d1870 legacy=0x7f44181b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:08.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.054 7f4416e1d700 1 -- 172.21.2.112:0/3990084277 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4418088070 con 0x7f44180cf580 2021-02-10T11:39:08.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.054 7f441561a700 1 -- 172.21.2.112:0/3990084277 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1673200646 0 0) 0x7f4408001190 con 0x7f44180d1870 2021-02-10T11:39:08.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.054 7f4415e1b700 1 --2- 172.21.2.112:0/3990084277 >> v2:172.21.2.120:3300/0 conn(0x7f44180d1c70 0x7f44180cef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:39:08.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.054 7f4416e1d700 1 --2- 172.21.2.112:0/3990084277 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f44180cf580 0x7f44180cf980 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f4410006e50 tx=0x7f4410009e70).ready entity=mon.2 client_cookie=523956619bc8379f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:08.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.054 7f441561a700 1 -- 172.21.2.112:0/3990084277 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4410006a20 con 0x7f44180cf580 2021-02-10T11:39:08.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.054 7f441561a700 1 -- 172.21.2.112:0/3990084277 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4410006790 con 0x7f44180cf580 2021-02-10T11:39:08.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.054 7f441561a700 1 -- 172.21.2.112:0/3990084277 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f441000b700 con 0x7f44180cf580 2021-02-10T11:39:08.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.054 7f441c858700 1 -- 172.21.2.112:0/3990084277 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f44180cf580 msgr2=0x7f44180cf980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:08.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.054 7f441c858700 1 --2- 172.21.2.112:0/3990084277 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f44180cf580 0x7f44180cf980 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f4410006e50 tx=0x7f4410009e70).stop 2021-02-10T11:39:08.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.055 7f441c858700 1 -- 172.21.2.112:0/3990084277 shutdown_connections 2021-02-10T11:39:08.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.055 7f441c858700 1 --2- 172.21.2.112:0/3990084277 >> v2:172.21.2.120:3300/0 conn(0x7f44180d1c70 0x7f44180cef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:08.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.055 7f441c858700 1 --2- 172.21.2.112:0/3990084277 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f44180cf580 0x7f44180cf980 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:08.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.055 7f441c858700 1 -- 172.21.2.112:0/3990084277 >> 172.21.2.112:0/3990084277 conn(0x7f44180ce420 msgr2=0x7f44180d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:08.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.055 7f441c858700 1 -- 172.21.2.112:0/3990084277 shutdown_connections 2021-02-10T11:39:08.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.055 7f441c858700 1 -- 172.21.2.112:0/3990084277 wait complete. 2021-02-10T11:39:08.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.070 7f441c858700 1 Processor -- start 2021-02-10T11:39:08.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.070 7f441c858700 1 -- start start 2021-02-10T11:39:08.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.070 7f441c858700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4418085bc0 0x7f44180863e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:08.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.070 7f441c858700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f44180954c0 0x7f4418096300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:08.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.070 7f441c858700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f44180958c0 0x7f4418096920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:08.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.070 7f441c858700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4418095d80 con 0x7f4418085bc0 2021-02-10T11:39:08.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.070 7f441c858700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44180bfa60 con 0x7f44180954c0 2021-02-10T11:39:08.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.070 7f441c858700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4418085fc0 con 0x7f44180958c0 2021-02-10T11:39:08.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.070 7f441661c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4418085bc0 0x7f44180863e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:08.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.070 7f441661c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4418085bc0 0x7f44180863e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:53510/0 (socket says 172.21.2.112:53510) 2021-02-10T11:39:08.068 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.070 7f441661c700 1 -- 172.21.2.112:0/802818243 learned_addr learned my addr 172.21.2.112:0/802818243 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:39:08.068 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.071 7f441661c700 1 -- 172.21.2.112:0/802818243 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f44180958c0 msgr2=0x7f4418096920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:08.068 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.071 7f441661c700 1 --2- 172.21.2.112:0/802818243 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f44180958c0 0x7f4418096920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:08.068 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.071 7f441661c700 1 -- 172.21.2.112:0/802818243 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f44180954c0 msgr2=0x7f4418096300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:08.068 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.071 7f441661c700 1 --2- 172.21.2.112:0/802818243 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f44180954c0 0x7f4418096300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:08.068 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.071 7f441661c700 1 -- 172.21.2.112:0/802818243 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4418086c10 con 0x7f4418085bc0 2021-02-10T11:39:08.068 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.071 7f441661c700 1 --2- 172.21.2.112:0/802818243 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4418085bc0 0x7f44180863e0 secure :-1 s=READY pgs=1873 cs=0 l=1 rev1=1 rx=0x7f440800a590 tx=0x7f440800a930).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:08.068 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.071 7f4406ffd700 1 -- 172.21.2.112:0/802818243 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4408001190 con 0x7f4418085bc0 2021-02-10T11:39:08.068 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.071 7f4406ffd700 1 -- 172.21.2.112:0/802818243 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44080070d0 con 0x7f4418085bc0 2021-02-10T11:39:08.068 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.071 7f4406ffd700 1 -- 172.21.2.112:0/802818243 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4408004530 con 0x7f4418085bc0 2021-02-10T11:39:08.068 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.071 7f441c858700 1 -- 172.21.2.112:0/802818243 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4418086c10 con 0x7f4418085bc0 2021-02-10T11:39:08.068 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.071 7f441c858700 1 -- 172.21.2.112:0/802818243 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4418097230 con 0x7f4418085bc0 2021-02-10T11:39:08.068 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.071 7f4406ffd700 1 -- 172.21.2.112:0/802818243 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4408004320 con 0x7f4418085bc0 2021-02-10T11:39:08.068 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.072 7f4406ffd700 1 --2- 172.21.2.112:0/802818243 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f43fc0186c0 0x7f43fc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:08.069 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.072 7f4406ffd700 1 -- 172.21.2.112:0/802818243 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f44080323e0 con 0x7f4418085bc0 2021-02-10T11:39:08.069 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.072 7f4415e1b700 1 --2- 172.21.2.112:0/802818243 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f43fc0186c0 0x7f43fc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:08.069 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.072 7f4415e1b700 1 --2- 172.21.2.112:0/802818243 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f43fc0186c0 0x7f43fc01abc0 crc :-1 s=READY pgs=881 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:08.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.102 7f441c858700 1 -- 172.21.2.112:0/802818243 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44180fe1c0 con 0x7f4418085bc0 2021-02-10T11:39:08.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.104 7f4406ffd700 1 -- 172.21.2.112:0/802818243 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4408001130 con 0x7f4418085bc0 2021-02-10T11:39:08.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.232 7f441c858700 1 -- 172.21.2.112:0/802818243 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f44180fe1c0 con 0x7f4418085bc0 2021-02-10T11:39:08.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.232 7f4406ffd700 1 -- 172.21.2.112:0/802818243 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v92) v1 ==== 100+0+10 (secure 0 0 0) 0x7f440800ad20 con 0x7f4418085bc0 2021-02-10T11:39:08.227 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:39:08.228 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.235 7f441c858700 1 -- 172.21.2.112:0/802818243 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f43fc0186c0 msgr2=0x7f43fc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:08.228 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.235 7f441c858700 1 --2- 172.21.2.112:0/802818243 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f43fc0186c0 0x7f43fc01abc0 crc :-1 s=READY pgs=881 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:08.228 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.235 7f441c858700 1 -- 172.21.2.112:0/802818243 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4418085bc0 msgr2=0x7f44180863e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:08.228 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.235 7f441c858700 1 --2- 172.21.2.112:0/802818243 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4418085bc0 0x7f44180863e0 secure :-1 s=READY pgs=1873 cs=0 l=1 rev1=1 rx=0x7f440800a590 tx=0x7f440800a930).stop 2021-02-10T11:39:08.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.235 7f441c858700 1 -- 172.21.2.112:0/802818243 shutdown_connections 2021-02-10T11:39:08.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.235 7f441c858700 1 --2- 172.21.2.112:0/802818243 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f43fc0186c0 0x7f43fc01abc0 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:08.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.235 7f441c858700 1 --2- 172.21.2.112:0/802818243 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f44180954c0 0x7f4418096300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:08.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.235 7f441c858700 1 --2- 172.21.2.112:0/802818243 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f44180958c0 0x7f4418096920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:08.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.235 7f441c858700 1 --2- 172.21.2.112:0/802818243 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4418085bc0 0x7f44180863e0 unknown :-1 s=CLOSED pgs=1873 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:08.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.235 7f441c858700 1 -- 172.21.2.112:0/802818243 >> 172.21.2.112:0/802818243 conn(0x7f44180bef00 msgr2=0x7f441809d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:08.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.235 7f441c858700 1 -- 172.21.2.112:0/802818243 shutdown_connections 2021-02-10T11:39:08.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.235 7f441c858700 1 -- 172.21.2.112:0/802818243 wait complete. 2021-02-10T11:39:08.237 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:39:08.237 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:39:08.665 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.655 7f8e72941700 1 Processor -- start 2021-02-10T11:39:08.665 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.655 7f8e72941700 1 -- start start 2021-02-10T11:39:08.665 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.655 7f8e72941700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f8e6c0d1870 0x7f8e6c1b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:08.665 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.655 7f8e72941700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f8e6c1b0a10 0x7f8e6c0cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:08.665 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.655 7f8e72941700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e6c0f0b10 con 0x7f8e6c0d1c70 2021-02-10T11:39:08.665 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.655 7f8e72941700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e6c0e5ed0 con 0x7f8e6c0d1870 2021-02-10T11:39:08.665 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.655 7f8e72941700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e6c0f0650 con 0x7f8e6c1b0a10 2021-02-10T11:39:08.665 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.656 7f8e6bfff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f8e6c0d1870 0x7f8e6c1b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:08.666 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.656 7f8e6bfff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f8e6c0d1870 0x7f8e6c1b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:53516/0 (socket says 172.21.2.112:53516) 2021-02-10T11:39:08.666 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.656 7f8e6bfff700 1 -- 172.21.2.112:0/3304827226 learned_addr learned my addr 172.21.2.112:0/3304827226 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:39:08.666 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.656 7f8e6bfff700 1 -- 172.21.2.112:0/3304827226 >> v2:172.21.2.120:3300/0 conn(0x7f8e6c1b0a10 msgr2=0x7f8e6c0cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:08.666 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.656 7f8e6bfff700 1 --2- 172.21.2.112:0/3304827226 >> v2:172.21.2.120:3300/0 conn(0x7f8e6c1b0a10 0x7f8e6c0cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:08.666 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.656 7f8e6bfff700 1 -- 172.21.2.112:0/3304827226 >> v1:172.21.2.120:6789/0 conn(0x7f8e6c0d1c70 legacy=0x7f8e6c0cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:08.666 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.656 7f8e6bfff700 1 -- 172.21.2.112:0/3304827226 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e6c0f0840 con 0x7f8e6c0d1870 2021-02-10T11:39:08.666 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.656 7f8e6bfff700 1 --2- 172.21.2.112:0/3304827226 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8e6c0d1870 0x7f8e6c1b03f0 secure :-1 s=READY pgs=1874 cs=0 l=1 rev1=1 rx=0x7f8e5c006700 tx=0x7f8e5c009b80).ready entity=mon.0 client_cookie=a7243e175c427048 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:08.666 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.656 7f8e6affd700 1 -- 172.21.2.112:0/3304827226 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8e5c000c90 con 0x7f8e6c0d1870 2021-02-10T11:39:08.666 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.656 7f8e6affd700 1 -- 172.21.2.112:0/3304827226 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e5c00c0c0 con 0x7f8e6c0d1870 2021-02-10T11:39:08.666 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.656 7f8e6affd700 1 -- 172.21.2.112:0/3304827226 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8e5c00c5a0 con 0x7f8e6c0d1870 2021-02-10T11:39:08.666 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.656 7f8e72941700 1 -- 172.21.2.112:0/3304827226 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8e6c0d1870 msgr2=0x7f8e6c1b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:08.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.656 7f8e72941700 1 --2- 172.21.2.112:0/3304827226 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8e6c0d1870 0x7f8e6c1b03f0 secure :-1 s=READY pgs=1874 cs=0 l=1 rev1=1 rx=0x7f8e5c006700 tx=0x7f8e5c009b80).stop 2021-02-10T11:39:08.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.657 7f8e72941700 1 -- 172.21.2.112:0/3304827226 shutdown_connections 2021-02-10T11:39:08.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.657 7f8e72941700 1 --2- 172.21.2.112:0/3304827226 >> v2:172.21.2.120:3300/0 conn(0x7f8e6c1b0a10 0x7f8e6c0cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:08.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.657 7f8e72941700 1 --2- 172.21.2.112:0/3304827226 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8e6c0d1870 0x7f8e6c1b03f0 unknown :-1 s=CLOSED pgs=1874 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:08.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.657 7f8e72941700 1 -- 172.21.2.112:0/3304827226 >> 172.21.2.112:0/3304827226 conn(0x7f8e6c0ce450 msgr2=0x7f8e6c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:08.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.657 7f8e72941700 1 -- 172.21.2.112:0/3304827226 shutdown_connections 2021-02-10T11:39:08.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.657 7f8e72941700 1 -- 172.21.2.112:0/3304827226 wait complete. 2021-02-10T11:39:08.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.672 7f8e72941700 1 Processor -- start 2021-02-10T11:39:08.669 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.672 7f8e72941700 1 -- start start 2021-02-10T11:39:08.669 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.672 7f8e72941700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8e6c0b0ad0 0x7f8e6c091f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:08.669 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.672 7f8e72941700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8e6c1a8850 0x7f8e6c1a8c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:08.669 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.672 7f8e72941700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8e6c1a6fa0 0x7f8e6c1a73a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:08.669 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.672 7f8e72941700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e6c1abfe0 con 0x7f8e6c0b0ad0 2021-02-10T11:39:08.669 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.672 7f8e72941700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e6c1a9c40 con 0x7f8e6c1a6fa0 2021-02-10T11:39:08.669 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.672 7f8e72941700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e6c1a9730 con 0x7f8e6c1a8850 2021-02-10T11:39:08.669 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.672 7f8e6bfff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8e6c0b0ad0 0x7f8e6c091f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:08.669 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.672 7f8e6bfff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8e6c0b0ad0 0x7f8e6c091f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:53520/0 (socket says 172.21.2.112:53520) 2021-02-10T11:39:08.669 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.672 7f8e6bfff700 1 -- 172.21.2.112:0/3364179298 learned_addr learned my addr 172.21.2.112:0/3364179298 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:39:08.669 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.673 7f8e6bfff700 1 -- 172.21.2.112:0/3364179298 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8e6c1a8850 msgr2=0x7f8e6c1a8c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:08.669 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.673 7f8e6bfff700 1 --2- 172.21.2.112:0/3364179298 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8e6c1a8850 0x7f8e6c1a8c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:08.670 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.673 7f8e6bfff700 1 -- 172.21.2.112:0/3364179298 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8e6c1a6fa0 msgr2=0x7f8e6c1a73a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:08.670 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.673 7f8e6bfff700 1 --2- 172.21.2.112:0/3364179298 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8e6c1a6fa0 0x7f8e6c1a73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:08.670 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.673 7f8e6bfff700 1 -- 172.21.2.112:0/3364179298 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e6c1a9270 con 0x7f8e6c0b0ad0 2021-02-10T11:39:08.670 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.673 7f8e6bfff700 1 --2- 172.21.2.112:0/3364179298 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8e6c0b0ad0 0x7f8e6c091f30 secure :-1 s=READY pgs=1875 cs=0 l=1 rev1=1 rx=0x7f8e5c006f00 tx=0x7f8e5c00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:08.670 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.673 7f8e68ff9700 1 -- 172.21.2.112:0/3364179298 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8e5c006b10 con 0x7f8e6c0b0ad0 2021-02-10T11:39:08.670 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.673 7f8e68ff9700 1 -- 172.21.2.112:0/3364179298 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e5c00be10 con 0x7f8e6c0b0ad0 2021-02-10T11:39:08.670 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.673 7f8e68ff9700 1 -- 172.21.2.112:0/3364179298 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8e5c0052e0 con 0x7f8e6c0b0ad0 2021-02-10T11:39:08.670 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.673 7f8e72941700 1 -- 172.21.2.112:0/3364179298 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e6c1a9270 con 0x7f8e6c0b0ad0 2021-02-10T11:39:08.670 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.673 7f8e72941700 1 -- 172.21.2.112:0/3364179298 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e6c1a7cb0 con 0x7f8e6c0b0ad0 2021-02-10T11:39:08.670 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.674 7f8e68ff9700 1 -- 172.21.2.112:0/3364179298 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8e5c006030 con 0x7f8e6c0b0ad0 2021-02-10T11:39:08.670 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.674 7f8e68ff9700 1 --2- 172.21.2.112:0/3364179298 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8e4c0186c0 0x7f8e4c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:08.670 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.674 7f8e68ff9700 1 -- 172.21.2.112:0/3364179298 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f8e5c032370 con 0x7f8e6c0b0ad0 2021-02-10T11:39:08.670 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.674 7f8e6b7fe700 1 --2- 172.21.2.112:0/3364179298 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8e4c0186c0 0x7f8e4c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:08.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.675 7f8e6b7fe700 1 --2- 172.21.2.112:0/3364179298 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8e4c0186c0 0x7f8e4c01abc0 crc :-1 s=READY pgs=882 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:08.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.705 7f8e72941700 1 -- 172.21.2.112:0/3364179298 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e6c1abfe0 con 0x7f8e6c0b0ad0 2021-02-10T11:39:08.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.708 7f8e68ff9700 1 -- 172.21.2.112:0/3364179298 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8e5c006ac0 con 0x7f8e6c0b0ad0 2021-02-10T11:39:08.830 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.836 7f8e72941700 1 -- 172.21.2.112:0/3364179298 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8e6c1abfe0 con 0x7f8e6c0b0ad0 2021-02-10T11:39:08.831 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.837 7f8e68ff9700 1 -- 172.21.2.112:0/3364179298 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+2101 (secure 0 0 0) 0x7f8e5c006030 con 0x7f8e6c0b0ad0 2021-02-10T11:39:08.831 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:39:08.831 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":72,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9621,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/1174368781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":1174368781},{"type":"v1","addr":"172.21.2.112:6833","nonce":1174368781}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":69}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:38:51.410458","modified":"2021-02-10 11:38:53.445268","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6380},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6380":{"gid":6380,"name":"b","rank":0,"incarnation":70,"state":"up:active","state_seq":3,"addr":"172.21.2.120:6835/2402714407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":2402714407},{"type":"v1","addr":"172.21.2.120:6835","nonce":2402714407}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":9}]} 2021-02-10T11:39:08.831 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 72 2021-02-10T11:39:08.833 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.839 7f8e72941700 1 -- 172.21.2.112:0/3364179298 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8e4c0186c0 msgr2=0x7f8e4c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:08.833 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.839 7f8e72941700 1 --2- 172.21.2.112:0/3364179298 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8e4c0186c0 0x7f8e4c01abc0 crc :-1 s=READY pgs=882 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:08.833 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.839 7f8e72941700 1 -- 172.21.2.112:0/3364179298 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8e6c0b0ad0 msgr2=0x7f8e6c091f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:08.833 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.839 7f8e72941700 1 --2- 172.21.2.112:0/3364179298 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8e6c0b0ad0 0x7f8e6c091f30 secure :-1 s=READY pgs=1875 cs=0 l=1 rev1=1 rx=0x7f8e5c006f00 tx=0x7f8e5c00b990).stop 2021-02-10T11:39:08.833 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.839 7f8e72941700 1 -- 172.21.2.112:0/3364179298 shutdown_connections 2021-02-10T11:39:08.833 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.839 7f8e72941700 1 --2- 172.21.2.112:0/3364179298 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8e4c0186c0 0x7f8e4c01abc0 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:08.833 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.839 7f8e72941700 1 --2- 172.21.2.112:0/3364179298 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8e6c1a6fa0 0x7f8e6c1a73a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:08.833 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.839 7f8e72941700 1 --2- 172.21.2.112:0/3364179298 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8e6c1a8850 0x7f8e6c1a8c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:08.834 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.839 7f8e72941700 1 --2- 172.21.2.112:0/3364179298 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8e6c0b0ad0 0x7f8e6c091f30 unknown :-1 s=CLOSED pgs=1875 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:08.834 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.839 7f8e72941700 1 -- 172.21.2.112:0/3364179298 >> 172.21.2.112:0/3364179298 conn(0x7f8e6c19f8f0 msgr2=0x7f8e6c1a5f80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:08.834 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.839 7f8e72941700 1 -- 172.21.2.112:0/3364179298 shutdown_connections 2021-02-10T11:39:08.834 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:08.839 7f8e72941700 1 -- 172.21.2.112:0/3364179298 wait complete. 2021-02-10T11:39:09.042 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 72, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:38:51.410458', 'modified': '2021-02-10 11:38:53.445268', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6380}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6380': {'gid': 6380, 'name': 'b', 'rank': 0, 'incarnation': 70, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.120:6835/2402714407', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.120:6834', 'nonce': 2402714407}, {'type': 'v1', 'addr': '172.21.2.120:6835', 'nonce': 2402714407}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:39:09.043 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:39:09.043 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:39:09.442 INFO:teuthology.orchestra.run.gibba020.stdout:{ 2021-02-10T11:39:09.443 INFO:teuthology.orchestra.run.gibba020.stdout: "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", 2021-02-10T11:39:09.443 INFO:teuthology.orchestra.run.gibba020.stdout: "whoami": 0, 2021-02-10T11:39:09.443 INFO:teuthology.orchestra.run.gibba020.stdout: "id": 6380, 2021-02-10T11:39:09.443 INFO:teuthology.orchestra.run.gibba020.stdout: "want_state": "up:active", 2021-02-10T11:39:09.443 INFO:teuthology.orchestra.run.gibba020.stdout: "state": "up:active", 2021-02-10T11:39:09.443 INFO:teuthology.orchestra.run.gibba020.stdout: "rank_uptime": 18.037169033000001, 2021-02-10T11:39:09.443 INFO:teuthology.orchestra.run.gibba020.stdout: "mdsmap_epoch": 72, 2021-02-10T11:39:09.443 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch": 92, 2021-02-10T11:39:09.443 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch_barrier": 92, 2021-02-10T11:39:09.443 INFO:teuthology.orchestra.run.gibba020.stdout: "uptime": 23.670201916 2021-02-10T11:39:09.443 INFO:teuthology.orchestra.run.gibba020.stdout:} 2021-02-10T11:39:09.447 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", "whoami": 0, "id": 6380, "want_state": "up:active", "state": "up:active", "rank_uptime": 18.037169033000001, "mdsmap_epoch": 72, "osdmap_epoch": 92, "osdmap_epoch_barrier": 92, "uptime": 23.670201916 } 2021-02-10T11:39:09.447 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:39:09.447 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2021-02-10T11:39:09.447 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.1 at ubuntu@gibba012.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2021-02-10T11:39:09.447 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:39:09.447 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:39:09.781 DEBUG:teuthology.orchestra.run.gibba012:> sudo modprobe fuse 2021-02-10T11:39:10.127 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:39:10.127 DEBUG:teuthology.orchestra.run.gibba012:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:39:10.471 INFO:teuthology.orchestra.run.gibba012.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:39:10.471 INFO:teuthology.orchestra.run.gibba012.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:39:10.471 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:39:10.472 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:39:10.472 DEBUG:teuthology.orchestra.run.gibba012:> ls /sys/fs/fuse/connections 2021-02-10T11:39:10.805 INFO:teuthology.orchestra.run.gibba012.stdout:42 2021-02-10T11:39:10.806 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [42] 2021-02-10T11:39:10.806 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.1 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:39:10.809 DEBUG:teuthology.orchestra.run.gibba012:> sudo modprobe fuse 2021-02-10T11:39:11.152 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:39:11.152 DEBUG:teuthology.orchestra.run.gibba012:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:39:11.218 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba012.stderr:ceph-fuse[150463]: starting ceph client 2021-02-10T11:39:11.218 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba012.stderr:2021-02-10 11:39:11.225 7efc90c89f80 -1 init, newargv = 0x55c8419961a0 newargc=9 2021-02-10T11:39:11.222 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba012.stderr:ceph-fuse[150463]: starting fuse 2021-02-10T11:39:11.494 INFO:teuthology.orchestra.run.gibba012.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:39:11.495 INFO:teuthology.orchestra.run.gibba012.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:39:11.496 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:39:11.496 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:39:11.497 DEBUG:teuthology.orchestra.run.gibba012:> ls /sys/fs/fuse/connections 2021-02-10T11:39:11.832 INFO:teuthology.orchestra.run.gibba012.stdout:42 2021-02-10T11:39:11.832 INFO:teuthology.orchestra.run.gibba012.stdout:43 2021-02-10T11:39:11.833 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42, 43] 2021-02-10T11:39:11.833 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:39:11.833 DEBUG:teuthology.orchestra.run.gibba012:> sudo python3 -c ' 2021-02-10T11:39:11.834 DEBUG:teuthology.orchestra.run.gibba012:> import glob 2021-02-10T11:39:11.834 DEBUG:teuthology.orchestra.run.gibba012:> import re 2021-02-10T11:39:11.834 DEBUG:teuthology.orchestra.run.gibba012:> import os 2021-02-10T11:39:11.834 DEBUG:teuthology.orchestra.run.gibba012:> import subprocess 2021-02-10T11:39:11.834 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:39:11.834 DEBUG:teuthology.orchestra.run.gibba012:> def find_socket(client_name): 2021-02-10T11:39:11.834 DEBUG:teuthology.orchestra.run.gibba012:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2021-02-10T11:39:11.834 DEBUG:teuthology.orchestra.run.gibba012:> files = glob.glob(asok_path) 2021-02-10T11:39:11.834 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:39:11.834 DEBUG:teuthology.orchestra.run.gibba012:> # Given a non-glob path, it better be there 2021-02-10T11:39:11.835 DEBUG:teuthology.orchestra.run.gibba012:> if "*" not in asok_path: 2021-02-10T11:39:11.835 DEBUG:teuthology.orchestra.run.gibba012:> assert(len(files) == 1) 2021-02-10T11:39:11.835 DEBUG:teuthology.orchestra.run.gibba012:> return files[0] 2021-02-10T11:39:11.835 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:39:11.835 DEBUG:teuthology.orchestra.run.gibba012:> for f in files: 2021-02-10T11:39:11.835 DEBUG:teuthology.orchestra.run.gibba012:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:39:11.835 DEBUG:teuthology.orchestra.run.gibba012:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:39:11.835 DEBUG:teuthology.orchestra.run.gibba012:> return f 2021-02-10T11:39:11.835 DEBUG:teuthology.orchestra.run.gibba012:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:39:11.836 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:39:11.836 DEBUG:teuthology.orchestra.run.gibba012:> print(find_socket("client.1")) 2021-02-10T11:39:11.836 DEBUG:teuthology.orchestra.run.gibba012:> ' 2021-02-10T11:39:12.198 INFO:teuthology.orchestra.run.gibba012.stdout:/var/run/ceph/ceph-client.1.150463.asok 2021-02-10T11:39:12.201 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.150463.asok 2021-02-10T11:39:12.201 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:39:12.202 DEBUG:teuthology.orchestra.run.gibba012:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.150463.asok status 2021-02-10T11:39:12.591 INFO:teuthology.orchestra.run.gibba012.stdout:{ 2021-02-10T11:39:12.592 INFO:teuthology.orchestra.run.gibba012.stdout: "metadata": { 2021-02-10T11:39:12.592 INFO:teuthology.orchestra.run.gibba012.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:39:12.592 INFO:teuthology.orchestra.run.gibba012.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:39:12.592 INFO:teuthology.orchestra.run.gibba012.stdout: "entity_id": "1", 2021-02-10T11:39:12.592 INFO:teuthology.orchestra.run.gibba012.stdout: "hostname": "gibba012", 2021-02-10T11:39:12.592 INFO:teuthology.orchestra.run.gibba012.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2021-02-10T11:39:12.592 INFO:teuthology.orchestra.run.gibba012.stdout: "pid": "150463", 2021-02-10T11:39:12.592 INFO:teuthology.orchestra.run.gibba012.stdout: "root": "/" 2021-02-10T11:39:12.592 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:39:12.592 INFO:teuthology.orchestra.run.gibba012.stdout: "dentry_count": 0, 2021-02-10T11:39:12.592 INFO:teuthology.orchestra.run.gibba012.stdout: "dentry_pinned_count": 0, 2021-02-10T11:39:12.593 INFO:teuthology.orchestra.run.gibba012.stdout: "id": 9825, 2021-02-10T11:39:12.593 INFO:teuthology.orchestra.run.gibba012.stdout: "inst": { 2021-02-10T11:39:12.593 INFO:teuthology.orchestra.run.gibba012.stdout: "name": { 2021-02-10T11:39:12.593 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "client", 2021-02-10T11:39:12.593 INFO:teuthology.orchestra.run.gibba012.stdout: "num": 9825 2021-02-10T11:39:12.593 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:39:12.593 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": { 2021-02-10T11:39:12.593 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:39:12.593 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:0", 2021-02-10T11:39:12.593 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 989775129 2021-02-10T11:39:12.593 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:39:12.593 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:39:12.594 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": { 2021-02-10T11:39:12.594 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:39:12.594 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:0", 2021-02-10T11:39:12.594 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 989775129 2021-02-10T11:39:12.594 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:39:12.594 INFO:teuthology.orchestra.run.gibba012.stdout: "inst_str": "client.9825 172.21.2.112:0/989775129", 2021-02-10T11:39:12.594 INFO:teuthology.orchestra.run.gibba012.stdout: "addr_str": "172.21.2.112:0/989775129", 2021-02-10T11:39:12.594 INFO:teuthology.orchestra.run.gibba012.stdout: "inode_count": 1, 2021-02-10T11:39:12.594 INFO:teuthology.orchestra.run.gibba012.stdout: "mds_epoch": 72, 2021-02-10T11:39:12.594 INFO:teuthology.orchestra.run.gibba012.stdout: "osd_epoch": 92, 2021-02-10T11:39:12.594 INFO:teuthology.orchestra.run.gibba012.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:39:12.594 INFO:teuthology.orchestra.run.gibba012.stdout: "blacklisted": false 2021-02-10T11:39:12.594 INFO:teuthology.orchestra.run.gibba012.stdout:} 2021-02-10T11:39:12.595 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:39:12.596 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:39:12.596 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:39:12.932 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:39:12.932 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:39:12.932 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:39:12.932 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:39:13.279 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2021-02-10T11:39:13.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:13.701 7f8b76662700 1 Processor -- start 2021-02-10T11:39:13.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:13.701 7f8b76662700 1 -- start start 2021-02-10T11:39:13.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:13.702 7f8b76662700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8b7010b000 0x7f8b700cf760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:13.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:13.702 7f8b76662700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b700fe1d0 con 0x7f8b700d1880 2021-02-10T11:39:13.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:13.702 7f8b76662700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b701b0cc0 con 0x7f8b700d1c80 2021-02-10T11:39:13.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:13.702 7f8b76662700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b70109a70 con 0x7f8b7010b000 2021-02-10T11:39:13.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:13.702 7f8b6ffff700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f8b700d1880 0x7f8b701b04d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:53002/0 (socket says 172.21.2.112:53002) 2021-02-10T11:39:13.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:13.702 7f8b6ffff700 1 -- 172.21.2.112:0/2089697889 learned_addr learned my addr 172.21.2.112:0/2089697889 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:39:13.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:13.702 7f8b6effd700 1 -- 172.21.2.112:0/2089697889 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 202094656 0 0) 0x7f8b60001180 con 0x7f8b700d1880 2021-02-10T11:39:13.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:13.702 7f8b6effd700 1 -- 172.21.2.112:0/2089697889 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b5c002fe0 con 0x7f8b700d1880 2021-02-10T11:39:13.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:13.702 7f8b74c7f700 1 --2- 172.21.2.112:0/2089697889 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8b7010b000 0x7f8b700cf760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:13.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:13.702 7f8b6effd700 1 -- 172.21.2.112:0/2089697889 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2550236368 0 0) 0x7f8b60001180 con 0x7f8b700d1880 2021-02-10T11:39:13.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:13.702 7f8b6effd700 1 -- 172.21.2.112:0/2089697889 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8b7010b000 msgr2=0x7f8b700cf760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:13.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:13.702 7f8b6effd700 1 --2- 172.21.2.112:0/2089697889 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8b7010b000 0x7f8b700cf760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:13.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:13.702 7f8b6effd700 1 -- 172.21.2.112:0/2089697889 >> v1:172.21.2.120:6789/0 conn(0x7f8b700d1c80 legacy=0x7f8b700cef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:13.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:13.702 7f8b6effd700 1 -- 172.21.2.112:0/2089697889 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b700d3370 con 0x7f8b700d1880 2021-02-10T11:39:13.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:13.702 7f8b6effd700 1 -- 172.21.2.112:0/2089697889 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f8b60001680 con 0x7f8b700d1880 2021-02-10T11:39:13.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:13.703 7f8b6effd700 1 -- 172.21.2.112:0/2089697889 >> v1:172.21.2.112:6789/0 conn(0x7f8b700d1880 legacy=0x7f8b701b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:13.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:13.703 7f8b6effd700 1 --2- 172.21.2.112:0/2089697889 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8b5c003aa0 0x7f8b5c005f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:13.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:13.703 7f8b6effd700 1 --2- 172.21.2.112:0/2089697889 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8b5c006780 0x7f8b5c008c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:13.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:13.703 7f8b6effd700 1 --2- 172.21.2.112:0/2089697889 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8b5c0094f0 0x7f8b5c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:13.714 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:13.703 7f8b6effd700 1 -- 172.21.2.112:0/2089697889 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b5c00c2c0 con 0x7f8b5c003aa0 2021-02-10T11:39:13.714 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:13.703 7f8b6effd700 1 -- 172.21.2.112:0/2089697889 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b5c00c4e0 con 0x7f8b5c0094f0 2021-02-10T11:39:13.714 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:13.703 7f8b6effd700 1 -- 172.21.2.112:0/2089697889 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b5c050710 con 0x7f8b5c006780 2021-02-10T11:39:13.714 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:13.703 7f8b6ffff700 1 --2- 172.21.2.112:0/2089697889 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8b5c003aa0 0x7f8b5c005f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:13.714 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:13.703 7f8b6f7fe700 1 --2- 172.21.2.112:0/2089697889 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8b5c006780 0x7f8b5c008c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:13.714 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:13.703 7f8b74c7f700 1 --2- 172.21.2.112:0/2089697889 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8b5c0094f0 0x7f8b5c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:13.714 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:13.703 7f8b6ffff700 1 -- 172.21.2.112:0/2089697889 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8b5c006780 msgr2=0x7f8b5c008c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:13.714 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:13.703 7f8b6ffff700 1 --2- 172.21.2.112:0/2089697889 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8b5c006780 0x7f8b5c008c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:13.714 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:13.703 7f8b6ffff700 1 -- 172.21.2.112:0/2089697889 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8b5c0094f0 msgr2=0x7f8b5c00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:13.714 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:13.703 7f8b6ffff700 1 --2- 172.21.2.112:0/2089697889 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8b5c0094f0 0x7f8b5c00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:13.714 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:13.703 7f8b6ffff700 1 -- 172.21.2.112:0/2089697889 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b5c050970 con 0x7f8b5c003aa0 2021-02-10T11:39:13.714 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:13.703 7f8b6ffff700 1 --2- 172.21.2.112:0/2089697889 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8b5c003aa0 0x7f8b5c005f60 secure :-1 s=READY pgs=1880 cs=0 l=1 rev1=1 rx=0x7f8b600014e0 tx=0x7f8b60007d20).ready entity=mon.0 client_cookie=a6eb0b3c86b3ed65 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:13.714 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:13.704 7f8b6effd700 1 -- 172.21.2.112:0/2089697889 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8b600070c0 con 0x7f8b5c003aa0 2021-02-10T11:39:13.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:13.704 7f8b6effd700 1 -- 172.21.2.112:0/2089697889 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b600043a0 con 0x7f8b5c003aa0 2021-02-10T11:39:13.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:13.704 7f8b6effd700 1 -- 172.21.2.112:0/2089697889 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8b600048c0 con 0x7f8b5c003aa0 2021-02-10T11:39:13.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:13.704 7f8b76662700 1 -- 172.21.2.112:0/2089697889 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8b5c003aa0 msgr2=0x7f8b5c005f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:13.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:13.704 7f8b76662700 1 --2- 172.21.2.112:0/2089697889 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8b5c003aa0 0x7f8b5c005f60 secure :-1 s=READY pgs=1880 cs=0 l=1 rev1=1 rx=0x7f8b600014e0 tx=0x7f8b60007d20).stop 2021-02-10T11:39:13.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:13.704 7f8b74c7f700 1 -- 172.21.2.112:0/2089697889 reap_dead start 2021-02-10T11:39:13.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:13.704 7f8b76662700 1 -- 172.21.2.112:0/2089697889 shutdown_connections 2021-02-10T11:39:13.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:13.704 7f8b76662700 1 -- 172.21.2.112:0/2089697889 >> 172.21.2.112:0/2089697889 conn(0x7f8b700ce430 msgr2=0x7f8b700d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:13.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:13.704 7f8b76662700 1 -- 172.21.2.112:0/2089697889 shutdown_connections 2021-02-10T11:39:13.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:13.704 7f8b76662700 1 -- 172.21.2.112:0/2089697889 wait complete. 2021-02-10T11:39:13.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:13.719 7f8b76662700 1 Processor -- start 2021-02-10T11:39:13.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:13.719 7f8b76662700 1 -- start start 2021-02-10T11:39:13.717 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:13.719 7f8b76662700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8b700913d0 0x7f8b701a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:13.717 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:13.719 7f8b76662700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8b701a3c30 0x7f8b700b1170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:13.717 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:13.720 7f8b76662700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8b700b1790 0x7f8b700b1b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:13.717 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:13.720 7f8b76662700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b70091850 con 0x7f8b700b1790 2021-02-10T11:39:13.717 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:13.720 7f8b76662700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b700b21b0 con 0x7f8b700913d0 2021-02-10T11:39:13.717 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:13.720 7f8b76662700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b700b23d0 con 0x7f8b701a3c30 2021-02-10T11:39:13.718 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:13.720 7f8b74c7f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8b700b1790 0x7f8b700b1b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:13.718 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:13.720 7f8b74c7f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8b700b1790 0x7f8b700b1b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:53556/0 (socket says 172.21.2.112:53556) 2021-02-10T11:39:13.718 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:13.720 7f8b74c7f700 1 -- 172.21.2.112:0/803257566 learned_addr learned my addr 172.21.2.112:0/803257566 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:39:13.718 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:13.720 7f8b6ffff700 1 --2- 172.21.2.112:0/803257566 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8b700913d0 0x7f8b701a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:13.718 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:13.720 7f8b6f7fe700 1 --2- 172.21.2.112:0/803257566 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8b701a3c30 0x7f8b700b1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:13.718 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:13.720 7f8b74c7f700 1 -- 172.21.2.112:0/803257566 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8b701a3c30 msgr2=0x7f8b700b1170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:13.718 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:13.720 7f8b74c7f700 1 --2- 172.21.2.112:0/803257566 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8b701a3c30 0x7f8b700b1170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:13.718 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:13.720 7f8b74c7f700 1 -- 172.21.2.112:0/803257566 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8b700913d0 msgr2=0x7f8b701a3610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:13.718 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:13.720 7f8b74c7f700 1 --2- 172.21.2.112:0/803257566 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8b700913d0 0x7f8b701a3610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:13.718 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:13.720 7f8b74c7f700 1 -- 172.21.2.112:0/803257566 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b700af2a0 con 0x7f8b700b1790 2021-02-10T11:39:13.718 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:13.720 7f8b6ffff700 1 --2- 172.21.2.112:0/803257566 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8b700913d0 0x7f8b701a3610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:39:13.718 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:13.720 7f8b6f7fe700 1 --2- 172.21.2.112:0/803257566 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8b701a3c30 0x7f8b700b1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:39:13.718 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:13.721 7f8b74c7f700 1 --2- 172.21.2.112:0/803257566 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8b700b1790 0x7f8b700b1b90 secure :-1 s=READY pgs=1881 cs=0 l=1 rev1=1 rx=0x7f8b6800a940 tx=0x7f8b6800ace0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:13.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:13.721 7f8b6cff9700 1 -- 172.21.2.112:0/803257566 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8b680066f0 con 0x7f8b700b1790 2021-02-10T11:39:13.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:13.721 7f8b6cff9700 1 -- 172.21.2.112:0/803257566 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b68003030 con 0x7f8b700b1790 2021-02-10T11:39:13.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:13.721 7f8b6cff9700 1 -- 172.21.2.112:0/803257566 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8b680015f0 con 0x7f8b700b1790 2021-02-10T11:39:13.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:13.721 7f8b76662700 1 -- 172.21.2.112:0/803257566 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b700af2a0 con 0x7f8b700b1790 2021-02-10T11:39:13.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:13.721 7f8b76662700 1 -- 172.21.2.112:0/803257566 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b700af830 con 0x7f8b700b1790 2021-02-10T11:39:13.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:13.722 7f8b6cff9700 1 -- 172.21.2.112:0/803257566 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8b680011f0 con 0x7f8b700b1790 2021-02-10T11:39:13.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:13.722 7f8b6cff9700 1 --2- 172.21.2.112:0/803257566 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8b540186c0 0x7f8b5401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:13.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:13.722 7f8b6cff9700 1 -- 172.21.2.112:0/803257566 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f8b68032910 con 0x7f8b700b1790 2021-02-10T11:39:13.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:13.722 7f8b6ffff700 1 --2- 172.21.2.112:0/803257566 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8b540186c0 0x7f8b5401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:13.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:13.722 7f8b6ffff700 1 --2- 172.21.2.112:0/803257566 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8b540186c0 0x7f8b5401abc0 crc :-1 s=READY pgs=883 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:13.744 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:13.751 7f8b76662700 1 -- 172.21.2.112:0/803257566 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b70082aa0 con 0x7f8b700b1790 2021-02-10T11:39:13.747 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:13.754 7f8b6cff9700 1 -- 172.21.2.112:0/803257566 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8b680056b0 con 0x7f8b700b1790 2021-02-10T11:39:13.883 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:13.891 7f8b76662700 1 -- 172.21.2.112:0/803257566 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f8b70082aa0 con 0x7f8b700b1790 2021-02-10T11:39:13.938 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:13.945 7f8b6cff9700 1 -- 172.21.2.112:0/803257566 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v73) v1 ==== 136+0+0 (secure 0 0 0) 0x7f8b6801f8c0 con 0x7f8b700b1790 2021-02-10T11:39:13.950 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:13.957 7f8b76662700 1 -- 172.21.2.112:0/803257566 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8b540186c0 msgr2=0x7f8b5401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:13.951 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:13.957 7f8b76662700 1 --2- 172.21.2.112:0/803257566 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8b540186c0 0x7f8b5401abc0 crc :-1 s=READY pgs=883 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:13.951 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:13.957 7f8b76662700 1 -- 172.21.2.112:0/803257566 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8b700b1790 msgr2=0x7f8b700b1b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:13.951 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:13.957 7f8b76662700 1 --2- 172.21.2.112:0/803257566 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8b700b1790 0x7f8b700b1b90 secure :-1 s=READY pgs=1881 cs=0 l=1 rev1=1 rx=0x7f8b6800a940 tx=0x7f8b6800ace0).stop 2021-02-10T11:39:13.951 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:13.958 7f8b76662700 1 -- 172.21.2.112:0/803257566 shutdown_connections 2021-02-10T11:39:13.951 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:13.958 7f8b76662700 1 --2- 172.21.2.112:0/803257566 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8b540186c0 0x7f8b5401abc0 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:13.951 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:13.958 7f8b76662700 1 --2- 172.21.2.112:0/803257566 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8b700b1790 0x7f8b700b1b90 unknown :-1 s=CLOSED pgs=1881 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:13.951 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:13.958 7f8b76662700 1 --2- 172.21.2.112:0/803257566 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8b701a3c30 0x7f8b700b1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:13.951 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:13.958 7f8b76662700 1 --2- 172.21.2.112:0/803257566 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8b700913d0 0x7f8b701a3610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:13.951 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:13.958 7f8b76662700 1 -- 172.21.2.112:0/803257566 >> 172.21.2.112:0/803257566 conn(0x7f8b70087360 msgr2=0x7f8b700b5980 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:13.951 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:13.958 7f8b76662700 1 -- 172.21.2.112:0/803257566 shutdown_connections 2021-02-10T11:39:13.951 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:13.958 7f8b76662700 1 -- 172.21.2.112:0/803257566 wait complete. 2021-02-10T11:39:13.958 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs volume ls 2021-02-10T11:39:14.385 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:14.376 7fa21da17700 1 Processor -- start 2021-02-10T11:39:14.385 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:14.376 7fa21da17700 1 -- start start 2021-02-10T11:39:14.385 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:14.376 7fa21da17700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fa2180d0340 0x7fa2180ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:14.385 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:14.376 7fa21da17700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fa2180d1c70 0x7fa2180efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:14.385 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:14.376 7fa21da17700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2180f0b10 con 0x7fa2180d1870 2021-02-10T11:39:14.386 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:14.376 7fa21da17700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2180e5ed0 con 0x7fa2180d1c70 2021-02-10T11:39:14.386 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:14.376 7fa21da17700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2181b0be0 con 0x7fa2180d0340 2021-02-10T11:39:14.386 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:14.377 7fa217fff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fa2180d1c70 0x7fa2180efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:14.386 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:14.377 7fa217fff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fa2180d1c70 0x7fa2180efc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:53560/0 (socket says 172.21.2.112:53560) 2021-02-10T11:39:14.386 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:14.377 7fa217fff700 1 -- 172.21.2.112:0/2330156332 learned_addr learned my addr 172.21.2.112:0/2330156332 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:39:14.386 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:14.377 7fa217fff700 1 -- 172.21.2.112:0/2330156332 >> v2:172.21.2.120:3300/0 conn(0x7fa2180d0340 msgr2=0x7fa2180ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:14.386 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:14.377 7fa217fff700 1 --2- 172.21.2.112:0/2330156332 >> v2:172.21.2.120:3300/0 conn(0x7fa2180d0340 0x7fa2180ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:14.386 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:14.377 7fa217fff700 1 -- 172.21.2.112:0/2330156332 >> v1:172.21.2.120:6789/0 conn(0x7fa2180d1870 legacy=0x7fa2181b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:14.386 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:14.377 7fa217fff700 1 -- 172.21.2.112:0/2330156332 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2180f20a0 con 0x7fa2180d1c70 2021-02-10T11:39:14.386 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:14.377 7fa217fff700 1 --2- 172.21.2.112:0/2330156332 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa2180d1c70 0x7fa2180efc60 secure :-1 s=READY pgs=1882 cs=0 l=1 rev1=1 rx=0x7fa210006f00 tx=0x7fa210009f10).ready entity=mon.0 client_cookie=cb763af2505a4124 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:14.386 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:14.377 7fa2167fc700 1 -- 172.21.2.112:0/2330156332 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa21000b3b0 con 0x7fa2180d1c70 2021-02-10T11:39:14.386 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:14.377 7fa2167fc700 1 -- 172.21.2.112:0/2330156332 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa21000b670 con 0x7fa2180d1c70 2021-02-10T11:39:14.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:14.377 7fa2167fc700 1 -- 172.21.2.112:0/2330156332 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa21000bb90 con 0x7fa2180d1c70 2021-02-10T11:39:14.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:14.377 7fa21da17700 1 -- 172.21.2.112:0/2330156332 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa2180d1c70 msgr2=0x7fa2180efc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:14.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:14.377 7fa21da17700 1 --2- 172.21.2.112:0/2330156332 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa2180d1c70 0x7fa2180efc60 secure :-1 s=READY pgs=1882 cs=0 l=1 rev1=1 rx=0x7fa210006f00 tx=0x7fa210009f10).stop 2021-02-10T11:39:14.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:14.378 7fa21da17700 1 -- 172.21.2.112:0/2330156332 shutdown_connections 2021-02-10T11:39:14.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:14.378 7fa21da17700 1 --2- 172.21.2.112:0/2330156332 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa2180d1c70 0x7fa2180efc60 unknown :-1 s=CLOSED pgs=1882 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:14.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:14.378 7fa21da17700 1 --2- 172.21.2.112:0/2330156332 >> v2:172.21.2.120:3300/0 conn(0x7fa2180d0340 0x7fa2180ef640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:14.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:14.378 7fa21da17700 1 -- 172.21.2.112:0/2330156332 >> 172.21.2.112:0/2330156332 conn(0x7fa2180ce450 msgr2=0x7fa2180d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:14.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:14.378 7fa21da17700 1 -- 172.21.2.112:0/2330156332 shutdown_connections 2021-02-10T11:39:14.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:14.378 7fa21da17700 1 -- 172.21.2.112:0/2330156332 wait complete. 2021-02-10T11:39:14.389 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:14.393 7fa21da17700 1 Processor -- start 2021-02-10T11:39:14.389 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:14.393 7fa21da17700 1 -- start start 2021-02-10T11:39:14.389 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:14.393 7fa21da17700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa21818b550 0x7fa21818b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:14.389 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:14.393 7fa21da17700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa2181a3530 0x7fa2181a3930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:14.389 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:14.393 7fa21da17700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa2180c1540 0x7fa2180c1940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:14.389 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:14.393 7fa21da17700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2180b5560 con 0x7fa2181a3530 2021-02-10T11:39:14.389 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:14.393 7fa21da17700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2180c1f60 con 0x7fa21818b550 2021-02-10T11:39:14.389 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:14.393 7fa21da17700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2180c2180 con 0x7fa2180c1540 2021-02-10T11:39:14.389 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:14.393 7fa216ffd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa2181a3530 0x7fa2181a3930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:14.389 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:14.393 7fa216ffd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa2181a3530 0x7fa2181a3930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:53570/0 (socket says 172.21.2.112:53570) 2021-02-10T11:39:14.390 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:14.393 7fa216ffd700 1 -- 172.21.2.112:0/1538499294 learned_addr learned my addr 172.21.2.112:0/1538499294 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:39:14.390 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:14.394 7fa216ffd700 1 -- 172.21.2.112:0/1538499294 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa2180c1540 msgr2=0x7fa2180c1940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:14.390 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:14.394 7fa217fff700 1 --2- 172.21.2.112:0/1538499294 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa2180c1540 0x7fa2180c1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:14.390 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:14.394 7fa216ffd700 1 --2- 172.21.2.112:0/1538499294 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa2180c1540 0x7fa2180c1940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:14.390 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:14.394 7fa216ffd700 1 -- 172.21.2.112:0/1538499294 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa21818b550 msgr2=0x7fa21818b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:14.390 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:14.394 7fa216ffd700 1 --2- 172.21.2.112:0/1538499294 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa21818b550 0x7fa21818b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:14.390 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:14.394 7fa216ffd700 1 -- 172.21.2.112:0/1538499294 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2180c23a0 con 0x7fa2181a3530 2021-02-10T11:39:14.390 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:14.394 7fa216ffd700 1 --2- 172.21.2.112:0/1538499294 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa2181a3530 0x7fa2181a3930 secure :-1 s=READY pgs=1883 cs=0 l=1 rev1=1 rx=0x7fa200006f00 tx=0x7fa20000b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:14.390 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:14.394 7fa20ffff700 1 -- 172.21.2.112:0/1538499294 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa200006b90 con 0x7fa2181a3530 2021-02-10T11:39:14.390 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:14.394 7fa20ffff700 1 -- 172.21.2.112:0/1538499294 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa20000bd30 con 0x7fa2181a3530 2021-02-10T11:39:14.390 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:14.394 7fa20ffff700 1 -- 172.21.2.112:0/1538499294 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa2000020d0 con 0x7fa2181a3530 2021-02-10T11:39:14.390 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:14.394 7fa21da17700 1 -- 172.21.2.112:0/1538499294 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2180c23a0 con 0x7fa2181a3530 2021-02-10T11:39:14.391 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:14.394 7fa21da17700 1 -- 172.21.2.112:0/1538499294 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2180c23a0 con 0x7fa2181a3530 2021-02-10T11:39:14.391 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:14.395 7fa20ffff700 1 -- 172.21.2.112:0/1538499294 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa200007000 con 0x7fa2181a3530 2021-02-10T11:39:14.391 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:14.395 7fa20ffff700 1 --2- 172.21.2.112:0/1538499294 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa1fc0186d0 0x7fa1fc01abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:14.391 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:14.395 7fa20ffff700 1 -- 172.21.2.112:0/1538499294 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fa200032790 con 0x7fa2181a3530 2021-02-10T11:39:14.391 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:14.395 7fa2177fe700 1 --2- 172.21.2.112:0/1538499294 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa1fc0186d0 0x7fa1fc01abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:14.391 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:14.396 7fa2177fe700 1 --2- 172.21.2.112:0/1538499294 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa1fc0186d0 0x7fa1fc01abd0 crc :-1 s=READY pgs=884 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:14.419 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:14.426 7fa21da17700 1 -- 172.21.2.112:0/1538499294 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2180ee0a0 con 0x7fa2181a3530 2021-02-10T11:39:14.421 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:14.429 7fa20ffff700 1 -- 172.21.2.112:0/1538499294 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa200006350 con 0x7fa2181a3530 2021-02-10T11:39:14.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:14.564 7fa21da17700 1 -- 172.21.2.112:0/1538499294 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs volume ls", "target": ["mgr", ""]}) v1 -- 0x7fa2180ce450 con 0x7fa1fc0186d0 2021-02-10T11:39:14.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:14.565 7fa20ffff700 1 -- 172.21.2.112:0/1538499294 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+41 (crc 0 0 0) 0x7fa208006a10 con 0x7fa1fc0186d0 2021-02-10T11:39:14.560 INFO:teuthology.orchestra.run.gibba012.stdout:[ 2021-02-10T11:39:14.560 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:39:14.560 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "cephfs" 2021-02-10T11:39:14.560 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:39:14.560 INFO:teuthology.orchestra.run.gibba012.stdout:] 2021-02-10T11:39:14.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:14.568 7fa21da17700 1 -- 172.21.2.112:0/1538499294 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa1fc0186d0 msgr2=0x7fa1fc01abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:14.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:14.568 7fa21da17700 1 --2- 172.21.2.112:0/1538499294 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa1fc0186d0 0x7fa1fc01abd0 crc :-1 s=READY pgs=884 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:14.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:14.568 7fa21da17700 1 -- 172.21.2.112:0/1538499294 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa2181a3530 msgr2=0x7fa2181a3930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:14.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:14.568 7fa21da17700 1 --2- 172.21.2.112:0/1538499294 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa2181a3530 0x7fa2181a3930 secure :-1 s=READY pgs=1883 cs=0 l=1 rev1=1 rx=0x7fa200006f00 tx=0x7fa20000b940).stop 2021-02-10T11:39:14.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:14.568 7fa21da17700 1 -- 172.21.2.112:0/1538499294 shutdown_connections 2021-02-10T11:39:14.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:14.568 7fa21da17700 1 --2- 172.21.2.112:0/1538499294 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa1fc0186d0 0x7fa1fc01abd0 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:14.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:14.568 7fa21da17700 1 --2- 172.21.2.112:0/1538499294 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa2180c1540 0x7fa2180c1940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:14.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:14.568 7fa21da17700 1 --2- 172.21.2.112:0/1538499294 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa2181a3530 0x7fa2181a3930 unknown :-1 s=CLOSED pgs=1883 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:14.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:14.568 7fa21da17700 1 --2- 172.21.2.112:0/1538499294 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa21818b550 0x7fa21818b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:14.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:14.568 7fa21da17700 1 -- 172.21.2.112:0/1538499294 >> 172.21.2.112:0/1538499294 conn(0x7fa2180a6870 msgr2=0x7fa21808a420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:14.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:14.569 7fa21da17700 1 -- 172.21.2.112:0/1538499294 shutdown_connections 2021-02-10T11:39:14.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:14.569 7fa21da17700 1 -- 172.21.2.112:0/1538499294 wait complete. 2021-02-10T11:39:14.570 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:39:15.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:14.992 7fa6e3ed0700 1 Processor -- start 2021-02-10T11:39:15.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:14.993 7fa6e3ed0700 1 -- start start 2021-02-10T11:39:15.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:14.993 7fa6e3ed0700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fa6dc10aff0 0x7fa6dc0cf750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:15.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:14.993 7fa6e3ed0700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6dc0fe1c0 con 0x7fa6dc0d1870 2021-02-10T11:39:15.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:14.993 7fa6e3ed0700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6dc1b0cb0 con 0x7fa6dc0d1c70 2021-02-10T11:39:15.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:14.993 7fa6e3ed0700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6dc109a60 con 0x7fa6dc10aff0 2021-02-10T11:39:15.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:14.993 7fa6e1cec700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fa6dc0d1870 0x7fa6dc1b04c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:53036/0 (socket says 172.21.2.112:53036) 2021-02-10T11:39:15.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:14.993 7fa6e1cec700 1 -- 172.21.2.112:0/3265611617 learned_addr learned my addr 172.21.2.112:0/3265611617 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:39:15.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:14.993 7fa6e24ed700 1 --2- 172.21.2.112:0/3265611617 >> v2:172.21.2.112:3300/0 conn(0x7fa6dc10aff0 0x7fa6dc0cf750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:15.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:14.994 7fa6e0cea700 1 -- 172.21.2.112:0/3265611617 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1826099998 0 0) 0x7fa6d8000ce0 con 0x7fa6dc0d1870 2021-02-10T11:39:15.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:14.994 7fa6e0cea700 1 -- 172.21.2.112:0/3265611617 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6c8002fe0 con 0x7fa6dc0d1870 2021-02-10T11:39:15.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:14.994 7fa6e0cea700 1 -- 172.21.2.112:0/3265611617 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3368271208 0 0) 0x7fa6cc000ff0 con 0x7fa6dc0d1c70 2021-02-10T11:39:15.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:14.994 7fa6e0cea700 1 -- 172.21.2.112:0/3265611617 --> v1:172.21.2.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6c80042c0 con 0x7fa6dc0d1c70 2021-02-10T11:39:15.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:14.994 7fa6e24ed700 1 -- 172.21.2.112:0/3265611617 >> v1:172.21.2.120:6789/0 conn(0x7fa6dc0d1c70 legacy=0x7fa6dc0cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:15.005 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:14.994 7fa6e24ed700 1 -- 172.21.2.112:0/3265611617 >> v1:172.21.2.112:6789/0 conn(0x7fa6dc0d1870 legacy=0x7fa6dc1b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:15.005 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:14.994 7fa6e24ed700 1 -- 172.21.2.112:0/3265611617 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6dc10b3f0 con 0x7fa6dc10aff0 2021-02-10T11:39:15.005 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:14.994 7fa6e24ed700 1 --2- 172.21.2.112:0/3265611617 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa6dc10aff0 0x7fa6dc0cf750 secure :-1 s=READY pgs=1885 cs=0 l=1 rev1=1 rx=0x7fa6d4006e30 tx=0x7fa6d4009de0).ready entity=mon.0 client_cookie=7ce4d98035756900 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:15.005 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:14.994 7fa6e0cea700 1 -- 172.21.2.112:0/3265611617 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa6d400b300 con 0x7fa6dc10aff0 2021-02-10T11:39:15.005 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:14.994 7fa6e0cea700 1 -- 172.21.2.112:0/3265611617 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6d400b580 con 0x7fa6dc10aff0 2021-02-10T11:39:15.005 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:14.994 7fa6e0cea700 1 -- 172.21.2.112:0/3265611617 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa6d400baa0 con 0x7fa6dc10aff0 2021-02-10T11:39:15.005 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:14.994 7fa6e3ed0700 1 -- 172.21.2.112:0/3265611617 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa6dc10aff0 msgr2=0x7fa6dc0cf750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:15.005 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:14.994 7fa6e3ed0700 1 --2- 172.21.2.112:0/3265611617 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa6dc10aff0 0x7fa6dc0cf750 secure :-1 s=READY pgs=1885 cs=0 l=1 rev1=1 rx=0x7fa6d4006e30 tx=0x7fa6d4009de0).stop 2021-02-10T11:39:15.005 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:14.995 7fa6e3ed0700 1 -- 172.21.2.112:0/3265611617 shutdown_connections 2021-02-10T11:39:15.005 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:14.995 7fa6e3ed0700 1 --2- 172.21.2.112:0/3265611617 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa6dc10aff0 0x7fa6dc0cf750 unknown :-1 s=CLOSED pgs=1885 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:15.005 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:14.995 7fa6e3ed0700 1 -- 172.21.2.112:0/3265611617 >> 172.21.2.112:0/3265611617 conn(0x7fa6dc0ce420 msgr2=0x7fa6dc0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:15.005 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:14.995 7fa6e3ed0700 1 -- 172.21.2.112:0/3265611617 shutdown_connections 2021-02-10T11:39:15.005 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:14.995 7fa6e3ed0700 1 -- 172.21.2.112:0/3265611617 wait complete. 2021-02-10T11:39:15.006 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.009 7fa6e3ed0700 1 Processor -- start 2021-02-10T11:39:15.006 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.009 7fa6e3ed0700 1 -- start start 2021-02-10T11:39:15.006 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.010 7fa6e3ed0700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa6dc191b10 0x7fa6dc0ae640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:15.006 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.010 7fa6e3ed0700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa6dc191f10 0x7fa6dc19ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:15.006 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.010 7fa6e3ed0700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa6dc1a05f0 0x7fa6dc082ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:15.006 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.010 7fa6e3ed0700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6dc192440 con 0x7fa6dc191f10 2021-02-10T11:39:15.006 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.010 7fa6e3ed0700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6dc1916e0 con 0x7fa6dc191b10 2021-02-10T11:39:15.006 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.010 7fa6e3ed0700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6dc191900 con 0x7fa6dc1a05f0 2021-02-10T11:39:15.006 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.010 7fa6e14eb700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa6dc191f10 0x7fa6dc19ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:15.006 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.010 7fa6e14eb700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa6dc191f10 0x7fa6dc19ffd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:53580/0 (socket says 172.21.2.112:53580) 2021-02-10T11:39:15.006 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.010 7fa6e14eb700 1 -- 172.21.2.112:0/3780242268 learned_addr learned my addr 172.21.2.112:0/3780242268 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:39:15.006 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.010 7fa6e24ed700 1 --2- 172.21.2.112:0/3780242268 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa6dc1a05f0 0x7fa6dc082ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:15.006 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.010 7fa6e1cec700 1 --2- 172.21.2.112:0/3780242268 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa6dc191b10 0x7fa6dc0ae640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:15.007 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.010 7fa6e14eb700 1 -- 172.21.2.112:0/3780242268 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa6dc1a05f0 msgr2=0x7fa6dc082ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:15.007 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.010 7fa6e14eb700 1 --2- 172.21.2.112:0/3780242268 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa6dc1a05f0 0x7fa6dc082ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:15.007 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.010 7fa6e14eb700 1 -- 172.21.2.112:0/3780242268 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa6dc191b10 msgr2=0x7fa6dc0ae640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:15.007 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.010 7fa6e14eb700 1 --2- 172.21.2.112:0/3780242268 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa6dc191b10 0x7fa6dc0ae640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:15.007 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.010 7fa6e14eb700 1 -- 172.21.2.112:0/3780242268 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6dc0834f0 con 0x7fa6dc191f10 2021-02-10T11:39:15.007 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.010 7fa6e24ed700 1 --2- 172.21.2.112:0/3780242268 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa6dc1a05f0 0x7fa6dc082ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:39:15.008 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.011 7fa6e14eb700 1 --2- 172.21.2.112:0/3780242268 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa6dc191f10 0x7fa6dc19ffd0 secure :-1 s=READY pgs=1886 cs=0 l=1 rev1=1 rx=0x7fa6cc006f20 tx=0x7fa6cc00b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:15.008 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.011 7fa6d27fc700 1 -- 172.21.2.112:0/3780242268 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa6cc00bd50 con 0x7fa6dc191f10 2021-02-10T11:39:15.008 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.011 7fa6d27fc700 1 -- 172.21.2.112:0/3780242268 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6cc001070 con 0x7fa6dc191f10 2021-02-10T11:39:15.008 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.011 7fa6d27fc700 1 -- 172.21.2.112:0/3780242268 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa6cc001590 con 0x7fa6dc191f10 2021-02-10T11:39:15.008 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.011 7fa6e3ed0700 1 -- 172.21.2.112:0/3780242268 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6dc0834f0 con 0x7fa6dc191f10 2021-02-10T11:39:15.008 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.011 7fa6e3ed0700 1 -- 172.21.2.112:0/3780242268 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6dc0834f0 con 0x7fa6dc191f10 2021-02-10T11:39:15.008 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.012 7fa6d27fc700 1 -- 172.21.2.112:0/3780242268 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa6cc00a000 con 0x7fa6dc191f10 2021-02-10T11:39:15.009 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.012 7fa6d27fc700 1 --2- 172.21.2.112:0/3780242268 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa6c40186c0 0x7fa6c401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:15.009 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.012 7fa6d27fc700 1 -- 172.21.2.112:0/3780242268 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fa6cc0325e0 con 0x7fa6dc191f10 2021-02-10T11:39:15.009 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.012 7fa6e1cec700 1 --2- 172.21.2.112:0/3780242268 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa6c40186c0 0x7fa6c401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:15.009 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.012 7fa6e1cec700 1 --2- 172.21.2.112:0/3780242268 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa6c40186c0 0x7fa6c401abc0 crc :-1 s=READY pgs=885 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:15.036 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.043 7fa6e3ed0700 1 -- 172.21.2.112:0/3780242268 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6dc191900 con 0x7fa6dc191f10 2021-02-10T11:39:15.038 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.045 7fa6d27fc700 1 -- 172.21.2.112:0/3780242268 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa6cc006c10 con 0x7fa6dc191f10 2021-02-10T11:39:15.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.195 7fa6e3ed0700 1 -- 172.21.2.112:0/3780242268 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fa6dc191900 con 0x7fa6dc191f10 2021-02-10T11:39:15.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.195 7fa6d27fc700 1 -- 172.21.2.112:0/3780242268 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v24)=0 v24) v1 ==== 122+0+0 (secure 0 0 0) 0x7fa6cc00a000 con 0x7fa6dc191f10 2021-02-10T11:39:15.190 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.197 7fa6e3ed0700 1 -- 172.21.2.112:0/3780242268 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa6c40186c0 msgr2=0x7fa6c401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:15.190 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.197 7fa6e3ed0700 1 --2- 172.21.2.112:0/3780242268 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa6c40186c0 0x7fa6c401abc0 crc :-1 s=READY pgs=885 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:15.190 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.197 7fa6e3ed0700 1 -- 172.21.2.112:0/3780242268 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa6dc191f10 msgr2=0x7fa6dc19ffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:15.190 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.197 7fa6e3ed0700 1 --2- 172.21.2.112:0/3780242268 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa6dc191f10 0x7fa6dc19ffd0 secure :-1 s=READY pgs=1886 cs=0 l=1 rev1=1 rx=0x7fa6cc006f20 tx=0x7fa6cc00b960).stop 2021-02-10T11:39:15.190 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.197 7fa6e3ed0700 1 -- 172.21.2.112:0/3780242268 shutdown_connections 2021-02-10T11:39:15.190 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.197 7fa6e3ed0700 1 --2- 172.21.2.112:0/3780242268 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa6c40186c0 0x7fa6c401abc0 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:15.190 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.197 7fa6e3ed0700 1 --2- 172.21.2.112:0/3780242268 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa6dc1a05f0 0x7fa6dc082ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:15.190 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.197 7fa6e3ed0700 1 --2- 172.21.2.112:0/3780242268 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa6dc191f10 0x7fa6dc19ffd0 unknown :-1 s=CLOSED pgs=1886 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:15.190 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.197 7fa6e3ed0700 1 --2- 172.21.2.112:0/3780242268 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa6dc191b10 0x7fa6dc0ae640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:15.190 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.197 7fa6e3ed0700 1 -- 172.21.2.112:0/3780242268 >> 172.21.2.112:0/3780242268 conn(0x7fa6dc0a02d0 msgr2=0x7fa6dc09f200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:15.191 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.197 7fa6e3ed0700 1 -- 172.21.2.112:0/3780242268 shutdown_connections 2021-02-10T11:39:15.191 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.197 7fa6e3ed0700 1 -- 172.21.2.112:0/3780242268 wait complete. 2021-02-10T11:39:15.199 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolumegroup create cephfs group_0000000000143913 2021-02-10T11:39:15.630 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.620 7f920475f700 1 Processor -- start 2021-02-10T11:39:15.630 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.620 7f920475f700 1 -- start start 2021-02-10T11:39:15.630 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.620 7f920475f700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f91fc0d1c60 0x7f91fc0cef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:15.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.620 7f920475f700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91fc0d0160 con 0x7f91fc0cf570 2021-02-10T11:39:15.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.620 7f920475f700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91fc0fe1b0 con 0x7f91fc0d1860 2021-02-10T11:39:15.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.620 7f920475f700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91fc1b0ca0 con 0x7f91fc0d1c60 2021-02-10T11:39:15.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.621 7f9202d7c700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f91fc0cf570 0x7f91fc0cf970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:53054/0 (socket says 172.21.2.112:53054) 2021-02-10T11:39:15.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.621 7f9202d7c700 1 -- 172.21.2.112:0/401233167 learned_addr learned my addr 172.21.2.112:0/401233167 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:39:15.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.621 7f9201579700 1 -- 172.21.2.112:0/401233167 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1404988522 0 0) 0x7f91f4000f90 con 0x7f91fc0cf570 2021-02-10T11:39:15.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.621 7f9201579700 1 -- 172.21.2.112:0/401233167 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91e8002fe0 con 0x7f91fc0cf570 2021-02-10T11:39:15.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.621 7f9201579700 1 -- 172.21.2.112:0/401233167 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2627677889 0 0) 0x7f91f4001460 con 0x7f91fc0cf570 2021-02-10T11:39:15.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.621 7f9201579700 1 -- 172.21.2.112:0/401233167 >> v2:172.21.2.120:3300/0 conn(0x7f91fc0d1c60 msgr2=0x7f91fc0cef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:15.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.621 7f9201579700 1 --2- 172.21.2.112:0/401233167 >> v2:172.21.2.120:3300/0 conn(0x7f91fc0d1c60 0x7f91fc0cef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:15.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.621 7f9201579700 1 -- 172.21.2.112:0/401233167 >> v1:172.21.2.120:6789/0 conn(0x7f91fc0d1860 legacy=0x7f91fc1b04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:15.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.621 7f9201579700 1 -- 172.21.2.112:0/401233167 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91fc109a50 con 0x7f91fc0cf570 2021-02-10T11:39:15.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.621 7f9201579700 1 -- 172.21.2.112:0/401233167 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f91f4001980 con 0x7f91fc0cf570 2021-02-10T11:39:15.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.621 7f9201579700 1 -- 172.21.2.112:0/401233167 >> v1:172.21.2.112:6789/0 conn(0x7f91fc0cf570 legacy=0x7f91fc0cf970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:15.632 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.621 7f9201579700 1 --2- 172.21.2.112:0/401233167 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f91e8003a60 0x7f91e8005f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:15.632 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.621 7f9201579700 1 --2- 172.21.2.112:0/401233167 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f91e8006780 0x7f91e8008ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:15.632 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.621 7f9201579700 1 --2- 172.21.2.112:0/401233167 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f91e80095a0 0x7f91e800baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:15.632 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.621 7f9201579700 1 -- 172.21.2.112:0/401233167 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91e800c2f0 con 0x7f91e8003a60 2021-02-10T11:39:15.632 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.621 7f9201579700 1 -- 172.21.2.112:0/401233167 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91e800c510 con 0x7f91e80095a0 2021-02-10T11:39:15.632 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.621 7f9201579700 1 -- 172.21.2.112:0/401233167 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91e800c730 con 0x7f91e8006780 2021-02-10T11:39:15.632 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.621 7f920257b700 1 --2- 172.21.2.112:0/401233167 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f91e8003a60 0x7f91e8005f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:15.632 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.622 7f9202d7c700 1 --2- 172.21.2.112:0/401233167 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f91e80095a0 0x7f91e800baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:15.632 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.622 7f9201d7a700 1 --2- 172.21.2.112:0/401233167 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f91e8006780 0x7f91e8008ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:15.632 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.622 7f920257b700 1 -- 172.21.2.112:0/401233167 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f91e8006780 msgr2=0x7f91e8008ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:15.632 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.622 7f920257b700 1 --2- 172.21.2.112:0/401233167 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f91e8006780 0x7f91e8008ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:15.632 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.622 7f920257b700 1 -- 172.21.2.112:0/401233167 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f91e80095a0 msgr2=0x7f91e800baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:15.632 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.622 7f920257b700 1 --2- 172.21.2.112:0/401233167 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f91e80095a0 0x7f91e800baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:15.633 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.622 7f9202d7c700 1 -- 172.21.2.112:0/401233167 reap_dead start 2021-02-10T11:39:15.633 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.622 7f920257b700 1 -- 172.21.2.112:0/401233167 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91e800c9d0 con 0x7f91e8003a60 2021-02-10T11:39:15.633 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.622 7f920257b700 1 --2- 172.21.2.112:0/401233167 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f91e8003a60 0x7f91e8005f60 secure :-1 s=READY pgs=1888 cs=0 l=1 rev1=1 rx=0x7f91f0006ab0 tx=0x7f91f004fb90).ready entity=mon.0 client_cookie=ba90948e69d2a940 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:15.633 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.622 7f9201579700 1 -- 172.21.2.112:0/401233167 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f91f0006240 con 0x7f91e8003a60 2021-02-10T11:39:15.633 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.622 7f9201579700 1 -- 172.21.2.112:0/401233167 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91f0006720 con 0x7f91e8003a60 2021-02-10T11:39:15.633 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.622 7f9201579700 1 -- 172.21.2.112:0/401233167 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f91f0005690 con 0x7f91e8003a60 2021-02-10T11:39:15.633 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.622 7f920475f700 1 -- 172.21.2.112:0/401233167 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f91e8003a60 msgr2=0x7f91e8005f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:15.633 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.622 7f920475f700 1 --2- 172.21.2.112:0/401233167 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f91e8003a60 0x7f91e8005f60 secure :-1 s=READY pgs=1888 cs=0 l=1 rev1=1 rx=0x7f91f0006ab0 tx=0x7f91f004fb90).stop 2021-02-10T11:39:15.633 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.623 7f920475f700 1 -- 172.21.2.112:0/401233167 shutdown_connections 2021-02-10T11:39:15.633 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.623 7f920475f700 1 --2- 172.21.2.112:0/401233167 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f91e8003a60 0x7f91e8005f60 unknown :-1 s=CLOSED pgs=1888 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:15.633 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.623 7f920475f700 1 -- 172.21.2.112:0/401233167 >> 172.21.2.112:0/401233167 conn(0x7f91fc0ce410 msgr2=0x7f91fc0d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:15.633 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.623 7f920475f700 1 -- 172.21.2.112:0/401233167 shutdown_connections 2021-02-10T11:39:15.633 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.623 7f920475f700 1 -- 172.21.2.112:0/401233167 wait complete. 2021-02-10T11:39:15.634 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.637 7f920475f700 1 Processor -- start 2021-02-10T11:39:15.634 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.637 7f920475f700 1 -- start start 2021-02-10T11:39:15.635 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.638 7f920475f700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f91fc0badd0 0x7f91fc09eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:15.635 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.638 7f920475f700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f91fc0adf50 0x7f91fc09f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:15.635 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.638 7f920475f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f91fc1a7470 0x7f91fc1a7870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:15.635 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.638 7f920475f700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91fc09e5a0 con 0x7f91fc1a7470 2021-02-10T11:39:15.635 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.638 7f920475f700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91fc09f6e0 con 0x7f91fc0adf50 2021-02-10T11:39:15.635 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.638 7f920475f700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91fc0bb220 con 0x7f91fc0badd0 2021-02-10T11:39:15.635 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.638 7f9202d7c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f91fc1a7470 0x7f91fc1a7870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:15.635 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.638 7f9202d7c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f91fc1a7470 0x7f91fc1a7870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:53604/0 (socket says 172.21.2.112:53604) 2021-02-10T11:39:15.635 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.638 7f9202d7c700 1 -- 172.21.2.112:0/1392303777 learned_addr learned my addr 172.21.2.112:0/1392303777 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:39:15.635 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.638 7f9202d7c700 1 -- 172.21.2.112:0/1392303777 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f91fc0badd0 msgr2=0x7f91fc09eaa0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:39:15.636 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.638 7f9202d7c700 1 --2- 172.21.2.112:0/1392303777 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f91fc0badd0 0x7f91fc09eaa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:15.636 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.638 7f9202d7c700 1 -- 172.21.2.112:0/1392303777 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f91fc0adf50 msgr2=0x7f91fc09f0c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:39:15.636 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.638 7f9202d7c700 1 --2- 172.21.2.112:0/1392303777 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f91fc0adf50 0x7f91fc09f0c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:15.636 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.638 7f9202d7c700 1 -- 172.21.2.112:0/1392303777 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91fc1a7e90 con 0x7f91fc1a7470 2021-02-10T11:39:15.636 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.639 7f9202d7c700 1 --2- 172.21.2.112:0/1392303777 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f91fc1a7470 0x7f91fc1a7870 secure :-1 s=READY pgs=1889 cs=0 l=1 rev1=1 rx=0x7f91f400b910 tx=0x7f91f400bcb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:15.636 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.639 7f91eeffd700 1 -- 172.21.2.112:0/1392303777 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f91f4007d20 con 0x7f91fc1a7470 2021-02-10T11:39:15.636 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.639 7f91eeffd700 1 -- 172.21.2.112:0/1392303777 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91f400c2f0 con 0x7f91fc1a7470 2021-02-10T11:39:15.636 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.639 7f91eeffd700 1 -- 172.21.2.112:0/1392303777 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f91f400c780 con 0x7f91fc1a7470 2021-02-10T11:39:15.636 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.639 7f920475f700 1 -- 172.21.2.112:0/1392303777 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91fc1a7e90 con 0x7f91fc1a7470 2021-02-10T11:39:15.636 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.639 7f920475f700 1 -- 172.21.2.112:0/1392303777 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91fc09d1f0 con 0x7f91fc1a7470 2021-02-10T11:39:15.636 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.639 7f91eeffd700 1 -- 172.21.2.112:0/1392303777 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f91f4007cc0 con 0x7f91fc1a7470 2021-02-10T11:39:15.636 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.640 7f91eeffd700 1 --2- 172.21.2.112:0/1392303777 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f91e40186c0 0x7f91e401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:15.636 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.640 7f91eeffd700 1 -- 172.21.2.112:0/1392303777 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f91f40328c0 con 0x7f91fc1a7470 2021-02-10T11:39:15.637 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.640 7f920257b700 1 --2- 172.21.2.112:0/1392303777 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f91e40186c0 0x7f91e401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:15.637 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.641 7f920257b700 1 --2- 172.21.2.112:0/1392303777 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f91e40186c0 0x7f91e401abc0 crc :-1 s=READY pgs=886 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:15.663 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.670 7f920475f700 1 -- 172.21.2.112:0/1392303777 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91fc082a20 con 0x7f91fc1a7470 2021-02-10T11:39:15.666 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.673 7f91eeffd700 1 -- 172.21.2.112:0/1392303777 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f91f40067f0 con 0x7f91fc1a7470 2021-02-10T11:39:15.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.802 7f920475f700 1 -- 172.21.2.112:0/1392303777 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "target": ["mgr", ""], "group_name": "group_0000000000143913"}) v1 -- 0x7f91fc09d1f0 con 0x7f91e40186c0 2021-02-10T11:39:15.798 INFO:tasks.ceph.mgr.x.gibba020.stderr:2021-02-10 11:39:15.803 7f6382eee700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:39:15.798 INFO:tasks.ceph.mgr.x.gibba020.stderr:2021-02-10 11:39:15.803 7f6382eee700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:39:15.798 INFO:tasks.ceph.mgr.x.gibba020.stderr:2021-02-10 11:39:15.803 7f6382eee700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:39:15.798 INFO:tasks.ceph.mgr.x.gibba020.stderr:2021-02-10 11:39:15.803 7f6382eee700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:39:15.798 INFO:tasks.ceph.mgr.x.gibba020.stderr:2021-02-10 11:39:15.803 7f6382eee700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:39:15.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.819 7f91eeffd700 1 -- 172.21.2.112:0/1392303777 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f91f0002040 con 0x7f91e40186c0 2021-02-10T11:39:15.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.835 7f920475f700 1 -- 172.21.2.112:0/1392303777 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f91e40186c0 msgr2=0x7f91e401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:15.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.835 7f920475f700 1 --2- 172.21.2.112:0/1392303777 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f91e40186c0 0x7f91e401abc0 crc :-1 s=READY pgs=886 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:15.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.835 7f920475f700 1 -- 172.21.2.112:0/1392303777 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f91fc1a7470 msgr2=0x7f91fc1a7870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:15.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.835 7f920475f700 1 --2- 172.21.2.112:0/1392303777 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f91fc1a7470 0x7f91fc1a7870 secure :-1 s=READY pgs=1889 cs=0 l=1 rev1=1 rx=0x7f91f400b910 tx=0x7f91f400bcb0).stop 2021-02-10T11:39:15.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.835 7f920475f700 1 -- 172.21.2.112:0/1392303777 shutdown_connections 2021-02-10T11:39:15.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.835 7f920475f700 1 --2- 172.21.2.112:0/1392303777 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f91e40186c0 0x7f91e401abc0 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:15.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.835 7f920475f700 1 --2- 172.21.2.112:0/1392303777 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f91fc0adf50 0x7f91fc09f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:15.829 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.835 7f920475f700 1 --2- 172.21.2.112:0/1392303777 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f91fc1a7470 0x7f91fc1a7870 unknown :-1 s=CLOSED pgs=1889 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:15.829 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.835 7f920475f700 1 --2- 172.21.2.112:0/1392303777 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f91fc0badd0 0x7f91fc09eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:15.829 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.835 7f920475f700 1 -- 172.21.2.112:0/1392303777 >> 172.21.2.112:0/1392303777 conn(0x7f91fc0bee90 msgr2=0x7f91fc0bb880 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:15.829 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.835 7f920475f700 1 -- 172.21.2.112:0/1392303777 shutdown_connections 2021-02-10T11:39:15.829 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:15.835 7f920475f700 1 -- 172.21.2.112:0/1392303777 wait complete. 2021-02-10T11:39:15.836 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000918084 --group_name group_0000000000143913 2021-02-10T11:39:16.268 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.257 7f1318244700 1 Processor -- start 2021-02-10T11:39:16.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.257 7f1318244700 1 -- start start 2021-02-10T11:39:16.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.258 7f1318244700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f13100d1880 0x7f13101b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:16.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.258 7f1318244700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13100fe1d0 con 0x7f13101b0af0 2021-02-10T11:39:16.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.258 7f1318244700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1310109a70 con 0x7f13100cff80 2021-02-10T11:39:16.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.258 7f1318244700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1310088070 con 0x7f13100d1880 2021-02-10T11:39:16.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.258 7f131585f700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f13101b0af0 0x7f1310108950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:53070/0 (socket says 172.21.2.112:53070) 2021-02-10T11:39:16.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.258 7f131585f700 1 -- 172.21.2.112:0/1826933464 learned_addr learned my addr 172.21.2.112:0/1826933464 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:39:16.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.259 7f131505e700 1 -- 172.21.2.112:0/1826933464 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1533146998 0 0) 0x7f13000010b0 con 0x7f13101b0af0 2021-02-10T11:39:16.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.259 7f131505e700 1 -- 172.21.2.112:0/1826933464 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12fc002fe0 con 0x7f13101b0af0 2021-02-10T11:39:16.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.259 7f1316060700 1 --2- 172.21.2.112:0/1826933464 >> v2:172.21.2.120:3300/0 conn(0x7f13100d1880 0x7f13101b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:16.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.259 7f131505e700 1 -- 172.21.2.112:0/1826933464 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4165202574 0 0) 0x7f13000010b0 con 0x7f13101b0af0 2021-02-10T11:39:16.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.259 7f131505e700 1 -- 172.21.2.112:0/1826933464 >> v2:172.21.2.120:3300/0 conn(0x7f13100d1880 msgr2=0x7f13101b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:16.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.259 7f131505e700 1 --2- 172.21.2.112:0/1826933464 >> v2:172.21.2.120:3300/0 conn(0x7f13100d1880 0x7f13101b04d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:16.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.259 7f131505e700 1 -- 172.21.2.112:0/1826933464 >> v1:172.21.2.120:6789/0 conn(0x7f13100cff80 legacy=0x7f1310109140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:16.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.259 7f131505e700 1 -- 172.21.2.112:0/1826933464 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1310088260 con 0x7f13101b0af0 2021-02-10T11:39:16.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.259 7f131505e700 1 -- 172.21.2.112:0/1826933464 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f13000015b0 con 0x7f13101b0af0 2021-02-10T11:39:16.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.259 7f131505e700 1 -- 172.21.2.112:0/1826933464 >> v1:172.21.2.112:6789/0 conn(0x7f13101b0af0 legacy=0x7f1310108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:16.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.259 7f131505e700 1 --2- 172.21.2.112:0/1826933464 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f12fc003af0 0x7f12fc005f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:16.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.259 7f131505e700 1 --2- 172.21.2.112:0/1826933464 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f12fc0067a0 0x7f12fc008ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:16.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.259 7f131505e700 1 --2- 172.21.2.112:0/1826933464 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f12fc0095a0 0x7f12fc00baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:16.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.259 7f131505e700 1 -- 172.21.2.112:0/1826933464 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12fc00c2f0 con 0x7f12fc003af0 2021-02-10T11:39:16.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.259 7f131505e700 1 -- 172.21.2.112:0/1826933464 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12fc00c510 con 0x7f12fc0095a0 2021-02-10T11:39:16.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.259 7f131505e700 1 -- 172.21.2.112:0/1826933464 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12fc00c730 con 0x7f12fc0067a0 2021-02-10T11:39:16.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.259 7f1316060700 1 --2- 172.21.2.112:0/1826933464 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f12fc003af0 0x7f12fc005f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:16.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.259 7f1316861700 1 --2- 172.21.2.112:0/1826933464 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f12fc0095a0 0x7f12fc00baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:16.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.260 7f131585f700 1 --2- 172.21.2.112:0/1826933464 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f12fc0067a0 0x7f12fc008ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:16.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.260 7f1316060700 1 -- 172.21.2.112:0/1826933464 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f12fc0067a0 msgr2=0x7f12fc008ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:16.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.260 7f1316060700 1 --2- 172.21.2.112:0/1826933464 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f12fc0067a0 0x7f12fc008ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:16.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.260 7f1316060700 1 -- 172.21.2.112:0/1826933464 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f12fc0095a0 msgr2=0x7f12fc00baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:16.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.260 7f1316060700 1 --2- 172.21.2.112:0/1826933464 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f12fc0095a0 0x7f12fc00baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:16.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.260 7f1316060700 1 -- 172.21.2.112:0/1826933464 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12fc0509a0 con 0x7f12fc003af0 2021-02-10T11:39:16.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.260 7f1316861700 1 --2- 172.21.2.112:0/1826933464 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f12fc0095a0 0x7f12fc00baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:39:16.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.260 7f131585f700 1 --2- 172.21.2.112:0/1826933464 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f12fc0067a0 0x7f12fc008ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:39:16.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.260 7f1316861700 1 -- 172.21.2.112:0/1826933464 reap_dead start 2021-02-10T11:39:16.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.260 7f1316060700 1 --2- 172.21.2.112:0/1826933464 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f12fc003af0 0x7f12fc005f80 secure :-1 s=READY pgs=1891 cs=0 l=1 rev1=1 rx=0x7f130c00c8f0 tx=0x7f130c00cc90).ready entity=mon.0 client_cookie=3e0d5abcc54bb495 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:16.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.260 7f131505e700 1 -- 172.21.2.112:0/1826933464 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f130c0009b0 con 0x7f12fc003af0 2021-02-10T11:39:16.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.260 7f131505e700 1 -- 172.21.2.112:0/1826933464 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f130c009310 con 0x7f12fc003af0 2021-02-10T11:39:16.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.260 7f131505e700 1 -- 172.21.2.112:0/1826933464 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f130c009830 con 0x7f12fc003af0 2021-02-10T11:39:16.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.260 7f1318244700 1 -- 172.21.2.112:0/1826933464 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f12fc003af0 msgr2=0x7f12fc005f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:16.272 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.260 7f1318244700 1 --2- 172.21.2.112:0/1826933464 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f12fc003af0 0x7f12fc005f80 secure :-1 s=READY pgs=1891 cs=0 l=1 rev1=1 rx=0x7f130c00c8f0 tx=0x7f130c00cc90).stop 2021-02-10T11:39:16.272 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.261 7f1318244700 1 -- 172.21.2.112:0/1826933464 shutdown_connections 2021-02-10T11:39:16.272 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.261 7f1318244700 1 --2- 172.21.2.112:0/1826933464 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f12fc003af0 0x7f12fc005f80 unknown :-1 s=CLOSED pgs=1891 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:16.272 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.261 7f1318244700 1 -- 172.21.2.112:0/1826933464 >> 172.21.2.112:0/1826933464 conn(0x7f13100ce430 msgr2=0x7f13100d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:16.272 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.261 7f1318244700 1 -- 172.21.2.112:0/1826933464 shutdown_connections 2021-02-10T11:39:16.272 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.261 7f1318244700 1 -- 172.21.2.112:0/1826933464 wait complete. 2021-02-10T11:39:16.273 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.276 7f1318244700 1 Processor -- start 2021-02-10T11:39:16.274 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.276 7f1318244700 1 -- start start 2021-02-10T11:39:16.274 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.276 7f1318244700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1310188400 0x7f1310188800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:16.274 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.276 7f1318244700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1310188e20 0x7f1310189220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:16.274 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.276 7f1318244700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f131009cbc0 0x7f131009cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:16.274 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.276 7f1318244700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1310187e80 con 0x7f1310188e20 2021-02-10T11:39:16.274 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.276 7f1318244700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13100b27c0 con 0x7f131009cbc0 2021-02-10T11:39:16.274 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.276 7f1318244700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13100b29a0 con 0x7f1310188400 2021-02-10T11:39:16.274 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.277 7f131585f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1310188e20 0x7f1310189220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:16.275 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.277 7f131585f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1310188e20 0x7f1310189220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:53620/0 (socket says 172.21.2.112:53620) 2021-02-10T11:39:16.275 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.277 7f131585f700 1 -- 172.21.2.112:0/3066160344 learned_addr learned my addr 172.21.2.112:0/3066160344 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:39:16.275 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.277 7f1316861700 1 --2- 172.21.2.112:0/3066160344 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f131009cbc0 0x7f131009cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:16.275 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.277 7f1316060700 1 --2- 172.21.2.112:0/3066160344 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1310188400 0x7f1310188800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:16.275 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.277 7f131585f700 1 -- 172.21.2.112:0/3066160344 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1310188400 msgr2=0x7f1310188800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:16.275 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.277 7f131585f700 1 --2- 172.21.2.112:0/3066160344 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1310188400 0x7f1310188800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:16.275 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.277 7f131585f700 1 -- 172.21.2.112:0/3066160344 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f131009cbc0 msgr2=0x7f131009cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:16.275 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.277 7f131585f700 1 --2- 172.21.2.112:0/3066160344 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f131009cbc0 0x7f131009cfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:16.275 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.277 7f131585f700 1 -- 172.21.2.112:0/3066160344 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f131009d5e0 con 0x7f1310188e20 2021-02-10T11:39:16.275 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.277 7f131585f700 1 --2- 172.21.2.112:0/3066160344 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1310188e20 0x7f1310189220 secure :-1 s=READY pgs=1892 cs=0 l=1 rev1=1 rx=0x7f130000a540 tx=0x7f130000a8e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:16.276 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.277 7f1306ffd700 1 -- 172.21.2.112:0/3066160344 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f13000009b0 con 0x7f1310188e20 2021-02-10T11:39:16.276 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.277 7f1306ffd700 1 -- 172.21.2.112:0/3066160344 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f130000acd0 con 0x7f1310188e20 2021-02-10T11:39:16.276 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.277 7f1318244700 1 -- 172.21.2.112:0/3066160344 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f131009d5e0 con 0x7f1310188e20 2021-02-10T11:39:16.276 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.277 7f1306ffd700 1 -- 172.21.2.112:0/3066160344 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1300004540 con 0x7f1310188e20 2021-02-10T11:39:16.276 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.277 7f1318244700 1 -- 172.21.2.112:0/3066160344 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f131009db70 con 0x7f1310188e20 2021-02-10T11:39:16.276 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.278 7f1306ffd700 1 -- 172.21.2.112:0/3066160344 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1300004140 con 0x7f1310188e20 2021-02-10T11:39:16.276 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.278 7f1306ffd700 1 --2- 172.21.2.112:0/3066160344 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f12f8018730 0x7f12f801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:16.276 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.278 7f1306ffd700 1 -- 172.21.2.112:0/3066160344 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f1300032620 con 0x7f1310188e20 2021-02-10T11:39:16.276 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.278 7f1316060700 1 --2- 172.21.2.112:0/3066160344 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f12f8018730 0x7f12f801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:16.276 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.279 7f1316060700 1 --2- 172.21.2.112:0/3066160344 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f12f8018730 0x7f12f801ac10 crc :-1 s=READY pgs=887 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:16.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.308 7f1318244700 1 -- 172.21.2.112:0/3066160344 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1310082a70 con 0x7f1310188e20 2021-02-10T11:39:16.303 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.310 7f1306ffd700 1 -- 172.21.2.112:0/3066160344 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f13000009b0 con 0x7f1310188e20 2021-02-10T11:39:16.432 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.439 7f1318244700 1 -- 172.21.2.112:0/3066160344 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000918084", "target": ["mgr", ""], "group_name": "group_0000000000143913"}) v1 -- 0x7f13100b27c0 con 0x7f12f8018730 2021-02-10T11:39:16.456 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.464 7f1306ffd700 1 -- 172.21.2.112:0/3066160344 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f130c006c50 con 0x7f12f8018730 2021-02-10T11:39:16.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.472 7f1318244700 1 -- 172.21.2.112:0/3066160344 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f12f8018730 msgr2=0x7f12f801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:16.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.472 7f1318244700 1 --2- 172.21.2.112:0/3066160344 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f12f8018730 0x7f12f801ac10 crc :-1 s=READY pgs=887 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:16.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.472 7f1318244700 1 -- 172.21.2.112:0/3066160344 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1310188e20 msgr2=0x7f1310189220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:16.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.472 7f1318244700 1 --2- 172.21.2.112:0/3066160344 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1310188e20 0x7f1310189220 secure :-1 s=READY pgs=1892 cs=0 l=1 rev1=1 rx=0x7f130000a540 tx=0x7f130000a8e0).stop 2021-02-10T11:39:16.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.472 7f1318244700 1 -- 172.21.2.112:0/3066160344 shutdown_connections 2021-02-10T11:39:16.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.472 7f1318244700 1 --2- 172.21.2.112:0/3066160344 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f12f8018730 0x7f12f801ac10 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:16.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.472 7f1318244700 1 --2- 172.21.2.112:0/3066160344 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f131009cbc0 0x7f131009cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:16.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.472 7f1318244700 1 --2- 172.21.2.112:0/3066160344 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1310188e20 0x7f1310189220 unknown :-1 s=CLOSED pgs=1892 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:16.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.472 7f1318244700 1 --2- 172.21.2.112:0/3066160344 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1310188400 0x7f1310188800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:16.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.472 7f1318244700 1 -- 172.21.2.112:0/3066160344 >> 172.21.2.112:0/3066160344 conn(0x7f13100c2cd0 msgr2=0x7f13100b5950 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:16.466 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.472 7f1318244700 1 -- 172.21.2.112:0/3066160344 shutdown_connections 2021-02-10T11:39:16.466 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.472 7f1318244700 1 -- 172.21.2.112:0/3066160344 wait complete. 2021-02-10T11:39:16.473 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume authorize cephfs subvolume_0000000000918084 guest1 --group_name group_0000000000143913 --tenant_id tenant1 2021-02-10T11:39:16.913 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.903 7fe5f1a7f700 1 Processor -- start 2021-02-10T11:39:16.914 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.903 7fe5f1a7f700 1 -- start start 2021-02-10T11:39:16.914 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.904 7fe5f1a7f700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fe5ec0e53a0 0x7fe5ec1b0550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:16.914 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.904 7fe5f1a7f700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe5ec1b0b70 0x7fe5ec1089d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:16.914 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.904 7fe5f1a7f700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5ec109ac0 con 0x7fe5ec108ff0 2021-02-10T11:39:16.914 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.904 7fe5f1a7f700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5ec0e57a0 con 0x7fe5ec0e53a0 2021-02-10T11:39:16.914 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.904 7fe5f1a7f700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5ec088070 con 0x7fe5ec1b0b70 2021-02-10T11:39:16.914 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.904 7fe5ebfff700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fe5ec108ff0 0x7fe5ec10a040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:53088/0 (socket says 172.21.2.112:53088) 2021-02-10T11:39:16.914 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.904 7fe5ebfff700 1 -- 172.21.2.112:0/1425826020 learned_addr learned my addr 172.21.2.112:0/1425826020 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:39:16.914 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.904 7fe5ea7fc700 1 -- 172.21.2.112:0/1425826020 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 823489865 0 0) 0x7fe5e4001500 con 0x7fe5ec108ff0 2021-02-10T11:39:16.914 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.904 7fe5ea7fc700 1 -- 172.21.2.112:0/1425826020 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5d8002fe0 con 0x7fe5ec108ff0 2021-02-10T11:39:16.914 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.904 7fe5eb7fe700 1 --2- 172.21.2.112:0/1425826020 >> v2:172.21.2.120:3300/0 conn(0x7fe5ec0e53a0 0x7fe5ec1b0550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:16.914 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.904 7fe5eaffd700 1 --2- 172.21.2.112:0/1425826020 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe5ec1b0b70 0x7fe5ec1089d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:16.914 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.905 7fe5ea7fc700 1 -- 172.21.2.112:0/1425826020 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1503918454 0 0) 0x7fe5e4001460 con 0x7fe5ec108ff0 2021-02-10T11:39:16.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.905 7fe5ea7fc700 1 -- 172.21.2.112:0/1425826020 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe5ec1b0b70 msgr2=0x7fe5ec1089d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:16.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.905 7fe5ea7fc700 1 --2- 172.21.2.112:0/1425826020 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe5ec1b0b70 0x7fe5ec1089d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:16.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.905 7fe5ea7fc700 1 -- 172.21.2.112:0/1425826020 >> v2:172.21.2.120:3300/0 conn(0x7fe5ec0e53a0 msgr2=0x7fe5ec1b0550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:16.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.905 7fe5ea7fc700 1 --2- 172.21.2.112:0/1425826020 >> v2:172.21.2.120:3300/0 conn(0x7fe5ec0e53a0 0x7fe5ec1b0550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:16.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.905 7fe5ea7fc700 1 -- 172.21.2.112:0/1425826020 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5ec1093f0 con 0x7fe5ec108ff0 2021-02-10T11:39:16.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.905 7fe5ea7fc700 1 -- 172.21.2.112:0/1425826020 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7fe5e4001980 con 0x7fe5ec108ff0 2021-02-10T11:39:16.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.905 7fe5ea7fc700 1 -- 172.21.2.112:0/1425826020 >> v1:172.21.2.112:6789/0 conn(0x7fe5ec108ff0 legacy=0x7fe5ec10a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:16.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.905 7fe5ea7fc700 1 --2- 172.21.2.112:0/1425826020 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe5d8003b10 0x7fe5d8006010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:16.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.905 7fe5ea7fc700 1 --2- 172.21.2.112:0/1425826020 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe5d8006820 0x7fe5d8008d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:16.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.905 7fe5ea7fc700 1 --2- 172.21.2.112:0/1425826020 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe5d8009640 0x7fe5d800bb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:16.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.905 7fe5ea7fc700 1 -- 172.21.2.112:0/1425826020 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5d8050370 con 0x7fe5d8003b10 2021-02-10T11:39:16.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.905 7fe5ea7fc700 1 -- 172.21.2.112:0/1425826020 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5d8050590 con 0x7fe5d8006820 2021-02-10T11:39:16.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.905 7fe5ea7fc700 1 -- 172.21.2.112:0/1425826020 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5d80507b0 con 0x7fe5d8009640 2021-02-10T11:39:16.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.905 7fe5eaffd700 1 --2- 172.21.2.112:0/1425826020 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe5ec1b0b70 0x7fe5ec1089d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:39:16.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.905 7fe5eb7fe700 1 --2- 172.21.2.112:0/1425826020 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe5d8003b10 0x7fe5d8006010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:16.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.905 7fe5ebfff700 1 --2- 172.21.2.112:0/1425826020 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe5d8009640 0x7fe5d800bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:16.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.905 7fe5eb7fe700 1 -- 172.21.2.112:0/1425826020 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe5d8009640 msgr2=0x7fe5d800bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:16.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.905 7fe5eb7fe700 1 --2- 172.21.2.112:0/1425826020 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe5d8009640 0x7fe5d800bb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:16.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.905 7fe5eb7fe700 1 -- 172.21.2.112:0/1425826020 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe5d8006820 msgr2=0x7fe5d8008d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:16.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.905 7fe5eb7fe700 1 --2- 172.21.2.112:0/1425826020 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe5d8006820 0x7fe5d8008d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:16.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.905 7fe5eb7fe700 1 -- 172.21.2.112:0/1425826020 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5d8050a50 con 0x7fe5d8003b10 2021-02-10T11:39:16.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.905 7fe5ebfff700 1 --2- 172.21.2.112:0/1425826020 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe5d8009640 0x7fe5d800bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:39:16.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.905 7fe5eb7fe700 1 --2- 172.21.2.112:0/1425826020 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe5d8003b10 0x7fe5d8006010 secure :-1 s=READY pgs=1894 cs=0 l=1 rev1=1 rx=0x7fe5dc00c8f0 tx=0x7fe5dc00cc90).ready entity=mon.0 client_cookie=edd59a3f8db0daf3 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:16.918 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.905 7fe5ea7fc700 1 -- 172.21.2.112:0/1425826020 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe5dc00cdd0 con 0x7fe5d8003b10 2021-02-10T11:39:16.918 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.905 7fe5ea7fc700 1 -- 172.21.2.112:0/1425826020 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5dc009310 con 0x7fe5d8003b10 2021-02-10T11:39:16.918 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.905 7fe5ea7fc700 1 -- 172.21.2.112:0/1425826020 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe5dc0097f0 con 0x7fe5d8003b10 2021-02-10T11:39:16.918 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.905 7fe5f1a7f700 1 -- 172.21.2.112:0/1425826020 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe5d8003b10 msgr2=0x7fe5d8006010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:16.918 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.905 7fe5f1a7f700 1 --2- 172.21.2.112:0/1425826020 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe5d8003b10 0x7fe5d8006010 secure :-1 s=READY pgs=1894 cs=0 l=1 rev1=1 rx=0x7fe5dc00c8f0 tx=0x7fe5dc00cc90).stop 2021-02-10T11:39:16.918 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.905 7fe5ebfff700 1 -- 172.21.2.112:0/1425826020 reap_dead start 2021-02-10T11:39:16.918 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.906 7fe5f1a7f700 1 -- 172.21.2.112:0/1425826020 shutdown_connections 2021-02-10T11:39:16.919 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.906 7fe5f1a7f700 1 -- 172.21.2.112:0/1425826020 >> 172.21.2.112:0/1425826020 conn(0x7fe5ec0ce4b0 msgr2=0x7fe5ec0e45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:16.919 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.906 7fe5f1a7f700 1 -- 172.21.2.112:0/1425826020 shutdown_connections 2021-02-10T11:39:16.919 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.906 7fe5f1a7f700 1 -- 172.21.2.112:0/1425826020 wait complete. 2021-02-10T11:39:16.919 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.921 7fe5f1a7f700 1 Processor -- start 2021-02-10T11:39:16.919 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.921 7fe5f1a7f700 1 -- start start 2021-02-10T11:39:16.919 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.921 7fe5f1a7f700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe5ec0a8f70 0x7fe5ec095e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:16.919 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.921 7fe5f1a7f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe5ec0964a0 0x7fe5ec0968a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:16.919 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.921 7fe5f1a7f700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe5ec096ec0 0x7fe5ec106520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:16.919 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.921 7fe5f1a7f700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5ec095900 con 0x7fe5ec0964a0 2021-02-10T11:39:16.919 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.921 7fe5f1a7f700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5ec107be0 con 0x7fe5ec0a8f70 2021-02-10T11:39:16.919 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.921 7fe5f1a7f700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5ec106b40 con 0x7fe5ec096ec0 2021-02-10T11:39:16.919 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.922 7fe5eaffd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe5ec0964a0 0x7fe5ec0968a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:16.919 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.922 7fe5eaffd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe5ec0964a0 0x7fe5ec0968a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:53636/0 (socket says 172.21.2.112:53636) 2021-02-10T11:39:16.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.922 7fe5eaffd700 1 -- 172.21.2.112:0/4244324649 learned_addr learned my addr 172.21.2.112:0/4244324649 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:39:16.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.922 7fe5eaffd700 1 -- 172.21.2.112:0/4244324649 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe5ec096ec0 msgr2=0x7fe5ec106520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:16.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.922 7fe5eaffd700 1 --2- 172.21.2.112:0/4244324649 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe5ec096ec0 0x7fe5ec106520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:16.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.922 7fe5eaffd700 1 -- 172.21.2.112:0/4244324649 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe5ec0a8f70 msgr2=0x7fe5ec095e80 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:39:16.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.922 7fe5eaffd700 1 --2- 172.21.2.112:0/4244324649 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe5ec0a8f70 0x7fe5ec095e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:16.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.922 7fe5eaffd700 1 -- 172.21.2.112:0/4244324649 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5ec106d60 con 0x7fe5ec0964a0 2021-02-10T11:39:16.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.922 7fe5eaffd700 1 --2- 172.21.2.112:0/4244324649 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe5ec0964a0 0x7fe5ec0968a0 secure :-1 s=READY pgs=1895 cs=0 l=1 rev1=1 rx=0x7fe5d4006f20 tx=0x7fe5d400b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:16.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.922 7fe5e3fff700 1 -- 172.21.2.112:0/4244324649 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe5d4006240 con 0x7fe5ec0964a0 2021-02-10T11:39:16.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.922 7fe5e3fff700 1 -- 172.21.2.112:0/4244324649 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5d4000940 con 0x7fe5ec0964a0 2021-02-10T11:39:16.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.922 7fe5f1a7f700 1 -- 172.21.2.112:0/4244324649 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5ec106d60 con 0x7fe5ec0964a0 2021-02-10T11:39:16.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.922 7fe5e3fff700 1 -- 172.21.2.112:0/4244324649 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe5d4000d90 con 0x7fe5ec0964a0 2021-02-10T11:39:16.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.922 7fe5f1a7f700 1 -- 172.21.2.112:0/4244324649 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5ec1072f0 con 0x7fe5ec0964a0 2021-02-10T11:39:16.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.923 7fe5e3fff700 1 -- 172.21.2.112:0/4244324649 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe5d4006030 con 0x7fe5ec0964a0 2021-02-10T11:39:16.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.923 7fe5e3fff700 1 --2- 172.21.2.112:0/4244324649 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe5d00186c0 0x7fe5d001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:16.921 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.923 7fe5e3fff700 1 -- 172.21.2.112:0/4244324649 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fe5d40323f0 con 0x7fe5ec0964a0 2021-02-10T11:39:16.921 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.923 7fe5eb7fe700 1 --2- 172.21.2.112:0/4244324649 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe5d00186c0 0x7fe5d001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:16.921 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.924 7fe5eb7fe700 1 --2- 172.21.2.112:0/4244324649 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe5d00186c0 0x7fe5d001abc0 crc :-1 s=READY pgs=888 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:16.946 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.953 7fe5f1a7f700 1 -- 172.21.2.112:0/4244324649 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5ec103750 con 0x7fe5ec0964a0 2021-02-10T11:39:16.949 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:16.956 7fe5e3fff700 1 -- 172.21.2.112:0/4244324649 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe5d40273d0 con 0x7fe5ec0964a0 2021-02-10T11:39:17.076 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:17.083 7fe5f1a7f700 1 -- 172.21.2.112:0/4244324649 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume authorize", "vol_name": "cephfs", "sub_name": "subvolume_0000000000918084", "target": ["mgr", ""], "tenant_id": "tenant1", "auth_id": "guest1", "group_name": "group_0000000000143913"}) v1 -- 0x7fe5ec109ac0 con 0x7fe5d00186c0 2021-02-10T11:39:17.081 INFO:tasks.ceph.mgr.x.gibba020.stderr:Creating meta for ID guest1 with tenant tenant1 2021-02-10T11:39:17.111 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:17.118 7fe5e3fff700 1 -- 172.21.2.112:0/4244324649 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+41 (crc 0 0 0) 0x7fe5dc00c140 con 0x7fe5d00186c0 2021-02-10T11:39:17.140 INFO:teuthology.orchestra.run.gibba012.stdout:AQDlxSNgMaQDBhAAO5GRoC6uEX0RAKa15nbbEQ== 2021-02-10T11:39:17.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:17.148 7fe5f1a7f700 1 -- 172.21.2.112:0/4244324649 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe5d00186c0 msgr2=0x7fe5d001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:17.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:17.148 7fe5f1a7f700 1 --2- 172.21.2.112:0/4244324649 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe5d00186c0 0x7fe5d001abc0 crc :-1 s=READY pgs=888 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:17.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:17.148 7fe5f1a7f700 1 -- 172.21.2.112:0/4244324649 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe5ec0964a0 msgr2=0x7fe5ec0968a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:17.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:17.148 7fe5f1a7f700 1 --2- 172.21.2.112:0/4244324649 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe5ec0964a0 0x7fe5ec0968a0 secure :-1 s=READY pgs=1895 cs=0 l=1 rev1=1 rx=0x7fe5d4006f20 tx=0x7fe5d400b990).stop 2021-02-10T11:39:17.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:17.149 7fe5f1a7f700 1 -- 172.21.2.112:0/4244324649 shutdown_connections 2021-02-10T11:39:17.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:17.149 7fe5f1a7f700 1 --2- 172.21.2.112:0/4244324649 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe5d00186c0 0x7fe5d001abc0 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:17.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:17.149 7fe5f1a7f700 1 --2- 172.21.2.112:0/4244324649 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe5ec096ec0 0x7fe5ec106520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:17.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:17.149 7fe5f1a7f700 1 --2- 172.21.2.112:0/4244324649 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe5ec0964a0 0x7fe5ec0968a0 unknown :-1 s=CLOSED pgs=1895 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:17.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:17.149 7fe5f1a7f700 1 --2- 172.21.2.112:0/4244324649 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe5ec0a8f70 0x7fe5ec095e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:17.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:17.149 7fe5f1a7f700 1 -- 172.21.2.112:0/4244324649 >> 172.21.2.112:0/4244324649 conn(0x7fe5ec09e480 msgr2=0x7fe5ec107f30 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:17.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:17.149 7fe5f1a7f700 1 -- 172.21.2.112:0/4244324649 shutdown_connections 2021-02-10T11:39:17.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:17.149 7fe5f1a7f700 1 -- 172.21.2.112:0/4244324649 wait complete. 2021-02-10T11:39:17.150 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000918084 --group_name group_0000000000143913 2021-02-10T11:39:17.581 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:17.572 7f8e7f71c700 1 Processor -- start 2021-02-10T11:39:17.581 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:17.572 7f8e7f71c700 1 -- start start 2021-02-10T11:39:17.581 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:17.573 7f8e7f71c700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8e780d1c80 0x7f8e780cef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:17.581 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:17.573 7f8e7f71c700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e780d0180 con 0x7f8e780cf590 2021-02-10T11:39:17.582 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:17.573 7f8e7f71c700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e780fe1d0 con 0x7f8e780d1880 2021-02-10T11:39:17.582 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:17.573 7f8e7f71c700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e781b0cc0 con 0x7f8e780d1c80 2021-02-10T11:39:17.582 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:17.573 7f8e7dd39700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f8e780cf590 0x7f8e780cf990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:53110/0 (socket says 172.21.2.112:53110) 2021-02-10T11:39:17.582 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:17.573 7f8e7dd39700 1 -- 172.21.2.112:0/2783135932 learned_addr learned my addr 172.21.2.112:0/2783135932 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:39:17.582 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:17.573 7f8e7cd37700 1 --2- 172.21.2.112:0/2783135932 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8e780d1c80 0x7f8e780cef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:17.582 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:17.573 7f8e6bfff700 1 -- 172.21.2.112:0/2783135932 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2205741394 0 0) 0x7f8e70001500 con 0x7f8e780cf590 2021-02-10T11:39:17.582 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:17.573 7f8e7cd37700 1 -- 172.21.2.112:0/2783135932 >> v1:172.21.2.120:6789/0 conn(0x7f8e780d1880 legacy=0x7f8e781b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:17.582 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:17.573 7f8e7cd37700 1 -- 172.21.2.112:0/2783135932 >> v1:172.21.2.112:6789/0 conn(0x7f8e780cf590 legacy=0x7f8e780cf990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:17.582 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:17.573 7f8e7cd37700 1 -- 172.21.2.112:0/2783135932 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e78109a70 con 0x7f8e780d1c80 2021-02-10T11:39:17.582 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:17.573 7f8e7cd37700 1 --2- 172.21.2.112:0/2783135932 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8e780d1c80 0x7f8e780cef70 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7f8e74006650 tx=0x7f8e74009a00).ready entity=mon.2 client_cookie=9a91d33d4d416f2 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:17.582 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:17.574 7f8e6bfff700 1 -- 172.21.2.112:0/2783135932 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8e740009b0 con 0x7f8e780d1c80 2021-02-10T11:39:17.582 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:17.574 7f8e6bfff700 1 -- 172.21.2.112:0/2783135932 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e7400c0c0 con 0x7f8e780d1c80 2021-02-10T11:39:17.583 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:17.574 7f8e6bfff700 1 -- 172.21.2.112:0/2783135932 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8e7400c5a0 con 0x7f8e780d1c80 2021-02-10T11:39:17.583 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:17.574 7f8e7f71c700 1 -- 172.21.2.112:0/2783135932 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8e780d1c80 msgr2=0x7f8e780cef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:17.583 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:17.574 7f8e7f71c700 1 --2- 172.21.2.112:0/2783135932 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8e780d1c80 0x7f8e780cef70 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7f8e74006650 tx=0x7f8e74009a00).stop 2021-02-10T11:39:17.583 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:17.574 7f8e7f71c700 1 -- 172.21.2.112:0/2783135932 shutdown_connections 2021-02-10T11:39:17.583 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:17.574 7f8e7f71c700 1 --2- 172.21.2.112:0/2783135932 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8e780d1c80 0x7f8e780cef70 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:17.583 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:17.574 7f8e7f71c700 1 -- 172.21.2.112:0/2783135932 >> 172.21.2.112:0/2783135932 conn(0x7f8e780ce430 msgr2=0x7f8e780d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:17.583 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:17.574 7f8e7f71c700 1 -- 172.21.2.112:0/2783135932 shutdown_connections 2021-02-10T11:39:17.583 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:17.574 7f8e7f71c700 1 -- 172.21.2.112:0/2783135932 wait complete. 2021-02-10T11:39:17.584 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:17.589 7f8e7f71c700 1 Processor -- start 2021-02-10T11:39:17.584 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:17.589 7f8e7f71c700 1 -- start start 2021-02-10T11:39:17.584 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:17.589 7f8e7f71c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8e7819e5b0 0x7f8e781afc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:17.584 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:17.589 7f8e7f71c700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8e7819e9b0 0x7f8e781b0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:17.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:17.589 7f8e7f71c700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8e781b0250 0x7f8e781b13e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:17.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:17.589 7f8e7f71c700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e78195bd0 con 0x7f8e7819e5b0 2021-02-10T11:39:17.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:17.589 7f8e7f71c700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e781b0840 con 0x7f8e781b0250 2021-02-10T11:39:17.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:17.589 7f8e7f71c700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e7819eed0 con 0x7f8e7819e9b0 2021-02-10T11:39:17.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:17.589 7f8e7d538700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8e7819e5b0 0x7f8e781afc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:17.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:17.589 7f8e7d538700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8e7819e5b0 0x7f8e781afc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:53650/0 (socket says 172.21.2.112:53650) 2021-02-10T11:39:17.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:17.589 7f8e7d538700 1 -- 172.21.2.112:0/2266413158 learned_addr learned my addr 172.21.2.112:0/2266413158 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:39:17.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:17.590 7f8e7d538700 1 -- 172.21.2.112:0/2266413158 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8e7819e9b0 msgr2=0x7f8e781b0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:17.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:17.590 7f8e7d538700 1 --2- 172.21.2.112:0/2266413158 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8e7819e9b0 0x7f8e781b0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:17.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:17.590 7f8e7d538700 1 -- 172.21.2.112:0/2266413158 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8e781b0250 msgr2=0x7f8e781b13e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:17.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:17.590 7f8e7d538700 1 --2- 172.21.2.112:0/2266413158 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8e781b0250 0x7f8e781b13e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:17.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:17.590 7f8e7d538700 1 -- 172.21.2.112:0/2266413158 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e7819f0b0 con 0x7f8e7819e5b0 2021-02-10T11:39:17.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:17.590 7f8e7d538700 1 --2- 172.21.2.112:0/2266413158 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8e7819e5b0 0x7f8e781afc30 secure :-1 s=READY pgs=1897 cs=0 l=1 rev1=1 rx=0x7f8e6c006f10 tx=0x7f8e6c00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:17.586 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:17.590 7f8e69ffb700 1 -- 172.21.2.112:0/2266413158 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8e6c0009b0 con 0x7f8e7819e5b0 2021-02-10T11:39:17.586 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:17.590 7f8e69ffb700 1 -- 172.21.2.112:0/2266413158 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e6c00bd80 con 0x7f8e7819e5b0 2021-02-10T11:39:17.586 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:17.590 7f8e69ffb700 1 -- 172.21.2.112:0/2266413158 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8e6c0013f0 con 0x7f8e7819e5b0 2021-02-10T11:39:17.586 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:17.590 7f8e7f71c700 1 -- 172.21.2.112:0/2266413158 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e7819f0b0 con 0x7f8e7819e5b0 2021-02-10T11:39:17.586 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:17.590 7f8e7f71c700 1 -- 172.21.2.112:0/2266413158 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e781b1cf0 con 0x7f8e7819e5b0 2021-02-10T11:39:17.586 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:17.591 7f8e69ffb700 1 -- 172.21.2.112:0/2266413158 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8e6c001200 con 0x7f8e7819e5b0 2021-02-10T11:39:17.586 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:17.591 7f8e69ffb700 1 --2- 172.21.2.112:0/2266413158 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8e580186c0 0x7f8e5801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:17.586 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:17.591 7f8e69ffb700 1 -- 172.21.2.112:0/2266413158 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f8e6c032560 con 0x7f8e7819e5b0 2021-02-10T11:39:17.587 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:17.592 7f8e7cd37700 1 --2- 172.21.2.112:0/2266413158 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8e580186c0 0x7f8e5801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:17.587 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:17.593 7f8e7cd37700 1 --2- 172.21.2.112:0/2266413158 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8e580186c0 0x7f8e5801abc0 crc :-1 s=READY pgs=889 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:17.615 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:17.623 7f8e7f71c700 1 -- 172.21.2.112:0/2266413158 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e78082aa0 con 0x7f8e7819e5b0 2021-02-10T11:39:17.618 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:17.625 7f8e69ffb700 1 -- 172.21.2.112:0/2266413158 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8e6c01c2b0 con 0x7f8e7819e5b0 2021-02-10T11:39:17.746 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:17.753 7f8e7f71c700 1 -- 172.21.2.112:0/2266413158 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000918084", "target": ["mgr", ""], "group_name": "group_0000000000143913"}) v1 -- 0x7f8e781b1cf0 con 0x7f8e580186c0 2021-02-10T11:39:17.749 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:17.756 7f8e69ffb700 1 -- 172.21.2.112:0/2266413158 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+96 (crc 0 0 0) 0x7f8e74006e30 con 0x7f8e580186c0 2021-02-10T11:39:17.749 INFO:teuthology.orchestra.run.gibba012.stdout:/volumes/group_0000000000143913/subvolume_0000000000918084/6682d7c1-1053-4b9a-9c54-24970fa60db6 2021-02-10T11:39:17.751 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:17.758 7f8e7f71c700 1 -- 172.21.2.112:0/2266413158 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8e580186c0 msgr2=0x7f8e5801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:17.751 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:17.758 7f8e7f71c700 1 --2- 172.21.2.112:0/2266413158 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8e580186c0 0x7f8e5801abc0 crc :-1 s=READY pgs=889 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:17.751 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:17.758 7f8e7f71c700 1 -- 172.21.2.112:0/2266413158 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8e7819e5b0 msgr2=0x7f8e781afc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:17.751 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:17.758 7f8e7f71c700 1 --2- 172.21.2.112:0/2266413158 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8e7819e5b0 0x7f8e781afc30 secure :-1 s=READY pgs=1897 cs=0 l=1 rev1=1 rx=0x7f8e6c006f10 tx=0x7f8e6c00b990).stop 2021-02-10T11:39:17.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:17.758 7f8e7f71c700 1 -- 172.21.2.112:0/2266413158 shutdown_connections 2021-02-10T11:39:17.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:17.758 7f8e7f71c700 1 --2- 172.21.2.112:0/2266413158 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8e580186c0 0x7f8e5801abc0 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:17.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:17.758 7f8e7f71c700 1 --2- 172.21.2.112:0/2266413158 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8e781b0250 0x7f8e781b13e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:17.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:17.758 7f8e7f71c700 1 --2- 172.21.2.112:0/2266413158 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8e7819e9b0 0x7f8e781b0dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:17.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:17.758 7f8e7f71c700 1 --2- 172.21.2.112:0/2266413158 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8e7819e5b0 0x7f8e781afc30 unknown :-1 s=CLOSED pgs=1897 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:17.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:17.759 7f8e7f71c700 1 -- 172.21.2.112:0/2266413158 >> 172.21.2.112:0/2266413158 conn(0x7f8e7819f940 msgr2=0x7f8e7818c290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:17.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:17.759 7f8e7f71c700 1 -- 172.21.2.112:0/2266413158 shutdown_connections 2021-02-10T11:39:17.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:17.759 7f8e7f71c700 1 -- 172.21.2.112:0/2266413158 wait complete. 2021-02-10T11:39:17.760 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.guest1 mds 'allow rw path=/volumes/group_0000000000143913, allow rw path=/volumes/group_0000000000143913/subvolume_0000000000918084/6682d7c1-1053-4b9a-9c54-24970fa60db6' osd 'allow rw pool=cephfs_data' mon 'allow r' mgr 'allow *' 2021-02-10T11:39:18.200 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.191 7ffbcc6e5700 1 Processor -- start 2021-02-10T11:39:18.200 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.191 7ffbcc6e5700 1 -- start start 2021-02-10T11:39:18.200 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.192 7ffbcc6e5700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7ffbc40e5350 0x7ffbc410d150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:18.200 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.192 7ffbcc6e5700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7ffbc410fce0 0x7ffbc410df60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:18.201 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.192 7ffbcc6e5700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbc410e750 con 0x7ffbc40e5750 2021-02-10T11:39:18.201 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.192 7ffbcc6e5700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffbc40e61d0 con 0x7ffbc410fce0 2021-02-10T11:39:18.201 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.192 7ffbcc6e5700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffbc41100e0 con 0x7ffbc40e5350 2021-02-10T11:39:18.201 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.192 7ffbcad02700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7ffbc410fce0 0x7ffbc410df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:18.201 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.192 7ffbcad02700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7ffbc410fce0 0x7ffbc410df60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:53658/0 (socket says 172.21.2.112:53658) 2021-02-10T11:39:18.201 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.192 7ffbcad02700 1 -- 172.21.2.112:0/960531431 learned_addr learned my addr 172.21.2.112:0/960531431 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:39:18.201 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.192 7ffbcad02700 1 -- 172.21.2.112:0/960531431 >> v2:172.21.2.120:3300/0 conn(0x7ffbc40e5350 msgr2=0x7ffbc410d150 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:39:18.201 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.192 7ffbcad02700 1 --2- 172.21.2.112:0/960531431 >> v2:172.21.2.120:3300/0 conn(0x7ffbc40e5350 0x7ffbc410d150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:18.201 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.192 7ffbcad02700 1 -- 172.21.2.112:0/960531431 >> v1:172.21.2.120:6789/0 conn(0x7ffbc40e5750 legacy=0x7ffbc410d770 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:39:18.201 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.192 7ffbcad02700 1 -- 172.21.2.112:0/960531431 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbc41b0be0 con 0x7ffbc410fce0 2021-02-10T11:39:18.201 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.192 7ffbcad02700 1 --2- 172.21.2.112:0/960531431 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ffbc410fce0 0x7ffbc410df60 secure :-1 s=READY pgs=1898 cs=0 l=1 rev1=1 rx=0x7ffbbc006f00 tx=0x7ffbbc009f10).ready entity=mon.0 client_cookie=6becb622d3a24611 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:18.201 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.193 7ffbc94ff700 1 -- 172.21.2.112:0/960531431 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffbbc00b3b0 con 0x7ffbc410fce0 2021-02-10T11:39:18.201 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.193 7ffbc94ff700 1 -- 172.21.2.112:0/960531431 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbbc00b630 con 0x7ffbc410fce0 2021-02-10T11:39:18.202 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.193 7ffbc94ff700 1 -- 172.21.2.112:0/960531431 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffbbc00bb50 con 0x7ffbc410fce0 2021-02-10T11:39:18.202 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.193 7ffbcc6e5700 1 -- 172.21.2.112:0/960531431 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ffbc410fce0 msgr2=0x7ffbc410df60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:18.202 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.193 7ffbcc6e5700 1 --2- 172.21.2.112:0/960531431 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ffbc410fce0 0x7ffbc410df60 secure :-1 s=READY pgs=1898 cs=0 l=1 rev1=1 rx=0x7ffbbc006f00 tx=0x7ffbbc009f10).stop 2021-02-10T11:39:18.202 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.193 7ffbcc6e5700 1 -- 172.21.2.112:0/960531431 shutdown_connections 2021-02-10T11:39:18.202 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.193 7ffbcc6e5700 1 --2- 172.21.2.112:0/960531431 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ffbc410fce0 0x7ffbc410df60 unknown :-1 s=CLOSED pgs=1898 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:18.202 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.193 7ffbcc6e5700 1 --2- 172.21.2.112:0/960531431 >> v2:172.21.2.120:3300/0 conn(0x7ffbc40e5350 0x7ffbc410d150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:18.202 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.193 7ffbcc6e5700 1 -- 172.21.2.112:0/960531431 >> 172.21.2.112:0/960531431 conn(0x7ffbc40cea00 msgr2=0x7ffbc40e4840 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:18.202 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.193 7ffbcc6e5700 1 -- 172.21.2.112:0/960531431 shutdown_connections 2021-02-10T11:39:18.202 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.194 7ffbcc6e5700 1 -- 172.21.2.112:0/960531431 wait complete. 2021-02-10T11:39:18.204 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.208 7ffbcc6e5700 1 Processor -- start 2021-02-10T11:39:18.204 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.208 7ffbcc6e5700 1 -- start start 2021-02-10T11:39:18.204 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.208 7ffbcc6e5700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ffbc40b0e80 0x7ffbc408fde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:18.204 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.208 7ffbcc6e5700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ffbc4191b70 0x7ffbc409b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:18.204 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.208 7ffbcc6e5700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ffbc4191f70 0x7ffbc409be70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:18.204 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.208 7ffbcc6e5700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbc4094cd0 con 0x7ffbc4191f70 2021-02-10T11:39:18.204 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.208 7ffbcc6e5700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbc4190270 con 0x7ffbc40b0e80 2021-02-10T11:39:18.204 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.208 7ffbcc6e5700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbc4190490 con 0x7ffbc4191b70 2021-02-10T11:39:18.204 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.208 7ffbcad02700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ffbc4191f70 0x7ffbc409be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:18.204 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.208 7ffbcad02700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ffbc4191f70 0x7ffbc409be70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:53666/0 (socket says 172.21.2.112:53666) 2021-02-10T11:39:18.205 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.208 7ffbcad02700 1 -- 172.21.2.112:0/1237685323 learned_addr learned my addr 172.21.2.112:0/1237685323 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:39:18.205 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.209 7ffbcad02700 1 -- 172.21.2.112:0/1237685323 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ffbc4191b70 msgr2=0x7ffbc409b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:18.205 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.209 7ffbcad02700 1 --2- 172.21.2.112:0/1237685323 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ffbc4191b70 0x7ffbc409b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:18.205 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.209 7ffbcad02700 1 -- 172.21.2.112:0/1237685323 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ffbc40b0e80 msgr2=0x7ffbc408fde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:18.206 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.209 7ffbcad02700 1 --2- 172.21.2.112:0/1237685323 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ffbc40b0e80 0x7ffbc408fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:18.206 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.209 7ffbcad02700 1 -- 172.21.2.112:0/1237685323 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbc41906b0 con 0x7ffbc4191f70 2021-02-10T11:39:18.206 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.209 7ffbcad02700 1 --2- 172.21.2.112:0/1237685323 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ffbc4191f70 0x7ffbc409be70 secure :-1 s=READY pgs=1899 cs=0 l=1 rev1=1 rx=0x7ffbbc009390 tx=0x7ffbbc00bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:18.206 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.209 7ffbb6ffd700 1 -- 172.21.2.112:0/1237685323 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffbbc009450 con 0x7ffbc4191f70 2021-02-10T11:39:18.206 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.209 7ffbb6ffd700 1 -- 172.21.2.112:0/1237685323 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbbc003030 con 0x7ffbc4191f70 2021-02-10T11:39:18.206 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.209 7ffbb6ffd700 1 -- 172.21.2.112:0/1237685323 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffbbc001480 con 0x7ffbc4191f70 2021-02-10T11:39:18.206 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.209 7ffbcc6e5700 1 -- 172.21.2.112:0/1237685323 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbc41906b0 con 0x7ffbc4191f70 2021-02-10T11:39:18.206 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.209 7ffbcc6e5700 1 -- 172.21.2.112:0/1237685323 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbc4190c40 con 0x7ffbc4191f70 2021-02-10T11:39:18.206 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.210 7ffbb6ffd700 1 -- 172.21.2.112:0/1237685323 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ffbbc001270 con 0x7ffbc4191f70 2021-02-10T11:39:18.206 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.210 7ffbb6ffd700 1 --2- 172.21.2.112:0/1237685323 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ffbac0186c0 0x7ffbac01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:18.206 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.210 7ffbb6ffd700 1 -- 172.21.2.112:0/1237685323 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5573+0+0 (secure 0 0 0) 0x7ffbbc032760 con 0x7ffbc4191f70 2021-02-10T11:39:18.206 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.210 7ffbca501700 1 --2- 172.21.2.112:0/1237685323 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ffbac0186c0 0x7ffbac01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:18.206 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.211 7ffbca501700 1 --2- 172.21.2.112:0/1237685323 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ffbac0186c0 0x7ffbac01abc0 crc :-1 s=READY pgs=890 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:18.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.242 7ffbcc6e5700 1 -- 172.21.2.112:0/1237685323 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbc4191640 con 0x7ffbc4191f70 2021-02-10T11:39:18.237 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.245 7ffbb6ffd700 1 -- 172.21.2.112:0/1237685323 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ffbbc0093d0 con 0x7ffbc4191f70 2021-02-10T11:39:18.368 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.375 7ffbcc6e5700 1 -- 172.21.2.112:0/1237685323 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.guest1", "caps": ["mds", "allow rw path=/volumes/group_0000000000143913, allow rw path=/volumes/group_0000000000143913/subvolume_0000000000918084/6682d7c1-1053-4b9a-9c54-24970fa60db6", "osd", "allow rw pool=cephfs_data", "mon", "allow r", "mgr", "allow *"]} v 0) v1 -- 0x7ffbc4191640 con 0x7ffbc4191f70 2021-02-10T11:39:18.371 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.378 7ffbb6ffd700 1 -- 172.21.2.112:0/1237685323 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.guest1", "caps": ["mds", "allow rw path=/volumes/group_0000000000143913, allow rw path=/volumes/group_0000000000143913/subvolume_0000000000918084/6682d7c1-1053-4b9a-9c54-24970fa60db6", "osd", "allow rw pool=cephfs_data", "mon", "allow r", "mgr", "allow *"]}]=0 updated caps for client.guest1 v58) v1 ==== 363+0+0 (secure 0 0 0) 0x7ffbbc01f8c0 con 0x7ffbc4191f70 2021-02-10T11:39:18.371 INFO:teuthology.orchestra.run.gibba012.stderr:updated caps for client.guest1 2021-02-10T11:39:18.373 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.380 7ffbcc6e5700 1 -- 172.21.2.112:0/1237685323 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ffbac0186c0 msgr2=0x7ffbac01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:18.373 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.380 7ffbcc6e5700 1 --2- 172.21.2.112:0/1237685323 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ffbac0186c0 0x7ffbac01abc0 crc :-1 s=READY pgs=890 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:18.373 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.380 7ffbcc6e5700 1 -- 172.21.2.112:0/1237685323 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ffbc4191f70 msgr2=0x7ffbc409be70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:18.373 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.380 7ffbcc6e5700 1 --2- 172.21.2.112:0/1237685323 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ffbc4191f70 0x7ffbc409be70 secure :-1 s=READY pgs=1899 cs=0 l=1 rev1=1 rx=0x7ffbbc009390 tx=0x7ffbbc00bd40).stop 2021-02-10T11:39:18.373 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.380 7ffbcc6e5700 1 -- 172.21.2.112:0/1237685323 shutdown_connections 2021-02-10T11:39:18.373 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.380 7ffbcc6e5700 1 --2- 172.21.2.112:0/1237685323 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ffbac0186c0 0x7ffbac01abc0 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:18.373 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.380 7ffbcc6e5700 1 --2- 172.21.2.112:0/1237685323 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ffbc4191f70 0x7ffbc409be70 unknown :-1 s=CLOSED pgs=1899 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:18.374 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.380 7ffbcc6e5700 1 --2- 172.21.2.112:0/1237685323 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ffbc4191b70 0x7ffbc409b850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:18.374 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.380 7ffbcc6e5700 1 --2- 172.21.2.112:0/1237685323 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ffbc40b0e80 0x7ffbc408fde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:18.374 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.380 7ffbcc6e5700 1 -- 172.21.2.112:0/1237685323 >> 172.21.2.112:0/1237685323 conn(0x7ffbc409b320 msgr2=0x7ffbc4095740 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:18.374 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.380 7ffbcc6e5700 1 -- 172.21.2.112:0/1237685323 shutdown_connections 2021-02-10T11:39:18.374 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.380 7ffbcc6e5700 1 -- 172.21.2.112:0/1237685323 wait complete. 2021-02-10T11:39:18.381 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume deauthorize cephfs subvolume_0000000000918084 guest1 --group_name group_0000000000143913 2021-02-10T11:39:18.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.803 7f6a23c8a700 1 Processor -- start 2021-02-10T11:39:18.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.803 7f6a23c8a700 1 -- start start 2021-02-10T11:39:18.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.803 7f6a23c8a700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f6a1c0dce80 0x7f6a1c0d14f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:18.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.803 7f6a23c8a700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f6a1c0eb220 0x7f6a1c0eb620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:18.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.803 7f6a23c8a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6a1c10aad0 0x7f6a1c10aed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:18.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.803 7f6a23c8a700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a1c10b570 con 0x7f6a1c0eb220 2021-02-10T11:39:18.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.803 7f6a23c8a700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a1c10a590 con 0x7f6a1c0dce80 2021-02-10T11:39:18.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.803 7f6a23c8a700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a1c0881d0 con 0x7f6a1c10aad0 2021-02-10T11:39:18.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.803 7f6a212a5700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f6a1c0eb220 0x7f6a1c0eb620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:18.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.803 7f6a212a5700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f6a1c0eb220 0x7f6a1c0eb620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:53670/0 (socket says 172.21.2.112:53670) 2021-02-10T11:39:18.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.803 7f6a212a5700 1 -- 172.21.2.112:0/3043412310 learned_addr learned my addr 172.21.2.112:0/3043412310 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:39:18.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.804 7f6a212a5700 1 -- 172.21.2.112:0/3043412310 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6a1c10aad0 msgr2=0x7f6a1c10aed0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:39:18.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.804 7f6a212a5700 1 --2- 172.21.2.112:0/3043412310 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6a1c10aad0 0x7f6a1c10aed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:18.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.804 7f6a212a5700 1 -- 172.21.2.112:0/3043412310 >> v2:172.21.2.120:3300/0 conn(0x7f6a1c0dce80 msgr2=0x7f6a1c0d14f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:39:18.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.804 7f6a212a5700 1 --2- 172.21.2.112:0/3043412310 >> v2:172.21.2.120:3300/0 conn(0x7f6a1c0dce80 0x7f6a1c0d14f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:18.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.804 7f6a212a5700 1 -- 172.21.2.112:0/3043412310 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a1c0d1f30 con 0x7f6a1c0eb220 2021-02-10T11:39:18.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.804 7f6a212a5700 1 --2- 172.21.2.112:0/3043412310 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6a1c0eb220 0x7f6a1c0eb620 secure :-1 s=READY pgs=1900 cs=0 l=1 rev1=1 rx=0x7f6a18006700 tx=0x7f6a18009b80).ready entity=mon.0 client_cookie=f5ffe7778f6edd83 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:18.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.804 7f6a20aa4700 1 -- 172.21.2.112:0/3043412310 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6a1800c0c0 con 0x7f6a1c0eb220 2021-02-10T11:39:18.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.804 7f6a20aa4700 1 -- 172.21.2.112:0/3043412310 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a1800c300 con 0x7f6a1c0eb220 2021-02-10T11:39:18.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.804 7f6a20aa4700 1 -- 172.21.2.112:0/3043412310 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6a1800c840 con 0x7f6a1c0eb220 2021-02-10T11:39:18.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.804 7f6a23c8a700 1 -- 172.21.2.112:0/3043412310 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6a1c0eb220 msgr2=0x7f6a1c0eb620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:18.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.804 7f6a23c8a700 1 --2- 172.21.2.112:0/3043412310 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6a1c0eb220 0x7f6a1c0eb620 secure :-1 s=READY pgs=1900 cs=0 l=1 rev1=1 rx=0x7f6a18006700 tx=0x7f6a18009b80).stop 2021-02-10T11:39:18.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.805 7f6a23c8a700 1 -- 172.21.2.112:0/3043412310 shutdown_connections 2021-02-10T11:39:18.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.805 7f6a23c8a700 1 --2- 172.21.2.112:0/3043412310 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6a1c10aad0 0x7f6a1c10aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:18.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.805 7f6a23c8a700 1 --2- 172.21.2.112:0/3043412310 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6a1c0eb220 0x7f6a1c0eb620 unknown :-1 s=CLOSED pgs=1900 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:18.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.805 7f6a23c8a700 1 --2- 172.21.2.112:0/3043412310 >> v2:172.21.2.120:3300/0 conn(0x7f6a1c0dce80 0x7f6a1c0d14f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:18.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.805 7f6a23c8a700 1 -- 172.21.2.112:0/3043412310 >> 172.21.2.112:0/3043412310 conn(0x7f6a1c0ea630 msgr2=0x7f6a1c0d0800 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:18.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.805 7f6a23c8a700 1 -- 172.21.2.112:0/3043412310 shutdown_connections 2021-02-10T11:39:18.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.805 7f6a23c8a700 1 -- 172.21.2.112:0/3043412310 wait complete. 2021-02-10T11:39:18.816 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.819 7f6a23c8a700 1 Processor -- start 2021-02-10T11:39:18.816 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.819 7f6a23c8a700 1 -- start start 2021-02-10T11:39:18.816 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.820 7f6a23c8a700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6a1c191630 0x7f6a1c0abae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:18.816 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.820 7f6a23c8a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6a1c0ac100 0x7f6a1c103380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:18.816 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.820 7f6a23c8a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6a1c1aa320 0x7f6a1c1039a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:18.816 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.820 7f6a23c8a700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a1c1a9da0 con 0x7f6a1c1aa320 2021-02-10T11:39:18.816 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.820 7f6a23c8a700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a1c103fc0 con 0x7f6a1c191630 2021-02-10T11:39:18.816 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.820 7f6a23c8a700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a1c1041a0 con 0x7f6a1c0ac100 2021-02-10T11:39:18.816 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.820 7f6a222a7700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6a1c1aa320 0x7f6a1c1039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:18.816 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.820 7f6a222a7700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6a1c1aa320 0x7f6a1c1039a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:53678/0 (socket says 172.21.2.112:53678) 2021-02-10T11:39:18.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.820 7f6a222a7700 1 -- 172.21.2.112:0/3823901930 learned_addr learned my addr 172.21.2.112:0/3823901930 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:39:18.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.820 7f6a222a7700 1 -- 172.21.2.112:0/3823901930 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6a1c0ac100 msgr2=0x7f6a1c103380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:18.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.820 7f6a222a7700 1 --2- 172.21.2.112:0/3823901930 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6a1c0ac100 0x7f6a1c103380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:18.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.820 7f6a222a7700 1 -- 172.21.2.112:0/3823901930 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6a1c191630 msgr2=0x7f6a1c0abae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:18.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.820 7f6a222a7700 1 --2- 172.21.2.112:0/3823901930 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6a1c191630 0x7f6a1c0abae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:18.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.820 7f6a222a7700 1 -- 172.21.2.112:0/3823901930 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a1c1a8710 con 0x7f6a1c1aa320 2021-02-10T11:39:18.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.821 7f6a222a7700 1 --2- 172.21.2.112:0/3823901930 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6a1c1aa320 0x7f6a1c1039a0 secure :-1 s=READY pgs=1901 cs=0 l=1 rev1=1 rx=0x7f6a14009960 tx=0x7f6a1400bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:18.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.821 7f6a0e7fc700 1 -- 172.21.2.112:0/3823901930 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6a14000f90 con 0x7f6a1c1aa320 2021-02-10T11:39:18.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.821 7f6a0e7fc700 1 -- 172.21.2.112:0/3823901930 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a14003030 con 0x7f6a1c1aa320 2021-02-10T11:39:18.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.821 7f6a0e7fc700 1 -- 172.21.2.112:0/3823901930 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6a14001590 con 0x7f6a1c1aa320 2021-02-10T11:39:18.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.821 7f6a23c8a700 1 -- 172.21.2.112:0/3823901930 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a1c1a8710 con 0x7f6a1c1aa320 2021-02-10T11:39:18.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.821 7f6a23c8a700 1 -- 172.21.2.112:0/3823901930 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a1c1a8710 con 0x7f6a1c1aa320 2021-02-10T11:39:18.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.821 7f6a0e7fc700 1 -- 172.21.2.112:0/3823901930 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6a14001310 con 0x7f6a1c1aa320 2021-02-10T11:39:18.818 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.822 7f6a0e7fc700 1 --2- 172.21.2.112:0/3823901930 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6a040186c0 0x7f6a0401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:18.818 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.822 7f6a0e7fc700 1 -- 172.21.2.112:0/3823901930 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f6a140327c0 con 0x7f6a1c1aa320 2021-02-10T11:39:18.818 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.822 7f6a21aa6700 1 --2- 172.21.2.112:0/3823901930 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6a040186c0 0x7f6a0401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:18.818 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.822 7f6a21aa6700 1 --2- 172.21.2.112:0/3823901930 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6a040186c0 0x7f6a0401abc0 crc :-1 s=READY pgs=891 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:18.846 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.853 7f6a23c8a700 1 -- 172.21.2.112:0/3823901930 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a1c10a550 con 0x7f6a1c1aa320 2021-02-10T11:39:18.848 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.855 7f6a0e7fc700 1 -- 172.21.2.112:0/3823901930 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6a14000f90 con 0x7f6a1c1aa320 2021-02-10T11:39:18.975 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:18.982 7f6a23c8a700 1 -- 172.21.2.112:0/3823901930 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume deauthorize", "vol_name": "cephfs", "sub_name": "subvolume_0000000000918084", "target": ["mgr", ""], "auth_id": "guest1", "group_name": "group_0000000000143913"}) v1 -- 0x7f6a1c10a550 con 0x7f6a040186c0 2021-02-10T11:39:19.011 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:19.018 7f6a0e7fc700 1 -- 172.21.2.112:0/3823901930 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f6a10006e60 con 0x7f6a040186c0 2021-02-10T11:39:19.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:19.048 7f6a23c8a700 1 -- 172.21.2.112:0/3823901930 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6a040186c0 msgr2=0x7f6a0401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:19.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:19.048 7f6a23c8a700 1 --2- 172.21.2.112:0/3823901930 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6a040186c0 0x7f6a0401abc0 crc :-1 s=READY pgs=891 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:19.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:19.048 7f6a23c8a700 1 -- 172.21.2.112:0/3823901930 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6a1c1aa320 msgr2=0x7f6a1c1039a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:19.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:19.048 7f6a23c8a700 1 --2- 172.21.2.112:0/3823901930 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6a1c1aa320 0x7f6a1c1039a0 secure :-1 s=READY pgs=1901 cs=0 l=1 rev1=1 rx=0x7f6a14009960 tx=0x7f6a1400bcf0).stop 2021-02-10T11:39:19.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:19.048 7f6a23c8a700 1 -- 172.21.2.112:0/3823901930 shutdown_connections 2021-02-10T11:39:19.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:19.048 7f6a23c8a700 1 --2- 172.21.2.112:0/3823901930 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6a040186c0 0x7f6a0401abc0 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:19.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:19.048 7f6a23c8a700 1 --2- 172.21.2.112:0/3823901930 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6a1c1aa320 0x7f6a1c1039a0 unknown :-1 s=CLOSED pgs=1901 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:19.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:19.048 7f6a23c8a700 1 --2- 172.21.2.112:0/3823901930 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6a1c0ac100 0x7f6a1c103380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:19.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:19.048 7f6a23c8a700 1 --2- 172.21.2.112:0/3823901930 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6a1c191630 0x7f6a1c0abae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:19.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:19.048 7f6a23c8a700 1 -- 172.21.2.112:0/3823901930 >> 172.21.2.112:0/3823901930 conn(0x7f6a1c19b3a0 msgr2=0x7f6a1c102370 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:19.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:19.048 7f6a23c8a700 1 -- 172.21.2.112:0/3823901930 shutdown_connections 2021-02-10T11:39:19.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:19.048 7f6a23c8a700 1 -- 172.21.2.112:0/3823901930 wait complete. 2021-02-10T11:39:19.049 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth get client.guest1 --format=json-pretty 2021-02-10T11:39:19.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:19.465 7f91bc286700 1 Processor -- start 2021-02-10T11:39:19.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:19.465 7f91bc286700 1 -- start start 2021-02-10T11:39:19.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:19.465 7f91bc286700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f91b40d1870 0x7f91b41b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:19.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:19.465 7f91bc286700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f91b40f0650 0x7f91b40cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:19.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:19.465 7f91bc286700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91b40f0b10 con 0x7f91b41b0a10 2021-02-10T11:39:19.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:19.465 7f91bc286700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91b40e5ed0 con 0x7f91b40f0650 2021-02-10T11:39:19.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:19.465 7f91bc286700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91b40d1c70 con 0x7f91b40d1870 2021-02-10T11:39:19.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:19.466 7f91ba8a3700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f91b40f0650 0x7f91b40cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:19.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:19.466 7f91ba8a3700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f91b40f0650 0x7f91b40cfb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:53682/0 (socket says 172.21.2.112:53682) 2021-02-10T11:39:19.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:19.466 7f91ba8a3700 1 -- 172.21.2.112:0/3113818266 learned_addr learned my addr 172.21.2.112:0/3113818266 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:39:19.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:19.466 7f91ba0a2700 1 --2- 172.21.2.112:0/3113818266 >> v2:172.21.2.120:3300/0 conn(0x7f91b40d1870 0x7f91b41b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:19.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:19.466 7f91ba8a3700 1 -- 172.21.2.112:0/3113818266 >> v2:172.21.2.120:3300/0 conn(0x7f91b40d1870 msgr2=0x7f91b41b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:19.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:19.466 7f91ba8a3700 1 --2- 172.21.2.112:0/3113818266 >> v2:172.21.2.120:3300/0 conn(0x7f91b40d1870 0x7f91b41b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:19.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:19.466 7f91ba8a3700 1 -- 172.21.2.112:0/3113818266 >> v1:172.21.2.120:6789/0 conn(0x7f91b41b0a10 legacy=0x7f91b40cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:19.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:19.466 7f91ba8a3700 1 -- 172.21.2.112:0/3113818266 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91b40d1e90 con 0x7f91b40f0650 2021-02-10T11:39:19.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:19.466 7f91ba8a3700 1 --2- 172.21.2.112:0/3113818266 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f91b40f0650 0x7f91b40cfb20 secure :-1 s=READY pgs=1902 cs=0 l=1 rev1=1 rx=0x7f91ac006f00 tx=0x7f91ac009f10).ready entity=mon.0 client_cookie=59f28ac01d1e0002 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:19.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:19.466 7f91b90a0700 1 -- 172.21.2.112:0/3113818266 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f91ac00b3b0 con 0x7f91b40f0650 2021-02-10T11:39:19.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:19.466 7f91b90a0700 1 -- 172.21.2.112:0/3113818266 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91ac00b630 con 0x7f91b40f0650 2021-02-10T11:39:19.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:19.466 7f91b90a0700 1 -- 172.21.2.112:0/3113818266 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f91ac00bb50 con 0x7f91b40f0650 2021-02-10T11:39:19.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:19.466 7f91bc286700 1 -- 172.21.2.112:0/3113818266 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f91b40f0650 msgr2=0x7f91b40cfb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:19.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:19.466 7f91bc286700 1 --2- 172.21.2.112:0/3113818266 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f91b40f0650 0x7f91b40cfb20 secure :-1 s=READY pgs=1902 cs=0 l=1 rev1=1 rx=0x7f91ac006f00 tx=0x7f91ac009f10).stop 2021-02-10T11:39:19.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:19.467 7f91bc286700 1 -- 172.21.2.112:0/3113818266 shutdown_connections 2021-02-10T11:39:19.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:19.467 7f91bc286700 1 --2- 172.21.2.112:0/3113818266 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f91b40f0650 0x7f91b40cfb20 unknown :-1 s=CLOSED pgs=1902 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:19.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:19.467 7f91bc286700 1 --2- 172.21.2.112:0/3113818266 >> v2:172.21.2.120:3300/0 conn(0x7f91b40d1870 0x7f91b41b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:19.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:19.467 7f91bc286700 1 -- 172.21.2.112:0/3113818266 >> 172.21.2.112:0/3113818266 conn(0x7f91b40ce450 msgr2=0x7f91b40d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:19.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:19.467 7f91bc286700 1 -- 172.21.2.112:0/3113818266 shutdown_connections 2021-02-10T11:39:19.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:19.467 7f91bc286700 1 -- 172.21.2.112:0/3113818266 wait complete. 2021-02-10T11:39:19.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:19.481 7f91bc286700 1 Processor -- start 2021-02-10T11:39:19.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:19.481 7f91bc286700 1 -- start start 2021-02-10T11:39:19.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:19.482 7f91bc286700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f91b40b7510 0x7f91b40b7910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:19.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:19.482 7f91bc286700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f91b409d530 0x7f91b409d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:19.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:19.482 7f91bc286700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f91b409df50 0x7f91b409eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:19.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:19.482 7f91bc286700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91b40b8140 con 0x7f91b409df50 2021-02-10T11:39:19.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:19.482 7f91bc286700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91b409e750 con 0x7f91b409d530 2021-02-10T11:39:19.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:19.482 7f91bc286700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91b409e350 con 0x7f91b40b7510 2021-02-10T11:39:19.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:19.482 7f91ba8a3700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f91b409df50 0x7f91b409eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:19.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:19.482 7f91ba8a3700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f91b409df50 0x7f91b409eca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:53686/0 (socket says 172.21.2.112:53686) 2021-02-10T11:39:19.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:19.482 7f91ba8a3700 1 -- 172.21.2.112:0/3873265603 learned_addr learned my addr 172.21.2.112:0/3873265603 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:39:19.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:19.482 7f91ba8a3700 1 -- 172.21.2.112:0/3873265603 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f91b40b7510 msgr2=0x7f91b40b7910 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:39:19.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:19.482 7f91ba8a3700 1 --2- 172.21.2.112:0/3873265603 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f91b40b7510 0x7f91b40b7910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:19.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:19.482 7f91ba8a3700 1 -- 172.21.2.112:0/3873265603 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f91b409d530 msgr2=0x7f91b409d930 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:39:19.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:19.482 7f91ba8a3700 1 --2- 172.21.2.112:0/3873265603 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f91b409d530 0x7f91b409d930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:19.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:19.482 7f91ba8a3700 1 -- 172.21.2.112:0/3873265603 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91b409f2c0 con 0x7f91b409df50 2021-02-10T11:39:19.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:19.482 7f91ba8a3700 1 --2- 172.21.2.112:0/3873265603 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f91b409df50 0x7f91b409eca0 secure :-1 s=READY pgs=1903 cs=0 l=1 rev1=1 rx=0x7f91ac009390 tx=0x7f91ac00bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:19.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:19.482 7f91a6ffd700 1 -- 172.21.2.112:0/3873265603 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f91ac009450 con 0x7f91b409df50 2021-02-10T11:39:19.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:19.483 7f91a6ffd700 1 -- 172.21.2.112:0/3873265603 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91ac0068e0 con 0x7f91b409df50 2021-02-10T11:39:19.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:19.483 7f91a6ffd700 1 -- 172.21.2.112:0/3873265603 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f91ac00c4e0 con 0x7f91b409df50 2021-02-10T11:39:19.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:19.483 7f91bc286700 1 -- 172.21.2.112:0/3873265603 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91b409f2c0 con 0x7f91b409df50 2021-02-10T11:39:19.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:19.483 7f91bc286700 1 -- 172.21.2.112:0/3873265603 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91b409f2c0 con 0x7f91b409df50 2021-02-10T11:39:19.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:19.483 7f91a6ffd700 1 -- 172.21.2.112:0/3873265603 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f91ac001270 con 0x7f91b409df50 2021-02-10T11:39:19.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:19.483 7f91a6ffd700 1 --2- 172.21.2.112:0/3873265603 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f919c0186c0 0x7f919c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:19.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:19.483 7f91a6ffd700 1 -- 172.21.2.112:0/3873265603 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f91ac032b00 con 0x7f91b409df50 2021-02-10T11:39:19.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:19.484 7f91ba0a2700 1 --2- 172.21.2.112:0/3873265603 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f919c0186c0 0x7f919c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:19.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:19.484 7f91ba0a2700 1 --2- 172.21.2.112:0/3873265603 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f919c0186c0 0x7f919c01abc0 crc :-1 s=READY pgs=892 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:19.509 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:19.515 7f91bc286700 1 -- 172.21.2.112:0/3873265603 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91b40ebc50 con 0x7f91b409df50 2021-02-10T11:39:19.511 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:19.518 7f91a6ffd700 1 -- 172.21.2.112:0/3873265603 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f91ac009450 con 0x7f91b409df50 2021-02-10T11:39:19.642 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:19.649 7f91bc286700 1 -- 172.21.2.112:0/3873265603 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "auth get", "entity": "client.guest1", "format": "json-pretty"} v 0) v1 -- 0x7f91b40f0b10 con 0x7f91b409df50 2021-02-10T11:39:19.643 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:19.650 7f91a6ffd700 1 -- 172.21.2.112:0/3873265603 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth get", "entity": "client.guest1", "format": "json-pretty"}]=0 exported keyring for client.guest1 v59) v1 ==== 142+0+235 (secure 0 0 0) 0x7f91ac0068e0 con 0x7f91b409df50 2021-02-10T11:39:19.643 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:39:19.643 INFO:teuthology.orchestra.run.gibba012.stdout:[ 2021-02-10T11:39:19.644 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:39:19.644 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.guest1", 2021-02-10T11:39:19.644 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQDlxSNgMaQDBhAAO5GRoC6uEX0RAKa15nbbEQ==", 2021-02-10T11:39:19.644 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:39:19.644 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow rw path=/volumes/group_0000000000143913", 2021-02-10T11:39:19.644 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *" 2021-02-10T11:39:19.644 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:39:19.644 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:39:19.644 INFO:teuthology.orchestra.run.gibba012.stdout:] 2021-02-10T11:39:19.644 INFO:teuthology.orchestra.run.gibba012.stderr:exported keyring for client.guest1 2021-02-10T11:39:19.645 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:19.652 7f91bc286700 1 -- 172.21.2.112:0/3873265603 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f919c0186c0 msgr2=0x7f919c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:19.645 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:19.652 7f91bc286700 1 --2- 172.21.2.112:0/3873265603 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f919c0186c0 0x7f919c01abc0 crc :-1 s=READY pgs=892 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:19.645 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:19.652 7f91bc286700 1 -- 172.21.2.112:0/3873265603 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f91b409df50 msgr2=0x7f91b409eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:19.645 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:19.652 7f91bc286700 1 --2- 172.21.2.112:0/3873265603 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f91b409df50 0x7f91b409eca0 secure :-1 s=READY pgs=1903 cs=0 l=1 rev1=1 rx=0x7f91ac009390 tx=0x7f91ac00bd40).stop 2021-02-10T11:39:19.645 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:19.652 7f91bc286700 1 -- 172.21.2.112:0/3873265603 shutdown_connections 2021-02-10T11:39:19.645 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:19.652 7f91bc286700 1 --2- 172.21.2.112:0/3873265603 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f919c0186c0 0x7f919c01abc0 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:19.646 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:19.652 7f91bc286700 1 --2- 172.21.2.112:0/3873265603 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f91b409d530 0x7f91b409d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:19.646 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:19.652 7f91bc286700 1 --2- 172.21.2.112:0/3873265603 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f91b409df50 0x7f91b409eca0 unknown :-1 s=CLOSED pgs=1903 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:19.646 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:19.652 7f91bc286700 1 --2- 172.21.2.112:0/3873265603 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f91b40b7510 0x7f91b40b7910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:19.646 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:19.652 7f91bc286700 1 -- 172.21.2.112:0/3873265603 >> 172.21.2.112:0/3873265603 conn(0x7f91b40b83b0 msgr2=0x7f91b40a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:19.646 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:19.652 7f91bc286700 1 -- 172.21.2.112:0/3873265603 shutdown_connections 2021-02-10T11:39:19.646 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:19.652 7f91bc286700 1 -- 172.21.2.112:0/3873265603 wait complete. 2021-02-10T11:39:19.653 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth rm client.guest1 2021-02-10T11:39:20.079 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.070 7f9862a95700 1 Processor -- start 2021-02-10T11:39:20.079 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.070 7f9862a95700 1 -- start start 2021-02-10T11:39:20.080 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.070 7f9862a95700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f985c0d0340 0x7f985c0ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:20.080 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.070 7f9862a95700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f985c0d1c70 0x7f985c0efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:20.080 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.070 7f9862a95700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f985c0f0b10 con 0x7f985c0d1870 2021-02-10T11:39:20.080 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.070 7f9862a95700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f985c0e5ed0 con 0x7f985c0d1c70 2021-02-10T11:39:20.080 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.070 7f9862a95700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f985c1b0be0 con 0x7f985c0d0340 2021-02-10T11:39:20.080 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.071 7f98610b2700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f985c0d1c70 0x7f985c0efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:20.080 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.071 7f98610b2700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f985c0d1c70 0x7f985c0efc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:53696/0 (socket says 172.21.2.112:53696) 2021-02-10T11:39:20.080 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.071 7f98610b2700 1 -- 172.21.2.112:0/984110345 learned_addr learned my addr 172.21.2.112:0/984110345 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:39:20.080 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.071 7f98610b2700 1 -- 172.21.2.112:0/984110345 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f985c0d0340 msgr2=0x7f985c0ef640 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:39:20.080 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.071 7f98610b2700 1 --2- 172.21.2.112:0/984110345 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f985c0d0340 0x7f985c0ef640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:20.080 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.071 7f98610b2700 1 -- 172.21.2.112:0/984110345 >> v1:172.21.2.120:6789/0 conn(0x7f985c0d1870 legacy=0x7f985c1b03f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:39:20.080 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.071 7f98610b2700 1 -- 172.21.2.112:0/984110345 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f985c0f20a0 con 0x7f985c0d1c70 2021-02-10T11:39:20.081 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.071 7f98610b2700 1 --2- 172.21.2.112:0/984110345 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f985c0d1c70 0x7f985c0efc60 secure :-1 s=READY pgs=1904 cs=0 l=1 rev1=1 rx=0x7f9854006b80 tx=0x7f9854009b90).ready entity=mon.0 client_cookie=62c13c346dabb092 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:20.081 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.071 7f985b7fe700 1 -- 172.21.2.112:0/984110345 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f98540022e0 con 0x7f985c0d1c70 2021-02-10T11:39:20.081 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.071 7f985b7fe700 1 -- 172.21.2.112:0/984110345 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9854002560 con 0x7f985c0d1c70 2021-02-10T11:39:20.081 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.071 7f985b7fe700 1 -- 172.21.2.112:0/984110345 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9854002a80 con 0x7f985c0d1c70 2021-02-10T11:39:20.081 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.071 7f9862a95700 1 -- 172.21.2.112:0/984110345 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f985c0d1c70 msgr2=0x7f985c0efc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:20.081 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.071 7f9862a95700 1 --2- 172.21.2.112:0/984110345 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f985c0d1c70 0x7f985c0efc60 secure :-1 s=READY pgs=1904 cs=0 l=1 rev1=1 rx=0x7f9854006b80 tx=0x7f9854009b90).stop 2021-02-10T11:39:20.081 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.072 7f9862a95700 1 -- 172.21.2.112:0/984110345 shutdown_connections 2021-02-10T11:39:20.081 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.072 7f9862a95700 1 --2- 172.21.2.112:0/984110345 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f985c0d1c70 0x7f985c0efc60 unknown :-1 s=CLOSED pgs=1904 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:20.081 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.072 7f9862a95700 1 --2- 172.21.2.112:0/984110345 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f985c0d0340 0x7f985c0ef640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:20.081 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.072 7f9862a95700 1 -- 172.21.2.112:0/984110345 >> 172.21.2.112:0/984110345 conn(0x7f985c0ce450 msgr2=0x7f985c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:20.081 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.072 7f9862a95700 1 -- 172.21.2.112:0/984110345 shutdown_connections 2021-02-10T11:39:20.081 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.072 7f9862a95700 1 -- 172.21.2.112:0/984110345 wait complete. 2021-02-10T11:39:20.083 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.087 7f9862a95700 1 Processor -- start 2021-02-10T11:39:20.083 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.087 7f9862a95700 1 -- start start 2021-02-10T11:39:20.083 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.087 7f9862a95700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f985c0a2fb0 0x7f985c0a33b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:20.083 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.087 7f9862a95700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f985c0a39d0 0x7f985c0a4860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:20.083 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.087 7f9862a95700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f985c0a3dd0 0x7f985c0a4e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:20.083 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.087 7f9862a95700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f985c0a2620 con 0x7f985c0a3dd0 2021-02-10T11:39:20.083 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.087 7f9862a95700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f985c0a2840 con 0x7f985c0a2fb0 2021-02-10T11:39:20.084 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.087 7f9862a95700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f985c0a2b80 con 0x7f985c0a39d0 2021-02-10T11:39:20.084 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.087 7f98610b2700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f985c0a3dd0 0x7f985c0a4e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:20.084 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.087 7f98610b2700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f985c0a3dd0 0x7f985c0a4e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:53698/0 (socket says 172.21.2.112:53698) 2021-02-10T11:39:20.084 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.087 7f98610b2700 1 -- 172.21.2.112:0/3914601134 learned_addr learned my addr 172.21.2.112:0/3914601134 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:39:20.084 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.088 7f98610b2700 1 -- 172.21.2.112:0/3914601134 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f985c0a39d0 msgr2=0x7f985c0a4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:20.084 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.088 7f98610b2700 1 --2- 172.21.2.112:0/3914601134 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f985c0a39d0 0x7f985c0a4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:20.084 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.088 7f98610b2700 1 -- 172.21.2.112:0/3914601134 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f985c0a2fb0 msgr2=0x7f985c0a33b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:20.084 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.088 7f98610b2700 1 --2- 172.21.2.112:0/3914601134 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f985c0a2fb0 0x7f985c0a33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:20.084 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.088 7f98610b2700 1 -- 172.21.2.112:0/3914601134 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f985c0a54a0 con 0x7f985c0a3dd0 2021-02-10T11:39:20.084 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.088 7f98610b2700 1 --2- 172.21.2.112:0/3914601134 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f985c0a3dd0 0x7f985c0a4e80 secure :-1 s=READY pgs=1905 cs=0 l=1 rev1=1 rx=0x7f9854009640 tx=0x7f985400b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:20.084 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.088 7f98597fa700 1 -- 172.21.2.112:0/3914601134 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f985400bd50 con 0x7f985c0a3dd0 2021-02-10T11:39:20.084 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.088 7f98597fa700 1 -- 172.21.2.112:0/3914601134 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9854006240 con 0x7f985c0a3dd0 2021-02-10T11:39:20.084 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.088 7f98597fa700 1 -- 172.21.2.112:0/3914601134 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f98540054a0 con 0x7f985c0a3dd0 2021-02-10T11:39:20.085 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.088 7f9862a95700 1 -- 172.21.2.112:0/3914601134 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f985c0a54a0 con 0x7f985c0a3dd0 2021-02-10T11:39:20.085 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.088 7f9862a95700 1 -- 172.21.2.112:0/3914601134 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f985c0a5a30 con 0x7f985c0a3dd0 2021-02-10T11:39:20.085 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.089 7f98597fa700 1 -- 172.21.2.112:0/3914601134 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9854006420 con 0x7f985c0a3dd0 2021-02-10T11:39:20.085 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.089 7f98597fa700 1 --2- 172.21.2.112:0/3914601134 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f98400186c0 0x7f984001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:20.085 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.089 7f98597fa700 1 -- 172.21.2.112:0/3914601134 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f9854032420 con 0x7f985c0a3dd0 2021-02-10T11:39:20.085 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.089 7f98608b1700 1 --2- 172.21.2.112:0/3914601134 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f98400186c0 0x7f984001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:20.085 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.090 7f98608b1700 1 --2- 172.21.2.112:0/3914601134 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f98400186c0 0x7f984001abc0 crc :-1 s=READY pgs=893 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:20.113 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.120 7f9862a95700 1 -- 172.21.2.112:0/3914601134 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f985c0ebc50 con 0x7f985c0a3dd0 2021-02-10T11:39:20.116 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.123 7f98597fa700 1 -- 172.21.2.112:0/3914601134 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9854026000 con 0x7f985c0a3dd0 2021-02-10T11:39:20.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.253 7f9862a95700 1 -- 172.21.2.112:0/3914601134 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "auth rm", "entity": "client.guest1"} v 0) v1 -- 0x7f985c0f0b10 con 0x7f985c0a3dd0 2021-02-10T11:39:20.248 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.256 7f98597fa700 1 -- 172.21.2.112:0/3914601134 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth rm", "entity": "client.guest1"}]=0 updated v60) v1 ==== 89+0+0 (secure 0 0 0) 0x7f9854026000 con 0x7f985c0a3dd0 2021-02-10T11:39:20.249 INFO:teuthology.orchestra.run.gibba012.stderr:updated 2021-02-10T11:39:20.250 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.257 7f9862a95700 1 -- 172.21.2.112:0/3914601134 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f98400186c0 msgr2=0x7f984001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:20.250 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.257 7f9862a95700 1 --2- 172.21.2.112:0/3914601134 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f98400186c0 0x7f984001abc0 crc :-1 s=READY pgs=893 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:20.250 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.257 7f9862a95700 1 -- 172.21.2.112:0/3914601134 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f985c0a3dd0 msgr2=0x7f985c0a4e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:20.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.257 7f9862a95700 1 --2- 172.21.2.112:0/3914601134 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f985c0a3dd0 0x7f985c0a4e80 secure :-1 s=READY pgs=1905 cs=0 l=1 rev1=1 rx=0x7f9854009640 tx=0x7f985400b960).stop 2021-02-10T11:39:20.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.258 7f9862a95700 1 -- 172.21.2.112:0/3914601134 shutdown_connections 2021-02-10T11:39:20.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.258 7f9862a95700 1 --2- 172.21.2.112:0/3914601134 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f98400186c0 0x7f984001abc0 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:20.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.258 7f9862a95700 1 --2- 172.21.2.112:0/3914601134 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f985c0a3dd0 0x7f985c0a4e80 unknown :-1 s=CLOSED pgs=1905 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:20.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.258 7f9862a95700 1 --2- 172.21.2.112:0/3914601134 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f985c0a39d0 0x7f985c0a4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:20.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.258 7f9862a95700 1 --2- 172.21.2.112:0/3914601134 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f985c0a2fb0 0x7f985c0a33b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:20.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.258 7f9862a95700 1 -- 172.21.2.112:0/3914601134 >> 172.21.2.112:0/3914601134 conn(0x7f985c08f030 msgr2=0x7f985c0b5e80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:20.252 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.258 7f9862a95700 1 -- 172.21.2.112:0/3914601134 shutdown_connections 2021-02-10T11:39:20.252 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.258 7f9862a95700 1 -- 172.21.2.112:0/3914601134 wait complete. 2021-02-10T11:39:20.259 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000918084 --group_name group_0000000000143913 2021-02-10T11:39:20.689 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.680 7f3096ee7700 1 Processor -- start 2021-02-10T11:39:20.689 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.681 7f3096ee7700 1 -- start start 2021-02-10T11:39:20.689 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.681 7f3096ee7700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f30900d1c80 0x7f30900cef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:20.690 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.681 7f3096ee7700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f30900cf590 0x7f30900cf990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:20.690 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.681 7f3096ee7700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30900fe1d0 con 0x7f30900d1880 2021-02-10T11:39:20.690 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.681 7f3096ee7700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30901b0cc0 con 0x7f30900cf590 2021-02-10T11:39:20.690 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.681 7f3096ee7700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3090109a70 con 0x7f30900d1c80 2021-02-10T11:39:20.690 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.681 7f3095504700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f30900cf590 0x7f30900cf990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:20.690 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.681 7f3095504700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f30900cf590 0x7f30900cf990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:53710/0 (socket says 172.21.2.112:53710) 2021-02-10T11:39:20.690 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.681 7f3095504700 1 -- 172.21.2.112:0/2339987287 learned_addr learned my addr 172.21.2.112:0/2339987287 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:39:20.690 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.681 7f3095504700 1 -- 172.21.2.112:0/2339987287 >> v2:172.21.2.120:3300/0 conn(0x7f30900d1c80 msgr2=0x7f30900cef70 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:39:20.690 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.681 7f3095504700 1 --2- 172.21.2.112:0/2339987287 >> v2:172.21.2.120:3300/0 conn(0x7f30900d1c80 0x7f30900cef70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:20.690 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.681 7f3095504700 1 -- 172.21.2.112:0/2339987287 >> v1:172.21.2.112:6789/0 conn(0x7f30900d1880 legacy=0x7f30901b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:20.690 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.681 7f3095504700 1 -- 172.21.2.112:0/2339987287 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3090088070 con 0x7f30900cf590 2021-02-10T11:39:20.690 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.681 7f3095504700 1 --2- 172.21.2.112:0/2339987287 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f30900cf590 0x7f30900cf990 secure :-1 s=READY pgs=1907 cs=0 l=1 rev1=1 rx=0x7f3088006f00 tx=0x7f3088009f10).ready entity=mon.0 client_cookie=9ae8e42f7418aeac server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:20.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.682 7f308f7fe700 1 -- 172.21.2.112:0/2339987287 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f308800b3b0 con 0x7f30900cf590 2021-02-10T11:39:20.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.682 7f308f7fe700 1 -- 172.21.2.112:0/2339987287 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f308800b3b0 con 0x7f30900cf590 2021-02-10T11:39:20.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.682 7f308f7fe700 1 -- 172.21.2.112:0/2339987287 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f308800b730 con 0x7f30900cf590 2021-02-10T11:39:20.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.682 7f3096ee7700 1 -- 172.21.2.112:0/2339987287 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f30900cf590 msgr2=0x7f30900cf990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:20.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.682 7f3096ee7700 1 --2- 172.21.2.112:0/2339987287 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f30900cf590 0x7f30900cf990 secure :-1 s=READY pgs=1907 cs=0 l=1 rev1=1 rx=0x7f3088006f00 tx=0x7f3088009f10).stop 2021-02-10T11:39:20.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.682 7f3096ee7700 1 -- 172.21.2.112:0/2339987287 shutdown_connections 2021-02-10T11:39:20.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.682 7f3096ee7700 1 --2- 172.21.2.112:0/2339987287 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f30900cf590 0x7f30900cf990 unknown :-1 s=CLOSED pgs=1907 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:20.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.682 7f3096ee7700 1 --2- 172.21.2.112:0/2339987287 >> v2:172.21.2.120:3300/0 conn(0x7f30900d1c80 0x7f30900cef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:20.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.682 7f3096ee7700 1 -- 172.21.2.112:0/2339987287 >> 172.21.2.112:0/2339987287 conn(0x7f30900ce430 msgr2=0x7f30900d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:20.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.682 7f3096ee7700 1 -- 172.21.2.112:0/2339987287 shutdown_connections 2021-02-10T11:39:20.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.682 7f3096ee7700 1 -- 172.21.2.112:0/2339987287 wait complete. 2021-02-10T11:39:20.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.697 7f3096ee7700 1 Processor -- start 2021-02-10T11:39:20.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.697 7f3096ee7700 1 -- start start 2021-02-10T11:39:20.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.697 7f3096ee7700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f309019e5b0 0x7f30901afc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:20.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.697 7f3096ee7700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f309019e9b0 0x7f30901b0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:20.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.697 7f3096ee7700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f30901b0250 0x7f30901b13e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:20.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.697 7f3096ee7700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3090195bd0 con 0x7f309019e9b0 2021-02-10T11:39:20.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.697 7f3096ee7700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30901b0840 con 0x7f30901b0250 2021-02-10T11:39:20.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.697 7f3096ee7700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f309019eed0 con 0x7f309019e5b0 2021-02-10T11:39:20.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.697 7f308ffff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f309019e9b0 0x7f30901b0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:20.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.697 7f308ffff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f309019e9b0 0x7f30901b0dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:53712/0 (socket says 172.21.2.112:53712) 2021-02-10T11:39:20.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.697 7f308ffff700 1 -- 172.21.2.112:0/794275744 learned_addr learned my addr 172.21.2.112:0/794275744 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:39:20.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.698 7f308ffff700 1 -- 172.21.2.112:0/794275744 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f309019e5b0 msgr2=0x7f30901afc30 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:39:20.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.698 7f308ffff700 1 --2- 172.21.2.112:0/794275744 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f309019e5b0 0x7f30901afc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:20.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.698 7f308ffff700 1 -- 172.21.2.112:0/794275744 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f30901b0250 msgr2=0x7f30901b13e0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:39:20.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.698 7f308ffff700 1 --2- 172.21.2.112:0/794275744 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f30901b0250 0x7f30901b13e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:20.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.698 7f308ffff700 1 -- 172.21.2.112:0/794275744 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f309019f0b0 con 0x7f309019e9b0 2021-02-10T11:39:20.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.698 7f308ffff700 1 --2- 172.21.2.112:0/794275744 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f309019e9b0 0x7f30901b0dc0 secure :-1 s=READY pgs=1908 cs=0 l=1 rev1=1 rx=0x7f308000b5c0 tx=0x7f308000b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:20.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.698 7f308d7fa700 1 -- 172.21.2.112:0/794275744 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f308000bd50 con 0x7f309019e9b0 2021-02-10T11:39:20.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.698 7f308d7fa700 1 -- 172.21.2.112:0/794275744 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3080006030 con 0x7f309019e9b0 2021-02-10T11:39:20.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.698 7f308d7fa700 1 -- 172.21.2.112:0/794275744 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f30800054d0 con 0x7f309019e9b0 2021-02-10T11:39:20.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.698 7f3096ee7700 1 -- 172.21.2.112:0/794275744 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f309019f0b0 con 0x7f309019e9b0 2021-02-10T11:39:20.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.698 7f3096ee7700 1 -- 172.21.2.112:0/794275744 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30901b1cf0 con 0x7f309019e9b0 2021-02-10T11:39:20.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.699 7f308d7fa700 1 -- 172.21.2.112:0/794275744 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3080006210 con 0x7f309019e9b0 2021-02-10T11:39:20.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.699 7f308d7fa700 1 --2- 172.21.2.112:0/794275744 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f30740186c0 0x7f307401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:20.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.699 7f308d7fa700 1 -- 172.21.2.112:0/794275744 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f30800325e0 con 0x7f309019e9b0 2021-02-10T11:39:20.695 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.699 7f3094d03700 1 --2- 172.21.2.112:0/794275744 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f30740186c0 0x7f307401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:20.695 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.700 7f3094d03700 1 --2- 172.21.2.112:0/794275744 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f30740186c0 0x7f307401abc0 crc :-1 s=READY pgs=894 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:20.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.731 7f3096ee7700 1 -- 172.21.2.112:0/794275744 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30901b0840 con 0x7f309019e9b0 2021-02-10T11:39:20.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.733 7f308d7fa700 1 -- 172.21.2.112:0/794275744 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3080000960 con 0x7f309019e9b0 2021-02-10T11:39:20.854 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.861 7f3096ee7700 1 -- 172.21.2.112:0/794275744 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000918084", "target": ["mgr", ""], "group_name": "group_0000000000143913"}) v1 -- 0x7f30901b1cf0 con 0x7f30740186c0 2021-02-10T11:39:20.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.872 7f308d7fa700 1 -- 172.21.2.112:0/794275744 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f3078006db0 con 0x7f30740186c0 2021-02-10T11:39:20.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.878 7f3096ee7700 1 -- 172.21.2.112:0/794275744 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f30740186c0 msgr2=0x7f307401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:20.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.878 7f3096ee7700 1 --2- 172.21.2.112:0/794275744 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f30740186c0 0x7f307401abc0 crc :-1 s=READY pgs=894 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:20.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.878 7f3096ee7700 1 -- 172.21.2.112:0/794275744 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f309019e9b0 msgr2=0x7f30901b0dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:20.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.878 7f3096ee7700 1 --2- 172.21.2.112:0/794275744 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f309019e9b0 0x7f30901b0dc0 secure :-1 s=READY pgs=1908 cs=0 l=1 rev1=1 rx=0x7f308000b5c0 tx=0x7f308000b960).stop 2021-02-10T11:39:20.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.878 7f3096ee7700 1 -- 172.21.2.112:0/794275744 shutdown_connections 2021-02-10T11:39:20.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.878 7f3096ee7700 1 --2- 172.21.2.112:0/794275744 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f30740186c0 0x7f307401abc0 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:20.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.878 7f3096ee7700 1 --2- 172.21.2.112:0/794275744 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f30901b0250 0x7f30901b13e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:20.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.878 7f3096ee7700 1 --2- 172.21.2.112:0/794275744 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f309019e9b0 0x7f30901b0dc0 unknown :-1 s=CLOSED pgs=1908 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:20.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.878 7f3096ee7700 1 --2- 172.21.2.112:0/794275744 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f309019e5b0 0x7f30901afc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:20.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.878 7f3096ee7700 1 -- 172.21.2.112:0/794275744 >> 172.21.2.112:0/794275744 conn(0x7f309019f940 msgr2=0x7f309018c290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:20.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.878 7f3096ee7700 1 -- 172.21.2.112:0/794275744 shutdown_connections 2021-02-10T11:39:20.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:20.878 7f3096ee7700 1 -- 172.21.2.112:0/794275744 wait complete. 2021-02-10T11:39:20.880 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolumegroup rm cephfs group_0000000000143913 2021-02-10T11:39:20.906 DEBUG:teuthology.orchestra.run.gibba012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:39:20.924 DEBUG:teuthology.orchestra.run.gibba020:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:39:21.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:21.304 7f2b85232700 1 Processor -- start 2021-02-10T11:39:21.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:21.304 7f2b85232700 1 -- start start 2021-02-10T11:39:21.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:21.304 7f2b85232700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f2b8010aff0 0x7f2b800cf750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:21.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:21.304 7f2b85232700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b800fe1c0 con 0x7f2b800d1c70 2021-02-10T11:39:21.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:21.304 7f2b85232700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b801b0cb0 con 0x7f2b800d1870 2021-02-10T11:39:21.314 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:21.304 7f2b85232700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b80109a60 con 0x7f2b8010aff0 2021-02-10T11:39:21.314 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:21.304 7f2b7e59c700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f2b800d1c70 0x7f2b800cef60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:53184/0 (socket says 172.21.2.112:53184) 2021-02-10T11:39:21.314 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:21.304 7f2b7e59c700 1 -- 172.21.2.112:0/1081511323 learned_addr learned my addr 172.21.2.112:0/1081511323 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:39:21.314 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:21.304 7f2b7f59e700 1 --2- 172.21.2.112:0/1081511323 >> v2:172.21.2.112:3300/0 conn(0x7f2b8010aff0 0x7f2b800cf750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:21.314 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:21.305 7f2b7dd9b700 1 -- 172.21.2.112:0/1081511323 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2341330610 0 0) 0x7f2b680010b0 con 0x7f2b800d1c70 2021-02-10T11:39:21.314 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:21.305 7f2b7dd9b700 1 -- 172.21.2.112:0/1081511323 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b6c002fe0 con 0x7f2b800d1c70 2021-02-10T11:39:21.314 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:21.305 7f2b7f59e700 1 -- 172.21.2.112:0/1081511323 >> v1:172.21.2.120:6789/0 conn(0x7f2b800d1870 legacy=0x7f2b801b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:21.314 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:21.305 7f2b7f59e700 1 -- 172.21.2.112:0/1081511323 >> v1:172.21.2.112:6789/0 conn(0x7f2b800d1c70 legacy=0x7f2b800cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:21.314 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:21.305 7f2b7f59e700 1 -- 172.21.2.112:0/1081511323 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b8010b3f0 con 0x7f2b8010aff0 2021-02-10T11:39:21.314 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:21.305 7f2b7f59e700 1 --2- 172.21.2.112:0/1081511323 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2b8010aff0 0x7f2b800cf750 secure :-1 s=READY pgs=1910 cs=0 l=1 rev1=1 rx=0x7f2b78006e30 tx=0x7f2b78009e60).ready entity=mon.0 client_cookie=116a750782a8fe2f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:21.314 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:21.305 7f2b7dd9b700 1 -- 172.21.2.112:0/1081511323 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2b7800b300 con 0x7f2b8010aff0 2021-02-10T11:39:21.315 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:21.305 7f2b7dd9b700 1 -- 172.21.2.112:0/1081511323 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b78000d30 con 0x7f2b8010aff0 2021-02-10T11:39:21.315 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:21.305 7f2b7dd9b700 1 -- 172.21.2.112:0/1081511323 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2b7800b830 con 0x7f2b8010aff0 2021-02-10T11:39:21.315 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:21.305 7f2b85232700 1 -- 172.21.2.112:0/1081511323 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2b8010aff0 msgr2=0x7f2b800cf750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:21.315 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:21.305 7f2b85232700 1 --2- 172.21.2.112:0/1081511323 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2b8010aff0 0x7f2b800cf750 secure :-1 s=READY pgs=1910 cs=0 l=1 rev1=1 rx=0x7f2b78006e30 tx=0x7f2b78009e60).stop 2021-02-10T11:39:21.315 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:21.306 7f2b85232700 1 -- 172.21.2.112:0/1081511323 shutdown_connections 2021-02-10T11:39:21.315 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:21.306 7f2b85232700 1 --2- 172.21.2.112:0/1081511323 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2b8010aff0 0x7f2b800cf750 unknown :-1 s=CLOSED pgs=1910 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:21.315 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:21.306 7f2b85232700 1 -- 172.21.2.112:0/1081511323 >> 172.21.2.112:0/1081511323 conn(0x7f2b800ce420 msgr2=0x7f2b800d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:21.315 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:21.306 7f2b85232700 1 -- 172.21.2.112:0/1081511323 shutdown_connections 2021-02-10T11:39:21.315 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:21.306 7f2b85232700 1 -- 172.21.2.112:0/1081511323 wait complete. 2021-02-10T11:39:21.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:21.320 7f2b85232700 1 Processor -- start 2021-02-10T11:39:21.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:21.320 7f2b85232700 1 -- start start 2021-02-10T11:39:21.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:21.321 7f2b85232700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2b800c3820 0x7f2b80084cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:21.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:21.321 7f2b85232700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2b800a0290 0x7f2b800852d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:21.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:21.321 7f2b85232700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2b80086270 0x7f2b80086670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:21.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:21.321 7f2b85232700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b800c0700 con 0x7f2b800a0290 2021-02-10T11:39:21.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:21.321 7f2b85232700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b800858f0 con 0x7f2b80086270 2021-02-10T11:39:21.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:21.321 7f2b85232700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b80085b10 con 0x7f2b800c3820 2021-02-10T11:39:21.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:21.321 7f2b7e59c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2b800a0290 0x7f2b800852d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:21.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:21.321 7f2b7e59c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2b800a0290 0x7f2b800852d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:53728/0 (socket says 172.21.2.112:53728) 2021-02-10T11:39:21.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:21.321 7f2b7e59c700 1 -- 172.21.2.112:0/699969961 learned_addr learned my addr 172.21.2.112:0/699969961 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:39:21.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:21.321 7f2b7e59c700 1 -- 172.21.2.112:0/699969961 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2b800c3820 msgr2=0x7f2b80084cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:21.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:21.321 7f2b7e59c700 1 --2- 172.21.2.112:0/699969961 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2b800c3820 0x7f2b80084cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:21.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:21.321 7f2b7e59c700 1 -- 172.21.2.112:0/699969961 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2b80086270 msgr2=0x7f2b80086670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:21.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:21.321 7f2b7e59c700 1 --2- 172.21.2.112:0/699969961 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2b80086270 0x7f2b80086670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:21.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:21.321 7f2b7e59c700 1 -- 172.21.2.112:0/699969961 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b80086c90 con 0x7f2b800a0290 2021-02-10T11:39:21.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:21.322 7f2b7e59c700 1 --2- 172.21.2.112:0/699969961 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2b800a0290 0x7f2b800852d0 secure :-1 s=READY pgs=1911 cs=0 l=1 rev1=1 rx=0x7f2b68001c60 tx=0x7f2b6800a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:21.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:21.322 7f2b777fe700 1 -- 172.21.2.112:0/699969961 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2b6800acc0 con 0x7f2b800a0290 2021-02-10T11:39:21.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:21.322 7f2b777fe700 1 -- 172.21.2.112:0/699969961 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b68004350 con 0x7f2b800a0290 2021-02-10T11:39:21.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:21.322 7f2b777fe700 1 -- 172.21.2.112:0/699969961 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2b68004830 con 0x7f2b800a0290 2021-02-10T11:39:21.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:21.322 7f2b85232700 1 -- 172.21.2.112:0/699969961 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b80086c90 con 0x7f2b800a0290 2021-02-10T11:39:21.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:21.322 7f2b85232700 1 -- 172.21.2.112:0/699969961 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b801a2f00 con 0x7f2b800a0290 2021-02-10T11:39:21.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:21.323 7f2b777fe700 1 -- 172.21.2.112:0/699969961 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2b68004530 con 0x7f2b800a0290 2021-02-10T11:39:21.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:21.323 7f2b777fe700 1 --2- 172.21.2.112:0/699969961 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2b640186c0 0x7f2b6401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:21.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:21.323 7f2b777fe700 1 -- 172.21.2.112:0/699969961 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f2b68032560 con 0x7f2b800a0290 2021-02-10T11:39:21.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:21.323 7f2b7ed9d700 1 --2- 172.21.2.112:0/699969961 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2b640186c0 0x7f2b6401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:21.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:21.324 7f2b7ed9d700 1 --2- 172.21.2.112:0/699969961 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2b640186c0 0x7f2b6401abc0 crc :-1 s=READY pgs=895 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:21.347 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:21.354 7f2b85232700 1 -- 172.21.2.112:0/699969961 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b80082a70 con 0x7f2b800a0290 2021-02-10T11:39:21.350 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:21.357 7f2b777fe700 1 -- 172.21.2.112:0/699969961 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2b68000960 con 0x7f2b800a0290 2021-02-10T11:39:21.481 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:21.488 7f2b85232700 1 -- 172.21.2.112:0/699969961 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolumegroup rm", "vol_name": "cephfs", "target": ["mgr", ""], "group_name": "group_0000000000143913"}) v1 -- 0x7f2b801a2f00 con 0x7f2b640186c0 2021-02-10T11:39:21.484 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:21.492 7f2b777fe700 1 -- 172.21.2.112:0/699969961 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f2b70006a10 con 0x7f2b640186c0 2021-02-10T11:39:21.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:21.497 7f2b85232700 1 -- 172.21.2.112:0/699969961 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2b640186c0 msgr2=0x7f2b6401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:21.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:21.497 7f2b85232700 1 --2- 172.21.2.112:0/699969961 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2b640186c0 0x7f2b6401abc0 crc :-1 s=READY pgs=895 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:21.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:21.497 7f2b85232700 1 -- 172.21.2.112:0/699969961 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2b800a0290 msgr2=0x7f2b800852d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:21.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:21.497 7f2b85232700 1 --2- 172.21.2.112:0/699969961 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2b800a0290 0x7f2b800852d0 secure :-1 s=READY pgs=1911 cs=0 l=1 rev1=1 rx=0x7f2b68001c60 tx=0x7f2b6800a8d0).stop 2021-02-10T11:39:21.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:21.497 7f2b85232700 1 -- 172.21.2.112:0/699969961 shutdown_connections 2021-02-10T11:39:21.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:21.497 7f2b85232700 1 --2- 172.21.2.112:0/699969961 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2b640186c0 0x7f2b6401abc0 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:21.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:21.497 7f2b85232700 1 --2- 172.21.2.112:0/699969961 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2b80086270 0x7f2b80086670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:21.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:21.497 7f2b85232700 1 --2- 172.21.2.112:0/699969961 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2b800a0290 0x7f2b800852d0 unknown :-1 s=CLOSED pgs=1911 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:21.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:21.497 7f2b85232700 1 --2- 172.21.2.112:0/699969961 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2b800c3820 0x7f2b80084cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:21.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:21.497 7f2b85232700 1 -- 172.21.2.112:0/699969961 >> 172.21.2.112:0/699969961 conn(0x7f2b800872c0 msgr2=0x7f2b80095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:21.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:21.497 7f2b85232700 1 -- 172.21.2.112:0/699969961 shutdown_connections 2021-02-10T11:39:21.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:21.497 7f2b85232700 1 -- 172.21.2.112:0/699969961 wait complete. 2021-02-10T11:39:21.498 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:39:21.499 DEBUG:teuthology.orchestra.run.gibba012:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:39:21.843 DEBUG:teuthology.orchestra.run.gibba020:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:39:22.187 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:39:22.187 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:39:22.187 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:39:22.187 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:39:22.522 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:39:22.522 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:39:22.522 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba012.front.sepia.ceph.com... 2021-02-10T11:39:22.522 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:39:22.523 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:39:22.875 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba012.stderr:ceph-fuse[149823]: fuse finished with error 0 and tester_r 0 2021-02-10T11:39:22.876 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:39:22.877 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:39:22.877 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:39:23.210 INFO:teuthology.orchestra.run.gibba012.stdout:ext2/ext3 2021-02-10T11:39:23.210 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:39:23.211 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:39:23.211 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:39:23.546 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:39:23.546 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:39:23.546 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:39:23.880 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:39:23.881 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:39:23.881 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba012.front.sepia.ceph.com... 2021-02-10T11:39:23.881 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:39:23.881 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:39:24.234 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba012.stderr:ceph-fuse[150463]: fuse finished with error 0 and tester_r 0 2021-02-10T11:39:24.236 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:39:24.236 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:39:24.236 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:39:24.569 INFO:teuthology.orchestra.run.gibba012.stdout:ext2/ext3 2021-02-10T11:39:24.569 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:39:24.570 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:39:24.570 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:39:24.905 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:39:24.906 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:39:24.906 DEBUG:teuthology.orchestra.run.gibba020:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:39:25.240 INFO:teuthology.orchestra.run.gibba020.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:39:25.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:39:25.240 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:39:25.241 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:39:25.241 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:39:25.574 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:39:25.574 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:39:25.574 DEBUG:teuthology.orchestra.run.gibba020:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:39:25.909 INFO:teuthology.orchestra.run.gibba020.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:39:25.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:39:25.909 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:39:25.910 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:39:25.910 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:39:26.243 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config rm mon mon_allow_pool_delete 2021-02-10T11:39:26.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:26.665 7f3eb3dd7700 1 Processor -- start 2021-02-10T11:39:26.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:26.665 7f3eb3dd7700 1 -- start start 2021-02-10T11:39:26.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:26.665 7f3eb3dd7700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3eac0d1870 0x7f3eac1b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:26.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:26.665 7f3eb3dd7700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3eac0f0b10 con 0x7f3eac0f0650 2021-02-10T11:39:26.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:26.665 7f3eb3dd7700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3eac0d0310 con 0x7f3eac1b0a10 2021-02-10T11:39:26.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:26.665 7f3eb3dd7700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3eac0e5ed0 con 0x7f3eac0d1870 2021-02-10T11:39:26.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:26.666 7f3eb23f4700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f3eac0f0650 0x7f3eac0cfb20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:53196/0 (socket says 172.21.2.112:53196) 2021-02-10T11:39:26.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:26.666 7f3eb23f4700 1 -- 172.21.2.112:0/2780930525 learned_addr learned my addr 172.21.2.112:0/2780930525 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:39:26.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:26.666 7f3eb0bf1700 1 -- 172.21.2.112:0/2780930525 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2695000462 0 0) 0x7f3ea4001500 con 0x7f3eac0f0650 2021-02-10T11:39:26.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:26.666 7f3eb0bf1700 1 -- 172.21.2.112:0/2780930525 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e98002fe0 con 0x7f3eac0f0650 2021-02-10T11:39:26.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:26.666 7f3eb1bf3700 1 --2- 172.21.2.112:0/2780930525 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3eac0d1870 0x7f3eac1b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:26.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:26.666 7f3eb0bf1700 1 -- 172.21.2.112:0/2780930525 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3276762951 0 0) 0x7f3ea4001460 con 0x7f3eac0f0650 2021-02-10T11:39:26.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:26.666 7f3eb0bf1700 1 -- 172.21.2.112:0/2780930525 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3eac0d1870 msgr2=0x7f3eac1b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:26.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:26.666 7f3eb0bf1700 1 --2- 172.21.2.112:0/2780930525 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3eac0d1870 0x7f3eac1b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:26.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:26.666 7f3eb0bf1700 1 -- 172.21.2.112:0/2780930525 >> v1:172.21.2.120:6789/0 conn(0x7f3eac1b0a10 legacy=0x7f3eac0cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:26.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:26.666 7f3eb0bf1700 1 -- 172.21.2.112:0/2780930525 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3eac0d0530 con 0x7f3eac0f0650 2021-02-10T11:39:26.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:26.666 7f3eb0bf1700 1 -- 172.21.2.112:0/2780930525 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f3ea4001980 con 0x7f3eac0f0650 2021-02-10T11:39:26.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:26.666 7f3eb0bf1700 1 -- 172.21.2.112:0/2780930525 >> v1:172.21.2.112:6789/0 conn(0x7f3eac0f0650 legacy=0x7f3eac0cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:26.678 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:26.666 7f3eb0bf1700 1 --2- 172.21.2.112:0/2780930525 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3e98003b10 0x7f3e98005f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:26.678 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:26.666 7f3eb0bf1700 1 --2- 172.21.2.112:0/2780930525 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3e98006820 0x7f3e98008d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:26.678 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:26.666 7f3eb0bf1700 1 --2- 172.21.2.112:0/2780930525 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3e98009620 0x7f3e9800bb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:26.678 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:26.666 7f3eb0bf1700 1 -- 172.21.2.112:0/2780930525 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e9800c360 con 0x7f3e98006820 2021-02-10T11:39:26.678 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:26.666 7f3eb0bf1700 1 -- 172.21.2.112:0/2780930525 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e9800c580 con 0x7f3e98003b10 2021-02-10T11:39:26.678 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:26.666 7f3eb0bf1700 1 -- 172.21.2.112:0/2780930525 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e9800c7a0 con 0x7f3e98009620 2021-02-10T11:39:26.678 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:26.666 7f3eb1bf3700 1 --2- 172.21.2.112:0/2780930525 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3eac0d1870 0x7f3eac1b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:39:26.678 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:26.667 7f3eb13f2700 1 --2- 172.21.2.112:0/2780930525 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3e98006820 0x7f3e98008d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:26.678 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:26.667 7f3eb23f4700 1 --2- 172.21.2.112:0/2780930525 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3e98009620 0x7f3e9800bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:26.678 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:26.667 7f3eb1bf3700 1 --2- 172.21.2.112:0/2780930525 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3e98003b10 0x7f3e98005f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:26.678 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:26.667 7f3eb13f2700 1 -- 172.21.2.112:0/2780930525 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3e98009620 msgr2=0x7f3e9800bb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:26.678 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:26.667 7f3eb13f2700 1 --2- 172.21.2.112:0/2780930525 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3e98009620 0x7f3e9800bb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:26.678 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:26.667 7f3eb13f2700 1 -- 172.21.2.112:0/2780930525 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3e98003b10 msgr2=0x7f3e98005f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:26.679 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:26.667 7f3eb13f2700 1 --2- 172.21.2.112:0/2780930525 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3e98003b10 0x7f3e98005f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:26.679 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:26.667 7f3eb13f2700 1 -- 172.21.2.112:0/2780930525 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e9800ca40 con 0x7f3e98006820 2021-02-10T11:39:26.679 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:26.667 7f3eb13f2700 1 --2- 172.21.2.112:0/2780930525 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3e98006820 0x7f3e98008d20 secure :-1 s=READY pgs=1913 cs=0 l=1 rev1=1 rx=0x7f3ea804aad0 tx=0x7f3ea804da50).ready entity=mon.0 client_cookie=b25cec0c9ba96542 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:26.679 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:26.667 7f3eb0bf1700 1 -- 172.21.2.112:0/2780930525 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3ea804b280 con 0x7f3e98006820 2021-02-10T11:39:26.679 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:26.667 7f3eb0bf1700 1 -- 172.21.2.112:0/2780930525 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ea804b580 con 0x7f3e98006820 2021-02-10T11:39:26.679 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:26.667 7f3eb0bf1700 1 -- 172.21.2.112:0/2780930525 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3ea804baa0 con 0x7f3e98006820 2021-02-10T11:39:26.679 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:26.667 7f3eb3dd7700 1 -- 172.21.2.112:0/2780930525 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3e98006820 msgr2=0x7f3e98008d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:26.679 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:26.667 7f3eb3dd7700 1 --2- 172.21.2.112:0/2780930525 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3e98006820 0x7f3e98008d20 secure :-1 s=READY pgs=1913 cs=0 l=1 rev1=1 rx=0x7f3ea804aad0 tx=0x7f3ea804da50).stop 2021-02-10T11:39:26.679 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:26.667 7f3eb23f4700 1 -- 172.21.2.112:0/2780930525 reap_dead start 2021-02-10T11:39:26.679 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:26.668 7f3eb3dd7700 1 -- 172.21.2.112:0/2780930525 shutdown_connections 2021-02-10T11:39:26.679 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:26.668 7f3eb3dd7700 1 -- 172.21.2.112:0/2780930525 >> 172.21.2.112:0/2780930525 conn(0x7f3eac0ce450 msgr2=0x7f3eac0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:26.679 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:26.668 7f3eb3dd7700 1 -- 172.21.2.112:0/2780930525 shutdown_connections 2021-02-10T11:39:26.679 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:26.668 7f3eb3dd7700 1 -- 172.21.2.112:0/2780930525 wait complete. 2021-02-10T11:39:26.680 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:26.683 7f3eb3dd7700 1 Processor -- start 2021-02-10T11:39:26.680 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:26.683 7f3eb3dd7700 1 -- start start 2021-02-10T11:39:26.681 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:26.683 7f3eb3dd7700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3eac1a0590 0x7f3eac0c1400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:26.681 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:26.683 7f3eb3dd7700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3eac0c21b0 0x7f3eac0c25b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:26.681 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:26.683 7f3eb3dd7700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3eac0c2bd0 0x7f3eac0c2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:26.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:26.683 7f3eb3dd7700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3eac09c5b0 con 0x7f3eac0c21b0 2021-02-10T11:39:26.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:26.683 7f3eb3dd7700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3eac0c1c30 con 0x7f3eac0c2bd0 2021-02-10T11:39:26.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:26.683 7f3eb3dd7700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3eac08fff0 con 0x7f3eac1a0590 2021-02-10T11:39:26.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:26.684 7f3eb13f2700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3eac0c21b0 0x7f3eac0c25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:26.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:26.684 7f3eb13f2700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3eac0c21b0 0x7f3eac0c25b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:53748/0 (socket says 172.21.2.112:53748) 2021-02-10T11:39:26.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:26.684 7f3eb13f2700 1 -- 172.21.2.112:0/118950521 learned_addr learned my addr 172.21.2.112:0/118950521 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:39:26.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:26.684 7f3eb1bf3700 1 --2- 172.21.2.112:0/118950521 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3eac1a0590 0x7f3eac0c1400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:26.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:26.684 7f3eb23f4700 1 --2- 172.21.2.112:0/118950521 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3eac0c2bd0 0x7f3eac0c2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:26.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:26.684 7f3eb13f2700 1 -- 172.21.2.112:0/118950521 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3eac1a0590 msgr2=0x7f3eac0c1400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:26.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:26.684 7f3eb13f2700 1 --2- 172.21.2.112:0/118950521 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3eac1a0590 0x7f3eac0c1400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:26.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:26.684 7f3eb13f2700 1 -- 172.21.2.112:0/118950521 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3eac0c2bd0 msgr2=0x7f3eac0c2fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:26.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:26.684 7f3eb13f2700 1 --2- 172.21.2.112:0/118950521 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3eac0c2bd0 0x7f3eac0c2fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:26.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:26.684 7f3eb13f2700 1 -- 172.21.2.112:0/118950521 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3eac0904b0 con 0x7f3eac0c21b0 2021-02-10T11:39:26.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:26.684 7f3eb1bf3700 1 --2- 172.21.2.112:0/118950521 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3eac1a0590 0x7f3eac0c1400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:39:26.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:26.684 7f3eb23f4700 1 --2- 172.21.2.112:0/118950521 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3eac0c2bd0 0x7f3eac0c2fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:39:26.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:26.684 7f3eb13f2700 1 --2- 172.21.2.112:0/118950521 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3eac0c21b0 0x7f3eac0c25b0 secure :-1 s=READY pgs=1914 cs=0 l=1 rev1=1 rx=0x7f3ea804f3f0 tx=0x7f3ea804b660).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:26.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:26.684 7f3e9e7fc700 1 -- 172.21.2.112:0/118950521 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3ea804d330 con 0x7f3eac0c21b0 2021-02-10T11:39:26.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:26.684 7f3e9e7fc700 1 -- 172.21.2.112:0/118950521 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ea804a5b0 con 0x7f3eac0c21b0 2021-02-10T11:39:26.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:26.684 7f3eb3dd7700 1 -- 172.21.2.112:0/118950521 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3eac0904b0 con 0x7f3eac0c21b0 2021-02-10T11:39:26.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:26.684 7f3e9e7fc700 1 -- 172.21.2.112:0/118950521 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3ea804d7a0 con 0x7f3eac0c21b0 2021-02-10T11:39:26.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:26.684 7f3eb3dd7700 1 -- 172.21.2.112:0/118950521 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3eac0904b0 con 0x7f3eac0c21b0 2021-02-10T11:39:26.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:26.685 7f3e9e7fc700 1 -- 172.21.2.112:0/118950521 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3ea804d4f0 con 0x7f3eac0c21b0 2021-02-10T11:39:26.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:26.685 7f3e9e7fc700 1 --2- 172.21.2.112:0/118950521 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3e940186d0 0x7f3e9401abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:26.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:26.685 7f3e9e7fc700 1 -- 172.21.2.112:0/118950521 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f3ea80767b0 con 0x7f3eac0c21b0 2021-02-10T11:39:26.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:26.685 7f3eb1bf3700 1 --2- 172.21.2.112:0/118950521 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3e940186d0 0x7f3e9401abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:26.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:26.686 7f3eb1bf3700 1 --2- 172.21.2.112:0/118950521 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3e940186d0 0x7f3e9401abd0 crc :-1 s=READY pgs=896 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:26.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:26.715 7f3eb3dd7700 1 -- 172.21.2.112:0/118950521 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3eac0ebcb0 con 0x7f3eac0c21b0 2021-02-10T11:39:26.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:26.718 7f3e9e7fc700 1 -- 172.21.2.112:0/118950521 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3ea804d2e0 con 0x7f3eac0c21b0 2021-02-10T11:39:26.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:26.869 7f3eb3dd7700 1 -- 172.21.2.112:0/118950521 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f3eac0f0b10 con 0x7f3eac0c21b0 2021-02-10T11:39:26.867 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:26.874 7f3e9e7fc700 1 -- 172.21.2.112:0/118950521 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v25) v1 ==== 104+0+0 (secure 0 0 0) 0x7f3ea804d4f0 con 0x7f3eac0c21b0 2021-02-10T11:39:26.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:26.878 7f3eb3dd7700 1 -- 172.21.2.112:0/118950521 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3e940186d0 msgr2=0x7f3e9401abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:26.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:26.878 7f3eb3dd7700 1 --2- 172.21.2.112:0/118950521 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3e940186d0 0x7f3e9401abd0 crc :-1 s=READY pgs=896 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:26.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:26.878 7f3eb3dd7700 1 -- 172.21.2.112:0/118950521 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3eac0c21b0 msgr2=0x7f3eac0c25b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:26.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:26.878 7f3eb3dd7700 1 --2- 172.21.2.112:0/118950521 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3eac0c21b0 0x7f3eac0c25b0 secure :-1 s=READY pgs=1914 cs=0 l=1 rev1=1 rx=0x7f3ea804f3f0 tx=0x7f3ea804b660).stop 2021-02-10T11:39:26.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:26.878 7f3eb3dd7700 1 -- 172.21.2.112:0/118950521 shutdown_connections 2021-02-10T11:39:26.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:26.878 7f3eb3dd7700 1 --2- 172.21.2.112:0/118950521 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3e940186d0 0x7f3e9401abd0 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:26.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:26.878 7f3eb3dd7700 1 --2- 172.21.2.112:0/118950521 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3eac0c2bd0 0x7f3eac0c2fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:26.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:26.878 7f3eb3dd7700 1 --2- 172.21.2.112:0/118950521 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3eac0c21b0 0x7f3eac0c25b0 unknown :-1 s=CLOSED pgs=1914 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:26.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:26.878 7f3eb3dd7700 1 --2- 172.21.2.112:0/118950521 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3eac1a0590 0x7f3eac0c1400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:26.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:26.878 7f3eb3dd7700 1 -- 172.21.2.112:0/118950521 >> 172.21.2.112:0/118950521 conn(0x7f3eac0c3e90 msgr2=0x7f3eac094790 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:26.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:26.878 7f3eb3dd7700 1 -- 172.21.2.112:0/118950521 shutdown_connections 2021-02-10T11:39:26.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:26.879 7f3eb3dd7700 1 -- 172.21.2.112:0/118950521 wait complete. 2021-02-10T11:39:26.880 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestVolumes.test_deauthorize_auth_id_after_out_of_band_update' 2021-02-10T11:39:27.307 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:27.298 7fce4d2fb700 1 Processor -- start 2021-02-10T11:39:27.307 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:27.298 7fce4d2fb700 1 -- start start 2021-02-10T11:39:27.307 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:27.298 7fce4d2fb700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fce480d1870 0x7fce481b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:27.308 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:27.298 7fce4d2fb700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fce480f0650 0x7fce480cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:27.308 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:27.298 7fce4d2fb700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce480f0b10 con 0x7fce481b0a10 2021-02-10T11:39:27.308 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:27.298 7fce4d2fb700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce480e5ed0 con 0x7fce480d1870 2021-02-10T11:39:27.308 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:27.298 7fce4d2fb700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce480d1c70 con 0x7fce480f0650 2021-02-10T11:39:27.308 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:27.298 7fce4707d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fce480d1870 0x7fce481b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:27.308 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:27.298 7fce4687c700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fce481b0a10 0x7fce480cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:53216/0 (socket says 172.21.2.112:53216) 2021-02-10T11:39:27.308 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:27.298 7fce4687c700 1 -- 172.21.2.112:0/2292857686 learned_addr learned my addr 172.21.2.112:0/2292857686 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:39:27.308 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:27.299 7fce4787e700 1 --2- 172.21.2.112:0/2292857686 >> v2:172.21.2.120:3300/0 conn(0x7fce480f0650 0x7fce480cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:27.308 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:27.299 7fce4707d700 1 -- 172.21.2.112:0/2292857686 >> v2:172.21.2.120:3300/0 conn(0x7fce480f0650 msgr2=0x7fce480cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:27.308 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:27.299 7fce4707d700 1 --2- 172.21.2.112:0/2292857686 >> v2:172.21.2.120:3300/0 conn(0x7fce480f0650 0x7fce480cfb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:27.308 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:27.299 7fce4707d700 1 -- 172.21.2.112:0/2292857686 >> v1:172.21.2.112:6789/0 conn(0x7fce481b0a10 legacy=0x7fce480cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:27.308 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:27.299 7fce4707d700 1 -- 172.21.2.112:0/2292857686 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce480d1e90 con 0x7fce480d1870 2021-02-10T11:39:27.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:27.299 7fce4707d700 1 --2- 172.21.2.112:0/2292857686 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fce480d1870 0x7fce481b03f0 secure :-1 s=READY pgs=1916 cs=0 l=1 rev1=1 rx=0x7fce30006650 tx=0x7fce30009a80).ready entity=mon.0 client_cookie=453db42997089e1e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:27.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:27.299 7fce4607b700 1 -- 172.21.2.112:0/2292857686 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fce3000c0c0 con 0x7fce480d1870 2021-02-10T11:39:27.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:27.299 7fce4607b700 1 -- 172.21.2.112:0/2292857686 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce3000c340 con 0x7fce480d1870 2021-02-10T11:39:27.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:27.299 7fce4607b700 1 -- 172.21.2.112:0/2292857686 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fce3000c860 con 0x7fce480d1870 2021-02-10T11:39:27.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:27.299 7fce4d2fb700 1 -- 172.21.2.112:0/2292857686 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fce480d1870 msgr2=0x7fce481b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:27.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:27.299 7fce4d2fb700 1 --2- 172.21.2.112:0/2292857686 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fce480d1870 0x7fce481b03f0 secure :-1 s=READY pgs=1916 cs=0 l=1 rev1=1 rx=0x7fce30006650 tx=0x7fce30009a80).stop 2021-02-10T11:39:27.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:27.300 7fce4d2fb700 1 -- 172.21.2.112:0/2292857686 shutdown_connections 2021-02-10T11:39:27.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:27.300 7fce4d2fb700 1 --2- 172.21.2.112:0/2292857686 >> v2:172.21.2.120:3300/0 conn(0x7fce480f0650 0x7fce480cfb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:27.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:27.300 7fce4d2fb700 1 --2- 172.21.2.112:0/2292857686 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fce480d1870 0x7fce481b03f0 unknown :-1 s=CLOSED pgs=1916 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:27.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:27.300 7fce4d2fb700 1 -- 172.21.2.112:0/2292857686 >> 172.21.2.112:0/2292857686 conn(0x7fce480ce450 msgr2=0x7fce480d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:27.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:27.300 7fce4d2fb700 1 -- 172.21.2.112:0/2292857686 shutdown_connections 2021-02-10T11:39:27.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:27.300 7fce4d2fb700 1 -- 172.21.2.112:0/2292857686 wait complete. 2021-02-10T11:39:27.311 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:27.314 7fce4d2fb700 1 Processor -- start 2021-02-10T11:39:27.311 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:27.315 7fce4d2fb700 1 -- start start 2021-02-10T11:39:27.311 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:27.315 7fce4d2fb700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fce480f0280 0x7fce480a2310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:27.311 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:27.315 7fce4d2fb700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fce480f0680 0x7fce480a2930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:27.311 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:27.315 7fce4d2fb700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fce480ec180 0x7fce480ec580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:27.311 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:27.315 7fce4d2fb700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce480efd00 con 0x7fce480ec180 2021-02-10T11:39:27.311 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:27.315 7fce4d2fb700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce480ecba0 con 0x7fce480f0680 2021-02-10T11:39:27.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:27.315 7fce4d2fb700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce480ecd80 con 0x7fce480f0280 2021-02-10T11:39:27.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:27.315 7fce4787e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fce480ec180 0x7fce480ec580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:27.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:27.315 7fce4787e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fce480ec180 0x7fce480ec580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:53758/0 (socket says 172.21.2.112:53758) 2021-02-10T11:39:27.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:27.315 7fce4787e700 1 -- 172.21.2.112:0/2145151094 learned_addr learned my addr 172.21.2.112:0/2145151094 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:39:27.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:27.315 7fce4787e700 1 -- 172.21.2.112:0/2145151094 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fce480f0280 msgr2=0x7fce480a2310 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:39:27.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:27.315 7fce4787e700 1 --2- 172.21.2.112:0/2145151094 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fce480f0280 0x7fce480a2310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:27.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:27.315 7fce4787e700 1 -- 172.21.2.112:0/2145151094 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fce480f0680 msgr2=0x7fce480a2930 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:39:27.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:27.315 7fce4787e700 1 --2- 172.21.2.112:0/2145151094 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fce480f0680 0x7fce480a2930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:27.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:27.315 7fce4787e700 1 -- 172.21.2.112:0/2145151094 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce480ecfa0 con 0x7fce480ec180 2021-02-10T11:39:27.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:27.315 7fce4787e700 1 --2- 172.21.2.112:0/2145151094 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fce480ec180 0x7fce480ec580 secure :-1 s=READY pgs=1917 cs=0 l=1 rev1=1 rx=0x7fce400099a0 tx=0x7fce4000bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:27.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:27.316 7fce3ffff700 1 -- 172.21.2.112:0/2145151094 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fce400066f0 con 0x7fce480ec180 2021-02-10T11:39:27.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:27.316 7fce3ffff700 1 -- 172.21.2.112:0/2145151094 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce40003030 con 0x7fce480ec180 2021-02-10T11:39:27.314 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:27.316 7fce4d2fb700 1 -- 172.21.2.112:0/2145151094 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce480ecfa0 con 0x7fce480ec180 2021-02-10T11:39:27.314 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:27.316 7fce4d2fb700 1 -- 172.21.2.112:0/2145151094 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce480ed530 con 0x7fce480ec180 2021-02-10T11:39:27.314 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:27.316 7fce3ffff700 1 -- 172.21.2.112:0/2145151094 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fce400013e0 con 0x7fce480ec180 2021-02-10T11:39:27.314 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:27.316 7fce3ffff700 1 -- 172.21.2.112:0/2145151094 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fce400011f0 con 0x7fce480ec180 2021-02-10T11:39:27.314 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:27.316 7fce3ffff700 1 --2- 172.21.2.112:0/2145151094 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fce2c0186c0 0x7fce2c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:27.314 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:27.316 7fce3ffff700 1 -- 172.21.2.112:0/2145151094 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fce400328f0 con 0x7fce480ec180 2021-02-10T11:39:27.314 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:27.317 7fce4707d700 1 --2- 172.21.2.112:0/2145151094 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fce2c0186c0 0x7fce2c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:27.314 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:27.317 7fce4707d700 1 --2- 172.21.2.112:0/2145151094 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fce2c0186c0 0x7fce2c01abc0 crc :-1 s=READY pgs=897 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:27.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:27.348 7fce4d2fb700 1 -- 172.21.2.112:0/2145151094 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce480ebc60 con 0x7fce480ec180 2021-02-10T11:39:27.344 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:27.351 7fce3ffff700 1 -- 172.21.2.112:0/2145151094 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fce40000f90 con 0x7fce480ec180 2021-02-10T11:39:27.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:27.482 7fce4d2fb700 1 -- 172.21.2.112:0/2145151094 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_deauthorize_auth_id_after_out_of_band_update"]} v 0) v1 -- 0x7fce480f0b10 con 0x7fce480ec180 2021-02-10T11:39:28.164 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:28.170 7fce3ffff700 1 -- 172.21.2.112:0/2145151094 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_deauthorize_auth_id_after_out_of_band_update"]}]=0 v504) v1 ==== 166+0+0 (secure 0 0 0) 0x7fce4001f850 con 0x7fce480ec180 2021-02-10T11:39:28.196 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:28.201 7fce4d2fb700 1 -- 172.21.2.112:0/2145151094 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fce2c0186c0 msgr2=0x7fce2c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:28.196 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:28.201 7fce4d2fb700 1 --2- 172.21.2.112:0/2145151094 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fce2c0186c0 0x7fce2c01abc0 crc :-1 s=READY pgs=897 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:28.196 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:28.201 7fce4d2fb700 1 -- 172.21.2.112:0/2145151094 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fce480ec180 msgr2=0x7fce480ec580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:28.196 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:28.201 7fce4d2fb700 1 --2- 172.21.2.112:0/2145151094 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fce480ec180 0x7fce480ec580 secure :-1 s=READY pgs=1917 cs=0 l=1 rev1=1 rx=0x7fce400099a0 tx=0x7fce4000bcf0).stop 2021-02-10T11:39:28.197 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:28.202 7fce4d2fb700 1 -- 172.21.2.112:0/2145151094 shutdown_connections 2021-02-10T11:39:28.197 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:28.202 7fce4d2fb700 1 --2- 172.21.2.112:0/2145151094 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fce2c0186c0 0x7fce2c01abc0 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:28.197 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:28.202 7fce4d2fb700 1 --2- 172.21.2.112:0/2145151094 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fce480f0680 0x7fce480a2930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:28.197 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:28.202 7fce4d2fb700 1 --2- 172.21.2.112:0/2145151094 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fce480ec180 0x7fce480ec580 unknown :-1 s=CLOSED pgs=1917 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:28.197 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:28.202 7fce4d2fb700 1 --2- 172.21.2.112:0/2145151094 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fce480f0280 0x7fce480a2310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:28.197 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:28.202 7fce4d2fb700 1 -- 172.21.2.112:0/2145151094 >> 172.21.2.112:0/2145151094 conn(0x7fce480c2130 msgr2=0x7fce4809e3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:28.197 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:28.202 7fce4d2fb700 1 -- 172.21.2.112:0/2145151094 shutdown_connections 2021-02-10T11:39:28.197 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:28.202 7fce4d2fb700 1 -- 172.21.2.112:0/2145151094 wait complete. 2021-02-10T11:39:28.204 INFO:tasks.cephfs_test_runner:test_deauthorize_auth_id_after_out_of_band_update (tasks.cephfs.test_volumes.TestVolumes) ... ok 2021-02-10T11:39:28.204 INFO:tasks.cephfs_test_runner:Starting test: test_default_uid_gid_subvolume (tasks.cephfs.test_volumes.TestVolumes) 2021-02-10T11:39:28.205 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestVolumes.test_default_uid_gid_subvolume' 2021-02-10T11:39:28.633 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:28.621 7fdf5cf24700 1 Processor -- start 2021-02-10T11:39:28.633 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:28.621 7fdf5cf24700 1 -- start start 2021-02-10T11:39:28.633 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:28.621 7fdf5cf24700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fdf580d1870 0x7fdf581b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:28.633 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:28.621 7fdf5cf24700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdf581b0a10 0x7fdf580cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:28.634 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:28.622 7fdf5cf24700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf580f0b10 con 0x7fdf580d1c70 2021-02-10T11:39:28.634 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:28.622 7fdf5cf24700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf580e5ed0 con 0x7fdf580d1870 2021-02-10T11:39:28.634 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:28.622 7fdf5cf24700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf580f0650 con 0x7fdf581b0a10 2021-02-10T11:39:28.634 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:28.623 7fdf57314700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7fdf580d1c70 0x7fdf580cf950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:46790/0 (socket says 172.21.2.112:46790) 2021-02-10T11:39:28.634 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:28.623 7fdf56312700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdf581b0a10 0x7fdf580cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:28.634 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:28.623 7fdf57314700 1 -- 172.21.2.112:0/1834103038 learned_addr learned my addr 172.21.2.112:0/1834103038 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:39:28.634 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:28.623 7fdf56b13700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fdf580d1870 0x7fdf581b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:28.634 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:28.623 7fdf56b13700 1 -- 172.21.2.112:0/1834103038 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdf581b0a10 msgr2=0x7fdf580cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:28.634 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:28.623 7fdf56b13700 1 --2- 172.21.2.112:0/1834103038 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdf581b0a10 0x7fdf580cf330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:28.634 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:28.623 7fdf56b13700 1 -- 172.21.2.112:0/1834103038 >> v1:172.21.2.120:6789/0 conn(0x7fdf580d1c70 legacy=0x7fdf580cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:28.634 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:28.623 7fdf56b13700 1 -- 172.21.2.112:0/1834103038 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf580f0870 con 0x7fdf580d1870 2021-02-10T11:39:28.634 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:28.623 7fdf56b13700 1 --2- 172.21.2.112:0/1834103038 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fdf580d1870 0x7fdf581b03f0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 rx=0x7fdf48006650 tx=0x7fdf48009a80).ready entity=mon.1 client_cookie=1a896c85fe424d9b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:28.635 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:28.625 7fdf55b11700 1 -- 172.21.2.112:0/1834103038 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdf4800c0c0 con 0x7fdf580d1870 2021-02-10T11:39:28.635 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:28.625 7fdf55b11700 1 -- 172.21.2.112:0/1834103038 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf4800c3c0 con 0x7fdf580d1870 2021-02-10T11:39:28.635 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:28.625 7fdf55b11700 1 -- 172.21.2.112:0/1834103038 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdf4800c820 con 0x7fdf580d1870 2021-02-10T11:39:28.635 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:28.625 7fdf5cf24700 1 -- 172.21.2.112:0/1834103038 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fdf580d1870 msgr2=0x7fdf581b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:28.635 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:28.625 7fdf5cf24700 1 --2- 172.21.2.112:0/1834103038 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fdf580d1870 0x7fdf581b03f0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 rx=0x7fdf48006650 tx=0x7fdf48009a80).stop 2021-02-10T11:39:28.635 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:28.625 7fdf5cf24700 1 -- 172.21.2.112:0/1834103038 shutdown_connections 2021-02-10T11:39:28.635 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:28.625 7fdf5cf24700 1 --2- 172.21.2.112:0/1834103038 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdf581b0a10 0x7fdf580cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:28.635 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:28.625 7fdf5cf24700 1 --2- 172.21.2.112:0/1834103038 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fdf580d1870 0x7fdf581b03f0 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:28.635 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:28.625 7fdf5cf24700 1 -- 172.21.2.112:0/1834103038 >> 172.21.2.112:0/1834103038 conn(0x7fdf580ce450 msgr2=0x7fdf580d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:28.635 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:28.625 7fdf5cf24700 1 -- 172.21.2.112:0/1834103038 shutdown_connections 2021-02-10T11:39:28.635 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:28.625 7fdf5cf24700 1 -- 172.21.2.112:0/1834103038 wait complete. 2021-02-10T11:39:28.637 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:28.639 7fdf5cf24700 1 Processor -- start 2021-02-10T11:39:28.637 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:28.639 7fdf5cf24700 1 -- start start 2021-02-10T11:39:28.637 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:28.640 7fdf5cf24700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fdf5818d870 0x7fdf580b19b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:28.637 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:28.640 7fdf5cf24700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdf580b1fd0 0x7fdf58099ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:28.637 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:28.640 7fdf5cf24700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdf580b23d0 0x7fdf5809a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:28.637 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:28.640 7fdf5cf24700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf5818dc70 con 0x7fdf580b1fd0 2021-02-10T11:39:28.637 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:28.640 7fdf5cf24700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf58199e60 con 0x7fdf5818d870 2021-02-10T11:39:28.637 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:28.640 7fdf5cf24700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf5809ab30 con 0x7fdf580b23d0 2021-02-10T11:39:28.638 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:28.640 7fdf56312700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdf580b1fd0 0x7fdf58099ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:28.638 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:28.640 7fdf56312700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdf580b1fd0 0x7fdf58099ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:53772/0 (socket says 172.21.2.112:53772) 2021-02-10T11:39:28.638 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:28.640 7fdf56312700 1 -- 172.21.2.112:0/2981576524 learned_addr learned my addr 172.21.2.112:0/2981576524 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:39:28.638 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:28.640 7fdf57314700 1 --2- 172.21.2.112:0/2981576524 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdf580b23d0 0x7fdf5809a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:28.638 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:28.640 7fdf56312700 1 -- 172.21.2.112:0/2981576524 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdf580b23d0 msgr2=0x7fdf5809a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:28.638 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:28.640 7fdf56b13700 1 --2- 172.21.2.112:0/2981576524 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fdf5818d870 0x7fdf580b19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:28.638 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:28.641 7fdf56312700 1 --2- 172.21.2.112:0/2981576524 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdf580b23d0 0x7fdf5809a510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:28.638 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:28.641 7fdf56312700 1 -- 172.21.2.112:0/2981576524 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fdf5818d870 msgr2=0x7fdf580b19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:28.638 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:28.641 7fdf56312700 1 --2- 172.21.2.112:0/2981576524 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fdf5818d870 0x7fdf580b19b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:28.638 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:28.641 7fdf56312700 1 -- 172.21.2.112:0/2981576524 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf5809ad50 con 0x7fdf580b1fd0 2021-02-10T11:39:28.638 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:28.641 7fdf56b13700 1 --2- 172.21.2.112:0/2981576524 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fdf5818d870 0x7fdf580b19b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:39:28.638 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:28.641 7fdf56312700 1 --2- 172.21.2.112:0/2981576524 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdf580b1fd0 0x7fdf58099ef0 secure :-1 s=READY pgs=1918 cs=0 l=1 rev1=1 rx=0x7fdf4c006f60 tx=0x7fdf4c00b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:28.638 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:28.641 7fdf477fe700 1 -- 172.21.2.112:0/2981576524 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdf4c0009b0 con 0x7fdf580b1fd0 2021-02-10T11:39:28.639 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:28.641 7fdf477fe700 1 -- 172.21.2.112:0/2981576524 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf4c00bdf0 con 0x7fdf580b1fd0 2021-02-10T11:39:28.639 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:28.641 7fdf477fe700 1 -- 172.21.2.112:0/2981576524 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdf4c002030 con 0x7fdf580b1fd0 2021-02-10T11:39:28.639 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:28.641 7fdf5cf24700 1 -- 172.21.2.112:0/2981576524 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf5809ad50 con 0x7fdf580b1fd0 2021-02-10T11:39:28.639 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:28.641 7fdf5cf24700 1 -- 172.21.2.112:0/2981576524 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf5818e3f0 con 0x7fdf580b1fd0 2021-02-10T11:39:28.639 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:28.642 7fdf477fe700 1 -- 172.21.2.112:0/2981576524 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fdf4c006b00 con 0x7fdf580b1fd0 2021-02-10T11:39:28.639 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:28.642 7fdf477fe700 1 --2- 172.21.2.112:0/2981576524 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fdf3c0186c0 0x7fdf3c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:28.639 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:28.642 7fdf477fe700 1 -- 172.21.2.112:0/2981576524 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fdf4c0323c0 con 0x7fdf580b1fd0 2021-02-10T11:39:28.639 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:28.642 7fdf56b13700 1 --2- 172.21.2.112:0/2981576524 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fdf3c0186c0 0x7fdf3c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:28.639 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:28.643 7fdf56b13700 1 --2- 172.21.2.112:0/2981576524 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fdf3c0186c0 0x7fdf3c01abc0 crc :-1 s=READY pgs=898 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:28.666 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:28.672 7fdf5cf24700 1 -- 172.21.2.112:0/2981576524 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf580999c0 con 0x7fdf580b1fd0 2021-02-10T11:39:28.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:28.674 7fdf477fe700 1 -- 172.21.2.112:0/2981576524 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fdf4c02f390 con 0x7fdf580b1fd0 2021-02-10T11:39:28.801 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:28.807 7fdf5cf24700 1 -- 172.21.2.112:0/2981576524 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_default_uid_gid_subvolume"]} v 0) v1 -- 0x7fdf580999c0 con 0x7fdf580b1fd0 2021-02-10T11:39:29.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:29.173 7fdf477fe700 1 -- 172.21.2.112:0/2981576524 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_default_uid_gid_subvolume"]}]=0 v505) v1 ==== 150+0+0 (secure 0 0 0) 0x7fdf4c02f390 con 0x7fdf580b1fd0 2021-02-10T11:39:29.218 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:29.223 7fdf5cf24700 1 -- 172.21.2.112:0/2981576524 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fdf3c0186c0 msgr2=0x7fdf3c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:29.218 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:29.223 7fdf5cf24700 1 --2- 172.21.2.112:0/2981576524 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fdf3c0186c0 0x7fdf3c01abc0 crc :-1 s=READY pgs=898 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:29.218 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:29.223 7fdf5cf24700 1 -- 172.21.2.112:0/2981576524 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdf580b1fd0 msgr2=0x7fdf58099ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:29.218 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:29.223 7fdf5cf24700 1 --2- 172.21.2.112:0/2981576524 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdf580b1fd0 0x7fdf58099ef0 secure :-1 s=READY pgs=1918 cs=0 l=1 rev1=1 rx=0x7fdf4c006f60 tx=0x7fdf4c00b970).stop 2021-02-10T11:39:29.218 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:29.224 7fdf5cf24700 1 -- 172.21.2.112:0/2981576524 shutdown_connections 2021-02-10T11:39:29.219 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:29.224 7fdf5cf24700 1 --2- 172.21.2.112:0/2981576524 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fdf3c0186c0 0x7fdf3c01abc0 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:29.219 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:29.224 7fdf5cf24700 1 --2- 172.21.2.112:0/2981576524 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdf580b23d0 0x7fdf5809a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:29.219 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:29.224 7fdf5cf24700 1 --2- 172.21.2.112:0/2981576524 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdf580b1fd0 0x7fdf58099ef0 unknown :-1 s=CLOSED pgs=1918 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:29.219 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:29.224 7fdf5cf24700 1 --2- 172.21.2.112:0/2981576524 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fdf5818d870 0x7fdf580b19b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:29.219 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:29.224 7fdf5cf24700 1 -- 172.21.2.112:0/2981576524 >> 172.21.2.112:0/2981576524 conn(0x7fdf580f0550 msgr2=0x7fdf58191730 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:29.219 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:29.224 7fdf5cf24700 1 -- 172.21.2.112:0/2981576524 shutdown_connections 2021-02-10T11:39:29.219 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:29.224 7fdf5cf24700 1 -- 172.21.2.112:0/2981576524 wait complete. 2021-02-10T11:39:29.227 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:39:29.658 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:29.647 7f38b9d4b700 1 Processor -- start 2021-02-10T11:39:29.658 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:29.647 7f38b9d4b700 1 -- start start 2021-02-10T11:39:29.658 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:29.647 7f38b9d4b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f38b40d1870 0x7f38b41b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:29.658 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:29.647 7f38b9d4b700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f38b40cff70 0x7f38b4109130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:29.658 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:29.647 7f38b9d4b700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38b4109a60 con 0x7f38b41b0ae0 2021-02-10T11:39:29.658 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:29.647 7f38b9d4b700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38b4109750 con 0x7f38b40cff70 2021-02-10T11:39:29.658 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:29.647 7f38b9d4b700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38b4088070 con 0x7f38b40d1870 2021-02-10T11:39:29.658 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:29.647 7f38b2ffd700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f38b41b0ae0 0x7f38b4108940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:53242/0 (socket says 172.21.2.112:53242) 2021-02-10T11:39:29.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:29.647 7f38b2ffd700 1 -- 172.21.2.112:0/2790146246 learned_addr learned my addr 172.21.2.112:0/2790146246 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:39:29.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:29.648 7f38b3fff700 1 --2- 172.21.2.112:0/2790146246 >> v2:172.21.2.120:3300/0 conn(0x7f38b40cff70 0x7f38b4109130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:29.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:29.648 7f38b37fe700 1 --2- 172.21.2.112:0/2790146246 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f38b40d1870 0x7f38b41b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:29.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:29.648 7f38b27fc700 1 -- 172.21.2.112:0/2790146246 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 627961915 0 0) 0x7f38a80010b0 con 0x7f38b41b0ae0 2021-02-10T11:39:29.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:29.648 7f38b27fc700 1 -- 172.21.2.112:0/2790146246 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f389c002fe0 con 0x7f38b41b0ae0 2021-02-10T11:39:29.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:29.648 7f38b37fe700 1 -- 172.21.2.112:0/2790146246 >> v2:172.21.2.120:3300/0 conn(0x7f38b40cff70 msgr2=0x7f38b4109130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:29.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:29.648 7f38b37fe700 1 --2- 172.21.2.112:0/2790146246 >> v2:172.21.2.120:3300/0 conn(0x7f38b40cff70 0x7f38b4109130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:29.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:29.648 7f38b37fe700 1 -- 172.21.2.112:0/2790146246 >> v1:172.21.2.112:6789/0 conn(0x7f38b41b0ae0 legacy=0x7f38b4108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:29.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:29.648 7f38b37fe700 1 -- 172.21.2.112:0/2790146246 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38b4088260 con 0x7f38b40d1870 2021-02-10T11:39:29.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:29.648 7f38b3fff700 1 --2- 172.21.2.112:0/2790146246 >> v2:172.21.2.120:3300/0 conn(0x7f38b40cff70 0x7f38b4109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:39:29.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:29.648 7f38b37fe700 1 --2- 172.21.2.112:0/2790146246 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f38b40d1870 0x7f38b41b04c0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7f38a4006650 tx=0x7f38a4009a80).ready entity=mon.2 client_cookie=ec9ce1a82cd5f767 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:29.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:29.649 7f38b27fc700 1 -- 172.21.2.112:0/2790146246 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f38a400c0c0 con 0x7f38b40d1870 2021-02-10T11:39:29.660 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:29.649 7f38b27fc700 1 -- 172.21.2.112:0/2790146246 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38a400c340 con 0x7f38b40d1870 2021-02-10T11:39:29.660 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:29.649 7f38b27fc700 1 -- 172.21.2.112:0/2790146246 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f38a400c860 con 0x7f38b40d1870 2021-02-10T11:39:29.660 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:29.649 7f38b9d4b700 1 -- 172.21.2.112:0/2790146246 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f38b40d1870 msgr2=0x7f38b41b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:29.660 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:29.649 7f38b9d4b700 1 --2- 172.21.2.112:0/2790146246 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f38b40d1870 0x7f38b41b04c0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7f38a4006650 tx=0x7f38a4009a80).stop 2021-02-10T11:39:29.660 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:29.649 7f38b9d4b700 1 -- 172.21.2.112:0/2790146246 shutdown_connections 2021-02-10T11:39:29.660 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:29.649 7f38b9d4b700 1 --2- 172.21.2.112:0/2790146246 >> v2:172.21.2.120:3300/0 conn(0x7f38b40cff70 0x7f38b4109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:29.660 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:29.649 7f38b9d4b700 1 --2- 172.21.2.112:0/2790146246 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f38b40d1870 0x7f38b41b04c0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:29.660 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:29.649 7f38b9d4b700 1 -- 172.21.2.112:0/2790146246 >> 172.21.2.112:0/2790146246 conn(0x7f38b40ce420 msgr2=0x7f38b40d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:29.660 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:29.649 7f38b9d4b700 1 -- 172.21.2.112:0/2790146246 shutdown_connections 2021-02-10T11:39:29.660 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:29.650 7f38b9d4b700 1 -- 172.21.2.112:0/2790146246 wait complete. 2021-02-10T11:39:29.662 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:29.664 7f38b9d4b700 1 Processor -- start 2021-02-10T11:39:29.662 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:29.664 7f38b9d4b700 1 -- start start 2021-02-10T11:39:29.662 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:29.664 7f38b9d4b700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f38b40c1850 0x7f38b40c1f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:29.662 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:29.664 7f38b9d4b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f38b40c2810 0x7f38b40c2c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:29.662 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:29.664 7f38b9d4b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f38b40c3230 0x7f38b40c3630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:29.662 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:29.664 7f38b9d4b700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38b40c1c50 con 0x7f38b40c3230 2021-02-10T11:39:29.662 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:29.664 7f38b9d4b700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38b4084920 con 0x7f38b40c1850 2021-02-10T11:39:29.662 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:29.664 7f38b9d4b700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38b4084b40 con 0x7f38b40c2810 2021-02-10T11:39:29.662 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:29.664 7f38b3fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f38b40c3230 0x7f38b40c3630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:29.662 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:29.664 7f38b3fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f38b40c3230 0x7f38b40c3630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:53786/0 (socket says 172.21.2.112:53786) 2021-02-10T11:39:29.662 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:29.664 7f38b3fff700 1 -- 172.21.2.112:0/1436412725 learned_addr learned my addr 172.21.2.112:0/1436412725 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:39:29.662 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:29.665 7f38b3fff700 1 -- 172.21.2.112:0/1436412725 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f38b40c2810 msgr2=0x7f38b40c2c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:29.663 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:29.665 7f38b3fff700 1 --2- 172.21.2.112:0/1436412725 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f38b40c2810 0x7f38b40c2c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:29.663 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:29.665 7f38b3fff700 1 -- 172.21.2.112:0/1436412725 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f38b40c1850 msgr2=0x7f38b40c1f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:29.663 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:29.665 7f38b3fff700 1 --2- 172.21.2.112:0/1436412725 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f38b40c1850 0x7f38b40c1f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:29.663 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:29.665 7f38b3fff700 1 -- 172.21.2.112:0/1436412725 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38b4084d60 con 0x7f38b40c3230 2021-02-10T11:39:29.663 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:29.665 7f38b3fff700 1 --2- 172.21.2.112:0/1436412725 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f38b40c3230 0x7f38b40c3630 secure :-1 s=READY pgs=1920 cs=0 l=1 rev1=1 rx=0x7f38ac00b910 tx=0x7f38ac00bcb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:29.663 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:29.665 7f389bfff700 1 -- 172.21.2.112:0/1436412725 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f38ac006aa0 con 0x7f38b40c3230 2021-02-10T11:39:29.663 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:29.665 7f389bfff700 1 -- 172.21.2.112:0/1436412725 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38ac0011f0 con 0x7f38b40c3230 2021-02-10T11:39:29.663 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:29.665 7f38b9d4b700 1 -- 172.21.2.112:0/1436412725 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38b4084d60 con 0x7f38b40c3230 2021-02-10T11:39:29.663 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:29.665 7f389bfff700 1 -- 172.21.2.112:0/1436412725 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f38ac001600 con 0x7f38b40c3230 2021-02-10T11:39:29.663 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:29.665 7f38b9d4b700 1 -- 172.21.2.112:0/1436412725 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38b4084d60 con 0x7f38b40c3230 2021-02-10T11:39:29.663 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:29.666 7f389bfff700 1 -- 172.21.2.112:0/1436412725 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f38ac006aa0 con 0x7f38b40c3230 2021-02-10T11:39:29.663 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:29.666 7f389bfff700 1 --2- 172.21.2.112:0/1436412725 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f38940186c0 0x7f389401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:29.663 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:29.666 7f389bfff700 1 -- 172.21.2.112:0/1436412725 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f38ac032760 con 0x7f38b40c3230 2021-02-10T11:39:29.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:29.666 7f38b37fe700 1 --2- 172.21.2.112:0/1436412725 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f38940186c0 0x7f389401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:29.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:29.667 7f38b37fe700 1 --2- 172.21.2.112:0/1436412725 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f38940186c0 0x7f389401abc0 crc :-1 s=READY pgs=899 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:29.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:29.697 7f38b9d4b700 1 -- 172.21.2.112:0/1436412725 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38b4109a60 con 0x7f38b40c3230 2021-02-10T11:39:29.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:29.700 7f389bfff700 1 -- 172.21.2.112:0/1436412725 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f38ac02d8a0 con 0x7f38b40c3230 2021-02-10T11:39:29.844 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:29.850 7f38b9d4b700 1 -- 172.21.2.112:0/1436412725 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f38b4109a60 con 0x7f38b40c3230 2021-02-10T11:39:29.848 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:29.854 7f389bfff700 1 -- 172.21.2.112:0/1436412725 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v26)=0 v26) v1 ==== 122+0+0 (secure 0 0 0) 0x7f38ac02d8a0 con 0x7f38b40c3230 2021-02-10T11:39:29.853 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:29.859 7f38b9d4b700 1 -- 172.21.2.112:0/1436412725 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f38940186c0 msgr2=0x7f389401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:29.854 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:29.859 7f38b9d4b700 1 --2- 172.21.2.112:0/1436412725 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f38940186c0 0x7f389401abc0 crc :-1 s=READY pgs=899 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:29.854 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:29.859 7f38b9d4b700 1 -- 172.21.2.112:0/1436412725 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f38b40c3230 msgr2=0x7f38b40c3630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:29.854 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:29.859 7f38b9d4b700 1 --2- 172.21.2.112:0/1436412725 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f38b40c3230 0x7f38b40c3630 secure :-1 s=READY pgs=1920 cs=0 l=1 rev1=1 rx=0x7f38ac00b910 tx=0x7f38ac00bcb0).stop 2021-02-10T11:39:29.854 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:29.859 7f38b9d4b700 1 -- 172.21.2.112:0/1436412725 shutdown_connections 2021-02-10T11:39:29.854 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:29.859 7f38b9d4b700 1 --2- 172.21.2.112:0/1436412725 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f38940186c0 0x7f389401abc0 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:29.854 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:29.859 7f38b9d4b700 1 --2- 172.21.2.112:0/1436412725 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f38b40c3230 0x7f38b40c3630 unknown :-1 s=CLOSED pgs=1920 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:29.854 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:29.859 7f38b9d4b700 1 --2- 172.21.2.112:0/1436412725 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f38b40c2810 0x7f38b40c2c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:29.854 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:29.859 7f38b9d4b700 1 --2- 172.21.2.112:0/1436412725 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f38b40c1850 0x7f38b40c1f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:29.854 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:29.859 7f38b9d4b700 1 -- 172.21.2.112:0/1436412725 >> 172.21.2.112:0/1436412725 conn(0x7f38b40af690 msgr2=0x7f38b4094c40 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:29.855 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:29.860 7f38b9d4b700 1 -- 172.21.2.112:0/1436412725 shutdown_connections 2021-02-10T11:39:29.855 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:29.860 7f38b9d4b700 1 -- 172.21.2.112:0/1436412725 wait complete. 2021-02-10T11:39:29.862 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:39:29.862 DEBUG:teuthology.orchestra.run.gibba012:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:39:30.207 DEBUG:teuthology.orchestra.run.gibba020:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:39:30.551 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:39:30.552 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:39:30.552 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:39:30.552 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:39:30.887 INFO:teuthology.orchestra.run.gibba012.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.0’: No such file or directory 2021-02-10T11:39:30.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:39:30.887 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2021-02-10T11:39:30.888 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:39:30.888 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:39:30.888 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:39:31.222 INFO:teuthology.orchestra.run.gibba012.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: No such file or directory 2021-02-10T11:39:31.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:39:31.223 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2021-02-10T11:39:31.223 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:39:31.223 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:39:31.223 DEBUG:teuthology.orchestra.run.gibba020:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:39:31.559 INFO:teuthology.orchestra.run.gibba020.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:39:31.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:39:31.560 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:39:31.560 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:39:31.560 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:39:31.560 DEBUG:teuthology.orchestra.run.gibba020:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:39:31.894 INFO:teuthology.orchestra.run.gibba020.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:39:31.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:39:31.894 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:39:31.894 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json-pretty 2021-02-10T11:39:32.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.311 7ff06204f700 1 Processor -- start 2021-02-10T11:39:32.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.311 7ff06204f700 1 -- start start 2021-02-10T11:39:32.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.312 7ff06204f700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff05c0d1870 0x7ff05c1b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:32.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.312 7ff06204f700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff05c0f0b10 con 0x7ff05c1b0a10 2021-02-10T11:39:32.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.312 7ff06204f700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff05c0d0310 con 0x7ff05c0f0650 2021-02-10T11:39:32.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.312 7ff06204f700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff05c0e5ed0 con 0x7ff05c0d1870 2021-02-10T11:39:32.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.312 7ff05affd700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7ff05c1b0a10 0x7ff05c0cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:53254/0 (socket says 172.21.2.112:53254) 2021-02-10T11:39:32.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.312 7ff05affd700 1 -- 172.21.2.112:0/2374311126 learned_addr learned my addr 172.21.2.112:0/2374311126 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:39:32.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.313 7ff05b7fe700 1 --2- 172.21.2.112:0/2374311126 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff05c0d1870 0x7ff05c1b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:32.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.313 7ff05a7fc700 1 -- 172.21.2.112:0/2374311126 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2733848637 0 0) 0x7ff0440010b0 con 0x7ff05c1b0a10 2021-02-10T11:39:32.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.313 7ff05a7fc700 1 -- 172.21.2.112:0/2374311126 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff048002fe0 con 0x7ff05c1b0a10 2021-02-10T11:39:32.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.313 7ff05a7fc700 1 -- 172.21.2.112:0/2374311126 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 881003915 0 0) 0x7ff0440010b0 con 0x7ff05c1b0a10 2021-02-10T11:39:32.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.313 7ff05a7fc700 1 -- 172.21.2.112:0/2374311126 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff05c0d1870 msgr2=0x7ff05c1b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:32.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.313 7ff05a7fc700 1 --2- 172.21.2.112:0/2374311126 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff05c0d1870 0x7ff05c1b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:32.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.313 7ff05a7fc700 1 -- 172.21.2.112:0/2374311126 >> v1:172.21.2.120:6789/0 conn(0x7ff05c0f0650 legacy=0x7ff05c0cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:32.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.313 7ff05a7fc700 1 -- 172.21.2.112:0/2374311126 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff05c0d0530 con 0x7ff05c1b0a10 2021-02-10T11:39:32.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.313 7ff05a7fc700 1 -- 172.21.2.112:0/2374311126 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7ff0440015b0 con 0x7ff05c1b0a10 2021-02-10T11:39:32.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.313 7ff05a7fc700 1 -- 172.21.2.112:0/2374311126 >> v1:172.21.2.112:6789/0 conn(0x7ff05c1b0a10 legacy=0x7ff05c0cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:32.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.313 7ff05a7fc700 1 --2- 172.21.2.112:0/2374311126 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff048003ac0 0x7ff048005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:32.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.313 7ff05a7fc700 1 --2- 172.21.2.112:0/2374311126 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff048006790 0x7ff048008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:32.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.313 7ff05a7fc700 1 --2- 172.21.2.112:0/2374311126 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff048009590 0x7ff04800ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:32.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.313 7ff05a7fc700 1 -- 172.21.2.112:0/2374311126 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff04800c2d0 con 0x7ff048003ac0 2021-02-10T11:39:32.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.313 7ff05a7fc700 1 -- 172.21.2.112:0/2374311126 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff04800c4f0 con 0x7ff048006790 2021-02-10T11:39:32.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.313 7ff05a7fc700 1 -- 172.21.2.112:0/2374311126 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff04800c710 con 0x7ff048009590 2021-02-10T11:39:32.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.313 7ff05b7fe700 1 --2- 172.21.2.112:0/2374311126 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff048003ac0 0x7ff048005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:32.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.313 7ff05affd700 1 --2- 172.21.2.112:0/2374311126 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff048006790 0x7ff048008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:32.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.313 7ff05bfff700 1 --2- 172.21.2.112:0/2374311126 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff048009590 0x7ff04800ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:32.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.313 7ff05b7fe700 1 -- 172.21.2.112:0/2374311126 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff048009590 msgr2=0x7ff04800ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:32.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.313 7ff05b7fe700 1 --2- 172.21.2.112:0/2374311126 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff048009590 0x7ff04800ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:32.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.314 7ff05b7fe700 1 -- 172.21.2.112:0/2374311126 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff048006790 msgr2=0x7ff048008c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:32.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.314 7ff05b7fe700 1 --2- 172.21.2.112:0/2374311126 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff048006790 0x7ff048008c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:32.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.314 7ff05b7fe700 1 -- 172.21.2.112:0/2374311126 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff048050980 con 0x7ff048003ac0 2021-02-10T11:39:32.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.314 7ff05b7fe700 1 --2- 172.21.2.112:0/2374311126 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff048003ac0 0x7ff048005ef0 secure :-1 s=READY pgs=1922 cs=0 l=1 rev1=1 rx=0x7ff04c009310 tx=0x7ff04c009a90).ready entity=mon.0 client_cookie=6a655af89edb2ab6 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:32.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.314 7ff05a7fc700 1 -- 172.21.2.112:0/2374311126 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff04c00c0c0 con 0x7ff048003ac0 2021-02-10T11:39:32.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.314 7ff05a7fc700 1 -- 172.21.2.112:0/2374311126 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff04c006400 con 0x7ff048003ac0 2021-02-10T11:39:32.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.314 7ff05a7fc700 1 -- 172.21.2.112:0/2374311126 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff04c00c0c0 con 0x7ff048003ac0 2021-02-10T11:39:32.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.314 7ff06204f700 1 -- 172.21.2.112:0/2374311126 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff048003ac0 msgr2=0x7ff048005ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:32.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.314 7ff06204f700 1 --2- 172.21.2.112:0/2374311126 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff048003ac0 0x7ff048005ef0 secure :-1 s=READY pgs=1922 cs=0 l=1 rev1=1 rx=0x7ff04c009310 tx=0x7ff04c009a90).stop 2021-02-10T11:39:32.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.314 7ff05bfff700 1 -- 172.21.2.112:0/2374311126 reap_dead start 2021-02-10T11:39:32.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.314 7ff06204f700 1 -- 172.21.2.112:0/2374311126 shutdown_connections 2021-02-10T11:39:32.326 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.314 7ff06204f700 1 -- 172.21.2.112:0/2374311126 >> 172.21.2.112:0/2374311126 conn(0x7ff05c0ce450 msgr2=0x7ff05c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:32.326 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.314 7ff06204f700 1 -- 172.21.2.112:0/2374311126 shutdown_connections 2021-02-10T11:39:32.326 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.314 7ff06204f700 1 -- 172.21.2.112:0/2374311126 wait complete. 2021-02-10T11:39:32.326 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.329 7ff06204f700 1 Processor -- start 2021-02-10T11:39:32.327 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.329 7ff06204f700 1 -- start start 2021-02-10T11:39:32.327 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.330 7ff06204f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff05c0baf60 0x7ff05c0b3070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:32.327 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.330 7ff06204f700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff05c0b3690 0x7ff05c0b8400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:32.327 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.330 7ff06204f700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff05c0b8a20 0x7ff05c0b6690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:32.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.330 7ff06204f700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff05c0b8e20 con 0x7ff05c0baf60 2021-02-10T11:39:32.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.330 7ff06204f700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff05c0bb360 con 0x7ff05c0b3690 2021-02-10T11:39:32.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.330 7ff06204f700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff05c0b6cb0 con 0x7ff05c0b8a20 2021-02-10T11:39:32.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.330 7ff05b7fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff05c0baf60 0x7ff05c0b3070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:32.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.330 7ff05b7fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff05c0baf60 0x7ff05c0b3070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:53802/0 (socket says 172.21.2.112:53802) 2021-02-10T11:39:32.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.330 7ff05b7fe700 1 -- 172.21.2.112:0/1758467428 learned_addr learned my addr 172.21.2.112:0/1758467428 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:39:32.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.330 7ff05b7fe700 1 -- 172.21.2.112:0/1758467428 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff05c0b8a20 msgr2=0x7ff05c0b6690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:32.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.330 7ff05b7fe700 1 --2- 172.21.2.112:0/1758467428 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff05c0b8a20 0x7ff05c0b6690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:32.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.330 7ff05b7fe700 1 -- 172.21.2.112:0/1758467428 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff05c0b3690 msgr2=0x7ff05c0b8400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:32.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.330 7ff05b7fe700 1 --2- 172.21.2.112:0/1758467428 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff05c0b3690 0x7ff05c0b8400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:32.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.330 7ff05b7fe700 1 -- 172.21.2.112:0/1758467428 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff05c0b6ed0 con 0x7ff05c0baf60 2021-02-10T11:39:32.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.330 7ff05b7fe700 1 --2- 172.21.2.112:0/1758467428 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff05c0baf60 0x7ff05c0b3070 secure :-1 s=READY pgs=1923 cs=0 l=1 rev1=1 rx=0x7ff04c009670 tx=0x7ff04c00b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:32.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.331 7ff053fff700 1 -- 172.21.2.112:0/1758467428 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff04c0093e0 con 0x7ff05c0baf60 2021-02-10T11:39:32.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.331 7ff053fff700 1 -- 172.21.2.112:0/1758467428 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff04c00bd50 con 0x7ff05c0baf60 2021-02-10T11:39:32.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.331 7ff053fff700 1 -- 172.21.2.112:0/1758467428 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff04c002300 con 0x7ff05c0baf60 2021-02-10T11:39:32.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.331 7ff06204f700 1 -- 172.21.2.112:0/1758467428 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff05c0b6ed0 con 0x7ff05c0baf60 2021-02-10T11:39:32.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.331 7ff06204f700 1 -- 172.21.2.112:0/1758467428 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff05c0b7460 con 0x7ff05c0baf60 2021-02-10T11:39:32.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.331 7ff053fff700 1 -- 172.21.2.112:0/1758467428 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff04c002020 con 0x7ff05c0baf60 2021-02-10T11:39:32.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.331 7ff053fff700 1 --2- 172.21.2.112:0/1758467428 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff0400186c0 0x7ff04001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:32.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.332 7ff053fff700 1 -- 172.21.2.112:0/1758467428 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5573+0+0 (secure 0 0 0) 0x7ff04c0325a0 con 0x7ff05c0baf60 2021-02-10T11:39:32.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.332 7ff05affd700 1 --2- 172.21.2.112:0/1758467428 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff0400186c0 0x7ff04001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:32.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.332 7ff05affd700 1 --2- 172.21.2.112:0/1758467428 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff0400186c0 0x7ff04001abc0 crc :-1 s=READY pgs=900 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:32.356 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.362 7ff06204f700 1 -- 172.21.2.112:0/1758467428 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff05c0ebcd0 con 0x7ff05c0baf60 2021-02-10T11:39:32.358 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.365 7ff053fff700 1 -- 172.21.2.112:0/1758467428 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff04c0093a0 con 0x7ff05c0baf60 2021-02-10T11:39:32.483 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.489 7ff06204f700 1 -- 172.21.2.112:0/1758467428 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7ff05c0f0b10 con 0x7ff05c0baf60 2021-02-10T11:39:32.483 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.490 7ff053fff700 1 -- 172.21.2.112:0/1758467428 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v92) v1 ==== 81+0+30363 (secure 0 0 0) 0x7ff04c030d50 con 0x7ff05c0baf60 2021-02-10T11:39:32.484 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:39:32.484 INFO:teuthology.orchestra.run.gibba012.stdout:{ 2021-02-10T11:39:32.485 INFO:teuthology.orchestra.run.gibba012.stdout: "epoch": 92, 2021-02-10T11:39:32.485 INFO:teuthology.orchestra.run.gibba012.stdout: "fsid": "f14f004e-0699-4584-ad81-3754258493de", 2021-02-10T11:39:32.485 INFO:teuthology.orchestra.run.gibba012.stdout: "created": "2021-02-10 11:13:26.501424", 2021-02-10T11:39:32.485 INFO:teuthology.orchestra.run.gibba012.stdout: "modified": "2021-02-10 11:38:52.412732", 2021-02-10T11:39:32.485 INFO:teuthology.orchestra.run.gibba012.stdout: "last_up_change": "2021-02-10 11:13:52.780316", 2021-02-10T11:39:32.485 INFO:teuthology.orchestra.run.gibba012.stdout: "last_in_change": "2021-02-10 11:13:52.780316", 2021-02-10T11:39:32.485 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-02-10T11:39:32.485 INFO:teuthology.orchestra.run.gibba012.stdout: "flags_num": 5799936, 2021-02-10T11:39:32.485 INFO:teuthology.orchestra.run.gibba012.stdout: "flags_set": [ 2021-02-10T11:39:32.485 INFO:teuthology.orchestra.run.gibba012.stdout: "pglog_hardlimit", 2021-02-10T11:39:32.485 INFO:teuthology.orchestra.run.gibba012.stdout: "purged_snapdirs", 2021-02-10T11:39:32.485 INFO:teuthology.orchestra.run.gibba012.stdout: "recovery_deletes", 2021-02-10T11:39:32.486 INFO:teuthology.orchestra.run.gibba012.stdout: "sortbitwise" 2021-02-10T11:39:32.486 INFO:teuthology.orchestra.run.gibba012.stdout: ], 2021-02-10T11:39:32.486 INFO:teuthology.orchestra.run.gibba012.stdout: "crush_version": 4, 2021-02-10T11:39:32.486 INFO:teuthology.orchestra.run.gibba012.stdout: "full_ratio": 0.94999998807907104, 2021-02-10T11:39:32.486 INFO:teuthology.orchestra.run.gibba012.stdout: "backfillfull_ratio": 0.89999997615814209, 2021-02-10T11:39:32.486 INFO:teuthology.orchestra.run.gibba012.stdout: "nearfull_ratio": 0.85000002384185791, 2021-02-10T11:39:32.486 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_snapshot": "", 2021-02-10T11:39:32.486 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_max": 19, 2021-02-10T11:39:32.486 INFO:teuthology.orchestra.run.gibba012.stdout: "max_osd": 8, 2021-02-10T11:39:32.486 INFO:teuthology.orchestra.run.gibba012.stdout: "require_min_compat_client": "jewel", 2021-02-10T11:39:32.486 INFO:teuthology.orchestra.run.gibba012.stdout: "min_compat_client": "jewel", 2021-02-10T11:39:32.486 INFO:teuthology.orchestra.run.gibba012.stdout: "require_osd_release": "nautilus", 2021-02-10T11:39:32.487 INFO:teuthology.orchestra.run.gibba012.stdout: "pools": [ 2021-02-10T11:39:32.487 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:39:32.487 INFO:teuthology.orchestra.run.gibba012.stdout: "pool": 1, 2021-02-10T11:39:32.487 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_name": "rbd", 2021-02-10T11:39:32.487 INFO:teuthology.orchestra.run.gibba012.stdout: "create_time": "2021-02-10 11:13:57.508758", 2021-02-10T11:39:32.487 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:39:32.487 INFO:teuthology.orchestra.run.gibba012.stdout: "flags_names": "hashpspool", 2021-02-10T11:39:32.487 INFO:teuthology.orchestra.run.gibba012.stdout: "type": 1, 2021-02-10T11:39:32.487 INFO:teuthology.orchestra.run.gibba012.stdout: "size": 2, 2021-02-10T11:39:32.487 INFO:teuthology.orchestra.run.gibba012.stdout: "min_size": 1, 2021-02-10T11:39:32.487 INFO:teuthology.orchestra.run.gibba012.stdout: "crush_rule": 0, 2021-02-10T11:39:32.487 INFO:teuthology.orchestra.run.gibba012.stdout: "object_hash": 2, 2021-02-10T11:39:32.487 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:39:32.488 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num": 8, 2021-02-10T11:39:32.488 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num": 8, 2021-02-10T11:39:32.488 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num_target": 8, 2021-02-10T11:39:32.488 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_target": 8, 2021-02-10T11:39:32.488 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_pending": 8, 2021-02-10T11:39:32.488 INFO:teuthology.orchestra.run.gibba012.stdout: "last_pg_merge_meta": { 2021-02-10T11:39:32.488 INFO:teuthology.orchestra.run.gibba012.stdout: "source_pgid": "0.0", 2021-02-10T11:39:32.488 INFO:teuthology.orchestra.run.gibba012.stdout: "ready_epoch": 0, 2021-02-10T11:39:32.488 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_started": 0, 2021-02-10T11:39:32.488 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_clean": 0, 2021-02-10T11:39:32.488 INFO:teuthology.orchestra.run.gibba012.stdout: "source_version": "0'0", 2021-02-10T11:39:32.488 INFO:teuthology.orchestra.run.gibba012.stdout: "target_version": "0'0" 2021-02-10T11:39:32.488 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:39:32.489 INFO:teuthology.orchestra.run.gibba012.stdout: "last_change": "16", 2021-02-10T11:39:32.489 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend": "0", 2021-02-10T11:39:32.489 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:39:32.489 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:39:32.489 INFO:teuthology.orchestra.run.gibba012.stdout: "auid": 0, 2021-02-10T11:39:32.489 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:39:32.489 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_seq": 0, 2021-02-10T11:39:32.489 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_epoch": 0, 2021-02-10T11:39:32.489 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_snaps": [], 2021-02-10T11:39:32.489 INFO:teuthology.orchestra.run.gibba012.stdout: "removed_snaps": "[]", 2021-02-10T11:39:32.489 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_bytes": 0, 2021-02-10T11:39:32.489 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_objects": 0, 2021-02-10T11:39:32.489 INFO:teuthology.orchestra.run.gibba012.stdout: "tiers": [], 2021-02-10T11:39:32.489 INFO:teuthology.orchestra.run.gibba012.stdout: "tier_of": -1, 2021-02-10T11:39:32.490 INFO:teuthology.orchestra.run.gibba012.stdout: "read_tier": -1, 2021-02-10T11:39:32.490 INFO:teuthology.orchestra.run.gibba012.stdout: "write_tier": -1, 2021-02-10T11:39:32.491 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_mode": "none", 2021-02-10T11:39:32.491 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_bytes": 0, 2021-02-10T11:39:32.491 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_objects": 0, 2021-02-10T11:39:32.491 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:39:32.491 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:39:32.491 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:39:32.491 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_flush_age": 0, 2021-02-10T11:39:32.491 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_evict_age": 0, 2021-02-10T11:39:32.491 INFO:teuthology.orchestra.run.gibba012.stdout: "erasure_code_profile": "", 2021-02-10T11:39:32.491 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_params": { 2021-02-10T11:39:32.491 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "none" 2021-02-10T11:39:32.492 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:39:32.492 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_period": 0, 2021-02-10T11:39:32.492 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_count": 0, 2021-02-10T11:39:32.492 INFO:teuthology.orchestra.run.gibba012.stdout: "use_gmt_hitset": true, 2021-02-10T11:39:32.492 INFO:teuthology.orchestra.run.gibba012.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:39:32.492 INFO:teuthology.orchestra.run.gibba012.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:39:32.492 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:39:32.492 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:39:32.492 INFO:teuthology.orchestra.run.gibba012.stdout: "grade_table": [], 2021-02-10T11:39:32.492 INFO:teuthology.orchestra.run.gibba012.stdout: "stripe_width": 0, 2021-02-10T11:39:32.492 INFO:teuthology.orchestra.run.gibba012.stdout: "expected_num_objects": 0, 2021-02-10T11:39:32.492 INFO:teuthology.orchestra.run.gibba012.stdout: "fast_read": false, 2021-02-10T11:39:32.492 INFO:teuthology.orchestra.run.gibba012.stdout: "options": {}, 2021-02-10T11:39:32.493 INFO:teuthology.orchestra.run.gibba012.stdout: "application_metadata": { 2021-02-10T11:39:32.493 INFO:teuthology.orchestra.run.gibba012.stdout: "rbd": {} 2021-02-10T11:39:32.493 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:39:32.493 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:39:32.493 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:39:32.493 INFO:teuthology.orchestra.run.gibba012.stdout: "pool": 18, 2021-02-10T11:39:32.493 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_name": "cephfs_metadata", 2021-02-10T11:39:32.493 INFO:teuthology.orchestra.run.gibba012.stdout: "create_time": "2021-02-10 11:38:49.384013", 2021-02-10T11:39:32.493 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:39:32.493 INFO:teuthology.orchestra.run.gibba012.stdout: "flags_names": "hashpspool", 2021-02-10T11:39:32.493 INFO:teuthology.orchestra.run.gibba012.stdout: "type": 1, 2021-02-10T11:39:32.493 INFO:teuthology.orchestra.run.gibba012.stdout: "size": 2, 2021-02-10T11:39:32.493 INFO:teuthology.orchestra.run.gibba012.stdout: "min_size": 1, 2021-02-10T11:39:32.494 INFO:teuthology.orchestra.run.gibba012.stdout: "crush_rule": 0, 2021-02-10T11:39:32.494 INFO:teuthology.orchestra.run.gibba012.stdout: "object_hash": 2, 2021-02-10T11:39:32.494 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:39:32.494 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num": 8, 2021-02-10T11:39:32.494 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num": 8, 2021-02-10T11:39:32.494 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num_target": 8, 2021-02-10T11:39:32.494 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_target": 8, 2021-02-10T11:39:32.494 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_pending": 8, 2021-02-10T11:39:32.494 INFO:teuthology.orchestra.run.gibba012.stdout: "last_pg_merge_meta": { 2021-02-10T11:39:32.494 INFO:teuthology.orchestra.run.gibba012.stdout: "source_pgid": "0.0", 2021-02-10T11:39:32.494 INFO:teuthology.orchestra.run.gibba012.stdout: "ready_epoch": 0, 2021-02-10T11:39:32.494 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_started": 0, 2021-02-10T11:39:32.494 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_clean": 0, 2021-02-10T11:39:32.495 INFO:teuthology.orchestra.run.gibba012.stdout: "source_version": "0'0", 2021-02-10T11:39:32.495 INFO:teuthology.orchestra.run.gibba012.stdout: "target_version": "0'0" 2021-02-10T11:39:32.495 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:39:32.495 INFO:teuthology.orchestra.run.gibba012.stdout: "last_change": "91", 2021-02-10T11:39:32.495 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend": "0", 2021-02-10T11:39:32.495 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:39:32.495 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:39:32.495 INFO:teuthology.orchestra.run.gibba012.stdout: "auid": 0, 2021-02-10T11:39:32.495 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:39:32.495 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_seq": 0, 2021-02-10T11:39:32.495 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_epoch": 0, 2021-02-10T11:39:32.495 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_snaps": [], 2021-02-10T11:39:32.495 INFO:teuthology.orchestra.run.gibba012.stdout: "removed_snaps": "[]", 2021-02-10T11:39:32.496 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_bytes": 0, 2021-02-10T11:39:32.496 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_objects": 0, 2021-02-10T11:39:32.496 INFO:teuthology.orchestra.run.gibba012.stdout: "tiers": [], 2021-02-10T11:39:32.496 INFO:teuthology.orchestra.run.gibba012.stdout: "tier_of": -1, 2021-02-10T11:39:32.496 INFO:teuthology.orchestra.run.gibba012.stdout: "read_tier": -1, 2021-02-10T11:39:32.496 INFO:teuthology.orchestra.run.gibba012.stdout: "write_tier": -1, 2021-02-10T11:39:32.496 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_mode": "none", 2021-02-10T11:39:32.496 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_bytes": 0, 2021-02-10T11:39:32.496 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_objects": 0, 2021-02-10T11:39:32.496 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:39:32.496 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:39:32.496 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:39:32.496 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_flush_age": 0, 2021-02-10T11:39:32.496 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_evict_age": 0, 2021-02-10T11:39:32.497 INFO:teuthology.orchestra.run.gibba012.stdout: "erasure_code_profile": "", 2021-02-10T11:39:32.497 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_params": { 2021-02-10T11:39:32.497 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "none" 2021-02-10T11:39:32.497 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:39:32.497 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_period": 0, 2021-02-10T11:39:32.497 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_count": 0, 2021-02-10T11:39:32.497 INFO:teuthology.orchestra.run.gibba012.stdout: "use_gmt_hitset": true, 2021-02-10T11:39:32.497 INFO:teuthology.orchestra.run.gibba012.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:39:32.497 INFO:teuthology.orchestra.run.gibba012.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:39:32.497 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:39:32.497 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:39:32.497 INFO:teuthology.orchestra.run.gibba012.stdout: "grade_table": [], 2021-02-10T11:39:32.498 INFO:teuthology.orchestra.run.gibba012.stdout: "stripe_width": 0, 2021-02-10T11:39:32.498 INFO:teuthology.orchestra.run.gibba012.stdout: "expected_num_objects": 0, 2021-02-10T11:39:32.498 INFO:teuthology.orchestra.run.gibba012.stdout: "fast_read": false, 2021-02-10T11:39:32.498 INFO:teuthology.orchestra.run.gibba012.stdout: "options": { 2021-02-10T11:39:32.498 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_autoscale_bias": 4, 2021-02-10T11:39:32.498 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_min": 16, 2021-02-10T11:39:32.498 INFO:teuthology.orchestra.run.gibba012.stdout: "recovery_priority": 5 2021-02-10T11:39:32.498 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:39:32.498 INFO:teuthology.orchestra.run.gibba012.stdout: "application_metadata": { 2021-02-10T11:39:32.498 INFO:teuthology.orchestra.run.gibba012.stdout: "cephfs": { 2021-02-10T11:39:32.498 INFO:teuthology.orchestra.run.gibba012.stdout: "metadata": "cephfs" 2021-02-10T11:39:32.498 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:39:32.498 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:39:32.499 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:39:32.499 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:39:32.499 INFO:teuthology.orchestra.run.gibba012.stdout: "pool": 19, 2021-02-10T11:39:32.499 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_name": "cephfs_data", 2021-02-10T11:39:32.499 INFO:teuthology.orchestra.run.gibba012.stdout: "create_time": "2021-02-10 11:38:50.454005", 2021-02-10T11:39:32.499 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:39:32.499 INFO:teuthology.orchestra.run.gibba012.stdout: "flags_names": "hashpspool", 2021-02-10T11:39:32.499 INFO:teuthology.orchestra.run.gibba012.stdout: "type": 1, 2021-02-10T11:39:32.499 INFO:teuthology.orchestra.run.gibba012.stdout: "size": 2, 2021-02-10T11:39:32.499 INFO:teuthology.orchestra.run.gibba012.stdout: "min_size": 1, 2021-02-10T11:39:32.499 INFO:teuthology.orchestra.run.gibba012.stdout: "crush_rule": 0, 2021-02-10T11:39:32.499 INFO:teuthology.orchestra.run.gibba012.stdout: "object_hash": 2, 2021-02-10T11:39:32.499 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:39:32.500 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num": 8, 2021-02-10T11:39:32.500 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num": 8, 2021-02-10T11:39:32.500 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num_target": 8, 2021-02-10T11:39:32.500 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_target": 8, 2021-02-10T11:39:32.501 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_pending": 8, 2021-02-10T11:39:32.501 INFO:teuthology.orchestra.run.gibba012.stdout: "last_pg_merge_meta": { 2021-02-10T11:39:32.501 INFO:teuthology.orchestra.run.gibba012.stdout: "source_pgid": "0.0", 2021-02-10T11:39:32.501 INFO:teuthology.orchestra.run.gibba012.stdout: "ready_epoch": 0, 2021-02-10T11:39:32.501 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_started": 0, 2021-02-10T11:39:32.501 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_clean": 0, 2021-02-10T11:39:32.501 INFO:teuthology.orchestra.run.gibba012.stdout: "source_version": "0'0", 2021-02-10T11:39:32.501 INFO:teuthology.orchestra.run.gibba012.stdout: "target_version": "0'0" 2021-02-10T11:39:32.501 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:39:32.501 INFO:teuthology.orchestra.run.gibba012.stdout: "last_change": "92", 2021-02-10T11:39:32.502 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend": "0", 2021-02-10T11:39:32.502 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:39:32.502 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:39:32.502 INFO:teuthology.orchestra.run.gibba012.stdout: "auid": 0, 2021-02-10T11:39:32.502 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:39:32.502 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_seq": 0, 2021-02-10T11:39:32.502 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_epoch": 0, 2021-02-10T11:39:32.502 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_snaps": [], 2021-02-10T11:39:32.502 INFO:teuthology.orchestra.run.gibba012.stdout: "removed_snaps": "[]", 2021-02-10T11:39:32.502 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_bytes": 0, 2021-02-10T11:39:32.502 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_objects": 0, 2021-02-10T11:39:32.503 INFO:teuthology.orchestra.run.gibba012.stdout: "tiers": [], 2021-02-10T11:39:32.503 INFO:teuthology.orchestra.run.gibba012.stdout: "tier_of": -1, 2021-02-10T11:39:32.503 INFO:teuthology.orchestra.run.gibba012.stdout: "read_tier": -1, 2021-02-10T11:39:32.503 INFO:teuthology.orchestra.run.gibba012.stdout: "write_tier": -1, 2021-02-10T11:39:32.503 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_mode": "none", 2021-02-10T11:39:32.503 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_bytes": 0, 2021-02-10T11:39:32.503 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_objects": 0, 2021-02-10T11:39:32.503 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:39:32.503 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:39:32.503 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:39:32.503 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_flush_age": 0, 2021-02-10T11:39:32.503 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_evict_age": 0, 2021-02-10T11:39:32.504 INFO:teuthology.orchestra.run.gibba012.stdout: "erasure_code_profile": "", 2021-02-10T11:39:32.504 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_params": { 2021-02-10T11:39:32.504 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "none" 2021-02-10T11:39:32.504 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:39:32.504 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_period": 0, 2021-02-10T11:39:32.504 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_count": 0, 2021-02-10T11:39:32.504 INFO:teuthology.orchestra.run.gibba012.stdout: "use_gmt_hitset": true, 2021-02-10T11:39:32.504 INFO:teuthology.orchestra.run.gibba012.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:39:32.504 INFO:teuthology.orchestra.run.gibba012.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:39:32.504 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:39:32.504 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:39:32.504 INFO:teuthology.orchestra.run.gibba012.stdout: "grade_table": [], 2021-02-10T11:39:32.505 INFO:teuthology.orchestra.run.gibba012.stdout: "stripe_width": 0, 2021-02-10T11:39:32.505 INFO:teuthology.orchestra.run.gibba012.stdout: "expected_num_objects": 0, 2021-02-10T11:39:32.505 INFO:teuthology.orchestra.run.gibba012.stdout: "fast_read": false, 2021-02-10T11:39:32.505 INFO:teuthology.orchestra.run.gibba012.stdout: "options": {}, 2021-02-10T11:39:32.505 INFO:teuthology.orchestra.run.gibba012.stdout: "application_metadata": { 2021-02-10T11:39:32.505 INFO:teuthology.orchestra.run.gibba012.stdout: "cephfs": { 2021-02-10T11:39:32.505 INFO:teuthology.orchestra.run.gibba012.stdout: "data": "cephfs" 2021-02-10T11:39:32.505 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:39:32.505 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:39:32.505 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:39:32.505 INFO:teuthology.orchestra.run.gibba012.stdout: ], 2021-02-10T11:39:32.505 INFO:teuthology.orchestra.run.gibba012.stdout: "osds": [ 2021-02-10T11:39:32.506 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:39:32.506 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 0, 2021-02-10T11:39:32.506 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "95444b8f-6370-4a1d-ae5f-df62651b77da", 2021-02-10T11:39:32.506 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:39:32.506 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:39:32.506 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:39:32.506 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:39:32.506 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:39:32.506 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:39:32.506 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:39:32.506 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 89, 2021-02-10T11:39:32.507 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:39:32.507 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:39:32.507 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:39:32.507 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:39:32.507 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:39:32.507 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:39:32.507 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6817", 2021-02-10T11:39:32.507 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:39:32.507 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:39:32.507 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:39:32.507 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:39:32.507 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6818", 2021-02-10T11:39:32.507 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:39:32.508 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:39:32.508 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:39:32.508 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:39:32.508 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:39:32.508 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:39:32.508 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:39:32.508 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:39:32.508 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6820", 2021-02-10T11:39:32.508 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:39:32.508 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:39:32.508 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:39:32.508 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:39:32.509 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6822", 2021-02-10T11:39:32.509 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:39:32.509 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:39:32.509 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:39:32.509 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:39:32.509 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:39:32.509 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:39:32.509 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:39:32.509 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:39:32.509 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6828", 2021-02-10T11:39:32.509 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:39:32.509 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:39:32.509 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:39:32.510 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:39:32.510 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6830", 2021-02-10T11:39:32.510 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:39:32.510 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:39:32.510 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:39:32.510 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:39:32.510 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:39:32.510 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:39:32.510 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:39:32.510 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:39:32.510 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6824", 2021-02-10T11:39:32.510 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:39:32.510 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:39:32.511 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:39:32.511 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:39:32.511 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6826", 2021-02-10T11:39:32.511 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:39:32.511 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:39:32.511 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:39:32.511 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:39:32.511 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.112:6818/36030", 2021-02-10T11:39:32.511 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.112:6822/36030", 2021-02-10T11:39:32.511 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.112:6830/36030", 2021-02-10T11:39:32.511 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.112:6826/36030", 2021-02-10T11:39:32.511 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:39:32.512 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:39:32.512 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:39:32.512 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:39:32.512 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:39:32.512 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:39:32.512 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 1, 2021-02-10T11:39:32.512 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "a23e217e-0d8e-486e-be07-712adfabd262", 2021-02-10T11:39:32.512 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:39:32.512 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:39:32.512 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:39:32.512 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:39:32.512 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:39:32.512 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:39:32.513 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:39:32.513 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 90, 2021-02-10T11:39:32.513 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:39:32.513 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:39:32.513 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:39:32.513 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:39:32.513 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:39:32.513 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:39:32.513 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6800", 2021-02-10T11:39:32.513 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:39:32.514 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:39:32.514 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:39:32.514 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:39:32.514 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6802", 2021-02-10T11:39:32.514 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:39:32.514 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:39:32.514 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:39:32.514 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:39:32.514 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:39:32.514 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:39:32.514 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:39:32.515 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:39:32.515 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6804", 2021-02-10T11:39:32.515 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:39:32.515 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:39:32.515 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:39:32.515 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:39:32.515 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6806", 2021-02-10T11:39:32.515 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:39:32.515 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:39:32.516 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:39:32.516 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:39:32.516 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:39:32.516 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:39:32.516 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:39:32.516 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:39:32.516 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6812", 2021-02-10T11:39:32.516 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:39:32.516 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:39:32.516 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:39:32.517 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:39:32.517 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6814", 2021-02-10T11:39:32.517 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:39:32.517 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:39:32.517 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:39:32.517 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:39:32.517 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:39:32.517 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:39:32.517 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:39:32.517 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:39:32.517 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6808", 2021-02-10T11:39:32.517 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:39:32.518 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:39:32.518 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:39:32.518 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:39:32.518 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6810", 2021-02-10T11:39:32.518 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:39:32.518 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:39:32.518 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:39:32.518 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:39:32.518 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.112:6802/36029", 2021-02-10T11:39:32.518 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.112:6806/36029", 2021-02-10T11:39:32.518 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.112:6814/36029", 2021-02-10T11:39:32.518 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.112:6810/36029", 2021-02-10T11:39:32.518 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:39:32.519 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:39:32.519 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:39:32.519 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:39:32.519 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:39:32.519 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:39:32.519 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 2, 2021-02-10T11:39:32.519 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "026afa8a-c0f1-4b35-9a76-53c00fe0c2fc", 2021-02-10T11:39:32.519 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:39:32.519 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:39:32.519 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:39:32.519 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:39:32.519 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:39:32.519 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:39:32.520 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:39:32.520 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 90, 2021-02-10T11:39:32.520 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:39:32.520 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:39:32.520 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:39:32.520 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:39:32.520 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:39:32.520 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:39:32.520 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6801", 2021-02-10T11:39:32.520 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:39:32.520 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:39:32.520 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:39:32.520 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:39:32.520 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6803", 2021-02-10T11:39:32.521 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:39:32.521 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:39:32.521 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:39:32.521 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:39:32.521 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:39:32.521 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:39:32.521 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:39:32.521 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:39:32.521 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6805", 2021-02-10T11:39:32.521 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:39:32.521 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:39:32.521 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:39:32.521 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:39:32.522 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6807", 2021-02-10T11:39:32.522 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:39:32.522 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:39:32.522 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:39:32.522 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:39:32.522 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:39:32.522 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:39:32.522 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:39:32.522 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:39:32.522 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6813", 2021-02-10T11:39:32.522 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:39:32.522 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:39:32.523 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:39:32.523 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:39:32.523 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6815", 2021-02-10T11:39:32.523 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:39:32.523 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:39:32.523 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:39:32.523 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:39:32.523 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:39:32.523 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:39:32.523 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:39:32.523 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:39:32.523 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6809", 2021-02-10T11:39:32.524 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:39:32.524 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:39:32.524 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:39:32.524 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:39:32.524 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6811", 2021-02-10T11:39:32.524 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:39:32.524 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:39:32.524 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:39:32.524 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:39:32.524 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.112:6803/36031", 2021-02-10T11:39:32.524 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.112:6807/36031", 2021-02-10T11:39:32.524 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.112:6815/36031", 2021-02-10T11:39:32.524 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.112:6811/36031", 2021-02-10T11:39:32.525 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:39:32.525 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:39:32.525 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:39:32.525 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:39:32.525 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:39:32.525 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:39:32.525 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 3, 2021-02-10T11:39:32.525 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "58795682-e00d-4173-bb84-f7075ca89a11", 2021-02-10T11:39:32.525 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:39:32.525 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:39:32.525 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:39:32.525 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:39:32.525 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:39:32.526 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:39:32.526 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:39:32.526 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 90, 2021-02-10T11:39:32.526 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:39:32.526 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:39:32.526 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:39:32.526 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:39:32.526 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:39:32.526 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:39:32.526 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6816", 2021-02-10T11:39:32.526 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:39:32.527 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:39:32.527 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:39:32.527 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:39:32.527 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6819", 2021-02-10T11:39:32.527 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:39:32.527 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:39:32.527 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:39:32.527 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:39:32.527 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:39:32.527 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:39:32.527 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:39:32.527 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:39:32.528 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6821", 2021-02-10T11:39:32.528 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:39:32.528 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:39:32.528 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:39:32.528 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:39:32.528 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6823", 2021-02-10T11:39:32.528 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:39:32.528 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:39:32.528 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:39:32.528 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:39:32.528 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:39:32.528 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:39:32.529 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:39:32.529 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:39:32.529 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6829", 2021-02-10T11:39:32.529 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:39:32.529 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:39:32.529 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:39:32.529 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:39:32.529 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6831", 2021-02-10T11:39:32.529 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:39:32.529 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:39:32.529 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:39:32.529 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:39:32.530 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:39:32.530 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:39:32.530 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:39:32.530 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:39:32.530 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6825", 2021-02-10T11:39:32.530 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:39:32.530 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:39:32.530 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:39:32.530 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:39:32.530 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6827", 2021-02-10T11:39:32.530 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:39:32.530 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:39:32.531 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:39:32.531 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:39:32.531 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.112:6819/36028", 2021-02-10T11:39:32.531 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.112:6823/36028", 2021-02-10T11:39:32.531 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.112:6831/36028", 2021-02-10T11:39:32.531 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.112:6827/36028", 2021-02-10T11:39:32.531 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:39:32.531 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:39:32.531 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:39:32.531 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:39:32.531 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:39:32.531 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:39:32.531 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 4, 2021-02-10T11:39:32.531 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "6763f8d6-53dc-4e45-bdf6-a6eabcf81e39", 2021-02-10T11:39:32.532 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:39:32.532 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:39:32.532 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:39:32.532 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:39:32.532 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:39:32.532 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:39:32.532 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:39:32.532 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 90, 2021-02-10T11:39:32.532 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:39:32.532 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:39:32.532 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:39:32.532 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:39:32.532 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:39:32.533 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:39:32.533 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6810", 2021-02-10T11:39:32.533 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:39:32.533 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:39:32.533 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:39:32.533 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:39:32.533 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6813", 2021-02-10T11:39:32.533 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:39:32.533 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:39:32.533 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:39:32.533 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:39:32.533 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:39:32.533 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:39:32.534 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:39:32.534 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:39:32.534 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6816", 2021-02-10T11:39:32.534 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:39:32.534 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:39:32.534 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:39:32.534 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:39:32.534 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6819", 2021-02-10T11:39:32.534 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:39:32.534 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:39:32.534 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:39:32.534 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:39:32.535 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:39:32.535 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:39:32.535 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:39:32.535 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:39:32.535 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6825", 2021-02-10T11:39:32.535 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:39:32.535 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:39:32.535 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:39:32.535 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:39:32.535 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6827", 2021-02-10T11:39:32.535 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:39:32.535 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:39:32.536 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:39:32.536 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:39:32.536 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:39:32.536 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:39:32.536 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:39:32.536 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:39:32.536 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6821", 2021-02-10T11:39:32.536 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:39:32.536 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:39:32.536 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:39:32.536 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:39:32.537 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6823", 2021-02-10T11:39:32.537 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:39:32.537 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:39:32.537 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:39:32.537 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:39:32.537 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.120:6813/36388", 2021-02-10T11:39:32.537 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.120:6819/36388", 2021-02-10T11:39:32.537 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.120:6827/36388", 2021-02-10T11:39:32.537 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.120:6823/36388", 2021-02-10T11:39:32.537 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:39:32.537 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:39:32.538 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:39:32.538 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:39:32.538 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:39:32.538 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:39:32.538 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 5, 2021-02-10T11:39:32.538 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "5d0e4dd6-0cb0-4bf8-8314-626acaa1b544", 2021-02-10T11:39:32.538 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:39:32.538 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:39:32.538 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:39:32.538 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:39:32.538 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:39:32.538 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:39:32.539 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:39:32.539 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 90, 2021-02-10T11:39:32.539 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:39:32.539 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:39:32.539 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:39:32.539 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:39:32.539 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:39:32.539 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:39:32.539 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6804", 2021-02-10T11:39:32.539 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:39:32.539 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:39:32.539 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:39:32.539 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:39:32.540 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6806", 2021-02-10T11:39:32.540 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:39:32.540 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:39:32.540 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:39:32.540 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:39:32.540 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:39:32.540 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:39:32.540 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:39:32.540 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:39:32.540 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6809", 2021-02-10T11:39:32.540 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:39:32.540 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:39:32.541 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:39:32.541 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:39:32.541 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6812", 2021-02-10T11:39:32.541 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:39:32.541 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:39:32.541 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:39:32.541 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:39:32.541 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:39:32.541 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:39:32.541 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:39:32.541 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:39:32.541 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6820", 2021-02-10T11:39:32.542 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:39:32.542 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:39:32.542 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:39:32.542 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:39:32.542 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6822", 2021-02-10T11:39:32.542 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:39:32.542 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:39:32.542 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:39:32.542 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:39:32.542 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:39:32.542 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:39:32.543 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:39:32.543 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:39:32.543 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6815", 2021-02-10T11:39:32.543 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:39:32.543 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:39:32.543 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:39:32.543 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:39:32.543 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6818", 2021-02-10T11:39:32.543 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:39:32.543 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:39:32.543 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:39:32.543 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:39:32.543 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.120:6806/36387", 2021-02-10T11:39:32.543 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.120:6812/36387", 2021-02-10T11:39:32.544 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.120:6822/36387", 2021-02-10T11:39:32.544 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.120:6818/36387", 2021-02-10T11:39:32.544 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:39:32.544 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:39:32.544 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:39:32.544 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:39:32.544 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:39:32.544 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:39:32.544 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 6, 2021-02-10T11:39:32.544 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "b010e187-4541-4b00-80dd-f948f6ab8f15", 2021-02-10T11:39:32.544 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:39:32.544 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:39:32.545 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:39:32.545 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:39:32.545 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:39:32.545 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:39:32.545 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:39:32.545 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 81, 2021-02-10T11:39:32.545 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:39:32.545 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:39:32.545 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:39:32.545 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:39:32.545 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:39:32.545 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:39:32.546 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6824", 2021-02-10T11:39:32.546 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:39:32.546 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:39:32.546 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:39:32.546 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:39:32.546 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6826", 2021-02-10T11:39:32.546 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:39:32.546 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:39:32.546 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:39:32.546 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:39:32.546 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:39:32.547 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:39:32.547 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:39:32.547 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:39:32.547 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6828", 2021-02-10T11:39:32.547 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:39:32.547 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:39:32.547 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:39:32.547 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:39:32.547 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6829", 2021-02-10T11:39:32.547 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:39:32.547 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:39:32.547 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:39:32.548 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:39:32.548 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:39:32.548 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:39:32.548 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:39:32.548 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:39:32.548 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6832", 2021-02-10T11:39:32.548 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:39:32.548 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:39:32.548 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:39:32.548 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:39:32.548 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6833", 2021-02-10T11:39:32.549 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:39:32.549 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:39:32.549 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:39:32.549 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:39:32.549 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:39:32.549 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:39:32.549 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:39:32.549 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:39:32.549 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6830", 2021-02-10T11:39:32.549 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:39:32.549 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:39:32.549 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:39:32.549 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:39:32.550 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6831", 2021-02-10T11:39:32.550 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:39:32.550 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:39:32.550 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:39:32.550 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:39:32.550 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.120:6826/36385", 2021-02-10T11:39:32.550 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.120:6829/36385", 2021-02-10T11:39:32.550 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.120:6833/36385", 2021-02-10T11:39:32.550 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.120:6831/36385", 2021-02-10T11:39:32.550 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:39:32.550 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:39:32.550 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:39:32.551 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:39:32.551 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:39:32.551 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:39:32.551 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 7, 2021-02-10T11:39:32.551 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "671e2772-33b7-497e-8e44-2178bd850bcf", 2021-02-10T11:39:32.551 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:39:32.551 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:39:32.551 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:39:32.551 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:39:32.551 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:39:32.551 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:39:32.552 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:39:32.552 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 90, 2021-02-10T11:39:32.552 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:39:32.552 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:39:32.552 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:39:32.552 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:39:32.552 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:39:32.552 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:39:32.552 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6802", 2021-02-10T11:39:32.552 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:39:32.552 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:39:32.552 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:39:32.552 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:39:32.553 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6803", 2021-02-10T11:39:32.553 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:39:32.553 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:39:32.553 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:39:32.553 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:39:32.553 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:39:32.553 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:39:32.553 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:39:32.553 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:39:32.553 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6805", 2021-02-10T11:39:32.553 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:39:32.553 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:39:32.554 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:39:32.554 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:39:32.554 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6807", 2021-02-10T11:39:32.554 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:39:32.554 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:39:32.554 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:39:32.554 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:39:32.554 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:39:32.554 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:39:32.554 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:39:32.554 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:39:32.554 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6814", 2021-02-10T11:39:32.555 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:39:32.555 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:39:32.555 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:39:32.555 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:39:32.555 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6817", 2021-02-10T11:39:32.555 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:39:32.555 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:39:32.555 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:39:32.555 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:39:32.555 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:39:32.555 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:39:32.555 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:39:32.555 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:39:32.556 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6808", 2021-02-10T11:39:32.556 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:39:32.556 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:39:32.556 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:39:32.556 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:39:32.556 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6811", 2021-02-10T11:39:32.556 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:39:32.556 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:39:32.556 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:39:32.556 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:39:32.556 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.120:6803/36386", 2021-02-10T11:39:32.556 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.120:6807/36386", 2021-02-10T11:39:32.556 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.120:6817/36386", 2021-02-10T11:39:32.557 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.120:6811/36386", 2021-02-10T11:39:32.557 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:39:32.557 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:39:32.557 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:39:32.557 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:39:32.557 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:39:32.557 INFO:teuthology.orchestra.run.gibba012.stdout: ], 2021-02-10T11:39:32.557 INFO:teuthology.orchestra.run.gibba012.stdout: "osd_xinfo": [ 2021-02-10T11:39:32.557 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:39:32.557 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 0, 2021-02-10T11:39:32.557 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:39:32.557 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:39:32.558 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:39:32.558 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:39:32.558 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:39:32.558 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:39:32.558 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:39:32.558 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 1, 2021-02-10T11:39:32.558 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:39:32.558 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:39:32.558 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:39:32.558 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:39:32.558 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:39:32.558 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:39:32.558 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:39:32.559 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 2, 2021-02-10T11:39:32.559 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:39:32.559 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:39:32.559 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:39:32.559 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:39:32.559 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:39:32.559 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:39:32.559 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:39:32.559 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 3, 2021-02-10T11:39:32.559 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:39:32.559 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:39:32.559 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:39:32.559 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:39:32.560 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:39:32.560 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:39:32.560 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:39:32.560 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 4, 2021-02-10T11:39:32.560 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:39:32.560 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:39:32.560 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:39:32.560 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:39:32.560 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:39:32.560 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:39:32.560 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:39:32.560 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 5, 2021-02-10T11:39:32.561 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:39:32.561 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:39:32.561 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:39:32.561 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:39:32.561 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:39:32.561 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:39:32.561 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:39:32.561 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 6, 2021-02-10T11:39:32.561 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:39:32.561 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:39:32.561 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:39:32.561 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:39:32.562 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:39:32.562 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:39:32.562 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:39:32.562 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 7, 2021-02-10T11:39:32.562 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:39:32.562 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:39:32.562 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:39:32.562 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:39:32.562 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:39:32.562 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:39:32.562 INFO:teuthology.orchestra.run.gibba012.stdout: ], 2021-02-10T11:39:32.563 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_upmap": [], 2021-02-10T11:39:32.563 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_upmap_items": [], 2021-02-10T11:39:32.563 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_temp": [], 2021-02-10T11:39:32.563 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_temp": [], 2021-02-10T11:39:32.563 INFO:teuthology.orchestra.run.gibba012.stdout: "blacklist": {}, 2021-02-10T11:39:32.563 INFO:teuthology.orchestra.run.gibba012.stdout: "erasure_code_profiles": { 2021-02-10T11:39:32.563 INFO:teuthology.orchestra.run.gibba012.stdout: "default": { 2021-02-10T11:39:32.563 INFO:teuthology.orchestra.run.gibba012.stdout: "crush-failure-domain": "osd", 2021-02-10T11:39:32.563 INFO:teuthology.orchestra.run.gibba012.stdout: "k": "2", 2021-02-10T11:39:32.563 INFO:teuthology.orchestra.run.gibba012.stdout: "m": "1", 2021-02-10T11:39:32.563 INFO:teuthology.orchestra.run.gibba012.stdout: "plugin": "jerasure", 2021-02-10T11:39:32.563 INFO:teuthology.orchestra.run.gibba012.stdout: "ruleset-failure-domain": "osd", 2021-02-10T11:39:32.563 INFO:teuthology.orchestra.run.gibba012.stdout: "technique": "reed_sol_van" 2021-02-10T11:39:32.563 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:39:32.564 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:39:32.564 INFO:teuthology.orchestra.run.gibba012.stdout: "removed_snaps_queue": [], 2021-02-10T11:39:32.564 INFO:teuthology.orchestra.run.gibba012.stdout: "new_removed_snaps": [], 2021-02-10T11:39:32.564 INFO:teuthology.orchestra.run.gibba012.stdout: "new_purged_snaps": [], 2021-02-10T11:39:32.564 INFO:teuthology.orchestra.run.gibba012.stdout: "crush_node_flags": {}, 2021-02-10T11:39:32.564 INFO:teuthology.orchestra.run.gibba012.stdout: "device_class_flags": {} 2021-02-10T11:39:32.564 INFO:teuthology.orchestra.run.gibba012.stdout:} 2021-02-10T11:39:32.565 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.492 7ff06204f700 1 -- 172.21.2.112:0/1758467428 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff0400186c0 msgr2=0x7ff04001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:32.565 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.492 7ff06204f700 1 --2- 172.21.2.112:0/1758467428 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff0400186c0 0x7ff04001abc0 crc :-1 s=READY pgs=900 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:32.565 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.492 7ff06204f700 1 -- 172.21.2.112:0/1758467428 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff05c0baf60 msgr2=0x7ff05c0b3070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:32.565 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.492 7ff06204f700 1 --2- 172.21.2.112:0/1758467428 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff05c0baf60 0x7ff05c0b3070 secure :-1 s=READY pgs=1923 cs=0 l=1 rev1=1 rx=0x7ff04c009670 tx=0x7ff04c00b960).stop 2021-02-10T11:39:32.566 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.492 7ff06204f700 1 -- 172.21.2.112:0/1758467428 shutdown_connections 2021-02-10T11:39:32.566 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.492 7ff06204f700 1 --2- 172.21.2.112:0/1758467428 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff0400186c0 0x7ff04001abc0 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:32.566 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.492 7ff06204f700 1 --2- 172.21.2.112:0/1758467428 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff05c0b3690 0x7ff05c0b8400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:32.566 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.492 7ff06204f700 1 --2- 172.21.2.112:0/1758467428 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff05c0b8a20 0x7ff05c0b6690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:32.566 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.492 7ff06204f700 1 --2- 172.21.2.112:0/1758467428 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff05c0baf60 0x7ff05c0b3070 unknown :-1 s=CLOSED pgs=1923 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:32.566 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.492 7ff06204f700 1 -- 172.21.2.112:0/1758467428 >> 172.21.2.112:0/1758467428 conn(0x7ff05c09ca60 msgr2=0x7ff05c0bbbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:32.566 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.492 7ff06204f700 1 -- 172.21.2.112:0/1758467428 shutdown_connections 2021-02-10T11:39:32.566 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.492 7ff06204f700 1 -- 172.21.2.112:0/1758467428 wait complete. 2021-02-10T11:39:32.567 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:39:32.996 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.985 7f8bdbbab700 1 Processor -- start 2021-02-10T11:39:32.996 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.985 7f8bdbbab700 1 -- start start 2021-02-10T11:39:32.996 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.985 7f8bdbbab700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8bd40d1c70 0x7f8bd40f20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:32.996 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.985 7f8bdbbab700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bd40f0b10 con 0x7f8bd40d0340 2021-02-10T11:39:32.996 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.985 7f8bdbbab700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bd41b0be0 con 0x7f8bd40d1870 2021-02-10T11:39:32.997 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.985 7f8bdbbab700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bd40e5ed0 con 0x7f8bd40d1c70 2021-02-10T11:39:32.997 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.985 7f8bd91c6700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f8bd40d0340 0x7f8bd40ef640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:53268/0 (socket says 172.21.2.112:53268) 2021-02-10T11:39:32.997 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.985 7f8bd91c6700 1 -- 172.21.2.112:0/3773474355 learned_addr learned my addr 172.21.2.112:0/3773474355 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:39:32.997 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.986 7f8bda1c8700 1 --2- 172.21.2.112:0/3773474355 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8bd40d1c70 0x7f8bd40f20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:32.997 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.986 7f8bd89c5700 1 -- 172.21.2.112:0/3773474355 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1208725286 0 0) 0x7f8bc80010b0 con 0x7f8bd40d0340 2021-02-10T11:39:32.997 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.986 7f8bd89c5700 1 -- 172.21.2.112:0/3773474355 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8bc0002fe0 con 0x7f8bd40d0340 2021-02-10T11:39:32.997 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.986 7f8bd89c5700 1 -- 172.21.2.112:0/3773474355 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3601784994 0 0) 0x7f8bd0001080 con 0x7f8bd40d1870 2021-02-10T11:39:32.997 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.986 7f8bd89c5700 1 -- 172.21.2.112:0/3773474355 --> v1:172.21.2.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8bc0003210 con 0x7f8bd40d1870 2021-02-10T11:39:32.997 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.986 7f8bd89c5700 1 -- 172.21.2.112:0/3773474355 <== mon.1 v1:172.21.2.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2153056659 0 0) 0x7f8bd0001010 con 0x7f8bd40d1870 2021-02-10T11:39:32.997 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.986 7f8bd89c5700 1 -- 172.21.2.112:0/3773474355 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8bd40d1c70 msgr2=0x7f8bd40f20a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:32.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.986 7f8bd89c5700 1 --2- 172.21.2.112:0/3773474355 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8bd40d1c70 0x7f8bd40f20a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:32.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.986 7f8bd89c5700 1 -- 172.21.2.112:0/3773474355 >> v1:172.21.2.112:6789/0 conn(0x7f8bd40d0340 legacy=0x7f8bd40ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:32.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.986 7f8bd89c5700 1 -- 172.21.2.112:0/3773474355 --> v1:172.21.2.120:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bd40efe30 con 0x7f8bd40d1870 2021-02-10T11:39:32.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.986 7f8bd89c5700 1 -- 172.21.2.112:0/3773474355 <== mon.1 v1:172.21.2.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f8bd0001450 con 0x7f8bd40d1870 2021-02-10T11:39:32.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.986 7f8bd89c5700 1 -- 172.21.2.112:0/3773474355 >> v1:172.21.2.120:6789/0 conn(0x7f8bd40d1870 legacy=0x7f8bd41b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:32.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.986 7f8bd89c5700 1 --2- 172.21.2.112:0/3773474355 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8bc0003ba0 0x7f8bc0005fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:32.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.986 7f8bd89c5700 1 --2- 172.21.2.112:0/3773474355 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8bc0006830 0x7f8bc0008d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:32.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.986 7f8bd89c5700 1 --2- 172.21.2.112:0/3773474355 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8bc0009650 0x7f8bc000bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:32.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.986 7f8bd89c5700 1 -- 172.21.2.112:0/3773474355 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bc000c390 con 0x7f8bc0003ba0 2021-02-10T11:39:32.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.986 7f8bd89c5700 1 -- 172.21.2.112:0/3773474355 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bc0050580 con 0x7f8bc0006830 2021-02-10T11:39:32.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.986 7f8bd89c5700 1 -- 172.21.2.112:0/3773474355 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bc00507a0 con 0x7f8bc0009650 2021-02-10T11:39:32.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.986 7f8bda1c8700 1 --2- 172.21.2.112:0/3773474355 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8bd40d1c70 0x7f8bd40f20a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:39:32.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.986 7f8bd91c6700 1 --2- 172.21.2.112:0/3773474355 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8bc0006830 0x7f8bc0008d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:32.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.986 7f8bd99c7700 1 --2- 172.21.2.112:0/3773474355 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8bc0003ba0 0x7f8bc0005fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:32.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.986 7f8bda1c8700 1 --2- 172.21.2.112:0/3773474355 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8bc0009650 0x7f8bc000bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:32.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.986 7f8bd99c7700 1 -- 172.21.2.112:0/3773474355 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8bc0009650 msgr2=0x7f8bc000bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:32.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.986 7f8bd99c7700 1 --2- 172.21.2.112:0/3773474355 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8bc0009650 0x7f8bc000bb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:32.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.986 7f8bd99c7700 1 -- 172.21.2.112:0/3773474355 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8bc0006830 msgr2=0x7f8bc0008d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:32.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.986 7f8bd99c7700 1 --2- 172.21.2.112:0/3773474355 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8bc0006830 0x7f8bc0008d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:32.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.987 7f8bd99c7700 1 -- 172.21.2.112:0/3773474355 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bc0050a40 con 0x7f8bc0003ba0 2021-02-10T11:39:32.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.987 7f8bd99c7700 1 --2- 172.21.2.112:0/3773474355 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8bc0003ba0 0x7f8bc0005fb0 secure :-1 s=READY pgs=1925 cs=0 l=1 rev1=1 rx=0x7f8bd0001410 tx=0x7f8bd0009ca0).ready entity=mon.0 client_cookie=128daacbdb5a17a7 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:32.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.987 7f8bd89c5700 1 -- 172.21.2.112:0/3773474355 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8bd0009de0 con 0x7f8bc0003ba0 2021-02-10T11:39:33.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.987 7f8bd89c5700 1 -- 172.21.2.112:0/3773474355 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bd0003030 con 0x7f8bc0003ba0 2021-02-10T11:39:33.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.987 7f8bd89c5700 1 -- 172.21.2.112:0/3773474355 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8bd0001650 con 0x7f8bc0003ba0 2021-02-10T11:39:33.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.987 7f8bdbbab700 1 -- 172.21.2.112:0/3773474355 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8bc0003ba0 msgr2=0x7f8bc0005fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:33.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.987 7f8bdbbab700 1 --2- 172.21.2.112:0/3773474355 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8bc0003ba0 0x7f8bc0005fb0 secure :-1 s=READY pgs=1925 cs=0 l=1 rev1=1 rx=0x7f8bd0001410 tx=0x7f8bd0009ca0).stop 2021-02-10T11:39:33.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.987 7f8bda1c8700 1 -- 172.21.2.112:0/3773474355 reap_dead start 2021-02-10T11:39:33.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.988 7f8bdbbab700 1 -- 172.21.2.112:0/3773474355 shutdown_connections 2021-02-10T11:39:33.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.988 7f8bdbbab700 1 -- 172.21.2.112:0/3773474355 >> 172.21.2.112:0/3773474355 conn(0x7f8bd40ce450 msgr2=0x7f8bd40d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:33.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.988 7f8bdbbab700 1 -- 172.21.2.112:0/3773474355 shutdown_connections 2021-02-10T11:39:33.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:32.988 7f8bdbbab700 1 -- 172.21.2.112:0/3773474355 wait complete. 2021-02-10T11:39:33.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.002 7f8bdbbab700 1 Processor -- start 2021-02-10T11:39:33.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.002 7f8bdbbab700 1 -- start start 2021-02-10T11:39:33.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.003 7f8bdbbab700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8bd4195650 0x7f8bd40b5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:33.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.003 7f8bdbbab700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8bd40b5b80 0x7f8bd41a2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:33.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.003 7f8bdbbab700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8bd40acfa0 0x7f8bd41a3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:33.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.003 7f8bdbbab700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bd40b6010 con 0x7f8bd40acfa0 2021-02-10T11:39:33.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.003 7f8bdbbab700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bd41a3a60 con 0x7f8bd40b5b80 2021-02-10T11:39:33.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.003 7f8bdbbab700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bd41a3c40 con 0x7f8bd4195650 2021-02-10T11:39:33.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.003 7f8bda1c8700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8bd40acfa0 0x7f8bd41a3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:33.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.003 7f8bda1c8700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8bd40acfa0 0x7f8bd41a3440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:53818/0 (socket says 172.21.2.112:53818) 2021-02-10T11:39:33.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.003 7f8bda1c8700 1 -- 172.21.2.112:0/1218715353 learned_addr learned my addr 172.21.2.112:0/1218715353 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:39:33.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.003 7f8bda1c8700 1 -- 172.21.2.112:0/1218715353 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8bd4195650 msgr2=0x7f8bd40b5560 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:39:33.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.003 7f8bda1c8700 1 --2- 172.21.2.112:0/1218715353 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8bd4195650 0x7f8bd40b5560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:33.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.003 7f8bda1c8700 1 -- 172.21.2.112:0/1218715353 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8bd40b5b80 msgr2=0x7f8bd41a2e20 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:39:33.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.003 7f8bda1c8700 1 --2- 172.21.2.112:0/1218715353 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8bd40b5b80 0x7f8bd41a2e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:33.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.003 7f8bda1c8700 1 -- 172.21.2.112:0/1218715353 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bd40bcae0 con 0x7f8bd40acfa0 2021-02-10T11:39:33.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.003 7f8bda1c8700 1 --2- 172.21.2.112:0/1218715353 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8bd40acfa0 0x7f8bd41a3440 secure :-1 s=READY pgs=1926 cs=0 l=1 rev1=1 rx=0x7f8bcc0099c0 tx=0x7f8bcc00bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:33.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.004 7f8bc67fc700 1 -- 172.21.2.112:0/1218715353 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8bcc0011f0 con 0x7f8bd40acfa0 2021-02-10T11:39:33.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.004 7f8bc67fc700 1 -- 172.21.2.112:0/1218715353 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bcc003030 con 0x7f8bd40acfa0 2021-02-10T11:39:33.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.004 7f8bc67fc700 1 -- 172.21.2.112:0/1218715353 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8bcc001720 con 0x7f8bd40acfa0 2021-02-10T11:39:33.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.004 7f8bdbbab700 1 -- 172.21.2.112:0/1218715353 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bd40bcae0 con 0x7f8bd40acfa0 2021-02-10T11:39:33.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.004 7f8bdbbab700 1 -- 172.21.2.112:0/1218715353 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bd40bcae0 con 0x7f8bd40acfa0 2021-02-10T11:39:33.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.004 7f8bc67fc700 1 -- 172.21.2.112:0/1218715353 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8bcc006a90 con 0x7f8bd40acfa0 2021-02-10T11:39:33.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.005 7f8bc67fc700 1 --2- 172.21.2.112:0/1218715353 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8bbc0186c0 0x7f8bbc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:33.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.005 7f8bc67fc700 1 -- 172.21.2.112:0/1218715353 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f8bcc032900 con 0x7f8bd40acfa0 2021-02-10T11:39:33.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.005 7f8bd99c7700 1 --2- 172.21.2.112:0/1218715353 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8bbc0186c0 0x7f8bbc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:33.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.006 7f8bd99c7700 1 --2- 172.21.2.112:0/1218715353 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8bbc0186c0 0x7f8bbc01abc0 crc :-1 s=READY pgs=901 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:33.028 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.035 7f8bdbbab700 1 -- 172.21.2.112:0/1218715353 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bd40b6010 con 0x7f8bd40acfa0 2021-02-10T11:39:33.031 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.038 7f8bc67fc700 1 -- 172.21.2.112:0/1218715353 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8bcc002f30 con 0x7f8bd40acfa0 2021-02-10T11:39:33.161 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.168 7f8bdbbab700 1 -- 172.21.2.112:0/1218715353 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8bd40b6010 con 0x7f8bd40acfa0 2021-02-10T11:39:33.161 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.168 7f8bc67fc700 1 -- 172.21.2.112:0/1218715353 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+2101 (secure 0 0 0) 0x7f8bcc002f30 con 0x7f8bd40acfa0 2021-02-10T11:39:33.162 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:39:33.163 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":73,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9621,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/1174368781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":1174368781},{"type":"v1","addr":"172.21.2.112:6833","nonce":1174368781}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":69}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:38:51.410458","modified":"2021-02-10 11:38:53.445268","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6380},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6380":{"gid":6380,"name":"b","rank":0,"incarnation":70,"state":"up:active","state_seq":3,"addr":"172.21.2.120:6835/2402714407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":2402714407},{"type":"v1","addr":"172.21.2.120:6835","nonce":2402714407}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":9}]} 2021-02-10T11:39:33.163 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 73 2021-02-10T11:39:33.164 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.170 7f8bdbbab700 1 -- 172.21.2.112:0/1218715353 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8bbc0186c0 msgr2=0x7f8bbc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:33.164 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.170 7f8bdbbab700 1 --2- 172.21.2.112:0/1218715353 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8bbc0186c0 0x7f8bbc01abc0 crc :-1 s=READY pgs=901 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:33.164 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.170 7f8bdbbab700 1 -- 172.21.2.112:0/1218715353 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8bd40acfa0 msgr2=0x7f8bd41a3440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:33.164 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.170 7f8bdbbab700 1 --2- 172.21.2.112:0/1218715353 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8bd40acfa0 0x7f8bd41a3440 secure :-1 s=READY pgs=1926 cs=0 l=1 rev1=1 rx=0x7f8bcc0099c0 tx=0x7f8bcc00bcf0).stop 2021-02-10T11:39:33.164 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.170 7f8bdbbab700 1 -- 172.21.2.112:0/1218715353 shutdown_connections 2021-02-10T11:39:33.164 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.170 7f8bdbbab700 1 --2- 172.21.2.112:0/1218715353 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8bbc0186c0 0x7f8bbc01abc0 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:33.165 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.170 7f8bdbbab700 1 --2- 172.21.2.112:0/1218715353 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8bd40b5b80 0x7f8bd41a2e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:33.165 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.170 7f8bdbbab700 1 --2- 172.21.2.112:0/1218715353 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8bd40acfa0 0x7f8bd41a3440 unknown :-1 s=CLOSED pgs=1926 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:33.165 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.170 7f8bdbbab700 1 --2- 172.21.2.112:0/1218715353 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8bd4195650 0x7f8bd40b5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:33.165 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.170 7f8bdbbab700 1 -- 172.21.2.112:0/1218715353 >> 172.21.2.112:0/1218715353 conn(0x7f8bd40b3040 msgr2=0x7f8bd40c3250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:33.165 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.170 7f8bdbbab700 1 -- 172.21.2.112:0/1218715353 shutdown_connections 2021-02-10T11:39:33.165 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.170 7f8bdbbab700 1 -- 172.21.2.112:0/1218715353 wait complete. 2021-02-10T11:39:33.172 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs fail cephfs 2021-02-10T11:39:33.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.588 7f1e773d5700 1 Processor -- start 2021-02-10T11:39:33.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.588 7f1e773d5700 1 -- start start 2021-02-10T11:39:33.600 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.588 7f1e773d5700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1e700d0340 0x7f1e700ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:33.600 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.588 7f1e773d5700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f1e700d1c70 0x7f1e700efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:33.600 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.588 7f1e773d5700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e700f0b10 con 0x7f1e700d1870 2021-02-10T11:39:33.600 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.588 7f1e773d5700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e700e5ed0 con 0x7f1e700d1c70 2021-02-10T11:39:33.600 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.588 7f1e773d5700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e701b0be0 con 0x7f1e700d0340 2021-02-10T11:39:33.600 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.588 7f1e751f1700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f1e700d1870 0x7f1e701b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:53288/0 (socket says 172.21.2.112:53288) 2021-02-10T11:39:33.600 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.588 7f1e751f1700 1 -- 172.21.2.112:0/2570259059 learned_addr learned my addr 172.21.2.112:0/2570259059 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:39:33.600 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.589 7f1e63fff700 1 -- 172.21.2.112:0/2570259059 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3328142307 0 0) 0x7f1e6c0010b0 con 0x7f1e700d1870 2021-02-10T11:39:33.600 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.589 7f1e63fff700 1 -- 172.21.2.112:0/2570259059 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e5c002fe0 con 0x7f1e700d1870 2021-02-10T11:39:33.600 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.589 7f1e749f0700 1 --2- 172.21.2.112:0/2570259059 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1e700d0340 0x7f1e700ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:33.600 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.589 7f1e759f2700 1 --2- 172.21.2.112:0/2570259059 >> v2:172.21.2.120:3300/0 conn(0x7f1e700d1c70 0x7f1e700efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:33.600 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.589 7f1e63fff700 1 -- 172.21.2.112:0/2570259059 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3750993591 0 0) 0x7f1e6c0010b0 con 0x7f1e700d1870 2021-02-10T11:39:33.600 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.589 7f1e63fff700 1 -- 172.21.2.112:0/2570259059 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1e700d0340 msgr2=0x7f1e700ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:33.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.589 7f1e63fff700 1 --2- 172.21.2.112:0/2570259059 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1e700d0340 0x7f1e700ef640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:33.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.589 7f1e63fff700 1 -- 172.21.2.112:0/2570259059 >> v2:172.21.2.120:3300/0 conn(0x7f1e700d1c70 msgr2=0x7f1e700efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:33.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.589 7f1e63fff700 1 --2- 172.21.2.112:0/2570259059 >> v2:172.21.2.120:3300/0 conn(0x7f1e700d1c70 0x7f1e700efc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:33.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.589 7f1e63fff700 1 -- 172.21.2.112:0/2570259059 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e700f20a0 con 0x7f1e700d1870 2021-02-10T11:39:33.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.589 7f1e63fff700 1 -- 172.21.2.112:0/2570259059 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f1e6c0015b0 con 0x7f1e700d1870 2021-02-10T11:39:33.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.589 7f1e759f2700 1 --2- 172.21.2.112:0/2570259059 >> v2:172.21.2.120:3300/0 conn(0x7f1e700d1c70 0x7f1e700efc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:39:33.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.589 7f1e63fff700 1 -- 172.21.2.112:0/2570259059 >> v1:172.21.2.112:6789/0 conn(0x7f1e700d1870 legacy=0x7f1e701b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:33.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.589 7f1e63fff700 1 --2- 172.21.2.112:0/2570259059 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1e5c003a60 0x7f1e5c005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:33.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.589 7f1e63fff700 1 --2- 172.21.2.112:0/2570259059 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1e5c006790 0x7f1e5c008c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:33.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.589 7f1e63fff700 1 --2- 172.21.2.112:0/2570259059 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1e5c009500 0x7f1e5c00ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:33.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.589 7f1e63fff700 1 -- 172.21.2.112:0/2570259059 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e5c0502e0 con 0x7f1e5c009500 2021-02-10T11:39:33.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.589 7f1e63fff700 1 -- 172.21.2.112:0/2570259059 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e5c050490 con 0x7f1e5c003a60 2021-02-10T11:39:33.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.590 7f1e63fff700 1 -- 172.21.2.112:0/2570259059 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e5c0506b0 con 0x7f1e5c006790 2021-02-10T11:39:33.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.590 7f1e759f2700 1 --2- 172.21.2.112:0/2570259059 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1e5c009500 0x7f1e5c00ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:33.602 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.590 7f1e751f1700 1 --2- 172.21.2.112:0/2570259059 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1e5c003a60 0x7f1e5c005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:33.602 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.590 7f1e749f0700 1 --2- 172.21.2.112:0/2570259059 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1e5c006790 0x7f1e5c008c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:33.602 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.590 7f1e759f2700 1 -- 172.21.2.112:0/2570259059 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1e5c006790 msgr2=0x7f1e5c008c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:33.602 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.590 7f1e759f2700 1 --2- 172.21.2.112:0/2570259059 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1e5c006790 0x7f1e5c008c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:33.602 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.590 7f1e759f2700 1 -- 172.21.2.112:0/2570259059 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1e5c003a60 msgr2=0x7f1e5c005ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:33.602 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.590 7f1e759f2700 1 --2- 172.21.2.112:0/2570259059 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1e5c003a60 0x7f1e5c005ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:33.602 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.590 7f1e759f2700 1 -- 172.21.2.112:0/2570259059 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e5c050950 con 0x7f1e5c009500 2021-02-10T11:39:33.602 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.590 7f1e759f2700 1 --2- 172.21.2.112:0/2570259059 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1e5c009500 0x7f1e5c00ba20 secure :-1 s=READY pgs=1928 cs=0 l=1 rev1=1 rx=0x7f1e680090f0 tx=0x7f1e6800b300).ready entity=mon.0 client_cookie=a147e47fe8b7fdfa server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:33.602 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.590 7f1e63fff700 1 -- 172.21.2.112:0/2570259059 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1e6800b6f0 con 0x7f1e5c009500 2021-02-10T11:39:33.602 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.590 7f1e63fff700 1 -- 172.21.2.112:0/2570259059 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e68003030 con 0x7f1e5c009500 2021-02-10T11:39:33.602 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.590 7f1e63fff700 1 -- 172.21.2.112:0/2570259059 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1e6800bcb0 con 0x7f1e5c009500 2021-02-10T11:39:33.602 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.590 7f1e773d5700 1 -- 172.21.2.112:0/2570259059 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1e5c009500 msgr2=0x7f1e5c00ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:33.602 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.590 7f1e773d5700 1 --2- 172.21.2.112:0/2570259059 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1e5c009500 0x7f1e5c00ba20 secure :-1 s=READY pgs=1928 cs=0 l=1 rev1=1 rx=0x7f1e680090f0 tx=0x7f1e6800b300).stop 2021-02-10T11:39:33.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.590 7f1e759f2700 1 -- 172.21.2.112:0/2570259059 reap_dead start 2021-02-10T11:39:33.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.591 7f1e773d5700 1 -- 172.21.2.112:0/2570259059 shutdown_connections 2021-02-10T11:39:33.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.591 7f1e773d5700 1 -- 172.21.2.112:0/2570259059 >> 172.21.2.112:0/2570259059 conn(0x7f1e700ce450 msgr2=0x7f1e700d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:33.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.591 7f1e773d5700 1 -- 172.21.2.112:0/2570259059 shutdown_connections 2021-02-10T11:39:33.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.591 7f1e773d5700 1 -- 172.21.2.112:0/2570259059 wait complete. 2021-02-10T11:39:33.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.606 7f1e773d5700 1 Processor -- start 2021-02-10T11:39:33.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.606 7f1e773d5700 1 -- start start 2021-02-10T11:39:33.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.606 7f1e773d5700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1e70097880 0x7f1e700a6ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:33.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.606 7f1e773d5700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1e70098c10 0x7f1e70099010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:33.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.606 7f1e773d5700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1e70099630 0x7f1e70093e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:33.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.606 7f1e773d5700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e70097380 con 0x7f1e70097880 2021-02-10T11:39:33.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.606 7f1e773d5700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e70099a70 con 0x7f1e70098c10 2021-02-10T11:39:33.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.606 7f1e773d5700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e70094470 con 0x7f1e70099630 2021-02-10T11:39:33.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.607 7f1e751f1700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1e70097880 0x7f1e700a6ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:33.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.607 7f1e751f1700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1e70097880 0x7f1e700a6ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:53840/0 (socket says 172.21.2.112:53840) 2021-02-10T11:39:33.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.607 7f1e751f1700 1 -- 172.21.2.112:0/238715708 learned_addr learned my addr 172.21.2.112:0/238715708 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:39:33.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.607 7f1e751f1700 1 -- 172.21.2.112:0/238715708 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1e70099630 msgr2=0x7f1e70093e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:33.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.607 7f1e751f1700 1 --2- 172.21.2.112:0/238715708 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1e70099630 0x7f1e70093e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:33.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.607 7f1e751f1700 1 -- 172.21.2.112:0/238715708 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1e70098c10 msgr2=0x7f1e70099010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:33.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.607 7f1e751f1700 1 --2- 172.21.2.112:0/238715708 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1e70098c10 0x7f1e70099010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:33.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.607 7f1e751f1700 1 -- 172.21.2.112:0/238715708 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e70094690 con 0x7f1e70097880 2021-02-10T11:39:33.606 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.607 7f1e751f1700 1 --2- 172.21.2.112:0/238715708 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1e70097880 0x7f1e700a6ff0 secure :-1 s=READY pgs=1929 cs=0 l=1 rev1=1 rx=0x7f1e6c001660 tx=0x7f1e6c00a930).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:33.606 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.607 7f1e61ffb700 1 -- 172.21.2.112:0/238715708 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1e6c00ad20 con 0x7f1e70097880 2021-02-10T11:39:33.606 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.607 7f1e61ffb700 1 -- 172.21.2.112:0/238715708 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e6c0016a0 con 0x7f1e70097880 2021-02-10T11:39:33.606 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.607 7f1e773d5700 1 -- 172.21.2.112:0/238715708 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e70094690 con 0x7f1e70097880 2021-02-10T11:39:33.606 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.607 7f1e773d5700 1 -- 172.21.2.112:0/238715708 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e70094c20 con 0x7f1e70097880 2021-02-10T11:39:33.606 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.607 7f1e61ffb700 1 -- 172.21.2.112:0/238715708 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1e6c004330 con 0x7f1e70097880 2021-02-10T11:39:33.606 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.608 7f1e61ffb700 1 -- 172.21.2.112:0/238715708 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1e6c004140 con 0x7f1e70097880 2021-02-10T11:39:33.606 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.608 7f1e61ffb700 1 --2- 172.21.2.112:0/238715708 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1e500186c0 0x7f1e5001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:33.606 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.608 7f1e61ffb700 1 -- 172.21.2.112:0/238715708 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f1e6c032540 con 0x7f1e70097880 2021-02-10T11:39:33.606 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.608 7f1e749f0700 1 --2- 172.21.2.112:0/238715708 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1e500186c0 0x7f1e5001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:33.606 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.609 7f1e749f0700 1 --2- 172.21.2.112:0/238715708 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1e500186c0 0x7f1e5001abc0 crc :-1 s=READY pgs=902 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:33.632 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.638 7f1e773d5700 1 -- 172.21.2.112:0/238715708 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e700ebc50 con 0x7f1e70097880 2021-02-10T11:39:33.634 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.641 7f1e61ffb700 1 -- 172.21.2.112:0/238715708 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1e6c0009b0 con 0x7f1e70097880 2021-02-10T11:39:33.763 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:33.769 7f1e773d5700 1 -- 172.21.2.112:0/238715708 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f1e700f0b10 con 0x7f1e70097880 2021-02-10T11:39:34.158 INFO:tasks.ceph.mon.a.gibba012.stderr:2021-02-10 11:39:34.164 7fc3fe6e2700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-02-10T11:39:34.164 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:34.170 7f1e61ffb700 1 -- 172.21.2.112:0/238715708 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v74) v1 ==== 161+0+0 (secure 0 0 0) 0x7f1e6c0044a0 con 0x7f1e70097880 2021-02-10T11:39:34.178 INFO:teuthology.orchestra.run.gibba012.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2021-02-10T11:39:34.180 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:34.187 7f1e773d5700 1 -- 172.21.2.112:0/238715708 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1e500186c0 msgr2=0x7f1e5001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:34.180 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:34.187 7f1e773d5700 1 --2- 172.21.2.112:0/238715708 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1e500186c0 0x7f1e5001abc0 crc :-1 s=READY pgs=902 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:34.181 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:34.187 7f1e773d5700 1 -- 172.21.2.112:0/238715708 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1e70097880 msgr2=0x7f1e700a6ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:34.181 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:34.187 7f1e773d5700 1 --2- 172.21.2.112:0/238715708 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1e70097880 0x7f1e700a6ff0 secure :-1 s=READY pgs=1929 cs=0 l=1 rev1=1 rx=0x7f1e6c001660 tx=0x7f1e6c00a930).stop 2021-02-10T11:39:34.181 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:34.187 7f1e773d5700 1 -- 172.21.2.112:0/238715708 shutdown_connections 2021-02-10T11:39:34.181 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:34.187 7f1e773d5700 1 --2- 172.21.2.112:0/238715708 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1e500186c0 0x7f1e5001abc0 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:34.181 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:34.187 7f1e773d5700 1 --2- 172.21.2.112:0/238715708 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1e70098c10 0x7f1e70099010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:34.182 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:34.187 7f1e773d5700 1 --2- 172.21.2.112:0/238715708 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1e70099630 0x7f1e70093e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:34.182 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:34.187 7f1e773d5700 1 --2- 172.21.2.112:0/238715708 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1e70097880 0x7f1e700a6ff0 unknown :-1 s=CLOSED pgs=1929 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:34.182 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:34.187 7f1e773d5700 1 -- 172.21.2.112:0/238715708 >> 172.21.2.112:0/238715708 conn(0x7f1e700beb40 msgr2=0x7f1e700bef40 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:34.182 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:34.188 7f1e773d5700 1 -- 172.21.2.112:0/238715708 shutdown_connections 2021-02-10T11:39:34.182 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:34.188 7f1e773d5700 1 -- 172.21.2.112:0/238715708 wait complete. 2021-02-10T11:39:34.189 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:39:34.214 INFO:tasks.ceph.mds.b.gibba020.stdout:starting mds.b at 2021-02-10T11:39:34.618 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:34.608 7fdeae08c700 1 Processor -- start 2021-02-10T11:39:34.618 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:34.608 7fdeae08c700 1 -- start start 2021-02-10T11:39:34.618 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:34.609 7fdeae08c700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fdea80d1870 0x7fdea81b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:34.618 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:34.609 7fdeae08c700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdea80f0650 0x7fdea80cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:34.618 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:34.609 7fdeae08c700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdea80f0b10 con 0x7fdea81b0a10 2021-02-10T11:39:34.618 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:34.609 7fdeae08c700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdea80e5ed0 con 0x7fdea80d1870 2021-02-10T11:39:34.618 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:34.609 7fdeae08c700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdea80d1c70 con 0x7fdea80f0650 2021-02-10T11:39:34.618 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:34.609 7fdea77fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fdea80d1870 0x7fdea81b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:34.618 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:34.609 7fdea77fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fdea80d1870 0x7fdea81b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:53846/0 (socket says 172.21.2.112:53846) 2021-02-10T11:39:34.619 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:34.609 7fdea77fe700 1 -- 172.21.2.112:0/540136663 learned_addr learned my addr 172.21.2.112:0/540136663 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:39:34.619 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:34.609 7fdea77fe700 1 -- 172.21.2.112:0/540136663 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdea80f0650 msgr2=0x7fdea80cfb20 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:39:34.619 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:34.609 7fdea77fe700 1 --2- 172.21.2.112:0/540136663 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdea80f0650 0x7fdea80cfb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:34.619 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:34.609 7fdea77fe700 1 -- 172.21.2.112:0/540136663 >> v1:172.21.2.120:6789/0 conn(0x7fdea81b0a10 legacy=0x7fdea80cf330 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:39:34.619 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:34.609 7fdea77fe700 1 -- 172.21.2.112:0/540136663 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdea80d1e90 con 0x7fdea80d1870 2021-02-10T11:39:34.619 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:34.609 7fdea77fe700 1 --2- 172.21.2.112:0/540136663 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdea80d1870 0x7fdea81b03f0 secure :-1 s=READY pgs=1931 cs=0 l=1 rev1=1 rx=0x7fde90006700 tx=0x7fde90009b80).ready entity=mon.0 client_cookie=95a8ad8c9393752b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:34.619 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:34.609 7fdea67fc700 1 -- 172.21.2.112:0/540136663 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fde9000c0c0 con 0x7fdea80d1870 2021-02-10T11:39:34.619 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:34.609 7fdea67fc700 1 -- 172.21.2.112:0/540136663 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde9000c340 con 0x7fdea80d1870 2021-02-10T11:39:34.619 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:34.609 7fdea67fc700 1 -- 172.21.2.112:0/540136663 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fde9000c820 con 0x7fdea80d1870 2021-02-10T11:39:34.619 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:34.609 7fdeae08c700 1 -- 172.21.2.112:0/540136663 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdea80d1870 msgr2=0x7fdea81b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:34.619 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:34.609 7fdeae08c700 1 --2- 172.21.2.112:0/540136663 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdea80d1870 0x7fdea81b03f0 secure :-1 s=READY pgs=1931 cs=0 l=1 rev1=1 rx=0x7fde90006700 tx=0x7fde90009b80).stop 2021-02-10T11:39:34.619 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:34.610 7fdeae08c700 1 -- 172.21.2.112:0/540136663 shutdown_connections 2021-02-10T11:39:34.620 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:34.610 7fdeae08c700 1 --2- 172.21.2.112:0/540136663 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdea80f0650 0x7fdea80cfb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:34.620 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:34.610 7fdeae08c700 1 --2- 172.21.2.112:0/540136663 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdea80d1870 0x7fdea81b03f0 unknown :-1 s=CLOSED pgs=1931 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:34.620 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:34.610 7fdeae08c700 1 -- 172.21.2.112:0/540136663 >> 172.21.2.112:0/540136663 conn(0x7fdea80ce450 msgr2=0x7fdea80d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:34.620 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:34.610 7fdeae08c700 1 -- 172.21.2.112:0/540136663 shutdown_connections 2021-02-10T11:39:34.620 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:34.610 7fdeae08c700 1 -- 172.21.2.112:0/540136663 wait complete. 2021-02-10T11:39:34.621 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:34.624 7fdeae08c700 1 Processor -- start 2021-02-10T11:39:34.621 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:34.625 7fdeae08c700 1 -- start start 2021-02-10T11:39:34.621 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:34.625 7fdeae08c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdea80b0ad0 0x7fdea8091f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:34.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:34.625 7fdeae08c700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdea81a8850 0x7fdea81a8c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:34.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:34.625 7fdeae08c700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fdea81a6fa0 0x7fdea81a73a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:34.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:34.625 7fdeae08c700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdea81abfe0 con 0x7fdea80b0ad0 2021-02-10T11:39:34.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:34.625 7fdeae08c700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdea81a9c40 con 0x7fdea81a6fa0 2021-02-10T11:39:34.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:34.625 7fdeae08c700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdea81a9730 con 0x7fdea81a8850 2021-02-10T11:39:34.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:34.625 7fdea77fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdea80b0ad0 0x7fdea8091f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:34.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:34.625 7fdea77fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdea80b0ad0 0x7fdea8091f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:53850/0 (socket says 172.21.2.112:53850) 2021-02-10T11:39:34.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:34.625 7fdea77fe700 1 -- 172.21.2.112:0/1020091633 learned_addr learned my addr 172.21.2.112:0/1020091633 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:39:34.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:34.625 7fdea77fe700 1 -- 172.21.2.112:0/1020091633 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdea81a8850 msgr2=0x7fdea81a8c50 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:39:34.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:34.625 7fdea77fe700 1 --2- 172.21.2.112:0/1020091633 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdea81a8850 0x7fdea81a8c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:34.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:34.625 7fdea77fe700 1 -- 172.21.2.112:0/1020091633 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fdea81a6fa0 msgr2=0x7fdea81a73a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:34.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:34.625 7fdea77fe700 1 --2- 172.21.2.112:0/1020091633 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fdea81a6fa0 0x7fdea81a73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:34.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:34.625 7fdea77fe700 1 -- 172.21.2.112:0/1020091633 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdea81a9270 con 0x7fdea80b0ad0 2021-02-10T11:39:34.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:34.626 7fdea77fe700 1 --2- 172.21.2.112:0/1020091633 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdea80b0ad0 0x7fdea8091f30 secure :-1 s=READY pgs=1932 cs=0 l=1 rev1=1 rx=0x7fde9000b580 tx=0x7fde9000b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:34.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:34.626 7fde9ffff700 1 -- 172.21.2.112:0/1020091633 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fde9000bd10 con 0x7fdea80b0ad0 2021-02-10T11:39:34.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:34.626 7fde9ffff700 1 -- 172.21.2.112:0/1020091633 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde90002030 con 0x7fdea80b0ad0 2021-02-10T11:39:34.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:34.626 7fdeae08c700 1 -- 172.21.2.112:0/1020091633 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdea81a9270 con 0x7fdea80b0ad0 2021-02-10T11:39:34.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:34.626 7fdeae08c700 1 -- 172.21.2.112:0/1020091633 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdea81a7cb0 con 0x7fdea80b0ad0 2021-02-10T11:39:34.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:34.626 7fde9ffff700 1 -- 172.21.2.112:0/1020091633 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fde90002480 con 0x7fdea80b0ad0 2021-02-10T11:39:34.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:34.626 7fde9ffff700 1 -- 172.21.2.112:0/1020091633 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fde90000c90 con 0x7fdea80b0ad0 2021-02-10T11:39:34.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:34.627 7fde9ffff700 1 --2- 172.21.2.112:0/1020091633 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fde8c0186c0 0x7fde8c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:34.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:34.627 7fde9ffff700 1 -- 172.21.2.112:0/1020091633 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5659+0+0 (secure 0 0 0) 0x7fde90032590 con 0x7fdea80b0ad0 2021-02-10T11:39:34.624 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:34.627 7fdea6ffd700 1 --2- 172.21.2.112:0/1020091633 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fde8c0186c0 0x7fde8c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:34.624 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:34.627 7fdea6ffd700 1 --2- 172.21.2.112:0/1020091633 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fde8c0186c0 0x7fde8c01abc0 crc :-1 s=READY pgs=903 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:34.652 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:34.659 7fdeae08c700 1 -- 172.21.2.112:0/1020091633 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdea80ce450 con 0x7fdea80b0ad0 2021-02-10T11:39:34.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:34.661 7fde9ffff700 1 -- 172.21.2.112:0/1020091633 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fde900009b0 con 0x7fdea80b0ad0 2021-02-10T11:39:34.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:34.790 7fdeae08c700 1 -- 172.21.2.112:0/1020091633 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdea80ce450 con 0x7fdea80b0ad0 2021-02-10T11:39:34.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:34.791 7fde9ffff700 1 -- 172.21.2.112:0/1020091633 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) v1 ==== 94+0+1756 (secure 0 0 0) 0x7fde9000bd10 con 0x7fdea80b0ad0 2021-02-10T11:39:34.785 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 74 2021-02-10T11:39:34.785 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:39:34.785 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":74,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9621,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/1174368781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":1174368781},{"type":"v1","addr":"172.21.2.112:6833","nonce":1174368781}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":69}],"filesystems":[{"mdsmap":{"epoch":74,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:38:51.410458","modified":"2021-02-10 11:39:34.165266","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":93,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":9}]} 2021-02-10T11:39:34.787 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:34.793 7fdeae08c700 1 -- 172.21.2.112:0/1020091633 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fde8c0186c0 msgr2=0x7fde8c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:34.787 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:34.793 7fdeae08c700 1 --2- 172.21.2.112:0/1020091633 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fde8c0186c0 0x7fde8c01abc0 crc :-1 s=READY pgs=903 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:34.787 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:34.793 7fdeae08c700 1 -- 172.21.2.112:0/1020091633 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdea80b0ad0 msgr2=0x7fdea8091f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:34.787 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:34.793 7fdeae08c700 1 --2- 172.21.2.112:0/1020091633 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdea80b0ad0 0x7fdea8091f30 secure :-1 s=READY pgs=1932 cs=0 l=1 rev1=1 rx=0x7fde9000b580 tx=0x7fde9000b920).stop 2021-02-10T11:39:34.787 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:34.793 7fdeae08c700 1 -- 172.21.2.112:0/1020091633 shutdown_connections 2021-02-10T11:39:34.787 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:34.793 7fdeae08c700 1 --2- 172.21.2.112:0/1020091633 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fde8c0186c0 0x7fde8c01abc0 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:34.787 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:34.793 7fdeae08c700 1 --2- 172.21.2.112:0/1020091633 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fdea81a6fa0 0x7fdea81a73a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:34.787 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:34.793 7fdeae08c700 1 --2- 172.21.2.112:0/1020091633 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdea81a8850 0x7fdea81a8c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:34.787 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:34.793 7fdeae08c700 1 --2- 172.21.2.112:0/1020091633 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdea80b0ad0 0x7fdea8091f30 unknown :-1 s=CLOSED pgs=1932 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:34.787 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:34.793 7fdeae08c700 1 -- 172.21.2.112:0/1020091633 >> 172.21.2.112:0/1020091633 conn(0x7fdea819f8f0 msgr2=0x7fdea81a5f80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:34.788 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:34.793 7fdeae08c700 1 -- 172.21.2.112:0/1020091633 shutdown_connections 2021-02-10T11:39:34.788 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:34.793 7fdeae08c700 1 -- 172.21.2.112:0/1020091633 wait complete. 2021-02-10T11:39:34.796 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2021-02-10T11:39:35.228 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:35.218 7f8bd80ba700 1 Processor -- start 2021-02-10T11:39:35.228 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:35.218 7f8bd80ba700 1 -- start start 2021-02-10T11:39:35.228 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:35.219 7f8bd80ba700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f8bd00d1c70 0x7f8bd00f20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:35.228 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:35.219 7f8bd80ba700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bd00f0b10 con 0x7f8bd00d1870 2021-02-10T11:39:35.228 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:35.219 7f8bd80ba700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bd01b0be0 con 0x7f8bd00d0340 2021-02-10T11:39:35.228 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:35.219 7f8bd80ba700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8bd00e5ed0 con 0x7f8bd00d1c70 2021-02-10T11:39:35.228 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:35.219 7f8bd5ed6700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f8bd00d1870 0x7f8bd01b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:53320/0 (socket says 172.21.2.112:53320) 2021-02-10T11:39:35.228 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:35.219 7f8bd5ed6700 1 -- 172.21.2.112:0/4110749139 learned_addr learned my addr 172.21.2.112:0/4110749139 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:39:35.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:35.219 7f8bd66d7700 1 --2- 172.21.2.112:0/4110749139 >> v2:172.21.2.112:3300/0 conn(0x7f8bd00d1c70 0x7f8bd00f20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:35.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:35.219 7f8bd66d7700 1 -- 172.21.2.112:0/4110749139 >> v1:172.21.2.120:6789/0 conn(0x7f8bd00d0340 legacy=0x7f8bd00ef640 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:39:35.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:35.219 7f8bd66d7700 1 -- 172.21.2.112:0/4110749139 >> v1:172.21.2.112:6789/0 conn(0x7f8bd00d1870 legacy=0x7f8bd01b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:35.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:35.219 7f8bd66d7700 1 -- 172.21.2.112:0/4110749139 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bd00efe30 con 0x7f8bd00d1c70 2021-02-10T11:39:35.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:35.219 7f8bd66d7700 1 --2- 172.21.2.112:0/4110749139 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8bd00d1c70 0x7f8bd00f20a0 secure :-1 s=READY pgs=1934 cs=0 l=1 rev1=1 rx=0x7f8bc8006e30 tx=0x7f8bc8009e60).ready entity=mon.0 client_cookie=ad337c86699d913b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:35.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:35.219 7f8bd4ed4700 1 -- 172.21.2.112:0/4110749139 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8bc800b300 con 0x7f8bd00d1c70 2021-02-10T11:39:35.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:35.219 7f8bd4ed4700 1 -- 172.21.2.112:0/4110749139 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bc800b580 con 0x7f8bd00d1c70 2021-02-10T11:39:35.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:35.219 7f8bd4ed4700 1 -- 172.21.2.112:0/4110749139 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8bc800ba10 con 0x7f8bd00d1c70 2021-02-10T11:39:35.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:35.219 7f8bd80ba700 1 -- 172.21.2.112:0/4110749139 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8bd00d1c70 msgr2=0x7f8bd00f20a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:35.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:35.219 7f8bd80ba700 1 --2- 172.21.2.112:0/4110749139 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8bd00d1c70 0x7f8bd00f20a0 secure :-1 s=READY pgs=1934 cs=0 l=1 rev1=1 rx=0x7f8bc8006e30 tx=0x7f8bc8009e60).stop 2021-02-10T11:39:35.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:35.220 7f8bd80ba700 1 -- 172.21.2.112:0/4110749139 shutdown_connections 2021-02-10T11:39:35.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:35.220 7f8bd80ba700 1 --2- 172.21.2.112:0/4110749139 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8bd00d1c70 0x7f8bd00f20a0 unknown :-1 s=CLOSED pgs=1934 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:35.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:35.220 7f8bd80ba700 1 -- 172.21.2.112:0/4110749139 >> 172.21.2.112:0/4110749139 conn(0x7f8bd00ce450 msgr2=0x7f8bd00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:35.230 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:35.220 7f8bd80ba700 1 -- 172.21.2.112:0/4110749139 shutdown_connections 2021-02-10T11:39:35.230 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:35.220 7f8bd80ba700 1 -- 172.21.2.112:0/4110749139 wait complete. 2021-02-10T11:39:35.231 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:35.235 7f8bd80ba700 1 Processor -- start 2021-02-10T11:39:35.231 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:35.235 7f8bd80ba700 1 -- start start 2021-02-10T11:39:35.231 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:35.235 7f8bd80ba700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8bd00a3a30 0x7f8bd00eda20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:35.231 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:35.235 7f8bd80ba700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8bd00ea400 0x7f8bd00ee040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:35.231 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:35.235 7f8bd80ba700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8bd00a1d90 0x7f8bd00a2190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:35.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:35.235 7f8bd80ba700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bd00a1630 con 0x7f8bd00a3a30 2021-02-10T11:39:35.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:35.235 7f8bd80ba700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bd00ece80 con 0x7f8bd00a1d90 2021-02-10T11:39:35.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:35.235 7f8bd80ba700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bd00a1810 con 0x7f8bd00ea400 2021-02-10T11:39:35.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:35.235 7f8bd5ed6700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8bd00a3a30 0x7f8bd00eda20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:35.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:35.235 7f8bd5ed6700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8bd00a3a30 0x7f8bd00eda20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:53868/0 (socket says 172.21.2.112:53868) 2021-02-10T11:39:35.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:35.235 7f8bd5ed6700 1 -- 172.21.2.112:0/1271756686 learned_addr learned my addr 172.21.2.112:0/1271756686 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:39:35.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:35.236 7f8bd66d7700 1 --2- 172.21.2.112:0/1271756686 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8bd00a1d90 0x7f8bd00a2190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:35.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:35.236 7f8bd5ed6700 1 -- 172.21.2.112:0/1271756686 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8bd00ea400 msgr2=0x7f8bd00ee040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:35.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:35.236 7f8bd5ed6700 1 --2- 172.21.2.112:0/1271756686 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8bd00ea400 0x7f8bd00ee040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:35.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:35.236 7f8bd5ed6700 1 -- 172.21.2.112:0/1271756686 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8bd00a1d90 msgr2=0x7f8bd00a2190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:35.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:35.236 7f8bd5ed6700 1 --2- 172.21.2.112:0/1271756686 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8bd00a1d90 0x7f8bd00a2190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:35.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:35.236 7f8bd5ed6700 1 -- 172.21.2.112:0/1271756686 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bd01c24c0 con 0x7f8bd00a3a30 2021-02-10T11:39:35.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:35.236 7f8bd66d7700 1 --2- 172.21.2.112:0/1271756686 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8bd00a1d90 0x7f8bd00a2190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:39:35.233 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:35.236 7f8bd5ed6700 1 --2- 172.21.2.112:0/1271756686 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8bd00a3a30 0x7f8bd00eda20 secure :-1 s=READY pgs=1935 cs=0 l=1 rev1=1 rx=0x7f8bcc001c60 tx=0x7f8bcc00a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:35.233 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:35.236 7f8bc27fc700 1 -- 172.21.2.112:0/1271756686 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8bcc00acc0 con 0x7f8bd00a3a30 2021-02-10T11:39:35.233 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:35.236 7f8bc27fc700 1 -- 172.21.2.112:0/1271756686 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bcc004140 con 0x7f8bd00a3a30 2021-02-10T11:39:35.233 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:35.236 7f8bc27fc700 1 -- 172.21.2.112:0/1271756686 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8bcc0045a0 con 0x7f8bd00a3a30 2021-02-10T11:39:35.233 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:35.236 7f8bd80ba700 1 -- 172.21.2.112:0/1271756686 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bd01c24c0 con 0x7f8bd00a3a30 2021-02-10T11:39:35.233 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:35.236 7f8bd80ba700 1 -- 172.21.2.112:0/1271756686 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bd01c24c0 con 0x7f8bd00a3a30 2021-02-10T11:39:35.233 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:35.237 7f8bc27fc700 1 -- 172.21.2.112:0/1271756686 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8bcc000d90 con 0x7f8bd00a3a30 2021-02-10T11:39:35.233 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:35.237 7f8bc27fc700 1 --2- 172.21.2.112:0/1271756686 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8bb80186c0 0x7f8bb801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:35.233 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:35.237 7f8bc27fc700 1 -- 172.21.2.112:0/1271756686 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5659+0+0 (secure 0 0 0) 0x7f8bcc0325c0 con 0x7f8bd00a3a30 2021-02-10T11:39:35.233 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:35.237 7f8bd56d5700 1 --2- 172.21.2.112:0/1271756686 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8bb80186c0 0x7f8bb801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:35.233 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:35.238 7f8bd56d5700 1 --2- 172.21.2.112:0/1271756686 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8bb80186c0 0x7f8bb801abc0 crc :-1 s=READY pgs=904 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:35.262 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:35.268 7f8bd80ba700 1 -- 172.21.2.112:0/1271756686 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bd00e7d30 con 0x7f8bd00a3a30 2021-02-10T11:39:35.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:35.271 7f8bc27fc700 1 -- 172.21.2.112:0/1271756686 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8bcc0009b0 con 0x7f8bd00a3a30 2021-02-10T11:39:35.391 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:35.397 7f8bd80ba700 1 -- 172.21.2.112:0/1271756686 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f8bd00e7d30 con 0x7f8bd00a3a30 2021-02-10T11:39:36.169 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:36.176 7f8bc27fc700 1 -- 172.21.2.112:0/1271756686 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v75) v1 ==== 104+0+0 (secure 0 0 0) 0x7f8bcc000d90 con 0x7f8bd00a3a30 2021-02-10T11:39:36.213 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:36.219 7f8bd80ba700 1 -- 172.21.2.112:0/1271756686 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8bb80186c0 msgr2=0x7f8bb801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:36.214 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:36.219 7f8bd80ba700 1 --2- 172.21.2.112:0/1271756686 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8bb80186c0 0x7f8bb801abc0 crc :-1 s=READY pgs=904 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:36.214 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:36.219 7f8bd80ba700 1 -- 172.21.2.112:0/1271756686 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8bd00a3a30 msgr2=0x7f8bd00eda20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:36.214 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:36.219 7f8bd80ba700 1 --2- 172.21.2.112:0/1271756686 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8bd00a3a30 0x7f8bd00eda20 secure :-1 s=READY pgs=1935 cs=0 l=1 rev1=1 rx=0x7f8bcc001c60 tx=0x7f8bcc00a8d0).stop 2021-02-10T11:39:36.214 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:36.219 7f8bd80ba700 1 -- 172.21.2.112:0/1271756686 shutdown_connections 2021-02-10T11:39:36.214 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:36.219 7f8bd80ba700 1 --2- 172.21.2.112:0/1271756686 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8bb80186c0 0x7f8bb801abc0 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:36.214 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:36.219 7f8bd80ba700 1 --2- 172.21.2.112:0/1271756686 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8bd00a1d90 0x7f8bd00a2190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:36.214 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:36.219 7f8bd80ba700 1 --2- 172.21.2.112:0/1271756686 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8bd00ea400 0x7f8bd00ee040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:36.214 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:36.219 7f8bd80ba700 1 --2- 172.21.2.112:0/1271756686 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8bd00a3a30 0x7f8bd00eda20 unknown :-1 s=CLOSED pgs=1935 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:36.214 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:36.219 7f8bd80ba700 1 -- 172.21.2.112:0/1271756686 >> 172.21.2.112:0/1271756686 conn(0x7f8bd0090420 msgr2=0x7f8bd00bea90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:36.214 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:36.220 7f8bd80ba700 1 -- 172.21.2.112:0/1271756686 shutdown_connections 2021-02-10T11:39:36.214 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:36.220 7f8bd80ba700 1 -- 172.21.2.112:0/1271756686 wait complete. 2021-02-10T11:39:36.221 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2021-02-10T11:39:36.651 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:36.641 7efee6afb700 1 Processor -- start 2021-02-10T11:39:36.651 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:36.641 7efee6afb700 1 -- start start 2021-02-10T11:39:36.651 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:36.641 7efee6afb700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7efee00d1880 0x7efee01b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:36.651 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:36.641 7efee6afb700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7efee00cff80 0x7efee0109140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:36.652 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:36.641 7efee6afb700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efee00fe1d0 con 0x7efee01b0af0 2021-02-10T11:39:36.652 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:36.641 7efee6afb700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efee0109ae0 con 0x7efee00d1880 2021-02-10T11:39:36.652 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:36.641 7efee6afb700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efee0109760 con 0x7efee00cff80 2021-02-10T11:39:36.652 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:36.642 7efedffff700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7efee01b0af0 0x7efee0108950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:53334/0 (socket says 172.21.2.112:53334) 2021-02-10T11:39:36.652 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:36.642 7efedffff700 1 -- 172.21.2.112:0/345382154 learned_addr learned my addr 172.21.2.112:0/345382154 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:39:36.652 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:36.642 7efee4917700 1 --2- 172.21.2.112:0/345382154 >> v2:172.21.2.112:3300/0 conn(0x7efee00d1880 0x7efee01b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:36.652 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:36.642 7efedf7fe700 1 -- 172.21.2.112:0/345382154 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3459289040 0 0) 0x7efed40010b0 con 0x7efee01b0af0 2021-02-10T11:39:36.652 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:36.642 7efedf7fe700 1 -- 172.21.2.112:0/345382154 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efec8002fe0 con 0x7efee01b0af0 2021-02-10T11:39:36.652 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:36.642 7efee5118700 1 --2- 172.21.2.112:0/345382154 >> v2:172.21.2.120:3300/0 conn(0x7efee00cff80 0x7efee0109140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:36.652 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:36.642 7efee4917700 1 -- 172.21.2.112:0/345382154 >> v2:172.21.2.120:3300/0 conn(0x7efee00cff80 msgr2=0x7efee0109140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:36.652 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:36.642 7efee4917700 1 --2- 172.21.2.112:0/345382154 >> v2:172.21.2.120:3300/0 conn(0x7efee00cff80 0x7efee0109140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:36.652 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:36.642 7efee4917700 1 -- 172.21.2.112:0/345382154 >> v1:172.21.2.112:6789/0 conn(0x7efee01b0af0 legacy=0x7efee0108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:36.652 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:36.642 7efee4917700 1 -- 172.21.2.112:0/345382154 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efee0088070 con 0x7efee00d1880 2021-02-10T11:39:36.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:36.642 7efee4917700 1 --2- 172.21.2.112:0/345382154 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efee00d1880 0x7efee01b04d0 secure :-1 s=READY pgs=1937 cs=0 l=1 rev1=1 rx=0x7efed0006650 tx=0x7efed0009a80).ready entity=mon.0 client_cookie=8c121f534346e236 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:36.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:36.642 7efedf7fe700 1 -- 172.21.2.112:0/345382154 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efed000c0c0 con 0x7efee00d1880 2021-02-10T11:39:36.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:36.642 7efedf7fe700 1 -- 172.21.2.112:0/345382154 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efed000c340 con 0x7efee00d1880 2021-02-10T11:39:36.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:36.642 7efedf7fe700 1 -- 172.21.2.112:0/345382154 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efed000c860 con 0x7efee00d1880 2021-02-10T11:39:36.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:36.642 7efee6afb700 1 -- 172.21.2.112:0/345382154 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efee00d1880 msgr2=0x7efee01b04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:36.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:36.642 7efee6afb700 1 --2- 172.21.2.112:0/345382154 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efee00d1880 0x7efee01b04d0 secure :-1 s=READY pgs=1937 cs=0 l=1 rev1=1 rx=0x7efed0006650 tx=0x7efed0009a80).stop 2021-02-10T11:39:36.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:36.643 7efee6afb700 1 -- 172.21.2.112:0/345382154 shutdown_connections 2021-02-10T11:39:36.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:36.643 7efee6afb700 1 --2- 172.21.2.112:0/345382154 >> v2:172.21.2.120:3300/0 conn(0x7efee00cff80 0x7efee0109140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:36.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:36.643 7efee6afb700 1 --2- 172.21.2.112:0/345382154 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efee00d1880 0x7efee01b04d0 unknown :-1 s=CLOSED pgs=1937 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:36.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:36.643 7efee6afb700 1 -- 172.21.2.112:0/345382154 >> 172.21.2.112:0/345382154 conn(0x7efee00ce430 msgr2=0x7efee00d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:36.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:36.643 7efee6afb700 1 -- 172.21.2.112:0/345382154 shutdown_connections 2021-02-10T11:39:36.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:36.643 7efee6afb700 1 -- 172.21.2.112:0/345382154 wait complete. 2021-02-10T11:39:36.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:36.658 7efee6afb700 1 Processor -- start 2021-02-10T11:39:36.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:36.658 7efee6afb700 1 -- start start 2021-02-10T11:39:36.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:36.659 7efee6afb700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efee01b58d0 0x7efee00aa4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:36.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:36.659 7efee6afb700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efee00aaaf0 0x7efee01b80a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:36.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:36.659 7efee6afb700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7efee01b86c0 0x7efee019de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:36.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:36.659 7efee6afb700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efee01b8ac0 con 0x7efee01b58d0 2021-02-10T11:39:36.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:36.659 7efee6afb700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efee01b8e20 con 0x7efee01b86c0 2021-02-10T11:39:36.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:36.659 7efee6afb700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efee01b5cd0 con 0x7efee00aaaf0 2021-02-10T11:39:36.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:36.659 7efee4917700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efee01b58d0 0x7efee00aa4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:36.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:36.659 7efee4917700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efee01b58d0 0x7efee00aa4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:53876/0 (socket says 172.21.2.112:53876) 2021-02-10T11:39:36.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:36.659 7efee4917700 1 -- 172.21.2.112:0/4194329094 learned_addr learned my addr 172.21.2.112:0/4194329094 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:39:36.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:36.659 7efee5118700 1 --2- 172.21.2.112:0/4194329094 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7efee01b86c0 0x7efee019de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:36.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:36.659 7efedffff700 1 --2- 172.21.2.112:0/4194329094 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efee00aaaf0 0x7efee01b80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:36.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:36.659 7efee4917700 1 -- 172.21.2.112:0/4194329094 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efee00aaaf0 msgr2=0x7efee01b80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:36.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:36.659 7efee4917700 1 --2- 172.21.2.112:0/4194329094 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efee00aaaf0 0x7efee01b80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:36.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:36.659 7efee4917700 1 -- 172.21.2.112:0/4194329094 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7efee01b86c0 msgr2=0x7efee019de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:36.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:36.659 7efee4917700 1 --2- 172.21.2.112:0/4194329094 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7efee01b86c0 0x7efee019de70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:36.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:36.659 7efee4917700 1 -- 172.21.2.112:0/4194329094 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efee0095710 con 0x7efee01b58d0 2021-02-10T11:39:36.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:36.659 7efee4917700 1 --2- 172.21.2.112:0/4194329094 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efee01b58d0 0x7efee00aa4d0 secure :-1 s=READY pgs=1938 cs=0 l=1 rev1=1 rx=0x7efed0009630 tx=0x7efed000b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:36.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:36.660 7efedd7fa700 1 -- 172.21.2.112:0/4194329094 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efed00093a0 con 0x7efee01b58d0 2021-02-10T11:39:36.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:36.660 7efedd7fa700 1 -- 172.21.2.112:0/4194329094 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efed000bd70 con 0x7efee01b58d0 2021-02-10T11:39:36.657 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:36.660 7efee6afb700 1 -- 172.21.2.112:0/4194329094 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efee0095710 con 0x7efee01b58d0 2021-02-10T11:39:36.657 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:36.660 7efee6afb700 1 -- 172.21.2.112:0/4194329094 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efee019e780 con 0x7efee01b58d0 2021-02-10T11:39:36.657 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:36.660 7efedd7fa700 1 -- 172.21.2.112:0/4194329094 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efed0005560 con 0x7efee01b58d0 2021-02-10T11:39:36.657 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:36.660 7efedd7fa700 1 -- 172.21.2.112:0/4194329094 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7efed000a000 con 0x7efee01b58d0 2021-02-10T11:39:36.657 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:36.661 7efedd7fa700 1 --2- 172.21.2.112:0/4194329094 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7efec00186c0 0x7efec001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:36.657 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:36.661 7efedd7fa700 1 -- 172.21.2.112:0/4194329094 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5659+0+0 (secure 0 0 0) 0x7efed000bd70 con 0x7efee01b58d0 2021-02-10T11:39:36.657 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:36.661 7efedffff700 1 --2- 172.21.2.112:0/4194329094 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7efec00186c0 0x7efec001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:36.657 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:36.661 7efedffff700 1 --2- 172.21.2.112:0/4194329094 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7efec00186c0 0x7efec001abc0 crc :-1 s=READY pgs=905 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:36.685 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:36.691 7efee6afb700 1 -- 172.21.2.112:0/4194329094 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efee0082aa0 con 0x7efee01b58d0 2021-02-10T11:39:36.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:36.694 7efedd7fa700 1 -- 172.21.2.112:0/4194329094 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7efed0009340 con 0x7efee01b58d0 2021-02-10T11:39:36.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:36.819 7efee6afb700 1 -- 172.21.2.112:0/4194329094 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7efee0082aa0 con 0x7efee01b58d0 2021-02-10T11:39:37.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:37.178 7efedd7fa700 1 -- 172.21.2.112:0/4194329094 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v94) v1 ==== 185+0+0 (secure 0 0 0) 0x7efed001f4e0 con 0x7efee01b58d0 2021-02-10T11:39:37.179 INFO:teuthology.orchestra.run.gibba012.stderr:pool 'cephfs_metadata' removed 2021-02-10T11:39:37.181 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:37.187 7efee6afb700 1 -- 172.21.2.112:0/4194329094 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7efec00186c0 msgr2=0x7efec001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:37.181 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:37.187 7efee6afb700 1 --2- 172.21.2.112:0/4194329094 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7efec00186c0 0x7efec001abc0 crc :-1 s=READY pgs=905 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:37.181 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:37.187 7efee6afb700 1 -- 172.21.2.112:0/4194329094 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efee01b58d0 msgr2=0x7efee00aa4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:37.181 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:37.187 7efee6afb700 1 --2- 172.21.2.112:0/4194329094 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efee01b58d0 0x7efee00aa4d0 secure :-1 s=READY pgs=1938 cs=0 l=1 rev1=1 rx=0x7efed0009630 tx=0x7efed000b980).stop 2021-02-10T11:39:37.181 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:37.187 7efee6afb700 1 -- 172.21.2.112:0/4194329094 shutdown_connections 2021-02-10T11:39:37.182 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:37.187 7efee6afb700 1 --2- 172.21.2.112:0/4194329094 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7efec00186c0 0x7efec001abc0 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:37.182 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:37.187 7efee6afb700 1 --2- 172.21.2.112:0/4194329094 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7efee01b86c0 0x7efee019de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:37.182 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:37.187 7efee6afb700 1 --2- 172.21.2.112:0/4194329094 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efee00aaaf0 0x7efee01b80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:37.182 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:37.187 7efee6afb700 1 --2- 172.21.2.112:0/4194329094 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efee01b58d0 0x7efee00aa4d0 unknown :-1 s=CLOSED pgs=1938 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:37.182 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:37.187 7efee6afb700 1 -- 172.21.2.112:0/4194329094 >> 172.21.2.112:0/4194329094 conn(0x7efee019f940 msgr2=0x7efee018c290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:37.182 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:37.188 7efee6afb700 1 -- 172.21.2.112:0/4194329094 shutdown_connections 2021-02-10T11:39:37.182 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:37.188 7efee6afb700 1 -- 172.21.2.112:0/4194329094 wait complete. 2021-02-10T11:39:37.190 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2021-02-10T11:39:37.621 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:37.611 7fed4e8fb700 1 Processor -- start 2021-02-10T11:39:37.621 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:37.611 7fed4e8fb700 1 -- start start 2021-02-10T11:39:37.621 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:37.611 7fed4e8fb700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fed4810b000 0x7fed480cf760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:37.621 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:37.611 7fed4e8fb700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed480fe1d0 con 0x7fed480d1c80 2021-02-10T11:39:37.621 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:37.612 7fed4e8fb700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed481b0cc0 con 0x7fed480d1880 2021-02-10T11:39:37.621 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:37.612 7fed4e8fb700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed48109a70 con 0x7fed4810b000 2021-02-10T11:39:37.621 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:37.612 7fed477fe700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fed480d1c80 0x7fed480cef70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:53344/0 (socket says 172.21.2.112:53344) 2021-02-10T11:39:37.621 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:37.612 7fed477fe700 1 -- 172.21.2.112:0/2731255501 learned_addr learned my addr 172.21.2.112:0/2731255501 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:39:37.621 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:37.612 7fed4cf18700 1 --2- 172.21.2.112:0/2731255501 >> v2:172.21.2.112:3300/0 conn(0x7fed4810b000 0x7fed480cf760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:37.621 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:37.612 7fed46ffd700 1 -- 172.21.2.112:0/2731255501 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3422546090 0 0) 0x7fed3c0010b0 con 0x7fed480d1c80 2021-02-10T11:39:37.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:37.612 7fed46ffd700 1 -- 172.21.2.112:0/2731255501 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed30002fe0 con 0x7fed480d1c80 2021-02-10T11:39:37.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:37.612 7fed4cf18700 1 -- 172.21.2.112:0/2731255501 >> v1:172.21.2.120:6789/0 conn(0x7fed480d1880 legacy=0x7fed481b04d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:39:37.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:37.612 7fed46ffd700 1 -- 172.21.2.112:0/2731255501 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1297833930 0 0) 0x7fed3c001010 con 0x7fed480d1c80 2021-02-10T11:39:37.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:37.612 7fed4cf18700 1 -- 172.21.2.112:0/2731255501 >> v1:172.21.2.112:6789/0 conn(0x7fed480d1c80 legacy=0x7fed480cef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:37.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:37.612 7fed4cf18700 1 -- 172.21.2.112:0/2731255501 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed4810b400 con 0x7fed4810b000 2021-02-10T11:39:37.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:37.612 7fed4cf18700 1 --2- 172.21.2.112:0/2731255501 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fed4810b000 0x7fed480cf760 secure :-1 s=READY pgs=1940 cs=0 l=1 rev1=1 rx=0x7fed40006e30 tx=0x7fed40009e60).ready entity=mon.0 client_cookie=3e197c4c7df9d69c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:37.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:37.612 7fed46ffd700 1 -- 172.21.2.112:0/2731255501 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fed40006a50 con 0x7fed4810b000 2021-02-10T11:39:37.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:37.612 7fed46ffd700 1 -- 172.21.2.112:0/2731255501 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed4000b300 con 0x7fed4810b000 2021-02-10T11:39:37.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:37.613 7fed46ffd700 1 -- 172.21.2.112:0/2731255501 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fed4000b820 con 0x7fed4810b000 2021-02-10T11:39:37.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:37.613 7fed4e8fb700 1 -- 172.21.2.112:0/2731255501 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fed4810b000 msgr2=0x7fed480cf760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:37.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:37.613 7fed4e8fb700 1 --2- 172.21.2.112:0/2731255501 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fed4810b000 0x7fed480cf760 secure :-1 s=READY pgs=1940 cs=0 l=1 rev1=1 rx=0x7fed40006e30 tx=0x7fed40009e60).stop 2021-02-10T11:39:37.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:37.613 7fed4e8fb700 1 -- 172.21.2.112:0/2731255501 shutdown_connections 2021-02-10T11:39:37.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:37.613 7fed4e8fb700 1 --2- 172.21.2.112:0/2731255501 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fed4810b000 0x7fed480cf760 unknown :-1 s=CLOSED pgs=1940 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:37.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:37.613 7fed4e8fb700 1 -- 172.21.2.112:0/2731255501 >> 172.21.2.112:0/2731255501 conn(0x7fed480ce430 msgr2=0x7fed480d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:37.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:37.613 7fed4e8fb700 1 -- 172.21.2.112:0/2731255501 shutdown_connections 2021-02-10T11:39:37.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:37.613 7fed4e8fb700 1 -- 172.21.2.112:0/2731255501 wait complete. 2021-02-10T11:39:37.624 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:37.628 7fed4e8fb700 1 Processor -- start 2021-02-10T11:39:37.624 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:37.628 7fed4e8fb700 1 -- start start 2021-02-10T11:39:37.624 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:37.628 7fed4e8fb700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fed481b58d0 0x7fed480aa4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:37.624 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:37.628 7fed4e8fb700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fed480aaaf0 0x7fed481b80a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:37.624 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:37.628 7fed4e8fb700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fed481b86c0 0x7fed4819de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:37.624 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:37.628 7fed4e8fb700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed481b8ac0 con 0x7fed480aaaf0 2021-02-10T11:39:37.624 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:37.628 7fed4e8fb700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed481b8e20 con 0x7fed481b58d0 2021-02-10T11:39:37.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:37.628 7fed4e8fb700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed481b5cd0 con 0x7fed481b86c0 2021-02-10T11:39:37.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:37.628 7fed477fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fed480aaaf0 0x7fed481b80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:37.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:37.628 7fed477fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fed480aaaf0 0x7fed481b80a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:53892/0 (socket says 172.21.2.112:53892) 2021-02-10T11:39:37.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:37.628 7fed477fe700 1 -- 172.21.2.112:0/3308895724 learned_addr learned my addr 172.21.2.112:0/3308895724 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:39:37.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:37.629 7fed477fe700 1 -- 172.21.2.112:0/3308895724 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fed481b86c0 msgr2=0x7fed4819de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:37.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:37.629 7fed477fe700 1 --2- 172.21.2.112:0/3308895724 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fed481b86c0 0x7fed4819de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:37.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:37.629 7fed477fe700 1 -- 172.21.2.112:0/3308895724 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fed481b58d0 msgr2=0x7fed480aa4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:37.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:37.629 7fed477fe700 1 --2- 172.21.2.112:0/3308895724 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fed481b58d0 0x7fed480aa4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:37.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:37.629 7fed477fe700 1 -- 172.21.2.112:0/3308895724 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed48095710 con 0x7fed480aaaf0 2021-02-10T11:39:37.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:37.629 7fed477fe700 1 --2- 172.21.2.112:0/3308895724 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fed480aaaf0 0x7fed481b80a0 secure :-1 s=READY pgs=1941 cs=0 l=1 rev1=1 rx=0x7fed3c001c60 tx=0x7fed3c00a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:37.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:37.629 7fed44ff9700 1 -- 172.21.2.112:0/3308895724 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fed3c00acc0 con 0x7fed480aaaf0 2021-02-10T11:39:37.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:37.629 7fed44ff9700 1 -- 172.21.2.112:0/3308895724 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed3c004140 con 0x7fed480aaaf0 2021-02-10T11:39:37.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:37.629 7fed44ff9700 1 -- 172.21.2.112:0/3308895724 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fed3c004660 con 0x7fed480aaaf0 2021-02-10T11:39:37.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:37.629 7fed4e8fb700 1 -- 172.21.2.112:0/3308895724 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed48095710 con 0x7fed480aaaf0 2021-02-10T11:39:37.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:37.629 7fed4e8fb700 1 -- 172.21.2.112:0/3308895724 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed4819e780 con 0x7fed480aaaf0 2021-02-10T11:39:37.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:37.630 7fed44ff9700 1 -- 172.21.2.112:0/3308895724 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fed3c004320 con 0x7fed480aaaf0 2021-02-10T11:39:37.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:37.630 7fed44ff9700 1 --2- 172.21.2.112:0/3308895724 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fed280186c0 0x7fed2801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:37.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:37.630 7fed44ff9700 1 -- 172.21.2.112:0/3308895724 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5221+0+0 (secure 0 0 0) 0x7fed3c0323c0 con 0x7fed480aaaf0 2021-02-10T11:39:37.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:37.630 7fed47fff700 1 --2- 172.21.2.112:0/3308895724 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fed280186c0 0x7fed2801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:37.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:37.631 7fed47fff700 1 --2- 172.21.2.112:0/3308895724 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fed280186c0 0x7fed2801abc0 crc :-1 s=READY pgs=906 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:37.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:37.661 7fed4e8fb700 1 -- 172.21.2.112:0/3308895724 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed48082aa0 con 0x7fed480aaaf0 2021-02-10T11:39:37.657 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:37.664 7fed44ff9700 1 -- 172.21.2.112:0/3308895724 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fed3c004f00 con 0x7fed480aaaf0 2021-02-10T11:39:37.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:37.790 7fed4e8fb700 1 -- 172.21.2.112:0/3308895724 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fed48082aa0 con 0x7fed480aaaf0 2021-02-10T11:39:38.176 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:38.182 7fed44ff9700 1 -- 172.21.2.112:0/3308895724 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v95) v1 ==== 173+0+0 (secure 0 0 0) 0x7fed3c000940 con 0x7fed480aaaf0 2021-02-10T11:39:38.199 INFO:teuthology.orchestra.run.gibba012.stderr:pool 'cephfs_data' removed 2021-02-10T11:39:38.201 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:38.207 7fed4e8fb700 1 -- 172.21.2.112:0/3308895724 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fed280186c0 msgr2=0x7fed2801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:38.201 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:38.207 7fed4e8fb700 1 --2- 172.21.2.112:0/3308895724 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fed280186c0 0x7fed2801abc0 crc :-1 s=READY pgs=906 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:38.201 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:38.207 7fed4e8fb700 1 -- 172.21.2.112:0/3308895724 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fed480aaaf0 msgr2=0x7fed481b80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:38.201 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:38.207 7fed4e8fb700 1 --2- 172.21.2.112:0/3308895724 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fed480aaaf0 0x7fed481b80a0 secure :-1 s=READY pgs=1941 cs=0 l=1 rev1=1 rx=0x7fed3c001c60 tx=0x7fed3c00a8d0).stop 2021-02-10T11:39:38.201 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:38.208 7fed4e8fb700 1 -- 172.21.2.112:0/3308895724 shutdown_connections 2021-02-10T11:39:38.201 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:38.208 7fed4e8fb700 1 --2- 172.21.2.112:0/3308895724 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fed280186c0 0x7fed2801abc0 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:38.202 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:38.208 7fed4e8fb700 1 --2- 172.21.2.112:0/3308895724 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fed481b86c0 0x7fed4819de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:38.202 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:38.208 7fed4e8fb700 1 --2- 172.21.2.112:0/3308895724 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fed480aaaf0 0x7fed481b80a0 unknown :-1 s=CLOSED pgs=1941 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:38.202 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:38.208 7fed4e8fb700 1 --2- 172.21.2.112:0/3308895724 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fed481b58d0 0x7fed480aa4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:38.202 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:38.208 7fed4e8fb700 1 -- 172.21.2.112:0/3308895724 >> 172.21.2.112:0/3308895724 conn(0x7fed4819f940 msgr2=0x7fed4818c290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:38.202 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:38.208 7fed4e8fb700 1 -- 172.21.2.112:0/3308895724 shutdown_connections 2021-02-10T11:39:38.202 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:38.208 7fed4e8fb700 1 -- 172.21.2.112:0/3308895724 wait complete. 2021-02-10T11:39:38.210 INFO:tasks.ceph.mds.a:Restarting daemon 2021-02-10T11:39:38.210 INFO:tasks.ceph.mds.a:Stopping old one... 2021-02-10T11:39:38.210 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2021-02-10T11:39:38.211 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:39:38.211 INFO:tasks.ceph.mds.b:Restarting daemon 2021-02-10T11:39:38.211 INFO:tasks.ceph.mds.b:Stopping old one... 2021-02-10T11:39:38.211 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2021-02-10T11:39:38.211 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:39:38.216 INFO:tasks.ceph.mds.a:Stopped 2021-02-10T11:39:38.217 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2021-02-10T11:39:38.217 INFO:tasks.ceph.mds.b:Stopped 2021-02-10T11:39:38.218 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2021-02-10T11:39:38.259 INFO:tasks.ceph.mds.a:Started 2021-02-10T11:39:38.259 DEBUG:teuthology.parallel:result is None 2021-02-10T11:39:38.261 INFO:tasks.ceph.mds.b:Started 2021-02-10T11:39:38.261 DEBUG:teuthology.parallel:result is None 2021-02-10T11:39:38.261 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd blacklist clear 2021-02-10T11:39:38.665 INFO:tasks.ceph.mds.b.gibba020.stdout:starting mds.b at 2021-02-10T11:39:38.668 INFO:tasks.ceph.mds.a.gibba012.stdout:starting mds.a at 2021-02-10T11:39:38.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:38.680 7f7ee3856700 1 Processor -- start 2021-02-10T11:39:38.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:38.680 7f7ee3856700 1 -- start start 2021-02-10T11:39:38.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:38.681 7f7ee3856700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f7edc0d1870 0x7f7edc1b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:38.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:38.681 7f7ee3856700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7edc0f0650 0x7f7edc0cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:38.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:38.681 7f7ee3856700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7edc0f0b10 con 0x7f7edc1b0a10 2021-02-10T11:39:38.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:38.681 7f7ee3856700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7edc0e5ed0 con 0x7f7edc0d1870 2021-02-10T11:39:38.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:38.681 7f7ee3856700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7edc0d1c70 con 0x7f7edc0f0650 2021-02-10T11:39:38.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:38.681 7f7ee0e71700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f7edc1b0a10 0x7f7edc0cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:53370/0 (socket says 172.21.2.112:53370) 2021-02-10T11:39:38.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:38.681 7f7ee0e71700 1 -- 172.21.2.112:0/687333110 learned_addr learned my addr 172.21.2.112:0/687333110 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:39:38.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:38.681 7f7ee1672700 1 --2- 172.21.2.112:0/687333110 >> v2:172.21.2.112:3300/0 conn(0x7f7edc0d1870 0x7f7edc1b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:38.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:38.681 7f7ee1e73700 1 --2- 172.21.2.112:0/687333110 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7edc0f0650 0x7f7edc0cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:38.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:38.682 7f7ed3fff700 1 -- 172.21.2.112:0/687333110 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4123175818 0 0) 0x7f7ecc0009b0 con 0x7f7edc1b0a10 2021-02-10T11:39:38.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:38.682 7f7ed3fff700 1 -- 172.21.2.112:0/687333110 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ec8002fe0 con 0x7f7edc1b0a10 2021-02-10T11:39:38.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:38.682 7f7ee1672700 1 -- 172.21.2.112:0/687333110 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7edc0f0650 msgr2=0x7f7edc0cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:38.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:38.682 7f7ee1672700 1 --2- 172.21.2.112:0/687333110 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7edc0f0650 0x7f7edc0cfb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:38.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:38.682 7f7ee1672700 1 -- 172.21.2.112:0/687333110 >> v1:172.21.2.112:6789/0 conn(0x7f7edc1b0a10 legacy=0x7f7edc0cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:38.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:38.682 7f7ee1672700 1 -- 172.21.2.112:0/687333110 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7edc0d1e90 con 0x7f7edc0d1870 2021-02-10T11:39:38.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:38.682 7f7ee1672700 1 --2- 172.21.2.112:0/687333110 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7edc0d1870 0x7f7edc1b03f0 secure :-1 s=READY pgs=1946 cs=0 l=1 rev1=1 rx=0x7f7ed40069d0 tx=0x7f7ed4009e40).ready entity=mon.0 client_cookie=caa9d8d6ff121385 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:38.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:38.682 7f7ed3fff700 1 -- 172.21.2.112:0/687333110 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7ed400b510 con 0x7f7edc0d1870 2021-02-10T11:39:38.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:38.682 7f7ed3fff700 1 -- 172.21.2.112:0/687333110 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ed400b890 con 0x7f7edc0d1870 2021-02-10T11:39:38.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:38.682 7f7ed3fff700 1 -- 172.21.2.112:0/687333110 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7ed400bdb0 con 0x7f7edc0d1870 2021-02-10T11:39:38.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:38.682 7f7ee3856700 1 -- 172.21.2.112:0/687333110 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7edc0d1870 msgr2=0x7f7edc1b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:38.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:38.682 7f7ee3856700 1 --2- 172.21.2.112:0/687333110 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7edc0d1870 0x7f7edc1b03f0 secure :-1 s=READY pgs=1946 cs=0 l=1 rev1=1 rx=0x7f7ed40069d0 tx=0x7f7ed4009e40).stop 2021-02-10T11:39:38.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:38.683 7f7ee3856700 1 -- 172.21.2.112:0/687333110 shutdown_connections 2021-02-10T11:39:38.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:38.683 7f7ee3856700 1 --2- 172.21.2.112:0/687333110 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7edc0f0650 0x7f7edc0cfb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:38.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:38.683 7f7ee3856700 1 --2- 172.21.2.112:0/687333110 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7edc0d1870 0x7f7edc1b03f0 unknown :-1 s=CLOSED pgs=1946 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:38.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:38.683 7f7ee3856700 1 -- 172.21.2.112:0/687333110 >> 172.21.2.112:0/687333110 conn(0x7f7edc0ce450 msgr2=0x7f7edc0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:38.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:38.683 7f7ee3856700 1 -- 172.21.2.112:0/687333110 shutdown_connections 2021-02-10T11:39:38.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:38.683 7f7ee3856700 1 -- 172.21.2.112:0/687333110 wait complete. 2021-02-10T11:39:38.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:38.698 7f7ee3856700 1 Processor -- start 2021-02-10T11:39:38.695 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:38.698 7f7ee3856700 1 -- start start 2021-02-10T11:39:38.695 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:38.698 7f7ee3856700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7edc19ffa0 0x7f7edc0eda10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:38.695 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:38.698 7f7ee3856700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7edc1a03a0 0x7f7edc0ee030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:38.695 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:38.698 7f7ee3856700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7edc0ee650 0x7f7edc0ebe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:38.695 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:38.698 7f7ee3856700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7edc0ed490 con 0x7f7edc19ffa0 2021-02-10T11:39:38.695 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:38.698 7f7ee3856700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7edc0eea50 con 0x7f7edc1a03a0 2021-02-10T11:39:38.695 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:38.698 7f7ee3856700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7edc0eec30 con 0x7f7edc0ee650 2021-02-10T11:39:38.695 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:38.698 7f7ee1672700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7edc19ffa0 0x7f7edc0eda10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:38.695 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:38.698 7f7ee1672700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7edc19ffa0 0x7f7edc0eda10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:53914/0 (socket says 172.21.2.112:53914) 2021-02-10T11:39:38.695 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:38.698 7f7ee1672700 1 -- 172.21.2.112:0/1003398769 learned_addr learned my addr 172.21.2.112:0/1003398769 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:39:38.695 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:38.698 7f7ee0e71700 1 --2- 172.21.2.112:0/1003398769 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7edc1a03a0 0x7f7edc0ee030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:38.695 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:38.698 7f7ee1e73700 1 --2- 172.21.2.112:0/1003398769 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7edc0ee650 0x7f7edc0ebe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:38.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:38.699 7f7ee1672700 1 -- 172.21.2.112:0/1003398769 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7edc0ee650 msgr2=0x7f7edc0ebe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:38.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:38.699 7f7ee1672700 1 --2- 172.21.2.112:0/1003398769 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7edc0ee650 0x7f7edc0ebe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:38.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:38.699 7f7ee1672700 1 -- 172.21.2.112:0/1003398769 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7edc1a03a0 msgr2=0x7f7edc0ee030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:38.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:38.699 7f7ee1672700 1 --2- 172.21.2.112:0/1003398769 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7edc1a03a0 0x7f7edc0ee030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:38.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:38.699 7f7ee1672700 1 -- 172.21.2.112:0/1003398769 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7edc0ec4a0 con 0x7f7edc19ffa0 2021-02-10T11:39:38.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:38.699 7f7ee1672700 1 --2- 172.21.2.112:0/1003398769 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7edc19ffa0 0x7f7edc0eda10 secure :-1 s=READY pgs=1947 cs=0 l=1 rev1=1 rx=0x7f7ed40099e0 tx=0x7f7ed400bd20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:38.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:38.699 7f7ed1ffb700 1 -- 172.21.2.112:0/1003398769 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7ed4006240 con 0x7f7edc19ffa0 2021-02-10T11:39:38.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:38.699 7f7ed1ffb700 1 -- 172.21.2.112:0/1003398769 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ed400be60 con 0x7f7edc19ffa0 2021-02-10T11:39:38.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:38.699 7f7ed1ffb700 1 -- 172.21.2.112:0/1003398769 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7ed4005660 con 0x7f7edc19ffa0 2021-02-10T11:39:38.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:38.699 7f7ee3856700 1 -- 172.21.2.112:0/1003398769 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7edc0ec4a0 con 0x7f7edc19ffa0 2021-02-10T11:39:38.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:38.699 7f7ee3856700 1 -- 172.21.2.112:0/1003398769 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7edc0eca30 con 0x7f7edc19ffa0 2021-02-10T11:39:38.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:38.700 7f7ed1ffb700 1 -- 172.21.2.112:0/1003398769 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7ed40064c0 con 0x7f7edc19ffa0 2021-02-10T11:39:38.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:38.700 7f7ed1ffb700 1 --2- 172.21.2.112:0/1003398769 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7ebc0186c0 0x7f7ebc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:38.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:38.700 7f7ed1ffb700 1 -- 172.21.2.112:0/1003398769 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 4839+0+0 (secure 0 0 0) 0x7f7ed40325e0 con 0x7f7edc19ffa0 2021-02-10T11:39:38.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:38.700 7f7ee0e71700 1 --2- 172.21.2.112:0/1003398769 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7ebc0186c0 0x7f7ebc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:38.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:38.700 7f7ee0e71700 1 --2- 172.21.2.112:0/1003398769 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7ebc0186c0 0x7f7ebc01abc0 crc :-1 s=READY pgs=907 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:38.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:38.730 7f7ee3856700 1 -- 172.21.2.112:0/1003398769 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7edc0ebc50 con 0x7f7edc19ffa0 2021-02-10T11:39:38.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:38.732 7f7ed1ffb700 1 -- 172.21.2.112:0/1003398769 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7ed4000d30 con 0x7f7edc19ffa0 2021-02-10T11:39:38.866 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:38.872 7f7ee3856700 1 -- 172.21.2.112:0/1003398769 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7f7edc0ebc50 con 0x7f7edc19ffa0 2021-02-10T11:39:39.178 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:39.184 7f7ed1ffb700 1 -- 172.21.2.112:0/1003398769 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v96) v1 ==== 97+0+0 (secure 0 0 0) 0x7f7ed40064c0 con 0x7f7edc19ffa0 2021-02-10T11:39:39.182 INFO:teuthology.orchestra.run.gibba012.stderr: removed all blacklist entries 2021-02-10T11:39:39.184 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:39.190 7f7ee3856700 1 -- 172.21.2.112:0/1003398769 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7ebc0186c0 msgr2=0x7f7ebc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:39.184 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:39.190 7f7ee3856700 1 --2- 172.21.2.112:0/1003398769 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7ebc0186c0 0x7f7ebc01abc0 crc :-1 s=READY pgs=907 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:39.184 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:39.190 7f7ee3856700 1 -- 172.21.2.112:0/1003398769 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7edc19ffa0 msgr2=0x7f7edc0eda10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:39.184 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:39.190 7f7ee3856700 1 --2- 172.21.2.112:0/1003398769 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7edc19ffa0 0x7f7edc0eda10 secure :-1 s=READY pgs=1947 cs=0 l=1 rev1=1 rx=0x7f7ed40099e0 tx=0x7f7ed400bd20).stop 2021-02-10T11:39:39.184 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:39.190 7f7ee3856700 1 -- 172.21.2.112:0/1003398769 shutdown_connections 2021-02-10T11:39:39.184 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:39.190 7f7ee3856700 1 --2- 172.21.2.112:0/1003398769 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7ebc0186c0 0x7f7ebc01abc0 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:39.185 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:39.190 7f7ee3856700 1 --2- 172.21.2.112:0/1003398769 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7edc1a03a0 0x7f7edc0ee030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:39.185 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:39.190 7f7ee3856700 1 --2- 172.21.2.112:0/1003398769 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7edc0ee650 0x7f7edc0ebe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:39.185 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:39.190 7f7ee3856700 1 --2- 172.21.2.112:0/1003398769 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7edc19ffa0 0x7f7edc0eda10 unknown :-1 s=CLOSED pgs=1947 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:39.185 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:39.190 7f7ee3856700 1 -- 172.21.2.112:0/1003398769 >> 172.21.2.112:0/1003398769 conn(0x7f7edc094d90 msgr2=0x7f7edc08a420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:39.185 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:39.191 7f7ee3856700 1 -- 172.21.2.112:0/1003398769 shutdown_connections 2021-02-10T11:39:39.185 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:39.191 7f7ee3856700 1 -- 172.21.2.112:0/1003398769 wait complete. 2021-02-10T11:39:39.193 INFO:tasks.cephfs.cephfs_test_case:['0', '1', '2', '3'] 2021-02-10T11:39:39.193 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth ls --format=json-pretty 2021-02-10T11:39:39.620 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:39.609 7f27023ac700 1 Processor -- start 2021-02-10T11:39:39.620 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:39.609 7f27023ac700 1 -- start start 2021-02-10T11:39:39.620 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:39.609 7f27023ac700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f26fc0d0340 0x7f26fc0ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:39.620 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:39.609 7f27023ac700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f26fc0d1c70 0x7f26fc0efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:39.620 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:39.609 7f27023ac700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26fc0f0b10 con 0x7f26fc0d1870 2021-02-10T11:39:39.620 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:39.609 7f27023ac700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26fc0e5ed0 con 0x7f26fc0d0340 2021-02-10T11:39:39.620 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:39.609 7f27023ac700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26fc1b0be0 con 0x7f26fc0d1c70 2021-02-10T11:39:39.620 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:39.610 7f26fbfff700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f26fc0d1870 0x7f26fc1b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:53384/0 (socket says 172.21.2.112:53384) 2021-02-10T11:39:39.621 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:39.610 7f26fbfff700 1 -- 172.21.2.112:0/987326790 learned_addr learned my addr 172.21.2.112:0/987326790 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:39:39.621 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:39.610 7f26faffd700 1 -- 172.21.2.112:0/987326790 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1346104076 0 0) 0x7f26ec0010b0 con 0x7f26fc0d1870 2021-02-10T11:39:39.621 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:39.610 7f26faffd700 1 -- 172.21.2.112:0/987326790 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26e4002fe0 con 0x7f26fc0d1870 2021-02-10T11:39:39.621 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:39.610 7f26fb7fe700 1 --2- 172.21.2.112:0/987326790 >> v2:172.21.2.120:3300/0 conn(0x7f26fc0d0340 0x7f26fc0ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:39.621 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:39.610 7f27009c9700 1 --2- 172.21.2.112:0/987326790 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f26fc0d1c70 0x7f26fc0efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:39.621 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:39.610 7f26faffd700 1 -- 172.21.2.112:0/987326790 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3131562555 0 0) 0x7f26ec0010b0 con 0x7f26fc0d1870 2021-02-10T11:39:39.621 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:39.610 7f26faffd700 1 -- 172.21.2.112:0/987326790 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f26fc0d1c70 msgr2=0x7f26fc0efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:39.621 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:39.610 7f26faffd700 1 --2- 172.21.2.112:0/987326790 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f26fc0d1c70 0x7f26fc0efc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:39.621 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:39.610 7f26faffd700 1 -- 172.21.2.112:0/987326790 >> v2:172.21.2.120:3300/0 conn(0x7f26fc0d0340 msgr2=0x7f26fc0ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:39.621 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:39.610 7f26faffd700 1 --2- 172.21.2.112:0/987326790 >> v2:172.21.2.120:3300/0 conn(0x7f26fc0d0340 0x7f26fc0ef640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:39.621 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:39.610 7f26faffd700 1 -- 172.21.2.112:0/987326790 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26fc0f20a0 con 0x7f26fc0d1870 2021-02-10T11:39:39.621 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:39.610 7f26faffd700 1 -- 172.21.2.112:0/987326790 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f26ec0015b0 con 0x7f26fc0d1870 2021-02-10T11:39:39.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:39.610 7f26faffd700 1 -- 172.21.2.112:0/987326790 >> v1:172.21.2.112:6789/0 conn(0x7f26fc0d1870 legacy=0x7f26fc1b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:39.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:39.610 7f26faffd700 1 --2- 172.21.2.112:0/987326790 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f26e4003ae0 0x7f26e4005f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:39.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:39.610 7f26faffd700 1 --2- 172.21.2.112:0/987326790 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f26e40067d0 0x7f26e4008cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:39.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:39.610 7f26faffd700 1 --2- 172.21.2.112:0/987326790 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f26e4009590 0x7f26e400bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:39.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:39.611 7f26faffd700 1 -- 172.21.2.112:0/987326790 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26e4050370 con 0x7f26e40067d0 2021-02-10T11:39:39.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:39.611 7f26faffd700 1 -- 172.21.2.112:0/987326790 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26e4050550 con 0x7f26e4009590 2021-02-10T11:39:39.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:39.611 7f26faffd700 1 -- 172.21.2.112:0/987326790 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26e4050770 con 0x7f26e4003ae0 2021-02-10T11:39:39.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:39.611 7f27009c9700 1 --2- 172.21.2.112:0/987326790 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f26fc0d1c70 0x7f26fc0efc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:39:39.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:39.611 7f26fb7fe700 1 --2- 172.21.2.112:0/987326790 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f26e40067d0 0x7f26e4008cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:39.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:39.611 7f26fbfff700 1 --2- 172.21.2.112:0/987326790 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f26e4003ae0 0x7f26e4005f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:39.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:39.611 7f27009c9700 1 --2- 172.21.2.112:0/987326790 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f26e4009590 0x7f26e400bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:39.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:39.611 7f26fb7fe700 1 -- 172.21.2.112:0/987326790 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f26e4003ae0 msgr2=0x7f26e4005f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:39.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:39.611 7f26fb7fe700 1 --2- 172.21.2.112:0/987326790 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f26e4003ae0 0x7f26e4005f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:39.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:39.611 7f26fb7fe700 1 -- 172.21.2.112:0/987326790 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f26e4009590 msgr2=0x7f26e400bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:39.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:39.611 7f26fb7fe700 1 --2- 172.21.2.112:0/987326790 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f26e4009590 0x7f26e400bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:39.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:39.611 7f26fb7fe700 1 -- 172.21.2.112:0/987326790 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26e4050a10 con 0x7f26e40067d0 2021-02-10T11:39:39.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:39.611 7f26fb7fe700 1 --2- 172.21.2.112:0/987326790 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f26e40067d0 0x7f26e4008cf0 secure :-1 s=READY pgs=1949 cs=0 l=1 rev1=1 rx=0x7f26f0006360 tx=0x7f26f0009c30).ready entity=mon.0 client_cookie=235ebf55354c7526 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:39.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:39.611 7f26faffd700 1 -- 172.21.2.112:0/987326790 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f26f00009b0 con 0x7f26e40067d0 2021-02-10T11:39:39.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:39.611 7f26faffd700 1 -- 172.21.2.112:0/987326790 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26f0000d20 con 0x7f26e40067d0 2021-02-10T11:39:39.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:39.611 7f26faffd700 1 -- 172.21.2.112:0/987326790 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f26f0002220 con 0x7f26e40067d0 2021-02-10T11:39:39.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:39.611 7f27023ac700 1 -- 172.21.2.112:0/987326790 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f26e40067d0 msgr2=0x7f26e4008cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:39.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:39.611 7f27023ac700 1 --2- 172.21.2.112:0/987326790 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f26e40067d0 0x7f26e4008cf0 secure :-1 s=READY pgs=1949 cs=0 l=1 rev1=1 rx=0x7f26f0006360 tx=0x7f26f0009c30).stop 2021-02-10T11:39:39.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:39.611 7f27009c9700 1 -- 172.21.2.112:0/987326790 reap_dead start 2021-02-10T11:39:39.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:39.612 7f27023ac700 1 -- 172.21.2.112:0/987326790 shutdown_connections 2021-02-10T11:39:39.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:39.612 7f27023ac700 1 -- 172.21.2.112:0/987326790 >> 172.21.2.112:0/987326790 conn(0x7f26fc0ce450 msgr2=0x7f26fc0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:39.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:39.612 7f27023ac700 1 -- 172.21.2.112:0/987326790 shutdown_connections 2021-02-10T11:39:39.624 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:39.612 7f27023ac700 1 -- 172.21.2.112:0/987326790 wait complete. 2021-02-10T11:39:39.624 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:39.626 7f27023ac700 1 Processor -- start 2021-02-10T11:39:39.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:39.627 7f27023ac700 1 -- start start 2021-02-10T11:39:39.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:39.627 7f27023ac700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f26fc1942b0 0x7f26fc0b3550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:39.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:39.627 7f27023ac700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f26fc195070 0x7f26fc195470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:39.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:39.627 7f27023ac700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f26fc0bb8e0 0x7f26fc0bbce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:39.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:39.627 7f27023ac700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26fc1946b0 con 0x7f26fc0bb8e0 2021-02-10T11:39:39.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:39.627 7f27023ac700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26fc0b3b70 con 0x7f26fc195070 2021-02-10T11:39:39.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:39.627 7f27023ac700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26fc0b3d90 con 0x7f26fc1942b0 2021-02-10T11:39:39.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:39.627 7f27009c9700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f26fc0bb8e0 0x7f26fc0bbce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:39.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:39.627 7f27009c9700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f26fc0bb8e0 0x7f26fc0bbce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:53938/0 (socket says 172.21.2.112:53938) 2021-02-10T11:39:39.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:39.627 7f27009c9700 1 -- 172.21.2.112:0/2009310430 learned_addr learned my addr 172.21.2.112:0/2009310430 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:39:39.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:39.628 7f27009c9700 1 -- 172.21.2.112:0/2009310430 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f26fc1942b0 msgr2=0x7f26fc0b3550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:39.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:39.628 7f27009c9700 1 --2- 172.21.2.112:0/2009310430 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f26fc1942b0 0x7f26fc0b3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:39.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:39.628 7f27009c9700 1 -- 172.21.2.112:0/2009310430 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f26fc195070 msgr2=0x7f26fc195470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:39.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:39.628 7f27009c9700 1 --2- 172.21.2.112:0/2009310430 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f26fc195070 0x7f26fc195470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:39.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:39.628 7f27009c9700 1 -- 172.21.2.112:0/2009310430 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26fc1a49e0 con 0x7f26fc0bb8e0 2021-02-10T11:39:39.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:39.628 7f27009c9700 1 --2- 172.21.2.112:0/2009310430 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f26fc0bb8e0 0x7f26fc0bbce0 secure :-1 s=READY pgs=1950 cs=0 l=1 rev1=1 rx=0x7f26f4009940 tx=0x7f26f400bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:39.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:39.628 7f26f8ff9700 1 -- 172.21.2.112:0/2009310430 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f26f4006030 con 0x7f26fc0bb8e0 2021-02-10T11:39:39.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:39.628 7f26f8ff9700 1 -- 172.21.2.112:0/2009310430 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26f4006330 con 0x7f26fc0bb8e0 2021-02-10T11:39:39.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:39.628 7f26f8ff9700 1 -- 172.21.2.112:0/2009310430 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f26f4005930 con 0x7f26fc0bb8e0 2021-02-10T11:39:39.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:39.628 7f27023ac700 1 -- 172.21.2.112:0/2009310430 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26fc1a49e0 con 0x7f26fc0bb8e0 2021-02-10T11:39:39.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:39.628 7f27023ac700 1 -- 172.21.2.112:0/2009310430 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26fc0bc5f0 con 0x7f26fc0bb8e0 2021-02-10T11:39:39.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:39.629 7f26f8ff9700 1 -- 172.21.2.112:0/2009310430 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f26f4006030 con 0x7f26fc0bb8e0 2021-02-10T11:39:39.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:39.629 7f26f8ff9700 1 --2- 172.21.2.112:0/2009310430 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f26dc0186c0 0x7f26dc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:39.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:39.629 7f26f8ff9700 1 -- 172.21.2.112:0/2009310430 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f26f40325e0 con 0x7f26fc0bb8e0 2021-02-10T11:39:39.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:39.629 7f26fbfff700 1 --2- 172.21.2.112:0/2009310430 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f26dc0186c0 0x7f26dc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:39.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:39.630 7f26fbfff700 1 --2- 172.21.2.112:0/2009310430 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f26dc0186c0 0x7f26dc01abc0 crc :-1 s=READY pgs=908 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:39.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:39.659 7f27023ac700 1 -- 172.21.2.112:0/2009310430 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26fc0bc5f0 con 0x7f26fc0bb8e0 2021-02-10T11:39:39.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:39.662 7f26f8ff9700 1 -- 172.21.2.112:0/2009310430 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f26f4000df0 con 0x7f26fc0bb8e0 2021-02-10T11:39:39.790 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:39.796 7f27023ac700 1 -- 172.21.2.112:0/2009310430 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f26fc0bc5f0 con 0x7f26fc0bb8e0 2021-02-10T11:39:39.790 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:39.796 7f26f8ff9700 1 -- 172.21.2.112:0/2009310430 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v60) v1 ==== 80+0+4513 (secure 0 0 0) 0x7f26f4006030 con 0x7f26fc0bb8e0 2021-02-10T11:39:39.790 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:39:39.791 INFO:teuthology.orchestra.run.gibba012.stdout:{ 2021-02-10T11:39:39.791 INFO:teuthology.orchestra.run.gibba012.stdout: "auth_dump": [ 2021-02-10T11:39:39.791 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:39:39.791 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "mds.a", 2021-02-10T11:39:39.791 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCKvyNgPkV4AxAAEmVNaghdccP6lWd8hvQnxw==", 2021-02-10T11:39:39.791 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:39:39.791 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:39:39.791 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:39:39.791 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:39:39.791 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:39:39.791 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:39:39.791 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:39:39.792 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:39:39.792 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "mds.b", 2021-02-10T11:39:39.792 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCKvyNguxONLhAAHIpry+R89MAwdhFJyEhVCw==", 2021-02-10T11:39:39.792 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:39:39.792 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:39:39.792 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:39:39.792 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:39:39.792 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:39:39.792 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:39:39.792 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:39:39.792 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:39:39.792 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.0", 2021-02-10T11:39:39.792 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCSvyNgXcKPLhAAP/Bj2F+06DCtPCzQDNq+cQ==", 2021-02-10T11:39:39.793 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:39:39.793 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:39:39.793 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:39:39.793 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:39:39.793 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:39:39.793 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:39:39.793 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:39:39.793 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.1", 2021-02-10T11:39:39.793 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCVvyNghfbcLhAA66PrRwR2M7o6VTF5UhU/Fg==", 2021-02-10T11:39:39.793 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:39:39.793 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:39:39.793 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:39:39.793 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:39:39.794 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:39:39.794 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:39:39.794 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:39:39.794 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.2", 2021-02-10T11:39:39.794 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCYvyNgwiPqLhAA66BVkRsgaUlGpAsQ15Nf8Q==", 2021-02-10T11:39:39.794 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:39:39.794 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:39:39.794 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:39:39.794 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:39:39.794 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:39:39.794 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:39:39.794 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:39:39.794 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.3", 2021-02-10T11:39:39.795 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCbvyNgqBZrLxAAERPe/1L3SXAEzI3d9ReHHg==", 2021-02-10T11:39:39.795 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:39:39.795 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:39:39.795 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:39:39.795 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:39:39.795 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:39:39.795 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:39:39.795 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:39:39.795 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.4", 2021-02-10T11:39:39.795 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCkvyNgDQsxIhAAHmVjVYnx20N4SDmKqvbcuA==", 2021-02-10T11:39:39.795 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:39:39.795 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:39:39.795 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:39:39.795 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:39:39.796 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:39:39.796 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:39:39.796 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:39:39.796 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.5", 2021-02-10T11:39:39.796 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCnvyNg8ITFIhAADd4aKybAV7btoGPx3kCg3A==", 2021-02-10T11:39:39.796 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:39:39.796 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:39:39.796 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:39:39.796 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:39:39.796 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:39:39.796 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:39:39.796 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:39:39.796 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.6", 2021-02-10T11:39:39.797 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCqvyNgxOhDIxAAzXixEa/nlDc1xXrg0aeLwg==", 2021-02-10T11:39:39.797 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:39:39.797 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:39:39.797 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:39:39.797 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:39:39.797 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:39:39.797 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:39:39.797 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:39:39.797 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.7", 2021-02-10T11:39:39.797 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCtvyNgxSqGIxAACxJQlwP7DpiVVCZHA3zn1A==", 2021-02-10T11:39:39.797 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:39:39.797 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:39:39.797 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:39:39.798 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:39:39.798 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:39:39.798 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:39:39.798 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:39:39.798 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.0", 2021-02-10T11:39:39.798 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCLvyNgDkNiHRAAygU8Za4PBusTqCgwAYWTiQ==", 2021-02-10T11:39:39.798 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:39:39.798 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:39:39.798 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow r", 2021-02-10T11:39:39.798 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:39:39.798 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:39:39.798 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:39:39.798 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:39:39.799 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.1", 2021-02-10T11:39:39.799 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCLvyNgxmLqMxAA1MlSDJ9LH9u4f2sL7mm8/Q==", 2021-02-10T11:39:39.799 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:39:39.799 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:39:39.799 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow r", 2021-02-10T11:39:39.799 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:39:39.799 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:39:39.799 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:39:39.799 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:39:39.799 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.2", 2021-02-10T11:39:39.799 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCMvyNgp8veDhAAniSVHYPtwwTog/pjbaRQmw==", 2021-02-10T11:39:39.799 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:39:39.799 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:39:39.800 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow r", 2021-02-10T11:39:39.800 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:39:39.800 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:39:39.800 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:39:39.800 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:39:39.800 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.3", 2021-02-10T11:39:39.800 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCMvyNgQEVoJRAApHcWdm1sldIxu+heyD4HdQ==", 2021-02-10T11:39:39.800 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:39:39.800 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:39:39.800 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow r", 2021-02-10T11:39:39.800 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:39:39.800 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:39:39.800 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:39:39.800 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:39:39.801 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.admin", 2021-02-10T11:39:39.801 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCGvyNg7jjJBRAAtot+ypUZOiTo1GbY+oVhVw==", 2021-02-10T11:39:39.801 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:39:39.801 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow *", 2021-02-10T11:39:39.801 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:39:39.801 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:39:39.801 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:39:39.801 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:39:39.801 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:39:39.801 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:39:39.801 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "mgr.x", 2021-02-10T11:39:39.801 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCJvyNgTPVwKBAAjduiBE8+RSXBv82tdPecWg==", 2021-02-10T11:39:39.801 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:39:39.802 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow *", 2021-02-10T11:39:39.802 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow profile mgr", 2021-02-10T11:39:39.802 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:39:39.802 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:39:39.802 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:39:39.802 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:39:39.802 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "mgr.y", 2021-02-10T11:39:39.802 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCJvyNgQ1tTDhAASwBssDy+Rz31ujw8fNlWow==", 2021-02-10T11:39:39.802 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:39:39.802 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow *", 2021-02-10T11:39:39.802 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow profile mgr", 2021-02-10T11:39:39.802 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:39:39.802 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:39:39.803 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:39:39.803 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:39:39.803 INFO:teuthology.orchestra.run.gibba012.stdout:} 2021-02-10T11:39:39.804 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:39.798 7f27023ac700 1 -- 172.21.2.112:0/2009310430 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f26dc0186c0 msgr2=0x7f26dc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:39.804 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:39.798 7f27023ac700 1 --2- 172.21.2.112:0/2009310430 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f26dc0186c0 0x7f26dc01abc0 crc :-1 s=READY pgs=908 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:39.804 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:39.798 7f27023ac700 1 -- 172.21.2.112:0/2009310430 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f26fc0bb8e0 msgr2=0x7f26fc0bbce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:39.804 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:39.798 7f27023ac700 1 --2- 172.21.2.112:0/2009310430 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f26fc0bb8e0 0x7f26fc0bbce0 secure :-1 s=READY pgs=1950 cs=0 l=1 rev1=1 rx=0x7f26f4009940 tx=0x7f26f400bcf0).stop 2021-02-10T11:39:39.804 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:39.798 7f27023ac700 1 -- 172.21.2.112:0/2009310430 shutdown_connections 2021-02-10T11:39:39.804 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:39.798 7f27023ac700 1 --2- 172.21.2.112:0/2009310430 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f26dc0186c0 0x7f26dc01abc0 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:39.804 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:39.798 7f27023ac700 1 --2- 172.21.2.112:0/2009310430 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f26fc195070 0x7f26fc195470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:39.804 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:39.798 7f27023ac700 1 --2- 172.21.2.112:0/2009310430 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f26fc0bb8e0 0x7f26fc0bbce0 unknown :-1 s=CLOSED pgs=1950 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:39.804 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:39.798 7f27023ac700 1 --2- 172.21.2.112:0/2009310430 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f26fc1942b0 0x7f26fc0b3550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:39.804 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:39.798 7f27023ac700 1 -- 172.21.2.112:0/2009310430 >> 172.21.2.112:0/2009310430 conn(0x7f26fc0b3040 msgr2=0x7f26fc0c3250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:39.805 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:39.799 7f27023ac700 1 -- 172.21.2.112:0/2009310430 shutdown_connections 2021-02-10T11:39:39.805 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:39.799 7f27023ac700 1 -- 172.21.2.112:0/2009310430 wait complete. 2021-02-10T11:39:39.805 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:39:40.238 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.222 7fe97d7a9700 1 Processor -- start 2021-02-10T11:39:40.238 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.223 7fe97d7a9700 1 -- start start 2021-02-10T11:39:40.238 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.223 7fe97d7a9700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe9780d1c70 0x7fe9780f20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:40.238 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.223 7fe97d7a9700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9780f0b10 con 0x7fe9780d1870 2021-02-10T11:39:40.239 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.223 7fe97d7a9700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9781b0be0 con 0x7fe9780d0340 2021-02-10T11:39:40.239 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.223 7fe97d7a9700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9780e5ed0 con 0x7fe9780d1c70 2021-02-10T11:39:40.239 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.223 7fe977574700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fe9780d1870 0x7fe9781b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:53404/0 (socket says 172.21.2.112:53404) 2021-02-10T11:39:40.239 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.223 7fe977574700 1 -- 172.21.2.112:0/3124259060 learned_addr learned my addr 172.21.2.112:0/3124259060 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:39:40.239 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.223 7fe977d75700 1 --2- 172.21.2.112:0/3124259060 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe9780d1c70 0x7fe9780f20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:40.239 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.224 7fe976572700 1 -- 172.21.2.112:0/3124259060 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 155508076 0 0) 0x7fe9680010b0 con 0x7fe9780d1870 2021-02-10T11:39:40.239 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.224 7fe976572700 1 -- 172.21.2.112:0/3124259060 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe964002fe0 con 0x7fe9780d1870 2021-02-10T11:39:40.240 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.224 7fe976572700 1 -- 172.21.2.112:0/3124259060 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3852696130 0 0) 0x7fe9680010b0 con 0x7fe9780d1870 2021-02-10T11:39:40.240 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.224 7fe976572700 1 -- 172.21.2.112:0/3124259060 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe9780d1c70 msgr2=0x7fe9780f20a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:40.240 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.224 7fe976572700 1 --2- 172.21.2.112:0/3124259060 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe9780d1c70 0x7fe9780f20a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:40.240 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.224 7fe976572700 1 -- 172.21.2.112:0/3124259060 >> v1:172.21.2.120:6789/0 conn(0x7fe9780d0340 legacy=0x7fe9780ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:40.240 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.224 7fe976572700 1 -- 172.21.2.112:0/3124259060 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9780efe30 con 0x7fe9780d1870 2021-02-10T11:39:40.240 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.224 7fe976572700 1 -- 172.21.2.112:0/3124259060 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7fe9680015b0 con 0x7fe9780d1870 2021-02-10T11:39:40.240 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.224 7fe976572700 1 -- 172.21.2.112:0/3124259060 >> v1:172.21.2.112:6789/0 conn(0x7fe9780d1870 legacy=0x7fe9781b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:40.240 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.224 7fe976572700 1 --2- 172.21.2.112:0/3124259060 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe964003aa0 0x7fe964005f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:40.241 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.224 7fe976572700 1 --2- 172.21.2.112:0/3124259060 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe964006780 0x7fe964008c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:40.241 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.224 7fe976572700 1 --2- 172.21.2.112:0/3124259060 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe964009510 0x7fe96400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:40.241 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.224 7fe976572700 1 -- 172.21.2.112:0/3124259060 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe96400c2c0 con 0x7fe964003aa0 2021-02-10T11:39:40.241 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.224 7fe976572700 1 -- 172.21.2.112:0/3124259060 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe96400c4e0 con 0x7fe964009510 2021-02-10T11:39:40.241 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.224 7fe976572700 1 -- 172.21.2.112:0/3124259060 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe964050710 con 0x7fe964006780 2021-02-10T11:39:40.241 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.224 7fe977574700 1 --2- 172.21.2.112:0/3124259060 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe964003aa0 0x7fe964005f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:40.241 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.224 7fe976d73700 1 --2- 172.21.2.112:0/3124259060 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe964006780 0x7fe964008c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:40.241 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.224 7fe977d75700 1 --2- 172.21.2.112:0/3124259060 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe964009510 0x7fe96400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:40.242 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.224 7fe977574700 1 -- 172.21.2.112:0/3124259060 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe964006780 msgr2=0x7fe964008c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:40.242 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.224 7fe977574700 1 --2- 172.21.2.112:0/3124259060 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe964006780 0x7fe964008c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:40.242 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.224 7fe977574700 1 -- 172.21.2.112:0/3124259060 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe964009510 msgr2=0x7fe96400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:40.242 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.224 7fe977574700 1 --2- 172.21.2.112:0/3124259060 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe964009510 0x7fe96400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:40.242 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.224 7fe977574700 1 -- 172.21.2.112:0/3124259060 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe964050970 con 0x7fe964003aa0 2021-02-10T11:39:40.242 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.224 7fe976d73700 1 --2- 172.21.2.112:0/3124259060 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe964006780 0x7fe964008c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:39:40.242 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.224 7fe977574700 1 --2- 172.21.2.112:0/3124259060 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe964003aa0 0x7fe964005f60 secure :-1 s=READY pgs=1952 cs=0 l=1 rev1=1 rx=0x7fe968007990 tx=0x7fe968007d30).ready entity=mon.0 client_cookie=d8e5cd2bca6e41b1 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:40.242 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.225 7fe976572700 1 -- 172.21.2.112:0/3124259060 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe9680009b0 con 0x7fe964003aa0 2021-02-10T11:39:40.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.225 7fe976572700 1 -- 172.21.2.112:0/3124259060 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe96800a310 con 0x7fe964003aa0 2021-02-10T11:39:40.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.225 7fe976572700 1 -- 172.21.2.112:0/3124259060 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe96800a830 con 0x7fe964003aa0 2021-02-10T11:39:40.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.225 7fe97d7a9700 1 -- 172.21.2.112:0/3124259060 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe964003aa0 msgr2=0x7fe964005f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:40.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.225 7fe97d7a9700 1 --2- 172.21.2.112:0/3124259060 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe964003aa0 0x7fe964005f60 secure :-1 s=READY pgs=1952 cs=0 l=1 rev1=1 rx=0x7fe968007990 tx=0x7fe968007d30).stop 2021-02-10T11:39:40.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.225 7fe977d75700 1 -- 172.21.2.112:0/3124259060 reap_dead start 2021-02-10T11:39:40.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.225 7fe97d7a9700 1 -- 172.21.2.112:0/3124259060 shutdown_connections 2021-02-10T11:39:40.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.225 7fe97d7a9700 1 -- 172.21.2.112:0/3124259060 >> 172.21.2.112:0/3124259060 conn(0x7fe9780ce450 msgr2=0x7fe9780d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:40.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.225 7fe97d7a9700 1 -- 172.21.2.112:0/3124259060 shutdown_connections 2021-02-10T11:39:40.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.225 7fe97d7a9700 1 -- 172.21.2.112:0/3124259060 wait complete. 2021-02-10T11:39:40.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.240 7fe97d7a9700 1 Processor -- start 2021-02-10T11:39:40.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.240 7fe97d7a9700 1 -- start start 2021-02-10T11:39:40.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.241 7fe97d7a9700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe9780b2860 0x7fe9781a4480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:40.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.241 7fe97d7a9700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe9781a4aa0 0x7fe9780ec620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:40.245 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.241 7fe97d7a9700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe9780ecc40 0x7fe978097790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:40.245 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.241 7fe97d7a9700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9780b2ce0 con 0x7fe9781a4aa0 2021-02-10T11:39:40.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.241 7fe97d7a9700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9780ed040 con 0x7fe9780ecc40 2021-02-10T11:39:40.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.241 7fe97d7a9700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9780ed260 con 0x7fe9780b2860 2021-02-10T11:39:40.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.241 7fe976d73700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe9781a4aa0 0x7fe9780ec620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:40.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.241 7fe976d73700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe9781a4aa0 0x7fe9780ec620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:53958/0 (socket says 172.21.2.112:53958) 2021-02-10T11:39:40.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.241 7fe976d73700 1 -- 172.21.2.112:0/2353638705 learned_addr learned my addr 172.21.2.112:0/2353638705 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:39:40.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.241 7fe976d73700 1 -- 172.21.2.112:0/2353638705 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe9780b2860 msgr2=0x7fe9781a4480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:40.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.241 7fe976d73700 1 --2- 172.21.2.112:0/2353638705 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe9780b2860 0x7fe9781a4480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:40.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.241 7fe976d73700 1 -- 172.21.2.112:0/2353638705 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe9780ecc40 msgr2=0x7fe978097790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:40.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.241 7fe976d73700 1 --2- 172.21.2.112:0/2353638705 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe9780ecc40 0x7fe978097790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:40.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.241 7fe976d73700 1 -- 172.21.2.112:0/2353638705 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe978097db0 con 0x7fe9781a4aa0 2021-02-10T11:39:40.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.241 7fe976d73700 1 --2- 172.21.2.112:0/2353638705 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe9781a4aa0 0x7fe9780ec620 secure :-1 s=READY pgs=1953 cs=0 l=1 rev1=1 rx=0x7fe96000b5b0 tx=0x7fe96000b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:40.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.241 7fe96ffff700 1 -- 172.21.2.112:0/2353638705 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe960000de0 con 0x7fe9781a4aa0 2021-02-10T11:39:40.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.241 7fe96ffff700 1 -- 172.21.2.112:0/2353638705 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe96000bd40 con 0x7fe9781a4aa0 2021-02-10T11:39:40.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.241 7fe96ffff700 1 -- 172.21.2.112:0/2353638705 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe9600054b0 con 0x7fe9781a4aa0 2021-02-10T11:39:40.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.241 7fe97d7a9700 1 -- 172.21.2.112:0/2353638705 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe978097db0 con 0x7fe9781a4aa0 2021-02-10T11:39:40.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.241 7fe97d7a9700 1 -- 172.21.2.112:0/2353638705 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe978097db0 con 0x7fe9781a4aa0 2021-02-10T11:39:40.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.242 7fe96ffff700 1 -- 172.21.2.112:0/2353638705 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe960006920 con 0x7fe9781a4aa0 2021-02-10T11:39:40.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.242 7fe96ffff700 1 --2- 172.21.2.112:0/2353638705 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe95c018730 0x7fe95c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:40.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.242 7fe96ffff700 1 -- 172.21.2.112:0/2353638705 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 4753+0+0 (secure 0 0 0) 0x7fe960032350 con 0x7fe9781a4aa0 2021-02-10T11:39:40.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.242 7fe977574700 1 --2- 172.21.2.112:0/2353638705 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe95c018730 0x7fe95c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:40.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.243 7fe977574700 1 --2- 172.21.2.112:0/2353638705 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe95c018730 0x7fe95c01abc0 crc :-1 s=READY pgs=909 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:40.266 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.273 7fe97d7a9700 1 -- 172.21.2.112:0/2353638705 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9780f0b10 con 0x7fe9781a4aa0 2021-02-10T11:39:40.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.275 7fe96ffff700 1 -- 172.21.2.112:0/2353638705 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe960000d90 con 0x7fe9781a4aa0 2021-02-10T11:39:40.394 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.400 7fe97d7a9700 1 -- 172.21.2.112:0/2353638705 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe9780f0b10 con 0x7fe9781a4aa0 2021-02-10T11:39:40.395 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.401 7fe96ffff700 1 -- 172.21.2.112:0/2353638705 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v96) v1 ==== 74+0+11145 (secure 0 0 0) 0x7fe960006920 con 0x7fe9781a4aa0 2021-02-10T11:39:40.395 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:39:40.395 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":96,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:39:39.182020","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:39:40.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.402 7fe97d7a9700 1 -- 172.21.2.112:0/2353638705 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe95c018730 msgr2=0x7fe95c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:40.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.402 7fe97d7a9700 1 --2- 172.21.2.112:0/2353638705 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe95c018730 0x7fe95c01abc0 crc :-1 s=READY pgs=909 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:40.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.402 7fe97d7a9700 1 -- 172.21.2.112:0/2353638705 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe9781a4aa0 msgr2=0x7fe9780ec620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:40.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.402 7fe97d7a9700 1 --2- 172.21.2.112:0/2353638705 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe9781a4aa0 0x7fe9780ec620 secure :-1 s=READY pgs=1953 cs=0 l=1 rev1=1 rx=0x7fe96000b5b0 tx=0x7fe96000b950).stop 2021-02-10T11:39:40.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.403 7fe97d7a9700 1 -- 172.21.2.112:0/2353638705 shutdown_connections 2021-02-10T11:39:40.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.403 7fe97d7a9700 1 --2- 172.21.2.112:0/2353638705 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe95c018730 0x7fe95c01abc0 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:40.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.403 7fe97d7a9700 1 --2- 172.21.2.112:0/2353638705 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe9780ecc40 0x7fe978097790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:40.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.403 7fe97d7a9700 1 --2- 172.21.2.112:0/2353638705 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe9781a4aa0 0x7fe9780ec620 unknown :-1 s=CLOSED pgs=1953 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:40.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.403 7fe97d7a9700 1 --2- 172.21.2.112:0/2353638705 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe9780b2860 0x7fe9781a4480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:40.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.403 7fe97d7a9700 1 -- 172.21.2.112:0/2353638705 >> 172.21.2.112:0/2353638705 conn(0x7fe9780bf500 msgr2=0x7fe9780b5050 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:40.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.403 7fe97d7a9700 1 -- 172.21.2.112:0/2353638705 shutdown_connections 2021-02-10T11:39:40.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.403 7fe97d7a9700 1 -- 172.21.2.112:0/2353638705 wait complete. 2021-02-10T11:39:40.405 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:13:57.508758', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2021-02-10T11:39:40.406 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:39:40.837 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.827 7f128759b700 1 Processor -- start 2021-02-10T11:39:40.837 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.827 7f128759b700 1 -- start start 2021-02-10T11:39:40.838 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.828 7f128759b700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f12800d1c60 0x7f12800cef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:40.838 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.828 7f128759b700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f12800cf570 0x7f12800cf970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:40.838 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.828 7f128759b700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12800fe1b0 con 0x7f12800d1860 2021-02-10T11:39:40.838 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.828 7f128759b700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12801b0ca0 con 0x7f12800cf570 2021-02-10T11:39:40.838 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.828 7f128759b700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1280109a50 con 0x7f12800d1c60 2021-02-10T11:39:40.838 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.828 7f12853b7700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f12800d1860 0x7f12801b04b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:53424/0 (socket says 172.21.2.112:53424) 2021-02-10T11:39:40.838 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.828 7f12853b7700 1 -- 172.21.2.112:0/2824368072 learned_addr learned my addr 172.21.2.112:0/2824368072 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:39:40.838 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.828 7f1285bb8700 1 --2- 172.21.2.112:0/2824368072 >> v2:172.21.2.112:3300/0 conn(0x7f12800cf570 0x7f12800cf970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:40.838 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.828 7f1285bb8700 1 -- 172.21.2.112:0/2824368072 >> v2:172.21.2.120:3300/0 conn(0x7f12800d1c60 msgr2=0x7f12800cef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:40.838 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.828 7f1285bb8700 1 --2- 172.21.2.112:0/2824368072 >> v2:172.21.2.120:3300/0 conn(0x7f12800d1c60 0x7f12800cef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:40.838 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.828 7f1285bb8700 1 -- 172.21.2.112:0/2824368072 >> v1:172.21.2.112:6789/0 conn(0x7f12800d1860 legacy=0x7f12801b04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:40.839 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.828 7f1285bb8700 1 -- 172.21.2.112:0/2824368072 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1280088070 con 0x7f12800cf570 2021-02-10T11:39:40.839 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.828 7f1285bb8700 1 --2- 172.21.2.112:0/2824368072 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f12800cf570 0x7f12800cf970 secure :-1 s=READY pgs=1955 cs=0 l=1 rev1=1 rx=0x7f1278006e30 tx=0x7f1278009e60).ready entity=mon.0 client_cookie=54350604796967c6 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:40.839 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.829 7f1273fff700 1 -- 172.21.2.112:0/2824368072 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1278006a50 con 0x7f12800cf570 2021-02-10T11:39:40.839 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.829 7f1273fff700 1 -- 172.21.2.112:0/2824368072 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1278006a50 con 0x7f12800cf570 2021-02-10T11:39:40.839 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.829 7f1273fff700 1 -- 172.21.2.112:0/2824368072 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f127800b300 con 0x7f12800cf570 2021-02-10T11:39:40.839 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.829 7f128759b700 1 -- 172.21.2.112:0/2824368072 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f12800cf570 msgr2=0x7f12800cf970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:40.839 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.829 7f128759b700 1 --2- 172.21.2.112:0/2824368072 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f12800cf570 0x7f12800cf970 secure :-1 s=READY pgs=1955 cs=0 l=1 rev1=1 rx=0x7f1278006e30 tx=0x7f1278009e60).stop 2021-02-10T11:39:40.839 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.829 7f128759b700 1 -- 172.21.2.112:0/2824368072 shutdown_connections 2021-02-10T11:39:40.839 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.829 7f128759b700 1 --2- 172.21.2.112:0/2824368072 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f12800cf570 0x7f12800cf970 unknown :-1 s=CLOSED pgs=1955 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:40.839 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.829 7f128759b700 1 --2- 172.21.2.112:0/2824368072 >> v2:172.21.2.120:3300/0 conn(0x7f12800d1c60 0x7f12800cef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:40.839 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.829 7f128759b700 1 -- 172.21.2.112:0/2824368072 >> 172.21.2.112:0/2824368072 conn(0x7f12800ce410 msgr2=0x7f12800d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:40.839 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.830 7f128759b700 1 -- 172.21.2.112:0/2824368072 shutdown_connections 2021-02-10T11:39:40.839 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.830 7f128759b700 1 -- 172.21.2.112:0/2824368072 wait complete. 2021-02-10T11:39:40.841 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.844 7f128759b700 1 Processor -- start 2021-02-10T11:39:40.841 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.844 7f128759b700 1 -- start start 2021-02-10T11:39:40.841 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.844 7f128759b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1280199ed0 0x7f12801a7c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:40.841 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.844 7f128759b700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f12801a8260 0x7f12800c1ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:40.842 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.844 7f128759b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f12800c22c0 0x7f12800c26c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:40.842 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.844 7f128759b700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12800c17a0 con 0x7f1280199ed0 2021-02-10T11:39:40.842 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.844 7f128759b700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12800b8c70 con 0x7f12801a8260 2021-02-10T11:39:40.842 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.844 7f128759b700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12800c2ce0 con 0x7f12800c22c0 2021-02-10T11:39:40.842 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.845 7f12853b7700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1280199ed0 0x7f12801a7c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:40.842 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.845 7f12853b7700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1280199ed0 0x7f12801a7c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:53968/0 (socket says 172.21.2.112:53968) 2021-02-10T11:39:40.842 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.845 7f12853b7700 1 -- 172.21.2.112:0/1566087382 learned_addr learned my addr 172.21.2.112:0/1566087382 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:39:40.842 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.845 7f12853b7700 1 -- 172.21.2.112:0/1566087382 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f12800c22c0 msgr2=0x7f12800c26c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:40.842 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.845 7f12853b7700 1 --2- 172.21.2.112:0/1566087382 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f12800c22c0 0x7f12800c26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:40.842 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.845 7f12853b7700 1 -- 172.21.2.112:0/1566087382 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f12801a8260 msgr2=0x7f12800c1ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:40.842 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.845 7f12853b7700 1 --2- 172.21.2.112:0/1566087382 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f12801a8260 0x7f12800c1ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:40.842 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.845 7f12853b7700 1 -- 172.21.2.112:0/1566087382 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12800b7200 con 0x7f1280199ed0 2021-02-10T11:39:40.842 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.845 7f12853b7700 1 --2- 172.21.2.112:0/1566087382 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1280199ed0 0x7f12801a7c40 secure :-1 s=READY pgs=1956 cs=0 l=1 rev1=1 rx=0x7f1274001c60 tx=0x7f127400a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:40.843 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.845 7f1271ffb700 1 -- 172.21.2.112:0/1566087382 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f127400acc0 con 0x7f1280199ed0 2021-02-10T11:39:40.843 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.845 7f1271ffb700 1 -- 172.21.2.112:0/1566087382 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1274004140 con 0x7f1280199ed0 2021-02-10T11:39:40.843 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.845 7f128759b700 1 -- 172.21.2.112:0/1566087382 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12800b7200 con 0x7f1280199ed0 2021-02-10T11:39:40.843 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.845 7f1271ffb700 1 -- 172.21.2.112:0/1566087382 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1274004620 con 0x7f1280199ed0 2021-02-10T11:39:40.843 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.846 7f128759b700 1 -- 172.21.2.112:0/1566087382 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12800b7200 con 0x7f1280199ed0 2021-02-10T11:39:40.843 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.846 7f1271ffb700 1 -- 172.21.2.112:0/1566087382 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1274000d90 con 0x7f1280199ed0 2021-02-10T11:39:40.843 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.846 7f1271ffb700 1 --2- 172.21.2.112:0/1566087382 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f12600186d0 0x7f126001abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:40.843 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.846 7f1271ffb700 1 -- 172.21.2.112:0/1566087382 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f1274032470 con 0x7f1280199ed0 2021-02-10T11:39:40.843 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.847 7f1284bb6700 1 --2- 172.21.2.112:0/1566087382 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f12600186d0 0x7f126001abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:40.843 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.847 7f1284bb6700 1 --2- 172.21.2.112:0/1566087382 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f12600186d0 0x7f126001abd0 crc :-1 s=READY pgs=910 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:40.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.877 7f128759b700 1 -- 172.21.2.112:0/1566087382 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12800fe1b0 con 0x7f1280199ed0 2021-02-10T11:39:40.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:40.880 7f1271ffb700 1 -- 172.21.2.112:0/1566087382 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f12740009b0 con 0x7f1280199ed0 2021-02-10T11:39:41.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:41.008 7f128759b700 1 -- 172.21.2.112:0/1566087382 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f12800b8c20 con 0x7f1280199ed0 2021-02-10T11:39:41.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:41.008 7f1271ffb700 1 -- 172.21.2.112:0/1566087382 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v96) v1 ==== 92+0+10 (secure 0 0 0) 0x7f1274000d90 con 0x7f1280199ed0 2021-02-10T11:39:41.002 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:39:41.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:41.010 7f128759b700 1 -- 172.21.2.112:0/1566087382 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f12600186d0 msgr2=0x7f126001abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:41.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:41.010 7f128759b700 1 --2- 172.21.2.112:0/1566087382 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f12600186d0 0x7f126001abd0 crc :-1 s=READY pgs=910 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:41.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:41.010 7f128759b700 1 -- 172.21.2.112:0/1566087382 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1280199ed0 msgr2=0x7f12801a7c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:41.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:41.010 7f128759b700 1 --2- 172.21.2.112:0/1566087382 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1280199ed0 0x7f12801a7c40 secure :-1 s=READY pgs=1956 cs=0 l=1 rev1=1 rx=0x7f1274001c60 tx=0x7f127400a8d0).stop 2021-02-10T11:39:41.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:41.010 7f128759b700 1 -- 172.21.2.112:0/1566087382 shutdown_connections 2021-02-10T11:39:41.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:41.010 7f128759b700 1 --2- 172.21.2.112:0/1566087382 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f12600186d0 0x7f126001abd0 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:41.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:41.010 7f128759b700 1 --2- 172.21.2.112:0/1566087382 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f12801a8260 0x7f12800c1ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:41.005 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:41.010 7f128759b700 1 --2- 172.21.2.112:0/1566087382 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f12800c22c0 0x7f12800c26c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:41.005 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:41.010 7f128759b700 1 --2- 172.21.2.112:0/1566087382 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1280199ed0 0x7f12801a7c40 unknown :-1 s=CLOSED pgs=1956 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:41.005 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:41.010 7f128759b700 1 -- 172.21.2.112:0/1566087382 >> 172.21.2.112:0/1566087382 conn(0x7f128009c130 msgr2=0x7f1280095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:41.005 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:41.010 7f128759b700 1 -- 172.21.2.112:0/1566087382 shutdown_connections 2021-02-10T11:39:41.005 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:41.010 7f128759b700 1 -- 172.21.2.112:0/1566087382 wait complete. 2021-02-10T11:39:41.013 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --format=json-pretty osd lspools 2021-02-10T11:39:41.439 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:41.429 7f661d59b700 1 Processor -- start 2021-02-10T11:39:41.439 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:41.429 7f661d59b700 1 -- start start 2021-02-10T11:39:41.439 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:41.429 7f661d59b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f66180d1870 0x7f66181b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:41.439 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:41.429 7f661d59b700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f66180f0650 0x7f66180cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:41.439 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:41.429 7f661d59b700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66180f0b10 con 0x7f66181b0a10 2021-02-10T11:39:41.440 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:41.429 7f661d59b700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66180fe230 con 0x7f66180f0650 2021-02-10T11:39:41.440 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:41.429 7f661d59b700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66180e5ed0 con 0x7f66180d1870 2021-02-10T11:39:41.440 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:41.430 7f6617ac0700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f66180f0650 0x7f66180cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:41.440 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:41.430 7f6617ac0700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f66180f0650 0x7f66180cfb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:53976/0 (socket says 172.21.2.112:53976) 2021-02-10T11:39:41.440 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:41.430 7f6617ac0700 1 -- 172.21.2.112:0/3542632137 learned_addr learned my addr 172.21.2.112:0/3542632137 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:39:41.440 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:41.430 7f6617ac0700 1 -- 172.21.2.112:0/3542632137 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f66180d1870 msgr2=0x7f66181b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:41.440 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:41.430 7f6617ac0700 1 --2- 172.21.2.112:0/3542632137 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f66180d1870 0x7f66181b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:41.440 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:41.430 7f6617ac0700 1 -- 172.21.2.112:0/3542632137 >> v1:172.21.2.120:6789/0 conn(0x7f66181b0a10 legacy=0x7f66180cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:41.440 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:41.430 7f6617ac0700 1 -- 172.21.2.112:0/3542632137 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66180f20a0 con 0x7f66180f0650 2021-02-10T11:39:41.440 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:41.430 7f6617ac0700 1 --2- 172.21.2.112:0/3542632137 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f66180f0650 0x7f66180cfb20 secure :-1 s=READY pgs=1957 cs=0 l=1 rev1=1 rx=0x7f6610006b80 tx=0x7f6610009b90).ready entity=mon.0 client_cookie=881be83e1feb3314 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:41.440 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:41.430 7f66162bd700 1 -- 172.21.2.112:0/3542632137 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f661000c0c0 con 0x7f66180f0650 2021-02-10T11:39:41.440 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:41.430 7f66162bd700 1 -- 172.21.2.112:0/3542632137 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f661000c340 con 0x7f66180f0650 2021-02-10T11:39:41.441 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:41.430 7f66162bd700 1 -- 172.21.2.112:0/3542632137 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f661000c820 con 0x7f66180f0650 2021-02-10T11:39:41.441 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:41.430 7f661d59b700 1 -- 172.21.2.112:0/3542632137 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f66180f0650 msgr2=0x7f66180cfb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:41.441 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:41.430 7f661d59b700 1 --2- 172.21.2.112:0/3542632137 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f66180f0650 0x7f66180cfb20 secure :-1 s=READY pgs=1957 cs=0 l=1 rev1=1 rx=0x7f6610006b80 tx=0x7f6610009b90).stop 2021-02-10T11:39:41.441 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:41.431 7f661d59b700 1 -- 172.21.2.112:0/3542632137 shutdown_connections 2021-02-10T11:39:41.441 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:41.431 7f661d59b700 1 --2- 172.21.2.112:0/3542632137 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f66180f0650 0x7f66180cfb20 unknown :-1 s=CLOSED pgs=1957 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:41.441 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:41.431 7f661d59b700 1 --2- 172.21.2.112:0/3542632137 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f66180d1870 0x7f66181b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:41.441 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:41.431 7f661d59b700 1 -- 172.21.2.112:0/3542632137 >> 172.21.2.112:0/3542632137 conn(0x7f66180ce450 msgr2=0x7f66180d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:41.441 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:41.431 7f661d59b700 1 -- 172.21.2.112:0/3542632137 shutdown_connections 2021-02-10T11:39:41.441 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:41.431 7f661d59b700 1 -- 172.21.2.112:0/3542632137 wait complete. 2021-02-10T11:39:41.443 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:41.446 7f661d59b700 1 Processor -- start 2021-02-10T11:39:41.443 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:41.446 7f661d59b700 1 -- start start 2021-02-10T11:39:41.443 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:41.446 7f661d59b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6618099d50 0x7f66181b31b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:41.443 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:41.446 7f661d59b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6618187e50 0x7f6618188250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:41.444 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:41.446 7f661d59b700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f66181b12a0 0x7f66181b16a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:41.444 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:41.446 7f661d59b700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6618188870 con 0x7f6618099d50 2021-02-10T11:39:41.444 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:41.446 7f661d59b700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66181b37d0 con 0x7f66181b12a0 2021-02-10T11:39:41.444 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:41.446 7f661d59b700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66181b1cc0 con 0x7f6618187e50 2021-02-10T11:39:41.444 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:41.446 7f66172bf700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6618099d50 0x7f66181b31b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:41.444 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:41.446 7f66172bf700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6618099d50 0x7f66181b31b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:53980/0 (socket says 172.21.2.112:53980) 2021-02-10T11:39:41.444 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:41.446 7f66172bf700 1 -- 172.21.2.112:0/3689139791 learned_addr learned my addr 172.21.2.112:0/3689139791 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:39:41.444 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:41.447 7f6617ac0700 1 --2- 172.21.2.112:0/3689139791 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f66181b12a0 0x7f66181b16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:41.444 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:41.447 7f6616abe700 1 --2- 172.21.2.112:0/3689139791 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6618187e50 0x7f6618188250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:41.444 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:41.447 7f66172bf700 1 -- 172.21.2.112:0/3689139791 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6618187e50 msgr2=0x7f6618188250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:41.444 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:41.447 7f66172bf700 1 --2- 172.21.2.112:0/3689139791 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6618187e50 0x7f6618188250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:41.444 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:41.447 7f66172bf700 1 -- 172.21.2.112:0/3689139791 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f66181b12a0 msgr2=0x7f66181b16a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:41.445 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:41.447 7f66172bf700 1 --2- 172.21.2.112:0/3689139791 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f66181b12a0 0x7f66181b16a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:41.445 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:41.447 7f66172bf700 1 -- 172.21.2.112:0/3689139791 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66181b1ee0 con 0x7f6618099d50 2021-02-10T11:39:41.445 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:41.447 7f66172bf700 1 --2- 172.21.2.112:0/3689139791 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6618099d50 0x7f66181b31b0 secure :-1 s=READY pgs=1958 cs=0 l=1 rev1=1 rx=0x7f6600009980 tx=0x7f660000bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:41.445 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:41.447 7f660ffff700 1 -- 172.21.2.112:0/3689139791 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f660000c2f0 con 0x7f6618099d50 2021-02-10T11:39:41.445 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:41.447 7f660ffff700 1 -- 172.21.2.112:0/3689139791 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f660000c570 con 0x7f6618099d50 2021-02-10T11:39:41.445 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:41.447 7f661d59b700 1 -- 172.21.2.112:0/3689139791 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66181b1ee0 con 0x7f6618099d50 2021-02-10T11:39:41.445 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:41.447 7f661d59b700 1 -- 172.21.2.112:0/3689139791 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66181b2470 con 0x7f6618099d50 2021-02-10T11:39:41.445 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:41.447 7f660ffff700 1 -- 172.21.2.112:0/3689139791 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f660000c9d0 con 0x7f6618099d50 2021-02-10T11:39:41.445 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:41.448 7f660ffff700 1 -- 172.21.2.112:0/3689139791 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6600001010 con 0x7f6618099d50 2021-02-10T11:39:41.445 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:41.448 7f660ffff700 1 --2- 172.21.2.112:0/3689139791 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f65fc018730 0x7f65fc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:41.445 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:41.448 7f660ffff700 1 -- 172.21.2.112:0/3689139791 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f6600032640 con 0x7f6618099d50 2021-02-10T11:39:41.445 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:41.448 7f6616abe700 1 --2- 172.21.2.112:0/3689139791 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f65fc018730 0x7f65fc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:41.445 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:41.449 7f6616abe700 1 --2- 172.21.2.112:0/3689139791 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f65fc018730 0x7f65fc01abc0 crc :-1 s=READY pgs=911 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:41.473 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:41.479 7f661d59b700 1 -- 172.21.2.112:0/3689139791 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f661809a1e0 con 0x7f6618099d50 2021-02-10T11:39:41.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:41.482 7f660ffff700 1 -- 172.21.2.112:0/3689139791 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6600000ce0 con 0x7f6618099d50 2021-02-10T11:39:41.598 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:41.604 7f661d59b700 1 -- 172.21.2.112:0/3689139791 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f661809a1e0 con 0x7f6618099d50 2021-02-10T11:39:41.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:41.605 7f660ffff700 1 -- 172.21.2.112:0/3689139791 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v96) v1 ==== 84+0+64 (secure 0 0 0) 0x7f6600001010 con 0x7f6618099d50 2021-02-10T11:39:41.599 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:39:41.599 INFO:teuthology.orchestra.run.gibba012.stdout:[ 2021-02-10T11:39:41.599 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:39:41.599 INFO:teuthology.orchestra.run.gibba012.stdout: "poolnum": 1, 2021-02-10T11:39:41.600 INFO:teuthology.orchestra.run.gibba012.stdout: "poolname": "rbd" 2021-02-10T11:39:41.600 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:39:41.600 INFO:teuthology.orchestra.run.gibba012.stdout:] 2021-02-10T11:39:41.600 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:41.607 7f661d59b700 1 -- 172.21.2.112:0/3689139791 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f65fc018730 msgr2=0x7f65fc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:41.600 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:41.607 7f661d59b700 1 --2- 172.21.2.112:0/3689139791 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f65fc018730 0x7f65fc01abc0 crc :-1 s=READY pgs=911 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:41.600 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:41.607 7f661d59b700 1 -- 172.21.2.112:0/3689139791 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6618099d50 msgr2=0x7f66181b31b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:41.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:41.607 7f661d59b700 1 --2- 172.21.2.112:0/3689139791 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6618099d50 0x7f66181b31b0 secure :-1 s=READY pgs=1958 cs=0 l=1 rev1=1 rx=0x7f6600009980 tx=0x7f660000bcd0).stop 2021-02-10T11:39:41.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:41.607 7f661d59b700 1 -- 172.21.2.112:0/3689139791 shutdown_connections 2021-02-10T11:39:41.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:41.607 7f661d59b700 1 --2- 172.21.2.112:0/3689139791 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f65fc018730 0x7f65fc01abc0 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:41.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:41.607 7f661d59b700 1 --2- 172.21.2.112:0/3689139791 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f66181b12a0 0x7f66181b16a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:41.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:41.607 7f661d59b700 1 --2- 172.21.2.112:0/3689139791 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6618187e50 0x7f6618188250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:41.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:41.607 7f661d59b700 1 --2- 172.21.2.112:0/3689139791 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6618099d50 0x7f66181b31b0 unknown :-1 s=CLOSED pgs=1958 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:41.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:41.607 7f661d59b700 1 -- 172.21.2.112:0/3689139791 >> 172.21.2.112:0/3689139791 conn(0x7f661819f8c0 msgr2=0x7f66181a5fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:41.602 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:41.607 7f661d59b700 1 -- 172.21.2.112:0/3689139791 shutdown_connections 2021-02-10T11:39:41.602 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:41.607 7f661d59b700 1 -- 172.21.2.112:0/3689139791 wait complete. 2021-02-10T11:39:41.609 INFO:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2021-02-10T11:39:41.609 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_metadata 8 2021-02-10T11:39:42.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:42.030 7f30a9b51700 1 Processor -- start 2021-02-10T11:39:42.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:42.030 7f30a9b51700 1 -- start start 2021-02-10T11:39:42.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:42.030 7f30a9b51700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f30a40d1870 0x7f30a41b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:42.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:42.030 7f30a9b51700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f30a41b0ae0 0x7f30a4108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:42.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:42.030 7f30a9b51700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f30a40d1c70 0x7f30a4108f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:42.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:42.031 7f30a9b51700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30a40fe1c0 con 0x7f30a40d1870 2021-02-10T11:39:42.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:42.031 7f30a9b51700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30a4109a60 con 0x7f30a41b0ae0 2021-02-10T11:39:42.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:42.031 7f30a9b51700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30a4088070 con 0x7f30a40d1c70 2021-02-10T11:39:42.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:42.031 7f30a37fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f30a40d1870 0x7f30a41b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:42.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:42.031 7f30a37fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f30a40d1870 0x7f30a41b04c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:53988/0 (socket says 172.21.2.112:53988) 2021-02-10T11:39:42.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:42.031 7f30a37fe700 1 -- 172.21.2.112:0/4093507280 learned_addr learned my addr 172.21.2.112:0/4093507280 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:39:42.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:42.031 7f30a2ffd700 1 --2- 172.21.2.112:0/4093507280 >> v2:172.21.2.120:3300/0 conn(0x7f30a41b0ae0 0x7f30a4108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:42.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:42.031 7f30a3fff700 1 --2- 172.21.2.112:0/4093507280 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f30a40d1c70 0x7f30a4108f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:42.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:42.031 7f30a3fff700 1 -- 172.21.2.112:0/4093507280 >> v2:172.21.2.120:3300/0 conn(0x7f30a41b0ae0 msgr2=0x7f30a4108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:42.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:42.031 7f30a3fff700 1 --2- 172.21.2.112:0/4093507280 >> v2:172.21.2.120:3300/0 conn(0x7f30a41b0ae0 0x7f30a4108940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:42.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:42.031 7f30a3fff700 1 -- 172.21.2.112:0/4093507280 >> v2:172.21.2.112:3300/0 conn(0x7f30a40d1870 msgr2=0x7f30a41b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:42.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:42.031 7f30a3fff700 1 --2- 172.21.2.112:0/4093507280 >> v2:172.21.2.112:3300/0 conn(0x7f30a40d1870 0x7f30a41b04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:42.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:42.031 7f30a3fff700 1 -- 172.21.2.112:0/4093507280 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30a4109580 con 0x7f30a40d1c70 2021-02-10T11:39:42.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:42.031 7f30a37fe700 1 --2- 172.21.2.112:0/4093507280 >> v2:172.21.2.112:3300/0 conn(0x7f30a40d1870 0x7f30a41b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:39:42.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:42.031 7f30a2ffd700 1 --2- 172.21.2.112:0/4093507280 >> v2:172.21.2.120:3300/0 conn(0x7f30a41b0ae0 0x7f30a4108940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:39:42.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:42.032 7f30a3fff700 1 --2- 172.21.2.112:0/4093507280 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f30a40d1c70 0x7f30a4108f60 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7f309c006ad0 tx=0x7f309c009ad0).ready entity=mon.2 client_cookie=ba8c34622b585884 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:42.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:42.032 7f30a27fc700 1 -- 172.21.2.112:0/4093507280 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f309c000c10 con 0x7f30a40d1c70 2021-02-10T11:39:42.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:42.032 7f30a27fc700 1 -- 172.21.2.112:0/4093507280 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f309c006370 con 0x7f30a40d1c70 2021-02-10T11:39:42.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:42.032 7f30a27fc700 1 -- 172.21.2.112:0/4093507280 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f309c002560 con 0x7f30a40d1c70 2021-02-10T11:39:42.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:42.032 7f30a9b51700 1 -- 172.21.2.112:0/4093507280 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f30a40d1c70 msgr2=0x7f30a4108f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:42.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:42.032 7f30a9b51700 1 --2- 172.21.2.112:0/4093507280 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f30a40d1c70 0x7f30a4108f60 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7f309c006ad0 tx=0x7f309c009ad0).stop 2021-02-10T11:39:42.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:42.033 7f30a9b51700 1 -- 172.21.2.112:0/4093507280 shutdown_connections 2021-02-10T11:39:42.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:42.033 7f30a9b51700 1 --2- 172.21.2.112:0/4093507280 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f30a40d1c70 0x7f30a4108f60 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:42.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:42.033 7f30a9b51700 1 --2- 172.21.2.112:0/4093507280 >> v2:172.21.2.120:3300/0 conn(0x7f30a41b0ae0 0x7f30a4108940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:42.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:42.033 7f30a9b51700 1 --2- 172.21.2.112:0/4093507280 >> v2:172.21.2.112:3300/0 conn(0x7f30a40d1870 0x7f30a41b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:42.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:42.033 7f30a9b51700 1 -- 172.21.2.112:0/4093507280 >> 172.21.2.112:0/4093507280 conn(0x7f30a40ce420 msgr2=0x7f30a40d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:42.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:42.033 7f30a9b51700 1 -- 172.21.2.112:0/4093507280 shutdown_connections 2021-02-10T11:39:42.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:42.033 7f30a9b51700 1 -- 172.21.2.112:0/4093507280 wait complete. 2021-02-10T11:39:42.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:42.047 7f30a9b51700 1 Processor -- start 2021-02-10T11:39:42.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:42.047 7f30a9b51700 1 -- start start 2021-02-10T11:39:42.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:42.048 7f30a9b51700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f30a4085bc0 0x7f30a40863e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:42.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:42.048 7f30a9b51700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f30a40954c0 0x7f30a4096300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:42.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:42.048 7f30a9b51700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f30a40958c0 0x7f30a4096920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:42.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:42.048 7f30a9b51700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30a4095d80 con 0x7f30a4085bc0 2021-02-10T11:39:42.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:42.048 7f30a9b51700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30a40bfa60 con 0x7f30a40958c0 2021-02-10T11:39:42.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:42.048 7f30a9b51700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30a4085fc0 con 0x7f30a40954c0 2021-02-10T11:39:42.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:42.048 7f30a37fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f30a4085bc0 0x7f30a40863e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:42.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:42.048 7f30a37fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f30a4085bc0 0x7f30a40863e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:53994/0 (socket says 172.21.2.112:53994) 2021-02-10T11:39:42.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:42.048 7f30a37fe700 1 -- 172.21.2.112:0/2030012275 learned_addr learned my addr 172.21.2.112:0/2030012275 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:39:42.047 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:42.048 7f30a2ffd700 1 --2- 172.21.2.112:0/2030012275 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f30a40954c0 0x7f30a4096300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:42.047 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:42.048 7f30a3fff700 1 --2- 172.21.2.112:0/2030012275 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f30a40958c0 0x7f30a4096920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:42.047 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:42.048 7f30a37fe700 1 -- 172.21.2.112:0/2030012275 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f30a40954c0 msgr2=0x7f30a4096300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:42.047 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:42.048 7f30a37fe700 1 --2- 172.21.2.112:0/2030012275 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f30a40954c0 0x7f30a4096300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:42.047 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:42.048 7f30a37fe700 1 -- 172.21.2.112:0/2030012275 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f30a40958c0 msgr2=0x7f30a4096920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:42.047 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:42.048 7f30a37fe700 1 --2- 172.21.2.112:0/2030012275 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f30a40958c0 0x7f30a4096920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:42.047 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:42.048 7f30a37fe700 1 -- 172.21.2.112:0/2030012275 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30a4086c10 con 0x7f30a4085bc0 2021-02-10T11:39:42.047 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:42.048 7f30a37fe700 1 --2- 172.21.2.112:0/2030012275 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f30a4085bc0 0x7f30a40863e0 secure :-1 s=READY pgs=1959 cs=0 l=1 rev1=1 rx=0x7f3094006ee0 tx=0x7f309400b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:42.047 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:42.049 7f309bfff700 1 -- 172.21.2.112:0/2030012275 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f309400bd50 con 0x7f30a4085bc0 2021-02-10T11:39:42.047 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:42.049 7f309bfff700 1 -- 172.21.2.112:0/2030012275 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30940052e0 con 0x7f30a4085bc0 2021-02-10T11:39:42.047 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:42.049 7f309bfff700 1 -- 172.21.2.112:0/2030012275 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3094005740 con 0x7f30a4085bc0 2021-02-10T11:39:42.048 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:42.049 7f30a9b51700 1 -- 172.21.2.112:0/2030012275 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30a4086c10 con 0x7f30a4085bc0 2021-02-10T11:39:42.048 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:42.049 7f30a9b51700 1 -- 172.21.2.112:0/2030012275 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30a4086c10 con 0x7f30a4085bc0 2021-02-10T11:39:42.048 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:42.050 7f309bfff700 1 -- 172.21.2.112:0/2030012275 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f30940054c0 con 0x7f30a4085bc0 2021-02-10T11:39:42.048 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:42.050 7f309bfff700 1 --2- 172.21.2.112:0/2030012275 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3088018730 0x7f308801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:42.048 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:42.050 7f309bfff700 1 -- 172.21.2.112:0/2030012275 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f30940322f0 con 0x7f30a4085bc0 2021-02-10T11:39:42.048 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:42.050 7f30a2ffd700 1 --2- 172.21.2.112:0/2030012275 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3088018730 0x7f308801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:42.048 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:42.050 7f30a2ffd700 1 --2- 172.21.2.112:0/2030012275 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3088018730 0x7f308801abc0 crc :-1 s=READY pgs=912 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:42.074 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:42.081 7f30a9b51700 1 -- 172.21.2.112:0/2030012275 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30a40fe1c0 con 0x7f30a4085bc0 2021-02-10T11:39:42.077 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:42.083 7f309bfff700 1 -- 172.21.2.112:0/2030012275 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f309401f4e0 con 0x7f30a4085bc0 2021-02-10T11:39:42.205 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:42.211 7f30a9b51700 1 -- 172.21.2.112:0/2030012275 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"} v 0) v1 -- 0x7f30a40fe1c0 con 0x7f30a4085bc0 2021-02-10T11:39:42.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:42.681 7f309bfff700 1 -- 172.21.2.112:0/2030012275 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"}]=0 pool 'cephfs_metadata' created v97) v1 ==== 133+0+0 (secure 0 0 0) 0x7f309401f6f0 con 0x7f30a4085bc0 2021-02-10T11:39:42.722 INFO:teuthology.orchestra.run.gibba012.stderr:pool 'cephfs_metadata' created 2021-02-10T11:39:42.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:42.730 7f30a9b51700 1 -- 172.21.2.112:0/2030012275 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3088018730 msgr2=0x7f308801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:42.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:42.730 7f30a9b51700 1 --2- 172.21.2.112:0/2030012275 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3088018730 0x7f308801abc0 crc :-1 s=READY pgs=912 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:42.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:42.730 7f30a9b51700 1 -- 172.21.2.112:0/2030012275 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f30a4085bc0 msgr2=0x7f30a40863e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:42.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:42.730 7f30a9b51700 1 --2- 172.21.2.112:0/2030012275 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f30a4085bc0 0x7f30a40863e0 secure :-1 s=READY pgs=1959 cs=0 l=1 rev1=1 rx=0x7f3094006ee0 tx=0x7f309400b960).stop 2021-02-10T11:39:42.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:42.730 7f30a9b51700 1 -- 172.21.2.112:0/2030012275 shutdown_connections 2021-02-10T11:39:42.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:42.730 7f30a9b51700 1 --2- 172.21.2.112:0/2030012275 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3088018730 0x7f308801abc0 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:42.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:42.730 7f30a9b51700 1 --2- 172.21.2.112:0/2030012275 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f30a40958c0 0x7f30a4096920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:42.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:42.730 7f30a9b51700 1 --2- 172.21.2.112:0/2030012275 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f30a40954c0 0x7f30a4096300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:42.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:42.730 7f30a9b51700 1 --2- 172.21.2.112:0/2030012275 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f30a4085bc0 0x7f30a40863e0 unknown :-1 s=CLOSED pgs=1959 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:42.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:42.730 7f30a9b51700 1 -- 172.21.2.112:0/2030012275 >> 172.21.2.112:0/2030012275 conn(0x7f30a40bef00 msgr2=0x7f30a409d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:42.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:42.731 7f30a9b51700 1 -- 172.21.2.112:0/2030012275 shutdown_connections 2021-02-10T11:39:42.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:42.731 7f30a9b51700 1 -- 172.21.2.112:0/2030012275 wait complete. 2021-02-10T11:39:42.733 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_data 8 2021-02-10T11:39:43.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:43.156 7f761394b700 1 Processor -- start 2021-02-10T11:39:43.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:43.156 7f761394b700 1 -- start start 2021-02-10T11:39:43.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:43.157 7f761394b700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f760c0d1c70 0x7f760c0cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:43.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:43.157 7f761394b700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f760c0d0170 con 0x7f760c0cf580 2021-02-10T11:39:43.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:43.157 7f761394b700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f760c0fe1c0 con 0x7f760c0d1870 2021-02-10T11:39:43.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:43.157 7f761394b700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f760c1b0cb0 con 0x7f760c0d1c70 2021-02-10T11:39:43.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:43.157 7f7610f66700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f760c0d1c70 0x7f760c0cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:43.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:43.157 7f7611f68700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f760c0cf580 0x7f760c0cf980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:53464/0 (socket says 172.21.2.112:53464) 2021-02-10T11:39:43.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:43.157 7f7611f68700 1 -- 172.21.2.112:0/3782491911 learned_addr learned my addr 172.21.2.112:0/3782491911 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:39:43.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:43.158 7f7610f66700 1 -- 172.21.2.112:0/3782491911 >> v1:172.21.2.120:6789/0 conn(0x7f760c0d1870 legacy=0x7f760c1b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:43.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:43.158 7f7610f66700 1 -- 172.21.2.112:0/3782491911 >> v1:172.21.2.112:6789/0 conn(0x7f760c0cf580 legacy=0x7f760c0cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:43.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:43.158 7f7610f66700 1 -- 172.21.2.112:0/3782491911 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f760c109a60 con 0x7f760c0d1c70 2021-02-10T11:39:43.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:43.158 7f7610f66700 1 --2- 172.21.2.112:0/3782491911 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f760c0d1c70 0x7f760c0cef60 secure :-1 s=READY pgs=1961 cs=0 l=1 rev1=1 rx=0x7f75fc006650 tx=0x7f75fc009a00).ready entity=mon.0 client_cookie=dd3a9934ef59dbbc server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:43.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:43.158 7f7603fff700 1 -- 172.21.2.112:0/3782491911 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f75fc0009b0 con 0x7f760c0d1c70 2021-02-10T11:39:43.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:43.158 7f7603fff700 1 -- 172.21.2.112:0/3782491911 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75fc00c0c0 con 0x7f760c0d1c70 2021-02-10T11:39:43.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:43.158 7f7603fff700 1 -- 172.21.2.112:0/3782491911 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f75fc00c5a0 con 0x7f760c0d1c70 2021-02-10T11:39:43.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:43.158 7f761394b700 1 -- 172.21.2.112:0/3782491911 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f760c0d1c70 msgr2=0x7f760c0cef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:43.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:43.158 7f761394b700 1 --2- 172.21.2.112:0/3782491911 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f760c0d1c70 0x7f760c0cef60 secure :-1 s=READY pgs=1961 cs=0 l=1 rev1=1 rx=0x7f75fc006650 tx=0x7f75fc009a00).stop 2021-02-10T11:39:43.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:43.159 7f761394b700 1 -- 172.21.2.112:0/3782491911 shutdown_connections 2021-02-10T11:39:43.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:43.159 7f761394b700 1 --2- 172.21.2.112:0/3782491911 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f760c0d1c70 0x7f760c0cef60 unknown :-1 s=CLOSED pgs=1961 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:43.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:43.159 7f761394b700 1 -- 172.21.2.112:0/3782491911 >> 172.21.2.112:0/3782491911 conn(0x7f760c0ce420 msgr2=0x7f760c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:43.169 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:43.159 7f761394b700 1 -- 172.21.2.112:0/3782491911 shutdown_connections 2021-02-10T11:39:43.169 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:43.159 7f761394b700 1 -- 172.21.2.112:0/3782491911 wait complete. 2021-02-10T11:39:43.170 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:43.173 7f761394b700 1 Processor -- start 2021-02-10T11:39:43.170 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:43.173 7f761394b700 1 -- start start 2021-02-10T11:39:43.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:43.174 7f761394b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f760c0c0740 0x7f760c1c5580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:43.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:43.174 7f761394b700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f760c1c5ba0 0x7f760c0a5c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:43.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:43.174 7f761394b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f760c0b6e60 0x7f760c0a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:43.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:43.174 7f761394b700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f760c0a6880 con 0x7f760c0c0740 2021-02-10T11:39:43.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:43.174 7f761394b700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f760c0a6aa0 con 0x7f760c1c5ba0 2021-02-10T11:39:43.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:43.174 7f761394b700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f760c0a6cc0 con 0x7f760c0b6e60 2021-02-10T11:39:43.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:43.174 7f7611767700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f760c0c0740 0x7f760c1c5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:43.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:43.174 7f7611767700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f760c0c0740 0x7f760c1c5580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:54012/0 (socket says 172.21.2.112:54012) 2021-02-10T11:39:43.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:43.174 7f7611767700 1 -- 172.21.2.112:0/1144152319 learned_addr learned my addr 172.21.2.112:0/1144152319 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:39:43.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:43.174 7f7610f66700 1 --2- 172.21.2.112:0/1144152319 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f760c1c5ba0 0x7f760c0a5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:43.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:43.175 7f7611f68700 1 --2- 172.21.2.112:0/1144152319 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f760c0b6e60 0x7f760c0a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:43.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:43.175 7f7611767700 1 -- 172.21.2.112:0/1144152319 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f760c0b6e60 msgr2=0x7f760c0a6260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:43.172 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:43.175 7f7611767700 1 --2- 172.21.2.112:0/1144152319 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f760c0b6e60 0x7f760c0a6260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:43.172 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:43.175 7f7611767700 1 -- 172.21.2.112:0/1144152319 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f760c1c5ba0 msgr2=0x7f760c0a5c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:43.172 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:43.175 7f7611767700 1 --2- 172.21.2.112:0/1144152319 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f760c1c5ba0 0x7f760c0a5c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:43.172 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:43.175 7f7611767700 1 -- 172.21.2.112:0/1144152319 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f760c0a6ee0 con 0x7f760c0c0740 2021-02-10T11:39:43.172 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:43.175 7f7611f68700 1 --2- 172.21.2.112:0/1144152319 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f760c0b6e60 0x7f760c0a6260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:39:43.172 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:43.175 7f7611767700 1 --2- 172.21.2.112:0/1144152319 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f760c0c0740 0x7f760c1c5580 secure :-1 s=READY pgs=1962 cs=0 l=1 rev1=1 rx=0x7f760800b5b0 tx=0x7f760800b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:43.172 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:43.175 7f7601ffb700 1 -- 172.21.2.112:0/1144152319 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f760800bd40 con 0x7f760c0c0740 2021-02-10T11:39:43.172 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:43.175 7f7601ffb700 1 -- 172.21.2.112:0/1144152319 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7608000ce0 con 0x7f760c0c0740 2021-02-10T11:39:43.172 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:43.175 7f761394b700 1 -- 172.21.2.112:0/1144152319 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f760c0a6ee0 con 0x7f760c0c0740 2021-02-10T11:39:43.172 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:43.175 7f7601ffb700 1 -- 172.21.2.112:0/1144152319 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f76080054a0 con 0x7f760c0c0740 2021-02-10T11:39:43.172 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:43.175 7f761394b700 1 -- 172.21.2.112:0/1144152319 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f760c0a6ee0 con 0x7f760c0c0740 2021-02-10T11:39:43.172 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:43.176 7f7601ffb700 1 -- 172.21.2.112:0/1144152319 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7608006030 con 0x7f760c0c0740 2021-02-10T11:39:43.172 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:43.176 7f7601ffb700 1 --2- 172.21.2.112:0/1144152319 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f75ec018730 0x7f75ec01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:43.173 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:43.176 7f7601ffb700 1 -- 172.21.2.112:0/1144152319 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5107+0+0 (secure 0 0 0) 0x7f76080324f0 con 0x7f760c0c0740 2021-02-10T11:39:43.173 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:43.176 7f7610f66700 1 --2- 172.21.2.112:0/1144152319 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f75ec018730 0x7f75ec01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:43.173 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:43.176 7f7610f66700 1 --2- 172.21.2.112:0/1144152319 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f75ec018730 0x7f75ec01ac10 crc :-1 s=READY pgs=913 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:43.199 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:43.205 7f761394b700 1 -- 172.21.2.112:0/1144152319 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f760c0807d0 con 0x7f760c0c0740 2021-02-10T11:39:43.201 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:43.208 7f7601ffb700 1 -- 172.21.2.112:0/1144152319 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f760800bd40 con 0x7f760c0c0740 2021-02-10T11:39:43.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:43.336 7f761394b700 1 -- 172.21.2.112:0/1144152319 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"} v 0) v1 -- 0x7f760c0807d0 con 0x7f760c0c0740 2021-02-10T11:39:43.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:43.688 7f7601ffb700 1 -- 172.21.2.112:0/1144152319 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"}]=0 pool 'cephfs_data' created v98) v1 ==== 125+0+0 (secure 0 0 0) 0x7f7608006030 con 0x7f760c0c0740 2021-02-10T11:39:43.696 INFO:teuthology.orchestra.run.gibba012.stderr:pool 'cephfs_data' created 2021-02-10T11:39:43.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:43.703 7f761394b700 1 -- 172.21.2.112:0/1144152319 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f75ec018730 msgr2=0x7f75ec01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:43.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:43.703 7f761394b700 1 --2- 172.21.2.112:0/1144152319 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f75ec018730 0x7f75ec01ac10 crc :-1 s=READY pgs=913 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:43.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:43.703 7f761394b700 1 -- 172.21.2.112:0/1144152319 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f760c0c0740 msgr2=0x7f760c1c5580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:43.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:43.703 7f761394b700 1 --2- 172.21.2.112:0/1144152319 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f760c0c0740 0x7f760c1c5580 secure :-1 s=READY pgs=1962 cs=0 l=1 rev1=1 rx=0x7f760800b5b0 tx=0x7f760800b950).stop 2021-02-10T11:39:43.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:43.704 7f761394b700 1 -- 172.21.2.112:0/1144152319 shutdown_connections 2021-02-10T11:39:43.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:43.704 7f761394b700 1 --2- 172.21.2.112:0/1144152319 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f75ec018730 0x7f75ec01ac10 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:43.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:43.704 7f761394b700 1 --2- 172.21.2.112:0/1144152319 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f760c1c5ba0 0x7f760c0a5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:43.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:43.704 7f761394b700 1 --2- 172.21.2.112:0/1144152319 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f760c0b6e60 0x7f760c0a6260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:43.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:43.704 7f761394b700 1 --2- 172.21.2.112:0/1144152319 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f760c0c0740 0x7f760c1c5580 unknown :-1 s=CLOSED pgs=1962 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:43.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:43.704 7f761394b700 1 -- 172.21.2.112:0/1144152319 >> 172.21.2.112:0/1144152319 conn(0x7f760c0c4120 msgr2=0x7f760c0ffd60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:43.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:43.704 7f761394b700 1 -- 172.21.2.112:0/1144152319 shutdown_connections 2021-02-10T11:39:43.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:43.704 7f761394b700 1 -- 172.21.2.112:0/1144152319 wait complete. 2021-02-10T11:39:43.706 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2021-02-10T11:39:44.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.128 7ff8b47df700 1 Processor -- start 2021-02-10T11:39:44.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.128 7ff8b47df700 1 -- start start 2021-02-10T11:39:44.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.128 7ff8b47df700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7ff8ac0d1880 0x7ff8ac1b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:44.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.128 7ff8b47df700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7ff8ac1b0af0 0x7ff8ac108950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:44.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.128 7ff8b47df700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8ac0fe1d0 con 0x7ff8ac0d1c80 2021-02-10T11:39:44.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.128 7ff8b47df700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8ac109a70 con 0x7ff8ac0d1880 2021-02-10T11:39:44.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.128 7ff8b47df700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8ac109760 con 0x7ff8ac1b0af0 2021-02-10T11:39:44.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.129 7ff8b25fb700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7ff8ac0d1880 0x7ff8ac1b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:44.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.129 7ff8b25fb700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7ff8ac0d1880 0x7ff8ac1b04d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:54018/0 (socket says 172.21.2.112:54018) 2021-02-10T11:39:44.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.129 7ff8b25fb700 1 -- 172.21.2.112:0/931396712 learned_addr learned my addr 172.21.2.112:0/931396712 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:39:44.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.129 7ff8b25fb700 1 -- 172.21.2.112:0/931396712 >> v2:172.21.2.120:3300/0 conn(0x7ff8ac1b0af0 msgr2=0x7ff8ac108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:44.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.129 7ff8b25fb700 1 --2- 172.21.2.112:0/931396712 >> v2:172.21.2.120:3300/0 conn(0x7ff8ac1b0af0 0x7ff8ac108950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:44.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.129 7ff8b25fb700 1 -- 172.21.2.112:0/931396712 >> v1:172.21.2.120:6789/0 conn(0x7ff8ac0d1c80 legacy=0x7ff8ac108f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:44.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.129 7ff8b25fb700 1 -- 172.21.2.112:0/931396712 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8ac088070 con 0x7ff8ac0d1880 2021-02-10T11:39:44.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.129 7ff8b25fb700 1 --2- 172.21.2.112:0/931396712 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff8ac0d1880 0x7ff8ac1b04d0 secure :-1 s=READY pgs=1963 cs=0 l=1 rev1=1 rx=0x7ff8a8006700 tx=0x7ff8a8009b80).ready entity=mon.0 client_cookie=f05208d2f23e4ea3 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:44.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.129 7ff8b15f9700 1 -- 172.21.2.112:0/931396712 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff8a800c0c0 con 0x7ff8ac0d1880 2021-02-10T11:39:44.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.129 7ff8b15f9700 1 -- 172.21.2.112:0/931396712 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8a800c440 con 0x7ff8ac0d1880 2021-02-10T11:39:44.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.129 7ff8b15f9700 1 -- 172.21.2.112:0/931396712 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff8a800c920 con 0x7ff8ac0d1880 2021-02-10T11:39:44.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.129 7ff8b47df700 1 -- 172.21.2.112:0/931396712 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff8ac0d1880 msgr2=0x7ff8ac1b04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:44.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.129 7ff8b47df700 1 --2- 172.21.2.112:0/931396712 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff8ac0d1880 0x7ff8ac1b04d0 secure :-1 s=READY pgs=1963 cs=0 l=1 rev1=1 rx=0x7ff8a8006700 tx=0x7ff8a8009b80).stop 2021-02-10T11:39:44.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.130 7ff8b47df700 1 -- 172.21.2.112:0/931396712 shutdown_connections 2021-02-10T11:39:44.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.130 7ff8b47df700 1 --2- 172.21.2.112:0/931396712 >> v2:172.21.2.120:3300/0 conn(0x7ff8ac1b0af0 0x7ff8ac108950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:44.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.130 7ff8b47df700 1 --2- 172.21.2.112:0/931396712 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff8ac0d1880 0x7ff8ac1b04d0 unknown :-1 s=CLOSED pgs=1963 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:44.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.130 7ff8b47df700 1 -- 172.21.2.112:0/931396712 >> 172.21.2.112:0/931396712 conn(0x7ff8ac0ce430 msgr2=0x7ff8ac0d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:44.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.130 7ff8b47df700 1 -- 172.21.2.112:0/931396712 shutdown_connections 2021-02-10T11:39:44.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.130 7ff8b47df700 1 -- 172.21.2.112:0/931396712 wait complete. 2021-02-10T11:39:44.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.145 7ff8b47df700 1 Processor -- start 2021-02-10T11:39:44.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.145 7ff8b47df700 1 -- start start 2021-02-10T11:39:44.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.145 7ff8b47df700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff8ac19e5b0 0x7ff8ac1afc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:44.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.145 7ff8b47df700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff8ac19e9b0 0x7ff8ac1b0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:44.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.145 7ff8b47df700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff8ac1b0250 0x7ff8ac1b13e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:44.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.145 7ff8b47df700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8ac195bd0 con 0x7ff8ac19e5b0 2021-02-10T11:39:44.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.145 7ff8b47df700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8ac1b0840 con 0x7ff8ac1b0250 2021-02-10T11:39:44.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.145 7ff8b47df700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8ac19eed0 con 0x7ff8ac19e9b0 2021-02-10T11:39:44.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.145 7ff8b25fb700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff8ac19e5b0 0x7ff8ac1afc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:44.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.145 7ff8b25fb700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff8ac19e5b0 0x7ff8ac1afc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:54024/0 (socket says 172.21.2.112:54024) 2021-02-10T11:39:44.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.145 7ff8b25fb700 1 -- 172.21.2.112:0/276000264 learned_addr learned my addr 172.21.2.112:0/276000264 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:39:44.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.145 7ff8b2dfc700 1 --2- 172.21.2.112:0/276000264 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff8ac1b0250 0x7ff8ac1b13e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:44.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.146 7ff8b1dfa700 1 --2- 172.21.2.112:0/276000264 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff8ac19e9b0 0x7ff8ac1b0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:44.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.146 7ff8b25fb700 1 -- 172.21.2.112:0/276000264 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff8ac19e9b0 msgr2=0x7ff8ac1b0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:44.145 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.146 7ff8b25fb700 1 --2- 172.21.2.112:0/276000264 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff8ac19e9b0 0x7ff8ac1b0dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:44.145 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.146 7ff8b25fb700 1 -- 172.21.2.112:0/276000264 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff8ac1b0250 msgr2=0x7ff8ac1b13e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:44.145 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.146 7ff8b25fb700 1 --2- 172.21.2.112:0/276000264 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff8ac1b0250 0x7ff8ac1b13e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:44.145 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.146 7ff8b25fb700 1 -- 172.21.2.112:0/276000264 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8ac19f0b0 con 0x7ff8ac19e5b0 2021-02-10T11:39:44.145 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.146 7ff8b25fb700 1 --2- 172.21.2.112:0/276000264 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff8ac19e5b0 0x7ff8ac1afc30 secure :-1 s=READY pgs=1964 cs=0 l=1 rev1=1 rx=0x7ff8a800b650 tx=0x7ff8a800b9f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:44.145 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.146 7ff89effd700 1 -- 172.21.2.112:0/276000264 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff8a800bde0 con 0x7ff8ac19e5b0 2021-02-10T11:39:44.145 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.146 7ff89effd700 1 -- 172.21.2.112:0/276000264 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8a8000c90 con 0x7ff8ac19e5b0 2021-02-10T11:39:44.145 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.146 7ff89effd700 1 -- 172.21.2.112:0/276000264 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff8a8002220 con 0x7ff8ac19e5b0 2021-02-10T11:39:44.145 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.146 7ff8b47df700 1 -- 172.21.2.112:0/276000264 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8ac19f0b0 con 0x7ff8ac19e5b0 2021-02-10T11:39:44.145 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.147 7ff8b47df700 1 -- 172.21.2.112:0/276000264 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8ac1b1cf0 con 0x7ff8ac19e5b0 2021-02-10T11:39:44.145 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.147 7ff89effd700 1 -- 172.21.2.112:0/276000264 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff8a8009f90 con 0x7ff8ac19e5b0 2021-02-10T11:39:44.145 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.147 7ff89effd700 1 --2- 172.21.2.112:0/276000264 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff8940186c0 0x7ff89401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:44.145 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.147 7ff89effd700 1 -- 172.21.2.112:0/276000264 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5457+0+0 (secure 0 0 0) 0x7ff8a8032500 con 0x7ff8ac19e5b0 2021-02-10T11:39:44.145 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.148 7ff8b1dfa700 1 --2- 172.21.2.112:0/276000264 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff8940186c0 0x7ff89401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:44.146 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.148 7ff8b1dfa700 1 --2- 172.21.2.112:0/276000264 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff8940186c0 0x7ff89401abc0 crc :-1 s=READY pgs=916 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:44.173 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.179 7ff8b47df700 1 -- 172.21.2.112:0/276000264 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8ac19f0b0 con 0x7ff8ac19e5b0 2021-02-10T11:39:44.175 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.182 7ff89effd700 1 -- 172.21.2.112:0/276000264 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff8a80009b0 con 0x7ff8ac19e5b0 2021-02-10T11:39:44.303 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.310 7ff8b47df700 1 -- 172.21.2.112:0/276000264 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"} v 0) v1 -- 0x7ff8ac19f0b0 con 0x7ff8ac19e5b0 2021-02-10T11:39:44.307 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.314 7ff89effd700 1 -- 172.21.2.112:0/276000264 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"}]=0 new fs with metadata pool 20 and data pool 21 v77) v1 ==== 189+0+0 (secure 0 0 0) 0x7ff8a800a2f0 con 0x7ff8ac19e5b0 2021-02-10T11:39:44.310 INFO:teuthology.orchestra.run.gibba012.stderr:new fs with metadata pool 20 and data pool 21 2021-02-10T11:39:44.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.319 7ff8b47df700 1 -- 172.21.2.112:0/276000264 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff8940186c0 msgr2=0x7ff89401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:44.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.319 7ff8b47df700 1 --2- 172.21.2.112:0/276000264 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff8940186c0 0x7ff89401abc0 crc :-1 s=READY pgs=916 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:44.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.319 7ff8b47df700 1 -- 172.21.2.112:0/276000264 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff8ac19e5b0 msgr2=0x7ff8ac1afc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:44.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.319 7ff8b47df700 1 --2- 172.21.2.112:0/276000264 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff8ac19e5b0 0x7ff8ac1afc30 secure :-1 s=READY pgs=1964 cs=0 l=1 rev1=1 rx=0x7ff8a800b650 tx=0x7ff8a800b9f0).stop 2021-02-10T11:39:44.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.319 7ff8b47df700 1 -- 172.21.2.112:0/276000264 shutdown_connections 2021-02-10T11:39:44.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.319 7ff8b47df700 1 --2- 172.21.2.112:0/276000264 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff8940186c0 0x7ff89401abc0 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:44.314 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.319 7ff8b47df700 1 --2- 172.21.2.112:0/276000264 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff8ac1b0250 0x7ff8ac1b13e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:44.314 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.319 7ff8b47df700 1 --2- 172.21.2.112:0/276000264 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff8ac19e9b0 0x7ff8ac1b0dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:44.314 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.319 7ff8b47df700 1 --2- 172.21.2.112:0/276000264 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff8ac19e5b0 0x7ff8ac1afc30 unknown :-1 s=CLOSED pgs=1964 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:44.314 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.319 7ff8b47df700 1 -- 172.21.2.112:0/276000264 >> 172.21.2.112:0/276000264 conn(0x7ff8ac19f940 msgr2=0x7ff8ac18c290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:44.314 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.319 7ff8b47df700 1 -- 172.21.2.112:0/276000264 shutdown_connections 2021-02-10T11:39:44.314 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.320 7ff8b47df700 1 -- 172.21.2.112:0/276000264 wait complete. 2021-02-10T11:39:44.321 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:39:44.749 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.739 7fd0e25a7700 1 Processor -- start 2021-02-10T11:39:44.749 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.739 7fd0e25a7700 1 -- start start 2021-02-10T11:39:44.749 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.739 7fd0e25a7700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fd0dc0d1c70 0x7fd0dc0f20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:44.749 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.739 7fd0e25a7700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0dc0f0b10 con 0x7fd0dc0d0340 2021-02-10T11:39:44.749 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.739 7fd0e25a7700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0dc1b0be0 con 0x7fd0dc0d1870 2021-02-10T11:39:44.749 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.739 7fd0e25a7700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0dc0e5ed0 con 0x7fd0dc0d1c70 2021-02-10T11:39:44.749 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.740 7fd0e0bc4700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fd0dc0d1c70 0x7fd0dc0f20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:44.750 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.740 7fd0dbfff700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7fd0dc0d1870 0x7fd0dc1b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:47050/0 (socket says 172.21.2.112:47050) 2021-02-10T11:39:44.750 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.740 7fd0dbfff700 1 -- 172.21.2.112:0/2271121705 learned_addr learned my addr 172.21.2.112:0/2271121705 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:39:44.750 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.740 7fd0db7fe700 1 --1- 172.21.2.112:0/2271121705 >> v1:172.21.2.112:6789/0 conn(0x7fd0dc0d0340 0x7fd0dc0ef640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:53492/0 (socket says 172.21.2.112:53492) 2021-02-10T11:39:44.750 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.740 7fd0daffd700 1 -- 172.21.2.112:0/2271121705 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3930538205 0 0) 0x7fd0cc0010c0 con 0x7fd0dc0d1870 2021-02-10T11:39:44.750 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.740 7fd0e0bc4700 1 -- 172.21.2.112:0/2271121705 >> v1:172.21.2.120:6789/0 conn(0x7fd0dc0d1870 legacy=0x7fd0dc1b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:44.750 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.740 7fd0e0bc4700 1 -- 172.21.2.112:0/2271121705 >> v1:172.21.2.112:6789/0 conn(0x7fd0dc0d0340 legacy=0x7fd0dc0ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:44.750 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.740 7fd0e0bc4700 1 -- 172.21.2.112:0/2271121705 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0dc0efe30 con 0x7fd0dc0d1c70 2021-02-10T11:39:44.750 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.740 7fd0e0bc4700 1 --2- 172.21.2.112:0/2271121705 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd0dc0d1c70 0x7fd0dc0f20a0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 rx=0x7fd0d4006e50 tx=0x7fd0d4009e70).ready entity=mon.1 client_cookie=449a34a23d8369cb server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:44.750 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.740 7fd0daffd700 1 -- 172.21.2.112:0/2271121705 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd0d400b510 con 0x7fd0dc0d1c70 2021-02-10T11:39:44.750 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.740 7fd0daffd700 1 -- 172.21.2.112:0/2271121705 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0d4006790 con 0x7fd0dc0d1c70 2021-02-10T11:39:44.750 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.740 7fd0daffd700 1 -- 172.21.2.112:0/2271121705 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd0d4000f90 con 0x7fd0dc0d1c70 2021-02-10T11:39:44.750 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.741 7fd0e25a7700 1 -- 172.21.2.112:0/2271121705 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd0dc0d1c70 msgr2=0x7fd0dc0f20a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:44.751 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.741 7fd0e25a7700 1 --2- 172.21.2.112:0/2271121705 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd0dc0d1c70 0x7fd0dc0f20a0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 rx=0x7fd0d4006e50 tx=0x7fd0d4009e70).stop 2021-02-10T11:39:44.751 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.741 7fd0e25a7700 1 -- 172.21.2.112:0/2271121705 shutdown_connections 2021-02-10T11:39:44.751 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.741 7fd0e25a7700 1 --2- 172.21.2.112:0/2271121705 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd0dc0d1c70 0x7fd0dc0f20a0 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:44.751 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.741 7fd0e25a7700 1 -- 172.21.2.112:0/2271121705 >> 172.21.2.112:0/2271121705 conn(0x7fd0dc0ce450 msgr2=0x7fd0dc0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:44.751 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.741 7fd0e25a7700 1 -- 172.21.2.112:0/2271121705 shutdown_connections 2021-02-10T11:39:44.751 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.741 7fd0e25a7700 1 -- 172.21.2.112:0/2271121705 wait complete. 2021-02-10T11:39:44.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.755 7fd0e25a7700 1 Processor -- start 2021-02-10T11:39:44.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.756 7fd0e25a7700 1 -- start start 2021-02-10T11:39:44.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.756 7fd0e25a7700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd0dc0b4e00 0x7fd0dc0a0fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:44.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.756 7fd0e25a7700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd0dc0a3a60 0x7fd0dc0a3e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:44.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.756 7fd0e25a7700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd0dc094750 0x7fd0dc094b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:44.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.756 7fd0e25a7700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0dc0a4550 con 0x7fd0dc094750 2021-02-10T11:39:44.754 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.756 7fd0e25a7700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0dc18bf30 con 0x7fd0dc0b4e00 2021-02-10T11:39:44.754 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.756 7fd0e25a7700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0dc18ba60 con 0x7fd0dc0a3a60 2021-02-10T11:39:44.754 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.756 7fd0e0bc4700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd0dc094750 0x7fd0dc094b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:44.754 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.756 7fd0e0bc4700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd0dc094750 0x7fd0dc094b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:54040/0 (socket says 172.21.2.112:54040) 2021-02-10T11:39:44.754 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.756 7fd0e0bc4700 1 -- 172.21.2.112:0/2923629241 learned_addr learned my addr 172.21.2.112:0/2923629241 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:39:44.754 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.756 7fd0db7fe700 1 --2- 172.21.2.112:0/2923629241 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd0dc0a3a60 0x7fd0dc0a3e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:44.754 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.757 7fd0dbfff700 1 --2- 172.21.2.112:0/2923629241 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd0dc0b4e00 0x7fd0dc0a0fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:44.754 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.757 7fd0e0bc4700 1 -- 172.21.2.112:0/2923629241 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd0dc0a3a60 msgr2=0x7fd0dc0a3e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:44.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.757 7fd0e0bc4700 1 --2- 172.21.2.112:0/2923629241 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd0dc0a3a60 0x7fd0dc0a3e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:44.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.757 7fd0e0bc4700 1 -- 172.21.2.112:0/2923629241 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd0dc0b4e00 msgr2=0x7fd0dc0a0fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:44.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.757 7fd0e0bc4700 1 --2- 172.21.2.112:0/2923629241 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd0dc0b4e00 0x7fd0dc0a0fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:44.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.757 7fd0e0bc4700 1 -- 172.21.2.112:0/2923629241 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0dc0a15c0 con 0x7fd0dc094750 2021-02-10T11:39:44.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.757 7fd0dbfff700 1 --2- 172.21.2.112:0/2923629241 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd0dc0b4e00 0x7fd0dc0a0fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:39:44.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.757 7fd0db7fe700 1 --2- 172.21.2.112:0/2923629241 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd0dc0a3a60 0x7fd0dc0a3e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:39:44.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.757 7fd0e0bc4700 1 --2- 172.21.2.112:0/2923629241 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd0dc094750 0x7fd0dc094b50 secure :-1 s=READY pgs=1966 cs=0 l=1 rev1=1 rx=0x7fd0d40099f0 tx=0x7fd0d400bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:44.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.757 7fd0d8ff9700 1 -- 172.21.2.112:0/2923629241 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd0d4009170 con 0x7fd0dc094750 2021-02-10T11:39:44.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.757 7fd0d8ff9700 1 -- 172.21.2.112:0/2923629241 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0d4009430 con 0x7fd0dc094750 2021-02-10T11:39:44.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.757 7fd0e25a7700 1 -- 172.21.2.112:0/2923629241 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0dc0a15c0 con 0x7fd0dc094750 2021-02-10T11:39:44.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.757 7fd0d8ff9700 1 -- 172.21.2.112:0/2923629241 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd0d400c570 con 0x7fd0dc094750 2021-02-10T11:39:44.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.757 7fd0e25a7700 1 -- 172.21.2.112:0/2923629241 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0dc095460 con 0x7fd0dc094750 2021-02-10T11:39:44.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.758 7fd0d8ff9700 1 -- 172.21.2.112:0/2923629241 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd0d400c2f0 con 0x7fd0dc094750 2021-02-10T11:39:44.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.758 7fd0d8ff9700 1 --2- 172.21.2.112:0/2923629241 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd0bc0186c0 0x7fd0bc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:44.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.758 7fd0d8ff9700 1 -- 172.21.2.112:0/2923629241 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fd0d4032900 con 0x7fd0dc094750 2021-02-10T11:39:44.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.758 7fd0dbfff700 1 --2- 172.21.2.112:0/2923629241 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd0bc0186c0 0x7fd0bc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:44.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.759 7fd0dbfff700 1 --2- 172.21.2.112:0/2923629241 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd0bc0186c0 0x7fd0bc01abc0 crc :-1 s=READY pgs=917 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:44.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.789 7fd0e25a7700 1 -- 172.21.2.112:0/2923629241 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0dc0ee0e0 con 0x7fd0dc094750 2021-02-10T11:39:44.786 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.792 7fd0d8ff9700 1 -- 172.21.2.112:0/2923629241 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd0d402cff0 con 0x7fd0dc094750 2021-02-10T11:39:44.910 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.916 7fd0e25a7700 1 -- 172.21.2.112:0/2923629241 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd0dc0ce450 con 0x7fd0dc094750 2021-02-10T11:39:44.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.916 7fd0d8ff9700 1 -- 172.21.2.112:0/2923629241 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v99) v1 ==== 74+0+14009 (secure 0 0 0) 0x7fd0d402cff0 con 0x7fd0dc094750 2021-02-10T11:39:44.911 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:39:44.911 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":99,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:39:44.311273","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:39:42.230237","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"99","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":21,"pool_name":"cephfs_data","create_time":"2021-02-10 11:39:43.354655","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"99","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:39:44.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.918 7fd0e25a7700 1 -- 172.21.2.112:0/2923629241 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd0bc0186c0 msgr2=0x7fd0bc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:44.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.918 7fd0e25a7700 1 --2- 172.21.2.112:0/2923629241 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd0bc0186c0 0x7fd0bc01abc0 crc :-1 s=READY pgs=917 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:44.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.918 7fd0e25a7700 1 -- 172.21.2.112:0/2923629241 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd0dc094750 msgr2=0x7fd0dc094b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:44.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.918 7fd0e25a7700 1 --2- 172.21.2.112:0/2923629241 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd0dc094750 0x7fd0dc094b50 secure :-1 s=READY pgs=1966 cs=0 l=1 rev1=1 rx=0x7fd0d40099f0 tx=0x7fd0d400bd10).stop 2021-02-10T11:39:44.913 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.918 7fd0e25a7700 1 -- 172.21.2.112:0/2923629241 shutdown_connections 2021-02-10T11:39:44.913 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.918 7fd0e25a7700 1 --2- 172.21.2.112:0/2923629241 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd0bc0186c0 0x7fd0bc01abc0 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:44.913 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.918 7fd0e25a7700 1 --2- 172.21.2.112:0/2923629241 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd0dc094750 0x7fd0dc094b50 unknown :-1 s=CLOSED pgs=1966 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:44.913 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.918 7fd0e25a7700 1 --2- 172.21.2.112:0/2923629241 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd0dc0a3a60 0x7fd0dc0a3e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:44.913 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.918 7fd0e25a7700 1 --2- 172.21.2.112:0/2923629241 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd0dc0b4e00 0x7fd0dc0a0fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:44.913 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.918 7fd0e25a7700 1 -- 172.21.2.112:0/2923629241 >> 172.21.2.112:0/2923629241 conn(0x7fd0dc18cab0 msgr2=0x7fd0dc092c60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:44.913 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.918 7fd0e25a7700 1 -- 172.21.2.112:0/2923629241 shutdown_connections 2021-02-10T11:39:44.913 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:44.918 7fd0e25a7700 1 -- 172.21.2.112:0/2923629241 wait complete. 2021-02-10T11:39:44.921 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:39:45.350 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.339 7fb263d43700 1 Processor -- start 2021-02-10T11:39:45.350 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.339 7fb263d43700 1 -- start start 2021-02-10T11:39:45.350 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.340 7fb263d43700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb25c0d1870 0x7fb25c1b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:45.350 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.340 7fb263d43700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fb25c1b0a10 0x7fb25c0cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:45.350 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.340 7fb263d43700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb25c0f0b10 con 0x7fb25c0d1c70 2021-02-10T11:39:45.351 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.340 7fb263d43700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb25c0fe230 con 0x7fb25c1b0a10 2021-02-10T11:39:45.351 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.340 7fb263d43700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb25c0e5ed0 con 0x7fb25c0d1870 2021-02-10T11:39:45.351 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.340 7fb261b5f700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb25c0d1870 0x7fb25c1b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:45.351 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.340 7fb262360700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7fb25c0d1c70 0x7fb25c0cf950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:47064/0 (socket says 172.21.2.112:47064) 2021-02-10T11:39:45.351 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.340 7fb262360700 1 -- 172.21.2.112:0/1289095438 learned_addr learned my addr 172.21.2.112:0/1289095438 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:39:45.351 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.340 7fb26135e700 1 --2- 172.21.2.112:0/1289095438 >> v2:172.21.2.120:3300/0 conn(0x7fb25c1b0a10 0x7fb25c0cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:45.351 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.341 7fb261b5f700 1 -- 172.21.2.112:0/1289095438 >> v2:172.21.2.120:3300/0 conn(0x7fb25c1b0a10 msgr2=0x7fb25c0cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:45.351 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.341 7fb261b5f700 1 --2- 172.21.2.112:0/1289095438 >> v2:172.21.2.120:3300/0 conn(0x7fb25c1b0a10 0x7fb25c0cf330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:45.351 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.341 7fb261b5f700 1 -- 172.21.2.112:0/1289095438 >> v1:172.21.2.120:6789/0 conn(0x7fb25c0d1c70 legacy=0x7fb25c0cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:45.351 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.341 7fb261b5f700 1 -- 172.21.2.112:0/1289095438 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb25c0f0650 con 0x7fb25c0d1870 2021-02-10T11:39:45.351 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.341 7fb261b5f700 1 --2- 172.21.2.112:0/1289095438 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb25c0d1870 0x7fb25c1b03f0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7fb258006650 tx=0x7fb258009a80).ready entity=mon.2 client_cookie=a827f8b40fd4b9b5 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:45.351 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.341 7fb260b5d700 1 -- 172.21.2.112:0/1289095438 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb25800c0c0 con 0x7fb25c0d1870 2021-02-10T11:39:45.351 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.341 7fb260b5d700 1 -- 172.21.2.112:0/1289095438 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb25800c340 con 0x7fb25c0d1870 2021-02-10T11:39:45.352 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.341 7fb260b5d700 1 -- 172.21.2.112:0/1289095438 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb25800c860 con 0x7fb25c0d1870 2021-02-10T11:39:45.352 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.341 7fb263d43700 1 -- 172.21.2.112:0/1289095438 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb25c0d1870 msgr2=0x7fb25c1b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:45.352 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.341 7fb263d43700 1 --2- 172.21.2.112:0/1289095438 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb25c0d1870 0x7fb25c1b03f0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7fb258006650 tx=0x7fb258009a80).stop 2021-02-10T11:39:45.352 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.342 7fb263d43700 1 -- 172.21.2.112:0/1289095438 shutdown_connections 2021-02-10T11:39:45.352 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.342 7fb263d43700 1 --2- 172.21.2.112:0/1289095438 >> v2:172.21.2.120:3300/0 conn(0x7fb25c1b0a10 0x7fb25c0cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:45.352 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.342 7fb263d43700 1 --2- 172.21.2.112:0/1289095438 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb25c0d1870 0x7fb25c1b03f0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:45.352 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.342 7fb263d43700 1 -- 172.21.2.112:0/1289095438 >> 172.21.2.112:0/1289095438 conn(0x7fb25c0ce450 msgr2=0x7fb25c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:45.352 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.342 7fb263d43700 1 -- 172.21.2.112:0/1289095438 shutdown_connections 2021-02-10T11:39:45.352 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.342 7fb263d43700 1 -- 172.21.2.112:0/1289095438 wait complete. 2021-02-10T11:39:45.354 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.356 7fb263d43700 1 Processor -- start 2021-02-10T11:39:45.354 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.357 7fb263d43700 1 -- start start 2021-02-10T11:39:45.354 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.357 7fb263d43700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb25c195650 0x7fb25c0b5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:45.354 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.357 7fb263d43700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb25c0b5b80 0x7fb25c1a2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:45.354 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.357 7fb263d43700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb25c0acfa0 0x7fb25c1a3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:45.354 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.357 7fb263d43700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb25c0b6010 con 0x7fb25c0b5b80 2021-02-10T11:39:45.354 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.357 7fb263d43700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb25c1a3a60 con 0x7fb25c0acfa0 2021-02-10T11:39:45.354 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.357 7fb263d43700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb25c1a3c40 con 0x7fb25c195650 2021-02-10T11:39:45.354 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.357 7fb26135e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb25c0b5b80 0x7fb25c1a2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:45.354 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.357 7fb26135e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb25c0b5b80 0x7fb25c1a2e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:54052/0 (socket says 172.21.2.112:54052) 2021-02-10T11:39:45.354 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.357 7fb26135e700 1 -- 172.21.2.112:0/3180209025 learned_addr learned my addr 172.21.2.112:0/3180209025 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:39:45.355 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.357 7fb261b5f700 1 --2- 172.21.2.112:0/3180209025 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb25c195650 0x7fb25c0b5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:45.355 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.358 7fb262360700 1 --2- 172.21.2.112:0/3180209025 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb25c0acfa0 0x7fb25c1a3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:45.355 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.358 7fb26135e700 1 -- 172.21.2.112:0/3180209025 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb25c195650 msgr2=0x7fb25c0b5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:45.355 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.358 7fb26135e700 1 --2- 172.21.2.112:0/3180209025 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb25c195650 0x7fb25c0b5560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:45.355 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.358 7fb26135e700 1 -- 172.21.2.112:0/3180209025 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb25c0acfa0 msgr2=0x7fb25c1a3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:45.355 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.358 7fb26135e700 1 --2- 172.21.2.112:0/3180209025 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb25c0acfa0 0x7fb25c1a3440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:45.355 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.358 7fb26135e700 1 -- 172.21.2.112:0/3180209025 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb25c0bcae0 con 0x7fb25c0b5b80 2021-02-10T11:39:45.355 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.358 7fb261b5f700 1 --2- 172.21.2.112:0/3180209025 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb25c195650 0x7fb25c0b5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:39:45.355 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.358 7fb26135e700 1 --2- 172.21.2.112:0/3180209025 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb25c0b5b80 0x7fb25c1a2e20 secure :-1 s=READY pgs=1967 cs=0 l=1 rev1=1 rx=0x7fb250006f60 tx=0x7fb25000b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:45.355 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.358 7fb24e7fc700 1 -- 172.21.2.112:0/3180209025 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb25000bd60 con 0x7fb25c0b5b80 2021-02-10T11:39:45.355 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.358 7fb24e7fc700 1 -- 172.21.2.112:0/3180209025 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb250000c90 con 0x7fb25c0b5b80 2021-02-10T11:39:45.355 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.358 7fb24e7fc700 1 -- 172.21.2.112:0/3180209025 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb250002220 con 0x7fb25c0b5b80 2021-02-10T11:39:45.355 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.358 7fb263d43700 1 -- 172.21.2.112:0/3180209025 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb25c0bcae0 con 0x7fb25c0b5b80 2021-02-10T11:39:45.356 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.359 7fb263d43700 1 -- 172.21.2.112:0/3180209025 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb25c0bcae0 con 0x7fb25c0b5b80 2021-02-10T11:39:45.356 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.359 7fb24e7fc700 1 -- 172.21.2.112:0/3180209025 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb2500052e0 con 0x7fb25c0b5b80 2021-02-10T11:39:45.356 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.359 7fb24e7fc700 1 --2- 172.21.2.112:0/3180209025 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb2440186d0 0x7fb24401abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:45.356 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.359 7fb24e7fc700 1 -- 172.21.2.112:0/3180209025 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fb250032640 con 0x7fb25c0b5b80 2021-02-10T11:39:45.356 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.359 7fb261b5f700 1 --2- 172.21.2.112:0/3180209025 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb2440186d0 0x7fb24401abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:45.356 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.360 7fb261b5f700 1 --2- 172.21.2.112:0/3180209025 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb2440186d0 0x7fb24401abd0 crc :-1 s=READY pgs=918 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:45.384 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.390 7fb263d43700 1 -- 172.21.2.112:0/3180209025 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb25c0b6010 con 0x7fb25c0b5b80 2021-02-10T11:39:45.386 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.392 7fb24e7fc700 1 -- 172.21.2.112:0/3180209025 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb250006b00 con 0x7fb25c0b5b80 2021-02-10T11:39:45.512 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.518 7fb263d43700 1 -- 172.21.2.112:0/3180209025 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb25c0b6010 con 0x7fb25c0b5b80 2021-02-10T11:39:45.512 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.519 7fb24e7fc700 1 -- 172.21.2.112:0/3180209025 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v100) v1 ==== 74+0+13998 (secure 0 0 0) 0x7fb2500052e0 con 0x7fb25c0b5b80 2021-02-10T11:39:45.513 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:39:45.513 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":100,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:39:45.313226","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:39:42.230237","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"99","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":21,"pool_name":"cephfs_data","create_time":"2021-02-10 11:39:43.354655","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:39:45.514 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.520 7fb263d43700 1 -- 172.21.2.112:0/3180209025 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb2440186d0 msgr2=0x7fb24401abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:45.515 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.520 7fb263d43700 1 --2- 172.21.2.112:0/3180209025 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb2440186d0 0x7fb24401abd0 crc :-1 s=READY pgs=918 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:45.515 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.520 7fb263d43700 1 -- 172.21.2.112:0/3180209025 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb25c0b5b80 msgr2=0x7fb25c1a2e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:45.515 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.520 7fb263d43700 1 --2- 172.21.2.112:0/3180209025 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb25c0b5b80 0x7fb25c1a2e20 secure :-1 s=READY pgs=1967 cs=0 l=1 rev1=1 rx=0x7fb250006f60 tx=0x7fb25000b970).stop 2021-02-10T11:39:45.515 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.520 7fb263d43700 1 -- 172.21.2.112:0/3180209025 shutdown_connections 2021-02-10T11:39:45.515 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.520 7fb263d43700 1 --2- 172.21.2.112:0/3180209025 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb2440186d0 0x7fb24401abd0 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:45.515 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.520 7fb263d43700 1 --2- 172.21.2.112:0/3180209025 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb25c0acfa0 0x7fb25c1a3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:45.515 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.520 7fb263d43700 1 --2- 172.21.2.112:0/3180209025 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb25c0b5b80 0x7fb25c1a2e20 unknown :-1 s=CLOSED pgs=1967 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:45.515 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.520 7fb263d43700 1 --2- 172.21.2.112:0/3180209025 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb25c195650 0x7fb25c0b5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:45.515 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.520 7fb263d43700 1 -- 172.21.2.112:0/3180209025 >> 172.21.2.112:0/3180209025 conn(0x7fb25c0b3040 msgr2=0x7fb25c0c3250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:45.515 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.521 7fb263d43700 1 -- 172.21.2.112:0/3180209025 shutdown_connections 2021-02-10T11:39:45.515 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.521 7fb263d43700 1 -- 172.21.2.112:0/3180209025 wait complete. 2021-02-10T11:39:45.523 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2021-02-10T11:39:45.954 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.943 7f6e597ab700 1 Processor -- start 2021-02-10T11:39:45.954 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.943 7f6e597ab700 1 -- start start 2021-02-10T11:39:45.954 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.944 7f6e597ab700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f6e540d1870 0x7f6e541b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:45.954 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.944 7f6e597ab700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6e541b0ae0 0x7f6e54108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:45.955 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.944 7f6e597ab700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e540fe1c0 con 0x7f6e540d1c70 2021-02-10T11:39:45.955 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.944 7f6e597ab700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e54109a60 con 0x7f6e540d1870 2021-02-10T11:39:45.955 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.944 7f6e597ab700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e54109750 con 0x7f6e541b0ae0 2021-02-10T11:39:45.955 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.944 7f6e53574700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f6e540d1870 0x7f6e541b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:45.955 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.944 7f6e53574700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f6e540d1870 0x7f6e541b04c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:54058/0 (socket says 172.21.2.112:54058) 2021-02-10T11:39:45.955 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.944 7f6e53574700 1 -- 172.21.2.112:0/2440427789 learned_addr learned my addr 172.21.2.112:0/2440427789 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:39:45.955 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.944 7f6e53574700 1 -- 172.21.2.112:0/2440427789 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6e541b0ae0 msgr2=0x7f6e54108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:45.955 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.944 7f6e53574700 1 --2- 172.21.2.112:0/2440427789 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6e541b0ae0 0x7f6e54108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:45.955 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.944 7f6e53574700 1 -- 172.21.2.112:0/2440427789 >> v1:172.21.2.120:6789/0 conn(0x7f6e540d1c70 legacy=0x7f6e54108f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:45.955 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.944 7f6e53574700 1 -- 172.21.2.112:0/2440427789 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e54088070 con 0x7f6e540d1870 2021-02-10T11:39:45.955 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.944 7f6e53574700 1 --2- 172.21.2.112:0/2440427789 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6e540d1870 0x7f6e541b04c0 secure :-1 s=READY pgs=1968 cs=0 l=1 rev1=1 rx=0x7f6e3c006700 tx=0x7f6e3c009b80).ready entity=mon.0 client_cookie=9074d9d60d41d994 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:45.955 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.945 7f6e52572700 1 -- 172.21.2.112:0/2440427789 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6e3c00c0c0 con 0x7f6e540d1870 2021-02-10T11:39:45.955 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.945 7f6e52572700 1 -- 172.21.2.112:0/2440427789 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e3c00c440 con 0x7f6e540d1870 2021-02-10T11:39:45.956 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.945 7f6e52572700 1 -- 172.21.2.112:0/2440427789 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6e3c00c8a0 con 0x7f6e540d1870 2021-02-10T11:39:45.956 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.945 7f6e597ab700 1 -- 172.21.2.112:0/2440427789 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6e540d1870 msgr2=0x7f6e541b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:45.956 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.945 7f6e597ab700 1 --2- 172.21.2.112:0/2440427789 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6e540d1870 0x7f6e541b04c0 secure :-1 s=READY pgs=1968 cs=0 l=1 rev1=1 rx=0x7f6e3c006700 tx=0x7f6e3c009b80).stop 2021-02-10T11:39:45.956 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.946 7f6e597ab700 1 -- 172.21.2.112:0/2440427789 shutdown_connections 2021-02-10T11:39:45.956 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.946 7f6e597ab700 1 --2- 172.21.2.112:0/2440427789 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6e541b0ae0 0x7f6e54108940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:45.956 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.946 7f6e597ab700 1 --2- 172.21.2.112:0/2440427789 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6e540d1870 0x7f6e541b04c0 unknown :-1 s=CLOSED pgs=1968 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:45.956 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.946 7f6e597ab700 1 -- 172.21.2.112:0/2440427789 >> 172.21.2.112:0/2440427789 conn(0x7f6e540ce420 msgr2=0x7f6e540d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:45.956 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.946 7f6e597ab700 1 -- 172.21.2.112:0/2440427789 shutdown_connections 2021-02-10T11:39:45.956 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.946 7f6e597ab700 1 -- 172.21.2.112:0/2440427789 wait complete. 2021-02-10T11:39:45.956 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.960 7f6e597ab700 1 Processor -- start 2021-02-10T11:39:45.956 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.960 7f6e597ab700 1 -- start start 2021-02-10T11:39:45.958 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.961 7f6e597ab700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6e540c3820 0x7f6e54084cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:45.958 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.961 7f6e597ab700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6e540a0290 0x7f6e540852d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:45.958 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.961 7f6e597ab700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6e54086270 0x7f6e54086670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:45.958 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.961 7f6e597ab700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e540c0700 con 0x7f6e540c3820 2021-02-10T11:39:45.958 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.961 7f6e597ab700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e540858f0 con 0x7f6e54086270 2021-02-10T11:39:45.958 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.961 7f6e597ab700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e54085b10 con 0x7f6e540a0290 2021-02-10T11:39:45.958 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.961 7f6e53574700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6e540c3820 0x7f6e54084cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:45.958 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.961 7f6e53574700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6e540c3820 0x7f6e54084cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:54066/0 (socket says 172.21.2.112:54066) 2021-02-10T11:39:45.958 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.961 7f6e53574700 1 -- 172.21.2.112:0/3294758754 learned_addr learned my addr 172.21.2.112:0/3294758754 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:39:45.959 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.961 7f6e52d73700 1 --2- 172.21.2.112:0/3294758754 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6e540a0290 0x7f6e540852d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:45.959 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.961 7f6e53d75700 1 --2- 172.21.2.112:0/3294758754 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6e54086270 0x7f6e54086670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:45.959 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.961 7f6e53574700 1 -- 172.21.2.112:0/3294758754 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6e540a0290 msgr2=0x7f6e540852d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:45.959 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.961 7f6e53574700 1 --2- 172.21.2.112:0/3294758754 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6e540a0290 0x7f6e540852d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:45.959 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.962 7f6e53574700 1 -- 172.21.2.112:0/3294758754 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6e54086270 msgr2=0x7f6e54086670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:45.959 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.962 7f6e53574700 1 --2- 172.21.2.112:0/3294758754 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6e54086270 0x7f6e54086670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:45.959 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.962 7f6e53574700 1 -- 172.21.2.112:0/3294758754 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e54086c90 con 0x7f6e540c3820 2021-02-10T11:39:45.959 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.962 7f6e52d73700 1 --2- 172.21.2.112:0/3294758754 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6e540a0290 0x7f6e540852d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:39:45.959 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.962 7f6e53d75700 1 --2- 172.21.2.112:0/3294758754 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6e54086270 0x7f6e54086670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:39:45.959 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.962 7f6e53574700 1 --2- 172.21.2.112:0/3294758754 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6e540c3820 0x7f6e54084cb0 secure :-1 s=READY pgs=1969 cs=0 l=1 rev1=1 rx=0x7f6e3c00b5e0 tx=0x7f6e3c00b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:45.959 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.962 7f6e4bfff700 1 -- 172.21.2.112:0/3294758754 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6e3c00bd70 con 0x7f6e540c3820 2021-02-10T11:39:45.959 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.962 7f6e4bfff700 1 -- 172.21.2.112:0/3294758754 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e3c002030 con 0x7f6e540c3820 2021-02-10T11:39:45.959 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.962 7f6e4bfff700 1 -- 172.21.2.112:0/3294758754 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6e3c0024d0 con 0x7f6e540c3820 2021-02-10T11:39:45.960 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.962 7f6e597ab700 1 -- 172.21.2.112:0/3294758754 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e54086c90 con 0x7f6e540c3820 2021-02-10T11:39:45.960 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.962 7f6e597ab700 1 -- 172.21.2.112:0/3294758754 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e541a2f00 con 0x7f6e540c3820 2021-02-10T11:39:45.960 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.963 7f6e4bfff700 1 -- 172.21.2.112:0/3294758754 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6e3c000c90 con 0x7f6e540c3820 2021-02-10T11:39:45.960 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.963 7f6e4bfff700 1 --2- 172.21.2.112:0/3294758754 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6e380186c0 0x7f6e3801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:45.960 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.963 7f6e4bfff700 1 -- 172.21.2.112:0/3294758754 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f6e3c032450 con 0x7f6e540c3820 2021-02-10T11:39:45.960 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.963 7f6e52d73700 1 --2- 172.21.2.112:0/3294758754 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6e380186c0 0x7f6e3801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:45.960 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.964 7f6e52d73700 1 --2- 172.21.2.112:0/3294758754 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6e380186c0 0x7f6e3801abc0 crc :-1 s=READY pgs=919 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:45.987 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.993 7f6e597ab700 1 -- 172.21.2.112:0/3294758754 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e54082a70 con 0x7f6e540c3820 2021-02-10T11:39:45.990 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:45.996 7f6e4bfff700 1 -- 172.21.2.112:0/3294758754 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6e3c030d80 con 0x7f6e540c3820 2021-02-10T11:39:46.127 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:46.133 7f6e597ab700 1 -- 172.21.2.112:0/3294758754 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"} v 0) v1 -- 0x7f6e54082a70 con 0x7f6e540c3820 2021-02-10T11:39:46.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:46.323 7f6e4bfff700 1 -- 172.21.2.112:0/3294758754 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"}]=0 v80) v1 ==== 118+0+0 (secure 0 0 0) 0x7f6e3c000c90 con 0x7f6e540c3820 2021-02-10T11:39:46.344 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:46.350 7f6e597ab700 1 -- 172.21.2.112:0/3294758754 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6e380186c0 msgr2=0x7f6e3801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:46.345 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:46.350 7f6e597ab700 1 --2- 172.21.2.112:0/3294758754 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6e380186c0 0x7f6e3801abc0 crc :-1 s=READY pgs=919 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:46.345 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:46.350 7f6e597ab700 1 -- 172.21.2.112:0/3294758754 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6e540c3820 msgr2=0x7f6e54084cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:46.345 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:46.350 7f6e597ab700 1 --2- 172.21.2.112:0/3294758754 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6e540c3820 0x7f6e54084cb0 secure :-1 s=READY pgs=1969 cs=0 l=1 rev1=1 rx=0x7f6e3c00b5e0 tx=0x7f6e3c00b980).stop 2021-02-10T11:39:46.345 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:46.351 7f6e597ab700 1 -- 172.21.2.112:0/3294758754 shutdown_connections 2021-02-10T11:39:46.345 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:46.351 7f6e597ab700 1 --2- 172.21.2.112:0/3294758754 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6e380186c0 0x7f6e3801abc0 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:46.345 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:46.351 7f6e597ab700 1 --2- 172.21.2.112:0/3294758754 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6e54086270 0x7f6e54086670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:46.345 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:46.351 7f6e597ab700 1 --2- 172.21.2.112:0/3294758754 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6e540a0290 0x7f6e540852d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:46.345 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:46.351 7f6e597ab700 1 --2- 172.21.2.112:0/3294758754 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6e540c3820 0x7f6e54084cb0 unknown :-1 s=CLOSED pgs=1969 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:46.345 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:46.351 7f6e597ab700 1 -- 172.21.2.112:0/3294758754 >> 172.21.2.112:0/3294758754 conn(0x7f6e540872c0 msgr2=0x7f6e54095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:46.346 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:46.351 7f6e597ab700 1 -- 172.21.2.112:0/3294758754 shutdown_connections 2021-02-10T11:39:46.346 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:46.351 7f6e597ab700 1 -- 172.21.2.112:0/3294758754 wait complete. 2021-02-10T11:39:46.353 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:39:46.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:46.770 7f9b1fb5f700 1 Processor -- start 2021-02-10T11:39:46.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:46.770 7f9b1fb5f700 1 -- start start 2021-02-10T11:39:46.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:46.770 7f9b1fb5f700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f9b180d0340 0x7f9b180ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:46.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:46.770 7f9b1fb5f700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f9b180d1c70 0x7f9b180efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:46.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:46.770 7f9b1fb5f700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b180f0b10 con 0x7f9b180d1870 2021-02-10T11:39:46.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:46.770 7f9b1fb5f700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b180e5ed0 con 0x7f9b180d0340 2021-02-10T11:39:46.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:46.770 7f9b1fb5f700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b181b0be0 con 0x7f9b180d1c70 2021-02-10T11:39:46.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:46.771 7f9b1d17a700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f9b180d0340 0x7f9b180ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:46.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:46.771 7f9b1d17a700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f9b180d0340 0x7f9b180ef640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:54072/0 (socket says 172.21.2.112:54072) 2021-02-10T11:39:46.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:46.771 7f9b1d17a700 1 -- 172.21.2.112:0/2413832271 learned_addr learned my addr 172.21.2.112:0/2413832271 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:39:46.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:46.771 7f9b1d17a700 1 -- 172.21.2.112:0/2413832271 >> v2:172.21.2.120:3300/0 conn(0x7f9b180d1c70 msgr2=0x7f9b180efc60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:39:46.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:46.771 7f9b1d17a700 1 --2- 172.21.2.112:0/2413832271 >> v2:172.21.2.120:3300/0 conn(0x7f9b180d1c70 0x7f9b180efc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:46.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:46.771 7f9b1d17a700 1 -- 172.21.2.112:0/2413832271 >> v1:172.21.2.120:6789/0 conn(0x7f9b180d1870 legacy=0x7f9b181b03f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:39:46.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:46.771 7f9b1d17a700 1 -- 172.21.2.112:0/2413832271 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b180f20a0 con 0x7f9b180d0340 2021-02-10T11:39:46.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:46.771 7f9b1d17a700 1 --2- 172.21.2.112:0/2413832271 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9b180d0340 0x7f9b180ef640 secure :-1 s=READY pgs=1970 cs=0 l=1 rev1=1 rx=0x7f9b14006700 tx=0x7f9b14009b00).ready entity=mon.0 client_cookie=46627de83ee76bdb server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:46.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:46.771 7f9b1c979700 1 -- 172.21.2.112:0/2413832271 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9b1400c0c0 con 0x7f9b180d0340 2021-02-10T11:39:46.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:46.771 7f9b1c979700 1 -- 172.21.2.112:0/2413832271 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b1400c400 con 0x7f9b180d0340 2021-02-10T11:39:46.782 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:46.771 7f9b1c979700 1 -- 172.21.2.112:0/2413832271 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9b1400c860 con 0x7f9b180d0340 2021-02-10T11:39:46.782 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:46.771 7f9b1fb5f700 1 -- 172.21.2.112:0/2413832271 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9b180d0340 msgr2=0x7f9b180ef640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:46.782 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:46.771 7f9b1fb5f700 1 --2- 172.21.2.112:0/2413832271 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9b180d0340 0x7f9b180ef640 secure :-1 s=READY pgs=1970 cs=0 l=1 rev1=1 rx=0x7f9b14006700 tx=0x7f9b14009b00).stop 2021-02-10T11:39:46.782 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:46.772 7f9b1fb5f700 1 -- 172.21.2.112:0/2413832271 shutdown_connections 2021-02-10T11:39:46.782 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:46.772 7f9b1fb5f700 1 --2- 172.21.2.112:0/2413832271 >> v2:172.21.2.120:3300/0 conn(0x7f9b180d1c70 0x7f9b180efc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:46.782 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:46.772 7f9b1fb5f700 1 --2- 172.21.2.112:0/2413832271 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9b180d0340 0x7f9b180ef640 unknown :-1 s=CLOSED pgs=1970 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:46.782 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:46.772 7f9b1fb5f700 1 -- 172.21.2.112:0/2413832271 >> 172.21.2.112:0/2413832271 conn(0x7f9b180ce450 msgr2=0x7f9b180d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:46.782 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:46.772 7f9b1fb5f700 1 -- 172.21.2.112:0/2413832271 shutdown_connections 2021-02-10T11:39:46.782 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:46.772 7f9b1fb5f700 1 -- 172.21.2.112:0/2413832271 wait complete. 2021-02-10T11:39:46.782 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:46.786 7f9b1fb5f700 1 Processor -- start 2021-02-10T11:39:46.782 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:46.786 7f9b1fb5f700 1 -- start start 2021-02-10T11:39:46.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:46.787 7f9b1fb5f700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9b1818f0a0 0x7f9b18196ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:46.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:46.787 7f9b1fb5f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9b181970d0 0x7f9b181974d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:46.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:46.787 7f9b1fb5f700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9b18094650 0x7f9b18094a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:46.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:46.787 7f9b1fb5f700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b18095bd0 con 0x7f9b181970d0 2021-02-10T11:39:46.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:46.787 7f9b1fb5f700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b180ad7c0 con 0x7f9b18094650 2021-02-10T11:39:46.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:46.787 7f9b1fb5f700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b1818ec70 con 0x7f9b1818f0a0 2021-02-10T11:39:46.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:46.787 7f9b1d17a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9b181970d0 0x7f9b181974d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:46.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:46.787 7f9b1d17a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9b181970d0 0x7f9b181974d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:54082/0 (socket says 172.21.2.112:54082) 2021-02-10T11:39:46.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:46.787 7f9b1d17a700 1 -- 172.21.2.112:0/735541316 learned_addr learned my addr 172.21.2.112:0/735541316 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:39:46.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:46.787 7f9b1d17a700 1 -- 172.21.2.112:0/735541316 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9b1818f0a0 msgr2=0x7f9b18196ab0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:39:46.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:46.787 7f9b1d17a700 1 --2- 172.21.2.112:0/735541316 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9b1818f0a0 0x7f9b18196ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:46.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:46.787 7f9b1d17a700 1 -- 172.21.2.112:0/735541316 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9b18094650 msgr2=0x7f9b18094a50 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:39:46.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:46.787 7f9b1d17a700 1 --2- 172.21.2.112:0/735541316 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9b18094650 0x7f9b18094a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:46.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:46.787 7f9b1d17a700 1 -- 172.21.2.112:0/735541316 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b18095070 con 0x7f9b181970d0 2021-02-10T11:39:46.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:46.787 7f9b1d17a700 1 --2- 172.21.2.112:0/735541316 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9b181970d0 0x7f9b181974d0 secure :-1 s=READY pgs=1971 cs=0 l=1 rev1=1 rx=0x7f9b1400b660 tx=0x7f9b1400ba00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:46.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:46.788 7f9b0a7fc700 1 -- 172.21.2.112:0/735541316 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9b14009440 con 0x7f9b181970d0 2021-02-10T11:39:46.786 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:46.788 7f9b0a7fc700 1 -- 172.21.2.112:0/735541316 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b14006390 con 0x7f9b181970d0 2021-02-10T11:39:46.786 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:46.788 7f9b1fb5f700 1 -- 172.21.2.112:0/735541316 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b18095070 con 0x7f9b181970d0 2021-02-10T11:39:46.786 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:46.788 7f9b0a7fc700 1 -- 172.21.2.112:0/735541316 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9b14002030 con 0x7f9b181970d0 2021-02-10T11:39:46.786 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:46.788 7f9b1fb5f700 1 -- 172.21.2.112:0/735541316 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b18095070 con 0x7f9b181970d0 2021-02-10T11:39:46.786 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:46.788 7f9b0a7fc700 1 -- 172.21.2.112:0/735541316 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9b140093e0 con 0x7f9b181970d0 2021-02-10T11:39:46.786 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:46.788 7f9b0a7fc700 1 --2- 172.21.2.112:0/735541316 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9b00018730 0x7f9b0001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:46.786 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:46.788 7f9b0a7fc700 1 -- 172.21.2.112:0/735541316 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f9b14032650 con 0x7f9b181970d0 2021-02-10T11:39:46.786 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:46.789 7f9b1d97b700 1 --2- 172.21.2.112:0/735541316 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9b00018730 0x7f9b0001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:46.786 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:46.789 7f9b1d97b700 1 --2- 172.21.2.112:0/735541316 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9b00018730 0x7f9b0001ac10 crc :-1 s=READY pgs=920 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:46.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:46.820 7f9b1fb5f700 1 -- 172.21.2.112:0/735541316 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b180ad790 con 0x7f9b181970d0 2021-02-10T11:39:46.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:46.823 7f9b0a7fc700 1 -- 172.21.2.112:0/735541316 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9b14006370 con 0x7f9b181970d0 2021-02-10T11:39:46.945 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:46.951 7f9b1fb5f700 1 -- 172.21.2.112:0/735541316 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9b180ad790 con 0x7f9b181970d0 2021-02-10T11:39:46.945 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:46.952 7f9b0a7fc700 1 -- 172.21.2.112:0/735541316 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+2104 (secure 0 0 0) 0x7f9b1400bdf0 con 0x7f9b181970d0 2021-02-10T11:39:46.946 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:39:46.946 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":80,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10011,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/3938237927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":3938237927},{"type":"v1","addr":"172.21.2.112:6833","nonce":3938237927}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":76}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:39:44.311361","modified":"2021-02-10 11:39:46.320259","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7633},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7633":{"gid":7633,"name":"b","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.2.120:6835/1537197833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":1537197833},{"type":"v1","addr":"172.21.2.120:6835","nonce":1537197833}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":10}]} 2021-02-10T11:39:46.947 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 80 2021-02-10T11:39:46.948 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:46.954 7f9b1fb5f700 1 -- 172.21.2.112:0/735541316 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9b00018730 msgr2=0x7f9b0001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:46.948 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:46.954 7f9b1fb5f700 1 --2- 172.21.2.112:0/735541316 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9b00018730 0x7f9b0001ac10 crc :-1 s=READY pgs=920 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:46.948 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:46.954 7f9b1fb5f700 1 -- 172.21.2.112:0/735541316 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9b181970d0 msgr2=0x7f9b181974d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:46.948 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:46.954 7f9b1fb5f700 1 --2- 172.21.2.112:0/735541316 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9b181970d0 0x7f9b181974d0 secure :-1 s=READY pgs=1971 cs=0 l=1 rev1=1 rx=0x7f9b1400b660 tx=0x7f9b1400ba00).stop 2021-02-10T11:39:46.948 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:46.954 7f9b1fb5f700 1 -- 172.21.2.112:0/735541316 shutdown_connections 2021-02-10T11:39:46.948 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:46.954 7f9b1fb5f700 1 --2- 172.21.2.112:0/735541316 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9b00018730 0x7f9b0001ac10 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:46.948 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:46.954 7f9b1fb5f700 1 --2- 172.21.2.112:0/735541316 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9b18094650 0x7f9b18094a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:46.949 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:46.954 7f9b1fb5f700 1 --2- 172.21.2.112:0/735541316 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9b181970d0 0x7f9b181974d0 unknown :-1 s=CLOSED pgs=1971 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:46.949 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:46.954 7f9b1fb5f700 1 --2- 172.21.2.112:0/735541316 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9b1818f0a0 0x7f9b18196ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:46.949 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:46.954 7f9b1fb5f700 1 -- 172.21.2.112:0/735541316 >> 172.21.2.112:0/735541316 conn(0x7f9b181a38c0 msgr2=0x7f9b181a2f00 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:46.949 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:46.954 7f9b1fb5f700 1 -- 172.21.2.112:0/735541316 shutdown_connections 2021-02-10T11:39:46.949 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:46.954 7f9b1fb5f700 1 -- 172.21.2.112:0/735541316 wait complete. 2021-02-10T11:39:46.957 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:39:47.385 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.375 7f3fee632700 1 Processor -- start 2021-02-10T11:39:47.385 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.375 7f3fee632700 1 -- start start 2021-02-10T11:39:47.385 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.375 7f3fee632700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3fe80d1870 0x7f3fe81b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:47.385 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.375 7f3fee632700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f3fe80f0650 0x7f3fe80cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:47.386 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.375 7f3fee632700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fe80f0b10 con 0x7f3fe81b0a10 2021-02-10T11:39:47.386 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.375 7f3fee632700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fe80fe230 con 0x7f3fe80f0650 2021-02-10T11:39:47.386 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.375 7f3fee632700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fe80e5ed0 con 0x7f3fe80d1870 2021-02-10T11:39:47.386 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.376 7f3fe77fe700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7f3fe81b0a10 0x7f3fe80cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:47106/0 (socket says 172.21.2.112:47106) 2021-02-10T11:39:47.386 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.376 7f3fe7fff700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3fe80d1870 0x7f3fe81b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:47.386 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.376 7f3fe77fe700 1 -- 172.21.2.112:0/691129308 learned_addr learned my addr 172.21.2.112:0/691129308 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:39:47.386 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.376 7f3fecc4f700 1 --2- 172.21.2.112:0/691129308 >> v2:172.21.2.120:3300/0 conn(0x7f3fe80f0650 0x7f3fe80cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:47.386 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.376 7f3fecc4f700 1 -- 172.21.2.112:0/691129308 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3fe80d1870 msgr2=0x7f3fe81b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:47.386 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.376 7f3fecc4f700 1 --2- 172.21.2.112:0/691129308 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3fe80d1870 0x7f3fe81b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:47.386 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.376 7f3fecc4f700 1 -- 172.21.2.112:0/691129308 >> v1:172.21.2.120:6789/0 conn(0x7f3fe81b0a10 legacy=0x7f3fe80cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:47.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.376 7f3fecc4f700 1 -- 172.21.2.112:0/691129308 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fe80f20a0 con 0x7f3fe80f0650 2021-02-10T11:39:47.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.376 7f3fecc4f700 1 --2- 172.21.2.112:0/691129308 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3fe80f0650 0x7f3fe80cfb20 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 rx=0x7f3fe0006e50 tx=0x7f3fe0009e70).ready entity=mon.1 client_cookie=1031840984e15982 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:47.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.377 7f3fe6ffd700 1 -- 172.21.2.112:0/691129308 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3fe0006a20 con 0x7f3fe80f0650 2021-02-10T11:39:47.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.377 7f3fe6ffd700 1 -- 172.21.2.112:0/691129308 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fe0006790 con 0x7f3fe80f0650 2021-02-10T11:39:47.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.377 7f3fe6ffd700 1 -- 172.21.2.112:0/691129308 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3fe000b700 con 0x7f3fe80f0650 2021-02-10T11:39:47.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.377 7f3fee632700 1 -- 172.21.2.112:0/691129308 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3fe80f0650 msgr2=0x7f3fe80cfb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:47.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.377 7f3fee632700 1 --2- 172.21.2.112:0/691129308 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3fe80f0650 0x7f3fe80cfb20 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 rx=0x7f3fe0006e50 tx=0x7f3fe0009e70).stop 2021-02-10T11:39:47.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.377 7f3fee632700 1 -- 172.21.2.112:0/691129308 shutdown_connections 2021-02-10T11:39:47.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.377 7f3fee632700 1 --2- 172.21.2.112:0/691129308 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3fe80f0650 0x7f3fe80cfb20 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:47.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.377 7f3fee632700 1 --2- 172.21.2.112:0/691129308 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3fe80d1870 0x7f3fe81b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:47.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.377 7f3fee632700 1 -- 172.21.2.112:0/691129308 >> 172.21.2.112:0/691129308 conn(0x7f3fe80ce450 msgr2=0x7f3fe80d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:47.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.377 7f3fee632700 1 -- 172.21.2.112:0/691129308 shutdown_connections 2021-02-10T11:39:47.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.377 7f3fee632700 1 -- 172.21.2.112:0/691129308 wait complete. 2021-02-10T11:39:47.389 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.392 7f3fee632700 1 Processor -- start 2021-02-10T11:39:47.389 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.392 7f3fee632700 1 -- start start 2021-02-10T11:39:47.389 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.392 7f3fee632700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3fe80ada90 0x7f3fe80ade90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:47.389 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.392 7f3fee632700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3fe8099d20 0x7f3fe809a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:47.390 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.392 7f3fee632700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3fe809a740 0x7f3fe8097bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:47.390 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.392 7f3fee632700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fe80997a0 con 0x7f3fe809a740 2021-02-10T11:39:47.390 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.392 7f3fee632700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fe818f0c0 con 0x7f3fe8099d20 2021-02-10T11:39:47.390 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.392 7f3fee632700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fe809ab40 con 0x7f3fe80ada90 2021-02-10T11:39:47.390 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.392 7f3fecc4f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3fe809a740 0x7f3fe8097bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:47.390 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.392 7f3fecc4f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3fe809a740 0x7f3fe8097bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:54094/0 (socket says 172.21.2.112:54094) 2021-02-10T11:39:47.390 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.392 7f3fecc4f700 1 -- 172.21.2.112:0/1394409370 learned_addr learned my addr 172.21.2.112:0/1394409370 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:39:47.390 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.393 7f3fe7fff700 1 --2- 172.21.2.112:0/1394409370 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3fe80ada90 0x7f3fe80ade90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:47.390 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.393 7f3fe77fe700 1 --2- 172.21.2.112:0/1394409370 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3fe8099d20 0x7f3fe809a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:47.390 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.393 7f3fecc4f700 1 -- 172.21.2.112:0/1394409370 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3fe80ada90 msgr2=0x7f3fe80ade90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:47.390 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.393 7f3fecc4f700 1 --2- 172.21.2.112:0/1394409370 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3fe80ada90 0x7f3fe80ade90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:47.390 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.393 7f3fecc4f700 1 -- 172.21.2.112:0/1394409370 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3fe8099d20 msgr2=0x7f3fe809a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:47.391 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.393 7f3fecc4f700 1 --2- 172.21.2.112:0/1394409370 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3fe8099d20 0x7f3fe809a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:47.391 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.393 7f3fecc4f700 1 -- 172.21.2.112:0/1394409370 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fe809ad60 con 0x7f3fe809a740 2021-02-10T11:39:47.391 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.393 7f3fe77fe700 1 --2- 172.21.2.112:0/1394409370 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3fe8099d20 0x7f3fe809a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:39:47.391 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.393 7f3fecc4f700 1 --2- 172.21.2.112:0/1394409370 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3fe809a740 0x7f3fe8097bc0 secure :-1 s=READY pgs=1972 cs=0 l=1 rev1=1 rx=0x7f3fe0006e60 tx=0x7f3fe000bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:47.391 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.393 7f3fe4ff9700 1 -- 172.21.2.112:0/1394409370 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3fe0009130 con 0x7f3fe809a740 2021-02-10T11:39:47.391 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.393 7f3fe4ff9700 1 -- 172.21.2.112:0/1394409370 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fe00093f0 con 0x7f3fe809a740 2021-02-10T11:39:47.391 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.393 7f3fe4ff9700 1 -- 172.21.2.112:0/1394409370 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3fe0001390 con 0x7f3fe809a740 2021-02-10T11:39:47.391 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.393 7f3fee632700 1 -- 172.21.2.112:0/1394409370 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fe809ad60 con 0x7f3fe809a740 2021-02-10T11:39:47.391 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.393 7f3fee632700 1 -- 172.21.2.112:0/1394409370 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fe80984d0 con 0x7f3fe809a740 2021-02-10T11:39:47.391 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.394 7f3fe4ff9700 1 -- 172.21.2.112:0/1394409370 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3fe00056b0 con 0x7f3fe809a740 2021-02-10T11:39:47.391 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.394 7f3fe4ff9700 1 --2- 172.21.2.112:0/1394409370 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3fc80186c0 0x7f3fc801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:47.391 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.394 7f3fe4ff9700 1 -- 172.21.2.112:0/1394409370 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f3fe00328c0 con 0x7f3fe809a740 2021-02-10T11:39:47.391 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.394 7f3fe7fff700 1 --2- 172.21.2.112:0/1394409370 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3fc80186c0 0x7f3fc801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:47.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.395 7f3fe7fff700 1 --2- 172.21.2.112:0/1394409370 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3fc80186c0 0x7f3fc801abc0 crc :-1 s=READY pgs=921 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:47.418 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.425 7f3fee632700 1 -- 172.21.2.112:0/1394409370 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fe80ebc00 con 0x7f3fe809a740 2021-02-10T11:39:47.421 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.427 7f3fe4ff9700 1 -- 172.21.2.112:0/1394409370 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3fe00011a0 con 0x7f3fe809a740 2021-02-10T11:39:47.546 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.552 7f3fee632700 1 -- 172.21.2.112:0/1394409370 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3fe80f0b10 con 0x7f3fe809a740 2021-02-10T11:39:47.546 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.553 7f3fe4ff9700 1 -- 172.21.2.112:0/1394409370 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v100) v1 ==== 74+0+13998 (secure 0 0 0) 0x7f3fe00056b0 con 0x7f3fe809a740 2021-02-10T11:39:47.547 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:39:47.547 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":100,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:39:45.313226","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:39:42.230237","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"99","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":21,"pool_name":"cephfs_data","create_time":"2021-02-10 11:39:43.354655","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:39:47.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.554 7f3fee632700 1 -- 172.21.2.112:0/1394409370 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3fc80186c0 msgr2=0x7f3fc801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:47.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.554 7f3fee632700 1 --2- 172.21.2.112:0/1394409370 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3fc80186c0 0x7f3fc801abc0 crc :-1 s=READY pgs=921 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:47.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.554 7f3fee632700 1 -- 172.21.2.112:0/1394409370 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3fe809a740 msgr2=0x7f3fe8097bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:47.549 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.554 7f3fee632700 1 --2- 172.21.2.112:0/1394409370 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3fe809a740 0x7f3fe8097bc0 secure :-1 s=READY pgs=1972 cs=0 l=1 rev1=1 rx=0x7f3fe0006e60 tx=0x7f3fe000bcf0).stop 2021-02-10T11:39:47.549 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.554 7f3fee632700 1 -- 172.21.2.112:0/1394409370 shutdown_connections 2021-02-10T11:39:47.549 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.554 7f3fee632700 1 --2- 172.21.2.112:0/1394409370 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3fc80186c0 0x7f3fc801abc0 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:47.549 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.554 7f3fee632700 1 --2- 172.21.2.112:0/1394409370 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3fe8099d20 0x7f3fe809a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:47.549 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.554 7f3fee632700 1 --2- 172.21.2.112:0/1394409370 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3fe809a740 0x7f3fe8097bc0 unknown :-1 s=CLOSED pgs=1972 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:47.549 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.554 7f3fee632700 1 --2- 172.21.2.112:0/1394409370 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3fe80ada90 0x7f3fe80ade90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:47.549 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.554 7f3fee632700 1 -- 172.21.2.112:0/1394409370 >> 172.21.2.112:0/1394409370 conn(0x7f3fe80f0550 msgr2=0x7f3fe8191730 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:47.549 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.555 7f3fee632700 1 -- 172.21.2.112:0/1394409370 shutdown_connections 2021-02-10T11:39:47.549 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.555 7f3fee632700 1 -- 172.21.2.112:0/1394409370 wait complete. 2021-02-10T11:39:47.557 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:39:47.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.987 7f7fbf2a2700 1 Processor -- start 2021-02-10T11:39:47.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.987 7f7fbf2a2700 1 -- start start 2021-02-10T11:39:47.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.988 7f7fbf2a2700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7fb80d1880 0x7f7fb81b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:48.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.988 7f7fbf2a2700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f7fb81b0af0 0x7f7fb8108950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:48.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.988 7f7fbf2a2700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fb8109a70 con 0x7f7fb80d1c80 2021-02-10T11:39:48.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.988 7f7fbf2a2700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fb80e5f00 con 0x7f7fb81b0af0 2021-02-10T11:39:48.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.988 7f7fbf2a2700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fb8109760 con 0x7f7fb80d1880 2021-02-10T11:39:48.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.988 7f7fbd8bf700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f7fb80d1c80 0x7f7fb8108f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:53562/0 (socket says 172.21.2.112:53562) 2021-02-10T11:39:48.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.988 7f7fbd8bf700 1 -- 172.21.2.112:0/682542018 learned_addr learned my addr 172.21.2.112:0/682542018 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:39:48.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.988 7f7fabfff700 1 -- 172.21.2.112:0/682542018 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2991546772 0 0) 0x7f7fb0001500 con 0x7f7fb80d1c80 2021-02-10T11:39:48.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.988 7f7fabfff700 1 -- 172.21.2.112:0/682542018 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fa4002fe0 con 0x7f7fb80d1c80 2021-02-10T11:39:48.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.989 7f7fbc8bd700 1 --2- 172.21.2.112:0/682542018 >> v2:172.21.2.120:3300/0 conn(0x7f7fb81b0af0 0x7f7fb8108950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:48.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.989 7f7fbd0be700 1 --2- 172.21.2.112:0/682542018 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7fb80d1880 0x7f7fb81b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:48.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.989 7f7fabfff700 1 -- 172.21.2.112:0/682542018 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 126959802 0 0) 0x7f7fb0001460 con 0x7f7fb80d1c80 2021-02-10T11:39:48.001 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.989 7f7fabfff700 1 -- 172.21.2.112:0/682542018 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7fb80d1880 msgr2=0x7f7fb81b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:48.001 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.989 7f7fabfff700 1 --2- 172.21.2.112:0/682542018 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7fb80d1880 0x7f7fb81b04d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:48.001 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.989 7f7fabfff700 1 -- 172.21.2.112:0/682542018 >> v2:172.21.2.120:3300/0 conn(0x7f7fb81b0af0 msgr2=0x7f7fb8108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:48.001 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.989 7f7fabfff700 1 --2- 172.21.2.112:0/682542018 >> v2:172.21.2.120:3300/0 conn(0x7f7fb81b0af0 0x7f7fb8108950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:48.001 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.989 7f7fabfff700 1 -- 172.21.2.112:0/682542018 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fb8088070 con 0x7f7fb80d1c80 2021-02-10T11:39:48.001 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.989 7f7fabfff700 1 -- 172.21.2.112:0/682542018 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f7fb0001980 con 0x7f7fb80d1c80 2021-02-10T11:39:48.001 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.989 7f7fabfff700 1 -- 172.21.2.112:0/682542018 >> v1:172.21.2.112:6789/0 conn(0x7f7fb80d1c80 legacy=0x7f7fb8108f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:48.001 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.989 7f7fabfff700 1 --2- 172.21.2.112:0/682542018 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7fa4003b10 0x7f7fa4005f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:48.001 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.989 7f7fabfff700 1 --2- 172.21.2.112:0/682542018 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7fa4006760 0x7f7fa4008c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:48.001 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.989 7f7fabfff700 1 --2- 172.21.2.112:0/682542018 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7fa4009520 0x7f7fa400ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:48.001 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.989 7f7fabfff700 1 -- 172.21.2.112:0/682542018 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fa400c2d0 con 0x7f7fa4006760 2021-02-10T11:39:48.001 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.989 7f7fabfff700 1 -- 172.21.2.112:0/682542018 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fa40504c0 con 0x7f7fa4009520 2021-02-10T11:39:48.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.989 7f7fabfff700 1 -- 172.21.2.112:0/682542018 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fa40506e0 con 0x7f7fa4003b10 2021-02-10T11:39:48.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.989 7f7fbc8bd700 1 --2- 172.21.2.112:0/682542018 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7fa4006760 0x7f7fa4008c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:48.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.989 7f7fbd8bf700 1 --2- 172.21.2.112:0/682542018 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7fa4009520 0x7f7fa400ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:48.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.989 7f7fbd0be700 1 --2- 172.21.2.112:0/682542018 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7fa4003b10 0x7f7fa4005f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:48.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.989 7f7fbc8bd700 1 -- 172.21.2.112:0/682542018 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7fa4003b10 msgr2=0x7f7fa4005f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:48.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.989 7f7fbc8bd700 1 --2- 172.21.2.112:0/682542018 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7fa4003b10 0x7f7fa4005f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:48.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.989 7f7fbc8bd700 1 -- 172.21.2.112:0/682542018 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7fa4009520 msgr2=0x7f7fa400ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:48.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.989 7f7fbc8bd700 1 --2- 172.21.2.112:0/682542018 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7fa4009520 0x7f7fa400ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:48.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.989 7f7fbc8bd700 1 -- 172.21.2.112:0/682542018 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fa4050980 con 0x7f7fa4006760 2021-02-10T11:39:48.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.990 7f7fbc8bd700 1 --2- 172.21.2.112:0/682542018 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7fa4006760 0x7f7fa4008c80 secure :-1 s=READY pgs=1974 cs=0 l=1 rev1=1 rx=0x7f7fac006360 tx=0x7f7fac009c30).ready entity=mon.0 client_cookie=bfd2ace0415fac59 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:48.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.990 7f7fabfff700 1 -- 172.21.2.112:0/682542018 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7fac0009b0 con 0x7f7fa4006760 2021-02-10T11:39:48.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.990 7f7fabfff700 1 -- 172.21.2.112:0/682542018 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fac000d20 con 0x7f7fa4006760 2021-02-10T11:39:48.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.990 7f7fabfff700 1 -- 172.21.2.112:0/682542018 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7fac002220 con 0x7f7fa4006760 2021-02-10T11:39:48.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.990 7f7fbf2a2700 1 -- 172.21.2.112:0/682542018 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7fa4006760 msgr2=0x7f7fa4008c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:48.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.990 7f7fbf2a2700 1 --2- 172.21.2.112:0/682542018 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7fa4006760 0x7f7fa4008c80 secure :-1 s=READY pgs=1974 cs=0 l=1 rev1=1 rx=0x7f7fac006360 tx=0x7f7fac009c30).stop 2021-02-10T11:39:48.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.990 7f7fbd8bf700 1 -- 172.21.2.112:0/682542018 reap_dead start 2021-02-10T11:39:48.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.990 7f7fbf2a2700 1 -- 172.21.2.112:0/682542018 shutdown_connections 2021-02-10T11:39:48.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.990 7f7fbf2a2700 1 -- 172.21.2.112:0/682542018 >> 172.21.2.112:0/682542018 conn(0x7f7fb80ce430 msgr2=0x7f7fb80d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:48.005 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.991 7f7fbf2a2700 1 -- 172.21.2.112:0/682542018 shutdown_connections 2021-02-10T11:39:48.005 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:47.991 7f7fbf2a2700 1 -- 172.21.2.112:0/682542018 wait complete. 2021-02-10T11:39:48.005 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.005 7f7fbf2a2700 1 Processor -- start 2021-02-10T11:39:48.005 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.005 7f7fbf2a2700 1 -- start start 2021-02-10T11:39:48.007 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.006 7f7fbf2a2700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7fb81a0580 0x7f7fb8086fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:48.007 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.006 7f7fbf2a2700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7fb80b66b0 0x7f7fb80b6ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:48.007 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.006 7f7fbf2a2700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7fb8082bb0 0x7f7fb8082fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:48.007 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.006 7f7fbf2a2700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fb80875d0 con 0x7f7fb81a0580 2021-02-10T11:39:48.007 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.006 7f7fbf2a2700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fb80835d0 con 0x7f7fb8082bb0 2021-02-10T11:39:48.007 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.006 7f7fbf2a2700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fb80837f0 con 0x7f7fb80b66b0 2021-02-10T11:39:48.007 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.006 7f7fbd0be700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7fb81a0580 0x7f7fb8086fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:48.007 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.006 7f7fbd0be700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7fb81a0580 0x7f7fb8086fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:54110/0 (socket says 172.21.2.112:54110) 2021-02-10T11:39:48.007 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.006 7f7fbd0be700 1 -- 172.21.2.112:0/3579711036 learned_addr learned my addr 172.21.2.112:0/3579711036 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:39:48.007 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.006 7f7fbd8bf700 1 --2- 172.21.2.112:0/3579711036 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7fb8082bb0 0x7f7fb8082fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:48.008 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.006 7f7fbc8bd700 1 --2- 172.21.2.112:0/3579711036 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7fb80b66b0 0x7f7fb80b6ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:48.008 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.006 7f7fbd0be700 1 -- 172.21.2.112:0/3579711036 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7fb80b66b0 msgr2=0x7f7fb80b6ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:48.008 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.006 7f7fbd0be700 1 --2- 172.21.2.112:0/3579711036 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7fb80b66b0 0x7f7fb80b6ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:48.008 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.006 7f7fbd0be700 1 -- 172.21.2.112:0/3579711036 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7fb8082bb0 msgr2=0x7f7fb8082fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:48.008 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.006 7f7fbd0be700 1 --2- 172.21.2.112:0/3579711036 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7fb8082bb0 0x7f7fb8082fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:48.008 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.006 7f7fbd0be700 1 -- 172.21.2.112:0/3579711036 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fb8083a10 con 0x7f7fb81a0580 2021-02-10T11:39:48.008 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.007 7f7fbd0be700 1 --2- 172.21.2.112:0/3579711036 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7fb81a0580 0x7f7fb8086fb0 secure :-1 s=READY pgs=1975 cs=0 l=1 rev1=1 rx=0x7f7fb400b5e0 tx=0x7f7fb400b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:48.008 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.007 7f7fa9ffb700 1 -- 172.21.2.112:0/3579711036 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7fb400bd70 con 0x7f7fb81a0580 2021-02-10T11:39:48.008 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.007 7f7fa9ffb700 1 -- 172.21.2.112:0/3579711036 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fb4002030 con 0x7f7fb81a0580 2021-02-10T11:39:48.008 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.007 7f7fa9ffb700 1 -- 172.21.2.112:0/3579711036 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7fb4002550 con 0x7f7fb81a0580 2021-02-10T11:39:48.008 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.007 7f7fbf2a2700 1 -- 172.21.2.112:0/3579711036 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fb8083a10 con 0x7f7fb81a0580 2021-02-10T11:39:48.008 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.007 7f7fbf2a2700 1 -- 172.21.2.112:0/3579711036 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fb8083fa0 con 0x7f7fb81a0580 2021-02-10T11:39:48.008 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.008 7f7fa9ffb700 1 -- 172.21.2.112:0/3579711036 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7fb4002210 con 0x7f7fb81a0580 2021-02-10T11:39:48.009 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.008 7f7fa9ffb700 1 --2- 172.21.2.112:0/3579711036 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7f980186c0 0x7f7f9801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:48.009 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.008 7f7fa9ffb700 1 -- 172.21.2.112:0/3579711036 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f7fb4032570 con 0x7f7fb81a0580 2021-02-10T11:39:48.009 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.008 7f7fbc8bd700 1 --2- 172.21.2.112:0/3579711036 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7f980186c0 0x7f7f9801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:48.009 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.009 7f7fbc8bd700 1 --2- 172.21.2.112:0/3579711036 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7f980186c0 0x7f7f9801abc0 crc :-1 s=READY pgs=922 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:48.031 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.037 7f7fbf2a2700 1 -- 172.21.2.112:0/3579711036 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fb80875d0 con 0x7f7fb81a0580 2021-02-10T11:39:48.034 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.040 7f7fa9ffb700 1 -- 172.21.2.112:0/3579711036 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7fb4000c90 con 0x7f7fb81a0580 2021-02-10T11:39:48.165 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.171 7f7fbf2a2700 1 -- 172.21.2.112:0/3579711036 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f7fb80875d0 con 0x7f7fb81a0580 2021-02-10T11:39:48.169 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.175 7f7fa9ffb700 1 -- 172.21.2.112:0/3579711036 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v61) v1 ==== 184+0+0 (secure 0 0 0) 0x7f7fb4002210 con 0x7f7fb81a0580 2021-02-10T11:39:48.172 INFO:teuthology.orchestra.run.gibba012.stderr:updated caps for client.0 2021-02-10T11:39:48.173 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.180 7f7fbf2a2700 1 -- 172.21.2.112:0/3579711036 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7f980186c0 msgr2=0x7f7f9801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:48.174 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.180 7f7fbf2a2700 1 --2- 172.21.2.112:0/3579711036 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7f980186c0 0x7f7f9801abc0 crc :-1 s=READY pgs=922 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:48.174 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.180 7f7fbf2a2700 1 -- 172.21.2.112:0/3579711036 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7fb81a0580 msgr2=0x7f7fb8086fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:48.174 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.180 7f7fbf2a2700 1 --2- 172.21.2.112:0/3579711036 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7fb81a0580 0x7f7fb8086fb0 secure :-1 s=READY pgs=1975 cs=0 l=1 rev1=1 rx=0x7f7fb400b5e0 tx=0x7f7fb400b980).stop 2021-02-10T11:39:48.174 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.180 7f7fbf2a2700 1 -- 172.21.2.112:0/3579711036 shutdown_connections 2021-02-10T11:39:48.174 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.180 7f7fbf2a2700 1 --2- 172.21.2.112:0/3579711036 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7f980186c0 0x7f7f9801abc0 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:48.174 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.180 7f7fbf2a2700 1 --2- 172.21.2.112:0/3579711036 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7fb8082bb0 0x7f7fb8082fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:48.174 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.180 7f7fbf2a2700 1 --2- 172.21.2.112:0/3579711036 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7fb80b66b0 0x7f7fb80b6ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:48.175 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.180 7f7fbf2a2700 1 --2- 172.21.2.112:0/3579711036 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7fb81a0580 0x7f7fb8086fb0 unknown :-1 s=CLOSED pgs=1975 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:48.175 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.180 7f7fbf2a2700 1 -- 172.21.2.112:0/3579711036 >> 172.21.2.112:0/3579711036 conn(0x7f7fb80afc60 msgr2=0x7f7fb80b50d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:48.175 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.180 7f7fbf2a2700 1 -- 172.21.2.112:0/3579711036 shutdown_connections 2021-02-10T11:39:48.175 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.180 7f7fbf2a2700 1 -- 172.21.2.112:0/3579711036 wait complete. 2021-02-10T11:39:48.182 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:39:48.621 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.610 7f1617c97700 1 Processor -- start 2021-02-10T11:39:48.621 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.610 7f1617c97700 1 -- start start 2021-02-10T11:39:48.621 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.611 7f1617c97700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f16100d1880 0x7f16101b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:48.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.611 7f1617c97700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f16101b0af0 0x7f1610108950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:48.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.611 7f1617c97700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f16100d1c80 0x7f1610108f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:48.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.611 7f1617c97700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16100fe1d0 con 0x7f16100d1880 2021-02-10T11:39:48.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.611 7f1617c97700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1610109a70 con 0x7f16100d1c80 2021-02-10T11:39:48.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.611 7f1617c97700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1610088070 con 0x7f16101b0af0 2021-02-10T11:39:48.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.611 7f1615ab3700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f16100d1880 0x7f16101b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:48.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.611 7f16162b4700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f16100d1c80 0x7f1610108f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:48.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.611 7f1615ab3700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f16100d1880 0x7f16101b04d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:54118/0 (socket says 172.21.2.112:54118) 2021-02-10T11:39:48.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.611 7f1615ab3700 1 -- 172.21.2.112:0/783383652 learned_addr learned my addr 172.21.2.112:0/783383652 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:39:48.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.611 7f16152b2700 1 --2- 172.21.2.112:0/783383652 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f16101b0af0 0x7f1610108950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:48.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.611 7f1615ab3700 1 -- 172.21.2.112:0/783383652 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f16101b0af0 msgr2=0x7f1610108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:48.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.611 7f1615ab3700 1 --2- 172.21.2.112:0/783383652 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f16101b0af0 0x7f1610108950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:48.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.611 7f1615ab3700 1 -- 172.21.2.112:0/783383652 >> v2:172.21.2.120:3300/0 conn(0x7f16100d1c80 msgr2=0x7f1610108f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:48.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.611 7f1615ab3700 1 --2- 172.21.2.112:0/783383652 >> v2:172.21.2.120:3300/0 conn(0x7f16100d1c80 0x7f1610108f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:48.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.611 7f1615ab3700 1 -- 172.21.2.112:0/783383652 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1610109590 con 0x7f16100d1880 2021-02-10T11:39:48.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.612 7f1615ab3700 1 --2- 172.21.2.112:0/783383652 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f16100d1880 0x7f16101b04d0 secure :-1 s=READY pgs=1976 cs=0 l=1 rev1=1 rx=0x7f160c006700 tx=0x7f160c009b80).ready entity=mon.0 client_cookie=6b393accc7a35ad7 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:48.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.612 7f1614ab1700 1 -- 172.21.2.112:0/783383652 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f160c00c0c0 con 0x7f16100d1880 2021-02-10T11:39:48.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.612 7f1614ab1700 1 -- 172.21.2.112:0/783383652 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f160c00c440 con 0x7f16100d1880 2021-02-10T11:39:48.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.612 7f1614ab1700 1 -- 172.21.2.112:0/783383652 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f160c00c960 con 0x7f16100d1880 2021-02-10T11:39:48.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.612 7f1617c97700 1 -- 172.21.2.112:0/783383652 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f16100d1880 msgr2=0x7f16101b04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:48.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.612 7f1617c97700 1 --2- 172.21.2.112:0/783383652 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f16100d1880 0x7f16101b04d0 secure :-1 s=READY pgs=1976 cs=0 l=1 rev1=1 rx=0x7f160c006700 tx=0x7f160c009b80).stop 2021-02-10T11:39:48.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.613 7f1617c97700 1 -- 172.21.2.112:0/783383652 shutdown_connections 2021-02-10T11:39:48.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.613 7f1617c97700 1 --2- 172.21.2.112:0/783383652 >> v2:172.21.2.120:3300/0 conn(0x7f16100d1c80 0x7f1610108f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:48.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.613 7f1617c97700 1 --2- 172.21.2.112:0/783383652 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f16101b0af0 0x7f1610108950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:48.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.613 7f1617c97700 1 --2- 172.21.2.112:0/783383652 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f16100d1880 0x7f16101b04d0 unknown :-1 s=CLOSED pgs=1976 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:48.624 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.613 7f1617c97700 1 -- 172.21.2.112:0/783383652 >> 172.21.2.112:0/783383652 conn(0x7f16100ce430 msgr2=0x7f16100d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:48.624 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.613 7f1617c97700 1 -- 172.21.2.112:0/783383652 shutdown_connections 2021-02-10T11:39:48.624 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.613 7f1617c97700 1 -- 172.21.2.112:0/783383652 wait complete. 2021-02-10T11:39:48.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.628 7f1617c97700 1 Processor -- start 2021-02-10T11:39:48.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.628 7f1617c97700 1 -- start start 2021-02-10T11:39:48.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.628 7f1617c97700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1610090e50 0x7f16101a44b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:48.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.628 7f1617c97700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f16101a4ad0 0x7f1610083200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:48.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.628 7f1617c97700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1610083820 0x7f1610083c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:48.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.628 7f1617c97700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1610091250 con 0x7f16101a4ad0 2021-02-10T11:39:48.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.628 7f1617c97700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1610084a40 con 0x7f1610083820 2021-02-10T11:39:48.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.628 7f1617c97700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1610084c60 con 0x7f1610090e50 2021-02-10T11:39:48.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.628 7f16152b2700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f16101a4ad0 0x7f1610083200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:48.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.628 7f16152b2700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f16101a4ad0 0x7f1610083200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:54122/0 (socket says 172.21.2.112:54122) 2021-02-10T11:39:48.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.628 7f16152b2700 1 -- 172.21.2.112:0/463273573 learned_addr learned my addr 172.21.2.112:0/463273573 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:39:48.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.629 7f16152b2700 1 -- 172.21.2.112:0/463273573 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1610090e50 msgr2=0x7f16101a44b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:48.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.629 7f16152b2700 1 --2- 172.21.2.112:0/463273573 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1610090e50 0x7f16101a44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:48.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.629 7f16152b2700 1 -- 172.21.2.112:0/463273573 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1610083820 msgr2=0x7f1610083c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:48.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.629 7f16152b2700 1 --2- 172.21.2.112:0/463273573 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1610083820 0x7f1610083c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:48.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.629 7f16152b2700 1 -- 172.21.2.112:0/463273573 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1610084e80 con 0x7f16101a4ad0 2021-02-10T11:39:48.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.629 7f16152b2700 1 --2- 172.21.2.112:0/463273573 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f16101a4ad0 0x7f1610083200 secure :-1 s=READY pgs=1977 cs=0 l=1 rev1=1 rx=0x7f1604006f60 tx=0x7f160400b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:48.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.629 7f16027fc700 1 -- 172.21.2.112:0/463273573 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f160400bd60 con 0x7f16101a4ad0 2021-02-10T11:39:48.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.629 7f16027fc700 1 -- 172.21.2.112:0/463273573 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1604000c90 con 0x7f16101a4ad0 2021-02-10T11:39:48.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.629 7f16027fc700 1 -- 172.21.2.112:0/463273573 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1604002220 con 0x7f16101a4ad0 2021-02-10T11:39:48.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.629 7f1617c97700 1 -- 172.21.2.112:0/463273573 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1610084e80 con 0x7f16101a4ad0 2021-02-10T11:39:48.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.629 7f1617c97700 1 -- 172.21.2.112:0/463273573 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1610084e80 con 0x7f16101a4ad0 2021-02-10T11:39:48.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.630 7f16027fc700 1 -- 172.21.2.112:0/463273573 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f16040052e0 con 0x7f16101a4ad0 2021-02-10T11:39:48.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.630 7f16027fc700 1 --2- 172.21.2.112:0/463273573 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f15f80186c0 0x7f15f801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:48.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.630 7f16027fc700 1 -- 172.21.2.112:0/463273573 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f1604032530 con 0x7f16101a4ad0 2021-02-10T11:39:48.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.630 7f1615ab3700 1 --2- 172.21.2.112:0/463273573 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f15f80186c0 0x7f15f801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:48.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.631 7f1615ab3700 1 --2- 172.21.2.112:0/463273573 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f15f80186c0 0x7f15f801abc0 crc :-1 s=READY pgs=923 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:48.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.660 7f1617c97700 1 -- 172.21.2.112:0/463273573 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1610084290 con 0x7f16101a4ad0 2021-02-10T11:39:48.657 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.663 7f16027fc700 1 -- 172.21.2.112:0/463273573 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1604006b00 con 0x7f16101a4ad0 2021-02-10T11:39:48.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.794 7f1617c97700 1 -- 172.21.2.112:0/463273573 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f1610084290 con 0x7f16101a4ad0 2021-02-10T11:39:48.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.797 7f16027fc700 1 -- 172.21.2.112:0/463273573 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v62) v1 ==== 184+0+0 (secure 0 0 0) 0x7f16040052e0 con 0x7f16101a4ad0 2021-02-10T11:39:48.796 INFO:teuthology.orchestra.run.gibba012.stderr:updated caps for client.1 2021-02-10T11:39:48.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.802 7f1617c97700 1 -- 172.21.2.112:0/463273573 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f15f80186c0 msgr2=0x7f15f801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:48.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.802 7f1617c97700 1 --2- 172.21.2.112:0/463273573 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f15f80186c0 0x7f15f801abc0 crc :-1 s=READY pgs=923 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:48.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.802 7f1617c97700 1 -- 172.21.2.112:0/463273573 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f16101a4ad0 msgr2=0x7f1610083200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:48.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.802 7f1617c97700 1 --2- 172.21.2.112:0/463273573 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f16101a4ad0 0x7f1610083200 secure :-1 s=READY pgs=1977 cs=0 l=1 rev1=1 rx=0x7f1604006f60 tx=0x7f160400b970).stop 2021-02-10T11:39:48.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.802 7f1617c97700 1 -- 172.21.2.112:0/463273573 shutdown_connections 2021-02-10T11:39:48.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.802 7f1617c97700 1 --2- 172.21.2.112:0/463273573 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f15f80186c0 0x7f15f801abc0 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:48.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.802 7f1617c97700 1 --2- 172.21.2.112:0/463273573 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1610083820 0x7f1610083c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:48.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.802 7f1617c97700 1 --2- 172.21.2.112:0/463273573 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f16101a4ad0 0x7f1610083200 unknown :-1 s=CLOSED pgs=1977 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:48.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.802 7f1617c97700 1 --2- 172.21.2.112:0/463273573 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1610090e50 0x7f16101a44b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:48.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.802 7f1617c97700 1 -- 172.21.2.112:0/463273573 >> 172.21.2.112:0/463273573 conn(0x7f16100a1100 msgr2=0x7f16100b5080 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:48.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.803 7f1617c97700 1 -- 172.21.2.112:0/463273573 shutdown_connections 2021-02-10T11:39:48.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:48.803 7f1617c97700 1 -- 172.21.2.112:0/463273573 wait complete. 2021-02-10T11:39:48.806 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.2 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:39:49.245 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.234 7f022fcf0700 1 Processor -- start 2021-02-10T11:39:49.245 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.234 7f022fcf0700 1 -- start start 2021-02-10T11:39:49.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.234 7f022fcf0700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f022810b000 0x7f02280cf760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:49.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.234 7f022fcf0700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02280fe1d0 con 0x7f02280d1880 2021-02-10T11:39:49.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.234 7f022fcf0700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02281b0cc0 con 0x7f02280d1c80 2021-02-10T11:39:49.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.234 7f022fcf0700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0228109a70 con 0x7f022810b000 2021-02-10T11:39:49.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.234 7f022db0c700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f02280d1880 0x7f02281b04d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:53590/0 (socket says 172.21.2.112:53590) 2021-02-10T11:39:49.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.234 7f022db0c700 1 -- 172.21.2.112:0/194024383 learned_addr learned my addr 172.21.2.112:0/194024383 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:39:49.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.234 7f022e30d700 1 --2- 172.21.2.112:0/194024383 >> v2:172.21.2.112:3300/0 conn(0x7f022810b000 0x7f02280cf760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:49.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.235 7f022e30d700 1 -- 172.21.2.112:0/194024383 >> v1:172.21.2.120:6789/0 conn(0x7f02280d1c80 legacy=0x7f02280cef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:49.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.235 7f022e30d700 1 -- 172.21.2.112:0/194024383 >> v1:172.21.2.112:6789/0 conn(0x7f02280d1880 legacy=0x7f02281b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:49.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.235 7f022e30d700 1 -- 172.21.2.112:0/194024383 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f022810b400 con 0x7f022810b000 2021-02-10T11:39:49.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.235 7f022e30d700 1 --2- 172.21.2.112:0/194024383 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f022810b000 0x7f02280cf760 secure :-1 s=READY pgs=1979 cs=0 l=1 rev1=1 rx=0x7f0220006e30 tx=0x7f0220009e60).ready entity=mon.0 client_cookie=54f72db60bd8b65c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:49.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.235 7f022cb0a700 1 -- 172.21.2.112:0/194024383 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0220006a50 con 0x7f022810b000 2021-02-10T11:39:49.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.235 7f022cb0a700 1 -- 172.21.2.112:0/194024383 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f022000b300 con 0x7f022810b000 2021-02-10T11:39:49.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.235 7f022cb0a700 1 -- 172.21.2.112:0/194024383 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f022000b820 con 0x7f022810b000 2021-02-10T11:39:49.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.235 7f022fcf0700 1 -- 172.21.2.112:0/194024383 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f022810b000 msgr2=0x7f02280cf760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:49.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.235 7f022fcf0700 1 --2- 172.21.2.112:0/194024383 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f022810b000 0x7f02280cf760 secure :-1 s=READY pgs=1979 cs=0 l=1 rev1=1 rx=0x7f0220006e30 tx=0x7f0220009e60).stop 2021-02-10T11:39:49.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.236 7f022fcf0700 1 -- 172.21.2.112:0/194024383 shutdown_connections 2021-02-10T11:39:49.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.236 7f022fcf0700 1 --2- 172.21.2.112:0/194024383 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f022810b000 0x7f02280cf760 unknown :-1 s=CLOSED pgs=1979 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:49.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.236 7f022fcf0700 1 -- 172.21.2.112:0/194024383 >> 172.21.2.112:0/194024383 conn(0x7f02280ce430 msgr2=0x7f02280d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:49.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.236 7f022fcf0700 1 -- 172.21.2.112:0/194024383 shutdown_connections 2021-02-10T11:39:49.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.236 7f022fcf0700 1 -- 172.21.2.112:0/194024383 wait complete. 2021-02-10T11:39:49.249 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.251 7f022fcf0700 1 Processor -- start 2021-02-10T11:39:49.249 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.251 7f022fcf0700 1 -- start start 2021-02-10T11:39:49.249 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.251 7f022fcf0700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f022808a480 0x7f0228108070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:49.249 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.251 7f022fcf0700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f02280ffcd0 0x7f022808af50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:49.249 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.251 7f022fcf0700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f02281000d0 0x7f022808b570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:49.249 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.251 7f022fcf0700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02281004d0 con 0x7f022808a480 2021-02-10T11:39:49.249 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.251 7f022fcf0700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02280a7f70 con 0x7f02281000d0 2021-02-10T11:39:49.249 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.251 7f022fcf0700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f022808a9d0 con 0x7f02280ffcd0 2021-02-10T11:39:49.249 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.251 7f022db0c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f022808a480 0x7f0228108070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:49.249 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.251 7f022db0c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f022808a480 0x7f0228108070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:54136/0 (socket says 172.21.2.112:54136) 2021-02-10T11:39:49.249 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.251 7f022db0c700 1 -- 172.21.2.112:0/3596610360 learned_addr learned my addr 172.21.2.112:0/3596610360 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:39:49.249 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.252 7f022db0c700 1 -- 172.21.2.112:0/3596610360 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f02280ffcd0 msgr2=0x7f022808af50 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:39:49.249 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.252 7f022db0c700 1 --2- 172.21.2.112:0/3596610360 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f02280ffcd0 0x7f022808af50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:49.250 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.252 7f022db0c700 1 -- 172.21.2.112:0/3596610360 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f02281000d0 msgr2=0x7f022808b570 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:39:49.250 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.252 7f022db0c700 1 --2- 172.21.2.112:0/3596610360 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f02281000d0 0x7f022808b570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:49.250 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.252 7f022db0c700 1 -- 172.21.2.112:0/3596610360 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0228108690 con 0x7f022808a480 2021-02-10T11:39:49.250 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.252 7f022db0c700 1 --2- 172.21.2.112:0/3596610360 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f022808a480 0x7f0228108070 secure :-1 s=READY pgs=1980 cs=0 l=1 rev1=1 rx=0x7f022400a570 tx=0x7f022400a910).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:49.250 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.252 7f021e7fc700 1 -- 172.21.2.112:0/3596610360 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f022400ad00 con 0x7f022808a480 2021-02-10T11:39:49.250 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.252 7f021e7fc700 1 -- 172.21.2.112:0/3596610360 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0224000f90 con 0x7f022808a480 2021-02-10T11:39:49.250 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.252 7f021e7fc700 1 -- 172.21.2.112:0/3596610360 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0224004830 con 0x7f022808a480 2021-02-10T11:39:49.250 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.252 7f022fcf0700 1 -- 172.21.2.112:0/3596610360 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0228108690 con 0x7f022808a480 2021-02-10T11:39:49.250 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.252 7f022fcf0700 1 -- 172.21.2.112:0/3596610360 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0228108690 con 0x7f022808a480 2021-02-10T11:39:49.250 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.253 7f021e7fc700 1 -- 172.21.2.112:0/3596610360 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0224001170 con 0x7f022808a480 2021-02-10T11:39:49.250 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.253 7f021e7fc700 1 --2- 172.21.2.112:0/3596610360 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f02100186c0 0x7f021001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:49.250 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.253 7f021e7fc700 1 -- 172.21.2.112:0/3596610360 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f02240325c0 con 0x7f022808a480 2021-02-10T11:39:49.250 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.253 7f022d30b700 1 --2- 172.21.2.112:0/3596610360 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f02100186c0 0x7f021001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:49.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.254 7f022d30b700 1 --2- 172.21.2.112:0/3596610360 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f02100186c0 0x7f021001abc0 crc :-1 s=READY pgs=924 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:49.279 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.284 7f022fcf0700 1 -- 172.21.2.112:0/3596610360 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02280a7f70 con 0x7f022808a480 2021-02-10T11:39:49.281 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.286 7f021e7fc700 1 -- 172.21.2.112:0/3596610360 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f022401f4e0 con 0x7f022808a480 2021-02-10T11:39:49.411 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.416 7f022fcf0700 1 -- 172.21.2.112:0/3596610360 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f02280a7f70 con 0x7f022808a480 2021-02-10T11:39:49.414 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.419 7f021e7fc700 1 -- 172.21.2.112:0/3596610360 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.2 v63) v1 ==== 184+0+0 (secure 0 0 0) 0x7f0224000f90 con 0x7f022808a480 2021-02-10T11:39:49.414 INFO:teuthology.orchestra.run.gibba012.stderr:updated caps for client.2 2021-02-10T11:39:49.416 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.421 7f022fcf0700 1 -- 172.21.2.112:0/3596610360 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f02100186c0 msgr2=0x7f021001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:49.416 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.421 7f022fcf0700 1 --2- 172.21.2.112:0/3596610360 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f02100186c0 0x7f021001abc0 crc :-1 s=READY pgs=924 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:49.416 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.421 7f022fcf0700 1 -- 172.21.2.112:0/3596610360 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f022808a480 msgr2=0x7f0228108070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:49.416 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.421 7f022fcf0700 1 --2- 172.21.2.112:0/3596610360 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f022808a480 0x7f0228108070 secure :-1 s=READY pgs=1980 cs=0 l=1 rev1=1 rx=0x7f022400a570 tx=0x7f022400a910).stop 2021-02-10T11:39:49.416 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.421 7f022fcf0700 1 -- 172.21.2.112:0/3596610360 shutdown_connections 2021-02-10T11:39:49.416 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.421 7f022fcf0700 1 --2- 172.21.2.112:0/3596610360 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f02100186c0 0x7f021001abc0 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:49.416 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.421 7f022fcf0700 1 --2- 172.21.2.112:0/3596610360 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f02281000d0 0x7f022808b570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:49.416 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.421 7f022fcf0700 1 --2- 172.21.2.112:0/3596610360 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f02280ffcd0 0x7f022808af50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:49.417 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.421 7f022fcf0700 1 --2- 172.21.2.112:0/3596610360 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f022808a480 0x7f0228108070 unknown :-1 s=CLOSED pgs=1980 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:49.417 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.421 7f022fcf0700 1 -- 172.21.2.112:0/3596610360 >> 172.21.2.112:0/3596610360 conn(0x7f0228100b70 msgr2=0x7f02280c3d00 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:49.417 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.421 7f022fcf0700 1 -- 172.21.2.112:0/3596610360 shutdown_connections 2021-02-10T11:39:49.417 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.421 7f022fcf0700 1 -- 172.21.2.112:0/3596610360 wait complete. 2021-02-10T11:39:49.425 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.3 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:39:49.867 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.852 7fc1e9d0b700 1 Processor -- start 2021-02-10T11:39:49.867 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.853 7fc1e9d0b700 1 -- start start 2021-02-10T11:39:49.868 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.853 7fc1e9d0b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc1e40d1880 0x7fc1e41b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:49.868 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.853 7fc1e9d0b700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fc1e41b0af0 0x7fc1e4108950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:49.868 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.853 7fc1e9d0b700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1e4109a70 con 0x7fc1e40d1c80 2021-02-10T11:39:49.868 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.853 7fc1e9d0b700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1e40e5f00 con 0x7fc1e41b0af0 2021-02-10T11:39:49.868 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.853 7fc1e9d0b700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1e4109760 con 0x7fc1e40d1880 2021-02-10T11:39:49.868 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.853 7fc1e3fff700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fc1e40d1c80 0x7fc1e4108f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:53604/0 (socket says 172.21.2.112:53604) 2021-02-10T11:39:49.868 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.853 7fc1e3fff700 1 -- 172.21.2.112:0/3511748129 learned_addr learned my addr 172.21.2.112:0/3511748129 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:39:49.868 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.854 7fc1e27fc700 1 -- 172.21.2.112:0/3511748129 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1528223422 0 0) 0x7fc1dc001500 con 0x7fc1e40d1c80 2021-02-10T11:39:49.868 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.854 7fc1e27fc700 1 -- 172.21.2.112:0/3511748129 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1c8002fe0 con 0x7fc1e40d1c80 2021-02-10T11:39:49.868 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.854 7fc1e2ffd700 1 --2- 172.21.2.112:0/3511748129 >> v2:172.21.2.120:3300/0 conn(0x7fc1e41b0af0 0x7fc1e4108950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:49.868 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.854 7fc1e37fe700 1 --2- 172.21.2.112:0/3511748129 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc1e40d1880 0x7fc1e41b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:49.868 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.854 7fc1e27fc700 1 -- 172.21.2.112:0/3511748129 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 658725376 0 0) 0x7fc1dc001460 con 0x7fc1e40d1c80 2021-02-10T11:39:49.868 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.854 7fc1e27fc700 1 -- 172.21.2.112:0/3511748129 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc1e40d1880 msgr2=0x7fc1e41b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:49.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.854 7fc1e27fc700 1 --2- 172.21.2.112:0/3511748129 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc1e40d1880 0x7fc1e41b04d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:49.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.854 7fc1e27fc700 1 -- 172.21.2.112:0/3511748129 >> v2:172.21.2.120:3300/0 conn(0x7fc1e41b0af0 msgr2=0x7fc1e4108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:49.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.854 7fc1e27fc700 1 --2- 172.21.2.112:0/3511748129 >> v2:172.21.2.120:3300/0 conn(0x7fc1e41b0af0 0x7fc1e4108950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:49.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.854 7fc1e27fc700 1 -- 172.21.2.112:0/3511748129 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1e4088070 con 0x7fc1e40d1c80 2021-02-10T11:39:49.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.854 7fc1e27fc700 1 -- 172.21.2.112:0/3511748129 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7fc1dc001980 con 0x7fc1e40d1c80 2021-02-10T11:39:49.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.854 7fc1e27fc700 1 -- 172.21.2.112:0/3511748129 >> v1:172.21.2.112:6789/0 conn(0x7fc1e40d1c80 legacy=0x7fc1e4108f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:49.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.854 7fc1e27fc700 1 --2- 172.21.2.112:0/3511748129 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc1c8003ae0 0x7fc1c8006000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:49.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.854 7fc1e27fc700 1 --2- 172.21.2.112:0/3511748129 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc1c8006820 0x7fc1c8008d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:49.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.854 7fc1e27fc700 1 --2- 172.21.2.112:0/3511748129 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc1c804d5a0 0x7fc1c804fac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:49.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.854 7fc1e27fc700 1 -- 172.21.2.112:0/3511748129 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1c8050370 con 0x7fc1c804d5a0 2021-02-10T11:39:49.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.854 7fc1e27fc700 1 -- 172.21.2.112:0/3511748129 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1c8050590 con 0x7fc1c8003ae0 2021-02-10T11:39:49.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.854 7fc1e27fc700 1 -- 172.21.2.112:0/3511748129 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1c80507b0 con 0x7fc1c8006820 2021-02-10T11:39:49.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.855 7fc1e37fe700 1 --2- 172.21.2.112:0/3511748129 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc1e40d1880 0x7fc1e41b04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:39:49.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.855 7fc1e2ffd700 1 --2- 172.21.2.112:0/3511748129 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc1c8006820 0x7fc1c8008d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:49.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.855 7fc1e3fff700 1 --2- 172.21.2.112:0/3511748129 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc1c804d5a0 0x7fc1c804fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:49.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.855 7fc1e3fff700 1 -- 172.21.2.112:0/3511748129 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc1c8006820 msgr2=0x7fc1c8008d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:49.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.855 7fc1e3fff700 1 --2- 172.21.2.112:0/3511748129 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc1c8006820 0x7fc1c8008d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:49.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.855 7fc1e3fff700 1 -- 172.21.2.112:0/3511748129 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc1c8003ae0 msgr2=0x7fc1c8006000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:49.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.855 7fc1e3fff700 1 --2- 172.21.2.112:0/3511748129 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc1c8003ae0 0x7fc1c8006000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:49.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.855 7fc1e3fff700 1 -- 172.21.2.112:0/3511748129 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1c8050a50 con 0x7fc1c804d5a0 2021-02-10T11:39:49.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.855 7fc1e3fff700 1 --2- 172.21.2.112:0/3511748129 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc1c804d5a0 0x7fc1c804fac0 secure :-1 s=READY pgs=1982 cs=0 l=1 rev1=1 rx=0x7fc1dc007d00 tx=0x7fc1dc00a310).ready entity=mon.0 client_cookie=15c8ddc5158c41f1 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:49.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.855 7fc1e27fc700 1 -- 172.21.2.112:0/3511748129 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc1dc001460 con 0x7fc1c804d5a0 2021-02-10T11:39:49.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.855 7fc1e27fc700 1 -- 172.21.2.112:0/3511748129 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1dc007dd0 con 0x7fc1c804d5a0 2021-02-10T11:39:49.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.855 7fc1e27fc700 1 -- 172.21.2.112:0/3511748129 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc1dc007370 con 0x7fc1c804d5a0 2021-02-10T11:39:49.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.855 7fc1e9d0b700 1 -- 172.21.2.112:0/3511748129 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc1c804d5a0 msgr2=0x7fc1c804fac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:49.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.855 7fc1e9d0b700 1 --2- 172.21.2.112:0/3511748129 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc1c804d5a0 0x7fc1c804fac0 secure :-1 s=READY pgs=1982 cs=0 l=1 rev1=1 rx=0x7fc1dc007d00 tx=0x7fc1dc00a310).stop 2021-02-10T11:39:49.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.855 7fc1e3fff700 1 -- 172.21.2.112:0/3511748129 reap_dead start 2021-02-10T11:39:49.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.856 7fc1e9d0b700 1 -- 172.21.2.112:0/3511748129 shutdown_connections 2021-02-10T11:39:49.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.856 7fc1e9d0b700 1 -- 172.21.2.112:0/3511748129 >> 172.21.2.112:0/3511748129 conn(0x7fc1e40ce430 msgr2=0x7fc1e40d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:49.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.856 7fc1e9d0b700 1 -- 172.21.2.112:0/3511748129 shutdown_connections 2021-02-10T11:39:49.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.856 7fc1e9d0b700 1 -- 172.21.2.112:0/3511748129 wait complete. 2021-02-10T11:39:49.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.871 7fc1e9d0b700 1 Processor -- start 2021-02-10T11:39:49.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.871 7fc1e9d0b700 1 -- start start 2021-02-10T11:39:49.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.871 7fc1e9d0b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc1e41a0580 0x7fc1e4086fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:49.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.871 7fc1e9d0b700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc1e40b66b0 0x7fc1e40b6ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:49.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.871 7fc1e9d0b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc1e4082bb0 0x7fc1e4082fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:49.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.871 7fc1e9d0b700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1e40875d0 con 0x7fc1e4082bb0 2021-02-10T11:39:49.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.871 7fc1e9d0b700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1e40835d0 con 0x7fc1e40b66b0 2021-02-10T11:39:49.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.871 7fc1e9d0b700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1e40837f0 con 0x7fc1e41a0580 2021-02-10T11:39:49.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.871 7fc1e3fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc1e4082bb0 0x7fc1e4082fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:49.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.871 7fc1e3fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc1e4082bb0 0x7fc1e4082fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:54158/0 (socket says 172.21.2.112:54158) 2021-02-10T11:39:49.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.871 7fc1e3fff700 1 -- 172.21.2.112:0/3385255328 learned_addr learned my addr 172.21.2.112:0/3385255328 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:39:49.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.872 7fc1e3fff700 1 -- 172.21.2.112:0/3385255328 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc1e41a0580 msgr2=0x7fc1e4086fb0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:39:49.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.872 7fc1e3fff700 1 --2- 172.21.2.112:0/3385255328 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc1e41a0580 0x7fc1e4086fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:49.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.872 7fc1e3fff700 1 -- 172.21.2.112:0/3385255328 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc1e40b66b0 msgr2=0x7fc1e40b6ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:49.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.872 7fc1e3fff700 1 --2- 172.21.2.112:0/3385255328 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc1e40b66b0 0x7fc1e40b6ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:49.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.872 7fc1e3fff700 1 -- 172.21.2.112:0/3385255328 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1e4083a10 con 0x7fc1e4082bb0 2021-02-10T11:39:49.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.872 7fc1e3fff700 1 --2- 172.21.2.112:0/3385255328 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc1e4082bb0 0x7fc1e4082fb0 secure :-1 s=READY pgs=1983 cs=0 l=1 rev1=1 rx=0x7fc1dc00a940 tx=0x7fc1dc00ace0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:49.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.872 7fc1dbfff700 1 -- 172.21.2.112:0/3385255328 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc1dc004930 con 0x7fc1e4082bb0 2021-02-10T11:39:49.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.872 7fc1dbfff700 1 -- 172.21.2.112:0/3385255328 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1dc004b30 con 0x7fc1e4082bb0 2021-02-10T11:39:49.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.872 7fc1e9d0b700 1 -- 172.21.2.112:0/3385255328 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1e4083a10 con 0x7fc1e4082bb0 2021-02-10T11:39:49.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.872 7fc1e9d0b700 1 -- 172.21.2.112:0/3385255328 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1e4083fa0 con 0x7fc1e4082bb0 2021-02-10T11:39:49.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.872 7fc1dbfff700 1 -- 172.21.2.112:0/3385255328 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc1dc007020 con 0x7fc1e4082bb0 2021-02-10T11:39:49.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.873 7fc1dbfff700 1 -- 172.21.2.112:0/3385255328 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc1dc004720 con 0x7fc1e4082bb0 2021-02-10T11:39:49.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.873 7fc1dbfff700 1 --2- 172.21.2.112:0/3385255328 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc1c00186c0 0x7fc1c001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:49.875 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.873 7fc1dbfff700 1 -- 172.21.2.112:0/3385255328 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fc1dc032b70 con 0x7fc1e4082bb0 2021-02-10T11:39:49.875 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.873 7fc1e37fe700 1 --2- 172.21.2.112:0/3385255328 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc1c00186c0 0x7fc1c001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:49.875 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.873 7fc1e37fe700 1 --2- 172.21.2.112:0/3385255328 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc1c00186c0 0x7fc1c001abc0 crc :-1 s=READY pgs=925 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:49.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.903 7fc1e9d0b700 1 -- 172.21.2.112:0/3385255328 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1e4082980 con 0x7fc1e4082bb0 2021-02-10T11:39:49.901 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:49.906 7fc1dbfff700 1 -- 172.21.2.112:0/3385255328 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc1dc001950 con 0x7fc1e4082bb0 2021-02-10T11:39:50.031 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:50.036 7fc1e9d0b700 1 -- 172.21.2.112:0/3385255328 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fc1e4082980 con 0x7fc1e4082bb0 2021-02-10T11:39:50.035 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:50.040 7fc1dbfff700 1 -- 172.21.2.112:0/3385255328 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.3 v64) v1 ==== 184+0+0 (secure 0 0 0) 0x7fc1dc004b10 con 0x7fc1e4082bb0 2021-02-10T11:39:50.038 INFO:teuthology.orchestra.run.gibba012.stderr:updated caps for client.3 2021-02-10T11:39:50.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:50.045 7fc1e9d0b700 1 -- 172.21.2.112:0/3385255328 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc1c00186c0 msgr2=0x7fc1c001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:50.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:50.045 7fc1e9d0b700 1 --2- 172.21.2.112:0/3385255328 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc1c00186c0 0x7fc1c001abc0 crc :-1 s=READY pgs=925 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:50.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:50.045 7fc1e9d0b700 1 -- 172.21.2.112:0/3385255328 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc1e4082bb0 msgr2=0x7fc1e4082fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:50.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:50.045 7fc1e9d0b700 1 --2- 172.21.2.112:0/3385255328 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc1e4082bb0 0x7fc1e4082fb0 secure :-1 s=READY pgs=1983 cs=0 l=1 rev1=1 rx=0x7fc1dc00a940 tx=0x7fc1dc00ace0).stop 2021-02-10T11:39:50.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:50.045 7fc1e9d0b700 1 -- 172.21.2.112:0/3385255328 shutdown_connections 2021-02-10T11:39:50.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:50.045 7fc1e9d0b700 1 --2- 172.21.2.112:0/3385255328 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc1c00186c0 0x7fc1c001abc0 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:50.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:50.045 7fc1e9d0b700 1 --2- 172.21.2.112:0/3385255328 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc1e40b66b0 0x7fc1e40b6ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:50.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:50.045 7fc1e9d0b700 1 --2- 172.21.2.112:0/3385255328 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc1e4082bb0 0x7fc1e4082fb0 unknown :-1 s=CLOSED pgs=1983 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:50.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:50.045 7fc1e9d0b700 1 --2- 172.21.2.112:0/3385255328 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc1e41a0580 0x7fc1e4086fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:50.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:50.045 7fc1e9d0b700 1 -- 172.21.2.112:0/3385255328 >> 172.21.2.112:0/3385255328 conn(0x7fc1e40afc60 msgr2=0x7fc1e40b50d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:50.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:50.045 7fc1e9d0b700 1 -- 172.21.2.112:0/3385255328 shutdown_connections 2021-02-10T11:39:50.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:50.045 7fc1e9d0b700 1 -- 172.21.2.112:0/3385255328 wait complete. 2021-02-10T11:39:50.049 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:39:50.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:50.462 7f451a49e700 1 Processor -- start 2021-02-10T11:39:50.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:50.462 7f451a49e700 1 -- start start 2021-02-10T11:39:50.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:50.463 7f451a49e700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f45140d0340 0x7f45140ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:50.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:50.463 7f451a49e700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f45140d1c70 0x7f45140efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:50.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:50.463 7f451a49e700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45140f0b10 con 0x7f45140d1870 2021-02-10T11:39:50.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:50.463 7f451a49e700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45140e5ed0 con 0x7f45140d0340 2021-02-10T11:39:50.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:50.463 7f451a49e700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45141b0be0 con 0x7f45140d1c70 2021-02-10T11:39:50.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:50.463 7f4513fff700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f45140d1870 0x7f45141b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:53624/0 (socket says 172.21.2.112:53624) 2021-02-10T11:39:50.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:50.463 7f4513fff700 1 -- 172.21.2.112:0/509599080 learned_addr learned my addr 172.21.2.112:0/509599080 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:39:50.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:50.463 7f4518abb700 1 --2- 172.21.2.112:0/509599080 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f45140d1c70 0x7f45140efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:50.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:50.463 7f45137fe700 1 --2- 172.21.2.112:0/509599080 >> v2:172.21.2.120:3300/0 conn(0x7f45140d0340 0x7f45140ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:50.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:50.464 7f4512ffd700 1 -- 172.21.2.112:0/509599080 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1767265132 0 0) 0x7f4504001180 con 0x7f45140d1870 2021-02-10T11:39:50.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:50.464 7f4512ffd700 1 -- 172.21.2.112:0/509599080 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44fc002fe0 con 0x7f45140d1870 2021-02-10T11:39:50.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:50.464 7f4518abb700 1 -- 172.21.2.112:0/509599080 >> v2:172.21.2.120:3300/0 conn(0x7f45140d0340 msgr2=0x7f45140ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:50.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:50.464 7f4518abb700 1 --2- 172.21.2.112:0/509599080 >> v2:172.21.2.120:3300/0 conn(0x7f45140d0340 0x7f45140ef640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:50.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:50.464 7f4518abb700 1 -- 172.21.2.112:0/509599080 >> v1:172.21.2.112:6789/0 conn(0x7f45140d1870 legacy=0x7f45141b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:50.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:50.464 7f4518abb700 1 -- 172.21.2.112:0/509599080 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45140f20a0 con 0x7f45140d1c70 2021-02-10T11:39:50.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:50.464 7f4518abb700 1 --2- 172.21.2.112:0/509599080 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f45140d1c70 0x7f45140efc60 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7f450c006e50 tx=0x7f450c009e70).ready entity=mon.2 client_cookie=d550af2072f3ac31 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:50.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:50.464 7f4512ffd700 1 -- 172.21.2.112:0/509599080 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f450c00b510 con 0x7f45140d1c70 2021-02-10T11:39:50.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:50.464 7f4512ffd700 1 -- 172.21.2.112:0/509599080 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f450c006790 con 0x7f45140d1c70 2021-02-10T11:39:50.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:50.464 7f4512ffd700 1 -- 172.21.2.112:0/509599080 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f450c00b940 con 0x7f45140d1c70 2021-02-10T11:39:50.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:50.464 7f451a49e700 1 -- 172.21.2.112:0/509599080 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f45140d1c70 msgr2=0x7f45140efc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:50.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:50.464 7f451a49e700 1 --2- 172.21.2.112:0/509599080 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f45140d1c70 0x7f45140efc60 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7f450c006e50 tx=0x7f450c009e70).stop 2021-02-10T11:39:50.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:50.465 7f451a49e700 1 -- 172.21.2.112:0/509599080 shutdown_connections 2021-02-10T11:39:50.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:50.465 7f451a49e700 1 --2- 172.21.2.112:0/509599080 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f45140d1c70 0x7f45140efc60 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:50.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:50.465 7f451a49e700 1 --2- 172.21.2.112:0/509599080 >> v2:172.21.2.120:3300/0 conn(0x7f45140d0340 0x7f45140ef640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:50.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:50.465 7f451a49e700 1 -- 172.21.2.112:0/509599080 >> 172.21.2.112:0/509599080 conn(0x7f45140ce450 msgr2=0x7f45140d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:50.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:50.465 7f451a49e700 1 -- 172.21.2.112:0/509599080 shutdown_connections 2021-02-10T11:39:50.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:50.465 7f451a49e700 1 -- 172.21.2.112:0/509599080 wait complete. 2021-02-10T11:39:50.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:50.480 7f451a49e700 1 Processor -- start 2021-02-10T11:39:50.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:50.480 7f451a49e700 1 -- start start 2021-02-10T11:39:50.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:50.480 7f451a49e700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f45140beeb0 0x7f45140bf2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:50.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:50.480 7f451a49e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f451418ef70 0x7f451418f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:50.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:50.480 7f451a49e700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f451418f990 0x7f45140c01d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:50.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:50.480 7f451a49e700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45140bfc50 con 0x7f451418ef70 2021-02-10T11:39:50.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:50.480 7f451a49e700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f451418fd90 con 0x7f451418f990 2021-02-10T11:39:50.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:50.480 7f451a49e700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45140bf8d0 con 0x7f45140beeb0 2021-02-10T11:39:50.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:50.480 7f45137fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f451418ef70 0x7f451418f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:50.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:50.480 7f45137fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f451418ef70 0x7f451418f370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:54168/0 (socket says 172.21.2.112:54168) 2021-02-10T11:39:50.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:50.480 7f45137fe700 1 -- 172.21.2.112:0/2744671349 learned_addr learned my addr 172.21.2.112:0/2744671349 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:39:50.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:50.481 7f45137fe700 1 -- 172.21.2.112:0/2744671349 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f45140beeb0 msgr2=0x7f45140bf2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:50.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:50.481 7f45137fe700 1 --2- 172.21.2.112:0/2744671349 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f45140beeb0 0x7f45140bf2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:50.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:50.481 7f45137fe700 1 -- 172.21.2.112:0/2744671349 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f451418f990 msgr2=0x7f45140c01d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:50.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:50.481 7f45137fe700 1 --2- 172.21.2.112:0/2744671349 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f451418f990 0x7f45140c01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:50.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:50.481 7f45137fe700 1 -- 172.21.2.112:0/2744671349 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45140c07f0 con 0x7f451418ef70 2021-02-10T11:39:50.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:50.481 7f45137fe700 1 --2- 172.21.2.112:0/2744671349 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f451418ef70 0x7f451418f370 secure :-1 s=READY pgs=1985 cs=0 l=1 rev1=1 rx=0x7f4508006f60 tx=0x7f450800b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:50.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:50.481 7f4510ff9700 1 -- 172.21.2.112:0/2744671349 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f45080009b0 con 0x7f451418ef70 2021-02-10T11:39:50.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:50.481 7f4510ff9700 1 -- 172.21.2.112:0/2744671349 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f450800bdf0 con 0x7f451418ef70 2021-02-10T11:39:50.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:50.481 7f4510ff9700 1 -- 172.21.2.112:0/2744671349 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4508002030 con 0x7f451418ef70 2021-02-10T11:39:50.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:50.481 7f451a49e700 1 -- 172.21.2.112:0/2744671349 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45140c07f0 con 0x7f451418ef70 2021-02-10T11:39:50.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:50.481 7f451a49e700 1 -- 172.21.2.112:0/2744671349 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45140c0d80 con 0x7f451418ef70 2021-02-10T11:39:50.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:50.482 7f4510ff9700 1 -- 172.21.2.112:0/2744671349 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4508006b00 con 0x7f451418ef70 2021-02-10T11:39:50.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:50.482 7f4510ff9700 1 --2- 172.21.2.112:0/2744671349 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f44f40186c0 0x7f44f401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:50.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:50.482 7f4510ff9700 1 -- 172.21.2.112:0/2744671349 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f4508032590 con 0x7f451418ef70 2021-02-10T11:39:50.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:50.482 7f4513fff700 1 --2- 172.21.2.112:0/2744671349 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f44f40186c0 0x7f44f401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:50.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:50.483 7f4513fff700 1 --2- 172.21.2.112:0/2744671349 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f44f40186c0 0x7f44f401abc0 crc :-1 s=READY pgs=926 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:50.508 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:50.513 7f451a49e700 1 -- 172.21.2.112:0/2744671349 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45140ebc60 con 0x7f451418ef70 2021-02-10T11:39:50.510 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:50.515 7f4510ff9700 1 -- 172.21.2.112:0/2744671349 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4508015410 con 0x7f451418ef70 2021-02-10T11:39:50.642 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:50.647 7f451a49e700 1 -- 172.21.2.112:0/2744671349 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f45140f0b10 con 0x7f451418ef70 2021-02-10T11:39:50.642 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:50.647 7f4510ff9700 1 -- 172.21.2.112:0/2744671349 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+2104 (secure 0 0 0) 0x7f4508015410 con 0x7f451418ef70 2021-02-10T11:39:50.643 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:39:50.643 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":80,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10011,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/3938237927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":3938237927},{"type":"v1","addr":"172.21.2.112:6833","nonce":3938237927}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":76}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:39:44.311361","modified":"2021-02-10 11:39:46.320259","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7633},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7633":{"gid":7633,"name":"b","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.2.120:6835/1537197833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":1537197833},{"type":"v1","addr":"172.21.2.120:6835","nonce":1537197833}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":10}]} 2021-02-10T11:39:50.643 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 80 2021-02-10T11:39:50.644 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:50.649 7f451a49e700 1 -- 172.21.2.112:0/2744671349 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f44f40186c0 msgr2=0x7f44f401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:50.644 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:50.649 7f451a49e700 1 --2- 172.21.2.112:0/2744671349 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f44f40186c0 0x7f44f401abc0 crc :-1 s=READY pgs=926 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:50.644 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:50.649 7f451a49e700 1 -- 172.21.2.112:0/2744671349 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f451418ef70 msgr2=0x7f451418f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:50.644 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:50.649 7f451a49e700 1 --2- 172.21.2.112:0/2744671349 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f451418ef70 0x7f451418f370 secure :-1 s=READY pgs=1985 cs=0 l=1 rev1=1 rx=0x7f4508006f60 tx=0x7f450800b970).stop 2021-02-10T11:39:50.645 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:50.649 7f451a49e700 1 -- 172.21.2.112:0/2744671349 shutdown_connections 2021-02-10T11:39:50.645 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:50.649 7f451a49e700 1 --2- 172.21.2.112:0/2744671349 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f44f40186c0 0x7f44f401abc0 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:50.645 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:50.649 7f451a49e700 1 --2- 172.21.2.112:0/2744671349 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f451418f990 0x7f45140c01d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:50.645 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:50.649 7f451a49e700 1 --2- 172.21.2.112:0/2744671349 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f451418ef70 0x7f451418f370 unknown :-1 s=CLOSED pgs=1985 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:50.645 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:50.649 7f451a49e700 1 --2- 172.21.2.112:0/2744671349 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f45140beeb0 0x7f45140bf2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:50.645 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:50.649 7f451a49e700 1 -- 172.21.2.112:0/2744671349 >> 172.21.2.112:0/2744671349 conn(0x7f45140c2130 msgr2=0x7f451409e3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:50.645 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:50.650 7f451a49e700 1 -- 172.21.2.112:0/2744671349 shutdown_connections 2021-02-10T11:39:50.645 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:50.650 7f451a49e700 1 -- 172.21.2.112:0/2744671349 wait complete. 2021-02-10T11:39:50.854 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 80, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:39:44.311361', 'modified': '2021-02-10 11:39:46.320259', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7633}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7633': {'gid': 7633, 'name': 'b', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.120:6835/1537197833', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.120:6834', 'nonce': 1537197833}, {'type': 'v1', 'addr': '172.21.2.120:6835', 'nonce': 1537197833}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:39:50.854 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:39:50.855 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:39:51.254 INFO:teuthology.orchestra.run.gibba020.stdout:{ 2021-02-10T11:39:51.254 INFO:teuthology.orchestra.run.gibba020.stdout: "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", 2021-02-10T11:39:51.254 INFO:teuthology.orchestra.run.gibba020.stdout: "whoami": 0, 2021-02-10T11:39:51.254 INFO:teuthology.orchestra.run.gibba020.stdout: "id": 7633, 2021-02-10T11:39:51.254 INFO:teuthology.orchestra.run.gibba020.stdout: "want_state": "up:active", 2021-02-10T11:39:51.254 INFO:teuthology.orchestra.run.gibba020.stdout: "state": "up:active", 2021-02-10T11:39:51.255 INFO:teuthology.orchestra.run.gibba020.stdout: "rank_uptime": 6.9468760039999999, 2021-02-10T11:39:51.255 INFO:teuthology.orchestra.run.gibba020.stdout: "mdsmap_epoch": 80, 2021-02-10T11:39:51.255 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch": 100, 2021-02-10T11:39:51.255 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch_barrier": 100, 2021-02-10T11:39:51.255 INFO:teuthology.orchestra.run.gibba020.stdout: "uptime": 12.5877205 2021-02-10T11:39:51.255 INFO:teuthology.orchestra.run.gibba020.stdout:} 2021-02-10T11:39:51.258 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", "whoami": 0, "id": 7633, "want_state": "up:active", "state": "up:active", "rank_uptime": 6.9468760039999999, "mdsmap_epoch": 80, "osdmap_epoch": 100, "osdmap_epoch_barrier": 100, "uptime": 12.5877205 } 2021-02-10T11:39:51.258 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:39:51.284 DEBUG:teuthology.orchestra.run.gibba012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:39:51.304 DEBUG:teuthology.orchestra.run.gibba020:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:39:51.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:51.677 7fed7a3b1700 1 Processor -- start 2021-02-10T11:39:51.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:51.677 7fed7a3b1700 1 -- start start 2021-02-10T11:39:51.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:51.677 7fed7a3b1700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fed740d1870 0x7fed741b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:51.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:51.677 7fed7a3b1700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fed741b0a10 0x7fed740cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:51.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:51.677 7fed7a3b1700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed740f0b10 con 0x7fed740d1c70 2021-02-10T11:39:51.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:51.677 7fed7a3b1700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed740e5ed0 con 0x7fed741b0a10 2021-02-10T11:39:51.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:51.677 7fed7a3b1700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed740f0650 con 0x7fed740d1870 2021-02-10T11:39:51.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:51.677 7fed737fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fed741b0a10 0x7fed740cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:51.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:51.677 7fed737fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fed741b0a10 0x7fed740cf330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:54180/0 (socket says 172.21.2.112:54180) 2021-02-10T11:39:51.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:51.677 7fed737fe700 1 -- 172.21.2.112:0/2429126840 learned_addr learned my addr 172.21.2.112:0/2429126840 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:39:51.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:51.678 7fed737fe700 1 -- 172.21.2.112:0/2429126840 >> v2:172.21.2.120:3300/0 conn(0x7fed740d1870 msgr2=0x7fed741b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:51.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:51.678 7fed737fe700 1 --2- 172.21.2.112:0/2429126840 >> v2:172.21.2.120:3300/0 conn(0x7fed740d1870 0x7fed741b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:51.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:51.678 7fed737fe700 1 -- 172.21.2.112:0/2429126840 >> v1:172.21.2.120:6789/0 conn(0x7fed740d1c70 legacy=0x7fed740cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:51.689 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:51.678 7fed737fe700 1 -- 172.21.2.112:0/2429126840 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed740f0840 con 0x7fed741b0a10 2021-02-10T11:39:51.689 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:51.678 7fed737fe700 1 --2- 172.21.2.112:0/2429126840 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fed741b0a10 0x7fed740cf330 secure :-1 s=READY pgs=1986 cs=0 l=1 rev1=1 rx=0x7fed68006700 tx=0x7fed68009b80).ready entity=mon.0 client_cookie=917f3706a7f09c2a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:51.689 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:51.678 7fed72ffd700 1 -- 172.21.2.112:0/2429126840 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fed6800c0c0 con 0x7fed741b0a10 2021-02-10T11:39:51.689 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:51.678 7fed72ffd700 1 -- 172.21.2.112:0/2429126840 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed6800c440 con 0x7fed741b0a10 2021-02-10T11:39:51.689 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:51.678 7fed72ffd700 1 -- 172.21.2.112:0/2429126840 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fed6800c8a0 con 0x7fed741b0a10 2021-02-10T11:39:51.689 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:51.678 7fed7a3b1700 1 -- 172.21.2.112:0/2429126840 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fed741b0a10 msgr2=0x7fed740cf330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:51.689 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:51.678 7fed7a3b1700 1 --2- 172.21.2.112:0/2429126840 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fed741b0a10 0x7fed740cf330 secure :-1 s=READY pgs=1986 cs=0 l=1 rev1=1 rx=0x7fed68006700 tx=0x7fed68009b80).stop 2021-02-10T11:39:51.689 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:51.679 7fed7a3b1700 1 -- 172.21.2.112:0/2429126840 shutdown_connections 2021-02-10T11:39:51.689 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:51.679 7fed7a3b1700 1 --2- 172.21.2.112:0/2429126840 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fed741b0a10 0x7fed740cf330 unknown :-1 s=CLOSED pgs=1986 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:51.689 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:51.679 7fed7a3b1700 1 --2- 172.21.2.112:0/2429126840 >> v2:172.21.2.120:3300/0 conn(0x7fed740d1870 0x7fed741b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:51.689 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:51.679 7fed7a3b1700 1 -- 172.21.2.112:0/2429126840 >> 172.21.2.112:0/2429126840 conn(0x7fed740ce450 msgr2=0x7fed740d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:51.689 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:51.679 7fed7a3b1700 1 -- 172.21.2.112:0/2429126840 shutdown_connections 2021-02-10T11:39:51.690 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:51.679 7fed7a3b1700 1 -- 172.21.2.112:0/2429126840 wait complete. 2021-02-10T11:39:51.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:51.694 7fed7a3b1700 1 Processor -- start 2021-02-10T11:39:51.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:51.694 7fed7a3b1700 1 -- start start 2021-02-10T11:39:51.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:51.694 7fed7a3b1700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fed741ae550 0x7fed741924f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:51.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:51.694 7fed7a3b1700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fed741ae950 0x7fed741a9aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:51.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:51.694 7fed7a3b1700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fed741aa0c0 0x7fed741aa4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:51.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:51.694 7fed7a3b1700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed741a9520 con 0x7fed741ae950 2021-02-10T11:39:51.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:51.694 7fed7a3b1700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed7418fee0 con 0x7fed741aa0c0 2021-02-10T11:39:51.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:51.694 7fed7a3b1700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed7418f9d0 con 0x7fed741ae550 2021-02-10T11:39:51.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:51.694 7fed737fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fed741ae950 0x7fed741a9aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:51.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:51.694 7fed737fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fed741ae950 0x7fed741a9aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:54182/0 (socket says 172.21.2.112:54182) 2021-02-10T11:39:51.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:51.694 7fed737fe700 1 -- 172.21.2.112:0/122773775 learned_addr learned my addr 172.21.2.112:0/122773775 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:39:51.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:51.695 7fed737fe700 1 -- 172.21.2.112:0/122773775 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fed741ae550 msgr2=0x7fed741924f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:51.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:51.695 7fed737fe700 1 --2- 172.21.2.112:0/122773775 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fed741ae550 0x7fed741924f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:51.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:51.695 7fed737fe700 1 -- 172.21.2.112:0/122773775 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fed741aa0c0 msgr2=0x7fed741aa4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:51.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:51.695 7fed737fe700 1 --2- 172.21.2.112:0/122773775 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fed741aa0c0 0x7fed741aa4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:51.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:51.695 7fed737fe700 1 -- 172.21.2.112:0/122773775 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed741a9170 con 0x7fed741ae950 2021-02-10T11:39:51.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:51.695 7fed737fe700 1 --2- 172.21.2.112:0/122773775 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fed741ae950 0x7fed741a9aa0 secure :-1 s=READY pgs=1987 cs=0 l=1 rev1=1 rx=0x7fed6800b5e0 tx=0x7fed6800b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:51.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:51.695 7fed70ff9700 1 -- 172.21.2.112:0/122773775 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fed6800bd70 con 0x7fed741ae950 2021-02-10T11:39:51.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:51.695 7fed70ff9700 1 -- 172.21.2.112:0/122773775 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed68002030 con 0x7fed741ae950 2021-02-10T11:39:51.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:51.695 7fed70ff9700 1 -- 172.21.2.112:0/122773775 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fed68002550 con 0x7fed741ae950 2021-02-10T11:39:51.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:51.695 7fed7a3b1700 1 -- 172.21.2.112:0/122773775 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed741a9170 con 0x7fed741ae950 2021-02-10T11:39:51.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:51.695 7fed7a3b1700 1 -- 172.21.2.112:0/122773775 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed741aadd0 con 0x7fed741ae950 2021-02-10T11:39:51.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:51.696 7fed70ff9700 1 -- 172.21.2.112:0/122773775 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fed68000c90 con 0x7fed741ae950 2021-02-10T11:39:51.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:51.696 7fed70ff9700 1 --2- 172.21.2.112:0/122773775 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fed540186c0 0x7fed5401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:51.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:51.696 7fed70ff9700 1 -- 172.21.2.112:0/122773775 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fed68032470 con 0x7fed741ae950 2021-02-10T11:39:51.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:51.696 7fed73fff700 1 --2- 172.21.2.112:0/122773775 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fed540186c0 0x7fed5401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:51.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:51.697 7fed73fff700 1 --2- 172.21.2.112:0/122773775 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fed540186c0 0x7fed5401abc0 crc :-1 s=READY pgs=927 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:51.721 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:51.727 7fed7a3b1700 1 -- 172.21.2.112:0/122773775 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed740ee060 con 0x7fed741ae950 2021-02-10T11:39:51.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:51.729 7fed70ff9700 1 -- 172.21.2.112:0/122773775 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fed68021f30 con 0x7fed741ae950 2021-02-10T11:39:51.850 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:51.856 7fed7a3b1700 1 -- 172.21.2.112:0/122773775 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fed740ee060 con 0x7fed741ae950 2021-02-10T11:39:51.850 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:51.856 7fed70ff9700 1 -- 172.21.2.112:0/122773775 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v100) v1 ==== 74+0+13998 (secure 0 0 0) 0x7fed68021f30 con 0x7fed741ae950 2021-02-10T11:39:51.851 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:39:51.851 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":100,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:39:45.313226","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:39:42.230237","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"99","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":21,"pool_name":"cephfs_data","create_time":"2021-02-10 11:39:43.354655","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:39:51.852 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:51.858 7fed7a3b1700 1 -- 172.21.2.112:0/122773775 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fed540186c0 msgr2=0x7fed5401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:51.853 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:51.858 7fed7a3b1700 1 --2- 172.21.2.112:0/122773775 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fed540186c0 0x7fed5401abc0 crc :-1 s=READY pgs=927 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:51.853 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:51.858 7fed7a3b1700 1 -- 172.21.2.112:0/122773775 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fed741ae950 msgr2=0x7fed741a9aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:51.853 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:51.858 7fed7a3b1700 1 --2- 172.21.2.112:0/122773775 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fed741ae950 0x7fed741a9aa0 secure :-1 s=READY pgs=1987 cs=0 l=1 rev1=1 rx=0x7fed6800b5e0 tx=0x7fed6800b980).stop 2021-02-10T11:39:51.853 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:51.858 7fed7a3b1700 1 -- 172.21.2.112:0/122773775 shutdown_connections 2021-02-10T11:39:51.853 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:51.858 7fed7a3b1700 1 --2- 172.21.2.112:0/122773775 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fed540186c0 0x7fed5401abc0 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:51.853 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:51.858 7fed7a3b1700 1 --2- 172.21.2.112:0/122773775 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fed741aa0c0 0x7fed741aa4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:51.853 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:51.858 7fed7a3b1700 1 --2- 172.21.2.112:0/122773775 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fed741ae950 0x7fed741a9aa0 unknown :-1 s=CLOSED pgs=1987 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:51.853 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:51.858 7fed7a3b1700 1 --2- 172.21.2.112:0/122773775 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fed741ae550 0x7fed741924f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:51.853 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:51.858 7fed7a3b1700 1 -- 172.21.2.112:0/122773775 >> 172.21.2.112:0/122773775 conn(0x7fed741a8c90 msgr2=0x7fed741b38a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:51.853 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:51.859 7fed7a3b1700 1 -- 172.21.2.112:0/122773775 shutdown_connections 2021-02-10T11:39:51.853 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:51.859 7fed7a3b1700 1 -- 172.21.2.112:0/122773775 wait complete. 2021-02-10T11:39:51.861 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:13:57.508758', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 20, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:39:42.230237', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '99', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 21, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:39:43.354655', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '100', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:39:51.862 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:39:52.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.319 7f47f1837700 1 Processor -- start 2021-02-10T11:39:52.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.319 7f47f1837700 1 -- start start 2021-02-10T11:39:52.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.319 7f47f1837700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f47ec0d1870 0x7f47ec1b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:52.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.319 7f47f1837700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f47ec0cff70 0x7f47ec109130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:52.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.319 7f47f1837700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47ec0fe1c0 con 0x7f47ec1b0ae0 2021-02-10T11:39:52.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.319 7f47f1837700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47ec109ad0 con 0x7f47ec0cff70 2021-02-10T11:39:52.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.319 7f47f1837700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47ec109750 con 0x7f47ec0d1870 2021-02-10T11:39:52.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.320 7f47ead73700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f47ec1b0ae0 0x7f47ec108940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:53652/0 (socket says 172.21.2.112:53652) 2021-02-10T11:39:52.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.320 7f47ead73700 1 -- 172.21.2.112:0/448777337 learned_addr learned my addr 172.21.2.112:0/448777337 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:39:52.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.320 7f47ebd75700 1 --2- 172.21.2.112:0/448777337 >> v2:172.21.2.112:3300/0 conn(0x7f47ec0cff70 0x7f47ec109130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:52.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.320 7f47eb574700 1 --2- 172.21.2.112:0/448777337 >> v2:172.21.2.120:3300/0 conn(0x7f47ec0d1870 0x7f47ec1b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:52.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.320 7f47ea572700 1 -- 172.21.2.112:0/448777337 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3902134625 0 0) 0x7f47d40010b0 con 0x7f47ec1b0ae0 2021-02-10T11:39:52.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.320 7f47ea572700 1 -- 172.21.2.112:0/448777337 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47d8002fe0 con 0x7f47ec1b0ae0 2021-02-10T11:39:52.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.320 7f47ebd75700 1 -- 172.21.2.112:0/448777337 >> v2:172.21.2.120:3300/0 conn(0x7f47ec0d1870 msgr2=0x7f47ec1b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:52.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.320 7f47ebd75700 1 --2- 172.21.2.112:0/448777337 >> v2:172.21.2.120:3300/0 conn(0x7f47ec0d1870 0x7f47ec1b04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:52.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.320 7f47ebd75700 1 -- 172.21.2.112:0/448777337 >> v1:172.21.2.112:6789/0 conn(0x7f47ec1b0ae0 legacy=0x7f47ec108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:52.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.320 7f47ebd75700 1 -- 172.21.2.112:0/448777337 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47ec088070 con 0x7f47ec0cff70 2021-02-10T11:39:52.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.320 7f47ebd75700 1 --2- 172.21.2.112:0/448777337 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f47ec0cff70 0x7f47ec109130 secure :-1 s=READY pgs=1989 cs=0 l=1 rev1=1 rx=0x7f47e4006e30 tx=0x7f47e4009de0).ready entity=mon.0 client_cookie=5445e72c2181b3c2 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:52.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.320 7f47ea572700 1 -- 172.21.2.112:0/448777337 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f47e400b300 con 0x7f47ec0cff70 2021-02-10T11:39:52.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.320 7f47ea572700 1 -- 172.21.2.112:0/448777337 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47e400b680 con 0x7f47ec0cff70 2021-02-10T11:39:52.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.320 7f47ea572700 1 -- 172.21.2.112:0/448777337 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f47e400bb60 con 0x7f47ec0cff70 2021-02-10T11:39:52.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.320 7f47f1837700 1 -- 172.21.2.112:0/448777337 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f47ec0cff70 msgr2=0x7f47ec109130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:52.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.320 7f47f1837700 1 --2- 172.21.2.112:0/448777337 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f47ec0cff70 0x7f47ec109130 secure :-1 s=READY pgs=1989 cs=0 l=1 rev1=1 rx=0x7f47e4006e30 tx=0x7f47e4009de0).stop 2021-02-10T11:39:52.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.321 7f47f1837700 1 -- 172.21.2.112:0/448777337 shutdown_connections 2021-02-10T11:39:52.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.321 7f47f1837700 1 --2- 172.21.2.112:0/448777337 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f47ec0cff70 0x7f47ec109130 unknown :-1 s=CLOSED pgs=1989 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:52.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.321 7f47f1837700 1 --2- 172.21.2.112:0/448777337 >> v2:172.21.2.120:3300/0 conn(0x7f47ec0d1870 0x7f47ec1b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:52.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.321 7f47f1837700 1 -- 172.21.2.112:0/448777337 >> 172.21.2.112:0/448777337 conn(0x7f47ec0ce420 msgr2=0x7f47ec0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:52.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.321 7f47f1837700 1 -- 172.21.2.112:0/448777337 shutdown_connections 2021-02-10T11:39:52.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.321 7f47f1837700 1 -- 172.21.2.112:0/448777337 wait complete. 2021-02-10T11:39:52.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.336 7f47f1837700 1 Processor -- start 2021-02-10T11:39:52.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.336 7f47f1837700 1 -- start start 2021-02-10T11:39:52.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.336 7f47f1837700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f47ec1a79a0 0x7f47ec1a7da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:52.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.336 7f47f1837700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f47ec0bdb40 0x7f47ec0bdf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:52.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.336 7f47f1837700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f47ec082ca0 0x7f47ec0830a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:52.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.336 7f47f1837700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47ec19d990 con 0x7f47ec1a79a0 2021-02-10T11:39:52.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.336 7f47f1837700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47ec19dbb0 con 0x7f47ec082ca0 2021-02-10T11:39:52.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.336 7f47f1837700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47ec0836c0 con 0x7f47ec0bdb40 2021-02-10T11:39:52.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.336 7f47eb574700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f47ec1a79a0 0x7f47ec1a7da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:52.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.336 7f47eb574700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f47ec1a79a0 0x7f47ec1a7da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:54196/0 (socket says 172.21.2.112:54196) 2021-02-10T11:39:52.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.336 7f47eb574700 1 -- 172.21.2.112:0/1731952788 learned_addr learned my addr 172.21.2.112:0/1731952788 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:39:52.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.337 7f47eb574700 1 -- 172.21.2.112:0/1731952788 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f47ec0bdb40 msgr2=0x7f47ec0bdf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:52.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.337 7f47eb574700 1 --2- 172.21.2.112:0/1731952788 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f47ec0bdb40 0x7f47ec0bdf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:52.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.337 7f47eb574700 1 -- 172.21.2.112:0/1731952788 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f47ec082ca0 msgr2=0x7f47ec0830a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:52.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.337 7f47eb574700 1 --2- 172.21.2.112:0/1731952788 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f47ec082ca0 0x7f47ec0830a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:52.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.337 7f47eb574700 1 -- 172.21.2.112:0/1731952788 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47ec0838e0 con 0x7f47ec1a79a0 2021-02-10T11:39:52.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.337 7f47eb574700 1 --2- 172.21.2.112:0/1731952788 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f47ec1a79a0 0x7f47ec1a7da0 secure :-1 s=READY pgs=1990 cs=0 l=1 rev1=1 rx=0x7f47dc006be0 tx=0x7f47dc00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:52.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.337 7f47e3fff700 1 -- 172.21.2.112:0/1731952788 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f47dc0064d0 con 0x7f47ec1a79a0 2021-02-10T11:39:52.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.337 7f47e3fff700 1 -- 172.21.2.112:0/1731952788 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47dc00bd80 con 0x7f47ec1a79a0 2021-02-10T11:39:52.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.337 7f47f1837700 1 -- 172.21.2.112:0/1731952788 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47ec0838e0 con 0x7f47ec1a79a0 2021-02-10T11:39:52.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.337 7f47e3fff700 1 -- 172.21.2.112:0/1731952788 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f47dc002030 con 0x7f47ec1a79a0 2021-02-10T11:39:52.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.337 7f47f1837700 1 -- 172.21.2.112:0/1731952788 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47ec0838e0 con 0x7f47ec1a79a0 2021-02-10T11:39:52.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.338 7f47e3fff700 1 -- 172.21.2.112:0/1731952788 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f47dc006c90 con 0x7f47ec1a79a0 2021-02-10T11:39:52.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.338 7f47e3fff700 1 --2- 172.21.2.112:0/1731952788 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f47d0018730 0x7f47d001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:52.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.338 7f47e3fff700 1 -- 172.21.2.112:0/1731952788 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f47dc032620 con 0x7f47ec1a79a0 2021-02-10T11:39:52.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.338 7f47ead73700 1 --2- 172.21.2.112:0/1731952788 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f47d0018730 0x7f47d001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:52.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.339 7f47ead73700 1 --2- 172.21.2.112:0/1731952788 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f47d0018730 0x7f47d001ac10 crc :-1 s=READY pgs=928 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:52.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.369 7f47f1837700 1 -- 172.21.2.112:0/1731952788 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47ec082a70 con 0x7f47ec1a79a0 2021-02-10T11:39:52.366 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.372 7f47e3fff700 1 -- 172.21.2.112:0/1731952788 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f47dc006820 con 0x7f47ec1a79a0 2021-02-10T11:39:52.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.498 7f47f1837700 1 -- 172.21.2.112:0/1731952788 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f47ec082a70 con 0x7f47ec1a79a0 2021-02-10T11:39:52.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.499 7f47e3fff700 1 -- 172.21.2.112:0/1731952788 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v100) v1 ==== 92+0+10 (secure 0 0 0) 0x7f47dc00bd80 con 0x7f47ec1a79a0 2021-02-10T11:39:52.493 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:39:52.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.500 7f47f1837700 1 -- 172.21.2.112:0/1731952788 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f47d0018730 msgr2=0x7f47d001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:52.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.500 7f47f1837700 1 --2- 172.21.2.112:0/1731952788 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f47d0018730 0x7f47d001ac10 crc :-1 s=READY pgs=928 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:52.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.500 7f47f1837700 1 -- 172.21.2.112:0/1731952788 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f47ec1a79a0 msgr2=0x7f47ec1a7da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:52.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.500 7f47f1837700 1 --2- 172.21.2.112:0/1731952788 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f47ec1a79a0 0x7f47ec1a7da0 secure :-1 s=READY pgs=1990 cs=0 l=1 rev1=1 rx=0x7f47dc006be0 tx=0x7f47dc00b990).stop 2021-02-10T11:39:52.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.501 7f47f1837700 1 -- 172.21.2.112:0/1731952788 shutdown_connections 2021-02-10T11:39:52.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.501 7f47f1837700 1 --2- 172.21.2.112:0/1731952788 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f47d0018730 0x7f47d001ac10 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:52.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.501 7f47f1837700 1 --2- 172.21.2.112:0/1731952788 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f47ec082ca0 0x7f47ec0830a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:52.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.501 7f47f1837700 1 --2- 172.21.2.112:0/1731952788 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f47ec0bdb40 0x7f47ec0bdf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:52.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.501 7f47f1837700 1 --2- 172.21.2.112:0/1731952788 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f47ec1a79a0 0x7f47ec1a7da0 unknown :-1 s=CLOSED pgs=1990 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:52.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.501 7f47f1837700 1 -- 172.21.2.112:0/1731952788 >> 172.21.2.112:0/1731952788 conn(0x7f47ec0bef00 msgr2=0x7f47ec09f200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:52.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.501 7f47f1837700 1 -- 172.21.2.112:0/1731952788 shutdown_connections 2021-02-10T11:39:52.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.501 7f47f1837700 1 -- 172.21.2.112:0/1731952788 wait complete. 2021-02-10T11:39:52.503 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:39:52.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.922 7fee10ef4700 1 Processor -- start 2021-02-10T11:39:52.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.922 7fee10ef4700 1 -- start start 2021-02-10T11:39:52.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.922 7fee10ef4700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fee0c10aff0 0x7fee0c0cf750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:52.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.922 7fee10ef4700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee0c0fe1c0 con 0x7fee0c0d1c70 2021-02-10T11:39:52.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.922 7fee10ef4700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee0c1b0cb0 con 0x7fee0c0d1870 2021-02-10T11:39:52.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.922 7fee10ef4700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee0c109a60 con 0x7fee0c10aff0 2021-02-10T11:39:52.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.922 7fee0a312700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fee0c0d1c70 0x7fee0c0cef60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:53664/0 (socket says 172.21.2.112:53664) 2021-02-10T11:39:52.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.922 7fee0a312700 1 -- 172.21.2.112:0/1222753131 learned_addr learned my addr 172.21.2.112:0/1222753131 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:39:52.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.922 7fee0b314700 1 --2- 172.21.2.112:0/1222753131 >> v2:172.21.2.120:3300/0 conn(0x7fee0c10aff0 0x7fee0c0cf750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:52.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.923 7fee0b314700 1 -- 172.21.2.112:0/1222753131 >> v1:172.21.2.120:6789/0 conn(0x7fee0c0d1870 legacy=0x7fee0c1b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:52.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.923 7fee0b314700 1 -- 172.21.2.112:0/1222753131 >> v1:172.21.2.112:6789/0 conn(0x7fee0c0d1c70 legacy=0x7fee0c0cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:52.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.923 7fee09b11700 1 -- 172.21.2.112:0/1222753131 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2187459138 0 0) 0x7fee000010b0 con 0x7fee0c0d1c70 2021-02-10T11:39:52.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.923 7fee0b314700 1 -- 172.21.2.112:0/1222753131 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee0c10b3f0 con 0x7fee0c10aff0 2021-02-10T11:39:52.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.923 7fee0b314700 1 --2- 172.21.2.112:0/1222753131 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fee0c10aff0 0x7fee0c0cf750 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 rx=0x7fedfc006ad0 tx=0x7fedfc009ad0).ready entity=mon.1 client_cookie=e1118cbd0a1df0d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:52.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.923 7fee09b11700 1 -- 172.21.2.112:0/1222753131 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fedfc0022e0 con 0x7fee0c10aff0 2021-02-10T11:39:52.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.923 7fee09b11700 1 -- 172.21.2.112:0/1222753131 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedfc006410 con 0x7fee0c10aff0 2021-02-10T11:39:52.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.923 7fee09b11700 1 -- 172.21.2.112:0/1222753131 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fedfc002560 con 0x7fee0c10aff0 2021-02-10T11:39:52.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.923 7fee10ef4700 1 -- 172.21.2.112:0/1222753131 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fee0c10aff0 msgr2=0x7fee0c0cf750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:52.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.923 7fee10ef4700 1 --2- 172.21.2.112:0/1222753131 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fee0c10aff0 0x7fee0c0cf750 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 rx=0x7fedfc006ad0 tx=0x7fedfc009ad0).stop 2021-02-10T11:39:52.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.924 7fee10ef4700 1 -- 172.21.2.112:0/1222753131 shutdown_connections 2021-02-10T11:39:52.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.924 7fee10ef4700 1 --2- 172.21.2.112:0/1222753131 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fee0c10aff0 0x7fee0c0cf750 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:52.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.924 7fee10ef4700 1 -- 172.21.2.112:0/1222753131 >> 172.21.2.112:0/1222753131 conn(0x7fee0c0ce420 msgr2=0x7fee0c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:52.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.924 7fee10ef4700 1 -- 172.21.2.112:0/1222753131 shutdown_connections 2021-02-10T11:39:52.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.924 7fee10ef4700 1 -- 172.21.2.112:0/1222753131 wait complete. 2021-02-10T11:39:52.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.939 7fee10ef4700 1 Processor -- start 2021-02-10T11:39:52.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.939 7fee10ef4700 1 -- start start 2021-02-10T11:39:52.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.939 7fee10ef4700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fee0c0c2370 0x7fee0c18f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:52.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.939 7fee10ef4700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fee0c084cb0 0x7fee0c1a7470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:52.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.939 7fee10ef4700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fee0c1a7a90 0x7fee0c085700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:52.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.939 7fee10ef4700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee0c0be840 con 0x7fee0c0c2370 2021-02-10T11:39:52.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.939 7fee10ef4700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee0c0c29a0 con 0x7fee0c084cb0 2021-02-10T11:39:52.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.939 7fee10ef4700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee0c1a7e90 con 0x7fee0c1a7a90 2021-02-10T11:39:52.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.939 7fee0ab13700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fee0c0c2370 0x7fee0c18f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:52.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.939 7fee0ab13700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fee0c0c2370 0x7fee0c18f870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:54206/0 (socket says 172.21.2.112:54206) 2021-02-10T11:39:52.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.939 7fee0ab13700 1 -- 172.21.2.112:0/1673986359 learned_addr learned my addr 172.21.2.112:0/1673986359 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:39:52.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.940 7fee0ab13700 1 -- 172.21.2.112:0/1673986359 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fee0c1a7a90 msgr2=0x7fee0c085700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:52.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.940 7fee0ab13700 1 --2- 172.21.2.112:0/1673986359 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fee0c1a7a90 0x7fee0c085700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:52.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.940 7fee0ab13700 1 -- 172.21.2.112:0/1673986359 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fee0c084cb0 msgr2=0x7fee0c1a7470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:52.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.940 7fee0ab13700 1 --2- 172.21.2.112:0/1673986359 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fee0c084cb0 0x7fee0c1a7470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:52.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.940 7fee0ab13700 1 -- 172.21.2.112:0/1673986359 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee0c1a80b0 con 0x7fee0c0c2370 2021-02-10T11:39:52.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.940 7fee0ab13700 1 --2- 172.21.2.112:0/1673986359 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fee0c0c2370 0x7fee0c18f870 secure :-1 s=READY pgs=1992 cs=0 l=1 rev1=1 rx=0x7fee04009940 tx=0x7fee0400bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:52.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.940 7fedfb7fe700 1 -- 172.21.2.112:0/1673986359 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fee04005ab0 con 0x7fee0c0c2370 2021-02-10T11:39:52.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.940 7fedfb7fe700 1 -- 172.21.2.112:0/1673986359 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee04005d30 con 0x7fee0c0c2370 2021-02-10T11:39:52.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.940 7fee10ef4700 1 -- 172.21.2.112:0/1673986359 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee0c1a80b0 con 0x7fee0c0c2370 2021-02-10T11:39:52.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.940 7fee10ef4700 1 -- 172.21.2.112:0/1673986359 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee0c086010 con 0x7fee0c0c2370 2021-02-10T11:39:52.938 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.940 7fedfb7fe700 1 -- 172.21.2.112:0/1673986359 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fee040062b0 con 0x7fee0c0c2370 2021-02-10T11:39:52.938 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.941 7fedfb7fe700 1 -- 172.21.2.112:0/1673986359 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fee04006030 con 0x7fee0c0c2370 2021-02-10T11:39:52.938 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.941 7fedfb7fe700 1 --2- 172.21.2.112:0/1673986359 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fedf0018730 0x7fedf001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:52.938 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.941 7fedfb7fe700 1 -- 172.21.2.112:0/1673986359 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fee04032b80 con 0x7fee0c0c2370 2021-02-10T11:39:52.938 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.941 7fee0a312700 1 --2- 172.21.2.112:0/1673986359 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fedf0018730 0x7fedf001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:52.938 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.942 7fee0a312700 1 --2- 172.21.2.112:0/1673986359 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fedf0018730 0x7fedf001ac10 crc :-1 s=READY pgs=929 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:52.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.972 7fee10ef4700 1 -- 172.21.2.112:0/1673986359 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee0c082a70 con 0x7fee0c0c2370 2021-02-10T11:39:52.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:52.975 7fedfb7fe700 1 -- 172.21.2.112:0/1673986359 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fee04006570 con 0x7fee0c0c2370 2021-02-10T11:39:53.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:53.101 7fee10ef4700 1 -- 172.21.2.112:0/1673986359 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7fee0c082a70 con 0x7fee0c0c2370 2021-02-10T11:39:53.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:53.102 7fedfb7fe700 1 -- 172.21.2.112:0/1673986359 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v100) v1 ==== 104+0+10 (secure 0 0 0) 0x7fee04006030 con 0x7fee0c0c2370 2021-02-10T11:39:53.097 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:39:53.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:53.104 7fee10ef4700 1 -- 172.21.2.112:0/1673986359 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fedf0018730 msgr2=0x7fedf001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:53.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:53.104 7fee10ef4700 1 --2- 172.21.2.112:0/1673986359 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fedf0018730 0x7fedf001ac10 crc :-1 s=READY pgs=929 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:53.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:53.104 7fee10ef4700 1 -- 172.21.2.112:0/1673986359 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fee0c0c2370 msgr2=0x7fee0c18f870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:53.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:53.104 7fee10ef4700 1 --2- 172.21.2.112:0/1673986359 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fee0c0c2370 0x7fee0c18f870 secure :-1 s=READY pgs=1992 cs=0 l=1 rev1=1 rx=0x7fee04009940 tx=0x7fee0400bcf0).stop 2021-02-10T11:39:53.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:53.104 7fee10ef4700 1 -- 172.21.2.112:0/1673986359 shutdown_connections 2021-02-10T11:39:53.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:53.104 7fee10ef4700 1 --2- 172.21.2.112:0/1673986359 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fedf0018730 0x7fedf001ac10 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:53.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:53.104 7fee10ef4700 1 --2- 172.21.2.112:0/1673986359 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fee0c084cb0 0x7fee0c1a7470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:53.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:53.104 7fee10ef4700 1 --2- 172.21.2.112:0/1673986359 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fee0c1a7a90 0x7fee0c085700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:53.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:53.104 7fee10ef4700 1 --2- 172.21.2.112:0/1673986359 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fee0c0c2370 0x7fee0c18f870 unknown :-1 s=CLOSED pgs=1992 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:53.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:53.104 7fee10ef4700 1 -- 172.21.2.112:0/1673986359 >> 172.21.2.112:0/1673986359 conn(0x7fee0c0b5dc0 msgr2=0x7fee0c0c2d90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:53.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:53.104 7fee10ef4700 1 -- 172.21.2.112:0/1673986359 shutdown_connections 2021-02-10T11:39:53.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:53.104 7fee10ef4700 1 -- 172.21.2.112:0/1673986359 wait complete. 2021-02-10T11:39:53.107 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:39:53.539 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:53.528 7f34aef3b700 1 Processor -- start 2021-02-10T11:39:53.539 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:53.528 7f34aef3b700 1 -- start start 2021-02-10T11:39:53.539 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:53.529 7f34aef3b700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f34a80d1870 0x7f34a81b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:53.539 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:53.529 7f34aef3b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f34a80cff70 0x7f34a8109130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:53.540 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:53.529 7f34aef3b700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34a8109a60 con 0x7f34a81b0ae0 2021-02-10T11:39:53.540 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:53.529 7f34aef3b700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34a8109750 con 0x7f34a80d1870 2021-02-10T11:39:53.540 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:53.529 7f34aef3b700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34a8088070 con 0x7f34a80cff70 2021-02-10T11:39:53.540 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:53.529 7f34acd57700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f34a80d1870 0x7f34a81b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:53.540 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:53.529 7f34a7fff700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f34a81b0ae0 0x7f34a8108940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:53676/0 (socket says 172.21.2.112:53676) 2021-02-10T11:39:53.540 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:53.529 7f34a7fff700 1 -- 172.21.2.112:0/3388451779 learned_addr learned my addr 172.21.2.112:0/3388451779 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:39:53.540 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:53.529 7f34ad558700 1 --2- 172.21.2.112:0/3388451779 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f34a80cff70 0x7f34a8109130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:53.540 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:53.529 7f34acd57700 1 -- 172.21.2.112:0/3388451779 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f34a80cff70 msgr2=0x7f34a8109130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:53.540 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:53.529 7f34acd57700 1 --2- 172.21.2.112:0/3388451779 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f34a80cff70 0x7f34a8109130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:53.540 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:53.529 7f34acd57700 1 -- 172.21.2.112:0/3388451779 >> v1:172.21.2.112:6789/0 conn(0x7f34a81b0ae0 legacy=0x7f34a8108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:53.540 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:53.529 7f34acd57700 1 -- 172.21.2.112:0/3388451779 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34a80d1c70 con 0x7f34a80d1870 2021-02-10T11:39:53.540 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:53.530 7f34acd57700 1 --2- 172.21.2.112:0/3388451779 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f34a80d1870 0x7f34a81b04c0 secure :-1 s=READY pgs=1994 cs=0 l=1 rev1=1 rx=0x7f3490006650 tx=0x7f3490009a80).ready entity=mon.0 client_cookie=450608890a1de000 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:53.540 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:53.530 7f34a77fe700 1 -- 172.21.2.112:0/3388451779 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f349000c0c0 con 0x7f34a80d1870 2021-02-10T11:39:53.541 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:53.530 7f34a77fe700 1 -- 172.21.2.112:0/3388451779 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f349000c340 con 0x7f34a80d1870 2021-02-10T11:39:53.541 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:53.530 7f34a77fe700 1 -- 172.21.2.112:0/3388451779 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f349000c860 con 0x7f34a80d1870 2021-02-10T11:39:53.541 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:53.530 7f34aef3b700 1 -- 172.21.2.112:0/3388451779 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f34a80d1870 msgr2=0x7f34a81b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:53.541 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:53.530 7f34aef3b700 1 --2- 172.21.2.112:0/3388451779 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f34a80d1870 0x7f34a81b04c0 secure :-1 s=READY pgs=1994 cs=0 l=1 rev1=1 rx=0x7f3490006650 tx=0x7f3490009a80).stop 2021-02-10T11:39:53.541 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:53.531 7f34aef3b700 1 -- 172.21.2.112:0/3388451779 shutdown_connections 2021-02-10T11:39:53.541 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:53.531 7f34aef3b700 1 --2- 172.21.2.112:0/3388451779 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f34a80cff70 0x7f34a8109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:53.541 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:53.531 7f34aef3b700 1 --2- 172.21.2.112:0/3388451779 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f34a80d1870 0x7f34a81b04c0 unknown :-1 s=CLOSED pgs=1994 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:53.541 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:53.531 7f34aef3b700 1 -- 172.21.2.112:0/3388451779 >> 172.21.2.112:0/3388451779 conn(0x7f34a80ce420 msgr2=0x7f34a80d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:53.541 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:53.531 7f34aef3b700 1 -- 172.21.2.112:0/3388451779 shutdown_connections 2021-02-10T11:39:53.541 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:53.531 7f34aef3b700 1 -- 172.21.2.112:0/3388451779 wait complete. 2021-02-10T11:39:53.543 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:53.545 7f34aef3b700 1 Processor -- start 2021-02-10T11:39:53.543 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:53.545 7f34aef3b700 1 -- start start 2021-02-10T11:39:53.543 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:53.546 7f34aef3b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f34a8191b10 0x7f34a80ae640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:53.543 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:53.546 7f34aef3b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f34a8191f10 0x7f34a819ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:53.543 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:53.546 7f34aef3b700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f34a81a05f0 0x7f34a8082ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:53.543 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:53.546 7f34aef3b700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34a8192440 con 0x7f34a8191b10 2021-02-10T11:39:53.543 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:53.546 7f34aef3b700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34a81916e0 con 0x7f34a81a05f0 2021-02-10T11:39:53.543 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:53.546 7f34aef3b700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34a8191900 con 0x7f34a8191f10 2021-02-10T11:39:53.543 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:53.546 7f34acd57700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f34a8191b10 0x7f34a80ae640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:53.544 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:53.546 7f34acd57700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f34a8191b10 0x7f34a80ae640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:54222/0 (socket says 172.21.2.112:54222) 2021-02-10T11:39:53.544 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:53.546 7f34acd57700 1 -- 172.21.2.112:0/266166368 learned_addr learned my addr 172.21.2.112:0/266166368 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:39:53.544 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:53.546 7f34acd57700 1 -- 172.21.2.112:0/266166368 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f34a8191f10 msgr2=0x7f34a819ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:53.544 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:53.546 7f34acd57700 1 --2- 172.21.2.112:0/266166368 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f34a8191f10 0x7f34a819ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:53.544 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:53.546 7f34acd57700 1 -- 172.21.2.112:0/266166368 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f34a81a05f0 msgr2=0x7f34a8082ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:53.544 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:53.546 7f34acd57700 1 --2- 172.21.2.112:0/266166368 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f34a81a05f0 0x7f34a8082ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:53.544 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:53.546 7f34acd57700 1 -- 172.21.2.112:0/266166368 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34a80834f0 con 0x7f34a8191b10 2021-02-10T11:39:53.544 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:53.546 7f34acd57700 1 --2- 172.21.2.112:0/266166368 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f34a8191b10 0x7f34a80ae640 secure :-1 s=READY pgs=1995 cs=0 l=1 rev1=1 rx=0x7f3490009630 tx=0x7f349000b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:53.544 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:53.547 7f34a57fa700 1 -- 172.21.2.112:0/266166368 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f34900093a0 con 0x7f34a8191b10 2021-02-10T11:39:53.544 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:53.547 7f34a57fa700 1 -- 172.21.2.112:0/266166368 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f349000bd70 con 0x7f34a8191b10 2021-02-10T11:39:53.544 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:53.547 7f34a57fa700 1 -- 172.21.2.112:0/266166368 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3490005560 con 0x7f34a8191b10 2021-02-10T11:39:53.544 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:53.547 7f34aef3b700 1 -- 172.21.2.112:0/266166368 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34a80834f0 con 0x7f34a8191b10 2021-02-10T11:39:53.544 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:53.547 7f34aef3b700 1 -- 172.21.2.112:0/266166368 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34a80834f0 con 0x7f34a8191b10 2021-02-10T11:39:53.545 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:53.547 7f34a57fa700 1 -- 172.21.2.112:0/266166368 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3490008240 con 0x7f34a8191b10 2021-02-10T11:39:53.545 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:53.548 7f34a57fa700 1 --2- 172.21.2.112:0/266166368 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f348c0186c0 0x7f348c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:53.545 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:53.548 7f34a57fa700 1 -- 172.21.2.112:0/266166368 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f3490032580 con 0x7f34a8191b10 2021-02-10T11:39:53.545 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:53.548 7f34a7fff700 1 --2- 172.21.2.112:0/266166368 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f348c0186c0 0x7f348c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:53.545 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:53.549 7f34a7fff700 1 --2- 172.21.2.112:0/266166368 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f348c0186c0 0x7f348c01abc0 crc :-1 s=READY pgs=930 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:53.572 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:53.579 7f34aef3b700 1 -- 172.21.2.112:0/266166368 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34a8192440 con 0x7f34a8191b10 2021-02-10T11:39:53.575 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:53.581 7f34a57fa700 1 -- 172.21.2.112:0/266166368 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f34900279e0 con 0x7f34a8191b10 2021-02-10T11:39:53.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:53.708 7f34aef3b700 1 -- 172.21.2.112:0/266166368 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f34a8192440 con 0x7f34a8191b10 2021-02-10T11:39:53.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:53.708 7f34a57fa700 1 -- 172.21.2.112:0/266166368 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v100) v1 ==== 100+0+10 (secure 0 0 0) 0x7f34900279e0 con 0x7f34a8191b10 2021-02-10T11:39:53.703 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:39:53.704 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:53.710 7f34aef3b700 1 -- 172.21.2.112:0/266166368 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f348c0186c0 msgr2=0x7f348c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:53.704 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:53.710 7f34aef3b700 1 --2- 172.21.2.112:0/266166368 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f348c0186c0 0x7f348c01abc0 crc :-1 s=READY pgs=930 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:53.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:53.710 7f34aef3b700 1 -- 172.21.2.112:0/266166368 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f34a8191b10 msgr2=0x7f34a80ae640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:53.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:53.710 7f34aef3b700 1 --2- 172.21.2.112:0/266166368 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f34a8191b10 0x7f34a80ae640 secure :-1 s=READY pgs=1995 cs=0 l=1 rev1=1 rx=0x7f3490009630 tx=0x7f349000b980).stop 2021-02-10T11:39:53.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:53.711 7f34aef3b700 1 -- 172.21.2.112:0/266166368 shutdown_connections 2021-02-10T11:39:53.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:53.711 7f34aef3b700 1 --2- 172.21.2.112:0/266166368 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f348c0186c0 0x7f348c01abc0 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:53.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:53.711 7f34aef3b700 1 --2- 172.21.2.112:0/266166368 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f34a81a05f0 0x7f34a8082ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:53.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:53.711 7f34aef3b700 1 --2- 172.21.2.112:0/266166368 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f34a8191f10 0x7f34a819ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:53.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:53.711 7f34aef3b700 1 --2- 172.21.2.112:0/266166368 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f34a8191b10 0x7f34a80ae640 unknown :-1 s=CLOSED pgs=1995 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:53.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:53.711 7f34aef3b700 1 -- 172.21.2.112:0/266166368 >> 172.21.2.112:0/266166368 conn(0x7f34a80a02d0 msgr2=0x7f34a809f200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:53.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:53.711 7f34aef3b700 1 -- 172.21.2.112:0/266166368 shutdown_connections 2021-02-10T11:39:53.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:53.711 7f34aef3b700 1 -- 172.21.2.112:0/266166368 wait complete. 2021-02-10T11:39:53.713 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:39:53.713 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:39:54.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:54.131 7f987098b700 1 Processor -- start 2021-02-10T11:39:54.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:54.131 7f987098b700 1 -- start start 2021-02-10T11:39:54.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:54.131 7f987098b700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f986c0d1870 0x7f986c1b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:54.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:54.131 7f987098b700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f986c1b0a10 0x7f986c0cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:54.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:54.131 7f987098b700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f986c0f0b10 con 0x7f986c0d1c70 2021-02-10T11:39:54.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:54.131 7f987098b700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f986c0e5ed0 con 0x7f986c1b0a10 2021-02-10T11:39:54.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:54.131 7f987098b700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f986c0f0650 con 0x7f986c0d1870 2021-02-10T11:39:54.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:54.132 7f9869e1b700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f986c1b0a10 0x7f986c0cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:54.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:54.132 7f986ae1d700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f986c0d1c70 0x7f986c0cf950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:53690/0 (socket says 172.21.2.112:53690) 2021-02-10T11:39:54.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:54.132 7f986ae1d700 1 -- 172.21.2.112:0/3261934590 learned_addr learned my addr 172.21.2.112:0/3261934590 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:39:54.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:54.132 7f9869e1b700 1 -- 172.21.2.112:0/3261934590 >> v2:172.21.2.120:3300/0 conn(0x7f986c0d1870 msgr2=0x7f986c1b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:54.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:54.132 7f9869e1b700 1 --2- 172.21.2.112:0/3261934590 >> v2:172.21.2.120:3300/0 conn(0x7f986c0d1870 0x7f986c1b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:54.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:54.132 7f9869e1b700 1 -- 172.21.2.112:0/3261934590 >> v1:172.21.2.112:6789/0 conn(0x7f986c0d1c70 legacy=0x7f986c0cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:54.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:54.132 7f9869e1b700 1 -- 172.21.2.112:0/3261934590 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f986c0f0840 con 0x7f986c1b0a10 2021-02-10T11:39:54.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:54.132 7f9869e1b700 1 --2- 172.21.2.112:0/3261934590 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f986c1b0a10 0x7f986c0cf330 secure :-1 s=READY pgs=1997 cs=0 l=1 rev1=1 rx=0x7f985c006650 tx=0x7f985c009a80).ready entity=mon.0 client_cookie=ec1bd080fbc00c9d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:54.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:54.132 7f986961a700 1 -- 172.21.2.112:0/3261934590 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f985c00c0c0 con 0x7f986c1b0a10 2021-02-10T11:39:54.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:54.132 7f986961a700 1 -- 172.21.2.112:0/3261934590 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f985c00c340 con 0x7f986c1b0a10 2021-02-10T11:39:54.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:54.133 7f986961a700 1 -- 172.21.2.112:0/3261934590 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f985c00c7a0 con 0x7f986c1b0a10 2021-02-10T11:39:54.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:54.133 7f987098b700 1 -- 172.21.2.112:0/3261934590 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f986c1b0a10 msgr2=0x7f986c0cf330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:54.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:54.133 7f987098b700 1 --2- 172.21.2.112:0/3261934590 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f986c1b0a10 0x7f986c0cf330 secure :-1 s=READY pgs=1997 cs=0 l=1 rev1=1 rx=0x7f985c006650 tx=0x7f985c009a80).stop 2021-02-10T11:39:54.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:54.133 7f987098b700 1 -- 172.21.2.112:0/3261934590 shutdown_connections 2021-02-10T11:39:54.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:54.133 7f987098b700 1 --2- 172.21.2.112:0/3261934590 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f986c1b0a10 0x7f986c0cf330 unknown :-1 s=CLOSED pgs=1997 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:54.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:54.133 7f987098b700 1 --2- 172.21.2.112:0/3261934590 >> v2:172.21.2.120:3300/0 conn(0x7f986c0d1870 0x7f986c1b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:54.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:54.133 7f987098b700 1 -- 172.21.2.112:0/3261934590 >> 172.21.2.112:0/3261934590 conn(0x7f986c0ce450 msgr2=0x7f986c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:54.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:54.133 7f987098b700 1 -- 172.21.2.112:0/3261934590 shutdown_connections 2021-02-10T11:39:54.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:54.134 7f987098b700 1 -- 172.21.2.112:0/3261934590 wait complete. 2021-02-10T11:39:54.145 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:54.148 7f987098b700 1 Processor -- start 2021-02-10T11:39:54.145 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:54.148 7f987098b700 1 -- start start 2021-02-10T11:39:54.145 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:54.148 7f987098b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f986c0b4f30 0x7f986c18b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:54.145 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:54.148 7f987098b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f986c093f40 0x7f986c094340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:54.146 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:54.148 7f987098b700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f986c094960 0x7f986c187e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:54.146 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:54.148 7f987098b700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f986c094e90 con 0x7f986c0b4f30 2021-02-10T11:39:54.146 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:54.148 7f987098b700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f986c18bff0 con 0x7f986c094960 2021-02-10T11:39:54.146 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:54.148 7f987098b700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f986c08f780 con 0x7f986c093f40 2021-02-10T11:39:54.146 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:54.148 7f986a61c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f986c0b4f30 0x7f986c18b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:54.146 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:54.148 7f986a61c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f986c0b4f30 0x7f986c18b9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:54236/0 (socket says 172.21.2.112:54236) 2021-02-10T11:39:54.146 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:54.149 7f986a61c700 1 -- 172.21.2.112:0/3364755456 learned_addr learned my addr 172.21.2.112:0/3364755456 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:39:54.146 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:54.149 7f9869e1b700 1 --2- 172.21.2.112:0/3364755456 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f986c093f40 0x7f986c094340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:54.146 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:54.149 7f986ae1d700 1 --2- 172.21.2.112:0/3364755456 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f986c094960 0x7f986c187e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:54.146 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:54.149 7f986a61c700 1 -- 172.21.2.112:0/3364755456 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f986c093f40 msgr2=0x7f986c094340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:54.146 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:54.149 7f986a61c700 1 --2- 172.21.2.112:0/3364755456 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f986c093f40 0x7f986c094340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:54.147 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:54.149 7f986a61c700 1 -- 172.21.2.112:0/3364755456 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f986c094960 msgr2=0x7f986c187e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:54.147 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:54.149 7f986a61c700 1 --2- 172.21.2.112:0/3364755456 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f986c094960 0x7f986c187e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:54.147 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:54.149 7f986a61c700 1 -- 172.21.2.112:0/3364755456 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f986c188470 con 0x7f986c0b4f30 2021-02-10T11:39:54.147 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:54.149 7f986ae1d700 1 --2- 172.21.2.112:0/3364755456 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f986c094960 0x7f986c187e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:39:54.147 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:54.149 7f986a61c700 1 --2- 172.21.2.112:0/3364755456 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f986c0b4f30 0x7f986c18b9d0 secure :-1 s=READY pgs=1998 cs=0 l=1 rev1=1 rx=0x7f9854009980 tx=0x7f985400bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:54.147 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:54.149 7f9862ffd700 1 -- 172.21.2.112:0/3364755456 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f98540066d0 con 0x7f986c0b4f30 2021-02-10T11:39:54.147 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:54.149 7f9862ffd700 1 -- 172.21.2.112:0/3364755456 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f985400c2f0 con 0x7f986c0b4f30 2021-02-10T11:39:54.147 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:54.149 7f9862ffd700 1 -- 172.21.2.112:0/3364755456 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f985400c810 con 0x7f986c0b4f30 2021-02-10T11:39:54.147 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:54.150 7f987098b700 1 -- 172.21.2.112:0/3364755456 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f986c188470 con 0x7f986c0b4f30 2021-02-10T11:39:54.147 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:54.150 7f987098b700 1 -- 172.21.2.112:0/3364755456 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f986c188470 con 0x7f986c0b4f30 2021-02-10T11:39:54.147 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:54.150 7f9862ffd700 1 -- 172.21.2.112:0/3364755456 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9854001010 con 0x7f986c0b4f30 2021-02-10T11:39:54.147 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:54.150 7f9862ffd700 1 --2- 172.21.2.112:0/3364755456 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f98500186c0 0x7f985001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:54.148 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:54.150 7f9862ffd700 1 -- 172.21.2.112:0/3364755456 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f98540328d0 con 0x7f986c0b4f30 2021-02-10T11:39:54.148 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:54.151 7f9869e1b700 1 --2- 172.21.2.112:0/3364755456 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f98500186c0 0x7f985001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:54.148 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:54.151 7f9869e1b700 1 --2- 172.21.2.112:0/3364755456 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f98500186c0 0x7f985001abc0 crc :-1 s=READY pgs=931 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:54.175 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:54.181 7f987098b700 1 -- 172.21.2.112:0/3364755456 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f986c0ce450 con 0x7f986c0b4f30 2021-02-10T11:39:54.178 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:54.184 7f9862ffd700 1 -- 172.21.2.112:0/3364755456 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f985402cf20 con 0x7f986c0b4f30 2021-02-10T11:39:54.306 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:54.312 7f987098b700 1 -- 172.21.2.112:0/3364755456 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f986c0ce450 con 0x7f986c0b4f30 2021-02-10T11:39:54.307 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:54.313 7f9862ffd700 1 -- 172.21.2.112:0/3364755456 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+2104 (secure 0 0 0) 0x7f985402ccc0 con 0x7f986c0b4f30 2021-02-10T11:39:54.307 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:39:54.308 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":80,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10011,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/3938237927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":3938237927},{"type":"v1","addr":"172.21.2.112:6833","nonce":3938237927}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":76}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:39:44.311361","modified":"2021-02-10 11:39:46.320259","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7633},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7633":{"gid":7633,"name":"b","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.2.120:6835/1537197833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":1537197833},{"type":"v1","addr":"172.21.2.120:6835","nonce":1537197833}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":10}]} 2021-02-10T11:39:54.308 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 80 2021-02-10T11:39:54.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:54.314 7f987098b700 1 -- 172.21.2.112:0/3364755456 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f98500186c0 msgr2=0x7f985001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:54.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:54.315 7f987098b700 1 --2- 172.21.2.112:0/3364755456 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f98500186c0 0x7f985001abc0 crc :-1 s=READY pgs=931 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:54.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:54.315 7f987098b700 1 -- 172.21.2.112:0/3364755456 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f986c0b4f30 msgr2=0x7f986c18b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:54.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:54.315 7f987098b700 1 --2- 172.21.2.112:0/3364755456 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f986c0b4f30 0x7f986c18b9d0 secure :-1 s=READY pgs=1998 cs=0 l=1 rev1=1 rx=0x7f9854009980 tx=0x7f985400bcd0).stop 2021-02-10T11:39:54.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:54.315 7f987098b700 1 -- 172.21.2.112:0/3364755456 shutdown_connections 2021-02-10T11:39:54.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:54.315 7f987098b700 1 --2- 172.21.2.112:0/3364755456 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f98500186c0 0x7f985001abc0 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:54.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:54.315 7f987098b700 1 --2- 172.21.2.112:0/3364755456 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f986c094960 0x7f986c187e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:54.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:54.315 7f987098b700 1 --2- 172.21.2.112:0/3364755456 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f986c093f40 0x7f986c094340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:54.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:54.315 7f987098b700 1 --2- 172.21.2.112:0/3364755456 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f986c0b4f30 0x7f986c18b9d0 unknown :-1 s=CLOSED pgs=1998 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:54.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:54.315 7f987098b700 1 -- 172.21.2.112:0/3364755456 >> 172.21.2.112:0/3364755456 conn(0x7f986c19f8f0 msgr2=0x7f986c1a5f80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:54.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:54.315 7f987098b700 1 -- 172.21.2.112:0/3364755456 shutdown_connections 2021-02-10T11:39:54.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:54.315 7f987098b700 1 -- 172.21.2.112:0/3364755456 wait complete. 2021-02-10T11:39:54.520 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 80, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:39:44.311361', 'modified': '2021-02-10 11:39:46.320259', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7633}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7633': {'gid': 7633, 'name': 'b', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.120:6835/1537197833', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.120:6834', 'nonce': 1537197833}, {'type': 'v1', 'addr': '172.21.2.120:6835', 'nonce': 1537197833}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:39:54.520 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:39:54.520 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:39:54.918 INFO:teuthology.orchestra.run.gibba020.stdout:{ 2021-02-10T11:39:54.918 INFO:teuthology.orchestra.run.gibba020.stdout: "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", 2021-02-10T11:39:54.918 INFO:teuthology.orchestra.run.gibba020.stdout: "whoami": 0, 2021-02-10T11:39:54.918 INFO:teuthology.orchestra.run.gibba020.stdout: "id": 7633, 2021-02-10T11:39:54.918 INFO:teuthology.orchestra.run.gibba020.stdout: "want_state": "up:active", 2021-02-10T11:39:54.918 INFO:teuthology.orchestra.run.gibba020.stdout: "state": "up:active", 2021-02-10T11:39:54.918 INFO:teuthology.orchestra.run.gibba020.stdout: "rank_uptime": 10.610737196000001, 2021-02-10T11:39:54.918 INFO:teuthology.orchestra.run.gibba020.stdout: "mdsmap_epoch": 80, 2021-02-10T11:39:54.919 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch": 100, 2021-02-10T11:39:54.919 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch_barrier": 100, 2021-02-10T11:39:54.919 INFO:teuthology.orchestra.run.gibba020.stdout: "uptime": 16.251583813 2021-02-10T11:39:54.919 INFO:teuthology.orchestra.run.gibba020.stdout:} 2021-02-10T11:39:54.922 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", "whoami": 0, "id": 7633, "want_state": "up:active", "state": "up:active", "rank_uptime": 10.610737196000001, "mdsmap_epoch": 80, "osdmap_epoch": 100, "osdmap_epoch_barrier": 100, "uptime": 16.251583813 } 2021-02-10T11:39:54.922 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:39:54.922 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2021-02-10T11:39:54.922 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@gibba012.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2021-02-10T11:39:54.923 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:39:54.923 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:39:55.258 DEBUG:teuthology.orchestra.run.gibba012:> sudo modprobe fuse 2021-02-10T11:39:55.602 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:39:55.602 DEBUG:teuthology.orchestra.run.gibba012:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:39:55.945 INFO:teuthology.orchestra.run.gibba012.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:39:55.946 INFO:teuthology.orchestra.run.gibba012.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:39:55.946 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:39:55.947 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:39:55.947 DEBUG:teuthology.orchestra.run.gibba012:> ls /sys/fs/fuse/connections 2021-02-10T11:39:56.281 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2021-02-10T11:39:56.282 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:39:56.284 DEBUG:teuthology.orchestra.run.gibba012:> sudo modprobe fuse 2021-02-10T11:39:56.629 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:39:56.629 DEBUG:teuthology.orchestra.run.gibba012:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:39:56.695 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba012.stderr:ceph-fuse[153931]: starting ceph client 2021-02-10T11:39:56.696 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba012.stderr:2021-02-10 11:39:56.701 7fac66d17f80 -1 init, newargv = 0x5574a8e831a0 newargc=9 2021-02-10T11:39:56.700 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba012.stderr:ceph-fuse[153931]: starting fuse 2021-02-10T11:39:56.971 INFO:teuthology.orchestra.run.gibba012.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:39:56.971 INFO:teuthology.orchestra.run.gibba012.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:39:56.971 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:39:56.972 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:39:56.972 DEBUG:teuthology.orchestra.run.gibba012:> ls /sys/fs/fuse/connections 2021-02-10T11:39:57.306 INFO:teuthology.orchestra.run.gibba012.stdout:42 2021-02-10T11:39:57.307 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2021-02-10T11:39:57.307 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:39:57.307 DEBUG:teuthology.orchestra.run.gibba012:> sudo python3 -c ' 2021-02-10T11:39:57.307 DEBUG:teuthology.orchestra.run.gibba012:> import glob 2021-02-10T11:39:57.307 DEBUG:teuthology.orchestra.run.gibba012:> import re 2021-02-10T11:39:57.307 DEBUG:teuthology.orchestra.run.gibba012:> import os 2021-02-10T11:39:57.307 DEBUG:teuthology.orchestra.run.gibba012:> import subprocess 2021-02-10T11:39:57.307 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:39:57.308 DEBUG:teuthology.orchestra.run.gibba012:> def find_socket(client_name): 2021-02-10T11:39:57.308 DEBUG:teuthology.orchestra.run.gibba012:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2021-02-10T11:39:57.308 DEBUG:teuthology.orchestra.run.gibba012:> files = glob.glob(asok_path) 2021-02-10T11:39:57.308 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:39:57.308 DEBUG:teuthology.orchestra.run.gibba012:> # Given a non-glob path, it better be there 2021-02-10T11:39:57.308 DEBUG:teuthology.orchestra.run.gibba012:> if "*" not in asok_path: 2021-02-10T11:39:57.308 DEBUG:teuthology.orchestra.run.gibba012:> assert(len(files) == 1) 2021-02-10T11:39:57.308 DEBUG:teuthology.orchestra.run.gibba012:> return files[0] 2021-02-10T11:39:57.308 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:39:57.308 DEBUG:teuthology.orchestra.run.gibba012:> for f in files: 2021-02-10T11:39:57.308 DEBUG:teuthology.orchestra.run.gibba012:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:39:57.308 DEBUG:teuthology.orchestra.run.gibba012:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:39:57.308 DEBUG:teuthology.orchestra.run.gibba012:> return f 2021-02-10T11:39:57.308 DEBUG:teuthology.orchestra.run.gibba012:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:39:57.309 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:39:57.309 DEBUG:teuthology.orchestra.run.gibba012:> print(find_socket("client.0")) 2021-02-10T11:39:57.309 DEBUG:teuthology.orchestra.run.gibba012:> ' 2021-02-10T11:39:57.671 INFO:teuthology.orchestra.run.gibba012.stdout:/var/run/ceph/ceph-client.0.153931.asok 2021-02-10T11:39:57.674 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.153931.asok 2021-02-10T11:39:57.674 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:39:57.674 DEBUG:teuthology.orchestra.run.gibba012:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.153931.asok status 2021-02-10T11:39:58.065 INFO:teuthology.orchestra.run.gibba012.stdout:{ 2021-02-10T11:39:58.065 INFO:teuthology.orchestra.run.gibba012.stdout: "metadata": { 2021-02-10T11:39:58.065 INFO:teuthology.orchestra.run.gibba012.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:39:58.065 INFO:teuthology.orchestra.run.gibba012.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:39:58.065 INFO:teuthology.orchestra.run.gibba012.stdout: "entity_id": "0", 2021-02-10T11:39:58.065 INFO:teuthology.orchestra.run.gibba012.stdout: "hostname": "gibba012", 2021-02-10T11:39:58.065 INFO:teuthology.orchestra.run.gibba012.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2021-02-10T11:39:58.066 INFO:teuthology.orchestra.run.gibba012.stdout: "pid": "153931", 2021-02-10T11:39:58.066 INFO:teuthology.orchestra.run.gibba012.stdout: "root": "/" 2021-02-10T11:39:58.066 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:39:58.066 INFO:teuthology.orchestra.run.gibba012.stdout: "dentry_count": 0, 2021-02-10T11:39:58.066 INFO:teuthology.orchestra.run.gibba012.stdout: "dentry_pinned_count": 0, 2021-02-10T11:39:58.066 INFO:teuthology.orchestra.run.gibba012.stdout: "id": 10170, 2021-02-10T11:39:58.066 INFO:teuthology.orchestra.run.gibba012.stdout: "inst": { 2021-02-10T11:39:58.066 INFO:teuthology.orchestra.run.gibba012.stdout: "name": { 2021-02-10T11:39:58.066 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "client", 2021-02-10T11:39:58.066 INFO:teuthology.orchestra.run.gibba012.stdout: "num": 10170 2021-02-10T11:39:58.066 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:39:58.066 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": { 2021-02-10T11:39:58.067 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:39:58.067 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:0", 2021-02-10T11:39:58.067 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 2368255529 2021-02-10T11:39:58.067 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:39:58.067 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:39:58.067 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": { 2021-02-10T11:39:58.067 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:39:58.067 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:0", 2021-02-10T11:39:58.067 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 2368255529 2021-02-10T11:39:58.067 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:39:58.067 INFO:teuthology.orchestra.run.gibba012.stdout: "inst_str": "client.10170 172.21.2.112:0/2368255529", 2021-02-10T11:39:58.067 INFO:teuthology.orchestra.run.gibba012.stdout: "addr_str": "172.21.2.112:0/2368255529", 2021-02-10T11:39:58.067 INFO:teuthology.orchestra.run.gibba012.stdout: "inode_count": 1, 2021-02-10T11:39:58.068 INFO:teuthology.orchestra.run.gibba012.stdout: "mds_epoch": 80, 2021-02-10T11:39:58.068 INFO:teuthology.orchestra.run.gibba012.stdout: "osd_epoch": 100, 2021-02-10T11:39:58.068 INFO:teuthology.orchestra.run.gibba012.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:39:58.068 INFO:teuthology.orchestra.run.gibba012.stdout: "blacklisted": false 2021-02-10T11:39:58.068 INFO:teuthology.orchestra.run.gibba012.stdout:} 2021-02-10T11:39:58.069 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:39:58.069 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:39:58.069 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:39:58.404 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:39:58.404 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:39:58.404 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:39:58.405 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:39:58.750 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:39:59.177 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.167 7f41d9a95700 1 Processor -- start 2021-02-10T11:39:59.177 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.167 7f41d9a95700 1 -- start start 2021-02-10T11:39:59.178 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.167 7f41d9a95700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f41d40d1870 0x7f41d41b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:59.178 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.167 7f41d9a95700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f41d41b0a10 0x7f41d40cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:59.178 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.167 7f41d9a95700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41d40f0b10 con 0x7f41d40d1c70 2021-02-10T11:39:59.178 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.167 7f41d9a95700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41d40fe230 con 0x7f41d41b0a10 2021-02-10T11:39:59.178 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.167 7f41d9a95700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41d40e5ed0 con 0x7f41d40d1870 2021-02-10T11:39:59.178 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.168 7f41d2ffd700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f41d41b0a10 0x7f41d40cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:59.178 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.168 7f41d2ffd700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f41d41b0a10 0x7f41d40cf330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:54260/0 (socket says 172.21.2.112:54260) 2021-02-10T11:39:59.178 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.168 7f41d2ffd700 1 -- 172.21.2.112:0/3200010883 learned_addr learned my addr 172.21.2.112:0/3200010883 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:39:59.178 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.168 7f41d2ffd700 1 -- 172.21.2.112:0/3200010883 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f41d40d1870 msgr2=0x7f41d41b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:59.178 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.168 7f41d2ffd700 1 --2- 172.21.2.112:0/3200010883 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f41d40d1870 0x7f41d41b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:59.178 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.168 7f41d2ffd700 1 -- 172.21.2.112:0/3200010883 >> v1:172.21.2.112:6789/0 conn(0x7f41d40d1c70 legacy=0x7f41d40cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:59.178 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.168 7f41d2ffd700 1 -- 172.21.2.112:0/3200010883 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41d40f0650 con 0x7f41d41b0a10 2021-02-10T11:39:59.179 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.168 7f41d2ffd700 1 --2- 172.21.2.112:0/3200010883 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f41d41b0a10 0x7f41d40cf330 secure :-1 s=READY pgs=2003 cs=0 l=1 rev1=1 rx=0x7f41c4006700 tx=0x7f41c4009b00).ready entity=mon.0 client_cookie=6320df5b79ce10e0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:59.179 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.168 7f41d27fc700 1 -- 172.21.2.112:0/3200010883 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f41c400c0c0 con 0x7f41d41b0a10 2021-02-10T11:39:59.179 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.168 7f41d27fc700 1 -- 172.21.2.112:0/3200010883 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41c400c400 con 0x7f41d41b0a10 2021-02-10T11:39:59.179 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.168 7f41d27fc700 1 -- 172.21.2.112:0/3200010883 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f41c400c920 con 0x7f41d41b0a10 2021-02-10T11:39:59.179 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.168 7f41d9a95700 1 -- 172.21.2.112:0/3200010883 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f41d41b0a10 msgr2=0x7f41d40cf330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:59.179 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.168 7f41d9a95700 1 --2- 172.21.2.112:0/3200010883 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f41d41b0a10 0x7f41d40cf330 secure :-1 s=READY pgs=2003 cs=0 l=1 rev1=1 rx=0x7f41c4006700 tx=0x7f41c4009b00).stop 2021-02-10T11:39:59.179 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.169 7f41d9a95700 1 -- 172.21.2.112:0/3200010883 shutdown_connections 2021-02-10T11:39:59.179 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.169 7f41d9a95700 1 --2- 172.21.2.112:0/3200010883 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f41d41b0a10 0x7f41d40cf330 unknown :-1 s=CLOSED pgs=2003 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:59.179 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.169 7f41d9a95700 1 --2- 172.21.2.112:0/3200010883 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f41d40d1870 0x7f41d41b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:59.179 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.169 7f41d9a95700 1 -- 172.21.2.112:0/3200010883 >> 172.21.2.112:0/3200010883 conn(0x7f41d40ce450 msgr2=0x7f41d40d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:59.179 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.169 7f41d9a95700 1 -- 172.21.2.112:0/3200010883 shutdown_connections 2021-02-10T11:39:59.179 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.169 7f41d9a95700 1 -- 172.21.2.112:0/3200010883 wait complete. 2021-02-10T11:39:59.181 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.183 7f41d9a95700 1 Processor -- start 2021-02-10T11:39:59.181 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.184 7f41d9a95700 1 -- start start 2021-02-10T11:39:59.181 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.184 7f41d9a95700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f41d40b4f30 0x7f41d418b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:59.181 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.184 7f41d9a95700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f41d4093f40 0x7f41d4094340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:59.181 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.184 7f41d9a95700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f41d4094960 0x7f41d4187e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:59.181 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.184 7f41d9a95700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41d4094e90 con 0x7f41d4093f40 2021-02-10T11:39:59.181 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.184 7f41d9a95700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41d418bff0 con 0x7f41d4094960 2021-02-10T11:39:59.181 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.184 7f41d9a95700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41d408f780 con 0x7f41d40b4f30 2021-02-10T11:39:59.181 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.184 7f41d2ffd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f41d4093f40 0x7f41d4094340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:59.181 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.184 7f41d2ffd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f41d4093f40 0x7f41d4094340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:54266/0 (socket says 172.21.2.112:54266) 2021-02-10T11:39:59.181 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.184 7f41d2ffd700 1 -- 172.21.2.112:0/2693817419 learned_addr learned my addr 172.21.2.112:0/2693817419 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:39:59.181 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.184 7f41d2ffd700 1 -- 172.21.2.112:0/2693817419 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f41d40b4f30 msgr2=0x7f41d418b9d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:39:59.182 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.184 7f41d2ffd700 1 --2- 172.21.2.112:0/2693817419 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f41d40b4f30 0x7f41d418b9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:59.182 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.184 7f41d2ffd700 1 -- 172.21.2.112:0/2693817419 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f41d4094960 msgr2=0x7f41d4187e50 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:39:59.182 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.184 7f41d2ffd700 1 --2- 172.21.2.112:0/2693817419 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f41d4094960 0x7f41d4187e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:59.182 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.184 7f41d2ffd700 1 -- 172.21.2.112:0/2693817419 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41d4188470 con 0x7f41d4093f40 2021-02-10T11:39:59.182 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.185 7f41d2ffd700 1 --2- 172.21.2.112:0/2693817419 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f41d4093f40 0x7f41d4094340 secure :-1 s=READY pgs=2004 cs=0 l=1 rev1=1 rx=0x7f41c400b660 tx=0x7f41c400ba00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:59.182 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.185 7f41cbfff700 1 -- 172.21.2.112:0/2693817419 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f41c4009440 con 0x7f41d4093f40 2021-02-10T11:39:59.182 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.185 7f41cbfff700 1 -- 172.21.2.112:0/2693817419 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41c4000c90 con 0x7f41d4093f40 2021-02-10T11:39:59.182 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.185 7f41cbfff700 1 -- 172.21.2.112:0/2693817419 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f41c4002030 con 0x7f41d4093f40 2021-02-10T11:39:59.182 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.185 7f41d9a95700 1 -- 172.21.2.112:0/2693817419 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41d4188470 con 0x7f41d4093f40 2021-02-10T11:39:59.182 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.185 7f41d9a95700 1 -- 172.21.2.112:0/2693817419 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41d4188a00 con 0x7f41d4093f40 2021-02-10T11:39:59.182 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.185 7f41cbfff700 1 -- 172.21.2.112:0/2693817419 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f41c40093e0 con 0x7f41d4093f40 2021-02-10T11:39:59.182 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.185 7f41cbfff700 1 --2- 172.21.2.112:0/2693817419 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f41b80186c0 0x7f41b801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:59.182 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.185 7f41cbfff700 1 -- 172.21.2.112:0/2693817419 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f41c4032550 con 0x7f41d4093f40 2021-02-10T11:39:59.183 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.187 7f41d37fe700 1 --2- 172.21.2.112:0/2693817419 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f41b80186c0 0x7f41b801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:59.183 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.187 7f41d37fe700 1 --2- 172.21.2.112:0/2693817419 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f41b80186c0 0x7f41b801abc0 crc :-1 s=READY pgs=932 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:59.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.217 7f41d9a95700 1 -- 172.21.2.112:0/2693817419 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41d40ce450 con 0x7f41d4093f40 2021-02-10T11:39:59.214 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.220 7f41cbfff700 1 -- 172.21.2.112:0/2693817419 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f41c4030d00 con 0x7f41d4093f40 2021-02-10T11:39:59.339 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.344 7f41d9a95700 1 -- 172.21.2.112:0/2693817419 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f41d40ce450 con 0x7f41d4093f40 2021-02-10T11:39:59.339 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.345 7f41cbfff700 1 -- 172.21.2.112:0/2693817419 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v100) v1 ==== 74+0+13998 (secure 0 0 0) 0x7f41c40093e0 con 0x7f41d4093f40 2021-02-10T11:39:59.340 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:39:59.340 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":100,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:39:45.313226","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:39:42.230237","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"99","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":21,"pool_name":"cephfs_data","create_time":"2021-02-10 11:39:43.354655","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:39:59.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.347 7f41d9a95700 1 -- 172.21.2.112:0/2693817419 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f41b80186c0 msgr2=0x7f41b801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:59.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.347 7f41d9a95700 1 --2- 172.21.2.112:0/2693817419 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f41b80186c0 0x7f41b801abc0 crc :-1 s=READY pgs=932 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:59.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.347 7f41d9a95700 1 -- 172.21.2.112:0/2693817419 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f41d4093f40 msgr2=0x7f41d4094340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:59.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.347 7f41d9a95700 1 --2- 172.21.2.112:0/2693817419 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f41d4093f40 0x7f41d4094340 secure :-1 s=READY pgs=2004 cs=0 l=1 rev1=1 rx=0x7f41c400b660 tx=0x7f41c400ba00).stop 2021-02-10T11:39:59.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.347 7f41d9a95700 1 -- 172.21.2.112:0/2693817419 shutdown_connections 2021-02-10T11:39:59.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.347 7f41d9a95700 1 --2- 172.21.2.112:0/2693817419 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f41b80186c0 0x7f41b801abc0 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:59.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.347 7f41d9a95700 1 --2- 172.21.2.112:0/2693817419 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f41d4094960 0x7f41d4187e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:59.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.347 7f41d9a95700 1 --2- 172.21.2.112:0/2693817419 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f41d4093f40 0x7f41d4094340 unknown :-1 s=CLOSED pgs=2004 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:59.342 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.347 7f41d9a95700 1 --2- 172.21.2.112:0/2693817419 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f41d40b4f30 0x7f41d418b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:59.342 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.347 7f41d9a95700 1 -- 172.21.2.112:0/2693817419 >> 172.21.2.112:0/2693817419 conn(0x7f41d419f8f0 msgr2=0x7f41d41a5f80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:59.342 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.347 7f41d9a95700 1 -- 172.21.2.112:0/2693817419 shutdown_connections 2021-02-10T11:39:59.342 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.347 7f41d9a95700 1 -- 172.21.2.112:0/2693817419 wait complete. 2021-02-10T11:39:59.350 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:13:57.508758', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 20, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:39:42.230237', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '99', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 21, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:39:43.354655', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '100', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:39:59.350 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:39:59.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.771 7f2e37693700 1 Processor -- start 2021-02-10T11:39:59.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.771 7f2e37693700 1 -- start start 2021-02-10T11:39:59.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.772 7f2e37693700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f2e3010aff0 0x7f2e300cf750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:59.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.772 7f2e37693700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e300fe1c0 con 0x7f2e300d1870 2021-02-10T11:39:59.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.772 7f2e37693700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e301b0cb0 con 0x7f2e300d1c70 2021-02-10T11:39:59.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.772 7f2e37693700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e30109a60 con 0x7f2e3010aff0 2021-02-10T11:39:59.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.772 7f2e354af700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f2e300d1870 0x7f2e301b04c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:53736/0 (socket says 172.21.2.112:53736) 2021-02-10T11:39:59.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.772 7f2e354af700 1 -- 172.21.2.112:0/1067468268 learned_addr learned my addr 172.21.2.112:0/1067468268 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:39:59.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.772 7f2e23fff700 1 -- 172.21.2.112:0/1067468268 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 121031561 0 0) 0x7f2e2c001180 con 0x7f2e300d1870 2021-02-10T11:39:59.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.772 7f2e23fff700 1 -- 172.21.2.112:0/1067468268 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e1c002fe0 con 0x7f2e300d1870 2021-02-10T11:39:59.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.772 7f2e35cb0700 1 --2- 172.21.2.112:0/1067468268 >> v2:172.21.2.120:3300/0 conn(0x7f2e3010aff0 0x7f2e300cf750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:59.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.773 7f2e23fff700 1 -- 172.21.2.112:0/1067468268 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3653261794 0 0) 0x7f2e2c001180 con 0x7f2e300d1870 2021-02-10T11:39:59.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.773 7f2e35cb0700 1 -- 172.21.2.112:0/1067468268 >> v1:172.21.2.120:6789/0 conn(0x7f2e300d1c70 legacy=0x7f2e300cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:59.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.773 7f2e35cb0700 1 -- 172.21.2.112:0/1067468268 >> v1:172.21.2.112:6789/0 conn(0x7f2e300d1870 legacy=0x7f2e301b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:59.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.773 7f2e35cb0700 1 -- 172.21.2.112:0/1067468268 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e3010b3f0 con 0x7f2e3010aff0 2021-02-10T11:39:59.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.773 7f2e35cb0700 1 --2- 172.21.2.112:0/1067468268 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2e3010aff0 0x7f2e300cf750 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 rx=0x7f2e28006e50 tx=0x7f2e28009e70).ready entity=mon.1 client_cookie=c2ac52cb743127da server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:59.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.773 7f2e23fff700 1 -- 172.21.2.112:0/1067468268 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2e2800b510 con 0x7f2e3010aff0 2021-02-10T11:39:59.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.773 7f2e23fff700 1 -- 172.21.2.112:0/1067468268 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e28006790 con 0x7f2e3010aff0 2021-02-10T11:39:59.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.773 7f2e23fff700 1 -- 172.21.2.112:0/1067468268 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2e2800b940 con 0x7f2e3010aff0 2021-02-10T11:39:59.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.773 7f2e37693700 1 -- 172.21.2.112:0/1067468268 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2e3010aff0 msgr2=0x7f2e300cf750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:59.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.773 7f2e37693700 1 --2- 172.21.2.112:0/1067468268 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2e3010aff0 0x7f2e300cf750 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 rx=0x7f2e28006e50 tx=0x7f2e28009e70).stop 2021-02-10T11:39:59.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.774 7f2e37693700 1 -- 172.21.2.112:0/1067468268 shutdown_connections 2021-02-10T11:39:59.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.774 7f2e37693700 1 --2- 172.21.2.112:0/1067468268 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2e3010aff0 0x7f2e300cf750 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:59.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.774 7f2e37693700 1 -- 172.21.2.112:0/1067468268 >> 172.21.2.112:0/1067468268 conn(0x7f2e300ce420 msgr2=0x7f2e300d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:59.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.774 7f2e37693700 1 -- 172.21.2.112:0/1067468268 shutdown_connections 2021-02-10T11:39:59.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.774 7f2e37693700 1 -- 172.21.2.112:0/1067468268 wait complete. 2021-02-10T11:39:59.787 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.789 7f2e37693700 1 Processor -- start 2021-02-10T11:39:59.787 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.789 7f2e37693700 1 -- start start 2021-02-10T11:39:59.787 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.790 7f2e37693700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2e300c3820 0x7f2e30084cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:59.787 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.790 7f2e37693700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2e300a0290 0x7f2e300852d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:59.788 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.790 7f2e37693700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2e30086270 0x7f2e30086670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:59.788 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.790 7f2e37693700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e300c0700 con 0x7f2e300a0290 2021-02-10T11:39:59.788 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.790 7f2e37693700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e300858f0 con 0x7f2e300c3820 2021-02-10T11:39:59.788 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.790 7f2e37693700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e30085b10 con 0x7f2e30086270 2021-02-10T11:39:59.788 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.790 7f2e34cae700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2e300a0290 0x7f2e300852d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:59.788 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.790 7f2e34cae700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2e300a0290 0x7f2e300852d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:54282/0 (socket says 172.21.2.112:54282) 2021-02-10T11:39:59.788 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.790 7f2e34cae700 1 -- 172.21.2.112:0/3811066599 learned_addr learned my addr 172.21.2.112:0/3811066599 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:39:59.788 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.790 7f2e354af700 1 --2- 172.21.2.112:0/3811066599 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2e300c3820 0x7f2e30084cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:59.788 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.790 7f2e35cb0700 1 --2- 172.21.2.112:0/3811066599 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2e30086270 0x7f2e30086670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:59.788 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.791 7f2e34cae700 1 -- 172.21.2.112:0/3811066599 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2e30086270 msgr2=0x7f2e30086670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:59.788 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.791 7f2e34cae700 1 --2- 172.21.2.112:0/3811066599 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2e30086270 0x7f2e30086670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:59.788 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.791 7f2e34cae700 1 -- 172.21.2.112:0/3811066599 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2e300c3820 msgr2=0x7f2e30084cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:59.788 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.791 7f2e34cae700 1 --2- 172.21.2.112:0/3811066599 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2e300c3820 0x7f2e30084cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:59.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.791 7f2e34cae700 1 -- 172.21.2.112:0/3811066599 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e30086c90 con 0x7f2e300a0290 2021-02-10T11:39:59.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.791 7f2e34cae700 1 --2- 172.21.2.112:0/3811066599 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2e300a0290 0x7f2e300852d0 secure :-1 s=READY pgs=2006 cs=0 l=1 rev1=1 rx=0x7f2e2400b5b0 tx=0x7f2e2400b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:59.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.791 7f2e21ffb700 1 -- 172.21.2.112:0/3811066599 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2e2400bd40 con 0x7f2e300a0290 2021-02-10T11:39:59.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.791 7f2e21ffb700 1 -- 172.21.2.112:0/3811066599 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e24000c60 con 0x7f2e300a0290 2021-02-10T11:39:59.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.791 7f2e21ffb700 1 -- 172.21.2.112:0/3811066599 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2e24006210 con 0x7f2e300a0290 2021-02-10T11:39:59.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.791 7f2e37693700 1 -- 172.21.2.112:0/3811066599 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e30086c90 con 0x7f2e300a0290 2021-02-10T11:39:59.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.791 7f2e37693700 1 -- 172.21.2.112:0/3811066599 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e30086c90 con 0x7f2e300a0290 2021-02-10T11:39:59.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.792 7f2e21ffb700 1 -- 172.21.2.112:0/3811066599 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2e24006020 con 0x7f2e300a0290 2021-02-10T11:39:59.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.792 7f2e21ffb700 1 --2- 172.21.2.112:0/3811066599 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2e100186d0 0x7f2e1001abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:59.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.792 7f2e21ffb700 1 -- 172.21.2.112:0/3811066599 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f2e240327d0 con 0x7f2e300a0290 2021-02-10T11:39:59.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.792 7f2e354af700 1 --2- 172.21.2.112:0/3811066599 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2e100186d0 0x7f2e1001abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:59.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.792 7f2e354af700 1 --2- 172.21.2.112:0/3811066599 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2e100186d0 0x7f2e1001abd0 crc :-1 s=READY pgs=933 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:59.816 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.821 7f2e37693700 1 -- 172.21.2.112:0/3811066599 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e30082a70 con 0x7f2e300a0290 2021-02-10T11:39:59.818 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.824 7f2e21ffb700 1 -- 172.21.2.112:0/3811066599 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2e2401f510 con 0x7f2e300a0290 2021-02-10T11:39:59.949 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.954 7f2e37693700 1 -- 172.21.2.112:0/3811066599 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f2e30082a70 con 0x7f2e300a0290 2021-02-10T11:39:59.949 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.955 7f2e21ffb700 1 -- 172.21.2.112:0/3811066599 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v100) v1 ==== 92+0+10 (secure 0 0 0) 0x7f2e2400bd40 con 0x7f2e300a0290 2021-02-10T11:39:59.950 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:39:59.951 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.957 7f2e37693700 1 -- 172.21.2.112:0/3811066599 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2e100186d0 msgr2=0x7f2e1001abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:59.951 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.957 7f2e37693700 1 --2- 172.21.2.112:0/3811066599 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2e100186d0 0x7f2e1001abd0 crc :-1 s=READY pgs=933 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:59.951 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.957 7f2e37693700 1 -- 172.21.2.112:0/3811066599 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2e300a0290 msgr2=0x7f2e300852d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:59.951 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.957 7f2e37693700 1 --2- 172.21.2.112:0/3811066599 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2e300a0290 0x7f2e300852d0 secure :-1 s=READY pgs=2006 cs=0 l=1 rev1=1 rx=0x7f2e2400b5b0 tx=0x7f2e2400b950).stop 2021-02-10T11:39:59.951 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.957 7f2e37693700 1 -- 172.21.2.112:0/3811066599 shutdown_connections 2021-02-10T11:39:59.951 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.957 7f2e37693700 1 --2- 172.21.2.112:0/3811066599 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2e100186d0 0x7f2e1001abd0 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:59.951 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.957 7f2e37693700 1 --2- 172.21.2.112:0/3811066599 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2e30086270 0x7f2e30086670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:59.951 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.957 7f2e37693700 1 --2- 172.21.2.112:0/3811066599 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2e300a0290 0x7f2e300852d0 unknown :-1 s=CLOSED pgs=2006 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:59.952 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.957 7f2e37693700 1 --2- 172.21.2.112:0/3811066599 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2e300c3820 0x7f2e30084cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:59.952 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.957 7f2e37693700 1 -- 172.21.2.112:0/3811066599 >> 172.21.2.112:0/3811066599 conn(0x7f2e300872c0 msgr2=0x7f2e30095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:59.952 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.957 7f2e37693700 1 -- 172.21.2.112:0/3811066599 shutdown_connections 2021-02-10T11:39:59.952 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:39:59.957 7f2e37693700 1 -- 172.21.2.112:0/3811066599 wait complete. 2021-02-10T11:39:59.959 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:40:00.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.381 7fdfc5fad700 1 Processor -- start 2021-02-10T11:40:00.393 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.381 7fdfc5fad700 1 -- start start 2021-02-10T11:40:00.393 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.382 7fdfc5fad700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdfc00d1870 0x7fdfc01b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:00.393 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.382 7fdfc5fad700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fdfc01b0ae0 0x7fdfc0108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:00.393 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.382 7fdfc5fad700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfc0109a60 con 0x7fdfc00d1c70 2021-02-10T11:40:00.393 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.382 7fdfc5fad700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfc00e5ef0 con 0x7fdfc01b0ae0 2021-02-10T11:40:00.393 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.382 7fdfc5fad700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfc0109750 con 0x7fdfc00d1870 2021-02-10T11:40:00.393 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.382 7fdfbffff700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fdfc00d1c70 0x7fdfc0108f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:53750/0 (socket says 172.21.2.112:53750) 2021-02-10T11:40:00.393 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.382 7fdfbffff700 1 -- 172.21.2.112:0/2100508824 learned_addr learned my addr 172.21.2.112:0/2100508824 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:40:00.393 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.382 7fdfbeffd700 1 --2- 172.21.2.112:0/2100508824 >> v2:172.21.2.120:3300/0 conn(0x7fdfc01b0ae0 0x7fdfc0108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:00.393 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.382 7fdfbf7fe700 1 --2- 172.21.2.112:0/2100508824 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdfc00d1870 0x7fdfc01b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:00.393 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.383 7fdfbe7fc700 1 -- 172.21.2.112:0/2100508824 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 198310077 0 0) 0x7fdfb8000f90 con 0x7fdfc00d1c70 2021-02-10T11:40:00.393 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.383 7fdfbe7fc700 1 -- 172.21.2.112:0/2100508824 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdfac002fe0 con 0x7fdfc00d1c70 2021-02-10T11:40:00.394 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.383 7fdfbeffd700 1 -- 172.21.2.112:0/2100508824 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdfc00d1870 msgr2=0x7fdfc01b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:00.394 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.383 7fdfbeffd700 1 --2- 172.21.2.112:0/2100508824 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdfc00d1870 0x7fdfc01b04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:00.394 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.383 7fdfbeffd700 1 -- 172.21.2.112:0/2100508824 >> v1:172.21.2.112:6789/0 conn(0x7fdfc00d1c70 legacy=0x7fdfc0108f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:00.394 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.383 7fdfbeffd700 1 -- 172.21.2.112:0/2100508824 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfc0088070 con 0x7fdfc01b0ae0 2021-02-10T11:40:00.394 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.383 7fdfbeffd700 1 --2- 172.21.2.112:0/2100508824 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fdfc01b0ae0 0x7fdfc0108940 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 rx=0x7fdfb0006650 tx=0x7fdfb0009a80).ready entity=mon.1 client_cookie=c207b9698a1fec83 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:00.394 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.383 7fdfbe7fc700 1 -- 172.21.2.112:0/2100508824 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdfb00009b0 con 0x7fdfc01b0ae0 2021-02-10T11:40:00.394 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.383 7fdfbe7fc700 1 -- 172.21.2.112:0/2100508824 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfb000c0c0 con 0x7fdfc01b0ae0 2021-02-10T11:40:00.394 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.383 7fdfbe7fc700 1 -- 172.21.2.112:0/2100508824 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdfb000c510 con 0x7fdfc01b0ae0 2021-02-10T11:40:00.394 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.383 7fdfc5fad700 1 -- 172.21.2.112:0/2100508824 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fdfc01b0ae0 msgr2=0x7fdfc0108940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:00.394 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.383 7fdfc5fad700 1 --2- 172.21.2.112:0/2100508824 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fdfc01b0ae0 0x7fdfc0108940 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 rx=0x7fdfb0006650 tx=0x7fdfb0009a80).stop 2021-02-10T11:40:00.394 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.384 7fdfc5fad700 1 -- 172.21.2.112:0/2100508824 shutdown_connections 2021-02-10T11:40:00.394 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.384 7fdfc5fad700 1 --2- 172.21.2.112:0/2100508824 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fdfc01b0ae0 0x7fdfc0108940 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:00.394 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.384 7fdfc5fad700 1 --2- 172.21.2.112:0/2100508824 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdfc00d1870 0x7fdfc01b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:00.395 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.384 7fdfc5fad700 1 -- 172.21.2.112:0/2100508824 >> 172.21.2.112:0/2100508824 conn(0x7fdfc00ce420 msgr2=0x7fdfc00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:00.395 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.384 7fdfc5fad700 1 -- 172.21.2.112:0/2100508824 shutdown_connections 2021-02-10T11:40:00.395 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.384 7fdfc5fad700 1 -- 172.21.2.112:0/2100508824 wait complete. 2021-02-10T11:40:00.395 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.398 7fdfc5fad700 1 Processor -- start 2021-02-10T11:40:00.395 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.398 7fdfc5fad700 1 -- start start 2021-02-10T11:40:00.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.399 7fdfc5fad700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdfc01a0580 0x7fdfc0093e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:00.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.399 7fdfc5fad700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fdfc00944a0 0x7fdfc00c19a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:00.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.399 7fdfc5fad700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdfc00c1fc0 0x7fdfc0085190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:00.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.399 7fdfc5fad700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfc0084c90 con 0x7fdfc01a0580 2021-02-10T11:40:00.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.399 7fdfc5fad700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfc008fe80 con 0x7fdfc00944a0 2021-02-10T11:40:00.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.399 7fdfc5fad700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfc00c23c0 con 0x7fdfc00c1fc0 2021-02-10T11:40:00.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.399 7fdfbf7fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdfc01a0580 0x7fdfc0093e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:00.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.399 7fdfbf7fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdfc01a0580 0x7fdfc0093e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:54292/0 (socket says 172.21.2.112:54292) 2021-02-10T11:40:00.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.399 7fdfbf7fe700 1 -- 172.21.2.112:0/3098374568 learned_addr learned my addr 172.21.2.112:0/3098374568 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:40:00.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.399 7fdfbeffd700 1 --2- 172.21.2.112:0/3098374568 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fdfc00944a0 0x7fdfc00c19a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:00.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.399 7fdfbffff700 1 --2- 172.21.2.112:0/3098374568 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdfc00c1fc0 0x7fdfc0085190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:00.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.399 7fdfbf7fe700 1 -- 172.21.2.112:0/3098374568 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdfc00c1fc0 msgr2=0x7fdfc0085190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:00.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.399 7fdfbf7fe700 1 --2- 172.21.2.112:0/3098374568 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdfc00c1fc0 0x7fdfc0085190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:00.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.399 7fdfbf7fe700 1 -- 172.21.2.112:0/3098374568 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fdfc00944a0 msgr2=0x7fdfc00c19a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:00.398 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.399 7fdfbf7fe700 1 --2- 172.21.2.112:0/3098374568 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fdfc00944a0 0x7fdfc00c19a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:00.398 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.399 7fdfbf7fe700 1 -- 172.21.2.112:0/3098374568 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfc00857b0 con 0x7fdfc01a0580 2021-02-10T11:40:00.398 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.399 7fdfbeffd700 1 --2- 172.21.2.112:0/3098374568 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fdfc00944a0 0x7fdfc00c19a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:40:00.398 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.399 7fdfbffff700 1 --2- 172.21.2.112:0/3098374568 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdfc00c1fc0 0x7fdfc0085190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:40:00.398 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.399 7fdfbf7fe700 1 --2- 172.21.2.112:0/3098374568 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdfc01a0580 0x7fdfc0093e80 secure :-1 s=READY pgs=2008 cs=0 l=1 rev1=1 rx=0x7fdfa8006f60 tx=0x7fdfa800b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:00.398 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.400 7fdfb7fff700 1 -- 172.21.2.112:0/3098374568 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdfa8006bb0 con 0x7fdfc01a0580 2021-02-10T11:40:00.398 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.400 7fdfb7fff700 1 -- 172.21.2.112:0/3098374568 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfa8006340 con 0x7fdfc01a0580 2021-02-10T11:40:00.398 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.400 7fdfb7fff700 1 -- 172.21.2.112:0/3098374568 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdfa8002030 con 0x7fdfc01a0580 2021-02-10T11:40:00.398 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.400 7fdfc5fad700 1 -- 172.21.2.112:0/3098374568 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfc00857b0 con 0x7fdfc01a0580 2021-02-10T11:40:00.398 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.400 7fdfc5fad700 1 -- 172.21.2.112:0/3098374568 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfc0085d40 con 0x7fdfc01a0580 2021-02-10T11:40:00.398 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.400 7fdfb7fff700 1 -- 172.21.2.112:0/3098374568 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fdfa8006b50 con 0x7fdfc01a0580 2021-02-10T11:40:00.398 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.400 7fdfb7fff700 1 --2- 172.21.2.112:0/3098374568 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fdfa40186c0 0x7fdfa401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:00.399 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.400 7fdfb7fff700 1 -- 172.21.2.112:0/3098374568 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fdfa8032560 con 0x7fdfc01a0580 2021-02-10T11:40:00.399 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.401 7fdfbeffd700 1 --2- 172.21.2.112:0/3098374568 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fdfa40186c0 0x7fdfa401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:00.399 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.401 7fdfbeffd700 1 --2- 172.21.2.112:0/3098374568 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fdfa40186c0 0x7fdfa401abc0 crc :-1 s=READY pgs=934 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:00.426 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.432 7fdfc5fad700 1 -- 172.21.2.112:0/3098374568 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfc0082980 con 0x7fdfc01a0580 2021-02-10T11:40:00.428 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.434 7fdfb7fff700 1 -- 172.21.2.112:0/3098374568 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fdfa8000940 con 0x7fdfc01a0580 2021-02-10T11:40:00.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.564 7fdfc5fad700 1 -- 172.21.2.112:0/3098374568 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7fdfc0082980 con 0x7fdfc01a0580 2021-02-10T11:40:00.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.565 7fdfb7fff700 1 -- 172.21.2.112:0/3098374568 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v100) v1 ==== 104+0+10 (secure 0 0 0) 0x7fdfa800bd60 con 0x7fdfc01a0580 2021-02-10T11:40:00.560 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:40:00.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.567 7fdfc5fad700 1 -- 172.21.2.112:0/3098374568 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fdfa40186c0 msgr2=0x7fdfa401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:00.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.567 7fdfc5fad700 1 --2- 172.21.2.112:0/3098374568 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fdfa40186c0 0x7fdfa401abc0 crc :-1 s=READY pgs=934 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:00.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.567 7fdfc5fad700 1 -- 172.21.2.112:0/3098374568 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdfc01a0580 msgr2=0x7fdfc0093e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:00.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.567 7fdfc5fad700 1 --2- 172.21.2.112:0/3098374568 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdfc01a0580 0x7fdfc0093e80 secure :-1 s=READY pgs=2008 cs=0 l=1 rev1=1 rx=0x7fdfa8006f60 tx=0x7fdfa800b970).stop 2021-02-10T11:40:00.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.567 7fdfc5fad700 1 -- 172.21.2.112:0/3098374568 shutdown_connections 2021-02-10T11:40:00.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.567 7fdfc5fad700 1 --2- 172.21.2.112:0/3098374568 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fdfa40186c0 0x7fdfa401abc0 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:00.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.567 7fdfc5fad700 1 --2- 172.21.2.112:0/3098374568 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fdfc00944a0 0x7fdfc00c19a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:00.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.567 7fdfc5fad700 1 --2- 172.21.2.112:0/3098374568 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdfc00c1fc0 0x7fdfc0085190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:00.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.567 7fdfc5fad700 1 --2- 172.21.2.112:0/3098374568 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdfc01a0580 0x7fdfc0093e80 unknown :-1 s=CLOSED pgs=2008 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:00.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.567 7fdfc5fad700 1 -- 172.21.2.112:0/3098374568 >> 172.21.2.112:0/3098374568 conn(0x7fdfc009e820 msgr2=0x7fdfc00c3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:00.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.567 7fdfc5fad700 1 -- 172.21.2.112:0/3098374568 shutdown_connections 2021-02-10T11:40:00.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.567 7fdfc5fad700 1 -- 172.21.2.112:0/3098374568 wait complete. 2021-02-10T11:40:00.569 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:40:01.001 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.989 7f4b54905700 1 Processor -- start 2021-02-10T11:40:01.001 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.989 7f4b54905700 1 -- start start 2021-02-10T11:40:01.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.990 7f4b54905700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f4b500d1c70 0x7f4b500cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:01.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.990 7f4b54905700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4b500cf580 0x7f4b500cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:01.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.990 7f4b54905700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b500fe1c0 con 0x7f4b500d1870 2021-02-10T11:40:01.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.990 7f4b54905700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b501b0cb0 con 0x7f4b500d1c70 2021-02-10T11:40:01.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.990 7f4b54905700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b50109a60 con 0x7f4b500cf580 2021-02-10T11:40:01.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.990 7f4b4e61c700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f4b500d1870 0x7f4b501b04c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:53762/0 (socket says 172.21.2.112:53762) 2021-02-10T11:40:01.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.990 7f4b4e61c700 1 -- 172.21.2.112:0/763661952 learned_addr learned my addr 172.21.2.112:0/763661952 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:40:01.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.990 7f4b4d61a700 1 -- 172.21.2.112:0/763661952 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1812382858 0 0) 0x7f4b38001180 con 0x7f4b500d1870 2021-02-10T11:40:01.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.990 7f4b4d61a700 1 -- 172.21.2.112:0/763661952 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b3c002fe0 con 0x7f4b500d1870 2021-02-10T11:40:01.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.990 7f4b4de1b700 1 --2- 172.21.2.112:0/763661952 >> v2:172.21.2.120:3300/0 conn(0x7f4b500d1c70 0x7f4b500cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:01.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.990 7f4b4ee1d700 1 --2- 172.21.2.112:0/763661952 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4b500cf580 0x7f4b500cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:01.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.990 7f4b4d61a700 1 -- 172.21.2.112:0/763661952 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1397052352 0 0) 0x7f4b38001180 con 0x7f4b500d1870 2021-02-10T11:40:01.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.990 7f4b4d61a700 1 -- 172.21.2.112:0/763661952 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4b500cf580 msgr2=0x7f4b500cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:01.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.990 7f4b4d61a700 1 --2- 172.21.2.112:0/763661952 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4b500cf580 0x7f4b500cf980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:01.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.990 7f4b4d61a700 1 -- 172.21.2.112:0/763661952 >> v2:172.21.2.120:3300/0 conn(0x7f4b500d1c70 msgr2=0x7f4b500cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:01.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.990 7f4b4d61a700 1 --2- 172.21.2.112:0/763661952 >> v2:172.21.2.120:3300/0 conn(0x7f4b500d1c70 0x7f4b500cef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:01.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.990 7f4b4d61a700 1 -- 172.21.2.112:0/763661952 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b50088070 con 0x7f4b500d1870 2021-02-10T11:40:01.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.990 7f4b4d61a700 1 -- 172.21.2.112:0/763661952 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f4b38001680 con 0x7f4b500d1870 2021-02-10T11:40:01.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.990 7f4b4d61a700 1 -- 172.21.2.112:0/763661952 >> v1:172.21.2.112:6789/0 conn(0x7f4b500d1870 legacy=0x7f4b501b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:01.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.990 7f4b4d61a700 1 --2- 172.21.2.112:0/763661952 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4b3c003b90 0x7f4b3c0060b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:01.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.990 7f4b4d61a700 1 --2- 172.21.2.112:0/763661952 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4b3c0068d0 0x7f4b3c008dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:01.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.990 7f4b4d61a700 1 --2- 172.21.2.112:0/763661952 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4b3c009660 0x7f4b3c00bb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:01.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.990 7f4b4d61a700 1 -- 172.21.2.112:0/763661952 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b3c00c430 con 0x7f4b3c009660 2021-02-10T11:40:01.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.990 7f4b4d61a700 1 -- 172.21.2.112:0/763661952 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b3c00c650 con 0x7f4b3c003b90 2021-02-10T11:40:01.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.990 7f4b4d61a700 1 -- 172.21.2.112:0/763661952 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b3c00c870 con 0x7f4b3c0068d0 2021-02-10T11:40:01.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.991 7f4b4de1b700 1 --2- 172.21.2.112:0/763661952 >> v2:172.21.2.120:3300/0 conn(0x7f4b500d1c70 0x7f4b500cef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:40:01.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.991 7f4b4ee1d700 1 --2- 172.21.2.112:0/763661952 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4b500cf580 0x7f4b500cf980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:40:01.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.991 7f4b4ee1d700 1 --2- 172.21.2.112:0/763661952 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4b3c009660 0x7f4b3c00bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:01.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.991 7f4b4e61c700 1 --2- 172.21.2.112:0/763661952 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4b3c003b90 0x7f4b3c0060b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:01.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.991 7f4b4de1b700 1 --2- 172.21.2.112:0/763661952 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4b3c0068d0 0x7f4b3c008dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:01.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.991 7f4b4de1b700 1 -- 172.21.2.112:0/763661952 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4b3c003b90 msgr2=0x7f4b3c0060b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:01.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.991 7f4b4de1b700 1 --2- 172.21.2.112:0/763661952 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4b3c003b90 0x7f4b3c0060b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:01.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.991 7f4b4de1b700 1 -- 172.21.2.112:0/763661952 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4b3c009660 msgr2=0x7f4b3c00bb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:01.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.991 7f4b4de1b700 1 --2- 172.21.2.112:0/763661952 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4b3c009660 0x7f4b3c00bb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:01.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.991 7f4b4de1b700 1 -- 172.21.2.112:0/763661952 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b3c050b20 con 0x7f4b3c0068d0 2021-02-10T11:40:01.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.991 7f4b4ee1d700 1 --2- 172.21.2.112:0/763661952 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4b3c009660 0x7f4b3c00bb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:40:01.005 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.991 7f4b4de1b700 1 --2- 172.21.2.112:0/763661952 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4b3c0068d0 0x7f4b3c008dc0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7f4b40009970 tx=0x7f4b40009d10).ready entity=mon.2 client_cookie=bb1c1fc57a660cb5 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:01.005 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.992 7f4b4d61a700 1 -- 172.21.2.112:0/763661952 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4b40006960 con 0x7f4b3c0068d0 2021-02-10T11:40:01.005 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.992 7f4b4d61a700 1 -- 172.21.2.112:0/763661952 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b40000c90 con 0x7f4b3c0068d0 2021-02-10T11:40:01.005 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.992 7f4b4d61a700 1 -- 172.21.2.112:0/763661952 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4b400022b0 con 0x7f4b3c0068d0 2021-02-10T11:40:01.005 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.992 7f4b54905700 1 -- 172.21.2.112:0/763661952 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4b3c0068d0 msgr2=0x7f4b3c008dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:01.005 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.992 7f4b54905700 1 --2- 172.21.2.112:0/763661952 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4b3c0068d0 0x7f4b3c008dc0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7f4b40009970 tx=0x7f4b40009d10).stop 2021-02-10T11:40:01.005 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.992 7f4b4ee1d700 1 -- 172.21.2.112:0/763661952 reap_dead start 2021-02-10T11:40:01.005 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.992 7f4b54905700 1 -- 172.21.2.112:0/763661952 shutdown_connections 2021-02-10T11:40:01.005 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.992 7f4b54905700 1 -- 172.21.2.112:0/763661952 >> 172.21.2.112:0/763661952 conn(0x7f4b500ce420 msgr2=0x7f4b500d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:01.005 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.992 7f4b54905700 1 -- 172.21.2.112:0/763661952 shutdown_connections 2021-02-10T11:40:01.005 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:00.992 7f4b54905700 1 -- 172.21.2.112:0/763661952 wait complete. 2021-02-10T11:40:01.005 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.007 7f4b54905700 1 Processor -- start 2021-02-10T11:40:01.005 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.007 7f4b54905700 1 -- start start 2021-02-10T11:40:01.008 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.007 7f4b54905700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4b50085bc0 0x7f4b500863e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:01.008 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.007 7f4b54905700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4b500954c0 0x7f4b50096300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:01.008 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.007 7f4b54905700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4b500958c0 0x7f4b50096920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:01.008 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.007 7f4b54905700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b50095d80 con 0x7f4b500958c0 2021-02-10T11:40:01.008 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.007 7f4b54905700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b500bfa60 con 0x7f4b50085bc0 2021-02-10T11:40:01.008 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.007 7f4b54905700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b50085fc0 con 0x7f4b500954c0 2021-02-10T11:40:01.008 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.008 7f4b4ee1d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4b500958c0 0x7f4b50096920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:01.009 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.008 7f4b4ee1d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4b500958c0 0x7f4b50096920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:54316/0 (socket says 172.21.2.112:54316) 2021-02-10T11:40:01.009 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.008 7f4b4ee1d700 1 -- 172.21.2.112:0/3196744382 learned_addr learned my addr 172.21.2.112:0/3196744382 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:40:01.009 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.008 7f4b4ee1d700 1 -- 172.21.2.112:0/3196744382 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4b500954c0 msgr2=0x7f4b50096300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:01.009 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.008 7f4b4ee1d700 1 --2- 172.21.2.112:0/3196744382 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4b500954c0 0x7f4b50096300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:01.009 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.008 7f4b4ee1d700 1 -- 172.21.2.112:0/3196744382 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4b50085bc0 msgr2=0x7f4b500863e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:01.009 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.008 7f4b4ee1d700 1 --2- 172.21.2.112:0/3196744382 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4b50085bc0 0x7f4b500863e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:01.009 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.008 7f4b4ee1d700 1 -- 172.21.2.112:0/3196744382 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b50086c10 con 0x7f4b500958c0 2021-02-10T11:40:01.009 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.008 7f4b4ee1d700 1 --2- 172.21.2.112:0/3196744382 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4b500958c0 0x7f4b50096920 secure :-1 s=READY pgs=2010 cs=0 l=1 rev1=1 rx=0x7f4b48006050 tx=0x7f4b4800ac80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:01.009 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.008 7f4b46ffd700 1 -- 172.21.2.112:0/3196744382 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4b480056b0 con 0x7f4b500958c0 2021-02-10T11:40:01.009 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.008 7f4b46ffd700 1 -- 172.21.2.112:0/3196744382 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b4800adc0 con 0x7f4b500958c0 2021-02-10T11:40:01.009 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.008 7f4b54905700 1 -- 172.21.2.112:0/3196744382 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b50086c10 con 0x7f4b500958c0 2021-02-10T11:40:01.009 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.008 7f4b46ffd700 1 -- 172.21.2.112:0/3196744382 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4b4800c4f0 con 0x7f4b500958c0 2021-02-10T11:40:01.009 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.008 7f4b54905700 1 -- 172.21.2.112:0/3196744382 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b50097230 con 0x7f4b500958c0 2021-02-10T11:40:01.009 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.009 7f4b46ffd700 1 -- 172.21.2.112:0/3196744382 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4b480011f0 con 0x7f4b500958c0 2021-02-10T11:40:01.010 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.009 7f4b46ffd700 1 --2- 172.21.2.112:0/3196744382 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4b340186c0 0x7f4b3401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:01.010 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.009 7f4b46ffd700 1 -- 172.21.2.112:0/3196744382 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f4b480328d0 con 0x7f4b500958c0 2021-02-10T11:40:01.010 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.009 7f4b4e61c700 1 --2- 172.21.2.112:0/3196744382 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4b340186c0 0x7f4b3401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:01.010 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.010 7f4b4e61c700 1 --2- 172.21.2.112:0/3196744382 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4b340186c0 0x7f4b3401abc0 crc :-1 s=READY pgs=935 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:01.034 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.040 7f4b54905700 1 -- 172.21.2.112:0/3196744382 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b500fe1c0 con 0x7f4b500958c0 2021-02-10T11:40:01.036 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.042 7f4b46ffd700 1 -- 172.21.2.112:0/3196744382 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4b48000e20 con 0x7f4b500958c0 2021-02-10T11:40:01.165 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.171 7f4b54905700 1 -- 172.21.2.112:0/3196744382 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f4b500fe1c0 con 0x7f4b500958c0 2021-02-10T11:40:01.166 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.171 7f4b46ffd700 1 -- 172.21.2.112:0/3196744382 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v100) v1 ==== 100+0+10 (secure 0 0 0) 0x7f4b480011f0 con 0x7f4b500958c0 2021-02-10T11:40:01.166 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:40:01.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.173 7f4b54905700 1 -- 172.21.2.112:0/3196744382 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4b340186c0 msgr2=0x7f4b3401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:01.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.173 7f4b54905700 1 --2- 172.21.2.112:0/3196744382 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4b340186c0 0x7f4b3401abc0 crc :-1 s=READY pgs=935 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:01.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.173 7f4b54905700 1 -- 172.21.2.112:0/3196744382 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4b500958c0 msgr2=0x7f4b50096920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:01.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.173 7f4b54905700 1 --2- 172.21.2.112:0/3196744382 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4b500958c0 0x7f4b50096920 secure :-1 s=READY pgs=2010 cs=0 l=1 rev1=1 rx=0x7f4b48006050 tx=0x7f4b4800ac80).stop 2021-02-10T11:40:01.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.173 7f4b54905700 1 -- 172.21.2.112:0/3196744382 shutdown_connections 2021-02-10T11:40:01.169 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.174 7f4b54905700 1 --2- 172.21.2.112:0/3196744382 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4b340186c0 0x7f4b3401abc0 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:01.169 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.174 7f4b54905700 1 --2- 172.21.2.112:0/3196744382 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4b500958c0 0x7f4b50096920 unknown :-1 s=CLOSED pgs=2010 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:01.169 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.174 7f4b54905700 1 --2- 172.21.2.112:0/3196744382 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4b500954c0 0x7f4b50096300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:01.169 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.174 7f4b54905700 1 --2- 172.21.2.112:0/3196744382 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4b50085bc0 0x7f4b500863e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:01.169 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.174 7f4b54905700 1 -- 172.21.2.112:0/3196744382 >> 172.21.2.112:0/3196744382 conn(0x7f4b500bef00 msgr2=0x7f4b5009d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:01.169 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.174 7f4b54905700 1 -- 172.21.2.112:0/3196744382 shutdown_connections 2021-02-10T11:40:01.169 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.174 7f4b54905700 1 -- 172.21.2.112:0/3196744382 wait complete. 2021-02-10T11:40:01.176 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:40:01.176 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:40:01.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.593 7fe42b667700 1 Processor -- start 2021-02-10T11:40:01.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.593 7fe42b667700 1 -- start start 2021-02-10T11:40:01.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.594 7fe42b667700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fe4241b0470 0x7fe4241b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:01.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.594 7fe42b667700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe4240f20e0 0x7fe4240f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:01.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.594 7fe42b667700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4240f0b50 con 0x7fe4240ef610 2021-02-10T11:40:01.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.594 7fe42b667700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4240f0820 con 0x7fe4241b0470 2021-02-10T11:40:01.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.594 7fe42b667700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4240f24e0 con 0x7fe4240f20e0 2021-02-10T11:40:01.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.594 7fe428c82700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fe4240ef610 0x7fe4240efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:53782/0 (socket says 172.21.2.112:53782) 2021-02-10T11:40:01.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.594 7fe428c82700 1 -- 172.21.2.112:0/703280843 learned_addr learned my addr 172.21.2.112:0/703280843 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:40:01.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.594 7fe429c84700 1 --2- 172.21.2.112:0/703280843 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe4240f20e0 0x7fe4240f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:01.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.594 7fe429483700 1 --2- 172.21.2.112:0/703280843 >> v2:172.21.2.120:3300/0 conn(0x7fe4241b0470 0x7fe4241b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:01.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.594 7fe417fff700 1 -- 172.21.2.112:0/703280843 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2968979278 0 0) 0x7fe4200010b0 con 0x7fe4240ef610 2021-02-10T11:40:01.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.594 7fe417fff700 1 -- 172.21.2.112:0/703280843 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe410002fe0 con 0x7fe4240ef610 2021-02-10T11:40:01.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.594 7fe429c84700 1 -- 172.21.2.112:0/703280843 >> v2:172.21.2.120:3300/0 conn(0x7fe4241b0470 msgr2=0x7fe4241b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:01.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.594 7fe429c84700 1 --2- 172.21.2.112:0/703280843 >> v2:172.21.2.120:3300/0 conn(0x7fe4241b0470 0x7fe4241b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:01.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.594 7fe429c84700 1 -- 172.21.2.112:0/703280843 >> v1:172.21.2.112:6789/0 conn(0x7fe4240ef610 legacy=0x7fe4240efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:01.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.594 7fe429c84700 1 -- 172.21.2.112:0/703280843 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4240e52e0 con 0x7fe4240f20e0 2021-02-10T11:40:01.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.595 7fe429c84700 1 --2- 172.21.2.112:0/703280843 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe4240f20e0 0x7fe4240f0200 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7fe41c006e50 tx=0x7fe41c009e70).ready entity=mon.2 client_cookie=6b40004cc0a8f1de server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:01.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.595 7fe417fff700 1 -- 172.21.2.112:0/703280843 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe41c006a20 con 0x7fe4240f20e0 2021-02-10T11:40:01.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.595 7fe417fff700 1 -- 172.21.2.112:0/703280843 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe41c006790 con 0x7fe4240f20e0 2021-02-10T11:40:01.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.595 7fe417fff700 1 -- 172.21.2.112:0/703280843 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe41c000f90 con 0x7fe4240f20e0 2021-02-10T11:40:01.606 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.595 7fe42b667700 1 -- 172.21.2.112:0/703280843 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe4240f20e0 msgr2=0x7fe4240f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:01.606 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.595 7fe42b667700 1 --2- 172.21.2.112:0/703280843 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe4240f20e0 0x7fe4240f0200 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7fe41c006e50 tx=0x7fe41c009e70).stop 2021-02-10T11:40:01.606 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.595 7fe42b667700 1 -- 172.21.2.112:0/703280843 shutdown_connections 2021-02-10T11:40:01.606 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.595 7fe42b667700 1 --2- 172.21.2.112:0/703280843 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe4240f20e0 0x7fe4240f0200 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:01.606 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.595 7fe42b667700 1 --2- 172.21.2.112:0/703280843 >> v2:172.21.2.120:3300/0 conn(0x7fe4241b0470 0x7fe4241b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:01.606 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.595 7fe42b667700 1 -- 172.21.2.112:0/703280843 >> 172.21.2.112:0/703280843 conn(0x7fe4240ce3e0 msgr2=0x7fe4240e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:01.606 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.595 7fe42b667700 1 -- 172.21.2.112:0/703280843 shutdown_connections 2021-02-10T11:40:01.606 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.595 7fe42b667700 1 -- 172.21.2.112:0/703280843 wait complete. 2021-02-10T11:40:01.607 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.610 7fe42b667700 1 Processor -- start 2021-02-10T11:40:01.608 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.610 7fe42b667700 1 -- start start 2021-02-10T11:40:01.608 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.610 7fe42b667700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe4241a8570 0x7fe42418e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:01.608 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.610 7fe42b667700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe4240a8780 0x7fe42418ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:01.608 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.610 7fe42b667700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe42418f3b0 0x7fe42418f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:01.608 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.610 7fe42b667700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe42418fe80 con 0x7fe42418f3b0 2021-02-10T11:40:01.608 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.610 7fe42b667700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe424190400 con 0x7fe4240a8780 2021-02-10T11:40:01.608 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.610 7fe42b667700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe424190620 con 0x7fe4241a8570 2021-02-10T11:40:01.608 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.611 7fe429c84700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe42418f3b0 0x7fe42418f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:01.608 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.611 7fe429c84700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe42418f3b0 0x7fe42418f7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:54330/0 (socket says 172.21.2.112:54330) 2021-02-10T11:40:01.608 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.611 7fe429c84700 1 -- 172.21.2.112:0/934107385 learned_addr learned my addr 172.21.2.112:0/934107385 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:40:01.608 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.611 7fe429483700 1 --2- 172.21.2.112:0/934107385 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe4241a8570 0x7fe42418e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:01.608 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.611 7fe428c82700 1 --2- 172.21.2.112:0/934107385 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe4240a8780 0x7fe42418ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:01.609 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.611 7fe429c84700 1 -- 172.21.2.112:0/934107385 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe4241a8570 msgr2=0x7fe42418e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:01.609 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.611 7fe429c84700 1 --2- 172.21.2.112:0/934107385 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe4241a8570 0x7fe42418e4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:01.609 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.611 7fe429c84700 1 -- 172.21.2.112:0/934107385 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe4240a8780 msgr2=0x7fe42418ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:01.609 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.611 7fe429c84700 1 --2- 172.21.2.112:0/934107385 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe4240a8780 0x7fe42418ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:01.609 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.611 7fe429c84700 1 -- 172.21.2.112:0/934107385 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe424190840 con 0x7fe42418f3b0 2021-02-10T11:40:01.609 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.611 7fe429483700 1 --2- 172.21.2.112:0/934107385 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe4241a8570 0x7fe42418e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:40:01.609 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.611 7fe428c82700 1 --2- 172.21.2.112:0/934107385 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe4240a8780 0x7fe42418ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:40:01.609 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.611 7fe429c84700 1 --2- 172.21.2.112:0/934107385 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe42418f3b0 0x7fe42418f7b0 secure :-1 s=READY pgs=2012 cs=0 l=1 rev1=1 rx=0x7fe41c006930 tx=0x7fe41c00bd50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:01.609 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.611 7fe415ffb700 1 -- 172.21.2.112:0/934107385 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe41c0093d0 con 0x7fe42418f3b0 2021-02-10T11:40:01.609 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.611 7fe415ffb700 1 -- 172.21.2.112:0/934107385 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe41c002030 con 0x7fe42418f3b0 2021-02-10T11:40:01.609 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.611 7fe42b667700 1 -- 172.21.2.112:0/934107385 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe424190840 con 0x7fe42418f3b0 2021-02-10T11:40:01.609 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.611 7fe415ffb700 1 -- 172.21.2.112:0/934107385 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe41c002550 con 0x7fe42418f3b0 2021-02-10T11:40:01.609 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.611 7fe42b667700 1 -- 172.21.2.112:0/934107385 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe424190dd0 con 0x7fe42418f3b0 2021-02-10T11:40:01.609 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.612 7fe415ffb700 1 -- 172.21.2.112:0/934107385 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe41c002210 con 0x7fe42418f3b0 2021-02-10T11:40:01.610 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.612 7fe415ffb700 1 --2- 172.21.2.112:0/934107385 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe4040186d0 0x7fe40401abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:01.610 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.612 7fe415ffb700 1 -- 172.21.2.112:0/934107385 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fe41c032b30 con 0x7fe42418f3b0 2021-02-10T11:40:01.610 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.612 7fe429483700 1 --2- 172.21.2.112:0/934107385 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe4040186d0 0x7fe40401abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:01.610 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.613 7fe429483700 1 --2- 172.21.2.112:0/934107385 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe4040186d0 0x7fe40401abd0 crc :-1 s=READY pgs=936 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:01.638 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.644 7fe42b667700 1 -- 172.21.2.112:0/934107385 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe424105ab0 con 0x7fe42418f3b0 2021-02-10T11:40:01.640 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.646 7fe415ffb700 1 -- 172.21.2.112:0/934107385 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe41c0093d0 con 0x7fe42418f3b0 2021-02-10T11:40:01.768 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.774 7fe42b667700 1 -- 172.21.2.112:0/934107385 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe4240f0b50 con 0x7fe42418f3b0 2021-02-10T11:40:01.769 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.774 7fe415ffb700 1 -- 172.21.2.112:0/934107385 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+2104 (secure 0 0 0) 0x7fe41c002030 con 0x7fe42418f3b0 2021-02-10T11:40:01.770 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 80 2021-02-10T11:40:01.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.776 7fe42b667700 1 -- 172.21.2.112:0/934107385 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe4040186d0 msgr2=0x7fe40401abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:01.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.776 7fe42b667700 1 --2- 172.21.2.112:0/934107385 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe4040186d0 0x7fe40401abd0 crc :-1 s=READY pgs=936 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:01.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.776 7fe42b667700 1 -- 172.21.2.112:0/934107385 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe42418f3b0 msgr2=0x7fe42418f7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:01.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.776 7fe42b667700 1 --2- 172.21.2.112:0/934107385 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe42418f3b0 0x7fe42418f7b0 secure :-1 s=READY pgs=2012 cs=0 l=1 rev1=1 rx=0x7fe41c006930 tx=0x7fe41c00bd50).stop 2021-02-10T11:40:01.771 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:40:01.771 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":80,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10011,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/3938237927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":3938237927},{"type":"v1","addr":"172.21.2.112:6833","nonce":3938237927}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":76}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:39:44.311361","modified":"2021-02-10 11:39:46.320259","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7633},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7633":{"gid":7633,"name":"b","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.2.120:6835/1537197833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":1537197833},{"type":"v1","addr":"172.21.2.120:6835","nonce":1537197833}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":10}]} 2021-02-10T11:40:01.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.777 7fe42b667700 1 -- 172.21.2.112:0/934107385 shutdown_connections 2021-02-10T11:40:01.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.777 7fe42b667700 1 --2- 172.21.2.112:0/934107385 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe4040186d0 0x7fe40401abd0 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:01.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.777 7fe42b667700 1 --2- 172.21.2.112:0/934107385 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe4240a8780 0x7fe42418ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:01.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.777 7fe42b667700 1 --2- 172.21.2.112:0/934107385 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe42418f3b0 0x7fe42418f7b0 unknown :-1 s=CLOSED pgs=2012 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:01.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.777 7fe42b667700 1 --2- 172.21.2.112:0/934107385 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe4241a8570 0x7fe42418e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:01.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.777 7fe42b667700 1 -- 172.21.2.112:0/934107385 >> 172.21.2.112:0/934107385 conn(0x7fe42419c4c0 msgr2=0x7fe4240916f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:01.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.777 7fe42b667700 1 -- 172.21.2.112:0/934107385 shutdown_connections 2021-02-10T11:40:01.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:01.777 7fe42b667700 1 -- 172.21.2.112:0/934107385 wait complete. 2021-02-10T11:40:01.981 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 80, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:39:44.311361', 'modified': '2021-02-10 11:39:46.320259', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7633}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7633': {'gid': 7633, 'name': 'b', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.120:6835/1537197833', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.120:6834', 'nonce': 1537197833}, {'type': 'v1', 'addr': '172.21.2.120:6835', 'nonce': 1537197833}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:40:01.981 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:40:01.982 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:40:02.383 INFO:teuthology.orchestra.run.gibba020.stdout:{ 2021-02-10T11:40:02.383 INFO:teuthology.orchestra.run.gibba020.stdout: "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", 2021-02-10T11:40:02.383 INFO:teuthology.orchestra.run.gibba020.stdout: "whoami": 0, 2021-02-10T11:40:02.384 INFO:teuthology.orchestra.run.gibba020.stdout: "id": 7633, 2021-02-10T11:40:02.384 INFO:teuthology.orchestra.run.gibba020.stdout: "want_state": "up:active", 2021-02-10T11:40:02.384 INFO:teuthology.orchestra.run.gibba020.stdout: "state": "up:active", 2021-02-10T11:40:02.384 INFO:teuthology.orchestra.run.gibba020.stdout: "rank_uptime": 18.076009575, 2021-02-10T11:40:02.384 INFO:teuthology.orchestra.run.gibba020.stdout: "mdsmap_epoch": 80, 2021-02-10T11:40:02.384 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch": 100, 2021-02-10T11:40:02.384 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch_barrier": 100, 2021-02-10T11:40:02.384 INFO:teuthology.orchestra.run.gibba020.stdout: "uptime": 23.716856180000001 2021-02-10T11:40:02.384 INFO:teuthology.orchestra.run.gibba020.stdout:} 2021-02-10T11:40:02.387 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", "whoami": 0, "id": 7633, "want_state": "up:active", "state": "up:active", "rank_uptime": 18.076009575, "mdsmap_epoch": 80, "osdmap_epoch": 100, "osdmap_epoch_barrier": 100, "uptime": 23.716856180000001 } 2021-02-10T11:40:02.387 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:40:02.387 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2021-02-10T11:40:02.388 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.1 at ubuntu@gibba012.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2021-02-10T11:40:02.388 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:40:02.388 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:40:02.730 DEBUG:teuthology.orchestra.run.gibba012:> sudo modprobe fuse 2021-02-10T11:40:03.077 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:40:03.077 DEBUG:teuthology.orchestra.run.gibba012:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:40:03.423 INFO:teuthology.orchestra.run.gibba012.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:40:03.423 INFO:teuthology.orchestra.run.gibba012.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:40:03.424 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:40:03.424 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:40:03.424 DEBUG:teuthology.orchestra.run.gibba012:> ls /sys/fs/fuse/connections 2021-02-10T11:40:03.761 INFO:teuthology.orchestra.run.gibba012.stdout:42 2021-02-10T11:40:03.761 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [42] 2021-02-10T11:40:03.761 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.1 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:40:03.764 DEBUG:teuthology.orchestra.run.gibba012:> sudo modprobe fuse 2021-02-10T11:40:04.110 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:40:04.110 DEBUG:teuthology.orchestra.run.gibba012:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:40:04.176 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba012.stderr:ceph-fuse[154636]: starting ceph client 2021-02-10T11:40:04.176 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba012.stderr:2021-02-10 11:40:04.181 7fd5f32faf80 -1 init, newargv = 0x558251e81220 newargc=9 2021-02-10T11:40:04.181 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba012.stderr:ceph-fuse[154636]: starting fuse 2021-02-10T11:40:04.454 INFO:teuthology.orchestra.run.gibba012.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:40:04.454 INFO:teuthology.orchestra.run.gibba012.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:40:04.454 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:40:04.454 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:40:04.454 DEBUG:teuthology.orchestra.run.gibba012:> ls /sys/fs/fuse/connections 2021-02-10T11:40:04.791 INFO:teuthology.orchestra.run.gibba012.stdout:42 2021-02-10T11:40:04.791 INFO:teuthology.orchestra.run.gibba012.stdout:43 2021-02-10T11:40:04.791 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42, 43] 2021-02-10T11:40:04.791 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:40:04.792 DEBUG:teuthology.orchestra.run.gibba012:> sudo python3 -c ' 2021-02-10T11:40:04.792 DEBUG:teuthology.orchestra.run.gibba012:> import glob 2021-02-10T11:40:04.792 DEBUG:teuthology.orchestra.run.gibba012:> import re 2021-02-10T11:40:04.792 DEBUG:teuthology.orchestra.run.gibba012:> import os 2021-02-10T11:40:04.792 DEBUG:teuthology.orchestra.run.gibba012:> import subprocess 2021-02-10T11:40:04.792 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:40:04.792 DEBUG:teuthology.orchestra.run.gibba012:> def find_socket(client_name): 2021-02-10T11:40:04.792 DEBUG:teuthology.orchestra.run.gibba012:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2021-02-10T11:40:04.792 DEBUG:teuthology.orchestra.run.gibba012:> files = glob.glob(asok_path) 2021-02-10T11:40:04.792 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:40:04.792 DEBUG:teuthology.orchestra.run.gibba012:> # Given a non-glob path, it better be there 2021-02-10T11:40:04.792 DEBUG:teuthology.orchestra.run.gibba012:> if "*" not in asok_path: 2021-02-10T11:40:04.792 DEBUG:teuthology.orchestra.run.gibba012:> assert(len(files) == 1) 2021-02-10T11:40:04.792 DEBUG:teuthology.orchestra.run.gibba012:> return files[0] 2021-02-10T11:40:04.793 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:40:04.793 DEBUG:teuthology.orchestra.run.gibba012:> for f in files: 2021-02-10T11:40:04.793 DEBUG:teuthology.orchestra.run.gibba012:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:40:04.793 DEBUG:teuthology.orchestra.run.gibba012:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:40:04.793 DEBUG:teuthology.orchestra.run.gibba012:> return f 2021-02-10T11:40:04.793 DEBUG:teuthology.orchestra.run.gibba012:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:40:04.793 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:40:04.793 DEBUG:teuthology.orchestra.run.gibba012:> print(find_socket("client.1")) 2021-02-10T11:40:04.793 DEBUG:teuthology.orchestra.run.gibba012:> ' 2021-02-10T11:40:05.155 INFO:teuthology.orchestra.run.gibba012.stdout:/var/run/ceph/ceph-client.1.154636.asok 2021-02-10T11:40:05.158 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.154636.asok 2021-02-10T11:40:05.158 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:40:05.158 DEBUG:teuthology.orchestra.run.gibba012:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.154636.asok status 2021-02-10T11:40:05.549 INFO:teuthology.orchestra.run.gibba012.stdout:{ 2021-02-10T11:40:05.549 INFO:teuthology.orchestra.run.gibba012.stdout: "metadata": { 2021-02-10T11:40:05.549 INFO:teuthology.orchestra.run.gibba012.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:40:05.549 INFO:teuthology.orchestra.run.gibba012.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:40:05.549 INFO:teuthology.orchestra.run.gibba012.stdout: "entity_id": "1", 2021-02-10T11:40:05.549 INFO:teuthology.orchestra.run.gibba012.stdout: "hostname": "gibba012", 2021-02-10T11:40:05.549 INFO:teuthology.orchestra.run.gibba012.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2021-02-10T11:40:05.550 INFO:teuthology.orchestra.run.gibba012.stdout: "pid": "154636", 2021-02-10T11:40:05.550 INFO:teuthology.orchestra.run.gibba012.stdout: "root": "/" 2021-02-10T11:40:05.550 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:40:05.550 INFO:teuthology.orchestra.run.gibba012.stdout: "dentry_count": 0, 2021-02-10T11:40:05.550 INFO:teuthology.orchestra.run.gibba012.stdout: "dentry_pinned_count": 0, 2021-02-10T11:40:05.550 INFO:teuthology.orchestra.run.gibba012.stdout: "id": 10209, 2021-02-10T11:40:05.550 INFO:teuthology.orchestra.run.gibba012.stdout: "inst": { 2021-02-10T11:40:05.550 INFO:teuthology.orchestra.run.gibba012.stdout: "name": { 2021-02-10T11:40:05.550 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "client", 2021-02-10T11:40:05.550 INFO:teuthology.orchestra.run.gibba012.stdout: "num": 10209 2021-02-10T11:40:05.550 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:40:05.550 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": { 2021-02-10T11:40:05.551 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:40:05.551 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:0", 2021-02-10T11:40:05.551 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 4067154506 2021-02-10T11:40:05.551 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:40:05.551 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:40:05.551 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": { 2021-02-10T11:40:05.551 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:40:05.551 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:0", 2021-02-10T11:40:05.551 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 4067154506 2021-02-10T11:40:05.551 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:40:05.551 INFO:teuthology.orchestra.run.gibba012.stdout: "inst_str": "client.10209 172.21.2.112:0/4067154506", 2021-02-10T11:40:05.551 INFO:teuthology.orchestra.run.gibba012.stdout: "addr_str": "172.21.2.112:0/4067154506", 2021-02-10T11:40:05.551 INFO:teuthology.orchestra.run.gibba012.stdout: "inode_count": 1, 2021-02-10T11:40:05.551 INFO:teuthology.orchestra.run.gibba012.stdout: "mds_epoch": 80, 2021-02-10T11:40:05.552 INFO:teuthology.orchestra.run.gibba012.stdout: "osd_epoch": 100, 2021-02-10T11:40:05.552 INFO:teuthology.orchestra.run.gibba012.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:40:05.552 INFO:teuthology.orchestra.run.gibba012.stdout: "blacklisted": false 2021-02-10T11:40:05.552 INFO:teuthology.orchestra.run.gibba012.stdout:} 2021-02-10T11:40:05.553 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:40:05.553 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:40:05.553 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:40:05.891 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:40:05.892 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:40:05.892 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:40:05.892 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:40:06.239 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2021-02-10T11:40:06.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:06.660 7f3db13e9700 1 Processor -- start 2021-02-10T11:40:06.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:06.660 7f3db13e9700 1 -- start start 2021-02-10T11:40:06.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:06.660 7f3db13e9700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3dac0d1880 0x7f3dac1b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:06.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:06.660 7f3db13e9700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f3dac0cff80 0x7f3dac109140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:06.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:06.660 7f3db13e9700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3dac109a70 con 0x7f3dac1b0af0 2021-02-10T11:40:06.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:06.660 7f3db13e9700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3dac109760 con 0x7f3dac0cff80 2021-02-10T11:40:06.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:06.660 7f3db13e9700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3dac088070 con 0x7f3dac0d1880 2021-02-10T11:40:06.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:06.660 7f3daa87c700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7f3dac1b0af0 0x7f3dac108950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:47392/0 (socket says 172.21.2.112:47392) 2021-02-10T11:40:06.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:06.660 7f3dab87e700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f3dac0cff80 0x7f3dac109140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:06.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:06.660 7f3dab07d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3dac0d1880 0x7f3dac1b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:06.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:06.660 7f3daa87c700 1 -- 172.21.2.112:0/1116193819 learned_addr learned my addr 172.21.2.112:0/1116193819 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:40:06.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:06.661 7f3daa07b700 1 -- 172.21.2.112:0/1116193819 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3159537824 0 0) 0x7f3d9c0010c0 con 0x7f3dac1b0af0 2021-02-10T11:40:06.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:06.661 7f3daa07b700 1 -- 172.21.2.112:0/1116193819 --> v1:172.21.2.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d98002fe0 con 0x7f3dac1b0af0 2021-02-10T11:40:06.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:06.661 7f3dab87e700 1 -- 172.21.2.112:0/1116193819 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3dac0d1880 msgr2=0x7f3dac1b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:06.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:06.661 7f3dab87e700 1 --2- 172.21.2.112:0/1116193819 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3dac0d1880 0x7f3dac1b04d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:06.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:06.661 7f3dab87e700 1 -- 172.21.2.112:0/1116193819 >> v1:172.21.2.120:6789/0 conn(0x7f3dac1b0af0 legacy=0x7f3dac108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:06.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:06.661 7f3dab87e700 1 -- 172.21.2.112:0/1116193819 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dac088260 con 0x7f3dac0cff80 2021-02-10T11:40:06.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:06.661 7f3dab07d700 1 --2- 172.21.2.112:0/1116193819 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3dac0d1880 0x7f3dac1b04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:40:06.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:06.661 7f3dab87e700 1 --2- 172.21.2.112:0/1116193819 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3dac0cff80 0x7f3dac109140 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 rx=0x7f3d94006ad0 tx=0x7f3d94009ad0).ready entity=mon.1 client_cookie=ae61fede0dc2063c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:06.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:06.661 7f3daa07b700 1 -- 172.21.2.112:0/1116193819 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3d94006370 con 0x7f3dac0cff80 2021-02-10T11:40:06.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:06.661 7f3daa07b700 1 -- 172.21.2.112:0/1116193819 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d940022e0 con 0x7f3dac0cff80 2021-02-10T11:40:06.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:06.661 7f3db13e9700 1 -- 172.21.2.112:0/1116193819 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3dac0cff80 msgr2=0x7f3dac109140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:06.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:06.661 7f3db13e9700 1 --2- 172.21.2.112:0/1116193819 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3dac0cff80 0x7f3dac109140 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 rx=0x7f3d94006ad0 tx=0x7f3d94009ad0).stop 2021-02-10T11:40:06.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:06.661 7f3daa07b700 1 -- 172.21.2.112:0/1116193819 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3d94002800 con 0x7f3dac0cff80 2021-02-10T11:40:06.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:06.662 7f3db13e9700 1 -- 172.21.2.112:0/1116193819 shutdown_connections 2021-02-10T11:40:06.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:06.662 7f3db13e9700 1 --2- 172.21.2.112:0/1116193819 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3dac0cff80 0x7f3dac109140 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:06.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:06.662 7f3db13e9700 1 --2- 172.21.2.112:0/1116193819 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3dac0d1880 0x7f3dac1b04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:06.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:06.662 7f3db13e9700 1 -- 172.21.2.112:0/1116193819 >> 172.21.2.112:0/1116193819 conn(0x7f3dac0ce430 msgr2=0x7f3dac0d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:06.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:06.662 7f3db13e9700 1 -- 172.21.2.112:0/1116193819 shutdown_connections 2021-02-10T11:40:06.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:06.662 7f3db13e9700 1 -- 172.21.2.112:0/1116193819 wait complete. 2021-02-10T11:40:06.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:06.677 7f3db13e9700 1 Processor -- start 2021-02-10T11:40:06.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:06.677 7f3db13e9700 1 -- start start 2021-02-10T11:40:06.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:06.677 7f3db13e9700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3dac09a690 0x7f3dac09aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:06.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:06.677 7f3db13e9700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3dac1b4160 0x7f3dac1b4560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:06.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:06.677 7f3db13e9700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3dac1b7dd0 0x7f3dac1b81d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:06.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:06.677 7f3db13e9700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dac1b4b80 con 0x7f3dac1b7dd0 2021-02-10T11:40:06.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:06.677 7f3db13e9700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dac1b87f0 con 0x7f3dac09a690 2021-02-10T11:40:06.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:06.677 7f3db13e9700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3dac1b8a10 con 0x7f3dac1b4160 2021-02-10T11:40:06.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:06.678 7f3dab87e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3dac1b7dd0 0x7f3dac1b81d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:06.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:06.678 7f3dab87e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3dac1b7dd0 0x7f3dac1b81d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:54380/0 (socket says 172.21.2.112:54380) 2021-02-10T11:40:06.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:06.678 7f3dab87e700 1 -- 172.21.2.112:0/3086259783 learned_addr learned my addr 172.21.2.112:0/3086259783 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:40:06.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:06.678 7f3daa87c700 1 --2- 172.21.2.112:0/3086259783 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3dac1b4160 0x7f3dac1b4560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:06.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:06.678 7f3dab07d700 1 --2- 172.21.2.112:0/3086259783 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3dac09a690 0x7f3dac09aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:06.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:06.678 7f3dab87e700 1 -- 172.21.2.112:0/3086259783 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3dac1b4160 msgr2=0x7f3dac1b4560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:06.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:06.678 7f3dab87e700 1 --2- 172.21.2.112:0/3086259783 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3dac1b4160 0x7f3dac1b4560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:06.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:06.678 7f3dab87e700 1 -- 172.21.2.112:0/3086259783 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3dac09a690 msgr2=0x7f3dac09aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:06.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:06.678 7f3dab87e700 1 --2- 172.21.2.112:0/3086259783 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3dac09a690 0x7f3dac09aa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:06.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:06.678 7f3dab87e700 1 -- 172.21.2.112:0/3086259783 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dac1b8c30 con 0x7f3dac1b7dd0 2021-02-10T11:40:06.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:06.678 7f3dab87e700 1 --2- 172.21.2.112:0/3086259783 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3dac1b7dd0 0x7f3dac1b81d0 secure :-1 s=READY pgs=2015 cs=0 l=1 rev1=1 rx=0x7f3d940091f0 tx=0x7f3d94002800).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:06.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:06.678 7f3da3fff700 1 -- 172.21.2.112:0/3086259783 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3d94008220 con 0x7f3dac1b7dd0 2021-02-10T11:40:06.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:06.678 7f3da3fff700 1 -- 172.21.2.112:0/3086259783 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d9400a030 con 0x7f3dac1b7dd0 2021-02-10T11:40:06.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:06.678 7f3da3fff700 1 -- 172.21.2.112:0/3086259783 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3d94006370 con 0x7f3dac1b7dd0 2021-02-10T11:40:06.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:06.678 7f3db13e9700 1 -- 172.21.2.112:0/3086259783 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3dac1b8c30 con 0x7f3dac1b7dd0 2021-02-10T11:40:06.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:06.678 7f3db13e9700 1 -- 172.21.2.112:0/3086259783 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3dac1b8c30 con 0x7f3dac1b7dd0 2021-02-10T11:40:06.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:06.679 7f3da3fff700 1 -- 172.21.2.112:0/3086259783 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3d940092e0 con 0x7f3dac1b7dd0 2021-02-10T11:40:06.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:06.679 7f3da3fff700 1 --2- 172.21.2.112:0/3086259783 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3d900186c0 0x7f3d9001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:06.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:06.679 7f3da3fff700 1 -- 172.21.2.112:0/3086259783 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f3d94032400 con 0x7f3dac1b7dd0 2021-02-10T11:40:06.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:06.679 7f3dab07d700 1 --2- 172.21.2.112:0/3086259783 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3d900186c0 0x7f3d9001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:06.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:06.679 7f3dab07d700 1 --2- 172.21.2.112:0/3086259783 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3d900186c0 0x7f3d9001abc0 crc :-1 s=READY pgs=937 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:06.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:06.711 7f3db13e9700 1 -- 172.21.2.112:0/3086259783 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3dac09a160 con 0x7f3dac1b7dd0 2021-02-10T11:40:06.708 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:06.713 7f3da3fff700 1 -- 172.21.2.112:0/3086259783 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3d9402cf00 con 0x7f3dac1b7dd0 2021-02-10T11:40:06.845 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:06.850 7f3db13e9700 1 -- 172.21.2.112:0/3086259783 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f3dac09a160 con 0x7f3dac1b7dd0 2021-02-10T11:40:07.172 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:07.177 7f3da3fff700 1 -- 172.21.2.112:0/3086259783 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v81) v1 ==== 136+0+0 (secure 0 0 0) 0x7f3d9402cf00 con 0x7f3dac1b7dd0 2021-02-10T11:40:07.214 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:07.219 7f3db13e9700 1 -- 172.21.2.112:0/3086259783 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3d900186c0 msgr2=0x7f3d9001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:07.214 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:07.219 7f3db13e9700 1 --2- 172.21.2.112:0/3086259783 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3d900186c0 0x7f3d9001abc0 crc :-1 s=READY pgs=937 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:07.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:07.219 7f3db13e9700 1 -- 172.21.2.112:0/3086259783 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3dac1b7dd0 msgr2=0x7f3dac1b81d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:07.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:07.219 7f3db13e9700 1 --2- 172.21.2.112:0/3086259783 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3dac1b7dd0 0x7f3dac1b81d0 secure :-1 s=READY pgs=2015 cs=0 l=1 rev1=1 rx=0x7f3d940091f0 tx=0x7f3d94002800).stop 2021-02-10T11:40:07.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:07.219 7f3db13e9700 1 -- 172.21.2.112:0/3086259783 shutdown_connections 2021-02-10T11:40:07.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:07.219 7f3db13e9700 1 --2- 172.21.2.112:0/3086259783 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3d900186c0 0x7f3d9001abc0 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:07.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:07.219 7f3db13e9700 1 --2- 172.21.2.112:0/3086259783 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3dac1b7dd0 0x7f3dac1b81d0 unknown :-1 s=CLOSED pgs=2015 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:07.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:07.219 7f3db13e9700 1 --2- 172.21.2.112:0/3086259783 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3dac1b4160 0x7f3dac1b4560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:07.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:07.219 7f3db13e9700 1 --2- 172.21.2.112:0/3086259783 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3dac09a690 0x7f3dac09aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:07.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:07.219 7f3db13e9700 1 -- 172.21.2.112:0/3086259783 >> 172.21.2.112:0/3086259783 conn(0x7f3dac18e870 msgr2=0x7f3dac18ecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:07.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:07.220 7f3db13e9700 1 -- 172.21.2.112:0/3086259783 shutdown_connections 2021-02-10T11:40:07.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:07.220 7f3db13e9700 1 -- 172.21.2.112:0/3086259783 wait complete. 2021-02-10T11:40:07.222 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs volume ls 2021-02-10T11:40:07.649 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:07.638 7fd5be4cb700 1 Processor -- start 2021-02-10T11:40:07.649 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:07.638 7fd5be4cb700 1 -- start start 2021-02-10T11:40:07.649 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:07.638 7fd5be4cb700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fd5b80d0340 0x7fd5b80ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:07.649 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:07.638 7fd5be4cb700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd5b80d1c70 0x7fd5b80efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:07.649 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:07.638 7fd5be4cb700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5b80f0b10 con 0x7fd5b80d1870 2021-02-10T11:40:07.649 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:07.638 7fd5be4cb700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5b80e5ed0 con 0x7fd5b80d0340 2021-02-10T11:40:07.650 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:07.638 7fd5be4cb700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5b81b0be0 con 0x7fd5b80d1c70 2021-02-10T11:40:07.650 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:07.638 7fd5b7fff700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fd5b80d1870 0x7fd5b81b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:53846/0 (socket says 172.21.2.112:53846) 2021-02-10T11:40:07.650 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:07.638 7fd5b7fff700 1 -- 172.21.2.112:0/4051476380 learned_addr learned my addr 172.21.2.112:0/4051476380 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:40:07.650 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:07.639 7fd5bcae8700 1 --2- 172.21.2.112:0/4051476380 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd5b80d1c70 0x7fd5b80efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:07.650 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:07.639 7fd5b77fe700 1 --2- 172.21.2.112:0/4051476380 >> v2:172.21.2.120:3300/0 conn(0x7fd5b80d0340 0x7fd5b80ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:07.650 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:07.639 7fd5b6ffd700 1 -- 172.21.2.112:0/4051476380 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1117077279 0 0) 0x7fd5a8000ce0 con 0x7fd5b80d1870 2021-02-10T11:40:07.650 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:07.639 7fd5b6ffd700 1 -- 172.21.2.112:0/4051476380 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5a0002fe0 con 0x7fd5b80d1870 2021-02-10T11:40:07.650 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:07.639 7fd5b6ffd700 1 -- 172.21.2.112:0/4051476380 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1123353571 0 0) 0x7fd5a80010e0 con 0x7fd5b80d1870 2021-02-10T11:40:07.650 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:07.639 7fd5b77fe700 1 -- 172.21.2.112:0/4051476380 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd5b80d1c70 msgr2=0x7fd5b80efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:07.650 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:07.639 7fd5b77fe700 1 --2- 172.21.2.112:0/4051476380 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd5b80d1c70 0x7fd5b80efc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:07.650 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:07.639 7fd5b77fe700 1 -- 172.21.2.112:0/4051476380 >> v1:172.21.2.112:6789/0 conn(0x7fd5b80d1870 legacy=0x7fd5b81b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:07.650 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:07.639 7fd5b77fe700 1 -- 172.21.2.112:0/4051476380 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5b80f20a0 con 0x7fd5b80d0340 2021-02-10T11:40:07.651 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:07.639 7fd5bcae8700 1 --2- 172.21.2.112:0/4051476380 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd5b80d1c70 0x7fd5b80efc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:40:07.651 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:07.639 7fd5b77fe700 1 --2- 172.21.2.112:0/4051476380 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd5b80d0340 0x7fd5b80ef640 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 rx=0x7fd5ac006650 tx=0x7fd5ac009a80).ready entity=mon.1 client_cookie=6a3375dc08315ac6 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:07.651 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:07.640 7fd5b6ffd700 1 -- 172.21.2.112:0/4051476380 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd5ac006390 con 0x7fd5b80d0340 2021-02-10T11:40:07.651 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:07.640 7fd5b6ffd700 1 -- 172.21.2.112:0/4051476380 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5ac0009b0 con 0x7fd5b80d0340 2021-02-10T11:40:07.651 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:07.640 7fd5b6ffd700 1 -- 172.21.2.112:0/4051476380 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd5ac00c2b0 con 0x7fd5b80d0340 2021-02-10T11:40:07.651 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:07.640 7fd5be4cb700 1 -- 172.21.2.112:0/4051476380 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd5b80d0340 msgr2=0x7fd5b80ef640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:07.651 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:07.640 7fd5be4cb700 1 --2- 172.21.2.112:0/4051476380 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd5b80d0340 0x7fd5b80ef640 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 rx=0x7fd5ac006650 tx=0x7fd5ac009a80).stop 2021-02-10T11:40:07.651 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:07.640 7fd5be4cb700 1 -- 172.21.2.112:0/4051476380 shutdown_connections 2021-02-10T11:40:07.651 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:07.640 7fd5be4cb700 1 --2- 172.21.2.112:0/4051476380 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd5b80d1c70 0x7fd5b80efc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:07.651 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:07.640 7fd5be4cb700 1 --2- 172.21.2.112:0/4051476380 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd5b80d0340 0x7fd5b80ef640 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:07.651 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:07.640 7fd5be4cb700 1 -- 172.21.2.112:0/4051476380 >> 172.21.2.112:0/4051476380 conn(0x7fd5b80ce450 msgr2=0x7fd5b80d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:07.651 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:07.641 7fd5be4cb700 1 -- 172.21.2.112:0/4051476380 shutdown_connections 2021-02-10T11:40:07.651 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:07.641 7fd5be4cb700 1 -- 172.21.2.112:0/4051476380 wait complete. 2021-02-10T11:40:07.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:07.655 7fd5be4cb700 1 Processor -- start 2021-02-10T11:40:07.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:07.655 7fd5be4cb700 1 -- start start 2021-02-10T11:40:07.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:07.655 7fd5be4cb700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd5b819c1d0 0x7fd5b80a2810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:07.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:07.655 7fd5be4cb700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd5b80a2e30 0x7fd5b80a3360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:07.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:07.655 7fd5be4cb700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd5b80a3980 0x7fd5b80a3d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:07.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:07.655 7fd5be4cb700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5b80a4740 con 0x7fd5b80a2e30 2021-02-10T11:40:07.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:07.655 7fd5be4cb700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5b80a43a0 con 0x7fd5b819c1d0 2021-02-10T11:40:07.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:07.655 7fd5be4cb700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5b80a4cc0 con 0x7fd5b80a3980 2021-02-10T11:40:07.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:07.656 7fd5b77fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd5b80a2e30 0x7fd5b80a3360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:07.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:07.656 7fd5b77fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd5b80a2e30 0x7fd5b80a3360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:54392/0 (socket says 172.21.2.112:54392) 2021-02-10T11:40:07.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:07.656 7fd5b77fe700 1 -- 172.21.2.112:0/453069848 learned_addr learned my addr 172.21.2.112:0/453069848 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:40:07.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:07.656 7fd5b7fff700 1 --2- 172.21.2.112:0/453069848 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd5b819c1d0 0x7fd5b80a2810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:07.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:07.656 7fd5bcae8700 1 --2- 172.21.2.112:0/453069848 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd5b80a3980 0x7fd5b80a3d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:07.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:07.656 7fd5b77fe700 1 -- 172.21.2.112:0/453069848 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd5b80a3980 msgr2=0x7fd5b80a3d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:07.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:07.656 7fd5b77fe700 1 --2- 172.21.2.112:0/453069848 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd5b80a3980 0x7fd5b80a3d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:07.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:07.656 7fd5b77fe700 1 -- 172.21.2.112:0/453069848 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd5b819c1d0 msgr2=0x7fd5b80a2810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:07.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:07.656 7fd5b77fe700 1 --2- 172.21.2.112:0/453069848 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd5b819c1d0 0x7fd5b80a2810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:07.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:07.656 7fd5b77fe700 1 -- 172.21.2.112:0/453069848 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5b80a4ee0 con 0x7fd5b80a2e30 2021-02-10T11:40:07.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:07.656 7fd5b77fe700 1 --2- 172.21.2.112:0/453069848 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd5b80a2e30 0x7fd5b80a3360 secure :-1 s=READY pgs=2017 cs=0 l=1 rev1=1 rx=0x7fd5ac006f70 tx=0x7fd5ac00b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:07.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:07.657 7fd5b4ff9700 1 -- 172.21.2.112:0/453069848 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd5ac0064a0 con 0x7fd5b80a2e30 2021-02-10T11:40:07.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:07.657 7fd5b4ff9700 1 -- 172.21.2.112:0/453069848 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5ac00bde0 con 0x7fd5b80a2e30 2021-02-10T11:40:07.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:07.657 7fd5b4ff9700 1 -- 172.21.2.112:0/453069848 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd5ac002030 con 0x7fd5b80a2e30 2021-02-10T11:40:07.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:07.657 7fd5be4cb700 1 -- 172.21.2.112:0/453069848 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5b80a4ee0 con 0x7fd5b80a2e30 2021-02-10T11:40:07.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:07.657 7fd5be4cb700 1 -- 172.21.2.112:0/453069848 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5b80a5470 con 0x7fd5b80a2e30 2021-02-10T11:40:07.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:07.657 7fd5b4ff9700 1 -- 172.21.2.112:0/453069848 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd5ac000c90 con 0x7fd5b80a2e30 2021-02-10T11:40:07.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:07.657 7fd5b4ff9700 1 --2- 172.21.2.112:0/453069848 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd5980186c0 0x7fd59801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:07.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:07.657 7fd5b4ff9700 1 -- 172.21.2.112:0/453069848 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fd5ac032550 con 0x7fd5b80a2e30 2021-02-10T11:40:07.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:07.658 7fd5b7fff700 1 --2- 172.21.2.112:0/453069848 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd5980186c0 0x7fd59801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:07.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:07.659 7fd5b7fff700 1 --2- 172.21.2.112:0/453069848 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd5980186c0 0x7fd59801abc0 crc :-1 s=READY pgs=938 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:07.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:07.688 7fd5be4cb700 1 -- 172.21.2.112:0/453069848 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5b80ebc50 con 0x7fd5b80a2e30 2021-02-10T11:40:07.685 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:07.691 7fd5b4ff9700 1 -- 172.21.2.112:0/453069848 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd5ac014300 con 0x7fd5b80a2e30 2021-02-10T11:40:07.819 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:07.824 7fd5be4cb700 1 -- 172.21.2.112:0/453069848 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs volume ls", "target": ["mgr", ""]}) v1 -- 0x7fd5b80a5470 con 0x7fd5980186c0 2021-02-10T11:40:07.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:07.826 7fd5b4ff9700 1 -- 172.21.2.112:0/453069848 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+41 (crc 0 0 0) 0x7fd5a8007180 con 0x7fd5980186c0 2021-02-10T11:40:07.822 INFO:teuthology.orchestra.run.gibba012.stdout:[ 2021-02-10T11:40:07.822 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:40:07.822 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "cephfs" 2021-02-10T11:40:07.822 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:40:07.822 INFO:teuthology.orchestra.run.gibba012.stdout:] 2021-02-10T11:40:07.824 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:07.829 7fd5be4cb700 1 -- 172.21.2.112:0/453069848 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd5980186c0 msgr2=0x7fd59801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:07.824 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:07.829 7fd5be4cb700 1 --2- 172.21.2.112:0/453069848 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd5980186c0 0x7fd59801abc0 crc :-1 s=READY pgs=938 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:07.824 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:07.829 7fd5be4cb700 1 -- 172.21.2.112:0/453069848 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd5b80a2e30 msgr2=0x7fd5b80a3360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:07.824 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:07.829 7fd5be4cb700 1 --2- 172.21.2.112:0/453069848 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd5b80a2e30 0x7fd5b80a3360 secure :-1 s=READY pgs=2017 cs=0 l=1 rev1=1 rx=0x7fd5ac006f70 tx=0x7fd5ac00b960).stop 2021-02-10T11:40:07.824 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:07.830 7fd5be4cb700 1 -- 172.21.2.112:0/453069848 shutdown_connections 2021-02-10T11:40:07.825 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:07.830 7fd5be4cb700 1 --2- 172.21.2.112:0/453069848 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd5980186c0 0x7fd59801abc0 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:07.825 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:07.830 7fd5be4cb700 1 --2- 172.21.2.112:0/453069848 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd5b80a3980 0x7fd5b80a3d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:07.825 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:07.830 7fd5be4cb700 1 --2- 172.21.2.112:0/453069848 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd5b80a2e30 0x7fd5b80a3360 unknown :-1 s=CLOSED pgs=2017 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:07.825 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:07.830 7fd5be4cb700 1 --2- 172.21.2.112:0/453069848 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd5b819c1d0 0x7fd5b80a2810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:07.825 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:07.830 7fd5be4cb700 1 -- 172.21.2.112:0/453069848 >> 172.21.2.112:0/453069848 conn(0x7fd5b80951b0 msgr2=0x7fd5b808f970 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:07.825 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:07.830 7fd5be4cb700 1 -- 172.21.2.112:0/453069848 shutdown_connections 2021-02-10T11:40:07.825 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:07.830 7fd5be4cb700 1 -- 172.21.2.112:0/453069848 wait complete. 2021-02-10T11:40:07.832 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:40:08.265 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.253 7f8eab0c9700 1 Processor -- start 2021-02-10T11:40:08.265 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.253 7f8eab0c9700 1 -- start start 2021-02-10T11:40:08.265 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.254 7f8eab0c9700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f8ea40d1870 0x7f8ea41b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:08.265 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.254 7f8eab0c9700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f8ea41b0ae0 0x7f8ea4108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:08.265 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.254 7f8eab0c9700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ea40fe1c0 con 0x7f8ea40d1c70 2021-02-10T11:40:08.265 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.254 7f8eab0c9700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ea4109a60 con 0x7f8ea40d1870 2021-02-10T11:40:08.265 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.254 7f8eab0c9700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ea4109750 con 0x7f8ea41b0ae0 2021-02-10T11:40:08.265 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.254 7f8ea96e6700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f8ea40d1c70 0x7f8ea4108f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:53858/0 (socket says 172.21.2.112:53858) 2021-02-10T11:40:08.265 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.254 7f8ea96e6700 1 -- 172.21.2.112:0/723744459 learned_addr learned my addr 172.21.2.112:0/723744459 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:40:08.265 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.254 7f8ea8ee5700 1 --2- 172.21.2.112:0/723744459 >> v2:172.21.2.112:3300/0 conn(0x7f8ea40d1870 0x7f8ea41b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:08.266 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.254 7f8ea3fff700 1 --2- 172.21.2.112:0/723744459 >> v2:172.21.2.120:3300/0 conn(0x7f8ea41b0ae0 0x7f8ea4108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:08.266 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.254 7f8ea37fe700 1 -- 172.21.2.112:0/723744459 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1973311654 0 0) 0x7f8e9c001500 con 0x7f8ea40d1c70 2021-02-10T11:40:08.266 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.254 7f8ea37fe700 1 -- 172.21.2.112:0/723744459 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e90002fe0 con 0x7f8ea40d1c70 2021-02-10T11:40:08.266 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.254 7f8ea8ee5700 1 -- 172.21.2.112:0/723744459 >> v2:172.21.2.120:3300/0 conn(0x7f8ea41b0ae0 msgr2=0x7f8ea4108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:08.266 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.254 7f8ea8ee5700 1 --2- 172.21.2.112:0/723744459 >> v2:172.21.2.120:3300/0 conn(0x7f8ea41b0ae0 0x7f8ea4108940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:08.266 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.254 7f8ea8ee5700 1 -- 172.21.2.112:0/723744459 >> v1:172.21.2.112:6789/0 conn(0x7f8ea40d1c70 legacy=0x7f8ea4108f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:08.266 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.254 7f8ea8ee5700 1 -- 172.21.2.112:0/723744459 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ea4088070 con 0x7f8ea40d1870 2021-02-10T11:40:08.266 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.255 7f8ea8ee5700 1 --2- 172.21.2.112:0/723744459 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8ea40d1870 0x7f8ea41b04c0 secure :-1 s=READY pgs=2019 cs=0 l=1 rev1=1 rx=0x7f8e8c006650 tx=0x7f8e8c009a80).ready entity=mon.0 client_cookie=d2eb20b1c9b5f24b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:08.266 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.255 7f8ea37fe700 1 -- 172.21.2.112:0/723744459 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8e8c00c0c0 con 0x7f8ea40d1870 2021-02-10T11:40:08.266 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.255 7f8ea37fe700 1 -- 172.21.2.112:0/723744459 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e8c00c340 con 0x7f8ea40d1870 2021-02-10T11:40:08.266 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.255 7f8ea37fe700 1 -- 172.21.2.112:0/723744459 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8e8c00c860 con 0x7f8ea40d1870 2021-02-10T11:40:08.266 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.255 7f8eab0c9700 1 -- 172.21.2.112:0/723744459 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8ea40d1870 msgr2=0x7f8ea41b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:08.267 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.255 7f8eab0c9700 1 --2- 172.21.2.112:0/723744459 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8ea40d1870 0x7f8ea41b04c0 secure :-1 s=READY pgs=2019 cs=0 l=1 rev1=1 rx=0x7f8e8c006650 tx=0x7f8e8c009a80).stop 2021-02-10T11:40:08.267 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.256 7f8eab0c9700 1 -- 172.21.2.112:0/723744459 shutdown_connections 2021-02-10T11:40:08.267 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.256 7f8eab0c9700 1 --2- 172.21.2.112:0/723744459 >> v2:172.21.2.120:3300/0 conn(0x7f8ea41b0ae0 0x7f8ea4108940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:08.267 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.256 7f8eab0c9700 1 --2- 172.21.2.112:0/723744459 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8ea40d1870 0x7f8ea41b04c0 unknown :-1 s=CLOSED pgs=2019 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:08.267 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.256 7f8eab0c9700 1 -- 172.21.2.112:0/723744459 >> 172.21.2.112:0/723744459 conn(0x7f8ea40ce420 msgr2=0x7f8ea40d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:08.267 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.256 7f8eab0c9700 1 -- 172.21.2.112:0/723744459 shutdown_connections 2021-02-10T11:40:08.267 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.256 7f8eab0c9700 1 -- 172.21.2.112:0/723744459 wait complete. 2021-02-10T11:40:08.268 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.270 7f8eab0c9700 1 Processor -- start 2021-02-10T11:40:08.268 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.270 7f8eab0c9700 1 -- start start 2021-02-10T11:40:08.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.271 7f8eab0c9700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8ea4085bc0 0x7f8ea40863e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:08.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.271 7f8eab0c9700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8ea40954c0 0x7f8ea4096300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:08.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.271 7f8eab0c9700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8ea40958c0 0x7f8ea4096920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:08.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.271 7f8eab0c9700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea4095d80 con 0x7f8ea40954c0 2021-02-10T11:40:08.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.271 7f8eab0c9700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea40bfa60 con 0x7f8ea4085bc0 2021-02-10T11:40:08.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.271 7f8eab0c9700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea4085fc0 con 0x7f8ea40958c0 2021-02-10T11:40:08.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.271 7f8ea3fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8ea40954c0 0x7f8ea4096300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:08.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.271 7f8ea3fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8ea40954c0 0x7f8ea4096300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:54400/0 (socket says 172.21.2.112:54400) 2021-02-10T11:40:08.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.271 7f8ea3fff700 1 -- 172.21.2.112:0/1804590138 learned_addr learned my addr 172.21.2.112:0/1804590138 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:40:08.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.271 7f8ea8ee5700 1 --2- 172.21.2.112:0/1804590138 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8ea4085bc0 0x7f8ea40863e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:08.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.271 7f8ea96e6700 1 --2- 172.21.2.112:0/1804590138 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8ea40958c0 0x7f8ea4096920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:08.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.271 7f8ea3fff700 1 -- 172.21.2.112:0/1804590138 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8ea40958c0 msgr2=0x7f8ea4096920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:08.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.271 7f8ea3fff700 1 --2- 172.21.2.112:0/1804590138 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8ea40958c0 0x7f8ea4096920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:08.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.271 7f8ea3fff700 1 -- 172.21.2.112:0/1804590138 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8ea4085bc0 msgr2=0x7f8ea40863e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:08.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.271 7f8ea3fff700 1 --2- 172.21.2.112:0/1804590138 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8ea4085bc0 0x7f8ea40863e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:08.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.271 7f8ea3fff700 1 -- 172.21.2.112:0/1804590138 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ea4086c10 con 0x7f8ea40954c0 2021-02-10T11:40:08.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.271 7f8ea8ee5700 1 --2- 172.21.2.112:0/1804590138 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8ea4085bc0 0x7f8ea40863e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:40:08.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.271 7f8ea3fff700 1 --2- 172.21.2.112:0/1804590138 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8ea40954c0 0x7f8ea4096300 secure :-1 s=READY pgs=2020 cs=0 l=1 rev1=1 rx=0x7f8e94006f60 tx=0x7f8e9400b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:08.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.272 7f8ea17fa700 1 -- 172.21.2.112:0/1804590138 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8e940009b0 con 0x7f8ea40954c0 2021-02-10T11:40:08.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.272 7f8ea17fa700 1 -- 172.21.2.112:0/1804590138 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e9400bdf0 con 0x7f8ea40954c0 2021-02-10T11:40:08.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.272 7f8eab0c9700 1 -- 172.21.2.112:0/1804590138 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ea4086c10 con 0x7f8ea40954c0 2021-02-10T11:40:08.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.272 7f8ea17fa700 1 -- 172.21.2.112:0/1804590138 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8e94002030 con 0x7f8ea40954c0 2021-02-10T11:40:08.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.272 7f8eab0c9700 1 -- 172.21.2.112:0/1804590138 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ea4097230 con 0x7f8ea40954c0 2021-02-10T11:40:08.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.272 7f8ea17fa700 1 -- 172.21.2.112:0/1804590138 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8e94006b00 con 0x7f8ea40954c0 2021-02-10T11:40:08.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.272 7f8ea17fa700 1 --2- 172.21.2.112:0/1804590138 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8e880186c0 0x7f8e8801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:08.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.272 7f8ea17fa700 1 -- 172.21.2.112:0/1804590138 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f8e94032570 con 0x7f8ea40954c0 2021-02-10T11:40:08.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.272 7f8ea8ee5700 1 --2- 172.21.2.112:0/1804590138 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8e880186c0 0x7f8e8801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:08.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.273 7f8ea8ee5700 1 --2- 172.21.2.112:0/1804590138 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8e880186c0 0x7f8e8801abc0 crc :-1 s=READY pgs=939 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:08.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.304 7f8eab0c9700 1 -- 172.21.2.112:0/1804590138 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ea40fe1c0 con 0x7f8ea40954c0 2021-02-10T11:40:08.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.306 7f8ea17fa700 1 -- 172.21.2.112:0/1804590138 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8e940009b0 con 0x7f8ea40954c0 2021-02-10T11:40:08.452 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.457 7f8eab0c9700 1 -- 172.21.2.112:0/1804590138 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f8ea40fe1c0 con 0x7f8ea40954c0 2021-02-10T11:40:08.452 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.457 7f8ea17fa700 1 -- 172.21.2.112:0/1804590138 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v26)=0 v26) v1 ==== 122+0+0 (secure 0 0 0) 0x7f8e94006b00 con 0x7f8ea40954c0 2021-02-10T11:40:08.454 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.459 7f8eab0c9700 1 -- 172.21.2.112:0/1804590138 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8e880186c0 msgr2=0x7f8e8801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:08.454 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.459 7f8eab0c9700 1 --2- 172.21.2.112:0/1804590138 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8e880186c0 0x7f8e8801abc0 crc :-1 s=READY pgs=939 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:08.454 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.459 7f8eab0c9700 1 -- 172.21.2.112:0/1804590138 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8ea40954c0 msgr2=0x7f8ea4096300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:08.454 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.459 7f8eab0c9700 1 --2- 172.21.2.112:0/1804590138 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8ea40954c0 0x7f8ea4096300 secure :-1 s=READY pgs=2020 cs=0 l=1 rev1=1 rx=0x7f8e94006f60 tx=0x7f8e9400b970).stop 2021-02-10T11:40:08.455 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.459 7f8eab0c9700 1 -- 172.21.2.112:0/1804590138 shutdown_connections 2021-02-10T11:40:08.455 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.459 7f8eab0c9700 1 --2- 172.21.2.112:0/1804590138 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8e880186c0 0x7f8e8801abc0 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:08.455 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.459 7f8eab0c9700 1 --2- 172.21.2.112:0/1804590138 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8ea40958c0 0x7f8ea4096920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:08.455 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.459 7f8eab0c9700 1 --2- 172.21.2.112:0/1804590138 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8ea40954c0 0x7f8ea4096300 unknown :-1 s=CLOSED pgs=2020 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:08.455 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.459 7f8eab0c9700 1 --2- 172.21.2.112:0/1804590138 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8ea4085bc0 0x7f8ea40863e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:08.455 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.459 7f8eab0c9700 1 -- 172.21.2.112:0/1804590138 >> 172.21.2.112:0/1804590138 conn(0x7f8ea40bef00 msgr2=0x7f8ea409d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:08.455 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.460 7f8eab0c9700 1 -- 172.21.2.112:0/1804590138 shutdown_connections 2021-02-10T11:40:08.455 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.460 7f8eab0c9700 1 -- 172.21.2.112:0/1804590138 wait complete. 2021-02-10T11:40:08.463 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000119244 2021-02-10T11:40:08.894 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.882 7f7fe7639700 1 Processor -- start 2021-02-10T11:40:08.894 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.882 7f7fe7639700 1 -- start start 2021-02-10T11:40:08.894 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.883 7f7fe7639700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7fe00d1c70 0x7f7fe00cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:08.894 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.883 7f7fe7639700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f7fe00cf580 0x7f7fe00cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:08.894 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.883 7f7fe7639700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fe00fe1c0 con 0x7f7fe00d1870 2021-02-10T11:40:08.894 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.883 7f7fe7639700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fe01b0cb0 con 0x7f7fe00cf580 2021-02-10T11:40:08.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.883 7f7fe7639700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fe0109a60 con 0x7f7fe00d1c70 2021-02-10T11:40:08.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.884 7f7fe5c56700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f7fe00cf580 0x7f7fe00cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:08.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.884 7f7fe4c54700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7fe00d1c70 0x7f7fe00cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:08.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.884 7f7fe5455700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7f7fe00d1870 0x7f7fe01b04c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:47428/0 (socket says 172.21.2.112:47428) 2021-02-10T11:40:08.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.884 7f7fe5455700 1 -- 172.21.2.112:0/2511968395 learned_addr learned my addr 172.21.2.112:0/2511968395 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:40:08.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.884 7f7fe5c56700 1 -- 172.21.2.112:0/2511968395 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7fe00d1c70 msgr2=0x7f7fe00cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:08.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.884 7f7fe5c56700 1 --2- 172.21.2.112:0/2511968395 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7fe00d1c70 0x7f7fe00cef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:08.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.884 7f7fe5c56700 1 -- 172.21.2.112:0/2511968395 >> v1:172.21.2.120:6789/0 conn(0x7f7fe00d1870 legacy=0x7f7fe01b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:08.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.884 7f7fe5c56700 1 -- 172.21.2.112:0/2511968395 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fe0088070 con 0x7f7fe00cf580 2021-02-10T11:40:08.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.884 7f7fe4c54700 1 --2- 172.21.2.112:0/2511968395 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7fe00d1c70 0x7f7fe00cef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:40:08.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.884 7f7fe5c56700 1 --2- 172.21.2.112:0/2511968395 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7fe00cf580 0x7f7fe00cf980 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 rx=0x7f7fd8006e50 tx=0x7f7fd8009e70).ready entity=mon.1 client_cookie=3cd1921652567fb0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:08.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.885 7f7fd7fff700 1 -- 172.21.2.112:0/2511968395 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7fd8000f90 con 0x7f7fe00cf580 2021-02-10T11:40:08.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.885 7f7fd7fff700 1 -- 172.21.2.112:0/2511968395 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fd80066f0 con 0x7f7fe00cf580 2021-02-10T11:40:08.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.885 7f7fd7fff700 1 -- 172.21.2.112:0/2511968395 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7fd800b700 con 0x7f7fe00cf580 2021-02-10T11:40:08.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.885 7f7fe7639700 1 -- 172.21.2.112:0/2511968395 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7fe00cf580 msgr2=0x7f7fe00cf980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:08.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.885 7f7fe7639700 1 --2- 172.21.2.112:0/2511968395 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7fe00cf580 0x7f7fe00cf980 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 rx=0x7f7fd8006e50 tx=0x7f7fd8009e70).stop 2021-02-10T11:40:08.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.885 7f7fe7639700 1 -- 172.21.2.112:0/2511968395 shutdown_connections 2021-02-10T11:40:08.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.885 7f7fe7639700 1 --2- 172.21.2.112:0/2511968395 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7fe00cf580 0x7f7fe00cf980 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:08.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.885 7f7fe7639700 1 --2- 172.21.2.112:0/2511968395 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7fe00d1c70 0x7f7fe00cef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:08.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.885 7f7fe7639700 1 -- 172.21.2.112:0/2511968395 >> 172.21.2.112:0/2511968395 conn(0x7f7fe00ce420 msgr2=0x7f7fe00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:08.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.886 7f7fe7639700 1 -- 172.21.2.112:0/2511968395 shutdown_connections 2021-02-10T11:40:08.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.886 7f7fe7639700 1 -- 172.21.2.112:0/2511968395 wait complete. 2021-02-10T11:40:08.897 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.900 7f7fe7639700 1 Processor -- start 2021-02-10T11:40:08.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.900 7f7fe7639700 1 -- start start 2021-02-10T11:40:08.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.900 7f7fe7639700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7fe0191b10 0x7f7fe00ae640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:08.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.900 7f7fe7639700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7fe0191f10 0x7f7fe019ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:08.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.900 7f7fe7639700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7fe01a05f0 0x7f7fe0082ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:08.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.900 7f7fe7639700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fe0192440 con 0x7f7fe01a05f0 2021-02-10T11:40:08.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.900 7f7fe7639700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fe01916e0 con 0x7f7fe0191b10 2021-02-10T11:40:08.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.900 7f7fe7639700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fe0191900 con 0x7f7fe0191f10 2021-02-10T11:40:08.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.901 7f7fe5c56700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7fe01a05f0 0x7f7fe0082ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:08.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.901 7f7fe5c56700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7fe01a05f0 0x7f7fe0082ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:54416/0 (socket says 172.21.2.112:54416) 2021-02-10T11:40:08.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.901 7f7fe5c56700 1 -- 172.21.2.112:0/2439663726 learned_addr learned my addr 172.21.2.112:0/2439663726 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:40:08.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.901 7f7fe5c56700 1 -- 172.21.2.112:0/2439663726 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7fe0191f10 msgr2=0x7f7fe019ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:08.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.901 7f7fe5c56700 1 --2- 172.21.2.112:0/2439663726 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7fe0191f10 0x7f7fe019ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:08.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.901 7f7fe5c56700 1 -- 172.21.2.112:0/2439663726 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7fe0191b10 msgr2=0x7f7fe00ae640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:08.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.901 7f7fe5c56700 1 --2- 172.21.2.112:0/2439663726 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7fe0191b10 0x7f7fe00ae640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:08.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.901 7f7fe5c56700 1 -- 172.21.2.112:0/2439663726 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fe00834f0 con 0x7f7fe01a05f0 2021-02-10T11:40:08.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.901 7f7fe5c56700 1 --2- 172.21.2.112:0/2439663726 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7fe01a05f0 0x7f7fe0082ed0 secure :-1 s=READY pgs=2021 cs=0 l=1 rev1=1 rx=0x7f7fd80099b0 tx=0x7f7fd800bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:08.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.901 7f7fd5ffb700 1 -- 172.21.2.112:0/2439663726 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7fd8002030 con 0x7f7fe01a05f0 2021-02-10T11:40:08.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.901 7f7fd5ffb700 1 -- 172.21.2.112:0/2439663726 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fd8002370 con 0x7f7fe01a05f0 2021-02-10T11:40:08.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.901 7f7fd5ffb700 1 -- 172.21.2.112:0/2439663726 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7fd8002890 con 0x7f7fe01a05f0 2021-02-10T11:40:08.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.901 7f7fe7639700 1 -- 172.21.2.112:0/2439663726 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fe00834f0 con 0x7f7fe01a05f0 2021-02-10T11:40:08.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.901 7f7fe7639700 1 -- 172.21.2.112:0/2439663726 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fe0083a80 con 0x7f7fe01a05f0 2021-02-10T11:40:08.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.902 7f7fd5ffb700 1 -- 172.21.2.112:0/2439663726 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7fd8002550 con 0x7f7fe01a05f0 2021-02-10T11:40:08.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.902 7f7fd5ffb700 1 --2- 172.21.2.112:0/2439663726 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7fc00186c0 0x7f7fc001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:08.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.902 7f7fd5ffb700 1 -- 172.21.2.112:0/2439663726 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f7fd8032b20 con 0x7f7fe01a05f0 2021-02-10T11:40:08.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.903 7f7fe5455700 1 --2- 172.21.2.112:0/2439663726 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7fc00186c0 0x7f7fc001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:08.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.903 7f7fe5455700 1 --2- 172.21.2.112:0/2439663726 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7fc00186c0 0x7f7fc001abc0 crc :-1 s=READY pgs=940 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:08.927 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.933 7f7fe7639700 1 -- 172.21.2.112:0/2439663726 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fe0192440 con 0x7f7fe01a05f0 2021-02-10T11:40:08.930 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:08.935 7f7fd5ffb700 1 -- 172.21.2.112:0/2439663726 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7fd800be10 con 0x7f7fe01a05f0 2021-02-10T11:40:09.056 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:09.062 7f7fe7639700 1 -- 172.21.2.112:0/2439663726 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000119244", "target": ["mgr", ""]}) v1 -- 0x7f7fe0083a80 con 0x7f7fc00186c0 2021-02-10T11:40:09.060 INFO:tasks.ceph.mgr.x.gibba020.stderr:2021-02-10 11:40:09.064 7f6382eee700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:40:09.061 INFO:tasks.ceph.mgr.x.gibba020.stderr:2021-02-10 11:40:09.064 7f6382eee700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:40:09.061 INFO:tasks.ceph.mgr.x.gibba020.stderr:2021-02-10 11:40:09.064 7f6382eee700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:40:09.061 INFO:tasks.ceph.mgr.x.gibba020.stderr:2021-02-10 11:40:09.064 7f6382eee700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:40:09.061 INFO:tasks.ceph.mgr.x.gibba020.stderr:2021-02-10 11:40:09.064 7f6382eee700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:40:09.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:09.098 7f7fd5ffb700 1 -- 172.21.2.112:0/2439663726 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f7fdc000e20 con 0x7f7fc00186c0 2021-02-10T11:40:09.122 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:09.127 7f7fe7639700 1 -- 172.21.2.112:0/2439663726 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7fc00186c0 msgr2=0x7f7fc001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:09.122 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:09.127 7f7fe7639700 1 --2- 172.21.2.112:0/2439663726 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7fc00186c0 0x7f7fc001abc0 crc :-1 s=READY pgs=940 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:09.122 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:09.127 7f7fe7639700 1 -- 172.21.2.112:0/2439663726 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7fe01a05f0 msgr2=0x7f7fe0082ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:09.122 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:09.127 7f7fe7639700 1 --2- 172.21.2.112:0/2439663726 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7fe01a05f0 0x7f7fe0082ed0 secure :-1 s=READY pgs=2021 cs=0 l=1 rev1=1 rx=0x7f7fd80099b0 tx=0x7f7fd800bcd0).stop 2021-02-10T11:40:09.123 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:09.127 7f7fe7639700 1 -- 172.21.2.112:0/2439663726 shutdown_connections 2021-02-10T11:40:09.123 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:09.127 7f7fe7639700 1 --2- 172.21.2.112:0/2439663726 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7fc00186c0 0x7f7fc001abc0 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:09.123 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:09.127 7f7fe7639700 1 --2- 172.21.2.112:0/2439663726 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7fe01a05f0 0x7f7fe0082ed0 unknown :-1 s=CLOSED pgs=2021 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:09.123 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:09.127 7f7fe7639700 1 --2- 172.21.2.112:0/2439663726 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7fe0191f10 0x7f7fe019ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:09.123 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:09.127 7f7fe7639700 1 --2- 172.21.2.112:0/2439663726 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7fe0191b10 0x7f7fe00ae640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:09.123 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:09.127 7f7fe7639700 1 -- 172.21.2.112:0/2439663726 >> 172.21.2.112:0/2439663726 conn(0x7f7fe00a02d0 msgr2=0x7f7fe009f200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:09.123 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:09.128 7f7fe7639700 1 -- 172.21.2.112:0/2439663726 shutdown_connections 2021-02-10T11:40:09.123 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:09.128 7f7fe7639700 1 -- 172.21.2.112:0/2439663726 wait complete. 2021-02-10T11:40:09.130 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000119244 2021-02-10T11:40:09.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:09.550 7f6f3c8f0700 1 Processor -- start 2021-02-10T11:40:09.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:09.550 7f6f3c8f0700 1 -- start start 2021-02-10T11:40:09.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:09.551 7f6f3c8f0700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f6f380d1c70 0x7f6f380cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:09.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:09.551 7f6f3c8f0700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f380d0170 con 0x7f6f380d1870 2021-02-10T11:40:09.563 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:09.551 7f6f3c8f0700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f380fe1c0 con 0x7f6f380cf580 2021-02-10T11:40:09.563 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:09.551 7f6f3c8f0700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f381b0cb0 con 0x7f6f380d1c70 2021-02-10T11:40:09.563 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:09.551 7f6f3661c700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f6f380d1870 0x7f6f381b04c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:53884/0 (socket says 172.21.2.112:53884) 2021-02-10T11:40:09.563 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:09.551 7f6f3661c700 1 -- 172.21.2.112:0/3013264243 learned_addr learned my addr 172.21.2.112:0/3013264243 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:40:09.563 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:09.552 7f6f35e1b700 1 --2- 172.21.2.112:0/3013264243 >> v2:172.21.2.120:3300/0 conn(0x7f6f380d1c70 0x7f6f380cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:09.563 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:09.552 7f6f3561a700 1 -- 172.21.2.112:0/3013264243 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 269848207 0 0) 0x7f6f20001180 con 0x7f6f380d1870 2021-02-10T11:40:09.563 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:09.552 7f6f3561a700 1 -- 172.21.2.112:0/3013264243 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f24002fe0 con 0x7f6f380d1870 2021-02-10T11:40:09.564 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:09.552 7f6f3561a700 1 -- 172.21.2.112:0/3013264243 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1112152015 0 0) 0x7f6f30001400 con 0x7f6f380cf580 2021-02-10T11:40:09.564 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:09.552 7f6f3561a700 1 -- 172.21.2.112:0/3013264243 --> v1:172.21.2.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f24003210 con 0x7f6f380cf580 2021-02-10T11:40:09.564 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:09.552 7f6f35e1b700 1 -- 172.21.2.112:0/3013264243 >> v1:172.21.2.120:6789/0 conn(0x7f6f380cf580 legacy=0x7f6f380cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:09.564 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:09.552 7f6f35e1b700 1 -- 172.21.2.112:0/3013264243 >> v1:172.21.2.112:6789/0 conn(0x7f6f380d1870 legacy=0x7f6f381b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:09.564 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:09.552 7f6f35e1b700 1 -- 172.21.2.112:0/3013264243 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f38109a60 con 0x7f6f380d1c70 2021-02-10T11:40:09.564 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:09.552 7f6f35e1b700 1 --2- 172.21.2.112:0/3013264243 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6f380d1c70 0x7f6f380cef60 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 rx=0x7f6f28006650 tx=0x7f6f28009a80).ready entity=mon.1 client_cookie=402d4fa5bcdbfec4 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:09.564 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:09.553 7f6f3561a700 1 -- 172.21.2.112:0/3013264243 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6f2800c0c0 con 0x7f6f380d1c70 2021-02-10T11:40:09.564 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:09.553 7f6f3561a700 1 -- 172.21.2.112:0/3013264243 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f2800c340 con 0x7f6f380d1c70 2021-02-10T11:40:09.564 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:09.553 7f6f3561a700 1 -- 172.21.2.112:0/3013264243 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6f2800c860 con 0x7f6f380d1c70 2021-02-10T11:40:09.564 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:09.553 7f6f3c8f0700 1 -- 172.21.2.112:0/3013264243 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6f380d1c70 msgr2=0x7f6f380cef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:09.564 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:09.553 7f6f3c8f0700 1 --2- 172.21.2.112:0/3013264243 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6f380d1c70 0x7f6f380cef60 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 rx=0x7f6f28006650 tx=0x7f6f28009a80).stop 2021-02-10T11:40:09.565 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:09.553 7f6f3c8f0700 1 -- 172.21.2.112:0/3013264243 shutdown_connections 2021-02-10T11:40:09.565 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:09.553 7f6f3c8f0700 1 --2- 172.21.2.112:0/3013264243 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6f380d1c70 0x7f6f380cef60 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:09.565 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:09.553 7f6f3c8f0700 1 -- 172.21.2.112:0/3013264243 >> 172.21.2.112:0/3013264243 conn(0x7f6f380ce420 msgr2=0x7f6f380d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:09.565 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:09.554 7f6f3c8f0700 1 -- 172.21.2.112:0/3013264243 shutdown_connections 2021-02-10T11:40:09.565 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:09.554 7f6f3c8f0700 1 -- 172.21.2.112:0/3013264243 wait complete. 2021-02-10T11:40:09.566 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:09.568 7f6f3c8f0700 1 Processor -- start 2021-02-10T11:40:09.566 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:09.568 7f6f3c8f0700 1 -- start start 2021-02-10T11:40:09.566 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:09.568 7f6f3c8f0700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6f380a7950 0x7f6f380b8df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:09.566 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:09.568 7f6f3c8f0700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6f381a7490 0x7f6f381a7890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:09.567 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:09.568 7f6f3c8f0700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6f381a7eb0 0x7f6f380ac630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:09.567 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:09.568 7f6f3c8f0700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f380a7fb0 con 0x7f6f381a7eb0 2021-02-10T11:40:09.567 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:09.568 7f6f3c8f0700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f380acc50 con 0x7f6f380a7950 2021-02-10T11:40:09.567 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:09.568 7f6f3c8f0700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f380ace30 con 0x7f6f381a7490 2021-02-10T11:40:09.567 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:09.569 7f6f36e1d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6f381a7eb0 0x7f6f380ac630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:09.567 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:09.569 7f6f36e1d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6f381a7eb0 0x7f6f380ac630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:54430/0 (socket says 172.21.2.112:54430) 2021-02-10T11:40:09.567 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:09.569 7f6f36e1d700 1 -- 172.21.2.112:0/1459381036 learned_addr learned my addr 172.21.2.112:0/1459381036 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:40:09.567 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:09.569 7f6f36e1d700 1 -- 172.21.2.112:0/1459381036 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6f381a7490 msgr2=0x7f6f381a7890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:09.567 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:09.569 7f6f36e1d700 1 --2- 172.21.2.112:0/1459381036 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6f381a7490 0x7f6f381a7890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:09.567 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:09.569 7f6f36e1d700 1 -- 172.21.2.112:0/1459381036 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6f380a7950 msgr2=0x7f6f380b8df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:09.567 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:09.569 7f6f36e1d700 1 --2- 172.21.2.112:0/1459381036 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6f380a7950 0x7f6f380b8df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:09.567 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:09.569 7f6f36e1d700 1 -- 172.21.2.112:0/1459381036 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f380ad050 con 0x7f6f381a7eb0 2021-02-10T11:40:09.567 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:09.569 7f6f36e1d700 1 --2- 172.21.2.112:0/1459381036 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6f381a7eb0 0x7f6f380ac630 secure :-1 s=READY pgs=2024 cs=0 l=1 rev1=1 rx=0x7f6f30009940 tx=0x7f6f3000bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:09.568 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:09.569 7f6f2effd700 1 -- 172.21.2.112:0/1459381036 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6f3000c2f0 con 0x7f6f381a7eb0 2021-02-10T11:40:09.568 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:09.569 7f6f2effd700 1 -- 172.21.2.112:0/1459381036 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f3000c570 con 0x7f6f381a7eb0 2021-02-10T11:40:09.568 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:09.569 7f6f2effd700 1 -- 172.21.2.112:0/1459381036 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6f3000c9d0 con 0x7f6f381a7eb0 2021-02-10T11:40:09.568 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:09.569 7f6f3c8f0700 1 -- 172.21.2.112:0/1459381036 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f380ad050 con 0x7f6f381a7eb0 2021-02-10T11:40:09.568 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:09.569 7f6f3c8f0700 1 -- 172.21.2.112:0/1459381036 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f380ad050 con 0x7f6f381a7eb0 2021-02-10T11:40:09.568 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:09.570 7f6f2effd700 1 -- 172.21.2.112:0/1459381036 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6f30002030 con 0x7f6f381a7eb0 2021-02-10T11:40:09.568 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:09.570 7f6f2effd700 1 --2- 172.21.2.112:0/1459381036 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6f1c0186c0 0x7f6f1c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:09.568 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:09.570 7f6f2effd700 1 -- 172.21.2.112:0/1459381036 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f6f30032940 con 0x7f6f381a7eb0 2021-02-10T11:40:09.572 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:09.577 7f6f3661c700 1 --2- 172.21.2.112:0/1459381036 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6f1c0186c0 0x7f6f1c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:09.573 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:09.579 7f6f3661c700 1 --2- 172.21.2.112:0/1459381036 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6f1c0186c0 0x7f6f1c01abc0 crc :-1 s=READY pgs=941 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:09.596 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:09.601 7f6f3c8f0700 1 -- 172.21.2.112:0/1459381036 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f38082a70 con 0x7f6f381a7eb0 2021-02-10T11:40:09.598 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:09.604 7f6f2effd700 1 -- 172.21.2.112:0/1459381036 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6f3000be30 con 0x7f6f381a7eb0 2021-02-10T11:40:09.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:09.734 7f6f3c8f0700 1 -- 172.21.2.112:0/1459381036 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000119244", "target": ["mgr", ""]}) v1 -- 0x7f6f38082a70 con 0x7f6f1c0186c0 2021-02-10T11:40:09.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:09.737 7f6f2effd700 1 -- 172.21.2.112:0/1459381036 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f6f20007780 con 0x7f6f1c0186c0 2021-02-10T11:40:09.736 INFO:teuthology.orchestra.run.gibba012.stdout:/volumes/_nogroup/subvolume_0000000000119244/c8e1f5a0-c998-4131-a8be-b62f10f5cee0 2021-02-10T11:40:09.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:09.743 7f6f3c8f0700 1 -- 172.21.2.112:0/1459381036 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6f1c0186c0 msgr2=0x7f6f1c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:09.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:09.743 7f6f3c8f0700 1 --2- 172.21.2.112:0/1459381036 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6f1c0186c0 0x7f6f1c01abc0 crc :-1 s=READY pgs=941 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:09.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:09.743 7f6f3c8f0700 1 -- 172.21.2.112:0/1459381036 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6f381a7eb0 msgr2=0x7f6f380ac630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:09.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:09.743 7f6f3c8f0700 1 --2- 172.21.2.112:0/1459381036 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6f381a7eb0 0x7f6f380ac630 secure :-1 s=READY pgs=2024 cs=0 l=1 rev1=1 rx=0x7f6f30009940 tx=0x7f6f3000bcf0).stop 2021-02-10T11:40:09.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:09.743 7f6f3c8f0700 1 -- 172.21.2.112:0/1459381036 shutdown_connections 2021-02-10T11:40:09.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:09.743 7f6f3c8f0700 1 --2- 172.21.2.112:0/1459381036 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6f1c0186c0 0x7f6f1c01abc0 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:09.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:09.743 7f6f3c8f0700 1 --2- 172.21.2.112:0/1459381036 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6f381a7eb0 0x7f6f380ac630 unknown :-1 s=CLOSED pgs=2024 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:09.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:09.743 7f6f3c8f0700 1 --2- 172.21.2.112:0/1459381036 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6f381a7490 0x7f6f381a7890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:09.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:09.743 7f6f3c8f0700 1 --2- 172.21.2.112:0/1459381036 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6f380a7950 0x7f6f380b8df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:09.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:09.743 7f6f3c8f0700 1 -- 172.21.2.112:0/1459381036 >> 172.21.2.112:0/1459381036 conn(0x7f6f380a02d0 msgr2=0x7f6f3809f200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:09.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:09.743 7f6f3c8f0700 1 -- 172.21.2.112:0/1459381036 shutdown_connections 2021-02-10T11:40:09.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:09.743 7f6f3c8f0700 1 -- 172.21.2.112:0/1459381036 wait complete. 2021-02-10T11:40:09.747 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2021-02-10T11:40:09.747 DEBUG:teuthology.orchestra.run.gibba012:> import os 2021-02-10T11:40:09.747 DEBUG:teuthology.orchestra.run.gibba012:> import stat 2021-02-10T11:40:09.747 DEBUG:teuthology.orchestra.run.gibba012:> import json 2021-02-10T11:40:09.747 DEBUG:teuthology.orchestra.run.gibba012:> import sys 2021-02-10T11:40:09.747 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:40:09.747 DEBUG:teuthology.orchestra.run.gibba012:> try: 2021-02-10T11:40:09.747 DEBUG:teuthology.orchestra.run.gibba012:> s = os.stat("/home/ubuntu/cephtest/mnt.0/volumes/_nogroup/subvolume_0000000000119244/c8e1f5a0-c998-4131-a8be-b62f10f5cee0") 2021-02-10T11:40:09.747 DEBUG:teuthology.orchestra.run.gibba012:> except OSError as e: 2021-02-10T11:40:09.748 DEBUG:teuthology.orchestra.run.gibba012:> sys.exit(e.errno) 2021-02-10T11:40:09.748 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:40:09.748 DEBUG:teuthology.orchestra.run.gibba012:> attrs = ["st_mode", "st_ino", "st_dev", "st_nlink", "st_uid", "st_gid", "st_size", "st_atime", "st_mtime", "st_ctime"] 2021-02-10T11:40:09.748 DEBUG:teuthology.orchestra.run.gibba012:> print(json.dumps( 2021-02-10T11:40:09.748 DEBUG:teuthology.orchestra.run.gibba012:> dict([(a, getattr(s, a)) for a in attrs]), 2021-02-10T11:40:09.748 DEBUG:teuthology.orchestra.run.gibba012:> indent=2)) 2021-02-10T11:40:09.748 DEBUG:teuthology.orchestra.run.gibba012:> ' 2021-02-10T11:40:10.162 INFO:teuthology.orchestra.run.gibba012.stdout:{ 2021-02-10T11:40:10.162 INFO:teuthology.orchestra.run.gibba012.stdout: "st_mode": 16877, 2021-02-10T11:40:10.162 INFO:teuthology.orchestra.run.gibba012.stdout: "st_ino": 1099511627779, 2021-02-10T11:40:10.162 INFO:teuthology.orchestra.run.gibba012.stdout: "st_dev": 42, 2021-02-10T11:40:10.162 INFO:teuthology.orchestra.run.gibba012.stdout: "st_nlink": 2, 2021-02-10T11:40:10.162 INFO:teuthology.orchestra.run.gibba012.stdout: "st_uid": 0, 2021-02-10T11:40:10.162 INFO:teuthology.orchestra.run.gibba012.stdout: "st_gid": 0, 2021-02-10T11:40:10.162 INFO:teuthology.orchestra.run.gibba012.stdout: "st_size": 0, 2021-02-10T11:40:10.163 INFO:teuthology.orchestra.run.gibba012.stdout: "st_atime": 1612957209.0771012, 2021-02-10T11:40:10.163 INFO:teuthology.orchestra.run.gibba012.stdout: "st_mtime": 1612957209.0771012, 2021-02-10T11:40:10.163 INFO:teuthology.orchestra.run.gibba012.stdout: "st_ctime": 1612957209.0862744 2021-02-10T11:40:10.163 INFO:teuthology.orchestra.run.gibba012.stdout:} 2021-02-10T11:40:10.322 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000119244 2021-02-10T11:40:10.793 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:10.780 7f316e326700 1 Processor -- start 2021-02-10T11:40:10.793 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:10.780 7f316e326700 1 -- start start 2021-02-10T11:40:10.793 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:10.780 7f316e326700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f31680d1870 0x7f31681b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:10.793 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:10.780 7f316e326700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f31681b0ae0 0x7f3168108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:10.793 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:10.780 7f316e326700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3168109a60 con 0x7f31680d1c70 2021-02-10T11:40:10.793 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:10.780 7f316e326700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31680e5ef0 con 0x7f31681b0ae0 2021-02-10T11:40:10.793 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:10.780 7f316e326700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3168109750 con 0x7f31680d1870 2021-02-10T11:40:10.793 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:10.780 7f316c943700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f31680d1c70 0x7f3168108f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:53898/0 (socket says 172.21.2.112:53898) 2021-02-10T11:40:10.793 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:10.780 7f316c943700 1 -- 172.21.2.112:0/2529050611 learned_addr learned my addr 172.21.2.112:0/2529050611 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:40:10.793 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:10.781 7f3166ffd700 1 -- 172.21.2.112:0/2529050611 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4031703632 0 0) 0x7f3160001500 con 0x7f31680d1c70 2021-02-10T11:40:10.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:10.781 7f3166ffd700 1 -- 172.21.2.112:0/2529050611 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3150002fe0 con 0x7f31680d1c70 2021-02-10T11:40:10.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:10.781 7f3167fff700 1 --2- 172.21.2.112:0/2529050611 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f31680d1870 0x7f31681b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:10.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:10.781 7f31677fe700 1 --2- 172.21.2.112:0/2529050611 >> v2:172.21.2.120:3300/0 conn(0x7f31681b0ae0 0x7f3168108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:10.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:10.781 7f3166ffd700 1 -- 172.21.2.112:0/2529050611 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2833095834 0 0) 0x7f3160001500 con 0x7f31680d1c70 2021-02-10T11:40:10.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:10.781 7f3166ffd700 1 -- 172.21.2.112:0/2529050611 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f31680d1870 msgr2=0x7f31681b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:10.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:10.781 7f3166ffd700 1 --2- 172.21.2.112:0/2529050611 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f31680d1870 0x7f31681b04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:10.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:10.781 7f3166ffd700 1 -- 172.21.2.112:0/2529050611 >> v2:172.21.2.120:3300/0 conn(0x7f31681b0ae0 msgr2=0x7f3168108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:10.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:10.781 7f3166ffd700 1 --2- 172.21.2.112:0/2529050611 >> v2:172.21.2.120:3300/0 conn(0x7f31681b0ae0 0x7f3168108940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:10.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:10.781 7f3166ffd700 1 -- 172.21.2.112:0/2529050611 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3168088070 con 0x7f31680d1c70 2021-02-10T11:40:10.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:10.781 7f3166ffd700 1 -- 172.21.2.112:0/2529050611 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f3160001a00 con 0x7f31680d1c70 2021-02-10T11:40:10.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:10.781 7f3166ffd700 1 -- 172.21.2.112:0/2529050611 >> v1:172.21.2.112:6789/0 conn(0x7f31680d1c70 legacy=0x7f3168108f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:10.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:10.781 7f3166ffd700 1 --2- 172.21.2.112:0/2529050611 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3150003b10 0x7f3150006010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:10.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:10.781 7f3166ffd700 1 --2- 172.21.2.112:0/2529050611 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3150006830 0x7f3150008d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:10.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:10.781 7f3166ffd700 1 --2- 172.21.2.112:0/2529050611 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3150009650 0x7f315000bb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:10.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:10.781 7f3166ffd700 1 -- 172.21.2.112:0/2529050611 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3150050380 con 0x7f3150003b10 2021-02-10T11:40:10.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:10.781 7f3166ffd700 1 -- 172.21.2.112:0/2529050611 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31500505a0 con 0x7f3150006830 2021-02-10T11:40:10.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:10.781 7f3166ffd700 1 -- 172.21.2.112:0/2529050611 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31500507c0 con 0x7f3150009650 2021-02-10T11:40:10.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:10.781 7f31677fe700 1 --2- 172.21.2.112:0/2529050611 >> v2:172.21.2.120:3300/0 conn(0x7f31681b0ae0 0x7f3168108940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:40:10.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:10.781 7f3167fff700 1 --2- 172.21.2.112:0/2529050611 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3150003b10 0x7f3150006010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:10.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:10.781 7f3167fff700 1 -- 172.21.2.112:0/2529050611 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3150009650 msgr2=0x7f315000bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:10.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:10.781 7f3167fff700 1 --2- 172.21.2.112:0/2529050611 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3150009650 0x7f315000bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:10.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:10.781 7f3167fff700 1 -- 172.21.2.112:0/2529050611 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3150006830 msgr2=0x7f3150008d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:10.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:10.781 7f3167fff700 1 --2- 172.21.2.112:0/2529050611 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3150006830 0x7f3150008d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:10.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:10.781 7f3167fff700 1 -- 172.21.2.112:0/2529050611 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3150050a60 con 0x7f3150003b10 2021-02-10T11:40:10.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:10.782 7f3167fff700 1 --2- 172.21.2.112:0/2529050611 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3150003b10 0x7f3150006010 secure :-1 s=READY pgs=2026 cs=0 l=1 rev1=1 rx=0x7f3158006ac0 tx=0x7f3158009ab0).ready entity=mon.0 client_cookie=b8ed175e4780e7b2 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:10.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:10.782 7f3166ffd700 1 -- 172.21.2.112:0/2529050611 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f315800c0c0 con 0x7f3150003b10 2021-02-10T11:40:10.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:10.782 7f3166ffd700 1 -- 172.21.2.112:0/2529050611 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f315800c400 con 0x7f3150003b10 2021-02-10T11:40:10.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:10.782 7f3166ffd700 1 -- 172.21.2.112:0/2529050611 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f315800c890 con 0x7f3150003b10 2021-02-10T11:40:10.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:10.782 7f316e326700 1 -- 172.21.2.112:0/2529050611 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3150003b10 msgr2=0x7f3150006010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:10.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:10.782 7f316e326700 1 --2- 172.21.2.112:0/2529050611 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3150003b10 0x7f3150006010 secure :-1 s=READY pgs=2026 cs=0 l=1 rev1=1 rx=0x7f3158006ac0 tx=0x7f3158009ab0).stop 2021-02-10T11:40:10.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:10.782 7f316c943700 1 -- 172.21.2.112:0/2529050611 reap_dead start 2021-02-10T11:40:10.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:10.782 7f316e326700 1 -- 172.21.2.112:0/2529050611 shutdown_connections 2021-02-10T11:40:10.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:10.782 7f316e326700 1 -- 172.21.2.112:0/2529050611 >> 172.21.2.112:0/2529050611 conn(0x7f31680ce420 msgr2=0x7f31680d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:10.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:10.783 7f316e326700 1 -- 172.21.2.112:0/2529050611 shutdown_connections 2021-02-10T11:40:10.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:10.783 7f316e326700 1 -- 172.21.2.112:0/2529050611 wait complete. 2021-02-10T11:40:10.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:10.797 7f316e326700 1 Processor -- start 2021-02-10T11:40:10.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:10.797 7f316e326700 1 -- start start 2021-02-10T11:40:10.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:10.798 7f316e326700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3168085bc0 0x7f31680863e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:10.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:10.798 7f316e326700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f31680954c0 0x7f3168096300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:10.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:10.798 7f316e326700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f31680958c0 0x7f3168096920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:10.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:10.798 7f316e326700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3168095d80 con 0x7f31680958c0 2021-02-10T11:40:10.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:10.798 7f316e326700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31680bfa60 con 0x7f31680954c0 2021-02-10T11:40:10.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:10.798 7f316e326700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3168085fc0 con 0x7f3168085bc0 2021-02-10T11:40:10.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:10.798 7f316c943700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f31680958c0 0x7f3168096920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:10.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:10.798 7f316c943700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f31680958c0 0x7f3168096920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:54452/0 (socket says 172.21.2.112:54452) 2021-02-10T11:40:10.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:10.798 7f316c943700 1 -- 172.21.2.112:0/2083824189 learned_addr learned my addr 172.21.2.112:0/2083824189 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:40:10.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:10.798 7f316c943700 1 -- 172.21.2.112:0/2083824189 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3168085bc0 msgr2=0x7f31680863e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:10.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:10.798 7f316c943700 1 --2- 172.21.2.112:0/2083824189 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3168085bc0 0x7f31680863e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:10.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:10.798 7f316c943700 1 -- 172.21.2.112:0/2083824189 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f31680954c0 msgr2=0x7f3168096300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:10.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:10.798 7f316c943700 1 --2- 172.21.2.112:0/2083824189 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f31680954c0 0x7f3168096300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:10.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:10.798 7f316c943700 1 -- 172.21.2.112:0/2083824189 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3168086c10 con 0x7f31680958c0 2021-02-10T11:40:10.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:10.799 7f316c943700 1 --2- 172.21.2.112:0/2083824189 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f31680958c0 0x7f3168096920 secure :-1 s=READY pgs=2027 cs=0 l=1 rev1=1 rx=0x7f3160007f90 tx=0x7f316000ac50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:10.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:10.799 7f3164ff9700 1 -- 172.21.2.112:0/2083824189 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3160000f20 con 0x7f31680958c0 2021-02-10T11:40:10.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:10.799 7f3164ff9700 1 -- 172.21.2.112:0/2083824189 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3160007020 con 0x7f31680958c0 2021-02-10T11:40:10.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:10.799 7f3164ff9700 1 -- 172.21.2.112:0/2083824189 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3160007470 con 0x7f31680958c0 2021-02-10T11:40:10.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:10.799 7f316e326700 1 -- 172.21.2.112:0/2083824189 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3168086c10 con 0x7f31680958c0 2021-02-10T11:40:10.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:10.799 7f316e326700 1 -- 172.21.2.112:0/2083824189 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3168086c10 con 0x7f31680958c0 2021-02-10T11:40:10.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:10.799 7f3164ff9700 1 -- 172.21.2.112:0/2083824189 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3160000d90 con 0x7f31680958c0 2021-02-10T11:40:10.800 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:10.799 7f3164ff9700 1 --2- 172.21.2.112:0/2083824189 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3148018730 0x7f314801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:10.800 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:10.799 7f3164ff9700 1 -- 172.21.2.112:0/2083824189 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f3160032960 con 0x7f31680958c0 2021-02-10T11:40:10.800 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:10.800 7f3167fff700 1 --2- 172.21.2.112:0/2083824189 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3148018730 0x7f314801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:10.800 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:10.800 7f3167fff700 1 --2- 172.21.2.112:0/2083824189 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3148018730 0x7f314801ac10 crc :-1 s=READY pgs=942 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:10.826 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:10.830 7f316e326700 1 -- 172.21.2.112:0/2083824189 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31680fe1c0 con 0x7f31680958c0 2021-02-10T11:40:10.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:10.832 7f3164ff9700 1 -- 172.21.2.112:0/2083824189 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3160007d00 con 0x7f31680958c0 2021-02-10T11:40:10.957 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:10.961 7f316e326700 1 -- 172.21.2.112:0/2083824189 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000119244", "target": ["mgr", ""]}) v1 -- 0x7f3168095d80 con 0x7f3148018730 2021-02-10T11:40:10.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:10.975 7f3164ff9700 1 -- 172.21.2.112:0/2083824189 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f3158006d60 con 0x7f3148018730 2021-02-10T11:40:10.975 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:10.979 7f316e326700 1 -- 172.21.2.112:0/2083824189 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3148018730 msgr2=0x7f314801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:10.975 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:10.979 7f316e326700 1 --2- 172.21.2.112:0/2083824189 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3148018730 0x7f314801ac10 crc :-1 s=READY pgs=942 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:10.975 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:10.979 7f316e326700 1 -- 172.21.2.112:0/2083824189 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f31680958c0 msgr2=0x7f3168096920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:10.975 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:10.979 7f316e326700 1 --2- 172.21.2.112:0/2083824189 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f31680958c0 0x7f3168096920 secure :-1 s=READY pgs=2027 cs=0 l=1 rev1=1 rx=0x7f3160007f90 tx=0x7f316000ac50).stop 2021-02-10T11:40:10.975 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:10.979 7f316e326700 1 -- 172.21.2.112:0/2083824189 shutdown_connections 2021-02-10T11:40:10.975 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:10.979 7f316e326700 1 --2- 172.21.2.112:0/2083824189 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3148018730 0x7f314801ac10 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:10.976 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:10.979 7f316e326700 1 --2- 172.21.2.112:0/2083824189 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f31680954c0 0x7f3168096300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:10.976 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:10.979 7f316e326700 1 --2- 172.21.2.112:0/2083824189 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f31680958c0 0x7f3168096920 unknown :-1 s=CLOSED pgs=2027 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:10.976 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:10.979 7f316e326700 1 --2- 172.21.2.112:0/2083824189 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3168085bc0 0x7f31680863e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:10.976 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:10.979 7f316e326700 1 -- 172.21.2.112:0/2083824189 >> 172.21.2.112:0/2083824189 conn(0x7f31680bef00 msgr2=0x7f316809d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:10.976 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:10.979 7f316e326700 1 -- 172.21.2.112:0/2083824189 shutdown_connections 2021-02-10T11:40:10.976 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:10.979 7f316e326700 1 -- 172.21.2.112:0/2083824189 wait complete. 2021-02-10T11:40:10.984 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo getfattr --only-values -n ceph.dir.entries /home/ubuntu/cephtest/mnt.0/./volumes/_deleting 2021-02-10T11:40:11.335 INFO:teuthology.orchestra.run.gibba012.stderr:getfattr: Removing leading '/' from absolute path names 2021-02-10T11:40:11.336 INFO:teuthology.orchestra.run.gibba012.stdout:0 2021-02-10T11:40:11.336 DEBUG:tasks.cephfs.mount:Directory ./volumes/_deleting seen empty from 0 after 0s 2021-02-10T11:40:11.336 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:40:11.336 DEBUG:teuthology.orchestra.run.gibba012:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:40:11.681 DEBUG:teuthology.orchestra.run.gibba020:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:40:12.024 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:40:12.025 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:40:12.025 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:40:12.025 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:40:12.361 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:40:12.362 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:40:12.362 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba012.front.sepia.ceph.com... 2021-02-10T11:40:12.362 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:40:12.362 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:40:12.720 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba012.stderr:ceph-fuse[153931]: fuse finished with error 0 and tester_r 0 2021-02-10T11:40:12.721 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:40:12.721 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:40:12.721 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:40:13.057 INFO:teuthology.orchestra.run.gibba012.stdout:ext2/ext3 2021-02-10T11:40:13.057 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:40:13.057 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:40:13.058 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:40:13.393 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:40:13.393 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:40:13.393 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:40:13.728 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:40:13.728 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:40:13.729 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba012.front.sepia.ceph.com... 2021-02-10T11:40:13.729 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:40:13.729 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:40:14.083 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba012.stderr:ceph-fuse[154636]: fuse finished with error 0 and tester_r 0 2021-02-10T11:40:14.084 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:40:14.085 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:40:14.085 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:40:14.420 INFO:teuthology.orchestra.run.gibba012.stdout:ext2/ext3 2021-02-10T11:40:14.420 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:40:14.420 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:40:14.421 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:40:14.755 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:40:14.755 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:40:14.755 DEBUG:teuthology.orchestra.run.gibba020:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:40:15.089 INFO:teuthology.orchestra.run.gibba020.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:40:15.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:40:15.089 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:40:15.090 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:40:15.090 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:40:15.424 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:40:15.424 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:40:15.424 DEBUG:teuthology.orchestra.run.gibba020:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:40:15.759 INFO:teuthology.orchestra.run.gibba020.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:40:15.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:40:15.759 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:40:15.759 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:40:15.760 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:40:16.093 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config rm mon mon_allow_pool_delete 2021-02-10T11:40:16.524 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:16.513 7f2a8bdae700 1 Processor -- start 2021-02-10T11:40:16.524 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:16.513 7f2a8bdae700 1 -- start start 2021-02-10T11:40:16.524 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:16.513 7f2a8bdae700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f2a840d0340 0x7f2a840ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:16.524 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:16.513 7f2a8bdae700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f2a840d1c70 0x7f2a840efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:16.524 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:16.513 7f2a8bdae700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a840f0b10 con 0x7f2a840d1870 2021-02-10T11:40:16.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:16.513 7f2a8bdae700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a840e5ed0 con 0x7f2a840d0340 2021-02-10T11:40:16.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:16.513 7f2a8bdae700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a841b0be0 con 0x7f2a840d1c70 2021-02-10T11:40:16.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:16.514 7f2a893c9700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f2a840d0340 0x7f2a840ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:16.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:16.514 7f2a893c9700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f2a840d0340 0x7f2a840ef640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:54456/0 (socket says 172.21.2.112:54456) 2021-02-10T11:40:16.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:16.514 7f2a893c9700 1 -- 172.21.2.112:0/2811534959 learned_addr learned my addr 172.21.2.112:0/2811534959 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:40:16.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:16.514 7f2a893c9700 1 -- 172.21.2.112:0/2811534959 >> v2:172.21.2.120:3300/0 conn(0x7f2a840d1c70 msgr2=0x7f2a840efc60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:40:16.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:16.514 7f2a893c9700 1 --2- 172.21.2.112:0/2811534959 >> v2:172.21.2.120:3300/0 conn(0x7f2a840d1c70 0x7f2a840efc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:16.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:16.514 7f2a893c9700 1 -- 172.21.2.112:0/2811534959 >> v1:172.21.2.120:6789/0 conn(0x7f2a840d1870 legacy=0x7f2a841b03f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:40:16.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:16.514 7f2a893c9700 1 -- 172.21.2.112:0/2811534959 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a840f20a0 con 0x7f2a840d0340 2021-02-10T11:40:16.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:16.514 7f2a893c9700 1 --2- 172.21.2.112:0/2811534959 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2a840d0340 0x7f2a840ef640 secure :-1 s=READY pgs=2028 cs=0 l=1 rev1=1 rx=0x7f2a78006700 tx=0x7f2a78009b80).ready entity=mon.0 client_cookie=a4a3e9fd782111d9 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:16.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:16.514 7f2a88bc8700 1 -- 172.21.2.112:0/2811534959 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2a7800c0c0 con 0x7f2a840d0340 2021-02-10T11:40:16.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:16.514 7f2a88bc8700 1 -- 172.21.2.112:0/2811534959 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a7800c440 con 0x7f2a840d0340 2021-02-10T11:40:16.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:16.514 7f2a88bc8700 1 -- 172.21.2.112:0/2811534959 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2a7800c960 con 0x7f2a840d0340 2021-02-10T11:40:16.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:16.514 7f2a8bdae700 1 -- 172.21.2.112:0/2811534959 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2a840d0340 msgr2=0x7f2a840ef640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:16.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:16.514 7f2a8bdae700 1 --2- 172.21.2.112:0/2811534959 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2a840d0340 0x7f2a840ef640 secure :-1 s=READY pgs=2028 cs=0 l=1 rev1=1 rx=0x7f2a78006700 tx=0x7f2a78009b80).stop 2021-02-10T11:40:16.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:16.515 7f2a8bdae700 1 -- 172.21.2.112:0/2811534959 shutdown_connections 2021-02-10T11:40:16.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:16.515 7f2a8bdae700 1 --2- 172.21.2.112:0/2811534959 >> v2:172.21.2.120:3300/0 conn(0x7f2a840d1c70 0x7f2a840efc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:16.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:16.515 7f2a8bdae700 1 --2- 172.21.2.112:0/2811534959 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2a840d0340 0x7f2a840ef640 unknown :-1 s=CLOSED pgs=2028 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:16.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:16.515 7f2a8bdae700 1 -- 172.21.2.112:0/2811534959 >> 172.21.2.112:0/2811534959 conn(0x7f2a840ce450 msgr2=0x7f2a840d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:16.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:16.515 7f2a8bdae700 1 -- 172.21.2.112:0/2811534959 shutdown_connections 2021-02-10T11:40:16.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:16.515 7f2a8bdae700 1 -- 172.21.2.112:0/2811534959 wait complete. 2021-02-10T11:40:16.528 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:16.530 7f2a8bdae700 1 Processor -- start 2021-02-10T11:40:16.528 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:16.530 7f2a8bdae700 1 -- start start 2021-02-10T11:40:16.528 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:16.530 7f2a8bdae700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2a840bbcf0 0x7f2a841a4400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:16.528 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:16.530 7f2a8bdae700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2a841a4a20 0x7f2a840bb020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:16.528 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:16.530 7f2a8bdae700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2a840ae5e0 0x7f2a840bb640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:16.528 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:16.530 7f2a8bdae700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a840bc2f0 con 0x7f2a841a4a20 2021-02-10T11:40:16.528 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:16.530 7f2a8bdae700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a84095700 con 0x7f2a840ae5e0 2021-02-10T11:40:16.528 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:16.530 7f2a8bdae700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a84095920 con 0x7f2a840bbcf0 2021-02-10T11:40:16.529 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:16.530 7f2a893c9700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2a841a4a20 0x7f2a840bb020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:16.529 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:16.530 7f2a893c9700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2a841a4a20 0x7f2a840bb020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:54460/0 (socket says 172.21.2.112:54460) 2021-02-10T11:40:16.529 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:16.530 7f2a893c9700 1 -- 172.21.2.112:0/1712753578 learned_addr learned my addr 172.21.2.112:0/1712753578 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:40:16.529 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:16.530 7f2a893c9700 1 -- 172.21.2.112:0/1712753578 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2a840bbcf0 msgr2=0x7f2a841a4400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:16.529 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:16.530 7f2a893c9700 1 --2- 172.21.2.112:0/1712753578 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2a840bbcf0 0x7f2a841a4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:16.529 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:16.531 7f2a893c9700 1 -- 172.21.2.112:0/1712753578 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2a840ae5e0 msgr2=0x7f2a840bb640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:16.529 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:16.531 7f2a893c9700 1 --2- 172.21.2.112:0/1712753578 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2a840ae5e0 0x7f2a840bb640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:16.529 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:16.531 7f2a893c9700 1 -- 172.21.2.112:0/1712753578 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a84095b40 con 0x7f2a841a4a20 2021-02-10T11:40:16.529 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:16.531 7f2a893c9700 1 --2- 172.21.2.112:0/1712753578 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2a841a4a20 0x7f2a840bb020 secure :-1 s=READY pgs=2029 cs=0 l=1 rev1=1 rx=0x7f2a7800b5e0 tx=0x7f2a7800b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:16.529 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:16.531 7f2a767fc700 1 -- 172.21.2.112:0/1712753578 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2a7800bd70 con 0x7f2a841a4a20 2021-02-10T11:40:16.529 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:16.531 7f2a767fc700 1 -- 172.21.2.112:0/1712753578 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a78002030 con 0x7f2a841a4a20 2021-02-10T11:40:16.530 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:16.531 7f2a767fc700 1 -- 172.21.2.112:0/1712753578 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2a780024d0 con 0x7f2a841a4a20 2021-02-10T11:40:16.530 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:16.531 7f2a8bdae700 1 -- 172.21.2.112:0/1712753578 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a84095b40 con 0x7f2a841a4a20 2021-02-10T11:40:16.530 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:16.531 7f2a8bdae700 1 -- 172.21.2.112:0/1712753578 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a84095b40 con 0x7f2a841a4a20 2021-02-10T11:40:16.530 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:16.532 7f2a767fc700 1 -- 172.21.2.112:0/1712753578 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2a78000c90 con 0x7f2a841a4a20 2021-02-10T11:40:16.530 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:16.532 7f2a767fc700 1 --2- 172.21.2.112:0/1712753578 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2a6c0186c0 0x7f2a6c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:16.530 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:16.532 7f2a767fc700 1 -- 172.21.2.112:0/1712753578 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f2a780325a0 con 0x7f2a841a4a20 2021-02-10T11:40:16.530 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:16.532 7f2a89bca700 1 --2- 172.21.2.112:0/1712753578 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2a6c0186c0 0x7f2a6c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:16.530 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:16.533 7f2a89bca700 1 --2- 172.21.2.112:0/1712753578 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2a6c0186c0 0x7f2a6c01abc0 crc :-1 s=READY pgs=943 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:16.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:16.563 7f2a8bdae700 1 -- 172.21.2.112:0/1712753578 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a840ce450 con 0x7f2a841a4a20 2021-02-10T11:40:16.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:16.566 7f2a767fc700 1 -- 172.21.2.112:0/1712753578 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2a78030d50 con 0x7f2a841a4a20 2021-02-10T11:40:16.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:16.714 7f2a8bdae700 1 -- 172.21.2.112:0/1712753578 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f2a840ce450 con 0x7f2a841a4a20 2021-02-10T11:40:16.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:16.719 7f2a767fc700 1 -- 172.21.2.112:0/1712753578 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v27) v1 ==== 104+0+0 (secure 0 0 0) 0x7f2a78000c90 con 0x7f2a841a4a20 2021-02-10T11:40:16.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:16.723 7f2a8bdae700 1 -- 172.21.2.112:0/1712753578 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2a6c0186c0 msgr2=0x7f2a6c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:16.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:16.723 7f2a8bdae700 1 --2- 172.21.2.112:0/1712753578 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2a6c0186c0 0x7f2a6c01abc0 crc :-1 s=READY pgs=943 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:16.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:16.724 7f2a8bdae700 1 -- 172.21.2.112:0/1712753578 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2a841a4a20 msgr2=0x7f2a840bb020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:16.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:16.724 7f2a8bdae700 1 --2- 172.21.2.112:0/1712753578 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2a841a4a20 0x7f2a840bb020 secure :-1 s=READY pgs=2029 cs=0 l=1 rev1=1 rx=0x7f2a7800b5e0 tx=0x7f2a7800b980).stop 2021-02-10T11:40:16.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:16.724 7f2a8bdae700 1 -- 172.21.2.112:0/1712753578 shutdown_connections 2021-02-10T11:40:16.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:16.724 7f2a8bdae700 1 --2- 172.21.2.112:0/1712753578 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2a6c0186c0 0x7f2a6c01abc0 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:16.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:16.724 7f2a8bdae700 1 --2- 172.21.2.112:0/1712753578 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2a840ae5e0 0x7f2a840bb640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:16.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:16.724 7f2a8bdae700 1 --2- 172.21.2.112:0/1712753578 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2a841a4a20 0x7f2a840bb020 unknown :-1 s=CLOSED pgs=2029 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:16.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:16.724 7f2a8bdae700 1 --2- 172.21.2.112:0/1712753578 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2a840bbcf0 0x7f2a841a4400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:16.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:16.724 7f2a8bdae700 1 -- 172.21.2.112:0/1712753578 >> 172.21.2.112:0/1712753578 conn(0x7f2a840a6620 msgr2=0x7f2a840b17a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:16.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:16.724 7f2a8bdae700 1 -- 172.21.2.112:0/1712753578 shutdown_connections 2021-02-10T11:40:16.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:16.724 7f2a8bdae700 1 -- 172.21.2.112:0/1712753578 wait complete. 2021-02-10T11:40:16.727 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestVolumes.test_default_uid_gid_subvolume' 2021-02-10T11:40:17.154 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:17.143 7fd864e43700 1 Processor -- start 2021-02-10T11:40:17.155 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:17.144 7fd864e43700 1 -- start start 2021-02-10T11:40:17.155 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:17.144 7fd864e43700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fd8600d0340 0x7fd8600ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:17.155 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:17.144 7fd864e43700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8600f0b10 con 0x7fd8600d1c70 2021-02-10T11:40:17.155 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:17.144 7fd864e43700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8601b0be0 con 0x7fd8600d1870 2021-02-10T11:40:17.155 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:17.144 7fd864e43700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8600e5ed0 con 0x7fd8600d0340 2021-02-10T11:40:17.155 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:17.144 7fd85e312700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fd8600d0340 0x7fd8600ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:17.155 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:17.144 7fd85e312700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fd8600d0340 0x7fd8600ef640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:54470/0 (socket says 172.21.2.112:54470) 2021-02-10T11:40:17.155 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:17.144 7fd85e312700 1 -- 172.21.2.112:0/2426551923 learned_addr learned my addr 172.21.2.112:0/2426551923 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:40:17.156 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:17.145 7fd85e312700 1 -- 172.21.2.112:0/2426551923 >> v1:172.21.2.120:6789/0 conn(0x7fd8600d1870 legacy=0x7fd8601b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:17.156 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:17.145 7fd85e312700 1 -- 172.21.2.112:0/2426551923 >> v1:172.21.2.112:6789/0 conn(0x7fd8600d1c70 legacy=0x7fd8600efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:17.156 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:17.145 7fd85e312700 1 -- 172.21.2.112:0/2426551923 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8600f0450 con 0x7fd8600d0340 2021-02-10T11:40:17.156 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:17.145 7fd85e312700 1 --2- 172.21.2.112:0/2426551923 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd8600d0340 0x7fd8600ef640 secure :-1 s=READY pgs=2031 cs=0 l=1 rev1=1 rx=0x7fd850006700 tx=0x7fd850009b00).ready entity=mon.0 client_cookie=5908112e5d2124dd server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:17.156 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:17.145 7fd85db11700 1 -- 172.21.2.112:0/2426551923 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd85000c0c0 con 0x7fd8600d0340 2021-02-10T11:40:17.156 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:17.145 7fd85db11700 1 -- 172.21.2.112:0/2426551923 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd85000c400 con 0x7fd8600d0340 2021-02-10T11:40:17.156 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:17.145 7fd85db11700 1 -- 172.21.2.112:0/2426551923 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd85000c8e0 con 0x7fd8600d0340 2021-02-10T11:40:17.156 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:17.145 7fd864e43700 1 -- 172.21.2.112:0/2426551923 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd8600d0340 msgr2=0x7fd8600ef640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:17.156 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:17.145 7fd864e43700 1 --2- 172.21.2.112:0/2426551923 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd8600d0340 0x7fd8600ef640 secure :-1 s=READY pgs=2031 cs=0 l=1 rev1=1 rx=0x7fd850006700 tx=0x7fd850009b00).stop 2021-02-10T11:40:17.156 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:17.146 7fd864e43700 1 -- 172.21.2.112:0/2426551923 shutdown_connections 2021-02-10T11:40:17.157 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:17.146 7fd864e43700 1 --2- 172.21.2.112:0/2426551923 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd8600d0340 0x7fd8600ef640 unknown :-1 s=CLOSED pgs=2031 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:17.157 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:17.146 7fd864e43700 1 -- 172.21.2.112:0/2426551923 >> 172.21.2.112:0/2426551923 conn(0x7fd8600ce450 msgr2=0x7fd8600d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:17.157 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:17.146 7fd864e43700 1 -- 172.21.2.112:0/2426551923 shutdown_connections 2021-02-10T11:40:17.157 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:17.146 7fd864e43700 1 -- 172.21.2.112:0/2426551923 wait complete. 2021-02-10T11:40:17.159 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:17.160 7fd864e43700 1 Processor -- start 2021-02-10T11:40:17.159 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:17.160 7fd864e43700 1 -- start start 2021-02-10T11:40:17.159 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:17.161 7fd864e43700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd8601be120 0x7fd860095400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:17.159 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:17.161 7fd864e43700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd8601040e0 0x7fd860095a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:17.159 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:17.161 7fd864e43700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd860102ee0 0x7fd8601032e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:17.159 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:17.161 7fd864e43700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd860103e90 con 0x7fd860102ee0 2021-02-10T11:40:17.159 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:17.161 7fd864e43700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd860103940 con 0x7fd8601040e0 2021-02-10T11:40:17.159 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:17.161 7fd864e43700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8600966a0 con 0x7fd8601be120 2021-02-10T11:40:17.159 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:17.161 7fd85f314700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd860102ee0 0x7fd8601032e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:17.159 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:17.161 7fd85f314700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd860102ee0 0x7fd8601032e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:54478/0 (socket says 172.21.2.112:54478) 2021-02-10T11:40:17.159 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:17.161 7fd85eb13700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd8601be120 0x7fd860095400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:17.160 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:17.161 7fd85f314700 1 -- 172.21.2.112:0/3140454762 learned_addr learned my addr 172.21.2.112:0/3140454762 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:40:17.160 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:17.161 7fd85e312700 1 --2- 172.21.2.112:0/3140454762 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd8601040e0 0x7fd860095a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:17.160 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:17.161 7fd85f314700 1 -- 172.21.2.112:0/3140454762 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd8601be120 msgr2=0x7fd860095400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:17.160 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:17.161 7fd85f314700 1 --2- 172.21.2.112:0/3140454762 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd8601be120 0x7fd860095400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:17.160 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:17.161 7fd85f314700 1 -- 172.21.2.112:0/3140454762 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd8601040e0 msgr2=0x7fd860095a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:17.160 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:17.161 7fd85f314700 1 --2- 172.21.2.112:0/3140454762 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd8601040e0 0x7fd860095a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:17.160 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:17.161 7fd85f314700 1 -- 172.21.2.112:0/3140454762 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8600968c0 con 0x7fd860102ee0 2021-02-10T11:40:17.160 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:17.161 7fd85eb13700 1 --2- 172.21.2.112:0/3140454762 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd8601be120 0x7fd860095400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:40:17.160 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:17.162 7fd85f314700 1 --2- 172.21.2.112:0/3140454762 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd860102ee0 0x7fd8601032e0 secure :-1 s=READY pgs=2032 cs=0 l=1 rev1=1 rx=0x7fd858009980 tx=0x7fd85800bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:17.160 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:17.162 7fd8577fe700 1 -- 172.21.2.112:0/3140454762 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd858005870 con 0x7fd860102ee0 2021-02-10T11:40:17.161 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:17.162 7fd8577fe700 1 -- 172.21.2.112:0/3140454762 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd858005a70 con 0x7fd860102ee0 2021-02-10T11:40:17.161 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:17.162 7fd8577fe700 1 -- 172.21.2.112:0/3140454762 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd858006030 con 0x7fd860102ee0 2021-02-10T11:40:17.161 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:17.162 7fd864e43700 1 -- 172.21.2.112:0/3140454762 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8600968c0 con 0x7fd860102ee0 2021-02-10T11:40:17.161 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:17.162 7fd864e43700 1 -- 172.21.2.112:0/3140454762 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8600968c0 con 0x7fd860102ee0 2021-02-10T11:40:17.161 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:17.163 7fd8577fe700 1 -- 172.21.2.112:0/3140454762 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd858005c50 con 0x7fd860102ee0 2021-02-10T11:40:17.161 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:17.163 7fd8577fe700 1 --2- 172.21.2.112:0/3140454762 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd8440186c0 0x7fd84401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:17.161 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:17.163 7fd8577fe700 1 -- 172.21.2.112:0/3140454762 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fd858032940 con 0x7fd860102ee0 2021-02-10T11:40:17.161 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:17.163 7fd85eb13700 1 --2- 172.21.2.112:0/3140454762 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd8440186c0 0x7fd84401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:17.161 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:17.163 7fd85eb13700 1 --2- 172.21.2.112:0/3140454762 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd8440186c0 0x7fd84401abc0 crc :-1 s=READY pgs=944 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:17.189 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:17.194 7fd864e43700 1 -- 172.21.2.112:0/3140454762 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8600ffb10 con 0x7fd860102ee0 2021-02-10T11:40:17.191 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:17.197 7fd8577fe700 1 -- 172.21.2.112:0/3140454762 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd85800be10 con 0x7fd860102ee0 2021-02-10T11:40:17.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:17.331 7fd864e43700 1 -- 172.21.2.112:0/3140454762 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_default_uid_gid_subvolume"]} v 0) v1 -- 0x7fd8600ffb10 con 0x7fd860102ee0 2021-02-10T11:40:18.175 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:18.180 7fd8577fe700 1 -- 172.21.2.112:0/3140454762 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_default_uid_gid_subvolume"]}]=0 v537) v1 ==== 147+0+0 (secure 0 0 0) 0x7fd85800be10 con 0x7fd860102ee0 2021-02-10T11:40:18.197 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:18.202 7fd864e43700 1 -- 172.21.2.112:0/3140454762 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd8440186c0 msgr2=0x7fd84401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:18.197 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:18.202 7fd864e43700 1 --2- 172.21.2.112:0/3140454762 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd8440186c0 0x7fd84401abc0 crc :-1 s=READY pgs=944 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:18.197 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:18.202 7fd864e43700 1 -- 172.21.2.112:0/3140454762 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd860102ee0 msgr2=0x7fd8601032e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:18.197 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:18.202 7fd864e43700 1 --2- 172.21.2.112:0/3140454762 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd860102ee0 0x7fd8601032e0 secure :-1 s=READY pgs=2032 cs=0 l=1 rev1=1 rx=0x7fd858009980 tx=0x7fd85800bcd0).stop 2021-02-10T11:40:18.197 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:18.202 7fd864e43700 1 -- 172.21.2.112:0/3140454762 shutdown_connections 2021-02-10T11:40:18.198 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:18.202 7fd864e43700 1 --2- 172.21.2.112:0/3140454762 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd8440186c0 0x7fd84401abc0 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:18.198 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:18.202 7fd864e43700 1 --2- 172.21.2.112:0/3140454762 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd8601040e0 0x7fd860095a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:18.198 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:18.202 7fd864e43700 1 --2- 172.21.2.112:0/3140454762 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd860102ee0 0x7fd8601032e0 unknown :-1 s=CLOSED pgs=2032 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:18.198 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:18.202 7fd864e43700 1 --2- 172.21.2.112:0/3140454762 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd8601be120 0x7fd860095400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:18.198 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:18.202 7fd864e43700 1 -- 172.21.2.112:0/3140454762 >> 172.21.2.112:0/3140454762 conn(0x7fd86010a820 msgr2=0x7fd8600c0be0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:18.198 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:18.202 7fd864e43700 1 -- 172.21.2.112:0/3140454762 shutdown_connections 2021-02-10T11:40:18.198 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:18.202 7fd864e43700 1 -- 172.21.2.112:0/3140454762 wait complete. 2021-02-10T11:40:18.206 INFO:tasks.cephfs_test_runner:test_default_uid_gid_subvolume (tasks.cephfs.test_volumes.TestVolumes) ... ok 2021-02-10T11:40:18.206 INFO:tasks.cephfs_test_runner:Starting test: test_default_uid_gid_subvolume_group (tasks.cephfs.test_volumes.TestVolumes) 2021-02-10T11:40:18.206 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestVolumes.test_default_uid_gid_subvolume_group' 2021-02-10T11:40:18.633 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:18.622 7f80b4686700 1 Processor -- start 2021-02-10T11:40:18.633 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:18.622 7f80b4686700 1 -- start start 2021-02-10T11:40:18.634 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:18.623 7f80b4686700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f80ac0d1870 0x7f80ac1b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:18.634 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:18.623 7f80b4686700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f80ac0f0650 0x7f80ac0cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:18.634 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:18.623 7f80b4686700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80ac0f0b10 con 0x7f80ac1b0a10 2021-02-10T11:40:18.634 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:18.623 7f80b4686700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80ac0e5ed0 con 0x7f80ac0d1870 2021-02-10T11:40:18.634 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:18.623 7f80b4686700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80ac0d1c70 con 0x7f80ac0f0650 2021-02-10T11:40:18.634 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:18.623 7f80b24a2700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f80ac0d1870 0x7f80ac1b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:18.634 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:18.623 7f80b24a2700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f80ac0d1870 0x7f80ac1b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:54482/0 (socket says 172.21.2.112:54482) 2021-02-10T11:40:18.634 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:18.623 7f80b24a2700 1 -- 172.21.2.112:0/3978975766 learned_addr learned my addr 172.21.2.112:0/3978975766 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:40:18.634 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:18.623 7f80b24a2700 1 -- 172.21.2.112:0/3978975766 >> v2:172.21.2.120:3300/0 conn(0x7f80ac0f0650 msgr2=0x7f80ac0cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:18.634 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:18.623 7f80b24a2700 1 --2- 172.21.2.112:0/3978975766 >> v2:172.21.2.120:3300/0 conn(0x7f80ac0f0650 0x7f80ac0cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:18.635 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:18.623 7f80b24a2700 1 -- 172.21.2.112:0/3978975766 >> v1:172.21.2.120:6789/0 conn(0x7f80ac1b0a10 legacy=0x7f80ac0cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:18.635 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:18.623 7f80b24a2700 1 -- 172.21.2.112:0/3978975766 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80ac0d1e90 con 0x7f80ac0d1870 2021-02-10T11:40:18.635 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:18.623 7f80b24a2700 1 --2- 172.21.2.112:0/3978975766 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f80ac0d1870 0x7f80ac1b03f0 secure :-1 s=READY pgs=2033 cs=0 l=1 rev1=1 rx=0x7f80a0006700 tx=0x7f80a0009b00).ready entity=mon.0 client_cookie=77b063d15c12d213 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:18.635 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:18.623 7f80b14a0700 1 -- 172.21.2.112:0/3978975766 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f80a000c0c0 con 0x7f80ac0d1870 2021-02-10T11:40:18.635 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:18.623 7f80b14a0700 1 -- 172.21.2.112:0/3978975766 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80a000c340 con 0x7f80ac0d1870 2021-02-10T11:40:18.635 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:18.623 7f80b14a0700 1 -- 172.21.2.112:0/3978975766 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f80a000c820 con 0x7f80ac0d1870 2021-02-10T11:40:18.635 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:18.623 7f80b4686700 1 -- 172.21.2.112:0/3978975766 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f80ac0d1870 msgr2=0x7f80ac1b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:18.635 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:18.623 7f80b4686700 1 --2- 172.21.2.112:0/3978975766 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f80ac0d1870 0x7f80ac1b03f0 secure :-1 s=READY pgs=2033 cs=0 l=1 rev1=1 rx=0x7f80a0006700 tx=0x7f80a0009b00).stop 2021-02-10T11:40:18.635 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:18.624 7f80b4686700 1 -- 172.21.2.112:0/3978975766 shutdown_connections 2021-02-10T11:40:18.635 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:18.624 7f80b4686700 1 --2- 172.21.2.112:0/3978975766 >> v2:172.21.2.120:3300/0 conn(0x7f80ac0f0650 0x7f80ac0cfb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:18.635 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:18.624 7f80b4686700 1 --2- 172.21.2.112:0/3978975766 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f80ac0d1870 0x7f80ac1b03f0 unknown :-1 s=CLOSED pgs=2033 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:18.636 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:18.624 7f80b4686700 1 -- 172.21.2.112:0/3978975766 >> 172.21.2.112:0/3978975766 conn(0x7f80ac0ce450 msgr2=0x7f80ac0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:18.636 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:18.624 7f80b4686700 1 -- 172.21.2.112:0/3978975766 shutdown_connections 2021-02-10T11:40:18.636 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:18.624 7f80b4686700 1 -- 172.21.2.112:0/3978975766 wait complete. 2021-02-10T11:40:18.637 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:18.639 7f80b4686700 1 Processor -- start 2021-02-10T11:40:18.637 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:18.639 7f80b4686700 1 -- start start 2021-02-10T11:40:18.638 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:18.639 7f80b4686700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f80ac19af00 0x7f80ac19b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:18.638 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:18.639 7f80b4686700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f80ac19ba80 0x7f80ac19be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:18.638 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:18.639 7f80b4686700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f80ac19c4a0 0x7f80ac08a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:18.638 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:18.639 7f80b4686700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80ac08fb50 con 0x7f80ac19ba80 2021-02-10T11:40:18.638 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:18.639 7f80b4686700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80ac08afb0 con 0x7f80ac19af00 2021-02-10T11:40:18.638 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:18.639 7f80b4686700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80ac08b1a0 con 0x7f80ac19c4a0 2021-02-10T11:40:18.638 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:18.639 7f80b1ca1700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f80ac19ba80 0x7f80ac19be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:18.638 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:18.639 7f80b1ca1700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f80ac19ba80 0x7f80ac19be80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:54488/0 (socket says 172.21.2.112:54488) 2021-02-10T11:40:18.638 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:18.639 7f80b1ca1700 1 -- 172.21.2.112:0/1400670113 learned_addr learned my addr 172.21.2.112:0/1400670113 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:40:18.638 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:18.640 7f80b1ca1700 1 -- 172.21.2.112:0/1400670113 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f80ac19c4a0 msgr2=0x7f80ac08a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:18.639 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:18.640 7f80b1ca1700 1 --2- 172.21.2.112:0/1400670113 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f80ac19c4a0 0x7f80ac08a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:18.639 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:18.640 7f80b1ca1700 1 -- 172.21.2.112:0/1400670113 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f80ac19af00 msgr2=0x7f80ac19b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:18.639 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:18.640 7f80b1ca1700 1 --2- 172.21.2.112:0/1400670113 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f80ac19af00 0x7f80ac19b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:18.639 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:18.640 7f80b1ca1700 1 -- 172.21.2.112:0/1400670113 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80ac08b3c0 con 0x7f80ac19ba80 2021-02-10T11:40:18.639 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:18.640 7f80b1ca1700 1 --2- 172.21.2.112:0/1400670113 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f80ac19ba80 0x7f80ac19be80 secure :-1 s=READY pgs=2034 cs=0 l=1 rev1=1 rx=0x7f80a800b5b0 tx=0x7f80a800b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:18.639 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:18.640 7f809effd700 1 -- 172.21.2.112:0/1400670113 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f80a800bd40 con 0x7f80ac19ba80 2021-02-10T11:40:18.639 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:18.640 7f809effd700 1 -- 172.21.2.112:0/1400670113 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80a8000e20 con 0x7f80ac19ba80 2021-02-10T11:40:18.639 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:18.640 7f809effd700 1 -- 172.21.2.112:0/1400670113 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f80a80054a0 con 0x7f80ac19ba80 2021-02-10T11:40:18.639 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:18.640 7f80b4686700 1 -- 172.21.2.112:0/1400670113 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80ac08b3c0 con 0x7f80ac19ba80 2021-02-10T11:40:18.639 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:18.640 7f80b4686700 1 -- 172.21.2.112:0/1400670113 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80ac08b950 con 0x7f80ac19ba80 2021-02-10T11:40:18.639 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:18.641 7f809effd700 1 -- 172.21.2.112:0/1400670113 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f80a8006030 con 0x7f80ac19ba80 2021-02-10T11:40:18.640 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:18.641 7f809effd700 1 --2- 172.21.2.112:0/1400670113 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f80940186c0 0x7f809401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:18.640 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:18.641 7f809effd700 1 -- 172.21.2.112:0/1400670113 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f80a80323d0 con 0x7f80ac19ba80 2021-02-10T11:40:18.640 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:18.641 7f80b24a2700 1 --2- 172.21.2.112:0/1400670113 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f80940186c0 0x7f809401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:18.640 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:18.642 7f80b24a2700 1 --2- 172.21.2.112:0/1400670113 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f80940186c0 0x7f809401abc0 crc :-1 s=READY pgs=945 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:18.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:18.672 7f80b4686700 1 -- 172.21.2.112:0/1400670113 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80ac0ee030 con 0x7f80ac19ba80 2021-02-10T11:40:18.670 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:18.675 7f809effd700 1 -- 172.21.2.112:0/1400670113 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f80a8015400 con 0x7f80ac19ba80 2021-02-10T11:40:18.806 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:18.811 7f80b4686700 1 -- 172.21.2.112:0/1400670113 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_default_uid_gid_subvolume_group"]} v 0) v1 -- 0x7f80ac0ce450 con 0x7f80ac19ba80 2021-02-10T11:40:19.178 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:19.183 7f809effd700 1 -- 172.21.2.112:0/1400670113 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_default_uid_gid_subvolume_group"]}]=0 v538) v1 ==== 156+0+0 (secure 0 0 0) 0x7f80a8006030 con 0x7f80ac19ba80 2021-02-10T11:40:19.223 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:19.227 7f80b4686700 1 -- 172.21.2.112:0/1400670113 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f80940186c0 msgr2=0x7f809401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:19.223 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:19.227 7f80b4686700 1 --2- 172.21.2.112:0/1400670113 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f80940186c0 0x7f809401abc0 crc :-1 s=READY pgs=945 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:19.223 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:19.227 7f80b4686700 1 -- 172.21.2.112:0/1400670113 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f80ac19ba80 msgr2=0x7f80ac19be80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:19.223 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:19.227 7f80b4686700 1 --2- 172.21.2.112:0/1400670113 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f80ac19ba80 0x7f80ac19be80 secure :-1 s=READY pgs=2034 cs=0 l=1 rev1=1 rx=0x7f80a800b5b0 tx=0x7f80a800b950).stop 2021-02-10T11:40:19.223 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:19.227 7f80b4686700 1 -- 172.21.2.112:0/1400670113 shutdown_connections 2021-02-10T11:40:19.223 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:19.227 7f80b4686700 1 --2- 172.21.2.112:0/1400670113 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f80940186c0 0x7f809401abc0 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:19.223 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:19.227 7f80b4686700 1 --2- 172.21.2.112:0/1400670113 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f80ac19c4a0 0x7f80ac08a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:19.223 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:19.227 7f80b4686700 1 --2- 172.21.2.112:0/1400670113 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f80ac19ba80 0x7f80ac19be80 unknown :-1 s=CLOSED pgs=2034 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:19.223 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:19.227 7f80b4686700 1 --2- 172.21.2.112:0/1400670113 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f80ac19af00 0x7f80ac19b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:19.223 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:19.227 7f80b4686700 1 -- 172.21.2.112:0/1400670113 >> 172.21.2.112:0/1400670113 conn(0x7f80ac19f8c0 msgr2=0x7f80ac1a5fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:19.223 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:19.228 7f80b4686700 1 -- 172.21.2.112:0/1400670113 shutdown_connections 2021-02-10T11:40:19.223 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:19.228 7f80b4686700 1 -- 172.21.2.112:0/1400670113 wait complete. 2021-02-10T11:40:19.231 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:40:19.662 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:19.651 7f31ea3c9700 1 Processor -- start 2021-02-10T11:40:19.662 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:19.651 7f31ea3c9700 1 -- start start 2021-02-10T11:40:19.663 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:19.651 7f31ea3c9700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f31e40d1c70 0x7f31e40cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:19.663 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:19.651 7f31ea3c9700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f31e40cf580 0x7f31e40cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:19.663 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:19.651 7f31ea3c9700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31e40fe1c0 con 0x7f31e40d1870 2021-02-10T11:40:19.663 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:19.651 7f31ea3c9700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31e41b0cb0 con 0x7f31e40d1c70 2021-02-10T11:40:19.663 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:19.651 7f31ea3c9700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31e4109a60 con 0x7f31e40cf580 2021-02-10T11:40:19.663 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:19.651 7f31e3fff700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f31e40d1870 0x7f31e41b04c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:53954/0 (socket says 172.21.2.112:53954) 2021-02-10T11:40:19.663 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:19.651 7f31e3fff700 1 -- 172.21.2.112:0/3552576154 learned_addr learned my addr 172.21.2.112:0/3552576154 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:40:19.663 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:19.652 7f31e37fe700 1 --2- 172.21.2.112:0/3552576154 >> v2:172.21.2.120:3300/0 conn(0x7f31e40d1c70 0x7f31e40cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:19.663 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:19.652 7f31e89e6700 1 --2- 172.21.2.112:0/3552576154 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f31e40cf580 0x7f31e40cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:19.663 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:19.652 7f31e2ffd700 1 -- 172.21.2.112:0/3552576154 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 628233567 0 0) 0x7f31d4000ce0 con 0x7f31e40d1870 2021-02-10T11:40:19.663 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:19.652 7f31e2ffd700 1 -- 172.21.2.112:0/3552576154 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31cc002fe0 con 0x7f31e40d1870 2021-02-10T11:40:19.663 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:19.652 7f31e37fe700 1 -- 172.21.2.112:0/3552576154 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f31e40cf580 msgr2=0x7f31e40cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:19.663 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:19.652 7f31e37fe700 1 --2- 172.21.2.112:0/3552576154 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f31e40cf580 0x7f31e40cf980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:19.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:19.652 7f31e37fe700 1 -- 172.21.2.112:0/3552576154 >> v1:172.21.2.112:6789/0 conn(0x7f31e40d1870 legacy=0x7f31e41b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:19.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:19.652 7f31e37fe700 1 -- 172.21.2.112:0/3552576154 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31e4088070 con 0x7f31e40d1c70 2021-02-10T11:40:19.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:19.652 7f31e37fe700 1 --2- 172.21.2.112:0/3552576154 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f31e40d1c70 0x7f31e40cef60 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 rx=0x7f31d8006650 tx=0x7f31d8009a80).ready entity=mon.1 client_cookie=287a56bc4dbff329 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:19.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:19.653 7f31e2ffd700 1 -- 172.21.2.112:0/3552576154 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f31d80009b0 con 0x7f31e40d1c70 2021-02-10T11:40:19.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:19.653 7f31e2ffd700 1 -- 172.21.2.112:0/3552576154 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31d800c0c0 con 0x7f31e40d1c70 2021-02-10T11:40:19.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:19.653 7f31e2ffd700 1 -- 172.21.2.112:0/3552576154 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f31d800c5e0 con 0x7f31e40d1c70 2021-02-10T11:40:19.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:19.653 7f31ea3c9700 1 -- 172.21.2.112:0/3552576154 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f31e40d1c70 msgr2=0x7f31e40cef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:19.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:19.653 7f31ea3c9700 1 --2- 172.21.2.112:0/3552576154 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f31e40d1c70 0x7f31e40cef60 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 rx=0x7f31d8006650 tx=0x7f31d8009a80).stop 2021-02-10T11:40:19.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:19.653 7f31ea3c9700 1 -- 172.21.2.112:0/3552576154 shutdown_connections 2021-02-10T11:40:19.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:19.653 7f31ea3c9700 1 --2- 172.21.2.112:0/3552576154 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f31e40cf580 0x7f31e40cf980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:19.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:19.653 7f31ea3c9700 1 --2- 172.21.2.112:0/3552576154 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f31e40d1c70 0x7f31e40cef60 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:19.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:19.653 7f31ea3c9700 1 -- 172.21.2.112:0/3552576154 >> 172.21.2.112:0/3552576154 conn(0x7f31e40ce420 msgr2=0x7f31e40d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:19.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:19.653 7f31ea3c9700 1 -- 172.21.2.112:0/3552576154 shutdown_connections 2021-02-10T11:40:19.665 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:19.653 7f31ea3c9700 1 -- 172.21.2.112:0/3552576154 wait complete. 2021-02-10T11:40:19.666 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:19.668 7f31ea3c9700 1 Processor -- start 2021-02-10T11:40:19.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:19.668 7f31ea3c9700 1 -- start start 2021-02-10T11:40:19.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:19.668 7f31ea3c9700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f31e409bcc0 0x7f31e409c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:19.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:19.668 7f31ea3c9700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f31e40a0820 0x7f31e40a0c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:19.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:19.668 7f31ea3c9700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f31e40a1240 0x7f31e40858d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:19.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:19.668 7f31ea3c9700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31e40a66b0 con 0x7f31e409bcc0 2021-02-10T11:40:19.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:19.668 7f31ea3c9700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31e41a49b0 con 0x7f31e40a1240 2021-02-10T11:40:19.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:19.668 7f31ea3c9700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31e41a4b90 con 0x7f31e40a0820 2021-02-10T11:40:19.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:19.668 7f31e3fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f31e409bcc0 0x7f31e409c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:19.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:19.668 7f31e3fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f31e409bcc0 0x7f31e409c0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:54496/0 (socket says 172.21.2.112:54496) 2021-02-10T11:40:19.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:19.668 7f31e3fff700 1 -- 172.21.2.112:0/4015197850 learned_addr learned my addr 172.21.2.112:0/4015197850 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:40:19.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:19.669 7f31e3fff700 1 -- 172.21.2.112:0/4015197850 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f31e40a0820 msgr2=0x7f31e40a0c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:19.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:19.669 7f31e3fff700 1 --2- 172.21.2.112:0/4015197850 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f31e40a0820 0x7f31e40a0c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:19.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:19.669 7f31e3fff700 1 -- 172.21.2.112:0/4015197850 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f31e40a1240 msgr2=0x7f31e40858d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:19.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:19.669 7f31e3fff700 1 --2- 172.21.2.112:0/4015197850 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f31e40a1240 0x7f31e40858d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:19.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:19.669 7f31e3fff700 1 -- 172.21.2.112:0/4015197850 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31e40a1640 con 0x7f31e409bcc0 2021-02-10T11:40:19.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:19.669 7f31e3fff700 1 --2- 172.21.2.112:0/4015197850 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f31e409bcc0 0x7f31e409c0c0 secure :-1 s=READY pgs=2036 cs=0 l=1 rev1=1 rx=0x7f31d400a5d0 tx=0x7f31d400a970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:19.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:19.669 7f31e0ff9700 1 -- 172.21.2.112:0/4015197850 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f31d4000ce0 con 0x7f31e409bcc0 2021-02-10T11:40:19.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:19.669 7f31e0ff9700 1 -- 172.21.2.112:0/4015197850 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31d40043a0 con 0x7f31e409bcc0 2021-02-10T11:40:19.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:19.669 7f31e0ff9700 1 -- 172.21.2.112:0/4015197850 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f31d40047f0 con 0x7f31e409bcc0 2021-02-10T11:40:19.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:19.669 7f31ea3c9700 1 -- 172.21.2.112:0/4015197850 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31e40a1640 con 0x7f31e409bcc0 2021-02-10T11:40:19.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:19.669 7f31ea3c9700 1 -- 172.21.2.112:0/4015197850 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31e40a1640 con 0x7f31e409bcc0 2021-02-10T11:40:19.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:19.670 7f31e0ff9700 1 -- 172.21.2.112:0/4015197850 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f31d4004790 con 0x7f31e409bcc0 2021-02-10T11:40:19.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:19.670 7f31e0ff9700 1 --2- 172.21.2.112:0/4015197850 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f31c40186c0 0x7f31c401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:19.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:19.670 7f31e0ff9700 1 -- 172.21.2.112:0/4015197850 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f31d4032780 con 0x7f31e409bcc0 2021-02-10T11:40:19.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:19.670 7f31e37fe700 1 --2- 172.21.2.112:0/4015197850 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f31c40186c0 0x7f31c401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:19.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:19.671 7f31e37fe700 1 --2- 172.21.2.112:0/4015197850 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f31c40186c0 0x7f31c401abc0 crc :-1 s=READY pgs=946 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:19.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:19.701 7f31ea3c9700 1 -- 172.21.2.112:0/4015197850 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31e40fe1c0 con 0x7f31e409bcc0 2021-02-10T11:40:19.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:19.704 7f31e0ff9700 1 -- 172.21.2.112:0/4015197850 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f31d40043a0 con 0x7f31e409bcc0 2021-02-10T11:40:19.846 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:19.851 7f31ea3c9700 1 -- 172.21.2.112:0/4015197850 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f31e40fe1c0 con 0x7f31e409bcc0 2021-02-10T11:40:19.850 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:19.855 7f31e0ff9700 1 -- 172.21.2.112:0/4015197850 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v28)=0 v28) v1 ==== 122+0+0 (secure 0 0 0) 0x7f31d40045b0 con 0x7f31e409bcc0 2021-02-10T11:40:19.855 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:19.860 7f31ea3c9700 1 -- 172.21.2.112:0/4015197850 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f31c40186c0 msgr2=0x7f31c401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:19.855 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:19.860 7f31ea3c9700 1 --2- 172.21.2.112:0/4015197850 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f31c40186c0 0x7f31c401abc0 crc :-1 s=READY pgs=946 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:19.855 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:19.860 7f31ea3c9700 1 -- 172.21.2.112:0/4015197850 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f31e409bcc0 msgr2=0x7f31e409c0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:19.855 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:19.860 7f31ea3c9700 1 --2- 172.21.2.112:0/4015197850 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f31e409bcc0 0x7f31e409c0c0 secure :-1 s=READY pgs=2036 cs=0 l=1 rev1=1 rx=0x7f31d400a5d0 tx=0x7f31d400a970).stop 2021-02-10T11:40:19.855 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:19.860 7f31ea3c9700 1 -- 172.21.2.112:0/4015197850 shutdown_connections 2021-02-10T11:40:19.855 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:19.860 7f31ea3c9700 1 --2- 172.21.2.112:0/4015197850 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f31c40186c0 0x7f31c401abc0 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:19.856 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:19.860 7f31ea3c9700 1 --2- 172.21.2.112:0/4015197850 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f31e40a1240 0x7f31e40858d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:19.856 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:19.860 7f31ea3c9700 1 --2- 172.21.2.112:0/4015197850 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f31e40a0820 0x7f31e40a0c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:19.856 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:19.860 7f31ea3c9700 1 --2- 172.21.2.112:0/4015197850 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f31e409bcc0 0x7f31e409c0c0 unknown :-1 s=CLOSED pgs=2036 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:19.856 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:19.860 7f31ea3c9700 1 -- 172.21.2.112:0/4015197850 >> 172.21.2.112:0/4015197850 conn(0x7f31e40bef00 msgr2=0x7f31e409d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:19.856 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:19.860 7f31ea3c9700 1 -- 172.21.2.112:0/4015197850 shutdown_connections 2021-02-10T11:40:19.856 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:19.860 7f31ea3c9700 1 -- 172.21.2.112:0/4015197850 wait complete. 2021-02-10T11:40:19.864 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:40:19.864 DEBUG:teuthology.orchestra.run.gibba012:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:40:20.209 DEBUG:teuthology.orchestra.run.gibba020:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:40:20.552 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:40:20.552 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:40:20.553 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:40:20.553 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:40:20.887 INFO:teuthology.orchestra.run.gibba012.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.0’: No such file or directory 2021-02-10T11:40:20.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:40:20.887 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2021-02-10T11:40:20.887 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:40:20.888 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:40:20.888 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:40:21.223 INFO:teuthology.orchestra.run.gibba012.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: No such file or directory 2021-02-10T11:40:21.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:40:21.224 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2021-02-10T11:40:21.224 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:40:21.224 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:40:21.224 DEBUG:teuthology.orchestra.run.gibba020:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:40:21.559 INFO:teuthology.orchestra.run.gibba020.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:40:21.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:40:21.559 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:40:21.560 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:40:21.560 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:40:21.560 DEBUG:teuthology.orchestra.run.gibba020:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:40:21.779 DEBUG:teuthology.orchestra.run.gibba012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:40:21.783 DEBUG:teuthology.orchestra.run.gibba020:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:40:21.894 INFO:teuthology.orchestra.run.gibba020.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:40:21.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:40:21.894 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:40:21.894 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json-pretty 2021-02-10T11:40:22.326 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.314 7f036fabc700 1 Processor -- start 2021-02-10T11:40:22.327 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.314 7f036fabc700 1 -- start start 2021-02-10T11:40:22.327 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.314 7f036fabc700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f03680d0340 0x7f03680ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:22.327 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.314 7f036fabc700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03680f0b10 con 0x7f03680d1870 2021-02-10T11:40:22.327 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.314 7f036fabc700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03681b0be0 con 0x7f03680d1c70 2021-02-10T11:40:22.327 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.314 7f036fabc700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03680e5ed0 con 0x7f03680d0340 2021-02-10T11:40:22.327 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.314 7f036d8d8700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f03680d1870 0x7f03681b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:53966/0 (socket says 172.21.2.112:53966) 2021-02-10T11:40:22.327 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.314 7f036d8d8700 1 -- 172.21.2.112:0/686161118 learned_addr learned my addr 172.21.2.112:0/686161118 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:40:22.327 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.315 7f036c8d6700 1 -- 172.21.2.112:0/686161118 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1957278568 0 0) 0x7f035c0010b0 con 0x7f03680d1870 2021-02-10T11:40:22.327 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.315 7f036c8d6700 1 -- 172.21.2.112:0/686161118 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0354002fe0 con 0x7f03680d1870 2021-02-10T11:40:22.327 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.315 7f036d0d7700 1 --2- 172.21.2.112:0/686161118 >> v2:172.21.2.120:3300/0 conn(0x7f03680d0340 0x7f03680ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:22.327 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.315 7f036c8d6700 1 -- 172.21.2.112:0/686161118 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3198396755 0 0) 0x7f035c0010b0 con 0x7f03680d1870 2021-02-10T11:40:22.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.315 7f036c8d6700 1 -- 172.21.2.112:0/686161118 >> v2:172.21.2.120:3300/0 conn(0x7f03680d0340 msgr2=0x7f03680ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:22.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.315 7f036c8d6700 1 --2- 172.21.2.112:0/686161118 >> v2:172.21.2.120:3300/0 conn(0x7f03680d0340 0x7f03680ef640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:22.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.315 7f036c8d6700 1 -- 172.21.2.112:0/686161118 >> v1:172.21.2.120:6789/0 conn(0x7f03680d1c70 legacy=0x7f03680efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:22.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.315 7f036c8d6700 1 -- 172.21.2.112:0/686161118 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03680f0450 con 0x7f03680d1870 2021-02-10T11:40:22.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.315 7f036c8d6700 1 -- 172.21.2.112:0/686161118 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f035c0015b0 con 0x7f03680d1870 2021-02-10T11:40:22.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.315 7f036c8d6700 1 -- 172.21.2.112:0/686161118 >> v1:172.21.2.112:6789/0 conn(0x7f03680d1870 legacy=0x7f03681b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:22.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.315 7f036c8d6700 1 --2- 172.21.2.112:0/686161118 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0354003b10 0x7f0354005fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:22.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.315 7f036c8d6700 1 --2- 172.21.2.112:0/686161118 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f03540067c0 0x7f0354008cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:22.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.315 7f036c8d6700 1 --2- 172.21.2.112:0/686161118 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f03540095c0 0x7f035400bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:22.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.315 7f036c8d6700 1 -- 172.21.2.112:0/686161118 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f035400c310 con 0x7f03540095c0 2021-02-10T11:40:22.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.315 7f036c8d6700 1 -- 172.21.2.112:0/686161118 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f035400c530 con 0x7f03540067c0 2021-02-10T11:40:22.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.315 7f036c8d6700 1 -- 172.21.2.112:0/686161118 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f035400c750 con 0x7f0354003b10 2021-02-10T11:40:22.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.315 7f036e0d9700 1 --2- 172.21.2.112:0/686161118 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f03540095c0 0x7f035400bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:22.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.316 7f036d8d8700 1 --2- 172.21.2.112:0/686161118 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0354003b10 0x7f0354005fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:22.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.316 7f036d0d7700 1 --2- 172.21.2.112:0/686161118 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f03540067c0 0x7f0354008cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:22.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.316 7f036e0d9700 1 -- 172.21.2.112:0/686161118 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0354003b10 msgr2=0x7f0354005fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:22.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.316 7f036e0d9700 1 --2- 172.21.2.112:0/686161118 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0354003b10 0x7f0354005fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:22.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.316 7f036e0d9700 1 -- 172.21.2.112:0/686161118 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f03540067c0 msgr2=0x7f0354008cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:22.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.316 7f036e0d9700 1 --2- 172.21.2.112:0/686161118 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f03540067c0 0x7f0354008cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:22.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.316 7f036e0d9700 1 -- 172.21.2.112:0/686161118 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f035400c9f0 con 0x7f03540095c0 2021-02-10T11:40:22.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.316 7f036e0d9700 1 -- 172.21.2.112:0/686161118 reap_dead start 2021-02-10T11:40:22.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.316 7f036e0d9700 1 --2- 172.21.2.112:0/686161118 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f03540095c0 0x7f035400bac0 secure :-1 s=READY pgs=2038 cs=0 l=1 rev1=1 rx=0x7f036004d670 tx=0x7f036004de50).ready entity=mon.0 client_cookie=6eb37843566d07cf server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:22.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.316 7f036c8d6700 1 -- 172.21.2.112:0/686161118 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0360006a30 con 0x7f03540095c0 2021-02-10T11:40:22.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.316 7f036c8d6700 1 -- 172.21.2.112:0/686161118 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f036004f300 con 0x7f03540095c0 2021-02-10T11:40:22.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.316 7f036c8d6700 1 -- 172.21.2.112:0/686161118 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f036004f820 con 0x7f03540095c0 2021-02-10T11:40:22.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.316 7f036fabc700 1 -- 172.21.2.112:0/686161118 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f03540095c0 msgr2=0x7f035400bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:22.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.316 7f036fabc700 1 --2- 172.21.2.112:0/686161118 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f03540095c0 0x7f035400bac0 secure :-1 s=READY pgs=2038 cs=0 l=1 rev1=1 rx=0x7f036004d670 tx=0x7f036004de50).stop 2021-02-10T11:40:22.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.317 7f036fabc700 1 -- 172.21.2.112:0/686161118 shutdown_connections 2021-02-10T11:40:22.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.317 7f036fabc700 1 --2- 172.21.2.112:0/686161118 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f03540095c0 0x7f035400bac0 unknown :-1 s=CLOSED pgs=2038 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:22.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.317 7f036fabc700 1 -- 172.21.2.112:0/686161118 >> 172.21.2.112:0/686161118 conn(0x7f03680ce450 msgr2=0x7f03680d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:22.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.317 7f036fabc700 1 -- 172.21.2.112:0/686161118 shutdown_connections 2021-02-10T11:40:22.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.317 7f036fabc700 1 -- 172.21.2.112:0/686161118 wait complete. 2021-02-10T11:40:22.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.331 7f036fabc700 1 Processor -- start 2021-02-10T11:40:22.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.332 7f036fabc700 1 -- start start 2021-02-10T11:40:22.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.332 7f036fabc700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0368195650 0x7f03680b5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:22.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.332 7f036fabc700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f03680b5b80 0x7f03681a2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:22.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.332 7f036fabc700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f03680acfa0 0x7f03681a3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:22.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.332 7f036fabc700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03680b6010 con 0x7f03680acfa0 2021-02-10T11:40:22.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.332 7f036fabc700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03681a3a60 con 0x7f0368195650 2021-02-10T11:40:22.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.332 7f036fabc700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03681a3c40 con 0x7f03680b5b80 2021-02-10T11:40:22.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.332 7f036e0d9700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f03680acfa0 0x7f03681a3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:22.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.332 7f036e0d9700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f03680acfa0 0x7f03681a3440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:54518/0 (socket says 172.21.2.112:54518) 2021-02-10T11:40:22.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.332 7f036e0d9700 1 -- 172.21.2.112:0/709810732 learned_addr learned my addr 172.21.2.112:0/709810732 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:40:22.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.333 7f036e0d9700 1 -- 172.21.2.112:0/709810732 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f03680b5b80 msgr2=0x7f03681a2e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:22.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.333 7f036e0d9700 1 --2- 172.21.2.112:0/709810732 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f03680b5b80 0x7f03681a2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:22.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.333 7f036e0d9700 1 -- 172.21.2.112:0/709810732 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0368195650 msgr2=0x7f03680b5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:22.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.333 7f036e0d9700 1 --2- 172.21.2.112:0/709810732 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0368195650 0x7f03680b5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:22.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.333 7f036e0d9700 1 -- 172.21.2.112:0/709810732 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03680bcae0 con 0x7f03680acfa0 2021-02-10T11:40:22.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.333 7f036e0d9700 1 --2- 172.21.2.112:0/709810732 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f03680acfa0 0x7f03681a3440 secure :-1 s=READY pgs=2039 cs=0 l=1 rev1=1 rx=0x7f036004d1c0 tx=0x7f0360050d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:22.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.333 7f035a7fc700 1 -- 172.21.2.112:0/709810732 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f036004d6a0 con 0x7f03680acfa0 2021-02-10T11:40:22.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.333 7f035a7fc700 1 -- 172.21.2.112:0/709810732 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f036004d200 con 0x7f03680acfa0 2021-02-10T11:40:22.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.333 7f035a7fc700 1 -- 172.21.2.112:0/709810732 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f036004b790 con 0x7f03680acfa0 2021-02-10T11:40:22.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.333 7f036fabc700 1 -- 172.21.2.112:0/709810732 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03680bcae0 con 0x7f03680acfa0 2021-02-10T11:40:22.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.333 7f036fabc700 1 -- 172.21.2.112:0/709810732 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03680bcae0 con 0x7f03680acfa0 2021-02-10T11:40:22.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.334 7f035a7fc700 1 -- 172.21.2.112:0/709810732 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0360005830 con 0x7f03680acfa0 2021-02-10T11:40:22.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.334 7f035a7fc700 1 --2- 172.21.2.112:0/709810732 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f03500186c0 0x7f035001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:22.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.334 7f035a7fc700 1 -- 172.21.2.112:0/709810732 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f0360076910 con 0x7f03680acfa0 2021-02-10T11:40:22.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.334 7f036d8d8700 1 --2- 172.21.2.112:0/709810732 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f03500186c0 0x7f035001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:22.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.335 7f036d8d8700 1 --2- 172.21.2.112:0/709810732 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f03500186c0 0x7f035001abc0 crc :-1 s=READY pgs=947 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:22.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.364 7f036fabc700 1 -- 172.21.2.112:0/709810732 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03680b6010 con 0x7f03680acfa0 2021-02-10T11:40:22.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.367 7f035a7fc700 1 -- 172.21.2.112:0/709810732 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f036004d3e0 con 0x7f03680acfa0 2021-02-10T11:40:22.487 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.492 7f036fabc700 1 -- 172.21.2.112:0/709810732 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f03680b6010 con 0x7f03680acfa0 2021-02-10T11:40:22.488 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.493 7f035a7fc700 1 -- 172.21.2.112:0/709810732 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v100) v1 ==== 81+0+30365 (secure 0 0 0) 0x7f03600607f0 con 0x7f03680acfa0 2021-02-10T11:40:22.489 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:40:22.489 INFO:teuthology.orchestra.run.gibba012.stdout:{ 2021-02-10T11:40:22.489 INFO:teuthology.orchestra.run.gibba012.stdout: "epoch": 100, 2021-02-10T11:40:22.489 INFO:teuthology.orchestra.run.gibba012.stdout: "fsid": "f14f004e-0699-4584-ad81-3754258493de", 2021-02-10T11:40:22.489 INFO:teuthology.orchestra.run.gibba012.stdout: "created": "2021-02-10 11:13:26.501424", 2021-02-10T11:40:22.489 INFO:teuthology.orchestra.run.gibba012.stdout: "modified": "2021-02-10 11:39:45.313226", 2021-02-10T11:40:22.489 INFO:teuthology.orchestra.run.gibba012.stdout: "last_up_change": "2021-02-10 11:13:52.780316", 2021-02-10T11:40:22.490 INFO:teuthology.orchestra.run.gibba012.stdout: "last_in_change": "2021-02-10 11:13:52.780316", 2021-02-10T11:40:22.490 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-02-10T11:40:22.490 INFO:teuthology.orchestra.run.gibba012.stdout: "flags_num": 5799936, 2021-02-10T11:40:22.490 INFO:teuthology.orchestra.run.gibba012.stdout: "flags_set": [ 2021-02-10T11:40:22.490 INFO:teuthology.orchestra.run.gibba012.stdout: "pglog_hardlimit", 2021-02-10T11:40:22.490 INFO:teuthology.orchestra.run.gibba012.stdout: "purged_snapdirs", 2021-02-10T11:40:22.490 INFO:teuthology.orchestra.run.gibba012.stdout: "recovery_deletes", 2021-02-10T11:40:22.490 INFO:teuthology.orchestra.run.gibba012.stdout: "sortbitwise" 2021-02-10T11:40:22.490 INFO:teuthology.orchestra.run.gibba012.stdout: ], 2021-02-10T11:40:22.490 INFO:teuthology.orchestra.run.gibba012.stdout: "crush_version": 4, 2021-02-10T11:40:22.490 INFO:teuthology.orchestra.run.gibba012.stdout: "full_ratio": 0.94999998807907104, 2021-02-10T11:40:22.491 INFO:teuthology.orchestra.run.gibba012.stdout: "backfillfull_ratio": 0.89999997615814209, 2021-02-10T11:40:22.491 INFO:teuthology.orchestra.run.gibba012.stdout: "nearfull_ratio": 0.85000002384185791, 2021-02-10T11:40:22.491 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_snapshot": "", 2021-02-10T11:40:22.491 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_max": 21, 2021-02-10T11:40:22.491 INFO:teuthology.orchestra.run.gibba012.stdout: "max_osd": 8, 2021-02-10T11:40:22.491 INFO:teuthology.orchestra.run.gibba012.stdout: "require_min_compat_client": "jewel", 2021-02-10T11:40:22.491 INFO:teuthology.orchestra.run.gibba012.stdout: "min_compat_client": "jewel", 2021-02-10T11:40:22.491 INFO:teuthology.orchestra.run.gibba012.stdout: "require_osd_release": "nautilus", 2021-02-10T11:40:22.491 INFO:teuthology.orchestra.run.gibba012.stdout: "pools": [ 2021-02-10T11:40:22.491 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:40:22.491 INFO:teuthology.orchestra.run.gibba012.stdout: "pool": 1, 2021-02-10T11:40:22.491 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_name": "rbd", 2021-02-10T11:40:22.491 INFO:teuthology.orchestra.run.gibba012.stdout: "create_time": "2021-02-10 11:13:57.508758", 2021-02-10T11:40:22.492 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:40:22.492 INFO:teuthology.orchestra.run.gibba012.stdout: "flags_names": "hashpspool", 2021-02-10T11:40:22.492 INFO:teuthology.orchestra.run.gibba012.stdout: "type": 1, 2021-02-10T11:40:22.492 INFO:teuthology.orchestra.run.gibba012.stdout: "size": 2, 2021-02-10T11:40:22.492 INFO:teuthology.orchestra.run.gibba012.stdout: "min_size": 1, 2021-02-10T11:40:22.492 INFO:teuthology.orchestra.run.gibba012.stdout: "crush_rule": 0, 2021-02-10T11:40:22.492 INFO:teuthology.orchestra.run.gibba012.stdout: "object_hash": 2, 2021-02-10T11:40:22.492 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:40:22.492 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num": 8, 2021-02-10T11:40:22.492 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num": 8, 2021-02-10T11:40:22.492 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num_target": 8, 2021-02-10T11:40:22.492 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_target": 8, 2021-02-10T11:40:22.492 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_pending": 8, 2021-02-10T11:40:22.492 INFO:teuthology.orchestra.run.gibba012.stdout: "last_pg_merge_meta": { 2021-02-10T11:40:22.493 INFO:teuthology.orchestra.run.gibba012.stdout: "source_pgid": "0.0", 2021-02-10T11:40:22.493 INFO:teuthology.orchestra.run.gibba012.stdout: "ready_epoch": 0, 2021-02-10T11:40:22.493 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_started": 0, 2021-02-10T11:40:22.493 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_clean": 0, 2021-02-10T11:40:22.493 INFO:teuthology.orchestra.run.gibba012.stdout: "source_version": "0'0", 2021-02-10T11:40:22.493 INFO:teuthology.orchestra.run.gibba012.stdout: "target_version": "0'0" 2021-02-10T11:40:22.493 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:40:22.493 INFO:teuthology.orchestra.run.gibba012.stdout: "last_change": "16", 2021-02-10T11:40:22.493 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend": "0", 2021-02-10T11:40:22.493 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:40:22.493 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:40:22.493 INFO:teuthology.orchestra.run.gibba012.stdout: "auid": 0, 2021-02-10T11:40:22.493 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:40:22.494 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_seq": 0, 2021-02-10T11:40:22.494 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_epoch": 0, 2021-02-10T11:40:22.494 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_snaps": [], 2021-02-10T11:40:22.494 INFO:teuthology.orchestra.run.gibba012.stdout: "removed_snaps": "[]", 2021-02-10T11:40:22.494 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_bytes": 0, 2021-02-10T11:40:22.494 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_objects": 0, 2021-02-10T11:40:22.494 INFO:teuthology.orchestra.run.gibba012.stdout: "tiers": [], 2021-02-10T11:40:22.494 INFO:teuthology.orchestra.run.gibba012.stdout: "tier_of": -1, 2021-02-10T11:40:22.494 INFO:teuthology.orchestra.run.gibba012.stdout: "read_tier": -1, 2021-02-10T11:40:22.494 INFO:teuthology.orchestra.run.gibba012.stdout: "write_tier": -1, 2021-02-10T11:40:22.494 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_mode": "none", 2021-02-10T11:40:22.494 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_bytes": 0, 2021-02-10T11:40:22.494 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_objects": 0, 2021-02-10T11:40:22.495 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:40:22.495 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:40:22.495 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:40:22.495 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_flush_age": 0, 2021-02-10T11:40:22.495 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_evict_age": 0, 2021-02-10T11:40:22.495 INFO:teuthology.orchestra.run.gibba012.stdout: "erasure_code_profile": "", 2021-02-10T11:40:22.495 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_params": { 2021-02-10T11:40:22.495 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "none" 2021-02-10T11:40:22.495 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:40:22.495 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_period": 0, 2021-02-10T11:40:22.495 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_count": 0, 2021-02-10T11:40:22.495 INFO:teuthology.orchestra.run.gibba012.stdout: "use_gmt_hitset": true, 2021-02-10T11:40:22.495 INFO:teuthology.orchestra.run.gibba012.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:40:22.496 INFO:teuthology.orchestra.run.gibba012.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:40:22.496 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:40:22.496 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:40:22.496 INFO:teuthology.orchestra.run.gibba012.stdout: "grade_table": [], 2021-02-10T11:40:22.496 INFO:teuthology.orchestra.run.gibba012.stdout: "stripe_width": 0, 2021-02-10T11:40:22.496 INFO:teuthology.orchestra.run.gibba012.stdout: "expected_num_objects": 0, 2021-02-10T11:40:22.496 INFO:teuthology.orchestra.run.gibba012.stdout: "fast_read": false, 2021-02-10T11:40:22.496 INFO:teuthology.orchestra.run.gibba012.stdout: "options": {}, 2021-02-10T11:40:22.496 INFO:teuthology.orchestra.run.gibba012.stdout: "application_metadata": { 2021-02-10T11:40:22.496 INFO:teuthology.orchestra.run.gibba012.stdout: "rbd": {} 2021-02-10T11:40:22.496 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:40:22.496 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:40:22.496 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:40:22.497 INFO:teuthology.orchestra.run.gibba012.stdout: "pool": 20, 2021-02-10T11:40:22.497 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_name": "cephfs_metadata", 2021-02-10T11:40:22.497 INFO:teuthology.orchestra.run.gibba012.stdout: "create_time": "2021-02-10 11:39:42.230237", 2021-02-10T11:40:22.497 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:40:22.497 INFO:teuthology.orchestra.run.gibba012.stdout: "flags_names": "hashpspool", 2021-02-10T11:40:22.497 INFO:teuthology.orchestra.run.gibba012.stdout: "type": 1, 2021-02-10T11:40:22.497 INFO:teuthology.orchestra.run.gibba012.stdout: "size": 2, 2021-02-10T11:40:22.497 INFO:teuthology.orchestra.run.gibba012.stdout: "min_size": 1, 2021-02-10T11:40:22.497 INFO:teuthology.orchestra.run.gibba012.stdout: "crush_rule": 0, 2021-02-10T11:40:22.497 INFO:teuthology.orchestra.run.gibba012.stdout: "object_hash": 2, 2021-02-10T11:40:22.497 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:40:22.497 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num": 8, 2021-02-10T11:40:22.497 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num": 8, 2021-02-10T11:40:22.498 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num_target": 8, 2021-02-10T11:40:22.498 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_target": 8, 2021-02-10T11:40:22.498 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_pending": 8, 2021-02-10T11:40:22.498 INFO:teuthology.orchestra.run.gibba012.stdout: "last_pg_merge_meta": { 2021-02-10T11:40:22.498 INFO:teuthology.orchestra.run.gibba012.stdout: "source_pgid": "0.0", 2021-02-10T11:40:22.498 INFO:teuthology.orchestra.run.gibba012.stdout: "ready_epoch": 0, 2021-02-10T11:40:22.498 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_started": 0, 2021-02-10T11:40:22.498 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_clean": 0, 2021-02-10T11:40:22.498 INFO:teuthology.orchestra.run.gibba012.stdout: "source_version": "0'0", 2021-02-10T11:40:22.498 INFO:teuthology.orchestra.run.gibba012.stdout: "target_version": "0'0" 2021-02-10T11:40:22.498 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:40:22.498 INFO:teuthology.orchestra.run.gibba012.stdout: "last_change": "99", 2021-02-10T11:40:22.498 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend": "0", 2021-02-10T11:40:22.499 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:40:22.499 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:40:22.499 INFO:teuthology.orchestra.run.gibba012.stdout: "auid": 0, 2021-02-10T11:40:22.499 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:40:22.499 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_seq": 0, 2021-02-10T11:40:22.499 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_epoch": 0, 2021-02-10T11:40:22.499 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_snaps": [], 2021-02-10T11:40:22.499 INFO:teuthology.orchestra.run.gibba012.stdout: "removed_snaps": "[]", 2021-02-10T11:40:22.499 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_bytes": 0, 2021-02-10T11:40:22.499 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_objects": 0, 2021-02-10T11:40:22.499 INFO:teuthology.orchestra.run.gibba012.stdout: "tiers": [], 2021-02-10T11:40:22.499 INFO:teuthology.orchestra.run.gibba012.stdout: "tier_of": -1, 2021-02-10T11:40:22.500 INFO:teuthology.orchestra.run.gibba012.stdout: "read_tier": -1, 2021-02-10T11:40:22.500 INFO:teuthology.orchestra.run.gibba012.stdout: "write_tier": -1, 2021-02-10T11:40:22.500 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_mode": "none", 2021-02-10T11:40:22.500 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_bytes": 0, 2021-02-10T11:40:22.500 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_objects": 0, 2021-02-10T11:40:22.500 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:40:22.500 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:40:22.500 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:40:22.500 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_flush_age": 0, 2021-02-10T11:40:22.500 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_evict_age": 0, 2021-02-10T11:40:22.500 INFO:teuthology.orchestra.run.gibba012.stdout: "erasure_code_profile": "", 2021-02-10T11:40:22.500 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_params": { 2021-02-10T11:40:22.500 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "none" 2021-02-10T11:40:22.501 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:40:22.501 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_period": 0, 2021-02-10T11:40:22.501 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_count": 0, 2021-02-10T11:40:22.501 INFO:teuthology.orchestra.run.gibba012.stdout: "use_gmt_hitset": true, 2021-02-10T11:40:22.501 INFO:teuthology.orchestra.run.gibba012.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:40:22.501 INFO:teuthology.orchestra.run.gibba012.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:40:22.501 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:40:22.501 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:40:22.501 INFO:teuthology.orchestra.run.gibba012.stdout: "grade_table": [], 2021-02-10T11:40:22.501 INFO:teuthology.orchestra.run.gibba012.stdout: "stripe_width": 0, 2021-02-10T11:40:22.501 INFO:teuthology.orchestra.run.gibba012.stdout: "expected_num_objects": 0, 2021-02-10T11:40:22.501 INFO:teuthology.orchestra.run.gibba012.stdout: "fast_read": false, 2021-02-10T11:40:22.501 INFO:teuthology.orchestra.run.gibba012.stdout: "options": { 2021-02-10T11:40:22.502 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_autoscale_bias": 4, 2021-02-10T11:40:22.502 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_min": 16, 2021-02-10T11:40:22.502 INFO:teuthology.orchestra.run.gibba012.stdout: "recovery_priority": 5 2021-02-10T11:40:22.502 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:40:22.502 INFO:teuthology.orchestra.run.gibba012.stdout: "application_metadata": { 2021-02-10T11:40:22.502 INFO:teuthology.orchestra.run.gibba012.stdout: "cephfs": { 2021-02-10T11:40:22.502 INFO:teuthology.orchestra.run.gibba012.stdout: "metadata": "cephfs" 2021-02-10T11:40:22.502 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:40:22.502 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:40:22.502 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:40:22.502 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:40:22.502 INFO:teuthology.orchestra.run.gibba012.stdout: "pool": 21, 2021-02-10T11:40:22.502 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_name": "cephfs_data", 2021-02-10T11:40:22.503 INFO:teuthology.orchestra.run.gibba012.stdout: "create_time": "2021-02-10 11:39:43.354655", 2021-02-10T11:40:22.503 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:40:22.503 INFO:teuthology.orchestra.run.gibba012.stdout: "flags_names": "hashpspool", 2021-02-10T11:40:22.503 INFO:teuthology.orchestra.run.gibba012.stdout: "type": 1, 2021-02-10T11:40:22.503 INFO:teuthology.orchestra.run.gibba012.stdout: "size": 2, 2021-02-10T11:40:22.503 INFO:teuthology.orchestra.run.gibba012.stdout: "min_size": 1, 2021-02-10T11:40:22.503 INFO:teuthology.orchestra.run.gibba012.stdout: "crush_rule": 0, 2021-02-10T11:40:22.503 INFO:teuthology.orchestra.run.gibba012.stdout: "object_hash": 2, 2021-02-10T11:40:22.503 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:40:22.503 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num": 8, 2021-02-10T11:40:22.503 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num": 8, 2021-02-10T11:40:22.503 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num_target": 8, 2021-02-10T11:40:22.503 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_target": 8, 2021-02-10T11:40:22.503 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_pending": 8, 2021-02-10T11:40:22.504 INFO:teuthology.orchestra.run.gibba012.stdout: "last_pg_merge_meta": { 2021-02-10T11:40:22.504 INFO:teuthology.orchestra.run.gibba012.stdout: "source_pgid": "0.0", 2021-02-10T11:40:22.504 INFO:teuthology.orchestra.run.gibba012.stdout: "ready_epoch": 0, 2021-02-10T11:40:22.504 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_started": 0, 2021-02-10T11:40:22.504 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_clean": 0, 2021-02-10T11:40:22.504 INFO:teuthology.orchestra.run.gibba012.stdout: "source_version": "0'0", 2021-02-10T11:40:22.504 INFO:teuthology.orchestra.run.gibba012.stdout: "target_version": "0'0" 2021-02-10T11:40:22.504 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:40:22.504 INFO:teuthology.orchestra.run.gibba012.stdout: "last_change": "100", 2021-02-10T11:40:22.504 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend": "0", 2021-02-10T11:40:22.504 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:40:22.504 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:40:22.504 INFO:teuthology.orchestra.run.gibba012.stdout: "auid": 0, 2021-02-10T11:40:22.505 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:40:22.505 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_seq": 0, 2021-02-10T11:40:22.505 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_epoch": 0, 2021-02-10T11:40:22.505 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_snaps": [], 2021-02-10T11:40:22.505 INFO:teuthology.orchestra.run.gibba012.stdout: "removed_snaps": "[]", 2021-02-10T11:40:22.505 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_bytes": 0, 2021-02-10T11:40:22.505 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_objects": 0, 2021-02-10T11:40:22.505 INFO:teuthology.orchestra.run.gibba012.stdout: "tiers": [], 2021-02-10T11:40:22.505 INFO:teuthology.orchestra.run.gibba012.stdout: "tier_of": -1, 2021-02-10T11:40:22.505 INFO:teuthology.orchestra.run.gibba012.stdout: "read_tier": -1, 2021-02-10T11:40:22.505 INFO:teuthology.orchestra.run.gibba012.stdout: "write_tier": -1, 2021-02-10T11:40:22.505 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_mode": "none", 2021-02-10T11:40:22.506 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_bytes": 0, 2021-02-10T11:40:22.506 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_objects": 0, 2021-02-10T11:40:22.506 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:40:22.506 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:40:22.506 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:40:22.506 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_flush_age": 0, 2021-02-10T11:40:22.506 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_evict_age": 0, 2021-02-10T11:40:22.506 INFO:teuthology.orchestra.run.gibba012.stdout: "erasure_code_profile": "", 2021-02-10T11:40:22.506 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_params": { 2021-02-10T11:40:22.506 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "none" 2021-02-10T11:40:22.506 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:40:22.506 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_period": 0, 2021-02-10T11:40:22.506 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_count": 0, 2021-02-10T11:40:22.507 INFO:teuthology.orchestra.run.gibba012.stdout: "use_gmt_hitset": true, 2021-02-10T11:40:22.507 INFO:teuthology.orchestra.run.gibba012.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:40:22.507 INFO:teuthology.orchestra.run.gibba012.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:40:22.507 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:40:22.507 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:40:22.507 INFO:teuthology.orchestra.run.gibba012.stdout: "grade_table": [], 2021-02-10T11:40:22.507 INFO:teuthology.orchestra.run.gibba012.stdout: "stripe_width": 0, 2021-02-10T11:40:22.507 INFO:teuthology.orchestra.run.gibba012.stdout: "expected_num_objects": 0, 2021-02-10T11:40:22.507 INFO:teuthology.orchestra.run.gibba012.stdout: "fast_read": false, 2021-02-10T11:40:22.507 INFO:teuthology.orchestra.run.gibba012.stdout: "options": {}, 2021-02-10T11:40:22.507 INFO:teuthology.orchestra.run.gibba012.stdout: "application_metadata": { 2021-02-10T11:40:22.507 INFO:teuthology.orchestra.run.gibba012.stdout: "cephfs": { 2021-02-10T11:40:22.507 INFO:teuthology.orchestra.run.gibba012.stdout: "data": "cephfs" 2021-02-10T11:40:22.507 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:40:22.508 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:40:22.508 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:40:22.508 INFO:teuthology.orchestra.run.gibba012.stdout: ], 2021-02-10T11:40:22.508 INFO:teuthology.orchestra.run.gibba012.stdout: "osds": [ 2021-02-10T11:40:22.508 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:40:22.508 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 0, 2021-02-10T11:40:22.508 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "95444b8f-6370-4a1d-ae5f-df62651b77da", 2021-02-10T11:40:22.508 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:40:22.508 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:40:22.508 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:40:22.508 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:40:22.508 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:40:22.509 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:40:22.509 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:40:22.509 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 97, 2021-02-10T11:40:22.509 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:40:22.509 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:40:22.509 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:40:22.509 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:40:22.509 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:40:22.509 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:40:22.509 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6817", 2021-02-10T11:40:22.509 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:40:22.509 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:40:22.509 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:40:22.509 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:40:22.510 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6818", 2021-02-10T11:40:22.510 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:40:22.510 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:40:22.510 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:40:22.510 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:40:22.510 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:40:22.510 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:40:22.510 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:40:22.510 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:40:22.510 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6820", 2021-02-10T11:40:22.510 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:40:22.510 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:40:22.510 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:40:22.511 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:40:22.511 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6822", 2021-02-10T11:40:22.511 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:40:22.511 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:40:22.511 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:40:22.511 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:40:22.511 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:40:22.511 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:40:22.511 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:40:22.511 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:40:22.511 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6828", 2021-02-10T11:40:22.511 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:40:22.511 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:40:22.512 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:40:22.512 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:40:22.512 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6830", 2021-02-10T11:40:22.512 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:40:22.512 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:40:22.512 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:40:22.512 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:40:22.512 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:40:22.512 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:40:22.512 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:40:22.512 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:40:22.512 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6824", 2021-02-10T11:40:22.512 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:40:22.513 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:40:22.513 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:40:22.513 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:40:22.513 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6826", 2021-02-10T11:40:22.513 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:40:22.513 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:40:22.513 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:40:22.513 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:40:22.513 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.112:6818/36030", 2021-02-10T11:40:22.513 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.112:6822/36030", 2021-02-10T11:40:22.513 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.112:6830/36030", 2021-02-10T11:40:22.513 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.112:6826/36030", 2021-02-10T11:40:22.513 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:40:22.514 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:40:22.514 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:40:22.514 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:40:22.514 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:40:22.514 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:40:22.514 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 1, 2021-02-10T11:40:22.514 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "a23e217e-0d8e-486e-be07-712adfabd262", 2021-02-10T11:40:22.514 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:40:22.514 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:40:22.514 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:40:22.514 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:40:22.514 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:40:22.515 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:40:22.515 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:40:22.515 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 98, 2021-02-10T11:40:22.515 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:40:22.515 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:40:22.515 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:40:22.515 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:40:22.515 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:40:22.515 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:40:22.515 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6800", 2021-02-10T11:40:22.515 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:40:22.515 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:40:22.516 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:40:22.516 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:40:22.516 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6802", 2021-02-10T11:40:22.516 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:40:22.516 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:40:22.516 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:40:22.516 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:40:22.516 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:40:22.516 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:40:22.516 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:40:22.516 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:40:22.517 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6804", 2021-02-10T11:40:22.517 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:40:22.517 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:40:22.517 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:40:22.517 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:40:22.517 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6806", 2021-02-10T11:40:22.517 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:40:22.517 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:40:22.517 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:40:22.517 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:40:22.517 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:40:22.517 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:40:22.517 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:40:22.518 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:40:22.518 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6812", 2021-02-10T11:40:22.518 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:40:22.518 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:40:22.518 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:40:22.518 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:40:22.518 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6814", 2021-02-10T11:40:22.518 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:40:22.518 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:40:22.518 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:40:22.518 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:40:22.518 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:40:22.519 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:40:22.519 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:40:22.519 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:40:22.519 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6808", 2021-02-10T11:40:22.519 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:40:22.519 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:40:22.519 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:40:22.519 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:40:22.519 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6810", 2021-02-10T11:40:22.519 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:40:22.519 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:40:22.519 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:40:22.520 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:40:22.520 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.112:6802/36029", 2021-02-10T11:40:22.520 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.112:6806/36029", 2021-02-10T11:40:22.520 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.112:6814/36029", 2021-02-10T11:40:22.520 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.112:6810/36029", 2021-02-10T11:40:22.520 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:40:22.520 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:40:22.520 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:40:22.520 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:40:22.520 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:40:22.520 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:40:22.520 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 2, 2021-02-10T11:40:22.521 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "026afa8a-c0f1-4b35-9a76-53c00fe0c2fc", 2021-02-10T11:40:22.521 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:40:22.521 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:40:22.521 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:40:22.521 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:40:22.521 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:40:22.521 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:40:22.521 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:40:22.521 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 98, 2021-02-10T11:40:22.521 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:40:22.521 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:40:22.521 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:40:22.521 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:40:22.522 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:40:22.522 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:40:22.522 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6801", 2021-02-10T11:40:22.522 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:40:22.522 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:40:22.522 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:40:22.522 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:40:22.522 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6803", 2021-02-10T11:40:22.522 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:40:22.522 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:40:22.522 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:40:22.522 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:40:22.522 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:40:22.523 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:40:22.523 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:40:22.523 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:40:22.523 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6805", 2021-02-10T11:40:22.523 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:40:22.523 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:40:22.523 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:40:22.523 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:40:22.523 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6807", 2021-02-10T11:40:22.523 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:40:22.523 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:40:22.523 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:40:22.523 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:40:22.524 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:40:22.524 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:40:22.524 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:40:22.524 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:40:22.524 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6813", 2021-02-10T11:40:22.524 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:40:22.524 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:40:22.524 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:40:22.524 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:40:22.524 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6815", 2021-02-10T11:40:22.524 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:40:22.524 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:40:22.524 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:40:22.525 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:40:22.525 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:40:22.525 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:40:22.525 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:40:22.525 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:40:22.525 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6809", 2021-02-10T11:40:22.525 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:40:22.525 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:40:22.525 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:40:22.525 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:40:22.525 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6811", 2021-02-10T11:40:22.525 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:40:22.525 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:40:22.526 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:40:22.526 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:40:22.526 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.112:6803/36031", 2021-02-10T11:40:22.526 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.112:6807/36031", 2021-02-10T11:40:22.526 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.112:6815/36031", 2021-02-10T11:40:22.526 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.112:6811/36031", 2021-02-10T11:40:22.526 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:40:22.526 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:40:22.526 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:40:22.526 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:40:22.526 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:40:22.527 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:40:22.527 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 3, 2021-02-10T11:40:22.527 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "58795682-e00d-4173-bb84-f7075ca89a11", 2021-02-10T11:40:22.527 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:40:22.527 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:40:22.527 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:40:22.527 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:40:22.527 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:40:22.527 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:40:22.527 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:40:22.527 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 98, 2021-02-10T11:40:22.527 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:40:22.528 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:40:22.528 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:40:22.528 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:40:22.528 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:40:22.528 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:40:22.528 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6816", 2021-02-10T11:40:22.528 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:40:22.528 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:40:22.528 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:40:22.528 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:40:22.528 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6819", 2021-02-10T11:40:22.528 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:40:22.528 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:40:22.528 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:40:22.529 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:40:22.529 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:40:22.529 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:40:22.529 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:40:22.529 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:40:22.529 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6821", 2021-02-10T11:40:22.529 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:40:22.529 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:40:22.529 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:40:22.529 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:40:22.529 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6823", 2021-02-10T11:40:22.529 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:40:22.529 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:40:22.530 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:40:22.530 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:40:22.530 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:40:22.530 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:40:22.530 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:40:22.530 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:40:22.530 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6829", 2021-02-10T11:40:22.530 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:40:22.530 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:40:22.530 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:40:22.530 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:40:22.530 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6831", 2021-02-10T11:40:22.531 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:40:22.531 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:40:22.531 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:40:22.531 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:40:22.531 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:40:22.531 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:40:22.531 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:40:22.531 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:40:22.531 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6825", 2021-02-10T11:40:22.531 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:40:22.531 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:40:22.531 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:40:22.531 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:40:22.532 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6827", 2021-02-10T11:40:22.532 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:40:22.532 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:40:22.532 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:40:22.532 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:40:22.532 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.112:6819/36028", 2021-02-10T11:40:22.532 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.112:6823/36028", 2021-02-10T11:40:22.532 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.112:6831/36028", 2021-02-10T11:40:22.532 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.112:6827/36028", 2021-02-10T11:40:22.532 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:40:22.532 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:40:22.532 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:40:22.532 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:40:22.533 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:40:22.533 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:40:22.533 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 4, 2021-02-10T11:40:22.533 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "6763f8d6-53dc-4e45-bdf6-a6eabcf81e39", 2021-02-10T11:40:22.533 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:40:22.533 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:40:22.533 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:40:22.533 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:40:22.533 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:40:22.533 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:40:22.533 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:40:22.533 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 90, 2021-02-10T11:40:22.533 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:40:22.533 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:40:22.534 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:40:22.534 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:40:22.534 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:40:22.534 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:40:22.534 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6810", 2021-02-10T11:40:22.534 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:40:22.534 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:40:22.534 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:40:22.534 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:40:22.534 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6813", 2021-02-10T11:40:22.534 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:40:22.534 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:40:22.535 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:40:22.535 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:40:22.535 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:40:22.535 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:40:22.535 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:40:22.535 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:40:22.535 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6816", 2021-02-10T11:40:22.535 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:40:22.535 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:40:22.535 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:40:22.535 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:40:22.535 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6819", 2021-02-10T11:40:22.535 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:40:22.535 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:40:22.536 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:40:22.536 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:40:22.536 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:40:22.536 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:40:22.536 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:40:22.536 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:40:22.536 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6825", 2021-02-10T11:40:22.536 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:40:22.536 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:40:22.536 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:40:22.536 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:40:22.536 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6827", 2021-02-10T11:40:22.537 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:40:22.537 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:40:22.537 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:40:22.537 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:40:22.537 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:40:22.537 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:40:22.537 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:40:22.537 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:40:22.537 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6821", 2021-02-10T11:40:22.537 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:40:22.537 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:40:22.537 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:40:22.537 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:40:22.537 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6823", 2021-02-10T11:40:22.538 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:40:22.538 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:40:22.538 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:40:22.538 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:40:22.538 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.120:6813/36388", 2021-02-10T11:40:22.538 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.120:6819/36388", 2021-02-10T11:40:22.538 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.120:6827/36388", 2021-02-10T11:40:22.538 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.120:6823/36388", 2021-02-10T11:40:22.538 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:40:22.538 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:40:22.538 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:40:22.538 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:40:22.539 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:40:22.539 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:40:22.539 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 5, 2021-02-10T11:40:22.539 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "5d0e4dd6-0cb0-4bf8-8314-626acaa1b544", 2021-02-10T11:40:22.539 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:40:22.539 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:40:22.539 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:40:22.539 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:40:22.539 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:40:22.539 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:40:22.539 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:40:22.539 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 98, 2021-02-10T11:40:22.540 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:40:22.540 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:40:22.540 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:40:22.540 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:40:22.540 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:40:22.540 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:40:22.540 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6804", 2021-02-10T11:40:22.540 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:40:22.540 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:40:22.540 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:40:22.540 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:40:22.540 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6806", 2021-02-10T11:40:22.540 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:40:22.541 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:40:22.541 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:40:22.541 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:40:22.541 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:40:22.541 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:40:22.541 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:40:22.541 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:40:22.541 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6809", 2021-02-10T11:40:22.541 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:40:22.541 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:40:22.541 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:40:22.541 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:40:22.541 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6812", 2021-02-10T11:40:22.542 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:40:22.542 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:40:22.542 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:40:22.542 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:40:22.542 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:40:22.542 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:40:22.542 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:40:22.542 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:40:22.542 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6820", 2021-02-10T11:40:22.542 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:40:22.542 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:40:22.542 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:40:22.542 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:40:22.543 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6822", 2021-02-10T11:40:22.543 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:40:22.543 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:40:22.543 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:40:22.543 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:40:22.543 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:40:22.543 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:40:22.543 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:40:22.543 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:40:22.543 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6815", 2021-02-10T11:40:22.543 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:40:22.543 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:40:22.544 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:40:22.544 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:40:22.544 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6818", 2021-02-10T11:40:22.544 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:40:22.544 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:40:22.544 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:40:22.544 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:40:22.544 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.120:6806/36387", 2021-02-10T11:40:22.544 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.120:6812/36387", 2021-02-10T11:40:22.544 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.120:6822/36387", 2021-02-10T11:40:22.544 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.120:6818/36387", 2021-02-10T11:40:22.544 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:40:22.544 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:40:22.544 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:40:22.545 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:40:22.545 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:40:22.545 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:40:22.545 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 6, 2021-02-10T11:40:22.545 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "b010e187-4541-4b00-80dd-f948f6ab8f15", 2021-02-10T11:40:22.545 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:40:22.545 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:40:22.545 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:40:22.545 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:40:22.545 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:40:22.545 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:40:22.545 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:40:22.545 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 98, 2021-02-10T11:40:22.545 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:40:22.546 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:40:22.546 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:40:22.546 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:40:22.546 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:40:22.546 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:40:22.546 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6824", 2021-02-10T11:40:22.546 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:40:22.546 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:40:22.546 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:40:22.546 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:40:22.546 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6826", 2021-02-10T11:40:22.546 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:40:22.546 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:40:22.547 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:40:22.547 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:40:22.547 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:40:22.547 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:40:22.547 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:40:22.547 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:40:22.547 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6828", 2021-02-10T11:40:22.547 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:40:22.547 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:40:22.547 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:40:22.547 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:40:22.547 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6829", 2021-02-10T11:40:22.547 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:40:22.548 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:40:22.548 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:40:22.548 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:40:22.548 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:40:22.548 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:40:22.548 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:40:22.548 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:40:22.548 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6832", 2021-02-10T11:40:22.548 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:40:22.548 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:40:22.548 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:40:22.548 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:40:22.549 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6833", 2021-02-10T11:40:22.549 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:40:22.549 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:40:22.549 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:40:22.549 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:40:22.549 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:40:22.549 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:40:22.549 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:40:22.549 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:40:22.549 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6830", 2021-02-10T11:40:22.549 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:40:22.549 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:40:22.549 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:40:22.550 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:40:22.550 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6831", 2021-02-10T11:40:22.550 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:40:22.550 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:40:22.550 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:40:22.550 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:40:22.550 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.120:6826/36385", 2021-02-10T11:40:22.550 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.120:6829/36385", 2021-02-10T11:40:22.550 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.120:6833/36385", 2021-02-10T11:40:22.550 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.120:6831/36385", 2021-02-10T11:40:22.550 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:40:22.550 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:40:22.550 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:40:22.551 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:40:22.551 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:40:22.551 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:40:22.551 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 7, 2021-02-10T11:40:22.551 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "671e2772-33b7-497e-8e44-2178bd850bcf", 2021-02-10T11:40:22.551 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:40:22.551 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:40:22.551 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:40:22.551 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:40:22.551 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:40:22.551 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:40:22.551 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:40:22.552 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 98, 2021-02-10T11:40:22.552 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:40:22.552 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:40:22.552 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:40:22.552 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:40:22.552 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:40:22.552 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:40:22.552 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6802", 2021-02-10T11:40:22.552 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:40:22.552 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:40:22.552 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:40:22.552 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:40:22.553 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6803", 2021-02-10T11:40:22.553 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:40:22.553 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:40:22.553 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:40:22.553 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:40:22.553 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:40:22.553 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:40:22.553 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:40:22.553 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:40:22.553 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6805", 2021-02-10T11:40:22.553 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:40:22.553 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:40:22.553 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:40:22.554 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:40:22.554 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6807", 2021-02-10T11:40:22.554 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:40:22.554 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:40:22.554 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:40:22.554 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:40:22.554 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:40:22.554 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:40:22.554 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:40:22.554 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:40:22.554 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6814", 2021-02-10T11:40:22.554 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:40:22.554 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:40:22.555 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:40:22.555 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:40:22.555 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6817", 2021-02-10T11:40:22.555 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:40:22.555 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:40:22.555 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:40:22.555 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:40:22.555 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:40:22.555 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:40:22.555 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:40:22.555 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:40:22.555 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6808", 2021-02-10T11:40:22.555 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:40:22.556 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:40:22.556 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:40:22.556 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:40:22.556 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6811", 2021-02-10T11:40:22.556 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:40:22.556 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:40:22.556 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:40:22.556 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:40:22.556 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.120:6803/36386", 2021-02-10T11:40:22.556 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.120:6807/36386", 2021-02-10T11:40:22.556 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.120:6817/36386", 2021-02-10T11:40:22.556 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.120:6811/36386", 2021-02-10T11:40:22.556 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:40:22.557 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:40:22.557 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:40:22.557 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:40:22.557 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:40:22.557 INFO:teuthology.orchestra.run.gibba012.stdout: ], 2021-02-10T11:40:22.557 INFO:teuthology.orchestra.run.gibba012.stdout: "osd_xinfo": [ 2021-02-10T11:40:22.557 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:40:22.557 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 0, 2021-02-10T11:40:22.557 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:40:22.557 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:40:22.557 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:40:22.557 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:40:22.558 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:40:22.558 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:40:22.558 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:40:22.558 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 1, 2021-02-10T11:40:22.558 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:40:22.558 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:40:22.558 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:40:22.558 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:40:22.558 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:40:22.558 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:40:22.558 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:40:22.559 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 2, 2021-02-10T11:40:22.559 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:40:22.559 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:40:22.559 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:40:22.559 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:40:22.559 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:40:22.559 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:40:22.559 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:40:22.559 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 3, 2021-02-10T11:40:22.559 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:40:22.559 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:40:22.559 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:40:22.560 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:40:22.560 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:40:22.560 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:40:22.560 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:40:22.560 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 4, 2021-02-10T11:40:22.560 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:40:22.560 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:40:22.560 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:40:22.560 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:40:22.560 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:40:22.560 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:40:22.560 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:40:22.561 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 5, 2021-02-10T11:40:22.561 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:40:22.561 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:40:22.561 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:40:22.561 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:40:22.561 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:40:22.561 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:40:22.561 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:40:22.561 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 6, 2021-02-10T11:40:22.561 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:40:22.561 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:40:22.561 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:40:22.561 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:40:22.561 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:40:22.562 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:40:22.562 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:40:22.562 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 7, 2021-02-10T11:40:22.562 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:40:22.562 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:40:22.562 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:40:22.562 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:40:22.562 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:40:22.562 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:40:22.562 INFO:teuthology.orchestra.run.gibba012.stdout: ], 2021-02-10T11:40:22.562 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_upmap": [], 2021-02-10T11:40:22.562 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_upmap_items": [], 2021-02-10T11:40:22.563 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_temp": [], 2021-02-10T11:40:22.563 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_temp": [], 2021-02-10T11:40:22.563 INFO:teuthology.orchestra.run.gibba012.stdout: "blacklist": {}, 2021-02-10T11:40:22.563 INFO:teuthology.orchestra.run.gibba012.stdout: "erasure_code_profiles": { 2021-02-10T11:40:22.563 INFO:teuthology.orchestra.run.gibba012.stdout: "default": { 2021-02-10T11:40:22.563 INFO:teuthology.orchestra.run.gibba012.stdout: "crush-failure-domain": "osd", 2021-02-10T11:40:22.563 INFO:teuthology.orchestra.run.gibba012.stdout: "k": "2", 2021-02-10T11:40:22.563 INFO:teuthology.orchestra.run.gibba012.stdout: "m": "1", 2021-02-10T11:40:22.563 INFO:teuthology.orchestra.run.gibba012.stdout: "plugin": "jerasure", 2021-02-10T11:40:22.563 INFO:teuthology.orchestra.run.gibba012.stdout: "ruleset-failure-domain": "osd", 2021-02-10T11:40:22.563 INFO:teuthology.orchestra.run.gibba012.stdout: "technique": "reed_sol_van" 2021-02-10T11:40:22.563 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:40:22.563 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:40:22.563 INFO:teuthology.orchestra.run.gibba012.stdout: "removed_snaps_queue": [], 2021-02-10T11:40:22.564 INFO:teuthology.orchestra.run.gibba012.stdout: "new_removed_snaps": [], 2021-02-10T11:40:22.564 INFO:teuthology.orchestra.run.gibba012.stdout: "new_purged_snaps": [], 2021-02-10T11:40:22.564 INFO:teuthology.orchestra.run.gibba012.stdout: "crush_node_flags": {}, 2021-02-10T11:40:22.564 INFO:teuthology.orchestra.run.gibba012.stdout: "device_class_flags": {} 2021-02-10T11:40:22.564 INFO:teuthology.orchestra.run.gibba012.stdout:} 2021-02-10T11:40:22.565 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.495 7f036fabc700 1 -- 172.21.2.112:0/709810732 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f03500186c0 msgr2=0x7f035001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:22.565 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.495 7f036fabc700 1 --2- 172.21.2.112:0/709810732 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f03500186c0 0x7f035001abc0 crc :-1 s=READY pgs=947 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:22.565 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.495 7f036fabc700 1 -- 172.21.2.112:0/709810732 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f03680acfa0 msgr2=0x7f03681a3440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:22.565 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.495 7f036fabc700 1 --2- 172.21.2.112:0/709810732 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f03680acfa0 0x7f03681a3440 secure :-1 s=READY pgs=2039 cs=0 l=1 rev1=1 rx=0x7f036004d1c0 tx=0x7f0360050d10).stop 2021-02-10T11:40:22.565 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.495 7f036fabc700 1 -- 172.21.2.112:0/709810732 shutdown_connections 2021-02-10T11:40:22.565 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.495 7f036fabc700 1 --2- 172.21.2.112:0/709810732 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f03500186c0 0x7f035001abc0 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:22.565 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.495 7f036fabc700 1 --2- 172.21.2.112:0/709810732 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f03680acfa0 0x7f03681a3440 unknown :-1 s=CLOSED pgs=2039 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:22.566 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.495 7f036fabc700 1 --2- 172.21.2.112:0/709810732 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f03680b5b80 0x7f03681a2e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:22.566 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.495 7f036fabc700 1 --2- 172.21.2.112:0/709810732 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0368195650 0x7f03680b5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:22.566 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.495 7f036fabc700 1 -- 172.21.2.112:0/709810732 >> 172.21.2.112:0/709810732 conn(0x7f03680b3040 msgr2=0x7f03680c3250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:22.566 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.495 7f036fabc700 1 -- 172.21.2.112:0/709810732 shutdown_connections 2021-02-10T11:40:22.566 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.495 7f036fabc700 1 -- 172.21.2.112:0/709810732 wait complete. 2021-02-10T11:40:22.567 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:40:22.995 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.982 7f223a35f700 1 Processor -- start 2021-02-10T11:40:22.995 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.982 7f223a35f700 1 -- start start 2021-02-10T11:40:22.995 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.983 7f223a35f700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f22340d1870 0x7f22341b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:22.995 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.983 7f223a35f700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22340f0b10 con 0x7f22340f0650 2021-02-10T11:40:22.996 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.983 7f223a35f700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22340d0310 con 0x7f22341b0a10 2021-02-10T11:40:22.996 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.983 7f223a35f700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22340d0530 con 0x7f22340d1870 2021-02-10T11:40:22.996 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.983 7f223897c700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f22340f0650 0x7f22340cfb20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:53986/0 (socket says 172.21.2.112:53986) 2021-02-10T11:40:22.996 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.983 7f223897c700 1 -- 172.21.2.112:0/2850852627 learned_addr learned my addr 172.21.2.112:0/2850852627 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:40:22.996 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.983 7f2233fff700 1 --2- 172.21.2.112:0/2850852627 >> v2:172.21.2.120:3300/0 conn(0x7f22340d1870 0x7f22341b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:22.996 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.983 7f2232ffd700 1 -- 172.21.2.112:0/2850852627 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2781145416 0 0) 0x7f222c000d30 con 0x7f22340f0650 2021-02-10T11:40:22.996 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.983 7f2232ffd700 1 -- 172.21.2.112:0/2850852627 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f221c002fe0 con 0x7f22340f0650 2021-02-10T11:40:22.996 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.983 7f2232ffd700 1 -- 172.21.2.112:0/2850852627 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3836666044 0 0) 0x7f222c001460 con 0x7f22340f0650 2021-02-10T11:40:22.996 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.984 7f2232ffd700 1 -- 172.21.2.112:0/2850852627 >> v2:172.21.2.120:3300/0 conn(0x7f22340d1870 msgr2=0x7f22341b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:22.996 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.984 7f2232ffd700 1 --2- 172.21.2.112:0/2850852627 >> v2:172.21.2.120:3300/0 conn(0x7f22340d1870 0x7f22341b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:22.996 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.984 7f2232ffd700 1 -- 172.21.2.112:0/2850852627 >> v1:172.21.2.120:6789/0 conn(0x7f22341b0a10 legacy=0x7f22340cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:22.996 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.984 7f2232ffd700 1 -- 172.21.2.112:0/2850852627 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22340d1c70 con 0x7f22340f0650 2021-02-10T11:40:22.997 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.984 7f2232ffd700 1 -- 172.21.2.112:0/2850852627 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f222c001980 con 0x7f22340f0650 2021-02-10T11:40:22.997 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.984 7f2232ffd700 1 -- 172.21.2.112:0/2850852627 >> v1:172.21.2.112:6789/0 conn(0x7f22340f0650 legacy=0x7f22340cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:22.997 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.984 7f2232ffd700 1 --2- 172.21.2.112:0/2850852627 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f221c003af0 0x7f221c005f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:22.997 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.984 7f2232ffd700 1 --2- 172.21.2.112:0/2850852627 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f221c0067a0 0x7f221c008c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:22.997 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.984 7f2232ffd700 1 --2- 172.21.2.112:0/2850852627 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f221c009580 0x7f221c00baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:22.997 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.984 7f2232ffd700 1 -- 172.21.2.112:0/2850852627 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f221c00c2f0 con 0x7f221c003af0 2021-02-10T11:40:22.997 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.984 7f2232ffd700 1 -- 172.21.2.112:0/2850852627 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f221c00c510 con 0x7f221c009580 2021-02-10T11:40:22.997 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.984 7f2232ffd700 1 -- 172.21.2.112:0/2850852627 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f221c050740 con 0x7f221c0067a0 2021-02-10T11:40:22.997 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.984 7f2233fff700 1 --2- 172.21.2.112:0/2850852627 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f221c003af0 0x7f221c005f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:22.997 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.984 7f223897c700 1 --2- 172.21.2.112:0/2850852627 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f221c009580 0x7f221c00baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:22.997 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.984 7f22337fe700 1 --2- 172.21.2.112:0/2850852627 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f221c0067a0 0x7f221c008c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:22.997 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.984 7f2233fff700 1 -- 172.21.2.112:0/2850852627 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f221c0067a0 msgr2=0x7f221c008c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:22.997 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.984 7f2233fff700 1 --2- 172.21.2.112:0/2850852627 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f221c0067a0 0x7f221c008c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:22.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.984 7f2233fff700 1 -- 172.21.2.112:0/2850852627 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f221c009580 msgr2=0x7f221c00baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:22.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.984 7f2233fff700 1 --2- 172.21.2.112:0/2850852627 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f221c009580 0x7f221c00baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:22.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.984 7f2233fff700 1 -- 172.21.2.112:0/2850852627 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f221c0509a0 con 0x7f221c003af0 2021-02-10T11:40:22.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.984 7f223897c700 1 -- 172.21.2.112:0/2850852627 reap_dead start 2021-02-10T11:40:22.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.984 7f2233fff700 1 --2- 172.21.2.112:0/2850852627 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f221c003af0 0x7f221c005f00 secure :-1 s=READY pgs=2041 cs=0 l=1 rev1=1 rx=0x7f222400c870 tx=0x7f222400cc10).ready entity=mon.0 client_cookie=68666b0916623eef server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:22.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.984 7f2232ffd700 1 -- 172.21.2.112:0/2850852627 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f22240090d0 con 0x7f221c003af0 2021-02-10T11:40:22.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.984 7f2232ffd700 1 -- 172.21.2.112:0/2850852627 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2224009450 con 0x7f221c003af0 2021-02-10T11:40:22.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.984 7f2232ffd700 1 -- 172.21.2.112:0/2850852627 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2224009930 con 0x7f221c003af0 2021-02-10T11:40:22.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.985 7f223a35f700 1 -- 172.21.2.112:0/2850852627 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f221c003af0 msgr2=0x7f221c005f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:22.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.985 7f223a35f700 1 --2- 172.21.2.112:0/2850852627 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f221c003af0 0x7f221c005f00 secure :-1 s=READY pgs=2041 cs=0 l=1 rev1=1 rx=0x7f222400c870 tx=0x7f222400cc10).stop 2021-02-10T11:40:22.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.985 7f223a35f700 1 -- 172.21.2.112:0/2850852627 shutdown_connections 2021-02-10T11:40:22.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.985 7f223a35f700 1 --2- 172.21.2.112:0/2850852627 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f221c003af0 0x7f221c005f00 unknown :-1 s=CLOSED pgs=2041 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:22.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.985 7f223a35f700 1 -- 172.21.2.112:0/2850852627 >> 172.21.2.112:0/2850852627 conn(0x7f22340ce450 msgr2=0x7f22340d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:22.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.985 7f223a35f700 1 -- 172.21.2.112:0/2850852627 shutdown_connections 2021-02-10T11:40:22.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:22.985 7f223a35f700 1 -- 172.21.2.112:0/2850852627 wait complete. 2021-02-10T11:40:22.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.000 7f223a35f700 1 Processor -- start 2021-02-10T11:40:22.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.000 7f223a35f700 1 -- start start 2021-02-10T11:40:23.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.000 7f223a35f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2234195650 0x7f22340b5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:23.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.000 7f223a35f700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f22340b5b80 0x7f22341a2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:23.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.000 7f223a35f700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f22340acfa0 0x7f22341a3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:23.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.000 7f223a35f700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22340b6010 con 0x7f2234195650 2021-02-10T11:40:23.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.000 7f223a35f700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22341a3a60 con 0x7f22340b5b80 2021-02-10T11:40:23.001 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.000 7f223a35f700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22341a3c40 con 0x7f22340acfa0 2021-02-10T11:40:23.001 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.000 7f2233fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2234195650 0x7f22340b5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:23.001 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.000 7f2233fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2234195650 0x7f22340b5560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:54536/0 (socket says 172.21.2.112:54536) 2021-02-10T11:40:23.001 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.000 7f2233fff700 1 -- 172.21.2.112:0/3147469355 learned_addr learned my addr 172.21.2.112:0/3147469355 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:40:23.001 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.001 7f2233fff700 1 -- 172.21.2.112:0/3147469355 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f22340acfa0 msgr2=0x7f22341a3440 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:40:23.001 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.001 7f2233fff700 1 --2- 172.21.2.112:0/3147469355 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f22340acfa0 0x7f22341a3440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:23.001 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.001 7f2233fff700 1 -- 172.21.2.112:0/3147469355 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f22340b5b80 msgr2=0x7f22341a2e20 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:40:23.001 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.001 7f2233fff700 1 --2- 172.21.2.112:0/3147469355 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f22340b5b80 0x7f22341a2e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:23.001 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.001 7f2233fff700 1 -- 172.21.2.112:0/3147469355 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22340bcae0 con 0x7f2234195650 2021-02-10T11:40:23.001 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.001 7f2233fff700 1 --2- 172.21.2.112:0/3147469355 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2234195650 0x7f22340b5560 secure :-1 s=READY pgs=2042 cs=0 l=1 rev1=1 rx=0x7f22240065a0 tx=0x7f222400b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:23.001 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.001 7f2230ff9700 1 -- 172.21.2.112:0/3147469355 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f222400bd50 con 0x7f2234195650 2021-02-10T11:40:23.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.001 7f2230ff9700 1 -- 172.21.2.112:0/3147469355 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22240060c0 con 0x7f2234195650 2021-02-10T11:40:23.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.001 7f223a35f700 1 -- 172.21.2.112:0/3147469355 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22340bcae0 con 0x7f2234195650 2021-02-10T11:40:23.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.001 7f223a35f700 1 -- 172.21.2.112:0/3147469355 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22340bd070 con 0x7f2234195650 2021-02-10T11:40:23.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.001 7f2230ff9700 1 -- 172.21.2.112:0/3147469355 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2224002210 con 0x7f2234195650 2021-02-10T11:40:23.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.002 7f2230ff9700 1 -- 172.21.2.112:0/3147469355 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2224006950 con 0x7f2234195650 2021-02-10T11:40:23.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.002 7f2230ff9700 1 --2- 172.21.2.112:0/3147469355 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f22140186c0 0x7f221401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:23.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.002 7f2230ff9700 1 -- 172.21.2.112:0/3147469355 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f22240327c0 con 0x7f2234195650 2021-02-10T11:40:23.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.002 7f22337fe700 1 --2- 172.21.2.112:0/3147469355 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f22140186c0 0x7f221401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:23.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.003 7f22337fe700 1 --2- 172.21.2.112:0/3147469355 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f22140186c0 0x7f221401abc0 crc :-1 s=READY pgs=948 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:23.027 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.032 7f223a35f700 1 -- 172.21.2.112:0/3147469355 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f220c002fd0 con 0x7f2234195650 2021-02-10T11:40:23.030 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.035 7f2230ff9700 1 -- 172.21.2.112:0/3147469355 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2224006740 con 0x7f2234195650 2021-02-10T11:40:23.159 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.164 7f223a35f700 1 -- 172.21.2.112:0/3147469355 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f220c002de0 con 0x7f2234195650 2021-02-10T11:40:23.159 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.164 7f2230ff9700 1 -- 172.21.2.112:0/3147469355 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+2104 (secure 0 0 0) 0x7f222400bd50 con 0x7f2234195650 2021-02-10T11:40:23.160 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:40:23.160 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":81,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10011,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/3938237927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":3938237927},{"type":"v1","addr":"172.21.2.112:6833","nonce":3938237927}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":76}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:39:44.311361","modified":"2021-02-10 11:39:46.320259","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7633},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7633":{"gid":7633,"name":"b","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.2.120:6835/1537197833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":1537197833},{"type":"v1","addr":"172.21.2.120:6835","nonce":1537197833}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":10}]} 2021-02-10T11:40:23.160 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 81 2021-02-10T11:40:23.161 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.166 7f223a35f700 1 -- 172.21.2.112:0/3147469355 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f22140186c0 msgr2=0x7f221401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:23.161 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.166 7f223a35f700 1 --2- 172.21.2.112:0/3147469355 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f22140186c0 0x7f221401abc0 crc :-1 s=READY pgs=948 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:23.161 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.166 7f223a35f700 1 -- 172.21.2.112:0/3147469355 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2234195650 msgr2=0x7f22340b5560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:23.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.166 7f223a35f700 1 --2- 172.21.2.112:0/3147469355 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2234195650 0x7f22340b5560 secure :-1 s=READY pgs=2042 cs=0 l=1 rev1=1 rx=0x7f22240065a0 tx=0x7f222400b960).stop 2021-02-10T11:40:23.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.166 7f223a35f700 1 -- 172.21.2.112:0/3147469355 shutdown_connections 2021-02-10T11:40:23.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.166 7f223a35f700 1 --2- 172.21.2.112:0/3147469355 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f22140186c0 0x7f221401abc0 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:23.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.166 7f223a35f700 1 --2- 172.21.2.112:0/3147469355 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f22340b5b80 0x7f22341a2e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:23.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.166 7f223a35f700 1 --2- 172.21.2.112:0/3147469355 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f22340acfa0 0x7f22341a3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:23.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.166 7f223a35f700 1 --2- 172.21.2.112:0/3147469355 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2234195650 0x7f22340b5560 unknown :-1 s=CLOSED pgs=2042 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:23.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.166 7f223a35f700 1 -- 172.21.2.112:0/3147469355 >> 172.21.2.112:0/3147469355 conn(0x7f22340b3040 msgr2=0x7f22340c3250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:23.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.167 7f223a35f700 1 -- 172.21.2.112:0/3147469355 shutdown_connections 2021-02-10T11:40:23.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.167 7f223a35f700 1 -- 172.21.2.112:0/3147469355 wait complete. 2021-02-10T11:40:23.171 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs fail cephfs 2021-02-10T11:40:23.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.586 7f8ecdb4d700 1 Processor -- start 2021-02-10T11:40:23.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.586 7f8ecdb4d700 1 -- start start 2021-02-10T11:40:23.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.587 7f8ecdb4d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8ec80d1870 0x7f8ec81b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:23.600 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.587 7f8ecdb4d700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ec80f0b10 con 0x7f8ec80f0650 2021-02-10T11:40:23.600 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.587 7f8ecdb4d700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ec80d0310 con 0x7f8ec81b0a10 2021-02-10T11:40:23.600 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.587 7f8ecdb4d700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ec80e5ed0 con 0x7f8ec80d1870 2021-02-10T11:40:23.600 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.587 7f8ec7fff700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f8ec80f0650 0x7f8ec80cfb20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:54006/0 (socket says 172.21.2.112:54006) 2021-02-10T11:40:23.600 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.587 7f8ec7fff700 1 -- 172.21.2.112:0/1571929261 learned_addr learned my addr 172.21.2.112:0/1571929261 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:40:23.600 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.587 7f8ec67fc700 1 -- 172.21.2.112:0/1571929261 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1285281239 0 0) 0x7f8ec0001500 con 0x7f8ec80f0650 2021-02-10T11:40:23.600 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.587 7f8ec67fc700 1 -- 172.21.2.112:0/1571929261 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8eb4002fe0 con 0x7f8ec80f0650 2021-02-10T11:40:23.600 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.587 7f8ec77fe700 1 --2- 172.21.2.112:0/1571929261 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8ec80d1870 0x7f8ec81b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:23.600 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.588 7f8ec67fc700 1 -- 172.21.2.112:0/1571929261 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 685500139 0 0) 0x7f8ec0001460 con 0x7f8ec80f0650 2021-02-10T11:40:23.600 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.588 7f8ec67fc700 1 -- 172.21.2.112:0/1571929261 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8ec80d1870 msgr2=0x7f8ec81b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:23.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.588 7f8ec67fc700 1 --2- 172.21.2.112:0/1571929261 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8ec80d1870 0x7f8ec81b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:23.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.588 7f8ec67fc700 1 -- 172.21.2.112:0/1571929261 >> v1:172.21.2.120:6789/0 conn(0x7f8ec81b0a10 legacy=0x7f8ec80cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:23.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.588 7f8ec67fc700 1 -- 172.21.2.112:0/1571929261 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ec80d0530 con 0x7f8ec80f0650 2021-02-10T11:40:23.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.588 7f8ec67fc700 1 -- 172.21.2.112:0/1571929261 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f8ec0001980 con 0x7f8ec80f0650 2021-02-10T11:40:23.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.588 7f8ec67fc700 1 -- 172.21.2.112:0/1571929261 >> v1:172.21.2.112:6789/0 conn(0x7f8ec80f0650 legacy=0x7f8ec80cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:23.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.588 7f8ec67fc700 1 --2- 172.21.2.112:0/1571929261 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8eb4003b90 0x7f8eb4005fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:23.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.588 7f8ec67fc700 1 --2- 172.21.2.112:0/1571929261 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8eb4006800 0x7f8eb4008d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:23.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.588 7f8ec67fc700 1 --2- 172.21.2.112:0/1571929261 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8eb40095c0 0x7f8eb400bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:23.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.588 7f8ec67fc700 1 -- 172.21.2.112:0/1571929261 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8eb400c370 con 0x7f8eb4006800 2021-02-10T11:40:23.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.588 7f8ec67fc700 1 -- 172.21.2.112:0/1571929261 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8eb400c590 con 0x7f8eb40095c0 2021-02-10T11:40:23.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.588 7f8ec67fc700 1 -- 172.21.2.112:0/1571929261 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8eb400c7b0 con 0x7f8eb4003b90 2021-02-10T11:40:23.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.588 7f8ec77fe700 1 --2- 172.21.2.112:0/1571929261 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8ec80d1870 0x7f8ec81b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:40:23.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.588 7f8ec6ffd700 1 --2- 172.21.2.112:0/1571929261 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8eb4006800 0x7f8eb4008d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:23.602 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.588 7f8ec77fe700 1 --2- 172.21.2.112:0/1571929261 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8eb4003b90 0x7f8eb4005fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:23.602 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.588 7f8ec7fff700 1 --2- 172.21.2.112:0/1571929261 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8eb40095c0 0x7f8eb400bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:23.602 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.588 7f8ec6ffd700 1 -- 172.21.2.112:0/1571929261 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8eb4003b90 msgr2=0x7f8eb4005fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:23.602 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.588 7f8ec6ffd700 1 --2- 172.21.2.112:0/1571929261 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8eb4003b90 0x7f8eb4005fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:23.602 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.588 7f8ec6ffd700 1 -- 172.21.2.112:0/1571929261 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8eb40095c0 msgr2=0x7f8eb400bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:23.602 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.588 7f8ec6ffd700 1 --2- 172.21.2.112:0/1571929261 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8eb40095c0 0x7f8eb400bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:23.602 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.588 7f8ec6ffd700 1 -- 172.21.2.112:0/1571929261 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8eb400ca50 con 0x7f8eb4006800 2021-02-10T11:40:23.602 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.588 7f8ec7fff700 1 --2- 172.21.2.112:0/1571929261 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8eb40095c0 0x7f8eb400bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:40:23.602 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.588 7f8ec6ffd700 1 --2- 172.21.2.112:0/1571929261 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8eb4006800 0x7f8eb4008d20 secure :-1 s=READY pgs=2044 cs=0 l=1 rev1=1 rx=0x7f8eb0006ab0 tx=0x7f8eb004dad0).ready entity=mon.0 client_cookie=94e5fb1f39fe19e2 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:23.602 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.589 7f8ec67fc700 1 -- 172.21.2.112:0/1571929261 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8eb0002280 con 0x7f8eb4006800 2021-02-10T11:40:23.602 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.589 7f8ec67fc700 1 -- 172.21.2.112:0/1571929261 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8eb0000e20 con 0x7f8eb4006800 2021-02-10T11:40:23.602 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.589 7f8ec67fc700 1 -- 172.21.2.112:0/1571929261 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8eb00009b0 con 0x7f8eb4006800 2021-02-10T11:40:23.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.589 7f8ecdb4d700 1 -- 172.21.2.112:0/1571929261 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8eb4006800 msgr2=0x7f8eb4008d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:23.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.589 7f8ecdb4d700 1 --2- 172.21.2.112:0/1571929261 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8eb4006800 0x7f8eb4008d20 secure :-1 s=READY pgs=2044 cs=0 l=1 rev1=1 rx=0x7f8eb0006ab0 tx=0x7f8eb004dad0).stop 2021-02-10T11:40:23.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.589 7f8ec7fff700 1 -- 172.21.2.112:0/1571929261 reap_dead start 2021-02-10T11:40:23.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.589 7f8ecdb4d700 1 -- 172.21.2.112:0/1571929261 shutdown_connections 2021-02-10T11:40:23.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.589 7f8ecdb4d700 1 -- 172.21.2.112:0/1571929261 >> 172.21.2.112:0/1571929261 conn(0x7f8ec80ce450 msgr2=0x7f8ec80d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:23.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.589 7f8ecdb4d700 1 -- 172.21.2.112:0/1571929261 shutdown_connections 2021-02-10T11:40:23.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.589 7f8ecdb4d700 1 -- 172.21.2.112:0/1571929261 wait complete. 2021-02-10T11:40:23.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.604 7f8ecdb4d700 1 Processor -- start 2021-02-10T11:40:23.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.604 7f8ecdb4d700 1 -- start start 2021-02-10T11:40:23.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.605 7f8ecdb4d700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8ec81a0550 0x7f8ec80c2390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:23.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.605 7f8ecdb4d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8ec80c29b0 0x7f8ec80ae8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:23.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.605 7f8ecdb4d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8ec80aef10 0x7f8ec80af310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:23.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.605 7f8ecdb4d700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ec80c2e30 con 0x7f8ec80aef10 2021-02-10T11:40:23.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.605 7f8ecdb4d700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ec80c31b0 con 0x7f8ec81a0550 2021-02-10T11:40:23.606 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.605 7f8ecdb4d700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ec80edf90 con 0x7f8ec80c29b0 2021-02-10T11:40:23.606 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.605 7f8ec7fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8ec80aef10 0x7f8ec80af310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:23.606 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.605 7f8ec7fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8ec80aef10 0x7f8ec80af310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:54556/0 (socket says 172.21.2.112:54556) 2021-02-10T11:40:23.606 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.605 7f8ec7fff700 1 -- 172.21.2.112:0/2988493332 learned_addr learned my addr 172.21.2.112:0/2988493332 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:40:23.606 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.605 7f8ec7fff700 1 -- 172.21.2.112:0/2988493332 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8ec80c29b0 msgr2=0x7f8ec80ae8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:23.606 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.605 7f8ec7fff700 1 --2- 172.21.2.112:0/2988493332 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8ec80c29b0 0x7f8ec80ae8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:23.606 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.605 7f8ec7fff700 1 -- 172.21.2.112:0/2988493332 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8ec81a0550 msgr2=0x7f8ec80c2390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:23.606 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.605 7f8ec7fff700 1 --2- 172.21.2.112:0/2988493332 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8ec81a0550 0x7f8ec80c2390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:23.606 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.605 7f8ec7fff700 1 -- 172.21.2.112:0/2988493332 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ec80ee1b0 con 0x7f8ec80aef10 2021-02-10T11:40:23.606 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.605 7f8ec7fff700 1 --2- 172.21.2.112:0/2988493332 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8ec80aef10 0x7f8ec80af310 secure :-1 s=READY pgs=2045 cs=0 l=1 rev1=1 rx=0x7f8ec0007f50 tx=0x7f8ec000aca0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:23.607 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.605 7f8ebffff700 1 -- 172.21.2.112:0/2988493332 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8ec0004930 con 0x7f8ec80aef10 2021-02-10T11:40:23.607 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.605 7f8ebffff700 1 -- 172.21.2.112:0/2988493332 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ec0001780 con 0x7f8ec80aef10 2021-02-10T11:40:23.607 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.605 7f8ebffff700 1 -- 172.21.2.112:0/2988493332 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8ec0007020 con 0x7f8ec80aef10 2021-02-10T11:40:23.607 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.605 7f8ecdb4d700 1 -- 172.21.2.112:0/2988493332 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ec80ee1b0 con 0x7f8ec80aef10 2021-02-10T11:40:23.607 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.606 7f8ecdb4d700 1 -- 172.21.2.112:0/2988493332 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ec80ee1b0 con 0x7f8ec80aef10 2021-02-10T11:40:23.607 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.606 7f8ebffff700 1 -- 172.21.2.112:0/2988493332 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8ec0007700 con 0x7f8ec80aef10 2021-02-10T11:40:23.607 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.606 7f8ebffff700 1 --2- 172.21.2.112:0/2988493332 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8eac0186c0 0x7f8eac01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:23.607 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.606 7f8ebffff700 1 -- 172.21.2.112:0/2988493332 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f8ec00327b0 con 0x7f8ec80aef10 2021-02-10T11:40:23.607 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.607 7f8ec77fe700 1 --2- 172.21.2.112:0/2988493332 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8eac0186c0 0x7f8eac01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:23.607 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.608 7f8ec77fe700 1 --2- 172.21.2.112:0/2988493332 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8eac0186c0 0x7f8eac01abc0 crc :-1 s=READY pgs=949 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:23.632 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.637 7f8ecdb4d700 1 -- 172.21.2.112:0/2988493332 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ec80f0b10 con 0x7f8ec80aef10 2021-02-10T11:40:23.635 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.640 7f8ebffff700 1 -- 172.21.2.112:0/2988493332 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8ec0024f70 con 0x7f8ec80aef10 2021-02-10T11:40:23.766 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:23.771 7f8ecdb4d700 1 -- 172.21.2.112:0/2988493332 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f8ec80f0b10 con 0x7f8ec80aef10 2021-02-10T11:40:24.171 INFO:tasks.ceph.mon.a.gibba012.stderr:2021-02-10 11:40:24.176 7fc3fe6e2700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-02-10T11:40:24.176 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:24.181 7f8ebffff700 1 -- 172.21.2.112:0/2988493332 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v82) v1 ==== 161+0+0 (secure 0 0 0) 0x7f8ec00076e0 con 0x7f8ec80aef10 2021-02-10T11:40:24.182 INFO:teuthology.orchestra.run.gibba012.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2021-02-10T11:40:24.183 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:24.188 7f8ecdb4d700 1 -- 172.21.2.112:0/2988493332 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8eac0186c0 msgr2=0x7f8eac01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:24.183 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:24.188 7f8ecdb4d700 1 --2- 172.21.2.112:0/2988493332 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8eac0186c0 0x7f8eac01abc0 crc :-1 s=READY pgs=949 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:24.184 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:24.188 7f8ecdb4d700 1 -- 172.21.2.112:0/2988493332 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8ec80aef10 msgr2=0x7f8ec80af310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:24.184 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:24.188 7f8ecdb4d700 1 --2- 172.21.2.112:0/2988493332 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8ec80aef10 0x7f8ec80af310 secure :-1 s=READY pgs=2045 cs=0 l=1 rev1=1 rx=0x7f8ec0007f50 tx=0x7f8ec000aca0).stop 2021-02-10T11:40:24.184 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:24.189 7f8ecdb4d700 1 -- 172.21.2.112:0/2988493332 shutdown_connections 2021-02-10T11:40:24.184 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:24.189 7f8ecdb4d700 1 --2- 172.21.2.112:0/2988493332 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8eac0186c0 0x7f8eac01abc0 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:24.184 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:24.189 7f8ecdb4d700 1 --2- 172.21.2.112:0/2988493332 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8ec80aef10 0x7f8ec80af310 unknown :-1 s=CLOSED pgs=2045 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:24.185 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:24.189 7f8ecdb4d700 1 --2- 172.21.2.112:0/2988493332 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8ec80c29b0 0x7f8ec80ae8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:24.185 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:24.189 7f8ecdb4d700 1 --2- 172.21.2.112:0/2988493332 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8ec81a0550 0x7f8ec80c2390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:24.185 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:24.189 7f8ecdb4d700 1 -- 172.21.2.112:0/2988493332 >> 172.21.2.112:0/2988493332 conn(0x7f8ec80beee0 msgr2=0x7f8ec80b8180 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:24.185 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:24.189 7f8ecdb4d700 1 -- 172.21.2.112:0/2988493332 shutdown_connections 2021-02-10T11:40:24.185 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:24.189 7f8ecdb4d700 1 -- 172.21.2.112:0/2988493332 wait complete. 2021-02-10T11:40:24.192 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:40:24.227 INFO:tasks.ceph.mds.b.gibba020.stdout:starting mds.b at 2021-02-10T11:40:24.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:24.608 7f0593bf9700 1 Processor -- start 2021-02-10T11:40:24.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:24.608 7f0593bf9700 1 -- start start 2021-02-10T11:40:24.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:24.609 7f0593bf9700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f058c0d1870 0x7f058c1b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:24.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:24.609 7f0593bf9700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f058c0f0b10 con 0x7f058c0f0650 2021-02-10T11:40:24.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:24.609 7f0593bf9700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f058c0d0310 con 0x7f058c1b0a10 2021-02-10T11:40:24.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:24.609 7f0593bf9700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f058c0d0530 con 0x7f058c0d1870 2021-02-10T11:40:24.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:24.609 7f0592216700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f058c0f0650 0x7f058c0cfb20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:54022/0 (socket says 172.21.2.112:54022) 2021-02-10T11:40:24.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:24.609 7f0592216700 1 -- 172.21.2.112:0/219049187 learned_addr learned my addr 172.21.2.112:0/219049187 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:40:24.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:24.609 7f0591a15700 1 --2- 172.21.2.112:0/219049187 >> v2:172.21.2.120:3300/0 conn(0x7f058c0d1870 0x7f058c1b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:24.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:24.610 7f0590a13700 1 -- 172.21.2.112:0/219049187 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1293848243 0 0) 0x7f0584001500 con 0x7f058c0f0650 2021-02-10T11:40:24.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:24.610 7f0590a13700 1 -- 172.21.2.112:0/219049187 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0578002fe0 con 0x7f058c0f0650 2021-02-10T11:40:24.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:24.610 7f0590a13700 1 -- 172.21.2.112:0/219049187 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1186052119 0 0) 0x7f0584001460 con 0x7f058c0f0650 2021-02-10T11:40:24.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:24.610 7f0590a13700 1 -- 172.21.2.112:0/219049187 >> v2:172.21.2.120:3300/0 conn(0x7f058c0d1870 msgr2=0x7f058c1b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:24.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:24.610 7f0590a13700 1 --2- 172.21.2.112:0/219049187 >> v2:172.21.2.120:3300/0 conn(0x7f058c0d1870 0x7f058c1b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:24.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:24.610 7f0590a13700 1 -- 172.21.2.112:0/219049187 >> v1:172.21.2.120:6789/0 conn(0x7f058c1b0a10 legacy=0x7f058c0cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:24.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:24.610 7f0590a13700 1 -- 172.21.2.112:0/219049187 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f058c0d1c70 con 0x7f058c0f0650 2021-02-10T11:40:24.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:24.610 7f0590a13700 1 -- 172.21.2.112:0/219049187 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f0584001980 con 0x7f058c0f0650 2021-02-10T11:40:24.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:24.610 7f0590a13700 1 -- 172.21.2.112:0/219049187 >> v1:172.21.2.112:6789/0 conn(0x7f058c0f0650 legacy=0x7f058c0cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:24.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:24.610 7f0590a13700 1 --2- 172.21.2.112:0/219049187 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0578003b20 0x7f0578005f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:24.624 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:24.610 7f0590a13700 1 --2- 172.21.2.112:0/219049187 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f05780067b0 0x7f0578008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:24.624 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:24.610 7f0590a13700 1 --2- 172.21.2.112:0/219049187 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0578009590 0x7f057800bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:24.624 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:24.610 7f0590a13700 1 -- 172.21.2.112:0/219049187 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f057800c300 con 0x7f05780067b0 2021-02-10T11:40:24.624 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:24.610 7f0590a13700 1 -- 172.21.2.112:0/219049187 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f057800c520 con 0x7f0578003b20 2021-02-10T11:40:24.624 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:24.610 7f0590a13700 1 -- 172.21.2.112:0/219049187 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f057800c740 con 0x7f0578009590 2021-02-10T11:40:24.624 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:24.610 7f0591214700 1 --2- 172.21.2.112:0/219049187 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f05780067b0 0x7f0578008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:24.624 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:24.610 7f0591a15700 1 --2- 172.21.2.112:0/219049187 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0578003b20 0x7f0578005f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:24.624 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:24.610 7f0592216700 1 --2- 172.21.2.112:0/219049187 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0578009590 0x7f057800bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:24.624 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:24.611 7f0591214700 1 -- 172.21.2.112:0/219049187 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0578009590 msgr2=0x7f057800bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:24.624 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:24.611 7f0591214700 1 --2- 172.21.2.112:0/219049187 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0578009590 0x7f057800bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:24.624 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:24.611 7f0591214700 1 -- 172.21.2.112:0/219049187 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0578003b20 msgr2=0x7f0578005f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:24.624 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:24.611 7f0591214700 1 --2- 172.21.2.112:0/219049187 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0578003b20 0x7f0578005f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:24.624 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:24.611 7f0591214700 1 -- 172.21.2.112:0/219049187 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05780509b0 con 0x7f05780067b0 2021-02-10T11:40:24.624 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:24.611 7f0592216700 1 -- 172.21.2.112:0/219049187 reap_dead start 2021-02-10T11:40:24.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:24.611 7f0591214700 1 --2- 172.21.2.112:0/219049187 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f05780067b0 0x7f0578008c90 secure :-1 s=READY pgs=2048 cs=0 l=1 rev1=1 rx=0x7f0588006660 tx=0x7f0588009a90).ready entity=mon.0 client_cookie=dc25697d47cf2915 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:24.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:24.611 7f0590a13700 1 -- 172.21.2.112:0/219049187 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0588008560 con 0x7f05780067b0 2021-02-10T11:40:24.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:24.611 7f0590a13700 1 -- 172.21.2.112:0/219049187 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05880087a0 con 0x7f05780067b0 2021-02-10T11:40:24.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:24.611 7f0590a13700 1 -- 172.21.2.112:0/219049187 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0588008cc0 con 0x7f05780067b0 2021-02-10T11:40:24.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:24.611 7f0593bf9700 1 -- 172.21.2.112:0/219049187 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f05780067b0 msgr2=0x7f0578008c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:24.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:24.611 7f0593bf9700 1 --2- 172.21.2.112:0/219049187 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f05780067b0 0x7f0578008c90 secure :-1 s=READY pgs=2048 cs=0 l=1 rev1=1 rx=0x7f0588006660 tx=0x7f0588009a90).stop 2021-02-10T11:40:24.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:24.612 7f0593bf9700 1 -- 172.21.2.112:0/219049187 shutdown_connections 2021-02-10T11:40:24.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:24.612 7f0593bf9700 1 --2- 172.21.2.112:0/219049187 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f05780067b0 0x7f0578008c90 unknown :-1 s=CLOSED pgs=2048 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:24.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:24.612 7f0593bf9700 1 -- 172.21.2.112:0/219049187 >> 172.21.2.112:0/219049187 conn(0x7f058c0ce450 msgr2=0x7f058c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:24.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:24.612 7f0593bf9700 1 -- 172.21.2.112:0/219049187 shutdown_connections 2021-02-10T11:40:24.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:24.612 7f0593bf9700 1 -- 172.21.2.112:0/219049187 wait complete. 2021-02-10T11:40:24.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:24.627 7f0593bf9700 1 Processor -- start 2021-02-10T11:40:24.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:24.627 7f0593bf9700 1 -- start start 2021-02-10T11:40:24.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:24.627 7f0593bf9700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f058c195650 0x7f058c0b5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:24.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:24.627 7f0593bf9700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f058c0b5b80 0x7f058c1a2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:24.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:24.627 7f0593bf9700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f058c0acfa0 0x7f058c1a3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:24.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:24.627 7f0593bf9700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f058c0b6010 con 0x7f058c0acfa0 2021-02-10T11:40:24.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:24.627 7f0593bf9700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f058c1a3a60 con 0x7f058c0b5b80 2021-02-10T11:40:24.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:24.627 7f0593bf9700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f058c1a3c40 con 0x7f058c195650 2021-02-10T11:40:24.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:24.627 7f0592216700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f058c0acfa0 0x7f058c1a3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:24.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:24.627 7f0592216700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f058c0acfa0 0x7f058c1a3440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:54574/0 (socket says 172.21.2.112:54574) 2021-02-10T11:40:24.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:24.627 7f0592216700 1 -- 172.21.2.112:0/1905817842 learned_addr learned my addr 172.21.2.112:0/1905817842 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:40:24.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:24.628 7f0592216700 1 -- 172.21.2.112:0/1905817842 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f058c195650 msgr2=0x7f058c0b5560 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:40:24.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:24.628 7f0592216700 1 --2- 172.21.2.112:0/1905817842 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f058c195650 0x7f058c0b5560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:24.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:24.628 7f0592216700 1 -- 172.21.2.112:0/1905817842 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f058c0b5b80 msgr2=0x7f058c1a2e20 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:40:24.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:24.628 7f0592216700 1 --2- 172.21.2.112:0/1905817842 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f058c0b5b80 0x7f058c1a2e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:24.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:24.628 7f0592216700 1 -- 172.21.2.112:0/1905817842 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f058c0bcae0 con 0x7f058c0acfa0 2021-02-10T11:40:24.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:24.628 7f0592216700 1 --2- 172.21.2.112:0/1905817842 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f058c0acfa0 0x7f058c1a3440 secure :-1 s=READY pgs=2049 cs=0 l=1 rev1=1 rx=0x7f0584001c30 tx=0x7f058400ac80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:24.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:24.628 7f057e7fc700 1 -- 172.21.2.112:0/1905817842 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0584000f90 con 0x7f058c0acfa0 2021-02-10T11:40:24.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:24.628 7f057e7fc700 1 -- 172.21.2.112:0/1905817842 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0584004930 con 0x7f058c0acfa0 2021-02-10T11:40:24.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:24.628 7f057e7fc700 1 -- 172.21.2.112:0/1905817842 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0584004dc0 con 0x7f058c0acfa0 2021-02-10T11:40:24.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:24.628 7f0593bf9700 1 -- 172.21.2.112:0/1905817842 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f058c0bcae0 con 0x7f058c0acfa0 2021-02-10T11:40:24.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:24.628 7f0593bf9700 1 -- 172.21.2.112:0/1905817842 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f058c0bcae0 con 0x7f058c0acfa0 2021-02-10T11:40:24.629 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:24.629 7f057e7fc700 1 -- 172.21.2.112:0/1905817842 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0584004b10 con 0x7f058c0acfa0 2021-02-10T11:40:24.629 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:24.629 7f057e7fc700 1 --2- 172.21.2.112:0/1905817842 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0574018700 0x7f057401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:24.629 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:24.629 7f057e7fc700 1 -- 172.21.2.112:0/1905817842 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5659+0+0 (secure 0 0 0) 0x7f0584032a00 con 0x7f058c0acfa0 2021-02-10T11:40:24.629 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:24.629 7f0591a15700 1 --2- 172.21.2.112:0/1905817842 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0574018700 0x7f057401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:24.629 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:24.630 7f0591a15700 1 --2- 172.21.2.112:0/1905817842 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0574018700 0x7f057401abc0 crc :-1 s=READY pgs=950 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:24.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:24.659 7f0593bf9700 1 -- 172.21.2.112:0/1905817842 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f058c0b6010 con 0x7f058c0acfa0 2021-02-10T11:40:24.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:24.661 7f057e7fc700 1 -- 172.21.2.112:0/1905817842 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0584004720 con 0x7f058c0acfa0 2021-02-10T11:40:24.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:24.789 7f0593bf9700 1 -- 172.21.2.112:0/1905817842 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f058c0b6010 con 0x7f058c0acfa0 2021-02-10T11:40:24.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:24.789 7f057e7fc700 1 -- 172.21.2.112:0/1905817842 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+1760 (secure 0 0 0) 0x7f0584004930 con 0x7f058c0acfa0 2021-02-10T11:40:24.785 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 82 2021-02-10T11:40:24.785 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:40:24.785 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":82,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10011,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/3938237927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":3938237927},{"type":"v1","addr":"172.21.2.112:6833","nonce":3938237927}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":76}],"filesystems":[{"mdsmap":{"epoch":82,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:39:44.311361","modified":"2021-02-10 11:40:24.176879","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":101,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":10}]} 2021-02-10T11:40:24.786 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:24.791 7f0593bf9700 1 -- 172.21.2.112:0/1905817842 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0574018700 msgr2=0x7f057401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:24.786 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:24.791 7f0593bf9700 1 --2- 172.21.2.112:0/1905817842 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0574018700 0x7f057401abc0 crc :-1 s=READY pgs=950 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:24.787 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:24.791 7f0593bf9700 1 -- 172.21.2.112:0/1905817842 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f058c0acfa0 msgr2=0x7f058c1a3440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:24.787 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:24.791 7f0593bf9700 1 --2- 172.21.2.112:0/1905817842 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f058c0acfa0 0x7f058c1a3440 secure :-1 s=READY pgs=2049 cs=0 l=1 rev1=1 rx=0x7f0584001c30 tx=0x7f058400ac80).stop 2021-02-10T11:40:24.787 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:24.792 7f0593bf9700 1 -- 172.21.2.112:0/1905817842 shutdown_connections 2021-02-10T11:40:24.787 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:24.792 7f0593bf9700 1 --2- 172.21.2.112:0/1905817842 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0574018700 0x7f057401abc0 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:24.788 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:24.792 7f0593bf9700 1 --2- 172.21.2.112:0/1905817842 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f058c0b5b80 0x7f058c1a2e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:24.788 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:24.792 7f0593bf9700 1 --2- 172.21.2.112:0/1905817842 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f058c0acfa0 0x7f058c1a3440 unknown :-1 s=CLOSED pgs=2049 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:24.788 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:24.792 7f0593bf9700 1 --2- 172.21.2.112:0/1905817842 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f058c195650 0x7f058c0b5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:24.788 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:24.792 7f0593bf9700 1 -- 172.21.2.112:0/1905817842 >> 172.21.2.112:0/1905817842 conn(0x7f058c0b3040 msgr2=0x7f058c0c3250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:24.788 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:24.792 7f0593bf9700 1 -- 172.21.2.112:0/1905817842 shutdown_connections 2021-02-10T11:40:24.788 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:24.792 7f0593bf9700 1 -- 172.21.2.112:0/1905817842 wait complete. 2021-02-10T11:40:24.795 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2021-02-10T11:40:25.227 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:25.215 7f29268ab700 1 Processor -- start 2021-02-10T11:40:25.227 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:25.215 7f29268ab700 1 -- start start 2021-02-10T11:40:25.227 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:25.215 7f29268ab700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f29200d1870 0x7f29201b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:25.227 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:25.215 7f29268ab700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f29201b0a10 0x7f29200cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:25.227 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:25.215 7f29268ab700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f29200d1c70 0x7f29200cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:25.227 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:25.215 7f29268ab700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29200e5ed0 con 0x7f29201b0a10 2021-02-10T11:40:25.227 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:25.215 7f29268ab700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29200f0b10 con 0x7f29200d1870 2021-02-10T11:40:25.228 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:25.215 7f29268ab700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29200cff70 con 0x7f29200d1c70 2021-02-10T11:40:25.228 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:25.215 7f291f7fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f29201b0a10 0x7f29200cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:25.228 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:25.215 7f291f7fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f29201b0a10 0x7f29200cf330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:54578/0 (socket says 172.21.2.112:54578) 2021-02-10T11:40:25.228 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:25.215 7f291f7fe700 1 -- 172.21.2.112:0/2918962694 learned_addr learned my addr 172.21.2.112:0/2918962694 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:40:25.228 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:25.216 7f2924ec8700 1 --2- 172.21.2.112:0/2918962694 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f29200d1c70 0x7f29200cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:25.228 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:25.216 7f291ffff700 1 --2- 172.21.2.112:0/2918962694 >> v2:172.21.2.120:3300/0 conn(0x7f29200d1870 0x7f29201b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:25.228 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:25.216 7f291f7fe700 1 -- 172.21.2.112:0/2918962694 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f29200d1c70 msgr2=0x7f29200cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:25.228 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:25.216 7f291f7fe700 1 --2- 172.21.2.112:0/2918962694 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f29200d1c70 0x7f29200cf950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:25.228 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:25.216 7f291f7fe700 1 -- 172.21.2.112:0/2918962694 >> v2:172.21.2.120:3300/0 conn(0x7f29200d1870 msgr2=0x7f29201b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:25.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:25.216 7f291f7fe700 1 --2- 172.21.2.112:0/2918962694 >> v2:172.21.2.120:3300/0 conn(0x7f29200d1870 0x7f29201b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:25.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:25.216 7f291f7fe700 1 -- 172.21.2.112:0/2918962694 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29200d0150 con 0x7f29201b0a10 2021-02-10T11:40:25.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:25.216 7f291ffff700 1 --2- 172.21.2.112:0/2918962694 >> v2:172.21.2.120:3300/0 conn(0x7f29200d1870 0x7f29201b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:40:25.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:25.216 7f291f7fe700 1 --2- 172.21.2.112:0/2918962694 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f29201b0a10 0x7f29200cf330 secure :-1 s=READY pgs=2050 cs=0 l=1 rev1=1 rx=0x7f2914006700 tx=0x7f2914009b00).ready entity=mon.0 client_cookie=7cfc2e312c8a7742 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:25.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:25.216 7f291effd700 1 -- 172.21.2.112:0/2918962694 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2914000c90 con 0x7f29201b0a10 2021-02-10T11:40:25.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:25.216 7f291effd700 1 -- 172.21.2.112:0/2918962694 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f291400c0c0 con 0x7f29201b0a10 2021-02-10T11:40:25.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:25.216 7f291effd700 1 -- 172.21.2.112:0/2918962694 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f291400c510 con 0x7f29201b0a10 2021-02-10T11:40:25.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:25.216 7f29268ab700 1 -- 172.21.2.112:0/2918962694 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f29201b0a10 msgr2=0x7f29200cf330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:25.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:25.216 7f29268ab700 1 --2- 172.21.2.112:0/2918962694 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f29201b0a10 0x7f29200cf330 secure :-1 s=READY pgs=2050 cs=0 l=1 rev1=1 rx=0x7f2914006700 tx=0x7f2914009b00).stop 2021-02-10T11:40:25.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:25.217 7f29268ab700 1 -- 172.21.2.112:0/2918962694 shutdown_connections 2021-02-10T11:40:25.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:25.217 7f29268ab700 1 --2- 172.21.2.112:0/2918962694 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f29200d1c70 0x7f29200cf950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:25.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:25.217 7f29268ab700 1 --2- 172.21.2.112:0/2918962694 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f29201b0a10 0x7f29200cf330 unknown :-1 s=CLOSED pgs=2050 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:25.230 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:25.217 7f29268ab700 1 --2- 172.21.2.112:0/2918962694 >> v2:172.21.2.120:3300/0 conn(0x7f29200d1870 0x7f29201b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:25.230 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:25.217 7f29268ab700 1 -- 172.21.2.112:0/2918962694 >> 172.21.2.112:0/2918962694 conn(0x7f29200ce450 msgr2=0x7f29200d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:25.230 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:25.217 7f29268ab700 1 -- 172.21.2.112:0/2918962694 shutdown_connections 2021-02-10T11:40:25.230 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:25.217 7f29268ab700 1 -- 172.21.2.112:0/2918962694 wait complete. 2021-02-10T11:40:25.230 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:25.232 7f29268ab700 1 Processor -- start 2021-02-10T11:40:25.231 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:25.232 7f29268ab700 1 -- start start 2021-02-10T11:40:25.231 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:25.232 7f29268ab700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2920094310 0x7f29200ebe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:25.231 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:25.232 7f29268ab700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2920094710 0x7f29200bca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:25.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:25.232 7f29268ab700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f29200ec490 0x7f29200bd050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:25.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:25.232 7f29268ab700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2920093d90 con 0x7f29200ec490 2021-02-10T11:40:25.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:25.232 7f29268ab700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29200bd670 con 0x7f2920094310 2021-02-10T11:40:25.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:25.232 7f29268ab700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29200939f0 con 0x7f2920094710 2021-02-10T11:40:25.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:25.232 7f291ffff700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2920094310 0x7f29200ebe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:25.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:25.233 7f2924ec8700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f29200ec490 0x7f29200bd050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:25.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:25.233 7f2924ec8700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f29200ec490 0x7f29200bd050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:54588/0 (socket says 172.21.2.112:54588) 2021-02-10T11:40:25.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:25.233 7f2924ec8700 1 -- 172.21.2.112:0/3978709308 learned_addr learned my addr 172.21.2.112:0/3978709308 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:40:25.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:25.233 7f291f7fe700 1 --2- 172.21.2.112:0/3978709308 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2920094710 0x7f29200bca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:25.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:25.233 7f2924ec8700 1 -- 172.21.2.112:0/3978709308 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2920094710 msgr2=0x7f29200bca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:25.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:25.233 7f2924ec8700 1 --2- 172.21.2.112:0/3978709308 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2920094710 0x7f29200bca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:25.233 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:25.233 7f2924ec8700 1 -- 172.21.2.112:0/3978709308 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2920094310 msgr2=0x7f29200ebe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:25.233 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:25.233 7f2924ec8700 1 --2- 172.21.2.112:0/3978709308 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2920094310 0x7f29200ebe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:25.233 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:25.233 7f2924ec8700 1 -- 172.21.2.112:0/3978709308 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29200bda80 con 0x7f29200ec490 2021-02-10T11:40:25.233 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:25.233 7f2924ec8700 1 --2- 172.21.2.112:0/3978709308 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f29200ec490 0x7f29200bd050 secure :-1 s=READY pgs=2051 cs=0 l=1 rev1=1 rx=0x7f291800b5e0 tx=0x7f291800b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:25.233 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:25.233 7f291cff9700 1 -- 172.21.2.112:0/3978709308 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f291800bd70 con 0x7f29200ec490 2021-02-10T11:40:25.233 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:25.233 7f291cff9700 1 -- 172.21.2.112:0/3978709308 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2918002030 con 0x7f29200ec490 2021-02-10T11:40:25.233 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:25.233 7f29268ab700 1 -- 172.21.2.112:0/3978709308 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29200bda80 con 0x7f29200ec490 2021-02-10T11:40:25.233 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:25.233 7f29268ab700 1 -- 172.21.2.112:0/3978709308 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29201a76d0 con 0x7f29200ec490 2021-02-10T11:40:25.233 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:25.233 7f291cff9700 1 -- 172.21.2.112:0/3978709308 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2918002550 con 0x7f29200ec490 2021-02-10T11:40:25.233 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:25.234 7f291cff9700 1 -- 172.21.2.112:0/3978709308 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2918002210 con 0x7f29200ec490 2021-02-10T11:40:25.233 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:25.234 7f291cff9700 1 --2- 172.21.2.112:0/3978709308 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2900018730 0x7f290001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:25.233 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:25.234 7f291cff9700 1 -- 172.21.2.112:0/3978709308 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5659+0+0 (secure 0 0 0) 0x7f29180326e0 con 0x7f29200ec490 2021-02-10T11:40:25.234 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:25.234 7f291ffff700 1 --2- 172.21.2.112:0/3978709308 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2900018730 0x7f290001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:25.234 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:25.235 7f291ffff700 1 --2- 172.21.2.112:0/3978709308 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2900018730 0x7f290001ac10 crc :-1 s=READY pgs=951 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:25.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:25.265 7f29268ab700 1 -- 172.21.2.112:0/3978709308 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29200ebc40 con 0x7f29200ec490 2021-02-10T11:40:25.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:25.268 7f291cff9700 1 -- 172.21.2.112:0/3978709308 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2918000c10 con 0x7f29200ec490 2021-02-10T11:40:25.391 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:25.396 7f29268ab700 1 -- 172.21.2.112:0/3978709308 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f29200f0b10 con 0x7f29200ec490 2021-02-10T11:40:26.182 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:26.187 7f291cff9700 1 -- 172.21.2.112:0/3978709308 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v83) v1 ==== 104+0+0 (secure 0 0 0) 0x7f2918006720 con 0x7f29200ec490 2021-02-10T11:40:26.211 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:26.216 7f29268ab700 1 -- 172.21.2.112:0/3978709308 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2900018730 msgr2=0x7f290001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:26.211 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:26.216 7f29268ab700 1 --2- 172.21.2.112:0/3978709308 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2900018730 0x7f290001ac10 crc :-1 s=READY pgs=951 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:26.211 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:26.216 7f29268ab700 1 -- 172.21.2.112:0/3978709308 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f29200ec490 msgr2=0x7f29200bd050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:26.211 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:26.216 7f29268ab700 1 --2- 172.21.2.112:0/3978709308 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f29200ec490 0x7f29200bd050 secure :-1 s=READY pgs=2051 cs=0 l=1 rev1=1 rx=0x7f291800b5e0 tx=0x7f291800b980).stop 2021-02-10T11:40:26.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:26.216 7f29268ab700 1 -- 172.21.2.112:0/3978709308 shutdown_connections 2021-02-10T11:40:26.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:26.216 7f29268ab700 1 --2- 172.21.2.112:0/3978709308 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2900018730 0x7f290001ac10 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:26.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:26.216 7f29268ab700 1 --2- 172.21.2.112:0/3978709308 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f29200ec490 0x7f29200bd050 unknown :-1 s=CLOSED pgs=2051 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:26.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:26.216 7f29268ab700 1 --2- 172.21.2.112:0/3978709308 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2920094710 0x7f29200bca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:26.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:26.216 7f29268ab700 1 --2- 172.21.2.112:0/3978709308 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2920094310 0x7f29200ebe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:26.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:26.216 7f29268ab700 1 -- 172.21.2.112:0/3978709308 >> 172.21.2.112:0/3978709308 conn(0x7f292008fa10 msgr2=0x7f29200bfa60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:26.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:26.216 7f29268ab700 1 -- 172.21.2.112:0/3978709308 shutdown_connections 2021-02-10T11:40:26.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:26.216 7f29268ab700 1 -- 172.21.2.112:0/3978709308 wait complete. 2021-02-10T11:40:26.219 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2021-02-10T11:40:26.651 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:26.639 7efd5f125700 1 Processor -- start 2021-02-10T11:40:26.651 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:26.639 7efd5f125700 1 -- start start 2021-02-10T11:40:26.651 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:26.640 7efd5f125700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efd580d1880 0x7efd581b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:26.651 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:26.640 7efd5f125700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7efd581b0af0 0x7efd58108950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:26.652 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:26.640 7efd5f125700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd58109a70 con 0x7efd580d1c80 2021-02-10T11:40:26.652 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:26.640 7efd5f125700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd580e5f00 con 0x7efd581b0af0 2021-02-10T11:40:26.652 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:26.640 7efd5f125700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd58109760 con 0x7efd580d1880 2021-02-10T11:40:26.652 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:26.640 7efd5d742700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7efd580d1c80 0x7efd58108f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:54054/0 (socket says 172.21.2.112:54054) 2021-02-10T11:40:26.652 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:26.640 7efd5d742700 1 -- 172.21.2.112:0/1882978053 learned_addr learned my addr 172.21.2.112:0/1882978053 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:40:26.652 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:26.640 7efd57fff700 1 --2- 172.21.2.112:0/1882978053 >> v2:172.21.2.120:3300/0 conn(0x7efd581b0af0 0x7efd58108950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:26.652 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:26.640 7efd5cf41700 1 --2- 172.21.2.112:0/1882978053 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efd580d1880 0x7efd581b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:26.652 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:26.640 7efd577fe700 1 -- 172.21.2.112:0/1882978053 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1376403413 0 0) 0x7efd50000c10 con 0x7efd580d1c80 2021-02-10T11:40:26.652 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:26.640 7efd577fe700 1 -- 172.21.2.112:0/1882978053 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd44002fe0 con 0x7efd580d1c80 2021-02-10T11:40:26.652 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:26.641 7efd57fff700 1 -- 172.21.2.112:0/1882978053 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efd580d1880 msgr2=0x7efd581b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:26.652 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:26.641 7efd57fff700 1 --2- 172.21.2.112:0/1882978053 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efd580d1880 0x7efd581b04d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:26.652 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:26.641 7efd57fff700 1 -- 172.21.2.112:0/1882978053 >> v1:172.21.2.112:6789/0 conn(0x7efd580d1c80 legacy=0x7efd58108f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:26.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:26.641 7efd57fff700 1 -- 172.21.2.112:0/1882978053 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd58088070 con 0x7efd581b0af0 2021-02-10T11:40:26.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:26.641 7efd57fff700 1 --2- 172.21.2.112:0/1882978053 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7efd581b0af0 0x7efd58108950 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 rx=0x7efd480069d0 tx=0x7efd48009e40).ready entity=mon.1 client_cookie=c236810007c7a1b4 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:26.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:26.641 7efd577fe700 1 -- 172.21.2.112:0/1882978053 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efd4800b510 con 0x7efd581b0af0 2021-02-10T11:40:26.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:26.641 7efd577fe700 1 -- 172.21.2.112:0/1882978053 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd4800b790 con 0x7efd581b0af0 2021-02-10T11:40:26.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:26.641 7efd577fe700 1 -- 172.21.2.112:0/1882978053 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efd4800bcb0 con 0x7efd581b0af0 2021-02-10T11:40:26.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:26.641 7efd5f125700 1 -- 172.21.2.112:0/1882978053 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7efd581b0af0 msgr2=0x7efd58108950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:26.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:26.641 7efd5f125700 1 --2- 172.21.2.112:0/1882978053 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7efd581b0af0 0x7efd58108950 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 rx=0x7efd480069d0 tx=0x7efd48009e40).stop 2021-02-10T11:40:26.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:26.642 7efd5f125700 1 -- 172.21.2.112:0/1882978053 shutdown_connections 2021-02-10T11:40:26.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:26.642 7efd5f125700 1 --2- 172.21.2.112:0/1882978053 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7efd581b0af0 0x7efd58108950 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:26.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:26.642 7efd5f125700 1 --2- 172.21.2.112:0/1882978053 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efd580d1880 0x7efd581b04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:26.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:26.642 7efd5f125700 1 -- 172.21.2.112:0/1882978053 >> 172.21.2.112:0/1882978053 conn(0x7efd580ce430 msgr2=0x7efd580d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:26.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:26.642 7efd5f125700 1 -- 172.21.2.112:0/1882978053 shutdown_connections 2021-02-10T11:40:26.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:26.642 7efd5f125700 1 -- 172.21.2.112:0/1882978053 wait complete. 2021-02-10T11:40:26.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:26.656 7efd5f125700 1 Processor -- start 2021-02-10T11:40:26.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:26.657 7efd5f125700 1 -- start start 2021-02-10T11:40:26.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:26.657 7efd5f125700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efd580855b0 0x7efd5819bc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:26.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:26.657 7efd5f125700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7efd58085e40 0x7efd58086240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:26.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:26.657 7efd5f125700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efd58086860 0x7efd580b0d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:26.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:26.657 7efd5f125700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd5819c240 con 0x7efd580855b0 2021-02-10T11:40:26.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:26.657 7efd5f125700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd58086c60 con 0x7efd58085e40 2021-02-10T11:40:26.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:26.657 7efd5f125700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd58086e80 con 0x7efd58086860 2021-02-10T11:40:26.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:26.657 7efd5cf41700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efd580855b0 0x7efd5819bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:26.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:26.657 7efd5cf41700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efd580855b0 0x7efd5819bc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:54598/0 (socket says 172.21.2.112:54598) 2021-02-10T11:40:26.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:26.657 7efd5cf41700 1 -- 172.21.2.112:0/3961606631 learned_addr learned my addr 172.21.2.112:0/3961606631 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:40:26.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:26.658 7efd5cf41700 1 -- 172.21.2.112:0/3961606631 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efd58086860 msgr2=0x7efd580b0d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:26.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:26.658 7efd5cf41700 1 --2- 172.21.2.112:0/3961606631 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efd58086860 0x7efd580b0d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:26.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:26.658 7efd5cf41700 1 -- 172.21.2.112:0/3961606631 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7efd58085e40 msgr2=0x7efd58086240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:26.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:26.658 7efd5cf41700 1 --2- 172.21.2.112:0/3961606631 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7efd58085e40 0x7efd58086240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:26.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:26.658 7efd5cf41700 1 -- 172.21.2.112:0/3961606631 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd580b1390 con 0x7efd580855b0 2021-02-10T11:40:26.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:26.658 7efd5cf41700 1 --2- 172.21.2.112:0/3961606631 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efd580855b0 0x7efd5819bc20 secure :-1 s=READY pgs=2053 cs=0 l=1 rev1=1 rx=0x7efd40006f60 tx=0x7efd4000b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:26.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:26.658 7efd557fa700 1 -- 172.21.2.112:0/3961606631 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efd4000bd60 con 0x7efd580855b0 2021-02-10T11:40:26.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:26.658 7efd557fa700 1 -- 172.21.2.112:0/3961606631 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd40000c90 con 0x7efd580855b0 2021-02-10T11:40:26.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:26.658 7efd557fa700 1 -- 172.21.2.112:0/3961606631 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efd400022b0 con 0x7efd580855b0 2021-02-10T11:40:26.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:26.658 7efd5f125700 1 -- 172.21.2.112:0/3961606631 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd580b1390 con 0x7efd580855b0 2021-02-10T11:40:26.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:26.658 7efd5f125700 1 -- 172.21.2.112:0/3961606631 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd580b1920 con 0x7efd580855b0 2021-02-10T11:40:26.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:26.659 7efd557fa700 1 -- 172.21.2.112:0/3961606631 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7efd40006b00 con 0x7efd580855b0 2021-02-10T11:40:26.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:26.659 7efd557fa700 1 --2- 172.21.2.112:0/3961606631 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7efd3c0186c0 0x7efd3c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:26.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:26.659 7efd557fa700 1 -- 172.21.2.112:0/3961606631 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5659+0+0 (secure 0 0 0) 0x7efd400325c0 con 0x7efd580855b0 2021-02-10T11:40:26.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:26.659 7efd57fff700 1 --2- 172.21.2.112:0/3961606631 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7efd3c0186c0 0x7efd3c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:26.657 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:26.661 7efd57fff700 1 --2- 172.21.2.112:0/3961606631 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7efd3c0186c0 0x7efd3c01abc0 crc :-1 s=READY pgs=952 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:26.685 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:26.690 7efd5f125700 1 -- 172.21.2.112:0/3961606631 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd58082a50 con 0x7efd580855b0 2021-02-10T11:40:26.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:26.692 7efd557fa700 1 -- 172.21.2.112:0/3961606631 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7efd40000940 con 0x7efd580855b0 2021-02-10T11:40:26.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:26.818 7efd5f125700 1 -- 172.21.2.112:0/3961606631 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7efd58109a70 con 0x7efd580855b0 2021-02-10T11:40:27.184 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:27.189 7efd557fa700 1 -- 172.21.2.112:0/3961606631 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v102) v1 ==== 185+0+0 (secure 0 0 0) 0x7efd40000940 con 0x7efd580855b0 2021-02-10T11:40:27.230 INFO:teuthology.orchestra.run.gibba012.stderr:pool 'cephfs_metadata' removed 2021-02-10T11:40:27.231 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:27.236 7efd5f125700 1 -- 172.21.2.112:0/3961606631 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7efd3c0186c0 msgr2=0x7efd3c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:27.231 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:27.236 7efd5f125700 1 --2- 172.21.2.112:0/3961606631 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7efd3c0186c0 0x7efd3c01abc0 crc :-1 s=READY pgs=952 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:27.231 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:27.236 7efd5f125700 1 -- 172.21.2.112:0/3961606631 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efd580855b0 msgr2=0x7efd5819bc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:27.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:27.236 7efd5f125700 1 --2- 172.21.2.112:0/3961606631 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efd580855b0 0x7efd5819bc20 secure :-1 s=READY pgs=2053 cs=0 l=1 rev1=1 rx=0x7efd40006f60 tx=0x7efd4000b970).stop 2021-02-10T11:40:27.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:27.237 7efd5f125700 1 -- 172.21.2.112:0/3961606631 shutdown_connections 2021-02-10T11:40:27.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:27.237 7efd5f125700 1 --2- 172.21.2.112:0/3961606631 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7efd3c0186c0 0x7efd3c01abc0 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:27.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:27.237 7efd5f125700 1 --2- 172.21.2.112:0/3961606631 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7efd58085e40 0x7efd58086240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:27.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:27.237 7efd5f125700 1 --2- 172.21.2.112:0/3961606631 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efd58086860 0x7efd580b0d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:27.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:27.237 7efd5f125700 1 --2- 172.21.2.112:0/3961606631 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efd580855b0 0x7efd5819bc20 unknown :-1 s=CLOSED pgs=2053 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:27.233 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:27.237 7efd5f125700 1 -- 172.21.2.112:0/3961606631 >> 172.21.2.112:0/3961606631 conn(0x7efd580a0f00 msgr2=0x7efd580b50f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:27.233 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:27.237 7efd5f125700 1 -- 172.21.2.112:0/3961606631 shutdown_connections 2021-02-10T11:40:27.233 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:27.237 7efd5f125700 1 -- 172.21.2.112:0/3961606631 wait complete. 2021-02-10T11:40:27.240 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2021-02-10T11:40:27.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:27.660 7f28efb74700 1 Processor -- start 2021-02-10T11:40:27.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:27.661 7f28efb74700 1 -- start start 2021-02-10T11:40:27.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:27.661 7f28efb74700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f28e80d1880 0x7f28e81b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:27.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:27.661 7f28efb74700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f28e81b0af0 0x7f28e8108950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:27.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:27.661 7f28efb74700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28e80fe1d0 con 0x7f28e80d1c80 2021-02-10T11:40:27.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:27.661 7f28efb74700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28e8109a70 con 0x7f28e80d1880 2021-02-10T11:40:27.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:27.661 7f28efb74700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28e8109760 con 0x7f28e81b0af0 2021-02-10T11:40:27.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:27.662 7f28ed990700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f28e80d1880 0x7f28e81b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:27.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:27.662 7f28ee191700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7f28e80d1c80 0x7f28e8108f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:47624/0 (socket says 172.21.2.112:47624) 2021-02-10T11:40:27.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:27.662 7f28ee191700 1 -- 172.21.2.112:0/411848308 learned_addr learned my addr 172.21.2.112:0/411848308 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:40:27.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:27.662 7f28ed18f700 1 --2- 172.21.2.112:0/411848308 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f28e81b0af0 0x7f28e8108950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:27.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:27.663 7f28ed990700 1 -- 172.21.2.112:0/411848308 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f28e81b0af0 msgr2=0x7f28e8108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:27.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:27.663 7f28ed990700 1 --2- 172.21.2.112:0/411848308 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f28e81b0af0 0x7f28e8108950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:27.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:27.663 7f28ed990700 1 -- 172.21.2.112:0/411848308 >> v1:172.21.2.120:6789/0 conn(0x7f28e80d1c80 legacy=0x7f28e8108f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:27.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:27.663 7f28ed990700 1 -- 172.21.2.112:0/411848308 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28e8088070 con 0x7f28e80d1880 2021-02-10T11:40:27.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:27.663 7f28ed990700 1 --2- 172.21.2.112:0/411848308 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f28e80d1880 0x7f28e81b04d0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 rx=0x7f28e4006650 tx=0x7f28e4009a80).ready entity=mon.1 client_cookie=5392705a97d425fb server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:27.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:27.663 7f28ec98e700 1 -- 172.21.2.112:0/411848308 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f28e400c0c0 con 0x7f28e80d1880 2021-02-10T11:40:27.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:27.663 7f28ec98e700 1 -- 172.21.2.112:0/411848308 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28e400c340 con 0x7f28e80d1880 2021-02-10T11:40:27.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:27.663 7f28ec98e700 1 -- 172.21.2.112:0/411848308 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f28e400c860 con 0x7f28e80d1880 2021-02-10T11:40:27.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:27.664 7f28efb74700 1 -- 172.21.2.112:0/411848308 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f28e80d1880 msgr2=0x7f28e81b04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:27.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:27.664 7f28efb74700 1 --2- 172.21.2.112:0/411848308 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f28e80d1880 0x7f28e81b04d0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 rx=0x7f28e4006650 tx=0x7f28e4009a80).stop 2021-02-10T11:40:27.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:27.664 7f28efb74700 1 -- 172.21.2.112:0/411848308 shutdown_connections 2021-02-10T11:40:27.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:27.664 7f28efb74700 1 --2- 172.21.2.112:0/411848308 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f28e81b0af0 0x7f28e8108950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:27.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:27.664 7f28efb74700 1 --2- 172.21.2.112:0/411848308 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f28e80d1880 0x7f28e81b04d0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:27.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:27.664 7f28efb74700 1 -- 172.21.2.112:0/411848308 >> 172.21.2.112:0/411848308 conn(0x7f28e80ce430 msgr2=0x7f28e80d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:27.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:27.664 7f28efb74700 1 -- 172.21.2.112:0/411848308 shutdown_connections 2021-02-10T11:40:27.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:27.664 7f28efb74700 1 -- 172.21.2.112:0/411848308 wait complete. 2021-02-10T11:40:27.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:27.679 7f28efb74700 1 Processor -- start 2021-02-10T11:40:27.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:27.679 7f28efb74700 1 -- start start 2021-02-10T11:40:27.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:27.679 7f28efb74700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f28e80ad520 0x7f28e80ad920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:27.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:27.679 7f28efb74700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f28e80adf40 0x7f28e8097c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:27.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:27.679 7f28efb74700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f28e8098250 0x7f28e8098650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:27.678 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:27.679 7f28efb74700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28e8098c70 con 0x7f28e80ad520 2021-02-10T11:40:27.678 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:27.679 7f28efb74700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28e80910f0 con 0x7f28e8098250 2021-02-10T11:40:27.678 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:27.679 7f28efb74700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28e8091310 con 0x7f28e80adf40 2021-02-10T11:40:27.678 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:27.679 7f28ed990700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f28e80ad520 0x7f28e80ad920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:27.678 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:27.679 7f28ed990700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f28e80ad520 0x7f28e80ad920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:54610/0 (socket says 172.21.2.112:54610) 2021-02-10T11:40:27.678 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:27.679 7f28ed990700 1 -- 172.21.2.112:0/2041416508 learned_addr learned my addr 172.21.2.112:0/2041416508 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:40:27.678 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:27.680 7f28ed990700 1 -- 172.21.2.112:0/2041416508 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f28e80adf40 msgr2=0x7f28e8097c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:27.678 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:27.680 7f28ed990700 1 --2- 172.21.2.112:0/2041416508 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f28e80adf40 0x7f28e8097c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:27.678 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:27.680 7f28ed990700 1 -- 172.21.2.112:0/2041416508 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f28e8098250 msgr2=0x7f28e8098650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:27.678 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:27.680 7f28ed990700 1 --2- 172.21.2.112:0/2041416508 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f28e8098250 0x7f28e8098650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:27.678 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:27.680 7f28ed990700 1 -- 172.21.2.112:0/2041416508 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28e819a3b0 con 0x7f28e80ad520 2021-02-10T11:40:27.678 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:27.680 7f28ed990700 1 --2- 172.21.2.112:0/2041416508 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f28e80ad520 0x7f28e80ad920 secure :-1 s=READY pgs=2054 cs=0 l=1 rev1=1 rx=0x7f28e4009630 tx=0x7f28e400b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:27.679 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:27.680 7f28de7fc700 1 -- 172.21.2.112:0/2041416508 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f28e40093a0 con 0x7f28e80ad520 2021-02-10T11:40:27.679 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:27.680 7f28de7fc700 1 -- 172.21.2.112:0/2041416508 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28e400bd70 con 0x7f28e80ad520 2021-02-10T11:40:27.679 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:27.680 7f28de7fc700 1 -- 172.21.2.112:0/2041416508 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f28e4005560 con 0x7f28e80ad520 2021-02-10T11:40:27.679 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:27.680 7f28efb74700 1 -- 172.21.2.112:0/2041416508 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28e819a3b0 con 0x7f28e80ad520 2021-02-10T11:40:27.679 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:27.680 7f28efb74700 1 -- 172.21.2.112:0/2041416508 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28e819a3b0 con 0x7f28e80ad520 2021-02-10T11:40:27.679 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:27.681 7f28de7fc700 1 -- 172.21.2.112:0/2041416508 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f28e4006020 con 0x7f28e80ad520 2021-02-10T11:40:27.679 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:27.681 7f28de7fc700 1 --2- 172.21.2.112:0/2041416508 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f28d0018730 0x7f28d001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:27.679 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:27.681 7f28de7fc700 1 -- 172.21.2.112:0/2041416508 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5221+0+0 (secure 0 0 0) 0x7f28e4032640 con 0x7f28e80ad520 2021-02-10T11:40:27.679 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:27.681 7f28ed18f700 1 --2- 172.21.2.112:0/2041416508 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f28d0018730 0x7f28d001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:27.679 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:27.681 7f28ed18f700 1 --2- 172.21.2.112:0/2041416508 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f28d0018730 0x7f28d001ac10 crc :-1 s=READY pgs=953 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:27.707 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:27.711 7f28efb74700 1 -- 172.21.2.112:0/2041416508 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28e8082a70 con 0x7f28e80ad520 2021-02-10T11:40:27.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:27.714 7f28de7fc700 1 -- 172.21.2.112:0/2041416508 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f28e4008030 con 0x7f28e80ad520 2021-02-10T11:40:27.836 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:27.841 7f28efb74700 1 -- 172.21.2.112:0/2041416508 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f28e8082a70 con 0x7f28e80ad520 2021-02-10T11:40:28.189 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:28.194 7f28de7fc700 1 -- 172.21.2.112:0/2041416508 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v103) v1 ==== 173+0+0 (secure 0 0 0) 0x7f28e4006020 con 0x7f28e80ad520 2021-02-10T11:40:28.202 INFO:teuthology.orchestra.run.gibba012.stderr:pool 'cephfs_data' removed 2021-02-10T11:40:28.204 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:28.208 7f28efb74700 1 -- 172.21.2.112:0/2041416508 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f28d0018730 msgr2=0x7f28d001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:28.204 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:28.208 7f28efb74700 1 --2- 172.21.2.112:0/2041416508 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f28d0018730 0x7f28d001ac10 crc :-1 s=READY pgs=953 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:28.204 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:28.208 7f28efb74700 1 -- 172.21.2.112:0/2041416508 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f28e80ad520 msgr2=0x7f28e80ad920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:28.204 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:28.208 7f28efb74700 1 --2- 172.21.2.112:0/2041416508 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f28e80ad520 0x7f28e80ad920 secure :-1 s=READY pgs=2054 cs=0 l=1 rev1=1 rx=0x7f28e4009630 tx=0x7f28e400b980).stop 2021-02-10T11:40:28.204 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:28.208 7f28efb74700 1 -- 172.21.2.112:0/2041416508 shutdown_connections 2021-02-10T11:40:28.204 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:28.208 7f28efb74700 1 --2- 172.21.2.112:0/2041416508 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f28d0018730 0x7f28d001ac10 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:28.204 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:28.208 7f28efb74700 1 --2- 172.21.2.112:0/2041416508 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f28e8098250 0x7f28e8098650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:28.204 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:28.208 7f28efb74700 1 --2- 172.21.2.112:0/2041416508 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f28e80adf40 0x7f28e8097c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:28.204 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:28.208 7f28efb74700 1 --2- 172.21.2.112:0/2041416508 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f28e80ad520 0x7f28e80ad920 unknown :-1 s=CLOSED pgs=2054 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:28.204 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:28.208 7f28efb74700 1 -- 172.21.2.112:0/2041416508 >> 172.21.2.112:0/2041416508 conn(0x7f28e818e810 msgr2=0x7f28e818c290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:28.204 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:28.208 7f28efb74700 1 -- 172.21.2.112:0/2041416508 shutdown_connections 2021-02-10T11:40:28.205 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:28.208 7f28efb74700 1 -- 172.21.2.112:0/2041416508 wait complete. 2021-02-10T11:40:28.212 INFO:tasks.ceph.mds.a:Restarting daemon 2021-02-10T11:40:28.213 INFO:tasks.ceph.mds.a:Stopping old one... 2021-02-10T11:40:28.213 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2021-02-10T11:40:28.213 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:40:28.213 INFO:tasks.ceph.mds.b:Restarting daemon 2021-02-10T11:40:28.213 INFO:tasks.ceph.mds.b:Stopping old one... 2021-02-10T11:40:28.213 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2021-02-10T11:40:28.214 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:40:28.219 INFO:tasks.ceph.mds.a:Stopped 2021-02-10T11:40:28.219 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2021-02-10T11:40:28.220 INFO:tasks.ceph.mds.b:Stopped 2021-02-10T11:40:28.220 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2021-02-10T11:40:28.261 INFO:tasks.ceph.mds.a:Started 2021-02-10T11:40:28.261 DEBUG:teuthology.parallel:result is None 2021-02-10T11:40:28.261 INFO:tasks.ceph.mds.b:Started 2021-02-10T11:40:28.262 DEBUG:teuthology.parallel:result is None 2021-02-10T11:40:28.262 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd blacklist clear 2021-02-10T11:40:28.666 INFO:tasks.ceph.mds.b.gibba020.stdout:starting mds.b at 2021-02-10T11:40:28.668 INFO:tasks.ceph.mds.a.gibba012.stdout:starting mds.a at 2021-02-10T11:40:28.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:28.680 7f748c3ca700 1 Processor -- start 2021-02-10T11:40:28.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:28.680 7f748c3ca700 1 -- start start 2021-02-10T11:40:28.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:28.680 7f748c3ca700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f74840d1870 0x7f74841b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:28.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:28.680 7f748c3ca700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f74841b0a10 0x7f74840cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:28.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:28.680 7f748c3ca700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f74840d1c70 0x7f74840cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:28.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:28.680 7f748c3ca700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74840fe230 con 0x7f74841b0a10 2021-02-10T11:40:28.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:28.680 7f748c3ca700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74840e5ed0 con 0x7f74840d1c70 2021-02-10T11:40:28.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:28.680 7f748c3ca700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74840f0b10 con 0x7f74840d1870 2021-02-10T11:40:28.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:28.680 7f74899e5700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f74841b0a10 0x7f74840cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:28.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:28.680 7f74899e5700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f74841b0a10 0x7f74840cf330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:54626/0 (socket says 172.21.2.112:54626) 2021-02-10T11:40:28.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:28.680 7f74899e5700 1 -- 172.21.2.112:0/970365608 learned_addr learned my addr 172.21.2.112:0/970365608 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:40:28.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:28.681 7f74899e5700 1 -- 172.21.2.112:0/970365608 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f74840d1870 msgr2=0x7f74841b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:28.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:28.681 7f74899e5700 1 --2- 172.21.2.112:0/970365608 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f74840d1870 0x7f74841b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:28.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:28.681 7f74899e5700 1 -- 172.21.2.112:0/970365608 >> v2:172.21.2.120:3300/0 conn(0x7f74840d1c70 msgr2=0x7f74840cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:28.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:28.681 7f74899e5700 1 --2- 172.21.2.112:0/970365608 >> v2:172.21.2.120:3300/0 conn(0x7f74840d1c70 0x7f74840cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:28.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:28.681 7f74899e5700 1 -- 172.21.2.112:0/970365608 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74840cff70 con 0x7f74841b0a10 2021-02-10T11:40:28.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:28.681 7f74899e5700 1 --2- 172.21.2.112:0/970365608 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f74841b0a10 0x7f74840cf330 secure :-1 s=READY pgs=2060 cs=0 l=1 rev1=1 rx=0x7f7480006700 tx=0x7f7480009b80).ready entity=mon.0 client_cookie=9f778bf0452cf423 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:28.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:28.681 7f74891e4700 1 -- 172.21.2.112:0/970365608 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f748000c0c0 con 0x7f74841b0a10 2021-02-10T11:40:28.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:28.681 7f74891e4700 1 -- 172.21.2.112:0/970365608 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f748000c440 con 0x7f74841b0a10 2021-02-10T11:40:28.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:28.681 7f74891e4700 1 -- 172.21.2.112:0/970365608 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f748000c920 con 0x7f74841b0a10 2021-02-10T11:40:28.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:28.681 7f748c3ca700 1 -- 172.21.2.112:0/970365608 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f74841b0a10 msgr2=0x7f74840cf330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:28.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:28.681 7f748c3ca700 1 --2- 172.21.2.112:0/970365608 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f74841b0a10 0x7f74840cf330 secure :-1 s=READY pgs=2060 cs=0 l=1 rev1=1 rx=0x7f7480006700 tx=0x7f7480009b80).stop 2021-02-10T11:40:28.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:28.682 7f748c3ca700 1 -- 172.21.2.112:0/970365608 shutdown_connections 2021-02-10T11:40:28.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:28.682 7f748c3ca700 1 --2- 172.21.2.112:0/970365608 >> v2:172.21.2.120:3300/0 conn(0x7f74840d1c70 0x7f74840cf950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:28.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:28.682 7f748c3ca700 1 --2- 172.21.2.112:0/970365608 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f74841b0a10 0x7f74840cf330 unknown :-1 s=CLOSED pgs=2060 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:28.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:28.682 7f748c3ca700 1 --2- 172.21.2.112:0/970365608 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f74840d1870 0x7f74841b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:28.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:28.682 7f748c3ca700 1 -- 172.21.2.112:0/970365608 >> 172.21.2.112:0/970365608 conn(0x7f74840ce450 msgr2=0x7f74840d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:28.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:28.682 7f748c3ca700 1 -- 172.21.2.112:0/970365608 shutdown_connections 2021-02-10T11:40:28.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:28.682 7f748c3ca700 1 -- 172.21.2.112:0/970365608 wait complete. 2021-02-10T11:40:28.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:28.696 7f748c3ca700 1 Processor -- start 2021-02-10T11:40:28.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:28.696 7f748c3ca700 1 -- start start 2021-02-10T11:40:28.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:28.697 7f748c3ca700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f74840a2fb0 0x7f74840a33b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:28.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:28.697 7f748c3ca700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f74840a39d0 0x7f74840a4860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:28.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:28.697 7f748c3ca700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f74840a3dd0 0x7f74840a4e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:28.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:28.697 7f748c3ca700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74840a2620 con 0x7f74840a39d0 2021-02-10T11:40:28.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:28.697 7f748c3ca700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74840a2840 con 0x7f74840a3dd0 2021-02-10T11:40:28.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:28.697 7f748c3ca700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74840a2b80 con 0x7f74840a2fb0 2021-02-10T11:40:28.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:28.697 7f74899e5700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f74840a39d0 0x7f74840a4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:28.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:28.697 7f74899e5700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f74840a39d0 0x7f74840a4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:54634/0 (socket says 172.21.2.112:54634) 2021-02-10T11:40:28.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:28.697 7f74899e5700 1 -- 172.21.2.112:0/3415748971 learned_addr learned my addr 172.21.2.112:0/3415748971 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:40:28.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:28.697 7f748a1e6700 1 --2- 172.21.2.112:0/3415748971 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f74840a2fb0 0x7f74840a33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:28.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:28.697 7f748a9e7700 1 --2- 172.21.2.112:0/3415748971 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f74840a3dd0 0x7f74840a4e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:28.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:28.697 7f74899e5700 1 -- 172.21.2.112:0/3415748971 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f74840a2fb0 msgr2=0x7f74840a33b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:28.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:28.697 7f74899e5700 1 --2- 172.21.2.112:0/3415748971 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f74840a2fb0 0x7f74840a33b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:28.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:28.697 7f74899e5700 1 -- 172.21.2.112:0/3415748971 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f74840a3dd0 msgr2=0x7f74840a4e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:28.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:28.697 7f74899e5700 1 --2- 172.21.2.112:0/3415748971 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f74840a3dd0 0x7f74840a4e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:28.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:28.697 7f74899e5700 1 -- 172.21.2.112:0/3415748971 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74840a54a0 con 0x7f74840a39d0 2021-02-10T11:40:28.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:28.697 7f74899e5700 1 --2- 172.21.2.112:0/3415748971 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f74840a39d0 0x7f74840a4860 secure :-1 s=READY pgs=2061 cs=0 l=1 rev1=1 rx=0x7f748000b580 tx=0x7f748000b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:28.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:28.698 7f7476ffd700 1 -- 172.21.2.112:0/3415748971 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f748000bd10 con 0x7f74840a39d0 2021-02-10T11:40:28.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:28.698 7f7476ffd700 1 -- 172.21.2.112:0/3415748971 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7480002030 con 0x7f74840a39d0 2021-02-10T11:40:28.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:28.698 7f748c3ca700 1 -- 172.21.2.112:0/3415748971 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74840a54a0 con 0x7f74840a39d0 2021-02-10T11:40:28.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:28.698 7f7476ffd700 1 -- 172.21.2.112:0/3415748971 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7480002490 con 0x7f74840a39d0 2021-02-10T11:40:28.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:28.698 7f748c3ca700 1 -- 172.21.2.112:0/3415748971 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74840a5a30 con 0x7f74840a39d0 2021-02-10T11:40:28.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:28.698 7f7476ffd700 1 -- 172.21.2.112:0/3415748971 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7480000c90 con 0x7f74840a39d0 2021-02-10T11:40:28.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:28.698 7f7476ffd700 1 --2- 172.21.2.112:0/3415748971 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f746c018730 0x7f746c01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:28.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:28.698 7f7476ffd700 1 -- 172.21.2.112:0/3415748971 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 4839+0+0 (secure 0 0 0) 0x7f7480032540 con 0x7f74840a39d0 2021-02-10T11:40:28.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:28.698 7f748a1e6700 1 --2- 172.21.2.112:0/3415748971 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f746c018730 0x7f746c01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:28.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:28.699 7f748a1e6700 1 --2- 172.21.2.112:0/3415748971 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f746c018730 0x7f746c01ac10 crc :-1 s=READY pgs=954 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:28.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:28.729 7f748c3ca700 1 -- 172.21.2.112:0/3415748971 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74840a4330 con 0x7f74840a39d0 2021-02-10T11:40:28.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:28.732 7f7476ffd700 1 -- 172.21.2.112:0/3415748971 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f748001f560 con 0x7f74840a39d0 2021-02-10T11:40:28.867 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:28.871 7f748c3ca700 1 -- 172.21.2.112:0/3415748971 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7f74840a4330 con 0x7f74840a39d0 2021-02-10T11:40:29.196 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:29.201 7f7476ffd700 1 -- 172.21.2.112:0/3415748971 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v104) v1 ==== 97+0+0 (secure 0 0 0) 0x7f748001f560 con 0x7f74840a39d0 2021-02-10T11:40:29.232 INFO:teuthology.orchestra.run.gibba012.stderr: removed all blacklist entries 2021-02-10T11:40:29.234 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:29.239 7f748c3ca700 1 -- 172.21.2.112:0/3415748971 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f746c018730 msgr2=0x7f746c01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:29.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:29.239 7f748c3ca700 1 --2- 172.21.2.112:0/3415748971 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f746c018730 0x7f746c01ac10 crc :-1 s=READY pgs=954 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:29.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:29.239 7f748c3ca700 1 -- 172.21.2.112:0/3415748971 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f74840a39d0 msgr2=0x7f74840a4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:29.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:29.239 7f748c3ca700 1 --2- 172.21.2.112:0/3415748971 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f74840a39d0 0x7f74840a4860 secure :-1 s=READY pgs=2061 cs=0 l=1 rev1=1 rx=0x7f748000b580 tx=0x7f748000b920).stop 2021-02-10T11:40:29.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:29.239 7f748c3ca700 1 -- 172.21.2.112:0/3415748971 shutdown_connections 2021-02-10T11:40:29.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:29.239 7f748c3ca700 1 --2- 172.21.2.112:0/3415748971 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f746c018730 0x7f746c01ac10 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:29.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:29.239 7f748c3ca700 1 --2- 172.21.2.112:0/3415748971 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f74840a3dd0 0x7f74840a4e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:29.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:29.239 7f748c3ca700 1 --2- 172.21.2.112:0/3415748971 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f74840a39d0 0x7f74840a4860 unknown :-1 s=CLOSED pgs=2061 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:29.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:29.239 7f748c3ca700 1 --2- 172.21.2.112:0/3415748971 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f74840a2fb0 0x7f74840a33b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:29.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:29.239 7f748c3ca700 1 -- 172.21.2.112:0/3415748971 >> 172.21.2.112:0/3415748971 conn(0x7f748408f030 msgr2=0x7f74840b5e80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:29.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:29.239 7f748c3ca700 1 -- 172.21.2.112:0/3415748971 shutdown_connections 2021-02-10T11:40:29.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:29.239 7f748c3ca700 1 -- 172.21.2.112:0/3415748971 wait complete. 2021-02-10T11:40:29.243 INFO:tasks.cephfs.cephfs_test_case:['0', '1', '2', '3'] 2021-02-10T11:40:29.243 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth ls --format=json-pretty 2021-02-10T11:40:29.670 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:29.656 7f00286a7700 1 Processor -- start 2021-02-10T11:40:29.670 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:29.656 7f00286a7700 1 -- start start 2021-02-10T11:40:29.670 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:29.657 7f00286a7700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f00201b0470 0x7f00201b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:29.670 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:29.657 7f00286a7700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f00200f20e0 0x7f00200f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:29.670 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:29.657 7f00286a7700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00200f0b50 con 0x7f00200ef610 2021-02-10T11:40:29.670 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:29.657 7f00286a7700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00200f0820 con 0x7f00200f20e0 2021-02-10T11:40:29.670 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:29.657 7f00286a7700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00200f24e0 con 0x7f00201b0470 2021-02-10T11:40:29.670 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:29.657 7f0026cc4700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f00200f20e0 0x7f00200f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:29.670 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:29.657 7f0026cc4700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f00200f20e0 0x7f00200f0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:54640/0 (socket says 172.21.2.112:54640) 2021-02-10T11:40:29.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:29.657 7f0026cc4700 1 -- 172.21.2.112:0/2128105077 learned_addr learned my addr 172.21.2.112:0/2128105077 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:40:29.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:29.658 7f0026cc4700 1 -- 172.21.2.112:0/2128105077 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f00201b0470 msgr2=0x7f00201b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:29.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:29.658 7f0026cc4700 1 --2- 172.21.2.112:0/2128105077 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f00201b0470 0x7f00201b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:29.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:29.658 7f0026cc4700 1 -- 172.21.2.112:0/2128105077 >> v1:172.21.2.120:6789/0 conn(0x7f00200ef610 legacy=0x7f00200efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:29.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:29.658 7f0026cc4700 1 -- 172.21.2.112:0/2128105077 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00200e52e0 con 0x7f00200f20e0 2021-02-10T11:40:29.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:29.658 7f0026cc4700 1 --2- 172.21.2.112:0/2128105077 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f00200f20e0 0x7f00200f0200 secure :-1 s=READY pgs=2062 cs=0 l=1 rev1=1 rx=0x7f0018006b80 tx=0x7f0018009b90).ready entity=mon.0 client_cookie=b280189a25a0aecd server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:29.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:29.658 7f00254c1700 1 -- 172.21.2.112:0/2128105077 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f00180022e0 con 0x7f00200f20e0 2021-02-10T11:40:29.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:29.658 7f00254c1700 1 -- 172.21.2.112:0/2128105077 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00180009b0 con 0x7f00200f20e0 2021-02-10T11:40:29.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:29.658 7f00254c1700 1 -- 172.21.2.112:0/2128105077 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f00180028e0 con 0x7f00200f20e0 2021-02-10T11:40:29.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:29.658 7f00286a7700 1 -- 172.21.2.112:0/2128105077 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f00200f20e0 msgr2=0x7f00200f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:29.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:29.658 7f00286a7700 1 --2- 172.21.2.112:0/2128105077 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f00200f20e0 0x7f00200f0200 secure :-1 s=READY pgs=2062 cs=0 l=1 rev1=1 rx=0x7f0018006b80 tx=0x7f0018009b90).stop 2021-02-10T11:40:29.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:29.659 7f00286a7700 1 -- 172.21.2.112:0/2128105077 shutdown_connections 2021-02-10T11:40:29.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:29.659 7f00286a7700 1 --2- 172.21.2.112:0/2128105077 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f00200f20e0 0x7f00200f0200 unknown :-1 s=CLOSED pgs=2062 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:29.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:29.659 7f00286a7700 1 --2- 172.21.2.112:0/2128105077 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f00201b0470 0x7f00201b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:29.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:29.659 7f00286a7700 1 -- 172.21.2.112:0/2128105077 >> 172.21.2.112:0/2128105077 conn(0x7f00200ce3e0 msgr2=0x7f00200e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:29.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:29.659 7f00286a7700 1 -- 172.21.2.112:0/2128105077 shutdown_connections 2021-02-10T11:40:29.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:29.659 7f00286a7700 1 -- 172.21.2.112:0/2128105077 wait complete. 2021-02-10T11:40:29.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:29.674 7f00286a7700 1 Processor -- start 2021-02-10T11:40:29.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:29.674 7f00286a7700 1 -- start start 2021-02-10T11:40:29.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:29.674 7f00286a7700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f00200b0970 0x7f0020092ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:29.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:29.674 7f00286a7700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f00200934f0 0x7f0020094dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:29.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:29.674 7f00286a7700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f00200938f0 0x7f00200953e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:29.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:29.674 7f00286a7700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0020195ad0 con 0x7f00200b0970 2021-02-10T11:40:29.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:29.674 7f00286a7700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0020095f40 con 0x7f00200934f0 2021-02-10T11:40:29.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:29.674 7f00286a7700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0020095a30 con 0x7f00200938f0 2021-02-10T11:40:29.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:29.674 7f00264c3700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f00200b0970 0x7f0020092ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:29.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:29.674 7f00264c3700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f00200b0970 0x7f0020092ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:54646/0 (socket says 172.21.2.112:54646) 2021-02-10T11:40:29.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:29.674 7f00264c3700 1 -- 172.21.2.112:0/2275929910 learned_addr learned my addr 172.21.2.112:0/2275929910 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:40:29.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:29.675 7f00264c3700 1 -- 172.21.2.112:0/2275929910 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f00200938f0 msgr2=0x7f00200953e0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:40:29.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:29.675 7f00264c3700 1 --2- 172.21.2.112:0/2275929910 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f00200938f0 0x7f00200953e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:29.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:29.675 7f00264c3700 1 -- 172.21.2.112:0/2275929910 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f00200934f0 msgr2=0x7f0020094dc0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:40:29.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:29.675 7f00264c3700 1 --2- 172.21.2.112:0/2275929910 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f00200934f0 0x7f0020094dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:29.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:29.675 7f00264c3700 1 -- 172.21.2.112:0/2275929910 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0020095c50 con 0x7f00200b0970 2021-02-10T11:40:29.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:29.675 7f00264c3700 1 --2- 172.21.2.112:0/2275929910 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f00200b0970 0x7f0020092ed0 secure :-1 s=READY pgs=2063 cs=0 l=1 rev1=1 rx=0x7f001000b530 tx=0x7f001000b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:29.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:29.675 7f001effd700 1 -- 172.21.2.112:0/2275929910 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f001000bcc0 con 0x7f00200b0970 2021-02-10T11:40:29.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:29.675 7f001effd700 1 -- 172.21.2.112:0/2275929910 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0010000c90 con 0x7f00200b0970 2021-02-10T11:40:29.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:29.675 7f00286a7700 1 -- 172.21.2.112:0/2275929910 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0020095c50 con 0x7f00200b0970 2021-02-10T11:40:29.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:29.675 7f00286a7700 1 -- 172.21.2.112:0/2275929910 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00200967b0 con 0x7f00200b0970 2021-02-10T11:40:29.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:29.675 7f001effd700 1 -- 172.21.2.112:0/2275929910 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f00100022a0 con 0x7f00200b0970 2021-02-10T11:40:29.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:29.676 7f001effd700 1 -- 172.21.2.112:0/2275929910 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0010005310 con 0x7f00200b0970 2021-02-10T11:40:29.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:29.676 7f001effd700 1 --2- 172.21.2.112:0/2275929910 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0000018730 0x7f000001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:29.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:29.676 7f001effd700 1 -- 172.21.2.112:0/2275929910 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f0010032440 con 0x7f00200b0970 2021-02-10T11:40:29.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:29.676 7f0025cc2700 1 --2- 172.21.2.112:0/2275929910 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0000018730 0x7f000001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:29.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:29.677 7f0025cc2700 1 --2- 172.21.2.112:0/2275929910 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0000018730 0x7f000001ac10 crc :-1 s=READY pgs=955 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:29.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:29.707 7f00286a7700 1 -- 172.21.2.112:0/2275929910 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00200e52e0 con 0x7f00200b0970 2021-02-10T11:40:29.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:29.709 7f001effd700 1 -- 172.21.2.112:0/2275929910 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0010006b90 con 0x7f00200b0970 2021-02-10T11:40:29.837 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:29.840 7f00286a7700 1 -- 172.21.2.112:0/2275929910 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f00200e52e0 con 0x7f00200b0970 2021-02-10T11:40:29.837 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:29.841 7f001effd700 1 -- 172.21.2.112:0/2275929910 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v64) v1 ==== 80+0+4513 (secure 0 0 0) 0x7f0010000c90 con 0x7f00200b0970 2021-02-10T11:40:29.838 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:40:29.838 INFO:teuthology.orchestra.run.gibba012.stdout:{ 2021-02-10T11:40:29.838 INFO:teuthology.orchestra.run.gibba012.stdout: "auth_dump": [ 2021-02-10T11:40:29.838 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:40:29.838 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "mds.a", 2021-02-10T11:40:29.838 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCKvyNgPkV4AxAAEmVNaghdccP6lWd8hvQnxw==", 2021-02-10T11:40:29.838 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:40:29.838 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:40:29.838 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:40:29.839 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:40:29.839 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:40:29.839 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:40:29.839 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:40:29.839 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:40:29.839 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "mds.b", 2021-02-10T11:40:29.839 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCKvyNguxONLhAAHIpry+R89MAwdhFJyEhVCw==", 2021-02-10T11:40:29.839 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:40:29.839 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:40:29.839 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:40:29.839 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:40:29.840 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:40:29.840 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:40:29.840 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:40:29.840 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:40:29.840 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.0", 2021-02-10T11:40:29.840 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCSvyNgXcKPLhAAP/Bj2F+06DCtPCzQDNq+cQ==", 2021-02-10T11:40:29.840 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:40:29.840 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:40:29.840 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:40:29.840 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:40:29.840 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:40:29.840 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:40:29.840 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:40:29.841 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.1", 2021-02-10T11:40:29.841 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCVvyNghfbcLhAA66PrRwR2M7o6VTF5UhU/Fg==", 2021-02-10T11:40:29.841 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:40:29.841 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:40:29.841 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:40:29.841 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:40:29.841 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:40:29.841 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:40:29.841 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:40:29.841 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.2", 2021-02-10T11:40:29.841 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCYvyNgwiPqLhAA66BVkRsgaUlGpAsQ15Nf8Q==", 2021-02-10T11:40:29.841 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:40:29.842 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:40:29.842 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:40:29.842 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:40:29.842 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:40:29.842 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:40:29.842 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:40:29.842 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.3", 2021-02-10T11:40:29.842 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCbvyNgqBZrLxAAERPe/1L3SXAEzI3d9ReHHg==", 2021-02-10T11:40:29.842 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:40:29.842 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:40:29.842 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:40:29.843 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:40:29.843 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:40:29.843 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:40:29.843 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:40:29.843 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.4", 2021-02-10T11:40:29.843 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCkvyNgDQsxIhAAHmVjVYnx20N4SDmKqvbcuA==", 2021-02-10T11:40:29.843 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:40:29.843 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:40:29.843 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:40:29.843 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:40:29.843 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:40:29.843 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:40:29.844 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:40:29.844 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.5", 2021-02-10T11:40:29.844 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCnvyNg8ITFIhAADd4aKybAV7btoGPx3kCg3A==", 2021-02-10T11:40:29.844 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:40:29.844 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:40:29.844 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:40:29.844 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:40:29.844 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:40:29.844 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:40:29.844 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:40:29.844 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.6", 2021-02-10T11:40:29.844 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCqvyNgxOhDIxAAzXixEa/nlDc1xXrg0aeLwg==", 2021-02-10T11:40:29.844 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:40:29.845 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:40:29.845 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:40:29.845 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:40:29.845 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:40:29.845 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:40:29.845 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:40:29.845 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.7", 2021-02-10T11:40:29.845 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCtvyNgxSqGIxAACxJQlwP7DpiVVCZHA3zn1A==", 2021-02-10T11:40:29.845 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:40:29.845 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:40:29.845 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:40:29.845 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:40:29.846 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:40:29.846 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:40:29.846 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:40:29.846 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.0", 2021-02-10T11:40:29.846 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCLvyNgDkNiHRAAygU8Za4PBusTqCgwAYWTiQ==", 2021-02-10T11:40:29.846 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:40:29.846 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:40:29.846 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow r", 2021-02-10T11:40:29.846 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:40:29.846 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:40:29.846 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:40:29.847 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:40:29.847 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.1", 2021-02-10T11:40:29.847 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCLvyNgxmLqMxAA1MlSDJ9LH9u4f2sL7mm8/Q==", 2021-02-10T11:40:29.847 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:40:29.847 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:40:29.847 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow r", 2021-02-10T11:40:29.847 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:40:29.847 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:40:29.847 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:40:29.847 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:40:29.847 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.2", 2021-02-10T11:40:29.847 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCMvyNgp8veDhAAniSVHYPtwwTog/pjbaRQmw==", 2021-02-10T11:40:29.848 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:40:29.848 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:40:29.848 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow r", 2021-02-10T11:40:29.848 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:40:29.848 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:40:29.848 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:40:29.848 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:40:29.848 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.3", 2021-02-10T11:40:29.848 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCMvyNgQEVoJRAApHcWdm1sldIxu+heyD4HdQ==", 2021-02-10T11:40:29.848 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:40:29.848 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:40:29.848 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow r", 2021-02-10T11:40:29.848 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:40:29.849 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:40:29.849 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:40:29.849 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:40:29.849 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.admin", 2021-02-10T11:40:29.849 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCGvyNg7jjJBRAAtot+ypUZOiTo1GbY+oVhVw==", 2021-02-10T11:40:29.849 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:40:29.849 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow *", 2021-02-10T11:40:29.849 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:40:29.849 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:40:29.849 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:40:29.849 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:40:29.849 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:40:29.849 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:40:29.850 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "mgr.x", 2021-02-10T11:40:29.850 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCJvyNgTPVwKBAAjduiBE8+RSXBv82tdPecWg==", 2021-02-10T11:40:29.850 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:40:29.850 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow *", 2021-02-10T11:40:29.850 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow profile mgr", 2021-02-10T11:40:29.850 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:40:29.850 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:40:29.850 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:40:29.850 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:40:29.850 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "mgr.y", 2021-02-10T11:40:29.850 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCJvyNgQ1tTDhAASwBssDy+Rz31ujw8fNlWow==", 2021-02-10T11:40:29.850 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:40:29.851 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow *", 2021-02-10T11:40:29.851 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow profile mgr", 2021-02-10T11:40:29.851 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:40:29.851 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:40:29.851 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:40:29.851 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:40:29.851 INFO:teuthology.orchestra.run.gibba012.stdout:} 2021-02-10T11:40:29.852 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:29.843 7f00286a7700 1 -- 172.21.2.112:0/2275929910 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0000018730 msgr2=0x7f000001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:29.852 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:29.843 7f00286a7700 1 --2- 172.21.2.112:0/2275929910 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0000018730 0x7f000001ac10 crc :-1 s=READY pgs=955 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:29.852 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:29.843 7f00286a7700 1 -- 172.21.2.112:0/2275929910 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f00200b0970 msgr2=0x7f0020092ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:29.852 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:29.843 7f00286a7700 1 --2- 172.21.2.112:0/2275929910 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f00200b0970 0x7f0020092ed0 secure :-1 s=READY pgs=2063 cs=0 l=1 rev1=1 rx=0x7f001000b530 tx=0x7f001000b8d0).stop 2021-02-10T11:40:29.852 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:29.843 7f00286a7700 1 -- 172.21.2.112:0/2275929910 shutdown_connections 2021-02-10T11:40:29.853 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:29.843 7f00286a7700 1 --2- 172.21.2.112:0/2275929910 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0000018730 0x7f000001ac10 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:29.853 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:29.843 7f00286a7700 1 --2- 172.21.2.112:0/2275929910 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f00200934f0 0x7f0020094dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:29.853 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:29.843 7f00286a7700 1 --2- 172.21.2.112:0/2275929910 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f00200938f0 0x7f00200953e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:29.853 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:29.843 7f00286a7700 1 --2- 172.21.2.112:0/2275929910 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f00200b0970 0x7f0020092ed0 unknown :-1 s=CLOSED pgs=2063 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:29.853 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:29.843 7f00286a7700 1 -- 172.21.2.112:0/2275929910 >> 172.21.2.112:0/2275929910 conn(0x7f002019f7a0 msgr2=0x7f00200940e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:29.853 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:29.843 7f00286a7700 1 -- 172.21.2.112:0/2275929910 shutdown_connections 2021-02-10T11:40:29.853 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:29.843 7f00286a7700 1 -- 172.21.2.112:0/2275929910 wait complete. 2021-02-10T11:40:29.854 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:40:30.283 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.270 7fd29d906700 1 Processor -- start 2021-02-10T11:40:30.283 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.270 7fd29d906700 1 -- start start 2021-02-10T11:40:30.283 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.271 7fd29d906700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fd2980d0340 0x7fd2980ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:30.283 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.271 7fd29d906700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2980f0b10 con 0x7fd2980d1c70 2021-02-10T11:40:30.284 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.271 7fd29d906700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2981b0be0 con 0x7fd2980d1870 2021-02-10T11:40:30.284 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.271 7fd29d906700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2980e5ed0 con 0x7fd2980d0340 2021-02-10T11:40:30.284 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.271 7fd297d75700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fd2980d1c70 0x7fd2980efc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:54118/0 (socket says 172.21.2.112:54118) 2021-02-10T11:40:30.284 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.271 7fd297d75700 1 -- 172.21.2.112:0/3050363641 learned_addr learned my addr 172.21.2.112:0/3050363641 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:40:30.284 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.271 7fd296d73700 1 --2- 172.21.2.112:0/3050363641 >> v2:172.21.2.120:3300/0 conn(0x7fd2980d0340 0x7fd2980ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:30.284 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.271 7fd296572700 1 -- 172.21.2.112:0/3050363641 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 557501071 0 0) 0x7fd290000f90 con 0x7fd2980d1c70 2021-02-10T11:40:30.284 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.271 7fd296572700 1 -- 172.21.2.112:0/3050363641 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd284002fe0 con 0x7fd2980d1c70 2021-02-10T11:40:30.284 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.271 7fd296d73700 1 -- 172.21.2.112:0/3050363641 >> v1:172.21.2.120:6789/0 conn(0x7fd2980d1870 legacy=0x7fd2981b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:30.284 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.271 7fd296d73700 1 -- 172.21.2.112:0/3050363641 >> v1:172.21.2.112:6789/0 conn(0x7fd2980d1c70 legacy=0x7fd2980efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:30.284 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.271 7fd296d73700 1 -- 172.21.2.112:0/3050363641 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2980f0450 con 0x7fd2980d0340 2021-02-10T11:40:30.284 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.271 7fd296d73700 1 --2- 172.21.2.112:0/3050363641 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd2980d0340 0x7fd2980ef640 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 rx=0x7fd280006650 tx=0x7fd280009a80).ready entity=mon.1 client_cookie=91b77294feabbf15 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:30.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.272 7fd296572700 1 -- 172.21.2.112:0/3050363641 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd2800009b0 con 0x7fd2980d0340 2021-02-10T11:40:30.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.272 7fd296572700 1 -- 172.21.2.112:0/3050363641 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd28000c0c0 con 0x7fd2980d0340 2021-02-10T11:40:30.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.272 7fd296572700 1 -- 172.21.2.112:0/3050363641 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd28000c5e0 con 0x7fd2980d0340 2021-02-10T11:40:30.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.272 7fd29d906700 1 -- 172.21.2.112:0/3050363641 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd2980d0340 msgr2=0x7fd2980ef640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:30.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.272 7fd29d906700 1 --2- 172.21.2.112:0/3050363641 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd2980d0340 0x7fd2980ef640 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 rx=0x7fd280006650 tx=0x7fd280009a80).stop 2021-02-10T11:40:30.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.273 7fd29d906700 1 -- 172.21.2.112:0/3050363641 shutdown_connections 2021-02-10T11:40:30.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.273 7fd29d906700 1 --2- 172.21.2.112:0/3050363641 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd2980d0340 0x7fd2980ef640 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:30.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.273 7fd29d906700 1 -- 172.21.2.112:0/3050363641 >> 172.21.2.112:0/3050363641 conn(0x7fd2980ce450 msgr2=0x7fd2980d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:30.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.273 7fd29d906700 1 -- 172.21.2.112:0/3050363641 shutdown_connections 2021-02-10T11:40:30.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.273 7fd29d906700 1 -- 172.21.2.112:0/3050363641 wait complete. 2021-02-10T11:40:30.287 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.287 7fd29d906700 1 Processor -- start 2021-02-10T11:40:30.287 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.287 7fd29d906700 1 -- start start 2021-02-10T11:40:30.287 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.288 7fd29d906700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd29818f0a0 0x7fd298196ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:30.287 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.288 7fd29d906700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd2981970d0 0x7fd2981974d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:30.287 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.288 7fd29d906700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd298094650 0x7fd298094a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:30.287 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.288 7fd29d906700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd298095bd0 con 0x7fd2981970d0 2021-02-10T11:40:30.287 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.288 7fd29d906700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2980ad7c0 con 0x7fd298094650 2021-02-10T11:40:30.287 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.288 7fd29d906700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd29818ec70 con 0x7fd29818f0a0 2021-02-10T11:40:30.287 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.288 7fd296d73700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd2981970d0 0x7fd2981974d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:30.287 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.288 7fd296d73700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd2981970d0 0x7fd2981974d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:54658/0 (socket says 172.21.2.112:54658) 2021-02-10T11:40:30.287 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.288 7fd296d73700 1 -- 172.21.2.112:0/2700796362 learned_addr learned my addr 172.21.2.112:0/2700796362 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:40:30.288 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.288 7fd297574700 1 --2- 172.21.2.112:0/2700796362 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd29818f0a0 0x7fd298196ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:30.288 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.288 7fd296d73700 1 -- 172.21.2.112:0/2700796362 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd29818f0a0 msgr2=0x7fd298196ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:30.288 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.288 7fd296d73700 1 --2- 172.21.2.112:0/2700796362 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd29818f0a0 0x7fd298196ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:30.288 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.288 7fd296d73700 1 -- 172.21.2.112:0/2700796362 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd298094650 msgr2=0x7fd298094a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:30.288 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.288 7fd296d73700 1 --2- 172.21.2.112:0/2700796362 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd298094650 0x7fd298094a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:30.288 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.288 7fd296d73700 1 -- 172.21.2.112:0/2700796362 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd298095070 con 0x7fd2981970d0 2021-02-10T11:40:30.288 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.288 7fd296d73700 1 --2- 172.21.2.112:0/2700796362 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd2981970d0 0x7fd2981974d0 secure :-1 s=READY pgs=2065 cs=0 l=1 rev1=1 rx=0x7fd280009630 tx=0x7fd28000b900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:30.288 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.289 7fd28ffff700 1 -- 172.21.2.112:0/2700796362 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd2800093a0 con 0x7fd2981970d0 2021-02-10T11:40:30.288 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.289 7fd28ffff700 1 -- 172.21.2.112:0/2700796362 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd28000bcf0 con 0x7fd2981970d0 2021-02-10T11:40:30.288 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.289 7fd28ffff700 1 -- 172.21.2.112:0/2700796362 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd280005560 con 0x7fd2981970d0 2021-02-10T11:40:30.288 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.289 7fd29d906700 1 -- 172.21.2.112:0/2700796362 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd298095070 con 0x7fd2981970d0 2021-02-10T11:40:30.288 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.289 7fd29d906700 1 -- 172.21.2.112:0/2700796362 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd298095600 con 0x7fd2981970d0 2021-02-10T11:40:30.288 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.289 7fd28ffff700 1 -- 172.21.2.112:0/2700796362 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd2800054f0 con 0x7fd2981970d0 2021-02-10T11:40:30.289 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.289 7fd28ffff700 1 --2- 172.21.2.112:0/2700796362 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd27c0186c0 0x7fd27c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:30.289 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.289 7fd28ffff700 1 -- 172.21.2.112:0/2700796362 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 4753+0+0 (secure 0 0 0) 0x7fd2800320b0 con 0x7fd2981970d0 2021-02-10T11:40:30.289 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.290 7fd297574700 1 --2- 172.21.2.112:0/2700796362 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd27c0186c0 0x7fd27c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:30.289 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.291 7fd297574700 1 --2- 172.21.2.112:0/2700796362 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd27c0186c0 0x7fd27c01abc0 crc :-1 s=READY pgs=956 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:30.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.321 7fd29d906700 1 -- 172.21.2.112:0/2700796362 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2980ad790 con 0x7fd2981970d0 2021-02-10T11:40:30.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.323 7fd28ffff700 1 -- 172.21.2.112:0/2700796362 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd28001c410 con 0x7fd2981970d0 2021-02-10T11:40:30.445 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.449 7fd29d906700 1 -- 172.21.2.112:0/2700796362 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd2980ad790 con 0x7fd2981970d0 2021-02-10T11:40:30.445 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.449 7fd28ffff700 1 -- 172.21.2.112:0/2700796362 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v104) v1 ==== 74+0+11146 (secure 0 0 0) 0x7fd28001c410 con 0x7fd2981970d0 2021-02-10T11:40:30.446 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:40:30.446 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":104,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:40:29.194745","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:40:30.447 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.451 7fd29d906700 1 -- 172.21.2.112:0/2700796362 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd27c0186c0 msgr2=0x7fd27c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:30.447 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.451 7fd29d906700 1 --2- 172.21.2.112:0/2700796362 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd27c0186c0 0x7fd27c01abc0 crc :-1 s=READY pgs=956 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:30.447 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.451 7fd29d906700 1 -- 172.21.2.112:0/2700796362 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd2981970d0 msgr2=0x7fd2981974d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:30.447 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.451 7fd29d906700 1 --2- 172.21.2.112:0/2700796362 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd2981970d0 0x7fd2981974d0 secure :-1 s=READY pgs=2065 cs=0 l=1 rev1=1 rx=0x7fd280009630 tx=0x7fd28000b900).stop 2021-02-10T11:40:30.447 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.451 7fd29d906700 1 -- 172.21.2.112:0/2700796362 shutdown_connections 2021-02-10T11:40:30.447 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.451 7fd29d906700 1 --2- 172.21.2.112:0/2700796362 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd27c0186c0 0x7fd27c01abc0 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:30.447 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.451 7fd29d906700 1 --2- 172.21.2.112:0/2700796362 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd298094650 0x7fd298094a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:30.448 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.451 7fd29d906700 1 --2- 172.21.2.112:0/2700796362 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd2981970d0 0x7fd2981974d0 unknown :-1 s=CLOSED pgs=2065 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:30.448 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.451 7fd29d906700 1 --2- 172.21.2.112:0/2700796362 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd29818f0a0 0x7fd298196ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:30.448 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.451 7fd29d906700 1 -- 172.21.2.112:0/2700796362 >> 172.21.2.112:0/2700796362 conn(0x7fd2981a38c0 msgr2=0x7fd2981a2f00 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:30.448 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.451 7fd29d906700 1 -- 172.21.2.112:0/2700796362 shutdown_connections 2021-02-10T11:40:30.448 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.451 7fd29d906700 1 -- 172.21.2.112:0/2700796362 wait complete. 2021-02-10T11:40:30.456 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:13:57.508758', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2021-02-10T11:40:30.456 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:40:30.887 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.875 7f1d6e142700 1 Processor -- start 2021-02-10T11:40:30.888 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.875 7f1d6e142700 1 -- start start 2021-02-10T11:40:30.888 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.876 7f1d6e142700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f1d680d1c70 0x7f1d680cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:30.888 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.876 7f1d6e142700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f1d680cf580 0x7f1d680cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:30.888 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.876 7f1d6e142700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d680fe1c0 con 0x7f1d680d1870 2021-02-10T11:40:30.888 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.876 7f1d6e142700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d681b0cb0 con 0x7f1d680cf580 2021-02-10T11:40:30.888 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.876 7f1d6e142700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d68109a60 con 0x7f1d680d1c70 2021-02-10T11:40:30.888 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.876 7f1d67fff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f1d680cf580 0x7f1d680cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:30.888 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.876 7f1d67fff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f1d680cf580 0x7f1d680cf980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:54670/0 (socket says 172.21.2.112:54670) 2021-02-10T11:40:30.888 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.876 7f1d67fff700 1 -- 172.21.2.112:0/690602870 learned_addr learned my addr 172.21.2.112:0/690602870 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:40:30.888 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.876 7f1d67fff700 1 -- 172.21.2.112:0/690602870 >> v2:172.21.2.120:3300/0 conn(0x7f1d680d1c70 msgr2=0x7f1d680cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:30.888 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.876 7f1d67fff700 1 --2- 172.21.2.112:0/690602870 >> v2:172.21.2.120:3300/0 conn(0x7f1d680d1c70 0x7f1d680cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:30.889 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.876 7f1d67fff700 1 -- 172.21.2.112:0/690602870 >> v1:172.21.2.120:6789/0 conn(0x7f1d680d1870 legacy=0x7f1d681b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:30.889 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.876 7f1d67fff700 1 -- 172.21.2.112:0/690602870 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d68088070 con 0x7f1d680cf580 2021-02-10T11:40:30.889 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.876 7f1d67fff700 1 --2- 172.21.2.112:0/690602870 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1d680cf580 0x7f1d680cf980 secure :-1 s=READY pgs=2066 cs=0 l=1 rev1=1 rx=0x7f1d60006f00 tx=0x7f1d60009f10).ready entity=mon.0 client_cookie=e2f0cbc251e2e6ff server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:30.889 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.876 7f1d667fc700 1 -- 172.21.2.112:0/690602870 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1d60003030 con 0x7f1d680cf580 2021-02-10T11:40:30.889 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.876 7f1d667fc700 1 -- 172.21.2.112:0/690602870 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d6000b3b0 con 0x7f1d680cf580 2021-02-10T11:40:30.889 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.876 7f1d667fc700 1 -- 172.21.2.112:0/690602870 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1d6000b8d0 con 0x7f1d680cf580 2021-02-10T11:40:30.889 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.876 7f1d6e142700 1 -- 172.21.2.112:0/690602870 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1d680cf580 msgr2=0x7f1d680cf980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:30.889 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.876 7f1d6e142700 1 --2- 172.21.2.112:0/690602870 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1d680cf580 0x7f1d680cf980 secure :-1 s=READY pgs=2066 cs=0 l=1 rev1=1 rx=0x7f1d60006f00 tx=0x7f1d60009f10).stop 2021-02-10T11:40:30.889 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.877 7f1d6e142700 1 -- 172.21.2.112:0/690602870 shutdown_connections 2021-02-10T11:40:30.889 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.877 7f1d6e142700 1 --2- 172.21.2.112:0/690602870 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1d680cf580 0x7f1d680cf980 unknown :-1 s=CLOSED pgs=2066 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:30.889 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.877 7f1d6e142700 1 --2- 172.21.2.112:0/690602870 >> v2:172.21.2.120:3300/0 conn(0x7f1d680d1c70 0x7f1d680cef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:30.889 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.877 7f1d6e142700 1 -- 172.21.2.112:0/690602870 >> 172.21.2.112:0/690602870 conn(0x7f1d680ce420 msgr2=0x7f1d680d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:30.889 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.877 7f1d6e142700 1 -- 172.21.2.112:0/690602870 shutdown_connections 2021-02-10T11:40:30.890 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.877 7f1d6e142700 1 -- 172.21.2.112:0/690602870 wait complete. 2021-02-10T11:40:30.891 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.892 7f1d6e142700 1 Processor -- start 2021-02-10T11:40:30.891 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.892 7f1d6e142700 1 -- start start 2021-02-10T11:40:30.891 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.892 7f1d6e142700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1d680c2370 0x7f1d6818f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:30.891 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.892 7f1d6e142700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1d68084cb0 0x7f1d681a7470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:30.891 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.892 7f1d6e142700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1d681a7a90 0x7f1d68085700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:30.891 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.892 7f1d6e142700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d680be840 con 0x7f1d68084cb0 2021-02-10T11:40:30.891 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.892 7f1d6e142700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d680c29a0 con 0x7f1d681a7a90 2021-02-10T11:40:30.892 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.892 7f1d6e142700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d681a7e90 con 0x7f1d680c2370 2021-02-10T11:40:30.892 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.892 7f1d66ffd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1d68084cb0 0x7f1d681a7470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:30.892 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.892 7f1d66ffd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1d68084cb0 0x7f1d681a7470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:54674/0 (socket says 172.21.2.112:54674) 2021-02-10T11:40:30.892 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.892 7f1d66ffd700 1 -- 172.21.2.112:0/3926348895 learned_addr learned my addr 172.21.2.112:0/3926348895 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:40:30.892 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.893 7f1d66ffd700 1 -- 172.21.2.112:0/3926348895 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1d680c2370 msgr2=0x7f1d6818f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:30.892 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.893 7f1d66ffd700 1 --2- 172.21.2.112:0/3926348895 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1d680c2370 0x7f1d6818f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:30.892 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.893 7f1d66ffd700 1 -- 172.21.2.112:0/3926348895 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1d681a7a90 msgr2=0x7f1d68085700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:30.892 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.893 7f1d66ffd700 1 --2- 172.21.2.112:0/3926348895 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1d681a7a90 0x7f1d68085700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:30.892 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.893 7f1d66ffd700 1 -- 172.21.2.112:0/3926348895 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d681a80b0 con 0x7f1d68084cb0 2021-02-10T11:40:30.892 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.893 7f1d66ffd700 1 --2- 172.21.2.112:0/3926348895 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1d68084cb0 0x7f1d681a7470 secure :-1 s=READY pgs=2067 cs=0 l=1 rev1=1 rx=0x7f1d58006f00 tx=0x7f1d5800b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:30.892 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.893 7f1d5ffff700 1 -- 172.21.2.112:0/3926348895 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1d58006350 con 0x7f1d68084cb0 2021-02-10T11:40:30.892 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.893 7f1d5ffff700 1 -- 172.21.2.112:0/3926348895 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d5800bdc0 con 0x7f1d68084cb0 2021-02-10T11:40:30.892 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.893 7f1d5ffff700 1 -- 172.21.2.112:0/3926348895 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1d58002020 con 0x7f1d68084cb0 2021-02-10T11:40:30.892 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.893 7f1d6e142700 1 -- 172.21.2.112:0/3926348895 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d681a80b0 con 0x7f1d68084cb0 2021-02-10T11:40:30.893 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.893 7f1d6e142700 1 -- 172.21.2.112:0/3926348895 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d681a80b0 con 0x7f1d68084cb0 2021-02-10T11:40:30.893 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.893 7f1d5ffff700 1 -- 172.21.2.112:0/3926348895 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1d58007020 con 0x7f1d68084cb0 2021-02-10T11:40:30.893 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.893 7f1d5ffff700 1 --2- 172.21.2.112:0/3926348895 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1d4c018730 0x7f1d4c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:30.893 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.894 7f1d5ffff700 1 -- 172.21.2.112:0/3926348895 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f1d580322f0 con 0x7f1d68084cb0 2021-02-10T11:40:30.893 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.894 7f1d677fe700 1 --2- 172.21.2.112:0/3926348895 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1d4c018730 0x7f1d4c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:30.893 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.894 7f1d677fe700 1 --2- 172.21.2.112:0/3926348895 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1d4c018730 0x7f1d4c01abc0 crc :-1 s=READY pgs=957 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:30.922 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.926 7f1d6e142700 1 -- 172.21.2.112:0/3926348895 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d680be840 con 0x7f1d68084cb0 2021-02-10T11:40:30.925 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:30.928 7f1d5ffff700 1 -- 172.21.2.112:0/3926348895 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1d58006b90 con 0x7f1d68084cb0 2021-02-10T11:40:31.053 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:31.056 7f1d6e142700 1 -- 172.21.2.112:0/3926348895 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f1d680be840 con 0x7f1d68084cb0 2021-02-10T11:40:31.053 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:31.057 7f1d5ffff700 1 -- 172.21.2.112:0/3926348895 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v104) v1 ==== 92+0+10 (secure 0 0 0) 0x7f1d5800bd30 con 0x7f1d68084cb0 2021-02-10T11:40:31.054 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:40:31.055 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:31.059 7f1d6e142700 1 -- 172.21.2.112:0/3926348895 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1d4c018730 msgr2=0x7f1d4c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:31.055 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:31.059 7f1d6e142700 1 --2- 172.21.2.112:0/3926348895 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1d4c018730 0x7f1d4c01abc0 crc :-1 s=READY pgs=957 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:31.055 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:31.059 7f1d6e142700 1 -- 172.21.2.112:0/3926348895 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1d68084cb0 msgr2=0x7f1d681a7470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:31.055 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:31.059 7f1d6e142700 1 --2- 172.21.2.112:0/3926348895 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1d68084cb0 0x7f1d681a7470 secure :-1 s=READY pgs=2067 cs=0 l=1 rev1=1 rx=0x7f1d58006f00 tx=0x7f1d5800b940).stop 2021-02-10T11:40:31.055 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:31.059 7f1d6e142700 1 -- 172.21.2.112:0/3926348895 shutdown_connections 2021-02-10T11:40:31.055 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:31.059 7f1d6e142700 1 --2- 172.21.2.112:0/3926348895 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1d4c018730 0x7f1d4c01abc0 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:31.055 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:31.059 7f1d6e142700 1 --2- 172.21.2.112:0/3926348895 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1d681a7a90 0x7f1d68085700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:31.056 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:31.059 7f1d6e142700 1 --2- 172.21.2.112:0/3926348895 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1d68084cb0 0x7f1d681a7470 unknown :-1 s=CLOSED pgs=2067 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:31.056 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:31.059 7f1d6e142700 1 --2- 172.21.2.112:0/3926348895 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1d680c2370 0x7f1d6818f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:31.056 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:31.059 7f1d6e142700 1 -- 172.21.2.112:0/3926348895 >> 172.21.2.112:0/3926348895 conn(0x7f1d680b5dc0 msgr2=0x7f1d680c2d90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:31.056 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:31.059 7f1d6e142700 1 -- 172.21.2.112:0/3926348895 shutdown_connections 2021-02-10T11:40:31.056 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:31.059 7f1d6e142700 1 -- 172.21.2.112:0/3926348895 wait complete. 2021-02-10T11:40:31.064 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --format=json-pretty osd lspools 2021-02-10T11:40:31.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:31.479 7fd6134c4700 1 Processor -- start 2021-02-10T11:40:31.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:31.479 7fd6134c4700 1 -- start start 2021-02-10T11:40:31.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:31.480 7fd6134c4700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fd60c0d1870 0x7fd60c1b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:31.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:31.480 7fd6134c4700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd60c1b0a10 0x7fd60c0cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:31.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:31.480 7fd6134c4700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd60c0f0b10 con 0x7fd60c0d1c70 2021-02-10T11:40:31.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:31.480 7fd6134c4700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd60c0e5ed0 con 0x7fd60c0d1870 2021-02-10T11:40:31.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:31.480 7fd6134c4700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd60c0f0650 con 0x7fd60c1b0a10 2021-02-10T11:40:31.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:31.480 7fd6112e0700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fd60c0d1870 0x7fd60c1b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:31.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:31.480 7fd6112e0700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fd60c0d1870 0x7fd60c1b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:54680/0 (socket says 172.21.2.112:54680) 2021-02-10T11:40:31.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:31.480 7fd6112e0700 1 -- 172.21.2.112:0/693607095 learned_addr learned my addr 172.21.2.112:0/693607095 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:40:31.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:31.480 7fd6112e0700 1 -- 172.21.2.112:0/693607095 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd60c1b0a10 msgr2=0x7fd60c0cf330 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:40:31.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:31.480 7fd6112e0700 1 --2- 172.21.2.112:0/693607095 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd60c1b0a10 0x7fd60c0cf330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:31.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:31.480 7fd6112e0700 1 -- 172.21.2.112:0/693607095 >> v1:172.21.2.120:6789/0 conn(0x7fd60c0d1c70 legacy=0x7fd60c0cf950 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:40:31.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:31.480 7fd6112e0700 1 -- 172.21.2.112:0/693607095 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd60c0f0870 con 0x7fd60c0d1870 2021-02-10T11:40:31.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:31.481 7fd6112e0700 1 --2- 172.21.2.112:0/693607095 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd60c0d1870 0x7fd60c1b03f0 secure :-1 s=READY pgs=2068 cs=0 l=1 rev1=1 rx=0x7fd600006700 tx=0x7fd600009b00).ready entity=mon.0 client_cookie=95b5a2d2b8d56f1a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:31.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:31.481 7fd5fffff700 1 -- 172.21.2.112:0/693607095 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd60000c0c0 con 0x7fd60c0d1870 2021-02-10T11:40:31.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:31.481 7fd5fffff700 1 -- 172.21.2.112:0/693607095 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd60000c340 con 0x7fd60c0d1870 2021-02-10T11:40:31.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:31.481 7fd5fffff700 1 -- 172.21.2.112:0/693607095 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd60000c820 con 0x7fd60c0d1870 2021-02-10T11:40:31.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:31.481 7fd6134c4700 1 -- 172.21.2.112:0/693607095 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd60c0d1870 msgr2=0x7fd60c1b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:31.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:31.481 7fd6134c4700 1 --2- 172.21.2.112:0/693607095 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd60c0d1870 0x7fd60c1b03f0 secure :-1 s=READY pgs=2068 cs=0 l=1 rev1=1 rx=0x7fd600006700 tx=0x7fd600009b00).stop 2021-02-10T11:40:31.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:31.481 7fd6134c4700 1 -- 172.21.2.112:0/693607095 shutdown_connections 2021-02-10T11:40:31.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:31.481 7fd6134c4700 1 --2- 172.21.2.112:0/693607095 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd60c1b0a10 0x7fd60c0cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:31.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:31.481 7fd6134c4700 1 --2- 172.21.2.112:0/693607095 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd60c0d1870 0x7fd60c1b03f0 unknown :-1 s=CLOSED pgs=2068 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:31.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:31.481 7fd6134c4700 1 -- 172.21.2.112:0/693607095 >> 172.21.2.112:0/693607095 conn(0x7fd60c0ce450 msgr2=0x7fd60c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:31.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:31.482 7fd6134c4700 1 -- 172.21.2.112:0/693607095 shutdown_connections 2021-02-10T11:40:31.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:31.482 7fd6134c4700 1 -- 172.21.2.112:0/693607095 wait complete. 2021-02-10T11:40:31.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:31.496 7fd6134c4700 1 Processor -- start 2021-02-10T11:40:31.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:31.496 7fd6134c4700 1 -- start start 2021-02-10T11:40:31.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:31.496 7fd6134c4700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd60c0b4f30 0x7fd60c18b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:31.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:31.496 7fd6134c4700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd60c093f40 0x7fd60c094340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:31.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:31.496 7fd6134c4700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd60c094960 0x7fd60c187e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:31.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:31.496 7fd6134c4700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd60c094e90 con 0x7fd60c094960 2021-02-10T11:40:31.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:31.496 7fd6134c4700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd60c18bff0 con 0x7fd60c0b4f30 2021-02-10T11:40:31.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:31.496 7fd6134c4700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd60c08f780 con 0x7fd60c093f40 2021-02-10T11:40:31.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:31.497 7fd611ae1700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd60c094960 0x7fd60c187e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:31.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:31.497 7fd611ae1700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd60c094960 0x7fd60c187e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:54688/0 (socket says 172.21.2.112:54688) 2021-02-10T11:40:31.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:31.497 7fd611ae1700 1 -- 172.21.2.112:0/3733680999 learned_addr learned my addr 172.21.2.112:0/3733680999 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:40:31.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:31.497 7fd610adf700 1 --2- 172.21.2.112:0/3733680999 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd60c093f40 0x7fd60c094340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:31.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:31.497 7fd6112e0700 1 --2- 172.21.2.112:0/3733680999 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd60c0b4f30 0x7fd60c18b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:31.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:31.497 7fd611ae1700 1 -- 172.21.2.112:0/3733680999 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd60c093f40 msgr2=0x7fd60c094340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:31.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:31.497 7fd611ae1700 1 --2- 172.21.2.112:0/3733680999 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd60c093f40 0x7fd60c094340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:31.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:31.497 7fd611ae1700 1 -- 172.21.2.112:0/3733680999 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd60c0b4f30 msgr2=0x7fd60c18b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:31.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:31.497 7fd611ae1700 1 --2- 172.21.2.112:0/3733680999 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd60c0b4f30 0x7fd60c18b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:31.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:31.497 7fd611ae1700 1 -- 172.21.2.112:0/3733680999 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd60c188470 con 0x7fd60c094960 2021-02-10T11:40:31.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:31.497 7fd611ae1700 1 --2- 172.21.2.112:0/3733680999 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd60c094960 0x7fd60c187e50 secure :-1 s=READY pgs=2069 cs=0 l=1 rev1=1 rx=0x7fd604009940 tx=0x7fd60400bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:31.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:31.498 7fd5fdffb700 1 -- 172.21.2.112:0/3733680999 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd604003030 con 0x7fd60c094960 2021-02-10T11:40:31.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:31.498 7fd5fdffb700 1 -- 172.21.2.112:0/3733680999 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd604001270 con 0x7fd60c094960 2021-02-10T11:40:31.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:31.498 7fd5fdffb700 1 -- 172.21.2.112:0/3733680999 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd6040016d0 con 0x7fd60c094960 2021-02-10T11:40:31.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:31.498 7fd6134c4700 1 -- 172.21.2.112:0/3733680999 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd60c188470 con 0x7fd60c094960 2021-02-10T11:40:31.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:31.498 7fd6134c4700 1 -- 172.21.2.112:0/3733680999 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd60c188a00 con 0x7fd60c094960 2021-02-10T11:40:31.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:31.498 7fd5fdffb700 1 -- 172.21.2.112:0/3733680999 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd604001450 con 0x7fd60c094960 2021-02-10T11:40:31.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:31.498 7fd5fdffb700 1 --2- 172.21.2.112:0/3733680999 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd5ec0186c0 0x7fd5ec01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:31.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:31.498 7fd5fdffb700 1 -- 172.21.2.112:0/3733680999 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 4753+0+0 (secure 0 0 0) 0x7fd6040328b0 con 0x7fd60c094960 2021-02-10T11:40:31.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:31.498 7fd6112e0700 1 --2- 172.21.2.112:0/3733680999 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd5ec0186c0 0x7fd5ec01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:31.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:31.499 7fd6112e0700 1 --2- 172.21.2.112:0/3733680999 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd5ec0186c0 0x7fd5ec01abc0 crc :-1 s=READY pgs=958 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:31.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:31.530 7fd6134c4700 1 -- 172.21.2.112:0/3733680999 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd60c0ce450 con 0x7fd60c094960 2021-02-10T11:40:31.529 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:31.533 7fd5fdffb700 1 -- 172.21.2.112:0/3733680999 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd604000f40 con 0x7fd60c094960 2021-02-10T11:40:31.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:31.659 7fd6134c4700 1 -- 172.21.2.112:0/3733680999 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fd60c0ce450 con 0x7fd60c094960 2021-02-10T11:40:31.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:31.659 7fd5fdffb700 1 -- 172.21.2.112:0/3733680999 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v104) v1 ==== 84+0+64 (secure 0 0 0) 0x7fd604001270 con 0x7fd60c094960 2021-02-10T11:40:31.656 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:40:31.656 INFO:teuthology.orchestra.run.gibba012.stdout:[ 2021-02-10T11:40:31.656 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:40:31.656 INFO:teuthology.orchestra.run.gibba012.stdout: "poolnum": 1, 2021-02-10T11:40:31.656 INFO:teuthology.orchestra.run.gibba012.stdout: "poolname": "rbd" 2021-02-10T11:40:31.656 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:40:31.656 INFO:teuthology.orchestra.run.gibba012.stdout:] 2021-02-10T11:40:31.658 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:31.662 7fd6134c4700 1 -- 172.21.2.112:0/3733680999 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd5ec0186c0 msgr2=0x7fd5ec01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:31.658 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:31.662 7fd6134c4700 1 --2- 172.21.2.112:0/3733680999 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd5ec0186c0 0x7fd5ec01abc0 crc :-1 s=READY pgs=958 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:31.658 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:31.662 7fd6134c4700 1 -- 172.21.2.112:0/3733680999 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd60c094960 msgr2=0x7fd60c187e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:31.658 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:31.662 7fd6134c4700 1 --2- 172.21.2.112:0/3733680999 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd60c094960 0x7fd60c187e50 secure :-1 s=READY pgs=2069 cs=0 l=1 rev1=1 rx=0x7fd604009940 tx=0x7fd60400bcf0).stop 2021-02-10T11:40:31.658 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:31.662 7fd6134c4700 1 -- 172.21.2.112:0/3733680999 shutdown_connections 2021-02-10T11:40:31.658 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:31.662 7fd6134c4700 1 --2- 172.21.2.112:0/3733680999 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd5ec0186c0 0x7fd5ec01abc0 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:31.658 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:31.662 7fd6134c4700 1 --2- 172.21.2.112:0/3733680999 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd60c094960 0x7fd60c187e50 unknown :-1 s=CLOSED pgs=2069 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:31.658 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:31.662 7fd6134c4700 1 --2- 172.21.2.112:0/3733680999 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd60c093f40 0x7fd60c094340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:31.658 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:31.662 7fd6134c4700 1 --2- 172.21.2.112:0/3733680999 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd60c0b4f30 0x7fd60c18b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:31.658 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:31.662 7fd6134c4700 1 -- 172.21.2.112:0/3733680999 >> 172.21.2.112:0/3733680999 conn(0x7fd60c19f8f0 msgr2=0x7fd60c1a5f80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:31.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:31.662 7fd6134c4700 1 -- 172.21.2.112:0/3733680999 shutdown_connections 2021-02-10T11:40:31.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:31.662 7fd6134c4700 1 -- 172.21.2.112:0/3733680999 wait complete. 2021-02-10T11:40:31.666 INFO:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2021-02-10T11:40:31.667 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_metadata 8 2021-02-10T11:40:32.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:32.085 7fd5bbbc7700 1 Processor -- start 2021-02-10T11:40:32.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:32.085 7fd5bbbc7700 1 -- start start 2021-02-10T11:40:32.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:32.085 7fd5bbbc7700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd5b40d1c70 0x7fd5b40cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:32.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:32.085 7fd5bbbc7700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fd5b40cf580 0x7fd5b40cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:32.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:32.085 7fd5bbbc7700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5b40fe1c0 con 0x7fd5b40d1870 2021-02-10T11:40:32.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:32.085 7fd5bbbc7700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5b41b0cb0 con 0x7fd5b40cf580 2021-02-10T11:40:32.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:32.085 7fd5bbbc7700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5b4109a60 con 0x7fd5b40d1c70 2021-02-10T11:40:32.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:32.086 7fd5b99e3700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fd5b40d1870 0x7fd5b41b04c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:54154/0 (socket says 172.21.2.112:54154) 2021-02-10T11:40:32.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:32.086 7fd5b99e3700 1 -- 172.21.2.112:0/1772499928 learned_addr learned my addr 172.21.2.112:0/1772499928 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:40:32.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:32.086 7fd5b89e1700 1 -- 172.21.2.112:0/1772499928 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1862601677 0 0) 0x7fd5b00010b0 con 0x7fd5b40d1870 2021-02-10T11:40:32.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:32.086 7fd5b89e1700 1 -- 172.21.2.112:0/1772499928 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5a0002fe0 con 0x7fd5b40d1870 2021-02-10T11:40:32.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:32.086 7fd5ba1e4700 1 --2- 172.21.2.112:0/1772499928 >> v2:172.21.2.120:3300/0 conn(0x7fd5b40cf580 0x7fd5b40cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:32.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:32.086 7fd5b91e2700 1 --2- 172.21.2.112:0/1772499928 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd5b40d1c70 0x7fd5b40cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:32.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:32.086 7fd5b89e1700 1 -- 172.21.2.112:0/1772499928 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2033664715 0 0) 0x7fd5b00010b0 con 0x7fd5b40d1870 2021-02-10T11:40:32.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:32.086 7fd5b89e1700 1 -- 172.21.2.112:0/1772499928 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd5b40d1c70 msgr2=0x7fd5b40cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:32.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:32.086 7fd5b89e1700 1 --2- 172.21.2.112:0/1772499928 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd5b40d1c70 0x7fd5b40cef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:32.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:32.086 7fd5b89e1700 1 -- 172.21.2.112:0/1772499928 >> v2:172.21.2.120:3300/0 conn(0x7fd5b40cf580 msgr2=0x7fd5b40cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:32.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:32.086 7fd5b89e1700 1 --2- 172.21.2.112:0/1772499928 >> v2:172.21.2.120:3300/0 conn(0x7fd5b40cf580 0x7fd5b40cf980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:32.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:32.086 7fd5b89e1700 1 -- 172.21.2.112:0/1772499928 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5b4088070 con 0x7fd5b40d1870 2021-02-10T11:40:32.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:32.086 7fd5b89e1700 1 -- 172.21.2.112:0/1772499928 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7fd5b00015b0 con 0x7fd5b40d1870 2021-02-10T11:40:32.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:32.086 7fd5b89e1700 1 -- 172.21.2.112:0/1772499928 >> v1:172.21.2.112:6789/0 conn(0x7fd5b40d1870 legacy=0x7fd5b41b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:32.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:32.086 7fd5b89e1700 1 --2- 172.21.2.112:0/1772499928 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd5a0003b10 0x7fd5a0005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:32.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:32.087 7fd5b89e1700 1 --2- 172.21.2.112:0/1772499928 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd5a004a800 0x7fd5a004cca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:32.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:32.087 7fd5b89e1700 1 --2- 172.21.2.112:0/1772499928 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd5a004d5a0 0x7fd5a004fa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:32.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:32.087 7fd5b89e1700 1 -- 172.21.2.112:0/1772499928 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5a00502e0 con 0x7fd5a004a800 2021-02-10T11:40:32.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:32.087 7fd5b89e1700 1 -- 172.21.2.112:0/1772499928 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5a0050500 con 0x7fd5a0003b10 2021-02-10T11:40:32.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:32.087 7fd5b89e1700 1 -- 172.21.2.112:0/1772499928 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5a0050720 con 0x7fd5a004d5a0 2021-02-10T11:40:32.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:32.087 7fd5b99e3700 1 --2- 172.21.2.112:0/1772499928 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd5a0003b10 0x7fd5a0005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:32.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:32.087 7fd5b91e2700 1 --2- 172.21.2.112:0/1772499928 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd5a004a800 0x7fd5a004cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:32.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:32.087 7fd5ba1e4700 1 --2- 172.21.2.112:0/1772499928 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd5a004d5a0 0x7fd5a004fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:32.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:32.087 7fd5b91e2700 1 -- 172.21.2.112:0/1772499928 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd5a004d5a0 msgr2=0x7fd5a004fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:32.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:32.087 7fd5b91e2700 1 --2- 172.21.2.112:0/1772499928 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd5a004d5a0 0x7fd5a004fa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:32.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:32.087 7fd5b91e2700 1 -- 172.21.2.112:0/1772499928 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd5a0003b10 msgr2=0x7fd5a0005fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:32.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:32.087 7fd5b91e2700 1 --2- 172.21.2.112:0/1772499928 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd5a0003b10 0x7fd5a0005fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:32.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:32.087 7fd5b91e2700 1 -- 172.21.2.112:0/1772499928 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5a00509c0 con 0x7fd5a004a800 2021-02-10T11:40:32.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:32.087 7fd5b91e2700 1 --2- 172.21.2.112:0/1772499928 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd5a004a800 0x7fd5a004cca0 secure :-1 s=READY pgs=2071 cs=0 l=1 rev1=1 rx=0x7fd5a4006ac0 tx=0x7fd5a4009ab0).ready entity=mon.0 client_cookie=a4005ab316868628 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:32.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:32.087 7fd5b89e1700 1 -- 172.21.2.112:0/1772499928 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd5a400c0c0 con 0x7fd5a004a800 2021-02-10T11:40:32.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:32.087 7fd5b89e1700 1 -- 172.21.2.112:0/1772499928 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5a400c3c0 con 0x7fd5a004a800 2021-02-10T11:40:32.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:32.087 7fd5b89e1700 1 -- 172.21.2.112:0/1772499928 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd5a400c0c0 con 0x7fd5a004a800 2021-02-10T11:40:32.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:32.087 7fd5bbbc7700 1 -- 172.21.2.112:0/1772499928 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd5a004a800 msgr2=0x7fd5a004cca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:32.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:32.087 7fd5bbbc7700 1 --2- 172.21.2.112:0/1772499928 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd5a004a800 0x7fd5a004cca0 secure :-1 s=READY pgs=2071 cs=0 l=1 rev1=1 rx=0x7fd5a4006ac0 tx=0x7fd5a4009ab0).stop 2021-02-10T11:40:32.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:32.087 7fd5ba1e4700 1 -- 172.21.2.112:0/1772499928 reap_dead start 2021-02-10T11:40:32.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:32.088 7fd5bbbc7700 1 -- 172.21.2.112:0/1772499928 shutdown_connections 2021-02-10T11:40:32.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:32.088 7fd5bbbc7700 1 -- 172.21.2.112:0/1772499928 >> 172.21.2.112:0/1772499928 conn(0x7fd5b40ce420 msgr2=0x7fd5b40d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:32.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:32.088 7fd5bbbc7700 1 -- 172.21.2.112:0/1772499928 shutdown_connections 2021-02-10T11:40:32.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:32.088 7fd5bbbc7700 1 -- 172.21.2.112:0/1772499928 wait complete. 2021-02-10T11:40:32.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:32.103 7fd5bbbc7700 1 Processor -- start 2021-02-10T11:40:32.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:32.103 7fd5bbbc7700 1 -- start start 2021-02-10T11:40:32.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:32.103 7fd5bbbc7700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd5b4085bc0 0x7fd5b40863e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:32.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:32.103 7fd5bbbc7700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd5b40954c0 0x7fd5b4096300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:32.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:32.103 7fd5bbbc7700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd5b40958c0 0x7fd5b4096920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:32.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:32.103 7fd5bbbc7700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5b4095d80 con 0x7fd5b4085bc0 2021-02-10T11:40:32.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:32.103 7fd5bbbc7700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5b40bfa60 con 0x7fd5b40958c0 2021-02-10T11:40:32.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:32.103 7fd5bbbc7700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5b4085fc0 con 0x7fd5b40954c0 2021-02-10T11:40:32.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:32.103 7fd5b99e3700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd5b4085bc0 0x7fd5b40863e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:32.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:32.103 7fd5b99e3700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd5b4085bc0 0x7fd5b40863e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:54704/0 (socket says 172.21.2.112:54704) 2021-02-10T11:40:32.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:32.103 7fd5b99e3700 1 -- 172.21.2.112:0/3550823985 learned_addr learned my addr 172.21.2.112:0/3550823985 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:40:32.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:32.104 7fd5b99e3700 1 -- 172.21.2.112:0/3550823985 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd5b40954c0 msgr2=0x7fd5b4096300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:32.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:32.104 7fd5b99e3700 1 --2- 172.21.2.112:0/3550823985 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd5b40954c0 0x7fd5b4096300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:32.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:32.104 7fd5b99e3700 1 -- 172.21.2.112:0/3550823985 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd5b40958c0 msgr2=0x7fd5b4096920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:32.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:32.104 7fd5b99e3700 1 --2- 172.21.2.112:0/3550823985 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd5b40958c0 0x7fd5b4096920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:32.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:32.104 7fd5b99e3700 1 -- 172.21.2.112:0/3550823985 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5b4086c10 con 0x7fd5b4085bc0 2021-02-10T11:40:32.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:32.104 7fd5b99e3700 1 --2- 172.21.2.112:0/3550823985 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd5b4085bc0 0x7fd5b40863e0 secure :-1 s=READY pgs=2072 cs=0 l=1 rev1=1 rx=0x7fd5b0007650 tx=0x7fd5b000a910).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:32.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:32.104 7fd5aa7fc700 1 -- 172.21.2.112:0/3550823985 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd5b000ad00 con 0x7fd5b4085bc0 2021-02-10T11:40:32.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:32.104 7fd5aa7fc700 1 -- 172.21.2.112:0/3550823985 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5b0004350 con 0x7fd5b4085bc0 2021-02-10T11:40:32.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:32.104 7fd5aa7fc700 1 -- 172.21.2.112:0/3550823985 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd5b0004870 con 0x7fd5b4085bc0 2021-02-10T11:40:32.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:32.104 7fd5bbbc7700 1 -- 172.21.2.112:0/3550823985 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5b4086c10 con 0x7fd5b4085bc0 2021-02-10T11:40:32.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:32.104 7fd5bbbc7700 1 -- 172.21.2.112:0/3550823985 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5b4086c10 con 0x7fd5b4085bc0 2021-02-10T11:40:32.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:32.105 7fd5aa7fc700 1 -- 172.21.2.112:0/3550823985 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd5b0004530 con 0x7fd5b4085bc0 2021-02-10T11:40:32.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:32.105 7fd5aa7fc700 1 --2- 172.21.2.112:0/3550823985 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd59c0186c0 0x7fd59c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:32.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:32.105 7fd5aa7fc700 1 -- 172.21.2.112:0/3550823985 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 4753+0+0 (secure 0 0 0) 0x7fd5b00322a0 con 0x7fd5b4085bc0 2021-02-10T11:40:32.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:32.105 7fd5b91e2700 1 --2- 172.21.2.112:0/3550823985 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd59c0186c0 0x7fd59c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:32.134 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:32.135 7fd5bbbc7700 1 -- 172.21.2.112:0/3550823985 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5b40fe1c0 con 0x7fd5b4085bc0 2021-02-10T11:40:32.135 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:32.138 7fd5aa7fc700 1 -- 172.21.2.112:0/3550823985 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd5b0004140 con 0x7fd5b4085bc0 2021-02-10T11:40:32.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:32.146 7fd5b91e2700 1 --2- 172.21.2.112:0/3550823985 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd59c0186c0 0x7fd59c01abc0 crc :-1 s=READY pgs=959 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:32.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:32.266 7fd5bbbc7700 1 -- 172.21.2.112:0/3550823985 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"} v 0) v1 -- 0x7fd5b40fe1c0 con 0x7fd5b4085bc0 2021-02-10T11:40:32.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:32.681 7fd5aa7fc700 1 -- 172.21.2.112:0/3550823985 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"}]=0 pool 'cephfs_metadata' created v105) v1 ==== 133+0+0 (secure 0 0 0) 0x7fd5b0004510 con 0x7fd5b4085bc0 2021-02-10T11:40:32.678 INFO:teuthology.orchestra.run.gibba012.stderr:pool 'cephfs_metadata' created 2021-02-10T11:40:32.680 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:32.684 7fd5bbbc7700 1 -- 172.21.2.112:0/3550823985 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd59c0186c0 msgr2=0x7fd59c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:32.680 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:32.685 7fd5bbbc7700 1 --2- 172.21.2.112:0/3550823985 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd59c0186c0 0x7fd59c01abc0 crc :-1 s=READY pgs=959 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:32.680 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:32.685 7fd5bbbc7700 1 -- 172.21.2.112:0/3550823985 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd5b4085bc0 msgr2=0x7fd5b40863e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:32.681 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:32.685 7fd5bbbc7700 1 --2- 172.21.2.112:0/3550823985 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd5b4085bc0 0x7fd5b40863e0 secure :-1 s=READY pgs=2072 cs=0 l=1 rev1=1 rx=0x7fd5b0007650 tx=0x7fd5b000a910).stop 2021-02-10T11:40:32.681 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:32.685 7fd5bbbc7700 1 -- 172.21.2.112:0/3550823985 shutdown_connections 2021-02-10T11:40:32.681 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:32.685 7fd5bbbc7700 1 --2- 172.21.2.112:0/3550823985 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd59c0186c0 0x7fd59c01abc0 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:32.681 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:32.685 7fd5bbbc7700 1 --2- 172.21.2.112:0/3550823985 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd5b40958c0 0x7fd5b4096920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:32.681 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:32.685 7fd5bbbc7700 1 --2- 172.21.2.112:0/3550823985 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd5b40954c0 0x7fd5b4096300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:32.681 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:32.685 7fd5bbbc7700 1 --2- 172.21.2.112:0/3550823985 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd5b4085bc0 0x7fd5b40863e0 unknown :-1 s=CLOSED pgs=2072 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:32.681 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:32.685 7fd5bbbc7700 1 -- 172.21.2.112:0/3550823985 >> 172.21.2.112:0/3550823985 conn(0x7fd5b40bef00 msgr2=0x7fd5b409d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:32.681 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:32.685 7fd5bbbc7700 1 -- 172.21.2.112:0/3550823985 shutdown_connections 2021-02-10T11:40:32.681 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:32.685 7fd5bbbc7700 1 -- 172.21.2.112:0/3550823985 wait complete. 2021-02-10T11:40:32.689 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_data 8 2021-02-10T11:40:33.118 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:33.107 7fa6555a0700 1 Processor -- start 2021-02-10T11:40:33.118 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:33.107 7fa6555a0700 1 -- start start 2021-02-10T11:40:33.119 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:33.107 7fa6555a0700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fa6500d1870 0x7fa6501b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:33.119 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:33.107 7fa6555a0700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fa6501b0ae0 0x7fa650108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:33.119 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:33.107 7fa6555a0700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa6500d1c70 0x7fa650108f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:33.119 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:33.107 7fa6555a0700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6500fe1c0 con 0x7fa6501b0ae0 2021-02-10T11:40:33.119 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:33.107 7fa6555a0700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa650109a60 con 0x7fa6500d1870 2021-02-10T11:40:33.119 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:33.107 7fa6555a0700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa650088070 con 0x7fa6500d1c70 2021-02-10T11:40:33.119 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:33.107 7fa64eabe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fa6501b0ae0 0x7fa650108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:33.119 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:33.108 7fa64eabe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fa6501b0ae0 0x7fa650108940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:54712/0 (socket says 172.21.2.112:54712) 2021-02-10T11:40:33.119 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:33.108 7fa64eabe700 1 -- 172.21.2.112:0/4285539568 learned_addr learned my addr 172.21.2.112:0/4285539568 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:40:33.119 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:33.108 7fa64eabe700 1 -- 172.21.2.112:0/4285539568 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa6500d1c70 msgr2=0x7fa650108f60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:40:33.119 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:33.108 7fa64eabe700 1 --2- 172.21.2.112:0/4285539568 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa6500d1c70 0x7fa650108f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:33.119 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:33.108 7fa64eabe700 1 -- 172.21.2.112:0/4285539568 >> v2:172.21.2.120:3300/0 conn(0x7fa6500d1870 msgr2=0x7fa6501b04c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:40:33.120 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:33.108 7fa64eabe700 1 --2- 172.21.2.112:0/4285539568 >> v2:172.21.2.120:3300/0 conn(0x7fa6500d1870 0x7fa6501b04c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:33.120 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:33.108 7fa64eabe700 1 -- 172.21.2.112:0/4285539568 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa650109580 con 0x7fa6501b0ae0 2021-02-10T11:40:33.120 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:33.108 7fa64eabe700 1 --2- 172.21.2.112:0/4285539568 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa6501b0ae0 0x7fa650108940 secure :-1 s=READY pgs=2073 cs=0 l=1 rev1=1 rx=0x7fa648006a80 tx=0x7fa648009f00).ready entity=mon.0 client_cookie=5c9f080c27181eb8 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:33.120 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:33.108 7fa64e2bd700 1 -- 172.21.2.112:0/4285539568 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa64800b5c0 con 0x7fa6501b0ae0 2021-02-10T11:40:33.120 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:33.108 7fa64e2bd700 1 -- 172.21.2.112:0/4285539568 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa64800b840 con 0x7fa6501b0ae0 2021-02-10T11:40:33.120 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:33.108 7fa64e2bd700 1 -- 172.21.2.112:0/4285539568 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa64800bd60 con 0x7fa6501b0ae0 2021-02-10T11:40:33.120 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:33.108 7fa6555a0700 1 -- 172.21.2.112:0/4285539568 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa6501b0ae0 msgr2=0x7fa650108940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:33.120 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:33.108 7fa6555a0700 1 --2- 172.21.2.112:0/4285539568 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa6501b0ae0 0x7fa650108940 secure :-1 s=READY pgs=2073 cs=0 l=1 rev1=1 rx=0x7fa648006a80 tx=0x7fa648009f00).stop 2021-02-10T11:40:33.120 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:33.109 7fa6555a0700 1 -- 172.21.2.112:0/4285539568 shutdown_connections 2021-02-10T11:40:33.120 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:33.109 7fa6555a0700 1 --2- 172.21.2.112:0/4285539568 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa6500d1c70 0x7fa650108f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:33.120 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:33.109 7fa6555a0700 1 --2- 172.21.2.112:0/4285539568 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa6501b0ae0 0x7fa650108940 unknown :-1 s=CLOSED pgs=2073 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:33.120 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:33.109 7fa6555a0700 1 --2- 172.21.2.112:0/4285539568 >> v2:172.21.2.120:3300/0 conn(0x7fa6500d1870 0x7fa6501b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:33.121 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:33.109 7fa6555a0700 1 -- 172.21.2.112:0/4285539568 >> 172.21.2.112:0/4285539568 conn(0x7fa6500ce420 msgr2=0x7fa6500d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:33.121 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:33.109 7fa6555a0700 1 -- 172.21.2.112:0/4285539568 shutdown_connections 2021-02-10T11:40:33.121 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:33.109 7fa6555a0700 1 -- 172.21.2.112:0/4285539568 wait complete. 2021-02-10T11:40:33.122 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:33.123 7fa6555a0700 1 Processor -- start 2021-02-10T11:40:33.122 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:33.123 7fa6555a0700 1 -- start start 2021-02-10T11:40:33.123 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:33.124 7fa6555a0700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa6500c2370 0x7fa65018f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:33.123 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:33.124 7fa6555a0700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa650084cb0 0x7fa6501a7470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:33.123 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:33.124 7fa6555a0700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa6501a7a90 0x7fa650085700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:33.123 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:33.124 7fa6555a0700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6500be840 con 0x7fa6500c2370 2021-02-10T11:40:33.123 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:33.124 7fa6555a0700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6500c29a0 con 0x7fa650084cb0 2021-02-10T11:40:33.123 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:33.124 7fa6555a0700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6501a7e90 con 0x7fa6501a7a90 2021-02-10T11:40:33.123 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:33.124 7fa64f2bf700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa6500c2370 0x7fa65018f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:33.123 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:33.124 7fa64f2bf700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa6500c2370 0x7fa65018f870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:54718/0 (socket says 172.21.2.112:54718) 2021-02-10T11:40:33.123 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:33.124 7fa64f2bf700 1 -- 172.21.2.112:0/76841745 learned_addr learned my addr 172.21.2.112:0/76841745 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:40:33.123 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:33.124 7fa64f2bf700 1 -- 172.21.2.112:0/76841745 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa6501a7a90 msgr2=0x7fa650085700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:33.123 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:33.124 7fa64f2bf700 1 --2- 172.21.2.112:0/76841745 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa6501a7a90 0x7fa650085700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:33.124 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:33.124 7fa64f2bf700 1 -- 172.21.2.112:0/76841745 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa650084cb0 msgr2=0x7fa6501a7470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:33.124 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:33.124 7fa64f2bf700 1 --2- 172.21.2.112:0/76841745 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa650084cb0 0x7fa6501a7470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:33.124 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:33.124 7fa64f2bf700 1 -- 172.21.2.112:0/76841745 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6501a80b0 con 0x7fa6500c2370 2021-02-10T11:40:33.124 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:33.125 7fa64f2bf700 1 --2- 172.21.2.112:0/76841745 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa6500c2370 0x7fa65018f870 secure :-1 s=READY pgs=2074 cs=0 l=1 rev1=1 rx=0x7fa64400b5c0 tx=0x7fa64400b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:33.124 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:33.125 7fa637fff700 1 -- 172.21.2.112:0/76841745 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa64400bd50 con 0x7fa6500c2370 2021-02-10T11:40:33.124 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:33.125 7fa637fff700 1 -- 172.21.2.112:0/76841745 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa644006030 con 0x7fa6500c2370 2021-02-10T11:40:33.124 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:33.125 7fa637fff700 1 -- 172.21.2.112:0/76841745 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa6440054d0 con 0x7fa6500c2370 2021-02-10T11:40:33.124 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:33.125 7fa6555a0700 1 -- 172.21.2.112:0/76841745 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6501a80b0 con 0x7fa6500c2370 2021-02-10T11:40:33.124 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:33.125 7fa6555a0700 1 -- 172.21.2.112:0/76841745 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6501a80b0 con 0x7fa6500c2370 2021-02-10T11:40:33.124 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:33.126 7fa637fff700 1 -- 172.21.2.112:0/76841745 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa644006210 con 0x7fa6500c2370 2021-02-10T11:40:33.124 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:33.126 7fa637fff700 1 --2- 172.21.2.112:0/76841745 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa6300186c0 0x7fa63001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:33.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:33.126 7fa637fff700 1 -- 172.21.2.112:0/76841745 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5107+0+0 (secure 0 0 0) 0x7fa6440323d0 con 0x7fa6500c2370 2021-02-10T11:40:33.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:33.126 7fa64eabe700 1 --2- 172.21.2.112:0/76841745 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa6300186c0 0x7fa63001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:33.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:33.126 7fa64eabe700 1 --2- 172.21.2.112:0/76841745 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa6300186c0 0x7fa63001abc0 crc :-1 s=READY pgs=960 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:33.153 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:33.157 7fa6555a0700 1 -- 172.21.2.112:0/76841745 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6500be840 con 0x7fa6500c2370 2021-02-10T11:40:33.155 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:33.160 7fa637fff700 1 -- 172.21.2.112:0/76841745 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa644006b80 con 0x7fa6500c2370 2021-02-10T11:40:33.284 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:33.288 7fa6555a0700 1 -- 172.21.2.112:0/76841745 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"} v 0) v1 -- 0x7fa6500be840 con 0x7fa6500c2370 2021-02-10T11:40:33.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:33.688 7fa637fff700 1 -- 172.21.2.112:0/76841745 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"}]=0 pool 'cephfs_data' created v106) v1 ==== 125+0+0 (secure 0 0 0) 0x7fa644006210 con 0x7fa6500c2370 2021-02-10T11:40:33.700 INFO:teuthology.orchestra.run.gibba012.stderr:pool 'cephfs_data' created 2021-02-10T11:40:33.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:33.706 7fa6555a0700 1 -- 172.21.2.112:0/76841745 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa6300186c0 msgr2=0x7fa63001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:33.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:33.706 7fa6555a0700 1 --2- 172.21.2.112:0/76841745 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa6300186c0 0x7fa63001abc0 crc :-1 s=READY pgs=960 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:33.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:33.706 7fa6555a0700 1 -- 172.21.2.112:0/76841745 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa6500c2370 msgr2=0x7fa65018f870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:33.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:33.706 7fa6555a0700 1 --2- 172.21.2.112:0/76841745 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa6500c2370 0x7fa65018f870 secure :-1 s=READY pgs=2074 cs=0 l=1 rev1=1 rx=0x7fa64400b5c0 tx=0x7fa64400b960).stop 2021-02-10T11:40:33.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:33.706 7fa6555a0700 1 -- 172.21.2.112:0/76841745 shutdown_connections 2021-02-10T11:40:33.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:33.706 7fa6555a0700 1 --2- 172.21.2.112:0/76841745 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa6300186c0 0x7fa63001abc0 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:33.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:33.706 7fa6555a0700 1 --2- 172.21.2.112:0/76841745 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa650084cb0 0x7fa6501a7470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:33.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:33.706 7fa6555a0700 1 --2- 172.21.2.112:0/76841745 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa6501a7a90 0x7fa650085700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:33.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:33.706 7fa6555a0700 1 --2- 172.21.2.112:0/76841745 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa6500c2370 0x7fa65018f870 unknown :-1 s=CLOSED pgs=2074 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:33.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:33.706 7fa6555a0700 1 -- 172.21.2.112:0/76841745 >> 172.21.2.112:0/76841745 conn(0x7fa6500b5dc0 msgr2=0x7fa6500c2d90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:33.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:33.706 7fa6555a0700 1 -- 172.21.2.112:0/76841745 shutdown_connections 2021-02-10T11:40:33.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:33.706 7fa6555a0700 1 -- 172.21.2.112:0/76841745 wait complete. 2021-02-10T11:40:33.710 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2021-02-10T11:40:34.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.129 7f2efaf86700 1 Processor -- start 2021-02-10T11:40:34.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.129 7f2efaf86700 1 -- start start 2021-02-10T11:40:34.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.130 7f2efaf86700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f2ef40d1880 0x7f2ef41b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:34.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.130 7f2efaf86700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2ef40cff80 0x7f2ef4109140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:34.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.130 7f2efaf86700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ef4109a70 con 0x7f2ef41b0af0 2021-02-10T11:40:34.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.130 7f2efaf86700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ef4109760 con 0x7f2ef40d1880 2021-02-10T11:40:34.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.130 7f2efaf86700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ef4088070 con 0x7f2ef40cff80 2021-02-10T11:40:34.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.130 7f2ef3fff700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f2ef41b0af0 0x7f2ef4108950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:54190/0 (socket says 172.21.2.112:54190) 2021-02-10T11:40:34.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.130 7f2ef3fff700 1 -- 172.21.2.112:0/2474153152 learned_addr learned my addr 172.21.2.112:0/2474153152 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:40:34.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.130 7f2ef37fe700 1 -- 172.21.2.112:0/2474153152 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2891248113 0 0) 0x7f2ee4001500 con 0x7f2ef41b0af0 2021-02-10T11:40:34.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.130 7f2ef37fe700 1 -- 172.21.2.112:0/2474153152 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ee0002fe0 con 0x7f2ef41b0af0 2021-02-10T11:40:34.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.130 7f2ef95a3700 1 --2- 172.21.2.112:0/2474153152 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2ef40cff80 0x7f2ef4109140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:34.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.130 7f2ef8da2700 1 --2- 172.21.2.112:0/2474153152 >> v2:172.21.2.120:3300/0 conn(0x7f2ef40d1880 0x7f2ef41b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:34.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.131 7f2ef37fe700 1 -- 172.21.2.112:0/2474153152 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1679322350 0 0) 0x7f2ee4001500 con 0x7f2ef41b0af0 2021-02-10T11:40:34.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.131 7f2ef37fe700 1 -- 172.21.2.112:0/2474153152 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2ef40cff80 msgr2=0x7f2ef4109140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:34.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.131 7f2ef37fe700 1 --2- 172.21.2.112:0/2474153152 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2ef40cff80 0x7f2ef4109140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:34.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.131 7f2ef37fe700 1 -- 172.21.2.112:0/2474153152 >> v2:172.21.2.120:3300/0 conn(0x7f2ef40d1880 msgr2=0x7f2ef41b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:34.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.131 7f2ef37fe700 1 --2- 172.21.2.112:0/2474153152 >> v2:172.21.2.120:3300/0 conn(0x7f2ef40d1880 0x7f2ef41b04d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:34.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.131 7f2ef37fe700 1 -- 172.21.2.112:0/2474153152 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ef40d1c80 con 0x7f2ef41b0af0 2021-02-10T11:40:34.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.131 7f2ef37fe700 1 -- 172.21.2.112:0/2474153152 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f2ee4001a00 con 0x7f2ef41b0af0 2021-02-10T11:40:34.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.131 7f2ef37fe700 1 -- 172.21.2.112:0/2474153152 >> v1:172.21.2.112:6789/0 conn(0x7f2ef41b0af0 legacy=0x7f2ef4108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:34.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.131 7f2ef37fe700 1 --2- 172.21.2.112:0/2474153152 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2ee0003a70 0x7f2ee0005ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:34.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.131 7f2ef37fe700 1 --2- 172.21.2.112:0/2474153152 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2ee004a790 0x7f2ee004cc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:34.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.131 7f2ef37fe700 1 --2- 172.21.2.112:0/2474153152 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2ee004d510 0x7f2ee004fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:34.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.131 7f2ef37fe700 1 -- 172.21.2.112:0/2474153152 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ee00502e0 con 0x7f2ee004d510 2021-02-10T11:40:34.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.131 7f2ef37fe700 1 -- 172.21.2.112:0/2474153152 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ee0050500 con 0x7f2ee004a790 2021-02-10T11:40:34.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.131 7f2ef37fe700 1 -- 172.21.2.112:0/2474153152 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ee0050720 con 0x7f2ee0003a70 2021-02-10T11:40:34.145 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.131 7f2ef95a3700 1 --2- 172.21.2.112:0/2474153152 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2ee004d510 0x7f2ee004fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:34.145 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.131 7f2ef8da2700 1 --2- 172.21.2.112:0/2474153152 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2ee0003a70 0x7f2ee0005ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:34.145 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.131 7f2ef3fff700 1 --2- 172.21.2.112:0/2474153152 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2ee004a790 0x7f2ee004cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:34.145 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.131 7f2ef95a3700 1 -- 172.21.2.112:0/2474153152 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2ee0003a70 msgr2=0x7f2ee0005ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:34.145 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.131 7f2ef95a3700 1 --2- 172.21.2.112:0/2474153152 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2ee0003a70 0x7f2ee0005ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:34.145 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.131 7f2ef95a3700 1 -- 172.21.2.112:0/2474153152 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2ee004a790 msgr2=0x7f2ee004cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:34.145 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.131 7f2ef95a3700 1 --2- 172.21.2.112:0/2474153152 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2ee004a790 0x7f2ee004cc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:34.145 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.131 7f2ef95a3700 1 -- 172.21.2.112:0/2474153152 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ee00509c0 con 0x7f2ee004d510 2021-02-10T11:40:34.145 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.131 7f2ef3fff700 1 --2- 172.21.2.112:0/2474153152 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2ee004a790 0x7f2ee004cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:40:34.145 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.131 7f2ef95a3700 1 --2- 172.21.2.112:0/2474153152 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2ee004d510 0x7f2ee004fa30 secure :-1 s=READY pgs=2076 cs=0 l=1 rev1=1 rx=0x7f2eec006b00 tx=0x7f2eec009b00).ready entity=mon.0 client_cookie=4b10047be4ee1370 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:34.145 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.132 7f2ef37fe700 1 -- 172.21.2.112:0/2474153152 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2eec000c10 con 0x7f2ee004d510 2021-02-10T11:40:34.145 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.132 7f2ef37fe700 1 -- 172.21.2.112:0/2474153152 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2eec00c0c0 con 0x7f2ee004d510 2021-02-10T11:40:34.145 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.132 7f2ef37fe700 1 -- 172.21.2.112:0/2474153152 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2eec00c5a0 con 0x7f2ee004d510 2021-02-10T11:40:34.145 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.132 7f2efaf86700 1 -- 172.21.2.112:0/2474153152 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2ee004d510 msgr2=0x7f2ee004fa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:34.146 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.132 7f2efaf86700 1 --2- 172.21.2.112:0/2474153152 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2ee004d510 0x7f2ee004fa30 secure :-1 s=READY pgs=2076 cs=0 l=1 rev1=1 rx=0x7f2eec006b00 tx=0x7f2eec009b00).stop 2021-02-10T11:40:34.146 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.132 7f2ef95a3700 1 -- 172.21.2.112:0/2474153152 reap_dead start 2021-02-10T11:40:34.146 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.132 7f2efaf86700 1 -- 172.21.2.112:0/2474153152 shutdown_connections 2021-02-10T11:40:34.146 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.132 7f2efaf86700 1 -- 172.21.2.112:0/2474153152 >> 172.21.2.112:0/2474153152 conn(0x7f2ef40ce430 msgr2=0x7f2ef40d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:34.146 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.132 7f2efaf86700 1 -- 172.21.2.112:0/2474153152 shutdown_connections 2021-02-10T11:40:34.146 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.132 7f2efaf86700 1 -- 172.21.2.112:0/2474153152 wait complete. 2021-02-10T11:40:34.146 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.147 7f2efaf86700 1 Processor -- start 2021-02-10T11:40:34.146 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.147 7f2efaf86700 1 -- start start 2021-02-10T11:40:34.150 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.148 7f2efaf86700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2ef40c3350 0x7f2ef40c3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:34.150 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.148 7f2efaf86700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2ef409f170 0x7f2ef409f570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:34.150 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.148 7f2efaf86700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2ef409fb90 0x7f2ef4083980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:34.150 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.148 7f2efaf86700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ef40b85c0 con 0x7f2ef409f170 2021-02-10T11:40:34.150 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.148 7f2efaf86700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ef409ffd0 con 0x7f2ef40c3350 2021-02-10T11:40:34.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.148 7f2efaf86700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ef4083fa0 con 0x7f2ef409fb90 2021-02-10T11:40:34.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.148 7f2ef3fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2ef409f170 0x7f2ef409f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:34.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.148 7f2ef3fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2ef409f170 0x7f2ef409f570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:54742/0 (socket says 172.21.2.112:54742) 2021-02-10T11:40:34.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.148 7f2ef3fff700 1 -- 172.21.2.112:0/2765056494 learned_addr learned my addr 172.21.2.112:0/2765056494 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:40:34.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.148 7f2ef3fff700 1 -- 172.21.2.112:0/2765056494 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2ef409fb90 msgr2=0x7f2ef4083980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:34.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.148 7f2ef3fff700 1 --2- 172.21.2.112:0/2765056494 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2ef409fb90 0x7f2ef4083980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:34.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.148 7f2ef3fff700 1 -- 172.21.2.112:0/2765056494 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2ef40c3350 msgr2=0x7f2ef40c3750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:34.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.148 7f2ef3fff700 1 --2- 172.21.2.112:0/2765056494 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2ef40c3350 0x7f2ef40c3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:34.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.148 7f2ef3fff700 1 -- 172.21.2.112:0/2765056494 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ef40841c0 con 0x7f2ef409f170 2021-02-10T11:40:34.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.148 7f2ef3fff700 1 --2- 172.21.2.112:0/2765056494 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2ef409f170 0x7f2ef409f570 secure :-1 s=READY pgs=2077 cs=0 l=1 rev1=1 rx=0x7f2ee400a940 tx=0x7f2ee400ace0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:34.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.149 7f2ef17fa700 1 -- 172.21.2.112:0/2765056494 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2ee40011b0 con 0x7f2ef409f170 2021-02-10T11:40:34.152 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.149 7f2ef17fa700 1 -- 172.21.2.112:0/2765056494 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ee4004930 con 0x7f2ef409f170 2021-02-10T11:40:34.152 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.149 7f2ef17fa700 1 -- 172.21.2.112:0/2765056494 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2ee4007020 con 0x7f2ef409f170 2021-02-10T11:40:34.153 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.149 7f2efaf86700 1 -- 172.21.2.112:0/2765056494 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ef40841c0 con 0x7f2ef409f170 2021-02-10T11:40:34.153 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.149 7f2efaf86700 1 -- 172.21.2.112:0/2765056494 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ef4084750 con 0x7f2ef409f170 2021-02-10T11:40:34.153 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.149 7f2ef17fa700 1 -- 172.21.2.112:0/2765056494 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2ee400ae20 con 0x7f2ef409f170 2021-02-10T11:40:34.153 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.149 7f2ef17fa700 1 --2- 172.21.2.112:0/2765056494 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2ed80186c0 0x7f2ed801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:34.154 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.149 7f2ef17fa700 1 -- 172.21.2.112:0/2765056494 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5457+0+0 (secure 0 0 0) 0x7f2ee4032890 con 0x7f2ef409f170 2021-02-10T11:40:34.154 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.150 7f2ef8da2700 1 --2- 172.21.2.112:0/2765056494 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2ed80186c0 0x7f2ed801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:34.154 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.150 7f2ef8da2700 1 --2- 172.21.2.112:0/2765056494 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2ed80186c0 0x7f2ed801abc0 crc :-1 s=READY pgs=963 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:34.176 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.180 7f2efaf86700 1 -- 172.21.2.112:0/2765056494 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ef4082a70 con 0x7f2ef409f170 2021-02-10T11:40:34.178 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.183 7f2ef17fa700 1 -- 172.21.2.112:0/2765056494 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2ee40011b0 con 0x7f2ef409f170 2021-02-10T11:40:34.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.313 7f2efaf86700 1 -- 172.21.2.112:0/2765056494 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"} v 0) v1 -- 0x7f2ef4082a70 con 0x7f2ef409f170 2021-02-10T11:40:34.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.317 7f2ef17fa700 1 -- 172.21.2.112:0/2765056494 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"}]=0 new fs with metadata pool 22 and data pool 23 v85) v1 ==== 189+0+0 (secure 0 0 0) 0x7f2ee400ae20 con 0x7f2ef409f170 2021-02-10T11:40:34.318 INFO:teuthology.orchestra.run.gibba012.stderr:new fs with metadata pool 22 and data pool 23 2021-02-10T11:40:34.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.322 7f2efaf86700 1 -- 172.21.2.112:0/2765056494 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2ed80186c0 msgr2=0x7f2ed801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:34.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.322 7f2efaf86700 1 --2- 172.21.2.112:0/2765056494 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2ed80186c0 0x7f2ed801abc0 crc :-1 s=READY pgs=963 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:34.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.322 7f2efaf86700 1 -- 172.21.2.112:0/2765056494 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2ef409f170 msgr2=0x7f2ef409f570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:34.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.322 7f2efaf86700 1 --2- 172.21.2.112:0/2765056494 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2ef409f170 0x7f2ef409f570 secure :-1 s=READY pgs=2077 cs=0 l=1 rev1=1 rx=0x7f2ee400a940 tx=0x7f2ee400ace0).stop 2021-02-10T11:40:34.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.323 7f2efaf86700 1 -- 172.21.2.112:0/2765056494 shutdown_connections 2021-02-10T11:40:34.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.323 7f2efaf86700 1 --2- 172.21.2.112:0/2765056494 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2ed80186c0 0x7f2ed801abc0 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:34.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.323 7f2efaf86700 1 --2- 172.21.2.112:0/2765056494 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2ef409fb90 0x7f2ef4083980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:34.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.323 7f2efaf86700 1 --2- 172.21.2.112:0/2765056494 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2ef409f170 0x7f2ef409f570 unknown :-1 s=CLOSED pgs=2077 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:34.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.323 7f2efaf86700 1 --2- 172.21.2.112:0/2765056494 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2ef40c3350 0x7f2ef40c3750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:34.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.323 7f2efaf86700 1 -- 172.21.2.112:0/2765056494 >> 172.21.2.112:0/2765056494 conn(0x7f2ef409ebd0 msgr2=0x7f2ef4082ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:34.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.323 7f2efaf86700 1 -- 172.21.2.112:0/2765056494 shutdown_connections 2021-02-10T11:40:34.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.323 7f2efaf86700 1 -- 172.21.2.112:0/2765056494 wait complete. 2021-02-10T11:40:34.326 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:40:34.754 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.741 7f742859a700 1 Processor -- start 2021-02-10T11:40:34.754 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.741 7f742859a700 1 -- start start 2021-02-10T11:40:34.754 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.742 7f742859a700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f74201b0470 0x7f74201b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:34.754 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.742 7f742859a700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74200f0b50 con 0x7f74200f20e0 2021-02-10T11:40:34.754 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.742 7f742859a700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74200f24e0 con 0x7f74200ef610 2021-02-10T11:40:34.754 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.742 7f742859a700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74200e5ea0 con 0x7f74201b0470 2021-02-10T11:40:34.754 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.742 7f7426bb7700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f74200f20e0 0x7f74200f0200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:54210/0 (socket says 172.21.2.112:54210) 2021-02-10T11:40:34.754 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.742 7f7426bb7700 1 -- 172.21.2.112:0/2441564490 learned_addr learned my addr 172.21.2.112:0/2441564490 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:40:34.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.742 7f74263b6700 1 --2- 172.21.2.112:0/2441564490 >> v2:172.21.2.120:3300/0 conn(0x7f74201b0470 0x7f74201b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:34.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.742 7f74253b4700 1 -- 172.21.2.112:0/2441564490 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3864432240 0 0) 0x7f7418001500 con 0x7f74200f20e0 2021-02-10T11:40:34.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.742 7f74253b4700 1 -- 172.21.2.112:0/2441564490 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f740c002fe0 con 0x7f74200f20e0 2021-02-10T11:40:34.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.742 7f74253b4700 1 -- 172.21.2.112:0/2441564490 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 243075168 0 0) 0x7f741c000ff0 con 0x7f74200ef610 2021-02-10T11:40:34.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.742 7f74253b4700 1 -- 172.21.2.112:0/2441564490 --> v1:172.21.2.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f740c003210 con 0x7f74200ef610 2021-02-10T11:40:34.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.743 7f74263b6700 1 -- 172.21.2.112:0/2441564490 >> v1:172.21.2.120:6789/0 conn(0x7f74200ef610 legacy=0x7f74200efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:34.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.743 7f74263b6700 1 -- 172.21.2.112:0/2441564490 >> v1:172.21.2.112:6789/0 conn(0x7f74200f20e0 legacy=0x7f74200f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:34.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.743 7f74263b6700 1 -- 172.21.2.112:0/2441564490 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74200e52e0 con 0x7f74201b0470 2021-02-10T11:40:34.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.743 7f74263b6700 1 --2- 172.21.2.112:0/2441564490 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f74201b0470 0x7f74201b0870 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 rx=0x7f7414006650 tx=0x7f7414009a80).ready entity=mon.1 client_cookie=eecd577c1b2384ee server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:34.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.743 7f74253b4700 1 -- 172.21.2.112:0/2441564490 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f741400c0c0 con 0x7f74201b0470 2021-02-10T11:40:34.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.743 7f74253b4700 1 -- 172.21.2.112:0/2441564490 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f741400c340 con 0x7f74201b0470 2021-02-10T11:40:34.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.743 7f74253b4700 1 -- 172.21.2.112:0/2441564490 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f741400c860 con 0x7f74201b0470 2021-02-10T11:40:34.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.743 7f742859a700 1 -- 172.21.2.112:0/2441564490 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f74201b0470 msgr2=0x7f74201b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:34.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.743 7f742859a700 1 --2- 172.21.2.112:0/2441564490 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f74201b0470 0x7f74201b0870 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 rx=0x7f7414006650 tx=0x7f7414009a80).stop 2021-02-10T11:40:34.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.744 7f742859a700 1 -- 172.21.2.112:0/2441564490 shutdown_connections 2021-02-10T11:40:34.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.744 7f742859a700 1 --2- 172.21.2.112:0/2441564490 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f74201b0470 0x7f74201b0870 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:34.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.744 7f742859a700 1 -- 172.21.2.112:0/2441564490 >> 172.21.2.112:0/2441564490 conn(0x7f74200ce3e0 msgr2=0x7f74200e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:34.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.744 7f742859a700 1 -- 172.21.2.112:0/2441564490 shutdown_connections 2021-02-10T11:40:34.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.744 7f742859a700 1 -- 172.21.2.112:0/2441564490 wait complete. 2021-02-10T11:40:34.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.759 7f742859a700 1 Processor -- start 2021-02-10T11:40:34.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.759 7f742859a700 1 -- start start 2021-02-10T11:40:34.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.759 7f742859a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f742008e110 0x7f74200b1ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:34.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.759 7f742859a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f74200b24f0 0x7f742008f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:34.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.759 7f742859a700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f742008fe00 0x7f742008c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:34.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.759 7f742859a700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f742008db90 con 0x7f742008e110 2021-02-10T11:40:34.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.759 7f742859a700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74201a83f0 con 0x7f742008fe00 2021-02-10T11:40:34.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.759 7f742859a700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74201a85d0 con 0x7f74200b24f0 2021-02-10T11:40:34.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.759 7f74263b6700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f742008e110 0x7f74200b1ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:34.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.759 7f74263b6700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f742008e110 0x7f74200b1ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:54754/0 (socket says 172.21.2.112:54754) 2021-02-10T11:40:34.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.759 7f74263b6700 1 -- 172.21.2.112:0/1272963989 learned_addr learned my addr 172.21.2.112:0/1272963989 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:40:34.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.760 7f7425bb5700 1 --2- 172.21.2.112:0/1272963989 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f74200b24f0 0x7f742008f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:34.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.760 7f7426bb7700 1 --2- 172.21.2.112:0/1272963989 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f742008fe00 0x7f742008c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:34.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.760 7f74263b6700 1 -- 172.21.2.112:0/1272963989 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f74200b24f0 msgr2=0x7f742008f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:34.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.760 7f74263b6700 1 --2- 172.21.2.112:0/1272963989 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f74200b24f0 0x7f742008f7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:34.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.760 7f74263b6700 1 -- 172.21.2.112:0/1272963989 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f742008fe00 msgr2=0x7f742008c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:34.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.760 7f74263b6700 1 --2- 172.21.2.112:0/1272963989 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f742008fe00 0x7f742008c690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:34.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.760 7f74263b6700 1 -- 172.21.2.112:0/1272963989 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74201a8890 con 0x7f742008e110 2021-02-10T11:40:34.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.760 7f7426bb7700 1 --2- 172.21.2.112:0/1272963989 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f742008fe00 0x7f742008c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:40:34.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.760 7f74263b6700 1 --2- 172.21.2.112:0/1272963989 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f742008e110 0x7f74200b1ed0 secure :-1 s=READY pgs=2079 cs=0 l=1 rev1=1 rx=0x7f7414009630 tx=0x7f741400b900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:34.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.760 7f7412ffd700 1 -- 172.21.2.112:0/1272963989 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f74140093a0 con 0x7f742008e110 2021-02-10T11:40:34.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.760 7f7412ffd700 1 -- 172.21.2.112:0/1272963989 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f741400bcf0 con 0x7f742008e110 2021-02-10T11:40:34.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.760 7f742859a700 1 -- 172.21.2.112:0/1272963989 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74201a8890 con 0x7f742008e110 2021-02-10T11:40:34.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.760 7f7412ffd700 1 -- 172.21.2.112:0/1272963989 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7414005560 con 0x7f742008e110 2021-02-10T11:40:34.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.761 7f742859a700 1 -- 172.21.2.112:0/1272963989 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74201a8890 con 0x7f742008e110 2021-02-10T11:40:34.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.761 7f7412ffd700 1 -- 172.21.2.112:0/1272963989 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7414005740 con 0x7f742008e110 2021-02-10T11:40:34.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.761 7f7412ffd700 1 --2- 172.21.2.112:0/1272963989 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f74080186c0 0x7f740801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:34.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.761 7f7412ffd700 1 -- 172.21.2.112:0/1272963989 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f74140325a0 con 0x7f742008e110 2021-02-10T11:40:34.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.762 7f7425bb5700 1 --2- 172.21.2.112:0/1272963989 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f74080186c0 0x7f740801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:34.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.762 7f7425bb5700 1 --2- 172.21.2.112:0/1272963989 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f74080186c0 0x7f740801abc0 crc :-1 s=READY pgs=964 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:34.787 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.792 7f742859a700 1 -- 172.21.2.112:0/1272963989 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7420105ab0 con 0x7f742008e110 2021-02-10T11:40:34.790 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.794 7f7412ffd700 1 -- 172.21.2.112:0/1272963989 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7414005530 con 0x7f742008e110 2021-02-10T11:40:34.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.920 7f742859a700 1 -- 172.21.2.112:0/1272963989 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7420105ab0 con 0x7f742008e110 2021-02-10T11:40:34.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.921 7f7412ffd700 1 -- 172.21.2.112:0/1272963989 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v107) v1 ==== 74+0+14019 (secure 0 0 0) 0x7f7414005740 con 0x7f742008e110 2021-02-10T11:40:34.917 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:40:34.917 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":107,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:40:34.314747","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:40:32.284968","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":23,"pool_name":"cephfs_data","create_time":"2021-02-10 11:40:33.307261","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:40:34.918 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.923 7f742859a700 1 -- 172.21.2.112:0/1272963989 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f74080186c0 msgr2=0x7f740801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:34.918 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.923 7f742859a700 1 --2- 172.21.2.112:0/1272963989 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f74080186c0 0x7f740801abc0 crc :-1 s=READY pgs=964 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:34.918 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.923 7f742859a700 1 -- 172.21.2.112:0/1272963989 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f742008e110 msgr2=0x7f74200b1ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:34.918 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.923 7f742859a700 1 --2- 172.21.2.112:0/1272963989 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f742008e110 0x7f74200b1ed0 secure :-1 s=READY pgs=2079 cs=0 l=1 rev1=1 rx=0x7f7414009630 tx=0x7f741400b900).stop 2021-02-10T11:40:34.919 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.923 7f742859a700 1 -- 172.21.2.112:0/1272963989 shutdown_connections 2021-02-10T11:40:34.919 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.923 7f742859a700 1 --2- 172.21.2.112:0/1272963989 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f74080186c0 0x7f740801abc0 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:34.919 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.923 7f742859a700 1 --2- 172.21.2.112:0/1272963989 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f742008fe00 0x7f742008c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:34.919 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.923 7f742859a700 1 --2- 172.21.2.112:0/1272963989 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f74200b24f0 0x7f742008f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:34.919 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.923 7f742859a700 1 --2- 172.21.2.112:0/1272963989 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f742008e110 0x7f74200b1ed0 unknown :-1 s=CLOSED pgs=2079 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:34.919 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.923 7f742859a700 1 -- 172.21.2.112:0/1272963989 >> 172.21.2.112:0/1272963989 conn(0x7f742019f7c0 msgr2=0x7f74201b6b60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:34.919 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.923 7f742859a700 1 -- 172.21.2.112:0/1272963989 shutdown_connections 2021-02-10T11:40:34.919 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:34.923 7f742859a700 1 -- 172.21.2.112:0/1272963989 wait complete. 2021-02-10T11:40:34.927 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:40:35.356 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.344 7f56d1fdf700 1 Processor -- start 2021-02-10T11:40:35.356 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.344 7f56d1fdf700 1 -- start start 2021-02-10T11:40:35.356 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.345 7f56d1fdf700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f56cc0d0340 0x7f56cc0ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:35.356 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.345 7f56d1fdf700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f56cc0d1c70 0x7f56cc0efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:35.356 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.345 7f56d1fdf700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56cc0f0b10 con 0x7f56cc0d1870 2021-02-10T11:40:35.357 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.345 7f56d1fdf700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56cc0e5ed0 con 0x7f56cc0d1c70 2021-02-10T11:40:35.357 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.345 7f56d1fdf700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56cc1b0be0 con 0x7f56cc0d0340 2021-02-10T11:40:35.357 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.345 7f56cb7fe700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7f56cc0d1870 0x7f56cc1b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:47780/0 (socket says 172.21.2.112:47780) 2021-02-10T11:40:35.357 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.345 7f56cb7fe700 1 -- 172.21.2.112:0/417683524 learned_addr learned my addr 172.21.2.112:0/417683524 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:40:35.357 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.345 7f56cbfff700 1 --2- 172.21.2.112:0/417683524 >> v2:172.21.2.120:3300/0 conn(0x7f56cc0d1c70 0x7f56cc0efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:35.357 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.345 7f56caffd700 1 --2- 172.21.2.112:0/417683524 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f56cc0d0340 0x7f56cc0ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:35.357 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.345 7f56cbfff700 1 -- 172.21.2.112:0/417683524 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f56cc0d0340 msgr2=0x7f56cc0ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:35.357 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.345 7f56cbfff700 1 --2- 172.21.2.112:0/417683524 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f56cc0d0340 0x7f56cc0ef640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:35.357 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.345 7f56ca7fc700 1 -- 172.21.2.112:0/417683524 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1117728626 0 0) 0x7f56c4001440 con 0x7f56cc0d1870 2021-02-10T11:40:35.357 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.345 7f56cbfff700 1 -- 172.21.2.112:0/417683524 >> v1:172.21.2.120:6789/0 conn(0x7f56cc0d1870 legacy=0x7f56cc1b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:35.357 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.345 7f56cbfff700 1 -- 172.21.2.112:0/417683524 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56cc0d2fe0 con 0x7f56cc0d1c70 2021-02-10T11:40:35.357 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.346 7f56cbfff700 1 --2- 172.21.2.112:0/417683524 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f56cc0d1c70 0x7f56cc0efc60 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 rx=0x7f56bc006ad0 tx=0x7f56bc009ad0).ready entity=mon.1 client_cookie=184efcb5f2268932 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:35.357 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.346 7f56ca7fc700 1 -- 172.21.2.112:0/417683524 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f56bc0022e0 con 0x7f56cc0d1c70 2021-02-10T11:40:35.358 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.346 7f56ca7fc700 1 -- 172.21.2.112:0/417683524 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56bc006410 con 0x7f56cc0d1c70 2021-02-10T11:40:35.358 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.346 7f56ca7fc700 1 -- 172.21.2.112:0/417683524 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f56bc000c10 con 0x7f56cc0d1c70 2021-02-10T11:40:35.358 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.346 7f56d1fdf700 1 -- 172.21.2.112:0/417683524 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f56cc0d1c70 msgr2=0x7f56cc0efc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:35.358 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.346 7f56d1fdf700 1 --2- 172.21.2.112:0/417683524 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f56cc0d1c70 0x7f56cc0efc60 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 rx=0x7f56bc006ad0 tx=0x7f56bc009ad0).stop 2021-02-10T11:40:35.358 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.346 7f56d1fdf700 1 -- 172.21.2.112:0/417683524 shutdown_connections 2021-02-10T11:40:35.358 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.346 7f56d1fdf700 1 --2- 172.21.2.112:0/417683524 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f56cc0d1c70 0x7f56cc0efc60 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:35.358 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.346 7f56d1fdf700 1 --2- 172.21.2.112:0/417683524 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f56cc0d0340 0x7f56cc0ef640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:35.358 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.346 7f56d1fdf700 1 -- 172.21.2.112:0/417683524 >> 172.21.2.112:0/417683524 conn(0x7f56cc0ce450 msgr2=0x7f56cc0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:35.358 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.346 7f56d1fdf700 1 -- 172.21.2.112:0/417683524 shutdown_connections 2021-02-10T11:40:35.358 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.346 7f56d1fdf700 1 -- 172.21.2.112:0/417683524 wait complete. 2021-02-10T11:40:35.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.361 7f56d1fdf700 1 Processor -- start 2021-02-10T11:40:35.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.361 7f56d1fdf700 1 -- start start 2021-02-10T11:40:35.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.362 7f56d1fdf700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f56cc0ada90 0x7f56cc0ade90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:35.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.362 7f56d1fdf700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f56cc099d20 0x7f56cc09a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:35.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.362 7f56d1fdf700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f56cc09a740 0x7f56cc097bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:35.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.362 7f56d1fdf700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56cc0997a0 con 0x7f56cc099d20 2021-02-10T11:40:35.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.362 7f56d1fdf700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56cc18f0c0 con 0x7f56cc09a740 2021-02-10T11:40:35.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.362 7f56d1fdf700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56cc09ab40 con 0x7f56cc0ada90 2021-02-10T11:40:35.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.362 7f56caffd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f56cc099d20 0x7f56cc09a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:35.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.362 7f56caffd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f56cc099d20 0x7f56cc09a120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:54770/0 (socket says 172.21.2.112:54770) 2021-02-10T11:40:35.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.362 7f56caffd700 1 -- 172.21.2.112:0/4234776714 learned_addr learned my addr 172.21.2.112:0/4234776714 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:40:35.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.362 7f56caffd700 1 -- 172.21.2.112:0/4234776714 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f56cc0ada90 msgr2=0x7f56cc0ade90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:35.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.362 7f56caffd700 1 --2- 172.21.2.112:0/4234776714 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f56cc0ada90 0x7f56cc0ade90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:35.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.362 7f56caffd700 1 -- 172.21.2.112:0/4234776714 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f56cc09a740 msgr2=0x7f56cc097bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:35.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.362 7f56caffd700 1 --2- 172.21.2.112:0/4234776714 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f56cc09a740 0x7f56cc097bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:35.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.362 7f56caffd700 1 -- 172.21.2.112:0/4234776714 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56cc09ad60 con 0x7f56cc099d20 2021-02-10T11:40:35.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.362 7f56caffd700 1 --2- 172.21.2.112:0/4234776714 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f56cc099d20 0x7f56cc09a120 secure :-1 s=READY pgs=2080 cs=0 l=1 rev1=1 rx=0x7f56c0006f60 tx=0x7f56c000b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:35.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.362 7f56b3fff700 1 -- 172.21.2.112:0/4234776714 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f56c00009b0 con 0x7f56cc099d20 2021-02-10T11:40:35.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.362 7f56b3fff700 1 -- 172.21.2.112:0/4234776714 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56c000bdf0 con 0x7f56cc099d20 2021-02-10T11:40:35.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.362 7f56d1fdf700 1 -- 172.21.2.112:0/4234776714 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56cc09ad60 con 0x7f56cc099d20 2021-02-10T11:40:35.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.362 7f56b3fff700 1 -- 172.21.2.112:0/4234776714 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f56c0002030 con 0x7f56cc099d20 2021-02-10T11:40:35.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.362 7f56d1fdf700 1 -- 172.21.2.112:0/4234776714 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56cc0984d0 con 0x7f56cc099d20 2021-02-10T11:40:35.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.363 7f56b3fff700 1 -- 172.21.2.112:0/4234776714 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f56c0000c90 con 0x7f56cc099d20 2021-02-10T11:40:35.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.363 7f56b3fff700 1 --2- 172.21.2.112:0/4234776714 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f56ac0186d0 0x7f56ac01abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:35.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.363 7f56b3fff700 1 -- 172.21.2.112:0/4234776714 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f56c00327f0 con 0x7f56cc099d20 2021-02-10T11:40:35.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.363 7f56cb7fe700 1 --2- 172.21.2.112:0/4234776714 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f56ac0186d0 0x7f56ac01abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:35.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.364 7f56cb7fe700 1 --2- 172.21.2.112:0/4234776714 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f56ac0186d0 0x7f56ac01abd0 crc :-1 s=READY pgs=965 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:35.390 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.394 7f56d1fdf700 1 -- 172.21.2.112:0/4234776714 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56cc0ebc00 con 0x7f56cc099d20 2021-02-10T11:40:35.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.397 7f56b3fff700 1 -- 172.21.2.112:0/4234776714 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f56c0000940 con 0x7f56cc099d20 2021-02-10T11:40:35.519 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.524 7f56d1fdf700 1 -- 172.21.2.112:0/4234776714 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f56cc0f0b10 con 0x7f56cc099d20 2021-02-10T11:40:35.520 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.524 7f56b3fff700 1 -- 172.21.2.112:0/4234776714 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v108) v1 ==== 74+0+14006 (secure 0 0 0) 0x7f56c0000c90 con 0x7f56cc099d20 2021-02-10T11:40:35.520 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:40:35.521 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":108,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:40:35.316267","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:40:32.284968","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":23,"pool_name":"cephfs_data","create_time":"2021-02-10 11:40:33.307261","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:40:35.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.526 7f56d1fdf700 1 -- 172.21.2.112:0/4234776714 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f56ac0186d0 msgr2=0x7f56ac01abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:35.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.526 7f56d1fdf700 1 --2- 172.21.2.112:0/4234776714 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f56ac0186d0 0x7f56ac01abd0 crc :-1 s=READY pgs=965 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:35.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.526 7f56d1fdf700 1 -- 172.21.2.112:0/4234776714 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f56cc099d20 msgr2=0x7f56cc09a120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:35.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.526 7f56d1fdf700 1 --2- 172.21.2.112:0/4234776714 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f56cc099d20 0x7f56cc09a120 secure :-1 s=READY pgs=2080 cs=0 l=1 rev1=1 rx=0x7f56c0006f60 tx=0x7f56c000b970).stop 2021-02-10T11:40:35.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.526 7f56d1fdf700 1 -- 172.21.2.112:0/4234776714 shutdown_connections 2021-02-10T11:40:35.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.526 7f56d1fdf700 1 --2- 172.21.2.112:0/4234776714 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f56ac0186d0 0x7f56ac01abd0 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:35.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.526 7f56d1fdf700 1 --2- 172.21.2.112:0/4234776714 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f56cc09a740 0x7f56cc097bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:35.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.526 7f56d1fdf700 1 --2- 172.21.2.112:0/4234776714 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f56cc099d20 0x7f56cc09a120 unknown :-1 s=CLOSED pgs=2080 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:35.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.526 7f56d1fdf700 1 --2- 172.21.2.112:0/4234776714 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f56cc0ada90 0x7f56cc0ade90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:35.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.526 7f56d1fdf700 1 -- 172.21.2.112:0/4234776714 >> 172.21.2.112:0/4234776714 conn(0x7f56cc0f0550 msgr2=0x7f56cc191730 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:35.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.526 7f56d1fdf700 1 -- 172.21.2.112:0/4234776714 shutdown_connections 2021-02-10T11:40:35.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.526 7f56d1fdf700 1 -- 172.21.2.112:0/4234776714 wait complete. 2021-02-10T11:40:35.530 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2021-02-10T11:40:35.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.953 7f75d801b700 1 Processor -- start 2021-02-10T11:40:35.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.953 7f75d801b700 1 -- start start 2021-02-10T11:40:35.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.953 7f75d801b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f75d010aff0 0x7f75d00cf750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:35.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.953 7f75d801b700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75d00fe1c0 con 0x7f75d00d1870 2021-02-10T11:40:35.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.953 7f75d801b700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75d01b0cb0 con 0x7f75d00d1c70 2021-02-10T11:40:35.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.953 7f75d801b700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75d0109a60 con 0x7f75d010aff0 2021-02-10T11:40:35.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.953 7f75d5e37700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f75d00d1870 0x7f75d01b04c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:54236/0 (socket says 172.21.2.112:54236) 2021-02-10T11:40:35.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.953 7f75d5e37700 1 -- 172.21.2.112:0/2527482057 learned_addr learned my addr 172.21.2.112:0/2527482057 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:40:35.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.954 7f75d6638700 1 --2- 172.21.2.112:0/2527482057 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f75d010aff0 0x7f75d00cf750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:35.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.954 7f75d4e35700 1 -- 172.21.2.112:0/2527482057 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 613505628 0 0) 0x7f75cc001180 con 0x7f75d00d1870 2021-02-10T11:40:35.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.954 7f75d4e35700 1 -- 172.21.2.112:0/2527482057 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75bc002fe0 con 0x7f75d00d1870 2021-02-10T11:40:35.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.954 7f75d4e35700 1 -- 172.21.2.112:0/2527482057 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 285859534 0 0) 0x7f75cc001180 con 0x7f75d00d1870 2021-02-10T11:40:35.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.954 7f75d4e35700 1 -- 172.21.2.112:0/2527482057 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f75d010aff0 msgr2=0x7f75d00cf750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:35.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.954 7f75d4e35700 1 --2- 172.21.2.112:0/2527482057 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f75d010aff0 0x7f75d00cf750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:35.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.954 7f75d4e35700 1 -- 172.21.2.112:0/2527482057 >> v1:172.21.2.120:6789/0 conn(0x7f75d00d1c70 legacy=0x7f75d00cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:35.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.954 7f75d4e35700 1 -- 172.21.2.112:0/2527482057 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75d010b3f0 con 0x7f75d00d1870 2021-02-10T11:40:35.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.954 7f75d4e35700 1 -- 172.21.2.112:0/2527482057 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f75cc001680 con 0x7f75d00d1870 2021-02-10T11:40:35.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.954 7f75d4e35700 1 -- 172.21.2.112:0/2527482057 >> v1:172.21.2.112:6789/0 conn(0x7f75d00d1870 legacy=0x7f75d01b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:35.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.955 7f75d4e35700 1 --2- 172.21.2.112:0/2527482057 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f75bc003a60 0x7f75bc005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:35.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.955 7f75d4e35700 1 --2- 172.21.2.112:0/2527482057 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f75bc006790 0x7f75bc008c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:35.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.955 7f75d4e35700 1 --2- 172.21.2.112:0/2527482057 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f75bc009500 0x7f75bc00ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:35.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.955 7f75d4e35700 1 -- 172.21.2.112:0/2527482057 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75bc00c2d0 con 0x7f75bc009500 2021-02-10T11:40:35.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.955 7f75d4e35700 1 -- 172.21.2.112:0/2527482057 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75bc00c4f0 con 0x7f75bc003a60 2021-02-10T11:40:35.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.955 7f75d4e35700 1 -- 172.21.2.112:0/2527482057 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75bc00c710 con 0x7f75bc006790 2021-02-10T11:40:35.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.955 7f75d6638700 1 --2- 172.21.2.112:0/2527482057 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f75bc009500 0x7f75bc00ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:35.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.955 7f75d5e37700 1 --2- 172.21.2.112:0/2527482057 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f75bc003a60 0x7f75bc005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:35.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.955 7f75d5636700 1 --2- 172.21.2.112:0/2527482057 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f75bc006790 0x7f75bc008c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:35.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.955 7f75d6638700 1 -- 172.21.2.112:0/2527482057 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f75bc006790 msgr2=0x7f75bc008c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:35.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.955 7f75d6638700 1 --2- 172.21.2.112:0/2527482057 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f75bc006790 0x7f75bc008c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:35.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.955 7f75d6638700 1 -- 172.21.2.112:0/2527482057 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f75bc003a60 msgr2=0x7f75bc005ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:35.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.955 7f75d6638700 1 --2- 172.21.2.112:0/2527482057 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f75bc003a60 0x7f75bc005ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:35.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.955 7f75d6638700 1 -- 172.21.2.112:0/2527482057 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75bc050980 con 0x7f75bc009500 2021-02-10T11:40:35.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.955 7f75d6638700 1 --2- 172.21.2.112:0/2527482057 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f75bc009500 0x7f75bc00ba20 secure :-1 s=READY pgs=2082 cs=0 l=1 rev1=1 rx=0x7f75c8009b10 tx=0x7f75c8009eb0).ready entity=mon.0 client_cookie=48543175fda6fee6 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:35.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.955 7f75d4e35700 1 -- 172.21.2.112:0/2527482057 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f75c800b510 con 0x7f75bc009500 2021-02-10T11:40:35.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.955 7f75d4e35700 1 -- 172.21.2.112:0/2527482057 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75c8003030 con 0x7f75bc009500 2021-02-10T11:40:35.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.955 7f75d4e35700 1 -- 172.21.2.112:0/2527482057 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f75c800b790 con 0x7f75bc009500 2021-02-10T11:40:35.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.955 7f75d801b700 1 -- 172.21.2.112:0/2527482057 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f75bc009500 msgr2=0x7f75bc00ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:35.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.955 7f75d801b700 1 --2- 172.21.2.112:0/2527482057 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f75bc009500 0x7f75bc00ba20 secure :-1 s=READY pgs=2082 cs=0 l=1 rev1=1 rx=0x7f75c8009b10 tx=0x7f75c8009eb0).stop 2021-02-10T11:40:35.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.955 7f75d6638700 1 -- 172.21.2.112:0/2527482057 reap_dead start 2021-02-10T11:40:35.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.956 7f75d801b700 1 -- 172.21.2.112:0/2527482057 shutdown_connections 2021-02-10T11:40:35.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.956 7f75d801b700 1 -- 172.21.2.112:0/2527482057 >> 172.21.2.112:0/2527482057 conn(0x7f75d00ce420 msgr2=0x7f75d00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:35.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.956 7f75d801b700 1 -- 172.21.2.112:0/2527482057 shutdown_connections 2021-02-10T11:40:35.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.956 7f75d801b700 1 -- 172.21.2.112:0/2527482057 wait complete. 2021-02-10T11:40:35.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.971 7f75d801b700 1 Processor -- start 2021-02-10T11:40:35.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.971 7f75d801b700 1 -- start start 2021-02-10T11:40:35.972 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.971 7f75d801b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f75d01a79a0 0x7f75d01a7da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:35.972 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.971 7f75d801b700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f75d00bdb40 0x7f75d00bdf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:35.972 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.971 7f75d801b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f75d0082ca0 0x7f75d00830a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:35.972 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.971 7f75d801b700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75d019d990 con 0x7f75d0082ca0 2021-02-10T11:40:35.972 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.971 7f75d801b700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75d019dbb0 con 0x7f75d00bdb40 2021-02-10T11:40:35.972 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.971 7f75d801b700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75d00836c0 con 0x7f75d01a79a0 2021-02-10T11:40:35.972 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.972 7f75d6638700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f75d0082ca0 0x7f75d00830a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:35.972 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.972 7f75d6638700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f75d0082ca0 0x7f75d00830a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:54788/0 (socket says 172.21.2.112:54788) 2021-02-10T11:40:35.972 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.972 7f75d6638700 1 -- 172.21.2.112:0/3720328765 learned_addr learned my addr 172.21.2.112:0/3720328765 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:40:35.972 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.972 7f75d6638700 1 -- 172.21.2.112:0/3720328765 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f75d01a79a0 msgr2=0x7f75d01a7da0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:40:35.972 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.972 7f75d6638700 1 --2- 172.21.2.112:0/3720328765 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f75d01a79a0 0x7f75d01a7da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:35.972 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.972 7f75d6638700 1 -- 172.21.2.112:0/3720328765 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f75d00bdb40 msgr2=0x7f75d00bdf40 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:40:35.972 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.972 7f75d6638700 1 --2- 172.21.2.112:0/3720328765 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f75d00bdb40 0x7f75d00bdf40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:35.973 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.972 7f75d6638700 1 -- 172.21.2.112:0/3720328765 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75d00838e0 con 0x7f75d0082ca0 2021-02-10T11:40:35.973 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.972 7f75d6638700 1 --2- 172.21.2.112:0/3720328765 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f75d0082ca0 0x7f75d00830a0 secure :-1 s=READY pgs=2083 cs=0 l=1 rev1=1 rx=0x7f75c80099f0 tx=0x7f75c800bd00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:35.973 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.972 7f75c67fc700 1 -- 172.21.2.112:0/3720328765 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f75c80093d0 con 0x7f75d0082ca0 2021-02-10T11:40:35.973 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.972 7f75c67fc700 1 -- 172.21.2.112:0/3720328765 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75c8007070 con 0x7f75d0082ca0 2021-02-10T11:40:35.973 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.972 7f75d801b700 1 -- 172.21.2.112:0/3720328765 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75d00838e0 con 0x7f75d0082ca0 2021-02-10T11:40:35.973 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.972 7f75c67fc700 1 -- 172.21.2.112:0/3720328765 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f75c8007590 con 0x7f75d0082ca0 2021-02-10T11:40:35.973 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.972 7f75d801b700 1 -- 172.21.2.112:0/3720328765 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75d00838e0 con 0x7f75d0082ca0 2021-02-10T11:40:35.973 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.973 7f75c67fc700 1 -- 172.21.2.112:0/3720328765 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f75c80056b0 con 0x7f75d0082ca0 2021-02-10T11:40:35.973 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.973 7f75c67fc700 1 --2- 172.21.2.112:0/3720328765 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f75b8018730 0x7f75b801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:35.973 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.973 7f75c67fc700 1 -- 172.21.2.112:0/3720328765 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f75c8032ba0 con 0x7f75d0082ca0 2021-02-10T11:40:35.973 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.973 7f75d5e37700 1 --2- 172.21.2.112:0/3720328765 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f75b8018730 0x7f75b801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:35.973 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:35.973 7f75d5e37700 1 --2- 172.21.2.112:0/3720328765 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f75b8018730 0x7f75b801ac10 crc :-1 s=READY pgs=966 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:35.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:36.003 7f75d801b700 1 -- 172.21.2.112:0/3720328765 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75d0082a70 con 0x7f75d0082ca0 2021-02-10T11:40:36.001 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:36.006 7f75c67fc700 1 -- 172.21.2.112:0/3720328765 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f75c8007070 con 0x7f75d0082ca0 2021-02-10T11:40:36.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:36.144 7f75d801b700 1 -- 172.21.2.112:0/3720328765 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"} v 0) v1 -- 0x7f75d0082a70 con 0x7f75d0082ca0 2021-02-10T11:40:36.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:36.326 7f75c67fc700 1 -- 172.21.2.112:0/3720328765 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"}]=0 v88) v1 ==== 118+0+0 (secure 0 0 0) 0x7f75c80056b0 con 0x7f75d0082ca0 2021-02-10T11:40:36.358 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:36.362 7f75d801b700 1 -- 172.21.2.112:0/3720328765 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f75b8018730 msgr2=0x7f75b801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:36.358 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:36.362 7f75d801b700 1 --2- 172.21.2.112:0/3720328765 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f75b8018730 0x7f75b801ac10 crc :-1 s=READY pgs=966 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:36.358 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:36.362 7f75d801b700 1 -- 172.21.2.112:0/3720328765 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f75d0082ca0 msgr2=0x7f75d00830a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:36.358 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:36.362 7f75d801b700 1 --2- 172.21.2.112:0/3720328765 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f75d0082ca0 0x7f75d00830a0 secure :-1 s=READY pgs=2083 cs=0 l=1 rev1=1 rx=0x7f75c80099f0 tx=0x7f75c800bd00).stop 2021-02-10T11:40:36.358 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:36.362 7f75d801b700 1 -- 172.21.2.112:0/3720328765 shutdown_connections 2021-02-10T11:40:36.358 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:36.362 7f75d801b700 1 --2- 172.21.2.112:0/3720328765 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f75b8018730 0x7f75b801ac10 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:36.358 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:36.362 7f75d801b700 1 --2- 172.21.2.112:0/3720328765 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f75d00bdb40 0x7f75d00bdf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:36.358 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:36.362 7f75d801b700 1 --2- 172.21.2.112:0/3720328765 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f75d0082ca0 0x7f75d00830a0 unknown :-1 s=CLOSED pgs=2083 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:36.358 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:36.362 7f75d801b700 1 --2- 172.21.2.112:0/3720328765 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f75d01a79a0 0x7f75d01a7da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:36.358 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:36.362 7f75d801b700 1 -- 172.21.2.112:0/3720328765 >> 172.21.2.112:0/3720328765 conn(0x7f75d00bef00 msgr2=0x7f75d009f200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:36.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:36.362 7f75d801b700 1 -- 172.21.2.112:0/3720328765 shutdown_connections 2021-02-10T11:40:36.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:36.362 7f75d801b700 1 -- 172.21.2.112:0/3720328765 wait complete. 2021-02-10T11:40:36.366 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:40:36.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:36.782 7fa14a5bd700 1 Processor -- start 2021-02-10T11:40:36.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:36.782 7fa14a5bd700 1 -- start start 2021-02-10T11:40:36.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:36.782 7fa14a5bd700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fa1440d0340 0x7fa1440ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:36.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:36.782 7fa14a5bd700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fa1440d1c70 0x7fa1440efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:36.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:36.782 7fa14a5bd700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1440f0b10 con 0x7fa1440d1870 2021-02-10T11:40:36.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:36.782 7fa14a5bd700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1440e5ed0 con 0x7fa1440d1c70 2021-02-10T11:40:36.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:36.782 7fa14a5bd700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1441b0be0 con 0x7fa1440d0340 2021-02-10T11:40:36.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:36.783 7fa148bda700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fa1440d1c70 0x7fa1440efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:36.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:36.783 7fa148bda700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fa1440d1c70 0x7fa1440efc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:54794/0 (socket says 172.21.2.112:54794) 2021-02-10T11:40:36.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:36.783 7fa148bda700 1 -- 172.21.2.112:0/3306096365 learned_addr learned my addr 172.21.2.112:0/3306096365 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:40:36.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:36.783 7fa148bda700 1 -- 172.21.2.112:0/3306096365 >> v2:172.21.2.120:3300/0 conn(0x7fa1440d0340 msgr2=0x7fa1440ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:36.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:36.783 7fa148bda700 1 --2- 172.21.2.112:0/3306096365 >> v2:172.21.2.120:3300/0 conn(0x7fa1440d0340 0x7fa1440ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:36.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:36.783 7fa148bda700 1 -- 172.21.2.112:0/3306096365 >> v1:172.21.2.120:6789/0 conn(0x7fa1440d1870 legacy=0x7fa1441b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:36.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:36.783 7fa148bda700 1 -- 172.21.2.112:0/3306096365 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1440f20a0 con 0x7fa1440d1c70 2021-02-10T11:40:36.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:36.783 7fa148bda700 1 --2- 172.21.2.112:0/3306096365 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa1440d1c70 0x7fa1440efc60 secure :-1 s=READY pgs=2084 cs=0 l=1 rev1=1 rx=0x7fa13c006f00 tx=0x7fa13c009f10).ready entity=mon.0 client_cookie=45b1913302a25fd4 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:36.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:36.783 7fa142ffd700 1 -- 172.21.2.112:0/3306096365 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa13c003030 con 0x7fa1440d1c70 2021-02-10T11:40:36.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:36.783 7fa142ffd700 1 -- 172.21.2.112:0/3306096365 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa13c00b3b0 con 0x7fa1440d1c70 2021-02-10T11:40:36.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:36.783 7fa142ffd700 1 -- 172.21.2.112:0/3306096365 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa13c00b8d0 con 0x7fa1440d1c70 2021-02-10T11:40:36.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:36.783 7fa14a5bd700 1 -- 172.21.2.112:0/3306096365 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa1440d1c70 msgr2=0x7fa1440efc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:36.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:36.783 7fa14a5bd700 1 --2- 172.21.2.112:0/3306096365 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa1440d1c70 0x7fa1440efc60 secure :-1 s=READY pgs=2084 cs=0 l=1 rev1=1 rx=0x7fa13c006f00 tx=0x7fa13c009f10).stop 2021-02-10T11:40:36.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:36.784 7fa14a5bd700 1 -- 172.21.2.112:0/3306096365 shutdown_connections 2021-02-10T11:40:36.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:36.784 7fa14a5bd700 1 --2- 172.21.2.112:0/3306096365 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa1440d1c70 0x7fa1440efc60 unknown :-1 s=CLOSED pgs=2084 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:36.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:36.784 7fa14a5bd700 1 --2- 172.21.2.112:0/3306096365 >> v2:172.21.2.120:3300/0 conn(0x7fa1440d0340 0x7fa1440ef640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:36.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:36.784 7fa14a5bd700 1 -- 172.21.2.112:0/3306096365 >> 172.21.2.112:0/3306096365 conn(0x7fa1440ce450 msgr2=0x7fa1440d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:36.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:36.784 7fa14a5bd700 1 -- 172.21.2.112:0/3306096365 shutdown_connections 2021-02-10T11:40:36.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:36.784 7fa14a5bd700 1 -- 172.21.2.112:0/3306096365 wait complete. 2021-02-10T11:40:36.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:36.799 7fa14a5bd700 1 Processor -- start 2021-02-10T11:40:36.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:36.799 7fa14a5bd700 1 -- start start 2021-02-10T11:40:36.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:36.799 7fa14a5bd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa1441947b0 0x7fa1440a69a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:36.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:36.799 7fa14a5bd700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa1440a6fc0 0x7fa1441a15d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:36.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:36.799 7fa14a5bd700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa1441a1bf0 0x7fa1441a1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:36.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:36.799 7fa14a5bd700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1440a73c0 con 0x7fa1441947b0 2021-02-10T11:40:36.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:36.799 7fa14a5bd700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa14418f170 con 0x7fa1441a1bf0 2021-02-10T11:40:36.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:36.799 7fa14a5bd700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa144194bb0 con 0x7fa1440a6fc0 2021-02-10T11:40:36.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:36.799 7fa143fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa1441947b0 0x7fa1440a69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:36.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:36.799 7fa143fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa1441947b0 0x7fa1440a69a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:54800/0 (socket says 172.21.2.112:54800) 2021-02-10T11:40:36.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:36.799 7fa143fff700 1 -- 172.21.2.112:0/2230828774 learned_addr learned my addr 172.21.2.112:0/2230828774 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:40:36.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:36.800 7fa148bda700 1 --2- 172.21.2.112:0/2230828774 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa1441a1bf0 0x7fa1441a1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:36.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:36.800 7fa143fff700 1 -- 172.21.2.112:0/2230828774 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa1440a6fc0 msgr2=0x7fa1441a15d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:36.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:36.800 7fa1437fe700 1 --2- 172.21.2.112:0/2230828774 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa1440a6fc0 0x7fa1441a15d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:36.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:36.800 7fa143fff700 1 --2- 172.21.2.112:0/2230828774 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa1440a6fc0 0x7fa1441a15d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:36.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:36.800 7fa143fff700 1 -- 172.21.2.112:0/2230828774 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa1441a1bf0 msgr2=0x7fa1441a1ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:36.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:36.800 7fa143fff700 1 --2- 172.21.2.112:0/2230828774 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa1441a1bf0 0x7fa1441a1ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:36.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:36.800 7fa143fff700 1 -- 172.21.2.112:0/2230828774 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa14418ec70 con 0x7fa1441947b0 2021-02-10T11:40:36.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:36.800 7fa148bda700 1 --2- 172.21.2.112:0/2230828774 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa1441a1bf0 0x7fa1441a1ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:40:36.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:36.800 7fa1437fe700 1 --2- 172.21.2.112:0/2230828774 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa1440a6fc0 0x7fa1441a15d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:40:36.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:36.800 7fa143fff700 1 --2- 172.21.2.112:0/2230828774 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa1441947b0 0x7fa1440a69a0 secure :-1 s=READY pgs=2085 cs=0 l=1 rev1=1 rx=0x7fa13400b5b0 tx=0x7fa13400b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:36.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:36.800 7fa140ff9700 1 -- 172.21.2.112:0/2230828774 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa13400bd40 con 0x7fa1441947b0 2021-02-10T11:40:36.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:36.800 7fa140ff9700 1 -- 172.21.2.112:0/2230828774 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1340009b0 con 0x7fa1441947b0 2021-02-10T11:40:36.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:36.800 7fa140ff9700 1 -- 172.21.2.112:0/2230828774 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa1340062a0 con 0x7fa1441947b0 2021-02-10T11:40:36.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:36.800 7fa14a5bd700 1 -- 172.21.2.112:0/2230828774 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa14418ec70 con 0x7fa1441947b0 2021-02-10T11:40:36.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:36.800 7fa14a5bd700 1 -- 172.21.2.112:0/2230828774 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1441a2900 con 0x7fa1441947b0 2021-02-10T11:40:36.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:36.801 7fa140ff9700 1 -- 172.21.2.112:0/2230828774 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa134006020 con 0x7fa1441947b0 2021-02-10T11:40:36.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:36.801 7fa140ff9700 1 --2- 172.21.2.112:0/2230828774 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa1240186c0 0x7fa12401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:36.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:36.801 7fa140ff9700 1 -- 172.21.2.112:0/2230828774 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fa1340323d0 con 0x7fa1441947b0 2021-02-10T11:40:36.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:36.802 7fa1437fe700 1 --2- 172.21.2.112:0/2230828774 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa1240186c0 0x7fa12401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:36.800 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:36.802 7fa1437fe700 1 --2- 172.21.2.112:0/2230828774 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa1240186c0 0x7fa12401abc0 crc :-1 s=READY pgs=967 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:36.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:36.832 7fa14a5bd700 1 -- 172.21.2.112:0/2230828774 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1440ee0e0 con 0x7fa1441947b0 2021-02-10T11:40:36.830 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:36.835 7fa140ff9700 1 -- 172.21.2.112:0/2230828774 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa134025f80 con 0x7fa1441947b0 2021-02-10T11:40:36.960 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:36.964 7fa14a5bd700 1 -- 172.21.2.112:0/2230828774 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa1440ce450 con 0x7fa1441947b0 2021-02-10T11:40:36.960 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:36.965 7fa140ff9700 1 -- 172.21.2.112:0/2230828774 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+2107 (secure 0 0 0) 0x7fa134025f80 con 0x7fa1441947b0 2021-02-10T11:40:36.961 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 88 2021-02-10T11:40:36.961 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:40:36.961 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":88,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10335,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/4159879144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":4159879144},{"type":"v1","addr":"172.21.2.112:6833","nonce":4159879144}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":84}],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:40:34.314832","modified":"2021-02-10 11:40:36.323257","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10332},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10332":{"gid":10332,"name":"b","rank":0,"incarnation":85,"state":"up:active","state_seq":3,"addr":"172.21.2.120:6835/3723311838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":3723311838},{"type":"v1","addr":"172.21.2.120:6835","nonce":3723311838}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":11}]} 2021-02-10T11:40:36.962 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:36.967 7fa14a5bd700 1 -- 172.21.2.112:0/2230828774 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa1240186c0 msgr2=0x7fa12401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:36.962 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:36.967 7fa14a5bd700 1 --2- 172.21.2.112:0/2230828774 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa1240186c0 0x7fa12401abc0 crc :-1 s=READY pgs=967 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:36.962 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:36.967 7fa14a5bd700 1 -- 172.21.2.112:0/2230828774 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa1441947b0 msgr2=0x7fa1440a69a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:36.963 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:36.967 7fa14a5bd700 1 --2- 172.21.2.112:0/2230828774 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa1441947b0 0x7fa1440a69a0 secure :-1 s=READY pgs=2085 cs=0 l=1 rev1=1 rx=0x7fa13400b5b0 tx=0x7fa13400b950).stop 2021-02-10T11:40:36.963 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:36.967 7fa14a5bd700 1 -- 172.21.2.112:0/2230828774 shutdown_connections 2021-02-10T11:40:36.963 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:36.967 7fa14a5bd700 1 --2- 172.21.2.112:0/2230828774 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa1240186c0 0x7fa12401abc0 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:36.963 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:36.967 7fa14a5bd700 1 --2- 172.21.2.112:0/2230828774 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa1441a1bf0 0x7fa1441a1ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:36.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:36.967 7fa14a5bd700 1 --2- 172.21.2.112:0/2230828774 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa1440a6fc0 0x7fa1441a15d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:36.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:36.967 7fa14a5bd700 1 --2- 172.21.2.112:0/2230828774 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa1441947b0 0x7fa1440a69a0 unknown :-1 s=CLOSED pgs=2085 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:36.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:36.967 7fa14a5bd700 1 -- 172.21.2.112:0/2230828774 >> 172.21.2.112:0/2230828774 conn(0x7fa1441a38c0 msgr2=0x7fa1441a2f00 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:36.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:36.967 7fa14a5bd700 1 -- 172.21.2.112:0/2230828774 shutdown_connections 2021-02-10T11:40:36.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:36.968 7fa14a5bd700 1 -- 172.21.2.112:0/2230828774 wait complete. 2021-02-10T11:40:36.972 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:40:37.400 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:37.389 7f6d07042700 1 Processor -- start 2021-02-10T11:40:37.400 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:37.389 7f6d07042700 1 -- start start 2021-02-10T11:40:37.401 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:37.389 7f6d07042700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6d000d1870 0x7f6d001b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:37.401 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:37.389 7f6d07042700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f6d000f0650 0x7f6d000cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:37.401 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:37.389 7f6d07042700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d000f0b10 con 0x7f6d001b0a10 2021-02-10T11:40:37.401 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:37.389 7f6d07042700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d000fe230 con 0x7f6d000f0650 2021-02-10T11:40:37.401 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:37.389 7f6d07042700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d000e5ed0 con 0x7f6d000d1870 2021-02-10T11:40:37.401 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:37.389 7f6d0565f700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f6d000f0650 0x7f6d000cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:37.401 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:37.389 7f6d0565f700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f6d000f0650 0x7f6d000cfb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:54812/0 (socket says 172.21.2.112:54812) 2021-02-10T11:40:37.401 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:37.389 7f6d0565f700 1 -- 172.21.2.112:0/2328695332 learned_addr learned my addr 172.21.2.112:0/2328695332 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:40:37.401 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:37.389 7f6d0565f700 1 -- 172.21.2.112:0/2328695332 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6d000d1870 msgr2=0x7f6d001b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:37.401 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:37.389 7f6d0565f700 1 --2- 172.21.2.112:0/2328695332 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6d000d1870 0x7f6d001b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:37.401 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:37.389 7f6d0565f700 1 -- 172.21.2.112:0/2328695332 >> v1:172.21.2.120:6789/0 conn(0x7f6d001b0a10 legacy=0x7f6d000cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:37.401 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:37.389 7f6d0565f700 1 -- 172.21.2.112:0/2328695332 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d000f20a0 con 0x7f6d000f0650 2021-02-10T11:40:37.401 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:37.390 7f6d0565f700 1 --2- 172.21.2.112:0/2328695332 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6d000f0650 0x7f6d000cfb20 secure :-1 s=READY pgs=2086 cs=0 l=1 rev1=1 rx=0x7f6cf8006b80 tx=0x7f6cf8009b90).ready entity=mon.0 client_cookie=a060d9186b18baa2 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:37.402 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:37.390 7f6cff7fe700 1 -- 172.21.2.112:0/2328695332 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6cf80022e0 con 0x7f6d000f0650 2021-02-10T11:40:37.402 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:37.390 7f6cff7fe700 1 -- 172.21.2.112:0/2328695332 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cf8002620 con 0x7f6d000f0650 2021-02-10T11:40:37.402 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:37.390 7f6cff7fe700 1 -- 172.21.2.112:0/2328695332 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6cf8002a80 con 0x7f6d000f0650 2021-02-10T11:40:37.402 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:37.390 7f6d07042700 1 -- 172.21.2.112:0/2328695332 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6d000f0650 msgr2=0x7f6d000cfb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:37.402 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:37.390 7f6d07042700 1 --2- 172.21.2.112:0/2328695332 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6d000f0650 0x7f6d000cfb20 secure :-1 s=READY pgs=2086 cs=0 l=1 rev1=1 rx=0x7f6cf8006b80 tx=0x7f6cf8009b90).stop 2021-02-10T11:40:37.402 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:37.390 7f6d07042700 1 -- 172.21.2.112:0/2328695332 shutdown_connections 2021-02-10T11:40:37.402 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:37.390 7f6d07042700 1 --2- 172.21.2.112:0/2328695332 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6d000f0650 0x7f6d000cfb20 unknown :-1 s=CLOSED pgs=2086 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:37.402 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:37.390 7f6d07042700 1 --2- 172.21.2.112:0/2328695332 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6d000d1870 0x7f6d001b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:37.402 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:37.390 7f6d07042700 1 -- 172.21.2.112:0/2328695332 >> 172.21.2.112:0/2328695332 conn(0x7f6d000ce450 msgr2=0x7f6d000d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:37.402 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:37.391 7f6d07042700 1 -- 172.21.2.112:0/2328695332 shutdown_connections 2021-02-10T11:40:37.402 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:37.391 7f6d07042700 1 -- 172.21.2.112:0/2328695332 wait complete. 2021-02-10T11:40:37.404 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:37.405 7f6d07042700 1 Processor -- start 2021-02-10T11:40:37.404 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:37.405 7f6d07042700 1 -- start start 2021-02-10T11:40:37.404 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:37.406 7f6d07042700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6d000b0ad0 0x7f6d00091f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:37.404 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:37.406 7f6d07042700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6d001a8850 0x7f6d001a8c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:37.404 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:37.406 7f6d07042700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6d001a6fa0 0x7f6d001a73a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:37.404 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:37.406 7f6d07042700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d001abfe0 con 0x7f6d001a8850 2021-02-10T11:40:37.405 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:37.406 7f6d07042700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d001a9c40 con 0x7f6d000b0ad0 2021-02-10T11:40:37.405 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:37.406 7f6d07042700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d001a9730 con 0x7f6d001a6fa0 2021-02-10T11:40:37.405 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:37.406 7f6cfffff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6d001a8850 0x7f6d001a8c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:37.405 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:37.406 7f6cfffff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6d001a8850 0x7f6d001a8c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:54816/0 (socket says 172.21.2.112:54816) 2021-02-10T11:40:37.405 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:37.406 7f6cfffff700 1 -- 172.21.2.112:0/779781179 learned_addr learned my addr 172.21.2.112:0/779781179 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:40:37.405 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:37.406 7f6d0565f700 1 --2- 172.21.2.112:0/779781179 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6d001a6fa0 0x7f6d001a73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:37.405 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:37.406 7f6d04e5e700 1 --2- 172.21.2.112:0/779781179 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6d000b0ad0 0x7f6d00091f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:37.405 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:37.406 7f6cfffff700 1 -- 172.21.2.112:0/779781179 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6d001a6fa0 msgr2=0x7f6d001a73a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:37.405 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:37.406 7f6cfffff700 1 --2- 172.21.2.112:0/779781179 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6d001a6fa0 0x7f6d001a73a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:37.405 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:37.406 7f6cfffff700 1 -- 172.21.2.112:0/779781179 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6d000b0ad0 msgr2=0x7f6d00091f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:37.405 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:37.406 7f6cfffff700 1 --2- 172.21.2.112:0/779781179 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6d000b0ad0 0x7f6d00091f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:37.405 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:37.406 7f6cfffff700 1 -- 172.21.2.112:0/779781179 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d001a9270 con 0x7f6d001a8850 2021-02-10T11:40:37.405 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:37.406 7f6d0565f700 1 --2- 172.21.2.112:0/779781179 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6d001a6fa0 0x7f6d001a73a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:40:37.406 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:37.406 7f6cfffff700 1 --2- 172.21.2.112:0/779781179 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6d001a8850 0x7f6d001a8c50 secure :-1 s=READY pgs=2087 cs=0 l=1 rev1=1 rx=0x7f6cf0009940 tx=0x7f6cf000bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:37.406 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:37.407 7f6cfd7fa700 1 -- 172.21.2.112:0/779781179 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6cf0000fe0 con 0x7f6d001a8850 2021-02-10T11:40:37.406 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:37.407 7f6cfd7fa700 1 -- 172.21.2.112:0/779781179 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cf000c2f0 con 0x7f6d001a8850 2021-02-10T11:40:37.406 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:37.407 7f6d07042700 1 -- 172.21.2.112:0/779781179 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d001a9270 con 0x7f6d001a8850 2021-02-10T11:40:37.406 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:37.407 7f6cfd7fa700 1 -- 172.21.2.112:0/779781179 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6cf000c7d0 con 0x7f6d001a8850 2021-02-10T11:40:37.406 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:37.407 7f6d07042700 1 -- 172.21.2.112:0/779781179 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d001a9270 con 0x7f6d001a8850 2021-02-10T11:40:37.406 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:37.407 7f6cfd7fa700 1 -- 172.21.2.112:0/779781179 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6cf0001230 con 0x7f6d001a8850 2021-02-10T11:40:37.406 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:37.407 7f6cfd7fa700 1 --2- 172.21.2.112:0/779781179 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6ce4018730 0x7f6ce401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:37.406 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:37.407 7f6cfd7fa700 1 -- 172.21.2.112:0/779781179 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f6cf0032bb0 con 0x7f6d001a8850 2021-02-10T11:40:37.406 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:37.408 7f6d04e5e700 1 --2- 172.21.2.112:0/779781179 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6ce4018730 0x7f6ce401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:37.406 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:37.408 7f6d04e5e700 1 --2- 172.21.2.112:0/779781179 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6ce4018730 0x7f6ce401abc0 crc :-1 s=READY pgs=968 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:37.434 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:37.439 7f6d07042700 1 -- 172.21.2.112:0/779781179 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d000ce450 con 0x7f6d001a8850 2021-02-10T11:40:37.437 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:37.441 7f6cfd7fa700 1 -- 172.21.2.112:0/779781179 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6cf000be30 con 0x7f6d001a8850 2021-02-10T11:40:37.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:37.566 7f6d07042700 1 -- 172.21.2.112:0/779781179 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6d000ce450 con 0x7f6d001a8850 2021-02-10T11:40:37.563 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:37.567 7f6cfd7fa700 1 -- 172.21.2.112:0/779781179 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v108) v1 ==== 74+0+14006 (secure 0 0 0) 0x7f6cf0001230 con 0x7f6d001a8850 2021-02-10T11:40:37.563 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:40:37.563 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":108,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:40:35.316267","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:40:32.284968","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":23,"pool_name":"cephfs_data","create_time":"2021-02-10 11:40:33.307261","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:40:37.564 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:37.568 7f6d07042700 1 -- 172.21.2.112:0/779781179 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6ce4018730 msgr2=0x7f6ce401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:37.564 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:37.568 7f6d07042700 1 --2- 172.21.2.112:0/779781179 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6ce4018730 0x7f6ce401abc0 crc :-1 s=READY pgs=968 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:37.564 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:37.568 7f6d07042700 1 -- 172.21.2.112:0/779781179 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6d001a8850 msgr2=0x7f6d001a8c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:37.564 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:37.568 7f6d07042700 1 --2- 172.21.2.112:0/779781179 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6d001a8850 0x7f6d001a8c50 secure :-1 s=READY pgs=2087 cs=0 l=1 rev1=1 rx=0x7f6cf0009940 tx=0x7f6cf000bcf0).stop 2021-02-10T11:40:37.565 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:37.569 7f6d07042700 1 -- 172.21.2.112:0/779781179 shutdown_connections 2021-02-10T11:40:37.565 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:37.569 7f6d07042700 1 --2- 172.21.2.112:0/779781179 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6ce4018730 0x7f6ce401abc0 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:37.565 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:37.569 7f6d07042700 1 --2- 172.21.2.112:0/779781179 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6d001a6fa0 0x7f6d001a73a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:37.565 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:37.569 7f6d07042700 1 --2- 172.21.2.112:0/779781179 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6d001a8850 0x7f6d001a8c50 unknown :-1 s=CLOSED pgs=2087 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:37.565 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:37.569 7f6d07042700 1 --2- 172.21.2.112:0/779781179 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6d000b0ad0 0x7f6d00091f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:37.565 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:37.569 7f6d07042700 1 -- 172.21.2.112:0/779781179 >> 172.21.2.112:0/779781179 conn(0x7f6d0019f8f0 msgr2=0x7f6d001a5f80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:37.565 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:37.569 7f6d07042700 1 -- 172.21.2.112:0/779781179 shutdown_connections 2021-02-10T11:40:37.565 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:37.569 7f6d07042700 1 -- 172.21.2.112:0/779781179 wait complete. 2021-02-10T11:40:37.574 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:40:38.012 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.000 7fd9ab611700 1 Processor -- start 2021-02-10T11:40:38.012 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.000 7fd9ab611700 1 -- start start 2021-02-10T11:40:38.012 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.000 7fd9ab611700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd9a40d1c80 0x7fd9a40cef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:38.012 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.000 7fd9ab611700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9a40d0180 con 0x7fd9a40cf590 2021-02-10T11:40:38.013 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.000 7fd9ab611700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9a40fe1d0 con 0x7fd9a40d1880 2021-02-10T11:40:38.013 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.000 7fd9ab611700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9a41b0cc0 con 0x7fd9a40d1c80 2021-02-10T11:40:38.013 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.000 7fd9a9c2e700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fd9a40cf590 0x7fd9a40cf990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:54288/0 (socket says 172.21.2.112:54288) 2021-02-10T11:40:38.013 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.000 7fd9a9c2e700 1 -- 172.21.2.112:0/1492173494 learned_addr learned my addr 172.21.2.112:0/1492173494 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:40:38.013 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.001 7fd9a8c2c700 1 --2- 172.21.2.112:0/1492173494 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd9a40d1c80 0x7fd9a40cef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:38.013 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.001 7fd997fff700 1 -- 172.21.2.112:0/1492173494 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3011459628 0 0) 0x7fd99c001500 con 0x7fd9a40cf590 2021-02-10T11:40:38.013 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.001 7fd997fff700 1 -- 172.21.2.112:0/1492173494 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd990002fe0 con 0x7fd9a40cf590 2021-02-10T11:40:38.013 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.001 7fd997fff700 1 -- 172.21.2.112:0/1492173494 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1408395351 0 0) 0x7fd998001080 con 0x7fd9a40d1880 2021-02-10T11:40:38.013 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.001 7fd9a8c2c700 1 -- 172.21.2.112:0/1492173494 >> v1:172.21.2.120:6789/0 conn(0x7fd9a40d1880 legacy=0x7fd9a41b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:38.013 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.001 7fd9a8c2c700 1 -- 172.21.2.112:0/1492173494 >> v1:172.21.2.112:6789/0 conn(0x7fd9a40cf590 legacy=0x7fd9a40cf990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:38.013 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.001 7fd9a8c2c700 1 -- 172.21.2.112:0/1492173494 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9a4109a70 con 0x7fd9a40d1c80 2021-02-10T11:40:38.013 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.001 7fd9a8c2c700 1 --2- 172.21.2.112:0/1492173494 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd9a40d1c80 0x7fd9a40cef70 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7fd9a0006650 tx=0x7fd9a0009a00).ready entity=mon.2 client_cookie=dea68fb797b5b20 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:38.013 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.002 7fd997fff700 1 -- 172.21.2.112:0/1492173494 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd9a00009b0 con 0x7fd9a40d1c80 2021-02-10T11:40:38.014 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.002 7fd997fff700 1 -- 172.21.2.112:0/1492173494 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9a000c0c0 con 0x7fd9a40d1c80 2021-02-10T11:40:38.014 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.002 7fd997fff700 1 -- 172.21.2.112:0/1492173494 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd9a000c5e0 con 0x7fd9a40d1c80 2021-02-10T11:40:38.014 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.002 7fd9ab611700 1 -- 172.21.2.112:0/1492173494 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd9a40d1c80 msgr2=0x7fd9a40cef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:38.014 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.002 7fd9ab611700 1 --2- 172.21.2.112:0/1492173494 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd9a40d1c80 0x7fd9a40cef70 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7fd9a0006650 tx=0x7fd9a0009a00).stop 2021-02-10T11:40:38.014 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.002 7fd9ab611700 1 -- 172.21.2.112:0/1492173494 shutdown_connections 2021-02-10T11:40:38.014 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.002 7fd9ab611700 1 --2- 172.21.2.112:0/1492173494 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd9a40d1c80 0x7fd9a40cef70 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:38.014 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.002 7fd9ab611700 1 -- 172.21.2.112:0/1492173494 >> 172.21.2.112:0/1492173494 conn(0x7fd9a40ce430 msgr2=0x7fd9a40d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:38.014 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.002 7fd9ab611700 1 -- 172.21.2.112:0/1492173494 shutdown_connections 2021-02-10T11:40:38.014 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.002 7fd9ab611700 1 -- 172.21.2.112:0/1492173494 wait complete. 2021-02-10T11:40:38.015 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.017 7fd9ab611700 1 Processor -- start 2021-02-10T11:40:38.015 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.017 7fd9ab611700 1 -- start start 2021-02-10T11:40:38.015 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.017 7fd9ab611700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd9a41b9280 0x7fd9a418f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:38.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.017 7fd9ab611700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd9a41b9680 0x7fd9a418fd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:38.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.017 7fd9ab611700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd9a41acd60 0x7fd9a41ad160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:38.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.017 7fd9ab611700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9a4190390 con 0x7fd9a41b9280 2021-02-10T11:40:38.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.017 7fd9ab611700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9a41907c0 con 0x7fd9a41b9680 2021-02-10T11:40:38.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.017 7fd9ab611700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9a41ac3d0 con 0x7fd9a41acd60 2021-02-10T11:40:38.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.018 7fd9a942d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd9a41b9280 0x7fd9a418f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:38.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.018 7fd9a942d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd9a41b9280 0x7fd9a418f750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:54828/0 (socket says 172.21.2.112:54828) 2021-02-10T11:40:38.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.018 7fd9a942d700 1 -- 172.21.2.112:0/2991659342 learned_addr learned my addr 172.21.2.112:0/2991659342 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:40:38.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.018 7fd9a942d700 1 -- 172.21.2.112:0/2991659342 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd9a41acd60 msgr2=0x7fd9a41ad160 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:40:38.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.018 7fd9a942d700 1 --2- 172.21.2.112:0/2991659342 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd9a41acd60 0x7fd9a41ad160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:38.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.018 7fd9a942d700 1 -- 172.21.2.112:0/2991659342 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd9a41b9680 msgr2=0x7fd9a418fd70 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:40:38.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.018 7fd9a942d700 1 --2- 172.21.2.112:0/2991659342 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd9a41b9680 0x7fd9a418fd70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:38.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.018 7fd9a942d700 1 -- 172.21.2.112:0/2991659342 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9a41ad780 con 0x7fd9a41b9280 2021-02-10T11:40:38.017 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.018 7fd9a942d700 1 --2- 172.21.2.112:0/2991659342 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd9a41b9280 0x7fd9a418f750 secure :-1 s=READY pgs=2089 cs=0 l=1 rev1=1 rx=0x7fd998006f30 tx=0x7fd99800b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:38.017 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.018 7fd995ffb700 1 -- 172.21.2.112:0/2991659342 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd99800bd80 con 0x7fd9a41b9280 2021-02-10T11:40:38.017 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.018 7fd995ffb700 1 -- 172.21.2.112:0/2991659342 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd998000e20 con 0x7fd9a41b9280 2021-02-10T11:40:38.017 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.018 7fd995ffb700 1 -- 172.21.2.112:0/2991659342 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd998005540 con 0x7fd9a41b9280 2021-02-10T11:40:38.017 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.018 7fd9ab611700 1 -- 172.21.2.112:0/2991659342 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9a41ad780 con 0x7fd9a41b9280 2021-02-10T11:40:38.017 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.018 7fd9ab611700 1 -- 172.21.2.112:0/2991659342 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9a41add10 con 0x7fd9a41b9280 2021-02-10T11:40:38.017 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.019 7fd995ffb700 1 -- 172.21.2.112:0/2991659342 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd998006030 con 0x7fd9a41b9280 2021-02-10T11:40:38.017 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.019 7fd995ffb700 1 --2- 172.21.2.112:0/2991659342 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd9840186c0 0x7fd98401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:38.017 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.019 7fd995ffb700 1 -- 172.21.2.112:0/2991659342 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fd9980323c0 con 0x7fd9a41b9280 2021-02-10T11:40:38.017 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.020 7fd9a8c2c700 1 --2- 172.21.2.112:0/2991659342 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd9840186c0 0x7fd98401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:38.018 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.020 7fd9a8c2c700 1 --2- 172.21.2.112:0/2991659342 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd9840186c0 0x7fd98401abc0 crc :-1 s=READY pgs=969 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:38.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.050 7fd9ab611700 1 -- 172.21.2.112:0/2991659342 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9a40829a0 con 0x7fd9a41b9280 2021-02-10T11:40:38.048 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.053 7fd995ffb700 1 -- 172.21.2.112:0/2991659342 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd998000bc0 con 0x7fd9a41b9280 2021-02-10T11:40:38.182 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.183 7fd9ab611700 1 -- 172.21.2.112:0/2991659342 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fd9a40829a0 con 0x7fd9a41b9280 2021-02-10T11:40:38.183 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.186 7fd995ffb700 1 -- 172.21.2.112:0/2991659342 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v65) v1 ==== 184+0+0 (secure 0 0 0) 0x7fd998005540 con 0x7fd9a41b9280 2021-02-10T11:40:38.185 INFO:teuthology.orchestra.run.gibba012.stderr:updated caps for client.0 2021-02-10T11:40:38.187 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.191 7fd9ab611700 1 -- 172.21.2.112:0/2991659342 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd9840186c0 msgr2=0x7fd98401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:38.187 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.191 7fd9ab611700 1 --2- 172.21.2.112:0/2991659342 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd9840186c0 0x7fd98401abc0 crc :-1 s=READY pgs=969 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:38.187 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.191 7fd9ab611700 1 -- 172.21.2.112:0/2991659342 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd9a41b9280 msgr2=0x7fd9a418f750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:38.187 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.191 7fd9ab611700 1 --2- 172.21.2.112:0/2991659342 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd9a41b9280 0x7fd9a418f750 secure :-1 s=READY pgs=2089 cs=0 l=1 rev1=1 rx=0x7fd998006f30 tx=0x7fd99800b990).stop 2021-02-10T11:40:38.187 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.192 7fd9ab611700 1 -- 172.21.2.112:0/2991659342 shutdown_connections 2021-02-10T11:40:38.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.192 7fd9ab611700 1 --2- 172.21.2.112:0/2991659342 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd9840186c0 0x7fd98401abc0 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:38.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.192 7fd9ab611700 1 --2- 172.21.2.112:0/2991659342 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd9a41b9680 0x7fd9a418fd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:38.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.192 7fd9ab611700 1 --2- 172.21.2.112:0/2991659342 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd9a41acd60 0x7fd9a41ad160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:38.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.192 7fd9ab611700 1 --2- 172.21.2.112:0/2991659342 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd9a41b9280 0x7fd9a418f750 unknown :-1 s=CLOSED pgs=2089 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:38.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.192 7fd9ab611700 1 -- 172.21.2.112:0/2991659342 >> 172.21.2.112:0/2991659342 conn(0x7fd9a40845b0 msgr2=0x7fd9a418c290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:38.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.192 7fd9ab611700 1 -- 172.21.2.112:0/2991659342 shutdown_connections 2021-02-10T11:40:38.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.192 7fd9ab611700 1 -- 172.21.2.112:0/2991659342 wait complete. 2021-02-10T11:40:38.196 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:40:38.636 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.622 7f60ac095700 1 Processor -- start 2021-02-10T11:40:38.636 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.622 7f60ac095700 1 -- start start 2021-02-10T11:40:38.636 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.623 7f60ac095700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f60a40d1880 0x7f60a41b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:38.636 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.623 7f60ac095700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f60a41b0af0 0x7f60a4108950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:38.636 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.623 7f60ac095700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60a40fe1d0 con 0x7f60a40d1c80 2021-02-10T11:40:38.636 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.623 7f60ac095700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60a4109a70 con 0x7f60a40d1880 2021-02-10T11:40:38.637 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.623 7f60ac095700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60a4109760 con 0x7f60a41b0af0 2021-02-10T11:40:38.637 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.623 7f60aa6b2700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f60a40d1c80 0x7f60a4108f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:54296/0 (socket says 172.21.2.112:54296) 2021-02-10T11:40:38.637 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.623 7f60aa6b2700 1 -- 172.21.2.112:0/3520348873 learned_addr learned my addr 172.21.2.112:0/3520348873 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:40:38.637 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.624 7f60a8eaf700 1 -- 172.21.2.112:0/3520348873 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2475524488 0 0) 0x7f609c001500 con 0x7f60a40d1c80 2021-02-10T11:40:38.637 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.624 7f60a8eaf700 1 -- 172.21.2.112:0/3520348873 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6090002fe0 con 0x7f60a40d1c80 2021-02-10T11:40:38.637 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.624 7f60a9eb1700 1 --2- 172.21.2.112:0/3520348873 >> v2:172.21.2.120:3300/0 conn(0x7f60a40d1880 0x7f60a41b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:38.637 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.624 7f60a96b0700 1 --2- 172.21.2.112:0/3520348873 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f60a41b0af0 0x7f60a4108950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:38.637 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.624 7f60a8eaf700 1 -- 172.21.2.112:0/3520348873 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1892738300 0 0) 0x7f609c001460 con 0x7f60a40d1c80 2021-02-10T11:40:38.637 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.624 7f60a8eaf700 1 -- 172.21.2.112:0/3520348873 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f60a41b0af0 msgr2=0x7f60a4108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:38.637 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.624 7f60a8eaf700 1 --2- 172.21.2.112:0/3520348873 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f60a41b0af0 0x7f60a4108950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:38.637 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.624 7f60a8eaf700 1 -- 172.21.2.112:0/3520348873 >> v2:172.21.2.120:3300/0 conn(0x7f60a40d1880 msgr2=0x7f60a41b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:38.637 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.624 7f60a8eaf700 1 --2- 172.21.2.112:0/3520348873 >> v2:172.21.2.120:3300/0 conn(0x7f60a40d1880 0x7f60a41b04d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:38.637 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.624 7f60a8eaf700 1 -- 172.21.2.112:0/3520348873 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60a4088070 con 0x7f60a40d1c80 2021-02-10T11:40:38.638 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.624 7f60a8eaf700 1 -- 172.21.2.112:0/3520348873 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f609c001980 con 0x7f60a40d1c80 2021-02-10T11:40:38.638 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.624 7f60a8eaf700 1 -- 172.21.2.112:0/3520348873 >> v1:172.21.2.112:6789/0 conn(0x7f60a40d1c80 legacy=0x7f60a4108f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:38.638 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.624 7f60a8eaf700 1 --2- 172.21.2.112:0/3520348873 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6090003ac0 0x7f6090005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:38.638 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.624 7f60a8eaf700 1 --2- 172.21.2.112:0/3520348873 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6090006770 0x7f6090008c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:38.638 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.624 7f60a8eaf700 1 --2- 172.21.2.112:0/3520348873 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6090009500 0x7f609000ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:38.638 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.624 7f60a8eaf700 1 -- 172.21.2.112:0/3520348873 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f609000c2d0 con 0x7f6090003ac0 2021-02-10T11:40:38.638 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.624 7f60a8eaf700 1 -- 172.21.2.112:0/3520348873 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6090050490 con 0x7f6090009500 2021-02-10T11:40:38.638 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.624 7f60a8eaf700 1 -- 172.21.2.112:0/3520348873 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60900506b0 con 0x7f6090006770 2021-02-10T11:40:38.638 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.624 7f60a9eb1700 1 --2- 172.21.2.112:0/3520348873 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6090003ac0 0x7f6090005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:38.638 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.624 7f60aa6b2700 1 --2- 172.21.2.112:0/3520348873 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6090009500 0x7f609000ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:38.638 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.624 7f60a96b0700 1 --2- 172.21.2.112:0/3520348873 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6090006770 0x7f6090008c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:38.638 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.624 7f60a9eb1700 1 -- 172.21.2.112:0/3520348873 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6090006770 msgr2=0x7f6090008c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:38.638 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.624 7f60a9eb1700 1 --2- 172.21.2.112:0/3520348873 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6090006770 0x7f6090008c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:38.639 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.624 7f60a9eb1700 1 -- 172.21.2.112:0/3520348873 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6090009500 msgr2=0x7f609000ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:38.639 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.624 7f60a9eb1700 1 --2- 172.21.2.112:0/3520348873 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6090009500 0x7f609000ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:38.639 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.625 7f60a9eb1700 1 -- 172.21.2.112:0/3520348873 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6090050950 con 0x7f6090003ac0 2021-02-10T11:40:38.639 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.625 7f60a9eb1700 1 --2- 172.21.2.112:0/3520348873 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6090003ac0 0x7f6090005ef0 secure :-1 s=READY pgs=2091 cs=0 l=1 rev1=1 rx=0x7f6098006360 tx=0x7f6098009c30).ready entity=mon.0 client_cookie=d5ef0e7e2b885800 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:38.639 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.625 7f60a8eaf700 1 -- 172.21.2.112:0/3520348873 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f60980009b0 con 0x7f6090003ac0 2021-02-10T11:40:38.639 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.625 7f60a8eaf700 1 -- 172.21.2.112:0/3520348873 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6098000d20 con 0x7f6090003ac0 2021-02-10T11:40:38.639 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.625 7f60a8eaf700 1 -- 172.21.2.112:0/3520348873 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6098002220 con 0x7f6090003ac0 2021-02-10T11:40:38.639 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.625 7f60ac095700 1 -- 172.21.2.112:0/3520348873 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6090003ac0 msgr2=0x7f6090005ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:38.639 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.625 7f60ac095700 1 --2- 172.21.2.112:0/3520348873 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6090003ac0 0x7f6090005ef0 secure :-1 s=READY pgs=2091 cs=0 l=1 rev1=1 rx=0x7f6098006360 tx=0x7f6098009c30).stop 2021-02-10T11:40:38.639 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.625 7f60aa6b2700 1 -- 172.21.2.112:0/3520348873 reap_dead start 2021-02-10T11:40:38.639 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.626 7f60ac095700 1 -- 172.21.2.112:0/3520348873 shutdown_connections 2021-02-10T11:40:38.639 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.626 7f60ac095700 1 -- 172.21.2.112:0/3520348873 >> 172.21.2.112:0/3520348873 conn(0x7f60a40ce430 msgr2=0x7f60a40d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:38.639 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.626 7f60ac095700 1 -- 172.21.2.112:0/3520348873 shutdown_connections 2021-02-10T11:40:38.639 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.626 7f60ac095700 1 -- 172.21.2.112:0/3520348873 wait complete. 2021-02-10T11:40:38.640 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.640 7f60ac095700 1 Processor -- start 2021-02-10T11:40:38.640 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.640 7f60ac095700 1 -- start start 2021-02-10T11:40:38.641 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.641 7f60ac095700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f60a40abd30 0x7f60a4082bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:38.641 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.641 7f60ac095700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f60a40831d0 0x7f60a408a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:38.641 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.641 7f60ac095700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f60a40ac130 0x7f60a408aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:38.641 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.641 7f60ac095700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60a4083700 con 0x7f60a40ac130 2021-02-10T11:40:38.642 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.641 7f60ac095700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60a408b0c0 con 0x7f60a40831d0 2021-02-10T11:40:38.642 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.641 7f60ac095700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60a408b2a0 con 0x7f60a40abd30 2021-02-10T11:40:38.642 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.641 7f60aa6b2700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f60a40ac130 0x7f60a408aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:38.642 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.641 7f60aa6b2700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f60a40ac130 0x7f60a408aaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:54846/0 (socket says 172.21.2.112:54846) 2021-02-10T11:40:38.642 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.641 7f60aa6b2700 1 -- 172.21.2.112:0/2480925885 learned_addr learned my addr 172.21.2.112:0/2480925885 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:40:38.642 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.641 7f60aa6b2700 1 -- 172.21.2.112:0/2480925885 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f60a40abd30 msgr2=0x7f60a4082bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:38.642 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.641 7f60aa6b2700 1 --2- 172.21.2.112:0/2480925885 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f60a40abd30 0x7f60a4082bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:38.642 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.641 7f60aa6b2700 1 -- 172.21.2.112:0/2480925885 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f60a40831d0 msgr2=0x7f60a408a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:38.642 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.641 7f60aa6b2700 1 --2- 172.21.2.112:0/2480925885 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f60a40831d0 0x7f60a408a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:38.642 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.642 7f60aa6b2700 1 -- 172.21.2.112:0/2480925885 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60a408b5f0 con 0x7f60a40ac130 2021-02-10T11:40:38.642 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.642 7f60aa6b2700 1 --2- 172.21.2.112:0/2480925885 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f60a40ac130 0x7f60a408aaa0 secure :-1 s=READY pgs=2092 cs=0 l=1 rev1=1 rx=0x7f609c00a8e0 tx=0x7f609c00ac80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:38.642 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.642 7f60967fc700 1 -- 172.21.2.112:0/2480925885 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f609c000d10 con 0x7f60a40ac130 2021-02-10T11:40:38.642 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.642 7f60967fc700 1 -- 172.21.2.112:0/2480925885 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f609c004930 con 0x7f60a40ac130 2021-02-10T11:40:38.643 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.642 7f60ac095700 1 -- 172.21.2.112:0/2480925885 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60a408b5f0 con 0x7f60a40ac130 2021-02-10T11:40:38.643 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.642 7f60967fc700 1 -- 172.21.2.112:0/2480925885 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f609c004d90 con 0x7f60a40ac130 2021-02-10T11:40:38.643 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.642 7f60ac095700 1 -- 172.21.2.112:0/2480925885 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60a408b5f0 con 0x7f60a40ac130 2021-02-10T11:40:38.643 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.643 7f60967fc700 1 -- 172.21.2.112:0/2480925885 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f609c000d10 con 0x7f60a40ac130 2021-02-10T11:40:38.643 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.643 7f60967fc700 1 --2- 172.21.2.112:0/2480925885 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f608c018730 0x7f608c01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:38.643 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.643 7f60967fc700 1 -- 172.21.2.112:0/2480925885 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f609c0329a0 con 0x7f60a40ac130 2021-02-10T11:40:38.643 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.643 7f60a9eb1700 1 --2- 172.21.2.112:0/2480925885 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f608c018730 0x7f608c01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:38.643 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.643 7f60a9eb1700 1 --2- 172.21.2.112:0/2480925885 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f608c018730 0x7f608c01ac10 crc :-1 s=READY pgs=970 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:38.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.673 7f60ac095700 1 -- 172.21.2.112:0/2480925885 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60a4082980 con 0x7f60a40ac130 2021-02-10T11:40:38.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.675 7f60967fc700 1 -- 172.21.2.112:0/2480925885 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f609c004720 con 0x7f60a40ac130 2021-02-10T11:40:38.803 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.808 7f60ac095700 1 -- 172.21.2.112:0/2480925885 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f60a4082980 con 0x7f60a40ac130 2021-02-10T11:40:38.825 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.810 7f60967fc700 1 -- 172.21.2.112:0/2480925885 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v66) v1 ==== 184+0+0 (secure 0 0 0) 0x7f609c000d10 con 0x7f60a40ac130 2021-02-10T11:40:38.826 INFO:teuthology.orchestra.run.gibba012.stderr:updated caps for client.1 2021-02-10T11:40:38.826 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.812 7f60ac095700 1 -- 172.21.2.112:0/2480925885 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f608c018730 msgr2=0x7f608c01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:38.826 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.812 7f60ac095700 1 --2- 172.21.2.112:0/2480925885 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f608c018730 0x7f608c01ac10 crc :-1 s=READY pgs=970 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:38.826 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.812 7f60ac095700 1 -- 172.21.2.112:0/2480925885 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f60a40ac130 msgr2=0x7f60a408aaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:38.826 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.812 7f60ac095700 1 --2- 172.21.2.112:0/2480925885 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f60a40ac130 0x7f60a408aaa0 secure :-1 s=READY pgs=2092 cs=0 l=1 rev1=1 rx=0x7f609c00a8e0 tx=0x7f609c00ac80).stop 2021-02-10T11:40:38.826 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.812 7f60ac095700 1 -- 172.21.2.112:0/2480925885 shutdown_connections 2021-02-10T11:40:38.826 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.812 7f60ac095700 1 --2- 172.21.2.112:0/2480925885 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f608c018730 0x7f608c01ac10 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:38.826 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.812 7f60ac095700 1 --2- 172.21.2.112:0/2480925885 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f60a40831d0 0x7f60a408a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:38.826 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.812 7f60ac095700 1 --2- 172.21.2.112:0/2480925885 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f60a40ac130 0x7f60a408aaa0 unknown :-1 s=CLOSED pgs=2092 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:38.826 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.812 7f60ac095700 1 --2- 172.21.2.112:0/2480925885 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f60a40abd30 0x7f60a4082bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:38.826 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.812 7f60ac095700 1 -- 172.21.2.112:0/2480925885 >> 172.21.2.112:0/2480925885 conn(0x7f60a40afc60 msgr2=0x7f60a40b50d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:38.826 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.812 7f60ac095700 1 -- 172.21.2.112:0/2480925885 shutdown_connections 2021-02-10T11:40:38.827 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:38.812 7f60ac095700 1 -- 172.21.2.112:0/2480925885 wait complete. 2021-02-10T11:40:38.827 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.2 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:40:39.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.246 7efbf845b700 1 Processor -- start 2021-02-10T11:40:39.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.246 7efbf845b700 1 -- start start 2021-02-10T11:40:39.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.246 7efbf845b700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7efbf00d1880 0x7efbf01b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:39.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.246 7efbf845b700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efbf00fe1d0 con 0x7efbf00cff80 2021-02-10T11:40:39.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.246 7efbf845b700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efbf0109a70 con 0x7efbf01b0af0 2021-02-10T11:40:39.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.246 7efbf845b700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efbf0088070 con 0x7efbf00d1880 2021-02-10T11:40:39.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.247 7efbf6277700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7efbf00d1880 0x7efbf01b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:39.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.247 7efbf6a78700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7efbf00cff80 0x7efbf0109140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:54314/0 (socket says 172.21.2.112:54314) 2021-02-10T11:40:39.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.247 7efbf6a78700 1 -- 172.21.2.112:0/2222792526 learned_addr learned my addr 172.21.2.112:0/2222792526 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:40:39.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.247 7efbf6277700 1 -- 172.21.2.112:0/2222792526 >> v1:172.21.2.120:6789/0 conn(0x7efbf01b0af0 legacy=0x7efbf0108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:39.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.247 7efbf6277700 1 -- 172.21.2.112:0/2222792526 >> v1:172.21.2.112:6789/0 conn(0x7efbf00cff80 legacy=0x7efbf0109140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:39.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.247 7efbf6277700 1 -- 172.21.2.112:0/2222792526 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efbf0088260 con 0x7efbf00d1880 2021-02-10T11:40:39.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.247 7efbf5275700 1 -- 172.21.2.112:0/2222792526 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2359805939 0 0) 0x7efbe8001500 con 0x7efbf00cff80 2021-02-10T11:40:39.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.247 7efbf6277700 1 --2- 172.21.2.112:0/2222792526 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efbf00d1880 0x7efbf01b04d0 secure :-1 s=READY pgs=2094 cs=0 l=1 rev1=1 rx=0x7efbec006650 tx=0x7efbec009a80).ready entity=mon.0 client_cookie=de766696e4be6e9 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:39.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.247 7efbf5275700 1 -- 172.21.2.112:0/2222792526 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efbec0009b0 con 0x7efbf00d1880 2021-02-10T11:40:39.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.247 7efbf5275700 1 -- 172.21.2.112:0/2222792526 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efbec00c0c0 con 0x7efbf00d1880 2021-02-10T11:40:39.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.247 7efbf5275700 1 -- 172.21.2.112:0/2222792526 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efbec00c5e0 con 0x7efbf00d1880 2021-02-10T11:40:39.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.247 7efbf845b700 1 -- 172.21.2.112:0/2222792526 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efbf00d1880 msgr2=0x7efbf01b04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:39.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.247 7efbf845b700 1 --2- 172.21.2.112:0/2222792526 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efbf00d1880 0x7efbf01b04d0 secure :-1 s=READY pgs=2094 cs=0 l=1 rev1=1 rx=0x7efbec006650 tx=0x7efbec009a80).stop 2021-02-10T11:40:39.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.248 7efbf845b700 1 -- 172.21.2.112:0/2222792526 shutdown_connections 2021-02-10T11:40:39.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.248 7efbf845b700 1 --2- 172.21.2.112:0/2222792526 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efbf00d1880 0x7efbf01b04d0 unknown :-1 s=CLOSED pgs=2094 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:39.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.248 7efbf845b700 1 -- 172.21.2.112:0/2222792526 >> 172.21.2.112:0/2222792526 conn(0x7efbf00ce430 msgr2=0x7efbf00d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:39.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.248 7efbf845b700 1 -- 172.21.2.112:0/2222792526 shutdown_connections 2021-02-10T11:40:39.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.248 7efbf845b700 1 -- 172.21.2.112:0/2222792526 wait complete. 2021-02-10T11:40:39.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.263 7efbf845b700 1 Processor -- start 2021-02-10T11:40:39.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.263 7efbf845b700 1 -- start start 2021-02-10T11:40:39.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.264 7efbf845b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efbf00a7500 0x7efbf00a7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:39.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.264 7efbf845b700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7efbf0094310 0x7efbf0094710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:39.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.264 7efbf845b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efbf00955c0 0x7efbf00959c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:39.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.264 7efbf845b700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efbf00950c0 con 0x7efbf00955c0 2021-02-10T11:40:39.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.264 7efbf845b700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efbf00a7940 con 0x7efbf0094310 2021-02-10T11:40:39.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.264 7efbf845b700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efbf00a8200 con 0x7efbf00a7500 2021-02-10T11:40:39.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.264 7efbf6a78700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efbf00955c0 0x7efbf00959c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:39.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.264 7efbf6a78700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efbf00955c0 0x7efbf00959c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:54860/0 (socket says 172.21.2.112:54860) 2021-02-10T11:40:39.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.264 7efbf6a78700 1 -- 172.21.2.112:0/2630428144 learned_addr learned my addr 172.21.2.112:0/2630428144 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:40:39.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.264 7efbf6277700 1 --2- 172.21.2.112:0/2630428144 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efbf00a7500 0x7efbf00a7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:39.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.264 7efbf5a76700 1 --2- 172.21.2.112:0/2630428144 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7efbf0094310 0x7efbf0094710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:39.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.265 7efbf6a78700 1 -- 172.21.2.112:0/2630428144 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efbf00a7500 msgr2=0x7efbf00a7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:39.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.265 7efbf6a78700 1 --2- 172.21.2.112:0/2630428144 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efbf00a7500 0x7efbf00a7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:39.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.265 7efbf6a78700 1 -- 172.21.2.112:0/2630428144 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7efbf0094310 msgr2=0x7efbf0094710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:39.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.265 7efbf6a78700 1 --2- 172.21.2.112:0/2630428144 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7efbf0094310 0x7efbf0094710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:39.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.265 7efbf6a78700 1 -- 172.21.2.112:0/2630428144 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efbf0094d30 con 0x7efbf00955c0 2021-02-10T11:40:39.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.265 7efbf5a76700 1 --2- 172.21.2.112:0/2630428144 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7efbf0094310 0x7efbf0094710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:40:39.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.265 7efbf6a78700 1 --2- 172.21.2.112:0/2630428144 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efbf00955c0 0x7efbf00959c0 secure :-1 s=READY pgs=2095 cs=0 l=1 rev1=1 rx=0x7efbe8001c10 tx=0x7efbe800ac60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:39.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.265 7efbe6ffd700 1 -- 172.21.2.112:0/2630428144 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efbe8000f90 con 0x7efbf00955c0 2021-02-10T11:40:39.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.265 7efbe6ffd700 1 -- 172.21.2.112:0/2630428144 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efbe8007020 con 0x7efbf00955c0 2021-02-10T11:40:39.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.265 7efbe6ffd700 1 -- 172.21.2.112:0/2630428144 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efbe80074c0 con 0x7efbf00955c0 2021-02-10T11:40:39.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.265 7efbf845b700 1 -- 172.21.2.112:0/2630428144 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efbf0094d30 con 0x7efbf00955c0 2021-02-10T11:40:39.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.266 7efbf845b700 1 -- 172.21.2.112:0/2630428144 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efbf0095fe0 con 0x7efbf00955c0 2021-02-10T11:40:39.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.266 7efbe6ffd700 1 -- 172.21.2.112:0/2630428144 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7efbe8007200 con 0x7efbf00955c0 2021-02-10T11:40:39.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.266 7efbe6ffd700 1 --2- 172.21.2.112:0/2630428144 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7efbd80186c0 0x7efbd801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:39.265 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.266 7efbe6ffd700 1 -- 172.21.2.112:0/2630428144 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5573+0+0 (secure 0 0 0) 0x7efbe8032770 con 0x7efbf00955c0 2021-02-10T11:40:39.265 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.266 7efbf6277700 1 --2- 172.21.2.112:0/2630428144 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7efbd80186c0 0x7efbd801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:39.265 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.267 7efbf6277700 1 --2- 172.21.2.112:0/2630428144 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7efbd80186c0 0x7efbd801abc0 crc :-1 s=READY pgs=971 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:39.291 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.296 7efbf845b700 1 -- 172.21.2.112:0/2630428144 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efbf0080730 con 0x7efbf00955c0 2021-02-10T11:40:39.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.298 7efbe6ffd700 1 -- 172.21.2.112:0/2630428144 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7efbe8000f90 con 0x7efbf00955c0 2021-02-10T11:40:39.425 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.429 7efbf845b700 1 -- 172.21.2.112:0/2630428144 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7efbf00ce430 con 0x7efbf00955c0 2021-02-10T11:40:39.427 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.431 7efbe6ffd700 1 -- 172.21.2.112:0/2630428144 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.2 v67) v1 ==== 184+0+0 (secure 0 0 0) 0x7efbe8007200 con 0x7efbf00955c0 2021-02-10T11:40:39.427 INFO:teuthology.orchestra.run.gibba012.stderr:updated caps for client.2 2021-02-10T11:40:39.429 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.433 7efbf845b700 1 -- 172.21.2.112:0/2630428144 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7efbd80186c0 msgr2=0x7efbd801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:39.429 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.433 7efbf845b700 1 --2- 172.21.2.112:0/2630428144 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7efbd80186c0 0x7efbd801abc0 crc :-1 s=READY pgs=971 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:39.429 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.433 7efbf845b700 1 -- 172.21.2.112:0/2630428144 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efbf00955c0 msgr2=0x7efbf00959c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:39.429 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.433 7efbf845b700 1 --2- 172.21.2.112:0/2630428144 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efbf00955c0 0x7efbf00959c0 secure :-1 s=READY pgs=2095 cs=0 l=1 rev1=1 rx=0x7efbe8001c10 tx=0x7efbe800ac60).stop 2021-02-10T11:40:39.429 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.433 7efbf845b700 1 -- 172.21.2.112:0/2630428144 shutdown_connections 2021-02-10T11:40:39.429 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.433 7efbf845b700 1 --2- 172.21.2.112:0/2630428144 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7efbd80186c0 0x7efbd801abc0 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:39.430 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.434 7efbf845b700 1 --2- 172.21.2.112:0/2630428144 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7efbf0094310 0x7efbf0094710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:39.430 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.434 7efbf845b700 1 --2- 172.21.2.112:0/2630428144 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efbf00955c0 0x7efbf00959c0 unknown :-1 s=CLOSED pgs=2095 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:39.430 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.434 7efbf845b700 1 --2- 172.21.2.112:0/2630428144 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efbf00a7500 0x7efbf00a7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:39.430 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.434 7efbf845b700 1 -- 172.21.2.112:0/2630428144 >> 172.21.2.112:0/2630428144 conn(0x7efbf01071a0 msgr2=0x7efbf0100cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:39.430 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.434 7efbf845b700 1 -- 172.21.2.112:0/2630428144 shutdown_connections 2021-02-10T11:40:39.430 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.434 7efbf845b700 1 -- 172.21.2.112:0/2630428144 wait complete. 2021-02-10T11:40:39.438 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.3 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:40:39.876 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.864 7f547577c700 1 Processor -- start 2021-02-10T11:40:39.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.865 7f547577c700 1 -- start start 2021-02-10T11:40:39.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.865 7f547577c700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f54700d1c80 0x7f54700cef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:39.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.865 7f547577c700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f54700cf590 0x7f54700cf990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:39.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.865 7f547577c700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54700fe1d0 con 0x7f54700d1880 2021-02-10T11:40:39.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.865 7f547577c700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54701b0cc0 con 0x7f54700d1c80 2021-02-10T11:40:39.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.865 7f547577c700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5470109a70 con 0x7f54700cf590 2021-02-10T11:40:39.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.865 7f546ed73700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f54700d1c80 0x7f54700cef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:39.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.865 7f546ed73700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f54700d1c80 0x7f54700cef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:54864/0 (socket says 172.21.2.112:54864) 2021-02-10T11:40:39.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.865 7f546ed73700 1 -- 172.21.2.112:0/446096303 learned_addr learned my addr 172.21.2.112:0/446096303 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:40:39.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.865 7f546ed73700 1 -- 172.21.2.112:0/446096303 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f54700cf590 msgr2=0x7f54700cf990 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:40:39.878 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.865 7f546ed73700 1 --2- 172.21.2.112:0/446096303 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f54700cf590 0x7f54700cf990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:39.878 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.865 7f546ed73700 1 -- 172.21.2.112:0/446096303 >> v1:172.21.2.120:6789/0 conn(0x7f54700d1880 legacy=0x7f54701b04d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:40:39.878 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.865 7f546ed73700 1 -- 172.21.2.112:0/446096303 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5470088070 con 0x7f54700d1c80 2021-02-10T11:40:39.878 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.866 7f546ed73700 1 --2- 172.21.2.112:0/446096303 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f54700d1c80 0x7f54700cef70 secure :-1 s=READY pgs=2096 cs=0 l=1 rev1=1 rx=0x7f5460006700 tx=0x7f5460009b80).ready entity=mon.0 client_cookie=b86fc045ffb4309c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:39.878 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.866 7f546e572700 1 -- 172.21.2.112:0/446096303 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f546000c0c0 con 0x7f54700d1c80 2021-02-10T11:40:39.878 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.866 7f546e572700 1 -- 172.21.2.112:0/446096303 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f546000c340 con 0x7f54700d1c80 2021-02-10T11:40:39.878 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.866 7f546e572700 1 -- 172.21.2.112:0/446096303 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f546000c860 con 0x7f54700d1c80 2021-02-10T11:40:39.878 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.866 7f547577c700 1 -- 172.21.2.112:0/446096303 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f54700d1c80 msgr2=0x7f54700cef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:39.878 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.866 7f547577c700 1 --2- 172.21.2.112:0/446096303 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f54700d1c80 0x7f54700cef70 secure :-1 s=READY pgs=2096 cs=0 l=1 rev1=1 rx=0x7f5460006700 tx=0x7f5460009b80).stop 2021-02-10T11:40:39.878 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.867 7f547577c700 1 -- 172.21.2.112:0/446096303 shutdown_connections 2021-02-10T11:40:39.878 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.867 7f547577c700 1 --2- 172.21.2.112:0/446096303 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f54700d1c80 0x7f54700cef70 unknown :-1 s=CLOSED pgs=2096 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:39.878 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.867 7f547577c700 1 --2- 172.21.2.112:0/446096303 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f54700cf590 0x7f54700cf990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:39.878 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.867 7f547577c700 1 -- 172.21.2.112:0/446096303 >> 172.21.2.112:0/446096303 conn(0x7f54700ce430 msgr2=0x7f54700d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:39.879 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.867 7f547577c700 1 -- 172.21.2.112:0/446096303 shutdown_connections 2021-02-10T11:40:39.879 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.867 7f547577c700 1 -- 172.21.2.112:0/446096303 wait complete. 2021-02-10T11:40:39.880 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.881 7f547577c700 1 Processor -- start 2021-02-10T11:40:39.880 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.881 7f547577c700 1 -- start start 2021-02-10T11:40:39.880 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.882 7f547577c700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5470086810 0x7f547019fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:39.880 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.882 7f547577c700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f54700b04d0 0x7f5470097450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:39.881 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.882 7f547577c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f54700b08d0 0x7f5470097a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:39.881 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.882 7f547577c700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54701a02c0 con 0x7f54700b08d0 2021-02-10T11:40:39.881 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.882 7f547577c700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5470098090 con 0x7f5470086810 2021-02-10T11:40:39.881 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.882 7f547577c700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54700982b0 con 0x7f54700b04d0 2021-02-10T11:40:39.881 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.882 7f546fd75700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f54700b08d0 0x7f5470097a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:39.881 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.882 7f546fd75700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f54700b08d0 0x7f5470097a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:54874/0 (socket says 172.21.2.112:54874) 2021-02-10T11:40:39.881 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.882 7f546fd75700 1 -- 172.21.2.112:0/2678164489 learned_addr learned my addr 172.21.2.112:0/2678164489 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:40:39.881 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.882 7f546fd75700 1 -- 172.21.2.112:0/2678164489 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f54700b04d0 msgr2=0x7f5470097450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:39.881 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.882 7f546fd75700 1 --2- 172.21.2.112:0/2678164489 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f54700b04d0 0x7f5470097450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:39.881 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.882 7f546fd75700 1 -- 172.21.2.112:0/2678164489 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5470086810 msgr2=0x7f547019fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:39.881 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.882 7f546fd75700 1 --2- 172.21.2.112:0/2678164489 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5470086810 0x7f547019fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:39.881 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.882 7f546fd75700 1 -- 172.21.2.112:0/2678164489 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54700984d0 con 0x7f54700b08d0 2021-02-10T11:40:39.881 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.882 7f546fd75700 1 --2- 172.21.2.112:0/2678164489 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f54700b08d0 0x7f5470097a70 secure :-1 s=READY pgs=2097 cs=0 l=1 rev1=1 rx=0x7f5468009960 tx=0x7f546800bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:39.882 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.883 7f5467fff700 1 -- 172.21.2.112:0/2678164489 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5468000d10 con 0x7f54700b08d0 2021-02-10T11:40:39.882 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.883 7f5467fff700 1 -- 172.21.2.112:0/2678164489 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54680030c0 con 0x7f54700b08d0 2021-02-10T11:40:39.882 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.883 7f547577c700 1 -- 172.21.2.112:0/2678164489 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54700984d0 con 0x7f54700b08d0 2021-02-10T11:40:39.882 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.883 7f547577c700 1 -- 172.21.2.112:0/2678164489 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5470098b80 con 0x7f54700b08d0 2021-02-10T11:40:39.882 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.883 7f5467fff700 1 -- 172.21.2.112:0/2678164489 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5468001500 con 0x7f54700b08d0 2021-02-10T11:40:39.882 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.883 7f5467fff700 1 -- 172.21.2.112:0/2678164489 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f546800a030 con 0x7f54700b08d0 2021-02-10T11:40:39.882 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.883 7f5467fff700 1 --2- 172.21.2.112:0/2678164489 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f54540186c0 0x7f545401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:39.882 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.883 7f5467fff700 1 -- 172.21.2.112:0/2678164489 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f5468032770 con 0x7f54700b08d0 2021-02-10T11:40:39.883 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.884 7f546f574700 1 --2- 172.21.2.112:0/2678164489 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f54540186c0 0x7f545401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:39.883 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.884 7f546f574700 1 --2- 172.21.2.112:0/2678164489 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f54540186c0 0x7f545401abc0 crc :-1 s=READY pgs=972 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:39.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.915 7f547577c700 1 -- 172.21.2.112:0/2678164489 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54700fe1d0 con 0x7f54700b08d0 2021-02-10T11:40:39.913 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:39.917 7f5467fff700 1 -- 172.21.2.112:0/2678164489 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5468000f40 con 0x7f54700b08d0 2021-02-10T11:40:40.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:40.048 7f547577c700 1 -- 172.21.2.112:0/2678164489 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f54700fe1d0 con 0x7f54700b08d0 2021-02-10T11:40:40.047 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:40.051 7f5467fff700 1 -- 172.21.2.112:0/2678164489 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.3 v68) v1 ==== 184+0+0 (secure 0 0 0) 0x7f546801f870 con 0x7f54700b08d0 2021-02-10T11:40:40.051 INFO:teuthology.orchestra.run.gibba012.stderr:updated caps for client.3 2021-02-10T11:40:40.052 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:40.056 7f547577c700 1 -- 172.21.2.112:0/2678164489 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f54540186c0 msgr2=0x7f545401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:40.052 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:40.056 7f547577c700 1 --2- 172.21.2.112:0/2678164489 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f54540186c0 0x7f545401abc0 crc :-1 s=READY pgs=972 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:40.052 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:40.056 7f547577c700 1 -- 172.21.2.112:0/2678164489 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f54700b08d0 msgr2=0x7f5470097a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:40.052 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:40.056 7f547577c700 1 --2- 172.21.2.112:0/2678164489 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f54700b08d0 0x7f5470097a70 secure :-1 s=READY pgs=2097 cs=0 l=1 rev1=1 rx=0x7f5468009960 tx=0x7f546800bcf0).stop 2021-02-10T11:40:40.052 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:40.057 7f547577c700 1 -- 172.21.2.112:0/2678164489 shutdown_connections 2021-02-10T11:40:40.053 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:40.057 7f547577c700 1 --2- 172.21.2.112:0/2678164489 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f54540186c0 0x7f545401abc0 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:40.053 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:40.057 7f547577c700 1 --2- 172.21.2.112:0/2678164489 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f54700b08d0 0x7f5470097a70 unknown :-1 s=CLOSED pgs=2097 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:40.053 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:40.057 7f547577c700 1 --2- 172.21.2.112:0/2678164489 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f54700b04d0 0x7f5470097450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:40.053 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:40.057 7f547577c700 1 --2- 172.21.2.112:0/2678164489 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5470086810 0x7f547019fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:40.053 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:40.057 7f547577c700 1 -- 172.21.2.112:0/2678164489 >> 172.21.2.112:0/2678164489 conn(0x7f547019f880 msgr2=0x7f547008f890 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:40.053 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:40.057 7f547577c700 1 -- 172.21.2.112:0/2678164489 shutdown_connections 2021-02-10T11:40:40.053 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:40.057 7f547577c700 1 -- 172.21.2.112:0/2678164489 wait complete. 2021-02-10T11:40:40.061 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:40:40.488 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:40.476 7fa32b333700 1 Processor -- start 2021-02-10T11:40:40.489 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:40.476 7fa32b333700 1 -- start start 2021-02-10T11:40:40.489 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:40.476 7fa32b333700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fa3241b0470 0x7fa3241b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:40.489 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:40.476 7fa32b333700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa3240ef610 0x7fa3240efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:40.489 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:40.476 7fa32b333700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3240f0b50 con 0x7fa3240f0030 2021-02-10T11:40:40.489 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:40.476 7fa32b333700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3240e52e0 con 0x7fa3241b0470 2021-02-10T11:40:40.489 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:40.476 7fa32b333700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3240e54d0 con 0x7fa3240ef610 2021-02-10T11:40:40.489 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:40.477 7fa32914f700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fa3241b0470 0x7fa3241b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:40.489 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:40.477 7fa32914f700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fa3241b0470 0x7fa3241b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:54878/0 (socket says 172.21.2.112:54878) 2021-02-10T11:40:40.489 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:40.477 7fa32914f700 1 -- 172.21.2.112:0/839016106 learned_addr learned my addr 172.21.2.112:0/839016106 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:40:40.489 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:40.477 7fa329950700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fa3240f0030 0x7fa3240f10d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:54340/0 (socket says 172.21.2.112:54340) 2021-02-10T11:40:40.489 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:40.477 7fa32914f700 1 -- 172.21.2.112:0/839016106 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa3240ef610 msgr2=0x7fa3240efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:40.489 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:40.477 7fa32914f700 1 --2- 172.21.2.112:0/839016106 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa3240ef610 0x7fa3240efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:40.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:40.477 7fa32914f700 1 -- 172.21.2.112:0/839016106 >> v1:172.21.2.112:6789/0 conn(0x7fa3240f0030 legacy=0x7fa3240f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:40.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:40.477 7fa32914f700 1 -- 172.21.2.112:0/839016106 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3240f0430 con 0x7fa3241b0470 2021-02-10T11:40:40.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:40.477 7fa32914f700 1 --2- 172.21.2.112:0/839016106 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa3241b0470 0x7fa3241b0870 secure :-1 s=READY pgs=2099 cs=0 l=1 rev1=1 rx=0x7fa318006700 tx=0x7fa318009b80).ready entity=mon.0 client_cookie=6bd2775bdded107 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:40.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:40.477 7fa317fff700 1 -- 172.21.2.112:0/839016106 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa31800c0c0 con 0x7fa3241b0470 2021-02-10T11:40:40.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:40.478 7fa317fff700 1 -- 172.21.2.112:0/839016106 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa31800c340 con 0x7fa3241b0470 2021-02-10T11:40:40.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:40.478 7fa317fff700 1 -- 172.21.2.112:0/839016106 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa31800c860 con 0x7fa3241b0470 2021-02-10T11:40:40.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:40.478 7fa32b333700 1 -- 172.21.2.112:0/839016106 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa3241b0470 msgr2=0x7fa3241b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:40.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:40.478 7fa32b333700 1 --2- 172.21.2.112:0/839016106 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa3241b0470 0x7fa3241b0870 secure :-1 s=READY pgs=2099 cs=0 l=1 rev1=1 rx=0x7fa318006700 tx=0x7fa318009b80).stop 2021-02-10T11:40:40.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:40.478 7fa32b333700 1 -- 172.21.2.112:0/839016106 shutdown_connections 2021-02-10T11:40:40.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:40.478 7fa32b333700 1 --2- 172.21.2.112:0/839016106 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa3240ef610 0x7fa3240efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:40.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:40.478 7fa32b333700 1 --2- 172.21.2.112:0/839016106 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa3241b0470 0x7fa3241b0870 unknown :-1 s=CLOSED pgs=2099 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:40.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:40.478 7fa32b333700 1 -- 172.21.2.112:0/839016106 >> 172.21.2.112:0/839016106 conn(0x7fa3240ce3e0 msgr2=0x7fa3240e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:40.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:40.479 7fa32b333700 1 -- 172.21.2.112:0/839016106 shutdown_connections 2021-02-10T11:40:40.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:40.479 7fa32b333700 1 -- 172.21.2.112:0/839016106 wait complete. 2021-02-10T11:40:40.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:40.493 7fa32b333700 1 Processor -- start 2021-02-10T11:40:40.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:40.493 7fa32b333700 1 -- start start 2021-02-10T11:40:40.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:40.494 7fa32b333700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa3241b9410 0x7fa32419bb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:40.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:40.494 7fa32b333700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa3241b8880 0x7fa32408a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:40.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:40.494 7fa32b333700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa324187e20 0x7fa324188220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:40.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:40.494 7fa32b333700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa32419c120 con 0x7fa3241b9410 2021-02-10T11:40:40.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:40.494 7fa32b333700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3241b9810 con 0x7fa324187e20 2021-02-10T11:40:40.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:40.494 7fa32b333700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3241b8e90 con 0x7fa3241b8880 2021-02-10T11:40:40.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:40.494 7fa32914f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa3241b9410 0x7fa32419bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:40.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:40.494 7fa32914f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa3241b9410 0x7fa32419bb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:54886/0 (socket says 172.21.2.112:54886) 2021-02-10T11:40:40.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:40.494 7fa32914f700 1 -- 172.21.2.112:0/2840960973 learned_addr learned my addr 172.21.2.112:0/2840960973 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:40:40.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:40.494 7fa32914f700 1 -- 172.21.2.112:0/2840960973 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa3241b8880 msgr2=0x7fa32408a740 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:40:40.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:40.494 7fa32914f700 1 --2- 172.21.2.112:0/2840960973 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa3241b8880 0x7fa32408a740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:40.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:40.494 7fa32914f700 1 -- 172.21.2.112:0/2840960973 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa324187e20 msgr2=0x7fa324188220 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:40:40.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:40.494 7fa32914f700 1 --2- 172.21.2.112:0/2840960973 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa324187e20 0x7fa324188220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:40.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:40.494 7fa32914f700 1 -- 172.21.2.112:0/2840960973 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa32408ad60 con 0x7fa3241b9410 2021-02-10T11:40:40.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:40.494 7fa32914f700 1 --2- 172.21.2.112:0/2840960973 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa3241b9410 0x7fa32419bb00 secure :-1 s=READY pgs=2100 cs=0 l=1 rev1=1 rx=0x7fa31800b650 tx=0x7fa31800b9f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:40.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:40.494 7fa315ffb700 1 -- 172.21.2.112:0/2840960973 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa31800bde0 con 0x7fa3241b9410 2021-02-10T11:40:40.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:40.494 7fa315ffb700 1 -- 172.21.2.112:0/2840960973 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa318002030 con 0x7fa3241b9410 2021-02-10T11:40:40.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:40.494 7fa32b333700 1 -- 172.21.2.112:0/2840960973 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa32408ad60 con 0x7fa3241b9410 2021-02-10T11:40:40.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:40.494 7fa315ffb700 1 -- 172.21.2.112:0/2840960973 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa318002490 con 0x7fa3241b9410 2021-02-10T11:40:40.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:40.494 7fa32b333700 1 -- 172.21.2.112:0/2840960973 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa324188840 con 0x7fa3241b9410 2021-02-10T11:40:40.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:40.495 7fa315ffb700 1 -- 172.21.2.112:0/2840960973 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa318000c90 con 0x7fa3241b9410 2021-02-10T11:40:40.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:40.495 7fa315ffb700 1 --2- 172.21.2.112:0/2840960973 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa304018730 0x7fa30401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:40.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:40.495 7fa315ffb700 1 -- 172.21.2.112:0/2840960973 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fa318032640 con 0x7fa3241b9410 2021-02-10T11:40:40.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:40.496 7fa32894e700 1 --2- 172.21.2.112:0/2840960973 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa304018730 0x7fa30401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:40.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:40.496 7fa32894e700 1 --2- 172.21.2.112:0/2840960973 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa304018730 0x7fa30401ac10 crc :-1 s=READY pgs=973 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:40.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:40.526 7fa32b333700 1 -- 172.21.2.112:0/2840960973 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3241b9810 con 0x7fa3241b9410 2021-02-10T11:40:40.524 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:40.529 7fa315ffb700 1 -- 172.21.2.112:0/2840960973 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa3180090b0 con 0x7fa3241b9410 2021-02-10T11:40:40.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:40.658 7fa32b333700 1 -- 172.21.2.112:0/2840960973 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa3241b9810 con 0x7fa3241b9410 2021-02-10T11:40:40.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:40.658 7fa315ffb700 1 -- 172.21.2.112:0/2840960973 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+2107 (secure 0 0 0) 0x7fa31800bde0 con 0x7fa3241b9410 2021-02-10T11:40:40.655 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 88 2021-02-10T11:40:40.655 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:40:40.655 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":88,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10335,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/4159879144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":4159879144},{"type":"v1","addr":"172.21.2.112:6833","nonce":4159879144}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":84}],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:40:34.314832","modified":"2021-02-10 11:40:36.323257","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10332},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10332":{"gid":10332,"name":"b","rank":0,"incarnation":85,"state":"up:active","state_seq":3,"addr":"172.21.2.120:6835/3723311838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":3723311838},{"type":"v1","addr":"172.21.2.120:6835","nonce":3723311838}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":11}]} 2021-02-10T11:40:40.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:40.660 7fa32b333700 1 -- 172.21.2.112:0/2840960973 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa304018730 msgr2=0x7fa30401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:40.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:40.660 7fa32b333700 1 --2- 172.21.2.112:0/2840960973 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa304018730 0x7fa30401ac10 crc :-1 s=READY pgs=973 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:40.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:40.660 7fa32b333700 1 -- 172.21.2.112:0/2840960973 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa3241b9410 msgr2=0x7fa32419bb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:40.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:40.660 7fa32b333700 1 --2- 172.21.2.112:0/2840960973 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa3241b9410 0x7fa32419bb00 secure :-1 s=READY pgs=2100 cs=0 l=1 rev1=1 rx=0x7fa31800b650 tx=0x7fa31800b9f0).stop 2021-02-10T11:40:40.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:40.660 7fa32b333700 1 -- 172.21.2.112:0/2840960973 shutdown_connections 2021-02-10T11:40:40.657 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:40.660 7fa32b333700 1 --2- 172.21.2.112:0/2840960973 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa304018730 0x7fa30401ac10 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:40.657 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:40.660 7fa32b333700 1 --2- 172.21.2.112:0/2840960973 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa324187e20 0x7fa324188220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:40.657 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:40.660 7fa32b333700 1 --2- 172.21.2.112:0/2840960973 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa3241b8880 0x7fa32408a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:40.657 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:40.660 7fa32b333700 1 --2- 172.21.2.112:0/2840960973 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa3241b9410 0x7fa32419bb00 unknown :-1 s=CLOSED pgs=2100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:40.657 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:40.660 7fa32b333700 1 -- 172.21.2.112:0/2840960973 >> 172.21.2.112:0/2840960973 conn(0x7fa32419f7c0 msgr2=0x7fa32418c230 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:40.657 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:40.661 7fa32b333700 1 -- 172.21.2.112:0/2840960973 shutdown_connections 2021-02-10T11:40:40.657 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:40.661 7fa32b333700 1 -- 172.21.2.112:0/2840960973 wait complete. 2021-02-10T11:40:40.866 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 88, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:40:34.314832', 'modified': '2021-02-10 11:40:36.323257', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 10332}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10332': {'gid': 10332, 'name': 'b', 'rank': 0, 'incarnation': 85, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.120:6835/3723311838', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.120:6834', 'nonce': 3723311838}, {'type': 'v1', 'addr': '172.21.2.120:6835', 'nonce': 3723311838}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [23], 'metadata_pool': 22, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:40:40.866 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:40:40.867 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:40:41.266 INFO:teuthology.orchestra.run.gibba020.stdout:{ 2021-02-10T11:40:41.266 INFO:teuthology.orchestra.run.gibba020.stdout: "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", 2021-02-10T11:40:41.266 INFO:teuthology.orchestra.run.gibba020.stdout: "whoami": 0, 2021-02-10T11:40:41.266 INFO:teuthology.orchestra.run.gibba020.stdout: "id": 10332, 2021-02-10T11:40:41.266 INFO:teuthology.orchestra.run.gibba020.stdout: "want_state": "up:active", 2021-02-10T11:40:41.267 INFO:teuthology.orchestra.run.gibba020.stdout: "state": "up:active", 2021-02-10T11:40:41.267 INFO:teuthology.orchestra.run.gibba020.stdout: "rank_uptime": 6.955030592, 2021-02-10T11:40:41.267 INFO:teuthology.orchestra.run.gibba020.stdout: "mdsmap_epoch": 88, 2021-02-10T11:40:41.267 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch": 108, 2021-02-10T11:40:41.267 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch_barrier": 108, 2021-02-10T11:40:41.267 INFO:teuthology.orchestra.run.gibba020.stdout: "uptime": 12.598421975999999 2021-02-10T11:40:41.267 INFO:teuthology.orchestra.run.gibba020.stdout:} 2021-02-10T11:40:41.270 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", "whoami": 0, "id": 10332, "want_state": "up:active", "state": "up:active", "rank_uptime": 6.955030592, "mdsmap_epoch": 88, "osdmap_epoch": 108, "osdmap_epoch_barrier": 108, "uptime": 12.598421975999999 } 2021-02-10T11:40:41.271 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:40:41.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:41.686 7f705b5f5700 1 Processor -- start 2021-02-10T11:40:41.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:41.686 7f705b5f5700 1 -- start start 2021-02-10T11:40:41.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:41.687 7f705b5f5700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f70540d1870 0x7f70541b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:41.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:41.687 7f705b5f5700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f70541b0a10 0x7f70540cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:41.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:41.687 7f705b5f5700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f70540d1c70 0x7f70540cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:41.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:41.687 7f705b5f5700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70540fe230 con 0x7f70541b0a10 2021-02-10T11:40:41.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:41.687 7f705b5f5700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70540e5ed0 con 0x7f70540d1c70 2021-02-10T11:40:41.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:41.687 7f705b5f5700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70540f0b10 con 0x7f70540d1870 2021-02-10T11:40:41.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:41.687 7f7058c10700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f70541b0a10 0x7f70540cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:41.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:41.687 7f7058c10700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f70541b0a10 0x7f70540cf330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:54890/0 (socket says 172.21.2.112:54890) 2021-02-10T11:40:41.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:41.687 7f7058c10700 1 -- 172.21.2.112:0/2266652 learned_addr learned my addr 172.21.2.112:0/2266652 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:40:41.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:41.687 7f7058c10700 1 -- 172.21.2.112:0/2266652 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f70540d1870 msgr2=0x7f70541b03f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:40:41.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:41.687 7f7058c10700 1 --2- 172.21.2.112:0/2266652 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f70540d1870 0x7f70541b03f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:41.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:41.687 7f7058c10700 1 -- 172.21.2.112:0/2266652 >> v2:172.21.2.120:3300/0 conn(0x7f70540d1c70 msgr2=0x7f70540cf950 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:40:41.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:41.687 7f7058c10700 1 --2- 172.21.2.112:0/2266652 >> v2:172.21.2.120:3300/0 conn(0x7f70540d1c70 0x7f70540cf950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:41.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:41.687 7f7058c10700 1 -- 172.21.2.112:0/2266652 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70540cff70 con 0x7f70541b0a10 2021-02-10T11:40:41.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:41.687 7f7058c10700 1 --2- 172.21.2.112:0/2266652 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f70541b0a10 0x7f70540cf330 secure :-1 s=READY pgs=2101 cs=0 l=1 rev1=1 rx=0x7f7050006700 tx=0x7f7050009b00).ready entity=mon.0 client_cookie=b945d12dc83bcc88 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:41.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:41.687 7f7047fff700 1 -- 172.21.2.112:0/2266652 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f705000c0c0 con 0x7f70541b0a10 2021-02-10T11:40:41.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:41.687 7f7047fff700 1 -- 172.21.2.112:0/2266652 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f705000c340 con 0x7f70541b0a10 2021-02-10T11:40:41.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:41.687 7f7047fff700 1 -- 172.21.2.112:0/2266652 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f705000c820 con 0x7f70541b0a10 2021-02-10T11:40:41.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:41.687 7f705b5f5700 1 -- 172.21.2.112:0/2266652 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f70541b0a10 msgr2=0x7f70540cf330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:41.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:41.687 7f705b5f5700 1 --2- 172.21.2.112:0/2266652 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f70541b0a10 0x7f70540cf330 secure :-1 s=READY pgs=2101 cs=0 l=1 rev1=1 rx=0x7f7050006700 tx=0x7f7050009b00).stop 2021-02-10T11:40:41.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:41.688 7f705b5f5700 1 -- 172.21.2.112:0/2266652 shutdown_connections 2021-02-10T11:40:41.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:41.688 7f705b5f5700 1 --2- 172.21.2.112:0/2266652 >> v2:172.21.2.120:3300/0 conn(0x7f70540d1c70 0x7f70540cf950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:41.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:41.688 7f705b5f5700 1 --2- 172.21.2.112:0/2266652 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f70541b0a10 0x7f70540cf330 unknown :-1 s=CLOSED pgs=2101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:41.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:41.688 7f705b5f5700 1 --2- 172.21.2.112:0/2266652 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f70540d1870 0x7f70541b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:41.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:41.688 7f705b5f5700 1 -- 172.21.2.112:0/2266652 >> 172.21.2.112:0/2266652 conn(0x7f70540ce450 msgr2=0x7f70540d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:41.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:41.688 7f705b5f5700 1 -- 172.21.2.112:0/2266652 shutdown_connections 2021-02-10T11:40:41.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:41.688 7f705b5f5700 1 -- 172.21.2.112:0/2266652 wait complete. 2021-02-10T11:40:41.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:41.703 7f705b5f5700 1 Processor -- start 2021-02-10T11:40:41.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:41.703 7f705b5f5700 1 -- start start 2021-02-10T11:40:41.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:41.703 7f705b5f5700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f70541b7cf0 0x7f705419cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:41.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:41.703 7f705b5f5700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f70541b80f0 0x7f70541b84f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:41.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:41.703 7f705b5f5700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f705419da20 0x7f705419de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:41.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:41.703 7f705b5f5700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70540a77c0 con 0x7f70541b7cf0 2021-02-10T11:40:41.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:41.703 7f705b5f5700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f705419d4a0 con 0x7f705419da20 2021-02-10T11:40:41.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:41.703 7f705b5f5700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70540954f0 con 0x7f70541b80f0 2021-02-10T11:40:41.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:41.703 7f7059411700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f70541b7cf0 0x7f705419cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:41.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:41.703 7f7059411700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f70541b7cf0 0x7f705419cdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:54896/0 (socket says 172.21.2.112:54896) 2021-02-10T11:40:41.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:41.703 7f7059411700 1 -- 172.21.2.112:0/650656703 learned_addr learned my addr 172.21.2.112:0/650656703 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:40:41.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:41.704 7f7059411700 1 -- 172.21.2.112:0/650656703 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f70541b80f0 msgr2=0x7f70541b84f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:41.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:41.704 7f7059411700 1 --2- 172.21.2.112:0/650656703 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f70541b80f0 0x7f70541b84f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:41.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:41.704 7f7059411700 1 -- 172.21.2.112:0/650656703 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f705419da20 msgr2=0x7f705419de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:41.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:41.704 7f7059411700 1 --2- 172.21.2.112:0/650656703 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f705419da20 0x7f705419de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:41.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:41.704 7f7059411700 1 -- 172.21.2.112:0/650656703 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70541b8d60 con 0x7f70541b7cf0 2021-02-10T11:40:41.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:41.704 7f7059411700 1 --2- 172.21.2.112:0/650656703 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f70541b7cf0 0x7f705419cdb0 secure :-1 s=READY pgs=2102 cs=0 l=1 rev1=1 rx=0x7f704800b5c0 tx=0x7f704800b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:41.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:41.704 7f7045ffb700 1 -- 172.21.2.112:0/650656703 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f704800bd50 con 0x7f70541b7cf0 2021-02-10T11:40:41.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:41.704 7f7045ffb700 1 -- 172.21.2.112:0/650656703 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7048006030 con 0x7f70541b7cf0 2021-02-10T11:40:41.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:41.704 7f7045ffb700 1 -- 172.21.2.112:0/650656703 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f70480054d0 con 0x7f70541b7cf0 2021-02-10T11:40:41.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:41.704 7f705b5f5700 1 -- 172.21.2.112:0/650656703 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70541b8d60 con 0x7f70541b7cf0 2021-02-10T11:40:41.704 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:41.704 7f705b5f5700 1 -- 172.21.2.112:0/650656703 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f705419e730 con 0x7f70541b7cf0 2021-02-10T11:40:41.704 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:41.705 7f7045ffb700 1 -- 172.21.2.112:0/650656703 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7048006210 con 0x7f70541b7cf0 2021-02-10T11:40:41.704 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:41.705 7f7045ffb700 1 --2- 172.21.2.112:0/650656703 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f70340186c0 0x7f703401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:41.704 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:41.705 7f7045ffb700 1 -- 172.21.2.112:0/650656703 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f70480325d0 con 0x7f70541b7cf0 2021-02-10T11:40:41.704 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:41.705 7f7058c10700 1 --2- 172.21.2.112:0/650656703 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f70340186c0 0x7f703401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:41.704 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:41.706 7f7058c10700 1 --2- 172.21.2.112:0/650656703 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f70340186c0 0x7f703401abc0 crc :-1 s=READY pgs=974 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:41.732 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:41.736 7f705b5f5700 1 -- 172.21.2.112:0/650656703 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70540ebc00 con 0x7f70541b7cf0 2021-02-10T11:40:41.734 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:41.739 7f7045ffb700 1 -- 172.21.2.112:0/650656703 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7048000940 con 0x7f70541b7cf0 2021-02-10T11:40:41.860 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:41.864 7f705b5f5700 1 -- 172.21.2.112:0/650656703 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f70540f0b10 con 0x7f70541b7cf0 2021-02-10T11:40:41.861 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:41.865 7f7045ffb700 1 -- 172.21.2.112:0/650656703 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v108) v1 ==== 74+0+14006 (secure 0 0 0) 0x7f70480061f0 con 0x7f70541b7cf0 2021-02-10T11:40:41.861 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:40:41.861 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":108,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:40:35.316267","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:40:32.284968","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":23,"pool_name":"cephfs_data","create_time":"2021-02-10 11:40:33.307261","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:40:41.862 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:41.866 7f705b5f5700 1 -- 172.21.2.112:0/650656703 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f70340186c0 msgr2=0x7f703401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:41.862 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:41.866 7f705b5f5700 1 --2- 172.21.2.112:0/650656703 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f70340186c0 0x7f703401abc0 crc :-1 s=READY pgs=974 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:41.862 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:41.866 7f705b5f5700 1 -- 172.21.2.112:0/650656703 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f70541b7cf0 msgr2=0x7f705419cdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:41.862 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:41.866 7f705b5f5700 1 --2- 172.21.2.112:0/650656703 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f70541b7cf0 0x7f705419cdb0 secure :-1 s=READY pgs=2102 cs=0 l=1 rev1=1 rx=0x7f704800b5c0 tx=0x7f704800b960).stop 2021-02-10T11:40:41.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:41.867 7f705b5f5700 1 -- 172.21.2.112:0/650656703 shutdown_connections 2021-02-10T11:40:41.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:41.867 7f705b5f5700 1 --2- 172.21.2.112:0/650656703 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f70340186c0 0x7f703401abc0 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:41.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:41.867 7f705b5f5700 1 --2- 172.21.2.112:0/650656703 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f705419da20 0x7f705419de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:41.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:41.867 7f705b5f5700 1 --2- 172.21.2.112:0/650656703 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f70541b80f0 0x7f70541b84f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:41.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:41.867 7f705b5f5700 1 --2- 172.21.2.112:0/650656703 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f70541b7cf0 0x7f705419cdb0 unknown :-1 s=CLOSED pgs=2102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:41.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:41.867 7f705b5f5700 1 -- 172.21.2.112:0/650656703 >> 172.21.2.112:0/650656703 conn(0x7f70540ed810 msgr2=0x7f7054094980 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:41.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:41.867 7f705b5f5700 1 -- 172.21.2.112:0/650656703 shutdown_connections 2021-02-10T11:40:41.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:41.867 7f705b5f5700 1 -- 172.21.2.112:0/650656703 wait complete. 2021-02-10T11:40:41.871 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:13:57.508758', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 22, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:40:32.284968', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '107', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 23, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:40:33.307261', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '108', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:40:41.871 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:40:42.303 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.291 7fb03ec16700 1 Processor -- start 2021-02-10T11:40:42.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.291 7fb03ec16700 1 -- start start 2021-02-10T11:40:42.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.292 7fb03ec16700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb0380d1870 0x7fb0381b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:42.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.292 7fb03ec16700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fb0381b0ae0 0x7fb038108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:42.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.292 7fb03ec16700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fb0380d1c70 0x7fb038108f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:42.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.292 7fb03ec16700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0380e5ef0 con 0x7fb0380d1c70 2021-02-10T11:40:42.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.292 7fb03ec16700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb038109a60 con 0x7fb0381b0ae0 2021-02-10T11:40:42.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.292 7fb03ec16700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb038088070 con 0x7fb0380d1870 2021-02-10T11:40:42.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.292 7fb03d233700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fb0380d1c70 0x7fb038108f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:42.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.292 7fb03d233700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fb0380d1c70 0x7fb038108f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:54908/0 (socket says 172.21.2.112:54908) 2021-02-10T11:40:42.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.292 7fb03d233700 1 -- 172.21.2.112:0/4271247486 learned_addr learned my addr 172.21.2.112:0/4271247486 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:40:42.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.292 7fb037fff700 1 --2- 172.21.2.112:0/4271247486 >> v2:172.21.2.120:3300/0 conn(0x7fb0381b0ae0 0x7fb038108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:42.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.292 7fb03ca32700 1 --2- 172.21.2.112:0/4271247486 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb0380d1870 0x7fb0381b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:42.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.292 7fb03d233700 1 -- 172.21.2.112:0/4271247486 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb0380d1870 msgr2=0x7fb0381b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:42.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.292 7fb03d233700 1 --2- 172.21.2.112:0/4271247486 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb0380d1870 0x7fb0381b04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:42.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.292 7fb03d233700 1 -- 172.21.2.112:0/4271247486 >> v2:172.21.2.120:3300/0 conn(0x7fb0381b0ae0 msgr2=0x7fb038108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:42.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.292 7fb03d233700 1 --2- 172.21.2.112:0/4271247486 >> v2:172.21.2.120:3300/0 conn(0x7fb0381b0ae0 0x7fb038108940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:42.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.293 7fb03d233700 1 -- 172.21.2.112:0/4271247486 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb038109580 con 0x7fb0380d1c70 2021-02-10T11:40:42.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.293 7fb03d233700 1 --2- 172.21.2.112:0/4271247486 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb0380d1c70 0x7fb038108f60 secure :-1 s=READY pgs=2103 cs=0 l=1 rev1=1 rx=0x7fb030006f00 tx=0x7fb030009f10).ready entity=mon.0 client_cookie=84b89a10d4354df3 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:42.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.293 7fb0377fe700 1 -- 172.21.2.112:0/4271247486 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb03000b3b0 con 0x7fb0380d1c70 2021-02-10T11:40:42.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.293 7fb0377fe700 1 -- 172.21.2.112:0/4271247486 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb030000d30 con 0x7fb0380d1c70 2021-02-10T11:40:42.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.293 7fb0377fe700 1 -- 172.21.2.112:0/4271247486 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb03000b820 con 0x7fb0380d1c70 2021-02-10T11:40:42.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.293 7fb03ec16700 1 -- 172.21.2.112:0/4271247486 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb0380d1c70 msgr2=0x7fb038108f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:42.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.293 7fb03ec16700 1 --2- 172.21.2.112:0/4271247486 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb0380d1c70 0x7fb038108f60 secure :-1 s=READY pgs=2103 cs=0 l=1 rev1=1 rx=0x7fb030006f00 tx=0x7fb030009f10).stop 2021-02-10T11:40:42.306 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.294 7fb03ec16700 1 -- 172.21.2.112:0/4271247486 shutdown_connections 2021-02-10T11:40:42.306 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.294 7fb03ec16700 1 --2- 172.21.2.112:0/4271247486 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb0380d1c70 0x7fb038108f60 unknown :-1 s=CLOSED pgs=2103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:42.306 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.294 7fb03ec16700 1 --2- 172.21.2.112:0/4271247486 >> v2:172.21.2.120:3300/0 conn(0x7fb0381b0ae0 0x7fb038108940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:42.306 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.294 7fb03ec16700 1 --2- 172.21.2.112:0/4271247486 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb0380d1870 0x7fb0381b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:42.306 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.294 7fb03ec16700 1 -- 172.21.2.112:0/4271247486 >> 172.21.2.112:0/4271247486 conn(0x7fb0380ce420 msgr2=0x7fb0380d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:42.306 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.294 7fb03ec16700 1 -- 172.21.2.112:0/4271247486 shutdown_connections 2021-02-10T11:40:42.306 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.294 7fb03ec16700 1 -- 172.21.2.112:0/4271247486 wait complete. 2021-02-10T11:40:42.307 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.308 7fb03ec16700 1 Processor -- start 2021-02-10T11:40:42.307 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.308 7fb03ec16700 1 -- start start 2021-02-10T11:40:42.307 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.309 7fb03ec16700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb0380abd60 0x7fb0380ac160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:42.308 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.309 7fb03ec16700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb0380ac780 0x7fb0380acb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:42.308 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.309 7fb03ec16700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb038083910 0x7fb038083d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:42.308 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.309 7fb03ec16700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb038087180 con 0x7fb0380abd60 2021-02-10T11:40:42.308 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.309 7fb03ec16700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb03818fcd0 con 0x7fb0380ac780 2021-02-10T11:40:42.308 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.309 7fb03ec16700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb038084330 con 0x7fb038083910 2021-02-10T11:40:42.308 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.309 7fb03ca32700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb0380abd60 0x7fb0380ac160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:42.308 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.309 7fb03ca32700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb0380abd60 0x7fb0380ac160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:54912/0 (socket says 172.21.2.112:54912) 2021-02-10T11:40:42.308 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.309 7fb03ca32700 1 -- 172.21.2.112:0/1073045702 learned_addr learned my addr 172.21.2.112:0/1073045702 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:40:42.308 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.309 7fb03d233700 1 --2- 172.21.2.112:0/1073045702 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb038083910 0x7fb038083d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:42.308 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.309 7fb03ca32700 1 -- 172.21.2.112:0/1073045702 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb038083910 msgr2=0x7fb038083d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:42.308 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.309 7fb03ca32700 1 --2- 172.21.2.112:0/1073045702 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb038083910 0x7fb038083d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:42.308 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.309 7fb03ca32700 1 -- 172.21.2.112:0/1073045702 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb0380ac780 msgr2=0x7fb0380acb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:42.308 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.309 7fb03ca32700 1 --2- 172.21.2.112:0/1073045702 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb0380ac780 0x7fb0380acb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:42.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.309 7fb03ca32700 1 -- 172.21.2.112:0/1073045702 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb038084550 con 0x7fb0380abd60 2021-02-10T11:40:42.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.309 7fb03ca32700 1 --2- 172.21.2.112:0/1073045702 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb0380abd60 0x7fb0380ac160 secure :-1 s=READY pgs=2104 cs=0 l=1 rev1=1 rx=0x7fb02000b580 tx=0x7fb02000b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:42.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.310 7fb0357fa700 1 -- 172.21.2.112:0/1073045702 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb0200064d0 con 0x7fb0380abd60 2021-02-10T11:40:42.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.310 7fb0357fa700 1 -- 172.21.2.112:0/1073045702 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb02000bde0 con 0x7fb0380abd60 2021-02-10T11:40:42.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.310 7fb0357fa700 1 -- 172.21.2.112:0/1073045702 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb020002030 con 0x7fb0380abd60 2021-02-10T11:40:42.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.310 7fb03ec16700 1 -- 172.21.2.112:0/1073045702 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb038084550 con 0x7fb0380abd60 2021-02-10T11:40:42.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.310 7fb03ec16700 1 -- 172.21.2.112:0/1073045702 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb038084ae0 con 0x7fb0380abd60 2021-02-10T11:40:42.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.310 7fb0357fa700 1 -- 172.21.2.112:0/1073045702 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb020000c90 con 0x7fb0380abd60 2021-02-10T11:40:42.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.310 7fb0357fa700 1 --2- 172.21.2.112:0/1073045702 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb01c0186c0 0x7fb01c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:42.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.311 7fb0357fa700 1 -- 172.21.2.112:0/1073045702 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fb020032580 con 0x7fb0380abd60 2021-02-10T11:40:42.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.311 7fb037fff700 1 --2- 172.21.2.112:0/1073045702 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb01c0186c0 0x7fb01c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:42.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.311 7fb037fff700 1 --2- 172.21.2.112:0/1073045702 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb01c0186c0 0x7fb01c01abc0 crc :-1 s=READY pgs=975 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:42.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.341 7fb03ec16700 1 -- 172.21.2.112:0/1073045702 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb038084d90 con 0x7fb0380abd60 2021-02-10T11:40:42.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.345 7fb0357fa700 1 -- 172.21.2.112:0/1073045702 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb020006d00 con 0x7fb0380abd60 2021-02-10T11:40:42.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.475 7fb03ec16700 1 -- 172.21.2.112:0/1073045702 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7fb038084d90 con 0x7fb0380abd60 2021-02-10T11:40:42.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.476 7fb0357fa700 1 -- 172.21.2.112:0/1073045702 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v108) v1 ==== 92+0+10 (secure 0 0 0) 0x7fb020000c90 con 0x7fb0380abd60 2021-02-10T11:40:42.472 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:40:42.473 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.478 7fb03ec16700 1 -- 172.21.2.112:0/1073045702 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb01c0186c0 msgr2=0x7fb01c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:42.473 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.478 7fb03ec16700 1 --2- 172.21.2.112:0/1073045702 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb01c0186c0 0x7fb01c01abc0 crc :-1 s=READY pgs=975 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:42.473 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.478 7fb03ec16700 1 -- 172.21.2.112:0/1073045702 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb0380abd60 msgr2=0x7fb0380ac160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:42.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.478 7fb03ec16700 1 --2- 172.21.2.112:0/1073045702 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb0380abd60 0x7fb0380ac160 secure :-1 s=READY pgs=2104 cs=0 l=1 rev1=1 rx=0x7fb02000b580 tx=0x7fb02000b920).stop 2021-02-10T11:40:42.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.478 7fb03ec16700 1 -- 172.21.2.112:0/1073045702 shutdown_connections 2021-02-10T11:40:42.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.478 7fb03ec16700 1 --2- 172.21.2.112:0/1073045702 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb01c0186c0 0x7fb01c01abc0 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:42.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.478 7fb03ec16700 1 --2- 172.21.2.112:0/1073045702 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb0380ac780 0x7fb0380acb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:42.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.478 7fb03ec16700 1 --2- 172.21.2.112:0/1073045702 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb038083910 0x7fb038083d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:42.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.478 7fb03ec16700 1 --2- 172.21.2.112:0/1073045702 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb0380abd60 0x7fb0380ac160 unknown :-1 s=CLOSED pgs=2104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:42.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.478 7fb03ec16700 1 -- 172.21.2.112:0/1073045702 >> 172.21.2.112:0/1073045702 conn(0x7fb0380a2b30 msgr2=0x7fb03809ce60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:42.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.478 7fb03ec16700 1 -- 172.21.2.112:0/1073045702 shutdown_connections 2021-02-10T11:40:42.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.478 7fb03ec16700 1 -- 172.21.2.112:0/1073045702 wait complete. 2021-02-10T11:40:42.482 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:40:42.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.902 7f5e3fa42700 1 Processor -- start 2021-02-10T11:40:42.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.902 7f5e3fa42700 1 -- start start 2021-02-10T11:40:42.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.902 7f5e3fa42700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f5e380d1c70 0x7f5e380cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:42.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.902 7f5e3fa42700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f5e380cf580 0x7f5e380cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:42.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.902 7f5e3fa42700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e380fe1c0 con 0x7f5e380d1870 2021-02-10T11:40:42.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.902 7f5e3fa42700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e381b0cb0 con 0x7f5e380d1c70 2021-02-10T11:40:42.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.903 7f5e3fa42700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e38109a60 con 0x7f5e380cf580 2021-02-10T11:40:42.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.903 7f5e3d05d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f5e380d1c70 0x7f5e380cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:42.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.903 7f5e3d05d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f5e380d1c70 0x7f5e380cef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:54918/0 (socket says 172.21.2.112:54918) 2021-02-10T11:40:42.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.903 7f5e3d05d700 1 -- 172.21.2.112:0/1993367179 learned_addr learned my addr 172.21.2.112:0/1993367179 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:40:42.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.903 7f5e3e05f700 1 --2- 172.21.2.112:0/1993367179 >> v2:172.21.2.120:3300/0 conn(0x7f5e380cf580 0x7f5e380cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:42.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.903 7f5e3d05d700 1 -- 172.21.2.112:0/1993367179 >> v2:172.21.2.120:3300/0 conn(0x7f5e380cf580 msgr2=0x7f5e380cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:42.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.903 7f5e3d05d700 1 --2- 172.21.2.112:0/1993367179 >> v2:172.21.2.120:3300/0 conn(0x7f5e380cf580 0x7f5e380cf980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:42.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.903 7f5e3d05d700 1 -- 172.21.2.112:0/1993367179 >> v1:172.21.2.120:6789/0 conn(0x7f5e380d1870 legacy=0x7f5e381b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:42.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.903 7f5e3d05d700 1 -- 172.21.2.112:0/1993367179 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e38088070 con 0x7f5e380d1c70 2021-02-10T11:40:42.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.903 7f5e3d05d700 1 --2- 172.21.2.112:0/1993367179 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5e380d1c70 0x7f5e380cef60 secure :-1 s=READY pgs=2105 cs=0 l=1 rev1=1 rx=0x7f5e2c006700 tx=0x7f5e2c009b80).ready entity=mon.0 client_cookie=4eac97518f53c95b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:42.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.904 7f5e3c85c700 1 -- 172.21.2.112:0/1993367179 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5e2c00c0c0 con 0x7f5e380d1c70 2021-02-10T11:40:42.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.904 7f5e3c85c700 1 -- 172.21.2.112:0/1993367179 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e2c00c400 con 0x7f5e380d1c70 2021-02-10T11:40:42.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.904 7f5e3c85c700 1 -- 172.21.2.112:0/1993367179 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5e2c00c920 con 0x7f5e380d1c70 2021-02-10T11:40:42.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.904 7f5e3fa42700 1 -- 172.21.2.112:0/1993367179 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5e380d1c70 msgr2=0x7f5e380cef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:42.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.904 7f5e3fa42700 1 --2- 172.21.2.112:0/1993367179 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5e380d1c70 0x7f5e380cef60 secure :-1 s=READY pgs=2105 cs=0 l=1 rev1=1 rx=0x7f5e2c006700 tx=0x7f5e2c009b80).stop 2021-02-10T11:40:42.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.904 7f5e3fa42700 1 -- 172.21.2.112:0/1993367179 shutdown_connections 2021-02-10T11:40:42.917 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.904 7f5e3fa42700 1 --2- 172.21.2.112:0/1993367179 >> v2:172.21.2.120:3300/0 conn(0x7f5e380cf580 0x7f5e380cf980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:42.917 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.904 7f5e3fa42700 1 --2- 172.21.2.112:0/1993367179 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5e380d1c70 0x7f5e380cef60 unknown :-1 s=CLOSED pgs=2105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:42.917 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.904 7f5e3fa42700 1 -- 172.21.2.112:0/1993367179 >> 172.21.2.112:0/1993367179 conn(0x7f5e380ce420 msgr2=0x7f5e380d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:42.917 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.905 7f5e3fa42700 1 -- 172.21.2.112:0/1993367179 shutdown_connections 2021-02-10T11:40:42.917 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.905 7f5e3fa42700 1 -- 172.21.2.112:0/1993367179 wait complete. 2021-02-10T11:40:42.918 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.919 7f5e3fa42700 1 Processor -- start 2021-02-10T11:40:42.919 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.919 7f5e3fa42700 1 -- start start 2021-02-10T11:40:42.919 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.920 7f5e3fa42700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5e38085bc0 0x7f5e380863e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:42.919 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.920 7f5e3fa42700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5e380954c0 0x7f5e38096300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:42.919 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.920 7f5e3fa42700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5e380958c0 0x7f5e38096920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:42.919 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.920 7f5e3fa42700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e38095d80 con 0x7f5e380954c0 2021-02-10T11:40:42.919 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.920 7f5e3fa42700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e380bfa60 con 0x7f5e380958c0 2021-02-10T11:40:42.919 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.920 7f5e3fa42700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e38085fc0 con 0x7f5e38085bc0 2021-02-10T11:40:42.919 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.920 7f5e3d05d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5e380954c0 0x7f5e38096300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:42.919 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.920 7f5e3d05d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5e380954c0 0x7f5e38096300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:54924/0 (socket says 172.21.2.112:54924) 2021-02-10T11:40:42.919 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.920 7f5e3d05d700 1 -- 172.21.2.112:0/2115800220 learned_addr learned my addr 172.21.2.112:0/2115800220 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:40:42.919 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.920 7f5e3d05d700 1 -- 172.21.2.112:0/2115800220 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5e38085bc0 msgr2=0x7f5e380863e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:42.919 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.920 7f5e3d05d700 1 --2- 172.21.2.112:0/2115800220 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5e38085bc0 0x7f5e380863e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:42.919 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.920 7f5e3d05d700 1 -- 172.21.2.112:0/2115800220 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5e380958c0 msgr2=0x7f5e38096920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:42.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.920 7f5e3d05d700 1 --2- 172.21.2.112:0/2115800220 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5e380958c0 0x7f5e38096920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:42.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.920 7f5e3d05d700 1 -- 172.21.2.112:0/2115800220 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e38086c10 con 0x7f5e380954c0 2021-02-10T11:40:42.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.920 7f5e3d05d700 1 --2- 172.21.2.112:0/2115800220 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5e380954c0 0x7f5e38096300 secure :-1 s=READY pgs=2106 cs=0 l=1 rev1=1 rx=0x7f5e2c0066a0 tx=0x7f5e2c00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:42.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.920 7f5e2a7fc700 1 -- 172.21.2.112:0/2115800220 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5e2c00bd80 con 0x7f5e380954c0 2021-02-10T11:40:42.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.921 7f5e2a7fc700 1 -- 172.21.2.112:0/2115800220 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e2c0052e0 con 0x7f5e380954c0 2021-02-10T11:40:42.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.921 7f5e2a7fc700 1 -- 172.21.2.112:0/2115800220 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5e2c0057c0 con 0x7f5e380954c0 2021-02-10T11:40:42.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.921 7f5e3fa42700 1 -- 172.21.2.112:0/2115800220 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e38086c10 con 0x7f5e380954c0 2021-02-10T11:40:42.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.921 7f5e3fa42700 1 -- 172.21.2.112:0/2115800220 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e38086c10 con 0x7f5e380954c0 2021-02-10T11:40:42.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.921 7f5e2a7fc700 1 -- 172.21.2.112:0/2115800220 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5e2c006240 con 0x7f5e380954c0 2021-02-10T11:40:42.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.921 7f5e2a7fc700 1 --2- 172.21.2.112:0/2115800220 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5e200186c0 0x7f5e2001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:42.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.921 7f5e2a7fc700 1 -- 172.21.2.112:0/2115800220 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f5e2c032640 con 0x7f5e380954c0 2021-02-10T11:40:42.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.922 7f5e3d85e700 1 --2- 172.21.2.112:0/2115800220 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5e200186c0 0x7f5e2001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:42.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.922 7f5e3d85e700 1 --2- 172.21.2.112:0/2115800220 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5e200186c0 0x7f5e2001abc0 crc :-1 s=READY pgs=976 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:42.948 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.952 7f5e3fa42700 1 -- 172.21.2.112:0/2115800220 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e380fe1c0 con 0x7f5e380954c0 2021-02-10T11:40:42.950 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:42.955 7f5e2a7fc700 1 -- 172.21.2.112:0/2115800220 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5e2c006030 con 0x7f5e380954c0 2021-02-10T11:40:43.078 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:43.082 7f5e3fa42700 1 -- 172.21.2.112:0/2115800220 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f5e380fe1c0 con 0x7f5e380954c0 2021-02-10T11:40:43.079 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:43.083 7f5e2a7fc700 1 -- 172.21.2.112:0/2115800220 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v108) v1 ==== 104+0+10 (secure 0 0 0) 0x7f5e2c00bd80 con 0x7f5e380954c0 2021-02-10T11:40:43.079 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:40:43.081 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:43.085 7f5e3fa42700 1 -- 172.21.2.112:0/2115800220 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5e200186c0 msgr2=0x7f5e2001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:43.081 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:43.085 7f5e3fa42700 1 --2- 172.21.2.112:0/2115800220 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5e200186c0 0x7f5e2001abc0 crc :-1 s=READY pgs=976 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:43.081 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:43.085 7f5e3fa42700 1 -- 172.21.2.112:0/2115800220 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5e380954c0 msgr2=0x7f5e38096300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:43.081 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:43.085 7f5e3fa42700 1 --2- 172.21.2.112:0/2115800220 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5e380954c0 0x7f5e38096300 secure :-1 s=READY pgs=2106 cs=0 l=1 rev1=1 rx=0x7f5e2c0066a0 tx=0x7f5e2c00b990).stop 2021-02-10T11:40:43.081 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:43.085 7f5e3fa42700 1 -- 172.21.2.112:0/2115800220 shutdown_connections 2021-02-10T11:40:43.081 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:43.085 7f5e3fa42700 1 --2- 172.21.2.112:0/2115800220 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5e200186c0 0x7f5e2001abc0 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:43.081 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:43.085 7f5e3fa42700 1 --2- 172.21.2.112:0/2115800220 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5e380958c0 0x7f5e38096920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:43.081 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:43.085 7f5e3fa42700 1 --2- 172.21.2.112:0/2115800220 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5e380954c0 0x7f5e38096300 unknown :-1 s=CLOSED pgs=2106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:43.081 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:43.085 7f5e3fa42700 1 --2- 172.21.2.112:0/2115800220 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5e38085bc0 0x7f5e380863e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:43.082 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:43.085 7f5e3fa42700 1 -- 172.21.2.112:0/2115800220 >> 172.21.2.112:0/2115800220 conn(0x7f5e380bef00 msgr2=0x7f5e3809d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:43.082 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:43.085 7f5e3fa42700 1 -- 172.21.2.112:0/2115800220 shutdown_connections 2021-02-10T11:40:43.082 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:43.085 7f5e3fa42700 1 -- 172.21.2.112:0/2115800220 wait complete. 2021-02-10T11:40:43.089 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:40:43.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:43.509 7f1a91611700 1 Processor -- start 2021-02-10T11:40:43.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:43.509 7f1a91611700 1 -- start start 2021-02-10T11:40:43.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:43.510 7f1a91611700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f1a8c0d1c70 0x7f1a8c0cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:43.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:43.510 7f1a91611700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a8c0d0170 con 0x7f1a8c0d1870 2021-02-10T11:40:43.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:43.510 7f1a91611700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a8c0fe1c0 con 0x7f1a8c0cf580 2021-02-10T11:40:43.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:43.510 7f1a91611700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a8c1b0cb0 con 0x7f1a8c0d1c70 2021-02-10T11:40:43.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:43.510 7f1a8aabe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f1a8c0d1c70 0x7f1a8c0cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:43.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:43.510 7f1a8b2bf700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f1a8c0d1870 0x7f1a8c1b04c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:54394/0 (socket says 172.21.2.112:54394) 2021-02-10T11:40:43.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:43.510 7f1a8b2bf700 1 -- 172.21.2.112:0/2789087291 learned_addr learned my addr 172.21.2.112:0/2789087291 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:40:43.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:43.510 7f1a8aabe700 1 -- 172.21.2.112:0/2789087291 >> v1:172.21.2.120:6789/0 conn(0x7f1a8c0cf580 legacy=0x7f1a8c0cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:43.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:43.510 7f1a8aabe700 1 -- 172.21.2.112:0/2789087291 >> v1:172.21.2.112:6789/0 conn(0x7f1a8c0d1870 legacy=0x7f1a8c1b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:43.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:43.510 7f1a8aabe700 1 -- 172.21.2.112:0/2789087291 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a8c109a60 con 0x7f1a8c0d1c70 2021-02-10T11:40:43.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:43.511 7f1a8aabe700 1 --2- 172.21.2.112:0/2789087291 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1a8c0d1c70 0x7f1a8c0cef60 secure :-1 s=READY pgs=2108 cs=0 l=1 rev1=1 rx=0x7f1a7c006650 tx=0x7f1a7c009a80).ready entity=mon.0 client_cookie=68917a6babb41914 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:43.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:43.511 7f1a8a2bd700 1 -- 172.21.2.112:0/2789087291 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1a7c0009b0 con 0x7f1a8c0d1c70 2021-02-10T11:40:43.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:43.511 7f1a8a2bd700 1 -- 172.21.2.112:0/2789087291 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a7c00c0c0 con 0x7f1a8c0d1c70 2021-02-10T11:40:43.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:43.511 7f1a8a2bd700 1 -- 172.21.2.112:0/2789087291 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1a7c00c550 con 0x7f1a8c0d1c70 2021-02-10T11:40:43.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:43.511 7f1a91611700 1 -- 172.21.2.112:0/2789087291 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1a8c0d1c70 msgr2=0x7f1a8c0cef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:43.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:43.511 7f1a91611700 1 --2- 172.21.2.112:0/2789087291 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1a8c0d1c70 0x7f1a8c0cef60 secure :-1 s=READY pgs=2108 cs=0 l=1 rev1=1 rx=0x7f1a7c006650 tx=0x7f1a7c009a80).stop 2021-02-10T11:40:43.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:43.511 7f1a91611700 1 -- 172.21.2.112:0/2789087291 shutdown_connections 2021-02-10T11:40:43.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:43.511 7f1a91611700 1 --2- 172.21.2.112:0/2789087291 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1a8c0d1c70 0x7f1a8c0cef60 unknown :-1 s=CLOSED pgs=2108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:43.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:43.511 7f1a91611700 1 -- 172.21.2.112:0/2789087291 >> 172.21.2.112:0/2789087291 conn(0x7f1a8c0ce420 msgr2=0x7f1a8c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:43.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:43.512 7f1a91611700 1 -- 172.21.2.112:0/2789087291 shutdown_connections 2021-02-10T11:40:43.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:43.512 7f1a91611700 1 -- 172.21.2.112:0/2789087291 wait complete. 2021-02-10T11:40:43.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:43.526 7f1a91611700 1 Processor -- start 2021-02-10T11:40:43.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:43.526 7f1a91611700 1 -- start start 2021-02-10T11:40:43.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:43.527 7f1a91611700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1a8c105400 0x7f1a8c105800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:43.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:43.527 7f1a91611700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1a8c099920 0x7f1a8c099d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:43.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:43.527 7f1a91611700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1a8c0bc300 0x7f1a8c0bc700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:43.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:43.527 7f1a91611700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a8c099420 con 0x7f1a8c099920 2021-02-10T11:40:43.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:43.527 7f1a91611700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a8c0c4060 con 0x7f1a8c0bc300 2021-02-10T11:40:43.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:43.527 7f1a91611700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a8c0bcd20 con 0x7f1a8c105400 2021-02-10T11:40:43.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:43.527 7f1a8aabe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1a8c099920 0x7f1a8c099d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:43.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:43.527 7f1a8aabe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1a8c099920 0x7f1a8c099d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:54942/0 (socket says 172.21.2.112:54942) 2021-02-10T11:40:43.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:43.527 7f1a8aabe700 1 -- 172.21.2.112:0/1267174656 learned_addr learned my addr 172.21.2.112:0/1267174656 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:40:43.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:43.527 7f1a8aabe700 1 -- 172.21.2.112:0/1267174656 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1a8c105400 msgr2=0x7f1a8c105800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:43.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:43.527 7f1a8aabe700 1 --2- 172.21.2.112:0/1267174656 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1a8c105400 0x7f1a8c105800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:43.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:43.527 7f1a8aabe700 1 -- 172.21.2.112:0/1267174656 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1a8c0bc300 msgr2=0x7f1a8c0bc700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:43.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:43.527 7f1a8aabe700 1 --2- 172.21.2.112:0/1267174656 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1a8c0bc300 0x7f1a8c0bc700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:43.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:43.527 7f1a8aabe700 1 -- 172.21.2.112:0/1267174656 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a8c0bcf40 con 0x7f1a8c099920 2021-02-10T11:40:43.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:43.527 7f1a8aabe700 1 --2- 172.21.2.112:0/1267174656 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1a8c099920 0x7f1a8c099d20 secure :-1 s=READY pgs=2109 cs=0 l=1 rev1=1 rx=0x7f1a7c009630 tx=0x7f1a7c00b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:43.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:43.527 7f1a83fff700 1 -- 172.21.2.112:0/1267174656 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1a7c0093a0 con 0x7f1a8c099920 2021-02-10T11:40:43.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:43.528 7f1a83fff700 1 -- 172.21.2.112:0/1267174656 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a7c00bd70 con 0x7f1a8c099920 2021-02-10T11:40:43.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:43.528 7f1a91611700 1 -- 172.21.2.112:0/1267174656 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a8c0bcf40 con 0x7f1a8c099920 2021-02-10T11:40:43.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:43.528 7f1a83fff700 1 -- 172.21.2.112:0/1267174656 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1a7c005560 con 0x7f1a8c099920 2021-02-10T11:40:43.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:43.528 7f1a91611700 1 -- 172.21.2.112:0/1267174656 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a8c0bd4f0 con 0x7f1a8c099920 2021-02-10T11:40:43.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:43.528 7f1a83fff700 1 -- 172.21.2.112:0/1267174656 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1a7c00a000 con 0x7f1a8c099920 2021-02-10T11:40:43.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:43.528 7f1a83fff700 1 --2- 172.21.2.112:0/1267174656 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1a700186c0 0x7f1a7001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:43.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:43.528 7f1a83fff700 1 -- 172.21.2.112:0/1267174656 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f1a7c032650 con 0x7f1a8c099920 2021-02-10T11:40:43.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:43.529 7f1a8b2bf700 1 --2- 172.21.2.112:0/1267174656 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1a700186c0 0x7f1a7001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:43.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:43.529 7f1a8b2bf700 1 --2- 172.21.2.112:0/1267174656 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1a700186c0 0x7f1a7001abc0 crc :-1 s=READY pgs=977 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:43.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:43.560 7f1a91611700 1 -- 172.21.2.112:0/1267174656 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a8c0807d0 con 0x7f1a8c099920 2021-02-10T11:40:43.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:43.563 7f1a83fff700 1 -- 172.21.2.112:0/1267174656 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1a7c009340 con 0x7f1a8c099920 2021-02-10T11:40:43.686 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:43.690 7f1a91611700 1 -- 172.21.2.112:0/1267174656 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f1a8c088070 con 0x7f1a8c099920 2021-02-10T11:40:43.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:43.691 7f1a83fff700 1 -- 172.21.2.112:0/1267174656 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v108) v1 ==== 100+0+10 (secure 0 0 0) 0x7f1a7c00bd70 con 0x7f1a8c099920 2021-02-10T11:40:43.687 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:40:43.689 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:43.693 7f1a91611700 1 -- 172.21.2.112:0/1267174656 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1a700186c0 msgr2=0x7f1a7001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:43.689 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:43.693 7f1a91611700 1 --2- 172.21.2.112:0/1267174656 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1a700186c0 0x7f1a7001abc0 crc :-1 s=READY pgs=977 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:43.689 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:43.693 7f1a91611700 1 -- 172.21.2.112:0/1267174656 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1a8c099920 msgr2=0x7f1a8c099d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:43.689 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:43.693 7f1a91611700 1 --2- 172.21.2.112:0/1267174656 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1a8c099920 0x7f1a8c099d20 secure :-1 s=READY pgs=2109 cs=0 l=1 rev1=1 rx=0x7f1a7c009630 tx=0x7f1a7c00b980).stop 2021-02-10T11:40:43.689 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:43.693 7f1a91611700 1 -- 172.21.2.112:0/1267174656 shutdown_connections 2021-02-10T11:40:43.689 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:43.693 7f1a91611700 1 --2- 172.21.2.112:0/1267174656 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1a700186c0 0x7f1a7001abc0 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:43.689 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:43.693 7f1a91611700 1 --2- 172.21.2.112:0/1267174656 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1a8c0bc300 0x7f1a8c0bc700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:43.689 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:43.693 7f1a91611700 1 --2- 172.21.2.112:0/1267174656 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1a8c099920 0x7f1a8c099d20 unknown :-1 s=CLOSED pgs=2109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:43.689 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:43.693 7f1a91611700 1 --2- 172.21.2.112:0/1267174656 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1a8c105400 0x7f1a8c105800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:43.689 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:43.693 7f1a91611700 1 -- 172.21.2.112:0/1267174656 >> 172.21.2.112:0/1267174656 conn(0x7f1a8c104fb0 msgr2=0x7f1a8c0bea50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:43.690 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:43.693 7f1a91611700 1 -- 172.21.2.112:0/1267174656 shutdown_connections 2021-02-10T11:40:43.690 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:43.693 7f1a91611700 1 -- 172.21.2.112:0/1267174656 wait complete. 2021-02-10T11:40:43.698 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:40:43.698 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:40:44.127 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:44.114 7fcb52280700 1 Processor -- start 2021-02-10T11:40:44.127 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:44.114 7fcb52280700 1 -- start start 2021-02-10T11:40:44.127 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:44.115 7fcb52280700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fcb4c0d1870 0x7fcb4c1b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:44.127 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:44.115 7fcb52280700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcb4c1b0a10 0x7fcb4c0cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:44.127 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:44.115 7fcb52280700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fcb4c0d1c70 0x7fcb4c0cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:44.127 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:44.115 7fcb52280700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb4c0fe200 con 0x7fcb4c0d1c70 2021-02-10T11:40:44.127 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:44.115 7fcb52280700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb4c0e5f10 con 0x7fcb4c0d1870 2021-02-10T11:40:44.127 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:44.115 7fcb52280700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb4c0f0b50 con 0x7fcb4c1b0a10 2021-02-10T11:40:44.127 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:44.115 7fcb5089d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fcb4c0d1c70 0x7fcb4c0cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:44.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:44.115 7fcb5089d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fcb4c0d1c70 0x7fcb4c0cf950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:54946/0 (socket says 172.21.2.112:54946) 2021-02-10T11:40:44.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:44.115 7fcb5089d700 1 -- 172.21.2.112:0/984866410 learned_addr learned my addr 172.21.2.112:0/984866410 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:40:44.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:44.115 7fcb4bfff700 1 --2- 172.21.2.112:0/984866410 >> v2:172.21.2.120:3300/0 conn(0x7fcb4c0d1870 0x7fcb4c1b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:44.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:44.115 7fcb4b7fe700 1 --2- 172.21.2.112:0/984866410 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcb4c1b0a10 0x7fcb4c0cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:44.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:44.115 7fcb5089d700 1 -- 172.21.2.112:0/984866410 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcb4c1b0a10 msgr2=0x7fcb4c0cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:44.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:44.115 7fcb5089d700 1 --2- 172.21.2.112:0/984866410 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcb4c1b0a10 0x7fcb4c0cf330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:44.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:44.115 7fcb5089d700 1 -- 172.21.2.112:0/984866410 >> v2:172.21.2.120:3300/0 conn(0x7fcb4c0d1870 msgr2=0x7fcb4c1b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:44.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:44.115 7fcb5089d700 1 --2- 172.21.2.112:0/984866410 >> v2:172.21.2.120:3300/0 conn(0x7fcb4c0d1870 0x7fcb4c1b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:44.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:44.115 7fcb5089d700 1 -- 172.21.2.112:0/984866410 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb4c0cff70 con 0x7fcb4c0d1c70 2021-02-10T11:40:44.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:44.116 7fcb5089d700 1 --2- 172.21.2.112:0/984866410 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcb4c0d1c70 0x7fcb4c0cf950 secure :-1 s=READY pgs=2110 cs=0 l=1 rev1=1 rx=0x7fcb44006f00 tx=0x7fcb44009f10).ready entity=mon.0 client_cookie=f5566881a27847b9 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:44.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:44.116 7fcb4affd700 1 -- 172.21.2.112:0/984866410 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcb4400b3b0 con 0x7fcb4c0d1c70 2021-02-10T11:40:44.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:44.116 7fcb4affd700 1 -- 172.21.2.112:0/984866410 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb4400b630 con 0x7fcb4c0d1c70 2021-02-10T11:40:44.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:44.116 7fcb4affd700 1 -- 172.21.2.112:0/984866410 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcb4400bb50 con 0x7fcb4c0d1c70 2021-02-10T11:40:44.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:44.116 7fcb52280700 1 -- 172.21.2.112:0/984866410 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcb4c0d1c70 msgr2=0x7fcb4c0cf950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:44.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:44.116 7fcb52280700 1 --2- 172.21.2.112:0/984866410 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcb4c0d1c70 0x7fcb4c0cf950 secure :-1 s=READY pgs=2110 cs=0 l=1 rev1=1 rx=0x7fcb44006f00 tx=0x7fcb44009f10).stop 2021-02-10T11:40:44.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:44.116 7fcb52280700 1 -- 172.21.2.112:0/984866410 shutdown_connections 2021-02-10T11:40:44.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:44.116 7fcb52280700 1 --2- 172.21.2.112:0/984866410 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcb4c0d1c70 0x7fcb4c0cf950 unknown :-1 s=CLOSED pgs=2110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:44.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:44.116 7fcb52280700 1 --2- 172.21.2.112:0/984866410 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcb4c1b0a10 0x7fcb4c0cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:44.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:44.116 7fcb52280700 1 --2- 172.21.2.112:0/984866410 >> v2:172.21.2.120:3300/0 conn(0x7fcb4c0d1870 0x7fcb4c1b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:44.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:44.116 7fcb52280700 1 -- 172.21.2.112:0/984866410 >> 172.21.2.112:0/984866410 conn(0x7fcb4c0ce450 msgr2=0x7fcb4c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:44.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:44.116 7fcb52280700 1 -- 172.21.2.112:0/984866410 shutdown_connections 2021-02-10T11:40:44.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:44.116 7fcb52280700 1 -- 172.21.2.112:0/984866410 wait complete. 2021-02-10T11:40:44.131 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:44.131 7fcb52280700 1 Processor -- start 2021-02-10T11:40:44.131 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:44.131 7fcb52280700 1 -- start start 2021-02-10T11:40:44.131 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:44.132 7fcb52280700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcb4c1942b0 0x7fcb4c0b3550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:44.131 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:44.132 7fcb52280700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcb4c195070 0x7fcb4c195470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:44.131 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:44.132 7fcb52280700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcb4c0bb8e0 0x7fcb4c0bbce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:44.131 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:44.132 7fcb52280700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb4c1946b0 con 0x7fcb4c195070 2021-02-10T11:40:44.131 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:44.132 7fcb52280700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb4c0b3b70 con 0x7fcb4c1942b0 2021-02-10T11:40:44.131 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:44.132 7fcb52280700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb4c0b3d90 con 0x7fcb4c0bb8e0 2021-02-10T11:40:44.131 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:44.132 7fcb4b7fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcb4c195070 0x7fcb4c195470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:44.131 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:44.132 7fcb4b7fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcb4c195070 0x7fcb4c195470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:54950/0 (socket says 172.21.2.112:54950) 2021-02-10T11:40:44.131 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:44.132 7fcb4b7fe700 1 -- 172.21.2.112:0/880911357 learned_addr learned my addr 172.21.2.112:0/880911357 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:40:44.131 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:44.132 7fcb4b7fe700 1 -- 172.21.2.112:0/880911357 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcb4c0bb8e0 msgr2=0x7fcb4c0bbce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:44.132 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:44.132 7fcb4b7fe700 1 --2- 172.21.2.112:0/880911357 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcb4c0bb8e0 0x7fcb4c0bbce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:44.132 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:44.132 7fcb4b7fe700 1 -- 172.21.2.112:0/880911357 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcb4c1942b0 msgr2=0x7fcb4c0b3550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:44.132 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:44.132 7fcb4b7fe700 1 --2- 172.21.2.112:0/880911357 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcb4c1942b0 0x7fcb4c0b3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:44.132 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:44.132 7fcb4b7fe700 1 -- 172.21.2.112:0/880911357 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb4c1a49e0 con 0x7fcb4c195070 2021-02-10T11:40:44.132 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:44.132 7fcb4b7fe700 1 --2- 172.21.2.112:0/880911357 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcb4c195070 0x7fcb4c195470 secure :-1 s=READY pgs=2111 cs=0 l=1 rev1=1 rx=0x7fcb38006f00 tx=0x7fcb3800b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:44.132 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:44.133 7fcb48ff9700 1 -- 172.21.2.112:0/880911357 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcb3800bd30 con 0x7fcb4c195070 2021-02-10T11:40:44.132 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:44.133 7fcb48ff9700 1 -- 172.21.2.112:0/880911357 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb38002030 con 0x7fcb4c195070 2021-02-10T11:40:44.132 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:44.133 7fcb48ff9700 1 -- 172.21.2.112:0/880911357 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcb38002550 con 0x7fcb4c195070 2021-02-10T11:40:44.132 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:44.133 7fcb52280700 1 -- 172.21.2.112:0/880911357 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb4c1a49e0 con 0x7fcb4c195070 2021-02-10T11:40:44.132 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:44.133 7fcb52280700 1 -- 172.21.2.112:0/880911357 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb4c0bc5f0 con 0x7fcb4c195070 2021-02-10T11:40:44.132 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:44.133 7fcb48ff9700 1 -- 172.21.2.112:0/880911357 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fcb38000c90 con 0x7fcb4c195070 2021-02-10T11:40:44.132 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:44.133 7fcb48ff9700 1 --2- 172.21.2.112:0/880911357 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcb280186c0 0x7fcb2801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:44.132 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:44.133 7fcb48ff9700 1 -- 172.21.2.112:0/880911357 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fcb380324f0 con 0x7fcb4c195070 2021-02-10T11:40:44.132 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:44.134 7fcb4bfff700 1 --2- 172.21.2.112:0/880911357 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcb280186c0 0x7fcb2801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:44.133 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:44.134 7fcb4bfff700 1 --2- 172.21.2.112:0/880911357 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcb280186c0 0x7fcb2801abc0 crc :-1 s=READY pgs=978 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:44.160 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:44.164 7fcb52280700 1 -- 172.21.2.112:0/880911357 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb4c0bc5f0 con 0x7fcb4c195070 2021-02-10T11:40:44.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:44.167 7fcb48ff9700 1 -- 172.21.2.112:0/880911357 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fcb38000960 con 0x7fcb4c195070 2021-02-10T11:40:44.291 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:44.295 7fcb52280700 1 -- 172.21.2.112:0/880911357 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcb4c0bc5f0 con 0x7fcb4c195070 2021-02-10T11:40:44.292 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:44.296 7fcb48ff9700 1 -- 172.21.2.112:0/880911357 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+2107 (secure 0 0 0) 0x7fcb3800bd30 con 0x7fcb4c195070 2021-02-10T11:40:44.292 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:40:44.292 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":88,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10335,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/4159879144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":4159879144},{"type":"v1","addr":"172.21.2.112:6833","nonce":4159879144}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":84}],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:40:34.314832","modified":"2021-02-10 11:40:36.323257","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10332},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10332":{"gid":10332,"name":"b","rank":0,"incarnation":85,"state":"up:active","state_seq":3,"addr":"172.21.2.120:6835/3723311838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":3723311838},{"type":"v1","addr":"172.21.2.120:6835","nonce":3723311838}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":11}]} 2021-02-10T11:40:44.292 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 88 2021-02-10T11:40:44.293 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:44.297 7fcb52280700 1 -- 172.21.2.112:0/880911357 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcb280186c0 msgr2=0x7fcb2801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:44.293 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:44.297 7fcb52280700 1 --2- 172.21.2.112:0/880911357 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcb280186c0 0x7fcb2801abc0 crc :-1 s=READY pgs=978 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:44.293 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:44.297 7fcb52280700 1 -- 172.21.2.112:0/880911357 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcb4c195070 msgr2=0x7fcb4c195470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:44.293 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:44.297 7fcb52280700 1 --2- 172.21.2.112:0/880911357 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcb4c195070 0x7fcb4c195470 secure :-1 s=READY pgs=2111 cs=0 l=1 rev1=1 rx=0x7fcb38006f00 tx=0x7fcb3800b940).stop 2021-02-10T11:40:44.293 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:44.297 7fcb52280700 1 -- 172.21.2.112:0/880911357 shutdown_connections 2021-02-10T11:40:44.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:44.298 7fcb52280700 1 --2- 172.21.2.112:0/880911357 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcb280186c0 0x7fcb2801abc0 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:44.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:44.298 7fcb52280700 1 --2- 172.21.2.112:0/880911357 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcb4c0bb8e0 0x7fcb4c0bbce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:44.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:44.298 7fcb52280700 1 --2- 172.21.2.112:0/880911357 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcb4c195070 0x7fcb4c195470 unknown :-1 s=CLOSED pgs=2111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:44.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:44.298 7fcb52280700 1 --2- 172.21.2.112:0/880911357 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcb4c1942b0 0x7fcb4c0b3550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:44.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:44.298 7fcb52280700 1 -- 172.21.2.112:0/880911357 >> 172.21.2.112:0/880911357 conn(0x7fcb4c0b3040 msgr2=0x7fcb4c0c3250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:44.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:44.298 7fcb52280700 1 -- 172.21.2.112:0/880911357 shutdown_connections 2021-02-10T11:40:44.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:44.298 7fcb52280700 1 -- 172.21.2.112:0/880911357 wait complete. 2021-02-10T11:40:44.503 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 88, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:40:34.314832', 'modified': '2021-02-10 11:40:36.323257', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 10332}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10332': {'gid': 10332, 'name': 'b', 'rank': 0, 'incarnation': 85, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.120:6835/3723311838', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.120:6834', 'nonce': 3723311838}, {'type': 'v1', 'addr': '172.21.2.120:6835', 'nonce': 3723311838}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [23], 'metadata_pool': 22, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:40:44.504 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:40:44.504 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:40:44.901 INFO:teuthology.orchestra.run.gibba020.stdout:{ 2021-02-10T11:40:44.902 INFO:teuthology.orchestra.run.gibba020.stdout: "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", 2021-02-10T11:40:44.902 INFO:teuthology.orchestra.run.gibba020.stdout: "whoami": 0, 2021-02-10T11:40:44.902 INFO:teuthology.orchestra.run.gibba020.stdout: "id": 10332, 2021-02-10T11:40:44.902 INFO:teuthology.orchestra.run.gibba020.stdout: "want_state": "up:active", 2021-02-10T11:40:44.902 INFO:teuthology.orchestra.run.gibba020.stdout: "state": "up:active", 2021-02-10T11:40:44.903 INFO:teuthology.orchestra.run.gibba020.stdout: "rank_uptime": 10.590093871000001, 2021-02-10T11:40:44.903 INFO:teuthology.orchestra.run.gibba020.stdout: "mdsmap_epoch": 88, 2021-02-10T11:40:44.903 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch": 108, 2021-02-10T11:40:44.903 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch_barrier": 108, 2021-02-10T11:40:44.903 INFO:teuthology.orchestra.run.gibba020.stdout: "uptime": 16.233488376 2021-02-10T11:40:44.903 INFO:teuthology.orchestra.run.gibba020.stdout:} 2021-02-10T11:40:44.905 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", "whoami": 0, "id": 10332, "want_state": "up:active", "state": "up:active", "rank_uptime": 10.590093871000001, "mdsmap_epoch": 88, "osdmap_epoch": 108, "osdmap_epoch_barrier": 108, "uptime": 16.233488376 } 2021-02-10T11:40:44.906 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:40:44.906 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2021-02-10T11:40:44.906 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@gibba012.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2021-02-10T11:40:44.906 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:40:44.906 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:40:45.243 DEBUG:teuthology.orchestra.run.gibba012:> sudo modprobe fuse 2021-02-10T11:40:45.586 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:40:45.587 DEBUG:teuthology.orchestra.run.gibba012:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:40:45.931 INFO:teuthology.orchestra.run.gibba012.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:40:45.931 INFO:teuthology.orchestra.run.gibba012.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:40:45.932 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:40:45.932 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:40:45.932 DEBUG:teuthology.orchestra.run.gibba012:> ls /sys/fs/fuse/connections 2021-02-10T11:40:46.267 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2021-02-10T11:40:46.267 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:40:46.271 DEBUG:teuthology.orchestra.run.gibba012:> sudo modprobe fuse 2021-02-10T11:40:46.614 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:40:46.614 DEBUG:teuthology.orchestra.run.gibba012:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:40:46.680 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba012.stderr:ceph-fuse[157734]: starting ceph client 2021-02-10T11:40:46.680 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba012.stderr:2021-02-10 11:40:46.684 7f4388fc0f80 -1 init, newargv = 0x55d953bf71a0 newargc=9 2021-02-10T11:40:46.686 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba012.stderr:ceph-fuse[157734]: starting fuse 2021-02-10T11:40:46.956 INFO:teuthology.orchestra.run.gibba012.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:40:46.956 INFO:teuthology.orchestra.run.gibba012.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:40:46.957 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:40:46.957 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:40:46.958 DEBUG:teuthology.orchestra.run.gibba012:> ls /sys/fs/fuse/connections 2021-02-10T11:40:47.293 INFO:teuthology.orchestra.run.gibba012.stdout:42 2021-02-10T11:40:47.293 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2021-02-10T11:40:47.294 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:40:47.294 DEBUG:teuthology.orchestra.run.gibba012:> sudo python3 -c ' 2021-02-10T11:40:47.294 DEBUG:teuthology.orchestra.run.gibba012:> import glob 2021-02-10T11:40:47.294 DEBUG:teuthology.orchestra.run.gibba012:> import re 2021-02-10T11:40:47.294 DEBUG:teuthology.orchestra.run.gibba012:> import os 2021-02-10T11:40:47.294 DEBUG:teuthology.orchestra.run.gibba012:> import subprocess 2021-02-10T11:40:47.294 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:40:47.294 DEBUG:teuthology.orchestra.run.gibba012:> def find_socket(client_name): 2021-02-10T11:40:47.294 DEBUG:teuthology.orchestra.run.gibba012:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2021-02-10T11:40:47.294 DEBUG:teuthology.orchestra.run.gibba012:> files = glob.glob(asok_path) 2021-02-10T11:40:47.294 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:40:47.294 DEBUG:teuthology.orchestra.run.gibba012:> # Given a non-glob path, it better be there 2021-02-10T11:40:47.295 DEBUG:teuthology.orchestra.run.gibba012:> if "*" not in asok_path: 2021-02-10T11:40:47.295 DEBUG:teuthology.orchestra.run.gibba012:> assert(len(files) == 1) 2021-02-10T11:40:47.295 DEBUG:teuthology.orchestra.run.gibba012:> return files[0] 2021-02-10T11:40:47.295 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:40:47.295 DEBUG:teuthology.orchestra.run.gibba012:> for f in files: 2021-02-10T11:40:47.295 DEBUG:teuthology.orchestra.run.gibba012:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:40:47.295 DEBUG:teuthology.orchestra.run.gibba012:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:40:47.295 DEBUG:teuthology.orchestra.run.gibba012:> return f 2021-02-10T11:40:47.295 DEBUG:teuthology.orchestra.run.gibba012:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:40:47.295 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:40:47.295 DEBUG:teuthology.orchestra.run.gibba012:> print(find_socket("client.0")) 2021-02-10T11:40:47.295 DEBUG:teuthology.orchestra.run.gibba012:> ' 2021-02-10T11:40:47.657 INFO:teuthology.orchestra.run.gibba012.stdout:/var/run/ceph/ceph-client.0.157734.asok 2021-02-10T11:40:47.660 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.157734.asok 2021-02-10T11:40:47.660 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:40:47.660 DEBUG:teuthology.orchestra.run.gibba012:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.157734.asok status 2021-02-10T11:40:48.050 INFO:teuthology.orchestra.run.gibba012.stdout:{ 2021-02-10T11:40:48.051 INFO:teuthology.orchestra.run.gibba012.stdout: "metadata": { 2021-02-10T11:40:48.051 INFO:teuthology.orchestra.run.gibba012.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:40:48.051 INFO:teuthology.orchestra.run.gibba012.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:40:48.051 INFO:teuthology.orchestra.run.gibba012.stdout: "entity_id": "0", 2021-02-10T11:40:48.052 INFO:teuthology.orchestra.run.gibba012.stdout: "hostname": "gibba012", 2021-02-10T11:40:48.052 INFO:teuthology.orchestra.run.gibba012.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2021-02-10T11:40:48.052 INFO:teuthology.orchestra.run.gibba012.stdout: "pid": "157734", 2021-02-10T11:40:48.052 INFO:teuthology.orchestra.run.gibba012.stdout: "root": "/" 2021-02-10T11:40:48.052 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:40:48.052 INFO:teuthology.orchestra.run.gibba012.stdout: "dentry_count": 0, 2021-02-10T11:40:48.053 INFO:teuthology.orchestra.run.gibba012.stdout: "dentry_pinned_count": 0, 2021-02-10T11:40:48.053 INFO:teuthology.orchestra.run.gibba012.stdout: "id": 10485, 2021-02-10T11:40:48.053 INFO:teuthology.orchestra.run.gibba012.stdout: "inst": { 2021-02-10T11:40:48.053 INFO:teuthology.orchestra.run.gibba012.stdout: "name": { 2021-02-10T11:40:48.053 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "client", 2021-02-10T11:40:48.054 INFO:teuthology.orchestra.run.gibba012.stdout: "num": 10485 2021-02-10T11:40:48.054 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:40:48.054 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": { 2021-02-10T11:40:48.054 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:40:48.054 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:0", 2021-02-10T11:40:48.054 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 3732934709 2021-02-10T11:40:48.055 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:40:48.055 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:40:48.055 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": { 2021-02-10T11:40:48.055 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:40:48.055 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:0", 2021-02-10T11:40:48.056 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 3732934709 2021-02-10T11:40:48.056 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:40:48.056 INFO:teuthology.orchestra.run.gibba012.stdout: "inst_str": "client.10485 172.21.2.112:0/3732934709", 2021-02-10T11:40:48.056 INFO:teuthology.orchestra.run.gibba012.stdout: "addr_str": "172.21.2.112:0/3732934709", 2021-02-10T11:40:48.056 INFO:teuthology.orchestra.run.gibba012.stdout: "inode_count": 1, 2021-02-10T11:40:48.056 INFO:teuthology.orchestra.run.gibba012.stdout: "mds_epoch": 88, 2021-02-10T11:40:48.057 INFO:teuthology.orchestra.run.gibba012.stdout: "osd_epoch": 108, 2021-02-10T11:40:48.057 INFO:teuthology.orchestra.run.gibba012.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:40:48.057 INFO:teuthology.orchestra.run.gibba012.stdout: "blacklisted": false 2021-02-10T11:40:48.057 INFO:teuthology.orchestra.run.gibba012.stdout:} 2021-02-10T11:40:48.059 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:40:48.059 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:40:48.060 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:40:48.397 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:40:48.397 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:40:48.398 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:40:48.398 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:40:48.742 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:40:49.170 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.158 7f1fe90d3700 1 Processor -- start 2021-02-10T11:40:49.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.158 7f1fe90d3700 1 -- start start 2021-02-10T11:40:49.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.159 7f1fe90d3700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f1fe40d1870 0x7f1fe41b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:49.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.159 7f1fe90d3700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1fe40f0650 0x7f1fe40cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:49.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.159 7f1fe90d3700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fe40f0b10 con 0x7f1fe41b0a10 2021-02-10T11:40:49.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.159 7f1fe90d3700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fe40e5ed0 con 0x7f1fe40d1870 2021-02-10T11:40:49.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.159 7f1fe90d3700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe40d1c70 con 0x7f1fe40f0650 2021-02-10T11:40:49.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.159 7f1fe2d9d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f1fe40d1870 0x7f1fe41b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:49.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.159 7f1fe2d9d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f1fe40d1870 0x7f1fe41b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:54972/0 (socket says 172.21.2.112:54972) 2021-02-10T11:40:49.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.159 7f1fe2d9d700 1 -- 172.21.2.112:0/1467542440 learned_addr learned my addr 172.21.2.112:0/1467542440 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:40:49.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.159 7f1fe359e700 1 --2- 172.21.2.112:0/1467542440 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1fe40f0650 0x7f1fe40cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:49.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.159 7f1fe2d9d700 1 -- 172.21.2.112:0/1467542440 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1fe40f0650 msgr2=0x7f1fe40cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:49.172 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.159 7f1fe2d9d700 1 --2- 172.21.2.112:0/1467542440 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1fe40f0650 0x7f1fe40cfb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:49.172 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.159 7f1fe2d9d700 1 -- 172.21.2.112:0/1467542440 >> v1:172.21.2.120:6789/0 conn(0x7f1fe41b0a10 legacy=0x7f1fe40cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:49.172 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.159 7f1fe2d9d700 1 -- 172.21.2.112:0/1467542440 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fe40d1e90 con 0x7f1fe40d1870 2021-02-10T11:40:49.172 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.159 7f1fe2d9d700 1 --2- 172.21.2.112:0/1467542440 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1fe40d1870 0x7f1fe41b03f0 secure :-1 s=READY pgs=2114 cs=0 l=1 rev1=1 rx=0x7f1fdc006a80 tx=0x7f1fdc009f00).ready entity=mon.0 client_cookie=82a2612cdaac2de7 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:49.172 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.160 7f1fe1d9b700 1 -- 172.21.2.112:0/1467542440 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1fdc00b3b0 con 0x7f1fe40d1870 2021-02-10T11:40:49.172 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.160 7f1fe1d9b700 1 -- 172.21.2.112:0/1467542440 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fdc00b6f0 con 0x7f1fe40d1870 2021-02-10T11:40:49.172 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.160 7f1fe1d9b700 1 -- 172.21.2.112:0/1467542440 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1fdc00bc10 con 0x7f1fe40d1870 2021-02-10T11:40:49.172 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.160 7f1fe90d3700 1 -- 172.21.2.112:0/1467542440 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1fe40d1870 msgr2=0x7f1fe41b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:49.172 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.160 7f1fe90d3700 1 --2- 172.21.2.112:0/1467542440 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1fe40d1870 0x7f1fe41b03f0 secure :-1 s=READY pgs=2114 cs=0 l=1 rev1=1 rx=0x7f1fdc006a80 tx=0x7f1fdc009f00).stop 2021-02-10T11:40:49.172 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.160 7f1fe90d3700 1 -- 172.21.2.112:0/1467542440 shutdown_connections 2021-02-10T11:40:49.172 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.160 7f1fe90d3700 1 --2- 172.21.2.112:0/1467542440 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1fe40f0650 0x7f1fe40cfb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:49.172 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.160 7f1fe90d3700 1 --2- 172.21.2.112:0/1467542440 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1fe40d1870 0x7f1fe41b03f0 unknown :-1 s=CLOSED pgs=2114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:49.172 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.160 7f1fe90d3700 1 -- 172.21.2.112:0/1467542440 >> 172.21.2.112:0/1467542440 conn(0x7f1fe40ce450 msgr2=0x7f1fe40d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:49.172 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.161 7f1fe90d3700 1 -- 172.21.2.112:0/1467542440 shutdown_connections 2021-02-10T11:40:49.173 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.161 7f1fe90d3700 1 -- 172.21.2.112:0/1467542440 wait complete. 2021-02-10T11:40:49.174 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.175 7f1fe90d3700 1 Processor -- start 2021-02-10T11:40:49.174 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.175 7f1fe90d3700 1 -- start start 2021-02-10T11:40:49.174 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.175 7f1fe90d3700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1fe4195650 0x7f1fe40b5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:49.174 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.175 7f1fe90d3700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1fe40b5b80 0x7f1fe41a2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:49.174 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.175 7f1fe90d3700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1fe40acfa0 0x7f1fe41a3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:49.174 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.175 7f1fe90d3700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe40b6010 con 0x7f1fe40acfa0 2021-02-10T11:40:49.175 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.175 7f1fe90d3700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe41a3a60 con 0x7f1fe4195650 2021-02-10T11:40:49.175 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.175 7f1fe90d3700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe41a3c40 con 0x7f1fe40b5b80 2021-02-10T11:40:49.175 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.175 7f1fe359e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1fe40acfa0 0x7f1fe41a3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:49.175 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.175 7f1fe359e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1fe40acfa0 0x7f1fe41a3440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:54976/0 (socket says 172.21.2.112:54976) 2021-02-10T11:40:49.175 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.175 7f1fe359e700 1 -- 172.21.2.112:0/3602303513 learned_addr learned my addr 172.21.2.112:0/3602303513 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:40:49.175 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.176 7f1fe2d9d700 1 --2- 172.21.2.112:0/3602303513 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1fe4195650 0x7f1fe40b5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:49.175 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.176 7f1fe259c700 1 --2- 172.21.2.112:0/3602303513 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1fe40b5b80 0x7f1fe41a2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:49.175 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.176 7f1fe359e700 1 -- 172.21.2.112:0/3602303513 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1fe40b5b80 msgr2=0x7f1fe41a2e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:49.175 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.176 7f1fe359e700 1 --2- 172.21.2.112:0/3602303513 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1fe40b5b80 0x7f1fe41a2e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:49.175 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.176 7f1fe359e700 1 -- 172.21.2.112:0/3602303513 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1fe4195650 msgr2=0x7f1fe40b5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:49.175 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.176 7f1fe359e700 1 --2- 172.21.2.112:0/3602303513 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1fe4195650 0x7f1fe40b5560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:49.175 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.176 7f1fe359e700 1 -- 172.21.2.112:0/3602303513 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fe40bcae0 con 0x7f1fe40acfa0 2021-02-10T11:40:49.175 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.176 7f1fe359e700 1 --2- 172.21.2.112:0/3602303513 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1fe40acfa0 0x7f1fe41a3440 secure :-1 s=READY pgs=2115 cs=0 l=1 rev1=1 rx=0x7f1fd400b5e0 tx=0x7f1fd400b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:49.176 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.176 7f1fdb7fe700 1 -- 172.21.2.112:0/3602303513 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1fd4006370 con 0x7f1fe40acfa0 2021-02-10T11:40:49.176 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.176 7f1fdb7fe700 1 -- 172.21.2.112:0/3602303513 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fd400bd70 con 0x7f1fe40acfa0 2021-02-10T11:40:49.176 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.176 7f1fe90d3700 1 -- 172.21.2.112:0/3602303513 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fe40bcae0 con 0x7f1fe40acfa0 2021-02-10T11:40:49.176 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.176 7f1fe90d3700 1 -- 172.21.2.112:0/3602303513 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fe40bd070 con 0x7f1fe40acfa0 2021-02-10T11:40:49.176 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.176 7f1fdb7fe700 1 -- 172.21.2.112:0/3602303513 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1fd4002560 con 0x7f1fe40acfa0 2021-02-10T11:40:49.176 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.177 7f1fdb7fe700 1 -- 172.21.2.112:0/3602303513 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1fd4005330 con 0x7f1fe40acfa0 2021-02-10T11:40:49.176 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.177 7f1fdb7fe700 1 --2- 172.21.2.112:0/3602303513 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1fc8018730 0x7f1fc801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:49.176 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.177 7f1fdb7fe700 1 -- 172.21.2.112:0/3602303513 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f1fd40327c0 con 0x7f1fe40acfa0 2021-02-10T11:40:49.176 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.177 7f1fe2d9d700 1 --2- 172.21.2.112:0/3602303513 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1fc8018730 0x7f1fc801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:49.176 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.178 7f1fe2d9d700 1 --2- 172.21.2.112:0/3602303513 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1fc8018730 0x7f1fc801ac10 crc :-1 s=READY pgs=979 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:49.204 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.208 7f1fe90d3700 1 -- 172.21.2.112:0/3602303513 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fe40f0ab0 con 0x7f1fe40acfa0 2021-02-10T11:40:49.206 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.210 7f1fdb7fe700 1 -- 172.21.2.112:0/3602303513 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1fd4000bc0 con 0x7f1fe40acfa0 2021-02-10T11:40:49.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.336 7f1fe90d3700 1 -- 172.21.2.112:0/3602303513 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1fe40f0ab0 con 0x7f1fe40acfa0 2021-02-10T11:40:49.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.337 7f1fdb7fe700 1 -- 172.21.2.112:0/3602303513 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v108) v1 ==== 74+0+14006 (secure 0 0 0) 0x7f1fd400bd70 con 0x7f1fe40acfa0 2021-02-10T11:40:49.333 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:40:49.334 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":108,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:40:35.316267","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:40:32.284968","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":23,"pool_name":"cephfs_data","create_time":"2021-02-10 11:40:33.307261","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:40:49.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.339 7f1fe90d3700 1 -- 172.21.2.112:0/3602303513 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1fc8018730 msgr2=0x7f1fc801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:49.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.339 7f1fe90d3700 1 --2- 172.21.2.112:0/3602303513 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1fc8018730 0x7f1fc801ac10 crc :-1 s=READY pgs=979 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:49.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.339 7f1fe90d3700 1 -- 172.21.2.112:0/3602303513 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1fe40acfa0 msgr2=0x7f1fe41a3440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:49.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.339 7f1fe90d3700 1 --2- 172.21.2.112:0/3602303513 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1fe40acfa0 0x7f1fe41a3440 secure :-1 s=READY pgs=2115 cs=0 l=1 rev1=1 rx=0x7f1fd400b5e0 tx=0x7f1fd400b980).stop 2021-02-10T11:40:49.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.339 7f1fe90d3700 1 -- 172.21.2.112:0/3602303513 shutdown_connections 2021-02-10T11:40:49.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.339 7f1fe90d3700 1 --2- 172.21.2.112:0/3602303513 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1fc8018730 0x7f1fc801ac10 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:49.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.339 7f1fe90d3700 1 --2- 172.21.2.112:0/3602303513 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1fe40acfa0 0x7f1fe41a3440 unknown :-1 s=CLOSED pgs=2115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:49.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.339 7f1fe90d3700 1 --2- 172.21.2.112:0/3602303513 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1fe40b5b80 0x7f1fe41a2e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:49.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.339 7f1fe90d3700 1 --2- 172.21.2.112:0/3602303513 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1fe4195650 0x7f1fe40b5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:49.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.339 7f1fe90d3700 1 -- 172.21.2.112:0/3602303513 >> 172.21.2.112:0/3602303513 conn(0x7f1fe40b3040 msgr2=0x7f1fe40c3250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:49.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.339 7f1fe90d3700 1 -- 172.21.2.112:0/3602303513 shutdown_connections 2021-02-10T11:40:49.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.339 7f1fe90d3700 1 -- 172.21.2.112:0/3602303513 wait complete. 2021-02-10T11:40:49.343 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:13:57.508758', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 22, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:40:32.284968', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '107', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 23, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:40:33.307261', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '108', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:40:49.344 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:40:49.774 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.761 7fa4ee07a700 1 Processor -- start 2021-02-10T11:40:49.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.761 7fa4ee07a700 1 -- start start 2021-02-10T11:40:49.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.762 7fa4ee07a700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fa4e80d1870 0x7fa4e81b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:49.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.762 7fa4ee07a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa4e81b0ae0 0x7fa4e8108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:49.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.762 7fa4ee07a700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4e80fe1c0 con 0x7fa4e80d1c70 2021-02-10T11:40:49.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.762 7fa4ee07a700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4e8109a60 con 0x7fa4e80d1870 2021-02-10T11:40:49.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.762 7fa4ee07a700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4e8109750 con 0x7fa4e81b0ae0 2021-02-10T11:40:49.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.762 7fa4e77fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fa4e80d1870 0x7fa4e81b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:49.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.762 7fa4e77fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fa4e80d1870 0x7fa4e81b04c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:54984/0 (socket says 172.21.2.112:54984) 2021-02-10T11:40:49.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.762 7fa4e77fe700 1 -- 172.21.2.112:0/4182311874 learned_addr learned my addr 172.21.2.112:0/4182311874 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:40:49.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.762 7fa4e77fe700 1 -- 172.21.2.112:0/4182311874 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa4e81b0ae0 msgr2=0x7fa4e8108940 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:40:49.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.762 7fa4e77fe700 1 --2- 172.21.2.112:0/4182311874 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa4e81b0ae0 0x7fa4e8108940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:49.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.762 7fa4e77fe700 1 -- 172.21.2.112:0/4182311874 >> v1:172.21.2.120:6789/0 conn(0x7fa4e80d1c70 legacy=0x7fa4e8108f60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:40:49.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.762 7fa4e77fe700 1 -- 172.21.2.112:0/4182311874 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4e8088070 con 0x7fa4e80d1870 2021-02-10T11:40:49.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.762 7fa4e77fe700 1 --2- 172.21.2.112:0/4182311874 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa4e80d1870 0x7fa4e81b04c0 secure :-1 s=READY pgs=2116 cs=0 l=1 rev1=1 rx=0x7fa4e0006a80 tx=0x7fa4e0009f00).ready entity=mon.0 client_cookie=ab947bf9945f3fcf server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:49.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.762 7fa4e67fc700 1 -- 172.21.2.112:0/4182311874 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa4e000b5c0 con 0x7fa4e80d1870 2021-02-10T11:40:49.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.762 7fa4e67fc700 1 -- 172.21.2.112:0/4182311874 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4e000b8c0 con 0x7fa4e80d1870 2021-02-10T11:40:49.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.762 7fa4e67fc700 1 -- 172.21.2.112:0/4182311874 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa4e000bde0 con 0x7fa4e80d1870 2021-02-10T11:40:49.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.762 7fa4ee07a700 1 -- 172.21.2.112:0/4182311874 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa4e80d1870 msgr2=0x7fa4e81b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:49.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.762 7fa4ee07a700 1 --2- 172.21.2.112:0/4182311874 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa4e80d1870 0x7fa4e81b04c0 secure :-1 s=READY pgs=2116 cs=0 l=1 rev1=1 rx=0x7fa4e0006a80 tx=0x7fa4e0009f00).stop 2021-02-10T11:40:49.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.763 7fa4ee07a700 1 -- 172.21.2.112:0/4182311874 shutdown_connections 2021-02-10T11:40:49.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.763 7fa4ee07a700 1 --2- 172.21.2.112:0/4182311874 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa4e81b0ae0 0x7fa4e8108940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:49.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.763 7fa4ee07a700 1 --2- 172.21.2.112:0/4182311874 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa4e80d1870 0x7fa4e81b04c0 unknown :-1 s=CLOSED pgs=2116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:49.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.763 7fa4ee07a700 1 -- 172.21.2.112:0/4182311874 >> 172.21.2.112:0/4182311874 conn(0x7fa4e80ce420 msgr2=0x7fa4e80d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:49.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.763 7fa4ee07a700 1 -- 172.21.2.112:0/4182311874 shutdown_connections 2021-02-10T11:40:49.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.763 7fa4ee07a700 1 -- 172.21.2.112:0/4182311874 wait complete. 2021-02-10T11:40:49.778 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.778 7fa4ee07a700 1 Processor -- start 2021-02-10T11:40:49.778 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.778 7fa4ee07a700 1 -- start start 2021-02-10T11:40:49.778 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.778 7fa4ee07a700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa4e80c3820 0x7fa4e8084cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:49.778 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.778 7fa4ee07a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa4e80a0290 0x7fa4e80852d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:49.778 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.778 7fa4ee07a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa4e8086270 0x7fa4e8086670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:49.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.778 7fa4ee07a700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4e80c0700 con 0x7fa4e8086270 2021-02-10T11:40:49.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.778 7fa4ee07a700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4e80858f0 con 0x7fa4e80c3820 2021-02-10T11:40:49.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.778 7fa4ee07a700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4e8085b10 con 0x7fa4e80a0290 2021-02-10T11:40:49.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.778 7fa4e7fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa4e8086270 0x7fa4e8086670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:49.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.779 7fa4e7fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa4e8086270 0x7fa4e8086670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:54990/0 (socket says 172.21.2.112:54990) 2021-02-10T11:40:49.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.779 7fa4e7fff700 1 -- 172.21.2.112:0/3240103253 learned_addr learned my addr 172.21.2.112:0/3240103253 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:40:49.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.779 7fa4e7fff700 1 -- 172.21.2.112:0/3240103253 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa4e80a0290 msgr2=0x7fa4e80852d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:40:49.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.779 7fa4e7fff700 1 --2- 172.21.2.112:0/3240103253 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa4e80a0290 0x7fa4e80852d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:49.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.779 7fa4e7fff700 1 -- 172.21.2.112:0/3240103253 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa4e80c3820 msgr2=0x7fa4e8084cb0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:40:49.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.779 7fa4e7fff700 1 --2- 172.21.2.112:0/3240103253 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa4e80c3820 0x7fa4e8084cb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:49.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.779 7fa4e7fff700 1 -- 172.21.2.112:0/3240103253 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4e8086c90 con 0x7fa4e8086270 2021-02-10T11:40:49.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.779 7fa4e7fff700 1 --2- 172.21.2.112:0/3240103253 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa4e8086270 0x7fa4e8086670 secure :-1 s=READY pgs=2117 cs=0 l=1 rev1=1 rx=0x7fa4d800b5b0 tx=0x7fa4d800b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:49.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.779 7fa4cffff700 1 -- 172.21.2.112:0/3240103253 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa4d800bd40 con 0x7fa4e8086270 2021-02-10T11:40:49.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.779 7fa4cffff700 1 -- 172.21.2.112:0/3240103253 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4d80022e0 con 0x7fa4e8086270 2021-02-10T11:40:49.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.779 7fa4cffff700 1 -- 172.21.2.112:0/3240103253 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa4d8002800 con 0x7fa4e8086270 2021-02-10T11:40:49.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.779 7fa4ee07a700 1 -- 172.21.2.112:0/3240103253 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4e8086c90 con 0x7fa4e8086270 2021-02-10T11:40:49.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.779 7fa4ee07a700 1 -- 172.21.2.112:0/3240103253 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4e81a2f00 con 0x7fa4e8086270 2021-02-10T11:40:49.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.780 7fa4cffff700 1 -- 172.21.2.112:0/3240103253 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa4d80024c0 con 0x7fa4e8086270 2021-02-10T11:40:49.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.780 7fa4cffff700 1 --2- 172.21.2.112:0/3240103253 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa4c80186c0 0x7fa4c801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:49.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.780 7fa4cffff700 1 -- 172.21.2.112:0/3240103253 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fa4d80325e0 con 0x7fa4e8086270 2021-02-10T11:40:49.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.781 7fa4e77fe700 1 --2- 172.21.2.112:0/3240103253 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa4c80186c0 0x7fa4c801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:49.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.781 7fa4e77fe700 1 --2- 172.21.2.112:0/3240103253 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa4c80186c0 0x7fa4c801abc0 crc :-1 s=READY pgs=980 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:49.809 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.812 7fa4ee07a700 1 -- 172.21.2.112:0/3240103253 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4e8082a70 con 0x7fa4e8086270 2021-02-10T11:40:49.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.815 7fa4cffff700 1 -- 172.21.2.112:0/3240103253 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa4d8000ef0 con 0x7fa4e8086270 2021-02-10T11:40:49.940 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.943 7fa4ee07a700 1 -- 172.21.2.112:0/3240103253 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7fa4e8082a70 con 0x7fa4e8086270 2021-02-10T11:40:49.940 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.943 7fa4cffff700 1 -- 172.21.2.112:0/3240103253 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v108) v1 ==== 92+0+10 (secure 0 0 0) 0x7fa4d80009b0 con 0x7fa4e8086270 2021-02-10T11:40:49.941 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:40:49.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.945 7fa4ee07a700 1 -- 172.21.2.112:0/3240103253 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa4c80186c0 msgr2=0x7fa4c801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:49.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.945 7fa4ee07a700 1 --2- 172.21.2.112:0/3240103253 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa4c80186c0 0x7fa4c801abc0 crc :-1 s=READY pgs=980 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:49.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.945 7fa4ee07a700 1 -- 172.21.2.112:0/3240103253 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa4e8086270 msgr2=0x7fa4e8086670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:49.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.945 7fa4ee07a700 1 --2- 172.21.2.112:0/3240103253 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa4e8086270 0x7fa4e8086670 secure :-1 s=READY pgs=2117 cs=0 l=1 rev1=1 rx=0x7fa4d800b5b0 tx=0x7fa4d800b950).stop 2021-02-10T11:40:49.943 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.945 7fa4ee07a700 1 -- 172.21.2.112:0/3240103253 shutdown_connections 2021-02-10T11:40:49.943 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.945 7fa4ee07a700 1 --2- 172.21.2.112:0/3240103253 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa4c80186c0 0x7fa4c801abc0 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:49.943 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.945 7fa4ee07a700 1 --2- 172.21.2.112:0/3240103253 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa4e8086270 0x7fa4e8086670 unknown :-1 s=CLOSED pgs=2117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:49.943 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.945 7fa4ee07a700 1 --2- 172.21.2.112:0/3240103253 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa4e80a0290 0x7fa4e80852d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:49.943 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.945 7fa4ee07a700 1 --2- 172.21.2.112:0/3240103253 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa4e80c3820 0x7fa4e8084cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:49.943 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.945 7fa4ee07a700 1 -- 172.21.2.112:0/3240103253 >> 172.21.2.112:0/3240103253 conn(0x7fa4e80872c0 msgr2=0x7fa4e8095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:49.943 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.945 7fa4ee07a700 1 -- 172.21.2.112:0/3240103253 shutdown_connections 2021-02-10T11:40:49.943 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:49.946 7fa4ee07a700 1 -- 172.21.2.112:0/3240103253 wait complete. 2021-02-10T11:40:49.951 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:40:50.382 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.368 7f49459ad700 1 Processor -- start 2021-02-10T11:40:50.382 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.368 7f49459ad700 1 -- start start 2021-02-10T11:40:50.382 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.369 7f49459ad700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f49400d1c70 0x7f49400cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:50.382 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.369 7f49459ad700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f49400cf580 0x7f49400cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:50.382 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.369 7f49459ad700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49400fe1c0 con 0x7f49400d1870 2021-02-10T11:40:50.382 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.369 7f49459ad700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49401b0cb0 con 0x7f49400cf580 2021-02-10T11:40:50.383 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.369 7f49459ad700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4940109a60 con 0x7f49400d1c70 2021-02-10T11:40:50.383 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.369 7f493fd75700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f49400cf580 0x7f49400cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:50.383 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.369 7f493fd75700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f49400cf580 0x7f49400cf980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:55002/0 (socket says 172.21.2.112:55002) 2021-02-10T11:40:50.383 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.369 7f493fd75700 1 -- 172.21.2.112:0/1729162972 learned_addr learned my addr 172.21.2.112:0/1729162972 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:40:50.383 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.369 7f493fd75700 1 -- 172.21.2.112:0/1729162972 >> v2:172.21.2.120:3300/0 conn(0x7f49400d1c70 msgr2=0x7f49400cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:50.383 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.369 7f493fd75700 1 --2- 172.21.2.112:0/1729162972 >> v2:172.21.2.120:3300/0 conn(0x7f49400d1c70 0x7f49400cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:50.383 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.369 7f493fd75700 1 -- 172.21.2.112:0/1729162972 >> v1:172.21.2.120:6789/0 conn(0x7f49400d1870 legacy=0x7f49401b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:50.383 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.369 7f493fd75700 1 -- 172.21.2.112:0/1729162972 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4940088070 con 0x7f49400cf580 2021-02-10T11:40:50.383 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.370 7f493fd75700 1 --2- 172.21.2.112:0/1729162972 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f49400cf580 0x7f49400cf980 secure :-1 s=READY pgs=2118 cs=0 l=1 rev1=1 rx=0x7f4938006f00 tx=0x7f4938009e90).ready entity=mon.0 client_cookie=9784c51c3d40f6b4 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:50.383 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.370 7f493e572700 1 -- 172.21.2.112:0/1729162972 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f493800b300 con 0x7f49400cf580 2021-02-10T11:40:50.383 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.370 7f493e572700 1 -- 172.21.2.112:0/1729162972 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f493800b580 con 0x7f49400cf580 2021-02-10T11:40:50.384 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.370 7f493e572700 1 -- 172.21.2.112:0/1729162972 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f493800b9d0 con 0x7f49400cf580 2021-02-10T11:40:50.384 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.370 7f49459ad700 1 -- 172.21.2.112:0/1729162972 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f49400cf580 msgr2=0x7f49400cf980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:50.384 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.370 7f49459ad700 1 --2- 172.21.2.112:0/1729162972 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f49400cf580 0x7f49400cf980 secure :-1 s=READY pgs=2118 cs=0 l=1 rev1=1 rx=0x7f4938006f00 tx=0x7f4938009e90).stop 2021-02-10T11:40:50.384 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.371 7f49459ad700 1 -- 172.21.2.112:0/1729162972 shutdown_connections 2021-02-10T11:40:50.384 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.371 7f49459ad700 1 --2- 172.21.2.112:0/1729162972 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f49400cf580 0x7f49400cf980 unknown :-1 s=CLOSED pgs=2118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:50.384 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.371 7f49459ad700 1 --2- 172.21.2.112:0/1729162972 >> v2:172.21.2.120:3300/0 conn(0x7f49400d1c70 0x7f49400cef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:50.384 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.371 7f49459ad700 1 -- 172.21.2.112:0/1729162972 >> 172.21.2.112:0/1729162972 conn(0x7f49400ce420 msgr2=0x7f49400d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:50.384 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.371 7f49459ad700 1 -- 172.21.2.112:0/1729162972 shutdown_connections 2021-02-10T11:40:50.385 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.371 7f49459ad700 1 -- 172.21.2.112:0/1729162972 wait complete. 2021-02-10T11:40:50.386 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.385 7f49459ad700 1 Processor -- start 2021-02-10T11:40:50.386 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.385 7f49459ad700 1 -- start start 2021-02-10T11:40:50.386 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.386 7f49459ad700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f49400c3820 0x7f4940084cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:50.386 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.386 7f49459ad700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f49400a0290 0x7f49400852d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:50.386 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.386 7f49459ad700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4940086270 0x7f4940086670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:50.386 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.386 7f49459ad700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49400c0700 con 0x7f49400a0290 2021-02-10T11:40:50.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.386 7f49459ad700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49400858f0 con 0x7f4940086270 2021-02-10T11:40:50.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.386 7f49459ad700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4940085b10 con 0x7f49400c3820 2021-02-10T11:40:50.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.386 7f493ed73700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f49400a0290 0x7f49400852d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:50.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.386 7f493ed73700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f49400a0290 0x7f49400852d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:55004/0 (socket says 172.21.2.112:55004) 2021-02-10T11:40:50.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.386 7f493ed73700 1 -- 172.21.2.112:0/2307622751 learned_addr learned my addr 172.21.2.112:0/2307622751 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:40:50.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.386 7f493fd75700 1 --2- 172.21.2.112:0/2307622751 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4940086270 0x7f4940086670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:50.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.386 7f493ed73700 1 -- 172.21.2.112:0/2307622751 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f49400c3820 msgr2=0x7f4940084cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:50.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.386 7f493ed73700 1 --2- 172.21.2.112:0/2307622751 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f49400c3820 0x7f4940084cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:50.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.386 7f493ed73700 1 -- 172.21.2.112:0/2307622751 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4940086270 msgr2=0x7f4940086670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:50.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.386 7f493ed73700 1 --2- 172.21.2.112:0/2307622751 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4940086270 0x7f4940086670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:50.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.386 7f493ed73700 1 -- 172.21.2.112:0/2307622751 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4940086c90 con 0x7f49400a0290 2021-02-10T11:40:50.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.386 7f493ed73700 1 --2- 172.21.2.112:0/2307622751 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f49400a0290 0x7f49400852d0 secure :-1 s=READY pgs=2119 cs=0 l=1 rev1=1 rx=0x7f4930006f00 tx=0x7f493000b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:50.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.387 7f4937fff700 1 -- 172.21.2.112:0/2307622751 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f49300009b0 con 0x7f49400a0290 2021-02-10T11:40:50.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.387 7f4937fff700 1 -- 172.21.2.112:0/2307622751 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4930006350 con 0x7f49400a0290 2021-02-10T11:40:50.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.387 7f4937fff700 1 -- 172.21.2.112:0/2307622751 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4930000c90 con 0x7f49400a0290 2021-02-10T11:40:50.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.387 7f49459ad700 1 -- 172.21.2.112:0/2307622751 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4940086c90 con 0x7f49400a0290 2021-02-10T11:40:50.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.387 7f49459ad700 1 -- 172.21.2.112:0/2307622751 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49401a2f00 con 0x7f49400a0290 2021-02-10T11:40:50.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.387 7f4937fff700 1 -- 172.21.2.112:0/2307622751 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4930007000 con 0x7f49400a0290 2021-02-10T11:40:50.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.387 7f4937fff700 1 --2- 172.21.2.112:0/2307622751 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f49240186c0 0x7f492401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:50.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.387 7f4937fff700 1 -- 172.21.2.112:0/2307622751 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f4930032560 con 0x7f49400a0290 2021-02-10T11:40:50.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.388 7f493f574700 1 --2- 172.21.2.112:0/2307622751 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f49240186c0 0x7f492401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:50.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.388 7f493f574700 1 --2- 172.21.2.112:0/2307622751 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f49240186c0 0x7f492401abc0 crc :-1 s=READY pgs=981 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:50.417 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.419 7f49459ad700 1 -- 172.21.2.112:0/2307622751 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4940082a70 con 0x7f49400a0290 2021-02-10T11:40:50.418 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.421 7f4937fff700 1 -- 172.21.2.112:0/2307622751 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4930000960 con 0x7f49400a0290 2021-02-10T11:40:50.546 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.549 7f49459ad700 1 -- 172.21.2.112:0/2307622751 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f4940082a70 con 0x7f49400a0290 2021-02-10T11:40:50.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.550 7f4937fff700 1 -- 172.21.2.112:0/2307622751 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v108) v1 ==== 104+0+10 (secure 0 0 0) 0x7f493000bd30 con 0x7f49400a0290 2021-02-10T11:40:50.547 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:40:50.549 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.552 7f49459ad700 1 -- 172.21.2.112:0/2307622751 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f49240186c0 msgr2=0x7f492401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:50.549 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.552 7f49459ad700 1 --2- 172.21.2.112:0/2307622751 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f49240186c0 0x7f492401abc0 crc :-1 s=READY pgs=981 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:50.549 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.552 7f49459ad700 1 -- 172.21.2.112:0/2307622751 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f49400a0290 msgr2=0x7f49400852d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:50.549 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.552 7f49459ad700 1 --2- 172.21.2.112:0/2307622751 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f49400a0290 0x7f49400852d0 secure :-1 s=READY pgs=2119 cs=0 l=1 rev1=1 rx=0x7f4930006f00 tx=0x7f493000b940).stop 2021-02-10T11:40:50.549 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.552 7f49459ad700 1 -- 172.21.2.112:0/2307622751 shutdown_connections 2021-02-10T11:40:50.549 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.552 7f49459ad700 1 --2- 172.21.2.112:0/2307622751 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f49240186c0 0x7f492401abc0 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:50.549 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.552 7f49459ad700 1 --2- 172.21.2.112:0/2307622751 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4940086270 0x7f4940086670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:50.550 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.552 7f49459ad700 1 --2- 172.21.2.112:0/2307622751 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f49400a0290 0x7f49400852d0 unknown :-1 s=CLOSED pgs=2119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:50.550 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.552 7f49459ad700 1 --2- 172.21.2.112:0/2307622751 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f49400c3820 0x7f4940084cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:50.550 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.552 7f49459ad700 1 -- 172.21.2.112:0/2307622751 >> 172.21.2.112:0/2307622751 conn(0x7f49400872c0 msgr2=0x7f4940095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:50.550 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.552 7f49459ad700 1 -- 172.21.2.112:0/2307622751 shutdown_connections 2021-02-10T11:40:50.550 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.552 7f49459ad700 1 -- 172.21.2.112:0/2307622751 wait complete. 2021-02-10T11:40:50.558 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:40:50.990 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.976 7f4fcfff8700 1 Processor -- start 2021-02-10T11:40:50.990 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.976 7f4fcfff8700 1 -- start start 2021-02-10T11:40:50.990 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.976 7f4fcfff8700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f4fc80d1c70 0x7f4fc80cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:50.990 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.976 7f4fcfff8700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4fc80cf580 0x7f4fc80cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:50.990 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.976 7f4fcfff8700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fc80fe1c0 con 0x7f4fc80d1870 2021-02-10T11:40:50.990 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.976 7f4fcfff8700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4fc81b0cb0 con 0x7f4fc80d1c70 2021-02-10T11:40:50.990 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.976 7f4fcfff8700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fc8109a60 con 0x7f4fc80cf580 2021-02-10T11:40:50.990 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.976 7f4fcde14700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f4fc80d1870 0x7f4fc81b04c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:54474/0 (socket says 172.21.2.112:54474) 2021-02-10T11:40:50.990 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.977 7f4fcde14700 1 -- 172.21.2.112:0/1743279864 learned_addr learned my addr 172.21.2.112:0/1743279864 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:40:50.991 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.977 7f4fcd613700 1 --2- 172.21.2.112:0/1743279864 >> v2:172.21.2.120:3300/0 conn(0x7f4fc80d1c70 0x7f4fc80cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:50.991 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.977 7f4fce615700 1 --2- 172.21.2.112:0/1743279864 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4fc80cf580 0x7f4fc80cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:50.991 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.977 7f4fcce12700 1 -- 172.21.2.112:0/1743279864 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1166134911 0 0) 0x7f4fc40010b0 con 0x7f4fc80d1870 2021-02-10T11:40:50.991 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.977 7f4fcce12700 1 -- 172.21.2.112:0/1743279864 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4fb4002fe0 con 0x7f4fc80d1870 2021-02-10T11:40:50.991 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.977 7f4fce615700 1 -- 172.21.2.112:0/1743279864 >> v2:172.21.2.120:3300/0 conn(0x7f4fc80d1c70 msgr2=0x7f4fc80cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:50.991 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.977 7f4fce615700 1 --2- 172.21.2.112:0/1743279864 >> v2:172.21.2.120:3300/0 conn(0x7f4fc80d1c70 0x7f4fc80cef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:50.991 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.977 7f4fce615700 1 -- 172.21.2.112:0/1743279864 >> v1:172.21.2.112:6789/0 conn(0x7f4fc80d1870 legacy=0x7f4fc81b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:50.991 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.977 7f4fce615700 1 -- 172.21.2.112:0/1743279864 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fc8088070 con 0x7f4fc80cf580 2021-02-10T11:40:50.991 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.977 7f4fce615700 1 --2- 172.21.2.112:0/1743279864 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4fc80cf580 0x7f4fc80cf980 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f4fc0006e50 tx=0x7f4fc0009e70).ready entity=mon.2 client_cookie=9e3ba72f43ce1015 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:50.991 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.977 7f4fcce12700 1 -- 172.21.2.112:0/1743279864 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4fc000b510 con 0x7f4fc80cf580 2021-02-10T11:40:50.992 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.977 7f4fcce12700 1 -- 172.21.2.112:0/1743279864 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fc0006790 con 0x7f4fc80cf580 2021-02-10T11:40:50.992 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.977 7f4fcce12700 1 -- 172.21.2.112:0/1743279864 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4fc000b940 con 0x7f4fc80cf580 2021-02-10T11:40:50.992 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.977 7f4fcfff8700 1 -- 172.21.2.112:0/1743279864 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4fc80cf580 msgr2=0x7f4fc80cf980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:50.992 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.977 7f4fcfff8700 1 --2- 172.21.2.112:0/1743279864 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4fc80cf580 0x7f4fc80cf980 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f4fc0006e50 tx=0x7f4fc0009e70).stop 2021-02-10T11:40:50.992 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.978 7f4fcfff8700 1 -- 172.21.2.112:0/1743279864 shutdown_connections 2021-02-10T11:40:50.992 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.978 7f4fcfff8700 1 --2- 172.21.2.112:0/1743279864 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4fc80cf580 0x7f4fc80cf980 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:50.992 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.978 7f4fcfff8700 1 --2- 172.21.2.112:0/1743279864 >> v2:172.21.2.120:3300/0 conn(0x7f4fc80d1c70 0x7f4fc80cef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:50.992 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.978 7f4fcfff8700 1 -- 172.21.2.112:0/1743279864 >> 172.21.2.112:0/1743279864 conn(0x7f4fc80ce420 msgr2=0x7f4fc80d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:50.992 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.978 7f4fcfff8700 1 -- 172.21.2.112:0/1743279864 shutdown_connections 2021-02-10T11:40:50.992 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.978 7f4fcfff8700 1 -- 172.21.2.112:0/1743279864 wait complete. 2021-02-10T11:40:50.994 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.993 7f4fcfff8700 1 Processor -- start 2021-02-10T11:40:50.994 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.993 7f4fcfff8700 1 -- start start 2021-02-10T11:40:50.994 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.993 7f4fcfff8700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4fc81a79a0 0x7f4fc81a7da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:50.994 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.993 7f4fcfff8700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4fc80bdb40 0x7f4fc80bdf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:50.994 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.993 7f4fcfff8700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4fc8082ca0 0x7f4fc80830a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:50.994 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.993 7f4fcfff8700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fc819d990 con 0x7f4fc8082ca0 2021-02-10T11:40:50.994 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.993 7f4fcfff8700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fc819dbb0 con 0x7f4fc80bdb40 2021-02-10T11:40:50.995 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.993 7f4fcfff8700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fc80836c0 con 0x7f4fc81a79a0 2021-02-10T11:40:50.995 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.994 7f4fce615700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4fc8082ca0 0x7f4fc80830a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:50.995 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.994 7f4fce615700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4fc8082ca0 0x7f4fc80830a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:55022/0 (socket says 172.21.2.112:55022) 2021-02-10T11:40:50.995 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.994 7f4fce615700 1 -- 172.21.2.112:0/2110844683 learned_addr learned my addr 172.21.2.112:0/2110844683 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:40:50.995 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.994 7f4fcd613700 1 --2- 172.21.2.112:0/2110844683 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4fc80bdb40 0x7f4fc80bdf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:50.996 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.994 7f4fcde14700 1 --2- 172.21.2.112:0/2110844683 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4fc81a79a0 0x7f4fc81a7da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:50.996 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.994 7f4fce615700 1 -- 172.21.2.112:0/2110844683 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4fc81a79a0 msgr2=0x7f4fc81a7da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:50.996 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.994 7f4fce615700 1 --2- 172.21.2.112:0/2110844683 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4fc81a79a0 0x7f4fc81a7da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:50.996 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.994 7f4fce615700 1 -- 172.21.2.112:0/2110844683 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4fc80bdb40 msgr2=0x7f4fc80bdf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:50.996 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.994 7f4fce615700 1 --2- 172.21.2.112:0/2110844683 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4fc80bdb40 0x7f4fc80bdf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:50.996 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.994 7f4fce615700 1 -- 172.21.2.112:0/2110844683 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fc80838e0 con 0x7f4fc8082ca0 2021-02-10T11:40:50.996 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.994 7f4fcd613700 1 --2- 172.21.2.112:0/2110844683 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4fc80bdb40 0x7f4fc80bdf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:40:50.996 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.994 7f4fce615700 1 --2- 172.21.2.112:0/2110844683 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4fc8082ca0 0x7f4fc80830a0 secure :-1 s=READY pgs=2121 cs=0 l=1 rev1=1 rx=0x7f4fc0009a90 tx=0x7f4fc000bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:50.996 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.994 7f4fba7fc700 1 -- 172.21.2.112:0/2110844683 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4fc00093d0 con 0x7f4fc8082ca0 2021-02-10T11:40:50.996 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.994 7f4fba7fc700 1 -- 172.21.2.112:0/2110844683 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fc0002f10 con 0x7f4fc8082ca0 2021-02-10T11:40:50.996 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.994 7f4fba7fc700 1 -- 172.21.2.112:0/2110844683 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4fc000c4e0 con 0x7f4fc8082ca0 2021-02-10T11:40:50.996 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.994 7f4fcfff8700 1 -- 172.21.2.112:0/2110844683 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fc80838e0 con 0x7f4fc8082ca0 2021-02-10T11:40:50.996 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.994 7f4fcfff8700 1 -- 172.21.2.112:0/2110844683 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fc80838e0 con 0x7f4fc8082ca0 2021-02-10T11:40:50.997 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.995 7f4fba7fc700 1 -- 172.21.2.112:0/2110844683 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4fc00056b0 con 0x7f4fc8082ca0 2021-02-10T11:40:50.997 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.995 7f4fba7fc700 1 --2- 172.21.2.112:0/2110844683 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4fb00186c0 0x7f4fb001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:50.997 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.995 7f4fba7fc700 1 -- 172.21.2.112:0/2110844683 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f4fc00328d0 con 0x7f4fc8082ca0 2021-02-10T11:40:50.997 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.995 7f4fcde14700 1 --2- 172.21.2.112:0/2110844683 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4fb00186c0 0x7f4fb001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:50.997 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:50.996 7f4fcde14700 1 --2- 172.21.2.112:0/2110844683 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4fb00186c0 0x7f4fb001abc0 crc :-1 s=READY pgs=982 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:51.024 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:51.027 7f4fcfff8700 1 -- 172.21.2.112:0/2110844683 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fc8082a70 con 0x7f4fc8082ca0 2021-02-10T11:40:51.026 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:51.029 7f4fba7fc700 1 -- 172.21.2.112:0/2110844683 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4fc000be30 con 0x7f4fc8082ca0 2021-02-10T11:40:51.154 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:51.157 7f4fcfff8700 1 -- 172.21.2.112:0/2110844683 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f4fc8082a70 con 0x7f4fc8082ca0 2021-02-10T11:40:51.155 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:51.158 7f4fba7fc700 1 -- 172.21.2.112:0/2110844683 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v108) v1 ==== 100+0+10 (secure 0 0 0) 0x7f4fc00056b0 con 0x7f4fc8082ca0 2021-02-10T11:40:51.156 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:40:51.157 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:51.160 7f4fcfff8700 1 -- 172.21.2.112:0/2110844683 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4fb00186c0 msgr2=0x7f4fb001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:51.157 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:51.160 7f4fcfff8700 1 --2- 172.21.2.112:0/2110844683 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4fb00186c0 0x7f4fb001abc0 crc :-1 s=READY pgs=982 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:51.157 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:51.160 7f4fcfff8700 1 -- 172.21.2.112:0/2110844683 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4fc8082ca0 msgr2=0x7f4fc80830a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:51.157 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:51.160 7f4fcfff8700 1 --2- 172.21.2.112:0/2110844683 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4fc8082ca0 0x7f4fc80830a0 secure :-1 s=READY pgs=2121 cs=0 l=1 rev1=1 rx=0x7f4fc0009a90 tx=0x7f4fc000bcf0).stop 2021-02-10T11:40:51.157 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:51.160 7f4fcfff8700 1 -- 172.21.2.112:0/2110844683 shutdown_connections 2021-02-10T11:40:51.157 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:51.160 7f4fcfff8700 1 --2- 172.21.2.112:0/2110844683 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4fb00186c0 0x7f4fb001abc0 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:51.158 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:51.160 7f4fcfff8700 1 --2- 172.21.2.112:0/2110844683 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4fc80bdb40 0x7f4fc80bdf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:51.158 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:51.160 7f4fcfff8700 1 --2- 172.21.2.112:0/2110844683 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4fc8082ca0 0x7f4fc80830a0 unknown :-1 s=CLOSED pgs=2121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:51.158 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:51.160 7f4fcfff8700 1 --2- 172.21.2.112:0/2110844683 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4fc81a79a0 0x7f4fc81a7da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:51.158 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:51.160 7f4fcfff8700 1 -- 172.21.2.112:0/2110844683 >> 172.21.2.112:0/2110844683 conn(0x7f4fc80bef00 msgr2=0x7f4fc809f200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:51.158 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:51.160 7f4fcfff8700 1 -- 172.21.2.112:0/2110844683 shutdown_connections 2021-02-10T11:40:51.158 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:51.160 7f4fcfff8700 1 -- 172.21.2.112:0/2110844683 wait complete. 2021-02-10T11:40:51.166 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:40:51.166 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:40:51.594 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:51.580 7f6c8bec3700 1 Processor -- start 2021-02-10T11:40:51.594 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:51.580 7f6c8bec3700 1 -- start start 2021-02-10T11:40:51.594 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:51.581 7f6c8bec3700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6c841b0470 0x7f6c841b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:51.594 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:51.581 7f6c8bec3700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f6c840ef610 0x7f6c840efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:51.594 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:51.581 7f6c8bec3700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f6c840f0030 0x7f6c840f0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:51.595 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:51.581 7f6c8bec3700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c840e5ea0 con 0x7f6c840ef610 2021-02-10T11:40:51.595 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:51.581 7f6c8bec3700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c840f0b50 con 0x7f6c840f0030 2021-02-10T11:40:51.595 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:51.581 7f6c8bec3700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c840e52e0 con 0x7f6c841b0470 2021-02-10T11:40:51.595 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:51.581 7f6c894de700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f6c840ef610 0x7f6c840efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:51.595 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:51.581 7f6c894de700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f6c840ef610 0x7f6c840efa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:55026/0 (socket says 172.21.2.112:55026) 2021-02-10T11:40:51.595 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:51.581 7f6c894de700 1 -- 172.21.2.112:0/2381700070 learned_addr learned my addr 172.21.2.112:0/2381700070 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:40:51.595 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:51.581 7f6c89cdf700 1 --2- 172.21.2.112:0/2381700070 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6c841b0470 0x7f6c841b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:51.595 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:51.581 7f6c8a4e0700 1 --2- 172.21.2.112:0/2381700070 >> v2:172.21.2.120:3300/0 conn(0x7f6c840f0030 0x7f6c840f0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:51.595 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:51.581 7f6c894de700 1 -- 172.21.2.112:0/2381700070 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6c841b0470 msgr2=0x7f6c841b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:51.595 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:51.581 7f6c894de700 1 --2- 172.21.2.112:0/2381700070 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6c841b0470 0x7f6c841b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:51.595 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:51.581 7f6c894de700 1 -- 172.21.2.112:0/2381700070 >> v2:172.21.2.120:3300/0 conn(0x7f6c840f0030 msgr2=0x7f6c840f0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:51.596 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:51.581 7f6c894de700 1 --2- 172.21.2.112:0/2381700070 >> v2:172.21.2.120:3300/0 conn(0x7f6c840f0030 0x7f6c840f0430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:51.596 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:51.581 7f6c894de700 1 -- 172.21.2.112:0/2381700070 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c840f10d0 con 0x7f6c840ef610 2021-02-10T11:40:51.596 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:51.581 7f6c894de700 1 --2- 172.21.2.112:0/2381700070 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6c840ef610 0x7f6c840efa10 secure :-1 s=READY pgs=2122 cs=0 l=1 rev1=1 rx=0x7f6c78006700 tx=0x7f6c78009b80).ready entity=mon.0 client_cookie=523b67fa2b625736 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:51.596 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:51.581 7f6c88cdd700 1 -- 172.21.2.112:0/2381700070 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6c7800c0c0 con 0x7f6c840ef610 2021-02-10T11:40:51.596 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:51.582 7f6c88cdd700 1 -- 172.21.2.112:0/2381700070 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c7800c340 con 0x7f6c840ef610 2021-02-10T11:40:51.596 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:51.582 7f6c88cdd700 1 -- 172.21.2.112:0/2381700070 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6c7800c860 con 0x7f6c840ef610 2021-02-10T11:40:51.596 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:51.582 7f6c8bec3700 1 -- 172.21.2.112:0/2381700070 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6c840ef610 msgr2=0x7f6c840efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:51.596 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:51.582 7f6c8bec3700 1 --2- 172.21.2.112:0/2381700070 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6c840ef610 0x7f6c840efa10 secure :-1 s=READY pgs=2122 cs=0 l=1 rev1=1 rx=0x7f6c78006700 tx=0x7f6c78009b80).stop 2021-02-10T11:40:51.596 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:51.582 7f6c8bec3700 1 -- 172.21.2.112:0/2381700070 shutdown_connections 2021-02-10T11:40:51.596 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:51.582 7f6c8bec3700 1 --2- 172.21.2.112:0/2381700070 >> v2:172.21.2.120:3300/0 conn(0x7f6c840f0030 0x7f6c840f0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:51.596 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:51.582 7f6c8bec3700 1 --2- 172.21.2.112:0/2381700070 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6c840ef610 0x7f6c840efa10 unknown :-1 s=CLOSED pgs=2122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:51.596 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:51.582 7f6c8bec3700 1 --2- 172.21.2.112:0/2381700070 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6c841b0470 0x7f6c841b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:51.597 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:51.582 7f6c8bec3700 1 -- 172.21.2.112:0/2381700070 >> 172.21.2.112:0/2381700070 conn(0x7f6c840ce3e0 msgr2=0x7f6c840e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:51.597 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:51.582 7f6c8bec3700 1 -- 172.21.2.112:0/2381700070 shutdown_connections 2021-02-10T11:40:51.597 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:51.582 7f6c8bec3700 1 -- 172.21.2.112:0/2381700070 wait complete. 2021-02-10T11:40:51.598 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:51.597 7f6c8bec3700 1 Processor -- start 2021-02-10T11:40:51.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:51.597 7f6c8bec3700 1 -- start start 2021-02-10T11:40:51.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:51.598 7f6c8bec3700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6c841a7a10 0x7f6c841a7e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:51.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:51.598 7f6c8bec3700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6c84109980 0x7f6c84109d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:51.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:51.598 7f6c8bec3700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6c8410a3a0 0x7f6c840a22e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:51.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:51.598 7f6c8bec3700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c840c3760 con 0x7f6c8410a3a0 2021-02-10T11:40:51.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:51.598 7f6c8bec3700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c840a2900 con 0x7f6c84109980 2021-02-10T11:40:51.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:51.598 7f6c8bec3700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c840a2b20 con 0x7f6c841a7a10 2021-02-10T11:40:51.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:51.598 7f6c8a4e0700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6c8410a3a0 0x7f6c840a22e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:51.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:51.598 7f6c8a4e0700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6c8410a3a0 0x7f6c840a22e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:55032/0 (socket says 172.21.2.112:55032) 2021-02-10T11:40:51.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:51.598 7f6c8a4e0700 1 -- 172.21.2.112:0/3740619957 learned_addr learned my addr 172.21.2.112:0/3740619957 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:40:51.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:51.598 7f6c894de700 1 --2- 172.21.2.112:0/3740619957 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6c84109980 0x7f6c84109d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:51.600 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:51.598 7f6c89cdf700 1 --2- 172.21.2.112:0/3740619957 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6c841a7a10 0x7f6c841a7e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:51.600 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:51.598 7f6c8a4e0700 1 -- 172.21.2.112:0/3740619957 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6c841a7a10 msgr2=0x7f6c841a7e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:51.600 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:51.598 7f6c8a4e0700 1 --2- 172.21.2.112:0/3740619957 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6c841a7a10 0x7f6c841a7e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:51.600 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:51.598 7f6c8a4e0700 1 -- 172.21.2.112:0/3740619957 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6c84109980 msgr2=0x7f6c84109d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:51.600 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:51.598 7f6c8a4e0700 1 --2- 172.21.2.112:0/3740619957 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6c84109980 0x7f6c84109d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:51.600 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:51.598 7f6c8a4e0700 1 -- 172.21.2.112:0/3740619957 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c840a2d40 con 0x7f6c8410a3a0 2021-02-10T11:40:51.600 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:51.598 7f6c894de700 1 --2- 172.21.2.112:0/3740619957 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6c84109980 0x7f6c84109d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:40:51.600 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:51.599 7f6c8a4e0700 1 --2- 172.21.2.112:0/3740619957 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6c8410a3a0 0x7f6c840a22e0 secure :-1 s=READY pgs=2123 cs=0 l=1 rev1=1 rx=0x7f6c7c0099a0 tx=0x7f6c7c00bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:51.600 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:51.599 7f6c767fc700 1 -- 172.21.2.112:0/3740619957 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6c7c0066f0 con 0x7f6c8410a3a0 2021-02-10T11:40:51.600 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:51.599 7f6c767fc700 1 -- 172.21.2.112:0/3740619957 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c7c003030 con 0x7f6c8410a3a0 2021-02-10T11:40:51.600 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:51.599 7f6c8bec3700 1 -- 172.21.2.112:0/3740619957 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c840a2d40 con 0x7f6c8410a3a0 2021-02-10T11:40:51.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:51.599 7f6c767fc700 1 -- 172.21.2.112:0/3740619957 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6c7c0013e0 con 0x7f6c8410a3a0 2021-02-10T11:40:51.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:51.599 7f6c8bec3700 1 -- 172.21.2.112:0/3740619957 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c840a2d40 con 0x7f6c8410a3a0 2021-02-10T11:40:51.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:51.599 7f6c767fc700 1 -- 172.21.2.112:0/3740619957 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6c7c0011f0 con 0x7f6c8410a3a0 2021-02-10T11:40:51.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:51.600 7f6c767fc700 1 --2- 172.21.2.112:0/3740619957 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6c6c018730 0x7f6c6c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:51.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:51.600 7f6c767fc700 1 -- 172.21.2.112:0/3740619957 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f6c7c0329c0 con 0x7f6c8410a3a0 2021-02-10T11:40:51.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:51.600 7f6c89cdf700 1 --2- 172.21.2.112:0/3740619957 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6c6c018730 0x7f6c6c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:51.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:51.600 7f6c89cdf700 1 --2- 172.21.2.112:0/3740619957 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6c6c018730 0x7f6c6c01abc0 crc :-1 s=READY pgs=983 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:51.629 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:51.631 7f6c8bec3700 1 -- 172.21.2.112:0/3740619957 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c840c3760 con 0x7f6c8410a3a0 2021-02-10T11:40:51.630 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:51.633 7f6c767fc700 1 -- 172.21.2.112:0/3740619957 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6c7c00be30 con 0x7f6c8410a3a0 2021-02-10T11:40:51.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:51.761 7f6c8bec3700 1 -- 172.21.2.112:0/3740619957 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6c840c3760 con 0x7f6c8410a3a0 2021-02-10T11:40:51.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:51.761 7f6c767fc700 1 -- 172.21.2.112:0/3740619957 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+2107 (secure 0 0 0) 0x7f6c7c001f00 con 0x7f6c8410a3a0 2021-02-10T11:40:51.759 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:40:51.759 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":88,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10335,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/4159879144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":4159879144},{"type":"v1","addr":"172.21.2.112:6833","nonce":4159879144}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":84}],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:40:34.314832","modified":"2021-02-10 11:40:36.323257","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10332},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10332":{"gid":10332,"name":"b","rank":0,"incarnation":85,"state":"up:active","state_seq":3,"addr":"172.21.2.120:6835/3723311838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":3723311838},{"type":"v1","addr":"172.21.2.120:6835","nonce":3723311838}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":11}]} 2021-02-10T11:40:51.759 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 88 2021-02-10T11:40:51.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:51.763 7f6c8bec3700 1 -- 172.21.2.112:0/3740619957 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6c6c018730 msgr2=0x7f6c6c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:51.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:51.763 7f6c8bec3700 1 --2- 172.21.2.112:0/3740619957 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6c6c018730 0x7f6c6c01abc0 crc :-1 s=READY pgs=983 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:51.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:51.763 7f6c8bec3700 1 -- 172.21.2.112:0/3740619957 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6c8410a3a0 msgr2=0x7f6c840a22e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:51.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:51.763 7f6c8bec3700 1 --2- 172.21.2.112:0/3740619957 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6c8410a3a0 0x7f6c840a22e0 secure :-1 s=READY pgs=2123 cs=0 l=1 rev1=1 rx=0x7f6c7c0099a0 tx=0x7f6c7c00bcf0).stop 2021-02-10T11:40:51.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:51.763 7f6c8bec3700 1 -- 172.21.2.112:0/3740619957 shutdown_connections 2021-02-10T11:40:51.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:51.763 7f6c8bec3700 1 --2- 172.21.2.112:0/3740619957 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6c6c018730 0x7f6c6c01abc0 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:51.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:51.763 7f6c8bec3700 1 --2- 172.21.2.112:0/3740619957 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6c84109980 0x7f6c84109d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:51.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:51.763 7f6c8bec3700 1 --2- 172.21.2.112:0/3740619957 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6c8410a3a0 0x7f6c840a22e0 unknown :-1 s=CLOSED pgs=2123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:51.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:51.763 7f6c8bec3700 1 --2- 172.21.2.112:0/3740619957 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6c841a7a10 0x7f6c841a7e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:51.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:51.764 7f6c8bec3700 1 -- 172.21.2.112:0/3740619957 >> 172.21.2.112:0/3740619957 conn(0x7f6c840a1dc0 msgr2=0x7f6c840b7960 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:51.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:51.764 7f6c8bec3700 1 -- 172.21.2.112:0/3740619957 shutdown_connections 2021-02-10T11:40:51.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:51.764 7f6c8bec3700 1 -- 172.21.2.112:0/3740619957 wait complete. 2021-02-10T11:40:51.970 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 88, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:40:34.314832', 'modified': '2021-02-10 11:40:36.323257', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 10332}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10332': {'gid': 10332, 'name': 'b', 'rank': 0, 'incarnation': 85, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.120:6835/3723311838', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.120:6834', 'nonce': 3723311838}, {'type': 'v1', 'addr': '172.21.2.120:6835', 'nonce': 3723311838}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [23], 'metadata_pool': 22, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:40:51.970 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:40:51.971 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:40:52.173 DEBUG:teuthology.orchestra.run.gibba012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:40:52.177 DEBUG:teuthology.orchestra.run.gibba020:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:40:52.369 INFO:teuthology.orchestra.run.gibba020.stdout:{ 2021-02-10T11:40:52.369 INFO:teuthology.orchestra.run.gibba020.stdout: "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", 2021-02-10T11:40:52.370 INFO:teuthology.orchestra.run.gibba020.stdout: "whoami": 0, 2021-02-10T11:40:52.370 INFO:teuthology.orchestra.run.gibba020.stdout: "id": 10332, 2021-02-10T11:40:52.370 INFO:teuthology.orchestra.run.gibba020.stdout: "want_state": "up:active", 2021-02-10T11:40:52.370 INFO:teuthology.orchestra.run.gibba020.stdout: "state": "up:active", 2021-02-10T11:40:52.370 INFO:teuthology.orchestra.run.gibba020.stdout: "rank_uptime": 18.058142718999999, 2021-02-10T11:40:52.370 INFO:teuthology.orchestra.run.gibba020.stdout: "mdsmap_epoch": 88, 2021-02-10T11:40:52.370 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch": 108, 2021-02-10T11:40:52.370 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch_barrier": 108, 2021-02-10T11:40:52.370 INFO:teuthology.orchestra.run.gibba020.stdout: "uptime": 23.701535164999999 2021-02-10T11:40:52.370 INFO:teuthology.orchestra.run.gibba020.stdout:} 2021-02-10T11:40:52.373 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", "whoami": 0, "id": 10332, "want_state": "up:active", "state": "up:active", "rank_uptime": 18.058142718999999, "mdsmap_epoch": 88, "osdmap_epoch": 108, "osdmap_epoch_barrier": 108, "uptime": 23.701535164999999 } 2021-02-10T11:40:52.373 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:40:52.374 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2021-02-10T11:40:52.374 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.1 at ubuntu@gibba012.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2021-02-10T11:40:52.374 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:40:52.374 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:40:52.710 DEBUG:teuthology.orchestra.run.gibba012:> sudo modprobe fuse 2021-02-10T11:40:53.054 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:40:53.054 DEBUG:teuthology.orchestra.run.gibba012:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:40:53.397 INFO:teuthology.orchestra.run.gibba012.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:40:53.397 INFO:teuthology.orchestra.run.gibba012.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:40:53.398 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:40:53.398 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:40:53.399 DEBUG:teuthology.orchestra.run.gibba012:> ls /sys/fs/fuse/connections 2021-02-10T11:40:53.734 INFO:teuthology.orchestra.run.gibba012.stdout:42 2021-02-10T11:40:53.734 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [42] 2021-02-10T11:40:53.735 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.1 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:40:53.737 DEBUG:teuthology.orchestra.run.gibba012:> sudo modprobe fuse 2021-02-10T11:40:54.081 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:40:54.082 DEBUG:teuthology.orchestra.run.gibba012:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:40:54.148 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba012.stderr:ceph-fuse[158401]: starting ceph client 2021-02-10T11:40:54.148 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba012.stderr:2021-02-10 11:40:54.152 7fcaf0b72f80 -1 init, newargv = 0x55be1afb1bb0 newargc=9 2021-02-10T11:40:54.154 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba012.stderr:ceph-fuse[158401]: starting fuse 2021-02-10T11:40:54.424 INFO:teuthology.orchestra.run.gibba012.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:40:54.424 INFO:teuthology.orchestra.run.gibba012.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:40:54.425 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:40:54.425 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:40:54.425 DEBUG:teuthology.orchestra.run.gibba012:> ls /sys/fs/fuse/connections 2021-02-10T11:40:54.768 INFO:teuthology.orchestra.run.gibba012.stdout:42 2021-02-10T11:40:54.769 INFO:teuthology.orchestra.run.gibba012.stdout:43 2021-02-10T11:40:54.769 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42, 43] 2021-02-10T11:40:54.769 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:40:54.769 DEBUG:teuthology.orchestra.run.gibba012:> sudo python3 -c ' 2021-02-10T11:40:54.769 DEBUG:teuthology.orchestra.run.gibba012:> import glob 2021-02-10T11:40:54.769 DEBUG:teuthology.orchestra.run.gibba012:> import re 2021-02-10T11:40:54.769 DEBUG:teuthology.orchestra.run.gibba012:> import os 2021-02-10T11:40:54.769 DEBUG:teuthology.orchestra.run.gibba012:> import subprocess 2021-02-10T11:40:54.769 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:40:54.769 DEBUG:teuthology.orchestra.run.gibba012:> def find_socket(client_name): 2021-02-10T11:40:54.770 DEBUG:teuthology.orchestra.run.gibba012:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2021-02-10T11:40:54.770 DEBUG:teuthology.orchestra.run.gibba012:> files = glob.glob(asok_path) 2021-02-10T11:40:54.770 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:40:54.770 DEBUG:teuthology.orchestra.run.gibba012:> # Given a non-glob path, it better be there 2021-02-10T11:40:54.770 DEBUG:teuthology.orchestra.run.gibba012:> if "*" not in asok_path: 2021-02-10T11:40:54.770 DEBUG:teuthology.orchestra.run.gibba012:> assert(len(files) == 1) 2021-02-10T11:40:54.770 DEBUG:teuthology.orchestra.run.gibba012:> return files[0] 2021-02-10T11:40:54.770 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:40:54.770 DEBUG:teuthology.orchestra.run.gibba012:> for f in files: 2021-02-10T11:40:54.770 DEBUG:teuthology.orchestra.run.gibba012:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:40:54.770 DEBUG:teuthology.orchestra.run.gibba012:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:40:54.770 DEBUG:teuthology.orchestra.run.gibba012:> return f 2021-02-10T11:40:54.770 DEBUG:teuthology.orchestra.run.gibba012:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:40:54.770 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:40:54.771 DEBUG:teuthology.orchestra.run.gibba012:> print(find_socket("client.1")) 2021-02-10T11:40:54.771 DEBUG:teuthology.orchestra.run.gibba012:> ' 2021-02-10T11:40:55.133 INFO:teuthology.orchestra.run.gibba012.stdout:/var/run/ceph/ceph-client.1.158401.asok 2021-02-10T11:40:55.137 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.158401.asok 2021-02-10T11:40:55.137 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:40:55.137 DEBUG:teuthology.orchestra.run.gibba012:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.158401.asok status 2021-02-10T11:40:55.527 INFO:teuthology.orchestra.run.gibba012.stdout:{ 2021-02-10T11:40:55.527 INFO:teuthology.orchestra.run.gibba012.stdout: "metadata": { 2021-02-10T11:40:55.527 INFO:teuthology.orchestra.run.gibba012.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:40:55.527 INFO:teuthology.orchestra.run.gibba012.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:40:55.527 INFO:teuthology.orchestra.run.gibba012.stdout: "entity_id": "1", 2021-02-10T11:40:55.527 INFO:teuthology.orchestra.run.gibba012.stdout: "hostname": "gibba012", 2021-02-10T11:40:55.527 INFO:teuthology.orchestra.run.gibba012.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2021-02-10T11:40:55.527 INFO:teuthology.orchestra.run.gibba012.stdout: "pid": "158401", 2021-02-10T11:40:55.527 INFO:teuthology.orchestra.run.gibba012.stdout: "root": "/" 2021-02-10T11:40:55.527 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:40:55.528 INFO:teuthology.orchestra.run.gibba012.stdout: "dentry_count": 0, 2021-02-10T11:40:55.528 INFO:teuthology.orchestra.run.gibba012.stdout: "dentry_pinned_count": 0, 2021-02-10T11:40:55.528 INFO:teuthology.orchestra.run.gibba012.stdout: "id": 10521, 2021-02-10T11:40:55.528 INFO:teuthology.orchestra.run.gibba012.stdout: "inst": { 2021-02-10T11:40:55.528 INFO:teuthology.orchestra.run.gibba012.stdout: "name": { 2021-02-10T11:40:55.528 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "client", 2021-02-10T11:40:55.528 INFO:teuthology.orchestra.run.gibba012.stdout: "num": 10521 2021-02-10T11:40:55.528 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:40:55.528 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": { 2021-02-10T11:40:55.528 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:40:55.528 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:0", 2021-02-10T11:40:55.528 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 985534932 2021-02-10T11:40:55.529 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:40:55.529 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:40:55.529 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": { 2021-02-10T11:40:55.529 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:40:55.529 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:0", 2021-02-10T11:40:55.529 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 985534932 2021-02-10T11:40:55.529 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:40:55.529 INFO:teuthology.orchestra.run.gibba012.stdout: "inst_str": "client.10521 172.21.2.112:0/985534932", 2021-02-10T11:40:55.529 INFO:teuthology.orchestra.run.gibba012.stdout: "addr_str": "172.21.2.112:0/985534932", 2021-02-10T11:40:55.529 INFO:teuthology.orchestra.run.gibba012.stdout: "inode_count": 1, 2021-02-10T11:40:55.529 INFO:teuthology.orchestra.run.gibba012.stdout: "mds_epoch": 88, 2021-02-10T11:40:55.529 INFO:teuthology.orchestra.run.gibba012.stdout: "osd_epoch": 108, 2021-02-10T11:40:55.529 INFO:teuthology.orchestra.run.gibba012.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:40:55.529 INFO:teuthology.orchestra.run.gibba012.stdout: "blacklisted": false 2021-02-10T11:40:55.530 INFO:teuthology.orchestra.run.gibba012.stdout:} 2021-02-10T11:40:55.531 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:40:55.531 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:40:55.531 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:40:55.867 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:40:55.867 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:40:55.867 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:40:55.867 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:40:56.213 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2021-02-10T11:40:56.646 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:56.633 7ff7d42c4700 1 Processor -- start 2021-02-10T11:40:56.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:56.634 7ff7d42c4700 1 -- start start 2021-02-10T11:40:56.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:56.634 7ff7d42c4700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7ff7cc0d1c80 0x7ff7cc0cef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:56.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:56.634 7ff7d42c4700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7cc0d0180 con 0x7ff7cc0d1880 2021-02-10T11:40:56.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:56.634 7ff7d42c4700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7cc0fe1d0 con 0x7ff7cc0cf590 2021-02-10T11:40:56.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:56.634 7ff7d42c4700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7cc1b0cc0 con 0x7ff7cc0d1c80 2021-02-10T11:40:56.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:56.634 7ff7d18df700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7ff7cc0d1c80 0x7ff7cc0cef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:56.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:56.634 7ff7d20e0700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7ff7cc0d1880 0x7ff7cc1b04d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:54512/0 (socket says 172.21.2.112:54512) 2021-02-10T11:40:56.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:56.634 7ff7d20e0700 1 -- 172.21.2.112:0/631500539 learned_addr learned my addr 172.21.2.112:0/631500539 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:40:56.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:56.635 7ff7d18df700 1 -- 172.21.2.112:0/631500539 >> v1:172.21.2.120:6789/0 conn(0x7ff7cc0cf590 legacy=0x7ff7cc0cf990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:56.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:56.635 7ff7d18df700 1 -- 172.21.2.112:0/631500539 >> v1:172.21.2.112:6789/0 conn(0x7ff7cc0d1880 legacy=0x7ff7cc1b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:56.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:56.635 7ff7d18df700 1 -- 172.21.2.112:0/631500539 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7cc109a70 con 0x7ff7cc0d1c80 2021-02-10T11:40:56.648 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:56.635 7ff7d18df700 1 --2- 172.21.2.112:0/631500539 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff7cc0d1c80 0x7ff7cc0cef70 secure :-1 s=READY pgs=2127 cs=0 l=1 rev1=1 rx=0x7ff7c0006650 tx=0x7ff7c0009a80).ready entity=mon.0 client_cookie=f7a92ffbaa79a305 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:56.648 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:56.635 7ff7d10de700 1 -- 172.21.2.112:0/631500539 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff7c00009b0 con 0x7ff7cc0d1c80 2021-02-10T11:40:56.648 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:56.635 7ff7d10de700 1 -- 172.21.2.112:0/631500539 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7c000c0c0 con 0x7ff7cc0d1c80 2021-02-10T11:40:56.648 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:56.635 7ff7d10de700 1 -- 172.21.2.112:0/631500539 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff7c000c510 con 0x7ff7cc0d1c80 2021-02-10T11:40:56.648 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:56.635 7ff7d42c4700 1 -- 172.21.2.112:0/631500539 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff7cc0d1c80 msgr2=0x7ff7cc0cef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:56.648 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:56.635 7ff7d42c4700 1 --2- 172.21.2.112:0/631500539 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff7cc0d1c80 0x7ff7cc0cef70 secure :-1 s=READY pgs=2127 cs=0 l=1 rev1=1 rx=0x7ff7c0006650 tx=0x7ff7c0009a80).stop 2021-02-10T11:40:56.648 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:56.636 7ff7d42c4700 1 -- 172.21.2.112:0/631500539 shutdown_connections 2021-02-10T11:40:56.648 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:56.636 7ff7d42c4700 1 --2- 172.21.2.112:0/631500539 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff7cc0d1c80 0x7ff7cc0cef70 unknown :-1 s=CLOSED pgs=2127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:56.648 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:56.636 7ff7d42c4700 1 -- 172.21.2.112:0/631500539 >> 172.21.2.112:0/631500539 conn(0x7ff7cc0ce430 msgr2=0x7ff7cc0d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:56.648 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:56.636 7ff7d42c4700 1 -- 172.21.2.112:0/631500539 shutdown_connections 2021-02-10T11:40:56.648 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:56.636 7ff7d42c4700 1 -- 172.21.2.112:0/631500539 wait complete. 2021-02-10T11:40:56.650 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:56.651 7ff7d42c4700 1 Processor -- start 2021-02-10T11:40:56.650 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:56.651 7ff7d42c4700 1 -- start start 2021-02-10T11:40:56.650 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:56.651 7ff7d42c4700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff7cc1b8e70 0x7ff7cc1b2140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:56.650 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:56.651 7ff7d42c4700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff7cc1b84b0 0x7ff7cc1b2760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:56.650 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:56.651 7ff7d42c4700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff7cc1b3100 0x7ff7cc092ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:56.650 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:56.651 7ff7d42c4700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7cc1b3680 con 0x7ff7cc1b3100 2021-02-10T11:40:56.650 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:56.651 7ff7d42c4700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7cc1b2d80 con 0x7ff7cc1b8e70 2021-02-10T11:40:56.650 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:56.651 7ff7d42c4700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7cc0934f0 con 0x7ff7cc1b84b0 2021-02-10T11:40:56.650 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:56.651 7ff7d28e1700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff7cc1b3100 0x7ff7cc092ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:56.650 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:56.651 7ff7d28e1700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff7cc1b3100 0x7ff7cc092ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:55060/0 (socket says 172.21.2.112:55060) 2021-02-10T11:40:56.650 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:56.651 7ff7d28e1700 1 -- 172.21.2.112:0/864162241 learned_addr learned my addr 172.21.2.112:0/864162241 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:40:56.651 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:56.652 7ff7d28e1700 1 -- 172.21.2.112:0/864162241 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff7cc1b84b0 msgr2=0x7ff7cc1b2760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:56.651 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:56.652 7ff7d28e1700 1 --2- 172.21.2.112:0/864162241 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff7cc1b84b0 0x7ff7cc1b2760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:56.651 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:56.652 7ff7d28e1700 1 -- 172.21.2.112:0/864162241 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff7cc1b8e70 msgr2=0x7ff7cc1b2140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:56.651 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:56.652 7ff7d28e1700 1 --2- 172.21.2.112:0/864162241 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff7cc1b8e70 0x7ff7cc1b2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:56.651 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:56.652 7ff7d28e1700 1 -- 172.21.2.112:0/864162241 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7cc093710 con 0x7ff7cc1b3100 2021-02-10T11:40:56.651 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:56.652 7ff7d28e1700 1 --2- 172.21.2.112:0/864162241 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff7cc1b3100 0x7ff7cc092ed0 secure :-1 s=READY pgs=2128 cs=0 l=1 rev1=1 rx=0x7ff7c400b8e0 tx=0x7ff7c400bc80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:56.651 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:56.652 7ff7beffd700 1 -- 172.21.2.112:0/864162241 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff7c4000f90 con 0x7ff7cc1b3100 2021-02-10T11:40:56.651 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:56.652 7ff7beffd700 1 -- 172.21.2.112:0/864162241 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7c400c2f0 con 0x7ff7cc1b3100 2021-02-10T11:40:56.651 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:56.652 7ff7beffd700 1 -- 172.21.2.112:0/864162241 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff7c400c810 con 0x7ff7cc1b3100 2021-02-10T11:40:56.651 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:56.652 7ff7d42c4700 1 -- 172.21.2.112:0/864162241 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7cc093710 con 0x7ff7cc1b3100 2021-02-10T11:40:56.651 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:56.652 7ff7d42c4700 1 -- 172.21.2.112:0/864162241 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7cc093ca0 con 0x7ff7cc1b3100 2021-02-10T11:40:56.651 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:56.653 7ff7beffd700 1 -- 172.21.2.112:0/864162241 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff7c4001220 con 0x7ff7cc1b3100 2021-02-10T11:40:56.651 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:56.653 7ff7beffd700 1 --2- 172.21.2.112:0/864162241 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff7b40186d0 0x7ff7b401abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:56.651 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:56.653 7ff7beffd700 1 -- 172.21.2.112:0/864162241 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5573+0+0 (secure 0 0 0) 0x7ff7c40329c0 con 0x7ff7cc1b3100 2021-02-10T11:40:56.652 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:56.654 7ff7d20e0700 1 --2- 172.21.2.112:0/864162241 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff7b40186d0 0x7ff7b401abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:56.652 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:56.654 7ff7d20e0700 1 --2- 172.21.2.112:0/864162241 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff7b40186d0 0x7ff7b401abd0 crc :-1 s=READY pgs=984 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:56.680 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:56.684 7ff7d42c4700 1 -- 172.21.2.112:0/864162241 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7cc080340 con 0x7ff7cc1b3100 2021-02-10T11:40:56.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:56.686 7ff7beffd700 1 -- 172.21.2.112:0/864162241 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff7c4000f40 con 0x7ff7cc1b3100 2021-02-10T11:40:56.821 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:56.825 7ff7d42c4700 1 -- 172.21.2.112:0/864162241 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7ff7cc088070 con 0x7ff7cc1b3100 2021-02-10T11:40:57.186 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:57.190 7ff7beffd700 1 -- 172.21.2.112:0/864162241 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v89) v1 ==== 136+0+0 (secure 0 0 0) 0x7ff7c401f860 con 0x7ff7cc1b3100 2021-02-10T11:40:57.189 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:57.193 7ff7d42c4700 1 -- 172.21.2.112:0/864162241 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff7b40186d0 msgr2=0x7ff7b401abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:57.189 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:57.193 7ff7d42c4700 1 --2- 172.21.2.112:0/864162241 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff7b40186d0 0x7ff7b401abd0 crc :-1 s=READY pgs=984 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:57.189 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:57.193 7ff7d42c4700 1 -- 172.21.2.112:0/864162241 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff7cc1b3100 msgr2=0x7ff7cc092ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:57.189 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:57.193 7ff7d42c4700 1 --2- 172.21.2.112:0/864162241 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff7cc1b3100 0x7ff7cc092ed0 secure :-1 s=READY pgs=2128 cs=0 l=1 rev1=1 rx=0x7ff7c400b8e0 tx=0x7ff7c400bc80).stop 2021-02-10T11:40:57.190 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:57.193 7ff7d42c4700 1 -- 172.21.2.112:0/864162241 shutdown_connections 2021-02-10T11:40:57.190 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:57.193 7ff7d42c4700 1 --2- 172.21.2.112:0/864162241 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff7b40186d0 0x7ff7b401abd0 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:57.190 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:57.193 7ff7d42c4700 1 --2- 172.21.2.112:0/864162241 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff7cc1b3100 0x7ff7cc092ed0 unknown :-1 s=CLOSED pgs=2128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:57.190 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:57.193 7ff7d42c4700 1 --2- 172.21.2.112:0/864162241 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff7cc1b84b0 0x7ff7cc1b2760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:57.190 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:57.193 7ff7d42c4700 1 --2- 172.21.2.112:0/864162241 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff7cc1b8e70 0x7ff7cc1b2140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:57.190 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:57.193 7ff7d42c4700 1 -- 172.21.2.112:0/864162241 >> 172.21.2.112:0/864162241 conn(0x7ff7cc0a0390 msgr2=0x7ff7cc098150 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:57.191 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:57.193 7ff7d42c4700 1 -- 172.21.2.112:0/864162241 shutdown_connections 2021-02-10T11:40:57.191 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:57.193 7ff7d42c4700 1 -- 172.21.2.112:0/864162241 wait complete. 2021-02-10T11:40:57.198 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs volume ls 2021-02-10T11:40:57.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:57.611 7f97a508d700 1 Processor -- start 2021-02-10T11:40:57.624 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:57.611 7f97a508d700 1 -- start start 2021-02-10T11:40:57.624 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:57.611 7f97a508d700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f97a00d0340 0x7f97a00ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:57.624 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:57.611 7f97a508d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f97a00d1c70 0x7f97a00efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:57.624 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:57.611 7f97a508d700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97a00f0b10 con 0x7f97a00d1870 2021-02-10T11:40:57.624 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:57.611 7f97a508d700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97a00e5ed0 con 0x7f97a00d1c70 2021-02-10T11:40:57.624 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:57.611 7f97a508d700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97a01b0be0 con 0x7f97a00d0340 2021-02-10T11:40:57.624 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:57.611 7f979ed9d700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f97a00d1870 0x7f97a01b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:54526/0 (socket says 172.21.2.112:54526) 2021-02-10T11:40:57.624 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:57.611 7f979ed9d700 1 -- 172.21.2.112:0/31421875 learned_addr learned my addr 172.21.2.112:0/31421875 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:40:57.624 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:57.611 7f979f59e700 1 --2- 172.21.2.112:0/31421875 >> v2:172.21.2.112:3300/0 conn(0x7f97a00d1c70 0x7f97a00efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:57.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:57.612 7f979f59e700 1 -- 172.21.2.112:0/31421875 >> v2:172.21.2.120:3300/0 conn(0x7f97a00d0340 msgr2=0x7f97a00ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:57.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:57.612 7f979f59e700 1 --2- 172.21.2.112:0/31421875 >> v2:172.21.2.120:3300/0 conn(0x7f97a00d0340 0x7f97a00ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:57.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:57.612 7f979f59e700 1 -- 172.21.2.112:0/31421875 >> v1:172.21.2.112:6789/0 conn(0x7f97a00d1870 legacy=0x7f97a01b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:57.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:57.612 7f979f59e700 1 -- 172.21.2.112:0/31421875 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97a00f20a0 con 0x7f97a00d1c70 2021-02-10T11:40:57.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:57.612 7f979dd9b700 1 -- 172.21.2.112:0/31421875 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 967646198 0 0) 0x7f9788000ce0 con 0x7f97a00d1870 2021-02-10T11:40:57.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:57.612 7f979f59e700 1 --2- 172.21.2.112:0/31421875 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f97a00d1c70 0x7f97a00efc60 secure :-1 s=READY pgs=2130 cs=0 l=1 rev1=1 rx=0x7f9798006e30 tx=0x7f9798009e60).ready entity=mon.0 client_cookie=bf5ad7354c656770 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:57.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:57.612 7f979dd9b700 1 -- 172.21.2.112:0/31421875 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9798006a50 con 0x7f97a00d1c70 2021-02-10T11:40:57.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:57.612 7f979dd9b700 1 -- 172.21.2.112:0/31421875 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f979800b300 con 0x7f97a00d1c70 2021-02-10T11:40:57.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:57.612 7f979dd9b700 1 -- 172.21.2.112:0/31421875 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f979800b820 con 0x7f97a00d1c70 2021-02-10T11:40:57.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:57.612 7f97a508d700 1 -- 172.21.2.112:0/31421875 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f97a00d1c70 msgr2=0x7f97a00efc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:57.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:57.612 7f97a508d700 1 --2- 172.21.2.112:0/31421875 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f97a00d1c70 0x7f97a00efc60 secure :-1 s=READY pgs=2130 cs=0 l=1 rev1=1 rx=0x7f9798006e30 tx=0x7f9798009e60).stop 2021-02-10T11:40:57.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:57.613 7f97a508d700 1 -- 172.21.2.112:0/31421875 shutdown_connections 2021-02-10T11:40:57.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:57.613 7f97a508d700 1 --2- 172.21.2.112:0/31421875 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f97a00d1c70 0x7f97a00efc60 unknown :-1 s=CLOSED pgs=2130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:57.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:57.613 7f97a508d700 1 --2- 172.21.2.112:0/31421875 >> v2:172.21.2.120:3300/0 conn(0x7f97a00d0340 0x7f97a00ef640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:57.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:57.613 7f97a508d700 1 -- 172.21.2.112:0/31421875 >> 172.21.2.112:0/31421875 conn(0x7f97a00ce450 msgr2=0x7f97a00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:57.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:57.613 7f97a508d700 1 -- 172.21.2.112:0/31421875 shutdown_connections 2021-02-10T11:40:57.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:57.613 7f97a508d700 1 -- 172.21.2.112:0/31421875 wait complete. 2021-02-10T11:40:57.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:57.628 7f97a508d700 1 Processor -- start 2021-02-10T11:40:57.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:57.628 7f97a508d700 1 -- start start 2021-02-10T11:40:57.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:57.628 7f97a508d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f97a01908b0 0x7f97a01a4440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:57.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:57.628 7f97a508d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f97a00bd920 0x7f97a00bdd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:57.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:57.628 7f97a508d700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f97a008d510 0x7f97a008d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:57.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:57.628 7f97a508d700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97a008cf90 con 0x7f97a00bd920 2021-02-10T11:40:57.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:57.628 7f97a508d700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97a00be340 con 0x7f97a008d510 2021-02-10T11:40:57.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:57.628 7f97a508d700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97a00be680 con 0x7f97a01908b0 2021-02-10T11:40:57.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:57.628 7f979e59c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f97a00bd920 0x7f97a00bdd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:57.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:57.628 7f979e59c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f97a00bd920 0x7f97a00bdd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:55072/0 (socket says 172.21.2.112:55072) 2021-02-10T11:40:57.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:57.628 7f979e59c700 1 -- 172.21.2.112:0/2492107259 learned_addr learned my addr 172.21.2.112:0/2492107259 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:40:57.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:57.629 7f979e59c700 1 -- 172.21.2.112:0/2492107259 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f97a01908b0 msgr2=0x7f97a01a4440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:57.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:57.629 7f979e59c700 1 --2- 172.21.2.112:0/2492107259 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f97a01908b0 0x7f97a01a4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:57.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:57.629 7f979e59c700 1 -- 172.21.2.112:0/2492107259 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f97a008d510 msgr2=0x7f97a008d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:57.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:57.629 7f979e59c700 1 --2- 172.21.2.112:0/2492107259 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f97a008d510 0x7f97a008d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:57.629 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:57.629 7f979e59c700 1 -- 172.21.2.112:0/2492107259 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97a01a4a60 con 0x7f97a00bd920 2021-02-10T11:40:57.629 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:57.629 7f979e59c700 1 --2- 172.21.2.112:0/2492107259 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f97a00bd920 0x7f97a00bdd20 secure :-1 s=READY pgs=2131 cs=0 l=1 rev1=1 rx=0x7f9790006f00 tx=0x7f979000b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:57.629 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:57.629 7f97977fe700 1 -- 172.21.2.112:0/2492107259 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f979000bd30 con 0x7f97a00bd920 2021-02-10T11:40:57.629 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:57.629 7f97977fe700 1 -- 172.21.2.112:0/2492107259 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9790000c90 con 0x7f97a00bd920 2021-02-10T11:40:57.629 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:57.629 7f97a508d700 1 -- 172.21.2.112:0/2492107259 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97a01a4a60 con 0x7f97a00bd920 2021-02-10T11:40:57.629 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:57.629 7f97a508d700 1 -- 172.21.2.112:0/2492107259 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97a008df30 con 0x7f97a00bd920 2021-02-10T11:40:57.629 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:57.629 7f97977fe700 1 -- 172.21.2.112:0/2492107259 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f97900022a0 con 0x7f97a00bd920 2021-02-10T11:40:57.629 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:57.630 7f97977fe700 1 -- 172.21.2.112:0/2492107259 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9790005310 con 0x7f97a00bd920 2021-02-10T11:40:57.629 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:57.630 7f97977fe700 1 --2- 172.21.2.112:0/2492107259 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9784018730 0x7f978401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:57.629 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:57.630 7f97977fe700 1 -- 172.21.2.112:0/2492107259 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f9790032770 con 0x7f97a00bd920 2021-02-10T11:40:57.629 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:57.631 7f979ed9d700 1 --2- 172.21.2.112:0/2492107259 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9784018730 0x7f978401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:57.629 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:57.631 7f979ed9d700 1 --2- 172.21.2.112:0/2492107259 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9784018730 0x7f978401ac10 crc :-1 s=READY pgs=985 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:57.657 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:57.661 7f97a508d700 1 -- 172.21.2.112:0/2492107259 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97a00ebc50 con 0x7f97a00bd920 2021-02-10T11:40:57.660 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:57.664 7f97977fe700 1 -- 172.21.2.112:0/2492107259 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9790000960 con 0x7f97a00bd920 2021-02-10T11:40:57.793 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:57.796 7f97a508d700 1 -- 172.21.2.112:0/2492107259 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs volume ls", "target": ["mgr", ""]}) v1 -- 0x7f97a008df30 con 0x7f9784018730 2021-02-10T11:40:57.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:57.798 7f97977fe700 1 -- 172.21.2.112:0/2492107259 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+41 (crc 0 0 0) 0x7f9788000c90 con 0x7f9784018730 2021-02-10T11:40:57.796 INFO:teuthology.orchestra.run.gibba012.stdout:[ 2021-02-10T11:40:57.796 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:40:57.796 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "cephfs" 2021-02-10T11:40:57.796 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:40:57.796 INFO:teuthology.orchestra.run.gibba012.stdout:] 2021-02-10T11:40:57.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:57.802 7f97a508d700 1 -- 172.21.2.112:0/2492107259 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9784018730 msgr2=0x7f978401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:57.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:57.802 7f97a508d700 1 --2- 172.21.2.112:0/2492107259 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9784018730 0x7f978401ac10 crc :-1 s=READY pgs=985 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:57.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:57.802 7f97a508d700 1 -- 172.21.2.112:0/2492107259 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f97a00bd920 msgr2=0x7f97a00bdd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:57.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:57.802 7f97a508d700 1 --2- 172.21.2.112:0/2492107259 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f97a00bd920 0x7f97a00bdd20 secure :-1 s=READY pgs=2131 cs=0 l=1 rev1=1 rx=0x7f9790006f00 tx=0x7f979000b940).stop 2021-02-10T11:40:57.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:57.802 7f97a508d700 1 -- 172.21.2.112:0/2492107259 shutdown_connections 2021-02-10T11:40:57.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:57.802 7f97a508d700 1 --2- 172.21.2.112:0/2492107259 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9784018730 0x7f978401ac10 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:57.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:57.802 7f97a508d700 1 --2- 172.21.2.112:0/2492107259 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f97a008d510 0x7f97a008d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:57.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:57.802 7f97a508d700 1 --2- 172.21.2.112:0/2492107259 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f97a00bd920 0x7f97a00bdd20 unknown :-1 s=CLOSED pgs=2131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:57.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:57.802 7f97a508d700 1 --2- 172.21.2.112:0/2492107259 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f97a01908b0 0x7f97a01a4440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:57.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:57.802 7f97a508d700 1 -- 172.21.2.112:0/2492107259 >> 172.21.2.112:0/2492107259 conn(0x7f97a008f030 msgr2=0x7f97a00b5e80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:57.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:57.802 7f97a508d700 1 -- 172.21.2.112:0/2492107259 shutdown_connections 2021-02-10T11:40:57.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:57.802 7f97a508d700 1 -- 172.21.2.112:0/2492107259 wait complete. 2021-02-10T11:40:57.806 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:40:58.237 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.224 7fad77239700 1 Processor -- start 2021-02-10T11:40:58.237 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.224 7fad77239700 1 -- start start 2021-02-10T11:40:58.237 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.225 7fad77239700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fad700d1c70 0x7fad700cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:58.237 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.225 7fad77239700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fad700cf580 0x7fad700cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:58.237 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.225 7fad77239700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad700fe1c0 con 0x7fad700d1870 2021-02-10T11:40:58.237 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.225 7fad77239700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad701b0cb0 con 0x7fad700cf580 2021-02-10T11:40:58.237 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.225 7fad77239700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad70109a60 con 0x7fad700d1c70 2021-02-10T11:40:58.238 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.225 7fad75856700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fad700cf580 0x7fad700cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:58.238 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.225 7fad75856700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fad700cf580 0x7fad700cf980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:55082/0 (socket says 172.21.2.112:55082) 2021-02-10T11:40:58.238 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.225 7fad75856700 1 -- 172.21.2.112:0/1581717040 learned_addr learned my addr 172.21.2.112:0/1581717040 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:40:58.238 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.225 7fad74854700 1 --2- 172.21.2.112:0/1581717040 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fad700d1c70 0x7fad700cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:58.238 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.225 7fad75856700 1 -- 172.21.2.112:0/1581717040 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fad700d1c70 msgr2=0x7fad700cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:58.238 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.225 7fad75856700 1 --2- 172.21.2.112:0/1581717040 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fad700d1c70 0x7fad700cef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:58.238 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.225 7fad75856700 1 -- 172.21.2.112:0/1581717040 >> v1:172.21.2.112:6789/0 conn(0x7fad700d1870 legacy=0x7fad701b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:58.238 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.225 7fad67fff700 1 -- 172.21.2.112:0/1581717040 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1833844943 0 0) 0x7fad60000ce0 con 0x7fad700d1870 2021-02-10T11:40:58.238 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.225 7fad75856700 1 -- 172.21.2.112:0/1581717040 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad70088070 con 0x7fad700cf580 2021-02-10T11:40:58.238 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.225 7fad74854700 1 --2- 172.21.2.112:0/1581717040 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fad700d1c70 0x7fad700cef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:40:58.238 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.226 7fad75856700 1 --2- 172.21.2.112:0/1581717040 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fad700cf580 0x7fad700cf980 secure :-1 s=READY pgs=2133 cs=0 l=1 rev1=1 rx=0x7fad6c006b80 tx=0x7fad6c009b90).ready entity=mon.0 client_cookie=f3ba4ac3c81fb4cc server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:58.239 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.226 7fad67fff700 1 -- 172.21.2.112:0/1581717040 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fad6c0022e0 con 0x7fad700cf580 2021-02-10T11:40:58.239 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.226 7fad67fff700 1 -- 172.21.2.112:0/1581717040 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad6c002560 con 0x7fad700cf580 2021-02-10T11:40:58.239 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.226 7fad67fff700 1 -- 172.21.2.112:0/1581717040 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fad6c002a80 con 0x7fad700cf580 2021-02-10T11:40:58.239 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.226 7fad77239700 1 -- 172.21.2.112:0/1581717040 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fad700cf580 msgr2=0x7fad700cf980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:58.239 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.226 7fad77239700 1 --2- 172.21.2.112:0/1581717040 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fad700cf580 0x7fad700cf980 secure :-1 s=READY pgs=2133 cs=0 l=1 rev1=1 rx=0x7fad6c006b80 tx=0x7fad6c009b90).stop 2021-02-10T11:40:58.239 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.226 7fad77239700 1 -- 172.21.2.112:0/1581717040 shutdown_connections 2021-02-10T11:40:58.239 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.226 7fad77239700 1 --2- 172.21.2.112:0/1581717040 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fad700cf580 0x7fad700cf980 unknown :-1 s=CLOSED pgs=2133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:58.239 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.226 7fad77239700 1 --2- 172.21.2.112:0/1581717040 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fad700d1c70 0x7fad700cef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:58.239 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.226 7fad77239700 1 -- 172.21.2.112:0/1581717040 >> 172.21.2.112:0/1581717040 conn(0x7fad700ce420 msgr2=0x7fad700d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:58.239 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.226 7fad77239700 1 -- 172.21.2.112:0/1581717040 shutdown_connections 2021-02-10T11:40:58.239 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.226 7fad77239700 1 -- 172.21.2.112:0/1581717040 wait complete. 2021-02-10T11:40:58.241 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.241 7fad77239700 1 Processor -- start 2021-02-10T11:40:58.241 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.241 7fad77239700 1 -- start start 2021-02-10T11:40:58.241 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.241 7fad77239700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fad700b04d0 0x7fad701a7a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:58.241 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.241 7fad77239700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fad701a8090 0x7fad700c23a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:58.242 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.241 7fad77239700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fad7008d680 0x7fad700c29c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:58.242 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.241 7fad77239700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad700b0a00 con 0x7fad701a8090 2021-02-10T11:40:58.242 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.241 7fad77239700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad700c2fe0 con 0x7fad700b04d0 2021-02-10T11:40:58.242 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.241 7fad77239700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad700c3200 con 0x7fad7008d680 2021-02-10T11:40:58.242 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.242 7fad74854700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fad701a8090 0x7fad700c23a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:58.242 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.242 7fad74854700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fad701a8090 0x7fad700c23a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:55086/0 (socket says 172.21.2.112:55086) 2021-02-10T11:40:58.242 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.242 7fad74854700 1 -- 172.21.2.112:0/84144645 learned_addr learned my addr 172.21.2.112:0/84144645 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:40:58.242 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.242 7fad74854700 1 -- 172.21.2.112:0/84144645 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fad7008d680 msgr2=0x7fad700c29c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:40:58.242 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.242 7fad74854700 1 --2- 172.21.2.112:0/84144645 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fad7008d680 0x7fad700c29c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:58.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.242 7fad74854700 1 -- 172.21.2.112:0/84144645 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fad700b04d0 msgr2=0x7fad701a7a70 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:40:58.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.242 7fad74854700 1 --2- 172.21.2.112:0/84144645 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fad700b04d0 0x7fad701a7a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:58.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.242 7fad74854700 1 -- 172.21.2.112:0/84144645 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad7008e080 con 0x7fad701a8090 2021-02-10T11:40:58.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.242 7fad74854700 1 --2- 172.21.2.112:0/84144645 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fad701a8090 0x7fad700c23a0 secure :-1 s=READY pgs=2134 cs=0 l=1 rev1=1 rx=0x7fad680099a0 tx=0x7fad6800bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:58.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.242 7fad65ffb700 1 -- 172.21.2.112:0/84144645 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fad6800c2f0 con 0x7fad701a8090 2021-02-10T11:40:58.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.242 7fad65ffb700 1 -- 172.21.2.112:0/84144645 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad6800c5b0 con 0x7fad701a8090 2021-02-10T11:40:58.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.242 7fad77239700 1 -- 172.21.2.112:0/84144645 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad7008e080 con 0x7fad701a8090 2021-02-10T11:40:58.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.242 7fad65ffb700 1 -- 172.21.2.112:0/84144645 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fad6800cad0 con 0x7fad701a8090 2021-02-10T11:40:58.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.242 7fad77239700 1 -- 172.21.2.112:0/84144645 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad7008e080 con 0x7fad701a8090 2021-02-10T11:40:58.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.243 7fad65ffb700 1 -- 172.21.2.112:0/84144645 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fad68001010 con 0x7fad701a8090 2021-02-10T11:40:58.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.243 7fad65ffb700 1 --2- 172.21.2.112:0/84144645 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fad500186d0 0x7fad5001abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:58.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.243 7fad65ffb700 1 -- 172.21.2.112:0/84144645 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fad68032b40 con 0x7fad701a8090 2021-02-10T11:40:58.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.243 7fad75055700 1 --2- 172.21.2.112:0/84144645 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fad500186d0 0x7fad5001abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:58.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.244 7fad75055700 1 --2- 172.21.2.112:0/84144645 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fad500186d0 0x7fad5001abd0 crc :-1 s=READY pgs=986 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:58.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.275 7fad77239700 1 -- 172.21.2.112:0/84144645 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad70082a70 con 0x7fad701a8090 2021-02-10T11:40:58.273 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.277 7fad65ffb700 1 -- 172.21.2.112:0/84144645 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fad6800be30 con 0x7fad701a8090 2021-02-10T11:40:58.421 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.425 7fad77239700 1 -- 172.21.2.112:0/84144645 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fad70082a70 con 0x7fad701a8090 2021-02-10T11:40:58.422 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.425 7fad65ffb700 1 -- 172.21.2.112:0/84144645 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v28)=0 v28) v1 ==== 122+0+0 (secure 0 0 0) 0x7fad68001010 con 0x7fad701a8090 2021-02-10T11:40:58.423 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.427 7fad77239700 1 -- 172.21.2.112:0/84144645 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fad500186d0 msgr2=0x7fad5001abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:58.423 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.427 7fad77239700 1 --2- 172.21.2.112:0/84144645 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fad500186d0 0x7fad5001abd0 crc :-1 s=READY pgs=986 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:58.424 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.427 7fad77239700 1 -- 172.21.2.112:0/84144645 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fad701a8090 msgr2=0x7fad700c23a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:58.424 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.427 7fad77239700 1 --2- 172.21.2.112:0/84144645 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fad701a8090 0x7fad700c23a0 secure :-1 s=READY pgs=2134 cs=0 l=1 rev1=1 rx=0x7fad680099a0 tx=0x7fad6800bcf0).stop 2021-02-10T11:40:58.424 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.427 7fad77239700 1 -- 172.21.2.112:0/84144645 shutdown_connections 2021-02-10T11:40:58.424 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.427 7fad77239700 1 --2- 172.21.2.112:0/84144645 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fad500186d0 0x7fad5001abd0 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:58.424 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.427 7fad77239700 1 --2- 172.21.2.112:0/84144645 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fad7008d680 0x7fad700c29c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:58.425 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.427 7fad77239700 1 --2- 172.21.2.112:0/84144645 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fad701a8090 0x7fad700c23a0 unknown :-1 s=CLOSED pgs=2134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:58.425 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.427 7fad77239700 1 --2- 172.21.2.112:0/84144645 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fad700b04d0 0x7fad701a7a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:58.425 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.427 7fad77239700 1 -- 172.21.2.112:0/84144645 >> 172.21.2.112:0/84144645 conn(0x7fad700be530 msgr2=0x7fad7009f1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:58.425 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.428 7fad77239700 1 -- 172.21.2.112:0/84144645 shutdown_connections 2021-02-10T11:40:58.425 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.428 7fad77239700 1 -- 172.21.2.112:0/84144645 wait complete. 2021-02-10T11:40:58.432 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolumegroup create cephfs group_0000000000741504 2021-02-10T11:40:58.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.849 7f0c7abe4700 1 Processor -- start 2021-02-10T11:40:58.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.849 7f0c7abe4700 1 -- start start 2021-02-10T11:40:58.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.849 7f0c7abe4700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f0c740d1860 0x7f0c741b04b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:58.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.849 7f0c7abe4700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c740fe1b0 con 0x7f0c740cff60 2021-02-10T11:40:58.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.849 7f0c7abe4700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c74109a50 con 0x7f0c741b0ad0 2021-02-10T11:40:58.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.849 7f0c7abe4700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c74088070 con 0x7f0c740d1860 2021-02-10T11:40:58.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.849 7f0c79201700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f0c740cff60 0x7f0c74109120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:54554/0 (socket says 172.21.2.112:54554) 2021-02-10T11:40:58.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.849 7f0c79201700 1 -- 172.21.2.112:0/2735437349 learned_addr learned my addr 172.21.2.112:0/2735437349 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:40:58.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.850 7f0c737fe700 1 -- 172.21.2.112:0/2735437349 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 859014669 0 0) 0x7f0c6c001480 con 0x7f0c740cff60 2021-02-10T11:40:58.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.850 7f0c737fe700 1 -- 172.21.2.112:0/2735437349 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c5c002fe0 con 0x7f0c740cff60 2021-02-10T11:40:58.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.850 7f0c78a00700 1 --2- 172.21.2.112:0/2735437349 >> v2:172.21.2.120:3300/0 conn(0x7f0c740d1860 0x7f0c741b04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:58.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.850 7f0c737fe700 1 -- 172.21.2.112:0/2735437349 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3435853699 0 0) 0x7f0c6c001460 con 0x7f0c740cff60 2021-02-10T11:40:58.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.850 7f0c737fe700 1 -- 172.21.2.112:0/2735437349 >> v2:172.21.2.120:3300/0 conn(0x7f0c740d1860 msgr2=0x7f0c741b04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:58.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.850 7f0c737fe700 1 --2- 172.21.2.112:0/2735437349 >> v2:172.21.2.120:3300/0 conn(0x7f0c740d1860 0x7f0c741b04b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:58.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.850 7f0c737fe700 1 -- 172.21.2.112:0/2735437349 >> v1:172.21.2.120:6789/0 conn(0x7f0c741b0ad0 legacy=0x7f0c74108930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:58.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.850 7f0c737fe700 1 -- 172.21.2.112:0/2735437349 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c74088260 con 0x7f0c740cff60 2021-02-10T11:40:58.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.850 7f0c737fe700 1 -- 172.21.2.112:0/2735437349 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f0c6c001980 con 0x7f0c740cff60 2021-02-10T11:40:58.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.850 7f0c737fe700 1 -- 172.21.2.112:0/2735437349 >> v1:172.21.2.112:6789/0 conn(0x7f0c740cff60 legacy=0x7f0c74109120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:58.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.850 7f0c737fe700 1 --2- 172.21.2.112:0/2735437349 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0c5c003aa0 0x7f0c5c005ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:58.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.850 7f0c737fe700 1 --2- 172.21.2.112:0/2735437349 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0c5c006770 0x7f0c5c008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:58.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.850 7f0c737fe700 1 --2- 172.21.2.112:0/2735437349 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0c5c009590 0x7f0c5c00bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:58.865 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.850 7f0c737fe700 1 -- 172.21.2.112:0/2735437349 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c5c00c300 con 0x7f0c5c006770 2021-02-10T11:40:58.865 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.850 7f0c737fe700 1 -- 172.21.2.112:0/2735437349 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c5c00c520 con 0x7f0c5c009590 2021-02-10T11:40:58.865 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.850 7f0c737fe700 1 -- 172.21.2.112:0/2735437349 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c5c050750 con 0x7f0c5c003aa0 2021-02-10T11:40:58.865 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.850 7f0c73fff700 1 --2- 172.21.2.112:0/2735437349 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0c5c006770 0x7f0c5c008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:58.865 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.851 7f0c79201700 1 --2- 172.21.2.112:0/2735437349 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0c5c009590 0x7f0c5c00bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:58.865 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.851 7f0c78a00700 1 --2- 172.21.2.112:0/2735437349 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0c5c003aa0 0x7f0c5c005ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:58.865 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.851 7f0c73fff700 1 -- 172.21.2.112:0/2735437349 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0c5c003aa0 msgr2=0x7f0c5c005ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:58.865 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.851 7f0c73fff700 1 --2- 172.21.2.112:0/2735437349 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0c5c003aa0 0x7f0c5c005ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:58.865 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.851 7f0c73fff700 1 -- 172.21.2.112:0/2735437349 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0c5c009590 msgr2=0x7f0c5c00bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:58.865 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.851 7f0c73fff700 1 --2- 172.21.2.112:0/2735437349 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0c5c009590 0x7f0c5c00bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:58.865 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.851 7f0c73fff700 1 -- 172.21.2.112:0/2735437349 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c5c0509b0 con 0x7f0c5c006770 2021-02-10T11:40:58.865 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.851 7f0c79201700 1 -- 172.21.2.112:0/2735437349 reap_dead start 2021-02-10T11:40:58.866 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.851 7f0c73fff700 1 --2- 172.21.2.112:0/2735437349 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0c5c006770 0x7f0c5c008c90 secure :-1 s=READY pgs=2136 cs=0 l=1 rev1=1 rx=0x7f0c68006ac0 tx=0x7f0c68009af0).ready entity=mon.0 client_cookie=7621af67e280ad7b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:58.866 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.851 7f0c737fe700 1 -- 172.21.2.112:0/2735437349 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0c6800a500 con 0x7f0c5c006770 2021-02-10T11:40:58.866 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.851 7f0c737fe700 1 -- 172.21.2.112:0/2735437349 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c680009b0 con 0x7f0c5c006770 2021-02-10T11:40:58.866 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.851 7f0c737fe700 1 -- 172.21.2.112:0/2735437349 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0c6800aa70 con 0x7f0c5c006770 2021-02-10T11:40:58.866 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.851 7f0c7abe4700 1 -- 172.21.2.112:0/2735437349 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0c5c006770 msgr2=0x7f0c5c008c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:58.866 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.851 7f0c7abe4700 1 --2- 172.21.2.112:0/2735437349 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0c5c006770 0x7f0c5c008c90 secure :-1 s=READY pgs=2136 cs=0 l=1 rev1=1 rx=0x7f0c68006ac0 tx=0x7f0c68009af0).stop 2021-02-10T11:40:58.866 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.852 7f0c7abe4700 1 -- 172.21.2.112:0/2735437349 shutdown_connections 2021-02-10T11:40:58.866 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.852 7f0c7abe4700 1 --2- 172.21.2.112:0/2735437349 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0c5c006770 0x7f0c5c008c90 unknown :-1 s=CLOSED pgs=2136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:58.866 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.852 7f0c7abe4700 1 -- 172.21.2.112:0/2735437349 >> 172.21.2.112:0/2735437349 conn(0x7f0c740ce410 msgr2=0x7f0c740d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:58.866 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.852 7f0c7abe4700 1 -- 172.21.2.112:0/2735437349 shutdown_connections 2021-02-10T11:40:58.866 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.852 7f0c7abe4700 1 -- 172.21.2.112:0/2735437349 wait complete. 2021-02-10T11:40:58.867 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.866 7f0c7abe4700 1 Processor -- start 2021-02-10T11:40:58.867 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.867 7f0c7abe4700 1 -- start start 2021-02-10T11:40:58.868 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.867 7f0c7abe4700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0c740adf50 0x7f0c740ae350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:58.868 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.867 7f0c7abe4700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0c740ae970 0x7f0c740aed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:58.868 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.867 7f0c7abe4700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0c740982c0 0x7f0c740986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:58.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.867 7f0c7abe4700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c7409f450 con 0x7f0c740adf50 2021-02-10T11:40:58.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.867 7f0c7abe4700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c74098ce0 con 0x7f0c740982c0 2021-02-10T11:40:58.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.867 7f0c7abe4700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c74098ed0 con 0x7f0c740ae970 2021-02-10T11:40:58.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.867 7f0c78a00700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0c740adf50 0x7f0c740ae350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:58.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.867 7f0c78a00700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0c740adf50 0x7f0c740ae350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:55104/0 (socket says 172.21.2.112:55104) 2021-02-10T11:40:58.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.867 7f0c78a00700 1 -- 172.21.2.112:0/314037918 learned_addr learned my addr 172.21.2.112:0/314037918 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:40:58.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.867 7f0c79201700 1 --2- 172.21.2.112:0/314037918 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0c740982c0 0x7f0c740986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:58.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.867 7f0c73fff700 1 --2- 172.21.2.112:0/314037918 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0c740ae970 0x7f0c740aed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:58.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.868 7f0c78a00700 1 -- 172.21.2.112:0/314037918 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0c740ae970 msgr2=0x7f0c740aed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:58.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.868 7f0c78a00700 1 --2- 172.21.2.112:0/314037918 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0c740ae970 0x7f0c740aed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:58.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.868 7f0c78a00700 1 -- 172.21.2.112:0/314037918 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0c740982c0 msgr2=0x7f0c740986c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:58.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.868 7f0c78a00700 1 --2- 172.21.2.112:0/314037918 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0c740982c0 0x7f0c740986c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:58.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.868 7f0c78a00700 1 -- 172.21.2.112:0/314037918 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c740990f0 con 0x7f0c740adf50 2021-02-10T11:40:58.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.868 7f0c79201700 1 --2- 172.21.2.112:0/314037918 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0c740982c0 0x7f0c740986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:40:58.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.868 7f0c78a00700 1 --2- 172.21.2.112:0/314037918 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0c740adf50 0x7f0c740ae350 secure :-1 s=READY pgs=2137 cs=0 l=1 rev1=1 rx=0x7f0c64006f60 tx=0x7f0c6400b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:58.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.868 7f0c717fa700 1 -- 172.21.2.112:0/314037918 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0c6400bd60 con 0x7f0c740adf50 2021-02-10T11:40:58.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.868 7f0c717fa700 1 -- 172.21.2.112:0/314037918 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c64000c90 con 0x7f0c740adf50 2021-02-10T11:40:58.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.868 7f0c7abe4700 1 -- 172.21.2.112:0/314037918 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c740990f0 con 0x7f0c740adf50 2021-02-10T11:40:58.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.868 7f0c7abe4700 1 -- 172.21.2.112:0/314037918 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c74099680 con 0x7f0c740adf50 2021-02-10T11:40:58.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.868 7f0c717fa700 1 -- 172.21.2.112:0/314037918 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0c64002220 con 0x7f0c740adf50 2021-02-10T11:40:58.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.869 7f0c717fa700 1 -- 172.21.2.112:0/314037918 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0c640052e0 con 0x7f0c740adf50 2021-02-10T11:40:58.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.869 7f0c717fa700 1 --2- 172.21.2.112:0/314037918 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0c540186c0 0x7f0c5401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:58.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.869 7f0c717fa700 1 -- 172.21.2.112:0/314037918 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f0c64032540 con 0x7f0c740adf50 2021-02-10T11:40:58.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.869 7f0c73fff700 1 --2- 172.21.2.112:0/314037918 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0c540186c0 0x7f0c5401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:58.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.869 7f0c73fff700 1 --2- 172.21.2.112:0/314037918 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0c540186c0 0x7f0c5401abc0 crc :-1 s=READY pgs=987 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:58.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.899 7f0c7abe4700 1 -- 172.21.2.112:0/314037918 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c74082a20 con 0x7f0c740adf50 2021-02-10T11:40:58.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:58.902 7f0c717fa700 1 -- 172.21.2.112:0/314037918 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0c64000940 con 0x7f0c740adf50 2021-02-10T11:40:59.026 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:59.029 7f0c7abe4700 1 -- 172.21.2.112:0/314037918 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "target": ["mgr", ""], "group_name": "group_0000000000741504"}) v1 -- 0x7f0c74099680 con 0x7f0c540186c0 2021-02-10T11:40:59.030 INFO:tasks.ceph.mgr.x.gibba020.stderr:2021-02-10 11:40:59.032 7f6382eee700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:40:59.030 INFO:tasks.ceph.mgr.x.gibba020.stderr:2021-02-10 11:40:59.032 7f6382eee700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:40:59.030 INFO:tasks.ceph.mgr.x.gibba020.stderr:2021-02-10 11:40:59.032 7f6382eee700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:40:59.030 INFO:tasks.ceph.mgr.x.gibba020.stderr:2021-02-10 11:40:59.032 7f6382eee700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:40:59.030 INFO:tasks.ceph.mgr.x.gibba020.stderr:2021-02-10 11:40:59.032 7f6382eee700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:40:59.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:59.046 7f0c717fa700 1 -- 172.21.2.112:0/314037918 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f0c680009b0 con 0x7f0c540186c0 2021-02-10T11:40:59.059 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:59.063 7f0c7abe4700 1 -- 172.21.2.112:0/314037918 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0c540186c0 msgr2=0x7f0c5401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:59.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:59.063 7f0c7abe4700 1 --2- 172.21.2.112:0/314037918 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0c540186c0 0x7f0c5401abc0 crc :-1 s=READY pgs=987 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:59.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:59.063 7f0c7abe4700 1 -- 172.21.2.112:0/314037918 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0c740adf50 msgr2=0x7f0c740ae350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:59.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:59.063 7f0c7abe4700 1 --2- 172.21.2.112:0/314037918 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0c740adf50 0x7f0c740ae350 secure :-1 s=READY pgs=2137 cs=0 l=1 rev1=1 rx=0x7f0c64006f60 tx=0x7f0c6400b970).stop 2021-02-10T11:40:59.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:59.063 7f0c7abe4700 1 -- 172.21.2.112:0/314037918 shutdown_connections 2021-02-10T11:40:59.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:59.063 7f0c7abe4700 1 --2- 172.21.2.112:0/314037918 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0c540186c0 0x7f0c5401abc0 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:59.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:59.063 7f0c7abe4700 1 --2- 172.21.2.112:0/314037918 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0c740982c0 0x7f0c740986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:59.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:59.063 7f0c7abe4700 1 --2- 172.21.2.112:0/314037918 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0c740ae970 0x7f0c740aed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:59.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:59.063 7f0c7abe4700 1 --2- 172.21.2.112:0/314037918 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0c740adf50 0x7f0c740ae350 unknown :-1 s=CLOSED pgs=2137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:59.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:59.063 7f0c7abe4700 1 -- 172.21.2.112:0/314037918 >> 172.21.2.112:0/314037918 conn(0x7f0c740bee90 msgr2=0x7f0c740bb880 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:59.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:59.063 7f0c7abe4700 1 -- 172.21.2.112:0/314037918 shutdown_connections 2021-02-10T11:40:59.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:59.063 7f0c7abe4700 1 -- 172.21.2.112:0/314037918 wait complete. 2021-02-10T11:40:59.067 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolumegroup getpath cephfs group_0000000000741504 2021-02-10T11:40:59.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:59.484 7f007fe9f700 1 Processor -- start 2021-02-10T11:40:59.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:59.485 7f007fe9f700 1 -- start start 2021-02-10T11:40:59.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:59.485 7f007fe9f700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f00780d1870 0x7f00781b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:59.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:59.485 7f007fe9f700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f00780cff70 0x7f0078109130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:59.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:59.485 7f007fe9f700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0078109a60 con 0x7f00781b0ae0 2021-02-10T11:40:59.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:59.485 7f007fe9f700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0078109750 con 0x7f00780d1870 2021-02-10T11:40:59.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:59.485 7f007fe9f700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0078088070 con 0x7f00780cff70 2021-02-10T11:40:59.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:59.485 7f007d4ba700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f00781b0ae0 0x7f0078108940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:54572/0 (socket says 172.21.2.112:54572) 2021-02-10T11:40:59.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:59.485 7f007d4ba700 1 -- 172.21.2.112:0/2670654286 learned_addr learned my addr 172.21.2.112:0/2670654286 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:40:59.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:59.485 7f007ccb9700 1 -- 172.21.2.112:0/2670654286 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 557890663 0 0) 0x7f00740010b0 con 0x7f00781b0ae0 2021-02-10T11:40:59.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:59.486 7f007ccb9700 1 -- 172.21.2.112:0/2670654286 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0064002fe0 con 0x7f00781b0ae0 2021-02-10T11:40:59.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:59.486 7f007dcbb700 1 --2- 172.21.2.112:0/2670654286 >> v2:172.21.2.120:3300/0 conn(0x7f00780d1870 0x7f00781b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:59.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:59.486 7f007e4bc700 1 --2- 172.21.2.112:0/2670654286 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f00780cff70 0x7f0078109130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:59.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:59.486 7f007ccb9700 1 -- 172.21.2.112:0/2670654286 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 537847812 0 0) 0x7f00740010b0 con 0x7f00781b0ae0 2021-02-10T11:40:59.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:59.486 7f007ccb9700 1 -- 172.21.2.112:0/2670654286 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f00780cff70 msgr2=0x7f0078109130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:59.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:59.486 7f007ccb9700 1 --2- 172.21.2.112:0/2670654286 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f00780cff70 0x7f0078109130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:59.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:59.486 7f007ccb9700 1 -- 172.21.2.112:0/2670654286 >> v2:172.21.2.120:3300/0 conn(0x7f00780d1870 msgr2=0x7f00781b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:59.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:59.486 7f007ccb9700 1 --2- 172.21.2.112:0/2670654286 >> v2:172.21.2.120:3300/0 conn(0x7f00780d1870 0x7f00781b04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:59.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:59.486 7f007ccb9700 1 -- 172.21.2.112:0/2670654286 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00780d1c70 con 0x7f00781b0ae0 2021-02-10T11:40:59.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:59.486 7f007ccb9700 1 -- 172.21.2.112:0/2670654286 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f00740015b0 con 0x7f00781b0ae0 2021-02-10T11:40:59.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:59.486 7f007ccb9700 1 -- 172.21.2.112:0/2670654286 >> v1:172.21.2.112:6789/0 conn(0x7f00781b0ae0 legacy=0x7f0078108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:59.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:59.486 7f007ccb9700 1 --2- 172.21.2.112:0/2670654286 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0064003af0 0x7f0064006010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:59.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:59.486 7f007ccb9700 1 --2- 172.21.2.112:0/2670654286 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f006404a7b0 0x7f006404cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:59.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:59.486 7f007ccb9700 1 --2- 172.21.2.112:0/2670654286 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f006404d590 0x7f006404fa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:59.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:59.486 7f007ccb9700 1 -- 172.21.2.112:0/2670654286 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00640502d0 con 0x7f0064003af0 2021-02-10T11:40:59.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:59.486 7f007ccb9700 1 -- 172.21.2.112:0/2670654286 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00640504f0 con 0x7f006404a7b0 2021-02-10T11:40:59.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:59.486 7f007ccb9700 1 -- 172.21.2.112:0/2670654286 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0064050710 con 0x7f006404d590 2021-02-10T11:40:59.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:59.486 7f007dcbb700 1 --2- 172.21.2.112:0/2670654286 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0064003af0 0x7f0064006010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:59.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:59.486 7f007dcbb700 1 -- 172.21.2.112:0/2670654286 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f006404d590 msgr2=0x7f006404fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:59.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:59.486 7f007dcbb700 1 --2- 172.21.2.112:0/2670654286 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f006404d590 0x7f006404fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:59.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:59.486 7f007dcbb700 1 -- 172.21.2.112:0/2670654286 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f006404a7b0 msgr2=0x7f006404cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:59.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:59.486 7f007dcbb700 1 --2- 172.21.2.112:0/2670654286 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f006404a7b0 0x7f006404cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:59.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:59.486 7f007dcbb700 1 -- 172.21.2.112:0/2670654286 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00640509b0 con 0x7f0064003af0 2021-02-10T11:40:59.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:59.486 7f007dcbb700 1 --2- 172.21.2.112:0/2670654286 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0064003af0 0x7f0064006010 secure :-1 s=READY pgs=2139 cs=0 l=1 rev1=1 rx=0x7f006800c8f0 tx=0x7f006800cc90).ready entity=mon.0 client_cookie=19f26f738f1cae27 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:59.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:59.486 7f007ccb9700 1 -- 172.21.2.112:0/2670654286 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f006800cdd0 con 0x7f0064003af0 2021-02-10T11:40:59.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:59.486 7f007ccb9700 1 -- 172.21.2.112:0/2670654286 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0068009310 con 0x7f0064003af0 2021-02-10T11:40:59.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:59.486 7f007ccb9700 1 -- 172.21.2.112:0/2670654286 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0068009830 con 0x7f0064003af0 2021-02-10T11:40:59.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:59.486 7f007fe9f700 1 -- 172.21.2.112:0/2670654286 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0064003af0 msgr2=0x7f0064006010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:59.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:59.486 7f007fe9f700 1 --2- 172.21.2.112:0/2670654286 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0064003af0 0x7f0064006010 secure :-1 s=READY pgs=2139 cs=0 l=1 rev1=1 rx=0x7f006800c8f0 tx=0x7f006800cc90).stop 2021-02-10T11:40:59.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:59.486 7f007e4bc700 1 -- 172.21.2.112:0/2670654286 reap_dead start 2021-02-10T11:40:59.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:59.487 7f007fe9f700 1 -- 172.21.2.112:0/2670654286 shutdown_connections 2021-02-10T11:40:59.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:59.487 7f007fe9f700 1 -- 172.21.2.112:0/2670654286 >> 172.21.2.112:0/2670654286 conn(0x7f00780ce420 msgr2=0x7f00780d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:59.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:59.487 7f007fe9f700 1 -- 172.21.2.112:0/2670654286 shutdown_connections 2021-02-10T11:40:59.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:59.487 7f007fe9f700 1 -- 172.21.2.112:0/2670654286 wait complete. 2021-02-10T11:40:59.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:59.502 7f007fe9f700 1 Processor -- start 2021-02-10T11:40:59.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:59.502 7f007fe9f700 1 -- start start 2021-02-10T11:40:59.505 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:59.502 7f007fe9f700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f00780a7460 0x7f0078083700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:59.505 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:59.502 7f007fe9f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0078187e70 0x7f0078098960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:59.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:59.502 7f007fe9f700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0078098f80 0x7f00781a7350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:59.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:59.502 7f007fe9f700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00780995b0 con 0x7f0078187e70 2021-02-10T11:40:59.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:59.502 7f007fe9f700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0078083280 con 0x7f00780a7460 2021-02-10T11:40:59.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:59.502 7f007fe9f700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0078083460 con 0x7f0078098f80 2021-02-10T11:40:59.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:59.502 7f007d4ba700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0078187e70 0x7f0078098960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:59.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:59.502 7f007d4ba700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0078187e70 0x7f0078098960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:55118/0 (socket says 172.21.2.112:55118) 2021-02-10T11:40:59.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:59.502 7f007d4ba700 1 -- 172.21.2.112:0/1526741662 learned_addr learned my addr 172.21.2.112:0/1526741662 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:40:59.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:59.502 7f007d4ba700 1 -- 172.21.2.112:0/1526741662 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0078098f80 msgr2=0x7f00781a7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:59.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:59.502 7f007d4ba700 1 --2- 172.21.2.112:0/1526741662 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0078098f80 0x7f00781a7350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:59.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:59.502 7f007d4ba700 1 -- 172.21.2.112:0/1526741662 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f00780a7460 msgr2=0x7f0078083700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:59.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:59.502 7f007d4ba700 1 --2- 172.21.2.112:0/1526741662 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f00780a7460 0x7f0078083700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:59.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:59.503 7f007d4ba700 1 -- 172.21.2.112:0/1526741662 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00781a7970 con 0x7f0078187e70 2021-02-10T11:40:59.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:59.503 7f007d4ba700 1 --2- 172.21.2.112:0/1526741662 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0078187e70 0x7f0078098960 secure :-1 s=READY pgs=2140 cs=0 l=1 rev1=1 rx=0x7f0074001c60 tx=0x7f007400a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:59.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:59.503 7f006e7fc700 1 -- 172.21.2.112:0/1526741662 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f007400acc0 con 0x7f0078187e70 2021-02-10T11:40:59.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:59.503 7f006e7fc700 1 -- 172.21.2.112:0/1526741662 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0074004140 con 0x7f0078187e70 2021-02-10T11:40:59.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:59.503 7f006e7fc700 1 -- 172.21.2.112:0/1526741662 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0074004660 con 0x7f0078187e70 2021-02-10T11:40:59.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:59.503 7f007fe9f700 1 -- 172.21.2.112:0/1526741662 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00781a7970 con 0x7f0078187e70 2021-02-10T11:40:59.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:59.503 7f007fe9f700 1 -- 172.21.2.112:0/1526741662 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00781a7970 con 0x7f0078187e70 2021-02-10T11:40:59.508 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:59.504 7f006e7fc700 1 -- 172.21.2.112:0/1526741662 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0074004320 con 0x7f0078187e70 2021-02-10T11:40:59.508 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:59.504 7f006e7fc700 1 --2- 172.21.2.112:0/1526741662 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f00600186c0 0x7f006001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:59.508 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:59.504 7f006e7fc700 1 -- 172.21.2.112:0/1526741662 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f0074032570 con 0x7f0078187e70 2021-02-10T11:40:59.508 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:59.504 7f007dcbb700 1 --2- 172.21.2.112:0/1526741662 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f00600186c0 0x7f006001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:59.508 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:59.504 7f007dcbb700 1 --2- 172.21.2.112:0/1526741662 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f00600186c0 0x7f006001abc0 crc :-1 s=READY pgs=988 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:59.532 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:59.535 7f007fe9f700 1 -- 172.21.2.112:0/1526741662 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0078082780 con 0x7f0078187e70 2021-02-10T11:40:59.533 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:59.537 7f006e7fc700 1 -- 172.21.2.112:0/1526741662 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0074030d20 con 0x7f0078187e70 2021-02-10T11:40:59.663 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:59.667 7f007fe9f700 1 -- 172.21.2.112:0/1526741662 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolumegroup getpath", "vol_name": "cephfs", "target": ["mgr", ""], "group_name": "group_0000000000741504"}) v1 -- 0x7f0078082780 con 0x7f00600186c0 2021-02-10T11:40:59.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:59.668 7f006e7fc700 1 -- 172.21.2.112:0/1526741662 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+32 (crc 0 0 0) 0x7f0068000940 con 0x7f00600186c0 2021-02-10T11:40:59.666 INFO:teuthology.orchestra.run.gibba012.stdout:/volumes/group_0000000000741504 2021-02-10T11:40:59.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:59.671 7f007fe9f700 1 -- 172.21.2.112:0/1526741662 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f00600186c0 msgr2=0x7f006001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:59.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:59.671 7f007fe9f700 1 --2- 172.21.2.112:0/1526741662 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f00600186c0 0x7f006001abc0 crc :-1 s=READY pgs=988 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:59.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:59.671 7f007fe9f700 1 -- 172.21.2.112:0/1526741662 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0078187e70 msgr2=0x7f0078098960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:59.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:59.671 7f007fe9f700 1 --2- 172.21.2.112:0/1526741662 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0078187e70 0x7f0078098960 secure :-1 s=READY pgs=2140 cs=0 l=1 rev1=1 rx=0x7f0074001c60 tx=0x7f007400a8d0).stop 2021-02-10T11:40:59.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:59.672 7f007fe9f700 1 -- 172.21.2.112:0/1526741662 shutdown_connections 2021-02-10T11:40:59.669 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:59.672 7f007fe9f700 1 --2- 172.21.2.112:0/1526741662 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f00600186c0 0x7f006001abc0 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:59.669 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:59.672 7f007fe9f700 1 --2- 172.21.2.112:0/1526741662 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0078098f80 0x7f00781a7350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:59.669 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:59.672 7f007fe9f700 1 --2- 172.21.2.112:0/1526741662 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0078187e70 0x7f0078098960 unknown :-1 s=CLOSED pgs=2140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:59.669 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:59.672 7f007fe9f700 1 --2- 172.21.2.112:0/1526741662 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f00780a7460 0x7f0078083700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:59.669 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:59.672 7f007fe9f700 1 -- 172.21.2.112:0/1526741662 >> 172.21.2.112:0/1526741662 conn(0x7f00780affc0 msgr2=0x7f007809ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:59.669 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:59.672 7f007fe9f700 1 -- 172.21.2.112:0/1526741662 shutdown_connections 2021-02-10T11:40:59.669 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:40:59.672 7f007fe9f700 1 -- 172.21.2.112:0/1526741662 wait complete. 2021-02-10T11:40:59.677 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2021-02-10T11:40:59.677 DEBUG:teuthology.orchestra.run.gibba012:> import os 2021-02-10T11:40:59.677 DEBUG:teuthology.orchestra.run.gibba012:> import stat 2021-02-10T11:40:59.677 DEBUG:teuthology.orchestra.run.gibba012:> import json 2021-02-10T11:40:59.677 DEBUG:teuthology.orchestra.run.gibba012:> import sys 2021-02-10T11:40:59.677 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:40:59.677 DEBUG:teuthology.orchestra.run.gibba012:> try: 2021-02-10T11:40:59.677 DEBUG:teuthology.orchestra.run.gibba012:> s = os.stat("/home/ubuntu/cephtest/mnt.0/volumes/group_0000000000741504") 2021-02-10T11:40:59.677 DEBUG:teuthology.orchestra.run.gibba012:> except OSError as e: 2021-02-10T11:40:59.677 DEBUG:teuthology.orchestra.run.gibba012:> sys.exit(e.errno) 2021-02-10T11:40:59.677 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:40:59.677 DEBUG:teuthology.orchestra.run.gibba012:> attrs = ["st_mode", "st_ino", "st_dev", "st_nlink", "st_uid", "st_gid", "st_size", "st_atime", "st_mtime", "st_ctime"] 2021-02-10T11:40:59.678 DEBUG:teuthology.orchestra.run.gibba012:> print(json.dumps( 2021-02-10T11:40:59.678 DEBUG:teuthology.orchestra.run.gibba012:> dict([(a, getattr(s, a)) for a in attrs]), 2021-02-10T11:40:59.678 DEBUG:teuthology.orchestra.run.gibba012:> indent=2)) 2021-02-10T11:40:59.678 DEBUG:teuthology.orchestra.run.gibba012:> ' 2021-02-10T11:41:00.068 INFO:teuthology.orchestra.run.gibba012.stdout:{ 2021-02-10T11:41:00.068 INFO:teuthology.orchestra.run.gibba012.stdout: "st_mode": 16877, 2021-02-10T11:41:00.068 INFO:teuthology.orchestra.run.gibba012.stdout: "st_ino": 1099511627777, 2021-02-10T11:41:00.068 INFO:teuthology.orchestra.run.gibba012.stdout: "st_dev": 42, 2021-02-10T11:41:00.069 INFO:teuthology.orchestra.run.gibba012.stdout: "st_nlink": 2, 2021-02-10T11:41:00.069 INFO:teuthology.orchestra.run.gibba012.stdout: "st_uid": 0, 2021-02-10T11:41:00.069 INFO:teuthology.orchestra.run.gibba012.stdout: "st_gid": 0, 2021-02-10T11:41:00.069 INFO:teuthology.orchestra.run.gibba012.stdout: "st_size": 0, 2021-02-10T11:41:00.069 INFO:teuthology.orchestra.run.gibba012.stdout: "st_atime": 1612957259.0409071, 2021-02-10T11:41:00.069 INFO:teuthology.orchestra.run.gibba012.stdout: "st_mtime": 1612957259.0409071, 2021-02-10T11:41:00.069 INFO:teuthology.orchestra.run.gibba012.stdout: "st_ctime": 1612957259.046201 2021-02-10T11:41:00.069 INFO:teuthology.orchestra.run.gibba012.stdout:} 2021-02-10T11:41:00.251 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolumegroup rm cephfs group_0000000000741504 2021-02-10T11:41:00.721 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:00.708 7f3e2b51f700 1 Processor -- start 2021-02-10T11:41:00.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:00.708 7f3e2b51f700 1 -- start start 2021-02-10T11:41:00.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:00.709 7f3e2b51f700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f3e240d1870 0x7f3e241b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:00.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:00.709 7f3e2b51f700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e240fe1c0 con 0x7f3e241b0ae0 2021-02-10T11:41:00.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:00.709 7f3e2b51f700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e24109a60 con 0x7f3e240cff70 2021-02-10T11:41:00.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:00.709 7f3e2b51f700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e24088070 con 0x7f3e240d1870 2021-02-10T11:41:00.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:00.709 7f3e28b3a700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f3e241b0ae0 0x7f3e24108940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:54588/0 (socket says 172.21.2.112:54588) 2021-02-10T11:41:00.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:00.709 7f3e28b3a700 1 -- 172.21.2.112:0/2897020081 learned_addr learned my addr 172.21.2.112:0/2897020081 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:41:00.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:00.709 7f3e2933b700 1 --2- 172.21.2.112:0/2897020081 >> v2:172.21.2.120:3300/0 conn(0x7f3e240d1870 0x7f3e241b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:00.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:00.709 7f3e1bfff700 1 -- 172.21.2.112:0/2897020081 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 205446752 0 0) 0x7f3e1c001500 con 0x7f3e241b0ae0 2021-02-10T11:41:00.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:00.709 7f3e1bfff700 1 -- 172.21.2.112:0/2897020081 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e10002fe0 con 0x7f3e241b0ae0 2021-02-10T11:41:00.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:00.709 7f3e1bfff700 1 -- 172.21.2.112:0/2897020081 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3654059262 0 0) 0x7f3e20001080 con 0x7f3e240cff70 2021-02-10T11:41:00.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:00.709 7f3e1bfff700 1 -- 172.21.2.112:0/2897020081 --> v1:172.21.2.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e10003210 con 0x7f3e240cff70 2021-02-10T11:41:00.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:00.710 7f3e1bfff700 1 -- 172.21.2.112:0/2897020081 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4196395368 0 0) 0x7f3e1c001250 con 0x7f3e241b0ae0 2021-02-10T11:41:00.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:00.710 7f3e2933b700 1 -- 172.21.2.112:0/2897020081 >> v1:172.21.2.120:6789/0 conn(0x7f3e240cff70 legacy=0x7f3e24109130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:00.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:00.710 7f3e2933b700 1 -- 172.21.2.112:0/2897020081 >> v1:172.21.2.112:6789/0 conn(0x7f3e241b0ae0 legacy=0x7f3e24108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:00.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:00.710 7f3e2933b700 1 -- 172.21.2.112:0/2897020081 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e24088260 con 0x7f3e240d1870 2021-02-10T11:41:00.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:00.710 7f3e2933b700 1 --2- 172.21.2.112:0/2897020081 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3e240d1870 0x7f3e241b04c0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 rx=0x7f3e14006650 tx=0x7f3e14009a80).ready entity=mon.1 client_cookie=ab7ddcfa7a78cd02 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:00.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:00.710 7f3e1bfff700 1 -- 172.21.2.112:0/2897020081 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3e1400c0c0 con 0x7f3e240d1870 2021-02-10T11:41:00.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:00.710 7f3e1bfff700 1 -- 172.21.2.112:0/2897020081 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e1400c340 con 0x7f3e240d1870 2021-02-10T11:41:00.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:00.710 7f3e1bfff700 1 -- 172.21.2.112:0/2897020081 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3e1400c820 con 0x7f3e240d1870 2021-02-10T11:41:00.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:00.710 7f3e2b51f700 1 -- 172.21.2.112:0/2897020081 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3e240d1870 msgr2=0x7f3e241b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:00.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:00.710 7f3e2b51f700 1 --2- 172.21.2.112:0/2897020081 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3e240d1870 0x7f3e241b04c0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 rx=0x7f3e14006650 tx=0x7f3e14009a80).stop 2021-02-10T11:41:00.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:00.711 7f3e2b51f700 1 -- 172.21.2.112:0/2897020081 shutdown_connections 2021-02-10T11:41:00.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:00.711 7f3e2b51f700 1 --2- 172.21.2.112:0/2897020081 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3e240d1870 0x7f3e241b04c0 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:00.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:00.711 7f3e2b51f700 1 -- 172.21.2.112:0/2897020081 >> 172.21.2.112:0/2897020081 conn(0x7f3e240ce420 msgr2=0x7f3e240d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:00.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:00.711 7f3e2b51f700 1 -- 172.21.2.112:0/2897020081 shutdown_connections 2021-02-10T11:41:00.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:00.711 7f3e2b51f700 1 -- 172.21.2.112:0/2897020081 wait complete. 2021-02-10T11:41:00.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:00.725 7f3e2b51f700 1 Processor -- start 2021-02-10T11:41:00.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:00.725 7f3e2b51f700 1 -- start start 2021-02-10T11:41:00.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:00.726 7f3e2b51f700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3e24085bc0 0x7f3e240863e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:00.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:00.726 7f3e2b51f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3e240954c0 0x7f3e24096300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:00.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:00.726 7f3e2b51f700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3e240958c0 0x7f3e24096920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:00.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:00.726 7f3e2b51f700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e24095d80 con 0x7f3e240954c0 2021-02-10T11:41:00.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:00.726 7f3e2b51f700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e240bfa60 con 0x7f3e24085bc0 2021-02-10T11:41:00.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:00.726 7f3e2b51f700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e24085fc0 con 0x7f3e240958c0 2021-02-10T11:41:00.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:00.726 7f3e28b3a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3e240954c0 0x7f3e24096300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:00.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:00.726 7f3e28b3a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3e240954c0 0x7f3e24096300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:55134/0 (socket says 172.21.2.112:55134) 2021-02-10T11:41:00.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:00.726 7f3e28b3a700 1 -- 172.21.2.112:0/902477865 learned_addr learned my addr 172.21.2.112:0/902477865 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:41:00.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:00.726 7f3e2933b700 1 --2- 172.21.2.112:0/902477865 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3e24085bc0 0x7f3e240863e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:00.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:00.726 7f3e29b3c700 1 --2- 172.21.2.112:0/902477865 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3e240958c0 0x7f3e24096920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:00.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:00.727 7f3e28b3a700 1 -- 172.21.2.112:0/902477865 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3e240958c0 msgr2=0x7f3e24096920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:00.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:00.727 7f3e28b3a700 1 --2- 172.21.2.112:0/902477865 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3e240958c0 0x7f3e24096920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:00.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:00.727 7f3e28b3a700 1 -- 172.21.2.112:0/902477865 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3e24085bc0 msgr2=0x7f3e240863e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:00.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:00.727 7f3e28b3a700 1 --2- 172.21.2.112:0/902477865 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3e24085bc0 0x7f3e240863e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:00.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:00.727 7f3e28b3a700 1 -- 172.21.2.112:0/902477865 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e24086c10 con 0x7f3e240954c0 2021-02-10T11:41:00.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:00.727 7f3e28b3a700 1 --2- 172.21.2.112:0/902477865 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3e240954c0 0x7f3e24096300 secure :-1 s=READY pgs=2142 cs=0 l=1 rev1=1 rx=0x7f3e1c00a940 tx=0x7f3e1c001ce0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:00.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:00.727 7f3e19ffb700 1 -- 172.21.2.112:0/902477865 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3e1c0070c0 con 0x7f3e240954c0 2021-02-10T11:41:00.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:00.727 7f3e19ffb700 1 -- 172.21.2.112:0/902477865 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e1c007400 con 0x7f3e240954c0 2021-02-10T11:41:00.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:00.727 7f3e19ffb700 1 -- 172.21.2.112:0/902477865 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3e1c004720 con 0x7f3e240954c0 2021-02-10T11:41:00.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:00.727 7f3e2b51f700 1 -- 172.21.2.112:0/902477865 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e24086c10 con 0x7f3e240954c0 2021-02-10T11:41:00.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:00.727 7f3e2b51f700 1 -- 172.21.2.112:0/902477865 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e24097230 con 0x7f3e240954c0 2021-02-10T11:41:00.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:00.728 7f3e19ffb700 1 -- 172.21.2.112:0/902477865 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3e1c0075e0 con 0x7f3e240954c0 2021-02-10T11:41:00.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:00.728 7f3e19ffb700 1 --2- 172.21.2.112:0/902477865 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3e040186c0 0x7f3e0401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:00.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:00.728 7f3e19ffb700 1 -- 172.21.2.112:0/902477865 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f3e1c0327a0 con 0x7f3e240954c0 2021-02-10T11:41:00.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:00.728 7f3e2933b700 1 --2- 172.21.2.112:0/902477865 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3e040186c0 0x7f3e0401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:00.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:00.728 7f3e2933b700 1 --2- 172.21.2.112:0/902477865 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3e040186c0 0x7f3e0401abc0 crc :-1 s=READY pgs=989 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:00.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:00.758 7f3e2b51f700 1 -- 172.21.2.112:0/902477865 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e240fe1c0 con 0x7f3e240954c0 2021-02-10T11:41:00.757 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:00.761 7f3e19ffb700 1 -- 172.21.2.112:0/902477865 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3e1c002060 con 0x7f3e240954c0 2021-02-10T11:41:00.884 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:00.887 7f3e2b51f700 1 -- 172.21.2.112:0/902477865 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolumegroup rm", "vol_name": "cephfs", "target": ["mgr", ""], "group_name": "group_0000000000741504"}) v1 -- 0x7f3e24097230 con 0x7f3e040186c0 2021-02-10T11:41:00.888 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:00.892 7f3e19ffb700 1 -- 172.21.2.112:0/902477865 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f3e14009180 con 0x7f3e040186c0 2021-02-10T11:41:00.893 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:00.896 7f3e2b51f700 1 -- 172.21.2.112:0/902477865 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3e040186c0 msgr2=0x7f3e0401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:00.893 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:00.897 7f3e2b51f700 1 --2- 172.21.2.112:0/902477865 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3e040186c0 0x7f3e0401abc0 crc :-1 s=READY pgs=989 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:00.893 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:00.897 7f3e2b51f700 1 -- 172.21.2.112:0/902477865 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3e240954c0 msgr2=0x7f3e24096300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:00.893 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:00.897 7f3e2b51f700 1 --2- 172.21.2.112:0/902477865 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3e240954c0 0x7f3e24096300 secure :-1 s=READY pgs=2142 cs=0 l=1 rev1=1 rx=0x7f3e1c00a940 tx=0x7f3e1c001ce0).stop 2021-02-10T11:41:00.894 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:00.897 7f3e2b51f700 1 -- 172.21.2.112:0/902477865 shutdown_connections 2021-02-10T11:41:00.894 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:00.897 7f3e2b51f700 1 --2- 172.21.2.112:0/902477865 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3e040186c0 0x7f3e0401abc0 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:00.894 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:00.897 7f3e2b51f700 1 --2- 172.21.2.112:0/902477865 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3e240958c0 0x7f3e24096920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:00.894 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:00.897 7f3e2b51f700 1 --2- 172.21.2.112:0/902477865 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3e240954c0 0x7f3e24096300 unknown :-1 s=CLOSED pgs=2142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:00.894 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:00.897 7f3e2b51f700 1 --2- 172.21.2.112:0/902477865 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3e24085bc0 0x7f3e240863e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:00.894 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:00.897 7f3e2b51f700 1 -- 172.21.2.112:0/902477865 >> 172.21.2.112:0/902477865 conn(0x7f3e240bef00 msgr2=0x7f3e2409d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:00.894 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:00.897 7f3e2b51f700 1 -- 172.21.2.112:0/902477865 shutdown_connections 2021-02-10T11:41:00.894 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:00.897 7f3e2b51f700 1 -- 172.21.2.112:0/902477865 wait complete. 2021-02-10T11:41:00.902 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:41:00.902 DEBUG:teuthology.orchestra.run.gibba012:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:41:01.248 DEBUG:teuthology.orchestra.run.gibba020:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:41:01.593 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:41:01.593 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:01.594 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:41:01.594 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:41:01.930 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:41:01.930 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:41:01.930 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba012.front.sepia.ceph.com... 2021-02-10T11:41:01.930 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:41:01.931 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:41:02.282 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba012.stderr:ceph-fuse[157734]: fuse finished with error 0 and tester_r 0 2021-02-10T11:41:02.283 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:02.284 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:41:02.284 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:41:02.618 INFO:teuthology.orchestra.run.gibba012.stdout:ext2/ext3 2021-02-10T11:41:02.619 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:41:02.619 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:41:02.619 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:41:02.954 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:02.954 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:41:02.954 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:41:03.291 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:41:03.292 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:41:03.292 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba012.front.sepia.ceph.com... 2021-02-10T11:41:03.292 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:41:03.292 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:41:03.646 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba012.stderr:ceph-fuse[158401]: fuse finished with error 0 and tester_r 0 2021-02-10T11:41:03.647 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:03.647 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:41:03.647 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:41:03.982 INFO:teuthology.orchestra.run.gibba012.stdout:ext2/ext3 2021-02-10T11:41:03.983 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:41:03.983 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:41:03.983 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:41:04.318 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:04.318 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:41:04.318 DEBUG:teuthology.orchestra.run.gibba020:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:41:04.652 INFO:teuthology.orchestra.run.gibba020.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:41:04.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:41:04.653 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:41:04.653 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:41:04.653 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:41:04.988 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:04.988 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:41:04.988 DEBUG:teuthology.orchestra.run.gibba020:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:41:05.322 INFO:teuthology.orchestra.run.gibba020.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:41:05.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:41:05.322 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:41:05.322 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:41:05.323 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:41:05.656 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config rm mon mon_allow_pool_delete 2021-02-10T11:41:06.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.076 7f433c447700 1 Processor -- start 2021-02-10T11:41:06.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.076 7f433c447700 1 -- start start 2021-02-10T11:41:06.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.076 7f433c447700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f43340d0340 0x7f43340ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:06.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.076 7f433c447700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43340f0b10 con 0x7f43340d1870 2021-02-10T11:41:06.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.076 7f433c447700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43341b0be0 con 0x7f43340d1c70 2021-02-10T11:41:06.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.076 7f433c447700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43340e5ed0 con 0x7f43340d0340 2021-02-10T11:41:06.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.077 7f4339a62700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f43340d0340 0x7f43340ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:06.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.077 7f433a263700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f43340d1870 0x7f43341b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:54602/0 (socket says 172.21.2.112:54602) 2021-02-10T11:41:06.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.077 7f433a263700 1 -- 172.21.2.112:0/3220794856 learned_addr learned my addr 172.21.2.112:0/3220794856 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:41:06.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.077 7f4339a62700 1 -- 172.21.2.112:0/3220794856 >> v1:172.21.2.120:6789/0 conn(0x7f43340d1c70 legacy=0x7f43340efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:06.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.077 7f4339a62700 1 -- 172.21.2.112:0/3220794856 >> v1:172.21.2.112:6789/0 conn(0x7f43340d1870 legacy=0x7f43341b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:06.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.077 7f4339a62700 1 -- 172.21.2.112:0/3220794856 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43340f0450 con 0x7f43340d0340 2021-02-10T11:41:06.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.077 7f4339a62700 1 --2- 172.21.2.112:0/3220794856 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f43340d0340 0x7f43340ef640 secure :-1 s=READY pgs=2144 cs=0 l=1 rev1=1 rx=0x7f4328006650 tx=0x7f4328009a80).ready entity=mon.0 client_cookie=e1aef5bf44577fd7 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:06.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.077 7f4339261700 1 -- 172.21.2.112:0/3220794856 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f432800c0c0 con 0x7f43340d0340 2021-02-10T11:41:06.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.077 7f4339261700 1 -- 172.21.2.112:0/3220794856 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f432800c340 con 0x7f43340d0340 2021-02-10T11:41:06.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.077 7f4339261700 1 -- 172.21.2.112:0/3220794856 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f432800c860 con 0x7f43340d0340 2021-02-10T11:41:06.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.077 7f433c447700 1 -- 172.21.2.112:0/3220794856 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f43340d0340 msgr2=0x7f43340ef640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:06.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.077 7f433c447700 1 --2- 172.21.2.112:0/3220794856 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f43340d0340 0x7f43340ef640 secure :-1 s=READY pgs=2144 cs=0 l=1 rev1=1 rx=0x7f4328006650 tx=0x7f4328009a80).stop 2021-02-10T11:41:06.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.078 7f433c447700 1 -- 172.21.2.112:0/3220794856 shutdown_connections 2021-02-10T11:41:06.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.078 7f433c447700 1 --2- 172.21.2.112:0/3220794856 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f43340d0340 0x7f43340ef640 unknown :-1 s=CLOSED pgs=2144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:06.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.078 7f433c447700 1 -- 172.21.2.112:0/3220794856 >> 172.21.2.112:0/3220794856 conn(0x7f43340ce450 msgr2=0x7f43340d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:06.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.078 7f433c447700 1 -- 172.21.2.112:0/3220794856 shutdown_connections 2021-02-10T11:41:06.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.078 7f433c447700 1 -- 172.21.2.112:0/3220794856 wait complete. 2021-02-10T11:41:06.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.093 7f433c447700 1 Processor -- start 2021-02-10T11:41:06.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.093 7f433c447700 1 -- start start 2021-02-10T11:41:06.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.093 7f433c447700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f43341c2540 0x7f43340a6490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:06.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.093 7f433c447700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f43340a11f0 0x7f43340be510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:06.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.093 7f433c447700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f43340beb30 0x7f43340a2070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:06.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.093 7f433c447700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4334109620 con 0x7f43340a11f0 2021-02-10T11:41:06.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.093 7f433c447700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43340a15f0 con 0x7f43341c2540 2021-02-10T11:41:06.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.093 7f433c447700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43340a1810 con 0x7f43340beb30 2021-02-10T11:41:06.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.094 7f4339a62700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f43340a11f0 0x7f43340be510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:06.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.094 7f4339a62700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f43340a11f0 0x7f43340be510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:55148/0 (socket says 172.21.2.112:55148) 2021-02-10T11:41:06.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.094 7f4339a62700 1 -- 172.21.2.112:0/3345577270 learned_addr learned my addr 172.21.2.112:0/3345577270 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:41:06.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.094 7f4339a62700 1 -- 172.21.2.112:0/3345577270 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f43340beb30 msgr2=0x7f43340a2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:06.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.094 7f4339a62700 1 --2- 172.21.2.112:0/3345577270 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f43340beb30 0x7f43340a2070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:06.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.094 7f4339a62700 1 -- 172.21.2.112:0/3345577270 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f43341c2540 msgr2=0x7f43340a6490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:06.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.094 7f4339a62700 1 --2- 172.21.2.112:0/3345577270 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f43341c2540 0x7f43340a6490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:06.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.094 7f4339a62700 1 -- 172.21.2.112:0/3345577270 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43340a2690 con 0x7f43340a11f0 2021-02-10T11:41:06.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.094 7f4339a62700 1 --2- 172.21.2.112:0/3345577270 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f43340a11f0 0x7f43340be510 secure :-1 s=READY pgs=2145 cs=0 l=1 rev1=1 rx=0x7f4328009650 tx=0x7f432800b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:06.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.094 7f4326ffd700 1 -- 172.21.2.112:0/3345577270 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f432800bd50 con 0x7f43340a11f0 2021-02-10T11:41:06.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.094 7f4326ffd700 1 -- 172.21.2.112:0/3345577270 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4328006030 con 0x7f43340a11f0 2021-02-10T11:41:06.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.094 7f4326ffd700 1 -- 172.21.2.112:0/3345577270 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4328005560 con 0x7f43340a11f0 2021-02-10T11:41:06.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.094 7f433c447700 1 -- 172.21.2.112:0/3345577270 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43340a2690 con 0x7f43340a11f0 2021-02-10T11:41:06.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.094 7f433c447700 1 -- 172.21.2.112:0/3345577270 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43340a2c20 con 0x7f43340a11f0 2021-02-10T11:41:06.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.095 7f4326ffd700 1 -- 172.21.2.112:0/3345577270 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4328009300 con 0x7f43340a11f0 2021-02-10T11:41:06.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.095 7f4326ffd700 1 --2- 172.21.2.112:0/3345577270 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f431c018730 0x7f431c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:06.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.095 7f4326ffd700 1 -- 172.21.2.112:0/3345577270 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f4328032640 con 0x7f43340a11f0 2021-02-10T11:41:06.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.096 7f433a263700 1 --2- 172.21.2.112:0/3345577270 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f431c018730 0x7f431c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:06.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.096 7f433a263700 1 --2- 172.21.2.112:0/3345577270 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f431c018730 0x7f431c01abc0 crc :-1 s=READY pgs=990 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:06.123 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.126 7f433c447700 1 -- 172.21.2.112:0/3345577270 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43340f06f0 con 0x7f43340a11f0 2021-02-10T11:41:06.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.129 7f4326ffd700 1 -- 172.21.2.112:0/3345577270 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f432801f500 con 0x7f43340a11f0 2021-02-10T11:41:06.276 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.279 7f433c447700 1 -- 172.21.2.112:0/3345577270 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f43340f06f0 con 0x7f43340a11f0 2021-02-10T11:41:06.279 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.282 7f4326ffd700 1 -- 172.21.2.112:0/3345577270 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v29) v1 ==== 104+0+0 (secure 0 0 0) 0x7f4328009300 con 0x7f43340a11f0 2021-02-10T11:41:06.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.288 7f433c447700 1 -- 172.21.2.112:0/3345577270 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f431c018730 msgr2=0x7f431c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:06.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.288 7f433c447700 1 --2- 172.21.2.112:0/3345577270 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f431c018730 0x7f431c01abc0 crc :-1 s=READY pgs=990 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:06.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.288 7f433c447700 1 -- 172.21.2.112:0/3345577270 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f43340a11f0 msgr2=0x7f43340be510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:06.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.288 7f433c447700 1 --2- 172.21.2.112:0/3345577270 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f43340a11f0 0x7f43340be510 secure :-1 s=READY pgs=2145 cs=0 l=1 rev1=1 rx=0x7f4328009650 tx=0x7f432800b960).stop 2021-02-10T11:41:06.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.288 7f433c447700 1 -- 172.21.2.112:0/3345577270 shutdown_connections 2021-02-10T11:41:06.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.288 7f433c447700 1 --2- 172.21.2.112:0/3345577270 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f431c018730 0x7f431c01abc0 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:06.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.288 7f433c447700 1 --2- 172.21.2.112:0/3345577270 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f43340beb30 0x7f43340a2070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:06.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.288 7f433c447700 1 --2- 172.21.2.112:0/3345577270 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f43340a11f0 0x7f43340be510 unknown :-1 s=CLOSED pgs=2145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:06.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.288 7f433c447700 1 --2- 172.21.2.112:0/3345577270 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f43341c2540 0x7f43340a6490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:06.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.288 7f433c447700 1 -- 172.21.2.112:0/3345577270 >> 172.21.2.112:0/3345577270 conn(0x7f43340a7190 msgr2=0x7f4334102a30 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:06.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.288 7f433c447700 1 -- 172.21.2.112:0/3345577270 shutdown_connections 2021-02-10T11:41:06.287 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.288 7f433c447700 1 -- 172.21.2.112:0/3345577270 wait complete. 2021-02-10T11:41:06.293 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestVolumes.test_default_uid_gid_subvolume_group' 2021-02-10T11:41:06.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.708 7f0c3b06b700 1 Processor -- start 2021-02-10T11:41:06.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.708 7f0c3b06b700 1 -- start start 2021-02-10T11:41:06.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.709 7f0c3b06b700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f0c340d0340 0x7f0c340ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:06.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.709 7f0c3b06b700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c340f0b10 con 0x7f0c340d1c70 2021-02-10T11:41:06.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.709 7f0c3b06b700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c341b0be0 con 0x7f0c340d1870 2021-02-10T11:41:06.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.709 7f0c3b06b700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c340e5ed0 con 0x7f0c340d0340 2021-02-10T11:41:06.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.709 7f0c39688700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f0c340d1c70 0x7f0c340efc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:54620/0 (socket says 172.21.2.112:54620) 2021-02-10T11:41:06.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.709 7f0c39688700 1 -- 172.21.2.112:0/3018506103 learned_addr learned my addr 172.21.2.112:0/3018506103 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:41:06.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.709 7f0c33fff700 1 --2- 172.21.2.112:0/3018506103 >> v2:172.21.2.112:3300/0 conn(0x7f0c340d0340 0x7f0c340ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:06.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.709 7f0c337fe700 1 -- 172.21.2.112:0/3018506103 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3644233155 0 0) 0x7f0c2c000f90 con 0x7f0c340d1c70 2021-02-10T11:41:06.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.709 7f0c337fe700 1 -- 172.21.2.112:0/3018506103 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c20002fe0 con 0x7f0c340d1c70 2021-02-10T11:41:06.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.709 7f0c33fff700 1 -- 172.21.2.112:0/3018506103 >> v1:172.21.2.120:6789/0 conn(0x7f0c340d1870 legacy=0x7f0c341b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:06.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.709 7f0c33fff700 1 -- 172.21.2.112:0/3018506103 >> v1:172.21.2.112:6789/0 conn(0x7f0c340d1c70 legacy=0x7f0c340efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:06.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.709 7f0c33fff700 1 -- 172.21.2.112:0/3018506103 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c340f0450 con 0x7f0c340d0340 2021-02-10T11:41:06.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.709 7f0c33fff700 1 --2- 172.21.2.112:0/3018506103 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0c340d0340 0x7f0c340ef640 secure :-1 s=READY pgs=2147 cs=0 l=1 rev1=1 rx=0x7f0c24006650 tx=0x7f0c24009a80).ready entity=mon.0 client_cookie=e755a143b049643 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:06.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.710 7f0c337fe700 1 -- 172.21.2.112:0/3018506103 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0c240009b0 con 0x7f0c340d0340 2021-02-10T11:41:06.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.710 7f0c337fe700 1 -- 172.21.2.112:0/3018506103 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c2400c0c0 con 0x7f0c340d0340 2021-02-10T11:41:06.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.710 7f0c337fe700 1 -- 172.21.2.112:0/3018506103 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0c2400c5e0 con 0x7f0c340d0340 2021-02-10T11:41:06.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.710 7f0c3b06b700 1 -- 172.21.2.112:0/3018506103 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0c340d0340 msgr2=0x7f0c340ef640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:06.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.710 7f0c3b06b700 1 --2- 172.21.2.112:0/3018506103 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0c340d0340 0x7f0c340ef640 secure :-1 s=READY pgs=2147 cs=0 l=1 rev1=1 rx=0x7f0c24006650 tx=0x7f0c24009a80).stop 2021-02-10T11:41:06.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.710 7f0c3b06b700 1 -- 172.21.2.112:0/3018506103 shutdown_connections 2021-02-10T11:41:06.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.710 7f0c3b06b700 1 --2- 172.21.2.112:0/3018506103 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0c340d0340 0x7f0c340ef640 unknown :-1 s=CLOSED pgs=2147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:06.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.710 7f0c3b06b700 1 -- 172.21.2.112:0/3018506103 >> 172.21.2.112:0/3018506103 conn(0x7f0c340ce450 msgr2=0x7f0c340d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:06.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.711 7f0c3b06b700 1 -- 172.21.2.112:0/3018506103 shutdown_connections 2021-02-10T11:41:06.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.711 7f0c3b06b700 1 -- 172.21.2.112:0/3018506103 wait complete. 2021-02-10T11:41:06.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.725 7f0c3b06b700 1 Processor -- start 2021-02-10T11:41:06.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.725 7f0c3b06b700 1 -- start start 2021-02-10T11:41:06.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.725 7f0c3b06b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0c340eef50 0x7f0c340ef350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:06.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.726 7f0c3b06b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0c340f1ae0 0x7f0c340f1ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:06.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.726 7f0c3b06b700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0c340f2500 0x7f0c3418f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:06.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.726 7f0c3b06b700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c340f2900 con 0x7f0c340f1ae0 2021-02-10T11:41:06.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.726 7f0c3b06b700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c3418fd50 con 0x7f0c340f2500 2021-02-10T11:41:06.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.726 7f0c3b06b700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c3418ff70 con 0x7f0c340eef50 2021-02-10T11:41:06.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.726 7f0c33fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0c340f1ae0 0x7f0c340f1ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:06.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.726 7f0c33fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0c340f1ae0 0x7f0c340f1ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:55160/0 (socket says 172.21.2.112:55160) 2021-02-10T11:41:06.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.726 7f0c33fff700 1 -- 172.21.2.112:0/1989675049 learned_addr learned my addr 172.21.2.112:0/1989675049 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:41:06.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.726 7f0c33fff700 1 -- 172.21.2.112:0/1989675049 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0c340eef50 msgr2=0x7f0c340ef350 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:41:06.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.726 7f0c33fff700 1 --2- 172.21.2.112:0/1989675049 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0c340eef50 0x7f0c340ef350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:06.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.726 7f0c33fff700 1 -- 172.21.2.112:0/1989675049 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0c340f2500 msgr2=0x7f0c3418f730 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:41:06.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.726 7f0c33fff700 1 --2- 172.21.2.112:0/1989675049 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0c340f2500 0x7f0c3418f730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:06.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.726 7f0c33fff700 1 -- 172.21.2.112:0/1989675049 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c34190190 con 0x7f0c340f1ae0 2021-02-10T11:41:06.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.726 7f0c33fff700 1 --2- 172.21.2.112:0/1989675049 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0c340f1ae0 0x7f0c340f1ee0 secure :-1 s=READY pgs=2148 cs=0 l=1 rev1=1 rx=0x7f0c24009630 tx=0x7f0c2400b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:06.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.726 7f0c317fa700 1 -- 172.21.2.112:0/1989675049 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0c240093a0 con 0x7f0c340f1ae0 2021-02-10T11:41:06.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.727 7f0c317fa700 1 -- 172.21.2.112:0/1989675049 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c2400bd70 con 0x7f0c340f1ae0 2021-02-10T11:41:06.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.727 7f0c317fa700 1 -- 172.21.2.112:0/1989675049 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0c240054d0 con 0x7f0c340f1ae0 2021-02-10T11:41:06.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.727 7f0c3b06b700 1 -- 172.21.2.112:0/1989675049 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c34190190 con 0x7f0c340f1ae0 2021-02-10T11:41:06.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.727 7f0c3b06b700 1 -- 172.21.2.112:0/1989675049 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c34190720 con 0x7f0c340f1ae0 2021-02-10T11:41:06.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.727 7f0c317fa700 1 -- 172.21.2.112:0/1989675049 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0c240052e0 con 0x7f0c340f1ae0 2021-02-10T11:41:06.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.727 7f0c317fa700 1 --2- 172.21.2.112:0/1989675049 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0c180186c0 0x7f0c1801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:06.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.727 7f0c317fa700 1 -- 172.21.2.112:0/1989675049 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f0c24032490 con 0x7f0c340f1ae0 2021-02-10T11:41:06.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.728 7f0c38e87700 1 --2- 172.21.2.112:0/1989675049 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0c180186c0 0x7f0c1801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:06.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.728 7f0c38e87700 1 --2- 172.21.2.112:0/1989675049 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0c180186c0 0x7f0c1801abc0 crc :-1 s=READY pgs=991 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:06.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.758 7f0c3b06b700 1 -- 172.21.2.112:0/1989675049 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c340ee080 con 0x7f0c340f1ae0 2021-02-10T11:41:06.757 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.761 7f0c317fa700 1 -- 172.21.2.112:0/1989675049 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0c24030dc0 con 0x7f0c340f1ae0 2021-02-10T11:41:06.890 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:06.893 7f0c3b06b700 1 -- 172.21.2.112:0/1989675049 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_default_uid_gid_subvolume_group"]} v 0) v1 -- 0x7f0c340ce450 con 0x7f0c340f1ae0 2021-02-10T11:41:07.184 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:07.188 7f0c317fa700 1 -- 172.21.2.112:0/1989675049 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_default_uid_gid_subvolume_group"]}]=0 v569) v1 ==== 153+0+0 (secure 0 0 0) 0x7f0c240052e0 con 0x7f0c340f1ae0 2021-02-10T11:41:07.210 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:07.212 7f0c3b06b700 1 -- 172.21.2.112:0/1989675049 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0c180186c0 msgr2=0x7f0c1801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:07.210 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:07.212 7f0c3b06b700 1 --2- 172.21.2.112:0/1989675049 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0c180186c0 0x7f0c1801abc0 crc :-1 s=READY pgs=991 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:07.210 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:07.212 7f0c3b06b700 1 -- 172.21.2.112:0/1989675049 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0c340f1ae0 msgr2=0x7f0c340f1ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:07.210 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:07.212 7f0c3b06b700 1 --2- 172.21.2.112:0/1989675049 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0c340f1ae0 0x7f0c340f1ee0 secure :-1 s=READY pgs=2148 cs=0 l=1 rev1=1 rx=0x7f0c24009630 tx=0x7f0c2400b980).stop 2021-02-10T11:41:07.210 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:07.213 7f0c3b06b700 1 -- 172.21.2.112:0/1989675049 shutdown_connections 2021-02-10T11:41:07.210 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:07.213 7f0c3b06b700 1 --2- 172.21.2.112:0/1989675049 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0c180186c0 0x7f0c1801abc0 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:07.210 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:07.213 7f0c3b06b700 1 --2- 172.21.2.112:0/1989675049 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0c340f2500 0x7f0c3418f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:07.210 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:07.213 7f0c3b06b700 1 --2- 172.21.2.112:0/1989675049 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0c340f1ae0 0x7f0c340f1ee0 unknown :-1 s=CLOSED pgs=2148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:07.210 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:07.213 7f0c3b06b700 1 --2- 172.21.2.112:0/1989675049 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0c340eef50 0x7f0c340ef350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:07.210 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:07.213 7f0c3b06b700 1 -- 172.21.2.112:0/1989675049 >> 172.21.2.112:0/1989675049 conn(0x7f0c3419ed80 msgr2=0x7f0c34092c60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:07.211 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:07.213 7f0c3b06b700 1 -- 172.21.2.112:0/1989675049 shutdown_connections 2021-02-10T11:41:07.211 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:07.213 7f0c3b06b700 1 -- 172.21.2.112:0/1989675049 wait complete. 2021-02-10T11:41:07.217 INFO:tasks.cephfs_test_runner:test_default_uid_gid_subvolume_group (tasks.cephfs.test_volumes.TestVolumes) ... ok 2021-02-10T11:41:07.218 INFO:tasks.cephfs_test_runner:Starting test: test_multitenant_subvolumes (tasks.cephfs.test_volumes.TestVolumes) 2021-02-10T11:41:07.218 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestVolumes.test_multitenant_subvolumes' 2021-02-10T11:41:07.646 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:07.632 7f6e37f5d700 1 Processor -- start 2021-02-10T11:41:07.646 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:07.633 7f6e37f5d700 1 -- start start 2021-02-10T11:41:07.646 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:07.633 7f6e37f5d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f6e300d1870 0x7f6e301b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:07.646 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:07.633 7f6e37f5d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6e301b0a10 0x7f6e300cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:07.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:07.633 7f6e37f5d700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e300f0b10 con 0x7f6e300d1c70 2021-02-10T11:41:07.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:07.633 7f6e37f5d700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e300e5ed0 con 0x7f6e300d1870 2021-02-10T11:41:07.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:07.633 7f6e37f5d700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e300f0650 con 0x7f6e301b0a10 2021-02-10T11:41:07.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:07.633 7f6e35d79700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f6e300d1870 0x7f6e301b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:07.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:07.633 7f6e35d79700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f6e300d1870 0x7f6e301b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:55168/0 (socket says 172.21.2.112:55168) 2021-02-10T11:41:07.648 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:07.633 7f6e35d79700 1 -- 172.21.2.112:0/1396997739 learned_addr learned my addr 172.21.2.112:0/1396997739 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:41:07.648 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:07.634 7f6e35d79700 1 -- 172.21.2.112:0/1396997739 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6e301b0a10 msgr2=0x7f6e300cf330 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:41:07.648 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:07.634 7f6e35d79700 1 --2- 172.21.2.112:0/1396997739 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6e301b0a10 0x7f6e300cf330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:07.648 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:07.634 7f6e35d79700 1 -- 172.21.2.112:0/1396997739 >> v1:172.21.2.120:6789/0 conn(0x7f6e300d1c70 legacy=0x7f6e300cf950 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:41:07.648 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:07.634 7f6e35d79700 1 -- 172.21.2.112:0/1396997739 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e300f0870 con 0x7f6e300d1870 2021-02-10T11:41:07.648 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:07.634 7f6e35d79700 1 --2- 172.21.2.112:0/1396997739 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6e300d1870 0x7f6e301b03f0 secure :-1 s=READY pgs=2149 cs=0 l=1 rev1=1 rx=0x7f6e2c006700 tx=0x7f6e2c009b00).ready entity=mon.0 client_cookie=715c61678a8ddaf1 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:07.649 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:07.634 7f6e34d77700 1 -- 172.21.2.112:0/1396997739 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6e2c00c0c0 con 0x7f6e300d1870 2021-02-10T11:41:07.649 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:07.634 7f6e34d77700 1 -- 172.21.2.112:0/1396997739 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e2c00c340 con 0x7f6e300d1870 2021-02-10T11:41:07.649 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:07.634 7f6e34d77700 1 -- 172.21.2.112:0/1396997739 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6e2c00c820 con 0x7f6e300d1870 2021-02-10T11:41:07.649 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:07.634 7f6e37f5d700 1 -- 172.21.2.112:0/1396997739 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6e300d1870 msgr2=0x7f6e301b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:07.649 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:07.634 7f6e37f5d700 1 --2- 172.21.2.112:0/1396997739 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6e300d1870 0x7f6e301b03f0 secure :-1 s=READY pgs=2149 cs=0 l=1 rev1=1 rx=0x7f6e2c006700 tx=0x7f6e2c009b00).stop 2021-02-10T11:41:07.650 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:07.635 7f6e37f5d700 1 -- 172.21.2.112:0/1396997739 shutdown_connections 2021-02-10T11:41:07.650 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:07.635 7f6e37f5d700 1 --2- 172.21.2.112:0/1396997739 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6e301b0a10 0x7f6e300cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:07.650 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:07.635 7f6e37f5d700 1 --2- 172.21.2.112:0/1396997739 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6e300d1870 0x7f6e301b03f0 unknown :-1 s=CLOSED pgs=2149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:07.650 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:07.635 7f6e37f5d700 1 -- 172.21.2.112:0/1396997739 >> 172.21.2.112:0/1396997739 conn(0x7f6e300ce450 msgr2=0x7f6e300d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:07.650 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:07.635 7f6e37f5d700 1 -- 172.21.2.112:0/1396997739 shutdown_connections 2021-02-10T11:41:07.650 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:07.635 7f6e37f5d700 1 -- 172.21.2.112:0/1396997739 wait complete. 2021-02-10T11:41:07.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:07.650 7f6e37f5d700 1 Processor -- start 2021-02-10T11:41:07.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:07.650 7f6e37f5d700 1 -- start start 2021-02-10T11:41:07.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:07.650 7f6e37f5d700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6e300aefa0 0x7f6e300af3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:07.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:07.650 7f6e37f5d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6e300af9c0 0x7f6e3009b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:07.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:07.650 7f6e37f5d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6e3009bb00 0x7f6e3009bf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:07.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:07.650 7f6e37f5d700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e3018fab0 con 0x7f6e300af9c0 2021-02-10T11:41:07.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:07.650 7f6e37f5d700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e300afdc0 con 0x7f6e300aefa0 2021-02-10T11:41:07.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:07.650 7f6e37f5d700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e3018f630 con 0x7f6e3009bb00 2021-02-10T11:41:07.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:07.650 7f6e35578700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6e300af9c0 0x7f6e3009b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:07.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:07.650 7f6e35578700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6e300af9c0 0x7f6e3009b4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:55172/0 (socket says 172.21.2.112:55172) 2021-02-10T11:41:07.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:07.650 7f6e35578700 1 -- 172.21.2.112:0/320395980 learned_addr learned my addr 172.21.2.112:0/320395980 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:41:07.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:07.651 7f6e35578700 1 -- 172.21.2.112:0/320395980 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6e3009bb00 msgr2=0x7f6e3009bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:07.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:07.651 7f6e35578700 1 --2- 172.21.2.112:0/320395980 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6e3009bb00 0x7f6e3009bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:07.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:07.651 7f6e35578700 1 -- 172.21.2.112:0/320395980 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6e300aefa0 msgr2=0x7f6e300af3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:07.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:07.651 7f6e35d79700 1 --2- 172.21.2.112:0/320395980 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6e300aefa0 0x7f6e300af3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:07.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:07.651 7f6e35578700 1 --2- 172.21.2.112:0/320395980 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6e300aefa0 0x7f6e300af3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:07.657 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:07.651 7f6e35578700 1 -- 172.21.2.112:0/320395980 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e3009cce0 con 0x7f6e300af9c0 2021-02-10T11:41:07.657 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:07.651 7f6e35d79700 1 --2- 172.21.2.112:0/320395980 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6e300aefa0 0x7f6e300af3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:41:07.657 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:07.651 7f6e35578700 1 --2- 172.21.2.112:0/320395980 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6e300af9c0 0x7f6e3009b4e0 secure :-1 s=READY pgs=2150 cs=0 l=1 rev1=1 rx=0x7f6e2000b560 tx=0x7f6e2000b900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:07.657 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:07.651 7f6e267fc700 1 -- 172.21.2.112:0/320395980 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6e2000bcf0 con 0x7f6e300af9c0 2021-02-10T11:41:07.657 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:07.651 7f6e267fc700 1 -- 172.21.2.112:0/320395980 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e20006030 con 0x7f6e300af9c0 2021-02-10T11:41:07.658 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:07.651 7f6e37f5d700 1 -- 172.21.2.112:0/320395980 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e3009cce0 con 0x7f6e300af9c0 2021-02-10T11:41:07.658 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:07.651 7f6e267fc700 1 -- 172.21.2.112:0/320395980 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6e200054d0 con 0x7f6e300af9c0 2021-02-10T11:41:07.658 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:07.652 7f6e37f5d700 1 -- 172.21.2.112:0/320395980 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e3009cce0 con 0x7f6e300af9c0 2021-02-10T11:41:07.658 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:07.652 7f6e267fc700 1 -- 172.21.2.112:0/320395980 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6e20006210 con 0x7f6e300af9c0 2021-02-10T11:41:07.658 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:07.652 7f6e267fc700 1 --2- 172.21.2.112:0/320395980 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6e18018730 0x7f6e1801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:07.658 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:07.652 7f6e267fc700 1 -- 172.21.2.112:0/320395980 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f6e200327b0 con 0x7f6e300af9c0 2021-02-10T11:41:07.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:07.653 7f6e35d79700 1 --2- 172.21.2.112:0/320395980 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6e18018730 0x7f6e1801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:07.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:07.653 7f6e35d79700 1 --2- 172.21.2.112:0/320395980 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6e18018730 0x7f6e1801abc0 crc :-1 s=READY pgs=992 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:07.680 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:07.684 7f6e37f5d700 1 -- 172.21.2.112:0/320395980 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e300ee060 con 0x7f6e300af9c0 2021-02-10T11:41:07.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:07.686 7f6e267fc700 1 -- 172.21.2.112:0/320395980 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6e20006b80 con 0x7f6e300af9c0 2021-02-10T11:41:07.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:07.815 7f6e37f5d700 1 -- 172.21.2.112:0/320395980 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_multitenant_subvolumes"]} v 0) v1 -- 0x7f6e300ee060 con 0x7f6e300af9c0 2021-02-10T11:41:08.189 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:08.192 7f6e267fc700 1 -- 172.21.2.112:0/320395980 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_multitenant_subvolumes"]}]=0 v570) v1 ==== 147+0+0 (secure 0 0 0) 0x7f6e20006030 con 0x7f6e300af9c0 2021-02-10T11:41:08.231 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:08.234 7f6e37f5d700 1 -- 172.21.2.112:0/320395980 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6e18018730 msgr2=0x7f6e1801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:08.231 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:08.234 7f6e37f5d700 1 --2- 172.21.2.112:0/320395980 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6e18018730 0x7f6e1801abc0 crc :-1 s=READY pgs=992 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:08.231 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:08.234 7f6e37f5d700 1 -- 172.21.2.112:0/320395980 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6e300af9c0 msgr2=0x7f6e3009b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:08.231 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:08.234 7f6e37f5d700 1 --2- 172.21.2.112:0/320395980 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6e300af9c0 0x7f6e3009b4e0 secure :-1 s=READY pgs=2150 cs=0 l=1 rev1=1 rx=0x7f6e2000b560 tx=0x7f6e2000b900).stop 2021-02-10T11:41:08.231 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:08.234 7f6e37f5d700 1 -- 172.21.2.112:0/320395980 shutdown_connections 2021-02-10T11:41:08.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:08.234 7f6e37f5d700 1 --2- 172.21.2.112:0/320395980 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6e18018730 0x7f6e1801abc0 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:08.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:08.234 7f6e37f5d700 1 --2- 172.21.2.112:0/320395980 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6e3009bb00 0x7f6e3009bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:08.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:08.234 7f6e37f5d700 1 --2- 172.21.2.112:0/320395980 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6e300af9c0 0x7f6e3009b4e0 unknown :-1 s=CLOSED pgs=2150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:08.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:08.234 7f6e37f5d700 1 --2- 172.21.2.112:0/320395980 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6e300aefa0 0x7f6e300af3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:08.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:08.234 7f6e37f5d700 1 -- 172.21.2.112:0/320395980 >> 172.21.2.112:0/320395980 conn(0x7f6e3019f8f0 msgr2=0x7f6e300906e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:08.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:08.234 7f6e37f5d700 1 -- 172.21.2.112:0/320395980 shutdown_connections 2021-02-10T11:41:08.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:08.234 7f6e37f5d700 1 -- 172.21.2.112:0/320395980 wait complete. 2021-02-10T11:41:08.239 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:41:08.670 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:08.656 7f227fba1700 1 Processor -- start 2021-02-10T11:41:08.670 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:08.656 7f227fba1700 1 -- start start 2021-02-10T11:41:08.670 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:08.657 7f227fba1700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f22780d1c60 0x7f22780cef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:08.670 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:08.657 7f227fba1700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f22780cf570 0x7f22780cf970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:08.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:08.657 7f227fba1700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22780fe1b0 con 0x7f22780d1860 2021-02-10T11:41:08.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:08.657 7f227fba1700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22781b0ca0 con 0x7f22780cf570 2021-02-10T11:41:08.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:08.657 7f227fba1700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2278109a50 con 0x7f22780d1c60 2021-02-10T11:41:08.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:08.657 7f227e1be700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f22780cf570 0x7f22780cf970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:08.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:08.657 7f227d9bd700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f22780d1860 0x7f22781b04b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:54640/0 (socket says 172.21.2.112:54640) 2021-02-10T11:41:08.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:08.657 7f227d9bd700 1 -- 172.21.2.112:0/3768404874 learned_addr learned my addr 172.21.2.112:0/3768404874 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:41:08.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:08.658 7f227e1be700 1 -- 172.21.2.112:0/3768404874 >> v2:172.21.2.120:3300/0 conn(0x7f22780d1c60 msgr2=0x7f22780cef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:08.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:08.658 7f227e1be700 1 --2- 172.21.2.112:0/3768404874 >> v2:172.21.2.120:3300/0 conn(0x7f22780d1c60 0x7f22780cef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:08.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:08.658 7f227e1be700 1 -- 172.21.2.112:0/3768404874 >> v1:172.21.2.112:6789/0 conn(0x7f22780d1860 legacy=0x7f22781b04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:08.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:08.658 7f227e1be700 1 -- 172.21.2.112:0/3768404874 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2278088070 con 0x7f22780cf570 2021-02-10T11:41:08.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:08.658 7f227e1be700 1 --2- 172.21.2.112:0/3768404874 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f22780cf570 0x7f22780cf970 secure :-1 s=READY pgs=2152 cs=0 l=1 rev1=1 rx=0x7f2270006e30 tx=0x7f2270009e60).ready entity=mon.0 client_cookie=641cefe9bf2384ee server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:08.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:08.658 7f227c9bb700 1 -- 172.21.2.112:0/3768404874 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2270006a50 con 0x7f22780cf570 2021-02-10T11:41:08.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:08.658 7f227c9bb700 1 -- 172.21.2.112:0/3768404874 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f227000b300 con 0x7f22780cf570 2021-02-10T11:41:08.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:08.658 7f227c9bb700 1 -- 172.21.2.112:0/3768404874 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f227000b7e0 con 0x7f22780cf570 2021-02-10T11:41:08.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:08.658 7f227fba1700 1 -- 172.21.2.112:0/3768404874 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f22780cf570 msgr2=0x7f22780cf970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:08.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:08.658 7f227fba1700 1 --2- 172.21.2.112:0/3768404874 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f22780cf570 0x7f22780cf970 secure :-1 s=READY pgs=2152 cs=0 l=1 rev1=1 rx=0x7f2270006e30 tx=0x7f2270009e60).stop 2021-02-10T11:41:08.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:08.659 7f227fba1700 1 -- 172.21.2.112:0/3768404874 shutdown_connections 2021-02-10T11:41:08.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:08.659 7f227fba1700 1 --2- 172.21.2.112:0/3768404874 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f22780cf570 0x7f22780cf970 unknown :-1 s=CLOSED pgs=2152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:08.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:08.659 7f227fba1700 1 --2- 172.21.2.112:0/3768404874 >> v2:172.21.2.120:3300/0 conn(0x7f22780d1c60 0x7f22780cef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:08.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:08.659 7f227fba1700 1 -- 172.21.2.112:0/3768404874 >> 172.21.2.112:0/3768404874 conn(0x7f22780ce410 msgr2=0x7f22780d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:08.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:08.659 7f227fba1700 1 -- 172.21.2.112:0/3768404874 shutdown_connections 2021-02-10T11:41:08.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:08.659 7f227fba1700 1 -- 172.21.2.112:0/3768404874 wait complete. 2021-02-10T11:41:08.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:08.674 7f227fba1700 1 Processor -- start 2021-02-10T11:41:08.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:08.674 7f227fba1700 1 -- start start 2021-02-10T11:41:08.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:08.674 7f227fba1700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f22780bfa10 0x7f22780b82a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:08.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:08.674 7f227fba1700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f22781a4a70 0x7f2278094c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:08.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:08.674 7f227fba1700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f22780c0410 0x7f22780c0810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:08.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:08.674 7f227fba1700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22780c3a50 con 0x7f22781a4a70 2021-02-10T11:41:08.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:08.674 7f227fba1700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22780b88c0 con 0x7f22780c0410 2021-02-10T11:41:08.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:08.674 7f227fba1700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2278095260 con 0x7f22780bfa10 2021-02-10T11:41:08.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:08.675 7f227d1bc700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f22781a4a70 0x7f2278094c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:08.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:08.675 7f227d1bc700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f22781a4a70 0x7f2278094c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:55188/0 (socket says 172.21.2.112:55188) 2021-02-10T11:41:08.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:08.675 7f227d1bc700 1 -- 172.21.2.112:0/3679257172 learned_addr learned my addr 172.21.2.112:0/3679257172 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:41:08.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:08.675 7f227d9bd700 1 --2- 172.21.2.112:0/3679257172 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f22780bfa10 0x7f22780b82a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:08.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:08.675 7f227e1be700 1 --2- 172.21.2.112:0/3679257172 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f22780c0410 0x7f22780c0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:08.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:08.675 7f227d1bc700 1 -- 172.21.2.112:0/3679257172 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f22780bfa10 msgr2=0x7f22780b82a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:08.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:08.675 7f227d1bc700 1 --2- 172.21.2.112:0/3679257172 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f22780bfa10 0x7f22780b82a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:08.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:08.675 7f227d1bc700 1 -- 172.21.2.112:0/3679257172 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f22780c0410 msgr2=0x7f22780c0810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:08.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:08.675 7f227d1bc700 1 --2- 172.21.2.112:0/3679257172 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f22780c0410 0x7f22780c0810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:08.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:08.675 7f227d1bc700 1 -- 172.21.2.112:0/3679257172 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22780c3540 con 0x7f22781a4a70 2021-02-10T11:41:08.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:08.675 7f227d9bd700 1 --2- 172.21.2.112:0/3679257172 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f22780bfa10 0x7f22780b82a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:41:08.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:08.675 7f227d1bc700 1 --2- 172.21.2.112:0/3679257172 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f22781a4a70 0x7f2278094c40 secure :-1 s=READY pgs=2153 cs=0 l=1 rev1=1 rx=0x7f2268006f00 tx=0x7f226800b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:08.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:08.675 7f226e7fc700 1 -- 172.21.2.112:0/3679257172 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f226800bd30 con 0x7f22781a4a70 2021-02-10T11:41:08.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:08.676 7f226e7fc700 1 -- 172.21.2.112:0/3679257172 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2268000c90 con 0x7f22781a4a70 2021-02-10T11:41:08.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:08.676 7f226e7fc700 1 -- 172.21.2.112:0/3679257172 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f22680022a0 con 0x7f22781a4a70 2021-02-10T11:41:08.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:08.676 7f227fba1700 1 -- 172.21.2.112:0/3679257172 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22780c3540 con 0x7f22781a4a70 2021-02-10T11:41:08.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:08.676 7f227fba1700 1 -- 172.21.2.112:0/3679257172 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22780c3540 con 0x7f22781a4a70 2021-02-10T11:41:08.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:08.676 7f226e7fc700 1 -- 172.21.2.112:0/3679257172 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2268005310 con 0x7f22781a4a70 2021-02-10T11:41:08.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:08.676 7f226e7fc700 1 --2- 172.21.2.112:0/3679257172 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f22600186c0 0x7f226001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:08.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:08.676 7f226e7fc700 1 -- 172.21.2.112:0/3679257172 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f2268032560 con 0x7f22781a4a70 2021-02-10T11:41:08.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:08.677 7f227d9bd700 1 --2- 172.21.2.112:0/3679257172 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f22600186c0 0x7f226001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:08.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:08.677 7f227d9bd700 1 --2- 172.21.2.112:0/3679257172 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f22600186c0 0x7f226001abc0 crc :-1 s=READY pgs=993 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:08.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:08.706 7f227fba1700 1 -- 172.21.2.112:0/3679257172 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22780bfee0 con 0x7f22781a4a70 2021-02-10T11:41:08.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:08.709 7f226e7fc700 1 -- 172.21.2.112:0/3679257172 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2268006b90 con 0x7f22781a4a70 2021-02-10T11:41:08.855 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:08.858 7f227fba1700 1 -- 172.21.2.112:0/3679257172 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f22780bfee0 con 0x7f22781a4a70 2021-02-10T11:41:08.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:08.861 7f226e7fc700 1 -- 172.21.2.112:0/3679257172 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v30)=0 v30) v1 ==== 122+0+0 (secure 0 0 0) 0x7f2268005310 con 0x7f22781a4a70 2021-02-10T11:41:08.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:08.867 7f227fba1700 1 -- 172.21.2.112:0/3679257172 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f22600186c0 msgr2=0x7f226001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:08.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:08.867 7f227fba1700 1 --2- 172.21.2.112:0/3679257172 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f22600186c0 0x7f226001abc0 crc :-1 s=READY pgs=993 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:08.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:08.867 7f227fba1700 1 -- 172.21.2.112:0/3679257172 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f22781a4a70 msgr2=0x7f2278094c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:08.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:08.867 7f227fba1700 1 --2- 172.21.2.112:0/3679257172 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f22781a4a70 0x7f2278094c40 secure :-1 s=READY pgs=2153 cs=0 l=1 rev1=1 rx=0x7f2268006f00 tx=0x7f226800b940).stop 2021-02-10T11:41:08.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:08.867 7f227fba1700 1 -- 172.21.2.112:0/3679257172 shutdown_connections 2021-02-10T11:41:08.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:08.867 7f227fba1700 1 --2- 172.21.2.112:0/3679257172 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f22600186c0 0x7f226001abc0 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:08.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:08.867 7f227fba1700 1 --2- 172.21.2.112:0/3679257172 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f22780c0410 0x7f22780c0810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:08.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:08.867 7f227fba1700 1 --2- 172.21.2.112:0/3679257172 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f22781a4a70 0x7f2278094c40 unknown :-1 s=CLOSED pgs=2153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:08.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:08.867 7f227fba1700 1 --2- 172.21.2.112:0/3679257172 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f22780bfa10 0x7f22780b82a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:08.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:08.867 7f227fba1700 1 -- 172.21.2.112:0/3679257172 >> 172.21.2.112:0/3679257172 conn(0x7f22780a6a80 msgr2=0x7f22780ae110 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:08.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:08.867 7f227fba1700 1 -- 172.21.2.112:0/3679257172 shutdown_connections 2021-02-10T11:41:08.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:08.867 7f227fba1700 1 -- 172.21.2.112:0/3679257172 wait complete. 2021-02-10T11:41:08.872 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:41:08.872 DEBUG:teuthology.orchestra.run.gibba012:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:41:09.217 DEBUG:teuthology.orchestra.run.gibba020:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:41:09.563 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:41:09.564 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:09.564 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:41:09.564 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:41:09.899 INFO:teuthology.orchestra.run.gibba012.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.0’: No such file or directory 2021-02-10T11:41:09.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:41:09.899 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2021-02-10T11:41:09.900 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:09.900 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:41:09.900 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:41:10.234 INFO:teuthology.orchestra.run.gibba012.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: No such file or directory 2021-02-10T11:41:10.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:41:10.234 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2021-02-10T11:41:10.235 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:10.235 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:41:10.235 DEBUG:teuthology.orchestra.run.gibba020:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:41:10.569 INFO:teuthology.orchestra.run.gibba020.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:41:10.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:41:10.569 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:41:10.569 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:10.570 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:41:10.570 DEBUG:teuthology.orchestra.run.gibba020:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:41:10.904 INFO:teuthology.orchestra.run.gibba020.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:41:10.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:41:10.904 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:41:10.905 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json-pretty 2021-02-10T11:41:11.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:11.318 7f4c45ddd700 1 Processor -- start 2021-02-10T11:41:11.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:11.318 7f4c45ddd700 1 -- start start 2021-02-10T11:41:11.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:11.318 7f4c45ddd700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4c400d1870 0x7f4c401b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:11.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:11.318 7f4c45ddd700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f4c401b0a10 0x7f4c400cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:11.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:11.318 7f4c45ddd700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c400f0b10 con 0x7f4c400d1c70 2021-02-10T11:41:11.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:11.318 7f4c45ddd700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c400fe230 con 0x7f4c401b0a10 2021-02-10T11:41:11.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:11.318 7f4c45ddd700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c400e5ed0 con 0x7f4c400d1870 2021-02-10T11:41:11.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:11.319 7f4c3ffff700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f4c400d1c70 0x7f4c400cf950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:54654/0 (socket says 172.21.2.112:54654) 2021-02-10T11:41:11.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:11.319 7f4c3ffff700 1 -- 172.21.2.112:0/380251515 learned_addr learned my addr 172.21.2.112:0/380251515 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:41:11.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:11.319 7f4c3effd700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f4c401b0a10 0x7f4c400cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:11.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:11.319 7f4c3e7fc700 1 -- 172.21.2.112:0/380251515 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2936910013 0 0) 0x7f4c38001500 con 0x7f4c400d1c70 2021-02-10T11:41:11.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:11.319 7f4c3e7fc700 1 -- 172.21.2.112:0/380251515 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c2c002fe0 con 0x7f4c400d1c70 2021-02-10T11:41:11.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:11.319 7f4c3effd700 1 -- 172.21.2.112:0/380251515 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4c400d1870 msgr2=0x7f4c401b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:11.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:11.319 7f4c3effd700 1 --2- 172.21.2.112:0/380251515 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4c400d1870 0x7f4c401b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:11.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:11.319 7f4c3effd700 1 -- 172.21.2.112:0/380251515 >> v1:172.21.2.112:6789/0 conn(0x7f4c400d1c70 legacy=0x7f4c400cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:11.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:11.319 7f4c3effd700 1 -- 172.21.2.112:0/380251515 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c400f0650 con 0x7f4c401b0a10 2021-02-10T11:41:11.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:11.319 7f4c3effd700 1 --2- 172.21.2.112:0/380251515 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4c401b0a10 0x7f4c400cf330 secure :-1 s=READY pgs=2155 cs=0 l=1 rev1=1 rx=0x7f4c30006650 tx=0x7f4c30009a80).ready entity=mon.0 client_cookie=4307b786d3262bbb server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:11.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:11.320 7f4c3e7fc700 1 -- 172.21.2.112:0/380251515 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4c300009b0 con 0x7f4c401b0a10 2021-02-10T11:41:11.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:11.320 7f4c3e7fc700 1 -- 172.21.2.112:0/380251515 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c3000c0c0 con 0x7f4c401b0a10 2021-02-10T11:41:11.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:11.320 7f4c3e7fc700 1 -- 172.21.2.112:0/380251515 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4c3000c5e0 con 0x7f4c401b0a10 2021-02-10T11:41:11.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:11.320 7f4c45ddd700 1 -- 172.21.2.112:0/380251515 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4c401b0a10 msgr2=0x7f4c400cf330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:11.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:11.320 7f4c45ddd700 1 --2- 172.21.2.112:0/380251515 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4c401b0a10 0x7f4c400cf330 secure :-1 s=READY pgs=2155 cs=0 l=1 rev1=1 rx=0x7f4c30006650 tx=0x7f4c30009a80).stop 2021-02-10T11:41:11.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:11.320 7f4c45ddd700 1 -- 172.21.2.112:0/380251515 shutdown_connections 2021-02-10T11:41:11.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:11.320 7f4c45ddd700 1 --2- 172.21.2.112:0/380251515 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4c401b0a10 0x7f4c400cf330 unknown :-1 s=CLOSED pgs=2155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:11.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:11.320 7f4c45ddd700 1 --2- 172.21.2.112:0/380251515 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4c400d1870 0x7f4c401b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:11.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:11.320 7f4c45ddd700 1 -- 172.21.2.112:0/380251515 >> 172.21.2.112:0/380251515 conn(0x7f4c400ce450 msgr2=0x7f4c400d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:11.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:11.320 7f4c45ddd700 1 -- 172.21.2.112:0/380251515 shutdown_connections 2021-02-10T11:41:11.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:11.320 7f4c45ddd700 1 -- 172.21.2.112:0/380251515 wait complete. 2021-02-10T11:41:11.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:11.335 7f4c45ddd700 1 Processor -- start 2021-02-10T11:41:11.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:11.335 7f4c45ddd700 1 -- start start 2021-02-10T11:41:11.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:11.335 7f4c45ddd700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4c4018d870 0x7f4c400b19b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:11.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:11.335 7f4c45ddd700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4c400b1fd0 0x7f4c40099ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:11.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:11.335 7f4c45ddd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4c400b23d0 0x7f4c4009a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:11.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:11.335 7f4c45ddd700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c4018dc70 con 0x7f4c400b23d0 2021-02-10T11:41:11.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:11.335 7f4c45ddd700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c40199e60 con 0x7f4c400b1fd0 2021-02-10T11:41:11.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:11.335 7f4c45ddd700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c4009ab30 con 0x7f4c4018d870 2021-02-10T11:41:11.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:11.335 7f4c3ffff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4c400b23d0 0x7f4c4009a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:11.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:11.335 7f4c3ffff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4c400b23d0 0x7f4c4009a510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:55202/0 (socket says 172.21.2.112:55202) 2021-02-10T11:41:11.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:11.335 7f4c3ffff700 1 -- 172.21.2.112:0/2518235275 learned_addr learned my addr 172.21.2.112:0/2518235275 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:41:11.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:11.336 7f4c3effd700 1 --2- 172.21.2.112:0/2518235275 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4c400b1fd0 0x7f4c40099ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:11.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:11.336 7f4c3f7fe700 1 --2- 172.21.2.112:0/2518235275 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4c4018d870 0x7f4c400b19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:11.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:11.336 7f4c3ffff700 1 -- 172.21.2.112:0/2518235275 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4c4018d870 msgr2=0x7f4c400b19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:11.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:11.336 7f4c3ffff700 1 --2- 172.21.2.112:0/2518235275 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4c4018d870 0x7f4c400b19b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:11.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:11.336 7f4c3ffff700 1 -- 172.21.2.112:0/2518235275 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4c400b1fd0 msgr2=0x7f4c40099ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:11.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:11.336 7f4c3ffff700 1 --2- 172.21.2.112:0/2518235275 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4c400b1fd0 0x7f4c40099ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:11.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:11.336 7f4c3ffff700 1 -- 172.21.2.112:0/2518235275 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c4009ad50 con 0x7f4c400b23d0 2021-02-10T11:41:11.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:11.336 7f4c3effd700 1 --2- 172.21.2.112:0/2518235275 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4c400b1fd0 0x7f4c40099ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:41:11.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:11.336 7f4c3ffff700 1 --2- 172.21.2.112:0/2518235275 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4c400b23d0 0x7f4c4009a510 secure :-1 s=READY pgs=2156 cs=0 l=1 rev1=1 rx=0x7f4c38009980 tx=0x7f4c3800bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:11.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:11.336 7f4c37fff700 1 -- 172.21.2.112:0/2518235275 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4c3800be10 con 0x7f4c400b23d0 2021-02-10T11:41:11.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:11.336 7f4c37fff700 1 -- 172.21.2.112:0/2518235275 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c38006030 con 0x7f4c400b23d0 2021-02-10T11:41:11.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:11.336 7f4c45ddd700 1 -- 172.21.2.112:0/2518235275 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c4009ad50 con 0x7f4c400b23d0 2021-02-10T11:41:11.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:11.336 7f4c45ddd700 1 -- 172.21.2.112:0/2518235275 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c4018e3f0 con 0x7f4c400b23d0 2021-02-10T11:41:11.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:11.337 7f4c37fff700 1 -- 172.21.2.112:0/2518235275 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4c38004720 con 0x7f4c400b23d0 2021-02-10T11:41:11.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:11.337 7f4c37fff700 1 -- 172.21.2.112:0/2518235275 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4c38006210 con 0x7f4c400b23d0 2021-02-10T11:41:11.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:11.337 7f4c37fff700 1 --2- 172.21.2.112:0/2518235275 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4c240186c0 0x7f4c2401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:11.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:11.337 7f4c37fff700 1 -- 172.21.2.112:0/2518235275 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f4c380328a0 con 0x7f4c400b23d0 2021-02-10T11:41:11.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:11.338 7f4c3f7fe700 1 --2- 172.21.2.112:0/2518235275 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4c240186c0 0x7f4c2401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:11.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:11.338 7f4c3f7fe700 1 --2- 172.21.2.112:0/2518235275 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4c240186c0 0x7f4c2401abc0 crc :-1 s=READY pgs=994 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:11.366 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:11.368 7f4c45ddd700 1 -- 172.21.2.112:0/2518235275 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c400999c0 con 0x7f4c400b23d0 2021-02-10T11:41:11.369 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:11.371 7f4c37fff700 1 -- 172.21.2.112:0/2518235275 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4c38025060 con 0x7f4c400b23d0 2021-02-10T11:41:11.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:11.494 7f4c45ddd700 1 -- 172.21.2.112:0/2518235275 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f4c400999c0 con 0x7f4c400b23d0 2021-02-10T11:41:11.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:11.495 7f4c37fff700 1 -- 172.21.2.112:0/2518235275 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v108) v1 ==== 81+0+30373 (secure 0 0 0) 0x7f4c38025060 con 0x7f4c400b23d0 2021-02-10T11:41:11.493 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:41:11.494 INFO:teuthology.orchestra.run.gibba012.stdout:{ 2021-02-10T11:41:11.494 INFO:teuthology.orchestra.run.gibba012.stdout: "epoch": 108, 2021-02-10T11:41:11.494 INFO:teuthology.orchestra.run.gibba012.stdout: "fsid": "f14f004e-0699-4584-ad81-3754258493de", 2021-02-10T11:41:11.494 INFO:teuthology.orchestra.run.gibba012.stdout: "created": "2021-02-10 11:13:26.501424", 2021-02-10T11:41:11.494 INFO:teuthology.orchestra.run.gibba012.stdout: "modified": "2021-02-10 11:40:35.316267", 2021-02-10T11:41:11.494 INFO:teuthology.orchestra.run.gibba012.stdout: "last_up_change": "2021-02-10 11:13:52.780316", 2021-02-10T11:41:11.494 INFO:teuthology.orchestra.run.gibba012.stdout: "last_in_change": "2021-02-10 11:13:52.780316", 2021-02-10T11:41:11.494 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-02-10T11:41:11.494 INFO:teuthology.orchestra.run.gibba012.stdout: "flags_num": 5799936, 2021-02-10T11:41:11.494 INFO:teuthology.orchestra.run.gibba012.stdout: "flags_set": [ 2021-02-10T11:41:11.494 INFO:teuthology.orchestra.run.gibba012.stdout: "pglog_hardlimit", 2021-02-10T11:41:11.494 INFO:teuthology.orchestra.run.gibba012.stdout: "purged_snapdirs", 2021-02-10T11:41:11.495 INFO:teuthology.orchestra.run.gibba012.stdout: "recovery_deletes", 2021-02-10T11:41:11.495 INFO:teuthology.orchestra.run.gibba012.stdout: "sortbitwise" 2021-02-10T11:41:11.495 INFO:teuthology.orchestra.run.gibba012.stdout: ], 2021-02-10T11:41:11.495 INFO:teuthology.orchestra.run.gibba012.stdout: "crush_version": 4, 2021-02-10T11:41:11.495 INFO:teuthology.orchestra.run.gibba012.stdout: "full_ratio": 0.94999998807907104, 2021-02-10T11:41:11.495 INFO:teuthology.orchestra.run.gibba012.stdout: "backfillfull_ratio": 0.89999997615814209, 2021-02-10T11:41:11.495 INFO:teuthology.orchestra.run.gibba012.stdout: "nearfull_ratio": 0.85000002384185791, 2021-02-10T11:41:11.495 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_snapshot": "", 2021-02-10T11:41:11.495 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_max": 23, 2021-02-10T11:41:11.495 INFO:teuthology.orchestra.run.gibba012.stdout: "max_osd": 8, 2021-02-10T11:41:11.495 INFO:teuthology.orchestra.run.gibba012.stdout: "require_min_compat_client": "jewel", 2021-02-10T11:41:11.495 INFO:teuthology.orchestra.run.gibba012.stdout: "min_compat_client": "jewel", 2021-02-10T11:41:11.496 INFO:teuthology.orchestra.run.gibba012.stdout: "require_osd_release": "nautilus", 2021-02-10T11:41:11.496 INFO:teuthology.orchestra.run.gibba012.stdout: "pools": [ 2021-02-10T11:41:11.496 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:41:11.496 INFO:teuthology.orchestra.run.gibba012.stdout: "pool": 1, 2021-02-10T11:41:11.496 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_name": "rbd", 2021-02-10T11:41:11.496 INFO:teuthology.orchestra.run.gibba012.stdout: "create_time": "2021-02-10 11:13:57.508758", 2021-02-10T11:41:11.496 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:41:11.496 INFO:teuthology.orchestra.run.gibba012.stdout: "flags_names": "hashpspool", 2021-02-10T11:41:11.496 INFO:teuthology.orchestra.run.gibba012.stdout: "type": 1, 2021-02-10T11:41:11.496 INFO:teuthology.orchestra.run.gibba012.stdout: "size": 2, 2021-02-10T11:41:11.496 INFO:teuthology.orchestra.run.gibba012.stdout: "min_size": 1, 2021-02-10T11:41:11.496 INFO:teuthology.orchestra.run.gibba012.stdout: "crush_rule": 0, 2021-02-10T11:41:11.496 INFO:teuthology.orchestra.run.gibba012.stdout: "object_hash": 2, 2021-02-10T11:41:11.496 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:41:11.497 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num": 8, 2021-02-10T11:41:11.497 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num": 8, 2021-02-10T11:41:11.497 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num_target": 8, 2021-02-10T11:41:11.497 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_target": 8, 2021-02-10T11:41:11.497 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_pending": 8, 2021-02-10T11:41:11.497 INFO:teuthology.orchestra.run.gibba012.stdout: "last_pg_merge_meta": { 2021-02-10T11:41:11.497 INFO:teuthology.orchestra.run.gibba012.stdout: "source_pgid": "0.0", 2021-02-10T11:41:11.497 INFO:teuthology.orchestra.run.gibba012.stdout: "ready_epoch": 0, 2021-02-10T11:41:11.497 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_started": 0, 2021-02-10T11:41:11.497 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_clean": 0, 2021-02-10T11:41:11.497 INFO:teuthology.orchestra.run.gibba012.stdout: "source_version": "0'0", 2021-02-10T11:41:11.497 INFO:teuthology.orchestra.run.gibba012.stdout: "target_version": "0'0" 2021-02-10T11:41:11.497 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:41:11.498 INFO:teuthology.orchestra.run.gibba012.stdout: "last_change": "16", 2021-02-10T11:41:11.498 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend": "0", 2021-02-10T11:41:11.498 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:41:11.498 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:41:11.498 INFO:teuthology.orchestra.run.gibba012.stdout: "auid": 0, 2021-02-10T11:41:11.498 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:41:11.498 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_seq": 0, 2021-02-10T11:41:11.498 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_epoch": 0, 2021-02-10T11:41:11.498 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_snaps": [], 2021-02-10T11:41:11.498 INFO:teuthology.orchestra.run.gibba012.stdout: "removed_snaps": "[]", 2021-02-10T11:41:11.498 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_bytes": 0, 2021-02-10T11:41:11.498 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_objects": 0, 2021-02-10T11:41:11.499 INFO:teuthology.orchestra.run.gibba012.stdout: "tiers": [], 2021-02-10T11:41:11.499 INFO:teuthology.orchestra.run.gibba012.stdout: "tier_of": -1, 2021-02-10T11:41:11.499 INFO:teuthology.orchestra.run.gibba012.stdout: "read_tier": -1, 2021-02-10T11:41:11.499 INFO:teuthology.orchestra.run.gibba012.stdout: "write_tier": -1, 2021-02-10T11:41:11.499 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_mode": "none", 2021-02-10T11:41:11.499 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_bytes": 0, 2021-02-10T11:41:11.499 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_objects": 0, 2021-02-10T11:41:11.499 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:41:11.499 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:41:11.499 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:41:11.499 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_flush_age": 0, 2021-02-10T11:41:11.499 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_evict_age": 0, 2021-02-10T11:41:11.499 INFO:teuthology.orchestra.run.gibba012.stdout: "erasure_code_profile": "", 2021-02-10T11:41:11.499 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_params": { 2021-02-10T11:41:11.500 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "none" 2021-02-10T11:41:11.500 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:41:11.500 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_period": 0, 2021-02-10T11:41:11.500 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_count": 0, 2021-02-10T11:41:11.500 INFO:teuthology.orchestra.run.gibba012.stdout: "use_gmt_hitset": true, 2021-02-10T11:41:11.500 INFO:teuthology.orchestra.run.gibba012.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:41:11.500 INFO:teuthology.orchestra.run.gibba012.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:41:11.500 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:41:11.500 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:41:11.500 INFO:teuthology.orchestra.run.gibba012.stdout: "grade_table": [], 2021-02-10T11:41:11.500 INFO:teuthology.orchestra.run.gibba012.stdout: "stripe_width": 0, 2021-02-10T11:41:11.500 INFO:teuthology.orchestra.run.gibba012.stdout: "expected_num_objects": 0, 2021-02-10T11:41:11.500 INFO:teuthology.orchestra.run.gibba012.stdout: "fast_read": false, 2021-02-10T11:41:11.501 INFO:teuthology.orchestra.run.gibba012.stdout: "options": {}, 2021-02-10T11:41:11.501 INFO:teuthology.orchestra.run.gibba012.stdout: "application_metadata": { 2021-02-10T11:41:11.501 INFO:teuthology.orchestra.run.gibba012.stdout: "rbd": {} 2021-02-10T11:41:11.501 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:41:11.501 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:41:11.501 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:41:11.501 INFO:teuthology.orchestra.run.gibba012.stdout: "pool": 22, 2021-02-10T11:41:11.501 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_name": "cephfs_metadata", 2021-02-10T11:41:11.501 INFO:teuthology.orchestra.run.gibba012.stdout: "create_time": "2021-02-10 11:40:32.284968", 2021-02-10T11:41:11.501 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:41:11.501 INFO:teuthology.orchestra.run.gibba012.stdout: "flags_names": "hashpspool", 2021-02-10T11:41:11.501 INFO:teuthology.orchestra.run.gibba012.stdout: "type": 1, 2021-02-10T11:41:11.501 INFO:teuthology.orchestra.run.gibba012.stdout: "size": 2, 2021-02-10T11:41:11.502 INFO:teuthology.orchestra.run.gibba012.stdout: "min_size": 1, 2021-02-10T11:41:11.502 INFO:teuthology.orchestra.run.gibba012.stdout: "crush_rule": 0, 2021-02-10T11:41:11.502 INFO:teuthology.orchestra.run.gibba012.stdout: "object_hash": 2, 2021-02-10T11:41:11.502 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:41:11.502 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num": 8, 2021-02-10T11:41:11.502 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num": 8, 2021-02-10T11:41:11.502 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num_target": 8, 2021-02-10T11:41:11.502 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_target": 8, 2021-02-10T11:41:11.502 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_pending": 8, 2021-02-10T11:41:11.502 INFO:teuthology.orchestra.run.gibba012.stdout: "last_pg_merge_meta": { 2021-02-10T11:41:11.502 INFO:teuthology.orchestra.run.gibba012.stdout: "source_pgid": "0.0", 2021-02-10T11:41:11.502 INFO:teuthology.orchestra.run.gibba012.stdout: "ready_epoch": 0, 2021-02-10T11:41:11.502 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_started": 0, 2021-02-10T11:41:11.503 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_clean": 0, 2021-02-10T11:41:11.503 INFO:teuthology.orchestra.run.gibba012.stdout: "source_version": "0'0", 2021-02-10T11:41:11.503 INFO:teuthology.orchestra.run.gibba012.stdout: "target_version": "0'0" 2021-02-10T11:41:11.503 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:41:11.503 INFO:teuthology.orchestra.run.gibba012.stdout: "last_change": "107", 2021-02-10T11:41:11.503 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend": "0", 2021-02-10T11:41:11.503 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:41:11.503 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:41:11.503 INFO:teuthology.orchestra.run.gibba012.stdout: "auid": 0, 2021-02-10T11:41:11.503 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:41:11.503 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_seq": 0, 2021-02-10T11:41:11.503 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_epoch": 0, 2021-02-10T11:41:11.503 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_snaps": [], 2021-02-10T11:41:11.504 INFO:teuthology.orchestra.run.gibba012.stdout: "removed_snaps": "[]", 2021-02-10T11:41:11.504 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_bytes": 0, 2021-02-10T11:41:11.504 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_objects": 0, 2021-02-10T11:41:11.504 INFO:teuthology.orchestra.run.gibba012.stdout: "tiers": [], 2021-02-10T11:41:11.504 INFO:teuthology.orchestra.run.gibba012.stdout: "tier_of": -1, 2021-02-10T11:41:11.504 INFO:teuthology.orchestra.run.gibba012.stdout: "read_tier": -1, 2021-02-10T11:41:11.504 INFO:teuthology.orchestra.run.gibba012.stdout: "write_tier": -1, 2021-02-10T11:41:11.504 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_mode": "none", 2021-02-10T11:41:11.504 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_bytes": 0, 2021-02-10T11:41:11.504 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_objects": 0, 2021-02-10T11:41:11.504 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:41:11.504 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:41:11.504 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:41:11.505 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_flush_age": 0, 2021-02-10T11:41:11.505 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_evict_age": 0, 2021-02-10T11:41:11.505 INFO:teuthology.orchestra.run.gibba012.stdout: "erasure_code_profile": "", 2021-02-10T11:41:11.505 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_params": { 2021-02-10T11:41:11.505 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "none" 2021-02-10T11:41:11.505 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:41:11.505 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_period": 0, 2021-02-10T11:41:11.505 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_count": 0, 2021-02-10T11:41:11.505 INFO:teuthology.orchestra.run.gibba012.stdout: "use_gmt_hitset": true, 2021-02-10T11:41:11.505 INFO:teuthology.orchestra.run.gibba012.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:41:11.505 INFO:teuthology.orchestra.run.gibba012.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:41:11.505 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:41:11.505 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:41:11.506 INFO:teuthology.orchestra.run.gibba012.stdout: "grade_table": [], 2021-02-10T11:41:11.506 INFO:teuthology.orchestra.run.gibba012.stdout: "stripe_width": 0, 2021-02-10T11:41:11.506 INFO:teuthology.orchestra.run.gibba012.stdout: "expected_num_objects": 0, 2021-02-10T11:41:11.506 INFO:teuthology.orchestra.run.gibba012.stdout: "fast_read": false, 2021-02-10T11:41:11.506 INFO:teuthology.orchestra.run.gibba012.stdout: "options": { 2021-02-10T11:41:11.506 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_autoscale_bias": 4, 2021-02-10T11:41:11.506 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_min": 16, 2021-02-10T11:41:11.506 INFO:teuthology.orchestra.run.gibba012.stdout: "recovery_priority": 5 2021-02-10T11:41:11.506 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:41:11.506 INFO:teuthology.orchestra.run.gibba012.stdout: "application_metadata": { 2021-02-10T11:41:11.506 INFO:teuthology.orchestra.run.gibba012.stdout: "cephfs": { 2021-02-10T11:41:11.506 INFO:teuthology.orchestra.run.gibba012.stdout: "metadata": "cephfs" 2021-02-10T11:41:11.506 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:41:11.507 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:41:11.507 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:41:11.507 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:41:11.507 INFO:teuthology.orchestra.run.gibba012.stdout: "pool": 23, 2021-02-10T11:41:11.507 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_name": "cephfs_data", 2021-02-10T11:41:11.507 INFO:teuthology.orchestra.run.gibba012.stdout: "create_time": "2021-02-10 11:40:33.307261", 2021-02-10T11:41:11.507 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:41:11.507 INFO:teuthology.orchestra.run.gibba012.stdout: "flags_names": "hashpspool", 2021-02-10T11:41:11.507 INFO:teuthology.orchestra.run.gibba012.stdout: "type": 1, 2021-02-10T11:41:11.507 INFO:teuthology.orchestra.run.gibba012.stdout: "size": 2, 2021-02-10T11:41:11.507 INFO:teuthology.orchestra.run.gibba012.stdout: "min_size": 1, 2021-02-10T11:41:11.507 INFO:teuthology.orchestra.run.gibba012.stdout: "crush_rule": 0, 2021-02-10T11:41:11.507 INFO:teuthology.orchestra.run.gibba012.stdout: "object_hash": 2, 2021-02-10T11:41:11.507 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:41:11.508 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num": 8, 2021-02-10T11:41:11.508 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num": 8, 2021-02-10T11:41:11.508 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num_target": 8, 2021-02-10T11:41:11.508 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_target": 8, 2021-02-10T11:41:11.508 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_pending": 8, 2021-02-10T11:41:11.508 INFO:teuthology.orchestra.run.gibba012.stdout: "last_pg_merge_meta": { 2021-02-10T11:41:11.508 INFO:teuthology.orchestra.run.gibba012.stdout: "source_pgid": "0.0", 2021-02-10T11:41:11.508 INFO:teuthology.orchestra.run.gibba012.stdout: "ready_epoch": 0, 2021-02-10T11:41:11.508 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_started": 0, 2021-02-10T11:41:11.508 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_clean": 0, 2021-02-10T11:41:11.508 INFO:teuthology.orchestra.run.gibba012.stdout: "source_version": "0'0", 2021-02-10T11:41:11.508 INFO:teuthology.orchestra.run.gibba012.stdout: "target_version": "0'0" 2021-02-10T11:41:11.508 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:41:11.509 INFO:teuthology.orchestra.run.gibba012.stdout: "last_change": "108", 2021-02-10T11:41:11.509 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend": "0", 2021-02-10T11:41:11.509 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:41:11.509 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:41:11.509 INFO:teuthology.orchestra.run.gibba012.stdout: "auid": 0, 2021-02-10T11:41:11.509 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:41:11.509 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_seq": 0, 2021-02-10T11:41:11.509 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_epoch": 0, 2021-02-10T11:41:11.509 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_snaps": [], 2021-02-10T11:41:11.509 INFO:teuthology.orchestra.run.gibba012.stdout: "removed_snaps": "[]", 2021-02-10T11:41:11.509 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_bytes": 0, 2021-02-10T11:41:11.509 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_objects": 0, 2021-02-10T11:41:11.509 INFO:teuthology.orchestra.run.gibba012.stdout: "tiers": [], 2021-02-10T11:41:11.509 INFO:teuthology.orchestra.run.gibba012.stdout: "tier_of": -1, 2021-02-10T11:41:11.510 INFO:teuthology.orchestra.run.gibba012.stdout: "read_tier": -1, 2021-02-10T11:41:11.510 INFO:teuthology.orchestra.run.gibba012.stdout: "write_tier": -1, 2021-02-10T11:41:11.510 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_mode": "none", 2021-02-10T11:41:11.510 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_bytes": 0, 2021-02-10T11:41:11.510 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_objects": 0, 2021-02-10T11:41:11.510 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:41:11.510 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:41:11.510 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:41:11.510 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_flush_age": 0, 2021-02-10T11:41:11.510 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_evict_age": 0, 2021-02-10T11:41:11.510 INFO:teuthology.orchestra.run.gibba012.stdout: "erasure_code_profile": "", 2021-02-10T11:41:11.510 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_params": { 2021-02-10T11:41:11.511 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "none" 2021-02-10T11:41:11.511 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:41:11.511 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_period": 0, 2021-02-10T11:41:11.511 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_count": 0, 2021-02-10T11:41:11.511 INFO:teuthology.orchestra.run.gibba012.stdout: "use_gmt_hitset": true, 2021-02-10T11:41:11.511 INFO:teuthology.orchestra.run.gibba012.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:41:11.511 INFO:teuthology.orchestra.run.gibba012.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:41:11.511 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:41:11.511 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:41:11.511 INFO:teuthology.orchestra.run.gibba012.stdout: "grade_table": [], 2021-02-10T11:41:11.511 INFO:teuthology.orchestra.run.gibba012.stdout: "stripe_width": 0, 2021-02-10T11:41:11.511 INFO:teuthology.orchestra.run.gibba012.stdout: "expected_num_objects": 0, 2021-02-10T11:41:11.511 INFO:teuthology.orchestra.run.gibba012.stdout: "fast_read": false, 2021-02-10T11:41:11.511 INFO:teuthology.orchestra.run.gibba012.stdout: "options": {}, 2021-02-10T11:41:11.512 INFO:teuthology.orchestra.run.gibba012.stdout: "application_metadata": { 2021-02-10T11:41:11.512 INFO:teuthology.orchestra.run.gibba012.stdout: "cephfs": { 2021-02-10T11:41:11.512 INFO:teuthology.orchestra.run.gibba012.stdout: "data": "cephfs" 2021-02-10T11:41:11.512 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:41:11.512 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:41:11.512 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:41:11.512 INFO:teuthology.orchestra.run.gibba012.stdout: ], 2021-02-10T11:41:11.512 INFO:teuthology.orchestra.run.gibba012.stdout: "osds": [ 2021-02-10T11:41:11.512 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:41:11.512 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 0, 2021-02-10T11:41:11.512 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "95444b8f-6370-4a1d-ae5f-df62651b77da", 2021-02-10T11:41:11.513 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:41:11.513 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:41:11.513 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:41:11.513 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:41:11.513 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:41:11.513 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:41:11.513 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:41:11.513 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 106, 2021-02-10T11:41:11.513 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:41:11.513 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:41:11.513 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:41:11.514 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:41:11.514 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:41:11.514 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:41:11.514 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6817", 2021-02-10T11:41:11.514 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:41:11.514 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:41:11.514 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:41:11.514 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:41:11.514 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6818", 2021-02-10T11:41:11.514 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:41:11.514 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:41:11.514 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:41:11.515 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:41:11.515 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:41:11.515 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:41:11.515 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:41:11.515 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:41:11.515 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6820", 2021-02-10T11:41:11.515 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:41:11.515 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:41:11.515 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:41:11.515 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:41:11.515 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6822", 2021-02-10T11:41:11.515 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:41:11.515 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:41:11.516 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:41:11.516 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:41:11.516 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:41:11.516 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:41:11.516 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:41:11.516 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:41:11.516 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6828", 2021-02-10T11:41:11.516 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:41:11.516 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:41:11.516 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:41:11.516 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:41:11.516 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6830", 2021-02-10T11:41:11.516 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:41:11.517 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:41:11.517 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:41:11.517 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:41:11.517 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:41:11.517 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:41:11.517 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:41:11.517 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:41:11.517 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6824", 2021-02-10T11:41:11.517 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:41:11.517 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:41:11.517 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:41:11.517 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:41:11.518 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6826", 2021-02-10T11:41:11.518 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:41:11.518 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:41:11.518 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:41:11.518 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:41:11.518 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.112:6818/36030", 2021-02-10T11:41:11.518 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.112:6822/36030", 2021-02-10T11:41:11.518 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.112:6830/36030", 2021-02-10T11:41:11.518 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.112:6826/36030", 2021-02-10T11:41:11.518 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:41:11.518 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:41:11.518 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:41:11.518 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:41:11.519 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:41:11.519 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:41:11.519 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 1, 2021-02-10T11:41:11.519 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "a23e217e-0d8e-486e-be07-712adfabd262", 2021-02-10T11:41:11.519 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:41:11.519 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:41:11.519 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:41:11.519 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:41:11.519 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:41:11.520 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:41:11.520 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:41:11.520 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 105, 2021-02-10T11:41:11.520 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:41:11.520 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:41:11.520 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:41:11.520 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:41:11.520 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:41:11.520 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:41:11.521 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6800", 2021-02-10T11:41:11.521 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:41:11.521 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:41:11.521 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:41:11.521 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:41:11.521 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6802", 2021-02-10T11:41:11.521 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:41:11.521 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:41:11.522 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:41:11.522 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:41:11.522 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:41:11.522 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:41:11.522 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:41:11.522 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:41:11.522 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6804", 2021-02-10T11:41:11.522 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:41:11.522 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:41:11.522 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:41:11.522 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:41:11.522 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6806", 2021-02-10T11:41:11.523 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:41:11.523 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:41:11.523 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:41:11.523 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:41:11.523 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:41:11.523 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:41:11.523 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:41:11.523 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:41:11.523 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6812", 2021-02-10T11:41:11.523 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:41:11.523 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:41:11.523 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:41:11.523 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:41:11.524 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6814", 2021-02-10T11:41:11.524 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:41:11.524 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:41:11.524 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:41:11.524 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:41:11.524 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:41:11.524 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:41:11.524 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:41:11.524 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:41:11.524 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6808", 2021-02-10T11:41:11.524 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:41:11.524 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:41:11.524 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:41:11.525 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:41:11.525 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6810", 2021-02-10T11:41:11.525 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:41:11.525 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:41:11.525 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:41:11.525 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:41:11.525 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.112:6802/36029", 2021-02-10T11:41:11.525 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.112:6806/36029", 2021-02-10T11:41:11.525 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.112:6814/36029", 2021-02-10T11:41:11.525 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.112:6810/36029", 2021-02-10T11:41:11.525 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:41:11.525 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:41:11.525 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:41:11.526 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:41:11.526 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:41:11.526 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:41:11.526 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 2, 2021-02-10T11:41:11.526 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "026afa8a-c0f1-4b35-9a76-53c00fe0c2fc", 2021-02-10T11:41:11.526 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:41:11.526 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:41:11.526 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:41:11.526 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:41:11.526 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:41:11.526 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:41:11.526 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:41:11.526 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 106, 2021-02-10T11:41:11.527 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:41:11.527 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:41:11.527 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:41:11.527 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:41:11.527 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:41:11.527 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:41:11.527 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6801", 2021-02-10T11:41:11.527 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:41:11.527 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:41:11.527 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:41:11.527 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:41:11.527 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6803", 2021-02-10T11:41:11.527 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:41:11.528 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:41:11.528 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:41:11.528 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:41:11.528 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:41:11.528 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:41:11.528 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:41:11.528 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:41:11.528 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6805", 2021-02-10T11:41:11.528 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:41:11.528 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:41:11.528 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:41:11.528 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:41:11.528 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6807", 2021-02-10T11:41:11.528 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:41:11.529 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:41:11.529 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:41:11.529 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:41:11.529 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:41:11.529 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:41:11.529 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:41:11.529 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:41:11.529 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6813", 2021-02-10T11:41:11.529 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:41:11.529 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:41:11.529 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:41:11.529 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:41:11.529 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6815", 2021-02-10T11:41:11.530 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:41:11.530 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:41:11.530 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:41:11.530 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:41:11.530 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:41:11.530 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:41:11.530 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:41:11.530 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:41:11.530 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6809", 2021-02-10T11:41:11.530 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:41:11.530 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:41:11.530 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:41:11.531 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:41:11.531 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6811", 2021-02-10T11:41:11.531 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:41:11.531 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:41:11.531 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:41:11.531 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:41:11.531 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.112:6803/36031", 2021-02-10T11:41:11.531 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.112:6807/36031", 2021-02-10T11:41:11.531 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.112:6815/36031", 2021-02-10T11:41:11.531 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.112:6811/36031", 2021-02-10T11:41:11.531 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:41:11.531 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:41:11.531 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:41:11.531 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:41:11.532 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:41:11.532 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:41:11.532 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 3, 2021-02-10T11:41:11.532 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "58795682-e00d-4173-bb84-f7075ca89a11", 2021-02-10T11:41:11.532 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:41:11.532 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:41:11.532 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:41:11.532 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:41:11.532 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:41:11.532 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:41:11.532 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:41:11.532 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 106, 2021-02-10T11:41:11.532 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:41:11.533 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:41:11.533 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:41:11.533 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:41:11.533 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:41:11.533 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:41:11.533 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6816", 2021-02-10T11:41:11.533 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:41:11.533 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:41:11.533 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:41:11.533 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:41:11.533 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6819", 2021-02-10T11:41:11.533 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:41:11.533 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:41:11.534 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:41:11.534 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:41:11.534 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:41:11.534 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:41:11.534 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:41:11.534 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:41:11.534 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6821", 2021-02-10T11:41:11.534 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:41:11.534 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:41:11.534 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:41:11.534 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:41:11.534 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6823", 2021-02-10T11:41:11.535 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:41:11.535 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:41:11.535 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:41:11.535 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:41:11.535 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:41:11.535 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:41:11.535 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:41:11.535 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:41:11.535 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6829", 2021-02-10T11:41:11.535 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:41:11.535 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:41:11.535 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:41:11.535 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:41:11.536 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6831", 2021-02-10T11:41:11.536 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:41:11.536 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:41:11.536 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:41:11.536 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:41:11.536 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:41:11.536 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:41:11.536 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:41:11.536 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:41:11.536 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6825", 2021-02-10T11:41:11.536 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:41:11.536 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:41:11.536 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:41:11.537 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:41:11.537 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6827", 2021-02-10T11:41:11.537 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:41:11.537 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:41:11.537 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:41:11.537 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:41:11.537 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.112:6819/36028", 2021-02-10T11:41:11.537 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.112:6823/36028", 2021-02-10T11:41:11.537 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.112:6831/36028", 2021-02-10T11:41:11.537 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.112:6827/36028", 2021-02-10T11:41:11.537 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:41:11.537 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:41:11.537 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:41:11.538 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:41:11.538 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:41:11.538 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:41:11.538 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 4, 2021-02-10T11:41:11.538 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "6763f8d6-53dc-4e45-bdf6-a6eabcf81e39", 2021-02-10T11:41:11.538 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:41:11.538 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:41:11.538 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:41:11.538 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:41:11.538 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:41:11.538 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:41:11.538 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:41:11.539 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 106, 2021-02-10T11:41:11.539 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:41:11.539 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:41:11.539 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:41:11.539 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:41:11.539 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:41:11.539 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:41:11.539 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6810", 2021-02-10T11:41:11.539 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:41:11.539 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:41:11.539 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:41:11.539 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:41:11.539 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6813", 2021-02-10T11:41:11.540 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:41:11.540 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:41:11.540 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:41:11.540 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:41:11.540 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:41:11.540 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:41:11.540 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:41:11.540 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:41:11.540 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6816", 2021-02-10T11:41:11.540 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:41:11.540 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:41:11.540 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:41:11.540 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:41:11.540 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6819", 2021-02-10T11:41:11.541 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:41:11.541 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:41:11.541 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:41:11.541 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:41:11.541 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:41:11.541 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:41:11.541 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:41:11.541 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:41:11.541 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6825", 2021-02-10T11:41:11.541 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:41:11.541 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:41:11.541 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:41:11.541 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:41:11.542 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6827", 2021-02-10T11:41:11.542 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:41:11.542 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:41:11.542 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:41:11.542 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:41:11.542 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:41:11.542 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:41:11.542 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:41:11.542 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:41:11.542 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6821", 2021-02-10T11:41:11.542 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:41:11.542 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:41:11.542 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:41:11.543 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:41:11.543 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6823", 2021-02-10T11:41:11.543 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:41:11.543 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:41:11.543 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:41:11.543 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:41:11.543 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.120:6813/36388", 2021-02-10T11:41:11.543 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.120:6819/36388", 2021-02-10T11:41:11.543 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.120:6827/36388", 2021-02-10T11:41:11.543 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.120:6823/36388", 2021-02-10T11:41:11.543 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:41:11.543 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:41:11.543 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:41:11.544 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:41:11.544 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:41:11.544 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:41:11.544 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 5, 2021-02-10T11:41:11.544 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "5d0e4dd6-0cb0-4bf8-8314-626acaa1b544", 2021-02-10T11:41:11.544 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:41:11.544 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:41:11.544 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:41:11.544 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:41:11.544 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:41:11.544 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:41:11.544 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:41:11.544 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 106, 2021-02-10T11:41:11.545 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:41:11.545 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:41:11.545 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:41:11.545 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:41:11.545 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:41:11.545 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:41:11.545 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6804", 2021-02-10T11:41:11.545 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:41:11.545 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:41:11.545 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:41:11.545 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:41:11.545 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6806", 2021-02-10T11:41:11.545 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:41:11.546 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:41:11.546 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:41:11.546 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:41:11.546 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:41:11.546 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:41:11.546 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:41:11.546 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:41:11.546 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6809", 2021-02-10T11:41:11.546 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:41:11.546 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:41:11.546 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:41:11.546 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:41:11.546 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6812", 2021-02-10T11:41:11.547 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:41:11.547 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:41:11.547 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:41:11.547 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:41:11.547 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:41:11.547 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:41:11.547 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:41:11.547 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:41:11.547 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6820", 2021-02-10T11:41:11.547 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:41:11.547 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:41:11.547 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:41:11.547 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:41:11.548 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6822", 2021-02-10T11:41:11.548 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:41:11.548 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:41:11.548 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:41:11.548 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:41:11.548 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:41:11.548 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:41:11.548 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:41:11.548 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:41:11.548 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6815", 2021-02-10T11:41:11.548 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:41:11.548 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:41:11.548 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:41:11.549 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:41:11.549 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6818", 2021-02-10T11:41:11.549 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:41:11.549 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:41:11.549 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:41:11.549 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:41:11.549 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.120:6806/36387", 2021-02-10T11:41:11.549 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.120:6812/36387", 2021-02-10T11:41:11.549 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.120:6822/36387", 2021-02-10T11:41:11.549 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.120:6818/36387", 2021-02-10T11:41:11.549 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:41:11.549 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:41:11.549 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:41:11.550 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:41:11.550 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:41:11.550 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:41:11.550 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 6, 2021-02-10T11:41:11.550 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "b010e187-4541-4b00-80dd-f948f6ab8f15", 2021-02-10T11:41:11.550 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:41:11.550 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:41:11.550 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:41:11.550 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:41:11.550 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:41:11.550 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:41:11.550 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:41:11.550 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 105, 2021-02-10T11:41:11.551 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:41:11.551 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:41:11.551 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:41:11.551 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:41:11.551 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:41:11.551 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:41:11.551 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6824", 2021-02-10T11:41:11.551 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:41:11.551 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:41:11.551 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:41:11.551 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:41:11.551 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6826", 2021-02-10T11:41:11.551 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:41:11.552 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:41:11.552 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:41:11.552 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:41:11.552 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:41:11.552 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:41:11.552 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:41:11.552 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:41:11.552 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6828", 2021-02-10T11:41:11.552 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:41:11.552 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:41:11.552 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:41:11.552 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:41:11.552 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6829", 2021-02-10T11:41:11.553 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:41:11.553 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:41:11.553 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:41:11.553 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:41:11.553 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:41:11.553 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:41:11.553 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:41:11.553 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:41:11.553 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6832", 2021-02-10T11:41:11.553 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:41:11.553 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:41:11.553 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:41:11.553 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:41:11.554 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6833", 2021-02-10T11:41:11.554 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:41:11.554 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:41:11.554 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:41:11.554 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:41:11.554 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:41:11.554 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:41:11.554 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:41:11.554 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:41:11.554 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6830", 2021-02-10T11:41:11.554 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:41:11.554 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:41:11.554 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:41:11.555 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:41:11.555 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6831", 2021-02-10T11:41:11.555 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:41:11.555 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:41:11.555 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:41:11.555 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:41:11.555 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.120:6826/36385", 2021-02-10T11:41:11.555 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.120:6829/36385", 2021-02-10T11:41:11.555 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.120:6833/36385", 2021-02-10T11:41:11.555 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.120:6831/36385", 2021-02-10T11:41:11.555 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:41:11.555 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:41:11.555 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:41:11.556 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:41:11.556 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:41:11.556 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:41:11.556 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 7, 2021-02-10T11:41:11.556 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "671e2772-33b7-497e-8e44-2178bd850bcf", 2021-02-10T11:41:11.556 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:41:11.556 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:41:11.556 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:41:11.556 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:41:11.556 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:41:11.556 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:41:11.556 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:41:11.556 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 98, 2021-02-10T11:41:11.557 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:41:11.557 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:41:11.557 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:41:11.557 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:41:11.557 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:41:11.557 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:41:11.557 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6802", 2021-02-10T11:41:11.557 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:41:11.557 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:41:11.557 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:41:11.557 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:41:11.557 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6803", 2021-02-10T11:41:11.557 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:41:11.558 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:41:11.558 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:41:11.558 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:41:11.558 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:41:11.558 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:41:11.558 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:41:11.558 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:41:11.558 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6805", 2021-02-10T11:41:11.558 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:41:11.558 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:41:11.558 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:41:11.558 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:41:11.558 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6807", 2021-02-10T11:41:11.559 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:41:11.559 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:41:11.559 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:41:11.559 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:41:11.559 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:41:11.559 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:41:11.559 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:41:11.559 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:41:11.559 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6814", 2021-02-10T11:41:11.559 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:41:11.559 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:41:11.559 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:41:11.559 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:41:11.560 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6817", 2021-02-10T11:41:11.560 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:41:11.560 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:41:11.560 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:41:11.560 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:41:11.560 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:41:11.560 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:41:11.560 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:41:11.560 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:41:11.560 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6808", 2021-02-10T11:41:11.560 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:41:11.560 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:41:11.560 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:41:11.560 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:41:11.561 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6811", 2021-02-10T11:41:11.561 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:41:11.561 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:41:11.561 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:41:11.561 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:41:11.561 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.120:6803/36386", 2021-02-10T11:41:11.561 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.120:6807/36386", 2021-02-10T11:41:11.561 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.120:6817/36386", 2021-02-10T11:41:11.561 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.120:6811/36386", 2021-02-10T11:41:11.561 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:41:11.561 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:41:11.561 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:41:11.561 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:41:11.562 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:41:11.562 INFO:teuthology.orchestra.run.gibba012.stdout: ], 2021-02-10T11:41:11.562 INFO:teuthology.orchestra.run.gibba012.stdout: "osd_xinfo": [ 2021-02-10T11:41:11.562 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:41:11.562 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 0, 2021-02-10T11:41:11.562 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:41:11.562 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:41:11.562 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:41:11.562 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:41:11.562 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:41:11.562 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:41:11.562 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:41:11.562 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 1, 2021-02-10T11:41:11.563 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:41:11.563 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:41:11.563 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:41:11.563 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:41:11.563 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:41:11.563 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:41:11.563 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:41:11.563 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 2, 2021-02-10T11:41:11.563 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:41:11.563 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:41:11.563 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:41:11.563 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:41:11.563 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:41:11.564 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:41:11.564 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:41:11.564 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 3, 2021-02-10T11:41:11.564 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:41:11.564 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:41:11.564 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:41:11.564 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:41:11.564 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:41:11.564 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:41:11.564 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:41:11.564 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 4, 2021-02-10T11:41:11.564 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:41:11.564 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:41:11.565 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:41:11.565 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:41:11.565 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:41:11.565 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:41:11.565 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:41:11.565 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 5, 2021-02-10T11:41:11.565 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:41:11.565 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:41:11.565 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:41:11.565 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:41:11.565 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:41:11.565 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:41:11.565 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:41:11.566 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 6, 2021-02-10T11:41:11.566 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:41:11.566 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:41:11.566 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:41:11.566 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:41:11.566 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:41:11.566 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:41:11.566 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:41:11.566 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 7, 2021-02-10T11:41:11.566 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:41:11.566 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:41:11.566 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:41:11.566 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:41:11.567 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:41:11.567 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:41:11.567 INFO:teuthology.orchestra.run.gibba012.stdout: ], 2021-02-10T11:41:11.567 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_upmap": [], 2021-02-10T11:41:11.567 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_upmap_items": [], 2021-02-10T11:41:11.567 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_temp": [], 2021-02-10T11:41:11.567 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_temp": [], 2021-02-10T11:41:11.567 INFO:teuthology.orchestra.run.gibba012.stdout: "blacklist": {}, 2021-02-10T11:41:11.567 INFO:teuthology.orchestra.run.gibba012.stdout: "erasure_code_profiles": { 2021-02-10T11:41:11.567 INFO:teuthology.orchestra.run.gibba012.stdout: "default": { 2021-02-10T11:41:11.567 INFO:teuthology.orchestra.run.gibba012.stdout: "crush-failure-domain": "osd", 2021-02-10T11:41:11.567 INFO:teuthology.orchestra.run.gibba012.stdout: "k": "2", 2021-02-10T11:41:11.567 INFO:teuthology.orchestra.run.gibba012.stdout: "m": "1", 2021-02-10T11:41:11.568 INFO:teuthology.orchestra.run.gibba012.stdout: "plugin": "jerasure", 2021-02-10T11:41:11.568 INFO:teuthology.orchestra.run.gibba012.stdout: "ruleset-failure-domain": "osd", 2021-02-10T11:41:11.568 INFO:teuthology.orchestra.run.gibba012.stdout: "technique": "reed_sol_van" 2021-02-10T11:41:11.568 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:41:11.568 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:41:11.568 INFO:teuthology.orchestra.run.gibba012.stdout: "removed_snaps_queue": [], 2021-02-10T11:41:11.568 INFO:teuthology.orchestra.run.gibba012.stdout: "new_removed_snaps": [], 2021-02-10T11:41:11.568 INFO:teuthology.orchestra.run.gibba012.stdout: "new_purged_snaps": [], 2021-02-10T11:41:11.568 INFO:teuthology.orchestra.run.gibba012.stdout: "crush_node_flags": {}, 2021-02-10T11:41:11.568 INFO:teuthology.orchestra.run.gibba012.stdout: "device_class_flags": {} 2021-02-10T11:41:11.568 INFO:teuthology.orchestra.run.gibba012.stdout:} 2021-02-10T11:41:11.570 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:11.496 7f4c45ddd700 1 -- 172.21.2.112:0/2518235275 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4c240186c0 msgr2=0x7f4c2401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:11.570 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:11.496 7f4c45ddd700 1 --2- 172.21.2.112:0/2518235275 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4c240186c0 0x7f4c2401abc0 crc :-1 s=READY pgs=994 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:11.570 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:11.496 7f4c45ddd700 1 -- 172.21.2.112:0/2518235275 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4c400b23d0 msgr2=0x7f4c4009a510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:11.570 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:11.496 7f4c45ddd700 1 --2- 172.21.2.112:0/2518235275 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4c400b23d0 0x7f4c4009a510 secure :-1 s=READY pgs=2156 cs=0 l=1 rev1=1 rx=0x7f4c38009980 tx=0x7f4c3800bcd0).stop 2021-02-10T11:41:11.570 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:11.496 7f4c45ddd700 1 -- 172.21.2.112:0/2518235275 shutdown_connections 2021-02-10T11:41:11.570 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:11.496 7f4c45ddd700 1 --2- 172.21.2.112:0/2518235275 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4c240186c0 0x7f4c2401abc0 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:11.570 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:11.496 7f4c45ddd700 1 --2- 172.21.2.112:0/2518235275 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4c400b1fd0 0x7f4c40099ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:11.570 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:11.496 7f4c45ddd700 1 --2- 172.21.2.112:0/2518235275 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4c400b23d0 0x7f4c4009a510 unknown :-1 s=CLOSED pgs=2156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:11.570 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:11.496 7f4c45ddd700 1 --2- 172.21.2.112:0/2518235275 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4c4018d870 0x7f4c400b19b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:11.570 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:11.496 7f4c45ddd700 1 -- 172.21.2.112:0/2518235275 >> 172.21.2.112:0/2518235275 conn(0x7f4c400f0550 msgr2=0x7f4c40191730 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:11.570 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:11.497 7f4c45ddd700 1 -- 172.21.2.112:0/2518235275 shutdown_connections 2021-02-10T11:41:11.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:11.497 7f4c45ddd700 1 -- 172.21.2.112:0/2518235275 wait complete. 2021-02-10T11:41:11.571 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:41:11.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:11.984 7f535f3eb700 1 Processor -- start 2021-02-10T11:41:11.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:11.984 7f535f3eb700 1 -- start start 2021-02-10T11:41:11.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:11.985 7f535f3eb700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f53580d0340 0x7f53580ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:11.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:11.985 7f535f3eb700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53580f0b10 con 0x7f53580d1870 2021-02-10T11:41:11.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:11.985 7f535f3eb700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53581b0be0 con 0x7f53580d1c70 2021-02-10T11:41:11.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:11.985 7f535f3eb700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53580e5ed0 con 0x7f53580d0340 2021-02-10T11:41:11.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:11.985 7f535ca06700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f53580d0340 0x7f53580ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:11.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:11.985 7f535d207700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f53580d1870 0x7f53581b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:54668/0 (socket says 172.21.2.112:54668) 2021-02-10T11:41:11.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:11.985 7f535d207700 1 -- 172.21.2.112:0/1085606668 learned_addr learned my addr 172.21.2.112:0/1085606668 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:41:11.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:11.985 7f535ca06700 1 -- 172.21.2.112:0/1085606668 >> v1:172.21.2.120:6789/0 conn(0x7f53580d1c70 legacy=0x7f53580efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:12.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:11.985 7f535ca06700 1 -- 172.21.2.112:0/1085606668 >> v1:172.21.2.112:6789/0 conn(0x7f53580d1870 legacy=0x7f53581b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:12.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:11.985 7f534bfff700 1 -- 172.21.2.112:0/1085606668 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1590152687 0 0) 0x7f534c001180 con 0x7f53580d1870 2021-02-10T11:41:12.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:11.985 7f535ca06700 1 -- 172.21.2.112:0/1085606668 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53580f0450 con 0x7f53580d0340 2021-02-10T11:41:12.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:11.986 7f535ca06700 1 --2- 172.21.2.112:0/1085606668 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f53580d0340 0x7f53580ef640 secure :-1 s=READY pgs=2158 cs=0 l=1 rev1=1 rx=0x7f5354006650 tx=0x7f5354009a80).ready entity=mon.0 client_cookie=3dfc22e2f0bd017f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:12.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:11.986 7f534bfff700 1 -- 172.21.2.112:0/1085606668 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f53540009b0 con 0x7f53580d0340 2021-02-10T11:41:12.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:11.986 7f534bfff700 1 -- 172.21.2.112:0/1085606668 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f535400c0c0 con 0x7f53580d0340 2021-02-10T11:41:12.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:11.986 7f534bfff700 1 -- 172.21.2.112:0/1085606668 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f535400c5e0 con 0x7f53580d0340 2021-02-10T11:41:12.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:11.986 7f535f3eb700 1 -- 172.21.2.112:0/1085606668 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f53580d0340 msgr2=0x7f53580ef640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:12.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:11.986 7f535f3eb700 1 --2- 172.21.2.112:0/1085606668 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f53580d0340 0x7f53580ef640 secure :-1 s=READY pgs=2158 cs=0 l=1 rev1=1 rx=0x7f5354006650 tx=0x7f5354009a80).stop 2021-02-10T11:41:12.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:11.986 7f535f3eb700 1 -- 172.21.2.112:0/1085606668 shutdown_connections 2021-02-10T11:41:12.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:11.986 7f535f3eb700 1 --2- 172.21.2.112:0/1085606668 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f53580d0340 0x7f53580ef640 unknown :-1 s=CLOSED pgs=2158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:12.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:11.986 7f535f3eb700 1 -- 172.21.2.112:0/1085606668 >> 172.21.2.112:0/1085606668 conn(0x7f53580ce450 msgr2=0x7f53580d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:12.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:11.987 7f535f3eb700 1 -- 172.21.2.112:0/1085606668 shutdown_connections 2021-02-10T11:41:12.001 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:11.987 7f535f3eb700 1 -- 172.21.2.112:0/1085606668 wait complete. 2021-02-10T11:41:12.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:12.001 7f535f3eb700 1 Processor -- start 2021-02-10T11:41:12.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:12.001 7f535f3eb700 1 -- start start 2021-02-10T11:41:12.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:12.002 7f535f3eb700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f535819a3d0 0x7f53581ab160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:12.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:12.002 7f535f3eb700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f53581ab780 0x7f53580b3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:12.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:12.002 7f535f3eb700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f53581abb80 0x7f53580b3c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:12.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:12.002 7f535f3eb700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53581ae200 con 0x7f535819a3d0 2021-02-10T11:41:12.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:12.002 7f535f3eb700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5358092930 con 0x7f53581abb80 2021-02-10T11:41:12.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:12.002 7f535f3eb700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53580b4290 con 0x7f53581ab780 2021-02-10T11:41:12.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:12.002 7f535d207700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f535819a3d0 0x7f53581ab160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:12.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:12.002 7f535d207700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f535819a3d0 0x7f53581ab160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:55214/0 (socket says 172.21.2.112:55214) 2021-02-10T11:41:12.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:12.002 7f535d207700 1 -- 172.21.2.112:0/1374402046 learned_addr learned my addr 172.21.2.112:0/1374402046 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:41:12.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:12.002 7f535ca06700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f53581ab780 0x7f53580b3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:12.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:12.002 7f535da08700 1 --2- 172.21.2.112:0/1374402046 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f53581abb80 0x7f53580b3c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:12.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:12.003 7f535d207700 1 -- 172.21.2.112:0/1374402046 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f53581ab780 msgr2=0x7f53580b3650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:12.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:12.003 7f535d207700 1 --2- 172.21.2.112:0/1374402046 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f53581ab780 0x7f53580b3650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:12.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:12.003 7f535d207700 1 -- 172.21.2.112:0/1374402046 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f53581abb80 msgr2=0x7f53580b3c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:12.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:12.003 7f535d207700 1 --2- 172.21.2.112:0/1374402046 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f53581abb80 0x7f53580b3c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:12.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:12.003 7f535d207700 1 -- 172.21.2.112:0/1374402046 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53580ee360 con 0x7f535819a3d0 2021-02-10T11:41:12.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:12.003 7f535ca06700 1 --2- 172.21.2.112:0/1374402046 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f53581ab780 0x7f53580b3650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:41:12.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:12.003 7f535d207700 1 --2- 172.21.2.112:0/1374402046 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f535819a3d0 0x7f53581ab160 secure :-1 s=READY pgs=2159 cs=0 l=1 rev1=1 rx=0x7f534c00a570 tx=0x7f534c00a910).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:12.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:12.003 7f5349ffb700 1 -- 172.21.2.112:0/1374402046 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f534c00ad00 con 0x7f535819a3d0 2021-02-10T11:41:12.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:12.003 7f5349ffb700 1 -- 172.21.2.112:0/1374402046 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f534c001180 con 0x7f535819a3d0 2021-02-10T11:41:12.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:12.003 7f535f3eb700 1 -- 172.21.2.112:0/1374402046 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53580ee360 con 0x7f535819a3d0 2021-02-10T11:41:12.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:12.003 7f5349ffb700 1 -- 172.21.2.112:0/1374402046 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f534c004620 con 0x7f535819a3d0 2021-02-10T11:41:12.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:12.003 7f535f3eb700 1 -- 172.21.2.112:0/1374402046 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53580ee8f0 con 0x7f535819a3d0 2021-02-10T11:41:12.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:12.004 7f5349ffb700 1 -- 172.21.2.112:0/1374402046 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f534c0043a0 con 0x7f535819a3d0 2021-02-10T11:41:12.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:12.004 7f5349ffb700 1 --2- 172.21.2.112:0/1374402046 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f53380186d0 0x7f533801abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:12.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:12.004 7f5349ffb700 1 -- 172.21.2.112:0/1374402046 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f534c0325f0 con 0x7f535819a3d0 2021-02-10T11:41:12.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:12.004 7f535ca06700 1 --2- 172.21.2.112:0/1374402046 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f53380186d0 0x7f533801abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:12.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:12.005 7f535ca06700 1 --2- 172.21.2.112:0/1374402046 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f53380186d0 0x7f533801abd0 crc :-1 s=READY pgs=995 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:12.033 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:12.035 7f535f3eb700 1 -- 172.21.2.112:0/1374402046 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53580ce450 con 0x7f535819a3d0 2021-02-10T11:41:12.035 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:12.037 7f5349ffb700 1 -- 172.21.2.112:0/1374402046 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f534c001160 con 0x7f535819a3d0 2021-02-10T11:41:12.166 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:12.169 7f535f3eb700 1 -- 172.21.2.112:0/1374402046 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f53580ce450 con 0x7f535819a3d0 2021-02-10T11:41:12.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:12.169 7f5349ffb700 1 -- 172.21.2.112:0/1374402046 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+2107 (secure 0 0 0) 0x7f534c006fd0 con 0x7f535819a3d0 2021-02-10T11:41:12.167 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 89 2021-02-10T11:41:12.168 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:41:12.168 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":89,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10335,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/4159879144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":4159879144},{"type":"v1","addr":"172.21.2.112:6833","nonce":4159879144}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":84}],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:40:34.314832","modified":"2021-02-10 11:40:36.323257","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10332},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10332":{"gid":10332,"name":"b","rank":0,"incarnation":85,"state":"up:active","state_seq":3,"addr":"172.21.2.120:6835/3723311838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":3723311838},{"type":"v1","addr":"172.21.2.120:6835","nonce":3723311838}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":11}]} 2021-02-10T11:41:12.169 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:12.171 7f535f3eb700 1 -- 172.21.2.112:0/1374402046 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f53380186d0 msgr2=0x7f533801abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:12.169 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:12.171 7f535f3eb700 1 --2- 172.21.2.112:0/1374402046 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f53380186d0 0x7f533801abd0 crc :-1 s=READY pgs=995 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:12.169 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:12.171 7f535f3eb700 1 -- 172.21.2.112:0/1374402046 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f535819a3d0 msgr2=0x7f53581ab160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:12.170 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:12.171 7f535f3eb700 1 --2- 172.21.2.112:0/1374402046 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f535819a3d0 0x7f53581ab160 secure :-1 s=READY pgs=2159 cs=0 l=1 rev1=1 rx=0x7f534c00a570 tx=0x7f534c00a910).stop 2021-02-10T11:41:12.170 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:12.171 7f535f3eb700 1 -- 172.21.2.112:0/1374402046 shutdown_connections 2021-02-10T11:41:12.170 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:12.171 7f535f3eb700 1 --2- 172.21.2.112:0/1374402046 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f53380186d0 0x7f533801abd0 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:12.170 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:12.171 7f535f3eb700 1 --2- 172.21.2.112:0/1374402046 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f53581abb80 0x7f53580b3c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:12.170 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:12.171 7f535f3eb700 1 --2- 172.21.2.112:0/1374402046 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f53581ab780 0x7f53580b3650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:12.170 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:12.171 7f535f3eb700 1 --2- 172.21.2.112:0/1374402046 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f535819a3d0 0x7f53581ab160 unknown :-1 s=CLOSED pgs=2159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:12.170 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:12.171 7f535f3eb700 1 -- 172.21.2.112:0/1374402046 >> 172.21.2.112:0/1374402046 conn(0x7f5358093640 msgr2=0x7f535818eb00 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:12.170 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:12.172 7f535f3eb700 1 -- 172.21.2.112:0/1374402046 shutdown_connections 2021-02-10T11:41:12.170 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:12.172 7f535f3eb700 1 -- 172.21.2.112:0/1374402046 wait complete. 2021-02-10T11:41:12.178 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs fail cephfs 2021-02-10T11:41:12.606 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:12.592 7f5cd5d9e700 1 Processor -- start 2021-02-10T11:41:12.607 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:12.592 7f5cd5d9e700 1 -- start start 2021-02-10T11:41:12.607 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:12.592 7f5cd5d9e700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f5cd00d1870 0x7f5cd01b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:12.607 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:12.592 7f5cd5d9e700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cd00f0b10 con 0x7f5cd01b0a10 2021-02-10T11:41:12.607 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:12.592 7f5cd5d9e700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cd00d0310 con 0x7f5cd00f0650 2021-02-10T11:41:12.607 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:12.592 7f5cd5d9e700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5cd00d0530 con 0x7f5cd00d1870 2021-02-10T11:41:12.607 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:12.593 7f5cceffd700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f5cd01b0a10 0x7f5cd00cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:54682/0 (socket says 172.21.2.112:54682) 2021-02-10T11:41:12.607 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:12.593 7f5cceffd700 1 -- 172.21.2.112:0/1666563846 learned_addr learned my addr 172.21.2.112:0/1666563846 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:41:12.607 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:12.593 7f5ccf7fe700 1 --2- 172.21.2.112:0/1666563846 >> v2:172.21.2.120:3300/0 conn(0x7f5cd00d1870 0x7f5cd01b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:12.607 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:12.593 7f5cce7fc700 1 -- 172.21.2.112:0/1666563846 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2016379433 0 0) 0x7f5cb80009b0 con 0x7f5cd01b0a10 2021-02-10T11:41:12.607 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:12.593 7f5cce7fc700 1 -- 172.21.2.112:0/1666563846 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5cbc002fe0 con 0x7f5cd01b0a10 2021-02-10T11:41:12.607 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:12.593 7f5ccf7fe700 1 -- 172.21.2.112:0/1666563846 >> v1:172.21.2.120:6789/0 conn(0x7f5cd00f0650 legacy=0x7f5cd00cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:12.608 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:12.593 7f5cce7fc700 1 -- 172.21.2.112:0/1666563846 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1519894115 0 0) 0x7f5cc8000f90 con 0x7f5cd00f0650 2021-02-10T11:41:12.608 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:12.593 7f5ccf7fe700 1 -- 172.21.2.112:0/1666563846 >> v1:172.21.2.112:6789/0 conn(0x7f5cd01b0a10 legacy=0x7f5cd00cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:12.608 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:12.593 7f5ccf7fe700 1 -- 172.21.2.112:0/1666563846 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cd00d1c70 con 0x7f5cd00d1870 2021-02-10T11:41:12.608 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:12.593 7f5ccf7fe700 1 --2- 172.21.2.112:0/1666563846 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5cd00d1870 0x7f5cd01b03f0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 rx=0x7f5cc0006650 tx=0x7f5cc0009a80).ready entity=mon.1 client_cookie=6f00a6e2eb9820b9 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:12.608 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:12.593 7f5cce7fc700 1 -- 172.21.2.112:0/1666563846 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5cc00009b0 con 0x7f5cd00d1870 2021-02-10T11:41:12.608 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:12.593 7f5cce7fc700 1 -- 172.21.2.112:0/1666563846 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cc000c0c0 con 0x7f5cd00d1870 2021-02-10T11:41:12.608 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:12.593 7f5cce7fc700 1 -- 172.21.2.112:0/1666563846 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5cc000c5e0 con 0x7f5cd00d1870 2021-02-10T11:41:12.608 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:12.593 7f5cd5d9e700 1 -- 172.21.2.112:0/1666563846 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5cd00d1870 msgr2=0x7f5cd01b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:12.608 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:12.593 7f5cd5d9e700 1 --2- 172.21.2.112:0/1666563846 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5cd00d1870 0x7f5cd01b03f0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 rx=0x7f5cc0006650 tx=0x7f5cc0009a80).stop 2021-02-10T11:41:12.608 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:12.594 7f5cd5d9e700 1 -- 172.21.2.112:0/1666563846 shutdown_connections 2021-02-10T11:41:12.608 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:12.594 7f5cd5d9e700 1 --2- 172.21.2.112:0/1666563846 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5cd00d1870 0x7f5cd01b03f0 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:12.609 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:12.594 7f5cd5d9e700 1 -- 172.21.2.112:0/1666563846 >> 172.21.2.112:0/1666563846 conn(0x7f5cd00ce450 msgr2=0x7f5cd00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:12.609 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:12.594 7f5cd5d9e700 1 -- 172.21.2.112:0/1666563846 shutdown_connections 2021-02-10T11:41:12.609 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:12.594 7f5cd5d9e700 1 -- 172.21.2.112:0/1666563846 wait complete. 2021-02-10T11:41:12.609 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:12.609 7f5cd5d9e700 1 Processor -- start 2021-02-10T11:41:12.610 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:12.609 7f5cd5d9e700 1 -- start start 2021-02-10T11:41:12.610 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:12.609 7f5cd5d9e700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5cd01908b0 0x7f5cd01a4440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:12.610 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:12.609 7f5cd5d9e700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5cd00bd920 0x7f5cd00bdd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:12.611 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:12.609 7f5cd5d9e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5cd008d510 0x7f5cd008d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:12.611 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:12.609 7f5cd5d9e700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cd008cf90 con 0x7f5cd008d510 2021-02-10T11:41:12.611 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:12.609 7f5cd5d9e700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cd00be340 con 0x7f5cd00bd920 2021-02-10T11:41:12.611 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:12.609 7f5cd5d9e700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cd00be680 con 0x7f5cd01908b0 2021-02-10T11:41:12.611 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:12.610 7f5ccf7fe700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5cd01908b0 0x7f5cd01a4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:12.611 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:12.610 7f5ccf7fe700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5cd01908b0 0x7f5cd01a4440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3301/0 says I am v2:172.21.2.112:54908/0 (socket says 172.21.2.112:54908) 2021-02-10T11:41:12.611 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:12.610 7f5ccf7fe700 1 -- 172.21.2.112:0/3325609870 learned_addr learned my addr 172.21.2.112:0/3325609870 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:41:12.611 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:12.610 7f5ccffff700 1 --2- 172.21.2.112:0/3325609870 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5cd008d510 0x7f5cd008d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:12.611 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:12.610 7f5cceffd700 1 --2- 172.21.2.112:0/3325609870 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5cd00bd920 0x7f5cd00bdd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:12.612 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:12.610 7f5ccffff700 1 -- 172.21.2.112:0/3325609870 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5cd01908b0 msgr2=0x7f5cd01a4440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:12.612 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:12.610 7f5ccffff700 1 --2- 172.21.2.112:0/3325609870 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5cd01908b0 0x7f5cd01a4440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:12.612 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:12.610 7f5ccffff700 1 -- 172.21.2.112:0/3325609870 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5cd00bd920 msgr2=0x7f5cd00bdd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:12.612 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:12.610 7f5ccffff700 1 --2- 172.21.2.112:0/3325609870 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5cd00bd920 0x7f5cd00bdd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:12.612 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:12.610 7f5ccffff700 1 -- 172.21.2.112:0/3325609870 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cd01a4a60 con 0x7f5cd008d510 2021-02-10T11:41:12.612 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:12.610 7f5ccffff700 1 --2- 172.21.2.112:0/3325609870 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5cd008d510 0x7f5cd008d910 secure :-1 s=READY pgs=2161 cs=0 l=1 rev1=1 rx=0x7f5cc800b8a0 tx=0x7f5cc800bc40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:12.612 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:12.610 7f5cc7fff700 1 -- 172.21.2.112:0/3325609870 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5cc800bd80 con 0x7f5cd008d510 2021-02-10T11:41:12.612 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:12.610 7f5cc7fff700 1 -- 172.21.2.112:0/3325609870 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cc80058b0 con 0x7f5cd008d510 2021-02-10T11:41:12.612 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:12.610 7f5cc7fff700 1 -- 172.21.2.112:0/3325609870 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5cc8005d40 con 0x7f5cd008d510 2021-02-10T11:41:12.612 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:12.610 7f5cd5d9e700 1 -- 172.21.2.112:0/3325609870 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cd01a4a60 con 0x7f5cd008d510 2021-02-10T11:41:12.612 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:12.610 7f5cd5d9e700 1 -- 172.21.2.112:0/3325609870 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cd008df30 con 0x7f5cd008d510 2021-02-10T11:41:12.612 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:12.611 7f5cc7fff700 1 -- 172.21.2.112:0/3325609870 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5cc8006020 con 0x7f5cd008d510 2021-02-10T11:41:12.613 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:12.611 7f5cc7fff700 1 --2- 172.21.2.112:0/3325609870 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5cb40186c0 0x7f5cb401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:12.613 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:12.611 7f5cc7fff700 1 -- 172.21.2.112:0/3325609870 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f5cc8032880 con 0x7f5cd008d510 2021-02-10T11:41:12.613 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:12.612 7f5ccf7fe700 1 --2- 172.21.2.112:0/3325609870 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5cb40186c0 0x7f5cb401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:12.613 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:12.612 7f5ccf7fe700 1 --2- 172.21.2.112:0/3325609870 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5cb40186c0 0x7f5cb401abc0 crc :-1 s=READY pgs=996 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:12.640 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:12.642 7f5cd5d9e700 1 -- 172.21.2.112:0/3325609870 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cd00ebc50 con 0x7f5cd008d510 2021-02-10T11:41:12.642 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:12.645 7f5cc7fff700 1 -- 172.21.2.112:0/3325609870 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5cc802af80 con 0x7f5cd008d510 2021-02-10T11:41:12.770 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:12.773 7f5cd5d9e700 1 -- 172.21.2.112:0/3325609870 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f5cd00f0b10 con 0x7f5cd008d510 2021-02-10T11:41:13.552 INFO:tasks.ceph.mon.a.gibba012.stderr:2021-02-10 11:41:13.555 7fc3fe6e2700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-02-10T11:41:13.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:13.562 7f5cc7fff700 1 -- 172.21.2.112:0/3325609870 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v90) v1 ==== 161+0+0 (secure 0 0 0) 0x7f5cc802af80 con 0x7f5cd008d510 2021-02-10T11:41:13.591 INFO:teuthology.orchestra.run.gibba012.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2021-02-10T11:41:13.593 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:13.596 7f5cd5d9e700 1 -- 172.21.2.112:0/3325609870 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5cb40186c0 msgr2=0x7f5cb401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:13.594 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:13.596 7f5cd5d9e700 1 --2- 172.21.2.112:0/3325609870 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5cb40186c0 0x7f5cb401abc0 crc :-1 s=READY pgs=996 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:13.594 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:13.596 7f5cd5d9e700 1 -- 172.21.2.112:0/3325609870 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5cd008d510 msgr2=0x7f5cd008d910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:13.594 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:13.596 7f5cd5d9e700 1 --2- 172.21.2.112:0/3325609870 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5cd008d510 0x7f5cd008d910 secure :-1 s=READY pgs=2161 cs=0 l=1 rev1=1 rx=0x7f5cc800b8a0 tx=0x7f5cc800bc40).stop 2021-02-10T11:41:13.594 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:13.597 7f5cd5d9e700 1 -- 172.21.2.112:0/3325609870 shutdown_connections 2021-02-10T11:41:13.594 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:13.597 7f5cd5d9e700 1 --2- 172.21.2.112:0/3325609870 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5cb40186c0 0x7f5cb401abc0 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:13.594 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:13.597 7f5cd5d9e700 1 --2- 172.21.2.112:0/3325609870 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5cd00bd920 0x7f5cd00bdd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:13.594 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:13.597 7f5cd5d9e700 1 --2- 172.21.2.112:0/3325609870 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5cd008d510 0x7f5cd008d910 unknown :-1 s=CLOSED pgs=2161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:13.594 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:13.597 7f5cd5d9e700 1 --2- 172.21.2.112:0/3325609870 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5cd01908b0 0x7f5cd01a4440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:13.594 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:13.597 7f5cd5d9e700 1 -- 172.21.2.112:0/3325609870 >> 172.21.2.112:0/3325609870 conn(0x7f5cd008f030 msgr2=0x7f5cd00b5e80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:13.594 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:13.597 7f5cd5d9e700 1 -- 172.21.2.112:0/3325609870 shutdown_connections 2021-02-10T11:41:13.594 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:13.597 7f5cd5d9e700 1 -- 172.21.2.112:0/3325609870 wait complete. 2021-02-10T11:41:13.602 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:41:13.605 INFO:tasks.ceph.mds.b.gibba020.stdout:starting mds.b at 2021-02-10T11:41:14.030 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.016 7f174946b700 1 Processor -- start 2021-02-10T11:41:14.030 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.016 7f174946b700 1 -- start start 2021-02-10T11:41:14.030 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.017 7f174946b700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f17440d0340 0x7f17440ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:14.030 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.017 7f174946b700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17440f0b10 con 0x7f17440d1870 2021-02-10T11:41:14.030 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.017 7f174946b700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17441b0be0 con 0x7f17440d1c70 2021-02-10T11:41:14.030 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.017 7f174946b700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17440e5ed0 con 0x7f17440d0340 2021-02-10T11:41:14.031 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.017 7f174307d700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f17440d1870 0x7f17441b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:54692/0 (socket says 172.21.2.112:54692) 2021-02-10T11:41:14.031 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.017 7f174307d700 1 -- 172.21.2.112:0/3944187702 learned_addr learned my addr 172.21.2.112:0/3944187702 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:41:14.031 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.017 7f174287c700 1 --2- 172.21.2.112:0/3944187702 >> v2:172.21.2.120:3300/0 conn(0x7f17440d0340 0x7f17440ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:14.031 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.017 7f174207b700 1 -- 172.21.2.112:0/3944187702 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1892225463 0 0) 0x7f17340009b0 con 0x7f17440d1870 2021-02-10T11:41:14.031 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.017 7f174207b700 1 -- 172.21.2.112:0/3944187702 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1730002fe0 con 0x7f17440d1870 2021-02-10T11:41:14.031 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.018 7f174287c700 1 -- 172.21.2.112:0/3944187702 >> v1:172.21.2.120:6789/0 conn(0x7f17440d1c70 legacy=0x7f17440efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:14.031 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.018 7f174287c700 1 -- 172.21.2.112:0/3944187702 >> v1:172.21.2.112:6789/0 conn(0x7f17440d1870 legacy=0x7f17441b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:14.031 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.018 7f174287c700 1 -- 172.21.2.112:0/3944187702 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17440f0450 con 0x7f17440d0340 2021-02-10T11:41:14.031 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.018 7f174287c700 1 --2- 172.21.2.112:0/3944187702 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f17440d0340 0x7f17440ef640 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 rx=0x7f172c006650 tx=0x7f172c009a80).ready entity=mon.1 client_cookie=4ca7742058d5881f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:14.031 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.018 7f174207b700 1 -- 172.21.2.112:0/3944187702 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f172c00c0c0 con 0x7f17440d0340 2021-02-10T11:41:14.031 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.018 7f174207b700 1 -- 172.21.2.112:0/3944187702 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f172c00c340 con 0x7f17440d0340 2021-02-10T11:41:14.032 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.018 7f174207b700 1 -- 172.21.2.112:0/3944187702 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f172c00c820 con 0x7f17440d0340 2021-02-10T11:41:14.032 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.018 7f174946b700 1 -- 172.21.2.112:0/3944187702 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f17440d0340 msgr2=0x7f17440ef640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:14.032 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.018 7f174946b700 1 --2- 172.21.2.112:0/3944187702 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f17440d0340 0x7f17440ef640 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 rx=0x7f172c006650 tx=0x7f172c009a80).stop 2021-02-10T11:41:14.032 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.019 7f174946b700 1 -- 172.21.2.112:0/3944187702 shutdown_connections 2021-02-10T11:41:14.032 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.019 7f174946b700 1 --2- 172.21.2.112:0/3944187702 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f17440d0340 0x7f17440ef640 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:14.032 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.019 7f174946b700 1 -- 172.21.2.112:0/3944187702 >> 172.21.2.112:0/3944187702 conn(0x7f17440ce450 msgr2=0x7f17440d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:14.032 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.019 7f174946b700 1 -- 172.21.2.112:0/3944187702 shutdown_connections 2021-02-10T11:41:14.032 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.019 7f174946b700 1 -- 172.21.2.112:0/3944187702 wait complete. 2021-02-10T11:41:14.033 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.033 7f174946b700 1 Processor -- start 2021-02-10T11:41:14.034 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.034 7f174946b700 1 -- start start 2021-02-10T11:41:14.034 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.034 7f174946b700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f17440b4e00 0x7f17440a0fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:14.034 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.034 7f174946b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f17440a3a60 0x7f17440a3e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:14.034 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.034 7f174946b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1744094750 0x7f1744094b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:14.034 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.034 7f174946b700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17440a4550 con 0x7f1744094750 2021-02-10T11:41:14.034 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.034 7f174946b700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f174418bf30 con 0x7f17440b4e00 2021-02-10T11:41:14.034 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.034 7f174946b700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f174418ba60 con 0x7f17440a3a60 2021-02-10T11:41:14.034 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.034 7f174387e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1744094750 0x7f1744094b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:14.034 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.034 7f174387e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1744094750 0x7f1744094b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:55234/0 (socket says 172.21.2.112:55234) 2021-02-10T11:41:14.034 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.034 7f174387e700 1 -- 172.21.2.112:0/3366082372 learned_addr learned my addr 172.21.2.112:0/3366082372 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:41:14.034 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.034 7f174307d700 1 --2- 172.21.2.112:0/3366082372 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f17440b4e00 0x7f17440a0fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:14.035 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.034 7f174287c700 1 --2- 172.21.2.112:0/3366082372 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f17440a3a60 0x7f17440a3e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:14.035 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.035 7f174387e700 1 -- 172.21.2.112:0/3366082372 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f17440a3a60 msgr2=0x7f17440a3e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:14.035 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.035 7f174387e700 1 --2- 172.21.2.112:0/3366082372 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f17440a3a60 0x7f17440a3e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:14.035 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.035 7f174387e700 1 -- 172.21.2.112:0/3366082372 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f17440b4e00 msgr2=0x7f17440a0fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:14.035 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.035 7f174387e700 1 --2- 172.21.2.112:0/3366082372 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f17440b4e00 0x7f17440a0fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:14.035 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.035 7f174387e700 1 -- 172.21.2.112:0/3366082372 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17440a15c0 con 0x7f1744094750 2021-02-10T11:41:14.035 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.035 7f174387e700 1 --2- 172.21.2.112:0/3366082372 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1744094750 0x7f1744094b50 secure :-1 s=READY pgs=2164 cs=0 l=1 rev1=1 rx=0x7f173c009940 tx=0x7f173c00bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:14.035 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.035 7f173bfff700 1 -- 172.21.2.112:0/3366082372 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f173c000ce0 con 0x7f1744094750 2021-02-10T11:41:14.035 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.035 7f173bfff700 1 -- 172.21.2.112:0/3366082372 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f173c000f90 con 0x7f1744094750 2021-02-10T11:41:14.035 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.035 7f174946b700 1 -- 172.21.2.112:0/3366082372 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17440a15c0 con 0x7f1744094750 2021-02-10T11:41:14.035 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.035 7f174946b700 1 -- 172.21.2.112:0/3366082372 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1744095460 con 0x7f1744094750 2021-02-10T11:41:14.035 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.035 7f173bfff700 1 -- 172.21.2.112:0/3366082372 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f173c00c570 con 0x7f1744094750 2021-02-10T11:41:14.035 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.036 7f173bfff700 1 -- 172.21.2.112:0/3366082372 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f173c001220 con 0x7f1744094750 2021-02-10T11:41:14.036 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.036 7f173bfff700 1 --2- 172.21.2.112:0/3366082372 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f17280186c0 0x7f172801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:14.036 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.036 7f173bfff700 1 -- 172.21.2.112:0/3366082372 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5659+0+0 (secure 0 0 0) 0x7f173c032960 con 0x7f1744094750 2021-02-10T11:41:14.036 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.036 7f174307d700 1 --2- 172.21.2.112:0/3366082372 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f17280186c0 0x7f172801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:14.036 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.036 7f174307d700 1 --2- 172.21.2.112:0/3366082372 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f17280186c0 0x7f172801abc0 crc :-1 s=READY pgs=997 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:14.063 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.066 7f174946b700 1 -- 172.21.2.112:0/3366082372 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17440ee0e0 con 0x7f1744094750 2021-02-10T11:41:14.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.069 7f173bfff700 1 -- 172.21.2.112:0/3366082372 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f173c030440 con 0x7f1744094750 2021-02-10T11:41:14.196 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.199 7f174946b700 1 -- 172.21.2.112:0/3366082372 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f17440ce450 con 0x7f1744094750 2021-02-10T11:41:14.196 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.200 7f173bfff700 1 -- 172.21.2.112:0/3366082372 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 90 v90) v1 ==== 94+0+1760 (secure 0 0 0) 0x7f173c030440 con 0x7f1744094750 2021-02-10T11:41:14.197 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:41:14.197 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":90,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10335,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/4159879144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":4159879144},{"type":"v1","addr":"172.21.2.112:6833","nonce":4159879144}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":84}],"filesystems":[{"mdsmap":{"epoch":90,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:40:34.314832","modified":"2021-02-10 11:41:13.556595","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":109,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":11}]} 2021-02-10T11:41:14.197 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 90 2021-02-10T11:41:14.198 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.201 7f174946b700 1 -- 172.21.2.112:0/3366082372 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f17280186c0 msgr2=0x7f172801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:14.198 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.201 7f174946b700 1 --2- 172.21.2.112:0/3366082372 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f17280186c0 0x7f172801abc0 crc :-1 s=READY pgs=997 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:14.198 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.201 7f174946b700 1 -- 172.21.2.112:0/3366082372 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1744094750 msgr2=0x7f1744094b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:14.198 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.201 7f174946b700 1 --2- 172.21.2.112:0/3366082372 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1744094750 0x7f1744094b50 secure :-1 s=READY pgs=2164 cs=0 l=1 rev1=1 rx=0x7f173c009940 tx=0x7f173c00bcf0).stop 2021-02-10T11:41:14.199 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.202 7f174946b700 1 -- 172.21.2.112:0/3366082372 shutdown_connections 2021-02-10T11:41:14.199 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.202 7f174946b700 1 --2- 172.21.2.112:0/3366082372 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f17280186c0 0x7f172801abc0 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:14.199 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.202 7f174946b700 1 --2- 172.21.2.112:0/3366082372 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1744094750 0x7f1744094b50 unknown :-1 s=CLOSED pgs=2164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:14.199 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.202 7f174946b700 1 --2- 172.21.2.112:0/3366082372 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f17440a3a60 0x7f17440a3e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:14.199 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.202 7f174946b700 1 --2- 172.21.2.112:0/3366082372 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f17440b4e00 0x7f17440a0fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:14.199 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.202 7f174946b700 1 -- 172.21.2.112:0/3366082372 >> 172.21.2.112:0/3366082372 conn(0x7f174418cab0 msgr2=0x7f1744092c60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:14.199 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.202 7f174946b700 1 -- 172.21.2.112:0/3366082372 shutdown_connections 2021-02-10T11:41:14.199 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.202 7f174946b700 1 -- 172.21.2.112:0/3366082372 wait complete. 2021-02-10T11:41:14.207 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2021-02-10T11:41:14.641 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.627 7fabcfa0e700 1 Processor -- start 2021-02-10T11:41:14.641 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.628 7fabcfa0e700 1 -- start start 2021-02-10T11:41:14.641 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.628 7fabcfa0e700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fabc80d1870 0x7fabc81b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:14.641 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.628 7fabcfa0e700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabc80f0b10 con 0x7fabc81b0a10 2021-02-10T11:41:14.641 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.628 7fabcfa0e700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabc80d0310 con 0x7fabc80f0650 2021-02-10T11:41:14.642 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.628 7fabcfa0e700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabc80d0530 con 0x7fabc80d1870 2021-02-10T11:41:14.642 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.628 7fabcd029700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fabc81b0a10 0x7fabc80cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:54702/0 (socket says 172.21.2.112:54702) 2021-02-10T11:41:14.642 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.628 7fabcd029700 1 -- 172.21.2.112:0/1482575755 learned_addr learned my addr 172.21.2.112:0/1482575755 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:41:14.642 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.628 7fabcd82a700 1 --2- 172.21.2.112:0/1482575755 >> v2:172.21.2.120:3300/0 conn(0x7fabc80d1870 0x7fabc81b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:14.642 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.629 7fabcc828700 1 -- 172.21.2.112:0/1482575755 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4217663076 0 0) 0x7fabc40010b0 con 0x7fabc81b0a10 2021-02-10T11:41:14.642 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.629 7fabcc828700 1 -- 172.21.2.112:0/1482575755 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabb4002fe0 con 0x7fabc81b0a10 2021-02-10T11:41:14.642 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.629 7fabcc828700 1 -- 172.21.2.112:0/1482575755 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3859791290 0 0) 0x7fabc0000f90 con 0x7fabc80f0650 2021-02-10T11:41:14.642 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.629 7fabcc828700 1 -- 172.21.2.112:0/1482575755 --> v1:172.21.2.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabb4004370 con 0x7fabc80f0650 2021-02-10T11:41:14.642 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.629 7fabcd82a700 1 -- 172.21.2.112:0/1482575755 >> v1:172.21.2.120:6789/0 conn(0x7fabc80f0650 legacy=0x7fabc80cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:14.642 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.629 7fabcd82a700 1 -- 172.21.2.112:0/1482575755 >> v1:172.21.2.112:6789/0 conn(0x7fabc81b0a10 legacy=0x7fabc80cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:14.643 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.629 7fabcd82a700 1 -- 172.21.2.112:0/1482575755 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabc80d1c70 con 0x7fabc80d1870 2021-02-10T11:41:14.643 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.629 7fabcd82a700 1 --2- 172.21.2.112:0/1482575755 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fabc80d1870 0x7fabc81b03f0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 rx=0x7fabbc006650 tx=0x7fabbc009a80).ready entity=mon.1 client_cookie=687e850b72a90b41 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:14.643 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.629 7fabcc828700 1 -- 172.21.2.112:0/1482575755 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fabbc00c0c0 con 0x7fabc80d1870 2021-02-10T11:41:14.643 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.629 7fabcc828700 1 -- 172.21.2.112:0/1482575755 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabbc00c340 con 0x7fabc80d1870 2021-02-10T11:41:14.643 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.629 7fabcc828700 1 -- 172.21.2.112:0/1482575755 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fabbc00c860 con 0x7fabc80d1870 2021-02-10T11:41:14.643 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.629 7fabcfa0e700 1 -- 172.21.2.112:0/1482575755 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fabc80d1870 msgr2=0x7fabc81b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:14.643 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.629 7fabcfa0e700 1 --2- 172.21.2.112:0/1482575755 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fabc80d1870 0x7fabc81b03f0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 rx=0x7fabbc006650 tx=0x7fabbc009a80).stop 2021-02-10T11:41:14.643 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.630 7fabcfa0e700 1 -- 172.21.2.112:0/1482575755 shutdown_connections 2021-02-10T11:41:14.643 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.630 7fabcfa0e700 1 --2- 172.21.2.112:0/1482575755 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fabc80d1870 0x7fabc81b03f0 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:14.643 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.630 7fabcfa0e700 1 -- 172.21.2.112:0/1482575755 >> 172.21.2.112:0/1482575755 conn(0x7fabc80ce450 msgr2=0x7fabc80d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:14.643 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.630 7fabcfa0e700 1 -- 172.21.2.112:0/1482575755 shutdown_connections 2021-02-10T11:41:14.643 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.630 7fabcfa0e700 1 -- 172.21.2.112:0/1482575755 wait complete. 2021-02-10T11:41:14.645 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.644 7fabcfa0e700 1 Processor -- start 2021-02-10T11:41:14.645 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.645 7fabcfa0e700 1 -- start start 2021-02-10T11:41:14.646 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.645 7fabcfa0e700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fabc81a06c0 0x7fabc80ebe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:14.646 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.645 7fabcfa0e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fabc80ec490 0x7fabc80c30f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:14.646 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.645 7fabcfa0e700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fabc80c3710 0x7fabc80bc070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:14.646 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.645 7fabcfa0e700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabc80c3b10 con 0x7fabc80ec490 2021-02-10T11:41:14.646 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.645 7fabcfa0e700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabc80eca90 con 0x7fabc81a06c0 2021-02-10T11:41:14.646 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.645 7fabcfa0e700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabc80bc690 con 0x7fabc80c3710 2021-02-10T11:41:14.646 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.645 7fabcd029700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fabc80ec490 0x7fabc80c30f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:14.646 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.645 7fabcd029700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fabc80ec490 0x7fabc80c30f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:55242/0 (socket says 172.21.2.112:55242) 2021-02-10T11:41:14.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.645 7fabcd029700 1 -- 172.21.2.112:0/2651866892 learned_addr learned my addr 172.21.2.112:0/2651866892 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:41:14.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.645 7fabce02b700 1 --2- 172.21.2.112:0/2651866892 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fabc80c3710 0x7fabc80bc070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:14.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.645 7fabcd82a700 1 --2- 172.21.2.112:0/2651866892 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fabc81a06c0 0x7fabc80ebe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:14.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.646 7fabcd029700 1 -- 172.21.2.112:0/2651866892 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fabc80c3710 msgr2=0x7fabc80bc070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:14.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.646 7fabcd029700 1 --2- 172.21.2.112:0/2651866892 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fabc80c3710 0x7fabc80bc070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:14.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.646 7fabcd029700 1 -- 172.21.2.112:0/2651866892 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fabc81a06c0 msgr2=0x7fabc80ebe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:14.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.646 7fabcd029700 1 --2- 172.21.2.112:0/2651866892 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fabc81a06c0 0x7fabc80ebe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:14.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.646 7fabcd029700 1 -- 172.21.2.112:0/2651866892 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabc80bc870 con 0x7fabc80ec490 2021-02-10T11:41:14.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.646 7fabcd029700 1 --2- 172.21.2.112:0/2651866892 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fabc80ec490 0x7fabc80c30f0 secure :-1 s=READY pgs=2166 cs=0 l=1 rev1=1 rx=0x7fabc4001c60 tx=0x7fabc400a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:14.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.646 7fabba7fc700 1 -- 172.21.2.112:0/2651866892 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fabc4000e00 con 0x7fabc80ec490 2021-02-10T11:41:14.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.646 7fabba7fc700 1 -- 172.21.2.112:0/2651866892 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabc400acc0 con 0x7fabc80ec490 2021-02-10T11:41:14.648 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.646 7fabba7fc700 1 -- 172.21.2.112:0/2651866892 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fabc40045d0 con 0x7fabc80ec490 2021-02-10T11:41:14.648 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.646 7fabcfa0e700 1 -- 172.21.2.112:0/2651866892 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabc80bc870 con 0x7fabc80ec490 2021-02-10T11:41:14.648 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.646 7fabcfa0e700 1 -- 172.21.2.112:0/2651866892 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabc80bc870 con 0x7fabc80ec490 2021-02-10T11:41:14.648 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.647 7fabba7fc700 1 -- 172.21.2.112:0/2651866892 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fabc4004140 con 0x7fabc80ec490 2021-02-10T11:41:14.648 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.647 7fabba7fc700 1 --2- 172.21.2.112:0/2651866892 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fabb00186c0 0x7fabb001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:14.648 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.647 7fabba7fc700 1 -- 172.21.2.112:0/2651866892 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5659+0+0 (secure 0 0 0) 0x7fabc40325d0 con 0x7fabc80ec490 2021-02-10T11:41:14.648 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.647 7fabcd82a700 1 --2- 172.21.2.112:0/2651866892 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fabb00186c0 0x7fabb001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:14.648 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.648 7fabcd82a700 1 --2- 172.21.2.112:0/2651866892 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fabb00186c0 0x7fabb001abc0 crc :-1 s=READY pgs=998 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:14.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.678 7fabcfa0e700 1 -- 172.21.2.112:0/2651866892 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabc80ebc40 con 0x7fabc80ec490 2021-02-10T11:41:14.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.681 7fabba7fc700 1 -- 172.21.2.112:0/2651866892 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fabc4030db0 con 0x7fabc80ec490 2021-02-10T11:41:14.804 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:14.807 7fabcfa0e700 1 -- 172.21.2.112:0/2651866892 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fabc80f0b10 con 0x7fabc80ec490 2021-02-10T11:41:15.567 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:15.570 7fabba7fc700 1 -- 172.21.2.112:0/2651866892 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v91) v1 ==== 104+0+0 (secure 0 0 0) 0x7fabc4004140 con 0x7fabc80ec490 2021-02-10T11:41:15.576 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:15.578 7fabcfa0e700 1 -- 172.21.2.112:0/2651866892 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fabb00186c0 msgr2=0x7fabb001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:15.576 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:15.578 7fabcfa0e700 1 --2- 172.21.2.112:0/2651866892 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fabb00186c0 0x7fabb001abc0 crc :-1 s=READY pgs=998 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:15.576 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:15.578 7fabcfa0e700 1 -- 172.21.2.112:0/2651866892 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fabc80ec490 msgr2=0x7fabc80c30f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:15.576 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:15.578 7fabcfa0e700 1 --2- 172.21.2.112:0/2651866892 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fabc80ec490 0x7fabc80c30f0 secure :-1 s=READY pgs=2166 cs=0 l=1 rev1=1 rx=0x7fabc4001c60 tx=0x7fabc400a8d0).stop 2021-02-10T11:41:15.576 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:15.579 7fabcfa0e700 1 -- 172.21.2.112:0/2651866892 shutdown_connections 2021-02-10T11:41:15.576 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:15.579 7fabcfa0e700 1 --2- 172.21.2.112:0/2651866892 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fabb00186c0 0x7fabb001abc0 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:15.576 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:15.579 7fabcfa0e700 1 --2- 172.21.2.112:0/2651866892 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fabc80c3710 0x7fabc80bc070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:15.576 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:15.579 7fabcfa0e700 1 --2- 172.21.2.112:0/2651866892 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fabc80ec490 0x7fabc80c30f0 unknown :-1 s=CLOSED pgs=2166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:15.576 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:15.579 7fabcfa0e700 1 --2- 172.21.2.112:0/2651866892 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fabc81a06c0 0x7fabc80ebe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:15.576 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:15.579 7fabcfa0e700 1 -- 172.21.2.112:0/2651866892 >> 172.21.2.112:0/2651866892 conn(0x7fabc80a6bb0 msgr2=0x7fabc8095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:15.577 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:15.579 7fabcfa0e700 1 -- 172.21.2.112:0/2651866892 shutdown_connections 2021-02-10T11:41:15.577 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:15.579 7fabcfa0e700 1 -- 172.21.2.112:0/2651866892 wait complete. 2021-02-10T11:41:15.584 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2021-02-10T11:41:16.014 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:16.001 7fdfe40a5700 1 Processor -- start 2021-02-10T11:41:16.014 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:16.001 7fdfe40a5700 1 -- start start 2021-02-10T11:41:16.015 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:16.001 7fdfe40a5700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fdfdc0d1880 0x7fdfdc1b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:16.015 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:16.001 7fdfe40a5700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fdfdc1b0af0 0x7fdfdc108950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:16.015 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:16.001 7fdfe40a5700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdfdc0d1c80 0x7fdfdc108f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:16.015 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:16.001 7fdfe40a5700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfdc0fe1d0 con 0x7fdfdc1b0af0 2021-02-10T11:41:16.015 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:16.001 7fdfe40a5700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfdc109a70 con 0x7fdfdc0d1880 2021-02-10T11:41:16.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:16.001 7fdfe40a5700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfdc088070 con 0x7fdfdc0d1c80 2021-02-10T11:41:16.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:16.002 7fdfe16c0700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fdfdc1b0af0 0x7fdfdc108950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:16.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:16.002 7fdfe16c0700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fdfdc1b0af0 0x7fdfdc108950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:55252/0 (socket says 172.21.2.112:55252) 2021-02-10T11:41:16.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:16.002 7fdfe16c0700 1 -- 172.21.2.112:0/1261096997 learned_addr learned my addr 172.21.2.112:0/1261096997 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:41:16.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:16.002 7fdfe1ec1700 1 --2- 172.21.2.112:0/1261096997 >> v2:172.21.2.120:3300/0 conn(0x7fdfdc0d1880 0x7fdfdc1b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:16.017 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:16.002 7fdfe26c2700 1 --2- 172.21.2.112:0/1261096997 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdfdc0d1c80 0x7fdfdc108f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:16.017 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:16.002 7fdfe16c0700 1 -- 172.21.2.112:0/1261096997 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdfdc0d1c80 msgr2=0x7fdfdc108f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:16.017 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:16.002 7fdfe16c0700 1 --2- 172.21.2.112:0/1261096997 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdfdc0d1c80 0x7fdfdc108f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:16.017 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:16.002 7fdfe16c0700 1 -- 172.21.2.112:0/1261096997 >> v2:172.21.2.120:3300/0 conn(0x7fdfdc0d1880 msgr2=0x7fdfdc1b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:16.017 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:16.002 7fdfe16c0700 1 --2- 172.21.2.112:0/1261096997 >> v2:172.21.2.120:3300/0 conn(0x7fdfdc0d1880 0x7fdfdc1b04d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:16.018 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:16.002 7fdfe16c0700 1 -- 172.21.2.112:0/1261096997 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfdc109590 con 0x7fdfdc1b0af0 2021-02-10T11:41:16.018 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:16.002 7fdfe16c0700 1 --2- 172.21.2.112:0/1261096997 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdfdc1b0af0 0x7fdfdc108950 secure :-1 s=READY pgs=2167 cs=0 l=1 rev1=1 rx=0x7fdfd0006700 tx=0x7fdfd0009b80).ready entity=mon.0 client_cookie=9a2b97965f2e3eb0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:16.018 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:16.002 7fdfe0ebf700 1 -- 172.21.2.112:0/1261096997 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdfd000c0c0 con 0x7fdfdc1b0af0 2021-02-10T11:41:16.018 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:16.002 7fdfe0ebf700 1 -- 172.21.2.112:0/1261096997 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfd000c340 con 0x7fdfdc1b0af0 2021-02-10T11:41:16.019 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:16.002 7fdfe0ebf700 1 -- 172.21.2.112:0/1261096997 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdfd000c860 con 0x7fdfdc1b0af0 2021-02-10T11:41:16.019 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:16.002 7fdfe40a5700 1 -- 172.21.2.112:0/1261096997 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdfdc1b0af0 msgr2=0x7fdfdc108950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:16.019 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:16.002 7fdfe40a5700 1 --2- 172.21.2.112:0/1261096997 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdfdc1b0af0 0x7fdfdc108950 secure :-1 s=READY pgs=2167 cs=0 l=1 rev1=1 rx=0x7fdfd0006700 tx=0x7fdfd0009b80).stop 2021-02-10T11:41:16.019 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:16.003 7fdfe40a5700 1 -- 172.21.2.112:0/1261096997 shutdown_connections 2021-02-10T11:41:16.019 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:16.003 7fdfe40a5700 1 --2- 172.21.2.112:0/1261096997 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdfdc0d1c80 0x7fdfdc108f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:16.019 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:16.003 7fdfe40a5700 1 --2- 172.21.2.112:0/1261096997 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdfdc1b0af0 0x7fdfdc108950 unknown :-1 s=CLOSED pgs=2167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:16.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:16.003 7fdfe40a5700 1 --2- 172.21.2.112:0/1261096997 >> v2:172.21.2.120:3300/0 conn(0x7fdfdc0d1880 0x7fdfdc1b04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:16.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:16.003 7fdfe40a5700 1 -- 172.21.2.112:0/1261096997 >> 172.21.2.112:0/1261096997 conn(0x7fdfdc0ce430 msgr2=0x7fdfdc0d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:16.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:16.003 7fdfe40a5700 1 -- 172.21.2.112:0/1261096997 shutdown_connections 2021-02-10T11:41:16.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:16.003 7fdfe40a5700 1 -- 172.21.2.112:0/1261096997 wait complete. 2021-02-10T11:41:16.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:16.017 7fdfe40a5700 1 Processor -- start 2021-02-10T11:41:16.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:16.017 7fdfe40a5700 1 -- start start 2021-02-10T11:41:16.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:16.018 7fdfe40a5700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdfdc1a7a70 0x7fdfdc1a7e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:16.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:16.018 7fdfe40a5700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fdfdc0bf730 0x7fdfdc0bfb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:16.024 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:16.018 7fdfe40a5700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdfdc0c0150 0x7fdfdc0909c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:16.024 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:16.018 7fdfe40a5700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfdc090fe0 con 0x7fdfdc1a7a70 2021-02-10T11:41:16.024 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:16.018 7fdfe40a5700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfdc091200 con 0x7fdfdc0bf730 2021-02-10T11:41:16.024 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:16.018 7fdfe40a5700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfdc091420 con 0x7fdfdc0c0150 2021-02-10T11:41:16.024 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:16.018 7fdfe1ec1700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdfdc1a7a70 0x7fdfdc1a7e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:16.024 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:16.018 7fdfe1ec1700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdfdc1a7a70 0x7fdfdc1a7e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:55260/0 (socket says 172.21.2.112:55260) 2021-02-10T11:41:16.025 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:16.018 7fdfe1ec1700 1 -- 172.21.2.112:0/1592855908 learned_addr learned my addr 172.21.2.112:0/1592855908 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:41:16.025 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:16.018 7fdfe1ec1700 1 -- 172.21.2.112:0/1592855908 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdfdc0c0150 msgr2=0x7fdfdc0909c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:16.025 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:16.018 7fdfe1ec1700 1 --2- 172.21.2.112:0/1592855908 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdfdc0c0150 0x7fdfdc0909c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:16.025 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:16.018 7fdfe1ec1700 1 -- 172.21.2.112:0/1592855908 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fdfdc0bf730 msgr2=0x7fdfdc0bfb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:16.025 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:16.018 7fdfe1ec1700 1 --2- 172.21.2.112:0/1592855908 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fdfdc0bf730 0x7fdfdc0bfb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:16.026 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:16.018 7fdfe1ec1700 1 -- 172.21.2.112:0/1592855908 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfdc091640 con 0x7fdfdc1a7a70 2021-02-10T11:41:16.026 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:16.018 7fdfe1ec1700 1 --2- 172.21.2.112:0/1592855908 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdfdc1a7a70 0x7fdfdc1a7e70 secure :-1 s=READY pgs=2168 cs=0 l=1 rev1=1 rx=0x7fdfd800b530 tx=0x7fdfd800b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:16.026 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:16.018 7fdfce7fc700 1 -- 172.21.2.112:0/1592855908 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdfd800bcc0 con 0x7fdfdc1a7a70 2021-02-10T11:41:16.026 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:16.018 7fdfce7fc700 1 -- 172.21.2.112:0/1592855908 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfd8002030 con 0x7fdfdc1a7a70 2021-02-10T11:41:16.026 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:16.018 7fdfce7fc700 1 -- 172.21.2.112:0/1592855908 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdfd8002490 con 0x7fdfdc1a7a70 2021-02-10T11:41:16.026 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:16.018 7fdfe40a5700 1 -- 172.21.2.112:0/1592855908 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfdc091640 con 0x7fdfdc1a7a70 2021-02-10T11:41:16.027 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:16.019 7fdfe40a5700 1 -- 172.21.2.112:0/1592855908 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfdc091640 con 0x7fdfdc1a7a70 2021-02-10T11:41:16.027 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:16.019 7fdfce7fc700 1 -- 172.21.2.112:0/1592855908 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fdfd8000c90 con 0x7fdfdc1a7a70 2021-02-10T11:41:16.027 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:16.019 7fdfce7fc700 1 --2- 172.21.2.112:0/1592855908 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fdfc4018730 0x7fdfc401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:16.027 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:16.019 7fdfce7fc700 1 -- 172.21.2.112:0/1592855908 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5659+0+0 (secure 0 0 0) 0x7fdfd8032680 con 0x7fdfdc1a7a70 2021-02-10T11:41:16.027 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:16.020 7fdfe16c0700 1 --2- 172.21.2.112:0/1592855908 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fdfc4018730 0x7fdfc401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:16.028 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:16.020 7fdfe16c0700 1 --2- 172.21.2.112:0/1592855908 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fdfc4018730 0x7fdfc401ac10 crc :-1 s=READY pgs=999 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:16.048 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:16.051 7fdfe40a5700 1 -- 172.21.2.112:0/1592855908 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfdc0fe1d0 con 0x7fdfdc1a7a70 2021-02-10T11:41:16.051 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:16.054 7fdfce7fc700 1 -- 172.21.2.112:0/1592855908 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fdfd801f4f0 con 0x7fdfdc1a7a70 2021-02-10T11:41:16.179 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:16.182 7fdfe40a5700 1 -- 172.21.2.112:0/1592855908 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fdfdc0fe1d0 con 0x7fdfdc1a7a70 2021-02-10T11:41:16.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:16.574 7fdfce7fc700 1 -- 172.21.2.112:0/1592855908 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v110) v1 ==== 185+0+0 (secure 0 0 0) 0x7fdfd801f700 con 0x7fdfdc1a7a70 2021-02-10T11:41:16.599 INFO:teuthology.orchestra.run.gibba012.stderr:pool 'cephfs_metadata' removed 2021-02-10T11:41:16.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:16.601 7fdfe40a5700 1 -- 172.21.2.112:0/1592855908 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fdfc4018730 msgr2=0x7fdfc401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:16.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:16.601 7fdfe40a5700 1 --2- 172.21.2.112:0/1592855908 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fdfc4018730 0x7fdfc401ac10 crc :-1 s=READY pgs=999 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:16.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:16.601 7fdfe40a5700 1 -- 172.21.2.112:0/1592855908 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdfdc1a7a70 msgr2=0x7fdfdc1a7e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:16.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:16.601 7fdfe40a5700 1 --2- 172.21.2.112:0/1592855908 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdfdc1a7a70 0x7fdfdc1a7e70 secure :-1 s=READY pgs=2168 cs=0 l=1 rev1=1 rx=0x7fdfd800b530 tx=0x7fdfd800b8d0).stop 2021-02-10T11:41:16.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:16.602 7fdfe40a5700 1 -- 172.21.2.112:0/1592855908 shutdown_connections 2021-02-10T11:41:16.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:16.602 7fdfe40a5700 1 --2- 172.21.2.112:0/1592855908 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fdfc4018730 0x7fdfc401ac10 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:16.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:16.602 7fdfe40a5700 1 --2- 172.21.2.112:0/1592855908 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fdfdc0bf730 0x7fdfdc0bfb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:16.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:16.602 7fdfe40a5700 1 --2- 172.21.2.112:0/1592855908 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdfdc0c0150 0x7fdfdc0909c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:16.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:16.602 7fdfe40a5700 1 --2- 172.21.2.112:0/1592855908 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdfdc1a7a70 0x7fdfdc1a7e70 unknown :-1 s=CLOSED pgs=2168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:16.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:16.602 7fdfe40a5700 1 -- 172.21.2.112:0/1592855908 >> 172.21.2.112:0/1592855908 conn(0x7fdfdc0a69d0 msgr2=0x7fdfdc0b5950 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:16.600 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:16.602 7fdfe40a5700 1 -- 172.21.2.112:0/1592855908 shutdown_connections 2021-02-10T11:41:16.600 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:16.602 7fdfe40a5700 1 -- 172.21.2.112:0/1592855908 wait complete. 2021-02-10T11:41:16.607 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2021-02-10T11:41:17.037 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:17.024 7ff00df38700 1 Processor -- start 2021-02-10T11:41:17.037 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:17.024 7ff00df38700 1 -- start start 2021-02-10T11:41:17.037 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:17.024 7ff00df38700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7ff0080d1880 0x7ff0081b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:17.038 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:17.024 7ff00df38700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff0080cff80 0x7ff008109140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:17.038 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:17.024 7ff00df38700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff008109a70 con 0x7ff0081b0af0 2021-02-10T11:41:17.038 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:17.024 7ff00df38700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff008109760 con 0x7ff0080d1880 2021-02-10T11:41:17.038 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:17.024 7ff00df38700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff008088070 con 0x7ff0080cff80 2021-02-10T11:41:17.038 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:17.024 7ff006ffd700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7ff0081b0af0 0x7ff008108950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:54726/0 (socket says 172.21.2.112:54726) 2021-02-10T11:41:17.038 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:17.024 7ff006ffd700 1 -- 172.21.2.112:0/2935189836 learned_addr learned my addr 172.21.2.112:0/2935189836 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:41:17.038 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:17.024 7ff007fff700 1 --2- 172.21.2.112:0/2935189836 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff0080cff80 0x7ff008109140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:17.038 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:17.025 7ff0077fe700 1 --2- 172.21.2.112:0/2935189836 >> v2:172.21.2.120:3300/0 conn(0x7ff0080d1880 0x7ff0081b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:17.038 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:17.025 7ff0067fc700 1 -- 172.21.2.112:0/2935189836 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3198718127 0 0) 0x7ff000001500 con 0x7ff0081b0af0 2021-02-10T11:41:17.038 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:17.025 7ff0067fc700 1 -- 172.21.2.112:0/2935189836 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feff4002fe0 con 0x7ff0081b0af0 2021-02-10T11:41:17.038 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:17.025 7ff007fff700 1 -- 172.21.2.112:0/2935189836 >> v2:172.21.2.120:3300/0 conn(0x7ff0080d1880 msgr2=0x7ff0081b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:17.038 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:17.025 7ff007fff700 1 --2- 172.21.2.112:0/2935189836 >> v2:172.21.2.120:3300/0 conn(0x7ff0080d1880 0x7ff0081b04d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:17.038 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:17.025 7ff007fff700 1 -- 172.21.2.112:0/2935189836 >> v1:172.21.2.112:6789/0 conn(0x7ff0081b0af0 legacy=0x7ff008108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:17.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:17.025 7ff007fff700 1 -- 172.21.2.112:0/2935189836 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0080d1c80 con 0x7ff0080cff80 2021-02-10T11:41:17.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:17.025 7ff007fff700 1 --2- 172.21.2.112:0/2935189836 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff0080cff80 0x7ff008109140 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7feff8006ad0 tx=0x7feff8009ad0).ready entity=mon.2 client_cookie=40aa2f3247598361 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:17.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:17.025 7ff0067fc700 1 -- 172.21.2.112:0/2935189836 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feff80022e0 con 0x7ff0080cff80 2021-02-10T11:41:17.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:17.025 7ff0067fc700 1 -- 172.21.2.112:0/2935189836 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feff8006410 con 0x7ff0080cff80 2021-02-10T11:41:17.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:17.026 7ff00df38700 1 -- 172.21.2.112:0/2935189836 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff0080cff80 msgr2=0x7ff008109140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:17.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:17.026 7ff00df38700 1 --2- 172.21.2.112:0/2935189836 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff0080cff80 0x7ff008109140 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7feff8006ad0 tx=0x7feff8009ad0).stop 2021-02-10T11:41:17.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:17.026 7ff0067fc700 1 -- 172.21.2.112:0/2935189836 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feff8000c10 con 0x7ff0080cff80 2021-02-10T11:41:17.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:17.026 7ff00df38700 1 -- 172.21.2.112:0/2935189836 shutdown_connections 2021-02-10T11:41:17.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:17.026 7ff00df38700 1 --2- 172.21.2.112:0/2935189836 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff0080cff80 0x7ff008109140 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:17.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:17.026 7ff00df38700 1 --2- 172.21.2.112:0/2935189836 >> v2:172.21.2.120:3300/0 conn(0x7ff0080d1880 0x7ff0081b04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:17.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:17.026 7ff00df38700 1 -- 172.21.2.112:0/2935189836 >> 172.21.2.112:0/2935189836 conn(0x7ff0080ce430 msgr2=0x7ff0080d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:17.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:17.026 7ff00df38700 1 -- 172.21.2.112:0/2935189836 shutdown_connections 2021-02-10T11:41:17.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:17.026 7ff00df38700 1 -- 172.21.2.112:0/2935189836 wait complete. 2021-02-10T11:41:17.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:17.041 7ff00df38700 1 Processor -- start 2021-02-10T11:41:17.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:17.041 7ff00df38700 1 -- start start 2021-02-10T11:41:17.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:17.041 7ff00df38700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff0080ad520 0x7ff0080ad920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:17.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:17.041 7ff00df38700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff0080adf40 0x7ff008097c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:17.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:17.041 7ff00df38700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff008098250 0x7ff008098650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:17.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:17.041 7ff00df38700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff008098c70 con 0x7ff008098250 2021-02-10T11:41:17.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:17.041 7ff00df38700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0080910f0 con 0x7ff0080adf40 2021-02-10T11:41:17.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:17.041 7ff00df38700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff008091310 con 0x7ff0080ad520 2021-02-10T11:41:17.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:17.041 7ff007fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff008098250 0x7ff008098650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:17.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:17.042 7ff007fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff008098250 0x7ff008098650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:55270/0 (socket says 172.21.2.112:55270) 2021-02-10T11:41:17.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:17.042 7ff007fff700 1 -- 172.21.2.112:0/3055242596 learned_addr learned my addr 172.21.2.112:0/3055242596 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:41:17.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:17.042 7ff0077fe700 1 --2- 172.21.2.112:0/3055242596 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff0080ad520 0x7ff0080ad920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:17.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:17.042 7ff006ffd700 1 --2- 172.21.2.112:0/3055242596 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff0080adf40 0x7ff008097c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:17.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:17.042 7ff007fff700 1 -- 172.21.2.112:0/3055242596 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff0080ad520 msgr2=0x7ff0080ad920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:17.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:17.042 7ff007fff700 1 --2- 172.21.2.112:0/3055242596 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff0080ad520 0x7ff0080ad920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:17.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:17.042 7ff007fff700 1 -- 172.21.2.112:0/3055242596 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff0080adf40 msgr2=0x7ff008097c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:17.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:17.042 7ff007fff700 1 --2- 172.21.2.112:0/3055242596 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff0080adf40 0x7ff008097c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:17.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:17.042 7ff007fff700 1 -- 172.21.2.112:0/3055242596 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff00819a3b0 con 0x7ff008098250 2021-02-10T11:41:17.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:17.042 7ff007fff700 1 --2- 172.21.2.112:0/3055242596 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff008098250 0x7ff008098650 secure :-1 s=READY pgs=2170 cs=0 l=1 rev1=1 rx=0x7feff8009690 tx=0x7feff800b9a0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:17.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:17.042 7feffffff700 1 -- 172.21.2.112:0/3055242596 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feff8000c10 con 0x7ff008098250 2021-02-10T11:41:17.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:17.042 7feffffff700 1 -- 172.21.2.112:0/3055242596 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feff80090b0 con 0x7ff008098250 2021-02-10T11:41:17.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:17.042 7feffffff700 1 -- 172.21.2.112:0/3055242596 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feff8002560 con 0x7ff008098250 2021-02-10T11:41:17.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:17.042 7ff00df38700 1 -- 172.21.2.112:0/3055242596 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff00819a3b0 con 0x7ff008098250 2021-02-10T11:41:17.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:17.042 7ff00df38700 1 -- 172.21.2.112:0/3055242596 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff00819a3b0 con 0x7ff008098250 2021-02-10T11:41:17.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:17.043 7feffffff700 1 -- 172.21.2.112:0/3055242596 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7feff8000c10 con 0x7ff008098250 2021-02-10T11:41:17.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:17.043 7feffffff700 1 --2- 172.21.2.112:0/3055242596 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fefec0186c0 0x7fefec01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:17.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:17.043 7feffffff700 1 -- 172.21.2.112:0/3055242596 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5221+0+0 (secure 0 0 0) 0x7feff8032430 con 0x7ff008098250 2021-02-10T11:41:17.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:17.043 7ff0077fe700 1 --2- 172.21.2.112:0/3055242596 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fefec0186c0 0x7fefec01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:17.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:17.044 7ff0077fe700 1 --2- 172.21.2.112:0/3055242596 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fefec0186c0 0x7fefec01abc0 crc :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:17.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:17.074 7ff00df38700 1 -- 172.21.2.112:0/3055242596 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff008098c70 con 0x7ff008098250 2021-02-10T11:41:17.073 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:17.077 7feffffff700 1 -- 172.21.2.112:0/3055242596 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7feff8030d40 con 0x7ff008098250 2021-02-10T11:41:17.200 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:17.203 7ff00df38700 1 -- 172.21.2.112:0/3055242596 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7ff008098c70 con 0x7ff008098250 2021-02-10T11:41:17.576 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:17.579 7feffffff700 1 -- 172.21.2.112:0/3055242596 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v111) v1 ==== 173+0+0 (secure 0 0 0) 0x7feff8000c10 con 0x7ff008098250 2021-02-10T11:41:17.616 INFO:teuthology.orchestra.run.gibba012.stderr:pool 'cephfs_data' removed 2021-02-10T11:41:17.618 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:17.621 7ff00df38700 1 -- 172.21.2.112:0/3055242596 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fefec0186c0 msgr2=0x7fefec01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:17.618 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:17.621 7ff00df38700 1 --2- 172.21.2.112:0/3055242596 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fefec0186c0 0x7fefec01abc0 crc :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:17.618 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:17.621 7ff00df38700 1 -- 172.21.2.112:0/3055242596 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff008098250 msgr2=0x7ff008098650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:17.618 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:17.621 7ff00df38700 1 --2- 172.21.2.112:0/3055242596 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff008098250 0x7ff008098650 secure :-1 s=READY pgs=2170 cs=0 l=1 rev1=1 rx=0x7feff8009690 tx=0x7feff800b9a0).stop 2021-02-10T11:41:17.618 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:17.621 7ff00df38700 1 -- 172.21.2.112:0/3055242596 shutdown_connections 2021-02-10T11:41:17.618 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:17.621 7ff00df38700 1 --2- 172.21.2.112:0/3055242596 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fefec0186c0 0x7fefec01abc0 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:17.618 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:17.621 7ff00df38700 1 --2- 172.21.2.112:0/3055242596 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff0080adf40 0x7ff008097c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:17.618 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:17.621 7ff00df38700 1 --2- 172.21.2.112:0/3055242596 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff008098250 0x7ff008098650 unknown :-1 s=CLOSED pgs=2170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:17.619 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:17.621 7ff00df38700 1 --2- 172.21.2.112:0/3055242596 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff0080ad520 0x7ff0080ad920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:17.619 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:17.621 7ff00df38700 1 -- 172.21.2.112:0/3055242596 >> 172.21.2.112:0/3055242596 conn(0x7ff00818e810 msgr2=0x7ff00818c290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:17.619 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:17.621 7ff00df38700 1 -- 172.21.2.112:0/3055242596 shutdown_connections 2021-02-10T11:41:17.619 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:17.621 7ff00df38700 1 -- 172.21.2.112:0/3055242596 wait complete. 2021-02-10T11:41:17.626 INFO:tasks.ceph.mds.a:Restarting daemon 2021-02-10T11:41:17.626 INFO:tasks.ceph.mds.a:Stopping old one... 2021-02-10T11:41:17.627 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2021-02-10T11:41:17.627 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:41:17.627 INFO:tasks.ceph.mds.b:Restarting daemon 2021-02-10T11:41:17.627 INFO:tasks.ceph.mds.b:Stopping old one... 2021-02-10T11:41:17.627 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2021-02-10T11:41:17.627 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:41:17.632 INFO:tasks.ceph.mds.a:Stopped 2021-02-10T11:41:17.633 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2021-02-10T11:41:17.634 INFO:tasks.ceph.mds.b:Stopped 2021-02-10T11:41:17.634 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2021-02-10T11:41:17.675 INFO:tasks.ceph.mds.a:Started 2021-02-10T11:41:17.676 DEBUG:teuthology.parallel:result is None 2021-02-10T11:41:17.676 INFO:tasks.ceph.mds.b:Started 2021-02-10T11:41:17.677 DEBUG:teuthology.parallel:result is None 2021-02-10T11:41:17.677 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd blacklist clear 2021-02-10T11:41:18.082 INFO:tasks.ceph.mds.b.gibba020.stdout:starting mds.b at 2021-02-10T11:41:18.083 INFO:tasks.ceph.mds.a.gibba012.stdout:starting mds.a at 2021-02-10T11:41:18.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:18.091 7f4aaf0bd700 1 Processor -- start 2021-02-10T11:41:18.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:18.091 7f4aaf0bd700 1 -- start start 2021-02-10T11:41:18.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:18.091 7f4aaf0bd700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f4aa80d0340 0x7f4aa80ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:18.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:18.091 7f4aaf0bd700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f4aa80d1c70 0x7f4aa80efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:18.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:18.091 7f4aaf0bd700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4aa80f0b10 con 0x7f4aa80d1870 2021-02-10T11:41:18.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:18.091 7f4aaf0bd700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4aa80e5ed0 con 0x7f4aa80d1c70 2021-02-10T11:41:18.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:18.091 7f4aaf0bd700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4aa81b0be0 con 0x7f4aa80d0340 2021-02-10T11:41:18.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:18.091 7f4aad6da700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f4aa80d1c70 0x7f4aa80efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:18.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:18.092 7f4aaced9700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f4aa80d1870 0x7f4aa81b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:54748/0 (socket says 172.21.2.112:54748) 2021-02-10T11:41:18.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:18.092 7f4aaced9700 1 -- 172.21.2.112:0/2588915879 learned_addr learned my addr 172.21.2.112:0/2588915879 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:41:18.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:18.092 7f4a9ffff700 1 --2- 172.21.2.112:0/2588915879 >> v2:172.21.2.120:3300/0 conn(0x7f4aa80d0340 0x7f4aa80ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:18.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:18.092 7f4aad6da700 1 -- 172.21.2.112:0/2588915879 >> v2:172.21.2.120:3300/0 conn(0x7f4aa80d0340 msgr2=0x7f4aa80ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:18.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:18.092 7f4aad6da700 1 --2- 172.21.2.112:0/2588915879 >> v2:172.21.2.120:3300/0 conn(0x7f4aa80d0340 0x7f4aa80ef640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:18.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:18.092 7f4aad6da700 1 -- 172.21.2.112:0/2588915879 >> v1:172.21.2.112:6789/0 conn(0x7f4aa80d1870 legacy=0x7f4aa81b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:18.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:18.092 7f4a9f7fe700 1 -- 172.21.2.112:0/2588915879 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 275678451 0 0) 0x7f4a98000ce0 con 0x7f4aa80d1870 2021-02-10T11:41:18.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:18.092 7f4aad6da700 1 -- 172.21.2.112:0/2588915879 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4aa80f20a0 con 0x7f4aa80d1c70 2021-02-10T11:41:18.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:18.092 7f4a9ffff700 1 --2- 172.21.2.112:0/2588915879 >> v2:172.21.2.120:3300/0 conn(0x7f4aa80d0340 0x7f4aa80ef640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:41:18.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:18.092 7f4aad6da700 1 --2- 172.21.2.112:0/2588915879 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4aa80d1c70 0x7f4aa80efc60 secure :-1 s=READY pgs=2175 cs=0 l=1 rev1=1 rx=0x7f4aa0006e30 tx=0x7f4aa0009e60).ready entity=mon.0 client_cookie=c5b3e27c32e84720 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:18.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:18.092 7f4a9f7fe700 1 -- 172.21.2.112:0/2588915879 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4aa0006a50 con 0x7f4aa80d1c70 2021-02-10T11:41:18.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:18.092 7f4a9f7fe700 1 -- 172.21.2.112:0/2588915879 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4aa000b300 con 0x7f4aa80d1c70 2021-02-10T11:41:18.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:18.092 7f4a9f7fe700 1 -- 172.21.2.112:0/2588915879 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4aa000b790 con 0x7f4aa80d1c70 2021-02-10T11:41:18.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:18.092 7f4aaf0bd700 1 -- 172.21.2.112:0/2588915879 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4aa80d1c70 msgr2=0x7f4aa80efc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:18.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:18.092 7f4aaf0bd700 1 --2- 172.21.2.112:0/2588915879 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4aa80d1c70 0x7f4aa80efc60 secure :-1 s=READY pgs=2175 cs=0 l=1 rev1=1 rx=0x7f4aa0006e30 tx=0x7f4aa0009e60).stop 2021-02-10T11:41:18.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:18.093 7f4aaf0bd700 1 -- 172.21.2.112:0/2588915879 shutdown_connections 2021-02-10T11:41:18.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:18.093 7f4aaf0bd700 1 --2- 172.21.2.112:0/2588915879 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4aa80d1c70 0x7f4aa80efc60 unknown :-1 s=CLOSED pgs=2175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:18.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:18.093 7f4aaf0bd700 1 --2- 172.21.2.112:0/2588915879 >> v2:172.21.2.120:3300/0 conn(0x7f4aa80d0340 0x7f4aa80ef640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:18.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:18.093 7f4aaf0bd700 1 -- 172.21.2.112:0/2588915879 >> 172.21.2.112:0/2588915879 conn(0x7f4aa80ce450 msgr2=0x7f4aa80d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:18.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:18.093 7f4aaf0bd700 1 -- 172.21.2.112:0/2588915879 shutdown_connections 2021-02-10T11:41:18.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:18.093 7f4aaf0bd700 1 -- 172.21.2.112:0/2588915879 wait complete. 2021-02-10T11:41:18.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:18.108 7f4aaf0bd700 1 Processor -- start 2021-02-10T11:41:18.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:18.108 7f4aaf0bd700 1 -- start start 2021-02-10T11:41:18.112 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:18.108 7f4aaf0bd700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4aa80950f0 0x7f4aa8091130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:18.112 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:18.108 7f4aaf0bd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4aa80954f0 0x7f4aa8090600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:18.112 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:18.108 7f4aaf0bd700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4aa8090c20 0x7f4aa80a4380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:18.112 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:18.108 7f4aaf0bd700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4aa819a010 con 0x7f4aa80954f0 2021-02-10T11:41:18.112 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:18.108 7f4aaf0bd700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4aa8094b70 con 0x7f4aa80950f0 2021-02-10T11:41:18.113 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:18.108 7f4aaf0bd700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4aa8091750 con 0x7f4aa8090c20 2021-02-10T11:41:18.113 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:18.108 7f4a9ffff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4aa80954f0 0x7f4aa8090600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:18.113 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:18.108 7f4a9ffff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4aa80954f0 0x7f4aa8090600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:55292/0 (socket says 172.21.2.112:55292) 2021-02-10T11:41:18.113 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:18.108 7f4a9ffff700 1 -- 172.21.2.112:0/2811395713 learned_addr learned my addr 172.21.2.112:0/2811395713 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:41:18.113 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:18.109 7f4aad6da700 1 --2- 172.21.2.112:0/2811395713 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4aa8090c20 0x7f4aa80a4380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:18.113 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:18.109 7f4aaced9700 1 --2- 172.21.2.112:0/2811395713 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4aa80950f0 0x7f4aa8091130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:18.114 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:18.109 7f4a9ffff700 1 -- 172.21.2.112:0/2811395713 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4aa8090c20 msgr2=0x7f4aa80a4380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:18.114 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:18.109 7f4a9ffff700 1 --2- 172.21.2.112:0/2811395713 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4aa8090c20 0x7f4aa80a4380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:18.114 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:18.109 7f4a9ffff700 1 -- 172.21.2.112:0/2811395713 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4aa80950f0 msgr2=0x7f4aa8091130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:18.114 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:18.109 7f4a9ffff700 1 --2- 172.21.2.112:0/2811395713 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4aa80950f0 0x7f4aa8091130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:18.114 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:18.109 7f4a9ffff700 1 -- 172.21.2.112:0/2811395713 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4aa80946e0 con 0x7f4aa80954f0 2021-02-10T11:41:18.115 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:18.109 7f4a9ffff700 1 --2- 172.21.2.112:0/2811395713 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4aa80954f0 0x7f4aa8090600 secure :-1 s=READY pgs=2176 cs=0 l=1 rev1=1 rx=0x7f4aa4006f20 tx=0x7f4aa400b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:18.115 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:18.109 7f4a9d7fa700 1 -- 172.21.2.112:0/2811395713 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4aa400bd80 con 0x7f4aa80954f0 2021-02-10T11:41:18.115 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:18.109 7f4a9d7fa700 1 -- 172.21.2.112:0/2811395713 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4aa4000c90 con 0x7f4aa80954f0 2021-02-10T11:41:18.115 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:18.109 7f4aaf0bd700 1 -- 172.21.2.112:0/2811395713 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4aa80946e0 con 0x7f4aa80954f0 2021-02-10T11:41:18.115 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:18.109 7f4aaf0bd700 1 -- 172.21.2.112:0/2811395713 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4aa80946e0 con 0x7f4aa80954f0 2021-02-10T11:41:18.115 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:18.109 7f4a9d7fa700 1 -- 172.21.2.112:0/2811395713 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4aa40022b0 con 0x7f4aa80954f0 2021-02-10T11:41:18.115 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:18.110 7f4a9d7fa700 1 -- 172.21.2.112:0/2811395713 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4aa400a000 con 0x7f4aa80954f0 2021-02-10T11:41:18.115 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:18.110 7f4a9d7fa700 1 --2- 172.21.2.112:0/2811395713 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4a88018730 0x7f4a8801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:18.116 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:18.110 7f4a9d7fa700 1 -- 172.21.2.112:0/2811395713 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 4839+0+0 (secure 0 0 0) 0x7f4aa4032520 con 0x7f4aa80954f0 2021-02-10T11:41:18.116 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:18.110 7f4aaced9700 1 --2- 172.21.2.112:0/2811395713 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4a88018730 0x7f4a8801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:18.116 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:18.111 7f4aaced9700 1 --2- 172.21.2.112:0/2811395713 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4a88018730 0x7f4a8801ac10 crc :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:18.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:18.140 7f4aaf0bd700 1 -- 172.21.2.112:0/2811395713 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4aa819a010 con 0x7f4aa80954f0 2021-02-10T11:41:18.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:18.142 7f4a9d7fa700 1 -- 172.21.2.112:0/2811395713 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4aa4006bb0 con 0x7f4aa80954f0 2021-02-10T11:41:18.279 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:18.282 7f4aaf0bd700 1 -- 172.21.2.112:0/2811395713 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7f4aa819a010 con 0x7f4aa80954f0 2021-02-10T11:41:18.616 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:18.619 7f4a9d7fa700 1 -- 172.21.2.112:0/2811395713 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v112) v1 ==== 97+0+0 (secure 0 0 0) 0x7f4aa400a000 con 0x7f4aa80954f0 2021-02-10T11:41:18.645 INFO:teuthology.orchestra.run.gibba012.stderr: removed all blacklist entries 2021-02-10T11:41:18.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:18.650 7f4aaf0bd700 1 -- 172.21.2.112:0/2811395713 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4a88018730 msgr2=0x7f4a8801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:18.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:18.650 7f4aaf0bd700 1 --2- 172.21.2.112:0/2811395713 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4a88018730 0x7f4a8801ac10 crc :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:18.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:18.650 7f4aaf0bd700 1 -- 172.21.2.112:0/2811395713 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4aa80954f0 msgr2=0x7f4aa8090600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:18.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:18.650 7f4aaf0bd700 1 --2- 172.21.2.112:0/2811395713 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4aa80954f0 0x7f4aa8090600 secure :-1 s=READY pgs=2176 cs=0 l=1 rev1=1 rx=0x7f4aa4006f20 tx=0x7f4aa400b990).stop 2021-02-10T11:41:18.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:18.650 7f4aaf0bd700 1 -- 172.21.2.112:0/2811395713 shutdown_connections 2021-02-10T11:41:18.648 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:18.650 7f4aaf0bd700 1 --2- 172.21.2.112:0/2811395713 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4a88018730 0x7f4a8801ac10 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:18.648 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:18.650 7f4aaf0bd700 1 --2- 172.21.2.112:0/2811395713 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4aa8090c20 0x7f4aa80a4380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:18.648 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:18.650 7f4aaf0bd700 1 --2- 172.21.2.112:0/2811395713 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4aa80954f0 0x7f4aa8090600 unknown :-1 s=CLOSED pgs=2176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:18.648 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:18.650 7f4aaf0bd700 1 --2- 172.21.2.112:0/2811395713 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4aa80950f0 0x7f4aa8091130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:18.648 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:18.650 7f4aaf0bd700 1 -- 172.21.2.112:0/2811395713 >> 172.21.2.112:0/2811395713 conn(0x7f4aa80bc0b0 msgr2=0x7f4aa80c29c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:18.648 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:18.650 7f4aaf0bd700 1 -- 172.21.2.112:0/2811395713 shutdown_connections 2021-02-10T11:41:18.648 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:18.650 7f4aaf0bd700 1 -- 172.21.2.112:0/2811395713 wait complete. 2021-02-10T11:41:18.655 INFO:tasks.cephfs.cephfs_test_case:['0', '1', '2', '3'] 2021-02-10T11:41:18.655 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth ls --format=json-pretty 2021-02-10T11:41:19.087 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.070 7f28592e3700 1 Processor -- start 2021-02-10T11:41:19.087 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.070 7f28592e3700 1 -- start start 2021-02-10T11:41:19.087 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.071 7f28592e3700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f28540d1870 0x7f28541b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:19.087 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.071 7f28592e3700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28540f0b10 con 0x7f28540f0650 2021-02-10T11:41:19.087 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.071 7f28592e3700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28540d0310 con 0x7f28541b0a10 2021-02-10T11:41:19.088 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.071 7f28592e3700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28540d0530 con 0x7f28540d1870 2021-02-10T11:41:19.088 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.071 7f285387e700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f28540f0650 0x7f28540cfb20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:54762/0 (socket says 172.21.2.112:54762) 2021-02-10T11:41:19.088 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.071 7f285387e700 1 -- 172.21.2.112:0/2826426574 learned_addr learned my addr 172.21.2.112:0/2826426574 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:41:19.088 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.071 7f285307d700 1 --2- 172.21.2.112:0/2826426574 >> v2:172.21.2.120:3300/0 conn(0x7f28540d1870 0x7f28541b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:19.088 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.071 7f285207b700 1 -- 172.21.2.112:0/2826426574 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1731661309 0 0) 0x7f284c001500 con 0x7f28540f0650 2021-02-10T11:41:19.088 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.071 7f285207b700 1 -- 172.21.2.112:0/2826426574 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2840002fe0 con 0x7f28540f0650 2021-02-10T11:41:19.088 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.071 7f285307d700 1 -- 172.21.2.112:0/2826426574 >> v1:172.21.2.120:6789/0 conn(0x7f28541b0a10 legacy=0x7f28540cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:19.088 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.071 7f285307d700 1 -- 172.21.2.112:0/2826426574 >> v1:172.21.2.112:6789/0 conn(0x7f28540f0650 legacy=0x7f28540cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:19.088 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.071 7f285307d700 1 -- 172.21.2.112:0/2826426574 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28540d1c70 con 0x7f28540d1870 2021-02-10T11:41:19.088 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.071 7f285307d700 1 --2- 172.21.2.112:0/2826426574 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f28540d1870 0x7f28541b03f0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 rx=0x7f2844006650 tx=0x7f2844009a80).ready entity=mon.1 client_cookie=402e65d1cb84eb7d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:19.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.072 7f285207b700 1 -- 172.21.2.112:0/2826426574 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f284400c0c0 con 0x7f28540d1870 2021-02-10T11:41:19.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.072 7f285207b700 1 -- 172.21.2.112:0/2826426574 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f284400c340 con 0x7f28540d1870 2021-02-10T11:41:19.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.072 7f285207b700 1 -- 172.21.2.112:0/2826426574 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f284400c860 con 0x7f28540d1870 2021-02-10T11:41:19.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.072 7f28592e3700 1 -- 172.21.2.112:0/2826426574 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f28540d1870 msgr2=0x7f28541b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:19.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.072 7f28592e3700 1 --2- 172.21.2.112:0/2826426574 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f28540d1870 0x7f28541b03f0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 rx=0x7f2844006650 tx=0x7f2844009a80).stop 2021-02-10T11:41:19.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.072 7f28592e3700 1 -- 172.21.2.112:0/2826426574 shutdown_connections 2021-02-10T11:41:19.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.072 7f28592e3700 1 --2- 172.21.2.112:0/2826426574 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f28540d1870 0x7f28541b03f0 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:19.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.072 7f28592e3700 1 -- 172.21.2.112:0/2826426574 >> 172.21.2.112:0/2826426574 conn(0x7f28540ce450 msgr2=0x7f28540d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:19.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.072 7f28592e3700 1 -- 172.21.2.112:0/2826426574 shutdown_connections 2021-02-10T11:41:19.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.072 7f28592e3700 1 -- 172.21.2.112:0/2826426574 wait complete. 2021-02-10T11:41:19.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.087 7f28592e3700 1 Processor -- start 2021-02-10T11:41:19.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.087 7f28592e3700 1 -- start start 2021-02-10T11:41:19.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.087 7f28592e3700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f28540b0ad0 0x7f2854091f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:19.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.087 7f28592e3700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f28541a8850 0x7f28541a8c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:19.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.087 7f28592e3700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f28541a6fa0 0x7f28541a73a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:19.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.087 7f28592e3700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28541abfe0 con 0x7f28541a6fa0 2021-02-10T11:41:19.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.087 7f28592e3700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28541a9c40 con 0x7f28541a8850 2021-02-10T11:41:19.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.087 7f28592e3700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28541a9730 con 0x7f28540b0ad0 2021-02-10T11:41:19.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.088 7f285387e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f28541a6fa0 0x7f28541a73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:19.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.088 7f285387e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f28541a6fa0 0x7f28541a73a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:55306/0 (socket says 172.21.2.112:55306) 2021-02-10T11:41:19.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.088 7f285387e700 1 -- 172.21.2.112:0/3248686650 learned_addr learned my addr 172.21.2.112:0/3248686650 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:41:19.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.088 7f285307d700 1 --2- 172.21.2.112:0/3248686650 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f28540b0ad0 0x7f2854091f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:19.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.088 7f285287c700 1 --2- 172.21.2.112:0/3248686650 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f28541a8850 0x7f28541a8c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:19.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.088 7f285387e700 1 -- 172.21.2.112:0/3248686650 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f28540b0ad0 msgr2=0x7f2854091f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:19.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.088 7f285387e700 1 --2- 172.21.2.112:0/3248686650 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f28540b0ad0 0x7f2854091f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:19.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.088 7f285387e700 1 -- 172.21.2.112:0/3248686650 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f28541a8850 msgr2=0x7f28541a8c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:19.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.088 7f285387e700 1 --2- 172.21.2.112:0/3248686650 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f28541a8850 0x7f28541a8c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:19.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.088 7f285387e700 1 -- 172.21.2.112:0/3248686650 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28541a9270 con 0x7f28541a6fa0 2021-02-10T11:41:19.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.088 7f285387e700 1 --2- 172.21.2.112:0/3248686650 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f28541a6fa0 0x7f28541a73a0 secure :-1 s=READY pgs=2178 cs=0 l=1 rev1=1 rx=0x7f284c009980 tx=0x7f284c00bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:19.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.088 7f284bfff700 1 -- 172.21.2.112:0/3248686650 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f284c00be10 con 0x7f28541a6fa0 2021-02-10T11:41:19.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.088 7f284bfff700 1 -- 172.21.2.112:0/3248686650 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f284c0060c0 con 0x7f28541a6fa0 2021-02-10T11:41:19.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.088 7f284bfff700 1 -- 172.21.2.112:0/3248686650 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f284c006550 con 0x7f28541a6fa0 2021-02-10T11:41:19.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.088 7f28592e3700 1 -- 172.21.2.112:0/3248686650 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28541a9270 con 0x7f28541a6fa0 2021-02-10T11:41:19.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.088 7f28592e3700 1 -- 172.21.2.112:0/3248686650 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28541a7cb0 con 0x7f28541a6fa0 2021-02-10T11:41:19.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.089 7f284bfff700 1 -- 172.21.2.112:0/3248686650 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f284c0062a0 con 0x7f28541a6fa0 2021-02-10T11:41:19.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.089 7f284bfff700 1 --2- 172.21.2.112:0/3248686650 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f28380186c0 0x7f283801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:19.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.089 7f284bfff700 1 -- 172.21.2.112:0/3248686650 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f284c0325c0 con 0x7f28541a6fa0 2021-02-10T11:41:19.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.089 7f285307d700 1 --2- 172.21.2.112:0/3248686650 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f28380186c0 0x7f283801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:19.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.090 7f285307d700 1 --2- 172.21.2.112:0/3248686650 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f28380186c0 0x7f283801abc0 crc :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:19.117 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.120 7f28592e3700 1 -- 172.21.2.112:0/3248686650 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28541a9710 con 0x7f28541a6fa0 2021-02-10T11:41:19.120 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.123 7f284bfff700 1 -- 172.21.2.112:0/3248686650 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f284c000f90 con 0x7f28541a6fa0 2021-02-10T11:41:19.253 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.256 7f28592e3700 1 -- 172.21.2.112:0/3248686650 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f28541a9710 con 0x7f28541a6fa0 2021-02-10T11:41:19.254 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.257 7f284bfff700 1 -- 172.21.2.112:0/3248686650 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v68) v1 ==== 80+0+4513 (secure 0 0 0) 0x7f284c0062a0 con 0x7f28541a6fa0 2021-02-10T11:41:19.254 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:41:19.254 INFO:teuthology.orchestra.run.gibba012.stdout:{ 2021-02-10T11:41:19.254 INFO:teuthology.orchestra.run.gibba012.stdout: "auth_dump": [ 2021-02-10T11:41:19.255 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:41:19.255 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "mds.a", 2021-02-10T11:41:19.255 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCKvyNgPkV4AxAAEmVNaghdccP6lWd8hvQnxw==", 2021-02-10T11:41:19.255 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:41:19.255 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:41:19.255 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:41:19.255 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:41:19.255 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:41:19.255 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:41:19.255 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:41:19.255 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:41:19.255 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "mds.b", 2021-02-10T11:41:19.255 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCKvyNguxONLhAAHIpry+R89MAwdhFJyEhVCw==", 2021-02-10T11:41:19.256 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:41:19.256 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:41:19.256 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:41:19.256 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:41:19.256 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:41:19.256 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:41:19.256 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:41:19.256 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:41:19.256 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.0", 2021-02-10T11:41:19.256 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCSvyNgXcKPLhAAP/Bj2F+06DCtPCzQDNq+cQ==", 2021-02-10T11:41:19.256 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:41:19.256 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:41:19.257 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:41:19.257 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:41:19.257 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:41:19.257 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:41:19.257 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:41:19.257 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.1", 2021-02-10T11:41:19.257 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCVvyNghfbcLhAA66PrRwR2M7o6VTF5UhU/Fg==", 2021-02-10T11:41:19.257 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:41:19.257 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:41:19.257 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:41:19.257 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:41:19.257 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:41:19.257 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:41:19.258 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:41:19.258 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.2", 2021-02-10T11:41:19.258 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCYvyNgwiPqLhAA66BVkRsgaUlGpAsQ15Nf8Q==", 2021-02-10T11:41:19.258 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:41:19.258 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:41:19.258 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:41:19.258 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:41:19.258 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:41:19.258 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:41:19.258 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:41:19.258 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.3", 2021-02-10T11:41:19.258 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCbvyNgqBZrLxAAERPe/1L3SXAEzI3d9ReHHg==", 2021-02-10T11:41:19.258 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:41:19.259 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:41:19.259 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:41:19.259 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:41:19.259 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:41:19.259 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:41:19.259 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:41:19.259 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.4", 2021-02-10T11:41:19.259 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCkvyNgDQsxIhAAHmVjVYnx20N4SDmKqvbcuA==", 2021-02-10T11:41:19.259 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:41:19.259 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:41:19.259 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:41:19.259 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:41:19.259 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:41:19.260 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:41:19.260 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:41:19.260 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.5", 2021-02-10T11:41:19.260 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCnvyNg8ITFIhAADd4aKybAV7btoGPx3kCg3A==", 2021-02-10T11:41:19.260 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:41:19.260 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:41:19.261 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:41:19.261 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:41:19.261 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:41:19.261 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:41:19.261 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:41:19.261 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.6", 2021-02-10T11:41:19.262 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCqvyNgxOhDIxAAzXixEa/nlDc1xXrg0aeLwg==", 2021-02-10T11:41:19.262 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:41:19.262 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:41:19.262 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:41:19.262 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:41:19.263 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:41:19.263 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:41:19.263 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:41:19.263 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.7", 2021-02-10T11:41:19.263 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCtvyNgxSqGIxAACxJQlwP7DpiVVCZHA3zn1A==", 2021-02-10T11:41:19.263 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:41:19.264 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:41:19.264 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:41:19.264 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:41:19.264 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:41:19.264 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:41:19.264 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:41:19.264 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.0", 2021-02-10T11:41:19.265 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCLvyNgDkNiHRAAygU8Za4PBusTqCgwAYWTiQ==", 2021-02-10T11:41:19.265 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:41:19.265 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:41:19.265 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow r", 2021-02-10T11:41:19.265 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:41:19.265 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:41:19.265 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:41:19.265 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:41:19.265 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.1", 2021-02-10T11:41:19.265 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCLvyNgxmLqMxAA1MlSDJ9LH9u4f2sL7mm8/Q==", 2021-02-10T11:41:19.265 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:41:19.265 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:41:19.265 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow r", 2021-02-10T11:41:19.266 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:41:19.266 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:41:19.266 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:41:19.266 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:41:19.266 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.2", 2021-02-10T11:41:19.266 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCMvyNgp8veDhAAniSVHYPtwwTog/pjbaRQmw==", 2021-02-10T11:41:19.266 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:41:19.266 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:41:19.266 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow r", 2021-02-10T11:41:19.266 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:41:19.266 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:41:19.267 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:41:19.267 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:41:19.267 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.3", 2021-02-10T11:41:19.267 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCMvyNgQEVoJRAApHcWdm1sldIxu+heyD4HdQ==", 2021-02-10T11:41:19.267 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:41:19.267 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:41:19.267 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow r", 2021-02-10T11:41:19.267 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:41:19.267 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:41:19.267 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:41:19.267 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:41:19.267 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.admin", 2021-02-10T11:41:19.267 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCGvyNg7jjJBRAAtot+ypUZOiTo1GbY+oVhVw==", 2021-02-10T11:41:19.268 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:41:19.268 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow *", 2021-02-10T11:41:19.268 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:41:19.268 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:41:19.268 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:41:19.268 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:41:19.268 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:41:19.268 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:41:19.268 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "mgr.x", 2021-02-10T11:41:19.268 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCJvyNgTPVwKBAAjduiBE8+RSXBv82tdPecWg==", 2021-02-10T11:41:19.268 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:41:19.268 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow *", 2021-02-10T11:41:19.268 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow profile mgr", 2021-02-10T11:41:19.269 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:41:19.269 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:41:19.269 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:41:19.269 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:41:19.269 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "mgr.y", 2021-02-10T11:41:19.269 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCJvyNgQ1tTDhAASwBssDy+Rz31ujw8fNlWow==", 2021-02-10T11:41:19.269 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:41:19.269 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow *", 2021-02-10T11:41:19.269 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow profile mgr", 2021-02-10T11:41:19.269 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:41:19.269 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:41:19.269 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:41:19.269 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:41:19.270 INFO:teuthology.orchestra.run.gibba012.stdout:} 2021-02-10T11:41:19.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.258 7f28592e3700 1 -- 172.21.2.112:0/3248686650 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f28380186c0 msgr2=0x7f283801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:19.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.258 7f28592e3700 1 --2- 172.21.2.112:0/3248686650 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f28380186c0 0x7f283801abc0 crc :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:19.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.258 7f28592e3700 1 -- 172.21.2.112:0/3248686650 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f28541a6fa0 msgr2=0x7f28541a73a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:19.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.258 7f28592e3700 1 --2- 172.21.2.112:0/3248686650 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f28541a6fa0 0x7f28541a73a0 secure :-1 s=READY pgs=2178 cs=0 l=1 rev1=1 rx=0x7f284c009980 tx=0x7f284c00bcd0).stop 2021-02-10T11:41:19.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.259 7f28592e3700 1 -- 172.21.2.112:0/3248686650 shutdown_connections 2021-02-10T11:41:19.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.259 7f28592e3700 1 --2- 172.21.2.112:0/3248686650 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f28380186c0 0x7f283801abc0 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:19.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.259 7f28592e3700 1 --2- 172.21.2.112:0/3248686650 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f28541a8850 0x7f28541a8c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:19.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.259 7f28592e3700 1 --2- 172.21.2.112:0/3248686650 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f28541a6fa0 0x7f28541a73a0 unknown :-1 s=CLOSED pgs=2178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:19.272 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.259 7f28592e3700 1 --2- 172.21.2.112:0/3248686650 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f28540b0ad0 0x7f2854091f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:19.272 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.259 7f28592e3700 1 -- 172.21.2.112:0/3248686650 >> 172.21.2.112:0/3248686650 conn(0x7f285419f8f0 msgr2=0x7f28541a5f80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:19.272 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.259 7f28592e3700 1 -- 172.21.2.112:0/3248686650 shutdown_connections 2021-02-10T11:41:19.272 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.259 7f28592e3700 1 -- 172.21.2.112:0/3248686650 wait complete. 2021-02-10T11:41:19.272 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:41:19.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.685 7fd64fbee700 1 Processor -- start 2021-02-10T11:41:19.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.685 7fd64fbee700 1 -- start start 2021-02-10T11:41:19.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.686 7fd64fbee700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fd6481b0470 0x7fd6481b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:19.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.686 7fd64fbee700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fd6480ef610 0x7fd6480efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:19.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.686 7fd64fbee700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6480f0b50 con 0x7fd6480f0030 2021-02-10T11:41:19.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.686 7fd64fbee700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6480e52e0 con 0x7fd6480ef610 2021-02-10T11:41:19.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.686 7fd64fbee700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6480e5490 con 0x7fd6481b0470 2021-02-10T11:41:19.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.686 7fd64d209700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fd6480ef610 0x7fd6480efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:19.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.686 7fd64d209700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fd6480ef610 0x7fd6480efa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:55310/0 (socket says 172.21.2.112:55310) 2021-02-10T11:41:19.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.686 7fd64d209700 1 -- 172.21.2.112:0/2975662758 learned_addr learned my addr 172.21.2.112:0/2975662758 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:41:19.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.686 7fd64da0a700 1 --2- 172.21.2.112:0/2975662758 >> v2:172.21.2.120:3300/0 conn(0x7fd6481b0470 0x7fd6481b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:19.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.686 7fd64d209700 1 -- 172.21.2.112:0/2975662758 >> v2:172.21.2.120:3300/0 conn(0x7fd6481b0470 msgr2=0x7fd6481b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:19.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.686 7fd64d209700 1 --2- 172.21.2.112:0/2975662758 >> v2:172.21.2.120:3300/0 conn(0x7fd6481b0470 0x7fd6481b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:19.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.686 7fd64d209700 1 -- 172.21.2.112:0/2975662758 >> v1:172.21.2.112:6789/0 conn(0x7fd6480f0030 legacy=0x7fd6480f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:19.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.686 7fd64d209700 1 -- 172.21.2.112:0/2975662758 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6480f0430 con 0x7fd6480ef610 2021-02-10T11:41:19.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.686 7fd64da0a700 1 --2- 172.21.2.112:0/2975662758 >> v2:172.21.2.120:3300/0 conn(0x7fd6481b0470 0x7fd6481b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:41:19.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.686 7fd64d209700 1 --2- 172.21.2.112:0/2975662758 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd6480ef610 0x7fd6480efa10 secure :-1 s=READY pgs=2180 cs=0 l=1 rev1=1 rx=0x7fd63c006700 tx=0x7fd63c009b80).ready entity=mon.0 client_cookie=1900202aca7fa8ba server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:19.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.687 7fd64ca08700 1 -- 172.21.2.112:0/2975662758 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd63c00c0c0 con 0x7fd6480ef610 2021-02-10T11:41:19.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.687 7fd64ca08700 1 -- 172.21.2.112:0/2975662758 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd63c00c400 con 0x7fd6480ef610 2021-02-10T11:41:19.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.687 7fd64ca08700 1 -- 172.21.2.112:0/2975662758 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd63c00c920 con 0x7fd6480ef610 2021-02-10T11:41:19.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.687 7fd64fbee700 1 -- 172.21.2.112:0/2975662758 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd6480ef610 msgr2=0x7fd6480efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:19.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.687 7fd64fbee700 1 --2- 172.21.2.112:0/2975662758 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd6480ef610 0x7fd6480efa10 secure :-1 s=READY pgs=2180 cs=0 l=1 rev1=1 rx=0x7fd63c006700 tx=0x7fd63c009b80).stop 2021-02-10T11:41:19.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.687 7fd64fbee700 1 -- 172.21.2.112:0/2975662758 shutdown_connections 2021-02-10T11:41:19.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.687 7fd64fbee700 1 --2- 172.21.2.112:0/2975662758 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd6480ef610 0x7fd6480efa10 unknown :-1 s=CLOSED pgs=2180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:19.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.687 7fd64fbee700 1 --2- 172.21.2.112:0/2975662758 >> v2:172.21.2.120:3300/0 conn(0x7fd6481b0470 0x7fd6481b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:19.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.687 7fd64fbee700 1 -- 172.21.2.112:0/2975662758 >> 172.21.2.112:0/2975662758 conn(0x7fd6480ce3e0 msgr2=0x7fd6480e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:19.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.687 7fd64fbee700 1 -- 172.21.2.112:0/2975662758 shutdown_connections 2021-02-10T11:41:19.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.688 7fd64fbee700 1 -- 172.21.2.112:0/2975662758 wait complete. 2021-02-10T11:41:19.704 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.702 7fd64fbee700 1 Processor -- start 2021-02-10T11:41:19.704 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.702 7fd64fbee700 1 -- start start 2021-02-10T11:41:19.704 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.702 7fd64fbee700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd6480aed00 0x7fd648096400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:19.704 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.702 7fd64fbee700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd64818ef20 0x7fd6480955e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:19.704 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.702 7fd64fbee700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd648095c00 0x7fd648093240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:19.704 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.702 7fd64fbee700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd648109ad0 con 0x7fd648095c00 2021-02-10T11:41:19.704 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.702 7fd64fbee700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd648096a20 con 0x7fd64818ef20 2021-02-10T11:41:19.704 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.702 7fd64fbee700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd648096000 con 0x7fd6480aed00 2021-02-10T11:41:19.704 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.703 7fd64e20b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd648095c00 0x7fd648093240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:19.704 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.703 7fd64e20b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd648095c00 0x7fd648093240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:55318/0 (socket says 172.21.2.112:55318) 2021-02-10T11:41:19.704 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.703 7fd64e20b700 1 -- 172.21.2.112:0/3582585455 learned_addr learned my addr 172.21.2.112:0/3582585455 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:41:19.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.703 7fd64e20b700 1 -- 172.21.2.112:0/3582585455 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd6480aed00 msgr2=0x7fd648096400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:19.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.703 7fd64e20b700 1 --2- 172.21.2.112:0/3582585455 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd6480aed00 0x7fd648096400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:19.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.703 7fd64e20b700 1 -- 172.21.2.112:0/3582585455 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd64818ef20 msgr2=0x7fd6480955e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:19.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.703 7fd64e20b700 1 --2- 172.21.2.112:0/3582585455 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd64818ef20 0x7fd6480955e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:19.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.703 7fd64e20b700 1 -- 172.21.2.112:0/3582585455 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd648093860 con 0x7fd648095c00 2021-02-10T11:41:19.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.703 7fd64e20b700 1 --2- 172.21.2.112:0/3582585455 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd648095c00 0x7fd648093240 secure :-1 s=READY pgs=2181 cs=0 l=1 rev1=1 rx=0x7fd640009980 tx=0x7fd64000bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:19.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.703 7fd63a7fc700 1 -- 172.21.2.112:0/3582585455 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd640005a60 con 0x7fd648095c00 2021-02-10T11:41:19.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.703 7fd63a7fc700 1 -- 172.21.2.112:0/3582585455 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd640005ce0 con 0x7fd648095c00 2021-02-10T11:41:19.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.703 7fd63a7fc700 1 -- 172.21.2.112:0/3582585455 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd6400062b0 con 0x7fd648095c00 2021-02-10T11:41:19.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.703 7fd64fbee700 1 -- 172.21.2.112:0/3582585455 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd648093860 con 0x7fd648095c00 2021-02-10T11:41:19.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.703 7fd64fbee700 1 -- 172.21.2.112:0/3582585455 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd648093860 con 0x7fd648095c00 2021-02-10T11:41:19.707 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.704 7fd63a7fc700 1 -- 172.21.2.112:0/3582585455 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd640005870 con 0x7fd648095c00 2021-02-10T11:41:19.707 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.704 7fd63a7fc700 1 --2- 172.21.2.112:0/3582585455 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd6300186c0 0x7fd63001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:19.707 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.704 7fd63a7fc700 1 -- 172.21.2.112:0/3582585455 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 4753+0+0 (secure 0 0 0) 0x7fd640032630 con 0x7fd648095c00 2021-02-10T11:41:19.707 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.704 7fd64da0a700 1 --2- 172.21.2.112:0/3582585455 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd6300186c0 0x7fd63001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:19.707 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.705 7fd64da0a700 1 --2- 172.21.2.112:0/3582585455 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd6300186c0 0x7fd63001abc0 crc :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:19.732 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.735 7fd64fbee700 1 -- 172.21.2.112:0/3582585455 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd648105a20 con 0x7fd648095c00 2021-02-10T11:41:19.735 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.738 7fd63a7fc700 1 -- 172.21.2.112:0/3582585455 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd64000be10 con 0x7fd648095c00 2021-02-10T11:41:19.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.862 7fd64fbee700 1 -- 172.21.2.112:0/3582585455 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd6480f0b50 con 0x7fd648095c00 2021-02-10T11:41:19.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.863 7fd63a7fc700 1 -- 172.21.2.112:0/3582585455 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v112) v1 ==== 74+0+11153 (secure 0 0 0) 0x7fd64000be10 con 0x7fd648095c00 2021-02-10T11:41:19.860 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:41:19.860 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":112,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:41:18.614763","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:41:19.862 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.864 7fd64fbee700 1 -- 172.21.2.112:0/3582585455 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd6300186c0 msgr2=0x7fd63001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:19.862 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.864 7fd64fbee700 1 --2- 172.21.2.112:0/3582585455 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd6300186c0 0x7fd63001abc0 crc :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:19.862 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.864 7fd64fbee700 1 -- 172.21.2.112:0/3582585455 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd648095c00 msgr2=0x7fd648093240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:19.862 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.864 7fd64fbee700 1 --2- 172.21.2.112:0/3582585455 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd648095c00 0x7fd648093240 secure :-1 s=READY pgs=2181 cs=0 l=1 rev1=1 rx=0x7fd640009980 tx=0x7fd64000bcd0).stop 2021-02-10T11:41:19.862 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.865 7fd64fbee700 1 -- 172.21.2.112:0/3582585455 shutdown_connections 2021-02-10T11:41:19.862 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.865 7fd64fbee700 1 --2- 172.21.2.112:0/3582585455 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd6300186c0 0x7fd63001abc0 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:19.862 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.865 7fd64fbee700 1 --2- 172.21.2.112:0/3582585455 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd64818ef20 0x7fd6480955e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:19.862 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.865 7fd64fbee700 1 --2- 172.21.2.112:0/3582585455 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd648095c00 0x7fd648093240 unknown :-1 s=CLOSED pgs=2181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:19.862 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.865 7fd64fbee700 1 --2- 172.21.2.112:0/3582585455 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd6480aed00 0x7fd648096400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:19.862 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.865 7fd64fbee700 1 -- 172.21.2.112:0/3582585455 >> 172.21.2.112:0/3582585455 conn(0x7fd6481a37a0 msgr2=0x7fd64819d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:19.862 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.865 7fd64fbee700 1 -- 172.21.2.112:0/3582585455 shutdown_connections 2021-02-10T11:41:19.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:19.865 7fd64fbee700 1 -- 172.21.2.112:0/3582585455 wait complete. 2021-02-10T11:41:19.870 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:13:57.508758', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2021-02-10T11:41:19.871 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:41:20.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.290 7f02d52dc700 1 Processor -- start 2021-02-10T11:41:20.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.290 7f02d52dc700 1 -- start start 2021-02-10T11:41:20.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.290 7f02d52dc700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f02d00d1c70 0x7f02d00cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:20.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.290 7f02d52dc700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02d00d0170 con 0x7f02d00cf580 2021-02-10T11:41:20.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.290 7f02d52dc700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02d00fe1c0 con 0x7f02d00d1870 2021-02-10T11:41:20.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.290 7f02d52dc700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02d01b0cb0 con 0x7f02d00d1c70 2021-02-10T11:41:20.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.290 7f02cf87e700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f02d00cf580 0x7f02d00cf980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:54784/0 (socket says 172.21.2.112:54784) 2021-02-10T11:41:20.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.290 7f02cf87e700 1 -- 172.21.2.112:0/3853468425 learned_addr learned my addr 172.21.2.112:0/3853468425 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:41:20.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.290 7f02ce07b700 1 -- 172.21.2.112:0/3853468425 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1579181943 0 0) 0x7f02c8001500 con 0x7f02d00cf580 2021-02-10T11:41:20.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.290 7f02ce87c700 1 --2- 172.21.2.112:0/3853468425 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f02d00d1c70 0x7f02d00cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:20.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.291 7f02ce07b700 1 -- 172.21.2.112:0/3853468425 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02bc002fe0 con 0x7f02d00cf580 2021-02-10T11:41:20.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.291 7f02ce07b700 1 -- 172.21.2.112:0/3853468425 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 957522449 0 0) 0x7f02c0000ff0 con 0x7f02d00d1870 2021-02-10T11:41:20.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.291 7f02ce07b700 1 -- 172.21.2.112:0/3853468425 --> v1:172.21.2.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02bc003210 con 0x7f02d00d1870 2021-02-10T11:41:20.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.291 7f02ce07b700 1 -- 172.21.2.112:0/3853468425 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2089816748 0 0) 0x7f02c8001460 con 0x7f02d00cf580 2021-02-10T11:41:20.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.291 7f02ce07b700 1 -- 172.21.2.112:0/3853468425 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f02d00d1c70 msgr2=0x7f02d00cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:20.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.291 7f02ce07b700 1 --2- 172.21.2.112:0/3853468425 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f02d00d1c70 0x7f02d00cef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:20.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.291 7f02ce07b700 1 -- 172.21.2.112:0/3853468425 >> v1:172.21.2.120:6789/0 conn(0x7f02d00d1870 legacy=0x7f02d01b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:20.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.291 7f02ce07b700 1 -- 172.21.2.112:0/3853468425 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02d0109a60 con 0x7f02d00cf580 2021-02-10T11:41:20.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.291 7f02ce07b700 1 -- 172.21.2.112:0/3853468425 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f02c8001980 con 0x7f02d00cf580 2021-02-10T11:41:20.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.291 7f02ce07b700 1 -- 172.21.2.112:0/3853468425 >> v1:172.21.2.112:6789/0 conn(0x7f02d00cf580 legacy=0x7f02d00cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:20.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.291 7f02ce07b700 1 --2- 172.21.2.112:0/3853468425 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f02bc0037e0 0x7f02bc003c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:20.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.291 7f02ce07b700 1 --2- 172.21.2.112:0/3853468425 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f02bc0067e0 0x7f02bc008c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:20.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.291 7f02ce07b700 1 --2- 172.21.2.112:0/3853468425 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f02bc009580 0x7f02bc00ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:20.306 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.291 7f02ce07b700 1 -- 172.21.2.112:0/3853468425 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02bc0502d0 con 0x7f02bc0067e0 2021-02-10T11:41:20.306 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.291 7f02ce07b700 1 -- 172.21.2.112:0/3853468425 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02bc0504b0 con 0x7f02bc0037e0 2021-02-10T11:41:20.306 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.291 7f02ce07b700 1 -- 172.21.2.112:0/3853468425 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02bc0506d0 con 0x7f02bc009580 2021-02-10T11:41:20.306 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.291 7f02ce87c700 1 --2- 172.21.2.112:0/3853468425 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f02bc0067e0 0x7f02bc008c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:20.306 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.291 7f02cf87e700 1 --2- 172.21.2.112:0/3853468425 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f02bc009580 0x7f02bc00ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:20.306 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.291 7f02cf07d700 1 --2- 172.21.2.112:0/3853468425 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f02bc0037e0 0x7f02bc003c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:20.306 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.291 7f02ce87c700 1 -- 172.21.2.112:0/3853468425 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f02bc009580 msgr2=0x7f02bc00ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:20.306 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.291 7f02ce87c700 1 --2- 172.21.2.112:0/3853468425 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f02bc009580 0x7f02bc00ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:20.306 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.291 7f02ce87c700 1 -- 172.21.2.112:0/3853468425 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f02bc0037e0 msgr2=0x7f02bc003c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:20.306 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.291 7f02ce87c700 1 --2- 172.21.2.112:0/3853468425 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f02bc0037e0 0x7f02bc003c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:20.306 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.291 7f02ce87c700 1 -- 172.21.2.112:0/3853468425 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02bc050970 con 0x7f02bc0067e0 2021-02-10T11:41:20.306 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.291 7f02ce87c700 1 --2- 172.21.2.112:0/3853468425 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f02bc0067e0 0x7f02bc008c80 secure :-1 s=READY pgs=2183 cs=0 l=1 rev1=1 rx=0x7f02b8006ac0 tx=0x7f02b8009a30).ready entity=mon.0 client_cookie=e74f324c63a56f04 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:20.306 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.291 7f02ce07b700 1 -- 172.21.2.112:0/3853468425 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f02b800c0c0 con 0x7f02bc0067e0 2021-02-10T11:41:20.307 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.291 7f02ce07b700 1 -- 172.21.2.112:0/3853468425 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02b8009e20 con 0x7f02bc0067e0 2021-02-10T11:41:20.307 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.291 7f02ce07b700 1 -- 172.21.2.112:0/3853468425 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f02b800c680 con 0x7f02bc0067e0 2021-02-10T11:41:20.307 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.291 7f02d52dc700 1 -- 172.21.2.112:0/3853468425 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f02bc0067e0 msgr2=0x7f02bc008c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:20.307 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.291 7f02d52dc700 1 --2- 172.21.2.112:0/3853468425 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f02bc0067e0 0x7f02bc008c80 secure :-1 s=READY pgs=2183 cs=0 l=1 rev1=1 rx=0x7f02b8006ac0 tx=0x7f02b8009a30).stop 2021-02-10T11:41:20.307 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.291 7f02cf87e700 1 -- 172.21.2.112:0/3853468425 reap_dead start 2021-02-10T11:41:20.307 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.292 7f02d52dc700 1 -- 172.21.2.112:0/3853468425 shutdown_connections 2021-02-10T11:41:20.307 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.292 7f02d52dc700 1 -- 172.21.2.112:0/3853468425 >> 172.21.2.112:0/3853468425 conn(0x7f02d00ce420 msgr2=0x7f02d00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:20.307 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.292 7f02d52dc700 1 -- 172.21.2.112:0/3853468425 shutdown_connections 2021-02-10T11:41:20.307 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.292 7f02d52dc700 1 -- 172.21.2.112:0/3853468425 wait complete. 2021-02-10T11:41:20.307 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.307 7f02d52dc700 1 Processor -- start 2021-02-10T11:41:20.307 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.307 7f02d52dc700 1 -- start start 2021-02-10T11:41:20.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.307 7f02d52dc700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f02d01a79a0 0x7f02d01a7da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:20.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.307 7f02d52dc700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f02d00bdb40 0x7f02d00bdf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:20.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.307 7f02d52dc700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f02d0082ca0 0x7f02d00830a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:20.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.307 7f02d52dc700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02d019d990 con 0x7f02d01a79a0 2021-02-10T11:41:20.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.307 7f02d52dc700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02d019dbb0 con 0x7f02d0082ca0 2021-02-10T11:41:20.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.307 7f02d52dc700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02d00836c0 con 0x7f02d00bdb40 2021-02-10T11:41:20.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.307 7f02cf07d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f02d01a79a0 0x7f02d01a7da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:20.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.308 7f02cf07d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f02d01a79a0 0x7f02d01a7da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:55336/0 (socket says 172.21.2.112:55336) 2021-02-10T11:41:20.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.308 7f02cf07d700 1 -- 172.21.2.112:0/1756836926 learned_addr learned my addr 172.21.2.112:0/1756836926 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:41:20.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.308 7f02ce87c700 1 --2- 172.21.2.112:0/1756836926 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f02d00bdb40 0x7f02d00bdf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:20.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.308 7f02cf87e700 1 --2- 172.21.2.112:0/1756836926 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f02d0082ca0 0x7f02d00830a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:20.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.308 7f02cf07d700 1 -- 172.21.2.112:0/1756836926 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f02d00bdb40 msgr2=0x7f02d00bdf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:20.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.308 7f02cf07d700 1 --2- 172.21.2.112:0/1756836926 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f02d00bdb40 0x7f02d00bdf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:20.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.308 7f02cf07d700 1 -- 172.21.2.112:0/1756836926 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f02d0082ca0 msgr2=0x7f02d00830a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:20.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.308 7f02cf07d700 1 --2- 172.21.2.112:0/1756836926 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f02d0082ca0 0x7f02d00830a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:20.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.308 7f02cf07d700 1 -- 172.21.2.112:0/1756836926 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02d00838e0 con 0x7f02d01a79a0 2021-02-10T11:41:20.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.308 7f02cf87e700 1 --2- 172.21.2.112:0/1756836926 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f02d0082ca0 0x7f02d00830a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:41:20.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.308 7f02cf07d700 1 --2- 172.21.2.112:0/1756836926 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f02d01a79a0 0x7f02d01a7da0 secure :-1 s=READY pgs=2184 cs=0 l=1 rev1=1 rx=0x7f02c00066d0 tx=0x7f02c000b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:20.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.308 7f02c7fff700 1 -- 172.21.2.112:0/1756836926 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f02c000bd80 con 0x7f02d01a79a0 2021-02-10T11:41:20.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.308 7f02c7fff700 1 -- 172.21.2.112:0/1756836926 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02c0006b50 con 0x7f02d01a79a0 2021-02-10T11:41:20.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.308 7f02c7fff700 1 -- 172.21.2.112:0/1756836926 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f02c0002080 con 0x7f02d01a79a0 2021-02-10T11:41:20.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.308 7f02d52dc700 1 -- 172.21.2.112:0/1756836926 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02d00838e0 con 0x7f02d01a79a0 2021-02-10T11:41:20.311 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.308 7f02d52dc700 1 -- 172.21.2.112:0/1756836926 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02d00838e0 con 0x7f02d01a79a0 2021-02-10T11:41:20.311 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.309 7f02c7fff700 1 -- 172.21.2.112:0/1756836926 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f02c0002020 con 0x7f02d01a79a0 2021-02-10T11:41:20.311 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.309 7f02c7fff700 1 --2- 172.21.2.112:0/1756836926 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f02b4018730 0x7f02b401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:20.311 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.309 7f02c7fff700 1 -- 172.21.2.112:0/1756836926 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f02c0032310 con 0x7f02d01a79a0 2021-02-10T11:41:20.311 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.309 7f02ce87c700 1 --2- 172.21.2.112:0/1756836926 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f02b4018730 0x7f02b401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:20.311 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.310 7f02ce87c700 1 --2- 172.21.2.112:0/1756836926 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f02b4018730 0x7f02b401abc0 crc :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:20.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.340 7f02d52dc700 1 -- 172.21.2.112:0/1756836926 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02d0082a70 con 0x7f02d01a79a0 2021-02-10T11:41:20.339 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.342 7f02c7fff700 1 -- 172.21.2.112:0/1756836926 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f02c0000de0 con 0x7f02d01a79a0 2021-02-10T11:41:20.470 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.473 7f02d52dc700 1 -- 172.21.2.112:0/1756836926 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f02d0082a70 con 0x7f02d01a79a0 2021-02-10T11:41:20.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.474 7f02c7fff700 1 -- 172.21.2.112:0/1756836926 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v112) v1 ==== 92+0+10 (secure 0 0 0) 0x7f02c000bd80 con 0x7f02d01a79a0 2021-02-10T11:41:20.471 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:41:20.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.475 7f02d52dc700 1 -- 172.21.2.112:0/1756836926 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f02b4018730 msgr2=0x7f02b401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:20.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.475 7f02d52dc700 1 --2- 172.21.2.112:0/1756836926 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f02b4018730 0x7f02b401abc0 crc :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:20.473 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.475 7f02d52dc700 1 -- 172.21.2.112:0/1756836926 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f02d01a79a0 msgr2=0x7f02d01a7da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:20.473 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.475 7f02d52dc700 1 --2- 172.21.2.112:0/1756836926 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f02d01a79a0 0x7f02d01a7da0 secure :-1 s=READY pgs=2184 cs=0 l=1 rev1=1 rx=0x7f02c00066d0 tx=0x7f02c000b990).stop 2021-02-10T11:41:20.473 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.476 7f02d52dc700 1 -- 172.21.2.112:0/1756836926 shutdown_connections 2021-02-10T11:41:20.473 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.476 7f02d52dc700 1 --2- 172.21.2.112:0/1756836926 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f02b4018730 0x7f02b401abc0 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:20.473 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.476 7f02d52dc700 1 --2- 172.21.2.112:0/1756836926 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f02d0082ca0 0x7f02d00830a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:20.473 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.476 7f02d52dc700 1 --2- 172.21.2.112:0/1756836926 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f02d00bdb40 0x7f02d00bdf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:20.473 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.476 7f02d52dc700 1 --2- 172.21.2.112:0/1756836926 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f02d01a79a0 0x7f02d01a7da0 unknown :-1 s=CLOSED pgs=2184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:20.473 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.476 7f02d52dc700 1 -- 172.21.2.112:0/1756836926 >> 172.21.2.112:0/1756836926 conn(0x7f02d00bef00 msgr2=0x7f02d009f200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:20.473 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.476 7f02d52dc700 1 -- 172.21.2.112:0/1756836926 shutdown_connections 2021-02-10T11:41:20.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.476 7f02d52dc700 1 -- 172.21.2.112:0/1756836926 wait complete. 2021-02-10T11:41:20.481 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --format=json-pretty osd lspools 2021-02-10T11:41:20.909 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.894 7fcc510e3700 1 Processor -- start 2021-02-10T11:41:20.909 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.894 7fcc510e3700 1 -- start start 2021-02-10T11:41:20.909 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.895 7fcc510e3700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcc4c1b0870 0x7fcc4c0f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:20.909 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.895 7fcc510e3700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc4c1b0c70 con 0x7fcc4c0efe00 2021-02-10T11:41:20.909 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.895 7fcc510e3700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc4c0f0b50 con 0x7fcc4c1b0470 2021-02-10T11:41:20.909 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.895 7fcc510e3700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc4c0e5ea0 con 0x7fcc4c1b0870 2021-02-10T11:41:20.909 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.895 7fcc4b59e700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fcc4c0efe00 0x7fcc4c0f0200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:54808/0 (socket says 172.21.2.112:54808) 2021-02-10T11:41:20.909 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.895 7fcc4b59e700 1 -- 172.21.2.112:0/945150842 learned_addr learned my addr 172.21.2.112:0/945150842 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:41:20.909 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.895 7fcc49d9b700 1 -- 172.21.2.112:0/945150842 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3374374104 0 0) 0x7fcc44001500 con 0x7fcc4c0efe00 2021-02-10T11:41:20.910 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.895 7fcc49d9b700 1 -- 172.21.2.112:0/945150842 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc38002fe0 con 0x7fcc4c0efe00 2021-02-10T11:41:20.910 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.895 7fcc4a59c700 1 --2- 172.21.2.112:0/945150842 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcc4c1b0870 0x7fcc4c0f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:20.910 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.896 7fcc49d9b700 1 -- 172.21.2.112:0/945150842 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1955686199 0 0) 0x7fcc44001500 con 0x7fcc4c0efe00 2021-02-10T11:41:20.910 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.896 7fcc49d9b700 1 -- 172.21.2.112:0/945150842 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcc4c1b0870 msgr2=0x7fcc4c0f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:20.910 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.896 7fcc49d9b700 1 --2- 172.21.2.112:0/945150842 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcc4c1b0870 0x7fcc4c0f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:20.910 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.896 7fcc49d9b700 1 -- 172.21.2.112:0/945150842 >> v1:172.21.2.120:6789/0 conn(0x7fcc4c1b0470 legacy=0x7fcc4c0ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:20.910 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.896 7fcc49d9b700 1 -- 172.21.2.112:0/945150842 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc4c0e52e0 con 0x7fcc4c0efe00 2021-02-10T11:41:20.910 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.896 7fcc49d9b700 1 -- 172.21.2.112:0/945150842 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7fcc44001a00 con 0x7fcc4c0efe00 2021-02-10T11:41:20.910 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.896 7fcc49d9b700 1 -- 172.21.2.112:0/945150842 >> v1:172.21.2.112:6789/0 conn(0x7fcc4c0efe00 legacy=0x7fcc4c0f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:20.910 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.896 7fcc49d9b700 1 --2- 172.21.2.112:0/945150842 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcc38003b10 0x7fcc38005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:20.910 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.896 7fcc49d9b700 1 --2- 172.21.2.112:0/945150842 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcc38006840 0x7fcc38008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:20.910 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.896 7fcc49d9b700 1 --2- 172.21.2.112:0/945150842 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcc38009530 0x7fcc3800ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:20.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.896 7fcc49d9b700 1 -- 172.21.2.112:0/945150842 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc3800c2e0 con 0x7fcc38003b10 2021-02-10T11:41:20.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.896 7fcc49d9b700 1 -- 172.21.2.112:0/945150842 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc3800c500 con 0x7fcc38009530 2021-02-10T11:41:20.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.896 7fcc49d9b700 1 -- 172.21.2.112:0/945150842 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc3800c720 con 0x7fcc38006840 2021-02-10T11:41:20.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.896 7fcc4ad9d700 1 --2- 172.21.2.112:0/945150842 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcc38003b10 0x7fcc38005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:20.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.896 7fcc4b59e700 1 --2- 172.21.2.112:0/945150842 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcc38009530 0x7fcc3800ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:20.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.896 7fcc4a59c700 1 --2- 172.21.2.112:0/945150842 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcc38006840 0x7fcc38008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:20.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.896 7fcc4ad9d700 1 -- 172.21.2.112:0/945150842 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcc38006840 msgr2=0x7fcc38008c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:20.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.896 7fcc4ad9d700 1 --2- 172.21.2.112:0/945150842 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcc38006840 0x7fcc38008c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:20.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.896 7fcc4ad9d700 1 -- 172.21.2.112:0/945150842 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcc38009530 msgr2=0x7fcc3800ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:20.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.896 7fcc4ad9d700 1 --2- 172.21.2.112:0/945150842 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcc38009530 0x7fcc3800ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:20.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.896 7fcc4ad9d700 1 -- 172.21.2.112:0/945150842 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc38050990 con 0x7fcc38003b10 2021-02-10T11:41:20.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.896 7fcc4ad9d700 1 --2- 172.21.2.112:0/945150842 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcc38003b10 0x7fcc38005fa0 secure :-1 s=READY pgs=2186 cs=0 l=1 rev1=1 rx=0x7fcc3c001040 tx=0x7fcc3c009aa0).ready entity=mon.0 client_cookie=88c625cdeea1af63 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:20.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.896 7fcc49d9b700 1 -- 172.21.2.112:0/945150842 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcc3c001290 con 0x7fcc38003b10 2021-02-10T11:41:20.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.896 7fcc49d9b700 1 -- 172.21.2.112:0/945150842 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc3c0066e0 con 0x7fcc38003b10 2021-02-10T11:41:20.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.896 7fcc49d9b700 1 -- 172.21.2.112:0/945150842 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcc3c000c60 con 0x7fcc38003b10 2021-02-10T11:41:20.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.896 7fcc510e3700 1 -- 172.21.2.112:0/945150842 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcc38003b10 msgr2=0x7fcc38005fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:20.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.896 7fcc510e3700 1 --2- 172.21.2.112:0/945150842 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcc38003b10 0x7fcc38005fa0 secure :-1 s=READY pgs=2186 cs=0 l=1 rev1=1 rx=0x7fcc3c001040 tx=0x7fcc3c009aa0).stop 2021-02-10T11:41:20.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.897 7fcc4b59e700 1 -- 172.21.2.112:0/945150842 reap_dead start 2021-02-10T11:41:20.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.897 7fcc510e3700 1 -- 172.21.2.112:0/945150842 shutdown_connections 2021-02-10T11:41:20.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.897 7fcc510e3700 1 -- 172.21.2.112:0/945150842 >> 172.21.2.112:0/945150842 conn(0x7fcc4c0ce3e0 msgr2=0x7fcc4c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:20.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.897 7fcc510e3700 1 -- 172.21.2.112:0/945150842 shutdown_connections 2021-02-10T11:41:20.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.897 7fcc510e3700 1 -- 172.21.2.112:0/945150842 wait complete. 2021-02-10T11:41:20.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.912 7fcc510e3700 1 Processor -- start 2021-02-10T11:41:20.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.912 7fcc510e3700 1 -- start start 2021-02-10T11:41:20.914 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.912 7fcc510e3700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcc4c108ce0 0x7fcc4c09da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:20.914 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.912 7fcc510e3700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcc4c1090e0 0x7fcc4c09e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:20.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.912 7fcc510e3700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcc4c09eee0 0x7fcc4c09f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:20.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.912 7fcc510e3700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc4c1094e0 con 0x7fcc4c108ce0 2021-02-10T11:41:20.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.912 7fcc510e3700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc4c09e960 con 0x7fcc4c09eee0 2021-02-10T11:41:20.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.912 7fcc510e3700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc4c09e690 con 0x7fcc4c1090e0 2021-02-10T11:41:20.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.912 7fcc4ad9d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcc4c108ce0 0x7fcc4c09da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:20.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.912 7fcc4ad9d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcc4c108ce0 0x7fcc4c09da50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:55356/0 (socket says 172.21.2.112:55356) 2021-02-10T11:41:20.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.913 7fcc4ad9d700 1 -- 172.21.2.112:0/3215552698 learned_addr learned my addr 172.21.2.112:0/3215552698 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:41:20.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.913 7fcc4ad9d700 1 -- 172.21.2.112:0/3215552698 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcc4c1090e0 msgr2=0x7fcc4c09e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:20.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.913 7fcc4ad9d700 1 --2- 172.21.2.112:0/3215552698 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcc4c1090e0 0x7fcc4c09e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:20.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.913 7fcc4ad9d700 1 -- 172.21.2.112:0/3215552698 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcc4c09eee0 msgr2=0x7fcc4c09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:20.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.913 7fcc4ad9d700 1 --2- 172.21.2.112:0/3215552698 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcc4c09eee0 0x7fcc4c09f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:20.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.913 7fcc4ad9d700 1 -- 172.21.2.112:0/3215552698 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc4c09f900 con 0x7fcc4c108ce0 2021-02-10T11:41:20.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.913 7fcc4ad9d700 1 --2- 172.21.2.112:0/3215552698 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcc4c108ce0 0x7fcc4c09da50 secure :-1 s=READY pgs=2187 cs=0 l=1 rev1=1 rx=0x7fcc3c009fc0 tx=0x7fcc3c00ba10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:20.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.913 7fcc437fe700 1 -- 172.21.2.112:0/3215552698 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcc3c0091d0 con 0x7fcc4c108ce0 2021-02-10T11:41:20.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.913 7fcc437fe700 1 -- 172.21.2.112:0/3215552698 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc3c006240 con 0x7fcc4c108ce0 2021-02-10T11:41:20.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.913 7fcc510e3700 1 -- 172.21.2.112:0/3215552698 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc4c09f900 con 0x7fcc4c108ce0 2021-02-10T11:41:20.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.913 7fcc510e3700 1 -- 172.21.2.112:0/3215552698 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc4c09f900 con 0x7fcc4c108ce0 2021-02-10T11:41:20.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.913 7fcc437fe700 1 -- 172.21.2.112:0/3215552698 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcc3c005730 con 0x7fcc4c108ce0 2021-02-10T11:41:20.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.914 7fcc437fe700 1 -- 172.21.2.112:0/3215552698 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fcc3c005540 con 0x7fcc4c108ce0 2021-02-10T11:41:20.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.914 7fcc437fe700 1 --2- 172.21.2.112:0/3215552698 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcc30018730 0x7fcc3001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:20.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.914 7fcc437fe700 1 -- 172.21.2.112:0/3215552698 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 4753+0+0 (secure 0 0 0) 0x7fcc3c032470 con 0x7fcc4c108ce0 2021-02-10T11:41:20.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.914 7fcc4a59c700 1 --2- 172.21.2.112:0/3215552698 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcc30018730 0x7fcc3001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:20.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.915 7fcc4a59c700 1 --2- 172.21.2.112:0/3215552698 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcc30018730 0x7fcc3001ac10 crc :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:20.941 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.944 7fcc510e3700 1 -- 172.21.2.112:0/3215552698 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc4c0f0b50 con 0x7fcc4c108ce0 2021-02-10T11:41:20.944 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:20.947 7fcc437fe700 1 -- 172.21.2.112:0/3215552698 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fcc3c0064a0 con 0x7fcc4c108ce0 2021-02-10T11:41:21.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:21.073 7fcc510e3700 1 -- 172.21.2.112:0/3215552698 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fcc4c0f0b50 con 0x7fcc4c108ce0 2021-02-10T11:41:21.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:21.073 7fcc437fe700 1 -- 172.21.2.112:0/3215552698 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v112) v1 ==== 84+0+64 (secure 0 0 0) 0x7fcc3c00be00 con 0x7fcc4c108ce0 2021-02-10T11:41:21.071 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:41:21.071 INFO:teuthology.orchestra.run.gibba012.stdout:[ 2021-02-10T11:41:21.071 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:41:21.071 INFO:teuthology.orchestra.run.gibba012.stdout: "poolnum": 1, 2021-02-10T11:41:21.071 INFO:teuthology.orchestra.run.gibba012.stdout: "poolname": "rbd" 2021-02-10T11:41:21.071 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:41:21.071 INFO:teuthology.orchestra.run.gibba012.stdout:] 2021-02-10T11:41:21.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:21.075 7fcc510e3700 1 -- 172.21.2.112:0/3215552698 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcc30018730 msgr2=0x7fcc3001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:21.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:21.075 7fcc510e3700 1 --2- 172.21.2.112:0/3215552698 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcc30018730 0x7fcc3001ac10 crc :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:21.073 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:21.075 7fcc510e3700 1 -- 172.21.2.112:0/3215552698 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcc4c108ce0 msgr2=0x7fcc4c09da50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:21.073 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:21.075 7fcc510e3700 1 --2- 172.21.2.112:0/3215552698 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcc4c108ce0 0x7fcc4c09da50 secure :-1 s=READY pgs=2187 cs=0 l=1 rev1=1 rx=0x7fcc3c009fc0 tx=0x7fcc3c00ba10).stop 2021-02-10T11:41:21.073 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:21.075 7fcc510e3700 1 -- 172.21.2.112:0/3215552698 shutdown_connections 2021-02-10T11:41:21.073 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:21.075 7fcc510e3700 1 --2- 172.21.2.112:0/3215552698 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcc30018730 0x7fcc3001ac10 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:21.073 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:21.075 7fcc510e3700 1 --2- 172.21.2.112:0/3215552698 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcc4c09eee0 0x7fcc4c09f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:21.073 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:21.075 7fcc510e3700 1 --2- 172.21.2.112:0/3215552698 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcc4c1090e0 0x7fcc4c09e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:21.073 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:21.075 7fcc510e3700 1 --2- 172.21.2.112:0/3215552698 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcc4c108ce0 0x7fcc4c09da50 unknown :-1 s=CLOSED pgs=2187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:21.073 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:21.075 7fcc510e3700 1 -- 172.21.2.112:0/3215552698 >> 172.21.2.112:0/3215552698 conn(0x7fcc4c0bc6c0 msgr2=0x7fcc4c0b1f70 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:21.073 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:21.076 7fcc510e3700 1 -- 172.21.2.112:0/3215552698 shutdown_connections 2021-02-10T11:41:21.073 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:21.076 7fcc510e3700 1 -- 172.21.2.112:0/3215552698 wait complete. 2021-02-10T11:41:21.081 INFO:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2021-02-10T11:41:21.081 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_metadata 8 2021-02-10T11:41:21.514 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:21.499 7fa587aab700 1 Processor -- start 2021-02-10T11:41:21.514 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:21.499 7fa587aab700 1 -- start start 2021-02-10T11:41:21.514 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:21.499 7fa587aab700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa58010afe0 0x7fa5800cf740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:21.514 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:21.499 7fa587aab700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5800fe1b0 con 0x7fa5800d1c60 2021-02-10T11:41:21.515 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:21.499 7fa587aab700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5801b0ca0 con 0x7fa5800d1860 2021-02-10T11:41:21.515 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:21.499 7fa587aab700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa580109a50 con 0x7fa58010afe0 2021-02-10T11:41:21.515 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:21.499 7fa5850c6700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fa5800d1c60 0x7fa5800cef50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:54824/0 (socket says 172.21.2.112:54824) 2021-02-10T11:41:21.515 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:21.499 7fa5850c6700 1 -- 172.21.2.112:0/4282991201 learned_addr learned my addr 172.21.2.112:0/4282991201 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:41:21.515 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:21.500 7fa5848c5700 1 -- 172.21.2.112:0/4282991201 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2865617582 0 0) 0x7fa5740009b0 con 0x7fa5800d1c60 2021-02-10T11:41:21.515 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:21.500 7fa5848c5700 1 -- 172.21.2.112:0/4282991201 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa56c002fe0 con 0x7fa5800d1c60 2021-02-10T11:41:21.515 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:21.500 7fa5860c8700 1 --2- 172.21.2.112:0/4282991201 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa58010afe0 0x7fa5800cf740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:21.515 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:21.500 7fa5848c5700 1 -- 172.21.2.112:0/4282991201 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 895071656 0 0) 0x7fa574001010 con 0x7fa5800d1c60 2021-02-10T11:41:21.515 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:21.500 7fa5848c5700 1 -- 172.21.2.112:0/4282991201 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa58010afe0 msgr2=0x7fa5800cf740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:21.515 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:21.500 7fa5848c5700 1 --2- 172.21.2.112:0/4282991201 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa58010afe0 0x7fa5800cf740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:21.515 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:21.500 7fa5848c5700 1 -- 172.21.2.112:0/4282991201 >> v1:172.21.2.120:6789/0 conn(0x7fa5800d1860 legacy=0x7fa5801b04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:21.515 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:21.500 7fa5848c5700 1 -- 172.21.2.112:0/4282991201 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa58010b3e0 con 0x7fa5800d1c60 2021-02-10T11:41:21.515 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:21.500 7fa5848c5700 1 -- 172.21.2.112:0/4282991201 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7fa574001590 con 0x7fa5800d1c60 2021-02-10T11:41:21.516 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:21.500 7fa5848c5700 1 -- 172.21.2.112:0/4282991201 >> v1:172.21.2.112:6789/0 conn(0x7fa5800d1c60 legacy=0x7fa5800cef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:21.516 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:21.500 7fa5848c5700 1 --2- 172.21.2.112:0/4282991201 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa56c003a60 0x7fa56c005f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:21.516 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:21.500 7fa5848c5700 1 --2- 172.21.2.112:0/4282991201 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa56c006780 0x7fa56c008ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:21.516 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:21.500 7fa5848c5700 1 --2- 172.21.2.112:0/4282991201 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa56c009580 0x7fa56c00ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:21.516 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:21.500 7fa5848c5700 1 -- 172.21.2.112:0/4282991201 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa56c00c2c0 con 0x7fa56c006780 2021-02-10T11:41:21.516 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:21.500 7fa5848c5700 1 -- 172.21.2.112:0/4282991201 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa56c00c4e0 con 0x7fa56c003a60 2021-02-10T11:41:21.516 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:21.500 7fa5848c5700 1 -- 172.21.2.112:0/4282991201 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa56c00c700 con 0x7fa56c009580 2021-02-10T11:41:21.516 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:21.500 7fa5850c6700 1 --2- 172.21.2.112:0/4282991201 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa56c006780 0x7fa56c008ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:21.516 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:21.500 7fa5860c8700 1 --2- 172.21.2.112:0/4282991201 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa56c009580 0x7fa56c00ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:21.516 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:21.500 7fa5858c7700 1 --2- 172.21.2.112:0/4282991201 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa56c003a60 0x7fa56c005f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:21.516 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:21.501 7fa5850c6700 1 -- 172.21.2.112:0/4282991201 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa56c009580 msgr2=0x7fa56c00ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:21.516 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:21.501 7fa5850c6700 1 --2- 172.21.2.112:0/4282991201 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa56c009580 0x7fa56c00ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:21.516 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:21.501 7fa5850c6700 1 -- 172.21.2.112:0/4282991201 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa56c003a60 msgr2=0x7fa56c005f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:21.517 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:21.501 7fa5850c6700 1 --2- 172.21.2.112:0/4282991201 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa56c003a60 0x7fa56c005f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:21.517 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:21.501 7fa5850c6700 1 -- 172.21.2.112:0/4282991201 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa56c050970 con 0x7fa56c006780 2021-02-10T11:41:21.517 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:21.501 7fa5850c6700 1 --2- 172.21.2.112:0/4282991201 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa56c006780 0x7fa56c008ca0 secure :-1 s=READY pgs=2189 cs=0 l=1 rev1=1 rx=0x7fa5740017d0 tx=0x7fa574007d30).ready entity=mon.0 client_cookie=e35d0931549c7684 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:21.517 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:21.501 7fa5848c5700 1 -- 172.21.2.112:0/4282991201 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa57400a310 con 0x7fa56c006780 2021-02-10T11:41:21.517 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:21.501 7fa5848c5700 1 -- 172.21.2.112:0/4282991201 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa57400a650 con 0x7fa56c006780 2021-02-10T11:41:21.517 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:21.501 7fa5848c5700 1 -- 172.21.2.112:0/4282991201 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa57400aab0 con 0x7fa56c006780 2021-02-10T11:41:21.517 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:21.501 7fa587aab700 1 -- 172.21.2.112:0/4282991201 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa56c006780 msgr2=0x7fa56c008ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:21.517 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:21.501 7fa587aab700 1 --2- 172.21.2.112:0/4282991201 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa56c006780 0x7fa56c008ca0 secure :-1 s=READY pgs=2189 cs=0 l=1 rev1=1 rx=0x7fa5740017d0 tx=0x7fa574007d30).stop 2021-02-10T11:41:21.517 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:21.501 7fa5860c8700 1 -- 172.21.2.112:0/4282991201 reap_dead start 2021-02-10T11:41:21.517 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:21.502 7fa587aab700 1 -- 172.21.2.112:0/4282991201 shutdown_connections 2021-02-10T11:41:21.517 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:21.502 7fa587aab700 1 -- 172.21.2.112:0/4282991201 >> 172.21.2.112:0/4282991201 conn(0x7fa5800ce410 msgr2=0x7fa5800d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:21.517 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:21.502 7fa587aab700 1 -- 172.21.2.112:0/4282991201 shutdown_connections 2021-02-10T11:41:21.518 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:21.502 7fa587aab700 1 -- 172.21.2.112:0/4282991201 wait complete. 2021-02-10T11:41:21.518 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:21.517 7fa587aab700 1 Processor -- start 2021-02-10T11:41:21.518 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:21.517 7fa587aab700 1 -- start start 2021-02-10T11:41:21.519 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:21.517 7fa587aab700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa5800badd0 0x7fa58009eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:21.520 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:21.517 7fa587aab700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa5800adf50 0x7fa58009f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:21.520 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:21.517 7fa587aab700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa5801a7470 0x7fa5801a7870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:21.520 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:21.517 7fa587aab700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa58009e5a0 con 0x7fa5800adf50 2021-02-10T11:41:21.520 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:21.517 7fa587aab700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa58009f6e0 con 0x7fa5801a7470 2021-02-10T11:41:21.520 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:21.517 7fa587aab700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5800bb220 con 0x7fa5800badd0 2021-02-10T11:41:21.520 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:21.518 7fa5850c6700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa5800adf50 0x7fa58009f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:21.520 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:21.518 7fa5850c6700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa5800adf50 0x7fa58009f0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:55376/0 (socket says 172.21.2.112:55376) 2021-02-10T11:41:21.520 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:21.518 7fa5850c6700 1 -- 172.21.2.112:0/216595344 learned_addr learned my addr 172.21.2.112:0/216595344 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:41:21.520 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:21.518 7fa5850c6700 1 -- 172.21.2.112:0/216595344 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa5800badd0 msgr2=0x7fa58009eaa0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:41:21.520 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:21.518 7fa5850c6700 1 --2- 172.21.2.112:0/216595344 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa5800badd0 0x7fa58009eaa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:21.520 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:21.518 7fa5850c6700 1 -- 172.21.2.112:0/216595344 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa5801a7470 msgr2=0x7fa5801a7870 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:41:21.520 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:21.518 7fa5850c6700 1 --2- 172.21.2.112:0/216595344 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa5801a7470 0x7fa5801a7870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:21.520 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:21.518 7fa5850c6700 1 -- 172.21.2.112:0/216595344 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5801a7e90 con 0x7fa5800adf50 2021-02-10T11:41:21.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:21.518 7fa5850c6700 1 --2- 172.21.2.112:0/216595344 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa5800adf50 0x7fa58009f0c0 secure :-1 s=READY pgs=2190 cs=0 l=1 rev1=1 rx=0x7fa57400a590 tx=0x7fa57400a930).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:21.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:21.518 7fa5727fc700 1 -- 172.21.2.112:0/216595344 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa5740009b0 con 0x7fa5800adf50 2021-02-10T11:41:21.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:21.518 7fa5727fc700 1 -- 172.21.2.112:0/216595344 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa57400ad20 con 0x7fa5800adf50 2021-02-10T11:41:21.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:21.518 7fa587aab700 1 -- 172.21.2.112:0/216595344 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5801a7e90 con 0x7fa5800adf50 2021-02-10T11:41:21.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:21.518 7fa5727fc700 1 -- 172.21.2.112:0/216595344 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa5740043c0 con 0x7fa5800adf50 2021-02-10T11:41:21.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:21.519 7fa587aab700 1 -- 172.21.2.112:0/216595344 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5801a7e90 con 0x7fa5800adf50 2021-02-10T11:41:21.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:21.519 7fa5727fc700 1 -- 172.21.2.112:0/216595344 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa574008000 con 0x7fa5800adf50 2021-02-10T11:41:21.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:21.519 7fa5727fc700 1 --2- 172.21.2.112:0/216595344 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa568018730 0x7fa56801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:21.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:21.519 7fa5727fc700 1 -- 172.21.2.112:0/216595344 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 4753+0+0 (secure 0 0 0) 0x7fa5740322f0 con 0x7fa5800adf50 2021-02-10T11:41:21.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:21.520 7fa5858c7700 1 --2- 172.21.2.112:0/216595344 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa568018730 0x7fa56801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:21.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:21.520 7fa5858c7700 1 --2- 172.21.2.112:0/216595344 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa568018730 0x7fa56801ac10 crc :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:21.546 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:21.549 7fa587aab700 1 -- 172.21.2.112:0/216595344 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa580082a20 con 0x7fa5800adf50 2021-02-10T11:41:21.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:21.551 7fa5727fc700 1 -- 172.21.2.112:0/216595344 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa574001510 con 0x7fa5800adf50 2021-02-10T11:41:21.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:21.679 7fa587aab700 1 -- 172.21.2.112:0/216595344 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"} v 0) v1 -- 0x7fa580082a20 con 0x7fa5800adf50 2021-02-10T11:41:22.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:22.093 7fa5727fc700 1 -- 172.21.2.112:0/216595344 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"}]=0 pool 'cephfs_metadata' created v113) v1 ==== 133+0+0 (secure 0 0 0) 0x7fa574008000 con 0x7fa5800adf50 2021-02-10T11:41:22.093 INFO:teuthology.orchestra.run.gibba012.stderr:pool 'cephfs_metadata' created 2021-02-10T11:41:22.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:22.098 7fa587aab700 1 -- 172.21.2.112:0/216595344 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa568018730 msgr2=0x7fa56801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:22.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:22.098 7fa587aab700 1 --2- 172.21.2.112:0/216595344 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa568018730 0x7fa56801ac10 crc :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:22.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:22.098 7fa587aab700 1 -- 172.21.2.112:0/216595344 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa5800adf50 msgr2=0x7fa58009f0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:22.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:22.098 7fa587aab700 1 --2- 172.21.2.112:0/216595344 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa5800adf50 0x7fa58009f0c0 secure :-1 s=READY pgs=2190 cs=0 l=1 rev1=1 rx=0x7fa57400a590 tx=0x7fa57400a930).stop 2021-02-10T11:41:22.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:22.098 7fa587aab700 1 -- 172.21.2.112:0/216595344 shutdown_connections 2021-02-10T11:41:22.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:22.098 7fa587aab700 1 --2- 172.21.2.112:0/216595344 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa568018730 0x7fa56801ac10 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:22.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:22.098 7fa587aab700 1 --2- 172.21.2.112:0/216595344 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa5801a7470 0x7fa5801a7870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:22.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:22.098 7fa587aab700 1 --2- 172.21.2.112:0/216595344 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa5800adf50 0x7fa58009f0c0 unknown :-1 s=CLOSED pgs=2190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:22.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:22.098 7fa587aab700 1 --2- 172.21.2.112:0/216595344 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa5800badd0 0x7fa58009eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:22.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:22.098 7fa587aab700 1 -- 172.21.2.112:0/216595344 >> 172.21.2.112:0/216595344 conn(0x7fa5800bee90 msgr2=0x7fa5800bb880 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:22.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:22.098 7fa587aab700 1 -- 172.21.2.112:0/216595344 shutdown_connections 2021-02-10T11:41:22.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:22.098 7fa587aab700 1 -- 172.21.2.112:0/216595344 wait complete. 2021-02-10T11:41:22.104 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_data 8 2021-02-10T11:41:22.537 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:22.520 7f4bcebe0700 1 Processor -- start 2021-02-10T11:41:22.537 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:22.520 7f4bcebe0700 1 -- start start 2021-02-10T11:41:22.537 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:22.521 7f4bcebe0700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4bc80d1870 0x7f4bc81b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:22.537 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:22.521 7f4bcebe0700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f4bc80cff70 0x7f4bc8109130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:22.538 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:22.521 7f4bcebe0700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bc8109a60 con 0x7f4bc81b0ae0 2021-02-10T11:41:22.538 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:22.521 7f4bcebe0700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bc8109750 con 0x7f4bc80cff70 2021-02-10T11:41:22.538 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:22.521 7f4bcebe0700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bc8088070 con 0x7f4bc80d1870 2021-02-10T11:41:22.538 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:22.521 7f4bc7fff700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f4bc81b0ae0 0x7f4bc8108940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:54844/0 (socket says 172.21.2.112:54844) 2021-02-10T11:41:22.538 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:22.521 7f4bc7fff700 1 -- 172.21.2.112:0/3300479538 learned_addr learned my addr 172.21.2.112:0/3300479538 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:41:22.538 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:22.521 7f4bc77fe700 1 -- 172.21.2.112:0/3300479538 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3735632025 0 0) 0x7f4bb8001060 con 0x7f4bc81b0ae0 2021-02-10T11:41:22.538 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:22.521 7f4bc77fe700 1 -- 172.21.2.112:0/3300479538 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4bb4002fe0 con 0x7f4bc81b0ae0 2021-02-10T11:41:22.538 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:22.521 7f4bc77fe700 1 -- 172.21.2.112:0/3300479538 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2981403801 0 0) 0x7f4bb8001460 con 0x7f4bc81b0ae0 2021-02-10T11:41:22.538 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:22.521 7f4bcc9fc700 1 --2- 172.21.2.112:0/3300479538 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4bc80d1870 0x7f4bc81b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:22.538 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:22.521 7f4bc77fe700 1 -- 172.21.2.112:0/3300479538 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4bc80d1870 msgr2=0x7f4bc81b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:22.538 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:22.521 7f4bcd1fd700 1 --2- 172.21.2.112:0/3300479538 >> v2:172.21.2.120:3300/0 conn(0x7f4bc80cff70 0x7f4bc8109130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:22.538 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:22.521 7f4bc77fe700 1 --2- 172.21.2.112:0/3300479538 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4bc80d1870 0x7f4bc81b04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:22.539 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:22.522 7f4bc77fe700 1 -- 172.21.2.112:0/3300479538 >> v2:172.21.2.120:3300/0 conn(0x7f4bc80cff70 msgr2=0x7f4bc8109130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:22.539 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:22.522 7f4bc77fe700 1 --2- 172.21.2.112:0/3300479538 >> v2:172.21.2.120:3300/0 conn(0x7f4bc80cff70 0x7f4bc8109130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:22.539 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:22.522 7f4bc77fe700 1 -- 172.21.2.112:0/3300479538 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bc8088260 con 0x7f4bc81b0ae0 2021-02-10T11:41:22.539 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:22.522 7f4bc77fe700 1 -- 172.21.2.112:0/3300479538 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f4bb8001980 con 0x7f4bc81b0ae0 2021-02-10T11:41:22.539 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:22.522 7f4bc77fe700 1 -- 172.21.2.112:0/3300479538 >> v1:172.21.2.112:6789/0 conn(0x7f4bc81b0ae0 legacy=0x7f4bc8108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:22.539 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:22.522 7f4bc77fe700 1 --2- 172.21.2.112:0/3300479538 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4bb4003b90 0x7f4bb40060b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:22.539 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:22.522 7f4bc77fe700 1 --2- 172.21.2.112:0/3300479538 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4bb4006950 0x7f4bb4008e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:22.539 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:22.522 7f4bc77fe700 1 --2- 172.21.2.112:0/3300479538 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4bb4009770 0x7f4bb400bc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:22.539 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:22.522 7f4bc77fe700 1 -- 172.21.2.112:0/3300479538 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bb400c430 con 0x7f4bb4006950 2021-02-10T11:41:22.539 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:22.522 7f4bc77fe700 1 -- 172.21.2.112:0/3300479538 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bb400c650 con 0x7f4bb4003b90 2021-02-10T11:41:22.539 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:22.522 7f4bc77fe700 1 -- 172.21.2.112:0/3300479538 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bb400c870 con 0x7f4bb4009770 2021-02-10T11:41:22.539 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:22.522 7f4bcd1fd700 1 --2- 172.21.2.112:0/3300479538 >> v2:172.21.2.120:3300/0 conn(0x7f4bc80cff70 0x7f4bc8109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:41:22.540 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:22.522 7f4bcc9fc700 1 --2- 172.21.2.112:0/3300479538 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4bc80d1870 0x7f4bc81b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:41:22.540 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:22.522 7f4bc7fff700 1 --2- 172.21.2.112:0/3300479538 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4bb4006950 0x7f4bb4008e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:22.540 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:22.522 7f4bc7fff700 1 -- 172.21.2.112:0/3300479538 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4bb4009770 msgr2=0x7f4bb400bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:22.540 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:22.522 7f4bc7fff700 1 --2- 172.21.2.112:0/3300479538 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4bb4009770 0x7f4bb400bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:22.540 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:22.522 7f4bc7fff700 1 -- 172.21.2.112:0/3300479538 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4bb4003b90 msgr2=0x7f4bb40060b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:22.540 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:22.522 7f4bc7fff700 1 --2- 172.21.2.112:0/3300479538 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4bb4003b90 0x7f4bb40060b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:22.540 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:22.522 7f4bc7fff700 1 -- 172.21.2.112:0/3300479538 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bb400cb10 con 0x7f4bb4006950 2021-02-10T11:41:22.540 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:22.522 7f4bc7fff700 1 --2- 172.21.2.112:0/3300479538 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4bb4006950 0x7f4bb4008e70 secure :-1 s=READY pgs=2192 cs=0 l=1 rev1=1 rx=0x7f4bb8002000 tx=0x7f4bb800a310).ready entity=mon.0 client_cookie=58ca83decc069d23 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:22.540 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:22.522 7f4bc77fe700 1 -- 172.21.2.112:0/3300479538 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4bb80070c0 con 0x7f4bb4006950 2021-02-10T11:41:22.540 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:22.522 7f4bc77fe700 1 -- 172.21.2.112:0/3300479538 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bb8007440 con 0x7f4bb4006950 2021-02-10T11:41:22.540 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:22.522 7f4bc77fe700 1 -- 172.21.2.112:0/3300479538 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4bb800ab90 con 0x7f4bb4006950 2021-02-10T11:41:22.541 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:22.522 7f4bcebe0700 1 -- 172.21.2.112:0/3300479538 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4bb4006950 msgr2=0x7f4bb4008e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:22.541 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:22.522 7f4bcebe0700 1 --2- 172.21.2.112:0/3300479538 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4bb4006950 0x7f4bb4008e70 secure :-1 s=READY pgs=2192 cs=0 l=1 rev1=1 rx=0x7f4bb8002000 tx=0x7f4bb800a310).stop 2021-02-10T11:41:22.541 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:22.522 7f4bcd1fd700 1 -- 172.21.2.112:0/3300479538 reap_dead start 2021-02-10T11:41:22.541 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:22.523 7f4bcebe0700 1 -- 172.21.2.112:0/3300479538 shutdown_connections 2021-02-10T11:41:22.541 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:22.523 7f4bcebe0700 1 -- 172.21.2.112:0/3300479538 >> 172.21.2.112:0/3300479538 conn(0x7f4bc80ce420 msgr2=0x7f4bc80d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:22.541 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:22.523 7f4bcebe0700 1 -- 172.21.2.112:0/3300479538 shutdown_connections 2021-02-10T11:41:22.541 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:22.523 7f4bcebe0700 1 -- 172.21.2.112:0/3300479538 wait complete. 2021-02-10T11:41:22.541 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:22.538 7f4bcebe0700 1 Processor -- start 2021-02-10T11:41:22.541 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:22.538 7f4bcebe0700 1 -- start start 2021-02-10T11:41:22.541 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:22.538 7f4bcebe0700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4bc80abd60 0x7f4bc80ac160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:22.541 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:22.538 7f4bcebe0700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4bc80ac780 0x7f4bc80acb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:22.541 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:22.538 7f4bcebe0700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4bc8083910 0x7f4bc8083d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:22.542 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:22.538 7f4bcebe0700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bc8087180 con 0x7f4bc80abd60 2021-02-10T11:41:22.542 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:22.538 7f4bcebe0700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bc818fcd0 con 0x7f4bc80ac780 2021-02-10T11:41:22.542 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:22.538 7f4bcebe0700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bc8084330 con 0x7f4bc8083910 2021-02-10T11:41:22.542 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:22.539 7f4bcc9fc700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4bc80abd60 0x7f4bc80ac160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:22.542 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:22.539 7f4bcc9fc700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4bc80abd60 0x7f4bc80ac160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:55396/0 (socket says 172.21.2.112:55396) 2021-02-10T11:41:22.542 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:22.539 7f4bcc9fc700 1 -- 172.21.2.112:0/2722966781 learned_addr learned my addr 172.21.2.112:0/2722966781 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:41:22.542 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:22.539 7f4bcc9fc700 1 -- 172.21.2.112:0/2722966781 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4bc8083910 msgr2=0x7f4bc8083d10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:41:22.542 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:22.539 7f4bcc9fc700 1 --2- 172.21.2.112:0/2722966781 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4bc8083910 0x7f4bc8083d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:22.542 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:22.539 7f4bcc9fc700 1 -- 172.21.2.112:0/2722966781 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4bc80ac780 msgr2=0x7f4bc80acb80 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:41:22.542 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:22.539 7f4bcc9fc700 1 --2- 172.21.2.112:0/2722966781 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4bc80ac780 0x7f4bc80acb80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:22.543 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:22.539 7f4bcc9fc700 1 -- 172.21.2.112:0/2722966781 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bc8084550 con 0x7f4bc80abd60 2021-02-10T11:41:22.543 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:22.539 7f4bcc9fc700 1 --2- 172.21.2.112:0/2722966781 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4bc80abd60 0x7f4bc80ac160 secure :-1 s=READY pgs=2193 cs=0 l=1 rev1=1 rx=0x7f4bb0006f30 tx=0x7f4bb000b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:22.543 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:22.539 7f4bc57fa700 1 -- 172.21.2.112:0/2722966781 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4bb0006240 con 0x7f4bc80abd60 2021-02-10T11:41:22.543 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:22.539 7f4bc57fa700 1 -- 172.21.2.112:0/2722966781 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bb0000940 con 0x7f4bc80abd60 2021-02-10T11:41:22.543 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:22.539 7f4bc57fa700 1 -- 172.21.2.112:0/2722966781 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4bb00020d0 con 0x7f4bc80abd60 2021-02-10T11:41:22.543 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:22.539 7f4bcebe0700 1 -- 172.21.2.112:0/2722966781 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bc8084550 con 0x7f4bc80abd60 2021-02-10T11:41:22.543 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:22.539 7f4bcebe0700 1 -- 172.21.2.112:0/2722966781 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bc8084550 con 0x7f4bc80abd60 2021-02-10T11:41:22.543 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:22.540 7f4bc57fa700 1 -- 172.21.2.112:0/2722966781 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4bb0006240 con 0x7f4bc80abd60 2021-02-10T11:41:22.543 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:22.540 7f4bc57fa700 1 --2- 172.21.2.112:0/2722966781 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4bac0186c0 0x7f4bac01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:22.543 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:22.540 7f4bc57fa700 1 -- 172.21.2.112:0/2722966781 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5107+0+0 (secure 0 0 0) 0x7f4bb00323e0 con 0x7f4bc80abd60 2021-02-10T11:41:22.544 DEBUG:teuthology.orchestra.run.gibba012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:41:22.545 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:22.541 7f4bc7fff700 1 --2- 172.21.2.112:0/2722966781 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4bac0186c0 0x7f4bac01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:22.545 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:22.542 7f4bc7fff700 1 --2- 172.21.2.112:0/2722966781 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4bac0186c0 0x7f4bac01abc0 crc :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:22.547 DEBUG:teuthology.orchestra.run.gibba020:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:41:22.567 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:22.570 7f4bcebe0700 1 -- 172.21.2.112:0/2722966781 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bc8082980 con 0x7f4bc80abd60 2021-02-10T11:41:22.570 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:22.573 7f4bc57fa700 1 -- 172.21.2.112:0/2722966781 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4bb002cdd0 con 0x7f4bc80abd60 2021-02-10T11:41:22.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:22.703 7f4bcebe0700 1 -- 172.21.2.112:0/2722966781 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"} v 0) v1 -- 0x7f4bc8109a60 con 0x7f4bc80abd60 2021-02-10T11:41:23.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:23.100 7f4bc57fa700 1 -- 172.21.2.112:0/2722966781 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"}]=0 pool 'cephfs_data' created v115) v1 ==== 125+0+0 (secure 0 0 0) 0x7f4bb002cdd0 con 0x7f4bc80abd60 2021-02-10T11:41:23.116 INFO:teuthology.orchestra.run.gibba012.stderr:pool 'cephfs_data' created 2021-02-10T11:41:23.118 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:23.120 7f4bcebe0700 1 -- 172.21.2.112:0/2722966781 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4bac0186c0 msgr2=0x7f4bac01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:23.118 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:23.120 7f4bcebe0700 1 --2- 172.21.2.112:0/2722966781 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4bac0186c0 0x7f4bac01abc0 crc :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:23.118 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:23.121 7f4bcebe0700 1 -- 172.21.2.112:0/2722966781 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4bc80abd60 msgr2=0x7f4bc80ac160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:23.118 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:23.121 7f4bcebe0700 1 --2- 172.21.2.112:0/2722966781 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4bc80abd60 0x7f4bc80ac160 secure :-1 s=READY pgs=2193 cs=0 l=1 rev1=1 rx=0x7f4bb0006f30 tx=0x7f4bb000b990).stop 2021-02-10T11:41:23.119 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:23.121 7f4bcebe0700 1 -- 172.21.2.112:0/2722966781 shutdown_connections 2021-02-10T11:41:23.119 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:23.121 7f4bcebe0700 1 --2- 172.21.2.112:0/2722966781 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4bac0186c0 0x7f4bac01abc0 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:23.119 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:23.121 7f4bcebe0700 1 --2- 172.21.2.112:0/2722966781 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4bc80ac780 0x7f4bc80acb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:23.119 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:23.121 7f4bcebe0700 1 --2- 172.21.2.112:0/2722966781 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4bc8083910 0x7f4bc8083d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:23.119 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:23.121 7f4bcebe0700 1 --2- 172.21.2.112:0/2722966781 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4bc80abd60 0x7f4bc80ac160 unknown :-1 s=CLOSED pgs=2193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:23.119 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:23.121 7f4bcebe0700 1 -- 172.21.2.112:0/2722966781 >> 172.21.2.112:0/2722966781 conn(0x7f4bc80a2b30 msgr2=0x7f4bc809ce60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:23.119 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:23.121 7f4bcebe0700 1 -- 172.21.2.112:0/2722966781 shutdown_connections 2021-02-10T11:41:23.119 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:23.121 7f4bcebe0700 1 -- 172.21.2.112:0/2722966781 wait complete. 2021-02-10T11:41:23.126 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2021-02-10T11:41:23.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:23.544 7f132b512700 1 Processor -- start 2021-02-10T11:41:23.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:23.545 7f132b512700 1 -- start start 2021-02-10T11:41:23.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:23.545 7f132b512700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f13240d1c80 0x7f13240cef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:23.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:23.545 7f132b512700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f13240cf590 0x7f13240cf990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:23.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:23.545 7f132b512700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13240fe1d0 con 0x7f13240d1880 2021-02-10T11:41:23.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:23.545 7f132b512700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13241b0cc0 con 0x7f13240cf590 2021-02-10T11:41:23.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:23.545 7f132b512700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1324109a70 con 0x7f13240d1c80 2021-02-10T11:41:23.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:23.545 7f1329b2f700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f13240cf590 0x7f13240cf990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:23.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:23.545 7f1329b2f700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f13240cf590 0x7f13240cf990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:55408/0 (socket says 172.21.2.112:55408) 2021-02-10T11:41:23.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:23.545 7f1329b2f700 1 -- 172.21.2.112:0/527496026 learned_addr learned my addr 172.21.2.112:0/527496026 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:41:23.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:23.546 7f1328b2d700 1 --2- 172.21.2.112:0/527496026 >> v2:172.21.2.120:3300/0 conn(0x7f13240d1c80 0x7f13240cef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:23.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:23.546 7f1329b2f700 1 -- 172.21.2.112:0/527496026 >> v2:172.21.2.120:3300/0 conn(0x7f13240d1c80 msgr2=0x7f13240cef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:23.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:23.546 7f1329b2f700 1 --2- 172.21.2.112:0/527496026 >> v2:172.21.2.120:3300/0 conn(0x7f13240d1c80 0x7f13240cef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:23.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:23.546 7f1329b2f700 1 -- 172.21.2.112:0/527496026 >> v1:172.21.2.112:6789/0 conn(0x7f13240d1880 legacy=0x7f13241b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:23.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:23.546 7f1329b2f700 1 -- 172.21.2.112:0/527496026 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1324088070 con 0x7f13240cf590 2021-02-10T11:41:23.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:23.546 7f1329b2f700 1 --2- 172.21.2.112:0/527496026 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f13240cf590 0x7f13240cf990 secure :-1 s=READY pgs=2195 cs=0 l=1 rev1=1 rx=0x7f131c006f00 tx=0x7f131c009f10).ready entity=mon.0 client_cookie=e85df47c3f47e054 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:23.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:23.546 7f131bfff700 1 -- 172.21.2.112:0/527496026 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f131c00b3b0 con 0x7f13240cf590 2021-02-10T11:41:23.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:23.546 7f131bfff700 1 -- 172.21.2.112:0/527496026 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f131c00b730 con 0x7f13240cf590 2021-02-10T11:41:23.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:23.546 7f131bfff700 1 -- 172.21.2.112:0/527496026 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f131c00bc10 con 0x7f13240cf590 2021-02-10T11:41:23.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:23.546 7f132b512700 1 -- 172.21.2.112:0/527496026 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f13240cf590 msgr2=0x7f13240cf990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:23.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:23.546 7f132b512700 1 --2- 172.21.2.112:0/527496026 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f13240cf590 0x7f13240cf990 secure :-1 s=READY pgs=2195 cs=0 l=1 rev1=1 rx=0x7f131c006f00 tx=0x7f131c009f10).stop 2021-02-10T11:41:23.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:23.547 7f132b512700 1 -- 172.21.2.112:0/527496026 shutdown_connections 2021-02-10T11:41:23.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:23.547 7f132b512700 1 --2- 172.21.2.112:0/527496026 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f13240cf590 0x7f13240cf990 unknown :-1 s=CLOSED pgs=2195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:23.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:23.547 7f132b512700 1 --2- 172.21.2.112:0/527496026 >> v2:172.21.2.120:3300/0 conn(0x7f13240d1c80 0x7f13240cef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:23.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:23.547 7f132b512700 1 -- 172.21.2.112:0/527496026 >> 172.21.2.112:0/527496026 conn(0x7f13240ce430 msgr2=0x7f13240d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:23.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:23.547 7f132b512700 1 -- 172.21.2.112:0/527496026 shutdown_connections 2021-02-10T11:41:23.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:23.547 7f132b512700 1 -- 172.21.2.112:0/527496026 wait complete. 2021-02-10T11:41:23.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:23.562 7f132b512700 1 Processor -- start 2021-02-10T11:41:23.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:23.562 7f132b512700 1 -- start start 2021-02-10T11:41:23.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:23.562 7f132b512700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f13241b58d0 0x7f13240aa4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:23.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:23.562 7f132b512700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f13240aaaf0 0x7f13241b80a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:23.563 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:23.562 7f132b512700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f13241b86c0 0x7f132419de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:23.563 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:23.562 7f132b512700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13241b8ac0 con 0x7f13241b86c0 2021-02-10T11:41:23.563 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:23.562 7f132b512700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13241b8e20 con 0x7f13241b58d0 2021-02-10T11:41:23.563 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:23.562 7f132b512700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13241b5cd0 con 0x7f13240aaaf0 2021-02-10T11:41:23.563 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:23.563 7f1329b2f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f13241b86c0 0x7f132419de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:23.563 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:23.563 7f1329b2f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f13241b86c0 0x7f132419de70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:55414/0 (socket says 172.21.2.112:55414) 2021-02-10T11:41:23.563 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:23.563 7f1329b2f700 1 -- 172.21.2.112:0/3898904521 learned_addr learned my addr 172.21.2.112:0/3898904521 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:41:23.563 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:23.563 7f1329b2f700 1 -- 172.21.2.112:0/3898904521 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f13240aaaf0 msgr2=0x7f13241b80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:23.563 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:23.563 7f1329b2f700 1 --2- 172.21.2.112:0/3898904521 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f13240aaaf0 0x7f13241b80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:23.563 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:23.563 7f1329b2f700 1 -- 172.21.2.112:0/3898904521 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f13241b58d0 msgr2=0x7f13240aa4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:23.563 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:23.563 7f1329b2f700 1 --2- 172.21.2.112:0/3898904521 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f13241b58d0 0x7f13240aa4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:23.563 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:23.563 7f1329b2f700 1 -- 172.21.2.112:0/3898904521 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1324095710 con 0x7f13241b86c0 2021-02-10T11:41:23.563 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:23.563 7f1329b2f700 1 --2- 172.21.2.112:0/3898904521 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f13241b86c0 0x7f132419de70 secure :-1 s=READY pgs=2196 cs=0 l=1 rev1=1 rx=0x7f131c009390 tx=0x7f131c00bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:23.564 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:23.563 7f1319ffb700 1 -- 172.21.2.112:0/3898904521 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f131c006920 con 0x7f13241b86c0 2021-02-10T11:41:23.564 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:23.563 7f1319ffb700 1 -- 172.21.2.112:0/3898904521 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f131c003030 con 0x7f13241b86c0 2021-02-10T11:41:23.564 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:23.563 7f1319ffb700 1 -- 172.21.2.112:0/3898904521 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f131c001670 con 0x7f13241b86c0 2021-02-10T11:41:23.564 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:23.564 7f132b512700 1 -- 172.21.2.112:0/3898904521 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1324095710 con 0x7f13241b86c0 2021-02-10T11:41:23.564 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:23.564 7f132b512700 1 -- 172.21.2.112:0/3898904521 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f132419e780 con 0x7f13241b86c0 2021-02-10T11:41:23.564 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:23.564 7f1319ffb700 1 -- 172.21.2.112:0/3898904521 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f131c001270 con 0x7f13241b86c0 2021-02-10T11:41:23.564 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:23.564 7f1319ffb700 1 --2- 172.21.2.112:0/3898904521 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f13040186c0 0x7f130401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:23.564 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:23.564 7f1319ffb700 1 -- 172.21.2.112:0/3898904521 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5457+0+0 (secure 0 0 0) 0x7f131c032870 con 0x7f13241b86c0 2021-02-10T11:41:23.564 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:23.565 7f132932e700 1 --2- 172.21.2.112:0/3898904521 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f13040186c0 0x7f130401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:23.564 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:23.565 7f132932e700 1 --2- 172.21.2.112:0/3898904521 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f13040186c0 0x7f130401abc0 crc :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:23.592 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:23.595 7f132b512700 1 -- 172.21.2.112:0/3898904521 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1324082aa0 con 0x7f13241b86c0 2021-02-10T11:41:23.595 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:23.598 7f1319ffb700 1 -- 172.21.2.112:0/3898904521 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f131c009450 con 0x7f13241b86c0 2021-02-10T11:41:23.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:23.730 7f132b512700 1 -- 172.21.2.112:0/3898904521 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"} v 0) v1 -- 0x7f1324082aa0 con 0x7f13241b86c0 2021-02-10T11:41:23.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:23.733 7f1319ffb700 1 -- 172.21.2.112:0/3898904521 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"}]=0 new fs with metadata pool 24 and data pool 25 v94) v1 ==== 189+0+0 (secure 0 0 0) 0x7f131c001270 con 0x7f13241b86c0 2021-02-10T11:41:23.736 INFO:teuthology.orchestra.run.gibba012.stderr:new fs with metadata pool 24 and data pool 25 2021-02-10T11:41:23.736 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:23.739 7f132b512700 1 -- 172.21.2.112:0/3898904521 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f13040186c0 msgr2=0x7f130401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:23.737 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:23.739 7f132b512700 1 --2- 172.21.2.112:0/3898904521 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f13040186c0 0x7f130401abc0 crc :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:23.737 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:23.739 7f132b512700 1 -- 172.21.2.112:0/3898904521 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f13241b86c0 msgr2=0x7f132419de70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:23.737 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:23.739 7f132b512700 1 --2- 172.21.2.112:0/3898904521 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f13241b86c0 0x7f132419de70 secure :-1 s=READY pgs=2196 cs=0 l=1 rev1=1 rx=0x7f131c009390 tx=0x7f131c00bd40).stop 2021-02-10T11:41:23.737 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:23.740 7f132b512700 1 -- 172.21.2.112:0/3898904521 shutdown_connections 2021-02-10T11:41:23.737 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:23.740 7f132b512700 1 --2- 172.21.2.112:0/3898904521 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f13040186c0 0x7f130401abc0 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:23.737 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:23.740 7f132b512700 1 --2- 172.21.2.112:0/3898904521 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f13241b86c0 0x7f132419de70 unknown :-1 s=CLOSED pgs=2196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:23.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:23.740 7f132b512700 1 --2- 172.21.2.112:0/3898904521 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f13240aaaf0 0x7f13241b80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:23.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:23.740 7f132b512700 1 --2- 172.21.2.112:0/3898904521 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f13241b58d0 0x7f13240aa4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:23.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:23.740 7f132b512700 1 -- 172.21.2.112:0/3898904521 >> 172.21.2.112:0/3898904521 conn(0x7f132419f940 msgr2=0x7f132418c290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:23.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:23.740 7f132b512700 1 -- 172.21.2.112:0/3898904521 shutdown_connections 2021-02-10T11:41:23.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:23.740 7f132b512700 1 -- 172.21.2.112:0/3898904521 wait complete. 2021-02-10T11:41:23.745 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:41:24.173 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.159 7fee46609700 1 Processor -- start 2021-02-10T11:41:24.173 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.159 7fee46609700 1 -- start start 2021-02-10T11:41:24.173 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.160 7fee46609700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fee400d1870 0x7fee401b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:24.173 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.160 7fee46609700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fee401b0a10 0x7fee400cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:24.173 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.160 7fee46609700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fee400d1c70 0x7fee400cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:24.173 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.160 7fee46609700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee400fe200 con 0x7fee400d1c70 2021-02-10T11:41:24.173 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.160 7fee46609700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee400e5f10 con 0x7fee400d1870 2021-02-10T11:41:24.173 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.160 7fee46609700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee400f0b50 con 0x7fee401b0a10 2021-02-10T11:41:24.173 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.160 7fee44c26700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fee400d1c70 0x7fee400cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:24.174 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.160 7fee44c26700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fee400d1c70 0x7fee400cf950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:55418/0 (socket says 172.21.2.112:55418) 2021-02-10T11:41:24.174 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.160 7fee44c26700 1 -- 172.21.2.112:0/2427679739 learned_addr learned my addr 172.21.2.112:0/2427679739 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:41:24.174 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.160 7fee3ffff700 1 --2- 172.21.2.112:0/2427679739 >> v2:172.21.2.120:3300/0 conn(0x7fee400d1870 0x7fee401b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:24.174 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.160 7fee3f7fe700 1 --2- 172.21.2.112:0/2427679739 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fee401b0a10 0x7fee400cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:24.174 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.160 7fee44c26700 1 -- 172.21.2.112:0/2427679739 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fee401b0a10 msgr2=0x7fee400cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:24.174 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.160 7fee44c26700 1 --2- 172.21.2.112:0/2427679739 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fee401b0a10 0x7fee400cf330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:24.174 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.160 7fee44c26700 1 -- 172.21.2.112:0/2427679739 >> v2:172.21.2.120:3300/0 conn(0x7fee400d1870 msgr2=0x7fee401b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:24.174 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.160 7fee44c26700 1 --2- 172.21.2.112:0/2427679739 >> v2:172.21.2.120:3300/0 conn(0x7fee400d1870 0x7fee401b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:24.174 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.160 7fee44c26700 1 -- 172.21.2.112:0/2427679739 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee400cff70 con 0x7fee400d1c70 2021-02-10T11:41:24.174 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.160 7fee44c26700 1 --2- 172.21.2.112:0/2427679739 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fee400d1c70 0x7fee400cf950 secure :-1 s=READY pgs=2197 cs=0 l=1 rev1=1 rx=0x7fee38006f00 tx=0x7fee38009f10).ready entity=mon.0 client_cookie=179e730f68ae8adc server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:24.174 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.161 7fee3effd700 1 -- 172.21.2.112:0/2427679739 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fee3800b3b0 con 0x7fee400d1c70 2021-02-10T11:41:24.174 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.161 7fee3effd700 1 -- 172.21.2.112:0/2427679739 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee3800b730 con 0x7fee400d1c70 2021-02-10T11:41:24.174 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.161 7fee3effd700 1 -- 172.21.2.112:0/2427679739 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fee3800bb90 con 0x7fee400d1c70 2021-02-10T11:41:24.175 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.161 7fee46609700 1 -- 172.21.2.112:0/2427679739 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fee400d1c70 msgr2=0x7fee400cf950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:24.175 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.161 7fee46609700 1 --2- 172.21.2.112:0/2427679739 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fee400d1c70 0x7fee400cf950 secure :-1 s=READY pgs=2197 cs=0 l=1 rev1=1 rx=0x7fee38006f00 tx=0x7fee38009f10).stop 2021-02-10T11:41:24.175 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.161 7fee46609700 1 -- 172.21.2.112:0/2427679739 shutdown_connections 2021-02-10T11:41:24.175 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.161 7fee46609700 1 --2- 172.21.2.112:0/2427679739 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fee400d1c70 0x7fee400cf950 unknown :-1 s=CLOSED pgs=2197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:24.175 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.161 7fee46609700 1 --2- 172.21.2.112:0/2427679739 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fee401b0a10 0x7fee400cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:24.175 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.161 7fee46609700 1 --2- 172.21.2.112:0/2427679739 >> v2:172.21.2.120:3300/0 conn(0x7fee400d1870 0x7fee401b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:24.175 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.161 7fee46609700 1 -- 172.21.2.112:0/2427679739 >> 172.21.2.112:0/2427679739 conn(0x7fee400ce450 msgr2=0x7fee400d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:24.175 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.161 7fee46609700 1 -- 172.21.2.112:0/2427679739 shutdown_connections 2021-02-10T11:41:24.175 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.161 7fee46609700 1 -- 172.21.2.112:0/2427679739 wait complete. 2021-02-10T11:41:24.175 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.176 7fee46609700 1 Processor -- start 2021-02-10T11:41:24.177 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.176 7fee46609700 1 -- start start 2021-02-10T11:41:24.177 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.177 7fee46609700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fee40195650 0x7fee400b5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:24.177 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.177 7fee46609700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fee400b5b80 0x7fee401a2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:24.177 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.177 7fee46609700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fee400acfa0 0x7fee401a3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:24.177 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.177 7fee46609700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee400b6010 con 0x7fee400b5b80 2021-02-10T11:41:24.177 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.177 7fee46609700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee401a3a60 con 0x7fee400acfa0 2021-02-10T11:41:24.177 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.177 7fee46609700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee401a3c40 con 0x7fee40195650 2021-02-10T11:41:24.177 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.177 7fee3f7fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fee400b5b80 0x7fee401a2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:24.177 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.177 7fee3f7fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fee400b5b80 0x7fee401a2e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:55422/0 (socket says 172.21.2.112:55422) 2021-02-10T11:41:24.178 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.177 7fee3f7fe700 1 -- 172.21.2.112:0/1707195100 learned_addr learned my addr 172.21.2.112:0/1707195100 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:41:24.178 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.177 7fee3f7fe700 1 -- 172.21.2.112:0/1707195100 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fee40195650 msgr2=0x7fee400b5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:24.178 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.177 7fee3f7fe700 1 --2- 172.21.2.112:0/1707195100 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fee40195650 0x7fee400b5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:24.178 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.177 7fee3f7fe700 1 -- 172.21.2.112:0/1707195100 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fee400acfa0 msgr2=0x7fee401a3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:24.178 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.177 7fee3f7fe700 1 --2- 172.21.2.112:0/1707195100 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fee400acfa0 0x7fee401a3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:24.178 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.177 7fee3f7fe700 1 -- 172.21.2.112:0/1707195100 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee400bcae0 con 0x7fee400b5b80 2021-02-10T11:41:24.178 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.177 7fee3f7fe700 1 --2- 172.21.2.112:0/1707195100 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fee400b5b80 0x7fee401a2e20 secure :-1 s=READY pgs=2198 cs=0 l=1 rev1=1 rx=0x7fee2c006f00 tx=0x7fee2c00b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:24.178 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.178 7fee3cff9700 1 -- 172.21.2.112:0/1707195100 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fee2c00bd30 con 0x7fee400b5b80 2021-02-10T11:41:24.178 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.178 7fee3cff9700 1 -- 172.21.2.112:0/1707195100 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee2c002030 con 0x7fee400b5b80 2021-02-10T11:41:24.178 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.178 7fee3cff9700 1 -- 172.21.2.112:0/1707195100 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fee2c002590 con 0x7fee400b5b80 2021-02-10T11:41:24.178 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.178 7fee46609700 1 -- 172.21.2.112:0/1707195100 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee400bcae0 con 0x7fee400b5b80 2021-02-10T11:41:24.178 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.178 7fee46609700 1 -- 172.21.2.112:0/1707195100 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee400bd070 con 0x7fee400b5b80 2021-02-10T11:41:24.178 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.178 7fee3cff9700 1 -- 172.21.2.112:0/1707195100 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fee2c000c90 con 0x7fee400b5b80 2021-02-10T11:41:24.179 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.178 7fee3cff9700 1 --2- 172.21.2.112:0/1707195100 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fee1c0186c0 0x7fee1c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:24.179 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.178 7fee3cff9700 1 -- 172.21.2.112:0/1707195100 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fee2c032470 con 0x7fee400b5b80 2021-02-10T11:41:24.179 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.179 7fee3ffff700 1 --2- 172.21.2.112:0/1707195100 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fee1c0186c0 0x7fee1c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:24.179 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.179 7fee3ffff700 1 --2- 172.21.2.112:0/1707195100 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fee1c0186c0 0x7fee1c01abc0 crc :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:24.207 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.209 7fee46609700 1 -- 172.21.2.112:0/1707195100 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee401a3c40 con 0x7fee400b5b80 2021-02-10T11:41:24.209 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.212 7fee3cff9700 1 -- 172.21.2.112:0/1707195100 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fee2c00bd30 con 0x7fee400b5b80 2021-02-10T11:41:24.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.338 7fee46609700 1 -- 172.21.2.112:0/1707195100 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fee401a3c40 con 0x7fee400b5b80 2021-02-10T11:41:24.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.338 7fee3cff9700 1 -- 172.21.2.112:0/1707195100 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v116) v1 ==== 74+0+14020 (secure 0 0 0) 0x7fee2c000c90 con 0x7fee400b5b80 2021-02-10T11:41:24.336 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:41:24.336 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":116,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:41:23.731486","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:41:21.697492","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"116","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":25,"pool_name":"cephfs_data","create_time":"2021-02-10 11:41:22.721757","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"116","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:41:24.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.340 7fee46609700 1 -- 172.21.2.112:0/1707195100 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fee1c0186c0 msgr2=0x7fee1c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:24.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.340 7fee46609700 1 --2- 172.21.2.112:0/1707195100 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fee1c0186c0 0x7fee1c01abc0 crc :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:24.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.340 7fee46609700 1 -- 172.21.2.112:0/1707195100 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fee400b5b80 msgr2=0x7fee401a2e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:24.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.340 7fee46609700 1 --2- 172.21.2.112:0/1707195100 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fee400b5b80 0x7fee401a2e20 secure :-1 s=READY pgs=2198 cs=0 l=1 rev1=1 rx=0x7fee2c006f00 tx=0x7fee2c00b940).stop 2021-02-10T11:41:24.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.340 7fee46609700 1 -- 172.21.2.112:0/1707195100 shutdown_connections 2021-02-10T11:41:24.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.340 7fee46609700 1 --2- 172.21.2.112:0/1707195100 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fee1c0186c0 0x7fee1c01abc0 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:24.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.340 7fee46609700 1 --2- 172.21.2.112:0/1707195100 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fee400acfa0 0x7fee401a3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:24.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.340 7fee46609700 1 --2- 172.21.2.112:0/1707195100 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fee400b5b80 0x7fee401a2e20 unknown :-1 s=CLOSED pgs=2198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:24.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.340 7fee46609700 1 --2- 172.21.2.112:0/1707195100 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fee40195650 0x7fee400b5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:24.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.340 7fee46609700 1 -- 172.21.2.112:0/1707195100 >> 172.21.2.112:0/1707195100 conn(0x7fee400b3040 msgr2=0x7fee400c3250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:24.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.340 7fee46609700 1 -- 172.21.2.112:0/1707195100 shutdown_connections 2021-02-10T11:41:24.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.340 7fee46609700 1 -- 172.21.2.112:0/1707195100 wait complete. 2021-02-10T11:41:24.346 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:41:24.777 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.763 7fef0699e700 1 Processor -- start 2021-02-10T11:41:24.777 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.763 7fef0699e700 1 -- start start 2021-02-10T11:41:24.777 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.764 7fef0699e700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fef001b0470 0x7fef001b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:24.777 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.764 7fef0699e700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fef000ef610 0x7fef000efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:24.777 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.764 7fef0699e700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef000f0b50 con 0x7fef000f0030 2021-02-10T11:41:24.777 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.764 7fef0699e700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef000e52e0 con 0x7fef000ef610 2021-02-10T11:41:24.777 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.764 7fef0699e700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef000e5490 con 0x7fef001b0470 2021-02-10T11:41:24.777 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.764 7feeff7fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fef000ef610 0x7fef000efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:24.777 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.764 7feeff7fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fef000ef610 0x7fef000efa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:55430/0 (socket says 172.21.2.112:55430) 2021-02-10T11:41:24.777 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.764 7feeff7fe700 1 -- 172.21.2.112:0/1327669436 learned_addr learned my addr 172.21.2.112:0/1327669436 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:41:24.777 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.764 7feefffff700 1 --2- 172.21.2.112:0/1327669436 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fef001b0470 0x7fef001b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:24.778 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.764 7feeff7fe700 1 -- 172.21.2.112:0/1327669436 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fef001b0470 msgr2=0x7fef001b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:24.778 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.764 7feeff7fe700 1 --2- 172.21.2.112:0/1327669436 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fef001b0470 0x7fef001b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:24.778 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.764 7feeff7fe700 1 -- 172.21.2.112:0/1327669436 >> v1:172.21.2.120:6789/0 conn(0x7fef000f0030 legacy=0x7fef000f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:24.778 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.764 7feeff7fe700 1 -- 172.21.2.112:0/1327669436 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef000f0430 con 0x7fef000ef610 2021-02-10T11:41:24.778 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.764 7feefffff700 1 --2- 172.21.2.112:0/1327669436 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fef001b0470 0x7fef001b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:41:24.778 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.764 7feeff7fe700 1 --2- 172.21.2.112:0/1327669436 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fef000ef610 0x7fef000efa10 secure :-1 s=READY pgs=2199 cs=0 l=1 rev1=1 rx=0x7feef4006700 tx=0x7feef4009b80).ready entity=mon.0 client_cookie=87631ae4596ac852 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:24.778 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.764 7feefeffd700 1 -- 172.21.2.112:0/1327669436 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feef400c0c0 con 0x7fef000ef610 2021-02-10T11:41:24.778 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.764 7feefeffd700 1 -- 172.21.2.112:0/1327669436 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feef400c340 con 0x7fef000ef610 2021-02-10T11:41:24.778 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.764 7feefeffd700 1 -- 172.21.2.112:0/1327669436 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feef400c7a0 con 0x7fef000ef610 2021-02-10T11:41:24.778 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.764 7fef0699e700 1 -- 172.21.2.112:0/1327669436 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fef000ef610 msgr2=0x7fef000efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:24.778 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.764 7fef0699e700 1 --2- 172.21.2.112:0/1327669436 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fef000ef610 0x7fef000efa10 secure :-1 s=READY pgs=2199 cs=0 l=1 rev1=1 rx=0x7feef4006700 tx=0x7feef4009b80).stop 2021-02-10T11:41:24.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.765 7fef0699e700 1 -- 172.21.2.112:0/1327669436 shutdown_connections 2021-02-10T11:41:24.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.765 7fef0699e700 1 --2- 172.21.2.112:0/1327669436 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fef000ef610 0x7fef000efa10 unknown :-1 s=CLOSED pgs=2199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:24.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.765 7fef0699e700 1 --2- 172.21.2.112:0/1327669436 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fef001b0470 0x7fef001b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:24.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.765 7fef0699e700 1 -- 172.21.2.112:0/1327669436 >> 172.21.2.112:0/1327669436 conn(0x7fef000ce3e0 msgr2=0x7fef000e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:24.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.765 7fef0699e700 1 -- 172.21.2.112:0/1327669436 shutdown_connections 2021-02-10T11:41:24.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.765 7fef0699e700 1 -- 172.21.2.112:0/1327669436 wait complete. 2021-02-10T11:41:24.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.780 7fef0699e700 1 Processor -- start 2021-02-10T11:41:24.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.780 7fef0699e700 1 -- start start 2021-02-10T11:41:24.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.780 7fef0699e700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fef001ac1a0 0x7fef0018e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:24.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.780 7fef0699e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fef001ab670 0x7fef0018ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:24.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.780 7fef0699e700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fef0019fee0 0x7fef001a02e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:24.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.780 7fef0699e700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef001abc20 con 0x7fef001ab670 2021-02-10T11:41:24.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.780 7fef0699e700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef001ac5a0 con 0x7fef0019fee0 2021-02-10T11:41:24.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.780 7fef0699e700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef001991c0 con 0x7fef001ac1a0 2021-02-10T11:41:24.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.780 7feeff7fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fef001ab670 0x7fef0018ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:24.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.780 7feeff7fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fef001ab670 0x7fef0018ed10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:55436/0 (socket says 172.21.2.112:55436) 2021-02-10T11:41:24.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.780 7feeff7fe700 1 -- 172.21.2.112:0/909773107 learned_addr learned my addr 172.21.2.112:0/909773107 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:41:24.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.780 7fef04fbb700 1 --2- 172.21.2.112:0/909773107 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fef0019fee0 0x7fef001a02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:24.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.780 7feefffff700 1 --2- 172.21.2.112:0/909773107 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fef001ac1a0 0x7fef0018e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:24.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.781 7feeff7fe700 1 -- 172.21.2.112:0/909773107 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fef001ac1a0 msgr2=0x7fef0018e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:24.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.781 7feeff7fe700 1 --2- 172.21.2.112:0/909773107 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fef001ac1a0 0x7fef0018e6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:24.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.781 7feeff7fe700 1 -- 172.21.2.112:0/909773107 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fef0019fee0 msgr2=0x7fef001a02e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:24.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.781 7feeff7fe700 1 --2- 172.21.2.112:0/909773107 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fef0019fee0 0x7fef001a02e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:24.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.781 7feeff7fe700 1 -- 172.21.2.112:0/909773107 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef00198d10 con 0x7fef001ab670 2021-02-10T11:41:24.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.781 7feeff7fe700 1 --2- 172.21.2.112:0/909773107 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fef001ab670 0x7fef0018ed10 secure :-1 s=READY pgs=2200 cs=0 l=1 rev1=1 rx=0x7feef4009720 tx=0x7feef400b8e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:24.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.781 7feefcff9700 1 -- 172.21.2.112:0/909773107 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feef40090c0 con 0x7fef001ab670 2021-02-10T11:41:24.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.781 7feefcff9700 1 -- 172.21.2.112:0/909773107 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feef4002030 con 0x7fef001ab670 2021-02-10T11:41:24.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.781 7feefcff9700 1 -- 172.21.2.112:0/909773107 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feef4002500 con 0x7fef001ab670 2021-02-10T11:41:24.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.781 7fef0699e700 1 -- 172.21.2.112:0/909773107 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef00198d10 con 0x7fef001ab670 2021-02-10T11:41:24.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.781 7fef0699e700 1 -- 172.21.2.112:0/909773107 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef00198d10 con 0x7fef001ab670 2021-02-10T11:41:24.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.782 7feefcff9700 1 -- 172.21.2.112:0/909773107 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7feef4000c90 con 0x7fef001ab670 2021-02-10T11:41:24.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.782 7feefcff9700 1 --2- 172.21.2.112:0/909773107 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7feee00186c0 0x7feee001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:24.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.782 7feefcff9700 1 -- 172.21.2.112:0/909773107 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5573+0+0 (secure 0 0 0) 0x7feef4032560 con 0x7fef001ab670 2021-02-10T11:41:24.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.782 7feefffff700 1 --2- 172.21.2.112:0/909773107 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7feee00186c0 0x7feee001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:24.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.782 7feefffff700 1 --2- 172.21.2.112:0/909773107 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7feee00186c0 0x7feee001abc0 crc :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:24.810 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.813 7fef0699e700 1 -- 172.21.2.112:0/909773107 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef00105a40 con 0x7fef001ab670 2021-02-10T11:41:24.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.816 7feefcff9700 1 -- 172.21.2.112:0/909773107 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7feef40063a0 con 0x7fef001ab670 2021-02-10T11:41:24.939 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.941 7fef0699e700 1 -- 172.21.2.112:0/909773107 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fef000f0b50 con 0x7fef001ab670 2021-02-10T11:41:24.940 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.942 7feefcff9700 1 -- 172.21.2.112:0/909773107 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v117) v1 ==== 74+0+14007 (secure 0 0 0) 0x7feef4000c90 con 0x7fef001ab670 2021-02-10T11:41:24.940 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:41:24.940 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":117,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:41:24.733802","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:41:21.697492","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"116","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":25,"pool_name":"cephfs_data","create_time":"2021-02-10 11:41:22.721757","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"117","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:41:24.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.944 7fef0699e700 1 -- 172.21.2.112:0/909773107 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7feee00186c0 msgr2=0x7feee001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:24.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.944 7fef0699e700 1 --2- 172.21.2.112:0/909773107 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7feee00186c0 0x7feee001abc0 crc :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:24.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.944 7fef0699e700 1 -- 172.21.2.112:0/909773107 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fef001ab670 msgr2=0x7fef0018ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:24.943 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.944 7fef0699e700 1 --2- 172.21.2.112:0/909773107 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fef001ab670 0x7fef0018ed10 secure :-1 s=READY pgs=2200 cs=0 l=1 rev1=1 rx=0x7feef4009720 tx=0x7feef400b8e0).stop 2021-02-10T11:41:24.943 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.944 7fef0699e700 1 -- 172.21.2.112:0/909773107 shutdown_connections 2021-02-10T11:41:24.943 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.944 7fef0699e700 1 --2- 172.21.2.112:0/909773107 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7feee00186c0 0x7feee001abc0 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:24.943 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.944 7fef0699e700 1 --2- 172.21.2.112:0/909773107 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fef0019fee0 0x7fef001a02e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:24.943 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.944 7fef0699e700 1 --2- 172.21.2.112:0/909773107 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fef001ab670 0x7fef0018ed10 unknown :-1 s=CLOSED pgs=2200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:24.944 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.944 7fef0699e700 1 --2- 172.21.2.112:0/909773107 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fef001ac1a0 0x7fef0018e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:24.944 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.944 7fef0699e700 1 -- 172.21.2.112:0/909773107 >> 172.21.2.112:0/909773107 conn(0x7fef001a37c0 msgr2=0x7fef001b9230 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:24.944 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.944 7fef0699e700 1 -- 172.21.2.112:0/909773107 shutdown_connections 2021-02-10T11:41:24.944 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:24.944 7fef0699e700 1 -- 172.21.2.112:0/909773107 wait complete. 2021-02-10T11:41:24.951 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2021-02-10T11:41:25.382 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:25.368 7f9fe1bc4700 1 Processor -- start 2021-02-10T11:41:25.382 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:25.368 7f9fe1bc4700 1 -- start start 2021-02-10T11:41:25.382 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:25.369 7f9fe1bc4700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f9fdc0d1c70 0x7f9fdc0cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:25.382 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:25.369 7f9fe1bc4700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f9fdc0cf580 0x7f9fdc0cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:25.382 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:25.369 7f9fe1bc4700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fdc0fe1c0 con 0x7f9fdc0d1870 2021-02-10T11:41:25.382 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:25.369 7f9fe1bc4700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fdc1b0cb0 con 0x7f9fdc0cf580 2021-02-10T11:41:25.382 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:25.369 7f9fe1bc4700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fdc109a60 con 0x7f9fdc0d1c70 2021-02-10T11:41:25.382 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:25.369 7f9fdbfff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f9fdc0cf580 0x7f9fdc0cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:25.383 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:25.369 7f9fdbfff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f9fdc0cf580 0x7f9fdc0cf980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:55448/0 (socket says 172.21.2.112:55448) 2021-02-10T11:41:25.383 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:25.369 7f9fdbfff700 1 -- 172.21.2.112:0/3892432577 learned_addr learned my addr 172.21.2.112:0/3892432577 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:41:25.383 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:25.369 7f9fdbfff700 1 -- 172.21.2.112:0/3892432577 >> v2:172.21.2.120:3300/0 conn(0x7f9fdc0d1c70 msgr2=0x7f9fdc0cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:25.383 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:25.369 7f9fdbfff700 1 --2- 172.21.2.112:0/3892432577 >> v2:172.21.2.120:3300/0 conn(0x7f9fdc0d1c70 0x7f9fdc0cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:25.383 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:25.369 7f9fdbfff700 1 -- 172.21.2.112:0/3892432577 >> v1:172.21.2.112:6789/0 conn(0x7f9fdc0d1870 legacy=0x7f9fdc1b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:25.383 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:25.369 7f9fdbfff700 1 -- 172.21.2.112:0/3892432577 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fdc088070 con 0x7f9fdc0cf580 2021-02-10T11:41:25.383 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:25.369 7f9fdbfff700 1 --2- 172.21.2.112:0/3892432577 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9fdc0cf580 0x7f9fdc0cf980 secure :-1 s=READY pgs=2202 cs=0 l=1 rev1=1 rx=0x7f9fd4006f00 tx=0x7f9fd4009f10).ready entity=mon.0 client_cookie=79250cd37a47b425 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:25.383 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:25.369 7f9fda7fc700 1 -- 172.21.2.112:0/3892432577 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9fd400b3b0 con 0x7f9fdc0cf580 2021-02-10T11:41:25.383 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:25.369 7f9fda7fc700 1 -- 172.21.2.112:0/3892432577 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fd400b630 con 0x7f9fdc0cf580 2021-02-10T11:41:25.383 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:25.369 7f9fda7fc700 1 -- 172.21.2.112:0/3892432577 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9fd400bb10 con 0x7f9fdc0cf580 2021-02-10T11:41:25.383 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:25.370 7f9fe1bc4700 1 -- 172.21.2.112:0/3892432577 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9fdc0cf580 msgr2=0x7f9fdc0cf980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:25.383 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:25.370 7f9fe1bc4700 1 --2- 172.21.2.112:0/3892432577 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9fdc0cf580 0x7f9fdc0cf980 secure :-1 s=READY pgs=2202 cs=0 l=1 rev1=1 rx=0x7f9fd4006f00 tx=0x7f9fd4009f10).stop 2021-02-10T11:41:25.383 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:25.370 7f9fe1bc4700 1 -- 172.21.2.112:0/3892432577 shutdown_connections 2021-02-10T11:41:25.384 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:25.370 7f9fe1bc4700 1 --2- 172.21.2.112:0/3892432577 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9fdc0cf580 0x7f9fdc0cf980 unknown :-1 s=CLOSED pgs=2202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:25.384 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:25.370 7f9fe1bc4700 1 --2- 172.21.2.112:0/3892432577 >> v2:172.21.2.120:3300/0 conn(0x7f9fdc0d1c70 0x7f9fdc0cef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:25.384 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:25.370 7f9fe1bc4700 1 -- 172.21.2.112:0/3892432577 >> 172.21.2.112:0/3892432577 conn(0x7f9fdc0ce420 msgr2=0x7f9fdc0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:25.384 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:25.370 7f9fe1bc4700 1 -- 172.21.2.112:0/3892432577 shutdown_connections 2021-02-10T11:41:25.384 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:25.370 7f9fe1bc4700 1 -- 172.21.2.112:0/3892432577 wait complete. 2021-02-10T11:41:25.384 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:25.385 7f9fe1bc4700 1 Processor -- start 2021-02-10T11:41:25.384 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:25.385 7f9fe1bc4700 1 -- start start 2021-02-10T11:41:25.385 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:25.385 7f9fe1bc4700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9fdc0c3820 0x7f9fdc084cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:25.385 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:25.385 7f9fe1bc4700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9fdc0a0290 0x7f9fdc0852d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:25.385 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:25.385 7f9fe1bc4700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9fdc086270 0x7f9fdc086670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:25.385 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:25.385 7f9fe1bc4700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fdc0c0700 con 0x7f9fdc0c3820 2021-02-10T11:41:25.385 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:25.385 7f9fe1bc4700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fdc0858f0 con 0x7f9fdc086270 2021-02-10T11:41:25.385 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:25.385 7f9fe1bc4700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fdc085b10 con 0x7f9fdc0a0290 2021-02-10T11:41:25.386 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:25.385 7f9fdb7fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9fdc0c3820 0x7f9fdc084cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:25.386 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:25.386 7f9fdb7fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9fdc0c3820 0x7f9fdc084cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:55452/0 (socket says 172.21.2.112:55452) 2021-02-10T11:41:25.386 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:25.386 7f9fdb7fe700 1 -- 172.21.2.112:0/2528213666 learned_addr learned my addr 172.21.2.112:0/2528213666 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:41:25.386 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:25.386 7f9fdb7fe700 1 -- 172.21.2.112:0/2528213666 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9fdc0a0290 msgr2=0x7f9fdc0852d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:25.386 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:25.386 7f9fdb7fe700 1 --2- 172.21.2.112:0/2528213666 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9fdc0a0290 0x7f9fdc0852d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:25.386 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:25.386 7f9fdb7fe700 1 -- 172.21.2.112:0/2528213666 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9fdc086270 msgr2=0x7f9fdc086670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:25.386 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:25.386 7f9fdb7fe700 1 --2- 172.21.2.112:0/2528213666 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9fdc086270 0x7f9fdc086670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:25.386 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:25.386 7f9fdb7fe700 1 -- 172.21.2.112:0/2528213666 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fdc086c90 con 0x7f9fdc0c3820 2021-02-10T11:41:25.386 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:25.386 7f9fdb7fe700 1 --2- 172.21.2.112:0/2528213666 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9fdc0c3820 0x7f9fdc084cb0 secure :-1 s=READY pgs=2203 cs=0 l=1 rev1=1 rx=0x7f9fcc006f00 tx=0x7f9fcc00b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:25.386 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:25.386 7f9fd3fff700 1 -- 172.21.2.112:0/2528213666 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9fcc000c60 con 0x7f9fdc0c3820 2021-02-10T11:41:25.386 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:25.386 7f9fd3fff700 1 -- 172.21.2.112:0/2528213666 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fcc006b90 con 0x7f9fdc0c3820 2021-02-10T11:41:25.386 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:25.386 7f9fe1bc4700 1 -- 172.21.2.112:0/2528213666 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fdc086c90 con 0x7f9fdc0c3820 2021-02-10T11:41:25.386 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:25.386 7f9fd3fff700 1 -- 172.21.2.112:0/2528213666 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9fcc001310 con 0x7f9fdc0c3820 2021-02-10T11:41:25.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:25.386 7f9fe1bc4700 1 -- 172.21.2.112:0/2528213666 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fdc1a2f00 con 0x7f9fdc0c3820 2021-02-10T11:41:25.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:25.387 7f9fd3fff700 1 -- 172.21.2.112:0/2528213666 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9fcc001100 con 0x7f9fdc0c3820 2021-02-10T11:41:25.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:25.387 7f9fd3fff700 1 --2- 172.21.2.112:0/2528213666 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9fc00186d0 0x7f9fc001abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:25.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:25.387 7f9fd3fff700 1 -- 172.21.2.112:0/2528213666 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f9fcc032660 con 0x7f9fdc0c3820 2021-02-10T11:41:25.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:25.387 7f9fdaffd700 1 --2- 172.21.2.112:0/2528213666 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9fc00186d0 0x7f9fc001abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:25.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:25.388 7f9fdaffd700 1 --2- 172.21.2.112:0/2528213666 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9fc00186d0 0x7f9fc001abd0 crc :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:25.415 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:25.418 7f9fe1bc4700 1 -- 172.21.2.112:0/2528213666 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fdc082a70 con 0x7f9fdc0c3820 2021-02-10T11:41:25.418 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:25.421 7f9fd3fff700 1 -- 172.21.2.112:0/2528213666 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9fcc000ed0 con 0x7f9fdc0c3820 2021-02-10T11:41:25.555 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:25.558 7f9fe1bc4700 1 -- 172.21.2.112:0/2528213666 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"} v 0) v1 -- 0x7f9fdc082a70 con 0x7f9fdc0c3820 2021-02-10T11:41:25.740 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:25.742 7f9fd3fff700 1 -- 172.21.2.112:0/2528213666 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"}]=0 v97) v1 ==== 118+0+0 (secure 0 0 0) 0x7f9fcc006b90 con 0x7f9fdc0c3820 2021-02-10T11:41:25.774 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:25.776 7f9fe1bc4700 1 -- 172.21.2.112:0/2528213666 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9fc00186d0 msgr2=0x7f9fc001abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:25.774 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:25.776 7f9fe1bc4700 1 --2- 172.21.2.112:0/2528213666 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9fc00186d0 0x7f9fc001abd0 crc :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:25.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:25.776 7f9fe1bc4700 1 -- 172.21.2.112:0/2528213666 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9fdc0c3820 msgr2=0x7f9fdc084cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:25.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:25.776 7f9fe1bc4700 1 --2- 172.21.2.112:0/2528213666 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9fdc0c3820 0x7f9fdc084cb0 secure :-1 s=READY pgs=2203 cs=0 l=1 rev1=1 rx=0x7f9fcc006f00 tx=0x7f9fcc00b940).stop 2021-02-10T11:41:25.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:25.777 7f9fe1bc4700 1 -- 172.21.2.112:0/2528213666 shutdown_connections 2021-02-10T11:41:25.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:25.777 7f9fe1bc4700 1 --2- 172.21.2.112:0/2528213666 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9fc00186d0 0x7f9fc001abd0 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:25.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:25.777 7f9fe1bc4700 1 --2- 172.21.2.112:0/2528213666 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9fdc086270 0x7f9fdc086670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:25.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:25.777 7f9fe1bc4700 1 --2- 172.21.2.112:0/2528213666 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9fdc0a0290 0x7f9fdc0852d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:25.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:25.777 7f9fe1bc4700 1 --2- 172.21.2.112:0/2528213666 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9fdc0c3820 0x7f9fdc084cb0 unknown :-1 s=CLOSED pgs=2203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:25.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:25.777 7f9fe1bc4700 1 -- 172.21.2.112:0/2528213666 >> 172.21.2.112:0/2528213666 conn(0x7f9fdc0872c0 msgr2=0x7f9fdc095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:25.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:25.777 7f9fe1bc4700 1 -- 172.21.2.112:0/2528213666 shutdown_connections 2021-02-10T11:41:25.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:25.777 7f9fe1bc4700 1 -- 172.21.2.112:0/2528213666 wait complete. 2021-02-10T11:41:25.783 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:41:26.211 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.198 7fd1b8d06700 1 Processor -- start 2021-02-10T11:41:26.211 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.198 7fd1b8d06700 1 -- start start 2021-02-10T11:41:26.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.199 7fd1b8d06700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fd1b40d1870 0x7fd1b41b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:26.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.199 7fd1b8d06700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd1b41b0a10 0x7fd1b40cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:26.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.199 7fd1b8d06700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1b40f0b10 con 0x7fd1b40d1c70 2021-02-10T11:41:26.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.199 7fd1b8d06700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1b40e5ed0 con 0x7fd1b40d1870 2021-02-10T11:41:26.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.199 7fd1b8d06700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1b40f0650 con 0x7fd1b41b0a10 2021-02-10T11:41:26.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.199 7fd1b2b13700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fd1b40d1870 0x7fd1b41b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:26.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.199 7fd1b2b13700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fd1b40d1870 0x7fd1b41b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:55458/0 (socket says 172.21.2.112:55458) 2021-02-10T11:41:26.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.199 7fd1b2b13700 1 -- 172.21.2.112:0/2968449093 learned_addr learned my addr 172.21.2.112:0/2968449093 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:41:26.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.199 7fd1b2b13700 1 -- 172.21.2.112:0/2968449093 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd1b41b0a10 msgr2=0x7fd1b40cf330 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:41:26.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.199 7fd1b2b13700 1 --2- 172.21.2.112:0/2968449093 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd1b41b0a10 0x7fd1b40cf330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:26.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.199 7fd1b2b13700 1 -- 172.21.2.112:0/2968449093 >> v1:172.21.2.120:6789/0 conn(0x7fd1b40d1c70 legacy=0x7fd1b40cf950 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:41:26.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.199 7fd1b2b13700 1 -- 172.21.2.112:0/2968449093 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1b40f0870 con 0x7fd1b40d1870 2021-02-10T11:41:26.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.199 7fd1b2b13700 1 --2- 172.21.2.112:0/2968449093 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd1b40d1870 0x7fd1b41b03f0 secure :-1 s=READY pgs=2204 cs=0 l=1 rev1=1 rx=0x7fd19c006700 tx=0x7fd19c009b80).ready entity=mon.0 client_cookie=355462e098fe3065 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:26.213 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.199 7fd1b1b11700 1 -- 172.21.2.112:0/2968449093 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd19c00c0c0 con 0x7fd1b40d1870 2021-02-10T11:41:26.213 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.199 7fd1b1b11700 1 -- 172.21.2.112:0/2968449093 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd19c00c340 con 0x7fd1b40d1870 2021-02-10T11:41:26.213 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.199 7fd1b1b11700 1 -- 172.21.2.112:0/2968449093 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd19c00c820 con 0x7fd1b40d1870 2021-02-10T11:41:26.213 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.200 7fd1b8d06700 1 -- 172.21.2.112:0/2968449093 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd1b40d1870 msgr2=0x7fd1b41b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:26.213 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.200 7fd1b8d06700 1 --2- 172.21.2.112:0/2968449093 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd1b40d1870 0x7fd1b41b03f0 secure :-1 s=READY pgs=2204 cs=0 l=1 rev1=1 rx=0x7fd19c006700 tx=0x7fd19c009b80).stop 2021-02-10T11:41:26.213 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.200 7fd1b8d06700 1 -- 172.21.2.112:0/2968449093 shutdown_connections 2021-02-10T11:41:26.213 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.200 7fd1b8d06700 1 --2- 172.21.2.112:0/2968449093 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd1b41b0a10 0x7fd1b40cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:26.213 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.200 7fd1b8d06700 1 --2- 172.21.2.112:0/2968449093 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd1b40d1870 0x7fd1b41b03f0 unknown :-1 s=CLOSED pgs=2204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:26.213 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.200 7fd1b8d06700 1 -- 172.21.2.112:0/2968449093 >> 172.21.2.112:0/2968449093 conn(0x7fd1b40ce450 msgr2=0x7fd1b40d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:26.213 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.200 7fd1b8d06700 1 -- 172.21.2.112:0/2968449093 shutdown_connections 2021-02-10T11:41:26.213 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.200 7fd1b8d06700 1 -- 172.21.2.112:0/2968449093 wait complete. 2021-02-10T11:41:26.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.214 7fd1b8d06700 1 Processor -- start 2021-02-10T11:41:26.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.214 7fd1b8d06700 1 -- start start 2021-02-10T11:41:26.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.215 7fd1b8d06700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd1b40b0ad0 0x7fd1b4091f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:26.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.215 7fd1b8d06700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd1b41a8850 0x7fd1b41a8c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:26.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.215 7fd1b8d06700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd1b41a6fa0 0x7fd1b41a73a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:26.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.215 7fd1b8d06700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1b41abfe0 con 0x7fd1b41a8850 2021-02-10T11:41:26.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.215 7fd1b8d06700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1b41a9c40 con 0x7fd1b41a6fa0 2021-02-10T11:41:26.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.215 7fd1b8d06700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1b41a9730 con 0x7fd1b40b0ad0 2021-02-10T11:41:26.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.215 7fd1b2312700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd1b41a8850 0x7fd1b41a8c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:26.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.215 7fd1b2312700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd1b41a8850 0x7fd1b41a8c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:55466/0 (socket says 172.21.2.112:55466) 2021-02-10T11:41:26.216 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.215 7fd1b2312700 1 -- 172.21.2.112:0/100033863 learned_addr learned my addr 172.21.2.112:0/100033863 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:41:26.216 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.215 7fd1b3314700 1 --2- 172.21.2.112:0/100033863 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd1b41a6fa0 0x7fd1b41a73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:26.216 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.215 7fd1b2b13700 1 --2- 172.21.2.112:0/100033863 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd1b40b0ad0 0x7fd1b4091f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:26.216 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.216 7fd1b2312700 1 -- 172.21.2.112:0/100033863 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd1b40b0ad0 msgr2=0x7fd1b4091f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:26.216 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.216 7fd1b2312700 1 --2- 172.21.2.112:0/100033863 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd1b40b0ad0 0x7fd1b4091f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:26.216 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.216 7fd1b2312700 1 -- 172.21.2.112:0/100033863 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd1b41a6fa0 msgr2=0x7fd1b41a73a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:26.216 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.216 7fd1b2312700 1 --2- 172.21.2.112:0/100033863 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd1b41a6fa0 0x7fd1b41a73a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:26.216 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.216 7fd1b2312700 1 -- 172.21.2.112:0/100033863 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1b41a9270 con 0x7fd1b41a8850 2021-02-10T11:41:26.216 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.216 7fd1b3314700 1 --2- 172.21.2.112:0/100033863 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd1b41a6fa0 0x7fd1b41a73a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:41:26.216 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.216 7fd1b2312700 1 --2- 172.21.2.112:0/100033863 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd1b41a8850 0x7fd1b41a8c50 secure :-1 s=READY pgs=2205 cs=0 l=1 rev1=1 rx=0x7fd1a400b5c0 tx=0x7fd1a400b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:26.216 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.216 7fd1ab7fe700 1 -- 172.21.2.112:0/100033863 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd1a400bd50 con 0x7fd1b41a8850 2021-02-10T11:41:26.216 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.216 7fd1ab7fe700 1 -- 172.21.2.112:0/100033863 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1a4006030 con 0x7fd1b41a8850 2021-02-10T11:41:26.216 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.216 7fd1ab7fe700 1 -- 172.21.2.112:0/100033863 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd1a40054d0 con 0x7fd1b41a8850 2021-02-10T11:41:26.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.216 7fd1b8d06700 1 -- 172.21.2.112:0/100033863 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1b41a9270 con 0x7fd1b41a8850 2021-02-10T11:41:26.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.216 7fd1b8d06700 1 -- 172.21.2.112:0/100033863 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1b41a7cb0 con 0x7fd1b41a8850 2021-02-10T11:41:26.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.217 7fd1ab7fe700 1 -- 172.21.2.112:0/100033863 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd1a4006210 con 0x7fd1b41a8850 2021-02-10T11:41:26.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.217 7fd1ab7fe700 1 --2- 172.21.2.112:0/100033863 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd1980186c0 0x7fd19801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:26.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.217 7fd1ab7fe700 1 -- 172.21.2.112:0/100033863 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fd1a40325a0 con 0x7fd1b41a8850 2021-02-10T11:41:26.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.218 7fd1b2b13700 1 --2- 172.21.2.112:0/100033863 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd1980186c0 0x7fd19801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:26.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.218 7fd1b2b13700 1 --2- 172.21.2.112:0/100033863 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd1980186c0 0x7fd19801abc0 crc :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:26.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.249 7fd1b8d06700 1 -- 172.21.2.112:0/100033863 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1b41a9710 con 0x7fd1b41a8850 2021-02-10T11:41:26.249 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.252 7fd1ab7fe700 1 -- 172.21.2.112:0/100033863 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd1a4000940 con 0x7fd1b41a8850 2021-02-10T11:41:26.377 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.380 7fd1b8d06700 1 -- 172.21.2.112:0/100033863 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd1b41a9710 con 0x7fd1b41a8850 2021-02-10T11:41:26.378 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.381 7fd1ab7fe700 1 -- 172.21.2.112:0/100033863 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) v1 ==== 94+0+2098 (secure 0 0 0) 0x7fd1a400bd50 con 0x7fd1b41a8850 2021-02-10T11:41:26.378 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 97 2021-02-10T11:41:26.378 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:41:26.378 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":97,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10671,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/668146927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":668146927},{"type":"v1","addr":"172.21.2.112:6833","nonce":668146927}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":93}],"filesystems":[{"mdsmap":{"epoch":97,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:41:23.731569","modified":"2021-02-10 11:41:25.740952","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8008},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8008":{"gid":8008,"name":"b","rank":0,"incarnation":94,"state":"up:active","state_seq":3,"addr":"172.21.2.120:6835/862287497","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":862287497},{"type":"v1","addr":"172.21.2.120:6835","nonce":862287497}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":12}]} 2021-02-10T11:41:26.380 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.382 7fd1b8d06700 1 -- 172.21.2.112:0/100033863 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd1980186c0 msgr2=0x7fd19801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:26.380 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.382 7fd1b8d06700 1 --2- 172.21.2.112:0/100033863 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd1980186c0 0x7fd19801abc0 crc :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:26.380 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.382 7fd1b8d06700 1 -- 172.21.2.112:0/100033863 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd1b41a8850 msgr2=0x7fd1b41a8c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:26.380 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.382 7fd1b8d06700 1 --2- 172.21.2.112:0/100033863 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd1b41a8850 0x7fd1b41a8c50 secure :-1 s=READY pgs=2205 cs=0 l=1 rev1=1 rx=0x7fd1a400b5c0 tx=0x7fd1a400b960).stop 2021-02-10T11:41:26.380 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.382 7fd1b8d06700 1 -- 172.21.2.112:0/100033863 shutdown_connections 2021-02-10T11:41:26.380 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.383 7fd1b8d06700 1 --2- 172.21.2.112:0/100033863 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd1980186c0 0x7fd19801abc0 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:26.380 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.383 7fd1b8d06700 1 --2- 172.21.2.112:0/100033863 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd1b41a6fa0 0x7fd1b41a73a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:26.380 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.383 7fd1b8d06700 1 --2- 172.21.2.112:0/100033863 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd1b41a8850 0x7fd1b41a8c50 unknown :-1 s=CLOSED pgs=2205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:26.380 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.383 7fd1b8d06700 1 --2- 172.21.2.112:0/100033863 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd1b40b0ad0 0x7fd1b4091f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:26.380 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.383 7fd1b8d06700 1 -- 172.21.2.112:0/100033863 >> 172.21.2.112:0/100033863 conn(0x7fd1b419f8f0 msgr2=0x7fd1b41a5f80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:26.380 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.383 7fd1b8d06700 1 -- 172.21.2.112:0/100033863 shutdown_connections 2021-02-10T11:41:26.381 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.383 7fd1b8d06700 1 -- 172.21.2.112:0/100033863 wait complete. 2021-02-10T11:41:26.388 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:41:26.818 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.804 7f6f3f429700 1 Processor -- start 2021-02-10T11:41:26.818 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.804 7f6f3f429700 1 -- start start 2021-02-10T11:41:26.818 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.804 7f6f3f429700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f6f381b0470 0x7f6f381b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:26.819 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.804 7f6f3f429700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f6f380ef610 0x7f6f380efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:26.819 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.804 7f6f3f429700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6f380f0030 0x7f6f380f0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:26.819 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.804 7f6f3f429700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f380e5ea0 con 0x7f6f380ef610 2021-02-10T11:41:26.819 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.804 7f6f3f429700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f380f0b50 con 0x7f6f381b0470 2021-02-10T11:41:26.819 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.804 7f6f3f429700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f380e52e0 con 0x7f6f380f0030 2021-02-10T11:41:26.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.805 7f6f3ca44700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f6f380ef610 0x7f6f380efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:26.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.805 7f6f3ca44700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f6f380ef610 0x7f6f380efa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:55472/0 (socket says 172.21.2.112:55472) 2021-02-10T11:41:26.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.805 7f6f3ca44700 1 -- 172.21.2.112:0/464233981 learned_addr learned my addr 172.21.2.112:0/464233981 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:41:26.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.805 7f6f3d245700 1 --2- 172.21.2.112:0/464233981 >> v2:172.21.2.120:3300/0 conn(0x7f6f381b0470 0x7f6f381b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:26.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.805 7f6f3da46700 1 --2- 172.21.2.112:0/464233981 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6f380f0030 0x7f6f380f0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:26.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.805 7f6f3ca44700 1 -- 172.21.2.112:0/464233981 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6f380f0030 msgr2=0x7f6f380f0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:26.821 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.805 7f6f3ca44700 1 --2- 172.21.2.112:0/464233981 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6f380f0030 0x7f6f380f0430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:26.821 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.805 7f6f3ca44700 1 -- 172.21.2.112:0/464233981 >> v2:172.21.2.120:3300/0 conn(0x7f6f381b0470 msgr2=0x7f6f381b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:26.821 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.805 7f6f3ca44700 1 --2- 172.21.2.112:0/464233981 >> v2:172.21.2.120:3300/0 conn(0x7f6f381b0470 0x7f6f381b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:26.821 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.805 7f6f3ca44700 1 -- 172.21.2.112:0/464233981 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f380f10d0 con 0x7f6f380ef610 2021-02-10T11:41:26.821 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.805 7f6f3ca44700 1 --2- 172.21.2.112:0/464233981 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6f380ef610 0x7f6f380efa10 secure :-1 s=READY pgs=2206 cs=0 l=1 rev1=1 rx=0x7f6f30006a80 tx=0x7f6f30009f00).ready entity=mon.0 client_cookie=818c5ed6471d3c49 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:26.821 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.805 7f6f2ffff700 1 -- 172.21.2.112:0/464233981 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6f3000b5c0 con 0x7f6f380ef610 2021-02-10T11:41:26.822 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.805 7f6f2ffff700 1 -- 172.21.2.112:0/464233981 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f3000b800 con 0x7f6f380ef610 2021-02-10T11:41:26.822 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.805 7f6f2ffff700 1 -- 172.21.2.112:0/464233981 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6f3000bd20 con 0x7f6f380ef610 2021-02-10T11:41:26.822 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.805 7f6f3f429700 1 -- 172.21.2.112:0/464233981 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6f380ef610 msgr2=0x7f6f380efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:26.822 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.805 7f6f3f429700 1 --2- 172.21.2.112:0/464233981 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6f380ef610 0x7f6f380efa10 secure :-1 s=READY pgs=2206 cs=0 l=1 rev1=1 rx=0x7f6f30006a80 tx=0x7f6f30009f00).stop 2021-02-10T11:41:26.822 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.806 7f6f3f429700 1 -- 172.21.2.112:0/464233981 shutdown_connections 2021-02-10T11:41:26.823 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.806 7f6f3f429700 1 --2- 172.21.2.112:0/464233981 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6f380f0030 0x7f6f380f0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:26.823 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.806 7f6f3f429700 1 --2- 172.21.2.112:0/464233981 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6f380ef610 0x7f6f380efa10 unknown :-1 s=CLOSED pgs=2206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:26.823 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.806 7f6f3f429700 1 --2- 172.21.2.112:0/464233981 >> v2:172.21.2.120:3300/0 conn(0x7f6f381b0470 0x7f6f381b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:26.823 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.806 7f6f3f429700 1 -- 172.21.2.112:0/464233981 >> 172.21.2.112:0/464233981 conn(0x7f6f380ce3e0 msgr2=0x7f6f380e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:26.823 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.806 7f6f3f429700 1 -- 172.21.2.112:0/464233981 shutdown_connections 2021-02-10T11:41:26.823 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.806 7f6f3f429700 1 -- 172.21.2.112:0/464233981 wait complete. 2021-02-10T11:41:26.824 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.821 7f6f3f429700 1 Processor -- start 2021-02-10T11:41:26.824 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.821 7f6f3f429700 1 -- start start 2021-02-10T11:41:26.827 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.821 7f6f3f429700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6f3818c720 0x7f6f3818cb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:26.827 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.821 7f6f3f429700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6f3818f2b0 0x7f6f3818f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:26.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.821 7f6f3f429700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6f38090970 0x7f6f38090d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:26.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.821 7f6f3f429700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f3818fcd0 con 0x7f6f38090970 2021-02-10T11:41:26.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.821 7f6f3f429700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f38090070 con 0x7f6f3818f2b0 2021-02-10T11:41:26.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.821 7f6f3f429700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f38091390 con 0x7f6f3818c720 2021-02-10T11:41:26.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.822 7f6f3da46700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6f38090970 0x7f6f38090d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:26.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.822 7f6f3da46700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6f38090970 0x7f6f38090d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:55482/0 (socket says 172.21.2.112:55482) 2021-02-10T11:41:26.829 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.822 7f6f3da46700 1 -- 172.21.2.112:0/1664983421 learned_addr learned my addr 172.21.2.112:0/1664983421 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:41:26.829 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.822 7f6f3da46700 1 -- 172.21.2.112:0/1664983421 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6f3818c720 msgr2=0x7f6f3818cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:26.829 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.822 7f6f3da46700 1 --2- 172.21.2.112:0/1664983421 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6f3818c720 0x7f6f3818cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:26.829 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.822 7f6f3da46700 1 -- 172.21.2.112:0/1664983421 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6f3818f2b0 msgr2=0x7f6f3818f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:26.829 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.822 7f6f3da46700 1 --2- 172.21.2.112:0/1664983421 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6f3818f2b0 0x7f6f3818f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:26.830 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.822 7f6f3da46700 1 -- 172.21.2.112:0/1664983421 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f380915b0 con 0x7f6f38090970 2021-02-10T11:41:26.831 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.822 7f6f3da46700 1 --2- 172.21.2.112:0/1664983421 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6f38090970 0x7f6f38090d70 secure :-1 s=READY pgs=2207 cs=0 l=1 rev1=1 rx=0x7f6f34006f20 tx=0x7f6f3400b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:26.831 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.822 7f6f2dffb700 1 -- 172.21.2.112:0/1664983421 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6f34006bb0 con 0x7f6f38090970 2021-02-10T11:41:26.831 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.822 7f6f2dffb700 1 -- 172.21.2.112:0/1664983421 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f3400bd80 con 0x7f6f38090970 2021-02-10T11:41:26.831 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.822 7f6f2dffb700 1 -- 172.21.2.112:0/1664983421 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6f340022b0 con 0x7f6f38090970 2021-02-10T11:41:26.831 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.822 7f6f3f429700 1 -- 172.21.2.112:0/1664983421 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f380915b0 con 0x7f6f38090970 2021-02-10T11:41:26.832 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.822 7f6f3f429700 1 -- 172.21.2.112:0/1664983421 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f380915b0 con 0x7f6f38090970 2021-02-10T11:41:26.832 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.823 7f6f2dffb700 1 -- 172.21.2.112:0/1664983421 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6f34005330 con 0x7f6f38090970 2021-02-10T11:41:26.832 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.823 7f6f2dffb700 1 --2- 172.21.2.112:0/1664983421 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6f180186c0 0x7f6f1801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:26.832 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.823 7f6f2dffb700 1 -- 172.21.2.112:0/1664983421 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f6f34032810 con 0x7f6f38090970 2021-02-10T11:41:26.833 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.823 7f6f3d245700 1 --2- 172.21.2.112:0/1664983421 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6f180186c0 0x7f6f1801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:26.833 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.824 7f6f3d245700 1 --2- 172.21.2.112:0/1664983421 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6f180186c0 0x7f6f1801abc0 crc :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:26.852 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.854 7f6f3f429700 1 -- 172.21.2.112:0/1664983421 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f38090070 con 0x7f6f38090970 2021-02-10T11:41:26.854 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.857 7f6f2dffb700 1 -- 172.21.2.112:0/1664983421 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6f34006370 con 0x7f6f38090970 2021-02-10T11:41:26.977 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.980 7f6f3f429700 1 -- 172.21.2.112:0/1664983421 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6f38090070 con 0x7f6f38090970 2021-02-10T11:41:26.978 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.981 7f6f2dffb700 1 -- 172.21.2.112:0/1664983421 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v117) v1 ==== 74+0+14007 (secure 0 0 0) 0x7f6f34005330 con 0x7f6f38090970 2021-02-10T11:41:26.978 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:41:26.978 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":117,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:41:24.733802","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:41:21.697492","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"116","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":25,"pool_name":"cephfs_data","create_time":"2021-02-10 11:41:22.721757","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"117","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:41:26.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.982 7f6f3f429700 1 -- 172.21.2.112:0/1664983421 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6f180186c0 msgr2=0x7f6f1801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:26.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.982 7f6f3f429700 1 --2- 172.21.2.112:0/1664983421 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6f180186c0 0x7f6f1801abc0 crc :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:26.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.982 7f6f3f429700 1 -- 172.21.2.112:0/1664983421 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6f38090970 msgr2=0x7f6f38090d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:26.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.982 7f6f3f429700 1 --2- 172.21.2.112:0/1664983421 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6f38090970 0x7f6f38090d70 secure :-1 s=READY pgs=2207 cs=0 l=1 rev1=1 rx=0x7f6f34006f20 tx=0x7f6f3400b990).stop 2021-02-10T11:41:26.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.983 7f6f3f429700 1 -- 172.21.2.112:0/1664983421 shutdown_connections 2021-02-10T11:41:26.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.983 7f6f3f429700 1 --2- 172.21.2.112:0/1664983421 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6f180186c0 0x7f6f1801abc0 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:26.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.983 7f6f3f429700 1 --2- 172.21.2.112:0/1664983421 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6f3818f2b0 0x7f6f3818f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:26.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.983 7f6f3f429700 1 --2- 172.21.2.112:0/1664983421 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6f38090970 0x7f6f38090d70 unknown :-1 s=CLOSED pgs=2207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:26.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.983 7f6f3f429700 1 --2- 172.21.2.112:0/1664983421 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6f3818c720 0x7f6f3818cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:26.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.983 7f6f3f429700 1 -- 172.21.2.112:0/1664983421 >> 172.21.2.112:0/1664983421 conn(0x7f6f3810a400 msgr2=0x7f6f380b1fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:26.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.983 7f6f3f429700 1 -- 172.21.2.112:0/1664983421 shutdown_connections 2021-02-10T11:41:26.981 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:26.983 7f6f3f429700 1 -- 172.21.2.112:0/1664983421 wait complete. 2021-02-10T11:41:26.988 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:41:27.426 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:27.413 7f07789e4700 1 Processor -- start 2021-02-10T11:41:27.427 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:27.413 7f07789e4700 1 -- start start 2021-02-10T11:41:27.427 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:27.413 7f07789e4700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f07740d1c80 0x7f07740cef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:27.427 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:27.413 7f07789e4700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07740d0180 con 0x7f07740cf590 2021-02-10T11:41:27.427 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:27.413 7f07789e4700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07740fe1d0 con 0x7f07740d1880 2021-02-10T11:41:27.427 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:27.413 7f07789e4700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07741b0cc0 con 0x7f07740d1c80 2021-02-10T11:41:27.427 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:27.413 7f0771e1b700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f07740d1c80 0x7f07740cef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:27.427 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:27.413 7f0771e1b700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f07740d1c80 0x7f07740cef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:55486/0 (socket says 172.21.2.112:55486) 2021-02-10T11:41:27.427 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:27.413 7f0771e1b700 1 -- 172.21.2.112:0/3633709332 learned_addr learned my addr 172.21.2.112:0/3633709332 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:41:27.427 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:27.413 7f0772e1d700 1 --1- 172.21.2.112:0/3633709332 >> v1:172.21.2.112:6789/0 conn(0x7f07740cf590 0x7f07740cf990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:54952/0 (socket says 172.21.2.112:54952) 2021-02-10T11:41:27.427 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:27.414 7f0771e1b700 1 -- 172.21.2.112:0/3633709332 >> v1:172.21.2.120:6789/0 conn(0x7f07740d1880 legacy=0x7f07741b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:27.428 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:27.414 7f0771e1b700 1 -- 172.21.2.112:0/3633709332 >> v1:172.21.2.112:6789/0 conn(0x7f07740cf590 legacy=0x7f07740cf990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:27.428 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:27.414 7f077161a700 1 -- 172.21.2.112:0/3633709332 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 13214123 0 0) 0x7f0764000c60 con 0x7f07740d1880 2021-02-10T11:41:27.428 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:27.414 7f0771e1b700 1 -- 172.21.2.112:0/3633709332 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0774109a70 con 0x7f07740d1c80 2021-02-10T11:41:27.428 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:27.414 7f0771e1b700 1 --2- 172.21.2.112:0/3633709332 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f07740d1c80 0x7f07740cef70 secure :-1 s=READY pgs=2209 cs=0 l=1 rev1=1 rx=0x7f075c006700 tx=0x7f075c009b80).ready entity=mon.0 client_cookie=c184311e955e743 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:27.428 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:27.414 7f077161a700 1 -- 172.21.2.112:0/3633709332 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f075c00c0c0 con 0x7f07740d1c80 2021-02-10T11:41:27.428 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:27.414 7f077161a700 1 -- 172.21.2.112:0/3633709332 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f075c00c400 con 0x7f07740d1c80 2021-02-10T11:41:27.428 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:27.414 7f077161a700 1 -- 172.21.2.112:0/3633709332 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f075c00c8e0 con 0x7f07740d1c80 2021-02-10T11:41:27.428 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:27.414 7f07789e4700 1 -- 172.21.2.112:0/3633709332 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f07740d1c80 msgr2=0x7f07740cef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:27.428 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:27.414 7f07789e4700 1 --2- 172.21.2.112:0/3633709332 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f07740d1c80 0x7f07740cef70 secure :-1 s=READY pgs=2209 cs=0 l=1 rev1=1 rx=0x7f075c006700 tx=0x7f075c009b80).stop 2021-02-10T11:41:27.428 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:27.415 7f07789e4700 1 -- 172.21.2.112:0/3633709332 shutdown_connections 2021-02-10T11:41:27.428 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:27.415 7f07789e4700 1 --2- 172.21.2.112:0/3633709332 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f07740d1c80 0x7f07740cef70 unknown :-1 s=CLOSED pgs=2209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:27.428 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:27.415 7f07789e4700 1 -- 172.21.2.112:0/3633709332 >> 172.21.2.112:0/3633709332 conn(0x7f07740ce430 msgr2=0x7f07740d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:27.428 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:27.415 7f07789e4700 1 -- 172.21.2.112:0/3633709332 shutdown_connections 2021-02-10T11:41:27.429 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:27.415 7f07789e4700 1 -- 172.21.2.112:0/3633709332 wait complete. 2021-02-10T11:41:27.430 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:27.430 7f07789e4700 1 Processor -- start 2021-02-10T11:41:27.430 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:27.430 7f07789e4700 1 -- start start 2021-02-10T11:41:27.430 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:27.430 7f07789e4700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0774086810 0x7f077419fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:27.430 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:27.430 7f07789e4700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f07740b04d0 0x7f0774097450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:27.430 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:27.430 7f07789e4700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f07740b08d0 0x7f0774097a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:27.431 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:27.430 7f07789e4700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07741a02c0 con 0x7f07740b04d0 2021-02-10T11:41:27.431 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:27.430 7f07789e4700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0774098090 con 0x7f0774086810 2021-02-10T11:41:27.431 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:27.430 7f07789e4700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07740982b0 con 0x7f07740b08d0 2021-02-10T11:41:27.431 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:27.431 7f0771e1b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f07740b04d0 0x7f0774097450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:27.431 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:27.431 7f0771e1b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f07740b04d0 0x7f0774097450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:55492/0 (socket says 172.21.2.112:55492) 2021-02-10T11:41:27.431 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:27.431 7f0771e1b700 1 -- 172.21.2.112:0/1125531378 learned_addr learned my addr 172.21.2.112:0/1125531378 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:41:27.431 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:27.431 7f077261c700 1 --2- 172.21.2.112:0/1125531378 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0774086810 0x7f077419fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:27.431 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:27.431 7f0772e1d700 1 --2- 172.21.2.112:0/1125531378 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f07740b08d0 0x7f0774097a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:27.431 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:27.431 7f0771e1b700 1 -- 172.21.2.112:0/1125531378 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f07740b08d0 msgr2=0x7f0774097a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:27.431 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:27.431 7f0771e1b700 1 --2- 172.21.2.112:0/1125531378 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f07740b08d0 0x7f0774097a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:27.431 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:27.431 7f0771e1b700 1 -- 172.21.2.112:0/1125531378 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0774086810 msgr2=0x7f077419fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:27.431 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:27.431 7f0771e1b700 1 --2- 172.21.2.112:0/1125531378 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0774086810 0x7f077419fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:27.431 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:27.431 7f0771e1b700 1 -- 172.21.2.112:0/1125531378 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07740984d0 con 0x7f07740b04d0 2021-02-10T11:41:27.432 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:27.431 7f0772e1d700 1 --2- 172.21.2.112:0/1125531378 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f07740b08d0 0x7f0774097a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:41:27.432 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:27.431 7f0771e1b700 1 --2- 172.21.2.112:0/1125531378 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f07740b04d0 0x7f0774097450 secure :-1 s=READY pgs=2210 cs=0 l=1 rev1=1 rx=0x7f075c0066a0 tx=0x7f075c00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:27.432 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:27.431 7f076affd700 1 -- 172.21.2.112:0/1125531378 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f075c00bd80 con 0x7f07740b04d0 2021-02-10T11:41:27.432 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:27.431 7f076affd700 1 -- 172.21.2.112:0/1125531378 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f075c0052e0 con 0x7f07740b04d0 2021-02-10T11:41:27.432 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:27.431 7f07789e4700 1 -- 172.21.2.112:0/1125531378 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07740984d0 con 0x7f07740b04d0 2021-02-10T11:41:27.432 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:27.431 7f076affd700 1 -- 172.21.2.112:0/1125531378 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f075c0057c0 con 0x7f07740b04d0 2021-02-10T11:41:27.432 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:27.432 7f07789e4700 1 -- 172.21.2.112:0/1125531378 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0774098b80 con 0x7f07740b04d0 2021-02-10T11:41:27.432 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:27.432 7f076affd700 1 -- 172.21.2.112:0/1125531378 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f075c006030 con 0x7f07740b04d0 2021-02-10T11:41:27.432 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:27.432 7f076affd700 1 --2- 172.21.2.112:0/1125531378 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f07580186c0 0x7f075801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:27.432 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:27.432 7f076affd700 1 -- 172.21.2.112:0/1125531378 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f075c032430 con 0x7f07740b04d0 2021-02-10T11:41:27.432 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:27.432 7f077261c700 1 --2- 172.21.2.112:0/1125531378 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f07580186c0 0x7f075801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:27.432 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:27.433 7f077261c700 1 --2- 172.21.2.112:0/1125531378 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f07580186c0 0x7f075801abc0 crc :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:27.461 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:27.463 7f07789e4700 1 -- 172.21.2.112:0/1125531378 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07740fe1d0 con 0x7f07740b04d0 2021-02-10T11:41:27.463 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:27.466 7f076affd700 1 -- 172.21.2.112:0/1125531378 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f075c030d60 con 0x7f07740b04d0 2021-02-10T11:41:27.594 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:27.597 7f07789e4700 1 -- 172.21.2.112:0/1125531378 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f07740fe1d0 con 0x7f07740b04d0 2021-02-10T11:41:27.598 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:27.600 7f076affd700 1 -- 172.21.2.112:0/1125531378 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v69) v1 ==== 184+0+0 (secure 0 0 0) 0x7f075c006030 con 0x7f07740b04d0 2021-02-10T11:41:27.601 INFO:teuthology.orchestra.run.gibba012.stderr:updated caps for client.0 2021-02-10T11:41:27.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:27.605 7f07789e4700 1 -- 172.21.2.112:0/1125531378 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f07580186c0 msgr2=0x7f075801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:27.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:27.605 7f07789e4700 1 --2- 172.21.2.112:0/1125531378 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f07580186c0 0x7f075801abc0 crc :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:27.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:27.605 7f07789e4700 1 -- 172.21.2.112:0/1125531378 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f07740b04d0 msgr2=0x7f0774097450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:27.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:27.605 7f07789e4700 1 --2- 172.21.2.112:0/1125531378 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f07740b04d0 0x7f0774097450 secure :-1 s=READY pgs=2210 cs=0 l=1 rev1=1 rx=0x7f075c0066a0 tx=0x7f075c00b990).stop 2021-02-10T11:41:27.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:27.605 7f07789e4700 1 -- 172.21.2.112:0/1125531378 shutdown_connections 2021-02-10T11:41:27.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:27.605 7f07789e4700 1 --2- 172.21.2.112:0/1125531378 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f07580186c0 0x7f075801abc0 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:27.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:27.605 7f07789e4700 1 --2- 172.21.2.112:0/1125531378 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f07740b08d0 0x7f0774097a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:27.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:27.605 7f07789e4700 1 --2- 172.21.2.112:0/1125531378 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f07740b04d0 0x7f0774097450 unknown :-1 s=CLOSED pgs=2210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:27.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:27.605 7f07789e4700 1 --2- 172.21.2.112:0/1125531378 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0774086810 0x7f077419fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:27.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:27.605 7f07789e4700 1 -- 172.21.2.112:0/1125531378 >> 172.21.2.112:0/1125531378 conn(0x7f077419f880 msgr2=0x7f077408f890 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:27.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:27.606 7f07789e4700 1 -- 172.21.2.112:0/1125531378 shutdown_connections 2021-02-10T11:41:27.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:27.606 7f07789e4700 1 -- 172.21.2.112:0/1125531378 wait complete. 2021-02-10T11:41:27.612 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:41:28.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.028 7f5c6051c700 1 Processor -- start 2021-02-10T11:41:28.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.028 7f5c6051c700 1 -- start start 2021-02-10T11:41:28.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.028 7f5c6051c700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f5c580d1880 0x7f5c581b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:28.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.028 7f5c6051c700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f5c581b0af0 0x7f5c58108950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:28.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.028 7f5c6051c700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c580fe1d0 con 0x7f5c580d1c80 2021-02-10T11:41:28.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.028 7f5c6051c700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c58109a70 con 0x7f5c581b0af0 2021-02-10T11:41:28.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.028 7f5c6051c700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c58109760 con 0x7f5c580d1880 2021-02-10T11:41:28.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.029 7f5c5db37700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f5c581b0af0 0x7f5c58108950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:28.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.029 7f5c5db37700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f5c581b0af0 0x7f5c58108950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:55500/0 (socket says 172.21.2.112:55500) 2021-02-10T11:41:28.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.029 7f5c5eb39700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f5c580d1c80 0x7f5c58108f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:54962/0 (socket says 172.21.2.112:54962) 2021-02-10T11:41:28.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.029 7f5c5db37700 1 -- 172.21.2.112:0/1165375763 learned_addr learned my addr 172.21.2.112:0/1165375763 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:41:28.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.029 7f5c5db37700 1 -- 172.21.2.112:0/1165375763 >> v2:172.21.2.120:3300/0 conn(0x7f5c580d1880 msgr2=0x7f5c581b04d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:41:28.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.029 7f5c5db37700 1 --2- 172.21.2.112:0/1165375763 >> v2:172.21.2.120:3300/0 conn(0x7f5c580d1880 0x7f5c581b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:28.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.029 7f5c5db37700 1 -- 172.21.2.112:0/1165375763 >> v1:172.21.2.112:6789/0 conn(0x7f5c580d1c80 legacy=0x7f5c58108f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:28.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.029 7f5c5db37700 1 -- 172.21.2.112:0/1165375763 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c58088070 con 0x7f5c581b0af0 2021-02-10T11:41:28.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.029 7f5c5db37700 1 --2- 172.21.2.112:0/1165375763 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5c581b0af0 0x7f5c58108950 secure :-1 s=READY pgs=2212 cs=0 l=1 rev1=1 rx=0x7f5c54006700 tx=0x7f5c54009b80).ready entity=mon.0 client_cookie=53978aa135c01387 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:28.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.029 7f5c5d336700 1 -- 172.21.2.112:0/1165375763 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5c5400c0c0 con 0x7f5c581b0af0 2021-02-10T11:41:28.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.029 7f5c5d336700 1 -- 172.21.2.112:0/1165375763 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c5400c400 con 0x7f5c581b0af0 2021-02-10T11:41:28.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.029 7f5c5d336700 1 -- 172.21.2.112:0/1165375763 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5c5400c920 con 0x7f5c581b0af0 2021-02-10T11:41:28.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.029 7f5c6051c700 1 -- 172.21.2.112:0/1165375763 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5c581b0af0 msgr2=0x7f5c58108950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:28.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.029 7f5c6051c700 1 --2- 172.21.2.112:0/1165375763 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5c581b0af0 0x7f5c58108950 secure :-1 s=READY pgs=2212 cs=0 l=1 rev1=1 rx=0x7f5c54006700 tx=0x7f5c54009b80).stop 2021-02-10T11:41:28.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.030 7f5c6051c700 1 -- 172.21.2.112:0/1165375763 shutdown_connections 2021-02-10T11:41:28.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.030 7f5c6051c700 1 --2- 172.21.2.112:0/1165375763 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5c581b0af0 0x7f5c58108950 unknown :-1 s=CLOSED pgs=2212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:28.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.030 7f5c6051c700 1 --2- 172.21.2.112:0/1165375763 >> v2:172.21.2.120:3300/0 conn(0x7f5c580d1880 0x7f5c581b04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:28.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.030 7f5c6051c700 1 -- 172.21.2.112:0/1165375763 >> 172.21.2.112:0/1165375763 conn(0x7f5c580ce430 msgr2=0x7f5c580d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:28.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.030 7f5c6051c700 1 -- 172.21.2.112:0/1165375763 shutdown_connections 2021-02-10T11:41:28.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.030 7f5c6051c700 1 -- 172.21.2.112:0/1165375763 wait complete. 2021-02-10T11:41:28.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.045 7f5c6051c700 1 Processor -- start 2021-02-10T11:41:28.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.045 7f5c6051c700 1 -- start start 2021-02-10T11:41:28.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.045 7f5c6051c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5c581b9280 0x7f5c5818f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:28.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.045 7f5c6051c700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5c581b9680 0x7f5c5818fd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:28.047 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.045 7f5c6051c700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5c581acd60 0x7f5c581ad160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:28.047 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.045 7f5c6051c700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c58190390 con 0x7f5c581b9280 2021-02-10T11:41:28.047 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.045 7f5c6051c700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c581907c0 con 0x7f5c581b9680 2021-02-10T11:41:28.047 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.045 7f5c6051c700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c581ac3d0 con 0x7f5c581acd60 2021-02-10T11:41:28.047 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.046 7f5c5e338700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5c581b9280 0x7f5c5818f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:28.047 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.046 7f5c5e338700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5c581b9280 0x7f5c5818f750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:55504/0 (socket says 172.21.2.112:55504) 2021-02-10T11:41:28.047 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.046 7f5c5e338700 1 -- 172.21.2.112:0/734154935 learned_addr learned my addr 172.21.2.112:0/734154935 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:41:28.047 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.046 7f5c5db37700 1 --2- 172.21.2.112:0/734154935 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5c581b9680 0x7f5c5818fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:28.047 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.046 7f5c5eb39700 1 --2- 172.21.2.112:0/734154935 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5c581acd60 0x7f5c581ad160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:28.047 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.046 7f5c5e338700 1 -- 172.21.2.112:0/734154935 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5c581acd60 msgr2=0x7f5c581ad160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:28.047 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.046 7f5c5e338700 1 --2- 172.21.2.112:0/734154935 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5c581acd60 0x7f5c581ad160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:28.047 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.046 7f5c5e338700 1 -- 172.21.2.112:0/734154935 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5c581b9680 msgr2=0x7f5c5818fd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:28.047 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.046 7f5c5e338700 1 --2- 172.21.2.112:0/734154935 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5c581b9680 0x7f5c5818fd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:28.048 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.046 7f5c5e338700 1 -- 172.21.2.112:0/734154935 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c581ad780 con 0x7f5c581b9280 2021-02-10T11:41:28.048 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.046 7f5c5eb39700 1 --2- 172.21.2.112:0/734154935 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5c581acd60 0x7f5c581ad160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:41:28.048 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.046 7f5c5e338700 1 --2- 172.21.2.112:0/734154935 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5c581b9280 0x7f5c5818f750 secure :-1 s=READY pgs=2213 cs=0 l=1 rev1=1 rx=0x7f5c4c006f00 tx=0x7f5c4c00b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:28.048 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.046 7f5c4affd700 1 -- 172.21.2.112:0/734154935 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5c4c00bd30 con 0x7f5c581b9280 2021-02-10T11:41:28.048 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.046 7f5c4affd700 1 -- 172.21.2.112:0/734154935 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c4c000c90 con 0x7f5c581b9280 2021-02-10T11:41:28.048 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.046 7f5c6051c700 1 -- 172.21.2.112:0/734154935 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c581ad780 con 0x7f5c581b9280 2021-02-10T11:41:28.048 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.046 7f5c4affd700 1 -- 172.21.2.112:0/734154935 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5c4c0022a0 con 0x7f5c581b9280 2021-02-10T11:41:28.048 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.047 7f5c6051c700 1 -- 172.21.2.112:0/734154935 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c581ad780 con 0x7f5c581b9280 2021-02-10T11:41:28.048 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.047 7f5c4affd700 1 -- 172.21.2.112:0/734154935 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5c4c005310 con 0x7f5c581b9280 2021-02-10T11:41:28.048 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.047 7f5c4affd700 1 --2- 172.21.2.112:0/734154935 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5c400186c0 0x7f5c4001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:28.048 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.047 7f5c4affd700 1 -- 172.21.2.112:0/734154935 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f5c4c0325f0 con 0x7f5c581b9280 2021-02-10T11:41:28.048 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.048 7f5c5db37700 1 --2- 172.21.2.112:0/734154935 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5c400186c0 0x7f5c4001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:28.048 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.048 7f5c5db37700 1 --2- 172.21.2.112:0/734154935 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5c400186c0 0x7f5c4001abc0 crc :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:28.075 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.078 7f5c6051c700 1 -- 172.21.2.112:0/734154935 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c58190390 con 0x7f5c581b9280 2021-02-10T11:41:28.078 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.081 7f5c4affd700 1 -- 172.21.2.112:0/734154935 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5c4c006b90 con 0x7f5c581b9280 2021-02-10T11:41:28.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.215 7f5c6051c700 1 -- 172.21.2.112:0/734154935 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f5c58190390 con 0x7f5c581b9280 2021-02-10T11:41:28.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.218 7f5c4affd700 1 -- 172.21.2.112:0/734154935 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v70) v1 ==== 184+0+0 (secure 0 0 0) 0x7f5c4c005310 con 0x7f5c581b9280 2021-02-10T11:41:28.215 INFO:teuthology.orchestra.run.gibba012.stderr:updated caps for client.1 2021-02-10T11:41:28.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.219 7f5c6051c700 1 -- 172.21.2.112:0/734154935 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5c400186c0 msgr2=0x7f5c4001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:28.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.219 7f5c6051c700 1 --2- 172.21.2.112:0/734154935 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5c400186c0 0x7f5c4001abc0 crc :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:28.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.219 7f5c6051c700 1 -- 172.21.2.112:0/734154935 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5c581b9280 msgr2=0x7f5c5818f750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:28.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.219 7f5c6051c700 1 --2- 172.21.2.112:0/734154935 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5c581b9280 0x7f5c5818f750 secure :-1 s=READY pgs=2213 cs=0 l=1 rev1=1 rx=0x7f5c4c006f00 tx=0x7f5c4c00b940).stop 2021-02-10T11:41:28.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.220 7f5c6051c700 1 -- 172.21.2.112:0/734154935 shutdown_connections 2021-02-10T11:41:28.218 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.220 7f5c6051c700 1 --2- 172.21.2.112:0/734154935 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5c400186c0 0x7f5c4001abc0 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:28.218 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.220 7f5c6051c700 1 --2- 172.21.2.112:0/734154935 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5c581b9680 0x7f5c5818fd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:28.218 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.220 7f5c6051c700 1 --2- 172.21.2.112:0/734154935 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5c581acd60 0x7f5c581ad160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:28.218 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.220 7f5c6051c700 1 --2- 172.21.2.112:0/734154935 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5c581b9280 0x7f5c5818f750 unknown :-1 s=CLOSED pgs=2213 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:28.218 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.220 7f5c6051c700 1 -- 172.21.2.112:0/734154935 >> 172.21.2.112:0/734154935 conn(0x7f5c580845b0 msgr2=0x7f5c5818c290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:28.218 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.220 7f5c6051c700 1 -- 172.21.2.112:0/734154935 shutdown_connections 2021-02-10T11:41:28.218 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.220 7f5c6051c700 1 -- 172.21.2.112:0/734154935 wait complete. 2021-02-10T11:41:28.225 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.2 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:41:28.657 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.643 7f43282be700 1 Processor -- start 2021-02-10T11:41:28.657 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.643 7f43282be700 1 -- start start 2021-02-10T11:41:28.657 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.643 7f43282be700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f43200d1c80 0x7f43200cef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:28.657 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.643 7f43282be700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f43200cf590 0x7f43200cf990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:28.657 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.643 7f43282be700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43200fe1d0 con 0x7f43200d1880 2021-02-10T11:41:28.657 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.643 7f43282be700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43201b0cc0 con 0x7f43200d1c80 2021-02-10T11:41:28.657 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.643 7f43282be700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4320109a70 con 0x7f43200cf590 2021-02-10T11:41:28.657 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.643 7f43258d9700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f43200d1c80 0x7f43200cef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:28.658 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.644 7f43260da700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f43200d1880 0x7f43201b04d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:54974/0 (socket says 172.21.2.112:54974) 2021-02-10T11:41:28.658 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.644 7f43258d9700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f43200d1c80 0x7f43200cef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:55512/0 (socket says 172.21.2.112:55512) 2021-02-10T11:41:28.658 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.644 7f43260da700 1 -- 172.21.2.112:0/2436483580 learned_addr learned my addr 172.21.2.112:0/2436483580 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:41:28.658 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.644 7f43268db700 1 --2- 172.21.2.112:0/2436483580 >> v2:172.21.2.120:3300/0 conn(0x7f43200cf590 0x7f43200cf990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:28.658 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.644 7f43258d9700 1 -- 172.21.2.112:0/2436483580 >> v2:172.21.2.120:3300/0 conn(0x7f43200cf590 msgr2=0x7f43200cf990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:28.658 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.644 7f43258d9700 1 --2- 172.21.2.112:0/2436483580 >> v2:172.21.2.120:3300/0 conn(0x7f43200cf590 0x7f43200cf990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:28.658 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.644 7f43258d9700 1 -- 172.21.2.112:0/2436483580 >> v1:172.21.2.112:6789/0 conn(0x7f43200d1880 legacy=0x7f43201b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:28.658 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.644 7f43250d8700 1 -- 172.21.2.112:0/2436483580 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3525752485 0 0) 0x7f4310000ce0 con 0x7f43200d1880 2021-02-10T11:41:28.658 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.644 7f43258d9700 1 -- 172.21.2.112:0/2436483580 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4320088070 con 0x7f43200d1c80 2021-02-10T11:41:28.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.644 7f43258d9700 1 --2- 172.21.2.112:0/2436483580 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f43200d1c80 0x7f43200cef70 secure :-1 s=READY pgs=2215 cs=0 l=1 rev1=1 rx=0x7f431c006650 tx=0x7f431c009a00).ready entity=mon.0 client_cookie=e5d7554fa0ea1560 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:28.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.644 7f43250d8700 1 -- 172.21.2.112:0/2436483580 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f431c009df0 con 0x7f43200d1c80 2021-02-10T11:41:28.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.644 7f43250d8700 1 -- 172.21.2.112:0/2436483580 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f431c0009b0 con 0x7f43200d1c80 2021-02-10T11:41:28.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.644 7f43250d8700 1 -- 172.21.2.112:0/2436483580 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f431c00c2b0 con 0x7f43200d1c80 2021-02-10T11:41:28.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.644 7f43282be700 1 -- 172.21.2.112:0/2436483580 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f43200d1c80 msgr2=0x7f43200cef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:28.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.644 7f43282be700 1 --2- 172.21.2.112:0/2436483580 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f43200d1c80 0x7f43200cef70 secure :-1 s=READY pgs=2215 cs=0 l=1 rev1=1 rx=0x7f431c006650 tx=0x7f431c009a00).stop 2021-02-10T11:41:28.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.645 7f43282be700 1 -- 172.21.2.112:0/2436483580 shutdown_connections 2021-02-10T11:41:28.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.645 7f43282be700 1 --2- 172.21.2.112:0/2436483580 >> v2:172.21.2.120:3300/0 conn(0x7f43200cf590 0x7f43200cf990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:28.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.645 7f43282be700 1 --2- 172.21.2.112:0/2436483580 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f43200d1c80 0x7f43200cef70 unknown :-1 s=CLOSED pgs=2215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:28.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.645 7f43282be700 1 -- 172.21.2.112:0/2436483580 >> 172.21.2.112:0/2436483580 conn(0x7f43200ce430 msgr2=0x7f43200d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:28.660 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.645 7f43282be700 1 -- 172.21.2.112:0/2436483580 shutdown_connections 2021-02-10T11:41:28.660 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.645 7f43282be700 1 -- 172.21.2.112:0/2436483580 wait complete. 2021-02-10T11:41:28.662 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.660 7f43282be700 1 Processor -- start 2021-02-10T11:41:28.663 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.660 7f43282be700 1 -- start start 2021-02-10T11:41:28.663 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.660 7f43282be700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f43201894e0 0x7f4320188990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:28.663 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.660 7f43282be700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f43201899e0 0x7f4320189de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:28.663 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.660 7f43282be700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f432008e790 0x7f432008eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:28.663 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.660 7f43282be700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f432008e210 con 0x7f43201894e0 2021-02-10T11:41:28.663 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.660 7f43282be700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4320188fb0 con 0x7f432008e790 2021-02-10T11:41:28.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.660 7f43282be700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4320189190 con 0x7f43201899e0 2021-02-10T11:41:28.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.660 7f43260da700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f43201894e0 0x7f4320188990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:28.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.660 7f43260da700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f43201894e0 0x7f4320188990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:55520/0 (socket says 172.21.2.112:55520) 2021-02-10T11:41:28.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.660 7f43260da700 1 -- 172.21.2.112:0/591358113 learned_addr learned my addr 172.21.2.112:0/591358113 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:41:28.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.660 7f43268db700 1 --2- 172.21.2.112:0/591358113 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f432008e790 0x7f432008eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:28.665 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.660 7f43258d9700 1 --2- 172.21.2.112:0/591358113 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f43201899e0 0x7f4320189de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:28.665 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.661 7f43260da700 1 -- 172.21.2.112:0/591358113 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f43201899e0 msgr2=0x7f4320189de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:28.665 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.661 7f43260da700 1 --2- 172.21.2.112:0/591358113 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f43201899e0 0x7f4320189de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:28.665 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.661 7f43260da700 1 -- 172.21.2.112:0/591358113 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f432008e790 msgr2=0x7f432008eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:28.665 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.661 7f43260da700 1 --2- 172.21.2.112:0/591358113 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f432008e790 0x7f432008eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:28.665 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.661 7f43260da700 1 -- 172.21.2.112:0/591358113 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f432008f1b0 con 0x7f43201894e0 2021-02-10T11:41:28.666 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.661 7f43260da700 1 --2- 172.21.2.112:0/591358113 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f43201894e0 0x7f4320188990 secure :-1 s=READY pgs=2216 cs=0 l=1 rev1=1 rx=0x7f4310006f10 tx=0x7f431000b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:28.666 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.661 7f4316ffd700 1 -- 172.21.2.112:0/591358113 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4310006020 con 0x7f43201894e0 2021-02-10T11:41:28.666 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.661 7f4316ffd700 1 -- 172.21.2.112:0/591358113 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43100062a0 con 0x7f43201894e0 2021-02-10T11:41:28.666 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.661 7f4316ffd700 1 -- 172.21.2.112:0/591358113 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4310004590 con 0x7f43201894e0 2021-02-10T11:41:28.666 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.661 7f43282be700 1 -- 172.21.2.112:0/591358113 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f432008f1b0 con 0x7f43201894e0 2021-02-10T11:41:28.666 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.661 7f43282be700 1 -- 172.21.2.112:0/591358113 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f432008f1b0 con 0x7f43201894e0 2021-02-10T11:41:28.666 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.662 7f4316ffd700 1 -- 172.21.2.112:0/591358113 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f43100043a0 con 0x7f43201894e0 2021-02-10T11:41:28.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.662 7f4316ffd700 1 --2- 172.21.2.112:0/591358113 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f43080186c0 0x7f430801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:28.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.662 7f4316ffd700 1 -- 172.21.2.112:0/591358113 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f43100327a0 con 0x7f43201894e0 2021-02-10T11:41:28.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.662 7f43258d9700 1 --2- 172.21.2.112:0/591358113 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f43080186c0 0x7f430801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:28.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.663 7f43258d9700 1 --2- 172.21.2.112:0/591358113 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f43080186c0 0x7f430801abc0 crc :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:28.690 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.693 7f43282be700 1 -- 172.21.2.112:0/591358113 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4320082980 con 0x7f43201894e0 2021-02-10T11:41:28.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.696 7f4316ffd700 1 -- 172.21.2.112:0/591358113 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4310006ba0 con 0x7f43201894e0 2021-02-10T11:41:28.826 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.829 7f43282be700 1 -- 172.21.2.112:0/591358113 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f4320082980 con 0x7f43201894e0 2021-02-10T11:41:28.830 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.832 7f4316ffd700 1 -- 172.21.2.112:0/591358113 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.2 v71) v1 ==== 184+0+0 (secure 0 0 0) 0x7f43100043a0 con 0x7f43201894e0 2021-02-10T11:41:28.833 INFO:teuthology.orchestra.run.gibba012.stderr:updated caps for client.2 2021-02-10T11:41:28.835 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.838 7f43282be700 1 -- 172.21.2.112:0/591358113 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f43080186c0 msgr2=0x7f430801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:28.835 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.838 7f43282be700 1 --2- 172.21.2.112:0/591358113 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f43080186c0 0x7f430801abc0 crc :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:28.836 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.838 7f43282be700 1 -- 172.21.2.112:0/591358113 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f43201894e0 msgr2=0x7f4320188990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:28.836 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.838 7f43282be700 1 --2- 172.21.2.112:0/591358113 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f43201894e0 0x7f4320188990 secure :-1 s=READY pgs=2216 cs=0 l=1 rev1=1 rx=0x7f4310006f10 tx=0x7f431000b990).stop 2021-02-10T11:41:28.836 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.838 7f43282be700 1 -- 172.21.2.112:0/591358113 shutdown_connections 2021-02-10T11:41:28.836 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.838 7f43282be700 1 --2- 172.21.2.112:0/591358113 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f43080186c0 0x7f430801abc0 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:28.836 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.838 7f43282be700 1 --2- 172.21.2.112:0/591358113 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f432008e790 0x7f432008eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:28.836 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.838 7f43282be700 1 --2- 172.21.2.112:0/591358113 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f43201899e0 0x7f4320189de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:28.836 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.838 7f43282be700 1 --2- 172.21.2.112:0/591358113 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f43201894e0 0x7f4320188990 unknown :-1 s=CLOSED pgs=2216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:28.836 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.838 7f43282be700 1 -- 172.21.2.112:0/591358113 >> 172.21.2.112:0/591358113 conn(0x7f432009eba0 msgr2=0x7f43200b5ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:28.836 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.838 7f43282be700 1 -- 172.21.2.112:0/591358113 shutdown_connections 2021-02-10T11:41:28.836 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:28.838 7f43282be700 1 -- 172.21.2.112:0/591358113 wait complete. 2021-02-10T11:41:28.844 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.3 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:41:29.284 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.270 7f11422c0700 1 Processor -- start 2021-02-10T11:41:29.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.270 7f11422c0700 1 -- start start 2021-02-10T11:41:29.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.271 7f11422c0700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f113c0d1c80 0x7f113c0cef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:29.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.271 7f11422c0700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f113c0d0180 con 0x7f113c0cf590 2021-02-10T11:41:29.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.271 7f11422c0700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f113c0fe1d0 con 0x7f113c0d1880 2021-02-10T11:41:29.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.271 7f11422c0700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f113c1b0cc0 con 0x7f113c0d1c80 2021-02-10T11:41:29.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.271 7f113b7fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f113c0d1c80 0x7f113c0cef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:29.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.271 7f113b7fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f113c0d1c80 0x7f113c0cef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:55528/0 (socket says 172.21.2.112:55528) 2021-02-10T11:41:29.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.271 7f11408dd700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f113c0cf590 0x7f113c0cf990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:54988/0 (socket says 172.21.2.112:54988) 2021-02-10T11:41:29.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.271 7f113b7fe700 1 -- 172.21.2.112:0/3265195163 learned_addr learned my addr 172.21.2.112:0/3265195163 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:41:29.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.271 7f113b7fe700 1 -- 172.21.2.112:0/3265195163 >> v1:172.21.2.120:6789/0 conn(0x7f113c0d1880 legacy=0x7f113c1b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:29.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.271 7f113b7fe700 1 -- 172.21.2.112:0/3265195163 >> v1:172.21.2.112:6789/0 conn(0x7f113c0cf590 legacy=0x7f113c0cf990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:29.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.271 7f113b7fe700 1 -- 172.21.2.112:0/3265195163 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f113c109a70 con 0x7f113c0d1c80 2021-02-10T11:41:29.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.272 7f113b7fe700 1 --2- 172.21.2.112:0/3265195163 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f113c0d1c80 0x7f113c0cef70 secure :-1 s=READY pgs=2218 cs=0 l=1 rev1=1 rx=0x7f1130006700 tx=0x7f1130009b80).ready entity=mon.0 client_cookie=7e3b230b4008cd8d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:29.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.272 7f113affd700 1 -- 172.21.2.112:0/3265195163 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f113000c0c0 con 0x7f113c0d1c80 2021-02-10T11:41:29.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.272 7f113affd700 1 -- 172.21.2.112:0/3265195163 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f113000c340 con 0x7f113c0d1c80 2021-02-10T11:41:29.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.272 7f113affd700 1 -- 172.21.2.112:0/3265195163 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f113000c860 con 0x7f113c0d1c80 2021-02-10T11:41:29.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.272 7f11422c0700 1 -- 172.21.2.112:0/3265195163 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f113c0d1c80 msgr2=0x7f113c0cef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:29.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.272 7f11422c0700 1 --2- 172.21.2.112:0/3265195163 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f113c0d1c80 0x7f113c0cef70 secure :-1 s=READY pgs=2218 cs=0 l=1 rev1=1 rx=0x7f1130006700 tx=0x7f1130009b80).stop 2021-02-10T11:41:29.287 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.272 7f11422c0700 1 -- 172.21.2.112:0/3265195163 shutdown_connections 2021-02-10T11:41:29.287 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.272 7f11422c0700 1 --2- 172.21.2.112:0/3265195163 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f113c0d1c80 0x7f113c0cef70 unknown :-1 s=CLOSED pgs=2218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:29.287 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.272 7f11422c0700 1 -- 172.21.2.112:0/3265195163 >> 172.21.2.112:0/3265195163 conn(0x7f113c0ce430 msgr2=0x7f113c0d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:29.287 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.273 7f11422c0700 1 -- 172.21.2.112:0/3265195163 shutdown_connections 2021-02-10T11:41:29.287 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.273 7f11422c0700 1 -- 172.21.2.112:0/3265195163 wait complete. 2021-02-10T11:41:29.287 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.287 7f11422c0700 1 Processor -- start 2021-02-10T11:41:29.287 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.287 7f11422c0700 1 -- start start 2021-02-10T11:41:29.288 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.287 7f11422c0700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f113c1b5c00 0x7f113c0c08a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:29.288 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.287 7f11422c0700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f113c1b50d0 0x7f113c1b54d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:29.288 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.287 7f11422c0700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f113c0c1680 0x7f113c0c1a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:29.288 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.287 7f11422c0700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f113c0c1100 con 0x7f113c1b50d0 2021-02-10T11:41:29.289 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.287 7f11422c0700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f113c1b6000 con 0x7f113c1b5c00 2021-02-10T11:41:29.289 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.287 7f11422c0700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f113c0c20a0 con 0x7f113c0c1680 2021-02-10T11:41:29.289 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.288 7f113b7fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f113c1b50d0 0x7f113c1b54d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:29.289 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.288 7f113b7fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f113c1b50d0 0x7f113c1b54d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:55530/0 (socket says 172.21.2.112:55530) 2021-02-10T11:41:29.289 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.288 7f113b7fe700 1 -- 172.21.2.112:0/1362894281 learned_addr learned my addr 172.21.2.112:0/1362894281 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:41:29.289 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.288 7f113b7fe700 1 -- 172.21.2.112:0/1362894281 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f113c0c1680 msgr2=0x7f113c0c1a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:29.289 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.288 7f113b7fe700 1 --2- 172.21.2.112:0/1362894281 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f113c0c1680 0x7f113c0c1a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:29.289 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.288 7f113b7fe700 1 -- 172.21.2.112:0/1362894281 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f113c1b5c00 msgr2=0x7f113c0c08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:29.289 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.288 7f113b7fe700 1 --2- 172.21.2.112:0/1362894281 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f113c1b5c00 0x7f113c0c08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:29.289 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.288 7f113b7fe700 1 -- 172.21.2.112:0/1362894281 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f113c0c22c0 con 0x7f113c1b50d0 2021-02-10T11:41:29.289 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.288 7f113b7fe700 1 --2- 172.21.2.112:0/1362894281 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f113c1b50d0 0x7f113c1b54d0 secure :-1 s=READY pgs=2219 cs=0 l=1 rev1=1 rx=0x7f113000b5e0 tx=0x7f113000b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:29.289 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.288 7f1138ff9700 1 -- 172.21.2.112:0/1362894281 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f113000bd70 con 0x7f113c1b50d0 2021-02-10T11:41:29.290 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.288 7f1138ff9700 1 -- 172.21.2.112:0/1362894281 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1130002030 con 0x7f113c1b50d0 2021-02-10T11:41:29.290 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.288 7f11422c0700 1 -- 172.21.2.112:0/1362894281 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f113c0c22c0 con 0x7f113c1b50d0 2021-02-10T11:41:29.290 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.288 7f1138ff9700 1 -- 172.21.2.112:0/1362894281 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f11300024d0 con 0x7f113c1b50d0 2021-02-10T11:41:29.290 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.288 7f11422c0700 1 -- 172.21.2.112:0/1362894281 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f113c0c2850 con 0x7f113c1b50d0 2021-02-10T11:41:29.290 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.289 7f1138ff9700 1 -- 172.21.2.112:0/1362894281 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1130000c90 con 0x7f113c1b50d0 2021-02-10T11:41:29.290 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.289 7f1138ff9700 1 --2- 172.21.2.112:0/1362894281 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f111c0186c0 0x7f111c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:29.290 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.289 7f1138ff9700 1 -- 172.21.2.112:0/1362894281 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f1130032590 con 0x7f113c1b50d0 2021-02-10T11:41:29.290 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.289 7f113bfff700 1 --2- 172.21.2.112:0/1362894281 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f111c0186c0 0x7f111c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:29.290 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.290 7f113bfff700 1 --2- 172.21.2.112:0/1362894281 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f111c0186c0 0x7f111c01abc0 crc :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:29.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.321 7f11422c0700 1 -- 172.21.2.112:0/1362894281 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f113c0c0ec0 con 0x7f113c1b50d0 2021-02-10T11:41:29.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.323 7f1138ff9700 1 -- 172.21.2.112:0/1362894281 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f113002df20 con 0x7f113c1b50d0 2021-02-10T11:41:29.449 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.451 7f11422c0700 1 -- 172.21.2.112:0/1362894281 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f113c0c0ec0 con 0x7f113c1b50d0 2021-02-10T11:41:29.452 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.455 7f1138ff9700 1 -- 172.21.2.112:0/1362894281 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.3 v72) v1 ==== 184+0+0 (secure 0 0 0) 0x7f113002df20 con 0x7f113c1b50d0 2021-02-10T11:41:29.456 INFO:teuthology.orchestra.run.gibba012.stderr:updated caps for client.3 2021-02-10T11:41:29.458 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.460 7f11422c0700 1 -- 172.21.2.112:0/1362894281 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f111c0186c0 msgr2=0x7f111c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:29.458 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.460 7f11422c0700 1 --2- 172.21.2.112:0/1362894281 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f111c0186c0 0x7f111c01abc0 crc :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:29.458 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.460 7f11422c0700 1 -- 172.21.2.112:0/1362894281 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f113c1b50d0 msgr2=0x7f113c1b54d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:29.458 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.461 7f11422c0700 1 --2- 172.21.2.112:0/1362894281 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f113c1b50d0 0x7f113c1b54d0 secure :-1 s=READY pgs=2219 cs=0 l=1 rev1=1 rx=0x7f113000b5e0 tx=0x7f113000b980).stop 2021-02-10T11:41:29.458 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.461 7f11422c0700 1 -- 172.21.2.112:0/1362894281 shutdown_connections 2021-02-10T11:41:29.458 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.461 7f11422c0700 1 --2- 172.21.2.112:0/1362894281 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f111c0186c0 0x7f111c01abc0 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:29.458 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.461 7f11422c0700 1 --2- 172.21.2.112:0/1362894281 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f113c0c1680 0x7f113c0c1a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:29.458 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.461 7f11422c0700 1 --2- 172.21.2.112:0/1362894281 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f113c1b50d0 0x7f113c1b54d0 unknown :-1 s=CLOSED pgs=2219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:29.458 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.461 7f11422c0700 1 --2- 172.21.2.112:0/1362894281 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f113c1b5c00 0x7f113c0c08a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:29.458 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.461 7f11422c0700 1 -- 172.21.2.112:0/1362894281 >> 172.21.2.112:0/1362894281 conn(0x7f113c092af0 msgr2=0x7f113c0fe180 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:29.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.461 7f11422c0700 1 -- 172.21.2.112:0/1362894281 shutdown_connections 2021-02-10T11:41:29.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.461 7f11422c0700 1 -- 172.21.2.112:0/1362894281 wait complete. 2021-02-10T11:41:29.467 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:41:29.893 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.880 7f9340998700 1 Processor -- start 2021-02-10T11:41:29.894 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.880 7f9340998700 1 -- start start 2021-02-10T11:41:29.894 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.881 7f9340998700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f933c0d1c70 0x7f933c0f20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:29.894 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.881 7f9340998700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f933c0f0b10 con 0x7f933c0d0340 2021-02-10T11:41:29.894 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.881 7f9340998700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f933c1b0be0 con 0x7f933c0d1870 2021-02-10T11:41:29.894 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.881 7f9340998700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f933c0e5ed0 con 0x7f933c0d1c70 2021-02-10T11:41:29.894 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.881 7f933ae1d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f933c0d1c70 0x7f933c0f20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:29.894 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.881 7f933ae1d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f933c0d1c70 0x7f933c0f20a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:55542/0 (socket says 172.21.2.112:55542) 2021-02-10T11:41:29.894 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.881 7f933ae1d700 1 -- 172.21.2.112:0/1356509543 learned_addr learned my addr 172.21.2.112:0/1356509543 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:41:29.894 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.881 7f933ae1d700 1 -- 172.21.2.112:0/1356509543 >> v1:172.21.2.120:6789/0 conn(0x7f933c0d1870 legacy=0x7f933c1b03f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:41:29.894 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.881 7f933ae1d700 1 -- 172.21.2.112:0/1356509543 >> v1:172.21.2.112:6789/0 conn(0x7f933c0d0340 legacy=0x7f933c0ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:29.894 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.881 7f933ae1d700 1 -- 172.21.2.112:0/1356509543 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f933c0efe30 con 0x7f933c0d1c70 2021-02-10T11:41:29.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.881 7f933ae1d700 1 --2- 172.21.2.112:0/1356509543 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f933c0d1c70 0x7f933c0f20a0 secure :-1 s=READY pgs=2221 cs=0 l=1 rev1=1 rx=0x7f9334006f00 tx=0x7f9334009f10).ready entity=mon.0 client_cookie=faa3f43dd9642727 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:29.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.881 7f933961a700 1 -- 172.21.2.112:0/1356509543 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f933400b3b0 con 0x7f933c0d1c70 2021-02-10T11:41:29.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.881 7f933961a700 1 -- 172.21.2.112:0/1356509543 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f933400b630 con 0x7f933c0d1c70 2021-02-10T11:41:29.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.881 7f933961a700 1 -- 172.21.2.112:0/1356509543 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f933400bb50 con 0x7f933c0d1c70 2021-02-10T11:41:29.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.881 7f9340998700 1 -- 172.21.2.112:0/1356509543 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f933c0d1c70 msgr2=0x7f933c0f20a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:29.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.881 7f9340998700 1 --2- 172.21.2.112:0/1356509543 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f933c0d1c70 0x7f933c0f20a0 secure :-1 s=READY pgs=2221 cs=0 l=1 rev1=1 rx=0x7f9334006f00 tx=0x7f9334009f10).stop 2021-02-10T11:41:29.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.882 7f9340998700 1 -- 172.21.2.112:0/1356509543 shutdown_connections 2021-02-10T11:41:29.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.882 7f9340998700 1 --2- 172.21.2.112:0/1356509543 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f933c0d1c70 0x7f933c0f20a0 unknown :-1 s=CLOSED pgs=2221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:29.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.882 7f9340998700 1 -- 172.21.2.112:0/1356509543 >> 172.21.2.112:0/1356509543 conn(0x7f933c0ce450 msgr2=0x7f933c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:29.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.882 7f9340998700 1 -- 172.21.2.112:0/1356509543 shutdown_connections 2021-02-10T11:41:29.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.882 7f9340998700 1 -- 172.21.2.112:0/1356509543 wait complete. 2021-02-10T11:41:29.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.896 7f9340998700 1 Processor -- start 2021-02-10T11:41:29.897 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.897 7f9340998700 1 -- start start 2021-02-10T11:41:29.897 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.897 7f9340998700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f933c0a35b0 0x7f933c0a2900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:29.897 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.897 7f9340998700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f933c1c6350 0x7f933c0b17a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:29.897 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.897 7f9340998700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f933c1c6750 0x7f933c0b1dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:29.897 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.897 7f9340998700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f933c0a3030 con 0x7f933c1c6750 2021-02-10T11:41:29.897 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.897 7f9340998700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f933c0f2c00 con 0x7f933c0a35b0 2021-02-10T11:41:29.897 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.897 7f9340998700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f933c09bfc0 con 0x7f933c1c6350 2021-02-10T11:41:29.897 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.897 7f933ae1d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f933c1c6750 0x7f933c0b1dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:29.897 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.897 7f933ae1d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f933c1c6750 0x7f933c0b1dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:55548/0 (socket says 172.21.2.112:55548) 2021-02-10T11:41:29.897 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.897 7f933ae1d700 1 -- 172.21.2.112:0/21292366 learned_addr learned my addr 172.21.2.112:0/21292366 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:41:29.897 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.897 7f933ae1d700 1 -- 172.21.2.112:0/21292366 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f933c1c6350 msgr2=0x7f933c0b17a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:41:29.897 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.897 7f933ae1d700 1 --2- 172.21.2.112:0/21292366 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f933c1c6350 0x7f933c0b17a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:29.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.897 7f933ae1d700 1 -- 172.21.2.112:0/21292366 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f933c0a35b0 msgr2=0x7f933c0a2900 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:41:29.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.897 7f933ae1d700 1 --2- 172.21.2.112:0/21292366 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f933c0a35b0 0x7f933c0a2900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:29.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.897 7f933ae1d700 1 -- 172.21.2.112:0/21292366 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f933c09c1e0 con 0x7f933c1c6750 2021-02-10T11:41:29.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.897 7f933ae1d700 1 --2- 172.21.2.112:0/21292366 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f933c1c6750 0x7f933c0b1dc0 secure :-1 s=READY pgs=2222 cs=0 l=1 rev1=1 rx=0x7f933400b9b0 tx=0x7f933400bd50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:29.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.898 7f932affd700 1 -- 172.21.2.112:0/21292366 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9334005820 con 0x7f933c1c6750 2021-02-10T11:41:29.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.898 7f932affd700 1 -- 172.21.2.112:0/21292366 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9334005aa0 con 0x7f933c1c6750 2021-02-10T11:41:29.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.898 7f932affd700 1 -- 172.21.2.112:0/21292366 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9334006030 con 0x7f933c1c6750 2021-02-10T11:41:29.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.898 7f9340998700 1 -- 172.21.2.112:0/21292366 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f933c09c1e0 con 0x7f933c1c6750 2021-02-10T11:41:29.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.898 7f9340998700 1 -- 172.21.2.112:0/21292366 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f933c09c1e0 con 0x7f933c1c6750 2021-02-10T11:41:29.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.898 7f932affd700 1 -- 172.21.2.112:0/21292366 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9334006030 con 0x7f933c1c6750 2021-02-10T11:41:29.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.898 7f932affd700 1 --2- 172.21.2.112:0/21292366 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f93200186c0 0x7f932001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:29.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.898 7f932affd700 1 -- 172.21.2.112:0/21292366 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f9334032b10 con 0x7f933c1c6750 2021-02-10T11:41:29.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.899 7f933a61c700 1 --2- 172.21.2.112:0/21292366 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f93200186c0 0x7f932001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:29.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.900 7f933a61c700 1 --2- 172.21.2.112:0/21292366 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f93200186c0 0x7f932001abc0 crc :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:29.928 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.930 7f9340998700 1 -- 172.21.2.112:0/21292366 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f933c0e7df0 con 0x7f933c1c6750 2021-02-10T11:41:29.930 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:29.933 7f932affd700 1 -- 172.21.2.112:0/21292366 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9334009450 con 0x7f933c1c6750 2021-02-10T11:41:30.058 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:30.060 7f9340998700 1 -- 172.21.2.112:0/21292366 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f933c0ce450 con 0x7f933c1c6750 2021-02-10T11:41:30.058 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:30.061 7f932affd700 1 -- 172.21.2.112:0/21292366 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) v1 ==== 94+0+2098 (secure 0 0 0) 0x7f9334006030 con 0x7f933c1c6750 2021-02-10T11:41:30.059 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 97 2021-02-10T11:41:30.059 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:41:30.059 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":97,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10671,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/668146927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":668146927},{"type":"v1","addr":"172.21.2.112:6833","nonce":668146927}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":93}],"filesystems":[{"mdsmap":{"epoch":97,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:41:23.731569","modified":"2021-02-10 11:41:25.740952","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8008},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8008":{"gid":8008,"name":"b","rank":0,"incarnation":94,"state":"up:active","state_seq":3,"addr":"172.21.2.120:6835/862287497","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":862287497},{"type":"v1","addr":"172.21.2.120:6835","nonce":862287497}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":12}]} 2021-02-10T11:41:30.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:30.063 7f9340998700 1 -- 172.21.2.112:0/21292366 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f93200186c0 msgr2=0x7f932001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:30.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:30.063 7f9340998700 1 --2- 172.21.2.112:0/21292366 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f93200186c0 0x7f932001abc0 crc :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:30.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:30.063 7f9340998700 1 -- 172.21.2.112:0/21292366 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f933c1c6750 msgr2=0x7f933c0b1dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:30.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:30.063 7f9340998700 1 --2- 172.21.2.112:0/21292366 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f933c1c6750 0x7f933c0b1dc0 secure :-1 s=READY pgs=2222 cs=0 l=1 rev1=1 rx=0x7f933400b9b0 tx=0x7f933400bd50).stop 2021-02-10T11:41:30.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:30.063 7f9340998700 1 -- 172.21.2.112:0/21292366 shutdown_connections 2021-02-10T11:41:30.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:30.063 7f9340998700 1 --2- 172.21.2.112:0/21292366 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f93200186c0 0x7f932001abc0 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:30.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:30.063 7f9340998700 1 --2- 172.21.2.112:0/21292366 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f933c1c6750 0x7f933c0b1dc0 unknown :-1 s=CLOSED pgs=2222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:30.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:30.063 7f9340998700 1 --2- 172.21.2.112:0/21292366 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f933c1c6350 0x7f933c0b17a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:30.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:30.063 7f9340998700 1 --2- 172.21.2.112:0/21292366 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f933c0a35b0 0x7f933c0a2900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:30.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:30.063 7f9340998700 1 -- 172.21.2.112:0/21292366 >> 172.21.2.112:0/21292366 conn(0x7f933c0a0030 msgr2=0x7f933c0ec020 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:30.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:30.063 7f9340998700 1 -- 172.21.2.112:0/21292366 shutdown_connections 2021-02-10T11:41:30.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:30.064 7f9340998700 1 -- 172.21.2.112:0/21292366 wait complete. 2021-02-10T11:41:30.270 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 97, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:41:23.731569', 'modified': '2021-02-10 11:41:25.740952', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8008}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8008': {'gid': 8008, 'name': 'b', 'rank': 0, 'incarnation': 94, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.120:6835/862287497', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.120:6834', 'nonce': 862287497}, {'type': 'v1', 'addr': '172.21.2.120:6835', 'nonce': 862287497}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [25], 'metadata_pool': 24, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:41:30.271 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:41:30.271 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:41:30.669 INFO:teuthology.orchestra.run.gibba020.stdout:{ 2021-02-10T11:41:30.669 INFO:teuthology.orchestra.run.gibba020.stdout: "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", 2021-02-10T11:41:30.669 INFO:teuthology.orchestra.run.gibba020.stdout: "whoami": 0, 2021-02-10T11:41:30.669 INFO:teuthology.orchestra.run.gibba020.stdout: "id": 8008, 2021-02-10T11:41:30.670 INFO:teuthology.orchestra.run.gibba020.stdout: "want_state": "up:active", 2021-02-10T11:41:30.670 INFO:teuthology.orchestra.run.gibba020.stdout: "state": "up:active", 2021-02-10T11:41:30.670 INFO:teuthology.orchestra.run.gibba020.stdout: "rank_uptime": 6.9385291679999996, 2021-02-10T11:41:30.670 INFO:teuthology.orchestra.run.gibba020.stdout: "mdsmap_epoch": 97, 2021-02-10T11:41:30.670 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch": 117, 2021-02-10T11:41:30.670 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch_barrier": 117, 2021-02-10T11:41:30.670 INFO:teuthology.orchestra.run.gibba020.stdout: "uptime": 12.585839453 2021-02-10T11:41:30.670 INFO:teuthology.orchestra.run.gibba020.stdout:} 2021-02-10T11:41:30.673 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", "whoami": 0, "id": 8008, "want_state": "up:active", "state": "up:active", "rank_uptime": 6.9385291679999996, "mdsmap_epoch": 97, "osdmap_epoch": 117, "osdmap_epoch_barrier": 117, "uptime": 12.585839453 } 2021-02-10T11:41:30.673 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:41:31.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.089 7f89887c8700 1 Processor -- start 2021-02-10T11:41:31.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.089 7f89887c8700 1 -- start start 2021-02-10T11:41:31.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.090 7f89887c8700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f89800d1870 0x7f89801b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:31.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.090 7f89887c8700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f89801b0a10 0x7f89800cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:31.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.090 7f89887c8700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f89800d1c70 0x7f89800cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:31.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.090 7f89887c8700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89800fe200 con 0x7f89800d1c70 2021-02-10T11:41:31.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.090 7f89887c8700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89800e5f10 con 0x7f89800d1870 2021-02-10T11:41:31.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.090 7f89887c8700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89800f0b50 con 0x7f89801b0a10 2021-02-10T11:41:31.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.090 7f8986de5700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f89800d1c70 0x7f89800cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:31.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.090 7f8986de5700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f89800d1c70 0x7f89800cf950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:55552/0 (socket says 172.21.2.112:55552) 2021-02-10T11:41:31.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.090 7f8986de5700 1 -- 172.21.2.112:0/2438869216 learned_addr learned my addr 172.21.2.112:0/2438869216 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:41:31.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.090 7f8986de5700 1 -- 172.21.2.112:0/2438869216 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f89801b0a10 msgr2=0x7f89800cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:31.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.090 7f8986de5700 1 --2- 172.21.2.112:0/2438869216 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f89801b0a10 0x7f89800cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:31.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.090 7f8986de5700 1 -- 172.21.2.112:0/2438869216 >> v2:172.21.2.120:3300/0 conn(0x7f89800d1870 msgr2=0x7f89801b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:31.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.090 7f8986de5700 1 --2- 172.21.2.112:0/2438869216 >> v2:172.21.2.120:3300/0 conn(0x7f89800d1870 0x7f89801b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:31.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.090 7f8986de5700 1 -- 172.21.2.112:0/2438869216 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89800cff70 con 0x7f89800d1c70 2021-02-10T11:41:31.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.090 7f8986de5700 1 --2- 172.21.2.112:0/2438869216 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f89800d1c70 0x7f89800cf950 secure :-1 s=READY pgs=2223 cs=0 l=1 rev1=1 rx=0x7f8978006f00 tx=0x7f8978009f10).ready entity=mon.0 client_cookie=34af4a4bedb1174f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:31.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.091 7f89855e2700 1 -- 172.21.2.112:0/2438869216 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f897800b3b0 con 0x7f89800d1c70 2021-02-10T11:41:31.109 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.091 7f89855e2700 1 -- 172.21.2.112:0/2438869216 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f897800b630 con 0x7f89800d1c70 2021-02-10T11:41:31.109 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.091 7f89855e2700 1 -- 172.21.2.112:0/2438869216 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f897800bb50 con 0x7f89800d1c70 2021-02-10T11:41:31.109 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.091 7f89887c8700 1 -- 172.21.2.112:0/2438869216 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f89800d1c70 msgr2=0x7f89800cf950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:31.109 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.091 7f89887c8700 1 --2- 172.21.2.112:0/2438869216 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f89800d1c70 0x7f89800cf950 secure :-1 s=READY pgs=2223 cs=0 l=1 rev1=1 rx=0x7f8978006f00 tx=0x7f8978009f10).stop 2021-02-10T11:41:31.109 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.091 7f89887c8700 1 -- 172.21.2.112:0/2438869216 shutdown_connections 2021-02-10T11:41:31.110 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.091 7f89887c8700 1 --2- 172.21.2.112:0/2438869216 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f89800d1c70 0x7f89800cf950 unknown :-1 s=CLOSED pgs=2223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:31.110 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.091 7f89887c8700 1 --2- 172.21.2.112:0/2438869216 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f89801b0a10 0x7f89800cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:31.111 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.091 7f89887c8700 1 --2- 172.21.2.112:0/2438869216 >> v2:172.21.2.120:3300/0 conn(0x7f89800d1870 0x7f89801b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:31.111 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.091 7f89887c8700 1 -- 172.21.2.112:0/2438869216 >> 172.21.2.112:0/2438869216 conn(0x7f89800ce450 msgr2=0x7f89800d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:31.111 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.091 7f89887c8700 1 -- 172.21.2.112:0/2438869216 shutdown_connections 2021-02-10T11:41:31.111 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.091 7f89887c8700 1 -- 172.21.2.112:0/2438869216 wait complete. 2021-02-10T11:41:31.111 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.106 7f89887c8700 1 Processor -- start 2021-02-10T11:41:31.111 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.106 7f89887c8700 1 -- start start 2021-02-10T11:41:31.111 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.106 7f89887c8700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f898018e930 0x7f89800aa4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:31.111 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.106 7f89887c8700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f89800aaaf0 0x7f8980093700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:31.111 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.106 7f89887c8700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f898018ed30 0x7f8980093d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:31.111 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.106 7f89887c8700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89800944e0 con 0x7f898018e930 2021-02-10T11:41:31.112 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.106 7f89887c8700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f898008a590 con 0x7f89800aaaf0 2021-02-10T11:41:31.112 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.106 7f89887c8700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f898008a770 con 0x7f898018ed30 2021-02-10T11:41:31.112 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.106 7f89865e4700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f898018e930 0x7f89800aa4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:31.112 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.106 7f89865e4700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f898018e930 0x7f89800aa4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:55562/0 (socket says 172.21.2.112:55562) 2021-02-10T11:41:31.112 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.106 7f89865e4700 1 -- 172.21.2.112:0/935378471 learned_addr learned my addr 172.21.2.112:0/935378471 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:41:31.112 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.107 7f89865e4700 1 -- 172.21.2.112:0/935378471 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f898018ed30 msgr2=0x7f8980093d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:31.112 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.107 7f89865e4700 1 --2- 172.21.2.112:0/935378471 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f898018ed30 0x7f8980093d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:31.112 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.107 7f89865e4700 1 -- 172.21.2.112:0/935378471 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f89800aaaf0 msgr2=0x7f8980093700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:31.112 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.107 7f89865e4700 1 --2- 172.21.2.112:0/935378471 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f89800aaaf0 0x7f8980093700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:31.112 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.107 7f89865e4700 1 -- 172.21.2.112:0/935378471 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f898008a990 con 0x7f898018e930 2021-02-10T11:41:31.112 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.107 7f89865e4700 1 --2- 172.21.2.112:0/935378471 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f898018e930 0x7f89800aa4d0 secure :-1 s=READY pgs=2224 cs=0 l=1 rev1=1 rx=0x7f8974006f00 tx=0x7f897400b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:31.112 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.107 7f8972ffd700 1 -- 172.21.2.112:0/935378471 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f897400bd30 con 0x7f898018e930 2021-02-10T11:41:31.112 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.107 7f8972ffd700 1 -- 172.21.2.112:0/935378471 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8974000c90 con 0x7f898018e930 2021-02-10T11:41:31.113 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.107 7f8972ffd700 1 -- 172.21.2.112:0/935378471 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f89740022a0 con 0x7f898018e930 2021-02-10T11:41:31.113 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.107 7f89887c8700 1 -- 172.21.2.112:0/935378471 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f898008a990 con 0x7f898018e930 2021-02-10T11:41:31.113 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.107 7f89887c8700 1 -- 172.21.2.112:0/935378471 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f898008af20 con 0x7f898018e930 2021-02-10T11:41:31.113 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.108 7f8972ffd700 1 -- 172.21.2.112:0/935378471 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8974005310 con 0x7f898018e930 2021-02-10T11:41:31.113 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.108 7f8972ffd700 1 --2- 172.21.2.112:0/935378471 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f89680186c0 0x7f896801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:31.114 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.108 7f8972ffd700 1 -- 172.21.2.112:0/935378471 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f8974032550 con 0x7f898018e930 2021-02-10T11:41:31.114 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.108 7f8985de3700 1 --2- 172.21.2.112:0/935378471 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f89680186c0 0x7f896801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:31.114 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.109 7f8985de3700 1 --2- 172.21.2.112:0/935378471 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f89680186c0 0x7f896801abc0 crc :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:31.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.140 7f89887c8700 1 -- 172.21.2.112:0/935378471 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89800ee060 con 0x7f898018e930 2021-02-10T11:41:31.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.142 7f8972ffd700 1 -- 172.21.2.112:0/935378471 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8974006b90 con 0x7f898018e930 2021-02-10T11:41:31.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.266 7f89887c8700 1 -- 172.21.2.112:0/935378471 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f89800ee060 con 0x7f898018e930 2021-02-10T11:41:31.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.266 7f8972ffd700 1 -- 172.21.2.112:0/935378471 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v117) v1 ==== 74+0+14007 (secure 0 0 0) 0x7f8974005310 con 0x7f898018e930 2021-02-10T11:41:31.264 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:41:31.264 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":117,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:41:24.733802","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:41:21.697492","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"116","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":25,"pool_name":"cephfs_data","create_time":"2021-02-10 11:41:22.721757","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"117","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:41:31.266 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.268 7f89887c8700 1 -- 172.21.2.112:0/935378471 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f89680186c0 msgr2=0x7f896801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:31.266 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.268 7f89887c8700 1 --2- 172.21.2.112:0/935378471 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f89680186c0 0x7f896801abc0 crc :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:31.266 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.268 7f89887c8700 1 -- 172.21.2.112:0/935378471 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f898018e930 msgr2=0x7f89800aa4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:31.266 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.268 7f89887c8700 1 --2- 172.21.2.112:0/935378471 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f898018e930 0x7f89800aa4d0 secure :-1 s=READY pgs=2224 cs=0 l=1 rev1=1 rx=0x7f8974006f00 tx=0x7f897400b940).stop 2021-02-10T11:41:31.266 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.268 7f89887c8700 1 -- 172.21.2.112:0/935378471 shutdown_connections 2021-02-10T11:41:31.266 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.268 7f89887c8700 1 --2- 172.21.2.112:0/935378471 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f89680186c0 0x7f896801abc0 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:31.266 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.268 7f89887c8700 1 --2- 172.21.2.112:0/935378471 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f89800aaaf0 0x7f8980093700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:31.266 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.268 7f89887c8700 1 --2- 172.21.2.112:0/935378471 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f898018ed30 0x7f8980093d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:31.266 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.268 7f89887c8700 1 --2- 172.21.2.112:0/935378471 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f898018e930 0x7f89800aa4d0 unknown :-1 s=CLOSED pgs=2224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:31.266 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.268 7f89887c8700 1 -- 172.21.2.112:0/935378471 >> 172.21.2.112:0/935378471 conn(0x7f898019f8f0 msgr2=0x7f898018c260 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:31.267 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.268 7f89887c8700 1 -- 172.21.2.112:0/935378471 shutdown_connections 2021-02-10T11:41:31.267 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.268 7f89887c8700 1 -- 172.21.2.112:0/935378471 wait complete. 2021-02-10T11:41:31.274 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:13:57.508758', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 24, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:41:21.697492', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '116', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 25, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:41:22.721757', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '117', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:41:31.275 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:41:31.708 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.693 7fabd0ccd700 1 Processor -- start 2021-02-10T11:41:31.708 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.693 7fabd0ccd700 1 -- start start 2021-02-10T11:41:31.708 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.693 7fabd0ccd700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fabcc0d1870 0x7fabcc1b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:31.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.693 7fabd0ccd700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabcc109a60 con 0x7fabcc0cff70 2021-02-10T11:41:31.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.693 7fabd0ccd700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabcc088070 con 0x7fabcc1b0ae0 2021-02-10T11:41:31.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.693 7fabd0ccd700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabcc0e5ef0 con 0x7fabcc0d1870 2021-02-10T11:41:31.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.693 7fabcb2e9700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fabcc0cff70 0x7fabcc109130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:55032/0 (socket says 172.21.2.112:55032) 2021-02-10T11:41:31.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.693 7fabcb2e9700 1 -- 172.21.2.112:0/4130511306 learned_addr learned my addr 172.21.2.112:0/4130511306 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:41:31.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.694 7fabc9ae6700 1 -- 172.21.2.112:0/4130511306 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1875464748 0 0) 0x7fabc4001500 con 0x7fabcc0cff70 2021-02-10T11:41:31.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.694 7fabc9ae6700 1 -- 172.21.2.112:0/4130511306 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabb8002fe0 con 0x7fabcc0cff70 2021-02-10T11:41:31.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.694 7fabc9ae6700 1 -- 172.21.2.112:0/4130511306 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2351735444 0 0) 0x7fabc4001460 con 0x7fabcc0cff70 2021-02-10T11:41:31.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.694 7fabc9ae6700 1 -- 172.21.2.112:0/4130511306 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fabcc0d1870 msgr2=0x7fabcc1b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:31.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.694 7fabcaae8700 1 --2- 172.21.2.112:0/4130511306 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fabcc0d1870 0x7fabcc1b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:31.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.694 7fabc9ae6700 1 --2- 172.21.2.112:0/4130511306 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fabcc0d1870 0x7fabcc1b04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:31.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.694 7fabc9ae6700 1 -- 172.21.2.112:0/4130511306 >> v1:172.21.2.120:6789/0 conn(0x7fabcc1b0ae0 legacy=0x7fabcc108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:31.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.694 7fabc9ae6700 1 -- 172.21.2.112:0/4130511306 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabcc0d1c70 con 0x7fabcc0cff70 2021-02-10T11:41:31.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.694 7fabc9ae6700 1 -- 172.21.2.112:0/4130511306 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7fabc4001980 con 0x7fabcc0cff70 2021-02-10T11:41:31.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.694 7fabcaae8700 1 --2- 172.21.2.112:0/4130511306 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fabcc0d1870 0x7fabcc1b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:41:31.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.694 7fabc9ae6700 1 -- 172.21.2.112:0/4130511306 >> v1:172.21.2.112:6789/0 conn(0x7fabcc0cff70 legacy=0x7fabcc109130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:31.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.694 7fabc9ae6700 1 --2- 172.21.2.112:0/4130511306 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fabb8003aa0 0x7fabb8005f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:31.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.694 7fabc9ae6700 1 --2- 172.21.2.112:0/4130511306 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fabb8006800 0x7fabb8008cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:31.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.694 7fabc9ae6700 1 --2- 172.21.2.112:0/4130511306 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fabb80095b0 0x7fabb800baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:31.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.694 7fabc9ae6700 1 -- 172.21.2.112:0/4130511306 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabb800c2d0 con 0x7fabb8003aa0 2021-02-10T11:41:31.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.694 7fabc9ae6700 1 -- 172.21.2.112:0/4130511306 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabb800c4f0 con 0x7fabb8006800 2021-02-10T11:41:31.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.694 7fabc9ae6700 1 -- 172.21.2.112:0/4130511306 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabb800c710 con 0x7fabb80095b0 2021-02-10T11:41:31.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.694 7fabcaae8700 1 --2- 172.21.2.112:0/4130511306 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fabb8003aa0 0x7fabb8005f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:31.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.694 7fabcaae8700 1 -- 172.21.2.112:0/4130511306 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fabb80095b0 msgr2=0x7fabb800baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:31.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.694 7fabcaae8700 1 --2- 172.21.2.112:0/4130511306 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fabb80095b0 0x7fabb800baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:31.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.694 7fabcaae8700 1 -- 172.21.2.112:0/4130511306 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fabb8006800 msgr2=0x7fabb8008cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:31.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.694 7fabcaae8700 1 --2- 172.21.2.112:0/4130511306 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fabb8006800 0x7fabb8008cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:31.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.694 7fabcaae8700 1 -- 172.21.2.112:0/4130511306 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabb800c9b0 con 0x7fabb8003aa0 2021-02-10T11:41:31.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.695 7fabcaae8700 1 --2- 172.21.2.112:0/4130511306 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fabb8003aa0 0x7fabb8005f60 secure :-1 s=READY pgs=2226 cs=0 l=1 rev1=1 rx=0x7fabb4006a90 tx=0x7fabb404da20).ready entity=mon.0 client_cookie=f2ed2c26962a41f2 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:31.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.695 7fabc9ae6700 1 -- 172.21.2.112:0/4130511306 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fabb40009b0 con 0x7fabb8003aa0 2021-02-10T11:41:31.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.695 7fabc9ae6700 1 -- 172.21.2.112:0/4130511306 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabb4002030 con 0x7fabb8003aa0 2021-02-10T11:41:31.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.695 7fabc9ae6700 1 -- 172.21.2.112:0/4130511306 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fabb40024e0 con 0x7fabb8003aa0 2021-02-10T11:41:31.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.695 7fabd0ccd700 1 -- 172.21.2.112:0/4130511306 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fabb8003aa0 msgr2=0x7fabb8005f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:31.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.695 7fabd0ccd700 1 --2- 172.21.2.112:0/4130511306 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fabb8003aa0 0x7fabb8005f60 secure :-1 s=READY pgs=2226 cs=0 l=1 rev1=1 rx=0x7fabb4006a90 tx=0x7fabb404da20).stop 2021-02-10T11:41:31.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.695 7fabcb2e9700 1 -- 172.21.2.112:0/4130511306 reap_dead start 2021-02-10T11:41:31.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.696 7fabd0ccd700 1 -- 172.21.2.112:0/4130511306 shutdown_connections 2021-02-10T11:41:31.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.696 7fabd0ccd700 1 -- 172.21.2.112:0/4130511306 >> 172.21.2.112:0/4130511306 conn(0x7fabcc0ce420 msgr2=0x7fabcc0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:31.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.696 7fabd0ccd700 1 -- 172.21.2.112:0/4130511306 shutdown_connections 2021-02-10T11:41:31.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.696 7fabd0ccd700 1 -- 172.21.2.112:0/4130511306 wait complete. 2021-02-10T11:41:31.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.710 7fabd0ccd700 1 Processor -- start 2021-02-10T11:41:31.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.710 7fabd0ccd700 1 -- start start 2021-02-10T11:41:31.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.710 7fabd0ccd700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fabcc085bc0 0x7fabcc0863e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:31.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.710 7fabd0ccd700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fabcc0954c0 0x7fabcc096300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:31.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.710 7fabd0ccd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fabcc0958c0 0x7fabcc096920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:31.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.710 7fabd0ccd700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabcc095d80 con 0x7fabcc0958c0 2021-02-10T11:41:31.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.710 7fabd0ccd700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabcc0bfa60 con 0x7fabcc085bc0 2021-02-10T11:41:31.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.710 7fabd0ccd700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabcc085fc0 con 0x7fabcc0954c0 2021-02-10T11:41:31.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.711 7fabcb2e9700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fabcc0958c0 0x7fabcc096920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:31.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.711 7fabcb2e9700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fabcc0958c0 0x7fabcc096920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:55578/0 (socket says 172.21.2.112:55578) 2021-02-10T11:41:31.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.711 7fabcb2e9700 1 -- 172.21.2.112:0/4104099444 learned_addr learned my addr 172.21.2.112:0/4104099444 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:41:31.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.711 7fabcb2e9700 1 -- 172.21.2.112:0/4104099444 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fabcc0954c0 msgr2=0x7fabcc096300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:31.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.711 7fabcb2e9700 1 --2- 172.21.2.112:0/4104099444 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fabcc0954c0 0x7fabcc096300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:31.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.711 7fabcb2e9700 1 -- 172.21.2.112:0/4104099444 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fabcc085bc0 msgr2=0x7fabcc0863e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:31.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.711 7fabcb2e9700 1 --2- 172.21.2.112:0/4104099444 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fabcc085bc0 0x7fabcc0863e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:31.714 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.711 7fabcb2e9700 1 -- 172.21.2.112:0/4104099444 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabcc086c10 con 0x7fabcc0958c0 2021-02-10T11:41:31.714 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.711 7fabcb2e9700 1 --2- 172.21.2.112:0/4104099444 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fabcc0958c0 0x7fabcc096920 secure :-1 s=READY pgs=2227 cs=0 l=1 rev1=1 rx=0x7fabc4001640 tx=0x7fabc400aca0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:31.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.711 7fabc37fe700 1 -- 172.21.2.112:0/4104099444 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fabc4004930 con 0x7fabcc0958c0 2021-02-10T11:41:31.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.711 7fabc37fe700 1 -- 172.21.2.112:0/4104099444 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabc4001cb0 con 0x7fabcc0958c0 2021-02-10T11:41:31.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.711 7fabc37fe700 1 -- 172.21.2.112:0/4104099444 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fabc4007020 con 0x7fabcc0958c0 2021-02-10T11:41:31.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.711 7fabd0ccd700 1 -- 172.21.2.112:0/4104099444 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabcc086c10 con 0x7fabcc0958c0 2021-02-10T11:41:31.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.711 7fabd0ccd700 1 -- 172.21.2.112:0/4104099444 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabcc086c10 con 0x7fabcc0958c0 2021-02-10T11:41:31.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.712 7fabc37fe700 1 -- 172.21.2.112:0/4104099444 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fabc4000e30 con 0x7fabcc0958c0 2021-02-10T11:41:31.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.712 7fabc37fe700 1 --2- 172.21.2.112:0/4104099444 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fabb00186c0 0x7fabb001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:31.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.712 7fabc37fe700 1 -- 172.21.2.112:0/4104099444 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fabc4032960 con 0x7fabcc0958c0 2021-02-10T11:41:31.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.712 7fabcaae8700 1 --2- 172.21.2.112:0/4104099444 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fabb00186c0 0x7fabb001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:31.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.713 7fabcaae8700 1 --2- 172.21.2.112:0/4104099444 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fabb00186c0 0x7fabb001abc0 crc :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:31.741 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.743 7fabd0ccd700 1 -- 172.21.2.112:0/4104099444 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabcc0fe1c0 con 0x7fabcc0958c0 2021-02-10T11:41:31.743 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.746 7fabc37fe700 1 -- 172.21.2.112:0/4104099444 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fabc402df00 con 0x7fabcc0958c0 2021-02-10T11:41:31.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.874 7fabd0ccd700 1 -- 172.21.2.112:0/4104099444 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7fabcc0fe1c0 con 0x7fabcc0958c0 2021-02-10T11:41:31.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.874 7fabc37fe700 1 -- 172.21.2.112:0/4104099444 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v117) v1 ==== 92+0+10 (secure 0 0 0) 0x7fabc4000e30 con 0x7fabcc0958c0 2021-02-10T11:41:31.872 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:41:31.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.876 7fabd0ccd700 1 -- 172.21.2.112:0/4104099444 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fabb00186c0 msgr2=0x7fabb001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:31.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.876 7fabd0ccd700 1 --2- 172.21.2.112:0/4104099444 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fabb00186c0 0x7fabb001abc0 crc :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:31.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.876 7fabd0ccd700 1 -- 172.21.2.112:0/4104099444 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fabcc0958c0 msgr2=0x7fabcc096920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:31.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.876 7fabd0ccd700 1 --2- 172.21.2.112:0/4104099444 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fabcc0958c0 0x7fabcc096920 secure :-1 s=READY pgs=2227 cs=0 l=1 rev1=1 rx=0x7fabc4001640 tx=0x7fabc400aca0).stop 2021-02-10T11:41:31.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.876 7fabd0ccd700 1 -- 172.21.2.112:0/4104099444 shutdown_connections 2021-02-10T11:41:31.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.876 7fabd0ccd700 1 --2- 172.21.2.112:0/4104099444 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fabb00186c0 0x7fabb001abc0 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:31.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.876 7fabd0ccd700 1 --2- 172.21.2.112:0/4104099444 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fabcc0958c0 0x7fabcc096920 unknown :-1 s=CLOSED pgs=2227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:31.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.876 7fabd0ccd700 1 --2- 172.21.2.112:0/4104099444 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fabcc0954c0 0x7fabcc096300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:31.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.876 7fabd0ccd700 1 --2- 172.21.2.112:0/4104099444 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fabcc085bc0 0x7fabcc0863e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:31.875 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.876 7fabd0ccd700 1 -- 172.21.2.112:0/4104099444 >> 172.21.2.112:0/4104099444 conn(0x7fabcc0bef00 msgr2=0x7fabcc09d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:31.875 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.876 7fabd0ccd700 1 -- 172.21.2.112:0/4104099444 shutdown_connections 2021-02-10T11:41:31.875 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:31.877 7fabd0ccd700 1 -- 172.21.2.112:0/4104099444 wait complete. 2021-02-10T11:41:31.882 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:41:32.314 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.298 7f08ea53b700 1 Processor -- start 2021-02-10T11:41:32.314 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.298 7f08ea53b700 1 -- start start 2021-02-10T11:41:32.315 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.298 7f08ea53b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f08e40d1870 0x7f08e41b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:32.315 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.298 7f08ea53b700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f08e40cff70 0x7f08e4109130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:32.315 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.298 7f08ea53b700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08e4109a60 con 0x7f08e41b0ae0 2021-02-10T11:41:32.315 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.298 7f08ea53b700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08e4109750 con 0x7f08e40cff70 2021-02-10T11:41:32.315 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.298 7f08ea53b700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08e4088070 con 0x7f08e40d1870 2021-02-10T11:41:32.315 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.300 7f08e3fff700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f08e40d1870 0x7f08e41b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:32.315 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.300 7f08e8b58700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f08e40cff70 0x7f08e4109130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:32.315 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.300 7f08e37fe700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7f08e41b0ae0 0x7f08e4108940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:48606/0 (socket says 172.21.2.112:48606) 2021-02-10T11:41:32.315 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.300 7f08e37fe700 1 -- 172.21.2.112:0/3522972913 learned_addr learned my addr 172.21.2.112:0/3522972913 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:41:32.315 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.300 7f08e8b58700 1 -- 172.21.2.112:0/3522972913 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f08e40d1870 msgr2=0x7f08e41b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:32.315 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.300 7f08e8b58700 1 --2- 172.21.2.112:0/3522972913 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f08e40d1870 0x7f08e41b04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:32.315 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.300 7f08e8b58700 1 -- 172.21.2.112:0/3522972913 >> v1:172.21.2.120:6789/0 conn(0x7f08e41b0ae0 legacy=0x7f08e4108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:32.316 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.300 7f08e8b58700 1 -- 172.21.2.112:0/3522972913 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08e4088260 con 0x7f08e40cff70 2021-02-10T11:41:32.316 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.301 7f08e8b58700 1 --2- 172.21.2.112:0/3522972913 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f08e40cff70 0x7f08e4109130 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 rx=0x7f08dc006e50 tx=0x7f08dc009e70).ready entity=mon.1 client_cookie=7ceaa59fae9688a1 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:32.316 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.301 7f08e2ffd700 1 -- 172.21.2.112:0/3522972913 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f08dc000f90 con 0x7f08e40cff70 2021-02-10T11:41:32.316 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.301 7f08e2ffd700 1 -- 172.21.2.112:0/3522972913 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08dc006790 con 0x7f08e40cff70 2021-02-10T11:41:32.316 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.301 7f08e2ffd700 1 -- 172.21.2.112:0/3522972913 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f08dc00b700 con 0x7f08e40cff70 2021-02-10T11:41:32.316 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.301 7f08ea53b700 1 -- 172.21.2.112:0/3522972913 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f08e40cff70 msgr2=0x7f08e4109130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:32.316 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.301 7f08ea53b700 1 --2- 172.21.2.112:0/3522972913 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f08e40cff70 0x7f08e4109130 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 rx=0x7f08dc006e50 tx=0x7f08dc009e70).stop 2021-02-10T11:41:32.316 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.302 7f08ea53b700 1 -- 172.21.2.112:0/3522972913 shutdown_connections 2021-02-10T11:41:32.316 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.302 7f08ea53b700 1 --2- 172.21.2.112:0/3522972913 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f08e40cff70 0x7f08e4109130 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:32.316 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.302 7f08ea53b700 1 --2- 172.21.2.112:0/3522972913 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f08e40d1870 0x7f08e41b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:32.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.302 7f08ea53b700 1 -- 172.21.2.112:0/3522972913 >> 172.21.2.112:0/3522972913 conn(0x7f08e40ce420 msgr2=0x7f08e40d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:32.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.302 7f08ea53b700 1 -- 172.21.2.112:0/3522972913 shutdown_connections 2021-02-10T11:41:32.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.302 7f08ea53b700 1 -- 172.21.2.112:0/3522972913 wait complete. 2021-02-10T11:41:32.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.316 7f08ea53b700 1 Processor -- start 2021-02-10T11:41:32.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.316 7f08ea53b700 1 -- start start 2021-02-10T11:41:32.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.317 7f08ea53b700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f08e408aa00 0x7f08e41a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:32.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.317 7f08ea53b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f08e409f580 0x7f08e409f980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:32.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.317 7f08ea53b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f08e40a0830 0x7f08e40a0c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:32.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.317 7f08ea53b700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08e40a0070 con 0x7f08e40a0830 2021-02-10T11:41:32.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.317 7f08ea53b700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08e408ae30 con 0x7f08e408aa00 2021-02-10T11:41:32.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.317 7f08ea53b700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08e408d2a0 con 0x7f08e409f580 2021-02-10T11:41:32.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.317 7f08e8b58700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f08e40a0830 0x7f08e40a0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:32.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.317 7f08e8b58700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f08e40a0830 0x7f08e40a0c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:55592/0 (socket says 172.21.2.112:55592) 2021-02-10T11:41:32.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.317 7f08e8b58700 1 -- 172.21.2.112:0/2124545251 learned_addr learned my addr 172.21.2.112:0/2124545251 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:41:32.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.317 7f08e37fe700 1 --2- 172.21.2.112:0/2124545251 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f08e409f580 0x7f08e409f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:32.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.318 7f08e3fff700 1 --2- 172.21.2.112:0/2124545251 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f08e408aa00 0x7f08e41a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:32.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.318 7f08e8b58700 1 -- 172.21.2.112:0/2124545251 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f08e409f580 msgr2=0x7f08e409f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:32.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.318 7f08e8b58700 1 --2- 172.21.2.112:0/2124545251 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f08e409f580 0x7f08e409f980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:32.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.318 7f08e8b58700 1 -- 172.21.2.112:0/2124545251 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f08e408aa00 msgr2=0x7f08e41a43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:32.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.318 7f08e8b58700 1 --2- 172.21.2.112:0/2124545251 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f08e408aa00 0x7f08e41a43b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:32.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.318 7f08e8b58700 1 -- 172.21.2.112:0/2124545251 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08e41a49d0 con 0x7f08e40a0830 2021-02-10T11:41:32.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.318 7f08e8b58700 1 --2- 172.21.2.112:0/2124545251 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f08e40a0830 0x7f08e40a0c30 secure :-1 s=READY pgs=2228 cs=0 l=1 rev1=1 rx=0x7f08dc006e60 tx=0x7f08dc00bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:32.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.318 7f08e0ff9700 1 -- 172.21.2.112:0/2124545251 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f08dc009130 con 0x7f08e40a0830 2021-02-10T11:41:32.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.318 7f08e0ff9700 1 -- 172.21.2.112:0/2124545251 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08dc000e80 con 0x7f08e40a0830 2021-02-10T11:41:32.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.318 7f08e0ff9700 1 -- 172.21.2.112:0/2124545251 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f08dc0013d0 con 0x7f08e40a0830 2021-02-10T11:41:32.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.318 7f08ea53b700 1 -- 172.21.2.112:0/2124545251 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08e41a49d0 con 0x7f08e40a0830 2021-02-10T11:41:32.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.318 7f08ea53b700 1 -- 172.21.2.112:0/2124545251 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08e40a1540 con 0x7f08e40a0830 2021-02-10T11:41:32.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.319 7f08e0ff9700 1 -- 172.21.2.112:0/2124545251 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f08dc0011a0 con 0x7f08e40a0830 2021-02-10T11:41:32.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.319 7f08e0ff9700 1 --2- 172.21.2.112:0/2124545251 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f08c40186c0 0x7f08c401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:32.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.319 7f08e0ff9700 1 -- 172.21.2.112:0/2124545251 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f08dc032900 con 0x7f08e40a0830 2021-02-10T11:41:32.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.319 7f08e3fff700 1 --2- 172.21.2.112:0/2124545251 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f08c40186c0 0x7f08c401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:32.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.320 7f08e3fff700 1 --2- 172.21.2.112:0/2124545251 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f08c40186c0 0x7f08c401abc0 crc :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:32.347 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.348 7f08ea53b700 1 -- 172.21.2.112:0/2124545251 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08e4082980 con 0x7f08e40a0830 2021-02-10T11:41:32.349 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.351 7f08e0ff9700 1 -- 172.21.2.112:0/2124545251 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f08dc024a80 con 0x7f08e40a0830 2021-02-10T11:41:32.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.478 7f08ea53b700 1 -- 172.21.2.112:0/2124545251 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f08e4109a60 con 0x7f08e40a0830 2021-02-10T11:41:32.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.478 7f08e0ff9700 1 -- 172.21.2.112:0/2124545251 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v117) v1 ==== 104+0+10 (secure 0 0 0) 0x7f08dc024a80 con 0x7f08e40a0830 2021-02-10T11:41:32.477 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:41:32.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.480 7f08ea53b700 1 -- 172.21.2.112:0/2124545251 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f08c40186c0 msgr2=0x7f08c401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:32.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.480 7f08ea53b700 1 --2- 172.21.2.112:0/2124545251 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f08c40186c0 0x7f08c401abc0 crc :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:32.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.480 7f08ea53b700 1 -- 172.21.2.112:0/2124545251 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f08e40a0830 msgr2=0x7f08e40a0c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:32.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.480 7f08ea53b700 1 --2- 172.21.2.112:0/2124545251 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f08e40a0830 0x7f08e40a0c30 secure :-1 s=READY pgs=2228 cs=0 l=1 rev1=1 rx=0x7f08dc006e60 tx=0x7f08dc00bcf0).stop 2021-02-10T11:41:32.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.480 7f08ea53b700 1 -- 172.21.2.112:0/2124545251 shutdown_connections 2021-02-10T11:41:32.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.480 7f08ea53b700 1 --2- 172.21.2.112:0/2124545251 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f08c40186c0 0x7f08c401abc0 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:32.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.480 7f08ea53b700 1 --2- 172.21.2.112:0/2124545251 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f08e40a0830 0x7f08e40a0c30 unknown :-1 s=CLOSED pgs=2228 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:32.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.480 7f08ea53b700 1 --2- 172.21.2.112:0/2124545251 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f08e409f580 0x7f08e409f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:32.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.480 7f08ea53b700 1 --2- 172.21.2.112:0/2124545251 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f08e408aa00 0x7f08e41a43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:32.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.480 7f08ea53b700 1 -- 172.21.2.112:0/2124545251 >> 172.21.2.112:0/2124545251 conn(0x7f08e40bfb90 msgr2=0x7f08e40a6a00 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:32.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.480 7f08ea53b700 1 -- 172.21.2.112:0/2124545251 shutdown_connections 2021-02-10T11:41:32.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.480 7f08ea53b700 1 -- 172.21.2.112:0/2124545251 wait complete. 2021-02-10T11:41:32.487 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:41:32.919 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.904 7f981a229700 1 Processor -- start 2021-02-10T11:41:32.919 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.904 7f981a229700 1 -- start start 2021-02-10T11:41:32.919 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.904 7f981a229700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f98140d1870 0x7f98141b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:32.919 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.904 7f981a229700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f98141b0ae0 0x7f9814108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:32.919 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.904 7f981a229700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9814109a60 con 0x7f98140d1c70 2021-02-10T11:41:32.919 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.904 7f981a229700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98140e5ef0 con 0x7f98141b0ae0 2021-02-10T11:41:32.919 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.904 7f981a229700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9814109750 con 0x7f98140d1870 2021-02-10T11:41:32.919 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.905 7f98137fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f98141b0ae0 0x7f9814108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:32.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.905 7f98137fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f98141b0ae0 0x7f9814108940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:55598/0 (socket says 172.21.2.112:55598) 2021-02-10T11:41:32.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.905 7f98137fe700 1 -- 172.21.2.112:0/1122831197 learned_addr learned my addr 172.21.2.112:0/1122831197 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:41:32.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.905 7f98137fe700 1 -- 172.21.2.112:0/1122831197 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f98140d1870 msgr2=0x7f98141b04c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:41:32.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.905 7f98137fe700 1 --2- 172.21.2.112:0/1122831197 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f98140d1870 0x7f98141b04c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:32.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.905 7f98137fe700 1 -- 172.21.2.112:0/1122831197 >> v1:172.21.2.112:6789/0 conn(0x7f98140d1c70 legacy=0x7f9814108f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:32.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.905 7f9812ffd700 1 -- 172.21.2.112:0/1122831197 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 907497159 0 0) 0x7f980c000f90 con 0x7f98140d1c70 2021-02-10T11:41:32.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.905 7f98137fe700 1 -- 172.21.2.112:0/1122831197 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9814088070 con 0x7f98141b0ae0 2021-02-10T11:41:32.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.905 7f98137fe700 1 --2- 172.21.2.112:0/1122831197 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f98141b0ae0 0x7f9814108940 secure :-1 s=READY pgs=2230 cs=0 l=1 rev1=1 rx=0x7f9808006700 tx=0x7f9808009b80).ready entity=mon.0 client_cookie=e8a08d51e6bd9075 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:32.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.905 7f9812ffd700 1 -- 172.21.2.112:0/1122831197 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f980800c0c0 con 0x7f98141b0ae0 2021-02-10T11:41:32.921 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.905 7f9812ffd700 1 -- 172.21.2.112:0/1122831197 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f980800c440 con 0x7f98141b0ae0 2021-02-10T11:41:32.921 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.905 7f9812ffd700 1 -- 172.21.2.112:0/1122831197 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f980800c8a0 con 0x7f98141b0ae0 2021-02-10T11:41:32.921 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.905 7f981a229700 1 -- 172.21.2.112:0/1122831197 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f98141b0ae0 msgr2=0x7f9814108940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:32.921 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.905 7f981a229700 1 --2- 172.21.2.112:0/1122831197 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f98141b0ae0 0x7f9814108940 secure :-1 s=READY pgs=2230 cs=0 l=1 rev1=1 rx=0x7f9808006700 tx=0x7f9808009b80).stop 2021-02-10T11:41:32.921 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.906 7f981a229700 1 -- 172.21.2.112:0/1122831197 shutdown_connections 2021-02-10T11:41:32.922 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.906 7f981a229700 1 --2- 172.21.2.112:0/1122831197 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f98141b0ae0 0x7f9814108940 unknown :-1 s=CLOSED pgs=2230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:32.922 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.906 7f981a229700 1 --2- 172.21.2.112:0/1122831197 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f98140d1870 0x7f98141b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:32.922 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.906 7f981a229700 1 -- 172.21.2.112:0/1122831197 >> 172.21.2.112:0/1122831197 conn(0x7f98140ce420 msgr2=0x7f98140d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:32.922 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.906 7f981a229700 1 -- 172.21.2.112:0/1122831197 shutdown_connections 2021-02-10T11:41:32.922 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.906 7f981a229700 1 -- 172.21.2.112:0/1122831197 wait complete. 2021-02-10T11:41:32.924 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.921 7f981a229700 1 Processor -- start 2021-02-10T11:41:32.925 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.921 7f981a229700 1 -- start start 2021-02-10T11:41:32.925 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.921 7f981a229700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f98141a7bf0 0x7f98141a7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:32.925 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.921 7f981a229700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9814195560 0x7f9814195960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:32.926 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.921 7f981a229700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9814195f80 0x7f9814196380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:32.926 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.921 7f981a229700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9814194fe0 con 0x7f9814195f80 2021-02-10T11:41:32.926 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.921 7f981a229700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9814193020 con 0x7f9814195560 2021-02-10T11:41:32.926 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.921 7f981a229700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9814193440 con 0x7f98141a7bf0 2021-02-10T11:41:32.926 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.921 7f9818846700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9814195f80 0x7f9814196380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:32.926 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.922 7f9818846700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9814195f80 0x7f9814196380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:55608/0 (socket says 172.21.2.112:55608) 2021-02-10T11:41:32.926 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.922 7f9818846700 1 -- 172.21.2.112:0/3391675394 learned_addr learned my addr 172.21.2.112:0/3391675394 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:41:32.926 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.922 7f9813fff700 1 --2- 172.21.2.112:0/3391675394 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f98141a7bf0 0x7f98141a7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:32.926 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.922 7f98137fe700 1 --2- 172.21.2.112:0/3391675394 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9814195560 0x7f9814195960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:32.926 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.922 7f9818846700 1 -- 172.21.2.112:0/3391675394 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f98141a7bf0 msgr2=0x7f98141a7ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:32.926 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.922 7f9818846700 1 --2- 172.21.2.112:0/3391675394 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f98141a7bf0 0x7f98141a7ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:32.926 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.922 7f9818846700 1 -- 172.21.2.112:0/3391675394 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9814195560 msgr2=0x7f9814195960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:32.926 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.922 7f9818846700 1 --2- 172.21.2.112:0/3391675394 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9814195560 0x7f9814195960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:32.927 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.922 7f9818846700 1 -- 172.21.2.112:0/3391675394 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9814193660 con 0x7f9814195f80 2021-02-10T11:41:32.927 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.922 7f9818846700 1 --2- 172.21.2.112:0/3391675394 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9814195f80 0x7f9814196380 secure :-1 s=READY pgs=2231 cs=0 l=1 rev1=1 rx=0x7f980c009980 tx=0x7f980c00bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:32.927 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.922 7f9810ff9700 1 -- 172.21.2.112:0/3391675394 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f980c005af0 con 0x7f9814195f80 2021-02-10T11:41:32.927 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.922 7f9810ff9700 1 -- 172.21.2.112:0/3391675394 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f980c005d70 con 0x7f9814195f80 2021-02-10T11:41:32.927 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.922 7f9810ff9700 1 -- 172.21.2.112:0/3391675394 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f980c0062b0 con 0x7f9814195f80 2021-02-10T11:41:32.927 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.922 7f981a229700 1 -- 172.21.2.112:0/3391675394 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9814193660 con 0x7f9814195f80 2021-02-10T11:41:32.927 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.922 7f981a229700 1 -- 172.21.2.112:0/3391675394 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9814193bf0 con 0x7f9814195f80 2021-02-10T11:41:32.927 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.923 7f9810ff9700 1 -- 172.21.2.112:0/3391675394 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f980c006030 con 0x7f9814195f80 2021-02-10T11:41:32.927 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.923 7f9810ff9700 1 --2- 172.21.2.112:0/3391675394 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f97f40186c0 0x7f97f401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:32.927 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.923 7f9810ff9700 1 -- 172.21.2.112:0/3391675394 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f980c032960 con 0x7f9814195f80 2021-02-10T11:41:32.927 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.923 7f9813fff700 1 --2- 172.21.2.112:0/3391675394 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f97f40186c0 0x7f97f401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:32.927 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.924 7f9813fff700 1 --2- 172.21.2.112:0/3391675394 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f97f40186c0 0x7f97f401abc0 crc :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:32.953 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.954 7f981a229700 1 -- 172.21.2.112:0/3391675394 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9814082980 con 0x7f9814195f80 2021-02-10T11:41:32.955 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:32.957 7f9810ff9700 1 -- 172.21.2.112:0/3391675394 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f980c006570 con 0x7f9814195f80 2021-02-10T11:41:33.083 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:33.085 7f981a229700 1 -- 172.21.2.112:0/3391675394 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f9814109a60 con 0x7f9814195f80 2021-02-10T11:41:33.083 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:33.085 7f9810ff9700 1 -- 172.21.2.112:0/3391675394 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v117) v1 ==== 100+0+10 (secure 0 0 0) 0x7f980c006570 con 0x7f9814195f80 2021-02-10T11:41:33.084 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:41:33.085 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:33.087 7f981a229700 1 -- 172.21.2.112:0/3391675394 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f97f40186c0 msgr2=0x7f97f401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:33.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:33.087 7f981a229700 1 --2- 172.21.2.112:0/3391675394 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f97f40186c0 0x7f97f401abc0 crc :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:33.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:33.087 7f981a229700 1 -- 172.21.2.112:0/3391675394 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9814195f80 msgr2=0x7f9814196380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:33.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:33.087 7f981a229700 1 --2- 172.21.2.112:0/3391675394 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9814195f80 0x7f9814196380 secure :-1 s=READY pgs=2231 cs=0 l=1 rev1=1 rx=0x7f980c009980 tx=0x7f980c00bcd0).stop 2021-02-10T11:41:33.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:33.087 7f981a229700 1 -- 172.21.2.112:0/3391675394 shutdown_connections 2021-02-10T11:41:33.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:33.087 7f981a229700 1 --2- 172.21.2.112:0/3391675394 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f97f40186c0 0x7f97f401abc0 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:33.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:33.087 7f981a229700 1 --2- 172.21.2.112:0/3391675394 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9814195560 0x7f9814195960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:33.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:33.087 7f981a229700 1 --2- 172.21.2.112:0/3391675394 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9814195f80 0x7f9814196380 unknown :-1 s=CLOSED pgs=2231 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:33.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:33.087 7f981a229700 1 --2- 172.21.2.112:0/3391675394 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f98141a7bf0 0x7f98141a7ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:33.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:33.087 7f981a229700 1 -- 172.21.2.112:0/3391675394 >> 172.21.2.112:0/3391675394 conn(0x7f98140a3270 msgr2=0x7f9814095970 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:33.087 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:33.088 7f981a229700 1 -- 172.21.2.112:0/3391675394 shutdown_connections 2021-02-10T11:41:33.087 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:33.088 7f981a229700 1 -- 172.21.2.112:0/3391675394 wait complete. 2021-02-10T11:41:33.094 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:41:33.095 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:41:33.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:33.507 7f84f71b7700 1 Processor -- start 2021-02-10T11:41:33.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:33.507 7f84f71b7700 1 -- start start 2021-02-10T11:41:33.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:33.508 7f84f71b7700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f84f00d0340 0x7f84f00ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:33.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:33.508 7f84f71b7700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f84f00d1c70 0x7f84f00efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:33.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:33.508 7f84f71b7700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84f00f0b10 con 0x7f84f00d1870 2021-02-10T11:41:33.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:33.508 7f84f71b7700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84f00e5ed0 con 0x7f84f00d1c70 2021-02-10T11:41:33.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:33.508 7f84f71b7700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84f01b0be0 con 0x7f84f00d0340 2021-02-10T11:41:33.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:33.508 7f84effff700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f84f00d0340 0x7f84f00ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:33.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:33.508 7f84f57d4700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f84f00d1c70 0x7f84f00efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:33.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:33.508 7f84f4fd3700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7f84f00d1870 0x7f84f01b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:48632/0 (socket says 172.21.2.112:48632) 2021-02-10T11:41:33.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:33.508 7f84f4fd3700 1 -- 172.21.2.112:0/3258566120 learned_addr learned my addr 172.21.2.112:0/3258566120 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:41:33.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:33.508 7f84ef7fe700 1 -- 172.21.2.112:0/3258566120 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1040413751 0 0) 0x7f84d80009b0 con 0x7f84f00d1870 2021-02-10T11:41:33.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:33.509 7f84ef7fe700 1 -- 172.21.2.112:0/3258566120 --> v1:172.21.2.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84dc002fe0 con 0x7f84f00d1870 2021-02-10T11:41:33.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:33.509 7f84effff700 1 -- 172.21.2.112:0/3258566120 >> v2:172.21.2.120:3300/0 conn(0x7f84f00d1c70 msgr2=0x7f84f00efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:33.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:33.509 7f84effff700 1 --2- 172.21.2.112:0/3258566120 >> v2:172.21.2.120:3300/0 conn(0x7f84f00d1c70 0x7f84f00efc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:33.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:33.509 7f84effff700 1 -- 172.21.2.112:0/3258566120 >> v1:172.21.2.120:6789/0 conn(0x7f84f00d1870 legacy=0x7f84f01b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:33.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:33.509 7f84effff700 1 -- 172.21.2.112:0/3258566120 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84f00f20a0 con 0x7f84f00d0340 2021-02-10T11:41:33.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:33.509 7f84effff700 1 --2- 172.21.2.112:0/3258566120 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f84f00d0340 0x7f84f00ef640 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7f84e00069d0 tx=0x7f84e0009e40).ready entity=mon.2 client_cookie=55f2e8cca5594f5f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:33.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:33.509 7f84ef7fe700 1 -- 172.21.2.112:0/3258566120 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f84e000b510 con 0x7f84f00d0340 2021-02-10T11:41:33.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:33.509 7f84ef7fe700 1 -- 172.21.2.112:0/3258566120 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84e000b790 con 0x7f84f00d0340 2021-02-10T11:41:33.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:33.509 7f84ef7fe700 1 -- 172.21.2.112:0/3258566120 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f84e000bcb0 con 0x7f84f00d0340 2021-02-10T11:41:33.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:33.509 7f84f71b7700 1 -- 172.21.2.112:0/3258566120 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f84f00d0340 msgr2=0x7f84f00ef640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:33.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:33.509 7f84f71b7700 1 --2- 172.21.2.112:0/3258566120 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f84f00d0340 0x7f84f00ef640 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7f84e00069d0 tx=0x7f84e0009e40).stop 2021-02-10T11:41:33.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:33.509 7f84f71b7700 1 -- 172.21.2.112:0/3258566120 shutdown_connections 2021-02-10T11:41:33.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:33.509 7f84f71b7700 1 --2- 172.21.2.112:0/3258566120 >> v2:172.21.2.120:3300/0 conn(0x7f84f00d1c70 0x7f84f00efc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:33.524 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:33.509 7f84f71b7700 1 --2- 172.21.2.112:0/3258566120 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f84f00d0340 0x7f84f00ef640 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:33.524 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:33.509 7f84f71b7700 1 -- 172.21.2.112:0/3258566120 >> 172.21.2.112:0/3258566120 conn(0x7f84f00ce450 msgr2=0x7f84f00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:33.524 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:33.509 7f84f71b7700 1 -- 172.21.2.112:0/3258566120 shutdown_connections 2021-02-10T11:41:33.524 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:33.509 7f84f71b7700 1 -- 172.21.2.112:0/3258566120 wait complete. 2021-02-10T11:41:33.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:33.524 7f84f71b7700 1 Processor -- start 2021-02-10T11:41:33.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:33.524 7f84f71b7700 1 -- start start 2021-02-10T11:41:33.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:33.524 7f84f71b7700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f84f0195650 0x7f84f00b5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:33.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:33.524 7f84f71b7700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f84f00b5b80 0x7f84f01a2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:33.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:33.524 7f84f71b7700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f84f00acfa0 0x7f84f01a3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:33.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:33.524 7f84f71b7700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84f00b6010 con 0x7f84f00acfa0 2021-02-10T11:41:33.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:33.524 7f84f71b7700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84f01a3a60 con 0x7f84f0195650 2021-02-10T11:41:33.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:33.524 7f84f71b7700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84f01a3c40 con 0x7f84f00b5b80 2021-02-10T11:41:33.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:33.524 7f84f57d4700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f84f00acfa0 0x7f84f01a3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:33.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:33.524 7f84f57d4700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f84f00acfa0 0x7f84f01a3440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:55622/0 (socket says 172.21.2.112:55622) 2021-02-10T11:41:33.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:33.524 7f84f57d4700 1 -- 172.21.2.112:0/4239128465 learned_addr learned my addr 172.21.2.112:0/4239128465 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:41:33.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:33.524 7f84f4fd3700 1 --2- 172.21.2.112:0/4239128465 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f84f0195650 0x7f84f00b5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:33.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:33.524 7f84effff700 1 --2- 172.21.2.112:0/4239128465 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f84f00b5b80 0x7f84f01a2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:33.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:33.525 7f84f57d4700 1 -- 172.21.2.112:0/4239128465 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f84f00b5b80 msgr2=0x7f84f01a2e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:33.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:33.525 7f84f57d4700 1 --2- 172.21.2.112:0/4239128465 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f84f00b5b80 0x7f84f01a2e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:33.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:33.525 7f84f57d4700 1 -- 172.21.2.112:0/4239128465 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f84f0195650 msgr2=0x7f84f00b5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:33.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:33.525 7f84f57d4700 1 --2- 172.21.2.112:0/4239128465 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f84f0195650 0x7f84f00b5560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:33.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:33.525 7f84f57d4700 1 -- 172.21.2.112:0/4239128465 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84f00bcae0 con 0x7f84f00acfa0 2021-02-10T11:41:33.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:33.525 7f84effff700 1 --2- 172.21.2.112:0/4239128465 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f84f00b5b80 0x7f84f01a2e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:41:33.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:33.525 7f84f57d4700 1 --2- 172.21.2.112:0/4239128465 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f84f00acfa0 0x7f84f01a3440 secure :-1 s=READY pgs=2232 cs=0 l=1 rev1=1 rx=0x7f84e800b5e0 tx=0x7f84e800b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:33.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:33.525 7f84ed7fa700 1 -- 172.21.2.112:0/4239128465 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f84e8006370 con 0x7f84f00acfa0 2021-02-10T11:41:33.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:33.525 7f84ed7fa700 1 -- 172.21.2.112:0/4239128465 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84e800bd70 con 0x7f84f00acfa0 2021-02-10T11:41:33.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:33.525 7f84ed7fa700 1 -- 172.21.2.112:0/4239128465 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f84e8002560 con 0x7f84f00acfa0 2021-02-10T11:41:33.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:33.525 7f84f71b7700 1 -- 172.21.2.112:0/4239128465 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84f00bcae0 con 0x7f84f00acfa0 2021-02-10T11:41:33.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:33.525 7f84f71b7700 1 -- 172.21.2.112:0/4239128465 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84f00bd070 con 0x7f84f00acfa0 2021-02-10T11:41:33.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:33.526 7f84ed7fa700 1 -- 172.21.2.112:0/4239128465 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f84e8005330 con 0x7f84f00acfa0 2021-02-10T11:41:33.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:33.526 7f84ed7fa700 1 --2- 172.21.2.112:0/4239128465 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f84d40186c0 0x7f84d401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:33.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:33.526 7f84ed7fa700 1 -- 172.21.2.112:0/4239128465 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f84e8032440 con 0x7f84f00acfa0 2021-02-10T11:41:33.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:33.526 7f84f4fd3700 1 --2- 172.21.2.112:0/4239128465 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f84d40186c0 0x7f84d401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:33.528 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:33.526 7f84f4fd3700 1 --2- 172.21.2.112:0/4239128465 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f84d40186c0 0x7f84d401abc0 crc :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:33.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:33.558 7f84f71b7700 1 -- 172.21.2.112:0/4239128465 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84f00b6010 con 0x7f84f00acfa0 2021-02-10T11:41:33.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:33.560 7f84ed7fa700 1 -- 172.21.2.112:0/4239128465 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f84e80067f0 con 0x7f84f00acfa0 2021-02-10T11:41:33.686 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:33.688 7f84f71b7700 1 -- 172.21.2.112:0/4239128465 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f84f00b6010 con 0x7f84f00acfa0 2021-02-10T11:41:33.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:33.688 7f84ed7fa700 1 -- 172.21.2.112:0/4239128465 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) v1 ==== 94+0+2098 (secure 0 0 0) 0x7f84e8000e70 con 0x7f84f00acfa0 2021-02-10T11:41:33.687 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:41:33.687 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":97,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10671,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/668146927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":668146927},{"type":"v1","addr":"172.21.2.112:6833","nonce":668146927}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":93}],"filesystems":[{"mdsmap":{"epoch":97,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:41:23.731569","modified":"2021-02-10 11:41:25.740952","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8008},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8008":{"gid":8008,"name":"b","rank":0,"incarnation":94,"state":"up:active","state_seq":3,"addr":"172.21.2.120:6835/862287497","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":862287497},{"type":"v1","addr":"172.21.2.120:6835","nonce":862287497}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":12}]} 2021-02-10T11:41:33.688 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 97 2021-02-10T11:41:33.689 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:33.690 7f84f71b7700 1 -- 172.21.2.112:0/4239128465 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f84d40186c0 msgr2=0x7f84d401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:33.689 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:33.690 7f84f71b7700 1 --2- 172.21.2.112:0/4239128465 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f84d40186c0 0x7f84d401abc0 crc :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:33.689 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:33.690 7f84f71b7700 1 -- 172.21.2.112:0/4239128465 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f84f00acfa0 msgr2=0x7f84f01a3440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:33.689 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:33.690 7f84f71b7700 1 --2- 172.21.2.112:0/4239128465 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f84f00acfa0 0x7f84f01a3440 secure :-1 s=READY pgs=2232 cs=0 l=1 rev1=1 rx=0x7f84e800b5e0 tx=0x7f84e800b980).stop 2021-02-10T11:41:33.690 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:33.691 7f84f71b7700 1 -- 172.21.2.112:0/4239128465 shutdown_connections 2021-02-10T11:41:33.690 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:33.691 7f84f71b7700 1 --2- 172.21.2.112:0/4239128465 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f84d40186c0 0x7f84d401abc0 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:33.690 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:33.691 7f84f71b7700 1 --2- 172.21.2.112:0/4239128465 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f84f00acfa0 0x7f84f01a3440 unknown :-1 s=CLOSED pgs=2232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:33.690 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:33.691 7f84f71b7700 1 --2- 172.21.2.112:0/4239128465 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f84f00b5b80 0x7f84f01a2e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:33.690 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:33.691 7f84f71b7700 1 --2- 172.21.2.112:0/4239128465 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f84f0195650 0x7f84f00b5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:33.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:33.691 7f84f71b7700 1 -- 172.21.2.112:0/4239128465 >> 172.21.2.112:0/4239128465 conn(0x7f84f00b3040 msgr2=0x7f84f00c3250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:33.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:33.691 7f84f71b7700 1 -- 172.21.2.112:0/4239128465 shutdown_connections 2021-02-10T11:41:33.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:33.691 7f84f71b7700 1 -- 172.21.2.112:0/4239128465 wait complete. 2021-02-10T11:41:33.900 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 97, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:41:23.731569', 'modified': '2021-02-10 11:41:25.740952', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8008}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8008': {'gid': 8008, 'name': 'b', 'rank': 0, 'incarnation': 94, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.120:6835/862287497', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.120:6834', 'nonce': 862287497}, {'type': 'v1', 'addr': '172.21.2.120:6835', 'nonce': 862287497}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [25], 'metadata_pool': 24, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:41:33.900 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:41:33.900 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:41:34.298 INFO:teuthology.orchestra.run.gibba020.stdout:{ 2021-02-10T11:41:34.298 INFO:teuthology.orchestra.run.gibba020.stdout: "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", 2021-02-10T11:41:34.299 INFO:teuthology.orchestra.run.gibba020.stdout: "whoami": 0, 2021-02-10T11:41:34.299 INFO:teuthology.orchestra.run.gibba020.stdout: "id": 8008, 2021-02-10T11:41:34.299 INFO:teuthology.orchestra.run.gibba020.stdout: "want_state": "up:active", 2021-02-10T11:41:34.299 INFO:teuthology.orchestra.run.gibba020.stdout: "state": "up:active", 2021-02-10T11:41:34.299 INFO:teuthology.orchestra.run.gibba020.stdout: "rank_uptime": 10.567753594999999, 2021-02-10T11:41:34.299 INFO:teuthology.orchestra.run.gibba020.stdout: "mdsmap_epoch": 97, 2021-02-10T11:41:34.299 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch": 117, 2021-02-10T11:41:34.299 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch_barrier": 117, 2021-02-10T11:41:34.299 INFO:teuthology.orchestra.run.gibba020.stdout: "uptime": 16.215064778999999 2021-02-10T11:41:34.299 INFO:teuthology.orchestra.run.gibba020.stdout:} 2021-02-10T11:41:34.302 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", "whoami": 0, "id": 8008, "want_state": "up:active", "state": "up:active", "rank_uptime": 10.567753594999999, "mdsmap_epoch": 97, "osdmap_epoch": 117, "osdmap_epoch_barrier": 117, "uptime": 16.215064778999999 } 2021-02-10T11:41:34.303 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:41:34.303 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2021-02-10T11:41:34.303 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@gibba012.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2021-02-10T11:41:34.303 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:34.303 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:41:34.639 DEBUG:teuthology.orchestra.run.gibba012:> sudo modprobe fuse 2021-02-10T11:41:34.984 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:41:34.984 DEBUG:teuthology.orchestra.run.gibba012:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:41:35.328 INFO:teuthology.orchestra.run.gibba012.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:41:35.329 INFO:teuthology.orchestra.run.gibba012.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:41:35.329 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:41:35.330 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:35.330 DEBUG:teuthology.orchestra.run.gibba012:> ls /sys/fs/fuse/connections 2021-02-10T11:41:35.665 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2021-02-10T11:41:35.666 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:41:35.668 DEBUG:teuthology.orchestra.run.gibba012:> sudo modprobe fuse 2021-02-10T11:41:36.012 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:41:36.012 DEBUG:teuthology.orchestra.run.gibba012:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:41:36.078 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba012.stderr:2021-02-10 11:41:36.081 7fa2417b2f80 -1 init, newargv = 0x55bf60435220 newargc=9 2021-02-10T11:41:36.078 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba012.stderr:ceph-fuse[161470]: starting ceph client 2021-02-10T11:41:36.083 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba012.stderr:ceph-fuse[161470]: starting fuse 2021-02-10T11:41:36.354 INFO:teuthology.orchestra.run.gibba012.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:41:36.354 INFO:teuthology.orchestra.run.gibba012.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:41:36.355 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:41:36.355 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:36.355 DEBUG:teuthology.orchestra.run.gibba012:> ls /sys/fs/fuse/connections 2021-02-10T11:41:36.690 INFO:teuthology.orchestra.run.gibba012.stdout:42 2021-02-10T11:41:36.690 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2021-02-10T11:41:36.690 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:36.690 DEBUG:teuthology.orchestra.run.gibba012:> sudo python3 -c ' 2021-02-10T11:41:36.691 DEBUG:teuthology.orchestra.run.gibba012:> import glob 2021-02-10T11:41:36.691 DEBUG:teuthology.orchestra.run.gibba012:> import re 2021-02-10T11:41:36.691 DEBUG:teuthology.orchestra.run.gibba012:> import os 2021-02-10T11:41:36.691 DEBUG:teuthology.orchestra.run.gibba012:> import subprocess 2021-02-10T11:41:36.691 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:41:36.691 DEBUG:teuthology.orchestra.run.gibba012:> def find_socket(client_name): 2021-02-10T11:41:36.691 DEBUG:teuthology.orchestra.run.gibba012:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2021-02-10T11:41:36.691 DEBUG:teuthology.orchestra.run.gibba012:> files = glob.glob(asok_path) 2021-02-10T11:41:36.691 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:41:36.691 DEBUG:teuthology.orchestra.run.gibba012:> # Given a non-glob path, it better be there 2021-02-10T11:41:36.691 DEBUG:teuthology.orchestra.run.gibba012:> if "*" not in asok_path: 2021-02-10T11:41:36.691 DEBUG:teuthology.orchestra.run.gibba012:> assert(len(files) == 1) 2021-02-10T11:41:36.691 DEBUG:teuthology.orchestra.run.gibba012:> return files[0] 2021-02-10T11:41:36.692 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:41:36.692 DEBUG:teuthology.orchestra.run.gibba012:> for f in files: 2021-02-10T11:41:36.692 DEBUG:teuthology.orchestra.run.gibba012:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:41:36.692 DEBUG:teuthology.orchestra.run.gibba012:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:41:36.692 DEBUG:teuthology.orchestra.run.gibba012:> return f 2021-02-10T11:41:36.692 DEBUG:teuthology.orchestra.run.gibba012:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:41:36.692 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:41:36.692 DEBUG:teuthology.orchestra.run.gibba012:> print(find_socket("client.0")) 2021-02-10T11:41:36.692 DEBUG:teuthology.orchestra.run.gibba012:> ' 2021-02-10T11:41:37.052 INFO:teuthology.orchestra.run.gibba012.stdout:/var/run/ceph/ceph-client.0.161470.asok 2021-02-10T11:41:37.055 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.161470.asok 2021-02-10T11:41:37.055 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:37.055 DEBUG:teuthology.orchestra.run.gibba012:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.161470.asok status 2021-02-10T11:41:37.445 INFO:teuthology.orchestra.run.gibba012.stdout:{ 2021-02-10T11:41:37.445 INFO:teuthology.orchestra.run.gibba012.stdout: "metadata": { 2021-02-10T11:41:37.445 INFO:teuthology.orchestra.run.gibba012.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:41:37.445 INFO:teuthology.orchestra.run.gibba012.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:41:37.445 INFO:teuthology.orchestra.run.gibba012.stdout: "entity_id": "0", 2021-02-10T11:41:37.445 INFO:teuthology.orchestra.run.gibba012.stdout: "hostname": "gibba012", 2021-02-10T11:41:37.446 INFO:teuthology.orchestra.run.gibba012.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2021-02-10T11:41:37.446 INFO:teuthology.orchestra.run.gibba012.stdout: "pid": "161470", 2021-02-10T11:41:37.446 INFO:teuthology.orchestra.run.gibba012.stdout: "root": "/" 2021-02-10T11:41:37.446 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:41:37.446 INFO:teuthology.orchestra.run.gibba012.stdout: "dentry_count": 0, 2021-02-10T11:41:37.446 INFO:teuthology.orchestra.run.gibba012.stdout: "dentry_pinned_count": 0, 2021-02-10T11:41:37.446 INFO:teuthology.orchestra.run.gibba012.stdout: "id": 6830, 2021-02-10T11:41:37.446 INFO:teuthology.orchestra.run.gibba012.stdout: "inst": { 2021-02-10T11:41:37.446 INFO:teuthology.orchestra.run.gibba012.stdout: "name": { 2021-02-10T11:41:37.446 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "client", 2021-02-10T11:41:37.446 INFO:teuthology.orchestra.run.gibba012.stdout: "num": 6830 2021-02-10T11:41:37.447 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:41:37.447 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": { 2021-02-10T11:41:37.447 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:41:37.447 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:0", 2021-02-10T11:41:37.447 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 1828670924 2021-02-10T11:41:37.447 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:41:37.447 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:41:37.447 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": { 2021-02-10T11:41:37.447 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:41:37.447 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:0", 2021-02-10T11:41:37.447 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 1828670924 2021-02-10T11:41:37.447 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:41:37.448 INFO:teuthology.orchestra.run.gibba012.stdout: "inst_str": "client.6830 172.21.2.112:0/1828670924", 2021-02-10T11:41:37.448 INFO:teuthology.orchestra.run.gibba012.stdout: "addr_str": "172.21.2.112:0/1828670924", 2021-02-10T11:41:37.448 INFO:teuthology.orchestra.run.gibba012.stdout: "inode_count": 1, 2021-02-10T11:41:37.448 INFO:teuthology.orchestra.run.gibba012.stdout: "mds_epoch": 97, 2021-02-10T11:41:37.448 INFO:teuthology.orchestra.run.gibba012.stdout: "osd_epoch": 117, 2021-02-10T11:41:37.448 INFO:teuthology.orchestra.run.gibba012.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:41:37.448 INFO:teuthology.orchestra.run.gibba012.stdout: "blacklisted": false 2021-02-10T11:41:37.448 INFO:teuthology.orchestra.run.gibba012.stdout:} 2021-02-10T11:41:37.449 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:37.449 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:41:37.449 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:41:37.787 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:41:37.787 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:41:37.788 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:37.788 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:41:38.134 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:41:38.567 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:38.548 7f079f82f700 1 Processor -- start 2021-02-10T11:41:38.568 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:38.549 7f079f82f700 1 -- start start 2021-02-10T11:41:38.568 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:38.549 7f079f82f700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f07980d1870 0x7f07981b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:38.568 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:38.549 7f079f82f700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07980f0b10 con 0x7f07981b0a10 2021-02-10T11:41:38.568 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:38.549 7f079f82f700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07980d0310 con 0x7f07980f0650 2021-02-10T11:41:38.568 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:38.549 7f079f82f700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07980d0530 con 0x7f07980d1870 2021-02-10T11:41:38.569 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:38.549 7f079ce4a700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f07981b0a10 0x7f07980cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:55100/0 (socket says 172.21.2.112:55100) 2021-02-10T11:41:38.569 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:38.549 7f079ce4a700 1 -- 172.21.2.112:0/1650141735 learned_addr learned my addr 172.21.2.112:0/1650141735 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:41:38.569 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:38.550 7f078ffff700 1 -- 172.21.2.112:0/1650141735 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2238753823 0 0) 0x7f07880009b0 con 0x7f07981b0a10 2021-02-10T11:41:38.569 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:38.550 7f078ffff700 1 -- 172.21.2.112:0/1650141735 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0784002fe0 con 0x7f07981b0a10 2021-02-10T11:41:38.569 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:38.550 7f079d64b700 1 --2- 172.21.2.112:0/1650141735 >> v2:172.21.2.120:3300/0 conn(0x7f07980d1870 0x7f07981b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:38.569 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:38.550 7f078ffff700 1 -- 172.21.2.112:0/1650141735 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3734520505 0 0) 0x7f07880010b0 con 0x7f07981b0a10 2021-02-10T11:41:38.570 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:38.550 7f078ffff700 1 -- 172.21.2.112:0/1650141735 >> v2:172.21.2.120:3300/0 conn(0x7f07980d1870 msgr2=0x7f07981b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:38.570 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:38.550 7f078ffff700 1 --2- 172.21.2.112:0/1650141735 >> v2:172.21.2.120:3300/0 conn(0x7f07980d1870 0x7f07981b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:38.570 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:38.550 7f078ffff700 1 -- 172.21.2.112:0/1650141735 >> v1:172.21.2.120:6789/0 conn(0x7f07980f0650 legacy=0x7f07980cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:38.570 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:38.550 7f078ffff700 1 -- 172.21.2.112:0/1650141735 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07980d1c70 con 0x7f07981b0a10 2021-02-10T11:41:38.570 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:38.550 7f078ffff700 1 -- 172.21.2.112:0/1650141735 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f07880015b0 con 0x7f07981b0a10 2021-02-10T11:41:38.570 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:38.550 7f078ffff700 1 -- 172.21.2.112:0/1650141735 >> v1:172.21.2.112:6789/0 conn(0x7f07981b0a10 legacy=0x7f07980cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:38.570 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:38.550 7f078ffff700 1 --2- 172.21.2.112:0/1650141735 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0784003ac0 0x7f0784005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:38.570 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:38.550 7f078ffff700 1 --2- 172.21.2.112:0/1650141735 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0784006790 0x7f0784008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:38.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:38.550 7f078ffff700 1 --2- 172.21.2.112:0/1650141735 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0784009590 0x7f078400bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:38.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:38.550 7f078ffff700 1 -- 172.21.2.112:0/1650141735 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f078400c300 con 0x7f0784006790 2021-02-10T11:41:38.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:38.550 7f078ffff700 1 -- 172.21.2.112:0/1650141735 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f078400c520 con 0x7f0784009590 2021-02-10T11:41:38.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:38.550 7f078ffff700 1 -- 172.21.2.112:0/1650141735 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f078400c740 con 0x7f0784003ac0 2021-02-10T11:41:38.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:38.550 7f079ce4a700 1 --2- 172.21.2.112:0/1650141735 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0784006790 0x7f0784008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:38.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:38.550 7f079de4c700 1 --2- 172.21.2.112:0/1650141735 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0784009590 0x7f078400bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:38.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:38.550 7f079d64b700 1 --2- 172.21.2.112:0/1650141735 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0784003ac0 0x7f0784005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:38.572 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:38.551 7f079ce4a700 1 -- 172.21.2.112:0/1650141735 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0784003ac0 msgr2=0x7f0784005ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:38.572 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:38.551 7f079ce4a700 1 --2- 172.21.2.112:0/1650141735 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0784003ac0 0x7f0784005ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:38.572 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:38.551 7f079ce4a700 1 -- 172.21.2.112:0/1650141735 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0784009590 msgr2=0x7f078400bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:38.572 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:38.551 7f079ce4a700 1 --2- 172.21.2.112:0/1650141735 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0784009590 0x7f078400bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:38.572 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:38.551 7f079ce4a700 1 -- 172.21.2.112:0/1650141735 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f078400c9e0 con 0x7f0784006790 2021-02-10T11:41:38.572 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:38.551 7f079de4c700 1 -- 172.21.2.112:0/1650141735 reap_dead start 2021-02-10T11:41:38.572 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:38.551 7f079ce4a700 1 --2- 172.21.2.112:0/1650141735 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0784006790 0x7f0784008c90 secure :-1 s=READY pgs=2235 cs=0 l=1 rev1=1 rx=0x7f0788001c60 tx=0x7f0788007d40).ready entity=mon.0 client_cookie=476c63899c45721b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:38.572 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:38.551 7f078ffff700 1 -- 172.21.2.112:0/1650141735 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f07880070d0 con 0x7f0784006790 2021-02-10T11:41:38.572 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:38.551 7f078ffff700 1 -- 172.21.2.112:0/1650141735 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07880009b0 con 0x7f0784006790 2021-02-10T11:41:38.572 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:38.551 7f078ffff700 1 -- 172.21.2.112:0/1650141735 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f078804e530 con 0x7f0784006790 2021-02-10T11:41:38.572 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:38.551 7f079f82f700 1 -- 172.21.2.112:0/1650141735 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0784006790 msgr2=0x7f0784008c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:38.573 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:38.551 7f079f82f700 1 --2- 172.21.2.112:0/1650141735 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0784006790 0x7f0784008c90 secure :-1 s=READY pgs=2235 cs=0 l=1 rev1=1 rx=0x7f0788001c60 tx=0x7f0788007d40).stop 2021-02-10T11:41:38.573 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:38.552 7f079f82f700 1 -- 172.21.2.112:0/1650141735 shutdown_connections 2021-02-10T11:41:38.573 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:38.552 7f079f82f700 1 --2- 172.21.2.112:0/1650141735 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0784006790 0x7f0784008c90 unknown :-1 s=CLOSED pgs=2235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:38.573 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:38.552 7f079f82f700 1 -- 172.21.2.112:0/1650141735 >> 172.21.2.112:0/1650141735 conn(0x7f07980ce450 msgr2=0x7f07980d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:38.573 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:38.552 7f079f82f700 1 -- 172.21.2.112:0/1650141735 shutdown_connections 2021-02-10T11:41:38.573 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:38.552 7f079f82f700 1 -- 172.21.2.112:0/1650141735 wait complete. 2021-02-10T11:41:38.573 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:38.567 7f079f82f700 1 Processor -- start 2021-02-10T11:41:38.573 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:38.567 7f079f82f700 1 -- start start 2021-02-10T11:41:38.573 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:38.567 7f079f82f700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f07980f0280 0x7f07980a2310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:38.573 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:38.567 7f079f82f700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f07980f0680 0x7f07980a2930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:38.573 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:38.567 7f079f82f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f07980ec180 0x7f07980ec580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:38.573 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:38.567 7f079f82f700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07980efd00 con 0x7f07980ec180 2021-02-10T11:41:38.573 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:38.567 7f079f82f700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07980ecba0 con 0x7f07980f0680 2021-02-10T11:41:38.574 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:38.567 7f079f82f700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07980ecd80 con 0x7f07980f0280 2021-02-10T11:41:38.574 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:38.568 7f079de4c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f07980ec180 0x7f07980ec580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:38.574 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:38.568 7f079de4c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f07980ec180 0x7f07980ec580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:55650/0 (socket says 172.21.2.112:55650) 2021-02-10T11:41:38.574 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:38.568 7f079de4c700 1 -- 172.21.2.112:0/17322267 learned_addr learned my addr 172.21.2.112:0/17322267 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:41:38.574 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:38.568 7f079de4c700 1 -- 172.21.2.112:0/17322267 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f07980f0280 msgr2=0x7f07980a2310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:38.574 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:38.568 7f079de4c700 1 --2- 172.21.2.112:0/17322267 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f07980f0280 0x7f07980a2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:38.574 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:38.568 7f079de4c700 1 -- 172.21.2.112:0/17322267 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f07980f0680 msgr2=0x7f07980a2930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:38.574 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:38.568 7f079de4c700 1 --2- 172.21.2.112:0/17322267 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f07980f0680 0x7f07980a2930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:38.574 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:38.568 7f079de4c700 1 -- 172.21.2.112:0/17322267 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07980ecfa0 con 0x7f07980ec180 2021-02-10T11:41:38.574 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:38.568 7f079de4c700 1 --2- 172.21.2.112:0/17322267 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f07980ec180 0x7f07980ec580 secure :-1 s=READY pgs=2236 cs=0 l=1 rev1=1 rx=0x7f079000b970 tx=0x7f079000bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:38.574 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:38.568 7f078dffb700 1 -- 172.21.2.112:0/17322267 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f07900066e0 con 0x7f07980ec180 2021-02-10T11:41:38.574 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:38.568 7f078dffb700 1 -- 172.21.2.112:0/17322267 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0790000ce0 con 0x7f07980ec180 2021-02-10T11:41:38.575 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:38.568 7f079f82f700 1 -- 172.21.2.112:0/17322267 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07980ecfa0 con 0x7f07980ec180 2021-02-10T11:41:38.575 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:38.568 7f079f82f700 1 -- 172.21.2.112:0/17322267 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07980ed530 con 0x7f07980ec180 2021-02-10T11:41:38.575 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:38.568 7f078dffb700 1 -- 172.21.2.112:0/17322267 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f079000c4e0 con 0x7f07980ec180 2021-02-10T11:41:38.575 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:38.569 7f078dffb700 1 -- 172.21.2.112:0/17322267 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f079000c2f0 con 0x7f07980ec180 2021-02-10T11:41:38.575 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:38.569 7f078dffb700 1 --2- 172.21.2.112:0/17322267 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0778018730 0x7f077801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:38.575 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:38.569 7f078dffb700 1 -- 172.21.2.112:0/17322267 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f0790032b70 con 0x7f07980ec180 2021-02-10T11:41:38.575 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:38.570 7f079d64b700 1 --2- 172.21.2.112:0/17322267 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0778018730 0x7f077801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:38.575 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:38.570 7f079d64b700 1 --2- 172.21.2.112:0/17322267 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0778018730 0x7f077801abc0 crc :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:38.596 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:38.599 7f079f82f700 1 -- 172.21.2.112:0/17322267 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07980ebc60 con 0x7f07980ec180 2021-02-10T11:41:38.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:38.601 7f078dffb700 1 -- 172.21.2.112:0/17322267 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0790000cc0 con 0x7f07980ec180 2021-02-10T11:41:38.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:38.727 7f079f82f700 1 -- 172.21.2.112:0/17322267 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f07980f0b10 con 0x7f07980ec180 2021-02-10T11:41:38.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:38.727 7f078dffb700 1 -- 172.21.2.112:0/17322267 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v117) v1 ==== 74+0+14007 (secure 0 0 0) 0x7f0790000cc0 con 0x7f07980ec180 2021-02-10T11:41:38.726 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:41:38.726 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":117,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:41:24.733802","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:41:21.697492","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"116","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":25,"pool_name":"cephfs_data","create_time":"2021-02-10 11:41:22.721757","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"117","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:41:38.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:38.729 7f079f82f700 1 -- 172.21.2.112:0/17322267 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0778018730 msgr2=0x7f077801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:38.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:38.729 7f079f82f700 1 --2- 172.21.2.112:0/17322267 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0778018730 0x7f077801abc0 crc :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:38.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:38.729 7f079f82f700 1 -- 172.21.2.112:0/17322267 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f07980ec180 msgr2=0x7f07980ec580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:38.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:38.729 7f079f82f700 1 --2- 172.21.2.112:0/17322267 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f07980ec180 0x7f07980ec580 secure :-1 s=READY pgs=2236 cs=0 l=1 rev1=1 rx=0x7f079000b970 tx=0x7f079000bd10).stop 2021-02-10T11:41:38.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:38.729 7f079f82f700 1 -- 172.21.2.112:0/17322267 shutdown_connections 2021-02-10T11:41:38.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:38.729 7f079f82f700 1 --2- 172.21.2.112:0/17322267 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0778018730 0x7f077801abc0 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:38.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:38.729 7f079f82f700 1 --2- 172.21.2.112:0/17322267 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f07980f0680 0x7f07980a2930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:38.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:38.729 7f079f82f700 1 --2- 172.21.2.112:0/17322267 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f07980ec180 0x7f07980ec580 unknown :-1 s=CLOSED pgs=2236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:38.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:38.729 7f079f82f700 1 --2- 172.21.2.112:0/17322267 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f07980f0280 0x7f07980a2310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:38.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:38.729 7f079f82f700 1 -- 172.21.2.112:0/17322267 >> 172.21.2.112:0/17322267 conn(0x7f07980c2130 msgr2=0x7f079809e3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:38.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:38.729 7f079f82f700 1 -- 172.21.2.112:0/17322267 shutdown_connections 2021-02-10T11:41:38.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:38.729 7f079f82f700 1 -- 172.21.2.112:0/17322267 wait complete. 2021-02-10T11:41:38.735 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:13:57.508758', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 24, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:41:21.697492', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '116', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 25, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:41:22.721757', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '117', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:41:38.736 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:41:39.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.153 7f60fbfe7700 1 Processor -- start 2021-02-10T11:41:39.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.153 7f60fbfe7700 1 -- start start 2021-02-10T11:41:39.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.153 7f60fbfe7700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f60f40d1870 0x7f60f41b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:39.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.153 7f60fbfe7700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f60f41b0ae0 0x7f60f4108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:39.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.153 7f60fbfe7700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f60f40d1c70 0x7f60f4108f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:39.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.153 7f60fbfe7700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60f40e5ef0 con 0x7f60f41b0ae0 2021-02-10T11:41:39.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.153 7f60fbfe7700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60f4109a60 con 0x7f60f40d1c70 2021-02-10T11:41:39.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.153 7f60fbfe7700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60f4088070 con 0x7f60f40d1870 2021-02-10T11:41:39.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.154 7f60f9602700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f60f41b0ae0 0x7f60f4108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:39.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.154 7f60f9602700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f60f41b0ae0 0x7f60f4108940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:55654/0 (socket says 172.21.2.112:55654) 2021-02-10T11:41:39.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.154 7f60f9602700 1 -- 172.21.2.112:0/1484203928 learned_addr learned my addr 172.21.2.112:0/1484203928 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:41:39.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.154 7f60f9602700 1 -- 172.21.2.112:0/1484203928 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f60f40d1870 msgr2=0x7f60f41b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:39.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.154 7f60f9e03700 1 --2- 172.21.2.112:0/1484203928 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f60f40d1870 0x7f60f41b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:39.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.154 7f60f9602700 1 --2- 172.21.2.112:0/1484203928 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f60f40d1870 0x7f60f41b04c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:39.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.154 7f60f9602700 1 -- 172.21.2.112:0/1484203928 >> v2:172.21.2.120:3300/0 conn(0x7f60f40d1c70 msgr2=0x7f60f4108f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:39.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.154 7f60fa604700 1 --2- 172.21.2.112:0/1484203928 >> v2:172.21.2.120:3300/0 conn(0x7f60f40d1c70 0x7f60f4108f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:39.169 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.154 7f60f9602700 1 --2- 172.21.2.112:0/1484203928 >> v2:172.21.2.120:3300/0 conn(0x7f60f40d1c70 0x7f60f4108f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:39.169 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.154 7f60f9602700 1 -- 172.21.2.112:0/1484203928 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60f4109580 con 0x7f60f41b0ae0 2021-02-10T11:41:39.169 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.154 7f60fa604700 1 --2- 172.21.2.112:0/1484203928 >> v2:172.21.2.120:3300/0 conn(0x7f60f40d1c70 0x7f60f4108f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:41:39.169 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.154 7f60f9602700 1 --2- 172.21.2.112:0/1484203928 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f60f41b0ae0 0x7f60f4108940 secure :-1 s=READY pgs=2237 cs=0 l=1 rev1=1 rx=0x7f60e8006a80 tx=0x7f60e8009e80).ready entity=mon.0 client_cookie=2ce3cb908b047a9d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:39.169 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.155 7f60f8e01700 1 -- 172.21.2.112:0/1484203928 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f60e800b510 con 0x7f60f41b0ae0 2021-02-10T11:41:39.169 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.155 7f60f8e01700 1 -- 172.21.2.112:0/1484203928 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60e800b7d0 con 0x7f60f41b0ae0 2021-02-10T11:41:39.169 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.155 7f60f8e01700 1 -- 172.21.2.112:0/1484203928 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f60e800bcb0 con 0x7f60f41b0ae0 2021-02-10T11:41:39.169 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.155 7f60fbfe7700 1 -- 172.21.2.112:0/1484203928 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f60f41b0ae0 msgr2=0x7f60f4108940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:39.169 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.155 7f60fbfe7700 1 --2- 172.21.2.112:0/1484203928 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f60f41b0ae0 0x7f60f4108940 secure :-1 s=READY pgs=2237 cs=0 l=1 rev1=1 rx=0x7f60e8006a80 tx=0x7f60e8009e80).stop 2021-02-10T11:41:39.169 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.155 7f60fbfe7700 1 -- 172.21.2.112:0/1484203928 shutdown_connections 2021-02-10T11:41:39.169 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.155 7f60fbfe7700 1 --2- 172.21.2.112:0/1484203928 >> v2:172.21.2.120:3300/0 conn(0x7f60f40d1c70 0x7f60f4108f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:39.169 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.155 7f60fbfe7700 1 --2- 172.21.2.112:0/1484203928 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f60f41b0ae0 0x7f60f4108940 unknown :-1 s=CLOSED pgs=2237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:39.169 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.155 7f60fbfe7700 1 --2- 172.21.2.112:0/1484203928 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f60f40d1870 0x7f60f41b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:39.169 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.155 7f60fbfe7700 1 -- 172.21.2.112:0/1484203928 >> 172.21.2.112:0/1484203928 conn(0x7f60f40ce420 msgr2=0x7f60f40d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:39.170 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.156 7f60fbfe7700 1 -- 172.21.2.112:0/1484203928 shutdown_connections 2021-02-10T11:41:39.170 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.156 7f60fbfe7700 1 -- 172.21.2.112:0/1484203928 wait complete. 2021-02-10T11:41:39.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.170 7f60fbfe7700 1 Processor -- start 2021-02-10T11:41:39.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.170 7f60fbfe7700 1 -- start start 2021-02-10T11:41:39.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.171 7f60fbfe7700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f60f40b1ba0 0x7f60f41a78d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:39.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.171 7f60fbfe7700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f60f41a7ef0 0x7f60f40a6a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:39.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.171 7f60fbfe7700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f60f40a7060 0x7f60f40b2920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:39.172 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.171 7f60fbfe7700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60f4190c20 con 0x7f60f40b1ba0 2021-02-10T11:41:39.172 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.171 7f60fbfe7700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60f4190710 con 0x7f60f41a7ef0 2021-02-10T11:41:39.172 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.171 7f60fbfe7700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60f40b1fa0 con 0x7f60f40a7060 2021-02-10T11:41:39.172 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.171 7f60f9e03700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f60f40b1ba0 0x7f60f41a78d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:39.172 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.171 7f60f9e03700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f60f40b1ba0 0x7f60f41a78d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:55658/0 (socket says 172.21.2.112:55658) 2021-02-10T11:41:39.172 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.171 7f60f9e03700 1 -- 172.21.2.112:0/38888332 learned_addr learned my addr 172.21.2.112:0/38888332 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:41:39.172 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.171 7f60f9e03700 1 -- 172.21.2.112:0/38888332 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f60f40a7060 msgr2=0x7f60f40b2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:39.172 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.171 7f60fa604700 1 --2- 172.21.2.112:0/38888332 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f60f40a7060 0x7f60f40b2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:39.172 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.171 7f60f9602700 1 --2- 172.21.2.112:0/38888332 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f60f41a7ef0 0x7f60f40a6a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:39.172 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.171 7f60f9e03700 1 --2- 172.21.2.112:0/38888332 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f60f40a7060 0x7f60f40b2920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:39.173 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.171 7f60f9e03700 1 -- 172.21.2.112:0/38888332 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f60f41a7ef0 msgr2=0x7f60f40a6a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:39.173 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.171 7f60f9e03700 1 --2- 172.21.2.112:0/38888332 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f60f41a7ef0 0x7f60f40a6a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:39.173 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.171 7f60f9e03700 1 -- 172.21.2.112:0/38888332 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60f40a7460 con 0x7f60f40b1ba0 2021-02-10T11:41:39.173 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.171 7f60fa604700 1 --2- 172.21.2.112:0/38888332 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f60f40a7060 0x7f60f40b2920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:41:39.173 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.172 7f60f9e03700 1 --2- 172.21.2.112:0/38888332 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f60f40b1ba0 0x7f60f41a78d0 secure :-1 s=READY pgs=2238 cs=0 l=1 rev1=1 rx=0x7f60e4006f70 tx=0x7f60e400b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:39.173 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.172 7f60f27fc700 1 -- 172.21.2.112:0/38888332 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f60e400bd50 con 0x7f60f40b1ba0 2021-02-10T11:41:39.173 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.172 7f60f27fc700 1 -- 172.21.2.112:0/38888332 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60e4000c90 con 0x7f60f40b1ba0 2021-02-10T11:41:39.173 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.172 7f60f27fc700 1 -- 172.21.2.112:0/38888332 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f60e4002220 con 0x7f60f40b1ba0 2021-02-10T11:41:39.173 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.172 7f60fbfe7700 1 -- 172.21.2.112:0/38888332 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60f40a7460 con 0x7f60f40b1ba0 2021-02-10T11:41:39.173 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.172 7f60fbfe7700 1 -- 172.21.2.112:0/38888332 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60f40a7460 con 0x7f60f40b1ba0 2021-02-10T11:41:39.173 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.172 7f60f27fc700 1 -- 172.21.2.112:0/38888332 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f60e400a000 con 0x7f60f40b1ba0 2021-02-10T11:41:39.173 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.173 7f60f27fc700 1 --2- 172.21.2.112:0/38888332 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f60d4018730 0x7f60d401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:39.173 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.173 7f60f27fc700 1 -- 172.21.2.112:0/38888332 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f60e4032630 con 0x7f60f40b1ba0 2021-02-10T11:41:39.174 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.173 7f60f9602700 1 --2- 172.21.2.112:0/38888332 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f60d4018730 0x7f60d401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:39.174 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.174 7f60f9602700 1 --2- 172.21.2.112:0/38888332 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f60d4018730 0x7f60d401abc0 crc :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:39.201 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.203 7f60fbfe7700 1 -- 172.21.2.112:0/38888332 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60f4082980 con 0x7f60f40b1ba0 2021-02-10T11:41:39.203 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.206 7f60f27fc700 1 -- 172.21.2.112:0/38888332 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f60e4006bc0 con 0x7f60f40b1ba0 2021-02-10T11:41:39.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.334 7f60fbfe7700 1 -- 172.21.2.112:0/38888332 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f60f4109a60 con 0x7f60f40b1ba0 2021-02-10T11:41:39.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.335 7f60f27fc700 1 -- 172.21.2.112:0/38888332 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v117) v1 ==== 92+0+10 (secure 0 0 0) 0x7f60e400a000 con 0x7f60f40b1ba0 2021-02-10T11:41:39.333 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:41:39.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.337 7f60fbfe7700 1 -- 172.21.2.112:0/38888332 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f60d4018730 msgr2=0x7f60d401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:39.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.337 7f60fbfe7700 1 --2- 172.21.2.112:0/38888332 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f60d4018730 0x7f60d401abc0 crc :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:39.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.337 7f60fbfe7700 1 -- 172.21.2.112:0/38888332 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f60f40b1ba0 msgr2=0x7f60f41a78d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:39.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.337 7f60fbfe7700 1 --2- 172.21.2.112:0/38888332 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f60f40b1ba0 0x7f60f41a78d0 secure :-1 s=READY pgs=2238 cs=0 l=1 rev1=1 rx=0x7f60e4006f70 tx=0x7f60e400b960).stop 2021-02-10T11:41:39.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.337 7f60fbfe7700 1 -- 172.21.2.112:0/38888332 shutdown_connections 2021-02-10T11:41:39.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.337 7f60fbfe7700 1 --2- 172.21.2.112:0/38888332 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f60d4018730 0x7f60d401abc0 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:39.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.337 7f60fbfe7700 1 --2- 172.21.2.112:0/38888332 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f60f41a7ef0 0x7f60f40a6a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:39.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.337 7f60fbfe7700 1 --2- 172.21.2.112:0/38888332 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f60f40a7060 0x7f60f40b2920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:39.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.337 7f60fbfe7700 1 --2- 172.21.2.112:0/38888332 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f60f40b1ba0 0x7f60f41a78d0 unknown :-1 s=CLOSED pgs=2238 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:39.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.337 7f60fbfe7700 1 -- 172.21.2.112:0/38888332 >> 172.21.2.112:0/38888332 conn(0x7f60f40b5dc0 msgr2=0x7f60f4095960 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:39.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.337 7f60fbfe7700 1 -- 172.21.2.112:0/38888332 shutdown_connections 2021-02-10T11:41:39.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.337 7f60fbfe7700 1 -- 172.21.2.112:0/38888332 wait complete. 2021-02-10T11:41:39.343 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:41:39.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.760 7f0af44fc700 1 Processor -- start 2021-02-10T11:41:39.774 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.760 7f0af44fc700 1 -- start start 2021-02-10T11:41:39.774 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.760 7f0af44fc700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f0aec0d1870 0x7f0aec1b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:39.774 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.760 7f0af44fc700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0aec0cff70 0x7f0aec109130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:39.774 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.760 7f0af44fc700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0aec109a60 con 0x7f0aec1b0ae0 2021-02-10T11:41:39.774 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.760 7f0af44fc700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0aec109750 con 0x7f0aec0d1870 2021-02-10T11:41:39.774 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.760 7f0af44fc700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0aec088070 con 0x7f0aec0cff70 2021-02-10T11:41:39.774 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.760 7f0af2318700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f0aec0d1870 0x7f0aec1b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:39.774 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.760 7f0af2318700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f0aec0d1870 0x7f0aec1b04c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:55666/0 (socket says 172.21.2.112:55666) 2021-02-10T11:41:39.774 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.760 7f0af2318700 1 -- 172.21.2.112:0/2286859105 learned_addr learned my addr 172.21.2.112:0/2286859105 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:41:39.774 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.761 7f0af2318700 1 -- 172.21.2.112:0/2286859105 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0aec0cff70 msgr2=0x7f0aec109130 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:41:39.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.761 7f0af2318700 1 --2- 172.21.2.112:0/2286859105 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0aec0cff70 0x7f0aec109130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:39.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.761 7f0af2318700 1 -- 172.21.2.112:0/2286859105 >> v1:172.21.2.120:6789/0 conn(0x7f0aec1b0ae0 legacy=0x7f0aec108940 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:41:39.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.761 7f0af2318700 1 -- 172.21.2.112:0/2286859105 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0aec0d1c70 con 0x7f0aec0d1870 2021-02-10T11:41:39.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.761 7f0af2318700 1 --2- 172.21.2.112:0/2286859105 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0aec0d1870 0x7f0aec1b04c0 secure :-1 s=READY pgs=2239 cs=0 l=1 rev1=1 rx=0x7f0ae0006700 tx=0x7f0ae0009b80).ready entity=mon.0 client_cookie=82c32785a9a50ec3 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:39.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.761 7f0af1316700 1 -- 172.21.2.112:0/2286859105 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0ae000c0c0 con 0x7f0aec0d1870 2021-02-10T11:41:39.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.761 7f0af1316700 1 -- 172.21.2.112:0/2286859105 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ae000c340 con 0x7f0aec0d1870 2021-02-10T11:41:39.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.761 7f0af1316700 1 -- 172.21.2.112:0/2286859105 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0ae000c860 con 0x7f0aec0d1870 2021-02-10T11:41:39.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.761 7f0af44fc700 1 -- 172.21.2.112:0/2286859105 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0aec0d1870 msgr2=0x7f0aec1b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:39.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.761 7f0af44fc700 1 --2- 172.21.2.112:0/2286859105 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0aec0d1870 0x7f0aec1b04c0 secure :-1 s=READY pgs=2239 cs=0 l=1 rev1=1 rx=0x7f0ae0006700 tx=0x7f0ae0009b80).stop 2021-02-10T11:41:39.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.762 7f0af44fc700 1 -- 172.21.2.112:0/2286859105 shutdown_connections 2021-02-10T11:41:39.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.762 7f0af44fc700 1 --2- 172.21.2.112:0/2286859105 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0aec0cff70 0x7f0aec109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:39.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.762 7f0af44fc700 1 --2- 172.21.2.112:0/2286859105 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0aec0d1870 0x7f0aec1b04c0 unknown :-1 s=CLOSED pgs=2239 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:39.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.762 7f0af44fc700 1 -- 172.21.2.112:0/2286859105 >> 172.21.2.112:0/2286859105 conn(0x7f0aec0ce420 msgr2=0x7f0aec0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:39.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.762 7f0af44fc700 1 -- 172.21.2.112:0/2286859105 shutdown_connections 2021-02-10T11:41:39.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.762 7f0af44fc700 1 -- 172.21.2.112:0/2286859105 wait complete. 2021-02-10T11:41:39.777 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.776 7f0af44fc700 1 Processor -- start 2021-02-10T11:41:39.778 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.776 7f0af44fc700 1 -- start start 2021-02-10T11:41:39.778 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.777 7f0af44fc700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0aec0c2370 0x7f0aec18f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:39.778 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.777 7f0af44fc700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0aec084cb0 0x7f0aec1a7470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:39.778 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.777 7f0af44fc700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0aec1a7a90 0x7f0aec085700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:39.778 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.777 7f0af44fc700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0aec0be840 con 0x7f0aec1a7a90 2021-02-10T11:41:39.778 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.777 7f0af44fc700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0aec0c29a0 con 0x7f0aec084cb0 2021-02-10T11:41:39.778 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.777 7f0af44fc700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0aec1a7e90 con 0x7f0aec0c2370 2021-02-10T11:41:39.778 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.777 7f0af2b19700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0aec1a7a90 0x7f0aec085700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:39.778 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.777 7f0af2b19700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0aec1a7a90 0x7f0aec085700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:55676/0 (socket says 172.21.2.112:55676) 2021-02-10T11:41:39.778 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.777 7f0af2b19700 1 -- 172.21.2.112:0/1104000922 learned_addr learned my addr 172.21.2.112:0/1104000922 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:41:39.778 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.777 7f0af2318700 1 --2- 172.21.2.112:0/1104000922 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0aec0c2370 0x7f0aec18f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:39.778 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.777 7f0af1b17700 1 --2- 172.21.2.112:0/1104000922 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0aec084cb0 0x7f0aec1a7470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:39.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.777 7f0af2b19700 1 -- 172.21.2.112:0/1104000922 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0aec0c2370 msgr2=0x7f0aec18f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:39.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.777 7f0af2b19700 1 --2- 172.21.2.112:0/1104000922 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0aec0c2370 0x7f0aec18f870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:39.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.777 7f0af2b19700 1 -- 172.21.2.112:0/1104000922 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0aec084cb0 msgr2=0x7f0aec1a7470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:39.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.777 7f0af2b19700 1 --2- 172.21.2.112:0/1104000922 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0aec084cb0 0x7f0aec1a7470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:39.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.777 7f0af2b19700 1 -- 172.21.2.112:0/1104000922 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0aec1a80b0 con 0x7f0aec1a7a90 2021-02-10T11:41:39.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.777 7f0af2318700 1 --2- 172.21.2.112:0/1104000922 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0aec0c2370 0x7f0aec18f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:41:39.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.778 7f0af2b19700 1 --2- 172.21.2.112:0/1104000922 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0aec1a7a90 0x7f0aec085700 secure :-1 s=READY pgs=2240 cs=0 l=1 rev1=1 rx=0x7f0ae4009960 tx=0x7f0ae400bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:39.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.778 7f0adeffd700 1 -- 172.21.2.112:0/1104000922 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0ae4003030 con 0x7f0aec1a7a90 2021-02-10T11:41:39.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.778 7f0adeffd700 1 -- 172.21.2.112:0/1104000922 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ae4001310 con 0x7f0aec1a7a90 2021-02-10T11:41:39.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.778 7f0adeffd700 1 -- 172.21.2.112:0/1104000922 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0ae4001830 con 0x7f0aec1a7a90 2021-02-10T11:41:39.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.778 7f0af44fc700 1 -- 172.21.2.112:0/1104000922 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0aec1a80b0 con 0x7f0aec1a7a90 2021-02-10T11:41:39.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.778 7f0af44fc700 1 -- 172.21.2.112:0/1104000922 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0aec1a80b0 con 0x7f0aec1a7a90 2021-02-10T11:41:39.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.779 7f0adeffd700 1 -- 172.21.2.112:0/1104000922 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0ae4000d10 con 0x7f0aec1a7a90 2021-02-10T11:41:39.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.779 7f0adeffd700 1 --2- 172.21.2.112:0/1104000922 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0ad4018700 0x7f0ad401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:39.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.779 7f0adeffd700 1 -- 172.21.2.112:0/1104000922 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f0ae4032b40 con 0x7f0aec1a7a90 2021-02-10T11:41:39.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.779 7f0af2318700 1 --2- 172.21.2.112:0/1104000922 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0ad4018700 0x7f0ad401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:39.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.779 7f0af2318700 1 --2- 172.21.2.112:0/1104000922 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0ad4018700 0x7f0ad401abc0 crc :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:39.807 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.810 7f0af44fc700 1 -- 172.21.2.112:0/1104000922 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0aec0be840 con 0x7f0aec1a7a90 2021-02-10T11:41:39.810 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.813 7f0adeffd700 1 -- 172.21.2.112:0/1104000922 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0ae400be30 con 0x7f0aec1a7a90 2021-02-10T11:41:39.939 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.941 7f0af44fc700 1 -- 172.21.2.112:0/1104000922 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f0aec0be840 con 0x7f0aec1a7a90 2021-02-10T11:41:39.939 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.942 7f0adeffd700 1 -- 172.21.2.112:0/1104000922 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v117) v1 ==== 104+0+10 (secure 0 0 0) 0x7f0ae4000d10 con 0x7f0aec1a7a90 2021-02-10T11:41:39.940 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:41:39.941 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.944 7f0af44fc700 1 -- 172.21.2.112:0/1104000922 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0ad4018700 msgr2=0x7f0ad401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:39.941 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.944 7f0af44fc700 1 --2- 172.21.2.112:0/1104000922 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0ad4018700 0x7f0ad401abc0 crc :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:39.941 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.944 7f0af44fc700 1 -- 172.21.2.112:0/1104000922 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0aec1a7a90 msgr2=0x7f0aec085700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:39.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.944 7f0af44fc700 1 --2- 172.21.2.112:0/1104000922 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0aec1a7a90 0x7f0aec085700 secure :-1 s=READY pgs=2240 cs=0 l=1 rev1=1 rx=0x7f0ae4009960 tx=0x7f0ae400bcf0).stop 2021-02-10T11:41:39.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.944 7f0af44fc700 1 -- 172.21.2.112:0/1104000922 shutdown_connections 2021-02-10T11:41:39.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.944 7f0af44fc700 1 --2- 172.21.2.112:0/1104000922 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0ad4018700 0x7f0ad401abc0 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:39.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.944 7f0af44fc700 1 --2- 172.21.2.112:0/1104000922 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0aec084cb0 0x7f0aec1a7470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:39.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.944 7f0af44fc700 1 --2- 172.21.2.112:0/1104000922 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0aec1a7a90 0x7f0aec085700 unknown :-1 s=CLOSED pgs=2240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:39.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.944 7f0af44fc700 1 --2- 172.21.2.112:0/1104000922 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0aec0c2370 0x7f0aec18f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:39.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.944 7f0af44fc700 1 -- 172.21.2.112:0/1104000922 >> 172.21.2.112:0/1104000922 conn(0x7f0aec0b5dc0 msgr2=0x7f0aec0c2d90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:39.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.944 7f0af44fc700 1 -- 172.21.2.112:0/1104000922 shutdown_connections 2021-02-10T11:41:39.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:39.944 7f0af44fc700 1 -- 172.21.2.112:0/1104000922 wait complete. 2021-02-10T11:41:39.949 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:41:40.380 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.366 7f3754f79700 1 Processor -- start 2021-02-10T11:41:40.380 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.366 7f3754f79700 1 -- start start 2021-02-10T11:41:40.380 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.366 7f3754f79700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f37500d1c70 0x7f37500cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:40.380 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.366 7f3754f79700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f37500cf580 0x7f37500cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:40.380 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.366 7f3754f79700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37500fe1c0 con 0x7f37500d1870 2021-02-10T11:41:40.380 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.366 7f3754f79700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37501b0cb0 con 0x7f37500cf580 2021-02-10T11:41:40.380 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.366 7f3754f79700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3750109a60 con 0x7f37500d1c70 2021-02-10T11:41:40.380 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.366 7f374eb13700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f37500d1870 0x7f37501b04c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:55142/0 (socket says 172.21.2.112:55142) 2021-02-10T11:41:40.380 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.366 7f374eb13700 1 -- 172.21.2.112:0/1518221557 learned_addr learned my addr 172.21.2.112:0/1518221557 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:41:40.380 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.367 7f374f314700 1 --2- 172.21.2.112:0/1518221557 >> v2:172.21.2.112:3300/0 conn(0x7f37500cf580 0x7f37500cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:40.381 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.367 7f374db11700 1 -- 172.21.2.112:0/1518221557 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1395072205 0 0) 0x7f3740001180 con 0x7f37500d1870 2021-02-10T11:41:40.381 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.367 7f374f314700 1 -- 172.21.2.112:0/1518221557 >> v2:172.21.2.120:3300/0 conn(0x7f37500d1c70 msgr2=0x7f37500cef60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:41:40.381 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.367 7f374f314700 1 --2- 172.21.2.112:0/1518221557 >> v2:172.21.2.120:3300/0 conn(0x7f37500d1c70 0x7f37500cef60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:40.381 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.367 7f374f314700 1 -- 172.21.2.112:0/1518221557 >> v1:172.21.2.112:6789/0 conn(0x7f37500d1870 legacy=0x7f37501b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:40.381 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.367 7f374f314700 1 -- 172.21.2.112:0/1518221557 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3750088070 con 0x7f37500cf580 2021-02-10T11:41:40.381 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.367 7f374f314700 1 --2- 172.21.2.112:0/1518221557 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f37500cf580 0x7f37500cf980 secure :-1 s=READY pgs=2242 cs=0 l=1 rev1=1 rx=0x7f3748006e30 tx=0x7f3748009e60).ready entity=mon.0 client_cookie=1c77eb7a982a6169 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:40.381 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.367 7f374db11700 1 -- 172.21.2.112:0/1518221557 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f374800b300 con 0x7f37500cf580 2021-02-10T11:41:40.381 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.367 7f374db11700 1 -- 172.21.2.112:0/1518221557 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3748000d30 con 0x7f37500cf580 2021-02-10T11:41:40.381 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.367 7f374db11700 1 -- 172.21.2.112:0/1518221557 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f374800b800 con 0x7f37500cf580 2021-02-10T11:41:40.381 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.367 7f3754f79700 1 -- 172.21.2.112:0/1518221557 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f37500cf580 msgr2=0x7f37500cf980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:40.381 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.367 7f3754f79700 1 --2- 172.21.2.112:0/1518221557 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f37500cf580 0x7f37500cf980 secure :-1 s=READY pgs=2242 cs=0 l=1 rev1=1 rx=0x7f3748006e30 tx=0x7f3748009e60).stop 2021-02-10T11:41:40.381 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.368 7f3754f79700 1 -- 172.21.2.112:0/1518221557 shutdown_connections 2021-02-10T11:41:40.381 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.368 7f3754f79700 1 --2- 172.21.2.112:0/1518221557 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f37500cf580 0x7f37500cf980 unknown :-1 s=CLOSED pgs=2242 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:40.382 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.368 7f3754f79700 1 --2- 172.21.2.112:0/1518221557 >> v2:172.21.2.120:3300/0 conn(0x7f37500d1c70 0x7f37500cef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:40.382 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.368 7f3754f79700 1 -- 172.21.2.112:0/1518221557 >> 172.21.2.112:0/1518221557 conn(0x7f37500ce420 msgr2=0x7f37500d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:40.382 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.368 7f3754f79700 1 -- 172.21.2.112:0/1518221557 shutdown_connections 2021-02-10T11:41:40.382 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.368 7f3754f79700 1 -- 172.21.2.112:0/1518221557 wait complete. 2021-02-10T11:41:40.383 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.382 7f3754f79700 1 Processor -- start 2021-02-10T11:41:40.383 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.382 7f3754f79700 1 -- start start 2021-02-10T11:41:40.383 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.383 7f3754f79700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f37500c3820 0x7f37500a7950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:40.383 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.383 7f3754f79700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f37501a44c0 0x7f37501a48c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:40.384 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.383 7f3754f79700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f37500863d0 0x7f37500867d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:40.384 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.383 7f3754f79700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3750085ed0 con 0x7f37500863d0 2021-02-10T11:41:40.384 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.383 7f3754f79700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37500c0740 con 0x7f37500c3820 2021-02-10T11:41:40.384 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.383 7f3754f79700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3750086df0 con 0x7f37501a44c0 2021-02-10T11:41:40.384 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.383 7f374f314700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f37500863d0 0x7f37500867d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:40.384 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.383 7f374f314700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f37500863d0 0x7f37500867d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:55688/0 (socket says 172.21.2.112:55688) 2021-02-10T11:41:40.384 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.383 7f374f314700 1 -- 172.21.2.112:0/2114894902 learned_addr learned my addr 172.21.2.112:0/2114894902 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:41:40.384 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.383 7f374f314700 1 -- 172.21.2.112:0/2114894902 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f37501a44c0 msgr2=0x7f37501a48c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:40.384 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.383 7f374f314700 1 --2- 172.21.2.112:0/2114894902 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f37501a44c0 0x7f37501a48c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:40.384 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.383 7f374f314700 1 -- 172.21.2.112:0/2114894902 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f37500c3820 msgr2=0x7f37500a7950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:40.384 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.383 7f374f314700 1 --2- 172.21.2.112:0/2114894902 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f37500c3820 0x7f37500a7950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:40.384 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.383 7f374f314700 1 -- 172.21.2.112:0/2114894902 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3750087010 con 0x7f37500863d0 2021-02-10T11:41:40.384 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.384 7f374f314700 1 --2- 172.21.2.112:0/2114894902 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f37500863d0 0x7f37500867d0 secure :-1 s=READY pgs=2243 cs=0 l=1 rev1=1 rx=0x7f374800b930 tx=0x7f374800bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:40.385 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.384 7f37477fe700 1 -- 172.21.2.112:0/2114894902 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3748009750 con 0x7f37500863d0 2021-02-10T11:41:40.385 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.384 7f37477fe700 1 -- 172.21.2.112:0/2114894902 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3748000d90 con 0x7f37500863d0 2021-02-10T11:41:40.385 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.384 7f37477fe700 1 -- 172.21.2.112:0/2114894902 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f374800c4e0 con 0x7f37500863d0 2021-02-10T11:41:40.385 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.384 7f3754f79700 1 -- 172.21.2.112:0/2114894902 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3750087010 con 0x7f37500863d0 2021-02-10T11:41:40.385 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.384 7f3754f79700 1 -- 172.21.2.112:0/2114894902 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3750087010 con 0x7f37500863d0 2021-02-10T11:41:40.385 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.384 7f37477fe700 1 -- 172.21.2.112:0/2114894902 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3748001270 con 0x7f37500863d0 2021-02-10T11:41:40.385 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.385 7f37477fe700 1 --2- 172.21.2.112:0/2114894902 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f37340186c0 0x7f373401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:40.385 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.385 7f37477fe700 1 -- 172.21.2.112:0/2114894902 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f3748032770 con 0x7f37500863d0 2021-02-10T11:41:40.385 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.385 7f374eb13700 1 --2- 172.21.2.112:0/2114894902 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f37340186c0 0x7f373401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:40.385 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.386 7f374eb13700 1 --2- 172.21.2.112:0/2114894902 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f37340186c0 0x7f373401abc0 crc :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:40.414 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.416 7f3754f79700 1 -- 172.21.2.112:0/2114894902 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37500fe1c0 con 0x7f37500863d0 2021-02-10T11:41:40.416 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.419 7f37477fe700 1 -- 172.21.2.112:0/2114894902 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f374802c250 con 0x7f37500863d0 2021-02-10T11:41:40.544 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.547 7f3754f79700 1 -- 172.21.2.112:0/2114894902 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f37500fe1c0 con 0x7f37500863d0 2021-02-10T11:41:40.545 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.547 7f37477fe700 1 -- 172.21.2.112:0/2114894902 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v117) v1 ==== 100+0+10 (secure 0 0 0) 0x7f374802c250 con 0x7f37500863d0 2021-02-10T11:41:40.545 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:41:40.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.549 7f3754f79700 1 -- 172.21.2.112:0/2114894902 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f37340186c0 msgr2=0x7f373401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:40.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.549 7f3754f79700 1 --2- 172.21.2.112:0/2114894902 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f37340186c0 0x7f373401abc0 crc :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:40.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.549 7f3754f79700 1 -- 172.21.2.112:0/2114894902 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f37500863d0 msgr2=0x7f37500867d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:40.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.549 7f3754f79700 1 --2- 172.21.2.112:0/2114894902 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f37500863d0 0x7f37500867d0 secure :-1 s=READY pgs=2243 cs=0 l=1 rev1=1 rx=0x7f374800b930 tx=0x7f374800bcd0).stop 2021-02-10T11:41:40.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.549 7f3754f79700 1 -- 172.21.2.112:0/2114894902 shutdown_connections 2021-02-10T11:41:40.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.549 7f3754f79700 1 --2- 172.21.2.112:0/2114894902 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f37340186c0 0x7f373401abc0 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:40.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.549 7f3754f79700 1 --2- 172.21.2.112:0/2114894902 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f37500863d0 0x7f37500867d0 unknown :-1 s=CLOSED pgs=2243 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:40.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.549 7f3754f79700 1 --2- 172.21.2.112:0/2114894902 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f37501a44c0 0x7f37501a48c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:40.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.549 7f3754f79700 1 --2- 172.21.2.112:0/2114894902 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f37500c3820 0x7f37500a7950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:40.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.549 7f3754f79700 1 -- 172.21.2.112:0/2114894902 >> 172.21.2.112:0/2114894902 conn(0x7f37500872c0 msgr2=0x7f3750095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:40.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.549 7f3754f79700 1 -- 172.21.2.112:0/2114894902 shutdown_connections 2021-02-10T11:41:40.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.549 7f3754f79700 1 -- 172.21.2.112:0/2114894902 wait complete. 2021-02-10T11:41:40.555 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:41:40.556 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:41:40.982 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.968 7f584937d700 1 Processor -- start 2021-02-10T11:41:40.982 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.968 7f584937d700 1 -- start start 2021-02-10T11:41:40.982 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.969 7f584937d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f58441b0870 0x7f58440f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:40.982 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.969 7f584937d700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58441b0c70 con 0x7f58440efe00 2021-02-10T11:41:40.982 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.969 7f584937d700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58440f0b50 con 0x7f58441b0470 2021-02-10T11:41:40.983 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.969 7f584937d700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58440e5ea0 con 0x7f58441b0870 2021-02-10T11:41:40.983 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.969 7f584287c700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f58441b0870 0x7f58440f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:40.983 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.969 7f584287c700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f58441b0870 0x7f58440f20e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:55694/0 (socket says 172.21.2.112:55694) 2021-02-10T11:41:40.983 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.969 7f584287c700 1 -- 172.21.2.112:0/3685899359 learned_addr learned my addr 172.21.2.112:0/3685899359 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:41:40.983 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.969 7f584287c700 1 -- 172.21.2.112:0/3685899359 >> v1:172.21.2.120:6789/0 conn(0x7f58441b0470 legacy=0x7f58440ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:40.983 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.969 7f584287c700 1 -- 172.21.2.112:0/3685899359 >> v1:172.21.2.112:6789/0 conn(0x7f58440efe00 legacy=0x7f58440f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:40.983 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.969 7f584287c700 1 -- 172.21.2.112:0/3685899359 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58440e52e0 con 0x7f58441b0870 2021-02-10T11:41:40.983 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.970 7f584287c700 1 --2- 172.21.2.112:0/3685899359 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f58441b0870 0x7f58440f20e0 secure :-1 s=READY pgs=2245 cs=0 l=1 rev1=1 rx=0x7f582c006700 tx=0x7f582c009b80).ready entity=mon.0 client_cookie=c744428287c75e1c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:40.983 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.970 7f584207b700 1 -- 172.21.2.112:0/3685899359 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f582c00c0c0 con 0x7f58441b0870 2021-02-10T11:41:40.983 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.970 7f584207b700 1 -- 172.21.2.112:0/3685899359 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f582c00c340 con 0x7f58441b0870 2021-02-10T11:41:40.983 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.970 7f584207b700 1 -- 172.21.2.112:0/3685899359 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f582c00c820 con 0x7f58441b0870 2021-02-10T11:41:40.983 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.970 7f584937d700 1 -- 172.21.2.112:0/3685899359 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f58441b0870 msgr2=0x7f58440f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:40.983 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.970 7f584937d700 1 --2- 172.21.2.112:0/3685899359 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f58441b0870 0x7f58440f20e0 secure :-1 s=READY pgs=2245 cs=0 l=1 rev1=1 rx=0x7f582c006700 tx=0x7f582c009b80).stop 2021-02-10T11:41:40.984 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.970 7f584937d700 1 -- 172.21.2.112:0/3685899359 shutdown_connections 2021-02-10T11:41:40.984 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.970 7f584937d700 1 --2- 172.21.2.112:0/3685899359 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f58441b0870 0x7f58440f20e0 unknown :-1 s=CLOSED pgs=2245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:40.984 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.970 7f584937d700 1 -- 172.21.2.112:0/3685899359 >> 172.21.2.112:0/3685899359 conn(0x7f58440ce3e0 msgr2=0x7f58440e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:40.984 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.970 7f584937d700 1 -- 172.21.2.112:0/3685899359 shutdown_connections 2021-02-10T11:41:40.984 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.970 7f584937d700 1 -- 172.21.2.112:0/3685899359 wait complete. 2021-02-10T11:41:40.986 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.985 7f584937d700 1 Processor -- start 2021-02-10T11:41:40.986 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.985 7f584937d700 1 -- start start 2021-02-10T11:41:40.986 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.985 7f584937d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5844092e30 0x7f58441c6320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:40.986 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.985 7f584937d700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f58440964a0 0x7f58440968a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:40.986 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.985 7f584937d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5844096ec0 0x7f5844084f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:40.986 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.985 7f584937d700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58440932b0 con 0x7f5844092e30 2021-02-10T11:41:40.986 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.986 7f584937d700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58441b5f70 con 0x7f58440964a0 2021-02-10T11:41:40.986 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.986 7f584937d700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58441b6150 con 0x7f5844096ec0 2021-02-10T11:41:40.987 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.986 7f584307d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5844092e30 0x7f58441c6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:40.987 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.986 7f584307d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5844092e30 0x7f58441c6320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:55700/0 (socket says 172.21.2.112:55700) 2021-02-10T11:41:40.987 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.986 7f584307d700 1 -- 172.21.2.112:0/910539306 learned_addr learned my addr 172.21.2.112:0/910539306 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:41:40.987 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.986 7f584387e700 1 --2- 172.21.2.112:0/910539306 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5844096ec0 0x7f5844084f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:40.987 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.986 7f584287c700 1 --2- 172.21.2.112:0/910539306 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f58440964a0 0x7f58440968a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:40.987 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.986 7f584307d700 1 -- 172.21.2.112:0/910539306 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5844096ec0 msgr2=0x7f5844084f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:40.987 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.986 7f584307d700 1 --2- 172.21.2.112:0/910539306 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5844096ec0 0x7f5844084f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:40.987 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.986 7f584307d700 1 -- 172.21.2.112:0/910539306 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f58440964a0 msgr2=0x7f58440968a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:40.987 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.986 7f584307d700 1 --2- 172.21.2.112:0/910539306 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f58440964a0 0x7f58440968a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:40.987 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.986 7f584307d700 1 -- 172.21.2.112:0/910539306 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58441c6970 con 0x7f5844092e30 2021-02-10T11:41:40.987 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.987 7f584307d700 1 --2- 172.21.2.112:0/910539306 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5844092e30 0x7f58441c6320 secure :-1 s=READY pgs=2246 cs=0 l=1 rev1=1 rx=0x7f5834006ed0 tx=0x7f583400b8e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:40.987 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.987 7f583bfff700 1 -- 172.21.2.112:0/910539306 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5834006240 con 0x7f5844092e30 2021-02-10T11:41:40.987 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.987 7f583bfff700 1 -- 172.21.2.112:0/910539306 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58340054b0 con 0x7f5844092e30 2021-02-10T11:41:40.987 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.987 7f583bfff700 1 -- 172.21.2.112:0/910539306 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5834005990 con 0x7f5844092e30 2021-02-10T11:41:40.988 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.987 7f584937d700 1 -- 172.21.2.112:0/910539306 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58441c6970 con 0x7f5844092e30 2021-02-10T11:41:40.988 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.987 7f584937d700 1 -- 172.21.2.112:0/910539306 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58441c6970 con 0x7f5844092e30 2021-02-10T11:41:40.988 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.988 7f583bfff700 1 -- 172.21.2.112:0/910539306 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5834006030 con 0x7f5844092e30 2021-02-10T11:41:40.988 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.988 7f583bfff700 1 --2- 172.21.2.112:0/910539306 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f58280186c0 0x7f582801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:40.988 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.988 7f583bfff700 1 -- 172.21.2.112:0/910539306 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f58340325d0 con 0x7f5844092e30 2021-02-10T11:41:40.988 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.988 7f584287c700 1 --2- 172.21.2.112:0/910539306 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f58280186c0 0x7f582801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:40.988 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:40.988 7f584287c700 1 --2- 172.21.2.112:0/910539306 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f58280186c0 0x7f582801abc0 crc :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:41.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:41.018 7f584937d700 1 -- 172.21.2.112:0/910539306 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f584407ed50 con 0x7f5844092e30 2021-02-10T11:41:41.018 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:41.021 7f583bfff700 1 -- 172.21.2.112:0/910539306 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5834000de0 con 0x7f5844092e30 2021-02-10T11:41:41.149 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:41.152 7f584937d700 1 -- 172.21.2.112:0/910539306 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f58440ce3e0 con 0x7f5844092e30 2021-02-10T11:41:41.150 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:41.152 7f583bfff700 1 -- 172.21.2.112:0/910539306 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) v1 ==== 94+0+2098 (secure 0 0 0) 0x7f5834006030 con 0x7f5844092e30 2021-02-10T11:41:41.150 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 97 2021-02-10T11:41:41.150 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:41:41.150 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":97,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10671,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/668146927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":668146927},{"type":"v1","addr":"172.21.2.112:6833","nonce":668146927}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":93}],"filesystems":[{"mdsmap":{"epoch":97,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:41:23.731569","modified":"2021-02-10 11:41:25.740952","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8008},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8008":{"gid":8008,"name":"b","rank":0,"incarnation":94,"state":"up:active","state_seq":3,"addr":"172.21.2.120:6835/862287497","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":862287497},{"type":"v1","addr":"172.21.2.120:6835","nonce":862287497}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":12}]} 2021-02-10T11:41:41.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:41.154 7f584937d700 1 -- 172.21.2.112:0/910539306 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f58280186c0 msgr2=0x7f582801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:41.152 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:41.154 7f584937d700 1 --2- 172.21.2.112:0/910539306 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f58280186c0 0x7f582801abc0 crc :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:41.152 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:41.154 7f584937d700 1 -- 172.21.2.112:0/910539306 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5844092e30 msgr2=0x7f58441c6320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:41.152 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:41.154 7f584937d700 1 --2- 172.21.2.112:0/910539306 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5844092e30 0x7f58441c6320 secure :-1 s=READY pgs=2246 cs=0 l=1 rev1=1 rx=0x7f5834006ed0 tx=0x7f583400b8e0).stop 2021-02-10T11:41:41.152 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:41.154 7f584937d700 1 -- 172.21.2.112:0/910539306 shutdown_connections 2021-02-10T11:41:41.152 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:41.154 7f584937d700 1 --2- 172.21.2.112:0/910539306 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f58280186c0 0x7f582801abc0 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:41.152 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:41.154 7f584937d700 1 --2- 172.21.2.112:0/910539306 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f58440964a0 0x7f58440968a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:41.153 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:41.154 7f584937d700 1 --2- 172.21.2.112:0/910539306 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5844096ec0 0x7f5844084f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:41.153 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:41.154 7f584937d700 1 --2- 172.21.2.112:0/910539306 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5844092e30 0x7f58441c6320 unknown :-1 s=CLOSED pgs=2246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:41.153 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:41.154 7f584937d700 1 -- 172.21.2.112:0/910539306 >> 172.21.2.112:0/910539306 conn(0x7f58440c3bd0 msgr2=0x7f5844089210 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:41.153 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:41.154 7f584937d700 1 -- 172.21.2.112:0/910539306 shutdown_connections 2021-02-10T11:41:41.153 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:41.155 7f584937d700 1 -- 172.21.2.112:0/910539306 wait complete. 2021-02-10T11:41:41.362 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 97, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:41:23.731569', 'modified': '2021-02-10 11:41:25.740952', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8008}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8008': {'gid': 8008, 'name': 'b', 'rank': 0, 'incarnation': 94, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.120:6835/862287497', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.120:6834', 'nonce': 862287497}, {'type': 'v1', 'addr': '172.21.2.120:6835', 'nonce': 862287497}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [25], 'metadata_pool': 24, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:41:41.362 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:41:41.362 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:41:41.761 INFO:teuthology.orchestra.run.gibba020.stdout:{ 2021-02-10T11:41:41.762 INFO:teuthology.orchestra.run.gibba020.stdout: "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", 2021-02-10T11:41:41.762 INFO:teuthology.orchestra.run.gibba020.stdout: "whoami": 0, 2021-02-10T11:41:41.762 INFO:teuthology.orchestra.run.gibba020.stdout: "id": 8008, 2021-02-10T11:41:41.762 INFO:teuthology.orchestra.run.gibba020.stdout: "want_state": "up:active", 2021-02-10T11:41:41.762 INFO:teuthology.orchestra.run.gibba020.stdout: "state": "up:active", 2021-02-10T11:41:41.762 INFO:teuthology.orchestra.run.gibba020.stdout: "rank_uptime": 18.030904150000001, 2021-02-10T11:41:41.762 INFO:teuthology.orchestra.run.gibba020.stdout: "mdsmap_epoch": 97, 2021-02-10T11:41:41.762 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch": 117, 2021-02-10T11:41:41.762 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch_barrier": 117, 2021-02-10T11:41:41.762 INFO:teuthology.orchestra.run.gibba020.stdout: "uptime": 23.678212680000001 2021-02-10T11:41:41.762 INFO:teuthology.orchestra.run.gibba020.stdout:} 2021-02-10T11:41:41.766 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", "whoami": 0, "id": 8008, "want_state": "up:active", "state": "up:active", "rank_uptime": 18.030904150000001, "mdsmap_epoch": 97, "osdmap_epoch": 117, "osdmap_epoch_barrier": 117, "uptime": 23.678212680000001 } 2021-02-10T11:41:41.766 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:41:41.766 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2021-02-10T11:41:41.766 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.1 at ubuntu@gibba012.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2021-02-10T11:41:41.767 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:41.767 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:41:42.102 DEBUG:teuthology.orchestra.run.gibba012:> sudo modprobe fuse 2021-02-10T11:41:42.447 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:41:42.447 DEBUG:teuthology.orchestra.run.gibba012:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:41:42.790 INFO:teuthology.orchestra.run.gibba012.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:41:42.790 INFO:teuthology.orchestra.run.gibba012.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:41:42.791 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:41:42.791 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:42.792 DEBUG:teuthology.orchestra.run.gibba012:> ls /sys/fs/fuse/connections 2021-02-10T11:41:43.125 INFO:teuthology.orchestra.run.gibba012.stdout:42 2021-02-10T11:41:43.126 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [42] 2021-02-10T11:41:43.126 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.1 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:41:43.129 DEBUG:teuthology.orchestra.run.gibba012:> sudo modprobe fuse 2021-02-10T11:41:43.472 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:41:43.472 DEBUG:teuthology.orchestra.run.gibba012:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:41:43.539 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba012.stderr:2021-02-10 11:41:43.541 7f738a764f80 -1 init, newargv = 0x55836803dd70 newargc=9 2021-02-10T11:41:43.539 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba012.stderr:ceph-fuse[162110]: starting ceph client 2021-02-10T11:41:43.545 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba012.stderr:ceph-fuse[162110]: starting fuse 2021-02-10T11:41:43.814 INFO:teuthology.orchestra.run.gibba012.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:41:43.814 INFO:teuthology.orchestra.run.gibba012.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:41:43.815 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:41:43.815 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:43.815 DEBUG:teuthology.orchestra.run.gibba012:> ls /sys/fs/fuse/connections 2021-02-10T11:41:44.150 INFO:teuthology.orchestra.run.gibba012.stdout:42 2021-02-10T11:41:44.150 INFO:teuthology.orchestra.run.gibba012.stdout:43 2021-02-10T11:41:44.150 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42, 43] 2021-02-10T11:41:44.150 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:44.150 DEBUG:teuthology.orchestra.run.gibba012:> sudo python3 -c ' 2021-02-10T11:41:44.150 DEBUG:teuthology.orchestra.run.gibba012:> import glob 2021-02-10T11:41:44.150 DEBUG:teuthology.orchestra.run.gibba012:> import re 2021-02-10T11:41:44.151 DEBUG:teuthology.orchestra.run.gibba012:> import os 2021-02-10T11:41:44.151 DEBUG:teuthology.orchestra.run.gibba012:> import subprocess 2021-02-10T11:41:44.151 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:41:44.151 DEBUG:teuthology.orchestra.run.gibba012:> def find_socket(client_name): 2021-02-10T11:41:44.151 DEBUG:teuthology.orchestra.run.gibba012:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2021-02-10T11:41:44.151 DEBUG:teuthology.orchestra.run.gibba012:> files = glob.glob(asok_path) 2021-02-10T11:41:44.151 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:41:44.151 DEBUG:teuthology.orchestra.run.gibba012:> # Given a non-glob path, it better be there 2021-02-10T11:41:44.151 DEBUG:teuthology.orchestra.run.gibba012:> if "*" not in asok_path: 2021-02-10T11:41:44.151 DEBUG:teuthology.orchestra.run.gibba012:> assert(len(files) == 1) 2021-02-10T11:41:44.151 DEBUG:teuthology.orchestra.run.gibba012:> return files[0] 2021-02-10T11:41:44.151 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:41:44.151 DEBUG:teuthology.orchestra.run.gibba012:> for f in files: 2021-02-10T11:41:44.151 DEBUG:teuthology.orchestra.run.gibba012:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:41:44.151 DEBUG:teuthology.orchestra.run.gibba012:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:41:44.152 DEBUG:teuthology.orchestra.run.gibba012:> return f 2021-02-10T11:41:44.152 DEBUG:teuthology.orchestra.run.gibba012:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:41:44.152 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:41:44.152 DEBUG:teuthology.orchestra.run.gibba012:> print(find_socket("client.1")) 2021-02-10T11:41:44.152 DEBUG:teuthology.orchestra.run.gibba012:> ' 2021-02-10T11:41:44.513 INFO:teuthology.orchestra.run.gibba012.stdout:/var/run/ceph/ceph-client.1.162110.asok 2021-02-10T11:41:44.516 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.162110.asok 2021-02-10T11:41:44.516 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:44.516 DEBUG:teuthology.orchestra.run.gibba012:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.162110.asok status 2021-02-10T11:41:44.908 INFO:teuthology.orchestra.run.gibba012.stdout:{ 2021-02-10T11:41:44.908 INFO:teuthology.orchestra.run.gibba012.stdout: "metadata": { 2021-02-10T11:41:44.908 INFO:teuthology.orchestra.run.gibba012.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:41:44.908 INFO:teuthology.orchestra.run.gibba012.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:41:44.908 INFO:teuthology.orchestra.run.gibba012.stdout: "entity_id": "1", 2021-02-10T11:41:44.908 INFO:teuthology.orchestra.run.gibba012.stdout: "hostname": "gibba012", 2021-02-10T11:41:44.909 INFO:teuthology.orchestra.run.gibba012.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2021-02-10T11:41:44.909 INFO:teuthology.orchestra.run.gibba012.stdout: "pid": "162110", 2021-02-10T11:41:44.909 INFO:teuthology.orchestra.run.gibba012.stdout: "root": "/" 2021-02-10T11:41:44.909 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:41:44.909 INFO:teuthology.orchestra.run.gibba012.stdout: "dentry_count": 0, 2021-02-10T11:41:44.909 INFO:teuthology.orchestra.run.gibba012.stdout: "dentry_pinned_count": 0, 2021-02-10T11:41:44.909 INFO:teuthology.orchestra.run.gibba012.stdout: "id": 10878, 2021-02-10T11:41:44.909 INFO:teuthology.orchestra.run.gibba012.stdout: "inst": { 2021-02-10T11:41:44.909 INFO:teuthology.orchestra.run.gibba012.stdout: "name": { 2021-02-10T11:41:44.909 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "client", 2021-02-10T11:41:44.909 INFO:teuthology.orchestra.run.gibba012.stdout: "num": 10878 2021-02-10T11:41:44.910 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:41:44.910 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": { 2021-02-10T11:41:44.910 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:41:44.910 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:0", 2021-02-10T11:41:44.910 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 3716806301 2021-02-10T11:41:44.910 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:41:44.910 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:41:44.910 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": { 2021-02-10T11:41:44.910 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:41:44.910 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:0", 2021-02-10T11:41:44.910 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 3716806301 2021-02-10T11:41:44.911 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:41:44.911 INFO:teuthology.orchestra.run.gibba012.stdout: "inst_str": "client.10878 172.21.2.112:0/3716806301", 2021-02-10T11:41:44.911 INFO:teuthology.orchestra.run.gibba012.stdout: "addr_str": "172.21.2.112:0/3716806301", 2021-02-10T11:41:44.911 INFO:teuthology.orchestra.run.gibba012.stdout: "inode_count": 1, 2021-02-10T11:41:44.911 INFO:teuthology.orchestra.run.gibba012.stdout: "mds_epoch": 97, 2021-02-10T11:41:44.911 INFO:teuthology.orchestra.run.gibba012.stdout: "osd_epoch": 117, 2021-02-10T11:41:44.911 INFO:teuthology.orchestra.run.gibba012.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:41:44.911 INFO:teuthology.orchestra.run.gibba012.stdout: "blacklisted": false 2021-02-10T11:41:44.911 INFO:teuthology.orchestra.run.gibba012.stdout:} 2021-02-10T11:41:44.912 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:44.912 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:41:44.913 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:41:45.247 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:41:45.248 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:41:45.248 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:45.248 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:41:45.596 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2021-02-10T11:41:46.027 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:46.013 7fd012f87700 1 Processor -- start 2021-02-10T11:41:46.027 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:46.013 7fd012f87700 1 -- start start 2021-02-10T11:41:46.027 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:46.013 7fd012f87700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fd00c0d1880 0x7fd00c1b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:46.027 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:46.013 7fd012f87700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd00c1b0af0 0x7fd00c108950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:46.028 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:46.013 7fd012f87700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd00c0fe1d0 con 0x7fd00c0d1c80 2021-02-10T11:41:46.028 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:46.013 7fd012f87700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd00c109a70 con 0x7fd00c0d1880 2021-02-10T11:41:46.028 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:46.013 7fd012f87700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd00c109760 con 0x7fd00c1b0af0 2021-02-10T11:41:46.028 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:46.013 7fd0115a4700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fd00c0d1c80 0x7fd00c108f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:55186/0 (socket says 172.21.2.112:55186) 2021-02-10T11:41:46.028 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:46.013 7fd0115a4700 1 -- 172.21.2.112:0/1179446070 learned_addr learned my addr 172.21.2.112:0/1179446070 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:41:46.028 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:46.013 7fd010da3700 1 --2- 172.21.2.112:0/1179446070 >> v2:172.21.2.112:3300/0 conn(0x7fd00c0d1880 0x7fd00c1b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:46.028 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:46.014 7fd00b7fe700 1 -- 172.21.2.112:0/1179446070 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3149151627 0 0) 0x7fd004001180 con 0x7fd00c0d1c80 2021-02-10T11:41:46.028 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:46.014 7fd00b7fe700 1 -- 172.21.2.112:0/1179446070 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcff8002fe0 con 0x7fd00c0d1c80 2021-02-10T11:41:46.028 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:46.014 7fd010da3700 1 -- 172.21.2.112:0/1179446070 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd00c1b0af0 msgr2=0x7fd00c108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:46.028 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:46.014 7fd010da3700 1 --2- 172.21.2.112:0/1179446070 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd00c1b0af0 0x7fd00c108950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:46.028 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:46.014 7fd010da3700 1 -- 172.21.2.112:0/1179446070 >> v1:172.21.2.112:6789/0 conn(0x7fd00c0d1c80 legacy=0x7fd00c108f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:46.028 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:46.014 7fd010da3700 1 -- 172.21.2.112:0/1179446070 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd00c088070 con 0x7fd00c0d1880 2021-02-10T11:41:46.028 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:46.014 7fd010da3700 1 --2- 172.21.2.112:0/1179446070 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd00c0d1880 0x7fd00c1b04d0 secure :-1 s=READY pgs=2249 cs=0 l=1 rev1=1 rx=0x7fcff4006650 tx=0x7fcff4009a80).ready entity=mon.0 client_cookie=67e73f37de335f5d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:46.029 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:46.014 7fd00b7fe700 1 -- 172.21.2.112:0/1179446070 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcff40009b0 con 0x7fd00c0d1880 2021-02-10T11:41:46.029 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:46.014 7fd00b7fe700 1 -- 172.21.2.112:0/1179446070 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcff400c0c0 con 0x7fd00c0d1880 2021-02-10T11:41:46.029 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:46.014 7fd00b7fe700 1 -- 172.21.2.112:0/1179446070 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcff400c5e0 con 0x7fd00c0d1880 2021-02-10T11:41:46.029 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:46.014 7fd012f87700 1 -- 172.21.2.112:0/1179446070 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd00c0d1880 msgr2=0x7fd00c1b04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:46.029 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:46.014 7fd012f87700 1 --2- 172.21.2.112:0/1179446070 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd00c0d1880 0x7fd00c1b04d0 secure :-1 s=READY pgs=2249 cs=0 l=1 rev1=1 rx=0x7fcff4006650 tx=0x7fcff4009a80).stop 2021-02-10T11:41:46.029 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:46.015 7fd012f87700 1 -- 172.21.2.112:0/1179446070 shutdown_connections 2021-02-10T11:41:46.029 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:46.015 7fd012f87700 1 --2- 172.21.2.112:0/1179446070 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd00c1b0af0 0x7fd00c108950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:46.029 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:46.015 7fd012f87700 1 --2- 172.21.2.112:0/1179446070 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd00c0d1880 0x7fd00c1b04d0 unknown :-1 s=CLOSED pgs=2249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:46.029 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:46.015 7fd012f87700 1 -- 172.21.2.112:0/1179446070 >> 172.21.2.112:0/1179446070 conn(0x7fd00c0ce430 msgr2=0x7fd00c0d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:46.029 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:46.015 7fd012f87700 1 -- 172.21.2.112:0/1179446070 shutdown_connections 2021-02-10T11:41:46.029 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:46.015 7fd012f87700 1 -- 172.21.2.112:0/1179446070 wait complete. 2021-02-10T11:41:46.031 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:46.030 7fd012f87700 1 Processor -- start 2021-02-10T11:41:46.031 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:46.030 7fd012f87700 1 -- start start 2021-02-10T11:41:46.031 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:46.030 7fd012f87700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd00c0ad520 0x7fd00c0ad920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:46.031 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:46.030 7fd012f87700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd00c0adf40 0x7fd00c097c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:46.031 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:46.030 7fd012f87700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd00c098250 0x7fd00c098650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:46.031 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:46.030 7fd012f87700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd00c098c70 con 0x7fd00c0adf40 2021-02-10T11:41:46.031 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:46.030 7fd012f87700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd00c0910f0 con 0x7fd00c0ad520 2021-02-10T11:41:46.031 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:46.030 7fd012f87700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd00c091310 con 0x7fd00c098250 2021-02-10T11:41:46.031 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:46.031 7fd0115a4700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd00c098250 0x7fd00c098650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:46.031 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:46.031 7fd00bfff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd00c0adf40 0x7fd00c097c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:46.032 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:46.031 7fd0115a4700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd00c098250 0x7fd00c098650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3301/0 says I am v2:172.21.2.112:55408/0 (socket says 172.21.2.112:55408) 2021-02-10T11:41:46.032 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:46.031 7fd00bfff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd00c0adf40 0x7fd00c097c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:55730/0 (socket says 172.21.2.112:55730) 2021-02-10T11:41:46.032 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:46.031 7fd0115a4700 1 -- 172.21.2.112:0/2577804553 learned_addr learned my addr 172.21.2.112:0/2577804553 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:41:46.032 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:46.031 7fd010da3700 1 --2- 172.21.2.112:0/2577804553 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd00c0ad520 0x7fd00c0ad920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:46.032 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:46.031 7fd00bfff700 1 -- 172.21.2.112:0/2577804553 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd00c098250 msgr2=0x7fd00c098650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:46.032 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:46.031 7fd00bfff700 1 --2- 172.21.2.112:0/2577804553 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd00c098250 0x7fd00c098650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:46.032 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:46.031 7fd00bfff700 1 -- 172.21.2.112:0/2577804553 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd00c0ad520 msgr2=0x7fd00c0ad920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:46.032 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:46.031 7fd00bfff700 1 --2- 172.21.2.112:0/2577804553 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd00c0ad520 0x7fd00c0ad920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:46.032 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:46.031 7fd00bfff700 1 -- 172.21.2.112:0/2577804553 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd00c19a3b0 con 0x7fd00c0adf40 2021-02-10T11:41:46.032 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:46.031 7fd00bfff700 1 --2- 172.21.2.112:0/2577804553 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd00c0adf40 0x7fd00c097c30 secure :-1 s=READY pgs=2250 cs=0 l=1 rev1=1 rx=0x7fcffc00b8a0 tx=0x7fcffc00bc40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:46.032 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:46.031 7fd0097fa700 1 -- 172.21.2.112:0/2577804553 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcffc006a00 con 0x7fd00c0adf40 2021-02-10T11:41:46.032 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:46.031 7fd0097fa700 1 -- 172.21.2.112:0/2577804553 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcffc007070 con 0x7fd00c0adf40 2021-02-10T11:41:46.032 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:46.031 7fd0097fa700 1 -- 172.21.2.112:0/2577804553 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcffc007590 con 0x7fd00c0adf40 2021-02-10T11:41:46.033 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:46.031 7fd012f87700 1 -- 172.21.2.112:0/2577804553 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd00c19a3b0 con 0x7fd00c0adf40 2021-02-10T11:41:46.033 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:46.031 7fd012f87700 1 -- 172.21.2.112:0/2577804553 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd00c19a940 con 0x7fd00c0adf40 2021-02-10T11:41:46.033 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:46.032 7fd0097fa700 1 -- 172.21.2.112:0/2577804553 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fcffc001010 con 0x7fd00c0adf40 2021-02-10T11:41:46.033 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:46.032 7fd0097fa700 1 --2- 172.21.2.112:0/2577804553 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcff00186c0 0x7fcff001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:46.033 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:46.032 7fd0097fa700 1 -- 172.21.2.112:0/2577804553 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fcffc0326d0 con 0x7fd00c0adf40 2021-02-10T11:41:46.033 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:46.032 7fd010da3700 1 --2- 172.21.2.112:0/2577804553 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcff00186c0 0x7fcff001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:46.033 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:46.033 7fd010da3700 1 --2- 172.21.2.112:0/2577804553 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcff00186c0 0x7fcff001abc0 crc :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:46.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:46.063 7fd012f87700 1 -- 172.21.2.112:0/2577804553 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd00c082a70 con 0x7fd00c0adf40 2021-02-10T11:41:46.063 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:46.065 7fd0097fa700 1 -- 172.21.2.112:0/2577804553 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fcffc001f00 con 0x7fd00c0adf40 2021-02-10T11:41:46.201 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:46.204 7fd012f87700 1 -- 172.21.2.112:0/2577804553 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fd00c082a70 con 0x7fd00c0adf40 2021-02-10T11:41:47.200 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:47.202 7fd0097fa700 1 -- 172.21.2.112:0/2577804553 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v98) v1 ==== 136+0+0 (secure 0 0 0) 0x7fcffc000cc0 con 0x7fd00c0adf40 2021-02-10T11:41:47.223 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:47.225 7fd012f87700 1 -- 172.21.2.112:0/2577804553 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcff00186c0 msgr2=0x7fcff001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:47.223 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:47.225 7fd012f87700 1 --2- 172.21.2.112:0/2577804553 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcff00186c0 0x7fcff001abc0 crc :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:47.223 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:47.225 7fd012f87700 1 -- 172.21.2.112:0/2577804553 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd00c0adf40 msgr2=0x7fd00c097c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:47.223 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:47.225 7fd012f87700 1 --2- 172.21.2.112:0/2577804553 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd00c0adf40 0x7fd00c097c30 secure :-1 s=READY pgs=2250 cs=0 l=1 rev1=1 rx=0x7fcffc00b8a0 tx=0x7fcffc00bc40).stop 2021-02-10T11:41:47.224 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:47.225 7fd012f87700 1 -- 172.21.2.112:0/2577804553 shutdown_connections 2021-02-10T11:41:47.224 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:47.225 7fd012f87700 1 --2- 172.21.2.112:0/2577804553 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcff00186c0 0x7fcff001abc0 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:47.224 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:47.225 7fd012f87700 1 --2- 172.21.2.112:0/2577804553 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd00c098250 0x7fd00c098650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:47.224 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:47.225 7fd012f87700 1 --2- 172.21.2.112:0/2577804553 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd00c0adf40 0x7fd00c097c30 unknown :-1 s=CLOSED pgs=2250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:47.224 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:47.225 7fd012f87700 1 --2- 172.21.2.112:0/2577804553 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd00c0ad520 0x7fd00c0ad920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:47.224 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:47.225 7fd012f87700 1 -- 172.21.2.112:0/2577804553 >> 172.21.2.112:0/2577804553 conn(0x7fd00c18e810 msgr2=0x7fd00c18c290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:47.224 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:47.225 7fd012f87700 1 -- 172.21.2.112:0/2577804553 shutdown_connections 2021-02-10T11:41:47.224 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:47.226 7fd012f87700 1 -- 172.21.2.112:0/2577804553 wait complete. 2021-02-10T11:41:47.231 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs volume ls 2021-02-10T11:41:47.660 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:47.644 7f16e3f7e700 1 Processor -- start 2021-02-10T11:41:47.661 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:47.645 7f16e3f7e700 1 -- start start 2021-02-10T11:41:47.661 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:47.645 7f16e3f7e700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f16dc0d0340 0x7f16dc0ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:47.661 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:47.645 7f16e3f7e700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f16dc0d1c70 0x7f16dc0efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:47.661 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:47.645 7f16e3f7e700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16dc0f0b10 con 0x7f16dc0d1870 2021-02-10T11:41:47.661 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:47.645 7f16e3f7e700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16dc0e5ed0 con 0x7f16dc0d0340 2021-02-10T11:41:47.661 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:47.645 7f16e3f7e700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16dc1b0be0 con 0x7f16dc0d1c70 2021-02-10T11:41:47.661 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:47.646 7f16e1599700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f16dc0d0340 0x7f16dc0ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:47.661 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:47.646 7f16e259b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f16dc0d1c70 0x7f16dc0efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:47.661 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:47.646 7f16e1d9a700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7f16dc0d1870 0x7f16dc1b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:48754/0 (socket says 172.21.2.112:48754) 2021-02-10T11:41:47.661 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:47.646 7f16e1d9a700 1 -- 172.21.2.112:0/440469831 learned_addr learned my addr 172.21.2.112:0/440469831 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:41:47.661 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:47.647 7f16e259b700 1 -- 172.21.2.112:0/440469831 >> v2:172.21.2.120:3300/0 conn(0x7f16dc0d0340 msgr2=0x7f16dc0ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:47.661 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:47.647 7f16e259b700 1 --2- 172.21.2.112:0/440469831 >> v2:172.21.2.120:3300/0 conn(0x7f16dc0d0340 0x7f16dc0ef640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:47.662 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:47.647 7f16e259b700 1 -- 172.21.2.112:0/440469831 >> v1:172.21.2.120:6789/0 conn(0x7f16dc0d1870 legacy=0x7f16dc1b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:47.662 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:47.647 7f16e259b700 1 -- 172.21.2.112:0/440469831 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16dc0f20a0 con 0x7f16dc0d1c70 2021-02-10T11:41:47.662 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:47.647 7f16e1599700 1 --2- 172.21.2.112:0/440469831 >> v2:172.21.2.120:3300/0 conn(0x7f16dc0d0340 0x7f16dc0ef640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:41:47.662 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:47.647 7f16e259b700 1 --2- 172.21.2.112:0/440469831 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f16dc0d1c70 0x7f16dc0efc60 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7f16d4006e50 tx=0x7f16d4009e70).ready entity=mon.2 client_cookie=afbdcc496c696b34 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:47.662 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:47.648 7f16e0d98700 1 -- 172.21.2.112:0/440469831 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f16d40066f0 con 0x7f16dc0d1c70 2021-02-10T11:41:47.662 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:47.648 7f16e0d98700 1 -- 172.21.2.112:0/440469831 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16d4003030 con 0x7f16dc0d1c70 2021-02-10T11:41:47.662 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:47.648 7f16e0d98700 1 -- 172.21.2.112:0/440469831 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f16d400b700 con 0x7f16dc0d1c70 2021-02-10T11:41:47.662 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:47.648 7f16e3f7e700 1 -- 172.21.2.112:0/440469831 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f16dc0d1c70 msgr2=0x7f16dc0efc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:47.662 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:47.648 7f16e3f7e700 1 --2- 172.21.2.112:0/440469831 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f16dc0d1c70 0x7f16dc0efc60 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7f16d4006e50 tx=0x7f16d4009e70).stop 2021-02-10T11:41:47.662 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:47.648 7f16e3f7e700 1 -- 172.21.2.112:0/440469831 shutdown_connections 2021-02-10T11:41:47.662 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:47.648 7f16e3f7e700 1 --2- 172.21.2.112:0/440469831 >> v2:172.21.2.120:3300/0 conn(0x7f16dc0d0340 0x7f16dc0ef640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:47.662 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:47.648 7f16e3f7e700 1 --2- 172.21.2.112:0/440469831 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f16dc0d1c70 0x7f16dc0efc60 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:47.663 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:47.648 7f16e3f7e700 1 -- 172.21.2.112:0/440469831 >> 172.21.2.112:0/440469831 conn(0x7f16dc0ce450 msgr2=0x7f16dc0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:47.663 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:47.649 7f16e3f7e700 1 -- 172.21.2.112:0/440469831 shutdown_connections 2021-02-10T11:41:47.663 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:47.649 7f16e3f7e700 1 -- 172.21.2.112:0/440469831 wait complete. 2021-02-10T11:41:47.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:47.663 7f16e3f7e700 1 Processor -- start 2021-02-10T11:41:47.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:47.663 7f16e3f7e700 1 -- start start 2021-02-10T11:41:47.665 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:47.663 7f16e3f7e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f16dc0b7510 0x7f16dc0b7910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:47.665 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:47.663 7f16e3f7e700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f16dc09d530 0x7f16dc09d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:47.665 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:47.663 7f16e3f7e700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f16dc09df50 0x7f16dc09eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:47.665 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:47.663 7f16e3f7e700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16dc0b8140 con 0x7f16dc0b7510 2021-02-10T11:41:47.665 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:47.663 7f16e3f7e700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16dc09e750 con 0x7f16dc09d530 2021-02-10T11:41:47.665 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:47.663 7f16e3f7e700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16dc09e350 con 0x7f16dc09df50 2021-02-10T11:41:47.665 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:47.664 7f16e1d9a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f16dc0b7510 0x7f16dc0b7910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:47.665 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:47.664 7f16e1d9a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f16dc0b7510 0x7f16dc0b7910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:55738/0 (socket says 172.21.2.112:55738) 2021-02-10T11:41:47.665 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:47.664 7f16e1d9a700 1 -- 172.21.2.112:0/1748785091 learned_addr learned my addr 172.21.2.112:0/1748785091 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:41:47.665 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:47.664 7f16e1599700 1 --2- 172.21.2.112:0/1748785091 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f16dc09d530 0x7f16dc09d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:47.665 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:47.664 7f16e259b700 1 --2- 172.21.2.112:0/1748785091 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f16dc09df50 0x7f16dc09eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:47.665 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:47.664 7f16e1d9a700 1 -- 172.21.2.112:0/1748785091 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f16dc09df50 msgr2=0x7f16dc09eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:47.666 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:47.664 7f16e1d9a700 1 --2- 172.21.2.112:0/1748785091 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f16dc09df50 0x7f16dc09eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:47.666 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:47.664 7f16e1d9a700 1 -- 172.21.2.112:0/1748785091 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f16dc09d530 msgr2=0x7f16dc09d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:47.666 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:47.664 7f16e1d9a700 1 --2- 172.21.2.112:0/1748785091 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f16dc09d530 0x7f16dc09d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:47.666 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:47.664 7f16e1d9a700 1 -- 172.21.2.112:0/1748785091 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16dc09f2c0 con 0x7f16dc0b7510 2021-02-10T11:41:47.666 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:47.664 7f16e1599700 1 --2- 172.21.2.112:0/1748785091 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f16dc09d530 0x7f16dc09d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:41:47.666 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:47.664 7f16e259b700 1 --2- 172.21.2.112:0/1748785091 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f16dc09df50 0x7f16dc09eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:41:47.666 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:47.664 7f16e1d9a700 1 --2- 172.21.2.112:0/1748785091 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f16dc0b7510 0x7f16dc0b7910 secure :-1 s=READY pgs=2251 cs=0 l=1 rev1=1 rx=0x7f16d800a580 tx=0x7f16d800a920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:47.666 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:47.665 7f16d27fc700 1 -- 172.21.2.112:0/1748785091 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f16d800ad10 con 0x7f16dc0b7510 2021-02-10T11:41:47.666 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:47.665 7f16d27fc700 1 -- 172.21.2.112:0/1748785091 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16d8000f00 con 0x7f16dc0b7510 2021-02-10T11:41:47.666 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:47.665 7f16d27fc700 1 -- 172.21.2.112:0/1748785091 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f16d8004510 con 0x7f16dc0b7510 2021-02-10T11:41:47.666 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:47.665 7f16e3f7e700 1 -- 172.21.2.112:0/1748785091 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16dc09f2c0 con 0x7f16dc0b7510 2021-02-10T11:41:47.666 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:47.665 7f16e3f7e700 1 -- 172.21.2.112:0/1748785091 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16dc09f2c0 con 0x7f16dc0b7510 2021-02-10T11:41:47.666 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:47.665 7f16d27fc700 1 -- 172.21.2.112:0/1748785091 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f16d8008000 con 0x7f16dc0b7510 2021-02-10T11:41:47.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:47.666 7f16d27fc700 1 --2- 172.21.2.112:0/1748785091 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f16c4018730 0x7f16c401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:47.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:47.666 7f16d27fc700 1 -- 172.21.2.112:0/1748785091 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f16d8008000 con 0x7f16dc0b7510 2021-02-10T11:41:47.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:47.666 7f16e1599700 1 --2- 172.21.2.112:0/1748785091 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f16c4018730 0x7f16c401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:47.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:47.667 7f16e1599700 1 --2- 172.21.2.112:0/1748785091 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f16c4018730 0x7f16c401ac10 crc :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:47.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:47.695 7f16e3f7e700 1 -- 172.21.2.112:0/1748785091 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16dc0ebc50 con 0x7f16dc0b7510 2021-02-10T11:41:47.695 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:47.698 7f16d27fc700 1 -- 172.21.2.112:0/1748785091 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f16d8000e90 con 0x7f16dc0b7510 2021-02-10T11:41:47.831 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:47.833 7f16e3f7e700 1 -- 172.21.2.112:0/1748785091 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs volume ls", "target": ["mgr", ""]}) v1 -- 0x7f16dc0f0b10 con 0x7f16c4018730 2021-02-10T11:41:47.832 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:47.835 7f16d27fc700 1 -- 172.21.2.112:0/1748785091 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+41 (crc 0 0 0) 0x7f16cc006de0 con 0x7f16c4018730 2021-02-10T11:41:47.832 INFO:teuthology.orchestra.run.gibba012.stdout:[ 2021-02-10T11:41:47.832 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:41:47.832 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "cephfs" 2021-02-10T11:41:47.832 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:41:47.832 INFO:teuthology.orchestra.run.gibba012.stdout:] 2021-02-10T11:41:47.834 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:47.836 7f16e3f7e700 1 -- 172.21.2.112:0/1748785091 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f16c4018730 msgr2=0x7f16c401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:47.834 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:47.836 7f16e3f7e700 1 --2- 172.21.2.112:0/1748785091 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f16c4018730 0x7f16c401ac10 crc :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:47.834 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:47.836 7f16e3f7e700 1 -- 172.21.2.112:0/1748785091 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f16dc0b7510 msgr2=0x7f16dc0b7910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:47.834 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:47.836 7f16e3f7e700 1 --2- 172.21.2.112:0/1748785091 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f16dc0b7510 0x7f16dc0b7910 secure :-1 s=READY pgs=2251 cs=0 l=1 rev1=1 rx=0x7f16d800a580 tx=0x7f16d800a920).stop 2021-02-10T11:41:47.834 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:47.836 7f16e3f7e700 1 -- 172.21.2.112:0/1748785091 shutdown_connections 2021-02-10T11:41:47.834 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:47.836 7f16e3f7e700 1 --2- 172.21.2.112:0/1748785091 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f16c4018730 0x7f16c401ac10 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:47.834 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:47.836 7f16e3f7e700 1 --2- 172.21.2.112:0/1748785091 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f16dc09d530 0x7f16dc09d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:47.834 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:47.836 7f16e3f7e700 1 --2- 172.21.2.112:0/1748785091 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f16dc09df50 0x7f16dc09eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:47.834 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:47.836 7f16e3f7e700 1 --2- 172.21.2.112:0/1748785091 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f16dc0b7510 0x7f16dc0b7910 unknown :-1 s=CLOSED pgs=2251 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:47.834 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:47.836 7f16e3f7e700 1 -- 172.21.2.112:0/1748785091 >> 172.21.2.112:0/1748785091 conn(0x7f16dc0b83b0 msgr2=0x7f16dc0a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:47.834 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:47.836 7f16e3f7e700 1 -- 172.21.2.112:0/1748785091 shutdown_connections 2021-02-10T11:41:47.835 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:47.836 7f16e3f7e700 1 -- 172.21.2.112:0/1748785091 wait complete. 2021-02-10T11:41:47.842 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:41:48.275 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.260 7f92ab832700 1 Processor -- start 2021-02-10T11:41:48.275 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.260 7f92ab832700 1 -- start start 2021-02-10T11:41:48.275 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.261 7f92ab832700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f92a40d1c70 0x7f92a40cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:48.275 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.261 7f92ab832700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f92a40cf580 0x7f92a40cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:48.276 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.261 7f92ab832700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92a40fe1c0 con 0x7f92a40d1870 2021-02-10T11:41:48.276 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.261 7f92ab832700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92a41b0cb0 con 0x7f92a40d1c70 2021-02-10T11:41:48.276 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.261 7f92ab832700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92a4109a60 con 0x7f92a40cf580 2021-02-10T11:41:48.276 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.261 7f92a9e4f700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f92a40cf580 0x7f92a40cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:48.276 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.261 7f92a8e4d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f92a40d1c70 0x7f92a40cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:48.276 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.261 7f92a8e4d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f92a40d1c70 0x7f92a40cef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:55746/0 (socket says 172.21.2.112:55746) 2021-02-10T11:41:48.276 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.261 7f92a8e4d700 1 -- 172.21.2.112:0/1241274194 learned_addr learned my addr 172.21.2.112:0/1241274194 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:41:48.276 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.261 7f92a8e4d700 1 -- 172.21.2.112:0/1241274194 >> v2:172.21.2.120:3300/0 conn(0x7f92a40cf580 msgr2=0x7f92a40cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:48.276 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.261 7f92a8e4d700 1 --2- 172.21.2.112:0/1241274194 >> v2:172.21.2.120:3300/0 conn(0x7f92a40cf580 0x7f92a40cf980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:48.276 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.261 7f92a8e4d700 1 -- 172.21.2.112:0/1241274194 >> v1:172.21.2.120:6789/0 conn(0x7f92a40d1870 legacy=0x7f92a41b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:48.276 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.261 7f92a8e4d700 1 -- 172.21.2.112:0/1241274194 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92a4088070 con 0x7f92a40d1c70 2021-02-10T11:41:48.277 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.262 7f92a8e4d700 1 --2- 172.21.2.112:0/1241274194 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f92a40d1c70 0x7f92a40cef60 secure :-1 s=READY pgs=2252 cs=0 l=1 rev1=1 rx=0x7f9298006700 tx=0x7f9298009b80).ready entity=mon.0 client_cookie=14bc34b3cd5faef4 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:48.277 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.262 7f9297fff700 1 -- 172.21.2.112:0/1241274194 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f929800c0c0 con 0x7f92a40d1c70 2021-02-10T11:41:48.277 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.262 7f9297fff700 1 -- 172.21.2.112:0/1241274194 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f929800c400 con 0x7f92a40d1c70 2021-02-10T11:41:48.277 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.262 7f9297fff700 1 -- 172.21.2.112:0/1241274194 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f929800c860 con 0x7f92a40d1c70 2021-02-10T11:41:48.277 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.262 7f92ab832700 1 -- 172.21.2.112:0/1241274194 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f92a40d1c70 msgr2=0x7f92a40cef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:48.277 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.262 7f92ab832700 1 --2- 172.21.2.112:0/1241274194 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f92a40d1c70 0x7f92a40cef60 secure :-1 s=READY pgs=2252 cs=0 l=1 rev1=1 rx=0x7f9298006700 tx=0x7f9298009b80).stop 2021-02-10T11:41:48.277 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.263 7f92ab832700 1 -- 172.21.2.112:0/1241274194 shutdown_connections 2021-02-10T11:41:48.277 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.263 7f92ab832700 1 --2- 172.21.2.112:0/1241274194 >> v2:172.21.2.120:3300/0 conn(0x7f92a40cf580 0x7f92a40cf980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:48.277 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.263 7f92ab832700 1 --2- 172.21.2.112:0/1241274194 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f92a40d1c70 0x7f92a40cef60 unknown :-1 s=CLOSED pgs=2252 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:48.277 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.263 7f92ab832700 1 -- 172.21.2.112:0/1241274194 >> 172.21.2.112:0/1241274194 conn(0x7f92a40ce420 msgr2=0x7f92a40d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:48.277 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.263 7f92ab832700 1 -- 172.21.2.112:0/1241274194 shutdown_connections 2021-02-10T11:41:48.277 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.263 7f92ab832700 1 -- 172.21.2.112:0/1241274194 wait complete. 2021-02-10T11:41:48.279 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.278 7f92ab832700 1 Processor -- start 2021-02-10T11:41:48.279 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.278 7f92ab832700 1 -- start start 2021-02-10T11:41:48.279 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.278 7f92ab832700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f92a41a79a0 0x7f92a41a7da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:48.279 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.278 7f92ab832700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f92a40bdb40 0x7f92a40bdf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:48.279 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.278 7f92ab832700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f92a4082ca0 0x7f92a40830a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:48.279 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.278 7f92ab832700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92a419d990 con 0x7f92a41a79a0 2021-02-10T11:41:48.279 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.278 7f92ab832700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92a419dbb0 con 0x7f92a4082ca0 2021-02-10T11:41:48.279 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.278 7f92ab832700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92a40836c0 con 0x7f92a40bdb40 2021-02-10T11:41:48.279 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.279 7f92a964e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f92a41a79a0 0x7f92a41a7da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:48.279 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.279 7f92a9e4f700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f92a4082ca0 0x7f92a40830a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:48.280 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.279 7f92a8e4d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f92a40bdb40 0x7f92a40bdf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:48.280 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.279 7f92a9e4f700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f92a4082ca0 0x7f92a40830a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3300/0 says I am v2:172.21.2.112:58036/0 (socket says 172.21.2.112:58036) 2021-02-10T11:41:48.280 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.279 7f92a9e4f700 1 -- 172.21.2.112:0/49250850 learned_addr learned my addr 172.21.2.112:0/49250850 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:41:48.280 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.279 7f92a964e700 1 -- 172.21.2.112:0/49250850 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f92a40bdb40 msgr2=0x7f92a40bdf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:48.280 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.279 7f92a964e700 1 --2- 172.21.2.112:0/49250850 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f92a40bdb40 0x7f92a40bdf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:48.280 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.279 7f92a964e700 1 -- 172.21.2.112:0/49250850 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f92a4082ca0 msgr2=0x7f92a40830a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:48.280 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.279 7f92a964e700 1 --2- 172.21.2.112:0/49250850 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f92a4082ca0 0x7f92a40830a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:48.280 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.279 7f92a964e700 1 -- 172.21.2.112:0/49250850 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92a40838e0 con 0x7f92a41a79a0 2021-02-10T11:41:48.280 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.279 7f92a9e4f700 1 --2- 172.21.2.112:0/49250850 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f92a4082ca0 0x7f92a40830a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:41:48.280 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.279 7f92a964e700 1 --2- 172.21.2.112:0/49250850 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f92a41a79a0 0x7f92a41a7da0 secure :-1 s=READY pgs=2253 cs=0 l=1 rev1=1 rx=0x7f92a000b510 tx=0x7f92a000b8b0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:48.280 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.280 7f9295ffb700 1 -- 172.21.2.112:0/49250850 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f92a0000960 con 0x7f92a41a79a0 2021-02-10T11:41:48.280 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.280 7f9295ffb700 1 -- 172.21.2.112:0/49250850 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92a000bca0 con 0x7f92a41a79a0 2021-02-10T11:41:48.280 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.280 7f9295ffb700 1 -- 172.21.2.112:0/49250850 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f92a0002500 con 0x7f92a41a79a0 2021-02-10T11:41:48.281 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.280 7f92ab832700 1 -- 172.21.2.112:0/49250850 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92a40838e0 con 0x7f92a41a79a0 2021-02-10T11:41:48.281 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.280 7f92ab832700 1 -- 172.21.2.112:0/49250850 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92a4084420 con 0x7f92a41a79a0 2021-02-10T11:41:48.281 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.280 7f9295ffb700 1 -- 172.21.2.112:0/49250850 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f92a0002070 con 0x7f92a41a79a0 2021-02-10T11:41:48.281 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.280 7f9295ffb700 1 --2- 172.21.2.112:0/49250850 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f92840186c0 0x7f928401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:48.281 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.280 7f9295ffb700 1 -- 172.21.2.112:0/49250850 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f92a0032380 con 0x7f92a41a79a0 2021-02-10T11:41:48.281 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.281 7f92a8e4d700 1 --2- 172.21.2.112:0/49250850 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f92840186c0 0x7f928401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:48.281 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.281 7f92a8e4d700 1 --2- 172.21.2.112:0/49250850 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f92840186c0 0x7f928401abc0 crc :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:48.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.311 7f92ab832700 1 -- 172.21.2.112:0/49250850 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92a4082a70 con 0x7f92a41a79a0 2021-02-10T11:41:48.311 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.314 7f9295ffb700 1 -- 172.21.2.112:0/49250850 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f92a0000940 con 0x7f92a41a79a0 2021-02-10T11:41:48.461 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.463 7f92ab832700 1 -- 172.21.2.112:0/49250850 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f92a4082a70 con 0x7f92a41a79a0 2021-02-10T11:41:48.461 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.464 7f9295ffb700 1 -- 172.21.2.112:0/49250850 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v30)=0 v30) v1 ==== 122+0+0 (secure 0 0 0) 0x7f92a001f470 con 0x7f92a41a79a0 2021-02-10T11:41:48.463 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.465 7f92ab832700 1 -- 172.21.2.112:0/49250850 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f92840186c0 msgr2=0x7f928401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:48.463 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.465 7f92ab832700 1 --2- 172.21.2.112:0/49250850 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f92840186c0 0x7f928401abc0 crc :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:48.463 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.465 7f92ab832700 1 -- 172.21.2.112:0/49250850 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f92a41a79a0 msgr2=0x7f92a41a7da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:48.463 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.465 7f92ab832700 1 --2- 172.21.2.112:0/49250850 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f92a41a79a0 0x7f92a41a7da0 secure :-1 s=READY pgs=2253 cs=0 l=1 rev1=1 rx=0x7f92a000b510 tx=0x7f92a000b8b0).stop 2021-02-10T11:41:48.464 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.466 7f92ab832700 1 -- 172.21.2.112:0/49250850 shutdown_connections 2021-02-10T11:41:48.464 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.466 7f92ab832700 1 --2- 172.21.2.112:0/49250850 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f92840186c0 0x7f928401abc0 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:48.464 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.466 7f92ab832700 1 --2- 172.21.2.112:0/49250850 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f92a4082ca0 0x7f92a40830a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:48.464 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.466 7f92ab832700 1 --2- 172.21.2.112:0/49250850 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f92a40bdb40 0x7f92a40bdf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:48.464 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.466 7f92ab832700 1 --2- 172.21.2.112:0/49250850 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f92a41a79a0 0x7f92a41a7da0 unknown :-1 s=CLOSED pgs=2253 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:48.464 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.466 7f92ab832700 1 -- 172.21.2.112:0/49250850 >> 172.21.2.112:0/49250850 conn(0x7f92a40bef00 msgr2=0x7f92a409f200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:48.464 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.466 7f92ab832700 1 -- 172.21.2.112:0/49250850 shutdown_connections 2021-02-10T11:41:48.464 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.466 7f92ab832700 1 -- 172.21.2.112:0/49250850 wait complete. 2021-02-10T11:41:48.472 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolumegroup create cephfs group_0000000000550905 2021-02-10T11:41:48.904 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.889 7f06cb853700 1 Processor -- start 2021-02-10T11:41:48.904 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.890 7f06cb853700 1 -- start start 2021-02-10T11:41:48.904 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.890 7f06cb853700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f06c40d1c70 0x7f06c40cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:48.904 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.890 7f06cb853700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f06c40cf580 0x7f06c40cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:48.904 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.890 7f06cb853700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06c40fe1c0 con 0x7f06c40d1870 2021-02-10T11:41:48.905 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.890 7f06cb853700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06c41b0cb0 con 0x7f06c40cf580 2021-02-10T11:41:48.905 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.890 7f06cb853700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06c4109a60 con 0x7f06c40d1c70 2021-02-10T11:41:48.905 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.890 7f06c8e6e700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f06c40d1c70 0x7f06c40cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:48.905 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.890 7f06c8e6e700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f06c40d1c70 0x7f06c40cef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3301/0 says I am v2:172.21.2.112:55442/0 (socket says 172.21.2.112:55442) 2021-02-10T11:41:48.905 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.890 7f06c8e6e700 1 -- 172.21.2.112:0/1387344559 learned_addr learned my addr 172.21.2.112:0/1387344559 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:41:48.905 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.890 7f06c9e70700 1 --2- 172.21.2.112:0/1387344559 >> v2:172.21.2.120:3300/0 conn(0x7f06c40cf580 0x7f06c40cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:48.905 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.891 7f06c8e6e700 1 -- 172.21.2.112:0/1387344559 >> v2:172.21.2.120:3300/0 conn(0x7f06c40cf580 msgr2=0x7f06c40cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:48.905 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.891 7f06c8e6e700 1 --2- 172.21.2.112:0/1387344559 >> v2:172.21.2.120:3300/0 conn(0x7f06c40cf580 0x7f06c40cf980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:48.905 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.891 7f06c8e6e700 1 -- 172.21.2.112:0/1387344559 >> v1:172.21.2.120:6789/0 conn(0x7f06c40d1870 legacy=0x7f06c41b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:48.905 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.891 7f06c8e6e700 1 -- 172.21.2.112:0/1387344559 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06c4088070 con 0x7f06c40d1c70 2021-02-10T11:41:48.905 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.891 7f06c8e6e700 1 --2- 172.21.2.112:0/1387344559 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f06c40d1c70 0x7f06c40cef60 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7f06b4006700 tx=0x7f06b4009b80).ready entity=mon.2 client_cookie=5a598e8271d15c7f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:48.905 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.891 7f06bbfff700 1 -- 172.21.2.112:0/1387344559 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f06b400c0c0 con 0x7f06c40d1c70 2021-02-10T11:41:48.906 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.891 7f06bbfff700 1 -- 172.21.2.112:0/1387344559 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06b400c340 con 0x7f06c40d1c70 2021-02-10T11:41:48.906 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.891 7f06bbfff700 1 -- 172.21.2.112:0/1387344559 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f06b400c860 con 0x7f06c40d1c70 2021-02-10T11:41:48.906 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.891 7f06cb853700 1 -- 172.21.2.112:0/1387344559 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f06c40d1c70 msgr2=0x7f06c40cef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:48.906 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.891 7f06cb853700 1 --2- 172.21.2.112:0/1387344559 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f06c40d1c70 0x7f06c40cef60 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7f06b4006700 tx=0x7f06b4009b80).stop 2021-02-10T11:41:48.906 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.892 7f06cb853700 1 -- 172.21.2.112:0/1387344559 shutdown_connections 2021-02-10T11:41:48.906 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.892 7f06cb853700 1 --2- 172.21.2.112:0/1387344559 >> v2:172.21.2.120:3300/0 conn(0x7f06c40cf580 0x7f06c40cf980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:48.906 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.892 7f06cb853700 1 --2- 172.21.2.112:0/1387344559 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f06c40d1c70 0x7f06c40cef60 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:48.906 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.892 7f06cb853700 1 -- 172.21.2.112:0/1387344559 >> 172.21.2.112:0/1387344559 conn(0x7f06c40ce420 msgr2=0x7f06c40d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:48.906 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.892 7f06cb853700 1 -- 172.21.2.112:0/1387344559 shutdown_connections 2021-02-10T11:41:48.906 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.892 7f06cb853700 1 -- 172.21.2.112:0/1387344559 wait complete. 2021-02-10T11:41:48.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.907 7f06cb853700 1 Processor -- start 2021-02-10T11:41:48.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.907 7f06cb853700 1 -- start start 2021-02-10T11:41:48.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.907 7f06cb853700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f06c40a7950 0x7f06c40b8df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:48.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.907 7f06cb853700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f06c41a7490 0x7f06c41a7890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:48.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.907 7f06cb853700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f06c41a7eb0 0x7f06c40ac630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:48.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.907 7f06cb853700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06c40a7fb0 con 0x7f06c41a7490 2021-02-10T11:41:48.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.907 7f06cb853700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06c40acc50 con 0x7f06c41a7eb0 2021-02-10T11:41:48.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.907 7f06cb853700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06c40ace30 con 0x7f06c40a7950 2021-02-10T11:41:48.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.907 7f06c8e6e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f06c41a7490 0x7f06c41a7890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:48.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.907 7f06c8e6e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f06c41a7490 0x7f06c41a7890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:55764/0 (socket says 172.21.2.112:55764) 2021-02-10T11:41:48.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.907 7f06c8e6e700 1 -- 172.21.2.112:0/2645496540 learned_addr learned my addr 172.21.2.112:0/2645496540 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:41:48.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.908 7f06c8e6e700 1 -- 172.21.2.112:0/2645496540 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f06c40a7950 msgr2=0x7f06c40b8df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:48.909 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.908 7f06c8e6e700 1 --2- 172.21.2.112:0/2645496540 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f06c40a7950 0x7f06c40b8df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:48.909 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.908 7f06c8e6e700 1 -- 172.21.2.112:0/2645496540 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f06c41a7eb0 msgr2=0x7f06c40ac630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:48.909 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.908 7f06c8e6e700 1 --2- 172.21.2.112:0/2645496540 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f06c41a7eb0 0x7f06c40ac630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:48.909 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.908 7f06c8e6e700 1 -- 172.21.2.112:0/2645496540 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06c40ad050 con 0x7f06c41a7490 2021-02-10T11:41:48.909 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.908 7f06c8e6e700 1 --2- 172.21.2.112:0/2645496540 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f06c41a7490 0x7f06c41a7890 secure :-1 s=READY pgs=2254 cs=0 l=1 rev1=1 rx=0x7f06b400b650 tx=0x7f06b400b9f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:48.909 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.908 7f06b9ffb700 1 -- 172.21.2.112:0/2645496540 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f06b400bde0 con 0x7f06c41a7490 2021-02-10T11:41:48.909 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.908 7f06b9ffb700 1 -- 172.21.2.112:0/2645496540 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06b4002030 con 0x7f06c41a7490 2021-02-10T11:41:48.909 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.908 7f06b9ffb700 1 -- 172.21.2.112:0/2645496540 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f06b4002550 con 0x7f06c41a7490 2021-02-10T11:41:48.909 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.908 7f06cb853700 1 -- 172.21.2.112:0/2645496540 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06c40ad050 con 0x7f06c41a7490 2021-02-10T11:41:48.909 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.908 7f06cb853700 1 -- 172.21.2.112:0/2645496540 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06c40a8530 con 0x7f06c41a7490 2021-02-10T11:41:48.909 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.909 7f06b9ffb700 1 -- 172.21.2.112:0/2645496540 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f06b4000c90 con 0x7f06c41a7490 2021-02-10T11:41:48.909 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.909 7f06b9ffb700 1 --2- 172.21.2.112:0/2645496540 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f06a40186c0 0x7f06a401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:48.909 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.909 7f06b9ffb700 1 -- 172.21.2.112:0/2645496540 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f06b40325b0 con 0x7f06c41a7490 2021-02-10T11:41:48.910 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.910 7f06c966f700 1 --2- 172.21.2.112:0/2645496540 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f06a40186c0 0x7f06a401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:48.910 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.910 7f06c966f700 1 --2- 172.21.2.112:0/2645496540 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f06a40186c0 0x7f06a401abc0 crc :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:48.938 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.940 7f06cb853700 1 -- 172.21.2.112:0/2645496540 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06c4082a70 con 0x7f06c41a7490 2021-02-10T11:41:48.940 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:48.943 7f06b9ffb700 1 -- 172.21.2.112:0/2645496540 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f06b4000940 con 0x7f06c41a7490 2021-02-10T11:41:49.069 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:49.071 7f06cb853700 1 -- 172.21.2.112:0/2645496540 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "target": ["mgr", ""], "group_name": "group_0000000000550905"}) v1 -- 0x7f06c40a7fb0 con 0x7f06a40186c0 2021-02-10T11:41:49.074 INFO:tasks.ceph.mgr.x.gibba020.stderr:2021-02-10 11:41:49.076 7f6382eee700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:41:49.074 INFO:tasks.ceph.mgr.x.gibba020.stderr:2021-02-10 11:41:49.076 7f6382eee700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:41:49.074 INFO:tasks.ceph.mgr.x.gibba020.stderr:2021-02-10 11:41:49.076 7f6382eee700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:41:49.074 INFO:tasks.ceph.mgr.x.gibba020.stderr:2021-02-10 11:41:49.076 7f6382eee700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:41:49.075 INFO:tasks.ceph.mgr.x.gibba020.stderr:2021-02-10 11:41:49.076 7f6382eee700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:41:49.088 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:49.091 7f06b9ffb700 1 -- 172.21.2.112:0/2645496540 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f06c0006a20 con 0x7f06a40186c0 2021-02-10T11:41:49.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:49.105 7f06cb853700 1 -- 172.21.2.112:0/2645496540 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f06a40186c0 msgr2=0x7f06a401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:49.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:49.105 7f06cb853700 1 --2- 172.21.2.112:0/2645496540 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f06a40186c0 0x7f06a401abc0 crc :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:49.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:49.105 7f06cb853700 1 -- 172.21.2.112:0/2645496540 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f06c41a7490 msgr2=0x7f06c41a7890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:49.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:49.105 7f06cb853700 1 --2- 172.21.2.112:0/2645496540 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f06c41a7490 0x7f06c41a7890 secure :-1 s=READY pgs=2254 cs=0 l=1 rev1=1 rx=0x7f06b400b650 tx=0x7f06b400b9f0).stop 2021-02-10T11:41:49.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:49.105 7f06cb853700 1 -- 172.21.2.112:0/2645496540 shutdown_connections 2021-02-10T11:41:49.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:49.105 7f06cb853700 1 --2- 172.21.2.112:0/2645496540 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f06a40186c0 0x7f06a401abc0 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:49.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:49.105 7f06cb853700 1 --2- 172.21.2.112:0/2645496540 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f06c41a7eb0 0x7f06c40ac630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:49.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:49.105 7f06cb853700 1 --2- 172.21.2.112:0/2645496540 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f06c41a7490 0x7f06c41a7890 unknown :-1 s=CLOSED pgs=2254 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:49.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:49.105 7f06cb853700 1 --2- 172.21.2.112:0/2645496540 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f06c40a7950 0x7f06c40b8df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:49.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:49.105 7f06cb853700 1 -- 172.21.2.112:0/2645496540 >> 172.21.2.112:0/2645496540 conn(0x7f06c40a02d0 msgr2=0x7f06c409f200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:49.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:49.106 7f06cb853700 1 -- 172.21.2.112:0/2645496540 shutdown_connections 2021-02-10T11:41:49.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:49.106 7f06cb853700 1 -- 172.21.2.112:0/2645496540 wait complete. 2021-02-10T11:41:49.112 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000814932 --group_name group_0000000000550905 2021-02-10T11:41:49.544 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:49.530 7f923a1b7700 1 Processor -- start 2021-02-10T11:41:49.544 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:49.530 7f923a1b7700 1 -- start start 2021-02-10T11:41:49.544 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:49.530 7f923a1b7700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f92340d1880 0x7f92341b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:49.544 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:49.530 7f923a1b7700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f92341b0af0 0x7f9234108950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:49.544 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:49.530 7f923a1b7700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92340fe1d0 con 0x7f92340d1c80 2021-02-10T11:41:49.544 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:49.530 7f923a1b7700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9234109a70 con 0x7f92340d1880 2021-02-10T11:41:49.545 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:49.530 7f923a1b7700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9234109760 con 0x7f92341b0af0 2021-02-10T11:41:49.545 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:49.530 7f92337fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f92340d1880 0x7f92341b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:49.545 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:49.530 7f92337fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f92340d1880 0x7f92341b04d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:55772/0 (socket says 172.21.2.112:55772) 2021-02-10T11:41:49.545 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:49.530 7f92337fe700 1 -- 172.21.2.112:0/757885638 learned_addr learned my addr 172.21.2.112:0/757885638 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:41:49.545 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:49.531 7f92337fe700 1 -- 172.21.2.112:0/757885638 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f92341b0af0 msgr2=0x7f9234108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:49.545 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:49.531 7f92337fe700 1 --2- 172.21.2.112:0/757885638 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f92341b0af0 0x7f9234108950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:49.545 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:49.531 7f92337fe700 1 -- 172.21.2.112:0/757885638 >> v1:172.21.2.120:6789/0 conn(0x7f92340d1c80 legacy=0x7f9234108f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:49.545 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:49.531 7f92337fe700 1 -- 172.21.2.112:0/757885638 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9234088070 con 0x7f92340d1880 2021-02-10T11:41:49.545 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:49.531 7f92337fe700 1 --2- 172.21.2.112:0/757885638 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f92340d1880 0x7f92341b04d0 secure :-1 s=READY pgs=2255 cs=0 l=1 rev1=1 rx=0x7f9224006700 tx=0x7f9224009b80).ready entity=mon.0 client_cookie=91fe6b7fe48ff44c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:49.545 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:49.531 7f92327fc700 1 -- 172.21.2.112:0/757885638 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f92240009b0 con 0x7f92340d1880 2021-02-10T11:41:49.545 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:49.531 7f92327fc700 1 -- 172.21.2.112:0/757885638 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f922400c0c0 con 0x7f92340d1880 2021-02-10T11:41:49.546 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:49.531 7f92327fc700 1 -- 172.21.2.112:0/757885638 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f922400c600 con 0x7f92340d1880 2021-02-10T11:41:49.546 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:49.531 7f923a1b7700 1 -- 172.21.2.112:0/757885638 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f92340d1880 msgr2=0x7f92341b04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:49.546 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:49.531 7f923a1b7700 1 --2- 172.21.2.112:0/757885638 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f92340d1880 0x7f92341b04d0 secure :-1 s=READY pgs=2255 cs=0 l=1 rev1=1 rx=0x7f9224006700 tx=0x7f9224009b80).stop 2021-02-10T11:41:49.546 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:49.532 7f923a1b7700 1 -- 172.21.2.112:0/757885638 shutdown_connections 2021-02-10T11:41:49.546 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:49.532 7f923a1b7700 1 --2- 172.21.2.112:0/757885638 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f92341b0af0 0x7f9234108950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:49.546 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:49.532 7f923a1b7700 1 --2- 172.21.2.112:0/757885638 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f92340d1880 0x7f92341b04d0 unknown :-1 s=CLOSED pgs=2255 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:49.546 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:49.532 7f923a1b7700 1 -- 172.21.2.112:0/757885638 >> 172.21.2.112:0/757885638 conn(0x7f92340ce430 msgr2=0x7f92340d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:49.546 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:49.532 7f923a1b7700 1 -- 172.21.2.112:0/757885638 shutdown_connections 2021-02-10T11:41:49.546 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:49.532 7f923a1b7700 1 -- 172.21.2.112:0/757885638 wait complete. 2021-02-10T11:41:49.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:49.547 7f923a1b7700 1 Processor -- start 2021-02-10T11:41:49.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:49.547 7f923a1b7700 1 -- start start 2021-02-10T11:41:49.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:49.547 7f923a1b7700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f923419e5b0 0x7f92341afc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:49.549 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:49.547 7f923a1b7700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f923419e9b0 0x7f92341b0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:49.549 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:49.547 7f923a1b7700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f92341b0250 0x7f92341b13e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:49.549 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:49.547 7f923a1b7700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9234195bd0 con 0x7f923419e9b0 2021-02-10T11:41:49.549 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:49.547 7f923a1b7700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92341b0840 con 0x7f92341b0250 2021-02-10T11:41:49.549 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:49.547 7f923a1b7700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f923419eed0 con 0x7f923419e5b0 2021-02-10T11:41:49.550 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:49.547 7f9232ffd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f923419e9b0 0x7f92341b0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:49.550 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:49.547 7f9232ffd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f923419e9b0 0x7f92341b0dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:55774/0 (socket says 172.21.2.112:55774) 2021-02-10T11:41:49.550 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:49.547 7f9232ffd700 1 -- 172.21.2.112:0/2955624209 learned_addr learned my addr 172.21.2.112:0/2955624209 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:41:49.550 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:49.548 7f9232ffd700 1 -- 172.21.2.112:0/2955624209 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f923419e5b0 msgr2=0x7f92341afc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:49.550 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:49.548 7f9232ffd700 1 --2- 172.21.2.112:0/2955624209 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f923419e5b0 0x7f92341afc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:49.550 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:49.548 7f9232ffd700 1 -- 172.21.2.112:0/2955624209 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f92341b0250 msgr2=0x7f92341b13e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:49.550 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:49.548 7f9232ffd700 1 --2- 172.21.2.112:0/2955624209 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f92341b0250 0x7f92341b13e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:49.550 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:49.548 7f9232ffd700 1 -- 172.21.2.112:0/2955624209 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f923419f0b0 con 0x7f923419e9b0 2021-02-10T11:41:49.550 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:49.548 7f9232ffd700 1 --2- 172.21.2.112:0/2955624209 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f923419e9b0 0x7f92341b0dc0 secure :-1 s=READY pgs=2256 cs=0 l=1 rev1=1 rx=0x7f921c006f00 tx=0x7f921c00b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:49.550 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:49.548 7f922bfff700 1 -- 172.21.2.112:0/2955624209 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f921c006350 con 0x7f923419e9b0 2021-02-10T11:41:49.550 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:49.548 7f922bfff700 1 -- 172.21.2.112:0/2955624209 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f921c00be00 con 0x7f923419e9b0 2021-02-10T11:41:49.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:49.548 7f923a1b7700 1 -- 172.21.2.112:0/2955624209 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f923419f0b0 con 0x7f923419e9b0 2021-02-10T11:41:49.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:49.548 7f922bfff700 1 -- 172.21.2.112:0/2955624209 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f921c002020 con 0x7f923419e9b0 2021-02-10T11:41:49.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:49.548 7f923a1b7700 1 -- 172.21.2.112:0/2955624209 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92341b1cf0 con 0x7f923419e9b0 2021-02-10T11:41:49.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:49.549 7f922bfff700 1 -- 172.21.2.112:0/2955624209 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f921c005310 con 0x7f923419e9b0 2021-02-10T11:41:49.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:49.549 7f922bfff700 1 --2- 172.21.2.112:0/2955624209 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f92180186c0 0x7f921801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:49.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:49.549 7f922bfff700 1 -- 172.21.2.112:0/2955624209 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f921c032540 con 0x7f923419e9b0 2021-02-10T11:41:49.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:49.550 7f92337fe700 1 --2- 172.21.2.112:0/2955624209 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f92180186c0 0x7f921801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:49.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:49.550 7f92337fe700 1 --2- 172.21.2.112:0/2955624209 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f92180186c0 0x7f921801abc0 crc :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:49.578 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:49.580 7f923a1b7700 1 -- 172.21.2.112:0/2955624209 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f923419f0b0 con 0x7f923419e9b0 2021-02-10T11:41:49.580 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:49.583 7f922bfff700 1 -- 172.21.2.112:0/2955624209 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f921c006b90 con 0x7f923419e9b0 2021-02-10T11:41:49.707 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:49.709 7f923a1b7700 1 -- 172.21.2.112:0/2955624209 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000814932", "target": ["mgr", ""], "group_name": "group_0000000000550905"}) v1 -- 0x7f92341b1cf0 con 0x7f92180186c0 2021-02-10T11:41:49.734 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:49.736 7f922bfff700 1 -- 172.21.2.112:0/2955624209 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f9224006390 con 0x7f92180186c0 2021-02-10T11:41:49.741 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:49.743 7f923a1b7700 1 -- 172.21.2.112:0/2955624209 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f92180186c0 msgr2=0x7f921801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:49.741 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:49.743 7f923a1b7700 1 --2- 172.21.2.112:0/2955624209 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f92180186c0 0x7f921801abc0 crc :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:49.741 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:49.743 7f923a1b7700 1 -- 172.21.2.112:0/2955624209 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f923419e9b0 msgr2=0x7f92341b0dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:49.741 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:49.743 7f923a1b7700 1 --2- 172.21.2.112:0/2955624209 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f923419e9b0 0x7f92341b0dc0 secure :-1 s=READY pgs=2256 cs=0 l=1 rev1=1 rx=0x7f921c006f00 tx=0x7f921c00b940).stop 2021-02-10T11:41:49.741 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:49.743 7f923a1b7700 1 -- 172.21.2.112:0/2955624209 shutdown_connections 2021-02-10T11:41:49.741 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:49.743 7f923a1b7700 1 --2- 172.21.2.112:0/2955624209 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f92180186c0 0x7f921801abc0 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:49.741 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:49.743 7f923a1b7700 1 --2- 172.21.2.112:0/2955624209 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f92341b0250 0x7f92341b13e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:49.741 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:49.743 7f923a1b7700 1 --2- 172.21.2.112:0/2955624209 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f923419e9b0 0x7f92341b0dc0 unknown :-1 s=CLOSED pgs=2256 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:49.741 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:49.743 7f923a1b7700 1 --2- 172.21.2.112:0/2955624209 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f923419e5b0 0x7f92341afc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:49.741 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:49.743 7f923a1b7700 1 -- 172.21.2.112:0/2955624209 >> 172.21.2.112:0/2955624209 conn(0x7f923419f940 msgr2=0x7f923418c290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:49.742 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:49.743 7f923a1b7700 1 -- 172.21.2.112:0/2955624209 shutdown_connections 2021-02-10T11:41:49.742 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:49.743 7f923a1b7700 1 -- 172.21.2.112:0/2955624209 wait complete. 2021-02-10T11:41:49.749 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.1 && sudo ls volumes 2021-02-10T11:41:50.101 INFO:teuthology.orchestra.run.gibba012.stdout:group_0000000000550905 2021-02-10T11:41:50.101 INFO:teuthology.orchestra.run.gibba012.stdout:_group_0000000000550905:subvolume_0000000000814932.meta 2021-02-10T11:41:50.102 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume authorize cephfs subvolume_0000000000814932 alice --group_name group_0000000000550905 --tenant_id tenant1 2021-02-10T11:41:50.543 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:50.528 7f6ecbe9c700 1 Processor -- start 2021-02-10T11:41:50.543 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:50.528 7f6ecbe9c700 1 -- start start 2021-02-10T11:41:50.543 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:50.529 7f6ecbe9c700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6ec40e53a0 0x7f6ec41b0550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:50.543 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:50.529 7f6ecbe9c700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f6ec410b050 0x7f6ec41091c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:50.543 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:50.529 7f6ecbe9c700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ec4109ac0 con 0x7f6ec41b0b70 2021-02-10T11:41:50.543 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:50.529 7f6ecbe9c700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ec40fe2b0 con 0x7f6ec410b050 2021-02-10T11:41:50.543 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:50.529 7f6ecbe9c700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ec40e5f80 con 0x7f6ec40e53a0 2021-02-10T11:41:50.543 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:50.529 7f6ec94b7700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f6ec41b0b70 0x7f6ec41089d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:55244/0 (socket says 172.21.2.112:55244) 2021-02-10T11:41:50.543 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:50.529 7f6ec94b7700 1 -- 172.21.2.112:0/1748334132 learned_addr learned my addr 172.21.2.112:0/1748334132 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:41:50.544 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:50.529 7f6eca4b9700 1 --2- 172.21.2.112:0/1748334132 >> v2:172.21.2.120:3300/0 conn(0x7f6ec410b050 0x7f6ec41091c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:50.544 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:50.529 7f6ec9cb8700 1 --2- 172.21.2.112:0/1748334132 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6ec40e53a0 0x7f6ec41b0550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:50.544 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:50.529 7f6ec8cb6700 1 -- 172.21.2.112:0/1748334132 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3382490002 0 0) 0x7f6ebc001060 con 0x7f6ec41b0b70 2021-02-10T11:41:50.544 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:50.529 7f6ec8cb6700 1 -- 172.21.2.112:0/1748334132 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6eac002fe0 con 0x7f6ec41b0b70 2021-02-10T11:41:50.544 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:50.530 7f6eca4b9700 1 -- 172.21.2.112:0/1748334132 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6ec40e53a0 msgr2=0x7f6ec41b0550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:50.544 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:50.530 7f6eca4b9700 1 --2- 172.21.2.112:0/1748334132 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6ec40e53a0 0x7f6ec41b0550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:50.544 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:50.530 7f6eca4b9700 1 -- 172.21.2.112:0/1748334132 >> v1:172.21.2.112:6789/0 conn(0x7f6ec41b0b70 legacy=0x7f6ec41089d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:50.544 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:50.530 7f6eca4b9700 1 -- 172.21.2.112:0/1748334132 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ec40e57a0 con 0x7f6ec410b050 2021-02-10T11:41:50.544 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:50.530 7f6eca4b9700 1 --2- 172.21.2.112:0/1748334132 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6ec410b050 0x7f6ec41091c0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 rx=0x7f6eb4006ad0 tx=0x7f6eb4009ad0).ready entity=mon.1 client_cookie=92e884f5f342b358 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:50.544 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:50.530 7f6ec8cb6700 1 -- 172.21.2.112:0/1748334132 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6eb40022e0 con 0x7f6ec410b050 2021-02-10T11:41:50.544 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:50.530 7f6ec8cb6700 1 -- 172.21.2.112:0/1748334132 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6eb4006410 con 0x7f6ec410b050 2021-02-10T11:41:50.544 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:50.530 7f6ecbe9c700 1 -- 172.21.2.112:0/1748334132 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6ec410b050 msgr2=0x7f6ec41091c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:50.545 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:50.530 7f6ecbe9c700 1 --2- 172.21.2.112:0/1748334132 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6ec410b050 0x7f6ec41091c0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 rx=0x7f6eb4006ad0 tx=0x7f6eb4009ad0).stop 2021-02-10T11:41:50.545 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:50.530 7f6ec8cb6700 1 -- 172.21.2.112:0/1748334132 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6eb4000c10 con 0x7f6ec410b050 2021-02-10T11:41:50.545 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:50.530 7f6ecbe9c700 1 -- 172.21.2.112:0/1748334132 shutdown_connections 2021-02-10T11:41:50.545 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:50.530 7f6ecbe9c700 1 --2- 172.21.2.112:0/1748334132 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6ec410b050 0x7f6ec41091c0 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:50.545 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:50.530 7f6ecbe9c700 1 --2- 172.21.2.112:0/1748334132 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6ec40e53a0 0x7f6ec41b0550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:50.545 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:50.530 7f6ecbe9c700 1 -- 172.21.2.112:0/1748334132 >> 172.21.2.112:0/1748334132 conn(0x7f6ec40ce4b0 msgr2=0x7f6ec40e45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:50.545 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:50.531 7f6ecbe9c700 1 -- 172.21.2.112:0/1748334132 shutdown_connections 2021-02-10T11:41:50.545 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:50.531 7f6ecbe9c700 1 -- 172.21.2.112:0/1748334132 wait complete. 2021-02-10T11:41:50.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:50.545 7f6ecbe9c700 1 Processor -- start 2021-02-10T11:41:50.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:50.545 7f6ecbe9c700 1 -- start start 2021-02-10T11:41:50.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:50.546 7f6ecbe9c700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6ec418bbe0 0x7f6ec418f380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:50.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:50.546 7f6ecbe9c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6ec418b0b0 0x7f6ec418b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:50.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:50.546 7f6ecbe9c700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6ec4190340 0x7f6ec4190740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:50.547 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:50.546 7f6ecbe9c700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ec418fdc0 con 0x7f6ec418b0b0 2021-02-10T11:41:50.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:50.546 7f6ecbe9c700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ec418f9a0 con 0x7f6ec418bbe0 2021-02-10T11:41:50.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:50.546 7f6ecbe9c700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ec418fb80 con 0x7f6ec4190340 2021-02-10T11:41:50.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:50.546 7f6ec94b7700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6ec418b0b0 0x7f6ec418b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:50.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:50.546 7f6ec94b7700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6ec418b0b0 0x7f6ec418b4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:55790/0 (socket says 172.21.2.112:55790) 2021-02-10T11:41:50.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:50.546 7f6ec94b7700 1 -- 172.21.2.112:0/1247262411 learned_addr learned my addr 172.21.2.112:0/1247262411 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:41:50.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:50.546 7f6ec9cb8700 1 --2- 172.21.2.112:0/1247262411 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6ec418bbe0 0x7f6ec418f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:50.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:50.546 7f6eca4b9700 1 --2- 172.21.2.112:0/1247262411 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6ec4190340 0x7f6ec4190740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:50.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:50.547 7f6ec94b7700 1 -- 172.21.2.112:0/1247262411 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6ec4190340 msgr2=0x7f6ec4190740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:50.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:50.547 7f6ec94b7700 1 --2- 172.21.2.112:0/1247262411 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6ec4190340 0x7f6ec4190740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:50.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:50.547 7f6ec94b7700 1 -- 172.21.2.112:0/1247262411 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6ec418bbe0 msgr2=0x7f6ec418f380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:50.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:50.547 7f6ec94b7700 1 --2- 172.21.2.112:0/1247262411 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6ec418bbe0 0x7f6ec418f380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:50.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:50.547 7f6ec94b7700 1 -- 172.21.2.112:0/1247262411 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ec4190d60 con 0x7f6ec418b0b0 2021-02-10T11:41:50.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:50.547 7f6ec94b7700 1 --2- 172.21.2.112:0/1247262411 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6ec418b0b0 0x7f6ec418b4b0 secure :-1 s=READY pgs=2258 cs=0 l=1 rev1=1 rx=0x7f6ebc009960 tx=0x7f6ebc00bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:50.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:50.547 7f6ec27fc700 1 -- 172.21.2.112:0/1247262411 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6ebc0062b0 con 0x7f6ec418b0b0 2021-02-10T11:41:50.549 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:50.547 7f6ec27fc700 1 -- 172.21.2.112:0/1247262411 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ebc004720 con 0x7f6ec418b0b0 2021-02-10T11:41:50.549 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:50.547 7f6ec27fc700 1 -- 172.21.2.112:0/1247262411 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6ebc004b70 con 0x7f6ec418b0b0 2021-02-10T11:41:50.549 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:50.547 7f6ecbe9c700 1 -- 172.21.2.112:0/1247262411 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ec4190d60 con 0x7f6ec418b0b0 2021-02-10T11:41:50.549 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:50.547 7f6ecbe9c700 1 -- 172.21.2.112:0/1247262411 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ec4190d60 con 0x7f6ec418b0b0 2021-02-10T11:41:50.549 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:50.548 7f6ec27fc700 1 -- 172.21.2.112:0/1247262411 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6ebc006570 con 0x7f6ec418b0b0 2021-02-10T11:41:50.549 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:50.548 7f6ec27fc700 1 --2- 172.21.2.112:0/1247262411 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6ea40186c0 0x7f6ea401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:50.549 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:50.548 7f6ec27fc700 1 -- 172.21.2.112:0/1247262411 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f6ebc032910 con 0x7f6ec418b0b0 2021-02-10T11:41:50.549 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:50.548 7f6ec9cb8700 1 --2- 172.21.2.112:0/1247262411 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6ea40186c0 0x7f6ea401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:50.549 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:50.549 7f6ec9cb8700 1 --2- 172.21.2.112:0/1247262411 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6ea40186c0 0x7f6ea401abc0 crc :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:50.577 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:50.579 7f6ecbe9c700 1 -- 172.21.2.112:0/1247262411 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ec4191660 con 0x7f6ec418b0b0 2021-02-10T11:41:50.579 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:50.581 7f6ec27fc700 1 -- 172.21.2.112:0/1247262411 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6ebc00be30 con 0x7f6ec418b0b0 2021-02-10T11:41:50.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:50.708 7f6ecbe9c700 1 -- 172.21.2.112:0/1247262411 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume authorize", "vol_name": "cephfs", "sub_name": "subvolume_0000000000814932", "target": ["mgr", ""], "tenant_id": "tenant1", "auth_id": "alice", "group_name": "group_0000000000550905"}) v1 -- 0x7f6ec4191660 con 0x7f6ea40186c0 2021-02-10T11:41:50.712 INFO:tasks.ceph.mgr.x.gibba020.stderr:Creating meta for ID alice with tenant tenant1 2021-02-10T11:41:50.741 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:50.743 7f6ec27fc700 1 -- 172.21.2.112:0/1247262411 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+41 (crc 0 0 0) 0x7f6eb8006a70 con 0x7f6ea40186c0 2021-02-10T11:41:50.770 INFO:teuthology.orchestra.run.gibba012.stdout:AQB+xiNg9vRRKxAA5aDIgQeoOtGfhiRx20y/jg== 2021-02-10T11:41:50.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:50.774 7f6ecbe9c700 1 -- 172.21.2.112:0/1247262411 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6ea40186c0 msgr2=0x7f6ea401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:50.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:50.774 7f6ecbe9c700 1 --2- 172.21.2.112:0/1247262411 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6ea40186c0 0x7f6ea401abc0 crc :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:50.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:50.774 7f6ecbe9c700 1 -- 172.21.2.112:0/1247262411 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6ec418b0b0 msgr2=0x7f6ec418b4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:50.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:50.774 7f6ecbe9c700 1 --2- 172.21.2.112:0/1247262411 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6ec418b0b0 0x7f6ec418b4b0 secure :-1 s=READY pgs=2258 cs=0 l=1 rev1=1 rx=0x7f6ebc009960 tx=0x7f6ebc00bcf0).stop 2021-02-10T11:41:50.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:50.774 7f6ecbe9c700 1 -- 172.21.2.112:0/1247262411 shutdown_connections 2021-02-10T11:41:50.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:50.774 7f6ecbe9c700 1 --2- 172.21.2.112:0/1247262411 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6ea40186c0 0x7f6ea401abc0 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:50.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:50.774 7f6ecbe9c700 1 --2- 172.21.2.112:0/1247262411 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6ec4190340 0x7f6ec4190740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:50.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:50.774 7f6ecbe9c700 1 --2- 172.21.2.112:0/1247262411 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6ec418b0b0 0x7f6ec418b4b0 unknown :-1 s=CLOSED pgs=2258 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:50.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:50.774 7f6ecbe9c700 1 --2- 172.21.2.112:0/1247262411 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6ec418bbe0 0x7f6ec418f380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:50.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:50.774 7f6ecbe9c700 1 -- 172.21.2.112:0/1247262411 >> 172.21.2.112:0/1247262411 conn(0x7f6ec418cf20 msgr2=0x7f6ec4106e70 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:50.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:50.774 7f6ecbe9c700 1 -- 172.21.2.112:0/1247262411 shutdown_connections 2021-02-10T11:41:50.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:50.774 7f6ecbe9c700 1 -- 172.21.2.112:0/1247262411 wait complete. 2021-02-10T11:41:50.780 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.1 && sudo ls volumes 2021-02-10T11:41:51.128 INFO:teuthology.orchestra.run.gibba012.stdout:$alice.meta 2021-02-10T11:41:51.128 INFO:teuthology.orchestra.run.gibba012.stdout:group_0000000000550905 2021-02-10T11:41:51.128 INFO:teuthology.orchestra.run.gibba012.stdout:_group_0000000000550905:subvolume_0000000000814932.meta 2021-02-10T11:41:51.129 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.1 && sudo sudo cat '/home/ubuntu/cephtest/mnt.1/volumes/$alice.meta' 2021-02-10T11:41:51.492 INFO:teuthology.orchestra.run.gibba012.stdout:{"compat_version": 1, "tenant_id": "tenant1", "version": 5, "dirty": false, "subvolumes": {"group_0000000000550905/subvolume_0000000000814932": {"access_level": "rw", "dirty": false}}} 2021-02-10T11:41:51.492 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.1 && sudo sudo cat /home/ubuntu/cephtest/mnt.1/volumes/_group_0000000000550905:subvolume_0000000000814932.meta 2021-02-10T11:41:51.853 INFO:teuthology.orchestra.run.gibba012.stdout:{"auths": {"alice": {"access_level": "rw", "dirty": false}}, "version": 5, "compat_version": 1} 2021-02-10T11:41:51.853 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume authorize cephfs subvolume_0000000000814932 alice --group_name group_0000000000550905 --tenant_id tenant2 2021-02-10T11:41:52.295 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.279 7fdd43173700 1 Processor -- start 2021-02-10T11:41:52.295 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.279 7fdd43173700 1 -- start start 2021-02-10T11:41:52.296 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.280 7fdd43173700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fdd3c0d18d0 0x7fdd3c1b3280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:52.296 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.280 7fdd43173700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdd3c0cffd0 0x7fdd3c0ef6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:52.296 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.280 7fdd43173700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd3c0f0b80 con 0x7fdd3c0d1cd0 2021-02-10T11:41:52.296 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.280 7fdd43173700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd3c1b4090 con 0x7fdd3c0d18d0 2021-02-10T11:41:52.296 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.280 7fdd43173700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd3c0f2110 con 0x7fdd3c0cffd0 2021-02-10T11:41:52.296 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.281 7fdd40f8f700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fdd3c0d18d0 0x7fdd3c1b3280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:52.296 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.281 7fdd41790700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdd3c0cffd0 0x7fdd3c0ef6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:52.296 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.281 7fdd40f8f700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fdd3c0d18d0 0x7fdd3c1b3280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3300/0 says I am v2:172.21.2.112:58082/0 (socket says 172.21.2.112:58082) 2021-02-10T11:41:52.296 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.281 7fdd3bfff700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7fdd3c0d1cd0 0x7fdd3c1b38a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:48820/0 (socket says 172.21.2.112:48820) 2021-02-10T11:41:52.296 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.281 7fdd40f8f700 1 -- 172.21.2.112:0/2755838255 learned_addr learned my addr 172.21.2.112:0/2755838255 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:41:52.296 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.281 7fdd41790700 1 -- 172.21.2.112:0/2755838255 >> v2:172.21.2.120:3300/0 conn(0x7fdd3c0d18d0 msgr2=0x7fdd3c1b3280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:52.296 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.281 7fdd41790700 1 --2- 172.21.2.112:0/2755838255 >> v2:172.21.2.120:3300/0 conn(0x7fdd3c0d18d0 0x7fdd3c1b3280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:52.297 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.281 7fdd41790700 1 -- 172.21.2.112:0/2755838255 >> v1:172.21.2.120:6789/0 conn(0x7fdd3c0d1cd0 legacy=0x7fdd3c1b38a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:52.297 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.281 7fdd41790700 1 -- 172.21.2.112:0/2755838255 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd3c0f2330 con 0x7fdd3c0cffd0 2021-02-10T11:41:52.297 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.282 7fdd40f8f700 1 --2- 172.21.2.112:0/2755838255 >> v2:172.21.2.120:3300/0 conn(0x7fdd3c0d18d0 0x7fdd3c1b3280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:41:52.297 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.282 7fdd41790700 1 --2- 172.21.2.112:0/2755838255 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdd3c0cffd0 0x7fdd3c0ef6c0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7fdd34006e50 tx=0x7fdd34009e70).ready entity=mon.2 client_cookie=16de7cf276431f4 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:52.297 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.282 7fdd3b7fe700 1 -- 172.21.2.112:0/2755838255 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdd34000f90 con 0x7fdd3c0cffd0 2021-02-10T11:41:52.297 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.282 7fdd3b7fe700 1 -- 172.21.2.112:0/2755838255 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd340066f0 con 0x7fdd3c0cffd0 2021-02-10T11:41:52.297 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.282 7fdd3b7fe700 1 -- 172.21.2.112:0/2755838255 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdd3400b700 con 0x7fdd3c0cffd0 2021-02-10T11:41:52.297 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.282 7fdd43173700 1 -- 172.21.2.112:0/2755838255 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdd3c0cffd0 msgr2=0x7fdd3c0ef6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:52.297 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.282 7fdd43173700 1 --2- 172.21.2.112:0/2755838255 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdd3c0cffd0 0x7fdd3c0ef6c0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7fdd34006e50 tx=0x7fdd34009e70).stop 2021-02-10T11:41:52.297 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.283 7fdd43173700 1 -- 172.21.2.112:0/2755838255 shutdown_connections 2021-02-10T11:41:52.297 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.283 7fdd43173700 1 --2- 172.21.2.112:0/2755838255 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdd3c0cffd0 0x7fdd3c0ef6c0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:52.297 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.283 7fdd43173700 1 --2- 172.21.2.112:0/2755838255 >> v2:172.21.2.120:3300/0 conn(0x7fdd3c0d18d0 0x7fdd3c1b3280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:52.297 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.283 7fdd43173700 1 -- 172.21.2.112:0/2755838255 >> 172.21.2.112:0/2755838255 conn(0x7fdd3c0ce480 msgr2=0x7fdd3c0d0840 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:52.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.283 7fdd43173700 1 -- 172.21.2.112:0/2755838255 shutdown_connections 2021-02-10T11:41:52.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.283 7fdd43173700 1 -- 172.21.2.112:0/2755838255 wait complete. 2021-02-10T11:41:52.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.298 7fdd43173700 1 Processor -- start 2021-02-10T11:41:52.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.298 7fdd43173700 1 -- start start 2021-02-10T11:41:52.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.298 7fdd43173700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdd3c0ae260 0x7fdd3c1a31e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:52.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.298 7fdd43173700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdd3c1a3800 0x7fdd3c08a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:52.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.298 7fdd43173700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fdd3c08ab30 0x7fdd3c190700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:52.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.298 7fdd43173700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd3c1a2c60 con 0x7fdd3c0ae260 2021-02-10T11:41:52.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.298 7fdd43173700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd3c0ae660 con 0x7fdd3c08ab30 2021-02-10T11:41:52.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.298 7fdd43173700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd3c0ae840 con 0x7fdd3c1a3800 2021-02-10T11:41:52.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.299 7fdd40f8f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdd3c0ae260 0x7fdd3c1a31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:52.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.299 7fdd40f8f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdd3c0ae260 0x7fdd3c1a31e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:55806/0 (socket says 172.21.2.112:55806) 2021-02-10T11:41:52.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.299 7fdd40f8f700 1 -- 172.21.2.112:0/638738705 learned_addr learned my addr 172.21.2.112:0/638738705 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:41:52.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.299 7fdd40f8f700 1 -- 172.21.2.112:0/638738705 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdd3c1a3800 msgr2=0x7fdd3c08a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:52.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.299 7fdd40f8f700 1 --2- 172.21.2.112:0/638738705 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdd3c1a3800 0x7fdd3c08a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:52.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.299 7fdd40f8f700 1 -- 172.21.2.112:0/638738705 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fdd3c08ab30 msgr2=0x7fdd3c190700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:52.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.299 7fdd40f8f700 1 --2- 172.21.2.112:0/638738705 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fdd3c08ab30 0x7fdd3c190700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:52.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.299 7fdd40f8f700 1 -- 172.21.2.112:0/638738705 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd3c0aea60 con 0x7fdd3c0ae260 2021-02-10T11:41:52.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.299 7fdd40f8f700 1 --2- 172.21.2.112:0/638738705 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdd3c0ae260 0x7fdd3c1a31e0 secure :-1 s=READY pgs=2259 cs=0 l=1 rev1=1 rx=0x7fdd24006ee0 tx=0x7fdd2400b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:52.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.299 7fdd397fa700 1 -- 172.21.2.112:0/638738705 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdd2400bd50 con 0x7fdd3c0ae260 2021-02-10T11:41:52.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.299 7fdd397fa700 1 -- 172.21.2.112:0/638738705 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd240052e0 con 0x7fdd3c0ae260 2021-02-10T11:41:52.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.299 7fdd397fa700 1 -- 172.21.2.112:0/638738705 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdd24005800 con 0x7fdd3c0ae260 2021-02-10T11:41:52.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.299 7fdd43173700 1 -- 172.21.2.112:0/638738705 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd3c0aea60 con 0x7fdd3c0ae260 2021-02-10T11:41:52.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.299 7fdd43173700 1 -- 172.21.2.112:0/638738705 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd3c191010 con 0x7fdd3c0ae260 2021-02-10T11:41:52.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.300 7fdd397fa700 1 -- 172.21.2.112:0/638738705 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fdd240054c0 con 0x7fdd3c0ae260 2021-02-10T11:41:52.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.300 7fdd397fa700 1 --2- 172.21.2.112:0/638738705 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fdd200186c0 0x7fdd2001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:52.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.300 7fdd397fa700 1 -- 172.21.2.112:0/638738705 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fdd24032530 con 0x7fdd3c0ae260 2021-02-10T11:41:52.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.300 7fdd3bfff700 1 --2- 172.21.2.112:0/638738705 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fdd200186c0 0x7fdd2001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:52.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.301 7fdd3bfff700 1 --2- 172.21.2.112:0/638738705 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fdd200186c0 0x7fdd2001abc0 crc :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:52.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.330 7fdd43173700 1 -- 172.21.2.112:0/638738705 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd3c0ebc30 con 0x7fdd3c0ae260 2021-02-10T11:41:52.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.333 7fdd397fa700 1 -- 172.21.2.112:0/638738705 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fdd240009b0 con 0x7fdd3c0ae260 2021-02-10T11:41:52.456 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.458 7fdd43173700 1 -- 172.21.2.112:0/638738705 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume authorize", "vol_name": "cephfs", "sub_name": "subvolume_0000000000814932", "target": ["mgr", ""], "tenant_id": "tenant2", "auth_id": "alice", "group_name": "group_0000000000550905"}) v1 -- 0x7fdd3c0f0b80 con 0x7fdd200186c0 2021-02-10T11:41:52.462 INFO:tasks.ceph.mgr.x.gibba020.stderr:2021-02-10 11:41:52.464 7f6382eee700 -1 mgr.server reply reply (1) Operation not permitted auth ID: alice is already in use 2021-02-10T11:41:52.463 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.465 7fdd397fa700 1 -- 172.21.2.112:0/638738705 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: -1 auth ID: alice is already in use) v1 ==== 40+0+0 (crc 0 0 0) 0x7fdd2c000e70 con 0x7fdd200186c0 2021-02-10T11:41:52.463 INFO:teuthology.orchestra.run.gibba012.stderr:Error EPERM: auth ID: alice is already in use 2021-02-10T11:41:52.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.466 7fdd43173700 1 -- 172.21.2.112:0/638738705 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fdd200186c0 msgr2=0x7fdd2001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:52.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.466 7fdd43173700 1 --2- 172.21.2.112:0/638738705 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fdd200186c0 0x7fdd2001abc0 crc :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:52.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.467 7fdd43173700 1 -- 172.21.2.112:0/638738705 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdd3c0ae260 msgr2=0x7fdd3c1a31e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:52.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.467 7fdd43173700 1 --2- 172.21.2.112:0/638738705 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdd3c0ae260 0x7fdd3c1a31e0 secure :-1 s=READY pgs=2259 cs=0 l=1 rev1=1 rx=0x7fdd24006ee0 tx=0x7fdd2400b960).stop 2021-02-10T11:41:52.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.467 7fdd43173700 1 -- 172.21.2.112:0/638738705 shutdown_connections 2021-02-10T11:41:52.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.467 7fdd43173700 1 --2- 172.21.2.112:0/638738705 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fdd200186c0 0x7fdd2001abc0 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:52.466 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.467 7fdd43173700 1 --2- 172.21.2.112:0/638738705 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fdd3c08ab30 0x7fdd3c190700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:52.466 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.467 7fdd43173700 1 --2- 172.21.2.112:0/638738705 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdd3c1a3800 0x7fdd3c08a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:52.466 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.467 7fdd43173700 1 --2- 172.21.2.112:0/638738705 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdd3c0ae260 0x7fdd3c1a31e0 unknown :-1 s=CLOSED pgs=2259 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:52.466 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.467 7fdd43173700 1 -- 172.21.2.112:0/638738705 >> 172.21.2.112:0/638738705 conn(0x7fdd3c19f7a0 msgr2=0x7fdd3c1a4080 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:52.466 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.467 7fdd43173700 1 -- 172.21.2.112:0/638738705 shutdown_connections 2021-02-10T11:41:52.466 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.467 7fdd43173700 1 -- 172.21.2.112:0/638738705 wait complete. 2021-02-10T11:41:52.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:41:52.474 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume deauthorize cephfs subvolume_0000000000814932 alice --group_name group_0000000000550905 2021-02-10T11:41:52.906 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.892 7f4f19357700 1 Processor -- start 2021-02-10T11:41:52.907 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.892 7f4f19357700 1 -- start start 2021-02-10T11:41:52.907 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.893 7f4f19357700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f4f140dce80 0x7f4f140d14f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:52.907 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.893 7f4f19357700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4f140eb220 0x7f4f140eb620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:52.907 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.893 7f4f19357700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f1410a550 con 0x7f4f1410aad0 2021-02-10T11:41:52.907 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.893 7f4f19357700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f14088240 con 0x7f4f140dce80 2021-02-10T11:41:52.907 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.893 7f4f19357700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f140d1f30 con 0x7f4f140eb220 2021-02-10T11:41:52.907 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.893 7f4f1387e700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f4f1410aad0 0x7f4f1410aed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:55276/0 (socket says 172.21.2.112:55276) 2021-02-10T11:41:52.907 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.893 7f4f1387e700 1 -- 172.21.2.112:0/1748782 learned_addr learned my addr 172.21.2.112:0/1748782 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:41:52.907 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.893 7f4f1307d700 1 --2- 172.21.2.112:0/1748782 >> v2:172.21.2.112:3300/0 conn(0x7f4f140dce80 0x7f4f140d14f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:52.907 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.893 7f4f1287c700 1 --2- 172.21.2.112:0/1748782 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4f140eb220 0x7f4f140eb620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:52.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.893 7f4f1207b700 1 -- 172.21.2.112:0/1748782 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 130658684 0 0) 0x7f4f0c001500 con 0x7f4f1410aad0 2021-02-10T11:41:52.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.893 7f4f1207b700 1 -- 172.21.2.112:0/1748782 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f00002fe0 con 0x7f4f1410aad0 2021-02-10T11:41:52.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.893 7f4f1307d700 1 -- 172.21.2.112:0/1748782 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4f140eb220 msgr2=0x7f4f140eb620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:52.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.893 7f4f1307d700 1 --2- 172.21.2.112:0/1748782 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4f140eb220 0x7f4f140eb620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:52.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.893 7f4f1307d700 1 -- 172.21.2.112:0/1748782 >> v1:172.21.2.112:6789/0 conn(0x7f4f1410aad0 legacy=0x7f4f1410aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:52.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.893 7f4f1307d700 1 -- 172.21.2.112:0/1748782 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f140dd280 con 0x7f4f140dce80 2021-02-10T11:41:52.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.894 7f4f1307d700 1 --2- 172.21.2.112:0/1748782 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4f140dce80 0x7f4f140d14f0 secure :-1 s=READY pgs=2261 cs=0 l=1 rev1=1 rx=0x7f4f04006650 tx=0x7f4f04009a00).ready entity=mon.0 client_cookie=62e5226d56db6831 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:52.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.894 7f4f1207b700 1 -- 172.21.2.112:0/1748782 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4f04009df0 con 0x7f4f140dce80 2021-02-10T11:41:52.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.894 7f4f1207b700 1 -- 172.21.2.112:0/1748782 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f040009b0 con 0x7f4f140dce80 2021-02-10T11:41:52.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.894 7f4f19357700 1 -- 172.21.2.112:0/1748782 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4f140dce80 msgr2=0x7f4f140d14f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:52.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.894 7f4f19357700 1 --2- 172.21.2.112:0/1748782 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4f140dce80 0x7f4f140d14f0 secure :-1 s=READY pgs=2261 cs=0 l=1 rev1=1 rx=0x7f4f04006650 tx=0x7f4f04009a00).stop 2021-02-10T11:41:52.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.894 7f4f1207b700 1 -- 172.21.2.112:0/1748782 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4f0400c2b0 con 0x7f4f140dce80 2021-02-10T11:41:52.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.894 7f4f19357700 1 -- 172.21.2.112:0/1748782 shutdown_connections 2021-02-10T11:41:52.909 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.894 7f4f19357700 1 --2- 172.21.2.112:0/1748782 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4f140eb220 0x7f4f140eb620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:52.909 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.894 7f4f19357700 1 --2- 172.21.2.112:0/1748782 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4f140dce80 0x7f4f140d14f0 unknown :-1 s=CLOSED pgs=2261 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:52.909 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.894 7f4f19357700 1 -- 172.21.2.112:0/1748782 >> 172.21.2.112:0/1748782 conn(0x7f4f140ea630 msgr2=0x7f4f140d0800 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:52.909 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.894 7f4f19357700 1 -- 172.21.2.112:0/1748782 shutdown_connections 2021-02-10T11:41:52.909 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.894 7f4f19357700 1 -- 172.21.2.112:0/1748782 wait complete. 2021-02-10T11:41:52.910 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.909 7f4f19357700 1 Processor -- start 2021-02-10T11:41:52.910 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.909 7f4f19357700 1 -- start start 2021-02-10T11:41:52.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.909 7f4f19357700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4f140a73d0 0x7f4f140943e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:52.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.909 7f4f19357700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4f140a77d0 0x7f4f14094a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:52.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.909 7f4f19357700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4f140950b0 0x7f4f140954b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:52.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.909 7f4f19357700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f14095b50 con 0x7f4f140a77d0 2021-02-10T11:41:52.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.909 7f4f19357700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f14095d70 con 0x7f4f140a73d0 2021-02-10T11:41:52.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.909 7f4f19357700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f14095f90 con 0x7f4f140950b0 2021-02-10T11:41:52.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.910 7f4f1287c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4f140a77d0 0x7f4f14094a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:52.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.910 7f4f1287c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4f140a77d0 0x7f4f14094a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:55818/0 (socket says 172.21.2.112:55818) 2021-02-10T11:41:52.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.910 7f4f1287c700 1 -- 172.21.2.112:0/1020802552 learned_addr learned my addr 172.21.2.112:0/1020802552 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:41:52.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.910 7f4f1287c700 1 -- 172.21.2.112:0/1020802552 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4f140950b0 msgr2=0x7f4f140954b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:41:52.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.910 7f4f1287c700 1 --2- 172.21.2.112:0/1020802552 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4f140950b0 0x7f4f140954b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:52.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.910 7f4f1287c700 1 -- 172.21.2.112:0/1020802552 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4f140a73d0 msgr2=0x7f4f140943e0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:41:52.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.910 7f4f1287c700 1 --2- 172.21.2.112:0/1020802552 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4f140a73d0 0x7f4f140943e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:52.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.910 7f4f1287c700 1 -- 172.21.2.112:0/1020802552 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f140961b0 con 0x7f4f140a77d0 2021-02-10T11:41:52.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.910 7f4f1287c700 1 --2- 172.21.2.112:0/1020802552 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4f140a77d0 0x7f4f14094a00 secure :-1 s=READY pgs=2262 cs=0 l=1 rev1=1 rx=0x7f4efc006f60 tx=0x7f4efc00b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:52.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.910 7f4f0bfff700 1 -- 172.21.2.112:0/1020802552 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4efc00bd60 con 0x7f4f140a77d0 2021-02-10T11:41:52.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.910 7f4f0bfff700 1 -- 172.21.2.112:0/1020802552 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4efc000c90 con 0x7f4f140a77d0 2021-02-10T11:41:52.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.910 7f4f0bfff700 1 -- 172.21.2.112:0/1020802552 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4efc002220 con 0x7f4f140a77d0 2021-02-10T11:41:52.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.910 7f4f19357700 1 -- 172.21.2.112:0/1020802552 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f140961b0 con 0x7f4f140a77d0 2021-02-10T11:41:52.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.910 7f4f19357700 1 -- 172.21.2.112:0/1020802552 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f140961b0 con 0x7f4f140a77d0 2021-02-10T11:41:52.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.911 7f4f0bfff700 1 -- 172.21.2.112:0/1020802552 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4efc0052e0 con 0x7f4f140a77d0 2021-02-10T11:41:52.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.911 7f4f0bfff700 1 --2- 172.21.2.112:0/1020802552 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4ef80186c0 0x7f4ef801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:52.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.911 7f4f0bfff700 1 -- 172.21.2.112:0/1020802552 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f4efc032780 con 0x7f4f140a77d0 2021-02-10T11:41:52.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.912 7f4f1307d700 1 --2- 172.21.2.112:0/1020802552 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4ef80186c0 0x7f4ef801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:52.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.912 7f4f1307d700 1 --2- 172.21.2.112:0/1020802552 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4ef80186c0 0x7f4ef801abc0 crc :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:52.914 DEBUG:teuthology.orchestra.run.gibba012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:41:52.917 DEBUG:teuthology.orchestra.run.gibba020:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:41:52.940 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.943 7f4f19357700 1 -- 172.21.2.112:0/1020802552 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f140f93d0 con 0x7f4f140a77d0 2021-02-10T11:41:52.943 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:52.945 7f4f0bfff700 1 -- 172.21.2.112:0/1020802552 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4efc0052e0 con 0x7f4f140a77d0 2021-02-10T11:41:53.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:53.073 7f4f19357700 1 -- 172.21.2.112:0/1020802552 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume deauthorize", "vol_name": "cephfs", "sub_name": "subvolume_0000000000814932", "target": ["mgr", ""], "auth_id": "alice", "group_name": "group_0000000000550905"}) v1 -- 0x7f4f140f93d0 con 0x7f4ef80186c0 2021-02-10T11:41:53.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:53.111 7f4f0bfff700 1 -- 172.21.2.112:0/1020802552 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f4f04000c90 con 0x7f4ef80186c0 2021-02-10T11:41:53.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:53.138 7f4f19357700 1 -- 172.21.2.112:0/1020802552 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4ef80186c0 msgr2=0x7f4ef801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:53.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:53.138 7f4f19357700 1 --2- 172.21.2.112:0/1020802552 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4ef80186c0 0x7f4ef801abc0 crc :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:53.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:53.138 7f4f19357700 1 -- 172.21.2.112:0/1020802552 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4f140a77d0 msgr2=0x7f4f14094a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:53.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:53.138 7f4f19357700 1 --2- 172.21.2.112:0/1020802552 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4f140a77d0 0x7f4f14094a00 secure :-1 s=READY pgs=2262 cs=0 l=1 rev1=1 rx=0x7f4efc006f60 tx=0x7f4efc00b970).stop 2021-02-10T11:41:53.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:53.139 7f4f19357700 1 -- 172.21.2.112:0/1020802552 shutdown_connections 2021-02-10T11:41:53.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:53.139 7f4f19357700 1 --2- 172.21.2.112:0/1020802552 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4ef80186c0 0x7f4ef801abc0 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:53.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:53.139 7f4f19357700 1 --2- 172.21.2.112:0/1020802552 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4f140950b0 0x7f4f140954b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:53.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:53.139 7f4f19357700 1 --2- 172.21.2.112:0/1020802552 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4f140a77d0 0x7f4f14094a00 unknown :-1 s=CLOSED pgs=2262 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:53.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:53.139 7f4f19357700 1 --2- 172.21.2.112:0/1020802552 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4f140a73d0 0x7f4f140943e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:53.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:53.139 7f4f19357700 1 -- 172.21.2.112:0/1020802552 >> 172.21.2.112:0/1020802552 conn(0x7f4f140fafe0 msgr2=0x7f4f14091950 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:53.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:53.139 7f4f19357700 1 -- 172.21.2.112:0/1020802552 shutdown_connections 2021-02-10T11:41:53.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:53.139 7f4f19357700 1 -- 172.21.2.112:0/1020802552 wait complete. 2021-02-10T11:41:53.145 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.1 && sudo ls volumes 2021-02-10T11:41:53.533 INFO:teuthology.orchestra.run.gibba012.stdout:group_0000000000550905 2021-02-10T11:41:53.533 INFO:teuthology.orchestra.run.gibba012.stdout:_group_0000000000550905:subvolume_0000000000814932.meta 2021-02-10T11:41:53.534 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000814932 --group_name group_0000000000550905 2021-02-10T11:41:53.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:53.951 7f15d9dd5700 1 Processor -- start 2021-02-10T11:41:53.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:53.951 7f15d9dd5700 1 -- start start 2021-02-10T11:41:53.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:53.951 7f15d9dd5700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f15d410b000 0x7f15d40cf760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:53.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:53.951 7f15d9dd5700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15d40fe1d0 con 0x7f15d40d1880 2021-02-10T11:41:53.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:53.951 7f15d9dd5700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15d41b0cc0 con 0x7f15d40d1c80 2021-02-10T11:41:53.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:53.951 7f15d9dd5700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15d4109a70 con 0x7f15d410b000 2021-02-10T11:41:53.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:53.952 7f15d3fff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f15d410b000 0x7f15d40cf760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:53.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:53.952 7f15d3fff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f15d410b000 0x7f15d40cf760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:55826/0 (socket says 172.21.2.112:55826) 2021-02-10T11:41:53.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:53.952 7f15d3fff700 1 -- 172.21.2.112:0/3477485431 learned_addr learned my addr 172.21.2.112:0/3477485431 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:41:53.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:53.952 7f15d37fe700 1 --1- 172.21.2.112:0/3477485431 >> v1:172.21.2.112:6789/0 conn(0x7f15d40d1880 0x7f15d41b04d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:55288/0 (socket says 172.21.2.112:55288) 2021-02-10T11:41:53.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:53.952 7f15d3fff700 1 -- 172.21.2.112:0/3477485431 >> v1:172.21.2.120:6789/0 conn(0x7f15d40d1c80 legacy=0x7f15d40cef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:53.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:53.952 7f15d3fff700 1 -- 172.21.2.112:0/3477485431 >> v1:172.21.2.112:6789/0 conn(0x7f15d40d1880 legacy=0x7f15d41b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:53.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:53.952 7f15d3fff700 1 -- 172.21.2.112:0/3477485431 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15d410b400 con 0x7f15d410b000 2021-02-10T11:41:53.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:53.952 7f15d3fff700 1 --2- 172.21.2.112:0/3477485431 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f15d410b000 0x7f15d40cf760 secure :-1 s=READY pgs=2264 cs=0 l=1 rev1=1 rx=0x7f15cc006f00 tx=0x7f15cc009f10).ready entity=mon.0 client_cookie=55fe5434809c937b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:53.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:53.952 7f15d27fc700 1 -- 172.21.2.112:0/3477485431 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f15cc00b3b0 con 0x7f15d410b000 2021-02-10T11:41:53.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:53.952 7f15d27fc700 1 -- 172.21.2.112:0/3477485431 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15cc00b630 con 0x7f15d410b000 2021-02-10T11:41:53.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:53.952 7f15d27fc700 1 -- 172.21.2.112:0/3477485431 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f15cc00bb50 con 0x7f15d410b000 2021-02-10T11:41:53.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:53.952 7f15d9dd5700 1 -- 172.21.2.112:0/3477485431 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f15d410b000 msgr2=0x7f15d40cf760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:53.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:53.952 7f15d9dd5700 1 --2- 172.21.2.112:0/3477485431 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f15d410b000 0x7f15d40cf760 secure :-1 s=READY pgs=2264 cs=0 l=1 rev1=1 rx=0x7f15cc006f00 tx=0x7f15cc009f10).stop 2021-02-10T11:41:53.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:53.953 7f15d9dd5700 1 -- 172.21.2.112:0/3477485431 shutdown_connections 2021-02-10T11:41:53.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:53.953 7f15d9dd5700 1 --2- 172.21.2.112:0/3477485431 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f15d410b000 0x7f15d40cf760 unknown :-1 s=CLOSED pgs=2264 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:53.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:53.953 7f15d9dd5700 1 -- 172.21.2.112:0/3477485431 >> 172.21.2.112:0/3477485431 conn(0x7f15d40ce430 msgr2=0x7f15d40d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:53.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:53.953 7f15d9dd5700 1 -- 172.21.2.112:0/3477485431 shutdown_connections 2021-02-10T11:41:53.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:53.953 7f15d9dd5700 1 -- 172.21.2.112:0/3477485431 wait complete. 2021-02-10T11:41:53.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:53.968 7f15d9dd5700 1 Processor -- start 2021-02-10T11:41:53.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:53.968 7f15d9dd5700 1 -- start start 2021-02-10T11:41:53.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:53.968 7f15d9dd5700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f15d41b04f0 0x7f15d41b08f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:53.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:53.968 7f15d9dd5700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f15d408ac80 0x7f15d408b080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:53.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:53.968 7f15d9dd5700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f15d408b6a0 0x7f15d408c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:53.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:53.968 7f15d9dd5700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15d408c760 con 0x7f15d408ac80 2021-02-10T11:41:53.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:53.968 7f15d9dd5700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15d408c980 con 0x7f15d408b6a0 2021-02-10T11:41:53.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:53.968 7f15d9dd5700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15d408cba0 con 0x7f15d41b04f0 2021-02-10T11:41:53.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:53.968 7f15d2ffd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f15d408ac80 0x7f15d408b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:53.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:53.968 7f15d2ffd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f15d408ac80 0x7f15d408b080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:55832/0 (socket says 172.21.2.112:55832) 2021-02-10T11:41:53.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:53.968 7f15d2ffd700 1 -- 172.21.2.112:0/2689665338 learned_addr learned my addr 172.21.2.112:0/2689665338 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:41:53.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:53.969 7f15d2ffd700 1 -- 172.21.2.112:0/2689665338 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f15d41b04f0 msgr2=0x7f15d41b08f0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:41:53.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:53.969 7f15d2ffd700 1 --2- 172.21.2.112:0/2689665338 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f15d41b04f0 0x7f15d41b08f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:53.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:53.969 7f15d2ffd700 1 -- 172.21.2.112:0/2689665338 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f15d408b6a0 msgr2=0x7f15d408c140 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:41:53.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:53.969 7f15d2ffd700 1 --2- 172.21.2.112:0/2689665338 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f15d408b6a0 0x7f15d408c140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:53.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:53.969 7f15d2ffd700 1 -- 172.21.2.112:0/2689665338 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15d408cdc0 con 0x7f15d408ac80 2021-02-10T11:41:53.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:53.969 7f15d2ffd700 1 --2- 172.21.2.112:0/2689665338 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f15d408ac80 0x7f15d408b080 secure :-1 s=READY pgs=2265 cs=0 l=1 rev1=1 rx=0x7f15c800b5b0 tx=0x7f15c800b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:53.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:53.969 7f15bbfff700 1 -- 172.21.2.112:0/2689665338 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f15c800bd40 con 0x7f15d408ac80 2021-02-10T11:41:53.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:53.969 7f15bbfff700 1 -- 172.21.2.112:0/2689665338 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15c8000e20 con 0x7f15d408ac80 2021-02-10T11:41:53.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:53.969 7f15bbfff700 1 -- 172.21.2.112:0/2689665338 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f15c8006210 con 0x7f15d408ac80 2021-02-10T11:41:53.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:53.969 7f15d9dd5700 1 -- 172.21.2.112:0/2689665338 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15d408cdc0 con 0x7f15d408ac80 2021-02-10T11:41:53.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:53.969 7f15d9dd5700 1 -- 172.21.2.112:0/2689665338 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15d408cdc0 con 0x7f15d408ac80 2021-02-10T11:41:53.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:53.970 7f15bbfff700 1 -- 172.21.2.112:0/2689665338 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f15c8006020 con 0x7f15d408ac80 2021-02-10T11:41:53.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:53.970 7f15bbfff700 1 --2- 172.21.2.112:0/2689665338 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f15b40186c0 0x7f15b401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:53.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:53.970 7f15bbfff700 1 -- 172.21.2.112:0/2689665338 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f15c8032590 con 0x7f15d408ac80 2021-02-10T11:41:53.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:53.970 7f15d37fe700 1 --2- 172.21.2.112:0/2689665338 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f15b40186c0 0x7f15b401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:53.971 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:53.971 7f15d37fe700 1 --2- 172.21.2.112:0/2689665338 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f15b40186c0 0x7f15b401abc0 crc :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:53.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:54.001 7f15d9dd5700 1 -- 172.21.2.112:0/2689665338 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15d408d640 con 0x7f15d408ac80 2021-02-10T11:41:54.001 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:54.004 7f15bbfff700 1 -- 172.21.2.112:0/2689665338 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f15c8030d40 con 0x7f15d408ac80 2021-02-10T11:41:54.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:54.132 7f15d9dd5700 1 -- 172.21.2.112:0/2689665338 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000814932", "target": ["mgr", ""], "group_name": "group_0000000000550905"}) v1 -- 0x7f15d408d640 con 0x7f15b40186c0 2021-02-10T11:41:54.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:54.147 7f15bbfff700 1 -- 172.21.2.112:0/2689665338 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f15c4000e20 con 0x7f15b40186c0 2021-02-10T11:41:54.149 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:54.150 7f15d9dd5700 1 -- 172.21.2.112:0/2689665338 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f15b40186c0 msgr2=0x7f15b401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:54.149 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:54.150 7f15d9dd5700 1 --2- 172.21.2.112:0/2689665338 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f15b40186c0 0x7f15b401abc0 crc :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:54.149 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:54.150 7f15d9dd5700 1 -- 172.21.2.112:0/2689665338 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f15d408ac80 msgr2=0x7f15d408b080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:54.149 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:54.150 7f15d9dd5700 1 --2- 172.21.2.112:0/2689665338 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f15d408ac80 0x7f15d408b080 secure :-1 s=READY pgs=2265 cs=0 l=1 rev1=1 rx=0x7f15c800b5b0 tx=0x7f15c800b950).stop 2021-02-10T11:41:54.150 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:54.150 7f15d9dd5700 1 -- 172.21.2.112:0/2689665338 shutdown_connections 2021-02-10T11:41:54.150 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:54.150 7f15d9dd5700 1 --2- 172.21.2.112:0/2689665338 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f15b40186c0 0x7f15b401abc0 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:54.150 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:54.150 7f15d9dd5700 1 --2- 172.21.2.112:0/2689665338 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f15d408b6a0 0x7f15d408c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:54.150 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:54.150 7f15d9dd5700 1 --2- 172.21.2.112:0/2689665338 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f15d408ac80 0x7f15d408b080 unknown :-1 s=CLOSED pgs=2265 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:54.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:54.150 7f15d9dd5700 1 --2- 172.21.2.112:0/2689665338 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f15d41b04f0 0x7f15d41b08f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:54.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:54.150 7f15d9dd5700 1 -- 172.21.2.112:0/2689665338 >> 172.21.2.112:0/2689665338 conn(0x7f15d409fb50 msgr2=0x7f15d40fed60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:54.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:54.150 7f15d9dd5700 1 -- 172.21.2.112:0/2689665338 shutdown_connections 2021-02-10T11:41:54.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:54.150 7f15d9dd5700 1 -- 172.21.2.112:0/2689665338 wait complete. 2021-02-10T11:41:54.157 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.1 && sudo ls volumes 2021-02-10T11:41:54.503 INFO:teuthology.orchestra.run.gibba012.stdout:_deleting 2021-02-10T11:41:54.503 INFO:teuthology.orchestra.run.gibba012.stdout:group_0000000000550905 2021-02-10T11:41:54.504 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:54.504 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:41:54.505 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:41:54.840 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:41:54.840 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:41:54.840 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba012.front.sepia.ceph.com... 2021-02-10T11:41:54.840 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:41:54.840 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:41:55.188 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba012.stderr:ceph-fuse[162110]: fuse finished with error 0 and tester_r 0 2021-02-10T11:41:55.190 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:55.190 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:41:55.190 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:41:55.526 INFO:teuthology.orchestra.run.gibba012.stdout:ext2/ext3 2021-02-10T11:41:55.527 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:41:55.527 INFO:teuthology.orchestra.run:waiting for 900 2021-02-10T11:41:55.527 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:41:55.527 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:41:55.862 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolumegroup rm cephfs group_0000000000550905 2021-02-10T11:41:56.293 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:56.278 7f92b2cb8700 1 Processor -- start 2021-02-10T11:41:56.293 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:56.278 7f92b2cb8700 1 -- start start 2021-02-10T11:41:56.293 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:56.278 7f92b2cb8700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f92ac0d1c70 0x7f92ac0cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:56.293 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:56.278 7f92b2cb8700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92ac0d0170 con 0x7f92ac0cf580 2021-02-10T11:41:56.293 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:56.278 7f92b2cb8700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92ac0fe1c0 con 0x7f92ac0d1870 2021-02-10T11:41:56.293 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:56.278 7f92b2cb8700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92ac1b0cb0 con 0x7f92ac0d1c70 2021-02-10T11:41:56.293 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:56.278 7f92abfff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f92ac0d1c70 0x7f92ac0cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:56.293 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:56.278 7f92b12d5700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f92ac0cf580 0x7f92ac0cf980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:55304/0 (socket says 172.21.2.112:55304) 2021-02-10T11:41:56.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:56.278 7f92b12d5700 1 -- 172.21.2.112:0/1214632639 learned_addr learned my addr 172.21.2.112:0/1214632639 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:41:56.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:56.279 7f92ab7fe700 1 -- 172.21.2.112:0/1214632639 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2387400150 0 0) 0x7f92a4001500 con 0x7f92ac0cf580 2021-02-10T11:41:56.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:56.279 7f92abfff700 1 -- 172.21.2.112:0/1214632639 >> v1:172.21.2.120:6789/0 conn(0x7f92ac0d1870 legacy=0x7f92ac1b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:56.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:56.279 7f92abfff700 1 -- 172.21.2.112:0/1214632639 >> v1:172.21.2.112:6789/0 conn(0x7f92ac0cf580 legacy=0x7f92ac0cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:56.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:56.279 7f92abfff700 1 -- 172.21.2.112:0/1214632639 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92ac109a60 con 0x7f92ac0d1c70 2021-02-10T11:41:56.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:56.279 7f92abfff700 1 --2- 172.21.2.112:0/1214632639 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f92ac0d1c70 0x7f92ac0cef60 secure :-1 s=READY pgs=2267 cs=0 l=1 rev1=1 rx=0x7f929c006650 tx=0x7f929c009a80).ready entity=mon.0 client_cookie=ea6e62cfc07c8115 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:56.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:56.279 7f92ab7fe700 1 -- 172.21.2.112:0/1214632639 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f929c0009b0 con 0x7f92ac0d1c70 2021-02-10T11:41:56.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:56.279 7f92ab7fe700 1 -- 172.21.2.112:0/1214632639 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f929c00c0c0 con 0x7f92ac0d1c70 2021-02-10T11:41:56.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:56.279 7f92ab7fe700 1 -- 172.21.2.112:0/1214632639 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f929c00c520 con 0x7f92ac0d1c70 2021-02-10T11:41:56.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:56.279 7f92b2cb8700 1 -- 172.21.2.112:0/1214632639 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f92ac0d1c70 msgr2=0x7f92ac0cef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:56.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:56.279 7f92b2cb8700 1 --2- 172.21.2.112:0/1214632639 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f92ac0d1c70 0x7f92ac0cef60 secure :-1 s=READY pgs=2267 cs=0 l=1 rev1=1 rx=0x7f929c006650 tx=0x7f929c009a80).stop 2021-02-10T11:41:56.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:56.280 7f92b2cb8700 1 -- 172.21.2.112:0/1214632639 shutdown_connections 2021-02-10T11:41:56.295 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:56.280 7f92b2cb8700 1 --2- 172.21.2.112:0/1214632639 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f92ac0d1c70 0x7f92ac0cef60 unknown :-1 s=CLOSED pgs=2267 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:56.295 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:56.280 7f92b2cb8700 1 -- 172.21.2.112:0/1214632639 >> 172.21.2.112:0/1214632639 conn(0x7f92ac0ce420 msgr2=0x7f92ac0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:56.295 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:56.280 7f92b2cb8700 1 -- 172.21.2.112:0/1214632639 shutdown_connections 2021-02-10T11:41:56.295 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:56.280 7f92b2cb8700 1 -- 172.21.2.112:0/1214632639 wait complete. 2021-02-10T11:41:56.296 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:56.294 7f92b2cb8700 1 Processor -- start 2021-02-10T11:41:56.296 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:56.294 7f92b2cb8700 1 -- start start 2021-02-10T11:41:56.296 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:56.295 7f92b2cb8700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f92ac0c2370 0x7f92ac18f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:56.296 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:56.295 7f92b2cb8700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f92ac084cb0 0x7f92ac1a7470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:56.296 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:56.295 7f92b2cb8700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f92ac1a7a90 0x7f92ac085700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:56.297 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:56.295 7f92b2cb8700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92ac0be840 con 0x7f92ac0c2370 2021-02-10T11:41:56.297 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:56.295 7f92b2cb8700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92ac0c29a0 con 0x7f92ac084cb0 2021-02-10T11:41:56.297 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:56.295 7f92b2cb8700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92ac1a7e90 con 0x7f92ac1a7a90 2021-02-10T11:41:56.297 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:56.295 7f92b0ad4700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f92ac0c2370 0x7f92ac18f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:56.297 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:56.295 7f92b0ad4700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f92ac0c2370 0x7f92ac18f870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:55844/0 (socket says 172.21.2.112:55844) 2021-02-10T11:41:56.297 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:56.295 7f92b0ad4700 1 -- 172.21.2.112:0/1588602280 learned_addr learned my addr 172.21.2.112:0/1588602280 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:41:56.297 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:56.295 7f92b0ad4700 1 -- 172.21.2.112:0/1588602280 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f92ac1a7a90 msgr2=0x7f92ac085700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:56.297 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:56.295 7f92b0ad4700 1 --2- 172.21.2.112:0/1588602280 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f92ac1a7a90 0x7f92ac085700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:56.297 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:56.295 7f92b0ad4700 1 -- 172.21.2.112:0/1588602280 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f92ac084cb0 msgr2=0x7f92ac1a7470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:56.297 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:56.295 7f92b0ad4700 1 --2- 172.21.2.112:0/1588602280 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f92ac084cb0 0x7f92ac1a7470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:56.297 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:56.295 7f92b0ad4700 1 -- 172.21.2.112:0/1588602280 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92ac1a80b0 con 0x7f92ac0c2370 2021-02-10T11:41:56.297 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:56.295 7f92b0ad4700 1 --2- 172.21.2.112:0/1588602280 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f92ac0c2370 0x7f92ac18f870 secure :-1 s=READY pgs=2268 cs=0 l=1 rev1=1 rx=0x7f929400b5b0 tx=0x7f929400b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:56.297 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:56.295 7f92a97fa700 1 -- 172.21.2.112:0/1588602280 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f929400bd40 con 0x7f92ac0c2370 2021-02-10T11:41:56.297 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:56.295 7f92a97fa700 1 -- 172.21.2.112:0/1588602280 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9294006020 con 0x7f92ac0c2370 2021-02-10T11:41:56.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:56.295 7f92a97fa700 1 -- 172.21.2.112:0/1588602280 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f92940054a0 con 0x7f92ac0c2370 2021-02-10T11:41:56.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:56.295 7f92b2cb8700 1 -- 172.21.2.112:0/1588602280 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92ac1a80b0 con 0x7f92ac0c2370 2021-02-10T11:41:56.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:56.296 7f92b2cb8700 1 -- 172.21.2.112:0/1588602280 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92ac1a80b0 con 0x7f92ac0c2370 2021-02-10T11:41:56.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:56.296 7f92a97fa700 1 -- 172.21.2.112:0/1588602280 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9294006200 con 0x7f92ac0c2370 2021-02-10T11:41:56.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:56.296 7f92a97fa700 1 --2- 172.21.2.112:0/1588602280 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9290018730 0x7f929001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:56.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:56.296 7f92a97fa700 1 -- 172.21.2.112:0/1588602280 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f92940327f0 con 0x7f92ac0c2370 2021-02-10T11:41:56.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:56.297 7f92abfff700 1 --2- 172.21.2.112:0/1588602280 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9290018730 0x7f929001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:56.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:56.297 7f92abfff700 1 --2- 172.21.2.112:0/1588602280 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9290018730 0x7f929001ac10 crc :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:56.327 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:56.328 7f92b2cb8700 1 -- 172.21.2.112:0/1588602280 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92ac0be840 con 0x7f92ac0c2370 2021-02-10T11:41:56.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:56.330 7f92a97fa700 1 -- 172.21.2.112:0/1588602280 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f92940009b0 con 0x7f92ac0c2370 2021-02-10T11:41:56.456 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:56.457 7f92b2cb8700 1 -- 172.21.2.112:0/1588602280 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolumegroup rm", "vol_name": "cephfs", "target": ["mgr", ""], "group_name": "group_0000000000550905"}) v1 -- 0x7f92ac0be840 con 0x7f9290018730 2021-02-10T11:41:56.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:56.461 7f92a97fa700 1 -- 172.21.2.112:0/1588602280 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f929c000940 con 0x7f9290018730 2021-02-10T11:41:56.461 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:56.462 7f92b2cb8700 1 -- 172.21.2.112:0/1588602280 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9290018730 msgr2=0x7f929001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:56.461 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:56.462 7f92b2cb8700 1 --2- 172.21.2.112:0/1588602280 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9290018730 0x7f929001ac10 crc :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:56.461 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:56.462 7f92b2cb8700 1 -- 172.21.2.112:0/1588602280 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f92ac0c2370 msgr2=0x7f92ac18f870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:56.461 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:56.462 7f92b2cb8700 1 --2- 172.21.2.112:0/1588602280 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f92ac0c2370 0x7f92ac18f870 secure :-1 s=READY pgs=2268 cs=0 l=1 rev1=1 rx=0x7f929400b5b0 tx=0x7f929400b950).stop 2021-02-10T11:41:56.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:56.462 7f92b2cb8700 1 -- 172.21.2.112:0/1588602280 shutdown_connections 2021-02-10T11:41:56.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:56.462 7f92b2cb8700 1 --2- 172.21.2.112:0/1588602280 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9290018730 0x7f929001ac10 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:56.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:56.462 7f92b2cb8700 1 --2- 172.21.2.112:0/1588602280 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f92ac084cb0 0x7f92ac1a7470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:56.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:56.462 7f92b2cb8700 1 --2- 172.21.2.112:0/1588602280 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f92ac1a7a90 0x7f92ac085700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:56.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:56.462 7f92b2cb8700 1 --2- 172.21.2.112:0/1588602280 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f92ac0c2370 0x7f92ac18f870 unknown :-1 s=CLOSED pgs=2268 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:56.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:56.462 7f92b2cb8700 1 -- 172.21.2.112:0/1588602280 >> 172.21.2.112:0/1588602280 conn(0x7f92ac0b5dc0 msgr2=0x7f92ac0c2d90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:56.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:56.462 7f92b2cb8700 1 -- 172.21.2.112:0/1588602280 shutdown_connections 2021-02-10T11:41:56.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:41:56.463 7f92b2cb8700 1 -- 172.21.2.112:0/1588602280 wait complete. 2021-02-10T11:41:56.469 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:41:56.469 DEBUG:teuthology.orchestra.run.gibba012:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:41:56.814 DEBUG:teuthology.orchestra.run.gibba020:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:41:57.159 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:41:57.159 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:57.159 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:41:57.159 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:41:57.495 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:41:57.495 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:41:57.495 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba012.front.sepia.ceph.com... 2021-02-10T11:41:57.495 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:41:57.496 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:41:57.844 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba012.stderr:ceph-fuse[161470]: fuse finished with error 0 and tester_r 0 2021-02-10T11:41:57.845 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:57.845 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:41:57.845 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:41:58.179 INFO:teuthology.orchestra.run.gibba012.stdout:ext2/ext3 2021-02-10T11:41:58.180 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:41:58.180 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:41:58.180 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:41:58.514 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:58.515 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:41:58.515 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:41:58.849 INFO:teuthology.orchestra.run.gibba012.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: No such file or directory 2021-02-10T11:41:58.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:41:58.849 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2021-02-10T11:41:58.850 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:41:58.850 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:41:59.189 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:59.190 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:41:59.190 DEBUG:teuthology.orchestra.run.gibba020:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:41:59.525 INFO:teuthology.orchestra.run.gibba020.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:41:59.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:41:59.526 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:41:59.526 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:41:59.526 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:41:59.861 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:59.861 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:41:59.861 DEBUG:teuthology.orchestra.run.gibba020:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:42:00.197 INFO:teuthology.orchestra.run.gibba020.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:42:00.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:42:00.197 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:42:00.197 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:42:00.197 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:42:00.533 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config rm mon mon_allow_pool_delete 2021-02-10T11:42:00.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:00.949 7fafd8ab2700 1 Processor -- start 2021-02-10T11:42:00.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:00.949 7fafd8ab2700 1 -- start start 2021-02-10T11:42:00.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:00.950 7fafd8ab2700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fafd40d1870 0x7fafd41b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:00.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:00.950 7fafd8ab2700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fafd41b0a10 0x7fafd40cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:00.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:00.950 7fafd8ab2700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafd40f0b10 con 0x7fafd40d1c70 2021-02-10T11:42:00.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:00.950 7fafd8ab2700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafd40e5ed0 con 0x7fafd41b0a10 2021-02-10T11:42:00.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:00.950 7fafd8ab2700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafd40f0650 con 0x7fafd40d1870 2021-02-10T11:42:00.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:00.950 7fafd205d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fafd41b0a10 0x7fafd40cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:00.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:00.950 7fafd205d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fafd41b0a10 0x7fafd40cf330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:55852/0 (socket says 172.21.2.112:55852) 2021-02-10T11:42:00.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:00.950 7fafd205d700 1 -- 172.21.2.112:0/4001260189 learned_addr learned my addr 172.21.2.112:0/4001260189 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:42:00.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:00.950 7fafd205d700 1 -- 172.21.2.112:0/4001260189 >> v2:172.21.2.120:3300/0 conn(0x7fafd40d1870 msgr2=0x7fafd41b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:00.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:00.950 7fafd205d700 1 --2- 172.21.2.112:0/4001260189 >> v2:172.21.2.120:3300/0 conn(0x7fafd40d1870 0x7fafd41b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:00.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:00.950 7fafd205d700 1 -- 172.21.2.112:0/4001260189 >> v1:172.21.2.120:6789/0 conn(0x7fafd40d1c70 legacy=0x7fafd40cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:00.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:00.950 7fafd205d700 1 -- 172.21.2.112:0/4001260189 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafd40f0840 con 0x7fafd41b0a10 2021-02-10T11:42:00.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:00.950 7fafd205d700 1 --2- 172.21.2.112:0/4001260189 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fafd41b0a10 0x7fafd40cf330 secure :-1 s=READY pgs=2269 cs=0 l=1 rev1=1 rx=0x7fafc8006700 tx=0x7fafc8009b80).ready entity=mon.0 client_cookie=6f1183fe119cdb98 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:00.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:00.951 7fafd185c700 1 -- 172.21.2.112:0/4001260189 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fafc800c0c0 con 0x7fafd41b0a10 2021-02-10T11:42:00.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:00.951 7fafd185c700 1 -- 172.21.2.112:0/4001260189 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafc800c440 con 0x7fafd41b0a10 2021-02-10T11:42:00.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:00.951 7fafd185c700 1 -- 172.21.2.112:0/4001260189 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fafc800c960 con 0x7fafd41b0a10 2021-02-10T11:42:00.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:00.951 7fafd8ab2700 1 -- 172.21.2.112:0/4001260189 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fafd41b0a10 msgr2=0x7fafd40cf330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:00.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:00.951 7fafd8ab2700 1 --2- 172.21.2.112:0/4001260189 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fafd41b0a10 0x7fafd40cf330 secure :-1 s=READY pgs=2269 cs=0 l=1 rev1=1 rx=0x7fafc8006700 tx=0x7fafc8009b80).stop 2021-02-10T11:42:00.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:00.952 7fafd8ab2700 1 -- 172.21.2.112:0/4001260189 shutdown_connections 2021-02-10T11:42:00.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:00.952 7fafd8ab2700 1 --2- 172.21.2.112:0/4001260189 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fafd41b0a10 0x7fafd40cf330 unknown :-1 s=CLOSED pgs=2269 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:00.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:00.952 7fafd8ab2700 1 --2- 172.21.2.112:0/4001260189 >> v2:172.21.2.120:3300/0 conn(0x7fafd40d1870 0x7fafd41b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:00.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:00.952 7fafd8ab2700 1 -- 172.21.2.112:0/4001260189 >> 172.21.2.112:0/4001260189 conn(0x7fafd40ce450 msgr2=0x7fafd40d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:00.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:00.952 7fafd8ab2700 1 -- 172.21.2.112:0/4001260189 shutdown_connections 2021-02-10T11:42:00.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:00.952 7fafd8ab2700 1 -- 172.21.2.112:0/4001260189 wait complete. 2021-02-10T11:42:00.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:00.966 7fafd8ab2700 1 Processor -- start 2021-02-10T11:42:00.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:00.966 7fafd8ab2700 1 -- start start 2021-02-10T11:42:00.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:00.967 7fafd8ab2700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fafd40a2fb0 0x7fafd40a33b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:00.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:00.967 7fafd8ab2700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fafd40a39d0 0x7fafd40a4860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:00.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:00.967 7fafd8ab2700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fafd40a3dd0 0x7fafd40a4e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:00.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:00.967 7fafd8ab2700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafd40a2620 con 0x7fafd40a39d0 2021-02-10T11:42:00.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:00.967 7fafd8ab2700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafd40a2840 con 0x7fafd40a3dd0 2021-02-10T11:42:00.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:00.967 7fafd8ab2700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafd40a2b80 con 0x7fafd40a2fb0 2021-02-10T11:42:00.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:00.967 7fafd205d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fafd40a39d0 0x7fafd40a4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:00.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:00.967 7fafd205d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fafd40a39d0 0x7fafd40a4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:55858/0 (socket says 172.21.2.112:55858) 2021-02-10T11:42:00.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:00.967 7fafd205d700 1 -- 172.21.2.112:0/3687293929 learned_addr learned my addr 172.21.2.112:0/3687293929 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:42:00.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:00.967 7fafd205d700 1 -- 172.21.2.112:0/3687293929 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fafd40a2fb0 msgr2=0x7fafd40a33b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:42:00.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:00.967 7fafd205d700 1 --2- 172.21.2.112:0/3687293929 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fafd40a2fb0 0x7fafd40a33b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:00.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:00.967 7fafd205d700 1 -- 172.21.2.112:0/3687293929 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fafd40a3dd0 msgr2=0x7fafd40a4e80 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:42:00.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:00.967 7fafd205d700 1 --2- 172.21.2.112:0/3687293929 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fafd40a3dd0 0x7fafd40a4e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:00.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:00.967 7fafd205d700 1 -- 172.21.2.112:0/3687293929 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafd40a54a0 con 0x7fafd40a39d0 2021-02-10T11:42:00.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:00.967 7fafd205d700 1 --2- 172.21.2.112:0/3687293929 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fafd40a39d0 0x7fafd40a4860 secure :-1 s=READY pgs=2270 cs=0 l=1 rev1=1 rx=0x7fafc800b5e0 tx=0x7fafc800b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:00.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:00.968 7fafc37fe700 1 -- 172.21.2.112:0/3687293929 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fafc800bd70 con 0x7fafd40a39d0 2021-02-10T11:42:00.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:00.968 7fafc37fe700 1 -- 172.21.2.112:0/3687293929 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafc8002030 con 0x7fafd40a39d0 2021-02-10T11:42:00.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:00.968 7fafc37fe700 1 -- 172.21.2.112:0/3687293929 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fafc8002550 con 0x7fafd40a39d0 2021-02-10T11:42:00.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:00.968 7fafd8ab2700 1 -- 172.21.2.112:0/3687293929 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafd40a54a0 con 0x7fafd40a39d0 2021-02-10T11:42:00.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:00.968 7fafd8ab2700 1 -- 172.21.2.112:0/3687293929 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafd40a5a30 con 0x7fafd40a39d0 2021-02-10T11:42:00.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:00.968 7fafc37fe700 1 -- 172.21.2.112:0/3687293929 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fafc8000c90 con 0x7fafd40a39d0 2021-02-10T11:42:00.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:00.968 7fafc37fe700 1 --2- 172.21.2.112:0/3687293929 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fafb80186c0 0x7fafb801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:00.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:00.968 7fafc37fe700 1 -- 172.21.2.112:0/3687293929 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fafc80325e0 con 0x7fafd40a39d0 2021-02-10T11:42:00.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:00.969 7fafd285e700 1 --2- 172.21.2.112:0/3687293929 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fafb80186c0 0x7fafb801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:00.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:00.969 7fafd285e700 1 --2- 172.21.2.112:0/3687293929 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fafb80186c0 0x7fafb801abc0 crc :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:00.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:01.000 7fafd8ab2700 1 -- 172.21.2.112:0/3687293929 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafd40ee060 con 0x7fafd40a39d0 2021-02-10T11:42:01.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:01.002 7fafc37fe700 1 -- 172.21.2.112:0/3687293929 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fafc8030d90 con 0x7fafd40a39d0 2021-02-10T11:42:01.150 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:01.152 7fafd8ab2700 1 -- 172.21.2.112:0/3687293929 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fafd40a5d20 con 0x7fafd40a39d0 2021-02-10T11:42:01.154 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:01.155 7fafc37fe700 1 -- 172.21.2.112:0/3687293929 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v31) v1 ==== 104+0+0 (secure 0 0 0) 0x7fafc8000c90 con 0x7fafd40a39d0 2021-02-10T11:42:01.159 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:01.160 7fafd8ab2700 1 -- 172.21.2.112:0/3687293929 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fafb80186c0 msgr2=0x7fafb801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:01.159 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:01.160 7fafd8ab2700 1 --2- 172.21.2.112:0/3687293929 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fafb80186c0 0x7fafb801abc0 crc :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:01.159 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:01.160 7fafd8ab2700 1 -- 172.21.2.112:0/3687293929 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fafd40a39d0 msgr2=0x7fafd40a4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:01.159 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:01.160 7fafd8ab2700 1 --2- 172.21.2.112:0/3687293929 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fafd40a39d0 0x7fafd40a4860 secure :-1 s=READY pgs=2270 cs=0 l=1 rev1=1 rx=0x7fafc800b5e0 tx=0x7fafc800b980).stop 2021-02-10T11:42:01.159 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:01.160 7fafd8ab2700 1 -- 172.21.2.112:0/3687293929 shutdown_connections 2021-02-10T11:42:01.159 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:01.161 7fafd8ab2700 1 --2- 172.21.2.112:0/3687293929 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fafb80186c0 0x7fafb801abc0 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:01.159 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:01.161 7fafd8ab2700 1 --2- 172.21.2.112:0/3687293929 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fafd40a3dd0 0x7fafd40a4e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:01.160 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:01.161 7fafd8ab2700 1 --2- 172.21.2.112:0/3687293929 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fafd40a39d0 0x7fafd40a4860 unknown :-1 s=CLOSED pgs=2270 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:01.160 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:01.161 7fafd8ab2700 1 --2- 172.21.2.112:0/3687293929 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fafd40a2fb0 0x7fafd40a33b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:01.160 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:01.161 7fafd8ab2700 1 -- 172.21.2.112:0/3687293929 >> 172.21.2.112:0/3687293929 conn(0x7fafd408f030 msgr2=0x7fafd40b5e80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:01.160 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:01.161 7fafd8ab2700 1 -- 172.21.2.112:0/3687293929 shutdown_connections 2021-02-10T11:42:01.160 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:01.161 7fafd8ab2700 1 -- 172.21.2.112:0/3687293929 wait complete. 2021-02-10T11:42:01.169 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestVolumes.test_multitenant_subvolumes' 2021-02-10T11:42:01.596 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:01.582 7fe8dd799700 1 Processor -- start 2021-02-10T11:42:01.596 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:01.582 7fe8dd799700 1 -- start start 2021-02-10T11:42:01.596 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:01.582 7fe8dd799700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fe8d81b0470 0x7fe8d81b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:01.596 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:01.582 7fe8dd799700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fe8d80ef610 0x7fe8d80efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:01.596 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:01.582 7fe8dd799700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8d80f0b50 con 0x7fe8d80f0030 2021-02-10T11:42:01.597 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:01.582 7fe8dd799700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8d80e52e0 con 0x7fe8d80ef610 2021-02-10T11:42:01.597 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:01.582 7fe8dd799700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8d80e5490 con 0x7fe8d81b0470 2021-02-10T11:42:01.597 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:01.582 7fe8ced73700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fe8d80ef610 0x7fe8d80efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:01.597 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:01.582 7fe8ced73700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fe8d80ef610 0x7fe8d80efa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:55864/0 (socket says 172.21.2.112:55864) 2021-02-10T11:42:01.597 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:01.582 7fe8ced73700 1 -- 172.21.2.112:0/3323375496 learned_addr learned my addr 172.21.2.112:0/3323375496 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:42:01.597 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:01.582 7fe8d7574700 1 --2- 172.21.2.112:0/3323375496 >> v2:172.21.2.120:3300/0 conn(0x7fe8d81b0470 0x7fe8d81b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:01.597 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:01.582 7fe8ced73700 1 -- 172.21.2.112:0/3323375496 >> v2:172.21.2.120:3300/0 conn(0x7fe8d81b0470 msgr2=0x7fe8d81b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:01.597 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:01.582 7fe8ced73700 1 --2- 172.21.2.112:0/3323375496 >> v2:172.21.2.120:3300/0 conn(0x7fe8d81b0470 0x7fe8d81b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:01.598 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:01.582 7fe8ced73700 1 -- 172.21.2.112:0/3323375496 >> v1:172.21.2.120:6789/0 conn(0x7fe8d80f0030 legacy=0x7fe8d80f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:01.598 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:01.582 7fe8ced73700 1 -- 172.21.2.112:0/3323375496 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8d80f0430 con 0x7fe8d80ef610 2021-02-10T11:42:01.598 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:01.583 7fe8ced73700 1 --2- 172.21.2.112:0/3323375496 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe8d80ef610 0x7fe8d80efa10 secure :-1 s=READY pgs=2271 cs=0 l=1 rev1=1 rx=0x7fe8c8006700 tx=0x7fe8c8009b80).ready entity=mon.0 client_cookie=295b25cf98b2ca33 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:01.598 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:01.583 7fe8d6d73700 1 -- 172.21.2.112:0/3323375496 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe8c800c0c0 con 0x7fe8d80ef610 2021-02-10T11:42:01.598 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:01.583 7fe8d6d73700 1 -- 172.21.2.112:0/3323375496 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8c800c440 con 0x7fe8d80ef610 2021-02-10T11:42:01.598 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:01.583 7fe8d6d73700 1 -- 172.21.2.112:0/3323375496 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe8c800c920 con 0x7fe8d80ef610 2021-02-10T11:42:01.598 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:01.583 7fe8dd799700 1 -- 172.21.2.112:0/3323375496 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe8d80ef610 msgr2=0x7fe8d80efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:01.598 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:01.583 7fe8dd799700 1 --2- 172.21.2.112:0/3323375496 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe8d80ef610 0x7fe8d80efa10 secure :-1 s=READY pgs=2271 cs=0 l=1 rev1=1 rx=0x7fe8c8006700 tx=0x7fe8c8009b80).stop 2021-02-10T11:42:01.598 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:01.583 7fe8dd799700 1 -- 172.21.2.112:0/3323375496 shutdown_connections 2021-02-10T11:42:01.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:01.583 7fe8dd799700 1 --2- 172.21.2.112:0/3323375496 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe8d80ef610 0x7fe8d80efa10 unknown :-1 s=CLOSED pgs=2271 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:01.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:01.583 7fe8dd799700 1 --2- 172.21.2.112:0/3323375496 >> v2:172.21.2.120:3300/0 conn(0x7fe8d81b0470 0x7fe8d81b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:01.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:01.583 7fe8dd799700 1 -- 172.21.2.112:0/3323375496 >> 172.21.2.112:0/3323375496 conn(0x7fe8d80ce3e0 msgr2=0x7fe8d80e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:01.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:01.583 7fe8dd799700 1 -- 172.21.2.112:0/3323375496 shutdown_connections 2021-02-10T11:42:01.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:01.583 7fe8dd799700 1 -- 172.21.2.112:0/3323375496 wait complete. 2021-02-10T11:42:01.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:01.598 7fe8dd799700 1 Processor -- start 2021-02-10T11:42:01.602 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:01.598 7fe8dd799700 1 -- start start 2021-02-10T11:42:01.602 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:01.598 7fe8dd799700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe8d80aa8d0 0x7fe8d819a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:01.602 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:01.598 7fe8dd799700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe8d819e9d0 0x7fe8d819a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:01.602 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:01.598 7fe8dd799700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe8d819ad40 0x7fe8d819b770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:01.602 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:01.598 7fe8dd799700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8d819b1f0 con 0x7fe8d819ad40 2021-02-10T11:42:01.602 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:01.598 7fe8dd799700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8d8109b30 con 0x7fe8d80aa8d0 2021-02-10T11:42:01.602 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:01.598 7fe8dd799700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8d8099620 con 0x7fe8d819e9d0 2021-02-10T11:42:01.602 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:01.598 7fe8d7d75700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe8d819ad40 0x7fe8d819b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:01.602 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:01.598 7fe8d7d75700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe8d819ad40 0x7fe8d819b770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:55870/0 (socket says 172.21.2.112:55870) 2021-02-10T11:42:01.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:01.598 7fe8d7d75700 1 -- 172.21.2.112:0/2353810723 learned_addr learned my addr 172.21.2.112:0/2353810723 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:42:01.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:01.598 7fe8d7574700 1 --2- 172.21.2.112:0/2353810723 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe8d80aa8d0 0x7fe8d819a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:01.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:01.599 7fe8ced73700 1 --2- 172.21.2.112:0/2353810723 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe8d819e9d0 0x7fe8d819a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:01.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:01.599 7fe8d7d75700 1 -- 172.21.2.112:0/2353810723 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe8d819e9d0 msgr2=0x7fe8d819a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:01.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:01.599 7fe8d7d75700 1 --2- 172.21.2.112:0/2353810723 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe8d819e9d0 0x7fe8d819a720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:01.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:01.599 7fe8d7d75700 1 -- 172.21.2.112:0/2353810723 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe8d80aa8d0 msgr2=0x7fe8d819a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:01.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:01.599 7fe8d7d75700 1 --2- 172.21.2.112:0/2353810723 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe8d80aa8d0 0x7fe8d819a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:01.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:01.599 7fe8d7d75700 1 -- 172.21.2.112:0/2353810723 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8d8109640 con 0x7fe8d819ad40 2021-02-10T11:42:01.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:01.599 7fe8d7d75700 1 --2- 172.21.2.112:0/2353810723 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe8d819ad40 0x7fe8d819b770 secure :-1 s=READY pgs=2272 cs=0 l=1 rev1=1 rx=0x7fe8d0009940 tx=0x7fe8d000bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:01.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:01.599 7fe8d4d6f700 1 -- 172.21.2.112:0/2353810723 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe8d0000d30 con 0x7fe8d819ad40 2021-02-10T11:42:01.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:01.599 7fe8d4d6f700 1 -- 172.21.2.112:0/2353810723 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8d000c2f0 con 0x7fe8d819ad40 2021-02-10T11:42:01.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:01.599 7fe8dd799700 1 -- 172.21.2.112:0/2353810723 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8d8109640 con 0x7fe8d819ad40 2021-02-10T11:42:01.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:01.599 7fe8dd799700 1 -- 172.21.2.112:0/2353810723 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8d8109640 con 0x7fe8d819ad40 2021-02-10T11:42:01.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:01.599 7fe8d4d6f700 1 -- 172.21.2.112:0/2353810723 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe8d000c7d0 con 0x7fe8d819ad40 2021-02-10T11:42:01.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:01.600 7fe8d4d6f700 1 -- 172.21.2.112:0/2353810723 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe8d0001220 con 0x7fe8d819ad40 2021-02-10T11:42:01.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:01.600 7fe8d4d6f700 1 --2- 172.21.2.112:0/2353810723 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe8bc018730 0x7fe8bc01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:01.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:01.600 7fe8d4d6f700 1 -- 172.21.2.112:0/2353810723 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fe8d0032b80 con 0x7fe8d819ad40 2021-02-10T11:42:01.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:01.600 7fe8d7574700 1 --2- 172.21.2.112:0/2353810723 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe8bc018730 0x7fe8bc01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:01.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:01.601 7fe8d7574700 1 --2- 172.21.2.112:0/2353810723 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe8bc018730 0x7fe8bc01ac10 crc :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:01.630 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:01.632 7fe8dd799700 1 -- 172.21.2.112:0/2353810723 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8d80f0b50 con 0x7fe8d819ad40 2021-02-10T11:42:01.633 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:01.634 7fe8d4d6f700 1 -- 172.21.2.112:0/2353810723 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe8d000be30 con 0x7fe8d819ad40 2021-02-10T11:42:01.766 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:01.768 7fe8dd799700 1 -- 172.21.2.112:0/2353810723 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_multitenant_subvolumes"]} v 0) v1 -- 0x7fe8d8109ae0 con 0x7fe8d819ad40 2021-02-10T11:42:02.660 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:02.661 7fe8d4d6f700 1 -- 172.21.2.112:0/2353810723 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_multitenant_subvolumes"]}]=0 v607) v1 ==== 144+0+0 (secure 0 0 0) 0x7fe8d0001220 con 0x7fe8d819ad40 2021-02-10T11:42:02.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:02.688 7fe8dd799700 1 -- 172.21.2.112:0/2353810723 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe8bc018730 msgr2=0x7fe8bc01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:02.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:02.688 7fe8dd799700 1 --2- 172.21.2.112:0/2353810723 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe8bc018730 0x7fe8bc01ac10 crc :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:02.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:02.688 7fe8dd799700 1 -- 172.21.2.112:0/2353810723 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe8d819ad40 msgr2=0x7fe8d819b770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:02.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:02.688 7fe8dd799700 1 --2- 172.21.2.112:0/2353810723 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe8d819ad40 0x7fe8d819b770 secure :-1 s=READY pgs=2272 cs=0 l=1 rev1=1 rx=0x7fe8d0009940 tx=0x7fe8d000bcf0).stop 2021-02-10T11:42:02.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:02.688 7fe8dd799700 1 -- 172.21.2.112:0/2353810723 shutdown_connections 2021-02-10T11:42:02.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:02.688 7fe8dd799700 1 --2- 172.21.2.112:0/2353810723 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe8bc018730 0x7fe8bc01ac10 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:02.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:02.688 7fe8dd799700 1 --2- 172.21.2.112:0/2353810723 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe8d819ad40 0x7fe8d819b770 unknown :-1 s=CLOSED pgs=2272 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:02.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:02.688 7fe8dd799700 1 --2- 172.21.2.112:0/2353810723 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe8d819e9d0 0x7fe8d819a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:02.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:02.688 7fe8dd799700 1 --2- 172.21.2.112:0/2353810723 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe8d80aa8d0 0x7fe8d819a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:02.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:02.688 7fe8dd799700 1 -- 172.21.2.112:0/2353810723 >> 172.21.2.112:0/2353810723 conn(0x7fe8d819f7b0 msgr2=0x7fe8d818f380 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:02.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:02.689 7fe8dd799700 1 -- 172.21.2.112:0/2353810723 shutdown_connections 2021-02-10T11:42:02.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:02.689 7fe8dd799700 1 -- 172.21.2.112:0/2353810723 wait complete. 2021-02-10T11:42:02.695 INFO:tasks.cephfs_test_runner:test_multitenant_subvolumes (tasks.cephfs.test_volumes.TestVolumes) ... ok 2021-02-10T11:42:02.695 INFO:tasks.cephfs_test_runner:Starting test: test_non_clone_status (tasks.cephfs.test_volumes.TestVolumes) 2021-02-10T11:42:02.695 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestVolumes.test_non_clone_status' 2021-02-10T11:42:03.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:03.107 7f1afc149700 1 Processor -- start 2021-02-10T11:42:03.126 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:03.107 7f1afc149700 1 -- start start 2021-02-10T11:42:03.126 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:03.108 7f1afc149700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f1af40d1c70 0x7f1af40f20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:03.126 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:03.108 7f1afc149700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1af40f0b10 con 0x7f1af40d1870 2021-02-10T11:42:03.126 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:03.108 7f1afc149700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1af41b0be0 con 0x7f1af40d0340 2021-02-10T11:42:03.126 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:03.108 7f1afc149700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1af40e5ed0 con 0x7f1af40d1c70 2021-02-10T11:42:03.127 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:03.108 7f1af9f65700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f1af40d1870 0x7f1af41b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:55340/0 (socket says 172.21.2.112:55340) 2021-02-10T11:42:03.127 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:03.108 7f1af9f65700 1 -- 172.21.2.112:0/1918032976 learned_addr learned my addr 172.21.2.112:0/1918032976 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:42:03.127 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:03.108 7f1af9764700 1 --1- 172.21.2.112:0/1918032976 >> v1:172.21.2.120:6789/0 conn(0x7f1af40d0340 0x7f1af40ef640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:48898/0 (socket says 172.21.2.112:48898) 2021-02-10T11:42:03.127 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:03.108 7f1afa766700 1 --2- 172.21.2.112:0/1918032976 >> v2:172.21.2.112:3300/0 conn(0x7f1af40d1c70 0x7f1af40f20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:03.127 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:03.108 7f1af8f63700 1 -- 172.21.2.112:0/1918032976 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2982857558 0 0) 0x7f1af0000c60 con 0x7f1af40d0340 2021-02-10T11:42:03.127 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:03.108 7f1af8f63700 1 -- 172.21.2.112:0/1918032976 --> v1:172.21.2.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ae0002fe0 con 0x7f1af40d0340 2021-02-10T11:42:03.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:03.108 7f1af8f63700 1 -- 172.21.2.112:0/1918032976 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 964668677 0 0) 0x7f1aec001060 con 0x7f1af40d1870 2021-02-10T11:42:03.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:03.108 7f1af8f63700 1 -- 172.21.2.112:0/1918032976 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ae00042c0 con 0x7f1af40d1870 2021-02-10T11:42:03.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:03.109 7f1afa766700 1 -- 172.21.2.112:0/1918032976 >> v1:172.21.2.120:6789/0 conn(0x7f1af40d0340 legacy=0x7f1af40ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:03.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:03.109 7f1afa766700 1 -- 172.21.2.112:0/1918032976 >> v1:172.21.2.112:6789/0 conn(0x7f1af40d1870 legacy=0x7f1af41b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:03.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:03.109 7f1afa766700 1 -- 172.21.2.112:0/1918032976 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1af40d3360 con 0x7f1af40d1c70 2021-02-10T11:42:03.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:03.109 7f1afa766700 1 --2- 172.21.2.112:0/1918032976 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1af40d1c70 0x7f1af40f20a0 secure :-1 s=READY pgs=2274 cs=0 l=1 rev1=1 rx=0x7f1ae4006ab0 tx=0x7f1ae4009b40).ready entity=mon.0 client_cookie=752e0b2def0985bd server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:03.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:03.109 7f1af8f63700 1 -- 172.21.2.112:0/1918032976 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1ae40022e0 con 0x7f1af40d1c70 2021-02-10T11:42:03.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:03.109 7f1af8f63700 1 -- 172.21.2.112:0/1918032976 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ae4002660 con 0x7f1af40d1c70 2021-02-10T11:42:03.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:03.109 7f1af8f63700 1 -- 172.21.2.112:0/1918032976 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1ae4002b40 con 0x7f1af40d1c70 2021-02-10T11:42:03.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:03.109 7f1afc149700 1 -- 172.21.2.112:0/1918032976 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1af40d1c70 msgr2=0x7f1af40f20a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:03.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:03.109 7f1afc149700 1 --2- 172.21.2.112:0/1918032976 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1af40d1c70 0x7f1af40f20a0 secure :-1 s=READY pgs=2274 cs=0 l=1 rev1=1 rx=0x7f1ae4006ab0 tx=0x7f1ae4009b40).stop 2021-02-10T11:42:03.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:03.110 7f1afc149700 1 -- 172.21.2.112:0/1918032976 shutdown_connections 2021-02-10T11:42:03.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:03.110 7f1afc149700 1 --2- 172.21.2.112:0/1918032976 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1af40d1c70 0x7f1af40f20a0 unknown :-1 s=CLOSED pgs=2274 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:03.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:03.110 7f1afc149700 1 -- 172.21.2.112:0/1918032976 >> 172.21.2.112:0/1918032976 conn(0x7f1af40ce450 msgr2=0x7f1af40d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:03.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:03.110 7f1afc149700 1 -- 172.21.2.112:0/1918032976 shutdown_connections 2021-02-10T11:42:03.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:03.110 7f1afc149700 1 -- 172.21.2.112:0/1918032976 wait complete. 2021-02-10T11:42:03.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:03.124 7f1afc149700 1 Processor -- start 2021-02-10T11:42:03.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:03.125 7f1afc149700 1 -- start start 2021-02-10T11:42:03.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:03.125 7f1afc149700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1af40ada90 0x7f1af40ade90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:03.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:03.125 7f1afc149700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1af4099d20 0x7f1af409a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:03.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:03.125 7f1afc149700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1af409a740 0x7f1af4097bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:03.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:03.125 7f1afc149700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1af40997a0 con 0x7f1af409a740 2021-02-10T11:42:03.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:03.125 7f1afc149700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1af418f0c0 con 0x7f1af4099d20 2021-02-10T11:42:03.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:03.125 7f1afc149700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1af409ab40 con 0x7f1af40ada90 2021-02-10T11:42:03.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:03.125 7f1afa766700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1af409a740 0x7f1af4097bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:03.131 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:03.125 7f1afa766700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1af409a740 0x7f1af4097bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:55888/0 (socket says 172.21.2.112:55888) 2021-02-10T11:42:03.131 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:03.125 7f1afa766700 1 -- 172.21.2.112:0/1205359163 learned_addr learned my addr 172.21.2.112:0/1205359163 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:42:03.131 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:03.125 7f1af9f65700 1 --2- 172.21.2.112:0/1205359163 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1af40ada90 0x7f1af40ade90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:03.131 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:03.125 7f1af9764700 1 --2- 172.21.2.112:0/1205359163 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1af4099d20 0x7f1af409a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:03.131 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:03.125 7f1afa766700 1 -- 172.21.2.112:0/1205359163 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1af40ada90 msgr2=0x7f1af40ade90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:03.131 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:03.125 7f1afa766700 1 --2- 172.21.2.112:0/1205359163 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1af40ada90 0x7f1af40ade90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:03.132 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:03.125 7f1afa766700 1 -- 172.21.2.112:0/1205359163 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1af4099d20 msgr2=0x7f1af409a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:03.132 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:03.125 7f1afa766700 1 --2- 172.21.2.112:0/1205359163 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1af4099d20 0x7f1af409a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:03.132 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:03.125 7f1afa766700 1 -- 172.21.2.112:0/1205359163 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1af409ad60 con 0x7f1af409a740 2021-02-10T11:42:03.132 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:03.125 7f1af9764700 1 --2- 172.21.2.112:0/1205359163 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1af4099d20 0x7f1af409a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:42:03.132 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:03.126 7f1afa766700 1 --2- 172.21.2.112:0/1205359163 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1af409a740 0x7f1af4097bc0 secure :-1 s=READY pgs=2275 cs=0 l=1 rev1=1 rx=0x7f1ae400b5e0 tx=0x7f1ae400b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:03.133 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:03.126 7f1aea7fc700 1 -- 172.21.2.112:0/1205359163 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1ae4002030 con 0x7f1af409a740 2021-02-10T11:42:03.133 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:03.126 7f1aea7fc700 1 -- 172.21.2.112:0/1205359163 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ae40022f0 con 0x7f1af409a740 2021-02-10T11:42:03.133 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:03.126 7f1aea7fc700 1 -- 172.21.2.112:0/1205359163 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1ae4002750 con 0x7f1af409a740 2021-02-10T11:42:03.133 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:03.126 7f1afc149700 1 -- 172.21.2.112:0/1205359163 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1af409ad60 con 0x7f1af409a740 2021-02-10T11:42:03.133 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:03.126 7f1afc149700 1 -- 172.21.2.112:0/1205359163 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1af409ad60 con 0x7f1af409a740 2021-02-10T11:42:03.133 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:03.127 7f1aea7fc700 1 -- 172.21.2.112:0/1205359163 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1ae4002020 con 0x7f1af409a740 2021-02-10T11:42:03.134 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:03.127 7f1aea7fc700 1 --2- 172.21.2.112:0/1205359163 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1adc018730 0x7f1adc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:03.134 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:03.127 7f1aea7fc700 1 -- 172.21.2.112:0/1205359163 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f1ae4032640 con 0x7f1af409a740 2021-02-10T11:42:03.134 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:03.127 7f1af9f65700 1 --2- 172.21.2.112:0/1205359163 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1adc018730 0x7f1adc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:03.135 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:03.127 7f1af9f65700 1 --2- 172.21.2.112:0/1205359163 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1adc018730 0x7f1adc01abc0 crc :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:03.156 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:03.157 7f1afc149700 1 -- 172.21.2.112:0/1205359163 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1af40ebc00 con 0x7f1af409a740 2021-02-10T11:42:03.158 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:03.160 7f1aea7fc700 1 -- 172.21.2.112:0/1205359163 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1ae4009000 con 0x7f1af409a740 2021-02-10T11:42:03.292 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:03.293 7f1afc149700 1 -- 172.21.2.112:0/1205359163 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_non_clone_status"]} v 0) v1 -- 0x7f1af40ebc00 con 0x7f1af409a740 2021-02-10T11:42:03.662 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:03.664 7f1aea7fc700 1 -- 172.21.2.112:0/1205359163 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_non_clone_status"]}]=0 v608) v1 ==== 141+0+0 (secure 0 0 0) 0x7f1ae401f500 con 0x7f1af409a740 2021-02-10T11:42:03.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:03.711 7f1afc149700 1 -- 172.21.2.112:0/1205359163 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1adc018730 msgr2=0x7f1adc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:03.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:03.711 7f1afc149700 1 --2- 172.21.2.112:0/1205359163 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1adc018730 0x7f1adc01abc0 crc :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:03.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:03.711 7f1afc149700 1 -- 172.21.2.112:0/1205359163 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1af409a740 msgr2=0x7f1af4097bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:03.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:03.711 7f1afc149700 1 --2- 172.21.2.112:0/1205359163 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1af409a740 0x7f1af4097bc0 secure :-1 s=READY pgs=2275 cs=0 l=1 rev1=1 rx=0x7f1ae400b5e0 tx=0x7f1ae400b980).stop 2021-02-10T11:42:03.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:03.711 7f1afc149700 1 -- 172.21.2.112:0/1205359163 shutdown_connections 2021-02-10T11:42:03.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:03.711 7f1afc149700 1 --2- 172.21.2.112:0/1205359163 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1adc018730 0x7f1adc01abc0 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:03.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:03.711 7f1afc149700 1 --2- 172.21.2.112:0/1205359163 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1af4099d20 0x7f1af409a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:03.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:03.711 7f1afc149700 1 --2- 172.21.2.112:0/1205359163 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1af409a740 0x7f1af4097bc0 unknown :-1 s=CLOSED pgs=2275 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:03.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:03.711 7f1afc149700 1 --2- 172.21.2.112:0/1205359163 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1af40ada90 0x7f1af40ade90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:03.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:03.711 7f1afc149700 1 -- 172.21.2.112:0/1205359163 >> 172.21.2.112:0/1205359163 conn(0x7f1af40f0550 msgr2=0x7f1af4191730 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:03.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:03.712 7f1afc149700 1 -- 172.21.2.112:0/1205359163 shutdown_connections 2021-02-10T11:42:03.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:03.712 7f1afc149700 1 -- 172.21.2.112:0/1205359163 wait complete. 2021-02-10T11:42:03.718 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:42:04.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:04.135 7f5569df9700 1 Processor -- start 2021-02-10T11:42:04.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:04.135 7f5569df9700 1 -- start start 2021-02-10T11:42:04.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:04.136 7f5569df9700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f55640d1870 0x7f55641b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:04.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:04.136 7f5569df9700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f55640cff70 0x7f5564109130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:04.152 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:04.136 7f5569df9700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5564109a60 con 0x7f55641b0ae0 2021-02-10T11:42:04.152 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:04.136 7f5569df9700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5564109750 con 0x7f55640d1870 2021-02-10T11:42:04.152 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:04.136 7f5569df9700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5564088070 con 0x7f55640cff70 2021-02-10T11:42:04.152 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:04.136 7f5562ffd700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f55641b0ae0 0x7f5564108940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:55354/0 (socket says 172.21.2.112:55354) 2021-02-10T11:42:04.152 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:04.136 7f5562ffd700 1 -- 172.21.2.112:0/1294445532 learned_addr learned my addr 172.21.2.112:0/1294445532 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:42:04.152 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:04.137 7f55627fc700 1 -- 172.21.2.112:0/1294445532 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1922198788 0 0) 0x7f55540010b0 con 0x7f55641b0ae0 2021-02-10T11:42:04.152 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:04.137 7f5563fff700 1 --2- 172.21.2.112:0/1294445532 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f55640cff70 0x7f5564109130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:04.152 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:04.137 7f55637fe700 1 --2- 172.21.2.112:0/1294445532 >> v2:172.21.2.120:3300/0 conn(0x7f55640d1870 0x7f55641b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:04.152 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:04.137 7f55627fc700 1 -- 172.21.2.112:0/1294445532 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5550002fe0 con 0x7f55641b0ae0 2021-02-10T11:42:04.152 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:04.137 7f55627fc700 1 -- 172.21.2.112:0/1294445532 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1826508183 0 0) 0x7f55540010b0 con 0x7f55641b0ae0 2021-02-10T11:42:04.152 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:04.137 7f55637fe700 1 -- 172.21.2.112:0/1294445532 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f55640cff70 msgr2=0x7f5564109130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:04.152 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:04.137 7f55637fe700 1 --2- 172.21.2.112:0/1294445532 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f55640cff70 0x7f5564109130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:04.152 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:04.137 7f55637fe700 1 -- 172.21.2.112:0/1294445532 >> v1:172.21.2.112:6789/0 conn(0x7f55641b0ae0 legacy=0x7f5564108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:04.153 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:04.137 7f55637fe700 1 -- 172.21.2.112:0/1294445532 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55640d1c70 con 0x7f55640d1870 2021-02-10T11:42:04.153 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:04.137 7f55637fe700 1 --2- 172.21.2.112:0/1294445532 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f55640d1870 0x7f55641b04c0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 rx=0x7f554c006650 tx=0x7f554c009a80).ready entity=mon.1 client_cookie=e444b37ea3b95a59 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:04.153 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:04.138 7f55627fc700 1 -- 172.21.2.112:0/1294445532 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f554c0009b0 con 0x7f55640d1870 2021-02-10T11:42:04.153 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:04.138 7f55627fc700 1 -- 172.21.2.112:0/1294445532 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f554c00c0c0 con 0x7f55640d1870 2021-02-10T11:42:04.153 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:04.138 7f55627fc700 1 -- 172.21.2.112:0/1294445532 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f554c00c5e0 con 0x7f55640d1870 2021-02-10T11:42:04.153 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:04.138 7f5569df9700 1 -- 172.21.2.112:0/1294445532 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f55640d1870 msgr2=0x7f55641b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:04.153 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:04.138 7f5569df9700 1 --2- 172.21.2.112:0/1294445532 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f55640d1870 0x7f55641b04c0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 rx=0x7f554c006650 tx=0x7f554c009a80).stop 2021-02-10T11:42:04.153 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:04.138 7f5569df9700 1 -- 172.21.2.112:0/1294445532 shutdown_connections 2021-02-10T11:42:04.153 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:04.138 7f5569df9700 1 --2- 172.21.2.112:0/1294445532 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f55640cff70 0x7f5564109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:04.153 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:04.138 7f5569df9700 1 --2- 172.21.2.112:0/1294445532 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f55640d1870 0x7f55641b04c0 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:04.153 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:04.138 7f5569df9700 1 -- 172.21.2.112:0/1294445532 >> 172.21.2.112:0/1294445532 conn(0x7f55640ce420 msgr2=0x7f55640d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:04.153 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:04.139 7f5569df9700 1 -- 172.21.2.112:0/1294445532 shutdown_connections 2021-02-10T11:42:04.153 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:04.139 7f5569df9700 1 -- 172.21.2.112:0/1294445532 wait complete. 2021-02-10T11:42:04.153 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:04.153 7f5569df9700 1 Processor -- start 2021-02-10T11:42:04.155 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:04.153 7f5569df9700 1 -- start start 2021-02-10T11:42:04.155 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:04.154 7f5569df9700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5564085bc0 0x7f55640863e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:04.156 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:04.154 7f5569df9700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f55640954c0 0x7f5564096300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:04.156 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:04.154 7f5569df9700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f55640958c0 0x7f5564096920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:04.156 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:04.154 7f5569df9700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5564095d80 con 0x7f5564085bc0 2021-02-10T11:42:04.156 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:04.154 7f5569df9700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55640bfa60 con 0x7f55640954c0 2021-02-10T11:42:04.156 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:04.154 7f5569df9700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5564085fc0 con 0x7f55640958c0 2021-02-10T11:42:04.156 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:04.154 7f55637fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5564085bc0 0x7f55640863e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:04.156 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:04.154 7f55637fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5564085bc0 0x7f55640863e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:55894/0 (socket says 172.21.2.112:55894) 2021-02-10T11:42:04.156 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:04.154 7f55637fe700 1 -- 172.21.2.112:0/3730499632 learned_addr learned my addr 172.21.2.112:0/3730499632 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:42:04.156 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:04.154 7f5562ffd700 1 --2- 172.21.2.112:0/3730499632 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f55640954c0 0x7f5564096300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:04.156 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:04.154 7f55637fe700 1 -- 172.21.2.112:0/3730499632 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f55640958c0 msgr2=0x7f5564096920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:04.156 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:04.154 7f5563fff700 1 --2- 172.21.2.112:0/3730499632 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f55640958c0 0x7f5564096920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:04.156 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:04.154 7f55637fe700 1 --2- 172.21.2.112:0/3730499632 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f55640958c0 0x7f5564096920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:04.156 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:04.154 7f55637fe700 1 -- 172.21.2.112:0/3730499632 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f55640954c0 msgr2=0x7f5564096300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:04.157 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:04.154 7f55637fe700 1 --2- 172.21.2.112:0/3730499632 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f55640954c0 0x7f5564096300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:04.157 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:04.155 7f55637fe700 1 -- 172.21.2.112:0/3730499632 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5564086c10 con 0x7f5564085bc0 2021-02-10T11:42:04.157 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:04.155 7f5562ffd700 1 --2- 172.21.2.112:0/3730499632 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f55640954c0 0x7f5564096300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:42:04.158 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:04.155 7f5563fff700 1 --2- 172.21.2.112:0/3730499632 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f55640958c0 0x7f5564096920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:42:04.158 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:04.155 7f55637fe700 1 --2- 172.21.2.112:0/3730499632 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5564085bc0 0x7f55640863e0 secure :-1 s=READY pgs=2277 cs=0 l=1 rev1=1 rx=0x7f554c009630 tx=0x7f554c00b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:04.158 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:04.155 7f555bfff700 1 -- 172.21.2.112:0/3730499632 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f554c0093a0 con 0x7f5564085bc0 2021-02-10T11:42:04.158 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:04.155 7f555bfff700 1 -- 172.21.2.112:0/3730499632 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f554c00bd70 con 0x7f5564085bc0 2021-02-10T11:42:04.158 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:04.155 7f555bfff700 1 -- 172.21.2.112:0/3730499632 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f554c0054d0 con 0x7f5564085bc0 2021-02-10T11:42:04.158 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:04.155 7f5569df9700 1 -- 172.21.2.112:0/3730499632 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5564086c10 con 0x7f5564085bc0 2021-02-10T11:42:04.158 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:04.155 7f5569df9700 1 -- 172.21.2.112:0/3730499632 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5564097230 con 0x7f5564085bc0 2021-02-10T11:42:04.158 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:04.156 7f555bfff700 1 -- 172.21.2.112:0/3730499632 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f554c006020 con 0x7f5564085bc0 2021-02-10T11:42:04.159 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:04.156 7f555bfff700 1 --2- 172.21.2.112:0/3730499632 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f55480186c0 0x7f554801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:04.159 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:04.156 7f555bfff700 1 -- 172.21.2.112:0/3730499632 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f554c032610 con 0x7f5564085bc0 2021-02-10T11:42:04.159 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:04.156 7f5562ffd700 1 --2- 172.21.2.112:0/3730499632 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f55480186c0 0x7f554801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:04.159 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:04.156 7f5562ffd700 1 --2- 172.21.2.112:0/3730499632 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f55480186c0 0x7f554801abc0 crc :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:04.184 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:04.186 7f5569df9700 1 -- 172.21.2.112:0/3730499632 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55640fe1c0 con 0x7f5564085bc0 2021-02-10T11:42:04.186 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:04.188 7f555bfff700 1 -- 172.21.2.112:0/3730499632 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f554c008030 con 0x7f5564085bc0 2021-02-10T11:42:04.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:04.340 7f5569df9700 1 -- 172.21.2.112:0/3730499632 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f55640fe1c0 con 0x7f5564085bc0 2021-02-10T11:42:04.342 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:04.343 7f555bfff700 1 -- 172.21.2.112:0/3730499632 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v32)=0 v32) v1 ==== 122+0+0 (secure 0 0 0) 0x7f554c006000 con 0x7f5564085bc0 2021-02-10T11:42:04.347 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:04.348 7f5569df9700 1 -- 172.21.2.112:0/3730499632 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f55480186c0 msgr2=0x7f554801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:04.347 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:04.348 7f5569df9700 1 --2- 172.21.2.112:0/3730499632 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f55480186c0 0x7f554801abc0 crc :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:04.347 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:04.348 7f5569df9700 1 -- 172.21.2.112:0/3730499632 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5564085bc0 msgr2=0x7f55640863e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:04.347 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:04.348 7f5569df9700 1 --2- 172.21.2.112:0/3730499632 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5564085bc0 0x7f55640863e0 secure :-1 s=READY pgs=2277 cs=0 l=1 rev1=1 rx=0x7f554c009630 tx=0x7f554c00b980).stop 2021-02-10T11:42:04.347 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:04.348 7f5569df9700 1 -- 172.21.2.112:0/3730499632 shutdown_connections 2021-02-10T11:42:04.347 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:04.348 7f5569df9700 1 --2- 172.21.2.112:0/3730499632 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f55480186c0 0x7f554801abc0 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:04.347 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:04.348 7f5569df9700 1 --2- 172.21.2.112:0/3730499632 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f55640954c0 0x7f5564096300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:04.347 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:04.349 7f5569df9700 1 --2- 172.21.2.112:0/3730499632 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f55640958c0 0x7f5564096920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:04.347 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:04.349 7f5569df9700 1 --2- 172.21.2.112:0/3730499632 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5564085bc0 0x7f55640863e0 unknown :-1 s=CLOSED pgs=2277 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:04.348 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:04.349 7f5569df9700 1 -- 172.21.2.112:0/3730499632 >> 172.21.2.112:0/3730499632 conn(0x7f55640bef00 msgr2=0x7f556409d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:04.348 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:04.349 7f5569df9700 1 -- 172.21.2.112:0/3730499632 shutdown_connections 2021-02-10T11:42:04.348 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:04.349 7f5569df9700 1 -- 172.21.2.112:0/3730499632 wait complete. 2021-02-10T11:42:04.355 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:42:04.355 DEBUG:teuthology.orchestra.run.gibba012:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:42:04.701 DEBUG:teuthology.orchestra.run.gibba020:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:42:05.047 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:42:05.048 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:42:05.048 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:42:05.048 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:42:05.384 INFO:teuthology.orchestra.run.gibba012.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.0’: No such file or directory 2021-02-10T11:42:05.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:42:05.385 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2021-02-10T11:42:05.385 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:42:05.385 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:42:05.385 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:42:05.721 INFO:teuthology.orchestra.run.gibba012.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: No such file or directory 2021-02-10T11:42:05.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:42:05.721 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2021-02-10T11:42:05.722 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:42:05.722 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:42:05.722 DEBUG:teuthology.orchestra.run.gibba020:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:42:06.057 INFO:teuthology.orchestra.run.gibba020.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:42:06.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:42:06.058 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:42:06.058 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:42:06.058 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:42:06.058 DEBUG:teuthology.orchestra.run.gibba020:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:42:06.394 INFO:teuthology.orchestra.run.gibba020.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:42:06.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:42:06.394 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:42:06.395 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json-pretty 2021-02-10T11:42:06.823 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:06.808 7fd8f8541700 1 Processor -- start 2021-02-10T11:42:06.823 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:06.808 7fd8f8541700 1 -- start start 2021-02-10T11:42:06.823 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:06.809 7fd8f8541700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fd8f00d1870 0x7fd8f01b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:06.823 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:06.809 7fd8f8541700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8f00f0b10 con 0x7fd8f01b0a10 2021-02-10T11:42:06.824 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:06.809 7fd8f8541700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8f00d0310 con 0x7fd8f00f0650 2021-02-10T11:42:06.824 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:06.809 7fd8f8541700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8f00d0530 con 0x7fd8f00d1870 2021-02-10T11:42:06.824 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:06.809 7fd8f635d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fd8f00d1870 0x7fd8f01b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:06.824 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:06.809 7fd8f635d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fd8f00d1870 0x7fd8f01b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:55902/0 (socket says 172.21.2.112:55902) 2021-02-10T11:42:06.824 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:06.809 7fd8f635d700 1 -- 172.21.2.112:0/4191718147 learned_addr learned my addr 172.21.2.112:0/4191718147 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:42:06.824 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:06.809 7fd8f635d700 1 -- 172.21.2.112:0/4191718147 >> v1:172.21.2.120:6789/0 conn(0x7fd8f00f0650 legacy=0x7fd8f00cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:06.824 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:06.809 7fd8f635d700 1 -- 172.21.2.112:0/4191718147 >> v1:172.21.2.112:6789/0 conn(0x7fd8f01b0a10 legacy=0x7fd8f00cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:06.824 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:06.809 7fd8f635d700 1 -- 172.21.2.112:0/4191718147 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8f00d1c70 con 0x7fd8f00d1870 2021-02-10T11:42:06.824 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:06.809 7fd8f635d700 1 --2- 172.21.2.112:0/4191718147 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd8f00d1870 0x7fd8f01b03f0 secure :-1 s=READY pgs=2279 cs=0 l=1 rev1=1 rx=0x7fd8ec006700 tx=0x7fd8ec009b00).ready entity=mon.0 client_cookie=1036fc3652f51d8a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:06.824 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:06.810 7fd8f535b700 1 -- 172.21.2.112:0/4191718147 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd8ec00c0c0 con 0x7fd8f00d1870 2021-02-10T11:42:06.824 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:06.810 7fd8f535b700 1 -- 172.21.2.112:0/4191718147 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8ec00c400 con 0x7fd8f00d1870 2021-02-10T11:42:06.824 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:06.810 7fd8f535b700 1 -- 172.21.2.112:0/4191718147 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd8ec00c920 con 0x7fd8f00d1870 2021-02-10T11:42:06.824 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:06.810 7fd8f8541700 1 -- 172.21.2.112:0/4191718147 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd8f00d1870 msgr2=0x7fd8f01b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:06.824 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:06.810 7fd8f8541700 1 --2- 172.21.2.112:0/4191718147 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd8f00d1870 0x7fd8f01b03f0 secure :-1 s=READY pgs=2279 cs=0 l=1 rev1=1 rx=0x7fd8ec006700 tx=0x7fd8ec009b00).stop 2021-02-10T11:42:06.825 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:06.810 7fd8f8541700 1 -- 172.21.2.112:0/4191718147 shutdown_connections 2021-02-10T11:42:06.825 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:06.810 7fd8f8541700 1 --2- 172.21.2.112:0/4191718147 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd8f00d1870 0x7fd8f01b03f0 unknown :-1 s=CLOSED pgs=2279 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:06.825 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:06.810 7fd8f8541700 1 -- 172.21.2.112:0/4191718147 >> 172.21.2.112:0/4191718147 conn(0x7fd8f00ce450 msgr2=0x7fd8f00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:06.825 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:06.811 7fd8f8541700 1 -- 172.21.2.112:0/4191718147 shutdown_connections 2021-02-10T11:42:06.825 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:06.811 7fd8f8541700 1 -- 172.21.2.112:0/4191718147 wait complete. 2021-02-10T11:42:06.827 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:06.825 7fd8f8541700 1 Processor -- start 2021-02-10T11:42:06.827 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:06.825 7fd8f8541700 1 -- start start 2021-02-10T11:42:06.827 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:06.826 7fd8f8541700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd8f009bca0 0x7fd8f009c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:06.827 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:06.826 7fd8f8541700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd8f009c6c0 0x7fd8f009cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:06.827 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:06.826 7fd8f8541700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd8f008c1a0 0x7fd8f008c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:06.827 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:06.826 7fd8f8541700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8f009b720 con 0x7fd8f009c6c0 2021-02-10T11:42:06.827 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:06.826 7fd8f8541700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8f00abcd0 con 0x7fd8f008c1a0 2021-02-10T11:42:06.827 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:06.826 7fd8f8541700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8f008cbc0 con 0x7fd8f009bca0 2021-02-10T11:42:06.827 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:06.826 7fd8f5b5c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd8f009c6c0 0x7fd8f009cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:06.827 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:06.826 7fd8f5b5c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd8f009c6c0 0x7fd8f009cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:55904/0 (socket says 172.21.2.112:55904) 2021-02-10T11:42:06.827 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:06.826 7fd8f5b5c700 1 -- 172.21.2.112:0/1846781544 learned_addr learned my addr 172.21.2.112:0/1846781544 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:42:06.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:06.826 7fd8f6b5e700 1 --2- 172.21.2.112:0/1846781544 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd8f008c1a0 0x7fd8f008c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:06.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:06.826 7fd8f5b5c700 1 -- 172.21.2.112:0/1846781544 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd8f009bca0 msgr2=0x7fd8f009c0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:06.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:06.826 7fd8f635d700 1 --2- 172.21.2.112:0/1846781544 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd8f009bca0 0x7fd8f009c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:06.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:06.826 7fd8f5b5c700 1 --2- 172.21.2.112:0/1846781544 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd8f009bca0 0x7fd8f009c0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:06.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:06.826 7fd8f5b5c700 1 -- 172.21.2.112:0/1846781544 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd8f008c1a0 msgr2=0x7fd8f008c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:06.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:06.826 7fd8f5b5c700 1 --2- 172.21.2.112:0/1846781544 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd8f008c1a0 0x7fd8f008c5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:06.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:06.826 7fd8f5b5c700 1 -- 172.21.2.112:0/1846781544 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8f008cde0 con 0x7fd8f009c6c0 2021-02-10T11:42:06.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:06.826 7fd8f635d700 1 --2- 172.21.2.112:0/1846781544 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd8f009bca0 0x7fd8f009c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:42:06.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:06.826 7fd8f5b5c700 1 --2- 172.21.2.112:0/1846781544 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd8f009c6c0 0x7fd8f009cac0 secure :-1 s=READY pgs=2280 cs=0 l=1 rev1=1 rx=0x7fd8e4006f10 tx=0x7fd8e400b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:06.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:06.827 7fd8e2ffd700 1 -- 172.21.2.112:0/1846781544 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd8e40009b0 con 0x7fd8f009c6c0 2021-02-10T11:42:06.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:06.827 7fd8e2ffd700 1 -- 172.21.2.112:0/1846781544 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8e400bd80 con 0x7fd8f009c6c0 2021-02-10T11:42:06.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:06.827 7fd8f8541700 1 -- 172.21.2.112:0/1846781544 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8f008cde0 con 0x7fd8f009c6c0 2021-02-10T11:42:06.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:06.827 7fd8e2ffd700 1 -- 172.21.2.112:0/1846781544 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd8e4001600 con 0x7fd8f009c6c0 2021-02-10T11:42:06.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:06.827 7fd8f8541700 1 -- 172.21.2.112:0/1846781544 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8f008cde0 con 0x7fd8f009c6c0 2021-02-10T11:42:06.829 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:06.827 7fd8e2ffd700 1 -- 172.21.2.112:0/1846781544 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd8e40056c0 con 0x7fd8f009c6c0 2021-02-10T11:42:06.829 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:06.827 7fd8e2ffd700 1 --2- 172.21.2.112:0/1846781544 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd8d8018730 0x7fd8d801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:06.829 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:06.827 7fd8e2ffd700 1 -- 172.21.2.112:0/1846781544 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fd8e4032600 con 0x7fd8f009c6c0 2021-02-10T11:42:06.829 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:06.828 7fd8f635d700 1 --2- 172.21.2.112:0/1846781544 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd8d8018730 0x7fd8d801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:06.830 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:06.828 7fd8f635d700 1 --2- 172.21.2.112:0/1846781544 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd8d8018730 0x7fd8d801abc0 crc :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:06.856 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:06.858 7fd8f8541700 1 -- 172.21.2.112:0/1846781544 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8f00d1ee0 con 0x7fd8f009c6c0 2021-02-10T11:42:06.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:06.861 7fd8e2ffd700 1 -- 172.21.2.112:0/1846781544 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd8e4001200 con 0x7fd8f009c6c0 2021-02-10T11:42:06.984 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:06.985 7fd8f8541700 1 -- 172.21.2.112:0/1846781544 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7fd8f008d660 con 0x7fd8f009c6c0 2021-02-10T11:42:06.984 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:06.986 7fd8e2ffd700 1 -- 172.21.2.112:0/1846781544 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v117) v1 ==== 81+0+30374 (secure 0 0 0) 0x7fd8e40056c0 con 0x7fd8f009c6c0 2021-02-10T11:42:06.985 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:42:06.985 INFO:teuthology.orchestra.run.gibba012.stdout:{ 2021-02-10T11:42:06.985 INFO:teuthology.orchestra.run.gibba012.stdout: "epoch": 117, 2021-02-10T11:42:06.985 INFO:teuthology.orchestra.run.gibba012.stdout: "fsid": "f14f004e-0699-4584-ad81-3754258493de", 2021-02-10T11:42:06.986 INFO:teuthology.orchestra.run.gibba012.stdout: "created": "2021-02-10 11:13:26.501424", 2021-02-10T11:42:06.986 INFO:teuthology.orchestra.run.gibba012.stdout: "modified": "2021-02-10 11:41:24.733802", 2021-02-10T11:42:06.986 INFO:teuthology.orchestra.run.gibba012.stdout: "last_up_change": "2021-02-10 11:13:52.780316", 2021-02-10T11:42:06.986 INFO:teuthology.orchestra.run.gibba012.stdout: "last_in_change": "2021-02-10 11:13:52.780316", 2021-02-10T11:42:06.986 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-02-10T11:42:06.986 INFO:teuthology.orchestra.run.gibba012.stdout: "flags_num": 5799936, 2021-02-10T11:42:06.986 INFO:teuthology.orchestra.run.gibba012.stdout: "flags_set": [ 2021-02-10T11:42:06.986 INFO:teuthology.orchestra.run.gibba012.stdout: "pglog_hardlimit", 2021-02-10T11:42:06.986 INFO:teuthology.orchestra.run.gibba012.stdout: "purged_snapdirs", 2021-02-10T11:42:06.986 INFO:teuthology.orchestra.run.gibba012.stdout: "recovery_deletes", 2021-02-10T11:42:06.986 INFO:teuthology.orchestra.run.gibba012.stdout: "sortbitwise" 2021-02-10T11:42:06.986 INFO:teuthology.orchestra.run.gibba012.stdout: ], 2021-02-10T11:42:06.987 INFO:teuthology.orchestra.run.gibba012.stdout: "crush_version": 4, 2021-02-10T11:42:06.987 INFO:teuthology.orchestra.run.gibba012.stdout: "full_ratio": 0.94999998807907104, 2021-02-10T11:42:06.987 INFO:teuthology.orchestra.run.gibba012.stdout: "backfillfull_ratio": 0.89999997615814209, 2021-02-10T11:42:06.987 INFO:teuthology.orchestra.run.gibba012.stdout: "nearfull_ratio": 0.85000002384185791, 2021-02-10T11:42:06.987 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_snapshot": "", 2021-02-10T11:42:06.987 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_max": 25, 2021-02-10T11:42:06.987 INFO:teuthology.orchestra.run.gibba012.stdout: "max_osd": 8, 2021-02-10T11:42:06.987 INFO:teuthology.orchestra.run.gibba012.stdout: "require_min_compat_client": "jewel", 2021-02-10T11:42:06.987 INFO:teuthology.orchestra.run.gibba012.stdout: "min_compat_client": "jewel", 2021-02-10T11:42:06.987 INFO:teuthology.orchestra.run.gibba012.stdout: "require_osd_release": "nautilus", 2021-02-10T11:42:06.987 INFO:teuthology.orchestra.run.gibba012.stdout: "pools": [ 2021-02-10T11:42:06.987 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:06.988 INFO:teuthology.orchestra.run.gibba012.stdout: "pool": 1, 2021-02-10T11:42:06.988 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_name": "rbd", 2021-02-10T11:42:06.988 INFO:teuthology.orchestra.run.gibba012.stdout: "create_time": "2021-02-10 11:13:57.508758", 2021-02-10T11:42:06.988 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:42:06.988 INFO:teuthology.orchestra.run.gibba012.stdout: "flags_names": "hashpspool", 2021-02-10T11:42:06.988 INFO:teuthology.orchestra.run.gibba012.stdout: "type": 1, 2021-02-10T11:42:06.988 INFO:teuthology.orchestra.run.gibba012.stdout: "size": 2, 2021-02-10T11:42:06.988 INFO:teuthology.orchestra.run.gibba012.stdout: "min_size": 1, 2021-02-10T11:42:06.988 INFO:teuthology.orchestra.run.gibba012.stdout: "crush_rule": 0, 2021-02-10T11:42:06.988 INFO:teuthology.orchestra.run.gibba012.stdout: "object_hash": 2, 2021-02-10T11:42:06.988 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:42:06.988 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num": 8, 2021-02-10T11:42:06.988 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num": 8, 2021-02-10T11:42:06.988 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num_target": 8, 2021-02-10T11:42:06.989 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_target": 8, 2021-02-10T11:42:06.989 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_pending": 8, 2021-02-10T11:42:06.989 INFO:teuthology.orchestra.run.gibba012.stdout: "last_pg_merge_meta": { 2021-02-10T11:42:06.989 INFO:teuthology.orchestra.run.gibba012.stdout: "source_pgid": "0.0", 2021-02-10T11:42:06.989 INFO:teuthology.orchestra.run.gibba012.stdout: "ready_epoch": 0, 2021-02-10T11:42:06.989 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_started": 0, 2021-02-10T11:42:06.989 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_clean": 0, 2021-02-10T11:42:06.989 INFO:teuthology.orchestra.run.gibba012.stdout: "source_version": "0'0", 2021-02-10T11:42:06.989 INFO:teuthology.orchestra.run.gibba012.stdout: "target_version": "0'0" 2021-02-10T11:42:06.989 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:06.989 INFO:teuthology.orchestra.run.gibba012.stdout: "last_change": "16", 2021-02-10T11:42:06.989 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend": "0", 2021-02-10T11:42:06.989 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:42:06.990 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:42:06.990 INFO:teuthology.orchestra.run.gibba012.stdout: "auid": 0, 2021-02-10T11:42:06.990 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:42:06.990 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_seq": 0, 2021-02-10T11:42:06.990 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_epoch": 0, 2021-02-10T11:42:06.990 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_snaps": [], 2021-02-10T11:42:06.990 INFO:teuthology.orchestra.run.gibba012.stdout: "removed_snaps": "[]", 2021-02-10T11:42:06.990 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_bytes": 0, 2021-02-10T11:42:06.990 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_objects": 0, 2021-02-10T11:42:06.990 INFO:teuthology.orchestra.run.gibba012.stdout: "tiers": [], 2021-02-10T11:42:06.990 INFO:teuthology.orchestra.run.gibba012.stdout: "tier_of": -1, 2021-02-10T11:42:06.990 INFO:teuthology.orchestra.run.gibba012.stdout: "read_tier": -1, 2021-02-10T11:42:06.991 INFO:teuthology.orchestra.run.gibba012.stdout: "write_tier": -1, 2021-02-10T11:42:06.991 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_mode": "none", 2021-02-10T11:42:06.991 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_bytes": 0, 2021-02-10T11:42:06.991 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_objects": 0, 2021-02-10T11:42:06.991 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:42:06.991 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:42:06.991 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:42:06.991 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_flush_age": 0, 2021-02-10T11:42:06.991 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_evict_age": 0, 2021-02-10T11:42:06.991 INFO:teuthology.orchestra.run.gibba012.stdout: "erasure_code_profile": "", 2021-02-10T11:42:06.991 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_params": { 2021-02-10T11:42:06.991 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "none" 2021-02-10T11:42:06.991 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:06.992 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_period": 0, 2021-02-10T11:42:06.992 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_count": 0, 2021-02-10T11:42:06.992 INFO:teuthology.orchestra.run.gibba012.stdout: "use_gmt_hitset": true, 2021-02-10T11:42:06.992 INFO:teuthology.orchestra.run.gibba012.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:42:06.992 INFO:teuthology.orchestra.run.gibba012.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:42:06.992 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:42:06.992 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:42:06.992 INFO:teuthology.orchestra.run.gibba012.stdout: "grade_table": [], 2021-02-10T11:42:06.992 INFO:teuthology.orchestra.run.gibba012.stdout: "stripe_width": 0, 2021-02-10T11:42:06.992 INFO:teuthology.orchestra.run.gibba012.stdout: "expected_num_objects": 0, 2021-02-10T11:42:06.992 INFO:teuthology.orchestra.run.gibba012.stdout: "fast_read": false, 2021-02-10T11:42:06.992 INFO:teuthology.orchestra.run.gibba012.stdout: "options": {}, 2021-02-10T11:42:06.992 INFO:teuthology.orchestra.run.gibba012.stdout: "application_metadata": { 2021-02-10T11:42:06.992 INFO:teuthology.orchestra.run.gibba012.stdout: "rbd": {} 2021-02-10T11:42:06.993 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:42:06.993 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:06.993 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:06.993 INFO:teuthology.orchestra.run.gibba012.stdout: "pool": 24, 2021-02-10T11:42:06.993 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_name": "cephfs_metadata", 2021-02-10T11:42:06.993 INFO:teuthology.orchestra.run.gibba012.stdout: "create_time": "2021-02-10 11:41:21.697492", 2021-02-10T11:42:06.993 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:42:06.993 INFO:teuthology.orchestra.run.gibba012.stdout: "flags_names": "hashpspool", 2021-02-10T11:42:06.993 INFO:teuthology.orchestra.run.gibba012.stdout: "type": 1, 2021-02-10T11:42:06.993 INFO:teuthology.orchestra.run.gibba012.stdout: "size": 2, 2021-02-10T11:42:06.993 INFO:teuthology.orchestra.run.gibba012.stdout: "min_size": 1, 2021-02-10T11:42:06.993 INFO:teuthology.orchestra.run.gibba012.stdout: "crush_rule": 0, 2021-02-10T11:42:06.993 INFO:teuthology.orchestra.run.gibba012.stdout: "object_hash": 2, 2021-02-10T11:42:06.994 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:42:06.994 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num": 8, 2021-02-10T11:42:06.994 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num": 8, 2021-02-10T11:42:06.994 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num_target": 8, 2021-02-10T11:42:06.994 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_target": 8, 2021-02-10T11:42:06.994 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_pending": 8, 2021-02-10T11:42:06.994 INFO:teuthology.orchestra.run.gibba012.stdout: "last_pg_merge_meta": { 2021-02-10T11:42:06.994 INFO:teuthology.orchestra.run.gibba012.stdout: "source_pgid": "0.0", 2021-02-10T11:42:06.994 INFO:teuthology.orchestra.run.gibba012.stdout: "ready_epoch": 0, 2021-02-10T11:42:06.994 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_started": 0, 2021-02-10T11:42:06.994 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_clean": 0, 2021-02-10T11:42:06.994 INFO:teuthology.orchestra.run.gibba012.stdout: "source_version": "0'0", 2021-02-10T11:42:06.995 INFO:teuthology.orchestra.run.gibba012.stdout: "target_version": "0'0" 2021-02-10T11:42:06.995 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:06.995 INFO:teuthology.orchestra.run.gibba012.stdout: "last_change": "116", 2021-02-10T11:42:06.995 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend": "0", 2021-02-10T11:42:06.995 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:42:06.995 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:42:06.995 INFO:teuthology.orchestra.run.gibba012.stdout: "auid": 0, 2021-02-10T11:42:06.995 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:42:06.995 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_seq": 0, 2021-02-10T11:42:06.995 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_epoch": 0, 2021-02-10T11:42:06.995 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_snaps": [], 2021-02-10T11:42:06.995 INFO:teuthology.orchestra.run.gibba012.stdout: "removed_snaps": "[]", 2021-02-10T11:42:06.995 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_bytes": 0, 2021-02-10T11:42:06.996 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_objects": 0, 2021-02-10T11:42:06.996 INFO:teuthology.orchestra.run.gibba012.stdout: "tiers": [], 2021-02-10T11:42:06.996 INFO:teuthology.orchestra.run.gibba012.stdout: "tier_of": -1, 2021-02-10T11:42:06.996 INFO:teuthology.orchestra.run.gibba012.stdout: "read_tier": -1, 2021-02-10T11:42:06.996 INFO:teuthology.orchestra.run.gibba012.stdout: "write_tier": -1, 2021-02-10T11:42:06.996 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_mode": "none", 2021-02-10T11:42:06.996 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_bytes": 0, 2021-02-10T11:42:06.996 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_objects": 0, 2021-02-10T11:42:06.996 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:42:06.996 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:42:06.996 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:42:06.996 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_flush_age": 0, 2021-02-10T11:42:06.997 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_evict_age": 0, 2021-02-10T11:42:06.997 INFO:teuthology.orchestra.run.gibba012.stdout: "erasure_code_profile": "", 2021-02-10T11:42:06.997 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_params": { 2021-02-10T11:42:06.997 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "none" 2021-02-10T11:42:06.997 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:06.997 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_period": 0, 2021-02-10T11:42:06.997 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_count": 0, 2021-02-10T11:42:06.997 INFO:teuthology.orchestra.run.gibba012.stdout: "use_gmt_hitset": true, 2021-02-10T11:42:06.998 INFO:teuthology.orchestra.run.gibba012.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:42:06.998 INFO:teuthology.orchestra.run.gibba012.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:42:06.998 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:42:06.998 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:42:06.998 INFO:teuthology.orchestra.run.gibba012.stdout: "grade_table": [], 2021-02-10T11:42:06.998 INFO:teuthology.orchestra.run.gibba012.stdout: "stripe_width": 0, 2021-02-10T11:42:06.998 INFO:teuthology.orchestra.run.gibba012.stdout: "expected_num_objects": 0, 2021-02-10T11:42:06.998 INFO:teuthology.orchestra.run.gibba012.stdout: "fast_read": false, 2021-02-10T11:42:06.998 INFO:teuthology.orchestra.run.gibba012.stdout: "options": { 2021-02-10T11:42:06.999 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_autoscale_bias": 4, 2021-02-10T11:42:06.999 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_min": 16, 2021-02-10T11:42:06.999 INFO:teuthology.orchestra.run.gibba012.stdout: "recovery_priority": 5 2021-02-10T11:42:06.999 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:06.999 INFO:teuthology.orchestra.run.gibba012.stdout: "application_metadata": { 2021-02-10T11:42:06.999 INFO:teuthology.orchestra.run.gibba012.stdout: "cephfs": { 2021-02-10T11:42:06.999 INFO:teuthology.orchestra.run.gibba012.stdout: "metadata": "cephfs" 2021-02-10T11:42:06.999 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:42:06.999 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:42:06.999 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:06.999 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:06.999 INFO:teuthology.orchestra.run.gibba012.stdout: "pool": 25, 2021-02-10T11:42:06.999 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_name": "cephfs_data", 2021-02-10T11:42:07.000 INFO:teuthology.orchestra.run.gibba012.stdout: "create_time": "2021-02-10 11:41:22.721757", 2021-02-10T11:42:07.000 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:42:07.000 INFO:teuthology.orchestra.run.gibba012.stdout: "flags_names": "hashpspool", 2021-02-10T11:42:07.000 INFO:teuthology.orchestra.run.gibba012.stdout: "type": 1, 2021-02-10T11:42:07.000 INFO:teuthology.orchestra.run.gibba012.stdout: "size": 2, 2021-02-10T11:42:07.000 INFO:teuthology.orchestra.run.gibba012.stdout: "min_size": 1, 2021-02-10T11:42:07.000 INFO:teuthology.orchestra.run.gibba012.stdout: "crush_rule": 0, 2021-02-10T11:42:07.000 INFO:teuthology.orchestra.run.gibba012.stdout: "object_hash": 2, 2021-02-10T11:42:07.000 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:42:07.000 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num": 8, 2021-02-10T11:42:07.000 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num": 8, 2021-02-10T11:42:07.000 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num_target": 8, 2021-02-10T11:42:07.000 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_target": 8, 2021-02-10T11:42:07.001 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_pending": 8, 2021-02-10T11:42:07.001 INFO:teuthology.orchestra.run.gibba012.stdout: "last_pg_merge_meta": { 2021-02-10T11:42:07.001 INFO:teuthology.orchestra.run.gibba012.stdout: "source_pgid": "0.0", 2021-02-10T11:42:07.001 INFO:teuthology.orchestra.run.gibba012.stdout: "ready_epoch": 0, 2021-02-10T11:42:07.001 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_started": 0, 2021-02-10T11:42:07.001 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_clean": 0, 2021-02-10T11:42:07.001 INFO:teuthology.orchestra.run.gibba012.stdout: "source_version": "0'0", 2021-02-10T11:42:07.001 INFO:teuthology.orchestra.run.gibba012.stdout: "target_version": "0'0" 2021-02-10T11:42:07.001 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:07.001 INFO:teuthology.orchestra.run.gibba012.stdout: "last_change": "117", 2021-02-10T11:42:07.001 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend": "0", 2021-02-10T11:42:07.001 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:42:07.001 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:42:07.002 INFO:teuthology.orchestra.run.gibba012.stdout: "auid": 0, 2021-02-10T11:42:07.002 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:42:07.002 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_seq": 0, 2021-02-10T11:42:07.002 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_epoch": 0, 2021-02-10T11:42:07.002 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_snaps": [], 2021-02-10T11:42:07.002 INFO:teuthology.orchestra.run.gibba012.stdout: "removed_snaps": "[]", 2021-02-10T11:42:07.002 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_bytes": 0, 2021-02-10T11:42:07.002 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_objects": 0, 2021-02-10T11:42:07.002 INFO:teuthology.orchestra.run.gibba012.stdout: "tiers": [], 2021-02-10T11:42:07.002 INFO:teuthology.orchestra.run.gibba012.stdout: "tier_of": -1, 2021-02-10T11:42:07.002 INFO:teuthology.orchestra.run.gibba012.stdout: "read_tier": -1, 2021-02-10T11:42:07.002 INFO:teuthology.orchestra.run.gibba012.stdout: "write_tier": -1, 2021-02-10T11:42:07.003 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_mode": "none", 2021-02-10T11:42:07.003 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_bytes": 0, 2021-02-10T11:42:07.003 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_objects": 0, 2021-02-10T11:42:07.003 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:42:07.003 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:42:07.003 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:42:07.003 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_flush_age": 0, 2021-02-10T11:42:07.003 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_evict_age": 0, 2021-02-10T11:42:07.003 INFO:teuthology.orchestra.run.gibba012.stdout: "erasure_code_profile": "", 2021-02-10T11:42:07.003 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_params": { 2021-02-10T11:42:07.003 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "none" 2021-02-10T11:42:07.003 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:07.004 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_period": 0, 2021-02-10T11:42:07.004 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_count": 0, 2021-02-10T11:42:07.004 INFO:teuthology.orchestra.run.gibba012.stdout: "use_gmt_hitset": true, 2021-02-10T11:42:07.004 INFO:teuthology.orchestra.run.gibba012.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:42:07.004 INFO:teuthology.orchestra.run.gibba012.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:42:07.004 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:42:07.004 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:42:07.004 INFO:teuthology.orchestra.run.gibba012.stdout: "grade_table": [], 2021-02-10T11:42:07.004 INFO:teuthology.orchestra.run.gibba012.stdout: "stripe_width": 0, 2021-02-10T11:42:07.004 INFO:teuthology.orchestra.run.gibba012.stdout: "expected_num_objects": 0, 2021-02-10T11:42:07.004 INFO:teuthology.orchestra.run.gibba012.stdout: "fast_read": false, 2021-02-10T11:42:07.004 INFO:teuthology.orchestra.run.gibba012.stdout: "options": {}, 2021-02-10T11:42:07.004 INFO:teuthology.orchestra.run.gibba012.stdout: "application_metadata": { 2021-02-10T11:42:07.005 INFO:teuthology.orchestra.run.gibba012.stdout: "cephfs": { 2021-02-10T11:42:07.005 INFO:teuthology.orchestra.run.gibba012.stdout: "data": "cephfs" 2021-02-10T11:42:07.005 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:42:07.005 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:42:07.005 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:42:07.005 INFO:teuthology.orchestra.run.gibba012.stdout: ], 2021-02-10T11:42:07.005 INFO:teuthology.orchestra.run.gibba012.stdout: "osds": [ 2021-02-10T11:42:07.005 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:07.005 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 0, 2021-02-10T11:42:07.005 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "95444b8f-6370-4a1d-ae5f-df62651b77da", 2021-02-10T11:42:07.005 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:42:07.005 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:42:07.006 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:42:07.006 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:42:07.006 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:42:07.006 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:42:07.006 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:42:07.006 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 115, 2021-02-10T11:42:07.006 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:42:07.006 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:42:07.006 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:42:07.006 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:42:07.006 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:07.006 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:42:07.006 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6817", 2021-02-10T11:42:07.007 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:42:07.007 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:07.007 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:07.007 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:42:07.007 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6818", 2021-02-10T11:42:07.007 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:42:07.007 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:42:07.007 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:42:07.007 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:07.007 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:42:07.007 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:42:07.007 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:07.007 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:42:07.008 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6820", 2021-02-10T11:42:07.008 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:42:07.008 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:07.008 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:07.008 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:42:07.008 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6822", 2021-02-10T11:42:07.008 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:42:07.008 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:42:07.008 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:42:07.008 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:07.008 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:42:07.008 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:42:07.008 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:07.009 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:42:07.009 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6828", 2021-02-10T11:42:07.009 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:42:07.009 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:07.009 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:07.009 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:42:07.009 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6830", 2021-02-10T11:42:07.009 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:42:07.009 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:42:07.009 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:42:07.009 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:07.009 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:42:07.009 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:42:07.010 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:07.010 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:42:07.010 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6824", 2021-02-10T11:42:07.010 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:42:07.010 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:07.010 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:07.010 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:42:07.010 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6826", 2021-02-10T11:42:07.010 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:42:07.010 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:42:07.010 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:42:07.010 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:07.011 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.112:6818/36030", 2021-02-10T11:42:07.011 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.112:6822/36030", 2021-02-10T11:42:07.011 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.112:6830/36030", 2021-02-10T11:42:07.011 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.112:6826/36030", 2021-02-10T11:42:07.011 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:42:07.011 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:42:07.011 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:42:07.011 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:42:07.011 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:07.011 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:07.011 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 1, 2021-02-10T11:42:07.011 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "a23e217e-0d8e-486e-be07-712adfabd262", 2021-02-10T11:42:07.011 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:42:07.011 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:42:07.012 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:42:07.012 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:42:07.012 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:42:07.012 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:42:07.012 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:42:07.012 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 113, 2021-02-10T11:42:07.012 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:42:07.012 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:42:07.012 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:42:07.012 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:42:07.012 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:07.012 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:42:07.012 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6800", 2021-02-10T11:42:07.013 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:42:07.013 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:07.013 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:07.013 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:42:07.013 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6802", 2021-02-10T11:42:07.013 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:42:07.013 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:42:07.013 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:42:07.013 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:07.013 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:42:07.013 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:42:07.013 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:07.013 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:42:07.014 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6804", 2021-02-10T11:42:07.014 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:42:07.014 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:07.014 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:07.014 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:42:07.014 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6806", 2021-02-10T11:42:07.014 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:42:07.014 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:42:07.014 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:42:07.014 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:07.014 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:42:07.014 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:42:07.015 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:07.015 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:42:07.015 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6812", 2021-02-10T11:42:07.015 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:42:07.015 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:07.015 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:07.015 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:42:07.015 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6814", 2021-02-10T11:42:07.015 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:42:07.015 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:42:07.015 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:42:07.015 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:07.015 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:42:07.016 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:42:07.016 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:07.016 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:42:07.016 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6808", 2021-02-10T11:42:07.016 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:42:07.016 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:07.016 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:07.016 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:42:07.016 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6810", 2021-02-10T11:42:07.016 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:42:07.016 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:42:07.016 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:42:07.016 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:07.017 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.112:6802/36029", 2021-02-10T11:42:07.017 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.112:6806/36029", 2021-02-10T11:42:07.017 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.112:6814/36029", 2021-02-10T11:42:07.017 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.112:6810/36029", 2021-02-10T11:42:07.017 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:42:07.017 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:42:07.017 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:42:07.017 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:42:07.017 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:07.017 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:07.017 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 2, 2021-02-10T11:42:07.017 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "026afa8a-c0f1-4b35-9a76-53c00fe0c2fc", 2021-02-10T11:42:07.017 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:42:07.017 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:42:07.018 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:42:07.018 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:42:07.018 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:42:07.018 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:42:07.018 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:42:07.018 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 115, 2021-02-10T11:42:07.018 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:42:07.018 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:42:07.018 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:42:07.018 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:42:07.018 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:07.018 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:42:07.019 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6801", 2021-02-10T11:42:07.019 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:42:07.019 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:07.019 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:07.019 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:42:07.019 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6803", 2021-02-10T11:42:07.019 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:42:07.019 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:42:07.019 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:42:07.019 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:07.019 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:42:07.019 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:42:07.019 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:07.020 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:42:07.020 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6805", 2021-02-10T11:42:07.020 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:42:07.020 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:07.020 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:07.020 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:42:07.020 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6807", 2021-02-10T11:42:07.020 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:42:07.020 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:42:07.020 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:42:07.020 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:07.020 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:42:07.020 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:42:07.020 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:07.021 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:42:07.021 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6813", 2021-02-10T11:42:07.021 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:42:07.021 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:07.021 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:07.021 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:42:07.021 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6815", 2021-02-10T11:42:07.021 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:42:07.021 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:42:07.021 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:42:07.021 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:07.021 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:42:07.021 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:42:07.022 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:07.022 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:42:07.022 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6809", 2021-02-10T11:42:07.022 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:42:07.022 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:07.022 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:07.022 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:42:07.022 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6811", 2021-02-10T11:42:07.022 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:42:07.022 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:42:07.022 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:42:07.022 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:07.023 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.112:6803/36031", 2021-02-10T11:42:07.023 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.112:6807/36031", 2021-02-10T11:42:07.023 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.112:6815/36031", 2021-02-10T11:42:07.023 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.112:6811/36031", 2021-02-10T11:42:07.023 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:42:07.023 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:42:07.023 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:42:07.023 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:42:07.023 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:07.023 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:07.023 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 3, 2021-02-10T11:42:07.023 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "58795682-e00d-4173-bb84-f7075ca89a11", 2021-02-10T11:42:07.023 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:42:07.024 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:42:07.024 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:42:07.024 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:42:07.024 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:42:07.024 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:42:07.024 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:42:07.024 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 115, 2021-02-10T11:42:07.024 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:42:07.024 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:42:07.024 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:42:07.024 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:42:07.024 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:07.024 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:42:07.025 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6816", 2021-02-10T11:42:07.025 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:42:07.025 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:07.025 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:07.025 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:42:07.025 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6819", 2021-02-10T11:42:07.025 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:42:07.025 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:42:07.025 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:42:07.025 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:07.025 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:42:07.025 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:42:07.025 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:07.026 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:42:07.026 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6821", 2021-02-10T11:42:07.026 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:42:07.026 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:07.026 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:07.026 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:42:07.026 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6823", 2021-02-10T11:42:07.026 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:42:07.026 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:42:07.026 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:42:07.026 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:07.026 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:42:07.027 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:42:07.027 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:07.027 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:42:07.027 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6829", 2021-02-10T11:42:07.027 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:42:07.027 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:07.027 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:07.027 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:42:07.027 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6831", 2021-02-10T11:42:07.027 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:42:07.027 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:42:07.027 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:42:07.027 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:07.028 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:42:07.028 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:42:07.028 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:07.028 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:42:07.028 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6825", 2021-02-10T11:42:07.028 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:42:07.028 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:07.028 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:07.028 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:42:07.028 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6827", 2021-02-10T11:42:07.028 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:42:07.028 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:42:07.028 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:42:07.029 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:07.029 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.112:6819/36028", 2021-02-10T11:42:07.029 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.112:6823/36028", 2021-02-10T11:42:07.029 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.112:6831/36028", 2021-02-10T11:42:07.029 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.112:6827/36028", 2021-02-10T11:42:07.029 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:42:07.029 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:42:07.029 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:42:07.029 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:42:07.029 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:07.029 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:07.029 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 4, 2021-02-10T11:42:07.029 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "6763f8d6-53dc-4e45-bdf6-a6eabcf81e39", 2021-02-10T11:42:07.029 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:42:07.030 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:42:07.030 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:42:07.030 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:42:07.030 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:42:07.030 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:42:07.030 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:42:07.030 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 106, 2021-02-10T11:42:07.030 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:42:07.030 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:42:07.030 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:42:07.030 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:42:07.030 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:07.031 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:42:07.031 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6810", 2021-02-10T11:42:07.031 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:42:07.031 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:07.031 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:07.031 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:42:07.031 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6813", 2021-02-10T11:42:07.031 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:42:07.031 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:42:07.031 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:42:07.031 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:07.031 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:42:07.031 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:42:07.032 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:07.032 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:42:07.032 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6816", 2021-02-10T11:42:07.032 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:42:07.032 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:07.032 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:07.032 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:42:07.032 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6819", 2021-02-10T11:42:07.032 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:42:07.032 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:42:07.032 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:42:07.032 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:07.032 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:42:07.032 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:42:07.033 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:07.033 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:42:07.033 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6825", 2021-02-10T11:42:07.033 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:42:07.033 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:07.033 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:07.033 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:42:07.033 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6827", 2021-02-10T11:42:07.033 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:42:07.033 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:42:07.033 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:42:07.033 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:07.033 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:42:07.034 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:42:07.034 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:07.034 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:42:07.034 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6821", 2021-02-10T11:42:07.034 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:42:07.034 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:07.034 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:07.034 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:42:07.034 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6823", 2021-02-10T11:42:07.034 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:42:07.034 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:42:07.034 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:42:07.035 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:07.035 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.120:6813/36388", 2021-02-10T11:42:07.035 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.120:6819/36388", 2021-02-10T11:42:07.035 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.120:6827/36388", 2021-02-10T11:42:07.035 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.120:6823/36388", 2021-02-10T11:42:07.035 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:42:07.035 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:42:07.035 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:42:07.035 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:42:07.035 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:07.035 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:07.035 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 5, 2021-02-10T11:42:07.035 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "5d0e4dd6-0cb0-4bf8-8314-626acaa1b544", 2021-02-10T11:42:07.035 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:42:07.036 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:42:07.036 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:42:07.036 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:42:07.036 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:42:07.036 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:42:07.036 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:42:07.036 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 113, 2021-02-10T11:42:07.036 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:42:07.036 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:42:07.036 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:42:07.036 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:42:07.036 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:07.036 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:42:07.037 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6804", 2021-02-10T11:42:07.037 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:42:07.037 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:07.037 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:07.037 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:42:07.037 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6806", 2021-02-10T11:42:07.037 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:42:07.037 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:42:07.037 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:42:07.037 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:07.037 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:42:07.037 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:42:07.037 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:07.038 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:42:07.038 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6809", 2021-02-10T11:42:07.038 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:42:07.038 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:07.038 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:07.038 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:42:07.038 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6812", 2021-02-10T11:42:07.038 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:42:07.038 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:42:07.038 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:42:07.038 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:07.038 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:42:07.039 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:42:07.039 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:07.039 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:42:07.039 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6820", 2021-02-10T11:42:07.039 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:42:07.039 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:07.039 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:07.039 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:42:07.039 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6822", 2021-02-10T11:42:07.039 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:42:07.039 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:42:07.039 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:42:07.039 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:07.039 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:42:07.040 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:42:07.040 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:07.040 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:42:07.040 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6815", 2021-02-10T11:42:07.040 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:42:07.040 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:07.040 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:07.040 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:42:07.040 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6818", 2021-02-10T11:42:07.040 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:42:07.040 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:42:07.040 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:42:07.040 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:07.041 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.120:6806/36387", 2021-02-10T11:42:07.041 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.120:6812/36387", 2021-02-10T11:42:07.041 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.120:6822/36387", 2021-02-10T11:42:07.041 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.120:6818/36387", 2021-02-10T11:42:07.041 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:42:07.041 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:42:07.041 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:42:07.041 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:42:07.041 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:07.041 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:07.041 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 6, 2021-02-10T11:42:07.041 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "b010e187-4541-4b00-80dd-f948f6ab8f15", 2021-02-10T11:42:07.041 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:42:07.042 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:42:07.042 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:42:07.042 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:42:07.042 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:42:07.042 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:42:07.042 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:42:07.042 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 115, 2021-02-10T11:42:07.042 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:42:07.042 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:42:07.042 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:42:07.042 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:42:07.042 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:07.042 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:42:07.043 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6824", 2021-02-10T11:42:07.043 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:42:07.043 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:07.043 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:07.043 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:42:07.043 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6826", 2021-02-10T11:42:07.043 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:42:07.043 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:42:07.043 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:42:07.043 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:07.043 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:42:07.043 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:42:07.044 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:07.044 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:42:07.044 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6828", 2021-02-10T11:42:07.044 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:42:07.044 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:07.044 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:07.044 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:42:07.044 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6829", 2021-02-10T11:42:07.044 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:42:07.044 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:42:07.044 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:42:07.044 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:07.044 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:42:07.045 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:42:07.045 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:07.045 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:42:07.045 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6832", 2021-02-10T11:42:07.045 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:42:07.045 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:07.045 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:07.045 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:42:07.045 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6833", 2021-02-10T11:42:07.045 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:42:07.045 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:42:07.045 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:42:07.045 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:07.046 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:42:07.046 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:42:07.046 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:07.046 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:42:07.046 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6830", 2021-02-10T11:42:07.046 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:42:07.046 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:07.046 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:07.046 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:42:07.046 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6831", 2021-02-10T11:42:07.046 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:42:07.046 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:42:07.046 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:42:07.047 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:07.047 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.120:6826/36385", 2021-02-10T11:42:07.047 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.120:6829/36385", 2021-02-10T11:42:07.047 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.120:6833/36385", 2021-02-10T11:42:07.047 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.120:6831/36385", 2021-02-10T11:42:07.047 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:42:07.047 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:42:07.047 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:42:07.047 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:42:07.047 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:07.047 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:07.047 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 7, 2021-02-10T11:42:07.047 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "671e2772-33b7-497e-8e44-2178bd850bcf", 2021-02-10T11:42:07.048 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:42:07.048 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:42:07.048 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:42:07.048 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:42:07.048 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:42:07.048 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:42:07.048 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:42:07.048 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 115, 2021-02-10T11:42:07.048 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:42:07.048 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:42:07.048 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:42:07.048 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:42:07.048 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:07.049 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:42:07.049 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6802", 2021-02-10T11:42:07.049 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:42:07.049 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:07.049 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:07.049 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:42:07.049 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6803", 2021-02-10T11:42:07.049 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:42:07.049 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:42:07.049 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:42:07.049 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:07.049 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:42:07.049 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:42:07.050 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:07.050 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:42:07.050 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6805", 2021-02-10T11:42:07.050 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:42:07.050 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:07.050 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:07.050 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:42:07.050 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6807", 2021-02-10T11:42:07.050 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:42:07.050 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:42:07.050 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:42:07.050 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:07.051 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:42:07.051 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:42:07.051 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:07.051 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:42:07.051 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6814", 2021-02-10T11:42:07.051 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:42:07.051 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:07.051 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:07.051 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:42:07.051 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6817", 2021-02-10T11:42:07.051 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:42:07.051 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:42:07.051 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:42:07.052 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:07.052 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:42:07.052 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:42:07.052 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:07.052 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:42:07.052 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6808", 2021-02-10T11:42:07.052 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:42:07.052 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:07.052 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:07.052 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:42:07.052 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6811", 2021-02-10T11:42:07.052 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:42:07.052 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:42:07.053 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:42:07.053 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:07.053 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.120:6803/36386", 2021-02-10T11:42:07.053 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.120:6807/36386", 2021-02-10T11:42:07.053 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.120:6817/36386", 2021-02-10T11:42:07.053 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.120:6811/36386", 2021-02-10T11:42:07.053 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:42:07.053 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:42:07.053 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:42:07.053 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:42:07.053 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:42:07.053 INFO:teuthology.orchestra.run.gibba012.stdout: ], 2021-02-10T11:42:07.053 INFO:teuthology.orchestra.run.gibba012.stdout: "osd_xinfo": [ 2021-02-10T11:42:07.054 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:07.054 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 0, 2021-02-10T11:42:07.054 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:42:07.054 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:42:07.054 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:42:07.054 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:42:07.054 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:42:07.054 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:07.054 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:07.054 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 1, 2021-02-10T11:42:07.054 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:42:07.054 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:42:07.055 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:42:07.055 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:42:07.055 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:42:07.055 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:07.055 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:07.055 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 2, 2021-02-10T11:42:07.055 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:42:07.055 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:42:07.055 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:42:07.055 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:42:07.055 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:42:07.055 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:07.055 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:07.056 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 3, 2021-02-10T11:42:07.056 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:42:07.056 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:42:07.056 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:42:07.056 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:42:07.056 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:42:07.056 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:07.056 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:07.056 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 4, 2021-02-10T11:42:07.056 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:42:07.056 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:42:07.056 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:42:07.056 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:42:07.057 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:42:07.057 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:07.057 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:07.057 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 5, 2021-02-10T11:42:07.057 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:42:07.057 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:42:07.057 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:42:07.057 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:42:07.057 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:42:07.057 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:07.057 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:07.057 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 6, 2021-02-10T11:42:07.057 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:42:07.058 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:42:07.058 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:42:07.058 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:42:07.058 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:42:07.058 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:07.058 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:07.058 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 7, 2021-02-10T11:42:07.058 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:42:07.058 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:42:07.058 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:42:07.058 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:42:07.058 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:42:07.058 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:42:07.059 INFO:teuthology.orchestra.run.gibba012.stdout: ], 2021-02-10T11:42:07.059 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_upmap": [], 2021-02-10T11:42:07.059 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_upmap_items": [], 2021-02-10T11:42:07.059 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_temp": [], 2021-02-10T11:42:07.059 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_temp": [], 2021-02-10T11:42:07.059 INFO:teuthology.orchestra.run.gibba012.stdout: "blacklist": {}, 2021-02-10T11:42:07.059 INFO:teuthology.orchestra.run.gibba012.stdout: "erasure_code_profiles": { 2021-02-10T11:42:07.059 INFO:teuthology.orchestra.run.gibba012.stdout: "default": { 2021-02-10T11:42:07.059 INFO:teuthology.orchestra.run.gibba012.stdout: "crush-failure-domain": "osd", 2021-02-10T11:42:07.059 INFO:teuthology.orchestra.run.gibba012.stdout: "k": "2", 2021-02-10T11:42:07.059 INFO:teuthology.orchestra.run.gibba012.stdout: "m": "1", 2021-02-10T11:42:07.059 INFO:teuthology.orchestra.run.gibba012.stdout: "plugin": "jerasure", 2021-02-10T11:42:07.059 INFO:teuthology.orchestra.run.gibba012.stdout: "ruleset-failure-domain": "osd", 2021-02-10T11:42:07.060 INFO:teuthology.orchestra.run.gibba012.stdout: "technique": "reed_sol_van" 2021-02-10T11:42:07.060 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:42:07.060 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:07.060 INFO:teuthology.orchestra.run.gibba012.stdout: "removed_snaps_queue": [], 2021-02-10T11:42:07.060 INFO:teuthology.orchestra.run.gibba012.stdout: "new_removed_snaps": [], 2021-02-10T11:42:07.060 INFO:teuthology.orchestra.run.gibba012.stdout: "new_purged_snaps": [], 2021-02-10T11:42:07.060 INFO:teuthology.orchestra.run.gibba012.stdout: "crush_node_flags": {}, 2021-02-10T11:42:07.060 INFO:teuthology.orchestra.run.gibba012.stdout: "device_class_flags": {} 2021-02-10T11:42:07.060 INFO:teuthology.orchestra.run.gibba012.stdout:} 2021-02-10T11:42:07.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:06.988 7fd8f8541700 1 -- 172.21.2.112:0/1846781544 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd8d8018730 msgr2=0x7fd8d801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:07.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:06.988 7fd8f8541700 1 --2- 172.21.2.112:0/1846781544 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd8d8018730 0x7fd8d801abc0 crc :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:07.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:06.988 7fd8f8541700 1 -- 172.21.2.112:0/1846781544 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd8f009c6c0 msgr2=0x7fd8f009cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:07.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:06.988 7fd8f8541700 1 --2- 172.21.2.112:0/1846781544 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd8f009c6c0 0x7fd8f009cac0 secure :-1 s=READY pgs=2280 cs=0 l=1 rev1=1 rx=0x7fd8e4006f10 tx=0x7fd8e400b990).stop 2021-02-10T11:42:07.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:06.988 7fd8f8541700 1 -- 172.21.2.112:0/1846781544 shutdown_connections 2021-02-10T11:42:07.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:06.988 7fd8f8541700 1 --2- 172.21.2.112:0/1846781544 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd8d8018730 0x7fd8d801abc0 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:07.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:06.988 7fd8f8541700 1 --2- 172.21.2.112:0/1846781544 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd8f008c1a0 0x7fd8f008c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:07.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:06.988 7fd8f8541700 1 --2- 172.21.2.112:0/1846781544 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd8f009c6c0 0x7fd8f009cac0 unknown :-1 s=CLOSED pgs=2280 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:07.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:06.988 7fd8f8541700 1 --2- 172.21.2.112:0/1846781544 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd8f009bca0 0x7fd8f009c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:07.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:06.988 7fd8f8541700 1 -- 172.21.2.112:0/1846781544 >> 172.21.2.112:0/1846781544 conn(0x7fd8f00a0450 msgr2=0x7fd8f00f0460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:07.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:06.988 7fd8f8541700 1 -- 172.21.2.112:0/1846781544 shutdown_connections 2021-02-10T11:42:07.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:06.988 7fd8f8541700 1 -- 172.21.2.112:0/1846781544 wait complete. 2021-02-10T11:42:07.063 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:42:07.488 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:07.474 7fa92d4e7700 1 Processor -- start 2021-02-10T11:42:07.489 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:07.474 7fa92d4e7700 1 -- start start 2021-02-10T11:42:07.489 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:07.474 7fa92d4e7700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa9280d1870 0x7fa9281b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:07.489 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:07.474 7fa92d4e7700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fa9281b0a10 0x7fa9280cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:07.489 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:07.474 7fa92d4e7700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fa9280d1c70 0x7fa9280cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:07.489 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:07.474 7fa92d4e7700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9280fe230 con 0x7fa9280d1c70 2021-02-10T11:42:07.489 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:07.474 7fa92d4e7700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9280e5ed0 con 0x7fa9281b0a10 2021-02-10T11:42:07.489 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:07.474 7fa92d4e7700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9280f0b10 con 0x7fa9280d1870 2021-02-10T11:42:07.489 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:07.474 7fa927ac0700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fa9280d1c70 0x7fa9280cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:07.489 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:07.474 7fa927ac0700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fa9280d1c70 0x7fa9280cf950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:55912/0 (socket says 172.21.2.112:55912) 2021-02-10T11:42:07.489 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:07.474 7fa927ac0700 1 -- 172.21.2.112:0/694100914 learned_addr learned my addr 172.21.2.112:0/694100914 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:42:07.489 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:07.475 7fa927ac0700 1 -- 172.21.2.112:0/694100914 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa9280d1870 msgr2=0x7fa9281b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:07.489 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:07.475 7fa927ac0700 1 --2- 172.21.2.112:0/694100914 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa9280d1870 0x7fa9281b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:07.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:07.475 7fa927ac0700 1 -- 172.21.2.112:0/694100914 >> v2:172.21.2.120:3300/0 conn(0x7fa9281b0a10 msgr2=0x7fa9280cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:07.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:07.475 7fa927ac0700 1 --2- 172.21.2.112:0/694100914 >> v2:172.21.2.120:3300/0 conn(0x7fa9281b0a10 0x7fa9280cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:07.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:07.475 7fa927ac0700 1 -- 172.21.2.112:0/694100914 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9280cff70 con 0x7fa9280d1c70 2021-02-10T11:42:07.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:07.475 7fa927ac0700 1 --2- 172.21.2.112:0/694100914 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa9280d1c70 0x7fa9280cf950 secure :-1 s=READY pgs=2281 cs=0 l=1 rev1=1 rx=0x7fa920006f00 tx=0x7fa920009f10).ready entity=mon.0 client_cookie=22fb947659b1f34a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:07.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:07.475 7fa9262bd700 1 -- 172.21.2.112:0/694100914 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa92000b3b0 con 0x7fa9280d1c70 2021-02-10T11:42:07.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:07.475 7fa9262bd700 1 -- 172.21.2.112:0/694100914 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa920000d30 con 0x7fa9280d1c70 2021-02-10T11:42:07.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:07.475 7fa9262bd700 1 -- 172.21.2.112:0/694100914 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa92000b8a0 con 0x7fa9280d1c70 2021-02-10T11:42:07.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:07.475 7fa92d4e7700 1 -- 172.21.2.112:0/694100914 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa9280d1c70 msgr2=0x7fa9280cf950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:07.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:07.475 7fa92d4e7700 1 --2- 172.21.2.112:0/694100914 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa9280d1c70 0x7fa9280cf950 secure :-1 s=READY pgs=2281 cs=0 l=1 rev1=1 rx=0x7fa920006f00 tx=0x7fa920009f10).stop 2021-02-10T11:42:07.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:07.476 7fa92d4e7700 1 -- 172.21.2.112:0/694100914 shutdown_connections 2021-02-10T11:42:07.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:07.476 7fa92d4e7700 1 --2- 172.21.2.112:0/694100914 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa9280d1c70 0x7fa9280cf950 unknown :-1 s=CLOSED pgs=2281 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:07.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:07.476 7fa92d4e7700 1 --2- 172.21.2.112:0/694100914 >> v2:172.21.2.120:3300/0 conn(0x7fa9281b0a10 0x7fa9280cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:07.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:07.476 7fa92d4e7700 1 --2- 172.21.2.112:0/694100914 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa9280d1870 0x7fa9281b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:07.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:07.476 7fa92d4e7700 1 -- 172.21.2.112:0/694100914 >> 172.21.2.112:0/694100914 conn(0x7fa9280ce450 msgr2=0x7fa9280d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:07.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:07.476 7fa92d4e7700 1 -- 172.21.2.112:0/694100914 shutdown_connections 2021-02-10T11:42:07.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:07.476 7fa92d4e7700 1 -- 172.21.2.112:0/694100914 wait complete. 2021-02-10T11:42:07.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:07.490 7fa92d4e7700 1 Processor -- start 2021-02-10T11:42:07.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:07.490 7fa92d4e7700 1 -- start start 2021-02-10T11:42:07.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:07.491 7fa92d4e7700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa928093060 0x7fa92819a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:07.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:07.491 7fa92d4e7700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa92819aeb0 0x7fa9280b1990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:07.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:07.491 7fa92d4e7700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa92819b2b0 0x7fa9280b1fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:07.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:07.491 7fa92d4e7700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9280efd70 con 0x7fa92819b2b0 2021-02-10T11:42:07.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:07.491 7fa92d4e7700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa92819b8a0 con 0x7fa928093060 2021-02-10T11:42:07.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:07.491 7fa92d4e7700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9280b25d0 con 0x7fa92819aeb0 2021-02-10T11:42:07.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:07.491 7fa927ac0700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa92819b2b0 0x7fa9280b1fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:07.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:07.491 7fa927ac0700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa92819b2b0 0x7fa9280b1fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:55914/0 (socket says 172.21.2.112:55914) 2021-02-10T11:42:07.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:07.491 7fa927ac0700 1 -- 172.21.2.112:0/387325616 learned_addr learned my addr 172.21.2.112:0/387325616 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:42:07.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:07.491 7fa927ac0700 1 -- 172.21.2.112:0/387325616 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa92819aeb0 msgr2=0x7fa9280b1990 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:42:07.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:07.491 7fa927ac0700 1 --2- 172.21.2.112:0/387325616 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa92819aeb0 0x7fa9280b1990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:07.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:07.491 7fa927ac0700 1 -- 172.21.2.112:0/387325616 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa928093060 msgr2=0x7fa92819a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:42:07.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:07.491 7fa927ac0700 1 --2- 172.21.2.112:0/387325616 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa928093060 0x7fa92819a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:07.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:07.491 7fa927ac0700 1 -- 172.21.2.112:0/387325616 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa92818e410 con 0x7fa92819b2b0 2021-02-10T11:42:07.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:07.491 7fa927ac0700 1 --2- 172.21.2.112:0/387325616 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa92819b2b0 0x7fa9280b1fb0 secure :-1 s=READY pgs=2282 cs=0 l=1 rev1=1 rx=0x7fa9200099c0 tx=0x7fa92000bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:07.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:07.492 7fa91ffff700 1 -- 172.21.2.112:0/387325616 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa920000f90 con 0x7fa92819b2b0 2021-02-10T11:42:07.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:07.492 7fa91ffff700 1 -- 172.21.2.112:0/387325616 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa920006cc0 con 0x7fa92819b2b0 2021-02-10T11:42:07.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:07.492 7fa92d4e7700 1 -- 172.21.2.112:0/387325616 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa92818e410 con 0x7fa92819b2b0 2021-02-10T11:42:07.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:07.492 7fa91ffff700 1 -- 172.21.2.112:0/387325616 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa9200014f0 con 0x7fa92819b2b0 2021-02-10T11:42:07.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:07.492 7fa92d4e7700 1 -- 172.21.2.112:0/387325616 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa92818e9a0 con 0x7fa92819b2b0 2021-02-10T11:42:07.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:07.492 7fa91ffff700 1 -- 172.21.2.112:0/387325616 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa920001270 con 0x7fa92819b2b0 2021-02-10T11:42:07.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:07.492 7fa91ffff700 1 --2- 172.21.2.112:0/387325616 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa90c0186c0 0x7fa90c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:07.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:07.492 7fa91ffff700 1 -- 172.21.2.112:0/387325616 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fa920032790 con 0x7fa92819b2b0 2021-02-10T11:42:07.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:07.493 7fa9272bf700 1 --2- 172.21.2.112:0/387325616 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa90c0186c0 0x7fa90c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:07.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:07.493 7fa9272bf700 1 --2- 172.21.2.112:0/387325616 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa90c0186c0 0x7fa90c01abc0 crc :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:07.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:07.524 7fa92d4e7700 1 -- 172.21.2.112:0/387325616 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa92818f230 con 0x7fa92819b2b0 2021-02-10T11:42:07.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:07.527 7fa91ffff700 1 -- 172.21.2.112:0/387325616 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa920000d30 con 0x7fa92819b2b0 2021-02-10T11:42:07.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:07.655 7fa92d4e7700 1 -- 172.21.2.112:0/387325616 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa92818f230 con 0x7fa92819b2b0 2021-02-10T11:42:07.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:07.655 7fa91ffff700 1 -- 172.21.2.112:0/387325616 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+2098 (secure 0 0 0) 0x7fa920001270 con 0x7fa92819b2b0 2021-02-10T11:42:07.654 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:42:07.654 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":98,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10671,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/668146927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":668146927},{"type":"v1","addr":"172.21.2.112:6833","nonce":668146927}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":93}],"filesystems":[{"mdsmap":{"epoch":97,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:41:23.731569","modified":"2021-02-10 11:41:25.740952","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8008},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8008":{"gid":8008,"name":"b","rank":0,"incarnation":94,"state":"up:active","state_seq":3,"addr":"172.21.2.120:6835/862287497","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":862287497},{"type":"v1","addr":"172.21.2.120:6835","nonce":862287497}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":12}]} 2021-02-10T11:42:07.654 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 98 2021-02-10T11:42:07.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:07.657 7fa92d4e7700 1 -- 172.21.2.112:0/387325616 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa90c0186c0 msgr2=0x7fa90c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:07.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:07.657 7fa92d4e7700 1 --2- 172.21.2.112:0/387325616 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa90c0186c0 0x7fa90c01abc0 crc :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:07.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:07.657 7fa92d4e7700 1 -- 172.21.2.112:0/387325616 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa92819b2b0 msgr2=0x7fa9280b1fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:07.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:07.657 7fa92d4e7700 1 --2- 172.21.2.112:0/387325616 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa92819b2b0 0x7fa9280b1fb0 secure :-1 s=READY pgs=2282 cs=0 l=1 rev1=1 rx=0x7fa9200099c0 tx=0x7fa92000bd10).stop 2021-02-10T11:42:07.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:07.657 7fa92d4e7700 1 -- 172.21.2.112:0/387325616 shutdown_connections 2021-02-10T11:42:07.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:07.657 7fa92d4e7700 1 --2- 172.21.2.112:0/387325616 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa90c0186c0 0x7fa90c01abc0 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:07.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:07.657 7fa92d4e7700 1 --2- 172.21.2.112:0/387325616 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa92819b2b0 0x7fa9280b1fb0 unknown :-1 s=CLOSED pgs=2282 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:07.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:07.657 7fa92d4e7700 1 --2- 172.21.2.112:0/387325616 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa92819aeb0 0x7fa9280b1990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:07.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:07.657 7fa92d4e7700 1 --2- 172.21.2.112:0/387325616 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa928093060 0x7fa92819a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:07.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:07.657 7fa92d4e7700 1 -- 172.21.2.112:0/387325616 >> 172.21.2.112:0/387325616 conn(0x7fa92819f880 msgr2=0x7fa928199a80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:07.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:07.657 7fa92d4e7700 1 -- 172.21.2.112:0/387325616 shutdown_connections 2021-02-10T11:42:07.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:07.658 7fa92d4e7700 1 -- 172.21.2.112:0/387325616 wait complete. 2021-02-10T11:42:07.664 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs fail cephfs 2021-02-10T11:42:08.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:08.076 7f2644d18700 1 Processor -- start 2021-02-10T11:42:08.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:08.076 7f2644d18700 1 -- start start 2021-02-10T11:42:08.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:08.077 7f2644d18700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f26400d1870 0x7f26401b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:08.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:08.077 7f2644d18700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f26401b0a10 0x7f26400cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:08.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:08.077 7f2644d18700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26400f0b10 con 0x7f26400d1c70 2021-02-10T11:42:08.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:08.077 7f2644d18700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26400fe230 con 0x7f26401b0a10 2021-02-10T11:42:08.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:08.077 7f2644d18700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26400e5ed0 con 0x7f26400d1870 2021-02-10T11:42:08.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:08.077 7f263e312700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f26401b0a10 0x7f26400cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:08.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:08.077 7f263e312700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f26401b0a10 0x7f26400cf330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:55924/0 (socket says 172.21.2.112:55924) 2021-02-10T11:42:08.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:08.077 7f263e312700 1 -- 172.21.2.112:0/4117185138 learned_addr learned my addr 172.21.2.112:0/4117185138 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:42:08.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:08.077 7f263e312700 1 -- 172.21.2.112:0/4117185138 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f26400d1870 msgr2=0x7f26401b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:08.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:08.077 7f263e312700 1 --2- 172.21.2.112:0/4117185138 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f26400d1870 0x7f26401b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:08.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:08.077 7f263e312700 1 -- 172.21.2.112:0/4117185138 >> v1:172.21.2.120:6789/0 conn(0x7f26400d1c70 legacy=0x7f26400cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:08.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:08.077 7f263e312700 1 -- 172.21.2.112:0/4117185138 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26400f0650 con 0x7f26401b0a10 2021-02-10T11:42:08.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:08.077 7f263e312700 1 --2- 172.21.2.112:0/4117185138 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f26401b0a10 0x7f26400cf330 secure :-1 s=READY pgs=2283 cs=0 l=1 rev1=1 rx=0x7f2628006700 tx=0x7f2628009b80).ready entity=mon.0 client_cookie=30fcb20ef829dae2 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:08.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:08.078 7f263db11700 1 -- 172.21.2.112:0/4117185138 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f262800c0c0 con 0x7f26401b0a10 2021-02-10T11:42:08.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:08.078 7f263db11700 1 -- 172.21.2.112:0/4117185138 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f262800c0c0 con 0x7f26401b0a10 2021-02-10T11:42:08.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:08.078 7f263db11700 1 -- 172.21.2.112:0/4117185138 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f262800c440 con 0x7f26401b0a10 2021-02-10T11:42:08.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:08.078 7f2644d18700 1 -- 172.21.2.112:0/4117185138 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f26401b0a10 msgr2=0x7f26400cf330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:08.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:08.078 7f2644d18700 1 --2- 172.21.2.112:0/4117185138 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f26401b0a10 0x7f26400cf330 secure :-1 s=READY pgs=2283 cs=0 l=1 rev1=1 rx=0x7f2628006700 tx=0x7f2628009b80).stop 2021-02-10T11:42:08.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:08.078 7f2644d18700 1 -- 172.21.2.112:0/4117185138 shutdown_connections 2021-02-10T11:42:08.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:08.078 7f2644d18700 1 --2- 172.21.2.112:0/4117185138 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f26401b0a10 0x7f26400cf330 unknown :-1 s=CLOSED pgs=2283 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:08.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:08.078 7f2644d18700 1 --2- 172.21.2.112:0/4117185138 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f26400d1870 0x7f26401b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:08.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:08.078 7f2644d18700 1 -- 172.21.2.112:0/4117185138 >> 172.21.2.112:0/4117185138 conn(0x7f26400ce450 msgr2=0x7f26400d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:08.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:08.079 7f2644d18700 1 -- 172.21.2.112:0/4117185138 shutdown_connections 2021-02-10T11:42:08.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:08.079 7f2644d18700 1 -- 172.21.2.112:0/4117185138 wait complete. 2021-02-10T11:42:08.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:08.093 7f2644d18700 1 Processor -- start 2021-02-10T11:42:08.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:08.093 7f2644d18700 1 -- start start 2021-02-10T11:42:08.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:08.094 7f2644d18700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2640190c80 0x7f26401a4400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:08.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:08.094 7f2644d18700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f26401a4a20 0x7f2640193260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:08.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:08.094 7f2644d18700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2640193880 0x7f26400f0830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:08.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:08.094 7f2644d18700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2640190790 con 0x7f2640193880 2021-02-10T11:42:08.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:08.094 7f2644d18700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2640190980 con 0x7f26401a4a20 2021-02-10T11:42:08.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:08.094 7f2644d18700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2640193c80 con 0x7f2640190c80 2021-02-10T11:42:08.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:08.094 7f263f314700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2640193880 0x7f26400f0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:08.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:08.094 7f263f314700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2640193880 0x7f26400f0830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:55932/0 (socket says 172.21.2.112:55932) 2021-02-10T11:42:08.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:08.094 7f263f314700 1 -- 172.21.2.112:0/3770190468 learned_addr learned my addr 172.21.2.112:0/3770190468 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:42:08.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:08.094 7f263e312700 1 --2- 172.21.2.112:0/3770190468 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f26401a4a20 0x7f2640193260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:08.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:08.094 7f263eb13700 1 --2- 172.21.2.112:0/3770190468 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2640190c80 0x7f26401a4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:08.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:08.094 7f263f314700 1 -- 172.21.2.112:0/3770190468 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2640190c80 msgr2=0x7f26401a4400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:08.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:08.094 7f263f314700 1 --2- 172.21.2.112:0/3770190468 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2640190c80 0x7f26401a4400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:08.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:08.094 7f263f314700 1 -- 172.21.2.112:0/3770190468 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f26401a4a20 msgr2=0x7f2640193260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:08.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:08.094 7f263f314700 1 --2- 172.21.2.112:0/3770190468 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f26401a4a20 0x7f2640193260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:08.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:08.094 7f263f314700 1 -- 172.21.2.112:0/3770190468 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2640193e60 con 0x7f2640193880 2021-02-10T11:42:08.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:08.094 7f263e312700 1 --2- 172.21.2.112:0/3770190468 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f26401a4a20 0x7f2640193260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:42:08.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:08.094 7f263eb13700 1 --2- 172.21.2.112:0/3770190468 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2640190c80 0x7f26401a4400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:42:08.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:08.094 7f263f314700 1 --2- 172.21.2.112:0/3770190468 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2640193880 0x7f26400f0830 secure :-1 s=READY pgs=2284 cs=0 l=1 rev1=1 rx=0x7f2638009940 tx=0x7f263800bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:08.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:08.095 7f26377fe700 1 -- 172.21.2.112:0/3770190468 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2638000f90 con 0x7f2640193880 2021-02-10T11:42:08.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:08.095 7f26377fe700 1 -- 172.21.2.112:0/3770190468 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f263800c2f0 con 0x7f2640193880 2021-02-10T11:42:08.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:08.095 7f2644d18700 1 -- 172.21.2.112:0/3770190468 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2640193e60 con 0x7f2640193880 2021-02-10T11:42:08.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:08.095 7f26377fe700 1 -- 172.21.2.112:0/3770190468 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f263800c810 con 0x7f2640193880 2021-02-10T11:42:08.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:08.095 7f2644d18700 1 -- 172.21.2.112:0/3770190468 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26400f1140 con 0x7f2640193880 2021-02-10T11:42:08.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:08.095 7f26377fe700 1 -- 172.21.2.112:0/3770190468 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2638001220 con 0x7f2640193880 2021-02-10T11:42:08.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:08.095 7f26377fe700 1 --2- 172.21.2.112:0/3770190468 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f26240186c0 0x7f262401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:08.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:08.095 7f26377fe700 1 -- 172.21.2.112:0/3770190468 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f26380327e0 con 0x7f2640193880 2021-02-10T11:42:08.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:08.096 7f263eb13700 1 --2- 172.21.2.112:0/3770190468 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f26240186c0 0x7f262401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:08.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:08.096 7f263eb13700 1 --2- 172.21.2.112:0/3770190468 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f26240186c0 0x7f262401abc0 crc :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:08.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:08.126 7f2644d18700 1 -- 172.21.2.112:0/3770190468 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26400ce450 con 0x7f2640193880 2021-02-10T11:42:08.127 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:08.129 7f26377fe700 1 -- 172.21.2.112:0/3770190468 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2638002ef0 con 0x7f2640193880 2021-02-10T11:42:08.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:08.258 7f2644d18700 1 -- 172.21.2.112:0/3770190468 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f26400ce450 con 0x7f2640193880 2021-02-10T11:42:09.207 INFO:tasks.ceph.mon.a.gibba012.stderr:2021-02-10 11:42:09.209 7fc3fe6e2700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-02-10T11:42:09.213 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:09.214 7f26377fe700 1 -- 172.21.2.112:0/3770190468 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v99) v1 ==== 161+0+0 (secure 0 0 0) 0x7f2638001220 con 0x7f2640193880 2021-02-10T11:42:09.228 INFO:teuthology.orchestra.run.gibba012.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2021-02-10T11:42:09.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:09.231 7f2644d18700 1 -- 172.21.2.112:0/3770190468 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f26240186c0 msgr2=0x7f262401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:09.230 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:09.231 7f2644d18700 1 --2- 172.21.2.112:0/3770190468 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f26240186c0 0x7f262401abc0 crc :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:09.230 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:09.231 7f2644d18700 1 -- 172.21.2.112:0/3770190468 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2640193880 msgr2=0x7f26400f0830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:09.230 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:09.231 7f2644d18700 1 --2- 172.21.2.112:0/3770190468 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2640193880 0x7f26400f0830 secure :-1 s=READY pgs=2284 cs=0 l=1 rev1=1 rx=0x7f2638009940 tx=0x7f263800bcf0).stop 2021-02-10T11:42:09.230 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:09.231 7f2644d18700 1 -- 172.21.2.112:0/3770190468 shutdown_connections 2021-02-10T11:42:09.230 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:09.231 7f2644d18700 1 --2- 172.21.2.112:0/3770190468 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f26240186c0 0x7f262401abc0 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:09.230 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:09.231 7f2644d18700 1 --2- 172.21.2.112:0/3770190468 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f26401a4a20 0x7f2640193260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:09.230 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:09.231 7f2644d18700 1 --2- 172.21.2.112:0/3770190468 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2640193880 0x7f26400f0830 unknown :-1 s=CLOSED pgs=2284 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:09.230 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:09.231 7f2644d18700 1 --2- 172.21.2.112:0/3770190468 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2640190c80 0x7f26401a4400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:09.230 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:09.231 7f2644d18700 1 -- 172.21.2.112:0/3770190468 >> 172.21.2.112:0/3770190468 conn(0x7f2640191110 msgr2=0x7f26400bf0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:09.231 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:09.232 7f2644d18700 1 -- 172.21.2.112:0/3770190468 shutdown_connections 2021-02-10T11:42:09.231 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:09.232 7f2644d18700 1 -- 172.21.2.112:0/3770190468 wait complete. 2021-02-10T11:42:09.238 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:42:09.265 INFO:tasks.ceph.mds.b.gibba020.stdout:starting mds.b at 2021-02-10T11:42:09.666 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:09.651 7fcc82d8f700 1 Processor -- start 2021-02-10T11:42:09.666 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:09.651 7fcc82d8f700 1 -- start start 2021-02-10T11:42:09.666 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:09.652 7fcc82d8f700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fcc7c1b0470 0x7fcc7c1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:09.666 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:09.652 7fcc82d8f700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fcc7c0f20e0 0x7fcc7c0f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:09.666 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:09.652 7fcc82d8f700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc7c0f0b50 con 0x7fcc7c0ef610 2021-02-10T11:42:09.666 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:09.652 7fcc82d8f700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc7c0f0820 con 0x7fcc7c0f20e0 2021-02-10T11:42:09.666 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:09.652 7fcc82d8f700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc7c0f24e0 con 0x7fcc7c1b0470 2021-02-10T11:42:09.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:09.652 7fcc813ac700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fcc7c0f20e0 0x7fcc7c0f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:09.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:09.652 7fcc813ac700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fcc7c0f20e0 0x7fcc7c0f0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:55940/0 (socket says 172.21.2.112:55940) 2021-02-10T11:42:09.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:09.652 7fcc813ac700 1 -- 172.21.2.112:0/1463037457 learned_addr learned my addr 172.21.2.112:0/1463037457 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:42:09.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:09.652 7fcc80bab700 1 --2- 172.21.2.112:0/1463037457 >> v2:172.21.2.120:3300/0 conn(0x7fcc7c1b0470 0x7fcc7c1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:09.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:09.652 7fcc813ac700 1 -- 172.21.2.112:0/1463037457 >> v2:172.21.2.120:3300/0 conn(0x7fcc7c1b0470 msgr2=0x7fcc7c1b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:09.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:09.652 7fcc813ac700 1 --2- 172.21.2.112:0/1463037457 >> v2:172.21.2.120:3300/0 conn(0x7fcc7c1b0470 0x7fcc7c1b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:09.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:09.652 7fcc813ac700 1 -- 172.21.2.112:0/1463037457 >> v1:172.21.2.112:6789/0 conn(0x7fcc7c0ef610 legacy=0x7fcc7c0efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:09.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:09.652 7fcc813ac700 1 -- 172.21.2.112:0/1463037457 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc7c0e52e0 con 0x7fcc7c0f20e0 2021-02-10T11:42:09.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:09.653 7fcc813ac700 1 --2- 172.21.2.112:0/1463037457 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcc7c0f20e0 0x7fcc7c0f0200 secure :-1 s=READY pgs=2286 cs=0 l=1 rev1=1 rx=0x7fcc74006f00 tx=0x7fcc74009f10).ready entity=mon.0 client_cookie=eb4632a9ee3b669c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:09.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:09.653 7fcc7b7fe700 1 -- 172.21.2.112:0/1463037457 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcc7400b3b0 con 0x7fcc7c0f20e0 2021-02-10T11:42:09.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:09.653 7fcc7b7fe700 1 -- 172.21.2.112:0/1463037457 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc7400b630 con 0x7fcc7c0f20e0 2021-02-10T11:42:09.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:09.653 7fcc7b7fe700 1 -- 172.21.2.112:0/1463037457 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcc7400bb50 con 0x7fcc7c0f20e0 2021-02-10T11:42:09.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:09.653 7fcc82d8f700 1 -- 172.21.2.112:0/1463037457 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcc7c0f20e0 msgr2=0x7fcc7c0f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:09.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:09.653 7fcc82d8f700 1 --2- 172.21.2.112:0/1463037457 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcc7c0f20e0 0x7fcc7c0f0200 secure :-1 s=READY pgs=2286 cs=0 l=1 rev1=1 rx=0x7fcc74006f00 tx=0x7fcc74009f10).stop 2021-02-10T11:42:09.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:09.653 7fcc82d8f700 1 -- 172.21.2.112:0/1463037457 shutdown_connections 2021-02-10T11:42:09.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:09.653 7fcc82d8f700 1 --2- 172.21.2.112:0/1463037457 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcc7c0f20e0 0x7fcc7c0f0200 unknown :-1 s=CLOSED pgs=2286 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:09.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:09.653 7fcc82d8f700 1 --2- 172.21.2.112:0/1463037457 >> v2:172.21.2.120:3300/0 conn(0x7fcc7c1b0470 0x7fcc7c1b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:09.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:09.653 7fcc82d8f700 1 -- 172.21.2.112:0/1463037457 >> 172.21.2.112:0/1463037457 conn(0x7fcc7c0ce3e0 msgr2=0x7fcc7c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:09.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:09.653 7fcc82d8f700 1 -- 172.21.2.112:0/1463037457 shutdown_connections 2021-02-10T11:42:09.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:09.654 7fcc82d8f700 1 -- 172.21.2.112:0/1463037457 wait complete. 2021-02-10T11:42:09.669 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:09.668 7fcc82d8f700 1 Processor -- start 2021-02-10T11:42:09.669 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:09.668 7fcc82d8f700 1 -- start start 2021-02-10T11:42:09.670 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:09.668 7fcc82d8f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcc7c1a48e0 0x7fcc7c0b65c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:09.670 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:09.669 7fcc82d8f700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcc7c090bc0 0x7fcc7c090fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:09.670 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:09.669 7fcc82d8f700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcc7c0915e0 0x7fcc7c0b73e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:09.670 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:09.669 7fcc82d8f700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc7c09d7b0 con 0x7fcc7c1a48e0 2021-02-10T11:42:09.670 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:09.669 7fcc82d8f700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc7c09d2f0 con 0x7fcc7c0915e0 2021-02-10T11:42:09.670 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:09.669 7fcc82d8f700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc7c0b7a00 con 0x7fcc7c090bc0 2021-02-10T11:42:09.670 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:09.669 7fcc80bab700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcc7c1a48e0 0x7fcc7c0b65c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:09.670 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:09.669 7fcc80bab700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcc7c1a48e0 0x7fcc7c0b65c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:55946/0 (socket says 172.21.2.112:55946) 2021-02-10T11:42:09.670 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:09.669 7fcc80bab700 1 -- 172.21.2.112:0/3749092497 learned_addr learned my addr 172.21.2.112:0/3749092497 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:42:09.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:09.669 7fcc80bab700 1 -- 172.21.2.112:0/3749092497 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcc7c090bc0 msgr2=0x7fcc7c090fc0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:42:09.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:09.669 7fcc80bab700 1 --2- 172.21.2.112:0/3749092497 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcc7c090bc0 0x7fcc7c090fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:09.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:09.669 7fcc80bab700 1 -- 172.21.2.112:0/3749092497 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcc7c0915e0 msgr2=0x7fcc7c0b73e0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:42:09.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:09.669 7fcc80bab700 1 --2- 172.21.2.112:0/3749092497 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcc7c0915e0 0x7fcc7c0b73e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:09.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:09.669 7fcc80bab700 1 -- 172.21.2.112:0/3749092497 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc7c0b7c20 con 0x7fcc7c1a48e0 2021-02-10T11:42:09.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:09.669 7fcc80bab700 1 --2- 172.21.2.112:0/3749092497 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcc7c1a48e0 0x7fcc7c0b65c0 secure :-1 s=READY pgs=2287 cs=0 l=1 rev1=1 rx=0x7fcc64006f00 tx=0x7fcc6400b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:09.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:09.669 7fcc797fa700 1 -- 172.21.2.112:0/3749092497 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcc6400bd30 con 0x7fcc7c1a48e0 2021-02-10T11:42:09.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:09.669 7fcc797fa700 1 -- 172.21.2.112:0/3749092497 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc64002030 con 0x7fcc7c1a48e0 2021-02-10T11:42:09.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:09.669 7fcc797fa700 1 -- 172.21.2.112:0/3749092497 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcc64002550 con 0x7fcc7c1a48e0 2021-02-10T11:42:09.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:09.669 7fcc82d8f700 1 -- 172.21.2.112:0/3749092497 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc7c0b7c20 con 0x7fcc7c1a48e0 2021-02-10T11:42:09.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:09.670 7fcc82d8f700 1 -- 172.21.2.112:0/3749092497 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc7c0b81b0 con 0x7fcc7c1a48e0 2021-02-10T11:42:09.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:09.670 7fcc797fa700 1 -- 172.21.2.112:0/3749092497 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fcc64000c90 con 0x7fcc7c1a48e0 2021-02-10T11:42:09.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:09.670 7fcc797fa700 1 --2- 172.21.2.112:0/3749092497 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcc600186c0 0x7fcc6001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:09.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:09.670 7fcc797fa700 1 -- 172.21.2.112:0/3749092497 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5659+0+0 (secure 0 0 0) 0x7fcc640325b0 con 0x7fcc7c1a48e0 2021-02-10T11:42:09.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:09.671 7fcc7bfff700 1 --2- 172.21.2.112:0/3749092497 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcc600186c0 0x7fcc6001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:09.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:09.671 7fcc7bfff700 1 --2- 172.21.2.112:0/3749092497 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcc600186c0 0x7fcc6001abc0 crc :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:09.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:09.701 7fcc82d8f700 1 -- 172.21.2.112:0/3749092497 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc7c105a20 con 0x7fcc7c1a48e0 2021-02-10T11:42:09.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:09.704 7fcc797fa700 1 -- 172.21.2.112:0/3749092497 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fcc640009b0 con 0x7fcc7c1a48e0 2021-02-10T11:42:09.831 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:09.833 7fcc82d8f700 1 -- 172.21.2.112:0/3749092497 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcc7c0f0b50 con 0x7fcc7c1a48e0 2021-02-10T11:42:09.832 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:09.833 7fcc797fa700 1 -- 172.21.2.112:0/3749092497 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+1757 (secure 0 0 0) 0x7fcc6400bd30 con 0x7fcc7c1a48e0 2021-02-10T11:42:09.832 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 99 2021-02-10T11:42:09.833 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:42:09.833 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":99,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10671,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/668146927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":668146927},{"type":"v1","addr":"172.21.2.112:6833","nonce":668146927}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":93}],"filesystems":[{"mdsmap":{"epoch":99,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:41:23.731569","modified":"2021-02-10 11:42:09.209783","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":118,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":12}]} 2021-02-10T11:42:09.834 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:09.835 7fcc82d8f700 1 -- 172.21.2.112:0/3749092497 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcc600186c0 msgr2=0x7fcc6001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:09.834 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:09.835 7fcc82d8f700 1 --2- 172.21.2.112:0/3749092497 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcc600186c0 0x7fcc6001abc0 crc :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:09.834 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:09.835 7fcc82d8f700 1 -- 172.21.2.112:0/3749092497 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcc7c1a48e0 msgr2=0x7fcc7c0b65c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:09.834 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:09.835 7fcc82d8f700 1 --2- 172.21.2.112:0/3749092497 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcc7c1a48e0 0x7fcc7c0b65c0 secure :-1 s=READY pgs=2287 cs=0 l=1 rev1=1 rx=0x7fcc64006f00 tx=0x7fcc6400b940).stop 2021-02-10T11:42:09.835 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:09.836 7fcc82d8f700 1 -- 172.21.2.112:0/3749092497 shutdown_connections 2021-02-10T11:42:09.835 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:09.836 7fcc82d8f700 1 --2- 172.21.2.112:0/3749092497 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcc600186c0 0x7fcc6001abc0 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:09.835 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:09.836 7fcc82d8f700 1 --2- 172.21.2.112:0/3749092497 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcc7c0915e0 0x7fcc7c0b73e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:09.835 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:09.836 7fcc82d8f700 1 --2- 172.21.2.112:0/3749092497 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcc7c090bc0 0x7fcc7c090fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:09.835 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:09.836 7fcc82d8f700 1 --2- 172.21.2.112:0/3749092497 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcc7c1a48e0 0x7fcc7c0b65c0 unknown :-1 s=CLOSED pgs=2287 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:09.835 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:09.836 7fcc82d8f700 1 -- 172.21.2.112:0/3749092497 >> 172.21.2.112:0/3749092497 conn(0x7fcc7c09cb20 msgr2=0x7fcc7c0c3c80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:09.835 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:09.836 7fcc82d8f700 1 -- 172.21.2.112:0/3749092497 shutdown_connections 2021-02-10T11:42:09.835 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:09.836 7fcc82d8f700 1 -- 172.21.2.112:0/3749092497 wait complete. 2021-02-10T11:42:09.843 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2021-02-10T11:42:10.276 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:10.261 7f0688009700 1 Processor -- start 2021-02-10T11:42:10.276 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:10.261 7f0688009700 1 -- start start 2021-02-10T11:42:10.276 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:10.262 7f0688009700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f06800d1c60 0x7f06800f2090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:10.276 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:10.262 7f0688009700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06800f0b00 con 0x7f06800d0330 2021-02-10T11:42:10.276 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:10.262 7f0688009700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06801b0bd0 con 0x7f06800d1860 2021-02-10T11:42:10.276 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:10.262 7f0688009700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06800e5ec0 con 0x7f06800d1c60 2021-02-10T11:42:10.277 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:10.262 7f0686626700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f06800d1c60 0x7f06800f2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:10.277 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:10.262 7f0686626700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f06800d1c60 0x7f06800f2090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:55954/0 (socket says 172.21.2.112:55954) 2021-02-10T11:42:10.277 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:10.262 7f0685624700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f06800d0330 0x7f06800ef630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:55412/0 (socket says 172.21.2.112:55412) 2021-02-10T11:42:10.277 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:10.262 7f0685624700 1 -- 172.21.2.112:0/505848009 learned_addr learned my addr 172.21.2.112:0/505848009 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:42:10.277 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:10.262 7f0684e23700 1 -- 172.21.2.112:0/505848009 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2610158688 0 0) 0x7f067c0010b0 con 0x7f06800d0330 2021-02-10T11:42:10.277 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:10.262 7f0684e23700 1 -- 172.21.2.112:0/505848009 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f066c002fe0 con 0x7f06800d0330 2021-02-10T11:42:10.277 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:10.262 7f0686626700 1 -- 172.21.2.112:0/505848009 >> v1:172.21.2.120:6789/0 conn(0x7f06800d1860 legacy=0x7f06801b03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:10.277 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:10.262 7f0686626700 1 -- 172.21.2.112:0/505848009 >> v1:172.21.2.112:6789/0 conn(0x7f06800d0330 legacy=0x7f06800ef630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:10.277 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:10.262 7f0686626700 1 -- 172.21.2.112:0/505848009 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06800efe20 con 0x7f06800d1c60 2021-02-10T11:42:10.277 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:10.262 7f0686626700 1 --2- 172.21.2.112:0/505848009 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f06800d1c60 0x7f06800f2090 secure :-1 s=READY pgs=2289 cs=0 l=1 rev1=1 rx=0x7f0678006e30 tx=0x7f0678009e60).ready entity=mon.0 client_cookie=69144c9cc313eec4 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:10.277 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:10.263 7f0684e23700 1 -- 172.21.2.112:0/505848009 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0678006a50 con 0x7f06800d1c60 2021-02-10T11:42:10.277 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:10.263 7f0684e23700 1 -- 172.21.2.112:0/505848009 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f067800b300 con 0x7f06800d1c60 2021-02-10T11:42:10.278 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:10.263 7f0684e23700 1 -- 172.21.2.112:0/505848009 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f067800b7e0 con 0x7f06800d1c60 2021-02-10T11:42:10.278 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:10.263 7f0688009700 1 -- 172.21.2.112:0/505848009 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f06800d1c60 msgr2=0x7f06800f2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:10.278 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:10.263 7f0688009700 1 --2- 172.21.2.112:0/505848009 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f06800d1c60 0x7f06800f2090 secure :-1 s=READY pgs=2289 cs=0 l=1 rev1=1 rx=0x7f0678006e30 tx=0x7f0678009e60).stop 2021-02-10T11:42:10.278 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:10.263 7f0688009700 1 -- 172.21.2.112:0/505848009 shutdown_connections 2021-02-10T11:42:10.278 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:10.263 7f0688009700 1 --2- 172.21.2.112:0/505848009 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f06800d1c60 0x7f06800f2090 unknown :-1 s=CLOSED pgs=2289 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:10.278 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:10.263 7f0688009700 1 -- 172.21.2.112:0/505848009 >> 172.21.2.112:0/505848009 conn(0x7f06800ce440 msgr2=0x7f06800d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:10.278 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:10.264 7f0688009700 1 -- 172.21.2.112:0/505848009 shutdown_connections 2021-02-10T11:42:10.278 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:10.264 7f0688009700 1 -- 172.21.2.112:0/505848009 wait complete. 2021-02-10T11:42:10.280 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:10.278 7f0688009700 1 Processor -- start 2021-02-10T11:42:10.280 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:10.278 7f0688009700 1 -- start start 2021-02-10T11:42:10.280 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:10.279 7f0688009700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f06800ee8a0 0x7f06800f02f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:10.280 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:10.279 7f0688009700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f06800edc20 0x7f068019ffc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:10.280 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:10.279 7f0688009700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f06801a05e0 0x7f0680187e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:10.280 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:10.279 7f0688009700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06800ed6a0 con 0x7f06800ee8a0 2021-02-10T11:42:10.280 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:10.279 7f0688009700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0680188490 con 0x7f06800edc20 2021-02-10T11:42:10.280 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:10.279 7f0688009700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0680188670 con 0x7f06801a05e0 2021-02-10T11:42:10.280 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:10.279 7f0685e25700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f06800ee8a0 0x7f06800f02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:10.280 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:10.279 7f0685e25700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f06800ee8a0 0x7f06800f02f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:55958/0 (socket says 172.21.2.112:55958) 2021-02-10T11:42:10.280 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:10.279 7f0685e25700 1 -- 172.21.2.112:0/2815244610 learned_addr learned my addr 172.21.2.112:0/2815244610 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:42:10.281 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:10.279 7f0685624700 1 --2- 172.21.2.112:0/2815244610 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f06800edc20 0x7f068019ffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:10.281 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:10.279 7f0686626700 1 --2- 172.21.2.112:0/2815244610 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f06801a05e0 0x7f0680187e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:10.281 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:10.279 7f0685e25700 1 -- 172.21.2.112:0/2815244610 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f06801a05e0 msgr2=0x7f0680187e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:10.281 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:10.279 7f0685e25700 1 --2- 172.21.2.112:0/2815244610 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f06801a05e0 0x7f0680187e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:10.281 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:10.279 7f0685e25700 1 -- 172.21.2.112:0/2815244610 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f06800edc20 msgr2=0x7f068019ffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:10.281 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:10.279 7f0685e25700 1 --2- 172.21.2.112:0/2815244610 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f06800edc20 0x7f068019ffc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:10.281 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:10.279 7f0685e25700 1 -- 172.21.2.112:0/2815244610 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0680188890 con 0x7f06800ee8a0 2021-02-10T11:42:10.281 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:10.280 7f0685e25700 1 --2- 172.21.2.112:0/2815244610 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f06800ee8a0 0x7f06800f02f0 secure :-1 s=READY pgs=2290 cs=0 l=1 rev1=1 rx=0x7f067000b5b0 tx=0x7f067000b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:10.281 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:10.280 7f06767fc700 1 -- 172.21.2.112:0/2815244610 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f067000bd40 con 0x7f06800ee8a0 2021-02-10T11:42:10.281 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:10.280 7f06767fc700 1 -- 172.21.2.112:0/2815244610 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0670006240 con 0x7f06800ee8a0 2021-02-10T11:42:10.281 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:10.280 7f0688009700 1 -- 172.21.2.112:0/2815244610 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0680188890 con 0x7f06800ee8a0 2021-02-10T11:42:10.281 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:10.280 7f06767fc700 1 -- 172.21.2.112:0/2815244610 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0670005690 con 0x7f06800ee8a0 2021-02-10T11:42:10.281 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:10.280 7f0688009700 1 -- 172.21.2.112:0/2815244610 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0680188890 con 0x7f06800ee8a0 2021-02-10T11:42:10.282 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:10.281 7f06767fc700 1 -- 172.21.2.112:0/2815244610 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f06700054a0 con 0x7f06800ee8a0 2021-02-10T11:42:10.282 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:10.281 7f06767fc700 1 --2- 172.21.2.112:0/2815244610 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0668018730 0x7f066801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:10.282 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:10.281 7f06767fc700 1 -- 172.21.2.112:0/2815244610 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5659+0+0 (secure 0 0 0) 0x7f0670032820 con 0x7f06800ee8a0 2021-02-10T11:42:10.282 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:10.281 7f0685624700 1 --2- 172.21.2.112:0/2815244610 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0668018730 0x7f066801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:10.282 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:10.281 7f0685624700 1 --2- 172.21.2.112:0/2815244610 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0668018730 0x7f066801ac10 crc :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:10.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:10.311 7f0688009700 1 -- 172.21.2.112:0/2815244610 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06800f0b00 con 0x7f06800ee8a0 2021-02-10T11:42:10.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:10.314 7f06767fc700 1 -- 172.21.2.112:0/2815244610 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f06700009b0 con 0x7f06800ee8a0 2021-02-10T11:42:10.440 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:10.441 7f0688009700 1 -- 172.21.2.112:0/2815244610 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f06800f0b00 con 0x7f06800ee8a0 2021-02-10T11:42:11.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:11.222 7f06767fc700 1 -- 172.21.2.112:0/2815244610 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v100) v1 ==== 104+0+0 (secure 0 0 0) 0x7f06700054a0 con 0x7f06800ee8a0 2021-02-10T11:42:11.262 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:11.262 7f0688009700 1 -- 172.21.2.112:0/2815244610 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0668018730 msgr2=0x7f066801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:11.262 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:11.262 7f0688009700 1 --2- 172.21.2.112:0/2815244610 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0668018730 0x7f066801ac10 crc :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:11.262 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:11.262 7f0688009700 1 -- 172.21.2.112:0/2815244610 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f06800ee8a0 msgr2=0x7f06800f02f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:11.262 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:11.262 7f0688009700 1 --2- 172.21.2.112:0/2815244610 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f06800ee8a0 0x7f06800f02f0 secure :-1 s=READY pgs=2290 cs=0 l=1 rev1=1 rx=0x7f067000b5b0 tx=0x7f067000b950).stop 2021-02-10T11:42:11.262 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:11.263 7f0688009700 1 -- 172.21.2.112:0/2815244610 shutdown_connections 2021-02-10T11:42:11.262 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:11.263 7f0688009700 1 --2- 172.21.2.112:0/2815244610 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0668018730 0x7f066801ac10 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:11.262 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:11.263 7f0688009700 1 --2- 172.21.2.112:0/2815244610 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f06800edc20 0x7f068019ffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:11.262 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:11.263 7f0688009700 1 --2- 172.21.2.112:0/2815244610 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f06801a05e0 0x7f0680187e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:11.262 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:11.263 7f0688009700 1 --2- 172.21.2.112:0/2815244610 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f06800ee8a0 0x7f06800f02f0 unknown :-1 s=CLOSED pgs=2290 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:11.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:11.263 7f0688009700 1 -- 172.21.2.112:0/2815244610 >> 172.21.2.112:0/2815244610 conn(0x7f0680094850 msgr2=0x7f06800b81a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:11.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:11.263 7f0688009700 1 -- 172.21.2.112:0/2815244610 shutdown_connections 2021-02-10T11:42:11.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:11.263 7f0688009700 1 -- 172.21.2.112:0/2815244610 wait complete. 2021-02-10T11:42:11.270 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2021-02-10T11:42:11.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:11.686 7fb280fa4700 1 Processor -- start 2021-02-10T11:42:11.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:11.686 7fb280fa4700 1 -- start start 2021-02-10T11:42:11.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:11.687 7fb280fa4700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb27c10aff0 0x7fb27c0cf750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:11.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:11.687 7fb280fa4700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb27c0fe1c0 con 0x7fb27c0d1c70 2021-02-10T11:42:11.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:11.687 7fb280fa4700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb27c1b0cb0 con 0x7fb27c0d1870 2021-02-10T11:42:11.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:11.687 7fb280fa4700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb27c109a60 con 0x7fb27c10aff0 2021-02-10T11:42:11.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:11.687 7fb27a59c700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fb27c0d1c70 0x7fb27c0cef60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:55426/0 (socket says 172.21.2.112:55426) 2021-02-10T11:42:11.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:11.687 7fb27a59c700 1 -- 172.21.2.112:0/3192388572 learned_addr learned my addr 172.21.2.112:0/3192388572 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:42:11.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:11.687 7fb279d9b700 1 -- 172.21.2.112:0/3192388572 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3908440810 0 0) 0x7fb2640009b0 con 0x7fb27c0d1c70 2021-02-10T11:42:11.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:11.687 7fb279d9b700 1 -- 172.21.2.112:0/3192388572 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb268002fe0 con 0x7fb27c0d1c70 2021-02-10T11:42:11.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:11.688 7fb27b59e700 1 --2- 172.21.2.112:0/3192388572 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb27c10aff0 0x7fb27c0cf750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:11.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:11.688 7fb279d9b700 1 -- 172.21.2.112:0/3192388572 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3921614709 0 0) 0x7fb264001010 con 0x7fb27c0d1c70 2021-02-10T11:42:11.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:11.688 7fb279d9b700 1 -- 172.21.2.112:0/3192388572 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb27c10aff0 msgr2=0x7fb27c0cf750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:11.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:11.688 7fb279d9b700 1 --2- 172.21.2.112:0/3192388572 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb27c10aff0 0x7fb27c0cf750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:11.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:11.688 7fb279d9b700 1 -- 172.21.2.112:0/3192388572 >> v1:172.21.2.120:6789/0 conn(0x7fb27c0d1870 legacy=0x7fb27c1b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:11.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:11.688 7fb279d9b700 1 -- 172.21.2.112:0/3192388572 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb27c10b3f0 con 0x7fb27c0d1c70 2021-02-10T11:42:11.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:11.688 7fb279d9b700 1 -- 172.21.2.112:0/3192388572 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7fb264001590 con 0x7fb27c0d1c70 2021-02-10T11:42:11.897 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:11.688 7fb279d9b700 1 -- 172.21.2.112:0/3192388572 >> v1:172.21.2.112:6789/0 conn(0x7fb27c0d1c70 legacy=0x7fb27c0cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:11.897 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:11.688 7fb279d9b700 1 --2- 172.21.2.112:0/3192388572 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb268003a40 0x7fb268005f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:11.897 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:11.688 7fb279d9b700 1 --2- 172.21.2.112:0/3192388572 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb268006780 0x7fb268008c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:11.897 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:11.688 7fb279d9b700 1 --2- 172.21.2.112:0/3192388572 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb2680094f0 0x7fb26800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:11.897 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:11.688 7fb279d9b700 1 -- 172.21.2.112:0/3192388572 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb26800c2c0 con 0x7fb2680094f0 2021-02-10T11:42:11.897 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:11.688 7fb279d9b700 1 -- 172.21.2.112:0/3192388572 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb26800c4e0 con 0x7fb268003a40 2021-02-10T11:42:11.897 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:11.688 7fb279d9b700 1 -- 172.21.2.112:0/3192388572 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb26800c700 con 0x7fb268006780 2021-02-10T11:42:11.897 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:11.688 7fb27b59e700 1 --2- 172.21.2.112:0/3192388572 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb2680094f0 0x7fb26800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:11.897 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:11.688 7fb27ad9d700 1 --2- 172.21.2.112:0/3192388572 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb268003a40 0x7fb268005f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:11.897 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:11.688 7fb27a59c700 1 --2- 172.21.2.112:0/3192388572 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb268006780 0x7fb268008c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:11.897 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:11.688 7fb27b59e700 1 -- 172.21.2.112:0/3192388572 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb268006780 msgr2=0x7fb268008c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:11.897 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:11.688 7fb27b59e700 1 --2- 172.21.2.112:0/3192388572 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb268006780 0x7fb268008c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:11.897 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:11.688 7fb27b59e700 1 -- 172.21.2.112:0/3192388572 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb268003a40 msgr2=0x7fb268005f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:11.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:11.688 7fb27b59e700 1 --2- 172.21.2.112:0/3192388572 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb268003a40 0x7fb268005f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:11.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:11.688 7fb27b59e700 1 -- 172.21.2.112:0/3192388572 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb26800c9a0 con 0x7fb2680094f0 2021-02-10T11:42:11.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:11.689 7fb27b59e700 1 --2- 172.21.2.112:0/3192388572 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb2680094f0 0x7fb26800ba10 secure :-1 s=READY pgs=2292 cs=0 l=1 rev1=1 rx=0x7fb274006e50 tx=0x7fb274009e70).ready entity=mon.0 client_cookie=527302a42f9a0a93 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:11.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:11.689 7fb279d9b700 1 -- 172.21.2.112:0/3192388572 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb274006a20 con 0x7fb2680094f0 2021-02-10T11:42:11.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:11.689 7fb279d9b700 1 -- 172.21.2.112:0/3192388572 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb274003030 con 0x7fb2680094f0 2021-02-10T11:42:11.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:11.689 7fb279d9b700 1 -- 172.21.2.112:0/3192388572 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb27400b5b0 con 0x7fb2680094f0 2021-02-10T11:42:11.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:11.689 7fb280fa4700 1 -- 172.21.2.112:0/3192388572 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb2680094f0 msgr2=0x7fb26800ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:11.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:11.689 7fb280fa4700 1 --2- 172.21.2.112:0/3192388572 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb2680094f0 0x7fb26800ba10 secure :-1 s=READY pgs=2292 cs=0 l=1 rev1=1 rx=0x7fb274006e50 tx=0x7fb274009e70).stop 2021-02-10T11:42:11.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:11.689 7fb27b59e700 1 -- 172.21.2.112:0/3192388572 reap_dead start 2021-02-10T11:42:11.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:11.689 7fb280fa4700 1 -- 172.21.2.112:0/3192388572 shutdown_connections 2021-02-10T11:42:11.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:11.689 7fb280fa4700 1 -- 172.21.2.112:0/3192388572 >> 172.21.2.112:0/3192388572 conn(0x7fb27c0ce420 msgr2=0x7fb27c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:11.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:11.689 7fb280fa4700 1 -- 172.21.2.112:0/3192388572 shutdown_connections 2021-02-10T11:42:11.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:11.689 7fb280fa4700 1 -- 172.21.2.112:0/3192388572 wait complete. 2021-02-10T11:42:11.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:11.704 7fb280fa4700 1 Processor -- start 2021-02-10T11:42:11.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:11.704 7fb280fa4700 1 -- start start 2021-02-10T11:42:11.900 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:11.704 7fb280fa4700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb27c1a05f0 0x7fb27c0a8f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:11.901 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:11.704 7fb280fa4700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb27c0aba50 0x7fb27c0b3090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:11.901 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:11.704 7fb280fa4700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb27c0b36b0 0x7fb27c0b42a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:11.901 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:11.704 7fb280fa4700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb27c0b3d20 con 0x7fb27c0aba50 2021-02-10T11:42:11.901 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:11.704 7fb280fa4700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb27c0abe50 con 0x7fb27c1a05f0 2021-02-10T11:42:11.901 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:11.704 7fb280fa4700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb27c0b23c0 con 0x7fb27c0b36b0 2021-02-10T11:42:11.901 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:11.705 7fb27a59c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb27c0aba50 0x7fb27c0b3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:11.901 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:11.705 7fb27a59c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb27c0aba50 0x7fb27c0b3090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:55976/0 (socket says 172.21.2.112:55976) 2021-02-10T11:42:11.901 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:11.705 7fb27a59c700 1 -- 172.21.2.112:0/3211305694 learned_addr learned my addr 172.21.2.112:0/3211305694 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:42:11.901 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:11.705 7fb27b59e700 1 --2- 172.21.2.112:0/3211305694 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb27c0b36b0 0x7fb27c0b42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:11.901 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:11.705 7fb27ad9d700 1 --2- 172.21.2.112:0/3211305694 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb27c1a05f0 0x7fb27c0a8f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:11.901 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:11.705 7fb27a59c700 1 -- 172.21.2.112:0/3211305694 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb27c0b36b0 msgr2=0x7fb27c0b42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:11.902 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:11.705 7fb27a59c700 1 --2- 172.21.2.112:0/3211305694 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb27c0b36b0 0x7fb27c0b42a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:11.902 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:11.705 7fb27a59c700 1 -- 172.21.2.112:0/3211305694 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb27c1a05f0 msgr2=0x7fb27c0a8f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:11.902 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:11.705 7fb27a59c700 1 --2- 172.21.2.112:0/3211305694 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb27c1a05f0 0x7fb27c0a8f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:11.902 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:11.705 7fb27a59c700 1 -- 172.21.2.112:0/3211305694 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb27c0b48c0 con 0x7fb27c0aba50 2021-02-10T11:42:11.902 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:11.705 7fb27ad9d700 1 --2- 172.21.2.112:0/3211305694 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb27c1a05f0 0x7fb27c0a8f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:42:11.902 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:11.705 7fb27a59c700 1 --2- 172.21.2.112:0/3211305694 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb27c0aba50 0x7fb27c0b3090 secure :-1 s=READY pgs=2293 cs=0 l=1 rev1=1 rx=0x7fb264007650 tx=0x7fb26400a880).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:11.902 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:11.705 7fb2737fe700 1 -- 172.21.2.112:0/3211305694 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb26400ac70 con 0x7fb27c0aba50 2021-02-10T11:42:11.902 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:11.705 7fb2737fe700 1 -- 172.21.2.112:0/3211305694 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2640016a0 con 0x7fb27c0aba50 2021-02-10T11:42:11.902 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:11.705 7fb2737fe700 1 -- 172.21.2.112:0/3211305694 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb2640043c0 con 0x7fb27c0aba50 2021-02-10T11:42:11.902 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:11.705 7fb280fa4700 1 -- 172.21.2.112:0/3211305694 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb27c0b48c0 con 0x7fb27c0aba50 2021-02-10T11:42:11.902 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:11.705 7fb280fa4700 1 -- 172.21.2.112:0/3211305694 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb27c0b48c0 con 0x7fb27c0aba50 2021-02-10T11:42:11.902 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:11.706 7fb2737fe700 1 -- 172.21.2.112:0/3211305694 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb264004140 con 0x7fb27c0aba50 2021-02-10T11:42:11.902 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:11.706 7fb2737fe700 1 --2- 172.21.2.112:0/3211305694 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb2600186c0 0x7fb26001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:11.903 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:11.706 7fb2737fe700 1 -- 172.21.2.112:0/3211305694 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5659+0+0 (secure 0 0 0) 0x7fb2640325d0 con 0x7fb27c0aba50 2021-02-10T11:42:11.903 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:11.707 7fb27ad9d700 1 --2- 172.21.2.112:0/3211305694 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb2600186c0 0x7fb26001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:11.903 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:11.707 7fb27ad9d700 1 --2- 172.21.2.112:0/3211305694 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb2600186c0 0x7fb26001abc0 crc :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:11.903 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:11.737 7fb280fa4700 1 -- 172.21.2.112:0/3211305694 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb27c0825d0 con 0x7fb27c0aba50 2021-02-10T11:42:11.903 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:11.739 7fb2737fe700 1 -- 172.21.2.112:0/3211305694 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb264001180 con 0x7fb27c0aba50 2021-02-10T11:42:11.903 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:11.867 7fb280fa4700 1 -- 172.21.2.112:0/3211305694 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fb27c0825d0 con 0x7fb27c0aba50 2021-02-10T11:42:12.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:12.227 7fb2737fe700 1 -- 172.21.2.112:0/3211305694 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v119) v1 ==== 185+0+0 (secure 0 0 0) 0x7fb26401f4f0 con 0x7fb27c0aba50 2021-02-10T11:42:12.231 INFO:teuthology.orchestra.run.gibba012.stderr:pool 'cephfs_metadata' removed 2021-02-10T11:42:12.233 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:12.235 7fb280fa4700 1 -- 172.21.2.112:0/3211305694 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb2600186c0 msgr2=0x7fb26001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:12.233 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:12.235 7fb280fa4700 1 --2- 172.21.2.112:0/3211305694 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb2600186c0 0x7fb26001abc0 crc :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:12.233 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:12.235 7fb280fa4700 1 -- 172.21.2.112:0/3211305694 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb27c0aba50 msgr2=0x7fb27c0b3090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:12.233 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:12.235 7fb280fa4700 1 --2- 172.21.2.112:0/3211305694 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb27c0aba50 0x7fb27c0b3090 secure :-1 s=READY pgs=2293 cs=0 l=1 rev1=1 rx=0x7fb264007650 tx=0x7fb26400a880).stop 2021-02-10T11:42:12.234 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:12.235 7fb280fa4700 1 -- 172.21.2.112:0/3211305694 shutdown_connections 2021-02-10T11:42:12.234 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:12.235 7fb280fa4700 1 --2- 172.21.2.112:0/3211305694 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb2600186c0 0x7fb26001abc0 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:12.234 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:12.235 7fb280fa4700 1 --2- 172.21.2.112:0/3211305694 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb27c0b36b0 0x7fb27c0b42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:12.234 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:12.235 7fb280fa4700 1 --2- 172.21.2.112:0/3211305694 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb27c0aba50 0x7fb27c0b3090 unknown :-1 s=CLOSED pgs=2293 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:12.234 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:12.235 7fb280fa4700 1 --2- 172.21.2.112:0/3211305694 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb27c1a05f0 0x7fb27c0a8f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:12.234 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:12.235 7fb280fa4700 1 -- 172.21.2.112:0/3211305694 >> 172.21.2.112:0/3211305694 conn(0x7fb27c0bc070 msgr2=0x7fb27c0ba770 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:12.234 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:12.235 7fb280fa4700 1 -- 172.21.2.112:0/3211305694 shutdown_connections 2021-02-10T11:42:12.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:12.235 7fb280fa4700 1 -- 172.21.2.112:0/3211305694 wait complete. 2021-02-10T11:42:12.242 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2021-02-10T11:42:12.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:12.659 7fda43777700 1 Processor -- start 2021-02-10T11:42:12.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:12.659 7fda43777700 1 -- start start 2021-02-10T11:42:12.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:12.660 7fda43777700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fda3c10b000 0x7fda3c0cf760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:12.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:12.660 7fda43777700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda3c0fe1d0 con 0x7fda3c0d1880 2021-02-10T11:42:12.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:12.660 7fda43777700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda3c1b0cc0 con 0x7fda3c0d1c80 2021-02-10T11:42:12.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:12.660 7fda43777700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda3c109a70 con 0x7fda3c10b000 2021-02-10T11:42:12.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:12.660 7fda41593700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fda3c0d1880 0x7fda3c1b04d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:55442/0 (socket says 172.21.2.112:55442) 2021-02-10T11:42:12.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:12.660 7fda41593700 1 -- 172.21.2.112:0/4079874456 learned_addr learned my addr 172.21.2.112:0/4079874456 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:42:12.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:12.660 7fda2ffff700 1 -- 172.21.2.112:0/4079874456 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 675121594 0 0) 0x7fda30001180 con 0x7fda3c0d1880 2021-02-10T11:42:12.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:12.660 7fda2ffff700 1 -- 172.21.2.112:0/4079874456 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda28002fe0 con 0x7fda3c0d1880 2021-02-10T11:42:12.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:12.660 7fda41d94700 1 --2- 172.21.2.112:0/4079874456 >> v2:172.21.2.120:3300/0 conn(0x7fda3c10b000 0x7fda3c0cf760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:12.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:12.661 7fda2ffff700 1 -- 172.21.2.112:0/4079874456 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2014489507 0 0) 0x7fda30001180 con 0x7fda3c0d1880 2021-02-10T11:42:12.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:12.661 7fda2ffff700 1 -- 172.21.2.112:0/4079874456 >> v2:172.21.2.120:3300/0 conn(0x7fda3c10b000 msgr2=0x7fda3c0cf760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:12.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:12.661 7fda2ffff700 1 --2- 172.21.2.112:0/4079874456 >> v2:172.21.2.120:3300/0 conn(0x7fda3c10b000 0x7fda3c0cf760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:12.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:12.661 7fda2ffff700 1 -- 172.21.2.112:0/4079874456 >> v1:172.21.2.120:6789/0 conn(0x7fda3c0d1c80 legacy=0x7fda3c0cef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:12.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:12.661 7fda2ffff700 1 -- 172.21.2.112:0/4079874456 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda3c10b400 con 0x7fda3c0d1880 2021-02-10T11:42:12.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:12.661 7fda2ffff700 1 -- 172.21.2.112:0/4079874456 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7fda30001680 con 0x7fda3c0d1880 2021-02-10T11:42:12.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:12.661 7fda2ffff700 1 -- 172.21.2.112:0/4079874456 >> v1:172.21.2.112:6789/0 conn(0x7fda3c0d1880 legacy=0x7fda3c1b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:12.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:12.661 7fda2ffff700 1 --2- 172.21.2.112:0/4079874456 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fda28003aa0 0x7fda28005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:12.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:12.661 7fda2ffff700 1 --2- 172.21.2.112:0/4079874456 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fda28006790 0x7fda28008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:12.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:12.661 7fda2ffff700 1 --2- 172.21.2.112:0/4079874456 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fda28009590 0x7fda2800bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:12.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:12.661 7fda2ffff700 1 -- 172.21.2.112:0/4079874456 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda2800c300 con 0x7fda28003aa0 2021-02-10T11:42:12.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:12.661 7fda2ffff700 1 -- 172.21.2.112:0/4079874456 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda2800c520 con 0x7fda28006790 2021-02-10T11:42:12.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:12.661 7fda2ffff700 1 -- 172.21.2.112:0/4079874456 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda2800c740 con 0x7fda28009590 2021-02-10T11:42:12.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:12.661 7fda41593700 1 --2- 172.21.2.112:0/4079874456 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fda28003aa0 0x7fda28005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:12.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:12.661 7fda40d92700 1 --2- 172.21.2.112:0/4079874456 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fda28006790 0x7fda28008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:12.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:12.661 7fda41d94700 1 --2- 172.21.2.112:0/4079874456 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fda28009590 0x7fda2800bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:12.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:12.661 7fda41593700 1 -- 172.21.2.112:0/4079874456 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fda28009590 msgr2=0x7fda2800bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:12.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:12.661 7fda41593700 1 --2- 172.21.2.112:0/4079874456 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fda28009590 0x7fda2800bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:12.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:12.661 7fda41593700 1 -- 172.21.2.112:0/4079874456 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fda28006790 msgr2=0x7fda28008c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:12.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:12.661 7fda41593700 1 --2- 172.21.2.112:0/4079874456 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fda28006790 0x7fda28008c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:12.678 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:12.661 7fda41d94700 1 -- 172.21.2.112:0/4079874456 reap_dead start 2021-02-10T11:42:12.678 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:12.661 7fda41593700 1 -- 172.21.2.112:0/4079874456 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda28050980 con 0x7fda28003aa0 2021-02-10T11:42:12.678 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:12.661 7fda41593700 1 --2- 172.21.2.112:0/4079874456 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fda28003aa0 0x7fda28005ef0 secure :-1 s=READY pgs=2295 cs=0 l=1 rev1=1 rx=0x7fda300014e0 tx=0x7fda30007c70).ready entity=mon.0 client_cookie=666da156b54197d0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:12.678 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:12.661 7fda2ffff700 1 -- 172.21.2.112:0/4079874456 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fda300070c0 con 0x7fda28003aa0 2021-02-10T11:42:12.678 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:12.661 7fda2ffff700 1 -- 172.21.2.112:0/4079874456 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda300043a0 con 0x7fda28003aa0 2021-02-10T11:42:12.678 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:12.662 7fda2ffff700 1 -- 172.21.2.112:0/4079874456 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fda300048c0 con 0x7fda28003aa0 2021-02-10T11:42:12.678 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:12.662 7fda43777700 1 -- 172.21.2.112:0/4079874456 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fda28003aa0 msgr2=0x7fda28005ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:12.678 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:12.662 7fda43777700 1 --2- 172.21.2.112:0/4079874456 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fda28003aa0 0x7fda28005ef0 secure :-1 s=READY pgs=2295 cs=0 l=1 rev1=1 rx=0x7fda300014e0 tx=0x7fda30007c70).stop 2021-02-10T11:42:12.678 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:12.662 7fda43777700 1 -- 172.21.2.112:0/4079874456 shutdown_connections 2021-02-10T11:42:12.678 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:12.662 7fda43777700 1 --2- 172.21.2.112:0/4079874456 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fda28003aa0 0x7fda28005ef0 unknown :-1 s=CLOSED pgs=2295 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:12.678 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:12.662 7fda43777700 1 -- 172.21.2.112:0/4079874456 >> 172.21.2.112:0/4079874456 conn(0x7fda3c0ce430 msgr2=0x7fda3c0d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:12.678 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:12.662 7fda43777700 1 -- 172.21.2.112:0/4079874456 shutdown_connections 2021-02-10T11:42:12.678 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:12.662 7fda43777700 1 -- 172.21.2.112:0/4079874456 wait complete. 2021-02-10T11:42:12.679 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:12.677 7fda43777700 1 Processor -- start 2021-02-10T11:42:12.679 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:12.677 7fda43777700 1 -- start start 2021-02-10T11:42:12.680 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:12.677 7fda43777700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fda3c1a3570 0x7fda3c1a3970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:12.680 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:12.677 7fda43777700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fda3c09c9f0 0x7fda3c09cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:12.680 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:12.677 7fda43777700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fda3c09d410 0x7fda3c098170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:12.680 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:12.677 7fda43777700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda3c09c470 con 0x7fda3c09d410 2021-02-10T11:42:12.680 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:12.677 7fda43777700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda3c0b2840 con 0x7fda3c09c9f0 2021-02-10T11:42:12.680 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:12.677 7fda43777700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda3c0b2a20 con 0x7fda3c1a3570 2021-02-10T11:42:12.680 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:12.677 7fda41d94700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fda3c09d410 0x7fda3c098170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:12.681 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:12.677 7fda41d94700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fda3c09d410 0x7fda3c098170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:55996/0 (socket says 172.21.2.112:55996) 2021-02-10T11:42:12.681 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:12.677 7fda41d94700 1 -- 172.21.2.112:0/1661941996 learned_addr learned my addr 172.21.2.112:0/1661941996 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:42:12.681 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:12.678 7fda41d94700 1 -- 172.21.2.112:0/1661941996 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fda3c1a3570 msgr2=0x7fda3c1a3970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:12.681 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:12.678 7fda41d94700 1 --2- 172.21.2.112:0/1661941996 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fda3c1a3570 0x7fda3c1a3970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:12.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:12.678 7fda41d94700 1 -- 172.21.2.112:0/1661941996 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fda3c09c9f0 msgr2=0x7fda3c09cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:12.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:12.678 7fda41d94700 1 --2- 172.21.2.112:0/1661941996 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fda3c09c9f0 0x7fda3c09cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:12.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:12.678 7fda41d94700 1 -- 172.21.2.112:0/1661941996 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda3c09d880 con 0x7fda3c09d410 2021-02-10T11:42:12.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:12.678 7fda41d94700 1 --2- 172.21.2.112:0/1661941996 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fda3c09d410 0x7fda3c098170 secure :-1 s=READY pgs=2296 cs=0 l=1 rev1=1 rx=0x7fda3400b910 tx=0x7fda3400bcb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:12.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:12.678 7fda2dffb700 1 -- 172.21.2.112:0/1661941996 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fda3400c2f0 con 0x7fda3c09d410 2021-02-10T11:42:12.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:12.678 7fda2dffb700 1 -- 172.21.2.112:0/1661941996 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda34006d20 con 0x7fda3c09d410 2021-02-10T11:42:12.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:12.678 7fda2dffb700 1 -- 172.21.2.112:0/1661941996 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fda3400c720 con 0x7fda3c09d410 2021-02-10T11:42:12.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:12.678 7fda43777700 1 -- 172.21.2.112:0/1661941996 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda3c09d880 con 0x7fda3c09d410 2021-02-10T11:42:12.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:12.678 7fda43777700 1 -- 172.21.2.112:0/1661941996 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda3c098a80 con 0x7fda3c09d410 2021-02-10T11:42:12.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:12.679 7fda2dffb700 1 -- 172.21.2.112:0/1661941996 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fda340011f0 con 0x7fda3c09d410 2021-02-10T11:42:12.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:12.679 7fda2dffb700 1 --2- 172.21.2.112:0/1661941996 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fda1c0186c0 0x7fda1c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:12.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:12.679 7fda2dffb700 1 -- 172.21.2.112:0/1661941996 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 5221+0+0 (secure 0 0 0) 0x7fda340327e0 con 0x7fda3c09d410 2021-02-10T11:42:12.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:12.679 7fda41593700 1 --2- 172.21.2.112:0/1661941996 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fda1c0186c0 0x7fda1c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:12.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:12.680 7fda41593700 1 --2- 172.21.2.112:0/1661941996 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fda1c0186c0 0x7fda1c01abc0 crc :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:12.708 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:12.710 7fda43777700 1 -- 172.21.2.112:0/1661941996 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda3c082a70 con 0x7fda3c09d410 2021-02-10T11:42:12.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:12.713 7fda2dffb700 1 -- 172.21.2.112:0/1661941996 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fda3401f8c0 con 0x7fda3c09d410 2021-02-10T11:42:12.839 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:12.840 7fda43777700 1 -- 172.21.2.112:0/1661941996 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fda3c082a70 con 0x7fda3c09d410 2021-02-10T11:42:13.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:13.233 7fda2dffb700 1 -- 172.21.2.112:0/1661941996 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v120) v1 ==== 173+0+0 (secure 0 0 0) 0x7fda340011f0 con 0x7fda3c09d410 2021-02-10T11:42:13.255 INFO:teuthology.orchestra.run.gibba012.stderr:pool 'cephfs_data' removed 2021-02-10T11:42:13.257 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:13.258 7fda43777700 1 -- 172.21.2.112:0/1661941996 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fda1c0186c0 msgr2=0x7fda1c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:13.257 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:13.258 7fda43777700 1 --2- 172.21.2.112:0/1661941996 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fda1c0186c0 0x7fda1c01abc0 crc :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:13.257 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:13.258 7fda43777700 1 -- 172.21.2.112:0/1661941996 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fda3c09d410 msgr2=0x7fda3c098170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:13.257 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:13.258 7fda43777700 1 --2- 172.21.2.112:0/1661941996 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fda3c09d410 0x7fda3c098170 secure :-1 s=READY pgs=2296 cs=0 l=1 rev1=1 rx=0x7fda3400b910 tx=0x7fda3400bcb0).stop 2021-02-10T11:42:13.258 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:13.259 7fda43777700 1 -- 172.21.2.112:0/1661941996 shutdown_connections 2021-02-10T11:42:13.258 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:13.259 7fda43777700 1 --2- 172.21.2.112:0/1661941996 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fda1c0186c0 0x7fda1c01abc0 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:13.258 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:13.259 7fda43777700 1 --2- 172.21.2.112:0/1661941996 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fda3c09c9f0 0x7fda3c09cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:13.258 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:13.259 7fda43777700 1 --2- 172.21.2.112:0/1661941996 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fda3c09d410 0x7fda3c098170 unknown :-1 s=CLOSED pgs=2296 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:13.258 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:13.259 7fda43777700 1 --2- 172.21.2.112:0/1661941996 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fda3c1a3570 0x7fda3c1a3970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:13.258 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:13.259 7fda43777700 1 -- 172.21.2.112:0/1661941996 >> 172.21.2.112:0/1661941996 conn(0x7fda3c0c2cd0 msgr2=0x7fda3c0b5950 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:13.258 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:13.259 7fda43777700 1 -- 172.21.2.112:0/1661941996 shutdown_connections 2021-02-10T11:42:13.258 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:13.259 7fda43777700 1 -- 172.21.2.112:0/1661941996 wait complete. 2021-02-10T11:42:13.266 INFO:tasks.ceph.mds.a:Restarting daemon 2021-02-10T11:42:13.266 INFO:tasks.ceph.mds.a:Stopping old one... 2021-02-10T11:42:13.266 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2021-02-10T11:42:13.266 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:42:13.267 INFO:tasks.ceph.mds.b:Restarting daemon 2021-02-10T11:42:13.267 INFO:tasks.ceph.mds.b:Stopping old one... 2021-02-10T11:42:13.267 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2021-02-10T11:42:13.267 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:42:13.272 INFO:tasks.ceph.mds.a:Stopped 2021-02-10T11:42:13.273 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2021-02-10T11:42:13.273 INFO:tasks.ceph.mds.b:Stopped 2021-02-10T11:42:13.274 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2021-02-10T11:42:13.315 INFO:tasks.ceph.mds.a:Started 2021-02-10T11:42:13.315 DEBUG:teuthology.parallel:result is None 2021-02-10T11:42:13.316 INFO:tasks.ceph.mds.b:Started 2021-02-10T11:42:13.316 DEBUG:teuthology.parallel:result is None 2021-02-10T11:42:13.316 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd blacklist clear 2021-02-10T11:42:13.719 INFO:tasks.ceph.mds.b.gibba020.stdout:starting mds.b at 2021-02-10T11:42:13.722 INFO:tasks.ceph.mds.a.gibba012.stdout:starting mds.a at 2021-02-10T11:42:13.741 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:13.726 7fdc89d45700 1 Processor -- start 2021-02-10T11:42:13.741 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:13.726 7fdc89d45700 1 -- start start 2021-02-10T11:42:13.741 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:13.726 7fdc89d45700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdc840d1870 0x7fdc841b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:13.741 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:13.727 7fdc89d45700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fdc840f0650 0x7fdc840cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:13.741 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:13.727 7fdc89d45700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc840f0b10 con 0x7fdc841b0a10 2021-02-10T11:42:13.741 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:13.727 7fdc89d45700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc840fe230 con 0x7fdc840f0650 2021-02-10T11:42:13.742 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:13.727 7fdc89d45700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc840e5ed0 con 0x7fdc840d1870 2021-02-10T11:42:13.742 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:13.727 7fdc83fff700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fdc840f0650 0x7fdc840cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:13.742 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:13.727 7fdc83fff700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fdc840f0650 0x7fdc840cfb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3300/0 says I am v2:172.21.2.112:58296/0 (socket says 172.21.2.112:58296) 2021-02-10T11:42:13.742 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:13.727 7fdc83fff700 1 -- 172.21.2.112:0/1405127158 learned_addr learned my addr 172.21.2.112:0/1405127158 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:42:13.742 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:13.727 7fdc837fe700 1 --2- 172.21.2.112:0/1405127158 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdc840d1870 0x7fdc841b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:13.742 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:13.727 7fdc83fff700 1 -- 172.21.2.112:0/1405127158 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdc840d1870 msgr2=0x7fdc841b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:13.742 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:13.727 7fdc83fff700 1 --2- 172.21.2.112:0/1405127158 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdc840d1870 0x7fdc841b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:13.742 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:13.727 7fdc83fff700 1 -- 172.21.2.112:0/1405127158 >> v1:172.21.2.120:6789/0 conn(0x7fdc841b0a10 legacy=0x7fdc840cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:13.742 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:13.727 7fdc83fff700 1 -- 172.21.2.112:0/1405127158 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc840d2fe0 con 0x7fdc840f0650 2021-02-10T11:42:13.742 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:13.727 7fdc83fff700 1 --2- 172.21.2.112:0/1405127158 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fdc840f0650 0x7fdc840cfb20 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 rx=0x7fdc7c006720 tx=0x7fdc7c009ba0).ready entity=mon.1 client_cookie=c0cf34d4c3f15930 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:13.742 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:13.727 7fdc827fc700 1 -- 172.21.2.112:0/1405127158 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdc7c0022e0 con 0x7fdc840f0650 2021-02-10T11:42:13.742 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:13.728 7fdc827fc700 1 -- 172.21.2.112:0/1405127158 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc7c002660 con 0x7fdc840f0650 2021-02-10T11:42:13.743 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:13.728 7fdc827fc700 1 -- 172.21.2.112:0/1405127158 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdc7c000c10 con 0x7fdc840f0650 2021-02-10T11:42:13.743 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:13.728 7fdc89d45700 1 -- 172.21.2.112:0/1405127158 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fdc840f0650 msgr2=0x7fdc840cfb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:13.743 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:13.728 7fdc89d45700 1 --2- 172.21.2.112:0/1405127158 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fdc840f0650 0x7fdc840cfb20 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 rx=0x7fdc7c006720 tx=0x7fdc7c009ba0).stop 2021-02-10T11:42:13.743 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:13.728 7fdc89d45700 1 -- 172.21.2.112:0/1405127158 shutdown_connections 2021-02-10T11:42:13.743 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:13.728 7fdc89d45700 1 --2- 172.21.2.112:0/1405127158 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fdc840f0650 0x7fdc840cfb20 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:13.743 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:13.728 7fdc89d45700 1 --2- 172.21.2.112:0/1405127158 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdc840d1870 0x7fdc841b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:13.743 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:13.728 7fdc89d45700 1 -- 172.21.2.112:0/1405127158 >> 172.21.2.112:0/1405127158 conn(0x7fdc840ce450 msgr2=0x7fdc840d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:13.743 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:13.728 7fdc89d45700 1 -- 172.21.2.112:0/1405127158 shutdown_connections 2021-02-10T11:42:13.743 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:13.728 7fdc89d45700 1 -- 172.21.2.112:0/1405127158 wait complete. 2021-02-10T11:42:13.745 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:13.743 7fdc89d45700 1 Processor -- start 2021-02-10T11:42:13.745 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:13.743 7fdc89d45700 1 -- start start 2021-02-10T11:42:13.745 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:13.743 7fdc89d45700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdc840bb770 0x7fdc840a23c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:13.745 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:13.744 7fdc89d45700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdc840b2c40 0x7fdc840a29e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:13.745 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:13.744 7fdc89d45700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fdc840b1420 0x7fdc840b1820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:13.745 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:13.744 7fdc89d45700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc840b26c0 con 0x7fdc840bb770 2021-02-10T11:42:13.745 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:13.744 7fdc89d45700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc84099c90 con 0x7fdc840b1420 2021-02-10T11:42:13.745 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:13.744 7fdc89d45700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc840a3000 con 0x7fdc840b2c40 2021-02-10T11:42:13.746 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:13.744 7fdc837fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdc840bb770 0x7fdc840a23c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:13.746 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:13.744 7fdc837fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdc840bb770 0x7fdc840a23c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:56022/0 (socket says 172.21.2.112:56022) 2021-02-10T11:42:13.746 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:13.744 7fdc837fe700 1 -- 172.21.2.112:0/3605121233 learned_addr learned my addr 172.21.2.112:0/3605121233 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:42:13.746 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:13.744 7fdc83fff700 1 --2- 172.21.2.112:0/3605121233 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fdc840b1420 0x7fdc840b1820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:13.746 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:13.744 7fdc82ffd700 1 --2- 172.21.2.112:0/3605121233 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdc840b2c40 0x7fdc840a29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:13.746 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:13.744 7fdc837fe700 1 -- 172.21.2.112:0/3605121233 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdc840b2c40 msgr2=0x7fdc840a29e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:13.746 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:13.744 7fdc837fe700 1 --2- 172.21.2.112:0/3605121233 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdc840b2c40 0x7fdc840a29e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:13.746 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:13.744 7fdc837fe700 1 -- 172.21.2.112:0/3605121233 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fdc840b1420 msgr2=0x7fdc840b1820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:13.746 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:13.744 7fdc837fe700 1 --2- 172.21.2.112:0/3605121233 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fdc840b1420 0x7fdc840b1820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:13.746 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:13.744 7fdc837fe700 1 -- 172.21.2.112:0/3605121233 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc840b1e40 con 0x7fdc840bb770 2021-02-10T11:42:13.746 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:13.744 7fdc837fe700 1 --2- 172.21.2.112:0/3605121233 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdc840bb770 0x7fdc840a23c0 secure :-1 s=READY pgs=2301 cs=0 l=1 rev1=1 rx=0x7fdc740099e0 tx=0x7fdc7400bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:13.746 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:13.744 7fdc6bfff700 1 -- 172.21.2.112:0/3605121233 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdc7400c2f0 con 0x7fdc840bb770 2021-02-10T11:42:13.747 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:13.744 7fdc6bfff700 1 -- 172.21.2.112:0/3605121233 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc7400c5f0 con 0x7fdc840bb770 2021-02-10T11:42:13.747 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:13.744 7fdc6bfff700 1 -- 172.21.2.112:0/3605121233 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdc7400ca50 con 0x7fdc840bb770 2021-02-10T11:42:13.747 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:13.744 7fdc89d45700 1 -- 172.21.2.112:0/3605121233 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc840b1e40 con 0x7fdc840bb770 2021-02-10T11:42:13.747 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:13.744 7fdc89d45700 1 -- 172.21.2.112:0/3605121233 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc840b1e40 con 0x7fdc840bb770 2021-02-10T11:42:13.747 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:13.745 7fdc6bfff700 1 -- 172.21.2.112:0/3605121233 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fdc74009fe0 con 0x7fdc840bb770 2021-02-10T11:42:13.747 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:13.745 7fdc6bfff700 1 --2- 172.21.2.112:0/3605121233 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fdc640186c0 0x7fdc6401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:13.747 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:13.745 7fdc6bfff700 1 -- 172.21.2.112:0/3605121233 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 4839+0+0 (secure 0 0 0) 0x7fdc74032720 con 0x7fdc840bb770 2021-02-10T11:42:13.747 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:13.745 7fdc82ffd700 1 --2- 172.21.2.112:0/3605121233 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fdc640186c0 0x7fdc6401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:13.747 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:13.746 7fdc82ffd700 1 --2- 172.21.2.112:0/3605121233 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fdc640186c0 0x7fdc6401abc0 crc :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:13.774 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:13.775 7fdc89d45700 1 -- 172.21.2.112:0/3605121233 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc840ebc50 con 0x7fdc840bb770 2021-02-10T11:42:13.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:13.778 7fdc6bfff700 1 -- 172.21.2.112:0/3605121233 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fdc7400c2f0 con 0x7fdc840bb770 2021-02-10T11:42:13.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:13.916 7fdc89d45700 1 -- 172.21.2.112:0/3605121233 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7fdc840ebc50 con 0x7fdc840bb770 2021-02-10T11:42:14.237 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:14.238 7fdc6bfff700 1 -- 172.21.2.112:0/3605121233 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v121) v1 ==== 97+0+0 (secure 0 0 0) 0x7fdc74009fe0 con 0x7fdc840bb770 2021-02-10T11:42:14.281 INFO:teuthology.orchestra.run.gibba012.stderr: removed all blacklist entries 2021-02-10T11:42:14.284 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:14.285 7fdc89d45700 1 -- 172.21.2.112:0/3605121233 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fdc640186c0 msgr2=0x7fdc6401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:14.284 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:14.285 7fdc89d45700 1 --2- 172.21.2.112:0/3605121233 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fdc640186c0 0x7fdc6401abc0 crc :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:14.284 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:14.285 7fdc89d45700 1 -- 172.21.2.112:0/3605121233 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdc840bb770 msgr2=0x7fdc840a23c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:14.284 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:14.285 7fdc89d45700 1 --2- 172.21.2.112:0/3605121233 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdc840bb770 0x7fdc840a23c0 secure :-1 s=READY pgs=2301 cs=0 l=1 rev1=1 rx=0x7fdc740099e0 tx=0x7fdc7400bd40).stop 2021-02-10T11:42:14.284 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:14.285 7fdc89d45700 1 -- 172.21.2.112:0/3605121233 shutdown_connections 2021-02-10T11:42:14.284 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:14.285 7fdc89d45700 1 --2- 172.21.2.112:0/3605121233 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fdc640186c0 0x7fdc6401abc0 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:14.284 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:14.285 7fdc89d45700 1 --2- 172.21.2.112:0/3605121233 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fdc840b1420 0x7fdc840b1820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:14.284 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:14.285 7fdc89d45700 1 --2- 172.21.2.112:0/3605121233 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdc840b2c40 0x7fdc840a29e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:14.284 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:14.285 7fdc89d45700 1 --2- 172.21.2.112:0/3605121233 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdc840bb770 0x7fdc840a23c0 unknown :-1 s=CLOSED pgs=2301 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:14.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:14.285 7fdc89d45700 1 -- 172.21.2.112:0/3605121233 >> 172.21.2.112:0/3605121233 conn(0x7fdc84091dd0 msgr2=0x7fdc840b3630 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:14.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:14.285 7fdc89d45700 1 -- 172.21.2.112:0/3605121233 shutdown_connections 2021-02-10T11:42:14.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:14.285 7fdc89d45700 1 -- 172.21.2.112:0/3605121233 wait complete. 2021-02-10T11:42:14.292 INFO:tasks.cephfs.cephfs_test_case:['0', '1', '2', '3'] 2021-02-10T11:42:14.292 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth ls --format=json-pretty 2021-02-10T11:42:14.718 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:14.702 7fcfa5578700 1 Processor -- start 2021-02-10T11:42:14.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:14.702 7fcfa5578700 1 -- start start 2021-02-10T11:42:14.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:14.703 7fcfa5578700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fcfa00d1870 0x7fcfa01b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:14.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:14.703 7fcfa5578700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfa00f0b10 con 0x7fcfa01b0a10 2021-02-10T11:42:14.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:14.703 7fcfa5578700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfa00d0310 con 0x7fcfa00f0650 2021-02-10T11:42:14.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:14.703 7fcfa5578700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfa00d0530 con 0x7fcfa00d1870 2021-02-10T11:42:14.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:14.703 7fcf9eabe700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fcfa01b0a10 0x7fcfa00cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:55488/0 (socket says 172.21.2.112:55488) 2021-02-10T11:42:14.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:14.703 7fcf9eabe700 1 -- 172.21.2.112:0/338079040 learned_addr learned my addr 172.21.2.112:0/338079040 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:42:14.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:14.703 7fcf9e2bd700 1 -- 172.21.2.112:0/338079040 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2531925376 0 0) 0x7fcf940010b0 con 0x7fcfa01b0a10 2021-02-10T11:42:14.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:14.703 7fcf9e2bd700 1 -- 172.21.2.112:0/338079040 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf88002fe0 con 0x7fcfa01b0a10 2021-02-10T11:42:14.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:14.703 7fcf9f2bf700 1 --2- 172.21.2.112:0/338079040 >> v2:172.21.2.120:3300/0 conn(0x7fcfa00d1870 0x7fcfa01b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:14.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:14.704 7fcf9e2bd700 1 -- 172.21.2.112:0/338079040 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1253129887 0 0) 0x7fcf940010b0 con 0x7fcfa01b0a10 2021-02-10T11:42:14.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:14.704 7fcf9e2bd700 1 -- 172.21.2.112:0/338079040 >> v2:172.21.2.120:3300/0 conn(0x7fcfa00d1870 msgr2=0x7fcfa01b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:14.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:14.704 7fcf9e2bd700 1 --2- 172.21.2.112:0/338079040 >> v2:172.21.2.120:3300/0 conn(0x7fcfa00d1870 0x7fcfa01b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:14.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:14.704 7fcf9e2bd700 1 -- 172.21.2.112:0/338079040 >> v1:172.21.2.120:6789/0 conn(0x7fcfa00f0650 legacy=0x7fcfa00cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:14.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:14.704 7fcf9e2bd700 1 -- 172.21.2.112:0/338079040 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfa00d1c70 con 0x7fcfa01b0a10 2021-02-10T11:42:14.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:14.704 7fcf9e2bd700 1 -- 172.21.2.112:0/338079040 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7fcf940015b0 con 0x7fcfa01b0a10 2021-02-10T11:42:14.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:14.704 7fcf9e2bd700 1 -- 172.21.2.112:0/338079040 >> v1:172.21.2.112:6789/0 conn(0x7fcfa01b0a10 legacy=0x7fcfa00cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:14.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:14.704 7fcf9e2bd700 1 --2- 172.21.2.112:0/338079040 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcf88003b10 0x7fcf88006010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:14.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:14.704 7fcf9e2bd700 1 --2- 172.21.2.112:0/338079040 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcf880068b0 0x7fcf88008dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:14.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:14.704 7fcf9e2bd700 1 --2- 172.21.2.112:0/338079040 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcf88009640 0x7fcf8800bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:14.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:14.704 7fcf9e2bd700 1 -- 172.21.2.112:0/338079040 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf8800c3b0 con 0x7fcf880068b0 2021-02-10T11:42:14.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:14.704 7fcf9e2bd700 1 -- 172.21.2.112:0/338079040 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf8800c5d0 con 0x7fcf88009640 2021-02-10T11:42:14.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:14.704 7fcf9e2bd700 1 -- 172.21.2.112:0/338079040 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf8800c7f0 con 0x7fcf88003b10 2021-02-10T11:42:14.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:14.704 7fcf9fac0700 1 --2- 172.21.2.112:0/338079040 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcf88009640 0x7fcf8800bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:14.721 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:14.704 7fcf9eabe700 1 --2- 172.21.2.112:0/338079040 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcf880068b0 0x7fcf88008dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:14.721 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:14.704 7fcf9f2bf700 1 --2- 172.21.2.112:0/338079040 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcf88003b10 0x7fcf88006010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:14.721 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:14.704 7fcf9eabe700 1 -- 172.21.2.112:0/338079040 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcf88003b10 msgr2=0x7fcf88006010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:14.721 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:14.704 7fcf9eabe700 1 --2- 172.21.2.112:0/338079040 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcf88003b10 0x7fcf88006010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:14.721 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:14.704 7fcf9eabe700 1 -- 172.21.2.112:0/338079040 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcf88009640 msgr2=0x7fcf8800bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:14.721 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:14.704 7fcf9eabe700 1 --2- 172.21.2.112:0/338079040 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcf88009640 0x7fcf8800bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:14.721 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:14.704 7fcf9eabe700 1 -- 172.21.2.112:0/338079040 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf88050a30 con 0x7fcf880068b0 2021-02-10T11:42:14.721 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:14.704 7fcf9fac0700 1 --2- 172.21.2.112:0/338079040 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcf88009640 0x7fcf8800bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:42:14.721 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:14.704 7fcf9fac0700 1 -- 172.21.2.112:0/338079040 reap_dead start 2021-02-10T11:42:14.721 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:14.704 7fcf9eabe700 1 --2- 172.21.2.112:0/338079040 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcf880068b0 0x7fcf88008dd0 secure :-1 s=READY pgs=2303 cs=0 l=1 rev1=1 rx=0x7fcf94001aa0 tx=0x7fcf94007d30).ready entity=mon.0 client_cookie=c3177c86e9862afb server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:14.721 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:14.704 7fcf9e2bd700 1 -- 172.21.2.112:0/338079040 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcf9400a310 con 0x7fcf880068b0 2021-02-10T11:42:14.721 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:14.704 7fcf9e2bd700 1 -- 172.21.2.112:0/338079040 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf9400a690 con 0x7fcf880068b0 2021-02-10T11:42:14.721 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:14.704 7fcf9e2bd700 1 -- 172.21.2.112:0/338079040 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcf9400ab70 con 0x7fcf880068b0 2021-02-10T11:42:14.721 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:14.704 7fcfa5578700 1 -- 172.21.2.112:0/338079040 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcf880068b0 msgr2=0x7fcf88008dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:14.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:14.704 7fcfa5578700 1 --2- 172.21.2.112:0/338079040 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcf880068b0 0x7fcf88008dd0 secure :-1 s=READY pgs=2303 cs=0 l=1 rev1=1 rx=0x7fcf94001aa0 tx=0x7fcf94007d30).stop 2021-02-10T11:42:14.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:14.705 7fcfa5578700 1 -- 172.21.2.112:0/338079040 shutdown_connections 2021-02-10T11:42:14.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:14.705 7fcfa5578700 1 --2- 172.21.2.112:0/338079040 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcf880068b0 0x7fcf88008dd0 unknown :-1 s=CLOSED pgs=2303 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:14.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:14.705 7fcfa5578700 1 -- 172.21.2.112:0/338079040 >> 172.21.2.112:0/338079040 conn(0x7fcfa00ce450 msgr2=0x7fcfa00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:14.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:14.705 7fcfa5578700 1 -- 172.21.2.112:0/338079040 shutdown_connections 2021-02-10T11:42:14.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:14.705 7fcfa5578700 1 -- 172.21.2.112:0/338079040 wait complete. 2021-02-10T11:42:14.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:14.720 7fcfa5578700 1 Processor -- start 2021-02-10T11:42:14.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:14.720 7fcfa5578700 1 -- start start 2021-02-10T11:42:14.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:14.721 7fcfa5578700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcfa018f360 0x7fcfa00a9940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:14.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:14.721 7fcfa5578700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcfa00a9f60 0x7fcfa00ecb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:14.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:14.721 7fcfa5578700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcfa00ed190 0x7fcfa00ed590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:14.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:14.721 7fcfa5578700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa018f760 con 0x7fcfa00a9f60 2021-02-10T11:42:14.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:14.721 7fcfa5578700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa018f9e0 con 0x7fcfa00ed190 2021-02-10T11:42:14.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:14.721 7fcfa5578700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa008d230 con 0x7fcfa018f360 2021-02-10T11:42:14.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:14.721 7fcf9eabe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcfa00a9f60 0x7fcfa00ecb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:14.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:14.721 7fcf9eabe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcfa00a9f60 0x7fcfa00ecb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:56040/0 (socket says 172.21.2.112:56040) 2021-02-10T11:42:14.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:14.721 7fcf9eabe700 1 -- 172.21.2.112:0/2986149691 learned_addr learned my addr 172.21.2.112:0/2986149691 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:42:14.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:14.721 7fcf9eabe700 1 -- 172.21.2.112:0/2986149691 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcfa018f360 msgr2=0x7fcfa00a9940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:14.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:14.721 7fcf9eabe700 1 --2- 172.21.2.112:0/2986149691 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcfa018f360 0x7fcfa00a9940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:14.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:14.721 7fcf9eabe700 1 -- 172.21.2.112:0/2986149691 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcfa00ed190 msgr2=0x7fcfa00ed590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:14.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:14.721 7fcf9eabe700 1 --2- 172.21.2.112:0/2986149691 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcfa00ed190 0x7fcfa00ed590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:14.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:14.721 7fcf9eabe700 1 -- 172.21.2.112:0/2986149691 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfa008d450 con 0x7fcfa00a9f60 2021-02-10T11:42:14.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:14.721 7fcf9eabe700 1 --2- 172.21.2.112:0/2986149691 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcfa00a9f60 0x7fcfa00ecb70 secure :-1 s=READY pgs=2304 cs=0 l=1 rev1=1 rx=0x7fcf94001310 tx=0x7fcf9400a8a0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:14.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:14.721 7fcf87fff700 1 -- 172.21.2.112:0/2986149691 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcf940072d0 con 0x7fcfa00a9f60 2021-02-10T11:42:14.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:14.721 7fcf87fff700 1 -- 172.21.2.112:0/2986149691 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf94004140 con 0x7fcfa00a9f60 2021-02-10T11:42:14.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:14.721 7fcf87fff700 1 -- 172.21.2.112:0/2986149691 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcf940045d0 con 0x7fcfa00a9f60 2021-02-10T11:42:14.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:14.721 7fcfa5578700 1 -- 172.21.2.112:0/2986149691 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfa008d450 con 0x7fcfa00a9f60 2021-02-10T11:42:14.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:14.722 7fcfa5578700 1 -- 172.21.2.112:0/2986149691 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfa008d450 con 0x7fcfa00a9f60 2021-02-10T11:42:14.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:14.722 7fcf87fff700 1 -- 172.21.2.112:0/2986149691 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fcf940072d0 con 0x7fcfa00a9f60 2021-02-10T11:42:14.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:14.722 7fcf87fff700 1 --2- 172.21.2.112:0/2986149691 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcf8001cad0 0x7fcf8001eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:14.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:14.722 7fcf87fff700 1 -- 172.21.2.112:0/2986149691 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 4753+0+0 (secure 0 0 0) 0x7fcf94032470 con 0x7fcfa00a9f60 2021-02-10T11:42:14.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:14.722 7fcf9f2bf700 1 --2- 172.21.2.112:0/2986149691 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcf8001cad0 0x7fcf8001eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:14.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:14.723 7fcf9f2bf700 1 --2- 172.21.2.112:0/2986149691 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcf8001cad0 0x7fcf8001eff0 crc :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:14.751 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:14.752 7fcfa5578700 1 -- 172.21.2.112:0/2986149691 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfa00ebc60 con 0x7fcfa00a9f60 2021-02-10T11:42:14.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:14.755 7fcf87fff700 1 -- 172.21.2.112:0/2986149691 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fcf940009f0 con 0x7fcfa00a9f60 2021-02-10T11:42:14.883 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:14.885 7fcfa5578700 1 -- 172.21.2.112:0/2986149691 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fcfa00ebc60 con 0x7fcfa00a9f60 2021-02-10T11:42:14.884 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:14.885 7fcf87fff700 1 -- 172.21.2.112:0/2986149691 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v74) v1 ==== 80+0+4513 (secure 0 0 0) 0x7fcf940072d0 con 0x7fcfa00a9f60 2021-02-10T11:42:14.884 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:42:14.884 INFO:teuthology.orchestra.run.gibba012.stdout:{ 2021-02-10T11:42:14.884 INFO:teuthology.orchestra.run.gibba012.stdout: "auth_dump": [ 2021-02-10T11:42:14.884 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:14.884 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "mds.a", 2021-02-10T11:42:14.884 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCKvyNgPkV4AxAAEmVNaghdccP6lWd8hvQnxw==", 2021-02-10T11:42:14.885 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:42:14.885 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:42:14.885 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:42:14.885 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:42:14.885 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:42:14.885 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:42:14.885 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:14.885 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:14.885 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "mds.b", 2021-02-10T11:42:14.885 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCKvyNguxONLhAAHIpry+R89MAwdhFJyEhVCw==", 2021-02-10T11:42:14.885 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:42:14.885 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:42:14.885 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:42:14.886 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:42:14.886 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:42:14.886 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:42:14.886 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:14.886 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:14.886 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.0", 2021-02-10T11:42:14.886 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCSvyNgXcKPLhAAP/Bj2F+06DCtPCzQDNq+cQ==", 2021-02-10T11:42:14.886 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:42:14.886 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:42:14.886 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:42:14.886 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:42:14.886 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:42:14.886 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:14.887 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:14.887 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.1", 2021-02-10T11:42:14.887 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCVvyNghfbcLhAA66PrRwR2M7o6VTF5UhU/Fg==", 2021-02-10T11:42:14.887 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:42:14.887 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:42:14.887 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:42:14.887 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:42:14.887 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:42:14.887 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:14.887 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:14.887 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.2", 2021-02-10T11:42:14.887 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCYvyNgwiPqLhAA66BVkRsgaUlGpAsQ15Nf8Q==", 2021-02-10T11:42:14.887 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:42:14.888 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:42:14.888 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:42:14.888 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:42:14.888 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:42:14.888 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:14.888 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:14.888 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.3", 2021-02-10T11:42:14.888 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCbvyNgqBZrLxAAERPe/1L3SXAEzI3d9ReHHg==", 2021-02-10T11:42:14.888 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:42:14.888 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:42:14.888 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:42:14.888 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:42:14.888 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:42:14.889 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:14.889 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:14.889 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.4", 2021-02-10T11:42:14.889 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCkvyNgDQsxIhAAHmVjVYnx20N4SDmKqvbcuA==", 2021-02-10T11:42:14.889 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:42:14.889 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:42:14.889 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:42:14.889 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:42:14.889 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:42:14.889 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:14.889 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:14.889 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.5", 2021-02-10T11:42:14.889 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCnvyNg8ITFIhAADd4aKybAV7btoGPx3kCg3A==", 2021-02-10T11:42:14.890 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:42:14.890 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:42:14.890 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:42:14.890 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:42:14.890 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:42:14.890 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:14.890 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:14.890 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.6", 2021-02-10T11:42:14.890 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCqvyNgxOhDIxAAzXixEa/nlDc1xXrg0aeLwg==", 2021-02-10T11:42:14.890 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:42:14.890 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:42:14.890 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:42:14.890 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:42:14.891 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:42:14.891 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:14.891 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:14.891 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.7", 2021-02-10T11:42:14.891 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCtvyNgxSqGIxAACxJQlwP7DpiVVCZHA3zn1A==", 2021-02-10T11:42:14.891 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:42:14.891 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:42:14.891 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:42:14.891 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:42:14.891 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:42:14.891 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:14.891 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:14.891 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.0", 2021-02-10T11:42:14.891 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCLvyNgDkNiHRAAygU8Za4PBusTqCgwAYWTiQ==", 2021-02-10T11:42:14.892 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:42:14.892 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:42:14.892 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow r", 2021-02-10T11:42:14.892 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:42:14.892 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:42:14.892 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:14.892 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:14.892 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.1", 2021-02-10T11:42:14.892 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCLvyNgxmLqMxAA1MlSDJ9LH9u4f2sL7mm8/Q==", 2021-02-10T11:42:14.892 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:42:14.892 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:42:14.892 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow r", 2021-02-10T11:42:14.892 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:42:14.893 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:42:14.893 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:14.893 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:14.893 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.2", 2021-02-10T11:42:14.893 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCMvyNgp8veDhAAniSVHYPtwwTog/pjbaRQmw==", 2021-02-10T11:42:14.893 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:42:14.893 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:42:14.893 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow r", 2021-02-10T11:42:14.893 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:42:14.893 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:42:14.893 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:14.893 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:14.893 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.3", 2021-02-10T11:42:14.894 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCMvyNgQEVoJRAApHcWdm1sldIxu+heyD4HdQ==", 2021-02-10T11:42:14.894 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:42:14.894 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:42:14.894 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow r", 2021-02-10T11:42:14.894 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:42:14.894 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:42:14.894 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:14.894 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:14.894 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.admin", 2021-02-10T11:42:14.894 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCGvyNg7jjJBRAAtot+ypUZOiTo1GbY+oVhVw==", 2021-02-10T11:42:14.894 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:42:14.894 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow *", 2021-02-10T11:42:14.895 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:42:14.895 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:42:14.895 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:42:14.895 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:42:14.895 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:14.895 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:14.895 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "mgr.x", 2021-02-10T11:42:14.895 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCJvyNgTPVwKBAAjduiBE8+RSXBv82tdPecWg==", 2021-02-10T11:42:14.895 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:42:14.895 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow *", 2021-02-10T11:42:14.895 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow profile mgr", 2021-02-10T11:42:14.895 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:42:14.896 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:42:14.896 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:14.896 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:14.896 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "mgr.y", 2021-02-10T11:42:14.896 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCJvyNgQ1tTDhAASwBssDy+Rz31ujw8fNlWow==", 2021-02-10T11:42:14.896 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:42:14.896 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow *", 2021-02-10T11:42:14.896 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow profile mgr", 2021-02-10T11:42:14.896 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:42:14.896 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:42:14.896 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:42:14.896 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:42:14.897 INFO:teuthology.orchestra.run.gibba012.stdout:} 2021-02-10T11:42:14.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:14.887 7fcfa5578700 1 -- 172.21.2.112:0/2986149691 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcf8001cad0 msgr2=0x7fcf8001eff0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:14.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:14.887 7fcfa5578700 1 --2- 172.21.2.112:0/2986149691 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcf8001cad0 0x7fcf8001eff0 crc :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:14.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:14.887 7fcfa5578700 1 -- 172.21.2.112:0/2986149691 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcfa00a9f60 msgr2=0x7fcfa00ecb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:14.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:14.887 7fcfa5578700 1 --2- 172.21.2.112:0/2986149691 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcfa00a9f60 0x7fcfa00ecb70 secure :-1 s=READY pgs=2304 cs=0 l=1 rev1=1 rx=0x7fcf94001310 tx=0x7fcf9400a8a0).stop 2021-02-10T11:42:14.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:14.887 7fcfa5578700 1 -- 172.21.2.112:0/2986149691 shutdown_connections 2021-02-10T11:42:14.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:14.887 7fcfa5578700 1 --2- 172.21.2.112:0/2986149691 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcf8001cad0 0x7fcf8001eff0 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:14.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:14.887 7fcfa5578700 1 --2- 172.21.2.112:0/2986149691 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcfa00ed190 0x7fcfa00ed590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:14.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:14.887 7fcfa5578700 1 --2- 172.21.2.112:0/2986149691 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcfa00a9f60 0x7fcfa00ecb70 unknown :-1 s=CLOSED pgs=2304 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:14.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:14.887 7fcfa5578700 1 --2- 172.21.2.112:0/2986149691 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcfa018f360 0x7fcfa00a9940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:14.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:14.887 7fcfa5578700 1 -- 172.21.2.112:0/2986149691 >> 172.21.2.112:0/2986149691 conn(0x7fcfa009eba0 msgr2=0x7fcfa00ebe90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:14.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:14.887 7fcfa5578700 1 -- 172.21.2.112:0/2986149691 shutdown_connections 2021-02-10T11:42:14.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:14.887 7fcfa5578700 1 -- 172.21.2.112:0/2986149691 wait complete. 2021-02-10T11:42:14.899 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:42:15.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.314 7efcc4c1c700 1 Processor -- start 2021-02-10T11:42:15.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.314 7efcc4c1c700 1 -- start start 2021-02-10T11:42:15.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.314 7efcc4c1c700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efcc00d1870 0x7efcc01b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:15.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.314 7efcc4c1c700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7efcc00f0650 0x7efcc00cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:15.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.314 7efcc4c1c700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcc00f0b10 con 0x7efcc01b0a10 2021-02-10T11:42:15.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.314 7efcc4c1c700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efcc00fe230 con 0x7efcc00f0650 2021-02-10T11:42:15.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.314 7efcc4c1c700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcc00e5ed0 con 0x7efcc00d1870 2021-02-10T11:42:15.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.315 7efcbe85e700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efcc00d1870 0x7efcc01b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:15.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.315 7efcbe05d700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7efcc01b0a10 0x7efcc00cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:49064/0 (socket says 172.21.2.112:49064) 2021-02-10T11:42:15.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.315 7efcbe05d700 1 -- 172.21.2.112:0/3281881702 learned_addr learned my addr 172.21.2.112:0/3281881702 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:42:15.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.315 7efcbf05f700 1 --2- 172.21.2.112:0/3281881702 >> v2:172.21.2.120:3300/0 conn(0x7efcc00f0650 0x7efcc00cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:15.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.315 7efcbe85e700 1 -- 172.21.2.112:0/3281881702 >> v2:172.21.2.120:3300/0 conn(0x7efcc00f0650 msgr2=0x7efcc00cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:15.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.315 7efcbe85e700 1 --2- 172.21.2.112:0/3281881702 >> v2:172.21.2.120:3300/0 conn(0x7efcc00f0650 0x7efcc00cfb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:15.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.315 7efcbe85e700 1 -- 172.21.2.112:0/3281881702 >> v1:172.21.2.120:6789/0 conn(0x7efcc01b0a10 legacy=0x7efcc00cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:15.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.315 7efcbd85c700 1 -- 172.21.2.112:0/3281881702 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2973997908 0 0) 0x7efca80010c0 con 0x7efcc01b0a10 2021-02-10T11:42:15.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.315 7efcbe85e700 1 -- 172.21.2.112:0/3281881702 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcc00f20a0 con 0x7efcc00d1870 2021-02-10T11:42:15.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.315 7efcbf05f700 1 --2- 172.21.2.112:0/3281881702 >> v2:172.21.2.120:3300/0 conn(0x7efcc00f0650 0x7efcc00cfb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:42:15.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.315 7efcbe85e700 1 --2- 172.21.2.112:0/3281881702 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efcc00d1870 0x7efcc01b03f0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7efcb00069d0 tx=0x7efcb0009e40).ready entity=mon.2 client_cookie=df08342d2b15d2eb server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:15.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.315 7efcbd85c700 1 -- 172.21.2.112:0/3281881702 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efcb000b510 con 0x7efcc00d1870 2021-02-10T11:42:15.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.315 7efcbd85c700 1 -- 172.21.2.112:0/3281881702 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcb000b510 con 0x7efcc00d1870 2021-02-10T11:42:15.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.315 7efcc4c1c700 1 -- 172.21.2.112:0/3281881702 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efcc00d1870 msgr2=0x7efcc01b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:15.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.316 7efcc4c1c700 1 --2- 172.21.2.112:0/3281881702 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efcc00d1870 0x7efcc01b03f0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7efcb00069d0 tx=0x7efcb0009e40).stop 2021-02-10T11:42:15.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.316 7efcbd85c700 1 -- 172.21.2.112:0/3281881702 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efcb000b890 con 0x7efcc00d1870 2021-02-10T11:42:15.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.316 7efcc4c1c700 1 -- 172.21.2.112:0/3281881702 shutdown_connections 2021-02-10T11:42:15.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.316 7efcc4c1c700 1 --2- 172.21.2.112:0/3281881702 >> v2:172.21.2.120:3300/0 conn(0x7efcc00f0650 0x7efcc00cfb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:15.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.316 7efcc4c1c700 1 --2- 172.21.2.112:0/3281881702 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efcc00d1870 0x7efcc01b03f0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:15.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.316 7efcc4c1c700 1 -- 172.21.2.112:0/3281881702 >> 172.21.2.112:0/3281881702 conn(0x7efcc00ce450 msgr2=0x7efcc00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:15.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.316 7efcc4c1c700 1 -- 172.21.2.112:0/3281881702 shutdown_connections 2021-02-10T11:42:15.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.316 7efcc4c1c700 1 -- 172.21.2.112:0/3281881702 wait complete. 2021-02-10T11:42:15.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.331 7efcc4c1c700 1 Processor -- start 2021-02-10T11:42:15.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.331 7efcc4c1c700 1 -- start start 2021-02-10T11:42:15.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.331 7efcc4c1c700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7efcc01a2c50 0x7efcc018b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:15.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.331 7efcc4c1c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efcc018bd70 0x7efcc01949d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:15.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.331 7efcc4c1c700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efcc0194ff0 0x7efcc01953f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:15.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.331 7efcc4c1c700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcc01a3050 con 0x7efcc018bd70 2021-02-10T11:42:15.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.331 7efcc4c1c700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcc01a3950 con 0x7efcc01a2c50 2021-02-10T11:42:15.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.331 7efcc4c1c700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcc01a3b70 con 0x7efcc0194ff0 2021-02-10T11:42:15.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.332 7efcbe85e700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7efcc01a2c50 0x7efcc018b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:15.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.332 7efcbe85e700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7efcc01a2c50 0x7efcc018b750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3300/0 says I am v2:172.21.2.112:58330/0 (socket says 172.21.2.112:58330) 2021-02-10T11:42:15.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.332 7efcbe85e700 1 -- 172.21.2.112:0/4174184362 learned_addr learned my addr 172.21.2.112:0/4174184362 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:42:15.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.332 7efcbe05d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efcc018bd70 0x7efcc01949d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:15.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.332 7efcbf05f700 1 --2- 172.21.2.112:0/4174184362 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efcc0194ff0 0x7efcc01953f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:15.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.332 7efcbe85e700 1 -- 172.21.2.112:0/4174184362 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efcc0194ff0 msgr2=0x7efcc01953f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:15.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.332 7efcbe85e700 1 --2- 172.21.2.112:0/4174184362 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efcc0194ff0 0x7efcc01953f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:15.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.332 7efcbe85e700 1 -- 172.21.2.112:0/4174184362 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efcc018bd70 msgr2=0x7efcc01949d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:15.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.332 7efcbe85e700 1 --2- 172.21.2.112:0/4174184362 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efcc018bd70 0x7efcc01949d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:15.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.332 7efcbe85e700 1 -- 172.21.2.112:0/4174184362 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcc01a3d90 con 0x7efcc01a2c50 2021-02-10T11:42:15.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.332 7efcbe85e700 1 --2- 172.21.2.112:0/4174184362 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7efcc01a2c50 0x7efcc018b750 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 rx=0x7efcb000b930 tx=0x7efcb000bcd0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:15.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.332 7efcb77fe700 1 -- 172.21.2.112:0/4174184362 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efcb0006c20 con 0x7efcc01a2c50 2021-02-10T11:42:15.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.332 7efcb77fe700 1 -- 172.21.2.112:0/4174184362 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcb0006030 con 0x7efcc01a2c50 2021-02-10T11:42:15.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.332 7efcc4c1c700 1 -- 172.21.2.112:0/4174184362 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcc01a3d90 con 0x7efcc01a2c50 2021-02-10T11:42:15.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.332 7efcb77fe700 1 -- 172.21.2.112:0/4174184362 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efcb0006480 con 0x7efcc01a2c50 2021-02-10T11:42:15.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.332 7efcc4c1c700 1 -- 172.21.2.112:0/4174184362 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcc01a3d90 con 0x7efcc01a2c50 2021-02-10T11:42:15.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.333 7efcb77fe700 1 -- 172.21.2.112:0/4174184362 <== mon.1 v2:172.21.2.120:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7efcb0006210 con 0x7efcc01a2c50 2021-02-10T11:42:15.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.333 7efcb77fe700 1 --2- 172.21.2.112:0/4174184362 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7efca4018730 0x7efca401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:15.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.333 7efcb77fe700 1 -- 172.21.2.112:0/4174184362 <== mon.1 v2:172.21.2.120:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 4753+0+0 (secure 0 0 0) 0x7efcb0032630 con 0x7efcc01a2c50 2021-02-10T11:42:15.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.333 7efcbe05d700 1 --2- 172.21.2.112:0/4174184362 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7efca4018730 0x7efca401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:15.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.334 7efcbe05d700 1 --2- 172.21.2.112:0/4174184362 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7efca4018730 0x7efca401ac10 crc :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:15.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.363 7efcc4c1c700 1 -- 172.21.2.112:0/4174184362 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcc00ebc60 con 0x7efcc01a2c50 2021-02-10T11:42:15.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.365 7efcb77fe700 1 -- 172.21.2.112:0/4174184362 <== mon.1 v2:172.21.2.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7efcb000be10 con 0x7efcc01a2c50 2021-02-10T11:42:15.489 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.491 7efcc4c1c700 1 -- 172.21.2.112:0/4174184362 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efcc00ebc60 con 0x7efcc01a2c50 2021-02-10T11:42:15.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.492 7efcb77fe700 1 -- 172.21.2.112:0/4174184362 <== mon.1 v2:172.21.2.120:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v121) v1 ==== 74+0+11154 (secure 0 0 0) 0x7efcb0006030 con 0x7efcc01a2c50 2021-02-10T11:42:15.493 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:42:15.493 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":121,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:42:14.233983","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:42:15.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.495 7efcc4c1c700 1 -- 172.21.2.112:0/4174184362 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7efca4018730 msgr2=0x7efca401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:15.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.495 7efcc4c1c700 1 --2- 172.21.2.112:0/4174184362 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7efca4018730 0x7efca401ac10 crc :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:15.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.495 7efcc4c1c700 1 -- 172.21.2.112:0/4174184362 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7efcc01a2c50 msgr2=0x7efcc018b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:15.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.495 7efcc4c1c700 1 --2- 172.21.2.112:0/4174184362 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7efcc01a2c50 0x7efcc018b750 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 rx=0x7efcb000b930 tx=0x7efcb000bcd0).stop 2021-02-10T11:42:15.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.495 7efcc4c1c700 1 -- 172.21.2.112:0/4174184362 shutdown_connections 2021-02-10T11:42:15.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.495 7efcc4c1c700 1 --2- 172.21.2.112:0/4174184362 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7efca4018730 0x7efca401ac10 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:15.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.495 7efcc4c1c700 1 --2- 172.21.2.112:0/4174184362 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efcc0194ff0 0x7efcc01953f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:15.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.495 7efcc4c1c700 1 --2- 172.21.2.112:0/4174184362 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efcc018bd70 0x7efcc01949d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:15.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.495 7efcc4c1c700 1 --2- 172.21.2.112:0/4174184362 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7efcc01a2c50 0x7efcc018b750 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:15.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.495 7efcc4c1c700 1 -- 172.21.2.112:0/4174184362 >> 172.21.2.112:0/4174184362 conn(0x7efcc019c5a0 msgr2=0x7efcc00ebe90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:15.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.496 7efcc4c1c700 1 -- 172.21.2.112:0/4174184362 shutdown_connections 2021-02-10T11:42:15.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.496 7efcc4c1c700 1 -- 172.21.2.112:0/4174184362 wait complete. 2021-02-10T11:42:15.503 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:13:57.508758', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2021-02-10T11:42:15.503 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:42:15.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.919 7fc7c16b2700 1 Processor -- start 2021-02-10T11:42:15.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.919 7fc7c16b2700 1 -- start start 2021-02-10T11:42:15.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.920 7fc7c16b2700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc7bc0d1870 0x7fc7bc1b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:15.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.920 7fc7c16b2700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fc7bc1b0ae0 0x7fc7bc108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:15.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.920 7fc7c16b2700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7bc109a60 con 0x7fc7bc0d1c70 2021-02-10T11:42:15.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.920 7fc7c16b2700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7bc0e5ef0 con 0x7fc7bc1b0ae0 2021-02-10T11:42:15.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.920 7fc7c16b2700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7bc109750 con 0x7fc7bc0d1870 2021-02-10T11:42:15.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.921 7fc7baabe700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fc7bc1b0ae0 0x7fc7bc108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:15.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.921 7fc7bb2bf700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc7bc0d1870 0x7fc7bc1b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:15.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.921 7fc7bbac0700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7fc7bc0d1c70 0x7fc7bc108f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:49078/0 (socket says 172.21.2.112:49078) 2021-02-10T11:42:15.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.921 7fc7bbac0700 1 -- 172.21.2.112:0/2029820584 learned_addr learned my addr 172.21.2.112:0/2029820584 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:42:15.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.922 7fc7bb2bf700 1 -- 172.21.2.112:0/2029820584 >> v2:172.21.2.120:3300/0 conn(0x7fc7bc1b0ae0 msgr2=0x7fc7bc108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:15.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.922 7fc7bb2bf700 1 --2- 172.21.2.112:0/2029820584 >> v2:172.21.2.120:3300/0 conn(0x7fc7bc1b0ae0 0x7fc7bc108940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:15.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.922 7fc7bb2bf700 1 -- 172.21.2.112:0/2029820584 >> v1:172.21.2.120:6789/0 conn(0x7fc7bc0d1c70 legacy=0x7fc7bc108f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:15.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.922 7fc7bb2bf700 1 -- 172.21.2.112:0/2029820584 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7bc088070 con 0x7fc7bc0d1870 2021-02-10T11:42:15.938 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.922 7fc7baabe700 1 --2- 172.21.2.112:0/2029820584 >> v2:172.21.2.120:3300/0 conn(0x7fc7bc1b0ae0 0x7fc7bc108940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:42:15.938 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.922 7fc7bb2bf700 1 --2- 172.21.2.112:0/2029820584 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc7bc0d1870 0x7fc7bc1b04c0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7fc7a4006650 tx=0x7fc7a4009a80).ready entity=mon.2 client_cookie=dccd563f6680dfd5 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:15.938 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.923 7fc7ba2bd700 1 -- 172.21.2.112:0/2029820584 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc7a4006390 con 0x7fc7bc0d1870 2021-02-10T11:42:15.938 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.923 7fc7ba2bd700 1 -- 172.21.2.112:0/2029820584 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7a40009b0 con 0x7fc7bc0d1870 2021-02-10T11:42:15.938 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.923 7fc7ba2bd700 1 -- 172.21.2.112:0/2029820584 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc7a400c2b0 con 0x7fc7bc0d1870 2021-02-10T11:42:15.938 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.923 7fc7c16b2700 1 -- 172.21.2.112:0/2029820584 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc7bc0d1870 msgr2=0x7fc7bc1b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:15.938 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.923 7fc7c16b2700 1 --2- 172.21.2.112:0/2029820584 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc7bc0d1870 0x7fc7bc1b04c0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7fc7a4006650 tx=0x7fc7a4009a80).stop 2021-02-10T11:42:15.938 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.924 7fc7c16b2700 1 -- 172.21.2.112:0/2029820584 shutdown_connections 2021-02-10T11:42:15.938 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.924 7fc7c16b2700 1 --2- 172.21.2.112:0/2029820584 >> v2:172.21.2.120:3300/0 conn(0x7fc7bc1b0ae0 0x7fc7bc108940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:15.938 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.924 7fc7c16b2700 1 --2- 172.21.2.112:0/2029820584 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc7bc0d1870 0x7fc7bc1b04c0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:15.938 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.924 7fc7c16b2700 1 -- 172.21.2.112:0/2029820584 >> 172.21.2.112:0/2029820584 conn(0x7fc7bc0ce420 msgr2=0x7fc7bc0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:15.938 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.924 7fc7c16b2700 1 -- 172.21.2.112:0/2029820584 shutdown_connections 2021-02-10T11:42:15.939 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.924 7fc7c16b2700 1 -- 172.21.2.112:0/2029820584 wait complete. 2021-02-10T11:42:15.941 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.938 7fc7c16b2700 1 Processor -- start 2021-02-10T11:42:15.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.938 7fc7c16b2700 1 -- start start 2021-02-10T11:42:15.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.939 7fc7c16b2700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc7bc0a58c0 0x7fc7bc0a4d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:15.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.939 7fc7c16b2700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc7bc0b1ba0 0x7fc7bc0b1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:15.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.939 7fc7c16b2700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc7bc0b25c0 0x7fc7bc0b3270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:15.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.939 7fc7c16b2700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7bc0b2a40 con 0x7fc7bc0b1ba0 2021-02-10T11:42:15.943 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.939 7fc7c16b2700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7bc0a5340 con 0x7fc7bc0a58c0 2021-02-10T11:42:15.943 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.939 7fc7c16b2700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7bc0a5560 con 0x7fc7bc0b25c0 2021-02-10T11:42:15.943 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.939 7fc7baabe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc7bc0b1ba0 0x7fc7bc0b1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:15.943 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.939 7fc7baabe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc7bc0b1ba0 0x7fc7bc0b1fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:56066/0 (socket says 172.21.2.112:56066) 2021-02-10T11:42:15.944 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.939 7fc7baabe700 1 -- 172.21.2.112:0/1799158744 learned_addr learned my addr 172.21.2.112:0/1799158744 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:42:15.944 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.939 7fc7baabe700 1 -- 172.21.2.112:0/1799158744 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc7bc0b25c0 msgr2=0x7fc7bc0b3270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:15.944 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.939 7fc7baabe700 1 --2- 172.21.2.112:0/1799158744 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc7bc0b25c0 0x7fc7bc0b3270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:15.944 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.939 7fc7baabe700 1 -- 172.21.2.112:0/1799158744 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc7bc0a58c0 msgr2=0x7fc7bc0a4d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:15.945 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.939 7fc7baabe700 1 --2- 172.21.2.112:0/1799158744 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc7bc0a58c0 0x7fc7bc0a4d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:15.945 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.939 7fc7baabe700 1 -- 172.21.2.112:0/1799158744 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7bc0b3890 con 0x7fc7bc0b1ba0 2021-02-10T11:42:15.945 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.939 7fc7baabe700 1 --2- 172.21.2.112:0/1799158744 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc7bc0b1ba0 0x7fc7bc0b1fa0 secure :-1 s=READY pgs=2305 cs=0 l=1 rev1=1 rx=0x7fc7ac006f40 tx=0x7fc7ac00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:15.945 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.940 7fc7b3fff700 1 -- 172.21.2.112:0/1799158744 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc7ac00bd80 con 0x7fc7bc0b1ba0 2021-02-10T11:42:15.945 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.940 7fc7b3fff700 1 -- 172.21.2.112:0/1799158744 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7ac000c90 con 0x7fc7bc0b1ba0 2021-02-10T11:42:15.945 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.940 7fc7c16b2700 1 -- 172.21.2.112:0/1799158744 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7bc0b3890 con 0x7fc7bc0b1ba0 2021-02-10T11:42:15.945 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.940 7fc7c16b2700 1 -- 172.21.2.112:0/1799158744 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7bc0b3890 con 0x7fc7bc0b1ba0 2021-02-10T11:42:15.945 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.940 7fc7b3fff700 1 -- 172.21.2.112:0/1799158744 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc7ac002220 con 0x7fc7bc0b1ba0 2021-02-10T11:42:15.945 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.940 7fc7b3fff700 1 -- 172.21.2.112:0/1799158744 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc7ac00a000 con 0x7fc7bc0b1ba0 2021-02-10T11:42:15.945 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.940 7fc7b3fff700 1 --2- 172.21.2.112:0/1799158744 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc7a0018730 0x7fc7a001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:15.946 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.940 7fc7b3fff700 1 -- 172.21.2.112:0/1799158744 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 4753+0+0 (secure 0 0 0) 0x7fc7ac032490 con 0x7fc7bc0b1ba0 2021-02-10T11:42:15.946 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.941 7fc7bb2bf700 1 --2- 172.21.2.112:0/1799158744 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc7a0018730 0x7fc7a001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:15.946 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.941 7fc7bb2bf700 1 --2- 172.21.2.112:0/1799158744 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc7a0018730 0x7fc7a001abc0 crc :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:15.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.970 7fc7c16b2700 1 -- 172.21.2.112:0/1799158744 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7bc082980 con 0x7fc7bc0b1ba0 2021-02-10T11:42:15.971 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:15.972 7fc7b3fff700 1 -- 172.21.2.112:0/1799158744 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc7ac006b00 con 0x7fc7bc0b1ba0 2021-02-10T11:42:16.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:16.099 7fc7c16b2700 1 -- 172.21.2.112:0/1799158744 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7fc7bc0b3890 con 0x7fc7bc0b1ba0 2021-02-10T11:42:16.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:16.100 7fc7b3fff700 1 -- 172.21.2.112:0/1799158744 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v121) v1 ==== 92+0+10 (secure 0 0 0) 0x7fc7ac00bd80 con 0x7fc7bc0b1ba0 2021-02-10T11:42:16.099 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:42:16.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:16.102 7fc7c16b2700 1 -- 172.21.2.112:0/1799158744 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc7a0018730 msgr2=0x7fc7a001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:16.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:16.102 7fc7c16b2700 1 --2- 172.21.2.112:0/1799158744 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc7a0018730 0x7fc7a001abc0 crc :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:16.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:16.102 7fc7c16b2700 1 -- 172.21.2.112:0/1799158744 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc7bc0b1ba0 msgr2=0x7fc7bc0b1fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:16.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:16.102 7fc7c16b2700 1 --2- 172.21.2.112:0/1799158744 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc7bc0b1ba0 0x7fc7bc0b1fa0 secure :-1 s=READY pgs=2305 cs=0 l=1 rev1=1 rx=0x7fc7ac006f40 tx=0x7fc7ac00b990).stop 2021-02-10T11:42:16.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:16.102 7fc7c16b2700 1 -- 172.21.2.112:0/1799158744 shutdown_connections 2021-02-10T11:42:16.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:16.102 7fc7c16b2700 1 --2- 172.21.2.112:0/1799158744 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc7a0018730 0x7fc7a001abc0 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:16.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:16.102 7fc7c16b2700 1 --2- 172.21.2.112:0/1799158744 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc7bc0b25c0 0x7fc7bc0b3270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:16.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:16.102 7fc7c16b2700 1 --2- 172.21.2.112:0/1799158744 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc7bc0b1ba0 0x7fc7bc0b1fa0 unknown :-1 s=CLOSED pgs=2305 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:16.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:16.102 7fc7c16b2700 1 --2- 172.21.2.112:0/1799158744 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc7bc0a58c0 0x7fc7bc0a4d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:16.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:16.102 7fc7c16b2700 1 -- 172.21.2.112:0/1799158744 >> 172.21.2.112:0/1799158744 conn(0x7fc7bc0b5dc0 msgr2=0x7fc7bc095960 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:16.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:16.102 7fc7c16b2700 1 -- 172.21.2.112:0/1799158744 shutdown_connections 2021-02-10T11:42:16.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:16.102 7fc7c16b2700 1 -- 172.21.2.112:0/1799158744 wait complete. 2021-02-10T11:42:16.110 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --format=json-pretty osd lspools 2021-02-10T11:42:16.539 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:16.523 7fea75068700 1 Processor -- start 2021-02-10T11:42:16.539 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:16.523 7fea75068700 1 -- start start 2021-02-10T11:42:16.539 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:16.523 7fea75068700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fea700d0340 0x7fea700ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:16.539 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:16.523 7fea75068700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea700f0b10 con 0x7fea700d1870 2021-02-10T11:42:16.539 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:16.523 7fea75068700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea701b0be0 con 0x7fea700d1c70 2021-02-10T11:42:16.539 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:16.523 7fea75068700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea700e5ed0 con 0x7fea700d0340 2021-02-10T11:42:16.539 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:16.523 7fea6ed9d700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fea700d1870 0x7fea701b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:55532/0 (socket says 172.21.2.112:55532) 2021-02-10T11:42:16.540 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:16.523 7fea6ed9d700 1 -- 172.21.2.112:0/890199540 learned_addr learned my addr 172.21.2.112:0/890199540 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:42:16.540 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:16.524 7fea6dd9b700 1 -- 172.21.2.112:0/890199540 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2831442381 0 0) 0x7fea58001180 con 0x7fea700d1870 2021-02-10T11:42:16.540 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:16.524 7fea6dd9b700 1 -- 172.21.2.112:0/890199540 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea5c002fe0 con 0x7fea700d1870 2021-02-10T11:42:16.540 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:16.524 7fea6e59c700 1 --2- 172.21.2.112:0/890199540 >> v2:172.21.2.120:3300/0 conn(0x7fea700d0340 0x7fea700ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:16.540 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:16.524 7fea6dd9b700 1 -- 172.21.2.112:0/890199540 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2934577143 0 0) 0x7fea58001180 con 0x7fea700d1870 2021-02-10T11:42:16.540 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:16.524 7fea6dd9b700 1 -- 172.21.2.112:0/890199540 >> v2:172.21.2.120:3300/0 conn(0x7fea700d0340 msgr2=0x7fea700ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:16.540 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:16.524 7fea6dd9b700 1 --2- 172.21.2.112:0/890199540 >> v2:172.21.2.120:3300/0 conn(0x7fea700d0340 0x7fea700ef640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:16.540 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:16.524 7fea6dd9b700 1 -- 172.21.2.112:0/890199540 >> v1:172.21.2.120:6789/0 conn(0x7fea700d1c70 legacy=0x7fea700efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:16.540 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:16.524 7fea6dd9b700 1 -- 172.21.2.112:0/890199540 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea700f0450 con 0x7fea700d1870 2021-02-10T11:42:16.540 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:16.524 7fea6dd9b700 1 -- 172.21.2.112:0/890199540 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7fea58001680 con 0x7fea700d1870 2021-02-10T11:42:16.540 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:16.524 7fea6dd9b700 1 -- 172.21.2.112:0/890199540 >> v1:172.21.2.112:6789/0 conn(0x7fea700d1870 legacy=0x7fea701b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:16.540 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:16.524 7fea6dd9b700 1 --2- 172.21.2.112:0/890199540 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fea5c003b10 0x7fea5c005f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:16.540 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:16.524 7fea6dd9b700 1 --2- 172.21.2.112:0/890199540 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fea5c0067a0 0x7fea5c008ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:16.541 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:16.524 7fea6dd9b700 1 --2- 172.21.2.112:0/890199540 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fea5c0095a0 0x7fea5c00bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:16.541 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:16.524 7fea6dd9b700 1 -- 172.21.2.112:0/890199540 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea5c00c310 con 0x7fea5c0095a0 2021-02-10T11:42:16.541 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:16.524 7fea6dd9b700 1 -- 172.21.2.112:0/890199540 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea5c00c530 con 0x7fea5c0067a0 2021-02-10T11:42:16.541 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:16.524 7fea6dd9b700 1 -- 172.21.2.112:0/890199540 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea5c00c750 con 0x7fea5c003b10 2021-02-10T11:42:16.541 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:16.524 7fea6f59e700 1 --2- 172.21.2.112:0/890199540 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fea5c0095a0 0x7fea5c00bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:16.541 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:16.525 7fea6e59c700 1 --2- 172.21.2.112:0/890199540 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fea5c0067a0 0x7fea5c008ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:16.541 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:16.525 7fea6ed9d700 1 --2- 172.21.2.112:0/890199540 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fea5c003b10 0x7fea5c005f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:16.541 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:16.525 7fea6f59e700 1 -- 172.21.2.112:0/890199540 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fea5c003b10 msgr2=0x7fea5c005f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:16.541 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:16.525 7fea6f59e700 1 --2- 172.21.2.112:0/890199540 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fea5c003b10 0x7fea5c005f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:16.541 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:16.525 7fea6f59e700 1 -- 172.21.2.112:0/890199540 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fea5c0067a0 msgr2=0x7fea5c008ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:16.541 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:16.525 7fea6f59e700 1 --2- 172.21.2.112:0/890199540 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fea5c0067a0 0x7fea5c008ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:16.541 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:16.525 7fea6f59e700 1 -- 172.21.2.112:0/890199540 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea5c0509c0 con 0x7fea5c0095a0 2021-02-10T11:42:16.541 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:16.525 7fea6f59e700 1 -- 172.21.2.112:0/890199540 reap_dead start 2021-02-10T11:42:16.542 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:16.525 7fea6f59e700 1 --2- 172.21.2.112:0/890199540 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fea5c0095a0 0x7fea5c00bac0 secure :-1 s=READY pgs=2307 cs=0 l=1 rev1=1 rx=0x7fea68006e40 tx=0x7fea68009e30).ready entity=mon.0 client_cookie=d7ed4d2c59198257 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:16.542 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:16.525 7fea6dd9b700 1 -- 172.21.2.112:0/890199540 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fea68006a60 con 0x7fea5c0095a0 2021-02-10T11:42:16.542 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:16.525 7fea6dd9b700 1 -- 172.21.2.112:0/890199540 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea6800b300 con 0x7fea5c0095a0 2021-02-10T11:42:16.542 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:16.525 7fea6dd9b700 1 -- 172.21.2.112:0/890199540 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fea6800b7e0 con 0x7fea5c0095a0 2021-02-10T11:42:16.542 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:16.525 7fea75068700 1 -- 172.21.2.112:0/890199540 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fea5c0095a0 msgr2=0x7fea5c00bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:16.542 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:16.525 7fea75068700 1 --2- 172.21.2.112:0/890199540 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fea5c0095a0 0x7fea5c00bac0 secure :-1 s=READY pgs=2307 cs=0 l=1 rev1=1 rx=0x7fea68006e40 tx=0x7fea68009e30).stop 2021-02-10T11:42:16.542 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:16.526 7fea75068700 1 -- 172.21.2.112:0/890199540 shutdown_connections 2021-02-10T11:42:16.543 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:16.526 7fea75068700 1 --2- 172.21.2.112:0/890199540 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fea5c0095a0 0x7fea5c00bac0 unknown :-1 s=CLOSED pgs=2307 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:16.543 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:16.526 7fea75068700 1 -- 172.21.2.112:0/890199540 >> 172.21.2.112:0/890199540 conn(0x7fea700ce450 msgr2=0x7fea700d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:16.543 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:16.526 7fea75068700 1 -- 172.21.2.112:0/890199540 shutdown_connections 2021-02-10T11:42:16.543 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:16.526 7fea75068700 1 -- 172.21.2.112:0/890199540 wait complete. 2021-02-10T11:42:16.544 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:16.541 7fea75068700 1 Processor -- start 2021-02-10T11:42:16.544 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:16.541 7fea75068700 1 -- start start 2021-02-10T11:42:16.545 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:16.541 7fea75068700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fea700f0280 0x7fea700a2310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:16.545 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:16.541 7fea75068700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fea700f0680 0x7fea700a2930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:16.545 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:16.541 7fea75068700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fea700ec180 0x7fea700ec580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:16.545 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:16.541 7fea75068700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea700efd00 con 0x7fea700f0680 2021-02-10T11:42:16.545 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:16.541 7fea75068700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea700ecba0 con 0x7fea700f0280 2021-02-10T11:42:16.545 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:16.541 7fea75068700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea700ecd80 con 0x7fea700ec180 2021-02-10T11:42:16.545 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:16.541 7fea6e59c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fea700f0680 0x7fea700a2930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:16.545 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:16.541 7fea6e59c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fea700f0680 0x7fea700a2930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:56082/0 (socket says 172.21.2.112:56082) 2021-02-10T11:42:16.545 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:16.541 7fea6e59c700 1 -- 172.21.2.112:0/1564140600 learned_addr learned my addr 172.21.2.112:0/1564140600 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:42:16.545 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:16.542 7fea6e59c700 1 -- 172.21.2.112:0/1564140600 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fea700ec180 msgr2=0x7fea700ec580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:16.545 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:16.542 7fea6e59c700 1 --2- 172.21.2.112:0/1564140600 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fea700ec180 0x7fea700ec580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:16.545 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:16.542 7fea6e59c700 1 -- 172.21.2.112:0/1564140600 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fea700f0280 msgr2=0x7fea700a2310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:16.545 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:16.542 7fea6e59c700 1 --2- 172.21.2.112:0/1564140600 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fea700f0280 0x7fea700a2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:16.546 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:16.542 7fea6e59c700 1 -- 172.21.2.112:0/1564140600 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea700ecfa0 con 0x7fea700f0680 2021-02-10T11:42:16.546 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:16.542 7fea6e59c700 1 --2- 172.21.2.112:0/1564140600 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fea700f0680 0x7fea700a2930 secure :-1 s=READY pgs=2308 cs=0 l=1 rev1=1 rx=0x7fea60006f60 tx=0x7fea6000b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:16.546 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:16.542 7fea677fe700 1 -- 172.21.2.112:0/1564140600 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fea600009b0 con 0x7fea700f0680 2021-02-10T11:42:16.546 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:16.542 7fea677fe700 1 -- 172.21.2.112:0/1564140600 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea6000bdf0 con 0x7fea700f0680 2021-02-10T11:42:16.546 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:16.542 7fea677fe700 1 -- 172.21.2.112:0/1564140600 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fea60002030 con 0x7fea700f0680 2021-02-10T11:42:16.546 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:16.542 7fea75068700 1 -- 172.21.2.112:0/1564140600 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea700ecfa0 con 0x7fea700f0680 2021-02-10T11:42:16.546 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:16.542 7fea75068700 1 -- 172.21.2.112:0/1564140600 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea700ed530 con 0x7fea700f0680 2021-02-10T11:42:16.546 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:16.543 7fea677fe700 1 -- 172.21.2.112:0/1564140600 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fea60000c90 con 0x7fea700f0680 2021-02-10T11:42:16.546 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:16.543 7fea677fe700 1 --2- 172.21.2.112:0/1564140600 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fea540186c0 0x7fea5401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:16.546 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:16.543 7fea677fe700 1 -- 172.21.2.112:0/1564140600 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 4753+0+0 (secure 0 0 0) 0x7fea60032120 con 0x7fea700f0680 2021-02-10T11:42:16.546 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:16.543 7fea6ed9d700 1 --2- 172.21.2.112:0/1564140600 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fea540186c0 0x7fea5401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:16.546 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:16.544 7fea6ed9d700 1 --2- 172.21.2.112:0/1564140600 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fea540186c0 0x7fea5401abc0 crc :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:16.572 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:16.573 7fea75068700 1 -- 172.21.2.112:0/1564140600 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea700ebc60 con 0x7fea700f0680 2021-02-10T11:42:16.574 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:16.576 7fea677fe700 1 -- 172.21.2.112:0/1564140600 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fea6002d110 con 0x7fea700f0680 2021-02-10T11:42:16.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:16.699 7fea75068700 1 -- 172.21.2.112:0/1564140600 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fea700ebc60 con 0x7fea700f0680 2021-02-10T11:42:16.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:16.700 7fea677fe700 1 -- 172.21.2.112:0/1564140600 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v121) v1 ==== 84+0+64 (secure 0 0 0) 0x7fea6002df30 con 0x7fea700f0680 2021-02-10T11:42:16.699 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:42:16.699 INFO:teuthology.orchestra.run.gibba012.stdout:[ 2021-02-10T11:42:16.699 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:16.699 INFO:teuthology.orchestra.run.gibba012.stdout: "poolnum": 1, 2021-02-10T11:42:16.699 INFO:teuthology.orchestra.run.gibba012.stdout: "poolname": "rbd" 2021-02-10T11:42:16.699 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:42:16.699 INFO:teuthology.orchestra.run.gibba012.stdout:] 2021-02-10T11:42:16.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:16.702 7fea75068700 1 -- 172.21.2.112:0/1564140600 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fea540186c0 msgr2=0x7fea5401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:16.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:16.702 7fea75068700 1 --2- 172.21.2.112:0/1564140600 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fea540186c0 0x7fea5401abc0 crc :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:16.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:16.702 7fea75068700 1 -- 172.21.2.112:0/1564140600 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fea700f0680 msgr2=0x7fea700a2930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:16.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:16.702 7fea75068700 1 --2- 172.21.2.112:0/1564140600 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fea700f0680 0x7fea700a2930 secure :-1 s=READY pgs=2308 cs=0 l=1 rev1=1 rx=0x7fea60006f60 tx=0x7fea6000b970).stop 2021-02-10T11:42:16.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:16.702 7fea75068700 1 -- 172.21.2.112:0/1564140600 shutdown_connections 2021-02-10T11:42:16.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:16.702 7fea75068700 1 --2- 172.21.2.112:0/1564140600 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fea540186c0 0x7fea5401abc0 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:16.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:16.702 7fea75068700 1 --2- 172.21.2.112:0/1564140600 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fea700ec180 0x7fea700ec580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:16.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:16.702 7fea75068700 1 --2- 172.21.2.112:0/1564140600 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fea700f0680 0x7fea700a2930 unknown :-1 s=CLOSED pgs=2308 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:16.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:16.702 7fea75068700 1 --2- 172.21.2.112:0/1564140600 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fea700f0280 0x7fea700a2310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:16.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:16.702 7fea75068700 1 -- 172.21.2.112:0/1564140600 >> 172.21.2.112:0/1564140600 conn(0x7fea700c2130 msgr2=0x7fea7009e3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:16.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:16.702 7fea75068700 1 -- 172.21.2.112:0/1564140600 shutdown_connections 2021-02-10T11:42:16.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:16.702 7fea75068700 1 -- 172.21.2.112:0/1564140600 wait complete. 2021-02-10T11:42:16.709 INFO:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2021-02-10T11:42:16.709 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_metadata 8 2021-02-10T11:42:17.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:17.124 7f14bc203700 1 Processor -- start 2021-02-10T11:42:17.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:17.124 7f14bc203700 1 -- start start 2021-02-10T11:42:17.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:17.124 7f14bc203700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f14b40d1c60 0x7f14b40cef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:17.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:17.124 7f14bc203700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f14b40cf570 0x7f14b40cf970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:17.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:17.124 7f14bc203700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14b40fe1b0 con 0x7f14b40d1860 2021-02-10T11:42:17.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:17.124 7f14bc203700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14b41b0ca0 con 0x7f14b40cf570 2021-02-10T11:42:17.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:17.124 7f14bc203700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14b4109a50 con 0x7f14b40d1c60 2021-02-10T11:42:17.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:17.125 7f14ba01f700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f14b40d1860 0x7f14b41b04b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:55552/0 (socket says 172.21.2.112:55552) 2021-02-10T11:42:17.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:17.125 7f14ba01f700 1 -- 172.21.2.112:0/1158102440 learned_addr learned my addr 172.21.2.112:0/1158102440 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:42:17.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:17.125 7f14b901d700 1 -- 172.21.2.112:0/1158102440 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1044976465 0 0) 0x7f14a8001180 con 0x7f14b40d1860 2021-02-10T11:42:17.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:17.125 7f14b901d700 1 -- 172.21.2.112:0/1158102440 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14a0002fe0 con 0x7f14b40d1860 2021-02-10T11:42:17.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:17.125 7f14ba820700 1 --2- 172.21.2.112:0/1158102440 >> v2:172.21.2.120:3300/0 conn(0x7f14b40cf570 0x7f14b40cf970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:17.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:17.125 7f14b981e700 1 --2- 172.21.2.112:0/1158102440 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f14b40d1c60 0x7f14b40cef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:17.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:17.125 7f14b901d700 1 -- 172.21.2.112:0/1158102440 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1825644725 0 0) 0x7f14a80010e0 con 0x7f14b40d1860 2021-02-10T11:42:17.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:17.125 7f14b901d700 1 -- 172.21.2.112:0/1158102440 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f14b40d1c60 msgr2=0x7f14b40cef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:17.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:17.125 7f14b901d700 1 --2- 172.21.2.112:0/1158102440 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f14b40d1c60 0x7f14b40cef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:17.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:17.125 7f14b901d700 1 -- 172.21.2.112:0/1158102440 >> v2:172.21.2.120:3300/0 conn(0x7f14b40cf570 msgr2=0x7f14b40cf970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:17.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:17.125 7f14b901d700 1 --2- 172.21.2.112:0/1158102440 >> v2:172.21.2.120:3300/0 conn(0x7f14b40cf570 0x7f14b40cf970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:17.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:17.126 7f14b901d700 1 -- 172.21.2.112:0/1158102440 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14b4088070 con 0x7f14b40d1860 2021-02-10T11:42:17.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:17.126 7f14b901d700 1 -- 172.21.2.112:0/1158102440 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f14a8001600 con 0x7f14b40d1860 2021-02-10T11:42:17.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:17.126 7f14b901d700 1 -- 172.21.2.112:0/1158102440 >> v1:172.21.2.112:6789/0 conn(0x7f14b40d1860 legacy=0x7f14b41b04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:17.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:17.126 7f14b901d700 1 --2- 172.21.2.112:0/1158102440 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f14a0003a70 0x7f14a0005f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:17.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:17.126 7f14b901d700 1 --2- 172.21.2.112:0/1158102440 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f14a00067a0 0x7f14a0008ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:17.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:17.126 7f14b901d700 1 --2- 172.21.2.112:0/1158102440 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f14a00095a0 0x7f14a000ba90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:17.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:17.126 7f14b901d700 1 -- 172.21.2.112:0/1158102440 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14a00502f0 con 0x7f14a00067a0 2021-02-10T11:42:17.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:17.126 7f14b901d700 1 -- 172.21.2.112:0/1158102440 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14a00504a0 con 0x7f14a0003a70 2021-02-10T11:42:17.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:17.126 7f14b901d700 1 -- 172.21.2.112:0/1158102440 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14a00506c0 con 0x7f14a00095a0 2021-02-10T11:42:17.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:17.126 7f14b981e700 1 --2- 172.21.2.112:0/1158102440 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f14a00067a0 0x7f14a0008ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:17.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:17.126 7f14ba01f700 1 --2- 172.21.2.112:0/1158102440 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f14a0003a70 0x7f14a0005f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:17.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:17.126 7f14ba820700 1 --2- 172.21.2.112:0/1158102440 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f14a00095a0 0x7f14a000ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:17.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:17.126 7f14ba01f700 1 -- 172.21.2.112:0/1158102440 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f14a00095a0 msgr2=0x7f14a000ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:17.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:17.126 7f14ba01f700 1 --2- 172.21.2.112:0/1158102440 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f14a00095a0 0x7f14a000ba90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:17.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:17.126 7f14ba01f700 1 -- 172.21.2.112:0/1158102440 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f14a00067a0 msgr2=0x7f14a0008ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:17.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:17.126 7f14ba01f700 1 --2- 172.21.2.112:0/1158102440 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f14a00067a0 0x7f14a0008ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:17.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:17.126 7f14ba01f700 1 -- 172.21.2.112:0/1158102440 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14a0050960 con 0x7f14a0003a70 2021-02-10T11:42:17.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:17.126 7f14ba820700 1 --2- 172.21.2.112:0/1158102440 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f14a00095a0 0x7f14a000ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:42:17.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:17.126 7f14b981e700 1 --2- 172.21.2.112:0/1158102440 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f14a00067a0 0x7f14a0008ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:42:17.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:17.126 7f14ba01f700 1 --2- 172.21.2.112:0/1158102440 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f14a0003a70 0x7f14a0005f00 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 rx=0x7f14a8000e80 tx=0x7f14a8007cf0).ready entity=mon.1 client_cookie=4b2da36315ce3754 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:17.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:17.127 7f14b901d700 1 -- 172.21.2.112:0/1158102440 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f14a80015c0 con 0x7f14a0003a70 2021-02-10T11:42:17.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:17.127 7f14b901d700 1 -- 172.21.2.112:0/1158102440 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14a80070c0 con 0x7f14a0003a70 2021-02-10T11:42:17.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:17.127 7f14bc203700 1 -- 172.21.2.112:0/1158102440 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f14a0003a70 msgr2=0x7f14a0005f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:17.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:17.127 7f14bc203700 1 --2- 172.21.2.112:0/1158102440 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f14a0003a70 0x7f14a0005f00 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 rx=0x7f14a8000e80 tx=0x7f14a8007cf0).stop 2021-02-10T11:42:17.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:17.127 7f14ba820700 1 -- 172.21.2.112:0/1158102440 reap_dead start 2021-02-10T11:42:17.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:17.127 7f14bc203700 1 -- 172.21.2.112:0/1158102440 shutdown_connections 2021-02-10T11:42:17.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:17.127 7f14bc203700 1 -- 172.21.2.112:0/1158102440 >> 172.21.2.112:0/1158102440 conn(0x7f14b40ce410 msgr2=0x7f14b40d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:17.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:17.127 7f14bc203700 1 -- 172.21.2.112:0/1158102440 shutdown_connections 2021-02-10T11:42:17.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:17.128 7f14bc203700 1 -- 172.21.2.112:0/1158102440 wait complete. 2021-02-10T11:42:17.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:17.142 7f14bc203700 1 Processor -- start 2021-02-10T11:42:17.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:17.142 7f14bc203700 1 -- start start 2021-02-10T11:42:17.145 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:17.142 7f14bc203700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f14b40badd0 0x7f14b409eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:17.146 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:17.142 7f14bc203700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f14b40adf50 0x7f14b409f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:17.146 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:17.142 7f14bc203700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f14b41a7470 0x7f14b41a7870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:17.146 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:17.142 7f14bc203700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14b409e5a0 con 0x7f14b40adf50 2021-02-10T11:42:17.146 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:17.142 7f14bc203700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14b409f6e0 con 0x7f14b40badd0 2021-02-10T11:42:17.146 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:17.142 7f14bc203700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14b40bb220 con 0x7f14b41a7470 2021-02-10T11:42:17.146 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:17.143 7f14b981e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f14b40adf50 0x7f14b409f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:17.146 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:17.143 7f14b981e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f14b40adf50 0x7f14b409f0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:56104/0 (socket says 172.21.2.112:56104) 2021-02-10T11:42:17.146 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:17.143 7f14b981e700 1 -- 172.21.2.112:0/1717650830 learned_addr learned my addr 172.21.2.112:0/1717650830 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:42:17.146 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:17.143 7f14b981e700 1 -- 172.21.2.112:0/1717650830 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f14b41a7470 msgr2=0x7f14b41a7870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:17.146 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:17.143 7f14b981e700 1 --2- 172.21.2.112:0/1717650830 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f14b41a7470 0x7f14b41a7870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:17.146 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:17.143 7f14b981e700 1 -- 172.21.2.112:0/1717650830 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f14b40badd0 msgr2=0x7f14b409eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:17.146 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:17.143 7f14b981e700 1 --2- 172.21.2.112:0/1717650830 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f14b40badd0 0x7f14b409eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:17.146 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:17.143 7f14b981e700 1 -- 172.21.2.112:0/1717650830 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14b41a7e90 con 0x7f14b40adf50 2021-02-10T11:42:17.147 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:17.143 7f14b981e700 1 --2- 172.21.2.112:0/1717650830 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f14b40adf50 0x7f14b409f0c0 secure :-1 s=READY pgs=2310 cs=0 l=1 rev1=1 rx=0x7f14b000b5e0 tx=0x7f14b000b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:17.147 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:17.143 7f14a6ffd700 1 -- 172.21.2.112:0/1717650830 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f14b000bd70 con 0x7f14b40adf50 2021-02-10T11:42:17.147 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:17.143 7f14a6ffd700 1 -- 172.21.2.112:0/1717650830 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14b0006720 con 0x7f14b40adf50 2021-02-10T11:42:17.147 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:17.143 7f14a6ffd700 1 -- 172.21.2.112:0/1717650830 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f14b00022b0 con 0x7f14b40adf50 2021-02-10T11:42:17.147 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:17.143 7f14bc203700 1 -- 172.21.2.112:0/1717650830 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14b41a7e90 con 0x7f14b40adf50 2021-02-10T11:42:17.147 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:17.143 7f14bc203700 1 -- 172.21.2.112:0/1717650830 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14b409d1f0 con 0x7f14b40adf50 2021-02-10T11:42:17.147 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:17.144 7f14a6ffd700 1 -- 172.21.2.112:0/1717650830 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f14b000a000 con 0x7f14b40adf50 2021-02-10T11:42:17.147 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:17.144 7f14a6ffd700 1 --2- 172.21.2.112:0/1717650830 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f149c0186d0 0x7f149c01abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:17.147 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:17.144 7f14a6ffd700 1 -- 172.21.2.112:0/1717650830 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f14b0032370 con 0x7f14b40adf50 2021-02-10T11:42:17.148 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:17.145 7f14ba01f700 1 --2- 172.21.2.112:0/1717650830 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f149c0186d0 0x7f149c01abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:17.148 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:17.145 7f14ba01f700 1 --2- 172.21.2.112:0/1717650830 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f149c0186d0 0x7f149c01abd0 crc :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:17.173 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:17.175 7f14bc203700 1 -- 172.21.2.112:0/1717650830 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14b4082a20 con 0x7f14b40adf50 2021-02-10T11:42:17.176 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:17.177 7f14a6ffd700 1 -- 172.21.2.112:0/1717650830 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f14b0000c90 con 0x7f14b40adf50 2021-02-10T11:42:17.302 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:17.304 7f14bc203700 1 -- 172.21.2.112:0/1717650830 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"} v 0) v1 -- 0x7f14b4082a20 con 0x7f14b40adf50 2021-02-10T11:42:17.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:17.732 7f14a6ffd700 1 -- 172.21.2.112:0/1717650830 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"}]=0 pool 'cephfs_metadata' created v122) v1 ==== 133+0+0 (secure 0 0 0) 0x7f14b000a000 con 0x7f14b40adf50 2021-02-10T11:42:17.769 INFO:teuthology.orchestra.run.gibba012.stderr:pool 'cephfs_metadata' created 2021-02-10T11:42:17.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:17.772 7f14bc203700 1 -- 172.21.2.112:0/1717650830 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f149c0186d0 msgr2=0x7f149c01abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:17.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:17.772 7f14bc203700 1 --2- 172.21.2.112:0/1717650830 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f149c0186d0 0x7f149c01abd0 crc :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:17.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:17.772 7f14bc203700 1 -- 172.21.2.112:0/1717650830 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f14b40adf50 msgr2=0x7f14b409f0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:17.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:17.772 7f14bc203700 1 --2- 172.21.2.112:0/1717650830 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f14b40adf50 0x7f14b409f0c0 secure :-1 s=READY pgs=2310 cs=0 l=1 rev1=1 rx=0x7f14b000b5e0 tx=0x7f14b000b980).stop 2021-02-10T11:42:17.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:17.772 7f14bc203700 1 -- 172.21.2.112:0/1717650830 shutdown_connections 2021-02-10T11:42:17.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:17.772 7f14bc203700 1 --2- 172.21.2.112:0/1717650830 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f149c0186d0 0x7f149c01abd0 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:17.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:17.772 7f14bc203700 1 --2- 172.21.2.112:0/1717650830 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f14b41a7470 0x7f14b41a7870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:17.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:17.772 7f14bc203700 1 --2- 172.21.2.112:0/1717650830 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f14b40adf50 0x7f14b409f0c0 unknown :-1 s=CLOSED pgs=2310 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:17.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:17.772 7f14bc203700 1 --2- 172.21.2.112:0/1717650830 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f14b40badd0 0x7f14b409eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:17.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:17.772 7f14bc203700 1 -- 172.21.2.112:0/1717650830 >> 172.21.2.112:0/1717650830 conn(0x7f14b40bee90 msgr2=0x7f14b40bb880 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:17.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:17.772 7f14bc203700 1 -- 172.21.2.112:0/1717650830 shutdown_connections 2021-02-10T11:42:17.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:17.772 7f14bc203700 1 -- 172.21.2.112:0/1717650830 wait complete. 2021-02-10T11:42:17.780 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_data 8 2021-02-10T11:42:18.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:18.196 7f2b56bff700 1 Processor -- start 2021-02-10T11:42:18.216 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:18.196 7f2b56bff700 1 -- start start 2021-02-10T11:42:18.216 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:18.196 7f2b56bff700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2b500d1c70 0x7f2b500cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:18.216 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:18.196 7f2b56bff700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b500d0170 con 0x7f2b500cf580 2021-02-10T11:42:18.216 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:18.196 7f2b56bff700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b500fe1c0 con 0x7f2b500d1870 2021-02-10T11:42:18.216 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:18.196 7f2b56bff700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b501b0cb0 con 0x7f2b500d1c70 2021-02-10T11:42:18.216 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:18.196 7f2b5521c700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f2b500cf580 0x7f2b500cf980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:55576/0 (socket says 172.21.2.112:55576) 2021-02-10T11:42:18.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:18.196 7f2b5521c700 1 -- 172.21.2.112:0/1962065808 learned_addr learned my addr 172.21.2.112:0/1962065808 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:42:18.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:18.197 7f2b4f7fe700 1 -- 172.21.2.112:0/1962065808 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3692332594 0 0) 0x7f2b48001180 con 0x7f2b500cf580 2021-02-10T11:42:18.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:18.197 7f2b4f7fe700 1 -- 172.21.2.112:0/1962065808 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b3c002fe0 con 0x7f2b500cf580 2021-02-10T11:42:18.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:18.197 7f2b4f7fe700 1 -- 172.21.2.112:0/1962065808 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2621936558 0 0) 0x7f2b480010e0 con 0x7f2b500cf580 2021-02-10T11:42:18.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:18.197 7f2b4f7fe700 1 -- 172.21.2.112:0/1962065808 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2b500d1c70 msgr2=0x7f2b500cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:18.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:18.197 7f2b4f7fe700 1 --2- 172.21.2.112:0/1962065808 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2b500d1c70 0x7f2b500cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:18.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:18.197 7f2b4f7fe700 1 -- 172.21.2.112:0/1962065808 >> v1:172.21.2.120:6789/0 conn(0x7f2b500d1870 legacy=0x7f2b501b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:18.218 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:18.197 7f2b4f7fe700 1 -- 172.21.2.112:0/1962065808 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b50109a60 con 0x7f2b500cf580 2021-02-10T11:42:18.218 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:18.197 7f2b4f7fe700 1 -- 172.21.2.112:0/1962065808 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f2b48001600 con 0x7f2b500cf580 2021-02-10T11:42:18.218 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:18.197 7f2b4f7fe700 1 -- 172.21.2.112:0/1962065808 >> v1:172.21.2.112:6789/0 conn(0x7f2b500cf580 legacy=0x7f2b500cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:18.218 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:18.197 7f2b4f7fe700 1 --2- 172.21.2.112:0/1962065808 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2b3c003af0 0x7f2b3c005f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:18.218 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:18.197 7f2b4f7fe700 1 --2- 172.21.2.112:0/1962065808 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2b3c0067a0 0x7f2b3c008c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:18.218 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:18.197 7f2b4f7fe700 1 --2- 172.21.2.112:0/1962065808 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2b3c0094f0 0x7f2b3c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:18.218 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:18.197 7f2b4f7fe700 1 -- 172.21.2.112:0/1962065808 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b3c00c2c0 con 0x7f2b3c003af0 2021-02-10T11:42:18.219 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:18.197 7f2b4f7fe700 1 -- 172.21.2.112:0/1962065808 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b3c00c4e0 con 0x7f2b3c0094f0 2021-02-10T11:42:18.219 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:18.197 7f2b4f7fe700 1 -- 172.21.2.112:0/1962065808 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b3c00c700 con 0x7f2b3c0067a0 2021-02-10T11:42:18.219 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:18.197 7f2b54a1b700 1 --2- 172.21.2.112:0/1962065808 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2b3c003af0 0x7f2b3c005f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:18.219 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:18.197 7f2b5521c700 1 --2- 172.21.2.112:0/1962065808 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2b3c0094f0 0x7f2b3c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:18.219 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:18.197 7f2b4ffff700 1 --2- 172.21.2.112:0/1962065808 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2b3c0067a0 0x7f2b3c008c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:18.219 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:18.197 7f2b54a1b700 1 -- 172.21.2.112:0/1962065808 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2b3c0067a0 msgr2=0x7f2b3c008c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:18.220 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:18.197 7f2b54a1b700 1 --2- 172.21.2.112:0/1962065808 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2b3c0067a0 0x7f2b3c008c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:18.220 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:18.197 7f2b54a1b700 1 -- 172.21.2.112:0/1962065808 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2b3c0094f0 msgr2=0x7f2b3c00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:18.220 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:18.197 7f2b54a1b700 1 --2- 172.21.2.112:0/1962065808 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2b3c0094f0 0x7f2b3c00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:18.220 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:18.197 7f2b54a1b700 1 -- 172.21.2.112:0/1962065808 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b3c00c9a0 con 0x7f2b3c003af0 2021-02-10T11:42:18.220 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:18.198 7f2b54a1b700 1 --2- 172.21.2.112:0/1962065808 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2b3c003af0 0x7f2b3c005f00 secure :-1 s=READY pgs=2312 cs=0 l=1 rev1=1 rx=0x7f2b38006ab0 tx=0x7f2b3804fb10).ready entity=mon.0 client_cookie=a975bc30bb57f288 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:18.220 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:18.198 7f2b4f7fe700 1 -- 172.21.2.112:0/1962065808 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2b38001290 con 0x7f2b3c003af0 2021-02-10T11:42:18.220 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:18.198 7f2b4f7fe700 1 -- 172.21.2.112:0/1962065808 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b380066f0 con 0x7f2b3c003af0 2021-02-10T11:42:18.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:18.198 7f2b4f7fe700 1 -- 172.21.2.112:0/1962065808 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2b38001790 con 0x7f2b3c003af0 2021-02-10T11:42:18.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:18.198 7f2b56bff700 1 -- 172.21.2.112:0/1962065808 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2b3c003af0 msgr2=0x7f2b3c005f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:18.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:18.198 7f2b56bff700 1 --2- 172.21.2.112:0/1962065808 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2b3c003af0 0x7f2b3c005f00 secure :-1 s=READY pgs=2312 cs=0 l=1 rev1=1 rx=0x7f2b38006ab0 tx=0x7f2b3804fb10).stop 2021-02-10T11:42:18.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:18.198 7f2b5521c700 1 -- 172.21.2.112:0/1962065808 reap_dead start 2021-02-10T11:42:18.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:18.198 7f2b56bff700 1 -- 172.21.2.112:0/1962065808 shutdown_connections 2021-02-10T11:42:18.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:18.199 7f2b56bff700 1 -- 172.21.2.112:0/1962065808 >> 172.21.2.112:0/1962065808 conn(0x7f2b500ce420 msgr2=0x7f2b500d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:18.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:18.199 7f2b56bff700 1 -- 172.21.2.112:0/1962065808 shutdown_connections 2021-02-10T11:42:18.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:18.199 7f2b56bff700 1 -- 172.21.2.112:0/1962065808 wait complete. 2021-02-10T11:42:18.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:18.213 7f2b56bff700 1 Processor -- start 2021-02-10T11:42:18.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:18.213 7f2b56bff700 1 -- start start 2021-02-10T11:42:18.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:18.214 7f2b56bff700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2b50085bc0 0x7f2b500863e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:18.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:18.214 7f2b56bff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2b500954c0 0x7f2b50096300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:18.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:18.214 7f2b56bff700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2b500958c0 0x7f2b50096920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:18.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:18.214 7f2b56bff700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b50095d80 con 0x7f2b500954c0 2021-02-10T11:42:18.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:18.214 7f2b56bff700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b500bfa60 con 0x7f2b500958c0 2021-02-10T11:42:18.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:18.214 7f2b56bff700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b50085fc0 con 0x7f2b50085bc0 2021-02-10T11:42:18.223 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:18.214 7f2b4ffff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2b500954c0 0x7f2b50096300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:18.223 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:18.214 7f2b4ffff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2b500954c0 0x7f2b50096300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:56126/0 (socket says 172.21.2.112:56126) 2021-02-10T11:42:18.223 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:18.214 7f2b4ffff700 1 -- 172.21.2.112:0/3720431560 learned_addr learned my addr 172.21.2.112:0/3720431560 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:42:18.223 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:18.214 7f2b4ffff700 1 -- 172.21.2.112:0/3720431560 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2b50085bc0 msgr2=0x7f2b500863e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:18.223 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:18.214 7f2b4ffff700 1 --2- 172.21.2.112:0/3720431560 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2b50085bc0 0x7f2b500863e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:18.223 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:18.214 7f2b4ffff700 1 -- 172.21.2.112:0/3720431560 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2b500958c0 msgr2=0x7f2b50096920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:18.223 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:18.214 7f2b4ffff700 1 --2- 172.21.2.112:0/3720431560 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2b500958c0 0x7f2b50096920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:18.223 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:18.214 7f2b4ffff700 1 -- 172.21.2.112:0/3720431560 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b50086c10 con 0x7f2b500954c0 2021-02-10T11:42:18.223 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:18.214 7f2b4ffff700 1 --2- 172.21.2.112:0/3720431560 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2b500954c0 0x7f2b50096300 secure :-1 s=READY pgs=2313 cs=0 l=1 rev1=1 rx=0x7f2b40009980 tx=0x7f2b4000bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:18.223 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:18.215 7f2b4d7fa700 1 -- 172.21.2.112:0/3720431560 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2b40006030 con 0x7f2b500954c0 2021-02-10T11:42:18.223 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:18.215 7f2b4d7fa700 1 -- 172.21.2.112:0/3720431560 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b40006370 con 0x7f2b500954c0 2021-02-10T11:42:18.223 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:18.215 7f2b4d7fa700 1 -- 172.21.2.112:0/3720431560 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2b40005660 con 0x7f2b500954c0 2021-02-10T11:42:18.223 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:18.215 7f2b56bff700 1 -- 172.21.2.112:0/3720431560 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b50086c10 con 0x7f2b500954c0 2021-02-10T11:42:18.223 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:18.215 7f2b56bff700 1 -- 172.21.2.112:0/3720431560 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b50086c10 con 0x7f2b500954c0 2021-02-10T11:42:18.224 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:18.215 7f2b4d7fa700 1 -- 172.21.2.112:0/3720431560 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2b40006550 con 0x7f2b500954c0 2021-02-10T11:42:18.224 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:18.215 7f2b4d7fa700 1 --2- 172.21.2.112:0/3720431560 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2b34018730 0x7f2b3401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:18.224 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:18.215 7f2b4d7fa700 1 -- 172.21.2.112:0/3720431560 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 5107+0+0 (secure 0 0 0) 0x7f2b40032850 con 0x7f2b500954c0 2021-02-10T11:42:18.224 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:18.216 7f2b54a1b700 1 --2- 172.21.2.112:0/3720431560 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2b34018730 0x7f2b3401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:18.224 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:18.216 7f2b54a1b700 1 --2- 172.21.2.112:0/3720431560 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2b34018730 0x7f2b3401abc0 crc :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:18.245 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:18.246 7f2b56bff700 1 -- 172.21.2.112:0/3720431560 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b500fe1c0 con 0x7f2b500954c0 2021-02-10T11:42:18.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:18.249 7f2b4d7fa700 1 -- 172.21.2.112:0/3720431560 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2b4000be10 con 0x7f2b500954c0 2021-02-10T11:42:18.376 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:18.377 7f2b56bff700 1 -- 172.21.2.112:0/3720431560 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"} v 0) v1 -- 0x7f2b500fe1c0 con 0x7f2b500954c0 2021-02-10T11:42:18.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:18.740 7f2b4d7fa700 1 -- 172.21.2.112:0/3720431560 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"}]=0 pool 'cephfs_data' created v123) v1 ==== 125+0+0 (secure 0 0 0) 0x7f2b4000be10 con 0x7f2b500954c0 2021-02-10T11:42:18.740 INFO:teuthology.orchestra.run.gibba012.stderr:pool 'cephfs_data' created 2021-02-10T11:42:18.743 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:18.744 7f2b56bff700 1 -- 172.21.2.112:0/3720431560 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2b34018730 msgr2=0x7f2b3401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:18.743 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:18.744 7f2b56bff700 1 --2- 172.21.2.112:0/3720431560 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2b34018730 0x7f2b3401abc0 crc :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:18.743 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:18.744 7f2b56bff700 1 -- 172.21.2.112:0/3720431560 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2b500954c0 msgr2=0x7f2b50096300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:18.743 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:18.744 7f2b56bff700 1 --2- 172.21.2.112:0/3720431560 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2b500954c0 0x7f2b50096300 secure :-1 s=READY pgs=2313 cs=0 l=1 rev1=1 rx=0x7f2b40009980 tx=0x7f2b4000bcd0).stop 2021-02-10T11:42:18.743 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:18.744 7f2b56bff700 1 -- 172.21.2.112:0/3720431560 shutdown_connections 2021-02-10T11:42:18.743 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:18.744 7f2b56bff700 1 --2- 172.21.2.112:0/3720431560 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2b34018730 0x7f2b3401abc0 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:18.743 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:18.744 7f2b56bff700 1 --2- 172.21.2.112:0/3720431560 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2b500958c0 0x7f2b50096920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:18.743 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:18.744 7f2b56bff700 1 --2- 172.21.2.112:0/3720431560 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2b500954c0 0x7f2b50096300 unknown :-1 s=CLOSED pgs=2313 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:18.743 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:18.744 7f2b56bff700 1 --2- 172.21.2.112:0/3720431560 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2b50085bc0 0x7f2b500863e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:18.743 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:18.744 7f2b56bff700 1 -- 172.21.2.112:0/3720431560 >> 172.21.2.112:0/3720431560 conn(0x7f2b500bef00 msgr2=0x7f2b5009d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:18.744 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:18.744 7f2b56bff700 1 -- 172.21.2.112:0/3720431560 shutdown_connections 2021-02-10T11:42:18.744 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:18.744 7f2b56bff700 1 -- 172.21.2.112:0/3720431560 wait complete. 2021-02-10T11:42:18.751 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2021-02-10T11:42:19.183 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.166 7fc69c663700 1 Processor -- start 2021-02-10T11:42:19.184 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.166 7fc69c663700 1 -- start start 2021-02-10T11:42:19.184 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.166 7fc69c663700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fc69410b000 0x7fc6940cf760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:19.184 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.166 7fc69c663700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6940fe1d0 con 0x7fc6940d1c80 2021-02-10T11:42:19.184 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.166 7fc69c663700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6941b0cc0 con 0x7fc6940d1880 2021-02-10T11:42:19.184 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.166 7fc69c663700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc694109a70 con 0x7fc69410b000 2021-02-10T11:42:19.184 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.166 7fc699c7e700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fc6940d1c80 0x7fc6940cef70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:55594/0 (socket says 172.21.2.112:55594) 2021-02-10T11:42:19.184 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.166 7fc699c7e700 1 -- 172.21.2.112:0/1387509001 learned_addr learned my addr 172.21.2.112:0/1387509001 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:42:19.184 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.167 7fc69947d700 1 -- 172.21.2.112:0/1387509001 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4168653266 0 0) 0x7fc684001180 con 0x7fc6940d1c80 2021-02-10T11:42:19.184 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.167 7fc69947d700 1 -- 172.21.2.112:0/1387509001 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc680002fe0 con 0x7fc6940d1c80 2021-02-10T11:42:19.184 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.167 7fc69ac80700 1 --2- 172.21.2.112:0/1387509001 >> v2:172.21.2.120:3300/0 conn(0x7fc69410b000 0x7fc6940cf760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:19.184 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.167 7fc69947d700 1 -- 172.21.2.112:0/1387509001 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 703340690 0 0) 0x7fc684001180 con 0x7fc6940d1c80 2021-02-10T11:42:19.185 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.167 7fc69947d700 1 -- 172.21.2.112:0/1387509001 >> v2:172.21.2.120:3300/0 conn(0x7fc69410b000 msgr2=0x7fc6940cf760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:19.185 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.167 7fc69947d700 1 --2- 172.21.2.112:0/1387509001 >> v2:172.21.2.120:3300/0 conn(0x7fc69410b000 0x7fc6940cf760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:19.185 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.167 7fc69947d700 1 -- 172.21.2.112:0/1387509001 >> v1:172.21.2.120:6789/0 conn(0x7fc6940d1880 legacy=0x7fc6941b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:19.185 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.167 7fc69947d700 1 -- 172.21.2.112:0/1387509001 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc69410b400 con 0x7fc6940d1c80 2021-02-10T11:42:19.185 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.167 7fc69947d700 1 -- 172.21.2.112:0/1387509001 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7fc684001680 con 0x7fc6940d1c80 2021-02-10T11:42:19.185 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.167 7fc69947d700 1 -- 172.21.2.112:0/1387509001 >> v1:172.21.2.112:6789/0 conn(0x7fc6940d1c80 legacy=0x7fc6940cef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:19.185 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.167 7fc69947d700 1 --2- 172.21.2.112:0/1387509001 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc680003b10 0x7fc680006010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:19.185 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.167 7fc69947d700 1 --2- 172.21.2.112:0/1387509001 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc680006830 0x7fc680008d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:19.185 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.167 7fc69947d700 1 --2- 172.21.2.112:0/1387509001 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc6800095d0 0x7fc68000bad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:19.185 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.167 7fc69947d700 1 -- 172.21.2.112:0/1387509001 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc68000c320 con 0x7fc680006830 2021-02-10T11:42:19.185 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.167 7fc69947d700 1 -- 172.21.2.112:0/1387509001 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc68000c540 con 0x7fc680003b10 2021-02-10T11:42:19.185 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.167 7fc69947d700 1 -- 172.21.2.112:0/1387509001 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc68000c760 con 0x7fc6800095d0 2021-02-10T11:42:19.185 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.167 7fc699c7e700 1 --2- 172.21.2.112:0/1387509001 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc680006830 0x7fc680008d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:19.186 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.167 7fc69a47f700 1 --2- 172.21.2.112:0/1387509001 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc680003b10 0x7fc680006010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:19.186 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.168 7fc69ac80700 1 --2- 172.21.2.112:0/1387509001 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc6800095d0 0x7fc68000bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:19.186 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.168 7fc69a47f700 1 -- 172.21.2.112:0/1387509001 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc6800095d0 msgr2=0x7fc68000bad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:19.186 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.168 7fc69a47f700 1 --2- 172.21.2.112:0/1387509001 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc6800095d0 0x7fc68000bad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:19.186 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.168 7fc69a47f700 1 -- 172.21.2.112:0/1387509001 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc680006830 msgr2=0x7fc680008d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:19.186 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.168 7fc69a47f700 1 --2- 172.21.2.112:0/1387509001 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc680006830 0x7fc680008d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:19.186 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.168 7fc69a47f700 1 -- 172.21.2.112:0/1387509001 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6800509a0 con 0x7fc680003b10 2021-02-10T11:42:19.186 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.168 7fc69ac80700 1 -- 172.21.2.112:0/1387509001 reap_dead start 2021-02-10T11:42:19.186 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.168 7fc69a47f700 1 --2- 172.21.2.112:0/1387509001 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc680003b10 0x7fc680006010 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 rx=0x7fc690006650 tx=0x7fc690009a80).ready entity=mon.1 client_cookie=5c04e2429933155b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:19.186 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.168 7fc69947d700 1 -- 172.21.2.112:0/1387509001 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc690002560 con 0x7fc680003b10 2021-02-10T11:42:19.186 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.168 7fc69947d700 1 -- 172.21.2.112:0/1387509001 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6900027e0 con 0x7fc680003b10 2021-02-10T11:42:19.186 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.168 7fc69947d700 1 -- 172.21.2.112:0/1387509001 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc690002c30 con 0x7fc680003b10 2021-02-10T11:42:19.186 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.168 7fc69c663700 1 -- 172.21.2.112:0/1387509001 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc680003b10 msgr2=0x7fc680006010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:19.187 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.168 7fc69c663700 1 --2- 172.21.2.112:0/1387509001 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc680003b10 0x7fc680006010 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 rx=0x7fc690006650 tx=0x7fc690009a80).stop 2021-02-10T11:42:19.187 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.169 7fc69c663700 1 -- 172.21.2.112:0/1387509001 shutdown_connections 2021-02-10T11:42:19.187 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.169 7fc69c663700 1 --2- 172.21.2.112:0/1387509001 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc680003b10 0x7fc680006010 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:19.187 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.169 7fc69c663700 1 -- 172.21.2.112:0/1387509001 >> 172.21.2.112:0/1387509001 conn(0x7fc6940ce430 msgr2=0x7fc6940d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:19.187 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.169 7fc69c663700 1 -- 172.21.2.112:0/1387509001 shutdown_connections 2021-02-10T11:42:19.187 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.169 7fc69c663700 1 -- 172.21.2.112:0/1387509001 wait complete. 2021-02-10T11:42:19.187 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.184 7fc69c663700 1 Processor -- start 2021-02-10T11:42:19.187 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.184 7fc69c663700 1 -- start start 2021-02-10T11:42:19.189 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.184 7fc69c663700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc6941a3570 0x7fc6941a3970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:19.189 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.184 7fc69c663700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc69409c9f0 0x7fc69409cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:19.189 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.184 7fc69c663700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc69409d410 0x7fc694098170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:19.189 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.184 7fc69c663700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc69409c470 con 0x7fc69409d410 2021-02-10T11:42:19.189 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.184 7fc69c663700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6940b2840 con 0x7fc69409c9f0 2021-02-10T11:42:19.189 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.184 7fc69c663700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6940b2a20 con 0x7fc6941a3570 2021-02-10T11:42:19.189 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.184 7fc69ac80700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc69409d410 0x7fc694098170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:19.189 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.184 7fc69ac80700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc69409d410 0x7fc694098170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:56148/0 (socket says 172.21.2.112:56148) 2021-02-10T11:42:19.190 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.184 7fc69ac80700 1 -- 172.21.2.112:0/2610227182 learned_addr learned my addr 172.21.2.112:0/2610227182 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:42:19.190 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.185 7fc69ac80700 1 -- 172.21.2.112:0/2610227182 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc6941a3570 msgr2=0x7fc6941a3970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:19.190 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.185 7fc69ac80700 1 --2- 172.21.2.112:0/2610227182 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc6941a3570 0x7fc6941a3970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:19.190 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.185 7fc69ac80700 1 -- 172.21.2.112:0/2610227182 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc69409c9f0 msgr2=0x7fc69409cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:19.190 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.185 7fc69ac80700 1 --2- 172.21.2.112:0/2610227182 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc69409c9f0 0x7fc69409cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:19.190 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.185 7fc69ac80700 1 -- 172.21.2.112:0/2610227182 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc69409d880 con 0x7fc69409d410 2021-02-10T11:42:19.190 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.185 7fc69ac80700 1 --2- 172.21.2.112:0/2610227182 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc69409d410 0x7fc694098170 secure :-1 s=READY pgs=2315 cs=0 l=1 rev1=1 rx=0x7fc68c00b970 tx=0x7fc68c00bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:19.190 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.185 7fc68affd700 1 -- 172.21.2.112:0/2610227182 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc68c000f90 con 0x7fc69409d410 2021-02-10T11:42:19.190 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.185 7fc69c663700 1 -- 172.21.2.112:0/2610227182 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc69409d880 con 0x7fc69409d410 2021-02-10T11:42:19.190 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.185 7fc68affd700 1 -- 172.21.2.112:0/2610227182 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc68c006bc0 con 0x7fc69409d410 2021-02-10T11:42:19.190 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.185 7fc68affd700 1 -- 172.21.2.112:0/2610227182 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc68c000f90 con 0x7fc69409d410 2021-02-10T11:42:19.190 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.185 7fc69c663700 1 -- 172.21.2.112:0/2610227182 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc69409d880 con 0x7fc69409d410 2021-02-10T11:42:19.190 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.186 7fc68affd700 1 -- 172.21.2.112:0/2610227182 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc68c0056b0 con 0x7fc69409d410 2021-02-10T11:42:19.191 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.186 7fc68affd700 1 --2- 172.21.2.112:0/2610227182 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc67c0186d0 0x7fc67c01abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:19.191 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.186 7fc68affd700 1 -- 172.21.2.112:0/2610227182 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 5457+0+0 (secure 0 0 0) 0x7fc68c032980 con 0x7fc69409d410 2021-02-10T11:42:19.191 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.187 7fc69a47f700 1 --2- 172.21.2.112:0/2610227182 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc67c0186d0 0x7fc67c01abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:19.191 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.187 7fc69a47f700 1 --2- 172.21.2.112:0/2610227182 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc67c0186d0 0x7fc67c01abd0 crc :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:19.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.216 7fc69c663700 1 -- 172.21.2.112:0/2610227182 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc694082a70 con 0x7fc69409d410 2021-02-10T11:42:19.218 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.218 7fc68affd700 1 -- 172.21.2.112:0/2610227182 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc68c000f00 con 0x7fc69409d410 2021-02-10T11:42:19.348 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.349 7fc69c663700 1 -- 172.21.2.112:0/2610227182 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"} v 0) v1 -- 0x7fc694082a70 con 0x7fc69409d410 2021-02-10T11:42:19.352 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.352 7fc68affd700 1 -- 172.21.2.112:0/2610227182 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"}]=0 new fs with metadata pool 26 and data pool 27 v102) v1 ==== 189+0+0 (secure 0 0 0) 0x7fc68c006ab0 con 0x7fc69409d410 2021-02-10T11:42:19.356 INFO:teuthology.orchestra.run.gibba012.stderr:new fs with metadata pool 26 and data pool 27 2021-02-10T11:42:19.358 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.358 7fc69c663700 1 -- 172.21.2.112:0/2610227182 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc67c0186d0 msgr2=0x7fc67c01abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:19.358 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.358 7fc69c663700 1 --2- 172.21.2.112:0/2610227182 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc67c0186d0 0x7fc67c01abd0 crc :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:19.358 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.358 7fc69c663700 1 -- 172.21.2.112:0/2610227182 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc69409d410 msgr2=0x7fc694098170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:19.358 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.358 7fc69c663700 1 --2- 172.21.2.112:0/2610227182 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc69409d410 0x7fc694098170 secure :-1 s=READY pgs=2315 cs=0 l=1 rev1=1 rx=0x7fc68c00b970 tx=0x7fc68c00bd10).stop 2021-02-10T11:42:19.358 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.358 7fc69c663700 1 -- 172.21.2.112:0/2610227182 shutdown_connections 2021-02-10T11:42:19.358 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.358 7fc69c663700 1 --2- 172.21.2.112:0/2610227182 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc67c0186d0 0x7fc67c01abd0 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:19.358 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.358 7fc69c663700 1 --2- 172.21.2.112:0/2610227182 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc69409c9f0 0x7fc69409cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:19.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.358 7fc69c663700 1 --2- 172.21.2.112:0/2610227182 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc69409d410 0x7fc694098170 unknown :-1 s=CLOSED pgs=2315 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:19.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.358 7fc69c663700 1 --2- 172.21.2.112:0/2610227182 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc6941a3570 0x7fc6941a3970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:19.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.358 7fc69c663700 1 -- 172.21.2.112:0/2610227182 >> 172.21.2.112:0/2610227182 conn(0x7fc6940c2cd0 msgr2=0x7fc6940b5950 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:19.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.358 7fc69c663700 1 -- 172.21.2.112:0/2610227182 shutdown_connections 2021-02-10T11:42:19.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.358 7fc69c663700 1 -- 172.21.2.112:0/2610227182 wait complete. 2021-02-10T11:42:19.366 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:42:19.793 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.777 7fd0b62df700 1 Processor -- start 2021-02-10T11:42:19.793 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.777 7fd0b62df700 1 -- start start 2021-02-10T11:42:19.793 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.778 7fd0b62df700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fd0b00d1870 0x7fd0b01b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:19.793 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.778 7fd0b62df700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd0b00f0650 0x7fd0b00cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:19.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.778 7fd0b62df700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0b00f0b10 con 0x7fd0b01b0a10 2021-02-10T11:42:19.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.778 7fd0b62df700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0b00e5ed0 con 0x7fd0b00d1870 2021-02-10T11:42:19.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.778 7fd0b62df700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0b00d1c70 con 0x7fd0b00f0650 2021-02-10T11:42:19.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.778 7fd0affff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fd0b00d1870 0x7fd0b01b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:19.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.778 7fd0af7fe700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fd0b01b0a10 0x7fd0b00cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:55616/0 (socket says 172.21.2.112:55616) 2021-02-10T11:42:19.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.778 7fd0af7fe700 1 -- 172.21.2.112:0/2513643258 learned_addr learned my addr 172.21.2.112:0/2513643258 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:42:19.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.778 7fd0affff700 1 -- 172.21.2.112:0/2513643258 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd0b00f0650 msgr2=0x7fd0b00cfb20 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:42:19.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.778 7fd0affff700 1 --2- 172.21.2.112:0/2513643258 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd0b00f0650 0x7fd0b00cfb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:19.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.778 7fd0affff700 1 -- 172.21.2.112:0/2513643258 >> v1:172.21.2.112:6789/0 conn(0x7fd0b01b0a10 legacy=0x7fd0b00cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:19.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.778 7fd0affff700 1 -- 172.21.2.112:0/2513643258 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0b00d1e90 con 0x7fd0b00d1870 2021-02-10T11:42:19.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.778 7fd0affff700 1 --2- 172.21.2.112:0/2513643258 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd0b00d1870 0x7fd0b01b03f0 secure :-1 s=READY pgs=2317 cs=0 l=1 rev1=1 rx=0x7fd0a0006650 tx=0x7fd0a0009a00).ready entity=mon.0 client_cookie=50ce6123f75ed9e4 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:19.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.779 7fd0aeffd700 1 -- 172.21.2.112:0/2513643258 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd0a0009df0 con 0x7fd0b00d1870 2021-02-10T11:42:19.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.779 7fd0aeffd700 1 -- 172.21.2.112:0/2513643258 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0a00009b0 con 0x7fd0b00d1870 2021-02-10T11:42:19.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.779 7fd0aeffd700 1 -- 172.21.2.112:0/2513643258 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd0a000c2b0 con 0x7fd0b00d1870 2021-02-10T11:42:19.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.779 7fd0b62df700 1 -- 172.21.2.112:0/2513643258 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd0b00d1870 msgr2=0x7fd0b01b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:19.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.779 7fd0b62df700 1 --2- 172.21.2.112:0/2513643258 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd0b00d1870 0x7fd0b01b03f0 secure :-1 s=READY pgs=2317 cs=0 l=1 rev1=1 rx=0x7fd0a0006650 tx=0x7fd0a0009a00).stop 2021-02-10T11:42:19.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.779 7fd0b62df700 1 -- 172.21.2.112:0/2513643258 shutdown_connections 2021-02-10T11:42:19.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.779 7fd0b62df700 1 --2- 172.21.2.112:0/2513643258 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd0b00f0650 0x7fd0b00cfb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:19.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.779 7fd0b62df700 1 --2- 172.21.2.112:0/2513643258 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd0b00d1870 0x7fd0b01b03f0 unknown :-1 s=CLOSED pgs=2317 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:19.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.779 7fd0b62df700 1 -- 172.21.2.112:0/2513643258 >> 172.21.2.112:0/2513643258 conn(0x7fd0b00ce450 msgr2=0x7fd0b00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:19.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.779 7fd0b62df700 1 -- 172.21.2.112:0/2513643258 shutdown_connections 2021-02-10T11:42:19.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.779 7fd0b62df700 1 -- 172.21.2.112:0/2513643258 wait complete. 2021-02-10T11:42:19.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.793 7fd0b62df700 1 Processor -- start 2021-02-10T11:42:19.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.793 7fd0b62df700 1 -- start start 2021-02-10T11:42:19.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.794 7fd0b62df700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd0b018a540 0x7fd0b0197480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:19.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.794 7fd0b62df700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd0b018aa50 0x7fd0b00b77d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:19.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.794 7fd0b62df700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd0b00b7df0 0x7fd0b00b81f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:19.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.794 7fd0b62df700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0b01a79d0 con 0x7fd0b018a540 2021-02-10T11:42:19.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.794 7fd0b62df700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0b01a9e20 con 0x7fd0b00b7df0 2021-02-10T11:42:19.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.794 7fd0b62df700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0b01aa780 con 0x7fd0b018aa50 2021-02-10T11:42:19.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.794 7fd0affff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd0b018a540 0x7fd0b0197480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:19.800 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.794 7fd0affff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd0b018a540 0x7fd0b0197480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:56158/0 (socket says 172.21.2.112:56158) 2021-02-10T11:42:19.800 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.794 7fd0affff700 1 -- 172.21.2.112:0/3291672588 learned_addr learned my addr 172.21.2.112:0/3291672588 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:42:19.800 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.794 7fd0affff700 1 -- 172.21.2.112:0/3291672588 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd0b018aa50 msgr2=0x7fd0b00b77d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:42:19.800 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.794 7fd0affff700 1 --2- 172.21.2.112:0/3291672588 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd0b018aa50 0x7fd0b00b77d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:19.800 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.794 7fd0affff700 1 -- 172.21.2.112:0/3291672588 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd0b00b7df0 msgr2=0x7fd0b00b81f0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:42:19.800 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.794 7fd0affff700 1 --2- 172.21.2.112:0/3291672588 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd0b00b7df0 0x7fd0b00b81f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:19.801 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.794 7fd0affff700 1 -- 172.21.2.112:0/3291672588 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0b01aa9a0 con 0x7fd0b018a540 2021-02-10T11:42:19.801 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.794 7fd0affff700 1 --2- 172.21.2.112:0/3291672588 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd0b018a540 0x7fd0b0197480 secure :-1 s=READY pgs=2318 cs=0 l=1 rev1=1 rx=0x7fd0a00096a0 tx=0x7fd0a000b8e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:19.801 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.794 7fd0acff9700 1 -- 172.21.2.112:0/3291672588 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd0a00093a0 con 0x7fd0b018a540 2021-02-10T11:42:19.801 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.794 7fd0acff9700 1 -- 172.21.2.112:0/3291672588 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0a00067f0 con 0x7fd0b018a540 2021-02-10T11:42:19.801 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.794 7fd0b62df700 1 -- 172.21.2.112:0/3291672588 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0b01aa9a0 con 0x7fd0b018a540 2021-02-10T11:42:19.801 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.794 7fd0acff9700 1 -- 172.21.2.112:0/3291672588 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd0a00022b0 con 0x7fd0b018a540 2021-02-10T11:42:19.801 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.794 7fd0b62df700 1 -- 172.21.2.112:0/3291672588 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0b01aa9a0 con 0x7fd0b018a540 2021-02-10T11:42:19.801 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.795 7fd0acff9700 1 -- 172.21.2.112:0/3291672588 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd0a00052e0 con 0x7fd0b018a540 2021-02-10T11:42:19.801 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.795 7fd0acff9700 1 --2- 172.21.2.112:0/3291672588 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd08c0186c0 0x7fd08c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:19.802 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.795 7fd0acff9700 1 -- 172.21.2.112:0/3291672588 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fd0a00325a0 con 0x7fd0b018a540 2021-02-10T11:42:19.802 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.795 7fd0b62df700 1 -- 172.21.2.112:0/3291672588 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0b00ce450 con 0x7fd0b018a540 2021-02-10T11:42:19.802 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.796 7fd0af7fe700 1 --2- 172.21.2.112:0/3291672588 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd08c0186c0 0x7fd08c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:19.802 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.796 7fd0af7fe700 1 --2- 172.21.2.112:0/3291672588 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd08c0186c0 0x7fd08c01abc0 crc :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:19.802 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.798 7fd0acff9700 1 -- 172.21.2.112:0/3291672588 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd0a0030d50 con 0x7fd0b018a540 2021-02-10T11:42:19.923 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.924 7fd0b62df700 1 -- 172.21.2.112:0/3291672588 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd0b00ce450 con 0x7fd0b018a540 2021-02-10T11:42:19.924 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.924 7fd0acff9700 1 -- 172.21.2.112:0/3291672588 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v124) v1 ==== 74+0+14020 (secure 0 0 0) 0x7fd0a00052e0 con 0x7fd0b018a540 2021-02-10T11:42:19.924 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:42:19.925 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":124,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:42:19.351264","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":27,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:42:17.322526","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"124","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":27,"pool_name":"cephfs_data","create_time":"2021-02-10 11:42:18.395545","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"124","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:42:19.926 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.926 7fd0b62df700 1 -- 172.21.2.112:0/3291672588 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd08c0186c0 msgr2=0x7fd08c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:19.926 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.926 7fd0b62df700 1 --2- 172.21.2.112:0/3291672588 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd08c0186c0 0x7fd08c01abc0 crc :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:19.926 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.926 7fd0b62df700 1 -- 172.21.2.112:0/3291672588 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd0b018a540 msgr2=0x7fd0b0197480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:19.926 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.926 7fd0b62df700 1 --2- 172.21.2.112:0/3291672588 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd0b018a540 0x7fd0b0197480 secure :-1 s=READY pgs=2318 cs=0 l=1 rev1=1 rx=0x7fd0a00096a0 tx=0x7fd0a000b8e0).stop 2021-02-10T11:42:19.926 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.926 7fd0b62df700 1 -- 172.21.2.112:0/3291672588 shutdown_connections 2021-02-10T11:42:19.926 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.926 7fd0b62df700 1 --2- 172.21.2.112:0/3291672588 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd08c0186c0 0x7fd08c01abc0 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:19.926 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.926 7fd0b62df700 1 --2- 172.21.2.112:0/3291672588 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd0b00b7df0 0x7fd0b00b81f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:19.927 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.926 7fd0b62df700 1 --2- 172.21.2.112:0/3291672588 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd0b018aa50 0x7fd0b00b77d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:19.927 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.926 7fd0b62df700 1 --2- 172.21.2.112:0/3291672588 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd0b018a540 0x7fd0b0197480 unknown :-1 s=CLOSED pgs=2318 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:19.927 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.926 7fd0b62df700 1 -- 172.21.2.112:0/3291672588 >> 172.21.2.112:0/3291672588 conn(0x7fd0b00efc70 msgr2=0x7fd0b00ef410 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:19.927 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.926 7fd0b62df700 1 -- 172.21.2.112:0/3291672588 shutdown_connections 2021-02-10T11:42:19.927 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:19.927 7fd0b62df700 1 -- 172.21.2.112:0/3291672588 wait complete. 2021-02-10T11:42:19.934 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:42:20.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.346 7f35bcbed700 1 Processor -- start 2021-02-10T11:42:20.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.346 7f35bcbed700 1 -- start start 2021-02-10T11:42:20.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.346 7f35bcbed700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f35b80d1c70 0x7f35b80f20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:20.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.346 7f35bcbed700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35b80f0b10 con 0x7f35b80d0340 2021-02-10T11:42:20.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.346 7f35bcbed700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35b81b0be0 con 0x7f35b80d1870 2021-02-10T11:42:20.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.346 7f35bcbed700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35b80e5ed0 con 0x7f35b80d1c70 2021-02-10T11:42:20.365 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.347 7f35b705f700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f35b80d1c70 0x7f35b80f20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:20.365 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.347 7f35b705f700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f35b80d1c70 0x7f35b80f20a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:56170/0 (socket says 172.21.2.112:56170) 2021-02-10T11:42:20.365 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.347 7f35b705f700 1 -- 172.21.2.112:0/745169706 learned_addr learned my addr 172.21.2.112:0/745169706 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:42:20.365 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.347 7f35b705f700 1 -- 172.21.2.112:0/745169706 >> v1:172.21.2.120:6789/0 conn(0x7f35b80d1870 legacy=0x7f35b81b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:20.365 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.347 7f35b705f700 1 -- 172.21.2.112:0/745169706 >> v1:172.21.2.112:6789/0 conn(0x7f35b80d0340 legacy=0x7f35b80ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:20.365 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.347 7f35b705f700 1 -- 172.21.2.112:0/745169706 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35b80efe30 con 0x7f35b80d1c70 2021-02-10T11:42:20.365 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.347 7f35b705f700 1 --2- 172.21.2.112:0/745169706 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f35b80d1c70 0x7f35b80f20a0 secure :-1 s=READY pgs=2320 cs=0 l=1 rev1=1 rx=0x7f35b0006f00 tx=0x7f35b0009f10).ready entity=mon.0 client_cookie=6248641f742a7b04 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:20.365 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.348 7f35b585c700 1 -- 172.21.2.112:0/745169706 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f35b000b3b0 con 0x7f35b80d1c70 2021-02-10T11:42:20.365 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.348 7f35b585c700 1 -- 172.21.2.112:0/745169706 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35b000b630 con 0x7f35b80d1c70 2021-02-10T11:42:20.365 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.348 7f35b585c700 1 -- 172.21.2.112:0/745169706 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f35b000bb50 con 0x7f35b80d1c70 2021-02-10T11:42:20.365 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.348 7f35bcbed700 1 -- 172.21.2.112:0/745169706 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f35b80d1c70 msgr2=0x7f35b80f20a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:20.365 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.348 7f35bcbed700 1 --2- 172.21.2.112:0/745169706 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f35b80d1c70 0x7f35b80f20a0 secure :-1 s=READY pgs=2320 cs=0 l=1 rev1=1 rx=0x7f35b0006f00 tx=0x7f35b0009f10).stop 2021-02-10T11:42:20.365 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.349 7f35bcbed700 1 -- 172.21.2.112:0/745169706 shutdown_connections 2021-02-10T11:42:20.365 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.349 7f35bcbed700 1 --2- 172.21.2.112:0/745169706 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f35b80d1c70 0x7f35b80f20a0 unknown :-1 s=CLOSED pgs=2320 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:20.366 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.349 7f35bcbed700 1 -- 172.21.2.112:0/745169706 >> 172.21.2.112:0/745169706 conn(0x7f35b80ce450 msgr2=0x7f35b80d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:20.366 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.349 7f35bcbed700 1 -- 172.21.2.112:0/745169706 shutdown_connections 2021-02-10T11:42:20.366 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.349 7f35bcbed700 1 -- 172.21.2.112:0/745169706 wait complete. 2021-02-10T11:42:20.366 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.364 7f35bcbed700 1 Processor -- start 2021-02-10T11:42:20.366 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.364 7f35bcbed700 1 -- start start 2021-02-10T11:42:20.367 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.365 7f35bcbed700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f35b80af040 0x7f35b80af440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:20.367 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.365 7f35bcbed700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f35b80afa60 0x7f35b80ada20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:20.368 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.365 7f35bcbed700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f35b80afe60 0x7f35b80ae040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:20.368 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.365 7f35bcbed700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35b80923c0 con 0x7f35b80afa60 2021-02-10T11:42:20.368 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.365 7f35bcbed700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35b80b0410 con 0x7f35b80afe60 2021-02-10T11:42:20.368 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.365 7f35bcbed700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35b8091e40 con 0x7f35b80af040 2021-02-10T11:42:20.368 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.365 7f35b605d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f35b80afa60 0x7f35b80ada20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:20.368 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.365 7f35b685e700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f35b80af040 0x7f35b80af440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:20.368 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.365 7f35b605d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f35b80afa60 0x7f35b80ada20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:56174/0 (socket says 172.21.2.112:56174) 2021-02-10T11:42:20.368 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.365 7f35b605d700 1 -- 172.21.2.112:0/506924182 learned_addr learned my addr 172.21.2.112:0/506924182 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:42:20.368 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.365 7f35b705f700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f35b80afe60 0x7f35b80ae040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:20.368 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.365 7f35b605d700 1 -- 172.21.2.112:0/506924182 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f35b80af040 msgr2=0x7f35b80af440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:20.368 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.365 7f35b605d700 1 --2- 172.21.2.112:0/506924182 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f35b80af040 0x7f35b80af440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:20.368 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.365 7f35b605d700 1 -- 172.21.2.112:0/506924182 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f35b80afe60 msgr2=0x7f35b80ae040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:20.369 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.365 7f35b605d700 1 --2- 172.21.2.112:0/506924182 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f35b80afe60 0x7f35b80ae040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:20.369 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.365 7f35b605d700 1 -- 172.21.2.112:0/506924182 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35b808f6d0 con 0x7f35b80afa60 2021-02-10T11:42:20.369 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.366 7f35b605d700 1 --2- 172.21.2.112:0/506924182 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f35b80afa60 0x7f35b80ada20 secure :-1 s=READY pgs=2321 cs=0 l=1 rev1=1 rx=0x7f35ac006ed0 tx=0x7f35ac00b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:20.369 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.366 7f35a77fe700 1 -- 172.21.2.112:0/506924182 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f35ac006030 con 0x7f35b80afa60 2021-02-10T11:42:20.369 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.366 7f35a77fe700 1 -- 172.21.2.112:0/506924182 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35ac006330 con 0x7f35b80afa60 2021-02-10T11:42:20.369 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.366 7f35a77fe700 1 -- 172.21.2.112:0/506924182 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f35ac005730 con 0x7f35b80afa60 2021-02-10T11:42:20.369 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.366 7f35bcbed700 1 -- 172.21.2.112:0/506924182 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35b808f6d0 con 0x7f35b80afa60 2021-02-10T11:42:20.369 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.366 7f35bcbed700 1 -- 172.21.2.112:0/506924182 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35b808f6d0 con 0x7f35b80afa60 2021-02-10T11:42:20.369 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.367 7f35a77fe700 1 -- 172.21.2.112:0/506924182 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f35ac006030 con 0x7f35b80afa60 2021-02-10T11:42:20.369 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.367 7f35a77fe700 1 --2- 172.21.2.112:0/506924182 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f359c0186c0 0x7f359c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:20.369 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.367 7f35a77fe700 1 -- 172.21.2.112:0/506924182 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f35ac032560 con 0x7f35b80afa60 2021-02-10T11:42:20.370 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.367 7f35b685e700 1 --2- 172.21.2.112:0/506924182 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f359c0186c0 0x7f359c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:20.370 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.367 7f35b685e700 1 --2- 172.21.2.112:0/506924182 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f359c0186c0 0x7f359c01abc0 crc :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:20.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.396 7f35bcbed700 1 -- 172.21.2.112:0/506924182 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35b80e7df0 con 0x7f35b80afa60 2021-02-10T11:42:20.398 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.399 7f35a77fe700 1 -- 172.21.2.112:0/506924182 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f35ac02eee0 con 0x7f35b80afa60 2021-02-10T11:42:20.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.526 7f35bcbed700 1 -- 172.21.2.112:0/506924182 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f35b80ce450 con 0x7f35b80afa60 2021-02-10T11:42:20.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.527 7f35a77fe700 1 -- 172.21.2.112:0/506924182 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v125) v1 ==== 74+0+14007 (secure 0 0 0) 0x7f35ac02eee0 con 0x7f35b80afa60 2021-02-10T11:42:20.527 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:42:20.527 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":125,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:42:20.353491","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":27,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:42:17.322526","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"124","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":27,"pool_name":"cephfs_data","create_time":"2021-02-10 11:42:18.395545","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"125","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:42:20.528 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.528 7f35bcbed700 1 -- 172.21.2.112:0/506924182 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f359c0186c0 msgr2=0x7f359c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:20.528 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.528 7f35bcbed700 1 --2- 172.21.2.112:0/506924182 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f359c0186c0 0x7f359c01abc0 crc :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:20.529 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.528 7f35bcbed700 1 -- 172.21.2.112:0/506924182 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f35b80afa60 msgr2=0x7f35b80ada20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:20.529 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.528 7f35bcbed700 1 --2- 172.21.2.112:0/506924182 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f35b80afa60 0x7f35b80ada20 secure :-1 s=READY pgs=2321 cs=0 l=1 rev1=1 rx=0x7f35ac006ed0 tx=0x7f35ac00b960).stop 2021-02-10T11:42:20.529 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.529 7f35bcbed700 1 -- 172.21.2.112:0/506924182 shutdown_connections 2021-02-10T11:42:20.529 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.529 7f35bcbed700 1 --2- 172.21.2.112:0/506924182 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f359c0186c0 0x7f359c01abc0 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:20.529 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.529 7f35bcbed700 1 --2- 172.21.2.112:0/506924182 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f35b80afe60 0x7f35b80ae040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:20.529 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.529 7f35bcbed700 1 --2- 172.21.2.112:0/506924182 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f35b80afa60 0x7f35b80ada20 unknown :-1 s=CLOSED pgs=2321 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:20.529 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.529 7f35bcbed700 1 --2- 172.21.2.112:0/506924182 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f35b80af040 0x7f35b80af440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:20.529 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.529 7f35bcbed700 1 -- 172.21.2.112:0/506924182 >> 172.21.2.112:0/506924182 conn(0x7f35b80a0030 msgr2=0x7f35b80f22b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:20.529 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.529 7f35bcbed700 1 -- 172.21.2.112:0/506924182 shutdown_connections 2021-02-10T11:42:20.529 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.529 7f35bcbed700 1 -- 172.21.2.112:0/506924182 wait complete. 2021-02-10T11:42:20.537 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2021-02-10T11:42:20.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.953 7fd401475700 1 Processor -- start 2021-02-10T11:42:20.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.953 7fd401475700 1 -- start start 2021-02-10T11:42:20.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.953 7fd401475700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd3fc10aff0 0x7fd3fc0cf750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:20.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.954 7fd401475700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3fc0fe1c0 con 0x7fd3fc0d1870 2021-02-10T11:42:20.971 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.954 7fd401475700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3fc1b0cb0 con 0x7fd3fc0d1c70 2021-02-10T11:42:20.971 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.954 7fd401475700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3fc109a60 con 0x7fd3fc10aff0 2021-02-10T11:42:20.971 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.954 7fd3fb07d700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fd3fc0d1870 0x7fd3fc1b04c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:55640/0 (socket says 172.21.2.112:55640) 2021-02-10T11:42:20.971 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.954 7fd3fb07d700 1 -- 172.21.2.112:0/1156109264 learned_addr learned my addr 172.21.2.112:0/1156109264 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:42:20.971 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.954 7fd3fa07b700 1 -- 172.21.2.112:0/1156109264 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3171265510 0 0) 0x7fd3ec001180 con 0x7fd3fc0d1870 2021-02-10T11:42:20.971 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.954 7fd3fa07b700 1 -- 172.21.2.112:0/1156109264 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3e4002fe0 con 0x7fd3fc0d1870 2021-02-10T11:42:20.971 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.954 7fd3fb87e700 1 --2- 172.21.2.112:0/1156109264 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd3fc10aff0 0x7fd3fc0cf750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:20.971 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.955 7fd3fa07b700 1 -- 172.21.2.112:0/1156109264 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1286872066 0 0) 0x7fd3ec001180 con 0x7fd3fc0d1870 2021-02-10T11:42:20.971 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.955 7fd3fa07b700 1 -- 172.21.2.112:0/1156109264 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd3fc10aff0 msgr2=0x7fd3fc0cf750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:20.971 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.955 7fd3fa07b700 1 --2- 172.21.2.112:0/1156109264 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd3fc10aff0 0x7fd3fc0cf750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:20.971 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.955 7fd3fa07b700 1 -- 172.21.2.112:0/1156109264 >> v1:172.21.2.120:6789/0 conn(0x7fd3fc0d1c70 legacy=0x7fd3fc0cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:20.972 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.955 7fd3fa07b700 1 -- 172.21.2.112:0/1156109264 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3fc10b3f0 con 0x7fd3fc0d1870 2021-02-10T11:42:20.972 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.955 7fd3fa07b700 1 -- 172.21.2.112:0/1156109264 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7fd3ec001680 con 0x7fd3fc0d1870 2021-02-10T11:42:20.972 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.955 7fd3fa07b700 1 -- 172.21.2.112:0/1156109264 >> v1:172.21.2.112:6789/0 conn(0x7fd3fc0d1870 legacy=0x7fd3fc1b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:20.972 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.955 7fd3fa07b700 1 --2- 172.21.2.112:0/1156109264 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd3e4003ae0 0x7fd3e4005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:20.972 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.955 7fd3fa07b700 1 --2- 172.21.2.112:0/1156109264 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd3e4006790 0x7fd3e4008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:20.972 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.955 7fd3fa07b700 1 --2- 172.21.2.112:0/1156109264 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd3e4009590 0x7fd3e400ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:20.972 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.955 7fd3fa07b700 1 -- 172.21.2.112:0/1156109264 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3e400c2d0 con 0x7fd3e4006790 2021-02-10T11:42:20.972 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.955 7fd3fa07b700 1 -- 172.21.2.112:0/1156109264 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3e400c4f0 con 0x7fd3e4003ae0 2021-02-10T11:42:20.972 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.955 7fd3fa07b700 1 -- 172.21.2.112:0/1156109264 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3e400c710 con 0x7fd3e4009590 2021-02-10T11:42:20.972 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.955 7fd3fa87c700 1 --2- 172.21.2.112:0/1156109264 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd3e4006790 0x7fd3e4008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:20.972 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.955 7fd3fb07d700 1 --2- 172.21.2.112:0/1156109264 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd3e4003ae0 0x7fd3e4005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:20.972 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.955 7fd3fb87e700 1 --2- 172.21.2.112:0/1156109264 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd3e4009590 0x7fd3e400ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:20.972 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.955 7fd3fa87c700 1 -- 172.21.2.112:0/1156109264 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd3e4009590 msgr2=0x7fd3e400ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:20.973 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.955 7fd3fa87c700 1 --2- 172.21.2.112:0/1156109264 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd3e4009590 0x7fd3e400ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:20.973 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.955 7fd3fa87c700 1 -- 172.21.2.112:0/1156109264 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd3e4003ae0 msgr2=0x7fd3e4005ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:20.973 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.955 7fd3fa87c700 1 --2- 172.21.2.112:0/1156109264 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd3e4003ae0 0x7fd3e4005ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:20.973 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.955 7fd3fa87c700 1 -- 172.21.2.112:0/1156109264 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3e4050980 con 0x7fd3e4006790 2021-02-10T11:42:20.973 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.955 7fd3fa87c700 1 --2- 172.21.2.112:0/1156109264 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd3e4006790 0x7fd3e4008c90 secure :-1 s=READY pgs=2323 cs=0 l=1 rev1=1 rx=0x7fd3f0006ab0 tx=0x7fd3f0009b40).ready entity=mon.0 client_cookie=2a37789716becf40 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:20.973 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.955 7fd3fa07b700 1 -- 172.21.2.112:0/1156109264 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd3f000b2e0 con 0x7fd3e4006790 2021-02-10T11:42:20.973 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.955 7fd3fa07b700 1 -- 172.21.2.112:0/1156109264 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3f000b5e0 con 0x7fd3e4006790 2021-02-10T11:42:20.973 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.955 7fd3fa07b700 1 -- 172.21.2.112:0/1156109264 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd3f0000960 con 0x7fd3e4006790 2021-02-10T11:42:20.973 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.955 7fd401475700 1 -- 172.21.2.112:0/1156109264 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd3e4006790 msgr2=0x7fd3e4008c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:20.973 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.955 7fd401475700 1 --2- 172.21.2.112:0/1156109264 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd3e4006790 0x7fd3e4008c90 secure :-1 s=READY pgs=2323 cs=0 l=1 rev1=1 rx=0x7fd3f0006ab0 tx=0x7fd3f0009b40).stop 2021-02-10T11:42:20.973 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.955 7fd3fb87e700 1 -- 172.21.2.112:0/1156109264 reap_dead start 2021-02-10T11:42:20.973 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.956 7fd401475700 1 -- 172.21.2.112:0/1156109264 shutdown_connections 2021-02-10T11:42:20.973 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.956 7fd401475700 1 -- 172.21.2.112:0/1156109264 >> 172.21.2.112:0/1156109264 conn(0x7fd3fc0ce420 msgr2=0x7fd3fc0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:20.973 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.956 7fd401475700 1 -- 172.21.2.112:0/1156109264 shutdown_connections 2021-02-10T11:42:20.974 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.956 7fd401475700 1 -- 172.21.2.112:0/1156109264 wait complete. 2021-02-10T11:42:20.974 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.971 7fd401475700 1 Processor -- start 2021-02-10T11:42:20.975 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.971 7fd401475700 1 -- start start 2021-02-10T11:42:20.976 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.971 7fd401475700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd3fc1a79a0 0x7fd3fc1a7da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:20.976 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.971 7fd401475700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd3fc0bdb40 0x7fd3fc0bdf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:20.976 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.971 7fd401475700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd3fc082ca0 0x7fd3fc0830a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:20.976 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.971 7fd401475700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3fc19d990 con 0x7fd3fc1a79a0 2021-02-10T11:42:20.976 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.971 7fd401475700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3fc19dbb0 con 0x7fd3fc0bdb40 2021-02-10T11:42:20.976 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.971 7fd401475700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3fc0836c0 con 0x7fd3fc082ca0 2021-02-10T11:42:20.976 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.971 7fd3fb07d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd3fc1a79a0 0x7fd3fc1a7da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:20.976 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.971 7fd3fb07d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd3fc1a79a0 0x7fd3fc1a7da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:56190/0 (socket says 172.21.2.112:56190) 2021-02-10T11:42:20.976 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.971 7fd3fb07d700 1 -- 172.21.2.112:0/1140942605 learned_addr learned my addr 172.21.2.112:0/1140942605 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:42:20.976 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.971 7fd3fb87e700 1 --2- 172.21.2.112:0/1140942605 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd3fc082ca0 0x7fd3fc0830a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:20.976 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.972 7fd3fa87c700 1 --2- 172.21.2.112:0/1140942605 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd3fc0bdb40 0x7fd3fc0bdf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:20.976 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.972 7fd3fb87e700 1 -- 172.21.2.112:0/1140942605 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd3fc0bdb40 msgr2=0x7fd3fc0bdf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:20.976 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.972 7fd3fb87e700 1 --2- 172.21.2.112:0/1140942605 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd3fc0bdb40 0x7fd3fc0bdf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:20.977 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.972 7fd3fb87e700 1 -- 172.21.2.112:0/1140942605 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd3fc1a79a0 msgr2=0x7fd3fc1a7da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:20.977 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.972 7fd3fb87e700 1 --2- 172.21.2.112:0/1140942605 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd3fc1a79a0 0x7fd3fc1a7da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:20.977 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.972 7fd3fb87e700 1 -- 172.21.2.112:0/1140942605 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3fc0838e0 con 0x7fd3fc082ca0 2021-02-10T11:42:20.977 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.972 7fd3fb87e700 1 --2- 172.21.2.112:0/1140942605 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd3fc082ca0 0x7fd3fc0830a0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7fd3f4006fc0 tx=0x7fd3f400bcf0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:20.977 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.972 7fd3e3fff700 1 -- 172.21.2.112:0/1140942605 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd3f4006c80 con 0x7fd3fc082ca0 2021-02-10T11:42:20.977 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.972 7fd3e3fff700 1 -- 172.21.2.112:0/1140942605 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3f4003030 con 0x7fd3fc082ca0 2021-02-10T11:42:20.977 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.972 7fd3e3fff700 1 -- 172.21.2.112:0/1140942605 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd3f4001490 con 0x7fd3fc082ca0 2021-02-10T11:42:20.977 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.972 7fd401475700 1 -- 172.21.2.112:0/1140942605 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3fc0838e0 con 0x7fd3fc082ca0 2021-02-10T11:42:20.977 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.972 7fd401475700 1 -- 172.21.2.112:0/1140942605 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3fc0838e0 con 0x7fd3fc082ca0 2021-02-10T11:42:20.977 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.973 7fd3e3fff700 1 -- 172.21.2.112:0/1140942605 <== mon.2 v2:172.21.2.120:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd3f4006c80 con 0x7fd3fc082ca0 2021-02-10T11:42:20.977 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.973 7fd3e3fff700 1 --2- 172.21.2.112:0/1140942605 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd3dc0186c0 0x7fd3dc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:20.977 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.973 7fd3e3fff700 1 -- 172.21.2.112:0/1140942605 <== mon.2 v2:172.21.2.120:3301/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fd3f40327e0 con 0x7fd3fc082ca0 2021-02-10T11:42:20.978 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.973 7fd3fb07d700 1 --2- 172.21.2.112:0/1140942605 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd3dc0186c0 0x7fd3dc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:20.978 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:20.973 7fd3fb07d700 1 --2- 172.21.2.112:0/1140942605 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd3dc0186c0 0x7fd3dc01abc0 crc :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:21.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:21.003 7fd401475700 1 -- 172.21.2.112:0/1140942605 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3fc082a70 con 0x7fd3fc082ca0 2021-02-10T11:42:21.006 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:21.006 7fd3e3fff700 1 -- 172.21.2.112:0/1140942605 <== mon.2 v2:172.21.2.120:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd3f400be30 con 0x7fd3fc082ca0 2021-02-10T11:42:21.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:21.143 7fd401475700 1 -- 172.21.2.112:0/1140942605 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_command({"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"} v 0) v1 -- 0x7fd3fc082a70 con 0x7fd3fc082ca0 2021-02-10T11:42:21.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:21.360 7fd3e3fff700 1 -- 172.21.2.112:0/1140942605 <== mon.2 v2:172.21.2.120:3301/0 7 ==== mon_command_ack([{"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"}]=0 v105) v1 ==== 118+0+0 (secure 0 0 0) 0x7fd3f4006c80 con 0x7fd3fc082ca0 2021-02-10T11:42:21.411 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:21.411 7fd401475700 1 -- 172.21.2.112:0/1140942605 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd3dc0186c0 msgr2=0x7fd3dc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:21.411 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:21.411 7fd401475700 1 --2- 172.21.2.112:0/1140942605 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd3dc0186c0 0x7fd3dc01abc0 crc :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:21.411 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:21.411 7fd401475700 1 -- 172.21.2.112:0/1140942605 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd3fc082ca0 msgr2=0x7fd3fc0830a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:21.411 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:21.411 7fd401475700 1 --2- 172.21.2.112:0/1140942605 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd3fc082ca0 0x7fd3fc0830a0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7fd3f4006fc0 tx=0x7fd3f400bcf0).stop 2021-02-10T11:42:21.411 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:21.411 7fd401475700 1 -- 172.21.2.112:0/1140942605 shutdown_connections 2021-02-10T11:42:21.411 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:21.411 7fd401475700 1 --2- 172.21.2.112:0/1140942605 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd3dc0186c0 0x7fd3dc01abc0 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:21.411 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:21.411 7fd401475700 1 --2- 172.21.2.112:0/1140942605 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd3fc0bdb40 0x7fd3fc0bdf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:21.412 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:21.411 7fd401475700 1 --2- 172.21.2.112:0/1140942605 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd3fc082ca0 0x7fd3fc0830a0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:21.412 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:21.411 7fd401475700 1 --2- 172.21.2.112:0/1140942605 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd3fc1a79a0 0x7fd3fc1a7da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:21.412 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:21.411 7fd401475700 1 -- 172.21.2.112:0/1140942605 >> 172.21.2.112:0/1140942605 conn(0x7fd3fc0bef00 msgr2=0x7fd3fc09f200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:21.412 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:21.412 7fd401475700 1 -- 172.21.2.112:0/1140942605 shutdown_connections 2021-02-10T11:42:21.412 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:21.412 7fd401475700 1 -- 172.21.2.112:0/1140942605 wait complete. 2021-02-10T11:42:21.419 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:42:21.847 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:21.831 7fefe703b700 1 Processor -- start 2021-02-10T11:42:21.848 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:21.831 7fefe703b700 1 -- start start 2021-02-10T11:42:21.848 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:21.832 7fefe703b700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fefe00d1870 0x7fefe01b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:21.848 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:21.832 7fefe703b700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fefe00f0650 0x7fefe00cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:21.848 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:21.832 7fefe703b700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefe00f0b10 con 0x7fefe01b0a10 2021-02-10T11:42:21.848 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:21.832 7fefe703b700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fefe00e5ed0 con 0x7fefe00f0650 2021-02-10T11:42:21.848 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:21.832 7fefe703b700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fefe00d1c70 con 0x7fefe00d1870 2021-02-10T11:42:21.848 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:21.832 7fefe5658700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fefe00f0650 0x7fefe00cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:21.848 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:21.832 7fefe5658700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fefe00f0650 0x7fefe00cfb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:56202/0 (socket says 172.21.2.112:56202) 2021-02-10T11:42:21.848 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:21.832 7fefe5658700 1 -- 172.21.2.112:0/3932037500 learned_addr learned my addr 172.21.2.112:0/3932037500 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:42:21.848 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:21.832 7fefe4e57700 1 --2- 172.21.2.112:0/3932037500 >> v2:172.21.2.120:3300/0 conn(0x7fefe00d1870 0x7fefe01b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:21.848 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:21.832 7fefe5658700 1 -- 172.21.2.112:0/3932037500 >> v2:172.21.2.120:3300/0 conn(0x7fefe00d1870 msgr2=0x7fefe01b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:21.849 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:21.832 7fefe5658700 1 --2- 172.21.2.112:0/3932037500 >> v2:172.21.2.120:3300/0 conn(0x7fefe00d1870 0x7fefe01b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:21.849 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:21.832 7fefe5658700 1 -- 172.21.2.112:0/3932037500 >> v1:172.21.2.120:6789/0 conn(0x7fefe01b0a10 legacy=0x7fefe00cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:21.849 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:21.832 7fefe5658700 1 -- 172.21.2.112:0/3932037500 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefe00d1e90 con 0x7fefe00f0650 2021-02-10T11:42:21.849 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:21.832 7fefe4e57700 1 --2- 172.21.2.112:0/3932037500 >> v2:172.21.2.120:3300/0 conn(0x7fefe00d1870 0x7fefe01b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:42:21.849 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:21.833 7fefe5658700 1 --2- 172.21.2.112:0/3932037500 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fefe00f0650 0x7fefe00cfb20 secure :-1 s=READY pgs=2324 cs=0 l=1 rev1=1 rx=0x7fefd8006b80 tx=0x7fefd8009b90).ready entity=mon.0 client_cookie=714c6a85da9b1932 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:21.849 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:21.833 7fefdf7fe700 1 -- 172.21.2.112:0/3932037500 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fefd80022e0 con 0x7fefe00f0650 2021-02-10T11:42:21.849 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:21.833 7fefdf7fe700 1 -- 172.21.2.112:0/3932037500 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefd8002620 con 0x7fefe00f0650 2021-02-10T11:42:21.849 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:21.833 7fefdf7fe700 1 -- 172.21.2.112:0/3932037500 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fefd8002b00 con 0x7fefe00f0650 2021-02-10T11:42:21.849 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:21.833 7fefe703b700 1 -- 172.21.2.112:0/3932037500 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fefe00f0650 msgr2=0x7fefe00cfb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:21.849 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:21.833 7fefe703b700 1 --2- 172.21.2.112:0/3932037500 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fefe00f0650 0x7fefe00cfb20 secure :-1 s=READY pgs=2324 cs=0 l=1 rev1=1 rx=0x7fefd8006b80 tx=0x7fefd8009b90).stop 2021-02-10T11:42:21.849 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:21.833 7fefe703b700 1 -- 172.21.2.112:0/3932037500 shutdown_connections 2021-02-10T11:42:21.849 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:21.834 7fefe703b700 1 --2- 172.21.2.112:0/3932037500 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fefe00f0650 0x7fefe00cfb20 unknown :-1 s=CLOSED pgs=2324 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:21.849 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:21.834 7fefe703b700 1 --2- 172.21.2.112:0/3932037500 >> v2:172.21.2.120:3300/0 conn(0x7fefe00d1870 0x7fefe01b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:21.850 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:21.834 7fefe703b700 1 -- 172.21.2.112:0/3932037500 >> 172.21.2.112:0/3932037500 conn(0x7fefe00ce450 msgr2=0x7fefe00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:21.850 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:21.834 7fefe703b700 1 -- 172.21.2.112:0/3932037500 shutdown_connections 2021-02-10T11:42:21.850 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:21.834 7fefe703b700 1 -- 172.21.2.112:0/3932037500 wait complete. 2021-02-10T11:42:21.851 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:21.848 7fefe703b700 1 Processor -- start 2021-02-10T11:42:21.851 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:21.848 7fefe703b700 1 -- start start 2021-02-10T11:42:21.852 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:21.849 7fefe703b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fefe01b7cf0 0x7fefe019cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:21.852 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:21.849 7fefe703b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fefe01b80f0 0x7fefe01b84f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:21.852 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:21.849 7fefe703b700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fefe019da20 0x7fefe019de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:21.852 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:21.849 7fefe703b700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefe00a77c0 con 0x7fefe01b80f0 2021-02-10T11:42:21.852 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:21.849 7fefe703b700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefe019d4a0 con 0x7fefe019da20 2021-02-10T11:42:21.852 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:21.849 7fefe703b700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefe00954f0 con 0x7fefe01b7cf0 2021-02-10T11:42:21.852 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:21.849 7fefdffff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fefe01b80f0 0x7fefe01b84f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:21.852 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:21.849 7fefdffff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fefe01b80f0 0x7fefe01b84f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:56206/0 (socket says 172.21.2.112:56206) 2021-02-10T11:42:21.852 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:21.849 7fefdffff700 1 -- 172.21.2.112:0/54997886 learned_addr learned my addr 172.21.2.112:0/54997886 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:42:21.852 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:21.849 7fefdffff700 1 -- 172.21.2.112:0/54997886 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fefe01b7cf0 msgr2=0x7fefe019cdb0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:42:21.852 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:21.849 7fefdffff700 1 --2- 172.21.2.112:0/54997886 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fefe01b7cf0 0x7fefe019cdb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:21.852 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:21.849 7fefdffff700 1 -- 172.21.2.112:0/54997886 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fefe019da20 msgr2=0x7fefe019de20 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:42:21.852 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:21.849 7fefdffff700 1 --2- 172.21.2.112:0/54997886 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fefe019da20 0x7fefe019de20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:21.853 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:21.849 7fefdffff700 1 -- 172.21.2.112:0/54997886 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefe01b8d60 con 0x7fefe01b80f0 2021-02-10T11:42:21.853 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:21.850 7fefdffff700 1 --2- 172.21.2.112:0/54997886 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fefe01b80f0 0x7fefe01b84f0 secure :-1 s=READY pgs=2325 cs=0 l=1 rev1=1 rx=0x7fefd0009940 tx=0x7fefd000bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:21.853 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:21.850 7fefdd7fa700 1 -- 172.21.2.112:0/54997886 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fefd0000ce0 con 0x7fefe01b80f0 2021-02-10T11:42:21.853 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:21.850 7fefdd7fa700 1 -- 172.21.2.112:0/54997886 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefd000c2f0 con 0x7fefe01b80f0 2021-02-10T11:42:21.853 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:21.850 7fefe703b700 1 -- 172.21.2.112:0/54997886 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefe01b8d60 con 0x7fefe01b80f0 2021-02-10T11:42:21.853 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:21.850 7fefe703b700 1 -- 172.21.2.112:0/54997886 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefe019e730 con 0x7fefe01b80f0 2021-02-10T11:42:21.853 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:21.850 7fefdd7fa700 1 -- 172.21.2.112:0/54997886 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fefd000c810 con 0x7fefe01b80f0 2021-02-10T11:42:21.853 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:21.851 7fefdd7fa700 1 -- 172.21.2.112:0/54997886 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fefd0001230 con 0x7fefe01b80f0 2021-02-10T11:42:21.853 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:21.851 7fefdd7fa700 1 --2- 172.21.2.112:0/54997886 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fefc40186c0 0x7fefc401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:21.853 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:21.851 7fefdd7fa700 1 -- 172.21.2.112:0/54997886 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fefd0032bb0 con 0x7fefe01b80f0 2021-02-10T11:42:21.853 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:21.851 7fefe4e57700 1 --2- 172.21.2.112:0/54997886 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fefc40186c0 0x7fefc401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:21.853 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:21.853 7fefe4e57700 1 --2- 172.21.2.112:0/54997886 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fefc40186c0 0x7fefc401abc0 crc :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:21.881 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:21.881 7fefe703b700 1 -- 172.21.2.112:0/54997886 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefe00ebc00 con 0x7fefe01b80f0 2021-02-10T11:42:21.883 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:21.885 7fefdd7fa700 1 -- 172.21.2.112:0/54997886 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fefd0000cc0 con 0x7fefe01b80f0 2021-02-10T11:42:22.009 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:22.009 7fefe703b700 1 -- 172.21.2.112:0/54997886 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fefe00f0b10 con 0x7fefe01b80f0 2021-02-10T11:42:22.009 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:22.010 7fefdd7fa700 1 -- 172.21.2.112:0/54997886 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+2108 (secure 0 0 0) 0x7fefd001f8f0 con 0x7fefe01b80f0 2021-02-10T11:42:22.010 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:42:22.010 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":105,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11040,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/1814390924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":1814390924},{"type":"v1","addr":"172.21.2.112:6833","nonce":1814390924}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":101}],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:42:19.351348","modified":"2021-02-10 11:42:21.357585","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8203},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8203":{"gid":8203,"name":"b","rank":0,"incarnation":102,"state":"up:active","state_seq":3,"addr":"172.21.2.120:6835/2761661070","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":2761661070},{"type":"v1","addr":"172.21.2.120:6835","nonce":2761661070}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":13}]} 2021-02-10T11:42:22.010 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 105 2021-02-10T11:42:22.011 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:22.012 7fefe703b700 1 -- 172.21.2.112:0/54997886 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fefc40186c0 msgr2=0x7fefc401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:22.011 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:22.012 7fefe703b700 1 --2- 172.21.2.112:0/54997886 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fefc40186c0 0x7fefc401abc0 crc :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:22.011 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:22.012 7fefe703b700 1 -- 172.21.2.112:0/54997886 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fefe01b80f0 msgr2=0x7fefe01b84f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:22.011 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:22.012 7fefe703b700 1 --2- 172.21.2.112:0/54997886 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fefe01b80f0 0x7fefe01b84f0 secure :-1 s=READY pgs=2325 cs=0 l=1 rev1=1 rx=0x7fefd0009940 tx=0x7fefd000bcf0).stop 2021-02-10T11:42:22.012 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:22.012 7fefe703b700 1 -- 172.21.2.112:0/54997886 shutdown_connections 2021-02-10T11:42:22.012 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:22.012 7fefe703b700 1 --2- 172.21.2.112:0/54997886 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fefc40186c0 0x7fefc401abc0 unknown :-1 s=CLOSED pgs=1065 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:22.012 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:22.012 7fefe703b700 1 --2- 172.21.2.112:0/54997886 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fefe019da20 0x7fefe019de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:22.012 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:22.012 7fefe703b700 1 --2- 172.21.2.112:0/54997886 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fefe01b80f0 0x7fefe01b84f0 unknown :-1 s=CLOSED pgs=2325 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:22.012 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:22.012 7fefe703b700 1 --2- 172.21.2.112:0/54997886 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fefe01b7cf0 0x7fefe019cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:22.012 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:22.012 7fefe703b700 1 -- 172.21.2.112:0/54997886 >> 172.21.2.112:0/54997886 conn(0x7fefe00ed810 msgr2=0x7fefe0094980 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:22.012 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:22.012 7fefe703b700 1 -- 172.21.2.112:0/54997886 shutdown_connections 2021-02-10T11:42:22.012 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:22.012 7fefe703b700 1 -- 172.21.2.112:0/54997886 wait complete. 2021-02-10T11:42:22.020 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:42:22.448 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:22.433 7fea0d40a700 1 Processor -- start 2021-02-10T11:42:22.448 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:22.433 7fea0d40a700 1 -- start start 2021-02-10T11:42:22.448 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:22.434 7fea0d40a700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fea080d1870 0x7fea081b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:22.448 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:22.434 7fea0d40a700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fea081b0a10 0x7fea080cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:22.448 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:22.434 7fea0d40a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fea080d1c70 0x7fea080cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:22.449 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:22.434 7fea0d40a700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea080e5ed0 con 0x7fea080d1870 2021-02-10T11:42:22.449 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:22.434 7fea0d40a700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea080f0b10 con 0x7fea081b0a10 2021-02-10T11:42:22.449 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:22.434 7fea0d40a700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea080cff70 con 0x7fea080d1c70 2021-02-10T11:42:22.449 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:22.434 7fea0707d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fea080d1870 0x7fea081b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:22.449 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:22.434 7fea0707d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fea080d1870 0x7fea081b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:56212/0 (socket says 172.21.2.112:56212) 2021-02-10T11:42:22.449 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:22.434 7fea0707d700 1 -- 172.21.2.112:0/3817596506 learned_addr learned my addr 172.21.2.112:0/3817596506 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:42:22.449 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:22.434 7fea0687c700 1 --2- 172.21.2.112:0/3817596506 >> v2:172.21.2.120:3300/0 conn(0x7fea081b0a10 0x7fea080cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:22.449 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:22.434 7fea0787e700 1 --2- 172.21.2.112:0/3817596506 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fea080d1c70 0x7fea080cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:22.449 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:22.434 7fea0707d700 1 -- 172.21.2.112:0/3817596506 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fea080d1c70 msgr2=0x7fea080cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:22.449 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:22.434 7fea0707d700 1 --2- 172.21.2.112:0/3817596506 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fea080d1c70 0x7fea080cf950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:22.449 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:22.434 7fea0707d700 1 -- 172.21.2.112:0/3817596506 >> v2:172.21.2.120:3300/0 conn(0x7fea081b0a10 msgr2=0x7fea080cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:22.450 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:22.434 7fea0707d700 1 --2- 172.21.2.112:0/3817596506 >> v2:172.21.2.120:3300/0 conn(0x7fea081b0a10 0x7fea080cf330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:22.450 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:22.434 7fea0707d700 1 -- 172.21.2.112:0/3817596506 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea080d0150 con 0x7fea080d1870 2021-02-10T11:42:22.450 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:22.434 7fea0687c700 1 --2- 172.21.2.112:0/3817596506 >> v2:172.21.2.120:3300/0 conn(0x7fea081b0a10 0x7fea080cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:42:22.450 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:22.435 7fea0707d700 1 --2- 172.21.2.112:0/3817596506 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fea080d1870 0x7fea081b03f0 secure :-1 s=READY pgs=2326 cs=0 l=1 rev1=1 rx=0x7fe9f8006700 tx=0x7fe9f8009b80).ready entity=mon.0 client_cookie=526dd76f08ff6e3c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:22.450 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:22.435 7fea0607b700 1 -- 172.21.2.112:0/3817596506 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe9f800c0c0 con 0x7fea080d1870 2021-02-10T11:42:22.450 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:22.435 7fea0607b700 1 -- 172.21.2.112:0/3817596506 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9f800c400 con 0x7fea080d1870 2021-02-10T11:42:22.450 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:22.435 7fea0607b700 1 -- 172.21.2.112:0/3817596506 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe9f800c920 con 0x7fea080d1870 2021-02-10T11:42:22.450 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:22.435 7fea0d40a700 1 -- 172.21.2.112:0/3817596506 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fea080d1870 msgr2=0x7fea081b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:22.450 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:22.435 7fea0d40a700 1 --2- 172.21.2.112:0/3817596506 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fea080d1870 0x7fea081b03f0 secure :-1 s=READY pgs=2326 cs=0 l=1 rev1=1 rx=0x7fe9f8006700 tx=0x7fe9f8009b80).stop 2021-02-10T11:42:22.450 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:22.435 7fea0d40a700 1 -- 172.21.2.112:0/3817596506 shutdown_connections 2021-02-10T11:42:22.450 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:22.435 7fea0d40a700 1 --2- 172.21.2.112:0/3817596506 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fea080d1c70 0x7fea080cf950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:22.450 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:22.435 7fea0d40a700 1 --2- 172.21.2.112:0/3817596506 >> v2:172.21.2.120:3300/0 conn(0x7fea081b0a10 0x7fea080cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:22.451 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:22.435 7fea0d40a700 1 --2- 172.21.2.112:0/3817596506 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fea080d1870 0x7fea081b03f0 unknown :-1 s=CLOSED pgs=2326 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:22.451 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:22.435 7fea0d40a700 1 -- 172.21.2.112:0/3817596506 >> 172.21.2.112:0/3817596506 conn(0x7fea080ce450 msgr2=0x7fea080d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:22.451 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:22.435 7fea0d40a700 1 -- 172.21.2.112:0/3817596506 shutdown_connections 2021-02-10T11:42:22.451 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:22.435 7fea0d40a700 1 -- 172.21.2.112:0/3817596506 wait complete. 2021-02-10T11:42:22.453 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:22.450 7fea0d40a700 1 Processor -- start 2021-02-10T11:42:22.453 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:22.450 7fea0d40a700 1 -- start start 2021-02-10T11:42:22.453 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:22.450 7fea0d40a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fea080b2860 0x7fea081a4480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:22.453 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:22.450 7fea0d40a700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fea081a4aa0 0x7fea080ec620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:22.453 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:22.450 7fea0d40a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fea080ecc40 0x7fea08097790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:22.453 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:22.450 7fea0d40a700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea080b2ce0 con 0x7fea080ecc40 2021-02-10T11:42:22.453 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:22.450 7fea0d40a700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea080ed040 con 0x7fea081a4aa0 2021-02-10T11:42:22.453 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:22.450 7fea0d40a700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea080ed260 con 0x7fea080b2860 2021-02-10T11:42:22.453 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:22.451 7fea0787e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fea080ecc40 0x7fea08097790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:22.453 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:22.451 7fea0787e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fea080ecc40 0x7fea08097790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:56218/0 (socket says 172.21.2.112:56218) 2021-02-10T11:42:22.453 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:22.451 7fea0787e700 1 -- 172.21.2.112:0/1272777628 learned_addr learned my addr 172.21.2.112:0/1272777628 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:42:22.454 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:22.451 7fea0687c700 1 --2- 172.21.2.112:0/1272777628 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fea081a4aa0 0x7fea080ec620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:22.454 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:22.451 7fea0707d700 1 --2- 172.21.2.112:0/1272777628 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fea080b2860 0x7fea081a4480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:22.454 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:22.451 7fea0787e700 1 -- 172.21.2.112:0/1272777628 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fea080b2860 msgr2=0x7fea081a4480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:22.454 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:22.451 7fea0787e700 1 --2- 172.21.2.112:0/1272777628 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fea080b2860 0x7fea081a4480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:22.454 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:22.451 7fea0787e700 1 -- 172.21.2.112:0/1272777628 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fea081a4aa0 msgr2=0x7fea080ec620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:22.454 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:22.451 7fea0787e700 1 --2- 172.21.2.112:0/1272777628 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fea081a4aa0 0x7fea080ec620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:22.454 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:22.451 7fea0787e700 1 -- 172.21.2.112:0/1272777628 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea08097db0 con 0x7fea080ecc40 2021-02-10T11:42:22.454 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:22.452 7fea0787e700 1 --2- 172.21.2.112:0/1272777628 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fea080ecc40 0x7fea08097790 secure :-1 s=READY pgs=2327 cs=0 l=1 rev1=1 rx=0x7fea00005fd0 tx=0x7fea0000ac50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:22.454 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:22.452 7fe9effff700 1 -- 172.21.2.112:0/1272777628 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fea0000c300 con 0x7fea080ecc40 2021-02-10T11:42:22.454 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:22.452 7fe9effff700 1 -- 172.21.2.112:0/1272777628 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea00003030 con 0x7fea080ecc40 2021-02-10T11:42:22.454 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:22.452 7fe9effff700 1 -- 172.21.2.112:0/1272777628 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fea0000c7b0 con 0x7fea080ecc40 2021-02-10T11:42:22.454 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:22.452 7fea0d40a700 1 -- 172.21.2.112:0/1272777628 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea08097db0 con 0x7fea080ecc40 2021-02-10T11:42:22.454 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:22.452 7fea0d40a700 1 -- 172.21.2.112:0/1272777628 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea08097db0 con 0x7fea080ecc40 2021-02-10T11:42:22.455 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:22.453 7fe9effff700 1 -- 172.21.2.112:0/1272777628 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fea0000b030 con 0x7fea080ecc40 2021-02-10T11:42:22.455 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:22.453 7fe9effff700 1 --2- 172.21.2.112:0/1272777628 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe9e80186c0 0x7fe9e801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:22.455 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:22.453 7fe9effff700 1 -- 172.21.2.112:0/1272777628 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fea000328d0 con 0x7fea080ecc40 2021-02-10T11:42:22.455 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:22.453 7fea0707d700 1 --2- 172.21.2.112:0/1272777628 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe9e80186c0 0x7fe9e801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:22.455 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:22.453 7fea0707d700 1 --2- 172.21.2.112:0/1272777628 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe9e80186c0 0x7fe9e801abc0 crc :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:22.482 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:22.484 7fea0d40a700 1 -- 172.21.2.112:0/1272777628 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea080f0b10 con 0x7fea080ecc40 2021-02-10T11:42:22.485 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:22.486 7fe9effff700 1 -- 172.21.2.112:0/1272777628 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fea0000ad90 con 0x7fea080ecc40 2021-02-10T11:42:22.609 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:22.610 7fea0d40a700 1 -- 172.21.2.112:0/1272777628 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fea080f0b10 con 0x7fea080ecc40 2021-02-10T11:42:22.610 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:22.611 7fe9effff700 1 -- 172.21.2.112:0/1272777628 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v125) v1 ==== 74+0+14007 (secure 0 0 0) 0x7fea0000b030 con 0x7fea080ecc40 2021-02-10T11:42:22.610 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:42:22.610 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":125,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:42:20.353491","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":27,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:42:17.322526","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"124","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":27,"pool_name":"cephfs_data","create_time":"2021-02-10 11:42:18.395545","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"125","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:42:22.612 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:22.613 7fea0d40a700 1 -- 172.21.2.112:0/1272777628 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe9e80186c0 msgr2=0x7fe9e801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:22.612 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:22.613 7fea0d40a700 1 --2- 172.21.2.112:0/1272777628 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe9e80186c0 0x7fe9e801abc0 crc :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:22.612 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:22.613 7fea0d40a700 1 -- 172.21.2.112:0/1272777628 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fea080ecc40 msgr2=0x7fea08097790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:22.612 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:22.613 7fea0d40a700 1 --2- 172.21.2.112:0/1272777628 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fea080ecc40 0x7fea08097790 secure :-1 s=READY pgs=2327 cs=0 l=1 rev1=1 rx=0x7fea00005fd0 tx=0x7fea0000ac50).stop 2021-02-10T11:42:22.612 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:22.613 7fea0d40a700 1 -- 172.21.2.112:0/1272777628 shutdown_connections 2021-02-10T11:42:22.612 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:22.613 7fea0d40a700 1 --2- 172.21.2.112:0/1272777628 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe9e80186c0 0x7fe9e801abc0 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:22.612 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:22.613 7fea0d40a700 1 --2- 172.21.2.112:0/1272777628 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fea081a4aa0 0x7fea080ec620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:22.612 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:22.613 7fea0d40a700 1 --2- 172.21.2.112:0/1272777628 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fea080ecc40 0x7fea08097790 unknown :-1 s=CLOSED pgs=2327 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:22.612 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:22.613 7fea0d40a700 1 --2- 172.21.2.112:0/1272777628 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fea080b2860 0x7fea081a4480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:22.612 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:22.613 7fea0d40a700 1 -- 172.21.2.112:0/1272777628 >> 172.21.2.112:0/1272777628 conn(0x7fea080bf500 msgr2=0x7fea080b5050 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:22.613 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:22.613 7fea0d40a700 1 -- 172.21.2.112:0/1272777628 shutdown_connections 2021-02-10T11:42:22.613 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:22.613 7fea0d40a700 1 -- 172.21.2.112:0/1272777628 wait complete. 2021-02-10T11:42:22.621 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:42:23.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.046 7fc8b8b3c700 1 Processor -- start 2021-02-10T11:42:23.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.047 7fc8b8b3c700 1 -- start start 2021-02-10T11:42:23.063 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.047 7fc8b8b3c700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fc8b40d1c80 0x7fc8b40cef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:23.063 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.047 7fc8b8b3c700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8b40d0180 con 0x7fc8b40cf590 2021-02-10T11:42:23.063 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.047 7fc8b8b3c700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8b40fe1d0 con 0x7fc8b40d1880 2021-02-10T11:42:23.063 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.047 7fc8b8b3c700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8b41b0cc0 con 0x7fc8b40d1c80 2021-02-10T11:42:23.063 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.047 7fc8b205d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fc8b40d1c80 0x7fc8b40cef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:23.063 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.047 7fc8b305f700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fc8b40cf590 0x7fc8b40cf990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:55686/0 (socket says 172.21.2.112:55686) 2021-02-10T11:42:23.063 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.047 7fc8b305f700 1 -- 172.21.2.112:0/98501929 learned_addr learned my addr 172.21.2.112:0/98501929 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:42:23.063 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.048 7fc8b205d700 1 -- 172.21.2.112:0/98501929 >> v1:172.21.2.120:6789/0 conn(0x7fc8b40d1880 legacy=0x7fc8b41b04d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:42:23.063 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.048 7fc8b205d700 1 -- 172.21.2.112:0/98501929 >> v1:172.21.2.112:6789/0 conn(0x7fc8b40cf590 legacy=0x7fc8b40cf990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:23.063 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.048 7fc8b205d700 1 -- 172.21.2.112:0/98501929 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8b4109a70 con 0x7fc8b40d1c80 2021-02-10T11:42:23.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.048 7fc8b185c700 1 -- 172.21.2.112:0/98501929 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3259586215 0 0) 0x7fc8ac000f90 con 0x7fc8b40cf590 2021-02-10T11:42:23.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.048 7fc8b205d700 1 --2- 172.21.2.112:0/98501929 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc8b40d1c80 0x7fc8b40cef70 secure :-1 s=READY pgs=2329 cs=0 l=1 rev1=1 rx=0x7fc8a8006650 tx=0x7fc8a8009a00).ready entity=mon.0 client_cookie=e1947ca530b83038 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:23.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.048 7fc8b185c700 1 -- 172.21.2.112:0/98501929 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc8a8009df0 con 0x7fc8b40d1c80 2021-02-10T11:42:23.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.048 7fc8b185c700 1 -- 172.21.2.112:0/98501929 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8a80009b0 con 0x7fc8b40d1c80 2021-02-10T11:42:23.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.048 7fc8b185c700 1 -- 172.21.2.112:0/98501929 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc8a800c2b0 con 0x7fc8b40d1c80 2021-02-10T11:42:23.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.048 7fc8b8b3c700 1 -- 172.21.2.112:0/98501929 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc8b40d1c80 msgr2=0x7fc8b40cef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:23.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.048 7fc8b8b3c700 1 --2- 172.21.2.112:0/98501929 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc8b40d1c80 0x7fc8b40cef70 secure :-1 s=READY pgs=2329 cs=0 l=1 rev1=1 rx=0x7fc8a8006650 tx=0x7fc8a8009a00).stop 2021-02-10T11:42:23.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.049 7fc8b8b3c700 1 -- 172.21.2.112:0/98501929 shutdown_connections 2021-02-10T11:42:23.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.049 7fc8b8b3c700 1 --2- 172.21.2.112:0/98501929 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc8b40d1c80 0x7fc8b40cef70 unknown :-1 s=CLOSED pgs=2329 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:23.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.049 7fc8b8b3c700 1 -- 172.21.2.112:0/98501929 >> 172.21.2.112:0/98501929 conn(0x7fc8b40ce430 msgr2=0x7fc8b40d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:23.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.049 7fc8b8b3c700 1 -- 172.21.2.112:0/98501929 shutdown_connections 2021-02-10T11:42:23.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.049 7fc8b8b3c700 1 -- 172.21.2.112:0/98501929 wait complete. 2021-02-10T11:42:23.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.064 7fc8b8b3c700 1 Processor -- start 2021-02-10T11:42:23.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.064 7fc8b8b3c700 1 -- start start 2021-02-10T11:42:23.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.064 7fc8b8b3c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc8b41a68f0 0x7fc8b419fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:23.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.064 7fc8b8b3c700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc8b41a5e00 0x7fc8b41a02c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:23.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.064 7fc8b8b3c700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc8b41a08e0 0x7fc8b41aae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:23.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.064 7fc8b8b3c700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8b41a6370 con 0x7fc8b41a68f0 2021-02-10T11:42:23.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.064 7fc8b8b3c700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8b41a3960 con 0x7fc8b41a08e0 2021-02-10T11:42:23.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.064 7fc8b8b3c700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8b41ab440 con 0x7fc8b41a5e00 2021-02-10T11:42:23.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.064 7fc8b285e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc8b41a68f0 0x7fc8b419fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:23.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.064 7fc8b285e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc8b41a68f0 0x7fc8b419fca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:56230/0 (socket says 172.21.2.112:56230) 2021-02-10T11:42:23.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.064 7fc8b285e700 1 -- 172.21.2.112:0/569610069 learned_addr learned my addr 172.21.2.112:0/569610069 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:42:23.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.065 7fc8b285e700 1 -- 172.21.2.112:0/569610069 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc8b41a5e00 msgr2=0x7fc8b41a02c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:42:23.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.065 7fc8b285e700 1 --2- 172.21.2.112:0/569610069 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc8b41a5e00 0x7fc8b41a02c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:23.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.065 7fc8b285e700 1 -- 172.21.2.112:0/569610069 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc8b41a08e0 msgr2=0x7fc8b41aae20 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:42:23.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.065 7fc8b285e700 1 --2- 172.21.2.112:0/569610069 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc8b41a08e0 0x7fc8b41aae20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:23.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.065 7fc8b285e700 1 -- 172.21.2.112:0/569610069 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8b41ab660 con 0x7fc8b41a68f0 2021-02-10T11:42:23.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.065 7fc8b285e700 1 --2- 172.21.2.112:0/569610069 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc8b41a68f0 0x7fc8b419fca0 secure :-1 s=READY pgs=2330 cs=0 l=1 rev1=1 rx=0x7fc8a400b5b0 tx=0x7fc8a400b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:23.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.065 7fc8a37fe700 1 -- 172.21.2.112:0/569610069 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc8a400bd40 con 0x7fc8b41a68f0 2021-02-10T11:42:23.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.065 7fc8a37fe700 1 -- 172.21.2.112:0/569610069 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8a40009b0 con 0x7fc8b41a68f0 2021-02-10T11:42:23.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.065 7fc8a37fe700 1 -- 172.21.2.112:0/569610069 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc8a4006210 con 0x7fc8b41a68f0 2021-02-10T11:42:23.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.065 7fc8b8b3c700 1 -- 172.21.2.112:0/569610069 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8b41ab660 con 0x7fc8b41a68f0 2021-02-10T11:42:23.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.065 7fc8b8b3c700 1 -- 172.21.2.112:0/569610069 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8b41abbf0 con 0x7fc8b41a68f0 2021-02-10T11:42:23.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.066 7fc8a37fe700 1 -- 172.21.2.112:0/569610069 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc8a4006020 con 0x7fc8b41a68f0 2021-02-10T11:42:23.068 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.066 7fc8a37fe700 1 --2- 172.21.2.112:0/569610069 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc8980186c0 0x7fc89801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:23.068 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.066 7fc8a37fe700 1 -- 172.21.2.112:0/569610069 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fc8a4032560 con 0x7fc8b41a68f0 2021-02-10T11:42:23.068 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.066 7fc8b205d700 1 --2- 172.21.2.112:0/569610069 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc8980186c0 0x7fc89801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:23.068 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.067 7fc8b205d700 1 --2- 172.21.2.112:0/569610069 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc8980186c0 0x7fc89801abc0 crc :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:23.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.097 7fc8b8b3c700 1 -- 172.21.2.112:0/569610069 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8b40829a0 con 0x7fc8b41a68f0 2021-02-10T11:42:23.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.099 7fc8a37fe700 1 -- 172.21.2.112:0/569610069 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc8a4026620 con 0x7fc8b41a68f0 2021-02-10T11:42:23.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.231 7fc8b8b3c700 1 -- 172.21.2.112:0/569610069 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fc8b40829a0 con 0x7fc8b41a68f0 2021-02-10T11:42:23.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.234 7fc8a37fe700 1 -- 172.21.2.112:0/569610069 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v75) v1 ==== 184+0+0 (secure 0 0 0) 0x7fc8a4006020 con 0x7fc8b41a68f0 2021-02-10T11:42:23.237 INFO:teuthology.orchestra.run.gibba012.stderr:updated caps for client.0 2021-02-10T11:42:23.238 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.239 7fc8b8b3c700 1 -- 172.21.2.112:0/569610069 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc8980186c0 msgr2=0x7fc89801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:23.238 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.239 7fc8b8b3c700 1 --2- 172.21.2.112:0/569610069 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc8980186c0 0x7fc89801abc0 crc :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:23.238 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.239 7fc8b8b3c700 1 -- 172.21.2.112:0/569610069 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc8b41a68f0 msgr2=0x7fc8b419fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:23.238 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.239 7fc8b8b3c700 1 --2- 172.21.2.112:0/569610069 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc8b41a68f0 0x7fc8b419fca0 secure :-1 s=READY pgs=2330 cs=0 l=1 rev1=1 rx=0x7fc8a400b5b0 tx=0x7fc8a400b950).stop 2021-02-10T11:42:23.239 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.240 7fc8b8b3c700 1 -- 172.21.2.112:0/569610069 shutdown_connections 2021-02-10T11:42:23.239 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.240 7fc8b8b3c700 1 --2- 172.21.2.112:0/569610069 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc8980186c0 0x7fc89801abc0 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:23.239 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.240 7fc8b8b3c700 1 --2- 172.21.2.112:0/569610069 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc8b41a08e0 0x7fc8b41aae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:23.239 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.240 7fc8b8b3c700 1 --2- 172.21.2.112:0/569610069 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc8b41a5e00 0x7fc8b41a02c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:23.239 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.240 7fc8b8b3c700 1 --2- 172.21.2.112:0/569610069 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc8b41a68f0 0x7fc8b419fca0 unknown :-1 s=CLOSED pgs=2330 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:23.239 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.240 7fc8b8b3c700 1 -- 172.21.2.112:0/569610069 >> 172.21.2.112:0/569610069 conn(0x7fc8b419f880 msgr2=0x7fc8b41a71a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:23.239 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.240 7fc8b8b3c700 1 -- 172.21.2.112:0/569610069 shutdown_connections 2021-02-10T11:42:23.239 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.240 7fc8b8b3c700 1 -- 172.21.2.112:0/569610069 wait complete. 2021-02-10T11:42:23.247 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:42:23.280 DEBUG:teuthology.orchestra.run.gibba012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:42:23.292 DEBUG:teuthology.orchestra.run.gibba020:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:42:23.679 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.664 7fa0dac63700 1 Processor -- start 2021-02-10T11:42:23.679 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.664 7fa0dac63700 1 -- start start 2021-02-10T11:42:23.679 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.664 7fa0dac63700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fa0d40d1860 0x7fa0d41b04b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:23.679 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.664 7fa0dac63700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0d40fe1b0 con 0x7fa0d41b0ad0 2021-02-10T11:42:23.679 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.664 7fa0dac63700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0d4109a50 con 0x7fa0d40cff60 2021-02-10T11:42:23.679 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.664 7fa0dac63700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0d4088070 con 0x7fa0d40d1860 2021-02-10T11:42:23.679 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.665 7fa0d3fff700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fa0d41b0ad0 0x7fa0d4108930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:55698/0 (socket says 172.21.2.112:55698) 2021-02-10T11:42:23.680 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.665 7fa0d3fff700 1 -- 172.21.2.112:0/4176168294 learned_addr learned my addr 172.21.2.112:0/4176168294 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:42:23.680 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.665 7fa0d8a7f700 1 --2- 172.21.2.112:0/4176168294 >> v2:172.21.2.120:3300/0 conn(0x7fa0d40d1860 0x7fa0d41b04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:23.680 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.665 7fa0d37fe700 1 -- 172.21.2.112:0/4176168294 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1833756446 0 0) 0x7fa0cc000ce0 con 0x7fa0d41b0ad0 2021-02-10T11:42:23.680 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.665 7fa0d37fe700 1 -- 172.21.2.112:0/4176168294 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0c0002fe0 con 0x7fa0d41b0ad0 2021-02-10T11:42:23.680 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.665 7fa0d8a7f700 1 -- 172.21.2.112:0/4176168294 >> v1:172.21.2.120:6789/0 conn(0x7fa0d40cff60 legacy=0x7fa0d4109120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:23.680 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.665 7fa0d8a7f700 1 -- 172.21.2.112:0/4176168294 >> v1:172.21.2.112:6789/0 conn(0x7fa0d41b0ad0 legacy=0x7fa0d4108930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:23.680 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.665 7fa0d8a7f700 1 -- 172.21.2.112:0/4176168294 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0d4088260 con 0x7fa0d40d1860 2021-02-10T11:42:23.680 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.665 7fa0d8a7f700 1 --2- 172.21.2.112:0/4176168294 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa0d40d1860 0x7fa0d41b04b0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 rx=0x7fa0c40069d0 tx=0x7fa0c4009e40).ready entity=mon.1 client_cookie=73b304638cffee2f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:23.680 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.666 7fa0d37fe700 1 -- 172.21.2.112:0/4176168294 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa0c400b510 con 0x7fa0d40d1860 2021-02-10T11:42:23.680 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.666 7fa0d37fe700 1 -- 172.21.2.112:0/4176168294 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0c400b890 con 0x7fa0d40d1860 2021-02-10T11:42:23.680 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.666 7fa0d37fe700 1 -- 172.21.2.112:0/4176168294 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa0c400bdb0 con 0x7fa0d40d1860 2021-02-10T11:42:23.680 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.666 7fa0dac63700 1 -- 172.21.2.112:0/4176168294 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa0d40d1860 msgr2=0x7fa0d41b04b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:23.680 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.666 7fa0dac63700 1 --2- 172.21.2.112:0/4176168294 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa0d40d1860 0x7fa0d41b04b0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 rx=0x7fa0c40069d0 tx=0x7fa0c4009e40).stop 2021-02-10T11:42:23.681 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.666 7fa0dac63700 1 -- 172.21.2.112:0/4176168294 shutdown_connections 2021-02-10T11:42:23.681 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.666 7fa0dac63700 1 --2- 172.21.2.112:0/4176168294 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa0d40d1860 0x7fa0d41b04b0 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:23.681 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.666 7fa0dac63700 1 -- 172.21.2.112:0/4176168294 >> 172.21.2.112:0/4176168294 conn(0x7fa0d40ce410 msgr2=0x7fa0d40d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:23.681 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.666 7fa0dac63700 1 -- 172.21.2.112:0/4176168294 shutdown_connections 2021-02-10T11:42:23.681 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.666 7fa0dac63700 1 -- 172.21.2.112:0/4176168294 wait complete. 2021-02-10T11:42:23.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.681 7fa0dac63700 1 Processor -- start 2021-02-10T11:42:23.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.681 7fa0dac63700 1 -- start start 2021-02-10T11:42:23.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.681 7fa0dac63700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa0d4195500 0x7fa0d40b0410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:23.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.681 7fa0dac63700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa0d40b0a30 0x7fa0d418f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:23.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.681 7fa0dac63700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa0d418f990 0x7fa0d4190680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:23.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.681 7fa0dac63700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0d4195a60 con 0x7fa0d40b0a30 2021-02-10T11:42:23.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.681 7fa0dac63700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0d41a5970 con 0x7fa0d418f990 2021-02-10T11:42:23.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.681 7fa0dac63700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0d418fd90 con 0x7fa0d4195500 2021-02-10T11:42:23.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.681 7fa0d3fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa0d40b0a30 0x7fa0d418f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:23.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.681 7fa0d3fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa0d40b0a30 0x7fa0d418f370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:56238/0 (socket says 172.21.2.112:56238) 2021-02-10T11:42:23.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.681 7fa0d3fff700 1 -- 172.21.2.112:0/3088931698 learned_addr learned my addr 172.21.2.112:0/3088931698 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:42:23.684 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.682 7fa0d9280700 1 --2- 172.21.2.112:0/3088931698 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa0d418f990 0x7fa0d4190680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:23.684 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.682 7fa0d3fff700 1 -- 172.21.2.112:0/3088931698 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa0d4195500 msgr2=0x7fa0d40b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:23.684 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.682 7fa0d8a7f700 1 --2- 172.21.2.112:0/3088931698 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa0d4195500 0x7fa0d40b0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:23.684 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.682 7fa0d3fff700 1 --2- 172.21.2.112:0/3088931698 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa0d4195500 0x7fa0d40b0410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:23.684 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.682 7fa0d3fff700 1 -- 172.21.2.112:0/3088931698 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa0d418f990 msgr2=0x7fa0d4190680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:23.684 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.682 7fa0d3fff700 1 --2- 172.21.2.112:0/3088931698 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa0d418f990 0x7fa0d4190680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:23.684 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.682 7fa0d3fff700 1 -- 172.21.2.112:0/3088931698 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0d4190ca0 con 0x7fa0d40b0a30 2021-02-10T11:42:23.684 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.682 7fa0d8a7f700 1 --2- 172.21.2.112:0/3088931698 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa0d4195500 0x7fa0d40b0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:42:23.684 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.682 7fa0d9280700 1 --2- 172.21.2.112:0/3088931698 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa0d418f990 0x7fa0d4190680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:42:23.684 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.682 7fa0d3fff700 1 --2- 172.21.2.112:0/3088931698 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa0d40b0a30 0x7fa0d418f370 secure :-1 s=READY pgs=2332 cs=0 l=1 rev1=1 rx=0x7fa0cc00b5c0 tx=0x7fa0cc00b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:23.684 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.682 7fa0d17fa700 1 -- 172.21.2.112:0/3088931698 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa0cc00bd50 con 0x7fa0d40b0a30 2021-02-10T11:42:23.684 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.682 7fa0d17fa700 1 -- 172.21.2.112:0/3088931698 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0cc006280 con 0x7fa0d40b0a30 2021-02-10T11:42:23.684 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.682 7fa0d17fa700 1 -- 172.21.2.112:0/3088931698 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa0cc004620 con 0x7fa0d40b0a30 2021-02-10T11:42:23.684 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.682 7fa0dac63700 1 -- 172.21.2.112:0/3088931698 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0d4190ca0 con 0x7fa0d40b0a30 2021-02-10T11:42:23.685 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.682 7fa0dac63700 1 -- 172.21.2.112:0/3088931698 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0d4190ca0 con 0x7fa0d40b0a30 2021-02-10T11:42:23.685 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.683 7fa0d17fa700 1 -- 172.21.2.112:0/3088931698 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa0cc00a000 con 0x7fa0d40b0a30 2021-02-10T11:42:23.685 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.683 7fa0d17fa700 1 --2- 172.21.2.112:0/3088931698 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa0b80186c0 0x7fa0b801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:23.685 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.683 7fa0d17fa700 1 -- 172.21.2.112:0/3088931698 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fa0cc032540 con 0x7fa0d40b0a30 2021-02-10T11:42:23.685 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.683 7fa0d8a7f700 1 --2- 172.21.2.112:0/3088931698 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa0b80186c0 0x7fa0b801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:23.685 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.683 7fa0d8a7f700 1 --2- 172.21.2.112:0/3088931698 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa0b80186c0 0x7fa0b801abc0 crc :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:23.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.714 7fa0dac63700 1 -- 172.21.2.112:0/3088931698 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0d4082980 con 0x7fa0d40b0a30 2021-02-10T11:42:23.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.717 7fa0d17fa700 1 -- 172.21.2.112:0/3088931698 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa0cc006020 con 0x7fa0d40b0a30 2021-02-10T11:42:23.846 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.847 7fa0dac63700 1 -- 172.21.2.112:0/3088931698 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fa0d4082980 con 0x7fa0d40b0a30 2021-02-10T11:42:23.849 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.850 7fa0d17fa700 1 -- 172.21.2.112:0/3088931698 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v76) v1 ==== 184+0+0 (secure 0 0 0) 0x7fa0cc009fb0 con 0x7fa0d40b0a30 2021-02-10T11:42:23.853 INFO:teuthology.orchestra.run.gibba012.stderr:updated caps for client.1 2021-02-10T11:42:23.854 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.855 7fa0dac63700 1 -- 172.21.2.112:0/3088931698 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa0b80186c0 msgr2=0x7fa0b801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:23.854 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.855 7fa0dac63700 1 --2- 172.21.2.112:0/3088931698 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa0b80186c0 0x7fa0b801abc0 crc :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:23.854 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.855 7fa0dac63700 1 -- 172.21.2.112:0/3088931698 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa0d40b0a30 msgr2=0x7fa0d418f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:23.854 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.855 7fa0dac63700 1 --2- 172.21.2.112:0/3088931698 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa0d40b0a30 0x7fa0d418f370 secure :-1 s=READY pgs=2332 cs=0 l=1 rev1=1 rx=0x7fa0cc00b5c0 tx=0x7fa0cc00b960).stop 2021-02-10T11:42:23.855 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.856 7fa0dac63700 1 -- 172.21.2.112:0/3088931698 shutdown_connections 2021-02-10T11:42:23.855 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.856 7fa0dac63700 1 --2- 172.21.2.112:0/3088931698 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa0b80186c0 0x7fa0b801abc0 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:23.855 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.856 7fa0dac63700 1 --2- 172.21.2.112:0/3088931698 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa0d418f990 0x7fa0d4190680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:23.855 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.856 7fa0dac63700 1 --2- 172.21.2.112:0/3088931698 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa0d40b0a30 0x7fa0d418f370 unknown :-1 s=CLOSED pgs=2332 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:23.855 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.856 7fa0dac63700 1 --2- 172.21.2.112:0/3088931698 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa0d4195500 0x7fa0d40b0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:23.855 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.856 7fa0dac63700 1 -- 172.21.2.112:0/3088931698 >> 172.21.2.112:0/3088931698 conn(0x7fa0d419f820 msgr2=0x7fa0d41a5cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:23.855 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.856 7fa0dac63700 1 -- 172.21.2.112:0/3088931698 shutdown_connections 2021-02-10T11:42:23.856 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:23.856 7fa0dac63700 1 -- 172.21.2.112:0/3088931698 wait complete. 2021-02-10T11:42:23.863 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.2 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:42:24.302 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.286 7f13e921d700 1 Processor -- start 2021-02-10T11:42:24.302 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.286 7f13e921d700 1 -- start start 2021-02-10T11:42:24.302 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.287 7f13e921d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f13e40d1880 0x7f13e41b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:24.302 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.287 7f13e921d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f13e41b0af0 0x7f13e4108950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:24.302 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.287 7f13e921d700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13e4109a70 con 0x7f13e40d1c80 2021-02-10T11:42:24.302 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.287 7f13e921d700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13e40e5f00 con 0x7f13e41b0af0 2021-02-10T11:42:24.302 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.287 7f13e921d700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13e4109760 con 0x7f13e40d1880 2021-02-10T11:42:24.302 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.287 7f13e259c700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f13e41b0af0 0x7f13e4108950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:24.302 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.287 7f13e259c700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f13e41b0af0 0x7f13e4108950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:56248/0 (socket says 172.21.2.112:56248) 2021-02-10T11:42:24.302 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.287 7f13e259c700 1 -- 172.21.2.112:0/2246689072 learned_addr learned my addr 172.21.2.112:0/2246689072 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:42:24.303 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.287 7f13e259c700 1 -- 172.21.2.112:0/2246689072 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f13e40d1880 msgr2=0x7f13e41b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:24.303 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.287 7f13e259c700 1 --2- 172.21.2.112:0/2246689072 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f13e40d1880 0x7f13e41b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:24.303 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.287 7f13e259c700 1 -- 172.21.2.112:0/2246689072 >> v1:172.21.2.120:6789/0 conn(0x7f13e40d1c80 legacy=0x7f13e4108f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:24.303 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.287 7f13e259c700 1 -- 172.21.2.112:0/2246689072 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13e4088070 con 0x7f13e41b0af0 2021-02-10T11:42:24.303 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.288 7f13e259c700 1 --2- 172.21.2.112:0/2246689072 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f13e41b0af0 0x7f13e4108950 secure :-1 s=READY pgs=2333 cs=0 l=1 rev1=1 rx=0x7f13cc006700 tx=0x7f13cc009b80).ready entity=mon.0 client_cookie=b8f21f7f5c1e2441 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:24.303 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.288 7f13e1d9b700 1 -- 172.21.2.112:0/2246689072 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f13cc000c90 con 0x7f13e41b0af0 2021-02-10T11:42:24.303 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.288 7f13e1d9b700 1 -- 172.21.2.112:0/2246689072 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13cc00c0c0 con 0x7f13e41b0af0 2021-02-10T11:42:24.303 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.288 7f13e1d9b700 1 -- 172.21.2.112:0/2246689072 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f13cc00c5e0 con 0x7f13e41b0af0 2021-02-10T11:42:24.303 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.288 7f13e921d700 1 -- 172.21.2.112:0/2246689072 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f13e41b0af0 msgr2=0x7f13e4108950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:24.303 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.288 7f13e921d700 1 --2- 172.21.2.112:0/2246689072 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f13e41b0af0 0x7f13e4108950 secure :-1 s=READY pgs=2333 cs=0 l=1 rev1=1 rx=0x7f13cc006700 tx=0x7f13cc009b80).stop 2021-02-10T11:42:24.303 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.289 7f13e921d700 1 -- 172.21.2.112:0/2246689072 shutdown_connections 2021-02-10T11:42:24.303 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.289 7f13e921d700 1 --2- 172.21.2.112:0/2246689072 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f13e41b0af0 0x7f13e4108950 unknown :-1 s=CLOSED pgs=2333 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:24.303 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.289 7f13e921d700 1 --2- 172.21.2.112:0/2246689072 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f13e40d1880 0x7f13e41b04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:24.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.289 7f13e921d700 1 -- 172.21.2.112:0/2246689072 >> 172.21.2.112:0/2246689072 conn(0x7f13e40ce430 msgr2=0x7f13e40d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:24.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.289 7f13e921d700 1 -- 172.21.2.112:0/2246689072 shutdown_connections 2021-02-10T11:42:24.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.289 7f13e921d700 1 -- 172.21.2.112:0/2246689072 wait complete. 2021-02-10T11:42:24.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.303 7f13e921d700 1 Processor -- start 2021-02-10T11:42:24.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.303 7f13e921d700 1 -- start start 2021-02-10T11:42:24.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.304 7f13e921d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f13e4086520 0x7f13e41a1cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:24.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.304 7f13e921d700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f13e4086920 0x7f13e4187e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:24.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.304 7f13e921d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f13e41a22d0 0x7f13e41884a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:24.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.304 7f13e921d700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13e4099600 con 0x7f13e4086520 2021-02-10T11:42:24.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.304 7f13e921d700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13e40999f0 con 0x7f13e4086920 2021-02-10T11:42:24.306 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.304 7f13e921d700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13e4188ac0 con 0x7f13e41a22d0 2021-02-10T11:42:24.306 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.304 7f13e2d9d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f13e4086520 0x7f13e41a1cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:24.306 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.304 7f13e2d9d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f13e4086520 0x7f13e41a1cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:56252/0 (socket says 172.21.2.112:56252) 2021-02-10T11:42:24.306 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.304 7f13e2d9d700 1 -- 172.21.2.112:0/3311732674 learned_addr learned my addr 172.21.2.112:0/3311732674 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:42:24.306 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.304 7f13e2d9d700 1 -- 172.21.2.112:0/3311732674 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f13e41a22d0 msgr2=0x7f13e41884a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:42:24.306 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.304 7f13e2d9d700 1 --2- 172.21.2.112:0/3311732674 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f13e41a22d0 0x7f13e41884a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:24.306 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.304 7f13e2d9d700 1 -- 172.21.2.112:0/3311732674 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f13e4086920 msgr2=0x7f13e4187e80 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:42:24.306 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.304 7f13e2d9d700 1 --2- 172.21.2.112:0/3311732674 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f13e4086920 0x7f13e4187e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:24.306 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.304 7f13e2d9d700 1 -- 172.21.2.112:0/3311732674 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13e4188ce0 con 0x7f13e4086520 2021-02-10T11:42:24.306 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.304 7f13e2d9d700 1 --2- 172.21.2.112:0/3311732674 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f13e4086520 0x7f13e41a1cb0 secure :-1 s=READY pgs=2334 cs=0 l=1 rev1=1 rx=0x7f13d4006f00 tx=0x7f13d400b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:24.306 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.305 7f13db7fe700 1 -- 172.21.2.112:0/3311732674 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f13d400bd30 con 0x7f13e4086520 2021-02-10T11:42:24.306 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.305 7f13db7fe700 1 -- 172.21.2.112:0/3311732674 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13d4006c20 con 0x7f13e4086520 2021-02-10T11:42:24.307 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.305 7f13e921d700 1 -- 172.21.2.112:0/3311732674 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13e4188ce0 con 0x7f13e4086520 2021-02-10T11:42:24.307 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.305 7f13e921d700 1 -- 172.21.2.112:0/3311732674 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13e4189270 con 0x7f13e4086520 2021-02-10T11:42:24.307 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.305 7f13db7fe700 1 -- 172.21.2.112:0/3311732674 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f13d4002020 con 0x7f13e4086520 2021-02-10T11:42:24.307 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.305 7f13db7fe700 1 -- 172.21.2.112:0/3311732674 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f13d4005310 con 0x7f13e4086520 2021-02-10T11:42:24.307 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.305 7f13db7fe700 1 --2- 172.21.2.112:0/3311732674 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f13c8018730 0x7f13c801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:24.307 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.305 7f13db7fe700 1 -- 172.21.2.112:0/3311732674 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f13d40327c0 con 0x7f13e4086520 2021-02-10T11:42:24.307 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.306 7f13e259c700 1 --2- 172.21.2.112:0/3311732674 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f13c8018730 0x7f13c801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:24.307 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.306 7f13e259c700 1 --2- 172.21.2.112:0/3311732674 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f13c8018730 0x7f13c801ac10 crc :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:24.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.336 7f13e921d700 1 -- 172.21.2.112:0/3311732674 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13e40fe1d0 con 0x7f13e4086520 2021-02-10T11:42:24.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.339 7f13db7fe700 1 -- 172.21.2.112:0/3311732674 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f13d40009b0 con 0x7f13e4086520 2021-02-10T11:42:24.469 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.470 7f13e921d700 1 -- 172.21.2.112:0/3311732674 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f13e40fe1d0 con 0x7f13e4086520 2021-02-10T11:42:24.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.473 7f13db7fe700 1 -- 172.21.2.112:0/3311732674 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.2 v77) v1 ==== 184+0+0 (secure 0 0 0) 0x7f13d4005310 con 0x7f13e4086520 2021-02-10T11:42:24.472 INFO:teuthology.orchestra.run.gibba012.stderr:updated caps for client.2 2021-02-10T11:42:24.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.475 7f13e921d700 1 -- 172.21.2.112:0/3311732674 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f13c8018730 msgr2=0x7f13c801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:24.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.475 7f13e921d700 1 --2- 172.21.2.112:0/3311732674 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f13c8018730 0x7f13c801ac10 crc :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:24.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.475 7f13e921d700 1 -- 172.21.2.112:0/3311732674 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f13e4086520 msgr2=0x7f13e41a1cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:24.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.475 7f13e921d700 1 --2- 172.21.2.112:0/3311732674 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f13e4086520 0x7f13e41a1cb0 secure :-1 s=READY pgs=2334 cs=0 l=1 rev1=1 rx=0x7f13d4006f00 tx=0x7f13d400b940).stop 2021-02-10T11:42:24.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.475 7f13e921d700 1 -- 172.21.2.112:0/3311732674 shutdown_connections 2021-02-10T11:42:24.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.475 7f13e921d700 1 --2- 172.21.2.112:0/3311732674 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f13c8018730 0x7f13c801ac10 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:24.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.475 7f13e921d700 1 --2- 172.21.2.112:0/3311732674 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f13e4086920 0x7f13e4187e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:24.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.475 7f13e921d700 1 --2- 172.21.2.112:0/3311732674 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f13e41a22d0 0x7f13e41884a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:24.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.475 7f13e921d700 1 --2- 172.21.2.112:0/3311732674 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f13e4086520 0x7f13e41a1cb0 unknown :-1 s=CLOSED pgs=2334 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:24.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.475 7f13e921d700 1 -- 172.21.2.112:0/3311732674 >> 172.21.2.112:0/3311732674 conn(0x7f13e419f880 msgr2=0x7f13e408f890 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:24.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.475 7f13e921d700 1 -- 172.21.2.112:0/3311732674 shutdown_connections 2021-02-10T11:42:24.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.475 7f13e921d700 1 -- 172.21.2.112:0/3311732674 wait complete. 2021-02-10T11:42:24.482 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.3 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:42:24.927 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.908 7f03fe67a700 1 Processor -- start 2021-02-10T11:42:24.927 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.908 7f03fe67a700 1 -- start start 2021-02-10T11:42:24.928 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.909 7f03fe67a700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f03f80d1c80 0x7f03f80cef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:24.928 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.909 7f03fe67a700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03f80d0180 con 0x7f03f80cf590 2021-02-10T11:42:24.928 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.909 7f03fe67a700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03f80fe1d0 con 0x7f03f80d1880 2021-02-10T11:42:24.928 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.909 7f03fe67a700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03f81b0cc0 con 0x7f03f80d1c80 2021-02-10T11:42:24.928 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.909 7f03fcc97700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f03f80cf590 0x7f03f80cf990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:55722/0 (socket says 172.21.2.112:55722) 2021-02-10T11:42:24.928 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.909 7f03fcc97700 1 -- 172.21.2.112:0/112654870 learned_addr learned my addr 172.21.2.112:0/112654870 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:42:24.928 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.909 7f03f6ffd700 1 -- 172.21.2.112:0/112654870 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4273523559 0 0) 0x7f03f0001500 con 0x7f03f80cf590 2021-02-10T11:42:24.928 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.909 7f03f6ffd700 1 -- 172.21.2.112:0/112654870 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03e0002fe0 con 0x7f03f80cf590 2021-02-10T11:42:24.929 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.909 7f03f77fe700 1 --2- 172.21.2.112:0/112654870 >> v2:172.21.2.120:3300/0 conn(0x7f03f80d1c80 0x7f03f80cef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:24.929 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.910 7f03f6ffd700 1 -- 172.21.2.112:0/112654870 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3126149897 0 0) 0x7f03f0001460 con 0x7f03f80cf590 2021-02-10T11:42:24.929 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.910 7f03f6ffd700 1 -- 172.21.2.112:0/112654870 >> v2:172.21.2.120:3300/0 conn(0x7f03f80d1c80 msgr2=0x7f03f80cef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:24.929 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.910 7f03f6ffd700 1 --2- 172.21.2.112:0/112654870 >> v2:172.21.2.120:3300/0 conn(0x7f03f80d1c80 0x7f03f80cef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:24.929 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.910 7f03f6ffd700 1 -- 172.21.2.112:0/112654870 >> v1:172.21.2.120:6789/0 conn(0x7f03f80d1880 legacy=0x7f03f81b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:24.930 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.910 7f03f6ffd700 1 -- 172.21.2.112:0/112654870 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03f8109a70 con 0x7f03f80cf590 2021-02-10T11:42:24.930 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.910 7f03f6ffd700 1 -- 172.21.2.112:0/112654870 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f03f0001980 con 0x7f03f80cf590 2021-02-10T11:42:24.930 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.910 7f03f6ffd700 1 -- 172.21.2.112:0/112654870 >> v1:172.21.2.112:6789/0 conn(0x7f03f80cf590 legacy=0x7f03f80cf990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:24.930 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.910 7f03f6ffd700 1 --2- 172.21.2.112:0/112654870 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f03e0003aa0 0x7f03e0005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:24.930 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.910 7f03f6ffd700 1 --2- 172.21.2.112:0/112654870 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f03e0006790 0x7f03e0008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:24.930 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.910 7f03f6ffd700 1 --2- 172.21.2.112:0/112654870 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f03e0009590 0x7f03e000bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:24.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.910 7f03f6ffd700 1 -- 172.21.2.112:0/112654870 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03e000c300 con 0x7f03e0003aa0 2021-02-10T11:42:24.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.910 7f03f6ffd700 1 -- 172.21.2.112:0/112654870 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03e000c520 con 0x7f03e0006790 2021-02-10T11:42:24.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.910 7f03f7fff700 1 --2- 172.21.2.112:0/112654870 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f03e0003aa0 0x7f03e0005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:24.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.910 7f03f6ffd700 1 -- 172.21.2.112:0/112654870 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03e0050750 con 0x7f03e0009590 2021-02-10T11:42:24.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.910 7f03f77fe700 1 --2- 172.21.2.112:0/112654870 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f03e0006790 0x7f03e0008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:24.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.910 7f03f7fff700 1 -- 172.21.2.112:0/112654870 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f03e0009590 msgr2=0x7f03e000bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:24.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.910 7f03f7fff700 1 --2- 172.21.2.112:0/112654870 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f03e0009590 0x7f03e000bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:24.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.910 7f03f7fff700 1 -- 172.21.2.112:0/112654870 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f03e0006790 msgr2=0x7f03e0008c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:24.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.910 7f03f7fff700 1 --2- 172.21.2.112:0/112654870 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f03e0006790 0x7f03e0008c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:24.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.910 7f03f7fff700 1 -- 172.21.2.112:0/112654870 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03e00509b0 con 0x7f03e0003aa0 2021-02-10T11:42:24.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.910 7f03fcc97700 1 -- 172.21.2.112:0/112654870 reap_dead start 2021-02-10T11:42:24.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.910 7f03f7fff700 1 --2- 172.21.2.112:0/112654870 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f03e0003aa0 0x7f03e0005ef0 secure :-1 s=READY pgs=2336 cs=0 l=1 rev1=1 rx=0x7f03e8006ab0 tx=0x7f03e8009b40).ready entity=mon.0 client_cookie=c3c2522a863ad913 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:24.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.910 7f03f6ffd700 1 -- 172.21.2.112:0/112654870 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f03e800b2e0 con 0x7f03e0003aa0 2021-02-10T11:42:24.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.911 7f03f6ffd700 1 -- 172.21.2.112:0/112654870 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03e800b660 con 0x7f03e0003aa0 2021-02-10T11:42:24.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.911 7f03f6ffd700 1 -- 172.21.2.112:0/112654870 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f03e800b2e0 con 0x7f03e0003aa0 2021-02-10T11:42:24.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.911 7f03fe67a700 1 -- 172.21.2.112:0/112654870 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f03e0003aa0 msgr2=0x7f03e0005ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:24.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.911 7f03fe67a700 1 --2- 172.21.2.112:0/112654870 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f03e0003aa0 0x7f03e0005ef0 secure :-1 s=READY pgs=2336 cs=0 l=1 rev1=1 rx=0x7f03e8006ab0 tx=0x7f03e8009b40).stop 2021-02-10T11:42:24.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.911 7f03fe67a700 1 -- 172.21.2.112:0/112654870 shutdown_connections 2021-02-10T11:42:24.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.911 7f03fe67a700 1 --2- 172.21.2.112:0/112654870 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f03e0003aa0 0x7f03e0005ef0 unknown :-1 s=CLOSED pgs=2336 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:24.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.911 7f03fe67a700 1 -- 172.21.2.112:0/112654870 >> 172.21.2.112:0/112654870 conn(0x7f03f80ce430 msgr2=0x7f03f80d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:24.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.911 7f03fe67a700 1 -- 172.21.2.112:0/112654870 shutdown_connections 2021-02-10T11:42:24.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.911 7f03fe67a700 1 -- 172.21.2.112:0/112654870 wait complete. 2021-02-10T11:42:24.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.926 7f03fe67a700 1 Processor -- start 2021-02-10T11:42:24.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.926 7f03fe67a700 1 -- start start 2021-02-10T11:42:24.935 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.926 7f03fe67a700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f03f80b38c0 0x7f03f80b3cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:24.935 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.926 7f03fe67a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f03f80b42e0 0x7f03f8094130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:24.935 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.926 7f03fe67a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f03f80b46e0 0x7f03f8094750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:24.935 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.926 7f03fe67a700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03f80c4040 con 0x7f03f80b42e0 2021-02-10T11:42:24.935 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.926 7f03fe67a700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03f8095c50 con 0x7f03f80b38c0 2021-02-10T11:42:24.935 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.926 7f03fe67a700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03f8095e70 con 0x7f03f80b46e0 2021-02-10T11:42:24.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.927 7f03f77fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f03f80b42e0 0x7f03f8094130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:24.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.927 7f03f77fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f03f80b42e0 0x7f03f8094130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:56272/0 (socket says 172.21.2.112:56272) 2021-02-10T11:42:24.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.927 7f03f77fe700 1 -- 172.21.2.112:0/3032645779 learned_addr learned my addr 172.21.2.112:0/3032645779 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:42:24.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.927 7f03f7fff700 1 --2- 172.21.2.112:0/3032645779 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f03f80b38c0 0x7f03f80b3cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:24.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.927 7f03fcc97700 1 --2- 172.21.2.112:0/3032645779 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f03f80b46e0 0x7f03f8094750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:24.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.927 7f03f77fe700 1 -- 172.21.2.112:0/3032645779 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f03f80b46e0 msgr2=0x7f03f8094750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:24.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.927 7f03f77fe700 1 --2- 172.21.2.112:0/3032645779 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f03f80b46e0 0x7f03f8094750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:24.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.927 7f03f77fe700 1 -- 172.21.2.112:0/3032645779 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f03f80b38c0 msgr2=0x7f03f80b3cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:24.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.927 7f03f77fe700 1 --2- 172.21.2.112:0/3032645779 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f03f80b38c0 0x7f03f80b3cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:24.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.927 7f03f77fe700 1 -- 172.21.2.112:0/3032645779 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03f8094d70 con 0x7f03f80b42e0 2021-02-10T11:42:24.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.927 7f03f7fff700 1 --2- 172.21.2.112:0/3032645779 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f03f80b38c0 0x7f03f80b3cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:42:24.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.927 7f03fcc97700 1 --2- 172.21.2.112:0/3032645779 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f03f80b46e0 0x7f03f8094750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:42:24.938 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.927 7f03f77fe700 1 --2- 172.21.2.112:0/3032645779 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f03f80b42e0 0x7f03f8094130 secure :-1 s=READY pgs=2337 cs=0 l=1 rev1=1 rx=0x7f03ec006f60 tx=0x7f03ec00b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:24.938 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.927 7f03f4ff9700 1 -- 172.21.2.112:0/3032645779 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f03ec0009b0 con 0x7f03f80b42e0 2021-02-10T11:42:24.938 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.927 7f03f4ff9700 1 -- 172.21.2.112:0/3032645779 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03ec006b00 con 0x7f03f80b42e0 2021-02-10T11:42:24.938 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.927 7f03f4ff9700 1 -- 172.21.2.112:0/3032645779 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f03ec005370 con 0x7f03f80b42e0 2021-02-10T11:42:24.938 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.927 7f03fe67a700 1 -- 172.21.2.112:0/3032645779 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03f8094d70 con 0x7f03f80b42e0 2021-02-10T11:42:24.938 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.927 7f03fe67a700 1 -- 172.21.2.112:0/3032645779 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03f8094d70 con 0x7f03f80b42e0 2021-02-10T11:42:24.939 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.928 7f03f4ff9700 1 -- 172.21.2.112:0/3032645779 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f03ec00a000 con 0x7f03f80b42e0 2021-02-10T11:42:24.939 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.928 7f03f4ff9700 1 --2- 172.21.2.112:0/3032645779 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f03d80186d0 0x7f03d801abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:24.939 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.928 7f03f4ff9700 1 -- 172.21.2.112:0/3032645779 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f03ec0327b0 con 0x7f03f80b42e0 2021-02-10T11:42:24.939 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.928 7f03f7fff700 1 --2- 172.21.2.112:0/3032645779 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f03d80186d0 0x7f03d801abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:24.939 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.929 7f03f7fff700 1 --2- 172.21.2.112:0/3032645779 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f03d80186d0 0x7f03d801abd0 crc :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:24.958 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.959 7f03fe67a700 1 -- 172.21.2.112:0/3032645779 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03f80fe1d0 con 0x7f03f80b42e0 2021-02-10T11:42:24.960 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:24.962 7f03f4ff9700 1 -- 172.21.2.112:0/3032645779 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f03ec0063a0 con 0x7f03f80b42e0 2021-02-10T11:42:25.088 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:25.089 7f03fe67a700 1 -- 172.21.2.112:0/3032645779 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f03f80fe1d0 con 0x7f03f80b42e0 2021-02-10T11:42:25.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:25.093 7f03f4ff9700 1 -- 172.21.2.112:0/3032645779 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.3 v78) v1 ==== 184+0+0 (secure 0 0 0) 0x7f03ec00a000 con 0x7f03f80b42e0 2021-02-10T11:42:25.095 INFO:teuthology.orchestra.run.gibba012.stderr:updated caps for client.3 2021-02-10T11:42:25.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:25.098 7f03fe67a700 1 -- 172.21.2.112:0/3032645779 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f03d80186d0 msgr2=0x7f03d801abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:25.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:25.098 7f03fe67a700 1 --2- 172.21.2.112:0/3032645779 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f03d80186d0 0x7f03d801abd0 crc :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:25.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:25.098 7f03fe67a700 1 -- 172.21.2.112:0/3032645779 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f03f80b42e0 msgr2=0x7f03f8094130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:25.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:25.098 7f03fe67a700 1 --2- 172.21.2.112:0/3032645779 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f03f80b42e0 0x7f03f8094130 secure :-1 s=READY pgs=2337 cs=0 l=1 rev1=1 rx=0x7f03ec006f60 tx=0x7f03ec00b970).stop 2021-02-10T11:42:25.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:25.098 7f03fe67a700 1 -- 172.21.2.112:0/3032645779 shutdown_connections 2021-02-10T11:42:25.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:25.098 7f03fe67a700 1 --2- 172.21.2.112:0/3032645779 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f03d80186d0 0x7f03d801abd0 unknown :-1 s=CLOSED pgs=1070 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:25.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:25.098 7f03fe67a700 1 --2- 172.21.2.112:0/3032645779 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f03f80b46e0 0x7f03f8094750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:25.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:25.098 7f03fe67a700 1 --2- 172.21.2.112:0/3032645779 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f03f80b42e0 0x7f03f8094130 unknown :-1 s=CLOSED pgs=2337 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:25.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:25.098 7f03fe67a700 1 --2- 172.21.2.112:0/3032645779 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f03f80b38c0 0x7f03f80b3cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:25.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:25.098 7f03fe67a700 1 -- 172.21.2.112:0/3032645779 >> 172.21.2.112:0/3032645779 conn(0x7f03f80ac780 msgr2=0x7f03f808a480 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:25.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:25.098 7f03fe67a700 1 -- 172.21.2.112:0/3032645779 shutdown_connections 2021-02-10T11:42:25.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:25.098 7f03fe67a700 1 -- 172.21.2.112:0/3032645779 wait complete. 2021-02-10T11:42:25.106 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:42:25.534 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:25.519 7f153cf74700 1 Processor -- start 2021-02-10T11:42:25.535 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:25.519 7f153cf74700 1 -- start start 2021-02-10T11:42:25.535 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:25.519 7f153cf74700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f15380d0340 0x7f15380ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:25.535 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:25.519 7f153cf74700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15380f0b10 con 0x7f15380d1870 2021-02-10T11:42:25.535 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:25.519 7f153cf74700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15381b0be0 con 0x7f15380d1c70 2021-02-10T11:42:25.535 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:25.519 7f153cf74700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15380e5ed0 con 0x7f15380d0340 2021-02-10T11:42:25.535 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:25.519 7f1536312700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f15380d0340 0x7f15380ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:25.535 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:25.519 7f1536312700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f15380d0340 0x7f15380ef640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:56282/0 (socket says 172.21.2.112:56282) 2021-02-10T11:42:25.535 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:25.519 7f1536312700 1 -- 172.21.2.112:0/1853374266 learned_addr learned my addr 172.21.2.112:0/1853374266 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:42:25.535 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:25.520 7f1536312700 1 -- 172.21.2.112:0/1853374266 >> v1:172.21.2.120:6789/0 conn(0x7f15380d1c70 legacy=0x7f15380efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:25.535 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:25.520 7f1536312700 1 -- 172.21.2.112:0/1853374266 >> v1:172.21.2.112:6789/0 conn(0x7f15380d1870 legacy=0x7f15381b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:25.535 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:25.520 7f1535b11700 1 -- 172.21.2.112:0/1853374266 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1489179399 0 0) 0x7f1520000ce0 con 0x7f15380d1870 2021-02-10T11:42:25.536 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:25.520 7f1536312700 1 -- 172.21.2.112:0/1853374266 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15380f0450 con 0x7f15380d0340 2021-02-10T11:42:25.536 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:25.520 7f1536312700 1 --2- 172.21.2.112:0/1853374266 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f15380d0340 0x7f15380ef640 secure :-1 s=READY pgs=2339 cs=0 l=1 rev1=1 rx=0x7f1528006700 tx=0x7f1528009b00).ready entity=mon.0 client_cookie=168f4842d7658529 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:25.536 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:25.520 7f1535b11700 1 -- 172.21.2.112:0/1853374266 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f152800c0c0 con 0x7f15380d0340 2021-02-10T11:42:25.536 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:25.520 7f1535b11700 1 -- 172.21.2.112:0/1853374266 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f152800c340 con 0x7f15380d0340 2021-02-10T11:42:25.536 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:25.520 7f1535b11700 1 -- 172.21.2.112:0/1853374266 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f152800c820 con 0x7f15380d0340 2021-02-10T11:42:25.536 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:25.520 7f153cf74700 1 -- 172.21.2.112:0/1853374266 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f15380d0340 msgr2=0x7f15380ef640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:25.536 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:25.520 7f153cf74700 1 --2- 172.21.2.112:0/1853374266 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f15380d0340 0x7f15380ef640 secure :-1 s=READY pgs=2339 cs=0 l=1 rev1=1 rx=0x7f1528006700 tx=0x7f1528009b00).stop 2021-02-10T11:42:25.536 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:25.521 7f153cf74700 1 -- 172.21.2.112:0/1853374266 shutdown_connections 2021-02-10T11:42:25.536 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:25.521 7f153cf74700 1 --2- 172.21.2.112:0/1853374266 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f15380d0340 0x7f15380ef640 unknown :-1 s=CLOSED pgs=2339 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:25.536 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:25.521 7f153cf74700 1 -- 172.21.2.112:0/1853374266 >> 172.21.2.112:0/1853374266 conn(0x7f15380ce450 msgr2=0x7f15380d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:25.536 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:25.521 7f153cf74700 1 -- 172.21.2.112:0/1853374266 shutdown_connections 2021-02-10T11:42:25.536 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:25.521 7f153cf74700 1 -- 172.21.2.112:0/1853374266 wait complete. 2021-02-10T11:42:25.537 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:25.536 7f153cf74700 1 Processor -- start 2021-02-10T11:42:25.537 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:25.536 7f153cf74700 1 -- start start 2021-02-10T11:42:25.538 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:25.536 7f153cf74700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f153818e810 0x7f15380ef200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:25.538 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:25.536 7f153cf74700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f15380ef820 0x7f153818d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:25.538 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:25.536 7f153cf74700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f153818ec10 0x7f153818ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:25.539 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:25.536 7f153cf74700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f153819e010 con 0x7f153818e810 2021-02-10T11:42:25.539 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:25.536 7f153cf74700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15380ac6c0 con 0x7f15380ef820 2021-02-10T11:42:25.539 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:25.536 7f153cf74700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f153818e3f0 con 0x7f153818ec10 2021-02-10T11:42:25.539 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:25.536 7f1536b13700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f153818e810 0x7f15380ef200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:25.539 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:25.536 7f1536b13700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f153818e810 0x7f15380ef200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:56286/0 (socket says 172.21.2.112:56286) 2021-02-10T11:42:25.539 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:25.536 7f1536b13700 1 -- 172.21.2.112:0/305151526 learned_addr learned my addr 172.21.2.112:0/305151526 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:42:25.539 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:25.537 7f1536b13700 1 -- 172.21.2.112:0/305151526 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f153818ec10 msgr2=0x7f153818ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:25.539 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:25.537 7f1536b13700 1 --2- 172.21.2.112:0/305151526 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f153818ec10 0x7f153818ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:25.539 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:25.537 7f1536b13700 1 -- 172.21.2.112:0/305151526 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f15380ef820 msgr2=0x7f153818d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:25.539 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:25.537 7f1536b13700 1 --2- 172.21.2.112:0/305151526 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f15380ef820 0x7f153818d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:25.540 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:25.537 7f1536b13700 1 -- 172.21.2.112:0/305151526 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1538095fe0 con 0x7f153818e810 2021-02-10T11:42:25.540 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:25.537 7f1536b13700 1 --2- 172.21.2.112:0/305151526 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f153818e810 0x7f15380ef200 secure :-1 s=READY pgs=2340 cs=0 l=1 rev1=1 rx=0x7f152000b580 tx=0x7f152000b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:25.540 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:25.537 7f152f7fe700 1 -- 172.21.2.112:0/305151526 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1520000ce0 con 0x7f153818e810 2021-02-10T11:42:25.540 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:25.537 7f152f7fe700 1 -- 172.21.2.112:0/305151526 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1520001100 con 0x7f153818e810 2021-02-10T11:42:25.540 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:25.537 7f152f7fe700 1 -- 172.21.2.112:0/305151526 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1520001620 con 0x7f153818e810 2021-02-10T11:42:25.540 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:25.537 7f153cf74700 1 -- 172.21.2.112:0/305151526 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1538095fe0 con 0x7f153818e810 2021-02-10T11:42:25.540 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:25.537 7f153cf74700 1 -- 172.21.2.112:0/305151526 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1538095fe0 con 0x7f153818e810 2021-02-10T11:42:25.540 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:25.538 7f152f7fe700 1 -- 172.21.2.112:0/305151526 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1520000c80 con 0x7f153818e810 2021-02-10T11:42:25.540 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:25.538 7f152f7fe700 1 --2- 172.21.2.112:0/305151526 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f151c018730 0x7f151c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:25.540 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:25.538 7f152f7fe700 1 -- 172.21.2.112:0/305151526 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f15200327f0 con 0x7f153818e810 2021-02-10T11:42:25.540 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:25.538 7f1536312700 1 --2- 172.21.2.112:0/305151526 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f151c018730 0x7f151c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:25.540 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:25.539 7f1536312700 1 --2- 172.21.2.112:0/305151526 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f151c018730 0x7f151c01abc0 crc :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:25.568 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:25.569 7f153cf74700 1 -- 172.21.2.112:0/305151526 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f153819e010 con 0x7f153818e810 2021-02-10T11:42:25.570 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:25.572 7f152f7fe700 1 -- 172.21.2.112:0/305151526 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1520006bb0 con 0x7f153818e810 2021-02-10T11:42:25.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:25.697 7f153cf74700 1 -- 172.21.2.112:0/305151526 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f153819e010 con 0x7f153818e810 2021-02-10T11:42:25.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:25.698 7f152f7fe700 1 -- 172.21.2.112:0/305151526 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+2108 (secure 0 0 0) 0x7f1520000c80 con 0x7f153818e810 2021-02-10T11:42:25.697 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:42:25.698 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":105,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11040,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/1814390924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":1814390924},{"type":"v1","addr":"172.21.2.112:6833","nonce":1814390924}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":101}],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:42:19.351348","modified":"2021-02-10 11:42:21.357585","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8203},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8203":{"gid":8203,"name":"b","rank":0,"incarnation":102,"state":"up:active","state_seq":3,"addr":"172.21.2.120:6835/2761661070","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":2761661070},{"type":"v1","addr":"172.21.2.120:6835","nonce":2761661070}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":13}]} 2021-02-10T11:42:25.698 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 105 2021-02-10T11:42:25.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:25.700 7f153cf74700 1 -- 172.21.2.112:0/305151526 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f151c018730 msgr2=0x7f151c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:25.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:25.700 7f153cf74700 1 --2- 172.21.2.112:0/305151526 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f151c018730 0x7f151c01abc0 crc :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:25.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:25.700 7f153cf74700 1 -- 172.21.2.112:0/305151526 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f153818e810 msgr2=0x7f15380ef200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:25.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:25.700 7f153cf74700 1 --2- 172.21.2.112:0/305151526 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f153818e810 0x7f15380ef200 secure :-1 s=READY pgs=2340 cs=0 l=1 rev1=1 rx=0x7f152000b580 tx=0x7f152000b920).stop 2021-02-10T11:42:25.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:25.700 7f153cf74700 1 -- 172.21.2.112:0/305151526 shutdown_connections 2021-02-10T11:42:25.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:25.700 7f153cf74700 1 --2- 172.21.2.112:0/305151526 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f151c018730 0x7f151c01abc0 unknown :-1 s=CLOSED pgs=1071 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:25.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:25.700 7f153cf74700 1 --2- 172.21.2.112:0/305151526 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f15380ef820 0x7f153818d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:25.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:25.700 7f153cf74700 1 --2- 172.21.2.112:0/305151526 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f153818ec10 0x7f153818ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:25.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:25.700 7f153cf74700 1 --2- 172.21.2.112:0/305151526 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f153818e810 0x7f15380ef200 unknown :-1 s=CLOSED pgs=2340 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:25.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:25.700 7f153cf74700 1 -- 172.21.2.112:0/305151526 >> 172.21.2.112:0/305151526 conn(0x7f15380efcf0 msgr2=0x7f15380f1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:25.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:25.700 7f153cf74700 1 -- 172.21.2.112:0/305151526 shutdown_connections 2021-02-10T11:42:25.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:25.700 7f153cf74700 1 -- 172.21.2.112:0/305151526 wait complete. 2021-02-10T11:42:25.910 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 105, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:42:19.351348', 'modified': '2021-02-10 11:42:21.357585', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8203}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8203': {'gid': 8203, 'name': 'b', 'rank': 0, 'incarnation': 102, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.120:6835/2761661070', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.120:6834', 'nonce': 2761661070}, {'type': 'v1', 'addr': '172.21.2.120:6835', 'nonce': 2761661070}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [27], 'metadata_pool': 26, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:42:25.910 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:42:25.910 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:42:26.308 INFO:teuthology.orchestra.run.gibba020.stdout:{ 2021-02-10T11:42:26.308 INFO:teuthology.orchestra.run.gibba020.stdout: "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", 2021-02-10T11:42:26.309 INFO:teuthology.orchestra.run.gibba020.stdout: "whoami": 0, 2021-02-10T11:42:26.309 INFO:teuthology.orchestra.run.gibba020.stdout: "id": 8203, 2021-02-10T11:42:26.309 INFO:teuthology.orchestra.run.gibba020.stdout: "want_state": "up:active", 2021-02-10T11:42:26.309 INFO:teuthology.orchestra.run.gibba020.stdout: "state": "up:active", 2021-02-10T11:42:26.309 INFO:teuthology.orchestra.run.gibba020.stdout: "rank_uptime": 6.9568132819999997, 2021-02-10T11:42:26.309 INFO:teuthology.orchestra.run.gibba020.stdout: "mdsmap_epoch": 105, 2021-02-10T11:42:26.309 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch": 125, 2021-02-10T11:42:26.309 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch_barrier": 125, 2021-02-10T11:42:26.309 INFO:teuthology.orchestra.run.gibba020.stdout: "uptime": 12.58788113 2021-02-10T11:42:26.309 INFO:teuthology.orchestra.run.gibba020.stdout:} 2021-02-10T11:42:26.312 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", "whoami": 0, "id": 8203, "want_state": "up:active", "state": "up:active", "rank_uptime": 6.9568132819999997, "mdsmap_epoch": 105, "osdmap_epoch": 125, "osdmap_epoch_barrier": 125, "uptime": 12.58788113 } 2021-02-10T11:42:26.313 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:42:26.741 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:26.726 7f95a80de700 1 Processor -- start 2021-02-10T11:42:26.741 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:26.726 7f95a80de700 1 -- start start 2021-02-10T11:42:26.742 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:26.726 7f95a80de700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f95a00d1870 0x7f95a01b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:26.742 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:26.726 7f95a80de700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f95a00f0650 0x7f95a00cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:26.742 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:26.726 7f95a80de700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95a00f0b10 con 0x7f95a01b0a10 2021-02-10T11:42:26.742 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:26.726 7f95a80de700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95a00e5ed0 con 0x7f95a00d1870 2021-02-10T11:42:26.742 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:26.726 7f95a80de700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95a00d1c70 con 0x7f95a00f0650 2021-02-10T11:42:26.742 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:26.726 7f95a5efa700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f95a00d1870 0x7f95a01b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:26.742 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:26.726 7f95a56f9700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f95a01b0a10 0x7f95a00cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:55756/0 (socket says 172.21.2.112:55756) 2021-02-10T11:42:26.742 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:26.726 7f95a5efa700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f95a00d1870 0x7f95a01b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:56294/0 (socket says 172.21.2.112:56294) 2021-02-10T11:42:26.742 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:26.726 7f95a56f9700 1 -- 172.21.2.112:0/1675814666 learned_addr learned my addr 172.21.2.112:0/1675814666 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:42:26.742 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:26.727 7f95a4ef8700 1 -- 172.21.2.112:0/1675814666 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 804491389 0 0) 0x7f9594001430 con 0x7f95a01b0a10 2021-02-10T11:42:26.742 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:26.727 7f95a66fb700 1 --2- 172.21.2.112:0/1675814666 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f95a00f0650 0x7f95a00cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:26.742 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:26.727 7f95a4ef8700 1 -- 172.21.2.112:0/1675814666 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f958c002fe0 con 0x7f95a01b0a10 2021-02-10T11:42:26.743 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:26.727 7f95a5efa700 1 -- 172.21.2.112:0/1675814666 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f95a00f0650 msgr2=0x7f95a00cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:26.743 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:26.727 7f95a5efa700 1 --2- 172.21.2.112:0/1675814666 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f95a00f0650 0x7f95a00cfb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:26.743 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:26.727 7f95a5efa700 1 -- 172.21.2.112:0/1675814666 >> v1:172.21.2.112:6789/0 conn(0x7f95a01b0a10 legacy=0x7f95a00cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:26.743 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:26.727 7f95a5efa700 1 -- 172.21.2.112:0/1675814666 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95a00d1e90 con 0x7f95a00d1870 2021-02-10T11:42:26.743 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:26.727 7f95a5efa700 1 --2- 172.21.2.112:0/1675814666 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f95a00d1870 0x7f95a01b03f0 secure :-1 s=READY pgs=2342 cs=0 l=1 rev1=1 rx=0x7f959c006650 tx=0x7f959c009a80).ready entity=mon.0 client_cookie=d15bb1d4f084cdcf server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:26.743 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:26.727 7f95a4ef8700 1 -- 172.21.2.112:0/1675814666 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f959c00c0c0 con 0x7f95a00d1870 2021-02-10T11:42:26.743 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:26.727 7f95a4ef8700 1 -- 172.21.2.112:0/1675814666 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f959c00c340 con 0x7f95a00d1870 2021-02-10T11:42:26.743 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:26.727 7f95a4ef8700 1 -- 172.21.2.112:0/1675814666 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f959c00c860 con 0x7f95a00d1870 2021-02-10T11:42:26.743 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:26.728 7f95a80de700 1 -- 172.21.2.112:0/1675814666 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f95a00d1870 msgr2=0x7f95a01b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:26.743 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:26.728 7f95a80de700 1 --2- 172.21.2.112:0/1675814666 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f95a00d1870 0x7f95a01b03f0 secure :-1 s=READY pgs=2342 cs=0 l=1 rev1=1 rx=0x7f959c006650 tx=0x7f959c009a80).stop 2021-02-10T11:42:26.743 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:26.728 7f95a80de700 1 -- 172.21.2.112:0/1675814666 shutdown_connections 2021-02-10T11:42:26.743 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:26.728 7f95a80de700 1 --2- 172.21.2.112:0/1675814666 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f95a00f0650 0x7f95a00cfb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:26.743 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:26.728 7f95a80de700 1 --2- 172.21.2.112:0/1675814666 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f95a00d1870 0x7f95a01b03f0 unknown :-1 s=CLOSED pgs=2342 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:26.743 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:26.728 7f95a80de700 1 -- 172.21.2.112:0/1675814666 >> 172.21.2.112:0/1675814666 conn(0x7f95a00ce450 msgr2=0x7f95a00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:26.744 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:26.728 7f95a80de700 1 -- 172.21.2.112:0/1675814666 shutdown_connections 2021-02-10T11:42:26.744 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:26.729 7f95a80de700 1 -- 172.21.2.112:0/1675814666 wait complete. 2021-02-10T11:42:26.745 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:26.743 7f95a80de700 1 Processor -- start 2021-02-10T11:42:26.745 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:26.743 7f95a80de700 1 -- start start 2021-02-10T11:42:26.746 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:26.743 7f95a80de700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f95a00ada90 0x7f95a00ade90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:26.746 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:26.743 7f95a80de700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f95a0099d20 0x7f95a009a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:26.746 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:26.743 7f95a80de700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f95a009a740 0x7f95a0097bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:26.746 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:26.743 7f95a80de700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95a00997a0 con 0x7f95a009a740 2021-02-10T11:42:26.746 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:26.743 7f95a80de700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95a018f0c0 con 0x7f95a00ada90 2021-02-10T11:42:26.746 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:26.743 7f95a80de700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95a009ab40 con 0x7f95a0099d20 2021-02-10T11:42:26.746 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:26.744 7f95a66fb700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f95a009a740 0x7f95a0097bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:26.746 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:26.744 7f95a66fb700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f95a009a740 0x7f95a0097bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:56300/0 (socket says 172.21.2.112:56300) 2021-02-10T11:42:26.746 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:26.744 7f95a66fb700 1 -- 172.21.2.112:0/3351955136 learned_addr learned my addr 172.21.2.112:0/3351955136 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:42:26.746 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:26.744 7f95a5efa700 1 --2- 172.21.2.112:0/3351955136 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f95a00ada90 0x7f95a00ade90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:26.747 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:26.744 7f95a56f9700 1 --2- 172.21.2.112:0/3351955136 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f95a0099d20 0x7f95a009a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:26.747 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:26.744 7f95a66fb700 1 -- 172.21.2.112:0/3351955136 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f95a0099d20 msgr2=0x7f95a009a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:26.747 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:26.744 7f95a66fb700 1 --2- 172.21.2.112:0/3351955136 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f95a0099d20 0x7f95a009a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:26.747 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:26.744 7f95a66fb700 1 -- 172.21.2.112:0/3351955136 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f95a00ada90 msgr2=0x7f95a00ade90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:26.747 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:26.744 7f95a66fb700 1 --2- 172.21.2.112:0/3351955136 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f95a00ada90 0x7f95a00ade90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:26.747 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:26.744 7f95a66fb700 1 -- 172.21.2.112:0/3351955136 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95a009ad60 con 0x7f95a009a740 2021-02-10T11:42:26.747 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:26.744 7f95a5efa700 1 --2- 172.21.2.112:0/3351955136 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f95a00ada90 0x7f95a00ade90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:42:26.747 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:26.744 7f95a56f9700 1 --2- 172.21.2.112:0/3351955136 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f95a0099d20 0x7f95a009a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:42:26.747 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:26.744 7f95a66fb700 1 --2- 172.21.2.112:0/3351955136 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f95a009a740 0x7f95a0097bc0 secure :-1 s=READY pgs=2343 cs=0 l=1 rev1=1 rx=0x7f9598006f20 tx=0x7f959800b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:26.747 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:26.744 7f95927fc700 1 -- 172.21.2.112:0/3351955136 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9598006bb0 con 0x7f95a009a740 2021-02-10T11:42:26.747 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:26.744 7f95927fc700 1 -- 172.21.2.112:0/3351955136 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f959800bd80 con 0x7f95a009a740 2021-02-10T11:42:26.747 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:26.745 7f95a80de700 1 -- 172.21.2.112:0/3351955136 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95a009ad60 con 0x7f95a009a740 2021-02-10T11:42:26.747 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:26.745 7f95927fc700 1 -- 172.21.2.112:0/3351955136 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f95980022b0 con 0x7f95a009a740 2021-02-10T11:42:26.748 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:26.745 7f95a80de700 1 -- 172.21.2.112:0/3351955136 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95a009ad60 con 0x7f95a009a740 2021-02-10T11:42:26.748 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:26.745 7f95927fc700 1 -- 172.21.2.112:0/3351955136 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9598005330 con 0x7f95a009a740 2021-02-10T11:42:26.748 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:26.745 7f95927fc700 1 --2- 172.21.2.112:0/3351955136 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9588018730 0x7f958801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:26.748 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:26.745 7f95927fc700 1 -- 172.21.2.112:0/3351955136 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f9598032810 con 0x7f95a009a740 2021-02-10T11:42:26.748 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:26.746 7f95a5efa700 1 --2- 172.21.2.112:0/3351955136 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9588018730 0x7f958801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:26.748 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:26.747 7f95a5efa700 1 --2- 172.21.2.112:0/3351955136 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9588018730 0x7f958801ac10 crc :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:26.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:26.776 7f95a80de700 1 -- 172.21.2.112:0/3351955136 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95a00ebc00 con 0x7f95a009a740 2021-02-10T11:42:26.777 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:26.779 7f95927fc700 1 -- 172.21.2.112:0/3351955136 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9598006370 con 0x7f95a009a740 2021-02-10T11:42:26.901 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:26.902 7f95a80de700 1 -- 172.21.2.112:0/3351955136 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f95a00f0b10 con 0x7f95a009a740 2021-02-10T11:42:26.902 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:26.903 7f95927fc700 1 -- 172.21.2.112:0/3351955136 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v125) v1 ==== 74+0+14007 (secure 0 0 0) 0x7f9598000e70 con 0x7f95a009a740 2021-02-10T11:42:26.902 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:42:26.902 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":125,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:42:20.353491","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":27,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:42:17.322526","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"124","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":27,"pool_name":"cephfs_data","create_time":"2021-02-10 11:42:18.395545","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"125","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:42:26.903 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:26.904 7f95a80de700 1 -- 172.21.2.112:0/3351955136 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9588018730 msgr2=0x7f958801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:26.903 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:26.905 7f95a80de700 1 --2- 172.21.2.112:0/3351955136 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9588018730 0x7f958801ac10 crc :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:26.904 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:26.905 7f95a80de700 1 -- 172.21.2.112:0/3351955136 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f95a009a740 msgr2=0x7f95a0097bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:26.904 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:26.905 7f95a80de700 1 --2- 172.21.2.112:0/3351955136 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f95a009a740 0x7f95a0097bc0 secure :-1 s=READY pgs=2343 cs=0 l=1 rev1=1 rx=0x7f9598006f20 tx=0x7f959800b990).stop 2021-02-10T11:42:26.904 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:26.905 7f95a80de700 1 -- 172.21.2.112:0/3351955136 shutdown_connections 2021-02-10T11:42:26.904 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:26.905 7f95a80de700 1 --2- 172.21.2.112:0/3351955136 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9588018730 0x7f958801ac10 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:26.904 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:26.905 7f95a80de700 1 --2- 172.21.2.112:0/3351955136 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f95a009a740 0x7f95a0097bc0 unknown :-1 s=CLOSED pgs=2343 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:26.904 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:26.905 7f95a80de700 1 --2- 172.21.2.112:0/3351955136 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f95a0099d20 0x7f95a009a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:26.904 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:26.905 7f95a80de700 1 --2- 172.21.2.112:0/3351955136 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f95a00ada90 0x7f95a00ade90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:26.904 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:26.905 7f95a80de700 1 -- 172.21.2.112:0/3351955136 >> 172.21.2.112:0/3351955136 conn(0x7f95a00f0550 msgr2=0x7f95a0191730 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:26.904 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:26.905 7f95a80de700 1 -- 172.21.2.112:0/3351955136 shutdown_connections 2021-02-10T11:42:26.904 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:26.905 7f95a80de700 1 -- 172.21.2.112:0/3351955136 wait complete. 2021-02-10T11:42:26.912 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:13:57.508758', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 26, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:42:17.322526', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '124', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 27, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:42:18.395545', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '125', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:42:26.913 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:42:27.344 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.329 7faa1bb6d700 1 Processor -- start 2021-02-10T11:42:27.344 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.329 7faa1bb6d700 1 -- start start 2021-02-10T11:42:27.344 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.329 7faa1bb6d700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7faa140d1870 0x7faa141b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:27.344 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.329 7faa1bb6d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7faa140cff70 0x7faa14109130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:27.344 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.329 7faa1bb6d700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa140fe1c0 con 0x7faa141b0ae0 2021-02-10T11:42:27.344 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.329 7faa1bb6d700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa14109ad0 con 0x7faa140cff70 2021-02-10T11:42:27.345 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.329 7faa1bb6d700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa14109750 con 0x7faa140d1870 2021-02-10T11:42:27.345 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.329 7faa1a18a700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7faa140cff70 0x7faa14109130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:27.345 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.329 7faa1a18a700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7faa140cff70 0x7faa14109130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:56308/0 (socket says 172.21.2.112:56308) 2021-02-10T11:42:27.345 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.329 7faa1a18a700 1 -- 172.21.2.112:0/2630890058 learned_addr learned my addr 172.21.2.112:0/2630890058 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:42:27.345 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.330 7faa1a18a700 1 -- 172.21.2.112:0/2630890058 >> v2:172.21.2.120:3300/0 conn(0x7faa140d1870 msgr2=0x7faa141b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:27.345 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.330 7faa1a18a700 1 --2- 172.21.2.112:0/2630890058 >> v2:172.21.2.120:3300/0 conn(0x7faa140d1870 0x7faa141b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:27.345 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.330 7faa1a18a700 1 -- 172.21.2.112:0/2630890058 >> v1:172.21.2.120:6789/0 conn(0x7faa141b0ae0 legacy=0x7faa14108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:27.345 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.330 7faa1a18a700 1 -- 172.21.2.112:0/2630890058 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa14088070 con 0x7faa140cff70 2021-02-10T11:42:27.345 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.330 7faa1a18a700 1 --2- 172.21.2.112:0/2630890058 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faa140cff70 0x7faa14109130 secure :-1 s=READY pgs=2344 cs=0 l=1 rev1=1 rx=0x7faa0c006f00 tx=0x7faa0c009f10).ready entity=mon.0 client_cookie=945dbd935b1daade server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:27.345 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.330 7faa18987700 1 -- 172.21.2.112:0/2630890058 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faa0c003030 con 0x7faa140cff70 2021-02-10T11:42:27.345 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.330 7faa18987700 1 -- 172.21.2.112:0/2630890058 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa0c00b3b0 con 0x7faa140cff70 2021-02-10T11:42:27.345 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.330 7faa18987700 1 -- 172.21.2.112:0/2630890058 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faa0c00b8d0 con 0x7faa140cff70 2021-02-10T11:42:27.345 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.330 7faa1bb6d700 1 -- 172.21.2.112:0/2630890058 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faa140cff70 msgr2=0x7faa14109130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:27.345 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.330 7faa1bb6d700 1 --2- 172.21.2.112:0/2630890058 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faa140cff70 0x7faa14109130 secure :-1 s=READY pgs=2344 cs=0 l=1 rev1=1 rx=0x7faa0c006f00 tx=0x7faa0c009f10).stop 2021-02-10T11:42:27.346 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.331 7faa1bb6d700 1 -- 172.21.2.112:0/2630890058 shutdown_connections 2021-02-10T11:42:27.346 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.331 7faa1bb6d700 1 --2- 172.21.2.112:0/2630890058 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faa140cff70 0x7faa14109130 unknown :-1 s=CLOSED pgs=2344 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:27.346 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.331 7faa1bb6d700 1 --2- 172.21.2.112:0/2630890058 >> v2:172.21.2.120:3300/0 conn(0x7faa140d1870 0x7faa141b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:27.346 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.331 7faa1bb6d700 1 -- 172.21.2.112:0/2630890058 >> 172.21.2.112:0/2630890058 conn(0x7faa140ce420 msgr2=0x7faa140d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:27.346 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.331 7faa1bb6d700 1 -- 172.21.2.112:0/2630890058 shutdown_connections 2021-02-10T11:42:27.346 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.331 7faa1bb6d700 1 -- 172.21.2.112:0/2630890058 wait complete. 2021-02-10T11:42:27.348 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.345 7faa1bb6d700 1 Processor -- start 2021-02-10T11:42:27.348 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.346 7faa1bb6d700 1 -- start start 2021-02-10T11:42:27.348 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.346 7faa1bb6d700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7faa140c3820 0x7faa140a7950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:27.348 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.346 7faa1bb6d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7faa141a44c0 0x7faa141a48c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:27.348 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.346 7faa1bb6d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faa140863d0 0x7faa140867d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:27.348 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.346 7faa1bb6d700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa14085ed0 con 0x7faa140863d0 2021-02-10T11:42:27.348 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.346 7faa1bb6d700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa140c0740 con 0x7faa140c3820 2021-02-10T11:42:27.348 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.346 7faa1bb6d700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa14086df0 con 0x7faa141a44c0 2021-02-10T11:42:27.348 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.346 7faa1a18a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faa140863d0 0x7faa140867d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:27.349 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.346 7faa1a18a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faa140863d0 0x7faa140867d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:56312/0 (socket says 172.21.2.112:56312) 2021-02-10T11:42:27.349 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.346 7faa1a18a700 1 -- 172.21.2.112:0/909650486 learned_addr learned my addr 172.21.2.112:0/909650486 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:42:27.349 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.347 7faa1a18a700 1 -- 172.21.2.112:0/909650486 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7faa141a44c0 msgr2=0x7faa141a48c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:27.349 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.347 7faa19989700 1 --2- 172.21.2.112:0/909650486 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7faa140c3820 0x7faa140a7950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:27.349 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.347 7faa19188700 1 --2- 172.21.2.112:0/909650486 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7faa141a44c0 0x7faa141a48c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:27.349 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.347 7faa1a18a700 1 --2- 172.21.2.112:0/909650486 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7faa141a44c0 0x7faa141a48c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:27.349 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.347 7faa1a18a700 1 -- 172.21.2.112:0/909650486 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7faa140c3820 msgr2=0x7faa140a7950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:27.349 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.347 7faa1a18a700 1 --2- 172.21.2.112:0/909650486 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7faa140c3820 0x7faa140a7950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:27.349 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.347 7faa1a18a700 1 -- 172.21.2.112:0/909650486 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa14087010 con 0x7faa140863d0 2021-02-10T11:42:27.349 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.347 7faa19989700 1 --2- 172.21.2.112:0/909650486 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7faa140c3820 0x7faa140a7950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:42:27.349 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.347 7faa1a18a700 1 --2- 172.21.2.112:0/909650486 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faa140863d0 0x7faa140867d0 secure :-1 s=READY pgs=2345 cs=0 l=1 rev1=1 rx=0x7faa0c00b940 tx=0x7faa0c00bce0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:27.349 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.347 7faa067fc700 1 -- 172.21.2.112:0/909650486 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faa0c009450 con 0x7faa140863d0 2021-02-10T11:42:27.350 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.347 7faa067fc700 1 -- 172.21.2.112:0/909650486 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa0c003030 con 0x7faa140863d0 2021-02-10T11:42:27.350 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.347 7faa1bb6d700 1 -- 172.21.2.112:0/909650486 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa14087010 con 0x7faa140863d0 2021-02-10T11:42:27.350 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.347 7faa067fc700 1 -- 172.21.2.112:0/909650486 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faa0c00c570 con 0x7faa140863d0 2021-02-10T11:42:27.350 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.347 7faa1bb6d700 1 -- 172.21.2.112:0/909650486 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa14087010 con 0x7faa140863d0 2021-02-10T11:42:27.350 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.348 7faa067fc700 1 -- 172.21.2.112:0/909650486 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7faa0c001270 con 0x7faa140863d0 2021-02-10T11:42:27.350 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.348 7faa067fc700 1 --2- 172.21.2.112:0/909650486 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa9fc018730 0x7fa9fc01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:27.350 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.348 7faa067fc700 1 -- 172.21.2.112:0/909650486 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5573+0+0 (secure 0 0 0) 0x7faa0c032ba0 con 0x7faa140863d0 2021-02-10T11:42:27.350 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.349 7faa19989700 1 --2- 172.21.2.112:0/909650486 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa9fc018730 0x7fa9fc01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:27.350 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.349 7faa19989700 1 --2- 172.21.2.112:0/909650486 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa9fc018730 0x7fa9fc01ac10 crc :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:27.378 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.379 7faa1bb6d700 1 -- 172.21.2.112:0/909650486 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa140fe1c0 con 0x7faa140863d0 2021-02-10T11:42:27.381 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.382 7faa067fc700 1 -- 172.21.2.112:0/909650486 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7faa0c00be20 con 0x7faa140863d0 2021-02-10T11:42:27.509 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.510 7faa1bb6d700 1 -- 172.21.2.112:0/909650486 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7faa140fe1c0 con 0x7faa140863d0 2021-02-10T11:42:27.509 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.510 7faa067fc700 1 -- 172.21.2.112:0/909650486 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v125) v1 ==== 92+0+10 (secure 0 0 0) 0x7faa0c001270 con 0x7faa140863d0 2021-02-10T11:42:27.509 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:42:27.511 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.512 7faa1bb6d700 1 -- 172.21.2.112:0/909650486 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa9fc018730 msgr2=0x7fa9fc01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:27.511 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.512 7faa1bb6d700 1 --2- 172.21.2.112:0/909650486 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa9fc018730 0x7fa9fc01ac10 crc :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:27.511 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.512 7faa1bb6d700 1 -- 172.21.2.112:0/909650486 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faa140863d0 msgr2=0x7faa140867d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:27.511 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.512 7faa1bb6d700 1 --2- 172.21.2.112:0/909650486 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faa140863d0 0x7faa140867d0 secure :-1 s=READY pgs=2345 cs=0 l=1 rev1=1 rx=0x7faa0c00b940 tx=0x7faa0c00bce0).stop 2021-02-10T11:42:27.512 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.512 7faa1bb6d700 1 -- 172.21.2.112:0/909650486 shutdown_connections 2021-02-10T11:42:27.512 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.512 7faa1bb6d700 1 --2- 172.21.2.112:0/909650486 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa9fc018730 0x7fa9fc01ac10 unknown :-1 s=CLOSED pgs=1073 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:27.512 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.512 7faa1bb6d700 1 --2- 172.21.2.112:0/909650486 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faa140863d0 0x7faa140867d0 unknown :-1 s=CLOSED pgs=2345 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:27.512 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.512 7faa1bb6d700 1 --2- 172.21.2.112:0/909650486 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7faa141a44c0 0x7faa141a48c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:27.512 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.512 7faa1bb6d700 1 --2- 172.21.2.112:0/909650486 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7faa140c3820 0x7faa140a7950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:27.512 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.512 7faa1bb6d700 1 -- 172.21.2.112:0/909650486 >> 172.21.2.112:0/909650486 conn(0x7faa140872c0 msgr2=0x7faa14095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:27.512 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.512 7faa1bb6d700 1 -- 172.21.2.112:0/909650486 shutdown_connections 2021-02-10T11:42:27.512 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.513 7faa1bb6d700 1 -- 172.21.2.112:0/909650486 wait complete. 2021-02-10T11:42:27.519 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:42:27.950 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.935 7f7f57cd5700 1 Processor -- start 2021-02-10T11:42:27.950 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.935 7f7f57cd5700 1 -- start start 2021-02-10T11:42:27.951 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.935 7f7f57cd5700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f7f500d1870 0x7f7f501b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:27.951 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.935 7f7f57cd5700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7f501b0ae0 0x7f7f50108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:27.951 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.935 7f7f57cd5700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f7f500d1c70 0x7f7f50108f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:27.951 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.935 7f7f57cd5700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f500fe1c0 con 0x7f7f500d1c70 2021-02-10T11:42:27.951 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.935 7f7f57cd5700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f50109a60 con 0x7f7f500d1870 2021-02-10T11:42:27.952 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.935 7f7f57cd5700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f50088070 con 0x7f7f501b0ae0 2021-02-10T11:42:27.952 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.936 7f7f562f2700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f7f500d1c70 0x7f7f50108f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:27.952 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.936 7f7f562f2700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f7f500d1c70 0x7f7f50108f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:56318/0 (socket says 172.21.2.112:56318) 2021-02-10T11:42:27.952 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.936 7f7f562f2700 1 -- 172.21.2.112:0/2791602483 learned_addr learned my addr 172.21.2.112:0/2791602483 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:42:27.952 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.936 7f7f562f2700 1 -- 172.21.2.112:0/2791602483 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7f501b0ae0 msgr2=0x7f7f50108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:27.952 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.936 7f7f562f2700 1 --2- 172.21.2.112:0/2791602483 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7f501b0ae0 0x7f7f50108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:27.953 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.936 7f7f562f2700 1 -- 172.21.2.112:0/2791602483 >> v2:172.21.2.120:3300/0 conn(0x7f7f500d1870 msgr2=0x7f7f501b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:27.953 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.936 7f7f562f2700 1 --2- 172.21.2.112:0/2791602483 >> v2:172.21.2.120:3300/0 conn(0x7f7f500d1870 0x7f7f501b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:27.953 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.936 7f7f562f2700 1 -- 172.21.2.112:0/2791602483 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f50109580 con 0x7f7f500d1c70 2021-02-10T11:42:27.953 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.936 7f7f562f2700 1 --2- 172.21.2.112:0/2791602483 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7f500d1c70 0x7f7f50108f60 secure :-1 s=READY pgs=2346 cs=0 l=1 rev1=1 rx=0x7f7f48006f00 tx=0x7f7f48009f10).ready entity=mon.0 client_cookie=2d4926e4060c9ba8 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:27.953 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.936 7f7f54aef700 1 -- 172.21.2.112:0/2791602483 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7f4800b3b0 con 0x7f7f500d1c70 2021-02-10T11:42:27.954 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.936 7f7f54aef700 1 -- 172.21.2.112:0/2791602483 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f4800b730 con 0x7f7f500d1c70 2021-02-10T11:42:27.954 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.936 7f7f54aef700 1 -- 172.21.2.112:0/2791602483 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7f4800bc50 con 0x7f7f500d1c70 2021-02-10T11:42:27.954 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.936 7f7f57cd5700 1 -- 172.21.2.112:0/2791602483 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7f500d1c70 msgr2=0x7f7f50108f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:27.954 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.936 7f7f57cd5700 1 --2- 172.21.2.112:0/2791602483 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7f500d1c70 0x7f7f50108f60 secure :-1 s=READY pgs=2346 cs=0 l=1 rev1=1 rx=0x7f7f48006f00 tx=0x7f7f48009f10).stop 2021-02-10T11:42:27.954 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.937 7f7f57cd5700 1 -- 172.21.2.112:0/2791602483 shutdown_connections 2021-02-10T11:42:27.954 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.937 7f7f57cd5700 1 --2- 172.21.2.112:0/2791602483 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7f500d1c70 0x7f7f50108f60 unknown :-1 s=CLOSED pgs=2346 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:27.955 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.937 7f7f57cd5700 1 --2- 172.21.2.112:0/2791602483 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7f501b0ae0 0x7f7f50108940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:27.955 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.937 7f7f57cd5700 1 --2- 172.21.2.112:0/2791602483 >> v2:172.21.2.120:3300/0 conn(0x7f7f500d1870 0x7f7f501b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:27.955 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.937 7f7f57cd5700 1 -- 172.21.2.112:0/2791602483 >> 172.21.2.112:0/2791602483 conn(0x7f7f500ce420 msgr2=0x7f7f500d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:27.955 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.937 7f7f57cd5700 1 -- 172.21.2.112:0/2791602483 shutdown_connections 2021-02-10T11:42:27.955 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.937 7f7f57cd5700 1 -- 172.21.2.112:0/2791602483 wait complete. 2021-02-10T11:42:27.959 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.952 7f7f57cd5700 1 Processor -- start 2021-02-10T11:42:27.959 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.952 7f7f57cd5700 1 -- start start 2021-02-10T11:42:27.959 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.952 7f7f57cd5700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7f500c3820 0x7f7f50084cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:27.959 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.952 7f7f57cd5700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7f500a0290 0x7f7f500852d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:27.959 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.952 7f7f57cd5700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7f50086270 0x7f7f50086670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:27.960 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.952 7f7f57cd5700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f500c0700 con 0x7f7f500c3820 2021-02-10T11:42:27.960 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.952 7f7f57cd5700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f500858f0 con 0x7f7f50086270 2021-02-10T11:42:27.960 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.952 7f7f57cd5700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f50085b10 con 0x7f7f500a0290 2021-02-10T11:42:27.960 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.952 7f7f55af1700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7f500c3820 0x7f7f50084cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:27.960 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.952 7f7f55af1700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7f500c3820 0x7f7f50084cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:56324/0 (socket says 172.21.2.112:56324) 2021-02-10T11:42:27.961 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.952 7f7f55af1700 1 -- 172.21.2.112:0/2258329337 learned_addr learned my addr 172.21.2.112:0/2258329337 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:42:27.961 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.953 7f7f562f2700 1 --2- 172.21.2.112:0/2258329337 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7f50086270 0x7f7f50086670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:27.961 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.953 7f7f55af1700 1 -- 172.21.2.112:0/2258329337 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7f500a0290 msgr2=0x7f7f500852d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:27.961 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.953 7f7f55af1700 1 --2- 172.21.2.112:0/2258329337 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7f500a0290 0x7f7f500852d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:27.961 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.953 7f7f55af1700 1 -- 172.21.2.112:0/2258329337 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7f50086270 msgr2=0x7f7f50086670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:27.961 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.953 7f7f55af1700 1 --2- 172.21.2.112:0/2258329337 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7f50086270 0x7f7f50086670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:27.962 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.953 7f7f55af1700 1 -- 172.21.2.112:0/2258329337 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f50086c90 con 0x7f7f500c3820 2021-02-10T11:42:27.962 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.953 7f7f562f2700 1 --2- 172.21.2.112:0/2258329337 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7f50086270 0x7f7f50086670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:42:27.962 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.953 7f7f55af1700 1 --2- 172.21.2.112:0/2258329337 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7f500c3820 0x7f7f50084cb0 secure :-1 s=READY pgs=2347 cs=0 l=1 rev1=1 rx=0x7f7f4c006f00 tx=0x7f7f4c00b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:27.962 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.953 7f7f467fc700 1 -- 172.21.2.112:0/2258329337 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7f4c00bd30 con 0x7f7f500c3820 2021-02-10T11:42:27.962 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.953 7f7f467fc700 1 -- 172.21.2.112:0/2258329337 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f4c000c90 con 0x7f7f500c3820 2021-02-10T11:42:27.963 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.953 7f7f467fc700 1 -- 172.21.2.112:0/2258329337 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7f4c0022a0 con 0x7f7f500c3820 2021-02-10T11:42:27.963 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.953 7f7f57cd5700 1 -- 172.21.2.112:0/2258329337 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f50086c90 con 0x7f7f500c3820 2021-02-10T11:42:27.963 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.953 7f7f57cd5700 1 -- 172.21.2.112:0/2258329337 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f50086c90 con 0x7f7f500c3820 2021-02-10T11:42:27.963 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.954 7f7f467fc700 1 -- 172.21.2.112:0/2258329337 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7f4c005310 con 0x7f7f500c3820 2021-02-10T11:42:27.963 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.954 7f7f467fc700 1 --2- 172.21.2.112:0/2258329337 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7f38018730 0x7f7f3801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:27.963 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.954 7f7f467fc700 1 -- 172.21.2.112:0/2258329337 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f7f4c032780 con 0x7f7f500c3820 2021-02-10T11:42:27.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.954 7f7f552f0700 1 --2- 172.21.2.112:0/2258329337 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7f38018730 0x7f7f3801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:27.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.956 7f7f552f0700 1 --2- 172.21.2.112:0/2258329337 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7f38018730 0x7f7f3801abc0 crc :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:27.984 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.985 7f7f57cd5700 1 -- 172.21.2.112:0/2258329337 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f50082a70 con 0x7f7f500c3820 2021-02-10T11:42:27.987 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:27.988 7f7f467fc700 1 -- 172.21.2.112:0/2258329337 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7f4c006b90 con 0x7f7f500c3820 2021-02-10T11:42:28.114 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:28.115 7f7f57cd5700 1 -- 172.21.2.112:0/2258329337 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f7f50082a70 con 0x7f7f500c3820 2021-02-10T11:42:28.114 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:28.115 7f7f467fc700 1 -- 172.21.2.112:0/2258329337 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v125) v1 ==== 104+0+10 (secure 0 0 0) 0x7f7f4c00bd30 con 0x7f7f500c3820 2021-02-10T11:42:28.115 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:42:28.117 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:28.117 7f7f57cd5700 1 -- 172.21.2.112:0/2258329337 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7f38018730 msgr2=0x7f7f3801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:28.117 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:28.117 7f7f57cd5700 1 --2- 172.21.2.112:0/2258329337 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7f38018730 0x7f7f3801abc0 crc :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:28.117 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:28.117 7f7f57cd5700 1 -- 172.21.2.112:0/2258329337 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7f500c3820 msgr2=0x7f7f50084cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:28.117 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:28.117 7f7f57cd5700 1 --2- 172.21.2.112:0/2258329337 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7f500c3820 0x7f7f50084cb0 secure :-1 s=READY pgs=2347 cs=0 l=1 rev1=1 rx=0x7f7f4c006f00 tx=0x7f7f4c00b940).stop 2021-02-10T11:42:28.117 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:28.117 7f7f57cd5700 1 -- 172.21.2.112:0/2258329337 shutdown_connections 2021-02-10T11:42:28.117 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:28.117 7f7f57cd5700 1 --2- 172.21.2.112:0/2258329337 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7f38018730 0x7f7f3801abc0 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:28.117 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:28.118 7f7f57cd5700 1 --2- 172.21.2.112:0/2258329337 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7f50086270 0x7f7f50086670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:28.117 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:28.118 7f7f57cd5700 1 --2- 172.21.2.112:0/2258329337 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7f500a0290 0x7f7f500852d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:28.117 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:28.118 7f7f57cd5700 1 --2- 172.21.2.112:0/2258329337 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7f500c3820 0x7f7f50084cb0 unknown :-1 s=CLOSED pgs=2347 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:28.118 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:28.118 7f7f57cd5700 1 -- 172.21.2.112:0/2258329337 >> 172.21.2.112:0/2258329337 conn(0x7f7f500872c0 msgr2=0x7f7f50095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:28.118 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:28.118 7f7f57cd5700 1 -- 172.21.2.112:0/2258329337 shutdown_connections 2021-02-10T11:42:28.118 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:28.118 7f7f57cd5700 1 -- 172.21.2.112:0/2258329337 wait complete. 2021-02-10T11:42:28.125 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:42:28.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:28.540 7f62de8e1700 1 Processor -- start 2021-02-10T11:42:28.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:28.540 7f62de8e1700 1 -- start start 2021-02-10T11:42:28.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:28.541 7f62de8e1700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f62d810aff0 0x7f62d80cf750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:28.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:28.541 7f62de8e1700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62d80fe1c0 con 0x7f62d80d1870 2021-02-10T11:42:28.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:28.541 7f62de8e1700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62d81b0cb0 con 0x7f62d80d1c70 2021-02-10T11:42:28.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:28.541 7f62de8e1700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62d8109a60 con 0x7f62d810aff0 2021-02-10T11:42:28.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:28.541 7f62d7fff700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f62d80d1870 0x7f62d81b04c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:55790/0 (socket says 172.21.2.112:55790) 2021-02-10T11:42:28.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:28.541 7f62d7fff700 1 -- 172.21.2.112:0/284590303 learned_addr learned my addr 172.21.2.112:0/284590303 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:42:28.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:28.541 7f62dcefe700 1 --2- 172.21.2.112:0/284590303 >> v2:172.21.2.120:3300/0 conn(0x7f62d810aff0 0x7f62d80cf750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:28.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:28.541 7f62d6ffd700 1 -- 172.21.2.112:0/284590303 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1057771492 0 0) 0x7f62c8000ce0 con 0x7f62d80d1870 2021-02-10T11:42:28.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:28.541 7f62d6ffd700 1 -- 172.21.2.112:0/284590303 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62c0002fe0 con 0x7f62d80d1870 2021-02-10T11:42:28.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:28.542 7f62dcefe700 1 -- 172.21.2.112:0/284590303 >> v1:172.21.2.120:6789/0 conn(0x7f62d80d1c70 legacy=0x7f62d80cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:28.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:28.542 7f62dcefe700 1 -- 172.21.2.112:0/284590303 >> v1:172.21.2.112:6789/0 conn(0x7f62d80d1870 legacy=0x7f62d81b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:28.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:28.542 7f62dcefe700 1 -- 172.21.2.112:0/284590303 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62d810b3f0 con 0x7f62d810aff0 2021-02-10T11:42:28.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:28.542 7f62dcefe700 1 --2- 172.21.2.112:0/284590303 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f62d810aff0 0x7f62d80cf750 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 rx=0x7f62d0006e50 tx=0x7f62d0009e70).ready entity=mon.1 client_cookie=f4da659a0922f9cf server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:28.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:28.542 7f62d6ffd700 1 -- 172.21.2.112:0/284590303 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f62d000b510 con 0x7f62d810aff0 2021-02-10T11:42:28.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:28.542 7f62d6ffd700 1 -- 172.21.2.112:0/284590303 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62d0006790 con 0x7f62d810aff0 2021-02-10T11:42:28.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:28.542 7f62d6ffd700 1 -- 172.21.2.112:0/284590303 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f62d000b940 con 0x7f62d810aff0 2021-02-10T11:42:28.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:28.542 7f62de8e1700 1 -- 172.21.2.112:0/284590303 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f62d810aff0 msgr2=0x7f62d80cf750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:28.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:28.542 7f62de8e1700 1 --2- 172.21.2.112:0/284590303 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f62d810aff0 0x7f62d80cf750 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 rx=0x7f62d0006e50 tx=0x7f62d0009e70).stop 2021-02-10T11:42:28.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:28.543 7f62de8e1700 1 -- 172.21.2.112:0/284590303 shutdown_connections 2021-02-10T11:42:28.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:28.543 7f62de8e1700 1 --2- 172.21.2.112:0/284590303 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f62d810aff0 0x7f62d80cf750 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:28.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:28.543 7f62de8e1700 1 -- 172.21.2.112:0/284590303 >> 172.21.2.112:0/284590303 conn(0x7f62d80ce420 msgr2=0x7f62d80d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:28.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:28.543 7f62de8e1700 1 -- 172.21.2.112:0/284590303 shutdown_connections 2021-02-10T11:42:28.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:28.543 7f62de8e1700 1 -- 172.21.2.112:0/284590303 wait complete. 2021-02-10T11:42:28.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:28.557 7f62de8e1700 1 Processor -- start 2021-02-10T11:42:28.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:28.557 7f62de8e1700 1 -- start start 2021-02-10T11:42:28.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:28.558 7f62de8e1700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f62d80c3820 0x7f62d8084cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:28.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:28.558 7f62de8e1700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f62d80a0290 0x7f62d80852d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:28.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:28.558 7f62de8e1700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f62d8086270 0x7f62d8086670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:28.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:28.558 7f62de8e1700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62d80c0700 con 0x7f62d80a0290 2021-02-10T11:42:28.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:28.558 7f62de8e1700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62d80858f0 con 0x7f62d8086270 2021-02-10T11:42:28.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:28.558 7f62de8e1700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62d8085b10 con 0x7f62d80c3820 2021-02-10T11:42:28.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:28.558 7f62d77fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f62d80a0290 0x7f62d80852d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:28.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:28.558 7f62d77fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f62d80a0290 0x7f62d80852d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:56336/0 (socket says 172.21.2.112:56336) 2021-02-10T11:42:28.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:28.558 7f62d77fe700 1 -- 172.21.2.112:0/2224457870 learned_addr learned my addr 172.21.2.112:0/2224457870 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:42:28.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:28.558 7f62d7fff700 1 --2- 172.21.2.112:0/2224457870 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f62d80c3820 0x7f62d8084cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:28.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:28.558 7f62dcefe700 1 --2- 172.21.2.112:0/2224457870 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f62d8086270 0x7f62d8086670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:28.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:28.558 7f62d77fe700 1 -- 172.21.2.112:0/2224457870 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f62d80c3820 msgr2=0x7f62d8084cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:28.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:28.558 7f62d77fe700 1 --2- 172.21.2.112:0/2224457870 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f62d80c3820 0x7f62d8084cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:28.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:28.558 7f62d77fe700 1 -- 172.21.2.112:0/2224457870 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f62d8086270 msgr2=0x7f62d8086670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:28.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:28.558 7f62d77fe700 1 --2- 172.21.2.112:0/2224457870 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f62d8086270 0x7f62d8086670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:28.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:28.559 7f62d77fe700 1 -- 172.21.2.112:0/2224457870 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62d8086c90 con 0x7f62d80a0290 2021-02-10T11:42:28.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:28.559 7f62d77fe700 1 --2- 172.21.2.112:0/2224457870 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f62d80a0290 0x7f62d80852d0 secure :-1 s=READY pgs=2349 cs=0 l=1 rev1=1 rx=0x7f62cc006ed0 tx=0x7f62cc00b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:28.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:28.559 7f62d4ff9700 1 -- 172.21.2.112:0/2224457870 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f62cc006020 con 0x7f62d80a0290 2021-02-10T11:42:28.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:28.559 7f62d4ff9700 1 -- 172.21.2.112:0/2224457870 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62cc006360 con 0x7f62d80a0290 2021-02-10T11:42:28.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:28.559 7f62d4ff9700 1 -- 172.21.2.112:0/2224457870 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f62cc005730 con 0x7f62d80a0290 2021-02-10T11:42:28.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:28.559 7f62de8e1700 1 -- 172.21.2.112:0/2224457870 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62d8086c90 con 0x7f62d80a0290 2021-02-10T11:42:28.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:28.559 7f62de8e1700 1 -- 172.21.2.112:0/2224457870 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62d8086c90 con 0x7f62d80a0290 2021-02-10T11:42:28.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:28.560 7f62d4ff9700 1 -- 172.21.2.112:0/2224457870 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f62cc0054b0 con 0x7f62d80a0290 2021-02-10T11:42:28.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:28.560 7f62d4ff9700 1 --2- 172.21.2.112:0/2224457870 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f62b80186c0 0x7f62b801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:28.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:28.560 7f62d4ff9700 1 -- 172.21.2.112:0/2224457870 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f62cc0325b0 con 0x7f62d80a0290 2021-02-10T11:42:28.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:28.560 7f62d7fff700 1 --2- 172.21.2.112:0/2224457870 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f62b80186c0 0x7f62b801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:28.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:28.560 7f62d7fff700 1 --2- 172.21.2.112:0/2224457870 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f62b80186c0 0x7f62b801abc0 crc :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:28.589 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:28.591 7f62de8e1700 1 -- 172.21.2.112:0/2224457870 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62d8082a70 con 0x7f62d80a0290 2021-02-10T11:42:28.592 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:28.593 7f62d4ff9700 1 -- 172.21.2.112:0/2224457870 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f62cc030d60 con 0x7f62d80a0290 2021-02-10T11:42:28.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:28.721 7f62de8e1700 1 -- 172.21.2.112:0/2224457870 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f62d8082a70 con 0x7f62d80a0290 2021-02-10T11:42:28.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:28.721 7f62d4ff9700 1 -- 172.21.2.112:0/2224457870 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v125) v1 ==== 100+0+10 (secure 0 0 0) 0x7f62cc0054b0 con 0x7f62d80a0290 2021-02-10T11:42:28.721 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:42:28.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:28.723 7f62de8e1700 1 -- 172.21.2.112:0/2224457870 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f62b80186c0 msgr2=0x7f62b801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:28.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:28.723 7f62de8e1700 1 --2- 172.21.2.112:0/2224457870 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f62b80186c0 0x7f62b801abc0 crc :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:28.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:28.723 7f62de8e1700 1 -- 172.21.2.112:0/2224457870 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f62d80a0290 msgr2=0x7f62d80852d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:28.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:28.723 7f62de8e1700 1 --2- 172.21.2.112:0/2224457870 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f62d80a0290 0x7f62d80852d0 secure :-1 s=READY pgs=2349 cs=0 l=1 rev1=1 rx=0x7f62cc006ed0 tx=0x7f62cc00b960).stop 2021-02-10T11:42:28.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:28.723 7f62de8e1700 1 -- 172.21.2.112:0/2224457870 shutdown_connections 2021-02-10T11:42:28.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:28.723 7f62de8e1700 1 --2- 172.21.2.112:0/2224457870 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f62b80186c0 0x7f62b801abc0 unknown :-1 s=CLOSED pgs=1075 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:28.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:28.723 7f62de8e1700 1 --2- 172.21.2.112:0/2224457870 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f62d8086270 0x7f62d8086670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:28.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:28.723 7f62de8e1700 1 --2- 172.21.2.112:0/2224457870 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f62d80a0290 0x7f62d80852d0 unknown :-1 s=CLOSED pgs=2349 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:28.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:28.723 7f62de8e1700 1 --2- 172.21.2.112:0/2224457870 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f62d80c3820 0x7f62d8084cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:28.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:28.723 7f62de8e1700 1 -- 172.21.2.112:0/2224457870 >> 172.21.2.112:0/2224457870 conn(0x7f62d80872c0 msgr2=0x7f62d8095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:28.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:28.724 7f62de8e1700 1 -- 172.21.2.112:0/2224457870 shutdown_connections 2021-02-10T11:42:28.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:28.724 7f62de8e1700 1 -- 172.21.2.112:0/2224457870 wait complete. 2021-02-10T11:42:28.731 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:42:28.732 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:42:29.160 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:29.143 7fcae529c700 1 Processor -- start 2021-02-10T11:42:29.160 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:29.144 7fcae529c700 1 -- start start 2021-02-10T11:42:29.160 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:29.144 7fcae529c700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fcae00d0340 0x7fcae00ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:29.160 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:29.144 7fcae529c700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcae00f0b10 con 0x7fcae00d1870 2021-02-10T11:42:29.160 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:29.144 7fcae529c700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcae01b0be0 con 0x7fcae00d1c70 2021-02-10T11:42:29.160 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:29.144 7fcae529c700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcae00e5ed0 con 0x7fcae00d0340 2021-02-10T11:42:29.160 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:29.144 7fcadf07d700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fcae00d1870 0x7fcae01b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:55804/0 (socket says 172.21.2.112:55804) 2021-02-10T11:42:29.160 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:29.144 7fcadf07d700 1 -- 172.21.2.112:0/1020468949 learned_addr learned my addr 172.21.2.112:0/1020468949 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:42:29.160 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:29.145 7fcade87c700 1 --2- 172.21.2.112:0/1020468949 >> v2:172.21.2.120:3300/0 conn(0x7fcae00d0340 0x7fcae00ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:29.160 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:29.145 7fcade07b700 1 -- 172.21.2.112:0/1020468949 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1736547312 0 0) 0x7fcad00010b0 con 0x7fcae00d1870 2021-02-10T11:42:29.161 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:29.145 7fcade07b700 1 -- 172.21.2.112:0/1020468949 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcac8002fe0 con 0x7fcae00d1870 2021-02-10T11:42:29.161 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:29.145 7fcade07b700 1 -- 172.21.2.112:0/1020468949 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 767519000 0 0) 0x7fcad0001010 con 0x7fcae00d1870 2021-02-10T11:42:29.161 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:29.145 7fcade07b700 1 -- 172.21.2.112:0/1020468949 >> v2:172.21.2.120:3300/0 conn(0x7fcae00d0340 msgr2=0x7fcae00ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:29.161 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:29.145 7fcade07b700 1 --2- 172.21.2.112:0/1020468949 >> v2:172.21.2.120:3300/0 conn(0x7fcae00d0340 0x7fcae00ef640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:29.161 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:29.145 7fcade07b700 1 -- 172.21.2.112:0/1020468949 >> v1:172.21.2.120:6789/0 conn(0x7fcae00d1c70 legacy=0x7fcae00efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:29.161 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:29.145 7fcade07b700 1 -- 172.21.2.112:0/1020468949 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcae00f0450 con 0x7fcae00d1870 2021-02-10T11:42:29.161 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:29.145 7fcade07b700 1 -- 172.21.2.112:0/1020468949 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7fcad0001530 con 0x7fcae00d1870 2021-02-10T11:42:29.161 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:29.145 7fcade07b700 1 -- 172.21.2.112:0/1020468949 >> v1:172.21.2.112:6789/0 conn(0x7fcae00d1870 legacy=0x7fcae01b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:29.161 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:29.145 7fcade07b700 1 --2- 172.21.2.112:0/1020468949 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcac8003aa0 0x7fcac8005ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:29.161 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:29.145 7fcade07b700 1 --2- 172.21.2.112:0/1020468949 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcac8006770 0x7fcac8008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:29.161 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:29.145 7fcade07b700 1 --2- 172.21.2.112:0/1020468949 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcac8009590 0x7fcac800bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:29.161 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:29.145 7fcade07b700 1 -- 172.21.2.112:0/1020468949 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcac800c300 con 0x7fcac8003aa0 2021-02-10T11:42:29.161 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:29.145 7fcade07b700 1 -- 172.21.2.112:0/1020468949 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcac800c520 con 0x7fcac8009590 2021-02-10T11:42:29.161 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:29.145 7fcade07b700 1 -- 172.21.2.112:0/1020468949 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcac800c740 con 0x7fcac8006770 2021-02-10T11:42:29.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:29.145 7fcadf07d700 1 --2- 172.21.2.112:0/1020468949 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcac8003aa0 0x7fcac8005ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:29.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:29.145 7fcade87c700 1 --2- 172.21.2.112:0/1020468949 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcac8006770 0x7fcac8008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:29.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:29.146 7fcadf87e700 1 --2- 172.21.2.112:0/1020468949 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcac8009590 0x7fcac800bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:29.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:29.146 7fcadf07d700 1 -- 172.21.2.112:0/1020468949 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcac8006770 msgr2=0x7fcac8008c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:29.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:29.146 7fcadf07d700 1 --2- 172.21.2.112:0/1020468949 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcac8006770 0x7fcac8008c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:29.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:29.146 7fcadf07d700 1 -- 172.21.2.112:0/1020468949 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcac8009590 msgr2=0x7fcac800bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:29.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:29.146 7fcadf07d700 1 --2- 172.21.2.112:0/1020468949 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcac8009590 0x7fcac800bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:29.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:29.146 7fcadf07d700 1 -- 172.21.2.112:0/1020468949 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcac8050980 con 0x7fcac8003aa0 2021-02-10T11:42:29.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:29.146 7fcadf87e700 1 -- 172.21.2.112:0/1020468949 reap_dead start 2021-02-10T11:42:29.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:29.146 7fcadf07d700 1 --2- 172.21.2.112:0/1020468949 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcac8003aa0 0x7fcac8005ed0 secure :-1 s=READY pgs=2351 cs=0 l=1 rev1=1 rx=0x7fcad00017d0 tx=0x7fcad0007d30).ready entity=mon.0 client_cookie=40830d6135bbc9cf server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:29.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:29.146 7fcade07b700 1 -- 172.21.2.112:0/1020468949 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcad00013a0 con 0x7fcac8003aa0 2021-02-10T11:42:29.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:29.146 7fcade07b700 1 -- 172.21.2.112:0/1020468949 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcad000a310 con 0x7fcac8003aa0 2021-02-10T11:42:29.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:29.146 7fcade07b700 1 -- 172.21.2.112:0/1020468949 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcad000a7f0 con 0x7fcac8003aa0 2021-02-10T11:42:29.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:29.146 7fcae529c700 1 -- 172.21.2.112:0/1020468949 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcac8003aa0 msgr2=0x7fcac8005ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:29.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:29.146 7fcae529c700 1 --2- 172.21.2.112:0/1020468949 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcac8003aa0 0x7fcac8005ed0 secure :-1 s=READY pgs=2351 cs=0 l=1 rev1=1 rx=0x7fcad00017d0 tx=0x7fcad0007d30).stop 2021-02-10T11:42:29.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:29.146 7fcae529c700 1 -- 172.21.2.112:0/1020468949 shutdown_connections 2021-02-10T11:42:29.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:29.146 7fcae529c700 1 --2- 172.21.2.112:0/1020468949 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcac8003aa0 0x7fcac8005ed0 unknown :-1 s=CLOSED pgs=2351 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:29.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:29.146 7fcae529c700 1 -- 172.21.2.112:0/1020468949 >> 172.21.2.112:0/1020468949 conn(0x7fcae00ce450 msgr2=0x7fcae00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:29.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:29.146 7fcae529c700 1 -- 172.21.2.112:0/1020468949 shutdown_connections 2021-02-10T11:42:29.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:29.146 7fcae529c700 1 -- 172.21.2.112:0/1020468949 wait complete. 2021-02-10T11:42:29.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:29.161 7fcae529c700 1 Processor -- start 2021-02-10T11:42:29.164 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:29.161 7fcae529c700 1 -- start start 2021-02-10T11:42:29.165 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:29.161 7fcae529c700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcae00beeb0 0x7fcae00bf2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:29.165 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:29.161 7fcae529c700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcae018ef70 0x7fcae018f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:29.165 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:29.161 7fcae529c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcae018f990 0x7fcae00c01d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:29.165 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:29.161 7fcae529c700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcae00bfc50 con 0x7fcae018f990 2021-02-10T11:42:29.165 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:29.161 7fcae529c700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcae018fd90 con 0x7fcae00beeb0 2021-02-10T11:42:29.165 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:29.161 7fcae529c700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcae00bf8d0 con 0x7fcae018ef70 2021-02-10T11:42:29.165 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:29.162 7fcadf87e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcae018f990 0x7fcae00c01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:29.165 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:29.162 7fcadf87e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcae018f990 0x7fcae00c01d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:56356/0 (socket says 172.21.2.112:56356) 2021-02-10T11:42:29.165 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:29.162 7fcadf87e700 1 -- 172.21.2.112:0/951590186 learned_addr learned my addr 172.21.2.112:0/951590186 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:42:29.165 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:29.162 7fcadf87e700 1 -- 172.21.2.112:0/951590186 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcae018ef70 msgr2=0x7fcae018f370 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:42:29.165 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:29.162 7fcadf87e700 1 --2- 172.21.2.112:0/951590186 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcae018ef70 0x7fcae018f370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:29.165 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:29.162 7fcadf87e700 1 -- 172.21.2.112:0/951590186 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcae00beeb0 msgr2=0x7fcae00bf2b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:42:29.165 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:29.162 7fcadf87e700 1 --2- 172.21.2.112:0/951590186 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcae00beeb0 0x7fcae00bf2b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:29.166 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:29.162 7fcadf87e700 1 -- 172.21.2.112:0/951590186 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcae00c07f0 con 0x7fcae018f990 2021-02-10T11:42:29.166 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:29.162 7fcadf87e700 1 --2- 172.21.2.112:0/951590186 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcae018f990 0x7fcae00c01d0 secure :-1 s=READY pgs=2352 cs=0 l=1 rev1=1 rx=0x7fcad80099a0 tx=0x7fcad800bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:29.166 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:29.162 7fcac7fff700 1 -- 172.21.2.112:0/951590186 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcad8006ac0 con 0x7fcae018f990 2021-02-10T11:42:29.166 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:29.162 7fcac7fff700 1 -- 172.21.2.112:0/951590186 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcad800c380 con 0x7fcae018f990 2021-02-10T11:42:29.166 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:29.162 7fcac7fff700 1 -- 172.21.2.112:0/951590186 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcad800c810 con 0x7fcae018f990 2021-02-10T11:42:29.166 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:29.162 7fcae529c700 1 -- 172.21.2.112:0/951590186 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcae00c07f0 con 0x7fcae018f990 2021-02-10T11:42:29.166 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:29.162 7fcae529c700 1 -- 172.21.2.112:0/951590186 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcae00c0d80 con 0x7fcae018f990 2021-02-10T11:42:29.166 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:29.163 7fcac7fff700 1 -- 172.21.2.112:0/951590186 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fcad8001010 con 0x7fcae018f990 2021-02-10T11:42:29.166 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:29.163 7fcac7fff700 1 --2- 172.21.2.112:0/951590186 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcac00186c0 0x7fcac001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:29.166 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:29.163 7fcac7fff700 1 -- 172.21.2.112:0/951590186 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fcad80328e0 con 0x7fcae018f990 2021-02-10T11:42:29.166 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:29.163 7fcadf07d700 1 --2- 172.21.2.112:0/951590186 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcac00186c0 0x7fcac001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:29.166 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:29.164 7fcadf07d700 1 --2- 172.21.2.112:0/951590186 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcac00186c0 0x7fcac001abc0 crc :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:29.193 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:29.194 7fcae529c700 1 -- 172.21.2.112:0/951590186 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcae00c1550 con 0x7fcae018f990 2021-02-10T11:42:29.195 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:29.196 7fcac7fff700 1 -- 172.21.2.112:0/951590186 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fcad800be30 con 0x7fcae018f990 2021-02-10T11:42:29.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:29.325 7fcae529c700 1 -- 172.21.2.112:0/951590186 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcae00c1550 con 0x7fcae018f990 2021-02-10T11:42:29.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:29.325 7fcac7fff700 1 -- 172.21.2.112:0/951590186 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+2108 (secure 0 0 0) 0x7fcad8001ee0 con 0x7fcae018f990 2021-02-10T11:42:29.325 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:42:29.325 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":105,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11040,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/1814390924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":1814390924},{"type":"v1","addr":"172.21.2.112:6833","nonce":1814390924}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":101}],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:42:19.351348","modified":"2021-02-10 11:42:21.357585","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8203},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8203":{"gid":8203,"name":"b","rank":0,"incarnation":102,"state":"up:active","state_seq":3,"addr":"172.21.2.120:6835/2761661070","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":2761661070},{"type":"v1","addr":"172.21.2.120:6835","nonce":2761661070}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":13}]} 2021-02-10T11:42:29.325 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 105 2021-02-10T11:42:29.326 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:29.327 7fcae529c700 1 -- 172.21.2.112:0/951590186 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcac00186c0 msgr2=0x7fcac001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:29.326 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:29.327 7fcae529c700 1 --2- 172.21.2.112:0/951590186 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcac00186c0 0x7fcac001abc0 crc :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:29.326 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:29.327 7fcae529c700 1 -- 172.21.2.112:0/951590186 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcae018f990 msgr2=0x7fcae00c01d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:29.326 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:29.327 7fcae529c700 1 --2- 172.21.2.112:0/951590186 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcae018f990 0x7fcae00c01d0 secure :-1 s=READY pgs=2352 cs=0 l=1 rev1=1 rx=0x7fcad80099a0 tx=0x7fcad800bcf0).stop 2021-02-10T11:42:29.326 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:29.327 7fcae529c700 1 -- 172.21.2.112:0/951590186 shutdown_connections 2021-02-10T11:42:29.326 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:29.327 7fcae529c700 1 --2- 172.21.2.112:0/951590186 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcac00186c0 0x7fcac001abc0 unknown :-1 s=CLOSED pgs=1076 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:29.327 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:29.327 7fcae529c700 1 --2- 172.21.2.112:0/951590186 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcae018f990 0x7fcae00c01d0 unknown :-1 s=CLOSED pgs=2352 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:29.327 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:29.327 7fcae529c700 1 --2- 172.21.2.112:0/951590186 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcae018ef70 0x7fcae018f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:29.327 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:29.327 7fcae529c700 1 --2- 172.21.2.112:0/951590186 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcae00beeb0 0x7fcae00bf2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:29.327 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:29.327 7fcae529c700 1 -- 172.21.2.112:0/951590186 >> 172.21.2.112:0/951590186 conn(0x7fcae00c2130 msgr2=0x7fcae009e3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:29.327 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:29.327 7fcae529c700 1 -- 172.21.2.112:0/951590186 shutdown_connections 2021-02-10T11:42:29.327 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:29.327 7fcae529c700 1 -- 172.21.2.112:0/951590186 wait complete. 2021-02-10T11:42:29.537 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 105, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:42:19.351348', 'modified': '2021-02-10 11:42:21.357585', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8203}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8203': {'gid': 8203, 'name': 'b', 'rank': 0, 'incarnation': 102, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.120:6835/2761661070', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.120:6834', 'nonce': 2761661070}, {'type': 'v1', 'addr': '172.21.2.120:6835', 'nonce': 2761661070}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [27], 'metadata_pool': 26, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:42:29.537 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:42:29.537 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:42:29.936 INFO:teuthology.orchestra.run.gibba020.stdout:{ 2021-02-10T11:42:29.936 INFO:teuthology.orchestra.run.gibba020.stdout: "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", 2021-02-10T11:42:29.936 INFO:teuthology.orchestra.run.gibba020.stdout: "whoami": 0, 2021-02-10T11:42:29.936 INFO:teuthology.orchestra.run.gibba020.stdout: "id": 8203, 2021-02-10T11:42:29.936 INFO:teuthology.orchestra.run.gibba020.stdout: "want_state": "up:active", 2021-02-10T11:42:29.937 INFO:teuthology.orchestra.run.gibba020.stdout: "state": "up:active", 2021-02-10T11:42:29.937 INFO:teuthology.orchestra.run.gibba020.stdout: "rank_uptime": 10.584247252999999, 2021-02-10T11:42:29.937 INFO:teuthology.orchestra.run.gibba020.stdout: "mdsmap_epoch": 105, 2021-02-10T11:42:29.937 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch": 125, 2021-02-10T11:42:29.937 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch_barrier": 125, 2021-02-10T11:42:29.937 INFO:teuthology.orchestra.run.gibba020.stdout: "uptime": 16.215315624999999 2021-02-10T11:42:29.937 INFO:teuthology.orchestra.run.gibba020.stdout:} 2021-02-10T11:42:29.940 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", "whoami": 0, "id": 8203, "want_state": "up:active", "state": "up:active", "rank_uptime": 10.584247252999999, "mdsmap_epoch": 105, "osdmap_epoch": 125, "osdmap_epoch_barrier": 125, "uptime": 16.215315624999999 } 2021-02-10T11:42:29.940 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:42:29.940 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2021-02-10T11:42:29.940 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@gibba012.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2021-02-10T11:42:29.940 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:42:29.941 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:42:30.276 DEBUG:teuthology.orchestra.run.gibba012:> sudo modprobe fuse 2021-02-10T11:42:30.621 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:42:30.621 DEBUG:teuthology.orchestra.run.gibba012:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:42:30.965 INFO:teuthology.orchestra.run.gibba012.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:42:30.966 INFO:teuthology.orchestra.run.gibba012.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:42:30.966 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:42:30.967 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:42:30.967 DEBUG:teuthology.orchestra.run.gibba012:> ls /sys/fs/fuse/connections 2021-02-10T11:42:31.301 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2021-02-10T11:42:31.301 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:42:31.304 DEBUG:teuthology.orchestra.run.gibba012:> sudo modprobe fuse 2021-02-10T11:42:31.649 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:42:31.649 DEBUG:teuthology.orchestra.run.gibba012:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:42:31.716 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba012.stderr:ceph-fuse[165630]: starting ceph client 2021-02-10T11:42:31.717 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba012.stderr:2021-02-10 11:42:31.718 7ff24822af80 -1 init, newargv = 0x55f85c0a71a0 newargc=9 2021-02-10T11:42:31.722 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba012.stderr:ceph-fuse[165630]: starting fuse 2021-02-10T11:42:31.992 INFO:teuthology.orchestra.run.gibba012.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:42:31.993 INFO:teuthology.orchestra.run.gibba012.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:42:31.993 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:42:31.994 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:42:31.994 DEBUG:teuthology.orchestra.run.gibba012:> ls /sys/fs/fuse/connections 2021-02-10T11:42:32.333 INFO:teuthology.orchestra.run.gibba012.stdout:42 2021-02-10T11:42:32.333 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2021-02-10T11:42:32.333 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:42:32.334 DEBUG:teuthology.orchestra.run.gibba012:> sudo python3 -c ' 2021-02-10T11:42:32.334 DEBUG:teuthology.orchestra.run.gibba012:> import glob 2021-02-10T11:42:32.334 DEBUG:teuthology.orchestra.run.gibba012:> import re 2021-02-10T11:42:32.334 DEBUG:teuthology.orchestra.run.gibba012:> import os 2021-02-10T11:42:32.334 DEBUG:teuthology.orchestra.run.gibba012:> import subprocess 2021-02-10T11:42:32.334 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:42:32.334 DEBUG:teuthology.orchestra.run.gibba012:> def find_socket(client_name): 2021-02-10T11:42:32.334 DEBUG:teuthology.orchestra.run.gibba012:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2021-02-10T11:42:32.334 DEBUG:teuthology.orchestra.run.gibba012:> files = glob.glob(asok_path) 2021-02-10T11:42:32.334 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:42:32.334 DEBUG:teuthology.orchestra.run.gibba012:> # Given a non-glob path, it better be there 2021-02-10T11:42:32.334 DEBUG:teuthology.orchestra.run.gibba012:> if "*" not in asok_path: 2021-02-10T11:42:32.334 DEBUG:teuthology.orchestra.run.gibba012:> assert(len(files) == 1) 2021-02-10T11:42:32.335 DEBUG:teuthology.orchestra.run.gibba012:> return files[0] 2021-02-10T11:42:32.335 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:42:32.335 DEBUG:teuthology.orchestra.run.gibba012:> for f in files: 2021-02-10T11:42:32.335 DEBUG:teuthology.orchestra.run.gibba012:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:42:32.335 DEBUG:teuthology.orchestra.run.gibba012:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:42:32.335 DEBUG:teuthology.orchestra.run.gibba012:> return f 2021-02-10T11:42:32.335 DEBUG:teuthology.orchestra.run.gibba012:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:42:32.335 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:42:32.335 DEBUG:teuthology.orchestra.run.gibba012:> print(find_socket("client.0")) 2021-02-10T11:42:32.335 DEBUG:teuthology.orchestra.run.gibba012:> ' 2021-02-10T11:42:32.699 INFO:teuthology.orchestra.run.gibba012.stdout:/var/run/ceph/ceph-client.0.165630.asok 2021-02-10T11:42:32.701 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.165630.asok 2021-02-10T11:42:32.701 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:42:32.702 DEBUG:teuthology.orchestra.run.gibba012:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.165630.asok status 2021-02-10T11:42:33.093 INFO:teuthology.orchestra.run.gibba012.stdout:{ 2021-02-10T11:42:33.093 INFO:teuthology.orchestra.run.gibba012.stdout: "metadata": { 2021-02-10T11:42:33.093 INFO:teuthology.orchestra.run.gibba012.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:42:33.093 INFO:teuthology.orchestra.run.gibba012.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:42:33.094 INFO:teuthology.orchestra.run.gibba012.stdout: "entity_id": "0", 2021-02-10T11:42:33.094 INFO:teuthology.orchestra.run.gibba012.stdout: "hostname": "gibba012", 2021-02-10T11:42:33.094 INFO:teuthology.orchestra.run.gibba012.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2021-02-10T11:42:33.094 INFO:teuthology.orchestra.run.gibba012.stdout: "pid": "165630", 2021-02-10T11:42:33.094 INFO:teuthology.orchestra.run.gibba012.stdout: "root": "/" 2021-02-10T11:42:33.094 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:33.094 INFO:teuthology.orchestra.run.gibba012.stdout: "dentry_count": 0, 2021-02-10T11:42:33.094 INFO:teuthology.orchestra.run.gibba012.stdout: "dentry_pinned_count": 0, 2021-02-10T11:42:33.094 INFO:teuthology.orchestra.run.gibba012.stdout: "id": 11190, 2021-02-10T11:42:33.094 INFO:teuthology.orchestra.run.gibba012.stdout: "inst": { 2021-02-10T11:42:33.094 INFO:teuthology.orchestra.run.gibba012.stdout: "name": { 2021-02-10T11:42:33.094 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "client", 2021-02-10T11:42:33.095 INFO:teuthology.orchestra.run.gibba012.stdout: "num": 11190 2021-02-10T11:42:33.095 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:33.095 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": { 2021-02-10T11:42:33.095 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:42:33.095 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:0", 2021-02-10T11:42:33.095 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 3923734033 2021-02-10T11:42:33.095 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:42:33.095 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:33.095 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": { 2021-02-10T11:42:33.095 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:42:33.095 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:0", 2021-02-10T11:42:33.095 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 3923734033 2021-02-10T11:42:33.095 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:33.096 INFO:teuthology.orchestra.run.gibba012.stdout: "inst_str": "client.11190 172.21.2.112:0/3923734033", 2021-02-10T11:42:33.096 INFO:teuthology.orchestra.run.gibba012.stdout: "addr_str": "172.21.2.112:0/3923734033", 2021-02-10T11:42:33.096 INFO:teuthology.orchestra.run.gibba012.stdout: "inode_count": 1, 2021-02-10T11:42:33.096 INFO:teuthology.orchestra.run.gibba012.stdout: "mds_epoch": 105, 2021-02-10T11:42:33.096 INFO:teuthology.orchestra.run.gibba012.stdout: "osd_epoch": 125, 2021-02-10T11:42:33.096 INFO:teuthology.orchestra.run.gibba012.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:42:33.096 INFO:teuthology.orchestra.run.gibba012.stdout: "blacklisted": false 2021-02-10T11:42:33.096 INFO:teuthology.orchestra.run.gibba012.stdout:} 2021-02-10T11:42:33.097 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:42:33.097 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:42:33.097 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:42:33.433 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:42:33.433 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:42:33.433 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:42:33.433 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:42:33.779 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:42:34.209 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.191 7fca47905700 1 Processor -- start 2021-02-10T11:42:34.209 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.191 7fca47905700 1 -- start start 2021-02-10T11:42:34.209 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.191 7fca47905700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fca400d1870 0x7fca401b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:34.209 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.191 7fca47905700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fca401b0a10 0x7fca400cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:34.209 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.191 7fca47905700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fca400d1c70 0x7fca400cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:34.209 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.191 7fca47905700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca400e5ed0 con 0x7fca400d1870 2021-02-10T11:42:34.209 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.191 7fca47905700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca400f0b10 con 0x7fca401b0a10 2021-02-10T11:42:34.209 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.191 7fca47905700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca400cff70 con 0x7fca400d1c70 2021-02-10T11:42:34.209 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.192 7fca45721700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fca400d1870 0x7fca401b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:34.209 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.192 7fca45721700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fca400d1870 0x7fca401b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:56378/0 (socket says 172.21.2.112:56378) 2021-02-10T11:42:34.210 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.192 7fca45721700 1 -- 172.21.2.112:0/321987996 learned_addr learned my addr 172.21.2.112:0/321987996 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:42:34.210 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.192 7fca45721700 1 -- 172.21.2.112:0/321987996 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fca400d1c70 msgr2=0x7fca400cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:34.210 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.192 7fca45721700 1 --2- 172.21.2.112:0/321987996 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fca400d1c70 0x7fca400cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:34.210 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.192 7fca45721700 1 -- 172.21.2.112:0/321987996 >> v2:172.21.2.120:3300/0 conn(0x7fca401b0a10 msgr2=0x7fca400cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:34.210 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.192 7fca45721700 1 --2- 172.21.2.112:0/321987996 >> v2:172.21.2.120:3300/0 conn(0x7fca401b0a10 0x7fca400cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:34.210 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.192 7fca45721700 1 -- 172.21.2.112:0/321987996 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca400d0150 con 0x7fca400d1870 2021-02-10T11:42:34.210 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.192 7fca45721700 1 --2- 172.21.2.112:0/321987996 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fca400d1870 0x7fca401b03f0 secure :-1 s=READY pgs=2356 cs=0 l=1 rev1=1 rx=0x7fca3c006700 tx=0x7fca3c009b80).ready entity=mon.0 client_cookie=ac7e84321a3a8a94 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:34.210 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.192 7fca33fff700 1 -- 172.21.2.112:0/321987996 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fca3c00c0c0 con 0x7fca400d1870 2021-02-10T11:42:34.210 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.193 7fca33fff700 1 -- 172.21.2.112:0/321987996 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca3c00c340 con 0x7fca400d1870 2021-02-10T11:42:34.210 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.193 7fca33fff700 1 -- 172.21.2.112:0/321987996 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fca3c00c860 con 0x7fca400d1870 2021-02-10T11:42:34.210 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.193 7fca47905700 1 -- 172.21.2.112:0/321987996 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fca400d1870 msgr2=0x7fca401b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:34.210 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.193 7fca47905700 1 --2- 172.21.2.112:0/321987996 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fca400d1870 0x7fca401b03f0 secure :-1 s=READY pgs=2356 cs=0 l=1 rev1=1 rx=0x7fca3c006700 tx=0x7fca3c009b80).stop 2021-02-10T11:42:34.211 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.193 7fca47905700 1 -- 172.21.2.112:0/321987996 shutdown_connections 2021-02-10T11:42:34.211 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.193 7fca47905700 1 --2- 172.21.2.112:0/321987996 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fca400d1c70 0x7fca400cf950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:34.211 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.193 7fca47905700 1 --2- 172.21.2.112:0/321987996 >> v2:172.21.2.120:3300/0 conn(0x7fca401b0a10 0x7fca400cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:34.211 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.193 7fca47905700 1 --2- 172.21.2.112:0/321987996 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fca400d1870 0x7fca401b03f0 unknown :-1 s=CLOSED pgs=2356 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:34.211 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.193 7fca47905700 1 -- 172.21.2.112:0/321987996 >> 172.21.2.112:0/321987996 conn(0x7fca400ce450 msgr2=0x7fca400d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:34.211 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.193 7fca47905700 1 -- 172.21.2.112:0/321987996 shutdown_connections 2021-02-10T11:42:34.211 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.193 7fca47905700 1 -- 172.21.2.112:0/321987996 wait complete. 2021-02-10T11:42:34.211 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.208 7fca47905700 1 Processor -- start 2021-02-10T11:42:34.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.208 7fca47905700 1 -- start start 2021-02-10T11:42:34.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.208 7fca47905700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fca401b8ad0 0x7fca40187e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:34.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.208 7fca47905700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fca401b8ed0 0x7fca400b1af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:34.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.208 7fca47905700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fca400b2110 0x7fca401b7a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:34.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.208 7fca47905700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca40188470 con 0x7fca400b2110 2021-02-10T11:42:34.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.208 7fca47905700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca4018be10 con 0x7fca401b8ed0 2021-02-10T11:42:34.213 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.208 7fca47905700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca4018c030 con 0x7fca401b8ad0 2021-02-10T11:42:34.213 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.209 7fca45f22700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fca400b2110 0x7fca401b7a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:34.213 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.209 7fca45f22700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fca400b2110 0x7fca401b7a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:56382/0 (socket says 172.21.2.112:56382) 2021-02-10T11:42:34.213 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.209 7fca45f22700 1 -- 172.21.2.112:0/1990462086 learned_addr learned my addr 172.21.2.112:0/1990462086 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:42:34.213 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.209 7fca45721700 1 --2- 172.21.2.112:0/1990462086 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fca401b8ad0 0x7fca40187e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:34.213 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.209 7fca44f20700 1 --2- 172.21.2.112:0/1990462086 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fca401b8ed0 0x7fca400b1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:34.213 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.209 7fca45f22700 1 -- 172.21.2.112:0/1990462086 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fca401b8ad0 msgr2=0x7fca40187e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:34.214 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.209 7fca45f22700 1 --2- 172.21.2.112:0/1990462086 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fca401b8ad0 0x7fca40187e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:34.214 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.209 7fca45f22700 1 -- 172.21.2.112:0/1990462086 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fca401b8ed0 msgr2=0x7fca400b1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:34.214 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.209 7fca45f22700 1 --2- 172.21.2.112:0/1990462086 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fca401b8ed0 0x7fca400b1af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:34.214 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.209 7fca45f22700 1 -- 172.21.2.112:0/1990462086 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca400b2510 con 0x7fca400b2110 2021-02-10T11:42:34.214 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.209 7fca45721700 1 --2- 172.21.2.112:0/1990462086 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fca401b8ad0 0x7fca40187e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:42:34.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.209 7fca45f22700 1 --2- 172.21.2.112:0/1990462086 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fca400b2110 0x7fca401b7a00 secure :-1 s=READY pgs=2357 cs=0 l=1 rev1=1 rx=0x7fca38009980 tx=0x7fca3800bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:34.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.209 7fca31ffb700 1 -- 172.21.2.112:0/1990462086 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fca38005a80 con 0x7fca400b2110 2021-02-10T11:42:34.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.209 7fca31ffb700 1 -- 172.21.2.112:0/1990462086 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca38005cc0 con 0x7fca400b2110 2021-02-10T11:42:34.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.209 7fca47905700 1 -- 172.21.2.112:0/1990462086 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca400b2510 con 0x7fca400b2110 2021-02-10T11:42:34.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.209 7fca31ffb700 1 -- 172.21.2.112:0/1990462086 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fca380062b0 con 0x7fca400b2110 2021-02-10T11:42:34.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.209 7fca47905700 1 -- 172.21.2.112:0/1990462086 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca400b2510 con 0x7fca400b2110 2021-02-10T11:42:34.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.210 7fca31ffb700 1 -- 172.21.2.112:0/1990462086 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fca38005870 con 0x7fca400b2110 2021-02-10T11:42:34.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.210 7fca31ffb700 1 --2- 172.21.2.112:0/1990462086 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fca20018730 0x7fca2001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:34.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.210 7fca31ffb700 1 -- 172.21.2.112:0/1990462086 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fca38032990 con 0x7fca400b2110 2021-02-10T11:42:34.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.210 7fca45721700 1 --2- 172.21.2.112:0/1990462086 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fca20018730 0x7fca2001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:34.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.211 7fca45721700 1 --2- 172.21.2.112:0/1990462086 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fca20018730 0x7fca2001ac10 crc :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:34.240 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.241 7fca47905700 1 -- 172.21.2.112:0/1990462086 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca400ebc00 con 0x7fca400b2110 2021-02-10T11:42:34.242 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.244 7fca31ffb700 1 -- 172.21.2.112:0/1990462086 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fca38005cc0 con 0x7fca400b2110 2021-02-10T11:42:34.367 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.368 7fca47905700 1 -- 172.21.2.112:0/1990462086 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fca400f0b10 con 0x7fca400b2110 2021-02-10T11:42:34.368 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.369 7fca31ffb700 1 -- 172.21.2.112:0/1990462086 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v125) v1 ==== 74+0+14007 (secure 0 0 0) 0x7fca38003020 con 0x7fca400b2110 2021-02-10T11:42:34.369 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:42:34.370 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":125,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:42:20.353491","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":27,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:42:17.322526","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"124","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":27,"pool_name":"cephfs_data","create_time":"2021-02-10 11:42:18.395545","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"125","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:42:34.371 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.371 7fca47905700 1 -- 172.21.2.112:0/1990462086 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fca20018730 msgr2=0x7fca2001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:34.371 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.371 7fca47905700 1 --2- 172.21.2.112:0/1990462086 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fca20018730 0x7fca2001ac10 crc :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:34.371 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.371 7fca47905700 1 -- 172.21.2.112:0/1990462086 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fca400b2110 msgr2=0x7fca401b7a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:34.371 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.371 7fca47905700 1 --2- 172.21.2.112:0/1990462086 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fca400b2110 0x7fca401b7a00 secure :-1 s=READY pgs=2357 cs=0 l=1 rev1=1 rx=0x7fca38009980 tx=0x7fca3800bcd0).stop 2021-02-10T11:42:34.371 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.371 7fca47905700 1 -- 172.21.2.112:0/1990462086 shutdown_connections 2021-02-10T11:42:34.371 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.371 7fca47905700 1 --2- 172.21.2.112:0/1990462086 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fca20018730 0x7fca2001ac10 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:34.371 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.371 7fca47905700 1 --2- 172.21.2.112:0/1990462086 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fca401b8ed0 0x7fca400b1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:34.371 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.371 7fca47905700 1 --2- 172.21.2.112:0/1990462086 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fca400b2110 0x7fca401b7a00 unknown :-1 s=CLOSED pgs=2357 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:34.372 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.371 7fca47905700 1 --2- 172.21.2.112:0/1990462086 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fca401b8ad0 0x7fca40187e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:34.372 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.371 7fca47905700 1 -- 172.21.2.112:0/1990462086 >> 172.21.2.112:0/1990462086 conn(0x7fca400ed810 msgr2=0x7fca40094980 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:34.372 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.371 7fca47905700 1 -- 172.21.2.112:0/1990462086 shutdown_connections 2021-02-10T11:42:34.372 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.371 7fca47905700 1 -- 172.21.2.112:0/1990462086 wait complete. 2021-02-10T11:42:34.379 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:13:57.508758', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 26, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:42:17.322526', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '124', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 27, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:42:18.395545', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '125', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:42:34.379 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:42:34.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.796 7f639994c700 1 Processor -- start 2021-02-10T11:42:34.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.796 7f639994c700 1 -- start start 2021-02-10T11:42:34.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.796 7f639994c700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f63940d1860 0x7f63941b04b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:34.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.796 7f639994c700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f63941b0ad0 0x7f6394108930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:34.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.796 7f639994c700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6394109a50 con 0x7f63940d1c60 2021-02-10T11:42:34.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.796 7f639994c700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63940e5ee0 con 0x7f63941b0ad0 2021-02-10T11:42:34.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.796 7f639994c700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6394109740 con 0x7f63940d1860 2021-02-10T11:42:34.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.796 7f6393d75700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f63940d1c60 0x7f6394108f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:55850/0 (socket says 172.21.2.112:55850) 2021-02-10T11:42:34.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.796 7f6393d75700 1 -- 172.21.2.112:0/757353904 learned_addr learned my addr 172.21.2.112:0/757353904 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:42:34.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.797 7f6392d73700 1 --2- 172.21.2.112:0/757353904 >> v2:172.21.2.120:3300/0 conn(0x7f63941b0ad0 0x7f6394108930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:34.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.797 7f6393574700 1 --2- 172.21.2.112:0/757353904 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f63940d1860 0x7f63941b04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:34.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.797 7f6392572700 1 -- 172.21.2.112:0/757353904 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4213270859 0 0) 0x7f638c000f90 con 0x7f63940d1c60 2021-02-10T11:42:34.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.797 7f6392572700 1 -- 172.21.2.112:0/757353904 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6380002fe0 con 0x7f63940d1c60 2021-02-10T11:42:34.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.797 7f6392572700 1 -- 172.21.2.112:0/757353904 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2883353590 0 0) 0x7f638c001270 con 0x7f63940d1c60 2021-02-10T11:42:34.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.797 7f6393574700 1 -- 172.21.2.112:0/757353904 >> v2:172.21.2.120:3300/0 conn(0x7f63941b0ad0 msgr2=0x7f6394108930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:34.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.797 7f6393574700 1 --2- 172.21.2.112:0/757353904 >> v2:172.21.2.120:3300/0 conn(0x7f63941b0ad0 0x7f6394108930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:34.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.797 7f6393574700 1 -- 172.21.2.112:0/757353904 >> v1:172.21.2.112:6789/0 conn(0x7f63940d1c60 legacy=0x7f6394108f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:34.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.797 7f6393574700 1 -- 172.21.2.112:0/757353904 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6394088070 con 0x7f63940d1860 2021-02-10T11:42:34.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.797 7f6392d73700 1 --2- 172.21.2.112:0/757353904 >> v2:172.21.2.120:3300/0 conn(0x7f63941b0ad0 0x7f6394108930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:42:34.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.797 7f6393574700 1 --2- 172.21.2.112:0/757353904 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f63940d1860 0x7f63941b04b0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7f637c006650 tx=0x7f637c009a80).ready entity=mon.2 client_cookie=2ca100e0c9fdd378 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:34.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.797 7f6392572700 1 -- 172.21.2.112:0/757353904 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f637c0009b0 con 0x7f63940d1860 2021-02-10T11:42:34.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.797 7f6392572700 1 -- 172.21.2.112:0/757353904 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f637c00c0c0 con 0x7f63940d1860 2021-02-10T11:42:34.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.797 7f6392572700 1 -- 172.21.2.112:0/757353904 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f637c00c5e0 con 0x7f63940d1860 2021-02-10T11:42:34.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.798 7f639994c700 1 -- 172.21.2.112:0/757353904 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f63940d1860 msgr2=0x7f63941b04b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:34.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.798 7f639994c700 1 --2- 172.21.2.112:0/757353904 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f63940d1860 0x7f63941b04b0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7f637c006650 tx=0x7f637c009a80).stop 2021-02-10T11:42:34.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.798 7f639994c700 1 -- 172.21.2.112:0/757353904 shutdown_connections 2021-02-10T11:42:34.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.798 7f639994c700 1 --2- 172.21.2.112:0/757353904 >> v2:172.21.2.120:3300/0 conn(0x7f63941b0ad0 0x7f6394108930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:34.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.798 7f639994c700 1 --2- 172.21.2.112:0/757353904 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f63940d1860 0x7f63941b04b0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:34.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.798 7f639994c700 1 -- 172.21.2.112:0/757353904 >> 172.21.2.112:0/757353904 conn(0x7f63940ce410 msgr2=0x7f63940d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:34.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.798 7f639994c700 1 -- 172.21.2.112:0/757353904 shutdown_connections 2021-02-10T11:42:34.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.798 7f639994c700 1 -- 172.21.2.112:0/757353904 wait complete. 2021-02-10T11:42:34.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.813 7f639994c700 1 Processor -- start 2021-02-10T11:42:34.816 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.813 7f639994c700 1 -- start start 2021-02-10T11:42:34.816 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.813 7f639994c700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f63941a4a40 0x7f6394094a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:34.816 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.813 7f639994c700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6394095050 0x7f6394095760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:34.816 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.813 7f639994c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6394095d80 0x7f6394096180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:34.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.813 7f639994c700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63940989c0 con 0x7f6394095d80 2021-02-10T11:42:34.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.813 7f639994c700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6394098bb0 con 0x7f6394095050 2021-02-10T11:42:34.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.813 7f639994c700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63940967a0 con 0x7f63941a4a40 2021-02-10T11:42:34.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.813 7f6393d75700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6394095d80 0x7f6394096180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:34.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.813 7f6393d75700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6394095d80 0x7f6394096180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:56394/0 (socket says 172.21.2.112:56394) 2021-02-10T11:42:34.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.813 7f6393d75700 1 -- 172.21.2.112:0/148416374 learned_addr learned my addr 172.21.2.112:0/148416374 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:42:34.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.814 7f6392d73700 1 --2- 172.21.2.112:0/148416374 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6394095050 0x7f6394095760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:34.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.814 7f6393574700 1 --2- 172.21.2.112:0/148416374 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f63941a4a40 0x7f6394094a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:34.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.814 7f6393d75700 1 -- 172.21.2.112:0/148416374 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f63941a4a40 msgr2=0x7f6394094a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:34.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.814 7f6393d75700 1 --2- 172.21.2.112:0/148416374 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f63941a4a40 0x7f6394094a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:34.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.814 7f6393d75700 1 -- 172.21.2.112:0/148416374 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6394095050 msgr2=0x7f6394095760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:34.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.814 7f6393d75700 1 --2- 172.21.2.112:0/148416374 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6394095050 0x7f6394095760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:34.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.814 7f6393d75700 1 -- 172.21.2.112:0/148416374 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63940969c0 con 0x7f6394095d80 2021-02-10T11:42:34.818 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.814 7f6393574700 1 --2- 172.21.2.112:0/148416374 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f63941a4a40 0x7f6394094a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:42:34.818 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.814 7f6393d75700 1 --2- 172.21.2.112:0/148416374 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6394095d80 0x7f6394096180 secure :-1 s=READY pgs=2359 cs=0 l=1 rev1=1 rx=0x7f638c001b20 tx=0x7f638c00ac90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:34.818 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.814 7f638bfff700 1 -- 172.21.2.112:0/148416374 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f638c004930 con 0x7f6394095d80 2021-02-10T11:42:34.818 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.814 7f638bfff700 1 -- 172.21.2.112:0/148416374 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f638c004c30 con 0x7f6394095d80 2021-02-10T11:42:34.818 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.814 7f639994c700 1 -- 172.21.2.112:0/148416374 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63940969c0 con 0x7f6394095d80 2021-02-10T11:42:34.818 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.814 7f639994c700 1 -- 172.21.2.112:0/148416374 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63940974d0 con 0x7f6394095d80 2021-02-10T11:42:34.818 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.814 7f638bfff700 1 -- 172.21.2.112:0/148416374 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f638c0072c0 con 0x7f6394095d80 2021-02-10T11:42:34.818 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.815 7f638bfff700 1 -- 172.21.2.112:0/148416374 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f638c004720 con 0x7f6394095d80 2021-02-10T11:42:34.818 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.815 7f638bfff700 1 --2- 172.21.2.112:0/148416374 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f63780186c0 0x7f637801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:34.818 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.815 7f638bfff700 1 -- 172.21.2.112:0/148416374 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f638c032b00 con 0x7f6394095d80 2021-02-10T11:42:34.818 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.815 7f6393574700 1 --2- 172.21.2.112:0/148416374 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f63780186c0 0x7f637801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:34.818 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.816 7f6393574700 1 --2- 172.21.2.112:0/148416374 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f63780186c0 0x7f637801abc0 crc :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:34.845 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.846 7f639994c700 1 -- 172.21.2.112:0/148416374 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6394082920 con 0x7f6394095d80 2021-02-10T11:42:34.848 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.849 7f638bfff700 1 -- 172.21.2.112:0/148416374 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f638c0015c0 con 0x7f6394095d80 2021-02-10T11:42:34.975 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.975 7f639994c700 1 -- 172.21.2.112:0/148416374 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f6394109a50 con 0x7f6394095d80 2021-02-10T11:42:34.975 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.976 7f638bfff700 1 -- 172.21.2.112:0/148416374 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v125) v1 ==== 92+0+10 (secure 0 0 0) 0x7f638c004c90 con 0x7f6394095d80 2021-02-10T11:42:34.975 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:42:34.977 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.978 7f639994c700 1 -- 172.21.2.112:0/148416374 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f63780186c0 msgr2=0x7f637801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:34.977 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.978 7f639994c700 1 --2- 172.21.2.112:0/148416374 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f63780186c0 0x7f637801abc0 crc :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:34.977 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.978 7f639994c700 1 -- 172.21.2.112:0/148416374 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6394095d80 msgr2=0x7f6394096180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:34.977 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.978 7f639994c700 1 --2- 172.21.2.112:0/148416374 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6394095d80 0x7f6394096180 secure :-1 s=READY pgs=2359 cs=0 l=1 rev1=1 rx=0x7f638c001b20 tx=0x7f638c00ac90).stop 2021-02-10T11:42:34.977 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.978 7f639994c700 1 -- 172.21.2.112:0/148416374 shutdown_connections 2021-02-10T11:42:34.977 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.978 7f639994c700 1 --2- 172.21.2.112:0/148416374 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f63780186c0 0x7f637801abc0 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:34.977 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.978 7f639994c700 1 --2- 172.21.2.112:0/148416374 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6394095050 0x7f6394095760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:34.977 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.978 7f639994c700 1 --2- 172.21.2.112:0/148416374 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6394095d80 0x7f6394096180 unknown :-1 s=CLOSED pgs=2359 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:34.977 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.978 7f639994c700 1 --2- 172.21.2.112:0/148416374 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f63941a4a40 0x7f6394094a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:34.977 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.978 7f639994c700 1 -- 172.21.2.112:0/148416374 >> 172.21.2.112:0/148416374 conn(0x7f63940c3e40 msgr2=0x7f63940916a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:34.978 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.978 7f639994c700 1 -- 172.21.2.112:0/148416374 shutdown_connections 2021-02-10T11:42:34.978 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:34.978 7f639994c700 1 -- 172.21.2.112:0/148416374 wait complete. 2021-02-10T11:42:34.987 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:42:35.416 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:35.401 7f5a2a1c0700 1 Processor -- start 2021-02-10T11:42:35.416 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:35.401 7f5a2a1c0700 1 -- start start 2021-02-10T11:42:35.416 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:35.401 7f5a2a1c0700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f5a240d1870 0x7f5a241b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:35.416 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:35.401 7f5a2a1c0700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a240fe1c0 con 0x7f5a240cff70 2021-02-10T11:42:35.416 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:35.401 7f5a2a1c0700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a24109a60 con 0x7f5a241b0ae0 2021-02-10T11:42:35.417 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:35.401 7f5a2a1c0700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a24088070 con 0x7f5a240d1870 2021-02-10T11:42:35.417 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:35.402 7f5a237fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f5a240d1870 0x7f5a241b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:35.417 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:35.402 7f5a23fff700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f5a240cff70 0x7f5a24109130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:55862/0 (socket says 172.21.2.112:55862) 2021-02-10T11:42:35.417 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:35.402 7f5a23fff700 1 -- 172.21.2.112:0/4216820892 learned_addr learned my addr 172.21.2.112:0/4216820892 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:42:35.417 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:35.402 7f5a237fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f5a240d1870 0x7f5a241b04c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:56400/0 (socket says 172.21.2.112:56400) 2021-02-10T11:42:35.417 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:35.402 7f5a237fe700 1 -- 172.21.2.112:0/4216820892 >> v1:172.21.2.120:6789/0 conn(0x7f5a241b0ae0 legacy=0x7f5a24108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:35.417 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:35.402 7f5a237fe700 1 -- 172.21.2.112:0/4216820892 >> v1:172.21.2.112:6789/0 conn(0x7f5a240cff70 legacy=0x7f5a24109130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:35.417 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:35.402 7f5a237fe700 1 -- 172.21.2.112:0/4216820892 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a24088260 con 0x7f5a240d1870 2021-02-10T11:42:35.417 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:35.402 7f5a237fe700 1 --2- 172.21.2.112:0/4216820892 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5a240d1870 0x7f5a241b04c0 secure :-1 s=READY pgs=2361 cs=0 l=1 rev1=1 rx=0x7f5a0c006650 tx=0x7f5a0c009a80).ready entity=mon.0 client_cookie=56c64fe71037ad61 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:35.417 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:35.402 7f5a227fc700 1 -- 172.21.2.112:0/4216820892 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5a0c0009b0 con 0x7f5a240d1870 2021-02-10T11:42:35.417 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:35.402 7f5a227fc700 1 -- 172.21.2.112:0/4216820892 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a0c00c0c0 con 0x7f5a240d1870 2021-02-10T11:42:35.417 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:35.402 7f5a227fc700 1 -- 172.21.2.112:0/4216820892 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5a0c00c5a0 con 0x7f5a240d1870 2021-02-10T11:42:35.417 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:35.403 7f5a2a1c0700 1 -- 172.21.2.112:0/4216820892 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5a240d1870 msgr2=0x7f5a241b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:35.418 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:35.403 7f5a2a1c0700 1 --2- 172.21.2.112:0/4216820892 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5a240d1870 0x7f5a241b04c0 secure :-1 s=READY pgs=2361 cs=0 l=1 rev1=1 rx=0x7f5a0c006650 tx=0x7f5a0c009a80).stop 2021-02-10T11:42:35.418 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:35.403 7f5a2a1c0700 1 -- 172.21.2.112:0/4216820892 shutdown_connections 2021-02-10T11:42:35.418 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:35.403 7f5a2a1c0700 1 --2- 172.21.2.112:0/4216820892 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5a240d1870 0x7f5a241b04c0 unknown :-1 s=CLOSED pgs=2361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:35.418 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:35.403 7f5a2a1c0700 1 -- 172.21.2.112:0/4216820892 >> 172.21.2.112:0/4216820892 conn(0x7f5a240ce420 msgr2=0x7f5a240d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:35.418 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:35.403 7f5a2a1c0700 1 -- 172.21.2.112:0/4216820892 shutdown_connections 2021-02-10T11:42:35.418 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:35.403 7f5a2a1c0700 1 -- 172.21.2.112:0/4216820892 wait complete. 2021-02-10T11:42:35.419 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:35.417 7f5a2a1c0700 1 Processor -- start 2021-02-10T11:42:35.420 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:35.418 7f5a2a1c0700 1 -- start start 2021-02-10T11:42:35.420 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:35.418 7f5a2a1c0700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5a2408a650 0x7f5a241c1f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:35.420 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:35.418 7f5a2a1c0700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5a24091e10 0x7f5a24092210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:35.420 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:35.418 7f5a2a1c0700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5a241076a0 0x7f5a24107aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:35.420 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:35.418 7f5a2a1c0700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a24092830 con 0x7f5a241076a0 2021-02-10T11:42:35.420 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:35.418 7f5a2a1c0700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a241003e0 con 0x7f5a24091e10 2021-02-10T11:42:35.420 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:35.418 7f5a2a1c0700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a241c2540 con 0x7f5a2408a650 2021-02-10T11:42:35.420 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:35.418 7f5a23fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5a241076a0 0x7f5a24107aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:35.420 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:35.418 7f5a23fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5a241076a0 0x7f5a24107aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:56410/0 (socket says 172.21.2.112:56410) 2021-02-10T11:42:35.420 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:35.418 7f5a23fff700 1 -- 172.21.2.112:0/3015770511 learned_addr learned my addr 172.21.2.112:0/3015770511 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:42:35.420 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:35.419 7f5a23fff700 1 -- 172.21.2.112:0/3015770511 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5a2408a650 msgr2=0x7f5a241c1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:35.420 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:35.419 7f5a23fff700 1 --2- 172.21.2.112:0/3015770511 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5a2408a650 0x7f5a241c1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:35.421 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:35.419 7f5a23fff700 1 -- 172.21.2.112:0/3015770511 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5a24091e10 msgr2=0x7f5a24092210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:35.421 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:35.419 7f5a23fff700 1 --2- 172.21.2.112:0/3015770511 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5a24091e10 0x7f5a24092210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:35.421 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:35.419 7f5a23fff700 1 -- 172.21.2.112:0/3015770511 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a2408aa50 con 0x7f5a241076a0 2021-02-10T11:42:35.421 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:35.419 7f5a23fff700 1 --2- 172.21.2.112:0/3015770511 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5a241076a0 0x7f5a24107aa0 secure :-1 s=READY pgs=2362 cs=0 l=1 rev1=1 rx=0x7f5a1c00a8a0 tx=0x7f5a1c00ac40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:35.421 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:35.419 7f5a1bfff700 1 -- 172.21.2.112:0/3015770511 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5a1c007020 con 0x7f5a241076a0 2021-02-10T11:42:35.421 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:35.419 7f5a1bfff700 1 -- 172.21.2.112:0/3015770511 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a1c007220 con 0x7f5a241076a0 2021-02-10T11:42:35.421 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:35.419 7f5a2a1c0700 1 -- 172.21.2.112:0/3015770511 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a2408aa50 con 0x7f5a241076a0 2021-02-10T11:42:35.421 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:35.419 7f5a2a1c0700 1 -- 172.21.2.112:0/3015770511 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a241083b0 con 0x7f5a241076a0 2021-02-10T11:42:35.421 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:35.419 7f5a1bfff700 1 -- 172.21.2.112:0/3015770511 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5a1c004720 con 0x7f5a241076a0 2021-02-10T11:42:35.421 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:35.419 7f5a1bfff700 1 -- 172.21.2.112:0/3015770511 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5a1c000d30 con 0x7f5a241076a0 2021-02-10T11:42:35.421 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:35.420 7f5a1bfff700 1 --2- 172.21.2.112:0/3015770511 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5a080186c0 0x7f5a0801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:35.421 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:35.420 7f5a1bfff700 1 -- 172.21.2.112:0/3015770511 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f5a1c0328f0 con 0x7f5a241076a0 2021-02-10T11:42:35.421 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:35.420 7f5a237fe700 1 --2- 172.21.2.112:0/3015770511 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5a080186c0 0x7f5a0801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:35.422 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:35.421 7f5a237fe700 1 --2- 172.21.2.112:0/3015770511 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5a080186c0 0x7f5a0801abc0 crc :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:35.450 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:35.451 7f5a2a1c0700 1 -- 172.21.2.112:0/3015770511 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a240918e0 con 0x7f5a241076a0 2021-02-10T11:42:35.453 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:35.454 7f5a1bfff700 1 -- 172.21.2.112:0/3015770511 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5a1c004920 con 0x7f5a241076a0 2021-02-10T11:42:35.580 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:35.581 7f5a2a1c0700 1 -- 172.21.2.112:0/3015770511 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f5a240918e0 con 0x7f5a241076a0 2021-02-10T11:42:35.580 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:35.581 7f5a1bfff700 1 -- 172.21.2.112:0/3015770511 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v125) v1 ==== 104+0+10 (secure 0 0 0) 0x7f5a1c000d30 con 0x7f5a241076a0 2021-02-10T11:42:35.581 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:42:35.582 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:35.583 7f5a2a1c0700 1 -- 172.21.2.112:0/3015770511 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5a080186c0 msgr2=0x7f5a0801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:35.582 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:35.583 7f5a2a1c0700 1 --2- 172.21.2.112:0/3015770511 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5a080186c0 0x7f5a0801abc0 crc :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:35.582 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:35.583 7f5a2a1c0700 1 -- 172.21.2.112:0/3015770511 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5a241076a0 msgr2=0x7f5a24107aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:35.582 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:35.583 7f5a2a1c0700 1 --2- 172.21.2.112:0/3015770511 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5a241076a0 0x7f5a24107aa0 secure :-1 s=READY pgs=2362 cs=0 l=1 rev1=1 rx=0x7f5a1c00a8a0 tx=0x7f5a1c00ac40).stop 2021-02-10T11:42:35.583 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:35.583 7f5a2a1c0700 1 -- 172.21.2.112:0/3015770511 shutdown_connections 2021-02-10T11:42:35.583 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:35.583 7f5a2a1c0700 1 --2- 172.21.2.112:0/3015770511 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5a080186c0 0x7f5a0801abc0 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:35.583 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:35.583 7f5a2a1c0700 1 --2- 172.21.2.112:0/3015770511 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5a24091e10 0x7f5a24092210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:35.583 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:35.583 7f5a2a1c0700 1 --2- 172.21.2.112:0/3015770511 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5a241076a0 0x7f5a24107aa0 unknown :-1 s=CLOSED pgs=2362 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:35.583 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:35.583 7f5a2a1c0700 1 --2- 172.21.2.112:0/3015770511 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5a2408a650 0x7f5a241c1f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:35.583 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:35.583 7f5a2a1c0700 1 -- 172.21.2.112:0/3015770511 >> 172.21.2.112:0/3015770511 conn(0x7f5a240bcff0 msgr2=0x7f5a2408e9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:35.583 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:35.584 7f5a2a1c0700 1 -- 172.21.2.112:0/3015770511 shutdown_connections 2021-02-10T11:42:35.583 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:35.584 7f5a2a1c0700 1 -- 172.21.2.112:0/3015770511 wait complete. 2021-02-10T11:42:35.591 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:42:36.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.006 7f8ff6955700 1 Processor -- start 2021-02-10T11:42:36.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.006 7f8ff6955700 1 -- start start 2021-02-10T11:42:36.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.007 7f8ff6955700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8ff00d1870 0x7f8ff01b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:36.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.007 7f8ff6955700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f8ff01b0ae0 0x7f8ff0108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:36.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.007 7f8ff6955700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ff0109a60 con 0x7f8ff00d1c70 2021-02-10T11:42:36.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.007 7f8ff6955700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ff00e5ef0 con 0x7f8ff01b0ae0 2021-02-10T11:42:36.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.007 7f8ff6955700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ff0109750 con 0x7f8ff00d1870 2021-02-10T11:42:36.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.007 7f8ff4f72700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7f8ff00d1c70 0x7f8ff0108f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:49436/0 (socket says 172.21.2.112:49436) 2021-02-10T11:42:36.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.007 7f8ff4f72700 1 -- 172.21.2.112:0/408224847 learned_addr learned my addr 172.21.2.112:0/408224847 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:42:36.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.007 7f8feffff700 1 --2- 172.21.2.112:0/408224847 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8ff00d1870 0x7f8ff01b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:36.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.007 7f8fef7fe700 1 --2- 172.21.2.112:0/408224847 >> v2:172.21.2.120:3300/0 conn(0x7f8ff01b0ae0 0x7f8ff0108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:36.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.008 7f8feffff700 1 -- 172.21.2.112:0/408224847 >> v2:172.21.2.120:3300/0 conn(0x7f8ff01b0ae0 msgr2=0x7f8ff0108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:36.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.008 7f8feffff700 1 --2- 172.21.2.112:0/408224847 >> v2:172.21.2.120:3300/0 conn(0x7f8ff01b0ae0 0x7f8ff0108940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:36.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.008 7f8feffff700 1 -- 172.21.2.112:0/408224847 >> v1:172.21.2.120:6789/0 conn(0x7f8ff00d1c70 legacy=0x7f8ff0108f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:36.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.008 7f8feeffd700 1 -- 172.21.2.112:0/408224847 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1434134541 0 0) 0x7f8fe8001440 con 0x7f8ff00d1c70 2021-02-10T11:42:36.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.008 7f8feffff700 1 -- 172.21.2.112:0/408224847 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ff0088070 con 0x7f8ff00d1870 2021-02-10T11:42:36.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.008 7f8feffff700 1 --2- 172.21.2.112:0/408224847 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8ff00d1870 0x7f8ff01b04c0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7f8fe0006650 tx=0x7f8fe0009a00).ready entity=mon.2 client_cookie=be34f64815591ac1 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:36.024 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.008 7f8feeffd700 1 -- 172.21.2.112:0/408224847 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8fe00009b0 con 0x7f8ff00d1870 2021-02-10T11:42:36.024 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.008 7f8feeffd700 1 -- 172.21.2.112:0/408224847 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fe000c0c0 con 0x7f8ff00d1870 2021-02-10T11:42:36.024 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.008 7f8ff6955700 1 -- 172.21.2.112:0/408224847 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8ff00d1870 msgr2=0x7f8ff01b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:36.024 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.008 7f8ff6955700 1 --2- 172.21.2.112:0/408224847 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8ff00d1870 0x7f8ff01b04c0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7f8fe0006650 tx=0x7f8fe0009a00).stop 2021-02-10T11:42:36.024 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.009 7f8ff6955700 1 -- 172.21.2.112:0/408224847 shutdown_connections 2021-02-10T11:42:36.024 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.009 7f8ff6955700 1 --2- 172.21.2.112:0/408224847 >> v2:172.21.2.120:3300/0 conn(0x7f8ff01b0ae0 0x7f8ff0108940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:36.024 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.009 7f8ff6955700 1 --2- 172.21.2.112:0/408224847 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8ff00d1870 0x7f8ff01b04c0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:36.024 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.009 7f8ff6955700 1 -- 172.21.2.112:0/408224847 >> 172.21.2.112:0/408224847 conn(0x7f8ff00ce420 msgr2=0x7f8ff00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:36.024 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.009 7f8ff6955700 1 -- 172.21.2.112:0/408224847 shutdown_connections 2021-02-10T11:42:36.024 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.009 7f8ff6955700 1 -- 172.21.2.112:0/408224847 wait complete. 2021-02-10T11:42:36.026 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.023 7f8ff6955700 1 Processor -- start 2021-02-10T11:42:36.026 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.023 7f8ff6955700 1 -- start start 2021-02-10T11:42:36.026 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.024 7f8ff6955700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8ff00a58c0 0x7f8ff00a4d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:36.026 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.024 7f8ff6955700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8ff00b1ba0 0x7f8ff00b1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:36.026 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.024 7f8ff6955700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8ff00b25c0 0x7f8ff00b3270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:36.026 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.024 7f8ff6955700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ff00b2a40 con 0x7f8ff00b25c0 2021-02-10T11:42:36.026 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.024 7f8ff6955700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ff00a5340 con 0x7f8ff00b1ba0 2021-02-10T11:42:36.026 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.024 7f8ff6955700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ff00a5560 con 0x7f8ff00a58c0 2021-02-10T11:42:36.026 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.024 7f8ff4f72700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8ff00b25c0 0x7f8ff00b3270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:36.027 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.024 7f8ff4f72700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8ff00b25c0 0x7f8ff00b3270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:56424/0 (socket says 172.21.2.112:56424) 2021-02-10T11:42:36.027 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.024 7f8ff4f72700 1 -- 172.21.2.112:0/477995413 learned_addr learned my addr 172.21.2.112:0/477995413 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:42:36.027 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.024 7f8fef7fe700 1 --2- 172.21.2.112:0/477995413 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8ff00b1ba0 0x7f8ff00b1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:36.027 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.024 7f8feffff700 1 --2- 172.21.2.112:0/477995413 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8ff00a58c0 0x7f8ff00a4d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:36.027 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.025 7f8fef7fe700 1 -- 172.21.2.112:0/477995413 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8ff00a58c0 msgr2=0x7f8ff00a4d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:36.027 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.025 7f8fef7fe700 1 --2- 172.21.2.112:0/477995413 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8ff00a58c0 0x7f8ff00a4d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:36.027 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.025 7f8fef7fe700 1 -- 172.21.2.112:0/477995413 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8ff00b25c0 msgr2=0x7f8ff00b3270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:36.027 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.025 7f8fef7fe700 1 --2- 172.21.2.112:0/477995413 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8ff00b25c0 0x7f8ff00b3270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:36.027 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.025 7f8fef7fe700 1 -- 172.21.2.112:0/477995413 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ff00b3890 con 0x7f8ff00b1ba0 2021-02-10T11:42:36.027 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.025 7f8fef7fe700 1 --2- 172.21.2.112:0/477995413 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8ff00b1ba0 0x7f8ff00b1fa0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 rx=0x7f8fe4009540 tx=0x7f8fe400b8e0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:36.027 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.025 7f8fecff9700 1 -- 172.21.2.112:0/477995413 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8fe40009b0 con 0x7f8ff00b1ba0 2021-02-10T11:42:36.027 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.025 7f8fecff9700 1 -- 172.21.2.112:0/477995413 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fe4006740 con 0x7f8ff00b1ba0 2021-02-10T11:42:36.027 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.025 7f8fecff9700 1 -- 172.21.2.112:0/477995413 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8fe4002030 con 0x7f8ff00b1ba0 2021-02-10T11:42:36.028 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.025 7f8ff6955700 1 -- 172.21.2.112:0/477995413 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ff00b3890 con 0x7f8ff00b1ba0 2021-02-10T11:42:36.028 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.025 7f8ff6955700 1 -- 172.21.2.112:0/477995413 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ff00b3890 con 0x7f8ff00b1ba0 2021-02-10T11:42:36.028 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.026 7f8fecff9700 1 -- 172.21.2.112:0/477995413 <== mon.1 v2:172.21.2.120:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8fe40052e0 con 0x7f8ff00b1ba0 2021-02-10T11:42:36.028 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.026 7f8fecff9700 1 --2- 172.21.2.112:0/477995413 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8fd00186c0 0x7f8fd001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:36.028 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.026 7f8fecff9700 1 -- 172.21.2.112:0/477995413 <== mon.1 v2:172.21.2.120:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f8fe4032440 con 0x7f8ff00b1ba0 2021-02-10T11:42:36.028 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.026 7f8feffff700 1 --2- 172.21.2.112:0/477995413 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8fd00186c0 0x7f8fd001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:36.028 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.027 7f8feffff700 1 --2- 172.21.2.112:0/477995413 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8fd00186c0 0x7f8fd001abc0 crc :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:36.055 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.056 7f8ff6955700 1 -- 172.21.2.112:0/477995413 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ff00b2ac0 con 0x7f8ff00b1ba0 2021-02-10T11:42:36.059 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.060 7f8fecff9700 1 -- 172.21.2.112:0/477995413 <== mon.1 v2:172.21.2.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8fe4000c90 con 0x7f8ff00b1ba0 2021-02-10T11:42:36.189 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.190 7f8ff6955700 1 -- 172.21.2.112:0/477995413 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f8ff00b2ac0 con 0x7f8ff00b1ba0 2021-02-10T11:42:36.190 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.191 7f8fecff9700 1 -- 172.21.2.112:0/477995413 <== mon.1 v2:172.21.2.120:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v125) v1 ==== 100+0+10 (secure 0 0 0) 0x7f8fe40052e0 con 0x7f8ff00b1ba0 2021-02-10T11:42:36.190 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:42:36.191 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.192 7f8ff6955700 1 -- 172.21.2.112:0/477995413 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8fd00186c0 msgr2=0x7f8fd001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:36.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.192 7f8ff6955700 1 --2- 172.21.2.112:0/477995413 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8fd00186c0 0x7f8fd001abc0 crc :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:36.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.192 7f8ff6955700 1 -- 172.21.2.112:0/477995413 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8ff00b1ba0 msgr2=0x7f8ff00b1fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:36.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.192 7f8ff6955700 1 --2- 172.21.2.112:0/477995413 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8ff00b1ba0 0x7f8ff00b1fa0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 rx=0x7f8fe4009540 tx=0x7f8fe400b8e0).stop 2021-02-10T11:42:36.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.193 7f8ff6955700 1 -- 172.21.2.112:0/477995413 shutdown_connections 2021-02-10T11:42:36.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.193 7f8ff6955700 1 --2- 172.21.2.112:0/477995413 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8fd00186c0 0x7f8fd001abc0 unknown :-1 s=CLOSED pgs=1080 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:36.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.193 7f8ff6955700 1 --2- 172.21.2.112:0/477995413 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8ff00b1ba0 0x7f8ff00b1fa0 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:36.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.193 7f8ff6955700 1 --2- 172.21.2.112:0/477995413 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8ff00b25c0 0x7f8ff00b3270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:36.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.193 7f8ff6955700 1 --2- 172.21.2.112:0/477995413 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8ff00a58c0 0x7f8ff00a4d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:36.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.193 7f8ff6955700 1 -- 172.21.2.112:0/477995413 >> 172.21.2.112:0/477995413 conn(0x7f8ff00b5dc0 msgr2=0x7f8ff0095960 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:36.193 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.193 7f8ff6955700 1 -- 172.21.2.112:0/477995413 shutdown_connections 2021-02-10T11:42:36.193 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.193 7f8ff6955700 1 -- 172.21.2.112:0/477995413 wait complete. 2021-02-10T11:42:36.201 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:42:36.201 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:42:36.630 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.614 7f61cd576700 1 Processor -- start 2021-02-10T11:42:36.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.614 7f61cd576700 1 -- start start 2021-02-10T11:42:36.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.615 7f61cd576700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f61c80d0340 0x7f61c80ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:36.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.615 7f61cd576700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f61c80d1c70 0x7f61c80efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:36.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.615 7f61cd576700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61c80f0b10 con 0x7f61c80d1870 2021-02-10T11:42:36.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.615 7f61cd576700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61c80e5ed0 con 0x7f61c80d0340 2021-02-10T11:42:36.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.615 7f61cd576700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61c81b0be0 con 0x7f61c80d1c70 2021-02-10T11:42:36.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.615 7f61c72bf700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f61c80d1870 0x7f61c81b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:55890/0 (socket says 172.21.2.112:55890) 2021-02-10T11:42:36.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.615 7f61c72bf700 1 -- 172.21.2.112:0/3311573690 learned_addr learned my addr 172.21.2.112:0/3311573690 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:42:36.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.615 7f61c62bd700 1 -- 172.21.2.112:0/3311573690 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1271812260 0 0) 0x7f61b00010b0 con 0x7f61c80d1870 2021-02-10T11:42:36.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.615 7f61c62bd700 1 -- 172.21.2.112:0/3311573690 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61b4002fe0 con 0x7f61c80d1870 2021-02-10T11:42:36.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.615 7f61c6abe700 1 --2- 172.21.2.112:0/3311573690 >> v2:172.21.2.120:3300/0 conn(0x7f61c80d0340 0x7f61c80ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:36.632 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.615 7f61c7ac0700 1 --2- 172.21.2.112:0/3311573690 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f61c80d1c70 0x7f61c80efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:36.632 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.615 7f61c62bd700 1 -- 172.21.2.112:0/3311573690 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 935808887 0 0) 0x7f61b00010b0 con 0x7f61c80d1870 2021-02-10T11:42:36.632 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.616 7f61c62bd700 1 -- 172.21.2.112:0/3311573690 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f61c80d1c70 msgr2=0x7f61c80efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:36.632 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.616 7f61c62bd700 1 --2- 172.21.2.112:0/3311573690 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f61c80d1c70 0x7f61c80efc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:36.632 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.616 7f61c62bd700 1 -- 172.21.2.112:0/3311573690 >> v2:172.21.2.120:3300/0 conn(0x7f61c80d0340 msgr2=0x7f61c80ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:36.632 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.616 7f61c62bd700 1 --2- 172.21.2.112:0/3311573690 >> v2:172.21.2.120:3300/0 conn(0x7f61c80d0340 0x7f61c80ef640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:36.632 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.616 7f61c62bd700 1 -- 172.21.2.112:0/3311573690 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61c80f20a0 con 0x7f61c80d1870 2021-02-10T11:42:36.632 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.616 7f61c62bd700 1 -- 172.21.2.112:0/3311573690 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f61b00015b0 con 0x7f61c80d1870 2021-02-10T11:42:36.632 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.616 7f61c62bd700 1 -- 172.21.2.112:0/3311573690 >> v1:172.21.2.112:6789/0 conn(0x7f61c80d1870 legacy=0x7f61c81b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:36.632 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.616 7f61c62bd700 1 --2- 172.21.2.112:0/3311573690 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f61b4003ae0 0x7f61b4005f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:36.632 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.616 7f61c62bd700 1 --2- 172.21.2.112:0/3311573690 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f61b4006770 0x7f61b4008c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:36.632 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.616 7f61c62bd700 1 --2- 172.21.2.112:0/3311573690 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f61b4009510 0x7f61b400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:36.632 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.616 7f61c62bd700 1 -- 172.21.2.112:0/3311573690 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61b400c2c0 con 0x7f61b4009510 2021-02-10T11:42:36.633 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.616 7f61c62bd700 1 -- 172.21.2.112:0/3311573690 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61b40504b0 con 0x7f61b4006770 2021-02-10T11:42:36.633 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.616 7f61c62bd700 1 -- 172.21.2.112:0/3311573690 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61b40506d0 con 0x7f61b4003ae0 2021-02-10T11:42:36.633 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.616 7f61c7ac0700 1 --2- 172.21.2.112:0/3311573690 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f61b4009510 0x7f61b400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:36.633 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.616 7f61c6abe700 1 --2- 172.21.2.112:0/3311573690 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f61b4006770 0x7f61b4008c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:36.633 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.616 7f61c72bf700 1 --2- 172.21.2.112:0/3311573690 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f61b4003ae0 0x7f61b4005f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:36.633 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.616 7f61c7ac0700 1 -- 172.21.2.112:0/3311573690 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f61b4003ae0 msgr2=0x7f61b4005f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:36.633 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.616 7f61c7ac0700 1 --2- 172.21.2.112:0/3311573690 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f61b4003ae0 0x7f61b4005f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:36.633 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.616 7f61c7ac0700 1 -- 172.21.2.112:0/3311573690 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f61b4006770 msgr2=0x7f61b4008c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:36.633 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.616 7f61c7ac0700 1 --2- 172.21.2.112:0/3311573690 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f61b4006770 0x7f61b4008c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:36.633 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.616 7f61c7ac0700 1 -- 172.21.2.112:0/3311573690 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61b4050970 con 0x7f61b4009510 2021-02-10T11:42:36.633 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.616 7f61c7ac0700 1 --2- 172.21.2.112:0/3311573690 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f61b4009510 0x7f61b400ba10 secure :-1 s=READY pgs=2364 cs=0 l=1 rev1=1 rx=0x7f61c0006e50 tx=0x7f61c0009e70).ready entity=mon.0 client_cookie=e9b8498d5cec6d47 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:36.633 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.616 7f61c62bd700 1 -- 172.21.2.112:0/3311573690 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f61c000b300 con 0x7f61b4009510 2021-02-10T11:42:36.633 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.616 7f61c62bd700 1 -- 172.21.2.112:0/3311573690 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61c0003030 con 0x7f61b4009510 2021-02-10T11:42:36.634 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.616 7f61c62bd700 1 -- 172.21.2.112:0/3311573690 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f61c000b730 con 0x7f61b4009510 2021-02-10T11:42:36.634 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.617 7f61cd576700 1 -- 172.21.2.112:0/3311573690 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f61b4009510 msgr2=0x7f61b400ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:36.634 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.617 7f61cd576700 1 --2- 172.21.2.112:0/3311573690 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f61b4009510 0x7f61b400ba10 secure :-1 s=READY pgs=2364 cs=0 l=1 rev1=1 rx=0x7f61c0006e50 tx=0x7f61c0009e70).stop 2021-02-10T11:42:36.634 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.617 7f61c7ac0700 1 -- 172.21.2.112:0/3311573690 reap_dead start 2021-02-10T11:42:36.634 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.617 7f61cd576700 1 -- 172.21.2.112:0/3311573690 shutdown_connections 2021-02-10T11:42:36.634 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.617 7f61cd576700 1 -- 172.21.2.112:0/3311573690 >> 172.21.2.112:0/3311573690 conn(0x7f61c80ce450 msgr2=0x7f61c80d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:36.634 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.617 7f61cd576700 1 -- 172.21.2.112:0/3311573690 shutdown_connections 2021-02-10T11:42:36.634 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.617 7f61cd576700 1 -- 172.21.2.112:0/3311573690 wait complete. 2021-02-10T11:42:36.634 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.631 7f61cd576700 1 Processor -- start 2021-02-10T11:42:36.634 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.631 7f61cd576700 1 -- start start 2021-02-10T11:42:36.636 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.632 7f61cd576700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f61c8195650 0x7f61c80b5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:36.636 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.632 7f61cd576700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f61c80b5b80 0x7f61c81a2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:36.636 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.632 7f61cd576700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f61c80acfa0 0x7f61c81a3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:36.636 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.632 7f61cd576700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61c80b6010 con 0x7f61c80b5b80 2021-02-10T11:42:36.636 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.632 7f61cd576700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61c81a3a60 con 0x7f61c80acfa0 2021-02-10T11:42:36.636 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.632 7f61cd576700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61c81a3c40 con 0x7f61c8195650 2021-02-10T11:42:36.636 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.632 7f61c6abe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f61c80b5b80 0x7f61c81a2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:36.636 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.632 7f61c6abe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f61c80b5b80 0x7f61c81a2e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:56440/0 (socket says 172.21.2.112:56440) 2021-02-10T11:42:36.636 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.632 7f61c6abe700 1 -- 172.21.2.112:0/3108877011 learned_addr learned my addr 172.21.2.112:0/3108877011 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:42:36.636 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.632 7f61c6abe700 1 -- 172.21.2.112:0/3108877011 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f61c8195650 msgr2=0x7f61c80b5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:36.637 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.632 7f61c6abe700 1 --2- 172.21.2.112:0/3108877011 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f61c8195650 0x7f61c80b5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:36.637 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.632 7f61c6abe700 1 -- 172.21.2.112:0/3108877011 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f61c80acfa0 msgr2=0x7f61c81a3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:36.637 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.632 7f61c6abe700 1 --2- 172.21.2.112:0/3108877011 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f61c80acfa0 0x7f61c81a3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:36.637 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.632 7f61c6abe700 1 -- 172.21.2.112:0/3108877011 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61c80bcae0 con 0x7f61c80b5b80 2021-02-10T11:42:36.637 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.633 7f61c6abe700 1 --2- 172.21.2.112:0/3108877011 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f61c80b5b80 0x7f61c81a2e20 secure :-1 s=READY pgs=2365 cs=0 l=1 rev1=1 rx=0x7f61b8006950 tx=0x7f61b800b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:36.638 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.633 7f61bffff700 1 -- 172.21.2.112:0/3108877011 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f61b800bd50 con 0x7f61c80b5b80 2021-02-10T11:42:36.638 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.633 7f61bffff700 1 -- 172.21.2.112:0/3108877011 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61b8000c90 con 0x7f61c80b5b80 2021-02-10T11:42:36.638 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.633 7f61bffff700 1 -- 172.21.2.112:0/3108877011 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f61b80022b0 con 0x7f61c80b5b80 2021-02-10T11:42:36.638 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.633 7f61cd576700 1 -- 172.21.2.112:0/3108877011 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61c80bcae0 con 0x7f61c80b5b80 2021-02-10T11:42:36.638 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.633 7f61cd576700 1 -- 172.21.2.112:0/3108877011 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61c80bd070 con 0x7f61c80b5b80 2021-02-10T11:42:36.638 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.633 7f61bffff700 1 -- 172.21.2.112:0/3108877011 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f61b8002020 con 0x7f61c80b5b80 2021-02-10T11:42:36.638 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.633 7f61bffff700 1 --2- 172.21.2.112:0/3108877011 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f61ac0186c0 0x7f61ac01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:36.638 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.634 7f61bffff700 1 -- 172.21.2.112:0/3108877011 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f61b8032580 con 0x7f61c80b5b80 2021-02-10T11:42:36.638 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.634 7f61c72bf700 1 --2- 172.21.2.112:0/3108877011 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f61ac0186c0 0x7f61ac01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:36.638 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.634 7f61c72bf700 1 --2- 172.21.2.112:0/3108877011 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f61ac0186c0 0x7f61ac01abc0 crc :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:36.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.665 7f61cd576700 1 -- 172.21.2.112:0/3108877011 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61c80b6010 con 0x7f61c80b5b80 2021-02-10T11:42:36.666 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.667 7f61bffff700 1 -- 172.21.2.112:0/3108877011 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f61b8000960 con 0x7f61c80b5b80 2021-02-10T11:42:36.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.795 7f61cd576700 1 -- 172.21.2.112:0/3108877011 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f61c80b6010 con 0x7f61c80b5b80 2021-02-10T11:42:36.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.795 7f61bffff700 1 -- 172.21.2.112:0/3108877011 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+2108 (secure 0 0 0) 0x7f61b800bd50 con 0x7f61c80b5b80 2021-02-10T11:42:36.795 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:42:36.795 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":105,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11040,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/1814390924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":1814390924},{"type":"v1","addr":"172.21.2.112:6833","nonce":1814390924}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":101}],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:42:19.351348","modified":"2021-02-10 11:42:21.357585","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8203},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8203":{"gid":8203,"name":"b","rank":0,"incarnation":102,"state":"up:active","state_seq":3,"addr":"172.21.2.120:6835/2761661070","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":2761661070},{"type":"v1","addr":"172.21.2.120:6835","nonce":2761661070}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":13}]} 2021-02-10T11:42:36.795 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 105 2021-02-10T11:42:36.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.797 7f61cd576700 1 -- 172.21.2.112:0/3108877011 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f61ac0186c0 msgr2=0x7f61ac01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:36.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.797 7f61cd576700 1 --2- 172.21.2.112:0/3108877011 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f61ac0186c0 0x7f61ac01abc0 crc :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:36.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.797 7f61cd576700 1 -- 172.21.2.112:0/3108877011 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f61c80b5b80 msgr2=0x7f61c81a2e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:36.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.797 7f61cd576700 1 --2- 172.21.2.112:0/3108877011 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f61c80b5b80 0x7f61c81a2e20 secure :-1 s=READY pgs=2365 cs=0 l=1 rev1=1 rx=0x7f61b8006950 tx=0x7f61b800b960).stop 2021-02-10T11:42:36.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.797 7f61cd576700 1 -- 172.21.2.112:0/3108877011 shutdown_connections 2021-02-10T11:42:36.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.797 7f61cd576700 1 --2- 172.21.2.112:0/3108877011 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f61ac0186c0 0x7f61ac01abc0 unknown :-1 s=CLOSED pgs=1081 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:36.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.797 7f61cd576700 1 --2- 172.21.2.112:0/3108877011 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f61c80acfa0 0x7f61c81a3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:36.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.797 7f61cd576700 1 --2- 172.21.2.112:0/3108877011 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f61c80b5b80 0x7f61c81a2e20 unknown :-1 s=CLOSED pgs=2365 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:36.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.797 7f61cd576700 1 --2- 172.21.2.112:0/3108877011 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f61c8195650 0x7f61c80b5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:36.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.797 7f61cd576700 1 -- 172.21.2.112:0/3108877011 >> 172.21.2.112:0/3108877011 conn(0x7f61c80b3040 msgr2=0x7f61c80c3250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:36.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.797 7f61cd576700 1 -- 172.21.2.112:0/3108877011 shutdown_connections 2021-02-10T11:42:36.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:36.797 7f61cd576700 1 -- 172.21.2.112:0/3108877011 wait complete. 2021-02-10T11:42:37.006 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 105, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:42:19.351348', 'modified': '2021-02-10 11:42:21.357585', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8203}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8203': {'gid': 8203, 'name': 'b', 'rank': 0, 'incarnation': 102, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.120:6835/2761661070', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.120:6834', 'nonce': 2761661070}, {'type': 'v1', 'addr': '172.21.2.120:6835', 'nonce': 2761661070}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [27], 'metadata_pool': 26, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:42:37.006 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:42:37.007 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:42:37.404 INFO:teuthology.orchestra.run.gibba020.stdout:{ 2021-02-10T11:42:37.404 INFO:teuthology.orchestra.run.gibba020.stdout: "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", 2021-02-10T11:42:37.405 INFO:teuthology.orchestra.run.gibba020.stdout: "whoami": 0, 2021-02-10T11:42:37.405 INFO:teuthology.orchestra.run.gibba020.stdout: "id": 8203, 2021-02-10T11:42:37.405 INFO:teuthology.orchestra.run.gibba020.stdout: "want_state": "up:active", 2021-02-10T11:42:37.405 INFO:teuthology.orchestra.run.gibba020.stdout: "state": "up:active", 2021-02-10T11:42:37.405 INFO:teuthology.orchestra.run.gibba020.stdout: "rank_uptime": 18.052603960999999, 2021-02-10T11:42:37.405 INFO:teuthology.orchestra.run.gibba020.stdout: "mdsmap_epoch": 105, 2021-02-10T11:42:37.405 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch": 125, 2021-02-10T11:42:37.405 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch_barrier": 125, 2021-02-10T11:42:37.405 INFO:teuthology.orchestra.run.gibba020.stdout: "uptime": 23.683672789999999 2021-02-10T11:42:37.405 INFO:teuthology.orchestra.run.gibba020.stdout:} 2021-02-10T11:42:37.409 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", "whoami": 0, "id": 8203, "want_state": "up:active", "state": "up:active", "rank_uptime": 18.052603960999999, "mdsmap_epoch": 105, "osdmap_epoch": 125, "osdmap_epoch_barrier": 125, "uptime": 23.683672789999999 } 2021-02-10T11:42:37.409 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:42:37.409 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2021-02-10T11:42:37.409 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.1 at ubuntu@gibba012.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2021-02-10T11:42:37.409 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:42:37.409 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:42:37.744 DEBUG:teuthology.orchestra.run.gibba012:> sudo modprobe fuse 2021-02-10T11:42:38.088 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:42:38.088 DEBUG:teuthology.orchestra.run.gibba012:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:42:38.430 INFO:teuthology.orchestra.run.gibba012.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:42:38.431 INFO:teuthology.orchestra.run.gibba012.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:42:38.431 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:42:38.432 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:42:38.432 DEBUG:teuthology.orchestra.run.gibba012:> ls /sys/fs/fuse/connections 2021-02-10T11:42:38.766 INFO:teuthology.orchestra.run.gibba012.stdout:42 2021-02-10T11:42:38.766 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [42] 2021-02-10T11:42:38.766 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.1 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:42:38.769 DEBUG:teuthology.orchestra.run.gibba012:> sudo modprobe fuse 2021-02-10T11:42:39.113 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:42:39.113 DEBUG:teuthology.orchestra.run.gibba012:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:42:39.179 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba012.stderr:ceph-fuse[166270]: starting ceph client 2021-02-10T11:42:39.179 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba012.stderr:2021-02-10 11:42:39.179 7f064d4caf80 -1 init, newargv = 0x5572b0e1b220 newargc=9 2021-02-10T11:42:39.185 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba012.stderr:ceph-fuse[166270]: starting fuse 2021-02-10T11:42:39.455 INFO:teuthology.orchestra.run.gibba012.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:42:39.455 INFO:teuthology.orchestra.run.gibba012.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:42:39.456 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:42:39.456 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:42:39.457 DEBUG:teuthology.orchestra.run.gibba012:> ls /sys/fs/fuse/connections 2021-02-10T11:42:39.791 INFO:teuthology.orchestra.run.gibba012.stdout:42 2021-02-10T11:42:39.791 INFO:teuthology.orchestra.run.gibba012.stdout:43 2021-02-10T11:42:39.792 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42, 43] 2021-02-10T11:42:39.792 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:42:39.792 DEBUG:teuthology.orchestra.run.gibba012:> sudo python3 -c ' 2021-02-10T11:42:39.792 DEBUG:teuthology.orchestra.run.gibba012:> import glob 2021-02-10T11:42:39.792 DEBUG:teuthology.orchestra.run.gibba012:> import re 2021-02-10T11:42:39.792 DEBUG:teuthology.orchestra.run.gibba012:> import os 2021-02-10T11:42:39.792 DEBUG:teuthology.orchestra.run.gibba012:> import subprocess 2021-02-10T11:42:39.792 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:42:39.792 DEBUG:teuthology.orchestra.run.gibba012:> def find_socket(client_name): 2021-02-10T11:42:39.792 DEBUG:teuthology.orchestra.run.gibba012:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2021-02-10T11:42:39.792 DEBUG:teuthology.orchestra.run.gibba012:> files = glob.glob(asok_path) 2021-02-10T11:42:39.793 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:42:39.793 DEBUG:teuthology.orchestra.run.gibba012:> # Given a non-glob path, it better be there 2021-02-10T11:42:39.793 DEBUG:teuthology.orchestra.run.gibba012:> if "*" not in asok_path: 2021-02-10T11:42:39.793 DEBUG:teuthology.orchestra.run.gibba012:> assert(len(files) == 1) 2021-02-10T11:42:39.793 DEBUG:teuthology.orchestra.run.gibba012:> return files[0] 2021-02-10T11:42:39.793 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:42:39.793 DEBUG:teuthology.orchestra.run.gibba012:> for f in files: 2021-02-10T11:42:39.793 DEBUG:teuthology.orchestra.run.gibba012:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:42:39.793 DEBUG:teuthology.orchestra.run.gibba012:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:42:39.793 DEBUG:teuthology.orchestra.run.gibba012:> return f 2021-02-10T11:42:39.793 DEBUG:teuthology.orchestra.run.gibba012:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:42:39.793 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:42:39.793 DEBUG:teuthology.orchestra.run.gibba012:> print(find_socket("client.1")) 2021-02-10T11:42:39.793 DEBUG:teuthology.orchestra.run.gibba012:> ' 2021-02-10T11:42:40.155 INFO:teuthology.orchestra.run.gibba012.stdout:/var/run/ceph/ceph-client.1.166270.asok 2021-02-10T11:42:40.158 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.166270.asok 2021-02-10T11:42:40.158 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:42:40.158 DEBUG:teuthology.orchestra.run.gibba012:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.166270.asok status 2021-02-10T11:42:40.550 INFO:teuthology.orchestra.run.gibba012.stdout:{ 2021-02-10T11:42:40.550 INFO:teuthology.orchestra.run.gibba012.stdout: "metadata": { 2021-02-10T11:42:40.550 INFO:teuthology.orchestra.run.gibba012.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:42:40.551 INFO:teuthology.orchestra.run.gibba012.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:42:40.551 INFO:teuthology.orchestra.run.gibba012.stdout: "entity_id": "1", 2021-02-10T11:42:40.551 INFO:teuthology.orchestra.run.gibba012.stdout: "hostname": "gibba012", 2021-02-10T11:42:40.551 INFO:teuthology.orchestra.run.gibba012.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2021-02-10T11:42:40.551 INFO:teuthology.orchestra.run.gibba012.stdout: "pid": "166270", 2021-02-10T11:42:40.551 INFO:teuthology.orchestra.run.gibba012.stdout: "root": "/" 2021-02-10T11:42:40.551 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:40.551 INFO:teuthology.orchestra.run.gibba012.stdout: "dentry_count": 0, 2021-02-10T11:42:40.551 INFO:teuthology.orchestra.run.gibba012.stdout: "dentry_pinned_count": 0, 2021-02-10T11:42:40.551 INFO:teuthology.orchestra.run.gibba012.stdout: "id": 11229, 2021-02-10T11:42:40.551 INFO:teuthology.orchestra.run.gibba012.stdout: "inst": { 2021-02-10T11:42:40.551 INFO:teuthology.orchestra.run.gibba012.stdout: "name": { 2021-02-10T11:42:40.551 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "client", 2021-02-10T11:42:40.552 INFO:teuthology.orchestra.run.gibba012.stdout: "num": 11229 2021-02-10T11:42:40.552 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:40.552 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": { 2021-02-10T11:42:40.552 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:42:40.552 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:0", 2021-02-10T11:42:40.552 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 317381829 2021-02-10T11:42:40.552 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:42:40.552 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:40.552 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": { 2021-02-10T11:42:40.552 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:42:40.552 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:0", 2021-02-10T11:42:40.552 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 317381829 2021-02-10T11:42:40.552 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:40.553 INFO:teuthology.orchestra.run.gibba012.stdout: "inst_str": "client.11229 172.21.2.112:0/317381829", 2021-02-10T11:42:40.553 INFO:teuthology.orchestra.run.gibba012.stdout: "addr_str": "172.21.2.112:0/317381829", 2021-02-10T11:42:40.553 INFO:teuthology.orchestra.run.gibba012.stdout: "inode_count": 1, 2021-02-10T11:42:40.553 INFO:teuthology.orchestra.run.gibba012.stdout: "mds_epoch": 105, 2021-02-10T11:42:40.553 INFO:teuthology.orchestra.run.gibba012.stdout: "osd_epoch": 125, 2021-02-10T11:42:40.553 INFO:teuthology.orchestra.run.gibba012.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:42:40.553 INFO:teuthology.orchestra.run.gibba012.stdout: "blacklisted": false 2021-02-10T11:42:40.553 INFO:teuthology.orchestra.run.gibba012.stdout:} 2021-02-10T11:42:40.554 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:42:40.555 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:42:40.555 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:42:40.890 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:42:40.890 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:42:40.890 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:42:40.890 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:42:41.237 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2021-02-10T11:42:41.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:41.654 7fe296fd6700 1 Processor -- start 2021-02-10T11:42:41.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:41.654 7fe296fd6700 1 -- start start 2021-02-10T11:42:41.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:41.654 7fe296fd6700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fe2900d1c80 0x7fe2900cef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:41.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:41.655 7fe296fd6700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2900d0180 con 0x7fe2900cf590 2021-02-10T11:42:41.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:41.655 7fe296fd6700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2900fe1d0 con 0x7fe2900d1880 2021-02-10T11:42:41.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:41.655 7fe296fd6700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2901b0cc0 con 0x7fe2900d1c80 2021-02-10T11:42:41.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:41.655 7fe2955f3700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fe2900cf590 0x7fe2900cf990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:55924/0 (socket says 172.21.2.112:55924) 2021-02-10T11:42:41.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:41.655 7fe2955f3700 1 -- 172.21.2.112:0/1662288873 learned_addr learned my addr 172.21.2.112:0/1662288873 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:42:41.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:41.655 7fe28f7fe700 1 -- 172.21.2.112:0/1662288873 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2749171143 0 0) 0x7fe288001180 con 0x7fe2900cf590 2021-02-10T11:42:41.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:41.655 7fe28f7fe700 1 -- 172.21.2.112:0/1662288873 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe27c002fe0 con 0x7fe2900cf590 2021-02-10T11:42:41.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:41.655 7fe28ffff700 1 --2- 172.21.2.112:0/1662288873 >> v2:172.21.2.120:3300/0 conn(0x7fe2900d1c80 0x7fe2900cef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:41.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:41.656 7fe28f7fe700 1 -- 172.21.2.112:0/1662288873 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 230111503 0 0) 0x7fe2880010e0 con 0x7fe2900cf590 2021-02-10T11:42:41.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:41.656 7fe28f7fe700 1 -- 172.21.2.112:0/1662288873 >> v2:172.21.2.120:3300/0 conn(0x7fe2900d1c80 msgr2=0x7fe2900cef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:41.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:41.656 7fe28f7fe700 1 --2- 172.21.2.112:0/1662288873 >> v2:172.21.2.120:3300/0 conn(0x7fe2900d1c80 0x7fe2900cef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:41.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:41.656 7fe28f7fe700 1 -- 172.21.2.112:0/1662288873 >> v1:172.21.2.120:6789/0 conn(0x7fe2900d1880 legacy=0x7fe2901b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:41.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:41.656 7fe28f7fe700 1 -- 172.21.2.112:0/1662288873 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe290109a70 con 0x7fe2900cf590 2021-02-10T11:42:41.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:41.656 7fe28f7fe700 1 -- 172.21.2.112:0/1662288873 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7fe288001600 con 0x7fe2900cf590 2021-02-10T11:42:41.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:41.656 7fe28f7fe700 1 -- 172.21.2.112:0/1662288873 >> v1:172.21.2.112:6789/0 conn(0x7fe2900cf590 legacy=0x7fe2900cf990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:41.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:41.656 7fe28f7fe700 1 --2- 172.21.2.112:0/1662288873 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe27c003aa0 0x7fe27c005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:41.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:41.656 7fe28f7fe700 1 --2- 172.21.2.112:0/1662288873 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe27c006770 0x7fe27c008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:41.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:41.656 7fe28f7fe700 1 --2- 172.21.2.112:0/1662288873 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe27c009590 0x7fe27c00bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:41.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:41.656 7fe28f7fe700 1 -- 172.21.2.112:0/1662288873 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe27c00c300 con 0x7fe27c006770 2021-02-10T11:42:41.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:41.656 7fe28f7fe700 1 -- 172.21.2.112:0/1662288873 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe27c00c520 con 0x7fe27c003aa0 2021-02-10T11:42:41.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:41.656 7fe28f7fe700 1 -- 172.21.2.112:0/1662288873 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe27c00c740 con 0x7fe27c009590 2021-02-10T11:42:41.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:41.656 7fe28ffff700 1 --2- 172.21.2.112:0/1662288873 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe27c006770 0x7fe27c008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:41.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:41.656 7fe294df2700 1 --2- 172.21.2.112:0/1662288873 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe27c003aa0 0x7fe27c005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:41.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:41.656 7fe2955f3700 1 --2- 172.21.2.112:0/1662288873 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe27c009590 0x7fe27c00bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:41.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:41.656 7fe28ffff700 1 -- 172.21.2.112:0/1662288873 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe27c009590 msgr2=0x7fe27c00bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:41.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:41.656 7fe28ffff700 1 --2- 172.21.2.112:0/1662288873 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe27c009590 0x7fe27c00bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:41.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:41.656 7fe28ffff700 1 -- 172.21.2.112:0/1662288873 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe27c003aa0 msgr2=0x7fe27c005ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:41.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:41.656 7fe28ffff700 1 --2- 172.21.2.112:0/1662288873 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe27c003aa0 0x7fe27c005ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:41.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:41.656 7fe28ffff700 1 -- 172.21.2.112:0/1662288873 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe27c050980 con 0x7fe27c006770 2021-02-10T11:42:41.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:41.656 7fe2955f3700 1 -- 172.21.2.112:0/1662288873 reap_dead start 2021-02-10T11:42:41.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:41.656 7fe28ffff700 1 --2- 172.21.2.112:0/1662288873 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe27c006770 0x7fe27c008c90 secure :-1 s=READY pgs=2370 cs=0 l=1 rev1=1 rx=0x7fe280006fc0 tx=0x7fe280009060).ready entity=mon.0 client_cookie=78b50f820b47c5a3 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:41.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:41.657 7fe28f7fe700 1 -- 172.21.2.112:0/1662288873 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe28000c420 con 0x7fe27c006770 2021-02-10T11:42:41.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:41.657 7fe28f7fe700 1 -- 172.21.2.112:0/1662288873 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe280009a70 con 0x7fe27c006770 2021-02-10T11:42:41.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:41.657 7fe28f7fe700 1 -- 172.21.2.112:0/1662288873 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe28000c9a0 con 0x7fe27c006770 2021-02-10T11:42:41.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:41.657 7fe296fd6700 1 -- 172.21.2.112:0/1662288873 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe27c006770 msgr2=0x7fe27c008c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:41.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:41.657 7fe296fd6700 1 --2- 172.21.2.112:0/1662288873 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe27c006770 0x7fe27c008c90 secure :-1 s=READY pgs=2370 cs=0 l=1 rev1=1 rx=0x7fe280006fc0 tx=0x7fe280009060).stop 2021-02-10T11:42:41.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:41.657 7fe296fd6700 1 -- 172.21.2.112:0/1662288873 shutdown_connections 2021-02-10T11:42:41.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:41.657 7fe296fd6700 1 --2- 172.21.2.112:0/1662288873 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe27c006770 0x7fe27c008c90 unknown :-1 s=CLOSED pgs=2370 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:41.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:41.657 7fe296fd6700 1 -- 172.21.2.112:0/1662288873 >> 172.21.2.112:0/1662288873 conn(0x7fe2900ce430 msgr2=0x7fe2900d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:41.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:41.657 7fe296fd6700 1 -- 172.21.2.112:0/1662288873 shutdown_connections 2021-02-10T11:42:41.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:41.657 7fe296fd6700 1 -- 172.21.2.112:0/1662288873 wait complete. 2021-02-10T11:42:41.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:41.672 7fe296fd6700 1 Processor -- start 2021-02-10T11:42:41.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:41.672 7fe296fd6700 1 -- start start 2021-02-10T11:42:41.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:41.673 7fe296fd6700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe290188400 0x7fe290188800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:41.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:41.673 7fe296fd6700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe290188e20 0x7fe290189220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:41.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:41.673 7fe296fd6700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe29009cbc0 0x7fe29009cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:41.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:41.673 7fe296fd6700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe290187e80 con 0x7fe290188e20 2021-02-10T11:42:41.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:41.673 7fe296fd6700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2900b27c0 con 0x7fe29009cbc0 2021-02-10T11:42:41.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:41.673 7fe296fd6700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2900b29a0 con 0x7fe290188400 2021-02-10T11:42:41.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:41.673 7fe28ffff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe290188e20 0x7fe290189220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:41.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:41.673 7fe28ffff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe290188e20 0x7fe290189220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:56472/0 (socket says 172.21.2.112:56472) 2021-02-10T11:42:41.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:41.673 7fe28ffff700 1 -- 172.21.2.112:0/424273412 learned_addr learned my addr 172.21.2.112:0/424273412 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:42:41.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:41.673 7fe28ffff700 1 -- 172.21.2.112:0/424273412 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe290188400 msgr2=0x7fe290188800 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:42:41.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:41.673 7fe28ffff700 1 --2- 172.21.2.112:0/424273412 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe290188400 0x7fe290188800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:41.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:41.673 7fe28ffff700 1 -- 172.21.2.112:0/424273412 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe29009cbc0 msgr2=0x7fe29009cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:41.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:41.673 7fe28ffff700 1 --2- 172.21.2.112:0/424273412 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe29009cbc0 0x7fe29009cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:41.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:41.673 7fe28ffff700 1 -- 172.21.2.112:0/424273412 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe29009d5e0 con 0x7fe290188e20 2021-02-10T11:42:41.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:41.673 7fe28ffff700 1 --2- 172.21.2.112:0/424273412 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe290188e20 0x7fe290189220 secure :-1 s=READY pgs=2371 cs=0 l=1 rev1=1 rx=0x7fe28000b8c0 tx=0x7fe28000bc60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:41.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:41.674 7fe28d7fa700 1 -- 172.21.2.112:0/424273412 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe280006030 con 0x7fe290188e20 2021-02-10T11:42:41.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:41.674 7fe28d7fa700 1 -- 172.21.2.112:0/424273412 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2800062b0 con 0x7fe290188e20 2021-02-10T11:42:41.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:41.674 7fe28d7fa700 1 -- 172.21.2.112:0/424273412 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe280005660 con 0x7fe290188e20 2021-02-10T11:42:41.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:41.674 7fe296fd6700 1 -- 172.21.2.112:0/424273412 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe29009d5e0 con 0x7fe290188e20 2021-02-10T11:42:41.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:41.674 7fe296fd6700 1 -- 172.21.2.112:0/424273412 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe29009d5e0 con 0x7fe290188e20 2021-02-10T11:42:41.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:41.674 7fe28d7fa700 1 -- 172.21.2.112:0/424273412 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe280006490 con 0x7fe290188e20 2021-02-10T11:42:41.678 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:41.674 7fe28d7fa700 1 --2- 172.21.2.112:0/424273412 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe2740186c0 0x7fe27401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:41.678 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:41.674 7fe28d7fa700 1 -- 172.21.2.112:0/424273412 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fe280032950 con 0x7fe290188e20 2021-02-10T11:42:41.678 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:41.675 7fe294df2700 1 --2- 172.21.2.112:0/424273412 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe2740186c0 0x7fe27401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:41.678 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:41.675 7fe294df2700 1 --2- 172.21.2.112:0/424273412 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe2740186c0 0x7fe27401abc0 crc :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:41.704 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:41.705 7fe296fd6700 1 -- 172.21.2.112:0/424273412 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe29009db70 con 0x7fe290188e20 2021-02-10T11:42:41.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:41.707 7fe28d7fa700 1 -- 172.21.2.112:0/424273412 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe280006030 con 0x7fe290188e20 2021-02-10T11:42:41.842 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:41.843 7fe296fd6700 1 -- 172.21.2.112:0/424273412 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fe290082a70 con 0x7fe290188e20 2021-02-10T11:42:41.897 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:41.898 7fe28d7fa700 1 -- 172.21.2.112:0/424273412 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v106) v1 ==== 136+0+0 (secure 0 0 0) 0x7fe280006490 con 0x7fe290188e20 2021-02-10T11:42:41.909 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:41.910 7fe296fd6700 1 -- 172.21.2.112:0/424273412 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe2740186c0 msgr2=0x7fe27401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:41.909 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:41.910 7fe296fd6700 1 --2- 172.21.2.112:0/424273412 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe2740186c0 0x7fe27401abc0 crc :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:41.909 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:41.910 7fe296fd6700 1 -- 172.21.2.112:0/424273412 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe290188e20 msgr2=0x7fe290189220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:41.909 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:41.910 7fe296fd6700 1 --2- 172.21.2.112:0/424273412 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe290188e20 0x7fe290189220 secure :-1 s=READY pgs=2371 cs=0 l=1 rev1=1 rx=0x7fe28000b8c0 tx=0x7fe28000bc60).stop 2021-02-10T11:42:41.910 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:41.910 7fe296fd6700 1 -- 172.21.2.112:0/424273412 shutdown_connections 2021-02-10T11:42:41.910 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:41.910 7fe296fd6700 1 --2- 172.21.2.112:0/424273412 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe2740186c0 0x7fe27401abc0 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:41.910 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:41.910 7fe296fd6700 1 --2- 172.21.2.112:0/424273412 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe29009cbc0 0x7fe29009cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:41.910 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:41.910 7fe296fd6700 1 --2- 172.21.2.112:0/424273412 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe290188e20 0x7fe290189220 unknown :-1 s=CLOSED pgs=2371 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:41.910 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:41.910 7fe296fd6700 1 --2- 172.21.2.112:0/424273412 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe290188400 0x7fe290188800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:41.910 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:41.910 7fe296fd6700 1 -- 172.21.2.112:0/424273412 >> 172.21.2.112:0/424273412 conn(0x7fe2900c2cd0 msgr2=0x7fe2900b5950 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:41.910 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:41.910 7fe296fd6700 1 -- 172.21.2.112:0/424273412 shutdown_connections 2021-02-10T11:42:41.910 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:41.910 7fe296fd6700 1 -- 172.21.2.112:0/424273412 wait complete. 2021-02-10T11:42:41.917 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs volume ls 2021-02-10T11:42:42.345 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.329 7ff5dbc57700 1 Processor -- start 2021-02-10T11:42:42.345 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.329 7ff5dbc57700 1 -- start start 2021-02-10T11:42:42.345 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.330 7ff5dbc57700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff5d40d1870 0x7ff5d41b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:42.345 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.330 7ff5dbc57700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7ff5d41b0a10 0x7ff5d40cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:42.345 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.330 7ff5dbc57700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5d40f0b10 con 0x7ff5d40d1c70 2021-02-10T11:42:42.345 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.330 7ff5dbc57700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5d40fe230 con 0x7ff5d41b0a10 2021-02-10T11:42:42.346 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.330 7ff5dbc57700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5d40e5ed0 con 0x7ff5d40d1870 2021-02-10T11:42:42.346 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.330 7ff5da274700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7ff5d40d1c70 0x7ff5d40cf950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:55944/0 (socket says 172.21.2.112:55944) 2021-02-10T11:42:42.346 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.330 7ff5da274700 1 -- 172.21.2.112:0/3083461226 learned_addr learned my addr 172.21.2.112:0/3083461226 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:42:42.346 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.330 7ff5d9272700 1 --2- 172.21.2.112:0/3083461226 >> v2:172.21.2.112:3300/0 conn(0x7ff5d41b0a10 0x7ff5d40cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:42.346 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.330 7ff5d9a73700 1 --2- 172.21.2.112:0/3083461226 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff5d40d1870 0x7ff5d41b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:42.346 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.330 7ff5d8a71700 1 -- 172.21.2.112:0/3083461226 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4202220610 0 0) 0x7ff5cc000d30 con 0x7ff5d40d1c70 2021-02-10T11:42:42.346 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.330 7ff5d9272700 1 -- 172.21.2.112:0/3083461226 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff5d40d1870 msgr2=0x7ff5d41b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:42.346 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.330 7ff5d9272700 1 --2- 172.21.2.112:0/3083461226 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff5d40d1870 0x7ff5d41b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:42.346 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.330 7ff5d9272700 1 -- 172.21.2.112:0/3083461226 >> v1:172.21.2.112:6789/0 conn(0x7ff5d40d1c70 legacy=0x7ff5d40cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:42.346 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.330 7ff5d9272700 1 -- 172.21.2.112:0/3083461226 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5d40f0650 con 0x7ff5d41b0a10 2021-02-10T11:42:42.346 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.331 7ff5d9272700 1 --2- 172.21.2.112:0/3083461226 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff5d41b0a10 0x7ff5d40cf330 secure :-1 s=READY pgs=2373 cs=0 l=1 rev1=1 rx=0x7ff5d0006650 tx=0x7ff5d0009a80).ready entity=mon.0 client_cookie=1ec5a79f89e9b578 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:42.346 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.331 7ff5d8a71700 1 -- 172.21.2.112:0/3083461226 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff5d00009b0 con 0x7ff5d41b0a10 2021-02-10T11:42:42.347 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.331 7ff5d8a71700 1 -- 172.21.2.112:0/3083461226 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5d000c0c0 con 0x7ff5d41b0a10 2021-02-10T11:42:42.347 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.331 7ff5d8a71700 1 -- 172.21.2.112:0/3083461226 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff5d000c520 con 0x7ff5d41b0a10 2021-02-10T11:42:42.347 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.331 7ff5dbc57700 1 -- 172.21.2.112:0/3083461226 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff5d41b0a10 msgr2=0x7ff5d40cf330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:42.347 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.331 7ff5dbc57700 1 --2- 172.21.2.112:0/3083461226 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff5d41b0a10 0x7ff5d40cf330 secure :-1 s=READY pgs=2373 cs=0 l=1 rev1=1 rx=0x7ff5d0006650 tx=0x7ff5d0009a80).stop 2021-02-10T11:42:42.347 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.332 7ff5dbc57700 1 -- 172.21.2.112:0/3083461226 shutdown_connections 2021-02-10T11:42:42.347 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.332 7ff5dbc57700 1 --2- 172.21.2.112:0/3083461226 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff5d41b0a10 0x7ff5d40cf330 unknown :-1 s=CLOSED pgs=2373 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:42.347 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.332 7ff5dbc57700 1 --2- 172.21.2.112:0/3083461226 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff5d40d1870 0x7ff5d41b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:42.347 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.332 7ff5dbc57700 1 -- 172.21.2.112:0/3083461226 >> 172.21.2.112:0/3083461226 conn(0x7ff5d40ce450 msgr2=0x7ff5d40d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:42.347 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.332 7ff5dbc57700 1 -- 172.21.2.112:0/3083461226 shutdown_connections 2021-02-10T11:42:42.347 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.332 7ff5dbc57700 1 -- 172.21.2.112:0/3083461226 wait complete. 2021-02-10T11:42:42.349 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.346 7ff5dbc57700 1 Processor -- start 2021-02-10T11:42:42.349 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.346 7ff5dbc57700 1 -- start start 2021-02-10T11:42:42.349 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.347 7ff5dbc57700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff5d41a06a0 0x7ff5d409f380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:42.349 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.347 7ff5dbc57700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff5d409f9a0 0x7ff5d408b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:42.349 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.347 7ff5dbc57700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff5d408bb80 0x7ff5d408bf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:42.349 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.347 7ff5dbc57700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5d408c780 con 0x7ff5d409f9a0 2021-02-10T11:42:42.349 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.347 7ff5dbc57700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5d409fda0 con 0x7ff5d41a06a0 2021-02-10T11:42:42.349 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.347 7ff5dbc57700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5d408cd00 con 0x7ff5d408bb80 2021-02-10T11:42:42.349 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.347 7ff5d9272700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff5d409f9a0 0x7ff5d408b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:42.349 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.347 7ff5d9272700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff5d409f9a0 0x7ff5d408b560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:56486/0 (socket says 172.21.2.112:56486) 2021-02-10T11:42:42.349 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.347 7ff5d9272700 1 -- 172.21.2.112:0/1795218217 learned_addr learned my addr 172.21.2.112:0/1795218217 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:42:42.350 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.348 7ff5da274700 1 --2- 172.21.2.112:0/1795218217 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff5d408bb80 0x7ff5d408bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:42.350 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.348 7ff5d9a73700 1 --2- 172.21.2.112:0/1795218217 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff5d41a06a0 0x7ff5d409f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:42.350 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.348 7ff5d9272700 1 -- 172.21.2.112:0/1795218217 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff5d408bb80 msgr2=0x7ff5d408bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:42.350 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.348 7ff5d9272700 1 --2- 172.21.2.112:0/1795218217 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff5d408bb80 0x7ff5d408bf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:42.350 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.348 7ff5d9272700 1 -- 172.21.2.112:0/1795218217 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff5d41a06a0 msgr2=0x7ff5d409f380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:42.350 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.348 7ff5d9272700 1 --2- 172.21.2.112:0/1795218217 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff5d41a06a0 0x7ff5d409f380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:42.350 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.348 7ff5d9272700 1 -- 172.21.2.112:0/1795218217 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5d408cf20 con 0x7ff5d409f9a0 2021-02-10T11:42:42.350 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.348 7ff5d9a73700 1 --2- 172.21.2.112:0/1795218217 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff5d41a06a0 0x7ff5d409f380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:42:42.350 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.348 7ff5d9272700 1 --2- 172.21.2.112:0/1795218217 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff5d409f9a0 0x7ff5d408b560 secure :-1 s=READY pgs=2374 cs=0 l=1 rev1=1 rx=0x7ff5d0009630 tx=0x7ff5d000b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:42.350 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.348 7ff5c67fc700 1 -- 172.21.2.112:0/1795218217 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff5d00093a0 con 0x7ff5d409f9a0 2021-02-10T11:42:42.350 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.348 7ff5c67fc700 1 -- 172.21.2.112:0/1795218217 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5d000bd70 con 0x7ff5d409f9a0 2021-02-10T11:42:42.350 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.348 7ff5c67fc700 1 -- 172.21.2.112:0/1795218217 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff5d00054d0 con 0x7ff5d409f9a0 2021-02-10T11:42:42.351 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.348 7ff5dbc57700 1 -- 172.21.2.112:0/1795218217 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5d408cf20 con 0x7ff5d409f9a0 2021-02-10T11:42:42.351 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.348 7ff5dbc57700 1 -- 172.21.2.112:0/1795218217 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5d408cf20 con 0x7ff5d409f9a0 2021-02-10T11:42:42.351 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.349 7ff5c67fc700 1 -- 172.21.2.112:0/1795218217 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff5d00054f0 con 0x7ff5d409f9a0 2021-02-10T11:42:42.351 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.349 7ff5c67fc700 1 --2- 172.21.2.112:0/1795218217 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff5bc0186c0 0x7ff5bc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:42.351 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.349 7ff5c67fc700 1 -- 172.21.2.112:0/1795218217 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5573+0+0 (secure 0 0 0) 0x7ff5d0032550 con 0x7ff5d409f9a0 2021-02-10T11:42:42.351 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.349 7ff5d9a73700 1 --2- 172.21.2.112:0/1795218217 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff5bc0186c0 0x7ff5bc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:42.351 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.350 7ff5d9a73700 1 --2- 172.21.2.112:0/1795218217 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff5bc0186c0 0x7ff5bc01abc0 crc :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:42.379 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.380 7ff5dbc57700 1 -- 172.21.2.112:0/1795218217 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5d40ebc50 con 0x7ff5d409f9a0 2021-02-10T11:42:42.381 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.383 7ff5c67fc700 1 -- 172.21.2.112:0/1795218217 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff5d0021e90 con 0x7ff5d409f9a0 2021-02-10T11:42:42.517 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.517 7ff5dbc57700 1 -- 172.21.2.112:0/1795218217 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs volume ls", "target": ["mgr", ""]}) v1 -- 0x7ff5d40f0b10 con 0x7ff5bc0186c0 2021-02-10T11:42:42.517 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.518 7ff5c67fc700 1 -- 172.21.2.112:0/1795218217 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+41 (crc 0 0 0) 0x7ff5c8006ae0 con 0x7ff5bc0186c0 2021-02-10T11:42:42.518 INFO:teuthology.orchestra.run.gibba012.stdout:[ 2021-02-10T11:42:42.518 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:42.518 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "cephfs" 2021-02-10T11:42:42.519 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:42:42.519 INFO:teuthology.orchestra.run.gibba012.stdout:] 2021-02-10T11:42:42.520 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.520 7ff5dbc57700 1 -- 172.21.2.112:0/1795218217 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff5bc0186c0 msgr2=0x7ff5bc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:42.520 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.520 7ff5dbc57700 1 --2- 172.21.2.112:0/1795218217 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff5bc0186c0 0x7ff5bc01abc0 crc :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:42.520 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.520 7ff5dbc57700 1 -- 172.21.2.112:0/1795218217 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff5d409f9a0 msgr2=0x7ff5d408b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:42.520 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.520 7ff5dbc57700 1 --2- 172.21.2.112:0/1795218217 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff5d409f9a0 0x7ff5d408b560 secure :-1 s=READY pgs=2374 cs=0 l=1 rev1=1 rx=0x7ff5d0009630 tx=0x7ff5d000b980).stop 2021-02-10T11:42:42.520 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.520 7ff5dbc57700 1 -- 172.21.2.112:0/1795218217 shutdown_connections 2021-02-10T11:42:42.520 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.520 7ff5dbc57700 1 --2- 172.21.2.112:0/1795218217 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff5bc0186c0 0x7ff5bc01abc0 unknown :-1 s=CLOSED pgs=1083 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:42.520 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.520 7ff5dbc57700 1 --2- 172.21.2.112:0/1795218217 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff5d408bb80 0x7ff5d408bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:42.520 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.520 7ff5dbc57700 1 --2- 172.21.2.112:0/1795218217 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff5d409f9a0 0x7ff5d408b560 unknown :-1 s=CLOSED pgs=2374 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:42.520 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.520 7ff5dbc57700 1 --2- 172.21.2.112:0/1795218217 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff5d41a06a0 0x7ff5d409f380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:42.520 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.520 7ff5dbc57700 1 -- 172.21.2.112:0/1795218217 >> 172.21.2.112:0/1795218217 conn(0x7ff5d40b83b0 msgr2=0x7ff5d40a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:42.520 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.520 7ff5dbc57700 1 -- 172.21.2.112:0/1795218217 shutdown_connections 2021-02-10T11:42:42.520 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.520 7ff5dbc57700 1 -- 172.21.2.112:0/1795218217 wait complete. 2021-02-10T11:42:42.527 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:42:42.962 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.945 7fea43545700 1 Processor -- start 2021-02-10T11:42:42.962 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.945 7fea43545700 1 -- start start 2021-02-10T11:42:42.962 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.946 7fea43545700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fea3c0d1870 0x7fea3c1b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:42.962 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.946 7fea43545700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fea3c1b0ae0 0x7fea3c108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:42.962 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.946 7fea43545700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea3c109a60 con 0x7fea3c0d1c70 2021-02-10T11:42:42.963 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.946 7fea43545700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea3c0e5ef0 con 0x7fea3c1b0ae0 2021-02-10T11:42:42.963 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.946 7fea43545700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea3c109750 con 0x7fea3c0d1870 2021-02-10T11:42:42.963 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.946 7fea40b60700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fea3c1b0ae0 0x7fea3c108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:42.963 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.946 7fea41361700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fea3c0d1870 0x7fea3c1b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:42.963 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.946 7fea40b60700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fea3c1b0ae0 0x7fea3c108940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3300/0 says I am v2:172.21.2.112:58772/0 (socket says 172.21.2.112:58772) 2021-02-10T11:42:42.963 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.946 7fea40b60700 1 -- 172.21.2.112:0/2863800596 learned_addr learned my addr 172.21.2.112:0/2863800596 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:42:42.963 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.946 7fea41b62700 1 --1- 172.21.2.112:0/2863800596 >> v1:172.21.2.120:6789/0 conn(0x7fea3c0d1c70 0x7fea3c108f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:49512/0 (socket says 172.21.2.112:49512) 2021-02-10T11:42:42.963 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.947 7fea40b60700 1 -- 172.21.2.112:0/2863800596 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fea3c0d1870 msgr2=0x7fea3c1b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:42.963 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.947 7fea40b60700 1 --2- 172.21.2.112:0/2863800596 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fea3c0d1870 0x7fea3c1b04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:42.963 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.947 7fea40b60700 1 -- 172.21.2.112:0/2863800596 >> v1:172.21.2.120:6789/0 conn(0x7fea3c0d1c70 legacy=0x7fea3c108f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:42.963 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.947 7fea40b60700 1 -- 172.21.2.112:0/2863800596 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea3c088070 con 0x7fea3c1b0ae0 2021-02-10T11:42:42.963 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.947 7fea41361700 1 --2- 172.21.2.112:0/2863800596 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fea3c0d1870 0x7fea3c1b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:42:42.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.947 7fea40b60700 1 --2- 172.21.2.112:0/2863800596 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fea3c1b0ae0 0x7fea3c108940 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 rx=0x7fea38006700 tx=0x7fea38009b80).ready entity=mon.1 client_cookie=afcc783972a2eea3 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:42.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.947 7fea2ffff700 1 -- 172.21.2.112:0/2863800596 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fea3800c0c0 con 0x7fea3c1b0ae0 2021-02-10T11:42:42.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.947 7fea2ffff700 1 -- 172.21.2.112:0/2863800596 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea3800c440 con 0x7fea3c1b0ae0 2021-02-10T11:42:42.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.947 7fea2ffff700 1 -- 172.21.2.112:0/2863800596 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fea3800c8a0 con 0x7fea3c1b0ae0 2021-02-10T11:42:42.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.947 7fea43545700 1 -- 172.21.2.112:0/2863800596 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fea3c1b0ae0 msgr2=0x7fea3c108940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:42.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.947 7fea43545700 1 --2- 172.21.2.112:0/2863800596 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fea3c1b0ae0 0x7fea3c108940 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 rx=0x7fea38006700 tx=0x7fea38009b80).stop 2021-02-10T11:42:42.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.948 7fea43545700 1 -- 172.21.2.112:0/2863800596 shutdown_connections 2021-02-10T11:42:42.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.948 7fea43545700 1 --2- 172.21.2.112:0/2863800596 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fea3c1b0ae0 0x7fea3c108940 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:42.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.948 7fea43545700 1 --2- 172.21.2.112:0/2863800596 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fea3c0d1870 0x7fea3c1b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:42.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.948 7fea43545700 1 -- 172.21.2.112:0/2863800596 >> 172.21.2.112:0/2863800596 conn(0x7fea3c0ce420 msgr2=0x7fea3c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:42.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.948 7fea43545700 1 -- 172.21.2.112:0/2863800596 shutdown_connections 2021-02-10T11:42:42.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.948 7fea43545700 1 -- 172.21.2.112:0/2863800596 wait complete. 2021-02-10T11:42:42.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.963 7fea43545700 1 Processor -- start 2021-02-10T11:42:42.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.963 7fea43545700 1 -- start start 2021-02-10T11:42:42.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.963 7fea43545700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fea3c0b1ba0 0x7fea3c1a78d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:42.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.963 7fea43545700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fea3c1a7ef0 0x7fea3c0a6a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:42.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.963 7fea43545700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fea3c0a7060 0x7fea3c0b2920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:42.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.963 7fea43545700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea3c190c20 con 0x7fea3c0b1ba0 2021-02-10T11:42:42.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.963 7fea43545700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea3c190710 con 0x7fea3c1a7ef0 2021-02-10T11:42:42.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.963 7fea43545700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea3c0b1fa0 con 0x7fea3c0a7060 2021-02-10T11:42:42.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.964 7fea41361700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fea3c0b1ba0 0x7fea3c1a78d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:42.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.964 7fea41361700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fea3c0b1ba0 0x7fea3c1a78d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:56496/0 (socket says 172.21.2.112:56496) 2021-02-10T11:42:42.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.964 7fea41361700 1 -- 172.21.2.112:0/1522197307 learned_addr learned my addr 172.21.2.112:0/1522197307 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:42:42.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.964 7fea41361700 1 -- 172.21.2.112:0/1522197307 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fea3c0a7060 msgr2=0x7fea3c0b2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:42.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.964 7fea41361700 1 --2- 172.21.2.112:0/1522197307 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fea3c0a7060 0x7fea3c0b2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:42.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.964 7fea41361700 1 -- 172.21.2.112:0/1522197307 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fea3c1a7ef0 msgr2=0x7fea3c0a6a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:42.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.964 7fea41361700 1 --2- 172.21.2.112:0/1522197307 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fea3c1a7ef0 0x7fea3c0a6a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:42.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.964 7fea41361700 1 -- 172.21.2.112:0/1522197307 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea3c0a7460 con 0x7fea3c0b1ba0 2021-02-10T11:42:42.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.964 7fea41361700 1 --2- 172.21.2.112:0/1522197307 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fea3c0b1ba0 0x7fea3c1a78d0 secure :-1 s=READY pgs=2375 cs=0 l=1 rev1=1 rx=0x7fea3000b580 tx=0x7fea3000b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:42.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.964 7fea2dffb700 1 -- 172.21.2.112:0/1522197307 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fea30006bb0 con 0x7fea3c0b1ba0 2021-02-10T11:42:42.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.964 7fea2dffb700 1 -- 172.21.2.112:0/1522197307 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea3000bd10 con 0x7fea3c0b1ba0 2021-02-10T11:42:42.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.964 7fea2dffb700 1 -- 172.21.2.112:0/1522197307 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fea300022b0 con 0x7fea3c0b1ba0 2021-02-10T11:42:42.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.964 7fea43545700 1 -- 172.21.2.112:0/1522197307 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea3c0a7460 con 0x7fea3c0b1ba0 2021-02-10T11:42:42.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.964 7fea43545700 1 -- 172.21.2.112:0/1522197307 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea3c0b3230 con 0x7fea3c0b1ba0 2021-02-10T11:42:42.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.965 7fea2dffb700 1 -- 172.21.2.112:0/1522197307 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fea30006b00 con 0x7fea3c0b1ba0 2021-02-10T11:42:42.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.965 7fea2dffb700 1 --2- 172.21.2.112:0/1522197307 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fea1c0186c0 0x7fea1c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:42.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.965 7fea2dffb700 1 -- 172.21.2.112:0/1522197307 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fea300323c0 con 0x7fea3c0b1ba0 2021-02-10T11:42:42.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.965 7fea40b60700 1 --2- 172.21.2.112:0/1522197307 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fea1c0186c0 0x7fea1c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:42.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.966 7fea40b60700 1 --2- 172.21.2.112:0/1522197307 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fea1c0186c0 0x7fea1c01abc0 crc :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:42.994 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.995 7fea43545700 1 -- 172.21.2.112:0/1522197307 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea3c082980 con 0x7fea3c0b1ba0 2021-02-10T11:42:42.997 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:42.998 7fea2dffb700 1 -- 172.21.2.112:0/1522197307 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fea30000940 con 0x7fea3c0b1ba0 2021-02-10T11:42:43.147 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:43.148 7fea43545700 1 -- 172.21.2.112:0/1522197307 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fea3c082980 con 0x7fea3c0b1ba0 2021-02-10T11:42:43.147 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:43.148 7fea2dffb700 1 -- 172.21.2.112:0/1522197307 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v32)=0 v32) v1 ==== 122+0+0 (secure 0 0 0) 0x7fea30006b00 con 0x7fea3c0b1ba0 2021-02-10T11:42:43.149 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:43.150 7fea43545700 1 -- 172.21.2.112:0/1522197307 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fea1c0186c0 msgr2=0x7fea1c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:43.149 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:43.150 7fea43545700 1 --2- 172.21.2.112:0/1522197307 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fea1c0186c0 0x7fea1c01abc0 crc :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:43.149 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:43.150 7fea43545700 1 -- 172.21.2.112:0/1522197307 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fea3c0b1ba0 msgr2=0x7fea3c1a78d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:43.149 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:43.150 7fea43545700 1 --2- 172.21.2.112:0/1522197307 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fea3c0b1ba0 0x7fea3c1a78d0 secure :-1 s=READY pgs=2375 cs=0 l=1 rev1=1 rx=0x7fea3000b580 tx=0x7fea3000b920).stop 2021-02-10T11:42:43.150 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:43.150 7fea43545700 1 -- 172.21.2.112:0/1522197307 shutdown_connections 2021-02-10T11:42:43.150 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:43.150 7fea43545700 1 --2- 172.21.2.112:0/1522197307 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fea1c0186c0 0x7fea1c01abc0 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:43.150 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:43.150 7fea43545700 1 --2- 172.21.2.112:0/1522197307 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fea3c1a7ef0 0x7fea3c0a6a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:43.150 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:43.150 7fea43545700 1 --2- 172.21.2.112:0/1522197307 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fea3c0a7060 0x7fea3c0b2920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:43.150 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:43.150 7fea43545700 1 --2- 172.21.2.112:0/1522197307 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fea3c0b1ba0 0x7fea3c1a78d0 unknown :-1 s=CLOSED pgs=2375 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:43.150 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:43.150 7fea43545700 1 -- 172.21.2.112:0/1522197307 >> 172.21.2.112:0/1522197307 conn(0x7fea3c0b5dc0 msgr2=0x7fea3c095960 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:43.150 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:43.150 7fea43545700 1 -- 172.21.2.112:0/1522197307 shutdown_connections 2021-02-10T11:42:43.150 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:43.150 7fea43545700 1 -- 172.21.2.112:0/1522197307 wait complete. 2021-02-10T11:42:43.157 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000918327 2021-02-10T11:42:43.590 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:43.573 7f4995af1700 1 Processor -- start 2021-02-10T11:42:43.590 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:43.573 7f4995af1700 1 -- start start 2021-02-10T11:42:43.590 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:43.574 7f4995af1700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f49900d1870 0x7f49901b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:43.590 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:43.574 7f4995af1700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f49901b0ae0 0x7f4990108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:43.590 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:43.574 7f4995af1700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49900fe1c0 con 0x7f49900d1c70 2021-02-10T11:42:43.590 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:43.574 7f4995af1700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4990109a60 con 0x7f49900d1870 2021-02-10T11:42:43.590 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:43.574 7f4995af1700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4990109750 con 0x7f49901b0ae0 2021-02-10T11:42:43.591 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:43.574 7f498ffff700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f49900d1c70 0x7f4990108f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:55966/0 (socket says 172.21.2.112:55966) 2021-02-10T11:42:43.591 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:43.574 7f498ffff700 1 -- 172.21.2.112:0/3491055778 learned_addr learned my addr 172.21.2.112:0/3491055778 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:42:43.591 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:43.574 7f498e7fc700 1 -- 172.21.2.112:0/3491055778 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3466143052 0 0) 0x7f4988001500 con 0x7f49900d1c70 2021-02-10T11:42:43.591 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:43.574 7f498e7fc700 1 -- 172.21.2.112:0/3491055778 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f497c002fe0 con 0x7f49900d1c70 2021-02-10T11:42:43.591 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:43.575 7f498f7fe700 1 --2- 172.21.2.112:0/3491055778 >> v2:172.21.2.120:3300/0 conn(0x7f49900d1870 0x7f49901b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:43.591 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:43.575 7f498effd700 1 --2- 172.21.2.112:0/3491055778 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f49901b0ae0 0x7f4990108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:43.591 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:43.575 7f498e7fc700 1 -- 172.21.2.112:0/3491055778 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3543383522 0 0) 0x7f4988001460 con 0x7f49900d1c70 2021-02-10T11:42:43.591 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:43.575 7f498e7fc700 1 -- 172.21.2.112:0/3491055778 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f49901b0ae0 msgr2=0x7f4990108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:43.591 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:43.575 7f498e7fc700 1 --2- 172.21.2.112:0/3491055778 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f49901b0ae0 0x7f4990108940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:43.591 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:43.575 7f498e7fc700 1 -- 172.21.2.112:0/3491055778 >> v2:172.21.2.120:3300/0 conn(0x7f49900d1870 msgr2=0x7f49901b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:43.591 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:43.575 7f498e7fc700 1 --2- 172.21.2.112:0/3491055778 >> v2:172.21.2.120:3300/0 conn(0x7f49900d1870 0x7f49901b04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:43.591 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:43.575 7f498e7fc700 1 -- 172.21.2.112:0/3491055778 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4990088070 con 0x7f49900d1c70 2021-02-10T11:42:43.591 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:43.575 7f498e7fc700 1 -- 172.21.2.112:0/3491055778 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f4988001980 con 0x7f49900d1c70 2021-02-10T11:42:43.592 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:43.575 7f498f7fe700 1 --2- 172.21.2.112:0/3491055778 >> v2:172.21.2.120:3300/0 conn(0x7f49900d1870 0x7f49901b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:42:43.592 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:43.575 7f498e7fc700 1 -- 172.21.2.112:0/3491055778 >> v1:172.21.2.112:6789/0 conn(0x7f49900d1c70 legacy=0x7f4990108f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:43.592 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:43.575 7f498e7fc700 1 --2- 172.21.2.112:0/3491055778 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f497c003a60 0x7f497c005e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:43.592 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:43.575 7f498e7fc700 1 --2- 172.21.2.112:0/3491055778 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f497c006730 0x7f497c008c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:43.592 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:43.575 7f498e7fc700 1 --2- 172.21.2.112:0/3491055778 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f497c0094f0 0x7f497c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:43.592 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:43.575 7f498e7fc700 1 -- 172.21.2.112:0/3491055778 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f497c0502d0 con 0x7f497c0094f0 2021-02-10T11:42:43.592 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:43.575 7f498e7fc700 1 -- 172.21.2.112:0/3491055778 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f497c0504b0 con 0x7f497c006730 2021-02-10T11:42:43.592 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:43.575 7f498e7fc700 1 -- 172.21.2.112:0/3491055778 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f497c0506d0 con 0x7f497c003a60 2021-02-10T11:42:43.592 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:43.575 7f498ffff700 1 --2- 172.21.2.112:0/3491055778 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f497c0094f0 0x7f497c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:43.592 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:43.575 7f498effd700 1 --2- 172.21.2.112:0/3491055778 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f497c006730 0x7f497c008c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:43.592 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:43.575 7f498f7fe700 1 --2- 172.21.2.112:0/3491055778 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f497c003a60 0x7f497c005e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:43.592 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:43.575 7f498ffff700 1 -- 172.21.2.112:0/3491055778 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f497c003a60 msgr2=0x7f497c005e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:43.592 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:43.575 7f498ffff700 1 --2- 172.21.2.112:0/3491055778 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f497c003a60 0x7f497c005e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:43.593 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:43.575 7f498ffff700 1 -- 172.21.2.112:0/3491055778 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f497c006730 msgr2=0x7f497c008c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:43.593 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:43.575 7f498ffff700 1 --2- 172.21.2.112:0/3491055778 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f497c006730 0x7f497c008c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:43.593 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:43.575 7f498ffff700 1 -- 172.21.2.112:0/3491055778 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f497c050970 con 0x7f497c0094f0 2021-02-10T11:42:43.593 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:43.575 7f498effd700 1 --2- 172.21.2.112:0/3491055778 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f497c006730 0x7f497c008c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:42:43.593 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:43.575 7f498ffff700 1 --2- 172.21.2.112:0/3491055778 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f497c0094f0 0x7f497c00ba10 secure :-1 s=READY pgs=2377 cs=0 l=1 rev1=1 rx=0x7f4988001160 tx=0x7f498800a310).ready entity=mon.0 client_cookie=7b3e4364fc7df0db server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:43.593 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:43.575 7f498e7fc700 1 -- 172.21.2.112:0/3491055778 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4988001460 con 0x7f497c0094f0 2021-02-10T11:42:43.593 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:43.575 7f498e7fc700 1 -- 172.21.2.112:0/3491055778 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49880070c0 con 0x7f497c0094f0 2021-02-10T11:42:43.593 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:43.575 7f498e7fc700 1 -- 172.21.2.112:0/3491055778 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4988007520 con 0x7f497c0094f0 2021-02-10T11:42:43.593 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:43.576 7f4995af1700 1 -- 172.21.2.112:0/3491055778 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f497c0094f0 msgr2=0x7f497c00ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:43.593 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:43.576 7f4995af1700 1 --2- 172.21.2.112:0/3491055778 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f497c0094f0 0x7f497c00ba10 secure :-1 s=READY pgs=2377 cs=0 l=1 rev1=1 rx=0x7f4988001160 tx=0x7f498800a310).stop 2021-02-10T11:42:43.593 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:43.576 7f498ffff700 1 -- 172.21.2.112:0/3491055778 reap_dead start 2021-02-10T11:42:43.593 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:43.576 7f4995af1700 1 -- 172.21.2.112:0/3491055778 shutdown_connections 2021-02-10T11:42:43.593 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:43.576 7f4995af1700 1 -- 172.21.2.112:0/3491055778 >> 172.21.2.112:0/3491055778 conn(0x7f49900ce420 msgr2=0x7f49900d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:43.594 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:43.576 7f4995af1700 1 -- 172.21.2.112:0/3491055778 shutdown_connections 2021-02-10T11:42:43.594 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:43.576 7f4995af1700 1 -- 172.21.2.112:0/3491055778 wait complete. 2021-02-10T11:42:43.594 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:43.591 7f4995af1700 1 Processor -- start 2021-02-10T11:42:43.594 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:43.591 7f4995af1700 1 -- start start 2021-02-10T11:42:43.595 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:43.591 7f4995af1700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f499009bcc0 0x7f499009c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:43.595 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:43.591 7f4995af1700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f49900a0820 0x7f49900a0c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:43.595 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:43.591 7f4995af1700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f49900a1240 0x7f49900858d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:43.595 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:43.591 7f4995af1700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49900a66b0 con 0x7f499009bcc0 2021-02-10T11:42:43.595 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:43.591 7f4995af1700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49901a49b0 con 0x7f49900a0820 2021-02-10T11:42:43.595 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:43.591 7f4995af1700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49901a4b90 con 0x7f49900a1240 2021-02-10T11:42:43.595 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:43.591 7f498f7fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f499009bcc0 0x7f499009c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:43.595 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:43.591 7f498f7fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f499009bcc0 0x7f499009c0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:56516/0 (socket says 172.21.2.112:56516) 2021-02-10T11:42:43.596 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:43.591 7f498f7fe700 1 -- 172.21.2.112:0/3060833674 learned_addr learned my addr 172.21.2.112:0/3060833674 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:42:43.596 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:43.592 7f498f7fe700 1 -- 172.21.2.112:0/3060833674 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f49900a1240 msgr2=0x7f49900858d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:43.596 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:43.592 7f498f7fe700 1 --2- 172.21.2.112:0/3060833674 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f49900a1240 0x7f49900858d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:43.596 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:43.592 7f498f7fe700 1 -- 172.21.2.112:0/3060833674 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f49900a0820 msgr2=0x7f49900a0c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:43.596 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:43.592 7f498f7fe700 1 --2- 172.21.2.112:0/3060833674 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f49900a0820 0x7f49900a0c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:43.596 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:43.592 7f498f7fe700 1 -- 172.21.2.112:0/3060833674 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49900a1640 con 0x7f499009bcc0 2021-02-10T11:42:43.596 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:43.592 7f498f7fe700 1 --2- 172.21.2.112:0/3060833674 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f499009bcc0 0x7f499009c0c0 secure :-1 s=READY pgs=2378 cs=0 l=1 rev1=1 rx=0x7f4978006f20 tx=0x7f497800b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:43.596 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:43.592 7f4987fff700 1 -- 172.21.2.112:0/3060833674 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f497800bd80 con 0x7f499009bcc0 2021-02-10T11:42:43.596 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:43.592 7f4987fff700 1 -- 172.21.2.112:0/3060833674 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4978000c90 con 0x7f499009bcc0 2021-02-10T11:42:43.596 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:43.592 7f4987fff700 1 -- 172.21.2.112:0/3060833674 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f49780022b0 con 0x7f499009bcc0 2021-02-10T11:42:43.596 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:43.592 7f4995af1700 1 -- 172.21.2.112:0/3060833674 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49900a1640 con 0x7f499009bcc0 2021-02-10T11:42:43.596 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:43.592 7f4995af1700 1 -- 172.21.2.112:0/3060833674 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49900861e0 con 0x7f499009bcc0 2021-02-10T11:42:43.596 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:43.593 7f4987fff700 1 -- 172.21.2.112:0/3060833674 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f497800a000 con 0x7f499009bcc0 2021-02-10T11:42:43.597 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:43.593 7f4987fff700 1 --2- 172.21.2.112:0/3060833674 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f49740186c0 0x7f497401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:43.597 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:43.593 7f4987fff700 1 -- 172.21.2.112:0/3060833674 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f49780323c0 con 0x7f499009bcc0 2021-02-10T11:42:43.597 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:43.593 7f498effd700 1 --2- 172.21.2.112:0/3060833674 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f49740186c0 0x7f497401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:43.597 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:43.593 7f498effd700 1 --2- 172.21.2.112:0/3060833674 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f49740186c0 0x7f497401abc0 crc :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:43.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:43.624 7f4995af1700 1 -- 172.21.2.112:0/3060833674 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4990086a70 con 0x7f499009bcc0 2021-02-10T11:42:43.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:43.626 7f4987fff700 1 -- 172.21.2.112:0/3060833674 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4978030640 con 0x7f499009bcc0 2021-02-10T11:42:43.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:43.753 7f4995af1700 1 -- 172.21.2.112:0/3060833674 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000918327", "target": ["mgr", ""]}) v1 -- 0x7f49900861e0 con 0x7f49740186c0 2021-02-10T11:42:43.756 INFO:tasks.ceph.mgr.x.gibba020.stderr:2021-02-10 11:42:43.758 7f6382eee700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:42:43.757 INFO:tasks.ceph.mgr.x.gibba020.stderr:2021-02-10 11:42:43.758 7f6382eee700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:42:43.757 INFO:tasks.ceph.mgr.x.gibba020.stderr:2021-02-10 11:42:43.758 7f6382eee700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:42:43.757 INFO:tasks.ceph.mgr.x.gibba020.stderr:2021-02-10 11:42:43.758 7f6382eee700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:42:43.757 INFO:tasks.ceph.mgr.x.gibba020.stderr:2021-02-10 11:42:43.758 7f6382eee700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:42:43.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:43.791 7f4987fff700 1 -- 172.21.2.112:0/3060833674 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f49800009b0 con 0x7f49740186c0 2021-02-10T11:42:43.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:43.818 7f4995af1700 1 -- 172.21.2.112:0/3060833674 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f49740186c0 msgr2=0x7f497401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:43.818 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:43.818 7f4995af1700 1 --2- 172.21.2.112:0/3060833674 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f49740186c0 0x7f497401abc0 crc :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:43.818 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:43.818 7f4995af1700 1 -- 172.21.2.112:0/3060833674 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f499009bcc0 msgr2=0x7f499009c0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:43.818 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:43.818 7f4995af1700 1 --2- 172.21.2.112:0/3060833674 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f499009bcc0 0x7f499009c0c0 secure :-1 s=READY pgs=2378 cs=0 l=1 rev1=1 rx=0x7f4978006f20 tx=0x7f497800b990).stop 2021-02-10T11:42:43.818 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:43.818 7f4995af1700 1 -- 172.21.2.112:0/3060833674 shutdown_connections 2021-02-10T11:42:43.818 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:43.818 7f4995af1700 1 --2- 172.21.2.112:0/3060833674 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f49740186c0 0x7f497401abc0 unknown :-1 s=CLOSED pgs=1085 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:43.818 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:43.818 7f4995af1700 1 --2- 172.21.2.112:0/3060833674 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f49900a0820 0x7f49900a0c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:43.818 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:43.818 7f4995af1700 1 --2- 172.21.2.112:0/3060833674 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f49900a1240 0x7f49900858d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:43.818 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:43.818 7f4995af1700 1 --2- 172.21.2.112:0/3060833674 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f499009bcc0 0x7f499009c0c0 unknown :-1 s=CLOSED pgs=2378 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:43.818 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:43.818 7f4995af1700 1 -- 172.21.2.112:0/3060833674 >> 172.21.2.112:0/3060833674 conn(0x7f49900bef00 msgr2=0x7f499009d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:43.819 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:43.819 7f4995af1700 1 -- 172.21.2.112:0/3060833674 shutdown_connections 2021-02-10T11:42:43.819 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:43.819 7f4995af1700 1 -- 172.21.2.112:0/3060833674 wait complete. 2021-02-10T11:42:43.826 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs clone status cephfs subvolume_0000000000918327 2021-02-10T11:42:44.276 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.240 7f2bf0fca700 1 Processor -- start 2021-02-10T11:42:44.276 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.241 7f2bf0fca700 1 -- start start 2021-02-10T11:42:44.276 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.241 7f2bf0fca700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f2bec10aff0 0x7f2bec0cf750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:44.276 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.241 7f2bf0fca700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bec0fe1c0 con 0x7f2bec0d1870 2021-02-10T11:42:44.276 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.241 7f2bf0fca700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bec1b0cb0 con 0x7f2bec0d1c70 2021-02-10T11:42:44.277 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.241 7f2bf0fca700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bec109a60 con 0x7f2bec10aff0 2021-02-10T11:42:44.277 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.241 7f2bead9d700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f2bec0d1870 0x7f2bec1b04c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:55984/0 (socket says 172.21.2.112:55984) 2021-02-10T11:42:44.277 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.241 7f2bead9d700 1 -- 172.21.2.112:0/2890371384 learned_addr learned my addr 172.21.2.112:0/2890371384 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:42:44.277 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.242 7f2be9d9b700 1 -- 172.21.2.112:0/2890371384 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3284991217 0 0) 0x7f2bdc001180 con 0x7f2bec0d1870 2021-02-10T11:42:44.277 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.242 7f2be9d9b700 1 -- 172.21.2.112:0/2890371384 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2bd4002fe0 con 0x7f2bec0d1870 2021-02-10T11:42:44.277 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.242 7f2beb59e700 1 --2- 172.21.2.112:0/2890371384 >> v2:172.21.2.120:3300/0 conn(0x7f2bec10aff0 0x7f2bec0cf750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:44.277 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.242 7f2be9d9b700 1 -- 172.21.2.112:0/2890371384 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 340311161 0 0) 0x7f2bdc0010e0 con 0x7f2bec0d1870 2021-02-10T11:42:44.277 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.242 7f2be9d9b700 1 -- 172.21.2.112:0/2890371384 >> v2:172.21.2.120:3300/0 conn(0x7f2bec10aff0 msgr2=0x7f2bec0cf750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:44.277 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.242 7f2be9d9b700 1 --2- 172.21.2.112:0/2890371384 >> v2:172.21.2.120:3300/0 conn(0x7f2bec10aff0 0x7f2bec0cf750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:44.277 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.242 7f2be9d9b700 1 -- 172.21.2.112:0/2890371384 >> v1:172.21.2.120:6789/0 conn(0x7f2bec0d1c70 legacy=0x7f2bec0cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:44.277 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.242 7f2be9d9b700 1 -- 172.21.2.112:0/2890371384 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bec10b3f0 con 0x7f2bec0d1870 2021-02-10T11:42:44.277 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.242 7f2be9d9b700 1 -- 172.21.2.112:0/2890371384 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f2bdc001600 con 0x7f2bec0d1870 2021-02-10T11:42:44.277 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.242 7f2be9d9b700 1 -- 172.21.2.112:0/2890371384 >> v1:172.21.2.112:6789/0 conn(0x7f2bec0d1870 legacy=0x7f2bec1b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:44.278 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.242 7f2be9d9b700 1 --2- 172.21.2.112:0/2890371384 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2bd4003aa0 0x7f2bd4005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:44.278 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.242 7f2be9d9b700 1 --2- 172.21.2.112:0/2890371384 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2bd4006790 0x7f2bd4008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:44.278 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.242 7f2be9d9b700 1 --2- 172.21.2.112:0/2890371384 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2bd4009590 0x7f2bd400bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:44.278 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.242 7f2be9d9b700 1 -- 172.21.2.112:0/2890371384 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bd400c300 con 0x7f2bd4003aa0 2021-02-10T11:42:44.278 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.242 7f2be9d9b700 1 -- 172.21.2.112:0/2890371384 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bd400c520 con 0x7f2bd4006790 2021-02-10T11:42:44.278 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.242 7f2be9d9b700 1 -- 172.21.2.112:0/2890371384 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bd400c740 con 0x7f2bd4009590 2021-02-10T11:42:44.278 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.242 7f2bead9d700 1 --2- 172.21.2.112:0/2890371384 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2bd4003aa0 0x7f2bd4005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:44.278 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.242 7f2bead9d700 1 -- 172.21.2.112:0/2890371384 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2bd4009590 msgr2=0x7f2bd400bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:44.278 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.242 7f2bead9d700 1 --2- 172.21.2.112:0/2890371384 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2bd4009590 0x7f2bd400bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:44.278 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.242 7f2bead9d700 1 -- 172.21.2.112:0/2890371384 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2bd4006790 msgr2=0x7f2bd4008c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:44.278 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.242 7f2bead9d700 1 --2- 172.21.2.112:0/2890371384 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2bd4006790 0x7f2bd4008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:44.278 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.242 7f2bead9d700 1 -- 172.21.2.112:0/2890371384 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bd400c9e0 con 0x7f2bd4003aa0 2021-02-10T11:42:44.278 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.242 7f2beb59e700 1 -- 172.21.2.112:0/2890371384 reap_dead start 2021-02-10T11:42:44.279 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.242 7f2bead9d700 1 --2- 172.21.2.112:0/2890371384 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2bd4003aa0 0x7f2bd4005ef0 secure :-1 s=READY pgs=2380 cs=0 l=1 rev1=1 rx=0x7f2bdc001e80 tx=0x7f2bdc007cf0).ready entity=mon.0 client_cookie=6991583ee42d73be server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:44.279 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.242 7f2be9d9b700 1 -- 172.21.2.112:0/2890371384 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2bdc001ac0 con 0x7f2bd4003aa0 2021-02-10T11:42:44.279 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.242 7f2be9d9b700 1 -- 172.21.2.112:0/2890371384 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bdc0071c0 con 0x7f2bd4003aa0 2021-02-10T11:42:44.279 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.242 7f2be9d9b700 1 -- 172.21.2.112:0/2890371384 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2bdc0043a0 con 0x7f2bd4003aa0 2021-02-10T11:42:44.279 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.242 7f2bf0fca700 1 -- 172.21.2.112:0/2890371384 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2bd4003aa0 msgr2=0x7f2bd4005ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:44.279 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.242 7f2bf0fca700 1 --2- 172.21.2.112:0/2890371384 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2bd4003aa0 0x7f2bd4005ef0 secure :-1 s=READY pgs=2380 cs=0 l=1 rev1=1 rx=0x7f2bdc001e80 tx=0x7f2bdc007cf0).stop 2021-02-10T11:42:44.279 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.243 7f2bf0fca700 1 -- 172.21.2.112:0/2890371384 shutdown_connections 2021-02-10T11:42:44.279 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.243 7f2bf0fca700 1 --2- 172.21.2.112:0/2890371384 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2bd4003aa0 0x7f2bd4005ef0 unknown :-1 s=CLOSED pgs=2380 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:44.279 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.243 7f2bf0fca700 1 -- 172.21.2.112:0/2890371384 >> 172.21.2.112:0/2890371384 conn(0x7f2bec0ce420 msgr2=0x7f2bec0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:44.279 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.243 7f2bf0fca700 1 -- 172.21.2.112:0/2890371384 shutdown_connections 2021-02-10T11:42:44.279 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.243 7f2bf0fca700 1 -- 172.21.2.112:0/2890371384 wait complete. 2021-02-10T11:42:44.279 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.277 7f2bf0fca700 1 Processor -- start 2021-02-10T11:42:44.279 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.277 7f2bf0fca700 1 -- start start 2021-02-10T11:42:44.281 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.277 7f2bf0fca700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2bec1a79a0 0x7f2bec1a7da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:44.281 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.277 7f2bf0fca700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2bec0bdb40 0x7f2bec0bdf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:44.281 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.277 7f2bf0fca700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2bec082ca0 0x7f2bec0830a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:44.282 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.277 7f2bf0fca700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bec19d990 con 0x7f2bec0bdb40 2021-02-10T11:42:44.282 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.277 7f2bf0fca700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bec19dbb0 con 0x7f2bec1a79a0 2021-02-10T11:42:44.282 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.277 7f2bf0fca700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bec0836c0 con 0x7f2bec082ca0 2021-02-10T11:42:44.282 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.277 7f2bea59c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2bec0bdb40 0x7f2bec0bdf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:44.282 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.278 7f2bea59c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2bec0bdb40 0x7f2bec0bdf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:56536/0 (socket says 172.21.2.112:56536) 2021-02-10T11:42:44.282 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.278 7f2bea59c700 1 -- 172.21.2.112:0/1725202986 learned_addr learned my addr 172.21.2.112:0/1725202986 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:42:44.282 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.278 7f2bea59c700 1 -- 172.21.2.112:0/1725202986 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2bec082ca0 msgr2=0x7f2bec0830a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:44.282 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.278 7f2bea59c700 1 --2- 172.21.2.112:0/1725202986 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2bec082ca0 0x7f2bec0830a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:44.282 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.278 7f2bea59c700 1 -- 172.21.2.112:0/1725202986 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2bec1a79a0 msgr2=0x7f2bec1a7da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:44.283 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.278 7f2bea59c700 1 --2- 172.21.2.112:0/1725202986 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2bec1a79a0 0x7f2bec1a7da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:44.283 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.278 7f2bea59c700 1 -- 172.21.2.112:0/1725202986 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bec0838e0 con 0x7f2bec0bdb40 2021-02-10T11:42:44.283 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.278 7f2bea59c700 1 --2- 172.21.2.112:0/1725202986 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2bec0bdb40 0x7f2bec0bdf40 secure :-1 s=READY pgs=2381 cs=0 l=1 rev1=1 rx=0x7f2be0006f30 tx=0x7f2be000b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:44.283 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.278 7f2bdb7fe700 1 -- 172.21.2.112:0/1725202986 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2be0000c60 con 0x7f2bec0bdb40 2021-02-10T11:42:44.283 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.278 7f2bdb7fe700 1 -- 172.21.2.112:0/1725202986 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2be0006030 con 0x7f2bec0bdb40 2021-02-10T11:42:44.283 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.278 7f2bdb7fe700 1 -- 172.21.2.112:0/1725202986 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2be0005540 con 0x7f2bec0bdb40 2021-02-10T11:42:44.283 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.278 7f2bf0fca700 1 -- 172.21.2.112:0/1725202986 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bec0838e0 con 0x7f2bec0bdb40 2021-02-10T11:42:44.283 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.278 7f2bf0fca700 1 -- 172.21.2.112:0/1725202986 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bec0838e0 con 0x7f2bec0bdb40 2021-02-10T11:42:44.283 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.279 7f2bdb7fe700 1 -- 172.21.2.112:0/1725202986 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2be0006210 con 0x7f2bec0bdb40 2021-02-10T11:42:44.283 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.279 7f2bdb7fe700 1 --2- 172.21.2.112:0/1725202986 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2bd00186c0 0x7f2bd001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:44.283 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.279 7f2bdb7fe700 1 -- 172.21.2.112:0/1725202986 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f2be00323e0 con 0x7f2bec0bdb40 2021-02-10T11:42:44.283 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.279 7f2bead9d700 1 --2- 172.21.2.112:0/1725202986 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2bd00186c0 0x7f2bd001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:44.283 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.280 7f2bead9d700 1 --2- 172.21.2.112:0/1725202986 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2bd00186c0 0x7f2bd001abc0 crc :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:44.290 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.291 7f2bf0fca700 1 -- 172.21.2.112:0/1725202986 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bec082a70 con 0x7f2bec0bdb40 2021-02-10T11:42:44.293 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.294 7f2bdb7fe700 1 -- 172.21.2.112:0/1725202986 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2be0026a90 con 0x7f2bec0bdb40 2021-02-10T11:42:44.425 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.425 7f2bf0fca700 1 -- 172.21.2.112:0/1725202986 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvolume_0000000000918327", "target": ["mgr", ""]}) v1 -- 0x7f2bec082a70 con 0x7f2bd00186c0 2021-02-10T11:42:44.428 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.429 7f2bdb7fe700 1 -- 172.21.2.112:0/1725202986 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: -95 operation 'clone-status' is not allowed on subvolume 'subvolume_0000000000918327' of type subvolume) v1 ==== 107+0+0 (crc 0 0 0) 0x7f2bdc001820 con 0x7f2bd00186c0 2021-02-10T11:42:44.428 INFO:tasks.ceph.mgr.x.gibba020.stderr:2021-02-10 11:42:44.429 7f6382eee700 -1 mgr.server reply reply (95) Operation not supported operation 'clone-status' is not allowed on subvolume 'subvolume_0000000000918327' of type subvolume 2021-02-10T11:42:44.429 INFO:teuthology.orchestra.run.gibba012.stderr:Error ENOTSUP: operation 'clone-status' is not allowed on subvolume 'subvolume_0000000000918327' of type subvolume 2021-02-10T11:42:44.430 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.430 7f2bf0fca700 1 -- 172.21.2.112:0/1725202986 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2bd00186c0 msgr2=0x7f2bd001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:44.430 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.430 7f2bf0fca700 1 --2- 172.21.2.112:0/1725202986 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2bd00186c0 0x7f2bd001abc0 crc :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:44.430 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.430 7f2bf0fca700 1 -- 172.21.2.112:0/1725202986 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2bec0bdb40 msgr2=0x7f2bec0bdf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:44.430 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.430 7f2bf0fca700 1 --2- 172.21.2.112:0/1725202986 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2bec0bdb40 0x7f2bec0bdf40 secure :-1 s=READY pgs=2381 cs=0 l=1 rev1=1 rx=0x7f2be0006f30 tx=0x7f2be000b990).stop 2021-02-10T11:42:44.430 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.430 7f2bf0fca700 1 -- 172.21.2.112:0/1725202986 shutdown_connections 2021-02-10T11:42:44.431 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.430 7f2bf0fca700 1 --2- 172.21.2.112:0/1725202986 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2bd00186c0 0x7f2bd001abc0 unknown :-1 s=CLOSED pgs=1086 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:44.431 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.430 7f2bf0fca700 1 --2- 172.21.2.112:0/1725202986 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2bec082ca0 0x7f2bec0830a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:44.431 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.430 7f2bf0fca700 1 --2- 172.21.2.112:0/1725202986 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2bec0bdb40 0x7f2bec0bdf40 unknown :-1 s=CLOSED pgs=2381 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:44.431 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.430 7f2bf0fca700 1 --2- 172.21.2.112:0/1725202986 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2bec1a79a0 0x7f2bec1a7da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:44.432 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.430 7f2bf0fca700 1 -- 172.21.2.112:0/1725202986 >> 172.21.2.112:0/1725202986 conn(0x7f2bec0bef00 msgr2=0x7f2bec09f200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:44.432 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.430 7f2bf0fca700 1 -- 172.21.2.112:0/1725202986 shutdown_connections 2021-02-10T11:42:44.432 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.430 7f2bf0fca700 1 -- 172.21.2.112:0/1725202986 wait complete. 2021-02-10T11:42:44.438 DEBUG:teuthology.orchestra.run:got remote process result: 95 2021-02-10T11:42:44.438 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000918327 2021-02-10T11:42:44.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.853 7f1f4d035700 1 Processor -- start 2021-02-10T11:42:44.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.853 7f1f4d035700 1 -- start start 2021-02-10T11:42:44.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.853 7f1f4d035700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f1f480d1c70 0x7f1f480cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:44.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.853 7f1f4d035700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1f480cf580 0x7f1f480cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:44.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.853 7f1f4d035700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f480fe1c0 con 0x7f1f480d1870 2021-02-10T11:42:44.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.854 7f1f4d035700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f481b0cb0 con 0x7f1f480d1c70 2021-02-10T11:42:44.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.854 7f1f4d035700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f48109a60 con 0x7f1f480cf580 2021-02-10T11:42:44.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.855 7f1f4659c700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f1f480d1c70 0x7f1f480cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:44.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.855 7f1f4759e700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1f480cf580 0x7f1f480cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:44.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.855 7f1f46d9d700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7f1f480d1870 0x7f1f481b04c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:49562/0 (socket says 172.21.2.112:49562) 2021-02-10T11:42:44.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.855 7f1f46d9d700 1 -- 172.21.2.112:0/1351464960 learned_addr learned my addr 172.21.2.112:0/1351464960 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:42:44.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.855 7f1f45d9b700 1 -- 172.21.2.112:0/1351464960 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1090895545 0 0) 0x7f1f380010c0 con 0x7f1f480d1870 2021-02-10T11:42:44.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.855 7f1f4759e700 1 -- 172.21.2.112:0/1351464960 >> v2:172.21.2.120:3300/0 conn(0x7f1f480d1c70 msgr2=0x7f1f480cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:44.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.855 7f1f4759e700 1 --2- 172.21.2.112:0/1351464960 >> v2:172.21.2.120:3300/0 conn(0x7f1f480d1c70 0x7f1f480cef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:44.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.855 7f1f4759e700 1 -- 172.21.2.112:0/1351464960 >> v1:172.21.2.120:6789/0 conn(0x7f1f480d1870 legacy=0x7f1f481b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:44.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.855 7f1f4759e700 1 -- 172.21.2.112:0/1351464960 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f48088070 con 0x7f1f480cf580 2021-02-10T11:42:44.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.856 7f1f4759e700 1 --2- 172.21.2.112:0/1351464960 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1f480cf580 0x7f1f480cf980 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7f1f40006e50 tx=0x7f1f40009e70).ready entity=mon.2 client_cookie=39125bc86446df4c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:44.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.856 7f1f45d9b700 1 -- 172.21.2.112:0/1351464960 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1f40006a20 con 0x7f1f480cf580 2021-02-10T11:42:44.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.856 7f1f45d9b700 1 -- 172.21.2.112:0/1351464960 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f40006790 con 0x7f1f480cf580 2021-02-10T11:42:44.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.856 7f1f45d9b700 1 -- 172.21.2.112:0/1351464960 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1f4000b700 con 0x7f1f480cf580 2021-02-10T11:42:44.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.856 7f1f4d035700 1 -- 172.21.2.112:0/1351464960 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1f480cf580 msgr2=0x7f1f480cf980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:44.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.856 7f1f4d035700 1 --2- 172.21.2.112:0/1351464960 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1f480cf580 0x7f1f480cf980 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7f1f40006e50 tx=0x7f1f40009e70).stop 2021-02-10T11:42:44.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.857 7f1f4d035700 1 -- 172.21.2.112:0/1351464960 shutdown_connections 2021-02-10T11:42:44.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.857 7f1f4d035700 1 --2- 172.21.2.112:0/1351464960 >> v2:172.21.2.120:3300/0 conn(0x7f1f480d1c70 0x7f1f480cef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:44.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.857 7f1f4d035700 1 --2- 172.21.2.112:0/1351464960 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1f480cf580 0x7f1f480cf980 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:44.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.857 7f1f4d035700 1 -- 172.21.2.112:0/1351464960 >> 172.21.2.112:0/1351464960 conn(0x7f1f480ce420 msgr2=0x7f1f480d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:44.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.857 7f1f4d035700 1 -- 172.21.2.112:0/1351464960 shutdown_connections 2021-02-10T11:42:44.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.857 7f1f4d035700 1 -- 172.21.2.112:0/1351464960 wait complete. 2021-02-10T11:42:44.875 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.871 7f1f4d035700 1 Processor -- start 2021-02-10T11:42:44.875 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.871 7f1f4d035700 1 -- start start 2021-02-10T11:42:44.875 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.872 7f1f4d035700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1f48191b10 0x7f1f480ae640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:44.875 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.872 7f1f4d035700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1f48191f10 0x7f1f4819ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:44.875 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.872 7f1f4d035700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1f481a05f0 0x7f1f48082ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:44.875 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.872 7f1f4d035700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f48192440 con 0x7f1f48191f10 2021-02-10T11:42:44.875 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.872 7f1f4d035700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f481916e0 con 0x7f1f481a05f0 2021-02-10T11:42:44.875 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.872 7f1f4d035700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f48191900 con 0x7f1f48191b10 2021-02-10T11:42:44.875 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.872 7f1f4659c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1f48191f10 0x7f1f4819ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:44.875 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.872 7f1f4659c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1f48191f10 0x7f1f4819ffd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:56550/0 (socket says 172.21.2.112:56550) 2021-02-10T11:42:44.875 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.872 7f1f4659c700 1 -- 172.21.2.112:0/2104629489 learned_addr learned my addr 172.21.2.112:0/2104629489 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:42:44.876 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.872 7f1f4759e700 1 --2- 172.21.2.112:0/2104629489 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1f481a05f0 0x7f1f48082ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:44.876 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.872 7f1f46d9d700 1 --2- 172.21.2.112:0/2104629489 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1f48191b10 0x7f1f480ae640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:44.876 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.872 7f1f4659c700 1 -- 172.21.2.112:0/2104629489 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1f48191b10 msgr2=0x7f1f480ae640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:44.876 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.872 7f1f4659c700 1 --2- 172.21.2.112:0/2104629489 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1f48191b10 0x7f1f480ae640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:44.876 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.872 7f1f4659c700 1 -- 172.21.2.112:0/2104629489 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1f481a05f0 msgr2=0x7f1f48082ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:44.876 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.872 7f1f4659c700 1 --2- 172.21.2.112:0/2104629489 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1f481a05f0 0x7f1f48082ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:44.876 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.872 7f1f4659c700 1 -- 172.21.2.112:0/2104629489 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f480834f0 con 0x7f1f48191f10 2021-02-10T11:42:44.876 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.873 7f1f4659c700 1 --2- 172.21.2.112:0/2104629489 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1f48191f10 0x7f1f4819ffd0 secure :-1 s=READY pgs=2382 cs=0 l=1 rev1=1 rx=0x7f1f3c006f60 tx=0x7f1f3c00b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:44.876 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.873 7f1f377fe700 1 -- 172.21.2.112:0/2104629489 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1f3c0009b0 con 0x7f1f48191f10 2021-02-10T11:42:44.876 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.873 7f1f377fe700 1 -- 172.21.2.112:0/2104629489 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f3c00bdf0 con 0x7f1f48191f10 2021-02-10T11:42:44.876 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.873 7f1f4d035700 1 -- 172.21.2.112:0/2104629489 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f480834f0 con 0x7f1f48191f10 2021-02-10T11:42:44.876 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.873 7f1f377fe700 1 -- 172.21.2.112:0/2104629489 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1f3c002030 con 0x7f1f48191f10 2021-02-10T11:42:44.876 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.873 7f1f4d035700 1 -- 172.21.2.112:0/2104629489 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f48083a80 con 0x7f1f48191f10 2021-02-10T11:42:44.876 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.874 7f1f377fe700 1 -- 172.21.2.112:0/2104629489 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1f3c006b00 con 0x7f1f48191f10 2021-02-10T11:42:44.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.874 7f1f377fe700 1 --2- 172.21.2.112:0/2104629489 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1f2c018730 0x7f1f2c01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:44.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.874 7f1f377fe700 1 -- 172.21.2.112:0/2104629489 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f1f3c032630 con 0x7f1f48191f10 2021-02-10T11:42:44.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.874 7f1f46d9d700 1 --2- 172.21.2.112:0/2104629489 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1f2c018730 0x7f1f2c01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:44.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.874 7f1f46d9d700 1 --2- 172.21.2.112:0/2104629489 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1f2c018730 0x7f1f2c01ac10 crc :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:44.903 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.903 7f1f4d035700 1 -- 172.21.2.112:0/2104629489 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f480fe1c0 con 0x7f1f48191f10 2021-02-10T11:42:44.906 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:44.906 7f1f377fe700 1 -- 172.21.2.112:0/2104629489 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1f3c0009b0 con 0x7f1f48191f10 2021-02-10T11:42:45.034 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:45.034 7f1f4d035700 1 -- 172.21.2.112:0/2104629489 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000918327", "target": ["mgr", ""]}) v1 -- 0x7f1f48083a80 con 0x7f1f2c018730 2021-02-10T11:42:45.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:45.043 7f1f377fe700 1 -- 172.21.2.112:0/2104629489 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f1f38006820 con 0x7f1f2c018730 2021-02-10T11:42:45.051 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:45.050 7f1f4d035700 1 -- 172.21.2.112:0/2104629489 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1f2c018730 msgr2=0x7f1f2c01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:45.051 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:45.050 7f1f4d035700 1 --2- 172.21.2.112:0/2104629489 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1f2c018730 0x7f1f2c01ac10 crc :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:45.051 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:45.050 7f1f4d035700 1 -- 172.21.2.112:0/2104629489 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1f48191f10 msgr2=0x7f1f4819ffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:45.051 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:45.050 7f1f4d035700 1 --2- 172.21.2.112:0/2104629489 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1f48191f10 0x7f1f4819ffd0 secure :-1 s=READY pgs=2382 cs=0 l=1 rev1=1 rx=0x7f1f3c006f60 tx=0x7f1f3c00b970).stop 2021-02-10T11:42:45.051 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:45.051 7f1f4d035700 1 -- 172.21.2.112:0/2104629489 shutdown_connections 2021-02-10T11:42:45.051 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:45.051 7f1f4d035700 1 --2- 172.21.2.112:0/2104629489 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1f2c018730 0x7f1f2c01ac10 unknown :-1 s=CLOSED pgs=1087 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:45.052 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:45.051 7f1f4d035700 1 --2- 172.21.2.112:0/2104629489 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1f481a05f0 0x7f1f48082ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:45.052 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:45.051 7f1f4d035700 1 --2- 172.21.2.112:0/2104629489 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1f48191f10 0x7f1f4819ffd0 unknown :-1 s=CLOSED pgs=2382 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:45.052 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:45.051 7f1f4d035700 1 --2- 172.21.2.112:0/2104629489 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1f48191b10 0x7f1f480ae640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:45.052 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:45.051 7f1f4d035700 1 -- 172.21.2.112:0/2104629489 >> 172.21.2.112:0/2104629489 conn(0x7f1f480a02d0 msgr2=0x7f1f4809f200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:45.052 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:45.051 7f1f4d035700 1 -- 172.21.2.112:0/2104629489 shutdown_connections 2021-02-10T11:42:45.052 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:45.051 7f1f4d035700 1 -- 172.21.2.112:0/2104629489 wait complete. 2021-02-10T11:42:45.059 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo getfattr --only-values -n ceph.dir.entries /home/ubuntu/cephtest/mnt.0/./volumes/_deleting 2021-02-10T11:42:45.407 INFO:teuthology.orchestra.run.gibba012.stderr:getfattr: Removing leading '/' from absolute path names 2021-02-10T11:42:45.408 INFO:teuthology.orchestra.run.gibba012.stdout:0 2021-02-10T11:42:45.408 DEBUG:tasks.cephfs.mount:Directory ./volumes/_deleting seen empty from 0 after 0s 2021-02-10T11:42:45.408 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:42:45.408 DEBUG:teuthology.orchestra.run.gibba012:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:42:45.754 DEBUG:teuthology.orchestra.run.gibba020:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:42:46.100 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:42:46.101 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:42:46.101 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:42:46.101 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:42:46.437 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:42:46.437 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:42:46.438 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba012.front.sepia.ceph.com... 2021-02-10T11:42:46.438 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:42:46.438 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:42:46.790 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba012.stderr:ceph-fuse[165630]: fuse finished with error 0 and tester_r 0 2021-02-10T11:42:46.791 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:42:46.792 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:42:46.792 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:42:47.129 INFO:teuthology.orchestra.run.gibba012.stdout:ext2/ext3 2021-02-10T11:42:47.129 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:42:47.129 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:42:47.130 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:42:47.466 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:42:47.466 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:42:47.466 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:42:47.802 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:42:47.802 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:42:47.802 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba012.front.sepia.ceph.com... 2021-02-10T11:42:47.803 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:42:47.803 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:42:48.153 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba012.stderr:ceph-fuse[166270]: fuse finished with error 0 and tester_r 0 2021-02-10T11:42:48.154 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:42:48.155 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:42:48.155 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:42:48.489 INFO:teuthology.orchestra.run.gibba012.stdout:ext2/ext3 2021-02-10T11:42:48.490 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:42:48.490 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:42:48.490 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:42:48.825 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:42:48.825 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:42:48.825 DEBUG:teuthology.orchestra.run.gibba020:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:42:49.160 INFO:teuthology.orchestra.run.gibba020.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:42:49.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:42:49.160 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:42:49.160 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:42:49.161 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:42:49.494 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:42:49.494 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:42:49.494 DEBUG:teuthology.orchestra.run.gibba020:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:42:49.829 INFO:teuthology.orchestra.run.gibba020.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:42:49.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:42:49.829 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:42:49.829 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:42:49.830 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:42:50.164 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config rm mon mon_allow_pool_delete 2021-02-10T11:42:50.597 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:50.581 7fa0a254e700 1 Processor -- start 2021-02-10T11:42:50.597 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:50.581 7fa0a254e700 1 -- start start 2021-02-10T11:42:50.597 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:50.581 7fa0a254e700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa09c0d1870 0x7fa09c1b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:50.597 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:50.581 7fa0a254e700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fa09c0f0650 0x7fa09c0cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:50.597 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:50.581 7fa0a254e700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa09c0f0b10 con 0x7fa09c1b0a10 2021-02-10T11:42:50.597 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:50.581 7fa0a254e700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa09c0fe230 con 0x7fa09c0f0650 2021-02-10T11:42:50.597 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:50.581 7fa0a254e700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa09c0e5ed0 con 0x7fa09c0d1870 2021-02-10T11:42:50.597 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:50.581 7fa09bfff700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa09c0d1870 0x7fa09c1b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:50.597 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:50.581 7fa09b7fe700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7fa09c1b0a10 0x7fa09c0cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:49576/0 (socket says 172.21.2.112:49576) 2021-02-10T11:42:50.598 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:50.581 7fa09b7fe700 1 -- 172.21.2.112:0/3152128221 learned_addr learned my addr 172.21.2.112:0/3152128221 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:42:50.598 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:50.581 7fa0a0b6b700 1 --2- 172.21.2.112:0/3152128221 >> v2:172.21.2.120:3300/0 conn(0x7fa09c0f0650 0x7fa09c0cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:50.598 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:50.582 7fa0a0b6b700 1 -- 172.21.2.112:0/3152128221 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa09c0d1870 msgr2=0x7fa09c1b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:50.598 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:50.582 7fa0a0b6b700 1 --2- 172.21.2.112:0/3152128221 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa09c0d1870 0x7fa09c1b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:50.598 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:50.582 7fa0a0b6b700 1 -- 172.21.2.112:0/3152128221 >> v1:172.21.2.120:6789/0 conn(0x7fa09c1b0a10 legacy=0x7fa09c0cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:50.598 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:50.582 7fa0a0b6b700 1 -- 172.21.2.112:0/3152128221 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa09c0f20a0 con 0x7fa09c0f0650 2021-02-10T11:42:50.598 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:50.582 7fa0a0b6b700 1 --2- 172.21.2.112:0/3152128221 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa09c0f0650 0x7fa09c0cfb20 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 rx=0x7fa094006e50 tx=0x7fa094009e70).ready entity=mon.1 client_cookie=8649e927a3aef135 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:50.598 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:50.582 7fa09affd700 1 -- 172.21.2.112:0/3152128221 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa09400b510 con 0x7fa09c0f0650 2021-02-10T11:42:50.598 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:50.582 7fa09affd700 1 -- 172.21.2.112:0/3152128221 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa094006790 con 0x7fa09c0f0650 2021-02-10T11:42:50.598 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:50.582 7fa09affd700 1 -- 172.21.2.112:0/3152128221 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa09400b940 con 0x7fa09c0f0650 2021-02-10T11:42:50.598 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:50.582 7fa0a254e700 1 -- 172.21.2.112:0/3152128221 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa09c0f0650 msgr2=0x7fa09c0cfb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:50.598 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:50.582 7fa0a254e700 1 --2- 172.21.2.112:0/3152128221 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa09c0f0650 0x7fa09c0cfb20 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 rx=0x7fa094006e50 tx=0x7fa094009e70).stop 2021-02-10T11:42:50.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:50.583 7fa0a254e700 1 -- 172.21.2.112:0/3152128221 shutdown_connections 2021-02-10T11:42:50.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:50.583 7fa0a254e700 1 --2- 172.21.2.112:0/3152128221 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa09c0f0650 0x7fa09c0cfb20 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:50.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:50.583 7fa0a254e700 1 --2- 172.21.2.112:0/3152128221 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa09c0d1870 0x7fa09c1b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:50.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:50.583 7fa0a254e700 1 -- 172.21.2.112:0/3152128221 >> 172.21.2.112:0/3152128221 conn(0x7fa09c0ce450 msgr2=0x7fa09c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:50.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:50.583 7fa0a254e700 1 -- 172.21.2.112:0/3152128221 shutdown_connections 2021-02-10T11:42:50.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:50.583 7fa0a254e700 1 -- 172.21.2.112:0/3152128221 wait complete. 2021-02-10T11:42:50.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:50.598 7fa0a254e700 1 Processor -- start 2021-02-10T11:42:50.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:50.598 7fa0a254e700 1 -- start start 2021-02-10T11:42:50.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:50.598 7fa0a254e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa09c0ebe70 0x7fa09c0ec270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:50.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:50.598 7fa0a254e700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa09c093830 0x7fa09c093c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:50.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:50.598 7fa0a254e700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa09c094250 0x7fa09c094650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:50.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:50.598 7fa0a254e700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa09c0ec940 con 0x7fa09c0ebe70 2021-02-10T11:42:50.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:50.598 7fa0a254e700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa09c0ece40 con 0x7fa09c094250 2021-02-10T11:42:50.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:50.598 7fa0a254e700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa09c0ed060 con 0x7fa09c093830 2021-02-10T11:42:50.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:50.598 7fa09bfff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa09c0ebe70 0x7fa09c0ec270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:50.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:50.598 7fa09bfff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa09c0ebe70 0x7fa09c0ec270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:56562/0 (socket says 172.21.2.112:56562) 2021-02-10T11:42:50.602 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:50.598 7fa09bfff700 1 -- 172.21.2.112:0/2064801310 learned_addr learned my addr 172.21.2.112:0/2064801310 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:42:50.602 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:50.599 7fa09b7fe700 1 --2- 172.21.2.112:0/2064801310 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa09c093830 0x7fa09c093c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:50.602 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:50.599 7fa0a0b6b700 1 --2- 172.21.2.112:0/2064801310 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa09c094250 0x7fa09c094650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:50.602 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:50.599 7fa09bfff700 1 -- 172.21.2.112:0/2064801310 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa09c093830 msgr2=0x7fa09c093c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:50.602 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:50.599 7fa09bfff700 1 --2- 172.21.2.112:0/2064801310 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa09c093830 0x7fa09c093c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:50.602 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:50.599 7fa09bfff700 1 -- 172.21.2.112:0/2064801310 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa09c094250 msgr2=0x7fa09c094650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:50.602 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:50.599 7fa09bfff700 1 --2- 172.21.2.112:0/2064801310 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa09c094250 0x7fa09c094650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:50.602 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:50.599 7fa09bfff700 1 -- 172.21.2.112:0/2064801310 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa09c0ed280 con 0x7fa09c0ebe70 2021-02-10T11:42:50.602 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:50.599 7fa0a0b6b700 1 --2- 172.21.2.112:0/2064801310 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa09c094250 0x7fa09c094650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:42:50.602 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:50.599 7fa09bfff700 1 --2- 172.21.2.112:0/2064801310 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa09c0ebe70 0x7fa09c0ec270 secure :-1 s=READY pgs=2383 cs=0 l=1 rev1=1 rx=0x7fa08c006f60 tx=0x7fa08c00b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:50.602 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:50.599 7fa098ff9700 1 -- 172.21.2.112:0/2064801310 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa08c0009b0 con 0x7fa09c0ebe70 2021-02-10T11:42:50.602 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:50.599 7fa098ff9700 1 -- 172.21.2.112:0/2064801310 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa08c00bd60 con 0x7fa09c0ebe70 2021-02-10T11:42:50.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:50.599 7fa098ff9700 1 -- 172.21.2.112:0/2064801310 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa08c002030 con 0x7fa09c0ebe70 2021-02-10T11:42:50.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:50.599 7fa0a254e700 1 -- 172.21.2.112:0/2064801310 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa09c0ed280 con 0x7fa09c0ebe70 2021-02-10T11:42:50.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:50.599 7fa0a254e700 1 -- 172.21.2.112:0/2064801310 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa09c0ed280 con 0x7fa09c0ebe70 2021-02-10T11:42:50.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:50.600 7fa098ff9700 1 -- 172.21.2.112:0/2064801310 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa08c006b00 con 0x7fa09c0ebe70 2021-02-10T11:42:50.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:50.600 7fa098ff9700 1 --2- 172.21.2.112:0/2064801310 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa07c0186d0 0x7fa07c01abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:50.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:50.600 7fa098ff9700 1 -- 172.21.2.112:0/2064801310 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fa08c0327c0 con 0x7fa09c0ebe70 2021-02-10T11:42:50.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:50.600 7fa09b7fe700 1 --2- 172.21.2.112:0/2064801310 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa07c0186d0 0x7fa07c01abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:50.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:50.600 7fa09b7fe700 1 --2- 172.21.2.112:0/2064801310 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa07c0186d0 0x7fa07c01abd0 crc :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:50.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:50.631 7fa0a254e700 1 -- 172.21.2.112:0/2064801310 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa09c0ebc40 con 0x7fa09c0ebe70 2021-02-10T11:42:50.633 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:50.634 7fa098ff9700 1 -- 172.21.2.112:0/2064801310 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa08c006340 con 0x7fa09c0ebe70 2021-02-10T11:42:50.782 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:50.783 7fa0a254e700 1 -- 172.21.2.112:0/2064801310 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fa09c0ebc40 con 0x7fa09c0ebe70 2021-02-10T11:42:50.786 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:50.787 7fa098ff9700 1 -- 172.21.2.112:0/2064801310 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v33) v1 ==== 104+0+0 (secure 0 0 0) 0x7fa08c006b00 con 0x7fa09c0ebe70 2021-02-10T11:42:50.791 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:50.792 7fa0a254e700 1 -- 172.21.2.112:0/2064801310 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa07c0186d0 msgr2=0x7fa07c01abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:50.791 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:50.792 7fa0a254e700 1 --2- 172.21.2.112:0/2064801310 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa07c0186d0 0x7fa07c01abd0 crc :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:50.791 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:50.792 7fa0a254e700 1 -- 172.21.2.112:0/2064801310 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa09c0ebe70 msgr2=0x7fa09c0ec270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:50.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:50.792 7fa0a254e700 1 --2- 172.21.2.112:0/2064801310 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa09c0ebe70 0x7fa09c0ec270 secure :-1 s=READY pgs=2383 cs=0 l=1 rev1=1 rx=0x7fa08c006f60 tx=0x7fa08c00b970).stop 2021-02-10T11:42:50.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:50.792 7fa0a254e700 1 -- 172.21.2.112:0/2064801310 shutdown_connections 2021-02-10T11:42:50.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:50.792 7fa0a254e700 1 --2- 172.21.2.112:0/2064801310 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa07c0186d0 0x7fa07c01abd0 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:50.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:50.792 7fa0a254e700 1 --2- 172.21.2.112:0/2064801310 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa09c094250 0x7fa09c094650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:50.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:50.792 7fa0a254e700 1 --2- 172.21.2.112:0/2064801310 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa09c093830 0x7fa09c093c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:50.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:50.792 7fa0a254e700 1 --2- 172.21.2.112:0/2064801310 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa09c0ebe70 0x7fa09c0ec270 unknown :-1 s=CLOSED pgs=2383 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:50.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:50.792 7fa0a254e700 1 -- 172.21.2.112:0/2064801310 >> 172.21.2.112:0/2064801310 conn(0x7fa09c08fa10 msgr2=0x7fa09c0bfa60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:50.793 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:50.792 7fa0a254e700 1 -- 172.21.2.112:0/2064801310 shutdown_connections 2021-02-10T11:42:50.793 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:50.793 7fa0a254e700 1 -- 172.21.2.112:0/2064801310 wait complete. 2021-02-10T11:42:50.800 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestVolumes.test_non_clone_status' 2021-02-10T11:42:51.227 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:51.211 7f413971f700 1 Processor -- start 2021-02-10T11:42:51.227 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:51.211 7f413971f700 1 -- start start 2021-02-10T11:42:51.228 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:51.212 7f413971f700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f41340f05f0 0x7f41340f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:51.228 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:51.212 7f413971f700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41341b0c70 con 0x7f41341b0470 2021-02-10T11:42:51.228 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:51.212 7f413971f700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41340f0b50 con 0x7f41341b0870 2021-02-10T11:42:51.228 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:51.212 7f413971f700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41340e5ea0 con 0x7f41340f05f0 2021-02-10T11:42:51.228 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:51.212 7f41332bf700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f41341b0470 0x7f41340ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:56030/0 (socket says 172.21.2.112:56030) 2021-02-10T11:42:51.228 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:51.212 7f41332bf700 1 -- 172.21.2.112:0/3625115925 learned_addr learned my addr 172.21.2.112:0/3625115925 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:42:51.228 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:51.212 7f41322bd700 1 -- 172.21.2.112:0/3625115925 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 520711977 0 0) 0x7f41240010b0 con 0x7f41341b0470 2021-02-10T11:42:51.228 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:51.212 7f41322bd700 1 -- 172.21.2.112:0/3625115925 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4120002fe0 con 0x7f41341b0470 2021-02-10T11:42:51.228 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:51.212 7f4133ac0700 1 --2- 172.21.2.112:0/3625115925 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f41340f05f0 0x7f41340f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:51.228 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:51.212 7f41322bd700 1 -- 172.21.2.112:0/3625115925 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 681878812 0 0) 0x7f41240010b0 con 0x7f41341b0470 2021-02-10T11:42:51.228 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:51.212 7f41322bd700 1 -- 172.21.2.112:0/3625115925 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f41340f05f0 msgr2=0x7f41340f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:51.228 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:51.212 7f41322bd700 1 --2- 172.21.2.112:0/3625115925 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f41340f05f0 0x7f41340f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:51.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:51.212 7f41322bd700 1 -- 172.21.2.112:0/3625115925 >> v1:172.21.2.120:6789/0 conn(0x7f41341b0870 legacy=0x7f41340efe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:51.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:51.212 7f41322bd700 1 -- 172.21.2.112:0/3625115925 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41340e52e0 con 0x7f41341b0470 2021-02-10T11:42:51.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:51.213 7f41322bd700 1 -- 172.21.2.112:0/3625115925 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f41240015b0 con 0x7f41341b0470 2021-02-10T11:42:51.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:51.213 7f41322bd700 1 -- 172.21.2.112:0/3625115925 >> v1:172.21.2.112:6789/0 conn(0x7f41341b0470 legacy=0x7f41340ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:51.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:51.213 7f41322bd700 1 --2- 172.21.2.112:0/3625115925 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4120003aa0 0x7f4120005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:51.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:51.213 7f41322bd700 1 --2- 172.21.2.112:0/3625115925 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4120006790 0x7f4120008c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:51.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:51.213 7f41322bd700 1 --2- 172.21.2.112:0/3625115925 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4120009500 0x7f412000ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:51.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:51.213 7f41322bd700 1 -- 172.21.2.112:0/3625115925 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f412000c2d0 con 0x7f4120003aa0 2021-02-10T11:42:51.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:51.213 7f41322bd700 1 -- 172.21.2.112:0/3625115925 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f412000c4f0 con 0x7f4120009500 2021-02-10T11:42:51.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:51.213 7f41322bd700 1 -- 172.21.2.112:0/3625115925 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4120050720 con 0x7f4120006790 2021-02-10T11:42:51.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:51.213 7f41332bf700 1 --2- 172.21.2.112:0/3625115925 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4120003aa0 0x7f4120005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:51.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:51.213 7f41332bf700 1 -- 172.21.2.112:0/3625115925 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4120006790 msgr2=0x7f4120008c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:51.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:51.213 7f41332bf700 1 --2- 172.21.2.112:0/3625115925 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4120006790 0x7f4120008c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:51.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:51.213 7f41332bf700 1 -- 172.21.2.112:0/3625115925 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4120009500 msgr2=0x7f412000ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:51.230 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:51.213 7f41332bf700 1 --2- 172.21.2.112:0/3625115925 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4120009500 0x7f412000ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:51.230 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:51.213 7f41332bf700 1 -- 172.21.2.112:0/3625115925 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4120050980 con 0x7f4120003aa0 2021-02-10T11:42:51.230 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:51.213 7f41332bf700 1 --2- 172.21.2.112:0/3625115925 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4120003aa0 0x7f4120005ef0 secure :-1 s=READY pgs=2385 cs=0 l=1 rev1=1 rx=0x7f4124000b50 tx=0x7f4124007d60).ready entity=mon.0 client_cookie=68f0a35ff92b0da3 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:51.230 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:51.213 7f41322bd700 1 -- 172.21.2.112:0/3625115925 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f412400a310 con 0x7f4120003aa0 2021-02-10T11:42:51.230 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:51.213 7f41322bd700 1 -- 172.21.2.112:0/3625115925 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f412400a590 con 0x7f4120003aa0 2021-02-10T11:42:51.230 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:51.213 7f41322bd700 1 -- 172.21.2.112:0/3625115925 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f412400a9f0 con 0x7f4120003aa0 2021-02-10T11:42:51.230 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:51.213 7f413971f700 1 -- 172.21.2.112:0/3625115925 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4120003aa0 msgr2=0x7f4120005ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:51.230 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:51.213 7f413971f700 1 --2- 172.21.2.112:0/3625115925 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4120003aa0 0x7f4120005ef0 secure :-1 s=READY pgs=2385 cs=0 l=1 rev1=1 rx=0x7f4124000b50 tx=0x7f4124007d60).stop 2021-02-10T11:42:51.230 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:51.213 7f4133ac0700 1 -- 172.21.2.112:0/3625115925 reap_dead start 2021-02-10T11:42:51.230 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:51.214 7f413971f700 1 -- 172.21.2.112:0/3625115925 shutdown_connections 2021-02-10T11:42:51.230 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:51.214 7f413971f700 1 -- 172.21.2.112:0/3625115925 >> 172.21.2.112:0/3625115925 conn(0x7f41340ce3e0 msgr2=0x7f41340e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:51.230 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:51.214 7f413971f700 1 -- 172.21.2.112:0/3625115925 shutdown_connections 2021-02-10T11:42:51.231 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:51.214 7f413971f700 1 -- 172.21.2.112:0/3625115925 wait complete. 2021-02-10T11:42:51.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:51.228 7f413971f700 1 Processor -- start 2021-02-10T11:42:51.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:51.228 7f413971f700 1 -- start start 2021-02-10T11:42:51.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:51.229 7f413971f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f41340bba80 0x7f4134108e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:51.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:51.229 7f413971f700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f41340bbe80 0x7f41340bd5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:51.232 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:51.229 7f413971f700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f41341094b0 0x7f41340bdbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:51.233 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:51.229 7f413971f700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41341089f0 con 0x7f41340bba80 2021-02-10T11:42:51.233 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:51.229 7f413971f700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41340c2ec0 con 0x7f41340bbe80 2021-02-10T11:42:51.233 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:51.229 7f413971f700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41340c30e0 con 0x7f41341094b0 2021-02-10T11:42:51.233 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:51.229 7f41332bf700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f41340bba80 0x7f4134108e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:51.233 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:51.229 7f41332bf700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f41340bba80 0x7f4134108e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:56580/0 (socket says 172.21.2.112:56580) 2021-02-10T11:42:51.233 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:51.229 7f41332bf700 1 -- 172.21.2.112:0/4231270730 learned_addr learned my addr 172.21.2.112:0/4231270730 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:42:51.233 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:51.229 7f4132abe700 1 --2- 172.21.2.112:0/4231270730 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f41340bbe80 0x7f41340bd5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:51.233 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:51.229 7f4133ac0700 1 --2- 172.21.2.112:0/4231270730 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f41341094b0 0x7f41340bdbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:51.233 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:51.229 7f41332bf700 1 -- 172.21.2.112:0/4231270730 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f41341094b0 msgr2=0x7f41340bdbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:51.233 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:51.229 7f41332bf700 1 --2- 172.21.2.112:0/4231270730 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f41341094b0 0x7f41340bdbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:51.233 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:51.229 7f41332bf700 1 -- 172.21.2.112:0/4231270730 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f41340bbe80 msgr2=0x7f41340bd5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:51.233 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:51.229 7f41332bf700 1 --2- 172.21.2.112:0/4231270730 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f41340bbe80 0x7f41340bd5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:51.234 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:51.230 7f41332bf700 1 -- 172.21.2.112:0/4231270730 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41340be200 con 0x7f41340bba80 2021-02-10T11:42:51.234 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:51.230 7f4133ac0700 1 --2- 172.21.2.112:0/4231270730 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f41341094b0 0x7f41340bdbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:42:51.234 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:51.230 7f4132abe700 1 --2- 172.21.2.112:0/4231270730 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f41340bbe80 0x7f41340bd5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:42:51.234 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:51.230 7f41332bf700 1 --2- 172.21.2.112:0/4231270730 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f41340bba80 0x7f4134108e90 secure :-1 s=READY pgs=2386 cs=0 l=1 rev1=1 rx=0x7f412400a570 tx=0x7f412400a910).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:51.234 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:51.230 7f412bfff700 1 -- 172.21.2.112:0/4231270730 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f412400ad00 con 0x7f41340bba80 2021-02-10T11:42:51.234 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:51.230 7f412bfff700 1 -- 172.21.2.112:0/4231270730 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4124004140 con 0x7f41340bba80 2021-02-10T11:42:51.234 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:51.230 7f413971f700 1 -- 172.21.2.112:0/4231270730 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41340be200 con 0x7f41340bba80 2021-02-10T11:42:51.234 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:51.230 7f412bfff700 1 -- 172.21.2.112:0/4231270730 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f41240045a0 con 0x7f41340bba80 2021-02-10T11:42:51.234 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:51.230 7f413971f700 1 -- 172.21.2.112:0/4231270730 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41340be200 con 0x7f41340bba80 2021-02-10T11:42:51.234 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:51.231 7f412bfff700 1 -- 172.21.2.112:0/4231270730 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4124008000 con 0x7f41340bba80 2021-02-10T11:42:51.234 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:51.231 7f412bfff700 1 --2- 172.21.2.112:0/4231270730 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f41180186d0 0x7f411801abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:51.234 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:51.231 7f412bfff700 1 -- 172.21.2.112:0/4231270730 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f4124032640 con 0x7f41340bba80 2021-02-10T11:42:51.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:51.231 7f4132abe700 1 --2- 172.21.2.112:0/4231270730 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f41180186d0 0x7f411801abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:51.237 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:51.232 7f4132abe700 1 --2- 172.21.2.112:0/4231270730 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f41180186d0 0x7f411801abd0 crc :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:51.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:51.261 7f413971f700 1 -- 172.21.2.112:0/4231270730 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41340f0b50 con 0x7f41340bba80 2021-02-10T11:42:51.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:51.264 7f412bfff700 1 -- 172.21.2.112:0/4231270730 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f412401f500 con 0x7f41340bba80 2021-02-10T11:42:51.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:51.397 7f413971f700 1 -- 172.21.2.112:0/4231270730 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_non_clone_status"]} v 0) v1 -- 0x7f41340f0b50 con 0x7f41340bba80 2021-02-10T11:42:51.657 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:51.657 7f412bfff700 1 -- 172.21.2.112:0/4231270730 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_non_clone_status"]}]=0 v641) v1 ==== 138+0+0 (secure 0 0 0) 0x7f4124008000 con 0x7f41340bba80 2021-02-10T11:42:51.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:51.664 7f413971f700 1 -- 172.21.2.112:0/4231270730 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f41180186d0 msgr2=0x7f411801abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:51.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:51.664 7f413971f700 1 --2- 172.21.2.112:0/4231270730 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f41180186d0 0x7f411801abd0 crc :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:51.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:51.664 7f413971f700 1 -- 172.21.2.112:0/4231270730 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f41340bba80 msgr2=0x7f4134108e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:51.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:51.664 7f413971f700 1 --2- 172.21.2.112:0/4231270730 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f41340bba80 0x7f4134108e90 secure :-1 s=READY pgs=2386 cs=0 l=1 rev1=1 rx=0x7f412400a570 tx=0x7f412400a910).stop 2021-02-10T11:42:51.665 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:51.665 7f413971f700 1 -- 172.21.2.112:0/4231270730 shutdown_connections 2021-02-10T11:42:51.665 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:51.665 7f413971f700 1 --2- 172.21.2.112:0/4231270730 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f41180186d0 0x7f411801abd0 unknown :-1 s=CLOSED pgs=1089 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:51.665 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:51.665 7f413971f700 1 --2- 172.21.2.112:0/4231270730 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f41340bbe80 0x7f41340bd5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:51.665 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:51.665 7f413971f700 1 --2- 172.21.2.112:0/4231270730 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f41341094b0 0x7f41340bdbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:51.665 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:51.665 7f413971f700 1 --2- 172.21.2.112:0/4231270730 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f41340bba80 0x7f4134108e90 unknown :-1 s=CLOSED pgs=2386 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:51.665 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:51.665 7f413971f700 1 -- 172.21.2.112:0/4231270730 >> 172.21.2.112:0/4231270730 conn(0x7f41340bc6c0 msgr2=0x7f41340b1f70 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:51.666 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:51.665 7f413971f700 1 -- 172.21.2.112:0/4231270730 shutdown_connections 2021-02-10T11:42:51.666 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:51.665 7f413971f700 1 -- 172.21.2.112:0/4231270730 wait complete. 2021-02-10T11:42:51.672 INFO:tasks.cephfs_test_runner:test_non_clone_status (tasks.cephfs.test_volumes.TestVolumes) ... ok 2021-02-10T11:42:51.673 INFO:tasks.cephfs_test_runner:Starting test: test_nonexistent_subvolume_group_create (tasks.cephfs.test_volumes.TestVolumes) 2021-02-10T11:42:51.673 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestVolumes.test_nonexistent_subvolume_group_create' 2021-02-10T11:42:52.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:52.084 7fe1e5a70700 1 Processor -- start 2021-02-10T11:42:52.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:52.084 7fe1e5a70700 1 -- start start 2021-02-10T11:42:52.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:52.084 7fe1e5a70700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fe1e00d1c70 0x7fe1e00f20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:52.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:52.084 7fe1e5a70700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1e00f0b10 con 0x7fe1e00d0340 2021-02-10T11:42:52.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:52.084 7fe1e5a70700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1e01b0be0 con 0x7fe1e00d1870 2021-02-10T11:42:52.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:52.084 7fe1e5a70700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1e00e5ed0 con 0x7fe1e00d1c70 2021-02-10T11:42:52.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:52.085 7fe1dffff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fe1e00d1c70 0x7fe1e00f20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:52.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:52.085 7fe1dffff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fe1e00d1c70 0x7fe1e00f20a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:56588/0 (socket says 172.21.2.112:56588) 2021-02-10T11:42:52.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:52.085 7fe1dffff700 1 -- 172.21.2.112:0/2377344910 learned_addr learned my addr 172.21.2.112:0/2377344910 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:42:52.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:52.085 7fe1dffff700 1 -- 172.21.2.112:0/2377344910 >> v1:172.21.2.120:6789/0 conn(0x7fe1e00d1870 legacy=0x7fe1e01b03f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:42:52.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:52.085 7fe1dffff700 1 -- 172.21.2.112:0/2377344910 >> v1:172.21.2.112:6789/0 conn(0x7fe1e00d0340 legacy=0x7fe1e00ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:52.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:52.085 7fe1dffff700 1 -- 172.21.2.112:0/2377344910 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1e00efe30 con 0x7fe1e00d1c70 2021-02-10T11:42:52.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:52.085 7fe1dffff700 1 --2- 172.21.2.112:0/2377344910 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe1e00d1c70 0x7fe1e00f20a0 secure :-1 s=READY pgs=2388 cs=0 l=1 rev1=1 rx=0x7fe1d8006f00 tx=0x7fe1d8009f10).ready entity=mon.0 client_cookie=a106e44112cdf027 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:52.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:52.085 7fe1de7fc700 1 -- 172.21.2.112:0/2377344910 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe1d800b3b0 con 0x7fe1e00d1c70 2021-02-10T11:42:52.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:52.085 7fe1de7fc700 1 -- 172.21.2.112:0/2377344910 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1d800b630 con 0x7fe1e00d1c70 2021-02-10T11:42:52.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:52.085 7fe1de7fc700 1 -- 172.21.2.112:0/2377344910 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe1d8000f90 con 0x7fe1e00d1c70 2021-02-10T11:42:52.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:52.085 7fe1e5a70700 1 -- 172.21.2.112:0/2377344910 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe1e00d1c70 msgr2=0x7fe1e00f20a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:52.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:52.085 7fe1e5a70700 1 --2- 172.21.2.112:0/2377344910 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe1e00d1c70 0x7fe1e00f20a0 secure :-1 s=READY pgs=2388 cs=0 l=1 rev1=1 rx=0x7fe1d8006f00 tx=0x7fe1d8009f10).stop 2021-02-10T11:42:52.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:52.086 7fe1e5a70700 1 -- 172.21.2.112:0/2377344910 shutdown_connections 2021-02-10T11:42:52.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:52.086 7fe1e5a70700 1 --2- 172.21.2.112:0/2377344910 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe1e00d1c70 0x7fe1e00f20a0 unknown :-1 s=CLOSED pgs=2388 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:52.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:52.086 7fe1e5a70700 1 -- 172.21.2.112:0/2377344910 >> 172.21.2.112:0/2377344910 conn(0x7fe1e00ce450 msgr2=0x7fe1e00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:52.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:52.086 7fe1e5a70700 1 -- 172.21.2.112:0/2377344910 shutdown_connections 2021-02-10T11:42:52.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:52.086 7fe1e5a70700 1 -- 172.21.2.112:0/2377344910 wait complete. 2021-02-10T11:42:52.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:52.101 7fe1e5a70700 1 Processor -- start 2021-02-10T11:42:52.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:52.101 7fe1e5a70700 1 -- start start 2021-02-10T11:42:52.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:52.102 7fe1e5a70700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe1e00c0250 0x7fe1e00e8d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:52.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:52.102 7fe1e5a70700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe1e00e9320 0x7fe1e00e9720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:52.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:52.102 7fe1e5a70700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe1e00c1070 0x7fe1e00c1470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:52.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:52.102 7fe1e5a70700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1e00c0650 con 0x7fe1e00c0250 2021-02-10T11:42:52.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:52.102 7fe1e5a70700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1e008ec70 con 0x7fe1e00c1070 2021-02-10T11:42:52.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:52.102 7fe1e5a70700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1e008f110 con 0x7fe1e00e9320 2021-02-10T11:42:52.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:52.102 7fe1df7fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe1e00c0250 0x7fe1e00e8d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:52.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:52.102 7fe1df7fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe1e00c0250 0x7fe1e00e8d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:56592/0 (socket says 172.21.2.112:56592) 2021-02-10T11:42:52.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:52.102 7fe1df7fe700 1 -- 172.21.2.112:0/907218013 learned_addr learned my addr 172.21.2.112:0/907218013 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:42:52.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:52.102 7fe1dffff700 1 --2- 172.21.2.112:0/907218013 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe1e00c1070 0x7fe1e00c1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:52.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:52.102 7fe1deffd700 1 --2- 172.21.2.112:0/907218013 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe1e00e9320 0x7fe1e00e9720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:52.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:52.102 7fe1dffff700 1 -- 172.21.2.112:0/907218013 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe1e00e9320 msgr2=0x7fe1e00e9720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:52.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:52.102 7fe1dffff700 1 --2- 172.21.2.112:0/907218013 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe1e00e9320 0x7fe1e00e9720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:52.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:52.102 7fe1dffff700 1 -- 172.21.2.112:0/907218013 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe1e00c0250 msgr2=0x7fe1e00e8d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:52.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:52.102 7fe1dffff700 1 --2- 172.21.2.112:0/907218013 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe1e00c0250 0x7fe1e00e8d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:52.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:52.102 7fe1dffff700 1 -- 172.21.2.112:0/907218013 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1e00c1a90 con 0x7fe1e00c1070 2021-02-10T11:42:52.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:52.102 7fe1df7fe700 1 --2- 172.21.2.112:0/907218013 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe1e00c0250 0x7fe1e00e8d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:42:52.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:52.103 7fe1dffff700 1 --2- 172.21.2.112:0/907218013 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe1e00c1070 0x7fe1e00c1470 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 rx=0x7fe1d800afc0 tx=0x7fe1d8002c20).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:52.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:52.103 7fe1d7fff700 1 -- 172.21.2.112:0/907218013 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe1d8005c20 con 0x7fe1e00c1070 2021-02-10T11:42:52.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:52.103 7fe1d7fff700 1 -- 172.21.2.112:0/907218013 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1d8005fa0 con 0x7fe1e00c1070 2021-02-10T11:42:52.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:52.103 7fe1d7fff700 1 -- 172.21.2.112:0/907218013 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe1d80064c0 con 0x7fe1e00c1070 2021-02-10T11:42:52.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:52.103 7fe1e5a70700 1 -- 172.21.2.112:0/907218013 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1e00c1a90 con 0x7fe1e00c1070 2021-02-10T11:42:52.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:52.103 7fe1e5a70700 1 -- 172.21.2.112:0/907218013 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1e00c1a90 con 0x7fe1e00c1070 2021-02-10T11:42:52.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:52.103 7fe1d7fff700 1 -- 172.21.2.112:0/907218013 <== mon.1 v2:172.21.2.120:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe1d8006390 con 0x7fe1e00c1070 2021-02-10T11:42:52.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:52.103 7fe1d7fff700 1 --2- 172.21.2.112:0/907218013 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe1c40186c0 0x7fe1c401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:52.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:52.103 7fe1d7fff700 1 -- 172.21.2.112:0/907218013 <== mon.1 v2:172.21.2.120:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fe1d80327f0 con 0x7fe1e00c1070 2021-02-10T11:42:52.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:52.104 7fe1df7fe700 1 --2- 172.21.2.112:0/907218013 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe1c40186c0 0x7fe1c401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:52.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:52.104 7fe1df7fe700 1 --2- 172.21.2.112:0/907218013 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe1c40186c0 0x7fe1c401abc0 crc :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:52.134 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:52.134 7fe1e5a70700 1 -- 172.21.2.112:0/907218013 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1e00c0650 con 0x7fe1e00c1070 2021-02-10T11:42:52.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:52.138 7fe1d7fff700 1 -- 172.21.2.112:0/907218013 <== mon.1 v2:172.21.2.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe1d8006160 con 0x7fe1e00c1070 2021-02-10T11:42:52.275 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:52.275 7fe1e5a70700 1 -- 172.21.2.112:0/907218013 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_nonexistent_subvolume_group_create"]} v 0) v1 -- 0x7fe1e00c0650 con 0x7fe1e00c1070 2021-02-10T11:42:52.661 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:52.662 7fe1d7fff700 1 -- 172.21.2.112:0/907218013 <== mon.1 v2:172.21.2.120:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_nonexistent_subvolume_group_create"]}]=0 v642) v1 ==== 159+0+0 (secure 0 0 0) 0x7fe1d8006390 con 0x7fe1e00c1070 2021-02-10T11:42:52.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:52.694 7fe1e5a70700 1 -- 172.21.2.112:0/907218013 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe1c40186c0 msgr2=0x7fe1c401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:52.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:52.694 7fe1e5a70700 1 --2- 172.21.2.112:0/907218013 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe1c40186c0 0x7fe1c401abc0 crc :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:52.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:52.694 7fe1e5a70700 1 -- 172.21.2.112:0/907218013 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe1e00c1070 msgr2=0x7fe1e00c1470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:52.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:52.694 7fe1e5a70700 1 --2- 172.21.2.112:0/907218013 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe1e00c1070 0x7fe1e00c1470 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 rx=0x7fe1d800afc0 tx=0x7fe1d8002c20).stop 2021-02-10T11:42:52.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:52.694 7fe1e5a70700 1 -- 172.21.2.112:0/907218013 shutdown_connections 2021-02-10T11:42:52.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:52.694 7fe1e5a70700 1 --2- 172.21.2.112:0/907218013 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe1c40186c0 0x7fe1c401abc0 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:52.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:52.694 7fe1e5a70700 1 --2- 172.21.2.112:0/907218013 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe1e00c1070 0x7fe1e00c1470 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:52.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:52.694 7fe1e5a70700 1 --2- 172.21.2.112:0/907218013 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe1e00e9320 0x7fe1e00e9720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:52.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:52.694 7fe1e5a70700 1 --2- 172.21.2.112:0/907218013 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe1e00c0250 0x7fe1e00e8d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:52.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:52.694 7fe1e5a70700 1 -- 172.21.2.112:0/907218013 >> 172.21.2.112:0/907218013 conn(0x7fe1e00b2470 msgr2=0x7fe1e00b6970 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:52.695 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:52.694 7fe1e5a70700 1 -- 172.21.2.112:0/907218013 shutdown_connections 2021-02-10T11:42:52.695 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:52.695 7fe1e5a70700 1 -- 172.21.2.112:0/907218013 wait complete. 2021-02-10T11:42:52.702 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:42:53.135 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:53.118 7f3170bd9700 1 Processor -- start 2021-02-10T11:42:53.135 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:53.119 7f3170bd9700 1 -- start start 2021-02-10T11:42:53.135 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:53.119 7f3170bd9700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f316c0d1c60 0x7f316c0cef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:53.135 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:53.119 7f3170bd9700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f316c0cf570 0x7f316c0cf970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:53.135 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:53.119 7f3170bd9700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f316c0fe1b0 con 0x7f316c0d1860 2021-02-10T11:42:53.135 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:53.119 7f3170bd9700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f316c1b0ca0 con 0x7f316c0d1c60 2021-02-10T11:42:53.135 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:53.119 7f3170bd9700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f316c109a50 con 0x7f316c0cf570 2021-02-10T11:42:53.135 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:53.119 7f316b05f700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f316c0cf570 0x7f316c0cf970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:53.135 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:53.119 7f316b05f700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f316c0cf570 0x7f316c0cf970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3301/0 says I am v2:172.21.2.112:56284/0 (socket says 172.21.2.112:56284) 2021-02-10T11:42:53.136 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:53.119 7f316b05f700 1 -- 172.21.2.112:0/233092898 learned_addr learned my addr 172.21.2.112:0/233092898 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:42:53.136 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:53.119 7f316a05d700 1 --2- 172.21.2.112:0/233092898 >> v2:172.21.2.120:3300/0 conn(0x7f316c0d1c60 0x7f316c0cef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:53.136 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:53.120 7f316b05f700 1 -- 172.21.2.112:0/233092898 >> v2:172.21.2.120:3300/0 conn(0x7f316c0d1c60 msgr2=0x7f316c0cef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:53.136 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:53.120 7f316b05f700 1 --2- 172.21.2.112:0/233092898 >> v2:172.21.2.120:3300/0 conn(0x7f316c0d1c60 0x7f316c0cef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:53.136 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:53.120 7f316b05f700 1 -- 172.21.2.112:0/233092898 >> v1:172.21.2.120:6789/0 conn(0x7f316c0d1860 legacy=0x7f316c1b04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:53.136 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:53.120 7f316b05f700 1 -- 172.21.2.112:0/233092898 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f316c088070 con 0x7f316c0cf570 2021-02-10T11:42:53.136 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:53.120 7f316b05f700 1 --2- 172.21.2.112:0/233092898 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f316c0cf570 0x7f316c0cf970 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7f3164006aa0 tx=0x7f3164009f20).ready entity=mon.2 client_cookie=39d28baef457ac32 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:53.136 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:53.120 7f316985c700 1 -- 172.21.2.112:0/233092898 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f316400b5c0 con 0x7f316c0cf570 2021-02-10T11:42:53.136 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:53.120 7f316985c700 1 -- 172.21.2.112:0/233092898 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3164003030 con 0x7f316c0cf570 2021-02-10T11:42:53.136 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:53.120 7f316985c700 1 -- 172.21.2.112:0/233092898 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f316400b9f0 con 0x7f316c0cf570 2021-02-10T11:42:53.136 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:53.120 7f3170bd9700 1 -- 172.21.2.112:0/233092898 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f316c0cf570 msgr2=0x7f316c0cf970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:53.136 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:53.120 7f3170bd9700 1 --2- 172.21.2.112:0/233092898 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f316c0cf570 0x7f316c0cf970 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7f3164006aa0 tx=0x7f3164009f20).stop 2021-02-10T11:42:53.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:53.121 7f3170bd9700 1 -- 172.21.2.112:0/233092898 shutdown_connections 2021-02-10T11:42:53.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:53.121 7f3170bd9700 1 --2- 172.21.2.112:0/233092898 >> v2:172.21.2.120:3300/0 conn(0x7f316c0d1c60 0x7f316c0cef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:53.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:53.121 7f3170bd9700 1 --2- 172.21.2.112:0/233092898 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f316c0cf570 0x7f316c0cf970 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:53.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:53.121 7f3170bd9700 1 -- 172.21.2.112:0/233092898 >> 172.21.2.112:0/233092898 conn(0x7f316c0ce410 msgr2=0x7f316c0d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:53.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:53.121 7f3170bd9700 1 -- 172.21.2.112:0/233092898 shutdown_connections 2021-02-10T11:42:53.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:53.121 7f3170bd9700 1 -- 172.21.2.112:0/233092898 wait complete. 2021-02-10T11:42:53.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:53.135 7f3170bd9700 1 Processor -- start 2021-02-10T11:42:53.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:53.135 7f3170bd9700 1 -- start start 2021-02-10T11:42:53.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:53.136 7f3170bd9700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f316c0badd0 0x7f316c09eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:53.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:53.136 7f3170bd9700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f316c0adf50 0x7f316c09f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:53.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:53.136 7f3170bd9700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f316c1a7470 0x7f316c1a7870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:53.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:53.136 7f3170bd9700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f316c09e5a0 con 0x7f316c0adf50 2021-02-10T11:42:53.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:53.136 7f3170bd9700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f316c09f6e0 con 0x7f316c1a7470 2021-02-10T11:42:53.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:53.136 7f3170bd9700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f316c0bb220 con 0x7f316c0badd0 2021-02-10T11:42:53.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:53.136 7f316a85e700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f316c0badd0 0x7f316c09eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:53.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:53.136 7f316a05d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f316c0adf50 0x7f316c09f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:53.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:53.136 7f316a85e700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f316c0badd0 0x7f316c09eaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3301/0 says I am v2:172.21.2.112:56286/0 (socket says 172.21.2.112:56286) 2021-02-10T11:42:53.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:53.136 7f316a85e700 1 -- 172.21.2.112:0/3291988087 learned_addr learned my addr 172.21.2.112:0/3291988087 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:42:53.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:53.136 7f316a05d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f316c0adf50 0x7f316c09f0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:56606/0 (socket says 172.21.2.112:56606) 2021-02-10T11:42:53.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:53.136 7f316b05f700 1 --2- 172.21.2.112:0/3291988087 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f316c1a7470 0x7f316c1a7870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:53.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:53.136 7f316a85e700 1 -- 172.21.2.112:0/3291988087 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f316c1a7470 msgr2=0x7f316c1a7870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:53.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:53.136 7f316a85e700 1 --2- 172.21.2.112:0/3291988087 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f316c1a7470 0x7f316c1a7870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:53.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:53.136 7f316a85e700 1 -- 172.21.2.112:0/3291988087 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f316c0adf50 msgr2=0x7f316c09f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:53.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:53.136 7f316a85e700 1 --2- 172.21.2.112:0/3291988087 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f316c0adf50 0x7f316c09f0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:53.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:53.136 7f316a85e700 1 -- 172.21.2.112:0/3291988087 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f316c1a7e90 con 0x7f316c0badd0 2021-02-10T11:42:53.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:53.137 7f316a85e700 1 --2- 172.21.2.112:0/3291988087 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f316c0badd0 0x7f316c09eaa0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7f315c006f00 tx=0x7f315c00b940).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:53.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:53.137 7f31637fe700 1 -- 172.21.2.112:0/3291988087 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f315c00bd30 con 0x7f316c0badd0 2021-02-10T11:42:53.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:53.137 7f31637fe700 1 -- 172.21.2.112:0/3291988087 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f315c000ca0 con 0x7f316c0badd0 2021-02-10T11:42:53.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:53.137 7f31637fe700 1 -- 172.21.2.112:0/3291988087 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f315c0022a0 con 0x7f316c0badd0 2021-02-10T11:42:53.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:53.137 7f3170bd9700 1 -- 172.21.2.112:0/3291988087 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f316c1a7e90 con 0x7f316c0badd0 2021-02-10T11:42:53.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:53.137 7f3170bd9700 1 -- 172.21.2.112:0/3291988087 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f316c1a7e90 con 0x7f316c0badd0 2021-02-10T11:42:53.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:53.137 7f31637fe700 1 -- 172.21.2.112:0/3291988087 <== mon.2 v2:172.21.2.120:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f315c0052e0 con 0x7f316c0badd0 2021-02-10T11:42:53.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:53.137 7f31637fe700 1 --2- 172.21.2.112:0/3291988087 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f31500186c0 0x7f315001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:53.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:53.137 7f31637fe700 1 -- 172.21.2.112:0/3291988087 <== mon.2 v2:172.21.2.120:3301/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f315c0323e0 con 0x7f316c0badd0 2021-02-10T11:42:53.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:53.138 7f316a05d700 1 --2- 172.21.2.112:0/3291988087 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f31500186c0 0x7f315001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:53.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:53.138 7f316a05d700 1 --2- 172.21.2.112:0/3291988087 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f31500186c0 0x7f315001abc0 crc :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:53.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:53.169 7f3170bd9700 1 -- 172.21.2.112:0/3291988087 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f316c09f6e0 con 0x7f316c0badd0 2021-02-10T11:42:53.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:53.172 7f31637fe700 1 -- 172.21.2.112:0/3291988087 <== mon.2 v2:172.21.2.120:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f315c02bd30 con 0x7f316c0badd0 2021-02-10T11:42:53.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:53.323 7f3170bd9700 1 -- 172.21.2.112:0/3291988087 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f316c09f6e0 con 0x7f316c0badd0 2021-02-10T11:42:53.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:53.330 7f31637fe700 1 -- 172.21.2.112:0/3291988087 <== mon.2 v2:172.21.2.120:3301/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v34)=0 v34) v1 ==== 122+0+0 (secure 0 0 0) 0x7f315c02bd30 con 0x7f316c0badd0 2021-02-10T11:42:53.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:53.332 7f3170bd9700 1 -- 172.21.2.112:0/3291988087 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f31500186c0 msgr2=0x7f315001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:53.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:53.332 7f3170bd9700 1 --2- 172.21.2.112:0/3291988087 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f31500186c0 0x7f315001abc0 crc :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:53.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:53.332 7f3170bd9700 1 -- 172.21.2.112:0/3291988087 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f316c0badd0 msgr2=0x7f316c09eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:53.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:53.332 7f3170bd9700 1 --2- 172.21.2.112:0/3291988087 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f316c0badd0 0x7f316c09eaa0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7f315c006f00 tx=0x7f315c00b940).stop 2021-02-10T11:42:53.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:53.332 7f3170bd9700 1 -- 172.21.2.112:0/3291988087 shutdown_connections 2021-02-10T11:42:53.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:53.332 7f3170bd9700 1 --2- 172.21.2.112:0/3291988087 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f31500186c0 0x7f315001abc0 unknown :-1 s=CLOSED pgs=1091 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:53.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:53.332 7f3170bd9700 1 --2- 172.21.2.112:0/3291988087 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f316c1a7470 0x7f316c1a7870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:53.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:53.332 7f3170bd9700 1 --2- 172.21.2.112:0/3291988087 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f316c0adf50 0x7f316c09f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:53.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:53.332 7f3170bd9700 1 --2- 172.21.2.112:0/3291988087 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f316c0badd0 0x7f316c09eaa0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:53.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:53.332 7f3170bd9700 1 -- 172.21.2.112:0/3291988087 >> 172.21.2.112:0/3291988087 conn(0x7f316c0bee90 msgr2=0x7f316c0bb880 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:53.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:53.333 7f3170bd9700 1 -- 172.21.2.112:0/3291988087 shutdown_connections 2021-02-10T11:42:53.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:53.333 7f3170bd9700 1 -- 172.21.2.112:0/3291988087 wait complete. 2021-02-10T11:42:53.340 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:42:53.340 DEBUG:teuthology.orchestra.run.gibba012:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:42:53.658 DEBUG:teuthology.orchestra.run.gibba012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:42:53.661 DEBUG:teuthology.orchestra.run.gibba020:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:42:53.687 DEBUG:teuthology.orchestra.run.gibba020:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:42:54.005 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:42:54.005 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:42:54.006 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:42:54.006 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:42:54.342 INFO:teuthology.orchestra.run.gibba012.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.0’: No such file or directory 2021-02-10T11:42:54.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:42:54.343 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2021-02-10T11:42:54.343 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:42:54.343 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:42:54.343 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:42:54.680 INFO:teuthology.orchestra.run.gibba012.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: No such file or directory 2021-02-10T11:42:54.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:42:54.680 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2021-02-10T11:42:54.680 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:42:54.680 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:42:54.680 DEBUG:teuthology.orchestra.run.gibba020:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:42:55.015 INFO:teuthology.orchestra.run.gibba020.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:42:55.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:42:55.015 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:42:55.015 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:42:55.015 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:42:55.016 DEBUG:teuthology.orchestra.run.gibba020:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:42:55.350 INFO:teuthology.orchestra.run.gibba020.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:42:55.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:42:55.351 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:42:55.351 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json-pretty 2021-02-10T11:42:55.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:55.764 7f41d89a2700 1 Processor -- start 2021-02-10T11:42:55.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:55.764 7f41d89a2700 1 -- start start 2021-02-10T11:42:55.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:55.764 7f41d89a2700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f41d40d1870 0x7f41d41b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:55.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:55.764 7f41d89a2700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f41d40f0650 0x7f41d40cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:55.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:55.764 7f41d89a2700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41d40f0b10 con 0x7f41d41b0a10 2021-02-10T11:42:55.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:55.764 7f41d89a2700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41d40fe230 con 0x7f41d40f0650 2021-02-10T11:42:55.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:55.764 7f41d89a2700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41d40e5ed0 con 0x7f41d40d1870 2021-02-10T11:42:55.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:55.764 7f41d2e1d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f41d40f0650 0x7f41d40cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:55.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:55.764 7f41d2e1d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f41d40f0650 0x7f41d40cfb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:56612/0 (socket says 172.21.2.112:56612) 2021-02-10T11:42:55.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:55.765 7f41d2e1d700 1 -- 172.21.2.112:0/3687127074 learned_addr learned my addr 172.21.2.112:0/3687127074 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:42:55.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:55.765 7f41d2e1d700 1 -- 172.21.2.112:0/3687127074 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f41d40d1870 msgr2=0x7f41d41b03f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:42:55.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:55.765 7f41d2e1d700 1 --2- 172.21.2.112:0/3687127074 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f41d40d1870 0x7f41d41b03f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:55.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:55.765 7f41d2e1d700 1 -- 172.21.2.112:0/3687127074 >> v1:172.21.2.112:6789/0 conn(0x7f41d41b0a10 legacy=0x7f41d40cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:55.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:55.765 7f41d2e1d700 1 -- 172.21.2.112:0/3687127074 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41d40f20a0 con 0x7f41d40f0650 2021-02-10T11:42:55.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:55.765 7f41d2e1d700 1 --2- 172.21.2.112:0/3687127074 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f41d40f0650 0x7f41d40cfb20 secure :-1 s=READY pgs=2390 cs=0 l=1 rev1=1 rx=0x7f41cc006f00 tx=0x7f41cc009f10).ready entity=mon.0 client_cookie=661a8e2b1a6eca68 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:55.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:55.765 7f41d161a700 1 -- 172.21.2.112:0/3687127074 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f41cc00b3b0 con 0x7f41d40f0650 2021-02-10T11:42:55.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:55.765 7f41d161a700 1 -- 172.21.2.112:0/3687127074 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41cc00b630 con 0x7f41d40f0650 2021-02-10T11:42:55.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:55.765 7f41d161a700 1 -- 172.21.2.112:0/3687127074 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f41cc00bb50 con 0x7f41d40f0650 2021-02-10T11:42:55.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:55.765 7f41d89a2700 1 -- 172.21.2.112:0/3687127074 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f41d40f0650 msgr2=0x7f41d40cfb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:55.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:55.765 7f41d89a2700 1 --2- 172.21.2.112:0/3687127074 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f41d40f0650 0x7f41d40cfb20 secure :-1 s=READY pgs=2390 cs=0 l=1 rev1=1 rx=0x7f41cc006f00 tx=0x7f41cc009f10).stop 2021-02-10T11:42:55.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:55.766 7f41d89a2700 1 -- 172.21.2.112:0/3687127074 shutdown_connections 2021-02-10T11:42:55.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:55.766 7f41d89a2700 1 --2- 172.21.2.112:0/3687127074 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f41d40f0650 0x7f41d40cfb20 unknown :-1 s=CLOSED pgs=2390 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:55.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:55.766 7f41d89a2700 1 --2- 172.21.2.112:0/3687127074 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f41d40d1870 0x7f41d41b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:55.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:55.766 7f41d89a2700 1 -- 172.21.2.112:0/3687127074 >> 172.21.2.112:0/3687127074 conn(0x7f41d40ce450 msgr2=0x7f41d40d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:55.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:55.766 7f41d89a2700 1 -- 172.21.2.112:0/3687127074 shutdown_connections 2021-02-10T11:42:55.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:55.766 7f41d89a2700 1 -- 172.21.2.112:0/3687127074 wait complete. 2021-02-10T11:42:55.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:55.780 7f41d89a2700 1 Processor -- start 2021-02-10T11:42:55.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:55.780 7f41d89a2700 1 -- start start 2021-02-10T11:42:55.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:55.781 7f41d89a2700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f41d40aefa0 0x7f41d40af3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:55.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:55.781 7f41d89a2700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f41d40af9c0 0x7f41d409b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:55.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:55.781 7f41d89a2700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f41d409bb00 0x7f41d409bf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:55.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:55.781 7f41d89a2700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41d418fab0 con 0x7f41d409bb00 2021-02-10T11:42:55.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:55.781 7f41d89a2700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41d40afdc0 con 0x7f41d40af9c0 2021-02-10T11:42:55.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:55.781 7f41d89a2700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41d418f630 con 0x7f41d40aefa0 2021-02-10T11:42:55.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:55.781 7f41d2e1d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f41d409bb00 0x7f41d409bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:55.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:55.781 7f41d2e1d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f41d409bb00 0x7f41d409bf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:56622/0 (socket says 172.21.2.112:56622) 2021-02-10T11:42:55.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:55.781 7f41d2e1d700 1 -- 172.21.2.112:0/3557384527 learned_addr learned my addr 172.21.2.112:0/3557384527 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:42:55.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:55.781 7f41d2e1d700 1 -- 172.21.2.112:0/3557384527 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f41d40aefa0 msgr2=0x7f41d40af3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:55.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:55.781 7f41d2e1d700 1 --2- 172.21.2.112:0/3557384527 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f41d40aefa0 0x7f41d40af3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:55.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:55.781 7f41d2e1d700 1 -- 172.21.2.112:0/3557384527 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f41d40af9c0 msgr2=0x7f41d409b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:55.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:55.781 7f41d2e1d700 1 --2- 172.21.2.112:0/3557384527 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f41d40af9c0 0x7f41d409b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:55.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:55.781 7f41d2e1d700 1 -- 172.21.2.112:0/3557384527 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41d409cce0 con 0x7f41d409bb00 2021-02-10T11:42:55.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:55.781 7f41d2e1d700 1 --2- 172.21.2.112:0/3557384527 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f41d409bb00 0x7f41d409bf00 secure :-1 s=READY pgs=2391 cs=0 l=1 rev1=1 rx=0x7f41cc00b9b0 tx=0x7f41cc00bd50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:55.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:55.782 7f41caffd700 1 -- 172.21.2.112:0/3557384527 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f41cc003030 con 0x7f41d409bb00 2021-02-10T11:42:55.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:55.782 7f41caffd700 1 -- 172.21.2.112:0/3557384527 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41cc00c2f0 con 0x7f41d409bb00 2021-02-10T11:42:55.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:55.782 7f41d89a2700 1 -- 172.21.2.112:0/3557384527 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41d409cce0 con 0x7f41d409bb00 2021-02-10T11:42:55.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:55.782 7f41d89a2700 1 -- 172.21.2.112:0/3557384527 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41d409d270 con 0x7f41d409bb00 2021-02-10T11:42:55.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:55.782 7f41caffd700 1 -- 172.21.2.112:0/3557384527 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f41cc00c810 con 0x7f41d409bb00 2021-02-10T11:42:55.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:55.783 7f41caffd700 1 -- 172.21.2.112:0/3557384527 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f41cc001270 con 0x7f41d409bb00 2021-02-10T11:42:55.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:55.783 7f41caffd700 1 --2- 172.21.2.112:0/3557384527 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f41b8018730 0x7f41b801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:55.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:55.783 7f41caffd700 1 -- 172.21.2.112:0/3557384527 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f41cc032b10 con 0x7f41d409bb00 2021-02-10T11:42:55.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:55.783 7f41d261c700 1 --2- 172.21.2.112:0/3557384527 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f41b8018730 0x7f41b801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:55.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:55.783 7f41d261c700 1 --2- 172.21.2.112:0/3557384527 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f41b8018730 0x7f41b801ac10 crc :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:55.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:55.814 7f41d89a2700 1 -- 172.21.2.112:0/3557384527 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41d40ee060 con 0x7f41d409bb00 2021-02-10T11:42:55.816 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:55.817 7f41caffd700 1 -- 172.21.2.112:0/3557384527 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f41cc009450 con 0x7f41d409bb00 2021-02-10T11:42:55.941 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:55.942 7f41d89a2700 1 -- 172.21.2.112:0/3557384527 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f41d40ce450 con 0x7f41d409bb00 2021-02-10T11:42:55.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:55.942 7f41caffd700 1 -- 172.21.2.112:0/3557384527 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v125) v1 ==== 81+0+30374 (secure 0 0 0) 0x7f41cc001270 con 0x7f41d409bb00 2021-02-10T11:42:55.942 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:42:55.943 INFO:teuthology.orchestra.run.gibba012.stdout:{ 2021-02-10T11:42:55.943 INFO:teuthology.orchestra.run.gibba012.stdout: "epoch": 125, 2021-02-10T11:42:55.943 INFO:teuthology.orchestra.run.gibba012.stdout: "fsid": "f14f004e-0699-4584-ad81-3754258493de", 2021-02-10T11:42:55.943 INFO:teuthology.orchestra.run.gibba012.stdout: "created": "2021-02-10 11:13:26.501424", 2021-02-10T11:42:55.943 INFO:teuthology.orchestra.run.gibba012.stdout: "modified": "2021-02-10 11:42:20.353491", 2021-02-10T11:42:55.943 INFO:teuthology.orchestra.run.gibba012.stdout: "last_up_change": "2021-02-10 11:13:52.780316", 2021-02-10T11:42:55.943 INFO:teuthology.orchestra.run.gibba012.stdout: "last_in_change": "2021-02-10 11:13:52.780316", 2021-02-10T11:42:55.943 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-02-10T11:42:55.943 INFO:teuthology.orchestra.run.gibba012.stdout: "flags_num": 5799936, 2021-02-10T11:42:55.943 INFO:teuthology.orchestra.run.gibba012.stdout: "flags_set": [ 2021-02-10T11:42:55.943 INFO:teuthology.orchestra.run.gibba012.stdout: "pglog_hardlimit", 2021-02-10T11:42:55.943 INFO:teuthology.orchestra.run.gibba012.stdout: "purged_snapdirs", 2021-02-10T11:42:55.944 INFO:teuthology.orchestra.run.gibba012.stdout: "recovery_deletes", 2021-02-10T11:42:55.944 INFO:teuthology.orchestra.run.gibba012.stdout: "sortbitwise" 2021-02-10T11:42:55.944 INFO:teuthology.orchestra.run.gibba012.stdout: ], 2021-02-10T11:42:55.944 INFO:teuthology.orchestra.run.gibba012.stdout: "crush_version": 4, 2021-02-10T11:42:55.944 INFO:teuthology.orchestra.run.gibba012.stdout: "full_ratio": 0.94999998807907104, 2021-02-10T11:42:55.944 INFO:teuthology.orchestra.run.gibba012.stdout: "backfillfull_ratio": 0.89999997615814209, 2021-02-10T11:42:55.944 INFO:teuthology.orchestra.run.gibba012.stdout: "nearfull_ratio": 0.85000002384185791, 2021-02-10T11:42:55.944 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_snapshot": "", 2021-02-10T11:42:55.944 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_max": 27, 2021-02-10T11:42:55.944 INFO:teuthology.orchestra.run.gibba012.stdout: "max_osd": 8, 2021-02-10T11:42:55.944 INFO:teuthology.orchestra.run.gibba012.stdout: "require_min_compat_client": "jewel", 2021-02-10T11:42:55.944 INFO:teuthology.orchestra.run.gibba012.stdout: "min_compat_client": "jewel", 2021-02-10T11:42:55.944 INFO:teuthology.orchestra.run.gibba012.stdout: "require_osd_release": "nautilus", 2021-02-10T11:42:55.945 INFO:teuthology.orchestra.run.gibba012.stdout: "pools": [ 2021-02-10T11:42:55.945 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:55.945 INFO:teuthology.orchestra.run.gibba012.stdout: "pool": 1, 2021-02-10T11:42:55.945 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_name": "rbd", 2021-02-10T11:42:55.945 INFO:teuthology.orchestra.run.gibba012.stdout: "create_time": "2021-02-10 11:13:57.508758", 2021-02-10T11:42:55.945 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:42:55.945 INFO:teuthology.orchestra.run.gibba012.stdout: "flags_names": "hashpspool", 2021-02-10T11:42:55.945 INFO:teuthology.orchestra.run.gibba012.stdout: "type": 1, 2021-02-10T11:42:55.945 INFO:teuthology.orchestra.run.gibba012.stdout: "size": 2, 2021-02-10T11:42:55.945 INFO:teuthology.orchestra.run.gibba012.stdout: "min_size": 1, 2021-02-10T11:42:55.945 INFO:teuthology.orchestra.run.gibba012.stdout: "crush_rule": 0, 2021-02-10T11:42:55.945 INFO:teuthology.orchestra.run.gibba012.stdout: "object_hash": 2, 2021-02-10T11:42:55.946 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:42:55.946 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num": 8, 2021-02-10T11:42:55.946 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num": 8, 2021-02-10T11:42:55.946 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num_target": 8, 2021-02-10T11:42:55.946 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_target": 8, 2021-02-10T11:42:55.946 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_pending": 8, 2021-02-10T11:42:55.946 INFO:teuthology.orchestra.run.gibba012.stdout: "last_pg_merge_meta": { 2021-02-10T11:42:55.946 INFO:teuthology.orchestra.run.gibba012.stdout: "source_pgid": "0.0", 2021-02-10T11:42:55.946 INFO:teuthology.orchestra.run.gibba012.stdout: "ready_epoch": 0, 2021-02-10T11:42:55.946 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_started": 0, 2021-02-10T11:42:55.946 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_clean": 0, 2021-02-10T11:42:55.946 INFO:teuthology.orchestra.run.gibba012.stdout: "source_version": "0'0", 2021-02-10T11:42:55.946 INFO:teuthology.orchestra.run.gibba012.stdout: "target_version": "0'0" 2021-02-10T11:42:55.947 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:55.947 INFO:teuthology.orchestra.run.gibba012.stdout: "last_change": "16", 2021-02-10T11:42:55.947 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend": "0", 2021-02-10T11:42:55.947 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:42:55.947 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:42:55.947 INFO:teuthology.orchestra.run.gibba012.stdout: "auid": 0, 2021-02-10T11:42:55.947 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:42:55.947 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_seq": 0, 2021-02-10T11:42:55.947 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_epoch": 0, 2021-02-10T11:42:55.947 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_snaps": [], 2021-02-10T11:42:55.947 INFO:teuthology.orchestra.run.gibba012.stdout: "removed_snaps": "[]", 2021-02-10T11:42:55.947 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_bytes": 0, 2021-02-10T11:42:55.947 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_objects": 0, 2021-02-10T11:42:55.948 INFO:teuthology.orchestra.run.gibba012.stdout: "tiers": [], 2021-02-10T11:42:55.948 INFO:teuthology.orchestra.run.gibba012.stdout: "tier_of": -1, 2021-02-10T11:42:55.948 INFO:teuthology.orchestra.run.gibba012.stdout: "read_tier": -1, 2021-02-10T11:42:55.948 INFO:teuthology.orchestra.run.gibba012.stdout: "write_tier": -1, 2021-02-10T11:42:55.948 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_mode": "none", 2021-02-10T11:42:55.948 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_bytes": 0, 2021-02-10T11:42:55.948 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_objects": 0, 2021-02-10T11:42:55.948 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:42:55.948 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:42:55.948 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:42:55.948 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_flush_age": 0, 2021-02-10T11:42:55.948 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_evict_age": 0, 2021-02-10T11:42:55.948 INFO:teuthology.orchestra.run.gibba012.stdout: "erasure_code_profile": "", 2021-02-10T11:42:55.949 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_params": { 2021-02-10T11:42:55.949 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "none" 2021-02-10T11:42:55.949 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:55.949 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_period": 0, 2021-02-10T11:42:55.949 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_count": 0, 2021-02-10T11:42:55.949 INFO:teuthology.orchestra.run.gibba012.stdout: "use_gmt_hitset": true, 2021-02-10T11:42:55.949 INFO:teuthology.orchestra.run.gibba012.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:42:55.949 INFO:teuthology.orchestra.run.gibba012.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:42:55.949 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:42:55.949 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:42:55.949 INFO:teuthology.orchestra.run.gibba012.stdout: "grade_table": [], 2021-02-10T11:42:55.949 INFO:teuthology.orchestra.run.gibba012.stdout: "stripe_width": 0, 2021-02-10T11:42:55.949 INFO:teuthology.orchestra.run.gibba012.stdout: "expected_num_objects": 0, 2021-02-10T11:42:55.949 INFO:teuthology.orchestra.run.gibba012.stdout: "fast_read": false, 2021-02-10T11:42:55.950 INFO:teuthology.orchestra.run.gibba012.stdout: "options": {}, 2021-02-10T11:42:55.950 INFO:teuthology.orchestra.run.gibba012.stdout: "application_metadata": { 2021-02-10T11:42:55.950 INFO:teuthology.orchestra.run.gibba012.stdout: "rbd": {} 2021-02-10T11:42:55.950 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:42:55.950 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:55.950 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:55.950 INFO:teuthology.orchestra.run.gibba012.stdout: "pool": 26, 2021-02-10T11:42:55.950 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_name": "cephfs_metadata", 2021-02-10T11:42:55.950 INFO:teuthology.orchestra.run.gibba012.stdout: "create_time": "2021-02-10 11:42:17.322526", 2021-02-10T11:42:55.950 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:42:55.950 INFO:teuthology.orchestra.run.gibba012.stdout: "flags_names": "hashpspool", 2021-02-10T11:42:55.950 INFO:teuthology.orchestra.run.gibba012.stdout: "type": 1, 2021-02-10T11:42:55.950 INFO:teuthology.orchestra.run.gibba012.stdout: "size": 2, 2021-02-10T11:42:55.951 INFO:teuthology.orchestra.run.gibba012.stdout: "min_size": 1, 2021-02-10T11:42:55.951 INFO:teuthology.orchestra.run.gibba012.stdout: "crush_rule": 0, 2021-02-10T11:42:55.951 INFO:teuthology.orchestra.run.gibba012.stdout: "object_hash": 2, 2021-02-10T11:42:55.951 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:42:55.951 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num": 8, 2021-02-10T11:42:55.951 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num": 8, 2021-02-10T11:42:55.951 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num_target": 8, 2021-02-10T11:42:55.951 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_target": 8, 2021-02-10T11:42:55.951 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_pending": 8, 2021-02-10T11:42:55.951 INFO:teuthology.orchestra.run.gibba012.stdout: "last_pg_merge_meta": { 2021-02-10T11:42:55.951 INFO:teuthology.orchestra.run.gibba012.stdout: "source_pgid": "0.0", 2021-02-10T11:42:55.951 INFO:teuthology.orchestra.run.gibba012.stdout: "ready_epoch": 0, 2021-02-10T11:42:55.952 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_started": 0, 2021-02-10T11:42:55.952 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_clean": 0, 2021-02-10T11:42:55.952 INFO:teuthology.orchestra.run.gibba012.stdout: "source_version": "0'0", 2021-02-10T11:42:55.952 INFO:teuthology.orchestra.run.gibba012.stdout: "target_version": "0'0" 2021-02-10T11:42:55.952 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:55.952 INFO:teuthology.orchestra.run.gibba012.stdout: "last_change": "124", 2021-02-10T11:42:55.952 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend": "0", 2021-02-10T11:42:55.952 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:42:55.952 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:42:55.952 INFO:teuthology.orchestra.run.gibba012.stdout: "auid": 0, 2021-02-10T11:42:55.952 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:42:55.952 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_seq": 0, 2021-02-10T11:42:55.953 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_epoch": 0, 2021-02-10T11:42:55.953 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_snaps": [], 2021-02-10T11:42:55.953 INFO:teuthology.orchestra.run.gibba012.stdout: "removed_snaps": "[]", 2021-02-10T11:42:55.953 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_bytes": 0, 2021-02-10T11:42:55.953 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_objects": 0, 2021-02-10T11:42:55.953 INFO:teuthology.orchestra.run.gibba012.stdout: "tiers": [], 2021-02-10T11:42:55.953 INFO:teuthology.orchestra.run.gibba012.stdout: "tier_of": -1, 2021-02-10T11:42:55.953 INFO:teuthology.orchestra.run.gibba012.stdout: "read_tier": -1, 2021-02-10T11:42:55.953 INFO:teuthology.orchestra.run.gibba012.stdout: "write_tier": -1, 2021-02-10T11:42:55.953 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_mode": "none", 2021-02-10T11:42:55.953 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_bytes": 0, 2021-02-10T11:42:55.953 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_objects": 0, 2021-02-10T11:42:55.953 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:42:55.954 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:42:55.954 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:42:55.954 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_flush_age": 0, 2021-02-10T11:42:55.954 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_evict_age": 0, 2021-02-10T11:42:55.954 INFO:teuthology.orchestra.run.gibba012.stdout: "erasure_code_profile": "", 2021-02-10T11:42:55.954 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_params": { 2021-02-10T11:42:55.955 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "none" 2021-02-10T11:42:55.955 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:55.955 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_period": 0, 2021-02-10T11:42:55.955 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_count": 0, 2021-02-10T11:42:55.955 INFO:teuthology.orchestra.run.gibba012.stdout: "use_gmt_hitset": true, 2021-02-10T11:42:55.955 INFO:teuthology.orchestra.run.gibba012.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:42:55.955 INFO:teuthology.orchestra.run.gibba012.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:42:55.955 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:42:55.955 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:42:55.955 INFO:teuthology.orchestra.run.gibba012.stdout: "grade_table": [], 2021-02-10T11:42:55.955 INFO:teuthology.orchestra.run.gibba012.stdout: "stripe_width": 0, 2021-02-10T11:42:55.955 INFO:teuthology.orchestra.run.gibba012.stdout: "expected_num_objects": 0, 2021-02-10T11:42:55.956 INFO:teuthology.orchestra.run.gibba012.stdout: "fast_read": false, 2021-02-10T11:42:55.956 INFO:teuthology.orchestra.run.gibba012.stdout: "options": { 2021-02-10T11:42:55.956 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_autoscale_bias": 4, 2021-02-10T11:42:55.956 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_min": 16, 2021-02-10T11:42:55.956 INFO:teuthology.orchestra.run.gibba012.stdout: "recovery_priority": 5 2021-02-10T11:42:55.956 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:55.956 INFO:teuthology.orchestra.run.gibba012.stdout: "application_metadata": { 2021-02-10T11:42:55.956 INFO:teuthology.orchestra.run.gibba012.stdout: "cephfs": { 2021-02-10T11:42:55.956 INFO:teuthology.orchestra.run.gibba012.stdout: "metadata": "cephfs" 2021-02-10T11:42:55.956 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:42:55.956 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:42:55.956 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:55.957 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:55.957 INFO:teuthology.orchestra.run.gibba012.stdout: "pool": 27, 2021-02-10T11:42:55.957 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_name": "cephfs_data", 2021-02-10T11:42:55.957 INFO:teuthology.orchestra.run.gibba012.stdout: "create_time": "2021-02-10 11:42:18.395545", 2021-02-10T11:42:55.957 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:42:55.957 INFO:teuthology.orchestra.run.gibba012.stdout: "flags_names": "hashpspool", 2021-02-10T11:42:55.957 INFO:teuthology.orchestra.run.gibba012.stdout: "type": 1, 2021-02-10T11:42:55.957 INFO:teuthology.orchestra.run.gibba012.stdout: "size": 2, 2021-02-10T11:42:55.957 INFO:teuthology.orchestra.run.gibba012.stdout: "min_size": 1, 2021-02-10T11:42:55.957 INFO:teuthology.orchestra.run.gibba012.stdout: "crush_rule": 0, 2021-02-10T11:42:55.957 INFO:teuthology.orchestra.run.gibba012.stdout: "object_hash": 2, 2021-02-10T11:42:55.957 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:42:55.957 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num": 8, 2021-02-10T11:42:55.958 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num": 8, 2021-02-10T11:42:55.958 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num_target": 8, 2021-02-10T11:42:55.958 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_target": 8, 2021-02-10T11:42:55.958 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_pending": 8, 2021-02-10T11:42:55.958 INFO:teuthology.orchestra.run.gibba012.stdout: "last_pg_merge_meta": { 2021-02-10T11:42:55.958 INFO:teuthology.orchestra.run.gibba012.stdout: "source_pgid": "0.0", 2021-02-10T11:42:55.958 INFO:teuthology.orchestra.run.gibba012.stdout: "ready_epoch": 0, 2021-02-10T11:42:55.958 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_started": 0, 2021-02-10T11:42:55.959 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_clean": 0, 2021-02-10T11:42:55.959 INFO:teuthology.orchestra.run.gibba012.stdout: "source_version": "0'0", 2021-02-10T11:42:55.959 INFO:teuthology.orchestra.run.gibba012.stdout: "target_version": "0'0" 2021-02-10T11:42:55.959 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:55.959 INFO:teuthology.orchestra.run.gibba012.stdout: "last_change": "125", 2021-02-10T11:42:55.959 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend": "0", 2021-02-10T11:42:55.959 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:42:55.959 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:42:55.959 INFO:teuthology.orchestra.run.gibba012.stdout: "auid": 0, 2021-02-10T11:42:55.959 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:42:55.959 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_seq": 0, 2021-02-10T11:42:55.959 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_epoch": 0, 2021-02-10T11:42:55.960 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_snaps": [], 2021-02-10T11:42:55.960 INFO:teuthology.orchestra.run.gibba012.stdout: "removed_snaps": "[]", 2021-02-10T11:42:55.960 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_bytes": 0, 2021-02-10T11:42:55.960 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_objects": 0, 2021-02-10T11:42:55.960 INFO:teuthology.orchestra.run.gibba012.stdout: "tiers": [], 2021-02-10T11:42:55.960 INFO:teuthology.orchestra.run.gibba012.stdout: "tier_of": -1, 2021-02-10T11:42:55.960 INFO:teuthology.orchestra.run.gibba012.stdout: "read_tier": -1, 2021-02-10T11:42:55.960 INFO:teuthology.orchestra.run.gibba012.stdout: "write_tier": -1, 2021-02-10T11:42:55.960 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_mode": "none", 2021-02-10T11:42:55.960 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_bytes": 0, 2021-02-10T11:42:55.960 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_objects": 0, 2021-02-10T11:42:55.960 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:42:55.960 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:42:55.961 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:42:55.961 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_flush_age": 0, 2021-02-10T11:42:55.961 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_evict_age": 0, 2021-02-10T11:42:55.961 INFO:teuthology.orchestra.run.gibba012.stdout: "erasure_code_profile": "", 2021-02-10T11:42:55.961 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_params": { 2021-02-10T11:42:55.961 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "none" 2021-02-10T11:42:55.961 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:55.961 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_period": 0, 2021-02-10T11:42:55.961 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_count": 0, 2021-02-10T11:42:55.961 INFO:teuthology.orchestra.run.gibba012.stdout: "use_gmt_hitset": true, 2021-02-10T11:42:55.961 INFO:teuthology.orchestra.run.gibba012.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:42:55.961 INFO:teuthology.orchestra.run.gibba012.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:42:55.961 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:42:55.962 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:42:55.962 INFO:teuthology.orchestra.run.gibba012.stdout: "grade_table": [], 2021-02-10T11:42:55.962 INFO:teuthology.orchestra.run.gibba012.stdout: "stripe_width": 0, 2021-02-10T11:42:55.962 INFO:teuthology.orchestra.run.gibba012.stdout: "expected_num_objects": 0, 2021-02-10T11:42:55.962 INFO:teuthology.orchestra.run.gibba012.stdout: "fast_read": false, 2021-02-10T11:42:55.962 INFO:teuthology.orchestra.run.gibba012.stdout: "options": {}, 2021-02-10T11:42:55.962 INFO:teuthology.orchestra.run.gibba012.stdout: "application_metadata": { 2021-02-10T11:42:55.962 INFO:teuthology.orchestra.run.gibba012.stdout: "cephfs": { 2021-02-10T11:42:55.962 INFO:teuthology.orchestra.run.gibba012.stdout: "data": "cephfs" 2021-02-10T11:42:55.962 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:42:55.962 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:42:55.962 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:42:55.963 INFO:teuthology.orchestra.run.gibba012.stdout: ], 2021-02-10T11:42:55.963 INFO:teuthology.orchestra.run.gibba012.stdout: "osds": [ 2021-02-10T11:42:55.963 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:55.963 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 0, 2021-02-10T11:42:55.964 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "95444b8f-6370-4a1d-ae5f-df62651b77da", 2021-02-10T11:42:55.964 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:42:55.964 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:42:55.964 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:42:55.964 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:42:55.964 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:42:55.964 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:42:55.964 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:42:55.964 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 123, 2021-02-10T11:42:55.964 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:42:55.964 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:42:55.964 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:42:55.964 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:42:55.965 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:55.965 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:42:55.965 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6817", 2021-02-10T11:42:55.965 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:42:55.965 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:55.965 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:55.965 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:42:55.965 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6818", 2021-02-10T11:42:55.965 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:42:55.965 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:42:55.965 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:42:55.965 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:55.965 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:42:55.966 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:42:55.966 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:55.966 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:42:55.966 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6820", 2021-02-10T11:42:55.966 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:42:55.966 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:55.966 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:55.966 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:42:55.966 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6822", 2021-02-10T11:42:55.966 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:42:55.966 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:42:55.966 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:42:55.966 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:55.967 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:42:55.967 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:42:55.967 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:55.967 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:42:55.967 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6828", 2021-02-10T11:42:55.967 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:42:55.967 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:55.967 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:55.967 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:42:55.967 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6830", 2021-02-10T11:42:55.967 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:42:55.967 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:42:55.967 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:42:55.968 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:55.968 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:42:55.968 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:42:55.968 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:55.968 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:42:55.968 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6824", 2021-02-10T11:42:55.968 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:42:55.968 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:55.968 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:55.968 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:42:55.968 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6826", 2021-02-10T11:42:55.968 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:42:55.968 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:42:55.969 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:42:55.969 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:55.969 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.112:6818/36030", 2021-02-10T11:42:55.969 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.112:6822/36030", 2021-02-10T11:42:55.969 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.112:6830/36030", 2021-02-10T11:42:55.969 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.112:6826/36030", 2021-02-10T11:42:55.969 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:42:55.969 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:42:55.969 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:42:55.969 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:42:55.969 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:55.969 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:55.969 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 1, 2021-02-10T11:42:55.970 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "a23e217e-0d8e-486e-be07-712adfabd262", 2021-02-10T11:42:55.970 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:42:55.970 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:42:55.970 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:42:55.970 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:42:55.970 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:42:55.970 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:42:55.970 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:42:55.970 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 122, 2021-02-10T11:42:55.970 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:42:55.970 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:42:55.970 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:42:55.971 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:42:55.971 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:55.971 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:42:55.971 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6800", 2021-02-10T11:42:55.971 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:42:55.971 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:55.971 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:55.971 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:42:55.971 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6802", 2021-02-10T11:42:55.971 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:42:55.971 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:42:55.971 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:42:55.971 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:55.972 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:42:55.972 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:42:55.972 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:55.972 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:42:55.972 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6804", 2021-02-10T11:42:55.972 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:42:55.972 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:55.972 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:55.972 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:42:55.972 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6806", 2021-02-10T11:42:55.972 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:42:55.972 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:42:55.972 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:42:55.973 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:55.973 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:42:55.973 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:42:55.973 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:55.973 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:42:55.973 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6812", 2021-02-10T11:42:55.973 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:42:55.973 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:55.973 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:55.973 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:42:55.973 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6814", 2021-02-10T11:42:55.973 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:42:55.973 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:42:55.974 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:42:55.974 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:55.974 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:42:55.974 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:42:55.974 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:55.974 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:42:55.974 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6808", 2021-02-10T11:42:55.974 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:42:55.974 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:55.974 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:55.974 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:42:55.974 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6810", 2021-02-10T11:42:55.974 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:42:55.975 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:42:55.975 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:42:55.975 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:55.975 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.112:6802/36029", 2021-02-10T11:42:55.975 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.112:6806/36029", 2021-02-10T11:42:55.975 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.112:6814/36029", 2021-02-10T11:42:55.975 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.112:6810/36029", 2021-02-10T11:42:55.975 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:42:55.975 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:42:55.975 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:42:55.975 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:42:55.975 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:55.975 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:55.976 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 2, 2021-02-10T11:42:55.976 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "026afa8a-c0f1-4b35-9a76-53c00fe0c2fc", 2021-02-10T11:42:55.976 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:42:55.976 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:42:55.976 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:42:55.976 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:42:55.976 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:42:55.976 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:42:55.976 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:42:55.976 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 123, 2021-02-10T11:42:55.976 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:42:55.976 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:42:55.976 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:42:55.977 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:42:55.977 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:55.977 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:42:55.977 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6801", 2021-02-10T11:42:55.977 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:42:55.977 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:55.977 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:55.977 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:42:55.977 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6803", 2021-02-10T11:42:55.977 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:42:55.977 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:42:55.977 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:42:55.977 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:55.977 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:42:55.978 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:42:55.978 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:55.978 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:42:55.978 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6805", 2021-02-10T11:42:55.978 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:42:55.978 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:55.978 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:55.978 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:42:55.978 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6807", 2021-02-10T11:42:55.978 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:42:55.978 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:42:55.978 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:42:55.979 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:55.979 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:42:55.979 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:42:55.979 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:55.979 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:42:55.979 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6813", 2021-02-10T11:42:55.979 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:42:55.979 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:55.979 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:55.979 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:42:55.979 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6815", 2021-02-10T11:42:55.979 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:42:55.979 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:42:55.979 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:42:55.980 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:55.980 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:42:55.980 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:42:55.980 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:55.980 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:42:55.980 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6809", 2021-02-10T11:42:55.980 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:42:55.980 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:55.980 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:55.980 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:42:55.980 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6811", 2021-02-10T11:42:55.980 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:42:55.980 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:42:55.981 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:42:55.981 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:55.981 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.112:6803/36031", 2021-02-10T11:42:55.981 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.112:6807/36031", 2021-02-10T11:42:55.981 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.112:6815/36031", 2021-02-10T11:42:55.981 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.112:6811/36031", 2021-02-10T11:42:55.981 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:42:55.981 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:42:55.981 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:42:55.981 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:42:55.981 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:55.981 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:55.981 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 3, 2021-02-10T11:42:55.982 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "58795682-e00d-4173-bb84-f7075ca89a11", 2021-02-10T11:42:55.982 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:42:55.982 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:42:55.982 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:42:55.982 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:42:55.982 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:42:55.982 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:42:55.982 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:42:55.982 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 115, 2021-02-10T11:42:55.982 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:42:55.982 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:42:55.982 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:42:55.982 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:42:55.983 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:55.983 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:42:55.983 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6816", 2021-02-10T11:42:55.983 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:42:55.983 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:55.983 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:55.983 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:42:55.983 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6819", 2021-02-10T11:42:55.983 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:42:55.983 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:42:55.983 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:42:55.983 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:55.983 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:42:55.984 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:42:55.984 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:55.984 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:42:55.984 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6821", 2021-02-10T11:42:55.984 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:42:55.984 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:55.984 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:55.984 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:42:55.984 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6823", 2021-02-10T11:42:55.984 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:42:55.984 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:42:55.984 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:42:55.985 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:55.985 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:42:55.985 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:42:55.985 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:55.985 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:42:55.985 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6829", 2021-02-10T11:42:55.985 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:42:55.985 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:55.985 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:55.985 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:42:55.985 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6831", 2021-02-10T11:42:55.985 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:42:55.985 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:42:55.986 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:42:55.986 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:55.986 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:42:55.986 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:42:55.986 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:55.986 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:42:55.986 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6825", 2021-02-10T11:42:55.986 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:42:55.986 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:55.986 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:55.986 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:42:55.986 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6827", 2021-02-10T11:42:55.986 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:42:55.987 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:42:55.987 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:42:55.987 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:55.987 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.112:6819/36028", 2021-02-10T11:42:55.987 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.112:6823/36028", 2021-02-10T11:42:55.987 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.112:6831/36028", 2021-02-10T11:42:55.987 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.112:6827/36028", 2021-02-10T11:42:55.987 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:42:55.987 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:42:55.987 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:42:55.987 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:42:55.987 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:55.987 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:55.988 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 4, 2021-02-10T11:42:55.988 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "6763f8d6-53dc-4e45-bdf6-a6eabcf81e39", 2021-02-10T11:42:55.988 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:42:55.988 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:42:55.988 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:42:55.988 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:42:55.988 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:42:55.988 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:42:55.988 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:42:55.988 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 123, 2021-02-10T11:42:55.988 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:42:55.988 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:42:55.988 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:42:55.989 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:42:55.989 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:55.989 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:42:55.989 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6810", 2021-02-10T11:42:55.989 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:42:55.989 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:55.989 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:55.989 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:42:55.989 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6813", 2021-02-10T11:42:55.989 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:42:55.989 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:42:55.989 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:42:55.989 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:55.990 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:42:55.990 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:42:55.990 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:55.990 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:42:55.990 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6816", 2021-02-10T11:42:55.990 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:42:55.990 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:55.990 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:55.990 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:42:55.990 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6819", 2021-02-10T11:42:55.990 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:42:55.990 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:42:55.991 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:42:55.991 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:55.991 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:42:55.991 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:42:55.991 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:55.991 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:42:55.991 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6825", 2021-02-10T11:42:55.991 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:42:55.991 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:55.991 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:55.991 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:42:55.991 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6827", 2021-02-10T11:42:55.991 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:42:55.992 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:42:55.992 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:42:55.992 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:55.992 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:42:55.992 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:42:55.992 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:55.992 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:42:55.992 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6821", 2021-02-10T11:42:55.992 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:42:55.992 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:55.992 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:55.992 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:42:55.992 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6823", 2021-02-10T11:42:55.993 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:42:55.993 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:42:55.993 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:42:55.993 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:55.993 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.120:6813/36388", 2021-02-10T11:42:55.993 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.120:6819/36388", 2021-02-10T11:42:55.993 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.120:6827/36388", 2021-02-10T11:42:55.993 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.120:6823/36388", 2021-02-10T11:42:55.993 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:42:55.993 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:42:55.993 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:42:55.993 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:42:55.993 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:55.994 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:55.994 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 5, 2021-02-10T11:42:55.994 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "5d0e4dd6-0cb0-4bf8-8314-626acaa1b544", 2021-02-10T11:42:55.994 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:42:55.994 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:42:55.994 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:42:55.994 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:42:55.994 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:42:55.994 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:42:55.994 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:42:55.994 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 123, 2021-02-10T11:42:55.994 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:42:55.995 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:42:55.995 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:42:55.995 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:42:55.995 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:55.995 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:42:55.995 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6804", 2021-02-10T11:42:55.995 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:42:55.995 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:55.995 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:55.995 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:42:55.995 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6806", 2021-02-10T11:42:55.996 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:42:55.996 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:42:55.996 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:42:55.996 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:55.996 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:42:55.996 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:42:55.996 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:55.996 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:42:55.996 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6809", 2021-02-10T11:42:55.996 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:42:55.996 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:55.996 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:55.996 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:42:55.996 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6812", 2021-02-10T11:42:55.997 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:42:55.997 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:42:55.997 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:42:55.997 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:55.997 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:42:55.997 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:42:55.997 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:55.997 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:42:55.997 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6820", 2021-02-10T11:42:55.997 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:42:55.997 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:55.997 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:55.997 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:42:55.998 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6822", 2021-02-10T11:42:55.998 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:42:55.998 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:42:55.998 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:42:55.998 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:55.998 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:42:55.998 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:42:55.998 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:55.998 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:42:55.998 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6815", 2021-02-10T11:42:55.998 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:42:55.998 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:55.998 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:55.999 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:42:55.999 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6818", 2021-02-10T11:42:55.999 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:42:55.999 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:42:55.999 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:42:55.999 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:55.999 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.120:6806/36387", 2021-02-10T11:42:55.999 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.120:6812/36387", 2021-02-10T11:42:55.999 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.120:6822/36387", 2021-02-10T11:42:55.999 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.120:6818/36387", 2021-02-10T11:42:55.999 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:42:56.000 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:42:56.000 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:42:56.000 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:42:56.000 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:56.000 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:56.000 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 6, 2021-02-10T11:42:56.000 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "b010e187-4541-4b00-80dd-f948f6ab8f15", 2021-02-10T11:42:56.000 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:42:56.000 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:42:56.001 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:42:56.001 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:42:56.001 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:42:56.001 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:42:56.001 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:42:56.001 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 123, 2021-02-10T11:42:56.001 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:42:56.001 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:42:56.001 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:42:56.001 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:42:56.002 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:56.002 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:42:56.002 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6824", 2021-02-10T11:42:56.002 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:42:56.002 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:56.002 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:56.002 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:42:56.002 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6826", 2021-02-10T11:42:56.002 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:42:56.002 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:42:56.002 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:42:56.002 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:56.003 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:42:56.003 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:42:56.003 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:56.003 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:42:56.003 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6828", 2021-02-10T11:42:56.003 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:42:56.003 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:56.003 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:56.003 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:42:56.003 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6829", 2021-02-10T11:42:56.003 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:42:56.003 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:42:56.003 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:42:56.003 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:56.004 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:42:56.004 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:42:56.004 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:56.004 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:42:56.004 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6832", 2021-02-10T11:42:56.004 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:42:56.004 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:56.004 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:56.004 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:42:56.004 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6833", 2021-02-10T11:42:56.004 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:42:56.004 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:42:56.005 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:42:56.005 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:56.005 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:42:56.005 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:42:56.005 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:56.005 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:42:56.005 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6830", 2021-02-10T11:42:56.005 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:42:56.005 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:56.005 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:56.005 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:42:56.005 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6831", 2021-02-10T11:42:56.005 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:42:56.005 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:42:56.006 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:42:56.006 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:56.006 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.120:6826/36385", 2021-02-10T11:42:56.006 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.120:6829/36385", 2021-02-10T11:42:56.006 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.120:6833/36385", 2021-02-10T11:42:56.006 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.120:6831/36385", 2021-02-10T11:42:56.006 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:42:56.006 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:42:56.006 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:42:56.006 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:42:56.006 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:56.007 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:56.007 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 7, 2021-02-10T11:42:56.007 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "671e2772-33b7-497e-8e44-2178bd850bcf", 2021-02-10T11:42:56.007 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:42:56.007 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:42:56.007 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:42:56.007 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:42:56.007 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:42:56.007 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:42:56.007 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:42:56.007 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 122, 2021-02-10T11:42:56.007 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:42:56.007 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:42:56.008 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:42:56.008 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:42:56.008 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:56.008 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:42:56.008 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6802", 2021-02-10T11:42:56.008 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:42:56.008 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:56.008 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:56.008 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:42:56.008 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6803", 2021-02-10T11:42:56.008 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:42:56.008 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:42:56.008 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:42:56.009 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:56.009 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:42:56.009 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:42:56.009 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:56.009 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:42:56.009 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6805", 2021-02-10T11:42:56.009 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:42:56.009 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:56.009 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:56.009 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:42:56.009 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6807", 2021-02-10T11:42:56.009 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:42:56.009 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:42:56.009 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:42:56.010 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:56.010 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:42:56.010 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:42:56.010 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:56.010 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:42:56.010 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6814", 2021-02-10T11:42:56.010 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:42:56.010 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:56.010 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:56.010 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:42:56.010 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6817", 2021-02-10T11:42:56.010 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:42:56.011 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:42:56.011 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:42:56.011 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:56.011 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:42:56.011 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:42:56.011 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:56.011 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:42:56.011 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6808", 2021-02-10T11:42:56.011 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:42:56.011 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:56.011 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:56.011 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:42:56.011 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6811", 2021-02-10T11:42:56.011 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:42:56.012 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:42:56.012 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:42:56.012 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:56.012 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.120:6803/36386", 2021-02-10T11:42:56.012 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.120:6807/36386", 2021-02-10T11:42:56.012 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.120:6817/36386", 2021-02-10T11:42:56.012 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.120:6811/36386", 2021-02-10T11:42:56.012 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:42:56.012 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:42:56.012 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:42:56.012 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:42:56.012 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:42:56.012 INFO:teuthology.orchestra.run.gibba012.stdout: ], 2021-02-10T11:42:56.013 INFO:teuthology.orchestra.run.gibba012.stdout: "osd_xinfo": [ 2021-02-10T11:42:56.013 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:56.013 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 0, 2021-02-10T11:42:56.013 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:42:56.013 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:42:56.013 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:42:56.013 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:42:56.013 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:42:56.013 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:56.013 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:56.013 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 1, 2021-02-10T11:42:56.013 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:42:56.013 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:42:56.014 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:42:56.014 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:42:56.014 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:42:56.014 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:56.014 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:56.014 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 2, 2021-02-10T11:42:56.014 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:42:56.014 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:42:56.014 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:42:56.014 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:42:56.014 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:42:56.014 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:56.014 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:56.015 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 3, 2021-02-10T11:42:56.015 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:42:56.015 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:42:56.015 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:42:56.015 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:42:56.015 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:42:56.015 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:56.015 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:56.015 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 4, 2021-02-10T11:42:56.015 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:42:56.015 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:42:56.015 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:42:56.015 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:42:56.016 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:42:56.016 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:56.016 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:56.016 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 5, 2021-02-10T11:42:56.016 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:42:56.016 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:42:56.016 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:42:56.016 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:42:56.016 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:42:56.016 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:56.016 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:56.016 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 6, 2021-02-10T11:42:56.016 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:42:56.016 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:42:56.017 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:42:56.017 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:42:56.017 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:42:56.017 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:56.017 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:42:56.017 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 7, 2021-02-10T11:42:56.017 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:42:56.017 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:42:56.017 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:42:56.017 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:42:56.017 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:42:56.017 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:42:56.017 INFO:teuthology.orchestra.run.gibba012.stdout: ], 2021-02-10T11:42:56.018 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_upmap": [], 2021-02-10T11:42:56.018 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_upmap_items": [], 2021-02-10T11:42:56.018 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_temp": [], 2021-02-10T11:42:56.018 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_temp": [], 2021-02-10T11:42:56.018 INFO:teuthology.orchestra.run.gibba012.stdout: "blacklist": {}, 2021-02-10T11:42:56.018 INFO:teuthology.orchestra.run.gibba012.stdout: "erasure_code_profiles": { 2021-02-10T11:42:56.018 INFO:teuthology.orchestra.run.gibba012.stdout: "default": { 2021-02-10T11:42:56.018 INFO:teuthology.orchestra.run.gibba012.stdout: "crush-failure-domain": "osd", 2021-02-10T11:42:56.018 INFO:teuthology.orchestra.run.gibba012.stdout: "k": "2", 2021-02-10T11:42:56.018 INFO:teuthology.orchestra.run.gibba012.stdout: "m": "1", 2021-02-10T11:42:56.018 INFO:teuthology.orchestra.run.gibba012.stdout: "plugin": "jerasure", 2021-02-10T11:42:56.018 INFO:teuthology.orchestra.run.gibba012.stdout: "ruleset-failure-domain": "osd", 2021-02-10T11:42:56.018 INFO:teuthology.orchestra.run.gibba012.stdout: "technique": "reed_sol_van" 2021-02-10T11:42:56.019 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:42:56.019 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:42:56.019 INFO:teuthology.orchestra.run.gibba012.stdout: "removed_snaps_queue": [], 2021-02-10T11:42:56.019 INFO:teuthology.orchestra.run.gibba012.stdout: "new_removed_snaps": [], 2021-02-10T11:42:56.019 INFO:teuthology.orchestra.run.gibba012.stdout: "new_purged_snaps": [], 2021-02-10T11:42:56.019 INFO:teuthology.orchestra.run.gibba012.stdout: "crush_node_flags": {}, 2021-02-10T11:42:56.019 INFO:teuthology.orchestra.run.gibba012.stdout: "device_class_flags": {} 2021-02-10T11:42:56.019 INFO:teuthology.orchestra.run.gibba012.stdout:} 2021-02-10T11:42:56.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:55.944 7f41d89a2700 1 -- 172.21.2.112:0/3557384527 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f41b8018730 msgr2=0x7f41b801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:56.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:55.944 7f41d89a2700 1 --2- 172.21.2.112:0/3557384527 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f41b8018730 0x7f41b801ac10 crc :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:56.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:55.944 7f41d89a2700 1 -- 172.21.2.112:0/3557384527 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f41d409bb00 msgr2=0x7f41d409bf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:56.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:55.944 7f41d89a2700 1 --2- 172.21.2.112:0/3557384527 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f41d409bb00 0x7f41d409bf00 secure :-1 s=READY pgs=2391 cs=0 l=1 rev1=1 rx=0x7f41cc00b9b0 tx=0x7f41cc00bd50).stop 2021-02-10T11:42:56.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:55.944 7f41d89a2700 1 -- 172.21.2.112:0/3557384527 shutdown_connections 2021-02-10T11:42:56.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:55.944 7f41d89a2700 1 --2- 172.21.2.112:0/3557384527 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f41b8018730 0x7f41b801ac10 unknown :-1 s=CLOSED pgs=1092 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:56.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:55.944 7f41d89a2700 1 --2- 172.21.2.112:0/3557384527 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f41d40af9c0 0x7f41d409b4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:56.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:55.944 7f41d89a2700 1 --2- 172.21.2.112:0/3557384527 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f41d409bb00 0x7f41d409bf00 unknown :-1 s=CLOSED pgs=2391 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:56.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:55.944 7f41d89a2700 1 --2- 172.21.2.112:0/3557384527 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f41d40aefa0 0x7f41d40af3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:56.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:55.944 7f41d89a2700 1 -- 172.21.2.112:0/3557384527 >> 172.21.2.112:0/3557384527 conn(0x7f41d419f8f0 msgr2=0x7f41d40906e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:56.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:55.944 7f41d89a2700 1 -- 172.21.2.112:0/3557384527 shutdown_connections 2021-02-10T11:42:56.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:55.944 7f41d89a2700 1 -- 172.21.2.112:0/3557384527 wait complete. 2021-02-10T11:42:56.022 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:42:56.450 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:56.433 7f3d68114700 1 Processor -- start 2021-02-10T11:42:56.450 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:56.433 7f3d68114700 1 -- start start 2021-02-10T11:42:56.450 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:56.434 7f3d68114700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f3d600d0340 0x7f3d600ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:56.450 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:56.434 7f3d68114700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3d600d1c70 0x7f3d600efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:56.450 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:56.434 7f3d68114700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d600f0b10 con 0x7f3d600d1870 2021-02-10T11:42:56.450 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:56.434 7f3d68114700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d600e5ed0 con 0x7f3d600d0340 2021-02-10T11:42:56.450 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:56.434 7f3d68114700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d601b0be0 con 0x7f3d600d1c70 2021-02-10T11:42:56.451 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:56.435 7f3d66731700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3d600d1c70 0x7f3d600efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:56.451 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:56.435 7f3d66731700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3d600d1c70 0x7f3d600efc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3301/0 says I am v2:172.21.2.112:56312/0 (socket says 172.21.2.112:56312) 2021-02-10T11:42:56.451 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:56.435 7f3d66731700 1 -- 172.21.2.112:0/281808652 learned_addr learned my addr 172.21.2.112:0/281808652 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:42:56.451 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:56.435 7f3d6572f700 1 --2- 172.21.2.112:0/281808652 >> v2:172.21.2.120:3300/0 conn(0x7f3d600d0340 0x7f3d600ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:56.451 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:56.435 7f3d66731700 1 -- 172.21.2.112:0/281808652 >> v2:172.21.2.120:3300/0 conn(0x7f3d600d0340 msgr2=0x7f3d600ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:56.451 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:56.435 7f3d66731700 1 --2- 172.21.2.112:0/281808652 >> v2:172.21.2.120:3300/0 conn(0x7f3d600d0340 0x7f3d600ef640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:56.451 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:56.435 7f3d66731700 1 -- 172.21.2.112:0/281808652 >> v1:172.21.2.120:6789/0 conn(0x7f3d600d1870 legacy=0x7f3d601b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:56.451 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:56.435 7f3d66731700 1 -- 172.21.2.112:0/281808652 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d600f20a0 con 0x7f3d600d1c70 2021-02-10T11:42:56.451 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:56.435 7f3d66731700 1 --2- 172.21.2.112:0/281808652 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3d600d1c70 0x7f3d600efc60 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7f3d58006aa0 tx=0x7f3d58009f20).ready entity=mon.2 client_cookie=63938724c6ce6c6f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:56.451 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:56.435 7f3d64f2e700 1 -- 172.21.2.112:0/281808652 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3d5800b5c0 con 0x7f3d600d1c70 2021-02-10T11:42:56.451 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:56.435 7f3d64f2e700 1 -- 172.21.2.112:0/281808652 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d58003030 con 0x7f3d600d1c70 2021-02-10T11:42:56.451 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:56.436 7f3d68114700 1 -- 172.21.2.112:0/281808652 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3d600d1c70 msgr2=0x7f3d600efc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:56.451 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:56.436 7f3d68114700 1 --2- 172.21.2.112:0/281808652 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3d600d1c70 0x7f3d600efc60 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7f3d58006aa0 tx=0x7f3d58009f20).stop 2021-02-10T11:42:56.452 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:56.436 7f3d68114700 1 -- 172.21.2.112:0/281808652 shutdown_connections 2021-02-10T11:42:56.452 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:56.436 7f3d68114700 1 --2- 172.21.2.112:0/281808652 >> v2:172.21.2.120:3300/0 conn(0x7f3d600d0340 0x7f3d600ef640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:56.452 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:56.436 7f3d68114700 1 --2- 172.21.2.112:0/281808652 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3d600d1c70 0x7f3d600efc60 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:56.452 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:56.436 7f3d68114700 1 -- 172.21.2.112:0/281808652 >> 172.21.2.112:0/281808652 conn(0x7f3d600ce450 msgr2=0x7f3d600d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:56.452 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:56.436 7f3d68114700 1 -- 172.21.2.112:0/281808652 shutdown_connections 2021-02-10T11:42:56.452 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:56.436 7f3d68114700 1 -- 172.21.2.112:0/281808652 wait complete. 2021-02-10T11:42:56.454 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:56.451 7f3d68114700 1 Processor -- start 2021-02-10T11:42:56.454 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:56.451 7f3d68114700 1 -- start start 2021-02-10T11:42:56.454 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:56.451 7f3d68114700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3d60187e50 0x7f3d60188250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:56.454 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:56.451 7f3d68114700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3d600ef210 0x7f3d600ef610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:56.454 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:56.451 7f3d68114700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3d60188e80 0x7f3d60189280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:56.454 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:56.451 7f3d68114700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d6018efb0 con 0x7f3d60188e80 2021-02-10T11:42:56.454 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:56.451 7f3d68114700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d6018f1d0 con 0x7f3d600ef210 2021-02-10T11:42:56.454 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:56.451 7f3d68114700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d601898a0 con 0x7f3d60187e50 2021-02-10T11:42:56.454 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:56.451 7f3d66731700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3d60188e80 0x7f3d60189280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:56.454 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:56.452 7f3d66731700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3d60188e80 0x7f3d60189280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:56634/0 (socket says 172.21.2.112:56634) 2021-02-10T11:42:56.454 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:56.452 7f3d66731700 1 -- 172.21.2.112:0/2316948653 learned_addr learned my addr 172.21.2.112:0/2316948653 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:42:56.455 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:56.452 7f3d6572f700 1 --2- 172.21.2.112:0/2316948653 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3d600ef210 0x7f3d600ef610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:56.455 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:56.452 7f3d65f30700 1 --2- 172.21.2.112:0/2316948653 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3d60187e50 0x7f3d60188250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:56.455 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:56.452 7f3d66731700 1 -- 172.21.2.112:0/2316948653 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3d60187e50 msgr2=0x7f3d60188250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:56.455 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:56.452 7f3d66731700 1 --2- 172.21.2.112:0/2316948653 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3d60187e50 0x7f3d60188250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:56.455 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:56.452 7f3d66731700 1 -- 172.21.2.112:0/2316948653 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3d600ef210 msgr2=0x7f3d600ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:56.455 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:56.452 7f3d66731700 1 --2- 172.21.2.112:0/2316948653 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3d600ef210 0x7f3d600ef610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:56.455 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:56.452 7f3d66731700 1 -- 172.21.2.112:0/2316948653 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d60189ac0 con 0x7f3d60188e80 2021-02-10T11:42:56.455 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:56.452 7f3d6572f700 1 --2- 172.21.2.112:0/2316948653 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3d600ef210 0x7f3d600ef610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:42:56.455 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:56.452 7f3d65f30700 1 --2- 172.21.2.112:0/2316948653 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3d60187e50 0x7f3d60188250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:42:56.455 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:56.452 7f3d66731700 1 --2- 172.21.2.112:0/2316948653 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3d60188e80 0x7f3d60189280 secure :-1 s=READY pgs=2392 cs=0 l=1 rev1=1 rx=0x7f3d58009f70 tx=0x7f3d5800bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:56.455 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:56.452 7f3d567fc700 1 -- 172.21.2.112:0/2316948653 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3d58009140 con 0x7f3d60188e80 2021-02-10T11:42:56.455 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:56.452 7f3d567fc700 1 -- 172.21.2.112:0/2316948653 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d58006820 con 0x7f3d60188e80 2021-02-10T11:42:56.455 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:56.452 7f3d567fc700 1 -- 172.21.2.112:0/2316948653 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3d5800c4e0 con 0x7f3d60188e80 2021-02-10T11:42:56.456 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:56.453 7f3d68114700 1 -- 172.21.2.112:0/2316948653 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d60189ac0 con 0x7f3d60188e80 2021-02-10T11:42:56.456 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:56.453 7f3d68114700 1 -- 172.21.2.112:0/2316948653 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d60189ac0 con 0x7f3d60188e80 2021-02-10T11:42:56.456 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:56.453 7f3d567fc700 1 -- 172.21.2.112:0/2316948653 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3d580056b0 con 0x7f3d60188e80 2021-02-10T11:42:56.456 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:56.453 7f3d567fc700 1 --2- 172.21.2.112:0/2316948653 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3d480186c0 0x7f3d4801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:56.456 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:56.453 7f3d567fc700 1 -- 172.21.2.112:0/2316948653 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f3d58032880 con 0x7f3d60188e80 2021-02-10T11:42:56.456 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:56.454 7f3d65f30700 1 --2- 172.21.2.112:0/2316948653 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3d480186c0 0x7f3d4801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:56.456 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:56.454 7f3d65f30700 1 --2- 172.21.2.112:0/2316948653 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3d480186c0 0x7f3d4801abc0 crc :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:56.483 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:56.484 7f3d68114700 1 -- 172.21.2.112:0/2316948653 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d600f0b10 con 0x7f3d60188e80 2021-02-10T11:42:56.485 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:56.486 7f3d567fc700 1 -- 172.21.2.112:0/2316948653 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3d5800be30 con 0x7f3d60188e80 2021-02-10T11:42:56.613 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:56.613 7f3d68114700 1 -- 172.21.2.112:0/2316948653 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3d600f0b10 con 0x7f3d60188e80 2021-02-10T11:42:56.613 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:56.614 7f3d567fc700 1 -- 172.21.2.112:0/2316948653 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+2108 (secure 0 0 0) 0x7f3d580068f0 con 0x7f3d60188e80 2021-02-10T11:42:56.614 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:42:56.614 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":106,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11040,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/1814390924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":1814390924},{"type":"v1","addr":"172.21.2.112:6833","nonce":1814390924}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":101}],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:42:19.351348","modified":"2021-02-10 11:42:21.357585","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8203},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8203":{"gid":8203,"name":"b","rank":0,"incarnation":102,"state":"up:active","state_seq":3,"addr":"172.21.2.120:6835/2761661070","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":2761661070},{"type":"v1","addr":"172.21.2.120:6835","nonce":2761661070}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":13}]} 2021-02-10T11:42:56.614 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 106 2021-02-10T11:42:56.615 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:56.616 7f3d68114700 1 -- 172.21.2.112:0/2316948653 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3d480186c0 msgr2=0x7f3d4801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:56.615 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:56.616 7f3d68114700 1 --2- 172.21.2.112:0/2316948653 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3d480186c0 0x7f3d4801abc0 crc :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:56.615 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:56.616 7f3d68114700 1 -- 172.21.2.112:0/2316948653 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3d60188e80 msgr2=0x7f3d60189280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:56.615 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:56.616 7f3d68114700 1 --2- 172.21.2.112:0/2316948653 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3d60188e80 0x7f3d60189280 secure :-1 s=READY pgs=2392 cs=0 l=1 rev1=1 rx=0x7f3d58009f70 tx=0x7f3d5800bcf0).stop 2021-02-10T11:42:56.616 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:56.616 7f3d68114700 1 -- 172.21.2.112:0/2316948653 shutdown_connections 2021-02-10T11:42:56.616 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:56.616 7f3d68114700 1 --2- 172.21.2.112:0/2316948653 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3d480186c0 0x7f3d4801abc0 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:56.616 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:56.616 7f3d68114700 1 --2- 172.21.2.112:0/2316948653 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3d600ef210 0x7f3d600ef610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:56.616 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:56.616 7f3d68114700 1 --2- 172.21.2.112:0/2316948653 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3d60188e80 0x7f3d60189280 unknown :-1 s=CLOSED pgs=2392 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:56.616 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:56.616 7f3d68114700 1 --2- 172.21.2.112:0/2316948653 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3d60187e50 0x7f3d60188250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:56.616 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:56.616 7f3d68114700 1 -- 172.21.2.112:0/2316948653 >> 172.21.2.112:0/2316948653 conn(0x7f3d6019f8c0 msgr2=0x7f3d60094980 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:56.616 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:56.616 7f3d68114700 1 -- 172.21.2.112:0/2316948653 shutdown_connections 2021-02-10T11:42:56.616 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:56.616 7f3d68114700 1 -- 172.21.2.112:0/2316948653 wait complete. 2021-02-10T11:42:56.624 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs fail cephfs 2021-02-10T11:42:57.055 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:57.039 7f780776b700 1 Processor -- start 2021-02-10T11:42:57.055 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:57.039 7f780776b700 1 -- start start 2021-02-10T11:42:57.055 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:57.039 7f780776b700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f78000d1870 0x7f78001b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:57.055 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:57.039 7f780776b700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f78001b0a10 0x7f78000cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:57.056 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:57.039 7f780776b700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78000f0b10 con 0x7f78000d1c70 2021-02-10T11:42:57.056 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:57.039 7f780776b700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78000e5ed0 con 0x7f78001b0a10 2021-02-10T11:42:57.056 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:57.039 7f780776b700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78000f0650 con 0x7f78000d1870 2021-02-10T11:42:57.056 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:57.040 7f7804d86700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f78001b0a10 0x7f78000cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:57.056 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:57.040 7f7804d86700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f78001b0a10 0x7f78000cf330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:56640/0 (socket says 172.21.2.112:56640) 2021-02-10T11:42:57.056 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:57.040 7f7804d86700 1 -- 172.21.2.112:0/3472026722 learned_addr learned my addr 172.21.2.112:0/3472026722 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:42:57.056 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:57.040 7f7804d86700 1 -- 172.21.2.112:0/3472026722 >> v2:172.21.2.120:3300/0 conn(0x7f78000d1870 msgr2=0x7f78001b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:57.056 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:57.040 7f7804d86700 1 --2- 172.21.2.112:0/3472026722 >> v2:172.21.2.120:3300/0 conn(0x7f78000d1870 0x7f78001b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:57.056 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:57.040 7f7804d86700 1 -- 172.21.2.112:0/3472026722 >> v1:172.21.2.120:6789/0 conn(0x7f78000d1c70 legacy=0x7f78000cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:57.056 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:57.040 7f7804d86700 1 -- 172.21.2.112:0/3472026722 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78000f0840 con 0x7f78001b0a10 2021-02-10T11:42:57.056 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:57.040 7f7804d86700 1 --2- 172.21.2.112:0/3472026722 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f78001b0a10 0x7f78000cf330 secure :-1 s=READY pgs=2393 cs=0 l=1 rev1=1 rx=0x7f77f0006700 tx=0x7f77f0009b80).ready entity=mon.0 client_cookie=fd3c31b400d94fec server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:57.056 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:57.041 7f77f7fff700 1 -- 172.21.2.112:0/3472026722 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f77f000c0c0 con 0x7f78001b0a10 2021-02-10T11:42:57.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:57.041 7f77f7fff700 1 -- 172.21.2.112:0/3472026722 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77f000c440 con 0x7f78001b0a10 2021-02-10T11:42:57.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:57.041 7f77f7fff700 1 -- 172.21.2.112:0/3472026722 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f77f000c960 con 0x7f78001b0a10 2021-02-10T11:42:57.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:57.041 7f780776b700 1 -- 172.21.2.112:0/3472026722 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f78001b0a10 msgr2=0x7f78000cf330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:57.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:57.041 7f780776b700 1 --2- 172.21.2.112:0/3472026722 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f78001b0a10 0x7f78000cf330 secure :-1 s=READY pgs=2393 cs=0 l=1 rev1=1 rx=0x7f77f0006700 tx=0x7f77f0009b80).stop 2021-02-10T11:42:57.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:57.041 7f780776b700 1 -- 172.21.2.112:0/3472026722 shutdown_connections 2021-02-10T11:42:57.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:57.041 7f780776b700 1 --2- 172.21.2.112:0/3472026722 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f78001b0a10 0x7f78000cf330 unknown :-1 s=CLOSED pgs=2393 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:57.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:57.041 7f780776b700 1 --2- 172.21.2.112:0/3472026722 >> v2:172.21.2.120:3300/0 conn(0x7f78000d1870 0x7f78001b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:57.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:57.041 7f780776b700 1 -- 172.21.2.112:0/3472026722 >> 172.21.2.112:0/3472026722 conn(0x7f78000ce450 msgr2=0x7f78000d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:57.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:57.042 7f780776b700 1 -- 172.21.2.112:0/3472026722 shutdown_connections 2021-02-10T11:42:57.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:57.042 7f780776b700 1 -- 172.21.2.112:0/3472026722 wait complete. 2021-02-10T11:42:57.059 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:57.056 7f780776b700 1 Processor -- start 2021-02-10T11:42:57.059 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:57.056 7f780776b700 1 -- start start 2021-02-10T11:42:57.059 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:57.057 7f780776b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7800190910 0x7f780018fd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:57.059 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:57.057 7f780776b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7800190d10 0x7f78000b7db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:57.059 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:57.057 7f780776b700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f78001903b0 0x7f78000b83d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:57.059 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:57.057 7f780776b700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78000b7520 con 0x7f7800190d10 2021-02-10T11:42:57.059 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:57.057 7f780776b700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78000f0cc0 con 0x7f78001903b0 2021-02-10T11:42:57.059 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:57.057 7f780776b700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78000f0ee0 con 0x7f7800190910 2021-02-10T11:42:57.059 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:57.057 7f7804d86700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7800190d10 0x7f78000b7db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:57.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:57.057 7f7804d86700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7800190d10 0x7f78000b7db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:56646/0 (socket says 172.21.2.112:56646) 2021-02-10T11:42:57.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:57.057 7f7804d86700 1 -- 172.21.2.112:0/1315323500 learned_addr learned my addr 172.21.2.112:0/1315323500 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:42:57.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:57.057 7f7804d86700 1 -- 172.21.2.112:0/1315323500 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7800190910 msgr2=0x7f780018fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:57.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:57.057 7f7804d86700 1 --2- 172.21.2.112:0/1315323500 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7800190910 0x7f780018fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:57.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:57.057 7f7804d86700 1 -- 172.21.2.112:0/1315323500 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f78001903b0 msgr2=0x7f78000b83d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:57.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:57.057 7f7804d86700 1 --2- 172.21.2.112:0/1315323500 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f78001903b0 0x7f78000b83d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:57.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:57.057 7f7804d86700 1 -- 172.21.2.112:0/1315323500 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78000b89f0 con 0x7f7800190d10 2021-02-10T11:42:57.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:57.057 7f7804d86700 1 --2- 172.21.2.112:0/1315323500 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7800190d10 0x7f78000b7db0 secure :-1 s=READY pgs=2394 cs=0 l=1 rev1=1 rx=0x7f77f000b5e0 tx=0x7f77f000b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:57.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:57.057 7f77f5ffb700 1 -- 172.21.2.112:0/1315323500 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f77f000bd70 con 0x7f7800190d10 2021-02-10T11:42:57.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:57.057 7f77f5ffb700 1 -- 172.21.2.112:0/1315323500 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77f0002030 con 0x7f7800190d10 2021-02-10T11:42:57.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:57.057 7f77f5ffb700 1 -- 172.21.2.112:0/1315323500 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f77f0002550 con 0x7f7800190d10 2021-02-10T11:42:57.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:57.057 7f780776b700 1 -- 172.21.2.112:0/1315323500 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78000b89f0 con 0x7f7800190d10 2021-02-10T11:42:57.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:57.057 7f780776b700 1 -- 172.21.2.112:0/1315323500 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78000b8f80 con 0x7f7800190d10 2021-02-10T11:42:57.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:57.058 7f77f5ffb700 1 -- 172.21.2.112:0/1315323500 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f77f0000c90 con 0x7f7800190d10 2021-02-10T11:42:57.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:57.058 7f77f5ffb700 1 --2- 172.21.2.112:0/1315323500 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f77e00186c0 0x7f77e001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:57.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:57.058 7f77f5ffb700 1 -- 172.21.2.112:0/1315323500 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f77f0032430 con 0x7f7800190d10 2021-02-10T11:42:57.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:57.058 7f7805587700 1 --2- 172.21.2.112:0/1315323500 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f77e00186c0 0x7f77e001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:57.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:57.059 7f7805587700 1 --2- 172.21.2.112:0/1315323500 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f77e00186c0 0x7f77e001abc0 crc :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:57.088 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:57.089 7f780776b700 1 -- 172.21.2.112:0/1315323500 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78000b9230 con 0x7f7800190d10 2021-02-10T11:42:57.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:57.091 7f77f5ffb700 1 -- 172.21.2.112:0/1315323500 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f77f00009b0 con 0x7f7800190d10 2021-02-10T11:42:57.220 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:57.220 7f780776b700 1 -- 172.21.2.112:0/1315323500 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f7800199000 con 0x7f7800190d10 2021-02-10T11:42:58.000 INFO:tasks.ceph.mon.a.gibba012.stderr:2021-02-10 11:42:58.001 7fc3fe6e2700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-02-10T11:42:58.006 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:58.007 7f77f5ffb700 1 -- 172.21.2.112:0/1315323500 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v107) v1 ==== 161+0+0 (secure 0 0 0) 0x7f77f0000c90 con 0x7f7800190d10 2021-02-10T11:42:58.039 INFO:teuthology.orchestra.run.gibba012.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2021-02-10T11:42:58.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:58.041 7f780776b700 1 -- 172.21.2.112:0/1315323500 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f77e00186c0 msgr2=0x7f77e001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:58.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:58.041 7f780776b700 1 --2- 172.21.2.112:0/1315323500 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f77e00186c0 0x7f77e001abc0 crc :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:58.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:58.041 7f780776b700 1 -- 172.21.2.112:0/1315323500 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7800190d10 msgr2=0x7f78000b7db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:58.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:58.041 7f780776b700 1 --2- 172.21.2.112:0/1315323500 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7800190d10 0x7f78000b7db0 secure :-1 s=READY pgs=2394 cs=0 l=1 rev1=1 rx=0x7f77f000b5e0 tx=0x7f77f000b980).stop 2021-02-10T11:42:58.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:58.042 7f780776b700 1 -- 172.21.2.112:0/1315323500 shutdown_connections 2021-02-10T11:42:58.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:58.042 7f780776b700 1 --2- 172.21.2.112:0/1315323500 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f77e00186c0 0x7f77e001abc0 unknown :-1 s=CLOSED pgs=1094 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:58.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:58.042 7f780776b700 1 --2- 172.21.2.112:0/1315323500 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f78001903b0 0x7f78000b83d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:58.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:58.042 7f780776b700 1 --2- 172.21.2.112:0/1315323500 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7800190d10 0x7f78000b7db0 unknown :-1 s=CLOSED pgs=2394 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:58.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:58.042 7f780776b700 1 --2- 172.21.2.112:0/1315323500 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7800190910 0x7f780018fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:58.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:58.042 7f780776b700 1 -- 172.21.2.112:0/1315323500 >> 172.21.2.112:0/1315323500 conn(0x7f780018e050 msgr2=0x7f78000953e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:58.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:58.042 7f780776b700 1 -- 172.21.2.112:0/1315323500 shutdown_connections 2021-02-10T11:42:58.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:58.042 7f780776b700 1 -- 172.21.2.112:0/1315323500 wait complete. 2021-02-10T11:42:58.050 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:42:58.057 INFO:tasks.ceph.mds.b.gibba020.stdout:starting mds.b at 2021-02-10T11:42:58.486 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:58.470 7fd73c8f1700 1 Processor -- start 2021-02-10T11:42:58.487 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:58.470 7fd73c8f1700 1 -- start start 2021-02-10T11:42:58.487 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:58.471 7fd73c8f1700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fd7380d1c70 0x7fd7380f20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:58.487 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:58.471 7fd73c8f1700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7380f0b10 con 0x7fd7380d1870 2021-02-10T11:42:58.487 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:58.471 7fd73c8f1700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7381b0be0 con 0x7fd7380d0340 2021-02-10T11:42:58.487 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:58.471 7fd73c8f1700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7380e5ed0 con 0x7fd7380d1c70 2021-02-10T11:42:58.487 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:58.471 7fd736e1d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fd7380d1c70 0x7fd7380f20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:58.487 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:58.471 7fd736e1d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fd7380d1c70 0x7fd7380f20a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:56654/0 (socket says 172.21.2.112:56654) 2021-02-10T11:42:58.487 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:58.471 7fd736e1d700 1 -- 172.21.2.112:0/285715263 learned_addr learned my addr 172.21.2.112:0/285715263 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:42:58.487 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:58.472 7fd736e1d700 1 -- 172.21.2.112:0/285715263 >> v1:172.21.2.120:6789/0 conn(0x7fd7380d0340 legacy=0x7fd7380ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:58.487 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:58.472 7fd736e1d700 1 -- 172.21.2.112:0/285715263 >> v1:172.21.2.112:6789/0 conn(0x7fd7380d1870 legacy=0x7fd7381b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:58.488 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:58.472 7fd736e1d700 1 -- 172.21.2.112:0/285715263 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7380efe30 con 0x7fd7380d1c70 2021-02-10T11:42:58.488 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:58.472 7fd736e1d700 1 --2- 172.21.2.112:0/285715263 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd7380d1c70 0x7fd7380f20a0 secure :-1 s=READY pgs=2397 cs=0 l=1 rev1=1 rx=0x7fd730006f00 tx=0x7fd730009e90).ready entity=mon.0 client_cookie=d82bfa287dbc776f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:58.488 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:58.472 7fd73561a700 1 -- 172.21.2.112:0/285715263 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd73000b300 con 0x7fd7380d1c70 2021-02-10T11:42:58.488 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:58.472 7fd73561a700 1 -- 172.21.2.112:0/285715263 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd73000b580 con 0x7fd7380d1c70 2021-02-10T11:42:58.488 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:58.472 7fd73561a700 1 -- 172.21.2.112:0/285715263 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd73000baa0 con 0x7fd7380d1c70 2021-02-10T11:42:58.488 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:58.472 7fd73c8f1700 1 -- 172.21.2.112:0/285715263 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd7380d1c70 msgr2=0x7fd7380f20a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:58.488 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:58.472 7fd73c8f1700 1 --2- 172.21.2.112:0/285715263 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd7380d1c70 0x7fd7380f20a0 secure :-1 s=READY pgs=2397 cs=0 l=1 rev1=1 rx=0x7fd730006f00 tx=0x7fd730009e90).stop 2021-02-10T11:42:58.488 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:58.473 7fd73c8f1700 1 -- 172.21.2.112:0/285715263 shutdown_connections 2021-02-10T11:42:58.488 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:58.473 7fd73c8f1700 1 --2- 172.21.2.112:0/285715263 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd7380d1c70 0x7fd7380f20a0 unknown :-1 s=CLOSED pgs=2397 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:58.488 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:58.473 7fd73c8f1700 1 -- 172.21.2.112:0/285715263 >> 172.21.2.112:0/285715263 conn(0x7fd7380ce450 msgr2=0x7fd7380d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:58.488 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:58.473 7fd73c8f1700 1 -- 172.21.2.112:0/285715263 shutdown_connections 2021-02-10T11:42:58.488 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:58.473 7fd73c8f1700 1 -- 172.21.2.112:0/285715263 wait complete. 2021-02-10T11:42:58.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:58.487 7fd73c8f1700 1 Processor -- start 2021-02-10T11:42:58.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:58.487 7fd73c8f1700 1 -- start start 2021-02-10T11:42:58.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:58.488 7fd73c8f1700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd738092010 0x7fd7380c33c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:58.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:58.488 7fd73c8f1700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd7380c39e0 0x7fd7380af040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:58.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:58.488 7fd73c8f1700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd7380af660 0x7fd7380afa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:58.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:58.488 7fd73c8f1700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7380b0500 con 0x7fd7380af660 2021-02-10T11:42:58.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:58.488 7fd73c8f1700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7380b0080 con 0x7fd7380c39e0 2021-02-10T11:42:58.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:58.488 7fd73c8f1700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7380b02a0 con 0x7fd738092010 2021-02-10T11:42:58.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:58.488 7fd736e1d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd7380af660 0x7fd7380afa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:58.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:58.488 7fd736e1d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd7380af660 0x7fd7380afa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:56660/0 (socket says 172.21.2.112:56660) 2021-02-10T11:42:58.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:58.488 7fd736e1d700 1 -- 172.21.2.112:0/1202797512 learned_addr learned my addr 172.21.2.112:0/1202797512 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:42:58.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:58.488 7fd736e1d700 1 -- 172.21.2.112:0/1202797512 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd738092010 msgr2=0x7fd7380c33c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:58.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:58.488 7fd736e1d700 1 --2- 172.21.2.112:0/1202797512 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd738092010 0x7fd7380c33c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:58.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:58.488 7fd736e1d700 1 -- 172.21.2.112:0/1202797512 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd7380c39e0 msgr2=0x7fd7380af040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:58.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:58.488 7fd736e1d700 1 --2- 172.21.2.112:0/1202797512 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd7380c39e0 0x7fd7380af040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:58.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:58.488 7fd736e1d700 1 -- 172.21.2.112:0/1202797512 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7380b0a70 con 0x7fd7380af660 2021-02-10T11:42:58.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:58.488 7fd736e1d700 1 --2- 172.21.2.112:0/1202797512 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd7380af660 0x7fd7380afa60 secure :-1 s=READY pgs=2398 cs=0 l=1 rev1=1 rx=0x7fd73000b8b0 tx=0x7fd73000bc50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:58.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:58.489 7fd72effd700 1 -- 172.21.2.112:0/1202797512 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd73000bd90 con 0x7fd7380af660 2021-02-10T11:42:58.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:58.489 7fd72effd700 1 -- 172.21.2.112:0/1202797512 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd730005820 con 0x7fd7380af660 2021-02-10T11:42:58.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:58.489 7fd72effd700 1 -- 172.21.2.112:0/1202797512 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd730005d00 con 0x7fd7380af660 2021-02-10T11:42:58.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:58.489 7fd73c8f1700 1 -- 172.21.2.112:0/1202797512 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7380b0a70 con 0x7fd7380af660 2021-02-10T11:42:58.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:58.489 7fd73c8f1700 1 -- 172.21.2.112:0/1202797512 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7380b1000 con 0x7fd7380af660 2021-02-10T11:42:58.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:58.489 7fd72effd700 1 -- 172.21.2.112:0/1202797512 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd730005c90 con 0x7fd7380af660 2021-02-10T11:42:58.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:58.490 7fd72effd700 1 --2- 172.21.2.112:0/1202797512 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd71c0186c0 0x7fd71c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:58.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:58.490 7fd72effd700 1 -- 172.21.2.112:0/1202797512 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 5659+0+0 (secure 0 0 0) 0x7fd730032960 con 0x7fd7380af660 2021-02-10T11:42:58.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:58.490 7fd73661c700 1 --2- 172.21.2.112:0/1202797512 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd71c0186c0 0x7fd71c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:58.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:58.491 7fd73661c700 1 --2- 172.21.2.112:0/1202797512 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd71c0186c0 0x7fd71c01abc0 crc :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:58.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:58.521 7fd73c8f1700 1 -- 172.21.2.112:0/1202797512 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7380e7df0 con 0x7fd7380af660 2021-02-10T11:42:58.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:58.523 7fd72effd700 1 -- 172.21.2.112:0/1202797512 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd7300058f0 con 0x7fd7380af660 2021-02-10T11:42:58.651 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:58.651 7fd73c8f1700 1 -- 172.21.2.112:0/1202797512 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd7380ce450 con 0x7fd7380af660 2021-02-10T11:42:58.651 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:58.651 7fd72effd700 1 -- 172.21.2.112:0/1202797512 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+1763 (secure 0 0 0) 0x7fd730005b00 con 0x7fd7380af660 2021-02-10T11:42:58.652 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:42:58.652 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":107,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11040,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/1814390924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":1814390924},{"type":"v1","addr":"172.21.2.112:6833","nonce":1814390924}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":101}],"filesystems":[{"mdsmap":{"epoch":107,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:42:19.351348","modified":"2021-02-10 11:42:58.002179","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":126,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":13}]} 2021-02-10T11:42:58.652 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 107 2021-02-10T11:42:58.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:58.653 7fd73c8f1700 1 -- 172.21.2.112:0/1202797512 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd71c0186c0 msgr2=0x7fd71c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:58.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:58.653 7fd73c8f1700 1 --2- 172.21.2.112:0/1202797512 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd71c0186c0 0x7fd71c01abc0 crc :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:58.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:58.653 7fd73c8f1700 1 -- 172.21.2.112:0/1202797512 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd7380af660 msgr2=0x7fd7380afa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:58.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:58.653 7fd73c8f1700 1 --2- 172.21.2.112:0/1202797512 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd7380af660 0x7fd7380afa60 secure :-1 s=READY pgs=2398 cs=0 l=1 rev1=1 rx=0x7fd73000b8b0 tx=0x7fd73000bc50).stop 2021-02-10T11:42:58.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:58.654 7fd73c8f1700 1 -- 172.21.2.112:0/1202797512 shutdown_connections 2021-02-10T11:42:58.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:58.654 7fd73c8f1700 1 --2- 172.21.2.112:0/1202797512 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd71c0186c0 0x7fd71c01abc0 unknown :-1 s=CLOSED pgs=1095 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:58.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:58.654 7fd73c8f1700 1 --2- 172.21.2.112:0/1202797512 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd7380c39e0 0x7fd7380af040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:58.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:58.654 7fd73c8f1700 1 --2- 172.21.2.112:0/1202797512 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd7380af660 0x7fd7380afa60 unknown :-1 s=CLOSED pgs=2398 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:58.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:58.654 7fd73c8f1700 1 --2- 172.21.2.112:0/1202797512 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd738092010 0x7fd7380c33c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:58.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:58.654 7fd73c8f1700 1 -- 172.21.2.112:0/1202797512 >> 172.21.2.112:0/1202797512 conn(0x7fd7380a0030 msgr2=0x7fd7380f22b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:58.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:58.654 7fd73c8f1700 1 -- 172.21.2.112:0/1202797512 shutdown_connections 2021-02-10T11:42:58.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:58.654 7fd73c8f1700 1 -- 172.21.2.112:0/1202797512 wait complete. 2021-02-10T11:42:58.662 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2021-02-10T11:42:59.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:59.080 7f458473c700 1 Processor -- start 2021-02-10T11:42:59.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:59.081 7f458473c700 1 -- start start 2021-02-10T11:42:59.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:59.081 7f458473c700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f457c0d1870 0x7f457c1b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:59.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:59.081 7f458473c700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f457c1b0a10 0x7f457c0cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:59.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:59.081 7f458473c700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f457c0d1c70 0x7f457c0cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:59.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:59.081 7f458473c700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f457c0fe200 con 0x7f457c0d1c70 2021-02-10T11:42:59.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:59.081 7f458473c700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f457c0e5f10 con 0x7f457c0d1870 2021-02-10T11:42:59.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:59.081 7f458473c700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f457c0f0b50 con 0x7f457c1b0a10 2021-02-10T11:42:59.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:59.081 7f4582d59700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f457c0d1c70 0x7f457c0cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:59.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:59.081 7f4582558700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f457c0d1870 0x7f457c1b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:59.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:59.081 7f4582558700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f457c0d1870 0x7f457c1b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3300/0 says I am v2:172.21.2.112:58944/0 (socket says 172.21.2.112:58944) 2021-02-10T11:42:59.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:59.081 7f4582d59700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f457c0d1c70 0x7f457c0cf950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:56664/0 (socket says 172.21.2.112:56664) 2021-02-10T11:42:59.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:59.081 7f4582558700 1 -- 172.21.2.112:0/4018301812 learned_addr learned my addr 172.21.2.112:0/4018301812 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:42:59.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:59.081 7f4581d57700 1 --2- 172.21.2.112:0/4018301812 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f457c1b0a10 0x7f457c0cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:59.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:59.082 7f4582d59700 1 -- 172.21.2.112:0/4018301812 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f457c1b0a10 msgr2=0x7f457c0cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:59.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:59.082 7f4582d59700 1 --2- 172.21.2.112:0/4018301812 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f457c1b0a10 0x7f457c0cf330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:59.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:59.082 7f4582d59700 1 -- 172.21.2.112:0/4018301812 >> v2:172.21.2.120:3300/0 conn(0x7f457c0d1870 msgr2=0x7f457c1b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:59.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:59.082 7f4582d59700 1 --2- 172.21.2.112:0/4018301812 >> v2:172.21.2.120:3300/0 conn(0x7f457c0d1870 0x7f457c1b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:59.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:59.082 7f4582d59700 1 -- 172.21.2.112:0/4018301812 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f457c0cff70 con 0x7f457c0d1c70 2021-02-10T11:42:59.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:59.082 7f4582558700 1 --2- 172.21.2.112:0/4018301812 >> v2:172.21.2.120:3300/0 conn(0x7f457c0d1870 0x7f457c1b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:42:59.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:59.082 7f4582d59700 1 --2- 172.21.2.112:0/4018301812 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f457c0d1c70 0x7f457c0cf950 secure :-1 s=READY pgs=2399 cs=0 l=1 rev1=1 rx=0x7f4574006e30 tx=0x7f4574009de0).ready entity=mon.0 client_cookie=7ce283b702e2803e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:59.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:59.082 7f4581556700 1 -- 172.21.2.112:0/4018301812 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f457400b300 con 0x7f457c0d1c70 2021-02-10T11:42:59.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:59.082 7f4581556700 1 -- 172.21.2.112:0/4018301812 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4574000d30 con 0x7f457c0d1c70 2021-02-10T11:42:59.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:59.082 7f4581556700 1 -- 172.21.2.112:0/4018301812 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f457400b770 con 0x7f457c0d1c70 2021-02-10T11:42:59.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:59.082 7f458473c700 1 -- 172.21.2.112:0/4018301812 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f457c0d1c70 msgr2=0x7f457c0cf950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:59.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:59.082 7f458473c700 1 --2- 172.21.2.112:0/4018301812 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f457c0d1c70 0x7f457c0cf950 secure :-1 s=READY pgs=2399 cs=0 l=1 rev1=1 rx=0x7f4574006e30 tx=0x7f4574009de0).stop 2021-02-10T11:42:59.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:59.083 7f458473c700 1 -- 172.21.2.112:0/4018301812 shutdown_connections 2021-02-10T11:42:59.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:59.083 7f458473c700 1 --2- 172.21.2.112:0/4018301812 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f457c0d1c70 0x7f457c0cf950 unknown :-1 s=CLOSED pgs=2399 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:59.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:59.083 7f458473c700 1 --2- 172.21.2.112:0/4018301812 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f457c1b0a10 0x7f457c0cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:59.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:59.083 7f458473c700 1 --2- 172.21.2.112:0/4018301812 >> v2:172.21.2.120:3300/0 conn(0x7f457c0d1870 0x7f457c1b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:59.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:59.083 7f458473c700 1 -- 172.21.2.112:0/4018301812 >> 172.21.2.112:0/4018301812 conn(0x7f457c0ce450 msgr2=0x7f457c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:59.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:59.083 7f458473c700 1 -- 172.21.2.112:0/4018301812 shutdown_connections 2021-02-10T11:42:59.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:59.083 7f458473c700 1 -- 172.21.2.112:0/4018301812 wait complete. 2021-02-10T11:42:59.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:59.098 7f458473c700 1 Processor -- start 2021-02-10T11:42:59.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:59.098 7f458473c700 1 -- start start 2021-02-10T11:42:59.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:59.098 7f458473c700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f457c0bfa90 0x7f457c18a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:59.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:59.098 7f458473c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f457c0bfe90 0x7f457c18a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:59.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:59.098 7f458473c700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f457c192510 0x7f457c192910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:59.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:59.098 7f458473c700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f457c189b50 con 0x7f457c0bfe90 2021-02-10T11:42:59.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:59.098 7f458473c700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f457c190d30 con 0x7f457c0bfa90 2021-02-10T11:42:59.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:59.098 7f458473c700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f457c193d70 con 0x7f457c192510 2021-02-10T11:42:59.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:59.098 7f4581d57700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f457c0bfe90 0x7f457c18a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:59.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:59.098 7f4581d57700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f457c0bfe90 0x7f457c18a670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:56668/0 (socket says 172.21.2.112:56668) 2021-02-10T11:42:59.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:59.098 7f4581d57700 1 -- 172.21.2.112:0/667275883 learned_addr learned my addr 172.21.2.112:0/667275883 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:42:59.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:59.098 7f4581d57700 1 -- 172.21.2.112:0/667275883 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f457c192510 msgr2=0x7f457c192910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:59.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:59.099 7f4581d57700 1 --2- 172.21.2.112:0/667275883 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f457c192510 0x7f457c192910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:59.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:59.099 7f4581d57700 1 -- 172.21.2.112:0/667275883 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f457c0bfa90 msgr2=0x7f457c18a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:59.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:59.099 7f4581d57700 1 --2- 172.21.2.112:0/667275883 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f457c0bfa90 0x7f457c18a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:59.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:59.099 7f4581d57700 1 -- 172.21.2.112:0/667275883 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f457c192f30 con 0x7f457c0bfe90 2021-02-10T11:42:59.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:59.099 7f4581d57700 1 --2- 172.21.2.112:0/667275883 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f457c0bfe90 0x7f457c18a670 secure :-1 s=READY pgs=2400 cs=0 l=1 rev1=1 rx=0x7f456c00b560 tx=0x7f456c00b900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:59.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:59.099 7f4572ffd700 1 -- 172.21.2.112:0/667275883 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f456c00bcf0 con 0x7f457c0bfe90 2021-02-10T11:42:59.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:59.099 7f4572ffd700 1 -- 172.21.2.112:0/667275883 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f456c000c90 con 0x7f457c0bfe90 2021-02-10T11:42:59.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:59.099 7f458473c700 1 -- 172.21.2.112:0/667275883 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f457c192f30 con 0x7f457c0bfe90 2021-02-10T11:42:59.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:59.099 7f458473c700 1 -- 172.21.2.112:0/667275883 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f457c192f30 con 0x7f457c0bfe90 2021-02-10T11:42:59.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:59.099 7f4572ffd700 1 -- 172.21.2.112:0/667275883 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f456c0022b0 con 0x7f457c0bfe90 2021-02-10T11:42:59.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:59.099 7f4572ffd700 1 -- 172.21.2.112:0/667275883 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f456c002280 con 0x7f457c0bfe90 2021-02-10T11:42:59.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:59.099 7f4572ffd700 1 --2- 172.21.2.112:0/667275883 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4564018730 0x7f456401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:59.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:59.099 7f4572ffd700 1 -- 172.21.2.112:0/667275883 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 5659+0+0 (secure 0 0 0) 0x7f456c032820 con 0x7f457c0bfe90 2021-02-10T11:42:59.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:59.100 7f4582558700 1 --2- 172.21.2.112:0/667275883 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4564018730 0x7f456401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:59.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:59.100 7f4582558700 1 --2- 172.21.2.112:0/667275883 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4564018730 0x7f456401ac10 crc :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:59.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:59.131 7f458473c700 1 -- 172.21.2.112:0/667275883 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f457c190d10 con 0x7f457c0bfe90 2021-02-10T11:42:59.132 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:59.133 7f4572ffd700 1 -- 172.21.2.112:0/667275883 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f456c000c90 con 0x7f457c0bfe90 2021-02-10T11:42:59.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:42:59.260 7f458473c700 1 -- 172.21.2.112:0/667275883 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f457c190d10 con 0x7f457c0bfe90 2021-02-10T11:43:00.018 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:00.018 7f4572ffd700 1 -- 172.21.2.112:0/667275883 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v108) v1 ==== 104+0+0 (secure 0 0 0) 0x7f456c002280 con 0x7f457c0bfe90 2021-02-10T11:43:00.031 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:00.032 7f458473c700 1 -- 172.21.2.112:0/667275883 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4564018730 msgr2=0x7f456401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:00.031 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:00.032 7f458473c700 1 --2- 172.21.2.112:0/667275883 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4564018730 0x7f456401ac10 crc :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:00.031 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:00.032 7f458473c700 1 -- 172.21.2.112:0/667275883 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f457c0bfe90 msgr2=0x7f457c18a670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:00.031 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:00.032 7f458473c700 1 --2- 172.21.2.112:0/667275883 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f457c0bfe90 0x7f457c18a670 secure :-1 s=READY pgs=2400 cs=0 l=1 rev1=1 rx=0x7f456c00b560 tx=0x7f456c00b900).stop 2021-02-10T11:43:00.032 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:00.032 7f458473c700 1 -- 172.21.2.112:0/667275883 shutdown_connections 2021-02-10T11:43:00.032 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:00.032 7f458473c700 1 --2- 172.21.2.112:0/667275883 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4564018730 0x7f456401ac10 unknown :-1 s=CLOSED pgs=1096 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:00.032 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:00.032 7f458473c700 1 --2- 172.21.2.112:0/667275883 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f457c192510 0x7f457c192910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:00.032 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:00.032 7f458473c700 1 --2- 172.21.2.112:0/667275883 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f457c0bfe90 0x7f457c18a670 unknown :-1 s=CLOSED pgs=2400 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:00.032 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:00.032 7f458473c700 1 --2- 172.21.2.112:0/667275883 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f457c0bfa90 0x7f457c18a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:00.032 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:00.032 7f458473c700 1 -- 172.21.2.112:0/667275883 >> 172.21.2.112:0/667275883 conn(0x7f457c0ed860 msgr2=0x7f457c0bf0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:00.032 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:00.032 7f458473c700 1 -- 172.21.2.112:0/667275883 shutdown_connections 2021-02-10T11:43:00.032 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:00.032 7f458473c700 1 -- 172.21.2.112:0/667275883 wait complete. 2021-02-10T11:43:00.040 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2021-02-10T11:43:00.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:00.454 7f8e8a9d9700 1 Processor -- start 2021-02-10T11:43:00.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:00.454 7f8e8a9d9700 1 -- start start 2021-02-10T11:43:00.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:00.455 7f8e8a9d9700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f8e8410b000 0x7f8e840cf760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:00.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:00.455 7f8e8a9d9700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e840fe1d0 con 0x7f8e840d1880 2021-02-10T11:43:00.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:00.455 7f8e8a9d9700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e841b0cc0 con 0x7f8e840d1c80 2021-02-10T11:43:00.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:00.455 7f8e8a9d9700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e84109a70 con 0x7f8e8410b000 2021-02-10T11:43:00.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:00.455 7f8e88ff6700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f8e8410b000 0x7f8e840cf760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:00.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:00.455 7f8e83fff700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f8e840d1880 0x7f8e841b04d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:56136/0 (socket says 172.21.2.112:56136) 2021-02-10T11:43:00.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:00.455 7f8e83fff700 1 -- 172.21.2.112:0/2620907098 learned_addr learned my addr 172.21.2.112:0/2620907098 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:43:00.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:00.456 7f8e88ff6700 1 -- 172.21.2.112:0/2620907098 >> v1:172.21.2.120:6789/0 conn(0x7f8e840d1c80 legacy=0x7f8e840cef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:00.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:00.456 7f8e88ff6700 1 -- 172.21.2.112:0/2620907098 >> v1:172.21.2.112:6789/0 conn(0x7f8e840d1880 legacy=0x7f8e841b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:00.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:00.456 7f8e82ffd700 1 -- 172.21.2.112:0/2620907098 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1781516155 0 0) 0x7f8e74001180 con 0x7f8e840d1880 2021-02-10T11:43:00.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:00.456 7f8e88ff6700 1 -- 172.21.2.112:0/2620907098 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e8410b400 con 0x7f8e8410b000 2021-02-10T11:43:00.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:00.456 7f8e88ff6700 1 --2- 172.21.2.112:0/2620907098 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8e8410b000 0x7f8e840cf760 secure :-1 s=READY pgs=2402 cs=0 l=1 rev1=1 rx=0x7f8e7c006e30 tx=0x7f8e7c009de0).ready entity=mon.0 client_cookie=39b8e6544d50906f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:00.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:00.456 7f8e82ffd700 1 -- 172.21.2.112:0/2620907098 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8e7c00b300 con 0x7f8e8410b000 2021-02-10T11:43:00.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:00.456 7f8e82ffd700 1 -- 172.21.2.112:0/2620907098 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e7c00b680 con 0x7f8e8410b000 2021-02-10T11:43:00.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:00.456 7f8e82ffd700 1 -- 172.21.2.112:0/2620907098 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8e7c00bb60 con 0x7f8e8410b000 2021-02-10T11:43:00.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:00.456 7f8e8a9d9700 1 -- 172.21.2.112:0/2620907098 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8e8410b000 msgr2=0x7f8e840cf760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:00.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:00.456 7f8e8a9d9700 1 --2- 172.21.2.112:0/2620907098 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8e8410b000 0x7f8e840cf760 secure :-1 s=READY pgs=2402 cs=0 l=1 rev1=1 rx=0x7f8e7c006e30 tx=0x7f8e7c009de0).stop 2021-02-10T11:43:00.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:00.457 7f8e8a9d9700 1 -- 172.21.2.112:0/2620907098 shutdown_connections 2021-02-10T11:43:00.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:00.457 7f8e8a9d9700 1 --2- 172.21.2.112:0/2620907098 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8e8410b000 0x7f8e840cf760 unknown :-1 s=CLOSED pgs=2402 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:00.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:00.457 7f8e8a9d9700 1 -- 172.21.2.112:0/2620907098 >> 172.21.2.112:0/2620907098 conn(0x7f8e840ce430 msgr2=0x7f8e840d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:00.473 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:00.457 7f8e8a9d9700 1 -- 172.21.2.112:0/2620907098 shutdown_connections 2021-02-10T11:43:00.473 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:00.457 7f8e8a9d9700 1 -- 172.21.2.112:0/2620907098 wait complete. 2021-02-10T11:43:00.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:00.471 7f8e8a9d9700 1 Processor -- start 2021-02-10T11:43:00.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:00.472 7f8e8a9d9700 1 -- start start 2021-02-10T11:43:00.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:00.472 7f8e8a9d9700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8e840c4190 0x7f8e840933e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:00.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:00.472 7f8e8a9d9700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8e84093a00 0x7f8e84093e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:00.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:00.472 7f8e8a9d9700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8e84094420 0x7f8e84100d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:00.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:00.472 7f8e8a9d9700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e84092b10 con 0x7f8e840c4190 2021-02-10T11:43:00.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:00.472 7f8e8a9d9700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e841c6970 con 0x7f8e84093a00 2021-02-10T11:43:00.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:00.472 7f8e8a9d9700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e841c6b90 con 0x7f8e84094420 2021-02-10T11:43:00.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:00.472 7f8e83fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8e840c4190 0x7f8e840933e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:00.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:00.472 7f8e83fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8e840c4190 0x7f8e840933e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:56680/0 (socket says 172.21.2.112:56680) 2021-02-10T11:43:00.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:00.472 7f8e83fff700 1 -- 172.21.2.112:0/4066173741 learned_addr learned my addr 172.21.2.112:0/4066173741 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:43:00.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:00.473 7f8e83fff700 1 -- 172.21.2.112:0/4066173741 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8e84094420 msgr2=0x7f8e84100d60 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:43:00.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:00.473 7f8e83fff700 1 --2- 172.21.2.112:0/4066173741 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8e84094420 0x7f8e84100d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:00.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:00.473 7f8e83fff700 1 -- 172.21.2.112:0/4066173741 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8e84093a00 msgr2=0x7f8e84093e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:00.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:00.473 7f8e83fff700 1 --2- 172.21.2.112:0/4066173741 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8e84093a00 0x7f8e84093e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:00.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:00.473 7f8e83fff700 1 -- 172.21.2.112:0/4066173741 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e84101380 con 0x7f8e840c4190 2021-02-10T11:43:00.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:00.473 7f8e83fff700 1 --2- 172.21.2.112:0/4066173741 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8e840c4190 0x7f8e840933e0 secure :-1 s=READY pgs=2403 cs=0 l=1 rev1=1 rx=0x7f8e7400a570 tx=0x7f8e7400a910).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:00.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:00.473 7f8e80ff9700 1 -- 172.21.2.112:0/4066173741 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8e7400ad00 con 0x7f8e840c4190 2021-02-10T11:43:00.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:00.473 7f8e80ff9700 1 -- 172.21.2.112:0/4066173741 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e74001180 con 0x7f8e840c4190 2021-02-10T11:43:00.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:00.473 7f8e80ff9700 1 -- 172.21.2.112:0/4066173741 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8e74004620 con 0x7f8e840c4190 2021-02-10T11:43:00.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:00.473 7f8e8a9d9700 1 -- 172.21.2.112:0/4066173741 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e84101380 con 0x7f8e840c4190 2021-02-10T11:43:00.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:00.473 7f8e8a9d9700 1 -- 172.21.2.112:0/4066173741 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e84101910 con 0x7f8e840c4190 2021-02-10T11:43:00.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:00.474 7f8e80ff9700 1 -- 172.21.2.112:0/4066173741 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8e740043a0 con 0x7f8e840c4190 2021-02-10T11:43:00.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:00.474 7f8e80ff9700 1 --2- 172.21.2.112:0/4066173741 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8e640186c0 0x7f8e6401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:00.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:00.474 7f8e80ff9700 1 -- 172.21.2.112:0/4066173741 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 5659+0+0 (secure 0 0 0) 0x7f8e74032590 con 0x7f8e840c4190 2021-02-10T11:43:00.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:00.474 7f8e837fe700 1 --2- 172.21.2.112:0/4066173741 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8e640186c0 0x7f8e6401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:00.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:00.474 7f8e837fe700 1 --2- 172.21.2.112:0/4066173741 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8e640186c0 0x7f8e6401abc0 crc :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:00.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:00.505 7f8e8a9d9700 1 -- 172.21.2.112:0/4066173741 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e84080340 con 0x7f8e840c4190 2021-02-10T11:43:00.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:00.507 7f8e80ff9700 1 -- 172.21.2.112:0/4066173741 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8e74006fd0 con 0x7f8e840c4190 2021-02-10T11:43:00.634 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:00.634 7f8e8a9d9700 1 -- 172.21.2.112:0/4066173741 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f8e84088070 con 0x7f8e840c4190 2021-02-10T11:43:01.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:01.024 7f8e80ff9700 1 -- 172.21.2.112:0/4066173741 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v127) v1 ==== 185+0+0 (secure 0 0 0) 0x7f8e7401f4b0 con 0x7f8e840c4190 2021-02-10T11:43:01.052 INFO:teuthology.orchestra.run.gibba012.stderr:pool 'cephfs_metadata' removed 2021-02-10T11:43:01.053 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:01.054 7f8e8a9d9700 1 -- 172.21.2.112:0/4066173741 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8e640186c0 msgr2=0x7f8e6401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:01.053 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:01.054 7f8e8a9d9700 1 --2- 172.21.2.112:0/4066173741 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8e640186c0 0x7f8e6401abc0 crc :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:01.054 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:01.054 7f8e8a9d9700 1 -- 172.21.2.112:0/4066173741 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8e840c4190 msgr2=0x7f8e840933e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:01.054 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:01.054 7f8e8a9d9700 1 --2- 172.21.2.112:0/4066173741 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8e840c4190 0x7f8e840933e0 secure :-1 s=READY pgs=2403 cs=0 l=1 rev1=1 rx=0x7f8e7400a570 tx=0x7f8e7400a910).stop 2021-02-10T11:43:01.054 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:01.054 7f8e8a9d9700 1 -- 172.21.2.112:0/4066173741 shutdown_connections 2021-02-10T11:43:01.054 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:01.054 7f8e8a9d9700 1 --2- 172.21.2.112:0/4066173741 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8e640186c0 0x7f8e6401abc0 unknown :-1 s=CLOSED pgs=1097 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:01.054 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:01.054 7f8e8a9d9700 1 --2- 172.21.2.112:0/4066173741 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8e84093a00 0x7f8e84093e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:01.055 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:01.054 7f8e8a9d9700 1 --2- 172.21.2.112:0/4066173741 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8e84094420 0x7f8e84100d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:01.055 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:01.054 7f8e8a9d9700 1 --2- 172.21.2.112:0/4066173741 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8e840c4190 0x7f8e840933e0 unknown :-1 s=CLOSED pgs=2403 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:01.055 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:01.054 7f8e8a9d9700 1 -- 172.21.2.112:0/4066173741 >> 172.21.2.112:0/4066173741 conn(0x7f8e8409fb30 msgr2=0x7f8e840ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:01.055 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:01.054 7f8e8a9d9700 1 -- 172.21.2.112:0/4066173741 shutdown_connections 2021-02-10T11:43:01.055 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:01.054 7f8e8a9d9700 1 -- 172.21.2.112:0/4066173741 wait complete. 2021-02-10T11:43:01.062 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2021-02-10T11:43:01.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:01.478 7fa629814700 1 Processor -- start 2021-02-10T11:43:01.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:01.478 7fa629814700 1 -- start start 2021-02-10T11:43:01.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:01.479 7fa629814700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa6240d1870 0x7fa6241b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:01.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:01.479 7fa629814700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fa6241b0ae0 0x7fa624108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:01.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:01.479 7fa629814700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa624109a60 con 0x7fa6240d1c70 2021-02-10T11:43:01.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:01.479 7fa629814700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6240e5ef0 con 0x7fa6241b0ae0 2021-02-10T11:43:01.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:01.479 7fa629814700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa624109750 con 0x7fa6240d1870 2021-02-10T11:43:01.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:01.479 7fa623d75700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fa6240d1c70 0x7fa624108f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:56150/0 (socket says 172.21.2.112:56150) 2021-02-10T11:43:01.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:01.479 7fa623d75700 1 -- 172.21.2.112:0/622141783 learned_addr learned my addr 172.21.2.112:0/622141783 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:43:01.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:01.480 7fa622572700 1 -- 172.21.2.112:0/622141783 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3073633912 0 0) 0x7fa61c001500 con 0x7fa6240d1c70 2021-02-10T11:43:01.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:01.480 7fa622572700 1 -- 172.21.2.112:0/622141783 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa610002fe0 con 0x7fa6240d1c70 2021-02-10T11:43:01.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:01.480 7fa623574700 1 --2- 172.21.2.112:0/622141783 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa6240d1870 0x7fa6241b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:01.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:01.480 7fa622d73700 1 --2- 172.21.2.112:0/622141783 >> v2:172.21.2.120:3300/0 conn(0x7fa6241b0ae0 0x7fa624108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:01.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:01.480 7fa622572700 1 -- 172.21.2.112:0/622141783 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1591702038 0 0) 0x7fa61c001460 con 0x7fa6240d1c70 2021-02-10T11:43:01.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:01.480 7fa622572700 1 -- 172.21.2.112:0/622141783 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa6240d1870 msgr2=0x7fa6241b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:01.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:01.480 7fa622572700 1 --2- 172.21.2.112:0/622141783 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa6240d1870 0x7fa6241b04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:01.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:01.480 7fa622572700 1 -- 172.21.2.112:0/622141783 >> v2:172.21.2.120:3300/0 conn(0x7fa6241b0ae0 msgr2=0x7fa624108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:01.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:01.480 7fa622572700 1 --2- 172.21.2.112:0/622141783 >> v2:172.21.2.120:3300/0 conn(0x7fa6241b0ae0 0x7fa624108940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:01.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:01.480 7fa622572700 1 -- 172.21.2.112:0/622141783 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa624088070 con 0x7fa6240d1c70 2021-02-10T11:43:01.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:01.480 7fa622572700 1 -- 172.21.2.112:0/622141783 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7fa61c001980 con 0x7fa6240d1c70 2021-02-10T11:43:01.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:01.480 7fa622572700 1 -- 172.21.2.112:0/622141783 >> v1:172.21.2.112:6789/0 conn(0x7fa6240d1c70 legacy=0x7fa624108f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:01.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:01.480 7fa622572700 1 --2- 172.21.2.112:0/622141783 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa610003a60 0x7fa610005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:01.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:01.480 7fa622572700 1 --2- 172.21.2.112:0/622141783 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa610006770 0x7fa610008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:01.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:01.480 7fa622572700 1 --2- 172.21.2.112:0/622141783 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa610009590 0x7fa61000ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:01.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:01.480 7fa622572700 1 -- 172.21.2.112:0/622141783 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa61000c2d0 con 0x7fa610006770 2021-02-10T11:43:01.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:01.480 7fa622572700 1 -- 172.21.2.112:0/622141783 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa610050490 con 0x7fa610003a60 2021-02-10T11:43:01.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:01.480 7fa622572700 1 -- 172.21.2.112:0/622141783 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6100506b0 con 0x7fa610009590 2021-02-10T11:43:01.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:01.480 7fa622d73700 1 --2- 172.21.2.112:0/622141783 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa610006770 0x7fa610008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:01.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:01.480 7fa622d73700 1 -- 172.21.2.112:0/622141783 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa610009590 msgr2=0x7fa61000ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:01.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:01.480 7fa623574700 1 --2- 172.21.2.112:0/622141783 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa610003a60 0x7fa610005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:01.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:01.480 7fa623d75700 1 --2- 172.21.2.112:0/622141783 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa610009590 0x7fa61000ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:01.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:01.480 7fa622d73700 1 --2- 172.21.2.112:0/622141783 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa610009590 0x7fa61000ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:01.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:01.480 7fa622d73700 1 -- 172.21.2.112:0/622141783 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa610003a60 msgr2=0x7fa610005ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:01.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:01.480 7fa622d73700 1 --2- 172.21.2.112:0/622141783 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa610003a60 0x7fa610005ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:01.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:01.480 7fa622d73700 1 -- 172.21.2.112:0/622141783 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa610050950 con 0x7fa610006770 2021-02-10T11:43:01.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:01.480 7fa623574700 1 --2- 172.21.2.112:0/622141783 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa610003a60 0x7fa610005ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:43:01.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:01.480 7fa623d75700 1 --2- 172.21.2.112:0/622141783 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa610009590 0x7fa61000ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:43:01.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:01.480 7fa622d73700 1 --2- 172.21.2.112:0/622141783 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa610006770 0x7fa610008c90 secure :-1 s=READY pgs=2405 cs=0 l=1 rev1=1 rx=0x7fa60c006360 tx=0x7fa60c009c30).ready entity=mon.0 client_cookie=6db780849dd0ac96 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:01.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:01.481 7fa622572700 1 -- 172.21.2.112:0/622141783 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa60c0009b0 con 0x7fa610006770 2021-02-10T11:43:01.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:01.481 7fa622572700 1 -- 172.21.2.112:0/622141783 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa60c000d20 con 0x7fa610006770 2021-02-10T11:43:01.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:01.481 7fa622572700 1 -- 172.21.2.112:0/622141783 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa60c002220 con 0x7fa610006770 2021-02-10T11:43:01.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:01.481 7fa629814700 1 -- 172.21.2.112:0/622141783 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa610006770 msgr2=0x7fa610008c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:01.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:01.481 7fa629814700 1 --2- 172.21.2.112:0/622141783 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa610006770 0x7fa610008c90 secure :-1 s=READY pgs=2405 cs=0 l=1 rev1=1 rx=0x7fa60c006360 tx=0x7fa60c009c30).stop 2021-02-10T11:43:01.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:01.481 7fa623d75700 1 -- 172.21.2.112:0/622141783 reap_dead start 2021-02-10T11:43:01.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:01.481 7fa629814700 1 -- 172.21.2.112:0/622141783 shutdown_connections 2021-02-10T11:43:01.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:01.481 7fa629814700 1 -- 172.21.2.112:0/622141783 >> 172.21.2.112:0/622141783 conn(0x7fa6240ce420 msgr2=0x7fa6240d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:01.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:01.481 7fa629814700 1 -- 172.21.2.112:0/622141783 shutdown_connections 2021-02-10T11:43:01.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:01.481 7fa629814700 1 -- 172.21.2.112:0/622141783 wait complete. 2021-02-10T11:43:01.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:01.496 7fa629814700 1 Processor -- start 2021-02-10T11:43:01.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:01.496 7fa629814700 1 -- start start 2021-02-10T11:43:01.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:01.496 7fa629814700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa6240a8f60 0x7fa6240b2250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:01.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:01.496 7fa629814700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa6241a7ae0 0x7fa6241a7ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:01.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:01.496 7fa629814700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa6240b50a0 0x7fa6240b54a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:01.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:01.496 7fa629814700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6240b5b40 con 0x7fa6240a8f60 2021-02-10T11:43:01.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:01.496 7fa629814700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6240b5d60 con 0x7fa6240b50a0 2021-02-10T11:43:01.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:01.496 7fa629814700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6240b5f80 con 0x7fa6241a7ae0 2021-02-10T11:43:01.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:01.497 7fa623574700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa6240a8f60 0x7fa6240b2250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:01.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:01.497 7fa623574700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa6240a8f60 0x7fa6240b2250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:56698/0 (socket says 172.21.2.112:56698) 2021-02-10T11:43:01.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:01.497 7fa623574700 1 -- 172.21.2.112:0/1460114701 learned_addr learned my addr 172.21.2.112:0/1460114701 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:43:01.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:01.497 7fa622d73700 1 --2- 172.21.2.112:0/1460114701 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa6241a7ae0 0x7fa6241a7ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:01.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:01.497 7fa623d75700 1 --2- 172.21.2.112:0/1460114701 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa6240b50a0 0x7fa6240b54a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:01.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:01.497 7fa623574700 1 -- 172.21.2.112:0/1460114701 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa6241a7ae0 msgr2=0x7fa6241a7ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:01.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:01.497 7fa623574700 1 --2- 172.21.2.112:0/1460114701 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa6241a7ae0 0x7fa6241a7ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:01.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:01.497 7fa623574700 1 -- 172.21.2.112:0/1460114701 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa6240b50a0 msgr2=0x7fa6240b54a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:01.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:01.497 7fa623574700 1 --2- 172.21.2.112:0/1460114701 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa6240b50a0 0x7fa6240b54a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:01.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:01.497 7fa623574700 1 -- 172.21.2.112:0/1460114701 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6240b6be0 con 0x7fa6240a8f60 2021-02-10T11:43:01.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:01.497 7fa622d73700 1 --2- 172.21.2.112:0/1460114701 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa6241a7ae0 0x7fa6241a7ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:43:01.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:01.497 7fa623574700 1 --2- 172.21.2.112:0/1460114701 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa6240a8f60 0x7fa6240b2250 secure :-1 s=READY pgs=2406 cs=0 l=1 rev1=1 rx=0x7fa614006f10 tx=0x7fa61400b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:01.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:01.497 7fa61bfff700 1 -- 172.21.2.112:0/1460114701 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa6140009b0 con 0x7fa6240a8f60 2021-02-10T11:43:01.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:01.497 7fa61bfff700 1 -- 172.21.2.112:0/1460114701 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa61400bd80 con 0x7fa6240a8f60 2021-02-10T11:43:01.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:01.497 7fa61bfff700 1 -- 172.21.2.112:0/1460114701 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa614002220 con 0x7fa6240a8f60 2021-02-10T11:43:01.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:01.497 7fa629814700 1 -- 172.21.2.112:0/1460114701 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6240b6be0 con 0x7fa6240a8f60 2021-02-10T11:43:01.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:01.497 7fa629814700 1 -- 172.21.2.112:0/1460114701 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6240b6be0 con 0x7fa6240a8f60 2021-02-10T11:43:01.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:01.498 7fa61bfff700 1 -- 172.21.2.112:0/1460114701 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa614000c90 con 0x7fa6240a8f60 2021-02-10T11:43:01.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:01.498 7fa61bfff700 1 --2- 172.21.2.112:0/1460114701 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa608018730 0x7fa60801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:01.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:01.498 7fa61bfff700 1 -- 172.21.2.112:0/1460114701 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5221+0+0 (secure 0 0 0) 0x7fa6140324c0 con 0x7fa6240a8f60 2021-02-10T11:43:01.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:01.498 7fa622d73700 1 --2- 172.21.2.112:0/1460114701 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa608018730 0x7fa60801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:01.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:01.499 7fa622d73700 1 --2- 172.21.2.112:0/1460114701 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa608018730 0x7fa60801ac10 crc :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:01.529 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:01.529 7fa629814700 1 -- 172.21.2.112:0/1460114701 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6240825d0 con 0x7fa6240a8f60 2021-02-10T11:43:01.531 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:01.532 7fa61bfff700 1 -- 172.21.2.112:0/1460114701 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa614006ba0 con 0x7fa6240a8f60 2021-02-10T11:43:01.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:01.659 7fa629814700 1 -- 172.21.2.112:0/1460114701 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fa6240825d0 con 0x7fa6240a8f60 2021-02-10T11:43:02.030 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:02.031 7fa61bfff700 1 -- 172.21.2.112:0/1460114701 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v128) v1 ==== 173+0+0 (secure 0 0 0) 0x7fa614000c90 con 0x7fa6240a8f60 2021-02-10T11:43:02.076 INFO:teuthology.orchestra.run.gibba012.stderr:pool 'cephfs_data' removed 2021-02-10T11:43:02.078 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:02.078 7fa629814700 1 -- 172.21.2.112:0/1460114701 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa608018730 msgr2=0x7fa60801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:02.078 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:02.078 7fa629814700 1 --2- 172.21.2.112:0/1460114701 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa608018730 0x7fa60801ac10 crc :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:02.078 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:02.079 7fa629814700 1 -- 172.21.2.112:0/1460114701 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa6240a8f60 msgr2=0x7fa6240b2250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:02.078 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:02.079 7fa629814700 1 --2- 172.21.2.112:0/1460114701 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa6240a8f60 0x7fa6240b2250 secure :-1 s=READY pgs=2406 cs=0 l=1 rev1=1 rx=0x7fa614006f10 tx=0x7fa61400b990).stop 2021-02-10T11:43:02.079 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:02.079 7fa629814700 1 -- 172.21.2.112:0/1460114701 shutdown_connections 2021-02-10T11:43:02.079 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:02.079 7fa629814700 1 --2- 172.21.2.112:0/1460114701 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa608018730 0x7fa60801ac10 unknown :-1 s=CLOSED pgs=1098 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:02.079 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:02.079 7fa629814700 1 --2- 172.21.2.112:0/1460114701 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa6240b50a0 0x7fa6240b54a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:02.079 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:02.079 7fa629814700 1 --2- 172.21.2.112:0/1460114701 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa6241a7ae0 0x7fa6241a7ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:02.079 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:02.079 7fa629814700 1 --2- 172.21.2.112:0/1460114701 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa6240a8f60 0x7fa6240b2250 unknown :-1 s=CLOSED pgs=2406 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:02.079 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:02.079 7fa629814700 1 -- 172.21.2.112:0/1460114701 >> 172.21.2.112:0/1460114701 conn(0x7fa6240bc070 msgr2=0x7fa6240ba770 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:02.079 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:02.079 7fa629814700 1 -- 172.21.2.112:0/1460114701 shutdown_connections 2021-02-10T11:43:02.079 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:02.079 7fa629814700 1 -- 172.21.2.112:0/1460114701 wait complete. 2021-02-10T11:43:02.087 INFO:tasks.ceph.mds.a:Restarting daemon 2021-02-10T11:43:02.087 INFO:tasks.ceph.mds.a:Stopping old one... 2021-02-10T11:43:02.087 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2021-02-10T11:43:02.087 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:43:02.087 INFO:tasks.ceph.mds.b:Restarting daemon 2021-02-10T11:43:02.088 INFO:tasks.ceph.mds.b:Stopping old one... 2021-02-10T11:43:02.088 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2021-02-10T11:43:02.088 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:43:02.093 INFO:tasks.ceph.mds.a:Stopped 2021-02-10T11:43:02.093 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2021-02-10T11:43:02.094 INFO:tasks.ceph.mds.b:Stopped 2021-02-10T11:43:02.094 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2021-02-10T11:43:02.136 INFO:tasks.ceph.mds.a:Started 2021-02-10T11:43:02.136 DEBUG:teuthology.parallel:result is None 2021-02-10T11:43:02.137 INFO:tasks.ceph.mds.b:Started 2021-02-10T11:43:02.137 DEBUG:teuthology.parallel:result is None 2021-02-10T11:43:02.137 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd blacklist clear 2021-02-10T11:43:02.541 INFO:tasks.ceph.mds.b.gibba020.stdout:starting mds.b at 2021-02-10T11:43:02.545 INFO:tasks.ceph.mds.a.gibba012.stdout:starting mds.a at 2021-02-10T11:43:02.569 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:02.552 7f1eb5ceb700 1 Processor -- start 2021-02-10T11:43:02.570 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:02.552 7f1eb5ceb700 1 -- start start 2021-02-10T11:43:02.570 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:02.553 7f1eb5ceb700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1eb00d0340 0x7f1eb00ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:02.570 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:02.553 7f1eb5ceb700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1eb00f0b10 con 0x7f1eb00d1870 2021-02-10T11:43:02.570 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:02.553 7f1eb5ceb700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1eb01b0be0 con 0x7f1eb00d1c70 2021-02-10T11:43:02.570 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:02.553 7f1eb5ceb700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb00e5ed0 con 0x7f1eb00d0340 2021-02-10T11:43:02.570 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:02.553 7f1eaf7fe700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f1eb00d1870 0x7f1eb01b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:56180/0 (socket says 172.21.2.112:56180) 2021-02-10T11:43:02.570 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:02.553 7f1eaf7fe700 1 -- 172.21.2.112:0/2261818901 learned_addr learned my addr 172.21.2.112:0/2261818901 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:43:02.570 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:02.553 7f1eaeffd700 1 --2- 172.21.2.112:0/2261818901 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1eb00d0340 0x7f1eb00ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:02.570 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:02.553 7f1eae7fc700 1 -- 172.21.2.112:0/2261818901 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1033353182 0 0) 0x7f1ea0000ce0 con 0x7f1eb00d1870 2021-02-10T11:43:02.570 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:02.553 7f1eae7fc700 1 -- 172.21.2.112:0/2261818901 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e98002fe0 con 0x7f1eb00d1870 2021-02-10T11:43:02.570 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:02.554 7f1eaeffd700 1 -- 172.21.2.112:0/2261818901 >> v1:172.21.2.120:6789/0 conn(0x7f1eb00d1c70 legacy=0x7f1eb00efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:02.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:02.554 7f1eae7fc700 1 -- 172.21.2.112:0/2261818901 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3768608486 0 0) 0x7f1ea8000f90 con 0x7f1eb00d1c70 2021-02-10T11:43:02.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:02.554 7f1eaeffd700 1 -- 172.21.2.112:0/2261818901 >> v1:172.21.2.112:6789/0 conn(0x7f1eb00d1870 legacy=0x7f1eb01b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:02.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:02.554 7f1eaeffd700 1 -- 172.21.2.112:0/2261818901 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1eb00f0450 con 0x7f1eb00d0340 2021-02-10T11:43:02.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:02.554 7f1eaeffd700 1 --2- 172.21.2.112:0/2261818901 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1eb00d0340 0x7f1eb00ef640 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7f1ea4006650 tx=0x7f1ea4009a80).ready entity=mon.2 client_cookie=46470fc2ad9e300 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:02.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:02.554 7f1eae7fc700 1 -- 172.21.2.112:0/2261818901 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1ea400c0c0 con 0x7f1eb00d0340 2021-02-10T11:43:02.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:02.554 7f1eae7fc700 1 -- 172.21.2.112:0/2261818901 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ea400c300 con 0x7f1eb00d0340 2021-02-10T11:43:02.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:02.554 7f1eae7fc700 1 -- 172.21.2.112:0/2261818901 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1ea400c820 con 0x7f1eb00d0340 2021-02-10T11:43:02.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:02.554 7f1eb5ceb700 1 -- 172.21.2.112:0/2261818901 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1eb00d0340 msgr2=0x7f1eb00ef640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:02.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:02.554 7f1eb5ceb700 1 --2- 172.21.2.112:0/2261818901 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1eb00d0340 0x7f1eb00ef640 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7f1ea4006650 tx=0x7f1ea4009a80).stop 2021-02-10T11:43:02.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:02.555 7f1eb5ceb700 1 -- 172.21.2.112:0/2261818901 shutdown_connections 2021-02-10T11:43:02.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:02.555 7f1eb5ceb700 1 --2- 172.21.2.112:0/2261818901 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1eb00d0340 0x7f1eb00ef640 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:02.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:02.555 7f1eb5ceb700 1 -- 172.21.2.112:0/2261818901 >> 172.21.2.112:0/2261818901 conn(0x7f1eb00ce450 msgr2=0x7f1eb00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:02.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:02.555 7f1eb5ceb700 1 -- 172.21.2.112:0/2261818901 shutdown_connections 2021-02-10T11:43:02.572 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:02.555 7f1eb5ceb700 1 -- 172.21.2.112:0/2261818901 wait complete. 2021-02-10T11:43:02.573 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:02.570 7f1eb5ceb700 1 Processor -- start 2021-02-10T11:43:02.573 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:02.571 7f1eb5ceb700 1 -- start start 2021-02-10T11:43:02.573 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:02.571 7f1eb5ceb700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1eb00a2fb0 0x7f1eb00a33b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:02.573 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:02.571 7f1eb5ceb700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1eb00a39d0 0x7f1eb00a4860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:02.573 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:02.571 7f1eb5ceb700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1eb00a3dd0 0x7f1eb00a4e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:02.573 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:02.571 7f1eb5ceb700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb00a2620 con 0x7f1eb00a2fb0 2021-02-10T11:43:02.573 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:02.571 7f1eb5ceb700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb00a2840 con 0x7f1eb00a3dd0 2021-02-10T11:43:02.574 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:02.571 7f1eb5ceb700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb00a2b80 con 0x7f1eb00a39d0 2021-02-10T11:43:02.574 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:02.571 7f1eaf7fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1eb00a2fb0 0x7f1eb00a33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:02.574 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:02.571 7f1eaf7fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1eb00a2fb0 0x7f1eb00a33b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:56726/0 (socket says 172.21.2.112:56726) 2021-02-10T11:43:02.574 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:02.571 7f1eaf7fe700 1 -- 172.21.2.112:0/2512781349 learned_addr learned my addr 172.21.2.112:0/2512781349 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:43:02.574 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:02.571 7f1eaeffd700 1 --2- 172.21.2.112:0/2512781349 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1eb00a39d0 0x7f1eb00a4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:02.574 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:02.571 7f1eaffff700 1 --2- 172.21.2.112:0/2512781349 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1eb00a3dd0 0x7f1eb00a4e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:02.574 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:02.571 7f1eaf7fe700 1 -- 172.21.2.112:0/2512781349 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1eb00a39d0 msgr2=0x7f1eb00a4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:02.574 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:02.571 7f1eaf7fe700 1 --2- 172.21.2.112:0/2512781349 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1eb00a39d0 0x7f1eb00a4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:02.574 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:02.571 7f1eaf7fe700 1 -- 172.21.2.112:0/2512781349 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1eb00a3dd0 msgr2=0x7f1eb00a4e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:02.574 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:02.571 7f1eaf7fe700 1 --2- 172.21.2.112:0/2512781349 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1eb00a3dd0 0x7f1eb00a4e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:02.574 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:02.571 7f1eaf7fe700 1 -- 172.21.2.112:0/2512781349 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1eb00a54a0 con 0x7f1eb00a2fb0 2021-02-10T11:43:02.574 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:02.571 7f1eaffff700 1 --2- 172.21.2.112:0/2512781349 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1eb00a3dd0 0x7f1eb00a4e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:43:02.575 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:02.572 7f1eaf7fe700 1 --2- 172.21.2.112:0/2512781349 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1eb00a2fb0 0x7f1eb00a33b0 secure :-1 s=READY pgs=2411 cs=0 l=1 rev1=1 rx=0x7f1ea0000b80 tx=0x7f1ea000a980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:02.575 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:02.572 7f1e97fff700 1 -- 172.21.2.112:0/2512781349 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1ea0001490 con 0x7f1eb00a2fb0 2021-02-10T11:43:02.575 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:02.572 7f1e97fff700 1 -- 172.21.2.112:0/2512781349 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ea000ad70 con 0x7f1eb00a2fb0 2021-02-10T11:43:02.575 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:02.572 7f1e97fff700 1 -- 172.21.2.112:0/2512781349 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1ea00045b0 con 0x7f1eb00a2fb0 2021-02-10T11:43:02.575 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:02.572 7f1eb5ceb700 1 -- 172.21.2.112:0/2512781349 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1eb00a54a0 con 0x7f1eb00a2fb0 2021-02-10T11:43:02.575 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:02.572 7f1eb5ceb700 1 -- 172.21.2.112:0/2512781349 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1eb00a54a0 con 0x7f1eb00a2fb0 2021-02-10T11:43:02.575 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:02.572 7f1e97fff700 1 -- 172.21.2.112:0/2512781349 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1ea00043a0 con 0x7f1eb00a2fb0 2021-02-10T11:43:02.575 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:02.572 7f1e97fff700 1 --2- 172.21.2.112:0/2512781349 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1e90018730 0x7f1e9001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:02.575 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:02.572 7f1e97fff700 1 -- 172.21.2.112:0/2512781349 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 4839+0+0 (secure 0 0 0) 0x7f1ea0032360 con 0x7f1eb00a2fb0 2021-02-10T11:43:02.575 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:02.572 7f1eaeffd700 1 --2- 172.21.2.112:0/2512781349 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1e90018730 0x7f1e9001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:02.575 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:02.573 7f1eaeffd700 1 --2- 172.21.2.112:0/2512781349 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1e90018730 0x7f1e9001ac10 crc :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:02.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:02.601 7f1eb5ceb700 1 -- 172.21.2.112:0/2512781349 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1eb00ebc50 con 0x7f1eb00a2fb0 2021-02-10T11:43:02.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:02.604 7f1e97fff700 1 -- 172.21.2.112:0/2512781349 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1ea0004f00 con 0x7f1eb00a2fb0 2021-02-10T11:43:02.742 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:02.743 7f1eb5ceb700 1 -- 172.21.2.112:0/2512781349 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7f1eb00f0b10 con 0x7f1eb00a2fb0 2021-02-10T11:43:03.068 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:03.068 7f1e97fff700 1 -- 172.21.2.112:0/2512781349 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v129) v1 ==== 97+0+0 (secure 0 0 0) 0x7f1ea0004f00 con 0x7f1eb00a2fb0 2021-02-10T11:43:03.108 INFO:teuthology.orchestra.run.gibba012.stderr: removed all blacklist entries 2021-02-10T11:43:03.110 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:03.110 7f1eb5ceb700 1 -- 172.21.2.112:0/2512781349 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1e90018730 msgr2=0x7f1e9001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:03.110 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:03.110 7f1eb5ceb700 1 --2- 172.21.2.112:0/2512781349 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1e90018730 0x7f1e9001ac10 crc :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:03.111 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:03.110 7f1eb5ceb700 1 -- 172.21.2.112:0/2512781349 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1eb00a2fb0 msgr2=0x7f1eb00a33b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:03.111 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:03.110 7f1eb5ceb700 1 --2- 172.21.2.112:0/2512781349 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1eb00a2fb0 0x7f1eb00a33b0 secure :-1 s=READY pgs=2411 cs=0 l=1 rev1=1 rx=0x7f1ea0000b80 tx=0x7f1ea000a980).stop 2021-02-10T11:43:03.111 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:03.111 7f1eb5ceb700 1 -- 172.21.2.112:0/2512781349 shutdown_connections 2021-02-10T11:43:03.111 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:03.111 7f1eb5ceb700 1 --2- 172.21.2.112:0/2512781349 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1e90018730 0x7f1e9001ac10 unknown :-1 s=CLOSED pgs=1099 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:03.111 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:03.111 7f1eb5ceb700 1 --2- 172.21.2.112:0/2512781349 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1eb00a3dd0 0x7f1eb00a4e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:03.111 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:03.111 7f1eb5ceb700 1 --2- 172.21.2.112:0/2512781349 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1eb00a39d0 0x7f1eb00a4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:03.111 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:03.111 7f1eb5ceb700 1 --2- 172.21.2.112:0/2512781349 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1eb00a2fb0 0x7f1eb00a33b0 unknown :-1 s=CLOSED pgs=2411 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:03.111 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:03.111 7f1eb5ceb700 1 -- 172.21.2.112:0/2512781349 >> 172.21.2.112:0/2512781349 conn(0x7f1eb008f030 msgr2=0x7f1eb00b5e80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:03.111 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:03.111 7f1eb5ceb700 1 -- 172.21.2.112:0/2512781349 shutdown_connections 2021-02-10T11:43:03.111 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:03.111 7f1eb5ceb700 1 -- 172.21.2.112:0/2512781349 wait complete. 2021-02-10T11:43:03.119 INFO:tasks.cephfs.cephfs_test_case:['0', '1', '2', '3'] 2021-02-10T11:43:03.119 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth ls --format=json-pretty 2021-02-10T11:43:03.555 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:03.531 7f02118de700 1 Processor -- start 2021-02-10T11:43:03.555 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:03.531 7f02118de700 1 -- start start 2021-02-10T11:43:03.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:03.532 7f02118de700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f020c0d1870 0x7f020c1b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:03.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:03.532 7f02118de700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f020c1b0a10 0x7f020c0cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:03.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:03.532 7f02118de700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f020c0f0b10 con 0x7f020c0d1c70 2021-02-10T11:43:03.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:03.532 7f02118de700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f020c0fe230 con 0x7f020c1b0a10 2021-02-10T11:43:03.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:03.532 7f02118de700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f020c0e5ed0 con 0x7f020c0d1870 2021-02-10T11:43:03.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:03.532 7f020b574700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f020c0d1870 0x7f020c1b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:03.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:03.533 7f020b574700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f020c0d1870 0x7f020c1b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3301/0 says I am v2:172.21.2.112:56414/0 (socket says 172.21.2.112:56414) 2021-02-10T11:43:03.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:03.533 7f020b574700 1 -- 172.21.2.112:0/3364995385 learned_addr learned my addr 172.21.2.112:0/3364995385 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:43:03.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:03.533 7f020ad73700 1 --2- 172.21.2.112:0/3364995385 >> v2:172.21.2.120:3300/0 conn(0x7f020c1b0a10 0x7f020c0cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:03.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:03.533 7f020b574700 1 -- 172.21.2.112:0/3364995385 >> v2:172.21.2.120:3300/0 conn(0x7f020c1b0a10 msgr2=0x7f020c0cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:03.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:03.533 7f020b574700 1 --2- 172.21.2.112:0/3364995385 >> v2:172.21.2.120:3300/0 conn(0x7f020c1b0a10 0x7f020c0cf330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:03.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:03.533 7f020b574700 1 -- 172.21.2.112:0/3364995385 >> v1:172.21.2.120:6789/0 conn(0x7f020c0d1c70 legacy=0x7f020c0cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:03.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:03.533 7f020b574700 1 -- 172.21.2.112:0/3364995385 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f020c0f0650 con 0x7f020c0d1870 2021-02-10T11:43:03.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:03.533 7f020ad73700 1 --2- 172.21.2.112:0/3364995385 >> v2:172.21.2.120:3300/0 conn(0x7f020c1b0a10 0x7f020c0cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:43:03.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:03.533 7f020b574700 1 --2- 172.21.2.112:0/3364995385 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f020c0d1870 0x7f020c1b03f0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7f01fc006700 tx=0x7f01fc009b80).ready entity=mon.2 client_cookie=e841d2d351ecaa5d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:03.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:03.533 7f020a572700 1 -- 172.21.2.112:0/3364995385 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f01fc00c0c0 con 0x7f020c0d1870 2021-02-10T11:43:03.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:03.534 7f020a572700 1 -- 172.21.2.112:0/3364995385 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01fc00c440 con 0x7f020c0d1870 2021-02-10T11:43:03.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:03.534 7f020a572700 1 -- 172.21.2.112:0/3364995385 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f01fc00c960 con 0x7f020c0d1870 2021-02-10T11:43:03.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:03.534 7f02118de700 1 -- 172.21.2.112:0/3364995385 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f020c0d1870 msgr2=0x7f020c1b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:03.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:03.534 7f02118de700 1 --2- 172.21.2.112:0/3364995385 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f020c0d1870 0x7f020c1b03f0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7f01fc006700 tx=0x7f01fc009b80).stop 2021-02-10T11:43:03.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:03.534 7f02118de700 1 -- 172.21.2.112:0/3364995385 shutdown_connections 2021-02-10T11:43:03.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:03.534 7f02118de700 1 --2- 172.21.2.112:0/3364995385 >> v2:172.21.2.120:3300/0 conn(0x7f020c1b0a10 0x7f020c0cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:03.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:03.534 7f02118de700 1 --2- 172.21.2.112:0/3364995385 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f020c0d1870 0x7f020c1b03f0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:03.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:03.534 7f02118de700 1 -- 172.21.2.112:0/3364995385 >> 172.21.2.112:0/3364995385 conn(0x7f020c0ce450 msgr2=0x7f020c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:03.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:03.534 7f02118de700 1 -- 172.21.2.112:0/3364995385 shutdown_connections 2021-02-10T11:43:03.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:03.534 7f02118de700 1 -- 172.21.2.112:0/3364995385 wait complete. 2021-02-10T11:43:03.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:03.549 7f02118de700 1 Processor -- start 2021-02-10T11:43:03.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:03.549 7f02118de700 1 -- start start 2021-02-10T11:43:03.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:03.549 7f02118de700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f020c0ada90 0x7f020c0ade90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:03.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:03.549 7f02118de700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f020c099d20 0x7f020c09a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:03.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:03.549 7f02118de700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f020c09a740 0x7f020c097bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:03.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:03.549 7f02118de700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f020c0997a0 con 0x7f020c09a740 2021-02-10T11:43:03.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:03.549 7f02118de700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f020c18f0c0 con 0x7f020c0ada90 2021-02-10T11:43:03.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:03.549 7f02118de700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f020c09ab40 con 0x7f020c099d20 2021-02-10T11:43:03.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:03.549 7f020bd75700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f020c09a740 0x7f020c097bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:03.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:03.549 7f020bd75700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f020c09a740 0x7f020c097bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:56740/0 (socket says 172.21.2.112:56740) 2021-02-10T11:43:03.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:03.550 7f020bd75700 1 -- 172.21.2.112:0/1719938806 learned_addr learned my addr 172.21.2.112:0/1719938806 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:43:03.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:03.550 7f020bd75700 1 -- 172.21.2.112:0/1719938806 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f020c099d20 msgr2=0x7f020c09a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:03.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:03.550 7f020bd75700 1 --2- 172.21.2.112:0/1719938806 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f020c099d20 0x7f020c09a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:03.560 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:03.550 7f020bd75700 1 -- 172.21.2.112:0/1719938806 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f020c0ada90 msgr2=0x7f020c0ade90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:03.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:03.550 7f020bd75700 1 --2- 172.21.2.112:0/1719938806 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f020c0ada90 0x7f020c0ade90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:03.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:03.550 7f020bd75700 1 -- 172.21.2.112:0/1719938806 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f020c09ad60 con 0x7f020c09a740 2021-02-10T11:43:03.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:03.550 7f020bd75700 1 --2- 172.21.2.112:0/1719938806 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f020c09a740 0x7f020c097bc0 secure :-1 s=READY pgs=2412 cs=0 l=1 rev1=1 rx=0x7f0204009940 tx=0x7f020400bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:03.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:03.550 7f01f3fff700 1 -- 172.21.2.112:0/1719938806 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0204000f90 con 0x7f020c09a740 2021-02-10T11:43:03.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:03.550 7f01f3fff700 1 -- 172.21.2.112:0/1719938806 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0204000ce0 con 0x7f020c09a740 2021-02-10T11:43:03.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:03.550 7f02118de700 1 -- 172.21.2.112:0/1719938806 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f020c09ad60 con 0x7f020c09a740 2021-02-10T11:43:03.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:03.550 7f02118de700 1 -- 172.21.2.112:0/1719938806 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f020c0984d0 con 0x7f020c09a740 2021-02-10T11:43:03.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:03.551 7f01f3fff700 1 -- 172.21.2.112:0/1719938806 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f020400c4e0 con 0x7f020c09a740 2021-02-10T11:43:03.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:03.551 7f01f3fff700 1 -- 172.21.2.112:0/1719938806 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0204001220 con 0x7f020c09a740 2021-02-10T11:43:03.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:03.551 7f01f3fff700 1 --2- 172.21.2.112:0/1719938806 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f01ec0186c0 0x7f01ec01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:03.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:03.551 7f01f3fff700 1 -- 172.21.2.112:0/1719938806 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f0204032630 con 0x7f020c09a740 2021-02-10T11:43:03.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:03.551 7f020b574700 1 --2- 172.21.2.112:0/1719938806 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f01ec0186c0 0x7f01ec01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:03.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:03.552 7f020b574700 1 --2- 172.21.2.112:0/1719938806 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f01ec0186c0 0x7f01ec01abc0 crc :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:03.581 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:03.582 7f02118de700 1 -- 172.21.2.112:0/1719938806 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f020c0ebc00 con 0x7f020c09a740 2021-02-10T11:43:03.584 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:03.584 7f01f3fff700 1 -- 172.21.2.112:0/1719938806 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0204000f90 con 0x7f020c09a740 2021-02-10T11:43:03.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:03.715 7f02118de700 1 -- 172.21.2.112:0/1719938806 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f020c0f0b10 con 0x7f020c09a740 2021-02-10T11:43:03.716 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:03.716 7f01f3fff700 1 -- 172.21.2.112:0/1719938806 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v78) v1 ==== 80+0+4513 (secure 0 0 0) 0x7f020401f8e0 con 0x7f020c09a740 2021-02-10T11:43:03.716 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:43:03.716 INFO:teuthology.orchestra.run.gibba012.stdout:{ 2021-02-10T11:43:03.716 INFO:teuthology.orchestra.run.gibba012.stdout: "auth_dump": [ 2021-02-10T11:43:03.717 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:03.717 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "mds.a", 2021-02-10T11:43:03.717 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCKvyNgPkV4AxAAEmVNaghdccP6lWd8hvQnxw==", 2021-02-10T11:43:03.717 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:43:03.717 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:43:03.718 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:43:03.718 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:43:03.718 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:43:03.718 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:43:03.718 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:03.718 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:03.718 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "mds.b", 2021-02-10T11:43:03.718 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCKvyNguxONLhAAHIpry+R89MAwdhFJyEhVCw==", 2021-02-10T11:43:03.718 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:43:03.718 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:43:03.718 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:43:03.719 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:43:03.719 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:43:03.719 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:43:03.719 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:03.719 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:03.719 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.0", 2021-02-10T11:43:03.719 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCSvyNgXcKPLhAAP/Bj2F+06DCtPCzQDNq+cQ==", 2021-02-10T11:43:03.719 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:43:03.719 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:43:03.719 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:43:03.719 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:43:03.719 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:43:03.719 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:03.719 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:03.720 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.1", 2021-02-10T11:43:03.720 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCVvyNghfbcLhAA66PrRwR2M7o6VTF5UhU/Fg==", 2021-02-10T11:43:03.720 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:43:03.720 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:43:03.720 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:43:03.720 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:43:03.720 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:43:03.720 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:03.720 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:03.720 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.2", 2021-02-10T11:43:03.720 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCYvyNgwiPqLhAA66BVkRsgaUlGpAsQ15Nf8Q==", 2021-02-10T11:43:03.720 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:43:03.720 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:43:03.721 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:43:03.721 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:43:03.721 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:43:03.721 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:03.721 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:03.721 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.3", 2021-02-10T11:43:03.721 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCbvyNgqBZrLxAAERPe/1L3SXAEzI3d9ReHHg==", 2021-02-10T11:43:03.721 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:43:03.721 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:43:03.721 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:43:03.721 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:43:03.721 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:43:03.721 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:03.722 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:03.722 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.4", 2021-02-10T11:43:03.722 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCkvyNgDQsxIhAAHmVjVYnx20N4SDmKqvbcuA==", 2021-02-10T11:43:03.722 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:43:03.722 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:43:03.722 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:43:03.722 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:43:03.722 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:43:03.722 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:03.722 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:03.722 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.5", 2021-02-10T11:43:03.722 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCnvyNg8ITFIhAADd4aKybAV7btoGPx3kCg3A==", 2021-02-10T11:43:03.722 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:43:03.723 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:43:03.723 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:43:03.723 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:43:03.723 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:43:03.723 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:03.723 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:03.723 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.6", 2021-02-10T11:43:03.723 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCqvyNgxOhDIxAAzXixEa/nlDc1xXrg0aeLwg==", 2021-02-10T11:43:03.723 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:43:03.723 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:43:03.723 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:43:03.723 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:43:03.723 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:43:03.724 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:03.724 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:03.724 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.7", 2021-02-10T11:43:03.724 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCtvyNgxSqGIxAACxJQlwP7DpiVVCZHA3zn1A==", 2021-02-10T11:43:03.724 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:43:03.724 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:43:03.724 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:43:03.724 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:43:03.724 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:43:03.724 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:03.724 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:03.724 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.0", 2021-02-10T11:43:03.724 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCLvyNgDkNiHRAAygU8Za4PBusTqCgwAYWTiQ==", 2021-02-10T11:43:03.724 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:43:03.725 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:43:03.725 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow r", 2021-02-10T11:43:03.725 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:43:03.725 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:43:03.725 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:03.725 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:03.725 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.1", 2021-02-10T11:43:03.725 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCLvyNgxmLqMxAA1MlSDJ9LH9u4f2sL7mm8/Q==", 2021-02-10T11:43:03.725 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:43:03.725 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:43:03.725 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow r", 2021-02-10T11:43:03.725 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:43:03.725 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:43:03.725 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:03.726 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:03.726 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.2", 2021-02-10T11:43:03.726 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCMvyNgp8veDhAAniSVHYPtwwTog/pjbaRQmw==", 2021-02-10T11:43:03.726 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:43:03.726 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:43:03.726 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow r", 2021-02-10T11:43:03.726 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:43:03.726 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:43:03.726 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:03.726 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:03.726 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.3", 2021-02-10T11:43:03.726 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCMvyNgQEVoJRAApHcWdm1sldIxu+heyD4HdQ==", 2021-02-10T11:43:03.727 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:43:03.727 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:43:03.727 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow r", 2021-02-10T11:43:03.727 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:43:03.727 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:43:03.727 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:03.727 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:03.727 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.admin", 2021-02-10T11:43:03.727 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCGvyNg7jjJBRAAtot+ypUZOiTo1GbY+oVhVw==", 2021-02-10T11:43:03.727 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:43:03.727 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow *", 2021-02-10T11:43:03.727 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:43:03.727 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:43:03.728 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:43:03.728 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:43:03.728 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:03.728 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:03.728 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "mgr.x", 2021-02-10T11:43:03.728 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCJvyNgTPVwKBAAjduiBE8+RSXBv82tdPecWg==", 2021-02-10T11:43:03.728 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:43:03.728 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow *", 2021-02-10T11:43:03.728 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow profile mgr", 2021-02-10T11:43:03.728 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:43:03.728 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:43:03.729 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:03.729 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:03.729 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "mgr.y", 2021-02-10T11:43:03.729 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCJvyNgQ1tTDhAASwBssDy+Rz31ujw8fNlWow==", 2021-02-10T11:43:03.729 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:43:03.729 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow *", 2021-02-10T11:43:03.729 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow profile mgr", 2021-02-10T11:43:03.729 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:43:03.729 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:43:03.729 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:43:03.729 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:43:03.730 INFO:teuthology.orchestra.run.gibba012.stdout:} 2021-02-10T11:43:03.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:03.718 7f02118de700 1 -- 172.21.2.112:0/1719938806 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f01ec0186c0 msgr2=0x7f01ec01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:03.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:03.718 7f02118de700 1 --2- 172.21.2.112:0/1719938806 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f01ec0186c0 0x7f01ec01abc0 crc :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:03.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:03.718 7f02118de700 1 -- 172.21.2.112:0/1719938806 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f020c09a740 msgr2=0x7f020c097bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:03.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:03.718 7f02118de700 1 --2- 172.21.2.112:0/1719938806 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f020c09a740 0x7f020c097bc0 secure :-1 s=READY pgs=2412 cs=0 l=1 rev1=1 rx=0x7f0204009940 tx=0x7f020400bcf0).stop 2021-02-10T11:43:03.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:03.718 7f02118de700 1 -- 172.21.2.112:0/1719938806 shutdown_connections 2021-02-10T11:43:03.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:03.718 7f02118de700 1 --2- 172.21.2.112:0/1719938806 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f01ec0186c0 0x7f01ec01abc0 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:03.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:03.718 7f02118de700 1 --2- 172.21.2.112:0/1719938806 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f020c09a740 0x7f020c097bc0 unknown :-1 s=CLOSED pgs=2412 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:03.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:03.718 7f02118de700 1 --2- 172.21.2.112:0/1719938806 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f020c099d20 0x7f020c09a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:03.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:03.718 7f02118de700 1 --2- 172.21.2.112:0/1719938806 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f020c0ada90 0x7f020c0ade90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:03.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:03.718 7f02118de700 1 -- 172.21.2.112:0/1719938806 >> 172.21.2.112:0/1719938806 conn(0x7f020c0f0550 msgr2=0x7f020c191730 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:03.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:03.718 7f02118de700 1 -- 172.21.2.112:0/1719938806 shutdown_connections 2021-02-10T11:43:03.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:03.718 7f02118de700 1 -- 172.21.2.112:0/1719938806 wait complete. 2021-02-10T11:43:03.732 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:43:04.164 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.144 7f94e768d700 1 Processor -- start 2021-02-10T11:43:04.165 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.144 7f94e768d700 1 -- start start 2021-02-10T11:43:04.165 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.144 7f94e768d700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f94e00d1870 0x7f94e01b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:04.165 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.144 7f94e768d700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94e00f0b10 con 0x7f94e01b0a10 2021-02-10T11:43:04.165 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.144 7f94e768d700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94e00d0310 con 0x7f94e00f0650 2021-02-10T11:43:04.165 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.144 7f94e768d700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94e00d0530 con 0x7f94e00d1870 2021-02-10T11:43:04.166 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.145 7f94e4ca8700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f94e01b0a10 0x7f94e00cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:56206/0 (socket says 172.21.2.112:56206) 2021-02-10T11:43:04.166 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.145 7f94e4ca8700 1 -- 172.21.2.112:0/3042633586 learned_addr learned my addr 172.21.2.112:0/3042633586 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:43:04.166 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.145 7f94d3fff700 1 -- 172.21.2.112:0/3042633586 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1545519574 0 0) 0x7f94d40010b0 con 0x7f94e01b0a10 2021-02-10T11:43:04.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.145 7f94d3fff700 1 -- 172.21.2.112:0/3042633586 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94cc002fe0 con 0x7f94e01b0a10 2021-02-10T11:43:04.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.145 7f94e54a9700 1 --2- 172.21.2.112:0/3042633586 >> v2:172.21.2.120:3300/0 conn(0x7f94e00d1870 0x7f94e01b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:04.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.145 7f94d3fff700 1 -- 172.21.2.112:0/3042633586 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3808558401 0 0) 0x7f94d40010b0 con 0x7f94e01b0a10 2021-02-10T11:43:04.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.145 7f94d3fff700 1 -- 172.21.2.112:0/3042633586 >> v2:172.21.2.120:3300/0 conn(0x7f94e00d1870 msgr2=0x7f94e01b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:04.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.145 7f94d3fff700 1 --2- 172.21.2.112:0/3042633586 >> v2:172.21.2.120:3300/0 conn(0x7f94e00d1870 0x7f94e01b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:04.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.145 7f94d3fff700 1 -- 172.21.2.112:0/3042633586 >> v1:172.21.2.120:6789/0 conn(0x7f94e00f0650 legacy=0x7f94e00cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:04.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.145 7f94d3fff700 1 -- 172.21.2.112:0/3042633586 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94e00d1c70 con 0x7f94e01b0a10 2021-02-10T11:43:04.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.145 7f94d3fff700 1 -- 172.21.2.112:0/3042633586 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f94d40015b0 con 0x7f94e01b0a10 2021-02-10T11:43:04.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.146 7f94d3fff700 1 -- 172.21.2.112:0/3042633586 >> v1:172.21.2.112:6789/0 conn(0x7f94e01b0a10 legacy=0x7f94e00cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:04.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.146 7f94d3fff700 1 --2- 172.21.2.112:0/3042633586 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f94cc003a60 0x7f94cc005f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:04.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.146 7f94d3fff700 1 --2- 172.21.2.112:0/3042633586 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f94cc006800 0x7f94cc008d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:04.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.146 7f94d3fff700 1 --2- 172.21.2.112:0/3042633586 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f94cc009620 0x7f94cc00bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:04.169 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.146 7f94d3fff700 1 -- 172.21.2.112:0/3042633586 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94cc00c310 con 0x7f94cc003a60 2021-02-10T11:43:04.169 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.146 7f94d3fff700 1 -- 172.21.2.112:0/3042633586 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94cc00c530 con 0x7f94cc009620 2021-02-10T11:43:04.169 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.146 7f94d3fff700 1 -- 172.21.2.112:0/3042633586 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94cc050760 con 0x7f94cc006800 2021-02-10T11:43:04.169 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.146 7f94e54a9700 1 --2- 172.21.2.112:0/3042633586 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f94cc003a60 0x7f94cc005f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:04.169 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.146 7f94e4ca8700 1 --2- 172.21.2.112:0/3042633586 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f94cc006800 0x7f94cc008d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:04.169 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.146 7f94e5caa700 1 --2- 172.21.2.112:0/3042633586 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f94cc009620 0x7f94cc00bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:04.170 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.146 7f94e54a9700 1 -- 172.21.2.112:0/3042633586 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f94cc006800 msgr2=0x7f94cc008d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:04.170 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.146 7f94e54a9700 1 --2- 172.21.2.112:0/3042633586 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f94cc006800 0x7f94cc008d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:04.170 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.146 7f94e54a9700 1 -- 172.21.2.112:0/3042633586 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f94cc009620 msgr2=0x7f94cc00bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:04.170 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.146 7f94e54a9700 1 --2- 172.21.2.112:0/3042633586 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f94cc009620 0x7f94cc00bb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:04.170 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.146 7f94e54a9700 1 -- 172.21.2.112:0/3042633586 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94cc0509c0 con 0x7f94cc003a60 2021-02-10T11:43:04.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.146 7f94e5caa700 1 -- 172.21.2.112:0/3042633586 reap_dead start 2021-02-10T11:43:04.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.146 7f94e54a9700 1 --2- 172.21.2.112:0/3042633586 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f94cc003a60 0x7f94cc005f60 secure :-1 s=READY pgs=2414 cs=0 l=1 rev1=1 rx=0x7f94dc00c470 tx=0x7f94dc006c20).ready entity=mon.0 client_cookie=61cc1d91b823ad1e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:04.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.146 7f94d3fff700 1 -- 172.21.2.112:0/3042633586 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f94dc0090d0 con 0x7f94cc003a60 2021-02-10T11:43:04.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.147 7f94d3fff700 1 -- 172.21.2.112:0/3042633586 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94dc009450 con 0x7f94cc003a60 2021-02-10T11:43:04.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.147 7f94d3fff700 1 -- 172.21.2.112:0/3042633586 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f94dc009970 con 0x7f94cc003a60 2021-02-10T11:43:04.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.147 7f94e768d700 1 -- 172.21.2.112:0/3042633586 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f94cc003a60 msgr2=0x7f94cc005f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:04.172 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.147 7f94e768d700 1 --2- 172.21.2.112:0/3042633586 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f94cc003a60 0x7f94cc005f60 secure :-1 s=READY pgs=2414 cs=0 l=1 rev1=1 rx=0x7f94dc00c470 tx=0x7f94dc006c20).stop 2021-02-10T11:43:04.172 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.147 7f94e768d700 1 -- 172.21.2.112:0/3042633586 shutdown_connections 2021-02-10T11:43:04.172 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.147 7f94e768d700 1 --2- 172.21.2.112:0/3042633586 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f94cc003a60 0x7f94cc005f60 unknown :-1 s=CLOSED pgs=2414 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:04.172 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.147 7f94e768d700 1 -- 172.21.2.112:0/3042633586 >> 172.21.2.112:0/3042633586 conn(0x7f94e00ce450 msgr2=0x7f94e00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:04.172 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.147 7f94e768d700 1 -- 172.21.2.112:0/3042633586 shutdown_connections 2021-02-10T11:43:04.173 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.147 7f94e768d700 1 -- 172.21.2.112:0/3042633586 wait complete. 2021-02-10T11:43:04.173 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.162 7f94e768d700 1 Processor -- start 2021-02-10T11:43:04.173 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.162 7f94e768d700 1 -- start start 2021-02-10T11:43:04.173 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.163 7f94e768d700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f94e0195650 0x7f94e00b5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:04.173 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.163 7f94e768d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f94e00b5b80 0x7f94e01a2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:04.174 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.163 7f94e768d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f94e00acfa0 0x7f94e01a3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:04.174 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.163 7f94e768d700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94e00b6010 con 0x7f94e00b5b80 2021-02-10T11:43:04.174 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.163 7f94e768d700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94e01a3a60 con 0x7f94e0195650 2021-02-10T11:43:04.174 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.163 7f94e768d700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94e01a3c40 con 0x7f94e00acfa0 2021-02-10T11:43:04.174 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.163 7f94e4ca8700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f94e00b5b80 0x7f94e01a2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:04.175 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.163 7f94e4ca8700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f94e00b5b80 0x7f94e01a2e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:56754/0 (socket says 172.21.2.112:56754) 2021-02-10T11:43:04.175 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.163 7f94e4ca8700 1 -- 172.21.2.112:0/3883861407 learned_addr learned my addr 172.21.2.112:0/3883861407 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:43:04.175 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.163 7f94e4ca8700 1 -- 172.21.2.112:0/3883861407 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f94e00acfa0 msgr2=0x7f94e01a3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:04.175 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.163 7f94e4ca8700 1 --2- 172.21.2.112:0/3883861407 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f94e00acfa0 0x7f94e01a3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:04.175 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.163 7f94e4ca8700 1 -- 172.21.2.112:0/3883861407 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f94e0195650 msgr2=0x7f94e00b5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:04.175 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.163 7f94e4ca8700 1 --2- 172.21.2.112:0/3883861407 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f94e0195650 0x7f94e00b5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:04.176 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.163 7f94e4ca8700 1 -- 172.21.2.112:0/3883861407 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94e00bcae0 con 0x7f94e00b5b80 2021-02-10T11:43:04.176 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.163 7f94e4ca8700 1 --2- 172.21.2.112:0/3883861407 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f94e00b5b80 0x7f94e01a2e20 secure :-1 s=READY pgs=2415 cs=0 l=1 rev1=1 rx=0x7f94d4001660 tx=0x7f94d400a930).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:04.176 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.164 7f94d1ffb700 1 -- 172.21.2.112:0/3883861407 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f94d400ad20 con 0x7f94e00b5b80 2021-02-10T11:43:04.176 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.164 7f94d1ffb700 1 -- 172.21.2.112:0/3883861407 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94d40016a0 con 0x7f94e00b5b80 2021-02-10T11:43:04.177 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.164 7f94d1ffb700 1 -- 172.21.2.112:0/3883861407 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f94d4004330 con 0x7f94e00b5b80 2021-02-10T11:43:04.177 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.164 7f94e768d700 1 -- 172.21.2.112:0/3883861407 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94e00bcae0 con 0x7f94e00b5b80 2021-02-10T11:43:04.177 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.164 7f94e768d700 1 -- 172.21.2.112:0/3883861407 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94e00bd070 con 0x7f94e00b5b80 2021-02-10T11:43:04.177 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.164 7f94d1ffb700 1 -- 172.21.2.112:0/3883861407 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f94d4004140 con 0x7f94e00b5b80 2021-02-10T11:43:04.177 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.164 7f94d1ffb700 1 --2- 172.21.2.112:0/3883861407 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f94c00186c0 0x7f94c001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:04.178 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.164 7f94d1ffb700 1 -- 172.21.2.112:0/3883861407 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f94d4032080 con 0x7f94e00b5b80 2021-02-10T11:43:04.178 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.165 7f94e54a9700 1 --2- 172.21.2.112:0/3883861407 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f94c00186c0 0x7f94c001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:04.179 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.166 7f94e54a9700 1 --2- 172.21.2.112:0/3883861407 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f94c00186c0 0x7f94c001abc0 crc :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:04.195 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.194 7f94e768d700 1 -- 172.21.2.112:0/3883861407 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94e01a3c40 con 0x7f94e00b5b80 2021-02-10T11:43:04.197 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.197 7f94d1ffb700 1 -- 172.21.2.112:0/3883861407 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f94d4025cd0 con 0x7f94e00b5b80 2021-02-10T11:43:04.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.322 7f94e768d700 1 -- 172.21.2.112:0/3883861407 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f94e01a3c40 con 0x7f94e00b5b80 2021-02-10T11:43:04.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.323 7f94d1ffb700 1 -- 172.21.2.112:0/3883861407 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v129) v1 ==== 74+0+11154 (secure 0 0 0) 0x7f94d4025cd0 con 0x7f94e00b5b80 2021-02-10T11:43:04.323 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:43:04.323 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":129,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:43:03.064453","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":27,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:43:04.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.324 7f94e768d700 1 -- 172.21.2.112:0/3883861407 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f94c00186c0 msgr2=0x7f94c001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:04.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.324 7f94e768d700 1 --2- 172.21.2.112:0/3883861407 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f94c00186c0 0x7f94c001abc0 crc :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:04.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.324 7f94e768d700 1 -- 172.21.2.112:0/3883861407 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f94e00b5b80 msgr2=0x7f94e01a2e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:04.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.324 7f94e768d700 1 --2- 172.21.2.112:0/3883861407 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f94e00b5b80 0x7f94e01a2e20 secure :-1 s=READY pgs=2415 cs=0 l=1 rev1=1 rx=0x7f94d4001660 tx=0x7f94d400a930).stop 2021-02-10T11:43:04.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.325 7f94e768d700 1 -- 172.21.2.112:0/3883861407 shutdown_connections 2021-02-10T11:43:04.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.325 7f94e768d700 1 --2- 172.21.2.112:0/3883861407 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f94c00186c0 0x7f94c001abc0 unknown :-1 s=CLOSED pgs=1101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:04.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.325 7f94e768d700 1 --2- 172.21.2.112:0/3883861407 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f94e00acfa0 0x7f94e01a3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:04.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.325 7f94e768d700 1 --2- 172.21.2.112:0/3883861407 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f94e00b5b80 0x7f94e01a2e20 unknown :-1 s=CLOSED pgs=2415 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:04.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.325 7f94e768d700 1 --2- 172.21.2.112:0/3883861407 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f94e0195650 0x7f94e00b5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:04.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.325 7f94e768d700 1 -- 172.21.2.112:0/3883861407 >> 172.21.2.112:0/3883861407 conn(0x7f94e00b3040 msgr2=0x7f94e00c3250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:04.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.325 7f94e768d700 1 -- 172.21.2.112:0/3883861407 shutdown_connections 2021-02-10T11:43:04.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.325 7f94e768d700 1 -- 172.21.2.112:0/3883861407 wait complete. 2021-02-10T11:43:04.334 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:13:57.508758', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2021-02-10T11:43:04.334 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:43:04.765 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.749 7fee570f7700 1 Processor -- start 2021-02-10T11:43:04.765 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.749 7fee570f7700 1 -- start start 2021-02-10T11:43:04.766 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.749 7fee570f7700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fee500d1870 0x7fee501b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:04.766 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.749 7fee570f7700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee500fe1c0 con 0x7fee501b0ae0 2021-02-10T11:43:04.766 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.749 7fee570f7700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee50109a60 con 0x7fee500cff70 2021-02-10T11:43:04.766 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.749 7fee570f7700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee50088070 con 0x7fee500d1870 2021-02-10T11:43:04.766 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.750 7fee4ffff700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fee501b0ae0 0x7fee50108940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:56224/0 (socket says 172.21.2.112:56224) 2021-02-10T11:43:04.766 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.750 7fee4ffff700 1 -- 172.21.2.112:0/1150731692 learned_addr learned my addr 172.21.2.112:0/1150731692 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:43:04.766 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.750 7fee54f13700 1 --2- 172.21.2.112:0/1150731692 >> v2:172.21.2.120:3300/0 conn(0x7fee500d1870 0x7fee501b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:04.766 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.750 7fee4f7fe700 1 -- 172.21.2.112:0/1150731692 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3886050627 0 0) 0x7fee40001500 con 0x7fee501b0ae0 2021-02-10T11:43:04.766 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.750 7fee4f7fe700 1 -- 172.21.2.112:0/1150731692 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee3c002fe0 con 0x7fee501b0ae0 2021-02-10T11:43:04.766 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.750 7fee54f13700 1 -- 172.21.2.112:0/1150731692 >> v1:172.21.2.120:6789/0 conn(0x7fee500cff70 legacy=0x7fee50109130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:04.766 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.750 7fee54f13700 1 -- 172.21.2.112:0/1150731692 >> v1:172.21.2.112:6789/0 conn(0x7fee501b0ae0 legacy=0x7fee50108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:04.766 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.750 7fee54f13700 1 -- 172.21.2.112:0/1150731692 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee50088260 con 0x7fee500d1870 2021-02-10T11:43:04.766 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.751 7fee54f13700 1 --2- 172.21.2.112:0/1150731692 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fee500d1870 0x7fee501b04c0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 rx=0x7fee38006650 tx=0x7fee38009a00).ready entity=mon.1 client_cookie=7c75522a82f747b1 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:04.767 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.751 7fee4f7fe700 1 -- 172.21.2.112:0/1150731692 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fee380009b0 con 0x7fee500d1870 2021-02-10T11:43:04.767 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.751 7fee4f7fe700 1 -- 172.21.2.112:0/1150731692 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee3800c0c0 con 0x7fee500d1870 2021-02-10T11:43:04.767 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.751 7fee4f7fe700 1 -- 172.21.2.112:0/1150731692 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fee3800c5a0 con 0x7fee500d1870 2021-02-10T11:43:04.767 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.751 7fee570f7700 1 -- 172.21.2.112:0/1150731692 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fee500d1870 msgr2=0x7fee501b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:04.767 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.751 7fee570f7700 1 --2- 172.21.2.112:0/1150731692 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fee500d1870 0x7fee501b04c0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 rx=0x7fee38006650 tx=0x7fee38009a00).stop 2021-02-10T11:43:04.767 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.751 7fee570f7700 1 -- 172.21.2.112:0/1150731692 shutdown_connections 2021-02-10T11:43:04.767 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.751 7fee570f7700 1 --2- 172.21.2.112:0/1150731692 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fee500d1870 0x7fee501b04c0 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:04.767 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.752 7fee570f7700 1 -- 172.21.2.112:0/1150731692 >> 172.21.2.112:0/1150731692 conn(0x7fee500ce420 msgr2=0x7fee500d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:04.767 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.752 7fee570f7700 1 -- 172.21.2.112:0/1150731692 shutdown_connections 2021-02-10T11:43:04.767 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.752 7fee570f7700 1 -- 172.21.2.112:0/1150731692 wait complete. 2021-02-10T11:43:04.768 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.766 7fee570f7700 1 Processor -- start 2021-02-10T11:43:04.769 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.766 7fee570f7700 1 -- start start 2021-02-10T11:43:04.769 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.766 7fee570f7700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fee501a0020 0x7fee501a0420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:04.769 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.766 7fee570f7700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fee50091330 0x7fee501917e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:04.769 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.766 7fee570f7700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fee50191e00 0x7fee50092d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:04.769 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.766 7fee570f7700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee50192200 con 0x7fee50191e00 2021-02-10T11:43:04.769 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.766 7fee570f7700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee50092790 con 0x7fee501a0020 2021-02-10T11:43:04.769 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.766 7fee570f7700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee50096180 con 0x7fee50091330 2021-02-10T11:43:04.769 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.767 7fee55714700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fee50191e00 0x7fee50092d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:04.769 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.767 7fee55714700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fee50191e00 0x7fee50092d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:56768/0 (socket says 172.21.2.112:56768) 2021-02-10T11:43:04.769 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.767 7fee55714700 1 -- 172.21.2.112:0/3493697716 learned_addr learned my addr 172.21.2.112:0/3493697716 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:43:04.769 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.767 7fee55714700 1 -- 172.21.2.112:0/3493697716 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fee50091330 msgr2=0x7fee501917e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:04.769 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.767 7fee55714700 1 --2- 172.21.2.112:0/3493697716 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fee50091330 0x7fee501917e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:04.769 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.767 7fee55714700 1 -- 172.21.2.112:0/3493697716 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fee501a0020 msgr2=0x7fee501a0420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:04.770 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.767 7fee55714700 1 --2- 172.21.2.112:0/3493697716 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fee501a0020 0x7fee501a0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:04.770 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.767 7fee55714700 1 -- 172.21.2.112:0/3493697716 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee50093330 con 0x7fee50191e00 2021-02-10T11:43:04.770 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.767 7fee55714700 1 --2- 172.21.2.112:0/3493697716 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fee50191e00 0x7fee50092d10 secure :-1 s=READY pgs=2417 cs=0 l=1 rev1=1 rx=0x7fee48006f20 tx=0x7fee4800b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:04.770 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.767 7fee4d7fa700 1 -- 172.21.2.112:0/3493697716 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fee48006bf0 con 0x7fee50191e00 2021-02-10T11:43:04.770 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.767 7fee4d7fa700 1 -- 172.21.2.112:0/3493697716 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee4800bd80 con 0x7fee50191e00 2021-02-10T11:43:04.770 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.767 7fee4d7fa700 1 -- 172.21.2.112:0/3493697716 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fee48005540 con 0x7fee50191e00 2021-02-10T11:43:04.770 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.767 7fee570f7700 1 -- 172.21.2.112:0/3493697716 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee50093330 con 0x7fee50191e00 2021-02-10T11:43:04.770 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.767 7fee570f7700 1 -- 172.21.2.112:0/3493697716 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee50093330 con 0x7fee50191e00 2021-02-10T11:43:04.770 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.768 7fee4d7fa700 1 -- 172.21.2.112:0/3493697716 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fee4800a000 con 0x7fee50191e00 2021-02-10T11:43:04.770 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.768 7fee4d7fa700 1 --2- 172.21.2.112:0/3493697716 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fee340186c0 0x7fee3401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:04.770 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.768 7fee4d7fa700 1 -- 172.21.2.112:0/3493697716 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 4753+0+0 (secure 0 0 0) 0x7fee480321f0 con 0x7fee50191e00 2021-02-10T11:43:04.770 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.768 7fee54f13700 1 --2- 172.21.2.112:0/3493697716 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fee340186c0 0x7fee3401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:04.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.769 7fee54f13700 1 --2- 172.21.2.112:0/3493697716 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fee340186c0 0x7fee3401abc0 crc :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:04.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.799 7fee570f7700 1 -- 172.21.2.112:0/3493697716 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee50192200 con 0x7fee50191e00 2021-02-10T11:43:04.801 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.802 7fee4d7fa700 1 -- 172.21.2.112:0/3493697716 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fee48026f20 con 0x7fee50191e00 2021-02-10T11:43:04.930 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.931 7fee570f7700 1 -- 172.21.2.112:0/3493697716 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7fee50192200 con 0x7fee50191e00 2021-02-10T11:43:04.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.931 7fee4d7fa700 1 -- 172.21.2.112:0/3493697716 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v129) v1 ==== 92+0+10 (secure 0 0 0) 0x7fee48026f20 con 0x7fee50191e00 2021-02-10T11:43:04.931 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:43:04.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.933 7fee570f7700 1 -- 172.21.2.112:0/3493697716 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fee340186c0 msgr2=0x7fee3401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:04.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.933 7fee570f7700 1 --2- 172.21.2.112:0/3493697716 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fee340186c0 0x7fee3401abc0 crc :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:04.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.933 7fee570f7700 1 -- 172.21.2.112:0/3493697716 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fee50191e00 msgr2=0x7fee50092d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:04.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.933 7fee570f7700 1 --2- 172.21.2.112:0/3493697716 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fee50191e00 0x7fee50092d10 secure :-1 s=READY pgs=2417 cs=0 l=1 rev1=1 rx=0x7fee48006f20 tx=0x7fee4800b990).stop 2021-02-10T11:43:04.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.933 7fee570f7700 1 -- 172.21.2.112:0/3493697716 shutdown_connections 2021-02-10T11:43:04.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.933 7fee570f7700 1 --2- 172.21.2.112:0/3493697716 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fee340186c0 0x7fee3401abc0 unknown :-1 s=CLOSED pgs=1102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:04.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.933 7fee570f7700 1 --2- 172.21.2.112:0/3493697716 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fee50191e00 0x7fee50092d10 unknown :-1 s=CLOSED pgs=2417 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:04.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.933 7fee570f7700 1 --2- 172.21.2.112:0/3493697716 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fee50091330 0x7fee501917e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:04.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.933 7fee570f7700 1 --2- 172.21.2.112:0/3493697716 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fee501a0020 0x7fee501a0420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:04.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.933 7fee570f7700 1 -- 172.21.2.112:0/3493697716 >> 172.21.2.112:0/3493697716 conn(0x7fee500bc0c0 msgr2=0x7fee50082ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:04.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.933 7fee570f7700 1 -- 172.21.2.112:0/3493697716 shutdown_connections 2021-02-10T11:43:04.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:04.933 7fee570f7700 1 -- 172.21.2.112:0/3493697716 wait complete. 2021-02-10T11:43:04.941 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --format=json-pretty osd lspools 2021-02-10T11:43:05.369 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.352 7fa6a607d700 1 Processor -- start 2021-02-10T11:43:05.369 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.352 7fa6a607d700 1 -- start start 2021-02-10T11:43:05.370 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.353 7fa6a607d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fa6a01b0470 0x7fa6a01b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:05.370 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.353 7fa6a607d700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fa6a00ef610 0x7fa6a00efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:05.370 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.353 7fa6a607d700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6a00f0b50 con 0x7fa6a00f0030 2021-02-10T11:43:05.370 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.353 7fa6a607d700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6a00e52e0 con 0x7fa6a01b0470 2021-02-10T11:43:05.370 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.353 7fa6a607d700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6a00e5490 con 0x7fa6a00ef610 2021-02-10T11:43:05.370 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.353 7fa69f7fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fa6a01b0470 0x7fa6a01b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:05.370 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.353 7fa69f7fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fa6a01b0470 0x7fa6a01b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:56776/0 (socket says 172.21.2.112:56776) 2021-02-10T11:43:05.370 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.353 7fa69f7fe700 1 -- 172.21.2.112:0/3561630775 learned_addr learned my addr 172.21.2.112:0/3561630775 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:43:05.370 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.354 7fa69f7fe700 1 -- 172.21.2.112:0/3561630775 >> v2:172.21.2.120:3300/0 conn(0x7fa6a00ef610 msgr2=0x7fa6a00efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:05.370 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.354 7fa69f7fe700 1 --2- 172.21.2.112:0/3561630775 >> v2:172.21.2.120:3300/0 conn(0x7fa6a00ef610 0x7fa6a00efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:05.370 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.354 7fa69f7fe700 1 -- 172.21.2.112:0/3561630775 >> v1:172.21.2.120:6789/0 conn(0x7fa6a00f0030 legacy=0x7fa6a00f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:05.371 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.354 7fa69f7fe700 1 -- 172.21.2.112:0/3561630775 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6a00f0430 con 0x7fa6a01b0470 2021-02-10T11:43:05.371 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.354 7fa69f7fe700 1 --2- 172.21.2.112:0/3561630775 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa6a01b0470 0x7fa6a01b0870 secure :-1 s=READY pgs=2418 cs=0 l=1 rev1=1 rx=0x7fa688006700 tx=0x7fa688009b80).ready entity=mon.0 client_cookie=5fc23ed91450e032 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:05.371 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.354 7fa69e7fc700 1 -- 172.21.2.112:0/3561630775 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa68800c0c0 con 0x7fa6a01b0470 2021-02-10T11:43:05.371 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.354 7fa69e7fc700 1 -- 172.21.2.112:0/3561630775 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa68800c340 con 0x7fa6a01b0470 2021-02-10T11:43:05.371 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.354 7fa69e7fc700 1 -- 172.21.2.112:0/3561630775 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa68800c820 con 0x7fa6a01b0470 2021-02-10T11:43:05.371 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.354 7fa6a607d700 1 -- 172.21.2.112:0/3561630775 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa6a01b0470 msgr2=0x7fa6a01b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:05.371 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.354 7fa6a607d700 1 --2- 172.21.2.112:0/3561630775 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa6a01b0470 0x7fa6a01b0870 secure :-1 s=READY pgs=2418 cs=0 l=1 rev1=1 rx=0x7fa688006700 tx=0x7fa688009b80).stop 2021-02-10T11:43:05.371 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.355 7fa6a607d700 1 -- 172.21.2.112:0/3561630775 shutdown_connections 2021-02-10T11:43:05.371 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.355 7fa6a607d700 1 --2- 172.21.2.112:0/3561630775 >> v2:172.21.2.120:3300/0 conn(0x7fa6a00ef610 0x7fa6a00efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:05.371 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.355 7fa6a607d700 1 --2- 172.21.2.112:0/3561630775 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa6a01b0470 0x7fa6a01b0870 unknown :-1 s=CLOSED pgs=2418 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:05.371 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.355 7fa6a607d700 1 -- 172.21.2.112:0/3561630775 >> 172.21.2.112:0/3561630775 conn(0x7fa6a00ce3e0 msgr2=0x7fa6a00e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:05.371 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.355 7fa6a607d700 1 -- 172.21.2.112:0/3561630775 shutdown_connections 2021-02-10T11:43:05.371 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.355 7fa6a607d700 1 -- 172.21.2.112:0/3561630775 wait complete. 2021-02-10T11:43:05.372 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.370 7fa6a607d700 1 Processor -- start 2021-02-10T11:43:05.372 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.370 7fa6a607d700 1 -- start start 2021-02-10T11:43:05.373 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.370 7fa6a607d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa6a01acc20 0x7fa6a01ad020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:05.373 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.370 7fa6a607d700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa6a01ad640 0x7fa6a01ada40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:05.373 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.370 7fa6a607d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa6a0188110 0x7fa6a0188510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:05.373 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.370 7fa6a607d700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6a01ac720 con 0x7fa6a0188110 2021-02-10T11:43:05.373 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.370 7fa6a607d700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6a00b1b10 con 0x7fa6a01ad640 2021-02-10T11:43:05.374 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.370 7fa6a607d700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6a0188b30 con 0x7fa6a01acc20 2021-02-10T11:43:05.374 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.370 7fa69ffff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa6a0188110 0x7fa6a0188510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:05.374 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.370 7fa69ffff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa6a0188110 0x7fa6a0188510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:56784/0 (socket says 172.21.2.112:56784) 2021-02-10T11:43:05.374 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.370 7fa69ffff700 1 -- 172.21.2.112:0/2087273262 learned_addr learned my addr 172.21.2.112:0/2087273262 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:43:05.374 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.371 7fa69ffff700 1 -- 172.21.2.112:0/2087273262 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa6a01acc20 msgr2=0x7fa6a01ad020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:05.374 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.371 7fa69ffff700 1 --2- 172.21.2.112:0/2087273262 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa6a01acc20 0x7fa6a01ad020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:05.374 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.371 7fa69ffff700 1 -- 172.21.2.112:0/2087273262 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa6a01ad640 msgr2=0x7fa6a01ada40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:05.374 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.371 7fa69ffff700 1 --2- 172.21.2.112:0/2087273262 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa6a01ad640 0x7fa6a01ada40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:05.374 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.371 7fa69ffff700 1 -- 172.21.2.112:0/2087273262 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6a0188d50 con 0x7fa6a0188110 2021-02-10T11:43:05.375 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.371 7fa69ffff700 1 --2- 172.21.2.112:0/2087273262 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa6a0188110 0x7fa6a0188510 secure :-1 s=READY pgs=2419 cs=0 l=1 rev1=1 rx=0x7fa69800b8e0 tx=0x7fa69800bc80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:05.375 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.371 7fa697fff700 1 -- 172.21.2.112:0/2087273262 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa698000f90 con 0x7fa6a0188110 2021-02-10T11:43:05.375 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.371 7fa697fff700 1 -- 172.21.2.112:0/2087273262 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa69800c2f0 con 0x7fa6a0188110 2021-02-10T11:43:05.375 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.371 7fa697fff700 1 -- 172.21.2.112:0/2087273262 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa69800c810 con 0x7fa6a0188110 2021-02-10T11:43:05.375 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.371 7fa6a607d700 1 -- 172.21.2.112:0/2087273262 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6a0188d50 con 0x7fa6a0188110 2021-02-10T11:43:05.375 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.371 7fa6a607d700 1 -- 172.21.2.112:0/2087273262 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6a01892e0 con 0x7fa6a0188110 2021-02-10T11:43:05.375 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.372 7fa697fff700 1 -- 172.21.2.112:0/2087273262 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa698001220 con 0x7fa6a0188110 2021-02-10T11:43:05.375 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.372 7fa697fff700 1 --2- 172.21.2.112:0/2087273262 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa6840186c0 0x7fa68401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:05.375 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.372 7fa697fff700 1 -- 172.21.2.112:0/2087273262 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 4753+0+0 (secure 0 0 0) 0x7fa698032630 con 0x7fa6a0188110 2021-02-10T11:43:05.375 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.372 7fa69f7fe700 1 --2- 172.21.2.112:0/2087273262 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa6840186c0 0x7fa68401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:05.375 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.373 7fa69f7fe700 1 --2- 172.21.2.112:0/2087273262 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa6840186c0 0x7fa68401abc0 crc :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:05.403 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.403 7fa6a607d700 1 -- 172.21.2.112:0/2087273262 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6a00eddf0 con 0x7fa6a0188110 2021-02-10T11:43:05.405 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.406 7fa697fff700 1 -- 172.21.2.112:0/2087273262 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa698023ef0 con 0x7fa6a0188110 2021-02-10T11:43:05.530 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.530 7fa6a607d700 1 -- 172.21.2.112:0/2087273262 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fa6a00e52e0 con 0x7fa6a0188110 2021-02-10T11:43:05.531 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.530 7fa697fff700 1 -- 172.21.2.112:0/2087273262 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v129) v1 ==== 84+0+64 (secure 0 0 0) 0x7fa698023ef0 con 0x7fa6a0188110 2021-02-10T11:43:05.531 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:43:05.531 INFO:teuthology.orchestra.run.gibba012.stdout:[ 2021-02-10T11:43:05.531 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:05.531 INFO:teuthology.orchestra.run.gibba012.stdout: "poolnum": 1, 2021-02-10T11:43:05.531 INFO:teuthology.orchestra.run.gibba012.stdout: "poolname": "rbd" 2021-02-10T11:43:05.531 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:43:05.532 INFO:teuthology.orchestra.run.gibba012.stdout:] 2021-02-10T11:43:05.532 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.532 7fa6a607d700 1 -- 172.21.2.112:0/2087273262 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa6840186c0 msgr2=0x7fa68401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:05.532 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.532 7fa6a607d700 1 --2- 172.21.2.112:0/2087273262 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa6840186c0 0x7fa68401abc0 crc :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:05.532 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.532 7fa6a607d700 1 -- 172.21.2.112:0/2087273262 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa6a0188110 msgr2=0x7fa6a0188510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:05.533 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.532 7fa6a607d700 1 --2- 172.21.2.112:0/2087273262 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa6a0188110 0x7fa6a0188510 secure :-1 s=READY pgs=2419 cs=0 l=1 rev1=1 rx=0x7fa69800b8e0 tx=0x7fa69800bc80).stop 2021-02-10T11:43:05.533 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.532 7fa6a607d700 1 -- 172.21.2.112:0/2087273262 shutdown_connections 2021-02-10T11:43:05.533 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.532 7fa6a607d700 1 --2- 172.21.2.112:0/2087273262 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa6840186c0 0x7fa68401abc0 unknown :-1 s=CLOSED pgs=1103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:05.533 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.532 7fa6a607d700 1 --2- 172.21.2.112:0/2087273262 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa6a01ad640 0x7fa6a01ada40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:05.533 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.532 7fa6a607d700 1 --2- 172.21.2.112:0/2087273262 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa6a0188110 0x7fa6a0188510 unknown :-1 s=CLOSED pgs=2419 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:05.533 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.532 7fa6a607d700 1 --2- 172.21.2.112:0/2087273262 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa6a01acc20 0x7fa6a01ad020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:05.533 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.532 7fa6a607d700 1 -- 172.21.2.112:0/2087273262 >> 172.21.2.112:0/2087273262 conn(0x7fa6a019f7a0 msgr2=0x7fa6a01922a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:05.533 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.533 7fa6a607d700 1 -- 172.21.2.112:0/2087273262 shutdown_connections 2021-02-10T11:43:05.533 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.533 7fa6a607d700 1 -- 172.21.2.112:0/2087273262 wait complete. 2021-02-10T11:43:05.541 INFO:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2021-02-10T11:43:05.541 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_metadata 8 2021-02-10T11:43:05.972 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.956 7efcae15a700 1 Processor -- start 2021-02-10T11:43:05.972 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.956 7efcae15a700 1 -- start start 2021-02-10T11:43:05.972 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.956 7efcae15a700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7efca80d1870 0x7efca81b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:05.972 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.956 7efcae15a700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7efca80cff70 0x7efca8109130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:05.972 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.956 7efcae15a700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efca80fe1c0 con 0x7efca81b0ae0 2021-02-10T11:43:05.973 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.956 7efcae15a700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efca8109ad0 con 0x7efca80cff70 2021-02-10T11:43:05.973 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.956 7efcae15a700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efca8109750 con 0x7efca80d1870 2021-02-10T11:43:05.973 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.956 7efca7fff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7efca80cff70 0x7efca8109130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:05.973 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.956 7efca7fff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7efca80cff70 0x7efca8109130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:56788/0 (socket says 172.21.2.112:56788) 2021-02-10T11:43:05.973 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.957 7efca7fff700 1 -- 172.21.2.112:0/3818120788 learned_addr learned my addr 172.21.2.112:0/3818120788 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:43:05.973 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.957 7efca7fff700 1 -- 172.21.2.112:0/3818120788 >> v2:172.21.2.120:3300/0 conn(0x7efca80d1870 msgr2=0x7efca81b04c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:43:05.973 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.957 7efca7fff700 1 --2- 172.21.2.112:0/3818120788 >> v2:172.21.2.120:3300/0 conn(0x7efca80d1870 0x7efca81b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:05.973 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.957 7efca7fff700 1 -- 172.21.2.112:0/3818120788 >> v1:172.21.2.112:6789/0 conn(0x7efca81b0ae0 legacy=0x7efca8108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:05.973 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.957 7efca7fff700 1 -- 172.21.2.112:0/3818120788 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efca8088070 con 0x7efca80cff70 2021-02-10T11:43:05.973 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.957 7efca67fc700 1 -- 172.21.2.112:0/3818120788 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3010058616 0 0) 0x7efc98000ff0 con 0x7efca81b0ae0 2021-02-10T11:43:05.973 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.957 7efca7fff700 1 --2- 172.21.2.112:0/3818120788 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efca80cff70 0x7efca8109130 secure :-1 s=READY pgs=2421 cs=0 l=1 rev1=1 rx=0x7efca0006f00 tx=0x7efca0009f10).ready entity=mon.0 client_cookie=f5c11b655adaaa0f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:05.973 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.957 7efca67fc700 1 -- 172.21.2.112:0/3818120788 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efca000b3b0 con 0x7efca80cff70 2021-02-10T11:43:05.973 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.957 7efca67fc700 1 -- 172.21.2.112:0/3818120788 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efca000b630 con 0x7efca80cff70 2021-02-10T11:43:05.974 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.957 7efca67fc700 1 -- 172.21.2.112:0/3818120788 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efca000bb50 con 0x7efca80cff70 2021-02-10T11:43:05.974 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.957 7efcae15a700 1 -- 172.21.2.112:0/3818120788 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efca80cff70 msgr2=0x7efca8109130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:05.974 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.957 7efcae15a700 1 --2- 172.21.2.112:0/3818120788 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efca80cff70 0x7efca8109130 secure :-1 s=READY pgs=2421 cs=0 l=1 rev1=1 rx=0x7efca0006f00 tx=0x7efca0009f10).stop 2021-02-10T11:43:05.974 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.958 7efcae15a700 1 -- 172.21.2.112:0/3818120788 shutdown_connections 2021-02-10T11:43:05.974 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.958 7efcae15a700 1 --2- 172.21.2.112:0/3818120788 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efca80cff70 0x7efca8109130 unknown :-1 s=CLOSED pgs=2421 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:05.974 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.958 7efcae15a700 1 --2- 172.21.2.112:0/3818120788 >> v2:172.21.2.120:3300/0 conn(0x7efca80d1870 0x7efca81b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:05.974 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.958 7efcae15a700 1 -- 172.21.2.112:0/3818120788 >> 172.21.2.112:0/3818120788 conn(0x7efca80ce420 msgr2=0x7efca80d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:05.974 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.958 7efcae15a700 1 -- 172.21.2.112:0/3818120788 shutdown_connections 2021-02-10T11:43:05.974 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.958 7efcae15a700 1 -- 172.21.2.112:0/3818120788 wait complete. 2021-02-10T11:43:05.976 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.973 7efcae15a700 1 Processor -- start 2021-02-10T11:43:05.976 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.973 7efcae15a700 1 -- start start 2021-02-10T11:43:05.976 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.973 7efcae15a700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7efca80c2370 0x7efca818f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:05.976 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.973 7efcae15a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efca8084cb0 0x7efca81a7470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:05.976 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.973 7efcae15a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efca81a7a90 0x7efca8085700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:05.976 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.973 7efcae15a700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efca80be840 con 0x7efca81a7a90 2021-02-10T11:43:05.976 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.973 7efcae15a700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efca80c29a0 con 0x7efca80c2370 2021-02-10T11:43:05.976 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.973 7efcae15a700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efca81a7e90 con 0x7efca8084cb0 2021-02-10T11:43:05.976 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.973 7efca7fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efca81a7a90 0x7efca8085700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:05.976 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.973 7efca7fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efca81a7a90 0x7efca8085700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:56790/0 (socket says 172.21.2.112:56790) 2021-02-10T11:43:05.976 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.973 7efca7fff700 1 -- 172.21.2.112:0/1471205650 learned_addr learned my addr 172.21.2.112:0/1471205650 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:43:05.977 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.974 7efca7fff700 1 -- 172.21.2.112:0/1471205650 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efca8084cb0 msgr2=0x7efca81a7470 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:43:05.977 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.974 7efca7fff700 1 --2- 172.21.2.112:0/1471205650 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efca8084cb0 0x7efca81a7470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:05.977 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.974 7efca7fff700 1 -- 172.21.2.112:0/1471205650 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7efca80c2370 msgr2=0x7efca818f870 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:43:05.977 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.974 7efca7fff700 1 --2- 172.21.2.112:0/1471205650 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7efca80c2370 0x7efca818f870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:05.977 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.974 7efca7fff700 1 -- 172.21.2.112:0/1471205650 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efca81a80b0 con 0x7efca81a7a90 2021-02-10T11:43:05.977 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.974 7efca7fff700 1 --2- 172.21.2.112:0/1471205650 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efca81a7a90 0x7efca8085700 secure :-1 s=READY pgs=2422 cs=0 l=1 rev1=1 rx=0x7efca0009390 tx=0x7efca000bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:05.977 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.974 7efc9ffff700 1 -- 172.21.2.112:0/1471205650 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efca0006960 con 0x7efca81a7a90 2021-02-10T11:43:05.977 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.974 7efc9ffff700 1 -- 172.21.2.112:0/1471205650 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efca0003030 con 0x7efca81a7a90 2021-02-10T11:43:05.977 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.974 7efc9ffff700 1 -- 172.21.2.112:0/1471205650 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efca0001480 con 0x7efca81a7a90 2021-02-10T11:43:05.977 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.974 7efcae15a700 1 -- 172.21.2.112:0/1471205650 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efca81a80b0 con 0x7efca81a7a90 2021-02-10T11:43:05.977 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.974 7efcae15a700 1 -- 172.21.2.112:0/1471205650 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efca81a80b0 con 0x7efca81a7a90 2021-02-10T11:43:05.977 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.975 7efc9ffff700 1 -- 172.21.2.112:0/1471205650 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7efca0001270 con 0x7efca81a7a90 2021-02-10T11:43:05.977 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.975 7efc9ffff700 1 --2- 172.21.2.112:0/1471205650 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7efc8c0186c0 0x7efc8c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:05.978 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.975 7efc9ffff700 1 -- 172.21.2.112:0/1471205650 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 4753+0+0 (secure 0 0 0) 0x7efca0032800 con 0x7efca81a7a90 2021-02-10T11:43:05.978 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.975 7efca77fe700 1 --2- 172.21.2.112:0/1471205650 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7efc8c0186c0 0x7efc8c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:05.978 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:05.976 7efca77fe700 1 --2- 172.21.2.112:0/1471205650 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7efc8c0186c0 0x7efc8c01abc0 crc :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:06.006 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:06.006 7efcae15a700 1 -- 172.21.2.112:0/1471205650 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efca80be840 con 0x7efca81a7a90 2021-02-10T11:43:06.008 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:06.009 7efc9ffff700 1 -- 172.21.2.112:0/1471205650 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7efca0000cc0 con 0x7efca81a7a90 2021-02-10T11:43:06.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:06.138 7efcae15a700 1 -- 172.21.2.112:0/1471205650 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"} v 0) v1 -- 0x7efca80be840 con 0x7efca81a7a90 2021-02-10T11:43:06.550 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:06.550 7efc9ffff700 1 -- 172.21.2.112:0/1471205650 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"}]=0 pool 'cephfs_metadata' created v130) v1 ==== 133+0+0 (secure 0 0 0) 0x7efca0003030 con 0x7efca81a7a90 2021-02-10T11:43:06.553 INFO:teuthology.orchestra.run.gibba012.stderr:pool 'cephfs_metadata' created 2021-02-10T11:43:06.555 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:06.555 7efcae15a700 1 -- 172.21.2.112:0/1471205650 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7efc8c0186c0 msgr2=0x7efc8c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:06.555 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:06.555 7efcae15a700 1 --2- 172.21.2.112:0/1471205650 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7efc8c0186c0 0x7efc8c01abc0 crc :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:06.555 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:06.555 7efcae15a700 1 -- 172.21.2.112:0/1471205650 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efca81a7a90 msgr2=0x7efca8085700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:06.555 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:06.555 7efcae15a700 1 --2- 172.21.2.112:0/1471205650 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efca81a7a90 0x7efca8085700 secure :-1 s=READY pgs=2422 cs=0 l=1 rev1=1 rx=0x7efca0009390 tx=0x7efca000bd40).stop 2021-02-10T11:43:06.555 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:06.555 7efcae15a700 1 -- 172.21.2.112:0/1471205650 shutdown_connections 2021-02-10T11:43:06.555 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:06.555 7efcae15a700 1 --2- 172.21.2.112:0/1471205650 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7efc8c0186c0 0x7efc8c01abc0 unknown :-1 s=CLOSED pgs=1104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:06.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:06.555 7efcae15a700 1 --2- 172.21.2.112:0/1471205650 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efca81a7a90 0x7efca8085700 unknown :-1 s=CLOSED pgs=2422 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:06.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:06.555 7efcae15a700 1 --2- 172.21.2.112:0/1471205650 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efca8084cb0 0x7efca81a7470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:06.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:06.555 7efcae15a700 1 --2- 172.21.2.112:0/1471205650 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7efca80c2370 0x7efca818f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:06.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:06.555 7efcae15a700 1 -- 172.21.2.112:0/1471205650 >> 172.21.2.112:0/1471205650 conn(0x7efca80b5dc0 msgr2=0x7efca80c2d90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:06.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:06.555 7efcae15a700 1 -- 172.21.2.112:0/1471205650 shutdown_connections 2021-02-10T11:43:06.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:06.555 7efcae15a700 1 -- 172.21.2.112:0/1471205650 wait complete. 2021-02-10T11:43:06.563 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_data 8 2021-02-10T11:43:06.994 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:06.977 7f41c4064700 1 Processor -- start 2021-02-10T11:43:06.994 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:06.977 7f41c4064700 1 -- start start 2021-02-10T11:43:06.995 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:06.978 7f41c4064700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f41bc0d1c60 0x7f41bc0cef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:06.995 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:06.978 7f41c4064700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f41bc0cf570 0x7f41bc0cf970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:06.995 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:06.978 7f41c4064700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41bc0fe1b0 con 0x7f41bc0d1860 2021-02-10T11:43:06.995 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:06.978 7f41c4064700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41bc1b0ca0 con 0x7f41bc0cf570 2021-02-10T11:43:06.995 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:06.978 7f41c4064700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41bc109a50 con 0x7f41bc0d1c60 2021-02-10T11:43:06.995 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:06.978 7f41c2681700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f41bc0cf570 0x7f41bc0cf970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:06.995 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:06.978 7f41c2681700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f41bc0cf570 0x7f41bc0cf970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3300/0 says I am v2:172.21.2.112:59082/0 (socket says 172.21.2.112:59082) 2021-02-10T11:43:06.995 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:06.978 7f41c2681700 1 -- 172.21.2.112:0/1044504021 learned_addr learned my addr 172.21.2.112:0/1044504021 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:43:06.995 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:06.978 7f41c167f700 1 --2- 172.21.2.112:0/1044504021 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f41bc0d1c60 0x7f41bc0cef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:06.995 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:06.979 7f41c2681700 1 -- 172.21.2.112:0/1044504021 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f41bc0d1c60 msgr2=0x7f41bc0cef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:06.995 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:06.979 7f41c2681700 1 --2- 172.21.2.112:0/1044504021 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f41bc0d1c60 0x7f41bc0cef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:06.995 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:06.979 7f41c2681700 1 -- 172.21.2.112:0/1044504021 >> v1:172.21.2.120:6789/0 conn(0x7f41bc0d1860 legacy=0x7f41bc1b04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:06.995 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:06.979 7f41c2681700 1 -- 172.21.2.112:0/1044504021 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41bc088070 con 0x7f41bc0cf570 2021-02-10T11:43:06.996 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:06.979 7f41c0e7e700 1 -- 172.21.2.112:0/1044504021 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1608477387 0 0) 0x7f41b0000c60 con 0x7f41bc0d1860 2021-02-10T11:43:06.996 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:06.979 7f41c2681700 1 --2- 172.21.2.112:0/1044504021 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f41bc0cf570 0x7f41bc0cf970 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 rx=0x7f41b4006aa0 tx=0x7f41b4009ea0).ready entity=mon.1 client_cookie=a8b4c668c5cb8df2 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:06.996 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:06.979 7f41c0e7e700 1 -- 172.21.2.112:0/1044504021 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f41b400b510 con 0x7f41bc0cf570 2021-02-10T11:43:06.996 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:06.979 7f41c0e7e700 1 -- 172.21.2.112:0/1044504021 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41b4003030 con 0x7f41bc0cf570 2021-02-10T11:43:06.996 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:06.979 7f41c0e7e700 1 -- 172.21.2.112:0/1044504021 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f41b400bad0 con 0x7f41bc0cf570 2021-02-10T11:43:06.996 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:06.980 7f41c4064700 1 -- 172.21.2.112:0/1044504021 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f41bc0cf570 msgr2=0x7f41bc0cf970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:06.996 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:06.980 7f41c4064700 1 --2- 172.21.2.112:0/1044504021 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f41bc0cf570 0x7f41bc0cf970 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 rx=0x7f41b4006aa0 tx=0x7f41b4009ea0).stop 2021-02-10T11:43:06.996 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:06.980 7f41c4064700 1 -- 172.21.2.112:0/1044504021 shutdown_connections 2021-02-10T11:43:06.996 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:06.980 7f41c4064700 1 --2- 172.21.2.112:0/1044504021 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f41bc0cf570 0x7f41bc0cf970 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:06.996 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:06.980 7f41c4064700 1 --2- 172.21.2.112:0/1044504021 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f41bc0d1c60 0x7f41bc0cef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:06.996 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:06.980 7f41c4064700 1 -- 172.21.2.112:0/1044504021 >> 172.21.2.112:0/1044504021 conn(0x7f41bc0ce410 msgr2=0x7f41bc0d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:06.996 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:06.980 7f41c4064700 1 -- 172.21.2.112:0/1044504021 shutdown_connections 2021-02-10T11:43:06.996 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:06.980 7f41c4064700 1 -- 172.21.2.112:0/1044504021 wait complete. 2021-02-10T11:43:06.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:06.995 7f41c4064700 1 Processor -- start 2021-02-10T11:43:06.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:06.995 7f41c4064700 1 -- start start 2021-02-10T11:43:06.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:06.995 7f41c4064700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f41bc0c2a30 0x7f41bc0c2e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:06.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:06.995 7f41c4064700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f41bc0a7950 0x7f41bc0a7d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:06.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:06.995 7f41c4064700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f41bc1a71f0 0x7f41bc1a75f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:06.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:06.995 7f41c4064700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41bc0a8440 con 0x7f41bc0a7950 2021-02-10T11:43:06.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:06.995 7f41c4064700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41bc0c3790 con 0x7f41bc0c2a30 2021-02-10T11:43:06.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:06.995 7f41c4064700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41bc1a7c10 con 0x7f41bc1a71f0 2021-02-10T11:43:06.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:06.996 7f41c167f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f41bc0a7950 0x7f41bc0a7d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:06.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:06.996 7f41c167f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f41bc0a7950 0x7f41bc0a7d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:56804/0 (socket says 172.21.2.112:56804) 2021-02-10T11:43:06.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:06.996 7f41c167f700 1 -- 172.21.2.112:0/2120925554 learned_addr learned my addr 172.21.2.112:0/2120925554 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:43:06.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:06.996 7f41c167f700 1 -- 172.21.2.112:0/2120925554 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f41bc1a71f0 msgr2=0x7f41bc1a75f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:06.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:06.996 7f41c167f700 1 --2- 172.21.2.112:0/2120925554 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f41bc1a71f0 0x7f41bc1a75f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:06.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:06.996 7f41c167f700 1 -- 172.21.2.112:0/2120925554 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f41bc0c2a30 msgr2=0x7f41bc0c2e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:06.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:06.996 7f41c167f700 1 --2- 172.21.2.112:0/2120925554 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f41bc0c2a30 0x7f41bc0c2e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:06.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:06.996 7f41c167f700 1 -- 172.21.2.112:0/2120925554 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41bc1a7e30 con 0x7f41bc0a7950 2021-02-10T11:43:06.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:06.996 7f41c167f700 1 --2- 172.21.2.112:0/2120925554 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f41bc0a7950 0x7f41bc0a7d50 secure :-1 s=READY pgs=2423 cs=0 l=1 rev1=1 rx=0x7f41b8006f60 tx=0x7f41b800b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:06.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:06.996 7f41ae7fc700 1 -- 172.21.2.112:0/2120925554 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f41b80009b0 con 0x7f41bc0a7950 2021-02-10T11:43:07.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:06.996 7f41ae7fc700 1 -- 172.21.2.112:0/2120925554 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41b800bd60 con 0x7f41bc0a7950 2021-02-10T11:43:07.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:06.996 7f41c4064700 1 -- 172.21.2.112:0/2120925554 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41bc1a7e30 con 0x7f41bc0a7950 2021-02-10T11:43:07.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:06.996 7f41ae7fc700 1 -- 172.21.2.112:0/2120925554 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f41b8002030 con 0x7f41bc0a7950 2021-02-10T11:43:07.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:06.997 7f41c4064700 1 -- 172.21.2.112:0/2120925554 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41bc1a7e30 con 0x7f41bc0a7950 2021-02-10T11:43:07.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:06.997 7f41ae7fc700 1 -- 172.21.2.112:0/2120925554 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f41b8000c90 con 0x7f41bc0a7950 2021-02-10T11:43:07.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:06.997 7f41ae7fc700 1 --2- 172.21.2.112:0/2120925554 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f41a4018730 0x7f41a401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:07.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:06.997 7f41ae7fc700 1 -- 172.21.2.112:0/2120925554 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5107+0+0 (secure 0 0 0) 0x7f41b80325d0 con 0x7f41bc0a7950 2021-02-10T11:43:07.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:06.997 7f41c1e80700 1 --2- 172.21.2.112:0/2120925554 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f41a4018730 0x7f41a401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:07.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:06.998 7f41c1e80700 1 --2- 172.21.2.112:0/2120925554 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f41a4018730 0x7f41a401ac10 crc :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:07.027 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:07.028 7f41c4064700 1 -- 172.21.2.112:0/2120925554 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41bc082a20 con 0x7f41bc0a7950 2021-02-10T11:43:07.030 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:07.031 7f41ae7fc700 1 -- 172.21.2.112:0/2120925554 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f41b8006b00 con 0x7f41bc0a7950 2021-02-10T11:43:07.159 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:07.159 7f41c4064700 1 -- 172.21.2.112:0/2120925554 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"} v 0) v1 -- 0x7f41bc082a20 con 0x7f41bc0a7950 2021-02-10T11:43:07.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:07.558 7f41ae7fc700 1 -- 172.21.2.112:0/2120925554 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"}]=0 pool 'cephfs_data' created v132) v1 ==== 125+0+0 (secure 0 0 0) 0x7f41b8000c90 con 0x7f41bc0a7950 2021-02-10T11:43:07.575 INFO:teuthology.orchestra.run.gibba012.stderr:pool 'cephfs_data' created 2021-02-10T11:43:07.577 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:07.577 7f41c4064700 1 -- 172.21.2.112:0/2120925554 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f41a4018730 msgr2=0x7f41a401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:07.577 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:07.577 7f41c4064700 1 --2- 172.21.2.112:0/2120925554 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f41a4018730 0x7f41a401ac10 crc :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:07.577 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:07.577 7f41c4064700 1 -- 172.21.2.112:0/2120925554 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f41bc0a7950 msgr2=0x7f41bc0a7d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:07.577 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:07.577 7f41c4064700 1 --2- 172.21.2.112:0/2120925554 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f41bc0a7950 0x7f41bc0a7d50 secure :-1 s=READY pgs=2423 cs=0 l=1 rev1=1 rx=0x7f41b8006f60 tx=0x7f41b800b970).stop 2021-02-10T11:43:07.577 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:07.577 7f41c4064700 1 -- 172.21.2.112:0/2120925554 shutdown_connections 2021-02-10T11:43:07.577 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:07.577 7f41c4064700 1 --2- 172.21.2.112:0/2120925554 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f41a4018730 0x7f41a401ac10 unknown :-1 s=CLOSED pgs=1105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:07.577 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:07.577 7f41c4064700 1 --2- 172.21.2.112:0/2120925554 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f41bc1a71f0 0x7f41bc1a75f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:07.577 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:07.577 7f41c4064700 1 --2- 172.21.2.112:0/2120925554 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f41bc0a7950 0x7f41bc0a7d50 unknown :-1 s=CLOSED pgs=2423 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:07.577 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:07.577 7f41c4064700 1 --2- 172.21.2.112:0/2120925554 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f41bc0c2a30 0x7f41bc0c2e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:07.578 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:07.577 7f41c4064700 1 -- 172.21.2.112:0/2120925554 >> 172.21.2.112:0/2120925554 conn(0x7f41bc0b5f10 msgr2=0x7f41bc09f200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:07.578 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:07.577 7f41c4064700 1 -- 172.21.2.112:0/2120925554 shutdown_connections 2021-02-10T11:43:07.578 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:07.577 7f41c4064700 1 -- 172.21.2.112:0/2120925554 wait complete. 2021-02-10T11:43:07.585 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2021-02-10T11:43:08.015 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:07.999 7f2144d80700 1 Processor -- start 2021-02-10T11:43:08.015 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:07.999 7f2144d80700 1 -- start start 2021-02-10T11:43:08.015 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.000 7f2144d80700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f21400d1870 0x7f21401b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:08.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.000 7f2144d80700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21400fe1c0 con 0x7f21400cff70 2021-02-10T11:43:08.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.000 7f2144d80700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2140109a60 con 0x7f21401b0ae0 2021-02-10T11:43:08.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.000 7f2144d80700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2140088070 con 0x7f21400d1870 2021-02-10T11:43:08.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.000 7f213f314700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f21400cff70 0x7f2140109130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:56278/0 (socket says 172.21.2.112:56278) 2021-02-10T11:43:08.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.000 7f213f314700 1 -- 172.21.2.112:0/644831726 learned_addr learned my addr 172.21.2.112:0/644831726 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:43:08.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.000 7f213eb13700 1 --2- 172.21.2.112:0/644831726 >> v2:172.21.2.112:3300/0 conn(0x7f21400d1870 0x7f21401b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:08.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.000 7f213db11700 1 -- 172.21.2.112:0/644831726 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 620435914 0 0) 0x7f2138000f90 con 0x7f21400cff70 2021-02-10T11:43:08.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.000 7f213eb13700 1 -- 172.21.2.112:0/644831726 >> v1:172.21.2.120:6789/0 conn(0x7f21401b0ae0 legacy=0x7f2140108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:08.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.000 7f213eb13700 1 -- 172.21.2.112:0/644831726 >> v1:172.21.2.112:6789/0 conn(0x7f21400cff70 legacy=0x7f2140109130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:08.017 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.000 7f213eb13700 1 -- 172.21.2.112:0/644831726 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2140088260 con 0x7f21400d1870 2021-02-10T11:43:08.017 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.001 7f213eb13700 1 --2- 172.21.2.112:0/644831726 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f21400d1870 0x7f21401b04c0 secure :-1 s=READY pgs=2425 cs=0 l=1 rev1=1 rx=0x7f2130006650 tx=0x7f2130009a80).ready entity=mon.0 client_cookie=fe97c84c4362e2e6 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:08.017 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.001 7f213db11700 1 -- 172.21.2.112:0/644831726 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f213000c0c0 con 0x7f21400d1870 2021-02-10T11:43:08.017 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.001 7f213db11700 1 -- 172.21.2.112:0/644831726 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f213000c340 con 0x7f21400d1870 2021-02-10T11:43:08.017 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.001 7f213db11700 1 -- 172.21.2.112:0/644831726 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f213000c860 con 0x7f21400d1870 2021-02-10T11:43:08.017 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.001 7f2144d80700 1 -- 172.21.2.112:0/644831726 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f21400d1870 msgr2=0x7f21401b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:08.017 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.001 7f2144d80700 1 --2- 172.21.2.112:0/644831726 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f21400d1870 0x7f21401b04c0 secure :-1 s=READY pgs=2425 cs=0 l=1 rev1=1 rx=0x7f2130006650 tx=0x7f2130009a80).stop 2021-02-10T11:43:08.017 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.002 7f2144d80700 1 -- 172.21.2.112:0/644831726 shutdown_connections 2021-02-10T11:43:08.017 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.002 7f2144d80700 1 --2- 172.21.2.112:0/644831726 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f21400d1870 0x7f21401b04c0 unknown :-1 s=CLOSED pgs=2425 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:08.017 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.002 7f2144d80700 1 -- 172.21.2.112:0/644831726 >> 172.21.2.112:0/644831726 conn(0x7f21400ce420 msgr2=0x7f21400d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:08.018 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.002 7f2144d80700 1 -- 172.21.2.112:0/644831726 shutdown_connections 2021-02-10T11:43:08.018 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.002 7f2144d80700 1 -- 172.21.2.112:0/644831726 wait complete. 2021-02-10T11:43:08.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.016 7f2144d80700 1 Processor -- start 2021-02-10T11:43:08.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.016 7f2144d80700 1 -- start start 2021-02-10T11:43:08.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.016 7f2144d80700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f21401b33e0 0x7f21401a2b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:08.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.016 7f2144d80700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f21400931c0 0x7f21401b2610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:08.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.016 7f2144d80700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2140187e70 0x7f2140188270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:08.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.016 7f2144d80700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21401b2e60 con 0x7f21401b33e0 2021-02-10T11:43:08.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.016 7f2144d80700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21401b37e0 con 0x7f21400931c0 2021-02-10T11:43:08.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.016 7f2144d80700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21400935c0 con 0x7f2140187e70 2021-02-10T11:43:08.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.017 7f213eb13700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f21401b33e0 0x7f21401a2b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:08.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.017 7f213eb13700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f21401b33e0 0x7f21401a2b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:56822/0 (socket says 172.21.2.112:56822) 2021-02-10T11:43:08.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.017 7f213eb13700 1 -- 172.21.2.112:0/3561470943 learned_addr learned my addr 172.21.2.112:0/3561470943 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:43:08.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.017 7f213e312700 1 --2- 172.21.2.112:0/3561470943 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f21400931c0 0x7f21401b2610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:08.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.017 7f213eb13700 1 -- 172.21.2.112:0/3561470943 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2140187e70 msgr2=0x7f2140188270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:08.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.017 7f213eb13700 1 --2- 172.21.2.112:0/3561470943 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2140187e70 0x7f2140188270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:08.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.017 7f213eb13700 1 -- 172.21.2.112:0/3561470943 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f21400931c0 msgr2=0x7f21401b2610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:08.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.017 7f213eb13700 1 --2- 172.21.2.112:0/3561470943 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f21400931c0 0x7f21401b2610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:08.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.017 7f213eb13700 1 -- 172.21.2.112:0/3561470943 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2140188890 con 0x7f21401b33e0 2021-02-10T11:43:08.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.017 7f213e312700 1 --2- 172.21.2.112:0/3561470943 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f21400931c0 0x7f21401b2610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:43:08.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.017 7f213eb13700 1 --2- 172.21.2.112:0/3561470943 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f21401b33e0 0x7f21401a2b50 secure :-1 s=READY pgs=2426 cs=0 l=1 rev1=1 rx=0x7f213000b5c0 tx=0x7f213000b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:08.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.017 7f21377fe700 1 -- 172.21.2.112:0/3561470943 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2130006740 con 0x7f21401b33e0 2021-02-10T11:43:08.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.017 7f21377fe700 1 -- 172.21.2.112:0/3561470943 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21300064d0 con 0x7f21401b33e0 2021-02-10T11:43:08.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.017 7f21377fe700 1 -- 172.21.2.112:0/3561470943 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2130002030 con 0x7f21401b33e0 2021-02-10T11:43:08.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.018 7f2144d80700 1 -- 172.21.2.112:0/3561470943 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2140188890 con 0x7f21401b33e0 2021-02-10T11:43:08.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.018 7f2144d80700 1 -- 172.21.2.112:0/3561470943 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2140188890 con 0x7f21401b33e0 2021-02-10T11:43:08.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.018 7f21377fe700 1 -- 172.21.2.112:0/3561470943 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f21300052e0 con 0x7f21401b33e0 2021-02-10T11:43:08.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.018 7f21377fe700 1 --2- 172.21.2.112:0/3561470943 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f21240186c0 0x7f212401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:08.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.018 7f21377fe700 1 -- 172.21.2.112:0/3561470943 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5457+0+0 (secure 0 0 0) 0x7f2130032340 con 0x7f21401b33e0 2021-02-10T11:43:08.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.019 7f213e312700 1 --2- 172.21.2.112:0/3561470943 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f21240186c0 0x7f212401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:08.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.019 7f213e312700 1 --2- 172.21.2.112:0/3561470943 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f21240186c0 0x7f212401abc0 crc :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:08.050 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.050 7f2144d80700 1 -- 172.21.2.112:0/3561470943 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21400825d0 con 0x7f21401b33e0 2021-02-10T11:43:08.052 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.053 7f21377fe700 1 -- 172.21.2.112:0/3561470943 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2130000c90 con 0x7f21401b33e0 2021-02-10T11:43:08.181 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.181 7f2144d80700 1 -- 172.21.2.112:0/3561470943 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"} v 0) v1 -- 0x7f21400825d0 con 0x7f21401b33e0 2021-02-10T11:43:08.185 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.186 7f21377fe700 1 -- 172.21.2.112:0/3561470943 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"}]=0 new fs with metadata pool 28 and data pool 29 v111) v1 ==== 189+0+0 (secure 0 0 0) 0x7f213001f4a0 con 0x7f21401b33e0 2021-02-10T11:43:08.188 INFO:teuthology.orchestra.run.gibba012.stderr:new fs with metadata pool 28 and data pool 29 2021-02-10T11:43:08.191 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.191 7f2144d80700 1 -- 172.21.2.112:0/3561470943 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f21240186c0 msgr2=0x7f212401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:08.191 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.191 7f2144d80700 1 --2- 172.21.2.112:0/3561470943 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f21240186c0 0x7f212401abc0 crc :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:08.191 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.191 7f2144d80700 1 -- 172.21.2.112:0/3561470943 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f21401b33e0 msgr2=0x7f21401a2b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:08.191 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.191 7f2144d80700 1 --2- 172.21.2.112:0/3561470943 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f21401b33e0 0x7f21401a2b50 secure :-1 s=READY pgs=2426 cs=0 l=1 rev1=1 rx=0x7f213000b5c0 tx=0x7f213000b960).stop 2021-02-10T11:43:08.191 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.191 7f2144d80700 1 -- 172.21.2.112:0/3561470943 shutdown_connections 2021-02-10T11:43:08.191 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.191 7f2144d80700 1 --2- 172.21.2.112:0/3561470943 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f21240186c0 0x7f212401abc0 unknown :-1 s=CLOSED pgs=1108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:08.191 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.191 7f2144d80700 1 --2- 172.21.2.112:0/3561470943 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f21400931c0 0x7f21401b2610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:08.191 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.191 7f2144d80700 1 --2- 172.21.2.112:0/3561470943 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2140187e70 0x7f2140188270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:08.191 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.191 7f2144d80700 1 --2- 172.21.2.112:0/3561470943 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f21401b33e0 0x7f21401a2b50 unknown :-1 s=CLOSED pgs=2426 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:08.191 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.191 7f2144d80700 1 -- 172.21.2.112:0/3561470943 >> 172.21.2.112:0/3561470943 conn(0x7f21400841e0 msgr2=0x7f2140082800 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:08.191 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.191 7f2144d80700 1 -- 172.21.2.112:0/3561470943 shutdown_connections 2021-02-10T11:43:08.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.191 7f2144d80700 1 -- 172.21.2.112:0/3561470943 wait complete. 2021-02-10T11:43:08.199 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:43:08.624 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.608 7fe4e5ecc700 1 Processor -- start 2021-02-10T11:43:08.624 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.608 7fe4e5ecc700 1 -- start start 2021-02-10T11:43:08.624 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.609 7fe4e5ecc700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fe4e00d1870 0x7fe4e01b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:08.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.609 7fe4e5ecc700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe4e01b0a10 0x7fe4e00cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:08.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.609 7fe4e5ecc700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4e00f0b10 con 0x7fe4e00d1c70 2021-02-10T11:43:08.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.609 7fe4e5ecc700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4e00e5ed0 con 0x7fe4e00d1870 2021-02-10T11:43:08.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.609 7fe4e5ecc700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4e00f0650 con 0x7fe4e01b0a10 2021-02-10T11:43:08.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.609 7fe4df7fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fe4e00d1870 0x7fe4e01b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:08.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.609 7fe4df7fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fe4e00d1870 0x7fe4e01b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:56826/0 (socket says 172.21.2.112:56826) 2021-02-10T11:43:08.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.609 7fe4df7fe700 1 -- 172.21.2.112:0/1428011819 learned_addr learned my addr 172.21.2.112:0/1428011819 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:43:08.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.609 7fe4df7fe700 1 -- 172.21.2.112:0/1428011819 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe4e01b0a10 msgr2=0x7fe4e00cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:08.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.609 7fe4df7fe700 1 --2- 172.21.2.112:0/1428011819 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe4e01b0a10 0x7fe4e00cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:08.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.609 7fe4df7fe700 1 -- 172.21.2.112:0/1428011819 >> v1:172.21.2.120:6789/0 conn(0x7fe4e00d1c70 legacy=0x7fe4e00cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:08.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.609 7fe4df7fe700 1 -- 172.21.2.112:0/1428011819 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4e00f0870 con 0x7fe4e00d1870 2021-02-10T11:43:08.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.609 7fe4df7fe700 1 --2- 172.21.2.112:0/1428011819 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe4e00d1870 0x7fe4e01b03f0 secure :-1 s=READY pgs=2427 cs=0 l=1 rev1=1 rx=0x7fe4c8006700 tx=0x7fe4c8009b80).ready entity=mon.0 client_cookie=ce7ca1d2f6d92449 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:08.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.610 7fe4de7fc700 1 -- 172.21.2.112:0/1428011819 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe4c8006340 con 0x7fe4e00d1870 2021-02-10T11:43:08.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.610 7fe4de7fc700 1 -- 172.21.2.112:0/1428011819 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4c800c0c0 con 0x7fe4e00d1870 2021-02-10T11:43:08.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.610 7fe4de7fc700 1 -- 172.21.2.112:0/1428011819 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe4c800c5e0 con 0x7fe4e00d1870 2021-02-10T11:43:08.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.610 7fe4e5ecc700 1 -- 172.21.2.112:0/1428011819 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe4e00d1870 msgr2=0x7fe4e01b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:08.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.610 7fe4e5ecc700 1 --2- 172.21.2.112:0/1428011819 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe4e00d1870 0x7fe4e01b03f0 secure :-1 s=READY pgs=2427 cs=0 l=1 rev1=1 rx=0x7fe4c8006700 tx=0x7fe4c8009b80).stop 2021-02-10T11:43:08.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.610 7fe4e5ecc700 1 -- 172.21.2.112:0/1428011819 shutdown_connections 2021-02-10T11:43:08.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.610 7fe4e5ecc700 1 --2- 172.21.2.112:0/1428011819 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe4e01b0a10 0x7fe4e00cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:08.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.610 7fe4e5ecc700 1 --2- 172.21.2.112:0/1428011819 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe4e00d1870 0x7fe4e01b03f0 unknown :-1 s=CLOSED pgs=2427 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:08.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.610 7fe4e5ecc700 1 -- 172.21.2.112:0/1428011819 >> 172.21.2.112:0/1428011819 conn(0x7fe4e00ce450 msgr2=0x7fe4e00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:08.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.610 7fe4e5ecc700 1 -- 172.21.2.112:0/1428011819 shutdown_connections 2021-02-10T11:43:08.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.610 7fe4e5ecc700 1 -- 172.21.2.112:0/1428011819 wait complete. 2021-02-10T11:43:08.629 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.625 7fe4e5ecc700 1 Processor -- start 2021-02-10T11:43:08.629 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.625 7fe4e5ecc700 1 -- start start 2021-02-10T11:43:08.629 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.625 7fe4e5ecc700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe4e00b0ad0 0x7fe4e0091f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:08.629 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.625 7fe4e5ecc700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe4e01a8850 0x7fe4e01a8c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:08.629 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.625 7fe4e5ecc700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe4e01a6fa0 0x7fe4e01a73a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:08.629 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.625 7fe4e5ecc700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4e01abfe0 con 0x7fe4e01a8850 2021-02-10T11:43:08.630 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.625 7fe4e5ecc700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4e01a9c40 con 0x7fe4e00b0ad0 2021-02-10T11:43:08.630 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.625 7fe4e5ecc700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4e01a9730 con 0x7fe4e01a6fa0 2021-02-10T11:43:08.630 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.625 7fe4deffd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe4e01a8850 0x7fe4e01a8c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:08.630 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.626 7fe4deffd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe4e01a8850 0x7fe4e01a8c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:56828/0 (socket says 172.21.2.112:56828) 2021-02-10T11:43:08.630 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.626 7fe4deffd700 1 -- 172.21.2.112:0/557767301 learned_addr learned my addr 172.21.2.112:0/557767301 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:43:08.630 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.626 7fe4deffd700 1 -- 172.21.2.112:0/557767301 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe4e01a6fa0 msgr2=0x7fe4e01a73a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:08.630 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.626 7fe4deffd700 1 --2- 172.21.2.112:0/557767301 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe4e01a6fa0 0x7fe4e01a73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:08.630 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.626 7fe4deffd700 1 -- 172.21.2.112:0/557767301 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe4e00b0ad0 msgr2=0x7fe4e0091f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:08.630 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.626 7fe4df7fe700 1 --2- 172.21.2.112:0/557767301 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe4e00b0ad0 0x7fe4e0091f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:08.630 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.626 7fe4deffd700 1 --2- 172.21.2.112:0/557767301 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe4e00b0ad0 0x7fe4e0091f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:08.630 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.626 7fe4deffd700 1 -- 172.21.2.112:0/557767301 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4e01a9270 con 0x7fe4e01a8850 2021-02-10T11:43:08.630 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.626 7fe4deffd700 1 --2- 172.21.2.112:0/557767301 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe4e01a8850 0x7fe4e01a8c50 secure :-1 s=READY pgs=2428 cs=0 l=1 rev1=1 rx=0x7fe4d000b530 tx=0x7fe4d000b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:08.630 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.626 7fe4d7fff700 1 -- 172.21.2.112:0/557767301 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe4d000bcc0 con 0x7fe4e01a8850 2021-02-10T11:43:08.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.626 7fe4d7fff700 1 -- 172.21.2.112:0/557767301 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4d0000c90 con 0x7fe4e01a8850 2021-02-10T11:43:08.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.626 7fe4e5ecc700 1 -- 172.21.2.112:0/557767301 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4e01a9270 con 0x7fe4e01a8850 2021-02-10T11:43:08.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.626 7fe4e5ecc700 1 -- 172.21.2.112:0/557767301 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4e01a9270 con 0x7fe4e01a8850 2021-02-10T11:43:08.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.626 7fe4d7fff700 1 -- 172.21.2.112:0/557767301 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe4d00022a0 con 0x7fe4e01a8850 2021-02-10T11:43:08.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.627 7fe4d7fff700 1 -- 172.21.2.112:0/557767301 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe4d0005310 con 0x7fe4e01a8850 2021-02-10T11:43:08.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.627 7fe4d7fff700 1 --2- 172.21.2.112:0/557767301 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe4c40186c0 0x7fe4c401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:08.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.627 7fe4d7fff700 1 -- 172.21.2.112:0/557767301 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fe4d0032530 con 0x7fe4e01a8850 2021-02-10T11:43:08.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.628 7fe4df7fe700 1 --2- 172.21.2.112:0/557767301 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe4c40186c0 0x7fe4c401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:08.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.628 7fe4df7fe700 1 --2- 172.21.2.112:0/557767301 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe4c40186c0 0x7fe4c401abc0 crc :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:08.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.659 7fe4e5ecc700 1 -- 172.21.2.112:0/557767301 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4e01abfe0 con 0x7fe4e01a8850 2021-02-10T11:43:08.661 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.662 7fe4d7fff700 1 -- 172.21.2.112:0/557767301 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe4d0006b90 con 0x7fe4e01a8850 2021-02-10T11:43:08.786 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.786 7fe4e5ecc700 1 -- 172.21.2.112:0/557767301 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe4e01abfe0 con 0x7fe4e01a8850 2021-02-10T11:43:08.786 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.787 7fe4d7fff700 1 -- 172.21.2.112:0/557767301 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v133) v1 ==== 74+0+14020 (secure 0 0 0) 0x7fe4d0005310 con 0x7fe4e01a8850 2021-02-10T11:43:08.787 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:43:08.787 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":133,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:43:08.182540","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":28,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:43:06.157376","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"133","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":29,"pool_name":"cephfs_data","create_time":"2021-02-10 11:43:07.178112","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"133","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":132,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":132,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":132,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":132,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:43:08.788 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.788 7fe4e5ecc700 1 -- 172.21.2.112:0/557767301 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe4c40186c0 msgr2=0x7fe4c401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:08.788 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.788 7fe4e5ecc700 1 --2- 172.21.2.112:0/557767301 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe4c40186c0 0x7fe4c401abc0 crc :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:08.788 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.788 7fe4e5ecc700 1 -- 172.21.2.112:0/557767301 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe4e01a8850 msgr2=0x7fe4e01a8c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:08.788 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.788 7fe4e5ecc700 1 --2- 172.21.2.112:0/557767301 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe4e01a8850 0x7fe4e01a8c50 secure :-1 s=READY pgs=2428 cs=0 l=1 rev1=1 rx=0x7fe4d000b530 tx=0x7fe4d000b8d0).stop 2021-02-10T11:43:08.788 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.788 7fe4e5ecc700 1 -- 172.21.2.112:0/557767301 shutdown_connections 2021-02-10T11:43:08.788 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.788 7fe4e5ecc700 1 --2- 172.21.2.112:0/557767301 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe4c40186c0 0x7fe4c401abc0 unknown :-1 s=CLOSED pgs=1109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:08.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.788 7fe4e5ecc700 1 --2- 172.21.2.112:0/557767301 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe4e01a6fa0 0x7fe4e01a73a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:08.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.788 7fe4e5ecc700 1 --2- 172.21.2.112:0/557767301 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe4e01a8850 0x7fe4e01a8c50 unknown :-1 s=CLOSED pgs=2428 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:08.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.788 7fe4e5ecc700 1 --2- 172.21.2.112:0/557767301 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe4e00b0ad0 0x7fe4e0091f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:08.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.788 7fe4e5ecc700 1 -- 172.21.2.112:0/557767301 >> 172.21.2.112:0/557767301 conn(0x7fe4e019f8f0 msgr2=0x7fe4e01a5f80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:08.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.789 7fe4e5ecc700 1 -- 172.21.2.112:0/557767301 shutdown_connections 2021-02-10T11:43:08.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:08.789 7fe4e5ecc700 1 -- 172.21.2.112:0/557767301 wait complete. 2021-02-10T11:43:08.797 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:43:09.224 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.208 7f8af8a62700 1 Processor -- start 2021-02-10T11:43:09.224 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.208 7f8af8a62700 1 -- start start 2021-02-10T11:43:09.224 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.209 7f8af8a62700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f8af41b0470 0x7f8af41b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:09.224 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.209 7f8af8a62700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8af40ef610 0x7f8af40efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:09.224 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.209 7f8af8a62700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8af40f0b50 con 0x7f8af40f0030 2021-02-10T11:43:09.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.209 7f8af8a62700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8af40e52e0 con 0x7f8af41b0470 2021-02-10T11:43:09.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.209 7f8af8a62700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8af40e54d0 con 0x7f8af40ef610 2021-02-10T11:43:09.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.209 7f8af305f700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7f8af40f0030 0x7f8af40f10d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:49860/0 (socket says 172.21.2.112:49860) 2021-02-10T11:43:09.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.209 7f8af305f700 1 -- 172.21.2.112:0/1443160645 learned_addr learned my addr 172.21.2.112:0/1443160645 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:43:09.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.209 7f8af285e700 1 --2- 172.21.2.112:0/1443160645 >> v2:172.21.2.120:3300/0 conn(0x7f8af41b0470 0x7f8af41b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:09.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.209 7f8af205d700 1 --2- 172.21.2.112:0/1443160645 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8af40ef610 0x7f8af40efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:09.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.209 7f8af185c700 1 -- 172.21.2.112:0/1443160645 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2719143231 0 0) 0x7f8aec001440 con 0x7f8af40f0030 2021-02-10T11:43:09.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.209 7f8af205d700 1 -- 172.21.2.112:0/1443160645 >> v2:172.21.2.120:3300/0 conn(0x7f8af41b0470 msgr2=0x7f8af41b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:09.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.209 7f8af205d700 1 --2- 172.21.2.112:0/1443160645 >> v2:172.21.2.120:3300/0 conn(0x7f8af41b0470 0x7f8af41b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:09.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.209 7f8af205d700 1 -- 172.21.2.112:0/1443160645 >> v1:172.21.2.120:6789/0 conn(0x7f8af40f0030 legacy=0x7f8af40f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:09.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.209 7f8af205d700 1 -- 172.21.2.112:0/1443160645 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8af40f0430 con 0x7f8af40ef610 2021-02-10T11:43:09.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.210 7f8af205d700 1 --2- 172.21.2.112:0/1443160645 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8af40ef610 0x7f8af40efa10 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7f8adc006650 tx=0x7f8adc009a80).ready entity=mon.2 client_cookie=161ad8e5a0c07b2a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:09.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.210 7f8af185c700 1 -- 172.21.2.112:0/1443160645 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8adc00c0c0 con 0x7f8af40ef610 2021-02-10T11:43:09.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.210 7f8af185c700 1 -- 172.21.2.112:0/1443160645 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8adc00c0c0 con 0x7f8af40ef610 2021-02-10T11:43:09.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.210 7f8af185c700 1 -- 172.21.2.112:0/1443160645 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8adc00c340 con 0x7f8af40ef610 2021-02-10T11:43:09.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.210 7f8af8a62700 1 -- 172.21.2.112:0/1443160645 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8af40ef610 msgr2=0x7f8af40efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:09.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.210 7f8af8a62700 1 --2- 172.21.2.112:0/1443160645 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8af40ef610 0x7f8af40efa10 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7f8adc006650 tx=0x7f8adc009a80).stop 2021-02-10T11:43:09.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.210 7f8af8a62700 1 -- 172.21.2.112:0/1443160645 shutdown_connections 2021-02-10T11:43:09.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.210 7f8af8a62700 1 --2- 172.21.2.112:0/1443160645 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8af40ef610 0x7f8af40efa10 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:09.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.210 7f8af8a62700 1 --2- 172.21.2.112:0/1443160645 >> v2:172.21.2.120:3300/0 conn(0x7f8af41b0470 0x7f8af41b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:09.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.210 7f8af8a62700 1 -- 172.21.2.112:0/1443160645 >> 172.21.2.112:0/1443160645 conn(0x7f8af40ce3e0 msgr2=0x7f8af40e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:09.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.211 7f8af8a62700 1 -- 172.21.2.112:0/1443160645 shutdown_connections 2021-02-10T11:43:09.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.211 7f8af8a62700 1 -- 172.21.2.112:0/1443160645 wait complete. 2021-02-10T11:43:09.228 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.225 7f8af8a62700 1 Processor -- start 2021-02-10T11:43:09.228 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.225 7f8af8a62700 1 -- start start 2021-02-10T11:43:09.228 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.225 7f8af8a62700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8af419fec0 0x7f8af41acfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:09.228 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.225 7f8af8a62700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8af41aea70 0x7f8af41aee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:09.228 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.225 7f8af8a62700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8af41af490 0x7f8af41a0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:09.228 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.225 7f8af8a62700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8af41a02c0 con 0x7f8af41aea70 2021-02-10T11:43:09.228 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.225 7f8af8a62700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8af41af890 con 0x7f8af419fec0 2021-02-10T11:43:09.228 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.225 7f8af8a62700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8af41afab0 con 0x7f8af41af490 2021-02-10T11:43:09.228 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.225 7f8af205d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8af41aea70 0x7f8af41aee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:09.228 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.225 7f8af205d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8af41aea70 0x7f8af41aee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:56842/0 (socket says 172.21.2.112:56842) 2021-02-10T11:43:09.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.225 7f8af205d700 1 -- 172.21.2.112:0/1215352174 learned_addr learned my addr 172.21.2.112:0/1215352174 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:43:09.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.226 7f8af285e700 1 --2- 172.21.2.112:0/1215352174 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8af419fec0 0x7f8af41acfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:09.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.226 7f8af205d700 1 -- 172.21.2.112:0/1215352174 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8af41af490 msgr2=0x7f8af41a0ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:09.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.226 7f8af205d700 1 --2- 172.21.2.112:0/1215352174 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8af41af490 0x7f8af41a0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:09.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.226 7f8af205d700 1 -- 172.21.2.112:0/1215352174 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8af419fec0 msgr2=0x7f8af41acfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:09.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.226 7f8af205d700 1 --2- 172.21.2.112:0/1215352174 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8af419fec0 0x7f8af41acfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:09.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.226 7f8af205d700 1 -- 172.21.2.112:0/1215352174 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8af41a1100 con 0x7f8af41aea70 2021-02-10T11:43:09.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.226 7f8af285e700 1 --2- 172.21.2.112:0/1215352174 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8af419fec0 0x7f8af41acfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:43:09.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.226 7f8af205d700 1 --2- 172.21.2.112:0/1215352174 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8af41aea70 0x7f8af41aee70 secure :-1 s=READY pgs=2429 cs=0 l=1 rev1=1 rx=0x7f8adc009650 tx=0x7f8adc00b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:09.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.226 7f8aeb7fe700 1 -- 172.21.2.112:0/1215352174 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8adc00bd50 con 0x7f8af41aea70 2021-02-10T11:43:09.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.226 7f8aeb7fe700 1 -- 172.21.2.112:0/1215352174 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8adc006030 con 0x7f8af41aea70 2021-02-10T11:43:09.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.226 7f8aeb7fe700 1 -- 172.21.2.112:0/1215352174 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8adc005560 con 0x7f8af41aea70 2021-02-10T11:43:09.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.226 7f8af8a62700 1 -- 172.21.2.112:0/1215352174 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8af41a1100 con 0x7f8af41aea70 2021-02-10T11:43:09.230 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.226 7f8af8a62700 1 -- 172.21.2.112:0/1215352174 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8af41a1690 con 0x7f8af41aea70 2021-02-10T11:43:09.230 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.227 7f8aeb7fe700 1 -- 172.21.2.112:0/1215352174 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8adc0052e0 con 0x7f8af41aea70 2021-02-10T11:43:09.230 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.227 7f8aeb7fe700 1 --2- 172.21.2.112:0/1215352174 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8ad80186c0 0x7f8ad801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:09.230 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.227 7f8aeb7fe700 1 -- 172.21.2.112:0/1215352174 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f8adc0323e0 con 0x7f8af41aea70 2021-02-10T11:43:09.230 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.228 7f8af285e700 1 --2- 172.21.2.112:0/1215352174 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8ad80186c0 0x7f8ad801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:09.230 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.228 7f8af285e700 1 --2- 172.21.2.112:0/1215352174 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8ad80186c0 0x7f8ad801abc0 crc :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:09.258 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.258 7f8af8a62700 1 -- 172.21.2.112:0/1215352174 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8af41a05b0 con 0x7f8af41aea70 2021-02-10T11:43:09.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.261 7f8aeb7fe700 1 -- 172.21.2.112:0/1215352174 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8adc009300 con 0x7f8af41aea70 2021-02-10T11:43:09.385 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.385 7f8af8a62700 1 -- 172.21.2.112:0/1215352174 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8af41a05b0 con 0x7f8af41aea70 2021-02-10T11:43:09.385 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.386 7f8aeb7fe700 1 -- 172.21.2.112:0/1215352174 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v134) v1 ==== 74+0+14007 (secure 0 0 0) 0x7f8adc0052e0 con 0x7f8af41aea70 2021-02-10T11:43:09.386 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:43:09.386 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":134,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:43:09.185424","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":28,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:43:06.157376","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"133","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":29,"pool_name":"cephfs_data","create_time":"2021-02-10 11:43:07.178112","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"134","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":132,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":132,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":132,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":132,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:43:09.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.387 7f8af8a62700 1 -- 172.21.2.112:0/1215352174 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8ad80186c0 msgr2=0x7f8ad801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:09.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.387 7f8af8a62700 1 --2- 172.21.2.112:0/1215352174 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8ad80186c0 0x7f8ad801abc0 crc :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:09.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.387 7f8af8a62700 1 -- 172.21.2.112:0/1215352174 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8af41aea70 msgr2=0x7f8af41aee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:09.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.387 7f8af8a62700 1 --2- 172.21.2.112:0/1215352174 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8af41aea70 0x7f8af41aee70 secure :-1 s=READY pgs=2429 cs=0 l=1 rev1=1 rx=0x7f8adc009650 tx=0x7f8adc00b960).stop 2021-02-10T11:43:09.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.388 7f8af8a62700 1 -- 172.21.2.112:0/1215352174 shutdown_connections 2021-02-10T11:43:09.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.388 7f8af8a62700 1 --2- 172.21.2.112:0/1215352174 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8ad80186c0 0x7f8ad801abc0 unknown :-1 s=CLOSED pgs=1110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:09.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.388 7f8af8a62700 1 --2- 172.21.2.112:0/1215352174 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8af41af490 0x7f8af41a0ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:09.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.388 7f8af8a62700 1 --2- 172.21.2.112:0/1215352174 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8af41aea70 0x7f8af41aee70 unknown :-1 s=CLOSED pgs=2429 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:09.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.388 7f8af8a62700 1 --2- 172.21.2.112:0/1215352174 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8af419fec0 0x7f8af41acfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:09.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.388 7f8af8a62700 1 -- 172.21.2.112:0/1215352174 >> 172.21.2.112:0/1215352174 conn(0x7f8af41a37a0 msgr2=0x7f8af418c230 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:09.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.388 7f8af8a62700 1 -- 172.21.2.112:0/1215352174 shutdown_connections 2021-02-10T11:43:09.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.388 7f8af8a62700 1 -- 172.21.2.112:0/1215352174 wait complete. 2021-02-10T11:43:09.397 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2021-02-10T11:43:09.829 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.812 7f8dc4b3a700 1 Processor -- start 2021-02-10T11:43:09.830 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.812 7f8dc4b3a700 1 -- start start 2021-02-10T11:43:09.830 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.812 7f8dc4b3a700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f8dc00d1870 0x7f8dc01b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:09.830 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.812 7f8dc4b3a700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8dc00fe1c0 con 0x7f8dc01b0ae0 2021-02-10T11:43:09.830 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.812 7f8dc4b3a700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8dc0109a60 con 0x7f8dc00cff70 2021-02-10T11:43:09.830 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.812 7f8dc4b3a700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8dc0088070 con 0x7f8dc00d1870 2021-02-10T11:43:09.830 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.813 7f8dbe05d700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f8dc01b0ae0 0x7f8dc0108940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:56314/0 (socket says 172.21.2.112:56314) 2021-02-10T11:43:09.830 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.813 7f8dbe05d700 1 -- 172.21.2.112:0/2399836582 learned_addr learned my addr 172.21.2.112:0/2399836582 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:43:09.830 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.813 7f8dbd85c700 1 -- 172.21.2.112:0/2399836582 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4290749984 0 0) 0x7f8db00010b0 con 0x7f8dc01b0ae0 2021-02-10T11:43:09.830 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.813 7f8dbe85e700 1 --2- 172.21.2.112:0/2399836582 >> v2:172.21.2.120:3300/0 conn(0x7f8dc00d1870 0x7f8dc01b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:09.830 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.813 7f8dbd85c700 1 -- 172.21.2.112:0/2399836582 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8dac002fe0 con 0x7f8dc01b0ae0 2021-02-10T11:43:09.830 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.813 7f8dbd85c700 1 -- 172.21.2.112:0/2399836582 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2391877287 0 0) 0x7f8db00010b0 con 0x7f8dc01b0ae0 2021-02-10T11:43:09.830 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.813 7f8dbd85c700 1 -- 172.21.2.112:0/2399836582 >> v2:172.21.2.120:3300/0 conn(0x7f8dc00d1870 msgr2=0x7f8dc01b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:09.831 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.813 7f8dbd85c700 1 --2- 172.21.2.112:0/2399836582 >> v2:172.21.2.120:3300/0 conn(0x7f8dc00d1870 0x7f8dc01b04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:09.831 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.813 7f8dbd85c700 1 -- 172.21.2.112:0/2399836582 >> v1:172.21.2.120:6789/0 conn(0x7f8dc00cff70 legacy=0x7f8dc0109130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:09.831 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.813 7f8dbd85c700 1 -- 172.21.2.112:0/2399836582 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dc0088260 con 0x7f8dc01b0ae0 2021-02-10T11:43:09.831 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.813 7f8dbd85c700 1 -- 172.21.2.112:0/2399836582 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f8db00015b0 con 0x7f8dc01b0ae0 2021-02-10T11:43:09.831 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.813 7f8dbd85c700 1 -- 172.21.2.112:0/2399836582 >> v1:172.21.2.112:6789/0 conn(0x7f8dc01b0ae0 legacy=0x7f8dc0108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:09.831 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.813 7f8dbd85c700 1 --2- 172.21.2.112:0/2399836582 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8dac003a40 0x7f8dac005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:09.831 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.813 7f8dbd85c700 1 --2- 172.21.2.112:0/2399836582 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8dac006790 0x7f8dac008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:09.831 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.813 7f8dbd85c700 1 --2- 172.21.2.112:0/2399836582 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8dac009590 0x7f8dac00bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:09.831 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.813 7f8dbd85c700 1 -- 172.21.2.112:0/2399836582 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dac00c300 con 0x7f8dac009590 2021-02-10T11:43:09.831 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.814 7f8dbd85c700 1 -- 172.21.2.112:0/2399836582 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dac00c520 con 0x7f8dac003a40 2021-02-10T11:43:09.831 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.814 7f8dbd85c700 1 -- 172.21.2.112:0/2399836582 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8dac00c740 con 0x7f8dac006790 2021-02-10T11:43:09.831 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.814 7f8dbe85e700 1 --2- 172.21.2.112:0/2399836582 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8dac003a40 0x7f8dac005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:09.831 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.814 7f8dbf05f700 1 --2- 172.21.2.112:0/2399836582 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8dac009590 0x7f8dac00bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:09.831 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.814 7f8dbe05d700 1 --2- 172.21.2.112:0/2399836582 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8dac006790 0x7f8dac008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:09.832 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.814 7f8dbf05f700 1 -- 172.21.2.112:0/2399836582 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8dac006790 msgr2=0x7f8dac008c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:09.832 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.814 7f8dbf05f700 1 --2- 172.21.2.112:0/2399836582 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8dac006790 0x7f8dac008c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:09.832 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.814 7f8dbf05f700 1 -- 172.21.2.112:0/2399836582 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8dac003a40 msgr2=0x7f8dac005ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:09.832 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.814 7f8dbf05f700 1 --2- 172.21.2.112:0/2399836582 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8dac003a40 0x7f8dac005ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:09.832 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.814 7f8dbf05f700 1 -- 172.21.2.112:0/2399836582 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dac0509b0 con 0x7f8dac009590 2021-02-10T11:43:09.832 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.814 7f8dbf05f700 1 -- 172.21.2.112:0/2399836582 reap_dead start 2021-02-10T11:43:09.832 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.814 7f8dbf05f700 1 --2- 172.21.2.112:0/2399836582 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8dac009590 0x7f8dac00bab0 secure :-1 s=READY pgs=2431 cs=0 l=1 rev1=1 rx=0x7f8db8006e40 tx=0x7f8db8009e30).ready entity=mon.0 client_cookie=ff00652a69e5d57d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:09.832 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.814 7f8dbd85c700 1 -- 172.21.2.112:0/2399836582 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8db8003020 con 0x7f8dac009590 2021-02-10T11:43:09.832 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.814 7f8dbd85c700 1 -- 172.21.2.112:0/2399836582 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8db800b510 con 0x7f8dac009590 2021-02-10T11:43:09.832 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.814 7f8dbd85c700 1 -- 172.21.2.112:0/2399836582 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8db8006a10 con 0x7f8dac009590 2021-02-10T11:43:09.832 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.814 7f8dc4b3a700 1 -- 172.21.2.112:0/2399836582 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8dac009590 msgr2=0x7f8dac00bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:09.832 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.814 7f8dc4b3a700 1 --2- 172.21.2.112:0/2399836582 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8dac009590 0x7f8dac00bab0 secure :-1 s=READY pgs=2431 cs=0 l=1 rev1=1 rx=0x7f8db8006e40 tx=0x7f8db8009e30).stop 2021-02-10T11:43:09.832 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.815 7f8dc4b3a700 1 -- 172.21.2.112:0/2399836582 shutdown_connections 2021-02-10T11:43:09.833 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.815 7f8dc4b3a700 1 --2- 172.21.2.112:0/2399836582 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8dac009590 0x7f8dac00bab0 unknown :-1 s=CLOSED pgs=2431 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:09.833 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.815 7f8dc4b3a700 1 -- 172.21.2.112:0/2399836582 >> 172.21.2.112:0/2399836582 conn(0x7f8dc00ce420 msgr2=0x7f8dc00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:09.833 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.815 7f8dc4b3a700 1 -- 172.21.2.112:0/2399836582 shutdown_connections 2021-02-10T11:43:09.833 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.815 7f8dc4b3a700 1 -- 172.21.2.112:0/2399836582 wait complete. 2021-02-10T11:43:09.833 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.830 7f8dc4b3a700 1 Processor -- start 2021-02-10T11:43:09.833 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.830 7f8dc4b3a700 1 -- start start 2021-02-10T11:43:09.834 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.830 7f8dc4b3a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8dc009bcc0 0x7f8dc009c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:09.834 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.830 7f8dc4b3a700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8dc00a0820 0x7f8dc00a0c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:09.834 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.830 7f8dc4b3a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8dc00a1240 0x7f8dc00858d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:09.834 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.830 7f8dc4b3a700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dc00a66b0 con 0x7f8dc009bcc0 2021-02-10T11:43:09.835 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.830 7f8dc4b3a700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dc01a49b0 con 0x7f8dc00a0820 2021-02-10T11:43:09.835 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.830 7f8dc4b3a700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8dc01a4b90 con 0x7f8dc00a1240 2021-02-10T11:43:09.835 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.830 7f8dbe85e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8dc009bcc0 0x7f8dc009c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:09.835 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.830 7f8dbe85e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8dc009bcc0 0x7f8dc009c0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:56862/0 (socket says 172.21.2.112:56862) 2021-02-10T11:43:09.835 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.830 7f8dbe85e700 1 -- 172.21.2.112:0/1053816005 learned_addr learned my addr 172.21.2.112:0/1053816005 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:43:09.835 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.831 7f8dbe85e700 1 -- 172.21.2.112:0/1053816005 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8dc00a1240 msgr2=0x7f8dc00858d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:09.835 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.831 7f8dbe85e700 1 --2- 172.21.2.112:0/1053816005 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8dc00a1240 0x7f8dc00858d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:09.835 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.831 7f8dbe85e700 1 -- 172.21.2.112:0/1053816005 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8dc00a0820 msgr2=0x7f8dc00a0c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:09.835 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.831 7f8dbe85e700 1 --2- 172.21.2.112:0/1053816005 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8dc00a0820 0x7f8dc00a0c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:09.835 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.831 7f8dbe85e700 1 -- 172.21.2.112:0/1053816005 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dc00a1640 con 0x7f8dc009bcc0 2021-02-10T11:43:09.835 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.831 7f8dbe85e700 1 --2- 172.21.2.112:0/1053816005 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8dc009bcc0 0x7f8dc009c0c0 secure :-1 s=READY pgs=2432 cs=0 l=1 rev1=1 rx=0x7f8da800b560 tx=0x7f8da800b900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:09.835 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.831 7f8db77fe700 1 -- 172.21.2.112:0/1053816005 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8da80009b0 con 0x7f8dc009bcc0 2021-02-10T11:43:09.835 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.831 7f8db77fe700 1 -- 172.21.2.112:0/1053816005 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8da800bd30 con 0x7f8dc009bcc0 2021-02-10T11:43:09.836 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.831 7f8db77fe700 1 -- 172.21.2.112:0/1053816005 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8da8002030 con 0x7f8dc009bcc0 2021-02-10T11:43:09.836 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.831 7f8dc4b3a700 1 -- 172.21.2.112:0/1053816005 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8dc00a1640 con 0x7f8dc009bcc0 2021-02-10T11:43:09.836 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.831 7f8dc4b3a700 1 -- 172.21.2.112:0/1053816005 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8dc00861e0 con 0x7f8dc009bcc0 2021-02-10T11:43:09.836 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.832 7f8db77fe700 1 -- 172.21.2.112:0/1053816005 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8da80052e0 con 0x7f8dc009bcc0 2021-02-10T11:43:09.836 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.832 7f8db77fe700 1 --2- 172.21.2.112:0/1053816005 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8da40186c0 0x7f8da401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:09.836 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.832 7f8db77fe700 1 -- 172.21.2.112:0/1053816005 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f8da8032560 con 0x7f8dc009bcc0 2021-02-10T11:43:09.836 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.832 7f8dbe05d700 1 --2- 172.21.2.112:0/1053816005 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8da40186c0 0x7f8da401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:09.836 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.833 7f8dbe05d700 1 --2- 172.21.2.112:0/1053816005 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8da40186c0 0x7f8da401abc0 crc :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:09.862 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.863 7f8dc4b3a700 1 -- 172.21.2.112:0/1053816005 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8dc00fe1c0 con 0x7f8dc009bcc0 2021-02-10T11:43:09.865 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:09.865 7f8db77fe700 1 -- 172.21.2.112:0/1053816005 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8da8023690 con 0x7f8dc009bcc0 2021-02-10T11:43:10.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:10.000 7f8dc4b3a700 1 -- 172.21.2.112:0/1053816005 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"} v 0) v1 -- 0x7f8dc00fe1c0 con 0x7f8dc009bcc0 2021-02-10T11:43:10.193 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:10.193 7f8db77fe700 1 -- 172.21.2.112:0/1053816005 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"}]=0 v114) v1 ==== 118+0+0 (secure 0 0 0) 0x7f8da8023690 con 0x7f8dc009bcc0 2021-02-10T11:43:10.218 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:10.217 7f8dc4b3a700 1 -- 172.21.2.112:0/1053816005 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8da40186c0 msgr2=0x7f8da401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:10.218 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:10.217 7f8dc4b3a700 1 --2- 172.21.2.112:0/1053816005 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8da40186c0 0x7f8da401abc0 crc :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:10.218 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:10.217 7f8dc4b3a700 1 -- 172.21.2.112:0/1053816005 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8dc009bcc0 msgr2=0x7f8dc009c0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:10.218 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:10.217 7f8dc4b3a700 1 --2- 172.21.2.112:0/1053816005 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8dc009bcc0 0x7f8dc009c0c0 secure :-1 s=READY pgs=2432 cs=0 l=1 rev1=1 rx=0x7f8da800b560 tx=0x7f8da800b900).stop 2021-02-10T11:43:10.219 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:10.217 7f8dc4b3a700 1 -- 172.21.2.112:0/1053816005 shutdown_connections 2021-02-10T11:43:10.219 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:10.217 7f8dc4b3a700 1 --2- 172.21.2.112:0/1053816005 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8da40186c0 0x7f8da401abc0 unknown :-1 s=CLOSED pgs=1111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:10.219 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:10.217 7f8dc4b3a700 1 --2- 172.21.2.112:0/1053816005 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8dc00a0820 0x7f8dc00a0c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:10.219 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:10.217 7f8dc4b3a700 1 --2- 172.21.2.112:0/1053816005 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8dc00a1240 0x7f8dc00858d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:10.219 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:10.217 7f8dc4b3a700 1 --2- 172.21.2.112:0/1053816005 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8dc009bcc0 0x7f8dc009c0c0 unknown :-1 s=CLOSED pgs=2432 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:10.219 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:10.217 7f8dc4b3a700 1 -- 172.21.2.112:0/1053816005 >> 172.21.2.112:0/1053816005 conn(0x7f8dc00bef00 msgr2=0x7f8dc009d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:10.220 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:10.217 7f8dc4b3a700 1 -- 172.21.2.112:0/1053816005 shutdown_connections 2021-02-10T11:43:10.220 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:10.218 7f8dc4b3a700 1 -- 172.21.2.112:0/1053816005 wait complete. 2021-02-10T11:43:10.226 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:43:10.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:10.635 7f15f37bc700 1 Processor -- start 2021-02-10T11:43:10.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:10.636 7f15f37bc700 1 -- start start 2021-02-10T11:43:10.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:10.636 7f15f37bc700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f15ec0d0340 0x7f15ec0ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:10.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:10.636 7f15f37bc700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f15ec0d1c70 0x7f15ec0efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:10.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:10.636 7f15f37bc700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15ec0f0b10 con 0x7f15ec0d1870 2021-02-10T11:43:10.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:10.636 7f15f37bc700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15ec0e5ed0 con 0x7f15ec0d0340 2021-02-10T11:43:10.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:10.636 7f15f37bc700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15ec1b0be0 con 0x7f15ec0d1c70 2021-02-10T11:43:10.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:10.636 7f15f0dd7700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f15ec0d0340 0x7f15ec0ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:10.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:10.636 7f15f15d8700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f15ec0d1870 0x7f15ec1b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:56332/0 (socket says 172.21.2.112:56332) 2021-02-10T11:43:10.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:10.636 7f15f0dd7700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f15ec0d0340 0x7f15ec0ef640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:56870/0 (socket says 172.21.2.112:56870) 2021-02-10T11:43:10.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:10.636 7f15f15d8700 1 -- 172.21.2.112:0/1093764176 learned_addr learned my addr 172.21.2.112:0/1093764176 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:43:10.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:10.636 7f15f1dd9700 1 --2- 172.21.2.112:0/1093764176 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f15ec0d1c70 0x7f15ec0efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:10.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:10.637 7f15f0dd7700 1 -- 172.21.2.112:0/1093764176 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f15ec0d1c70 msgr2=0x7f15ec0efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:10.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:10.637 7f15f0dd7700 1 --2- 172.21.2.112:0/1093764176 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f15ec0d1c70 0x7f15ec0efc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:10.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:10.637 7f15f0dd7700 1 -- 172.21.2.112:0/1093764176 >> v1:172.21.2.112:6789/0 conn(0x7f15ec0d1870 legacy=0x7f15ec1b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:10.657 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:10.637 7f15f0dd7700 1 -- 172.21.2.112:0/1093764176 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15ec0f20a0 con 0x7f15ec0d0340 2021-02-10T11:43:10.657 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:10.637 7f15f0dd7700 1 --2- 172.21.2.112:0/1093764176 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f15ec0d0340 0x7f15ec0ef640 secure :-1 s=READY pgs=2434 cs=0 l=1 rev1=1 rx=0x7f15dc006650 tx=0x7f15dc009a80).ready entity=mon.0 client_cookie=99bea328fe9d59b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:10.657 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:10.637 7f15e3fff700 1 -- 172.21.2.112:0/1093764176 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f15dc00c0c0 con 0x7f15ec0d0340 2021-02-10T11:43:10.657 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:10.637 7f15e3fff700 1 -- 172.21.2.112:0/1093764176 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15dc00c340 con 0x7f15ec0d0340 2021-02-10T11:43:10.657 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:10.637 7f15e3fff700 1 -- 172.21.2.112:0/1093764176 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f15dc00c860 con 0x7f15ec0d0340 2021-02-10T11:43:10.658 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:10.637 7f15f37bc700 1 -- 172.21.2.112:0/1093764176 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f15ec0d0340 msgr2=0x7f15ec0ef640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:10.658 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:10.637 7f15f37bc700 1 --2- 172.21.2.112:0/1093764176 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f15ec0d0340 0x7f15ec0ef640 secure :-1 s=READY pgs=2434 cs=0 l=1 rev1=1 rx=0x7f15dc006650 tx=0x7f15dc009a80).stop 2021-02-10T11:43:10.658 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:10.638 7f15f37bc700 1 -- 172.21.2.112:0/1093764176 shutdown_connections 2021-02-10T11:43:10.658 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:10.638 7f15f37bc700 1 --2- 172.21.2.112:0/1093764176 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f15ec0d1c70 0x7f15ec0efc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:10.658 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:10.638 7f15f37bc700 1 --2- 172.21.2.112:0/1093764176 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f15ec0d0340 0x7f15ec0ef640 unknown :-1 s=CLOSED pgs=2434 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:10.658 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:10.638 7f15f37bc700 1 -- 172.21.2.112:0/1093764176 >> 172.21.2.112:0/1093764176 conn(0x7f15ec0ce450 msgr2=0x7f15ec0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:10.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:10.638 7f15f37bc700 1 -- 172.21.2.112:0/1093764176 shutdown_connections 2021-02-10T11:43:10.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:10.638 7f15f37bc700 1 -- 172.21.2.112:0/1093764176 wait complete. 2021-02-10T11:43:10.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:10.653 7f15f37bc700 1 Processor -- start 2021-02-10T11:43:10.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:10.653 7f15f37bc700 1 -- start start 2021-02-10T11:43:10.662 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:10.653 7f15f37bc700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f15ec094040 0x7f15ec093370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:10.662 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:10.653 7f15f37bc700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f15ec093990 0x7f15ec0ad700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:10.662 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:10.653 7f15f37bc700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f15ec0add20 0x7f15ec1a2770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:10.663 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:10.653 7f15f37bc700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15ec093d90 con 0x7f15ec0add20 2021-02-10T11:43:10.663 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:10.653 7f15f37bc700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15ec1a1e50 con 0x7f15ec094040 2021-02-10T11:43:10.663 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:10.653 7f15f37bc700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15ec094590 con 0x7f15ec093990 2021-02-10T11:43:10.663 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:10.654 7f15f1dd9700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f15ec0add20 0x7f15ec1a2770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:10.663 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:10.654 7f15f1dd9700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f15ec0add20 0x7f15ec1a2770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:56876/0 (socket says 172.21.2.112:56876) 2021-02-10T11:43:10.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:10.654 7f15f1dd9700 1 -- 172.21.2.112:0/1790654381 learned_addr learned my addr 172.21.2.112:0/1790654381 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:43:10.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:10.654 7f15f0dd7700 1 --2- 172.21.2.112:0/1790654381 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f15ec093990 0x7f15ec0ad700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:10.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:10.654 7f15f15d8700 1 --2- 172.21.2.112:0/1790654381 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f15ec094040 0x7f15ec093370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:10.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:10.654 7f15f1dd9700 1 -- 172.21.2.112:0/1790654381 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f15ec093990 msgr2=0x7f15ec0ad700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:10.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:10.654 7f15f1dd9700 1 --2- 172.21.2.112:0/1790654381 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f15ec093990 0x7f15ec0ad700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:10.665 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:10.654 7f15f1dd9700 1 -- 172.21.2.112:0/1790654381 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f15ec094040 msgr2=0x7f15ec093370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:10.665 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:10.654 7f15f1dd9700 1 --2- 172.21.2.112:0/1790654381 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f15ec094040 0x7f15ec093370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:10.665 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:10.654 7f15f1dd9700 1 -- 172.21.2.112:0/1790654381 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15ec0ae120 con 0x7f15ec0add20 2021-02-10T11:43:10.665 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:10.654 7f15f1dd9700 1 --2- 172.21.2.112:0/1790654381 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f15ec0add20 0x7f15ec1a2770 secure :-1 s=READY pgs=2435 cs=0 l=1 rev1=1 rx=0x7f15e400b970 tx=0x7f15e400bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:10.665 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:10.654 7f15e1ffb700 1 -- 172.21.2.112:0/1790654381 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f15e4000f90 con 0x7f15ec0add20 2021-02-10T11:43:10.665 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:10.654 7f15e1ffb700 1 -- 172.21.2.112:0/1790654381 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15e4003030 con 0x7f15ec0add20 2021-02-10T11:43:10.666 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:10.654 7f15f37bc700 1 -- 172.21.2.112:0/1790654381 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15ec0ae120 con 0x7f15ec0add20 2021-02-10T11:43:10.666 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:10.654 7f15e1ffb700 1 -- 172.21.2.112:0/1790654381 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f15e4001680 con 0x7f15ec0add20 2021-02-10T11:43:10.666 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:10.654 7f15f37bc700 1 -- 172.21.2.112:0/1790654381 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15ec0ae120 con 0x7f15ec0add20 2021-02-10T11:43:10.666 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:10.655 7f15e1ffb700 1 -- 172.21.2.112:0/1790654381 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f15e4000df0 con 0x7f15ec0add20 2021-02-10T11:43:10.666 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:10.655 7f15e1ffb700 1 --2- 172.21.2.112:0/1790654381 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f15cc018730 0x7f15cc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:10.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:10.655 7f15e1ffb700 1 -- 172.21.2.112:0/1790654381 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f15e40329b0 con 0x7f15ec0add20 2021-02-10T11:43:10.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:10.656 7f15f15d8700 1 --2- 172.21.2.112:0/1790654381 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f15cc018730 0x7f15cc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:10.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:10.656 7f15f15d8700 1 --2- 172.21.2.112:0/1790654381 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f15cc018730 0x7f15cc01abc0 crc :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:10.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:10.686 7f15f37bc700 1 -- 172.21.2.112:0/1790654381 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15ec093d90 con 0x7f15ec0add20 2021-02-10T11:43:10.690 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:10.689 7f15e1ffb700 1 -- 172.21.2.112:0/1790654381 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f15e40066f0 con 0x7f15ec0add20 2021-02-10T11:43:10.818 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:10.816 7f15f37bc700 1 -- 172.21.2.112:0/1790654381 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f15ec093d90 con 0x7f15ec0add20 2021-02-10T11:43:10.818 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:10.817 7f15e1ffb700 1 -- 172.21.2.112:0/1790654381 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 114 v114) v1 ==== 95+0+2108 (secure 0 0 0) 0x7f15e4000df0 con 0x7f15ec0add20 2021-02-10T11:43:10.819 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:43:10.819 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":114,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11349,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/2093257968","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":2093257968},{"type":"v1","addr":"172.21.2.112:6833","nonce":2093257968}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":110}],"filesystems":[{"mdsmap":{"epoch":114,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:43:08.182625","modified":"2021-02-10 11:43:10.189979","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7025},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7025":{"gid":7025,"name":"b","rank":0,"incarnation":111,"state":"up:active","state_seq":3,"addr":"172.21.2.120:6835/2552826693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":2552826693},{"type":"v1","addr":"172.21.2.120:6835","nonce":2552826693}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":14}]} 2021-02-10T11:43:10.819 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 114 2021-02-10T11:43:10.821 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:10.819 7f15f37bc700 1 -- 172.21.2.112:0/1790654381 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f15cc018730 msgr2=0x7f15cc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:10.821 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:10.819 7f15f37bc700 1 --2- 172.21.2.112:0/1790654381 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f15cc018730 0x7f15cc01abc0 crc :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:10.821 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:10.819 7f15f37bc700 1 -- 172.21.2.112:0/1790654381 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f15ec0add20 msgr2=0x7f15ec1a2770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:10.821 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:10.819 7f15f37bc700 1 --2- 172.21.2.112:0/1790654381 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f15ec0add20 0x7f15ec1a2770 secure :-1 s=READY pgs=2435 cs=0 l=1 rev1=1 rx=0x7f15e400b970 tx=0x7f15e400bd10).stop 2021-02-10T11:43:10.821 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:10.819 7f15f37bc700 1 -- 172.21.2.112:0/1790654381 shutdown_connections 2021-02-10T11:43:10.822 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:10.819 7f15f37bc700 1 --2- 172.21.2.112:0/1790654381 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f15cc018730 0x7f15cc01abc0 unknown :-1 s=CLOSED pgs=1112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:10.822 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:10.819 7f15f37bc700 1 --2- 172.21.2.112:0/1790654381 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f15ec0add20 0x7f15ec1a2770 unknown :-1 s=CLOSED pgs=2435 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:10.822 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:10.819 7f15f37bc700 1 --2- 172.21.2.112:0/1790654381 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f15ec093990 0x7f15ec0ad700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:10.822 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:10.819 7f15f37bc700 1 --2- 172.21.2.112:0/1790654381 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f15ec094040 0x7f15ec093370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:10.822 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:10.819 7f15f37bc700 1 -- 172.21.2.112:0/1790654381 >> 172.21.2.112:0/1790654381 conn(0x7f15ec19f8a0 msgr2=0x7f15ec094980 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:10.822 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:10.819 7f15f37bc700 1 -- 172.21.2.112:0/1790654381 shutdown_connections 2021-02-10T11:43:10.823 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:10.819 7f15f37bc700 1 -- 172.21.2.112:0/1790654381 wait complete. 2021-02-10T11:43:10.830 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:43:11.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.241 7f6e626ae700 1 Processor -- start 2021-02-10T11:43:11.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.242 7f6e626ae700 1 -- start start 2021-02-10T11:43:11.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.242 7f6e626ae700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f6e5c0d1870 0x7f6e5c1b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:11.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.242 7f6e626ae700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6e5c1b0a10 0x7f6e5c0cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:11.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.242 7f6e626ae700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e5c0f0b10 con 0x7f6e5c0d1c70 2021-02-10T11:43:11.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.242 7f6e626ae700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e5c0e5ed0 con 0x7f6e5c0d1870 2021-02-10T11:43:11.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.242 7f6e626ae700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e5c0f0650 con 0x7f6e5c1b0a10 2021-02-10T11:43:11.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.242 7f6e5bfff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f6e5c0d1870 0x7f6e5c1b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:11.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.242 7f6e5bfff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f6e5c0d1870 0x7f6e5c1b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:56882/0 (socket says 172.21.2.112:56882) 2021-02-10T11:43:11.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.242 7f6e5bfff700 1 -- 172.21.2.112:0/2213639258 learned_addr learned my addr 172.21.2.112:0/2213639258 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:43:11.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.242 7f6e5b7fe700 1 --2- 172.21.2.112:0/2213639258 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6e5c1b0a10 0x7f6e5c0cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:11.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.243 7f6e5bfff700 1 -- 172.21.2.112:0/2213639258 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6e5c1b0a10 msgr2=0x7f6e5c0cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:11.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.243 7f6e5bfff700 1 --2- 172.21.2.112:0/2213639258 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6e5c1b0a10 0x7f6e5c0cf330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:11.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.243 7f6e5bfff700 1 -- 172.21.2.112:0/2213639258 >> v1:172.21.2.120:6789/0 conn(0x7f6e5c0d1c70 legacy=0x7f6e5c0cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:11.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.243 7f6e5bfff700 1 -- 172.21.2.112:0/2213639258 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e5c0f0870 con 0x7f6e5c0d1870 2021-02-10T11:43:11.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.243 7f6e5bfff700 1 --2- 172.21.2.112:0/2213639258 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6e5c0d1870 0x7f6e5c1b03f0 secure :-1 s=READY pgs=2436 cs=0 l=1 rev1=1 rx=0x7f6e4c006700 tx=0x7f6e4c009b80).ready entity=mon.0 client_cookie=d6a5e0196da34089 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:11.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.243 7f6e5affd700 1 -- 172.21.2.112:0/2213639258 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6e4c00c0c0 con 0x7f6e5c0d1870 2021-02-10T11:43:11.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.243 7f6e5affd700 1 -- 172.21.2.112:0/2213639258 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e4c00c340 con 0x7f6e5c0d1870 2021-02-10T11:43:11.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.243 7f6e5affd700 1 -- 172.21.2.112:0/2213639258 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6e4c00c860 con 0x7f6e5c0d1870 2021-02-10T11:43:11.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.243 7f6e626ae700 1 -- 172.21.2.112:0/2213639258 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6e5c0d1870 msgr2=0x7f6e5c1b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:11.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.243 7f6e626ae700 1 --2- 172.21.2.112:0/2213639258 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6e5c0d1870 0x7f6e5c1b03f0 secure :-1 s=READY pgs=2436 cs=0 l=1 rev1=1 rx=0x7f6e4c006700 tx=0x7f6e4c009b80).stop 2021-02-10T11:43:11.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.244 7f6e626ae700 1 -- 172.21.2.112:0/2213639258 shutdown_connections 2021-02-10T11:43:11.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.244 7f6e626ae700 1 --2- 172.21.2.112:0/2213639258 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6e5c1b0a10 0x7f6e5c0cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:11.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.244 7f6e626ae700 1 --2- 172.21.2.112:0/2213639258 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6e5c0d1870 0x7f6e5c1b03f0 unknown :-1 s=CLOSED pgs=2436 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:11.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.244 7f6e626ae700 1 -- 172.21.2.112:0/2213639258 >> 172.21.2.112:0/2213639258 conn(0x7f6e5c0ce450 msgr2=0x7f6e5c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:11.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.244 7f6e626ae700 1 -- 172.21.2.112:0/2213639258 shutdown_connections 2021-02-10T11:43:11.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.244 7f6e626ae700 1 -- 172.21.2.112:0/2213639258 wait complete. 2021-02-10T11:43:11.262 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.258 7f6e626ae700 1 Processor -- start 2021-02-10T11:43:11.262 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.258 7f6e626ae700 1 -- start start 2021-02-10T11:43:11.262 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.259 7f6e626ae700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6e5c195650 0x7f6e5c0b5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:11.262 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.259 7f6e626ae700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6e5c0b5b80 0x7f6e5c1a2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:11.262 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.259 7f6e626ae700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6e5c0acfa0 0x7f6e5c1a3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:11.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.259 7f6e626ae700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e5c0b6010 con 0x7f6e5c195650 2021-02-10T11:43:11.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.259 7f6e626ae700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e5c1a3a60 con 0x7f6e5c0b5b80 2021-02-10T11:43:11.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.259 7f6e626ae700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e5c1a3c40 con 0x7f6e5c0acfa0 2021-02-10T11:43:11.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.259 7f6e5bfff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6e5c195650 0x7f6e5c0b5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:11.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.259 7f6e5bfff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6e5c195650 0x7f6e5c0b5560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:56886/0 (socket says 172.21.2.112:56886) 2021-02-10T11:43:11.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.259 7f6e5bfff700 1 -- 172.21.2.112:0/924673079 learned_addr learned my addr 172.21.2.112:0/924673079 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:43:11.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.259 7f6e5bfff700 1 -- 172.21.2.112:0/924673079 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6e5c0acfa0 msgr2=0x7f6e5c1a3440 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:43:11.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.259 7f6e5bfff700 1 --2- 172.21.2.112:0/924673079 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6e5c0acfa0 0x7f6e5c1a3440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:11.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.259 7f6e5bfff700 1 -- 172.21.2.112:0/924673079 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6e5c0b5b80 msgr2=0x7f6e5c1a2e20 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:43:11.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.259 7f6e5bfff700 1 --2- 172.21.2.112:0/924673079 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6e5c0b5b80 0x7f6e5c1a2e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:11.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.259 7f6e5bfff700 1 -- 172.21.2.112:0/924673079 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e5c0bcae0 con 0x7f6e5c195650 2021-02-10T11:43:11.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.259 7f6e5bfff700 1 --2- 172.21.2.112:0/924673079 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6e5c195650 0x7f6e5c0b5560 secure :-1 s=READY pgs=2437 cs=0 l=1 rev1=1 rx=0x7f6e4c00b5e0 tx=0x7f6e4c00b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:11.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.259 7f6e58ff9700 1 -- 172.21.2.112:0/924673079 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6e4c00bd70 con 0x7f6e5c195650 2021-02-10T11:43:11.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.259 7f6e58ff9700 1 -- 172.21.2.112:0/924673079 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e4c002030 con 0x7f6e5c195650 2021-02-10T11:43:11.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.259 7f6e626ae700 1 -- 172.21.2.112:0/924673079 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e5c0bcae0 con 0x7f6e5c195650 2021-02-10T11:43:11.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.259 7f6e58ff9700 1 -- 172.21.2.112:0/924673079 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6e4c002550 con 0x7f6e5c195650 2021-02-10T11:43:11.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.260 7f6e626ae700 1 -- 172.21.2.112:0/924673079 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e5c0bd070 con 0x7f6e5c195650 2021-02-10T11:43:11.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.260 7f6e58ff9700 1 -- 172.21.2.112:0/924673079 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6e4c002210 con 0x7f6e5c195650 2021-02-10T11:43:11.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.260 7f6e58ff9700 1 --2- 172.21.2.112:0/924673079 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6e3c0186c0 0x7f6e3c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:11.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.260 7f6e58ff9700 1 -- 172.21.2.112:0/924673079 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f6e4c0323f0 con 0x7f6e5c195650 2021-02-10T11:43:11.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.261 7f6e5b7fe700 1 --2- 172.21.2.112:0/924673079 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6e3c0186c0 0x7f6e3c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:11.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.261 7f6e5b7fe700 1 --2- 172.21.2.112:0/924673079 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6e3c0186c0 0x7f6e3c01abc0 crc :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:11.293 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.292 7f6e626ae700 1 -- 172.21.2.112:0/924673079 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e5c0b6010 con 0x7f6e5c195650 2021-02-10T11:43:11.295 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.294 7f6e58ff9700 1 -- 172.21.2.112:0/924673079 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6e4c030d20 con 0x7f6e5c195650 2021-02-10T11:43:11.421 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.419 7f6e626ae700 1 -- 172.21.2.112:0/924673079 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6e5c0b6010 con 0x7f6e5c195650 2021-02-10T11:43:11.421 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.420 7f6e58ff9700 1 -- 172.21.2.112:0/924673079 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v134) v1 ==== 74+0+14007 (secure 0 0 0) 0x7f6e4c0021f0 con 0x7f6e5c195650 2021-02-10T11:43:11.422 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:43:11.422 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":134,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:43:09.185424","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":28,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:43:06.157376","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"133","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":29,"pool_name":"cephfs_data","create_time":"2021-02-10 11:43:07.178112","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"134","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":132,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":132,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":132,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":132,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:43:11.422 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.422 7f6e626ae700 1 -- 172.21.2.112:0/924673079 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6e3c0186c0 msgr2=0x7f6e3c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:11.422 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.422 7f6e626ae700 1 --2- 172.21.2.112:0/924673079 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6e3c0186c0 0x7f6e3c01abc0 crc :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:11.422 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.422 7f6e626ae700 1 -- 172.21.2.112:0/924673079 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6e5c195650 msgr2=0x7f6e5c0b5560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:11.423 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.422 7f6e626ae700 1 --2- 172.21.2.112:0/924673079 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6e5c195650 0x7f6e5c0b5560 secure :-1 s=READY pgs=2437 cs=0 l=1 rev1=1 rx=0x7f6e4c00b5e0 tx=0x7f6e4c00b980).stop 2021-02-10T11:43:11.423 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.422 7f6e626ae700 1 -- 172.21.2.112:0/924673079 shutdown_connections 2021-02-10T11:43:11.423 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.422 7f6e626ae700 1 --2- 172.21.2.112:0/924673079 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6e3c0186c0 0x7f6e3c01abc0 unknown :-1 s=CLOSED pgs=1113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:11.423 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.422 7f6e626ae700 1 --2- 172.21.2.112:0/924673079 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6e5c0b5b80 0x7f6e5c1a2e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:11.423 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.422 7f6e626ae700 1 --2- 172.21.2.112:0/924673079 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6e5c0acfa0 0x7f6e5c1a3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:11.423 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.422 7f6e626ae700 1 --2- 172.21.2.112:0/924673079 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6e5c195650 0x7f6e5c0b5560 unknown :-1 s=CLOSED pgs=2437 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:11.423 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.422 7f6e626ae700 1 -- 172.21.2.112:0/924673079 >> 172.21.2.112:0/924673079 conn(0x7f6e5c0b3040 msgr2=0x7f6e5c0c3250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:11.423 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.422 7f6e626ae700 1 -- 172.21.2.112:0/924673079 shutdown_connections 2021-02-10T11:43:11.423 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.422 7f6e626ae700 1 -- 172.21.2.112:0/924673079 wait complete. 2021-02-10T11:43:11.431 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:43:11.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.855 7facedf62700 1 Processor -- start 2021-02-10T11:43:11.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.855 7facedf62700 1 -- start start 2021-02-10T11:43:11.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.855 7facedf62700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7face80d1860 0x7face81b04b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:11.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.855 7facedf62700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7face81b0ad0 0x7face8108930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:11.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.855 7facedf62700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7face80d1c60 0x7face8108f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:11.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.855 7facedf62700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7face80fe1b0 con 0x7face81b0ad0 2021-02-10T11:43:11.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.855 7facedf62700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7face8109a50 con 0x7face80d1860 2021-02-10T11:43:11.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.855 7facedf62700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7face8088070 con 0x7face80d1c60 2021-02-10T11:43:11.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.855 7face6ffd700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7face81b0ad0 0x7face8108930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:11.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.855 7face6ffd700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7face81b0ad0 0x7face8108930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:56894/0 (socket says 172.21.2.112:56894) 2021-02-10T11:43:11.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.855 7face6ffd700 1 -- 172.21.2.112:0/2957135473 learned_addr learned my addr 172.21.2.112:0/2957135473 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:43:11.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.855 7face6ffd700 1 -- 172.21.2.112:0/2957135473 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7face80d1c60 msgr2=0x7face8108f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:11.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.855 7face6ffd700 1 --2- 172.21.2.112:0/2957135473 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7face80d1c60 0x7face8108f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:11.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.855 7face6ffd700 1 -- 172.21.2.112:0/2957135473 >> v2:172.21.2.120:3300/0 conn(0x7face80d1860 msgr2=0x7face81b04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:11.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.855 7face6ffd700 1 --2- 172.21.2.112:0/2957135473 >> v2:172.21.2.120:3300/0 conn(0x7face80d1860 0x7face81b04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:11.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.855 7face6ffd700 1 -- 172.21.2.112:0/2957135473 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7face8109570 con 0x7face81b0ad0 2021-02-10T11:43:11.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.856 7face6ffd700 1 --2- 172.21.2.112:0/2957135473 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7face81b0ad0 0x7face8108930 secure :-1 s=READY pgs=2438 cs=0 l=1 rev1=1 rx=0x7facd0006700 tx=0x7facd0009b00).ready entity=mon.0 client_cookie=9364d66bb4e1a1f4 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:11.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.856 7face67fc700 1 -- 172.21.2.112:0/2957135473 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7facd000c0c0 con 0x7face81b0ad0 2021-02-10T11:43:11.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.856 7face67fc700 1 -- 172.21.2.112:0/2957135473 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facd000c400 con 0x7face81b0ad0 2021-02-10T11:43:11.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.856 7face67fc700 1 -- 172.21.2.112:0/2957135473 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7facd000c920 con 0x7face81b0ad0 2021-02-10T11:43:11.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.856 7facedf62700 1 -- 172.21.2.112:0/2957135473 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7face81b0ad0 msgr2=0x7face8108930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:11.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.856 7facedf62700 1 --2- 172.21.2.112:0/2957135473 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7face81b0ad0 0x7face8108930 secure :-1 s=READY pgs=2438 cs=0 l=1 rev1=1 rx=0x7facd0006700 tx=0x7facd0009b00).stop 2021-02-10T11:43:11.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.856 7facedf62700 1 -- 172.21.2.112:0/2957135473 shutdown_connections 2021-02-10T11:43:11.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.856 7facedf62700 1 --2- 172.21.2.112:0/2957135473 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7face80d1c60 0x7face8108f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:11.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.856 7facedf62700 1 --2- 172.21.2.112:0/2957135473 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7face81b0ad0 0x7face8108930 unknown :-1 s=CLOSED pgs=2438 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:11.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.856 7facedf62700 1 --2- 172.21.2.112:0/2957135473 >> v2:172.21.2.120:3300/0 conn(0x7face80d1860 0x7face81b04b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:11.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.856 7facedf62700 1 -- 172.21.2.112:0/2957135473 >> 172.21.2.112:0/2957135473 conn(0x7face80ce410 msgr2=0x7face80d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:11.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.857 7facedf62700 1 -- 172.21.2.112:0/2957135473 shutdown_connections 2021-02-10T11:43:11.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.857 7facedf62700 1 -- 172.21.2.112:0/2957135473 wait complete. 2021-02-10T11:43:11.876 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.871 7facedf62700 1 Processor -- start 2021-02-10T11:43:11.876 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.872 7facedf62700 1 -- start start 2021-02-10T11:43:11.876 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.872 7facedf62700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7face8099b10 0x7face80ae7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:11.876 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.872 7facedf62700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7face80aee00 0x7face819b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:11.876 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.872 7facedf62700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7face819bd10 0x7face819c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:11.876 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.872 7facedf62700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7face819c1f0 con 0x7face8099b10 2021-02-10T11:43:11.876 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.872 7facedf62700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7face809a4d0 con 0x7face80aee00 2021-02-10T11:43:11.876 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.872 7facedf62700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7face81b9520 con 0x7face819bd10 2021-02-10T11:43:11.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.872 7face77fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7face8099b10 0x7face80ae7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:11.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.872 7face77fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7face8099b10 0x7face80ae7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:56900/0 (socket says 172.21.2.112:56900) 2021-02-10T11:43:11.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.872 7face77fe700 1 -- 172.21.2.112:0/3781379033 learned_addr learned my addr 172.21.2.112:0/3781379033 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:43:11.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.872 7face6ffd700 1 --2- 172.21.2.112:0/3781379033 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7face80aee00 0x7face819b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:11.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.872 7face7fff700 1 --2- 172.21.2.112:0/3781379033 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7face819bd10 0x7face819c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:11.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.873 7face77fe700 1 -- 172.21.2.112:0/3781379033 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7face819bd10 msgr2=0x7face819c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:11.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.873 7face77fe700 1 --2- 172.21.2.112:0/3781379033 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7face819bd10 0x7face819c6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:11.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.873 7face77fe700 1 -- 172.21.2.112:0/3781379033 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7face80aee00 msgr2=0x7face819b6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:11.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.873 7face77fe700 1 --2- 172.21.2.112:0/3781379033 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7face80aee00 0x7face819b6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:11.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.873 7face77fe700 1 -- 172.21.2.112:0/3781379033 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7face8099f10 con 0x7face8099b10 2021-02-10T11:43:11.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.873 7face6ffd700 1 --2- 172.21.2.112:0/3781379033 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7face80aee00 0x7face819b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:43:11.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.873 7face77fe700 1 --2- 172.21.2.112:0/3781379033 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7face8099b10 0x7face80ae7e0 secure :-1 s=READY pgs=2439 cs=0 l=1 rev1=1 rx=0x7facd8006f00 tx=0x7facd800b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:11.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.873 7facdffff700 1 -- 172.21.2.112:0/3781379033 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7facd800bd30 con 0x7face8099b10 2021-02-10T11:43:11.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.873 7facdffff700 1 -- 172.21.2.112:0/3781379033 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facd8000c90 con 0x7face8099b10 2021-02-10T11:43:11.878 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.873 7facedf62700 1 -- 172.21.2.112:0/3781379033 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7face8099f10 con 0x7face8099b10 2021-02-10T11:43:11.878 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.873 7facdffff700 1 -- 172.21.2.112:0/3781379033 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7facd80022a0 con 0x7face8099b10 2021-02-10T11:43:11.878 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.873 7facedf62700 1 -- 172.21.2.112:0/3781379033 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7face819d000 con 0x7face8099b10 2021-02-10T11:43:11.878 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.874 7facdffff700 1 -- 172.21.2.112:0/3781379033 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7facd8005310 con 0x7face8099b10 2021-02-10T11:43:11.878 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.874 7facdffff700 1 --2- 172.21.2.112:0/3781379033 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7faccc018730 0x7faccc01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:11.878 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.874 7facdffff700 1 -- 172.21.2.112:0/3781379033 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5573+0+0 (secure 0 0 0) 0x7facd80325f0 con 0x7face8099b10 2021-02-10T11:43:11.878 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.874 7face6ffd700 1 --2- 172.21.2.112:0/3781379033 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7faccc018730 0x7faccc01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:11.878 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.874 7face6ffd700 1 --2- 172.21.2.112:0/3781379033 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7faccc018730 0x7faccc01ac10 crc :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:11.905 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.905 7facedf62700 1 -- 172.21.2.112:0/3781379033 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7face8082980 con 0x7face8099b10 2021-02-10T11:43:11.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:11.907 7facdffff700 1 -- 172.21.2.112:0/3781379033 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7facd8006b90 con 0x7face8099b10 2021-02-10T11:43:12.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:12.039 7facedf62700 1 -- 172.21.2.112:0/3781379033 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7face8082980 con 0x7face8099b10 2021-02-10T11:43:12.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:12.042 7facdffff700 1 -- 172.21.2.112:0/3781379033 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v79) v1 ==== 184+0+0 (secure 0 0 0) 0x7facd8005310 con 0x7face8099b10 2021-02-10T11:43:12.047 INFO:teuthology.orchestra.run.gibba012.stderr:updated caps for client.0 2021-02-10T11:43:12.049 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:12.048 7facedf62700 1 -- 172.21.2.112:0/3781379033 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7faccc018730 msgr2=0x7faccc01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:12.049 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:12.048 7facedf62700 1 --2- 172.21.2.112:0/3781379033 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7faccc018730 0x7faccc01ac10 crc :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:12.049 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:12.048 7facedf62700 1 -- 172.21.2.112:0/3781379033 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7face8099b10 msgr2=0x7face80ae7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:12.049 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:12.048 7facedf62700 1 --2- 172.21.2.112:0/3781379033 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7face8099b10 0x7face80ae7e0 secure :-1 s=READY pgs=2439 cs=0 l=1 rev1=1 rx=0x7facd8006f00 tx=0x7facd800b940).stop 2021-02-10T11:43:12.049 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:12.048 7facedf62700 1 -- 172.21.2.112:0/3781379033 shutdown_connections 2021-02-10T11:43:12.049 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:12.048 7facedf62700 1 --2- 172.21.2.112:0/3781379033 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7faccc018730 0x7faccc01ac10 unknown :-1 s=CLOSED pgs=1114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:12.049 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:12.048 7facedf62700 1 --2- 172.21.2.112:0/3781379033 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7face80aee00 0x7face819b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:12.049 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:12.048 7facedf62700 1 --2- 172.21.2.112:0/3781379033 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7face819bd10 0x7face819c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:12.049 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:12.048 7facedf62700 1 --2- 172.21.2.112:0/3781379033 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7face8099b10 0x7face80ae7e0 unknown :-1 s=CLOSED pgs=2439 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:12.050 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:12.048 7facedf62700 1 -- 172.21.2.112:0/3781379033 >> 172.21.2.112:0/3781379033 conn(0x7face819f820 msgr2=0x7face818c270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:12.050 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:12.048 7facedf62700 1 -- 172.21.2.112:0/3781379033 shutdown_connections 2021-02-10T11:43:12.050 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:12.048 7facedf62700 1 -- 172.21.2.112:0/3781379033 wait complete. 2021-02-10T11:43:12.058 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:43:12.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:12.485 7fa143b9c700 1 Processor -- start 2021-02-10T11:43:12.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:12.485 7fa143b9c700 1 -- start start 2021-02-10T11:43:12.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:12.486 7fa143b9c700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa13c0d1c60 0x7fa13c0cef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:12.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:12.486 7fa143b9c700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fa13c0cf570 0x7fa13c0cf970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:12.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:12.486 7fa143b9c700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa13c0fe1b0 con 0x7fa13c0d1860 2021-02-10T11:43:12.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:12.486 7fa143b9c700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa13c1b0ca0 con 0x7fa13c0cf570 2021-02-10T11:43:12.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:12.486 7fa143b9c700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa13c109a50 con 0x7fa13c0d1c60 2021-02-10T11:43:12.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:12.486 7fa1421b9700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fa13c0cf570 0x7fa13c0cf970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:12.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:12.486 7fa1421b9700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fa13c0cf570 0x7fa13c0cf970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:56910/0 (socket says 172.21.2.112:56910) 2021-02-10T11:43:12.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:12.486 7fa1421b9700 1 -- 172.21.2.112:0/1567912532 learned_addr learned my addr 172.21.2.112:0/1567912532 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:43:12.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:12.486 7fa1411b7700 1 --2- 172.21.2.112:0/1567912532 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa13c0d1c60 0x7fa13c0cef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:12.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:12.486 7fa1411b7700 1 -- 172.21.2.112:0/1567912532 >> v2:172.21.2.112:3300/0 conn(0x7fa13c0cf570 msgr2=0x7fa13c0cf970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:12.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:12.486 7fa1411b7700 1 --2- 172.21.2.112:0/1567912532 >> v2:172.21.2.112:3300/0 conn(0x7fa13c0cf570 0x7fa13c0cf970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:12.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:12.486 7fa1411b7700 1 -- 172.21.2.112:0/1567912532 >> v1:172.21.2.112:6789/0 conn(0x7fa13c0d1860 legacy=0x7fa13c1b04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:12.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:12.486 7fa1411b7700 1 -- 172.21.2.112:0/1567912532 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa13c088070 con 0x7fa13c0d1c60 2021-02-10T11:43:12.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:12.486 7fa1411b7700 1 --2- 172.21.2.112:0/1567912532 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa13c0d1c60 0x7fa13c0cef50 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7fa130006650 tx=0x7fa130009a00).ready entity=mon.2 client_cookie=c2cb2cbe46a611c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:12.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:12.487 7fa1409b6700 1 -- 172.21.2.112:0/1567912532 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa130009df0 con 0x7fa13c0d1c60 2021-02-10T11:43:12.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:12.487 7fa1409b6700 1 -- 172.21.2.112:0/1567912532 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa130006740 con 0x7fa13c0d1c60 2021-02-10T11:43:12.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:12.487 7fa1409b6700 1 -- 172.21.2.112:0/1567912532 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa130009df0 con 0x7fa13c0d1c60 2021-02-10T11:43:12.505 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:12.487 7fa143b9c700 1 -- 172.21.2.112:0/1567912532 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa13c0d1c60 msgr2=0x7fa13c0cef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:12.505 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:12.487 7fa143b9c700 1 --2- 172.21.2.112:0/1567912532 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa13c0d1c60 0x7fa13c0cef50 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7fa130006650 tx=0x7fa130009a00).stop 2021-02-10T11:43:12.505 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:12.487 7fa143b9c700 1 -- 172.21.2.112:0/1567912532 shutdown_connections 2021-02-10T11:43:12.505 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:12.487 7fa143b9c700 1 --2- 172.21.2.112:0/1567912532 >> v2:172.21.2.112:3300/0 conn(0x7fa13c0cf570 0x7fa13c0cf970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:12.505 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:12.487 7fa143b9c700 1 --2- 172.21.2.112:0/1567912532 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa13c0d1c60 0x7fa13c0cef50 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:12.505 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:12.487 7fa143b9c700 1 -- 172.21.2.112:0/1567912532 >> 172.21.2.112:0/1567912532 conn(0x7fa13c0ce410 msgr2=0x7fa13c0d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:12.505 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:12.488 7fa143b9c700 1 -- 172.21.2.112:0/1567912532 shutdown_connections 2021-02-10T11:43:12.505 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:12.488 7fa143b9c700 1 -- 172.21.2.112:0/1567912532 wait complete. 2021-02-10T11:43:12.505 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:12.502 7fa143b9c700 1 Processor -- start 2021-02-10T11:43:12.505 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:12.502 7fa143b9c700 1 -- start start 2021-02-10T11:43:12.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:12.503 7fa143b9c700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa13c0bcee0 0x7fa13c0b5de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:12.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:12.503 7fa143b9c700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa13c09bce0 0x7fa13c1a7620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:12.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:12.503 7fa143b9c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa13c1a7c40 0x7fa13c09c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:12.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:12.503 7fa143b9c700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa13c0b6400 con 0x7fa13c1a7c40 2021-02-10T11:43:12.508 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:12.503 7fa143b9c700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa13c09c0e0 con 0x7fa13c09bce0 2021-02-10T11:43:12.508 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:12.503 7fa143b9c700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa13c09c320 con 0x7fa13c0bcee0 2021-02-10T11:43:12.508 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:12.503 7fa1421b9700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa13c1a7c40 0x7fa13c09c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:12.508 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:12.503 7fa1421b9700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa13c1a7c40 0x7fa13c09c8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:56914/0 (socket says 172.21.2.112:56914) 2021-02-10T11:43:12.508 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:12.503 7fa1421b9700 1 -- 172.21.2.112:0/2106586866 learned_addr learned my addr 172.21.2.112:0/2106586866 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:43:12.508 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:12.503 7fa1419b8700 1 --2- 172.21.2.112:0/2106586866 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa13c0bcee0 0x7fa13c0b5de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:12.508 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:12.503 7fa1411b7700 1 --2- 172.21.2.112:0/2106586866 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa13c09bce0 0x7fa13c1a7620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:12.508 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:12.503 7fa1421b9700 1 -- 172.21.2.112:0/2106586866 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa13c0bcee0 msgr2=0x7fa13c0b5de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:12.508 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:12.503 7fa1421b9700 1 --2- 172.21.2.112:0/2106586866 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa13c0bcee0 0x7fa13c0b5de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:12.508 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:12.503 7fa1421b9700 1 -- 172.21.2.112:0/2106586866 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa13c09bce0 msgr2=0x7fa13c1a7620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:12.508 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:12.503 7fa1421b9700 1 --2- 172.21.2.112:0/2106586866 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa13c09bce0 0x7fa13c1a7620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:12.508 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:12.503 7fa1421b9700 1 -- 172.21.2.112:0/2106586866 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa13c0bd2e0 con 0x7fa13c1a7c40 2021-02-10T11:43:12.509 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:12.503 7fa1411b7700 1 --2- 172.21.2.112:0/2106586866 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa13c09bce0 0x7fa13c1a7620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:43:12.509 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:12.503 7fa1421b9700 1 --2- 172.21.2.112:0/2106586866 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa13c1a7c40 0x7fa13c09c8a0 secure :-1 s=READY pgs=2441 cs=0 l=1 rev1=1 rx=0x7fa13400b8e0 tx=0x7fa13400bc80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:12.509 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:12.503 7fa12e7fc700 1 -- 172.21.2.112:0/2106586866 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa134000f90 con 0x7fa13c1a7c40 2021-02-10T11:43:12.509 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:12.504 7fa12e7fc700 1 -- 172.21.2.112:0/2106586866 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa13400bdc0 con 0x7fa13c1a7c40 2021-02-10T11:43:12.509 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:12.504 7fa12e7fc700 1 -- 172.21.2.112:0/2106586866 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa1340014f0 con 0x7fa13c1a7c40 2021-02-10T11:43:12.509 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:12.504 7fa143b9c700 1 -- 172.21.2.112:0/2106586866 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa13c0bd2e0 con 0x7fa13c1a7c40 2021-02-10T11:43:12.509 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:12.504 7fa143b9c700 1 -- 172.21.2.112:0/2106586866 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa13c0bd2e0 con 0x7fa13c1a7c40 2021-02-10T11:43:12.509 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:12.504 7fa12e7fc700 1 -- 172.21.2.112:0/2106586866 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa134001270 con 0x7fa13c1a7c40 2021-02-10T11:43:12.509 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:12.504 7fa12e7fc700 1 --2- 172.21.2.112:0/2106586866 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa124018730 0x7fa12401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:12.509 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:12.504 7fa12e7fc700 1 -- 172.21.2.112:0/2106586866 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fa1340329b0 con 0x7fa13c1a7c40 2021-02-10T11:43:12.509 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:12.504 7fa1419b8700 1 --2- 172.21.2.112:0/2106586866 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa124018730 0x7fa12401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:12.509 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:12.505 7fa1419b8700 1 --2- 172.21.2.112:0/2106586866 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa124018730 0x7fa12401ac10 crc :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:12.537 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:12.536 7fa143b9c700 1 -- 172.21.2.112:0/2106586866 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa13c09c0e0 con 0x7fa13c1a7c40 2021-02-10T11:43:12.539 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:12.538 7fa12e7fc700 1 -- 172.21.2.112:0/2106586866 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa134000f90 con 0x7fa13c1a7c40 2021-02-10T11:43:12.669 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:12.668 7fa143b9c700 1 -- 172.21.2.112:0/2106586866 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fa13c09c0e0 con 0x7fa13c1a7c40 2021-02-10T11:43:12.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:12.672 7fa12e7fc700 1 -- 172.21.2.112:0/2106586866 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v80) v1 ==== 184+0+0 (secure 0 0 0) 0x7fa134001270 con 0x7fa13c1a7c40 2021-02-10T11:43:12.677 INFO:teuthology.orchestra.run.gibba012.stderr:updated caps for client.1 2021-02-10T11:43:12.678 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:12.677 7fa143b9c700 1 -- 172.21.2.112:0/2106586866 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa124018730 msgr2=0x7fa12401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:12.678 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:12.677 7fa143b9c700 1 --2- 172.21.2.112:0/2106586866 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa124018730 0x7fa12401ac10 crc :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:12.678 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:12.677 7fa143b9c700 1 -- 172.21.2.112:0/2106586866 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa13c1a7c40 msgr2=0x7fa13c09c8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:12.678 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:12.677 7fa143b9c700 1 --2- 172.21.2.112:0/2106586866 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa13c1a7c40 0x7fa13c09c8a0 secure :-1 s=READY pgs=2441 cs=0 l=1 rev1=1 rx=0x7fa13400b8e0 tx=0x7fa13400bc80).stop 2021-02-10T11:43:12.679 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:12.677 7fa143b9c700 1 -- 172.21.2.112:0/2106586866 shutdown_connections 2021-02-10T11:43:12.679 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:12.677 7fa143b9c700 1 --2- 172.21.2.112:0/2106586866 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa124018730 0x7fa12401ac10 unknown :-1 s=CLOSED pgs=1115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:12.679 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:12.677 7fa143b9c700 1 --2- 172.21.2.112:0/2106586866 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa13c09bce0 0x7fa13c1a7620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:12.679 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:12.677 7fa143b9c700 1 --2- 172.21.2.112:0/2106586866 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa13c1a7c40 0x7fa13c09c8a0 unknown :-1 s=CLOSED pgs=2441 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:12.679 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:12.677 7fa143b9c700 1 --2- 172.21.2.112:0/2106586866 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa13c0bcee0 0x7fa13c0b5de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:12.679 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:12.677 7fa143b9c700 1 -- 172.21.2.112:0/2106586866 >> 172.21.2.112:0/2106586866 conn(0x7fa13c09e660 msgr2=0x7fa13c0a9550 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:12.679 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:12.678 7fa143b9c700 1 -- 172.21.2.112:0/2106586866 shutdown_connections 2021-02-10T11:43:12.679 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:12.678 7fa143b9c700 1 -- 172.21.2.112:0/2106586866 wait complete. 2021-02-10T11:43:12.686 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.2 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:43:13.127 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.110 7fe4f73e3700 1 Processor -- start 2021-02-10T11:43:13.127 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.111 7fe4f73e3700 1 -- start start 2021-02-10T11:43:13.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.111 7fe4f73e3700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fe4f00d1880 0x7fe4f01b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:13.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.111 7fe4f73e3700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fe4f01b0af0 0x7fe4f0108950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:13.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.111 7fe4f73e3700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4f00fe1d0 con 0x7fe4f00d1c80 2021-02-10T11:43:13.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.111 7fe4f73e3700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4f0109a70 con 0x7fe4f01b0af0 2021-02-10T11:43:13.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.111 7fe4f73e3700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4f0109760 con 0x7fe4f00d1880 2021-02-10T11:43:13.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.111 7fe4f49fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fe4f01b0af0 0x7fe4f0108950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:13.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.111 7fe4f49fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fe4f01b0af0 0x7fe4f0108950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:56920/0 (socket says 172.21.2.112:56920) 2021-02-10T11:43:13.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.111 7fe4f49fe700 1 -- 172.21.2.112:0/541157983 learned_addr learned my addr 172.21.2.112:0/541157983 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:43:13.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.111 7fe4f49fe700 1 -- 172.21.2.112:0/541157983 >> v2:172.21.2.120:3300/0 conn(0x7fe4f00d1880 msgr2=0x7fe4f01b04d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:43:13.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.111 7fe4f49fe700 1 --2- 172.21.2.112:0/541157983 >> v2:172.21.2.120:3300/0 conn(0x7fe4f00d1880 0x7fe4f01b04d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:13.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.111 7fe4f49fe700 1 -- 172.21.2.112:0/541157983 >> v1:172.21.2.112:6789/0 conn(0x7fe4f00d1c80 legacy=0x7fe4f0108f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:13.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.111 7fe4f49fe700 1 -- 172.21.2.112:0/541157983 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4f0088070 con 0x7fe4f01b0af0 2021-02-10T11:43:13.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.112 7fe4f49fe700 1 --2- 172.21.2.112:0/541157983 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe4f01b0af0 0x7fe4f0108950 secure :-1 s=READY pgs=2443 cs=0 l=1 rev1=1 rx=0x7fe4e0006700 tx=0x7fe4e0009b00).ready entity=mon.0 client_cookie=9d002c78f8d13290 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:13.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.112 7fe4e7fff700 1 -- 172.21.2.112:0/541157983 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe4e000c0c0 con 0x7fe4f01b0af0 2021-02-10T11:43:13.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.112 7fe4e7fff700 1 -- 172.21.2.112:0/541157983 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4e000c340 con 0x7fe4f01b0af0 2021-02-10T11:43:13.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.112 7fe4e7fff700 1 -- 172.21.2.112:0/541157983 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe4e000c820 con 0x7fe4f01b0af0 2021-02-10T11:43:13.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.112 7fe4f73e3700 1 -- 172.21.2.112:0/541157983 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe4f01b0af0 msgr2=0x7fe4f0108950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:13.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.112 7fe4f73e3700 1 --2- 172.21.2.112:0/541157983 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe4f01b0af0 0x7fe4f0108950 secure :-1 s=READY pgs=2443 cs=0 l=1 rev1=1 rx=0x7fe4e0006700 tx=0x7fe4e0009b00).stop 2021-02-10T11:43:13.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.112 7fe4f73e3700 1 -- 172.21.2.112:0/541157983 shutdown_connections 2021-02-10T11:43:13.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.112 7fe4f73e3700 1 --2- 172.21.2.112:0/541157983 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe4f01b0af0 0x7fe4f0108950 unknown :-1 s=CLOSED pgs=2443 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:13.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.112 7fe4f73e3700 1 --2- 172.21.2.112:0/541157983 >> v2:172.21.2.120:3300/0 conn(0x7fe4f00d1880 0x7fe4f01b04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:13.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.112 7fe4f73e3700 1 -- 172.21.2.112:0/541157983 >> 172.21.2.112:0/541157983 conn(0x7fe4f00ce430 msgr2=0x7fe4f00d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:13.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.113 7fe4f73e3700 1 -- 172.21.2.112:0/541157983 shutdown_connections 2021-02-10T11:43:13.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.113 7fe4f73e3700 1 -- 172.21.2.112:0/541157983 wait complete. 2021-02-10T11:43:13.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.127 7fe4f73e3700 1 Processor -- start 2021-02-10T11:43:13.131 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.127 7fe4f73e3700 1 -- start start 2021-02-10T11:43:13.131 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.127 7fe4f73e3700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe4f0086520 0x7fe4f01a1cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:13.131 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.127 7fe4f73e3700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe4f0086920 0x7fe4f0187e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:13.131 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.127 7fe4f73e3700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe4f01a22d0 0x7fe4f01884a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:13.131 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.127 7fe4f73e3700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4f0099600 con 0x7fe4f0086920 2021-02-10T11:43:13.131 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.127 7fe4f73e3700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4f00999f0 con 0x7fe4f01a22d0 2021-02-10T11:43:13.131 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.127 7fe4f73e3700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4f0188ac0 con 0x7fe4f0086520 2021-02-10T11:43:13.131 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.127 7fe4f49fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe4f0086920 0x7fe4f0187e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:13.131 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.128 7fe4f49fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe4f0086920 0x7fe4f0187e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:56928/0 (socket says 172.21.2.112:56928) 2021-02-10T11:43:13.131 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.128 7fe4f49fe700 1 -- 172.21.2.112:0/598673917 learned_addr learned my addr 172.21.2.112:0/598673917 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:43:13.131 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.128 7fe4f49fe700 1 -- 172.21.2.112:0/598673917 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe4f0086520 msgr2=0x7fe4f01a1cb0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:43:13.131 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.128 7fe4f49fe700 1 --2- 172.21.2.112:0/598673917 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe4f0086520 0x7fe4f01a1cb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:13.132 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.128 7fe4f49fe700 1 -- 172.21.2.112:0/598673917 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe4f01a22d0 msgr2=0x7fe4f01884a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:43:13.132 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.128 7fe4f49fe700 1 --2- 172.21.2.112:0/598673917 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe4f01a22d0 0x7fe4f01884a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:13.132 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.128 7fe4f49fe700 1 -- 172.21.2.112:0/598673917 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4f0188ce0 con 0x7fe4f0086920 2021-02-10T11:43:13.132 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.128 7fe4f49fe700 1 --2- 172.21.2.112:0/598673917 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe4f0086920 0x7fe4f0187e80 secure :-1 s=READY pgs=2444 cs=0 l=1 rev1=1 rx=0x7fe4e0006f20 tx=0x7fe4e000b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:13.132 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.128 7fe4e5ffb700 1 -- 172.21.2.112:0/598673917 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe4e0006390 con 0x7fe4f0086920 2021-02-10T11:43:13.132 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.128 7fe4e5ffb700 1 -- 172.21.2.112:0/598673917 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4e000bd80 con 0x7fe4f0086920 2021-02-10T11:43:13.132 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.128 7fe4f73e3700 1 -- 172.21.2.112:0/598673917 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4f0188ce0 con 0x7fe4f0086920 2021-02-10T11:43:13.132 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.128 7fe4f73e3700 1 -- 172.21.2.112:0/598673917 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4f0189270 con 0x7fe4f0086920 2021-02-10T11:43:13.132 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.128 7fe4e5ffb700 1 -- 172.21.2.112:0/598673917 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe4e00054d0 con 0x7fe4f0086920 2021-02-10T11:43:13.132 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.129 7fe4e5ffb700 1 -- 172.21.2.112:0/598673917 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe4e00052e0 con 0x7fe4f0086920 2021-02-10T11:43:13.132 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.129 7fe4e5ffb700 1 --2- 172.21.2.112:0/598673917 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe4d00186c0 0x7fe4d001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:13.132 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.129 7fe4e5ffb700 1 -- 172.21.2.112:0/598673917 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fe4e0032550 con 0x7fe4f0086920 2021-02-10T11:43:13.132 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.129 7fe4f51ff700 1 --2- 172.21.2.112:0/598673917 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe4d00186c0 0x7fe4d001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:13.132 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.130 7fe4f51ff700 1 --2- 172.21.2.112:0/598673917 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe4d00186c0 0x7fe4d001abc0 crc :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:13.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.161 7fe4f73e3700 1 -- 172.21.2.112:0/598673917 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4f00fe1d0 con 0x7fe4f0086920 2021-02-10T11:43:13.164 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.163 7fe4e5ffb700 1 -- 172.21.2.112:0/598673917 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe4e002d220 con 0x7fe4f0086920 2021-02-10T11:43:13.296 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.295 7fe4f73e3700 1 -- 172.21.2.112:0/598673917 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fe4f00999a0 con 0x7fe4f0086920 2021-02-10T11:43:13.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.298 7fe4e5ffb700 1 -- 172.21.2.112:0/598673917 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.2 v81) v1 ==== 184+0+0 (secure 0 0 0) 0x7fe4e00052e0 con 0x7fe4f0086920 2021-02-10T11:43:13.303 INFO:teuthology.orchestra.run.gibba012.stderr:updated caps for client.2 2021-02-10T11:43:13.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.304 7fe4f73e3700 1 -- 172.21.2.112:0/598673917 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe4d00186c0 msgr2=0x7fe4d001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:13.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.304 7fe4f73e3700 1 --2- 172.21.2.112:0/598673917 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe4d00186c0 0x7fe4d001abc0 crc :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:13.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.304 7fe4f73e3700 1 -- 172.21.2.112:0/598673917 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe4f0086920 msgr2=0x7fe4f0187e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:13.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.304 7fe4f73e3700 1 --2- 172.21.2.112:0/598673917 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe4f0086920 0x7fe4f0187e80 secure :-1 s=READY pgs=2444 cs=0 l=1 rev1=1 rx=0x7fe4e0006f20 tx=0x7fe4e000b990).stop 2021-02-10T11:43:13.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.304 7fe4f73e3700 1 -- 172.21.2.112:0/598673917 shutdown_connections 2021-02-10T11:43:13.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.304 7fe4f73e3700 1 --2- 172.21.2.112:0/598673917 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe4d00186c0 0x7fe4d001abc0 unknown :-1 s=CLOSED pgs=1116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:13.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.304 7fe4f73e3700 1 --2- 172.21.2.112:0/598673917 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe4f01a22d0 0x7fe4f01884a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:13.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.304 7fe4f73e3700 1 --2- 172.21.2.112:0/598673917 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe4f0086920 0x7fe4f0187e80 unknown :-1 s=CLOSED pgs=2444 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:13.306 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.304 7fe4f73e3700 1 --2- 172.21.2.112:0/598673917 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe4f0086520 0x7fe4f01a1cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:13.306 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.304 7fe4f73e3700 1 -- 172.21.2.112:0/598673917 >> 172.21.2.112:0/598673917 conn(0x7fe4f019f880 msgr2=0x7fe4f008f890 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:13.306 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.304 7fe4f73e3700 1 -- 172.21.2.112:0/598673917 shutdown_connections 2021-02-10T11:43:13.306 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.304 7fe4f73e3700 1 -- 172.21.2.112:0/598673917 wait complete. 2021-02-10T11:43:13.313 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.3 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:43:13.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.735 7fe9721a3700 1 Processor -- start 2021-02-10T11:43:14.074 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.736 7fe9721a3700 1 -- start start 2021-02-10T11:43:14.074 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.736 7fe9721a3700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fe96c0d1880 0x7fe96c1b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:14.074 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.736 7fe9721a3700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe96c0fe1d0 con 0x7fe96c1b0af0 2021-02-10T11:43:14.074 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.736 7fe9721a3700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe96c109a70 con 0x7fe96c0cff80 2021-02-10T11:43:14.074 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.736 7fe9721a3700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe96c088070 con 0x7fe96c0d1880 2021-02-10T11:43:14.074 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.736 7fe96b7fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fe96c0d1880 0x7fe96c1b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:14.074 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.736 7fe96affd700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fe96c1b0af0 0x7fe96c108950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:56396/0 (socket says 172.21.2.112:56396) 2021-02-10T11:43:14.075 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.736 7fe96b7fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fe96c0d1880 0x7fe96c1b04d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:56934/0 (socket says 172.21.2.112:56934) 2021-02-10T11:43:14.075 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.736 7fe96b7fe700 1 -- 172.21.2.112:0/3599069679 learned_addr learned my addr 172.21.2.112:0/3599069679 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:43:14.075 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.737 7fe96b7fe700 1 -- 172.21.2.112:0/3599069679 >> v1:172.21.2.120:6789/0 conn(0x7fe96c0cff80 legacy=0x7fe96c109140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:14.075 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.737 7fe96a7fc700 1 -- 172.21.2.112:0/3599069679 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 702786491 0 0) 0x7fe9540010b0 con 0x7fe96c1b0af0 2021-02-10T11:43:14.075 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.737 7fe96b7fe700 1 -- 172.21.2.112:0/3599069679 >> v1:172.21.2.112:6789/0 conn(0x7fe96c1b0af0 legacy=0x7fe96c108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:14.075 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.737 7fe96b7fe700 1 -- 172.21.2.112:0/3599069679 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe96c088260 con 0x7fe96c0d1880 2021-02-10T11:43:14.075 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.737 7fe96b7fe700 1 --2- 172.21.2.112:0/3599069679 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe96c0d1880 0x7fe96c1b04d0 secure :-1 s=READY pgs=2446 cs=0 l=1 rev1=1 rx=0x7fe95c006700 tx=0x7fe95c009b80).ready entity=mon.0 client_cookie=cfab814acac02630 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:14.075 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.737 7fe96a7fc700 1 -- 172.21.2.112:0/3599069679 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe95c00c0c0 con 0x7fe96c0d1880 2021-02-10T11:43:14.075 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.737 7fe96a7fc700 1 -- 172.21.2.112:0/3599069679 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe95c00c340 con 0x7fe96c0d1880 2021-02-10T11:43:14.075 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.737 7fe96a7fc700 1 -- 172.21.2.112:0/3599069679 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe95c00c860 con 0x7fe96c0d1880 2021-02-10T11:43:14.075 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.737 7fe9721a3700 1 -- 172.21.2.112:0/3599069679 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe96c0d1880 msgr2=0x7fe96c1b04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:14.075 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.737 7fe9721a3700 1 --2- 172.21.2.112:0/3599069679 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe96c0d1880 0x7fe96c1b04d0 secure :-1 s=READY pgs=2446 cs=0 l=1 rev1=1 rx=0x7fe95c006700 tx=0x7fe95c009b80).stop 2021-02-10T11:43:14.076 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.739 7fe9721a3700 1 -- 172.21.2.112:0/3599069679 shutdown_connections 2021-02-10T11:43:14.076 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.739 7fe9721a3700 1 --2- 172.21.2.112:0/3599069679 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe96c0d1880 0x7fe96c1b04d0 unknown :-1 s=CLOSED pgs=2446 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:14.076 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.739 7fe9721a3700 1 -- 172.21.2.112:0/3599069679 >> 172.21.2.112:0/3599069679 conn(0x7fe96c0ce430 msgr2=0x7fe96c0d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:14.076 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.739 7fe9721a3700 1 -- 172.21.2.112:0/3599069679 shutdown_connections 2021-02-10T11:43:14.076 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.739 7fe9721a3700 1 -- 172.21.2.112:0/3599069679 wait complete. 2021-02-10T11:43:14.082 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.753 7fe9721a3700 1 Processor -- start 2021-02-10T11:43:14.082 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.753 7fe9721a3700 1 -- start start 2021-02-10T11:43:14.082 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.753 7fe9721a3700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe96c086520 0x7fe96c1a1cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:14.083 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.753 7fe9721a3700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe96c086920 0x7fe96c187e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:14.083 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.753 7fe9721a3700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe96c1a22d0 0x7fe96c1884a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:14.083 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.753 7fe9721a3700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe96c099600 con 0x7fe96c086920 2021-02-10T11:43:14.083 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.753 7fe9721a3700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe96c0999f0 con 0x7fe96c086520 2021-02-10T11:43:14.083 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.753 7fe9721a3700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe96c188ac0 con 0x7fe96c1a22d0 2021-02-10T11:43:14.083 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.753 7fe96affd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe96c086920 0x7fe96c187e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:14.083 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.753 7fe96affd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe96c086920 0x7fe96c187e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:56942/0 (socket says 172.21.2.112:56942) 2021-02-10T11:43:14.084 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.753 7fe96affd700 1 -- 172.21.2.112:0/2440301315 learned_addr learned my addr 172.21.2.112:0/2440301315 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:43:14.084 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.754 7fe96bfff700 1 --2- 172.21.2.112:0/2440301315 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe96c1a22d0 0x7fe96c1884a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:14.084 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.754 7fe96b7fe700 1 --2- 172.21.2.112:0/2440301315 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe96c086520 0x7fe96c1a1cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:14.084 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.754 7fe96affd700 1 -- 172.21.2.112:0/2440301315 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe96c1a22d0 msgr2=0x7fe96c1884a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:14.084 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.754 7fe96affd700 1 --2- 172.21.2.112:0/2440301315 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe96c1a22d0 0x7fe96c1884a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:14.084 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.754 7fe96affd700 1 -- 172.21.2.112:0/2440301315 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe96c086520 msgr2=0x7fe96c1a1cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:14.085 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.754 7fe96affd700 1 --2- 172.21.2.112:0/2440301315 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe96c086520 0x7fe96c1a1cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:14.085 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.754 7fe96affd700 1 -- 172.21.2.112:0/2440301315 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe96c188ce0 con 0x7fe96c086920 2021-02-10T11:43:14.085 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.754 7fe96b7fe700 1 --2- 172.21.2.112:0/2440301315 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe96c086520 0x7fe96c1a1cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:43:14.085 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.754 7fe96affd700 1 --2- 172.21.2.112:0/2440301315 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe96c086920 0x7fe96c187e80 secure :-1 s=READY pgs=2447 cs=0 l=1 rev1=1 rx=0x7fe954001c10 tx=0x7fe95400a8f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:14.085 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.754 7fe963fff700 1 -- 172.21.2.112:0/2440301315 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe9540009b0 con 0x7fe96c086920 2021-02-10T11:43:14.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.754 7fe963fff700 1 -- 172.21.2.112:0/2440301315 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9540012c0 con 0x7fe96c086920 2021-02-10T11:43:14.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.754 7fe963fff700 1 -- 172.21.2.112:0/2440301315 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe954004350 con 0x7fe96c086920 2021-02-10T11:43:14.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.754 7fe9721a3700 1 -- 172.21.2.112:0/2440301315 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe96c188ce0 con 0x7fe96c086920 2021-02-10T11:43:14.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.754 7fe9721a3700 1 -- 172.21.2.112:0/2440301315 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe96c189270 con 0x7fe96c086920 2021-02-10T11:43:14.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.755 7fe963fff700 1 -- 172.21.2.112:0/2440301315 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe954007fd0 con 0x7fe96c086920 2021-02-10T11:43:14.087 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.755 7fe963fff700 1 --2- 172.21.2.112:0/2440301315 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe9500186c0 0x7fe95001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:14.087 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.755 7fe963fff700 1 -- 172.21.2.112:0/2440301315 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fe9540323b0 con 0x7fe96c086920 2021-02-10T11:43:14.087 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.755 7fe96b7fe700 1 --2- 172.21.2.112:0/2440301315 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe9500186c0 0x7fe95001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:14.087 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.756 7fe96b7fe700 1 --2- 172.21.2.112:0/2440301315 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe9500186c0 0x7fe95001abc0 crc :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:14.087 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.786 7fe9721a3700 1 -- 172.21.2.112:0/2440301315 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe96c0fe1d0 con 0x7fe96c086920 2021-02-10T11:43:14.087 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.789 7fe963fff700 1 -- 172.21.2.112:0/2440301315 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe9540009b0 con 0x7fe96c086920 2021-02-10T11:43:14.088 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.919 7fe9721a3700 1 -- 172.21.2.112:0/2440301315 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fe96c0999a0 con 0x7fe96c086920 2021-02-10T11:43:14.088 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.923 7fe963fff700 1 -- 172.21.2.112:0/2440301315 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.3 v82) v1 ==== 184+0+0 (secure 0 0 0) 0x7fe954007fd0 con 0x7fe96c086920 2021-02-10T11:43:14.088 INFO:teuthology.orchestra.run.gibba012.stderr:updated caps for client.3 2021-02-10T11:43:14.088 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.927 7fe9721a3700 1 -- 172.21.2.112:0/2440301315 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe9500186c0 msgr2=0x7fe95001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:14.088 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.927 7fe9721a3700 1 --2- 172.21.2.112:0/2440301315 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe9500186c0 0x7fe95001abc0 crc :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:14.088 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.927 7fe9721a3700 1 -- 172.21.2.112:0/2440301315 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe96c086920 msgr2=0x7fe96c187e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:14.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.927 7fe9721a3700 1 --2- 172.21.2.112:0/2440301315 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe96c086920 0x7fe96c187e80 secure :-1 s=READY pgs=2447 cs=0 l=1 rev1=1 rx=0x7fe954001c10 tx=0x7fe95400a8f0).stop 2021-02-10T11:43:14.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.928 7fe9721a3700 1 -- 172.21.2.112:0/2440301315 shutdown_connections 2021-02-10T11:43:14.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.928 7fe9721a3700 1 --2- 172.21.2.112:0/2440301315 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe9500186c0 0x7fe95001abc0 unknown :-1 s=CLOSED pgs=1117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:14.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.928 7fe9721a3700 1 --2- 172.21.2.112:0/2440301315 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe96c1a22d0 0x7fe96c1884a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:14.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.928 7fe9721a3700 1 --2- 172.21.2.112:0/2440301315 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe96c086920 0x7fe96c187e80 unknown :-1 s=CLOSED pgs=2447 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:14.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.928 7fe9721a3700 1 --2- 172.21.2.112:0/2440301315 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe96c086520 0x7fe96c1a1cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:14.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.928 7fe9721a3700 1 -- 172.21.2.112:0/2440301315 >> 172.21.2.112:0/2440301315 conn(0x7fe96c19f880 msgr2=0x7fe96c08f890 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:14.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.928 7fe9721a3700 1 -- 172.21.2.112:0/2440301315 shutdown_connections 2021-02-10T11:43:14.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:13.928 7fe9721a3700 1 -- 172.21.2.112:0/2440301315 wait complete. 2021-02-10T11:43:14.091 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:43:14.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:14.506 7fc654815700 1 Processor -- start 2021-02-10T11:43:14.524 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:14.506 7fc654815700 1 -- start start 2021-02-10T11:43:14.524 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:14.507 7fc654815700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fc6501b0470 0x7fc6501b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:14.524 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:14.507 7fc654815700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fc6500ef610 0x7fc6500efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:14.524 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:14.507 7fc654815700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6500f0b50 con 0x7fc6500f0030 2021-02-10T11:43:14.524 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:14.507 7fc654815700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6500e52e0 con 0x7fc6500ef610 2021-02-10T11:43:14.524 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:14.507 7fc654815700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6500e5490 con 0x7fc6501b0470 2021-02-10T11:43:14.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:14.507 7fc64de1b700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fc6500ef610 0x7fc6500efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:14.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:14.507 7fc64de1b700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fc6500ef610 0x7fc6500efa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:56946/0 (socket says 172.21.2.112:56946) 2021-02-10T11:43:14.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:14.507 7fc64de1b700 1 -- 172.21.2.112:0/3718282111 learned_addr learned my addr 172.21.2.112:0/3718282111 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:43:14.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:14.507 7fc64de1b700 1 -- 172.21.2.112:0/3718282111 >> v2:172.21.2.120:3300/0 conn(0x7fc6501b0470 msgr2=0x7fc6501b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:14.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:14.507 7fc64de1b700 1 --2- 172.21.2.112:0/3718282111 >> v2:172.21.2.120:3300/0 conn(0x7fc6501b0470 0x7fc6501b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:14.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:14.507 7fc64de1b700 1 -- 172.21.2.112:0/3718282111 >> v1:172.21.2.120:6789/0 conn(0x7fc6500f0030 legacy=0x7fc6500f10d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:43:14.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:14.508 7fc64de1b700 1 -- 172.21.2.112:0/3718282111 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6500f0430 con 0x7fc6500ef610 2021-02-10T11:43:14.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:14.508 7fc64de1b700 1 --2- 172.21.2.112:0/3718282111 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc6500ef610 0x7fc6500efa10 secure :-1 s=READY pgs=2448 cs=0 l=1 rev1=1 rx=0x7fc638006700 tx=0x7fc638009b00).ready entity=mon.0 client_cookie=8719c6d2be338678 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:14.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:14.508 7fc64d61a700 1 -- 172.21.2.112:0/3718282111 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc63800c0c0 con 0x7fc6500ef610 2021-02-10T11:43:14.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:14.508 7fc64d61a700 1 -- 172.21.2.112:0/3718282111 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc63800c340 con 0x7fc6500ef610 2021-02-10T11:43:14.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:14.508 7fc64d61a700 1 -- 172.21.2.112:0/3718282111 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc63800c820 con 0x7fc6500ef610 2021-02-10T11:43:14.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:14.508 7fc654815700 1 -- 172.21.2.112:0/3718282111 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc6500ef610 msgr2=0x7fc6500efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:14.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:14.508 7fc654815700 1 --2- 172.21.2.112:0/3718282111 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc6500ef610 0x7fc6500efa10 secure :-1 s=READY pgs=2448 cs=0 l=1 rev1=1 rx=0x7fc638006700 tx=0x7fc638009b00).stop 2021-02-10T11:43:14.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:14.509 7fc654815700 1 -- 172.21.2.112:0/3718282111 shutdown_connections 2021-02-10T11:43:14.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:14.509 7fc654815700 1 --2- 172.21.2.112:0/3718282111 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc6500ef610 0x7fc6500efa10 unknown :-1 s=CLOSED pgs=2448 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:14.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:14.509 7fc654815700 1 --2- 172.21.2.112:0/3718282111 >> v2:172.21.2.120:3300/0 conn(0x7fc6501b0470 0x7fc6501b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:14.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:14.509 7fc654815700 1 -- 172.21.2.112:0/3718282111 >> 172.21.2.112:0/3718282111 conn(0x7fc6500ce3e0 msgr2=0x7fc6500e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:14.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:14.509 7fc654815700 1 -- 172.21.2.112:0/3718282111 shutdown_connections 2021-02-10T11:43:14.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:14.509 7fc654815700 1 -- 172.21.2.112:0/3718282111 wait complete. 2021-02-10T11:43:14.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:14.523 7fc654815700 1 Processor -- start 2021-02-10T11:43:14.528 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:14.523 7fc654815700 1 -- start start 2021-02-10T11:43:14.530 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:14.524 7fc654815700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc6500b0970 0x7fc650092ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:14.530 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:14.524 7fc654815700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc6500934f0 0x7fc650094dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:14.531 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:14.524 7fc654815700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc6500938f0 0x7fc6500953e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:14.531 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:14.524 7fc654815700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc650195ad0 con 0x7fc6500938f0 2021-02-10T11:43:14.531 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:14.524 7fc654815700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc650095f40 con 0x7fc6500b0970 2021-02-10T11:43:14.531 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:14.524 7fc654815700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc650095a30 con 0x7fc6500934f0 2021-02-10T11:43:14.531 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:14.524 7fc64ee1d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc6500938f0 0x7fc6500953e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:14.531 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:14.524 7fc64ee1d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc6500938f0 0x7fc6500953e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:56954/0 (socket says 172.21.2.112:56954) 2021-02-10T11:43:14.532 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:14.524 7fc64ee1d700 1 -- 172.21.2.112:0/1443716644 learned_addr learned my addr 172.21.2.112:0/1443716644 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:43:14.532 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:14.524 7fc64ee1d700 1 -- 172.21.2.112:0/1443716644 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc6500934f0 msgr2=0x7fc650094dc0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:43:14.532 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:14.524 7fc64ee1d700 1 --2- 172.21.2.112:0/1443716644 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc6500934f0 0x7fc650094dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:14.532 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:14.524 7fc64ee1d700 1 -- 172.21.2.112:0/1443716644 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc6500b0970 msgr2=0x7fc650092ed0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:43:14.533 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:14.524 7fc64ee1d700 1 --2- 172.21.2.112:0/1443716644 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc6500b0970 0x7fc650092ed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:14.533 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:14.524 7fc64ee1d700 1 -- 172.21.2.112:0/1443716644 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc650095c50 con 0x7fc6500938f0 2021-02-10T11:43:14.533 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:14.525 7fc64ee1d700 1 --2- 172.21.2.112:0/1443716644 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc6500938f0 0x7fc6500953e0 secure :-1 s=READY pgs=2449 cs=0 l=1 rev1=1 rx=0x7fc648009940 tx=0x7fc64800bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:14.533 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:14.525 7fc646ffd700 1 -- 172.21.2.112:0/1443716644 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc648000f90 con 0x7fc6500938f0 2021-02-10T11:43:14.533 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:14.525 7fc646ffd700 1 -- 172.21.2.112:0/1443716644 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc64800c2f0 con 0x7fc6500938f0 2021-02-10T11:43:14.533 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:14.525 7fc646ffd700 1 -- 172.21.2.112:0/1443716644 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc64800c7d0 con 0x7fc6500938f0 2021-02-10T11:43:14.534 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:14.525 7fc654815700 1 -- 172.21.2.112:0/1443716644 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc650095c50 con 0x7fc6500938f0 2021-02-10T11:43:14.534 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:14.525 7fc654815700 1 -- 172.21.2.112:0/1443716644 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc650095c50 con 0x7fc6500938f0 2021-02-10T11:43:14.534 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:14.525 7fc646ffd700 1 -- 172.21.2.112:0/1443716644 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc648001220 con 0x7fc6500938f0 2021-02-10T11:43:14.534 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:14.525 7fc646ffd700 1 --2- 172.21.2.112:0/1443716644 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc634018730 0x7fc63401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:14.534 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:14.525 7fc646ffd700 1 -- 172.21.2.112:0/1443716644 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fc648032b50 con 0x7fc6500938f0 2021-02-10T11:43:14.534 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:14.526 7fc64e61c700 1 --2- 172.21.2.112:0/1443716644 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc634018730 0x7fc63401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:14.534 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:14.526 7fc64e61c700 1 --2- 172.21.2.112:0/1443716644 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc634018730 0x7fc63401ac10 crc :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:14.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:14.557 7fc654815700 1 -- 172.21.2.112:0/1443716644 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6500e52e0 con 0x7fc6500938f0 2021-02-10T11:43:14.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:14.560 7fc646ffd700 1 -- 172.21.2.112:0/1443716644 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc64800be30 con 0x7fc6500938f0 2021-02-10T11:43:14.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:14.687 7fc654815700 1 -- 172.21.2.112:0/1443716644 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc6500e52e0 con 0x7fc6500938f0 2021-02-10T11:43:14.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:14.688 7fc646ffd700 1 -- 172.21.2.112:0/1443716644 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 114 v114) v1 ==== 95+0+2108 (secure 0 0 0) 0x7fc648001220 con 0x7fc6500938f0 2021-02-10T11:43:14.688 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 114 2021-02-10T11:43:14.688 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:43:14.689 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":114,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11349,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/2093257968","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":2093257968},{"type":"v1","addr":"172.21.2.112:6833","nonce":2093257968}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":110}],"filesystems":[{"mdsmap":{"epoch":114,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:43:08.182625","modified":"2021-02-10 11:43:10.189979","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7025},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7025":{"gid":7025,"name":"b","rank":0,"incarnation":111,"state":"up:active","state_seq":3,"addr":"172.21.2.120:6835/2552826693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":2552826693},{"type":"v1","addr":"172.21.2.120:6835","nonce":2552826693}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":14}]} 2021-02-10T11:43:14.690 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:14.690 7fc654815700 1 -- 172.21.2.112:0/1443716644 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc634018730 msgr2=0x7fc63401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:14.690 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:14.690 7fc654815700 1 --2- 172.21.2.112:0/1443716644 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc634018730 0x7fc63401ac10 crc :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:14.690 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:14.690 7fc654815700 1 -- 172.21.2.112:0/1443716644 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc6500938f0 msgr2=0x7fc6500953e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:14.690 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:14.690 7fc654815700 1 --2- 172.21.2.112:0/1443716644 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc6500938f0 0x7fc6500953e0 secure :-1 s=READY pgs=2449 cs=0 l=1 rev1=1 rx=0x7fc648009940 tx=0x7fc64800bcf0).stop 2021-02-10T11:43:14.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:14.690 7fc654815700 1 -- 172.21.2.112:0/1443716644 shutdown_connections 2021-02-10T11:43:14.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:14.690 7fc654815700 1 --2- 172.21.2.112:0/1443716644 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc634018730 0x7fc63401ac10 unknown :-1 s=CLOSED pgs=1118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:14.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:14.690 7fc654815700 1 --2- 172.21.2.112:0/1443716644 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc6500938f0 0x7fc6500953e0 unknown :-1 s=CLOSED pgs=2449 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:14.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:14.690 7fc654815700 1 --2- 172.21.2.112:0/1443716644 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc6500934f0 0x7fc650094dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:14.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:14.690 7fc654815700 1 --2- 172.21.2.112:0/1443716644 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc6500b0970 0x7fc650092ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:14.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:14.690 7fc654815700 1 -- 172.21.2.112:0/1443716644 >> 172.21.2.112:0/1443716644 conn(0x7fc65019f7a0 msgr2=0x7fc6500940e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:14.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:14.690 7fc654815700 1 -- 172.21.2.112:0/1443716644 shutdown_connections 2021-02-10T11:43:14.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:14.690 7fc654815700 1 -- 172.21.2.112:0/1443716644 wait complete. 2021-02-10T11:43:14.900 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 114, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:43:08.182625', 'modified': '2021-02-10 11:43:10.189979', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7025}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7025': {'gid': 7025, 'name': 'b', 'rank': 0, 'incarnation': 111, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.120:6835/2552826693', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.120:6834', 'nonce': 2552826693}, {'type': 'v1', 'addr': '172.21.2.120:6835', 'nonce': 2552826693}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [29], 'metadata_pool': 28, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:43:14.900 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:43:14.900 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:43:15.299 INFO:teuthology.orchestra.run.gibba020.stdout:{ 2021-02-10T11:43:15.299 INFO:teuthology.orchestra.run.gibba020.stdout: "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", 2021-02-10T11:43:15.299 INFO:teuthology.orchestra.run.gibba020.stdout: "whoami": 0, 2021-02-10T11:43:15.299 INFO:teuthology.orchestra.run.gibba020.stdout: "id": 7025, 2021-02-10T11:43:15.299 INFO:teuthology.orchestra.run.gibba020.stdout: "want_state": "up:active", 2021-02-10T11:43:15.299 INFO:teuthology.orchestra.run.gibba020.stdout: "state": "up:active", 2021-02-10T11:43:15.299 INFO:teuthology.orchestra.run.gibba020.stdout: "rank_uptime": 7.1135747289999998, 2021-02-10T11:43:15.299 INFO:teuthology.orchestra.run.gibba020.stdout: "mdsmap_epoch": 114, 2021-02-10T11:43:15.299 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch": 134, 2021-02-10T11:43:15.300 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch_barrier": 134, 2021-02-10T11:43:15.300 INFO:teuthology.orchestra.run.gibba020.stdout: "uptime": 12.756230930999999 2021-02-10T11:43:15.300 INFO:teuthology.orchestra.run.gibba020.stdout:} 2021-02-10T11:43:15.303 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", "whoami": 0, "id": 7025, "want_state": "up:active", "state": "up:active", "rank_uptime": 7.1135747289999998, "mdsmap_epoch": 114, "osdmap_epoch": 134, "osdmap_epoch_barrier": 134, "uptime": 12.756230930999999 } 2021-02-10T11:43:15.303 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:43:15.732 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:15.715 7fa528b1d700 1 Processor -- start 2021-02-10T11:43:15.733 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:15.715 7fa528b1d700 1 -- start start 2021-02-10T11:43:15.733 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:15.716 7fa528b1d700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fa5240d1870 0x7fa5241b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:15.733 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:15.716 7fa528b1d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fa5241b0a10 0x7fa5240cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:15.733 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:15.716 7fa528b1d700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5240f0b10 con 0x7fa5240d1c70 2021-02-10T11:43:15.733 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:15.716 7fa528b1d700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5240e5ed0 con 0x7fa5241b0a10 2021-02-10T11:43:15.733 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:15.716 7fa528b1d700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5240f0650 con 0x7fa5240d1870 2021-02-10T11:43:15.733 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:15.716 7fa52205d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fa5241b0a10 0x7fa5240cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:15.733 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:15.716 7fa52305f700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fa5240d1c70 0x7fa5240cf950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:56422/0 (socket says 172.21.2.112:56422) 2021-02-10T11:43:15.734 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:15.716 7fa52305f700 1 -- 172.21.2.112:0/4017607369 learned_addr learned my addr 172.21.2.112:0/4017607369 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:43:15.734 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:15.716 7fa52285e700 1 --2- 172.21.2.112:0/4017607369 >> v2:172.21.2.120:3300/0 conn(0x7fa5240d1870 0x7fa5241b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:15.734 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:15.716 7fa52205d700 1 -- 172.21.2.112:0/4017607369 >> v2:172.21.2.120:3300/0 conn(0x7fa5240d1870 msgr2=0x7fa5241b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:15.734 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:15.716 7fa52205d700 1 --2- 172.21.2.112:0/4017607369 >> v2:172.21.2.120:3300/0 conn(0x7fa5240d1870 0x7fa5241b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:15.734 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:15.716 7fa52205d700 1 -- 172.21.2.112:0/4017607369 >> v1:172.21.2.112:6789/0 conn(0x7fa5240d1c70 legacy=0x7fa5240cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:15.734 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:15.716 7fa52205d700 1 -- 172.21.2.112:0/4017607369 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5240f0840 con 0x7fa5241b0a10 2021-02-10T11:43:15.734 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:15.716 7fa52285e700 1 --2- 172.21.2.112:0/4017607369 >> v2:172.21.2.120:3300/0 conn(0x7fa5240d1870 0x7fa5241b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:43:15.734 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:15.717 7fa52205d700 1 --2- 172.21.2.112:0/4017607369 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa5241b0a10 0x7fa5240cf330 secure :-1 s=READY pgs=2451 cs=0 l=1 rev1=1 rx=0x7fa50c006650 tx=0x7fa50c009a80).ready entity=mon.0 client_cookie=5377fcd307106a35 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:15.735 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:15.717 7fa52185c700 1 -- 172.21.2.112:0/4017607369 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa50c00c0c0 con 0x7fa5241b0a10 2021-02-10T11:43:15.735 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:15.717 7fa52185c700 1 -- 172.21.2.112:0/4017607369 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa50c00c340 con 0x7fa5241b0a10 2021-02-10T11:43:15.735 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:15.717 7fa52185c700 1 -- 172.21.2.112:0/4017607369 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa50c00c860 con 0x7fa5241b0a10 2021-02-10T11:43:15.735 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:15.717 7fa528b1d700 1 -- 172.21.2.112:0/4017607369 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa5241b0a10 msgr2=0x7fa5240cf330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:15.735 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:15.717 7fa528b1d700 1 --2- 172.21.2.112:0/4017607369 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa5241b0a10 0x7fa5240cf330 secure :-1 s=READY pgs=2451 cs=0 l=1 rev1=1 rx=0x7fa50c006650 tx=0x7fa50c009a80).stop 2021-02-10T11:43:15.735 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:15.718 7fa528b1d700 1 -- 172.21.2.112:0/4017607369 shutdown_connections 2021-02-10T11:43:15.735 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:15.718 7fa528b1d700 1 --2- 172.21.2.112:0/4017607369 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa5241b0a10 0x7fa5240cf330 unknown :-1 s=CLOSED pgs=2451 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:15.735 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:15.718 7fa528b1d700 1 --2- 172.21.2.112:0/4017607369 >> v2:172.21.2.120:3300/0 conn(0x7fa5240d1870 0x7fa5241b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:15.735 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:15.718 7fa528b1d700 1 -- 172.21.2.112:0/4017607369 >> 172.21.2.112:0/4017607369 conn(0x7fa5240ce450 msgr2=0x7fa5240d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:15.736 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:15.718 7fa528b1d700 1 -- 172.21.2.112:0/4017607369 shutdown_connections 2021-02-10T11:43:15.736 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:15.718 7fa528b1d700 1 -- 172.21.2.112:0/4017607369 wait complete. 2021-02-10T11:43:15.736 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:15.732 7fa528b1d700 1 Processor -- start 2021-02-10T11:43:15.736 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:15.733 7fa528b1d700 1 -- start start 2021-02-10T11:43:15.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:15.733 7fa528b1d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa5241b8ad0 0x7fa524187e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:15.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:15.733 7fa528b1d700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa5241b8ed0 0x7fa5240b1af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:15.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:15.733 7fa528b1d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa5240b2110 0x7fa5241b7a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:15.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:15.733 7fa528b1d700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa524188470 con 0x7fa5241b8ad0 2021-02-10T11:43:15.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:15.733 7fa528b1d700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa52418be10 con 0x7fa5241b8ed0 2021-02-10T11:43:15.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:15.733 7fa528b1d700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa52418c030 con 0x7fa5240b2110 2021-02-10T11:43:15.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:15.733 7fa52285e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa5241b8ad0 0x7fa524187e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:15.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:15.733 7fa52285e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa5241b8ad0 0x7fa524187e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:56964/0 (socket says 172.21.2.112:56964) 2021-02-10T11:43:15.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:15.733 7fa52285e700 1 -- 172.21.2.112:0/418171353 learned_addr learned my addr 172.21.2.112:0/418171353 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:43:15.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:15.734 7fa52285e700 1 -- 172.21.2.112:0/418171353 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa5240b2110 msgr2=0x7fa5241b7a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:15.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:15.734 7fa52285e700 1 --2- 172.21.2.112:0/418171353 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa5240b2110 0x7fa5241b7a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:15.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:15.734 7fa52285e700 1 -- 172.21.2.112:0/418171353 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa5241b8ed0 msgr2=0x7fa5240b1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:15.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:15.734 7fa52285e700 1 --2- 172.21.2.112:0/418171353 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa5241b8ed0 0x7fa5240b1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:15.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:15.734 7fa52285e700 1 -- 172.21.2.112:0/418171353 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5240b2510 con 0x7fa5241b8ad0 2021-02-10T11:43:15.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:15.734 7fa52285e700 1 --2- 172.21.2.112:0/418171353 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa5241b8ad0 0x7fa524187e50 secure :-1 s=READY pgs=2452 cs=0 l=1 rev1=1 rx=0x7fa514006f20 tx=0x7fa51400b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:15.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:15.734 7fa51b7fe700 1 -- 172.21.2.112:0/418171353 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa51400bd80 con 0x7fa5241b8ad0 2021-02-10T11:43:15.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:15.734 7fa51b7fe700 1 -- 172.21.2.112:0/418171353 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa514000c90 con 0x7fa5241b8ad0 2021-02-10T11:43:15.740 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:15.734 7fa51b7fe700 1 -- 172.21.2.112:0/418171353 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa5140022b0 con 0x7fa5241b8ad0 2021-02-10T11:43:15.740 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:15.734 7fa528b1d700 1 -- 172.21.2.112:0/418171353 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5240b2510 con 0x7fa5241b8ad0 2021-02-10T11:43:15.740 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:15.734 7fa528b1d700 1 -- 172.21.2.112:0/418171353 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5240b2510 con 0x7fa5241b8ad0 2021-02-10T11:43:15.740 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:15.735 7fa51b7fe700 1 -- 172.21.2.112:0/418171353 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa51400a000 con 0x7fa5241b8ad0 2021-02-10T11:43:15.740 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:15.735 7fa51b7fe700 1 --2- 172.21.2.112:0/418171353 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa5080186c0 0x7fa50801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:15.740 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:15.735 7fa51b7fe700 1 -- 172.21.2.112:0/418171353 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fa5140323f0 con 0x7fa5241b8ad0 2021-02-10T11:43:15.740 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:15.735 7fa52205d700 1 --2- 172.21.2.112:0/418171353 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa5080186c0 0x7fa50801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:15.740 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:15.736 7fa52205d700 1 --2- 172.21.2.112:0/418171353 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa5080186c0 0x7fa50801abc0 crc :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:15.766 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:15.766 7fa528b1d700 1 -- 172.21.2.112:0/418171353 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5240ebc00 con 0x7fa5241b8ad0 2021-02-10T11:43:15.768 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:15.768 7fa51b7fe700 1 -- 172.21.2.112:0/418171353 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa51402ded0 con 0x7fa5241b8ad0 2021-02-10T11:43:15.893 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:15.893 7fa528b1d700 1 -- 172.21.2.112:0/418171353 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa5240ebc00 con 0x7fa5241b8ad0 2021-02-10T11:43:15.894 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:15.894 7fa51b7fe700 1 -- 172.21.2.112:0/418171353 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v134) v1 ==== 74+0+14007 (secure 0 0 0) 0x7fa51402ded0 con 0x7fa5241b8ad0 2021-02-10T11:43:15.894 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:43:15.895 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":134,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:43:09.185424","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":28,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:43:06.157376","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"133","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":29,"pool_name":"cephfs_data","create_time":"2021-02-10 11:43:07.178112","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"134","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":132,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":132,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":132,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":132,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:43:15.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:15.895 7fa528b1d700 1 -- 172.21.2.112:0/418171353 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa5080186c0 msgr2=0x7fa50801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:15.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:15.895 7fa528b1d700 1 --2- 172.21.2.112:0/418171353 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa5080186c0 0x7fa50801abc0 crc :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:15.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:15.895 7fa528b1d700 1 -- 172.21.2.112:0/418171353 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa5241b8ad0 msgr2=0x7fa524187e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:15.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:15.895 7fa528b1d700 1 --2- 172.21.2.112:0/418171353 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa5241b8ad0 0x7fa524187e50 secure :-1 s=READY pgs=2452 cs=0 l=1 rev1=1 rx=0x7fa514006f20 tx=0x7fa51400b990).stop 2021-02-10T11:43:15.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:15.896 7fa528b1d700 1 -- 172.21.2.112:0/418171353 shutdown_connections 2021-02-10T11:43:15.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:15.896 7fa528b1d700 1 --2- 172.21.2.112:0/418171353 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa5080186c0 0x7fa50801abc0 unknown :-1 s=CLOSED pgs=1119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:15.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:15.896 7fa528b1d700 1 --2- 172.21.2.112:0/418171353 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa5241b8ed0 0x7fa5240b1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:15.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:15.896 7fa528b1d700 1 --2- 172.21.2.112:0/418171353 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa5240b2110 0x7fa5241b7a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:15.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:15.896 7fa528b1d700 1 --2- 172.21.2.112:0/418171353 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa5241b8ad0 0x7fa524187e50 unknown :-1 s=CLOSED pgs=2452 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:15.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:15.896 7fa528b1d700 1 -- 172.21.2.112:0/418171353 >> 172.21.2.112:0/418171353 conn(0x7fa5240ed810 msgr2=0x7fa524094980 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:15.897 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:15.896 7fa528b1d700 1 -- 172.21.2.112:0/418171353 shutdown_connections 2021-02-10T11:43:15.897 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:15.896 7fa528b1d700 1 -- 172.21.2.112:0/418171353 wait complete. 2021-02-10T11:43:15.904 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:13:57.508758', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 28, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:43:06.157376', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '133', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 29, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:43:07.178112', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '134', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:43:15.905 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:43:16.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.320 7fc51913c700 1 Processor -- start 2021-02-10T11:43:16.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.320 7fc51913c700 1 -- start start 2021-02-10T11:43:16.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.321 7fc51913c700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fc51410afe0 0x7fc5140cf740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:16.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.321 7fc51913c700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5140fe1b0 con 0x7fc5140d1860 2021-02-10T11:43:16.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.321 7fc51913c700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5141b0ca0 con 0x7fc5140d1c60 2021-02-10T11:43:16.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.321 7fc51913c700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc514109a50 con 0x7fc51410afe0 2021-02-10T11:43:16.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.321 7fc512d9d700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fc5140d1860 0x7fc5141b04b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:56432/0 (socket says 172.21.2.112:56432) 2021-02-10T11:43:16.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.321 7fc512d9d700 1 -- 172.21.2.112:0/55184864 learned_addr learned my addr 172.21.2.112:0/55184864 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:43:16.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.321 7fc511d9b700 1 -- 172.21.2.112:0/55184864 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4016993072 0 0) 0x7fc504000ce0 con 0x7fc5140d1860 2021-02-10T11:43:16.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.321 7fc511d9b700 1 -- 172.21.2.112:0/55184864 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc500002fe0 con 0x7fc5140d1860 2021-02-10T11:43:16.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.322 7fc511d9b700 1 -- 172.21.2.112:0/55184864 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2754077195 0 0) 0x7fc504001180 con 0x7fc5140d1860 2021-02-10T11:43:16.339 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.322 7fc511d9b700 1 -- 172.21.2.112:0/55184864 >> v2:172.21.2.120:3300/0 conn(0x7fc51410afe0 msgr2=0x7fc5140cf740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:16.339 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.322 7fc511d9b700 1 --2- 172.21.2.112:0/55184864 >> v2:172.21.2.120:3300/0 conn(0x7fc51410afe0 0x7fc5140cf740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:16.339 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.322 7fc511d9b700 1 -- 172.21.2.112:0/55184864 >> v1:172.21.2.120:6789/0 conn(0x7fc5140d1c60 legacy=0x7fc5140cef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:16.339 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.322 7fc511d9b700 1 -- 172.21.2.112:0/55184864 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc51410b3e0 con 0x7fc5140d1860 2021-02-10T11:43:16.339 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.322 7fc511d9b700 1 -- 172.21.2.112:0/55184864 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7fc504001680 con 0x7fc5140d1860 2021-02-10T11:43:16.339 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.322 7fc511d9b700 1 -- 172.21.2.112:0/55184864 >> v1:172.21.2.112:6789/0 conn(0x7fc5140d1860 legacy=0x7fc5141b04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:16.339 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.322 7fc511d9b700 1 --2- 172.21.2.112:0/55184864 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc500003a70 0x7fc500005f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:16.339 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.322 7fc511d9b700 1 --2- 172.21.2.112:0/55184864 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc500006780 0x7fc500008ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:16.339 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.322 7fc511d9b700 1 --2- 172.21.2.112:0/55184864 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc5000095a0 0x7fc50000bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:16.339 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.322 7fc511d9b700 1 -- 172.21.2.112:0/55184864 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc50000c310 con 0x7fc500006780 2021-02-10T11:43:16.339 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.322 7fc511d9b700 1 -- 172.21.2.112:0/55184864 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc50000c530 con 0x7fc500003a70 2021-02-10T11:43:16.339 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.322 7fc511d9b700 1 -- 172.21.2.112:0/55184864 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc50000c750 con 0x7fc5000095a0 2021-02-10T11:43:16.339 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.322 7fc51259c700 1 --2- 172.21.2.112:0/55184864 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc500006780 0x7fc500008ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:16.339 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.322 7fc512d9d700 1 --2- 172.21.2.112:0/55184864 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc500003a70 0x7fc500005f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:16.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.322 7fc51359e700 1 --2- 172.21.2.112:0/55184864 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc5000095a0 0x7fc50000bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:16.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.322 7fc51259c700 1 -- 172.21.2.112:0/55184864 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc5000095a0 msgr2=0x7fc50000bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:16.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.322 7fc51259c700 1 --2- 172.21.2.112:0/55184864 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc5000095a0 0x7fc50000bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:16.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.322 7fc51259c700 1 -- 172.21.2.112:0/55184864 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc500003a70 msgr2=0x7fc500005f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:16.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.322 7fc51259c700 1 --2- 172.21.2.112:0/55184864 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc500003a70 0x7fc500005f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:16.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.322 7fc51259c700 1 -- 172.21.2.112:0/55184864 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc50000c9f0 con 0x7fc500006780 2021-02-10T11:43:16.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.322 7fc51359e700 1 -- 172.21.2.112:0/55184864 reap_dead start 2021-02-10T11:43:16.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.322 7fc51259c700 1 --2- 172.21.2.112:0/55184864 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc500006780 0x7fc500008ca0 secure :-1 s=READY pgs=2454 cs=0 l=1 rev1=1 rx=0x7fc4fc006ab0 tx=0x7fc4fc04fb10).ready entity=mon.0 client_cookie=702db6fd51dc5917 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:16.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.322 7fc511d9b700 1 -- 172.21.2.112:0/55184864 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc4fc001290 con 0x7fc500006780 2021-02-10T11:43:16.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.322 7fc511d9b700 1 -- 172.21.2.112:0/55184864 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4fc0066f0 con 0x7fc500006780 2021-02-10T11:43:16.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.322 7fc511d9b700 1 -- 172.21.2.112:0/55184864 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc4fc001740 con 0x7fc500006780 2021-02-10T11:43:16.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.322 7fc51913c700 1 -- 172.21.2.112:0/55184864 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc500006780 msgr2=0x7fc500008ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:16.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.322 7fc51913c700 1 --2- 172.21.2.112:0/55184864 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc500006780 0x7fc500008ca0 secure :-1 s=READY pgs=2454 cs=0 l=1 rev1=1 rx=0x7fc4fc006ab0 tx=0x7fc4fc04fb10).stop 2021-02-10T11:43:16.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.323 7fc51913c700 1 -- 172.21.2.112:0/55184864 shutdown_connections 2021-02-10T11:43:16.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.323 7fc51913c700 1 --2- 172.21.2.112:0/55184864 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc500006780 0x7fc500008ca0 unknown :-1 s=CLOSED pgs=2454 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:16.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.323 7fc51913c700 1 -- 172.21.2.112:0/55184864 >> 172.21.2.112:0/55184864 conn(0x7fc5140ce410 msgr2=0x7fc5140d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:16.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.323 7fc51913c700 1 -- 172.21.2.112:0/55184864 shutdown_connections 2021-02-10T11:43:16.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.323 7fc51913c700 1 -- 172.21.2.112:0/55184864 wait complete. 2021-02-10T11:43:16.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.338 7fc51913c700 1 Processor -- start 2021-02-10T11:43:16.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.338 7fc51913c700 1 -- start start 2021-02-10T11:43:16.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.338 7fc51913c700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc5140adf50 0x7fc5140ae350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:16.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.338 7fc51913c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc5140ae970 0x7fc5140aed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:16.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.338 7fc51913c700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc5140982c0 0x7fc5140986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:16.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.338 7fc51913c700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc51409f450 con 0x7fc5140ae970 2021-02-10T11:43:16.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.338 7fc51913c700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc514098ce0 con 0x7fc5140982c0 2021-02-10T11:43:16.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.338 7fc51913c700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc514098ed0 con 0x7fc5140adf50 2021-02-10T11:43:16.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.338 7fc51259c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc5140ae970 0x7fc5140aed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:16.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.338 7fc51259c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc5140ae970 0x7fc5140aed70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:56984/0 (socket says 172.21.2.112:56984) 2021-02-10T11:43:16.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.338 7fc51259c700 1 -- 172.21.2.112:0/3074763755 learned_addr learned my addr 172.21.2.112:0/3074763755 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:43:16.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.339 7fc512d9d700 1 --2- 172.21.2.112:0/3074763755 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc5140adf50 0x7fc5140ae350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:16.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.339 7fc51359e700 1 --2- 172.21.2.112:0/3074763755 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc5140982c0 0x7fc5140986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:16.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.339 7fc51259c700 1 -- 172.21.2.112:0/3074763755 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc5140adf50 msgr2=0x7fc5140ae350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:16.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.339 7fc51259c700 1 --2- 172.21.2.112:0/3074763755 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc5140adf50 0x7fc5140ae350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:16.344 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.339 7fc51259c700 1 -- 172.21.2.112:0/3074763755 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc5140982c0 msgr2=0x7fc5140986c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:16.344 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.339 7fc51259c700 1 --2- 172.21.2.112:0/3074763755 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc5140982c0 0x7fc5140986c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:16.344 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.339 7fc51259c700 1 -- 172.21.2.112:0/3074763755 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5140990f0 con 0x7fc5140ae970 2021-02-10T11:43:16.344 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.339 7fc51359e700 1 --2- 172.21.2.112:0/3074763755 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc5140982c0 0x7fc5140986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:43:16.344 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.339 7fc51259c700 1 --2- 172.21.2.112:0/3074763755 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc5140ae970 0x7fc5140aed70 secure :-1 s=READY pgs=2455 cs=0 l=1 rev1=1 rx=0x7fc4fc04f6b0 tx=0x7fc4fc0099f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:16.344 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.339 7fc50b7fe700 1 -- 172.21.2.112:0/3074763755 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc4fc04e050 con 0x7fc5140ae970 2021-02-10T11:43:16.344 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.339 7fc50b7fe700 1 -- 172.21.2.112:0/3074763755 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4fc000cb0 con 0x7fc5140ae970 2021-02-10T11:43:16.344 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.339 7fc51913c700 1 -- 172.21.2.112:0/3074763755 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5140990f0 con 0x7fc5140ae970 2021-02-10T11:43:16.344 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.339 7fc51913c700 1 -- 172.21.2.112:0/3074763755 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc514099680 con 0x7fc5140ae970 2021-02-10T11:43:16.344 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.339 7fc50b7fe700 1 -- 172.21.2.112:0/3074763755 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc4fc04e440 con 0x7fc5140ae970 2021-02-10T11:43:16.345 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.340 7fc50b7fe700 1 -- 172.21.2.112:0/3074763755 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc4fc04e050 con 0x7fc5140ae970 2021-02-10T11:43:16.345 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.340 7fc50b7fe700 1 --2- 172.21.2.112:0/3074763755 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc4f8018730 0x7fc4f801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:16.345 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.340 7fc50b7fe700 1 -- 172.21.2.112:0/3074763755 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fc4fc076820 con 0x7fc5140ae970 2021-02-10T11:43:16.345 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.340 7fc512d9d700 1 --2- 172.21.2.112:0/3074763755 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc4f8018730 0x7fc4f801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:16.345 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.341 7fc512d9d700 1 --2- 172.21.2.112:0/3074763755 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc4f8018730 0x7fc4f801abc0 crc :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:16.371 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.371 7fc51913c700 1 -- 172.21.2.112:0/3074763755 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc514082a20 con 0x7fc5140ae970 2021-02-10T11:43:16.373 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.373 7fc50b7fe700 1 -- 172.21.2.112:0/3074763755 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc4fc007f00 con 0x7fc5140ae970 2021-02-10T11:43:16.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.501 7fc51913c700 1 -- 172.21.2.112:0/3074763755 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7fc51409f3d0 con 0x7fc5140ae970 2021-02-10T11:43:16.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.502 7fc50b7fe700 1 -- 172.21.2.112:0/3074763755 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v134) v1 ==== 92+0+10 (secure 0 0 0) 0x7fc4fc04e700 con 0x7fc5140ae970 2021-02-10T11:43:16.502 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:43:16.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.504 7fc51913c700 1 -- 172.21.2.112:0/3074763755 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc4f8018730 msgr2=0x7fc4f801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:16.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.504 7fc51913c700 1 --2- 172.21.2.112:0/3074763755 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc4f8018730 0x7fc4f801abc0 crc :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:16.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.504 7fc51913c700 1 -- 172.21.2.112:0/3074763755 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc5140ae970 msgr2=0x7fc5140aed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:16.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.504 7fc51913c700 1 --2- 172.21.2.112:0/3074763755 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc5140ae970 0x7fc5140aed70 secure :-1 s=READY pgs=2455 cs=0 l=1 rev1=1 rx=0x7fc4fc04f6b0 tx=0x7fc4fc0099f0).stop 2021-02-10T11:43:16.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.504 7fc51913c700 1 -- 172.21.2.112:0/3074763755 shutdown_connections 2021-02-10T11:43:16.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.504 7fc51913c700 1 --2- 172.21.2.112:0/3074763755 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc4f8018730 0x7fc4f801abc0 unknown :-1 s=CLOSED pgs=1120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:16.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.504 7fc51913c700 1 --2- 172.21.2.112:0/3074763755 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc5140982c0 0x7fc5140986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:16.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.504 7fc51913c700 1 --2- 172.21.2.112:0/3074763755 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc5140ae970 0x7fc5140aed70 unknown :-1 s=CLOSED pgs=2455 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:16.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.504 7fc51913c700 1 --2- 172.21.2.112:0/3074763755 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc5140adf50 0x7fc5140ae350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:16.505 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.504 7fc51913c700 1 -- 172.21.2.112:0/3074763755 >> 172.21.2.112:0/3074763755 conn(0x7fc5140bee90 msgr2=0x7fc5140bb880 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:16.505 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.504 7fc51913c700 1 -- 172.21.2.112:0/3074763755 shutdown_connections 2021-02-10T11:43:16.505 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.504 7fc51913c700 1 -- 172.21.2.112:0/3074763755 wait complete. 2021-02-10T11:43:16.512 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:43:16.943 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.927 7f9261ca8700 1 Processor -- start 2021-02-10T11:43:16.943 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.927 7f9261ca8700 1 -- start start 2021-02-10T11:43:16.944 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.928 7f9261ca8700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f925c0d1870 0x7f925c1b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:16.944 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.928 7f9261ca8700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f925c0cff70 0x7f925c109130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:16.944 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.928 7f9261ca8700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f925c0fe1c0 con 0x7f925c1b0ae0 2021-02-10T11:43:16.944 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.928 7f9261ca8700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f925c109ad0 con 0x7f925c0d1870 2021-02-10T11:43:16.944 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.928 7f9261ca8700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f925c109750 con 0x7f925c0cff70 2021-02-10T11:43:16.944 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.928 7f925b7fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f925c0d1870 0x7f925c1b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:16.944 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.928 7f925affd700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f925c1b0ae0 0x7f925c108940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:56450/0 (socket says 172.21.2.112:56450) 2021-02-10T11:43:16.944 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.928 7f925affd700 1 -- 172.21.2.112:0/1793660474 learned_addr learned my addr 172.21.2.112:0/1793660474 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:43:16.944 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.928 7f925b7fe700 1 -- 172.21.2.112:0/1793660474 >> v2:172.21.2.120:3300/0 conn(0x7f925c0cff70 msgr2=0x7f925c109130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:16.944 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.928 7f925b7fe700 1 --2- 172.21.2.112:0/1793660474 >> v2:172.21.2.120:3300/0 conn(0x7f925c0cff70 0x7f925c109130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:16.944 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.928 7f925b7fe700 1 -- 172.21.2.112:0/1793660474 >> v1:172.21.2.112:6789/0 conn(0x7f925c1b0ae0 legacy=0x7f925c108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:16.944 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.928 7f925a7fc700 1 -- 172.21.2.112:0/1793660474 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2642018783 0 0) 0x7f92440009b0 con 0x7f925c1b0ae0 2021-02-10T11:43:16.944 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.928 7f925b7fe700 1 -- 172.21.2.112:0/1793660474 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f925c088070 con 0x7f925c0d1870 2021-02-10T11:43:16.945 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.928 7f925b7fe700 1 --2- 172.21.2.112:0/1793660474 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f925c0d1870 0x7f925c1b04c0 secure :-1 s=READY pgs=2457 cs=0 l=1 rev1=1 rx=0x7f924c006650 tx=0x7f924c009a80).ready entity=mon.0 client_cookie=1e51831cf9a81e22 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:16.945 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.929 7f925a7fc700 1 -- 172.21.2.112:0/1793660474 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f924c0009b0 con 0x7f925c0d1870 2021-02-10T11:43:16.945 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.929 7f925a7fc700 1 -- 172.21.2.112:0/1793660474 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f924c00c0c0 con 0x7f925c0d1870 2021-02-10T11:43:16.945 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.929 7f925a7fc700 1 -- 172.21.2.112:0/1793660474 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f924c00c5e0 con 0x7f925c0d1870 2021-02-10T11:43:16.945 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.929 7f9261ca8700 1 -- 172.21.2.112:0/1793660474 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f925c0d1870 msgr2=0x7f925c1b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:16.945 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.929 7f9261ca8700 1 --2- 172.21.2.112:0/1793660474 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f925c0d1870 0x7f925c1b04c0 secure :-1 s=READY pgs=2457 cs=0 l=1 rev1=1 rx=0x7f924c006650 tx=0x7f924c009a80).stop 2021-02-10T11:43:16.945 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.929 7f9261ca8700 1 -- 172.21.2.112:0/1793660474 shutdown_connections 2021-02-10T11:43:16.945 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.929 7f9261ca8700 1 --2- 172.21.2.112:0/1793660474 >> v2:172.21.2.120:3300/0 conn(0x7f925c0cff70 0x7f925c109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:16.945 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.929 7f9261ca8700 1 --2- 172.21.2.112:0/1793660474 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f925c0d1870 0x7f925c1b04c0 unknown :-1 s=CLOSED pgs=2457 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:16.945 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.929 7f9261ca8700 1 -- 172.21.2.112:0/1793660474 >> 172.21.2.112:0/1793660474 conn(0x7f925c0ce420 msgr2=0x7f925c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:16.945 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.929 7f9261ca8700 1 -- 172.21.2.112:0/1793660474 shutdown_connections 2021-02-10T11:43:16.945 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.929 7f9261ca8700 1 -- 172.21.2.112:0/1793660474 wait complete. 2021-02-10T11:43:16.947 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.944 7f9261ca8700 1 Processor -- start 2021-02-10T11:43:16.947 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.944 7f9261ca8700 1 -- start start 2021-02-10T11:43:16.947 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.944 7f9261ca8700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f925c0c3820 0x7f925c0a7950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:16.947 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.944 7f9261ca8700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f925c1a44c0 0x7f925c1a48c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:16.947 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.944 7f9261ca8700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f925c0863d0 0x7f925c0867d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:16.948 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.944 7f9261ca8700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f925c085ed0 con 0x7f925c1a44c0 2021-02-10T11:43:16.948 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.944 7f9261ca8700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f925c0c0740 con 0x7f925c0863d0 2021-02-10T11:43:16.948 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.944 7f9261ca8700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f925c086df0 con 0x7f925c0c3820 2021-02-10T11:43:16.948 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.945 7f925affd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f925c1a44c0 0x7f925c1a48c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:16.948 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.945 7f925affd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f925c1a44c0 0x7f925c1a48c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:56992/0 (socket says 172.21.2.112:56992) 2021-02-10T11:43:16.948 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.945 7f925affd700 1 -- 172.21.2.112:0/1281775910 learned_addr learned my addr 172.21.2.112:0/1281775910 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:43:16.948 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.945 7f925b7fe700 1 --2- 172.21.2.112:0/1281775910 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f925c0c3820 0x7f925c0a7950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:16.948 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.945 7f925bfff700 1 --2- 172.21.2.112:0/1281775910 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f925c0863d0 0x7f925c0867d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:16.948 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.945 7f925affd700 1 -- 172.21.2.112:0/1281775910 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f925c0c3820 msgr2=0x7f925c0a7950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:16.948 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.945 7f925affd700 1 --2- 172.21.2.112:0/1281775910 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f925c0c3820 0x7f925c0a7950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:16.948 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.945 7f925affd700 1 -- 172.21.2.112:0/1281775910 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f925c0863d0 msgr2=0x7f925c0867d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:16.948 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.945 7f925affd700 1 --2- 172.21.2.112:0/1281775910 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f925c0863d0 0x7f925c0867d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:16.948 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.945 7f925affd700 1 -- 172.21.2.112:0/1281775910 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f925c087010 con 0x7f925c1a44c0 2021-02-10T11:43:16.949 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.945 7f925affd700 1 --2- 172.21.2.112:0/1281775910 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f925c1a44c0 0x7f925c1a48c0 secure :-1 s=READY pgs=2458 cs=0 l=1 rev1=1 rx=0x7f9244001c60 tx=0x7f924400a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:16.949 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.945 7f9253fff700 1 -- 172.21.2.112:0/1281775910 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f924400acc0 con 0x7f925c1a44c0 2021-02-10T11:43:16.949 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.945 7f9253fff700 1 -- 172.21.2.112:0/1281775910 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9244004140 con 0x7f925c1a44c0 2021-02-10T11:43:16.949 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.945 7f9253fff700 1 -- 172.21.2.112:0/1281775910 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9244004620 con 0x7f925c1a44c0 2021-02-10T11:43:16.949 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.945 7f9261ca8700 1 -- 172.21.2.112:0/1281775910 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f925c087010 con 0x7f925c1a44c0 2021-02-10T11:43:16.949 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.945 7f9261ca8700 1 -- 172.21.2.112:0/1281775910 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f925c087010 con 0x7f925c1a44c0 2021-02-10T11:43:16.949 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.946 7f9253fff700 1 -- 172.21.2.112:0/1281775910 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9244000dd0 con 0x7f925c1a44c0 2021-02-10T11:43:16.949 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.946 7f9253fff700 1 --2- 172.21.2.112:0/1281775910 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f92400186c0 0x7f924001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:16.949 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.946 7f9253fff700 1 -- 172.21.2.112:0/1281775910 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f92440327a0 con 0x7f925c1a44c0 2021-02-10T11:43:16.949 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.946 7f925b7fe700 1 --2- 172.21.2.112:0/1281775910 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f92400186c0 0x7f924001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:16.949 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.947 7f925b7fe700 1 --2- 172.21.2.112:0/1281775910 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f92400186c0 0x7f924001abc0 crc :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:16.978 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.978 7f9261ca8700 1 -- 172.21.2.112:0/1281775910 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f925c0fe1c0 con 0x7f925c1a44c0 2021-02-10T11:43:16.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:16.981 7f9253fff700 1 -- 172.21.2.112:0/1281775910 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f92440009b0 con 0x7f925c1a44c0 2021-02-10T11:43:17.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:17.108 7f9261ca8700 1 -- 172.21.2.112:0/1281775910 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f925c0fe1c0 con 0x7f925c1a44c0 2021-02-10T11:43:17.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:17.108 7f9253fff700 1 -- 172.21.2.112:0/1281775910 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v134) v1 ==== 104+0+10 (secure 0 0 0) 0x7f9244000dd0 con 0x7f925c1a44c0 2021-02-10T11:43:17.109 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:43:17.110 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:17.110 7f9261ca8700 1 -- 172.21.2.112:0/1281775910 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f92400186c0 msgr2=0x7f924001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:17.110 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:17.110 7f9261ca8700 1 --2- 172.21.2.112:0/1281775910 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f92400186c0 0x7f924001abc0 crc :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:17.110 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:17.110 7f9261ca8700 1 -- 172.21.2.112:0/1281775910 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f925c1a44c0 msgr2=0x7f925c1a48c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:17.110 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:17.110 7f9261ca8700 1 --2- 172.21.2.112:0/1281775910 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f925c1a44c0 0x7f925c1a48c0 secure :-1 s=READY pgs=2458 cs=0 l=1 rev1=1 rx=0x7f9244001c60 tx=0x7f924400a8d0).stop 2021-02-10T11:43:17.110 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:17.110 7f9261ca8700 1 -- 172.21.2.112:0/1281775910 shutdown_connections 2021-02-10T11:43:17.110 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:17.110 7f9261ca8700 1 --2- 172.21.2.112:0/1281775910 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f92400186c0 0x7f924001abc0 unknown :-1 s=CLOSED pgs=1121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:17.111 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:17.110 7f9261ca8700 1 --2- 172.21.2.112:0/1281775910 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f925c0863d0 0x7f925c0867d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:17.111 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:17.110 7f9261ca8700 1 --2- 172.21.2.112:0/1281775910 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f925c1a44c0 0x7f925c1a48c0 unknown :-1 s=CLOSED pgs=2458 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:17.111 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:17.110 7f9261ca8700 1 --2- 172.21.2.112:0/1281775910 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f925c0c3820 0x7f925c0a7950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:17.111 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:17.110 7f9261ca8700 1 -- 172.21.2.112:0/1281775910 >> 172.21.2.112:0/1281775910 conn(0x7f925c0872c0 msgr2=0x7f925c095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:17.111 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:17.110 7f9261ca8700 1 -- 172.21.2.112:0/1281775910 shutdown_connections 2021-02-10T11:43:17.111 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:17.110 7f9261ca8700 1 -- 172.21.2.112:0/1281775910 wait complete. 2021-02-10T11:43:17.119 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:43:17.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:17.534 7fe0d52c1700 1 Processor -- start 2021-02-10T11:43:17.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:17.535 7fe0d52c1700 1 -- start start 2021-02-10T11:43:17.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:17.535 7fe0d52c1700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fe0d00d1860 0x7fe0d01b04b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:17.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:17.535 7fe0d52c1700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0d00fe1b0 con 0x7fe0d00cff60 2021-02-10T11:43:17.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:17.535 7fe0d52c1700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0d0109a50 con 0x7fe0d01b0ad0 2021-02-10T11:43:17.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:17.535 7fe0d52c1700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0d0088070 con 0x7fe0d00d1860 2021-02-10T11:43:17.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:17.535 7fe0cf87e700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fe0d00cff60 0x7fe0d0109120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:56462/0 (socket says 172.21.2.112:56462) 2021-02-10T11:43:17.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:17.535 7fe0cf87e700 1 -- 172.21.2.112:0/3316757224 learned_addr learned my addr 172.21.2.112:0/3316757224 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:43:17.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:17.536 7fe0cf07d700 1 --2- 172.21.2.112:0/3316757224 >> v2:172.21.2.120:3300/0 conn(0x7fe0d00d1860 0x7fe0d01b04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:17.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:17.536 7fe0ce07b700 1 -- 172.21.2.112:0/3316757224 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3057984669 0 0) 0x7fe0c8001500 con 0x7fe0d00cff60 2021-02-10T11:43:17.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:17.536 7fe0ce07b700 1 -- 172.21.2.112:0/3316757224 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0b8002fe0 con 0x7fe0d00cff60 2021-02-10T11:43:17.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:17.536 7fe0ce07b700 1 -- 172.21.2.112:0/3316757224 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2961981547 0 0) 0x7fe0c8001270 con 0x7fe0d00cff60 2021-02-10T11:43:17.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:17.536 7fe0cf07d700 1 -- 172.21.2.112:0/3316757224 >> v1:172.21.2.120:6789/0 conn(0x7fe0d01b0ad0 legacy=0x7fe0d0108930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:17.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:17.536 7fe0cf07d700 1 -- 172.21.2.112:0/3316757224 >> v1:172.21.2.112:6789/0 conn(0x7fe0d00cff60 legacy=0x7fe0d0109120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:17.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:17.536 7fe0cf07d700 1 -- 172.21.2.112:0/3316757224 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0d0088260 con 0x7fe0d00d1860 2021-02-10T11:43:17.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:17.536 7fe0cf07d700 1 --2- 172.21.2.112:0/3316757224 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe0d00d1860 0x7fe0d01b04b0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 rx=0x7fe0c0006650 tx=0x7fe0c0009a00).ready entity=mon.1 client_cookie=b5f7aa79d5d67d1c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:17.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:17.537 7fe0ce07b700 1 -- 172.21.2.112:0/3316757224 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe0c0009df0 con 0x7fe0d00d1860 2021-02-10T11:43:17.553 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:17.537 7fe0ce07b700 1 -- 172.21.2.112:0/3316757224 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0c00009b0 con 0x7fe0d00d1860 2021-02-10T11:43:17.553 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:17.537 7fe0ce07b700 1 -- 172.21.2.112:0/3316757224 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe0c000c2b0 con 0x7fe0d00d1860 2021-02-10T11:43:17.553 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:17.537 7fe0d52c1700 1 -- 172.21.2.112:0/3316757224 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe0d00d1860 msgr2=0x7fe0d01b04b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:17.553 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:17.537 7fe0d52c1700 1 --2- 172.21.2.112:0/3316757224 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe0d00d1860 0x7fe0d01b04b0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 rx=0x7fe0c0006650 tx=0x7fe0c0009a00).stop 2021-02-10T11:43:17.553 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:17.537 7fe0d52c1700 1 -- 172.21.2.112:0/3316757224 shutdown_connections 2021-02-10T11:43:17.553 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:17.537 7fe0d52c1700 1 --2- 172.21.2.112:0/3316757224 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe0d00d1860 0x7fe0d01b04b0 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:17.553 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:17.537 7fe0d52c1700 1 -- 172.21.2.112:0/3316757224 >> 172.21.2.112:0/3316757224 conn(0x7fe0d00ce410 msgr2=0x7fe0d00d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:17.553 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:17.537 7fe0d52c1700 1 -- 172.21.2.112:0/3316757224 shutdown_connections 2021-02-10T11:43:17.553 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:17.537 7fe0d52c1700 1 -- 172.21.2.112:0/3316757224 wait complete. 2021-02-10T11:43:17.555 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:17.552 7fe0d52c1700 1 Processor -- start 2021-02-10T11:43:17.555 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:17.552 7fe0d52c1700 1 -- start start 2021-02-10T11:43:17.555 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:17.552 7fe0d52c1700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe0d00bfa10 0x7fe0d00b82a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:17.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:17.552 7fe0d52c1700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe0d01a4a70 0x7fe0d0094c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:17.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:17.552 7fe0d52c1700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe0d00c0410 0x7fe0d00c0810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:17.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:17.552 7fe0d52c1700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0d00c3a50 con 0x7fe0d01a4a70 2021-02-10T11:43:17.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:17.552 7fe0d52c1700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0d00b88c0 con 0x7fe0d00c0410 2021-02-10T11:43:17.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:17.552 7fe0d52c1700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0d0095260 con 0x7fe0d00bfa10 2021-02-10T11:43:17.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:17.552 7fe0ce87c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe0d01a4a70 0x7fe0d0094c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:17.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:17.552 7fe0ce87c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe0d01a4a70 0x7fe0d0094c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:57004/0 (socket says 172.21.2.112:57004) 2021-02-10T11:43:17.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:17.552 7fe0ce87c700 1 -- 172.21.2.112:0/1800410382 learned_addr learned my addr 172.21.2.112:0/1800410382 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:43:17.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:17.553 7fe0cf07d700 1 --2- 172.21.2.112:0/1800410382 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe0d00bfa10 0x7fe0d00b82a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:17.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:17.553 7fe0cf87e700 1 --2- 172.21.2.112:0/1800410382 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe0d00c0410 0x7fe0d00c0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:17.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:17.553 7fe0ce87c700 1 -- 172.21.2.112:0/1800410382 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe0d00bfa10 msgr2=0x7fe0d00b82a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:17.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:17.553 7fe0ce87c700 1 --2- 172.21.2.112:0/1800410382 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe0d00bfa10 0x7fe0d00b82a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:17.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:17.553 7fe0ce87c700 1 -- 172.21.2.112:0/1800410382 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe0d00c0410 msgr2=0x7fe0d00c0810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:17.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:17.553 7fe0ce87c700 1 --2- 172.21.2.112:0/1800410382 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe0d00c0410 0x7fe0d00c0810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:17.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:17.553 7fe0ce87c700 1 -- 172.21.2.112:0/1800410382 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0d00c3540 con 0x7fe0d01a4a70 2021-02-10T11:43:17.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:17.553 7fe0cf87e700 1 --2- 172.21.2.112:0/1800410382 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe0d00c0410 0x7fe0d00c0810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:43:17.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:17.553 7fe0ce87c700 1 --2- 172.21.2.112:0/1800410382 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe0d01a4a70 0x7fe0d0094c40 secure :-1 s=READY pgs=2460 cs=0 l=1 rev1=1 rx=0x7fe0c4006f10 tx=0x7fe0c400b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:17.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:17.553 7fe0b7fff700 1 -- 172.21.2.112:0/1800410382 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe0c40009b0 con 0x7fe0d01a4a70 2021-02-10T11:43:17.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:17.553 7fe0b7fff700 1 -- 172.21.2.112:0/1800410382 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0c400bd80 con 0x7fe0d01a4a70 2021-02-10T11:43:17.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:17.553 7fe0b7fff700 1 -- 172.21.2.112:0/1800410382 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe0c40013f0 con 0x7fe0d01a4a70 2021-02-10T11:43:17.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:17.553 7fe0d52c1700 1 -- 172.21.2.112:0/1800410382 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0d00c3540 con 0x7fe0d01a4a70 2021-02-10T11:43:17.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:17.553 7fe0d52c1700 1 -- 172.21.2.112:0/1800410382 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0d00c3540 con 0x7fe0d01a4a70 2021-02-10T11:43:17.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:17.554 7fe0b7fff700 1 -- 172.21.2.112:0/1800410382 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe0c4001200 con 0x7fe0d01a4a70 2021-02-10T11:43:17.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:17.554 7fe0b7fff700 1 --2- 172.21.2.112:0/1800410382 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe0b00186c0 0x7fe0b001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:17.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:17.554 7fe0b7fff700 1 -- 172.21.2.112:0/1800410382 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fe0c40327a0 con 0x7fe0d01a4a70 2021-02-10T11:43:17.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:17.554 7fe0cf07d700 1 --2- 172.21.2.112:0/1800410382 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe0b00186c0 0x7fe0b001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:17.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:17.555 7fe0cf07d700 1 --2- 172.21.2.112:0/1800410382 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe0b00186c0 0x7fe0b001abc0 crc :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:17.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:17.585 7fe0d52c1700 1 -- 172.21.2.112:0/1800410382 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0d00bfee0 con 0x7fe0d01a4a70 2021-02-10T11:43:17.588 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:17.588 7fe0b7fff700 1 -- 172.21.2.112:0/1800410382 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe0c4006ba0 con 0x7fe0d01a4a70 2021-02-10T11:43:17.716 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:17.716 7fe0d52c1700 1 -- 172.21.2.112:0/1800410382 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7fe0d00bfee0 con 0x7fe0d01a4a70 2021-02-10T11:43:17.717 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:17.716 7fe0b7fff700 1 -- 172.21.2.112:0/1800410382 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v134) v1 ==== 100+0+10 (secure 0 0 0) 0x7fe0c400bd80 con 0x7fe0d01a4a70 2021-02-10T11:43:17.717 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:43:17.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:17.718 7fe0d52c1700 1 -- 172.21.2.112:0/1800410382 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe0b00186c0 msgr2=0x7fe0b001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:17.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:17.718 7fe0d52c1700 1 --2- 172.21.2.112:0/1800410382 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe0b00186c0 0x7fe0b001abc0 crc :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:17.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:17.718 7fe0d52c1700 1 -- 172.21.2.112:0/1800410382 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe0d01a4a70 msgr2=0x7fe0d0094c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:17.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:17.718 7fe0d52c1700 1 --2- 172.21.2.112:0/1800410382 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe0d01a4a70 0x7fe0d0094c40 secure :-1 s=READY pgs=2460 cs=0 l=1 rev1=1 rx=0x7fe0c4006f10 tx=0x7fe0c400b990).stop 2021-02-10T11:43:17.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:17.719 7fe0d52c1700 1 -- 172.21.2.112:0/1800410382 shutdown_connections 2021-02-10T11:43:17.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:17.719 7fe0d52c1700 1 --2- 172.21.2.112:0/1800410382 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe0b00186c0 0x7fe0b001abc0 unknown :-1 s=CLOSED pgs=1122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:17.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:17.719 7fe0d52c1700 1 --2- 172.21.2.112:0/1800410382 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe0d00c0410 0x7fe0d00c0810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:17.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:17.719 7fe0d52c1700 1 --2- 172.21.2.112:0/1800410382 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe0d01a4a70 0x7fe0d0094c40 unknown :-1 s=CLOSED pgs=2460 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:17.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:17.719 7fe0d52c1700 1 --2- 172.21.2.112:0/1800410382 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe0d00bfa10 0x7fe0d00b82a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:17.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:17.719 7fe0d52c1700 1 -- 172.21.2.112:0/1800410382 >> 172.21.2.112:0/1800410382 conn(0x7fe0d00a6a80 msgr2=0x7fe0d00ae110 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:17.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:17.719 7fe0d52c1700 1 -- 172.21.2.112:0/1800410382 shutdown_connections 2021-02-10T11:43:17.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:17.719 7fe0d52c1700 1 -- 172.21.2.112:0/1800410382 wait complete. 2021-02-10T11:43:17.727 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:43:17.727 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:43:18.153 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:18.137 7f83b6f96700 1 Processor -- start 2021-02-10T11:43:18.153 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:18.137 7f83b6f96700 1 -- start start 2021-02-10T11:43:18.153 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:18.138 7f83b6f96700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f83b00d1870 0x7f83b01b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:18.153 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:18.138 7f83b6f96700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83b00f0b10 con 0x7f83b01b0a10 2021-02-10T11:43:18.154 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:18.138 7f83b6f96700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83b00d0310 con 0x7f83b00f0650 2021-02-10T11:43:18.154 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:18.138 7f83b6f96700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83b00d0530 con 0x7f83b00d1870 2021-02-10T11:43:18.154 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:18.138 7f83b4db2700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f83b00d1870 0x7f83b01b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:18.154 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:18.138 7f83affff700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f83b01b0a10 0x7f83b00cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:56474/0 (socket says 172.21.2.112:56474) 2021-02-10T11:43:18.154 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:18.138 7f83affff700 1 -- 172.21.2.112:0/760057682 learned_addr learned my addr 172.21.2.112:0/760057682 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:43:18.154 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:18.138 7f83b4db2700 1 -- 172.21.2.112:0/760057682 >> v1:172.21.2.120:6789/0 conn(0x7f83b00f0650 legacy=0x7f83b00cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:18.154 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:18.138 7f83b4db2700 1 -- 172.21.2.112:0/760057682 >> v1:172.21.2.112:6789/0 conn(0x7f83b01b0a10 legacy=0x7f83b00cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:18.154 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:18.138 7f83b4db2700 1 -- 172.21.2.112:0/760057682 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83b00d1c70 con 0x7f83b00d1870 2021-02-10T11:43:18.154 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:18.138 7f83b4db2700 1 --2- 172.21.2.112:0/760057682 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f83b00d1870 0x7f83b01b03f0 secure :-1 s=READY pgs=2462 cs=0 l=1 rev1=1 rx=0x7f83a0006650 tx=0x7f83a0009a80).ready entity=mon.0 client_cookie=8496ae0e6ea20f3d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:18.154 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:18.139 7f83af7fe700 1 -- 172.21.2.112:0/760057682 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f83a00009b0 con 0x7f83b00d1870 2021-02-10T11:43:18.154 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:18.139 7f83af7fe700 1 -- 172.21.2.112:0/760057682 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83a000c0c0 con 0x7f83b00d1870 2021-02-10T11:43:18.154 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:18.139 7f83af7fe700 1 -- 172.21.2.112:0/760057682 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f83a000c5e0 con 0x7f83b00d1870 2021-02-10T11:43:18.155 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:18.139 7f83b6f96700 1 -- 172.21.2.112:0/760057682 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f83b00d1870 msgr2=0x7f83b01b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:18.155 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:18.139 7f83b6f96700 1 --2- 172.21.2.112:0/760057682 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f83b00d1870 0x7f83b01b03f0 secure :-1 s=READY pgs=2462 cs=0 l=1 rev1=1 rx=0x7f83a0006650 tx=0x7f83a0009a80).stop 2021-02-10T11:43:18.155 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:18.139 7f83b6f96700 1 -- 172.21.2.112:0/760057682 shutdown_connections 2021-02-10T11:43:18.155 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:18.139 7f83b6f96700 1 --2- 172.21.2.112:0/760057682 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f83b00d1870 0x7f83b01b03f0 unknown :-1 s=CLOSED pgs=2462 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:18.155 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:18.139 7f83b6f96700 1 -- 172.21.2.112:0/760057682 >> 172.21.2.112:0/760057682 conn(0x7f83b00ce450 msgr2=0x7f83b00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:18.155 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:18.139 7f83b6f96700 1 -- 172.21.2.112:0/760057682 shutdown_connections 2021-02-10T11:43:18.155 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:18.139 7f83b6f96700 1 -- 172.21.2.112:0/760057682 wait complete. 2021-02-10T11:43:18.156 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:18.154 7f83b6f96700 1 Processor -- start 2021-02-10T11:43:18.157 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:18.154 7f83b6f96700 1 -- start start 2021-02-10T11:43:18.157 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:18.154 7f83b6f96700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f83b00ae6b0 0x7f83b00a8bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:18.157 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:18.154 7f83b6f96700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f83b00adbd0 0x7f83b00a9a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:18.157 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:18.154 7f83b6f96700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f83b00adfd0 0x7f83b00aa090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:18.157 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:18.154 7f83b6f96700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83b01bf350 con 0x7f83b00adbd0 2021-02-10T11:43:18.157 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:18.154 7f83b6f96700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83b00a9560 con 0x7f83b00adfd0 2021-02-10T11:43:18.157 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:18.154 7f83b6f96700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83b00a9210 con 0x7f83b00ae6b0 2021-02-10T11:43:18.157 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:18.155 7f83affff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f83b00adbd0 0x7f83b00a9a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:18.157 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:18.155 7f83affff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f83b00adbd0 0x7f83b00a9a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:57016/0 (socket says 172.21.2.112:57016) 2021-02-10T11:43:18.157 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:18.155 7f83affff700 1 -- 172.21.2.112:0/248991276 learned_addr learned my addr 172.21.2.112:0/248991276 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:43:18.157 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:18.155 7f83b4db2700 1 --2- 172.21.2.112:0/248991276 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f83b00ae6b0 0x7f83b00a8bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:18.158 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:18.155 7f83b55b3700 1 --2- 172.21.2.112:0/248991276 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f83b00adfd0 0x7f83b00aa090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:18.158 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:18.155 7f83affff700 1 -- 172.21.2.112:0/248991276 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f83b00ae6b0 msgr2=0x7f83b00a8bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:18.158 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:18.155 7f83affff700 1 --2- 172.21.2.112:0/248991276 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f83b00ae6b0 0x7f83b00a8bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:18.158 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:18.155 7f83affff700 1 -- 172.21.2.112:0/248991276 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f83b00adfd0 msgr2=0x7f83b00aa090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:18.158 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:18.155 7f83affff700 1 --2- 172.21.2.112:0/248991276 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f83b00adfd0 0x7f83b00aa090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:18.158 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:18.155 7f83affff700 1 -- 172.21.2.112:0/248991276 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83b00aeab0 con 0x7f83b00adbd0 2021-02-10T11:43:18.158 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:18.155 7f83b4db2700 1 --2- 172.21.2.112:0/248991276 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f83b00ae6b0 0x7f83b00a8bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:43:18.158 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:18.155 7f83affff700 1 --2- 172.21.2.112:0/248991276 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f83b00adbd0 0x7f83b00a9a70 secure :-1 s=READY pgs=2463 cs=0 l=1 rev1=1 rx=0x7f83a4001c60 tx=0x7f83a400a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:18.158 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:18.155 7f83ad7fa700 1 -- 172.21.2.112:0/248991276 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f83a400acc0 con 0x7f83b00adbd0 2021-02-10T11:43:18.158 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:18.156 7f83ad7fa700 1 -- 172.21.2.112:0/248991276 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83a4004140 con 0x7f83b00adbd0 2021-02-10T11:43:18.158 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:18.156 7f83b6f96700 1 -- 172.21.2.112:0/248991276 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83b00aeab0 con 0x7f83b00adbd0 2021-02-10T11:43:18.158 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:18.156 7f83b6f96700 1 -- 172.21.2.112:0/248991276 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83b00aa9a0 con 0x7f83b00adbd0 2021-02-10T11:43:18.158 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:18.156 7f83ad7fa700 1 -- 172.21.2.112:0/248991276 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f83a4004660 con 0x7f83b00adbd0 2021-02-10T11:43:18.159 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:18.156 7f83ad7fa700 1 -- 172.21.2.112:0/248991276 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f83a4004320 con 0x7f83b00adbd0 2021-02-10T11:43:18.159 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:18.156 7f83ad7fa700 1 --2- 172.21.2.112:0/248991276 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f83900186c0 0x7f839001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:18.159 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:18.156 7f83ad7fa700 1 -- 172.21.2.112:0/248991276 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f83a40327b0 con 0x7f83b00adbd0 2021-02-10T11:43:18.159 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:18.157 7f83b4db2700 1 --2- 172.21.2.112:0/248991276 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f83900186c0 0x7f839001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:18.159 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:18.157 7f83b4db2700 1 --2- 172.21.2.112:0/248991276 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f83900186c0 0x7f839001abc0 crc :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:18.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:18.188 7f83b6f96700 1 -- 172.21.2.112:0/248991276 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83b00e7de0 con 0x7f83b00adbd0 2021-02-10T11:43:18.190 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:18.190 7f83ad7fa700 1 -- 172.21.2.112:0/248991276 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f83a4000dd0 con 0x7f83b00adbd0 2021-02-10T11:43:18.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:18.320 7f83b6f96700 1 -- 172.21.2.112:0/248991276 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f83b00d1ee0 con 0x7f83b00adbd0 2021-02-10T11:43:18.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:18.320 7f83ad7fa700 1 -- 172.21.2.112:0/248991276 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 114 v114) v1 ==== 95+0+2108 (secure 0 0 0) 0x7f83a40042d0 con 0x7f83b00adbd0 2021-02-10T11:43:18.321 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:43:18.321 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":114,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11349,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/2093257968","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":2093257968},{"type":"v1","addr":"172.21.2.112:6833","nonce":2093257968}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":110}],"filesystems":[{"mdsmap":{"epoch":114,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:43:08.182625","modified":"2021-02-10 11:43:10.189979","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7025},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7025":{"gid":7025,"name":"b","rank":0,"incarnation":111,"state":"up:active","state_seq":3,"addr":"172.21.2.120:6835/2552826693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":2552826693},{"type":"v1","addr":"172.21.2.120:6835","nonce":2552826693}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":14}]} 2021-02-10T11:43:18.321 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 114 2021-02-10T11:43:18.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:18.322 7f83b6f96700 1 -- 172.21.2.112:0/248991276 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f83900186c0 msgr2=0x7f839001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:18.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:18.322 7f83b6f96700 1 --2- 172.21.2.112:0/248991276 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f83900186c0 0x7f839001abc0 crc :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:18.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:18.322 7f83b6f96700 1 -- 172.21.2.112:0/248991276 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f83b00adbd0 msgr2=0x7f83b00a9a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:18.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:18.322 7f83b6f96700 1 --2- 172.21.2.112:0/248991276 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f83b00adbd0 0x7f83b00a9a70 secure :-1 s=READY pgs=2463 cs=0 l=1 rev1=1 rx=0x7f83a4001c60 tx=0x7f83a400a8d0).stop 2021-02-10T11:43:18.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:18.323 7f83b6f96700 1 -- 172.21.2.112:0/248991276 shutdown_connections 2021-02-10T11:43:18.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:18.323 7f83b6f96700 1 --2- 172.21.2.112:0/248991276 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f83900186c0 0x7f839001abc0 unknown :-1 s=CLOSED pgs=1123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:18.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:18.323 7f83b6f96700 1 --2- 172.21.2.112:0/248991276 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f83b00adfd0 0x7f83b00aa090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:18.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:18.323 7f83b6f96700 1 --2- 172.21.2.112:0/248991276 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f83b00adbd0 0x7f83b00a9a70 unknown :-1 s=CLOSED pgs=2463 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:18.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:18.323 7f83b6f96700 1 --2- 172.21.2.112:0/248991276 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f83b00ae6b0 0x7f83b00a8bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:18.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:18.323 7f83b6f96700 1 -- 172.21.2.112:0/248991276 >> 172.21.2.112:0/248991276 conn(0x7f83b00a0030 msgr2=0x7f83b00eb1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:18.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:18.323 7f83b6f96700 1 -- 172.21.2.112:0/248991276 shutdown_connections 2021-02-10T11:43:18.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:18.323 7f83b6f96700 1 -- 172.21.2.112:0/248991276 wait complete. 2021-02-10T11:43:18.532 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 114, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:43:08.182625', 'modified': '2021-02-10 11:43:10.189979', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7025}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7025': {'gid': 7025, 'name': 'b', 'rank': 0, 'incarnation': 111, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.120:6835/2552826693', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.120:6834', 'nonce': 2552826693}, {'type': 'v1', 'addr': '172.21.2.120:6835', 'nonce': 2552826693}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [29], 'metadata_pool': 28, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:43:18.532 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:43:18.533 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:43:18.932 INFO:teuthology.orchestra.run.gibba020.stdout:{ 2021-02-10T11:43:18.933 INFO:teuthology.orchestra.run.gibba020.stdout: "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", 2021-02-10T11:43:18.933 INFO:teuthology.orchestra.run.gibba020.stdout: "whoami": 0, 2021-02-10T11:43:18.933 INFO:teuthology.orchestra.run.gibba020.stdout: "id": 7025, 2021-02-10T11:43:18.933 INFO:teuthology.orchestra.run.gibba020.stdout: "want_state": "up:active", 2021-02-10T11:43:18.933 INFO:teuthology.orchestra.run.gibba020.stdout: "state": "up:active", 2021-02-10T11:43:18.933 INFO:teuthology.orchestra.run.gibba020.stdout: "rank_uptime": 10.74734451, 2021-02-10T11:43:18.933 INFO:teuthology.orchestra.run.gibba020.stdout: "mdsmap_epoch": 114, 2021-02-10T11:43:18.933 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch": 134, 2021-02-10T11:43:18.933 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch_barrier": 134, 2021-02-10T11:43:18.933 INFO:teuthology.orchestra.run.gibba020.stdout: "uptime": 16.390003368999999 2021-02-10T11:43:18.933 INFO:teuthology.orchestra.run.gibba020.stdout:} 2021-02-10T11:43:18.936 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", "whoami": 0, "id": 7025, "want_state": "up:active", "state": "up:active", "rank_uptime": 10.74734451, "mdsmap_epoch": 114, "osdmap_epoch": 134, "osdmap_epoch_barrier": 134, "uptime": 16.390003368999999 } 2021-02-10T11:43:18.937 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:43:18.937 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2021-02-10T11:43:18.937 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@gibba012.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2021-02-10T11:43:18.937 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:43:18.937 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:43:19.273 DEBUG:teuthology.orchestra.run.gibba012:> sudo modprobe fuse 2021-02-10T11:43:19.618 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:43:19.619 DEBUG:teuthology.orchestra.run.gibba012:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:43:19.961 INFO:teuthology.orchestra.run.gibba012.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:43:19.961 INFO:teuthology.orchestra.run.gibba012.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:43:19.962 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:43:19.962 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:43:19.962 DEBUG:teuthology.orchestra.run.gibba012:> ls /sys/fs/fuse/connections 2021-02-10T11:43:20.298 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2021-02-10T11:43:20.298 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:43:20.301 DEBUG:teuthology.orchestra.run.gibba012:> sudo modprobe fuse 2021-02-10T11:43:20.645 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:43:20.645 DEBUG:teuthology.orchestra.run.gibba012:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:43:20.712 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba012.stderr:ceph-fuse[169337]: starting ceph client 2021-02-10T11:43:20.712 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba012.stderr:2021-02-10 11:43:20.712 7f85fe388f80 -1 init, newargv = 0x5651d0edc1a0 newargc=9 2021-02-10T11:43:20.716 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba012.stderr:ceph-fuse[169337]: starting fuse 2021-02-10T11:43:20.988 INFO:teuthology.orchestra.run.gibba012.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:43:20.989 INFO:teuthology.orchestra.run.gibba012.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:43:20.989 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:43:20.990 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:43:20.990 DEBUG:teuthology.orchestra.run.gibba012:> ls /sys/fs/fuse/connections 2021-02-10T11:43:21.325 INFO:teuthology.orchestra.run.gibba012.stdout:42 2021-02-10T11:43:21.325 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2021-02-10T11:43:21.326 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:43:21.326 DEBUG:teuthology.orchestra.run.gibba012:> sudo python3 -c ' 2021-02-10T11:43:21.326 DEBUG:teuthology.orchestra.run.gibba012:> import glob 2021-02-10T11:43:21.326 DEBUG:teuthology.orchestra.run.gibba012:> import re 2021-02-10T11:43:21.326 DEBUG:teuthology.orchestra.run.gibba012:> import os 2021-02-10T11:43:21.326 DEBUG:teuthology.orchestra.run.gibba012:> import subprocess 2021-02-10T11:43:21.326 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:43:21.326 DEBUG:teuthology.orchestra.run.gibba012:> def find_socket(client_name): 2021-02-10T11:43:21.326 DEBUG:teuthology.orchestra.run.gibba012:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2021-02-10T11:43:21.326 DEBUG:teuthology.orchestra.run.gibba012:> files = glob.glob(asok_path) 2021-02-10T11:43:21.326 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:43:21.327 DEBUG:teuthology.orchestra.run.gibba012:> # Given a non-glob path, it better be there 2021-02-10T11:43:21.327 DEBUG:teuthology.orchestra.run.gibba012:> if "*" not in asok_path: 2021-02-10T11:43:21.327 DEBUG:teuthology.orchestra.run.gibba012:> assert(len(files) == 1) 2021-02-10T11:43:21.327 DEBUG:teuthology.orchestra.run.gibba012:> return files[0] 2021-02-10T11:43:21.327 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:43:21.327 DEBUG:teuthology.orchestra.run.gibba012:> for f in files: 2021-02-10T11:43:21.327 DEBUG:teuthology.orchestra.run.gibba012:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:43:21.327 DEBUG:teuthology.orchestra.run.gibba012:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:43:21.327 DEBUG:teuthology.orchestra.run.gibba012:> return f 2021-02-10T11:43:21.327 DEBUG:teuthology.orchestra.run.gibba012:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:43:21.327 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:43:21.327 DEBUG:teuthology.orchestra.run.gibba012:> print(find_socket("client.0")) 2021-02-10T11:43:21.327 DEBUG:teuthology.orchestra.run.gibba012:> ' 2021-02-10T11:43:21.690 INFO:teuthology.orchestra.run.gibba012.stdout:/var/run/ceph/ceph-client.0.169337.asok 2021-02-10T11:43:21.693 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.169337.asok 2021-02-10T11:43:21.693 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:43:21.693 DEBUG:teuthology.orchestra.run.gibba012:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.169337.asok status 2021-02-10T11:43:22.084 INFO:teuthology.orchestra.run.gibba012.stdout:{ 2021-02-10T11:43:22.084 INFO:teuthology.orchestra.run.gibba012.stdout: "metadata": { 2021-02-10T11:43:22.084 INFO:teuthology.orchestra.run.gibba012.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:43:22.084 INFO:teuthology.orchestra.run.gibba012.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:43:22.084 INFO:teuthology.orchestra.run.gibba012.stdout: "entity_id": "0", 2021-02-10T11:43:22.084 INFO:teuthology.orchestra.run.gibba012.stdout: "hostname": "gibba012", 2021-02-10T11:43:22.085 INFO:teuthology.orchestra.run.gibba012.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2021-02-10T11:43:22.085 INFO:teuthology.orchestra.run.gibba012.stdout: "pid": "169337", 2021-02-10T11:43:22.085 INFO:teuthology.orchestra.run.gibba012.stdout: "root": "/" 2021-02-10T11:43:22.085 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:22.085 INFO:teuthology.orchestra.run.gibba012.stdout: "dentry_count": 0, 2021-02-10T11:43:22.085 INFO:teuthology.orchestra.run.gibba012.stdout: "dentry_pinned_count": 0, 2021-02-10T11:43:22.085 INFO:teuthology.orchestra.run.gibba012.stdout: "id": 11514, 2021-02-10T11:43:22.085 INFO:teuthology.orchestra.run.gibba012.stdout: "inst": { 2021-02-10T11:43:22.085 INFO:teuthology.orchestra.run.gibba012.stdout: "name": { 2021-02-10T11:43:22.086 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "client", 2021-02-10T11:43:22.086 INFO:teuthology.orchestra.run.gibba012.stdout: "num": 11514 2021-02-10T11:43:22.086 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:22.086 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": { 2021-02-10T11:43:22.086 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:43:22.086 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:0", 2021-02-10T11:43:22.086 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 517024409 2021-02-10T11:43:22.086 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:43:22.086 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:22.086 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": { 2021-02-10T11:43:22.087 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:43:22.087 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:0", 2021-02-10T11:43:22.087 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 517024409 2021-02-10T11:43:22.087 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:22.087 INFO:teuthology.orchestra.run.gibba012.stdout: "inst_str": "client.11514 172.21.2.112:0/517024409", 2021-02-10T11:43:22.087 INFO:teuthology.orchestra.run.gibba012.stdout: "addr_str": "172.21.2.112:0/517024409", 2021-02-10T11:43:22.087 INFO:teuthology.orchestra.run.gibba012.stdout: "inode_count": 1, 2021-02-10T11:43:22.087 INFO:teuthology.orchestra.run.gibba012.stdout: "mds_epoch": 114, 2021-02-10T11:43:22.087 INFO:teuthology.orchestra.run.gibba012.stdout: "osd_epoch": 134, 2021-02-10T11:43:22.088 INFO:teuthology.orchestra.run.gibba012.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:43:22.088 INFO:teuthology.orchestra.run.gibba012.stdout: "blacklisted": false 2021-02-10T11:43:22.088 INFO:teuthology.orchestra.run.gibba012.stdout:} 2021-02-10T11:43:22.089 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:43:22.089 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:43:22.089 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:43:22.426 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:43:22.426 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:43:22.426 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:43:22.426 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:43:22.773 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:43:23.200 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.183 7fdec68b2700 1 Processor -- start 2021-02-10T11:43:23.200 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.183 7fdec68b2700 1 -- start start 2021-02-10T11:43:23.200 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.184 7fdec68b2700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdec01b0470 0x7fdec01b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:23.200 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.184 7fdec68b2700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fdec00f20e0 0x7fdec00f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:23.200 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.184 7fdec68b2700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdec00f0b50 con 0x7fdec00ef610 2021-02-10T11:43:23.200 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.184 7fdec68b2700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdec00f0820 con 0x7fdec00f20e0 2021-02-10T11:43:23.200 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.184 7fdec68b2700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdec00f24e0 con 0x7fdec01b0470 2021-02-10T11:43:23.200 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.184 7fdec4ecf700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fdec00f20e0 0x7fdec00f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:23.200 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.184 7fdec4ecf700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fdec00f20e0 0x7fdec00f0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:57040/0 (socket says 172.21.2.112:57040) 2021-02-10T11:43:23.201 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.184 7fdec4ecf700 1 -- 172.21.2.112:0/980319456 learned_addr learned my addr 172.21.2.112:0/980319456 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:43:23.201 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.184 7fdec4ecf700 1 -- 172.21.2.112:0/980319456 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdec01b0470 msgr2=0x7fdec01b0870 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:43:23.201 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.184 7fdec4ecf700 1 --2- 172.21.2.112:0/980319456 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdec01b0470 0x7fdec01b0870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:23.201 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.184 7fdec4ecf700 1 -- 172.21.2.112:0/980319456 >> v1:172.21.2.112:6789/0 conn(0x7fdec00ef610 legacy=0x7fdec00efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:23.201 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.184 7fdec4ecf700 1 -- 172.21.2.112:0/980319456 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdec00e52e0 con 0x7fdec00f20e0 2021-02-10T11:43:23.201 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.184 7fdec4ecf700 1 --2- 172.21.2.112:0/980319456 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdec00f20e0 0x7fdec00f0200 secure :-1 s=READY pgs=2468 cs=0 l=1 rev1=1 rx=0x7fdeb8006f00 tx=0x7fdeb8009e90).ready entity=mon.0 client_cookie=7f74d7a92223de7b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:23.201 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.185 7fdebeffd700 1 -- 172.21.2.112:0/980319456 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdeb800b300 con 0x7fdec00f20e0 2021-02-10T11:43:23.201 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.185 7fdebeffd700 1 -- 172.21.2.112:0/980319456 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdeb800b640 con 0x7fdec00f20e0 2021-02-10T11:43:23.201 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.185 7fdebeffd700 1 -- 172.21.2.112:0/980319456 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdeb800ba90 con 0x7fdec00f20e0 2021-02-10T11:43:23.201 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.185 7fdec68b2700 1 -- 172.21.2.112:0/980319456 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdec00f20e0 msgr2=0x7fdec00f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:23.201 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.185 7fdec68b2700 1 --2- 172.21.2.112:0/980319456 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdec00f20e0 0x7fdec00f0200 secure :-1 s=READY pgs=2468 cs=0 l=1 rev1=1 rx=0x7fdeb8006f00 tx=0x7fdeb8009e90).stop 2021-02-10T11:43:23.201 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.185 7fdec68b2700 1 -- 172.21.2.112:0/980319456 shutdown_connections 2021-02-10T11:43:23.201 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.185 7fdec68b2700 1 --2- 172.21.2.112:0/980319456 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdec00f20e0 0x7fdec00f0200 unknown :-1 s=CLOSED pgs=2468 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:23.201 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.185 7fdec68b2700 1 --2- 172.21.2.112:0/980319456 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdec01b0470 0x7fdec01b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:23.202 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.185 7fdec68b2700 1 -- 172.21.2.112:0/980319456 >> 172.21.2.112:0/980319456 conn(0x7fdec00ce3e0 msgr2=0x7fdec00e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:23.202 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.185 7fdec68b2700 1 -- 172.21.2.112:0/980319456 shutdown_connections 2021-02-10T11:43:23.202 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.185 7fdec68b2700 1 -- 172.21.2.112:0/980319456 wait complete. 2021-02-10T11:43:23.203 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.200 7fdec68b2700 1 Processor -- start 2021-02-10T11:43:23.204 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.200 7fdec68b2700 1 -- start start 2021-02-10T11:43:23.204 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.200 7fdec68b2700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fdec01acc20 0x7fdec01ad020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:23.205 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.200 7fdec68b2700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdec01ad640 0x7fdec01ada40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:23.205 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.200 7fdec68b2700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdec0188110 0x7fdec0188510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:23.205 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.200 7fdec68b2700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdec01ac720 con 0x7fdec01ad640 2021-02-10T11:43:23.205 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.200 7fdec68b2700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdec00b1b10 con 0x7fdec01acc20 2021-02-10T11:43:23.205 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.200 7fdec68b2700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdec0188b30 con 0x7fdec0188110 2021-02-10T11:43:23.205 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.201 7fdebf7fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdec01ad640 0x7fdec01ada40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:23.205 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.201 7fdebf7fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdec01ad640 0x7fdec01ada40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:57046/0 (socket says 172.21.2.112:57046) 2021-02-10T11:43:23.205 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.201 7fdebf7fe700 1 -- 172.21.2.112:0/3445619915 learned_addr learned my addr 172.21.2.112:0/3445619915 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:43:23.205 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.201 7fdebf7fe700 1 -- 172.21.2.112:0/3445619915 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdec0188110 msgr2=0x7fdec0188510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:23.205 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.201 7fdebf7fe700 1 --2- 172.21.2.112:0/3445619915 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdec0188110 0x7fdec0188510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:23.205 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.201 7fdebf7fe700 1 -- 172.21.2.112:0/3445619915 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fdec01acc20 msgr2=0x7fdec01ad020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:23.205 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.201 7fdebf7fe700 1 --2- 172.21.2.112:0/3445619915 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fdec01acc20 0x7fdec01ad020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:23.205 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.201 7fdebf7fe700 1 -- 172.21.2.112:0/3445619915 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdec0188d50 con 0x7fdec01ad640 2021-02-10T11:43:23.206 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.201 7fdebf7fe700 1 --2- 172.21.2.112:0/3445619915 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdec01ad640 0x7fdec01ada40 secure :-1 s=READY pgs=2469 cs=0 l=1 rev1=1 rx=0x7fdeb4006ed0 tx=0x7fdeb400b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:23.206 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.202 7fdebcff9700 1 -- 172.21.2.112:0/3445619915 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdeb4006240 con 0x7fdec01ad640 2021-02-10T11:43:23.206 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.202 7fdebcff9700 1 -- 172.21.2.112:0/3445619915 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdeb40054b0 con 0x7fdec01ad640 2021-02-10T11:43:23.206 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.202 7fdebcff9700 1 -- 172.21.2.112:0/3445619915 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdeb4005990 con 0x7fdec01ad640 2021-02-10T11:43:23.206 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.202 7fdec68b2700 1 -- 172.21.2.112:0/3445619915 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdec0188d50 con 0x7fdec01ad640 2021-02-10T11:43:23.206 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.202 7fdec68b2700 1 -- 172.21.2.112:0/3445619915 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdec01892e0 con 0x7fdec01ad640 2021-02-10T11:43:23.206 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.202 7fdebcff9700 1 -- 172.21.2.112:0/3445619915 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fdeb4006030 con 0x7fdec01ad640 2021-02-10T11:43:23.206 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.202 7fdebcff9700 1 --2- 172.21.2.112:0/3445619915 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fdea00186c0 0x7fdea001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:23.206 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.202 7fdebcff9700 1 -- 172.21.2.112:0/3445619915 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fdeb4032510 con 0x7fdec01ad640 2021-02-10T11:43:23.206 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.203 7fdebffff700 1 --2- 172.21.2.112:0/3445619915 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fdea00186c0 0x7fdea001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:23.206 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.203 7fdebffff700 1 --2- 172.21.2.112:0/3445619915 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fdea00186c0 0x7fdea001abc0 crc :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:23.233 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.233 7fdec68b2700 1 -- 172.21.2.112:0/3445619915 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdec00eddf0 con 0x7fdec01ad640 2021-02-10T11:43:23.236 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.236 7fdebcff9700 1 -- 172.21.2.112:0/3445619915 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fdeb4030cc0 con 0x7fdec01ad640 2021-02-10T11:43:23.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.359 7fdec68b2700 1 -- 172.21.2.112:0/3445619915 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdec00e52e0 con 0x7fdec01ad640 2021-02-10T11:43:23.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.360 7fdebcff9700 1 -- 172.21.2.112:0/3445619915 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v134) v1 ==== 74+0+14007 (secure 0 0 0) 0x7fdeb4006030 con 0x7fdec01ad640 2021-02-10T11:43:23.360 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:43:23.361 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":134,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:43:09.185424","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":28,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:43:06.157376","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"133","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":29,"pool_name":"cephfs_data","create_time":"2021-02-10 11:43:07.178112","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"134","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":132,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":132,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":132,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":132,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:43:23.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.361 7fdec68b2700 1 -- 172.21.2.112:0/3445619915 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fdea00186c0 msgr2=0x7fdea001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:23.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.361 7fdec68b2700 1 --2- 172.21.2.112:0/3445619915 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fdea00186c0 0x7fdea001abc0 crc :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:23.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.361 7fdec68b2700 1 -- 172.21.2.112:0/3445619915 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdec01ad640 msgr2=0x7fdec01ada40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:23.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.361 7fdec68b2700 1 --2- 172.21.2.112:0/3445619915 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdec01ad640 0x7fdec01ada40 secure :-1 s=READY pgs=2469 cs=0 l=1 rev1=1 rx=0x7fdeb4006ed0 tx=0x7fdeb400b960).stop 2021-02-10T11:43:23.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.362 7fdec68b2700 1 -- 172.21.2.112:0/3445619915 shutdown_connections 2021-02-10T11:43:23.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.362 7fdec68b2700 1 --2- 172.21.2.112:0/3445619915 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fdea00186c0 0x7fdea001abc0 unknown :-1 s=CLOSED pgs=1124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:23.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.362 7fdec68b2700 1 --2- 172.21.2.112:0/3445619915 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdec0188110 0x7fdec0188510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:23.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.362 7fdec68b2700 1 --2- 172.21.2.112:0/3445619915 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdec01ad640 0x7fdec01ada40 unknown :-1 s=CLOSED pgs=2469 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:23.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.362 7fdec68b2700 1 --2- 172.21.2.112:0/3445619915 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fdec01acc20 0x7fdec01ad020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:23.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.362 7fdec68b2700 1 -- 172.21.2.112:0/3445619915 >> 172.21.2.112:0/3445619915 conn(0x7fdec019f7a0 msgr2=0x7fdec01922a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:23.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.362 7fdec68b2700 1 -- 172.21.2.112:0/3445619915 shutdown_connections 2021-02-10T11:43:23.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.362 7fdec68b2700 1 -- 172.21.2.112:0/3445619915 wait complete. 2021-02-10T11:43:23.370 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:13:57.508758', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 28, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:43:06.157376', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '133', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 29, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:43:07.178112', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '134', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:43:23.371 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:43:23.804 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.786 7f33996a1700 1 Processor -- start 2021-02-10T11:43:23.804 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.786 7f33996a1700 1 -- start start 2021-02-10T11:43:23.804 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.786 7f33996a1700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f339410aff0 0x7f33940cf750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:23.804 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.786 7f33996a1700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33940fe1c0 con 0x7f33940d1870 2021-02-10T11:43:23.805 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.786 7f33996a1700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33941b0cb0 con 0x7f33940d1c70 2021-02-10T11:43:23.805 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.786 7f33996a1700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3394109a60 con 0x7f339410aff0 2021-02-10T11:43:23.805 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.787 7f33932bf700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f33940d1870 0x7f33941b04c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:56512/0 (socket says 172.21.2.112:56512) 2021-02-10T11:43:23.805 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.787 7f33932bf700 1 -- 172.21.2.112:0/2982358421 learned_addr learned my addr 172.21.2.112:0/2982358421 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:43:23.805 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.787 7f33922bd700 1 -- 172.21.2.112:0/2982358421 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 14590159 0 0) 0x7f337c001180 con 0x7f33940d1870 2021-02-10T11:43:23.805 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.787 7f33922bd700 1 -- 172.21.2.112:0/2982358421 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3380002fe0 con 0x7f33940d1870 2021-02-10T11:43:23.805 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.787 7f3393ac0700 1 --2- 172.21.2.112:0/2982358421 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f339410aff0 0x7f33940cf750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:23.805 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.787 7f33922bd700 1 -- 172.21.2.112:0/2982358421 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2094887320 0 0) 0x7f337c001180 con 0x7f33940d1870 2021-02-10T11:43:23.805 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.787 7f33922bd700 1 -- 172.21.2.112:0/2982358421 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f339410aff0 msgr2=0x7f33940cf750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:23.805 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.787 7f33922bd700 1 --2- 172.21.2.112:0/2982358421 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f339410aff0 0x7f33940cf750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:23.805 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.787 7f33922bd700 1 -- 172.21.2.112:0/2982358421 >> v1:172.21.2.120:6789/0 conn(0x7f33940d1c70 legacy=0x7f33940cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:23.805 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.787 7f33922bd700 1 -- 172.21.2.112:0/2982358421 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f339410b3f0 con 0x7f33940d1870 2021-02-10T11:43:23.805 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.787 7f33922bd700 1 -- 172.21.2.112:0/2982358421 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f337c001680 con 0x7f33940d1870 2021-02-10T11:43:23.806 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.787 7f33922bd700 1 -- 172.21.2.112:0/2982358421 >> v1:172.21.2.112:6789/0 conn(0x7f33940d1870 legacy=0x7f33941b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:23.806 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.788 7f33922bd700 1 --2- 172.21.2.112:0/2982358421 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3380003af0 0x7f3380006010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:23.806 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.788 7f33922bd700 1 --2- 172.21.2.112:0/2982358421 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3380006830 0x7f3380008d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:23.806 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.788 7f33922bd700 1 --2- 172.21.2.112:0/2982358421 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3380009650 0x7f338000bb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:23.806 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.788 7f33922bd700 1 -- 172.21.2.112:0/2982358421 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f338000c370 con 0x7f3380006830 2021-02-10T11:43:23.806 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.788 7f33922bd700 1 -- 172.21.2.112:0/2982358421 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f338000c590 con 0x7f3380003af0 2021-02-10T11:43:23.806 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.788 7f33922bd700 1 -- 172.21.2.112:0/2982358421 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f338000c7b0 con 0x7f3380009650 2021-02-10T11:43:23.806 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.788 7f3393ac0700 1 --2- 172.21.2.112:0/2982358421 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f339410aff0 0x7f33940cf750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:43:23.806 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.788 7f3392abe700 1 --2- 172.21.2.112:0/2982358421 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3380006830 0x7f3380008d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:23.806 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.788 7f33932bf700 1 --2- 172.21.2.112:0/2982358421 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3380003af0 0x7f3380006010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:23.806 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.788 7f3393ac0700 1 --2- 172.21.2.112:0/2982358421 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3380009650 0x7f338000bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:23.806 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.788 7f3392abe700 1 -- 172.21.2.112:0/2982358421 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3380009650 msgr2=0x7f338000bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:23.806 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.788 7f3392abe700 1 --2- 172.21.2.112:0/2982358421 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3380009650 0x7f338000bb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:23.807 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.788 7f3392abe700 1 -- 172.21.2.112:0/2982358421 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3380003af0 msgr2=0x7f3380006010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:23.807 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.788 7f3392abe700 1 --2- 172.21.2.112:0/2982358421 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3380003af0 0x7f3380006010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:23.807 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.788 7f3392abe700 1 -- 172.21.2.112:0/2982358421 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f338000ca50 con 0x7f3380006830 2021-02-10T11:43:23.807 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.788 7f3392abe700 1 --2- 172.21.2.112:0/2982358421 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3380006830 0x7f3380008d50 secure :-1 s=READY pgs=2471 cs=0 l=1 rev1=1 rx=0x7f3384006ab0 tx=0x7f3384009b40).ready entity=mon.0 client_cookie=f2f6f5481aea8a2b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:23.807 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.788 7f33922bd700 1 -- 172.21.2.112:0/2982358421 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f338400b2e0 con 0x7f3380006830 2021-02-10T11:43:23.807 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.788 7f33922bd700 1 -- 172.21.2.112:0/2982358421 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3384000c60 con 0x7f3380006830 2021-02-10T11:43:23.807 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.788 7f33922bd700 1 -- 172.21.2.112:0/2982358421 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f338400b710 con 0x7f3380006830 2021-02-10T11:43:23.807 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.788 7f33996a1700 1 -- 172.21.2.112:0/2982358421 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3380006830 msgr2=0x7f3380008d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:23.807 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.788 7f33996a1700 1 --2- 172.21.2.112:0/2982358421 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3380006830 0x7f3380008d50 secure :-1 s=READY pgs=2471 cs=0 l=1 rev1=1 rx=0x7f3384006ab0 tx=0x7f3384009b40).stop 2021-02-10T11:43:23.807 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.788 7f3393ac0700 1 -- 172.21.2.112:0/2982358421 reap_dead start 2021-02-10T11:43:23.807 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.789 7f33996a1700 1 -- 172.21.2.112:0/2982358421 shutdown_connections 2021-02-10T11:43:23.807 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.789 7f33996a1700 1 -- 172.21.2.112:0/2982358421 >> 172.21.2.112:0/2982358421 conn(0x7f33940ce420 msgr2=0x7f33940d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:23.807 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.789 7f33996a1700 1 -- 172.21.2.112:0/2982358421 shutdown_connections 2021-02-10T11:43:23.807 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.789 7f33996a1700 1 -- 172.21.2.112:0/2982358421 wait complete. 2021-02-10T11:43:23.808 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.804 7f33996a1700 1 Processor -- start 2021-02-10T11:43:23.808 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.804 7f33996a1700 1 -- start start 2021-02-10T11:43:23.810 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.805 7f33996a1700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f339409bcc0 0x7f339409c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:23.810 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.805 7f33996a1700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f33940a0820 0x7f33940a0c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:23.810 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.805 7f33996a1700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f33940a1240 0x7f33940858d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:23.810 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.805 7f33996a1700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33940a66b0 con 0x7f33940a0820 2021-02-10T11:43:23.810 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.805 7f33996a1700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33941a49b0 con 0x7f339409bcc0 2021-02-10T11:43:23.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.805 7f33996a1700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33941a4b90 con 0x7f33940a1240 2021-02-10T11:43:23.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.805 7f3392abe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f33940a0820 0x7f33940a0c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:23.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.805 7f3392abe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f33940a0820 0x7f33940a0c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:57062/0 (socket says 172.21.2.112:57062) 2021-02-10T11:43:23.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.805 7f3392abe700 1 -- 172.21.2.112:0/669023393 learned_addr learned my addr 172.21.2.112:0/669023393 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:43:23.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.805 7f33932bf700 1 --2- 172.21.2.112:0/669023393 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f339409bcc0 0x7f339409c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:23.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.805 7f3393ac0700 1 --2- 172.21.2.112:0/669023393 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f33940a1240 0x7f33940858d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:23.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.805 7f3392abe700 1 -- 172.21.2.112:0/669023393 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f33940a1240 msgr2=0x7f33940858d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:23.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.805 7f3392abe700 1 --2- 172.21.2.112:0/669023393 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f33940a1240 0x7f33940858d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:23.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.805 7f3392abe700 1 -- 172.21.2.112:0/669023393 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f339409bcc0 msgr2=0x7f339409c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:23.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.805 7f3392abe700 1 --2- 172.21.2.112:0/669023393 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f339409bcc0 0x7f339409c0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:23.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.805 7f3392abe700 1 -- 172.21.2.112:0/669023393 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33940a1640 con 0x7f33940a0820 2021-02-10T11:43:23.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.805 7f33932bf700 1 --2- 172.21.2.112:0/669023393 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f339409bcc0 0x7f339409c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:43:23.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.805 7f3392abe700 1 --2- 172.21.2.112:0/669023393 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f33940a0820 0x7f33940a0c20 secure :-1 s=READY pgs=2472 cs=0 l=1 rev1=1 rx=0x7f3384006f50 tx=0x7f338400b9e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:23.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.806 7f338bfff700 1 -- 172.21.2.112:0/669023393 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f338400bdd0 con 0x7f33940a0820 2021-02-10T11:43:23.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.806 7f338bfff700 1 -- 172.21.2.112:0/669023393 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3384006030 con 0x7f33940a0820 2021-02-10T11:43:23.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.806 7f33996a1700 1 -- 172.21.2.112:0/669023393 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33940a1640 con 0x7f33940a0820 2021-02-10T11:43:23.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.806 7f33996a1700 1 -- 172.21.2.112:0/669023393 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33940861e0 con 0x7f33940a0820 2021-02-10T11:43:23.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.806 7f338bfff700 1 -- 172.21.2.112:0/669023393 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3384006550 con 0x7f33940a0820 2021-02-10T11:43:23.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.806 7f338bfff700 1 -- 172.21.2.112:0/669023393 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3384006210 con 0x7f33940a0820 2021-02-10T11:43:23.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.806 7f338bfff700 1 --2- 172.21.2.112:0/669023393 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f33780186c0 0x7f337801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:23.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.806 7f338bfff700 1 -- 172.21.2.112:0/669023393 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f3384032560 con 0x7f33940a0820 2021-02-10T11:43:23.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.807 7f33932bf700 1 --2- 172.21.2.112:0/669023393 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f33780186c0 0x7f337801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:23.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.807 7f33932bf700 1 --2- 172.21.2.112:0/669023393 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f33780186c0 0x7f337801abc0 crc :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:23.836 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.836 7f33996a1700 1 -- 172.21.2.112:0/669023393 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3394086a70 con 0x7f33940a0820 2021-02-10T11:43:23.839 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.839 7f338bfff700 1 -- 172.21.2.112:0/669023393 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3384030d10 con 0x7f33940a0820 2021-02-10T11:43:23.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.967 7f33996a1700 1 -- 172.21.2.112:0/669023393 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f3394086a70 con 0x7f33940a0820 2021-02-10T11:43:23.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.968 7f338bfff700 1 -- 172.21.2.112:0/669023393 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v134) v1 ==== 92+0+10 (secure 0 0 0) 0x7f338400bdd0 con 0x7f33940a0820 2021-02-10T11:43:23.969 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:43:23.971 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.969 7f33996a1700 1 -- 172.21.2.112:0/669023393 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f33780186c0 msgr2=0x7f337801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:23.971 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.969 7f33996a1700 1 --2- 172.21.2.112:0/669023393 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f33780186c0 0x7f337801abc0 crc :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:23.971 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.969 7f33996a1700 1 -- 172.21.2.112:0/669023393 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f33940a0820 msgr2=0x7f33940a0c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:23.971 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.969 7f33996a1700 1 --2- 172.21.2.112:0/669023393 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f33940a0820 0x7f33940a0c20 secure :-1 s=READY pgs=2472 cs=0 l=1 rev1=1 rx=0x7f3384006f50 tx=0x7f338400b9e0).stop 2021-02-10T11:43:23.971 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.970 7f33996a1700 1 -- 172.21.2.112:0/669023393 shutdown_connections 2021-02-10T11:43:23.971 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.970 7f33996a1700 1 --2- 172.21.2.112:0/669023393 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f33780186c0 0x7f337801abc0 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:23.972 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.970 7f33996a1700 1 --2- 172.21.2.112:0/669023393 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f33940a1240 0x7f33940858d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:23.972 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.970 7f33996a1700 1 --2- 172.21.2.112:0/669023393 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f33940a0820 0x7f33940a0c20 unknown :-1 s=CLOSED pgs=2472 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:23.972 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.970 7f33996a1700 1 --2- 172.21.2.112:0/669023393 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f339409bcc0 0x7f339409c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:23.972 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.970 7f33996a1700 1 -- 172.21.2.112:0/669023393 >> 172.21.2.112:0/669023393 conn(0x7f33940bef00 msgr2=0x7f339409d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:23.972 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.970 7f33996a1700 1 -- 172.21.2.112:0/669023393 shutdown_connections 2021-02-10T11:43:23.973 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:23.970 7f33996a1700 1 -- 172.21.2.112:0/669023393 wait complete. 2021-02-10T11:43:23.979 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:43:24.028 DEBUG:teuthology.orchestra.run.gibba012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:43:24.031 DEBUG:teuthology.orchestra.run.gibba020:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:43:24.412 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:24.395 7f8d9a64e700 1 Processor -- start 2021-02-10T11:43:24.412 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:24.395 7f8d9a64e700 1 -- start start 2021-02-10T11:43:24.412 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:24.395 7f8d9a64e700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8d940d1870 0x7f8d941b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:24.412 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:24.395 7f8d9a64e700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f8d940cff70 0x7f8d94109130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:24.412 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:24.396 7f8d9a64e700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d94109a60 con 0x7f8d941b0ae0 2021-02-10T11:43:24.412 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:24.396 7f8d9a64e700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d94109750 con 0x7f8d940cff70 2021-02-10T11:43:24.412 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:24.396 7f8d9a64e700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d94088070 con 0x7f8d940d1870 2021-02-10T11:43:24.412 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:24.396 7f8d98c6b700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f8d940cff70 0x7f8d94109130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:24.412 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:24.396 7f8d98c6b700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f8d940cff70 0x7f8d94109130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3300/0 says I am v2:172.21.2.112:59354/0 (socket says 172.21.2.112:59354) 2021-02-10T11:43:24.413 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:24.396 7f8d98c6b700 1 -- 172.21.2.112:0/3329371963 learned_addr learned my addr 172.21.2.112:0/3329371963 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:43:24.413 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:24.396 7f8d93fff700 1 --2- 172.21.2.112:0/3329371963 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8d940d1870 0x7f8d941b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:24.413 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:24.396 7f8d93fff700 1 -- 172.21.2.112:0/3329371963 >> v2:172.21.2.120:3300/0 conn(0x7f8d940cff70 msgr2=0x7f8d94109130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:24.413 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:24.396 7f8d93fff700 1 --2- 172.21.2.112:0/3329371963 >> v2:172.21.2.120:3300/0 conn(0x7f8d940cff70 0x7f8d94109130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:24.413 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:24.396 7f8d93fff700 1 -- 172.21.2.112:0/3329371963 >> v1:172.21.2.120:6789/0 conn(0x7f8d941b0ae0 legacy=0x7f8d94108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:24.413 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:24.396 7f8d93fff700 1 -- 172.21.2.112:0/3329371963 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d94088260 con 0x7f8d940d1870 2021-02-10T11:43:24.413 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:24.396 7f8d98c6b700 1 --2- 172.21.2.112:0/3329371963 >> v2:172.21.2.120:3300/0 conn(0x7f8d940cff70 0x7f8d94109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:43:24.413 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:24.397 7f8d93fff700 1 --2- 172.21.2.112:0/3329371963 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8d940d1870 0x7f8d941b04c0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7f8d84006650 tx=0x7f8d84009a80).ready entity=mon.2 client_cookie=55cb48a4d2f0cd4c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:24.413 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:24.397 7f8d92ffd700 1 -- 172.21.2.112:0/3329371963 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8d840009b0 con 0x7f8d940d1870 2021-02-10T11:43:24.413 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:24.397 7f8d92ffd700 1 -- 172.21.2.112:0/3329371963 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d8400c0c0 con 0x7f8d940d1870 2021-02-10T11:43:24.413 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:24.397 7f8d92ffd700 1 -- 172.21.2.112:0/3329371963 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8d8400c520 con 0x7f8d940d1870 2021-02-10T11:43:24.413 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:24.397 7f8d9a64e700 1 -- 172.21.2.112:0/3329371963 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8d940d1870 msgr2=0x7f8d941b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:24.413 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:24.397 7f8d9a64e700 1 --2- 172.21.2.112:0/3329371963 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8d940d1870 0x7f8d941b04c0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7f8d84006650 tx=0x7f8d84009a80).stop 2021-02-10T11:43:24.414 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:24.397 7f8d9a64e700 1 -- 172.21.2.112:0/3329371963 shutdown_connections 2021-02-10T11:43:24.414 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:24.397 7f8d9a64e700 1 --2- 172.21.2.112:0/3329371963 >> v2:172.21.2.120:3300/0 conn(0x7f8d940cff70 0x7f8d94109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:24.414 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:24.397 7f8d9a64e700 1 --2- 172.21.2.112:0/3329371963 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8d940d1870 0x7f8d941b04c0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:24.414 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:24.397 7f8d9a64e700 1 -- 172.21.2.112:0/3329371963 >> 172.21.2.112:0/3329371963 conn(0x7f8d940ce420 msgr2=0x7f8d940d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:24.414 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:24.397 7f8d9a64e700 1 -- 172.21.2.112:0/3329371963 shutdown_connections 2021-02-10T11:43:24.414 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:24.397 7f8d9a64e700 1 -- 172.21.2.112:0/3329371963 wait complete. 2021-02-10T11:43:24.416 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:24.412 7f8d9a64e700 1 Processor -- start 2021-02-10T11:43:24.416 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:24.412 7f8d9a64e700 1 -- start start 2021-02-10T11:43:24.416 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:24.413 7f8d9a64e700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8d940abd60 0x7f8d940ac160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:24.416 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:24.413 7f8d9a64e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8d940ac780 0x7f8d940acb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:24.416 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:24.413 7f8d9a64e700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8d94083910 0x7f8d94083d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:24.416 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:24.413 7f8d9a64e700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d94087180 con 0x7f8d940ac780 2021-02-10T11:43:24.417 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:24.413 7f8d9a64e700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d9418fcd0 con 0x7f8d940abd60 2021-02-10T11:43:24.417 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:24.413 7f8d9a64e700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d94084330 con 0x7f8d94083910 2021-02-10T11:43:24.417 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:24.413 7f8d937fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8d940ac780 0x7f8d940acb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:24.417 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:24.413 7f8d937fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8d940ac780 0x7f8d940acb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:57076/0 (socket says 172.21.2.112:57076) 2021-02-10T11:43:24.417 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:24.413 7f8d937fe700 1 -- 172.21.2.112:0/3693757978 learned_addr learned my addr 172.21.2.112:0/3693757978 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:43:24.417 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:24.413 7f8d93fff700 1 --2- 172.21.2.112:0/3693757978 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8d940abd60 0x7f8d940ac160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:24.417 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:24.413 7f8d98c6b700 1 --2- 172.21.2.112:0/3693757978 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8d94083910 0x7f8d94083d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:24.417 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:24.413 7f8d937fe700 1 -- 172.21.2.112:0/3693757978 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8d94083910 msgr2=0x7f8d94083d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:24.417 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:24.413 7f8d937fe700 1 --2- 172.21.2.112:0/3693757978 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8d94083910 0x7f8d94083d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:24.417 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:24.413 7f8d937fe700 1 -- 172.21.2.112:0/3693757978 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8d940abd60 msgr2=0x7f8d940ac160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:24.417 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:24.413 7f8d937fe700 1 --2- 172.21.2.112:0/3693757978 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8d940abd60 0x7f8d940ac160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:24.418 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:24.413 7f8d937fe700 1 -- 172.21.2.112:0/3693757978 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d94084550 con 0x7f8d940ac780 2021-02-10T11:43:24.418 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:24.413 7f8d98c6b700 1 --2- 172.21.2.112:0/3693757978 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8d94083910 0x7f8d94083d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:43:24.418 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:24.413 7f8d937fe700 1 --2- 172.21.2.112:0/3693757978 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8d940ac780 0x7f8d940acb80 secure :-1 s=READY pgs=2473 cs=0 l=1 rev1=1 rx=0x7f8d88006ed0 tx=0x7f8d8800b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:24.418 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:24.413 7f8d90ff9700 1 -- 172.21.2.112:0/3693757978 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8d88006240 con 0x7f8d940ac780 2021-02-10T11:43:24.418 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:24.413 7f8d90ff9700 1 -- 172.21.2.112:0/3693757978 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d880009b0 con 0x7f8d940ac780 2021-02-10T11:43:24.418 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:24.413 7f8d90ff9700 1 -- 172.21.2.112:0/3693757978 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8d88005730 con 0x7f8d940ac780 2021-02-10T11:43:24.419 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:24.413 7f8d9a64e700 1 -- 172.21.2.112:0/3693757978 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d94084550 con 0x7f8d940ac780 2021-02-10T11:43:24.419 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:24.414 7f8d9a64e700 1 -- 172.21.2.112:0/3693757978 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d94084550 con 0x7f8d940ac780 2021-02-10T11:43:24.419 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:24.414 7f8d90ff9700 1 -- 172.21.2.112:0/3693757978 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8d88006030 con 0x7f8d940ac780 2021-02-10T11:43:24.419 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:24.414 7f8d90ff9700 1 --2- 172.21.2.112:0/3693757978 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8d74018700 0x7f8d7401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:24.419 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:24.414 7f8d90ff9700 1 -- 172.21.2.112:0/3693757978 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f8d88032800 con 0x7f8d940ac780 2021-02-10T11:43:24.419 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:24.414 7f8d93fff700 1 --2- 172.21.2.112:0/3693757978 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8d74018700 0x7f8d7401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:24.419 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:24.415 7f8d93fff700 1 --2- 172.21.2.112:0/3693757978 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8d74018700 0x7f8d7401abc0 crc :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:24.445 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:24.445 7f8d9a64e700 1 -- 172.21.2.112:0/3693757978 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d94082980 con 0x7f8d940ac780 2021-02-10T11:43:24.447 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:24.447 7f8d90ff9700 1 -- 172.21.2.112:0/3693757978 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8d88000de0 con 0x7f8d940ac780 2021-02-10T11:43:24.576 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:24.575 7f8d9a64e700 1 -- 172.21.2.112:0/3693757978 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f8d94109a60 con 0x7f8d940ac780 2021-02-10T11:43:24.576 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:24.576 7f8d90ff9700 1 -- 172.21.2.112:0/3693757978 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v134) v1 ==== 104+0+10 (secure 0 0 0) 0x7f8d880009b0 con 0x7f8d940ac780 2021-02-10T11:43:24.576 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:43:24.578 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:24.578 7f8d9a64e700 1 -- 172.21.2.112:0/3693757978 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8d74018700 msgr2=0x7f8d7401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:24.578 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:24.578 7f8d9a64e700 1 --2- 172.21.2.112:0/3693757978 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8d74018700 0x7f8d7401abc0 crc :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:24.578 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:24.578 7f8d9a64e700 1 -- 172.21.2.112:0/3693757978 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8d940ac780 msgr2=0x7f8d940acb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:24.578 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:24.578 7f8d9a64e700 1 --2- 172.21.2.112:0/3693757978 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8d940ac780 0x7f8d940acb80 secure :-1 s=READY pgs=2473 cs=0 l=1 rev1=1 rx=0x7f8d88006ed0 tx=0x7f8d8800b960).stop 2021-02-10T11:43:24.578 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:24.578 7f8d9a64e700 1 -- 172.21.2.112:0/3693757978 shutdown_connections 2021-02-10T11:43:24.578 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:24.578 7f8d9a64e700 1 --2- 172.21.2.112:0/3693757978 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8d74018700 0x7f8d7401abc0 unknown :-1 s=CLOSED pgs=1126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:24.578 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:24.578 7f8d9a64e700 1 --2- 172.21.2.112:0/3693757978 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8d94083910 0x7f8d94083d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:24.579 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:24.578 7f8d9a64e700 1 --2- 172.21.2.112:0/3693757978 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8d940ac780 0x7f8d940acb80 unknown :-1 s=CLOSED pgs=2473 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:24.579 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:24.578 7f8d9a64e700 1 --2- 172.21.2.112:0/3693757978 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8d940abd60 0x7f8d940ac160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:24.579 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:24.578 7f8d9a64e700 1 -- 172.21.2.112:0/3693757978 >> 172.21.2.112:0/3693757978 conn(0x7f8d940a2b30 msgr2=0x7f8d9409ce60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:24.579 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:24.578 7f8d9a64e700 1 -- 172.21.2.112:0/3693757978 shutdown_connections 2021-02-10T11:43:24.579 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:24.578 7f8d9a64e700 1 -- 172.21.2.112:0/3693757978 wait complete. 2021-02-10T11:43:24.586 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:43:25.018 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.001 7ff0e28df700 1 Processor -- start 2021-02-10T11:43:25.018 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.001 7ff0e28df700 1 -- start start 2021-02-10T11:43:25.018 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.002 7ff0e28df700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff0dc0d1c70 0x7ff0dc0cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:25.018 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.002 7ff0e28df700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7ff0dc0cf580 0x7ff0dc0cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:25.019 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.002 7ff0e28df700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0dc0fe1c0 con 0x7ff0dc0d1870 2021-02-10T11:43:25.019 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.002 7ff0e28df700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0dc1b0cb0 con 0x7ff0dc0cf580 2021-02-10T11:43:25.019 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.002 7ff0e28df700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0dc109a60 con 0x7ff0dc0d1c70 2021-02-10T11:43:25.019 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.002 7ff0dbfff700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7ff0dc0d1870 0x7ff0dc1b04c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:50104/0 (socket says 172.21.2.112:50104) 2021-02-10T11:43:25.019 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.002 7ff0db7fe700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff0dc0d1c70 0x7ff0dc0cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:25.019 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.002 7ff0dbfff700 1 -- 172.21.2.112:0/2457248666 learned_addr learned my addr 172.21.2.112:0/2457248666 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:43:25.019 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.002 7ff0e0efc700 1 --2- 172.21.2.112:0/2457248666 >> v2:172.21.2.120:3300/0 conn(0x7ff0dc0cf580 0x7ff0dc0cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:25.019 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.003 7ff0e0efc700 1 -- 172.21.2.112:0/2457248666 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff0dc0d1c70 msgr2=0x7ff0dc0cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:25.019 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.003 7ff0e0efc700 1 --2- 172.21.2.112:0/2457248666 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff0dc0d1c70 0x7ff0dc0cef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:25.019 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.003 7ff0e0efc700 1 -- 172.21.2.112:0/2457248666 >> v1:172.21.2.120:6789/0 conn(0x7ff0dc0d1870 legacy=0x7ff0dc1b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:25.019 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.003 7ff0e0efc700 1 -- 172.21.2.112:0/2457248666 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0dc088070 con 0x7ff0dc0cf580 2021-02-10T11:43:25.019 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.003 7ff0e0efc700 1 --2- 172.21.2.112:0/2457248666 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff0dc0cf580 0x7ff0dc0cf980 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 rx=0x7ff0d4006e50 tx=0x7ff0d4009e70).ready entity=mon.1 client_cookie=f58ac64e7d9b4c31 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:25.019 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.003 7ff0daffd700 1 -- 172.21.2.112:0/2457248666 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff0d400b510 con 0x7ff0dc0cf580 2021-02-10T11:43:25.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.003 7ff0daffd700 1 -- 172.21.2.112:0/2457248666 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0d4006790 con 0x7ff0dc0cf580 2021-02-10T11:43:25.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.003 7ff0daffd700 1 -- 172.21.2.112:0/2457248666 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff0d400b9d0 con 0x7ff0dc0cf580 2021-02-10T11:43:25.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.004 7ff0e28df700 1 -- 172.21.2.112:0/2457248666 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff0dc0cf580 msgr2=0x7ff0dc0cf980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:25.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.004 7ff0e28df700 1 --2- 172.21.2.112:0/2457248666 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff0dc0cf580 0x7ff0dc0cf980 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 rx=0x7ff0d4006e50 tx=0x7ff0d4009e70).stop 2021-02-10T11:43:25.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.004 7ff0e28df700 1 -- 172.21.2.112:0/2457248666 shutdown_connections 2021-02-10T11:43:25.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.004 7ff0e28df700 1 --2- 172.21.2.112:0/2457248666 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff0dc0cf580 0x7ff0dc0cf980 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:25.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.004 7ff0e28df700 1 --2- 172.21.2.112:0/2457248666 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff0dc0d1c70 0x7ff0dc0cef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:25.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.004 7ff0e28df700 1 -- 172.21.2.112:0/2457248666 >> 172.21.2.112:0/2457248666 conn(0x7ff0dc0ce420 msgr2=0x7ff0dc0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:25.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.004 7ff0e28df700 1 -- 172.21.2.112:0/2457248666 shutdown_connections 2021-02-10T11:43:25.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.004 7ff0e28df700 1 -- 172.21.2.112:0/2457248666 wait complete. 2021-02-10T11:43:25.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.018 7ff0e28df700 1 Processor -- start 2021-02-10T11:43:25.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.019 7ff0e28df700 1 -- start start 2021-02-10T11:43:25.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.019 7ff0e28df700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff0dc085bc0 0x7ff0dc0863e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:25.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.019 7ff0e28df700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff0dc0954c0 0x7ff0dc096300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:25.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.019 7ff0e28df700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff0dc0958c0 0x7ff0dc096920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:25.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.019 7ff0e28df700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0dc095d80 con 0x7ff0dc0958c0 2021-02-10T11:43:25.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.019 7ff0e28df700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0dc0bfa60 con 0x7ff0dc0954c0 2021-02-10T11:43:25.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.019 7ff0e28df700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0dc085fc0 con 0x7ff0dc085bc0 2021-02-10T11:43:25.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.019 7ff0e0efc700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff0dc0958c0 0x7ff0dc096920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:25.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.019 7ff0e0efc700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff0dc0958c0 0x7ff0dc096920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:57090/0 (socket says 172.21.2.112:57090) 2021-02-10T11:43:25.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.019 7ff0e0efc700 1 -- 172.21.2.112:0/1482166267 learned_addr learned my addr 172.21.2.112:0/1482166267 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:43:25.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.019 7ff0db7fe700 1 --2- 172.21.2.112:0/1482166267 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff0dc0954c0 0x7ff0dc096300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:25.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.019 7ff0dbfff700 1 --2- 172.21.2.112:0/1482166267 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff0dc085bc0 0x7ff0dc0863e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:25.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.020 7ff0db7fe700 1 -- 172.21.2.112:0/1482166267 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff0dc085bc0 msgr2=0x7ff0dc0863e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:25.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.020 7ff0db7fe700 1 --2- 172.21.2.112:0/1482166267 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff0dc085bc0 0x7ff0dc0863e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:25.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.020 7ff0db7fe700 1 -- 172.21.2.112:0/1482166267 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff0dc0958c0 msgr2=0x7ff0dc096920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:25.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.020 7ff0db7fe700 1 --2- 172.21.2.112:0/1482166267 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff0dc0958c0 0x7ff0dc096920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:25.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.020 7ff0db7fe700 1 -- 172.21.2.112:0/1482166267 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0dc086c10 con 0x7ff0dc0954c0 2021-02-10T11:43:25.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.020 7ff0e0efc700 1 --2- 172.21.2.112:0/1482166267 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff0dc0958c0 0x7ff0dc096920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:43:25.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.020 7ff0db7fe700 1 --2- 172.21.2.112:0/1482166267 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff0dc0954c0 0x7ff0dc096300 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 rx=0x7ff0d0009540 tx=0x7ff0d000b8e0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:25.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.020 7ff0d8ff9700 1 -- 172.21.2.112:0/1482166267 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff0d00009b0 con 0x7ff0dc0954c0 2021-02-10T11:43:25.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.020 7ff0d8ff9700 1 -- 172.21.2.112:0/1482166267 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0d0006340 con 0x7ff0dc0954c0 2021-02-10T11:43:25.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.020 7ff0e28df700 1 -- 172.21.2.112:0/1482166267 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0dc086c10 con 0x7ff0dc0954c0 2021-02-10T11:43:25.024 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.020 7ff0d8ff9700 1 -- 172.21.2.112:0/1482166267 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff0d00009b0 con 0x7ff0dc0954c0 2021-02-10T11:43:25.024 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.020 7ff0e28df700 1 -- 172.21.2.112:0/1482166267 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0dc086c10 con 0x7ff0dc0954c0 2021-02-10T11:43:25.024 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.021 7ff0d8ff9700 1 -- 172.21.2.112:0/1482166267 <== mon.1 v2:172.21.2.120:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff0d00052e0 con 0x7ff0dc0954c0 2021-02-10T11:43:25.024 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.021 7ff0d8ff9700 1 --2- 172.21.2.112:0/1482166267 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff0bc0186c0 0x7ff0bc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:25.024 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.021 7ff0d8ff9700 1 -- 172.21.2.112:0/1482166267 <== mon.1 v2:172.21.2.120:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5573+0+0 (secure 0 0 0) 0x7ff0d0032330 con 0x7ff0dc0954c0 2021-02-10T11:43:25.024 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.021 7ff0dbfff700 1 --2- 172.21.2.112:0/1482166267 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff0bc0186c0 0x7ff0bc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:25.024 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.022 7ff0dbfff700 1 --2- 172.21.2.112:0/1482166267 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff0bc0186c0 0x7ff0bc01abc0 crc :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:25.051 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.052 7ff0e28df700 1 -- 172.21.2.112:0/1482166267 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0dc0fe1c0 con 0x7ff0dc0954c0 2021-02-10T11:43:25.055 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.055 7ff0d8ff9700 1 -- 172.21.2.112:0/1482166267 <== mon.1 v2:172.21.2.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff0d0000c90 con 0x7ff0dc0954c0 2021-02-10T11:43:25.186 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.186 7ff0e28df700 1 -- 172.21.2.112:0/1482166267 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7ff0dc0fe1c0 con 0x7ff0dc0954c0 2021-02-10T11:43:25.186 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.186 7ff0d8ff9700 1 -- 172.21.2.112:0/1482166267 <== mon.1 v2:172.21.2.120:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v134) v1 ==== 100+0+10 (secure 0 0 0) 0x7ff0d0000c90 con 0x7ff0dc0954c0 2021-02-10T11:43:25.187 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:43:25.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.188 7ff0e28df700 1 -- 172.21.2.112:0/1482166267 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff0bc0186c0 msgr2=0x7ff0bc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:25.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.188 7ff0e28df700 1 --2- 172.21.2.112:0/1482166267 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff0bc0186c0 0x7ff0bc01abc0 crc :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:25.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.188 7ff0e28df700 1 -- 172.21.2.112:0/1482166267 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff0dc0954c0 msgr2=0x7ff0dc096300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:25.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.188 7ff0e28df700 1 --2- 172.21.2.112:0/1482166267 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff0dc0954c0 0x7ff0dc096300 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 rx=0x7ff0d0009540 tx=0x7ff0d000b8e0).stop 2021-02-10T11:43:25.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.188 7ff0e28df700 1 -- 172.21.2.112:0/1482166267 shutdown_connections 2021-02-10T11:43:25.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.188 7ff0e28df700 1 --2- 172.21.2.112:0/1482166267 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff0bc0186c0 0x7ff0bc01abc0 unknown :-1 s=CLOSED pgs=1127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:25.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.188 7ff0e28df700 1 --2- 172.21.2.112:0/1482166267 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff0dc0954c0 0x7ff0dc096300 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:25.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.188 7ff0e28df700 1 --2- 172.21.2.112:0/1482166267 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff0dc0958c0 0x7ff0dc096920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:25.189 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.188 7ff0e28df700 1 --2- 172.21.2.112:0/1482166267 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff0dc085bc0 0x7ff0dc0863e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:25.189 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.188 7ff0e28df700 1 -- 172.21.2.112:0/1482166267 >> 172.21.2.112:0/1482166267 conn(0x7ff0dc0bef00 msgr2=0x7ff0dc09d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:25.189 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.188 7ff0e28df700 1 -- 172.21.2.112:0/1482166267 shutdown_connections 2021-02-10T11:43:25.189 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.188 7ff0e28df700 1 -- 172.21.2.112:0/1482166267 wait complete. 2021-02-10T11:43:25.197 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:43:25.197 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:43:25.624 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.608 7fa6368b8700 1 Processor -- start 2021-02-10T11:43:25.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.609 7fa6368b8700 1 -- start start 2021-02-10T11:43:25.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.609 7fa6368b8700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fa6300d1870 0x7fa6301b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:25.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.609 7fa6368b8700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fa6300f0650 0x7fa6300cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:25.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.609 7fa6368b8700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6300f0b10 con 0x7fa6301b0a10 2021-02-10T11:43:25.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.609 7fa6368b8700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6300e5ed0 con 0x7fa6300f0650 2021-02-10T11:43:25.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.609 7fa6368b8700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6300d1c70 con 0x7fa6300d1870 2021-02-10T11:43:25.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.609 7fa634ed5700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fa6300f0650 0x7fa6300cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:25.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.609 7fa634ed5700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fa6300f0650 0x7fa6300cfb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:57094/0 (socket says 172.21.2.112:57094) 2021-02-10T11:43:25.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.609 7fa634ed5700 1 -- 172.21.2.112:0/2133962587 learned_addr learned my addr 172.21.2.112:0/2133962587 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:43:25.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.609 7fa634ed5700 1 -- 172.21.2.112:0/2133962587 >> v2:172.21.2.120:3300/0 conn(0x7fa6300d1870 msgr2=0x7fa6301b03f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:43:25.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.609 7fa634ed5700 1 --2- 172.21.2.112:0/2133962587 >> v2:172.21.2.120:3300/0 conn(0x7fa6300d1870 0x7fa6301b03f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:25.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.609 7fa634ed5700 1 -- 172.21.2.112:0/2133962587 >> v1:172.21.2.120:6789/0 conn(0x7fa6301b0a10 legacy=0x7fa6300cf330 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:43:25.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.609 7fa634ed5700 1 -- 172.21.2.112:0/2133962587 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6300d1e90 con 0x7fa6300f0650 2021-02-10T11:43:25.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.609 7fa634ed5700 1 --2- 172.21.2.112:0/2133962587 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa6300f0650 0x7fa6300cfb20 secure :-1 s=READY pgs=2474 cs=0 l=1 rev1=1 rx=0x7fa628006f00 tx=0x7fa628009f10).ready entity=mon.0 client_cookie=120964e38557e86a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:25.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.610 7fa62effd700 1 -- 172.21.2.112:0/2133962587 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa62800b3b0 con 0x7fa6300f0650 2021-02-10T11:43:25.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.610 7fa62effd700 1 -- 172.21.2.112:0/2133962587 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa62800b730 con 0x7fa6300f0650 2021-02-10T11:43:25.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.610 7fa62effd700 1 -- 172.21.2.112:0/2133962587 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa62800bc10 con 0x7fa6300f0650 2021-02-10T11:43:25.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.610 7fa6368b8700 1 -- 172.21.2.112:0/2133962587 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa6300f0650 msgr2=0x7fa6300cfb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:25.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.610 7fa6368b8700 1 --2- 172.21.2.112:0/2133962587 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa6300f0650 0x7fa6300cfb20 secure :-1 s=READY pgs=2474 cs=0 l=1 rev1=1 rx=0x7fa628006f00 tx=0x7fa628009f10).stop 2021-02-10T11:43:25.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.610 7fa6368b8700 1 -- 172.21.2.112:0/2133962587 shutdown_connections 2021-02-10T11:43:25.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.610 7fa6368b8700 1 --2- 172.21.2.112:0/2133962587 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa6300f0650 0x7fa6300cfb20 unknown :-1 s=CLOSED pgs=2474 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:25.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.610 7fa6368b8700 1 --2- 172.21.2.112:0/2133962587 >> v2:172.21.2.120:3300/0 conn(0x7fa6300d1870 0x7fa6301b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:25.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.610 7fa6368b8700 1 -- 172.21.2.112:0/2133962587 >> 172.21.2.112:0/2133962587 conn(0x7fa6300ce450 msgr2=0x7fa6300d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:25.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.610 7fa6368b8700 1 -- 172.21.2.112:0/2133962587 shutdown_connections 2021-02-10T11:43:25.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.610 7fa6368b8700 1 -- 172.21.2.112:0/2133962587 wait complete. 2021-02-10T11:43:25.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.625 7fa6368b8700 1 Processor -- start 2021-02-10T11:43:25.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.625 7fa6368b8700 1 -- start start 2021-02-10T11:43:25.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.625 7fa6368b8700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa63018a540 0x7fa630197480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:25.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.625 7fa6368b8700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa63018aa50 0x7fa6300b77d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:25.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.625 7fa6368b8700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa6300b7df0 0x7fa6300b81f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:25.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.625 7fa6368b8700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6301a79d0 con 0x7fa6300b7df0 2021-02-10T11:43:25.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.625 7fa6368b8700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6301a9e20 con 0x7fa63018a540 2021-02-10T11:43:25.629 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.625 7fa6368b8700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6301aa780 con 0x7fa63018aa50 2021-02-10T11:43:25.629 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.625 7fa634ed5700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa6300b7df0 0x7fa6300b81f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:25.629 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.625 7fa634ed5700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa6300b7df0 0x7fa6300b81f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:57100/0 (socket says 172.21.2.112:57100) 2021-02-10T11:43:25.629 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.625 7fa634ed5700 1 -- 172.21.2.112:0/1124867216 learned_addr learned my addr 172.21.2.112:0/1124867216 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:43:25.629 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.626 7fa634ed5700 1 -- 172.21.2.112:0/1124867216 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa63018aa50 msgr2=0x7fa6300b77d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:43:25.629 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.626 7fa634ed5700 1 --2- 172.21.2.112:0/1124867216 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa63018aa50 0x7fa6300b77d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:25.629 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.626 7fa634ed5700 1 -- 172.21.2.112:0/1124867216 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa63018a540 msgr2=0x7fa630197480 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:43:25.629 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.626 7fa634ed5700 1 --2- 172.21.2.112:0/1124867216 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa63018a540 0x7fa630197480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:25.629 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.626 7fa634ed5700 1 -- 172.21.2.112:0/1124867216 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6301aa9a0 con 0x7fa6300b7df0 2021-02-10T11:43:25.629 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.626 7fa634ed5700 1 --2- 172.21.2.112:0/1124867216 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa6300b7df0 0x7fa6300b81f0 secure :-1 s=READY pgs=2475 cs=0 l=1 rev1=1 rx=0x7fa628009390 tx=0x7fa62800bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:25.629 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.626 7fa62cff9700 1 -- 172.21.2.112:0/1124867216 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa628009450 con 0x7fa6300b7df0 2021-02-10T11:43:25.629 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.626 7fa62cff9700 1 -- 172.21.2.112:0/1124867216 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa628006960 con 0x7fa6300b7df0 2021-02-10T11:43:25.629 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.626 7fa62cff9700 1 -- 172.21.2.112:0/1124867216 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa62800c4e0 con 0x7fa6300b7df0 2021-02-10T11:43:25.630 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.626 7fa6368b8700 1 -- 172.21.2.112:0/1124867216 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6301aa9a0 con 0x7fa6300b7df0 2021-02-10T11:43:25.630 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.626 7fa6368b8700 1 -- 172.21.2.112:0/1124867216 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6301aaf30 con 0x7fa6300b7df0 2021-02-10T11:43:25.630 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.627 7fa62cff9700 1 -- 172.21.2.112:0/1124867216 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa628001270 con 0x7fa6300b7df0 2021-02-10T11:43:25.630 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.627 7fa62cff9700 1 --2- 172.21.2.112:0/1124867216 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa6100186d0 0x7fa61001abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:25.630 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.627 7fa62cff9700 1 -- 172.21.2.112:0/1124867216 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fa628032b40 con 0x7fa6300b7df0 2021-02-10T11:43:25.630 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.627 7fa62ffff700 1 --2- 172.21.2.112:0/1124867216 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa6100186d0 0x7fa61001abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:25.630 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.628 7fa62ffff700 1 --2- 172.21.2.112:0/1124867216 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa6100186d0 0x7fa61001abd0 crc :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:25.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.659 7fa6368b8700 1 -- 172.21.2.112:0/1124867216 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6301aa250 con 0x7fa6300b7df0 2021-02-10T11:43:25.661 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.661 7fa62cff9700 1 -- 172.21.2.112:0/1124867216 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa628002fe0 con 0x7fa6300b7df0 2021-02-10T11:43:25.790 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.790 7fa6368b8700 1 -- 172.21.2.112:0/1124867216 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa6301aa250 con 0x7fa6300b7df0 2021-02-10T11:43:25.790 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.790 7fa62cff9700 1 -- 172.21.2.112:0/1124867216 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 114 v114) v1 ==== 95+0+2108 (secure 0 0 0) 0x7fa628001270 con 0x7fa6300b7df0 2021-02-10T11:43:25.791 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:43:25.791 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":114,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11349,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/2093257968","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":2093257968},{"type":"v1","addr":"172.21.2.112:6833","nonce":2093257968}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":110}],"filesystems":[{"mdsmap":{"epoch":114,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:43:08.182625","modified":"2021-02-10 11:43:10.189979","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7025},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7025":{"gid":7025,"name":"b","rank":0,"incarnation":111,"state":"up:active","state_seq":3,"addr":"172.21.2.120:6835/2552826693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":2552826693},{"type":"v1","addr":"172.21.2.120:6835","nonce":2552826693}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":14}]} 2021-02-10T11:43:25.791 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 114 2021-02-10T11:43:25.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.792 7fa6368b8700 1 -- 172.21.2.112:0/1124867216 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa6100186d0 msgr2=0x7fa61001abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:25.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.792 7fa6368b8700 1 --2- 172.21.2.112:0/1124867216 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa6100186d0 0x7fa61001abd0 crc :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:25.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.792 7fa6368b8700 1 -- 172.21.2.112:0/1124867216 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa6300b7df0 msgr2=0x7fa6300b81f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:25.793 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.792 7fa6368b8700 1 --2- 172.21.2.112:0/1124867216 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa6300b7df0 0x7fa6300b81f0 secure :-1 s=READY pgs=2475 cs=0 l=1 rev1=1 rx=0x7fa628009390 tx=0x7fa62800bd40).stop 2021-02-10T11:43:25.793 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.792 7fa6368b8700 1 -- 172.21.2.112:0/1124867216 shutdown_connections 2021-02-10T11:43:25.793 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.792 7fa6368b8700 1 --2- 172.21.2.112:0/1124867216 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa6100186d0 0x7fa61001abd0 unknown :-1 s=CLOSED pgs=1128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:25.793 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.792 7fa6368b8700 1 --2- 172.21.2.112:0/1124867216 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa6300b7df0 0x7fa6300b81f0 unknown :-1 s=CLOSED pgs=2475 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:25.793 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.792 7fa6368b8700 1 --2- 172.21.2.112:0/1124867216 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa63018aa50 0x7fa6300b77d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:25.793 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.792 7fa6368b8700 1 --2- 172.21.2.112:0/1124867216 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa63018a540 0x7fa630197480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:25.793 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.792 7fa6368b8700 1 -- 172.21.2.112:0/1124867216 >> 172.21.2.112:0/1124867216 conn(0x7fa6300efc70 msgr2=0x7fa6300ef410 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:25.793 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.793 7fa6368b8700 1 -- 172.21.2.112:0/1124867216 shutdown_connections 2021-02-10T11:43:25.793 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:25.793 7fa6368b8700 1 -- 172.21.2.112:0/1124867216 wait complete. 2021-02-10T11:43:26.003 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 114, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:43:08.182625', 'modified': '2021-02-10 11:43:10.189979', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7025}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7025': {'gid': 7025, 'name': 'b', 'rank': 0, 'incarnation': 111, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.120:6835/2552826693', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.120:6834', 'nonce': 2552826693}, {'type': 'v1', 'addr': '172.21.2.120:6835', 'nonce': 2552826693}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [29], 'metadata_pool': 28, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:43:26.003 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:43:26.003 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:43:26.401 INFO:teuthology.orchestra.run.gibba020.stdout:{ 2021-02-10T11:43:26.401 INFO:teuthology.orchestra.run.gibba020.stdout: "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", 2021-02-10T11:43:26.401 INFO:teuthology.orchestra.run.gibba020.stdout: "whoami": 0, 2021-02-10T11:43:26.401 INFO:teuthology.orchestra.run.gibba020.stdout: "id": 7025, 2021-02-10T11:43:26.401 INFO:teuthology.orchestra.run.gibba020.stdout: "want_state": "up:active", 2021-02-10T11:43:26.401 INFO:teuthology.orchestra.run.gibba020.stdout: "state": "up:active", 2021-02-10T11:43:26.401 INFO:teuthology.orchestra.run.gibba020.stdout: "rank_uptime": 18.215361499, 2021-02-10T11:43:26.401 INFO:teuthology.orchestra.run.gibba020.stdout: "mdsmap_epoch": 114, 2021-02-10T11:43:26.402 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch": 134, 2021-02-10T11:43:26.402 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch_barrier": 134, 2021-02-10T11:43:26.402 INFO:teuthology.orchestra.run.gibba020.stdout: "uptime": 23.858020290999999 2021-02-10T11:43:26.402 INFO:teuthology.orchestra.run.gibba020.stdout:} 2021-02-10T11:43:26.405 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", "whoami": 0, "id": 7025, "want_state": "up:active", "state": "up:active", "rank_uptime": 18.215361499, "mdsmap_epoch": 114, "osdmap_epoch": 134, "osdmap_epoch_barrier": 134, "uptime": 23.858020290999999 } 2021-02-10T11:43:26.405 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:43:26.405 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2021-02-10T11:43:26.405 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.1 at ubuntu@gibba012.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2021-02-10T11:43:26.405 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:43:26.405 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:43:26.741 DEBUG:teuthology.orchestra.run.gibba012:> sudo modprobe fuse 2021-02-10T11:43:27.086 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:43:27.087 DEBUG:teuthology.orchestra.run.gibba012:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:43:27.430 INFO:teuthology.orchestra.run.gibba012.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:43:27.430 INFO:teuthology.orchestra.run.gibba012.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:43:27.431 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:43:27.431 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:43:27.432 DEBUG:teuthology.orchestra.run.gibba012:> ls /sys/fs/fuse/connections 2021-02-10T11:43:27.766 INFO:teuthology.orchestra.run.gibba012.stdout:42 2021-02-10T11:43:27.766 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [42] 2021-02-10T11:43:27.767 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.1 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:43:27.769 DEBUG:teuthology.orchestra.run.gibba012:> sudo modprobe fuse 2021-02-10T11:43:28.114 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:43:28.115 DEBUG:teuthology.orchestra.run.gibba012:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:43:28.180 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba012.stderr:ceph-fuse[170005]: starting ceph client 2021-02-10T11:43:28.181 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba012.stderr:2021-02-10 11:43:28.180 7fb9b8a38f80 -1 init, newargv = 0x5638af1a7d80 newargc=9 2021-02-10T11:43:28.186 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba012.stderr:ceph-fuse[170005]: starting fuse 2021-02-10T11:43:28.457 INFO:teuthology.orchestra.run.gibba012.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:43:28.457 INFO:teuthology.orchestra.run.gibba012.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:43:28.458 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:43:28.458 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:43:28.458 DEBUG:teuthology.orchestra.run.gibba012:> ls /sys/fs/fuse/connections 2021-02-10T11:43:28.792 INFO:teuthology.orchestra.run.gibba012.stdout:42 2021-02-10T11:43:28.792 INFO:teuthology.orchestra.run.gibba012.stdout:43 2021-02-10T11:43:28.792 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42, 43] 2021-02-10T11:43:28.792 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:43:28.792 DEBUG:teuthology.orchestra.run.gibba012:> sudo python3 -c ' 2021-02-10T11:43:28.793 DEBUG:teuthology.orchestra.run.gibba012:> import glob 2021-02-10T11:43:28.793 DEBUG:teuthology.orchestra.run.gibba012:> import re 2021-02-10T11:43:28.793 DEBUG:teuthology.orchestra.run.gibba012:> import os 2021-02-10T11:43:28.793 DEBUG:teuthology.orchestra.run.gibba012:> import subprocess 2021-02-10T11:43:28.793 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:43:28.793 DEBUG:teuthology.orchestra.run.gibba012:> def find_socket(client_name): 2021-02-10T11:43:28.793 DEBUG:teuthology.orchestra.run.gibba012:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2021-02-10T11:43:28.793 DEBUG:teuthology.orchestra.run.gibba012:> files = glob.glob(asok_path) 2021-02-10T11:43:28.793 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:43:28.793 DEBUG:teuthology.orchestra.run.gibba012:> # Given a non-glob path, it better be there 2021-02-10T11:43:28.793 DEBUG:teuthology.orchestra.run.gibba012:> if "*" not in asok_path: 2021-02-10T11:43:28.793 DEBUG:teuthology.orchestra.run.gibba012:> assert(len(files) == 1) 2021-02-10T11:43:28.793 DEBUG:teuthology.orchestra.run.gibba012:> return files[0] 2021-02-10T11:43:28.793 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:43:28.794 DEBUG:teuthology.orchestra.run.gibba012:> for f in files: 2021-02-10T11:43:28.794 DEBUG:teuthology.orchestra.run.gibba012:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:43:28.794 DEBUG:teuthology.orchestra.run.gibba012:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:43:28.794 DEBUG:teuthology.orchestra.run.gibba012:> return f 2021-02-10T11:43:28.794 DEBUG:teuthology.orchestra.run.gibba012:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:43:28.794 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:43:28.794 DEBUG:teuthology.orchestra.run.gibba012:> print(find_socket("client.1")) 2021-02-10T11:43:28.794 DEBUG:teuthology.orchestra.run.gibba012:> ' 2021-02-10T11:43:29.157 INFO:teuthology.orchestra.run.gibba012.stdout:/var/run/ceph/ceph-client.1.170005.asok 2021-02-10T11:43:29.159 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.170005.asok 2021-02-10T11:43:29.159 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:43:29.160 DEBUG:teuthology.orchestra.run.gibba012:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.170005.asok status 2021-02-10T11:43:29.550 INFO:teuthology.orchestra.run.gibba012.stdout:{ 2021-02-10T11:43:29.550 INFO:teuthology.orchestra.run.gibba012.stdout: "metadata": { 2021-02-10T11:43:29.550 INFO:teuthology.orchestra.run.gibba012.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:43:29.550 INFO:teuthology.orchestra.run.gibba012.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:43:29.551 INFO:teuthology.orchestra.run.gibba012.stdout: "entity_id": "1", 2021-02-10T11:43:29.551 INFO:teuthology.orchestra.run.gibba012.stdout: "hostname": "gibba012", 2021-02-10T11:43:29.551 INFO:teuthology.orchestra.run.gibba012.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2021-02-10T11:43:29.551 INFO:teuthology.orchestra.run.gibba012.stdout: "pid": "170005", 2021-02-10T11:43:29.551 INFO:teuthology.orchestra.run.gibba012.stdout: "root": "/" 2021-02-10T11:43:29.551 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:29.551 INFO:teuthology.orchestra.run.gibba012.stdout: "dentry_count": 0, 2021-02-10T11:43:29.551 INFO:teuthology.orchestra.run.gibba012.stdout: "dentry_pinned_count": 0, 2021-02-10T11:43:29.551 INFO:teuthology.orchestra.run.gibba012.stdout: "id": 11547, 2021-02-10T11:43:29.551 INFO:teuthology.orchestra.run.gibba012.stdout: "inst": { 2021-02-10T11:43:29.551 INFO:teuthology.orchestra.run.gibba012.stdout: "name": { 2021-02-10T11:43:29.551 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "client", 2021-02-10T11:43:29.552 INFO:teuthology.orchestra.run.gibba012.stdout: "num": 11547 2021-02-10T11:43:29.552 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:29.552 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": { 2021-02-10T11:43:29.552 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:43:29.552 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:0", 2021-02-10T11:43:29.552 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 3439978697 2021-02-10T11:43:29.552 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:43:29.552 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:29.552 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": { 2021-02-10T11:43:29.552 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:43:29.552 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:0", 2021-02-10T11:43:29.552 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 3439978697 2021-02-10T11:43:29.552 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:29.552 INFO:teuthology.orchestra.run.gibba012.stdout: "inst_str": "client.11547 172.21.2.112:0/3439978697", 2021-02-10T11:43:29.553 INFO:teuthology.orchestra.run.gibba012.stdout: "addr_str": "172.21.2.112:0/3439978697", 2021-02-10T11:43:29.553 INFO:teuthology.orchestra.run.gibba012.stdout: "inode_count": 1, 2021-02-10T11:43:29.553 INFO:teuthology.orchestra.run.gibba012.stdout: "mds_epoch": 114, 2021-02-10T11:43:29.553 INFO:teuthology.orchestra.run.gibba012.stdout: "osd_epoch": 134, 2021-02-10T11:43:29.553 INFO:teuthology.orchestra.run.gibba012.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:43:29.553 INFO:teuthology.orchestra.run.gibba012.stdout: "blacklisted": false 2021-02-10T11:43:29.553 INFO:teuthology.orchestra.run.gibba012.stdout:} 2021-02-10T11:43:29.554 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:43:29.554 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:43:29.555 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:43:29.890 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:43:29.891 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:43:29.891 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:43:29.891 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:43:30.237 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2021-02-10T11:43:30.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:30.652 7f0e00501700 1 Processor -- start 2021-02-10T11:43:30.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:30.652 7f0e00501700 1 -- start start 2021-02-10T11:43:30.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:30.653 7f0e00501700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f0df80d1c80 0x7f0df80cef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:30.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:30.653 7f0e00501700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0df80cf590 0x7f0df80cf990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:30.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:30.653 7f0e00501700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0df80fe1d0 con 0x7f0df80d1880 2021-02-10T11:43:30.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:30.653 7f0e00501700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0df81b0cc0 con 0x7f0df80d1c80 2021-02-10T11:43:30.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:30.653 7f0e00501700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0df8109a70 con 0x7f0df80cf590 2021-02-10T11:43:30.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:30.653 7f0dfdb1c700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f0df80d1c80 0x7f0df80cef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:30.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:30.653 7f0dfe31d700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7f0df80d1880 0x7f0df81b04d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:50146/0 (socket says 172.21.2.112:50146) 2021-02-10T11:43:30.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:30.653 7f0dfdb1c700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f0df80d1c80 0x7f0df80cef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:57128/0 (socket says 172.21.2.112:57128) 2021-02-10T11:43:30.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:30.653 7f0dfe31d700 1 -- 172.21.2.112:0/462274151 learned_addr learned my addr 172.21.2.112:0/462274151 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:43:30.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:30.654 7f0dfeb1e700 1 --2- 172.21.2.112:0/462274151 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0df80cf590 0x7f0df80cf990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:30.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:30.654 7f0dfdb1c700 1 -- 172.21.2.112:0/462274151 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0df80cf590 msgr2=0x7f0df80cf990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:30.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:30.654 7f0dfdb1c700 1 --2- 172.21.2.112:0/462274151 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0df80cf590 0x7f0df80cf990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:30.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:30.654 7f0dfdb1c700 1 -- 172.21.2.112:0/462274151 >> v1:172.21.2.120:6789/0 conn(0x7f0df80d1880 legacy=0x7f0df81b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:30.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:30.654 7f0dfdb1c700 1 -- 172.21.2.112:0/462274151 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0df8088070 con 0x7f0df80d1c80 2021-02-10T11:43:30.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:30.654 7f0dfdb1c700 1 --2- 172.21.2.112:0/462274151 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0df80d1c80 0x7f0df80cef70 secure :-1 s=READY pgs=2478 cs=0 l=1 rev1=1 rx=0x7f0de8006650 tx=0x7f0de8009a80).ready entity=mon.0 client_cookie=ba1819cca02e2856 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:30.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:30.655 7f0dfd31b700 1 -- 172.21.2.112:0/462274151 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0de80009b0 con 0x7f0df80d1c80 2021-02-10T11:43:30.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:30.655 7f0dfd31b700 1 -- 172.21.2.112:0/462274151 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0de800c0c0 con 0x7f0df80d1c80 2021-02-10T11:43:30.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:30.655 7f0dfd31b700 1 -- 172.21.2.112:0/462274151 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0de800c5e0 con 0x7f0df80d1c80 2021-02-10T11:43:30.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:30.655 7f0e00501700 1 -- 172.21.2.112:0/462274151 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0df80d1c80 msgr2=0x7f0df80cef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:30.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:30.655 7f0e00501700 1 --2- 172.21.2.112:0/462274151 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0df80d1c80 0x7f0df80cef70 secure :-1 s=READY pgs=2478 cs=0 l=1 rev1=1 rx=0x7f0de8006650 tx=0x7f0de8009a80).stop 2021-02-10T11:43:30.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:30.656 7f0e00501700 1 -- 172.21.2.112:0/462274151 shutdown_connections 2021-02-10T11:43:30.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:30.656 7f0e00501700 1 --2- 172.21.2.112:0/462274151 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0df80d1c80 0x7f0df80cef70 unknown :-1 s=CLOSED pgs=2478 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:30.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:30.656 7f0e00501700 1 --2- 172.21.2.112:0/462274151 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0df80cf590 0x7f0df80cf990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:30.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:30.656 7f0e00501700 1 -- 172.21.2.112:0/462274151 >> 172.21.2.112:0/462274151 conn(0x7f0df80ce430 msgr2=0x7f0df80d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:30.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:30.656 7f0e00501700 1 -- 172.21.2.112:0/462274151 shutdown_connections 2021-02-10T11:43:30.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:30.656 7f0e00501700 1 -- 172.21.2.112:0/462274151 wait complete. 2021-02-10T11:43:30.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:30.671 7f0e00501700 1 Processor -- start 2021-02-10T11:43:30.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:30.671 7f0e00501700 1 -- start start 2021-02-10T11:43:30.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:30.671 7f0e00501700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0df81a7a70 0x7f0df81a7e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:30.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:30.671 7f0e00501700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0df80bf730 0x7f0df80bfb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:30.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:30.671 7f0e00501700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0df80c0150 0x7f0df80909c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:30.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:30.671 7f0e00501700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0df8090fe0 con 0x7f0df80bf730 2021-02-10T11:43:30.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:30.671 7f0e00501700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0df8091200 con 0x7f0df80c0150 2021-02-10T11:43:30.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:30.671 7f0e00501700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0df8091420 con 0x7f0df81a7a70 2021-02-10T11:43:30.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:30.671 7f0dfdb1c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0df80bf730 0x7f0df80bfb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:30.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:30.671 7f0dfdb1c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0df80bf730 0x7f0df80bfb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:57136/0 (socket says 172.21.2.112:57136) 2021-02-10T11:43:30.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:30.671 7f0dfdb1c700 1 -- 172.21.2.112:0/494599697 learned_addr learned my addr 172.21.2.112:0/494599697 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:43:30.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:30.672 7f0dfe31d700 1 --2- 172.21.2.112:0/494599697 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0df81a7a70 0x7f0df81a7e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:30.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:30.672 7f0dfeb1e700 1 --2- 172.21.2.112:0/494599697 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0df80c0150 0x7f0df80909c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:30.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:30.672 7f0dfdb1c700 1 -- 172.21.2.112:0/494599697 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0df81a7a70 msgr2=0x7f0df81a7e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:30.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:30.672 7f0dfdb1c700 1 --2- 172.21.2.112:0/494599697 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0df81a7a70 0x7f0df81a7e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:30.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:30.672 7f0dfdb1c700 1 -- 172.21.2.112:0/494599697 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0df80c0150 msgr2=0x7f0df80909c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:30.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:30.672 7f0dfdb1c700 1 --2- 172.21.2.112:0/494599697 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0df80c0150 0x7f0df80909c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:30.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:30.672 7f0dfdb1c700 1 -- 172.21.2.112:0/494599697 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0df8091640 con 0x7f0df80bf730 2021-02-10T11:43:30.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:30.672 7f0dfdb1c700 1 --2- 172.21.2.112:0/494599697 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0df80bf730 0x7f0df80bfb30 secure :-1 s=READY pgs=2479 cs=0 l=1 rev1=1 rx=0x7f0de8009630 tx=0x7f0de800b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:30.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:30.672 7f0deeffd700 1 -- 172.21.2.112:0/494599697 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0de80093a0 con 0x7f0df80bf730 2021-02-10T11:43:30.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:30.672 7f0deeffd700 1 -- 172.21.2.112:0/494599697 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0de800bd70 con 0x7f0df80bf730 2021-02-10T11:43:30.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:30.672 7f0deeffd700 1 -- 172.21.2.112:0/494599697 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0de80054d0 con 0x7f0df80bf730 2021-02-10T11:43:30.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:30.672 7f0e00501700 1 -- 172.21.2.112:0/494599697 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0df8091640 con 0x7f0df80bf730 2021-02-10T11:43:30.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:30.672 7f0e00501700 1 -- 172.21.2.112:0/494599697 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0df80c0e70 con 0x7f0df80bf730 2021-02-10T11:43:30.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:30.673 7f0deeffd700 1 -- 172.21.2.112:0/494599697 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0de800a000 con 0x7f0df80bf730 2021-02-10T11:43:30.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:30.673 7f0deeffd700 1 --2- 172.21.2.112:0/494599697 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0de00186c0 0x7f0de001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:30.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:30.673 7f0deeffd700 1 -- 172.21.2.112:0/494599697 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f0de8032580 con 0x7f0df80bf730 2021-02-10T11:43:30.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:30.674 7f0dfe31d700 1 --2- 172.21.2.112:0/494599697 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0de00186c0 0x7f0de001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:30.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:30.674 7f0dfe31d700 1 --2- 172.21.2.112:0/494599697 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0de00186c0 0x7f0de001abc0 crc :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:30.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:30.703 7f0e00501700 1 -- 172.21.2.112:0/494599697 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0df80fe1d0 con 0x7f0df80bf730 2021-02-10T11:43:30.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:30.705 7f0deeffd700 1 -- 172.21.2.112:0/494599697 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0de8009340 con 0x7f0df80bf730 2021-02-10T11:43:30.841 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:30.841 7f0e00501700 1 -- 172.21.2.112:0/494599697 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f0df80fe1d0 con 0x7f0df80bf730 2021-02-10T11:43:31.663 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:31.662 7f0deeffd700 1 -- 172.21.2.112:0/494599697 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v115) v1 ==== 136+0+0 (secure 0 0 0) 0x7f0de800a000 con 0x7f0df80bf730 2021-02-10T11:43:31.714 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:31.713 7f0e00501700 1 -- 172.21.2.112:0/494599697 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0de00186c0 msgr2=0x7f0de001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:31.714 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:31.713 7f0e00501700 1 --2- 172.21.2.112:0/494599697 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0de00186c0 0x7f0de001abc0 crc :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:31.714 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:31.713 7f0e00501700 1 -- 172.21.2.112:0/494599697 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0df80bf730 msgr2=0x7f0df80bfb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:31.714 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:31.713 7f0e00501700 1 --2- 172.21.2.112:0/494599697 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0df80bf730 0x7f0df80bfb30 secure :-1 s=READY pgs=2479 cs=0 l=1 rev1=1 rx=0x7f0de8009630 tx=0x7f0de800b980).stop 2021-02-10T11:43:31.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:31.714 7f0e00501700 1 -- 172.21.2.112:0/494599697 shutdown_connections 2021-02-10T11:43:31.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:31.714 7f0e00501700 1 --2- 172.21.2.112:0/494599697 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0de00186c0 0x7f0de001abc0 unknown :-1 s=CLOSED pgs=1135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:31.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:31.714 7f0e00501700 1 --2- 172.21.2.112:0/494599697 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0df80c0150 0x7f0df80909c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:31.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:31.714 7f0e00501700 1 --2- 172.21.2.112:0/494599697 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0df80bf730 0x7f0df80bfb30 unknown :-1 s=CLOSED pgs=2479 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:31.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:31.714 7f0e00501700 1 --2- 172.21.2.112:0/494599697 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0df81a7a70 0x7f0df81a7e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:31.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:31.714 7f0e00501700 1 -- 172.21.2.112:0/494599697 >> 172.21.2.112:0/494599697 conn(0x7f0df80a69d0 msgr2=0x7f0df80b5950 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:31.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:31.714 7f0e00501700 1 -- 172.21.2.112:0/494599697 shutdown_connections 2021-02-10T11:43:31.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:31.714 7f0e00501700 1 -- 172.21.2.112:0/494599697 wait complete. 2021-02-10T11:43:31.722 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs volume ls 2021-02-10T11:43:32.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.133 7f9817c40700 1 Processor -- start 2021-02-10T11:43:32.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.133 7f9817c40700 1 -- start start 2021-02-10T11:43:32.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.134 7f9817c40700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f98100d0330 0x7f98100ef630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:32.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.134 7f9817c40700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f98100d1c60 0x7f98100efc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:32.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.134 7f9817c40700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98100f0b00 con 0x7f98100d1860 2021-02-10T11:43:32.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.134 7f9817c40700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98100e5ec0 con 0x7f98100d1c60 2021-02-10T11:43:32.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.134 7f9817c40700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98101b0bd0 con 0x7f98100d0330 2021-02-10T11:43:32.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.134 7f981625d700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f98100d1c60 0x7f98100efc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:32.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.134 7f981525b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f98100d0330 0x7f98100ef630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:32.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.134 7f9815a5c700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7f98100d1860 0x7f98101b03e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:50160/0 (socket says 172.21.2.112:50160) 2021-02-10T11:43:32.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.135 7f9815a5c700 1 -- 172.21.2.112:0/665634146 learned_addr learned my addr 172.21.2.112:0/665634146 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:43:32.152 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.135 7f981625d700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f98100d1c60 0x7f98100efc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3300/0 says I am v2:172.21.2.112:59420/0 (socket says 172.21.2.112:59420) 2021-02-10T11:43:32.152 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.135 7f981625d700 1 -- 172.21.2.112:0/665634146 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f98100d0330 msgr2=0x7f98100ef630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:32.152 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.135 7f981625d700 1 --2- 172.21.2.112:0/665634146 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f98100d0330 0x7f98100ef630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:32.152 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.135 7f981625d700 1 -- 172.21.2.112:0/665634146 >> v1:172.21.2.120:6789/0 conn(0x7f98100d1860 legacy=0x7f98101b03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:32.152 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.135 7f981625d700 1 -- 172.21.2.112:0/665634146 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98100f2090 con 0x7f98100d1c60 2021-02-10T11:43:32.152 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.135 7f981525b700 1 --2- 172.21.2.112:0/665634146 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f98100d0330 0x7f98100ef630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:43:32.152 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.135 7f981625d700 1 --2- 172.21.2.112:0/665634146 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f98100d1c60 0x7f98100efc50 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 rx=0x7f9808006e50 tx=0x7f9808009e70).ready entity=mon.1 client_cookie=19b1e5e9f6138fa6 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:32.152 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.135 7f9814a5a700 1 -- 172.21.2.112:0/665634146 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9808000f90 con 0x7f98100d1c60 2021-02-10T11:43:32.152 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.135 7f9814a5a700 1 -- 172.21.2.112:0/665634146 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98080066f0 con 0x7f98100d1c60 2021-02-10T11:43:32.152 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.135 7f9814a5a700 1 -- 172.21.2.112:0/665634146 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f980800b700 con 0x7f98100d1c60 2021-02-10T11:43:32.152 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.136 7f9817c40700 1 -- 172.21.2.112:0/665634146 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f98100d1c60 msgr2=0x7f98100efc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:32.152 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.136 7f9817c40700 1 --2- 172.21.2.112:0/665634146 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f98100d1c60 0x7f98100efc50 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 rx=0x7f9808006e50 tx=0x7f9808009e70).stop 2021-02-10T11:43:32.152 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.136 7f9817c40700 1 -- 172.21.2.112:0/665634146 shutdown_connections 2021-02-10T11:43:32.153 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.136 7f9817c40700 1 --2- 172.21.2.112:0/665634146 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f98100d1c60 0x7f98100efc50 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:32.153 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.136 7f9817c40700 1 --2- 172.21.2.112:0/665634146 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f98100d0330 0x7f98100ef630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:32.153 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.136 7f9817c40700 1 -- 172.21.2.112:0/665634146 >> 172.21.2.112:0/665634146 conn(0x7f98100ce440 msgr2=0x7f98100d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:32.153 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.136 7f9817c40700 1 -- 172.21.2.112:0/665634146 shutdown_connections 2021-02-10T11:43:32.153 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.136 7f9817c40700 1 -- 172.21.2.112:0/665634146 wait complete. 2021-02-10T11:43:32.154 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.151 7f9817c40700 1 Processor -- start 2021-02-10T11:43:32.154 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.151 7f9817c40700 1 -- start start 2021-02-10T11:43:32.154 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.151 7f9817c40700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f98101a35c0 0x7f98101a39c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:32.155 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.151 7f9817c40700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f981009d0e0 0x7f981009d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:32.155 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.151 7f9817c40700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f981009db00 0x7f981009df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:32.155 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.151 7f9817c40700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f981009cb60 con 0x7f98101a35c0 2021-02-10T11:43:32.155 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.151 7f9817c40700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98100c3730 con 0x7f981009d0e0 2021-02-10T11:43:32.155 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.151 7f9817c40700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98100c3910 con 0x7f981009db00 2021-02-10T11:43:32.155 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.152 7f9815a5c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f98101a35c0 0x7f98101a39c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:32.155 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.152 7f9815a5c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f98101a35c0 0x7f98101a39c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:57148/0 (socket says 172.21.2.112:57148) 2021-02-10T11:43:32.155 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.152 7f9815a5c700 1 -- 172.21.2.112:0/1359114883 learned_addr learned my addr 172.21.2.112:0/1359114883 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:43:32.155 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.152 7f981625d700 1 --2- 172.21.2.112:0/1359114883 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f981009db00 0x7f981009df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:32.155 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.152 7f981525b700 1 --2- 172.21.2.112:0/1359114883 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f981009d0e0 0x7f981009d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:32.155 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.152 7f9815a5c700 1 -- 172.21.2.112:0/1359114883 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f981009db00 msgr2=0x7f981009df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:32.155 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.152 7f9815a5c700 1 --2- 172.21.2.112:0/1359114883 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f981009db00 0x7f981009df00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:32.155 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.152 7f9815a5c700 1 -- 172.21.2.112:0/1359114883 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f981009d0e0 msgr2=0x7f981009d4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:32.156 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.152 7f9815a5c700 1 --2- 172.21.2.112:0/1359114883 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f981009d0e0 0x7f981009d4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:32.156 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.152 7f9815a5c700 1 -- 172.21.2.112:0/1359114883 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9810093e60 con 0x7f98101a35c0 2021-02-10T11:43:32.156 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.152 7f981625d700 1 --2- 172.21.2.112:0/1359114883 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f981009db00 0x7f981009df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:43:32.156 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.152 7f981525b700 1 --2- 172.21.2.112:0/1359114883 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f981009d0e0 0x7f981009d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:43:32.156 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.152 7f9815a5c700 1 --2- 172.21.2.112:0/1359114883 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f98101a35c0 0x7f98101a39c0 secure :-1 s=READY pgs=2480 cs=0 l=1 rev1=1 rx=0x7f980c001c60 tx=0x7f980c00a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:32.156 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.153 7f98067fc700 1 -- 172.21.2.112:0/1359114883 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f980c00acc0 con 0x7f98101a35c0 2021-02-10T11:43:32.156 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.153 7f98067fc700 1 -- 172.21.2.112:0/1359114883 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f980c004140 con 0x7f98101a35c0 2021-02-10T11:43:32.156 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.153 7f98067fc700 1 -- 172.21.2.112:0/1359114883 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f980c004620 con 0x7f98101a35c0 2021-02-10T11:43:32.156 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.153 7f9817c40700 1 -- 172.21.2.112:0/1359114883 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9810093e60 con 0x7f98101a35c0 2021-02-10T11:43:32.156 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.153 7f9817c40700 1 -- 172.21.2.112:0/1359114883 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9810093e60 con 0x7f98101a35c0 2021-02-10T11:43:32.156 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.153 7f98067fc700 1 -- 172.21.2.112:0/1359114883 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f980c000d90 con 0x7f98101a35c0 2021-02-10T11:43:32.156 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.153 7f98067fc700 1 --2- 172.21.2.112:0/1359114883 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f97f80186c0 0x7f97f801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:32.156 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.154 7f98067fc700 1 -- 172.21.2.112:0/1359114883 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f980c032560 con 0x7f98101a35c0 2021-02-10T11:43:32.156 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.154 7f981525b700 1 --2- 172.21.2.112:0/1359114883 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f97f80186c0 0x7f97f801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:32.157 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.154 7f981525b700 1 --2- 172.21.2.112:0/1359114883 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f97f80186c0 0x7f97f801abc0 crc :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:32.184 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.183 7f9817c40700 1 -- 172.21.2.112:0/1359114883 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98100ebbe0 con 0x7f98101a35c0 2021-02-10T11:43:32.186 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.186 7f98067fc700 1 -- 172.21.2.112:0/1359114883 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f980c0009b0 con 0x7f98101a35c0 2021-02-10T11:43:32.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.318 7f9817c40700 1 -- 172.21.2.112:0/1359114883 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs volume ls", "target": ["mgr", ""]}) v1 -- 0x7f98100ebbe0 con 0x7f97f80186c0 2021-02-10T11:43:32.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.320 7f98067fc700 1 -- 172.21.2.112:0/1359114883 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+41 (crc 0 0 0) 0x7f9800006de0 con 0x7f97f80186c0 2021-02-10T11:43:32.321 INFO:teuthology.orchestra.run.gibba012.stdout:[ 2021-02-10T11:43:32.322 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:32.322 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "cephfs" 2021-02-10T11:43:32.322 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:43:32.322 INFO:teuthology.orchestra.run.gibba012.stdout:] 2021-02-10T11:43:32.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.323 7f9817c40700 1 -- 172.21.2.112:0/1359114883 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f97f80186c0 msgr2=0x7f97f801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:32.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.323 7f9817c40700 1 --2- 172.21.2.112:0/1359114883 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f97f80186c0 0x7f97f801abc0 crc :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:32.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.323 7f9817c40700 1 -- 172.21.2.112:0/1359114883 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f98101a35c0 msgr2=0x7f98101a39c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:32.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.323 7f9817c40700 1 --2- 172.21.2.112:0/1359114883 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f98101a35c0 0x7f98101a39c0 secure :-1 s=READY pgs=2480 cs=0 l=1 rev1=1 rx=0x7f980c001c60 tx=0x7f980c00a8d0).stop 2021-02-10T11:43:32.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.323 7f9817c40700 1 -- 172.21.2.112:0/1359114883 shutdown_connections 2021-02-10T11:43:32.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.323 7f9817c40700 1 --2- 172.21.2.112:0/1359114883 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f97f80186c0 0x7f97f801abc0 unknown :-1 s=CLOSED pgs=1136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:32.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.323 7f9817c40700 1 --2- 172.21.2.112:0/1359114883 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f981009d0e0 0x7f981009d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:32.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.323 7f9817c40700 1 --2- 172.21.2.112:0/1359114883 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f981009db00 0x7f981009df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:32.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.323 7f9817c40700 1 --2- 172.21.2.112:0/1359114883 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f98101a35c0 0x7f98101a39c0 unknown :-1 s=CLOSED pgs=2480 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:32.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.323 7f9817c40700 1 -- 172.21.2.112:0/1359114883 >> 172.21.2.112:0/1359114883 conn(0x7f98100a6b00 msgr2=0x7f9810095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:32.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.323 7f9817c40700 1 -- 172.21.2.112:0/1359114883 shutdown_connections 2021-02-10T11:43:32.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.323 7f9817c40700 1 -- 172.21.2.112:0/1359114883 wait complete. 2021-02-10T11:43:32.332 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:43:32.765 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.748 7f2856907700 1 Processor -- start 2021-02-10T11:43:32.765 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.748 7f2856907700 1 -- start start 2021-02-10T11:43:32.765 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.748 7f2856907700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f28500d1c70 0x7f28500cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:32.765 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.748 7f2856907700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f28500cf580 0x7f28500cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:32.765 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.748 7f2856907700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28500fe1c0 con 0x7f28500d1870 2021-02-10T11:43:32.765 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.748 7f2856907700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28501b0cb0 con 0x7f28500d1c70 2021-02-10T11:43:32.765 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.748 7f2856907700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2850109a60 con 0x7f28500cf580 2021-02-10T11:43:32.765 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.748 7f284ffff700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f28500d1870 0x7f28501b04c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:56616/0 (socket says 172.21.2.112:56616) 2021-02-10T11:43:32.765 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.748 7f284ffff700 1 -- 172.21.2.112:0/3947779915 learned_addr learned my addr 172.21.2.112:0/3947779915 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:43:32.765 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.748 7f284f7fe700 1 --2- 172.21.2.112:0/3947779915 >> v2:172.21.2.112:3300/0 conn(0x7f28500d1c70 0x7f28500cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:32.766 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.749 7f2854f24700 1 --2- 172.21.2.112:0/3947779915 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f28500cf580 0x7f28500cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:32.766 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.749 7f284f7fe700 1 -- 172.21.2.112:0/3947779915 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f28500cf580 msgr2=0x7f28500cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:32.766 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.749 7f284f7fe700 1 --2- 172.21.2.112:0/3947779915 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f28500cf580 0x7f28500cf980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:32.766 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.749 7f284f7fe700 1 -- 172.21.2.112:0/3947779915 >> v1:172.21.2.112:6789/0 conn(0x7f28500d1870 legacy=0x7f28501b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:32.766 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.749 7f284f7fe700 1 -- 172.21.2.112:0/3947779915 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2850088070 con 0x7f28500d1c70 2021-02-10T11:43:32.766 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.749 7f284f7fe700 1 --2- 172.21.2.112:0/3947779915 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f28500d1c70 0x7f28500cef60 secure :-1 s=READY pgs=2482 cs=0 l=1 rev1=1 rx=0x7f2844006650 tx=0x7f2844009a00).ready entity=mon.0 client_cookie=b15a59b00005bffc server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:32.766 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.750 7f284effd700 1 -- 172.21.2.112:0/3947779915 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f28440009b0 con 0x7f28500d1c70 2021-02-10T11:43:32.766 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.750 7f284effd700 1 -- 172.21.2.112:0/3947779915 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f284400c0c0 con 0x7f28500d1c70 2021-02-10T11:43:32.766 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.750 7f284effd700 1 -- 172.21.2.112:0/3947779915 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f284400c5a0 con 0x7f28500d1c70 2021-02-10T11:43:32.766 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.750 7f2856907700 1 -- 172.21.2.112:0/3947779915 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f28500d1c70 msgr2=0x7f28500cef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:32.766 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.750 7f2856907700 1 --2- 172.21.2.112:0/3947779915 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f28500d1c70 0x7f28500cef60 secure :-1 s=READY pgs=2482 cs=0 l=1 rev1=1 rx=0x7f2844006650 tx=0x7f2844009a00).stop 2021-02-10T11:43:32.766 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.750 7f2856907700 1 -- 172.21.2.112:0/3947779915 shutdown_connections 2021-02-10T11:43:32.766 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.750 7f2856907700 1 --2- 172.21.2.112:0/3947779915 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f28500cf580 0x7f28500cf980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:32.767 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.750 7f2856907700 1 --2- 172.21.2.112:0/3947779915 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f28500d1c70 0x7f28500cef60 unknown :-1 s=CLOSED pgs=2482 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:32.767 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.750 7f2856907700 1 -- 172.21.2.112:0/3947779915 >> 172.21.2.112:0/3947779915 conn(0x7f28500ce420 msgr2=0x7f28500d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:32.767 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.750 7f2856907700 1 -- 172.21.2.112:0/3947779915 shutdown_connections 2021-02-10T11:43:32.767 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.750 7f2856907700 1 -- 172.21.2.112:0/3947779915 wait complete. 2021-02-10T11:43:32.768 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.765 7f2856907700 1 Processor -- start 2021-02-10T11:43:32.768 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.765 7f2856907700 1 -- start start 2021-02-10T11:43:32.768 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.765 7f2856907700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f285009bcc0 0x7f285009c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:32.768 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.765 7f2856907700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f28500a0820 0x7f28500a0c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:32.768 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.765 7f2856907700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f28500a1240 0x7f28500858d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:32.769 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.765 7f2856907700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28500a66b0 con 0x7f285009bcc0 2021-02-10T11:43:32.769 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.765 7f2856907700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28501a49b0 con 0x7f28500a0820 2021-02-10T11:43:32.769 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.765 7f2856907700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28501a4b90 con 0x7f28500a1240 2021-02-10T11:43:32.769 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.766 7f284ffff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f285009bcc0 0x7f285009c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:32.769 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.766 7f284ffff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f285009bcc0 0x7f285009c0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:57158/0 (socket says 172.21.2.112:57158) 2021-02-10T11:43:32.769 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.766 7f284ffff700 1 -- 172.21.2.112:0/3353063847 learned_addr learned my addr 172.21.2.112:0/3353063847 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:43:32.769 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.766 7f284ffff700 1 -- 172.21.2.112:0/3353063847 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f28500a1240 msgr2=0x7f28500858d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:43:32.769 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.766 7f284ffff700 1 --2- 172.21.2.112:0/3353063847 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f28500a1240 0x7f28500858d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:32.769 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.766 7f284ffff700 1 -- 172.21.2.112:0/3353063847 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f28500a0820 msgr2=0x7f28500a0c20 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:43:32.769 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.766 7f284ffff700 1 --2- 172.21.2.112:0/3353063847 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f28500a0820 0x7f28500a0c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:32.769 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.766 7f284ffff700 1 -- 172.21.2.112:0/3353063847 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28500a1640 con 0x7f285009bcc0 2021-02-10T11:43:32.769 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.766 7f284ffff700 1 --2- 172.21.2.112:0/3353063847 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f285009bcc0 0x7f285009c0c0 secure :-1 s=READY pgs=2483 cs=0 l=1 rev1=1 rx=0x7f2840001c60 tx=0x7f284000a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:32.769 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.766 7f284cff9700 1 -- 172.21.2.112:0/3353063847 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f284000acc0 con 0x7f285009bcc0 2021-02-10T11:43:32.770 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.766 7f284cff9700 1 -- 172.21.2.112:0/3353063847 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2840004140 con 0x7f285009bcc0 2021-02-10T11:43:32.770 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.766 7f284cff9700 1 -- 172.21.2.112:0/3353063847 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2840004660 con 0x7f285009bcc0 2021-02-10T11:43:32.770 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.766 7f2856907700 1 -- 172.21.2.112:0/3353063847 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28500a1640 con 0x7f285009bcc0 2021-02-10T11:43:32.770 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.766 7f2856907700 1 -- 172.21.2.112:0/3353063847 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28500861e0 con 0x7f285009bcc0 2021-02-10T11:43:32.770 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.767 7f284cff9700 1 -- 172.21.2.112:0/3353063847 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2840004320 con 0x7f285009bcc0 2021-02-10T11:43:32.770 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.767 7f284cff9700 1 --2- 172.21.2.112:0/3353063847 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f28300186c0 0x7f283001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:32.770 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.767 7f284cff9700 1 -- 172.21.2.112:0/3353063847 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f28400323b0 con 0x7f285009bcc0 2021-02-10T11:43:32.770 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.768 7f284f7fe700 1 --2- 172.21.2.112:0/3353063847 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f28300186c0 0x7f283001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:32.770 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.768 7f284f7fe700 1 --2- 172.21.2.112:0/3353063847 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f28300186c0 0x7f283001abc0 crc :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:32.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.798 7f2856907700 1 -- 172.21.2.112:0/3353063847 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28500fe1c0 con 0x7f285009bcc0 2021-02-10T11:43:32.801 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.801 7f284cff9700 1 -- 172.21.2.112:0/3353063847 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2840000e20 con 0x7f285009bcc0 2021-02-10T11:43:32.951 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.951 7f2856907700 1 -- 172.21.2.112:0/3353063847 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f28500fe1c0 con 0x7f285009bcc0 2021-02-10T11:43:32.952 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.951 7f284cff9700 1 -- 172.21.2.112:0/3353063847 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v34)=0 v34) v1 ==== 122+0+0 (secure 0 0 0) 0x7f2840004320 con 0x7f285009bcc0 2021-02-10T11:43:32.954 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.953 7f2856907700 1 -- 172.21.2.112:0/3353063847 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f28300186c0 msgr2=0x7f283001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:32.954 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.953 7f2856907700 1 --2- 172.21.2.112:0/3353063847 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f28300186c0 0x7f283001abc0 crc :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:32.954 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.953 7f2856907700 1 -- 172.21.2.112:0/3353063847 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f285009bcc0 msgr2=0x7f285009c0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:32.954 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.953 7f2856907700 1 --2- 172.21.2.112:0/3353063847 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f285009bcc0 0x7f285009c0c0 secure :-1 s=READY pgs=2483 cs=0 l=1 rev1=1 rx=0x7f2840001c60 tx=0x7f284000a8d0).stop 2021-02-10T11:43:32.954 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.953 7f2856907700 1 -- 172.21.2.112:0/3353063847 shutdown_connections 2021-02-10T11:43:32.954 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.953 7f2856907700 1 --2- 172.21.2.112:0/3353063847 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f28300186c0 0x7f283001abc0 unknown :-1 s=CLOSED pgs=1137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:32.954 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.953 7f2856907700 1 --2- 172.21.2.112:0/3353063847 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f28500a0820 0x7f28500a0c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:32.954 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.953 7f2856907700 1 --2- 172.21.2.112:0/3353063847 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f28500a1240 0x7f28500858d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:32.954 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.953 7f2856907700 1 --2- 172.21.2.112:0/3353063847 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f285009bcc0 0x7f285009c0c0 unknown :-1 s=CLOSED pgs=2483 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:32.955 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.953 7f2856907700 1 -- 172.21.2.112:0/3353063847 >> 172.21.2.112:0/3353063847 conn(0x7f28500bef00 msgr2=0x7f285009d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:32.955 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.954 7f2856907700 1 -- 172.21.2.112:0/3353063847 shutdown_connections 2021-02-10T11:43:32.955 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:32.954 7f2856907700 1 -- 172.21.2.112:0/3353063847 wait complete. 2021-02-10T11:43:32.962 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000999951 --group_name non_existent_group 2021-02-10T11:43:33.395 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:33.378 7f4513b07700 1 Processor -- start 2021-02-10T11:43:33.395 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:33.378 7f4513b07700 1 -- start start 2021-02-10T11:43:33.395 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:33.379 7f4513b07700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f450c0d1870 0x7f450c1b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:33.395 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:33.379 7f4513b07700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f450c1b0ae0 0x7f450c108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:33.395 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:33.379 7f4513b07700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f450c0fe1c0 con 0x7f450c0d1c70 2021-02-10T11:43:33.395 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:33.379 7f4513b07700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f450c109a60 con 0x7f450c0d1870 2021-02-10T11:43:33.395 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:33.379 7f4513b07700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f450c109750 con 0x7f450c1b0ae0 2021-02-10T11:43:33.395 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:33.379 7f4512124700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f450c0d1c70 0x7f450c108f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:56632/0 (socket says 172.21.2.112:56632) 2021-02-10T11:43:33.395 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:33.379 7f4512124700 1 -- 172.21.2.112:0/2164251636 learned_addr learned my addr 172.21.2.112:0/2164251636 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:43:33.395 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:33.379 7f4511923700 1 --2- 172.21.2.112:0/2164251636 >> v2:172.21.2.112:3300/0 conn(0x7f450c0d1870 0x7f450c1b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:33.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:33.379 7f4510921700 1 -- 172.21.2.112:0/2164251636 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 243717479 0 0) 0x7f4504001500 con 0x7f450c0d1c70 2021-02-10T11:43:33.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:33.379 7f4510921700 1 -- 172.21.2.112:0/2164251636 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44f8002fe0 con 0x7f450c0d1c70 2021-02-10T11:43:33.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:33.379 7f4511923700 1 -- 172.21.2.112:0/2164251636 >> v2:172.21.2.120:3300/0 conn(0x7f450c1b0ae0 msgr2=0x7f450c108940 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:43:33.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:33.379 7f4511923700 1 --2- 172.21.2.112:0/2164251636 >> v2:172.21.2.120:3300/0 conn(0x7f450c1b0ae0 0x7f450c108940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:33.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:33.379 7f4511923700 1 -- 172.21.2.112:0/2164251636 >> v1:172.21.2.112:6789/0 conn(0x7f450c0d1c70 legacy=0x7f450c108f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:33.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:33.379 7f4511923700 1 -- 172.21.2.112:0/2164251636 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f450c088070 con 0x7f450c0d1870 2021-02-10T11:43:33.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:33.379 7f4511923700 1 --2- 172.21.2.112:0/2164251636 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f450c0d1870 0x7f450c1b04c0 secure :-1 s=READY pgs=2485 cs=0 l=1 rev1=1 rx=0x7f4500006650 tx=0x7f4500009a80).ready entity=mon.0 client_cookie=ac3e258a972f7767 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:33.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:33.380 7f4510921700 1 -- 172.21.2.112:0/2164251636 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f450000c0c0 con 0x7f450c0d1870 2021-02-10T11:43:33.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:33.380 7f4510921700 1 -- 172.21.2.112:0/2164251636 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f450000c340 con 0x7f450c0d1870 2021-02-10T11:43:33.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:33.380 7f4510921700 1 -- 172.21.2.112:0/2164251636 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f450000c860 con 0x7f450c0d1870 2021-02-10T11:43:33.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:33.380 7f4513b07700 1 -- 172.21.2.112:0/2164251636 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f450c0d1870 msgr2=0x7f450c1b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:33.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:33.380 7f4513b07700 1 --2- 172.21.2.112:0/2164251636 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f450c0d1870 0x7f450c1b04c0 secure :-1 s=READY pgs=2485 cs=0 l=1 rev1=1 rx=0x7f4500006650 tx=0x7f4500009a80).stop 2021-02-10T11:43:33.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:33.380 7f4513b07700 1 -- 172.21.2.112:0/2164251636 shutdown_connections 2021-02-10T11:43:33.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:33.380 7f4513b07700 1 --2- 172.21.2.112:0/2164251636 >> v2:172.21.2.120:3300/0 conn(0x7f450c1b0ae0 0x7f450c108940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:33.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:33.380 7f4513b07700 1 --2- 172.21.2.112:0/2164251636 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f450c0d1870 0x7f450c1b04c0 unknown :-1 s=CLOSED pgs=2485 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:33.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:33.380 7f4513b07700 1 -- 172.21.2.112:0/2164251636 >> 172.21.2.112:0/2164251636 conn(0x7f450c0ce420 msgr2=0x7f450c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:33.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:33.380 7f4513b07700 1 -- 172.21.2.112:0/2164251636 shutdown_connections 2021-02-10T11:43:33.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:33.380 7f4513b07700 1 -- 172.21.2.112:0/2164251636 wait complete. 2021-02-10T11:43:33.398 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:33.395 7f4513b07700 1 Processor -- start 2021-02-10T11:43:33.398 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:33.395 7f4513b07700 1 -- start start 2021-02-10T11:43:33.398 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:33.395 7f4513b07700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f450c19e5c0 0x7f450c099fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:33.399 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:33.395 7f4513b07700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f450c09a5d0 0x7f450c09a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:33.399 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:33.395 7f4513b07700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f450c0982a0 0x7f450c0986a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:33.399 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:33.395 7f4513b07700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f450c19feb0 con 0x7f450c09a5d0 2021-02-10T11:43:33.399 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:33.395 7f4513b07700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f450c099a30 con 0x7f450c19e5c0 2021-02-10T11:43:33.399 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:33.395 7f4513b07700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f450c19fb30 con 0x7f450c0982a0 2021-02-10T11:43:33.399 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:33.396 7f4511122700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f450c09a5d0 0x7f450c09a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:33.399 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:33.396 7f4511122700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f450c09a5d0 0x7f450c09a9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:57172/0 (socket says 172.21.2.112:57172) 2021-02-10T11:43:33.399 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:33.396 7f4511122700 1 -- 172.21.2.112:0/900330868 learned_addr learned my addr 172.21.2.112:0/900330868 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:43:33.399 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:33.396 7f4511122700 1 -- 172.21.2.112:0/900330868 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f450c0982a0 msgr2=0x7f450c0986a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:33.399 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:33.396 7f4511122700 1 --2- 172.21.2.112:0/900330868 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f450c0982a0 0x7f450c0986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:33.399 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:33.396 7f4511122700 1 -- 172.21.2.112:0/900330868 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f450c19e5c0 msgr2=0x7f450c099fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:33.399 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:33.396 7f4511122700 1 --2- 172.21.2.112:0/900330868 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f450c19e5c0 0x7f450c099fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:33.400 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:33.396 7f4511122700 1 -- 172.21.2.112:0/900330868 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f450c19e9c0 con 0x7f450c09a5d0 2021-02-10T11:43:33.400 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:33.396 7f4511122700 1 --2- 172.21.2.112:0/900330868 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f450c09a5d0 0x7f450c09a9d0 secure :-1 s=READY pgs=2486 cs=0 l=1 rev1=1 rx=0x7f450800b5c0 tx=0x7f450800b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:33.400 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:33.396 7f44fe7fc700 1 -- 172.21.2.112:0/900330868 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f450800bd50 con 0x7f450c09a5d0 2021-02-10T11:43:33.400 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:33.396 7f44fe7fc700 1 -- 172.21.2.112:0/900330868 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4508006030 con 0x7f450c09a5d0 2021-02-10T11:43:33.400 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:33.396 7f44fe7fc700 1 -- 172.21.2.112:0/900330868 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f45080054d0 con 0x7f450c09a5d0 2021-02-10T11:43:33.400 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:33.396 7f4513b07700 1 -- 172.21.2.112:0/900330868 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f450c19e9c0 con 0x7f450c09a5d0 2021-02-10T11:43:33.400 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:33.396 7f4513b07700 1 -- 172.21.2.112:0/900330868 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f450c19e9c0 con 0x7f450c09a5d0 2021-02-10T11:43:33.400 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:33.397 7f44fe7fc700 1 -- 172.21.2.112:0/900330868 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4508006210 con 0x7f450c09a5d0 2021-02-10T11:43:33.400 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:33.397 7f44fe7fc700 1 --2- 172.21.2.112:0/900330868 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f44f40186c0 0x7f44f401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:33.400 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:33.397 7f44fe7fc700 1 -- 172.21.2.112:0/900330868 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f45080325a0 con 0x7f450c09a5d0 2021-02-10T11:43:33.400 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:33.397 7f4511923700 1 --2- 172.21.2.112:0/900330868 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f44f40186c0 0x7f44f401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:33.400 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:33.398 7f4511923700 1 --2- 172.21.2.112:0/900330868 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f44f40186c0 0x7f44f401abc0 crc :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:33.429 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:33.429 7f4513b07700 1 -- 172.21.2.112:0/900330868 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f450c0825d0 con 0x7f450c09a5d0 2021-02-10T11:43:33.432 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:33.431 7f44fe7fc700 1 -- 172.21.2.112:0/900330868 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4508006b80 con 0x7f450c09a5d0 2021-02-10T11:43:33.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:33.559 7f4513b07700 1 -- 172.21.2.112:0/900330868 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000999951", "target": ["mgr", ""], "group_name": "non_existent_group"}) v1 -- 0x7f450c0825d0 con 0x7f44f40186c0 2021-02-10T11:43:33.564 INFO:tasks.ceph.mgr.x.gibba020.stderr:2021-02-10 11:43:33.564 7f6382eee700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:43:33.564 INFO:tasks.ceph.mgr.x.gibba020.stderr:2021-02-10 11:43:33.564 7f6382eee700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:43:33.565 INFO:tasks.ceph.mgr.x.gibba020.stderr:2021-02-10 11:43:33.564 7f6382eee700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:43:33.565 INFO:tasks.ceph.mgr.x.gibba020.stderr:2021-02-10 11:43:33.564 7f6382eee700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:43:33.565 INFO:tasks.ceph.mgr.x.gibba020.stderr:2021-02-10 11:43:33.564 7f6382eee700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:43:33.569 INFO:tasks.ceph.mgr.x.gibba020.stderr:2021-02-10 11:43:33.569 7f6382eee700 -1 mgr.server reply reply (2) No such file or directory subvolume group 'non_existent_group' does not exist 2021-02-10T11:43:33.569 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:33.569 7f44fe7fc700 1 -- 172.21.2.112:0/900330868 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: -2 subvolume group 'non_existent_group' does not exist) v1 ==== 59+0+0 (crc 0 0 0) 0x7f4500000940 con 0x7f44f40186c0 2021-02-10T11:43:33.575 INFO:teuthology.orchestra.run.gibba012.stderr:Error ENOENT: subvolume group 'non_existent_group' does not exist 2021-02-10T11:43:33.577 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:33.577 7f4513b07700 1 -- 172.21.2.112:0/900330868 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f44f40186c0 msgr2=0x7f44f401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:33.577 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:33.577 7f4513b07700 1 --2- 172.21.2.112:0/900330868 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f44f40186c0 0x7f44f401abc0 crc :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:33.577 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:33.577 7f4513b07700 1 -- 172.21.2.112:0/900330868 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f450c09a5d0 msgr2=0x7f450c09a9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:33.577 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:33.577 7f4513b07700 1 --2- 172.21.2.112:0/900330868 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f450c09a5d0 0x7f450c09a9d0 secure :-1 s=READY pgs=2486 cs=0 l=1 rev1=1 rx=0x7f450800b5c0 tx=0x7f450800b960).stop 2021-02-10T11:43:33.577 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:33.577 7f4513b07700 1 -- 172.21.2.112:0/900330868 shutdown_connections 2021-02-10T11:43:33.577 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:33.577 7f4513b07700 1 --2- 172.21.2.112:0/900330868 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f44f40186c0 0x7f44f401abc0 unknown :-1 s=CLOSED pgs=1138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:33.578 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:33.577 7f4513b07700 1 --2- 172.21.2.112:0/900330868 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f450c0982a0 0x7f450c0986a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:33.578 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:33.577 7f4513b07700 1 --2- 172.21.2.112:0/900330868 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f450c09a5d0 0x7f450c09a9d0 unknown :-1 s=CLOSED pgs=2486 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:33.578 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:33.577 7f4513b07700 1 --2- 172.21.2.112:0/900330868 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f450c19e5c0 0x7f450c099fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:33.578 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:33.577 7f4513b07700 1 -- 172.21.2.112:0/900330868 >> 172.21.2.112:0/900330868 conn(0x7f450c1b3560 msgr2=0x7f450c1b90e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:33.578 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:33.577 7f4513b07700 1 -- 172.21.2.112:0/900330868 shutdown_connections 2021-02-10T11:43:33.578 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:33.577 7f4513b07700 1 -- 172.21.2.112:0/900330868 wait complete. 2021-02-10T11:43:33.585 DEBUG:teuthology.orchestra.run:got remote process result: 2 2021-02-10T11:43:33.585 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:43:33.585 DEBUG:teuthology.orchestra.run.gibba012:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:43:33.932 DEBUG:teuthology.orchestra.run.gibba020:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:43:34.276 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:43:34.277 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:43:34.277 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:43:34.277 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:43:34.612 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:43:34.612 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:43:34.612 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba012.front.sepia.ceph.com... 2021-02-10T11:43:34.613 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:43:34.613 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:43:34.962 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba012.stderr:ceph-fuse[169337]: fuse finished with error 0 and tester_r 0 2021-02-10T11:43:34.963 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:43:34.964 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:43:34.964 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:43:35.299 INFO:teuthology.orchestra.run.gibba012.stdout:ext2/ext3 2021-02-10T11:43:35.299 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:43:35.299 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:43:35.299 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:43:35.635 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:43:35.635 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:43:35.635 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:43:35.970 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:43:35.970 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:43:35.970 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba012.front.sepia.ceph.com... 2021-02-10T11:43:35.970 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:43:35.971 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:43:36.327 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba012.stderr:ceph-fuse[170005]: fuse finished with error 0 and tester_r 0 2021-02-10T11:43:36.329 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:43:36.329 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:43:36.329 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:43:36.664 INFO:teuthology.orchestra.run.gibba012.stdout:ext2/ext3 2021-02-10T11:43:36.664 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:43:36.664 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:43:36.664 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:43:36.999 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:43:36.999 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:43:36.999 DEBUG:teuthology.orchestra.run.gibba020:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:43:37.334 INFO:teuthology.orchestra.run.gibba020.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:43:37.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:43:37.334 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:43:37.335 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:43:37.335 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:43:37.669 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:43:37.670 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:43:37.670 DEBUG:teuthology.orchestra.run.gibba020:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:43:38.003 INFO:teuthology.orchestra.run.gibba020.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:43:38.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:43:38.003 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:43:38.004 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:43:38.004 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:43:38.337 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config rm mon mon_allow_pool_delete 2021-02-10T11:43:38.770 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:38.753 7fab571bf700 1 Processor -- start 2021-02-10T11:43:38.770 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:38.753 7fab571bf700 1 -- start start 2021-02-10T11:43:38.770 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:38.753 7fab571bf700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fab500d1870 0x7fab501b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:38.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:38.753 7fab571bf700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fab501b0a10 0x7fab500cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:38.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:38.753 7fab571bf700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab500f0b10 con 0x7fab500d1c70 2021-02-10T11:43:38.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:38.753 7fab571bf700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab500e5ed0 con 0x7fab500d1870 2021-02-10T11:43:38.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:38.753 7fab571bf700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab500f0650 con 0x7fab501b0a10 2021-02-10T11:43:38.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:38.754 7fab54fdb700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fab500d1870 0x7fab501b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:38.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:38.754 7fab54fdb700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fab500d1870 0x7fab501b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:57180/0 (socket says 172.21.2.112:57180) 2021-02-10T11:43:38.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:38.754 7fab54fdb700 1 -- 172.21.2.112:0/3034470171 learned_addr learned my addr 172.21.2.112:0/3034470171 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:43:38.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:38.754 7fab4ffff700 1 --2- 172.21.2.112:0/3034470171 >> v2:172.21.2.120:3300/0 conn(0x7fab501b0a10 0x7fab500cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:38.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:38.754 7fab54fdb700 1 -- 172.21.2.112:0/3034470171 >> v2:172.21.2.120:3300/0 conn(0x7fab501b0a10 msgr2=0x7fab500cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:38.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:38.754 7fab54fdb700 1 --2- 172.21.2.112:0/3034470171 >> v2:172.21.2.120:3300/0 conn(0x7fab501b0a10 0x7fab500cf330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:38.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:38.754 7fab54fdb700 1 -- 172.21.2.112:0/3034470171 >> v1:172.21.2.112:6789/0 conn(0x7fab500d1c70 legacy=0x7fab500cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:38.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:38.754 7fab54fdb700 1 -- 172.21.2.112:0/3034470171 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab500f0840 con 0x7fab500d1870 2021-02-10T11:43:38.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:38.754 7fab4ffff700 1 --2- 172.21.2.112:0/3034470171 >> v2:172.21.2.120:3300/0 conn(0x7fab501b0a10 0x7fab500cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:43:38.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:38.755 7fab54fdb700 1 --2- 172.21.2.112:0/3034470171 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fab500d1870 0x7fab501b03f0 secure :-1 s=READY pgs=2488 cs=0 l=1 rev1=1 rx=0x7fab38006700 tx=0x7fab38009b80).ready entity=mon.0 client_cookie=d172282ef59311ea server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:38.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:38.755 7fab4f7fe700 1 -- 172.21.2.112:0/3034470171 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fab3800c0c0 con 0x7fab500d1870 2021-02-10T11:43:38.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:38.755 7fab4f7fe700 1 -- 172.21.2.112:0/3034470171 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab3800c440 con 0x7fab500d1870 2021-02-10T11:43:38.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:38.755 7fab4f7fe700 1 -- 172.21.2.112:0/3034470171 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fab3800c960 con 0x7fab500d1870 2021-02-10T11:43:38.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:38.755 7fab571bf700 1 -- 172.21.2.112:0/3034470171 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fab500d1870 msgr2=0x7fab501b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:38.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:38.755 7fab571bf700 1 --2- 172.21.2.112:0/3034470171 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fab500d1870 0x7fab501b03f0 secure :-1 s=READY pgs=2488 cs=0 l=1 rev1=1 rx=0x7fab38006700 tx=0x7fab38009b80).stop 2021-02-10T11:43:38.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:38.756 7fab571bf700 1 -- 172.21.2.112:0/3034470171 shutdown_connections 2021-02-10T11:43:38.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:38.756 7fab571bf700 1 --2- 172.21.2.112:0/3034470171 >> v2:172.21.2.120:3300/0 conn(0x7fab501b0a10 0x7fab500cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:38.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:38.756 7fab571bf700 1 --2- 172.21.2.112:0/3034470171 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fab500d1870 0x7fab501b03f0 unknown :-1 s=CLOSED pgs=2488 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:38.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:38.756 7fab571bf700 1 -- 172.21.2.112:0/3034470171 >> 172.21.2.112:0/3034470171 conn(0x7fab500ce450 msgr2=0x7fab500d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:38.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:38.756 7fab571bf700 1 -- 172.21.2.112:0/3034470171 shutdown_connections 2021-02-10T11:43:38.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:38.756 7fab571bf700 1 -- 172.21.2.112:0/3034470171 wait complete. 2021-02-10T11:43:38.774 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:38.770 7fab571bf700 1 Processor -- start 2021-02-10T11:43:38.774 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:38.770 7fab571bf700 1 -- start start 2021-02-10T11:43:38.774 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:38.771 7fab571bf700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fab501a4a80 0x7fab500906b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:38.774 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:38.771 7fab571bf700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fab500c30f0 0x7fab500c34f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:38.774 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:38.771 7fab571bf700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fab500ef5a0 0x7fab500ef9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:38.774 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:38.771 7fab571bf700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab500c3b10 con 0x7fab500ef5a0 2021-02-10T11:43:38.774 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:38.771 7fab571bf700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab500ef020 con 0x7fab500c30f0 2021-02-10T11:43:38.774 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:38.771 7fab571bf700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab500effc0 con 0x7fab501a4a80 2021-02-10T11:43:38.774 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:38.771 7fab557dc700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fab500ef5a0 0x7fab500ef9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:38.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:38.771 7fab557dc700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fab500ef5a0 0x7fab500ef9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:57188/0 (socket says 172.21.2.112:57188) 2021-02-10T11:43:38.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:38.771 7fab557dc700 1 -- 172.21.2.112:0/1753848414 learned_addr learned my addr 172.21.2.112:0/1753848414 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:43:38.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:38.771 7fab557dc700 1 -- 172.21.2.112:0/1753848414 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fab501a4a80 msgr2=0x7fab500906b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:38.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:38.771 7fab557dc700 1 --2- 172.21.2.112:0/1753848414 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fab501a4a80 0x7fab500906b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:38.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:38.771 7fab557dc700 1 -- 172.21.2.112:0/1753848414 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fab500c30f0 msgr2=0x7fab500c34f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:38.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:38.771 7fab557dc700 1 --2- 172.21.2.112:0/1753848414 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fab500c30f0 0x7fab500c34f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:38.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:38.771 7fab557dc700 1 -- 172.21.2.112:0/1753848414 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab500ebe70 con 0x7fab500ef5a0 2021-02-10T11:43:38.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:38.772 7fab557dc700 1 --2- 172.21.2.112:0/1753848414 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fab500ef5a0 0x7fab500ef9a0 secure :-1 s=READY pgs=2489 cs=0 l=1 rev1=1 rx=0x7fab48006f00 tx=0x7fab4800b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:38.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:38.772 7fab4d7fa700 1 -- 172.21.2.112:0/1753848414 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fab48006030 con 0x7fab500ef5a0 2021-02-10T11:43:38.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:38.772 7fab4d7fa700 1 -- 172.21.2.112:0/1753848414 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab480062f0 con 0x7fab500ef5a0 2021-02-10T11:43:38.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:38.772 7fab571bf700 1 -- 172.21.2.112:0/1753848414 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab500ebe70 con 0x7fab500ef5a0 2021-02-10T11:43:38.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:38.772 7fab4d7fa700 1 -- 172.21.2.112:0/1753848414 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fab48005770 con 0x7fab500ef5a0 2021-02-10T11:43:38.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:38.772 7fab571bf700 1 -- 172.21.2.112:0/1753848414 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab500ec400 con 0x7fab500ef5a0 2021-02-10T11:43:38.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:38.772 7fab4d7fa700 1 -- 172.21.2.112:0/1753848414 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fab480054f0 con 0x7fab500ef5a0 2021-02-10T11:43:38.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:38.773 7fab4d7fa700 1 --2- 172.21.2.112:0/1753848414 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fab34018730 0x7fab3401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:38.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:38.773 7fab4d7fa700 1 -- 172.21.2.112:0/1753848414 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fab480327c0 con 0x7fab500ef5a0 2021-02-10T11:43:38.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:38.773 7fab54fdb700 1 --2- 172.21.2.112:0/1753848414 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fab34018730 0x7fab3401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:38.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:38.774 7fab54fdb700 1 --2- 172.21.2.112:0/1753848414 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fab34018730 0x7fab3401ac10 crc :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:38.804 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:38.804 7fab571bf700 1 -- 172.21.2.112:0/1753848414 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab500f0b10 con 0x7fab500ef5a0 2021-02-10T11:43:38.806 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:38.806 7fab4d7fa700 1 -- 172.21.2.112:0/1753848414 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fab480054f0 con 0x7fab500ef5a0 2021-02-10T11:43:38.957 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:38.957 7fab571bf700 1 -- 172.21.2.112:0/1753848414 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fab500f0b10 con 0x7fab500ef5a0 2021-02-10T11:43:38.962 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:38.961 7fab4d7fa700 1 -- 172.21.2.112:0/1753848414 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v35) v1 ==== 104+0+0 (secure 0 0 0) 0x7fab480062f0 con 0x7fab500ef5a0 2021-02-10T11:43:38.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:38.966 7fab571bf700 1 -- 172.21.2.112:0/1753848414 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fab34018730 msgr2=0x7fab3401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:38.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:38.966 7fab571bf700 1 --2- 172.21.2.112:0/1753848414 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fab34018730 0x7fab3401ac10 crc :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:38.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:38.966 7fab571bf700 1 -- 172.21.2.112:0/1753848414 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fab500ef5a0 msgr2=0x7fab500ef9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:38.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:38.966 7fab571bf700 1 --2- 172.21.2.112:0/1753848414 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fab500ef5a0 0x7fab500ef9a0 secure :-1 s=READY pgs=2489 cs=0 l=1 rev1=1 rx=0x7fab48006f00 tx=0x7fab4800b940).stop 2021-02-10T11:43:38.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:38.967 7fab571bf700 1 -- 172.21.2.112:0/1753848414 shutdown_connections 2021-02-10T11:43:38.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:38.967 7fab571bf700 1 --2- 172.21.2.112:0/1753848414 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fab34018730 0x7fab3401ac10 unknown :-1 s=CLOSED pgs=1139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:38.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:38.967 7fab571bf700 1 --2- 172.21.2.112:0/1753848414 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fab500c30f0 0x7fab500c34f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:38.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:38.967 7fab571bf700 1 --2- 172.21.2.112:0/1753848414 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fab500ef5a0 0x7fab500ef9a0 unknown :-1 s=CLOSED pgs=2489 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:38.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:38.967 7fab571bf700 1 --2- 172.21.2.112:0/1753848414 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fab501a4a80 0x7fab500906b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:38.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:38.967 7fab571bf700 1 -- 172.21.2.112:0/1753848414 >> 172.21.2.112:0/1753848414 conn(0x7fab500a6bb0 msgr2=0x7fab50095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:38.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:38.967 7fab571bf700 1 -- 172.21.2.112:0/1753848414 shutdown_connections 2021-02-10T11:43:38.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:38.967 7fab571bf700 1 -- 172.21.2.112:0/1753848414 wait complete. 2021-02-10T11:43:38.975 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestVolumes.test_nonexistent_subvolume_group_create' 2021-02-10T11:43:39.404 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:39.386 7f03c4bc9700 1 Processor -- start 2021-02-10T11:43:39.405 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:39.386 7f03c4bc9700 1 -- start start 2021-02-10T11:43:39.405 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:39.387 7f03c4bc9700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f03c00d0340 0x7f03c00ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:39.405 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:39.387 7f03c4bc9700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f03c00d1c70 0x7f03c00efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:39.405 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:39.387 7f03c4bc9700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03c00f0b10 con 0x7f03c00d1870 2021-02-10T11:43:39.405 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:39.387 7f03c4bc9700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03c00e5ed0 con 0x7f03c00d0340 2021-02-10T11:43:39.405 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:39.387 7f03c4bc9700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03c01b0be0 con 0x7f03c00d1c70 2021-02-10T11:43:39.405 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:39.387 7f03be85e700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f03c00d1870 0x7f03c01b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:56654/0 (socket says 172.21.2.112:56654) 2021-02-10T11:43:39.405 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:39.387 7f03be85e700 1 -- 172.21.2.112:0/2642804905 learned_addr learned my addr 172.21.2.112:0/2642804905 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:43:39.405 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:39.388 7f03bd85c700 1 -- 172.21.2.112:0/2642804905 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2120351422 0 0) 0x7f03b0001180 con 0x7f03c00d1870 2021-02-10T11:43:39.405 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:39.388 7f03bd85c700 1 -- 172.21.2.112:0/2642804905 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03a8002fe0 con 0x7f03c00d1870 2021-02-10T11:43:39.405 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:39.388 7f03be05d700 1 --2- 172.21.2.112:0/2642804905 >> v2:172.21.2.120:3300/0 conn(0x7f03c00d0340 0x7f03c00ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:39.405 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:39.388 7f03bf05f700 1 --2- 172.21.2.112:0/2642804905 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f03c00d1c70 0x7f03c00efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:39.405 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:39.388 7f03bd85c700 1 -- 172.21.2.112:0/2642804905 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 963013188 0 0) 0x7f03b00010e0 con 0x7f03c00d1870 2021-02-10T11:43:39.406 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:39.388 7f03bd85c700 1 -- 172.21.2.112:0/2642804905 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f03c00d1c70 msgr2=0x7f03c00efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:39.406 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:39.388 7f03bd85c700 1 --2- 172.21.2.112:0/2642804905 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f03c00d1c70 0x7f03c00efc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:39.406 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:39.388 7f03bd85c700 1 -- 172.21.2.112:0/2642804905 >> v2:172.21.2.120:3300/0 conn(0x7f03c00d0340 msgr2=0x7f03c00ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:39.406 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:39.388 7f03bd85c700 1 --2- 172.21.2.112:0/2642804905 >> v2:172.21.2.120:3300/0 conn(0x7f03c00d0340 0x7f03c00ef640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:39.406 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:39.388 7f03bd85c700 1 -- 172.21.2.112:0/2642804905 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03c00f20a0 con 0x7f03c00d1870 2021-02-10T11:43:39.406 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:39.388 7f03bd85c700 1 -- 172.21.2.112:0/2642804905 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f03b0001600 con 0x7f03c00d1870 2021-02-10T11:43:39.406 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:39.388 7f03bd85c700 1 -- 172.21.2.112:0/2642804905 >> v1:172.21.2.112:6789/0 conn(0x7f03c00d1870 legacy=0x7f03c01b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:39.406 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:39.388 7f03bd85c700 1 --2- 172.21.2.112:0/2642804905 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f03a8003a60 0x7f03a8005f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:39.406 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:39.388 7f03bd85c700 1 --2- 172.21.2.112:0/2642804905 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f03a804a790 0x7f03a804cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:39.406 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:39.388 7f03bd85c700 1 --2- 172.21.2.112:0/2642804905 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f03a804d500 0x7f03a804fa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:39.406 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:39.388 7f03bd85c700 1 -- 172.21.2.112:0/2642804905 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03a80502d0 con 0x7f03a8003a60 2021-02-10T11:43:39.406 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:39.388 7f03bd85c700 1 -- 172.21.2.112:0/2642804905 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03a80504f0 con 0x7f03a804d500 2021-02-10T11:43:39.407 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:39.388 7f03bd85c700 1 -- 172.21.2.112:0/2642804905 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03a8050710 con 0x7f03a804a790 2021-02-10T11:43:39.407 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:39.388 7f03be85e700 1 --2- 172.21.2.112:0/2642804905 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f03a8003a60 0x7f03a8005f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:39.407 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:39.388 7f03be05d700 1 --2- 172.21.2.112:0/2642804905 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f03a804a790 0x7f03a804cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:39.407 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:39.388 7f03bf05f700 1 --2- 172.21.2.112:0/2642804905 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f03a804d500 0x7f03a804fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:39.407 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:39.389 7f03be85e700 1 -- 172.21.2.112:0/2642804905 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f03a804a790 msgr2=0x7f03a804cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:39.407 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:39.389 7f03be85e700 1 --2- 172.21.2.112:0/2642804905 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f03a804a790 0x7f03a804cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:39.407 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:39.389 7f03be85e700 1 -- 172.21.2.112:0/2642804905 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f03a804d500 msgr2=0x7f03a804fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:39.407 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:39.389 7f03be85e700 1 --2- 172.21.2.112:0/2642804905 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f03a804d500 0x7f03a804fa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:39.407 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:39.389 7f03be85e700 1 -- 172.21.2.112:0/2642804905 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03a80509b0 con 0x7f03a8003a60 2021-02-10T11:43:39.407 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:39.389 7f03be85e700 1 --2- 172.21.2.112:0/2642804905 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f03a8003a60 0x7f03a8005f60 secure :-1 s=READY pgs=2491 cs=0 l=1 rev1=1 rx=0x7f03b00018b0 tx=0x7f03b0007d40).ready entity=mon.0 client_cookie=f37ce6b04c2acc6a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:39.407 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:39.389 7f03bd85c700 1 -- 172.21.2.112:0/2642804905 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f03b00070c0 con 0x7f03a8003a60 2021-02-10T11:43:39.407 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:39.389 7f03bd85c700 1 -- 172.21.2.112:0/2642804905 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03b0001240 con 0x7f03a8003a60 2021-02-10T11:43:39.407 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:39.389 7f03bd85c700 1 -- 172.21.2.112:0/2642804905 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f03b000a7a0 con 0x7f03a8003a60 2021-02-10T11:43:39.408 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:39.389 7f03c4bc9700 1 -- 172.21.2.112:0/2642804905 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f03a8003a60 msgr2=0x7f03a8005f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:39.408 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:39.389 7f03c4bc9700 1 --2- 172.21.2.112:0/2642804905 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f03a8003a60 0x7f03a8005f60 secure :-1 s=READY pgs=2491 cs=0 l=1 rev1=1 rx=0x7f03b00018b0 tx=0x7f03b0007d40).stop 2021-02-10T11:43:39.408 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:39.389 7f03bf05f700 1 -- 172.21.2.112:0/2642804905 reap_dead start 2021-02-10T11:43:39.408 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:39.389 7f03c4bc9700 1 -- 172.21.2.112:0/2642804905 shutdown_connections 2021-02-10T11:43:39.408 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:39.389 7f03c4bc9700 1 -- 172.21.2.112:0/2642804905 >> 172.21.2.112:0/2642804905 conn(0x7f03c00ce450 msgr2=0x7f03c00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:39.408 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:39.390 7f03c4bc9700 1 -- 172.21.2.112:0/2642804905 shutdown_connections 2021-02-10T11:43:39.408 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:39.390 7f03c4bc9700 1 -- 172.21.2.112:0/2642804905 wait complete. 2021-02-10T11:43:39.408 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:39.404 7f03c4bc9700 1 Processor -- start 2021-02-10T11:43:39.408 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:39.404 7f03c4bc9700 1 -- start start 2021-02-10T11:43:39.409 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:39.405 7f03c4bc9700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f03c01a4a60 0x7f03c00c1c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:39.409 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:39.405 7f03c4bc9700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f03c00c2290 0x7f03c00c2690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:39.409 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:39.405 7f03c4bc9700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f03c00c2cb0 0x7f03c00bfd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:39.410 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:39.405 7f03c4bc9700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03c00f05b0 con 0x7f03c00c2290 2021-02-10T11:43:39.410 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:39.405 7f03c4bc9700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03c00f07d0 con 0x7f03c00c2cb0 2021-02-10T11:43:39.410 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:39.405 7f03c4bc9700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03c00c0340 con 0x7f03c01a4a60 2021-02-10T11:43:39.410 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:39.405 7f03be05d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f03c00c2290 0x7f03c00c2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:39.410 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:39.405 7f03be05d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f03c00c2290 0x7f03c00c2690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:57204/0 (socket says 172.21.2.112:57204) 2021-02-10T11:43:39.410 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:39.405 7f03be05d700 1 -- 172.21.2.112:0/3069786303 learned_addr learned my addr 172.21.2.112:0/3069786303 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:43:39.410 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:39.405 7f03bf05f700 1 --2- 172.21.2.112:0/3069786303 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f03c00c2cb0 0x7f03c00bfd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:39.410 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:39.405 7f03be85e700 1 --2- 172.21.2.112:0/3069786303 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f03c01a4a60 0x7f03c00c1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:39.410 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:39.405 7f03bf05f700 1 -- 172.21.2.112:0/3069786303 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f03c01a4a60 msgr2=0x7f03c00c1c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:39.410 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:39.405 7f03bf05f700 1 --2- 172.21.2.112:0/3069786303 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f03c01a4a60 0x7f03c00c1c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:39.410 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:39.405 7f03bf05f700 1 -- 172.21.2.112:0/3069786303 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f03c00c2290 msgr2=0x7f03c00c2690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:39.410 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:39.405 7f03bf05f700 1 --2- 172.21.2.112:0/3069786303 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f03c00c2290 0x7f03c00c2690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:39.411 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:39.405 7f03bf05f700 1 -- 172.21.2.112:0/3069786303 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03c00c0560 con 0x7f03c00c2cb0 2021-02-10T11:43:39.411 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:39.405 7f03bf05f700 1 --2- 172.21.2.112:0/3069786303 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f03c00c2cb0 0x7f03c00bfd20 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 rx=0x7f03b8006fc0 tx=0x7f03b800bcf0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:39.411 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:39.406 7f03af7fe700 1 -- 172.21.2.112:0/3069786303 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f03b8006c80 con 0x7f03c00c2cb0 2021-02-10T11:43:39.411 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:39.406 7f03af7fe700 1 -- 172.21.2.112:0/3069786303 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03b8003030 con 0x7f03c00c2cb0 2021-02-10T11:43:39.411 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:39.406 7f03c4bc9700 1 -- 172.21.2.112:0/3069786303 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03c00c0560 con 0x7f03c00c2cb0 2021-02-10T11:43:39.411 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:39.406 7f03c4bc9700 1 -- 172.21.2.112:0/3069786303 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03c00c0af0 con 0x7f03c00c2cb0 2021-02-10T11:43:39.411 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:39.406 7f03af7fe700 1 -- 172.21.2.112:0/3069786303 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f03b8006c80 con 0x7f03c00c2cb0 2021-02-10T11:43:39.411 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:39.406 7f03af7fe700 1 -- 172.21.2.112:0/3069786303 <== mon.1 v2:172.21.2.120:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f03b8006a00 con 0x7f03c00c2cb0 2021-02-10T11:43:39.411 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:39.406 7f03af7fe700 1 --2- 172.21.2.112:0/3069786303 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f03a40186c0 0x7f03a401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:39.411 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:39.406 7f03af7fe700 1 -- 172.21.2.112:0/3069786303 <== mon.1 v2:172.21.2.120:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f03b8032810 con 0x7f03c00c2cb0 2021-02-10T11:43:39.411 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:39.407 7f03be85e700 1 --2- 172.21.2.112:0/3069786303 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f03a40186c0 0x7f03a401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:39.411 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:39.407 7f03be85e700 1 --2- 172.21.2.112:0/3069786303 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f03a40186c0 0x7f03a401abc0 crc :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:39.437 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:39.437 7f03c4bc9700 1 -- 172.21.2.112:0/3069786303 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03c00f0b10 con 0x7f03c00c2cb0 2021-02-10T11:43:39.440 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:39.440 7f03af7fe700 1 -- 172.21.2.112:0/3069786303 <== mon.1 v2:172.21.2.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f03b80011f0 con 0x7f03c00c2cb0 2021-02-10T11:43:39.575 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:39.575 7f03c4bc9700 1 -- 172.21.2.112:0/3069786303 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_nonexistent_subvolume_group_create"]} v 0) v1 -- 0x7f03c00f0b10 con 0x7f03c00c2cb0 2021-02-10T11:43:40.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:40.235 7f03af7fe700 1 -- 172.21.2.112:0/3069786303 <== mon.1 v2:172.21.2.120:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_nonexistent_subvolume_group_create"]}]=0 v677) v1 ==== 156+0+0 (secure 0 0 0) 0x7f03b80056b0 con 0x7f03c00c2cb0 2021-02-10T11:43:40.245 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:40.245 7f03c4bc9700 1 -- 172.21.2.112:0/3069786303 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f03a40186c0 msgr2=0x7f03a401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:40.245 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:40.245 7f03c4bc9700 1 --2- 172.21.2.112:0/3069786303 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f03a40186c0 0x7f03a401abc0 crc :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:40.245 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:40.245 7f03c4bc9700 1 -- 172.21.2.112:0/3069786303 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f03c00c2cb0 msgr2=0x7f03c00bfd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:40.245 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:40.245 7f03c4bc9700 1 --2- 172.21.2.112:0/3069786303 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f03c00c2cb0 0x7f03c00bfd20 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 rx=0x7f03b8006fc0 tx=0x7f03b800bcf0).stop 2021-02-10T11:43:40.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:40.245 7f03c4bc9700 1 -- 172.21.2.112:0/3069786303 shutdown_connections 2021-02-10T11:43:40.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:40.245 7f03c4bc9700 1 --2- 172.21.2.112:0/3069786303 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f03a40186c0 0x7f03a401abc0 unknown :-1 s=CLOSED pgs=1140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:40.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:40.245 7f03c4bc9700 1 --2- 172.21.2.112:0/3069786303 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f03c00c2cb0 0x7f03c00bfd20 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:40.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:40.245 7f03c4bc9700 1 --2- 172.21.2.112:0/3069786303 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f03c00c2290 0x7f03c00c2690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:40.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:40.245 7f03c4bc9700 1 --2- 172.21.2.112:0/3069786303 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f03c01a4a60 0x7f03c00c1c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:40.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:40.245 7f03c4bc9700 1 -- 172.21.2.112:0/3069786303 >> 172.21.2.112:0/3069786303 conn(0x7f03c00ebe30 msgr2=0x7f03c00bf200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:40.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:40.245 7f03c4bc9700 1 -- 172.21.2.112:0/3069786303 shutdown_connections 2021-02-10T11:43:40.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:40.245 7f03c4bc9700 1 -- 172.21.2.112:0/3069786303 wait complete. 2021-02-10T11:43:40.254 INFO:tasks.cephfs_test_runner:test_nonexistent_subvolume_group_create (tasks.cephfs.test_volumes.TestVolumes) ... ok 2021-02-10T11:43:40.254 INFO:tasks.cephfs_test_runner:Starting test: test_nonexistent_subvolume_group_rm (tasks.cephfs.test_volumes.TestVolumes) 2021-02-10T11:43:40.254 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestVolumes.test_nonexistent_subvolume_group_rm' 2021-02-10T11:43:40.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:40.665 7f742f311700 1 Processor -- start 2021-02-10T11:43:40.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:40.665 7f742f311700 1 -- start start 2021-02-10T11:43:40.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:40.666 7f742f311700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f74280d1870 0x7f74281b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:40.684 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:40.666 7f742f311700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f74281b0a10 0x7f74280cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:40.684 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:40.666 7f742f311700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74280f0b10 con 0x7f74280d1c70 2021-02-10T11:43:40.684 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:40.666 7f742f311700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74280e5ed0 con 0x7f74280d1870 2021-02-10T11:43:40.684 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:40.666 7f742f311700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74280f0650 con 0x7f74281b0a10 2021-02-10T11:43:40.684 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:40.667 7f742d12d700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f74280d1870 0x7f74281b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:40.684 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:40.667 7f742c92c700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f74281b0a10 0x7f74280cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:40.684 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:40.667 7f742d92e700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7f74280d1c70 0x7f74280cf950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:50232/0 (socket says 172.21.2.112:50232) 2021-02-10T11:43:40.684 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:40.667 7f742d92e700 1 -- 172.21.2.112:0/2180870495 learned_addr learned my addr 172.21.2.112:0/2180870495 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:43:40.684 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:40.667 7f742d12d700 1 -- 172.21.2.112:0/2180870495 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f74281b0a10 msgr2=0x7f74280cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:40.684 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:40.667 7f742d12d700 1 --2- 172.21.2.112:0/2180870495 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f74281b0a10 0x7f74280cf330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:40.684 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:40.667 7f742d12d700 1 -- 172.21.2.112:0/2180870495 >> v1:172.21.2.120:6789/0 conn(0x7f74280d1c70 legacy=0x7f74280cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:40.684 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:40.667 7f742d12d700 1 -- 172.21.2.112:0/2180870495 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74280f0870 con 0x7f74280d1870 2021-02-10T11:43:40.685 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:40.668 7f742c92c700 1 --2- 172.21.2.112:0/2180870495 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f74281b0a10 0x7f74280cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:43:40.685 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:40.668 7f742d12d700 1 --2- 172.21.2.112:0/2180870495 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f74280d1870 0x7f74281b03f0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 rx=0x7f741c006650 tx=0x7f741c009a80).ready entity=mon.1 client_cookie=c40e7e9296f8e771 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:40.685 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:40.668 7f741bfff700 1 -- 172.21.2.112:0/2180870495 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f741c0009b0 con 0x7f74280d1870 2021-02-10T11:43:40.685 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:40.668 7f741bfff700 1 -- 172.21.2.112:0/2180870495 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f741c00c0c0 con 0x7f74280d1870 2021-02-10T11:43:40.685 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:40.668 7f741bfff700 1 -- 172.21.2.112:0/2180870495 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f741c00c5e0 con 0x7f74280d1870 2021-02-10T11:43:40.685 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:40.668 7f742f311700 1 -- 172.21.2.112:0/2180870495 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f74280d1870 msgr2=0x7f74281b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:40.685 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:40.668 7f742f311700 1 --2- 172.21.2.112:0/2180870495 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f74280d1870 0x7f74281b03f0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 rx=0x7f741c006650 tx=0x7f741c009a80).stop 2021-02-10T11:43:40.685 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:40.669 7f742f311700 1 -- 172.21.2.112:0/2180870495 shutdown_connections 2021-02-10T11:43:40.685 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:40.669 7f742f311700 1 --2- 172.21.2.112:0/2180870495 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f74281b0a10 0x7f74280cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:40.685 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:40.669 7f742f311700 1 --2- 172.21.2.112:0/2180870495 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f74280d1870 0x7f74281b03f0 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:40.685 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:40.669 7f742f311700 1 -- 172.21.2.112:0/2180870495 >> 172.21.2.112:0/2180870495 conn(0x7f74280ce450 msgr2=0x7f74280d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:40.685 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:40.669 7f742f311700 1 -- 172.21.2.112:0/2180870495 shutdown_connections 2021-02-10T11:43:40.685 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:40.669 7f742f311700 1 -- 172.21.2.112:0/2180870495 wait complete. 2021-02-10T11:43:40.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:40.682 7f742f311700 1 Processor -- start 2021-02-10T11:43:40.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:40.682 7f742f311700 1 -- start start 2021-02-10T11:43:40.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:40.683 7f742f311700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f742818d870 0x7f74280b19b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:40.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:40.683 7f742f311700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f74280b1fd0 0x7f7428099ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:40.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:40.683 7f742f311700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f74280b23d0 0x7f742809a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:40.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:40.683 7f742f311700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f742818dc70 con 0x7f74280b1fd0 2021-02-10T11:43:40.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:40.683 7f742f311700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7428199e60 con 0x7f74280b23d0 2021-02-10T11:43:40.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:40.683 7f742f311700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f742809ab30 con 0x7f742818d870 2021-02-10T11:43:40.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:40.683 7f742c92c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f74280b1fd0 0x7f7428099ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:40.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:40.683 7f742c92c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f74280b1fd0 0x7f7428099ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:57218/0 (socket says 172.21.2.112:57218) 2021-02-10T11:43:40.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:40.683 7f742c92c700 1 -- 172.21.2.112:0/4252716036 learned_addr learned my addr 172.21.2.112:0/4252716036 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:43:40.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:40.683 7f742c92c700 1 -- 172.21.2.112:0/4252716036 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f742818d870 msgr2=0x7f74280b19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:40.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:40.683 7f742c92c700 1 --2- 172.21.2.112:0/4252716036 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f742818d870 0x7f74280b19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:40.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:40.683 7f742c92c700 1 -- 172.21.2.112:0/4252716036 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f74280b23d0 msgr2=0x7f742809a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:40.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:40.683 7f742c92c700 1 --2- 172.21.2.112:0/4252716036 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f74280b23d0 0x7f742809a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:40.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:40.683 7f742c92c700 1 -- 172.21.2.112:0/4252716036 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f742809ad50 con 0x7f74280b1fd0 2021-02-10T11:43:40.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:40.684 7f742c92c700 1 --2- 172.21.2.112:0/4252716036 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f74280b1fd0 0x7f7428099ef0 secure :-1 s=READY pgs=2492 cs=0 l=1 rev1=1 rx=0x7f7424006f40 tx=0x7f742400b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:40.689 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:40.684 7f7419ffb700 1 -- 172.21.2.112:0/4252716036 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f742400bd80 con 0x7f74280b1fd0 2021-02-10T11:43:40.689 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:40.684 7f7419ffb700 1 -- 172.21.2.112:0/4252716036 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7424000c90 con 0x7f74280b1fd0 2021-02-10T11:43:40.689 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:40.684 7f7419ffb700 1 -- 172.21.2.112:0/4252716036 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7424002220 con 0x7f74280b1fd0 2021-02-10T11:43:40.689 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:40.684 7f742f311700 1 -- 172.21.2.112:0/4252716036 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f742809ad50 con 0x7f74280b1fd0 2021-02-10T11:43:40.689 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:40.684 7f742f311700 1 -- 172.21.2.112:0/4252716036 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f742818e3f0 con 0x7f74280b1fd0 2021-02-10T11:43:40.689 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:40.684 7f7419ffb700 1 -- 172.21.2.112:0/4252716036 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f742400a000 con 0x7f74280b1fd0 2021-02-10T11:43:40.689 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:40.684 7f7419ffb700 1 --2- 172.21.2.112:0/4252716036 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f74080186c0 0x7f740801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:40.689 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:40.684 7f7419ffb700 1 -- 172.21.2.112:0/4252716036 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f74240323f0 con 0x7f74280b1fd0 2021-02-10T11:43:40.689 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:40.685 7f742d12d700 1 --2- 172.21.2.112:0/4252716036 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f74080186c0 0x7f740801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:40.689 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:40.685 7f742d12d700 1 --2- 172.21.2.112:0/4252716036 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f74080186c0 0x7f740801abc0 crc :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:40.716 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:40.715 7f742f311700 1 -- 172.21.2.112:0/4252716036 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74280999c0 con 0x7f74280b1fd0 2021-02-10T11:43:40.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:40.718 7f7419ffb700 1 -- 172.21.2.112:0/4252716036 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7424000940 con 0x7f74280b1fd0 2021-02-10T11:43:40.852 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:40.850 7f742f311700 1 -- 172.21.2.112:0/4252716036 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_nonexistent_subvolume_group_rm"]} v 0) v1 -- 0x7f74280999c0 con 0x7f74280b1fd0 2021-02-10T11:43:41.239 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:41.237 7f7419ffb700 1 -- 172.21.2.112:0/4252716036 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_nonexistent_subvolume_group_rm"]}]=0 v678) v1 ==== 155+0+0 (secure 0 0 0) 0x7f742400a000 con 0x7f74280b1fd0 2021-02-10T11:43:41.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:41.269 7f742f311700 1 -- 172.21.2.112:0/4252716036 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f74080186c0 msgr2=0x7f740801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:41.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:41.269 7f742f311700 1 --2- 172.21.2.112:0/4252716036 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f74080186c0 0x7f740801abc0 crc :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:41.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:41.269 7f742f311700 1 -- 172.21.2.112:0/4252716036 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f74280b1fd0 msgr2=0x7f7428099ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:41.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:41.269 7f742f311700 1 --2- 172.21.2.112:0/4252716036 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f74280b1fd0 0x7f7428099ef0 secure :-1 s=READY pgs=2492 cs=0 l=1 rev1=1 rx=0x7f7424006f40 tx=0x7f742400b990).stop 2021-02-10T11:43:41.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:41.269 7f742f311700 1 -- 172.21.2.112:0/4252716036 shutdown_connections 2021-02-10T11:43:41.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:41.269 7f742f311700 1 --2- 172.21.2.112:0/4252716036 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f74080186c0 0x7f740801abc0 unknown :-1 s=CLOSED pgs=1141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:41.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:41.269 7f742f311700 1 --2- 172.21.2.112:0/4252716036 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f74280b23d0 0x7f742809a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:41.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:41.269 7f742f311700 1 --2- 172.21.2.112:0/4252716036 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f74280b1fd0 0x7f7428099ef0 unknown :-1 s=CLOSED pgs=2492 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:41.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:41.269 7f742f311700 1 --2- 172.21.2.112:0/4252716036 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f742818d870 0x7f74280b19b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:41.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:41.269 7f742f311700 1 -- 172.21.2.112:0/4252716036 >> 172.21.2.112:0/4252716036 conn(0x7f74280f0550 msgr2=0x7f7428191730 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:41.272 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:41.269 7f742f311700 1 -- 172.21.2.112:0/4252716036 shutdown_connections 2021-02-10T11:43:41.272 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:41.269 7f742f311700 1 -- 172.21.2.112:0/4252716036 wait complete. 2021-02-10T11:43:41.279 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:43:41.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:41.692 7fb71217a700 1 Processor -- start 2021-02-10T11:43:41.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:41.692 7fb71217a700 1 -- start start 2021-02-10T11:43:41.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:41.693 7fb71217a700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fb70c10aff0 0x7fb70c0cf750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:41.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:41.693 7fb71217a700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb70c0fe1c0 con 0x7fb70c0d1c70 2021-02-10T11:43:41.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:41.693 7fb71217a700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb70c1b0cb0 con 0x7fb70c0d1870 2021-02-10T11:43:41.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:41.693 7fb71217a700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb70c109a60 con 0x7fb70c10aff0 2021-02-10T11:43:41.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:41.693 7fb70affd700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fb70c0d1c70 0x7fb70c0cef60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:56686/0 (socket says 172.21.2.112:56686) 2021-02-10T11:43:41.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:41.693 7fb70affd700 1 -- 172.21.2.112:0/180041811 learned_addr learned my addr 172.21.2.112:0/180041811 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:43:41.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:41.693 7fb70a7fc700 1 -- 172.21.2.112:0/180041811 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1882449588 0 0) 0x7fb6f40009b0 con 0x7fb70c0d1c70 2021-02-10T11:43:41.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:41.693 7fb70a7fc700 1 -- 172.21.2.112:0/180041811 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6f8002fe0 con 0x7fb70c0d1c70 2021-02-10T11:43:41.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:41.694 7fb70bfff700 1 --2- 172.21.2.112:0/180041811 >> v2:172.21.2.120:3300/0 conn(0x7fb70c10aff0 0x7fb70c0cf750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:41.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:41.694 7fb70a7fc700 1 -- 172.21.2.112:0/180041811 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1424680874 0 0) 0x7fb6f4001010 con 0x7fb70c0d1c70 2021-02-10T11:43:41.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:41.694 7fb70a7fc700 1 -- 172.21.2.112:0/180041811 >> v2:172.21.2.120:3300/0 conn(0x7fb70c10aff0 msgr2=0x7fb70c0cf750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:41.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:41.694 7fb70a7fc700 1 --2- 172.21.2.112:0/180041811 >> v2:172.21.2.120:3300/0 conn(0x7fb70c10aff0 0x7fb70c0cf750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:41.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:41.694 7fb70a7fc700 1 -- 172.21.2.112:0/180041811 >> v1:172.21.2.120:6789/0 conn(0x7fb70c0d1870 legacy=0x7fb70c1b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:41.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:41.694 7fb70a7fc700 1 -- 172.21.2.112:0/180041811 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb70c10b3f0 con 0x7fb70c0d1c70 2021-02-10T11:43:41.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:41.694 7fb70a7fc700 1 -- 172.21.2.112:0/180041811 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7fb6f4001530 con 0x7fb70c0d1c70 2021-02-10T11:43:41.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:41.694 7fb70a7fc700 1 -- 172.21.2.112:0/180041811 >> v1:172.21.2.112:6789/0 conn(0x7fb70c0d1c70 legacy=0x7fb70c0cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:41.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:41.694 7fb70a7fc700 1 --2- 172.21.2.112:0/180041811 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb6f8003b10 0x7fb6f8005f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:41.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:41.694 7fb70a7fc700 1 --2- 172.21.2.112:0/180041811 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb6f80067a0 0x7fb6f8008ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:41.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:41.694 7fb70a7fc700 1 --2- 172.21.2.112:0/180041811 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb6f80095a0 0x7fb6f800baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:41.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:41.694 7fb70a7fc700 1 -- 172.21.2.112:0/180041811 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6f800c2f0 con 0x7fb6f80067a0 2021-02-10T11:43:41.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:41.694 7fb70a7fc700 1 -- 172.21.2.112:0/180041811 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6f800c510 con 0x7fb6f80095a0 2021-02-10T11:43:41.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:41.694 7fb70a7fc700 1 -- 172.21.2.112:0/180041811 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6f800c730 con 0x7fb6f8003b10 2021-02-10T11:43:41.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:41.694 7fb70affd700 1 --2- 172.21.2.112:0/180041811 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb6f80067a0 0x7fb6f8008ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:41.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:41.694 7fb70b7fe700 1 --2- 172.21.2.112:0/180041811 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb6f8003b10 0x7fb6f8005f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:41.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:41.694 7fb70bfff700 1 --2- 172.21.2.112:0/180041811 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb6f80095a0 0x7fb6f800baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:41.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:41.694 7fb70affd700 1 -- 172.21.2.112:0/180041811 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb6f8003b10 msgr2=0x7fb6f8005f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:41.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:41.694 7fb70affd700 1 --2- 172.21.2.112:0/180041811 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb6f8003b10 0x7fb6f8005f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:41.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:41.694 7fb70affd700 1 -- 172.21.2.112:0/180041811 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb6f80095a0 msgr2=0x7fb6f800baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:41.714 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:41.694 7fb70affd700 1 --2- 172.21.2.112:0/180041811 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb6f80095a0 0x7fb6f800baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:41.714 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:41.694 7fb70bfff700 1 -- 172.21.2.112:0/180041811 reap_dead start 2021-02-10T11:43:41.714 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:41.694 7fb70affd700 1 -- 172.21.2.112:0/180041811 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6f800c9d0 con 0x7fb6f80067a0 2021-02-10T11:43:41.714 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:41.694 7fb70affd700 1 --2- 172.21.2.112:0/180041811 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb6f80067a0 0x7fb6f8008ca0 secure :-1 s=READY pgs=2494 cs=0 l=1 rev1=1 rx=0x7fb6f40017d0 tx=0x7fb6f404bd40).ready entity=mon.0 client_cookie=207d16fa66eb36e3 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:41.714 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:41.694 7fb70a7fc700 1 -- 172.21.2.112:0/180041811 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb6f40013a0 con 0x7fb6f80067a0 2021-02-10T11:43:41.714 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:41.694 7fb70a7fc700 1 -- 172.21.2.112:0/180041811 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6f404e310 con 0x7fb6f80067a0 2021-02-10T11:43:41.714 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:41.694 7fb70a7fc700 1 -- 172.21.2.112:0/180041811 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb6f40009b0 con 0x7fb6f80067a0 2021-02-10T11:43:41.714 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:41.694 7fb71217a700 1 -- 172.21.2.112:0/180041811 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb6f80067a0 msgr2=0x7fb6f8008ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:41.714 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:41.694 7fb71217a700 1 --2- 172.21.2.112:0/180041811 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb6f80067a0 0x7fb6f8008ca0 secure :-1 s=READY pgs=2494 cs=0 l=1 rev1=1 rx=0x7fb6f40017d0 tx=0x7fb6f404bd40).stop 2021-02-10T11:43:41.714 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:41.695 7fb71217a700 1 -- 172.21.2.112:0/180041811 shutdown_connections 2021-02-10T11:43:41.714 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:41.695 7fb71217a700 1 --2- 172.21.2.112:0/180041811 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb6f80067a0 0x7fb6f8008ca0 unknown :-1 s=CLOSED pgs=2494 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:41.714 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:41.695 7fb71217a700 1 -- 172.21.2.112:0/180041811 >> 172.21.2.112:0/180041811 conn(0x7fb70c0ce420 msgr2=0x7fb70c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:41.714 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:41.695 7fb71217a700 1 -- 172.21.2.112:0/180041811 shutdown_connections 2021-02-10T11:43:41.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:41.695 7fb71217a700 1 -- 172.21.2.112:0/180041811 wait complete. 2021-02-10T11:43:41.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:41.710 7fb71217a700 1 Processor -- start 2021-02-10T11:43:41.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:41.710 7fb71217a700 1 -- start start 2021-02-10T11:43:41.716 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:41.710 7fb71217a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb70c09bcc0 0x7fb70c09c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:41.716 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:41.710 7fb71217a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb70c0a0820 0x7fb70c0a0c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:41.716 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:41.710 7fb71217a700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb70c0a1240 0x7fb70c0858d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:41.716 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:41.710 7fb71217a700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb70c0a66b0 con 0x7fb70c0a0820 2021-02-10T11:43:41.716 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:41.710 7fb71217a700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb70c1a49b0 con 0x7fb70c0a1240 2021-02-10T11:43:41.716 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:41.710 7fb71217a700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb70c1a4b90 con 0x7fb70c09bcc0 2021-02-10T11:43:41.717 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:41.710 7fb70affd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb70c0a0820 0x7fb70c0a0c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:41.717 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:41.710 7fb70affd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb70c0a0820 0x7fb70c0a0c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:57240/0 (socket says 172.21.2.112:57240) 2021-02-10T11:43:41.717 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:41.710 7fb70affd700 1 -- 172.21.2.112:0/3932484227 learned_addr learned my addr 172.21.2.112:0/3932484227 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:43:41.717 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:41.711 7fb70affd700 1 -- 172.21.2.112:0/3932484227 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb70c09bcc0 msgr2=0x7fb70c09c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:41.717 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:41.711 7fb70affd700 1 --2- 172.21.2.112:0/3932484227 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb70c09bcc0 0x7fb70c09c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:41.717 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:41.711 7fb70affd700 1 -- 172.21.2.112:0/3932484227 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb70c0a1240 msgr2=0x7fb70c0858d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:41.717 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:41.711 7fb70affd700 1 --2- 172.21.2.112:0/3932484227 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb70c0a1240 0x7fb70c0858d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:41.717 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:41.711 7fb70affd700 1 -- 172.21.2.112:0/3932484227 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb70c0a1640 con 0x7fb70c0a0820 2021-02-10T11:43:41.717 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:41.711 7fb70affd700 1 --2- 172.21.2.112:0/3932484227 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb70c0a0820 0x7fb70c0a0c20 secure :-1 s=READY pgs=2495 cs=0 l=1 rev1=1 rx=0x7fb6f4005610 tx=0x7fb6f40059b0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:41.717 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:41.711 7fb703fff700 1 -- 172.21.2.112:0/3932484227 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb6f4005da0 con 0x7fb70c0a0820 2021-02-10T11:43:41.717 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:41.711 7fb703fff700 1 -- 172.21.2.112:0/3932484227 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6f404f0d0 con 0x7fb70c0a0820 2021-02-10T11:43:41.717 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:41.711 7fb703fff700 1 -- 172.21.2.112:0/3932484227 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb6f404f5f0 con 0x7fb70c0a0820 2021-02-10T11:43:41.717 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:41.711 7fb71217a700 1 -- 172.21.2.112:0/3932484227 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb70c0a1640 con 0x7fb70c0a0820 2021-02-10T11:43:41.718 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:41.711 7fb71217a700 1 -- 172.21.2.112:0/3932484227 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb70c0a1640 con 0x7fb70c0a0820 2021-02-10T11:43:41.718 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:41.711 7fb703fff700 1 -- 172.21.2.112:0/3932484227 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb6f4004140 con 0x7fb70c0a0820 2021-02-10T11:43:41.718 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:41.712 7fb703fff700 1 --2- 172.21.2.112:0/3932484227 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb6f0018730 0x7fb6f001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:41.718 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:41.712 7fb703fff700 1 -- 172.21.2.112:0/3932484227 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fb6f4076630 con 0x7fb70c0a0820 2021-02-10T11:43:41.718 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:41.712 7fb70b7fe700 1 --2- 172.21.2.112:0/3932484227 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb6f0018730 0x7fb6f001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:41.718 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:41.712 7fb70b7fe700 1 --2- 172.21.2.112:0/3932484227 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb6f0018730 0x7fb6f001abc0 crc :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:41.744 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:41.743 7fb71217a700 1 -- 172.21.2.112:0/3932484227 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb70c0fe1c0 con 0x7fb70c0a0820 2021-02-10T11:43:41.746 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:41.745 7fb703fff700 1 -- 172.21.2.112:0/3932484227 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb6f4001790 con 0x7fb70c0a0820 2021-02-10T11:43:41.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:41.896 7fb71217a700 1 -- 172.21.2.112:0/3932484227 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fb70c0fe1c0 con 0x7fb70c0a0820 2021-02-10T11:43:41.901 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:41.900 7fb703fff700 1 -- 172.21.2.112:0/3932484227 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v36)=0 v36) v1 ==== 122+0+0 (secure 0 0 0) 0x7fb6f4004140 con 0x7fb70c0a0820 2021-02-10T11:43:41.906 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:41.905 7fb71217a700 1 -- 172.21.2.112:0/3932484227 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb6f0018730 msgr2=0x7fb6f001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:41.906 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:41.905 7fb71217a700 1 --2- 172.21.2.112:0/3932484227 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb6f0018730 0x7fb6f001abc0 crc :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:41.906 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:41.905 7fb71217a700 1 -- 172.21.2.112:0/3932484227 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb70c0a0820 msgr2=0x7fb70c0a0c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:41.907 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:41.905 7fb71217a700 1 --2- 172.21.2.112:0/3932484227 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb70c0a0820 0x7fb70c0a0c20 secure :-1 s=READY pgs=2495 cs=0 l=1 rev1=1 rx=0x7fb6f4005610 tx=0x7fb6f40059b0).stop 2021-02-10T11:43:41.907 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:41.905 7fb71217a700 1 -- 172.21.2.112:0/3932484227 shutdown_connections 2021-02-10T11:43:41.907 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:41.905 7fb71217a700 1 --2- 172.21.2.112:0/3932484227 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb6f0018730 0x7fb6f001abc0 unknown :-1 s=CLOSED pgs=1142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:41.907 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:41.905 7fb71217a700 1 --2- 172.21.2.112:0/3932484227 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb70c0a1240 0x7fb70c0858d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:41.907 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:41.905 7fb71217a700 1 --2- 172.21.2.112:0/3932484227 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb70c0a0820 0x7fb70c0a0c20 unknown :-1 s=CLOSED pgs=2495 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:41.907 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:41.905 7fb71217a700 1 --2- 172.21.2.112:0/3932484227 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb70c09bcc0 0x7fb70c09c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:41.907 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:41.905 7fb71217a700 1 -- 172.21.2.112:0/3932484227 >> 172.21.2.112:0/3932484227 conn(0x7fb70c0bef00 msgr2=0x7fb70c09d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:41.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:41.906 7fb71217a700 1 -- 172.21.2.112:0/3932484227 shutdown_connections 2021-02-10T11:43:41.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:41.906 7fb71217a700 1 -- 172.21.2.112:0/3932484227 wait complete. 2021-02-10T11:43:41.915 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:43:41.916 DEBUG:teuthology.orchestra.run.gibba012:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:43:42.261 DEBUG:teuthology.orchestra.run.gibba020:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:43:42.605 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:43:42.606 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:43:42.606 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:43:42.606 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:43:42.941 INFO:teuthology.orchestra.run.gibba012.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.0’: No such file or directory 2021-02-10T11:43:42.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:43:42.942 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2021-02-10T11:43:42.942 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:43:42.942 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:43:42.942 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:43:43.279 INFO:teuthology.orchestra.run.gibba012.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: No such file or directory 2021-02-10T11:43:43.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:43:43.279 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2021-02-10T11:43:43.279 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:43:43.279 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:43:43.279 DEBUG:teuthology.orchestra.run.gibba020:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:43:43.613 INFO:teuthology.orchestra.run.gibba020.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:43:43.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:43:43.614 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:43:43.614 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:43:43.614 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:43:43.614 DEBUG:teuthology.orchestra.run.gibba020:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:43:43.948 INFO:teuthology.orchestra.run.gibba020.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:43:43.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:43:43.949 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:43:43.949 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json-pretty 2021-02-10T11:43:44.375 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:44.358 7f81fa43f700 1 Processor -- start 2021-02-10T11:43:44.376 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:44.358 7f81fa43f700 1 -- start start 2021-02-10T11:43:44.376 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:44.358 7f81fa43f700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f81f40d0340 0x7f81f40ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:44.376 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:44.358 7f81fa43f700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81f40f0b10 con 0x7f81f40d1c70 2021-02-10T11:43:44.376 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:44.358 7f81fa43f700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81f41b0be0 con 0x7f81f40d1870 2021-02-10T11:43:44.376 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:44.358 7f81fa43f700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81f40e5ed0 con 0x7f81f40d0340 2021-02-10T11:43:44.376 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:44.358 7f81f37fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f81f40d0340 0x7f81f40ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:44.376 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:44.358 7f81f8a5c700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f81f40d1c70 0x7f81f40efc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:56710/0 (socket says 172.21.2.112:56710) 2021-02-10T11:43:44.376 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:44.358 7f81f37fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f81f40d0340 0x7f81f40ef640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:57244/0 (socket says 172.21.2.112:57244) 2021-02-10T11:43:44.376 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:44.358 7f81f8a5c700 1 -- 172.21.2.112:0/3369625629 learned_addr learned my addr 172.21.2.112:0/3369625629 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:43:44.376 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:44.359 7f81f37fe700 1 -- 172.21.2.112:0/3369625629 >> v1:172.21.2.120:6789/0 conn(0x7f81f40d1870 legacy=0x7f81f41b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:44.377 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:44.359 7f81f37fe700 1 -- 172.21.2.112:0/3369625629 >> v1:172.21.2.112:6789/0 conn(0x7f81f40d1c70 legacy=0x7f81f40efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:44.377 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:44.359 7f81f37fe700 1 -- 172.21.2.112:0/3369625629 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81f40f0450 con 0x7f81f40d0340 2021-02-10T11:43:44.377 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:44.359 7f81f37fe700 1 --2- 172.21.2.112:0/3369625629 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f81f40d0340 0x7f81f40ef640 secure :-1 s=READY pgs=2497 cs=0 l=1 rev1=1 rx=0x7f81e8006650 tx=0x7f81e8009a80).ready entity=mon.0 client_cookie=d6051c63fb04cced server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:44.377 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:44.359 7f81f2ffd700 1 -- 172.21.2.112:0/3369625629 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f81e800c0c0 con 0x7f81f40d0340 2021-02-10T11:43:44.377 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:44.359 7f81f2ffd700 1 -- 172.21.2.112:0/3369625629 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81e800c340 con 0x7f81f40d0340 2021-02-10T11:43:44.377 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:44.359 7f81f2ffd700 1 -- 172.21.2.112:0/3369625629 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f81e800c860 con 0x7f81f40d0340 2021-02-10T11:43:44.377 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:44.359 7f81fa43f700 1 -- 172.21.2.112:0/3369625629 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f81f40d0340 msgr2=0x7f81f40ef640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:44.377 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:44.359 7f81fa43f700 1 --2- 172.21.2.112:0/3369625629 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f81f40d0340 0x7f81f40ef640 secure :-1 s=READY pgs=2497 cs=0 l=1 rev1=1 rx=0x7f81e8006650 tx=0x7f81e8009a80).stop 2021-02-10T11:43:44.377 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:44.360 7f81fa43f700 1 -- 172.21.2.112:0/3369625629 shutdown_connections 2021-02-10T11:43:44.377 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:44.360 7f81fa43f700 1 --2- 172.21.2.112:0/3369625629 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f81f40d0340 0x7f81f40ef640 unknown :-1 s=CLOSED pgs=2497 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:44.377 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:44.360 7f81fa43f700 1 -- 172.21.2.112:0/3369625629 >> 172.21.2.112:0/3369625629 conn(0x7f81f40ce450 msgr2=0x7f81f40d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:44.377 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:44.360 7f81fa43f700 1 -- 172.21.2.112:0/3369625629 shutdown_connections 2021-02-10T11:43:44.377 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:44.360 7f81fa43f700 1 -- 172.21.2.112:0/3369625629 wait complete. 2021-02-10T11:43:44.377 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:44.374 7f81fa43f700 1 Processor -- start 2021-02-10T11:43:44.378 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:44.374 7f81fa43f700 1 -- start start 2021-02-10T11:43:44.379 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:44.375 7f81fa43f700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f81f4103520 0x7f81f40c3640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:44.379 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:44.375 7f81fa43f700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f81f4103920 0x7f81f40b32d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:44.379 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:44.375 7f81fa43f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f81f408a450 0x7f81f408a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:44.379 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:44.375 7f81fa43f700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81f40c3c60 con 0x7f81f408a450 2021-02-10T11:43:44.379 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:44.375 7f81fa43f700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81f40b38f0 con 0x7f81f4103520 2021-02-10T11:43:44.379 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:44.375 7f81fa43f700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81f40c0be0 con 0x7f81f4103920 2021-02-10T11:43:44.379 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:44.375 7f81f8a5c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f81f408a450 0x7f81f408a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:44.379 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:44.375 7f81f8a5c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f81f408a450 0x7f81f408a850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:57254/0 (socket says 172.21.2.112:57254) 2021-02-10T11:43:44.379 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:44.375 7f81f8a5c700 1 -- 172.21.2.112:0/1740257479 learned_addr learned my addr 172.21.2.112:0/1740257479 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:43:44.379 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:44.375 7f81f8a5c700 1 -- 172.21.2.112:0/1740257479 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f81f4103920 msgr2=0x7f81f40b32d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:44.380 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:44.375 7f81f8a5c700 1 --2- 172.21.2.112:0/1740257479 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f81f4103920 0x7f81f40b32d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:44.380 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:44.375 7f81f8a5c700 1 -- 172.21.2.112:0/1740257479 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f81f4103520 msgr2=0x7f81f40c3640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:44.380 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:44.375 7f81f8a5c700 1 --2- 172.21.2.112:0/1740257479 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f81f4103520 0x7f81f40c3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:44.380 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:44.375 7f81f8a5c700 1 -- 172.21.2.112:0/1740257479 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81f408ae70 con 0x7f81f408a450 2021-02-10T11:43:44.380 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:44.375 7f81f8a5c700 1 --2- 172.21.2.112:0/1740257479 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f81f408a450 0x7f81f408a850 secure :-1 s=READY pgs=2498 cs=0 l=1 rev1=1 rx=0x7f81ec00a900 tx=0x7f81ec00aca0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:44.380 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:44.376 7f81f0ff9700 1 -- 172.21.2.112:0/1740257479 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f81ec001270 con 0x7f81f408a450 2021-02-10T11:43:44.380 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:44.376 7f81f0ff9700 1 -- 172.21.2.112:0/1740257479 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81ec000d30 con 0x7f81f408a450 2021-02-10T11:43:44.380 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:44.376 7f81f0ff9700 1 -- 172.21.2.112:0/1740257479 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f81ec004bb0 con 0x7f81f408a450 2021-02-10T11:43:44.380 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:44.376 7f81fa43f700 1 -- 172.21.2.112:0/1740257479 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81f408ae70 con 0x7f81f408a450 2021-02-10T11:43:44.380 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:44.376 7f81fa43f700 1 -- 172.21.2.112:0/1740257479 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81f408b400 con 0x7f81f408a450 2021-02-10T11:43:44.380 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:44.376 7f81f0ff9700 1 -- 172.21.2.112:0/1740257479 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f81ec00ade0 con 0x7f81f408a450 2021-02-10T11:43:44.380 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:44.376 7f81f0ff9700 1 --2- 172.21.2.112:0/1740257479 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f81d40186c0 0x7f81d401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:44.380 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:44.376 7f81f0ff9700 1 -- 172.21.2.112:0/1740257479 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f81ec0328d0 con 0x7f81f408a450 2021-02-10T11:43:44.380 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:44.378 7f81f3fff700 1 --2- 172.21.2.112:0/1740257479 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f81d40186c0 0x7f81d401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:44.381 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:44.378 7f81f3fff700 1 --2- 172.21.2.112:0/1740257479 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f81d40186c0 0x7f81d401abc0 crc :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:44.410 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:44.409 7f81fa43f700 1 -- 172.21.2.112:0/1740257479 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81f40b38f0 con 0x7f81f408a450 2021-02-10T11:43:44.412 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:44.411 7f81f0ff9700 1 -- 172.21.2.112:0/1740257479 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f81ec001270 con 0x7f81f408a450 2021-02-10T11:43:44.537 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:44.536 7f81fa43f700 1 -- 172.21.2.112:0/1740257479 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f81f40b38f0 con 0x7f81f408a450 2021-02-10T11:43:44.538 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:44.537 7f81f0ff9700 1 -- 172.21.2.112:0/1740257479 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v134) v1 ==== 81+0+30374 (secure 0 0 0) 0x7f81ec00ade0 con 0x7f81f408a450 2021-02-10T11:43:44.539 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:43:44.539 INFO:teuthology.orchestra.run.gibba012.stdout:{ 2021-02-10T11:43:44.539 INFO:teuthology.orchestra.run.gibba012.stdout: "epoch": 134, 2021-02-10T11:43:44.539 INFO:teuthology.orchestra.run.gibba012.stdout: "fsid": "f14f004e-0699-4584-ad81-3754258493de", 2021-02-10T11:43:44.539 INFO:teuthology.orchestra.run.gibba012.stdout: "created": "2021-02-10 11:13:26.501424", 2021-02-10T11:43:44.539 INFO:teuthology.orchestra.run.gibba012.stdout: "modified": "2021-02-10 11:43:09.185424", 2021-02-10T11:43:44.539 INFO:teuthology.orchestra.run.gibba012.stdout: "last_up_change": "2021-02-10 11:13:52.780316", 2021-02-10T11:43:44.539 INFO:teuthology.orchestra.run.gibba012.stdout: "last_in_change": "2021-02-10 11:13:52.780316", 2021-02-10T11:43:44.539 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-02-10T11:43:44.540 INFO:teuthology.orchestra.run.gibba012.stdout: "flags_num": 5799936, 2021-02-10T11:43:44.540 INFO:teuthology.orchestra.run.gibba012.stdout: "flags_set": [ 2021-02-10T11:43:44.540 INFO:teuthology.orchestra.run.gibba012.stdout: "pglog_hardlimit", 2021-02-10T11:43:44.540 INFO:teuthology.orchestra.run.gibba012.stdout: "purged_snapdirs", 2021-02-10T11:43:44.540 INFO:teuthology.orchestra.run.gibba012.stdout: "recovery_deletes", 2021-02-10T11:43:44.540 INFO:teuthology.orchestra.run.gibba012.stdout: "sortbitwise" 2021-02-10T11:43:44.540 INFO:teuthology.orchestra.run.gibba012.stdout: ], 2021-02-10T11:43:44.540 INFO:teuthology.orchestra.run.gibba012.stdout: "crush_version": 4, 2021-02-10T11:43:44.540 INFO:teuthology.orchestra.run.gibba012.stdout: "full_ratio": 0.94999998807907104, 2021-02-10T11:43:44.540 INFO:teuthology.orchestra.run.gibba012.stdout: "backfillfull_ratio": 0.89999997615814209, 2021-02-10T11:43:44.540 INFO:teuthology.orchestra.run.gibba012.stdout: "nearfull_ratio": 0.85000002384185791, 2021-02-10T11:43:44.540 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_snapshot": "", 2021-02-10T11:43:44.540 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_max": 29, 2021-02-10T11:43:44.541 INFO:teuthology.orchestra.run.gibba012.stdout: "max_osd": 8, 2021-02-10T11:43:44.541 INFO:teuthology.orchestra.run.gibba012.stdout: "require_min_compat_client": "jewel", 2021-02-10T11:43:44.541 INFO:teuthology.orchestra.run.gibba012.stdout: "min_compat_client": "jewel", 2021-02-10T11:43:44.541 INFO:teuthology.orchestra.run.gibba012.stdout: "require_osd_release": "nautilus", 2021-02-10T11:43:44.541 INFO:teuthology.orchestra.run.gibba012.stdout: "pools": [ 2021-02-10T11:43:44.541 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:44.541 INFO:teuthology.orchestra.run.gibba012.stdout: "pool": 1, 2021-02-10T11:43:44.541 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_name": "rbd", 2021-02-10T11:43:44.541 INFO:teuthology.orchestra.run.gibba012.stdout: "create_time": "2021-02-10 11:13:57.508758", 2021-02-10T11:43:44.541 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:43:44.541 INFO:teuthology.orchestra.run.gibba012.stdout: "flags_names": "hashpspool", 2021-02-10T11:43:44.541 INFO:teuthology.orchestra.run.gibba012.stdout: "type": 1, 2021-02-10T11:43:44.541 INFO:teuthology.orchestra.run.gibba012.stdout: "size": 2, 2021-02-10T11:43:44.541 INFO:teuthology.orchestra.run.gibba012.stdout: "min_size": 1, 2021-02-10T11:43:44.542 INFO:teuthology.orchestra.run.gibba012.stdout: "crush_rule": 0, 2021-02-10T11:43:44.542 INFO:teuthology.orchestra.run.gibba012.stdout: "object_hash": 2, 2021-02-10T11:43:44.542 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:43:44.542 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num": 8, 2021-02-10T11:43:44.542 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num": 8, 2021-02-10T11:43:44.542 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num_target": 8, 2021-02-10T11:43:44.542 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_target": 8, 2021-02-10T11:43:44.542 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_pending": 8, 2021-02-10T11:43:44.542 INFO:teuthology.orchestra.run.gibba012.stdout: "last_pg_merge_meta": { 2021-02-10T11:43:44.542 INFO:teuthology.orchestra.run.gibba012.stdout: "source_pgid": "0.0", 2021-02-10T11:43:44.542 INFO:teuthology.orchestra.run.gibba012.stdout: "ready_epoch": 0, 2021-02-10T11:43:44.542 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_started": 0, 2021-02-10T11:43:44.542 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_clean": 0, 2021-02-10T11:43:44.543 INFO:teuthology.orchestra.run.gibba012.stdout: "source_version": "0'0", 2021-02-10T11:43:44.543 INFO:teuthology.orchestra.run.gibba012.stdout: "target_version": "0'0" 2021-02-10T11:43:44.543 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:44.543 INFO:teuthology.orchestra.run.gibba012.stdout: "last_change": "16", 2021-02-10T11:43:44.543 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend": "0", 2021-02-10T11:43:44.543 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:43:44.543 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:43:44.543 INFO:teuthology.orchestra.run.gibba012.stdout: "auid": 0, 2021-02-10T11:43:44.543 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:43:44.543 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_seq": 0, 2021-02-10T11:43:44.543 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_epoch": 0, 2021-02-10T11:43:44.543 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_snaps": [], 2021-02-10T11:43:44.544 INFO:teuthology.orchestra.run.gibba012.stdout: "removed_snaps": "[]", 2021-02-10T11:43:44.544 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_bytes": 0, 2021-02-10T11:43:44.544 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_objects": 0, 2021-02-10T11:43:44.544 INFO:teuthology.orchestra.run.gibba012.stdout: "tiers": [], 2021-02-10T11:43:44.544 INFO:teuthology.orchestra.run.gibba012.stdout: "tier_of": -1, 2021-02-10T11:43:44.544 INFO:teuthology.orchestra.run.gibba012.stdout: "read_tier": -1, 2021-02-10T11:43:44.544 INFO:teuthology.orchestra.run.gibba012.stdout: "write_tier": -1, 2021-02-10T11:43:44.544 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_mode": "none", 2021-02-10T11:43:44.544 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_bytes": 0, 2021-02-10T11:43:44.544 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_objects": 0, 2021-02-10T11:43:44.544 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:43:44.544 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:43:44.544 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:43:44.545 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_flush_age": 0, 2021-02-10T11:43:44.545 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_evict_age": 0, 2021-02-10T11:43:44.545 INFO:teuthology.orchestra.run.gibba012.stdout: "erasure_code_profile": "", 2021-02-10T11:43:44.545 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_params": { 2021-02-10T11:43:44.545 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "none" 2021-02-10T11:43:44.545 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:44.545 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_period": 0, 2021-02-10T11:43:44.545 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_count": 0, 2021-02-10T11:43:44.545 INFO:teuthology.orchestra.run.gibba012.stdout: "use_gmt_hitset": true, 2021-02-10T11:43:44.545 INFO:teuthology.orchestra.run.gibba012.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:43:44.545 INFO:teuthology.orchestra.run.gibba012.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:43:44.545 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:43:44.545 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:43:44.545 INFO:teuthology.orchestra.run.gibba012.stdout: "grade_table": [], 2021-02-10T11:43:44.546 INFO:teuthology.orchestra.run.gibba012.stdout: "stripe_width": 0, 2021-02-10T11:43:44.546 INFO:teuthology.orchestra.run.gibba012.stdout: "expected_num_objects": 0, 2021-02-10T11:43:44.546 INFO:teuthology.orchestra.run.gibba012.stdout: "fast_read": false, 2021-02-10T11:43:44.546 INFO:teuthology.orchestra.run.gibba012.stdout: "options": {}, 2021-02-10T11:43:44.546 INFO:teuthology.orchestra.run.gibba012.stdout: "application_metadata": { 2021-02-10T11:43:44.546 INFO:teuthology.orchestra.run.gibba012.stdout: "rbd": {} 2021-02-10T11:43:44.546 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:43:44.546 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:44.546 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:44.546 INFO:teuthology.orchestra.run.gibba012.stdout: "pool": 28, 2021-02-10T11:43:44.546 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_name": "cephfs_metadata", 2021-02-10T11:43:44.546 INFO:teuthology.orchestra.run.gibba012.stdout: "create_time": "2021-02-10 11:43:06.157376", 2021-02-10T11:43:44.546 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:43:44.547 INFO:teuthology.orchestra.run.gibba012.stdout: "flags_names": "hashpspool", 2021-02-10T11:43:44.547 INFO:teuthology.orchestra.run.gibba012.stdout: "type": 1, 2021-02-10T11:43:44.547 INFO:teuthology.orchestra.run.gibba012.stdout: "size": 2, 2021-02-10T11:43:44.547 INFO:teuthology.orchestra.run.gibba012.stdout: "min_size": 1, 2021-02-10T11:43:44.547 INFO:teuthology.orchestra.run.gibba012.stdout: "crush_rule": 0, 2021-02-10T11:43:44.547 INFO:teuthology.orchestra.run.gibba012.stdout: "object_hash": 2, 2021-02-10T11:43:44.547 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:43:44.547 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num": 8, 2021-02-10T11:43:44.547 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num": 8, 2021-02-10T11:43:44.547 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num_target": 8, 2021-02-10T11:43:44.547 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_target": 8, 2021-02-10T11:43:44.547 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_pending": 8, 2021-02-10T11:43:44.547 INFO:teuthology.orchestra.run.gibba012.stdout: "last_pg_merge_meta": { 2021-02-10T11:43:44.548 INFO:teuthology.orchestra.run.gibba012.stdout: "source_pgid": "0.0", 2021-02-10T11:43:44.548 INFO:teuthology.orchestra.run.gibba012.stdout: "ready_epoch": 0, 2021-02-10T11:43:44.548 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_started": 0, 2021-02-10T11:43:44.548 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_clean": 0, 2021-02-10T11:43:44.548 INFO:teuthology.orchestra.run.gibba012.stdout: "source_version": "0'0", 2021-02-10T11:43:44.548 INFO:teuthology.orchestra.run.gibba012.stdout: "target_version": "0'0" 2021-02-10T11:43:44.548 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:44.548 INFO:teuthology.orchestra.run.gibba012.stdout: "last_change": "133", 2021-02-10T11:43:44.548 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend": "0", 2021-02-10T11:43:44.548 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:43:44.548 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:43:44.548 INFO:teuthology.orchestra.run.gibba012.stdout: "auid": 0, 2021-02-10T11:43:44.548 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:43:44.549 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_seq": 0, 2021-02-10T11:43:44.549 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_epoch": 0, 2021-02-10T11:43:44.549 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_snaps": [], 2021-02-10T11:43:44.549 INFO:teuthology.orchestra.run.gibba012.stdout: "removed_snaps": "[]", 2021-02-10T11:43:44.549 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_bytes": 0, 2021-02-10T11:43:44.549 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_objects": 0, 2021-02-10T11:43:44.549 INFO:teuthology.orchestra.run.gibba012.stdout: "tiers": [], 2021-02-10T11:43:44.549 INFO:teuthology.orchestra.run.gibba012.stdout: "tier_of": -1, 2021-02-10T11:43:44.549 INFO:teuthology.orchestra.run.gibba012.stdout: "read_tier": -1, 2021-02-10T11:43:44.549 INFO:teuthology.orchestra.run.gibba012.stdout: "write_tier": -1, 2021-02-10T11:43:44.549 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_mode": "none", 2021-02-10T11:43:44.549 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_bytes": 0, 2021-02-10T11:43:44.549 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_objects": 0, 2021-02-10T11:43:44.549 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:43:44.550 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:43:44.550 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:43:44.550 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_flush_age": 0, 2021-02-10T11:43:44.550 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_evict_age": 0, 2021-02-10T11:43:44.550 INFO:teuthology.orchestra.run.gibba012.stdout: "erasure_code_profile": "", 2021-02-10T11:43:44.550 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_params": { 2021-02-10T11:43:44.550 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "none" 2021-02-10T11:43:44.550 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:44.550 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_period": 0, 2021-02-10T11:43:44.550 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_count": 0, 2021-02-10T11:43:44.550 INFO:teuthology.orchestra.run.gibba012.stdout: "use_gmt_hitset": true, 2021-02-10T11:43:44.550 INFO:teuthology.orchestra.run.gibba012.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:43:44.551 INFO:teuthology.orchestra.run.gibba012.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:43:44.551 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:43:44.551 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:43:44.551 INFO:teuthology.orchestra.run.gibba012.stdout: "grade_table": [], 2021-02-10T11:43:44.551 INFO:teuthology.orchestra.run.gibba012.stdout: "stripe_width": 0, 2021-02-10T11:43:44.551 INFO:teuthology.orchestra.run.gibba012.stdout: "expected_num_objects": 0, 2021-02-10T11:43:44.551 INFO:teuthology.orchestra.run.gibba012.stdout: "fast_read": false, 2021-02-10T11:43:44.551 INFO:teuthology.orchestra.run.gibba012.stdout: "options": { 2021-02-10T11:43:44.551 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_autoscale_bias": 4, 2021-02-10T11:43:44.551 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_min": 16, 2021-02-10T11:43:44.551 INFO:teuthology.orchestra.run.gibba012.stdout: "recovery_priority": 5 2021-02-10T11:43:44.551 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:44.551 INFO:teuthology.orchestra.run.gibba012.stdout: "application_metadata": { 2021-02-10T11:43:44.551 INFO:teuthology.orchestra.run.gibba012.stdout: "cephfs": { 2021-02-10T11:43:44.552 INFO:teuthology.orchestra.run.gibba012.stdout: "metadata": "cephfs" 2021-02-10T11:43:44.552 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:43:44.552 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:43:44.552 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:44.552 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:44.552 INFO:teuthology.orchestra.run.gibba012.stdout: "pool": 29, 2021-02-10T11:43:44.552 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_name": "cephfs_data", 2021-02-10T11:43:44.552 INFO:teuthology.orchestra.run.gibba012.stdout: "create_time": "2021-02-10 11:43:07.178112", 2021-02-10T11:43:44.552 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:43:44.552 INFO:teuthology.orchestra.run.gibba012.stdout: "flags_names": "hashpspool", 2021-02-10T11:43:44.552 INFO:teuthology.orchestra.run.gibba012.stdout: "type": 1, 2021-02-10T11:43:44.552 INFO:teuthology.orchestra.run.gibba012.stdout: "size": 2, 2021-02-10T11:43:44.553 INFO:teuthology.orchestra.run.gibba012.stdout: "min_size": 1, 2021-02-10T11:43:44.553 INFO:teuthology.orchestra.run.gibba012.stdout: "crush_rule": 0, 2021-02-10T11:43:44.553 INFO:teuthology.orchestra.run.gibba012.stdout: "object_hash": 2, 2021-02-10T11:43:44.553 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:43:44.553 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num": 8, 2021-02-10T11:43:44.553 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num": 8, 2021-02-10T11:43:44.553 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num_target": 8, 2021-02-10T11:43:44.553 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_target": 8, 2021-02-10T11:43:44.553 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_pending": 8, 2021-02-10T11:43:44.553 INFO:teuthology.orchestra.run.gibba012.stdout: "last_pg_merge_meta": { 2021-02-10T11:43:44.553 INFO:teuthology.orchestra.run.gibba012.stdout: "source_pgid": "0.0", 2021-02-10T11:43:44.553 INFO:teuthology.orchestra.run.gibba012.stdout: "ready_epoch": 0, 2021-02-10T11:43:44.553 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_started": 0, 2021-02-10T11:43:44.554 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_clean": 0, 2021-02-10T11:43:44.554 INFO:teuthology.orchestra.run.gibba012.stdout: "source_version": "0'0", 2021-02-10T11:43:44.554 INFO:teuthology.orchestra.run.gibba012.stdout: "target_version": "0'0" 2021-02-10T11:43:44.554 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:44.554 INFO:teuthology.orchestra.run.gibba012.stdout: "last_change": "134", 2021-02-10T11:43:44.554 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend": "0", 2021-02-10T11:43:44.554 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:43:44.554 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:43:44.554 INFO:teuthology.orchestra.run.gibba012.stdout: "auid": 0, 2021-02-10T11:43:44.554 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:43:44.554 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_seq": 0, 2021-02-10T11:43:44.554 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_epoch": 0, 2021-02-10T11:43:44.554 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_snaps": [], 2021-02-10T11:43:44.555 INFO:teuthology.orchestra.run.gibba012.stdout: "removed_snaps": "[]", 2021-02-10T11:43:44.555 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_bytes": 0, 2021-02-10T11:43:44.555 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_objects": 0, 2021-02-10T11:43:44.555 INFO:teuthology.orchestra.run.gibba012.stdout: "tiers": [], 2021-02-10T11:43:44.555 INFO:teuthology.orchestra.run.gibba012.stdout: "tier_of": -1, 2021-02-10T11:43:44.555 INFO:teuthology.orchestra.run.gibba012.stdout: "read_tier": -1, 2021-02-10T11:43:44.555 INFO:teuthology.orchestra.run.gibba012.stdout: "write_tier": -1, 2021-02-10T11:43:44.555 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_mode": "none", 2021-02-10T11:43:44.555 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_bytes": 0, 2021-02-10T11:43:44.555 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_objects": 0, 2021-02-10T11:43:44.555 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:43:44.555 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:43:44.555 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:43:44.556 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_flush_age": 0, 2021-02-10T11:43:44.556 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_evict_age": 0, 2021-02-10T11:43:44.556 INFO:teuthology.orchestra.run.gibba012.stdout: "erasure_code_profile": "", 2021-02-10T11:43:44.556 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_params": { 2021-02-10T11:43:44.556 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "none" 2021-02-10T11:43:44.556 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:44.556 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_period": 0, 2021-02-10T11:43:44.556 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_count": 0, 2021-02-10T11:43:44.556 INFO:teuthology.orchestra.run.gibba012.stdout: "use_gmt_hitset": true, 2021-02-10T11:43:44.556 INFO:teuthology.orchestra.run.gibba012.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:43:44.556 INFO:teuthology.orchestra.run.gibba012.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:43:44.556 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:43:44.556 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:43:44.557 INFO:teuthology.orchestra.run.gibba012.stdout: "grade_table": [], 2021-02-10T11:43:44.557 INFO:teuthology.orchestra.run.gibba012.stdout: "stripe_width": 0, 2021-02-10T11:43:44.557 INFO:teuthology.orchestra.run.gibba012.stdout: "expected_num_objects": 0, 2021-02-10T11:43:44.557 INFO:teuthology.orchestra.run.gibba012.stdout: "fast_read": false, 2021-02-10T11:43:44.557 INFO:teuthology.orchestra.run.gibba012.stdout: "options": {}, 2021-02-10T11:43:44.557 INFO:teuthology.orchestra.run.gibba012.stdout: "application_metadata": { 2021-02-10T11:43:44.557 INFO:teuthology.orchestra.run.gibba012.stdout: "cephfs": { 2021-02-10T11:43:44.557 INFO:teuthology.orchestra.run.gibba012.stdout: "data": "cephfs" 2021-02-10T11:43:44.557 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:43:44.557 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:43:44.557 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:43:44.557 INFO:teuthology.orchestra.run.gibba012.stdout: ], 2021-02-10T11:43:44.557 INFO:teuthology.orchestra.run.gibba012.stdout: "osds": [ 2021-02-10T11:43:44.558 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:44.558 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 0, 2021-02-10T11:43:44.558 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "95444b8f-6370-4a1d-ae5f-df62651b77da", 2021-02-10T11:43:44.558 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:43:44.558 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:43:44.558 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:43:44.558 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:43:44.558 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:43:44.558 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:43:44.558 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:43:44.558 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 130, 2021-02-10T11:43:44.558 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:43:44.559 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:43:44.559 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:43:44.559 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:43:44.559 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:44.559 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:43:44.559 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6817", 2021-02-10T11:43:44.559 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:43:44.559 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:44.559 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:44.559 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:43:44.559 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6818", 2021-02-10T11:43:44.559 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:43:44.559 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:43:44.560 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:43:44.560 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:44.560 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:43:44.560 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:43:44.560 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:44.560 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:43:44.560 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6820", 2021-02-10T11:43:44.560 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:43:44.560 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:44.560 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:44.560 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:43:44.560 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6822", 2021-02-10T11:43:44.560 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:43:44.561 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:43:44.561 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:43:44.561 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:44.561 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:43:44.561 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:43:44.561 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:44.561 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:43:44.561 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6828", 2021-02-10T11:43:44.561 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:43:44.561 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:44.561 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:44.561 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:43:44.561 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6830", 2021-02-10T11:43:44.562 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:43:44.562 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:43:44.562 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:43:44.562 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:44.562 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:43:44.562 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:43:44.562 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:44.562 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:43:44.562 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6824", 2021-02-10T11:43:44.562 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:43:44.562 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:44.562 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:44.562 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:43:44.563 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6826", 2021-02-10T11:43:44.563 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:43:44.563 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:43:44.563 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:43:44.563 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:44.563 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.112:6818/36030", 2021-02-10T11:43:44.563 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.112:6822/36030", 2021-02-10T11:43:44.563 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.112:6830/36030", 2021-02-10T11:43:44.563 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.112:6826/36030", 2021-02-10T11:43:44.563 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:43:44.563 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:43:44.563 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:43:44.563 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:43:44.564 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:44.564 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:44.564 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 1, 2021-02-10T11:43:44.564 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "a23e217e-0d8e-486e-be07-712adfabd262", 2021-02-10T11:43:44.564 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:43:44.564 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:43:44.564 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:43:44.564 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:43:44.564 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:43:44.564 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:43:44.564 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:43:44.564 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 132, 2021-02-10T11:43:44.564 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:43:44.565 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:43:44.565 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:43:44.565 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:43:44.565 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:44.565 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:43:44.565 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6800", 2021-02-10T11:43:44.565 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:43:44.565 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:44.565 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:44.565 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:43:44.565 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6802", 2021-02-10T11:43:44.565 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:43:44.565 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:43:44.566 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:43:44.566 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:44.566 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:43:44.566 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:43:44.566 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:44.566 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:43:44.566 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6804", 2021-02-10T11:43:44.566 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:43:44.566 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:44.566 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:44.566 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:43:44.566 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6806", 2021-02-10T11:43:44.566 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:43:44.567 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:43:44.567 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:43:44.567 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:44.567 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:43:44.567 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:43:44.567 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:44.567 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:43:44.567 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6812", 2021-02-10T11:43:44.567 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:43:44.567 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:44.567 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:44.567 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:43:44.567 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6814", 2021-02-10T11:43:44.568 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:43:44.568 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:43:44.568 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:43:44.568 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:44.568 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:43:44.568 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:43:44.568 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:44.568 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:43:44.568 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6808", 2021-02-10T11:43:44.568 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:43:44.568 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:44.568 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:44.568 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:43:44.569 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6810", 2021-02-10T11:43:44.569 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:43:44.569 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:43:44.569 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:43:44.569 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:44.569 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.112:6802/36029", 2021-02-10T11:43:44.569 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.112:6806/36029", 2021-02-10T11:43:44.569 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.112:6814/36029", 2021-02-10T11:43:44.569 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.112:6810/36029", 2021-02-10T11:43:44.569 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:43:44.569 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:43:44.569 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:43:44.569 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:43:44.570 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:44.570 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:44.570 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 2, 2021-02-10T11:43:44.570 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "026afa8a-c0f1-4b35-9a76-53c00fe0c2fc", 2021-02-10T11:43:44.570 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:43:44.570 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:43:44.570 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:43:44.570 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:43:44.570 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:43:44.570 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:43:44.570 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:43:44.570 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 123, 2021-02-10T11:43:44.570 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:43:44.571 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:43:44.571 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:43:44.571 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:43:44.571 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:44.571 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:43:44.571 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6801", 2021-02-10T11:43:44.571 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:43:44.571 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:44.571 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:44.571 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:43:44.571 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6803", 2021-02-10T11:43:44.571 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:43:44.571 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:43:44.572 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:43:44.572 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:44.572 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:43:44.572 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:43:44.572 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:44.572 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:43:44.572 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6805", 2021-02-10T11:43:44.572 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:43:44.572 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:44.572 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:44.572 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:43:44.572 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6807", 2021-02-10T11:43:44.572 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:43:44.573 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:43:44.573 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:43:44.573 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:44.573 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:43:44.573 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:43:44.573 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:44.573 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:43:44.573 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6813", 2021-02-10T11:43:44.573 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:43:44.573 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:44.573 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:44.573 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:43:44.573 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6815", 2021-02-10T11:43:44.574 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:43:44.574 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:43:44.574 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:43:44.574 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:44.574 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:43:44.574 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:43:44.574 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:44.574 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:43:44.574 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6809", 2021-02-10T11:43:44.574 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:43:44.574 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:44.574 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:44.575 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:43:44.575 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6811", 2021-02-10T11:43:44.575 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:43:44.575 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:43:44.575 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:43:44.575 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:44.575 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.112:6803/36031", 2021-02-10T11:43:44.575 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.112:6807/36031", 2021-02-10T11:43:44.575 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.112:6815/36031", 2021-02-10T11:43:44.575 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.112:6811/36031", 2021-02-10T11:43:44.575 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:43:44.575 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:43:44.575 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:43:44.576 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:43:44.576 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:44.576 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:44.576 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 3, 2021-02-10T11:43:44.576 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "58795682-e00d-4173-bb84-f7075ca89a11", 2021-02-10T11:43:44.576 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:43:44.576 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:43:44.576 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:43:44.576 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:43:44.576 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:43:44.576 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:43:44.576 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:43:44.576 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 115, 2021-02-10T11:43:44.576 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:43:44.577 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:43:44.577 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:43:44.577 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:43:44.577 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:44.577 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:43:44.577 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6816", 2021-02-10T11:43:44.577 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:43:44.577 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:44.577 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:44.577 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:43:44.577 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6819", 2021-02-10T11:43:44.577 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:43:44.578 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:43:44.578 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:43:44.578 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:44.578 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:43:44.578 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:43:44.578 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:44.578 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:43:44.578 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6821", 2021-02-10T11:43:44.578 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:43:44.578 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:44.578 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:44.578 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:43:44.579 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6823", 2021-02-10T11:43:44.579 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:43:44.579 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:43:44.579 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:43:44.638 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:44.639 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:43:44.639 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:43:44.639 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:44.639 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:43:44.639 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6829", 2021-02-10T11:43:44.639 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:43:44.639 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:44.639 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:44.639 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:43:44.639 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6831", 2021-02-10T11:43:44.639 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:43:44.640 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:43:44.640 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:43:44.640 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:44.640 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:43:44.640 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:43:44.640 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:44.640 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:43:44.640 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6825", 2021-02-10T11:43:44.640 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:43:44.640 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:44.640 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:44.640 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:43:44.640 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6827", 2021-02-10T11:43:44.641 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:43:44.641 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:43:44.641 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:43:44.641 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:44.641 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.112:6819/36028", 2021-02-10T11:43:44.641 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.112:6823/36028", 2021-02-10T11:43:44.641 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.112:6831/36028", 2021-02-10T11:43:44.641 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.112:6827/36028", 2021-02-10T11:43:44.641 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:43:44.641 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:43:44.641 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:43:44.641 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:43:44.641 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:44.642 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:44.642 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 4, 2021-02-10T11:43:44.642 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "6763f8d6-53dc-4e45-bdf6-a6eabcf81e39", 2021-02-10T11:43:44.642 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:43:44.642 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:43:44.642 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:43:44.642 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:43:44.642 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:43:44.642 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:43:44.642 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:43:44.642 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 132, 2021-02-10T11:43:44.642 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:43:44.643 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:43:44.643 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:43:44.643 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:43:44.643 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:44.643 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:43:44.643 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6810", 2021-02-10T11:43:44.643 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:43:44.643 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:44.643 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:44.643 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:43:44.643 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6813", 2021-02-10T11:43:44.643 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:43:44.643 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:43:44.644 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:43:44.644 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:44.644 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:43:44.644 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:43:44.644 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:44.644 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:43:44.644 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6816", 2021-02-10T11:43:44.644 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:43:44.644 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:44.644 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:44.644 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:43:44.644 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6819", 2021-02-10T11:43:44.645 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:43:44.645 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:43:44.645 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:43:44.645 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:44.645 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:43:44.645 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:43:44.645 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:44.645 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:43:44.645 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6825", 2021-02-10T11:43:44.645 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:43:44.645 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:44.645 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:44.645 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:43:44.646 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6827", 2021-02-10T11:43:44.646 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:43:44.646 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:43:44.646 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:43:44.646 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:44.646 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:43:44.646 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:43:44.646 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:44.646 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:43:44.646 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6821", 2021-02-10T11:43:44.646 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:43:44.646 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:44.646 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:44.647 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:43:44.647 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6823", 2021-02-10T11:43:44.647 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:43:44.647 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:43:44.647 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:43:44.647 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:44.647 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.120:6813/36388", 2021-02-10T11:43:44.647 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.120:6819/36388", 2021-02-10T11:43:44.647 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.120:6827/36388", 2021-02-10T11:43:44.647 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.120:6823/36388", 2021-02-10T11:43:44.647 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:43:44.647 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:43:44.647 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:43:44.648 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:43:44.648 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:44.648 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:44.648 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 5, 2021-02-10T11:43:44.648 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "5d0e4dd6-0cb0-4bf8-8314-626acaa1b544", 2021-02-10T11:43:44.648 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:43:44.648 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:43:44.648 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:43:44.648 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:43:44.648 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:43:44.648 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:43:44.648 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:43:44.648 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 130, 2021-02-10T11:43:44.649 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:43:44.649 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:43:44.649 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:43:44.649 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:43:44.649 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:44.649 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:43:44.649 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6804", 2021-02-10T11:43:44.649 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:43:44.649 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:44.649 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:44.649 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:43:44.649 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6806", 2021-02-10T11:43:44.649 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:43:44.649 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:43:44.650 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:43:44.650 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:44.650 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:43:44.650 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:43:44.650 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:44.650 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:43:44.650 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6809", 2021-02-10T11:43:44.650 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:43:44.650 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:44.650 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:44.650 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:43:44.650 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6812", 2021-02-10T11:43:44.650 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:43:44.651 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:43:44.651 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:43:44.651 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:44.651 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:43:44.651 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:43:44.651 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:44.651 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:43:44.651 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6820", 2021-02-10T11:43:44.651 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:43:44.651 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:44.651 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:44.651 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:43:44.651 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6822", 2021-02-10T11:43:44.652 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:43:44.652 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:43:44.652 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:43:44.652 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:44.652 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:43:44.652 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:43:44.652 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:44.652 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:43:44.652 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6815", 2021-02-10T11:43:44.652 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:43:44.652 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:44.652 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:44.652 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:43:44.653 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6818", 2021-02-10T11:43:44.653 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:43:44.653 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:43:44.653 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:43:44.653 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:44.653 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.120:6806/36387", 2021-02-10T11:43:44.653 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.120:6812/36387", 2021-02-10T11:43:44.653 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.120:6822/36387", 2021-02-10T11:43:44.653 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.120:6818/36387", 2021-02-10T11:43:44.653 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:43:44.653 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:43:44.653 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:43:44.653 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:43:44.653 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:44.654 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:44.654 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 6, 2021-02-10T11:43:44.654 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "b010e187-4541-4b00-80dd-f948f6ab8f15", 2021-02-10T11:43:44.654 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:43:44.654 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:43:44.654 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:43:44.654 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:43:44.654 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:43:44.654 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:43:44.654 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:43:44.654 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 132, 2021-02-10T11:43:44.654 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:43:44.654 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:43:44.655 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:43:44.655 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:43:44.655 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:44.655 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:43:44.655 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6824", 2021-02-10T11:43:44.655 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:43:44.655 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:44.655 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:44.655 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:43:44.655 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6826", 2021-02-10T11:43:44.655 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:43:44.655 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:43:44.655 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:43:44.656 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:44.656 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:43:44.656 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:43:44.656 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:44.656 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:43:44.656 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6828", 2021-02-10T11:43:44.656 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:43:44.656 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:44.656 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:44.656 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:43:44.656 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6829", 2021-02-10T11:43:44.656 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:43:44.656 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:43:44.657 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:43:44.657 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:44.657 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:43:44.657 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:43:44.657 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:44.657 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:43:44.657 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6832", 2021-02-10T11:43:44.657 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:43:44.657 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:44.657 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:44.657 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:43:44.657 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6833", 2021-02-10T11:43:44.657 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:43:44.658 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:43:44.658 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:43:44.658 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:44.658 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:43:44.658 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:43:44.658 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:44.658 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:43:44.658 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6830", 2021-02-10T11:43:44.658 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:43:44.658 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:44.658 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:44.658 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:43:44.658 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6831", 2021-02-10T11:43:44.659 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:43:44.659 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:43:44.659 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:43:44.659 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:44.659 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.120:6826/36385", 2021-02-10T11:43:44.659 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.120:6829/36385", 2021-02-10T11:43:44.659 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.120:6833/36385", 2021-02-10T11:43:44.659 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.120:6831/36385", 2021-02-10T11:43:44.659 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:43:44.659 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:43:44.659 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:43:44.659 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:43:44.659 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:44.660 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:44.660 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 7, 2021-02-10T11:43:44.660 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "671e2772-33b7-497e-8e44-2178bd850bcf", 2021-02-10T11:43:44.660 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:43:44.660 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:43:44.660 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:43:44.660 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:43:44.660 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:43:44.660 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:43:44.660 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:43:44.660 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 132, 2021-02-10T11:43:44.660 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:43:44.660 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:43:44.661 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:43:44.661 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:43:44.661 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:44.661 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:43:44.661 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6802", 2021-02-10T11:43:44.661 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:43:44.661 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:44.661 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:44.661 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:43:44.661 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6803", 2021-02-10T11:43:44.661 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:43:44.661 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:43:44.661 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:43:44.662 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:44.662 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:43:44.662 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:43:44.662 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:44.662 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:43:44.662 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6805", 2021-02-10T11:43:44.662 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:43:44.662 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:44.662 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:44.662 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:43:44.662 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6807", 2021-02-10T11:43:44.662 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:43:44.663 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:43:44.663 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:43:44.663 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:44.663 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:43:44.663 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:43:44.663 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:44.663 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:43:44.663 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6814", 2021-02-10T11:43:44.663 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:43:44.663 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:44.663 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:44.663 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:43:44.663 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6817", 2021-02-10T11:43:44.663 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:43:44.664 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:43:44.664 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:43:44.664 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:44.664 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:43:44.664 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:43:44.664 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:44.664 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:43:44.664 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6808", 2021-02-10T11:43:44.664 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:43:44.664 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:44.664 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:44.664 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:43:44.664 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6811", 2021-02-10T11:43:44.665 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:43:44.665 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:43:44.665 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:43:44.665 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:44.665 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.120:6803/36386", 2021-02-10T11:43:44.665 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.120:6807/36386", 2021-02-10T11:43:44.665 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.120:6817/36386", 2021-02-10T11:43:44.665 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.120:6811/36386", 2021-02-10T11:43:44.665 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:43:44.665 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:43:44.665 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:43:44.665 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:43:44.665 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:43:44.666 INFO:teuthology.orchestra.run.gibba012.stdout: ], 2021-02-10T11:43:44.666 INFO:teuthology.orchestra.run.gibba012.stdout: "osd_xinfo": [ 2021-02-10T11:43:44.666 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:44.666 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 0, 2021-02-10T11:43:44.666 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:43:44.666 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:43:44.666 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:43:44.666 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:43:44.666 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:43:44.666 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:44.666 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:44.666 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 1, 2021-02-10T11:43:44.666 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:43:44.667 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:43:44.667 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:43:44.667 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:43:44.667 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:43:44.667 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:44.667 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:44.667 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 2, 2021-02-10T11:43:44.667 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:43:44.667 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:43:44.667 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:43:44.667 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:43:44.667 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:43:44.667 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:44.667 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:44.668 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 3, 2021-02-10T11:43:44.668 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:43:44.668 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:43:44.668 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:43:44.668 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:43:44.668 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:43:44.668 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:44.668 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:44.668 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 4, 2021-02-10T11:43:44.668 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:43:44.668 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:43:44.668 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:43:44.668 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:43:44.669 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:43:44.669 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:44.669 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:44.669 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 5, 2021-02-10T11:43:44.669 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:43:44.669 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:43:44.669 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:43:44.669 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:43:44.669 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:43:44.669 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:44.669 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:44.669 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 6, 2021-02-10T11:43:44.669 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:43:44.670 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:43:44.670 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:43:44.670 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:43:44.670 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:43:44.670 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:44.670 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:44.670 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 7, 2021-02-10T11:43:44.670 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:43:44.670 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:43:44.670 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:43:44.670 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:43:44.670 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:43:44.670 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:43:44.671 INFO:teuthology.orchestra.run.gibba012.stdout: ], 2021-02-10T11:43:44.671 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_upmap": [], 2021-02-10T11:43:44.671 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_upmap_items": [], 2021-02-10T11:43:44.671 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_temp": [], 2021-02-10T11:43:44.671 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_temp": [], 2021-02-10T11:43:44.671 INFO:teuthology.orchestra.run.gibba012.stdout: "blacklist": {}, 2021-02-10T11:43:44.671 INFO:teuthology.orchestra.run.gibba012.stdout: "erasure_code_profiles": { 2021-02-10T11:43:44.671 INFO:teuthology.orchestra.run.gibba012.stdout: "default": { 2021-02-10T11:43:44.671 INFO:teuthology.orchestra.run.gibba012.stdout: "crush-failure-domain": "osd", 2021-02-10T11:43:44.671 INFO:teuthology.orchestra.run.gibba012.stdout: "k": "2", 2021-02-10T11:43:44.671 INFO:teuthology.orchestra.run.gibba012.stdout: "m": "1", 2021-02-10T11:43:44.671 INFO:teuthology.orchestra.run.gibba012.stdout: "plugin": "jerasure", 2021-02-10T11:43:44.671 INFO:teuthology.orchestra.run.gibba012.stdout: "ruleset-failure-domain": "osd", 2021-02-10T11:43:44.671 INFO:teuthology.orchestra.run.gibba012.stdout: "technique": "reed_sol_van" 2021-02-10T11:43:44.672 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:43:44.672 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:44.672 INFO:teuthology.orchestra.run.gibba012.stdout: "removed_snaps_queue": [], 2021-02-10T11:43:44.672 INFO:teuthology.orchestra.run.gibba012.stdout: "new_removed_snaps": [], 2021-02-10T11:43:44.672 INFO:teuthology.orchestra.run.gibba012.stdout: "new_purged_snaps": [], 2021-02-10T11:43:44.672 INFO:teuthology.orchestra.run.gibba012.stdout: "crush_node_flags": {}, 2021-02-10T11:43:44.672 INFO:teuthology.orchestra.run.gibba012.stdout: "device_class_flags": {} 2021-02-10T11:43:44.672 INFO:teuthology.orchestra.run.gibba012.stdout:} 2021-02-10T11:43:44.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:44.538 7f81fa43f700 1 -- 172.21.2.112:0/1740257479 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f81d40186c0 msgr2=0x7f81d401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:44.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:44.538 7f81fa43f700 1 --2- 172.21.2.112:0/1740257479 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f81d40186c0 0x7f81d401abc0 crc :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:44.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:44.538 7f81fa43f700 1 -- 172.21.2.112:0/1740257479 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f81f408a450 msgr2=0x7f81f408a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:44.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:44.538 7f81fa43f700 1 --2- 172.21.2.112:0/1740257479 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f81f408a450 0x7f81f408a850 secure :-1 s=READY pgs=2498 cs=0 l=1 rev1=1 rx=0x7f81ec00a900 tx=0x7f81ec00aca0).stop 2021-02-10T11:43:44.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:44.538 7f81fa43f700 1 -- 172.21.2.112:0/1740257479 shutdown_connections 2021-02-10T11:43:44.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:44.538 7f81fa43f700 1 --2- 172.21.2.112:0/1740257479 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f81d40186c0 0x7f81d401abc0 unknown :-1 s=CLOSED pgs=1143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:44.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:44.538 7f81fa43f700 1 --2- 172.21.2.112:0/1740257479 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f81f408a450 0x7f81f408a850 unknown :-1 s=CLOSED pgs=2498 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:44.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:44.538 7f81fa43f700 1 --2- 172.21.2.112:0/1740257479 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f81f4103920 0x7f81f40b32d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:44.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:44.538 7f81fa43f700 1 --2- 172.21.2.112:0/1740257479 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f81f4103520 0x7f81f40c3640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:44.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:44.538 7f81fa43f700 1 -- 172.21.2.112:0/1740257479 >> 172.21.2.112:0/1740257479 conn(0x7f81f410a820 msgr2=0x7f81f4108150 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:44.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:44.539 7f81fa43f700 1 -- 172.21.2.112:0/1740257479 shutdown_connections 2021-02-10T11:43:44.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:44.539 7f81fa43f700 1 -- 172.21.2.112:0/1740257479 wait complete. 2021-02-10T11:43:44.675 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:43:45.127 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.109 7feeef08a700 1 Processor -- start 2021-02-10T11:43:45.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.109 7feeef08a700 1 -- start start 2021-02-10T11:43:45.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.109 7feeef08a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7feee80d1870 0x7feee81b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:45.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.109 7feeef08a700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7feee81b0a10 0x7feee80cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:45.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.109 7feeef08a700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feee80f0b10 con 0x7feee80d1c70 2021-02-10T11:43:45.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.109 7feeef08a700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feee80fe230 con 0x7feee81b0a10 2021-02-10T11:43:45.132 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.109 7feeef08a700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feee80e5ed0 con 0x7feee80d1870 2021-02-10T11:43:45.132 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.109 7feeed6a7700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7feee80d1c70 0x7feee80cf950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:56724/0 (socket says 172.21.2.112:56724) 2021-02-10T11:43:45.133 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.109 7feeed6a7700 1 -- 172.21.2.112:0/3752302108 learned_addr learned my addr 172.21.2.112:0/3752302108 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:43:45.133 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.109 7feeecea6700 1 --2- 172.21.2.112:0/3752302108 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7feee80d1870 0x7feee81b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:45.133 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.109 7feee7fff700 1 --2- 172.21.2.112:0/3752302108 >> v2:172.21.2.120:3300/0 conn(0x7feee81b0a10 0x7feee80cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:45.133 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.110 7feee77fe700 1 -- 172.21.2.112:0/3752302108 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2008832862 0 0) 0x7feee0001500 con 0x7feee80d1c70 2021-02-10T11:43:45.133 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.110 7feeecea6700 1 -- 172.21.2.112:0/3752302108 >> v2:172.21.2.120:3300/0 conn(0x7feee81b0a10 msgr2=0x7feee80cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:45.133 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.110 7feeecea6700 1 --2- 172.21.2.112:0/3752302108 >> v2:172.21.2.120:3300/0 conn(0x7feee81b0a10 0x7feee80cf330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:45.133 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.110 7feeecea6700 1 -- 172.21.2.112:0/3752302108 >> v1:172.21.2.112:6789/0 conn(0x7feee80d1c70 legacy=0x7feee80cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:45.133 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.110 7feeecea6700 1 -- 172.21.2.112:0/3752302108 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feee80f0650 con 0x7feee80d1870 2021-02-10T11:43:45.134 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.110 7feeecea6700 1 --2- 172.21.2.112:0/3752302108 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7feee80d1870 0x7feee81b03f0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7feed0006650 tx=0x7feed0009a80).ready entity=mon.2 client_cookie=d091ebc4f5d75c60 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:45.134 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.110 7feee77fe700 1 -- 172.21.2.112:0/3752302108 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feed000c0c0 con 0x7feee80d1870 2021-02-10T11:43:45.134 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.110 7feee77fe700 1 -- 172.21.2.112:0/3752302108 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feed000c340 con 0x7feee80d1870 2021-02-10T11:43:45.134 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.110 7feeef08a700 1 -- 172.21.2.112:0/3752302108 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7feee80d1870 msgr2=0x7feee81b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:45.134 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.110 7feeef08a700 1 --2- 172.21.2.112:0/3752302108 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7feee80d1870 0x7feee81b03f0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7feed0006650 tx=0x7feed0009a80).stop 2021-02-10T11:43:45.134 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.110 7feee77fe700 1 -- 172.21.2.112:0/3752302108 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feed00009b0 con 0x7feee80d1870 2021-02-10T11:43:45.135 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.111 7feeef08a700 1 -- 172.21.2.112:0/3752302108 shutdown_connections 2021-02-10T11:43:45.135 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.111 7feeef08a700 1 --2- 172.21.2.112:0/3752302108 >> v2:172.21.2.120:3300/0 conn(0x7feee81b0a10 0x7feee80cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:45.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.111 7feeef08a700 1 --2- 172.21.2.112:0/3752302108 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7feee80d1870 0x7feee81b03f0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:45.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.111 7feeef08a700 1 -- 172.21.2.112:0/3752302108 >> 172.21.2.112:0/3752302108 conn(0x7feee80ce450 msgr2=0x7feee80d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:45.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.111 7feeef08a700 1 -- 172.21.2.112:0/3752302108 shutdown_connections 2021-02-10T11:43:45.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.111 7feeef08a700 1 -- 172.21.2.112:0/3752302108 wait complete. 2021-02-10T11:43:45.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.126 7feeef08a700 1 Processor -- start 2021-02-10T11:43:45.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.126 7feeef08a700 1 -- start start 2021-02-10T11:43:45.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.126 7feeef08a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7feee808a740 0x7feee808ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:45.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.127 7feeef08a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7feee808b160 0x7feee8194990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:45.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.127 7feeef08a700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7feee80913c0 0x7feee8194fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:45.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.127 7feeef08a700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feee81ac380 con 0x7feee808b160 2021-02-10T11:43:45.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.127 7feeef08a700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feee8091970 con 0x7feee80913c0 2021-02-10T11:43:45.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.127 7feeef08a700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feee81abe70 con 0x7feee808a740 2021-02-10T11:43:45.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.127 7feeecea6700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7feee808a740 0x7feee808ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:45.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.127 7feeecea6700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7feee808a740 0x7feee808ab40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3301/0 says I am v2:172.21.2.112:56946/0 (socket says 172.21.2.112:56946) 2021-02-10T11:43:45.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.127 7feee7fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7feee808b160 0x7feee8194990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:45.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.127 7feeecea6700 1 -- 172.21.2.112:0/2039002564 learned_addr learned my addr 172.21.2.112:0/2039002564 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:43:45.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.127 7feeed6a7700 1 --2- 172.21.2.112:0/2039002564 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7feee80913c0 0x7feee8194fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:45.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.127 7feeecea6700 1 -- 172.21.2.112:0/2039002564 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7feee80913c0 msgr2=0x7feee8194fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:45.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.127 7feeecea6700 1 --2- 172.21.2.112:0/2039002564 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7feee80913c0 0x7feee8194fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:45.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.127 7feeecea6700 1 -- 172.21.2.112:0/2039002564 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7feee808b160 msgr2=0x7feee8194990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:45.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.127 7feeecea6700 1 --2- 172.21.2.112:0/2039002564 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7feee808b160 0x7feee8194990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:45.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.127 7feeecea6700 1 -- 172.21.2.112:0/2039002564 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feee81955d0 con 0x7feee808a740 2021-02-10T11:43:45.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.127 7feeecea6700 1 --2- 172.21.2.112:0/2039002564 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7feee808a740 0x7feee808ab40 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7feed000b660 tx=0x7feed000ba00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:45.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.127 7feee57fa700 1 -- 172.21.2.112:0/2039002564 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feed000bdf0 con 0x7feee808a740 2021-02-10T11:43:45.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.127 7feee57fa700 1 -- 172.21.2.112:0/2039002564 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feed00009b0 con 0x7feee808a740 2021-02-10T11:43:45.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.127 7feeef08a700 1 -- 172.21.2.112:0/2039002564 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feee81955d0 con 0x7feee808a740 2021-02-10T11:43:45.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.129 7feee57fa700 1 -- 172.21.2.112:0/2039002564 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feed000bdf0 con 0x7feee808a740 2021-02-10T11:43:45.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.129 7feeef08a700 1 -- 172.21.2.112:0/2039002564 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feee81955d0 con 0x7feee808a740 2021-02-10T11:43:45.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.129 7feee57fa700 1 -- 172.21.2.112:0/2039002564 <== mon.2 v2:172.21.2.120:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7feed00052e0 con 0x7feee808a740 2021-02-10T11:43:45.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.129 7feee57fa700 1 --2- 172.21.2.112:0/2039002564 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7feecc018730 0x7feecc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:45.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.129 7feee57fa700 1 -- 172.21.2.112:0/2039002564 <== mon.2 v2:172.21.2.120:3301/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5573+0+0 (secure 0 0 0) 0x7feed0032630 con 0x7feee808a740 2021-02-10T11:43:45.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.129 7feee7fff700 1 --2- 172.21.2.112:0/2039002564 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7feecc018730 0x7feecc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:45.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.129 7feee7fff700 1 --2- 172.21.2.112:0/2039002564 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7feecc018730 0x7feecc01abc0 crc :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:45.160 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.159 7feeef08a700 1 -- 172.21.2.112:0/2039002564 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feee80ebc00 con 0x7feee808a740 2021-02-10T11:43:45.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.162 7feee57fa700 1 -- 172.21.2.112:0/2039002564 <== mon.2 v2:172.21.2.120:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7feed0000940 con 0x7feee808a740 2021-02-10T11:43:45.296 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.296 7feeef08a700 1 -- 172.21.2.112:0/2039002564 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feee80f0b10 con 0x7feee808a740 2021-02-10T11:43:45.297 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.297 7feee57fa700 1 -- 172.21.2.112:0/2039002564 <== mon.2 v2:172.21.2.120:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 115 v115) v1 ==== 95+0+2108 (secure 0 0 0) 0x7feed000bdf0 con 0x7feee808a740 2021-02-10T11:43:45.297 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:43:45.298 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":115,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11349,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/2093257968","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":2093257968},{"type":"v1","addr":"172.21.2.112:6833","nonce":2093257968}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":110}],"filesystems":[{"mdsmap":{"epoch":114,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:43:08.182625","modified":"2021-02-10 11:43:10.189979","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7025},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7025":{"gid":7025,"name":"b","rank":0,"incarnation":111,"state":"up:active","state_seq":3,"addr":"172.21.2.120:6835/2552826693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":2552826693},{"type":"v1","addr":"172.21.2.120:6835","nonce":2552826693}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":14}]} 2021-02-10T11:43:45.298 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 115 2021-02-10T11:43:45.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.298 7feeef08a700 1 -- 172.21.2.112:0/2039002564 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7feecc018730 msgr2=0x7feecc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:45.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.298 7feeef08a700 1 --2- 172.21.2.112:0/2039002564 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7feecc018730 0x7feecc01abc0 crc :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:45.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.298 7feeef08a700 1 -- 172.21.2.112:0/2039002564 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7feee808a740 msgr2=0x7feee808ab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:45.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.298 7feeef08a700 1 --2- 172.21.2.112:0/2039002564 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7feee808a740 0x7feee808ab40 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7feed000b660 tx=0x7feed000ba00).stop 2021-02-10T11:43:45.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.299 7feeef08a700 1 -- 172.21.2.112:0/2039002564 shutdown_connections 2021-02-10T11:43:45.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.299 7feeef08a700 1 --2- 172.21.2.112:0/2039002564 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7feecc018730 0x7feecc01abc0 unknown :-1 s=CLOSED pgs=1144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:45.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.299 7feeef08a700 1 --2- 172.21.2.112:0/2039002564 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7feee80913c0 0x7feee8194fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:45.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.299 7feeef08a700 1 --2- 172.21.2.112:0/2039002564 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7feee808b160 0x7feee8194990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:45.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.299 7feeef08a700 1 --2- 172.21.2.112:0/2039002564 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7feee808a740 0x7feee808ab40 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:45.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.299 7feeef08a700 1 -- 172.21.2.112:0/2039002564 >> 172.21.2.112:0/2039002564 conn(0x7feee819c560 msgr2=0x7feee80ebe30 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:45.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.299 7feeef08a700 1 -- 172.21.2.112:0/2039002564 shutdown_connections 2021-02-10T11:43:45.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.299 7feeef08a700 1 -- 172.21.2.112:0/2039002564 wait complete. 2021-02-10T11:43:45.308 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs fail cephfs 2021-02-10T11:43:45.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.721 7fbf5a07c700 1 Processor -- start 2021-02-10T11:43:45.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.721 7fbf5a07c700 1 -- start start 2021-02-10T11:43:45.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.722 7fbf5a07c700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fbf540d1870 0x7fbf541b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:45.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.722 7fbf5a07c700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf540f0b10 con 0x7fbf541b0a10 2021-02-10T11:43:45.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.722 7fbf5a07c700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf540d0310 con 0x7fbf540f0650 2021-02-10T11:43:45.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.722 7fbf5a07c700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf540d0530 con 0x7fbf540d1870 2021-02-10T11:43:45.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.722 7fbf52ffd700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fbf541b0a10 0x7fbf540cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:56734/0 (socket says 172.21.2.112:56734) 2021-02-10T11:43:45.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.722 7fbf52ffd700 1 -- 172.21.2.112:0/1693843923 learned_addr learned my addr 172.21.2.112:0/1693843923 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:43:45.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.722 7fbf537fe700 1 --2- 172.21.2.112:0/1693843923 >> v2:172.21.2.120:3300/0 conn(0x7fbf540d1870 0x7fbf541b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:45.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.722 7fbf527fc700 1 -- 172.21.2.112:0/1693843923 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1170655749 0 0) 0x7fbf4c000f90 con 0x7fbf540f0650 2021-02-10T11:43:45.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.722 7fbf527fc700 1 -- 172.21.2.112:0/1693843923 --> v1:172.21.2.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf40002fe0 con 0x7fbf540f0650 2021-02-10T11:43:45.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.722 7fbf527fc700 1 -- 172.21.2.112:0/1693843923 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3190003285 0 0) 0x7fbf440010b0 con 0x7fbf541b0a10 2021-02-10T11:43:45.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.722 7fbf527fc700 1 -- 172.21.2.112:0/1693843923 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf400042c0 con 0x7fbf541b0a10 2021-02-10T11:43:45.740 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.723 7fbf537fe700 1 -- 172.21.2.112:0/1693843923 >> v1:172.21.2.120:6789/0 conn(0x7fbf540f0650 legacy=0x7fbf540cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:45.740 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.723 7fbf537fe700 1 -- 172.21.2.112:0/1693843923 >> v1:172.21.2.112:6789/0 conn(0x7fbf541b0a10 legacy=0x7fbf540cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:45.740 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.723 7fbf537fe700 1 -- 172.21.2.112:0/1693843923 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf540d1c70 con 0x7fbf540d1870 2021-02-10T11:43:45.740 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.723 7fbf537fe700 1 --2- 172.21.2.112:0/1693843923 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fbf540d1870 0x7fbf541b03f0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 rx=0x7fbf3c006650 tx=0x7fbf3c009a80).ready entity=mon.1 client_cookie=8354207502ef9313 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:45.740 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.723 7fbf527fc700 1 -- 172.21.2.112:0/1693843923 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbf3c00c0c0 con 0x7fbf540d1870 2021-02-10T11:43:45.740 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.723 7fbf527fc700 1 -- 172.21.2.112:0/1693843923 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf3c00c340 con 0x7fbf540d1870 2021-02-10T11:43:45.740 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.723 7fbf527fc700 1 -- 172.21.2.112:0/1693843923 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbf3c00c860 con 0x7fbf540d1870 2021-02-10T11:43:45.740 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.723 7fbf5a07c700 1 -- 172.21.2.112:0/1693843923 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fbf540d1870 msgr2=0x7fbf541b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:45.740 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.723 7fbf5a07c700 1 --2- 172.21.2.112:0/1693843923 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fbf540d1870 0x7fbf541b03f0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 rx=0x7fbf3c006650 tx=0x7fbf3c009a80).stop 2021-02-10T11:43:45.740 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.724 7fbf5a07c700 1 -- 172.21.2.112:0/1693843923 shutdown_connections 2021-02-10T11:43:45.740 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.724 7fbf5a07c700 1 --2- 172.21.2.112:0/1693843923 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fbf540d1870 0x7fbf541b03f0 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:45.740 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.724 7fbf5a07c700 1 -- 172.21.2.112:0/1693843923 >> 172.21.2.112:0/1693843923 conn(0x7fbf540ce450 msgr2=0x7fbf540d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:45.740 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.724 7fbf5a07c700 1 -- 172.21.2.112:0/1693843923 shutdown_connections 2021-02-10T11:43:45.741 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.724 7fbf5a07c700 1 -- 172.21.2.112:0/1693843923 wait complete. 2021-02-10T11:43:45.743 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.739 7fbf5a07c700 1 Processor -- start 2021-02-10T11:43:45.743 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.739 7fbf5a07c700 1 -- start start 2021-02-10T11:43:45.743 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.739 7fbf5a07c700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbf541a06a0 0x7fbf5409f380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:45.743 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.739 7fbf5a07c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbf5409f9a0 0x7fbf5408b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:45.743 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.739 7fbf5a07c700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fbf5408bb80 0x7fbf5408bf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:45.743 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.739 7fbf5a07c700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf5408c780 con 0x7fbf5409f9a0 2021-02-10T11:43:45.743 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.739 7fbf5a07c700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf5409fda0 con 0x7fbf5408bb80 2021-02-10T11:43:45.743 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.739 7fbf5a07c700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf5408cd00 con 0x7fbf541a06a0 2021-02-10T11:43:45.743 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.739 7fbf52ffd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbf5409f9a0 0x7fbf5408b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:45.743 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.739 7fbf52ffd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbf5409f9a0 0x7fbf5408b560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:57274/0 (socket says 172.21.2.112:57274) 2021-02-10T11:43:45.743 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.739 7fbf52ffd700 1 -- 172.21.2.112:0/1305754982 learned_addr learned my addr 172.21.2.112:0/1305754982 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:43:45.744 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.740 7fbf537fe700 1 --2- 172.21.2.112:0/1305754982 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbf541a06a0 0x7fbf5409f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:45.744 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.740 7fbf53fff700 1 --2- 172.21.2.112:0/1305754982 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fbf5408bb80 0x7fbf5408bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:45.744 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.740 7fbf52ffd700 1 -- 172.21.2.112:0/1305754982 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbf541a06a0 msgr2=0x7fbf5409f380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:45.744 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.740 7fbf52ffd700 1 --2- 172.21.2.112:0/1305754982 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbf541a06a0 0x7fbf5409f380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:45.744 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.740 7fbf52ffd700 1 -- 172.21.2.112:0/1305754982 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fbf5408bb80 msgr2=0x7fbf5408bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:45.744 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.740 7fbf52ffd700 1 --2- 172.21.2.112:0/1305754982 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fbf5408bb80 0x7fbf5408bf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:45.744 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.740 7fbf52ffd700 1 -- 172.21.2.112:0/1305754982 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf5408cf20 con 0x7fbf5409f9a0 2021-02-10T11:43:45.744 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.740 7fbf53fff700 1 --2- 172.21.2.112:0/1305754982 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fbf5408bb80 0x7fbf5408bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:43:45.744 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.740 7fbf52ffd700 1 --2- 172.21.2.112:0/1305754982 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbf5409f9a0 0x7fbf5408b560 secure :-1 s=READY pgs=2501 cs=0 l=1 rev1=1 rx=0x7fbf4400a4c0 tx=0x7fbf4400a860).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:45.744 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.740 7fbf4bfff700 1 -- 172.21.2.112:0/1305754982 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbf44000e50 con 0x7fbf5409f9a0 2021-02-10T11:43:45.744 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.740 7fbf4bfff700 1 -- 172.21.2.112:0/1305754982 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf4400ac50 con 0x7fbf5409f9a0 2021-02-10T11:43:45.744 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.740 7fbf4bfff700 1 -- 172.21.2.112:0/1305754982 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbf440045d0 con 0x7fbf5409f9a0 2021-02-10T11:43:45.744 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.740 7fbf5a07c700 1 -- 172.21.2.112:0/1305754982 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf5408cf20 con 0x7fbf5409f9a0 2021-02-10T11:43:45.744 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.740 7fbf5a07c700 1 -- 172.21.2.112:0/1305754982 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf5408cf20 con 0x7fbf5409f9a0 2021-02-10T11:43:45.745 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.741 7fbf4bfff700 1 -- 172.21.2.112:0/1305754982 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fbf44004350 con 0x7fbf5409f9a0 2021-02-10T11:43:45.745 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.741 7fbf4bfff700 1 --2- 172.21.2.112:0/1305754982 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fbf38018730 0x7fbf3801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:45.745 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.741 7fbf4bfff700 1 -- 172.21.2.112:0/1305754982 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fbf44032790 con 0x7fbf5409f9a0 2021-02-10T11:43:45.745 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.741 7fbf537fe700 1 --2- 172.21.2.112:0/1305754982 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fbf38018730 0x7fbf3801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:45.745 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.742 7fbf537fe700 1 --2- 172.21.2.112:0/1305754982 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fbf38018730 0x7fbf3801ac10 crc :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:45.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.772 7fbf5a07c700 1 -- 172.21.2.112:0/1305754982 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf540ebc50 con 0x7fbf5409f9a0 2021-02-10T11:43:45.774 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.774 7fbf4bfff700 1 -- 172.21.2.112:0/1305754982 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fbf44004140 con 0x7fbf5409f9a0 2021-02-10T11:43:45.901 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:45.901 7fbf5a07c700 1 -- 172.21.2.112:0/1305754982 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fbf540f0b10 con 0x7fbf5409f9a0 2021-02-10T11:43:46.229 INFO:tasks.ceph.mon.a.gibba012.stderr:2021-02-10 11:43:46.228 7fc3fe6e2700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-02-10T11:43:46.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:46.234 7fbf4bfff700 1 -- 172.21.2.112:0/1305754982 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v116) v1 ==== 161+0+0 (secure 0 0 0) 0x7fbf44004350 con 0x7fbf5409f9a0 2021-02-10T11:43:46.266 INFO:teuthology.orchestra.run.gibba012.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2021-02-10T11:43:46.268 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:46.267 7fbf5a07c700 1 -- 172.21.2.112:0/1305754982 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fbf38018730 msgr2=0x7fbf3801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:46.268 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:46.267 7fbf5a07c700 1 --2- 172.21.2.112:0/1305754982 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fbf38018730 0x7fbf3801ac10 crc :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:46.268 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:46.267 7fbf5a07c700 1 -- 172.21.2.112:0/1305754982 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbf5409f9a0 msgr2=0x7fbf5408b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:46.268 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:46.267 7fbf5a07c700 1 --2- 172.21.2.112:0/1305754982 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbf5409f9a0 0x7fbf5408b560 secure :-1 s=READY pgs=2501 cs=0 l=1 rev1=1 rx=0x7fbf4400a4c0 tx=0x7fbf4400a860).stop 2021-02-10T11:43:46.268 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:46.268 7fbf5a07c700 1 -- 172.21.2.112:0/1305754982 shutdown_connections 2021-02-10T11:43:46.268 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:46.268 7fbf5a07c700 1 --2- 172.21.2.112:0/1305754982 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fbf38018730 0x7fbf3801ac10 unknown :-1 s=CLOSED pgs=1145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:46.268 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:46.268 7fbf5a07c700 1 --2- 172.21.2.112:0/1305754982 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fbf5408bb80 0x7fbf5408bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:46.268 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:46.268 7fbf5a07c700 1 --2- 172.21.2.112:0/1305754982 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbf5409f9a0 0x7fbf5408b560 unknown :-1 s=CLOSED pgs=2501 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:46.268 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:46.268 7fbf5a07c700 1 --2- 172.21.2.112:0/1305754982 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbf541a06a0 0x7fbf5409f380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:46.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:46.268 7fbf5a07c700 1 -- 172.21.2.112:0/1305754982 >> 172.21.2.112:0/1305754982 conn(0x7fbf540b83b0 msgr2=0x7fbf540a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:46.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:46.268 7fbf5a07c700 1 -- 172.21.2.112:0/1305754982 shutdown_connections 2021-02-10T11:43:46.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:46.268 7fbf5a07c700 1 -- 172.21.2.112:0/1305754982 wait complete. 2021-02-10T11:43:46.276 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:43:46.278 INFO:tasks.ceph.mds.b.gibba020.stdout:starting mds.b at 2021-02-10T11:43:46.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:46.688 7ff3d6559700 1 Processor -- start 2021-02-10T11:43:46.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:46.688 7ff3d6559700 1 -- start start 2021-02-10T11:43:46.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:46.688 7ff3d6559700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff3d00d1870 0x7ff3d01b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:46.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:46.688 7ff3d6559700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3d00f0b10 con 0x7ff3d01b0a10 2021-02-10T11:43:46.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:46.688 7ff3d6559700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3d00d0310 con 0x7ff3d00f0650 2021-02-10T11:43:46.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:46.688 7ff3d6559700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3d00e5ed0 con 0x7ff3d00d1870 2021-02-10T11:43:46.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:46.689 7ff3c7fff700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7ff3d01b0a10 0x7ff3d00cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:56744/0 (socket says 172.21.2.112:56744) 2021-02-10T11:43:46.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:46.689 7ff3c7fff700 1 -- 172.21.2.112:0/516259565 learned_addr learned my addr 172.21.2.112:0/516259565 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:43:46.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:46.689 7ff3cffff700 1 --2- 172.21.2.112:0/516259565 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff3d00d1870 0x7ff3d01b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:46.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:46.689 7ff3cf7fe700 1 -- 172.21.2.112:0/516259565 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2229034848 0 0) 0x7ff3c0000ce0 con 0x7ff3d01b0a10 2021-02-10T11:43:46.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:46.689 7ff3cf7fe700 1 -- 172.21.2.112:0/516259565 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3bc002fe0 con 0x7ff3d01b0a10 2021-02-10T11:43:46.707 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:46.689 7ff3cf7fe700 1 -- 172.21.2.112:0/516259565 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1557728616 0 0) 0x7ff3c00010e0 con 0x7ff3d01b0a10 2021-02-10T11:43:46.707 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:46.689 7ff3cf7fe700 1 -- 172.21.2.112:0/516259565 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff3d00d1870 msgr2=0x7ff3d01b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:46.707 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:46.689 7ff3cf7fe700 1 --2- 172.21.2.112:0/516259565 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff3d00d1870 0x7ff3d01b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:46.707 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:46.689 7ff3cf7fe700 1 -- 172.21.2.112:0/516259565 >> v1:172.21.2.120:6789/0 conn(0x7ff3d00f0650 legacy=0x7ff3d00cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:46.707 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:46.689 7ff3cf7fe700 1 -- 172.21.2.112:0/516259565 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3d00d0530 con 0x7ff3d01b0a10 2021-02-10T11:43:46.707 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:46.689 7ff3cf7fe700 1 -- 172.21.2.112:0/516259565 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7ff3c0001660 con 0x7ff3d01b0a10 2021-02-10T11:43:46.707 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:46.689 7ff3cf7fe700 1 -- 172.21.2.112:0/516259565 >> v1:172.21.2.112:6789/0 conn(0x7ff3d01b0a10 legacy=0x7ff3d00cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:46.707 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:46.689 7ff3cf7fe700 1 --2- 172.21.2.112:0/516259565 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff3bc003ac0 0x7ff3bc005f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:46.707 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:46.689 7ff3cf7fe700 1 --2- 172.21.2.112:0/516259565 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff3bc006780 0x7ff3bc008ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:46.707 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:46.689 7ff3cf7fe700 1 --2- 172.21.2.112:0/516259565 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff3bc009580 0x7ff3bc00ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:46.707 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:46.689 7ff3cf7fe700 1 -- 172.21.2.112:0/516259565 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3bc00c2c0 con 0x7ff3bc006780 2021-02-10T11:43:46.707 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:46.689 7ff3cf7fe700 1 -- 172.21.2.112:0/516259565 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3bc00c4e0 con 0x7ff3bc003ac0 2021-02-10T11:43:46.707 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:46.689 7ff3cf7fe700 1 -- 172.21.2.112:0/516259565 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3bc00c700 con 0x7ff3bc009580 2021-02-10T11:43:46.707 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:46.689 7ff3c7fff700 1 --2- 172.21.2.112:0/516259565 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff3bc006780 0x7ff3bc008ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:46.708 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:46.690 7ff3cffff700 1 --2- 172.21.2.112:0/516259565 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff3bc003ac0 0x7ff3bc005f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:46.708 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:46.690 7ff3c7fff700 1 -- 172.21.2.112:0/516259565 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff3bc009580 msgr2=0x7ff3bc00ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:46.708 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:46.690 7ff3d4b76700 1 --2- 172.21.2.112:0/516259565 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff3bc009580 0x7ff3bc00ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:46.708 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:46.690 7ff3c7fff700 1 --2- 172.21.2.112:0/516259565 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff3bc009580 0x7ff3bc00ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:46.708 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:46.690 7ff3c7fff700 1 -- 172.21.2.112:0/516259565 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff3bc003ac0 msgr2=0x7ff3bc005f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:46.708 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:46.690 7ff3c7fff700 1 --2- 172.21.2.112:0/516259565 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff3bc003ac0 0x7ff3bc005f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:46.708 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:46.690 7ff3c7fff700 1 -- 172.21.2.112:0/516259565 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3bc050970 con 0x7ff3bc006780 2021-02-10T11:43:46.708 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:46.690 7ff3cffff700 1 --2- 172.21.2.112:0/516259565 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff3bc003ac0 0x7ff3bc005f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:43:46.708 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:46.690 7ff3d4b76700 1 --2- 172.21.2.112:0/516259565 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff3bc009580 0x7ff3bc00ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:43:46.708 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:46.690 7ff3c7fff700 1 --2- 172.21.2.112:0/516259565 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff3bc006780 0x7ff3bc008ca0 secure :-1 s=READY pgs=2504 cs=0 l=1 rev1=1 rx=0x7ff3c0001280 tx=0x7ff3c0007d00).ready entity=mon.0 client_cookie=9c6869392409431a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:46.708 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:46.690 7ff3cf7fe700 1 -- 172.21.2.112:0/516259565 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff3c00014a0 con 0x7ff3bc006780 2021-02-10T11:43:46.708 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:46.690 7ff3cf7fe700 1 -- 172.21.2.112:0/516259565 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3c00070c0 con 0x7ff3bc006780 2021-02-10T11:43:46.708 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:46.690 7ff3cf7fe700 1 -- 172.21.2.112:0/516259565 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff3c000a7a0 con 0x7ff3bc006780 2021-02-10T11:43:46.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:46.690 7ff3d6559700 1 -- 172.21.2.112:0/516259565 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff3bc006780 msgr2=0x7ff3bc008ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:46.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:46.690 7ff3d6559700 1 --2- 172.21.2.112:0/516259565 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff3bc006780 0x7ff3bc008ca0 secure :-1 s=READY pgs=2504 cs=0 l=1 rev1=1 rx=0x7ff3c0001280 tx=0x7ff3c0007d00).stop 2021-02-10T11:43:46.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:46.690 7ff3d4b76700 1 -- 172.21.2.112:0/516259565 reap_dead start 2021-02-10T11:43:46.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:46.691 7ff3d6559700 1 -- 172.21.2.112:0/516259565 shutdown_connections 2021-02-10T11:43:46.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:46.691 7ff3d6559700 1 -- 172.21.2.112:0/516259565 >> 172.21.2.112:0/516259565 conn(0x7ff3d00ce450 msgr2=0x7ff3d00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:46.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:46.691 7ff3d6559700 1 -- 172.21.2.112:0/516259565 shutdown_connections 2021-02-10T11:43:46.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:46.691 7ff3d6559700 1 -- 172.21.2.112:0/516259565 wait complete. 2021-02-10T11:43:46.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:46.705 7ff3d6559700 1 Processor -- start 2021-02-10T11:43:46.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:46.705 7ff3d6559700 1 -- start start 2021-02-10T11:43:46.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:46.706 7ff3d6559700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff3d00a2560 0x7ff3d00b4fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:46.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:46.706 7ff3d6559700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff3d00b55e0 0x7ff3d00b59e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:46.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:46.706 7ff3d6559700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff3d00ee200 0x7ff3d00ee600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:46.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:46.706 7ff3d6559700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3d018fda0 con 0x7ff3d00a2560 2021-02-10T11:43:46.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:46.706 7ff3d6559700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3d00a20c0 con 0x7ff3d00ee200 2021-02-10T11:43:46.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:46.706 7ff3d6559700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3d00b6000 con 0x7ff3d00b55e0 2021-02-10T11:43:46.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:46.706 7ff3cffff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff3d00a2560 0x7ff3d00b4fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:46.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:46.706 7ff3cffff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff3d00a2560 0x7ff3d00b4fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:57298/0 (socket says 172.21.2.112:57298) 2021-02-10T11:43:46.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:46.706 7ff3cffff700 1 -- 172.21.2.112:0/890047727 learned_addr learned my addr 172.21.2.112:0/890047727 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:43:46.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:46.706 7ff3c7fff700 1 --2- 172.21.2.112:0/890047727 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff3d00b55e0 0x7ff3d00b59e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:46.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:46.706 7ff3d4b76700 1 --2- 172.21.2.112:0/890047727 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff3d00ee200 0x7ff3d00ee600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:46.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:46.706 7ff3cffff700 1 -- 172.21.2.112:0/890047727 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff3d00b55e0 msgr2=0x7ff3d00b59e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:46.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:46.706 7ff3cffff700 1 --2- 172.21.2.112:0/890047727 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff3d00b55e0 0x7ff3d00b59e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:46.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:46.706 7ff3cffff700 1 -- 172.21.2.112:0/890047727 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff3d00ee200 msgr2=0x7ff3d00ee600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:46.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:46.706 7ff3cffff700 1 --2- 172.21.2.112:0/890047727 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff3d00ee200 0x7ff3d00ee600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:46.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:46.706 7ff3cffff700 1 -- 172.21.2.112:0/890047727 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3d00eec20 con 0x7ff3d00a2560 2021-02-10T11:43:46.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:46.707 7ff3d4b76700 1 --2- 172.21.2.112:0/890047727 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff3d00ee200 0x7ff3d00ee600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:43:46.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:46.707 7ff3cffff700 1 --2- 172.21.2.112:0/890047727 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff3d00a2560 0x7ff3d00b4fc0 secure :-1 s=READY pgs=2505 cs=0 l=1 rev1=1 rx=0x7ff3b800b590 tx=0x7ff3b800b930).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:46.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:46.707 7ff3cd7fa700 1 -- 172.21.2.112:0/890047727 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff3b8006bb0 con 0x7ff3d00a2560 2021-02-10T11:43:46.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:46.707 7ff3cd7fa700 1 -- 172.21.2.112:0/890047727 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3b800bd20 con 0x7ff3d00a2560 2021-02-10T11:43:46.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:46.707 7ff3cd7fa700 1 -- 172.21.2.112:0/890047727 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff3b8006220 con 0x7ff3d00a2560 2021-02-10T11:43:46.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:46.707 7ff3d6559700 1 -- 172.21.2.112:0/890047727 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3d00eec20 con 0x7ff3d00a2560 2021-02-10T11:43:46.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:46.707 7ff3d6559700 1 -- 172.21.2.112:0/890047727 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3d00eec20 con 0x7ff3d00a2560 2021-02-10T11:43:46.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:46.708 7ff3cd7fa700 1 -- 172.21.2.112:0/890047727 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff3b8006030 con 0x7ff3d00a2560 2021-02-10T11:43:46.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:46.708 7ff3cd7fa700 1 --2- 172.21.2.112:0/890047727 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff3b40186c0 0x7ff3b401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:46.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:46.708 7ff3cd7fa700 1 -- 172.21.2.112:0/890047727 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 5659+0+0 (secure 0 0 0) 0x7ff3b8032590 con 0x7ff3d00a2560 2021-02-10T11:43:46.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:46.708 7ff3c7fff700 1 --2- 172.21.2.112:0/890047727 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff3b40186c0 0x7ff3b401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:46.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:46.709 7ff3c7fff700 1 --2- 172.21.2.112:0/890047727 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff3b40186c0 0x7ff3b401abc0 crc :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:46.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:46.738 7ff3d6559700 1 -- 172.21.2.112:0/890047727 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3d00f0b10 con 0x7ff3d00a2560 2021-02-10T11:43:46.741 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:46.740 7ff3cd7fa700 1 -- 172.21.2.112:0/890047727 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff3b8006bb0 con 0x7ff3d00a2560 2021-02-10T11:43:46.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:46.872 7ff3d6559700 1 -- 172.21.2.112:0/890047727 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff3d00f0b10 con 0x7ff3d00a2560 2021-02-10T11:43:46.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:46.872 7ff3cd7fa700 1 -- 172.21.2.112:0/890047727 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 116 v116) v1 ==== 95+0+1763 (secure 0 0 0) 0x7ff3b8006030 con 0x7ff3d00a2560 2021-02-10T11:43:46.873 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 116 2021-02-10T11:43:46.874 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:43:46.874 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":116,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11349,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/2093257968","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":2093257968},{"type":"v1","addr":"172.21.2.112:6833","nonce":2093257968}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":110}],"filesystems":[{"mdsmap":{"epoch":116,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:43:08.182625","modified":"2021-02-10 11:43:46.229395","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":135,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":14}]} 2021-02-10T11:43:46.875 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:46.874 7ff3d6559700 1 -- 172.21.2.112:0/890047727 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff3b40186c0 msgr2=0x7ff3b401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:46.875 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:46.874 7ff3d6559700 1 --2- 172.21.2.112:0/890047727 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff3b40186c0 0x7ff3b401abc0 crc :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:46.875 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:46.874 7ff3d6559700 1 -- 172.21.2.112:0/890047727 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff3d00a2560 msgr2=0x7ff3d00b4fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:46.875 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:46.874 7ff3d6559700 1 --2- 172.21.2.112:0/890047727 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff3d00a2560 0x7ff3d00b4fc0 secure :-1 s=READY pgs=2505 cs=0 l=1 rev1=1 rx=0x7ff3b800b590 tx=0x7ff3b800b930).stop 2021-02-10T11:43:46.875 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:46.874 7ff3d6559700 1 -- 172.21.2.112:0/890047727 shutdown_connections 2021-02-10T11:43:46.875 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:46.874 7ff3d6559700 1 --2- 172.21.2.112:0/890047727 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff3b40186c0 0x7ff3b401abc0 unknown :-1 s=CLOSED pgs=1146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:46.875 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:46.874 7ff3d6559700 1 --2- 172.21.2.112:0/890047727 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff3d00ee200 0x7ff3d00ee600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:46.876 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:46.874 7ff3d6559700 1 --2- 172.21.2.112:0/890047727 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff3d00b55e0 0x7ff3d00b59e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:46.876 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:46.874 7ff3d6559700 1 --2- 172.21.2.112:0/890047727 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff3d00a2560 0x7ff3d00b4fc0 unknown :-1 s=CLOSED pgs=2505 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:46.876 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:46.874 7ff3d6559700 1 -- 172.21.2.112:0/890047727 >> 172.21.2.112:0/890047727 conn(0x7ff3d00ac990 msgr2=0x7ff3d0091b30 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:46.876 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:46.875 7ff3d6559700 1 -- 172.21.2.112:0/890047727 shutdown_connections 2021-02-10T11:43:46.876 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:46.875 7ff3d6559700 1 -- 172.21.2.112:0/890047727 wait complete. 2021-02-10T11:43:46.883 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2021-02-10T11:43:47.315 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:47.299 7f2f2e126700 1 Processor -- start 2021-02-10T11:43:47.316 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:47.299 7f2f2e126700 1 -- start start 2021-02-10T11:43:47.316 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:47.299 7f2f2e126700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2f280d1870 0x7f2f281b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:47.316 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:47.299 7f2f2e126700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f2f281b0a10 0x7f2f280cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:47.316 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:47.299 7f2f2e126700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f280f0b10 con 0x7f2f280d1c70 2021-02-10T11:43:47.316 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:47.299 7f2f2e126700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f280fe230 con 0x7f2f281b0a10 2021-02-10T11:43:47.316 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:47.299 7f2f2e126700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f280e5ed0 con 0x7f2f280d1870 2021-02-10T11:43:47.316 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:47.300 7f2f26ffd700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f2f281b0a10 0x7f2f280cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:47.316 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:47.300 7f2f26ffd700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f2f281b0a10 0x7f2f280cf330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:57302/0 (socket says 172.21.2.112:57302) 2021-02-10T11:43:47.316 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:47.300 7f2f26ffd700 1 -- 172.21.2.112:0/4247946143 learned_addr learned my addr 172.21.2.112:0/4247946143 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:43:47.316 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:47.300 7f2f26ffd700 1 -- 172.21.2.112:0/4247946143 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2f280d1870 msgr2=0x7f2f281b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:47.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:47.300 7f2f26ffd700 1 --2- 172.21.2.112:0/4247946143 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2f280d1870 0x7f2f281b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:47.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:47.300 7f2f26ffd700 1 -- 172.21.2.112:0/4247946143 >> v1:172.21.2.120:6789/0 conn(0x7f2f280d1c70 legacy=0x7f2f280cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:47.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:47.300 7f2f26ffd700 1 -- 172.21.2.112:0/4247946143 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f280f0650 con 0x7f2f281b0a10 2021-02-10T11:43:47.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:47.300 7f2f26ffd700 1 --2- 172.21.2.112:0/4247946143 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2f281b0a10 0x7f2f280cf330 secure :-1 s=READY pgs=2506 cs=0 l=1 rev1=1 rx=0x7f2f18006700 tx=0x7f2f18009b80).ready entity=mon.0 client_cookie=2707c71e43249401 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:47.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:47.300 7f2f267fc700 1 -- 172.21.2.112:0/4247946143 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2f1800c0c0 con 0x7f2f281b0a10 2021-02-10T11:43:47.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:47.300 7f2f267fc700 1 -- 172.21.2.112:0/4247946143 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f1800c340 con 0x7f2f281b0a10 2021-02-10T11:43:47.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:47.300 7f2f267fc700 1 -- 172.21.2.112:0/4247946143 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2f1800c860 con 0x7f2f281b0a10 2021-02-10T11:43:47.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:47.300 7f2f2e126700 1 -- 172.21.2.112:0/4247946143 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2f281b0a10 msgr2=0x7f2f280cf330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:47.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:47.300 7f2f2e126700 1 --2- 172.21.2.112:0/4247946143 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2f281b0a10 0x7f2f280cf330 secure :-1 s=READY pgs=2506 cs=0 l=1 rev1=1 rx=0x7f2f18006700 tx=0x7f2f18009b80).stop 2021-02-10T11:43:47.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:47.301 7f2f2e126700 1 -- 172.21.2.112:0/4247946143 shutdown_connections 2021-02-10T11:43:47.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:47.301 7f2f2e126700 1 --2- 172.21.2.112:0/4247946143 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2f281b0a10 0x7f2f280cf330 unknown :-1 s=CLOSED pgs=2506 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:47.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:47.301 7f2f2e126700 1 --2- 172.21.2.112:0/4247946143 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2f280d1870 0x7f2f281b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:47.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:47.301 7f2f2e126700 1 -- 172.21.2.112:0/4247946143 >> 172.21.2.112:0/4247946143 conn(0x7f2f280ce450 msgr2=0x7f2f280d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:47.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:47.301 7f2f2e126700 1 -- 172.21.2.112:0/4247946143 shutdown_connections 2021-02-10T11:43:47.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:47.301 7f2f2e126700 1 -- 172.21.2.112:0/4247946143 wait complete. 2021-02-10T11:43:47.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:47.316 7f2f2e126700 1 Processor -- start 2021-02-10T11:43:47.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:47.316 7f2f2e126700 1 -- start start 2021-02-10T11:43:47.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:47.316 7f2f2e126700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2f281908b0 0x7f2f281a4440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:47.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:47.316 7f2f2e126700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2f280bd920 0x7f2f280bdd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:47.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:47.316 7f2f2e126700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2f2808d510 0x7f2f2808d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:47.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:47.316 7f2f2e126700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f2808cf90 con 0x7f2f2808d510 2021-02-10T11:43:47.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:47.316 7f2f2e126700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f280be340 con 0x7f2f281908b0 2021-02-10T11:43:47.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:47.316 7f2f2e126700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f280be5f0 con 0x7f2f280bd920 2021-02-10T11:43:47.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:47.316 7f2f27fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2f2808d510 0x7f2f2808d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:47.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:47.316 7f2f27fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2f2808d510 0x7f2f2808d910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:57308/0 (socket says 172.21.2.112:57308) 2021-02-10T11:43:47.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:47.316 7f2f27fff700 1 -- 172.21.2.112:0/3826488866 learned_addr learned my addr 172.21.2.112:0/3826488866 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:43:47.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:47.316 7f2f27fff700 1 -- 172.21.2.112:0/3826488866 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2f280bd920 msgr2=0x7f2f280bdd20 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:43:47.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:47.316 7f2f27fff700 1 --2- 172.21.2.112:0/3826488866 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2f280bd920 0x7f2f280bdd20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:47.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:47.316 7f2f27fff700 1 -- 172.21.2.112:0/3826488866 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2f281908b0 msgr2=0x7f2f281a4440 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:43:47.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:47.316 7f2f27fff700 1 --2- 172.21.2.112:0/3826488866 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2f281908b0 0x7f2f281a4440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:47.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:47.316 7f2f27fff700 1 -- 172.21.2.112:0/3826488866 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f281a4a60 con 0x7f2f2808d510 2021-02-10T11:43:47.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:47.317 7f2f27fff700 1 --2- 172.21.2.112:0/3826488866 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2f2808d510 0x7f2f2808d910 secure :-1 s=READY pgs=2507 cs=0 l=1 rev1=1 rx=0x7f2f20009940 tx=0x7f2f2000bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:47.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:47.317 7f2f1ffff700 1 -- 172.21.2.112:0/3826488866 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2f20000f90 con 0x7f2f2808d510 2021-02-10T11:43:47.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:47.317 7f2f1ffff700 1 -- 172.21.2.112:0/3826488866 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f2000c2f0 con 0x7f2f2808d510 2021-02-10T11:43:47.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:47.317 7f2f1ffff700 1 -- 172.21.2.112:0/3826488866 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2f2000c810 con 0x7f2f2808d510 2021-02-10T11:43:47.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:47.317 7f2f2e126700 1 -- 172.21.2.112:0/3826488866 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f281a4a60 con 0x7f2f2808d510 2021-02-10T11:43:47.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:47.317 7f2f2e126700 1 -- 172.21.2.112:0/3826488866 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f2808df30 con 0x7f2f2808d510 2021-02-10T11:43:47.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:47.318 7f2f1ffff700 1 -- 172.21.2.112:0/3826488866 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2f20001220 con 0x7f2f2808d510 2021-02-10T11:43:47.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:47.318 7f2f1ffff700 1 --2- 172.21.2.112:0/3826488866 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2f0c0186c0 0x7f2f0c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:47.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:47.318 7f2f1ffff700 1 -- 172.21.2.112:0/3826488866 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 5659+0+0 (secure 0 0 0) 0x7f2f20032950 con 0x7f2f2808d510 2021-02-10T11:43:47.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:47.318 7f2f277fe700 1 --2- 172.21.2.112:0/3826488866 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2f0c0186c0 0x7f2f0c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:47.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:47.319 7f2f277fe700 1 --2- 172.21.2.112:0/3826488866 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2f0c0186c0 0x7f2f0c01abc0 crc :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:47.350 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:47.349 7f2f2e126700 1 -- 172.21.2.112:0/3826488866 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f280ee060 con 0x7f2f2808d510 2021-02-10T11:43:47.352 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:47.352 7f2f1ffff700 1 -- 172.21.2.112:0/3826488866 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2f20000f40 con 0x7f2f2808d510 2021-02-10T11:43:47.482 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:47.481 7f2f2e126700 1 -- 172.21.2.112:0/3826488866 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f2f280ce450 con 0x7f2f2808d510 2021-02-10T11:43:48.240 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:48.239 7f2f1ffff700 1 -- 172.21.2.112:0/3826488866 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v117) v1 ==== 104+0+0 (secure 0 0 0) 0x7f2f2001f6a0 con 0x7f2f2808d510 2021-02-10T11:43:48.254 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:48.253 7f2f2e126700 1 -- 172.21.2.112:0/3826488866 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2f0c0186c0 msgr2=0x7f2f0c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:48.254 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:48.253 7f2f2e126700 1 --2- 172.21.2.112:0/3826488866 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2f0c0186c0 0x7f2f0c01abc0 crc :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:48.254 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:48.253 7f2f2e126700 1 -- 172.21.2.112:0/3826488866 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2f2808d510 msgr2=0x7f2f2808d910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:48.254 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:48.253 7f2f2e126700 1 --2- 172.21.2.112:0/3826488866 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2f2808d510 0x7f2f2808d910 secure :-1 s=READY pgs=2507 cs=0 l=1 rev1=1 rx=0x7f2f20009940 tx=0x7f2f2000bcf0).stop 2021-02-10T11:43:48.254 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:48.253 7f2f2e126700 1 -- 172.21.2.112:0/3826488866 shutdown_connections 2021-02-10T11:43:48.254 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:48.253 7f2f2e126700 1 --2- 172.21.2.112:0/3826488866 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2f0c0186c0 0x7f2f0c01abc0 unknown :-1 s=CLOSED pgs=1147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:48.254 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:48.253 7f2f2e126700 1 --2- 172.21.2.112:0/3826488866 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2f2808d510 0x7f2f2808d910 unknown :-1 s=CLOSED pgs=2507 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:48.254 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:48.253 7f2f2e126700 1 --2- 172.21.2.112:0/3826488866 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2f280bd920 0x7f2f280bdd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:48.255 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:48.253 7f2f2e126700 1 --2- 172.21.2.112:0/3826488866 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2f281908b0 0x7f2f281a4440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:48.255 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:48.253 7f2f2e126700 1 -- 172.21.2.112:0/3826488866 >> 172.21.2.112:0/3826488866 conn(0x7f2f2808f030 msgr2=0x7f2f280b5e80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:48.255 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:48.254 7f2f2e126700 1 -- 172.21.2.112:0/3826488866 shutdown_connections 2021-02-10T11:43:48.255 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:48.254 7f2f2e126700 1 -- 172.21.2.112:0/3826488866 wait complete. 2021-02-10T11:43:48.262 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2021-02-10T11:43:48.695 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:48.678 7f4d7706f700 1 Processor -- start 2021-02-10T11:43:48.695 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:48.678 7f4d7706f700 1 -- start start 2021-02-10T11:43:48.695 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:48.679 7f4d7706f700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f4d700d1880 0x7f4d701b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:48.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:48.679 7f4d7706f700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4d701b0af0 0x7f4d70108950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:48.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:48.679 7f4d7706f700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d700fe1d0 con 0x7f4d700d1c80 2021-02-10T11:43:48.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:48.679 7f4d7706f700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d70109a70 con 0x7f4d700d1880 2021-02-10T11:43:48.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:48.679 7f4d7706f700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d70109760 con 0x7f4d701b0af0 2021-02-10T11:43:48.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:48.679 7f4d74e8b700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f4d700d1880 0x7f4d701b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:48.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:48.679 7f4d7568c700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f4d700d1c80 0x7f4d70108f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:56782/0 (socket says 172.21.2.112:56782) 2021-02-10T11:43:48.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:48.679 7f4d7568c700 1 -- 172.21.2.112:0/651187904 learned_addr learned my addr 172.21.2.112:0/651187904 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:43:48.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:48.679 7f4d74e8b700 1 -- 172.21.2.112:0/651187904 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4d701b0af0 msgr2=0x7f4d70108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:48.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:48.679 7f4d74e8b700 1 --2- 172.21.2.112:0/651187904 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4d701b0af0 0x7f4d70108950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:48.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:48.679 7f4d74e8b700 1 -- 172.21.2.112:0/651187904 >> v1:172.21.2.112:6789/0 conn(0x7f4d700d1c80 legacy=0x7f4d70108f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:48.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:48.679 7f4d6f7fe700 1 -- 172.21.2.112:0/651187904 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 253017910 0 0) 0x7f4d68001500 con 0x7f4d700d1c80 2021-02-10T11:43:48.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:48.679 7f4d74e8b700 1 -- 172.21.2.112:0/651187904 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d70088070 con 0x7f4d700d1880 2021-02-10T11:43:48.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:48.680 7f4d74e8b700 1 --2- 172.21.2.112:0/651187904 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4d700d1880 0x7f4d701b04d0 secure :-1 s=READY pgs=2509 cs=0 l=1 rev1=1 rx=0x7f4d58006650 tx=0x7f4d58009a80).ready entity=mon.0 client_cookie=239b2b60320d5192 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:48.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:48.680 7f4d6f7fe700 1 -- 172.21.2.112:0/651187904 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4d5800c0c0 con 0x7f4d700d1880 2021-02-10T11:43:48.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:48.680 7f4d6f7fe700 1 -- 172.21.2.112:0/651187904 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d5800c340 con 0x7f4d700d1880 2021-02-10T11:43:48.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:48.680 7f4d6f7fe700 1 -- 172.21.2.112:0/651187904 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4d5800c860 con 0x7f4d700d1880 2021-02-10T11:43:48.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:48.680 7f4d7706f700 1 -- 172.21.2.112:0/651187904 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4d700d1880 msgr2=0x7f4d701b04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:48.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:48.680 7f4d7706f700 1 --2- 172.21.2.112:0/651187904 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4d700d1880 0x7f4d701b04d0 secure :-1 s=READY pgs=2509 cs=0 l=1 rev1=1 rx=0x7f4d58006650 tx=0x7f4d58009a80).stop 2021-02-10T11:43:48.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:48.680 7f4d7706f700 1 -- 172.21.2.112:0/651187904 shutdown_connections 2021-02-10T11:43:48.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:48.680 7f4d7706f700 1 --2- 172.21.2.112:0/651187904 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4d701b0af0 0x7f4d70108950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:48.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:48.680 7f4d7706f700 1 --2- 172.21.2.112:0/651187904 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4d700d1880 0x7f4d701b04d0 unknown :-1 s=CLOSED pgs=2509 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:48.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:48.680 7f4d7706f700 1 -- 172.21.2.112:0/651187904 >> 172.21.2.112:0/651187904 conn(0x7f4d700ce430 msgr2=0x7f4d700d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:48.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:48.681 7f4d7706f700 1 -- 172.21.2.112:0/651187904 shutdown_connections 2021-02-10T11:43:48.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:48.681 7f4d7706f700 1 -- 172.21.2.112:0/651187904 wait complete. 2021-02-10T11:43:48.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:48.695 7f4d7706f700 1 Processor -- start 2021-02-10T11:43:48.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:48.695 7f4d7706f700 1 -- start start 2021-02-10T11:43:48.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:48.696 7f4d7706f700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4d701b58d0 0x7f4d700aa4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:48.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:48.696 7f4d7706f700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4d700aaaf0 0x7f4d701b80a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:48.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:48.696 7f4d7706f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4d701b86c0 0x7f4d7019de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:48.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:48.696 7f4d7706f700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d701b8ac0 con 0x7f4d701b86c0 2021-02-10T11:43:48.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:48.696 7f4d7706f700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d701b8e20 con 0x7f4d700aaaf0 2021-02-10T11:43:48.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:48.696 7f4d7706f700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d701b5cd0 con 0x7f4d701b58d0 2021-02-10T11:43:48.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:48.696 7f4d7568c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4d701b86c0 0x7f4d7019de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:48.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:48.696 7f4d7568c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4d701b86c0 0x7f4d7019de70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:57322/0 (socket says 172.21.2.112:57322) 2021-02-10T11:43:48.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:48.696 7f4d7568c700 1 -- 172.21.2.112:0/3077244976 learned_addr learned my addr 172.21.2.112:0/3077244976 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:43:48.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:48.696 7f4d7568c700 1 -- 172.21.2.112:0/3077244976 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4d701b58d0 msgr2=0x7f4d700aa4d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:43:48.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:48.696 7f4d7568c700 1 --2- 172.21.2.112:0/3077244976 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4d701b58d0 0x7f4d700aa4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:48.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:48.696 7f4d7568c700 1 -- 172.21.2.112:0/3077244976 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4d700aaaf0 msgr2=0x7f4d701b80a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:43:48.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:48.696 7f4d7568c700 1 --2- 172.21.2.112:0/3077244976 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4d700aaaf0 0x7f4d701b80a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:48.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:48.696 7f4d7568c700 1 -- 172.21.2.112:0/3077244976 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d70095710 con 0x7f4d701b86c0 2021-02-10T11:43:48.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:48.696 7f4d7568c700 1 --2- 172.21.2.112:0/3077244976 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4d701b86c0 0x7f4d7019de70 secure :-1 s=READY pgs=2510 cs=0 l=1 rev1=1 rx=0x7f4d68001c10 tx=0x7f4d6800ac60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:48.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:48.697 7f4d6d7fa700 1 -- 172.21.2.112:0/3077244976 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4d680070d0 con 0x7f4d701b86c0 2021-02-10T11:43:48.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:48.697 7f4d6d7fa700 1 -- 172.21.2.112:0/3077244976 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d68007350 con 0x7f4d701b86c0 2021-02-10T11:43:48.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:48.697 7f4d6d7fa700 1 -- 172.21.2.112:0/3077244976 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4d680049a0 con 0x7f4d701b86c0 2021-02-10T11:43:48.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:48.697 7f4d7706f700 1 -- 172.21.2.112:0/3077244976 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d70095710 con 0x7f4d701b86c0 2021-02-10T11:43:48.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:48.697 7f4d7706f700 1 -- 172.21.2.112:0/3077244976 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d7019e780 con 0x7f4d701b86c0 2021-02-10T11:43:48.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:48.697 7f4d6d7fa700 1 -- 172.21.2.112:0/3077244976 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4d68004720 con 0x7f4d701b86c0 2021-02-10T11:43:48.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:48.698 7f4d6d7fa700 1 --2- 172.21.2.112:0/3077244976 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4d540186c0 0x7f4d5401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:48.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:48.698 7f4d6d7fa700 1 -- 172.21.2.112:0/3077244976 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 5659+0+0 (secure 0 0 0) 0x7f4d68032910 con 0x7f4d701b86c0 2021-02-10T11:43:48.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:48.698 7f4d74e8b700 1 --2- 172.21.2.112:0/3077244976 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4d540186c0 0x7f4d5401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:48.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:48.699 7f4d74e8b700 1 --2- 172.21.2.112:0/3077244976 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4d540186c0 0x7f4d5401abc0 crc :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:48.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:48.728 7f4d7706f700 1 -- 172.21.2.112:0/3077244976 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d70082aa0 con 0x7f4d701b86c0 2021-02-10T11:43:48.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:48.731 7f4d6d7fa700 1 -- 172.21.2.112:0/3077244976 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4d68007530 con 0x7f4d701b86c0 2021-02-10T11:43:48.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:48.858 7f4d7706f700 1 -- 172.21.2.112:0/3077244976 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f4d70082aa0 con 0x7f4d701b86c0 2021-02-10T11:43:49.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:49.246 7f4d6d7fa700 1 -- 172.21.2.112:0/3077244976 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v136) v1 ==== 185+0+0 (secure 0 0 0) 0x7f4d68004720 con 0x7f4d701b86c0 2021-02-10T11:43:49.275 INFO:teuthology.orchestra.run.gibba012.stderr:pool 'cephfs_metadata' removed 2021-02-10T11:43:49.277 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:49.277 7f4d7706f700 1 -- 172.21.2.112:0/3077244976 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4d540186c0 msgr2=0x7f4d5401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:49.277 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:49.277 7f4d7706f700 1 --2- 172.21.2.112:0/3077244976 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4d540186c0 0x7f4d5401abc0 crc :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:49.277 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:49.277 7f4d7706f700 1 -- 172.21.2.112:0/3077244976 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4d701b86c0 msgr2=0x7f4d7019de70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:49.277 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:49.277 7f4d7706f700 1 --2- 172.21.2.112:0/3077244976 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4d701b86c0 0x7f4d7019de70 secure :-1 s=READY pgs=2510 cs=0 l=1 rev1=1 rx=0x7f4d68001c10 tx=0x7f4d6800ac60).stop 2021-02-10T11:43:49.278 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:49.277 7f4d7706f700 1 -- 172.21.2.112:0/3077244976 shutdown_connections 2021-02-10T11:43:49.278 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:49.277 7f4d7706f700 1 --2- 172.21.2.112:0/3077244976 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4d540186c0 0x7f4d5401abc0 unknown :-1 s=CLOSED pgs=1148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:49.278 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:49.277 7f4d7706f700 1 --2- 172.21.2.112:0/3077244976 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4d700aaaf0 0x7f4d701b80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:49.278 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:49.277 7f4d7706f700 1 --2- 172.21.2.112:0/3077244976 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4d701b86c0 0x7f4d7019de70 unknown :-1 s=CLOSED pgs=2510 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:49.278 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:49.277 7f4d7706f700 1 --2- 172.21.2.112:0/3077244976 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4d701b58d0 0x7f4d700aa4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:49.278 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:49.277 7f4d7706f700 1 -- 172.21.2.112:0/3077244976 >> 172.21.2.112:0/3077244976 conn(0x7f4d7019f940 msgr2=0x7f4d7018c290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:49.278 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:49.277 7f4d7706f700 1 -- 172.21.2.112:0/3077244976 shutdown_connections 2021-02-10T11:43:49.278 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:49.277 7f4d7706f700 1 -- 172.21.2.112:0/3077244976 wait complete. 2021-02-10T11:43:49.285 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2021-02-10T11:43:49.716 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:49.699 7f9d84656700 1 Processor -- start 2021-02-10T11:43:49.716 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:49.699 7f9d84656700 1 -- start start 2021-02-10T11:43:49.716 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:49.700 7f9d84656700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f9d7c0d1c80 0x7f9d7c0cef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:49.716 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:49.700 7f9d84656700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9d7c0cf590 0x7f9d7c0cf990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:49.717 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:49.700 7f9d84656700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d7c0fe1d0 con 0x7f9d7c0d1880 2021-02-10T11:43:49.717 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:49.700 7f9d84656700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d7c1b0cc0 con 0x7f9d7c0d1c80 2021-02-10T11:43:49.717 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:49.700 7f9d84656700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d7c109a70 con 0x7f9d7c0cf590 2021-02-10T11:43:49.717 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:49.700 7f9d81c71700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f9d7c0d1c80 0x7f9d7c0cef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:49.717 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:49.700 7f9d81c71700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f9d7c0d1c80 0x7f9d7c0cef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:57330/0 (socket says 172.21.2.112:57330) 2021-02-10T11:43:49.717 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:49.700 7f9d81c71700 1 -- 172.21.2.112:0/4153078843 learned_addr learned my addr 172.21.2.112:0/4153078843 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:43:49.717 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:49.700 7f9d81c71700 1 -- 172.21.2.112:0/4153078843 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9d7c0cf590 msgr2=0x7f9d7c0cf990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:49.717 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:49.700 7f9d81c71700 1 --2- 172.21.2.112:0/4153078843 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9d7c0cf590 0x7f9d7c0cf990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:49.717 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:49.701 7f9d81c71700 1 -- 172.21.2.112:0/4153078843 >> v1:172.21.2.120:6789/0 conn(0x7f9d7c0d1880 legacy=0x7f9d7c1b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:49.717 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:49.701 7f9d81c71700 1 -- 172.21.2.112:0/4153078843 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d7c088070 con 0x7f9d7c0d1c80 2021-02-10T11:43:49.717 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:49.701 7f9d81c71700 1 --2- 172.21.2.112:0/4153078843 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9d7c0d1c80 0x7f9d7c0cef70 secure :-1 s=READY pgs=2511 cs=0 l=1 rev1=1 rx=0x7f9d78006700 tx=0x7f9d78009b00).ready entity=mon.0 client_cookie=fdfc0f69e6c3894d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:49.717 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:49.701 7f9d81470700 1 -- 172.21.2.112:0/4153078843 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9d7800c0c0 con 0x7f9d7c0d1c80 2021-02-10T11:43:49.717 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:49.701 7f9d81470700 1 -- 172.21.2.112:0/4153078843 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d7800c400 con 0x7f9d7c0d1c80 2021-02-10T11:43:49.718 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:49.701 7f9d81470700 1 -- 172.21.2.112:0/4153078843 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9d7800c920 con 0x7f9d7c0d1c80 2021-02-10T11:43:49.718 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:49.701 7f9d84656700 1 -- 172.21.2.112:0/4153078843 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9d7c0d1c80 msgr2=0x7f9d7c0cef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:49.718 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:49.701 7f9d84656700 1 --2- 172.21.2.112:0/4153078843 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9d7c0d1c80 0x7f9d7c0cef70 secure :-1 s=READY pgs=2511 cs=0 l=1 rev1=1 rx=0x7f9d78006700 tx=0x7f9d78009b00).stop 2021-02-10T11:43:49.718 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:49.702 7f9d84656700 1 -- 172.21.2.112:0/4153078843 shutdown_connections 2021-02-10T11:43:49.718 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:49.702 7f9d84656700 1 --2- 172.21.2.112:0/4153078843 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9d7c0d1c80 0x7f9d7c0cef70 unknown :-1 s=CLOSED pgs=2511 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:49.718 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:49.702 7f9d84656700 1 --2- 172.21.2.112:0/4153078843 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9d7c0cf590 0x7f9d7c0cf990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:49.718 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:49.702 7f9d84656700 1 -- 172.21.2.112:0/4153078843 >> 172.21.2.112:0/4153078843 conn(0x7f9d7c0ce430 msgr2=0x7f9d7c0d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:49.718 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:49.702 7f9d84656700 1 -- 172.21.2.112:0/4153078843 shutdown_connections 2021-02-10T11:43:49.718 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:49.702 7f9d84656700 1 -- 172.21.2.112:0/4153078843 wait complete. 2021-02-10T11:43:49.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:49.716 7f9d84656700 1 Processor -- start 2021-02-10T11:43:49.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:49.716 7f9d84656700 1 -- start start 2021-02-10T11:43:49.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:49.717 7f9d84656700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9d7c19e5b0 0x7f9d7c1afc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:49.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:49.717 7f9d84656700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9d7c19e9b0 0x7f9d7c1b0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:49.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:49.717 7f9d84656700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9d7c1b0250 0x7f9d7c1b13e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:49.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:49.717 7f9d84656700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d7c195bd0 con 0x7f9d7c19e5b0 2021-02-10T11:43:49.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:49.717 7f9d84656700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d7c1b0840 con 0x7f9d7c19e9b0 2021-02-10T11:43:49.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:49.717 7f9d84656700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d7c19eed0 con 0x7f9d7c1b0250 2021-02-10T11:43:49.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:49.717 7f9d82472700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9d7c19e5b0 0x7f9d7c1afc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:49.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:49.717 7f9d82472700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9d7c19e5b0 0x7f9d7c1afc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:57336/0 (socket says 172.21.2.112:57336) 2021-02-10T11:43:49.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:49.717 7f9d82472700 1 -- 172.21.2.112:0/2623254869 learned_addr learned my addr 172.21.2.112:0/2623254869 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:43:49.721 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:49.717 7f9d82472700 1 -- 172.21.2.112:0/2623254869 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9d7c1b0250 msgr2=0x7f9d7c1b13e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:49.721 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:49.717 7f9d82472700 1 --2- 172.21.2.112:0/2623254869 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9d7c1b0250 0x7f9d7c1b13e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:49.721 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:49.717 7f9d82472700 1 -- 172.21.2.112:0/2623254869 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9d7c19e9b0 msgr2=0x7f9d7c1b0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:49.721 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:49.717 7f9d82472700 1 --2- 172.21.2.112:0/2623254869 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9d7c19e9b0 0x7f9d7c1b0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:49.721 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:49.717 7f9d82472700 1 -- 172.21.2.112:0/2623254869 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d7c19f0b0 con 0x7f9d7c19e5b0 2021-02-10T11:43:49.721 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:49.717 7f9d82472700 1 --2- 172.21.2.112:0/2623254869 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9d7c19e5b0 0x7f9d7c1afc30 secure :-1 s=READY pgs=2512 cs=0 l=1 rev1=1 rx=0x7f9d7000b5b0 tx=0x7f9d7000b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:49.721 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:49.718 7f9d6effd700 1 -- 172.21.2.112:0/2623254869 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9d7000bd40 con 0x7f9d7c19e5b0 2021-02-10T11:43:49.721 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:49.718 7f9d6effd700 1 -- 172.21.2.112:0/2623254869 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d70006240 con 0x7f9d7c19e5b0 2021-02-10T11:43:49.721 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:49.718 7f9d84656700 1 -- 172.21.2.112:0/2623254869 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d7c19f0b0 con 0x7f9d7c19e5b0 2021-02-10T11:43:49.721 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:49.718 7f9d6effd700 1 -- 172.21.2.112:0/2623254869 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9d70005720 con 0x7f9d7c19e5b0 2021-02-10T11:43:49.721 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:49.718 7f9d84656700 1 -- 172.21.2.112:0/2623254869 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d7c1b1cf0 con 0x7f9d7c19e5b0 2021-02-10T11:43:49.721 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:49.718 7f9d6effd700 1 -- 172.21.2.112:0/2623254869 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9d700054a0 con 0x7f9d7c19e5b0 2021-02-10T11:43:49.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:49.719 7f9d6effd700 1 --2- 172.21.2.112:0/2623254869 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9d64018700 0x7f9d6401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:49.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:49.719 7f9d6effd700 1 -- 172.21.2.112:0/2623254869 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 5221+0+0 (secure 0 0 0) 0x7f9d70032690 con 0x7f9d7c19e5b0 2021-02-10T11:43:49.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:49.719 7f9d81c71700 1 --2- 172.21.2.112:0/2623254869 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9d64018700 0x7f9d6401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:49.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:49.719 7f9d81c71700 1 --2- 172.21.2.112:0/2623254869 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9d64018700 0x7f9d6401abc0 crc :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:49.750 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:49.750 7f9d84656700 1 -- 172.21.2.112:0/2623254869 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d7c1b0840 con 0x7f9d7c19e5b0 2021-02-10T11:43:49.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:49.752 7f9d6effd700 1 -- 172.21.2.112:0/2623254869 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9d70000c60 con 0x7f9d7c19e5b0 2021-02-10T11:43:49.880 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:49.879 7f9d84656700 1 -- 172.21.2.112:0/2623254869 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f9d7c1b0840 con 0x7f9d7c19e5b0 2021-02-10T11:43:50.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:50.250 7f9d6effd700 1 -- 172.21.2.112:0/2623254869 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v137) v1 ==== 173+0+0 (secure 0 0 0) 0x7f9d700054a0 con 0x7f9d7c19e5b0 2021-02-10T11:43:50.297 INFO:teuthology.orchestra.run.gibba012.stderr:pool 'cephfs_data' removed 2021-02-10T11:43:50.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:50.299 7f9d84656700 1 -- 172.21.2.112:0/2623254869 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9d64018700 msgr2=0x7f9d6401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:50.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:50.299 7f9d84656700 1 --2- 172.21.2.112:0/2623254869 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9d64018700 0x7f9d6401abc0 crc :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:50.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:50.299 7f9d84656700 1 -- 172.21.2.112:0/2623254869 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9d7c19e5b0 msgr2=0x7f9d7c1afc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:50.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:50.299 7f9d84656700 1 --2- 172.21.2.112:0/2623254869 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9d7c19e5b0 0x7f9d7c1afc30 secure :-1 s=READY pgs=2512 cs=0 l=1 rev1=1 rx=0x7f9d7000b5b0 tx=0x7f9d7000b950).stop 2021-02-10T11:43:50.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:50.299 7f9d84656700 1 -- 172.21.2.112:0/2623254869 shutdown_connections 2021-02-10T11:43:50.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:50.299 7f9d84656700 1 --2- 172.21.2.112:0/2623254869 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9d64018700 0x7f9d6401abc0 unknown :-1 s=CLOSED pgs=1149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:50.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:50.299 7f9d84656700 1 --2- 172.21.2.112:0/2623254869 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9d7c19e9b0 0x7f9d7c1b0dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:50.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:50.299 7f9d84656700 1 --2- 172.21.2.112:0/2623254869 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9d7c1b0250 0x7f9d7c1b13e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:50.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:50.299 7f9d84656700 1 --2- 172.21.2.112:0/2623254869 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9d7c19e5b0 0x7f9d7c1afc30 unknown :-1 s=CLOSED pgs=2512 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:50.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:50.299 7f9d84656700 1 -- 172.21.2.112:0/2623254869 >> 172.21.2.112:0/2623254869 conn(0x7f9d7c19f940 msgr2=0x7f9d7c18c290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:50.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:50.299 7f9d84656700 1 -- 172.21.2.112:0/2623254869 shutdown_connections 2021-02-10T11:43:50.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:50.299 7f9d84656700 1 -- 172.21.2.112:0/2623254869 wait complete. 2021-02-10T11:43:50.308 INFO:tasks.ceph.mds.a:Restarting daemon 2021-02-10T11:43:50.308 INFO:tasks.ceph.mds.a:Stopping old one... 2021-02-10T11:43:50.308 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2021-02-10T11:43:50.308 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:43:50.308 INFO:tasks.ceph.mds.b:Restarting daemon 2021-02-10T11:43:50.309 INFO:tasks.ceph.mds.b:Stopping old one... 2021-02-10T11:43:50.309 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2021-02-10T11:43:50.309 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:43:50.315 INFO:tasks.ceph.mds.a:Stopped 2021-02-10T11:43:50.315 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2021-02-10T11:43:50.317 INFO:tasks.ceph.mds.b:Stopped 2021-02-10T11:43:50.317 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2021-02-10T11:43:50.357 INFO:tasks.ceph.mds.a:Started 2021-02-10T11:43:50.357 DEBUG:teuthology.parallel:result is None 2021-02-10T11:43:50.362 INFO:tasks.ceph.mds.b:Started 2021-02-10T11:43:50.362 DEBUG:teuthology.parallel:result is None 2021-02-10T11:43:50.362 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd blacklist clear 2021-02-10T11:43:50.765 INFO:tasks.ceph.mds.b.gibba020.stdout:starting mds.b at 2021-02-10T11:43:50.767 INFO:tasks.ceph.mds.a.gibba012.stdout:starting mds.a at 2021-02-10T11:43:50.787 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:50.770 7f9f0d41d700 1 Processor -- start 2021-02-10T11:43:50.787 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:50.770 7f9f0d41d700 1 -- start start 2021-02-10T11:43:50.787 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:50.771 7f9f0d41d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f9f080d1870 0x7f9f081b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:50.787 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:50.771 7f9f0d41d700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f9f081b0a10 0x7f9f080cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:50.787 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:50.771 7f9f0d41d700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f080f0b10 con 0x7f9f080d1c70 2021-02-10T11:43:50.787 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:50.771 7f9f0d41d700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f080e5ed0 con 0x7f9f080d1870 2021-02-10T11:43:50.787 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:50.771 7f9f0d41d700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f080f0650 con 0x7f9f081b0a10 2021-02-10T11:43:50.787 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:50.771 7f9f0707d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f9f080d1870 0x7f9f081b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:50.787 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:50.771 7f9f0707d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f9f080d1870 0x7f9f081b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:57354/0 (socket says 172.21.2.112:57354) 2021-02-10T11:43:50.788 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:50.771 7f9f0707d700 1 -- 172.21.2.112:0/2788996027 learned_addr learned my addr 172.21.2.112:0/2788996027 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:43:50.788 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:50.771 7f9f0687c700 1 --2- 172.21.2.112:0/2788996027 >> v2:172.21.2.120:3300/0 conn(0x7f9f081b0a10 0x7f9f080cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:50.788 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:50.771 7f9f0707d700 1 -- 172.21.2.112:0/2788996027 >> v2:172.21.2.120:3300/0 conn(0x7f9f081b0a10 msgr2=0x7f9f080cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:50.788 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:50.771 7f9f0707d700 1 --2- 172.21.2.112:0/2788996027 >> v2:172.21.2.120:3300/0 conn(0x7f9f081b0a10 0x7f9f080cf330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:50.788 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:50.771 7f9f0707d700 1 -- 172.21.2.112:0/2788996027 >> v1:172.21.2.112:6789/0 conn(0x7f9f080d1c70 legacy=0x7f9f080cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:50.788 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:50.771 7f9f0707d700 1 -- 172.21.2.112:0/2788996027 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f080f0840 con 0x7f9f080d1870 2021-02-10T11:43:50.788 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:50.771 7f9f0687c700 1 --2- 172.21.2.112:0/2788996027 >> v2:172.21.2.120:3300/0 conn(0x7f9f081b0a10 0x7f9f080cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:43:50.788 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:50.771 7f9f0707d700 1 --2- 172.21.2.112:0/2788996027 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9f080d1870 0x7f9f081b03f0 secure :-1 s=READY pgs=2517 cs=0 l=1 rev1=1 rx=0x7f9efc006700 tx=0x7f9efc009b80).ready entity=mon.0 client_cookie=2491fb1cc5b2d48c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:50.788 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:50.771 7f9f0607b700 1 -- 172.21.2.112:0/2788996027 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9efc00c0c0 con 0x7f9f080d1870 2021-02-10T11:43:50.788 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:50.771 7f9f0607b700 1 -- 172.21.2.112:0/2788996027 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9efc00c440 con 0x7f9f080d1870 2021-02-10T11:43:50.788 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:50.771 7f9f0607b700 1 -- 172.21.2.112:0/2788996027 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9efc00c920 con 0x7f9f080d1870 2021-02-10T11:43:50.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:50.771 7f9f0d41d700 1 -- 172.21.2.112:0/2788996027 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9f080d1870 msgr2=0x7f9f081b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:50.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:50.771 7f9f0d41d700 1 --2- 172.21.2.112:0/2788996027 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9f080d1870 0x7f9f081b03f0 secure :-1 s=READY pgs=2517 cs=0 l=1 rev1=1 rx=0x7f9efc006700 tx=0x7f9efc009b80).stop 2021-02-10T11:43:50.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:50.772 7f9f0d41d700 1 -- 172.21.2.112:0/2788996027 shutdown_connections 2021-02-10T11:43:50.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:50.772 7f9f0d41d700 1 --2- 172.21.2.112:0/2788996027 >> v2:172.21.2.120:3300/0 conn(0x7f9f081b0a10 0x7f9f080cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:50.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:50.772 7f9f0d41d700 1 --2- 172.21.2.112:0/2788996027 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9f080d1870 0x7f9f081b03f0 unknown :-1 s=CLOSED pgs=2517 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:50.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:50.772 7f9f0d41d700 1 -- 172.21.2.112:0/2788996027 >> 172.21.2.112:0/2788996027 conn(0x7f9f080ce450 msgr2=0x7f9f080d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:50.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:50.772 7f9f0d41d700 1 -- 172.21.2.112:0/2788996027 shutdown_connections 2021-02-10T11:43:50.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:50.772 7f9f0d41d700 1 -- 172.21.2.112:0/2788996027 wait complete. 2021-02-10T11:43:50.791 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:50.787 7f9f0d41d700 1 Processor -- start 2021-02-10T11:43:50.791 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:50.787 7f9f0d41d700 1 -- start start 2021-02-10T11:43:50.791 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:50.787 7f9f0d41d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9f08189d00 0x7f9f0818a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:50.791 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:50.787 7f9f0d41d700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9f0818a720 0x7f9f0818ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:50.791 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:50.787 7f9f0d41d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9f08187e50 0x7f9f08188250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:50.791 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:50.787 7f9f0d41d700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f080c37e0 con 0x7f9f08187e50 2021-02-10T11:43:50.791 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:50.787 7f9f0d41d700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f080c39d0 con 0x7f9f0818a720 2021-02-10T11:43:50.791 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:50.787 7f9f0d41d700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f08188870 con 0x7f9f08189d00 2021-02-10T11:43:50.791 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:50.787 7f9f0787e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9f08187e50 0x7f9f08188250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:50.791 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:50.787 7f9f0787e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9f08187e50 0x7f9f08188250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:57364/0 (socket says 172.21.2.112:57364) 2021-02-10T11:43:50.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:50.787 7f9f0787e700 1 -- 172.21.2.112:0/7576505 learned_addr learned my addr 172.21.2.112:0/7576505 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:43:50.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:50.787 7f9f0707d700 1 --2- 172.21.2.112:0/7576505 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9f08189d00 0x7f9f0818a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:50.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:50.788 7f9f0687c700 1 --2- 172.21.2.112:0/7576505 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9f0818a720 0x7f9f0818ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:50.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:50.788 7f9f0787e700 1 -- 172.21.2.112:0/7576505 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9f08189d00 msgr2=0x7f9f0818a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:50.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:50.788 7f9f0787e700 1 --2- 172.21.2.112:0/7576505 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9f08189d00 0x7f9f0818a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:50.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:50.788 7f9f0787e700 1 -- 172.21.2.112:0/7576505 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9f0818a720 msgr2=0x7f9f0818ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:50.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:50.788 7f9f0787e700 1 --2- 172.21.2.112:0/7576505 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9f0818a720 0x7f9f0818ab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:50.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:50.788 7f9f0787e700 1 -- 172.21.2.112:0/7576505 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f08188a90 con 0x7f9f08187e50 2021-02-10T11:43:50.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:50.788 7f9f0787e700 1 --2- 172.21.2.112:0/7576505 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9f08187e50 0x7f9f08188250 secure :-1 s=READY pgs=2518 cs=0 l=1 rev1=1 rx=0x7f9f0000b530 tx=0x7f9f0000b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:50.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:50.788 7f9eeffff700 1 -- 172.21.2.112:0/7576505 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9f00006030 con 0x7f9f08187e50 2021-02-10T11:43:50.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:50.788 7f9eeffff700 1 -- 172.21.2.112:0/7576505 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f000062f0 con 0x7f9f08187e50 2021-02-10T11:43:50.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:50.788 7f9eeffff700 1 -- 172.21.2.112:0/7576505 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9f00005770 con 0x7f9f08187e50 2021-02-10T11:43:50.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:50.788 7f9f0d41d700 1 -- 172.21.2.112:0/7576505 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f08188a90 con 0x7f9f08187e50 2021-02-10T11:43:50.793 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:50.788 7f9f0d41d700 1 -- 172.21.2.112:0/7576505 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f08188a90 con 0x7f9f08187e50 2021-02-10T11:43:50.793 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:50.788 7f9eeffff700 1 -- 172.21.2.112:0/7576505 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9f000054f0 con 0x7f9f08187e50 2021-02-10T11:43:50.793 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:50.789 7f9eeffff700 1 --2- 172.21.2.112:0/7576505 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9ee8018730 0x7f9ee801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:50.793 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:50.789 7f9eeffff700 1 -- 172.21.2.112:0/7576505 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 4839+0+0 (secure 0 0 0) 0x7f9f000323a0 con 0x7f9f08187e50 2021-02-10T11:43:50.793 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:50.789 7f9f0707d700 1 --2- 172.21.2.112:0/7576505 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9ee8018730 0x7f9ee801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:50.793 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:50.789 7f9f0707d700 1 --2- 172.21.2.112:0/7576505 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9ee8018730 0x7f9ee801ac10 crc :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:50.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:50.819 7f9f0d41d700 1 -- 172.21.2.112:0/7576505 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f080ebc50 con 0x7f9f08187e50 2021-02-10T11:43:50.822 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:50.822 7f9eeffff700 1 -- 172.21.2.112:0/7576505 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9f00006b90 con 0x7f9f08187e50 2021-02-10T11:43:50.963 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:50.963 7f9f0d41d700 1 -- 172.21.2.112:0/7576505 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7f9f080f0b10 con 0x7f9f08187e50 2021-02-10T11:43:51.286 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:51.285 7f9eeffff700 1 -- 172.21.2.112:0/7576505 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v138) v1 ==== 97+0+0 (secure 0 0 0) 0x7f9f000054f0 con 0x7f9f08187e50 2021-02-10T11:43:51.329 INFO:teuthology.orchestra.run.gibba012.stderr: removed all blacklist entries 2021-02-10T11:43:51.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:51.330 7f9f0d41d700 1 -- 172.21.2.112:0/7576505 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9ee8018730 msgr2=0x7f9ee801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:51.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:51.330 7f9f0d41d700 1 --2- 172.21.2.112:0/7576505 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9ee8018730 0x7f9ee801ac10 crc :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:51.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:51.330 7f9f0d41d700 1 -- 172.21.2.112:0/7576505 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9f08187e50 msgr2=0x7f9f08188250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:51.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:51.330 7f9f0d41d700 1 --2- 172.21.2.112:0/7576505 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9f08187e50 0x7f9f08188250 secure :-1 s=READY pgs=2518 cs=0 l=1 rev1=1 rx=0x7f9f0000b530 tx=0x7f9f0000b8d0).stop 2021-02-10T11:43:51.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:51.330 7f9f0d41d700 1 -- 172.21.2.112:0/7576505 shutdown_connections 2021-02-10T11:43:51.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:51.330 7f9f0d41d700 1 --2- 172.21.2.112:0/7576505 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9ee8018730 0x7f9ee801ac10 unknown :-1 s=CLOSED pgs=1150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:51.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:51.330 7f9f0d41d700 1 --2- 172.21.2.112:0/7576505 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9f0818a720 0x7f9f0818ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:51.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:51.330 7f9f0d41d700 1 --2- 172.21.2.112:0/7576505 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9f08187e50 0x7f9f08188250 unknown :-1 s=CLOSED pgs=2518 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:51.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:51.330 7f9f0d41d700 1 --2- 172.21.2.112:0/7576505 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9f08189d00 0x7f9f0818a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:51.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:51.330 7f9f0d41d700 1 -- 172.21.2.112:0/7576505 >> 172.21.2.112:0/7576505 conn(0x7f9f080aed50 msgr2=0x7f9f0809c9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:51.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:51.331 7f9f0d41d700 1 -- 172.21.2.112:0/7576505 shutdown_connections 2021-02-10T11:43:51.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:51.331 7f9f0d41d700 1 -- 172.21.2.112:0/7576505 wait complete. 2021-02-10T11:43:51.339 INFO:tasks.cephfs.cephfs_test_case:['0', '1', '2', '3'] 2021-02-10T11:43:51.340 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth ls --format=json-pretty 2021-02-10T11:43:51.767 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:51.751 7f29ca702700 1 Processor -- start 2021-02-10T11:43:51.768 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:51.751 7f29ca702700 1 -- start start 2021-02-10T11:43:51.768 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:51.751 7f29ca702700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f29c40d0340 0x7f29c40ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:51.768 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:51.751 7f29ca702700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29c40f0b10 con 0x7f29c40d1870 2021-02-10T11:43:51.768 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:51.751 7f29ca702700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29c41b0be0 con 0x7f29c40d1c70 2021-02-10T11:43:51.768 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:51.751 7f29ca702700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29c40e5ed0 con 0x7f29c40d0340 2021-02-10T11:43:51.768 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:51.751 7f29c37fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f29c40d0340 0x7f29c40ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:51.768 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:51.751 7f29c3fff700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f29c40d1870 0x7f29c41b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:56830/0 (socket says 172.21.2.112:56830) 2021-02-10T11:43:51.768 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:51.751 7f29c3fff700 1 -- 172.21.2.112:0/1554528480 learned_addr learned my addr 172.21.2.112:0/1554528480 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:43:51.768 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:51.752 7f29c37fe700 1 -- 172.21.2.112:0/1554528480 >> v1:172.21.2.120:6789/0 conn(0x7f29c40d1c70 legacy=0x7f29c40efc60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:43:51.768 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:51.752 7f29c37fe700 1 -- 172.21.2.112:0/1554528480 >> v1:172.21.2.112:6789/0 conn(0x7f29c40d1870 legacy=0x7f29c41b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:51.768 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:51.752 7f29c37fe700 1 -- 172.21.2.112:0/1554528480 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29c40f0450 con 0x7f29c40d0340 2021-02-10T11:43:51.769 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:51.752 7f29c37fe700 1 --2- 172.21.2.112:0/1554528480 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f29c40d0340 0x7f29c40ef640 secure :-1 s=READY pgs=2520 cs=0 l=1 rev1=1 rx=0x7f29b8006650 tx=0x7f29b8009a80).ready entity=mon.0 client_cookie=f05aa41f5c335849 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:51.769 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:51.752 7f29c2ffd700 1 -- 172.21.2.112:0/1554528480 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f29b800c0c0 con 0x7f29c40d0340 2021-02-10T11:43:51.769 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:51.752 7f29c2ffd700 1 -- 172.21.2.112:0/1554528480 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29b800c340 con 0x7f29c40d0340 2021-02-10T11:43:51.769 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:51.752 7f29c2ffd700 1 -- 172.21.2.112:0/1554528480 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f29b800c860 con 0x7f29c40d0340 2021-02-10T11:43:51.769 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:51.752 7f29ca702700 1 -- 172.21.2.112:0/1554528480 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f29c40d0340 msgr2=0x7f29c40ef640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:51.769 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:51.752 7f29ca702700 1 --2- 172.21.2.112:0/1554528480 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f29c40d0340 0x7f29c40ef640 secure :-1 s=READY pgs=2520 cs=0 l=1 rev1=1 rx=0x7f29b8006650 tx=0x7f29b8009a80).stop 2021-02-10T11:43:51.769 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:51.753 7f29ca702700 1 -- 172.21.2.112:0/1554528480 shutdown_connections 2021-02-10T11:43:51.769 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:51.753 7f29ca702700 1 --2- 172.21.2.112:0/1554528480 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f29c40d0340 0x7f29c40ef640 unknown :-1 s=CLOSED pgs=2520 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:51.769 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:51.753 7f29ca702700 1 -- 172.21.2.112:0/1554528480 >> 172.21.2.112:0/1554528480 conn(0x7f29c40ce450 msgr2=0x7f29c40d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:51.769 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:51.753 7f29ca702700 1 -- 172.21.2.112:0/1554528480 shutdown_connections 2021-02-10T11:43:51.769 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:51.753 7f29ca702700 1 -- 172.21.2.112:0/1554528480 wait complete. 2021-02-10T11:43:51.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:51.767 7f29ca702700 1 Processor -- start 2021-02-10T11:43:51.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:51.768 7f29ca702700 1 -- start start 2021-02-10T11:43:51.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:51.768 7f29ca702700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f29c40c0060 0x7f29c40c2bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:51.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:51.768 7f29ca702700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f29c41c69b0 0x7f29c40c3210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:51.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:51.768 7f29ca702700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f29c40c3830 0x7f29c40b3970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:51.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:51.768 7f29ca702700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29c40b33f0 con 0x7f29c41c69b0 2021-02-10T11:43:51.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:51.768 7f29ca702700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29c40c3c30 con 0x7f29c40c3830 2021-02-10T11:43:51.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:51.768 7f29ca702700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29c40bfbc0 con 0x7f29c40c0060 2021-02-10T11:43:51.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:51.768 7f29c37fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f29c41c69b0 0x7f29c40c3210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:51.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:51.768 7f29c37fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f29c41c69b0 0x7f29c40c3210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:57374/0 (socket says 172.21.2.112:57374) 2021-02-10T11:43:51.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:51.768 7f29c37fe700 1 -- 172.21.2.112:0/664425494 learned_addr learned my addr 172.21.2.112:0/664425494 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:43:51.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:51.768 7f29c37fe700 1 -- 172.21.2.112:0/664425494 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f29c40c0060 msgr2=0x7f29c40c2bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:43:51.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:51.768 7f29c37fe700 1 --2- 172.21.2.112:0/664425494 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f29c40c0060 0x7f29c40c2bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:51.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:51.768 7f29c37fe700 1 -- 172.21.2.112:0/664425494 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f29c40c3830 msgr2=0x7f29c40b3970 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:43:51.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:51.768 7f29c37fe700 1 --2- 172.21.2.112:0/664425494 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f29c40c3830 0x7f29c40b3970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:51.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:51.768 7f29c37fe700 1 -- 172.21.2.112:0/664425494 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29c40b3f90 con 0x7f29c41c69b0 2021-02-10T11:43:51.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:51.769 7f29c37fe700 1 --2- 172.21.2.112:0/664425494 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f29c41c69b0 0x7f29c40c3210 secure :-1 s=READY pgs=2521 cs=0 l=1 rev1=1 rx=0x7f29b8009630 tx=0x7f29b800b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:51.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:51.769 7f29c0ff9700 1 -- 172.21.2.112:0/664425494 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f29b80093a0 con 0x7f29c41c69b0 2021-02-10T11:43:51.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:51.769 7f29c0ff9700 1 -- 172.21.2.112:0/664425494 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29b800bd70 con 0x7f29c41c69b0 2021-02-10T11:43:51.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:51.769 7f29c0ff9700 1 -- 172.21.2.112:0/664425494 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f29b80054d0 con 0x7f29c41c69b0 2021-02-10T11:43:51.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:51.769 7f29ca702700 1 -- 172.21.2.112:0/664425494 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29c40b3f90 con 0x7f29c41c69b0 2021-02-10T11:43:51.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:51.769 7f29ca702700 1 -- 172.21.2.112:0/664425494 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29c40b3f90 con 0x7f29c41c69b0 2021-02-10T11:43:51.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:51.770 7f29c0ff9700 1 -- 172.21.2.112:0/664425494 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f29b800a000 con 0x7f29c41c69b0 2021-02-10T11:43:51.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:51.770 7f29c0ff9700 1 --2- 172.21.2.112:0/664425494 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f29a40186c0 0x7f29a401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:51.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:51.770 7f29c0ff9700 1 -- 172.21.2.112:0/664425494 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f29b80324d0 con 0x7f29c41c69b0 2021-02-10T11:43:51.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:51.770 7f29c3fff700 1 --2- 172.21.2.112:0/664425494 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f29a40186c0 0x7f29a401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:51.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:51.771 7f29c3fff700 1 --2- 172.21.2.112:0/664425494 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f29a40186c0 0x7f29a401abc0 crc :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:51.802 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:51.801 7f29ca702700 1 -- 172.21.2.112:0/664425494 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29c40f06f0 con 0x7f29c41c69b0 2021-02-10T11:43:51.804 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:51.804 7f29c0ff9700 1 -- 172.21.2.112:0/664425494 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f29b800a330 con 0x7f29c41c69b0 2021-02-10T11:43:51.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:51.936 7f29ca702700 1 -- 172.21.2.112:0/664425494 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f29c40f06f0 con 0x7f29c41c69b0 2021-02-10T11:43:51.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:51.937 7f29c0ff9700 1 -- 172.21.2.112:0/664425494 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v82) v1 ==== 80+0+4513 (secure 0 0 0) 0x7f29b800a000 con 0x7f29c41c69b0 2021-02-10T11:43:51.938 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:43:51.938 INFO:teuthology.orchestra.run.gibba012.stdout:{ 2021-02-10T11:43:51.938 INFO:teuthology.orchestra.run.gibba012.stdout: "auth_dump": [ 2021-02-10T11:43:51.938 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:51.938 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "mds.a", 2021-02-10T11:43:51.938 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCKvyNgPkV4AxAAEmVNaghdccP6lWd8hvQnxw==", 2021-02-10T11:43:51.938 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:43:51.938 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:43:51.939 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:43:51.939 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:43:51.939 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:43:51.939 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:43:51.939 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:51.939 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:51.939 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "mds.b", 2021-02-10T11:43:51.939 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCKvyNguxONLhAAHIpry+R89MAwdhFJyEhVCw==", 2021-02-10T11:43:51.939 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:43:51.939 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:43:51.939 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:43:51.939 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:43:51.939 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:43:51.940 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:43:51.940 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:51.940 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:51.940 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.0", 2021-02-10T11:43:51.940 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCSvyNgXcKPLhAAP/Bj2F+06DCtPCzQDNq+cQ==", 2021-02-10T11:43:51.940 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:43:51.940 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:43:51.940 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:43:51.940 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:43:51.940 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:43:51.940 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:51.940 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:51.940 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.1", 2021-02-10T11:43:51.941 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCVvyNghfbcLhAA66PrRwR2M7o6VTF5UhU/Fg==", 2021-02-10T11:43:51.941 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:43:51.941 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:43:51.941 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:43:51.941 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:43:51.941 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:43:51.941 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:51.941 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:51.941 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.2", 2021-02-10T11:43:51.941 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCYvyNgwiPqLhAA66BVkRsgaUlGpAsQ15Nf8Q==", 2021-02-10T11:43:51.941 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:43:51.941 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:43:51.941 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:43:51.942 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:43:51.942 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:43:51.942 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:51.942 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:51.942 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.3", 2021-02-10T11:43:51.942 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCbvyNgqBZrLxAAERPe/1L3SXAEzI3d9ReHHg==", 2021-02-10T11:43:51.942 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:43:51.942 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:43:51.942 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:43:51.942 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:43:51.942 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:43:51.942 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:51.942 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:51.943 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.4", 2021-02-10T11:43:51.943 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCkvyNgDQsxIhAAHmVjVYnx20N4SDmKqvbcuA==", 2021-02-10T11:43:51.943 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:43:51.943 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:43:51.943 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:43:51.943 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:43:51.943 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:43:51.943 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:51.943 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:51.943 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.5", 2021-02-10T11:43:51.943 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCnvyNg8ITFIhAADd4aKybAV7btoGPx3kCg3A==", 2021-02-10T11:43:51.943 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:43:51.943 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:43:51.943 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:43:51.944 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:43:51.944 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:43:51.944 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:51.944 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:51.944 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.6", 2021-02-10T11:43:51.944 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCqvyNgxOhDIxAAzXixEa/nlDc1xXrg0aeLwg==", 2021-02-10T11:43:51.944 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:43:51.944 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:43:51.944 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:43:51.944 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:43:51.944 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:43:51.944 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:51.944 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:51.944 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.7", 2021-02-10T11:43:51.945 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCtvyNgxSqGIxAACxJQlwP7DpiVVCZHA3zn1A==", 2021-02-10T11:43:51.945 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:43:51.945 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:43:51.945 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:43:51.945 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:43:51.945 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:43:51.945 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:51.945 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:51.945 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.0", 2021-02-10T11:43:51.945 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCLvyNgDkNiHRAAygU8Za4PBusTqCgwAYWTiQ==", 2021-02-10T11:43:51.945 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:43:51.945 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:43:51.945 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow r", 2021-02-10T11:43:51.945 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:43:51.946 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:43:51.946 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:51.946 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:51.946 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.1", 2021-02-10T11:43:51.946 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCLvyNgxmLqMxAA1MlSDJ9LH9u4f2sL7mm8/Q==", 2021-02-10T11:43:51.946 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:43:51.946 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:43:51.946 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow r", 2021-02-10T11:43:51.946 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:43:51.946 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:43:51.946 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:51.946 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:51.947 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.2", 2021-02-10T11:43:51.947 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCMvyNgp8veDhAAniSVHYPtwwTog/pjbaRQmw==", 2021-02-10T11:43:51.947 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:43:51.947 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:43:51.947 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow r", 2021-02-10T11:43:51.947 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:43:51.947 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:43:51.947 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:51.947 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:51.947 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.3", 2021-02-10T11:43:51.947 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCMvyNgQEVoJRAApHcWdm1sldIxu+heyD4HdQ==", 2021-02-10T11:43:51.947 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:43:51.947 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:43:51.948 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow r", 2021-02-10T11:43:51.948 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:43:51.948 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:43:51.948 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:51.948 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:51.948 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.admin", 2021-02-10T11:43:51.948 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCGvyNg7jjJBRAAtot+ypUZOiTo1GbY+oVhVw==", 2021-02-10T11:43:51.948 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:43:51.948 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow *", 2021-02-10T11:43:51.948 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:43:51.948 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:43:51.948 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:43:51.948 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:43:51.948 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:51.949 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:51.949 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "mgr.x", 2021-02-10T11:43:51.949 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCJvyNgTPVwKBAAjduiBE8+RSXBv82tdPecWg==", 2021-02-10T11:43:51.949 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:43:51.949 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow *", 2021-02-10T11:43:51.949 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow profile mgr", 2021-02-10T11:43:51.949 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:43:51.949 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:43:51.949 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:43:51.949 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:51.949 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "mgr.y", 2021-02-10T11:43:51.949 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCJvyNgQ1tTDhAASwBssDy+Rz31ujw8fNlWow==", 2021-02-10T11:43:51.949 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:43:51.949 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow *", 2021-02-10T11:43:51.950 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow profile mgr", 2021-02-10T11:43:51.950 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:43:51.950 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:43:51.950 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:43:51.950 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:43:51.950 INFO:teuthology.orchestra.run.gibba012.stdout:} 2021-02-10T11:43:51.951 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:51.938 7f29ca702700 1 -- 172.21.2.112:0/664425494 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f29a40186c0 msgr2=0x7f29a401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:51.951 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:51.938 7f29ca702700 1 --2- 172.21.2.112:0/664425494 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f29a40186c0 0x7f29a401abc0 crc :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:51.951 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:51.938 7f29ca702700 1 -- 172.21.2.112:0/664425494 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f29c41c69b0 msgr2=0x7f29c40c3210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:51.951 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:51.938 7f29ca702700 1 --2- 172.21.2.112:0/664425494 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f29c41c69b0 0x7f29c40c3210 secure :-1 s=READY pgs=2521 cs=0 l=1 rev1=1 rx=0x7f29b8009630 tx=0x7f29b800b980).stop 2021-02-10T11:43:51.951 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:51.939 7f29ca702700 1 -- 172.21.2.112:0/664425494 shutdown_connections 2021-02-10T11:43:51.951 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:51.939 7f29ca702700 1 --2- 172.21.2.112:0/664425494 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f29a40186c0 0x7f29a401abc0 unknown :-1 s=CLOSED pgs=1151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:51.952 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:51.939 7f29ca702700 1 --2- 172.21.2.112:0/664425494 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f29c40c3830 0x7f29c40b3970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:51.952 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:51.939 7f29ca702700 1 --2- 172.21.2.112:0/664425494 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f29c41c69b0 0x7f29c40c3210 unknown :-1 s=CLOSED pgs=2521 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:51.952 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:51.939 7f29ca702700 1 --2- 172.21.2.112:0/664425494 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f29c40c0060 0x7f29c40c2bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:51.952 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:51.939 7f29ca702700 1 -- 172.21.2.112:0/664425494 >> 172.21.2.112:0/664425494 conn(0x7f29c410a820 msgr2=0x7f29c4108150 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:51.952 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:51.939 7f29ca702700 1 -- 172.21.2.112:0/664425494 shutdown_connections 2021-02-10T11:43:51.952 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:51.939 7f29ca702700 1 -- 172.21.2.112:0/664425494 wait complete. 2021-02-10T11:43:51.952 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:43:52.385 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.367 7f0b1f19f700 1 Processor -- start 2021-02-10T11:43:52.385 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.367 7f0b1f19f700 1 -- start start 2021-02-10T11:43:52.385 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.368 7f0b1f19f700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f0b180d1870 0x7f0b181b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:52.385 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.368 7f0b1f19f700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b180f0b10 con 0x7f0b180f0650 2021-02-10T11:43:52.385 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.368 7f0b1f19f700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b180d0310 con 0x7f0b181b0a10 2021-02-10T11:43:52.386 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.368 7f0b1f19f700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b180d0530 con 0x7f0b180d1870 2021-02-10T11:43:52.386 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.368 7f0b1d7bc700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f0b180f0650 0x7f0b180cfb20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:56844/0 (socket says 172.21.2.112:56844) 2021-02-10T11:43:52.386 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.368 7f0b1d7bc700 1 -- 172.21.2.112:0/202424210 learned_addr learned my addr 172.21.2.112:0/202424210 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:43:52.386 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.368 7f0b1cfbb700 1 --2- 172.21.2.112:0/202424210 >> v2:172.21.2.120:3300/0 conn(0x7f0b180d1870 0x7f0b181b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:52.386 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.368 7f0b177fe700 1 -- 172.21.2.112:0/202424210 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4199234152 0 0) 0x7f0b10000d30 con 0x7f0b180f0650 2021-02-10T11:43:52.386 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.368 7f0b177fe700 1 -- 172.21.2.112:0/202424210 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b04002fe0 con 0x7f0b180f0650 2021-02-10T11:43:52.386 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.368 7f0b177fe700 1 -- 172.21.2.112:0/202424210 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2201467730 0 0) 0x7f0b10001460 con 0x7f0b180f0650 2021-02-10T11:43:52.386 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.368 7f0b177fe700 1 -- 172.21.2.112:0/202424210 >> v2:172.21.2.120:3300/0 conn(0x7f0b180d1870 msgr2=0x7f0b181b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:52.386 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.368 7f0b177fe700 1 --2- 172.21.2.112:0/202424210 >> v2:172.21.2.120:3300/0 conn(0x7f0b180d1870 0x7f0b181b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:52.386 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.368 7f0b177fe700 1 -- 172.21.2.112:0/202424210 >> v1:172.21.2.120:6789/0 conn(0x7f0b181b0a10 legacy=0x7f0b180cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:52.386 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.368 7f0b177fe700 1 -- 172.21.2.112:0/202424210 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b180d1c70 con 0x7f0b180f0650 2021-02-10T11:43:52.386 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.368 7f0b177fe700 1 -- 172.21.2.112:0/202424210 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f0b10001980 con 0x7f0b180f0650 2021-02-10T11:43:52.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.368 7f0b177fe700 1 -- 172.21.2.112:0/202424210 >> v1:172.21.2.112:6789/0 conn(0x7f0b180f0650 legacy=0x7f0b180cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:52.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.368 7f0b177fe700 1 --2- 172.21.2.112:0/202424210 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0b04003aa0 0x7f0b04005ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:52.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.369 7f0b177fe700 1 --2- 172.21.2.112:0/202424210 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0b04006770 0x7f0b04008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:52.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.369 7f0b177fe700 1 --2- 172.21.2.112:0/202424210 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0b04009590 0x7f0b0400bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:52.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.369 7f0b177fe700 1 -- 172.21.2.112:0/202424210 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b0400c300 con 0x7f0b04009590 2021-02-10T11:43:52.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.369 7f0b177fe700 1 -- 172.21.2.112:0/202424210 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b0400c520 con 0x7f0b04003aa0 2021-02-10T11:43:52.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.369 7f0b177fe700 1 -- 172.21.2.112:0/202424210 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b04050750 con 0x7f0b04006770 2021-02-10T11:43:52.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.369 7f0b1d7bc700 1 --2- 172.21.2.112:0/202424210 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0b04009590 0x7f0b0400bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:52.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.369 7f0b1cfbb700 1 --2- 172.21.2.112:0/202424210 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0b04003aa0 0x7f0b04005ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:52.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.369 7f0b17fff700 1 --2- 172.21.2.112:0/202424210 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0b04006770 0x7f0b04008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:52.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.369 7f0b1d7bc700 1 -- 172.21.2.112:0/202424210 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0b04006770 msgr2=0x7f0b04008c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:52.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.369 7f0b1d7bc700 1 --2- 172.21.2.112:0/202424210 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0b04006770 0x7f0b04008c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:52.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.369 7f0b1d7bc700 1 -- 172.21.2.112:0/202424210 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0b04003aa0 msgr2=0x7f0b04005ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:52.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.369 7f0b1d7bc700 1 --2- 172.21.2.112:0/202424210 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0b04003aa0 0x7f0b04005ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:52.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.369 7f0b1d7bc700 1 -- 172.21.2.112:0/202424210 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b04050980 con 0x7f0b04009590 2021-02-10T11:43:52.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.369 7f0b1cfbb700 1 --2- 172.21.2.112:0/202424210 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0b04003aa0 0x7f0b04005ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:43:52.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.369 7f0b1d7bc700 1 -- 172.21.2.112:0/202424210 reap_dead start 2021-02-10T11:43:52.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.369 7f0b1d7bc700 1 --2- 172.21.2.112:0/202424210 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0b04009590 0x7f0b0400bab0 secure :-1 s=READY pgs=2523 cs=0 l=1 rev1=1 rx=0x7f0b10001c80 tx=0x7f0b1000a310).ready entity=mon.0 client_cookie=7317830d4cdd88b4 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:52.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.369 7f0b177fe700 1 -- 172.21.2.112:0/202424210 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0b100017f0 con 0x7f0b04009590 2021-02-10T11:43:52.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.369 7f0b177fe700 1 -- 172.21.2.112:0/202424210 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b100070c0 con 0x7f0b04009590 2021-02-10T11:43:52.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.369 7f0b177fe700 1 -- 172.21.2.112:0/202424210 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0b10007510 con 0x7f0b04009590 2021-02-10T11:43:52.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.369 7f0b1f19f700 1 -- 172.21.2.112:0/202424210 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0b04009590 msgr2=0x7f0b0400bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:52.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.369 7f0b1f19f700 1 --2- 172.21.2.112:0/202424210 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0b04009590 0x7f0b0400bab0 secure :-1 s=READY pgs=2523 cs=0 l=1 rev1=1 rx=0x7f0b10001c80 tx=0x7f0b1000a310).stop 2021-02-10T11:43:52.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.370 7f0b1f19f700 1 -- 172.21.2.112:0/202424210 shutdown_connections 2021-02-10T11:43:52.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.370 7f0b1f19f700 1 --2- 172.21.2.112:0/202424210 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0b04009590 0x7f0b0400bab0 unknown :-1 s=CLOSED pgs=2523 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:52.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.370 7f0b1f19f700 1 -- 172.21.2.112:0/202424210 >> 172.21.2.112:0/202424210 conn(0x7f0b180ce450 msgr2=0x7f0b180d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:52.389 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.370 7f0b1f19f700 1 -- 172.21.2.112:0/202424210 shutdown_connections 2021-02-10T11:43:52.389 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.370 7f0b1f19f700 1 -- 172.21.2.112:0/202424210 wait complete. 2021-02-10T11:43:52.389 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.385 7f0b1f19f700 1 Processor -- start 2021-02-10T11:43:52.389 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.385 7f0b1f19f700 1 -- start start 2021-02-10T11:43:52.390 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.385 7f0b1f19f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0b180aa700 0x7f0b18090ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:52.390 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.385 7f0b1f19f700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0b180aab00 0x7f0b180910e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:52.390 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.385 7f0b1f19f700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0b180ecb10 0x7f0b180ecf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:52.390 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.385 7f0b1f19f700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b18090540 con 0x7f0b180aa700 2021-02-10T11:43:52.390 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.385 7f0b1f19f700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b180c0170 con 0x7f0b180aab00 2021-02-10T11:43:52.390 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.385 7f0b1f19f700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b180ed530 con 0x7f0b180ecb10 2021-02-10T11:43:52.390 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.385 7f0b1cfbb700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0b180aa700 0x7f0b18090ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:52.391 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.385 7f0b1cfbb700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0b180aa700 0x7f0b18090ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:57396/0 (socket says 172.21.2.112:57396) 2021-02-10T11:43:52.391 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.385 7f0b1cfbb700 1 -- 172.21.2.112:0/1897287059 learned_addr learned my addr 172.21.2.112:0/1897287059 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:43:52.391 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.385 7f0b17fff700 1 --2- 172.21.2.112:0/1897287059 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0b180aab00 0x7f0b180910e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:52.391 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.385 7f0b1d7bc700 1 --2- 172.21.2.112:0/1897287059 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0b180ecb10 0x7f0b180ecf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:52.391 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.386 7f0b1cfbb700 1 -- 172.21.2.112:0/1897287059 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0b180ecb10 msgr2=0x7f0b180ecf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:52.391 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.386 7f0b1cfbb700 1 --2- 172.21.2.112:0/1897287059 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0b180ecb10 0x7f0b180ecf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:52.391 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.386 7f0b1cfbb700 1 -- 172.21.2.112:0/1897287059 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0b180aab00 msgr2=0x7f0b180910e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:52.391 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.386 7f0b1cfbb700 1 --2- 172.21.2.112:0/1897287059 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0b180aab00 0x7f0b180910e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:52.391 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.386 7f0b1cfbb700 1 -- 172.21.2.112:0/1897287059 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b180ed750 con 0x7f0b180aa700 2021-02-10T11:43:52.391 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.386 7f0b1cfbb700 1 --2- 172.21.2.112:0/1897287059 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0b180aa700 0x7f0b18090ac0 secure :-1 s=READY pgs=2524 cs=0 l=1 rev1=1 rx=0x7f0b0000b5e0 tx=0x7f0b0000b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:52.391 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.386 7f0b157fa700 1 -- 172.21.2.112:0/1897287059 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0b0000bd70 con 0x7f0b180aa700 2021-02-10T11:43:52.391 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.386 7f0b157fa700 1 -- 172.21.2.112:0/1897287059 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b00000c90 con 0x7f0b180aa700 2021-02-10T11:43:52.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.386 7f0b1f19f700 1 -- 172.21.2.112:0/1897287059 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b180ed750 con 0x7f0b180aa700 2021-02-10T11:43:52.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.386 7f0b157fa700 1 -- 172.21.2.112:0/1897287059 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0b000022b0 con 0x7f0b180aa700 2021-02-10T11:43:52.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.386 7f0b1f19f700 1 -- 172.21.2.112:0/1897287059 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b1808d110 con 0x7f0b180aa700 2021-02-10T11:43:52.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.386 7f0b157fa700 1 -- 172.21.2.112:0/1897287059 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0b00006740 con 0x7f0b180aa700 2021-02-10T11:43:52.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.386 7f0b157fa700 1 --2- 172.21.2.112:0/1897287059 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0afc0186d0 0x7f0afc01abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:52.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.386 7f0b157fa700 1 -- 172.21.2.112:0/1897287059 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f0b000324b0 con 0x7f0b180aa700 2021-02-10T11:43:52.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.387 7f0b17fff700 1 --2- 172.21.2.112:0/1897287059 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0afc0186d0 0x7f0afc01abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:52.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.387 7f0b17fff700 1 --2- 172.21.2.112:0/1897287059 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0afc0186d0 0x7f0afc01abd0 crc :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:52.417 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.417 7f0b1f19f700 1 -- 172.21.2.112:0/1897287059 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b180ed9f0 con 0x7f0b180aa700 2021-02-10T11:43:52.420 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.420 7f0b157fa700 1 -- 172.21.2.112:0/1897287059 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0b00000940 con 0x7f0b180aa700 2021-02-10T11:43:52.549 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.549 7f0b1f19f700 1 -- 172.21.2.112:0/1897287059 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0b180ed9f0 con 0x7f0b180aa700 2021-02-10T11:43:52.550 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.549 7f0b157fa700 1 -- 172.21.2.112:0/1897287059 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v138) v1 ==== 74+0+11154 (secure 0 0 0) 0x7f0b00006740 con 0x7f0b180aa700 2021-02-10T11:43:52.550 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:43:52.550 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":138,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:43:51.284026","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":132,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":132,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":132,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":132,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:43:52.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.551 7f0b1f19f700 1 -- 172.21.2.112:0/1897287059 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0afc0186d0 msgr2=0x7f0afc01abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:52.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.551 7f0b1f19f700 1 --2- 172.21.2.112:0/1897287059 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0afc0186d0 0x7f0afc01abd0 crc :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:52.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.551 7f0b1f19f700 1 -- 172.21.2.112:0/1897287059 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0b180aa700 msgr2=0x7f0b18090ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:52.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.551 7f0b1f19f700 1 --2- 172.21.2.112:0/1897287059 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0b180aa700 0x7f0b18090ac0 secure :-1 s=READY pgs=2524 cs=0 l=1 rev1=1 rx=0x7f0b0000b5e0 tx=0x7f0b0000b980).stop 2021-02-10T11:43:52.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.551 7f0b1f19f700 1 -- 172.21.2.112:0/1897287059 shutdown_connections 2021-02-10T11:43:52.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.551 7f0b1f19f700 1 --2- 172.21.2.112:0/1897287059 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0afc0186d0 0x7f0afc01abd0 unknown :-1 s=CLOSED pgs=1152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:52.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.551 7f0b1f19f700 1 --2- 172.21.2.112:0/1897287059 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0b180aab00 0x7f0b180910e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:52.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.551 7f0b1f19f700 1 --2- 172.21.2.112:0/1897287059 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0b180ecb10 0x7f0b180ecf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:52.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.551 7f0b1f19f700 1 --2- 172.21.2.112:0/1897287059 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0b180aa700 0x7f0b18090ac0 unknown :-1 s=CLOSED pgs=2524 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:52.553 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.551 7f0b1f19f700 1 -- 172.21.2.112:0/1897287059 >> 172.21.2.112:0/1897287059 conn(0x7f0b180b7e70 msgr2=0x7f0b180c3850 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:52.553 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.551 7f0b1f19f700 1 -- 172.21.2.112:0/1897287059 shutdown_connections 2021-02-10T11:43:52.553 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.551 7f0b1f19f700 1 -- 172.21.2.112:0/1897287059 wait complete. 2021-02-10T11:43:52.560 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:13:57.508758', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2021-02-10T11:43:52.561 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:43:52.993 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.976 7f1f2b5a1700 1 Processor -- start 2021-02-10T11:43:52.993 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.976 7f1f2b5a1700 1 -- start start 2021-02-10T11:43:52.993 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.976 7f1f2b5a1700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f1f240d1870 0x7f1f241b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:52.993 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.976 7f1f2b5a1700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1f240cff70 0x7f1f24109130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:52.993 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.976 7f1f2b5a1700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f24109a60 con 0x7f1f241b0ae0 2021-02-10T11:43:52.993 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.976 7f1f2b5a1700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f24109750 con 0x7f1f240d1870 2021-02-10T11:43:52.993 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.976 7f1f2b5a1700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f24088070 con 0x7f1f240cff70 2021-02-10T11:43:52.994 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.977 7f1f293bd700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f1f240d1870 0x7f1f241b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:52.994 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.977 7f1f28bbc700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f1f241b0ae0 0x7f1f24108940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:56864/0 (socket says 172.21.2.112:56864) 2021-02-10T11:43:52.994 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.977 7f1f28bbc700 1 -- 172.21.2.112:0/1139966090 learned_addr learned my addr 172.21.2.112:0/1139966090 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:43:52.994 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.977 7f1f293bd700 1 -- 172.21.2.112:0/1139966090 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1f240cff70 msgr2=0x7f1f24109130 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:43:52.994 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.977 7f1f293bd700 1 --2- 172.21.2.112:0/1139966090 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1f240cff70 0x7f1f24109130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:52.994 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.977 7f1f293bd700 1 -- 172.21.2.112:0/1139966090 >> v1:172.21.2.112:6789/0 conn(0x7f1f241b0ae0 legacy=0x7f1f24108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:52.994 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.977 7f1f293bd700 1 -- 172.21.2.112:0/1139966090 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f240d1c70 con 0x7f1f240d1870 2021-02-10T11:43:52.994 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.977 7f1f293bd700 1 --2- 172.21.2.112:0/1139966090 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1f240d1870 0x7f1f241b04c0 secure :-1 s=READY pgs=2526 cs=0 l=1 rev1=1 rx=0x7f1f18006650 tx=0x7f1f18009a80).ready entity=mon.0 client_cookie=ce35df9e2d03e1ae server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:52.994 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.977 7f1f17fff700 1 -- 172.21.2.112:0/1139966090 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1f1800c0c0 con 0x7f1f240d1870 2021-02-10T11:43:52.994 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.977 7f1f17fff700 1 -- 172.21.2.112:0/1139966090 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f1800c340 con 0x7f1f240d1870 2021-02-10T11:43:52.994 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.977 7f1f17fff700 1 -- 172.21.2.112:0/1139966090 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1f1800c860 con 0x7f1f240d1870 2021-02-10T11:43:52.995 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.977 7f1f2b5a1700 1 -- 172.21.2.112:0/1139966090 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1f240d1870 msgr2=0x7f1f241b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:52.995 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.977 7f1f2b5a1700 1 --2- 172.21.2.112:0/1139966090 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1f240d1870 0x7f1f241b04c0 secure :-1 s=READY pgs=2526 cs=0 l=1 rev1=1 rx=0x7f1f18006650 tx=0x7f1f18009a80).stop 2021-02-10T11:43:52.995 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.978 7f1f2b5a1700 1 -- 172.21.2.112:0/1139966090 shutdown_connections 2021-02-10T11:43:52.995 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.978 7f1f2b5a1700 1 --2- 172.21.2.112:0/1139966090 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1f240cff70 0x7f1f24109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:52.995 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.978 7f1f2b5a1700 1 --2- 172.21.2.112:0/1139966090 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1f240d1870 0x7f1f241b04c0 unknown :-1 s=CLOSED pgs=2526 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:52.995 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.978 7f1f2b5a1700 1 -- 172.21.2.112:0/1139966090 >> 172.21.2.112:0/1139966090 conn(0x7f1f240ce420 msgr2=0x7f1f240d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:52.995 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.978 7f1f2b5a1700 1 -- 172.21.2.112:0/1139966090 shutdown_connections 2021-02-10T11:43:52.995 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.978 7f1f2b5a1700 1 -- 172.21.2.112:0/1139966090 wait complete. 2021-02-10T11:43:52.996 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.993 7f1f2b5a1700 1 Processor -- start 2021-02-10T11:43:52.996 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.993 7f1f2b5a1700 1 -- start start 2021-02-10T11:43:52.997 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.993 7f1f2b5a1700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1f240c3820 0x7f1f240a7950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:52.997 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.993 7f1f2b5a1700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1f241a44c0 0x7f1f241a48c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:52.997 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.993 7f1f2b5a1700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1f240863d0 0x7f1f240867d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:52.997 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.993 7f1f2b5a1700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f24085ed0 con 0x7f1f240863d0 2021-02-10T11:43:52.997 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.993 7f1f2b5a1700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f240c0740 con 0x7f1f240c3820 2021-02-10T11:43:52.997 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.993 7f1f2b5a1700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f24086df0 con 0x7f1f241a44c0 2021-02-10T11:43:52.997 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.994 7f1f29bbe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1f240863d0 0x7f1f240867d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:52.997 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.994 7f1f29bbe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1f240863d0 0x7f1f240867d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:57404/0 (socket says 172.21.2.112:57404) 2021-02-10T11:43:52.997 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.994 7f1f29bbe700 1 -- 172.21.2.112:0/4086646473 learned_addr learned my addr 172.21.2.112:0/4086646473 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:43:52.997 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.994 7f1f29bbe700 1 -- 172.21.2.112:0/4086646473 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1f241a44c0 msgr2=0x7f1f241a48c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:43:52.997 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.994 7f1f29bbe700 1 --2- 172.21.2.112:0/4086646473 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1f241a44c0 0x7f1f241a48c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:52.997 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.994 7f1f29bbe700 1 -- 172.21.2.112:0/4086646473 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1f240c3820 msgr2=0x7f1f240a7950 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:43:52.997 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.994 7f1f29bbe700 1 --2- 172.21.2.112:0/4086646473 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1f240c3820 0x7f1f240a7950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:52.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.994 7f1f29bbe700 1 -- 172.21.2.112:0/4086646473 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f24087010 con 0x7f1f240863d0 2021-02-10T11:43:52.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.994 7f1f29bbe700 1 --2- 172.21.2.112:0/4086646473 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1f240863d0 0x7f1f240867d0 secure :-1 s=READY pgs=2527 cs=0 l=1 rev1=1 rx=0x7f1f1c009960 tx=0x7f1f1c00bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:52.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.994 7f1f15ffb700 1 -- 172.21.2.112:0/4086646473 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1f1c003030 con 0x7f1f240863d0 2021-02-10T11:43:52.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.994 7f1f15ffb700 1 -- 172.21.2.112:0/4086646473 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f1c001310 con 0x7f1f240863d0 2021-02-10T11:43:52.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.994 7f1f2b5a1700 1 -- 172.21.2.112:0/4086646473 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f24087010 con 0x7f1f240863d0 2021-02-10T11:43:52.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.994 7f1f2b5a1700 1 -- 172.21.2.112:0/4086646473 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f24083000 con 0x7f1f240863d0 2021-02-10T11:43:52.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.994 7f1f15ffb700 1 -- 172.21.2.112:0/4086646473 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1f1c001830 con 0x7f1f240863d0 2021-02-10T11:43:52.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.995 7f1f15ffb700 1 -- 172.21.2.112:0/4086646473 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1f1c0014f0 con 0x7f1f240863d0 2021-02-10T11:43:52.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.995 7f1f15ffb700 1 --2- 172.21.2.112:0/4086646473 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1f040186c0 0x7f1f0401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:52.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.995 7f1f15ffb700 1 -- 172.21.2.112:0/4086646473 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f1f1c032630 con 0x7f1f240863d0 2021-02-10T11:43:52.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.995 7f1f293bd700 1 --2- 172.21.2.112:0/4086646473 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1f040186c0 0x7f1f0401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:52.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:52.996 7f1f293bd700 1 --2- 172.21.2.112:0/4086646473 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1f040186c0 0x7f1f0401abc0 crc :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:53.027 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:53.027 7f1f2b5a1700 1 -- 172.21.2.112:0/4086646473 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f24085e50 con 0x7f1f240863d0 2021-02-10T11:43:53.030 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:53.030 7f1f15ffb700 1 -- 172.21.2.112:0/4086646473 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1f1c000f40 con 0x7f1f240863d0 2021-02-10T11:43:53.159 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:53.159 7f1f2b5a1700 1 -- 172.21.2.112:0/4086646473 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f1f24085e50 con 0x7f1f240863d0 2021-02-10T11:43:53.160 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:53.159 7f1f15ffb700 1 -- 172.21.2.112:0/4086646473 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v138) v1 ==== 92+0+10 (secure 0 0 0) 0x7f1f1c001340 con 0x7f1f240863d0 2021-02-10T11:43:53.160 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:43:53.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:53.161 7f1f2b5a1700 1 -- 172.21.2.112:0/4086646473 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1f040186c0 msgr2=0x7f1f0401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:53.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:53.161 7f1f2b5a1700 1 --2- 172.21.2.112:0/4086646473 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1f040186c0 0x7f1f0401abc0 crc :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:53.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:53.161 7f1f2b5a1700 1 -- 172.21.2.112:0/4086646473 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1f240863d0 msgr2=0x7f1f240867d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:53.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:53.161 7f1f2b5a1700 1 --2- 172.21.2.112:0/4086646473 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1f240863d0 0x7f1f240867d0 secure :-1 s=READY pgs=2527 cs=0 l=1 rev1=1 rx=0x7f1f1c009960 tx=0x7f1f1c00bcf0).stop 2021-02-10T11:43:53.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:53.161 7f1f2b5a1700 1 -- 172.21.2.112:0/4086646473 shutdown_connections 2021-02-10T11:43:53.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:53.161 7f1f2b5a1700 1 --2- 172.21.2.112:0/4086646473 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1f040186c0 0x7f1f0401abc0 unknown :-1 s=CLOSED pgs=1153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:53.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:53.161 7f1f2b5a1700 1 --2- 172.21.2.112:0/4086646473 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1f240863d0 0x7f1f240867d0 unknown :-1 s=CLOSED pgs=2527 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:53.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:53.161 7f1f2b5a1700 1 --2- 172.21.2.112:0/4086646473 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1f241a44c0 0x7f1f241a48c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:53.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:53.161 7f1f2b5a1700 1 --2- 172.21.2.112:0/4086646473 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1f240c3820 0x7f1f240a7950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:53.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:53.161 7f1f2b5a1700 1 -- 172.21.2.112:0/4086646473 >> 172.21.2.112:0/4086646473 conn(0x7f1f240872c0 msgr2=0x7f1f24095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:53.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:53.161 7f1f2b5a1700 1 -- 172.21.2.112:0/4086646473 shutdown_connections 2021-02-10T11:43:53.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:53.161 7f1f2b5a1700 1 -- 172.21.2.112:0/4086646473 wait complete. 2021-02-10T11:43:53.170 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --format=json-pretty osd lspools 2021-02-10T11:43:53.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:53.582 7f6c2dfb6700 1 Processor -- start 2021-02-10T11:43:53.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:53.582 7f6c2dfb6700 1 -- start start 2021-02-10T11:43:53.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:53.582 7f6c2dfb6700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f6c280d1870 0x7f6c281b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:53.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:53.582 7f6c2dfb6700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6c280f0650 0x7f6c280cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:53.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:53.582 7f6c2dfb6700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c280f0b10 con 0x7f6c281b0a10 2021-02-10T11:43:53.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:53.582 7f6c2dfb6700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c280e5ed0 con 0x7f6c280d1870 2021-02-10T11:43:53.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:53.582 7f6c2dfb6700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c280d1c70 con 0x7f6c280f0650 2021-02-10T11:43:53.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:53.583 7f6c277fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f6c280d1870 0x7f6c281b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:53.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:53.583 7f6c277fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f6c280d1870 0x7f6c281b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:57412/0 (socket says 172.21.2.112:57412) 2021-02-10T11:43:53.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:53.583 7f6c277fe700 1 -- 172.21.2.112:0/1679493093 learned_addr learned my addr 172.21.2.112:0/1679493093 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:43:53.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:53.583 7f6c27fff700 1 --2- 172.21.2.112:0/1679493093 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6c280f0650 0x7f6c280cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:53.600 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:53.583 7f6c277fe700 1 -- 172.21.2.112:0/1679493093 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6c280f0650 msgr2=0x7f6c280cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:53.600 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:53.583 7f6c277fe700 1 --2- 172.21.2.112:0/1679493093 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6c280f0650 0x7f6c280cfb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:53.600 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:53.583 7f6c277fe700 1 -- 172.21.2.112:0/1679493093 >> v1:172.21.2.120:6789/0 conn(0x7f6c281b0a10 legacy=0x7f6c280cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:53.600 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:53.583 7f6c277fe700 1 -- 172.21.2.112:0/1679493093 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c280d1e90 con 0x7f6c280d1870 2021-02-10T11:43:53.600 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:53.583 7f6c27fff700 1 --2- 172.21.2.112:0/1679493093 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6c280f0650 0x7f6c280cfb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:43:53.600 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:53.583 7f6c277fe700 1 --2- 172.21.2.112:0/1679493093 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6c280d1870 0x7f6c281b03f0 secure :-1 s=READY pgs=2528 cs=0 l=1 rev1=1 rx=0x7f6c18006700 tx=0x7f6c18009b80).ready entity=mon.0 client_cookie=640c9167da0bad49 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:53.600 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:53.583 7f6c267fc700 1 -- 172.21.2.112:0/1679493093 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6c1800c0c0 con 0x7f6c280d1870 2021-02-10T11:43:53.600 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:53.583 7f6c267fc700 1 -- 172.21.2.112:0/1679493093 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c1800c340 con 0x7f6c280d1870 2021-02-10T11:43:53.600 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:53.583 7f6c267fc700 1 -- 172.21.2.112:0/1679493093 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6c1800c860 con 0x7f6c280d1870 2021-02-10T11:43:53.600 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:53.583 7f6c2dfb6700 1 -- 172.21.2.112:0/1679493093 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6c280d1870 msgr2=0x7f6c281b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:53.600 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:53.583 7f6c2dfb6700 1 --2- 172.21.2.112:0/1679493093 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6c280d1870 0x7f6c281b03f0 secure :-1 s=READY pgs=2528 cs=0 l=1 rev1=1 rx=0x7f6c18006700 tx=0x7f6c18009b80).stop 2021-02-10T11:43:53.600 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:53.584 7f6c2dfb6700 1 -- 172.21.2.112:0/1679493093 shutdown_connections 2021-02-10T11:43:53.600 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:53.584 7f6c2dfb6700 1 --2- 172.21.2.112:0/1679493093 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6c280f0650 0x7f6c280cfb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:53.600 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:53.584 7f6c2dfb6700 1 --2- 172.21.2.112:0/1679493093 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6c280d1870 0x7f6c281b03f0 unknown :-1 s=CLOSED pgs=2528 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:53.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:53.584 7f6c2dfb6700 1 -- 172.21.2.112:0/1679493093 >> 172.21.2.112:0/1679493093 conn(0x7f6c280ce450 msgr2=0x7f6c280d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:53.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:53.584 7f6c2dfb6700 1 -- 172.21.2.112:0/1679493093 shutdown_connections 2021-02-10T11:43:53.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:53.584 7f6c2dfb6700 1 -- 172.21.2.112:0/1679493093 wait complete. 2021-02-10T11:43:53.602 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:53.598 7f6c2dfb6700 1 Processor -- start 2021-02-10T11:43:53.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:53.599 7f6c2dfb6700 1 -- start start 2021-02-10T11:43:53.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:53.599 7f6c2dfb6700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6c281971e0 0x7f6c2818eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:53.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:53.599 7f6c2dfb6700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6c281b8180 0x7f6c281b8580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:53.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:53.599 7f6c2dfb6700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6c281b8ba0 0x7f6c280b17b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:53.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:53.599 7f6c2dfb6700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c281b9750 con 0x7f6c281971e0 2021-02-10T11:43:53.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:53.599 7f6c2dfb6700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c2818e930 con 0x7f6c281b8ba0 2021-02-10T11:43:53.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:53.599 7f6c2dfb6700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c280b1dd0 con 0x7f6c281b8180 2021-02-10T11:43:53.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:53.599 7f6c277fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6c281971e0 0x7f6c2818eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:53.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:53.599 7f6c277fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6c281971e0 0x7f6c2818eda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:57416/0 (socket says 172.21.2.112:57416) 2021-02-10T11:43:53.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:53.599 7f6c277fe700 1 -- 172.21.2.112:0/1744030936 learned_addr learned my addr 172.21.2.112:0/1744030936 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:43:53.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:53.599 7f6c277fe700 1 -- 172.21.2.112:0/1744030936 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6c281b8180 msgr2=0x7f6c281b8580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:53.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:53.599 7f6c277fe700 1 --2- 172.21.2.112:0/1744030936 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6c281b8180 0x7f6c281b8580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:53.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:53.599 7f6c277fe700 1 -- 172.21.2.112:0/1744030936 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6c281b8ba0 msgr2=0x7f6c280b17b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:53.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:53.599 7f6c277fe700 1 --2- 172.21.2.112:0/1744030936 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6c281b8ba0 0x7f6c280b17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:53.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:53.599 7f6c277fe700 1 -- 172.21.2.112:0/1744030936 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c280b1ff0 con 0x7f6c281971e0 2021-02-10T11:43:53.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:53.600 7f6c277fe700 1 --2- 172.21.2.112:0/1744030936 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6c281971e0 0x7f6c2818eda0 secure :-1 s=READY pgs=2529 cs=0 l=1 rev1=1 rx=0x7f6c18000bd0 tx=0x7f6c1800b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:53.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:53.600 7f6c1ffff700 1 -- 172.21.2.112:0/1744030936 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6c1800bd50 con 0x7f6c281971e0 2021-02-10T11:43:53.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:53.600 7f6c1ffff700 1 -- 172.21.2.112:0/1744030936 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c180052e0 con 0x7f6c281971e0 2021-02-10T11:43:53.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:53.600 7f6c1ffff700 1 -- 172.21.2.112:0/1744030936 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6c18005800 con 0x7f6c281971e0 2021-02-10T11:43:53.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:53.600 7f6c2dfb6700 1 -- 172.21.2.112:0/1744030936 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c280b1ff0 con 0x7f6c281971e0 2021-02-10T11:43:53.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:53.600 7f6c2dfb6700 1 -- 172.21.2.112:0/1744030936 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c280b2580 con 0x7f6c281971e0 2021-02-10T11:43:53.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:53.601 7f6c1ffff700 1 -- 172.21.2.112:0/1744030936 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6c18006030 con 0x7f6c281971e0 2021-02-10T11:43:53.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:53.601 7f6c1ffff700 1 --2- 172.21.2.112:0/1744030936 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6c0c0186c0 0x7f6c0c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:53.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:53.601 7f6c1ffff700 1 -- 172.21.2.112:0/1744030936 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f6c18032240 con 0x7f6c281971e0 2021-02-10T11:43:53.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:53.601 7f6c26ffd700 1 --2- 172.21.2.112:0/1744030936 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6c0c0186c0 0x7f6c0c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:53.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:53.602 7f6c26ffd700 1 --2- 172.21.2.112:0/1744030936 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6c0c0186c0 0x7f6c0c01abc0 crc :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:53.633 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:53.632 7f6c2dfb6700 1 -- 172.21.2.112:0/1744030936 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c280ee060 con 0x7f6c281971e0 2021-02-10T11:43:53.636 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:53.635 7f6c1ffff700 1 -- 172.21.2.112:0/1744030936 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6c18030d20 con 0x7f6c281971e0 2021-02-10T11:43:53.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:53.759 7f6c2dfb6700 1 -- 172.21.2.112:0/1744030936 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f6c280ee060 con 0x7f6c281971e0 2021-02-10T11:43:53.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:53.759 7f6c1ffff700 1 -- 172.21.2.112:0/1744030936 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v138) v1 ==== 84+0+64 (secure 0 0 0) 0x7f6c1800bd50 con 0x7f6c281971e0 2021-02-10T11:43:53.760 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:43:53.760 INFO:teuthology.orchestra.run.gibba012.stdout:[ 2021-02-10T11:43:53.760 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:43:53.760 INFO:teuthology.orchestra.run.gibba012.stdout: "poolnum": 1, 2021-02-10T11:43:53.760 INFO:teuthology.orchestra.run.gibba012.stdout: "poolname": "rbd" 2021-02-10T11:43:53.760 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:43:53.761 INFO:teuthology.orchestra.run.gibba012.stdout:] 2021-02-10T11:43:53.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:53.761 7f6c2dfb6700 1 -- 172.21.2.112:0/1744030936 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6c0c0186c0 msgr2=0x7f6c0c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:53.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:53.761 7f6c2dfb6700 1 --2- 172.21.2.112:0/1744030936 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6c0c0186c0 0x7f6c0c01abc0 crc :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:53.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:53.761 7f6c2dfb6700 1 -- 172.21.2.112:0/1744030936 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6c281971e0 msgr2=0x7f6c2818eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:53.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:53.761 7f6c2dfb6700 1 --2- 172.21.2.112:0/1744030936 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6c281971e0 0x7f6c2818eda0 secure :-1 s=READY pgs=2529 cs=0 l=1 rev1=1 rx=0x7f6c18000bd0 tx=0x7f6c1800b960).stop 2021-02-10T11:43:53.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:53.761 7f6c2dfb6700 1 -- 172.21.2.112:0/1744030936 shutdown_connections 2021-02-10T11:43:53.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:53.761 7f6c2dfb6700 1 --2- 172.21.2.112:0/1744030936 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6c0c0186c0 0x7f6c0c01abc0 unknown :-1 s=CLOSED pgs=1154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:53.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:53.761 7f6c2dfb6700 1 --2- 172.21.2.112:0/1744030936 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6c281b8ba0 0x7f6c280b17b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:53.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:53.761 7f6c2dfb6700 1 --2- 172.21.2.112:0/1744030936 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6c281b8180 0x7f6c281b8580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:53.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:53.761 7f6c2dfb6700 1 --2- 172.21.2.112:0/1744030936 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6c281971e0 0x7f6c2818eda0 unknown :-1 s=CLOSED pgs=2529 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:53.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:53.761 7f6c2dfb6700 1 -- 172.21.2.112:0/1744030936 >> 172.21.2.112:0/1744030936 conn(0x7f6c2819f8f0 msgr2=0x7f6c2818c260 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:53.767 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:53.761 7f6c2dfb6700 1 -- 172.21.2.112:0/1744030936 shutdown_connections 2021-02-10T11:43:53.767 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:53.761 7f6c2dfb6700 1 -- 172.21.2.112:0/1744030936 wait complete. 2021-02-10T11:43:53.771 INFO:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2021-02-10T11:43:53.771 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_metadata 8 2021-02-10T11:43:54.204 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:54.188 7f9d879c6700 1 Processor -- start 2021-02-10T11:43:54.204 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:54.188 7f9d879c6700 1 -- start start 2021-02-10T11:43:54.204 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:54.188 7f9d879c6700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f9d800d1870 0x7f9d801b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:54.205 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:54.188 7f9d879c6700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f9d801b0ae0 0x7f9d80108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:54.205 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:54.188 7f9d879c6700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9d800d1c70 0x7f9d80108f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:54.205 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:54.188 7f9d879c6700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d800fe1c0 con 0x7f9d801b0ae0 2021-02-10T11:43:54.205 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:54.188 7f9d879c6700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d80109a60 con 0x7f9d800d1870 2021-02-10T11:43:54.205 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:54.188 7f9d879c6700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d80088070 con 0x7f9d800d1c70 2021-02-10T11:43:54.205 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:54.188 7f9d84fe1700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f9d801b0ae0 0x7f9d80108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:54.205 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:54.188 7f9d84fe1700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f9d801b0ae0 0x7f9d80108940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:57430/0 (socket says 172.21.2.112:57430) 2021-02-10T11:43:54.205 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:54.188 7f9d84fe1700 1 -- 172.21.2.112:0/966105481 learned_addr learned my addr 172.21.2.112:0/966105481 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:43:54.205 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:54.189 7f9d84fe1700 1 -- 172.21.2.112:0/966105481 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9d800d1c70 msgr2=0x7f9d80108f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:54.205 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:54.189 7f9d84fe1700 1 --2- 172.21.2.112:0/966105481 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9d800d1c70 0x7f9d80108f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:54.205 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:54.189 7f9d84fe1700 1 -- 172.21.2.112:0/966105481 >> v2:172.21.2.120:3300/0 conn(0x7f9d800d1870 msgr2=0x7f9d801b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:54.206 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:54.189 7f9d84fe1700 1 --2- 172.21.2.112:0/966105481 >> v2:172.21.2.120:3300/0 conn(0x7f9d800d1870 0x7f9d801b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:54.206 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:54.189 7f9d84fe1700 1 -- 172.21.2.112:0/966105481 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d80109580 con 0x7f9d801b0ae0 2021-02-10T11:43:54.206 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:54.189 7f9d84fe1700 1 --2- 172.21.2.112:0/966105481 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9d801b0ae0 0x7f9d80108940 secure :-1 s=READY pgs=2530 cs=0 l=1 rev1=1 rx=0x7f9d70006700 tx=0x7f9d70009b00).ready entity=mon.0 client_cookie=28018bdfdb067970 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:54.206 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:54.189 7f9d77fff700 1 -- 172.21.2.112:0/966105481 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9d7000c0c0 con 0x7f9d801b0ae0 2021-02-10T11:43:54.206 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:54.189 7f9d77fff700 1 -- 172.21.2.112:0/966105481 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d7000c340 con 0x7f9d801b0ae0 2021-02-10T11:43:54.206 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:54.189 7f9d77fff700 1 -- 172.21.2.112:0/966105481 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9d7000c820 con 0x7f9d801b0ae0 2021-02-10T11:43:54.206 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:54.189 7f9d879c6700 1 -- 172.21.2.112:0/966105481 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9d801b0ae0 msgr2=0x7f9d80108940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:54.206 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:54.189 7f9d879c6700 1 --2- 172.21.2.112:0/966105481 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9d801b0ae0 0x7f9d80108940 secure :-1 s=READY pgs=2530 cs=0 l=1 rev1=1 rx=0x7f9d70006700 tx=0x7f9d70009b00).stop 2021-02-10T11:43:54.206 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:54.190 7f9d879c6700 1 -- 172.21.2.112:0/966105481 shutdown_connections 2021-02-10T11:43:54.206 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:54.190 7f9d879c6700 1 --2- 172.21.2.112:0/966105481 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9d800d1c70 0x7f9d80108f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:54.206 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:54.190 7f9d879c6700 1 --2- 172.21.2.112:0/966105481 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9d801b0ae0 0x7f9d80108940 unknown :-1 s=CLOSED pgs=2530 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:54.206 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:54.190 7f9d879c6700 1 --2- 172.21.2.112:0/966105481 >> v2:172.21.2.120:3300/0 conn(0x7f9d800d1870 0x7f9d801b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:54.207 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:54.190 7f9d879c6700 1 -- 172.21.2.112:0/966105481 >> 172.21.2.112:0/966105481 conn(0x7f9d800ce420 msgr2=0x7f9d800d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:54.207 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:54.190 7f9d879c6700 1 -- 172.21.2.112:0/966105481 shutdown_connections 2021-02-10T11:43:54.207 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:54.190 7f9d879c6700 1 -- 172.21.2.112:0/966105481 wait complete. 2021-02-10T11:43:54.208 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:54.204 7f9d879c6700 1 Processor -- start 2021-02-10T11:43:54.208 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:54.204 7f9d879c6700 1 -- start start 2021-02-10T11:43:54.208 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:54.205 7f9d879c6700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9d800c3820 0x7f9d800a7950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:54.208 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:54.205 7f9d879c6700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9d801a44c0 0x7f9d801a48c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:54.209 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:54.205 7f9d879c6700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9d800863d0 0x7f9d800867d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:54.209 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:54.205 7f9d879c6700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d80085ed0 con 0x7f9d800863d0 2021-02-10T11:43:54.209 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:54.205 7f9d879c6700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d800c0740 con 0x7f9d801a44c0 2021-02-10T11:43:54.209 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:54.205 7f9d879c6700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d80086df0 con 0x7f9d800c3820 2021-02-10T11:43:54.209 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:54.205 7f9d85fe3700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9d800863d0 0x7f9d800867d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:54.209 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:54.205 7f9d85fe3700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9d800863d0 0x7f9d800867d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:57432/0 (socket says 172.21.2.112:57432) 2021-02-10T11:43:54.209 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:54.205 7f9d85fe3700 1 -- 172.21.2.112:0/3151144368 learned_addr learned my addr 172.21.2.112:0/3151144368 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:43:54.209 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:54.205 7f9d85fe3700 1 -- 172.21.2.112:0/3151144368 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9d800c3820 msgr2=0x7f9d800a7950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:54.209 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:54.205 7f9d85fe3700 1 --2- 172.21.2.112:0/3151144368 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9d800c3820 0x7f9d800a7950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:54.209 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:54.205 7f9d85fe3700 1 -- 172.21.2.112:0/3151144368 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9d801a44c0 msgr2=0x7f9d801a48c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:43:54.209 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:54.205 7f9d85fe3700 1 --2- 172.21.2.112:0/3151144368 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9d801a44c0 0x7f9d801a48c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:54.209 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:54.205 7f9d85fe3700 1 -- 172.21.2.112:0/3151144368 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d80087010 con 0x7f9d800863d0 2021-02-10T11:43:54.210 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:54.205 7f9d85fe3700 1 --2- 172.21.2.112:0/3151144368 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9d800863d0 0x7f9d800867d0 secure :-1 s=READY pgs=2531 cs=0 l=1 rev1=1 rx=0x7f9d78009980 tx=0x7f9d7800bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:54.210 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:54.205 7f9d75ffb700 1 -- 172.21.2.112:0/3151144368 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9d78005af0 con 0x7f9d800863d0 2021-02-10T11:43:54.210 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:54.205 7f9d75ffb700 1 -- 172.21.2.112:0/3151144368 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d78005d70 con 0x7f9d800863d0 2021-02-10T11:43:54.210 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:54.205 7f9d879c6700 1 -- 172.21.2.112:0/3151144368 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d80087010 con 0x7f9d800863d0 2021-02-10T11:43:54.210 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:54.205 7f9d75ffb700 1 -- 172.21.2.112:0/3151144368 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9d780062b0 con 0x7f9d800863d0 2021-02-10T11:43:54.210 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:54.206 7f9d879c6700 1 -- 172.21.2.112:0/3151144368 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d80087010 con 0x7f9d800863d0 2021-02-10T11:43:54.210 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:54.206 7f9d75ffb700 1 -- 172.21.2.112:0/3151144368 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9d78006030 con 0x7f9d800863d0 2021-02-10T11:43:54.210 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:54.206 7f9d75ffb700 1 --2- 172.21.2.112:0/3151144368 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9d600186d0 0x7f9d6001abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:54.210 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:54.206 7f9d75ffb700 1 -- 172.21.2.112:0/3151144368 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f9d78032800 con 0x7f9d800863d0 2021-02-10T11:43:54.210 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:54.206 7f9d857e2700 1 --2- 172.21.2.112:0/3151144368 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9d600186d0 0x7f9d6001abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:54.210 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:54.207 7f9d857e2700 1 --2- 172.21.2.112:0/3151144368 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9d600186d0 0x7f9d6001abd0 crc :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:54.238 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:54.238 7f9d879c6700 1 -- 172.21.2.112:0/3151144368 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d800fe1c0 con 0x7f9d800863d0 2021-02-10T11:43:54.241 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:54.240 7f9d75ffb700 1 -- 172.21.2.112:0/3151144368 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9d7800be10 con 0x7f9d800863d0 2021-02-10T11:43:54.368 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:54.367 7f9d879c6700 1 -- 172.21.2.112:0/3151144368 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"} v 0) v1 -- 0x7f9d800fe1c0 con 0x7f9d800863d0 2021-02-10T11:43:54.420 DEBUG:teuthology.orchestra.run.gibba012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:43:54.423 DEBUG:teuthology.orchestra.run.gibba020:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:43:54.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:54.772 7f9d75ffb700 1 -- 172.21.2.112:0/3151144368 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"}]=0 pool 'cephfs_metadata' created v139) v1 ==== 133+0+0 (secure 0 0 0) 0x7f9d78006030 con 0x7f9d800863d0 2021-02-10T11:43:54.783 INFO:teuthology.orchestra.run.gibba012.stderr:pool 'cephfs_metadata' created 2021-02-10T11:43:54.786 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:54.785 7f9d879c6700 1 -- 172.21.2.112:0/3151144368 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9d600186d0 msgr2=0x7f9d6001abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:54.786 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:54.785 7f9d879c6700 1 --2- 172.21.2.112:0/3151144368 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9d600186d0 0x7f9d6001abd0 crc :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:54.786 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:54.785 7f9d879c6700 1 -- 172.21.2.112:0/3151144368 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9d800863d0 msgr2=0x7f9d800867d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:54.786 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:54.785 7f9d879c6700 1 --2- 172.21.2.112:0/3151144368 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9d800863d0 0x7f9d800867d0 secure :-1 s=READY pgs=2531 cs=0 l=1 rev1=1 rx=0x7f9d78009980 tx=0x7f9d7800bcd0).stop 2021-02-10T11:43:54.786 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:54.785 7f9d879c6700 1 -- 172.21.2.112:0/3151144368 shutdown_connections 2021-02-10T11:43:54.787 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:54.785 7f9d879c6700 1 --2- 172.21.2.112:0/3151144368 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9d600186d0 0x7f9d6001abd0 unknown :-1 s=CLOSED pgs=1160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:54.787 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:54.785 7f9d879c6700 1 --2- 172.21.2.112:0/3151144368 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9d801a44c0 0x7f9d801a48c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:54.787 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:54.785 7f9d879c6700 1 --2- 172.21.2.112:0/3151144368 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9d800863d0 0x7f9d800867d0 unknown :-1 s=CLOSED pgs=2531 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:54.787 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:54.785 7f9d879c6700 1 --2- 172.21.2.112:0/3151144368 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9d800c3820 0x7f9d800a7950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:54.787 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:54.785 7f9d879c6700 1 -- 172.21.2.112:0/3151144368 >> 172.21.2.112:0/3151144368 conn(0x7f9d800872c0 msgr2=0x7f9d80095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:54.787 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:54.785 7f9d879c6700 1 -- 172.21.2.112:0/3151144368 shutdown_connections 2021-02-10T11:43:54.787 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:54.786 7f9d879c6700 1 -- 172.21.2.112:0/3151144368 wait complete. 2021-02-10T11:43:54.794 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_data 8 2021-02-10T11:43:55.266 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:55.249 7f8cb8268700 1 Processor -- start 2021-02-10T11:43:55.266 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:55.249 7f8cb8268700 1 -- start start 2021-02-10T11:43:55.266 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:55.249 7f8cb8268700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8cb00d1870 0x7f8cb01b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:55.266 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:55.250 7f8cb8268700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f8cb00cff70 0x7f8cb0109130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:55.266 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:55.250 7f8cb8268700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cb0109a60 con 0x7f8cb01b0ae0 2021-02-10T11:43:55.266 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:55.250 7f8cb8268700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cb0109750 con 0x7f8cb00cff70 2021-02-10T11:43:55.266 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:55.250 7f8cb8268700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cb0088070 con 0x7f8cb00d1870 2021-02-10T11:43:55.267 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:55.250 7f8cb6885700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f8cb00cff70 0x7f8cb0109130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:55.267 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:55.250 7f8cb6885700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f8cb00cff70 0x7f8cb0109130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:57440/0 (socket says 172.21.2.112:57440) 2021-02-10T11:43:55.267 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:55.250 7f8cb6885700 1 -- 172.21.2.112:0/1165997660 learned_addr learned my addr 172.21.2.112:0/1165997660 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:43:55.267 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:55.250 7f8cb6885700 1 -- 172.21.2.112:0/1165997660 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8cb00d1870 msgr2=0x7f8cb01b04c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:43:55.267 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:55.250 7f8cb6885700 1 --2- 172.21.2.112:0/1165997660 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8cb00d1870 0x7f8cb01b04c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:55.267 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:55.250 7f8cb6885700 1 -- 172.21.2.112:0/1165997660 >> v1:172.21.2.120:6789/0 conn(0x7f8cb01b0ae0 legacy=0x7f8cb0108940 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:43:55.267 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:55.250 7f8cb6885700 1 -- 172.21.2.112:0/1165997660 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cb0088260 con 0x7f8cb00cff70 2021-02-10T11:43:55.267 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:55.250 7f8cb6885700 1 --2- 172.21.2.112:0/1165997660 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8cb00cff70 0x7f8cb0109130 secure :-1 s=READY pgs=2532 cs=0 l=1 rev1=1 rx=0x7f8ca8006f00 tx=0x7f8ca8009f10).ready entity=mon.0 client_cookie=d8460093d68b551d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:55.267 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:55.250 7f8cb5082700 1 -- 172.21.2.112:0/1165997660 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8ca800b3b0 con 0x7f8cb00cff70 2021-02-10T11:43:55.267 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:55.250 7f8cb5082700 1 -- 172.21.2.112:0/1165997660 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ca800b630 con 0x7f8cb00cff70 2021-02-10T11:43:55.267 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:55.250 7f8cb5082700 1 -- 172.21.2.112:0/1165997660 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8ca800bb50 con 0x7f8cb00cff70 2021-02-10T11:43:55.267 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:55.250 7f8cb8268700 1 -- 172.21.2.112:0/1165997660 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8cb00cff70 msgr2=0x7f8cb0109130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:55.267 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:55.250 7f8cb8268700 1 --2- 172.21.2.112:0/1165997660 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8cb00cff70 0x7f8cb0109130 secure :-1 s=READY pgs=2532 cs=0 l=1 rev1=1 rx=0x7f8ca8006f00 tx=0x7f8ca8009f10).stop 2021-02-10T11:43:55.268 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:55.251 7f8cb8268700 1 -- 172.21.2.112:0/1165997660 shutdown_connections 2021-02-10T11:43:55.268 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:55.251 7f8cb8268700 1 --2- 172.21.2.112:0/1165997660 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8cb00cff70 0x7f8cb0109130 unknown :-1 s=CLOSED pgs=2532 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:55.268 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:55.251 7f8cb8268700 1 --2- 172.21.2.112:0/1165997660 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8cb00d1870 0x7f8cb01b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:55.268 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:55.251 7f8cb8268700 1 -- 172.21.2.112:0/1165997660 >> 172.21.2.112:0/1165997660 conn(0x7f8cb00ce420 msgr2=0x7f8cb00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:55.268 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:55.251 7f8cb8268700 1 -- 172.21.2.112:0/1165997660 shutdown_connections 2021-02-10T11:43:55.268 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:55.251 7f8cb8268700 1 -- 172.21.2.112:0/1165997660 wait complete. 2021-02-10T11:43:55.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:55.266 7f8cb8268700 1 Processor -- start 2021-02-10T11:43:55.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:55.266 7f8cb8268700 1 -- start start 2021-02-10T11:43:55.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:55.266 7f8cb8268700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8cb00c3820 0x7f8cb0084cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:55.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:55.266 7f8cb8268700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8cb00a0290 0x7f8cb00852d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:55.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:55.266 7f8cb8268700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8cb0086270 0x7f8cb0086670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:55.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:55.266 7f8cb8268700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cb00c0700 con 0x7f8cb0086270 2021-02-10T11:43:55.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:55.266 7f8cb8268700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cb00858f0 con 0x7f8cb00c3820 2021-02-10T11:43:55.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:55.266 7f8cb8268700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cb0085b10 con 0x7f8cb00a0290 2021-02-10T11:43:55.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:55.266 7f8cb6885700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8cb0086270 0x7f8cb0086670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:55.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:55.266 7f8cb6885700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8cb0086270 0x7f8cb0086670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:57448/0 (socket says 172.21.2.112:57448) 2021-02-10T11:43:55.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:55.266 7f8cb6885700 1 -- 172.21.2.112:0/2162304690 learned_addr learned my addr 172.21.2.112:0/2162304690 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:43:55.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:55.267 7f8cb6885700 1 -- 172.21.2.112:0/2162304690 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8cb00a0290 msgr2=0x7f8cb00852d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:55.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:55.267 7f8cb6885700 1 --2- 172.21.2.112:0/2162304690 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8cb00a0290 0x7f8cb00852d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:55.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:55.267 7f8cb6885700 1 -- 172.21.2.112:0/2162304690 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8cb00c3820 msgr2=0x7f8cb0084cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:55.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:55.267 7f8cb6885700 1 --2- 172.21.2.112:0/2162304690 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8cb00c3820 0x7f8cb0084cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:55.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:55.267 7f8cb6885700 1 -- 172.21.2.112:0/2162304690 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cb0086c90 con 0x7f8cb0086270 2021-02-10T11:43:55.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:55.267 7f8cb6885700 1 --2- 172.21.2.112:0/2162304690 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8cb0086270 0x7f8cb0086670 secure :-1 s=READY pgs=2533 cs=0 l=1 rev1=1 rx=0x7f8ca8009390 tx=0x7f8ca800bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:55.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:55.267 7f8ca6ffd700 1 -- 172.21.2.112:0/2162304690 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8ca8003030 con 0x7f8cb0086270 2021-02-10T11:43:55.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:55.267 7f8ca6ffd700 1 -- 172.21.2.112:0/2162304690 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ca800c2f0 con 0x7f8cb0086270 2021-02-10T11:43:55.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:55.267 7f8ca6ffd700 1 -- 172.21.2.112:0/2162304690 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8ca800c7d0 con 0x7f8cb0086270 2021-02-10T11:43:55.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:55.267 7f8cb8268700 1 -- 172.21.2.112:0/2162304690 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cb0086c90 con 0x7f8cb0086270 2021-02-10T11:43:55.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:55.267 7f8cb8268700 1 -- 172.21.2.112:0/2162304690 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cb0086c90 con 0x7f8cb0086270 2021-02-10T11:43:55.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:55.268 7f8ca6ffd700 1 -- 172.21.2.112:0/2162304690 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8ca8001270 con 0x7f8cb0086270 2021-02-10T11:43:55.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:55.268 7f8ca6ffd700 1 --2- 172.21.2.112:0/2162304690 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8c980186c0 0x7f8c9801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:55.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:55.268 7f8ca6ffd700 1 -- 172.21.2.112:0/2162304690 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 5107+0+0 (secure 0 0 0) 0x7f8ca80325a0 con 0x7f8cb0086270 2021-02-10T11:43:55.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:55.268 7f8cb6084700 1 --2- 172.21.2.112:0/2162304690 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8c980186c0 0x7f8c9801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:55.272 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:55.269 7f8cb6084700 1 --2- 172.21.2.112:0/2162304690 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8c980186c0 0x7f8c9801abc0 crc :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:55.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:55.299 7f8cb8268700 1 -- 172.21.2.112:0/2162304690 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cb0082a70 con 0x7f8cb0086270 2021-02-10T11:43:55.303 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:55.302 7f8ca6ffd700 1 -- 172.21.2.112:0/2162304690 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8ca8000cc0 con 0x7f8cb0086270 2021-02-10T11:43:55.436 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:55.435 7f8cb8268700 1 -- 172.21.2.112:0/2162304690 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"} v 0) v1 -- 0x7f8cb0082a70 con 0x7f8cb0086270 2021-02-10T11:43:55.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:55.779 7f8ca6ffd700 1 -- 172.21.2.112:0/2162304690 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"}]=0 pool 'cephfs_data' created v141) v1 ==== 125+0+0 (secure 0 0 0) 0x7f8ca8001270 con 0x7f8cb0086270 2021-02-10T11:43:55.801 INFO:teuthology.orchestra.run.gibba012.stderr:pool 'cephfs_data' created 2021-02-10T11:43:55.802 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:55.802 7f8cb8268700 1 -- 172.21.2.112:0/2162304690 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8c980186c0 msgr2=0x7f8c9801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:55.803 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:55.802 7f8cb8268700 1 --2- 172.21.2.112:0/2162304690 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8c980186c0 0x7f8c9801abc0 crc :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:55.803 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:55.802 7f8cb8268700 1 -- 172.21.2.112:0/2162304690 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8cb0086270 msgr2=0x7f8cb0086670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:55.803 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:55.802 7f8cb8268700 1 --2- 172.21.2.112:0/2162304690 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8cb0086270 0x7f8cb0086670 secure :-1 s=READY pgs=2533 cs=0 l=1 rev1=1 rx=0x7f8ca8009390 tx=0x7f8ca800bd40).stop 2021-02-10T11:43:55.803 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:55.802 7f8cb8268700 1 -- 172.21.2.112:0/2162304690 shutdown_connections 2021-02-10T11:43:55.803 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:55.802 7f8cb8268700 1 --2- 172.21.2.112:0/2162304690 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8c980186c0 0x7f8c9801abc0 unknown :-1 s=CLOSED pgs=1161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:55.803 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:55.802 7f8cb8268700 1 --2- 172.21.2.112:0/2162304690 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8cb0086270 0x7f8cb0086670 unknown :-1 s=CLOSED pgs=2533 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:55.803 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:55.802 7f8cb8268700 1 --2- 172.21.2.112:0/2162304690 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8cb00a0290 0x7f8cb00852d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:55.803 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:55.802 7f8cb8268700 1 --2- 172.21.2.112:0/2162304690 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8cb00c3820 0x7f8cb0084cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:55.803 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:55.802 7f8cb8268700 1 -- 172.21.2.112:0/2162304690 >> 172.21.2.112:0/2162304690 conn(0x7f8cb00872c0 msgr2=0x7f8cb0095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:55.803 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:55.802 7f8cb8268700 1 -- 172.21.2.112:0/2162304690 shutdown_connections 2021-02-10T11:43:55.803 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:55.802 7f8cb8268700 1 -- 172.21.2.112:0/2162304690 wait complete. 2021-02-10T11:43:55.811 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2021-02-10T11:43:56.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.226 7fe156ad7700 1 Processor -- start 2021-02-10T11:43:56.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.226 7fe156ad7700 1 -- start start 2021-02-10T11:43:56.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.226 7fe156ad7700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fe1500d1c80 0x7fe1500cef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:56.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.226 7fe156ad7700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fe1500cf590 0x7fe1500cf990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:56.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.226 7fe156ad7700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1500fe1d0 con 0x7fe1500d1880 2021-02-10T11:43:56.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.226 7fe156ad7700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1501b0cc0 con 0x7fe1500cf590 2021-02-10T11:43:56.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.226 7fe156ad7700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe150109a70 con 0x7fe1500d1c80 2021-02-10T11:43:56.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.227 7fe1550f4700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fe1500cf590 0x7fe1500cf990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:56.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.227 7fe1550f4700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fe1500cf590 0x7fe1500cf990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:57454/0 (socket says 172.21.2.112:57454) 2021-02-10T11:43:56.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.227 7fe1550f4700 1 -- 172.21.2.112:0/2143956297 learned_addr learned my addr 172.21.2.112:0/2143956297 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:43:56.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.227 7fe14ffff700 1 --2- 172.21.2.112:0/2143956297 >> v2:172.21.2.120:3300/0 conn(0x7fe1500d1c80 0x7fe1500cef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:56.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.227 7fe1550f4700 1 -- 172.21.2.112:0/2143956297 >> v2:172.21.2.120:3300/0 conn(0x7fe1500d1c80 msgr2=0x7fe1500cef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:56.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.227 7fe1550f4700 1 --2- 172.21.2.112:0/2143956297 >> v2:172.21.2.120:3300/0 conn(0x7fe1500d1c80 0x7fe1500cef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:56.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.227 7fe1550f4700 1 -- 172.21.2.112:0/2143956297 >> v1:172.21.2.120:6789/0 conn(0x7fe1500d1880 legacy=0x7fe1501b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:56.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.227 7fe1550f4700 1 -- 172.21.2.112:0/2143956297 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe150088070 con 0x7fe1500cf590 2021-02-10T11:43:56.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.227 7fe1550f4700 1 --2- 172.21.2.112:0/2143956297 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe1500cf590 0x7fe1500cf990 secure :-1 s=READY pgs=2534 cs=0 l=1 rev1=1 rx=0x7fe148006f00 tx=0x7fe148009f10).ready entity=mon.0 client_cookie=706deb74bf44aa64 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:56.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.227 7fe14f7fe700 1 -- 172.21.2.112:0/2143956297 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe148003030 con 0x7fe1500cf590 2021-02-10T11:43:56.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.227 7fe14f7fe700 1 -- 172.21.2.112:0/2143956297 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe148003030 con 0x7fe1500cf590 2021-02-10T11:43:56.245 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.227 7fe14f7fe700 1 -- 172.21.2.112:0/2143956297 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe14800b3b0 con 0x7fe1500cf590 2021-02-10T11:43:56.245 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.227 7fe156ad7700 1 -- 172.21.2.112:0/2143956297 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe1500cf590 msgr2=0x7fe1500cf990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:56.245 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.227 7fe156ad7700 1 --2- 172.21.2.112:0/2143956297 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe1500cf590 0x7fe1500cf990 secure :-1 s=READY pgs=2534 cs=0 l=1 rev1=1 rx=0x7fe148006f00 tx=0x7fe148009f10).stop 2021-02-10T11:43:56.245 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.228 7fe156ad7700 1 -- 172.21.2.112:0/2143956297 shutdown_connections 2021-02-10T11:43:56.245 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.228 7fe156ad7700 1 --2- 172.21.2.112:0/2143956297 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe1500cf590 0x7fe1500cf990 unknown :-1 s=CLOSED pgs=2534 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:56.245 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.228 7fe156ad7700 1 --2- 172.21.2.112:0/2143956297 >> v2:172.21.2.120:3300/0 conn(0x7fe1500d1c80 0x7fe1500cef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:56.245 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.228 7fe156ad7700 1 -- 172.21.2.112:0/2143956297 >> 172.21.2.112:0/2143956297 conn(0x7fe1500ce430 msgr2=0x7fe1500d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:56.245 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.228 7fe156ad7700 1 -- 172.21.2.112:0/2143956297 shutdown_connections 2021-02-10T11:43:56.245 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.228 7fe156ad7700 1 -- 172.21.2.112:0/2143956297 wait complete. 2021-02-10T11:43:56.245 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.242 7fe156ad7700 1 Processor -- start 2021-02-10T11:43:56.245 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.243 7fe156ad7700 1 -- start start 2021-02-10T11:43:56.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.243 7fe156ad7700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe1500a87a0 0x7fe15018b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:56.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.243 7fe156ad7700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe150086930 0x7fe1501aab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:56.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.243 7fe156ad7700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe150086d30 0x7fe1501ab160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:56.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.243 7fe156ad7700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe15018be10 con 0x7fe150086930 2021-02-10T11:43:56.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.243 7fe156ad7700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1501b98c0 con 0x7fe150086d30 2021-02-10T11:43:56.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.243 7fe156ad7700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1501b9d60 con 0x7fe1500a87a0 2021-02-10T11:43:56.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.243 7fe14ffff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe150086930 0x7fe1501aab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:56.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.243 7fe14ffff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe150086930 0x7fe1501aab40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:57460/0 (socket says 172.21.2.112:57460) 2021-02-10T11:43:56.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.243 7fe14ffff700 1 -- 172.21.2.112:0/2218615652 learned_addr learned my addr 172.21.2.112:0/2218615652 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:43:56.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.243 7fe1548f3700 1 --2- 172.21.2.112:0/2218615652 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe1500a87a0 0x7fe15018b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:56.248 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.243 7fe1550f4700 1 --2- 172.21.2.112:0/2218615652 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe150086d30 0x7fe1501ab160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:56.248 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.244 7fe14ffff700 1 -- 172.21.2.112:0/2218615652 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe1500a87a0 msgr2=0x7fe15018b7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:56.248 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.244 7fe14ffff700 1 --2- 172.21.2.112:0/2218615652 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe1500a87a0 0x7fe15018b7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:56.248 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.244 7fe14ffff700 1 -- 172.21.2.112:0/2218615652 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe150086d30 msgr2=0x7fe1501ab160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:56.248 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.244 7fe14ffff700 1 --2- 172.21.2.112:0/2218615652 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe150086d30 0x7fe1501ab160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:56.248 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.244 7fe14ffff700 1 -- 172.21.2.112:0/2218615652 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1501b9f80 con 0x7fe150086930 2021-02-10T11:43:56.248 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.244 7fe14ffff700 1 --2- 172.21.2.112:0/2218615652 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe150086930 0x7fe1501aab40 secure :-1 s=READY pgs=2535 cs=0 l=1 rev1=1 rx=0x7fe140006f60 tx=0x7fe14000b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:56.248 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.244 7fe14d7fa700 1 -- 172.21.2.112:0/2218615652 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe1400009b0 con 0x7fe150086930 2021-02-10T11:43:56.248 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.244 7fe14d7fa700 1 -- 172.21.2.112:0/2218615652 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe14000bdf0 con 0x7fe150086930 2021-02-10T11:43:56.248 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.244 7fe14d7fa700 1 -- 172.21.2.112:0/2218615652 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe140002030 con 0x7fe150086930 2021-02-10T11:43:56.248 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.244 7fe156ad7700 1 -- 172.21.2.112:0/2218615652 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1501b9f80 con 0x7fe150086930 2021-02-10T11:43:56.248 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.244 7fe156ad7700 1 -- 172.21.2.112:0/2218615652 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1501b9f80 con 0x7fe150086930 2021-02-10T11:43:56.248 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.245 7fe14d7fa700 1 -- 172.21.2.112:0/2218615652 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe140006b00 con 0x7fe150086930 2021-02-10T11:43:56.249 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.245 7fe14d7fa700 1 --2- 172.21.2.112:0/2218615652 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe1340186c0 0x7fe13401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:56.249 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.245 7fe14d7fa700 1 -- 172.21.2.112:0/2218615652 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 5457+0+0 (secure 0 0 0) 0x7fe1400324f0 con 0x7fe150086930 2021-02-10T11:43:56.249 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.245 7fe1548f3700 1 --2- 172.21.2.112:0/2218615652 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe1340186c0 0x7fe13401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:56.249 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.246 7fe1548f3700 1 --2- 172.21.2.112:0/2218615652 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe1340186c0 0x7fe13401abc0 crc :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:56.277 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.276 7fe156ad7700 1 -- 172.21.2.112:0/2218615652 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe150082780 con 0x7fe150086930 2021-02-10T11:43:56.280 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.279 7fe14d7fa700 1 -- 172.21.2.112:0/2218615652 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe140006340 con 0x7fe150086930 2021-02-10T11:43:56.409 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.408 7fe156ad7700 1 -- 172.21.2.112:0/2218615652 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"} v 0) v1 -- 0x7fe150082780 con 0x7fe150086930 2021-02-10T11:43:56.412 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.412 7fe14d7fa700 1 -- 172.21.2.112:0/2218615652 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"}]=0 new fs with metadata pool 30 and data pool 31 v120) v1 ==== 189+0+0 (secure 0 0 0) 0x7fe14002dc60 con 0x7fe150086930 2021-02-10T11:43:56.416 INFO:teuthology.orchestra.run.gibba012.stderr:new fs with metadata pool 30 and data pool 31 2021-02-10T11:43:56.418 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.417 7fe156ad7700 1 -- 172.21.2.112:0/2218615652 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe1340186c0 msgr2=0x7fe13401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:56.418 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.417 7fe156ad7700 1 --2- 172.21.2.112:0/2218615652 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe1340186c0 0x7fe13401abc0 crc :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:56.418 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.417 7fe156ad7700 1 -- 172.21.2.112:0/2218615652 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe150086930 msgr2=0x7fe1501aab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:56.418 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.417 7fe156ad7700 1 --2- 172.21.2.112:0/2218615652 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe150086930 0x7fe1501aab40 secure :-1 s=READY pgs=2535 cs=0 l=1 rev1=1 rx=0x7fe140006f60 tx=0x7fe14000b970).stop 2021-02-10T11:43:56.419 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.418 7fe156ad7700 1 -- 172.21.2.112:0/2218615652 shutdown_connections 2021-02-10T11:43:56.419 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.418 7fe156ad7700 1 --2- 172.21.2.112:0/2218615652 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe1340186c0 0x7fe13401abc0 unknown :-1 s=CLOSED pgs=1164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:56.419 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.418 7fe156ad7700 1 --2- 172.21.2.112:0/2218615652 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe150086d30 0x7fe1501ab160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:56.419 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.418 7fe156ad7700 1 --2- 172.21.2.112:0/2218615652 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe150086930 0x7fe1501aab40 unknown :-1 s=CLOSED pgs=2535 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:56.419 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.418 7fe156ad7700 1 --2- 172.21.2.112:0/2218615652 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe1500a87a0 0x7fe15018b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:56.419 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.418 7fe156ad7700 1 -- 172.21.2.112:0/2218615652 >> 172.21.2.112:0/2218615652 conn(0x7fe15019d590 msgr2=0x7fe1501b1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:56.419 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.418 7fe156ad7700 1 -- 172.21.2.112:0/2218615652 shutdown_connections 2021-02-10T11:43:56.419 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.418 7fe156ad7700 1 -- 172.21.2.112:0/2218615652 wait complete. 2021-02-10T11:43:56.427 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:43:56.856 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.838 7f00e88cb700 1 Processor -- start 2021-02-10T11:43:56.856 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.838 7f00e88cb700 1 -- start start 2021-02-10T11:43:56.857 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.838 7f00e88cb700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f00e40d1c70 0x7f00e40f20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:56.857 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.838 7f00e88cb700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00e40f0b10 con 0x7f00e40d0340 2021-02-10T11:43:56.857 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.838 7f00e88cb700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00e41b0be0 con 0x7f00e40d1870 2021-02-10T11:43:56.857 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.838 7f00e88cb700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00e40e5ed0 con 0x7f00e40d1c70 2021-02-10T11:43:56.857 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.839 7f00e1e1b700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f00e40d0340 0x7f00e40ef640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:56932/0 (socket says 172.21.2.112:56932) 2021-02-10T11:43:56.857 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.839 7f00e1e1b700 1 -- 172.21.2.112:0/428641938 learned_addr learned my addr 172.21.2.112:0/428641938 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:43:56.857 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.839 7f00e2e1d700 1 --2- 172.21.2.112:0/428641938 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f00e40d1c70 0x7f00e40f20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:56.857 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.839 7f00e161a700 1 -- 172.21.2.112:0/428641938 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2363966407 0 0) 0x7f00cc001180 con 0x7f00e40d0340 2021-02-10T11:43:56.857 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.839 7f00e161a700 1 -- 172.21.2.112:0/428641938 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00d0002fe0 con 0x7f00e40d0340 2021-02-10T11:43:56.857 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.839 7f00e161a700 1 -- 172.21.2.112:0/428641938 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3645609442 0 0) 0x7f00d4001080 con 0x7f00e40d1870 2021-02-10T11:43:56.857 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.840 7f00e161a700 1 -- 172.21.2.112:0/428641938 --> v1:172.21.2.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00d0003210 con 0x7f00e40d1870 2021-02-10T11:43:56.857 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.840 7f00e161a700 1 -- 172.21.2.112:0/428641938 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2646499037 0 0) 0x7f00cc001180 con 0x7f00e40d0340 2021-02-10T11:43:56.857 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.840 7f00e161a700 1 -- 172.21.2.112:0/428641938 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f00e40d1c70 msgr2=0x7f00e40f20a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:56.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.840 7f00e161a700 1 --2- 172.21.2.112:0/428641938 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f00e40d1c70 0x7f00e40f20a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:56.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.840 7f00e161a700 1 -- 172.21.2.112:0/428641938 >> v1:172.21.2.120:6789/0 conn(0x7f00e40d1870 legacy=0x7f00e41b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:56.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.840 7f00e161a700 1 -- 172.21.2.112:0/428641938 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00e40d2fe0 con 0x7f00e40d0340 2021-02-10T11:43:56.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.840 7f00e161a700 1 -- 172.21.2.112:0/428641938 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f00cc001680 con 0x7f00e40d0340 2021-02-10T11:43:56.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.840 7f00e161a700 1 -- 172.21.2.112:0/428641938 >> v1:172.21.2.112:6789/0 conn(0x7f00e40d0340 legacy=0x7f00e40ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:56.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.840 7f00e161a700 1 --2- 172.21.2.112:0/428641938 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f00d0003b90 0x7f00d0005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:56.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.840 7f00e161a700 1 --2- 172.21.2.112:0/428641938 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f00d0006770 0x7f00d0008c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:56.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.840 7f00e161a700 1 --2- 172.21.2.112:0/428641938 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f00d0009510 0x7f00d000ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:56.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.840 7f00e161a700 1 -- 172.21.2.112:0/428641938 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00d000c2e0 con 0x7f00d0009510 2021-02-10T11:43:56.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.840 7f00e161a700 1 -- 172.21.2.112:0/428641938 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00d00504d0 con 0x7f00d0006770 2021-02-10T11:43:56.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.840 7f00e161a700 1 -- 172.21.2.112:0/428641938 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00d00506f0 con 0x7f00d0003b90 2021-02-10T11:43:56.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.840 7f00e2e1d700 1 --2- 172.21.2.112:0/428641938 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f00d0009510 0x7f00d000ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:56.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.840 7f00e1e1b700 1 --2- 172.21.2.112:0/428641938 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f00d0006770 0x7f00d0008c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:56.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.840 7f00e261c700 1 --2- 172.21.2.112:0/428641938 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f00d0003b90 0x7f00d0005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:56.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.840 7f00e2e1d700 1 -- 172.21.2.112:0/428641938 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f00d0003b90 msgr2=0x7f00d0005fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:56.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.840 7f00e2e1d700 1 --2- 172.21.2.112:0/428641938 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f00d0003b90 0x7f00d0005fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:56.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.840 7f00e2e1d700 1 -- 172.21.2.112:0/428641938 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f00d0006770 msgr2=0x7f00d0008c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:56.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.840 7f00e2e1d700 1 --2- 172.21.2.112:0/428641938 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f00d0006770 0x7f00d0008c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:56.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.840 7f00e2e1d700 1 -- 172.21.2.112:0/428641938 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00d0050990 con 0x7f00d0009510 2021-02-10T11:43:56.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.840 7f00e2e1d700 1 --2- 172.21.2.112:0/428641938 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f00d0009510 0x7f00d000ba30 secure :-1 s=READY pgs=2537 cs=0 l=1 rev1=1 rx=0x7f00dc009b10 tx=0x7f00dc009eb0).ready entity=mon.0 client_cookie=db4a25665498037 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:56.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.841 7f00e161a700 1 -- 172.21.2.112:0/428641938 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f00dc00b300 con 0x7f00d0009510 2021-02-10T11:43:56.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.841 7f00e161a700 1 -- 172.21.2.112:0/428641938 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00dc003030 con 0x7f00d0009510 2021-02-10T11:43:56.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.841 7f00e161a700 1 -- 172.21.2.112:0/428641938 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f00dc000f90 con 0x7f00d0009510 2021-02-10T11:43:56.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.841 7f00e88cb700 1 -- 172.21.2.112:0/428641938 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f00d0009510 msgr2=0x7f00d000ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:56.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.841 7f00e88cb700 1 --2- 172.21.2.112:0/428641938 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f00d0009510 0x7f00d000ba30 secure :-1 s=READY pgs=2537 cs=0 l=1 rev1=1 rx=0x7f00dc009b10 tx=0x7f00dc009eb0).stop 2021-02-10T11:43:56.860 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.841 7f00e2e1d700 1 -- 172.21.2.112:0/428641938 reap_dead start 2021-02-10T11:43:56.860 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.841 7f00e88cb700 1 -- 172.21.2.112:0/428641938 shutdown_connections 2021-02-10T11:43:56.860 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.841 7f00e88cb700 1 -- 172.21.2.112:0/428641938 >> 172.21.2.112:0/428641938 conn(0x7f00e40ce450 msgr2=0x7f00e40d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:56.860 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.841 7f00e88cb700 1 -- 172.21.2.112:0/428641938 shutdown_connections 2021-02-10T11:43:56.860 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.841 7f00e88cb700 1 -- 172.21.2.112:0/428641938 wait complete. 2021-02-10T11:43:56.860 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.856 7f00e88cb700 1 Processor -- start 2021-02-10T11:43:56.861 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.856 7f00e88cb700 1 -- start start 2021-02-10T11:43:56.862 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.856 7f00e88cb700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f00e4195650 0x7f00e40b5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:56.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.856 7f00e88cb700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f00e40b5b80 0x7f00e41a2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:56.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.856 7f00e88cb700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f00e40acfa0 0x7f00e41a3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:56.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.856 7f00e88cb700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00e40b6010 con 0x7f00e40b5b80 2021-02-10T11:43:56.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.856 7f00e88cb700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00e41a3a60 con 0x7f00e40acfa0 2021-02-10T11:43:56.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.856 7f00e88cb700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00e41a3c40 con 0x7f00e4195650 2021-02-10T11:43:56.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.856 7f00e1e1b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f00e40b5b80 0x7f00e41a2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:56.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.856 7f00e1e1b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f00e40b5b80 0x7f00e41a2e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:57480/0 (socket says 172.21.2.112:57480) 2021-02-10T11:43:56.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.856 7f00e1e1b700 1 -- 172.21.2.112:0/2485890161 learned_addr learned my addr 172.21.2.112:0/2485890161 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:43:56.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.857 7f00e1e1b700 1 -- 172.21.2.112:0/2485890161 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f00e4195650 msgr2=0x7f00e40b5560 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:43:56.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.857 7f00e1e1b700 1 --2- 172.21.2.112:0/2485890161 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f00e4195650 0x7f00e40b5560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:56.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.857 7f00e1e1b700 1 -- 172.21.2.112:0/2485890161 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f00e40acfa0 msgr2=0x7f00e41a3440 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:43:56.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.857 7f00e1e1b700 1 --2- 172.21.2.112:0/2485890161 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f00e40acfa0 0x7f00e41a3440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:56.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.857 7f00e1e1b700 1 -- 172.21.2.112:0/2485890161 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00e40bcae0 con 0x7f00e40b5b80 2021-02-10T11:43:56.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.857 7f00e1e1b700 1 --2- 172.21.2.112:0/2485890161 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f00e40b5b80 0x7f00e41a2e20 secure :-1 s=READY pgs=2538 cs=0 l=1 rev1=1 rx=0x7f00cc009610 tx=0x7f00cc0099b0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:56.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.857 7f00daffd700 1 -- 172.21.2.112:0/2485890161 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f00cc0070c0 con 0x7f00e40b5b80 2021-02-10T11:43:56.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.857 7f00daffd700 1 -- 172.21.2.112:0/2485890161 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00cc0072c0 con 0x7f00e40b5b80 2021-02-10T11:43:56.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.857 7f00daffd700 1 -- 172.21.2.112:0/2485890161 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f00cc004590 con 0x7f00e40b5b80 2021-02-10T11:43:56.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.857 7f00e88cb700 1 -- 172.21.2.112:0/2485890161 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00e40bcae0 con 0x7f00e40b5b80 2021-02-10T11:43:56.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.857 7f00e88cb700 1 -- 172.21.2.112:0/2485890161 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00e40bcae0 con 0x7f00e40b5b80 2021-02-10T11:43:56.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.858 7f00daffd700 1 -- 172.21.2.112:0/2485890161 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f00cc008000 con 0x7f00e40b5b80 2021-02-10T11:43:56.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.858 7f00daffd700 1 --2- 172.21.2.112:0/2485890161 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f00c80186c0 0x7f00c801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:56.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.858 7f00daffd700 1 -- 172.21.2.112:0/2485890161 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f00cc032550 con 0x7f00e40b5b80 2021-02-10T11:43:56.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.859 7f00e261c700 1 --2- 172.21.2.112:0/2485890161 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f00c80186c0 0x7f00c801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:56.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.859 7f00e261c700 1 --2- 172.21.2.112:0/2485890161 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f00c80186c0 0x7f00c801abc0 crc :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:56.888 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.888 7f00e88cb700 1 -- 172.21.2.112:0/2485890161 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00e41a3c40 con 0x7f00e40b5b80 2021-02-10T11:43:56.891 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:56.891 7f00daffd700 1 -- 172.21.2.112:0/2485890161 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f00cc009da0 con 0x7f00e40b5b80 2021-02-10T11:43:57.017 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:57.016 7f00e88cb700 1 -- 172.21.2.112:0/2485890161 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f00e41a3c40 con 0x7f00e40b5b80 2021-02-10T11:43:57.017 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:57.017 7f00daffd700 1 -- 172.21.2.112:0/2485890161 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v142) v1 ==== 74+0+14020 (secure 0 0 0) 0x7f00cc008000 con 0x7f00e40b5b80 2021-02-10T11:43:57.018 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:43:57.018 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":142,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:43:56.409750","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":31,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":30,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:43:54.385734","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"142","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":31,"pool_name":"cephfs_data","create_time":"2021-02-10 11:43:55.453740","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"142","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":139,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":139,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:43:57.019 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:57.019 7f00e88cb700 1 -- 172.21.2.112:0/2485890161 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f00c80186c0 msgr2=0x7f00c801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:57.019 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:57.019 7f00e88cb700 1 --2- 172.21.2.112:0/2485890161 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f00c80186c0 0x7f00c801abc0 crc :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:57.019 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:57.019 7f00e88cb700 1 -- 172.21.2.112:0/2485890161 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f00e40b5b80 msgr2=0x7f00e41a2e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:57.019 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:57.019 7f00e88cb700 1 --2- 172.21.2.112:0/2485890161 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f00e40b5b80 0x7f00e41a2e20 secure :-1 s=READY pgs=2538 cs=0 l=1 rev1=1 rx=0x7f00cc009610 tx=0x7f00cc0099b0).stop 2021-02-10T11:43:57.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:57.019 7f00e88cb700 1 -- 172.21.2.112:0/2485890161 shutdown_connections 2021-02-10T11:43:57.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:57.019 7f00e88cb700 1 --2- 172.21.2.112:0/2485890161 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f00c80186c0 0x7f00c801abc0 unknown :-1 s=CLOSED pgs=1165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:57.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:57.019 7f00e88cb700 1 --2- 172.21.2.112:0/2485890161 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f00e40acfa0 0x7f00e41a3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:57.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:57.019 7f00e88cb700 1 --2- 172.21.2.112:0/2485890161 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f00e40b5b80 0x7f00e41a2e20 unknown :-1 s=CLOSED pgs=2538 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:57.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:57.019 7f00e88cb700 1 --2- 172.21.2.112:0/2485890161 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f00e4195650 0x7f00e40b5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:57.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:57.019 7f00e88cb700 1 -- 172.21.2.112:0/2485890161 >> 172.21.2.112:0/2485890161 conn(0x7f00e40b3040 msgr2=0x7f00e40c3250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:57.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:57.019 7f00e88cb700 1 -- 172.21.2.112:0/2485890161 shutdown_connections 2021-02-10T11:43:57.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:57.019 7f00e88cb700 1 -- 172.21.2.112:0/2485890161 wait complete. 2021-02-10T11:43:57.028 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:43:57.457 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:57.439 7fe328e25700 1 Processor -- start 2021-02-10T11:43:57.457 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:57.439 7fe328e25700 1 -- start start 2021-02-10T11:43:57.457 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:57.440 7fe328e25700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fe3240d0340 0x7fe3240ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:57.457 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:57.440 7fe328e25700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe3240d1c70 0x7fe3240efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:57.457 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:57.440 7fe328e25700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3240f0b10 con 0x7fe3240d1870 2021-02-10T11:43:57.457 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:57.440 7fe328e25700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3240e5ed0 con 0x7fe3240d0340 2021-02-10T11:43:57.457 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:57.440 7fe328e25700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3241b0be0 con 0x7fe3240d1c70 2021-02-10T11:43:57.457 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:57.440 7fe322312700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fe3240d0340 0x7fe3240ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:57.458 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:57.440 7fe322312700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fe3240d0340 0x7fe3240ef640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:57488/0 (socket says 172.21.2.112:57488) 2021-02-10T11:43:57.458 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:57.440 7fe322312700 1 -- 172.21.2.112:0/2082982802 learned_addr learned my addr 172.21.2.112:0/2082982802 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:43:57.458 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:57.440 7fe322b13700 1 --1- 172.21.2.112:0/2082982802 >> v1:172.21.2.112:6789/0 conn(0x7fe3240d1870 0x7fe3241b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:56950/0 (socket says 172.21.2.112:56950) 2021-02-10T11:43:57.458 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:57.440 7fe323314700 1 --2- 172.21.2.112:0/2082982802 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe3240d1c70 0x7fe3240efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:57.458 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:57.441 7fe323314700 1 -- 172.21.2.112:0/2082982802 >> v2:172.21.2.112:3300/0 conn(0x7fe3240d0340 msgr2=0x7fe3240ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:57.458 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:57.441 7fe323314700 1 --2- 172.21.2.112:0/2082982802 >> v2:172.21.2.112:3300/0 conn(0x7fe3240d0340 0x7fe3240ef640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:57.458 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:57.441 7fe323314700 1 -- 172.21.2.112:0/2082982802 >> v1:172.21.2.112:6789/0 conn(0x7fe3240d1870 legacy=0x7fe3241b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:57.458 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:57.441 7fe323314700 1 -- 172.21.2.112:0/2082982802 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3240f20a0 con 0x7fe3240d1c70 2021-02-10T11:43:57.458 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:57.441 7fe322312700 1 --2- 172.21.2.112:0/2082982802 >> v2:172.21.2.112:3300/0 conn(0x7fe3240d0340 0x7fe3240ef640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:43:57.458 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:57.441 7fe323314700 1 --2- 172.21.2.112:0/2082982802 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe3240d1c70 0x7fe3240efc60 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7fe31c006e50 tx=0x7fe31c009e70).ready entity=mon.2 client_cookie=945d3bed98a03ee1 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:57.458 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:57.441 7fe321b11700 1 -- 172.21.2.112:0/2082982802 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe31c006a20 con 0x7fe3240d1c70 2021-02-10T11:43:57.458 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:57.441 7fe321b11700 1 -- 172.21.2.112:0/2082982802 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe31c006790 con 0x7fe3240d1c70 2021-02-10T11:43:57.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:57.441 7fe321b11700 1 -- 172.21.2.112:0/2082982802 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe31c00b700 con 0x7fe3240d1c70 2021-02-10T11:43:57.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:57.441 7fe328e25700 1 -- 172.21.2.112:0/2082982802 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe3240d1c70 msgr2=0x7fe3240efc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:57.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:57.441 7fe328e25700 1 --2- 172.21.2.112:0/2082982802 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe3240d1c70 0x7fe3240efc60 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7fe31c006e50 tx=0x7fe31c009e70).stop 2021-02-10T11:43:57.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:57.442 7fe328e25700 1 -- 172.21.2.112:0/2082982802 shutdown_connections 2021-02-10T11:43:57.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:57.442 7fe328e25700 1 --2- 172.21.2.112:0/2082982802 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe3240d1c70 0x7fe3240efc60 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:57.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:57.442 7fe328e25700 1 --2- 172.21.2.112:0/2082982802 >> v2:172.21.2.112:3300/0 conn(0x7fe3240d0340 0x7fe3240ef640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:57.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:57.442 7fe328e25700 1 -- 172.21.2.112:0/2082982802 >> 172.21.2.112:0/2082982802 conn(0x7fe3240ce450 msgr2=0x7fe3240d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:57.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:57.442 7fe328e25700 1 -- 172.21.2.112:0/2082982802 shutdown_connections 2021-02-10T11:43:57.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:57.442 7fe328e25700 1 -- 172.21.2.112:0/2082982802 wait complete. 2021-02-10T11:43:57.461 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:57.457 7fe328e25700 1 Processor -- start 2021-02-10T11:43:57.461 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:57.457 7fe328e25700 1 -- start start 2021-02-10T11:43:57.461 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:57.457 7fe328e25700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe324093f10 0x7fe324094310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:57.461 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:57.457 7fe328e25700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe3240ae9e0 0x7fe3240aede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:57.461 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:57.457 7fe328e25700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe324092d90 0x7fe324093190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:57.461 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:57.457 7fe328e25700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe32419ebc0 con 0x7fe324092d90 2021-02-10T11:43:57.461 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:57.457 7fe328e25700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3241b8760 con 0x7fe324093f10 2021-02-10T11:43:57.461 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:57.457 7fe328e25700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3240937b0 con 0x7fe3240ae9e0 2021-02-10T11:43:57.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:57.457 7fe323314700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe324092d90 0x7fe324093190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:57.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:57.457 7fe323314700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe324092d90 0x7fe324093190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:57496/0 (socket says 172.21.2.112:57496) 2021-02-10T11:43:57.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:57.457 7fe323314700 1 -- 172.21.2.112:0/66612340 learned_addr learned my addr 172.21.2.112:0/66612340 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:43:57.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:57.458 7fe322b13700 1 --2- 172.21.2.112:0/66612340 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe324093f10 0x7fe324094310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:57.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:57.458 7fe322312700 1 --2- 172.21.2.112:0/66612340 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe3240ae9e0 0x7fe3240aede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:57.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:57.458 7fe323314700 1 -- 172.21.2.112:0/66612340 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe3240ae9e0 msgr2=0x7fe3240aede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:57.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:57.458 7fe323314700 1 --2- 172.21.2.112:0/66612340 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe3240ae9e0 0x7fe3240aede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:57.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:57.458 7fe323314700 1 -- 172.21.2.112:0/66612340 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe324093f10 msgr2=0x7fe324094310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:57.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:57.458 7fe323314700 1 --2- 172.21.2.112:0/66612340 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe324093f10 0x7fe324094310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:57.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:57.458 7fe323314700 1 -- 172.21.2.112:0/66612340 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3240939d0 con 0x7fe324092d90 2021-02-10T11:43:57.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:57.458 7fe322b13700 1 --2- 172.21.2.112:0/66612340 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe324093f10 0x7fe324094310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:43:57.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:57.458 7fe323314700 1 --2- 172.21.2.112:0/66612340 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe324092d90 0x7fe324093190 secure :-1 s=READY pgs=2540 cs=0 l=1 rev1=1 rx=0x7fe31c0099f0 tx=0x7fe31c00bc90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:57.463 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:57.458 7fe31b7fe700 1 -- 172.21.2.112:0/66612340 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe31c0090c0 con 0x7fe324092d90 2021-02-10T11:43:57.463 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:57.458 7fe31b7fe700 1 -- 172.21.2.112:0/66612340 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe31c000df0 con 0x7fe324092d90 2021-02-10T11:43:57.463 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:57.458 7fe328e25700 1 -- 172.21.2.112:0/66612340 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3240939d0 con 0x7fe324092d90 2021-02-10T11:43:57.463 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:57.458 7fe328e25700 1 -- 172.21.2.112:0/66612340 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe324189cd0 con 0x7fe324092d90 2021-02-10T11:43:57.463 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:57.458 7fe31b7fe700 1 -- 172.21.2.112:0/66612340 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe31c001420 con 0x7fe324092d90 2021-02-10T11:43:57.463 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:57.459 7fe31b7fe700 1 -- 172.21.2.112:0/66612340 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe31c0011a0 con 0x7fe324092d90 2021-02-10T11:43:57.463 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:57.459 7fe31b7fe700 1 --2- 172.21.2.112:0/66612340 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe3080186d0 0x7fe30801abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:57.463 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:57.459 7fe31b7fe700 1 -- 172.21.2.112:0/66612340 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fe31c032b30 con 0x7fe324092d90 2021-02-10T11:43:57.463 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:57.459 7fe322b13700 1 --2- 172.21.2.112:0/66612340 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe3080186d0 0x7fe30801abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:57.463 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:57.460 7fe322b13700 1 --2- 172.21.2.112:0/66612340 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe3080186d0 0x7fe30801abd0 crc :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:57.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:57.490 7fe328e25700 1 -- 172.21.2.112:0/66612340 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3240ebc60 con 0x7fe324092d90 2021-02-10T11:43:57.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:57.492 7fe31b7fe700 1 -- 172.21.2.112:0/66612340 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe31c0093d0 con 0x7fe324092d90 2021-02-10T11:43:57.619 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:57.619 7fe328e25700 1 -- 172.21.2.112:0/66612340 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe3240f0b10 con 0x7fe324092d90 2021-02-10T11:43:57.620 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:57.619 7fe31b7fe700 1 -- 172.21.2.112:0/66612340 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v143) v1 ==== 74+0+14007 (secure 0 0 0) 0x7fe31c0011a0 con 0x7fe324092d90 2021-02-10T11:43:57.620 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:43:57.620 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":143,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:43:57.411337","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":31,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":30,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:43:54.385734","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"142","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":31,"pool_name":"cephfs_data","create_time":"2021-02-10 11:43:55.453740","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"143","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":139,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":139,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:43:57.621 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:57.621 7fe328e25700 1 -- 172.21.2.112:0/66612340 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe3080186d0 msgr2=0x7fe30801abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:57.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:57.621 7fe328e25700 1 --2- 172.21.2.112:0/66612340 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe3080186d0 0x7fe30801abd0 crc :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:57.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:57.621 7fe328e25700 1 -- 172.21.2.112:0/66612340 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe324092d90 msgr2=0x7fe324093190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:57.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:57.621 7fe328e25700 1 --2- 172.21.2.112:0/66612340 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe324092d90 0x7fe324093190 secure :-1 s=READY pgs=2540 cs=0 l=1 rev1=1 rx=0x7fe31c0099f0 tx=0x7fe31c00bc90).stop 2021-02-10T11:43:57.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:57.621 7fe328e25700 1 -- 172.21.2.112:0/66612340 shutdown_connections 2021-02-10T11:43:57.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:57.621 7fe328e25700 1 --2- 172.21.2.112:0/66612340 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe3080186d0 0x7fe30801abd0 unknown :-1 s=CLOSED pgs=1166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:57.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:57.621 7fe328e25700 1 --2- 172.21.2.112:0/66612340 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe324092d90 0x7fe324093190 unknown :-1 s=CLOSED pgs=2540 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:57.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:57.621 7fe328e25700 1 --2- 172.21.2.112:0/66612340 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe3240ae9e0 0x7fe3240aede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:57.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:57.621 7fe328e25700 1 --2- 172.21.2.112:0/66612340 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe324093f10 0x7fe324094310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:57.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:57.621 7fe328e25700 1 -- 172.21.2.112:0/66612340 >> 172.21.2.112:0/66612340 conn(0x7fe32419f8a0 msgr2=0x7fe324094980 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:57.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:57.621 7fe328e25700 1 -- 172.21.2.112:0/66612340 shutdown_connections 2021-02-10T11:43:57.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:57.621 7fe328e25700 1 -- 172.21.2.112:0/66612340 wait complete. 2021-02-10T11:43:57.631 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2021-02-10T11:43:58.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.045 7f3fb224c700 1 Processor -- start 2021-02-10T11:43:58.063 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.045 7f3fb224c700 1 -- start start 2021-02-10T11:43:58.063 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.046 7f3fb224c700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f3fac0d1870 0x7f3fac1b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:58.063 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.046 7f3fb224c700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3fac1b0ae0 0x7f3fac108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:58.063 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.046 7f3fb224c700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f3fac0d1c70 0x7f3fac108f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:58.063 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.046 7f3fb224c700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fac0fe1c0 con 0x7f3fac0d1870 2021-02-10T11:43:58.063 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.046 7f3fb224c700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fac109a60 con 0x7f3fac0d1c70 2021-02-10T11:43:58.063 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.046 7f3fb224c700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fac088070 con 0x7f3fac1b0ae0 2021-02-10T11:43:58.063 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.046 7f3fabfff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f3fac0d1870 0x7f3fac1b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:58.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.046 7f3fabfff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f3fac0d1870 0x7f3fac1b04c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:57502/0 (socket says 172.21.2.112:57502) 2021-02-10T11:43:58.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.046 7f3fabfff700 1 -- 172.21.2.112:0/2969956485 learned_addr learned my addr 172.21.2.112:0/2969956485 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:43:58.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.046 7f3fabfff700 1 -- 172.21.2.112:0/2969956485 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3fac1b0ae0 msgr2=0x7f3fac108940 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:43:58.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.046 7f3fabfff700 1 --2- 172.21.2.112:0/2969956485 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3fac1b0ae0 0x7f3fac108940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:58.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.046 7f3fabfff700 1 -- 172.21.2.112:0/2969956485 >> v2:172.21.2.120:3300/0 conn(0x7f3fac0d1c70 msgr2=0x7f3fac108f60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:43:58.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.046 7f3fabfff700 1 --2- 172.21.2.112:0/2969956485 >> v2:172.21.2.120:3300/0 conn(0x7f3fac0d1c70 0x7f3fac108f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:58.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.046 7f3fabfff700 1 -- 172.21.2.112:0/2969956485 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fac109580 con 0x7f3fac0d1870 2021-02-10T11:43:58.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.046 7f3fabfff700 1 --2- 172.21.2.112:0/2969956485 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3fac0d1870 0x7f3fac1b04c0 secure :-1 s=READY pgs=2541 cs=0 l=1 rev1=1 rx=0x7f3f9c006a80 tx=0x7f3f9c009f00).ready entity=mon.0 client_cookie=2ed548ccade9ac4 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:58.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.047 7f3faaffd700 1 -- 172.21.2.112:0/2969956485 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3f9c00b5c0 con 0x7f3fac0d1870 2021-02-10T11:43:58.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.047 7f3faaffd700 1 -- 172.21.2.112:0/2969956485 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f9c00b8c0 con 0x7f3fac0d1870 2021-02-10T11:43:58.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.047 7f3faaffd700 1 -- 172.21.2.112:0/2969956485 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3f9c00bde0 con 0x7f3fac0d1870 2021-02-10T11:43:58.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.047 7f3fb224c700 1 -- 172.21.2.112:0/2969956485 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3fac0d1870 msgr2=0x7f3fac1b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:58.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.047 7f3fb224c700 1 --2- 172.21.2.112:0/2969956485 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3fac0d1870 0x7f3fac1b04c0 secure :-1 s=READY pgs=2541 cs=0 l=1 rev1=1 rx=0x7f3f9c006a80 tx=0x7f3f9c009f00).stop 2021-02-10T11:43:58.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.048 7f3fb224c700 1 -- 172.21.2.112:0/2969956485 shutdown_connections 2021-02-10T11:43:58.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.048 7f3fb224c700 1 --2- 172.21.2.112:0/2969956485 >> v2:172.21.2.120:3300/0 conn(0x7f3fac0d1c70 0x7f3fac108f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:58.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.048 7f3fb224c700 1 --2- 172.21.2.112:0/2969956485 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3fac1b0ae0 0x7f3fac108940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:58.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.048 7f3fb224c700 1 --2- 172.21.2.112:0/2969956485 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3fac0d1870 0x7f3fac1b04c0 unknown :-1 s=CLOSED pgs=2541 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:58.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.048 7f3fb224c700 1 -- 172.21.2.112:0/2969956485 >> 172.21.2.112:0/2969956485 conn(0x7f3fac0ce420 msgr2=0x7f3fac0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:58.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.048 7f3fb224c700 1 -- 172.21.2.112:0/2969956485 shutdown_connections 2021-02-10T11:43:58.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.048 7f3fb224c700 1 -- 172.21.2.112:0/2969956485 wait complete. 2021-02-10T11:43:58.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.062 7f3fb224c700 1 Processor -- start 2021-02-10T11:43:58.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.062 7f3fb224c700 1 -- start start 2021-02-10T11:43:58.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.063 7f3fb224c700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3fac0c3820 0x7f3fac084cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:58.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.063 7f3fb224c700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3fac0a0290 0x7f3fac0852d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:58.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.063 7f3fb224c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3fac086270 0x7f3fac086670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:58.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.063 7f3fb224c700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fac0c0700 con 0x7f3fac086270 2021-02-10T11:43:58.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.063 7f3fb224c700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fac0858f0 con 0x7f3fac0c3820 2021-02-10T11:43:58.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.063 7f3fb224c700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fac085b10 con 0x7f3fac0a0290 2021-02-10T11:43:58.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.063 7f3fb0869700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3fac086270 0x7f3fac086670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:58.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.063 7f3fb0869700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3fac086270 0x7f3fac086670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:57510/0 (socket says 172.21.2.112:57510) 2021-02-10T11:43:58.068 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.063 7f3fb0869700 1 -- 172.21.2.112:0/2523677393 learned_addr learned my addr 172.21.2.112:0/2523677393 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:43:58.068 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.063 7f3fb0869700 1 -- 172.21.2.112:0/2523677393 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3fac0a0290 msgr2=0x7f3fac0852d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:58.068 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.063 7f3fb0869700 1 --2- 172.21.2.112:0/2523677393 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3fac0a0290 0x7f3fac0852d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:58.068 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.063 7f3fb0869700 1 -- 172.21.2.112:0/2523677393 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3fac0c3820 msgr2=0x7f3fac084cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:58.068 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.063 7f3fb0869700 1 --2- 172.21.2.112:0/2523677393 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3fac0c3820 0x7f3fac084cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:58.068 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.063 7f3fb0869700 1 -- 172.21.2.112:0/2523677393 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fac086c90 con 0x7f3fac086270 2021-02-10T11:43:58.068 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.064 7f3fb0869700 1 --2- 172.21.2.112:0/2523677393 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3fac086270 0x7f3fac086670 secure :-1 s=READY pgs=2542 cs=0 l=1 rev1=1 rx=0x7f3fa400b5c0 tx=0x7f3fa400b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:58.068 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.064 7f3fa8ff9700 1 -- 172.21.2.112:0/2523677393 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3fa400bd50 con 0x7f3fac086270 2021-02-10T11:43:58.068 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.064 7f3fa8ff9700 1 -- 172.21.2.112:0/2523677393 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fa4006030 con 0x7f3fac086270 2021-02-10T11:43:58.068 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.064 7f3fb224c700 1 -- 172.21.2.112:0/2523677393 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fac086c90 con 0x7f3fac086270 2021-02-10T11:43:58.068 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.064 7f3fb224c700 1 -- 172.21.2.112:0/2523677393 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fac1a2f00 con 0x7f3fac086270 2021-02-10T11:43:58.069 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.064 7f3fa8ff9700 1 -- 172.21.2.112:0/2523677393 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3fa40054d0 con 0x7f3fac086270 2021-02-10T11:43:58.069 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.065 7f3fa8ff9700 1 -- 172.21.2.112:0/2523677393 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3fa4006210 con 0x7f3fac086270 2021-02-10T11:43:58.069 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.065 7f3fa8ff9700 1 --2- 172.21.2.112:0/2523677393 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3f900186c0 0x7f3f9001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:58.069 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.065 7f3fa8ff9700 1 -- 172.21.2.112:0/2523677393 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f3fa4032590 con 0x7f3fac086270 2021-02-10T11:43:58.069 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.065 7f3fabfff700 1 --2- 172.21.2.112:0/2523677393 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3f900186c0 0x7f3f9001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:58.069 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.065 7f3fabfff700 1 --2- 172.21.2.112:0/2523677393 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3f900186c0 0x7f3f9001abc0 crc :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:58.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.095 7f3fb224c700 1 -- 172.21.2.112:0/2523677393 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fac082a70 con 0x7f3fac086270 2021-02-10T11:43:58.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.098 7f3fa8ff9700 1 -- 172.21.2.112:0/2523677393 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3fa4000940 con 0x7f3fac086270 2021-02-10T11:43:58.237 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.237 7f3fb224c700 1 -- 172.21.2.112:0/2523677393 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"} v 0) v1 -- 0x7f3fac082a70 con 0x7f3fac086270 2021-02-10T11:43:58.419 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.418 7f3fa8ff9700 1 -- 172.21.2.112:0/2523677393 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"}]=0 v122) v1 ==== 118+0+0 (secure 0 0 0) 0x7f3fa4006210 con 0x7f3fac086270 2021-02-10T11:43:58.453 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.453 7f3fb224c700 1 -- 172.21.2.112:0/2523677393 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3f900186c0 msgr2=0x7f3f9001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:58.453 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.453 7f3fb224c700 1 --2- 172.21.2.112:0/2523677393 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3f900186c0 0x7f3f9001abc0 crc :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:58.454 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.453 7f3fb224c700 1 -- 172.21.2.112:0/2523677393 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3fac086270 msgr2=0x7f3fac086670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:58.454 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.453 7f3fb224c700 1 --2- 172.21.2.112:0/2523677393 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3fac086270 0x7f3fac086670 secure :-1 s=READY pgs=2542 cs=0 l=1 rev1=1 rx=0x7f3fa400b5c0 tx=0x7f3fa400b960).stop 2021-02-10T11:43:58.454 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.453 7f3fb224c700 1 -- 172.21.2.112:0/2523677393 shutdown_connections 2021-02-10T11:43:58.454 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.453 7f3fb224c700 1 --2- 172.21.2.112:0/2523677393 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3f900186c0 0x7f3f9001abc0 unknown :-1 s=CLOSED pgs=1167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:58.454 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.453 7f3fb224c700 1 --2- 172.21.2.112:0/2523677393 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3fac086270 0x7f3fac086670 unknown :-1 s=CLOSED pgs=2542 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:58.454 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.453 7f3fb224c700 1 --2- 172.21.2.112:0/2523677393 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3fac0a0290 0x7f3fac0852d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:58.454 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.453 7f3fb224c700 1 --2- 172.21.2.112:0/2523677393 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3fac0c3820 0x7f3fac084cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:58.454 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.453 7f3fb224c700 1 -- 172.21.2.112:0/2523677393 >> 172.21.2.112:0/2523677393 conn(0x7f3fac0872c0 msgr2=0x7f3fac095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:58.454 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.453 7f3fb224c700 1 -- 172.21.2.112:0/2523677393 shutdown_connections 2021-02-10T11:43:58.455 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.453 7f3fb224c700 1 -- 172.21.2.112:0/2523677393 wait complete. 2021-02-10T11:43:58.462 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:43:58.891 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.873 7f7301cf6700 1 Processor -- start 2021-02-10T11:43:58.892 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.874 7f7301cf6700 1 -- start start 2021-02-10T11:43:58.892 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.874 7f7301cf6700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f72fc0d1870 0x7f72fc1b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:58.892 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.874 7f7301cf6700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f72fc0f0650 0x7f72fc0cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:58.892 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.874 7f7301cf6700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72fc0f0b10 con 0x7f72fc1b0a10 2021-02-10T11:43:58.892 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.874 7f7301cf6700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72fc0fe230 con 0x7f72fc0f0650 2021-02-10T11:43:58.892 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.874 7f7301cf6700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72fc0e5ed0 con 0x7f72fc0d1870 2021-02-10T11:43:58.892 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.874 7f72faffd700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f72fc1b0a10 0x7f72fc0cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:56976/0 (socket says 172.21.2.112:56976) 2021-02-10T11:43:58.892 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.874 7f72faffd700 1 -- 172.21.2.112:0/3962418402 learned_addr learned my addr 172.21.2.112:0/3962418402 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:43:58.892 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.874 7f72fa7fc700 1 -- 172.21.2.112:0/3962418402 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 734052800 0 0) 0x7f72e40010b0 con 0x7f72fc1b0a10 2021-02-10T11:43:58.893 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.875 7f72fa7fc700 1 -- 172.21.2.112:0/3962418402 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72e8002fe0 con 0x7f72fc1b0a10 2021-02-10T11:43:58.893 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.875 7f72fbfff700 1 --2- 172.21.2.112:0/3962418402 >> v2:172.21.2.120:3300/0 conn(0x7f72fc0f0650 0x7f72fc0cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:58.893 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.875 7f72fb7fe700 1 --2- 172.21.2.112:0/3962418402 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f72fc0d1870 0x7f72fc1b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:58.893 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.875 7f72fa7fc700 1 -- 172.21.2.112:0/3962418402 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 184894683 0 0) 0x7f72e40010b0 con 0x7f72fc1b0a10 2021-02-10T11:43:58.893 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.875 7f72fa7fc700 1 -- 172.21.2.112:0/3962418402 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f72fc0d1870 msgr2=0x7f72fc1b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:58.893 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.875 7f72fa7fc700 1 --2- 172.21.2.112:0/3962418402 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f72fc0d1870 0x7f72fc1b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:58.893 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.875 7f72fa7fc700 1 -- 172.21.2.112:0/3962418402 >> v2:172.21.2.120:3300/0 conn(0x7f72fc0f0650 msgr2=0x7f72fc0cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:58.893 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.875 7f72fa7fc700 1 --2- 172.21.2.112:0/3962418402 >> v2:172.21.2.120:3300/0 conn(0x7f72fc0f0650 0x7f72fc0cfb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:58.893 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.875 7f72fa7fc700 1 -- 172.21.2.112:0/3962418402 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72fc0f20a0 con 0x7f72fc1b0a10 2021-02-10T11:43:58.893 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.875 7f72fa7fc700 1 -- 172.21.2.112:0/3962418402 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f72e40015b0 con 0x7f72fc1b0a10 2021-02-10T11:43:58.893 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.875 7f72fa7fc700 1 -- 172.21.2.112:0/3962418402 >> v1:172.21.2.112:6789/0 conn(0x7f72fc1b0a10 legacy=0x7f72fc0cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:58.893 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.875 7f72fa7fc700 1 --2- 172.21.2.112:0/3962418402 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f72e8003a60 0x7f72e8005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:58.893 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.875 7f72fa7fc700 1 --2- 172.21.2.112:0/3962418402 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f72e804a7a0 0x7f72e804cca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:58.893 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.875 7f72fa7fc700 1 --2- 172.21.2.112:0/3962418402 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f72e804d5a0 0x7f72e804fa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:58.894 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.875 7f72fa7fc700 1 -- 172.21.2.112:0/3962418402 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72e80502e0 con 0x7f72e804a7a0 2021-02-10T11:43:58.894 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.875 7f72fa7fc700 1 -- 172.21.2.112:0/3962418402 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72e8050500 con 0x7f72e8003a60 2021-02-10T11:43:58.894 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.875 7f72fa7fc700 1 -- 172.21.2.112:0/3962418402 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72e8050720 con 0x7f72e804d5a0 2021-02-10T11:43:58.894 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.875 7f72faffd700 1 --2- 172.21.2.112:0/3962418402 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f72e804a7a0 0x7f72e804cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:58.894 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.875 7f72fbfff700 1 --2- 172.21.2.112:0/3962418402 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f72e804d5a0 0x7f72e804fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:58.894 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.875 7f72fb7fe700 1 --2- 172.21.2.112:0/3962418402 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f72e8003a60 0x7f72e8005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:58.894 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.875 7f72faffd700 1 -- 172.21.2.112:0/3962418402 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f72e804d5a0 msgr2=0x7f72e804fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:58.894 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.875 7f72faffd700 1 --2- 172.21.2.112:0/3962418402 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f72e804d5a0 0x7f72e804fa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:58.894 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.875 7f72faffd700 1 -- 172.21.2.112:0/3962418402 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f72e8003a60 msgr2=0x7f72e8005ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:58.894 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.875 7f72faffd700 1 --2- 172.21.2.112:0/3962418402 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f72e8003a60 0x7f72e8005ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:58.894 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.875 7f72faffd700 1 -- 172.21.2.112:0/3962418402 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72e80509c0 con 0x7f72e804a7a0 2021-02-10T11:43:58.894 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.876 7f72faffd700 1 --2- 172.21.2.112:0/3962418402 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f72e804a7a0 0x7f72e804cca0 secure :-1 s=READY pgs=2544 cs=0 l=1 rev1=1 rx=0x7f72e40017d0 tx=0x7f72e4007cb0).ready entity=mon.0 client_cookie=7bf3b1c08868aa88 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:58.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.876 7f72fa7fc700 1 -- 172.21.2.112:0/3962418402 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f72e400a310 con 0x7f72e804a7a0 2021-02-10T11:43:58.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.876 7f72fa7fc700 1 -- 172.21.2.112:0/3962418402 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72e400a590 con 0x7f72e804a7a0 2021-02-10T11:43:58.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.876 7f72fa7fc700 1 -- 172.21.2.112:0/3962418402 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f72e400a9e0 con 0x7f72e804a7a0 2021-02-10T11:43:58.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.876 7f7301cf6700 1 -- 172.21.2.112:0/3962418402 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f72e804a7a0 msgr2=0x7f72e804cca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:58.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.876 7f7301cf6700 1 --2- 172.21.2.112:0/3962418402 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f72e804a7a0 0x7f72e804cca0 secure :-1 s=READY pgs=2544 cs=0 l=1 rev1=1 rx=0x7f72e40017d0 tx=0x7f72e4007cb0).stop 2021-02-10T11:43:58.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.876 7f72fbfff700 1 -- 172.21.2.112:0/3962418402 reap_dead start 2021-02-10T11:43:58.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.876 7f7301cf6700 1 -- 172.21.2.112:0/3962418402 shutdown_connections 2021-02-10T11:43:58.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.876 7f7301cf6700 1 -- 172.21.2.112:0/3962418402 >> 172.21.2.112:0/3962418402 conn(0x7f72fc0ce450 msgr2=0x7f72fc0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:58.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.876 7f7301cf6700 1 -- 172.21.2.112:0/3962418402 shutdown_connections 2021-02-10T11:43:58.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.876 7f7301cf6700 1 -- 172.21.2.112:0/3962418402 wait complete. 2021-02-10T11:43:58.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.891 7f7301cf6700 1 Processor -- start 2021-02-10T11:43:58.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.891 7f7301cf6700 1 -- start start 2021-02-10T11:43:58.897 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.891 7f7301cf6700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f72fc0b2860 0x7f72fc1a4480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:58.897 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.891 7f7301cf6700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f72fc1a4aa0 0x7f72fc0ec620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:58.897 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.891 7f7301cf6700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f72fc0ecc40 0x7f72fc097790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:58.897 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.891 7f7301cf6700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72fc0b2ce0 con 0x7f72fc0ecc40 2021-02-10T11:43:58.897 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.891 7f7301cf6700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72fc0ed040 con 0x7f72fc1a4aa0 2021-02-10T11:43:58.897 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.891 7f7301cf6700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72fc0ed260 con 0x7f72fc0b2860 2021-02-10T11:43:58.897 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.892 7f72fbfff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f72fc0ecc40 0x7f72fc097790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:58.897 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.892 7f72fbfff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f72fc0ecc40 0x7f72fc097790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:57530/0 (socket says 172.21.2.112:57530) 2021-02-10T11:43:58.897 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.892 7f72fbfff700 1 -- 172.21.2.112:0/2105948870 learned_addr learned my addr 172.21.2.112:0/2105948870 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:43:58.897 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.892 7f72fbfff700 1 -- 172.21.2.112:0/2105948870 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f72fc0b2860 msgr2=0x7f72fc1a4480 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:43:58.897 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.892 7f72fbfff700 1 --2- 172.21.2.112:0/2105948870 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f72fc0b2860 0x7f72fc1a4480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:58.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.892 7f72fbfff700 1 -- 172.21.2.112:0/2105948870 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f72fc1a4aa0 msgr2=0x7f72fc0ec620 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:43:58.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.892 7f72fbfff700 1 --2- 172.21.2.112:0/2105948870 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f72fc1a4aa0 0x7f72fc0ec620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:58.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.892 7f72fbfff700 1 -- 172.21.2.112:0/2105948870 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72fc097db0 con 0x7f72fc0ecc40 2021-02-10T11:43:58.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.892 7f72fbfff700 1 --2- 172.21.2.112:0/2105948870 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f72fc0ecc40 0x7f72fc097790 secure :-1 s=READY pgs=2545 cs=0 l=1 rev1=1 rx=0x7f72f40099e0 tx=0x7f72f400bd20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:58.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.892 7f72f3fff700 1 -- 172.21.2.112:0/2105948870 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f72f4001400 con 0x7f72fc0ecc40 2021-02-10T11:43:58.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.892 7f72f3fff700 1 -- 172.21.2.112:0/2105948870 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72f400be60 con 0x7f72fc0ecc40 2021-02-10T11:43:58.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.892 7f72f3fff700 1 -- 172.21.2.112:0/2105948870 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f72f4001680 con 0x7f72fc0ecc40 2021-02-10T11:43:58.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.892 7f7301cf6700 1 -- 172.21.2.112:0/2105948870 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72fc097db0 con 0x7f72fc0ecc40 2021-02-10T11:43:58.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.892 7f7301cf6700 1 -- 172.21.2.112:0/2105948870 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72fc097db0 con 0x7f72fc0ecc40 2021-02-10T11:43:58.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.893 7f72f3fff700 1 -- 172.21.2.112:0/2105948870 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f72f40011f0 con 0x7f72fc0ecc40 2021-02-10T11:43:58.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.893 7f72f3fff700 1 --2- 172.21.2.112:0/2105948870 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f72e00186d0 0x7f72e001abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:58.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.893 7f72f3fff700 1 -- 172.21.2.112:0/2105948870 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f72f4032b40 con 0x7f72fc0ecc40 2021-02-10T11:43:58.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.894 7f72fb7fe700 1 --2- 172.21.2.112:0/2105948870 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f72e00186d0 0x7f72e001abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:58.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.894 7f72fb7fe700 1 --2- 172.21.2.112:0/2105948870 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f72e00186d0 0x7f72e001abd0 crc :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:58.924 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.924 7f7301cf6700 1 -- 172.21.2.112:0/2105948870 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72fc0f0b10 con 0x7f72fc0ecc40 2021-02-10T11:43:58.927 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:58.926 7f72f3fff700 1 -- 172.21.2.112:0/2105948870 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f72f4006d40 con 0x7f72fc0ecc40 2021-02-10T11:43:59.056 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:59.055 7f7301cf6700 1 -- 172.21.2.112:0/2105948870 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f72fc0f0b10 con 0x7f72fc0ecc40 2021-02-10T11:43:59.056 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:59.056 7f72f3fff700 1 -- 172.21.2.112:0/2105948870 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) v1 ==== 95+0+2108 (secure 0 0 0) 0x7f72f401f880 con 0x7f72fc0ecc40 2021-02-10T11:43:59.057 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 122 2021-02-10T11:43:59.057 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:43:59.057 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":122,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11670,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/1286703435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":1286703435},{"type":"v1","addr":"172.21.2.112:6833","nonce":1286703435}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":119}],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:43:56.409867","modified":"2021-02-10 11:43:58.413644","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8578},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8578":{"gid":8578,"name":"b","rank":0,"incarnation":120,"state":"up:active","state_seq":3,"addr":"172.21.2.120:6835/1504781062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":1504781062},{"type":"v1","addr":"172.21.2.120:6835","nonce":1504781062}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":15}]} 2021-02-10T11:43:59.059 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:59.058 7f7301cf6700 1 -- 172.21.2.112:0/2105948870 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f72e00186d0 msgr2=0x7f72e001abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:59.059 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:59.058 7f7301cf6700 1 --2- 172.21.2.112:0/2105948870 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f72e00186d0 0x7f72e001abd0 crc :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:59.059 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:59.058 7f7301cf6700 1 -- 172.21.2.112:0/2105948870 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f72fc0ecc40 msgr2=0x7f72fc097790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:59.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:59.058 7f7301cf6700 1 --2- 172.21.2.112:0/2105948870 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f72fc0ecc40 0x7f72fc097790 secure :-1 s=READY pgs=2545 cs=0 l=1 rev1=1 rx=0x7f72f40099e0 tx=0x7f72f400bd20).stop 2021-02-10T11:43:59.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:59.058 7f7301cf6700 1 -- 172.21.2.112:0/2105948870 shutdown_connections 2021-02-10T11:43:59.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:59.058 7f7301cf6700 1 --2- 172.21.2.112:0/2105948870 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f72e00186d0 0x7f72e001abd0 unknown :-1 s=CLOSED pgs=1168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:59.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:59.058 7f7301cf6700 1 --2- 172.21.2.112:0/2105948870 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f72fc1a4aa0 0x7f72fc0ec620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:59.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:59.058 7f7301cf6700 1 --2- 172.21.2.112:0/2105948870 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f72fc0ecc40 0x7f72fc097790 unknown :-1 s=CLOSED pgs=2545 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:59.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:59.058 7f7301cf6700 1 --2- 172.21.2.112:0/2105948870 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f72fc0b2860 0x7f72fc1a4480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:59.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:59.058 7f7301cf6700 1 -- 172.21.2.112:0/2105948870 >> 172.21.2.112:0/2105948870 conn(0x7f72fc0bf500 msgr2=0x7f72fc0b5050 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:59.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:59.058 7f7301cf6700 1 -- 172.21.2.112:0/2105948870 shutdown_connections 2021-02-10T11:43:59.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:59.058 7f7301cf6700 1 -- 172.21.2.112:0/2105948870 wait complete. 2021-02-10T11:43:59.067 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:43:59.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:59.479 7f8a08994700 1 Processor -- start 2021-02-10T11:43:59.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:59.479 7f8a08994700 1 -- start start 2021-02-10T11:43:59.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:59.480 7f8a08994700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f8a040d1870 0x7f8a041b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:59.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:59.480 7f8a08994700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f8a040f0650 0x7f8a040cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:59.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:59.480 7f8a08994700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a040f0b10 con 0x7f8a041b0a10 2021-02-10T11:43:59.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:59.480 7f8a08994700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a040e5ed0 con 0x7f8a040f0650 2021-02-10T11:43:59.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:59.480 7f8a08994700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a040d1c70 con 0x7f8a040d1870 2021-02-10T11:43:59.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:59.480 7f8a02e1d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f8a040f0650 0x7f8a040cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:59.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:59.480 7f8a02e1d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f8a040f0650 0x7f8a040cfb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:57534/0 (socket says 172.21.2.112:57534) 2021-02-10T11:43:59.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:59.480 7f8a02e1d700 1 -- 172.21.2.112:0/834349304 learned_addr learned my addr 172.21.2.112:0/834349304 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:43:59.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:59.480 7f8a02e1d700 1 -- 172.21.2.112:0/834349304 >> v2:172.21.2.120:3300/0 conn(0x7f8a040d1870 msgr2=0x7f8a041b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:59.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:59.480 7f8a02e1d700 1 --2- 172.21.2.112:0/834349304 >> v2:172.21.2.120:3300/0 conn(0x7f8a040d1870 0x7f8a041b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:59.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:59.480 7f8a02e1d700 1 -- 172.21.2.112:0/834349304 >> v1:172.21.2.112:6789/0 conn(0x7f8a041b0a10 legacy=0x7f8a040cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:59.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:59.480 7f8a02e1d700 1 -- 172.21.2.112:0/834349304 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a040d1e90 con 0x7f8a040f0650 2021-02-10T11:43:59.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:59.480 7f8a02e1d700 1 --2- 172.21.2.112:0/834349304 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8a040f0650 0x7f8a040cfb20 secure :-1 s=READY pgs=2547 cs=0 l=1 rev1=1 rx=0x7f89fc006f00 tx=0x7f89fc009f10).ready entity=mon.0 client_cookie=a93b3ba33aa660e5 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:59.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:59.480 7f8a0161a700 1 -- 172.21.2.112:0/834349304 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f89fc003030 con 0x7f8a040f0650 2021-02-10T11:43:59.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:59.480 7f8a0161a700 1 -- 172.21.2.112:0/834349304 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89fc00b3b0 con 0x7f8a040f0650 2021-02-10T11:43:59.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:59.480 7f8a0161a700 1 -- 172.21.2.112:0/834349304 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f89fc000f90 con 0x7f8a040f0650 2021-02-10T11:43:59.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:59.480 7f8a08994700 1 -- 172.21.2.112:0/834349304 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8a040f0650 msgr2=0x7f8a040cfb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:59.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:59.480 7f8a08994700 1 --2- 172.21.2.112:0/834349304 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8a040f0650 0x7f8a040cfb20 secure :-1 s=READY pgs=2547 cs=0 l=1 rev1=1 rx=0x7f89fc006f00 tx=0x7f89fc009f10).stop 2021-02-10T11:43:59.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:59.481 7f8a08994700 1 -- 172.21.2.112:0/834349304 shutdown_connections 2021-02-10T11:43:59.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:59.481 7f8a08994700 1 --2- 172.21.2.112:0/834349304 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8a040f0650 0x7f8a040cfb20 unknown :-1 s=CLOSED pgs=2547 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:59.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:59.481 7f8a08994700 1 --2- 172.21.2.112:0/834349304 >> v2:172.21.2.120:3300/0 conn(0x7f8a040d1870 0x7f8a041b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:59.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:59.481 7f8a08994700 1 -- 172.21.2.112:0/834349304 >> 172.21.2.112:0/834349304 conn(0x7f8a040ce450 msgr2=0x7f8a040d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:59.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:59.481 7f8a08994700 1 -- 172.21.2.112:0/834349304 shutdown_connections 2021-02-10T11:43:59.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:59.481 7f8a08994700 1 -- 172.21.2.112:0/834349304 wait complete. 2021-02-10T11:43:59.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:59.496 7f8a08994700 1 Processor -- start 2021-02-10T11:43:59.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:59.496 7f8a08994700 1 -- start start 2021-02-10T11:43:59.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:59.496 7f8a08994700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8a040b0ad0 0x7f8a04091f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:59.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:59.496 7f8a08994700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8a041a8850 0x7f8a041a8c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:59.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:59.496 7f8a08994700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8a041a6fa0 0x7f8a041a73a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:59.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:59.496 7f8a08994700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a041abfe0 con 0x7f8a040b0ad0 2021-02-10T11:43:59.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:59.496 7f8a08994700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a041a9c40 con 0x7f8a041a6fa0 2021-02-10T11:43:59.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:59.496 7f8a08994700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a041a9730 con 0x7f8a041a8850 2021-02-10T11:43:59.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:59.496 7f8a0261c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8a040b0ad0 0x7f8a04091f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:59.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:59.496 7f8a0261c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8a040b0ad0 0x7f8a04091f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:57538/0 (socket says 172.21.2.112:57538) 2021-02-10T11:43:59.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:59.496 7f8a0261c700 1 -- 172.21.2.112:0/4067538291 learned_addr learned my addr 172.21.2.112:0/4067538291 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:43:59.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:59.496 7f8a02e1d700 1 --2- 172.21.2.112:0/4067538291 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8a041a6fa0 0x7f8a041a73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:59.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:59.497 7f8a01e1b700 1 --2- 172.21.2.112:0/4067538291 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8a041a8850 0x7f8a041a8c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:59.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:59.497 7f8a0261c700 1 -- 172.21.2.112:0/4067538291 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8a041a8850 msgr2=0x7f8a041a8c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:59.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:59.497 7f8a0261c700 1 --2- 172.21.2.112:0/4067538291 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8a041a8850 0x7f8a041a8c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:59.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:59.497 7f8a0261c700 1 -- 172.21.2.112:0/4067538291 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8a041a6fa0 msgr2=0x7f8a041a73a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:59.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:59.497 7f8a0261c700 1 --2- 172.21.2.112:0/4067538291 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8a041a6fa0 0x7f8a041a73a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:59.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:59.497 7f8a0261c700 1 -- 172.21.2.112:0/4067538291 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a041a9270 con 0x7f8a040b0ad0 2021-02-10T11:43:59.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:59.497 7f8a0261c700 1 --2- 172.21.2.112:0/4067538291 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8a040b0ad0 0x7f8a04091f30 secure :-1 s=READY pgs=2548 cs=0 l=1 rev1=1 rx=0x7f89f4006f00 tx=0x7f89f400b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:59.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:59.497 7f89faffd700 1 -- 172.21.2.112:0/4067538291 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f89f400bd30 con 0x7f8a040b0ad0 2021-02-10T11:43:59.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:59.497 7f89faffd700 1 -- 172.21.2.112:0/4067538291 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89f4000c90 con 0x7f8a040b0ad0 2021-02-10T11:43:59.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:59.497 7f89faffd700 1 -- 172.21.2.112:0/4067538291 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f89f40022a0 con 0x7f8a040b0ad0 2021-02-10T11:43:59.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:59.497 7f8a08994700 1 -- 172.21.2.112:0/4067538291 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a041a9270 con 0x7f8a040b0ad0 2021-02-10T11:43:59.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:59.497 7f8a08994700 1 -- 172.21.2.112:0/4067538291 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a041a7cb0 con 0x7f8a040b0ad0 2021-02-10T11:43:59.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:59.498 7f89faffd700 1 -- 172.21.2.112:0/4067538291 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f89f4005310 con 0x7f8a040b0ad0 2021-02-10T11:43:59.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:59.498 7f89faffd700 1 --2- 172.21.2.112:0/4067538291 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f89e80186c0 0x7f89e801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:59.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:59.498 7f89faffd700 1 -- 172.21.2.112:0/4067538291 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f89f4032560 con 0x7f8a040b0ad0 2021-02-10T11:43:59.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:59.498 7f8a01e1b700 1 --2- 172.21.2.112:0/4067538291 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f89e80186c0 0x7f89e801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:59.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:59.499 7f8a01e1b700 1 --2- 172.21.2.112:0/4067538291 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f89e80186c0 0x7f89e801abc0 crc :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:59.530 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:59.529 7f8a08994700 1 -- 172.21.2.112:0/4067538291 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a040ce450 con 0x7f8a040b0ad0 2021-02-10T11:43:59.532 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:59.532 7f89faffd700 1 -- 172.21.2.112:0/4067538291 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f89f4000960 con 0x7f8a040b0ad0 2021-02-10T11:43:59.657 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:59.657 7f8a08994700 1 -- 172.21.2.112:0/4067538291 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8a040ce450 con 0x7f8a040b0ad0 2021-02-10T11:43:59.658 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:59.657 7f89faffd700 1 -- 172.21.2.112:0/4067538291 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v143) v1 ==== 74+0+14007 (secure 0 0 0) 0x7f89f4005310 con 0x7f8a040b0ad0 2021-02-10T11:43:59.658 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:43:59.659 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":143,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:43:57.411337","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":31,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":30,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:43:54.385734","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"142","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":31,"pool_name":"cephfs_data","create_time":"2021-02-10 11:43:55.453740","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"143","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":139,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":139,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:43:59.660 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:59.659 7f8a08994700 1 -- 172.21.2.112:0/4067538291 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f89e80186c0 msgr2=0x7f89e801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:59.660 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:59.659 7f8a08994700 1 --2- 172.21.2.112:0/4067538291 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f89e80186c0 0x7f89e801abc0 crc :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:59.660 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:59.659 7f8a08994700 1 -- 172.21.2.112:0/4067538291 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8a040b0ad0 msgr2=0x7f8a04091f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:59.660 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:59.659 7f8a08994700 1 --2- 172.21.2.112:0/4067538291 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8a040b0ad0 0x7f8a04091f30 secure :-1 s=READY pgs=2548 cs=0 l=1 rev1=1 rx=0x7f89f4006f00 tx=0x7f89f400b940).stop 2021-02-10T11:43:59.661 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:59.659 7f8a08994700 1 -- 172.21.2.112:0/4067538291 shutdown_connections 2021-02-10T11:43:59.661 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:59.659 7f8a08994700 1 --2- 172.21.2.112:0/4067538291 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f89e80186c0 0x7f89e801abc0 unknown :-1 s=CLOSED pgs=1169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:59.661 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:59.659 7f8a08994700 1 --2- 172.21.2.112:0/4067538291 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8a041a6fa0 0x7f8a041a73a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:59.661 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:59.659 7f8a08994700 1 --2- 172.21.2.112:0/4067538291 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8a041a8850 0x7f8a041a8c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:59.661 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:59.659 7f8a08994700 1 --2- 172.21.2.112:0/4067538291 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8a040b0ad0 0x7f8a04091f30 unknown :-1 s=CLOSED pgs=2548 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:59.661 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:59.659 7f8a08994700 1 -- 172.21.2.112:0/4067538291 >> 172.21.2.112:0/4067538291 conn(0x7f8a0419f8f0 msgr2=0x7f8a041a5f80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:59.661 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:59.660 7f8a08994700 1 -- 172.21.2.112:0/4067538291 shutdown_connections 2021-02-10T11:43:59.661 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:43:59.660 7f8a08994700 1 -- 172.21.2.112:0/4067538291 wait complete. 2021-02-10T11:43:59.669 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:44:00.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.083 7f7d622a2700 1 Processor -- start 2021-02-10T11:44:00.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.083 7f7d622a2700 1 -- start start 2021-02-10T11:44:00.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.083 7f7d622a2700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f7d5c10afe0 0x7f7d5c0cf740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:00.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.083 7f7d622a2700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d5c0fe1b0 con 0x7f7d5c0d1860 2021-02-10T11:44:00.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.083 7f7d622a2700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d5c1b0ca0 con 0x7f7d5c0d1c60 2021-02-10T11:44:00.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.083 7f7d622a2700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d5c109a50 con 0x7f7d5c10afe0 2021-02-10T11:44:00.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.083 7f7d5bfff700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f7d5c0d1860 0x7f7d5c1b04b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:57008/0 (socket says 172.21.2.112:57008) 2021-02-10T11:44:00.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.083 7f7d5bfff700 1 -- 172.21.2.112:0/1108679804 learned_addr learned my addr 172.21.2.112:0/1108679804 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:44:00.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.084 7f7d608bf700 1 --2- 172.21.2.112:0/1108679804 >> v2:172.21.2.120:3300/0 conn(0x7f7d5c10afe0 0x7f7d5c0cf740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:00.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.084 7f7d5affd700 1 -- 172.21.2.112:0/1108679804 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1166619048 0 0) 0x7f7d4c001180 con 0x7f7d5c0d1860 2021-02-10T11:44:00.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.084 7f7d5affd700 1 -- 172.21.2.112:0/1108679804 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d44002fe0 con 0x7f7d5c0d1860 2021-02-10T11:44:00.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.084 7f7d608bf700 1 -- 172.21.2.112:0/1108679804 >> v1:172.21.2.120:6789/0 conn(0x7f7d5c0d1c60 legacy=0x7f7d5c0cef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:00.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.084 7f7d608bf700 1 -- 172.21.2.112:0/1108679804 >> v1:172.21.2.112:6789/0 conn(0x7f7d5c0d1860 legacy=0x7f7d5c1b04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:00.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.084 7f7d608bf700 1 -- 172.21.2.112:0/1108679804 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d5c10b3e0 con 0x7f7d5c10afe0 2021-02-10T11:44:00.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.084 7f7d608bf700 1 --2- 172.21.2.112:0/1108679804 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7d5c10afe0 0x7f7d5c0cf740 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 rx=0x7f7d54006e50 tx=0x7f7d54009e70).ready entity=mon.1 client_cookie=aaa1d0ac209111ef server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:00.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.085 7f7d5affd700 1 -- 172.21.2.112:0/1108679804 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7d54006a20 con 0x7f7d5c10afe0 2021-02-10T11:44:00.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.085 7f7d5affd700 1 -- 172.21.2.112:0/1108679804 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d54006790 con 0x7f7d5c10afe0 2021-02-10T11:44:00.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.085 7f7d5affd700 1 -- 172.21.2.112:0/1108679804 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7d5400b700 con 0x7f7d5c10afe0 2021-02-10T11:44:00.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.085 7f7d622a2700 1 -- 172.21.2.112:0/1108679804 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7d5c10afe0 msgr2=0x7f7d5c0cf740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:00.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.085 7f7d622a2700 1 --2- 172.21.2.112:0/1108679804 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7d5c10afe0 0x7f7d5c0cf740 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 rx=0x7f7d54006e50 tx=0x7f7d54009e70).stop 2021-02-10T11:44:00.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.085 7f7d622a2700 1 -- 172.21.2.112:0/1108679804 shutdown_connections 2021-02-10T11:44:00.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.085 7f7d622a2700 1 --2- 172.21.2.112:0/1108679804 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7d5c10afe0 0x7f7d5c0cf740 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:00.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.085 7f7d622a2700 1 -- 172.21.2.112:0/1108679804 >> 172.21.2.112:0/1108679804 conn(0x7f7d5c0ce410 msgr2=0x7f7d5c0d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:00.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.085 7f7d622a2700 1 -- 172.21.2.112:0/1108679804 shutdown_connections 2021-02-10T11:44:00.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.085 7f7d622a2700 1 -- 172.21.2.112:0/1108679804 wait complete. 2021-02-10T11:44:00.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.100 7f7d622a2700 1 Processor -- start 2021-02-10T11:44:00.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.100 7f7d622a2700 1 -- start start 2021-02-10T11:44:00.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.100 7f7d622a2700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7d5c099860 0x7f7d5c194f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:00.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.100 7f7d622a2700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7d5c099c60 0x7f7d5c195540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:00.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.100 7f7d622a2700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7d5c18a540 0x7f7d5c18a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:00.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.100 7f7d622a2700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d5c189fc0 con 0x7f7d5c18a540 2021-02-10T11:44:00.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.100 7f7d622a2700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d5c090820 con 0x7f7d5c099860 2021-02-10T11:44:00.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.100 7f7d622a2700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d5c090a00 con 0x7f7d5c099c60 2021-02-10T11:44:00.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.101 7f7d608bf700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7d5c18a540 0x7f7d5c18a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:00.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.101 7f7d608bf700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7d5c18a540 0x7f7d5c18a940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:57556/0 (socket says 172.21.2.112:57556) 2021-02-10T11:44:00.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.101 7f7d608bf700 1 -- 172.21.2.112:0/2005903877 learned_addr learned my addr 172.21.2.112:0/2005903877 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:44:00.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.101 7f7d608bf700 1 -- 172.21.2.112:0/2005903877 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7d5c099c60 msgr2=0x7f7d5c195540 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:44:00.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.101 7f7d608bf700 1 --2- 172.21.2.112:0/2005903877 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7d5c099c60 0x7f7d5c195540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:00.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.101 7f7d608bf700 1 -- 172.21.2.112:0/2005903877 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7d5c099860 msgr2=0x7f7d5c194f20 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:44:00.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.101 7f7d608bf700 1 --2- 172.21.2.112:0/2005903877 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7d5c099860 0x7f7d5c194f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:00.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.101 7f7d608bf700 1 -- 172.21.2.112:0/2005903877 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d5c18af60 con 0x7f7d5c18a540 2021-02-10T11:44:00.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.101 7f7d608bf700 1 --2- 172.21.2.112:0/2005903877 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7d5c18a540 0x7f7d5c18a940 secure :-1 s=READY pgs=2550 cs=0 l=1 rev1=1 rx=0x7f7d54006e60 tx=0x7f7d5400bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:00.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.101 7f7d58ff9700 1 -- 172.21.2.112:0/2005903877 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7d54009130 con 0x7f7d5c18a540 2021-02-10T11:44:00.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.101 7f7d58ff9700 1 -- 172.21.2.112:0/2005903877 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d540093f0 con 0x7f7d5c18a540 2021-02-10T11:44:00.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.101 7f7d58ff9700 1 -- 172.21.2.112:0/2005903877 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7d540013b0 con 0x7f7d5c18a540 2021-02-10T11:44:00.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.101 7f7d622a2700 1 -- 172.21.2.112:0/2005903877 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d5c18af60 con 0x7f7d5c18a540 2021-02-10T11:44:00.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.101 7f7d622a2700 1 -- 172.21.2.112:0/2005903877 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d5c18af60 con 0x7f7d5c18a540 2021-02-10T11:44:00.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.102 7f7d58ff9700 1 -- 172.21.2.112:0/2005903877 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7d540056b0 con 0x7f7d5c18a540 2021-02-10T11:44:00.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.102 7f7d58ff9700 1 --2- 172.21.2.112:0/2005903877 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7d3c0186d0 0x7f7d3c01abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:00.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.102 7f7d58ff9700 1 -- 172.21.2.112:0/2005903877 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f7d540329c0 con 0x7f7d5c18a540 2021-02-10T11:44:00.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.102 7f7d5bfff700 1 --2- 172.21.2.112:0/2005903877 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7d3c0186d0 0x7f7d3c01abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:00.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.103 7f7d5bfff700 1 --2- 172.21.2.112:0/2005903877 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7d3c0186d0 0x7f7d3c01abd0 crc :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:00.133 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.133 7f7d622a2700 1 -- 172.21.2.112:0/2005903877 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d5c082980 con 0x7f7d5c18a540 2021-02-10T11:44:00.136 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.136 7f7d58ff9700 1 -- 172.21.2.112:0/2005903877 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7d5400be30 con 0x7f7d5c18a540 2021-02-10T11:44:00.266 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.265 7f7d622a2700 1 -- 172.21.2.112:0/2005903877 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f7d5c082980 con 0x7f7d5c18a540 2021-02-10T11:44:00.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.269 7f7d58ff9700 1 -- 172.21.2.112:0/2005903877 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v83) v1 ==== 184+0+0 (secure 0 0 0) 0x7f7d540093d0 con 0x7f7d5c18a540 2021-02-10T11:44:00.273 INFO:teuthology.orchestra.run.gibba012.stderr:updated caps for client.0 2021-02-10T11:44:00.275 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.274 7f7d622a2700 1 -- 172.21.2.112:0/2005903877 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7d3c0186d0 msgr2=0x7f7d3c01abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:00.275 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.274 7f7d622a2700 1 --2- 172.21.2.112:0/2005903877 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7d3c0186d0 0x7f7d3c01abd0 crc :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:00.275 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.274 7f7d622a2700 1 -- 172.21.2.112:0/2005903877 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7d5c18a540 msgr2=0x7f7d5c18a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:00.275 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.274 7f7d622a2700 1 --2- 172.21.2.112:0/2005903877 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7d5c18a540 0x7f7d5c18a940 secure :-1 s=READY pgs=2550 cs=0 l=1 rev1=1 rx=0x7f7d54006e60 tx=0x7f7d5400bcf0).stop 2021-02-10T11:44:00.275 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.275 7f7d622a2700 1 -- 172.21.2.112:0/2005903877 shutdown_connections 2021-02-10T11:44:00.275 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.275 7f7d622a2700 1 --2- 172.21.2.112:0/2005903877 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7d3c0186d0 0x7f7d3c01abd0 unknown :-1 s=CLOSED pgs=1170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:00.276 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.275 7f7d622a2700 1 --2- 172.21.2.112:0/2005903877 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7d5c18a540 0x7f7d5c18a940 unknown :-1 s=CLOSED pgs=2550 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:00.276 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.275 7f7d622a2700 1 --2- 172.21.2.112:0/2005903877 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7d5c099c60 0x7f7d5c195540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:00.276 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.275 7f7d622a2700 1 --2- 172.21.2.112:0/2005903877 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7d5c099860 0x7f7d5c194f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:00.276 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.275 7f7d622a2700 1 -- 172.21.2.112:0/2005903877 >> 172.21.2.112:0/2005903877 conn(0x7f7d5c19f820 msgr2=0x7f7d5c1a5cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:00.276 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.275 7f7d622a2700 1 -- 172.21.2.112:0/2005903877 shutdown_connections 2021-02-10T11:44:00.276 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.275 7f7d622a2700 1 -- 172.21.2.112:0/2005903877 wait complete. 2021-02-10T11:44:00.284 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:44:00.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.706 7f7bc67f5700 1 Processor -- start 2021-02-10T11:44:00.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.706 7f7bc67f5700 1 -- start start 2021-02-10T11:44:00.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.707 7f7bc67f5700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f7bc00d1880 0x7f7bc01b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:00.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.707 7f7bc67f5700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f7bc01b0af0 0x7f7bc0108950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:00.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.707 7f7bc67f5700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bc00fe1d0 con 0x7f7bc00d1c80 2021-02-10T11:44:00.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.707 7f7bc67f5700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7bc0109a70 con 0x7f7bc00d1880 2021-02-10T11:44:00.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.707 7f7bc67f5700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7bc0109760 con 0x7f7bc01b0af0 2021-02-10T11:44:00.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.707 7f7bbffff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f7bc00d1880 0x7f7bc01b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:00.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.707 7f7bbffff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f7bc00d1880 0x7f7bc01b04d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:57560/0 (socket says 172.21.2.112:57560) 2021-02-10T11:44:00.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.707 7f7bc4e12700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f7bc00d1c80 0x7f7bc0108f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:57022/0 (socket says 172.21.2.112:57022) 2021-02-10T11:44:00.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.707 7f7bbffff700 1 -- 172.21.2.112:0/1096015408 learned_addr learned my addr 172.21.2.112:0/1096015408 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:44:00.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.707 7f7bbeffd700 1 -- 172.21.2.112:0/1096015408 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2255339027 0 0) 0x7f7bb8001480 con 0x7f7bc00d1c80 2021-02-10T11:44:00.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.707 7f7bbffff700 1 -- 172.21.2.112:0/1096015408 >> v2:172.21.2.120:3300/0 conn(0x7f7bc01b0af0 msgr2=0x7f7bc0108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:00.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.707 7f7bbffff700 1 --2- 172.21.2.112:0/1096015408 >> v2:172.21.2.120:3300/0 conn(0x7f7bc01b0af0 0x7f7bc0108950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:00.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.707 7f7bbffff700 1 -- 172.21.2.112:0/1096015408 >> v1:172.21.2.112:6789/0 conn(0x7f7bc00d1c80 legacy=0x7f7bc0108f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:00.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.707 7f7bbffff700 1 -- 172.21.2.112:0/1096015408 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bc0088070 con 0x7f7bc00d1880 2021-02-10T11:44:00.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.708 7f7bbffff700 1 --2- 172.21.2.112:0/1096015408 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7bc00d1880 0x7f7bc01b04d0 secure :-1 s=READY pgs=2552 cs=0 l=1 rev1=1 rx=0x7f7bb0006700 tx=0x7f7bb0009b80).ready entity=mon.0 client_cookie=3c34e008f5ee1ea8 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:00.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.708 7f7bbeffd700 1 -- 172.21.2.112:0/1096015408 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7bb000c0c0 con 0x7f7bc00d1880 2021-02-10T11:44:00.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.708 7f7bbeffd700 1 -- 172.21.2.112:0/1096015408 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bb000c340 con 0x7f7bc00d1880 2021-02-10T11:44:00.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.708 7f7bbeffd700 1 -- 172.21.2.112:0/1096015408 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7bb000c860 con 0x7f7bc00d1880 2021-02-10T11:44:00.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.708 7f7bc67f5700 1 -- 172.21.2.112:0/1096015408 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7bc00d1880 msgr2=0x7f7bc01b04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:00.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.708 7f7bc67f5700 1 --2- 172.21.2.112:0/1096015408 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7bc00d1880 0x7f7bc01b04d0 secure :-1 s=READY pgs=2552 cs=0 l=1 rev1=1 rx=0x7f7bb0006700 tx=0x7f7bb0009b80).stop 2021-02-10T11:44:00.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.709 7f7bc67f5700 1 -- 172.21.2.112:0/1096015408 shutdown_connections 2021-02-10T11:44:00.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.709 7f7bc67f5700 1 --2- 172.21.2.112:0/1096015408 >> v2:172.21.2.120:3300/0 conn(0x7f7bc01b0af0 0x7f7bc0108950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:00.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.709 7f7bc67f5700 1 --2- 172.21.2.112:0/1096015408 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7bc00d1880 0x7f7bc01b04d0 unknown :-1 s=CLOSED pgs=2552 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:00.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.709 7f7bc67f5700 1 -- 172.21.2.112:0/1096015408 >> 172.21.2.112:0/1096015408 conn(0x7f7bc00ce430 msgr2=0x7f7bc00d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:00.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.709 7f7bc67f5700 1 -- 172.21.2.112:0/1096015408 shutdown_connections 2021-02-10T11:44:00.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.709 7f7bc67f5700 1 -- 172.21.2.112:0/1096015408 wait complete. 2021-02-10T11:44:00.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.724 7f7bc67f5700 1 Processor -- start 2021-02-10T11:44:00.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.724 7f7bc67f5700 1 -- start start 2021-02-10T11:44:00.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.724 7f7bc67f5700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7bc01b9470 0x7f7bc01ae6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:00.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.724 7f7bc67f5700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7bc01aecf0 0x7f7bc00957f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:00.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.724 7f7bc67f5700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7bc01af0f0 0x7f7bc0095e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:00.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.724 7f7bc67f5700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bc0096d70 con 0x7f7bc01b9470 2021-02-10T11:44:00.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.724 7f7bc67f5700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bc00a7780 con 0x7f7bc01af0f0 2021-02-10T11:44:00.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.724 7f7bc67f5700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bc01b9870 con 0x7f7bc01aecf0 2021-02-10T11:44:00.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.724 7f7bbffff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7bc01b9470 0x7f7bc01ae6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:00.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.724 7f7bbffff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7bc01b9470 0x7f7bc01ae6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:57566/0 (socket says 172.21.2.112:57566) 2021-02-10T11:44:00.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.724 7f7bbffff700 1 -- 172.21.2.112:0/1534450496 learned_addr learned my addr 172.21.2.112:0/1534450496 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:44:00.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.725 7f7bbffff700 1 -- 172.21.2.112:0/1534450496 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7bc01aecf0 msgr2=0x7f7bc00957f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:00.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.725 7f7bbffff700 1 --2- 172.21.2.112:0/1534450496 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7bc01aecf0 0x7f7bc00957f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:00.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.725 7f7bbffff700 1 -- 172.21.2.112:0/1534450496 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7bc01af0f0 msgr2=0x7f7bc0095e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:00.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.725 7f7bbffff700 1 --2- 172.21.2.112:0/1534450496 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7bc01af0f0 0x7f7bc0095e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:00.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.725 7f7bbffff700 1 -- 172.21.2.112:0/1534450496 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bc019e6b0 con 0x7f7bc01b9470 2021-02-10T11:44:00.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.725 7f7bbffff700 1 --2- 172.21.2.112:0/1534450496 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7bc01b9470 0x7f7bc01ae6d0 secure :-1 s=READY pgs=2553 cs=0 l=1 rev1=1 rx=0x7f7bb000b5e0 tx=0x7f7bb000b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:00.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.725 7f7bbcff9700 1 -- 172.21.2.112:0/1534450496 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7bb000bd70 con 0x7f7bc01b9470 2021-02-10T11:44:00.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.725 7f7bbcff9700 1 -- 172.21.2.112:0/1534450496 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bb0002030 con 0x7f7bc01b9470 2021-02-10T11:44:00.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.725 7f7bc67f5700 1 -- 172.21.2.112:0/1534450496 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bc019e6b0 con 0x7f7bc01b9470 2021-02-10T11:44:00.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.725 7f7bc67f5700 1 -- 172.21.2.112:0/1534450496 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bc0096720 con 0x7f7bc01b9470 2021-02-10T11:44:00.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.725 7f7bbcff9700 1 -- 172.21.2.112:0/1534450496 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7bb0002550 con 0x7f7bc01b9470 2021-02-10T11:44:00.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.726 7f7bbcff9700 1 -- 172.21.2.112:0/1534450496 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7bb0000c90 con 0x7f7bc01b9470 2021-02-10T11:44:00.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.726 7f7bbcff9700 1 --2- 172.21.2.112:0/1534450496 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7ba00186c0 0x7f7ba001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:00.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.726 7f7bbcff9700 1 -- 172.21.2.112:0/1534450496 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f7bb00325d0 con 0x7f7bc01b9470 2021-02-10T11:44:00.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.726 7f7bbf7fe700 1 --2- 172.21.2.112:0/1534450496 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7ba00186c0 0x7f7ba001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:00.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.727 7f7bbf7fe700 1 --2- 172.21.2.112:0/1534450496 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7ba00186c0 0x7f7ba001abc0 crc :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:00.757 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.756 7f7bc67f5700 1 -- 172.21.2.112:0/1534450496 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bc00829a0 con 0x7f7bc01b9470 2021-02-10T11:44:00.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.759 7f7bbcff9700 1 -- 172.21.2.112:0/1534450496 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7bb0030d80 con 0x7f7bc01b9470 2021-02-10T11:44:00.893 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.892 7f7bc67f5700 1 -- 172.21.2.112:0/1534450496 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f7bc00829a0 con 0x7f7bc01b9470 2021-02-10T11:44:00.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.895 7f7bbcff9700 1 -- 172.21.2.112:0/1534450496 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v84) v1 ==== 184+0+0 (secure 0 0 0) 0x7f7bb0000c90 con 0x7f7bc01b9470 2021-02-10T11:44:00.900 INFO:teuthology.orchestra.run.gibba012.stderr:updated caps for client.1 2021-02-10T11:44:00.901 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.901 7f7bc67f5700 1 -- 172.21.2.112:0/1534450496 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7ba00186c0 msgr2=0x7f7ba001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:00.901 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.901 7f7bc67f5700 1 --2- 172.21.2.112:0/1534450496 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7ba00186c0 0x7f7ba001abc0 crc :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:00.902 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.901 7f7bc67f5700 1 -- 172.21.2.112:0/1534450496 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7bc01b9470 msgr2=0x7f7bc01ae6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:00.902 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.901 7f7bc67f5700 1 --2- 172.21.2.112:0/1534450496 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7bc01b9470 0x7f7bc01ae6d0 secure :-1 s=READY pgs=2553 cs=0 l=1 rev1=1 rx=0x7f7bb000b5e0 tx=0x7f7bb000b980).stop 2021-02-10T11:44:00.902 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.901 7f7bc67f5700 1 -- 172.21.2.112:0/1534450496 shutdown_connections 2021-02-10T11:44:00.902 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.901 7f7bc67f5700 1 --2- 172.21.2.112:0/1534450496 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7ba00186c0 0x7f7ba001abc0 unknown :-1 s=CLOSED pgs=1171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:00.902 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.901 7f7bc67f5700 1 --2- 172.21.2.112:0/1534450496 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7bc01af0f0 0x7f7bc0095e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:00.902 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.901 7f7bc67f5700 1 --2- 172.21.2.112:0/1534450496 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7bc01aecf0 0x7f7bc00957f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:00.902 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.901 7f7bc67f5700 1 --2- 172.21.2.112:0/1534450496 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7bc01b9470 0x7f7bc01ae6d0 unknown :-1 s=CLOSED pgs=2553 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:00.902 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.901 7f7bc67f5700 1 -- 172.21.2.112:0/1534450496 >> 172.21.2.112:0/1534450496 conn(0x7f7bc00845b0 msgr2=0x7f7bc018c290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:00.903 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.901 7f7bc67f5700 1 -- 172.21.2.112:0/1534450496 shutdown_connections 2021-02-10T11:44:00.903 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:00.901 7f7bc67f5700 1 -- 172.21.2.112:0/1534450496 wait complete. 2021-02-10T11:44:00.910 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.2 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:44:01.350 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.332 7fa31fdee700 1 Processor -- start 2021-02-10T11:44:01.350 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.333 7fa31fdee700 1 -- start start 2021-02-10T11:44:01.351 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.333 7fa31fdee700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fa3180d1880 0x7fa3181b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:01.351 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.333 7fa31fdee700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3180fe1d0 con 0x7fa3181b0af0 2021-02-10T11:44:01.351 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.333 7fa31fdee700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa318109a70 con 0x7fa3180cff80 2021-02-10T11:44:01.351 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.333 7fa31fdee700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa318088070 con 0x7fa3180d1880 2021-02-10T11:44:01.351 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.333 7fa31dc0a700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fa3180d1880 0x7fa3181b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:01.351 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.333 7fa31d409700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fa3181b0af0 0x7fa318108950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:57034/0 (socket says 172.21.2.112:57034) 2021-02-10T11:44:01.351 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.333 7fa31d409700 1 -- 172.21.2.112:0/4065652887 learned_addr learned my addr 172.21.2.112:0/4065652887 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:44:01.351 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.334 7fa31dc0a700 1 -- 172.21.2.112:0/4065652887 >> v1:172.21.2.120:6789/0 conn(0x7fa3180cff80 legacy=0x7fa318109140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:01.351 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.334 7fa31dc0a700 1 -- 172.21.2.112:0/4065652887 >> v1:172.21.2.112:6789/0 conn(0x7fa3181b0af0 legacy=0x7fa318108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:01.351 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.334 7fa31cc08700 1 -- 172.21.2.112:0/4065652887 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2374587589 0 0) 0x7fa30c0010b0 con 0x7fa3181b0af0 2021-02-10T11:44:01.351 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.334 7fa31dc0a700 1 -- 172.21.2.112:0/4065652887 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa318088260 con 0x7fa3180d1880 2021-02-10T11:44:01.351 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.334 7fa31dc0a700 1 --2- 172.21.2.112:0/4065652887 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa3180d1880 0x7fa3181b04d0 secure :-1 s=READY pgs=2555 cs=0 l=1 rev1=1 rx=0x7fa314006650 tx=0x7fa314009a00).ready entity=mon.0 client_cookie=2073c951217d4900 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:01.352 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.334 7fa31cc08700 1 -- 172.21.2.112:0/4065652887 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa3140009b0 con 0x7fa3180d1880 2021-02-10T11:44:01.352 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.334 7fa31cc08700 1 -- 172.21.2.112:0/4065652887 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa31400c0c0 con 0x7fa3180d1880 2021-02-10T11:44:01.352 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.334 7fa31cc08700 1 -- 172.21.2.112:0/4065652887 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa31400c5a0 con 0x7fa3180d1880 2021-02-10T11:44:01.352 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.334 7fa31fdee700 1 -- 172.21.2.112:0/4065652887 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa3180d1880 msgr2=0x7fa3181b04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:01.352 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.334 7fa31fdee700 1 --2- 172.21.2.112:0/4065652887 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa3180d1880 0x7fa3181b04d0 secure :-1 s=READY pgs=2555 cs=0 l=1 rev1=1 rx=0x7fa314006650 tx=0x7fa314009a00).stop 2021-02-10T11:44:01.352 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.335 7fa31fdee700 1 -- 172.21.2.112:0/4065652887 shutdown_connections 2021-02-10T11:44:01.352 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.335 7fa31fdee700 1 --2- 172.21.2.112:0/4065652887 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa3180d1880 0x7fa3181b04d0 unknown :-1 s=CLOSED pgs=2555 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:01.352 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.335 7fa31fdee700 1 -- 172.21.2.112:0/4065652887 >> 172.21.2.112:0/4065652887 conn(0x7fa3180ce430 msgr2=0x7fa3180d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:01.352 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.336 7fa31fdee700 1 -- 172.21.2.112:0/4065652887 shutdown_connections 2021-02-10T11:44:01.352 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.336 7fa31fdee700 1 -- 172.21.2.112:0/4065652887 wait complete. 2021-02-10T11:44:01.354 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.350 7fa31fdee700 1 Processor -- start 2021-02-10T11:44:01.354 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.350 7fa31fdee700 1 -- start start 2021-02-10T11:44:01.354 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.351 7fa31fdee700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa318086520 0x7fa3181a1cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:01.354 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.351 7fa31fdee700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa318086920 0x7fa318187e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:01.354 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.351 7fa31fdee700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa3181a22d0 0x7fa3181884a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:01.354 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.351 7fa31fdee700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa318099600 con 0x7fa318086520 2021-02-10T11:44:01.355 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.351 7fa31fdee700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3180999f0 con 0x7fa318086920 2021-02-10T11:44:01.355 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.351 7fa31fdee700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa318188ac0 con 0x7fa3181a22d0 2021-02-10T11:44:01.355 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.351 7fa31dc0a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa318086520 0x7fa3181a1cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:01.355 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.351 7fa31dc0a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa318086520 0x7fa3181a1cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:57578/0 (socket says 172.21.2.112:57578) 2021-02-10T11:44:01.355 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.351 7fa31dc0a700 1 -- 172.21.2.112:0/2846628659 learned_addr learned my addr 172.21.2.112:0/2846628659 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:44:01.355 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.351 7fa31d409700 1 --2- 172.21.2.112:0/2846628659 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa318086920 0x7fa318187e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:01.355 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.351 7fa31e40b700 1 --2- 172.21.2.112:0/2846628659 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa3181a22d0 0x7fa3181884a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:01.355 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.351 7fa31dc0a700 1 -- 172.21.2.112:0/2846628659 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa3181a22d0 msgr2=0x7fa3181884a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:01.355 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.351 7fa31dc0a700 1 --2- 172.21.2.112:0/2846628659 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa3181a22d0 0x7fa3181884a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:01.355 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.351 7fa31dc0a700 1 -- 172.21.2.112:0/2846628659 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa318086920 msgr2=0x7fa318187e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:01.355 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.351 7fa31dc0a700 1 --2- 172.21.2.112:0/2846628659 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa318086920 0x7fa318187e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:01.355 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.351 7fa31dc0a700 1 -- 172.21.2.112:0/2846628659 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa318188ce0 con 0x7fa318086520 2021-02-10T11:44:01.356 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.351 7fa31e40b700 1 --2- 172.21.2.112:0/2846628659 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa3181a22d0 0x7fa3181884a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:44:01.356 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.351 7fa31d409700 1 --2- 172.21.2.112:0/2846628659 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa318086920 0x7fa318187e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:44:01.356 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.351 7fa31dc0a700 1 --2- 172.21.2.112:0/2846628659 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa318086520 0x7fa3181a1cb0 secure :-1 s=READY pgs=2556 cs=0 l=1 rev1=1 rx=0x7fa31400b640 tx=0x7fa31400b9e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:01.356 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.352 7fa30a7fc700 1 -- 172.21.2.112:0/2846628659 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa31400bdd0 con 0x7fa318086520 2021-02-10T11:44:01.356 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.352 7fa30a7fc700 1 -- 172.21.2.112:0/2846628659 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa314000c90 con 0x7fa318086520 2021-02-10T11:44:01.356 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.352 7fa31fdee700 1 -- 172.21.2.112:0/2846628659 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa318188ce0 con 0x7fa318086520 2021-02-10T11:44:01.356 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.352 7fa30a7fc700 1 -- 172.21.2.112:0/2846628659 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa3140022b0 con 0x7fa318086520 2021-02-10T11:44:01.356 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.352 7fa31fdee700 1 -- 172.21.2.112:0/2846628659 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa318188ce0 con 0x7fa318086520 2021-02-10T11:44:01.356 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.353 7fa30a7fc700 1 -- 172.21.2.112:0/2846628659 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa314006a10 con 0x7fa318086520 2021-02-10T11:44:01.356 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.353 7fa30a7fc700 1 --2- 172.21.2.112:0/2846628659 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa300018730 0x7fa30001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:01.356 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.353 7fa30a7fc700 1 -- 172.21.2.112:0/2846628659 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fa3140326b0 con 0x7fa318086520 2021-02-10T11:44:01.356 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.353 7fa31d409700 1 --2- 172.21.2.112:0/2846628659 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa300018730 0x7fa30001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:01.356 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.353 7fa31d409700 1 --2- 172.21.2.112:0/2846628659 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa300018730 0x7fa30001ac10 crc :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:01.384 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.383 7fa31fdee700 1 -- 172.21.2.112:0/2846628659 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3180fe1d0 con 0x7fa318086520 2021-02-10T11:44:01.386 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.386 7fa30a7fc700 1 -- 172.21.2.112:0/2846628659 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa314000c90 con 0x7fa318086520 2021-02-10T11:44:01.515 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.514 7fa31fdee700 1 -- 172.21.2.112:0/2846628659 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fa3180fe1d0 con 0x7fa318086520 2021-02-10T11:44:01.518 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.518 7fa30a7fc700 1 -- 172.21.2.112:0/2846628659 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.2 v85) v1 ==== 184+0+0 (secure 0 0 0) 0x7fa314006a10 con 0x7fa318086520 2021-02-10T11:44:01.522 INFO:teuthology.orchestra.run.gibba012.stderr:updated caps for client.2 2021-02-10T11:44:01.524 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.523 7fa31fdee700 1 -- 172.21.2.112:0/2846628659 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa300018730 msgr2=0x7fa30001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:01.524 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.523 7fa31fdee700 1 --2- 172.21.2.112:0/2846628659 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa300018730 0x7fa30001ac10 crc :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:01.524 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.523 7fa31fdee700 1 -- 172.21.2.112:0/2846628659 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa318086520 msgr2=0x7fa3181a1cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:01.524 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.523 7fa31fdee700 1 --2- 172.21.2.112:0/2846628659 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa318086520 0x7fa3181a1cb0 secure :-1 s=READY pgs=2556 cs=0 l=1 rev1=1 rx=0x7fa31400b640 tx=0x7fa31400b9e0).stop 2021-02-10T11:44:01.524 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.523 7fa31fdee700 1 -- 172.21.2.112:0/2846628659 shutdown_connections 2021-02-10T11:44:01.524 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.523 7fa31fdee700 1 --2- 172.21.2.112:0/2846628659 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa300018730 0x7fa30001ac10 unknown :-1 s=CLOSED pgs=1172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:01.524 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.523 7fa31fdee700 1 --2- 172.21.2.112:0/2846628659 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa318086920 0x7fa318187e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:01.524 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.523 7fa31fdee700 1 --2- 172.21.2.112:0/2846628659 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa3181a22d0 0x7fa3181884a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:01.524 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.523 7fa31fdee700 1 --2- 172.21.2.112:0/2846628659 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa318086520 0x7fa3181a1cb0 unknown :-1 s=CLOSED pgs=2556 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:01.524 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.523 7fa31fdee700 1 -- 172.21.2.112:0/2846628659 >> 172.21.2.112:0/2846628659 conn(0x7fa31819f880 msgr2=0x7fa31808f890 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:01.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.524 7fa31fdee700 1 -- 172.21.2.112:0/2846628659 shutdown_connections 2021-02-10T11:44:01.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.524 7fa31fdee700 1 -- 172.21.2.112:0/2846628659 wait complete. 2021-02-10T11:44:01.532 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.3 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:44:01.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.946 7ff8ef322700 1 Processor -- start 2021-02-10T11:44:01.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.947 7ff8ef322700 1 -- start start 2021-02-10T11:44:01.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.947 7ff8ef322700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7ff8e80d1c80 0x7ff8e80cef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:01.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.947 7ff8ef322700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff8e80cf590 0x7ff8e80cf990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:01.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.947 7ff8ef322700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8e80fe1d0 con 0x7ff8e80d1880 2021-02-10T11:44:01.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.947 7ff8ef322700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8e81b0cc0 con 0x7ff8e80d1c80 2021-02-10T11:44:01.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.947 7ff8ef322700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8e8109a70 con 0x7ff8e80cf590 2021-02-10T11:44:01.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.947 7ff8ec93d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7ff8e80d1c80 0x7ff8e80cef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:01.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.947 7ff8ec93d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7ff8e80d1c80 0x7ff8e80cef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:57582/0 (socket says 172.21.2.112:57582) 2021-02-10T11:44:01.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.947 7ff8ec93d700 1 -- 172.21.2.112:0/2398071150 learned_addr learned my addr 172.21.2.112:0/2398071150 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:44:01.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.947 7ff8ed93f700 1 --2- 172.21.2.112:0/2398071150 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff8e80cf590 0x7ff8e80cf990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:01.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.948 7ff8ec93d700 1 -- 172.21.2.112:0/2398071150 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff8e80cf590 msgr2=0x7ff8e80cf990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:01.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.948 7ff8ec93d700 1 --2- 172.21.2.112:0/2398071150 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff8e80cf590 0x7ff8e80cf990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:01.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.948 7ff8ec93d700 1 -- 172.21.2.112:0/2398071150 >> v1:172.21.2.120:6789/0 conn(0x7ff8e80d1880 legacy=0x7ff8e81b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:01.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.948 7ff8ec93d700 1 -- 172.21.2.112:0/2398071150 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8e8088070 con 0x7ff8e80d1c80 2021-02-10T11:44:01.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.948 7ff8ec93d700 1 --2- 172.21.2.112:0/2398071150 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff8e80d1c80 0x7ff8e80cef70 secure :-1 s=READY pgs=2557 cs=0 l=1 rev1=1 rx=0x7ff8d8006700 tx=0x7ff8d8009b80).ready entity=mon.0 client_cookie=91e47464939b73be server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:01.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.948 7ff8dffff700 1 -- 172.21.2.112:0/2398071150 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff8d800c0c0 con 0x7ff8e80d1c80 2021-02-10T11:44:01.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.948 7ff8dffff700 1 -- 172.21.2.112:0/2398071150 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8d800c440 con 0x7ff8e80d1c80 2021-02-10T11:44:01.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.948 7ff8dffff700 1 -- 172.21.2.112:0/2398071150 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff8d800c960 con 0x7ff8e80d1c80 2021-02-10T11:44:01.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.948 7ff8ef322700 1 -- 172.21.2.112:0/2398071150 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff8e80d1c80 msgr2=0x7ff8e80cef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:01.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.948 7ff8ef322700 1 --2- 172.21.2.112:0/2398071150 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff8e80d1c80 0x7ff8e80cef70 secure :-1 s=READY pgs=2557 cs=0 l=1 rev1=1 rx=0x7ff8d8006700 tx=0x7ff8d8009b80).stop 2021-02-10T11:44:01.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.949 7ff8ef322700 1 -- 172.21.2.112:0/2398071150 shutdown_connections 2021-02-10T11:44:01.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.949 7ff8ef322700 1 --2- 172.21.2.112:0/2398071150 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff8e80d1c80 0x7ff8e80cef70 unknown :-1 s=CLOSED pgs=2557 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:01.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.949 7ff8ef322700 1 --2- 172.21.2.112:0/2398071150 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff8e80cf590 0x7ff8e80cf990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:01.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.949 7ff8ef322700 1 -- 172.21.2.112:0/2398071150 >> 172.21.2.112:0/2398071150 conn(0x7ff8e80ce430 msgr2=0x7ff8e80d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:01.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.949 7ff8ef322700 1 -- 172.21.2.112:0/2398071150 shutdown_connections 2021-02-10T11:44:01.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.949 7ff8ef322700 1 -- 172.21.2.112:0/2398071150 wait complete. 2021-02-10T11:44:01.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.964 7ff8ef322700 1 Processor -- start 2021-02-10T11:44:01.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.964 7ff8ef322700 1 -- start start 2021-02-10T11:44:01.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.964 7ff8ef322700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff8e808d090 0x7ff8e808c490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:01.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.964 7ff8ef322700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff8e808d490 0x7ff8e80beec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:01.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.964 7ff8ef322700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff8e80bf4e0 0x7ff8e809fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:01.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.964 7ff8ef322700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8e80bf8e0 con 0x7ff8e808d490 2021-02-10T11:44:01.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.964 7ff8ef322700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8e80a0c90 con 0x7ff8e808d090 2021-02-10T11:44:01.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.964 7ff8ef322700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8e80a0eb0 con 0x7ff8e80bf4e0 2021-02-10T11:44:01.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.964 7ff8ec93d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff8e808d490 0x7ff8e80beec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:01.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.964 7ff8ec93d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff8e808d490 0x7ff8e80beec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:57592/0 (socket says 172.21.2.112:57592) 2021-02-10T11:44:01.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.964 7ff8ec93d700 1 -- 172.21.2.112:0/1402578707 learned_addr learned my addr 172.21.2.112:0/1402578707 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:44:01.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.964 7ff8ed13e700 1 --2- 172.21.2.112:0/1402578707 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff8e808d090 0x7ff8e808c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:01.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.964 7ff8ed93f700 1 --2- 172.21.2.112:0/1402578707 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff8e80bf4e0 0x7ff8e809fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:01.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.965 7ff8ec93d700 1 -- 172.21.2.112:0/1402578707 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff8e80bf4e0 msgr2=0x7ff8e809fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:01.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.965 7ff8ec93d700 1 --2- 172.21.2.112:0/1402578707 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff8e80bf4e0 0x7ff8e809fc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:01.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.965 7ff8ec93d700 1 -- 172.21.2.112:0/1402578707 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff8e808d090 msgr2=0x7ff8e808c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:01.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.965 7ff8ec93d700 1 --2- 172.21.2.112:0/1402578707 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff8e808d090 0x7ff8e808c490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:01.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.965 7ff8ec93d700 1 -- 172.21.2.112:0/1402578707 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8e80a0260 con 0x7ff8e808d490 2021-02-10T11:44:01.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.965 7ff8ec93d700 1 --2- 172.21.2.112:0/1402578707 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff8e808d490 0x7ff8e80beec0 secure :-1 s=READY pgs=2558 cs=0 l=1 rev1=1 rx=0x7ff8d800b5e0 tx=0x7ff8d800b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:01.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.965 7ff8ddffb700 1 -- 172.21.2.112:0/1402578707 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff8d800bd70 con 0x7ff8e808d490 2021-02-10T11:44:01.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.965 7ff8ddffb700 1 -- 172.21.2.112:0/1402578707 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8d8002030 con 0x7ff8e808d490 2021-02-10T11:44:01.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.965 7ff8ddffb700 1 -- 172.21.2.112:0/1402578707 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff8d8002550 con 0x7ff8e808d490 2021-02-10T11:44:01.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.965 7ff8ef322700 1 -- 172.21.2.112:0/1402578707 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8e80a0260 con 0x7ff8e808d490 2021-02-10T11:44:01.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.965 7ff8ef322700 1 -- 172.21.2.112:0/1402578707 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8e80a0260 con 0x7ff8e808d490 2021-02-10T11:44:01.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.966 7ff8ddffb700 1 -- 172.21.2.112:0/1402578707 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff8d8002210 con 0x7ff8e808d490 2021-02-10T11:44:01.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.966 7ff8ddffb700 1 --2- 172.21.2.112:0/1402578707 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff8c80186c0 0x7ff8c801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:01.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.966 7ff8ddffb700 1 -- 172.21.2.112:0/1402578707 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5573+0+0 (secure 0 0 0) 0x7ff8d8032650 con 0x7ff8e808d490 2021-02-10T11:44:01.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.966 7ff8ed13e700 1 --2- 172.21.2.112:0/1402578707 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff8c80186c0 0x7ff8c801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:01.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.966 7ff8ed13e700 1 --2- 172.21.2.112:0/1402578707 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff8c80186c0 0x7ff8c801abc0 crc :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:01.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:01.997 7ff8ef322700 1 -- 172.21.2.112:0/1402578707 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8e8082980 con 0x7ff8e808d490 2021-02-10T11:44:02.001 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:02.000 7ff8ddffb700 1 -- 172.21.2.112:0/1402578707 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff8d801f510 con 0x7ff8e808d490 2021-02-10T11:44:02.133 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:02.132 7ff8ef322700 1 -- 172.21.2.112:0/1402578707 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7ff8e8082980 con 0x7ff8e808d490 2021-02-10T11:44:02.136 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:02.135 7ff8ddffb700 1 -- 172.21.2.112:0/1402578707 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.3 v86) v1 ==== 184+0+0 (secure 0 0 0) 0x7ff8d8002030 con 0x7ff8e808d490 2021-02-10T11:44:02.140 INFO:teuthology.orchestra.run.gibba012.stderr:updated caps for client.3 2021-02-10T11:44:02.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:02.141 7ff8ef322700 1 -- 172.21.2.112:0/1402578707 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff8c80186c0 msgr2=0x7ff8c801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:02.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:02.141 7ff8ef322700 1 --2- 172.21.2.112:0/1402578707 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff8c80186c0 0x7ff8c801abc0 crc :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:02.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:02.141 7ff8ef322700 1 -- 172.21.2.112:0/1402578707 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff8e808d490 msgr2=0x7ff8e80beec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:02.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:02.141 7ff8ef322700 1 --2- 172.21.2.112:0/1402578707 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff8e808d490 0x7ff8e80beec0 secure :-1 s=READY pgs=2558 cs=0 l=1 rev1=1 rx=0x7ff8d800b5e0 tx=0x7ff8d800b980).stop 2021-02-10T11:44:02.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:02.141 7ff8ef322700 1 -- 172.21.2.112:0/1402578707 shutdown_connections 2021-02-10T11:44:02.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:02.141 7ff8ef322700 1 --2- 172.21.2.112:0/1402578707 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff8c80186c0 0x7ff8c801abc0 unknown :-1 s=CLOSED pgs=1173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:02.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:02.141 7ff8ef322700 1 --2- 172.21.2.112:0/1402578707 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff8e80bf4e0 0x7ff8e809fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:02.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:02.141 7ff8ef322700 1 --2- 172.21.2.112:0/1402578707 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff8e808d490 0x7ff8e80beec0 unknown :-1 s=CLOSED pgs=2558 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:02.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:02.141 7ff8ef322700 1 --2- 172.21.2.112:0/1402578707 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff8e808d090 0x7ff8e808c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:02.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:02.141 7ff8ef322700 1 -- 172.21.2.112:0/1402578707 >> 172.21.2.112:0/1402578707 conn(0x7ff8e8087390 msgr2=0x7ff8e80a6770 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:02.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:02.142 7ff8ef322700 1 -- 172.21.2.112:0/1402578707 shutdown_connections 2021-02-10T11:44:02.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:02.142 7ff8ef322700 1 -- 172.21.2.112:0/1402578707 wait complete. 2021-02-10T11:44:02.150 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:44:02.577 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:02.560 7fdb616c9700 1 Processor -- start 2021-02-10T11:44:02.577 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:02.560 7fdb616c9700 1 -- start start 2021-02-10T11:44:02.577 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:02.561 7fdb616c9700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdb5c0d0340 0x7fdb5c0ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:02.577 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:02.561 7fdb616c9700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb5c0f0b10 con 0x7fdb5c0d1c70 2021-02-10T11:44:02.578 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:02.561 7fdb616c9700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb5c1b0be0 con 0x7fdb5c0d1870 2021-02-10T11:44:02.578 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:02.561 7fdb616c9700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb5c0e5ed0 con 0x7fdb5c0d0340 2021-02-10T11:44:02.578 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:02.561 7fdb5bac0700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fdb5c0d1c70 0x7fdb5c0efc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:57062/0 (socket says 172.21.2.112:57062) 2021-02-10T11:44:02.578 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:02.561 7fdb5bac0700 1 -- 172.21.2.112:0/868036802 learned_addr learned my addr 172.21.2.112:0/868036802 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:44:02.578 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:02.561 7fdb5aabe700 1 --2- 172.21.2.112:0/868036802 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdb5c0d0340 0x7fdb5c0ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:02.578 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:02.561 7fdb5a2bd700 1 -- 172.21.2.112:0/868036802 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2563933735 0 0) 0x7fdb54001500 con 0x7fdb5c0d1c70 2021-02-10T11:44:02.578 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:02.561 7fdb5a2bd700 1 -- 172.21.2.112:0/868036802 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb48002fe0 con 0x7fdb5c0d1c70 2021-02-10T11:44:02.578 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:02.561 7fdb5aabe700 1 -- 172.21.2.112:0/868036802 >> v1:172.21.2.120:6789/0 conn(0x7fdb5c0d1870 legacy=0x7fdb5c1b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:02.578 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:02.561 7fdb5aabe700 1 -- 172.21.2.112:0/868036802 >> v1:172.21.2.112:6789/0 conn(0x7fdb5c0d1c70 legacy=0x7fdb5c0efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:02.579 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:02.561 7fdb5aabe700 1 -- 172.21.2.112:0/868036802 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb5c0f0450 con 0x7fdb5c0d0340 2021-02-10T11:44:02.579 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:02.562 7fdb5aabe700 1 --2- 172.21.2.112:0/868036802 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdb5c0d0340 0x7fdb5c0ef640 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7fdb4c006650 tx=0x7fdb4c009a80).ready entity=mon.2 client_cookie=5789ae1fc1e815a4 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:02.579 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:02.562 7fdb5a2bd700 1 -- 172.21.2.112:0/868036802 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdb4c00c0c0 con 0x7fdb5c0d0340 2021-02-10T11:44:02.579 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:02.562 7fdb5a2bd700 1 -- 172.21.2.112:0/868036802 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb4c00c340 con 0x7fdb5c0d0340 2021-02-10T11:44:02.579 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:02.562 7fdb5a2bd700 1 -- 172.21.2.112:0/868036802 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdb4c00c860 con 0x7fdb5c0d0340 2021-02-10T11:44:02.579 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:02.562 7fdb616c9700 1 -- 172.21.2.112:0/868036802 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdb5c0d0340 msgr2=0x7fdb5c0ef640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:02.579 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:02.562 7fdb616c9700 1 --2- 172.21.2.112:0/868036802 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdb5c0d0340 0x7fdb5c0ef640 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7fdb4c006650 tx=0x7fdb4c009a80).stop 2021-02-10T11:44:02.579 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:02.562 7fdb616c9700 1 -- 172.21.2.112:0/868036802 shutdown_connections 2021-02-10T11:44:02.579 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:02.562 7fdb616c9700 1 --2- 172.21.2.112:0/868036802 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdb5c0d0340 0x7fdb5c0ef640 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:02.579 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:02.562 7fdb616c9700 1 -- 172.21.2.112:0/868036802 >> 172.21.2.112:0/868036802 conn(0x7fdb5c0ce450 msgr2=0x7fdb5c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:02.580 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:02.563 7fdb616c9700 1 -- 172.21.2.112:0/868036802 shutdown_connections 2021-02-10T11:44:02.580 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:02.563 7fdb616c9700 1 -- 172.21.2.112:0/868036802 wait complete. 2021-02-10T11:44:02.581 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:02.577 7fdb616c9700 1 Processor -- start 2021-02-10T11:44:02.581 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:02.577 7fdb616c9700 1 -- start start 2021-02-10T11:44:02.581 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:02.577 7fdb616c9700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdb5c1a9bd0 0x7fdb5c0ef6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:02.582 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:02.577 7fdb616c9700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fdb5c08c0f0 0x7fdb5c08c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:02.582 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:02.577 7fdb616c9700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdb5c08cb10 0x7fdb5c08cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:02.582 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:02.577 7fdb616c9700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb5c0efd10 con 0x7fdb5c1a9bd0 2021-02-10T11:44:02.582 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:02.577 7fdb616c9700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb5c0ee000 con 0x7fdb5c08c0f0 2021-02-10T11:44:02.582 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:02.577 7fdb616c9700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb5c0ee220 con 0x7fdb5c08cb10 2021-02-10T11:44:02.582 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:02.577 7fdb5b2bf700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdb5c1a9bd0 0x7fdb5c0ef6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:02.582 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:02.577 7fdb5b2bf700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdb5c1a9bd0 0x7fdb5c0ef6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:57606/0 (socket says 172.21.2.112:57606) 2021-02-10T11:44:02.582 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:02.577 7fdb5b2bf700 1 -- 172.21.2.112:0/2021094826 learned_addr learned my addr 172.21.2.112:0/2021094826 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:44:02.582 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:02.578 7fdb5b2bf700 1 -- 172.21.2.112:0/2021094826 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdb5c08cb10 msgr2=0x7fdb5c08cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:02.582 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:02.578 7fdb5b2bf700 1 --2- 172.21.2.112:0/2021094826 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdb5c08cb10 0x7fdb5c08cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:02.582 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:02.578 7fdb5b2bf700 1 -- 172.21.2.112:0/2021094826 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fdb5c08c0f0 msgr2=0x7fdb5c08c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:02.582 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:02.578 7fdb5b2bf700 1 --2- 172.21.2.112:0/2021094826 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fdb5c08c0f0 0x7fdb5c08c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:02.583 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:02.578 7fdb5b2bf700 1 -- 172.21.2.112:0/2021094826 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb5c0ee440 con 0x7fdb5c1a9bd0 2021-02-10T11:44:02.583 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:02.578 7fdb5b2bf700 1 --2- 172.21.2.112:0/2021094826 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdb5c1a9bd0 0x7fdb5c0ef6f0 secure :-1 s=READY pgs=2560 cs=0 l=1 rev1=1 rx=0x7fdb4400a570 tx=0x7fdb4400a910).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:02.583 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:02.578 7fdb53fff700 1 -- 172.21.2.112:0/2021094826 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdb4400ad00 con 0x7fdb5c1a9bd0 2021-02-10T11:44:02.583 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:02.578 7fdb53fff700 1 -- 172.21.2.112:0/2021094826 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb4400c0d0 con 0x7fdb5c1a9bd0 2021-02-10T11:44:02.583 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:02.578 7fdb53fff700 1 -- 172.21.2.112:0/2021094826 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdb4400c520 con 0x7fdb5c1a9bd0 2021-02-10T11:44:02.583 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:02.578 7fdb616c9700 1 -- 172.21.2.112:0/2021094826 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb5c0ee440 con 0x7fdb5c1a9bd0 2021-02-10T11:44:02.583 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:02.578 7fdb616c9700 1 -- 172.21.2.112:0/2021094826 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb5c0ee440 con 0x7fdb5c1a9bd0 2021-02-10T11:44:02.583 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:02.579 7fdb53fff700 1 -- 172.21.2.112:0/2021094826 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fdb44008030 con 0x7fdb5c1a9bd0 2021-02-10T11:44:02.583 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:02.579 7fdb53fff700 1 --2- 172.21.2.112:0/2021094826 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fdb400186c0 0x7fdb4001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:02.583 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:02.579 7fdb53fff700 1 -- 172.21.2.112:0/2021094826 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fdb440325e0 con 0x7fdb5c1a9bd0 2021-02-10T11:44:02.583 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:02.579 7fdb5aabe700 1 --2- 172.21.2.112:0/2021094826 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fdb400186c0 0x7fdb4001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:02.583 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:02.579 7fdb5aabe700 1 --2- 172.21.2.112:0/2021094826 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fdb400186c0 0x7fdb4001abc0 crc :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:02.611 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:02.610 7fdb616c9700 1 -- 172.21.2.112:0/2021094826 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb5c0efd10 con 0x7fdb5c1a9bd0 2021-02-10T11:44:02.613 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:02.613 7fdb53fff700 1 -- 172.21.2.112:0/2021094826 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fdb44030d90 con 0x7fdb5c1a9bd0 2021-02-10T11:44:02.742 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:02.741 7fdb616c9700 1 -- 172.21.2.112:0/2021094826 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdb5c0efd10 con 0x7fdb5c1a9bd0 2021-02-10T11:44:02.743 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:02.742 7fdb53fff700 1 -- 172.21.2.112:0/2021094826 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) v1 ==== 95+0+2108 (secure 0 0 0) 0x7fdb4401f560 con 0x7fdb5c1a9bd0 2021-02-10T11:44:02.743 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:44:02.743 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":122,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11670,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/1286703435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":1286703435},{"type":"v1","addr":"172.21.2.112:6833","nonce":1286703435}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":119}],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:43:56.409867","modified":"2021-02-10 11:43:58.413644","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8578},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8578":{"gid":8578,"name":"b","rank":0,"incarnation":120,"state":"up:active","state_seq":3,"addr":"172.21.2.120:6835/1504781062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":1504781062},{"type":"v1","addr":"172.21.2.120:6835","nonce":1504781062}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":15}]} 2021-02-10T11:44:02.743 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 122 2021-02-10T11:44:02.744 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:02.744 7fdb616c9700 1 -- 172.21.2.112:0/2021094826 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fdb400186c0 msgr2=0x7fdb4001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:02.744 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:02.744 7fdb616c9700 1 --2- 172.21.2.112:0/2021094826 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fdb400186c0 0x7fdb4001abc0 crc :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:02.744 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:02.744 7fdb616c9700 1 -- 172.21.2.112:0/2021094826 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdb5c1a9bd0 msgr2=0x7fdb5c0ef6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:02.744 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:02.744 7fdb616c9700 1 --2- 172.21.2.112:0/2021094826 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdb5c1a9bd0 0x7fdb5c0ef6f0 secure :-1 s=READY pgs=2560 cs=0 l=1 rev1=1 rx=0x7fdb4400a570 tx=0x7fdb4400a910).stop 2021-02-10T11:44:02.745 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:02.744 7fdb616c9700 1 -- 172.21.2.112:0/2021094826 shutdown_connections 2021-02-10T11:44:02.745 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:02.744 7fdb616c9700 1 --2- 172.21.2.112:0/2021094826 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fdb400186c0 0x7fdb4001abc0 unknown :-1 s=CLOSED pgs=1174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:02.745 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:02.744 7fdb616c9700 1 --2- 172.21.2.112:0/2021094826 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fdb5c08c0f0 0x7fdb5c08c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:02.745 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:02.744 7fdb616c9700 1 --2- 172.21.2.112:0/2021094826 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdb5c08cb10 0x7fdb5c08cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:02.745 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:02.744 7fdb616c9700 1 --2- 172.21.2.112:0/2021094826 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdb5c1a9bd0 0x7fdb5c0ef6f0 unknown :-1 s=CLOSED pgs=2560 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:02.745 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:02.744 7fdb616c9700 1 -- 172.21.2.112:0/2021094826 >> 172.21.2.112:0/2021094826 conn(0x7fdb5c19f8c0 msgr2=0x7fdb5c1aa180 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:02.746 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:02.744 7fdb616c9700 1 -- 172.21.2.112:0/2021094826 shutdown_connections 2021-02-10T11:44:02.746 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:02.744 7fdb616c9700 1 -- 172.21.2.112:0/2021094826 wait complete. 2021-02-10T11:44:02.955 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 122, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:43:56.409867', 'modified': '2021-02-10 11:43:58.413644', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8578}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8578': {'gid': 8578, 'name': 'b', 'rank': 0, 'incarnation': 120, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.120:6835/1504781062', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.120:6834', 'nonce': 1504781062}, {'type': 'v1', 'addr': '172.21.2.120:6835', 'nonce': 1504781062}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [31], 'metadata_pool': 30, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:44:02.955 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:44:02.955 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:44:03.354 INFO:teuthology.orchestra.run.gibba020.stdout:{ 2021-02-10T11:44:03.355 INFO:teuthology.orchestra.run.gibba020.stdout: "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", 2021-02-10T11:44:03.355 INFO:teuthology.orchestra.run.gibba020.stdout: "whoami": 0, 2021-02-10T11:44:03.355 INFO:teuthology.orchestra.run.gibba020.stdout: "id": 8578, 2021-02-10T11:44:03.355 INFO:teuthology.orchestra.run.gibba020.stdout: "want_state": "up:active", 2021-02-10T11:44:03.355 INFO:teuthology.orchestra.run.gibba020.stdout: "state": "up:active", 2021-02-10T11:44:03.355 INFO:teuthology.orchestra.run.gibba020.stdout: "rank_uptime": 6.9426199500000001, 2021-02-10T11:44:03.355 INFO:teuthology.orchestra.run.gibba020.stdout: "mdsmap_epoch": 122, 2021-02-10T11:44:03.355 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch": 143, 2021-02-10T11:44:03.355 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch_barrier": 143, 2021-02-10T11:44:03.355 INFO:teuthology.orchestra.run.gibba020.stdout: "uptime": 12.588123295000001 2021-02-10T11:44:03.355 INFO:teuthology.orchestra.run.gibba020.stdout:} 2021-02-10T11:44:03.359 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", "whoami": 0, "id": 8578, "want_state": "up:active", "state": "up:active", "rank_uptime": 6.9426199500000001, "mdsmap_epoch": 122, "osdmap_epoch": 143, "osdmap_epoch_barrier": 143, "uptime": 12.588123295000001 } 2021-02-10T11:44:03.359 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:44:03.788 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:03.771 7f362378f700 1 Processor -- start 2021-02-10T11:44:03.788 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:03.771 7f362378f700 1 -- start start 2021-02-10T11:44:03.788 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:03.771 7f362378f700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f361c1b0870 0x7f361c0f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:03.788 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:03.771 7f362378f700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f361c1b0c70 con 0x7f361c0efe00 2021-02-10T11:44:03.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:03.771 7f362378f700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f361c0f0b50 con 0x7f361c1b0470 2021-02-10T11:44:03.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:03.771 7f362378f700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f361c0e5ea0 con 0x7f361c1b0870 2021-02-10T11:44:03.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:03.771 7f3620daa700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f361c1b0870 0x7f361c0f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:03.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:03.771 7f3620daa700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f361c1b0870 0x7f361c0f20e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:57610/0 (socket says 172.21.2.112:57610) 2021-02-10T11:44:03.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:03.771 7f3620daa700 1 -- 172.21.2.112:0/1233354413 learned_addr learned my addr 172.21.2.112:0/1233354413 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:44:03.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:03.771 7f3621dac700 1 --1- 172.21.2.112:0/1233354413 >> v1:172.21.2.112:6789/0 conn(0x7f361c0efe00 0x7f361c0f0200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:57076/0 (socket says 172.21.2.112:57076) 2021-02-10T11:44:03.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:03.772 7f3620daa700 1 -- 172.21.2.112:0/1233354413 >> v1:172.21.2.120:6789/0 conn(0x7f361c1b0470 legacy=0x7f361c0ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:03.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:03.772 7f3620daa700 1 -- 172.21.2.112:0/1233354413 >> v1:172.21.2.112:6789/0 conn(0x7f361c0efe00 legacy=0x7f361c0f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:03.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:03.772 7f360ffff700 1 -- 172.21.2.112:0/1233354413 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 825339672 0 0) 0x7f3614000f90 con 0x7f361c0efe00 2021-02-10T11:44:03.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:03.772 7f3620daa700 1 -- 172.21.2.112:0/1233354413 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f361c0e52e0 con 0x7f361c1b0870 2021-02-10T11:44:03.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:03.772 7f3620daa700 1 --2- 172.21.2.112:0/1233354413 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f361c1b0870 0x7f361c0f20e0 secure :-1 s=READY pgs=2562 cs=0 l=1 rev1=1 rx=0x7f3618006700 tx=0x7f3618009b80).ready entity=mon.0 client_cookie=32e70d30728a5908 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:03.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:03.772 7f360ffff700 1 -- 172.21.2.112:0/1233354413 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f361800c0c0 con 0x7f361c1b0870 2021-02-10T11:44:03.790 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:03.772 7f360ffff700 1 -- 172.21.2.112:0/1233354413 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f361800c440 con 0x7f361c1b0870 2021-02-10T11:44:03.790 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:03.772 7f360ffff700 1 -- 172.21.2.112:0/1233354413 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f361800c960 con 0x7f361c1b0870 2021-02-10T11:44:03.790 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:03.773 7f362378f700 1 -- 172.21.2.112:0/1233354413 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f361c1b0870 msgr2=0x7f361c0f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:03.790 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:03.773 7f362378f700 1 --2- 172.21.2.112:0/1233354413 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f361c1b0870 0x7f361c0f20e0 secure :-1 s=READY pgs=2562 cs=0 l=1 rev1=1 rx=0x7f3618006700 tx=0x7f3618009b80).stop 2021-02-10T11:44:03.790 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:03.773 7f362378f700 1 -- 172.21.2.112:0/1233354413 shutdown_connections 2021-02-10T11:44:03.790 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:03.773 7f362378f700 1 --2- 172.21.2.112:0/1233354413 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f361c1b0870 0x7f361c0f20e0 unknown :-1 s=CLOSED pgs=2562 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:03.790 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:03.773 7f362378f700 1 -- 172.21.2.112:0/1233354413 >> 172.21.2.112:0/1233354413 conn(0x7f361c0ce3e0 msgr2=0x7f361c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:03.790 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:03.773 7f362378f700 1 -- 172.21.2.112:0/1233354413 shutdown_connections 2021-02-10T11:44:03.790 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:03.773 7f362378f700 1 -- 172.21.2.112:0/1233354413 wait complete. 2021-02-10T11:44:03.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:03.788 7f362378f700 1 Processor -- start 2021-02-10T11:44:03.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:03.788 7f362378f700 1 -- start start 2021-02-10T11:44:03.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:03.789 7f362378f700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f361c1aa660 0x7f361c097060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:03.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:03.789 7f362378f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f361c0b35b0 0x7f361c0b39b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:03.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:03.789 7f362378f700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f361c0b3fd0 0x7f361c095fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:03.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:03.789 7f362378f700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f361c097700 con 0x7f361c0b35b0 2021-02-10T11:44:03.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:03.789 7f362378f700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f361c093680 con 0x7f361c1aa660 2021-02-10T11:44:03.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:03.789 7f362378f700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f361c1aaab0 con 0x7f361c0b3fd0 2021-02-10T11:44:03.793 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:03.789 7f3620daa700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f361c0b35b0 0x7f361c0b39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:03.793 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:03.789 7f3620daa700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f361c0b35b0 0x7f361c0b39b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:57618/0 (socket says 172.21.2.112:57618) 2021-02-10T11:44:03.793 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:03.789 7f3620daa700 1 -- 172.21.2.112:0/1644577124 learned_addr learned my addr 172.21.2.112:0/1644577124 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:44:03.793 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:03.789 7f36215ab700 1 --2- 172.21.2.112:0/1644577124 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f361c1aa660 0x7f361c097060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:03.793 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:03.789 7f3621dac700 1 --2- 172.21.2.112:0/1644577124 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f361c0b3fd0 0x7f361c095fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:03.793 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:03.789 7f3620daa700 1 -- 172.21.2.112:0/1644577124 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f361c0b3fd0 msgr2=0x7f361c095fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:03.793 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:03.789 7f3620daa700 1 --2- 172.21.2.112:0/1644577124 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f361c0b3fd0 0x7f361c095fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:03.793 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:03.789 7f3620daa700 1 -- 172.21.2.112:0/1644577124 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f361c1aa660 msgr2=0x7f361c097060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:03.793 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:03.789 7f3620daa700 1 --2- 172.21.2.112:0/1644577124 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f361c1aa660 0x7f361c097060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:03.793 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:03.789 7f3620daa700 1 -- 172.21.2.112:0/1644577124 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f361c0965e0 con 0x7f361c0b35b0 2021-02-10T11:44:03.793 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:03.789 7f36215ab700 1 --2- 172.21.2.112:0/1644577124 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f361c1aa660 0x7f361c097060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:44:03.793 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:03.789 7f3620daa700 1 --2- 172.21.2.112:0/1644577124 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f361c0b35b0 0x7f361c0b39b0 secure :-1 s=READY pgs=2563 cs=0 l=1 rev1=1 rx=0x7f361800b5e0 tx=0x7f361800b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:03.793 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:03.789 7f360dffb700 1 -- 172.21.2.112:0/1644577124 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f361800bd70 con 0x7f361c0b35b0 2021-02-10T11:44:03.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:03.789 7f360dffb700 1 -- 172.21.2.112:0/1644577124 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3618002030 con 0x7f361c0b35b0 2021-02-10T11:44:03.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:03.789 7f362378f700 1 -- 172.21.2.112:0/1644577124 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f361c0965e0 con 0x7f361c0b35b0 2021-02-10T11:44:03.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:03.790 7f360dffb700 1 -- 172.21.2.112:0/1644577124 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3618002550 con 0x7f361c0b35b0 2021-02-10T11:44:03.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:03.790 7f362378f700 1 -- 172.21.2.112:0/1644577124 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f361c096b70 con 0x7f361c0b35b0 2021-02-10T11:44:03.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:03.790 7f360dffb700 1 -- 172.21.2.112:0/1644577124 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3618002210 con 0x7f361c0b35b0 2021-02-10T11:44:03.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:03.790 7f360dffb700 1 --2- 172.21.2.112:0/1644577124 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f35fc018730 0x7f35fc01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:03.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:03.790 7f360dffb700 1 -- 172.21.2.112:0/1644577124 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f3618032620 con 0x7f361c0b35b0 2021-02-10T11:44:03.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:03.790 7f36215ab700 1 --2- 172.21.2.112:0/1644577124 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f35fc018730 0x7f35fc01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:03.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:03.791 7f36215ab700 1 --2- 172.21.2.112:0/1644577124 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f35fc018730 0x7f35fc01ac10 crc :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:03.822 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:03.821 7f362378f700 1 -- 172.21.2.112:0/1644577124 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f361c096e60 con 0x7f361c0b35b0 2021-02-10T11:44:03.824 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:03.824 7f360dffb700 1 -- 172.21.2.112:0/1644577124 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3618002240 con 0x7f361c0b35b0 2021-02-10T11:44:03.949 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:03.949 7f362378f700 1 -- 172.21.2.112:0/1644577124 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f361c096e60 con 0x7f361c0b35b0 2021-02-10T11:44:03.950 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:03.949 7f360dffb700 1 -- 172.21.2.112:0/1644577124 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v143) v1 ==== 74+0+14007 (secure 0 0 0) 0x7f3618002030 con 0x7f361c0b35b0 2021-02-10T11:44:03.950 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:44:03.951 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":143,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:43:57.411337","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":31,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":30,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:43:54.385734","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"142","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":31,"pool_name":"cephfs_data","create_time":"2021-02-10 11:43:55.453740","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"143","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":139,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":139,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:44:03.952 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:03.951 7f362378f700 1 -- 172.21.2.112:0/1644577124 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f35fc018730 msgr2=0x7f35fc01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:03.952 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:03.951 7f362378f700 1 --2- 172.21.2.112:0/1644577124 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f35fc018730 0x7f35fc01ac10 crc :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:03.952 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:03.951 7f362378f700 1 -- 172.21.2.112:0/1644577124 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f361c0b35b0 msgr2=0x7f361c0b39b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:03.952 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:03.951 7f362378f700 1 --2- 172.21.2.112:0/1644577124 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f361c0b35b0 0x7f361c0b39b0 secure :-1 s=READY pgs=2563 cs=0 l=1 rev1=1 rx=0x7f361800b5e0 tx=0x7f361800b980).stop 2021-02-10T11:44:03.952 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:03.951 7f362378f700 1 -- 172.21.2.112:0/1644577124 shutdown_connections 2021-02-10T11:44:03.952 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:03.951 7f362378f700 1 --2- 172.21.2.112:0/1644577124 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f35fc018730 0x7f35fc01ac10 unknown :-1 s=CLOSED pgs=1175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:03.953 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:03.951 7f362378f700 1 --2- 172.21.2.112:0/1644577124 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f361c0b3fd0 0x7f361c095fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:03.953 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:03.951 7f362378f700 1 --2- 172.21.2.112:0/1644577124 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f361c0b35b0 0x7f361c0b39b0 unknown :-1 s=CLOSED pgs=2563 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:03.953 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:03.951 7f362378f700 1 --2- 172.21.2.112:0/1644577124 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f361c1aa660 0x7f361c097060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:03.953 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:03.951 7f362378f700 1 -- 172.21.2.112:0/1644577124 >> 172.21.2.112:0/1644577124 conn(0x7f361c094250 msgr2=0x7f361c18c230 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:03.953 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:03.951 7f362378f700 1 -- 172.21.2.112:0/1644577124 shutdown_connections 2021-02-10T11:44:03.953 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:03.951 7f362378f700 1 -- 172.21.2.112:0/1644577124 wait complete. 2021-02-10T11:44:03.960 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:13:57.508758', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 30, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:43:54.385734', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '142', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 31, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:43:55.453740', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '143', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:44:03.961 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:44:04.394 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:04.377 7f87deb04700 1 Processor -- start 2021-02-10T11:44:04.394 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:04.377 7f87deb04700 1 -- start start 2021-02-10T11:44:04.394 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:04.377 7f87deb04700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f87d80d1860 0x7f87d81b04b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:04.394 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:04.377 7f87deb04700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f87d81b0ad0 0x7f87d8108930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:04.394 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:04.378 7f87deb04700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87d80fe1b0 con 0x7f87d80d1c60 2021-02-10T11:44:04.394 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:04.378 7f87deb04700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87d8109a50 con 0x7f87d80d1860 2021-02-10T11:44:04.394 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:04.378 7f87deb04700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87d8109740 con 0x7f87d81b0ad0 2021-02-10T11:44:04.394 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:04.378 7f87dd121700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f87d80d1c60 0x7f87d8108f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:57088/0 (socket says 172.21.2.112:57088) 2021-02-10T11:44:04.394 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:04.378 7f87dd121700 1 -- 172.21.2.112:0/3803084700 learned_addr learned my addr 172.21.2.112:0/3803084700 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:44:04.395 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:04.378 7f87dc920700 1 --2- 172.21.2.112:0/3803084700 >> v2:172.21.2.120:3300/0 conn(0x7f87d80d1860 0x7f87d81b04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:04.395 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:04.378 7f87cffff700 1 --2- 172.21.2.112:0/3803084700 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f87d81b0ad0 0x7f87d8108930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:04.395 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:04.378 7f87dc920700 1 -- 172.21.2.112:0/3803084700 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f87d81b0ad0 msgr2=0x7f87d8108930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:04.395 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:04.378 7f87dc920700 1 --2- 172.21.2.112:0/3803084700 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f87d81b0ad0 0x7f87d8108930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:04.395 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:04.378 7f87cf7fe700 1 -- 172.21.2.112:0/3803084700 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2407947613 0 0) 0x7f87d0001500 con 0x7f87d80d1c60 2021-02-10T11:44:04.395 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:04.378 7f87dc920700 1 -- 172.21.2.112:0/3803084700 >> v1:172.21.2.112:6789/0 conn(0x7f87d80d1c60 legacy=0x7f87d8108f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:04.395 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:04.378 7f87dc920700 1 -- 172.21.2.112:0/3803084700 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87d80d2fd0 con 0x7f87d80d1860 2021-02-10T11:44:04.395 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:04.378 7f87dc920700 1 --2- 172.21.2.112:0/3803084700 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f87d80d1860 0x7f87d81b04b0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 rx=0x7f87c8006650 tx=0x7f87c8009a00).ready entity=mon.1 client_cookie=abd252524aee7f23 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:04.395 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:04.378 7f87cf7fe700 1 -- 172.21.2.112:0/3803084700 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f87c80009b0 con 0x7f87d80d1860 2021-02-10T11:44:04.395 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:04.378 7f87cf7fe700 1 -- 172.21.2.112:0/3803084700 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87c8009df0 con 0x7f87d80d1860 2021-02-10T11:44:04.395 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:04.379 7f87cf7fe700 1 -- 172.21.2.112:0/3803084700 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f87c800c0c0 con 0x7f87d80d1860 2021-02-10T11:44:04.395 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:04.379 7f87deb04700 1 -- 172.21.2.112:0/3803084700 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f87d80d1860 msgr2=0x7f87d81b04b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:04.395 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:04.379 7f87deb04700 1 --2- 172.21.2.112:0/3803084700 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f87d80d1860 0x7f87d81b04b0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 rx=0x7f87c8006650 tx=0x7f87c8009a00).stop 2021-02-10T11:44:04.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:04.379 7f87deb04700 1 -- 172.21.2.112:0/3803084700 shutdown_connections 2021-02-10T11:44:04.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:04.379 7f87deb04700 1 --2- 172.21.2.112:0/3803084700 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f87d81b0ad0 0x7f87d8108930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:04.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:04.379 7f87deb04700 1 --2- 172.21.2.112:0/3803084700 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f87d80d1860 0x7f87d81b04b0 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:04.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:04.379 7f87deb04700 1 -- 172.21.2.112:0/3803084700 >> 172.21.2.112:0/3803084700 conn(0x7f87d80ce410 msgr2=0x7f87d80d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:04.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:04.379 7f87deb04700 1 -- 172.21.2.112:0/3803084700 shutdown_connections 2021-02-10T11:44:04.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:04.379 7f87deb04700 1 -- 172.21.2.112:0/3803084700 wait complete. 2021-02-10T11:44:04.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:04.393 7f87deb04700 1 Processor -- start 2021-02-10T11:44:04.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:04.393 7f87deb04700 1 -- start start 2021-02-10T11:44:04.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:04.394 7f87deb04700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f87d819e3c0 0x7f87d80c3310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:04.398 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:04.394 7f87deb04700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f87d80c3930 0x7f87d8187e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:04.398 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:04.394 7f87deb04700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f87d8188480 0x7f87d8083930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:04.398 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:04.394 7f87deb04700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87d80c3e90 con 0x7f87d80c3930 2021-02-10T11:44:04.398 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:04.394 7f87deb04700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87d8188880 con 0x7f87d8188480 2021-02-10T11:44:04.398 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:04.394 7f87deb04700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87d8188a70 con 0x7f87d819e3c0 2021-02-10T11:44:04.398 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:04.394 7f87cffff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f87d80c3930 0x7f87d8187e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:04.398 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:04.394 7f87cffff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f87d80c3930 0x7f87d8187e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:57630/0 (socket says 172.21.2.112:57630) 2021-02-10T11:44:04.398 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:04.394 7f87cffff700 1 -- 172.21.2.112:0/3842415254 learned_addr learned my addr 172.21.2.112:0/3842415254 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:44:04.398 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:04.394 7f87dd121700 1 --2- 172.21.2.112:0/3842415254 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f87d8188480 0x7f87d8083930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:04.398 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:04.394 7f87dc920700 1 --2- 172.21.2.112:0/3842415254 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f87d819e3c0 0x7f87d80c3310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:04.398 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:04.394 7f87cffff700 1 -- 172.21.2.112:0/3842415254 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f87d819e3c0 msgr2=0x7f87d80c3310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:04.398 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:04.394 7f87cffff700 1 --2- 172.21.2.112:0/3842415254 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f87d819e3c0 0x7f87d80c3310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:04.398 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:04.394 7f87cffff700 1 -- 172.21.2.112:0/3842415254 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f87d8188480 msgr2=0x7f87d8083930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:04.399 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:04.394 7f87cffff700 1 --2- 172.21.2.112:0/3842415254 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f87d8188480 0x7f87d8083930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:04.399 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:04.394 7f87cffff700 1 -- 172.21.2.112:0/3842415254 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87d8083f50 con 0x7f87d80c3930 2021-02-10T11:44:04.399 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:04.395 7f87cffff700 1 --2- 172.21.2.112:0/3842415254 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f87d80c3930 0x7f87d8187e60 secure :-1 s=READY pgs=2565 cs=0 l=1 rev1=1 rx=0x7f87d4006f60 tx=0x7f87d400b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:04.399 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:04.395 7f87cd7fa700 1 -- 172.21.2.112:0/3842415254 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f87d400bd60 con 0x7f87d80c3930 2021-02-10T11:44:04.399 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:04.395 7f87cd7fa700 1 -- 172.21.2.112:0/3842415254 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87d4000c90 con 0x7f87d80c3930 2021-02-10T11:44:04.399 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:04.395 7f87cd7fa700 1 -- 172.21.2.112:0/3842415254 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f87d4002220 con 0x7f87d80c3930 2021-02-10T11:44:04.399 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:04.395 7f87deb04700 1 -- 172.21.2.112:0/3842415254 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87d8083f50 con 0x7f87d80c3930 2021-02-10T11:44:04.399 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:04.395 7f87deb04700 1 -- 172.21.2.112:0/3842415254 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87d8083f50 con 0x7f87d80c3930 2021-02-10T11:44:04.399 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:04.395 7f87cd7fa700 1 -- 172.21.2.112:0/3842415254 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f87d40052e0 con 0x7f87d80c3930 2021-02-10T11:44:04.399 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:04.395 7f87cd7fa700 1 --2- 172.21.2.112:0/3842415254 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f87b80186d0 0x7f87b801abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:04.399 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:04.395 7f87cd7fa700 1 -- 172.21.2.112:0/3842415254 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f87d4032640 con 0x7f87d80c3930 2021-02-10T11:44:04.399 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:04.396 7f87dc920700 1 --2- 172.21.2.112:0/3842415254 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f87b80186d0 0x7f87b801abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:04.399 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:04.396 7f87dc920700 1 --2- 172.21.2.112:0/3842415254 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f87b80186d0 0x7f87b801abd0 crc :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:04.427 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:04.427 7f87deb04700 1 -- 172.21.2.112:0/3842415254 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87d8082a20 con 0x7f87d80c3930 2021-02-10T11:44:04.430 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:04.429 7f87cd7fa700 1 -- 172.21.2.112:0/3842415254 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f87d4000c90 con 0x7f87d80c3930 2021-02-10T11:44:04.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:04.558 7f87deb04700 1 -- 172.21.2.112:0/3842415254 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f87d8083f50 con 0x7f87d80c3930 2021-02-10T11:44:04.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:04.558 7f87cd7fa700 1 -- 172.21.2.112:0/3842415254 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v143) v1 ==== 92+0+10 (secure 0 0 0) 0x7f87d4006b00 con 0x7f87d80c3930 2021-02-10T11:44:04.559 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:44:04.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:04.560 7f87deb04700 1 -- 172.21.2.112:0/3842415254 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f87b80186d0 msgr2=0x7f87b801abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:04.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:04.560 7f87deb04700 1 --2- 172.21.2.112:0/3842415254 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f87b80186d0 0x7f87b801abd0 crc :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:04.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:04.560 7f87deb04700 1 -- 172.21.2.112:0/3842415254 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f87d80c3930 msgr2=0x7f87d8187e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:04.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:04.560 7f87deb04700 1 --2- 172.21.2.112:0/3842415254 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f87d80c3930 0x7f87d8187e60 secure :-1 s=READY pgs=2565 cs=0 l=1 rev1=1 rx=0x7f87d4006f60 tx=0x7f87d400b970).stop 2021-02-10T11:44:04.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:04.560 7f87deb04700 1 -- 172.21.2.112:0/3842415254 shutdown_connections 2021-02-10T11:44:04.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:04.560 7f87deb04700 1 --2- 172.21.2.112:0/3842415254 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f87b80186d0 0x7f87b801abd0 unknown :-1 s=CLOSED pgs=1176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:04.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:04.560 7f87deb04700 1 --2- 172.21.2.112:0/3842415254 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f87d8188480 0x7f87d8083930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:04.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:04.560 7f87deb04700 1 --2- 172.21.2.112:0/3842415254 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f87d80c3930 0x7f87d8187e60 unknown :-1 s=CLOSED pgs=2565 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:04.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:04.560 7f87deb04700 1 --2- 172.21.2.112:0/3842415254 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f87d819e3c0 0x7f87d80c3310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:04.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:04.560 7f87deb04700 1 -- 172.21.2.112:0/3842415254 >> 172.21.2.112:0/3842415254 conn(0x7f87d80b6e50 msgr2=0x7f87d8082c50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:04.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:04.561 7f87deb04700 1 -- 172.21.2.112:0/3842415254 shutdown_connections 2021-02-10T11:44:04.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:04.561 7f87deb04700 1 -- 172.21.2.112:0/3842415254 wait complete. 2021-02-10T11:44:04.569 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:44:05.001 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:04.984 7fea08ab4700 1 Processor -- start 2021-02-10T11:44:05.001 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:04.984 7fea08ab4700 1 -- start start 2021-02-10T11:44:05.001 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:04.984 7fea08ab4700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fea040d1870 0x7fea041b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:05.001 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:04.984 7fea08ab4700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fea041b0ae0 0x7fea04108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:05.001 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:04.984 7fea08ab4700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fea040d1c70 0x7fea04108f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:05.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:04.984 7fea08ab4700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea040fe1c0 con 0x7fea041b0ae0 2021-02-10T11:44:05.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:04.984 7fea08ab4700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea04109a60 con 0x7fea040d1870 2021-02-10T11:44:05.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:04.984 7fea08ab4700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea04088070 con 0x7fea040d1c70 2021-02-10T11:44:05.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:04.985 7fea0205d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fea041b0ae0 0x7fea04108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:05.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:04.985 7fea0205d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fea041b0ae0 0x7fea04108940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:57640/0 (socket says 172.21.2.112:57640) 2021-02-10T11:44:05.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:04.985 7fea0205d700 1 -- 172.21.2.112:0/4047728247 learned_addr learned my addr 172.21.2.112:0/4047728247 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:44:05.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:04.985 7fea0205d700 1 -- 172.21.2.112:0/4047728247 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fea040d1c70 msgr2=0x7fea04108f60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:44:05.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:04.985 7fea0205d700 1 --2- 172.21.2.112:0/4047728247 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fea040d1c70 0x7fea04108f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:05.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:04.985 7fea0205d700 1 -- 172.21.2.112:0/4047728247 >> v2:172.21.2.120:3300/0 conn(0x7fea040d1870 msgr2=0x7fea041b04c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:44:05.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:04.985 7fea0205d700 1 --2- 172.21.2.112:0/4047728247 >> v2:172.21.2.120:3300/0 conn(0x7fea040d1870 0x7fea041b04c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:05.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:04.985 7fea0205d700 1 -- 172.21.2.112:0/4047728247 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea04109580 con 0x7fea041b0ae0 2021-02-10T11:44:05.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:04.985 7fea0205d700 1 --2- 172.21.2.112:0/4047728247 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fea041b0ae0 0x7fea04108940 secure :-1 s=READY pgs=2566 cs=0 l=1 rev1=1 rx=0x7fe9f4006700 tx=0x7fe9f4009b80).ready entity=mon.0 client_cookie=c872db323761508d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:05.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:04.985 7fea0185c700 1 -- 172.21.2.112:0/4047728247 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe9f400c0c0 con 0x7fea041b0ae0 2021-02-10T11:44:05.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:04.985 7fea0185c700 1 -- 172.21.2.112:0/4047728247 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9f400c400 con 0x7fea041b0ae0 2021-02-10T11:44:05.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:04.985 7fea0185c700 1 -- 172.21.2.112:0/4047728247 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe9f400c920 con 0x7fea041b0ae0 2021-02-10T11:44:05.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:04.985 7fea08ab4700 1 -- 172.21.2.112:0/4047728247 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fea041b0ae0 msgr2=0x7fea04108940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:05.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:04.985 7fea08ab4700 1 --2- 172.21.2.112:0/4047728247 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fea041b0ae0 0x7fea04108940 secure :-1 s=READY pgs=2566 cs=0 l=1 rev1=1 rx=0x7fe9f4006700 tx=0x7fe9f4009b80).stop 2021-02-10T11:44:05.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:04.986 7fea08ab4700 1 -- 172.21.2.112:0/4047728247 shutdown_connections 2021-02-10T11:44:05.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:04.986 7fea08ab4700 1 --2- 172.21.2.112:0/4047728247 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fea040d1c70 0x7fea04108f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:05.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:04.986 7fea08ab4700 1 --2- 172.21.2.112:0/4047728247 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fea041b0ae0 0x7fea04108940 unknown :-1 s=CLOSED pgs=2566 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:05.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:04.986 7fea08ab4700 1 --2- 172.21.2.112:0/4047728247 >> v2:172.21.2.120:3300/0 conn(0x7fea040d1870 0x7fea041b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:05.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:04.986 7fea08ab4700 1 -- 172.21.2.112:0/4047728247 >> 172.21.2.112:0/4047728247 conn(0x7fea040ce420 msgr2=0x7fea040d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:05.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:04.986 7fea08ab4700 1 -- 172.21.2.112:0/4047728247 shutdown_connections 2021-02-10T11:44:05.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:04.986 7fea08ab4700 1 -- 172.21.2.112:0/4047728247 wait complete. 2021-02-10T11:44:05.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:05.000 7fea08ab4700 1 Processor -- start 2021-02-10T11:44:05.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:05.000 7fea08ab4700 1 -- start start 2021-02-10T11:44:05.005 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:05.001 7fea08ab4700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fea040c3820 0x7fea04084cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:05.005 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:05.001 7fea08ab4700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fea040a0290 0x7fea040852d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:05.005 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:05.001 7fea08ab4700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fea04086270 0x7fea04086670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:05.006 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:05.001 7fea08ab4700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea040c0700 con 0x7fea04086270 2021-02-10T11:44:05.006 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:05.001 7fea08ab4700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea040858f0 con 0x7fea040a0290 2021-02-10T11:44:05.006 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:05.001 7fea08ab4700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea04085b10 con 0x7fea040c3820 2021-02-10T11:44:05.006 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:05.001 7fea0305f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fea04086270 0x7fea04086670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:05.006 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:05.001 7fea0305f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fea04086270 0x7fea04086670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:57646/0 (socket says 172.21.2.112:57646) 2021-02-10T11:44:05.006 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:05.001 7fea0305f700 1 -- 172.21.2.112:0/3341883321 learned_addr learned my addr 172.21.2.112:0/3341883321 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:44:05.006 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:05.001 7fea0305f700 1 -- 172.21.2.112:0/3341883321 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fea040c3820 msgr2=0x7fea04084cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:05.006 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:05.001 7fea0305f700 1 --2- 172.21.2.112:0/3341883321 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fea040c3820 0x7fea04084cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:05.006 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:05.001 7fea0305f700 1 -- 172.21.2.112:0/3341883321 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fea040a0290 msgr2=0x7fea040852d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:05.006 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:05.001 7fea0305f700 1 --2- 172.21.2.112:0/3341883321 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fea040a0290 0x7fea040852d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:05.006 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:05.001 7fea0305f700 1 -- 172.21.2.112:0/3341883321 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea04086c90 con 0x7fea04086270 2021-02-10T11:44:05.006 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:05.001 7fea0305f700 1 --2- 172.21.2.112:0/3341883321 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fea04086270 0x7fea04086670 secure :-1 s=READY pgs=2567 cs=0 l=1 rev1=1 rx=0x7fe9fc009960 tx=0x7fe9fc00bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:05.007 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:05.002 7fe9fb7fe700 1 -- 172.21.2.112:0/3341883321 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe9fc000d10 con 0x7fea04086270 2021-02-10T11:44:05.007 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:05.002 7fe9fb7fe700 1 -- 172.21.2.112:0/3341883321 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9fc0030c0 con 0x7fea04086270 2021-02-10T11:44:05.007 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:05.002 7fe9fb7fe700 1 -- 172.21.2.112:0/3341883321 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe9fc001500 con 0x7fea04086270 2021-02-10T11:44:05.007 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:05.002 7fea08ab4700 1 -- 172.21.2.112:0/3341883321 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea04086c90 con 0x7fea04086270 2021-02-10T11:44:05.007 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:05.002 7fea08ab4700 1 -- 172.21.2.112:0/3341883321 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea041a2f00 con 0x7fea04086270 2021-02-10T11:44:05.007 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:05.002 7fe9fb7fe700 1 -- 172.21.2.112:0/3341883321 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe9fc000d10 con 0x7fea04086270 2021-02-10T11:44:05.007 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:05.002 7fe9fb7fe700 1 --2- 172.21.2.112:0/3341883321 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe9e80186c0 0x7fe9e801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:05.007 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:05.002 7fe9fb7fe700 1 -- 172.21.2.112:0/3341883321 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fe9fc032940 con 0x7fea04086270 2021-02-10T11:44:05.007 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:05.003 7fea0285e700 1 --2- 172.21.2.112:0/3341883321 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe9e80186c0 0x7fe9e801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:05.007 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:05.004 7fea0285e700 1 --2- 172.21.2.112:0/3341883321 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe9e80186c0 0x7fe9e801abc0 crc :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:05.035 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:05.034 7fea08ab4700 1 -- 172.21.2.112:0/3341883321 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea04082a70 con 0x7fea04086270 2021-02-10T11:44:05.038 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:05.037 7fe9fb7fe700 1 -- 172.21.2.112:0/3341883321 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe9fc00be30 con 0x7fea04086270 2021-02-10T11:44:05.165 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:05.165 7fea08ab4700 1 -- 172.21.2.112:0/3341883321 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7fea04082a70 con 0x7fea04086270 2021-02-10T11:44:05.166 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:05.165 7fe9fb7fe700 1 -- 172.21.2.112:0/3341883321 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v143) v1 ==== 104+0+10 (secure 0 0 0) 0x7fe9fc003180 con 0x7fea04086270 2021-02-10T11:44:05.166 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:44:05.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:05.167 7fea08ab4700 1 -- 172.21.2.112:0/3341883321 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe9e80186c0 msgr2=0x7fe9e801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:05.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:05.167 7fea08ab4700 1 --2- 172.21.2.112:0/3341883321 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe9e80186c0 0x7fe9e801abc0 crc :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:05.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:05.167 7fea08ab4700 1 -- 172.21.2.112:0/3341883321 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fea04086270 msgr2=0x7fea04086670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:05.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:05.167 7fea08ab4700 1 --2- 172.21.2.112:0/3341883321 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fea04086270 0x7fea04086670 secure :-1 s=READY pgs=2567 cs=0 l=1 rev1=1 rx=0x7fe9fc009960 tx=0x7fe9fc00bcf0).stop 2021-02-10T11:44:05.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:05.167 7fea08ab4700 1 -- 172.21.2.112:0/3341883321 shutdown_connections 2021-02-10T11:44:05.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:05.167 7fea08ab4700 1 --2- 172.21.2.112:0/3341883321 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe9e80186c0 0x7fe9e801abc0 unknown :-1 s=CLOSED pgs=1177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:05.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:05.167 7fea08ab4700 1 --2- 172.21.2.112:0/3341883321 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fea040a0290 0x7fea040852d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:05.169 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:05.167 7fea08ab4700 1 --2- 172.21.2.112:0/3341883321 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fea04086270 0x7fea04086670 unknown :-1 s=CLOSED pgs=2567 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:05.169 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:05.167 7fea08ab4700 1 --2- 172.21.2.112:0/3341883321 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fea040c3820 0x7fea04084cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:05.169 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:05.167 7fea08ab4700 1 -- 172.21.2.112:0/3341883321 >> 172.21.2.112:0/3341883321 conn(0x7fea040872c0 msgr2=0x7fea04095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:05.169 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:05.167 7fea08ab4700 1 -- 172.21.2.112:0/3341883321 shutdown_connections 2021-02-10T11:44:05.169 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:05.167 7fea08ab4700 1 -- 172.21.2.112:0/3341883321 wait complete. 2021-02-10T11:44:05.176 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:44:05.608 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:05.590 7f4010f27700 1 Processor -- start 2021-02-10T11:44:05.609 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:05.590 7f4010f27700 1 -- start start 2021-02-10T11:44:05.609 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:05.590 7f4010f27700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f400c0d1860 0x7f400c1b04b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:05.609 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:05.590 7f4010f27700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f400c0cff60 0x7f400c109120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:05.609 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:05.590 7f4010f27700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f400c109a50 con 0x7f400c1b0ad0 2021-02-10T11:44:05.609 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:05.591 7f4010f27700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f400c109740 con 0x7f400c0cff60 2021-02-10T11:44:05.609 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:05.591 7f4010f27700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f400c088070 con 0x7f400c0d1860 2021-02-10T11:44:05.610 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:05.591 7f400a312700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7f400c1b0ad0 0x7f400c108930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:50674/0 (socket says 172.21.2.112:50674) 2021-02-10T11:44:05.610 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:05.591 7f400a312700 1 -- 172.21.2.112:0/1782393526 learned_addr learned my addr 172.21.2.112:0/1782393526 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:44:05.610 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:05.591 7f400b314700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f400c0cff60 0x7f400c109120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:05.610 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:05.591 7f400ab13700 1 --2- 172.21.2.112:0/1782393526 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f400c0d1860 0x7f400c1b04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:05.610 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:05.592 7f400b314700 1 -- 172.21.2.112:0/1782393526 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f400c0d1860 msgr2=0x7f400c1b04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:05.610 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:05.592 7f400b314700 1 --2- 172.21.2.112:0/1782393526 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f400c0d1860 0x7f400c1b04b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:05.610 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:05.592 7f400b314700 1 -- 172.21.2.112:0/1782393526 >> v1:172.21.2.120:6789/0 conn(0x7f400c1b0ad0 legacy=0x7f400c108930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:05.610 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:05.592 7f400b314700 1 -- 172.21.2.112:0/1782393526 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f400c088260 con 0x7f400c0cff60 2021-02-10T11:44:05.611 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:05.592 7f4009b11700 1 -- 172.21.2.112:0/1782393526 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1876663145 0 0) 0x7f3ff40009b0 con 0x7f400c1b0ad0 2021-02-10T11:44:05.611 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:05.592 7f400ab13700 1 --2- 172.21.2.112:0/1782393526 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f400c0d1860 0x7f400c1b04b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:44:05.611 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:05.592 7f400b314700 1 --2- 172.21.2.112:0/1782393526 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f400c0cff60 0x7f400c109120 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 rx=0x7f4004006e50 tx=0x7f4004009e70).ready entity=mon.1 client_cookie=5fd98d1c0819c8a8 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:05.611 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:05.593 7f4009b11700 1 -- 172.21.2.112:0/1782393526 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f40040066f0 con 0x7f400c0cff60 2021-02-10T11:44:05.611 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:05.593 7f4009b11700 1 -- 172.21.2.112:0/1782393526 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4004003030 con 0x7f400c0cff60 2021-02-10T11:44:05.611 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:05.593 7f4009b11700 1 -- 172.21.2.112:0/1782393526 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f400400b790 con 0x7f400c0cff60 2021-02-10T11:44:05.611 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:05.593 7f4010f27700 1 -- 172.21.2.112:0/1782393526 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f400c0cff60 msgr2=0x7f400c109120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:05.611 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:05.593 7f4010f27700 1 --2- 172.21.2.112:0/1782393526 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f400c0cff60 0x7f400c109120 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 rx=0x7f4004006e50 tx=0x7f4004009e70).stop 2021-02-10T11:44:05.611 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:05.593 7f4010f27700 1 -- 172.21.2.112:0/1782393526 shutdown_connections 2021-02-10T11:44:05.611 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:05.593 7f4010f27700 1 --2- 172.21.2.112:0/1782393526 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f400c0cff60 0x7f400c109120 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:05.611 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:05.593 7f4010f27700 1 --2- 172.21.2.112:0/1782393526 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f400c0d1860 0x7f400c1b04b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:05.611 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:05.593 7f4010f27700 1 -- 172.21.2.112:0/1782393526 >> 172.21.2.112:0/1782393526 conn(0x7f400c0ce410 msgr2=0x7f400c0d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:05.611 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:05.594 7f4010f27700 1 -- 172.21.2.112:0/1782393526 shutdown_connections 2021-02-10T11:44:05.611 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:05.594 7f4010f27700 1 -- 172.21.2.112:0/1782393526 wait complete. 2021-02-10T11:44:05.612 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:05.608 7f4010f27700 1 Processor -- start 2021-02-10T11:44:05.612 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:05.608 7f4010f27700 1 -- start start 2021-02-10T11:44:05.613 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:05.608 7f4010f27700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f400c0b1b40 0x7f400c0b1f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:05.613 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:05.608 7f4010f27700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f400c0b2560 0x7f400c0b7f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:05.613 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:05.608 7f4010f27700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f400c0b8520 0x7f400c0b8920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:05.613 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:05.608 7f4010f27700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f400c192d00 con 0x7f400c0b1b40 2021-02-10T11:44:05.613 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:05.608 7f4010f27700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f400c0b2960 con 0x7f400c0b2560 2021-02-10T11:44:05.613 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:05.608 7f4010f27700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f400c0a23c0 con 0x7f400c0b8520 2021-02-10T11:44:05.613 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:05.608 7f400ab13700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f400c0b1b40 0x7f400c0b1f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:05.613 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:05.608 7f400ab13700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f400c0b1b40 0x7f400c0b1f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:57660/0 (socket says 172.21.2.112:57660) 2021-02-10T11:44:05.614 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:05.608 7f400ab13700 1 -- 172.21.2.112:0/4156493497 learned_addr learned my addr 172.21.2.112:0/4156493497 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:44:05.614 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:05.609 7f400ab13700 1 -- 172.21.2.112:0/4156493497 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f400c0b8520 msgr2=0x7f400c0b8920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:05.614 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:05.609 7f400ab13700 1 --2- 172.21.2.112:0/4156493497 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f400c0b8520 0x7f400c0b8920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:05.614 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:05.609 7f400ab13700 1 -- 172.21.2.112:0/4156493497 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f400c0b2560 msgr2=0x7f400c0b7f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:05.614 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:05.609 7f400ab13700 1 --2- 172.21.2.112:0/4156493497 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f400c0b2560 0x7f400c0b7f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:05.614 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:05.609 7f400ab13700 1 -- 172.21.2.112:0/4156493497 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f400c193200 con 0x7f400c0b1b40 2021-02-10T11:44:05.614 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:05.609 7f400ab13700 1 --2- 172.21.2.112:0/4156493497 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f400c0b1b40 0x7f400c0b1f40 secure :-1 s=READY pgs=2568 cs=0 l=1 rev1=1 rx=0x7f3ffc006f40 tx=0x7f3ffc00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:05.614 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:05.609 7f40037fe700 1 -- 172.21.2.112:0/4156493497 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3ffc00bd80 con 0x7f400c0b1b40 2021-02-10T11:44:05.614 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:05.609 7f40037fe700 1 -- 172.21.2.112:0/4156493497 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ffc000c90 con 0x7f400c0b1b40 2021-02-10T11:44:05.614 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:05.609 7f4010f27700 1 -- 172.21.2.112:0/4156493497 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f400c193200 con 0x7f400c0b1b40 2021-02-10T11:44:05.614 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:05.609 7f40037fe700 1 -- 172.21.2.112:0/4156493497 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3ffc002220 con 0x7f400c0b1b40 2021-02-10T11:44:05.614 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:05.609 7f4010f27700 1 -- 172.21.2.112:0/4156493497 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f400c193200 con 0x7f400c0b1b40 2021-02-10T11:44:05.614 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:05.610 7f40037fe700 1 -- 172.21.2.112:0/4156493497 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3ffc006b00 con 0x7f400c0b1b40 2021-02-10T11:44:05.615 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:05.610 7f40037fe700 1 --2- 172.21.2.112:0/4156493497 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3ff00186c0 0x7f3ff001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:05.615 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:05.610 7f40037fe700 1 -- 172.21.2.112:0/4156493497 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f3ffc0327e0 con 0x7f400c0b1b40 2021-02-10T11:44:05.615 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:05.611 7f400a312700 1 --2- 172.21.2.112:0/4156493497 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3ff00186c0 0x7f3ff001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:05.615 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:05.611 7f400a312700 1 --2- 172.21.2.112:0/4156493497 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3ff00186c0 0x7f3ff001abc0 crc :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:05.641 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:05.640 7f4010f27700 1 -- 172.21.2.112:0/4156493497 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f400c082920 con 0x7f400c0b1b40 2021-02-10T11:44:05.643 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:05.643 7f40037fe700 1 -- 172.21.2.112:0/4156493497 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3ffc000c90 con 0x7f400c0b1b40 2021-02-10T11:44:05.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:05.770 7f4010f27700 1 -- 172.21.2.112:0/4156493497 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f400c109a50 con 0x7f400c0b1b40 2021-02-10T11:44:05.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:05.770 7f40037fe700 1 -- 172.21.2.112:0/4156493497 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v143) v1 ==== 100+0+10 (secure 0 0 0) 0x7f3ffc00bd80 con 0x7f400c0b1b40 2021-02-10T11:44:05.771 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:44:05.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:05.772 7f4010f27700 1 -- 172.21.2.112:0/4156493497 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3ff00186c0 msgr2=0x7f3ff001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:05.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:05.772 7f4010f27700 1 --2- 172.21.2.112:0/4156493497 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3ff00186c0 0x7f3ff001abc0 crc :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:05.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:05.772 7f4010f27700 1 -- 172.21.2.112:0/4156493497 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f400c0b1b40 msgr2=0x7f400c0b1f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:05.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:05.772 7f4010f27700 1 --2- 172.21.2.112:0/4156493497 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f400c0b1b40 0x7f400c0b1f40 secure :-1 s=READY pgs=2568 cs=0 l=1 rev1=1 rx=0x7f3ffc006f40 tx=0x7f3ffc00b990).stop 2021-02-10T11:44:05.774 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:05.772 7f4010f27700 1 -- 172.21.2.112:0/4156493497 shutdown_connections 2021-02-10T11:44:05.774 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:05.772 7f4010f27700 1 --2- 172.21.2.112:0/4156493497 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3ff00186c0 0x7f3ff001abc0 unknown :-1 s=CLOSED pgs=1178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:05.774 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:05.772 7f4010f27700 1 --2- 172.21.2.112:0/4156493497 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f400c0b2560 0x7f400c0b7f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:05.774 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:05.772 7f4010f27700 1 --2- 172.21.2.112:0/4156493497 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f400c0b8520 0x7f400c0b8920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:05.774 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:05.772 7f4010f27700 1 --2- 172.21.2.112:0/4156493497 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f400c0b1b40 0x7f400c0b1f40 unknown :-1 s=CLOSED pgs=2568 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:05.774 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:05.772 7f4010f27700 1 -- 172.21.2.112:0/4156493497 >> 172.21.2.112:0/4156493497 conn(0x7f400c0a28e0 msgr2=0x7f400c094c40 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:05.774 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:05.772 7f4010f27700 1 -- 172.21.2.112:0/4156493497 shutdown_connections 2021-02-10T11:44:05.774 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:05.773 7f4010f27700 1 -- 172.21.2.112:0/4156493497 wait complete. 2021-02-10T11:44:05.782 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:44:05.782 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:44:06.209 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:06.192 7ff93d57a700 1 Processor -- start 2021-02-10T11:44:06.210 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:06.192 7ff93d57a700 1 -- start start 2021-02-10T11:44:06.210 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:06.193 7ff93d57a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff9380d1870 0x7ff9381b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:06.210 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:06.193 7ff93d57a700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7ff9380f0650 0x7ff9380cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:06.210 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:06.193 7ff93d57a700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9380f0b10 con 0x7ff9381b0a10 2021-02-10T11:44:06.211 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:06.193 7ff93d57a700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9380fe230 con 0x7ff9380f0650 2021-02-10T11:44:06.211 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:06.193 7ff93d57a700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9380e5ed0 con 0x7ff9380d1870 2021-02-10T11:44:06.211 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:06.193 7ff937ac0700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7ff9380f0650 0x7ff9380cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:06.211 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:06.193 7ff937ac0700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7ff9380f0650 0x7ff9380cfb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:57664/0 (socket says 172.21.2.112:57664) 2021-02-10T11:44:06.211 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:06.193 7ff937ac0700 1 -- 172.21.2.112:0/4150682758 learned_addr learned my addr 172.21.2.112:0/4150682758 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:44:06.211 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:06.193 7ff937ac0700 1 -- 172.21.2.112:0/4150682758 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff9380d1870 msgr2=0x7ff9381b03f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:44:06.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:06.193 7ff937ac0700 1 --2- 172.21.2.112:0/4150682758 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff9380d1870 0x7ff9381b03f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:06.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:06.193 7ff937ac0700 1 -- 172.21.2.112:0/4150682758 >> v1:172.21.2.112:6789/0 conn(0x7ff9381b0a10 legacy=0x7ff9380cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:06.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:06.193 7ff937ac0700 1 -- 172.21.2.112:0/4150682758 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9380f20a0 con 0x7ff9380f0650 2021-02-10T11:44:06.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:06.193 7ff937ac0700 1 --2- 172.21.2.112:0/4150682758 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff9380f0650 0x7ff9380cfb20 secure :-1 s=READY pgs=2570 cs=0 l=1 rev1=1 rx=0x7ff930006f00 tx=0x7ff930009f10).ready entity=mon.0 client_cookie=b49bc4a66aaad214 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:06.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:06.193 7ff9362bd700 1 -- 172.21.2.112:0/4150682758 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff93000b3b0 con 0x7ff9380f0650 2021-02-10T11:44:06.213 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:06.194 7ff9362bd700 1 -- 172.21.2.112:0/4150682758 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff93000b730 con 0x7ff9380f0650 2021-02-10T11:44:06.213 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:06.194 7ff9362bd700 1 -- 172.21.2.112:0/4150682758 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff93000bc10 con 0x7ff9380f0650 2021-02-10T11:44:06.213 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:06.194 7ff93d57a700 1 -- 172.21.2.112:0/4150682758 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff9380f0650 msgr2=0x7ff9380cfb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:06.213 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:06.194 7ff93d57a700 1 --2- 172.21.2.112:0/4150682758 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff9380f0650 0x7ff9380cfb20 secure :-1 s=READY pgs=2570 cs=0 l=1 rev1=1 rx=0x7ff930006f00 tx=0x7ff930009f10).stop 2021-02-10T11:44:06.213 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:06.194 7ff93d57a700 1 -- 172.21.2.112:0/4150682758 shutdown_connections 2021-02-10T11:44:06.213 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:06.194 7ff93d57a700 1 --2- 172.21.2.112:0/4150682758 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff9380f0650 0x7ff9380cfb20 unknown :-1 s=CLOSED pgs=2570 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:06.213 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:06.194 7ff93d57a700 1 --2- 172.21.2.112:0/4150682758 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff9380d1870 0x7ff9381b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:06.214 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:06.194 7ff93d57a700 1 -- 172.21.2.112:0/4150682758 >> 172.21.2.112:0/4150682758 conn(0x7ff9380ce450 msgr2=0x7ff9380d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:06.214 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:06.194 7ff93d57a700 1 -- 172.21.2.112:0/4150682758 shutdown_connections 2021-02-10T11:44:06.214 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:06.194 7ff93d57a700 1 -- 172.21.2.112:0/4150682758 wait complete. 2021-02-10T11:44:06.214 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:06.209 7ff93d57a700 1 Processor -- start 2021-02-10T11:44:06.214 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:06.209 7ff93d57a700 1 -- start start 2021-02-10T11:44:06.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:06.209 7ff93d57a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff9380b0ad0 0x7ff938091f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:06.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:06.209 7ff93d57a700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff9381a8850 0x7ff9381a8c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:06.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:06.209 7ff93d57a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff9381a6fa0 0x7ff9381a73a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:06.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:06.209 7ff93d57a700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9381abfe0 con 0x7ff9381a6fa0 2021-02-10T11:44:06.218 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:06.209 7ff93d57a700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9381a9c40 con 0x7ff9381a8850 2021-02-10T11:44:06.218 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:06.209 7ff93d57a700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9381a9730 con 0x7ff9380b0ad0 2021-02-10T11:44:06.218 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:06.209 7ff937ac0700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff9381a6fa0 0x7ff9381a73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:06.218 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:06.209 7ff937ac0700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff9381a6fa0 0x7ff9381a73a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:57674/0 (socket says 172.21.2.112:57674) 2021-02-10T11:44:06.218 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:06.209 7ff937ac0700 1 -- 172.21.2.112:0/1284783590 learned_addr learned my addr 172.21.2.112:0/1284783590 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:44:06.218 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:06.210 7ff937ac0700 1 -- 172.21.2.112:0/1284783590 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff9380b0ad0 msgr2=0x7ff938091f30 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:44:06.219 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:06.210 7ff937ac0700 1 --2- 172.21.2.112:0/1284783590 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff9380b0ad0 0x7ff938091f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:06.219 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:06.210 7ff937ac0700 1 -- 172.21.2.112:0/1284783590 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff9381a8850 msgr2=0x7ff9381a8c50 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:44:06.219 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:06.210 7ff937ac0700 1 --2- 172.21.2.112:0/1284783590 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff9381a8850 0x7ff9381a8c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:06.219 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:06.210 7ff937ac0700 1 -- 172.21.2.112:0/1284783590 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9381a9270 con 0x7ff9381a6fa0 2021-02-10T11:44:06.219 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:06.210 7ff937ac0700 1 --2- 172.21.2.112:0/1284783590 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff9381a6fa0 0x7ff9381a73a0 secure :-1 s=READY pgs=2571 cs=0 l=1 rev1=1 rx=0x7ff93000b9b0 tx=0x7ff93000bd50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:06.219 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:06.210 7ff92ffff700 1 -- 172.21.2.112:0/1284783590 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff930005820 con 0x7ff9381a6fa0 2021-02-10T11:44:06.220 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:06.210 7ff92ffff700 1 -- 172.21.2.112:0/1284783590 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff930005a60 con 0x7ff9381a6fa0 2021-02-10T11:44:06.220 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:06.210 7ff92ffff700 1 -- 172.21.2.112:0/1284783590 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff930006030 con 0x7ff9381a6fa0 2021-02-10T11:44:06.220 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:06.210 7ff93d57a700 1 -- 172.21.2.112:0/1284783590 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9381a9270 con 0x7ff9381a6fa0 2021-02-10T11:44:06.220 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:06.210 7ff93d57a700 1 -- 172.21.2.112:0/1284783590 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9381a9270 con 0x7ff9381a6fa0 2021-02-10T11:44:06.220 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:06.211 7ff92ffff700 1 -- 172.21.2.112:0/1284783590 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff930009450 con 0x7ff9381a6fa0 2021-02-10T11:44:06.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:06.211 7ff92ffff700 1 --2- 172.21.2.112:0/1284783590 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff91c0186c0 0x7ff91c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:06.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:06.211 7ff92ffff700 1 -- 172.21.2.112:0/1284783590 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5573+0+0 (secure 0 0 0) 0x7ff930032b70 con 0x7ff9381a6fa0 2021-02-10T11:44:06.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:06.211 7ff9372bf700 1 --2- 172.21.2.112:0/1284783590 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff91c0186c0 0x7ff91c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:06.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:06.212 7ff9372bf700 1 --2- 172.21.2.112:0/1284783590 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff91c0186c0 0x7ff91c01abc0 crc :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:06.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:06.243 7ff93d57a700 1 -- 172.21.2.112:0/1284783590 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9381abfe0 con 0x7ff9381a6fa0 2021-02-10T11:44:06.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:06.245 7ff92ffff700 1 -- 172.21.2.112:0/1284783590 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff930000f90 con 0x7ff9381a6fa0 2021-02-10T11:44:06.374 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:06.373 7ff93d57a700 1 -- 172.21.2.112:0/1284783590 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff9381abfe0 con 0x7ff9381a6fa0 2021-02-10T11:44:06.374 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:06.374 7ff92ffff700 1 -- 172.21.2.112:0/1284783590 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) v1 ==== 95+0+2108 (secure 0 0 0) 0x7ff930005a60 con 0x7ff9381a6fa0 2021-02-10T11:44:06.375 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 122 2021-02-10T11:44:06.375 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:44:06.375 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":122,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11670,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/1286703435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":1286703435},{"type":"v1","addr":"172.21.2.112:6833","nonce":1286703435}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":119}],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:43:56.409867","modified":"2021-02-10 11:43:58.413644","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8578},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8578":{"gid":8578,"name":"b","rank":0,"incarnation":120,"state":"up:active","state_seq":3,"addr":"172.21.2.120:6835/1504781062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":1504781062},{"type":"v1","addr":"172.21.2.120:6835","nonce":1504781062}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":15}]} 2021-02-10T11:44:06.376 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:06.375 7ff93d57a700 1 -- 172.21.2.112:0/1284783590 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff91c0186c0 msgr2=0x7ff91c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:06.376 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:06.375 7ff93d57a700 1 --2- 172.21.2.112:0/1284783590 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff91c0186c0 0x7ff91c01abc0 crc :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:06.376 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:06.375 7ff93d57a700 1 -- 172.21.2.112:0/1284783590 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff9381a6fa0 msgr2=0x7ff9381a73a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:06.376 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:06.375 7ff93d57a700 1 --2- 172.21.2.112:0/1284783590 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff9381a6fa0 0x7ff9381a73a0 secure :-1 s=READY pgs=2571 cs=0 l=1 rev1=1 rx=0x7ff93000b9b0 tx=0x7ff93000bd50).stop 2021-02-10T11:44:06.376 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:06.375 7ff93d57a700 1 -- 172.21.2.112:0/1284783590 shutdown_connections 2021-02-10T11:44:06.376 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:06.375 7ff93d57a700 1 --2- 172.21.2.112:0/1284783590 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff91c0186c0 0x7ff91c01abc0 unknown :-1 s=CLOSED pgs=1179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:06.376 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:06.375 7ff93d57a700 1 --2- 172.21.2.112:0/1284783590 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff9381a8850 0x7ff9381a8c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:06.377 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:06.375 7ff93d57a700 1 --2- 172.21.2.112:0/1284783590 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff9381a6fa0 0x7ff9381a73a0 unknown :-1 s=CLOSED pgs=2571 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:06.377 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:06.375 7ff93d57a700 1 --2- 172.21.2.112:0/1284783590 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff9380b0ad0 0x7ff938091f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:06.377 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:06.375 7ff93d57a700 1 -- 172.21.2.112:0/1284783590 >> 172.21.2.112:0/1284783590 conn(0x7ff93819f8f0 msgr2=0x7ff9381a5f80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:06.377 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:06.376 7ff93d57a700 1 -- 172.21.2.112:0/1284783590 shutdown_connections 2021-02-10T11:44:06.377 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:06.376 7ff93d57a700 1 -- 172.21.2.112:0/1284783590 wait complete. 2021-02-10T11:44:06.587 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 122, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:43:56.409867', 'modified': '2021-02-10 11:43:58.413644', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8578}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8578': {'gid': 8578, 'name': 'b', 'rank': 0, 'incarnation': 120, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.120:6835/1504781062', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.120:6834', 'nonce': 1504781062}, {'type': 'v1', 'addr': '172.21.2.120:6835', 'nonce': 1504781062}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [31], 'metadata_pool': 30, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:44:06.587 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:44:06.587 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:44:06.985 INFO:teuthology.orchestra.run.gibba020.stdout:{ 2021-02-10T11:44:06.986 INFO:teuthology.orchestra.run.gibba020.stdout: "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", 2021-02-10T11:44:06.986 INFO:teuthology.orchestra.run.gibba020.stdout: "whoami": 0, 2021-02-10T11:44:06.986 INFO:teuthology.orchestra.run.gibba020.stdout: "id": 8578, 2021-02-10T11:44:06.986 INFO:teuthology.orchestra.run.gibba020.stdout: "want_state": "up:active", 2021-02-10T11:44:06.986 INFO:teuthology.orchestra.run.gibba020.stdout: "state": "up:active", 2021-02-10T11:44:06.986 INFO:teuthology.orchestra.run.gibba020.stdout: "rank_uptime": 10.573555300000001, 2021-02-10T11:44:06.986 INFO:teuthology.orchestra.run.gibba020.stdout: "mdsmap_epoch": 122, 2021-02-10T11:44:06.986 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch": 143, 2021-02-10T11:44:06.986 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch_barrier": 143, 2021-02-10T11:44:06.986 INFO:teuthology.orchestra.run.gibba020.stdout: "uptime": 16.219058722 2021-02-10T11:44:06.986 INFO:teuthology.orchestra.run.gibba020.stdout:} 2021-02-10T11:44:06.990 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", "whoami": 0, "id": 8578, "want_state": "up:active", "state": "up:active", "rank_uptime": 10.573555300000001, "mdsmap_epoch": 122, "osdmap_epoch": 143, "osdmap_epoch_barrier": 143, "uptime": 16.219058722 } 2021-02-10T11:44:06.990 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:44:06.990 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2021-02-10T11:44:06.990 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@gibba012.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2021-02-10T11:44:06.990 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:44:06.990 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:44:07.325 DEBUG:teuthology.orchestra.run.gibba012:> sudo modprobe fuse 2021-02-10T11:44:07.671 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:44:07.671 DEBUG:teuthology.orchestra.run.gibba012:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:44:08.013 INFO:teuthology.orchestra.run.gibba012.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:44:08.013 INFO:teuthology.orchestra.run.gibba012.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:44:08.014 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:44:08.014 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:44:08.015 DEBUG:teuthology.orchestra.run.gibba012:> ls /sys/fs/fuse/connections 2021-02-10T11:44:08.350 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2021-02-10T11:44:08.350 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:44:08.352 DEBUG:teuthology.orchestra.run.gibba012:> sudo modprobe fuse 2021-02-10T11:44:08.695 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:44:08.695 DEBUG:teuthology.orchestra.run.gibba012:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:44:08.762 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba012.stderr:ceph-fuse[172931]: starting ceph client 2021-02-10T11:44:08.762 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba012.stderr:2021-02-10 11:44:08.761 7f9ea9964f80 -1 init, newargv = 0x55865ff671f0 newargc=9 2021-02-10T11:44:08.767 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba012.stderr:ceph-fuse[172931]: starting fuse 2021-02-10T11:44:09.037 INFO:teuthology.orchestra.run.gibba012.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:44:09.037 INFO:teuthology.orchestra.run.gibba012.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:44:09.038 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:44:09.038 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:44:09.039 DEBUG:teuthology.orchestra.run.gibba012:> ls /sys/fs/fuse/connections 2021-02-10T11:44:09.374 INFO:teuthology.orchestra.run.gibba012.stdout:42 2021-02-10T11:44:09.374 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2021-02-10T11:44:09.374 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:44:09.374 DEBUG:teuthology.orchestra.run.gibba012:> sudo python3 -c ' 2021-02-10T11:44:09.374 DEBUG:teuthology.orchestra.run.gibba012:> import glob 2021-02-10T11:44:09.375 DEBUG:teuthology.orchestra.run.gibba012:> import re 2021-02-10T11:44:09.375 DEBUG:teuthology.orchestra.run.gibba012:> import os 2021-02-10T11:44:09.375 DEBUG:teuthology.orchestra.run.gibba012:> import subprocess 2021-02-10T11:44:09.375 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:44:09.375 DEBUG:teuthology.orchestra.run.gibba012:> def find_socket(client_name): 2021-02-10T11:44:09.375 DEBUG:teuthology.orchestra.run.gibba012:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2021-02-10T11:44:09.375 DEBUG:teuthology.orchestra.run.gibba012:> files = glob.glob(asok_path) 2021-02-10T11:44:09.375 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:44:09.375 DEBUG:teuthology.orchestra.run.gibba012:> # Given a non-glob path, it better be there 2021-02-10T11:44:09.375 DEBUG:teuthology.orchestra.run.gibba012:> if "*" not in asok_path: 2021-02-10T11:44:09.375 DEBUG:teuthology.orchestra.run.gibba012:> assert(len(files) == 1) 2021-02-10T11:44:09.375 DEBUG:teuthology.orchestra.run.gibba012:> return files[0] 2021-02-10T11:44:09.375 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:44:09.375 DEBUG:teuthology.orchestra.run.gibba012:> for f in files: 2021-02-10T11:44:09.376 DEBUG:teuthology.orchestra.run.gibba012:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:44:09.376 DEBUG:teuthology.orchestra.run.gibba012:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:44:09.376 DEBUG:teuthology.orchestra.run.gibba012:> return f 2021-02-10T11:44:09.376 DEBUG:teuthology.orchestra.run.gibba012:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:44:09.376 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:44:09.376 DEBUG:teuthology.orchestra.run.gibba012:> print(find_socket("client.0")) 2021-02-10T11:44:09.376 DEBUG:teuthology.orchestra.run.gibba012:> ' 2021-02-10T11:44:09.738 INFO:teuthology.orchestra.run.gibba012.stdout:/var/run/ceph/ceph-client.0.172931.asok 2021-02-10T11:44:09.741 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.172931.asok 2021-02-10T11:44:09.741 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:44:09.741 DEBUG:teuthology.orchestra.run.gibba012:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.172931.asok status 2021-02-10T11:44:10.130 INFO:teuthology.orchestra.run.gibba012.stdout:{ 2021-02-10T11:44:10.131 INFO:teuthology.orchestra.run.gibba012.stdout: "metadata": { 2021-02-10T11:44:10.131 INFO:teuthology.orchestra.run.gibba012.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:44:10.131 INFO:teuthology.orchestra.run.gibba012.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:44:10.131 INFO:teuthology.orchestra.run.gibba012.stdout: "entity_id": "0", 2021-02-10T11:44:10.131 INFO:teuthology.orchestra.run.gibba012.stdout: "hostname": "gibba012", 2021-02-10T11:44:10.131 INFO:teuthology.orchestra.run.gibba012.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2021-02-10T11:44:10.131 INFO:teuthology.orchestra.run.gibba012.stdout: "pid": "172931", 2021-02-10T11:44:10.131 INFO:teuthology.orchestra.run.gibba012.stdout: "root": "/" 2021-02-10T11:44:10.131 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:44:10.131 INFO:teuthology.orchestra.run.gibba012.stdout: "dentry_count": 0, 2021-02-10T11:44:10.131 INFO:teuthology.orchestra.run.gibba012.stdout: "dentry_pinned_count": 0, 2021-02-10T11:44:10.131 INFO:teuthology.orchestra.run.gibba012.stdout: "id": 11838, 2021-02-10T11:44:10.131 INFO:teuthology.orchestra.run.gibba012.stdout: "inst": { 2021-02-10T11:44:10.131 INFO:teuthology.orchestra.run.gibba012.stdout: "name": { 2021-02-10T11:44:10.132 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "client", 2021-02-10T11:44:10.132 INFO:teuthology.orchestra.run.gibba012.stdout: "num": 11838 2021-02-10T11:44:10.132 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:44:10.132 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": { 2021-02-10T11:44:10.132 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:44:10.132 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:0", 2021-02-10T11:44:10.132 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 1138679101 2021-02-10T11:44:10.132 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:44:10.132 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:44:10.132 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": { 2021-02-10T11:44:10.132 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:44:10.132 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:0", 2021-02-10T11:44:10.132 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 1138679101 2021-02-10T11:44:10.133 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:44:10.133 INFO:teuthology.orchestra.run.gibba012.stdout: "inst_str": "client.11838 172.21.2.112:0/1138679101", 2021-02-10T11:44:10.133 INFO:teuthology.orchestra.run.gibba012.stdout: "addr_str": "172.21.2.112:0/1138679101", 2021-02-10T11:44:10.133 INFO:teuthology.orchestra.run.gibba012.stdout: "inode_count": 1, 2021-02-10T11:44:10.133 INFO:teuthology.orchestra.run.gibba012.stdout: "mds_epoch": 122, 2021-02-10T11:44:10.133 INFO:teuthology.orchestra.run.gibba012.stdout: "osd_epoch": 143, 2021-02-10T11:44:10.133 INFO:teuthology.orchestra.run.gibba012.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:44:10.133 INFO:teuthology.orchestra.run.gibba012.stdout: "blacklisted": false 2021-02-10T11:44:10.133 INFO:teuthology.orchestra.run.gibba012.stdout:} 2021-02-10T11:44:10.135 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:44:10.135 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:44:10.135 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:44:10.471 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:44:10.472 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:44:10.472 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:44:10.472 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:44:10.819 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:44:11.245 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.228 7ffab0ef3700 1 Processor -- start 2021-02-10T11:44:11.245 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.228 7ffab0ef3700 1 -- start start 2021-02-10T11:44:11.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.229 7ffab0ef3700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7ffaac0d0340 0x7ffaac0ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:11.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.229 7ffab0ef3700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7ffaac0d1c70 0x7ffaac0efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:11.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.229 7ffab0ef3700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffaac0f0b10 con 0x7ffaac0d1870 2021-02-10T11:44:11.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.229 7ffab0ef3700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffaac0e5ed0 con 0x7ffaac0d0340 2021-02-10T11:44:11.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.229 7ffab0ef3700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffaac1b0be0 con 0x7ffaac0d1c70 2021-02-10T11:44:11.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.229 7ffaaa312700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7ffaac0d0340 0x7ffaac0ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:11.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.229 7ffaaa312700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7ffaac0d0340 0x7ffaac0ef640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:57694/0 (socket says 172.21.2.112:57694) 2021-02-10T11:44:11.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.229 7ffaaa312700 1 -- 172.21.2.112:0/4174482111 learned_addr learned my addr 172.21.2.112:0/4174482111 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:44:11.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.229 7ffaaa312700 1 -- 172.21.2.112:0/4174482111 >> v2:172.21.2.120:3300/0 conn(0x7ffaac0d1c70 msgr2=0x7ffaac0efc60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:44:11.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.229 7ffaaa312700 1 --2- 172.21.2.112:0/4174482111 >> v2:172.21.2.120:3300/0 conn(0x7ffaac0d1c70 0x7ffaac0efc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:11.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.229 7ffaaa312700 1 -- 172.21.2.112:0/4174482111 >> v1:172.21.2.120:6789/0 conn(0x7ffaac0d1870 legacy=0x7ffaac1b03f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:44:11.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.229 7ffaaa312700 1 -- 172.21.2.112:0/4174482111 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaac0f20a0 con 0x7ffaac0d0340 2021-02-10T11:44:11.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.229 7ffaaa312700 1 --2- 172.21.2.112:0/4174482111 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ffaac0d0340 0x7ffaac0ef640 secure :-1 s=READY pgs=2573 cs=0 l=1 rev1=1 rx=0x7ffaa0006700 tx=0x7ffaa0009b80).ready entity=mon.0 client_cookie=8f237f248a54cc6b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:11.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.230 7ffaa9b11700 1 -- 172.21.2.112:0/4174482111 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffaa000c0c0 con 0x7ffaac0d0340 2021-02-10T11:44:11.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.230 7ffaa9b11700 1 -- 172.21.2.112:0/4174482111 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffaa000c440 con 0x7ffaac0d0340 2021-02-10T11:44:11.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.230 7ffaa9b11700 1 -- 172.21.2.112:0/4174482111 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffaa000c960 con 0x7ffaac0d0340 2021-02-10T11:44:11.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.230 7ffab0ef3700 1 -- 172.21.2.112:0/4174482111 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ffaac0d0340 msgr2=0x7ffaac0ef640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:11.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.230 7ffab0ef3700 1 --2- 172.21.2.112:0/4174482111 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ffaac0d0340 0x7ffaac0ef640 secure :-1 s=READY pgs=2573 cs=0 l=1 rev1=1 rx=0x7ffaa0006700 tx=0x7ffaa0009b80).stop 2021-02-10T11:44:11.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.231 7ffab0ef3700 1 -- 172.21.2.112:0/4174482111 shutdown_connections 2021-02-10T11:44:11.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.231 7ffab0ef3700 1 --2- 172.21.2.112:0/4174482111 >> v2:172.21.2.120:3300/0 conn(0x7ffaac0d1c70 0x7ffaac0efc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:11.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.231 7ffab0ef3700 1 --2- 172.21.2.112:0/4174482111 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ffaac0d0340 0x7ffaac0ef640 unknown :-1 s=CLOSED pgs=2573 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:11.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.231 7ffab0ef3700 1 -- 172.21.2.112:0/4174482111 >> 172.21.2.112:0/4174482111 conn(0x7ffaac0ce450 msgr2=0x7ffaac0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:11.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.231 7ffab0ef3700 1 -- 172.21.2.112:0/4174482111 shutdown_connections 2021-02-10T11:44:11.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.231 7ffab0ef3700 1 -- 172.21.2.112:0/4174482111 wait complete. 2021-02-10T11:44:11.249 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.245 7ffab0ef3700 1 Processor -- start 2021-02-10T11:44:11.249 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.245 7ffab0ef3700 1 -- start start 2021-02-10T11:44:11.249 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.246 7ffab0ef3700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ffaac19a3d0 0x7ffaac1ab160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:11.249 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.246 7ffab0ef3700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ffaac1ab780 0x7ffaac0b3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:11.249 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.246 7ffab0ef3700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ffaac1abb80 0x7ffaac0b3c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:11.249 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.246 7ffab0ef3700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaac1ae200 con 0x7ffaac19a3d0 2021-02-10T11:44:11.249 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.246 7ffab0ef3700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaac092930 con 0x7ffaac1abb80 2021-02-10T11:44:11.249 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.246 7ffab0ef3700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffaac0b4290 con 0x7ffaac1ab780 2021-02-10T11:44:11.249 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.246 7ffaaab13700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ffaac19a3d0 0x7ffaac1ab160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:11.250 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.246 7ffaaab13700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ffaac19a3d0 0x7ffaac1ab160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:57698/0 (socket says 172.21.2.112:57698) 2021-02-10T11:44:11.250 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.246 7ffaaab13700 1 -- 172.21.2.112:0/815430789 learned_addr learned my addr 172.21.2.112:0/815430789 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:44:11.250 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.246 7ffaaa312700 1 --2- 172.21.2.112:0/815430789 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ffaac1ab780 0x7ffaac0b3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:11.250 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.246 7ffaab314700 1 --2- 172.21.2.112:0/815430789 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ffaac1abb80 0x7ffaac0b3c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:11.250 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.246 7ffaaab13700 1 -- 172.21.2.112:0/815430789 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ffaac1ab780 msgr2=0x7ffaac0b3650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:11.250 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.246 7ffaaab13700 1 --2- 172.21.2.112:0/815430789 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ffaac1ab780 0x7ffaac0b3650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:11.250 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.246 7ffaaab13700 1 -- 172.21.2.112:0/815430789 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ffaac1abb80 msgr2=0x7ffaac0b3c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:11.250 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.246 7ffaaab13700 1 --2- 172.21.2.112:0/815430789 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ffaac1abb80 0x7ffaac0b3c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:11.250 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.247 7ffaaab13700 1 -- 172.21.2.112:0/815430789 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaac0ee360 con 0x7ffaac19a3d0 2021-02-10T11:44:11.250 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.247 7ffaaab13700 1 --2- 172.21.2.112:0/815430789 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ffaac19a3d0 0x7ffaac1ab160 secure :-1 s=READY pgs=2574 cs=0 l=1 rev1=1 rx=0x7ffa9c00b5b0 tx=0x7ffa9c00b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:11.250 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.247 7ffa9b7fe700 1 -- 172.21.2.112:0/815430789 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffa9c00bd40 con 0x7ffaac19a3d0 2021-02-10T11:44:11.250 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.247 7ffa9b7fe700 1 -- 172.21.2.112:0/815430789 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa9c006240 con 0x7ffaac19a3d0 2021-02-10T11:44:11.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.247 7ffab0ef3700 1 -- 172.21.2.112:0/815430789 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffaac0ee360 con 0x7ffaac19a3d0 2021-02-10T11:44:11.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.247 7ffab0ef3700 1 -- 172.21.2.112:0/815430789 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffaac0ee8f0 con 0x7ffaac19a3d0 2021-02-10T11:44:11.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.247 7ffa9b7fe700 1 -- 172.21.2.112:0/815430789 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffa9c005720 con 0x7ffaac19a3d0 2021-02-10T11:44:11.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.248 7ffa9b7fe700 1 -- 172.21.2.112:0/815430789 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ffa9c0054a0 con 0x7ffaac19a3d0 2021-02-10T11:44:11.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.248 7ffa9b7fe700 1 --2- 172.21.2.112:0/815430789 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ffa90018730 0x7ffa9001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:11.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.248 7ffa9b7fe700 1 -- 172.21.2.112:0/815430789 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5573+0+0 (secure 0 0 0) 0x7ffa9c0327f0 con 0x7ffaac19a3d0 2021-02-10T11:44:11.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.248 7ffaaa312700 1 --2- 172.21.2.112:0/815430789 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ffa90018730 0x7ffa9001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:11.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.249 7ffaaa312700 1 --2- 172.21.2.112:0/815430789 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ffa90018730 0x7ffa9001ac10 crc :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:11.280 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.279 7ffab0ef3700 1 -- 172.21.2.112:0/815430789 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffaac0ce450 con 0x7ffaac19a3d0 2021-02-10T11:44:11.282 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.282 7ffa9b7fe700 1 -- 172.21.2.112:0/815430789 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ffa9c000c60 con 0x7ffaac19a3d0 2021-02-10T11:44:11.408 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.407 7ffab0ef3700 1 -- 172.21.2.112:0/815430789 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffaac0ce450 con 0x7ffaac19a3d0 2021-02-10T11:44:11.409 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.408 7ffa9b7fe700 1 -- 172.21.2.112:0/815430789 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v143) v1 ==== 74+0+14007 (secure 0 0 0) 0x7ffa9c0054a0 con 0x7ffaac19a3d0 2021-02-10T11:44:11.409 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:44:11.409 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":143,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:43:57.411337","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":31,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":30,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:43:54.385734","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"142","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":31,"pool_name":"cephfs_data","create_time":"2021-02-10 11:43:55.453740","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"143","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":139,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":139,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:44:11.410 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.410 7ffab0ef3700 1 -- 172.21.2.112:0/815430789 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ffa90018730 msgr2=0x7ffa9001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:11.411 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.410 7ffab0ef3700 1 --2- 172.21.2.112:0/815430789 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ffa90018730 0x7ffa9001ac10 crc :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:11.411 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.410 7ffab0ef3700 1 -- 172.21.2.112:0/815430789 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ffaac19a3d0 msgr2=0x7ffaac1ab160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:11.411 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.410 7ffab0ef3700 1 --2- 172.21.2.112:0/815430789 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ffaac19a3d0 0x7ffaac1ab160 secure :-1 s=READY pgs=2574 cs=0 l=1 rev1=1 rx=0x7ffa9c00b5b0 tx=0x7ffa9c00b950).stop 2021-02-10T11:44:11.411 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.410 7ffab0ef3700 1 -- 172.21.2.112:0/815430789 shutdown_connections 2021-02-10T11:44:11.411 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.410 7ffab0ef3700 1 --2- 172.21.2.112:0/815430789 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ffa90018730 0x7ffa9001ac10 unknown :-1 s=CLOSED pgs=1180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:11.411 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.410 7ffab0ef3700 1 --2- 172.21.2.112:0/815430789 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ffaac1abb80 0x7ffaac0b3c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:11.411 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.410 7ffab0ef3700 1 --2- 172.21.2.112:0/815430789 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ffaac1ab780 0x7ffaac0b3650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:11.411 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.410 7ffab0ef3700 1 --2- 172.21.2.112:0/815430789 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ffaac19a3d0 0x7ffaac1ab160 unknown :-1 s=CLOSED pgs=2574 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:11.411 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.410 7ffab0ef3700 1 -- 172.21.2.112:0/815430789 >> 172.21.2.112:0/815430789 conn(0x7ffaac093640 msgr2=0x7ffaac18eb00 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:11.411 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.410 7ffab0ef3700 1 -- 172.21.2.112:0/815430789 shutdown_connections 2021-02-10T11:44:11.411 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.410 7ffab0ef3700 1 -- 172.21.2.112:0/815430789 wait complete. 2021-02-10T11:44:11.419 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:13:57.508758', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 30, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:43:54.385734', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '142', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 31, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:43:55.453740', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '143', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:44:11.419 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:44:11.853 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.835 7f924172a700 1 Processor -- start 2021-02-10T11:44:11.853 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.835 7f924172a700 1 -- start start 2021-02-10T11:44:11.853 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.835 7f924172a700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f923c0d1870 0x7f923c1b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:11.853 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.835 7f924172a700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f923c0fe1c0 con 0x7f923c0cff70 2021-02-10T11:44:11.853 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.835 7f924172a700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f923c109a60 con 0x7f923c1b0ae0 2021-02-10T11:44:11.853 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.835 7f924172a700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f923c088070 con 0x7f923c0d1870 2021-02-10T11:44:11.854 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.835 7f923bac0700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f923c0cff70 0x7f923c109130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:57168/0 (socket says 172.21.2.112:57168) 2021-02-10T11:44:11.854 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.835 7f923bac0700 1 -- 172.21.2.112:0/2210888471 learned_addr learned my addr 172.21.2.112:0/2210888471 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:44:11.854 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.836 7f923a2bd700 1 -- 172.21.2.112:0/2210888471 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3185651279 0 0) 0x7f9234001180 con 0x7f923c0cff70 2021-02-10T11:44:11.854 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.836 7f923a2bd700 1 -- 172.21.2.112:0/2210888471 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9228002fe0 con 0x7f923c0cff70 2021-02-10T11:44:11.854 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.836 7f923b2bf700 1 --2- 172.21.2.112:0/2210888471 >> v2:172.21.2.120:3300/0 conn(0x7f923c0d1870 0x7f923c1b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:11.854 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.836 7f923a2bd700 1 -- 172.21.2.112:0/2210888471 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 261517785 0 0) 0x7f92340010e0 con 0x7f923c0cff70 2021-02-10T11:44:11.854 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.836 7f923a2bd700 1 -- 172.21.2.112:0/2210888471 >> v2:172.21.2.120:3300/0 conn(0x7f923c0d1870 msgr2=0x7f923c1b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:11.854 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.836 7f923a2bd700 1 --2- 172.21.2.112:0/2210888471 >> v2:172.21.2.120:3300/0 conn(0x7f923c0d1870 0x7f923c1b04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:11.854 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.836 7f923a2bd700 1 -- 172.21.2.112:0/2210888471 >> v1:172.21.2.120:6789/0 conn(0x7f923c1b0ae0 legacy=0x7f923c108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:11.854 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.836 7f923a2bd700 1 -- 172.21.2.112:0/2210888471 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f923c088260 con 0x7f923c0cff70 2021-02-10T11:44:11.854 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.836 7f923a2bd700 1 -- 172.21.2.112:0/2210888471 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f9234001600 con 0x7f923c0cff70 2021-02-10T11:44:11.854 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.836 7f923a2bd700 1 -- 172.21.2.112:0/2210888471 >> v1:172.21.2.112:6789/0 conn(0x7f923c0cff70 legacy=0x7f923c109130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:11.854 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.836 7f923a2bd700 1 --2- 172.21.2.112:0/2210888471 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9228003aa0 0x7f9228005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:11.855 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.836 7f923a2bd700 1 --2- 172.21.2.112:0/2210888471 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9228006770 0x7f9228008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:11.855 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.836 7f923a2bd700 1 --2- 172.21.2.112:0/2210888471 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9228009590 0x7f922800bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:11.855 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.836 7f923a2bd700 1 -- 172.21.2.112:0/2210888471 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f922800c300 con 0x7f9228009590 2021-02-10T11:44:11.855 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.836 7f923a2bd700 1 -- 172.21.2.112:0/2210888471 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f922800c520 con 0x7f9228006770 2021-02-10T11:44:11.855 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.836 7f923a2bd700 1 -- 172.21.2.112:0/2210888471 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f922800c740 con 0x7f9228003aa0 2021-02-10T11:44:11.855 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.837 7f923bac0700 1 --2- 172.21.2.112:0/2210888471 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9228009590 0x7f922800bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:11.855 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.837 7f923aabe700 1 --2- 172.21.2.112:0/2210888471 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9228006770 0x7f9228008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:11.855 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.837 7f923b2bf700 1 --2- 172.21.2.112:0/2210888471 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9228003aa0 0x7f9228005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:11.855 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.837 7f923bac0700 1 -- 172.21.2.112:0/2210888471 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9228003aa0 msgr2=0x7f9228005ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:11.855 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.837 7f923bac0700 1 --2- 172.21.2.112:0/2210888471 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9228003aa0 0x7f9228005ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:11.855 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.837 7f923bac0700 1 -- 172.21.2.112:0/2210888471 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9228006770 msgr2=0x7f9228008c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:11.855 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.837 7f923bac0700 1 --2- 172.21.2.112:0/2210888471 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9228006770 0x7f9228008c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:11.855 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.837 7f923bac0700 1 -- 172.21.2.112:0/2210888471 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9228050980 con 0x7f9228009590 2021-02-10T11:44:11.855 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.837 7f923bac0700 1 -- 172.21.2.112:0/2210888471 reap_dead start 2021-02-10T11:44:11.856 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.837 7f923bac0700 1 --2- 172.21.2.112:0/2210888471 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9228009590 0x7f922800bab0 secure :-1 s=READY pgs=2576 cs=0 l=1 rev1=1 rx=0x7f9234000de0 tx=0x7f9234007d00).ready entity=mon.0 client_cookie=bfda7401be856090 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:11.856 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.837 7f923a2bd700 1 -- 172.21.2.112:0/2210888471 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9234001680 con 0x7f9228009590 2021-02-10T11:44:11.856 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.837 7f923a2bd700 1 -- 172.21.2.112:0/2210888471 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92340070c0 con 0x7f9228009590 2021-02-10T11:44:11.856 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.837 7f923a2bd700 1 -- 172.21.2.112:0/2210888471 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f92340045b0 con 0x7f9228009590 2021-02-10T11:44:11.856 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.837 7f924172a700 1 -- 172.21.2.112:0/2210888471 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9228009590 msgr2=0x7f922800bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:11.856 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.837 7f924172a700 1 --2- 172.21.2.112:0/2210888471 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9228009590 0x7f922800bab0 secure :-1 s=READY pgs=2576 cs=0 l=1 rev1=1 rx=0x7f9234000de0 tx=0x7f9234007d00).stop 2021-02-10T11:44:11.856 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.838 7f924172a700 1 -- 172.21.2.112:0/2210888471 shutdown_connections 2021-02-10T11:44:11.856 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.838 7f924172a700 1 --2- 172.21.2.112:0/2210888471 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9228009590 0x7f922800bab0 unknown :-1 s=CLOSED pgs=2576 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:11.856 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.838 7f924172a700 1 -- 172.21.2.112:0/2210888471 >> 172.21.2.112:0/2210888471 conn(0x7f923c0ce420 msgr2=0x7f923c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:11.856 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.838 7f924172a700 1 -- 172.21.2.112:0/2210888471 shutdown_connections 2021-02-10T11:44:11.856 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.838 7f924172a700 1 -- 172.21.2.112:0/2210888471 wait complete. 2021-02-10T11:44:11.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.853 7f924172a700 1 Processor -- start 2021-02-10T11:44:11.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.853 7f924172a700 1 -- start start 2021-02-10T11:44:11.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.853 7f924172a700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f923c085bc0 0x7f923c0863e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:11.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.853 7f924172a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f923c0954c0 0x7f923c096300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:11.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.853 7f924172a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f923c0958c0 0x7f923c096920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:11.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.853 7f924172a700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f923c095d80 con 0x7f923c0954c0 2021-02-10T11:44:11.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.853 7f924172a700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f923c0bfa60 con 0x7f923c085bc0 2021-02-10T11:44:11.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.853 7f924172a700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f923c085fc0 con 0x7f923c0958c0 2021-02-10T11:44:11.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.853 7f923aabe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f923c0954c0 0x7f923c096300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:11.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.853 7f923aabe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f923c0954c0 0x7f923c096300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:57718/0 (socket says 172.21.2.112:57718) 2021-02-10T11:44:11.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.853 7f923aabe700 1 -- 172.21.2.112:0/2577889644 learned_addr learned my addr 172.21.2.112:0/2577889644 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:44:11.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.854 7f923b2bf700 1 --2- 172.21.2.112:0/2577889644 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f923c085bc0 0x7f923c0863e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:11.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.854 7f923bac0700 1 --2- 172.21.2.112:0/2577889644 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f923c0958c0 0x7f923c096920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:11.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.854 7f923aabe700 1 -- 172.21.2.112:0/2577889644 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f923c0958c0 msgr2=0x7f923c096920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:11.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.854 7f923aabe700 1 --2- 172.21.2.112:0/2577889644 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f923c0958c0 0x7f923c096920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:11.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.854 7f923aabe700 1 -- 172.21.2.112:0/2577889644 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f923c085bc0 msgr2=0x7f923c0863e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:11.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.854 7f923aabe700 1 --2- 172.21.2.112:0/2577889644 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f923c085bc0 0x7f923c0863e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:11.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.854 7f923aabe700 1 -- 172.21.2.112:0/2577889644 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f923c086c10 con 0x7f923c0954c0 2021-02-10T11:44:11.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.854 7f923bac0700 1 --2- 172.21.2.112:0/2577889644 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f923c0958c0 0x7f923c096920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:44:11.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.854 7f923aabe700 1 --2- 172.21.2.112:0/2577889644 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f923c0954c0 0x7f923c096300 secure :-1 s=READY pgs=2577 cs=0 l=1 rev1=1 rx=0x7f9224009e90 tx=0x7f922400bce0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:11.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.854 7f9233fff700 1 -- 172.21.2.112:0/2577889644 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9224003020 con 0x7f923c0954c0 2021-02-10T11:44:11.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.854 7f9233fff700 1 -- 172.21.2.112:0/2577889644 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9224009090 con 0x7f923c0954c0 2021-02-10T11:44:11.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.854 7f924172a700 1 -- 172.21.2.112:0/2577889644 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f923c086c10 con 0x7f923c0954c0 2021-02-10T11:44:11.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.854 7f9233fff700 1 -- 172.21.2.112:0/2577889644 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f922400c4e0 con 0x7f923c0954c0 2021-02-10T11:44:11.860 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.854 7f924172a700 1 -- 172.21.2.112:0/2577889644 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f923c086c10 con 0x7f923c0954c0 2021-02-10T11:44:11.860 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.855 7f9233fff700 1 -- 172.21.2.112:0/2577889644 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9224007020 con 0x7f923c0954c0 2021-02-10T11:44:11.860 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.855 7f9233fff700 1 --2- 172.21.2.112:0/2577889644 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f92200186c0 0x7f922001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:11.860 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.855 7f9233fff700 1 -- 172.21.2.112:0/2577889644 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f92240328f0 con 0x7f923c0954c0 2021-02-10T11:44:11.860 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.855 7f923b2bf700 1 --2- 172.21.2.112:0/2577889644 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f92200186c0 0x7f922001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:11.860 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.856 7f923b2bf700 1 --2- 172.21.2.112:0/2577889644 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f92200186c0 0x7f922001abc0 crc :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:11.886 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.885 7f924172a700 1 -- 172.21.2.112:0/2577889644 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f923c0fe1c0 con 0x7f923c0954c0 2021-02-10T11:44:11.888 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:11.888 7f9233fff700 1 -- 172.21.2.112:0/2577889644 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f922401e8d0 con 0x7f923c0954c0 2021-02-10T11:44:12.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:12.019 7f924172a700 1 -- 172.21.2.112:0/2577889644 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f923c0fe1c0 con 0x7f923c0954c0 2021-02-10T11:44:12.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:12.019 7f9233fff700 1 -- 172.21.2.112:0/2577889644 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v143) v1 ==== 92+0+10 (secure 0 0 0) 0x7f922401e8d0 con 0x7f923c0954c0 2021-02-10T11:44:12.021 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:44:12.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:12.021 7f924172a700 1 -- 172.21.2.112:0/2577889644 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f92200186c0 msgr2=0x7f922001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:12.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:12.021 7f924172a700 1 --2- 172.21.2.112:0/2577889644 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f92200186c0 0x7f922001abc0 crc :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:12.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:12.021 7f924172a700 1 -- 172.21.2.112:0/2577889644 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f923c0954c0 msgr2=0x7f923c096300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:12.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:12.021 7f924172a700 1 --2- 172.21.2.112:0/2577889644 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f923c0954c0 0x7f923c096300 secure :-1 s=READY pgs=2577 cs=0 l=1 rev1=1 rx=0x7f9224009e90 tx=0x7f922400bce0).stop 2021-02-10T11:44:12.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:12.021 7f924172a700 1 -- 172.21.2.112:0/2577889644 shutdown_connections 2021-02-10T11:44:12.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:12.022 7f924172a700 1 --2- 172.21.2.112:0/2577889644 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f92200186c0 0x7f922001abc0 unknown :-1 s=CLOSED pgs=1181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:12.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:12.022 7f924172a700 1 --2- 172.21.2.112:0/2577889644 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f923c0958c0 0x7f923c096920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:12.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:12.022 7f924172a700 1 --2- 172.21.2.112:0/2577889644 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f923c0954c0 0x7f923c096300 unknown :-1 s=CLOSED pgs=2577 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:12.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:12.022 7f924172a700 1 --2- 172.21.2.112:0/2577889644 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f923c085bc0 0x7f923c0863e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:12.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:12.022 7f924172a700 1 -- 172.21.2.112:0/2577889644 >> 172.21.2.112:0/2577889644 conn(0x7f923c0bef00 msgr2=0x7f923c09d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:12.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:12.022 7f924172a700 1 -- 172.21.2.112:0/2577889644 shutdown_connections 2021-02-10T11:44:12.024 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:12.022 7f924172a700 1 -- 172.21.2.112:0/2577889644 wait complete. 2021-02-10T11:44:12.031 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:44:12.464 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:12.446 7ff5a16ad700 1 Processor -- start 2021-02-10T11:44:12.464 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:12.446 7ff5a16ad700 1 -- start start 2021-02-10T11:44:12.464 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:12.446 7ff5a16ad700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7ff59c0d1870 0x7ff59c1b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:12.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:12.446 7ff5a16ad700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff59c0cff70 0x7ff59c109130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:12.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:12.446 7ff5a16ad700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff59c109a60 con 0x7ff59c1b0ae0 2021-02-10T11:44:12.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:12.446 7ff5a16ad700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff59c109750 con 0x7ff59c0d1870 2021-02-10T11:44:12.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:12.447 7ff5a16ad700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff59c088070 con 0x7ff59c0cff70 2021-02-10T11:44:12.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:12.447 7ff59aabe700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7ff59c1b0ae0 0x7ff59c108940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:57186/0 (socket says 172.21.2.112:57186) 2021-02-10T11:44:12.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:12.447 7ff59aabe700 1 -- 172.21.2.112:0/1279105878 learned_addr learned my addr 172.21.2.112:0/1279105878 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:44:12.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:12.447 7ff59a2bd700 1 -- 172.21.2.112:0/1279105878 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1887347523 0 0) 0x7ff5900009b0 con 0x7ff59c1b0ae0 2021-02-10T11:44:12.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:12.447 7ff59a2bd700 1 -- 172.21.2.112:0/1279105878 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff584002fe0 con 0x7ff59c1b0ae0 2021-02-10T11:44:12.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:12.447 7ff59b2bf700 1 --2- 172.21.2.112:0/1279105878 >> v2:172.21.2.120:3300/0 conn(0x7ff59c0d1870 0x7ff59c1b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:12.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:12.447 7ff59bac0700 1 --2- 172.21.2.112:0/1279105878 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff59c0cff70 0x7ff59c109130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:12.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:12.448 7ff59a2bd700 1 -- 172.21.2.112:0/1279105878 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 132370989 0 0) 0x7ff5900010b0 con 0x7ff59c1b0ae0 2021-02-10T11:44:12.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:12.448 7ff59a2bd700 1 -- 172.21.2.112:0/1279105878 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff59c0cff70 msgr2=0x7ff59c109130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:12.466 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:12.448 7ff59a2bd700 1 --2- 172.21.2.112:0/1279105878 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff59c0cff70 0x7ff59c109130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:12.466 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:12.448 7ff59a2bd700 1 -- 172.21.2.112:0/1279105878 >> v2:172.21.2.120:3300/0 conn(0x7ff59c0d1870 msgr2=0x7ff59c1b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:12.466 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:12.448 7ff59a2bd700 1 --2- 172.21.2.112:0/1279105878 >> v2:172.21.2.120:3300/0 conn(0x7ff59c0d1870 0x7ff59c1b04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:12.466 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:12.448 7ff59a2bd700 1 -- 172.21.2.112:0/1279105878 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff59c0d1c70 con 0x7ff59c1b0ae0 2021-02-10T11:44:12.466 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:12.448 7ff59a2bd700 1 -- 172.21.2.112:0/1279105878 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7ff5900015b0 con 0x7ff59c1b0ae0 2021-02-10T11:44:12.466 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:12.448 7ff59a2bd700 1 -- 172.21.2.112:0/1279105878 >> v1:172.21.2.112:6789/0 conn(0x7ff59c1b0ae0 legacy=0x7ff59c108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:12.466 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:12.448 7ff59a2bd700 1 --2- 172.21.2.112:0/1279105878 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff584003a60 0x7ff584005f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:12.466 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:12.448 7ff59a2bd700 1 --2- 172.21.2.112:0/1279105878 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff58404a790 0x7ff58404cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:12.466 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:12.448 7ff59a2bd700 1 --2- 172.21.2.112:0/1279105878 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff58404d590 0x7ff58404fa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:12.466 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:12.448 7ff59a2bd700 1 -- 172.21.2.112:0/1279105878 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5840502d0 con 0x7ff58404a790 2021-02-10T11:44:12.466 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:12.448 7ff59a2bd700 1 -- 172.21.2.112:0/1279105878 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5840504f0 con 0x7ff584003a60 2021-02-10T11:44:12.466 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:12.448 7ff59a2bd700 1 -- 172.21.2.112:0/1279105878 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff584050710 con 0x7ff58404d590 2021-02-10T11:44:12.466 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:12.448 7ff59aabe700 1 --2- 172.21.2.112:0/1279105878 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff58404a790 0x7ff58404cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:12.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:12.448 7ff59b2bf700 1 --2- 172.21.2.112:0/1279105878 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff584003a60 0x7ff584005f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:12.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:12.448 7ff59bac0700 1 --2- 172.21.2.112:0/1279105878 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff58404d590 0x7ff58404fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:12.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:12.448 7ff59aabe700 1 -- 172.21.2.112:0/1279105878 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff58404d590 msgr2=0x7ff58404fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:12.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:12.448 7ff59aabe700 1 --2- 172.21.2.112:0/1279105878 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff58404d590 0x7ff58404fa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:12.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:12.448 7ff59aabe700 1 -- 172.21.2.112:0/1279105878 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff584003a60 msgr2=0x7ff584005f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:12.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:12.448 7ff59aabe700 1 --2- 172.21.2.112:0/1279105878 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff584003a60 0x7ff584005f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:12.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:12.448 7ff59aabe700 1 -- 172.21.2.112:0/1279105878 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5840509b0 con 0x7ff58404a790 2021-02-10T11:44:12.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:12.448 7ff59bac0700 1 --2- 172.21.2.112:0/1279105878 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff58404d590 0x7ff58404fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:44:12.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:12.448 7ff59aabe700 1 --2- 172.21.2.112:0/1279105878 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff58404a790 0x7ff58404cc90 secure :-1 s=READY pgs=2579 cs=0 l=1 rev1=1 rx=0x7ff5900017d0 tx=0x7ff590007d30).ready entity=mon.0 client_cookie=483d063e1bd6d89e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:12.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:12.448 7ff59a2bd700 1 -- 172.21.2.112:0/1279105878 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff5900013f0 con 0x7ff58404a790 2021-02-10T11:44:12.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:12.448 7ff59a2bd700 1 -- 172.21.2.112:0/1279105878 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff59000a310 con 0x7ff58404a790 2021-02-10T11:44:12.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:12.449 7ff59a2bd700 1 -- 172.21.2.112:0/1279105878 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff59000a7f0 con 0x7ff58404a790 2021-02-10T11:44:12.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:12.449 7ff5a16ad700 1 -- 172.21.2.112:0/1279105878 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff58404a790 msgr2=0x7ff58404cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:12.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:12.449 7ff5a16ad700 1 --2- 172.21.2.112:0/1279105878 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff58404a790 0x7ff58404cc90 secure :-1 s=READY pgs=2579 cs=0 l=1 rev1=1 rx=0x7ff5900017d0 tx=0x7ff590007d30).stop 2021-02-10T11:44:12.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:12.449 7ff59bac0700 1 -- 172.21.2.112:0/1279105878 reap_dead start 2021-02-10T11:44:12.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:12.449 7ff5a16ad700 1 -- 172.21.2.112:0/1279105878 shutdown_connections 2021-02-10T11:44:12.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:12.449 7ff5a16ad700 1 -- 172.21.2.112:0/1279105878 >> 172.21.2.112:0/1279105878 conn(0x7ff59c0ce420 msgr2=0x7ff59c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:12.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:12.449 7ff5a16ad700 1 -- 172.21.2.112:0/1279105878 shutdown_connections 2021-02-10T11:44:12.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:12.449 7ff5a16ad700 1 -- 172.21.2.112:0/1279105878 wait complete. 2021-02-10T11:44:12.469 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:12.464 7ff5a16ad700 1 Processor -- start 2021-02-10T11:44:12.469 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:12.464 7ff5a16ad700 1 -- start start 2021-02-10T11:44:12.469 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:12.464 7ff5a16ad700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff59c085bc0 0x7ff59c0863e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:12.469 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:12.464 7ff5a16ad700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff59c0954c0 0x7ff59c096300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:12.470 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:12.464 7ff5a16ad700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff59c0958c0 0x7ff59c096920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:12.470 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:12.464 7ff5a16ad700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff59c095d80 con 0x7ff59c085bc0 2021-02-10T11:44:12.470 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:12.464 7ff5a16ad700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff59c0bfa60 con 0x7ff59c0954c0 2021-02-10T11:44:12.470 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:12.464 7ff5a16ad700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff59c085fc0 con 0x7ff59c0958c0 2021-02-10T11:44:12.470 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:12.464 7ff59b2bf700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff59c085bc0 0x7ff59c0863e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:12.470 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:12.465 7ff59b2bf700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff59c085bc0 0x7ff59c0863e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:57734/0 (socket says 172.21.2.112:57734) 2021-02-10T11:44:12.470 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:12.465 7ff59b2bf700 1 -- 172.21.2.112:0/1050712975 learned_addr learned my addr 172.21.2.112:0/1050712975 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:44:12.470 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:12.465 7ff59aabe700 1 --2- 172.21.2.112:0/1050712975 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff59c0954c0 0x7ff59c096300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:12.470 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:12.465 7ff59bac0700 1 --2- 172.21.2.112:0/1050712975 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff59c0958c0 0x7ff59c096920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:12.470 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:12.465 7ff59b2bf700 1 -- 172.21.2.112:0/1050712975 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff59c0958c0 msgr2=0x7ff59c096920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:12.470 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:12.465 7ff59b2bf700 1 --2- 172.21.2.112:0/1050712975 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff59c0958c0 0x7ff59c096920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:12.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:12.465 7ff59b2bf700 1 -- 172.21.2.112:0/1050712975 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff59c0954c0 msgr2=0x7ff59c096300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:12.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:12.465 7ff59b2bf700 1 --2- 172.21.2.112:0/1050712975 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff59c0954c0 0x7ff59c096300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:12.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:12.465 7ff59b2bf700 1 -- 172.21.2.112:0/1050712975 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff59c086c10 con 0x7ff59c085bc0 2021-02-10T11:44:12.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:12.465 7ff59b2bf700 1 --2- 172.21.2.112:0/1050712975 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff59c085bc0 0x7ff59c0863e0 secure :-1 s=READY pgs=2580 cs=0 l=1 rev1=1 rx=0x7ff58c006f60 tx=0x7ff58c00b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:12.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:12.465 7ff583fff700 1 -- 172.21.2.112:0/1050712975 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff58c0009b0 con 0x7ff59c085bc0 2021-02-10T11:44:12.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:12.465 7ff583fff700 1 -- 172.21.2.112:0/1050712975 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff58c00bd60 con 0x7ff59c085bc0 2021-02-10T11:44:12.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:12.465 7ff5a16ad700 1 -- 172.21.2.112:0/1050712975 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff59c086c10 con 0x7ff59c085bc0 2021-02-10T11:44:12.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:12.465 7ff583fff700 1 -- 172.21.2.112:0/1050712975 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff58c002030 con 0x7ff59c085bc0 2021-02-10T11:44:12.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:12.465 7ff5a16ad700 1 -- 172.21.2.112:0/1050712975 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff59c097230 con 0x7ff59c085bc0 2021-02-10T11:44:12.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:12.466 7ff583fff700 1 -- 172.21.2.112:0/1050712975 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff58c000c90 con 0x7ff59c085bc0 2021-02-10T11:44:12.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:12.466 7ff583fff700 1 --2- 172.21.2.112:0/1050712975 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff57c0186c0 0x7ff57c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:12.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:12.466 7ff583fff700 1 -- 172.21.2.112:0/1050712975 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5573+0+0 (secure 0 0 0) 0x7ff58c032570 con 0x7ff59c085bc0 2021-02-10T11:44:12.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:12.466 7ff59aabe700 1 --2- 172.21.2.112:0/1050712975 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff57c0186c0 0x7ff57c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:12.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:12.466 7ff59aabe700 1 --2- 172.21.2.112:0/1050712975 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff57c0186c0 0x7ff57c01abc0 crc :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:12.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:12.496 7ff5a16ad700 1 -- 172.21.2.112:0/1050712975 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff59c0fe1c0 con 0x7ff59c085bc0 2021-02-10T11:44:12.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:12.499 7ff583fff700 1 -- 172.21.2.112:0/1050712975 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff58c02ae00 con 0x7ff59c085bc0 2021-02-10T11:44:12.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:12.627 7ff5a16ad700 1 -- 172.21.2.112:0/1050712975 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7ff59c0fe1c0 con 0x7ff59c085bc0 2021-02-10T11:44:12.629 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:12.628 7ff583fff700 1 -- 172.21.2.112:0/1050712975 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v143) v1 ==== 104+0+10 (secure 0 0 0) 0x7ff58c02ae00 con 0x7ff59c085bc0 2021-02-10T11:44:12.629 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:44:12.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:12.629 7ff5a16ad700 1 -- 172.21.2.112:0/1050712975 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff57c0186c0 msgr2=0x7ff57c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:12.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:12.629 7ff5a16ad700 1 --2- 172.21.2.112:0/1050712975 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff57c0186c0 0x7ff57c01abc0 crc :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:12.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:12.629 7ff5a16ad700 1 -- 172.21.2.112:0/1050712975 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff59c085bc0 msgr2=0x7ff59c0863e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:12.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:12.629 7ff5a16ad700 1 --2- 172.21.2.112:0/1050712975 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff59c085bc0 0x7ff59c0863e0 secure :-1 s=READY pgs=2580 cs=0 l=1 rev1=1 rx=0x7ff58c006f60 tx=0x7ff58c00b970).stop 2021-02-10T11:44:12.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:12.630 7ff5a16ad700 1 -- 172.21.2.112:0/1050712975 shutdown_connections 2021-02-10T11:44:12.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:12.630 7ff5a16ad700 1 --2- 172.21.2.112:0/1050712975 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff57c0186c0 0x7ff57c01abc0 unknown :-1 s=CLOSED pgs=1182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:12.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:12.630 7ff5a16ad700 1 --2- 172.21.2.112:0/1050712975 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff59c0954c0 0x7ff59c096300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:12.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:12.630 7ff5a16ad700 1 --2- 172.21.2.112:0/1050712975 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff59c0958c0 0x7ff59c096920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:12.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:12.630 7ff5a16ad700 1 --2- 172.21.2.112:0/1050712975 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff59c085bc0 0x7ff59c0863e0 unknown :-1 s=CLOSED pgs=2580 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:12.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:12.630 7ff5a16ad700 1 -- 172.21.2.112:0/1050712975 >> 172.21.2.112:0/1050712975 conn(0x7ff59c0bef00 msgr2=0x7ff59c09d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:12.632 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:12.630 7ff5a16ad700 1 -- 172.21.2.112:0/1050712975 shutdown_connections 2021-02-10T11:44:12.632 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:12.630 7ff5a16ad700 1 -- 172.21.2.112:0/1050712975 wait complete. 2021-02-10T11:44:12.639 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:44:13.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.052 7fb49171d700 1 Processor -- start 2021-02-10T11:44:13.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.052 7fb49171d700 1 -- start start 2021-02-10T11:44:13.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.053 7fb49171d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb48c0d1870 0x7fb48c1b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:13.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.053 7fb49171d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fb48c0cff70 0x7fb48c109130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:13.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.053 7fb49171d700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb48c109a60 con 0x7fb48c1b0ae0 2021-02-10T11:44:13.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.053 7fb49171d700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb48c109750 con 0x7fb48c0cff70 2021-02-10T11:44:13.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.053 7fb49171d700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb48c088070 con 0x7fb48c0d1870 2021-02-10T11:44:13.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.053 7fb48bac0700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fb48c0cff70 0x7fb48c109130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:13.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.053 7fb48bac0700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fb48c0cff70 0x7fb48c109130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:57742/0 (socket says 172.21.2.112:57742) 2021-02-10T11:44:13.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.053 7fb48aabe700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fb48c1b0ae0 0x7fb48c108940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:57204/0 (socket says 172.21.2.112:57204) 2021-02-10T11:44:13.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.053 7fb48bac0700 1 -- 172.21.2.112:0/2281943088 learned_addr learned my addr 172.21.2.112:0/2281943088 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:44:13.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.053 7fb48b2bf700 1 --2- 172.21.2.112:0/2281943088 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb48c0d1870 0x7fb48c1b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:13.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.053 7fb48bac0700 1 -- 172.21.2.112:0/2281943088 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb48c0d1870 msgr2=0x7fb48c1b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:13.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.053 7fb48bac0700 1 --2- 172.21.2.112:0/2281943088 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb48c0d1870 0x7fb48c1b04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:13.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.053 7fb48bac0700 1 -- 172.21.2.112:0/2281943088 >> v1:172.21.2.112:6789/0 conn(0x7fb48c1b0ae0 legacy=0x7fb48c108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:13.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.053 7fb48bac0700 1 -- 172.21.2.112:0/2281943088 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb48c088260 con 0x7fb48c0cff70 2021-02-10T11:44:13.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.054 7fb48bac0700 1 --2- 172.21.2.112:0/2281943088 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb48c0cff70 0x7fb48c109130 secure :-1 s=READY pgs=2582 cs=0 l=1 rev1=1 rx=0x7fb484006f00 tx=0x7fb484009f10).ready entity=mon.0 client_cookie=d6cd750e3a4dae84 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:13.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.054 7fb48a2bd700 1 -- 172.21.2.112:0/2281943088 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb48400b3b0 con 0x7fb48c0cff70 2021-02-10T11:44:13.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.054 7fb48a2bd700 1 -- 172.21.2.112:0/2281943088 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb48400b630 con 0x7fb48c0cff70 2021-02-10T11:44:13.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.054 7fb48a2bd700 1 -- 172.21.2.112:0/2281943088 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb48400bb50 con 0x7fb48c0cff70 2021-02-10T11:44:13.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.054 7fb49171d700 1 -- 172.21.2.112:0/2281943088 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb48c0cff70 msgr2=0x7fb48c109130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:13.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.054 7fb49171d700 1 --2- 172.21.2.112:0/2281943088 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb48c0cff70 0x7fb48c109130 secure :-1 s=READY pgs=2582 cs=0 l=1 rev1=1 rx=0x7fb484006f00 tx=0x7fb484009f10).stop 2021-02-10T11:44:13.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.055 7fb49171d700 1 -- 172.21.2.112:0/2281943088 shutdown_connections 2021-02-10T11:44:13.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.055 7fb49171d700 1 --2- 172.21.2.112:0/2281943088 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb48c0cff70 0x7fb48c109130 unknown :-1 s=CLOSED pgs=2582 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:13.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.055 7fb49171d700 1 --2- 172.21.2.112:0/2281943088 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb48c0d1870 0x7fb48c1b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:13.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.055 7fb49171d700 1 -- 172.21.2.112:0/2281943088 >> 172.21.2.112:0/2281943088 conn(0x7fb48c0ce420 msgr2=0x7fb48c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:13.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.055 7fb49171d700 1 -- 172.21.2.112:0/2281943088 shutdown_connections 2021-02-10T11:44:13.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.055 7fb49171d700 1 -- 172.21.2.112:0/2281943088 wait complete. 2021-02-10T11:44:13.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.069 7fb49171d700 1 Processor -- start 2021-02-10T11:44:13.073 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.070 7fb49171d700 1 -- start start 2021-02-10T11:44:13.074 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.070 7fb49171d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb48c0b1ba0 0x7fb48c1a78d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:13.074 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.070 7fb49171d700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb48c1a7ef0 0x7fb48c0a6a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:13.074 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.070 7fb49171d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb48c0a7060 0x7fb48c0b2920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:13.074 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.070 7fb49171d700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb48c190c20 con 0x7fb48c0b1ba0 2021-02-10T11:44:13.074 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.070 7fb49171d700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb48c190710 con 0x7fb48c1a7ef0 2021-02-10T11:44:13.074 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.070 7fb49171d700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb48c0b1fa0 con 0x7fb48c0a7060 2021-02-10T11:44:13.074 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.070 7fb48b2bf700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb48c0b1ba0 0x7fb48c1a78d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:13.074 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.070 7fb48b2bf700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb48c0b1ba0 0x7fb48c1a78d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:57750/0 (socket says 172.21.2.112:57750) 2021-02-10T11:44:13.074 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.070 7fb48b2bf700 1 -- 172.21.2.112:0/2039533532 learned_addr learned my addr 172.21.2.112:0/2039533532 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:44:13.075 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.070 7fb48bac0700 1 --2- 172.21.2.112:0/2039533532 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb48c0a7060 0x7fb48c0b2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:13.075 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.070 7fb48aabe700 1 --2- 172.21.2.112:0/2039533532 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb48c1a7ef0 0x7fb48c0a6a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:13.075 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.070 7fb48b2bf700 1 -- 172.21.2.112:0/2039533532 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb48c0a7060 msgr2=0x7fb48c0b2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:13.075 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.070 7fb48b2bf700 1 --2- 172.21.2.112:0/2039533532 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb48c0a7060 0x7fb48c0b2920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:13.075 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.070 7fb48b2bf700 1 -- 172.21.2.112:0/2039533532 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb48c1a7ef0 msgr2=0x7fb48c0a6a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:13.075 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.070 7fb48b2bf700 1 --2- 172.21.2.112:0/2039533532 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb48c1a7ef0 0x7fb48c0a6a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:13.075 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.071 7fb48b2bf700 1 -- 172.21.2.112:0/2039533532 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb48c0a7460 con 0x7fb48c0b1ba0 2021-02-10T11:44:13.075 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.071 7fb48b2bf700 1 --2- 172.21.2.112:0/2039533532 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb48c0b1ba0 0x7fb48c1a78d0 secure :-1 s=READY pgs=2583 cs=0 l=1 rev1=1 rx=0x7fb47c006f60 tx=0x7fb47c00b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:13.075 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.071 7fb473fff700 1 -- 172.21.2.112:0/2039533532 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb47c00bd60 con 0x7fb48c0b1ba0 2021-02-10T11:44:13.075 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.071 7fb473fff700 1 -- 172.21.2.112:0/2039533532 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb47c000c90 con 0x7fb48c0b1ba0 2021-02-10T11:44:13.075 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.071 7fb473fff700 1 -- 172.21.2.112:0/2039533532 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb47c002220 con 0x7fb48c0b1ba0 2021-02-10T11:44:13.075 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.071 7fb49171d700 1 -- 172.21.2.112:0/2039533532 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb48c0a7460 con 0x7fb48c0b1ba0 2021-02-10T11:44:13.075 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.071 7fb49171d700 1 -- 172.21.2.112:0/2039533532 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb48c0b3230 con 0x7fb48c0b1ba0 2021-02-10T11:44:13.075 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.071 7fb473fff700 1 -- 172.21.2.112:0/2039533532 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb47c0052e0 con 0x7fb48c0b1ba0 2021-02-10T11:44:13.076 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.071 7fb473fff700 1 --2- 172.21.2.112:0/2039533532 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb46c0186c0 0x7fb46c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:13.076 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.072 7fb473fff700 1 -- 172.21.2.112:0/2039533532 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fb47c0323d0 con 0x7fb48c0b1ba0 2021-02-10T11:44:13.076 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.072 7fb48aabe700 1 --2- 172.21.2.112:0/2039533532 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb46c0186c0 0x7fb46c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:13.076 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.072 7fb48aabe700 1 --2- 172.21.2.112:0/2039533532 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb46c0186c0 0x7fb46c01abc0 crc :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:13.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.103 7fb49171d700 1 -- 172.21.2.112:0/2039533532 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb48c082980 con 0x7fb48c0b1ba0 2021-02-10T11:44:13.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.105 7fb473fff700 1 -- 172.21.2.112:0/2039533532 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb47c000940 con 0x7fb48c0b1ba0 2021-02-10T11:44:13.234 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.233 7fb49171d700 1 -- 172.21.2.112:0/2039533532 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7fb48c082980 con 0x7fb48c0b1ba0 2021-02-10T11:44:13.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.233 7fb473fff700 1 -- 172.21.2.112:0/2039533532 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v143) v1 ==== 100+0+10 (secure 0 0 0) 0x7fb47c00bd60 con 0x7fb48c0b1ba0 2021-02-10T11:44:13.235 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:44:13.236 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.235 7fb49171d700 1 -- 172.21.2.112:0/2039533532 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb46c0186c0 msgr2=0x7fb46c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:13.236 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.235 7fb49171d700 1 --2- 172.21.2.112:0/2039533532 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb46c0186c0 0x7fb46c01abc0 crc :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:13.236 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.235 7fb49171d700 1 -- 172.21.2.112:0/2039533532 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb48c0b1ba0 msgr2=0x7fb48c1a78d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:13.236 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.235 7fb49171d700 1 --2- 172.21.2.112:0/2039533532 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb48c0b1ba0 0x7fb48c1a78d0 secure :-1 s=READY pgs=2583 cs=0 l=1 rev1=1 rx=0x7fb47c006f60 tx=0x7fb47c00b970).stop 2021-02-10T11:44:13.236 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.235 7fb49171d700 1 -- 172.21.2.112:0/2039533532 shutdown_connections 2021-02-10T11:44:13.237 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.236 7fb49171d700 1 --2- 172.21.2.112:0/2039533532 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb46c0186c0 0x7fb46c01abc0 unknown :-1 s=CLOSED pgs=1183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:13.237 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.236 7fb49171d700 1 --2- 172.21.2.112:0/2039533532 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb48c1a7ef0 0x7fb48c0a6a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:13.237 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.236 7fb49171d700 1 --2- 172.21.2.112:0/2039533532 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb48c0a7060 0x7fb48c0b2920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:13.237 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.236 7fb49171d700 1 --2- 172.21.2.112:0/2039533532 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb48c0b1ba0 0x7fb48c1a78d0 unknown :-1 s=CLOSED pgs=2583 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:13.237 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.236 7fb49171d700 1 -- 172.21.2.112:0/2039533532 >> 172.21.2.112:0/2039533532 conn(0x7fb48c0b5dc0 msgr2=0x7fb48c095960 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:13.237 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.236 7fb49171d700 1 -- 172.21.2.112:0/2039533532 shutdown_connections 2021-02-10T11:44:13.237 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.236 7fb49171d700 1 -- 172.21.2.112:0/2039533532 wait complete. 2021-02-10T11:44:13.244 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:44:13.245 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:44:13.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.654 7f2066c34700 1 Processor -- start 2021-02-10T11:44:13.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.654 7f2066c34700 1 -- start start 2021-02-10T11:44:13.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.655 7f2066c34700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f20600d1870 0x7f20601b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:13.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.655 7f2066c34700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f20601b0a10 0x7f20600cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:13.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.655 7f2066c34700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20600f0b10 con 0x7f20600d1c70 2021-02-10T11:44:13.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.655 7f2066c34700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20600fe230 con 0x7f20601b0a10 2021-02-10T11:44:13.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.655 7f2066c34700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20600e5ed0 con 0x7f20600d1870 2021-02-10T11:44:13.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.655 7f2065251700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f20600d1c70 0x7f20600cf950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:57220/0 (socket says 172.21.2.112:57220) 2021-02-10T11:44:13.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.655 7f2065251700 1 -- 172.21.2.112:0/464363793 learned_addr learned my addr 172.21.2.112:0/464363793 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:44:13.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.655 7f2057fff700 1 --2- 172.21.2.112:0/464363793 >> v2:172.21.2.112:3300/0 conn(0x7f20601b0a10 0x7f20600cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:13.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.655 7f2064a50700 1 --2- 172.21.2.112:0/464363793 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f20600d1870 0x7f20601b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:13.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.655 7f2057fff700 1 -- 172.21.2.112:0/464363793 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f20600d1870 msgr2=0x7f20601b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:13.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.655 7f2057fff700 1 --2- 172.21.2.112:0/464363793 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f20600d1870 0x7f20601b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:13.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.655 7f2057fff700 1 -- 172.21.2.112:0/464363793 >> v1:172.21.2.112:6789/0 conn(0x7f20600d1c70 legacy=0x7f20600cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:13.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.655 7f2057fff700 1 -- 172.21.2.112:0/464363793 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20600f0650 con 0x7f20601b0a10 2021-02-10T11:44:13.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.655 7f2057fff700 1 --2- 172.21.2.112:0/464363793 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f20601b0a10 0x7f20600cf330 secure :-1 s=READY pgs=2585 cs=0 l=1 rev1=1 rx=0x7f2050006650 tx=0x7f2050009a80).ready entity=mon.0 client_cookie=aae0d79173508107 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:13.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.656 7f20577fe700 1 -- 172.21.2.112:0/464363793 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f20500009b0 con 0x7f20601b0a10 2021-02-10T11:44:13.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.656 7f20577fe700 1 -- 172.21.2.112:0/464363793 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f205000c0c0 con 0x7f20601b0a10 2021-02-10T11:44:13.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.656 7f20577fe700 1 -- 172.21.2.112:0/464363793 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f205000c5e0 con 0x7f20601b0a10 2021-02-10T11:44:13.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.656 7f2066c34700 1 -- 172.21.2.112:0/464363793 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f20601b0a10 msgr2=0x7f20600cf330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:13.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.656 7f2066c34700 1 --2- 172.21.2.112:0/464363793 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f20601b0a10 0x7f20600cf330 secure :-1 s=READY pgs=2585 cs=0 l=1 rev1=1 rx=0x7f2050006650 tx=0x7f2050009a80).stop 2021-02-10T11:44:13.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.656 7f2066c34700 1 -- 172.21.2.112:0/464363793 shutdown_connections 2021-02-10T11:44:13.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.656 7f2066c34700 1 --2- 172.21.2.112:0/464363793 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f20601b0a10 0x7f20600cf330 unknown :-1 s=CLOSED pgs=2585 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:13.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.656 7f2066c34700 1 --2- 172.21.2.112:0/464363793 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f20600d1870 0x7f20601b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:13.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.656 7f2066c34700 1 -- 172.21.2.112:0/464363793 >> 172.21.2.112:0/464363793 conn(0x7f20600ce450 msgr2=0x7f20600d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:13.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.656 7f2066c34700 1 -- 172.21.2.112:0/464363793 shutdown_connections 2021-02-10T11:44:13.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.656 7f2066c34700 1 -- 172.21.2.112:0/464363793 wait complete. 2021-02-10T11:44:13.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.671 7f2066c34700 1 Processor -- start 2021-02-10T11:44:13.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.671 7f2066c34700 1 -- start start 2021-02-10T11:44:13.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.672 7f2066c34700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f206018d870 0x7f20600b19b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:13.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.672 7f2066c34700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f20600b1fd0 0x7f2060099ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:13.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.672 7f2066c34700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f20600b23d0 0x7f206009a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:13.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.672 7f2066c34700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f206018dc70 con 0x7f206018d870 2021-02-10T11:44:13.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.672 7f2066c34700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2060199e60 con 0x7f20600b23d0 2021-02-10T11:44:13.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.672 7f2066c34700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f206009ab30 con 0x7f20600b1fd0 2021-02-10T11:44:13.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.672 7f2064a50700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f206018d870 0x7f20600b19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:13.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.672 7f2064a50700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f206018d870 0x7f20600b19b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:57762/0 (socket says 172.21.2.112:57762) 2021-02-10T11:44:13.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.672 7f2064a50700 1 -- 172.21.2.112:0/1342230581 learned_addr learned my addr 172.21.2.112:0/1342230581 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:44:13.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.672 7f2065251700 1 --2- 172.21.2.112:0/1342230581 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f20600b23d0 0x7f206009a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:13.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.672 7f2057fff700 1 --2- 172.21.2.112:0/1342230581 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f20600b1fd0 0x7f2060099ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:13.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.672 7f2064a50700 1 -- 172.21.2.112:0/1342230581 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f20600b1fd0 msgr2=0x7f2060099ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:13.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.672 7f2064a50700 1 --2- 172.21.2.112:0/1342230581 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f20600b1fd0 0x7f2060099ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:13.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.672 7f2064a50700 1 -- 172.21.2.112:0/1342230581 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f20600b23d0 msgr2=0x7f206009a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:13.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.672 7f2064a50700 1 --2- 172.21.2.112:0/1342230581 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f20600b23d0 0x7f206009a510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:13.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.672 7f2064a50700 1 -- 172.21.2.112:0/1342230581 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f206009ad50 con 0x7f206018d870 2021-02-10T11:44:13.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.672 7f2065251700 1 --2- 172.21.2.112:0/1342230581 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f20600b23d0 0x7f206009a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:44:13.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.673 7f2064a50700 1 --2- 172.21.2.112:0/1342230581 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f206018d870 0x7f20600b19b0 secure :-1 s=READY pgs=2586 cs=0 l=1 rev1=1 rx=0x7f205c006fa0 tx=0x7f205c00bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:13.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.673 7f20557fa700 1 -- 172.21.2.112:0/1342230581 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f205c00c2f0 con 0x7f206018d870 2021-02-10T11:44:13.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.673 7f20557fa700 1 -- 172.21.2.112:0/1342230581 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f205c00c630 con 0x7f206018d870 2021-02-10T11:44:13.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.673 7f2066c34700 1 -- 172.21.2.112:0/1342230581 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f206009ad50 con 0x7f206018d870 2021-02-10T11:44:13.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.673 7f20557fa700 1 -- 172.21.2.112:0/1342230581 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f205c00ca90 con 0x7f206018d870 2021-02-10T11:44:13.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.673 7f2066c34700 1 -- 172.21.2.112:0/1342230581 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f206009ad50 con 0x7f206018d870 2021-02-10T11:44:13.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.673 7f20557fa700 1 -- 172.21.2.112:0/1342230581 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f205c00c2f0 con 0x7f206018d870 2021-02-10T11:44:13.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.674 7f20557fa700 1 --2- 172.21.2.112:0/1342230581 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f20400186c0 0x7f204001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:13.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.674 7f20557fa700 1 -- 172.21.2.112:0/1342230581 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f205c032920 con 0x7f206018d870 2021-02-10T11:44:13.678 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.674 7f2057fff700 1 --2- 172.21.2.112:0/1342230581 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f20400186c0 0x7f204001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:13.678 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.674 7f2057fff700 1 --2- 172.21.2.112:0/1342230581 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f20400186c0 0x7f204001abc0 crc :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:13.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.705 7f2066c34700 1 -- 172.21.2.112:0/1342230581 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20600999c0 con 0x7f206018d870 2021-02-10T11:44:13.708 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.707 7f20557fa700 1 -- 172.21.2.112:0/1342230581 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f205c000cc0 con 0x7f206018d870 2021-02-10T11:44:13.836 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.835 7f2066c34700 1 -- 172.21.2.112:0/1342230581 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f20600999c0 con 0x7f206018d870 2021-02-10T11:44:13.837 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.836 7f20557fa700 1 -- 172.21.2.112:0/1342230581 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) v1 ==== 95+0+2108 (secure 0 0 0) 0x7f205c00c2f0 con 0x7f206018d870 2021-02-10T11:44:13.837 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 122 2021-02-10T11:44:13.837 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:44:13.838 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":122,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11670,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/1286703435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":1286703435},{"type":"v1","addr":"172.21.2.112:6833","nonce":1286703435}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":119}],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:43:56.409867","modified":"2021-02-10 11:43:58.413644","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8578},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8578":{"gid":8578,"name":"b","rank":0,"incarnation":120,"state":"up:active","state_seq":3,"addr":"172.21.2.120:6835/1504781062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":1504781062},{"type":"v1","addr":"172.21.2.120:6835","nonce":1504781062}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":15}]} 2021-02-10T11:44:13.839 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.838 7f2066c34700 1 -- 172.21.2.112:0/1342230581 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f20400186c0 msgr2=0x7f204001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:13.839 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.838 7f2066c34700 1 --2- 172.21.2.112:0/1342230581 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f20400186c0 0x7f204001abc0 crc :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:13.839 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.838 7f2066c34700 1 -- 172.21.2.112:0/1342230581 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f206018d870 msgr2=0x7f20600b19b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:13.839 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.838 7f2066c34700 1 --2- 172.21.2.112:0/1342230581 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f206018d870 0x7f20600b19b0 secure :-1 s=READY pgs=2586 cs=0 l=1 rev1=1 rx=0x7f205c006fa0 tx=0x7f205c00bd40).stop 2021-02-10T11:44:13.839 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.838 7f2066c34700 1 -- 172.21.2.112:0/1342230581 shutdown_connections 2021-02-10T11:44:13.839 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.838 7f2066c34700 1 --2- 172.21.2.112:0/1342230581 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f20400186c0 0x7f204001abc0 unknown :-1 s=CLOSED pgs=1184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:13.839 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.838 7f2066c34700 1 --2- 172.21.2.112:0/1342230581 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f20600b23d0 0x7f206009a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:13.839 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.838 7f2066c34700 1 --2- 172.21.2.112:0/1342230581 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f20600b1fd0 0x7f2060099ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:13.839 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.838 7f2066c34700 1 --2- 172.21.2.112:0/1342230581 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f206018d870 0x7f20600b19b0 unknown :-1 s=CLOSED pgs=2586 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:13.839 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.838 7f2066c34700 1 -- 172.21.2.112:0/1342230581 >> 172.21.2.112:0/1342230581 conn(0x7f20600f0550 msgr2=0x7f2060191730 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:13.839 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.838 7f2066c34700 1 -- 172.21.2.112:0/1342230581 shutdown_connections 2021-02-10T11:44:13.840 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:13.838 7f2066c34700 1 -- 172.21.2.112:0/1342230581 wait complete. 2021-02-10T11:44:14.049 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 122, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:43:56.409867', 'modified': '2021-02-10 11:43:58.413644', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8578}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8578': {'gid': 8578, 'name': 'b', 'rank': 0, 'incarnation': 120, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.120:6835/1504781062', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.120:6834', 'nonce': 1504781062}, {'type': 'v1', 'addr': '172.21.2.120:6835', 'nonce': 1504781062}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [31], 'metadata_pool': 30, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:44:14.049 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:44:14.049 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:44:14.448 INFO:teuthology.orchestra.run.gibba020.stdout:{ 2021-02-10T11:44:14.449 INFO:teuthology.orchestra.run.gibba020.stdout: "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", 2021-02-10T11:44:14.449 INFO:teuthology.orchestra.run.gibba020.stdout: "whoami": 0, 2021-02-10T11:44:14.449 INFO:teuthology.orchestra.run.gibba020.stdout: "id": 8578, 2021-02-10T11:44:14.449 INFO:teuthology.orchestra.run.gibba020.stdout: "want_state": "up:active", 2021-02-10T11:44:14.449 INFO:teuthology.orchestra.run.gibba020.stdout: "state": "up:active", 2021-02-10T11:44:14.449 INFO:teuthology.orchestra.run.gibba020.stdout: "rank_uptime": 18.036424236999999, 2021-02-10T11:44:14.449 INFO:teuthology.orchestra.run.gibba020.stdout: "mdsmap_epoch": 122, 2021-02-10T11:44:14.449 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch": 143, 2021-02-10T11:44:14.449 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch_barrier": 143, 2021-02-10T11:44:14.449 INFO:teuthology.orchestra.run.gibba020.stdout: "uptime": 23.681926063999999 2021-02-10T11:44:14.449 INFO:teuthology.orchestra.run.gibba020.stdout:} 2021-02-10T11:44:14.453 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", "whoami": 0, "id": 8578, "want_state": "up:active", "state": "up:active", "rank_uptime": 18.036424236999999, "mdsmap_epoch": 122, "osdmap_epoch": 143, "osdmap_epoch_barrier": 143, "uptime": 23.681926063999999 } 2021-02-10T11:44:14.453 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:44:14.453 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2021-02-10T11:44:14.453 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.1 at ubuntu@gibba012.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2021-02-10T11:44:14.453 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:44:14.453 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:44:14.788 DEBUG:teuthology.orchestra.run.gibba012:> sudo modprobe fuse 2021-02-10T11:44:15.131 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:44:15.132 DEBUG:teuthology.orchestra.run.gibba012:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:44:15.475 INFO:teuthology.orchestra.run.gibba012.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:44:15.475 INFO:teuthology.orchestra.run.gibba012.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:44:15.475 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:44:15.476 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:44:15.476 DEBUG:teuthology.orchestra.run.gibba012:> ls /sys/fs/fuse/connections 2021-02-10T11:44:15.811 INFO:teuthology.orchestra.run.gibba012.stdout:42 2021-02-10T11:44:15.811 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [42] 2021-02-10T11:44:15.811 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.1 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:44:15.813 DEBUG:teuthology.orchestra.run.gibba012:> sudo modprobe fuse 2021-02-10T11:44:16.158 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:44:16.158 DEBUG:teuthology.orchestra.run.gibba012:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:44:16.223 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba012.stderr:ceph-fuse[173571]: starting ceph client 2021-02-10T11:44:16.224 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba012.stderr:2021-02-10 11:44:16.223 7fa3e31fff80 -1 init, newargv = 0x55ef61c9b1a0 newargc=9 2021-02-10T11:44:16.230 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba012.stderr:ceph-fuse[173571]: starting fuse 2021-02-10T11:44:16.500 INFO:teuthology.orchestra.run.gibba012.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:44:16.501 INFO:teuthology.orchestra.run.gibba012.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:44:16.501 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:44:16.502 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:44:16.502 DEBUG:teuthology.orchestra.run.gibba012:> ls /sys/fs/fuse/connections 2021-02-10T11:44:16.837 INFO:teuthology.orchestra.run.gibba012.stdout:42 2021-02-10T11:44:16.837 INFO:teuthology.orchestra.run.gibba012.stdout:43 2021-02-10T11:44:16.837 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42, 43] 2021-02-10T11:44:16.838 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:44:16.838 DEBUG:teuthology.orchestra.run.gibba012:> sudo python3 -c ' 2021-02-10T11:44:16.838 DEBUG:teuthology.orchestra.run.gibba012:> import glob 2021-02-10T11:44:16.838 DEBUG:teuthology.orchestra.run.gibba012:> import re 2021-02-10T11:44:16.838 DEBUG:teuthology.orchestra.run.gibba012:> import os 2021-02-10T11:44:16.838 DEBUG:teuthology.orchestra.run.gibba012:> import subprocess 2021-02-10T11:44:16.838 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:44:16.838 DEBUG:teuthology.orchestra.run.gibba012:> def find_socket(client_name): 2021-02-10T11:44:16.838 DEBUG:teuthology.orchestra.run.gibba012:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2021-02-10T11:44:16.838 DEBUG:teuthology.orchestra.run.gibba012:> files = glob.glob(asok_path) 2021-02-10T11:44:16.838 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:44:16.839 DEBUG:teuthology.orchestra.run.gibba012:> # Given a non-glob path, it better be there 2021-02-10T11:44:16.839 DEBUG:teuthology.orchestra.run.gibba012:> if "*" not in asok_path: 2021-02-10T11:44:16.839 DEBUG:teuthology.orchestra.run.gibba012:> assert(len(files) == 1) 2021-02-10T11:44:16.839 DEBUG:teuthology.orchestra.run.gibba012:> return files[0] 2021-02-10T11:44:16.839 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:44:16.839 DEBUG:teuthology.orchestra.run.gibba012:> for f in files: 2021-02-10T11:44:16.839 DEBUG:teuthology.orchestra.run.gibba012:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:44:16.839 DEBUG:teuthology.orchestra.run.gibba012:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:44:16.839 DEBUG:teuthology.orchestra.run.gibba012:> return f 2021-02-10T11:44:16.839 DEBUG:teuthology.orchestra.run.gibba012:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:44:16.839 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:44:16.839 DEBUG:teuthology.orchestra.run.gibba012:> print(find_socket("client.1")) 2021-02-10T11:44:16.839 DEBUG:teuthology.orchestra.run.gibba012:> ' 2021-02-10T11:44:17.201 INFO:teuthology.orchestra.run.gibba012.stdout:/var/run/ceph/ceph-client.1.173571.asok 2021-02-10T11:44:17.203 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.173571.asok 2021-02-10T11:44:17.203 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:44:17.204 DEBUG:teuthology.orchestra.run.gibba012:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.173571.asok status 2021-02-10T11:44:17.593 INFO:teuthology.orchestra.run.gibba012.stdout:{ 2021-02-10T11:44:17.593 INFO:teuthology.orchestra.run.gibba012.stdout: "metadata": { 2021-02-10T11:44:17.593 INFO:teuthology.orchestra.run.gibba012.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:44:17.594 INFO:teuthology.orchestra.run.gibba012.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:44:17.594 INFO:teuthology.orchestra.run.gibba012.stdout: "entity_id": "1", 2021-02-10T11:44:17.594 INFO:teuthology.orchestra.run.gibba012.stdout: "hostname": "gibba012", 2021-02-10T11:44:17.594 INFO:teuthology.orchestra.run.gibba012.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2021-02-10T11:44:17.594 INFO:teuthology.orchestra.run.gibba012.stdout: "pid": "173571", 2021-02-10T11:44:17.594 INFO:teuthology.orchestra.run.gibba012.stdout: "root": "/" 2021-02-10T11:44:17.594 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:44:17.594 INFO:teuthology.orchestra.run.gibba012.stdout: "dentry_count": 0, 2021-02-10T11:44:17.594 INFO:teuthology.orchestra.run.gibba012.stdout: "dentry_pinned_count": 0, 2021-02-10T11:44:17.594 INFO:teuthology.orchestra.run.gibba012.stdout: "id": 11883, 2021-02-10T11:44:17.594 INFO:teuthology.orchestra.run.gibba012.stdout: "inst": { 2021-02-10T11:44:17.594 INFO:teuthology.orchestra.run.gibba012.stdout: "name": { 2021-02-10T11:44:17.594 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "client", 2021-02-10T11:44:17.595 INFO:teuthology.orchestra.run.gibba012.stdout: "num": 11883 2021-02-10T11:44:17.595 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:44:17.595 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": { 2021-02-10T11:44:17.595 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:44:17.595 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:0", 2021-02-10T11:44:17.595 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 2932951995 2021-02-10T11:44:17.595 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:44:17.595 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:44:17.595 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": { 2021-02-10T11:44:17.595 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:44:17.595 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:0", 2021-02-10T11:44:17.595 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 2932951995 2021-02-10T11:44:17.595 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:44:17.595 INFO:teuthology.orchestra.run.gibba012.stdout: "inst_str": "client.11883 172.21.2.112:0/2932951995", 2021-02-10T11:44:17.596 INFO:teuthology.orchestra.run.gibba012.stdout: "addr_str": "172.21.2.112:0/2932951995", 2021-02-10T11:44:17.596 INFO:teuthology.orchestra.run.gibba012.stdout: "inode_count": 1, 2021-02-10T11:44:17.596 INFO:teuthology.orchestra.run.gibba012.stdout: "mds_epoch": 122, 2021-02-10T11:44:17.596 INFO:teuthology.orchestra.run.gibba012.stdout: "osd_epoch": 143, 2021-02-10T11:44:17.596 INFO:teuthology.orchestra.run.gibba012.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:44:17.596 INFO:teuthology.orchestra.run.gibba012.stdout: "blacklisted": false 2021-02-10T11:44:17.596 INFO:teuthology.orchestra.run.gibba012.stdout:} 2021-02-10T11:44:17.597 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:44:17.597 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:44:17.597 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:44:17.933 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:44:17.933 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:44:17.934 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:44:17.934 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:44:18.280 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2021-02-10T11:44:18.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:18.694 7f72a3599700 1 Processor -- start 2021-02-10T11:44:18.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:18.694 7f72a3599700 1 -- start start 2021-02-10T11:44:18.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:18.694 7f72a3599700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f729c0d1880 0x7f729c1b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:18.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:18.694 7f72a3599700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f729c1b0af0 0x7f729c108950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:18.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:18.694 7f72a3599700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f729c0d1c80 0x7f729c108f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:18.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:18.694 7f72a3599700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f729c0e5f00 con 0x7f729c0d1c80 2021-02-10T11:44:18.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:18.694 7f72a3599700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f729c109a70 con 0x7f729c1b0af0 2021-02-10T11:44:18.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:18.694 7f72a3599700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f729c088070 con 0x7f729c0d1880 2021-02-10T11:44:18.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:18.694 7f72a1bb6700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f729c0d1c80 0x7f729c108f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:18.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:18.694 7f72a1bb6700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f729c0d1c80 0x7f729c108f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:57784/0 (socket says 172.21.2.112:57784) 2021-02-10T11:44:18.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:18.694 7f72a1bb6700 1 -- 172.21.2.112:0/300890048 learned_addr learned my addr 172.21.2.112:0/300890048 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:44:18.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:18.695 7f72a1bb6700 1 -- 172.21.2.112:0/300890048 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f729c0d1880 msgr2=0x7f729c1b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:18.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:18.695 7f72a1bb6700 1 --2- 172.21.2.112:0/300890048 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f729c0d1880 0x7f729c1b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:18.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:18.695 7f72a1bb6700 1 -- 172.21.2.112:0/300890048 >> v2:172.21.2.120:3300/0 conn(0x7f729c1b0af0 msgr2=0x7f729c108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:18.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:18.695 7f72a1bb6700 1 --2- 172.21.2.112:0/300890048 >> v2:172.21.2.120:3300/0 conn(0x7f729c1b0af0 0x7f729c108950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:18.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:18.695 7f72a1bb6700 1 -- 172.21.2.112:0/300890048 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f729c109590 con 0x7f729c0d1c80 2021-02-10T11:44:18.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:18.695 7f72a1bb6700 1 --2- 172.21.2.112:0/300890048 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f729c0d1c80 0x7f729c108f70 secure :-1 s=READY pgs=2590 cs=0 l=1 rev1=1 rx=0x7f7294006f00 tx=0x7f7294009f10).ready entity=mon.0 client_cookie=64c214bcf0b8653b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:18.714 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:18.695 7f7293fff700 1 -- 172.21.2.112:0/300890048 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f729400b3b0 con 0x7f729c0d1c80 2021-02-10T11:44:18.714 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:18.695 7f7293fff700 1 -- 172.21.2.112:0/300890048 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f729400b630 con 0x7f729c0d1c80 2021-02-10T11:44:18.714 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:18.695 7f7293fff700 1 -- 172.21.2.112:0/300890048 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f729400bb50 con 0x7f729c0d1c80 2021-02-10T11:44:18.714 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:18.695 7f72a3599700 1 -- 172.21.2.112:0/300890048 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f729c0d1c80 msgr2=0x7f729c108f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:18.714 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:18.695 7f72a3599700 1 --2- 172.21.2.112:0/300890048 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f729c0d1c80 0x7f729c108f70 secure :-1 s=READY pgs=2590 cs=0 l=1 rev1=1 rx=0x7f7294006f00 tx=0x7f7294009f10).stop 2021-02-10T11:44:18.714 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:18.696 7f72a3599700 1 -- 172.21.2.112:0/300890048 shutdown_connections 2021-02-10T11:44:18.714 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:18.696 7f72a3599700 1 --2- 172.21.2.112:0/300890048 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f729c0d1c80 0x7f729c108f70 unknown :-1 s=CLOSED pgs=2590 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:18.714 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:18.696 7f72a3599700 1 --2- 172.21.2.112:0/300890048 >> v2:172.21.2.120:3300/0 conn(0x7f729c1b0af0 0x7f729c108950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:18.714 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:18.696 7f72a3599700 1 --2- 172.21.2.112:0/300890048 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f729c0d1880 0x7f729c1b04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:18.714 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:18.696 7f72a3599700 1 -- 172.21.2.112:0/300890048 >> 172.21.2.112:0/300890048 conn(0x7f729c0ce430 msgr2=0x7f729c0d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:18.714 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:18.696 7f72a3599700 1 -- 172.21.2.112:0/300890048 shutdown_connections 2021-02-10T11:44:18.714 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:18.696 7f72a3599700 1 -- 172.21.2.112:0/300890048 wait complete. 2021-02-10T11:44:18.716 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:18.711 7f72a3599700 1 Processor -- start 2021-02-10T11:44:18.716 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:18.711 7f72a3599700 1 -- start start 2021-02-10T11:44:18.716 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:18.711 7f72a3599700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f729c0aee20 0x7f729c1b5370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:18.716 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:18.711 7f72a3599700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f729c1b5990 0x7f729c1a85b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:18.716 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:18.711 7f72a3599700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f729c1a8bd0 0x7f729c1b4160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:18.716 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:18.711 7f72a3599700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f729c1b73f0 con 0x7f729c1a8bd0 2021-02-10T11:44:18.716 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:18.711 7f72a3599700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f729c1b4780 con 0x7f729c0aee20 2021-02-10T11:44:18.716 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:18.711 7f72a3599700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f729c1b4960 con 0x7f729c1b5990 2021-02-10T11:44:18.716 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:18.711 7f72a1bb6700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f729c1a8bd0 0x7f729c1b4160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:18.716 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:18.711 7f72a1bb6700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f729c1a8bd0 0x7f729c1b4160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:57792/0 (socket says 172.21.2.112:57792) 2021-02-10T11:44:18.716 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:18.711 7f72a1bb6700 1 -- 172.21.2.112:0/2531955539 learned_addr learned my addr 172.21.2.112:0/2531955539 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:44:18.717 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:18.712 7f72a1bb6700 1 -- 172.21.2.112:0/2531955539 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f729c1b5990 msgr2=0x7f729c1a85b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:44:18.717 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:18.712 7f72a1bb6700 1 --2- 172.21.2.112:0/2531955539 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f729c1b5990 0x7f729c1a85b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:18.717 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:18.712 7f72a1bb6700 1 -- 172.21.2.112:0/2531955539 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f729c0aee20 msgr2=0x7f729c1b5370 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:44:18.717 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:18.712 7f72a1bb6700 1 --2- 172.21.2.112:0/2531955539 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f729c0aee20 0x7f729c1b5370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:18.717 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:18.712 7f72a1bb6700 1 -- 172.21.2.112:0/2531955539 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f729c1b4b80 con 0x7f729c1a8bd0 2021-02-10T11:44:18.717 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:18.712 7f72a1bb6700 1 --2- 172.21.2.112:0/2531955539 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f729c1a8bd0 0x7f729c1b4160 secure :-1 s=READY pgs=2591 cs=0 l=1 rev1=1 rx=0x7f7294009390 tx=0x7f729400bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:18.717 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:18.712 7f7291ffb700 1 -- 172.21.2.112:0/2531955539 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7294003030 con 0x7f729c1a8bd0 2021-02-10T11:44:18.717 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:18.712 7f7291ffb700 1 -- 172.21.2.112:0/2531955539 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72940068e0 con 0x7f729c1a8bd0 2021-02-10T11:44:18.717 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:18.712 7f7291ffb700 1 -- 172.21.2.112:0/2531955539 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f729400c570 con 0x7f729c1a8bd0 2021-02-10T11:44:18.717 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:18.712 7f72a3599700 1 -- 172.21.2.112:0/2531955539 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f729c1b4b80 con 0x7f729c1a8bd0 2021-02-10T11:44:18.717 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:18.712 7f72a3599700 1 -- 172.21.2.112:0/2531955539 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f729c1b5f50 con 0x7f729c1a8bd0 2021-02-10T11:44:18.717 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:18.713 7f7291ffb700 1 -- 172.21.2.112:0/2531955539 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7294001270 con 0x7f729c1a8bd0 2021-02-10T11:44:18.718 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:18.713 7f7291ffb700 1 --2- 172.21.2.112:0/2531955539 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f727c0186c0 0x7f727c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:18.718 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:18.713 7f7291ffb700 1 -- 172.21.2.112:0/2531955539 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f7294032900 con 0x7f729c1a8bd0 2021-02-10T11:44:18.718 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:18.714 7f72a13b5700 1 --2- 172.21.2.112:0/2531955539 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f727c0186c0 0x7f727c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:18.718 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:18.715 7f72a13b5700 1 --2- 172.21.2.112:0/2531955539 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f727c0186c0 0x7f727c01abc0 crc :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:18.746 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:18.744 7f72a3599700 1 -- 172.21.2.112:0/2531955539 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f729c082a30 con 0x7f729c1a8bd0 2021-02-10T11:44:18.748 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:18.747 7f7291ffb700 1 -- 172.21.2.112:0/2531955539 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7294009450 con 0x7f729c1a8bd0 2021-02-10T11:44:18.886 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:18.884 7f72a3599700 1 -- 172.21.2.112:0/2531955539 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f729c109a70 con 0x7f729c1a8bd0 2021-02-10T11:44:19.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:19.241 7f7291ffb700 1 -- 172.21.2.112:0/2531955539 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v123) v1 ==== 136+0+0 (secure 0 0 0) 0x7f7294001270 con 0x7f729c1a8bd0 2021-02-10T11:44:19.254 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:19.253 7f72a3599700 1 -- 172.21.2.112:0/2531955539 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f727c0186c0 msgr2=0x7f727c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:19.254 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:19.253 7f72a3599700 1 --2- 172.21.2.112:0/2531955539 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f727c0186c0 0x7f727c01abc0 crc :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:19.254 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:19.253 7f72a3599700 1 -- 172.21.2.112:0/2531955539 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f729c1a8bd0 msgr2=0x7f729c1b4160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:19.255 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:19.253 7f72a3599700 1 --2- 172.21.2.112:0/2531955539 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f729c1a8bd0 0x7f729c1b4160 secure :-1 s=READY pgs=2591 cs=0 l=1 rev1=1 rx=0x7f7294009390 tx=0x7f729400bd40).stop 2021-02-10T11:44:19.255 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:19.253 7f72a3599700 1 -- 172.21.2.112:0/2531955539 shutdown_connections 2021-02-10T11:44:19.255 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:19.253 7f72a3599700 1 --2- 172.21.2.112:0/2531955539 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f727c0186c0 0x7f727c01abc0 unknown :-1 s=CLOSED pgs=1185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:19.255 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:19.253 7f72a3599700 1 --2- 172.21.2.112:0/2531955539 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f729c1a8bd0 0x7f729c1b4160 unknown :-1 s=CLOSED pgs=2591 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:19.255 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:19.253 7f72a3599700 1 --2- 172.21.2.112:0/2531955539 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f729c1b5990 0x7f729c1a85b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:19.255 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:19.253 7f72a3599700 1 --2- 172.21.2.112:0/2531955539 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f729c0aee20 0x7f729c1b5370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:19.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:19.253 7f72a3599700 1 -- 172.21.2.112:0/2531955539 >> 172.21.2.112:0/2531955539 conn(0x7f729c19e990 msgr2=0x7f729c1918e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:19.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:19.253 7f72a3599700 1 -- 172.21.2.112:0/2531955539 shutdown_connections 2021-02-10T11:44:19.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:19.253 7f72a3599700 1 -- 172.21.2.112:0/2531955539 wait complete. 2021-02-10T11:44:19.263 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs volume ls 2021-02-10T11:44:19.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:19.672 7f7417a04700 1 Processor -- start 2021-02-10T11:44:19.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:19.672 7f7417a04700 1 -- start start 2021-02-10T11:44:19.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:19.672 7f7417a04700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f74100d1c70 0x7f74100f20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:19.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:19.672 7f7417a04700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74100f0b10 con 0x7f74100d1870 2021-02-10T11:44:19.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:19.672 7f7417a04700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74101b0be0 con 0x7f74100d0340 2021-02-10T11:44:19.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:19.672 7f7417a04700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74100e5ed0 con 0x7f74100d1c70 2021-02-10T11:44:19.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:19.672 7f7415820700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f74100d1870 0x7f74101b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:57258/0 (socket says 172.21.2.112:57258) 2021-02-10T11:44:19.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:19.672 7f7415820700 1 -- 172.21.2.112:0/2457014110 learned_addr learned my addr 172.21.2.112:0/2457014110 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:44:19.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:19.673 7f741481e700 1 -- 172.21.2.112:0/2457014110 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1207370977 0 0) 0x7f740c0010b0 con 0x7f74100d1870 2021-02-10T11:44:19.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:19.673 7f741481e700 1 -- 172.21.2.112:0/2457014110 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73fc002fe0 con 0x7f74100d1870 2021-02-10T11:44:19.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:19.673 7f7416021700 1 --2- 172.21.2.112:0/2457014110 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f74100d1c70 0x7f74100f20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:19.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:19.673 7f741481e700 1 -- 172.21.2.112:0/2457014110 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2037307492 0 0) 0x7f740c0010b0 con 0x7f74100d1870 2021-02-10T11:44:19.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:19.673 7f741481e700 1 -- 172.21.2.112:0/2457014110 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f74100d1c70 msgr2=0x7f74100f20a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:19.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:19.673 7f741481e700 1 --2- 172.21.2.112:0/2457014110 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f74100d1c70 0x7f74100f20a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:19.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:19.673 7f741481e700 1 -- 172.21.2.112:0/2457014110 >> v1:172.21.2.120:6789/0 conn(0x7f74100d0340 legacy=0x7f74100ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:19.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:19.673 7f741481e700 1 -- 172.21.2.112:0/2457014110 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74100efe30 con 0x7f74100d1870 2021-02-10T11:44:19.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:19.673 7f741481e700 1 -- 172.21.2.112:0/2457014110 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f740c0015b0 con 0x7f74100d1870 2021-02-10T11:44:19.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:19.673 7f741481e700 1 -- 172.21.2.112:0/2457014110 >> v1:172.21.2.112:6789/0 conn(0x7f74100d1870 legacy=0x7f74101b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:19.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:19.673 7f741481e700 1 --2- 172.21.2.112:0/2457014110 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f73fc003aa0 0x7f73fc005eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:19.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:19.673 7f741481e700 1 --2- 172.21.2.112:0/2457014110 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f73fc006750 0x7f73fc008c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:19.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:19.673 7f741481e700 1 --2- 172.21.2.112:0/2457014110 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f73fc0094f0 0x7f73fc00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:19.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:19.673 7f741481e700 1 -- 172.21.2.112:0/2457014110 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73fc00c2c0 con 0x7f73fc006750 2021-02-10T11:44:19.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:19.673 7f741481e700 1 -- 172.21.2.112:0/2457014110 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73fc00c4e0 con 0x7f73fc0094f0 2021-02-10T11:44:19.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:19.673 7f741481e700 1 -- 172.21.2.112:0/2457014110 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73fc00c700 con 0x7f73fc003aa0 2021-02-10T11:44:19.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:19.674 7f741501f700 1 --2- 172.21.2.112:0/2457014110 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f73fc006750 0x7f73fc008c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:19.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:19.674 7f7415820700 1 --2- 172.21.2.112:0/2457014110 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f73fc003aa0 0x7f73fc005eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:19.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:19.674 7f7416021700 1 --2- 172.21.2.112:0/2457014110 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f73fc0094f0 0x7f73fc00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:19.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:19.674 7f741501f700 1 -- 172.21.2.112:0/2457014110 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f73fc003aa0 msgr2=0x7f73fc005eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:19.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:19.674 7f741501f700 1 --2- 172.21.2.112:0/2457014110 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f73fc003aa0 0x7f73fc005eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:19.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:19.674 7f741501f700 1 -- 172.21.2.112:0/2457014110 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f73fc0094f0 msgr2=0x7f73fc00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:19.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:19.674 7f741501f700 1 --2- 172.21.2.112:0/2457014110 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f73fc0094f0 0x7f73fc00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:19.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:19.674 7f741501f700 1 -- 172.21.2.112:0/2457014110 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73fc050940 con 0x7f73fc006750 2021-02-10T11:44:19.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:19.674 7f7416021700 1 --2- 172.21.2.112:0/2457014110 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f73fc0094f0 0x7f73fc00ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:44:19.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:19.674 7f741501f700 1 --2- 172.21.2.112:0/2457014110 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f73fc006750 0x7f73fc008c50 secure :-1 s=READY pgs=2593 cs=0 l=1 rev1=1 rx=0x7f7404006ac0 tx=0x7f7404009a70).ready entity=mon.0 client_cookie=9d77f83c68b7240e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:19.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:19.674 7f741481e700 1 -- 172.21.2.112:0/2457014110 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7404007280 con 0x7f73fc006750 2021-02-10T11:44:19.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:19.674 7f741481e700 1 -- 172.21.2.112:0/2457014110 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7404007580 con 0x7f73fc006750 2021-02-10T11:44:19.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:19.674 7f741481e700 1 -- 172.21.2.112:0/2457014110 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7404007280 con 0x7f73fc006750 2021-02-10T11:44:19.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:19.674 7f7417a04700 1 -- 172.21.2.112:0/2457014110 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f73fc006750 msgr2=0x7f73fc008c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:19.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:19.674 7f7417a04700 1 --2- 172.21.2.112:0/2457014110 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f73fc006750 0x7f73fc008c50 secure :-1 s=READY pgs=2593 cs=0 l=1 rev1=1 rx=0x7f7404006ac0 tx=0x7f7404009a70).stop 2021-02-10T11:44:19.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:19.674 7f7416021700 1 -- 172.21.2.112:0/2457014110 reap_dead start 2021-02-10T11:44:19.695 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:19.675 7f7417a04700 1 -- 172.21.2.112:0/2457014110 shutdown_connections 2021-02-10T11:44:19.695 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:19.675 7f7417a04700 1 -- 172.21.2.112:0/2457014110 >> 172.21.2.112:0/2457014110 conn(0x7f74100ce450 msgr2=0x7f74100d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:19.695 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:19.675 7f7417a04700 1 -- 172.21.2.112:0/2457014110 shutdown_connections 2021-02-10T11:44:19.695 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:19.675 7f7417a04700 1 -- 172.21.2.112:0/2457014110 wait complete. 2021-02-10T11:44:19.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:19.690 7f7417a04700 1 Processor -- start 2021-02-10T11:44:19.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:19.690 7f7417a04700 1 -- start start 2021-02-10T11:44:19.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:19.690 7f7417a04700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f741008e8f0 0x7f741008ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:19.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:19.690 7f7417a04700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f74100ebe80 0x7f74100ec280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:19.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:19.690 7f7417a04700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f74100ec8a0 0x7f74100ed650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:19.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:19.690 7f7417a04700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74100a0a50 con 0x7f741008e8f0 2021-02-10T11:44:19.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:19.690 7f7417a04700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f741008f310 con 0x7f74100ebe80 2021-02-10T11:44:19.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:19.690 7f7417a04700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f741008f530 con 0x7f74100ec8a0 2021-02-10T11:44:19.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:19.690 7f7415820700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f741008e8f0 0x7f741008ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:19.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:19.690 7f7415820700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f741008e8f0 0x7f741008ecf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:57810/0 (socket says 172.21.2.112:57810) 2021-02-10T11:44:19.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:19.690 7f7415820700 1 -- 172.21.2.112:0/4059201468 learned_addr learned my addr 172.21.2.112:0/4059201468 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:44:19.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:19.691 7f7415820700 1 -- 172.21.2.112:0/4059201468 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f74100ec8a0 msgr2=0x7f74100ed650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:19.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:19.691 7f7415820700 1 --2- 172.21.2.112:0/4059201468 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f74100ec8a0 0x7f74100ed650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:19.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:19.691 7f7415820700 1 -- 172.21.2.112:0/4059201468 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f74100ebe80 msgr2=0x7f74100ec280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:19.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:19.691 7f7415820700 1 --2- 172.21.2.112:0/4059201468 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f74100ebe80 0x7f74100ec280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:19.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:19.691 7f7415820700 1 -- 172.21.2.112:0/4059201468 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74100ecca0 con 0x7f741008e8f0 2021-02-10T11:44:19.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:19.691 7f7415820700 1 --2- 172.21.2.112:0/4059201468 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f741008e8f0 0x7f741008ecf0 secure :-1 s=READY pgs=2594 cs=0 l=1 rev1=1 rx=0x7f740c007650 tx=0x7f740c00a900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:19.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:19.691 7f74027fc700 1 -- 172.21.2.112:0/4059201468 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f740c00acf0 con 0x7f741008e8f0 2021-02-10T11:44:19.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:19.691 7f74027fc700 1 -- 172.21.2.112:0/4059201468 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f740c004140 con 0x7f741008e8f0 2021-02-10T11:44:19.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:19.691 7f74027fc700 1 -- 172.21.2.112:0/4059201468 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f740c0045e0 con 0x7f741008e8f0 2021-02-10T11:44:19.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:19.691 7f7417a04700 1 -- 172.21.2.112:0/4059201468 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74100ecca0 con 0x7f741008e8f0 2021-02-10T11:44:19.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:19.691 7f7417a04700 1 -- 172.21.2.112:0/4059201468 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74100ecca0 con 0x7f741008e8f0 2021-02-10T11:44:19.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:19.692 7f74027fc700 1 -- 172.21.2.112:0/4059201468 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f740c004320 con 0x7f741008e8f0 2021-02-10T11:44:19.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:19.692 7f74027fc700 1 --2- 172.21.2.112:0/4059201468 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f73f80186c0 0x7f73f801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:19.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:19.692 7f74027fc700 1 -- 172.21.2.112:0/4059201468 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f740c032550 con 0x7f741008e8f0 2021-02-10T11:44:19.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:19.692 7f741501f700 1 --2- 172.21.2.112:0/4059201468 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f73f80186c0 0x7f73f801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:19.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:19.693 7f741501f700 1 --2- 172.21.2.112:0/4059201468 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f73f80186c0 0x7f73f801abc0 crc :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:19.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:19.723 7f7417a04700 1 -- 172.21.2.112:0/4059201468 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74100ebc50 con 0x7f741008e8f0 2021-02-10T11:44:19.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:19.725 7f74027fc700 1 -- 172.21.2.112:0/4059201468 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f740c001650 con 0x7f741008e8f0 2021-02-10T11:44:19.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:19.861 7f7417a04700 1 -- 172.21.2.112:0/4059201468 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs volume ls", "target": ["mgr", ""]}) v1 -- 0x7f741008f310 con 0x7f73f80186c0 2021-02-10T11:44:19.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:19.862 7f74027fc700 1 -- 172.21.2.112:0/4059201468 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+41 (crc 0 0 0) 0x7f7404006a90 con 0x7f73f80186c0 2021-02-10T11:44:19.866 INFO:teuthology.orchestra.run.gibba012.stdout:[ 2021-02-10T11:44:19.866 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:44:19.866 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "cephfs" 2021-02-10T11:44:19.866 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:44:19.866 INFO:teuthology.orchestra.run.gibba012.stdout:] 2021-02-10T11:44:19.868 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:19.866 7f7417a04700 1 -- 172.21.2.112:0/4059201468 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f73f80186c0 msgr2=0x7f73f801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:19.868 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:19.866 7f7417a04700 1 --2- 172.21.2.112:0/4059201468 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f73f80186c0 0x7f73f801abc0 crc :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:19.868 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:19.866 7f7417a04700 1 -- 172.21.2.112:0/4059201468 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f741008e8f0 msgr2=0x7f741008ecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:19.868 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:19.866 7f7417a04700 1 --2- 172.21.2.112:0/4059201468 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f741008e8f0 0x7f741008ecf0 secure :-1 s=READY pgs=2594 cs=0 l=1 rev1=1 rx=0x7f740c007650 tx=0x7f740c00a900).stop 2021-02-10T11:44:19.868 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:19.866 7f7417a04700 1 -- 172.21.2.112:0/4059201468 shutdown_connections 2021-02-10T11:44:19.868 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:19.866 7f7417a04700 1 --2- 172.21.2.112:0/4059201468 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f73f80186c0 0x7f73f801abc0 unknown :-1 s=CLOSED pgs=1186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:19.868 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:19.866 7f7417a04700 1 --2- 172.21.2.112:0/4059201468 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f74100ebe80 0x7f74100ec280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:19.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:19.866 7f7417a04700 1 --2- 172.21.2.112:0/4059201468 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f74100ec8a0 0x7f74100ed650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:19.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:19.866 7f7417a04700 1 --2- 172.21.2.112:0/4059201468 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f741008e8f0 0x7f741008ecf0 unknown :-1 s=CLOSED pgs=2594 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:19.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:19.866 7f7417a04700 1 -- 172.21.2.112:0/4059201468 >> 172.21.2.112:0/4059201468 conn(0x7f7410094d90 msgr2=0x7f741008a420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:19.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:19.866 7f7417a04700 1 -- 172.21.2.112:0/4059201468 shutdown_connections 2021-02-10T11:44:19.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:19.866 7f7417a04700 1 -- 172.21.2.112:0/4059201468 wait complete. 2021-02-10T11:44:19.876 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:44:20.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.292 7f084b3b8700 1 Processor -- start 2021-02-10T11:44:20.311 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.292 7f084b3b8700 1 -- start start 2021-02-10T11:44:20.311 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.292 7f084b3b8700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f08440d1860 0x7f08441b04b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:20.311 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.292 7f084b3b8700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f08440cff60 0x7f0844109120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:20.311 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.292 7f084b3b8700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0844109a50 con 0x7f08441b0ad0 2021-02-10T11:44:20.311 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.292 7f084b3b8700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0844109740 con 0x7f08440cff60 2021-02-10T11:44:20.311 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.292 7f084b3b8700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0844088070 con 0x7f08440d1860 2021-02-10T11:44:20.311 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.292 7f08489d3700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f08441b0ad0 0x7f0844108930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:57278/0 (socket says 172.21.2.112:57278) 2021-02-10T11:44:20.311 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.292 7f08489d3700 1 -- 172.21.2.112:0/1625284292 learned_addr learned my addr 172.21.2.112:0/1625284292 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:44:20.311 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.293 7f08499d5700 1 --2- 172.21.2.112:0/1625284292 >> v2:172.21.2.120:3300/0 conn(0x7f08440cff60 0x7f0844109120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:20.311 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.293 7f08491d4700 1 --2- 172.21.2.112:0/1625284292 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f08440d1860 0x7f08441b04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:20.311 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.293 7f083bfff700 1 -- 172.21.2.112:0/1625284292 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 359110366 0 0) 0x7f08340010b0 con 0x7f08441b0ad0 2021-02-10T11:44:20.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.293 7f083bfff700 1 -- 172.21.2.112:0/1625284292 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0830002fe0 con 0x7f08441b0ad0 2021-02-10T11:44:20.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.293 7f08499d5700 1 -- 172.21.2.112:0/1625284292 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f08440d1860 msgr2=0x7f08441b04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:20.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.293 7f08499d5700 1 --2- 172.21.2.112:0/1625284292 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f08440d1860 0x7f08441b04b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:20.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.293 7f08499d5700 1 -- 172.21.2.112:0/1625284292 >> v1:172.21.2.112:6789/0 conn(0x7f08441b0ad0 legacy=0x7f0844108930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:20.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.293 7f08499d5700 1 -- 172.21.2.112:0/1625284292 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0844088260 con 0x7f08440cff60 2021-02-10T11:44:20.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.293 7f08499d5700 1 --2- 172.21.2.112:0/1625284292 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f08440cff60 0x7f0844109120 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 rx=0x7f083c006e50 tx=0x7f083c009e70).ready entity=mon.1 client_cookie=48c649b20108b4b3 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:20.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.293 7f083bfff700 1 -- 172.21.2.112:0/1625284292 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f083c00b510 con 0x7f08440cff60 2021-02-10T11:44:20.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.293 7f083bfff700 1 -- 172.21.2.112:0/1625284292 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f083c006790 con 0x7f08440cff60 2021-02-10T11:44:20.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.293 7f083bfff700 1 -- 172.21.2.112:0/1625284292 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f083c00b940 con 0x7f08440cff60 2021-02-10T11:44:20.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.293 7f084b3b8700 1 -- 172.21.2.112:0/1625284292 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f08440cff60 msgr2=0x7f0844109120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:20.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.293 7f084b3b8700 1 --2- 172.21.2.112:0/1625284292 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f08440cff60 0x7f0844109120 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 rx=0x7f083c006e50 tx=0x7f083c009e70).stop 2021-02-10T11:44:20.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.294 7f084b3b8700 1 -- 172.21.2.112:0/1625284292 shutdown_connections 2021-02-10T11:44:20.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.294 7f084b3b8700 1 --2- 172.21.2.112:0/1625284292 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f08440cff60 0x7f0844109120 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:20.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.294 7f084b3b8700 1 --2- 172.21.2.112:0/1625284292 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f08440d1860 0x7f08441b04b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:20.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.294 7f084b3b8700 1 -- 172.21.2.112:0/1625284292 >> 172.21.2.112:0/1625284292 conn(0x7f08440ce410 msgr2=0x7f08440d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:20.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.294 7f084b3b8700 1 -- 172.21.2.112:0/1625284292 shutdown_connections 2021-02-10T11:44:20.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.294 7f084b3b8700 1 -- 172.21.2.112:0/1625284292 wait complete. 2021-02-10T11:44:20.314 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.309 7f084b3b8700 1 Processor -- start 2021-02-10T11:44:20.314 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.309 7f084b3b8700 1 -- start start 2021-02-10T11:44:20.314 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.309 7f084b3b8700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0844094d20 0x7f08440a3e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:20.314 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.309 7f084b3b8700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f08440a4460 0x7f08440a4860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:20.314 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.309 7f084b3b8700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f08440bf9e0 0x7f08440bfde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:20.314 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.309 7f084b3b8700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08440a7560 con 0x7f0844094d20 2021-02-10T11:44:20.314 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.309 7f084b3b8700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08440b0bc0 con 0x7f08440bf9e0 2021-02-10T11:44:20.315 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.309 7f084b3b8700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0844095120 con 0x7f08440a4460 2021-02-10T11:44:20.315 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.310 7f08491d4700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0844094d20 0x7f08440a3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:20.315 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.310 7f08491d4700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0844094d20 0x7f08440a3e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:57822/0 (socket says 172.21.2.112:57822) 2021-02-10T11:44:20.315 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.310 7f08491d4700 1 -- 172.21.2.112:0/3396362508 learned_addr learned my addr 172.21.2.112:0/3396362508 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:44:20.315 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.310 7f08491d4700 1 -- 172.21.2.112:0/3396362508 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f08440a4460 msgr2=0x7f08440a4860 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:44:20.315 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.310 7f08491d4700 1 --2- 172.21.2.112:0/3396362508 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f08440a4460 0x7f08440a4860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:20.315 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.310 7f08491d4700 1 -- 172.21.2.112:0/3396362508 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f08440bf9e0 msgr2=0x7f08440bfde0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:44:20.315 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.310 7f08491d4700 1 --2- 172.21.2.112:0/3396362508 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f08440bf9e0 0x7f08440bfde0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:20.315 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.310 7f08491d4700 1 -- 172.21.2.112:0/3396362508 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08440c0400 con 0x7f0844094d20 2021-02-10T11:44:20.315 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.310 7f08491d4700 1 --2- 172.21.2.112:0/3396362508 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0844094d20 0x7f08440a3e40 secure :-1 s=READY pgs=2596 cs=0 l=1 rev1=1 rx=0x7f0840006f60 tx=0x7f084000b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:20.315 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.310 7f0839ffb700 1 -- 172.21.2.112:0/3396362508 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f084000bd60 con 0x7f0844094d20 2021-02-10T11:44:20.315 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.310 7f0839ffb700 1 -- 172.21.2.112:0/3396362508 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0840000c90 con 0x7f0844094d20 2021-02-10T11:44:20.315 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.310 7f084b3b8700 1 -- 172.21.2.112:0/3396362508 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08440c0400 con 0x7f0844094d20 2021-02-10T11:44:20.316 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.310 7f084b3b8700 1 -- 172.21.2.112:0/3396362508 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08440c0990 con 0x7f0844094d20 2021-02-10T11:44:20.316 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.310 7f0839ffb700 1 -- 172.21.2.112:0/3396362508 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0840002220 con 0x7f0844094d20 2021-02-10T11:44:20.316 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.311 7f0839ffb700 1 -- 172.21.2.112:0/3396362508 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f08400052e0 con 0x7f0844094d20 2021-02-10T11:44:20.316 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.311 7f0839ffb700 1 --2- 172.21.2.112:0/3396362508 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f08240186c0 0x7f082401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:20.316 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.311 7f0839ffb700 1 -- 172.21.2.112:0/3396362508 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f0840032560 con 0x7f0844094d20 2021-02-10T11:44:20.316 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.311 7f08489d3700 1 --2- 172.21.2.112:0/3396362508 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f08240186c0 0x7f082401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:20.316 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.314 7f08489d3700 1 --2- 172.21.2.112:0/3396362508 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f08240186c0 0x7f082401abc0 crc :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:20.344 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.342 7f084b3b8700 1 -- 172.21.2.112:0/3396362508 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0844082920 con 0x7f0844094d20 2021-02-10T11:44:20.347 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.346 7f0839ffb700 1 -- 172.21.2.112:0/3396362508 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f084002dad0 con 0x7f0844094d20 2021-02-10T11:44:20.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.495 7f084b3b8700 1 -- 172.21.2.112:0/3396362508 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f0844109a50 con 0x7f0844094d20 2021-02-10T11:44:20.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.495 7f0839ffb700 1 -- 172.21.2.112:0/3396362508 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v36)=0 v36) v1 ==== 122+0+0 (secure 0 0 0) 0x7f084002dad0 con 0x7f0844094d20 2021-02-10T11:44:20.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.497 7f084b3b8700 1 -- 172.21.2.112:0/3396362508 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f08240186c0 msgr2=0x7f082401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:20.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.497 7f084b3b8700 1 --2- 172.21.2.112:0/3396362508 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f08240186c0 0x7f082401abc0 crc :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:20.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.497 7f084b3b8700 1 -- 172.21.2.112:0/3396362508 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0844094d20 msgr2=0x7f08440a3e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:20.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.497 7f084b3b8700 1 --2- 172.21.2.112:0/3396362508 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0844094d20 0x7f08440a3e40 secure :-1 s=READY pgs=2596 cs=0 l=1 rev1=1 rx=0x7f0840006f60 tx=0x7f084000b970).stop 2021-02-10T11:44:20.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.497 7f084b3b8700 1 -- 172.21.2.112:0/3396362508 shutdown_connections 2021-02-10T11:44:20.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.497 7f084b3b8700 1 --2- 172.21.2.112:0/3396362508 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f08240186c0 0x7f082401abc0 unknown :-1 s=CLOSED pgs=1187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:20.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.497 7f084b3b8700 1 --2- 172.21.2.112:0/3396362508 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f08440bf9e0 0x7f08440bfde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:20.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.497 7f084b3b8700 1 --2- 172.21.2.112:0/3396362508 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f08440a4460 0x7f08440a4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:20.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.497 7f084b3b8700 1 --2- 172.21.2.112:0/3396362508 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0844094d20 0x7f08440a3e40 unknown :-1 s=CLOSED pgs=2596 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:20.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.497 7f084b3b8700 1 -- 172.21.2.112:0/3396362508 >> 172.21.2.112:0/3396362508 conn(0x7f08440c3e40 msgr2=0x7f08440916a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:20.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.498 7f084b3b8700 1 -- 172.21.2.112:0/3396362508 shutdown_connections 2021-02-10T11:44:20.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.498 7f084b3b8700 1 -- 172.21.2.112:0/3396362508 wait complete. 2021-02-10T11:44:20.507 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolumegroup rm cephfs non_existent_group 2021-02-10T11:44:20.938 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.920 7f9e3be21700 1 Processor -- start 2021-02-10T11:44:20.938 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.920 7f9e3be21700 1 -- start start 2021-02-10T11:44:20.938 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.920 7f9e3be21700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9e340d1870 0x7f9e341b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:20.939 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.920 7f9e3be21700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f9e341b0ae0 0x7f9e34108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:20.939 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.920 7f9e3be21700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e34109a60 con 0x7f9e340d1c70 2021-02-10T11:44:20.939 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.920 7f9e3be21700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e340e5ef0 con 0x7f9e341b0ae0 2021-02-10T11:44:20.939 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.920 7f9e3be21700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e34109750 con 0x7f9e340d1870 2021-02-10T11:44:20.939 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.920 7f9e3943c700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f9e341b0ae0 0x7f9e34108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:20.939 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.920 7f9e3a43e700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f9e340d1c70 0x7f9e34108f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:57292/0 (socket says 172.21.2.112:57292) 2021-02-10T11:44:20.939 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.920 7f9e3a43e700 1 -- 172.21.2.112:0/1806191018 learned_addr learned my addr 172.21.2.112:0/1806191018 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:44:20.939 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.921 7f9e3943c700 1 -- 172.21.2.112:0/1806191018 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9e340d1870 msgr2=0x7f9e341b04c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:44:20.939 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.921 7f9e3943c700 1 --2- 172.21.2.112:0/1806191018 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9e340d1870 0x7f9e341b04c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:20.939 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.921 7f9e3943c700 1 -- 172.21.2.112:0/1806191018 >> v1:172.21.2.112:6789/0 conn(0x7f9e340d1c70 legacy=0x7f9e34108f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:20.939 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.921 7f9e3943c700 1 -- 172.21.2.112:0/1806191018 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e34088070 con 0x7f9e341b0ae0 2021-02-10T11:44:20.939 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.921 7f9e3943c700 1 --2- 172.21.2.112:0/1806191018 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9e341b0ae0 0x7f9e34108940 secure :-1 s=READY pgs=2598 cs=0 l=1 rev1=1 rx=0x7f9e28006650 tx=0x7f9e28009a00).ready entity=mon.0 client_cookie=f539e8c484dda3fa server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:20.939 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.921 7f9e38c3b700 1 -- 172.21.2.112:0/1806191018 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9e28009df0 con 0x7f9e341b0ae0 2021-02-10T11:44:20.940 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.921 7f9e38c3b700 1 -- 172.21.2.112:0/1806191018 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e280009b0 con 0x7f9e341b0ae0 2021-02-10T11:44:20.940 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.921 7f9e38c3b700 1 -- 172.21.2.112:0/1806191018 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9e2800c2b0 con 0x7f9e341b0ae0 2021-02-10T11:44:20.940 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.921 7f9e3be21700 1 -- 172.21.2.112:0/1806191018 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9e341b0ae0 msgr2=0x7f9e34108940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:20.940 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.921 7f9e3be21700 1 --2- 172.21.2.112:0/1806191018 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9e341b0ae0 0x7f9e34108940 secure :-1 s=READY pgs=2598 cs=0 l=1 rev1=1 rx=0x7f9e28006650 tx=0x7f9e28009a00).stop 2021-02-10T11:44:20.940 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.923 7f9e3be21700 1 -- 172.21.2.112:0/1806191018 shutdown_connections 2021-02-10T11:44:20.940 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.923 7f9e3be21700 1 --2- 172.21.2.112:0/1806191018 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9e341b0ae0 0x7f9e34108940 unknown :-1 s=CLOSED pgs=2598 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:20.940 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.923 7f9e3be21700 1 --2- 172.21.2.112:0/1806191018 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9e340d1870 0x7f9e341b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:20.940 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.923 7f9e3be21700 1 -- 172.21.2.112:0/1806191018 >> 172.21.2.112:0/1806191018 conn(0x7f9e340ce420 msgr2=0x7f9e340d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:20.940 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.923 7f9e3be21700 1 -- 172.21.2.112:0/1806191018 shutdown_connections 2021-02-10T11:44:20.940 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.923 7f9e3be21700 1 -- 172.21.2.112:0/1806191018 wait complete. 2021-02-10T11:44:20.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.937 7f9e3be21700 1 Processor -- start 2021-02-10T11:44:20.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.937 7f9e3be21700 1 -- start start 2021-02-10T11:44:20.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.937 7f9e3be21700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9e340c3820 0x7f9e340a7950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:20.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.937 7f9e3be21700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9e341a44c0 0x7f9e341a48c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:20.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.937 7f9e3be21700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9e340863d0 0x7f9e340867d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:20.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.937 7f9e3be21700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e34085ed0 con 0x7f9e340c3820 2021-02-10T11:44:20.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.937 7f9e3be21700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e340c0740 con 0x7f9e341a44c0 2021-02-10T11:44:20.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.937 7f9e3be21700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e34086df0 con 0x7f9e340863d0 2021-02-10T11:44:20.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.939 7f9e39c3d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9e340c3820 0x7f9e340a7950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:20.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.939 7f9e39c3d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9e340c3820 0x7f9e340a7950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:57832/0 (socket says 172.21.2.112:57832) 2021-02-10T11:44:20.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.939 7f9e39c3d700 1 -- 172.21.2.112:0/1123543157 learned_addr learned my addr 172.21.2.112:0/1123543157 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:44:20.943 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.939 7f9e39c3d700 1 -- 172.21.2.112:0/1123543157 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9e340863d0 msgr2=0x7f9e340867d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:20.943 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.939 7f9e39c3d700 1 --2- 172.21.2.112:0/1123543157 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9e340863d0 0x7f9e340867d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:20.943 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.939 7f9e39c3d700 1 -- 172.21.2.112:0/1123543157 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9e341a44c0 msgr2=0x7f9e341a48c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:20.943 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.939 7f9e39c3d700 1 --2- 172.21.2.112:0/1123543157 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9e341a44c0 0x7f9e341a48c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:20.943 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.939 7f9e39c3d700 1 -- 172.21.2.112:0/1123543157 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e34087010 con 0x7f9e340c3820 2021-02-10T11:44:20.943 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.939 7f9e39c3d700 1 --2- 172.21.2.112:0/1123543157 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9e340c3820 0x7f9e340a7950 secure :-1 s=READY pgs=2599 cs=0 l=1 rev1=1 rx=0x7f9e3000b5c0 tx=0x7f9e3000b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:20.943 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.939 7f9e267fc700 1 -- 172.21.2.112:0/1123543157 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9e30006b80 con 0x7f9e340c3820 2021-02-10T11:44:20.943 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.939 7f9e267fc700 1 -- 172.21.2.112:0/1123543157 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e3000bd50 con 0x7f9e340c3820 2021-02-10T11:44:20.943 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.939 7f9e3be21700 1 -- 172.21.2.112:0/1123543157 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e34087010 con 0x7f9e340c3820 2021-02-10T11:44:20.943 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.939 7f9e267fc700 1 -- 172.21.2.112:0/1123543157 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9e300054d0 con 0x7f9e340c3820 2021-02-10T11:44:20.943 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.940 7f9e3be21700 1 -- 172.21.2.112:0/1123543157 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e34087010 con 0x7f9e340c3820 2021-02-10T11:44:20.944 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.940 7f9e267fc700 1 -- 172.21.2.112:0/1123543157 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9e30006030 con 0x7f9e340c3820 2021-02-10T11:44:20.944 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.940 7f9e267fc700 1 --2- 172.21.2.112:0/1123543157 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9e1c018730 0x7f9e1c01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:20.944 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.940 7f9e267fc700 1 -- 172.21.2.112:0/1123543157 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f9e30032690 con 0x7f9e340c3820 2021-02-10T11:44:20.944 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.940 7f9e3943c700 1 --2- 172.21.2.112:0/1123543157 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9e1c018730 0x7f9e1c01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:20.944 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.940 7f9e3943c700 1 --2- 172.21.2.112:0/1123543157 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9e1c018730 0x7f9e1c01ac10 crc :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:20.973 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.971 7f9e3be21700 1 -- 172.21.2.112:0/1123543157 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e340fe1c0 con 0x7f9e340c3820 2021-02-10T11:44:20.975 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:20.973 7f9e267fc700 1 -- 172.21.2.112:0/1123543157 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9e30006b80 con 0x7f9e340c3820 2021-02-10T11:44:21.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:21.099 7f9e3be21700 1 -- 172.21.2.112:0/1123543157 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolumegroup rm", "vol_name": "cephfs", "target": ["mgr", ""], "group_name": "non_existent_group"}) v1 -- 0x7f9e340fe1c0 con 0x7f9e1c018730 2021-02-10T11:44:21.106 INFO:tasks.ceph.mgr.x.gibba020.stderr:2021-02-10 11:44:21.105 7f6382eee700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:44:21.106 INFO:tasks.ceph.mgr.x.gibba020.stderr:2021-02-10 11:44:21.105 7f6382eee700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:44:21.106 INFO:tasks.ceph.mgr.x.gibba020.stderr:2021-02-10 11:44:21.105 7f6382eee700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:44:21.106 INFO:tasks.ceph.mgr.x.gibba020.stderr:2021-02-10 11:44:21.105 7f6382eee700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:44:21.106 INFO:tasks.ceph.mgr.x.gibba020.stderr:2021-02-10 11:44:21.105 7f6382eee700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:44:21.110 INFO:tasks.ceph.mgr.x.gibba020.stderr:2021-02-10 11:44:21.109 7f6382eee700 -1 mgr.server reply reply (2) No such file or directory subvolume group 'non_existent_group' does not exist 2021-02-10T11:44:21.111 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:21.109 7f9e267fc700 1 -- 172.21.2.112:0/1123543157 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: -2 subvolume group 'non_existent_group' does not exist) v1 ==== 59+0+0 (crc 0 0 0) 0x7f9e280092d0 con 0x7f9e1c018730 2021-02-10T11:44:21.116 INFO:teuthology.orchestra.run.gibba012.stderr:Error ENOENT: subvolume group 'non_existent_group' does not exist 2021-02-10T11:44:21.118 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:21.117 7f9e3be21700 1 -- 172.21.2.112:0/1123543157 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9e1c018730 msgr2=0x7f9e1c01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:21.118 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:21.117 7f9e3be21700 1 --2- 172.21.2.112:0/1123543157 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9e1c018730 0x7f9e1c01ac10 crc :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:21.118 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:21.117 7f9e3be21700 1 -- 172.21.2.112:0/1123543157 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9e340c3820 msgr2=0x7f9e340a7950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:21.118 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:21.117 7f9e3be21700 1 --2- 172.21.2.112:0/1123543157 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9e340c3820 0x7f9e340a7950 secure :-1 s=READY pgs=2599 cs=0 l=1 rev1=1 rx=0x7f9e3000b5c0 tx=0x7f9e3000b960).stop 2021-02-10T11:44:21.119 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:21.118 7f9e3be21700 1 -- 172.21.2.112:0/1123543157 shutdown_connections 2021-02-10T11:44:21.119 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:21.118 7f9e3be21700 1 --2- 172.21.2.112:0/1123543157 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9e1c018730 0x7f9e1c01ac10 unknown :-1 s=CLOSED pgs=1188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:21.119 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:21.118 7f9e3be21700 1 --2- 172.21.2.112:0/1123543157 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9e341a44c0 0x7f9e341a48c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:21.119 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:21.118 7f9e3be21700 1 --2- 172.21.2.112:0/1123543157 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9e340863d0 0x7f9e340867d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:21.119 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:21.118 7f9e3be21700 1 --2- 172.21.2.112:0/1123543157 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9e340c3820 0x7f9e340a7950 unknown :-1 s=CLOSED pgs=2599 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:21.119 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:21.118 7f9e3be21700 1 -- 172.21.2.112:0/1123543157 >> 172.21.2.112:0/1123543157 conn(0x7f9e340872c0 msgr2=0x7f9e34095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:21.119 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:21.118 7f9e3be21700 1 -- 172.21.2.112:0/1123543157 shutdown_connections 2021-02-10T11:44:21.119 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:21.118 7f9e3be21700 1 -- 172.21.2.112:0/1123543157 wait complete. 2021-02-10T11:44:21.127 DEBUG:teuthology.orchestra.run:got remote process result: 2 2021-02-10T11:44:21.127 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:44:21.127 DEBUG:teuthology.orchestra.run.gibba012:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:44:21.472 DEBUG:teuthology.orchestra.run.gibba020:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:44:21.817 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:44:21.817 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:44:21.817 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:44:21.817 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:44:22.153 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:44:22.153 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:44:22.153 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba012.front.sepia.ceph.com... 2021-02-10T11:44:22.153 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:44:22.153 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:44:22.505 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba012.stderr:ceph-fuse[172931]: fuse finished with error 0 and tester_r 0 2021-02-10T11:44:22.506 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:44:22.506 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:44:22.506 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:44:22.840 INFO:teuthology.orchestra.run.gibba012.stdout:ext2/ext3 2021-02-10T11:44:22.840 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:44:22.841 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:44:22.841 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:44:23.176 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:44:23.176 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:44:23.176 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:44:23.512 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:44:23.512 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:44:23.512 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba012.front.sepia.ceph.com... 2021-02-10T11:44:23.512 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:44:23.512 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:44:23.865 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba012.stderr:ceph-fuse[173571]: fuse finished with error 0 and tester_r 0 2021-02-10T11:44:23.867 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:44:23.867 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:44:23.867 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:44:24.202 INFO:teuthology.orchestra.run.gibba012.stdout:ext2/ext3 2021-02-10T11:44:24.203 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:44:24.203 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:44:24.203 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:44:24.538 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:44:24.538 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:44:24.539 DEBUG:teuthology.orchestra.run.gibba020:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:44:24.808 DEBUG:teuthology.orchestra.run.gibba012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:44:24.810 DEBUG:teuthology.orchestra.run.gibba020:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:44:24.872 INFO:teuthology.orchestra.run.gibba020.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:44:24.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:44:24.873 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:44:24.873 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:44:24.873 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:44:25.146 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:44:25.146 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:44:25.146 DEBUG:teuthology.orchestra.run.gibba020:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:44:25.480 INFO:teuthology.orchestra.run.gibba020.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:44:25.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:44:25.481 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:44:25.481 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:44:25.481 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:44:25.816 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config rm mon mon_allow_pool_delete 2021-02-10T11:44:26.248 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.229 7f8cd74f8700 1 Processor -- start 2021-02-10T11:44:26.248 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.229 7f8cd74f8700 1 -- start start 2021-02-10T11:44:26.248 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.230 7f8cd74f8700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8cd00d1870 0x7f8cd01b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:26.248 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.230 7f8cd74f8700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f8cd00f0650 0x7f8cd00cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:26.248 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.230 7f8cd74f8700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cd00f0b10 con 0x7f8cd01b0a10 2021-02-10T11:44:26.248 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.230 7f8cd74f8700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cd00fe230 con 0x7f8cd00f0650 2021-02-10T11:44:26.249 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.230 7f8cd74f8700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cd00e5ed0 con 0x7f8cd00d1870 2021-02-10T11:44:26.249 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.230 7f8cd5b15700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f8cd00f0650 0x7f8cd00cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:26.249 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.230 7f8cd5b15700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f8cd00f0650 0x7f8cd00cfb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:57840/0 (socket says 172.21.2.112:57840) 2021-02-10T11:44:26.249 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.230 7f8cd5b15700 1 -- 172.21.2.112:0/2954035153 learned_addr learned my addr 172.21.2.112:0/2954035153 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:44:26.249 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.230 7f8cd5b15700 1 -- 172.21.2.112:0/2954035153 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8cd00d1870 msgr2=0x7f8cd01b03f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:44:26.249 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.230 7f8cd5b15700 1 --2- 172.21.2.112:0/2954035153 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8cd00d1870 0x7f8cd01b03f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:26.249 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.230 7f8cd5b15700 1 -- 172.21.2.112:0/2954035153 >> v1:172.21.2.120:6789/0 conn(0x7f8cd01b0a10 legacy=0x7f8cd00cf330 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:44:26.249 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.230 7f8cd5b15700 1 -- 172.21.2.112:0/2954035153 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cd00f20a0 con 0x7f8cd00f0650 2021-02-10T11:44:26.249 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.230 7f8cd5b15700 1 --2- 172.21.2.112:0/2954035153 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8cd00f0650 0x7f8cd00cfb20 secure :-1 s=READY pgs=2600 cs=0 l=1 rev1=1 rx=0x7f8cc8006f00 tx=0x7f8cc8009f10).ready entity=mon.0 client_cookie=384da8f683e9b7ce server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:26.249 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.231 7f8cc7fff700 1 -- 172.21.2.112:0/2954035153 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8cc800b3b0 con 0x7f8cd00f0650 2021-02-10T11:44:26.249 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.231 7f8cc7fff700 1 -- 172.21.2.112:0/2954035153 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cc800b730 con 0x7f8cd00f0650 2021-02-10T11:44:26.249 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.231 7f8cc7fff700 1 -- 172.21.2.112:0/2954035153 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8cc800bc50 con 0x7f8cd00f0650 2021-02-10T11:44:26.250 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.231 7f8cd74f8700 1 -- 172.21.2.112:0/2954035153 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8cd00f0650 msgr2=0x7f8cd00cfb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:26.250 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.231 7f8cd74f8700 1 --2- 172.21.2.112:0/2954035153 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8cd00f0650 0x7f8cd00cfb20 secure :-1 s=READY pgs=2600 cs=0 l=1 rev1=1 rx=0x7f8cc8006f00 tx=0x7f8cc8009f10).stop 2021-02-10T11:44:26.250 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.231 7f8cd74f8700 1 -- 172.21.2.112:0/2954035153 shutdown_connections 2021-02-10T11:44:26.250 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.231 7f8cd74f8700 1 --2- 172.21.2.112:0/2954035153 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8cd00f0650 0x7f8cd00cfb20 unknown :-1 s=CLOSED pgs=2600 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:26.250 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.231 7f8cd74f8700 1 --2- 172.21.2.112:0/2954035153 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8cd00d1870 0x7f8cd01b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:26.250 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.231 7f8cd74f8700 1 -- 172.21.2.112:0/2954035153 >> 172.21.2.112:0/2954035153 conn(0x7f8cd00ce450 msgr2=0x7f8cd00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:26.250 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.232 7f8cd74f8700 1 -- 172.21.2.112:0/2954035153 shutdown_connections 2021-02-10T11:44:26.250 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.232 7f8cd74f8700 1 -- 172.21.2.112:0/2954035153 wait complete. 2021-02-10T11:44:26.250 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.247 7f8cd74f8700 1 Processor -- start 2021-02-10T11:44:26.250 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.247 7f8cd74f8700 1 -- start start 2021-02-10T11:44:26.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.247 7f8cd74f8700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8cd00a2fb0 0x7f8cd00a33b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:26.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.247 7f8cd74f8700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8cd00a39d0 0x7f8cd00a4860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:26.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.247 7f8cd74f8700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8cd00a3dd0 0x7f8cd00a4e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:26.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.247 7f8cd74f8700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cd00a2620 con 0x7f8cd00a2fb0 2021-02-10T11:44:26.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.247 7f8cd74f8700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cd00a2840 con 0x7f8cd00a3dd0 2021-02-10T11:44:26.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.247 7f8cd74f8700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cd00a2b80 con 0x7f8cd00a39d0 2021-02-10T11:44:26.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.247 7f8cd5314700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8cd00a2fb0 0x7f8cd00a33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:26.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.247 7f8cd5314700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8cd00a2fb0 0x7f8cd00a33b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:57844/0 (socket says 172.21.2.112:57844) 2021-02-10T11:44:26.252 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.247 7f8cd5314700 1 -- 172.21.2.112:0/2628389952 learned_addr learned my addr 172.21.2.112:0/2628389952 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:44:26.252 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.248 7f8cd4b13700 1 --2- 172.21.2.112:0/2628389952 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8cd00a39d0 0x7f8cd00a4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:26.252 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.248 7f8cd5b15700 1 --2- 172.21.2.112:0/2628389952 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8cd00a3dd0 0x7f8cd00a4e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:26.258 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.248 7f8cd5314700 1 -- 172.21.2.112:0/2628389952 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8cd00a39d0 msgr2=0x7f8cd00a4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:26.258 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.248 7f8cd5314700 1 --2- 172.21.2.112:0/2628389952 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8cd00a39d0 0x7f8cd00a4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:26.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.248 7f8cd5314700 1 -- 172.21.2.112:0/2628389952 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8cd00a3dd0 msgr2=0x7f8cd00a4e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:26.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.248 7f8cd5314700 1 --2- 172.21.2.112:0/2628389952 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8cd00a3dd0 0x7f8cd00a4e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:26.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.248 7f8cd5314700 1 -- 172.21.2.112:0/2628389952 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cd00a54a0 con 0x7f8cd00a2fb0 2021-02-10T11:44:26.262 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.248 7f8cd5b15700 1 --2- 172.21.2.112:0/2628389952 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8cd00a3dd0 0x7f8cd00a4e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:44:26.262 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.248 7f8cd5314700 1 --2- 172.21.2.112:0/2628389952 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8cd00a2fb0 0x7f8cd00a33b0 secure :-1 s=READY pgs=2601 cs=0 l=1 rev1=1 rx=0x7f8ccc00b560 tx=0x7f8ccc00b900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:26.262 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.248 7f8cc5ffb700 1 -- 172.21.2.112:0/2628389952 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8ccc000c40 con 0x7f8cd00a2fb0 2021-02-10T11:44:26.262 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.248 7f8cc5ffb700 1 -- 172.21.2.112:0/2628389952 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ccc00bcf0 con 0x7f8cd00a2fb0 2021-02-10T11:44:26.262 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.249 7f8cd74f8700 1 -- 172.21.2.112:0/2628389952 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cd00a54a0 con 0x7f8cd00a2fb0 2021-02-10T11:44:26.262 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.249 7f8cd74f8700 1 -- 172.21.2.112:0/2628389952 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cd00a5a30 con 0x7f8cd00a2fb0 2021-02-10T11:44:26.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.249 7f8cc5ffb700 1 -- 172.21.2.112:0/2628389952 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8ccc0054d0 con 0x7f8cd00a2fb0 2021-02-10T11:44:26.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.249 7f8cc5ffb700 1 -- 172.21.2.112:0/2628389952 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8ccc006030 con 0x7f8cd00a2fb0 2021-02-10T11:44:26.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.249 7f8cc5ffb700 1 --2- 172.21.2.112:0/2628389952 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8cb00186c0 0x7f8cb001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:26.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.249 7f8cc5ffb700 1 -- 172.21.2.112:0/2628389952 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f8ccc032660 con 0x7f8cd00a2fb0 2021-02-10T11:44:26.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.250 7f8cd4b13700 1 --2- 172.21.2.112:0/2628389952 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8cb00186c0 0x7f8cb001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:26.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.250 7f8cd4b13700 1 --2- 172.21.2.112:0/2628389952 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8cb00186c0 0x7f8cb001abc0 crc :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:26.280 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.280 7f8cd74f8700 1 -- 172.21.2.112:0/2628389952 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cd00a4330 con 0x7f8cd00a2fb0 2021-02-10T11:44:26.283 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.282 7f8cc5ffb700 1 -- 172.21.2.112:0/2628389952 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8ccc000940 con 0x7f8cd00a2fb0 2021-02-10T11:44:26.430 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.429 7f8cd74f8700 1 -- 172.21.2.112:0/2628389952 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f8cd00a4330 con 0x7f8cd00a2fb0 2021-02-10T11:44:26.434 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.433 7f8cc5ffb700 1 -- 172.21.2.112:0/2628389952 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v37) v1 ==== 104+0+0 (secure 0 0 0) 0x7f8ccc006030 con 0x7f8cd00a2fb0 2021-02-10T11:44:26.439 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.438 7f8cd74f8700 1 -- 172.21.2.112:0/2628389952 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8cb00186c0 msgr2=0x7f8cb001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:26.439 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.438 7f8cd74f8700 1 --2- 172.21.2.112:0/2628389952 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8cb00186c0 0x7f8cb001abc0 crc :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:26.439 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.438 7f8cd74f8700 1 -- 172.21.2.112:0/2628389952 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8cd00a2fb0 msgr2=0x7f8cd00a33b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:26.439 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.438 7f8cd74f8700 1 --2- 172.21.2.112:0/2628389952 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8cd00a2fb0 0x7f8cd00a33b0 secure :-1 s=READY pgs=2601 cs=0 l=1 rev1=1 rx=0x7f8ccc00b560 tx=0x7f8ccc00b900).stop 2021-02-10T11:44:26.439 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.438 7f8cd74f8700 1 -- 172.21.2.112:0/2628389952 shutdown_connections 2021-02-10T11:44:26.439 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.438 7f8cd74f8700 1 --2- 172.21.2.112:0/2628389952 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8cb00186c0 0x7f8cb001abc0 unknown :-1 s=CLOSED pgs=1189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:26.439 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.438 7f8cd74f8700 1 --2- 172.21.2.112:0/2628389952 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8cd00a3dd0 0x7f8cd00a4e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:26.440 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.438 7f8cd74f8700 1 --2- 172.21.2.112:0/2628389952 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8cd00a39d0 0x7f8cd00a4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:26.440 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.438 7f8cd74f8700 1 --2- 172.21.2.112:0/2628389952 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8cd00a2fb0 0x7f8cd00a33b0 unknown :-1 s=CLOSED pgs=2601 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:26.440 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.438 7f8cd74f8700 1 -- 172.21.2.112:0/2628389952 >> 172.21.2.112:0/2628389952 conn(0x7f8cd008f030 msgr2=0x7f8cd00b5e80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:26.440 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.438 7f8cd74f8700 1 -- 172.21.2.112:0/2628389952 shutdown_connections 2021-02-10T11:44:26.440 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.438 7f8cd74f8700 1 -- 172.21.2.112:0/2628389952 wait complete. 2021-02-10T11:44:26.448 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestVolumes.test_nonexistent_subvolume_group_rm' 2021-02-10T11:44:26.902 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.883 7f7bafc36700 1 Processor -- start 2021-02-10T11:44:26.902 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.883 7f7bafc36700 1 -- start start 2021-02-10T11:44:26.902 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.884 7f7bafc36700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7ba80d1870 0x7f7ba81b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:26.902 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.884 7f7bafc36700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ba80f0b10 con 0x7f7ba80f0650 2021-02-10T11:44:26.902 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.884 7f7bafc36700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ba80d0310 con 0x7f7ba81b0a10 2021-02-10T11:44:26.902 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.884 7f7bafc36700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ba80e5ed0 con 0x7f7ba80d1870 2021-02-10T11:44:26.902 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.884 7f7bae253700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f7ba80f0650 0x7f7ba80cfb20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:57314/0 (socket says 172.21.2.112:57314) 2021-02-10T11:44:26.902 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.884 7f7bae253700 1 -- 172.21.2.112:0/3356165310 learned_addr learned my addr 172.21.2.112:0/3356165310 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:44:26.903 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.884 7f7baca50700 1 -- 172.21.2.112:0/3356165310 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4291018850 0 0) 0x7f7ba0001500 con 0x7f7ba80f0650 2021-02-10T11:44:26.903 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.884 7f7baca50700 1 -- 172.21.2.112:0/3356165310 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b94002fe0 con 0x7f7ba80f0650 2021-02-10T11:44:26.903 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.884 7f7bada52700 1 --2- 172.21.2.112:0/3356165310 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7ba80d1870 0x7f7ba81b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:26.903 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.884 7f7baca50700 1 -- 172.21.2.112:0/3356165310 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1695875434 0 0) 0x7f7ba0001460 con 0x7f7ba80f0650 2021-02-10T11:44:26.903 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.884 7f7baca50700 1 -- 172.21.2.112:0/3356165310 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7ba80d1870 msgr2=0x7f7ba81b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:26.903 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.884 7f7baca50700 1 --2- 172.21.2.112:0/3356165310 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7ba80d1870 0x7f7ba81b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:26.903 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.884 7f7baca50700 1 -- 172.21.2.112:0/3356165310 >> v1:172.21.2.120:6789/0 conn(0x7f7ba81b0a10 legacy=0x7f7ba80cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:26.903 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.884 7f7baca50700 1 -- 172.21.2.112:0/3356165310 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ba80d0530 con 0x7f7ba80f0650 2021-02-10T11:44:26.904 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.884 7f7baca50700 1 -- 172.21.2.112:0/3356165310 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f7ba0001980 con 0x7f7ba80f0650 2021-02-10T11:44:26.904 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.884 7f7baca50700 1 -- 172.21.2.112:0/3356165310 >> v1:172.21.2.112:6789/0 conn(0x7f7ba80f0650 legacy=0x7f7ba80cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:26.904 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.884 7f7baca50700 1 --2- 172.21.2.112:0/3356165310 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7b94003ae0 0x7f7b94005f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:26.904 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.884 7f7baca50700 1 --2- 172.21.2.112:0/3356165310 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7b94006830 0x7f7b94008d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:26.904 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.884 7f7baca50700 1 --2- 172.21.2.112:0/3356165310 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7b940095a0 0x7f7b9400bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:26.904 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.884 7f7baca50700 1 -- 172.21.2.112:0/3356165310 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b9400c370 con 0x7f7b940095a0 2021-02-10T11:44:26.904 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.884 7f7baca50700 1 -- 172.21.2.112:0/3356165310 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b9400c590 con 0x7f7b94003ae0 2021-02-10T11:44:26.904 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.884 7f7baca50700 1 -- 172.21.2.112:0/3356165310 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b9400c7b0 con 0x7f7b94006830 2021-02-10T11:44:26.904 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.885 7f7bada52700 1 --2- 172.21.2.112:0/3356165310 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7ba80d1870 0x7f7ba81b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:44:26.904 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.885 7f7bae253700 1 --2- 172.21.2.112:0/3356165310 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7b940095a0 0x7f7b9400bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:26.904 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.885 7f7bad251700 1 --2- 172.21.2.112:0/3356165310 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7b94006830 0x7f7b94008d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:26.904 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.885 7f7bae253700 1 -- 172.21.2.112:0/3356165310 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7b94006830 msgr2=0x7f7b94008d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:26.905 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.885 7f7bae253700 1 --2- 172.21.2.112:0/3356165310 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7b94006830 0x7f7b94008d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:26.905 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.885 7f7bae253700 1 -- 172.21.2.112:0/3356165310 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7b94003ae0 msgr2=0x7f7b94005f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:26.905 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.885 7f7bae253700 1 --2- 172.21.2.112:0/3356165310 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7b94003ae0 0x7f7b94005f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:26.905 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.885 7f7bae253700 1 -- 172.21.2.112:0/3356165310 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b9400ca50 con 0x7f7b940095a0 2021-02-10T11:44:26.905 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.885 7f7bae253700 1 --2- 172.21.2.112:0/3356165310 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7b940095a0 0x7f7b9400bac0 secure :-1 s=READY pgs=2603 cs=0 l=1 rev1=1 rx=0x7f7ba0001ae0 tx=0x7f7ba000a060).ready entity=mon.0 client_cookie=381851ba28dacc40 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:26.905 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.885 7f7baca50700 1 -- 172.21.2.112:0/3356165310 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7ba00070c0 con 0x7f7b940095a0 2021-02-10T11:44:26.905 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.885 7f7baca50700 1 -- 172.21.2.112:0/3356165310 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ba0007300 con 0x7f7b940095a0 2021-02-10T11:44:26.906 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.885 7f7baca50700 1 -- 172.21.2.112:0/3356165310 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7ba0004720 con 0x7f7b940095a0 2021-02-10T11:44:26.906 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.885 7f7bafc36700 1 -- 172.21.2.112:0/3356165310 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7b940095a0 msgr2=0x7f7b9400bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:26.906 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.885 7f7bafc36700 1 --2- 172.21.2.112:0/3356165310 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7b940095a0 0x7f7b9400bac0 secure :-1 s=READY pgs=2603 cs=0 l=1 rev1=1 rx=0x7f7ba0001ae0 tx=0x7f7ba000a060).stop 2021-02-10T11:44:26.906 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.885 7f7bae253700 1 -- 172.21.2.112:0/3356165310 reap_dead start 2021-02-10T11:44:26.906 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.886 7f7bafc36700 1 -- 172.21.2.112:0/3356165310 shutdown_connections 2021-02-10T11:44:26.906 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.886 7f7bafc36700 1 -- 172.21.2.112:0/3356165310 >> 172.21.2.112:0/3356165310 conn(0x7f7ba80ce450 msgr2=0x7f7ba80d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:26.906 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.886 7f7bafc36700 1 -- 172.21.2.112:0/3356165310 shutdown_connections 2021-02-10T11:44:26.906 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.886 7f7bafc36700 1 -- 172.21.2.112:0/3356165310 wait complete. 2021-02-10T11:44:26.906 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.901 7f7bafc36700 1 Processor -- start 2021-02-10T11:44:26.906 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.901 7f7bafc36700 1 -- start start 2021-02-10T11:44:26.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.901 7f7bafc36700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7ba818f360 0x7f7ba80a9940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:26.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.901 7f7bafc36700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7ba80a9f60 0x7f7ba80ecb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:26.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.901 7f7bafc36700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7ba80ed190 0x7f7ba80ed590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:26.909 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.901 7f7bafc36700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ba818f760 con 0x7f7ba818f360 2021-02-10T11:44:26.909 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.901 7f7bafc36700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ba818f9e0 con 0x7f7ba80ed190 2021-02-10T11:44:26.909 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.901 7f7bafc36700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ba808d230 con 0x7f7ba80a9f60 2021-02-10T11:44:26.909 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.901 7f7bada52700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7ba818f360 0x7f7ba80a9940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:26.909 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.901 7f7bada52700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7ba818f360 0x7f7ba80a9940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:57866/0 (socket says 172.21.2.112:57866) 2021-02-10T11:44:26.909 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.901 7f7bada52700 1 -- 172.21.2.112:0/2963360546 learned_addr learned my addr 172.21.2.112:0/2963360546 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:44:26.909 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.902 7f7bada52700 1 -- 172.21.2.112:0/2963360546 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7ba80a9f60 msgr2=0x7f7ba80ecb70 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:44:26.909 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.902 7f7bada52700 1 --2- 172.21.2.112:0/2963360546 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7ba80a9f60 0x7f7ba80ecb70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:26.909 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.902 7f7bada52700 1 -- 172.21.2.112:0/2963360546 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7ba80ed190 msgr2=0x7f7ba80ed590 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:44:26.909 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.902 7f7bada52700 1 --2- 172.21.2.112:0/2963360546 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7ba80ed190 0x7f7ba80ed590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:26.909 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.902 7f7bada52700 1 -- 172.21.2.112:0/2963360546 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ba808d450 con 0x7f7ba818f360 2021-02-10T11:44:26.910 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.902 7f7bada52700 1 --2- 172.21.2.112:0/2963360546 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7ba818f360 0x7f7ba80a9940 secure :-1 s=READY pgs=2604 cs=0 l=1 rev1=1 rx=0x7f7b9c006e90 tx=0x7f7b9c04e970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:26.910 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.902 7f7b9a7fc700 1 -- 172.21.2.112:0/2963360546 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7b9c006b20 con 0x7f7ba818f360 2021-02-10T11:44:26.910 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.902 7f7b9a7fc700 1 -- 172.21.2.112:0/2963360546 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b9c000c90 con 0x7f7ba818f360 2021-02-10T11:44:26.910 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.902 7f7b9a7fc700 1 -- 172.21.2.112:0/2963360546 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7b9c0022b0 con 0x7f7ba818f360 2021-02-10T11:44:26.910 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.902 7f7bafc36700 1 -- 172.21.2.112:0/2963360546 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ba808d450 con 0x7f7ba818f360 2021-02-10T11:44:26.910 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.902 7f7bafc36700 1 -- 172.21.2.112:0/2963360546 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ba808d450 con 0x7f7ba818f360 2021-02-10T11:44:26.910 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.903 7f7b9a7fc700 1 -- 172.21.2.112:0/2963360546 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7b9c0052e0 con 0x7f7ba818f360 2021-02-10T11:44:26.910 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.903 7f7b9a7fc700 1 --2- 172.21.2.112:0/2963360546 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7b900186c0 0x7f7b9001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:26.910 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.903 7f7b9a7fc700 1 -- 172.21.2.112:0/2963360546 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f7b9c076590 con 0x7f7ba818f360 2021-02-10T11:44:26.910 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.904 7f7bad251700 1 --2- 172.21.2.112:0/2963360546 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7b900186c0 0x7f7b9001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:26.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.904 7f7bad251700 1 --2- 172.21.2.112:0/2963360546 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7b900186c0 0x7f7b9001abc0 crc :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:26.935 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.933 7f7bafc36700 1 -- 172.21.2.112:0/2963360546 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ba80ebc60 con 0x7f7ba818f360 2021-02-10T11:44:26.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:26.936 7f7b9a7fc700 1 -- 172.21.2.112:0/2963360546 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7b9c006b20 con 0x7f7ba818f360 2021-02-10T11:44:27.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:27.069 7f7bafc36700 1 -- 172.21.2.112:0/2963360546 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_nonexistent_subvolume_group_rm"]} v 0) v1 -- 0x7f7ba80ebc60 con 0x7f7ba818f360 2021-02-10T11:44:27.241 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:27.240 7f7b9a7fc700 1 -- 172.21.2.112:0/2963360546 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_nonexistent_subvolume_group_rm"]}]=0 v709) v1 ==== 152+0+0 (secure 0 0 0) 0x7f7b9c0052e0 con 0x7f7ba818f360 2021-02-10T11:44:27.287 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:27.286 7f7bafc36700 1 -- 172.21.2.112:0/2963360546 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7b900186c0 msgr2=0x7f7b9001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:27.287 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:27.286 7f7bafc36700 1 --2- 172.21.2.112:0/2963360546 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7b900186c0 0x7f7b9001abc0 crc :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:27.288 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:27.286 7f7bafc36700 1 -- 172.21.2.112:0/2963360546 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7ba818f360 msgr2=0x7f7ba80a9940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:27.288 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:27.286 7f7bafc36700 1 --2- 172.21.2.112:0/2963360546 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7ba818f360 0x7f7ba80a9940 secure :-1 s=READY pgs=2604 cs=0 l=1 rev1=1 rx=0x7f7b9c006e90 tx=0x7f7b9c04e970).stop 2021-02-10T11:44:27.288 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:27.286 7f7bafc36700 1 -- 172.21.2.112:0/2963360546 shutdown_connections 2021-02-10T11:44:27.288 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:27.286 7f7bafc36700 1 --2- 172.21.2.112:0/2963360546 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7b900186c0 0x7f7b9001abc0 unknown :-1 s=CLOSED pgs=1190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:27.288 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:27.286 7f7bafc36700 1 --2- 172.21.2.112:0/2963360546 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7ba80ed190 0x7f7ba80ed590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:27.288 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:27.286 7f7bafc36700 1 --2- 172.21.2.112:0/2963360546 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7ba80a9f60 0x7f7ba80ecb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:27.288 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:27.286 7f7bafc36700 1 --2- 172.21.2.112:0/2963360546 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7ba818f360 0x7f7ba80a9940 unknown :-1 s=CLOSED pgs=2604 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:27.288 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:27.286 7f7bafc36700 1 -- 172.21.2.112:0/2963360546 >> 172.21.2.112:0/2963360546 conn(0x7f7ba809eba0 msgr2=0x7f7ba80ebe90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:27.288 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:27.286 7f7bafc36700 1 -- 172.21.2.112:0/2963360546 shutdown_connections 2021-02-10T11:44:27.288 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:27.286 7f7bafc36700 1 -- 172.21.2.112:0/2963360546 wait complete. 2021-02-10T11:44:27.295 INFO:tasks.cephfs_test_runner:test_nonexistent_subvolume_group_rm (tasks.cephfs.test_volumes.TestVolumes) ... ok 2021-02-10T11:44:27.295 INFO:tasks.cephfs_test_runner:Starting test: test_nonexistent_subvolume_group_snapshot_rm (tasks.cephfs.test_volumes.TestVolumes) 2021-02-10T11:44:27.296 INFO:tasks.cephfs_test_runner:test_nonexistent_subvolume_group_snapshot_rm (tasks.cephfs.test_volumes.TestVolumes) ... skipped 'skipping subvolumegroup snapshot tests' 2021-02-10T11:44:27.296 INFO:tasks.cephfs_test_runner:Starting test: test_nonexistent_subvolume_rm (tasks.cephfs.test_volumes.TestVolumes) 2021-02-10T11:44:27.296 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestVolumes.test_nonexistent_subvolume_rm' 2021-02-10T11:44:27.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:27.705 7f4c171ee700 1 Processor -- start 2021-02-10T11:44:27.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:27.705 7f4c171ee700 1 -- start start 2021-02-10T11:44:27.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:27.706 7f4c171ee700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f4c100d0340 0x7f4c100ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:27.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:27.706 7f4c171ee700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4c100d1c70 0x7f4c100efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:27.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:27.706 7f4c171ee700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c100f0b10 con 0x7f4c100d1870 2021-02-10T11:44:27.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:27.706 7f4c171ee700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c100e5ed0 con 0x7f4c100d0340 2021-02-10T11:44:27.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:27.706 7f4c171ee700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c101b0be0 con 0x7f4c100d1c70 2021-02-10T11:44:27.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:27.706 7f4c14809700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f4c100d0340 0x7f4c100ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:27.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:27.706 7f4c14809700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f4c100d0340 0x7f4c100ef640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:57870/0 (socket says 172.21.2.112:57870) 2021-02-10T11:44:27.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:27.706 7f4c14809700 1 -- 172.21.2.112:0/1272094523 learned_addr learned my addr 172.21.2.112:0/1272094523 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:44:27.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:27.706 7f4c1580b700 1 --2- 172.21.2.112:0/1272094523 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4c100d1c70 0x7f4c100efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:27.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:27.706 7f4c14809700 1 -- 172.21.2.112:0/1272094523 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4c100d1c70 msgr2=0x7f4c100efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:27.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:27.706 7f4c14809700 1 --2- 172.21.2.112:0/1272094523 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4c100d1c70 0x7f4c100efc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:27.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:27.706 7f4c14809700 1 -- 172.21.2.112:0/1272094523 >> v1:172.21.2.120:6789/0 conn(0x7f4c100d1870 legacy=0x7f4c101b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:27.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:27.706 7f4c14809700 1 -- 172.21.2.112:0/1272094523 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c100f20a0 con 0x7f4c100d0340 2021-02-10T11:44:27.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:27.706 7f4c14809700 1 --2- 172.21.2.112:0/1272094523 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4c100d0340 0x7f4c100ef640 secure :-1 s=READY pgs=2605 cs=0 l=1 rev1=1 rx=0x7f4c0c006700 tx=0x7f4c0c009b80).ready entity=mon.0 client_cookie=72e584ff2968accf server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:27.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:27.707 7f4c03fff700 1 -- 172.21.2.112:0/1272094523 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4c0c00c0c0 con 0x7f4c100d0340 2021-02-10T11:44:27.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:27.707 7f4c03fff700 1 -- 172.21.2.112:0/1272094523 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c0c00c340 con 0x7f4c100d0340 2021-02-10T11:44:27.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:27.707 7f4c03fff700 1 -- 172.21.2.112:0/1272094523 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4c0c00c860 con 0x7f4c100d0340 2021-02-10T11:44:27.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:27.707 7f4c171ee700 1 -- 172.21.2.112:0/1272094523 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4c100d0340 msgr2=0x7f4c100ef640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:27.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:27.707 7f4c171ee700 1 --2- 172.21.2.112:0/1272094523 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4c100d0340 0x7f4c100ef640 secure :-1 s=READY pgs=2605 cs=0 l=1 rev1=1 rx=0x7f4c0c006700 tx=0x7f4c0c009b80).stop 2021-02-10T11:44:27.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:27.707 7f4c171ee700 1 -- 172.21.2.112:0/1272094523 shutdown_connections 2021-02-10T11:44:27.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:27.707 7f4c171ee700 1 --2- 172.21.2.112:0/1272094523 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4c100d0340 0x7f4c100ef640 unknown :-1 s=CLOSED pgs=2605 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:27.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:27.707 7f4c171ee700 1 --2- 172.21.2.112:0/1272094523 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4c100d1c70 0x7f4c100efc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:27.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:27.707 7f4c171ee700 1 -- 172.21.2.112:0/1272094523 >> 172.21.2.112:0/1272094523 conn(0x7f4c100ce450 msgr2=0x7f4c100d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:27.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:27.707 7f4c171ee700 1 -- 172.21.2.112:0/1272094523 shutdown_connections 2021-02-10T11:44:27.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:27.707 7f4c171ee700 1 -- 172.21.2.112:0/1272094523 wait complete. 2021-02-10T11:44:27.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:27.722 7f4c171ee700 1 Processor -- start 2021-02-10T11:44:27.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:27.722 7f4c171ee700 1 -- start start 2021-02-10T11:44:27.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:27.722 7f4c171ee700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4c10195650 0x7f4c100b5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:27.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:27.722 7f4c171ee700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4c100b5b80 0x7f4c101a2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:27.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:27.722 7f4c171ee700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4c100acfa0 0x7f4c101a3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:27.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:27.722 7f4c171ee700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c100b6010 con 0x7f4c100acfa0 2021-02-10T11:44:27.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:27.722 7f4c171ee700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c101a3a60 con 0x7f4c10195650 2021-02-10T11:44:27.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:27.722 7f4c171ee700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c101a3c40 con 0x7f4c100b5b80 2021-02-10T11:44:27.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:27.722 7f4c1580b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4c100acfa0 0x7f4c101a3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:27.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:27.723 7f4c1580b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4c100acfa0 0x7f4c101a3440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:57880/0 (socket says 172.21.2.112:57880) 2021-02-10T11:44:27.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:27.723 7f4c1580b700 1 -- 172.21.2.112:0/2867916307 learned_addr learned my addr 172.21.2.112:0/2867916307 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:44:27.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:27.723 7f4c1580b700 1 -- 172.21.2.112:0/2867916307 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4c100b5b80 msgr2=0x7f4c101a2e20 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:44:27.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:27.723 7f4c1580b700 1 --2- 172.21.2.112:0/2867916307 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4c100b5b80 0x7f4c101a2e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:27.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:27.723 7f4c1580b700 1 -- 172.21.2.112:0/2867916307 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4c10195650 msgr2=0x7f4c100b5560 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:44:27.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:27.723 7f4c1580b700 1 --2- 172.21.2.112:0/2867916307 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4c10195650 0x7f4c100b5560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:27.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:27.723 7f4c1580b700 1 -- 172.21.2.112:0/2867916307 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c100bcae0 con 0x7f4c100acfa0 2021-02-10T11:44:27.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:27.723 7f4c1580b700 1 --2- 172.21.2.112:0/2867916307 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4c100acfa0 0x7f4c101a3440 secure :-1 s=READY pgs=2606 cs=0 l=1 rev1=1 rx=0x7f4c0800b970 tx=0x7f4c0800bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:27.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:27.723 7f4c01ffb700 1 -- 172.21.2.112:0/2867916307 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4c08000f90 con 0x7f4c100acfa0 2021-02-10T11:44:27.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:27.723 7f4c01ffb700 1 -- 172.21.2.112:0/2867916307 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c08006aa0 con 0x7f4c100acfa0 2021-02-10T11:44:27.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:27.723 7f4c01ffb700 1 -- 172.21.2.112:0/2867916307 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4c08001680 con 0x7f4c100acfa0 2021-02-10T11:44:27.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:27.723 7f4c171ee700 1 -- 172.21.2.112:0/2867916307 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c100bcae0 con 0x7f4c100acfa0 2021-02-10T11:44:27.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:27.723 7f4c171ee700 1 -- 172.21.2.112:0/2867916307 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c100bd070 con 0x7f4c100acfa0 2021-02-10T11:44:27.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:27.724 7f4c01ffb700 1 -- 172.21.2.112:0/2867916307 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4c080011f0 con 0x7f4c100acfa0 2021-02-10T11:44:27.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:27.724 7f4c01ffb700 1 --2- 172.21.2.112:0/2867916307 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4bf00186c0 0x7f4bf001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:27.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:27.724 7f4c01ffb700 1 -- 172.21.2.112:0/2867916307 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f4c080329c0 con 0x7f4c100acfa0 2021-02-10T11:44:27.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:27.724 7f4c1500a700 1 --2- 172.21.2.112:0/2867916307 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4bf00186c0 0x7f4bf001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:27.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:27.725 7f4c1500a700 1 --2- 172.21.2.112:0/2867916307 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4bf00186c0 0x7f4bf001abc0 crc :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:27.757 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:27.756 7f4c171ee700 1 -- 172.21.2.112:0/2867916307 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c100f0ab0 con 0x7f4c100acfa0 2021-02-10T11:44:27.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:27.759 7f4c01ffb700 1 -- 172.21.2.112:0/2867916307 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4c08000f20 con 0x7f4c100acfa0 2021-02-10T11:44:27.891 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:27.890 7f4c171ee700 1 -- 172.21.2.112:0/2867916307 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_nonexistent_subvolume_rm"]} v 0) v1 -- 0x7f4c100f0ab0 con 0x7f4c100acfa0 2021-02-10T11:44:28.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:28.245 7f4c01ffb700 1 -- 172.21.2.112:0/2867916307 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_nonexistent_subvolume_rm"]}]=0 v710) v1 ==== 149+0+0 (secure 0 0 0) 0x7f4c080011f0 con 0x7f4c100acfa0 2021-02-10T11:44:28.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:28.258 7f4c171ee700 1 -- 172.21.2.112:0/2867916307 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4bf00186c0 msgr2=0x7f4bf001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:28.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:28.258 7f4c171ee700 1 --2- 172.21.2.112:0/2867916307 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4bf00186c0 0x7f4bf001abc0 crc :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:28.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:28.259 7f4c171ee700 1 -- 172.21.2.112:0/2867916307 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4c100acfa0 msgr2=0x7f4c101a3440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:28.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:28.259 7f4c171ee700 1 --2- 172.21.2.112:0/2867916307 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4c100acfa0 0x7f4c101a3440 secure :-1 s=READY pgs=2606 cs=0 l=1 rev1=1 rx=0x7f4c0800b970 tx=0x7f4c0800bd10).stop 2021-02-10T11:44:28.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:28.259 7f4c171ee700 1 -- 172.21.2.112:0/2867916307 shutdown_connections 2021-02-10T11:44:28.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:28.259 7f4c171ee700 1 --2- 172.21.2.112:0/2867916307 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4bf00186c0 0x7f4bf001abc0 unknown :-1 s=CLOSED pgs=1191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:28.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:28.259 7f4c171ee700 1 --2- 172.21.2.112:0/2867916307 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4c100acfa0 0x7f4c101a3440 unknown :-1 s=CLOSED pgs=2606 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:28.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:28.259 7f4c171ee700 1 --2- 172.21.2.112:0/2867916307 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4c100b5b80 0x7f4c101a2e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:28.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:28.259 7f4c171ee700 1 --2- 172.21.2.112:0/2867916307 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4c10195650 0x7f4c100b5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:28.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:28.259 7f4c171ee700 1 -- 172.21.2.112:0/2867916307 >> 172.21.2.112:0/2867916307 conn(0x7f4c100b3040 msgr2=0x7f4c100c3250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:28.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:28.259 7f4c171ee700 1 -- 172.21.2.112:0/2867916307 shutdown_connections 2021-02-10T11:44:28.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:28.259 7f4c171ee700 1 -- 172.21.2.112:0/2867916307 wait complete. 2021-02-10T11:44:28.268 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:44:28.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:28.684 7f3eb7c4d700 1 Processor -- start 2021-02-10T11:44:28.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:28.684 7f3eb7c4d700 1 -- start start 2021-02-10T11:44:28.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:28.684 7f3eb7c4d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f3eb010afe0 0x7f3eb00cf740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:28.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:28.684 7f3eb7c4d700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3eb00fe1b0 con 0x7f3eb00d1860 2021-02-10T11:44:28.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:28.684 7f3eb7c4d700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3eb01b0ca0 con 0x7f3eb00d1c60 2021-02-10T11:44:28.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:28.684 7f3eb7c4d700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3eb0109a50 con 0x7f3eb010afe0 2021-02-10T11:44:28.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:28.685 7f3eb626a700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f3eb010afe0 0x7f3eb00cf740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:28.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:28.685 7f3eb626a700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f3eb010afe0 0x7f3eb00cf740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:57888/0 (socket says 172.21.2.112:57888) 2021-02-10T11:44:28.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:28.685 7f3eb626a700 1 -- 172.21.2.112:0/92515980 learned_addr learned my addr 172.21.2.112:0/92515980 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:44:28.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:28.685 7f3eb626a700 1 -- 172.21.2.112:0/92515980 >> v1:172.21.2.120:6789/0 conn(0x7f3eb00d1c60 legacy=0x7f3eb00cef50 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:44:28.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:28.685 7f3eb626a700 1 -- 172.21.2.112:0/92515980 >> v1:172.21.2.112:6789/0 conn(0x7f3eb00d1860 legacy=0x7f3eb01b04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:28.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:28.685 7f3eb626a700 1 -- 172.21.2.112:0/92515980 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3eb010b3e0 con 0x7f3eb010afe0 2021-02-10T11:44:28.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:28.685 7f3eb626a700 1 --2- 172.21.2.112:0/92515980 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3eb010afe0 0x7f3eb00cf740 secure :-1 s=READY pgs=2608 cs=0 l=1 rev1=1 rx=0x7f3ea8006f00 tx=0x7f3ea8009f10).ready entity=mon.0 client_cookie=1a38f8e93bd7cc8c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:28.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:28.685 7f3eb4a67700 1 -- 172.21.2.112:0/92515980 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3ea800b3b0 con 0x7f3eb010afe0 2021-02-10T11:44:28.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:28.685 7f3eb4a67700 1 -- 172.21.2.112:0/92515980 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ea800b730 con 0x7f3eb010afe0 2021-02-10T11:44:28.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:28.685 7f3eb4a67700 1 -- 172.21.2.112:0/92515980 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3ea800bc10 con 0x7f3eb010afe0 2021-02-10T11:44:28.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:28.685 7f3eb7c4d700 1 -- 172.21.2.112:0/92515980 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3eb010afe0 msgr2=0x7f3eb00cf740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:28.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:28.685 7f3eb7c4d700 1 --2- 172.21.2.112:0/92515980 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3eb010afe0 0x7f3eb00cf740 secure :-1 s=READY pgs=2608 cs=0 l=1 rev1=1 rx=0x7f3ea8006f00 tx=0x7f3ea8009f10).stop 2021-02-10T11:44:28.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:28.686 7f3eb7c4d700 1 -- 172.21.2.112:0/92515980 shutdown_connections 2021-02-10T11:44:28.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:28.686 7f3eb7c4d700 1 --2- 172.21.2.112:0/92515980 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3eb010afe0 0x7f3eb00cf740 unknown :-1 s=CLOSED pgs=2608 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:28.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:28.686 7f3eb7c4d700 1 -- 172.21.2.112:0/92515980 >> 172.21.2.112:0/92515980 conn(0x7f3eb00ce410 msgr2=0x7f3eb00d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:28.704 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:28.686 7f3eb7c4d700 1 -- 172.21.2.112:0/92515980 shutdown_connections 2021-02-10T11:44:28.704 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:28.686 7f3eb7c4d700 1 -- 172.21.2.112:0/92515980 wait complete. 2021-02-10T11:44:28.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:28.701 7f3eb7c4d700 1 Processor -- start 2021-02-10T11:44:28.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:28.701 7f3eb7c4d700 1 -- start start 2021-02-10T11:44:28.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:28.701 7f3eb7c4d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3eb00c3500 0x7f3eb00a2e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:28.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:28.701 7f3eb7c4d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3eb00b79a0 0x7f3eb00b7da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:28.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:28.701 7f3eb7c4d700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3eb00b9e40 0x7f3eb00ba240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:28.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:28.701 7f3eb7c4d700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb01065b0 con 0x7f3eb00b79a0 2021-02-10T11:44:28.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:28.701 7f3eb7c4d700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb00b70f0 con 0x7f3eb00b9e40 2021-02-10T11:44:28.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:28.701 7f3eb7c4d700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb00a3450 con 0x7f3eb00c3500 2021-02-10T11:44:28.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:28.701 7f3eb5268700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3eb00b79a0 0x7f3eb00b7da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:28.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:28.701 7f3eb5268700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3eb00b79a0 0x7f3eb00b7da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:57890/0 (socket says 172.21.2.112:57890) 2021-02-10T11:44:28.707 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:28.701 7f3eb5268700 1 -- 172.21.2.112:0/2834755786 learned_addr learned my addr 172.21.2.112:0/2834755786 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:44:28.707 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:28.701 7f3eb5268700 1 -- 172.21.2.112:0/2834755786 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3eb00c3500 msgr2=0x7f3eb00a2e30 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:44:28.707 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:28.701 7f3eb5268700 1 --2- 172.21.2.112:0/2834755786 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3eb00c3500 0x7f3eb00a2e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:28.707 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:28.701 7f3eb5268700 1 -- 172.21.2.112:0/2834755786 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3eb00b9e40 msgr2=0x7f3eb00ba240 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:44:28.707 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:28.701 7f3eb5268700 1 --2- 172.21.2.112:0/2834755786 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3eb00b9e40 0x7f3eb00ba240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:28.707 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:28.702 7f3eb5268700 1 -- 172.21.2.112:0/2834755786 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3eb00b83c0 con 0x7f3eb00b79a0 2021-02-10T11:44:28.707 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:28.702 7f3eb5268700 1 --2- 172.21.2.112:0/2834755786 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3eb00b79a0 0x7f3eb00b7da0 secure :-1 s=READY pgs=2609 cs=0 l=1 rev1=1 rx=0x7f3ea400b5b0 tx=0x7f3ea400b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:28.707 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:28.702 7f3ea27fc700 1 -- 172.21.2.112:0/2834755786 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3ea400bd40 con 0x7f3eb00b79a0 2021-02-10T11:44:28.707 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:28.702 7f3ea27fc700 1 -- 172.21.2.112:0/2834755786 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ea4002080 con 0x7f3eb00b79a0 2021-02-10T11:44:28.707 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:28.702 7f3eb7c4d700 1 -- 172.21.2.112:0/2834755786 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3eb00b83c0 con 0x7f3eb00b79a0 2021-02-10T11:44:28.708 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:28.702 7f3ea27fc700 1 -- 172.21.2.112:0/2834755786 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3ea4002560 con 0x7f3eb00b79a0 2021-02-10T11:44:28.708 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:28.702 7f3eb7c4d700 1 -- 172.21.2.112:0/2834755786 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3eb00b83c0 con 0x7f3eb00b79a0 2021-02-10T11:44:28.708 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:28.702 7f3ea27fc700 1 -- 172.21.2.112:0/2834755786 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3ea4002260 con 0x7f3eb00b79a0 2021-02-10T11:44:28.708 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:28.703 7f3ea27fc700 1 --2- 172.21.2.112:0/2834755786 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3e98018730 0x7f3e9801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:28.708 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:28.703 7f3ea27fc700 1 -- 172.21.2.112:0/2834755786 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f3ea4032800 con 0x7f3eb00b79a0 2021-02-10T11:44:28.708 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:28.703 7f3eb5a69700 1 --2- 172.21.2.112:0/2834755786 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3e98018730 0x7f3e9801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:28.708 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:28.704 7f3eb5a69700 1 --2- 172.21.2.112:0/2834755786 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3e98018730 0x7f3e9801abc0 crc :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:28.735 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:28.734 7f3eb7c4d700 1 -- 172.21.2.112:0/2834755786 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3eb00babd0 con 0x7f3eb00b79a0 2021-02-10T11:44:28.737 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:28.737 7f3ea27fc700 1 -- 172.21.2.112:0/2834755786 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3ea4000c60 con 0x7f3eb00b79a0 2021-02-10T11:44:28.887 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:28.886 7f3eb7c4d700 1 -- 172.21.2.112:0/2834755786 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f3eb007ffa0 con 0x7f3eb00b79a0 2021-02-10T11:44:28.891 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:28.890 7f3ea27fc700 1 -- 172.21.2.112:0/2834755786 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v38)=0 v38) v1 ==== 122+0+0 (secure 0 0 0) 0x7f3ea4002060 con 0x7f3eb00b79a0 2021-02-10T11:44:28.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:28.895 7f3eb7c4d700 1 -- 172.21.2.112:0/2834755786 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3e98018730 msgr2=0x7f3e9801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:28.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:28.895 7f3eb7c4d700 1 --2- 172.21.2.112:0/2834755786 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3e98018730 0x7f3e9801abc0 crc :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:28.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:28.895 7f3eb7c4d700 1 -- 172.21.2.112:0/2834755786 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3eb00b79a0 msgr2=0x7f3eb00b7da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:28.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:28.895 7f3eb7c4d700 1 --2- 172.21.2.112:0/2834755786 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3eb00b79a0 0x7f3eb00b7da0 secure :-1 s=READY pgs=2609 cs=0 l=1 rev1=1 rx=0x7f3ea400b5b0 tx=0x7f3ea400b950).stop 2021-02-10T11:44:28.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:28.895 7f3eb7c4d700 1 -- 172.21.2.112:0/2834755786 shutdown_connections 2021-02-10T11:44:28.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:28.895 7f3eb7c4d700 1 --2- 172.21.2.112:0/2834755786 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3e98018730 0x7f3e9801abc0 unknown :-1 s=CLOSED pgs=1192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:28.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:28.895 7f3eb7c4d700 1 --2- 172.21.2.112:0/2834755786 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3eb00b9e40 0x7f3eb00ba240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:28.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:28.895 7f3eb7c4d700 1 --2- 172.21.2.112:0/2834755786 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3eb00b79a0 0x7f3eb00b7da0 unknown :-1 s=CLOSED pgs=2609 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:28.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:28.895 7f3eb7c4d700 1 --2- 172.21.2.112:0/2834755786 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3eb00c3500 0x7f3eb00a2e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:28.897 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:28.895 7f3eb7c4d700 1 -- 172.21.2.112:0/2834755786 >> 172.21.2.112:0/2834755786 conn(0x7f3eb0080ba0 msgr2=0x7f3eb00ffca0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:28.897 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:28.895 7f3eb7c4d700 1 -- 172.21.2.112:0/2834755786 shutdown_connections 2021-02-10T11:44:28.897 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:28.895 7f3eb7c4d700 1 -- 172.21.2.112:0/2834755786 wait complete. 2021-02-10T11:44:28.904 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:44:28.905 DEBUG:teuthology.orchestra.run.gibba012:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:44:29.251 DEBUG:teuthology.orchestra.run.gibba020:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:44:29.597 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:44:29.597 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:44:29.597 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:44:29.598 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:44:29.932 INFO:teuthology.orchestra.run.gibba012.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.0’: No such file or directory 2021-02-10T11:44:29.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:44:29.933 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2021-02-10T11:44:29.933 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:44:29.933 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:44:29.933 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:44:30.266 INFO:teuthology.orchestra.run.gibba012.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: No such file or directory 2021-02-10T11:44:30.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:44:30.267 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2021-02-10T11:44:30.267 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:44:30.267 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:44:30.267 DEBUG:teuthology.orchestra.run.gibba020:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:44:30.600 INFO:teuthology.orchestra.run.gibba020.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:44:30.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:44:30.601 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:44:30.601 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:44:30.601 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:44:30.601 DEBUG:teuthology.orchestra.run.gibba020:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:44:30.936 INFO:teuthology.orchestra.run.gibba020.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:44:30.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:44:30.937 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:44:30.937 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json-pretty 2021-02-10T11:44:31.365 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:31.348 7f1912eb7700 1 Processor -- start 2021-02-10T11:44:31.366 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:31.348 7f1912eb7700 1 -- start start 2021-02-10T11:44:31.366 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:31.349 7f1912eb7700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f190c0d0340 0x7f190c0ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:31.366 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:31.349 7f1912eb7700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f190c0d1c70 0x7f190c0efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:31.366 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:31.349 7f1912eb7700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f190c0f0b10 con 0x7f190c0d1870 2021-02-10T11:44:31.366 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:31.349 7f1912eb7700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f190c0e5ed0 con 0x7f190c0d0340 2021-02-10T11:44:31.366 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:31.349 7f1912eb7700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f190c1b0be0 con 0x7f190c0d1c70 2021-02-10T11:44:31.366 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:31.349 7f190bfff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f190c0d0340 0x7f190c0ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:31.366 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:31.349 7f1910cd3700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f190c0d1870 0x7f190c1b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:57360/0 (socket says 172.21.2.112:57360) 2021-02-10T11:44:31.366 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:31.349 7f1910cd3700 1 -- 172.21.2.112:0/3056585608 learned_addr learned my addr 172.21.2.112:0/3056585608 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:44:31.367 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:31.349 7f19114d4700 1 --2- 172.21.2.112:0/3056585608 >> v2:172.21.2.120:3300/0 conn(0x7f190c0d1c70 0x7f190c0efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:31.367 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:31.349 7f190bfff700 1 -- 172.21.2.112:0/3056585608 >> v2:172.21.2.120:3300/0 conn(0x7f190c0d1c70 msgr2=0x7f190c0efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:31.367 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:31.349 7f190bfff700 1 --2- 172.21.2.112:0/3056585608 >> v2:172.21.2.120:3300/0 conn(0x7f190c0d1c70 0x7f190c0efc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:31.367 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:31.349 7f190bfff700 1 -- 172.21.2.112:0/3056585608 >> v1:172.21.2.112:6789/0 conn(0x7f190c0d1870 legacy=0x7f190c1b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:31.367 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:31.349 7f190b7fe700 1 -- 172.21.2.112:0/3056585608 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3112544978 0 0) 0x7f18f4000ce0 con 0x7f190c0d1870 2021-02-10T11:44:31.367 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:31.349 7f190bfff700 1 -- 172.21.2.112:0/3056585608 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f190c0f20a0 con 0x7f190c0d0340 2021-02-10T11:44:31.367 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:31.350 7f190bfff700 1 --2- 172.21.2.112:0/3056585608 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f190c0d0340 0x7f190c0ef640 secure :-1 s=READY pgs=2611 cs=0 l=1 rev1=1 rx=0x7f18fc006650 tx=0x7f18fc009a00).ready entity=mon.0 client_cookie=b279c527e71b471 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:31.367 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:31.350 7f190b7fe700 1 -- 172.21.2.112:0/3056585608 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f18fc0009b0 con 0x7f190c0d0340 2021-02-10T11:44:31.367 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:31.350 7f190b7fe700 1 -- 172.21.2.112:0/3056585608 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18fc00c0c0 con 0x7f190c0d0340 2021-02-10T11:44:31.367 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:31.350 7f190b7fe700 1 -- 172.21.2.112:0/3056585608 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f18fc00c520 con 0x7f190c0d0340 2021-02-10T11:44:31.367 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:31.350 7f1912eb7700 1 -- 172.21.2.112:0/3056585608 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f190c0d0340 msgr2=0x7f190c0ef640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:31.367 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:31.350 7f1912eb7700 1 --2- 172.21.2.112:0/3056585608 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f190c0d0340 0x7f190c0ef640 secure :-1 s=READY pgs=2611 cs=0 l=1 rev1=1 rx=0x7f18fc006650 tx=0x7f18fc009a00).stop 2021-02-10T11:44:31.367 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:31.351 7f1912eb7700 1 -- 172.21.2.112:0/3056585608 shutdown_connections 2021-02-10T11:44:31.368 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:31.351 7f1912eb7700 1 --2- 172.21.2.112:0/3056585608 >> v2:172.21.2.120:3300/0 conn(0x7f190c0d1c70 0x7f190c0efc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:31.368 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:31.351 7f1912eb7700 1 --2- 172.21.2.112:0/3056585608 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f190c0d0340 0x7f190c0ef640 unknown :-1 s=CLOSED pgs=2611 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:31.368 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:31.351 7f1912eb7700 1 -- 172.21.2.112:0/3056585608 >> 172.21.2.112:0/3056585608 conn(0x7f190c0ce450 msgr2=0x7f190c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:31.368 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:31.351 7f1912eb7700 1 -- 172.21.2.112:0/3056585608 shutdown_connections 2021-02-10T11:44:31.368 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:31.351 7f1912eb7700 1 -- 172.21.2.112:0/3056585608 wait complete. 2021-02-10T11:44:31.371 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:31.365 7f1912eb7700 1 Processor -- start 2021-02-10T11:44:31.372 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:31.365 7f1912eb7700 1 -- start start 2021-02-10T11:44:31.372 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:31.366 7f1912eb7700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f190c0970b0 0x7f190c0ebe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:31.372 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:31.366 7f1912eb7700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f190c098810 0x7f190c098c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:31.372 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:31.366 7f1912eb7700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f190c099230 0x7f190c0b18c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:31.372 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:31.366 7f1912eb7700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f190c0ec6d0 con 0x7f190c098810 2021-02-10T11:44:31.372 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:31.366 7f1912eb7700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f190c0974b0 con 0x7f190c0970b0 2021-02-10T11:44:31.372 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:31.366 7f1912eb7700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f190c0b1ee0 con 0x7f190c099230 2021-02-10T11:44:31.372 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:31.366 7f190bfff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f190c098810 0x7f190c098c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:31.373 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:31.366 7f190bfff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f190c098810 0x7f190c098c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:57908/0 (socket says 172.21.2.112:57908) 2021-02-10T11:44:31.373 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:31.366 7f190bfff700 1 -- 172.21.2.112:0/3994973022 learned_addr learned my addr 172.21.2.112:0/3994973022 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:44:31.373 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:31.366 7f190bfff700 1 -- 172.21.2.112:0/3994973022 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f190c099230 msgr2=0x7f190c0b18c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:44:31.373 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:31.366 7f190bfff700 1 --2- 172.21.2.112:0/3994973022 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f190c099230 0x7f190c0b18c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:31.373 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:31.366 7f190bfff700 1 -- 172.21.2.112:0/3994973022 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f190c0970b0 msgr2=0x7f190c0ebe30 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:44:31.373 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:31.366 7f190bfff700 1 --2- 172.21.2.112:0/3994973022 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f190c0970b0 0x7f190c0ebe30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:31.373 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:31.366 7f190bfff700 1 -- 172.21.2.112:0/3994973022 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f190c0b2100 con 0x7f190c098810 2021-02-10T11:44:31.374 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:31.366 7f190bfff700 1 --2- 172.21.2.112:0/3994973022 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f190c098810 0x7f190c098c10 secure :-1 s=READY pgs=2612 cs=0 l=1 rev1=1 rx=0x7f18fc00b5e0 tx=0x7f18fc00b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:31.374 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:31.366 7f19097fa700 1 -- 172.21.2.112:0/3994973022 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f18fc0090c0 con 0x7f190c098810 2021-02-10T11:44:31.374 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:31.366 7f19097fa700 1 -- 172.21.2.112:0/3994973022 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18fc0009b0 con 0x7f190c098810 2021-02-10T11:44:31.374 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:31.366 7f19097fa700 1 -- 172.21.2.112:0/3994973022 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f18fc0022a0 con 0x7f190c098810 2021-02-10T11:44:31.374 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:31.366 7f1912eb7700 1 -- 172.21.2.112:0/3994973022 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f190c0b2100 con 0x7f190c098810 2021-02-10T11:44:31.374 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:31.366 7f1912eb7700 1 -- 172.21.2.112:0/3994973022 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f190c0b2690 con 0x7f190c098810 2021-02-10T11:44:31.375 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:31.367 7f19097fa700 1 -- 172.21.2.112:0/3994973022 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f18fc000c90 con 0x7f190c098810 2021-02-10T11:44:31.375 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:31.367 7f19097fa700 1 --2- 172.21.2.112:0/3994973022 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f18f00186d0 0x7f18f001abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:31.375 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:31.367 7f19097fa700 1 -- 172.21.2.112:0/3994973022 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f18fc0327f0 con 0x7f190c098810 2021-02-10T11:44:31.375 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:31.367 7f1910cd3700 1 --2- 172.21.2.112:0/3994973022 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f18f00186d0 0x7f18f001abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:31.375 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:31.368 7f1910cd3700 1 --2- 172.21.2.112:0/3994973022 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f18f00186d0 0x7f18f001abd0 crc :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:31.399 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:31.398 7f1912eb7700 1 -- 172.21.2.112:0/3994973022 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f190c0ebc00 con 0x7f190c098810 2021-02-10T11:44:31.402 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:31.401 7f19097fa700 1 -- 172.21.2.112:0/3994973022 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f18fc0090c0 con 0x7f190c098810 2021-02-10T11:44:31.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:31.525 7f1912eb7700 1 -- 172.21.2.112:0/3994973022 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f190c0f0b10 con 0x7f190c098810 2021-02-10T11:44:31.528 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:31.526 7f19097fa700 1 -- 172.21.2.112:0/3994973022 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v143) v1 ==== 81+0+30374 (secure 0 0 0) 0x7f18fc000940 con 0x7f190c098810 2021-02-10T11:44:31.528 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:44:31.528 INFO:teuthology.orchestra.run.gibba012.stdout:{ 2021-02-10T11:44:31.528 INFO:teuthology.orchestra.run.gibba012.stdout: "epoch": 143, 2021-02-10T11:44:31.528 INFO:teuthology.orchestra.run.gibba012.stdout: "fsid": "f14f004e-0699-4584-ad81-3754258493de", 2021-02-10T11:44:31.528 INFO:teuthology.orchestra.run.gibba012.stdout: "created": "2021-02-10 11:13:26.501424", 2021-02-10T11:44:31.528 INFO:teuthology.orchestra.run.gibba012.stdout: "modified": "2021-02-10 11:43:57.411337", 2021-02-10T11:44:31.528 INFO:teuthology.orchestra.run.gibba012.stdout: "last_up_change": "2021-02-10 11:13:52.780316", 2021-02-10T11:44:31.528 INFO:teuthology.orchestra.run.gibba012.stdout: "last_in_change": "2021-02-10 11:13:52.780316", 2021-02-10T11:44:31.528 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-02-10T11:44:31.529 INFO:teuthology.orchestra.run.gibba012.stdout: "flags_num": 5799936, 2021-02-10T11:44:31.529 INFO:teuthology.orchestra.run.gibba012.stdout: "flags_set": [ 2021-02-10T11:44:31.529 INFO:teuthology.orchestra.run.gibba012.stdout: "pglog_hardlimit", 2021-02-10T11:44:31.529 INFO:teuthology.orchestra.run.gibba012.stdout: "purged_snapdirs", 2021-02-10T11:44:31.529 INFO:teuthology.orchestra.run.gibba012.stdout: "recovery_deletes", 2021-02-10T11:44:31.529 INFO:teuthology.orchestra.run.gibba012.stdout: "sortbitwise" 2021-02-10T11:44:31.529 INFO:teuthology.orchestra.run.gibba012.stdout: ], 2021-02-10T11:44:31.529 INFO:teuthology.orchestra.run.gibba012.stdout: "crush_version": 4, 2021-02-10T11:44:31.529 INFO:teuthology.orchestra.run.gibba012.stdout: "full_ratio": 0.94999998807907104, 2021-02-10T11:44:31.529 INFO:teuthology.orchestra.run.gibba012.stdout: "backfillfull_ratio": 0.89999997615814209, 2021-02-10T11:44:31.529 INFO:teuthology.orchestra.run.gibba012.stdout: "nearfull_ratio": 0.85000002384185791, 2021-02-10T11:44:31.529 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_snapshot": "", 2021-02-10T11:44:31.530 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_max": 31, 2021-02-10T11:44:31.530 INFO:teuthology.orchestra.run.gibba012.stdout: "max_osd": 8, 2021-02-10T11:44:31.530 INFO:teuthology.orchestra.run.gibba012.stdout: "require_min_compat_client": "jewel", 2021-02-10T11:44:31.530 INFO:teuthology.orchestra.run.gibba012.stdout: "min_compat_client": "jewel", 2021-02-10T11:44:31.530 INFO:teuthology.orchestra.run.gibba012.stdout: "require_osd_release": "nautilus", 2021-02-10T11:44:31.530 INFO:teuthology.orchestra.run.gibba012.stdout: "pools": [ 2021-02-10T11:44:31.530 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:44:31.530 INFO:teuthology.orchestra.run.gibba012.stdout: "pool": 1, 2021-02-10T11:44:31.530 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_name": "rbd", 2021-02-10T11:44:31.530 INFO:teuthology.orchestra.run.gibba012.stdout: "create_time": "2021-02-10 11:13:57.508758", 2021-02-10T11:44:31.530 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:44:31.530 INFO:teuthology.orchestra.run.gibba012.stdout: "flags_names": "hashpspool", 2021-02-10T11:44:31.530 INFO:teuthology.orchestra.run.gibba012.stdout: "type": 1, 2021-02-10T11:44:31.531 INFO:teuthology.orchestra.run.gibba012.stdout: "size": 2, 2021-02-10T11:44:31.531 INFO:teuthology.orchestra.run.gibba012.stdout: "min_size": 1, 2021-02-10T11:44:31.531 INFO:teuthology.orchestra.run.gibba012.stdout: "crush_rule": 0, 2021-02-10T11:44:31.531 INFO:teuthology.orchestra.run.gibba012.stdout: "object_hash": 2, 2021-02-10T11:44:31.531 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:44:31.531 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num": 8, 2021-02-10T11:44:31.531 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num": 8, 2021-02-10T11:44:31.531 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num_target": 8, 2021-02-10T11:44:31.531 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_target": 8, 2021-02-10T11:44:31.531 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_pending": 8, 2021-02-10T11:44:31.531 INFO:teuthology.orchestra.run.gibba012.stdout: "last_pg_merge_meta": { 2021-02-10T11:44:31.531 INFO:teuthology.orchestra.run.gibba012.stdout: "source_pgid": "0.0", 2021-02-10T11:44:31.532 INFO:teuthology.orchestra.run.gibba012.stdout: "ready_epoch": 0, 2021-02-10T11:44:31.532 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_started": 0, 2021-02-10T11:44:31.532 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_clean": 0, 2021-02-10T11:44:31.532 INFO:teuthology.orchestra.run.gibba012.stdout: "source_version": "0'0", 2021-02-10T11:44:31.532 INFO:teuthology.orchestra.run.gibba012.stdout: "target_version": "0'0" 2021-02-10T11:44:31.532 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:44:31.532 INFO:teuthology.orchestra.run.gibba012.stdout: "last_change": "16", 2021-02-10T11:44:31.532 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend": "0", 2021-02-10T11:44:31.532 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:44:31.532 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:44:31.532 INFO:teuthology.orchestra.run.gibba012.stdout: "auid": 0, 2021-02-10T11:44:31.532 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:44:31.532 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_seq": 0, 2021-02-10T11:44:31.533 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_epoch": 0, 2021-02-10T11:44:31.533 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_snaps": [], 2021-02-10T11:44:31.533 INFO:teuthology.orchestra.run.gibba012.stdout: "removed_snaps": "[]", 2021-02-10T11:44:31.533 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_bytes": 0, 2021-02-10T11:44:31.533 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_objects": 0, 2021-02-10T11:44:31.533 INFO:teuthology.orchestra.run.gibba012.stdout: "tiers": [], 2021-02-10T11:44:31.533 INFO:teuthology.orchestra.run.gibba012.stdout: "tier_of": -1, 2021-02-10T11:44:31.533 INFO:teuthology.orchestra.run.gibba012.stdout: "read_tier": -1, 2021-02-10T11:44:31.533 INFO:teuthology.orchestra.run.gibba012.stdout: "write_tier": -1, 2021-02-10T11:44:31.533 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_mode": "none", 2021-02-10T11:44:31.533 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_bytes": 0, 2021-02-10T11:44:31.533 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_objects": 0, 2021-02-10T11:44:31.533 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:44:31.534 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:44:31.534 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:44:31.534 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_flush_age": 0, 2021-02-10T11:44:31.534 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_evict_age": 0, 2021-02-10T11:44:31.534 INFO:teuthology.orchestra.run.gibba012.stdout: "erasure_code_profile": "", 2021-02-10T11:44:31.534 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_params": { 2021-02-10T11:44:31.534 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "none" 2021-02-10T11:44:31.534 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:44:31.534 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_period": 0, 2021-02-10T11:44:31.534 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_count": 0, 2021-02-10T11:44:31.534 INFO:teuthology.orchestra.run.gibba012.stdout: "use_gmt_hitset": true, 2021-02-10T11:44:31.534 INFO:teuthology.orchestra.run.gibba012.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:44:31.534 INFO:teuthology.orchestra.run.gibba012.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:44:31.535 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:44:31.535 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:44:31.535 INFO:teuthology.orchestra.run.gibba012.stdout: "grade_table": [], 2021-02-10T11:44:31.535 INFO:teuthology.orchestra.run.gibba012.stdout: "stripe_width": 0, 2021-02-10T11:44:31.535 INFO:teuthology.orchestra.run.gibba012.stdout: "expected_num_objects": 0, 2021-02-10T11:44:31.535 INFO:teuthology.orchestra.run.gibba012.stdout: "fast_read": false, 2021-02-10T11:44:31.535 INFO:teuthology.orchestra.run.gibba012.stdout: "options": {}, 2021-02-10T11:44:31.535 INFO:teuthology.orchestra.run.gibba012.stdout: "application_metadata": { 2021-02-10T11:44:31.535 INFO:teuthology.orchestra.run.gibba012.stdout: "rbd": {} 2021-02-10T11:44:31.535 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:44:31.535 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:44:31.535 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:44:31.535 INFO:teuthology.orchestra.run.gibba012.stdout: "pool": 30, 2021-02-10T11:44:31.536 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_name": "cephfs_metadata", 2021-02-10T11:44:31.536 INFO:teuthology.orchestra.run.gibba012.stdout: "create_time": "2021-02-10 11:43:54.385734", 2021-02-10T11:44:31.536 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:44:31.536 INFO:teuthology.orchestra.run.gibba012.stdout: "flags_names": "hashpspool", 2021-02-10T11:44:31.536 INFO:teuthology.orchestra.run.gibba012.stdout: "type": 1, 2021-02-10T11:44:31.536 INFO:teuthology.orchestra.run.gibba012.stdout: "size": 2, 2021-02-10T11:44:31.536 INFO:teuthology.orchestra.run.gibba012.stdout: "min_size": 1, 2021-02-10T11:44:31.536 INFO:teuthology.orchestra.run.gibba012.stdout: "crush_rule": 0, 2021-02-10T11:44:31.536 INFO:teuthology.orchestra.run.gibba012.stdout: "object_hash": 2, 2021-02-10T11:44:31.536 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:44:31.536 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num": 8, 2021-02-10T11:44:31.536 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num": 8, 2021-02-10T11:44:31.536 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num_target": 8, 2021-02-10T11:44:31.536 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_target": 8, 2021-02-10T11:44:31.537 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_pending": 8, 2021-02-10T11:44:31.537 INFO:teuthology.orchestra.run.gibba012.stdout: "last_pg_merge_meta": { 2021-02-10T11:44:31.537 INFO:teuthology.orchestra.run.gibba012.stdout: "source_pgid": "0.0", 2021-02-10T11:44:31.537 INFO:teuthology.orchestra.run.gibba012.stdout: "ready_epoch": 0, 2021-02-10T11:44:31.537 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_started": 0, 2021-02-10T11:44:31.537 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_clean": 0, 2021-02-10T11:44:31.537 INFO:teuthology.orchestra.run.gibba012.stdout: "source_version": "0'0", 2021-02-10T11:44:31.537 INFO:teuthology.orchestra.run.gibba012.stdout: "target_version": "0'0" 2021-02-10T11:44:31.537 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:44:31.537 INFO:teuthology.orchestra.run.gibba012.stdout: "last_change": "142", 2021-02-10T11:44:31.537 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend": "0", 2021-02-10T11:44:31.537 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:44:31.537 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:44:31.538 INFO:teuthology.orchestra.run.gibba012.stdout: "auid": 0, 2021-02-10T11:44:31.538 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:44:31.538 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_seq": 0, 2021-02-10T11:44:31.538 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_epoch": 0, 2021-02-10T11:44:31.538 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_snaps": [], 2021-02-10T11:44:31.538 INFO:teuthology.orchestra.run.gibba012.stdout: "removed_snaps": "[]", 2021-02-10T11:44:31.538 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_bytes": 0, 2021-02-10T11:44:31.538 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_objects": 0, 2021-02-10T11:44:31.538 INFO:teuthology.orchestra.run.gibba012.stdout: "tiers": [], 2021-02-10T11:44:31.538 INFO:teuthology.orchestra.run.gibba012.stdout: "tier_of": -1, 2021-02-10T11:44:31.538 INFO:teuthology.orchestra.run.gibba012.stdout: "read_tier": -1, 2021-02-10T11:44:31.538 INFO:teuthology.orchestra.run.gibba012.stdout: "write_tier": -1, 2021-02-10T11:44:31.538 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_mode": "none", 2021-02-10T11:44:31.539 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_bytes": 0, 2021-02-10T11:44:31.539 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_objects": 0, 2021-02-10T11:44:31.539 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:44:31.539 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:44:31.539 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:44:31.539 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_flush_age": 0, 2021-02-10T11:44:31.539 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_evict_age": 0, 2021-02-10T11:44:31.539 INFO:teuthology.orchestra.run.gibba012.stdout: "erasure_code_profile": "", 2021-02-10T11:44:31.539 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_params": { 2021-02-10T11:44:31.539 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "none" 2021-02-10T11:44:31.539 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:44:31.539 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_period": 0, 2021-02-10T11:44:31.539 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_count": 0, 2021-02-10T11:44:31.540 INFO:teuthology.orchestra.run.gibba012.stdout: "use_gmt_hitset": true, 2021-02-10T11:44:31.540 INFO:teuthology.orchestra.run.gibba012.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:44:31.540 INFO:teuthology.orchestra.run.gibba012.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:44:31.540 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:44:31.540 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:44:31.540 INFO:teuthology.orchestra.run.gibba012.stdout: "grade_table": [], 2021-02-10T11:44:31.540 INFO:teuthology.orchestra.run.gibba012.stdout: "stripe_width": 0, 2021-02-10T11:44:31.540 INFO:teuthology.orchestra.run.gibba012.stdout: "expected_num_objects": 0, 2021-02-10T11:44:31.540 INFO:teuthology.orchestra.run.gibba012.stdout: "fast_read": false, 2021-02-10T11:44:31.540 INFO:teuthology.orchestra.run.gibba012.stdout: "options": { 2021-02-10T11:44:31.540 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_autoscale_bias": 4, 2021-02-10T11:44:31.540 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_min": 16, 2021-02-10T11:44:31.540 INFO:teuthology.orchestra.run.gibba012.stdout: "recovery_priority": 5 2021-02-10T11:44:31.541 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:44:31.541 INFO:teuthology.orchestra.run.gibba012.stdout: "application_metadata": { 2021-02-10T11:44:31.541 INFO:teuthology.orchestra.run.gibba012.stdout: "cephfs": { 2021-02-10T11:44:31.541 INFO:teuthology.orchestra.run.gibba012.stdout: "metadata": "cephfs" 2021-02-10T11:44:31.541 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:44:31.541 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:44:31.541 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:44:31.541 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:44:31.541 INFO:teuthology.orchestra.run.gibba012.stdout: "pool": 31, 2021-02-10T11:44:31.541 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_name": "cephfs_data", 2021-02-10T11:44:31.541 INFO:teuthology.orchestra.run.gibba012.stdout: "create_time": "2021-02-10 11:43:55.453740", 2021-02-10T11:44:31.541 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:44:31.541 INFO:teuthology.orchestra.run.gibba012.stdout: "flags_names": "hashpspool", 2021-02-10T11:44:31.542 INFO:teuthology.orchestra.run.gibba012.stdout: "type": 1, 2021-02-10T11:44:31.542 INFO:teuthology.orchestra.run.gibba012.stdout: "size": 2, 2021-02-10T11:44:31.542 INFO:teuthology.orchestra.run.gibba012.stdout: "min_size": 1, 2021-02-10T11:44:31.542 INFO:teuthology.orchestra.run.gibba012.stdout: "crush_rule": 0, 2021-02-10T11:44:31.542 INFO:teuthology.orchestra.run.gibba012.stdout: "object_hash": 2, 2021-02-10T11:44:31.542 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:44:31.542 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num": 8, 2021-02-10T11:44:31.542 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num": 8, 2021-02-10T11:44:31.542 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num_target": 8, 2021-02-10T11:44:31.542 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_target": 8, 2021-02-10T11:44:31.542 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_pending": 8, 2021-02-10T11:44:31.542 INFO:teuthology.orchestra.run.gibba012.stdout: "last_pg_merge_meta": { 2021-02-10T11:44:31.542 INFO:teuthology.orchestra.run.gibba012.stdout: "source_pgid": "0.0", 2021-02-10T11:44:31.543 INFO:teuthology.orchestra.run.gibba012.stdout: "ready_epoch": 0, 2021-02-10T11:44:31.543 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_started": 0, 2021-02-10T11:44:31.543 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_clean": 0, 2021-02-10T11:44:31.543 INFO:teuthology.orchestra.run.gibba012.stdout: "source_version": "0'0", 2021-02-10T11:44:31.543 INFO:teuthology.orchestra.run.gibba012.stdout: "target_version": "0'0" 2021-02-10T11:44:31.543 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:44:31.543 INFO:teuthology.orchestra.run.gibba012.stdout: "last_change": "143", 2021-02-10T11:44:31.543 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend": "0", 2021-02-10T11:44:31.543 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:44:31.543 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:44:31.543 INFO:teuthology.orchestra.run.gibba012.stdout: "auid": 0, 2021-02-10T11:44:31.543 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:44:31.543 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_seq": 0, 2021-02-10T11:44:31.544 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_epoch": 0, 2021-02-10T11:44:31.544 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_snaps": [], 2021-02-10T11:44:31.544 INFO:teuthology.orchestra.run.gibba012.stdout: "removed_snaps": "[]", 2021-02-10T11:44:31.544 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_bytes": 0, 2021-02-10T11:44:31.544 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_objects": 0, 2021-02-10T11:44:31.544 INFO:teuthology.orchestra.run.gibba012.stdout: "tiers": [], 2021-02-10T11:44:31.544 INFO:teuthology.orchestra.run.gibba012.stdout: "tier_of": -1, 2021-02-10T11:44:31.544 INFO:teuthology.orchestra.run.gibba012.stdout: "read_tier": -1, 2021-02-10T11:44:31.544 INFO:teuthology.orchestra.run.gibba012.stdout: "write_tier": -1, 2021-02-10T11:44:31.544 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_mode": "none", 2021-02-10T11:44:31.544 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_bytes": 0, 2021-02-10T11:44:31.544 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_objects": 0, 2021-02-10T11:44:31.544 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:44:31.544 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:44:31.545 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:44:31.545 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_flush_age": 0, 2021-02-10T11:44:31.545 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_evict_age": 0, 2021-02-10T11:44:31.545 INFO:teuthology.orchestra.run.gibba012.stdout: "erasure_code_profile": "", 2021-02-10T11:44:31.545 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_params": { 2021-02-10T11:44:31.545 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "none" 2021-02-10T11:44:31.545 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:44:31.545 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_period": 0, 2021-02-10T11:44:31.545 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_count": 0, 2021-02-10T11:44:31.545 INFO:teuthology.orchestra.run.gibba012.stdout: "use_gmt_hitset": true, 2021-02-10T11:44:31.545 INFO:teuthology.orchestra.run.gibba012.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:44:31.545 INFO:teuthology.orchestra.run.gibba012.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:44:31.546 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:44:31.546 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:44:31.546 INFO:teuthology.orchestra.run.gibba012.stdout: "grade_table": [], 2021-02-10T11:44:31.546 INFO:teuthology.orchestra.run.gibba012.stdout: "stripe_width": 0, 2021-02-10T11:44:31.546 INFO:teuthology.orchestra.run.gibba012.stdout: "expected_num_objects": 0, 2021-02-10T11:44:31.546 INFO:teuthology.orchestra.run.gibba012.stdout: "fast_read": false, 2021-02-10T11:44:31.546 INFO:teuthology.orchestra.run.gibba012.stdout: "options": {}, 2021-02-10T11:44:31.546 INFO:teuthology.orchestra.run.gibba012.stdout: "application_metadata": { 2021-02-10T11:44:31.546 INFO:teuthology.orchestra.run.gibba012.stdout: "cephfs": { 2021-02-10T11:44:31.546 INFO:teuthology.orchestra.run.gibba012.stdout: "data": "cephfs" 2021-02-10T11:44:31.546 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:44:31.546 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:44:31.546 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:44:31.547 INFO:teuthology.orchestra.run.gibba012.stdout: ], 2021-02-10T11:44:31.547 INFO:teuthology.orchestra.run.gibba012.stdout: "osds": [ 2021-02-10T11:44:31.547 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:44:31.547 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 0, 2021-02-10T11:44:31.547 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "95444b8f-6370-4a1d-ae5f-df62651b77da", 2021-02-10T11:44:31.547 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:44:31.547 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:44:31.547 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:44:31.547 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:44:31.547 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:44:31.547 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:44:31.547 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:44:31.547 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 141, 2021-02-10T11:44:31.548 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:44:31.548 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:44:31.548 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:44:31.548 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:44:31.548 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:44:31.548 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:44:31.548 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6817", 2021-02-10T11:44:31.548 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:44:31.548 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:44:31.548 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:44:31.548 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:44:31.548 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6818", 2021-02-10T11:44:31.548 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:44:31.549 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:44:31.549 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:44:31.549 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:44:31.549 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:44:31.549 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:44:31.549 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:44:31.549 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:44:31.549 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6820", 2021-02-10T11:44:31.549 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:44:31.549 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:44:31.549 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:44:31.549 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:44:31.550 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6822", 2021-02-10T11:44:31.550 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:44:31.550 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:44:31.550 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:44:31.550 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:44:31.550 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:44:31.550 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:44:31.550 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:44:31.550 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:44:31.550 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6828", 2021-02-10T11:44:31.550 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:44:31.550 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:44:31.550 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:44:31.551 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:44:31.551 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6830", 2021-02-10T11:44:31.551 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:44:31.551 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:44:31.551 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:44:31.551 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:44:31.551 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:44:31.551 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:44:31.551 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:44:31.551 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:44:31.551 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6824", 2021-02-10T11:44:31.551 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:44:31.551 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:44:31.552 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:44:31.552 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:44:31.552 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6826", 2021-02-10T11:44:31.552 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:44:31.552 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:44:31.552 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:44:31.552 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:44:31.552 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.112:6818/36030", 2021-02-10T11:44:31.552 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.112:6822/36030", 2021-02-10T11:44:31.552 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.112:6830/36030", 2021-02-10T11:44:31.552 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.112:6826/36030", 2021-02-10T11:44:31.552 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:44:31.552 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:44:31.553 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:44:31.553 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:44:31.553 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:44:31.553 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:44:31.553 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 1, 2021-02-10T11:44:31.553 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "a23e217e-0d8e-486e-be07-712adfabd262", 2021-02-10T11:44:31.553 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:44:31.553 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:44:31.553 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:44:31.553 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:44:31.553 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:44:31.553 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:44:31.553 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:44:31.554 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 141, 2021-02-10T11:44:31.554 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:44:31.554 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:44:31.554 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:44:31.554 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:44:31.554 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:44:31.554 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:44:31.554 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6800", 2021-02-10T11:44:31.554 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:44:31.554 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:44:31.554 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:44:31.554 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:44:31.554 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6802", 2021-02-10T11:44:31.555 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:44:31.555 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:44:31.555 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:44:31.555 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:44:31.555 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:44:31.555 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:44:31.555 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:44:31.555 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:44:31.555 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6804", 2021-02-10T11:44:31.555 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:44:31.555 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:44:31.555 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:44:31.555 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:44:31.556 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6806", 2021-02-10T11:44:31.556 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:44:31.556 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:44:31.556 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:44:31.556 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:44:31.556 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:44:31.556 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:44:31.556 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:44:31.556 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:44:31.556 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6812", 2021-02-10T11:44:31.556 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:44:31.556 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:44:31.556 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:44:31.557 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:44:31.557 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6814", 2021-02-10T11:44:31.557 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:44:31.557 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:44:31.557 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:44:31.557 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:44:31.557 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:44:31.557 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:44:31.557 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:44:31.557 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:44:31.557 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6808", 2021-02-10T11:44:31.557 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:44:31.557 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:44:31.558 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:44:31.558 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:44:31.558 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6810", 2021-02-10T11:44:31.558 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:44:31.558 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:44:31.558 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:44:31.558 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:44:31.558 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.112:6802/36029", 2021-02-10T11:44:31.558 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.112:6806/36029", 2021-02-10T11:44:31.558 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.112:6814/36029", 2021-02-10T11:44:31.558 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.112:6810/36029", 2021-02-10T11:44:31.558 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:44:31.559 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:44:31.559 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:44:31.559 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:44:31.559 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:44:31.559 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:44:31.559 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 2, 2021-02-10T11:44:31.559 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "026afa8a-c0f1-4b35-9a76-53c00fe0c2fc", 2021-02-10T11:44:31.559 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:44:31.559 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:44:31.559 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:44:31.559 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:44:31.559 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:44:31.559 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:44:31.560 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:44:31.560 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 140, 2021-02-10T11:44:31.560 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:44:31.560 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:44:31.560 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:44:31.560 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:44:31.560 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:44:31.560 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:44:31.560 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6801", 2021-02-10T11:44:31.560 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:44:31.560 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:44:31.560 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:44:31.560 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:44:31.561 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6803", 2021-02-10T11:44:31.561 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:44:31.561 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:44:31.561 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:44:31.561 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:44:31.561 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:44:31.561 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:44:31.561 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:44:31.561 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:44:31.561 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6805", 2021-02-10T11:44:31.561 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:44:31.561 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:44:31.561 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:44:31.562 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:44:31.562 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6807", 2021-02-10T11:44:31.562 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:44:31.562 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:44:31.562 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:44:31.562 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:44:31.562 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:44:31.562 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:44:31.562 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:44:31.562 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:44:31.562 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6813", 2021-02-10T11:44:31.562 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:44:31.562 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:44:31.563 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:44:31.563 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:44:31.563 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6815", 2021-02-10T11:44:31.563 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:44:31.563 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:44:31.563 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:44:31.563 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:44:31.563 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:44:31.563 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:44:31.563 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:44:31.563 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:44:31.563 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6809", 2021-02-10T11:44:31.563 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:44:31.563 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:44:31.564 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:44:31.564 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:44:31.564 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6811", 2021-02-10T11:44:31.564 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:44:31.564 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:44:31.564 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:44:31.564 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:44:31.564 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.112:6803/36031", 2021-02-10T11:44:31.564 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.112:6807/36031", 2021-02-10T11:44:31.564 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.112:6815/36031", 2021-02-10T11:44:31.564 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.112:6811/36031", 2021-02-10T11:44:31.564 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:44:31.565 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:44:31.565 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:44:31.565 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:44:31.565 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:44:31.565 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:44:31.565 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 3, 2021-02-10T11:44:31.565 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "58795682-e00d-4173-bb84-f7075ca89a11", 2021-02-10T11:44:31.565 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:44:31.565 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:44:31.565 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:44:31.565 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:44:31.565 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:44:31.565 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:44:31.566 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:44:31.566 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 139, 2021-02-10T11:44:31.566 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:44:31.566 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:44:31.566 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:44:31.566 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:44:31.566 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:44:31.566 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:44:31.566 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6816", 2021-02-10T11:44:31.566 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:44:31.566 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:44:31.566 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:44:31.566 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:44:31.567 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6819", 2021-02-10T11:44:31.567 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:44:31.567 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:44:31.567 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:44:31.567 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:44:31.567 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:44:31.567 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:44:31.567 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:44:31.567 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:44:31.567 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6821", 2021-02-10T11:44:31.567 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:44:31.567 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:44:31.567 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:44:31.568 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:44:31.568 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6823", 2021-02-10T11:44:31.568 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:44:31.568 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:44:31.568 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:44:31.568 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:44:31.568 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:44:31.568 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:44:31.568 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:44:31.568 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:44:31.568 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6829", 2021-02-10T11:44:31.568 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:44:31.568 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:44:31.569 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:44:31.569 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:44:31.569 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6831", 2021-02-10T11:44:31.569 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:44:31.569 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:44:31.569 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:44:31.569 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:44:31.569 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:44:31.569 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:44:31.569 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:44:31.569 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:44:31.569 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6825", 2021-02-10T11:44:31.569 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:44:31.570 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:44:31.570 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:44:31.570 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:44:31.570 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6827", 2021-02-10T11:44:31.570 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:44:31.570 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:44:31.570 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:44:31.570 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:44:31.570 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.112:6819/36028", 2021-02-10T11:44:31.570 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.112:6823/36028", 2021-02-10T11:44:31.570 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.112:6831/36028", 2021-02-10T11:44:31.570 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.112:6827/36028", 2021-02-10T11:44:31.570 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:44:31.571 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:44:31.571 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:44:31.571 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:44:31.571 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:44:31.571 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:44:31.571 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 4, 2021-02-10T11:44:31.571 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "6763f8d6-53dc-4e45-bdf6-a6eabcf81e39", 2021-02-10T11:44:31.571 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:44:31.571 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:44:31.571 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:44:31.571 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:44:31.571 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:44:31.571 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:44:31.572 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:44:31.572 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 141, 2021-02-10T11:44:31.572 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:44:31.572 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:44:31.572 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:44:31.572 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:44:31.572 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:44:31.572 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:44:31.572 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6810", 2021-02-10T11:44:31.572 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:44:31.572 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:44:31.572 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:44:31.572 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:44:31.573 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6813", 2021-02-10T11:44:31.573 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:44:31.573 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:44:31.573 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:44:31.573 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:44:31.573 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:44:31.573 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:44:31.573 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:44:31.573 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:44:31.573 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6816", 2021-02-10T11:44:31.573 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:44:31.573 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:44:31.573 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:44:31.573 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:44:31.574 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6819", 2021-02-10T11:44:31.574 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:44:31.574 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:44:31.574 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:44:31.574 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:44:31.574 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:44:31.574 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:44:31.574 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:44:31.574 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:44:31.574 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6825", 2021-02-10T11:44:31.575 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:44:31.575 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:44:31.575 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:44:31.575 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:44:31.575 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6827", 2021-02-10T11:44:31.575 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:44:31.575 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:44:31.575 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:44:31.576 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:44:31.576 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:44:31.576 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:44:31.576 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:44:31.576 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:44:31.576 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6821", 2021-02-10T11:44:31.576 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:44:31.576 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:44:31.577 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:44:31.577 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:44:31.577 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6823", 2021-02-10T11:44:31.577 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:44:31.577 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:44:31.577 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:44:31.577 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:44:31.577 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.120:6813/36388", 2021-02-10T11:44:31.577 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.120:6819/36388", 2021-02-10T11:44:31.578 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.120:6827/36388", 2021-02-10T11:44:31.578 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.120:6823/36388", 2021-02-10T11:44:31.578 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:44:31.578 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:44:31.578 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:44:31.578 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:44:31.578 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:44:31.578 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:44:31.579 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 5, 2021-02-10T11:44:31.579 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "5d0e4dd6-0cb0-4bf8-8314-626acaa1b544", 2021-02-10T11:44:31.579 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:44:31.579 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:44:31.579 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:44:31.579 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:44:31.579 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:44:31.580 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:44:31.580 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:44:31.580 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 139, 2021-02-10T11:44:31.580 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:44:31.580 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:44:31.580 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:44:31.580 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:44:31.581 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:44:31.581 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:44:31.581 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6804", 2021-02-10T11:44:31.581 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:44:31.581 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:44:31.581 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:44:31.581 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:44:31.581 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6806", 2021-02-10T11:44:31.582 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:44:31.582 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:44:31.582 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:44:31.582 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:44:31.582 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:44:31.582 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:44:31.582 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:44:31.583 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:44:31.583 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6809", 2021-02-10T11:44:31.583 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:44:31.583 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:44:31.583 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:44:31.583 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:44:31.583 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6812", 2021-02-10T11:44:31.584 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:44:31.584 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:44:31.584 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:44:31.584 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:44:31.584 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:44:31.584 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:44:31.584 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:44:31.584 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:44:31.584 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6820", 2021-02-10T11:44:31.584 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:44:31.584 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:44:31.584 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:44:31.584 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:44:31.584 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6822", 2021-02-10T11:44:31.585 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:44:31.585 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:44:31.585 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:44:31.585 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:44:31.585 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:44:31.585 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:44:31.585 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:44:31.585 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:44:31.585 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6815", 2021-02-10T11:44:31.585 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:44:31.585 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:44:31.585 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:44:31.585 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:44:31.585 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6818", 2021-02-10T11:44:31.586 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:44:31.586 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:44:31.586 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:44:31.586 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:44:31.586 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.120:6806/36387", 2021-02-10T11:44:31.586 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.120:6812/36387", 2021-02-10T11:44:31.586 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.120:6822/36387", 2021-02-10T11:44:31.586 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.120:6818/36387", 2021-02-10T11:44:31.586 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:44:31.586 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:44:31.586 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:44:31.586 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:44:31.587 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:44:31.587 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:44:31.587 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 6, 2021-02-10T11:44:31.587 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "b010e187-4541-4b00-80dd-f948f6ab8f15", 2021-02-10T11:44:31.587 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:44:31.587 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:44:31.587 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:44:31.587 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:44:31.587 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:44:31.587 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:44:31.587 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:44:31.587 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 141, 2021-02-10T11:44:31.587 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:44:31.588 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:44:31.588 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:44:31.588 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:44:31.588 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:44:31.588 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:44:31.588 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6824", 2021-02-10T11:44:31.588 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:44:31.588 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:44:31.588 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:44:31.588 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:44:31.588 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6826", 2021-02-10T11:44:31.588 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:44:31.588 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:44:31.588 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:44:31.589 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:44:31.589 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:44:31.589 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:44:31.589 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:44:31.589 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:44:31.589 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6828", 2021-02-10T11:44:31.589 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:44:31.589 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:44:31.589 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:44:31.589 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:44:31.589 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6829", 2021-02-10T11:44:31.589 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:44:31.589 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:44:31.590 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:44:31.590 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:44:31.590 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:44:31.590 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:44:31.590 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:44:31.590 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:44:31.590 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6832", 2021-02-10T11:44:31.590 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:44:31.590 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:44:31.590 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:44:31.590 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:44:31.590 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6833", 2021-02-10T11:44:31.591 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:44:31.591 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:44:31.591 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:44:31.591 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:44:31.591 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:44:31.591 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:44:31.591 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:44:31.591 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:44:31.591 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6830", 2021-02-10T11:44:31.591 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:44:31.591 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:44:31.591 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:44:31.591 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:44:31.592 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6831", 2021-02-10T11:44:31.592 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:44:31.592 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:44:31.592 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:44:31.592 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:44:31.592 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.120:6826/36385", 2021-02-10T11:44:31.592 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.120:6829/36385", 2021-02-10T11:44:31.592 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.120:6833/36385", 2021-02-10T11:44:31.592 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.120:6831/36385", 2021-02-10T11:44:31.592 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:44:31.592 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:44:31.592 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:44:31.592 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:44:31.592 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:44:31.593 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:44:31.593 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 7, 2021-02-10T11:44:31.593 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "671e2772-33b7-497e-8e44-2178bd850bcf", 2021-02-10T11:44:31.593 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:44:31.593 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:44:31.593 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:44:31.593 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:44:31.593 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:44:31.593 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:44:31.593 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:44:31.593 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 140, 2021-02-10T11:44:31.593 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:44:31.593 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:44:31.594 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:44:31.594 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:44:31.594 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:44:31.594 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:44:31.594 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6802", 2021-02-10T11:44:31.594 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:44:31.594 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:44:31.594 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:44:31.594 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:44:31.594 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6803", 2021-02-10T11:44:31.594 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:44:31.594 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:44:31.595 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:44:31.595 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:44:31.595 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:44:31.595 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:44:31.595 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:44:31.595 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:44:31.595 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6805", 2021-02-10T11:44:31.595 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:44:31.595 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:44:31.595 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:44:31.595 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:44:31.595 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6807", 2021-02-10T11:44:31.595 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:44:31.595 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:44:31.596 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:44:31.596 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:44:31.596 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:44:31.596 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:44:31.596 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:44:31.596 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:44:31.596 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6814", 2021-02-10T11:44:31.596 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:44:31.596 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:44:31.596 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:44:31.596 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:44:31.597 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6817", 2021-02-10T11:44:31.597 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:44:31.597 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:44:31.597 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:44:31.597 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:44:31.597 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:44:31.597 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:44:31.597 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:44:31.597 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:44:31.597 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6808", 2021-02-10T11:44:31.597 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:44:31.597 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:44:31.597 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:44:31.597 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:44:31.598 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6811", 2021-02-10T11:44:31.598 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:44:31.598 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:44:31.598 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:44:31.598 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:44:31.598 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.120:6803/36386", 2021-02-10T11:44:31.598 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.120:6807/36386", 2021-02-10T11:44:31.598 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.120:6817/36386", 2021-02-10T11:44:31.598 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.120:6811/36386", 2021-02-10T11:44:31.598 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:44:31.598 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:44:31.598 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:44:31.598 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:44:31.599 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:44:31.599 INFO:teuthology.orchestra.run.gibba012.stdout: ], 2021-02-10T11:44:31.599 INFO:teuthology.orchestra.run.gibba012.stdout: "osd_xinfo": [ 2021-02-10T11:44:31.599 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:44:31.599 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 0, 2021-02-10T11:44:31.599 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:44:31.599 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:44:31.599 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:44:31.599 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:44:31.599 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:44:31.599 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:44:31.599 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:44:31.599 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 1, 2021-02-10T11:44:31.599 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:44:31.600 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:44:31.600 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:44:31.600 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:44:31.600 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:44:31.600 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:44:31.600 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:44:31.600 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 2, 2021-02-10T11:44:31.600 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:44:31.600 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:44:31.600 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:44:31.600 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:44:31.600 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:44:31.600 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:44:31.601 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:44:31.601 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 3, 2021-02-10T11:44:31.601 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:44:31.601 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:44:31.601 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:44:31.601 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:44:31.601 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:44:31.601 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:44:31.601 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:44:31.601 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 4, 2021-02-10T11:44:31.601 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:44:31.601 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:44:31.601 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:44:31.602 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:44:31.602 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:44:31.602 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:44:31.602 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:44:31.602 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 5, 2021-02-10T11:44:31.602 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:44:31.602 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:44:31.602 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:44:31.602 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:44:31.602 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:44:31.602 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:44:31.602 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:44:31.603 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 6, 2021-02-10T11:44:31.603 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:44:31.603 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:44:31.603 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:44:31.603 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:44:31.603 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:44:31.603 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:44:31.603 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:44:31.603 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 7, 2021-02-10T11:44:31.603 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:44:31.603 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:44:31.603 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:44:31.603 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:44:31.604 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:44:31.604 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:44:31.604 INFO:teuthology.orchestra.run.gibba012.stdout: ], 2021-02-10T11:44:31.604 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_upmap": [], 2021-02-10T11:44:31.604 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_upmap_items": [], 2021-02-10T11:44:31.604 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_temp": [], 2021-02-10T11:44:31.604 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_temp": [], 2021-02-10T11:44:31.604 INFO:teuthology.orchestra.run.gibba012.stdout: "blacklist": {}, 2021-02-10T11:44:31.604 INFO:teuthology.orchestra.run.gibba012.stdout: "erasure_code_profiles": { 2021-02-10T11:44:31.604 INFO:teuthology.orchestra.run.gibba012.stdout: "default": { 2021-02-10T11:44:31.604 INFO:teuthology.orchestra.run.gibba012.stdout: "crush-failure-domain": "osd", 2021-02-10T11:44:31.604 INFO:teuthology.orchestra.run.gibba012.stdout: "k": "2", 2021-02-10T11:44:31.604 INFO:teuthology.orchestra.run.gibba012.stdout: "m": "1", 2021-02-10T11:44:31.604 INFO:teuthology.orchestra.run.gibba012.stdout: "plugin": "jerasure", 2021-02-10T11:44:31.605 INFO:teuthology.orchestra.run.gibba012.stdout: "ruleset-failure-domain": "osd", 2021-02-10T11:44:31.605 INFO:teuthology.orchestra.run.gibba012.stdout: "technique": "reed_sol_van" 2021-02-10T11:44:31.605 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:44:31.605 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:44:31.605 INFO:teuthology.orchestra.run.gibba012.stdout: "removed_snaps_queue": [], 2021-02-10T11:44:31.605 INFO:teuthology.orchestra.run.gibba012.stdout: "new_removed_snaps": [], 2021-02-10T11:44:31.605 INFO:teuthology.orchestra.run.gibba012.stdout: "new_purged_snaps": [], 2021-02-10T11:44:31.605 INFO:teuthology.orchestra.run.gibba012.stdout: "crush_node_flags": {}, 2021-02-10T11:44:31.605 INFO:teuthology.orchestra.run.gibba012.stdout: "device_class_flags": {} 2021-02-10T11:44:31.605 INFO:teuthology.orchestra.run.gibba012.stdout:} 2021-02-10T11:44:31.606 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:31.528 7f1912eb7700 1 -- 172.21.2.112:0/3994973022 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f18f00186d0 msgr2=0x7f18f001abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:31.607 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:31.528 7f1912eb7700 1 --2- 172.21.2.112:0/3994973022 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f18f00186d0 0x7f18f001abd0 crc :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:31.607 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:31.528 7f1912eb7700 1 -- 172.21.2.112:0/3994973022 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f190c098810 msgr2=0x7f190c098c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:31.607 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:31.528 7f1912eb7700 1 --2- 172.21.2.112:0/3994973022 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f190c098810 0x7f190c098c10 secure :-1 s=READY pgs=2612 cs=0 l=1 rev1=1 rx=0x7f18fc00b5e0 tx=0x7f18fc00b980).stop 2021-02-10T11:44:31.607 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:31.528 7f1912eb7700 1 -- 172.21.2.112:0/3994973022 shutdown_connections 2021-02-10T11:44:31.607 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:31.528 7f1912eb7700 1 --2- 172.21.2.112:0/3994973022 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f18f00186d0 0x7f18f001abd0 unknown :-1 s=CLOSED pgs=1193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:31.607 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:31.528 7f1912eb7700 1 --2- 172.21.2.112:0/3994973022 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f190c099230 0x7f190c0b18c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:31.607 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:31.528 7f1912eb7700 1 --2- 172.21.2.112:0/3994973022 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f190c098810 0x7f190c098c10 unknown :-1 s=CLOSED pgs=2612 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:31.607 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:31.528 7f1912eb7700 1 --2- 172.21.2.112:0/3994973022 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f190c0970b0 0x7f190c0ebe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:31.607 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:31.528 7f1912eb7700 1 -- 172.21.2.112:0/3994973022 >> 172.21.2.112:0/3994973022 conn(0x7f190c19a730 msgr2=0x7f190c19e480 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:31.607 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:31.528 7f1912eb7700 1 -- 172.21.2.112:0/3994973022 shutdown_connections 2021-02-10T11:44:31.607 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:31.528 7f1912eb7700 1 -- 172.21.2.112:0/3994973022 wait complete. 2021-02-10T11:44:31.608 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:44:32.035 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.018 7f17106cc700 1 Processor -- start 2021-02-10T11:44:32.035 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.018 7f17106cc700 1 -- start start 2021-02-10T11:44:32.036 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.019 7f17106cc700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f17080d0340 0x7f17080ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:32.036 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.019 7f17106cc700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f17080d1c70 0x7f17080efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:32.036 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.019 7f17106cc700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17080f0b10 con 0x7f17080d1870 2021-02-10T11:44:32.036 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.019 7f17106cc700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17080e5ed0 con 0x7f17080d1c70 2021-02-10T11:44:32.036 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.019 7f17106cc700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17081b0be0 con 0x7f17080d0340 2021-02-10T11:44:32.036 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.019 7f170ece9700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f17080d1c70 0x7f17080efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:32.036 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.019 7f170ece9700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f17080d1c70 0x7f17080efc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:57916/0 (socket says 172.21.2.112:57916) 2021-02-10T11:44:32.036 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.019 7f170ece9700 1 -- 172.21.2.112:0/1679494951 learned_addr learned my addr 172.21.2.112:0/1679494951 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:44:32.036 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.019 7f170ece9700 1 -- 172.21.2.112:0/1679494951 >> v2:172.21.2.120:3300/0 conn(0x7f17080d0340 msgr2=0x7f17080ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:32.036 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.019 7f170ece9700 1 --2- 172.21.2.112:0/1679494951 >> v2:172.21.2.120:3300/0 conn(0x7f17080d0340 0x7f17080ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:32.037 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.019 7f170ece9700 1 -- 172.21.2.112:0/1679494951 >> v1:172.21.2.120:6789/0 conn(0x7f17080d1870 legacy=0x7f17081b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:32.037 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.019 7f170ece9700 1 -- 172.21.2.112:0/1679494951 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17080f20a0 con 0x7f17080d1c70 2021-02-10T11:44:32.037 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.019 7f170ece9700 1 --2- 172.21.2.112:0/1679494951 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f17080d1c70 0x7f17080efc60 secure :-1 s=READY pgs=2613 cs=0 l=1 rev1=1 rx=0x7f1700006f00 tx=0x7f1700009f10).ready entity=mon.0 client_cookie=2392da38ee1ea59e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:32.037 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.020 7f170d4e6700 1 -- 172.21.2.112:0/1679494951 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f170000b3b0 con 0x7f17080d1c70 2021-02-10T11:44:32.037 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.020 7f170d4e6700 1 -- 172.21.2.112:0/1679494951 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f170000b630 con 0x7f17080d1c70 2021-02-10T11:44:32.037 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.020 7f170d4e6700 1 -- 172.21.2.112:0/1679494951 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f170000bb50 con 0x7f17080d1c70 2021-02-10T11:44:32.037 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.020 7f17106cc700 1 -- 172.21.2.112:0/1679494951 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f17080d1c70 msgr2=0x7f17080efc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:32.037 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.020 7f17106cc700 1 --2- 172.21.2.112:0/1679494951 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f17080d1c70 0x7f17080efc60 secure :-1 s=READY pgs=2613 cs=0 l=1 rev1=1 rx=0x7f1700006f00 tx=0x7f1700009f10).stop 2021-02-10T11:44:32.037 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.020 7f17106cc700 1 -- 172.21.2.112:0/1679494951 shutdown_connections 2021-02-10T11:44:32.037 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.020 7f17106cc700 1 --2- 172.21.2.112:0/1679494951 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f17080d1c70 0x7f17080efc60 unknown :-1 s=CLOSED pgs=2613 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:32.037 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.020 7f17106cc700 1 --2- 172.21.2.112:0/1679494951 >> v2:172.21.2.120:3300/0 conn(0x7f17080d0340 0x7f17080ef640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:32.037 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.020 7f17106cc700 1 -- 172.21.2.112:0/1679494951 >> 172.21.2.112:0/1679494951 conn(0x7f17080ce450 msgr2=0x7f17080d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:32.037 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.020 7f17106cc700 1 -- 172.21.2.112:0/1679494951 shutdown_connections 2021-02-10T11:44:32.038 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.020 7f17106cc700 1 -- 172.21.2.112:0/1679494951 wait complete. 2021-02-10T11:44:32.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.035 7f17106cc700 1 Processor -- start 2021-02-10T11:44:32.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.035 7f17106cc700 1 -- start start 2021-02-10T11:44:32.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.035 7f17106cc700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f170818f0a0 0x7f1708196ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:32.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.035 7f17106cc700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f17081970d0 0x7f17081974d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:32.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.035 7f17106cc700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1708094650 0x7f1708094a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:32.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.035 7f17106cc700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1708095bd0 con 0x7f170818f0a0 2021-02-10T11:44:32.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.035 7f17106cc700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17080ad7c0 con 0x7f17081970d0 2021-02-10T11:44:32.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.035 7f17106cc700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f170818ec70 con 0x7f1708094650 2021-02-10T11:44:32.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.035 7f170e4e8700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f170818f0a0 0x7f1708196ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:32.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.035 7f170e4e8700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f170818f0a0 0x7f1708196ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:57918/0 (socket says 172.21.2.112:57918) 2021-02-10T11:44:32.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.035 7f170e4e8700 1 -- 172.21.2.112:0/2236936246 learned_addr learned my addr 172.21.2.112:0/2236936246 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:44:32.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.036 7f170e4e8700 1 -- 172.21.2.112:0/2236936246 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1708094650 msgr2=0x7f1708094a50 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:44:32.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.036 7f170e4e8700 1 --2- 172.21.2.112:0/2236936246 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1708094650 0x7f1708094a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:32.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.036 7f170e4e8700 1 -- 172.21.2.112:0/2236936246 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f17081970d0 msgr2=0x7f17081974d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:44:32.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.036 7f170e4e8700 1 --2- 172.21.2.112:0/2236936246 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f17081970d0 0x7f17081974d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:32.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.036 7f170e4e8700 1 -- 172.21.2.112:0/2236936246 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1708095070 con 0x7f170818f0a0 2021-02-10T11:44:32.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.036 7f170e4e8700 1 --2- 172.21.2.112:0/2236936246 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f170818f0a0 0x7f1708196ab0 secure :-1 s=READY pgs=2614 cs=0 l=1 rev1=1 rx=0x7f16fc00b5b0 tx=0x7f16fc00b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:32.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.036 7f16faffd700 1 -- 172.21.2.112:0/2236936246 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f16fc00bd40 con 0x7f170818f0a0 2021-02-10T11:44:32.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.036 7f16faffd700 1 -- 172.21.2.112:0/2236936246 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16fc000e20 con 0x7f170818f0a0 2021-02-10T11:44:32.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.036 7f17106cc700 1 -- 172.21.2.112:0/2236936246 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1708095070 con 0x7f170818f0a0 2021-02-10T11:44:32.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.036 7f16faffd700 1 -- 172.21.2.112:0/2236936246 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f16fc0054a0 con 0x7f170818f0a0 2021-02-10T11:44:32.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.036 7f17106cc700 1 -- 172.21.2.112:0/2236936246 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1708095600 con 0x7f170818f0a0 2021-02-10T11:44:32.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.036 7f16faffd700 1 -- 172.21.2.112:0/2236936246 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f16fc006030 con 0x7f170818f0a0 2021-02-10T11:44:32.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.037 7f16faffd700 1 --2- 172.21.2.112:0/2236936246 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f16f0018730 0x7f16f001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:32.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.037 7f16faffd700 1 -- 172.21.2.112:0/2236936246 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f16fc032670 con 0x7f170818f0a0 2021-02-10T11:44:32.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.037 7f170dce7700 1 --2- 172.21.2.112:0/2236936246 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f16f0018730 0x7f16f001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:32.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.038 7f170dce7700 1 --2- 172.21.2.112:0/2236936246 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f16f0018730 0x7f16f001ac10 crc :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:32.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.069 7f17106cc700 1 -- 172.21.2.112:0/2236936246 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17080ee0e0 con 0x7f170818f0a0 2021-02-10T11:44:32.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.071 7f16faffd700 1 -- 172.21.2.112:0/2236936246 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f16fc006240 con 0x7f170818f0a0 2021-02-10T11:44:32.201 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.200 7f17106cc700 1 -- 172.21.2.112:0/2236936246 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f17080ce450 con 0x7f170818f0a0 2021-02-10T11:44:32.202 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.201 7f16faffd700 1 -- 172.21.2.112:0/2236936246 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) v1 ==== 95+0+2108 (secure 0 0 0) 0x7f16fc006030 con 0x7f170818f0a0 2021-02-10T11:44:32.202 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:44:32.202 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":123,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11670,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/1286703435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":1286703435},{"type":"v1","addr":"172.21.2.112:6833","nonce":1286703435}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":119}],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:43:56.409867","modified":"2021-02-10 11:43:58.413644","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8578},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8578":{"gid":8578,"name":"b","rank":0,"incarnation":120,"state":"up:active","state_seq":3,"addr":"172.21.2.120:6835/1504781062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":1504781062},{"type":"v1","addr":"172.21.2.120:6835","nonce":1504781062}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":15}]} 2021-02-10T11:44:32.203 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 123 2021-02-10T11:44:32.204 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.203 7f17106cc700 1 -- 172.21.2.112:0/2236936246 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f16f0018730 msgr2=0x7f16f001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:32.204 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.203 7f17106cc700 1 --2- 172.21.2.112:0/2236936246 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f16f0018730 0x7f16f001ac10 crc :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:32.204 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.203 7f17106cc700 1 -- 172.21.2.112:0/2236936246 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f170818f0a0 msgr2=0x7f1708196ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:32.204 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.203 7f17106cc700 1 --2- 172.21.2.112:0/2236936246 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f170818f0a0 0x7f1708196ab0 secure :-1 s=READY pgs=2614 cs=0 l=1 rev1=1 rx=0x7f16fc00b5b0 tx=0x7f16fc00b950).stop 2021-02-10T11:44:32.204 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.203 7f17106cc700 1 -- 172.21.2.112:0/2236936246 shutdown_connections 2021-02-10T11:44:32.204 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.203 7f17106cc700 1 --2- 172.21.2.112:0/2236936246 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f16f0018730 0x7f16f001ac10 unknown :-1 s=CLOSED pgs=1194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:32.204 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.203 7f17106cc700 1 --2- 172.21.2.112:0/2236936246 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f17081970d0 0x7f17081974d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:32.204 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.203 7f17106cc700 1 --2- 172.21.2.112:0/2236936246 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1708094650 0x7f1708094a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:32.204 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.203 7f17106cc700 1 --2- 172.21.2.112:0/2236936246 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f170818f0a0 0x7f1708196ab0 unknown :-1 s=CLOSED pgs=2614 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:32.204 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.203 7f17106cc700 1 -- 172.21.2.112:0/2236936246 >> 172.21.2.112:0/2236936246 conn(0x7f17081a38c0 msgr2=0x7f17081a2f00 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:32.205 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.203 7f17106cc700 1 -- 172.21.2.112:0/2236936246 shutdown_connections 2021-02-10T11:44:32.205 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.203 7f17106cc700 1 -- 172.21.2.112:0/2236936246 wait complete. 2021-02-10T11:44:32.212 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs fail cephfs 2021-02-10T11:44:32.642 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.625 7f2d7d9b8700 1 Processor -- start 2021-02-10T11:44:32.643 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.625 7f2d7d9b8700 1 -- start start 2021-02-10T11:44:32.643 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.626 7f2d7d9b8700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f2d780d1870 0x7f2d781b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:32.643 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.626 7f2d7d9b8700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2d780f0650 0x7f2d780cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:32.643 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.626 7f2d7d9b8700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d780f0b10 con 0x7f2d781b0a10 2021-02-10T11:44:32.643 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.626 7f2d7d9b8700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d780e5ed0 con 0x7f2d780d1870 2021-02-10T11:44:32.643 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.626 7f2d7d9b8700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d780d1c70 con 0x7f2d780f0650 2021-02-10T11:44:32.643 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.626 7f2d77574700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f2d780d1870 0x7f2d781b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:32.643 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.626 7f2d77574700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f2d780d1870 0x7f2d781b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:57926/0 (socket says 172.21.2.112:57926) 2021-02-10T11:44:32.643 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.626 7f2d77574700 1 -- 172.21.2.112:0/2554763230 learned_addr learned my addr 172.21.2.112:0/2554763230 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:44:32.644 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.626 7f2d77d75700 1 --2- 172.21.2.112:0/2554763230 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2d780f0650 0x7f2d780cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:32.644 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.626 7f2d77574700 1 -- 172.21.2.112:0/2554763230 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2d780f0650 msgr2=0x7f2d780cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:32.644 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.626 7f2d77574700 1 --2- 172.21.2.112:0/2554763230 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2d780f0650 0x7f2d780cfb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:32.644 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.626 7f2d77574700 1 -- 172.21.2.112:0/2554763230 >> v1:172.21.2.120:6789/0 conn(0x7f2d781b0a10 legacy=0x7f2d780cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:32.644 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.626 7f2d77574700 1 -- 172.21.2.112:0/2554763230 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d780d1e90 con 0x7f2d780d1870 2021-02-10T11:44:32.644 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.627 7f2d77574700 1 --2- 172.21.2.112:0/2554763230 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2d780d1870 0x7f2d781b03f0 secure :-1 s=READY pgs=2615 cs=0 l=1 rev1=1 rx=0x7f2d68006700 tx=0x7f2d68009b80).ready entity=mon.0 client_cookie=cf2613a8dd148ed3 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:32.644 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.627 7f2d76572700 1 -- 172.21.2.112:0/2554763230 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2d6800c0c0 con 0x7f2d780d1870 2021-02-10T11:44:32.644 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.627 7f2d76572700 1 -- 172.21.2.112:0/2554763230 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d6800c340 con 0x7f2d780d1870 2021-02-10T11:44:32.644 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.627 7f2d76572700 1 -- 172.21.2.112:0/2554763230 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2d6800c860 con 0x7f2d780d1870 2021-02-10T11:44:32.644 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.627 7f2d7d9b8700 1 -- 172.21.2.112:0/2554763230 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2d780d1870 msgr2=0x7f2d781b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:32.644 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.627 7f2d7d9b8700 1 --2- 172.21.2.112:0/2554763230 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2d780d1870 0x7f2d781b03f0 secure :-1 s=READY pgs=2615 cs=0 l=1 rev1=1 rx=0x7f2d68006700 tx=0x7f2d68009b80).stop 2021-02-10T11:44:32.644 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.627 7f2d7d9b8700 1 -- 172.21.2.112:0/2554763230 shutdown_connections 2021-02-10T11:44:32.645 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.627 7f2d7d9b8700 1 --2- 172.21.2.112:0/2554763230 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2d780f0650 0x7f2d780cfb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:32.645 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.627 7f2d7d9b8700 1 --2- 172.21.2.112:0/2554763230 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2d780d1870 0x7f2d781b03f0 unknown :-1 s=CLOSED pgs=2615 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:32.645 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.627 7f2d7d9b8700 1 -- 172.21.2.112:0/2554763230 >> 172.21.2.112:0/2554763230 conn(0x7f2d780ce450 msgr2=0x7f2d780d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:32.645 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.627 7f2d7d9b8700 1 -- 172.21.2.112:0/2554763230 shutdown_connections 2021-02-10T11:44:32.645 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.627 7f2d7d9b8700 1 -- 172.21.2.112:0/2554763230 wait complete. 2021-02-10T11:44:32.646 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.642 7f2d7d9b8700 1 Processor -- start 2021-02-10T11:44:32.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.642 7f2d7d9b8700 1 -- start start 2021-02-10T11:44:32.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.642 7f2d7d9b8700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2d781a4990 0x7f2d780b2530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:32.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.642 7f2d7d9b8700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2d780b2b50 0x7f2d7809b990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:32.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.643 7f2d7d9b8700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2d7809bfb0 0x7f2d78189aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:32.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.643 7f2d7d9b8700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d7809c3b0 con 0x7f2d780b2b50 2021-02-10T11:44:32.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.643 7f2d7d9b8700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d7818a0c0 con 0x7f2d781a4990 2021-02-10T11:44:32.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.643 7f2d7d9b8700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d7818a2e0 con 0x7f2d7809bfb0 2021-02-10T11:44:32.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.643 7f2d76d73700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2d780b2b50 0x7f2d7809b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:32.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.643 7f2d77574700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2d781a4990 0x7f2d780b2530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:32.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.643 7f2d76d73700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2d780b2b50 0x7f2d7809b990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:57936/0 (socket says 172.21.2.112:57936) 2021-02-10T11:44:32.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.643 7f2d76d73700 1 -- 172.21.2.112:0/53607929 learned_addr learned my addr 172.21.2.112:0/53607929 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:44:32.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.643 7f2d77d75700 1 --2- 172.21.2.112:0/53607929 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2d7809bfb0 0x7f2d78189aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:32.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.643 7f2d77574700 1 -- 172.21.2.112:0/53607929 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2d7809bfb0 msgr2=0x7f2d78189aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:32.648 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.643 7f2d77574700 1 --2- 172.21.2.112:0/53607929 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2d7809bfb0 0x7f2d78189aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:32.648 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.643 7f2d77574700 1 -- 172.21.2.112:0/53607929 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2d780b2b50 msgr2=0x7f2d7809b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:32.648 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.643 7f2d77574700 1 --2- 172.21.2.112:0/53607929 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2d780b2b50 0x7f2d7809b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:32.648 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.643 7f2d77574700 1 -- 172.21.2.112:0/53607929 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d7818a500 con 0x7f2d781a4990 2021-02-10T11:44:32.648 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.643 7f2d77574700 1 --2- 172.21.2.112:0/53607929 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2d781a4990 0x7f2d780b2530 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 rx=0x7f2d68006f20 tx=0x7f2d6800b8e0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:32.648 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.644 7f2d5ffff700 1 -- 172.21.2.112:0/53607929 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2d6800bcd0 con 0x7f2d781a4990 2021-02-10T11:44:32.648 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.644 7f2d7d9b8700 1 -- 172.21.2.112:0/53607929 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d7818a500 con 0x7f2d781a4990 2021-02-10T11:44:32.648 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.644 7f2d5ffff700 1 -- 172.21.2.112:0/53607929 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d6800bcd0 con 0x7f2d781a4990 2021-02-10T11:44:32.648 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.644 7f2d7d9b8700 1 -- 172.21.2.112:0/53607929 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d7818a500 con 0x7f2d781a4990 2021-02-10T11:44:32.648 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.644 7f2d5ffff700 1 -- 172.21.2.112:0/53607929 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2d68006030 con 0x7f2d781a4990 2021-02-10T11:44:32.648 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.644 7f2d5ffff700 1 -- 172.21.2.112:0/53607929 <== mon.1 v2:172.21.2.120:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2d68009100 con 0x7f2d781a4990 2021-02-10T11:44:32.648 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.644 7f2d5ffff700 1 --2- 172.21.2.112:0/53607929 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2d58018730 0x7f2d5801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:32.649 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.644 7f2d5ffff700 1 -- 172.21.2.112:0/53607929 <== mon.1 v2:172.21.2.120:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f2d680326f0 con 0x7f2d781a4990 2021-02-10T11:44:32.649 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.645 7f2d76d73700 1 --2- 172.21.2.112:0/53607929 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2d58018730 0x7f2d5801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:32.649 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.645 7f2d76d73700 1 --2- 172.21.2.112:0/53607929 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2d58018730 0x7f2d5801ac10 crc :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:32.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.675 7f2d7d9b8700 1 -- 172.21.2.112:0/53607929 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d780ebc90 con 0x7f2d781a4990 2021-02-10T11:44:32.679 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.678 7f2d5ffff700 1 -- 172.21.2.112:0/53607929 <== mon.1 v2:172.21.2.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2d680064e0 con 0x7f2d781a4990 2021-02-10T11:44:32.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:32.811 7f2d7d9b8700 1 -- 172.21.2.112:0/53607929 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f2d780f0b10 con 0x7f2d781a4990 2021-02-10T11:44:33.585 INFO:tasks.ceph.mon.a.gibba012.stderr:2021-02-10 11:44:33.584 7fc3fe6e2700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-02-10T11:44:33.589 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:33.589 7f2d5ffff700 1 -- 172.21.2.112:0/53607929 <== mon.1 v2:172.21.2.120:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v124) v1 ==== 161+0+0 (secure 0 0 0) 0x7f2d68009100 con 0x7f2d781a4990 2021-02-10T11:44:33.632 INFO:teuthology.orchestra.run.gibba012.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2021-02-10T11:44:33.632 INFO:tasks.ceph.mds.b.gibba020.stdout:starting mds.b at 2021-02-10T11:44:33.634 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:33.633 7f2d7d9b8700 1 -- 172.21.2.112:0/53607929 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2d58018730 msgr2=0x7f2d5801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:33.634 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:33.633 7f2d7d9b8700 1 --2- 172.21.2.112:0/53607929 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2d58018730 0x7f2d5801ac10 crc :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:33.634 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:33.633 7f2d7d9b8700 1 -- 172.21.2.112:0/53607929 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2d781a4990 msgr2=0x7f2d780b2530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:33.634 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:33.633 7f2d7d9b8700 1 --2- 172.21.2.112:0/53607929 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2d781a4990 0x7f2d780b2530 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 rx=0x7f2d68006f20 tx=0x7f2d6800b8e0).stop 2021-02-10T11:44:33.634 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:33.633 7f2d7d9b8700 1 -- 172.21.2.112:0/53607929 shutdown_connections 2021-02-10T11:44:33.634 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:33.633 7f2d7d9b8700 1 --2- 172.21.2.112:0/53607929 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2d58018730 0x7f2d5801ac10 unknown :-1 s=CLOSED pgs=1195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:33.634 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:33.633 7f2d7d9b8700 1 --2- 172.21.2.112:0/53607929 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2d7809bfb0 0x7f2d78189aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:33.635 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:33.633 7f2d7d9b8700 1 --2- 172.21.2.112:0/53607929 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2d780b2b50 0x7f2d7809b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:33.635 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:33.633 7f2d7d9b8700 1 --2- 172.21.2.112:0/53607929 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2d781a4990 0x7f2d780b2530 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:33.635 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:33.633 7f2d7d9b8700 1 -- 172.21.2.112:0/53607929 >> 172.21.2.112:0/53607929 conn(0x7f2d780a6870 msgr2=0x7f2d7808a420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:33.635 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:33.633 7f2d7d9b8700 1 -- 172.21.2.112:0/53607929 shutdown_connections 2021-02-10T11:44:33.635 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:33.633 7f2d7d9b8700 1 -- 172.21.2.112:0/53607929 wait complete. 2021-02-10T11:44:33.642 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:44:34.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.053 7fd8f8852700 1 Processor -- start 2021-02-10T11:44:34.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.053 7fd8f8852700 1 -- start start 2021-02-10T11:44:34.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.053 7fd8f8852700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fd8f40d0340 0x7fd8f40ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:34.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.054 7fd8f8852700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8f40f0b10 con 0x7fd8f40d1c70 2021-02-10T11:44:34.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.054 7fd8f8852700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8f41b0be0 con 0x7fd8f40d1870 2021-02-10T11:44:34.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.054 7fd8f8852700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8f40e5ed0 con 0x7fd8f40d0340 2021-02-10T11:44:34.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.054 7fd8f1e1b700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fd8f40d0340 0x7fd8f40ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:34.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.054 7fd8f1e1b700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fd8f40d0340 0x7fd8f40ef640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:57942/0 (socket says 172.21.2.112:57942) 2021-02-10T11:44:34.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.054 7fd8f1e1b700 1 -- 172.21.2.112:0/2858410285 learned_addr learned my addr 172.21.2.112:0/2858410285 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:44:34.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.054 7fd8f1e1b700 1 -- 172.21.2.112:0/2858410285 >> v1:172.21.2.120:6789/0 conn(0x7fd8f40d1870 legacy=0x7fd8f41b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:34.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.054 7fd8f1e1b700 1 -- 172.21.2.112:0/2858410285 >> v1:172.21.2.112:6789/0 conn(0x7fd8f40d1c70 legacy=0x7fd8f40efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:34.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.054 7fd8f1e1b700 1 -- 172.21.2.112:0/2858410285 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8f40f0450 con 0x7fd8f40d0340 2021-02-10T11:44:34.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.054 7fd8f1e1b700 1 --2- 172.21.2.112:0/2858410285 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd8f40d0340 0x7fd8f40ef640 secure :-1 s=READY pgs=2617 cs=0 l=1 rev1=1 rx=0x7fd8e4006700 tx=0x7fd8e4009b00).ready entity=mon.0 client_cookie=4dda8c1d56e1313 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:34.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.054 7fd8f161a700 1 -- 172.21.2.112:0/2858410285 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd8e400c0c0 con 0x7fd8f40d0340 2021-02-10T11:44:34.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.055 7fd8f161a700 1 -- 172.21.2.112:0/2858410285 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8e400c340 con 0x7fd8f40d0340 2021-02-10T11:44:34.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.055 7fd8f161a700 1 -- 172.21.2.112:0/2858410285 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd8e400c7a0 con 0x7fd8f40d0340 2021-02-10T11:44:34.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.055 7fd8f8852700 1 -- 172.21.2.112:0/2858410285 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd8f40d0340 msgr2=0x7fd8f40ef640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:34.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.055 7fd8f8852700 1 --2- 172.21.2.112:0/2858410285 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd8f40d0340 0x7fd8f40ef640 secure :-1 s=READY pgs=2617 cs=0 l=1 rev1=1 rx=0x7fd8e4006700 tx=0x7fd8e4009b00).stop 2021-02-10T11:44:34.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.055 7fd8f8852700 1 -- 172.21.2.112:0/2858410285 shutdown_connections 2021-02-10T11:44:34.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.055 7fd8f8852700 1 --2- 172.21.2.112:0/2858410285 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd8f40d0340 0x7fd8f40ef640 unknown :-1 s=CLOSED pgs=2617 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:34.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.055 7fd8f8852700 1 -- 172.21.2.112:0/2858410285 >> 172.21.2.112:0/2858410285 conn(0x7fd8f40ce450 msgr2=0x7fd8f40d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:34.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.055 7fd8f8852700 1 -- 172.21.2.112:0/2858410285 shutdown_connections 2021-02-10T11:44:34.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.055 7fd8f8852700 1 -- 172.21.2.112:0/2858410285 wait complete. 2021-02-10T11:44:34.074 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.070 7fd8f8852700 1 Processor -- start 2021-02-10T11:44:34.074 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.070 7fd8f8852700 1 -- start start 2021-02-10T11:44:34.074 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.070 7fd8f8852700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd8f40c0060 0x7fd8f40c2bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:34.074 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.070 7fd8f8852700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd8f41c69b0 0x7fd8f40c3210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:34.074 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.070 7fd8f8852700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd8f40c3830 0x7fd8f40b3970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:34.074 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.070 7fd8f8852700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8f40b33f0 con 0x7fd8f41c69b0 2021-02-10T11:44:34.074 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.070 7fd8f8852700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8f40c3c30 con 0x7fd8f40c3830 2021-02-10T11:44:34.074 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.070 7fd8f8852700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8f40bfbc0 con 0x7fd8f40c0060 2021-02-10T11:44:34.074 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.070 7fd8f1e1b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd8f41c69b0 0x7fd8f40c3210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:34.074 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.070 7fd8f1e1b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd8f41c69b0 0x7fd8f40c3210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:57946/0 (socket says 172.21.2.112:57946) 2021-02-10T11:44:34.075 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.070 7fd8f1e1b700 1 -- 172.21.2.112:0/1732951725 learned_addr learned my addr 172.21.2.112:0/1732951725 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:44:34.075 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.071 7fd8f1e1b700 1 -- 172.21.2.112:0/1732951725 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd8f40c0060 msgr2=0x7fd8f40c2bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:34.075 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.071 7fd8f1e1b700 1 --2- 172.21.2.112:0/1732951725 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd8f40c0060 0x7fd8f40c2bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:34.075 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.071 7fd8f1e1b700 1 -- 172.21.2.112:0/1732951725 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd8f40c3830 msgr2=0x7fd8f40b3970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:34.075 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.071 7fd8f1e1b700 1 --2- 172.21.2.112:0/1732951725 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd8f40c3830 0x7fd8f40b3970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:34.075 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.071 7fd8f1e1b700 1 -- 172.21.2.112:0/1732951725 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8f40b3f90 con 0x7fd8f41c69b0 2021-02-10T11:44:34.075 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.071 7fd8f1e1b700 1 --2- 172.21.2.112:0/1732951725 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd8f41c69b0 0x7fd8f40c3210 secure :-1 s=READY pgs=2618 cs=0 l=1 rev1=1 rx=0x7fd8e4006f20 tx=0x7fd8e400b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:34.075 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.071 7fd8eaffd700 1 -- 172.21.2.112:0/1732951725 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd8e4000c90 con 0x7fd8f41c69b0 2021-02-10T11:44:34.075 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.071 7fd8eaffd700 1 -- 172.21.2.112:0/1732951725 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8e4009100 con 0x7fd8f41c69b0 2021-02-10T11:44:34.075 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.071 7fd8eaffd700 1 -- 172.21.2.112:0/1732951725 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd8e4002220 con 0x7fd8f41c69b0 2021-02-10T11:44:34.075 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.071 7fd8f8852700 1 -- 172.21.2.112:0/1732951725 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8f40b3f90 con 0x7fd8f41c69b0 2021-02-10T11:44:34.075 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.071 7fd8f8852700 1 -- 172.21.2.112:0/1732951725 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8f40b4520 con 0x7fd8f41c69b0 2021-02-10T11:44:34.076 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.072 7fd8eaffd700 1 -- 172.21.2.112:0/1732951725 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd8e400a000 con 0x7fd8f41c69b0 2021-02-10T11:44:34.076 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.072 7fd8eaffd700 1 --2- 172.21.2.112:0/1732951725 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd8d80186c0 0x7fd8d801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:34.076 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.072 7fd8eaffd700 1 -- 172.21.2.112:0/1732951725 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 5659+0+0 (secure 0 0 0) 0x7fd8e40325e0 con 0x7fd8f41c69b0 2021-02-10T11:44:34.076 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.072 7fd8f261c700 1 --2- 172.21.2.112:0/1732951725 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd8d80186c0 0x7fd8d801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:34.076 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.073 7fd8f261c700 1 --2- 172.21.2.112:0/1732951725 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd8d80186c0 0x7fd8d801abc0 crc :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:34.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.103 7fd8f8852700 1 -- 172.21.2.112:0/1732951725 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8f40f06f0 con 0x7fd8f41c69b0 2021-02-10T11:44:34.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.106 7fd8eaffd700 1 -- 172.21.2.112:0/1732951725 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd8e4024210 con 0x7fd8f41c69b0 2021-02-10T11:44:34.237 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.236 7fd8f8852700 1 -- 172.21.2.112:0/1732951725 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd8f40f06f0 con 0x7fd8f41c69b0 2021-02-10T11:44:34.237 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.236 7fd8eaffd700 1 -- 172.21.2.112:0/1732951725 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 124 v124) v1 ==== 95+0+1763 (secure 0 0 0) 0x7fd8e4024210 con 0x7fd8f41c69b0 2021-02-10T11:44:34.238 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:44:34.238 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":124,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11670,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/1286703435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":1286703435},{"type":"v1","addr":"172.21.2.112:6833","nonce":1286703435}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":119}],"filesystems":[{"mdsmap":{"epoch":124,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:43:56.409867","modified":"2021-02-10 11:44:33.585045","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":144,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":15}]} 2021-02-10T11:44:34.238 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 124 2021-02-10T11:44:34.239 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.238 7fd8f8852700 1 -- 172.21.2.112:0/1732951725 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd8d80186c0 msgr2=0x7fd8d801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:34.239 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.238 7fd8f8852700 1 --2- 172.21.2.112:0/1732951725 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd8d80186c0 0x7fd8d801abc0 crc :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:34.239 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.238 7fd8f8852700 1 -- 172.21.2.112:0/1732951725 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd8f41c69b0 msgr2=0x7fd8f40c3210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:34.239 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.238 7fd8f8852700 1 --2- 172.21.2.112:0/1732951725 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd8f41c69b0 0x7fd8f40c3210 secure :-1 s=READY pgs=2618 cs=0 l=1 rev1=1 rx=0x7fd8e4006f20 tx=0x7fd8e400b990).stop 2021-02-10T11:44:34.239 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.238 7fd8f8852700 1 -- 172.21.2.112:0/1732951725 shutdown_connections 2021-02-10T11:44:34.239 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.238 7fd8f8852700 1 --2- 172.21.2.112:0/1732951725 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd8d80186c0 0x7fd8d801abc0 unknown :-1 s=CLOSED pgs=1196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:34.239 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.238 7fd8f8852700 1 --2- 172.21.2.112:0/1732951725 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd8f40c3830 0x7fd8f40b3970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:34.239 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.238 7fd8f8852700 1 --2- 172.21.2.112:0/1732951725 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd8f41c69b0 0x7fd8f40c3210 unknown :-1 s=CLOSED pgs=2618 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:34.239 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.238 7fd8f8852700 1 --2- 172.21.2.112:0/1732951725 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd8f40c0060 0x7fd8f40c2bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:34.240 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.238 7fd8f8852700 1 -- 172.21.2.112:0/1732951725 >> 172.21.2.112:0/1732951725 conn(0x7fd8f410a820 msgr2=0x7fd8f4108150 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:34.240 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.238 7fd8f8852700 1 -- 172.21.2.112:0/1732951725 shutdown_connections 2021-02-10T11:44:34.240 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.238 7fd8f8852700 1 -- 172.21.2.112:0/1732951725 wait complete. 2021-02-10T11:44:34.248 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2021-02-10T11:44:34.681 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.664 7f1cb1aa1700 1 Processor -- start 2021-02-10T11:44:34.681 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.664 7f1cb1aa1700 1 -- start start 2021-02-10T11:44:34.681 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.664 7f1cb1aa1700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1cac0d1c70 0x7f1cac0f20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:34.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.664 7f1cb1aa1700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cac0f0b10 con 0x7f1cac0d1870 2021-02-10T11:44:34.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.664 7f1cb1aa1700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cac1b0be0 con 0x7f1cac0d0340 2021-02-10T11:44:34.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.664 7f1cb1aa1700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cac0e5ed0 con 0x7f1cac0d1c70 2021-02-10T11:44:34.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.664 7f1cab7fe700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f1cac0d1870 0x7f1cac1b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:57416/0 (socket says 172.21.2.112:57416) 2021-02-10T11:44:34.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.665 7f1cab7fe700 1 -- 172.21.2.112:0/1258801316 learned_addr learned my addr 172.21.2.112:0/1258801316 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:44:34.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.665 7f1cabfff700 1 --2- 172.21.2.112:0/1258801316 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1cac0d1c70 0x7f1cac0f20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:34.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.665 7f1caa7fc700 1 -- 172.21.2.112:0/1258801316 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 220093791 0 0) 0x7f1c9c0010b0 con 0x7f1cac0d1870 2021-02-10T11:44:34.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.665 7f1caa7fc700 1 -- 172.21.2.112:0/1258801316 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c98002fe0 con 0x7f1cac0d1870 2021-02-10T11:44:34.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.665 7f1caa7fc700 1 -- 172.21.2.112:0/1258801316 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3751871809 0 0) 0x7f1c94000ff0 con 0x7f1cac0d0340 2021-02-10T11:44:34.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.665 7f1caa7fc700 1 -- 172.21.2.112:0/1258801316 --> v1:172.21.2.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c98004370 con 0x7f1cac0d0340 2021-02-10T11:44:34.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.665 7f1cabfff700 1 -- 172.21.2.112:0/1258801316 >> v1:172.21.2.120:6789/0 conn(0x7f1cac0d0340 legacy=0x7f1cac0ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:34.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.665 7f1cabfff700 1 -- 172.21.2.112:0/1258801316 >> v1:172.21.2.112:6789/0 conn(0x7f1cac0d1870 legacy=0x7f1cac1b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:34.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.665 7f1cabfff700 1 -- 172.21.2.112:0/1258801316 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cac0efe30 con 0x7f1cac0d1c70 2021-02-10T11:44:34.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.665 7f1cabfff700 1 --2- 172.21.2.112:0/1258801316 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1cac0d1c70 0x7f1cac0f20a0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0x7f1ca4006e50 tx=0x7f1ca4009e70).ready entity=mon.2 client_cookie=84ca20e1d143396f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:34.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.666 7f1caa7fc700 1 -- 172.21.2.112:0/1258801316 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1ca4006a20 con 0x7f1cac0d1c70 2021-02-10T11:44:34.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.666 7f1caa7fc700 1 -- 172.21.2.112:0/1258801316 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ca4006790 con 0x7f1cac0d1c70 2021-02-10T11:44:34.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.666 7f1caa7fc700 1 -- 172.21.2.112:0/1258801316 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1ca400b700 con 0x7f1cac0d1c70 2021-02-10T11:44:34.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.666 7f1cb1aa1700 1 -- 172.21.2.112:0/1258801316 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1cac0d1c70 msgr2=0x7f1cac0f20a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:34.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.666 7f1cb1aa1700 1 --2- 172.21.2.112:0/1258801316 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1cac0d1c70 0x7f1cac0f20a0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0x7f1ca4006e50 tx=0x7f1ca4009e70).stop 2021-02-10T11:44:34.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.666 7f1cb1aa1700 1 -- 172.21.2.112:0/1258801316 shutdown_connections 2021-02-10T11:44:34.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.666 7f1cb1aa1700 1 --2- 172.21.2.112:0/1258801316 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1cac0d1c70 0x7f1cac0f20a0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:34.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.666 7f1cb1aa1700 1 -- 172.21.2.112:0/1258801316 >> 172.21.2.112:0/1258801316 conn(0x7f1cac0ce450 msgr2=0x7f1cac0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:34.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.666 7f1cb1aa1700 1 -- 172.21.2.112:0/1258801316 shutdown_connections 2021-02-10T11:44:34.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.666 7f1cb1aa1700 1 -- 172.21.2.112:0/1258801316 wait complete. 2021-02-10T11:44:34.685 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.681 7f1cb1aa1700 1 Processor -- start 2021-02-10T11:44:34.685 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.681 7f1cb1aa1700 1 -- start start 2021-02-10T11:44:34.685 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.681 7f1cb1aa1700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1cac1a06c0 0x7f1cac0ebe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:34.685 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.681 7f1cb1aa1700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1cac0ec490 0x7f1cac0c30f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:34.686 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.681 7f1cb1aa1700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1cac0c3710 0x7f1cac0bc070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:34.686 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.681 7f1cb1aa1700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cac0c3b10 con 0x7f1cac1a06c0 2021-02-10T11:44:34.686 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.681 7f1cb1aa1700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cac0eca90 con 0x7f1cac0c3710 2021-02-10T11:44:34.686 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.681 7f1cb1aa1700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cac0bc690 con 0x7f1cac0ec490 2021-02-10T11:44:34.686 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.681 7f1cab7fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1cac1a06c0 0x7f1cac0ebe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:34.686 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.681 7f1cab7fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1cac1a06c0 0x7f1cac0ebe70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:57962/0 (socket says 172.21.2.112:57962) 2021-02-10T11:44:34.686 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.681 7f1cab7fe700 1 -- 172.21.2.112:0/2529059770 learned_addr learned my addr 172.21.2.112:0/2529059770 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:44:34.686 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.681 7f1cabfff700 1 --2- 172.21.2.112:0/2529059770 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1cac0c3710 0x7f1cac0bc070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:34.686 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.681 7f1caaffd700 1 --2- 172.21.2.112:0/2529059770 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1cac0ec490 0x7f1cac0c30f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:34.686 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.682 7f1cab7fe700 1 -- 172.21.2.112:0/2529059770 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1cac0ec490 msgr2=0x7f1cac0c30f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:34.686 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.682 7f1cab7fe700 1 --2- 172.21.2.112:0/2529059770 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1cac0ec490 0x7f1cac0c30f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:34.686 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.682 7f1cab7fe700 1 -- 172.21.2.112:0/2529059770 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1cac0c3710 msgr2=0x7f1cac0bc070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:34.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.682 7f1cab7fe700 1 --2- 172.21.2.112:0/2529059770 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1cac0c3710 0x7f1cac0bc070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:34.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.682 7f1cab7fe700 1 -- 172.21.2.112:0/2529059770 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cac0bc870 con 0x7f1cac1a06c0 2021-02-10T11:44:34.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.682 7f1cab7fe700 1 --2- 172.21.2.112:0/2529059770 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1cac1a06c0 0x7f1cac0ebe70 secure :-1 s=READY pgs=2620 cs=0 l=1 rev1=1 rx=0x7f1c9c001c60 tx=0x7f1c9c00a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:34.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.682 7f1ca3fff700 1 -- 172.21.2.112:0/2529059770 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1c9c000e00 con 0x7f1cac1a06c0 2021-02-10T11:44:34.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.682 7f1ca3fff700 1 -- 172.21.2.112:0/2529059770 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c9c00acc0 con 0x7f1cac1a06c0 2021-02-10T11:44:34.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.682 7f1ca3fff700 1 -- 172.21.2.112:0/2529059770 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1c9c0045d0 con 0x7f1cac1a06c0 2021-02-10T11:44:34.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.682 7f1cb1aa1700 1 -- 172.21.2.112:0/2529059770 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cac0bc870 con 0x7f1cac1a06c0 2021-02-10T11:44:34.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.682 7f1cb1aa1700 1 -- 172.21.2.112:0/2529059770 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cac0bce00 con 0x7f1cac1a06c0 2021-02-10T11:44:34.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.683 7f1ca3fff700 1 -- 172.21.2.112:0/2529059770 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1c9c004140 con 0x7f1cac1a06c0 2021-02-10T11:44:34.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.683 7f1ca3fff700 1 --2- 172.21.2.112:0/2529059770 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1c900186c0 0x7f1c9001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:34.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.683 7f1ca3fff700 1 -- 172.21.2.112:0/2529059770 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 5659+0+0 (secure 0 0 0) 0x7f1c9c0325c0 con 0x7f1cac1a06c0 2021-02-10T11:44:34.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.683 7f1caaffd700 1 --2- 172.21.2.112:0/2529059770 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1c900186c0 0x7f1c9001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:34.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.683 7f1caaffd700 1 --2- 172.21.2.112:0/2529059770 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1c900186c0 0x7f1c9001abc0 crc :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:34.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.714 7f1cb1aa1700 1 -- 172.21.2.112:0/2529059770 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cac0ebc40 con 0x7f1cac1a06c0 2021-02-10T11:44:34.718 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.717 7f1ca3fff700 1 -- 172.21.2.112:0/2529059770 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1c9c000940 con 0x7f1cac1a06c0 2021-02-10T11:44:34.847 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:34.846 7f1cb1aa1700 1 -- 172.21.2.112:0/2529059770 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f1cac0c3b10 con 0x7f1cac1a06c0 2021-02-10T11:44:35.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:35.602 7f1ca3fff700 1 -- 172.21.2.112:0/2529059770 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v125) v1 ==== 104+0+0 (secure 0 0 0) 0x7f1c9c004140 con 0x7f1cac1a06c0 2021-02-10T11:44:35.619 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:35.618 7f1cb1aa1700 1 -- 172.21.2.112:0/2529059770 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1c900186c0 msgr2=0x7f1c9001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:35.619 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:35.618 7f1cb1aa1700 1 --2- 172.21.2.112:0/2529059770 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1c900186c0 0x7f1c9001abc0 crc :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:35.620 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:35.618 7f1cb1aa1700 1 -- 172.21.2.112:0/2529059770 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1cac1a06c0 msgr2=0x7f1cac0ebe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:35.620 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:35.618 7f1cb1aa1700 1 --2- 172.21.2.112:0/2529059770 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1cac1a06c0 0x7f1cac0ebe70 secure :-1 s=READY pgs=2620 cs=0 l=1 rev1=1 rx=0x7f1c9c001c60 tx=0x7f1c9c00a8d0).stop 2021-02-10T11:44:35.620 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:35.619 7f1cb1aa1700 1 -- 172.21.2.112:0/2529059770 shutdown_connections 2021-02-10T11:44:35.620 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:35.619 7f1cb1aa1700 1 --2- 172.21.2.112:0/2529059770 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1c900186c0 0x7f1c9001abc0 unknown :-1 s=CLOSED pgs=1197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:35.620 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:35.619 7f1cb1aa1700 1 --2- 172.21.2.112:0/2529059770 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1cac0c3710 0x7f1cac0bc070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:35.621 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:35.619 7f1cb1aa1700 1 --2- 172.21.2.112:0/2529059770 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1cac0ec490 0x7f1cac0c30f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:35.621 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:35.619 7f1cb1aa1700 1 --2- 172.21.2.112:0/2529059770 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1cac1a06c0 0x7f1cac0ebe70 unknown :-1 s=CLOSED pgs=2620 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:35.621 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:35.619 7f1cb1aa1700 1 -- 172.21.2.112:0/2529059770 >> 172.21.2.112:0/2529059770 conn(0x7f1cac0a6bb0 msgr2=0x7f1cac095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:35.621 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:35.619 7f1cb1aa1700 1 -- 172.21.2.112:0/2529059770 shutdown_connections 2021-02-10T11:44:35.621 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:35.619 7f1cb1aa1700 1 -- 172.21.2.112:0/2529059770 wait complete. 2021-02-10T11:44:35.628 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2021-02-10T11:44:36.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:36.042 7f4e43336700 1 Processor -- start 2021-02-10T11:44:36.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:36.042 7f4e43336700 1 -- start start 2021-02-10T11:44:36.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:36.043 7f4e43336700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4e3c0d1c80 0x7f4e3c0cef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:36.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:36.043 7f4e43336700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f4e3c0cf590 0x7f4e3c0cf990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:36.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:36.043 7f4e43336700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e3c0fe1d0 con 0x7f4e3c0d1880 2021-02-10T11:44:36.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:36.043 7f4e43336700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e3c1b0cc0 con 0x7f4e3c0cf590 2021-02-10T11:44:36.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:36.043 7f4e43336700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e3c109a70 con 0x7f4e3c0d1c80 2021-02-10T11:44:36.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:36.043 7f4e41953700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f4e3c0cf590 0x7f4e3c0cf990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:36.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:36.043 7f4e41953700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f4e3c0cf590 0x7f4e3c0cf990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:57968/0 (socket says 172.21.2.112:57968) 2021-02-10T11:44:36.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:36.043 7f4e41953700 1 -- 172.21.2.112:0/3465323528 learned_addr learned my addr 172.21.2.112:0/3465323528 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:44:36.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:36.043 7f4e41953700 1 -- 172.21.2.112:0/3465323528 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4e3c0d1c80 msgr2=0x7f4e3c0cef70 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:44:36.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:36.043 7f4e41953700 1 --2- 172.21.2.112:0/3465323528 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4e3c0d1c80 0x7f4e3c0cef70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:36.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:36.043 7f4e41953700 1 -- 172.21.2.112:0/3465323528 >> v1:172.21.2.120:6789/0 conn(0x7f4e3c0d1880 legacy=0x7f4e3c1b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:36.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:36.044 7f4e41953700 1 -- 172.21.2.112:0/3465323528 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e3c088070 con 0x7f4e3c0cf590 2021-02-10T11:44:36.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:36.044 7f4e41953700 1 --2- 172.21.2.112:0/3465323528 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4e3c0cf590 0x7f4e3c0cf990 secure :-1 s=READY pgs=2621 cs=0 l=1 rev1=1 rx=0x7f4e34006f00 tx=0x7f4e34009e90).ready entity=mon.0 client_cookie=e2c46cdd90808b45 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:36.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:36.044 7f4e33fff700 1 -- 172.21.2.112:0/3465323528 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4e3400b300 con 0x7f4e3c0cf590 2021-02-10T11:44:36.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:36.044 7f4e33fff700 1 -- 172.21.2.112:0/3465323528 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e3400b640 con 0x7f4e3c0cf590 2021-02-10T11:44:36.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:36.044 7f4e33fff700 1 -- 172.21.2.112:0/3465323528 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4e3400ba90 con 0x7f4e3c0cf590 2021-02-10T11:44:36.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:36.044 7f4e43336700 1 -- 172.21.2.112:0/3465323528 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4e3c0cf590 msgr2=0x7f4e3c0cf990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:36.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:36.044 7f4e43336700 1 --2- 172.21.2.112:0/3465323528 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4e3c0cf590 0x7f4e3c0cf990 secure :-1 s=READY pgs=2621 cs=0 l=1 rev1=1 rx=0x7f4e34006f00 tx=0x7f4e34009e90).stop 2021-02-10T11:44:36.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:36.045 7f4e43336700 1 -- 172.21.2.112:0/3465323528 shutdown_connections 2021-02-10T11:44:36.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:36.045 7f4e43336700 1 --2- 172.21.2.112:0/3465323528 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4e3c0cf590 0x7f4e3c0cf990 unknown :-1 s=CLOSED pgs=2621 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:36.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:36.045 7f4e43336700 1 --2- 172.21.2.112:0/3465323528 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4e3c0d1c80 0x7f4e3c0cef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:36.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:36.045 7f4e43336700 1 -- 172.21.2.112:0/3465323528 >> 172.21.2.112:0/3465323528 conn(0x7f4e3c0ce430 msgr2=0x7f4e3c0d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:36.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:36.045 7f4e43336700 1 -- 172.21.2.112:0/3465323528 shutdown_connections 2021-02-10T11:44:36.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:36.045 7f4e43336700 1 -- 172.21.2.112:0/3465323528 wait complete. 2021-02-10T11:44:36.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:36.059 7f4e43336700 1 Processor -- start 2021-02-10T11:44:36.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:36.060 7f4e43336700 1 -- start start 2021-02-10T11:44:36.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:36.060 7f4e43336700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4e3c187e80 0x7f4e3c188280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:36.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:36.060 7f4e43336700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4e3c1888a0 0x7f4e3c189490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:36.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:36.060 7f4e43336700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4e3c189ab0 0x7f4e3c189eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:36.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:36.060 7f4e43336700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e3c188ca0 con 0x7f4e3c189ab0 2021-02-10T11:44:36.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:36.060 7f4e43336700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e3c188f10 con 0x7f4e3c1888a0 2021-02-10T11:44:36.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:36.060 7f4e43336700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e3c099520 con 0x7f4e3c187e80 2021-02-10T11:44:36.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:36.060 7f4e41953700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4e3c189ab0 0x7f4e3c189eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:36.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:36.060 7f4e41953700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4e3c189ab0 0x7f4e3c189eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:57976/0 (socket says 172.21.2.112:57976) 2021-02-10T11:44:36.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:36.060 7f4e41953700 1 -- 172.21.2.112:0/2917882651 learned_addr learned my addr 172.21.2.112:0/2917882651 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:44:36.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:36.061 7f4e41953700 1 -- 172.21.2.112:0/2917882651 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4e3c187e80 msgr2=0x7f4e3c188280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:36.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:36.061 7f4e41953700 1 --2- 172.21.2.112:0/2917882651 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4e3c187e80 0x7f4e3c188280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:36.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:36.061 7f4e41953700 1 -- 172.21.2.112:0/2917882651 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4e3c1888a0 msgr2=0x7f4e3c189490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:36.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:36.061 7f4e41953700 1 --2- 172.21.2.112:0/2917882651 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4e3c1888a0 0x7f4e3c189490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:36.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:36.061 7f4e41953700 1 -- 172.21.2.112:0/2917882651 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e3c0999c0 con 0x7f4e3c189ab0 2021-02-10T11:44:36.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:36.061 7f4e41953700 1 --2- 172.21.2.112:0/2917882651 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4e3c189ab0 0x7f4e3c189eb0 secure :-1 s=READY pgs=2622 cs=0 l=1 rev1=1 rx=0x7f4e3400b910 tx=0x7f4e3400bcb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:36.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:36.061 7f4e31ffb700 1 -- 172.21.2.112:0/2917882651 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4e34000cc0 con 0x7f4e3c189ab0 2021-02-10T11:44:36.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:36.061 7f4e31ffb700 1 -- 172.21.2.112:0/2917882651 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e3400bdf0 con 0x7f4e3c189ab0 2021-02-10T11:44:36.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:36.061 7f4e31ffb700 1 -- 172.21.2.112:0/2917882651 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4e34005a10 con 0x7f4e3c189ab0 2021-02-10T11:44:36.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:36.061 7f4e43336700 1 -- 172.21.2.112:0/2917882651 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e3c0999c0 con 0x7f4e3c189ab0 2021-02-10T11:44:36.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:36.061 7f4e43336700 1 -- 172.21.2.112:0/2917882651 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e3c18a7c0 con 0x7f4e3c189ab0 2021-02-10T11:44:36.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:36.062 7f4e31ffb700 1 -- 172.21.2.112:0/2917882651 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4e34005820 con 0x7f4e3c189ab0 2021-02-10T11:44:36.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:36.062 7f4e31ffb700 1 --2- 172.21.2.112:0/2917882651 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4e1c0186c0 0x7f4e1c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:36.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:36.062 7f4e31ffb700 1 -- 172.21.2.112:0/2917882651 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 5659+0+0 (secure 0 0 0) 0x7f4e34032970 con 0x7f4e3c189ab0 2021-02-10T11:44:36.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:36.062 7f4e41152700 1 --2- 172.21.2.112:0/2917882651 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4e1c0186c0 0x7f4e1c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:36.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:36.063 7f4e41152700 1 --2- 172.21.2.112:0/2917882651 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4e1c0186c0 0x7f4e1c01abc0 crc :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:36.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:36.093 7f4e43336700 1 -- 172.21.2.112:0/2917882651 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e3c195bd0 con 0x7f4e3c189ab0 2021-02-10T11:44:36.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:36.095 7f4e31ffb700 1 -- 172.21.2.112:0/2917882651 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4e34017f20 con 0x7f4e3c189ab0 2021-02-10T11:44:36.224 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:36.223 7f4e43336700 1 -- 172.21.2.112:0/2917882651 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f4e3c195bd0 con 0x7f4e3c189ab0 2021-02-10T11:44:36.608 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:36.606 7f4e31ffb700 1 -- 172.21.2.112:0/2917882651 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v145) v1 ==== 185+0+0 (secure 0 0 0) 0x7f4e34017f20 con 0x7f4e3c189ab0 2021-02-10T11:44:36.640 INFO:teuthology.orchestra.run.gibba012.stderr:pool 'cephfs_metadata' removed 2021-02-10T11:44:36.642 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:36.641 7f4e43336700 1 -- 172.21.2.112:0/2917882651 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4e1c0186c0 msgr2=0x7f4e1c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:36.642 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:36.641 7f4e43336700 1 --2- 172.21.2.112:0/2917882651 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4e1c0186c0 0x7f4e1c01abc0 crc :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:36.642 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:36.641 7f4e43336700 1 -- 172.21.2.112:0/2917882651 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4e3c189ab0 msgr2=0x7f4e3c189eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:36.642 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:36.641 7f4e43336700 1 --2- 172.21.2.112:0/2917882651 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4e3c189ab0 0x7f4e3c189eb0 secure :-1 s=READY pgs=2622 cs=0 l=1 rev1=1 rx=0x7f4e3400b910 tx=0x7f4e3400bcb0).stop 2021-02-10T11:44:36.642 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:36.641 7f4e43336700 1 -- 172.21.2.112:0/2917882651 shutdown_connections 2021-02-10T11:44:36.643 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:36.641 7f4e43336700 1 --2- 172.21.2.112:0/2917882651 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4e1c0186c0 0x7f4e1c01abc0 unknown :-1 s=CLOSED pgs=1198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:36.643 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:36.641 7f4e43336700 1 --2- 172.21.2.112:0/2917882651 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4e3c1888a0 0x7f4e3c189490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:36.643 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:36.641 7f4e43336700 1 --2- 172.21.2.112:0/2917882651 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4e3c189ab0 0x7f4e3c189eb0 unknown :-1 s=CLOSED pgs=2622 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:36.643 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:36.641 7f4e43336700 1 --2- 172.21.2.112:0/2917882651 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4e3c187e80 0x7f4e3c188280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:36.643 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:36.641 7f4e43336700 1 -- 172.21.2.112:0/2917882651 >> 172.21.2.112:0/2917882651 conn(0x7f4e3c19f940 msgr2=0x7f4e3c18c290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:36.643 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:36.641 7f4e43336700 1 -- 172.21.2.112:0/2917882651 shutdown_connections 2021-02-10T11:44:36.643 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:36.641 7f4e43336700 1 -- 172.21.2.112:0/2917882651 wait complete. 2021-02-10T11:44:36.651 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2021-02-10T11:44:37.083 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:37.066 7f6a3c086700 1 Processor -- start 2021-02-10T11:44:37.084 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:37.066 7f6a3c086700 1 -- start start 2021-02-10T11:44:37.084 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:37.067 7f6a3c086700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f6a340d1880 0x7f6a341b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:37.084 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:37.067 7f6a3c086700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f6a340cff80 0x7f6a34109140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:37.084 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:37.067 7f6a3c086700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a340fe1d0 con 0x7f6a341b0af0 2021-02-10T11:44:37.084 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:37.067 7f6a3c086700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a34109ae0 con 0x7f6a340cff80 2021-02-10T11:44:37.084 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:37.067 7f6a3c086700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a34109760 con 0x7f6a340d1880 2021-02-10T11:44:37.084 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:37.067 7f6a3a6a3700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f6a340cff80 0x7f6a34109140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:37.084 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:37.067 7f6a3a6a3700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f6a340cff80 0x7f6a34109140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:57980/0 (socket says 172.21.2.112:57980) 2021-02-10T11:44:37.084 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:37.067 7f6a3a6a3700 1 -- 172.21.2.112:0/813838052 learned_addr learned my addr 172.21.2.112:0/813838052 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:44:37.084 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:37.067 7f6a3a6a3700 1 -- 172.21.2.112:0/813838052 >> v2:172.21.2.120:3300/0 conn(0x7f6a340d1880 msgr2=0x7f6a341b04d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:44:37.085 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:37.067 7f6a3a6a3700 1 --2- 172.21.2.112:0/813838052 >> v2:172.21.2.120:3300/0 conn(0x7f6a340d1880 0x7f6a341b04d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:37.085 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:37.067 7f6a3a6a3700 1 -- 172.21.2.112:0/813838052 >> v1:172.21.2.120:6789/0 conn(0x7f6a341b0af0 legacy=0x7f6a34108950 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:44:37.085 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:37.067 7f6a3a6a3700 1 -- 172.21.2.112:0/813838052 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a34088070 con 0x7f6a340cff80 2021-02-10T11:44:37.085 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:37.067 7f6a3a6a3700 1 --2- 172.21.2.112:0/813838052 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6a340cff80 0x7f6a34109140 secure :-1 s=READY pgs=2623 cs=0 l=1 rev1=1 rx=0x7f6a2c006f00 tx=0x7f6a2c009f10).ready entity=mon.0 client_cookie=cc2f38812f93ba38 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:37.085 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:37.067 7f6a38ea0700 1 -- 172.21.2.112:0/813838052 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6a2c00b3b0 con 0x7f6a340cff80 2021-02-10T11:44:37.085 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:37.067 7f6a38ea0700 1 -- 172.21.2.112:0/813838052 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a2c00b730 con 0x7f6a340cff80 2021-02-10T11:44:37.085 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:37.067 7f6a38ea0700 1 -- 172.21.2.112:0/813838052 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6a2c00bc10 con 0x7f6a340cff80 2021-02-10T11:44:37.085 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:37.067 7f6a3c086700 1 -- 172.21.2.112:0/813838052 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6a340cff80 msgr2=0x7f6a34109140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:37.085 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:37.067 7f6a3c086700 1 --2- 172.21.2.112:0/813838052 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6a340cff80 0x7f6a34109140 secure :-1 s=READY pgs=2623 cs=0 l=1 rev1=1 rx=0x7f6a2c006f00 tx=0x7f6a2c009f10).stop 2021-02-10T11:44:37.085 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:37.068 7f6a3c086700 1 -- 172.21.2.112:0/813838052 shutdown_connections 2021-02-10T11:44:37.085 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:37.068 7f6a3c086700 1 --2- 172.21.2.112:0/813838052 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6a340cff80 0x7f6a34109140 unknown :-1 s=CLOSED pgs=2623 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:37.085 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:37.068 7f6a3c086700 1 --2- 172.21.2.112:0/813838052 >> v2:172.21.2.120:3300/0 conn(0x7f6a340d1880 0x7f6a341b04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:37.085 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:37.068 7f6a3c086700 1 -- 172.21.2.112:0/813838052 >> 172.21.2.112:0/813838052 conn(0x7f6a340ce430 msgr2=0x7f6a340d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:37.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:37.068 7f6a3c086700 1 -- 172.21.2.112:0/813838052 shutdown_connections 2021-02-10T11:44:37.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:37.068 7f6a3c086700 1 -- 172.21.2.112:0/813838052 wait complete. 2021-02-10T11:44:37.087 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:37.083 7f6a3c086700 1 Processor -- start 2021-02-10T11:44:37.088 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:37.083 7f6a3c086700 1 -- start start 2021-02-10T11:44:37.088 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:37.083 7f6a3c086700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6a34187e80 0x7f6a34188280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:37.088 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:37.083 7f6a3c086700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6a341888a0 0x7f6a34189490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:37.088 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:37.083 7f6a3c086700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6a34189ab0 0x7f6a34189eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:37.088 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:37.083 7f6a3c086700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a34188ca0 con 0x7f6a341888a0 2021-02-10T11:44:37.088 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:37.083 7f6a3c086700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a34188f10 con 0x7f6a34189ab0 2021-02-10T11:44:37.088 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:37.083 7f6a3c086700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a34099520 con 0x7f6a34187e80 2021-02-10T11:44:37.088 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:37.083 7f6a396a1700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6a341888a0 0x7f6a34189490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:37.088 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:37.083 7f6a396a1700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6a341888a0 0x7f6a34189490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:57984/0 (socket says 172.21.2.112:57984) 2021-02-10T11:44:37.088 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:37.083 7f6a396a1700 1 -- 172.21.2.112:0/3153231625 learned_addr learned my addr 172.21.2.112:0/3153231625 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:44:37.088 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:37.084 7f6a396a1700 1 -- 172.21.2.112:0/3153231625 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6a34187e80 msgr2=0x7f6a34188280 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:44:37.088 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:37.084 7f6a396a1700 1 --2- 172.21.2.112:0/3153231625 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6a34187e80 0x7f6a34188280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:37.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:37.084 7f6a396a1700 1 -- 172.21.2.112:0/3153231625 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6a34189ab0 msgr2=0x7f6a34189eb0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:44:37.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:37.084 7f6a396a1700 1 --2- 172.21.2.112:0/3153231625 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6a34189ab0 0x7f6a34189eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:37.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:37.084 7f6a396a1700 1 -- 172.21.2.112:0/3153231625 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a340999c0 con 0x7f6a341888a0 2021-02-10T11:44:37.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:37.084 7f6a396a1700 1 --2- 172.21.2.112:0/3153231625 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6a341888a0 0x7f6a34189490 secure :-1 s=READY pgs=2624 cs=0 l=1 rev1=1 rx=0x7f6a3000b5b0 tx=0x7f6a3000b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:37.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:37.084 7f6a267fc700 1 -- 172.21.2.112:0/3153231625 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6a3000bd40 con 0x7f6a341888a0 2021-02-10T11:44:37.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:37.084 7f6a267fc700 1 -- 172.21.2.112:0/3153231625 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a30000c60 con 0x7f6a341888a0 2021-02-10T11:44:37.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:37.084 7f6a267fc700 1 -- 172.21.2.112:0/3153231625 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6a300062a0 con 0x7f6a341888a0 2021-02-10T11:44:37.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:37.084 7f6a3c086700 1 -- 172.21.2.112:0/3153231625 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a340999c0 con 0x7f6a341888a0 2021-02-10T11:44:37.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:37.084 7f6a3c086700 1 -- 172.21.2.112:0/3153231625 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a340999c0 con 0x7f6a341888a0 2021-02-10T11:44:37.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:37.085 7f6a267fc700 1 -- 172.21.2.112:0/3153231625 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6a30006020 con 0x7f6a341888a0 2021-02-10T11:44:37.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:37.085 7f6a267fc700 1 --2- 172.21.2.112:0/3153231625 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6a1c0186c0 0x7f6a1c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:37.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:37.085 7f6a267fc700 1 -- 172.21.2.112:0/3153231625 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5221+0+0 (secure 0 0 0) 0x7f6a30032450 con 0x7f6a341888a0 2021-02-10T11:44:37.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:37.086 7f6a39ea2700 1 --2- 172.21.2.112:0/3153231625 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6a1c0186c0 0x7f6a1c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:37.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:37.086 7f6a39ea2700 1 --2- 172.21.2.112:0/3153231625 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6a1c0186c0 0x7f6a1c01abc0 crc :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:37.118 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:37.117 7f6a3c086700 1 -- 172.21.2.112:0/3153231625 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a34195bd0 con 0x7f6a341888a0 2021-02-10T11:44:37.120 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:37.119 7f6a267fc700 1 -- 172.21.2.112:0/3153231625 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6a30030d60 con 0x7f6a341888a0 2021-02-10T11:44:37.248 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:37.247 7f6a3c086700 1 -- 172.21.2.112:0/3153231625 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f6a34195bd0 con 0x7f6a341888a0 2021-02-10T11:44:37.612 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:37.611 7f6a267fc700 1 -- 172.21.2.112:0/3153231625 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v146) v1 ==== 173+0+0 (secure 0 0 0) 0x7f6a30006020 con 0x7f6a341888a0 2021-02-10T11:44:37.614 INFO:teuthology.orchestra.run.gibba012.stderr:pool 'cephfs_data' removed 2021-02-10T11:44:37.616 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:37.615 7f6a3c086700 1 -- 172.21.2.112:0/3153231625 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6a1c0186c0 msgr2=0x7f6a1c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:37.616 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:37.615 7f6a3c086700 1 --2- 172.21.2.112:0/3153231625 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6a1c0186c0 0x7f6a1c01abc0 crc :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:37.616 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:37.615 7f6a3c086700 1 -- 172.21.2.112:0/3153231625 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6a341888a0 msgr2=0x7f6a34189490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:37.616 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:37.615 7f6a3c086700 1 --2- 172.21.2.112:0/3153231625 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6a341888a0 0x7f6a34189490 secure :-1 s=READY pgs=2624 cs=0 l=1 rev1=1 rx=0x7f6a3000b5b0 tx=0x7f6a3000b950).stop 2021-02-10T11:44:37.617 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:37.615 7f6a3c086700 1 -- 172.21.2.112:0/3153231625 shutdown_connections 2021-02-10T11:44:37.617 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:37.615 7f6a3c086700 1 --2- 172.21.2.112:0/3153231625 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6a1c0186c0 0x7f6a1c01abc0 unknown :-1 s=CLOSED pgs=1199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:37.617 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:37.615 7f6a3c086700 1 --2- 172.21.2.112:0/3153231625 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6a34189ab0 0x7f6a34189eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:37.617 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:37.615 7f6a3c086700 1 --2- 172.21.2.112:0/3153231625 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6a341888a0 0x7f6a34189490 unknown :-1 s=CLOSED pgs=2624 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:37.617 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:37.615 7f6a3c086700 1 --2- 172.21.2.112:0/3153231625 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6a34187e80 0x7f6a34188280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:37.617 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:37.615 7f6a3c086700 1 -- 172.21.2.112:0/3153231625 >> 172.21.2.112:0/3153231625 conn(0x7f6a3419f940 msgr2=0x7f6a3418c290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:37.617 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:37.615 7f6a3c086700 1 -- 172.21.2.112:0/3153231625 shutdown_connections 2021-02-10T11:44:37.617 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:37.615 7f6a3c086700 1 -- 172.21.2.112:0/3153231625 wait complete. 2021-02-10T11:44:37.625 INFO:tasks.ceph.mds.a:Restarting daemon 2021-02-10T11:44:37.625 INFO:tasks.ceph.mds.a:Stopping old one... 2021-02-10T11:44:37.625 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2021-02-10T11:44:37.625 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:44:37.626 INFO:tasks.ceph.mds.b:Restarting daemon 2021-02-10T11:44:37.626 INFO:tasks.ceph.mds.b:Stopping old one... 2021-02-10T11:44:37.626 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2021-02-10T11:44:37.626 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:44:37.631 INFO:tasks.ceph.mds.a:Stopped 2021-02-10T11:44:37.631 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2021-02-10T11:44:37.632 INFO:tasks.ceph.mds.b:Stopped 2021-02-10T11:44:37.633 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2021-02-10T11:44:37.674 INFO:tasks.ceph.mds.a:Started 2021-02-10T11:44:37.674 DEBUG:teuthology.parallel:result is None 2021-02-10T11:44:37.675 INFO:tasks.ceph.mds.b:Started 2021-02-10T11:44:37.675 DEBUG:teuthology.parallel:result is None 2021-02-10T11:44:37.675 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd blacklist clear 2021-02-10T11:44:38.079 INFO:tasks.ceph.mds.b.gibba020.stdout:starting mds.b at 2021-02-10T11:44:38.082 INFO:tasks.ceph.mds.a.gibba012.stdout:starting mds.a at 2021-02-10T11:44:38.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:38.084 7fce7f027700 1 Processor -- start 2021-02-10T11:44:38.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:38.084 7fce7f027700 1 -- start start 2021-02-10T11:44:38.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:38.084 7fce7f027700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fce780d0330 0x7fce780ef630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:38.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:38.084 7fce7f027700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fce780d1c60 0x7fce780efc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:38.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:38.084 7fce7f027700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce780f0b00 con 0x7fce780d1860 2021-02-10T11:44:38.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:38.084 7fce7f027700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce780e5ec0 con 0x7fce780d1c60 2021-02-10T11:44:38.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:38.084 7fce7f027700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce780d3350 con 0x7fce780d0330 2021-02-10T11:44:38.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:38.084 7fce7ce43700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fce780d1860 0x7fce781b03e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:57468/0 (socket says 172.21.2.112:57468) 2021-02-10T11:44:38.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:38.084 7fce7ce43700 1 -- 172.21.2.112:0/126861567 learned_addr learned my addr 172.21.2.112:0/126861567 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:44:38.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:38.084 7fce77fff700 1 --2- 172.21.2.112:0/126861567 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fce780d0330 0x7fce780ef630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:38.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:38.084 7fce7d644700 1 --2- 172.21.2.112:0/126861567 >> v2:172.21.2.120:3300/0 conn(0x7fce780d1c60 0x7fce780efc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:38.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:38.085 7fce77fff700 1 -- 172.21.2.112:0/126861567 >> v2:172.21.2.120:3300/0 conn(0x7fce780d1c60 msgr2=0x7fce780efc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:38.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:38.085 7fce77fff700 1 --2- 172.21.2.112:0/126861567 >> v2:172.21.2.120:3300/0 conn(0x7fce780d1c60 0x7fce780efc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:38.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:38.085 7fce77fff700 1 -- 172.21.2.112:0/126861567 >> v1:172.21.2.112:6789/0 conn(0x7fce780d1860 legacy=0x7fce781b03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:38.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:38.085 7fce77fff700 1 -- 172.21.2.112:0/126861567 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce781b0bd0 con 0x7fce780d0330 2021-02-10T11:44:38.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:38.085 7fce77fff700 1 --2- 172.21.2.112:0/126861567 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fce780d0330 0x7fce780ef630 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7fce6c006650 tx=0x7fce6c009a80).ready entity=mon.2 client_cookie=97fc7888a0ab5a0c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:38.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:38.085 7fce777fe700 1 -- 172.21.2.112:0/126861567 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fce6c00c0c0 con 0x7fce780d0330 2021-02-10T11:44:38.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:38.085 7fce777fe700 1 -- 172.21.2.112:0/126861567 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce6c00c340 con 0x7fce780d0330 2021-02-10T11:44:38.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:38.085 7fce777fe700 1 -- 172.21.2.112:0/126861567 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fce6c00c7a0 con 0x7fce780d0330 2021-02-10T11:44:38.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:38.085 7fce7f027700 1 -- 172.21.2.112:0/126861567 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fce780d0330 msgr2=0x7fce780ef630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:38.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:38.085 7fce7f027700 1 --2- 172.21.2.112:0/126861567 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fce780d0330 0x7fce780ef630 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7fce6c006650 tx=0x7fce6c009a80).stop 2021-02-10T11:44:38.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:38.086 7fce7f027700 1 -- 172.21.2.112:0/126861567 shutdown_connections 2021-02-10T11:44:38.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:38.086 7fce7f027700 1 --2- 172.21.2.112:0/126861567 >> v2:172.21.2.120:3300/0 conn(0x7fce780d1c60 0x7fce780efc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:38.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:38.086 7fce7f027700 1 --2- 172.21.2.112:0/126861567 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fce780d0330 0x7fce780ef630 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:38.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:38.086 7fce7f027700 1 -- 172.21.2.112:0/126861567 >> 172.21.2.112:0/126861567 conn(0x7fce780ce440 msgr2=0x7fce780d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:38.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:38.086 7fce7f027700 1 -- 172.21.2.112:0/126861567 shutdown_connections 2021-02-10T11:44:38.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:38.086 7fce7f027700 1 -- 172.21.2.112:0/126861567 wait complete. 2021-02-10T11:44:38.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:38.100 7fce7f027700 1 Processor -- start 2021-02-10T11:44:38.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:38.100 7fce7f027700 1 -- start start 2021-02-10T11:44:38.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:38.101 7fce7f027700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fce781a3550 0x7fce781a3950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:38.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:38.101 7fce7f027700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fce780bfb30 0x7fce780bff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:38.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:38.101 7fce7f027700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fce780c0550 0x7fce780c0950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:38.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:38.101 7fce7f027700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce7808e180 con 0x7fce780c0550 2021-02-10T11:44:38.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:38.101 7fce7f027700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce78090c10 con 0x7fce780bfb30 2021-02-10T11:44:38.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:38.101 7fce7f027700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce78090e30 con 0x7fce781a3550 2021-02-10T11:44:38.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:38.101 7fce7d644700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fce780c0550 0x7fce780c0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:38.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:38.101 7fce7d644700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fce780c0550 0x7fce780c0950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:58012/0 (socket says 172.21.2.112:58012) 2021-02-10T11:44:38.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:38.101 7fce7d644700 1 -- 172.21.2.112:0/969796050 learned_addr learned my addr 172.21.2.112:0/969796050 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:44:38.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:38.101 7fce7ce43700 1 --2- 172.21.2.112:0/969796050 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fce781a3550 0x7fce781a3950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:38.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:38.101 7fce77fff700 1 --2- 172.21.2.112:0/969796050 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fce780bfb30 0x7fce780bff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:38.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:38.101 7fce7d644700 1 -- 172.21.2.112:0/969796050 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fce781a3550 msgr2=0x7fce781a3950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:38.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:38.101 7fce7d644700 1 --2- 172.21.2.112:0/969796050 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fce781a3550 0x7fce781a3950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:38.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:38.102 7fce7d644700 1 -- 172.21.2.112:0/969796050 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fce780bfb30 msgr2=0x7fce780bff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:38.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:38.102 7fce7d644700 1 --2- 172.21.2.112:0/969796050 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fce780bfb30 0x7fce780bff30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:38.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:38.102 7fce7d644700 1 -- 172.21.2.112:0/969796050 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce78091050 con 0x7fce780c0550 2021-02-10T11:44:38.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:38.102 7fce7ce43700 1 --2- 172.21.2.112:0/969796050 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fce781a3550 0x7fce781a3950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:44:38.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:38.102 7fce7d644700 1 --2- 172.21.2.112:0/969796050 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fce780c0550 0x7fce780c0950 secure :-1 s=READY pgs=2629 cs=0 l=1 rev1=1 rx=0x7fce700099a0 tx=0x7fce7000bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:38.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:38.102 7fce757fa700 1 -- 172.21.2.112:0/969796050 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fce700066f0 con 0x7fce780c0550 2021-02-10T11:44:38.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:38.102 7fce757fa700 1 -- 172.21.2.112:0/969796050 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce70003030 con 0x7fce780c0550 2021-02-10T11:44:38.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:38.102 7fce757fa700 1 -- 172.21.2.112:0/969796050 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fce70001470 con 0x7fce780c0550 2021-02-10T11:44:38.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:38.102 7fce7f027700 1 -- 172.21.2.112:0/969796050 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce78091050 con 0x7fce780c0550 2021-02-10T11:44:38.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:38.102 7fce7f027700 1 -- 172.21.2.112:0/969796050 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce78091050 con 0x7fce780c0550 2021-02-10T11:44:38.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:38.102 7fce757fa700 1 -- 172.21.2.112:0/969796050 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fce700011f0 con 0x7fce780c0550 2021-02-10T11:44:38.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:38.102 7fce757fa700 1 --2- 172.21.2.112:0/969796050 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fce58018730 0x7fce5801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:38.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:38.102 7fce757fa700 1 -- 172.21.2.112:0/969796050 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 4839+0+0 (secure 0 0 0) 0x7fce700326b0 con 0x7fce780c0550 2021-02-10T11:44:38.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:38.103 7fce7ce43700 1 --2- 172.21.2.112:0/969796050 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fce58018730 0x7fce5801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:38.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:38.103 7fce7ce43700 1 --2- 172.21.2.112:0/969796050 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fce58018730 0x7fce5801ac10 crc :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:38.134 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:38.133 7fce7f027700 1 -- 172.21.2.112:0/969796050 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce780ebbe0 con 0x7fce780c0550 2021-02-10T11:44:38.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:38.136 7fce757fa700 1 -- 172.21.2.112:0/969796050 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fce7000be30 con 0x7fce780c0550 2021-02-10T11:44:38.276 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:38.275 7fce7f027700 1 -- 172.21.2.112:0/969796050 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7fce780f0b00 con 0x7fce780c0550 2021-02-10T11:44:38.614 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:38.613 7fce757fa700 1 -- 172.21.2.112:0/969796050 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v147) v1 ==== 97+0+0 (secure 0 0 0) 0x7fce70002f30 con 0x7fce780c0550 2021-02-10T11:44:38.641 INFO:teuthology.orchestra.run.gibba012.stderr: removed all blacklist entries 2021-02-10T11:44:38.643 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:38.642 7fce7f027700 1 -- 172.21.2.112:0/969796050 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fce58018730 msgr2=0x7fce5801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:38.643 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:38.642 7fce7f027700 1 --2- 172.21.2.112:0/969796050 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fce58018730 0x7fce5801ac10 crc :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:38.643 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:38.642 7fce7f027700 1 -- 172.21.2.112:0/969796050 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fce780c0550 msgr2=0x7fce780c0950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:38.643 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:38.642 7fce7f027700 1 --2- 172.21.2.112:0/969796050 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fce780c0550 0x7fce780c0950 secure :-1 s=READY pgs=2629 cs=0 l=1 rev1=1 rx=0x7fce700099a0 tx=0x7fce7000bcf0).stop 2021-02-10T11:44:38.643 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:38.642 7fce7f027700 1 -- 172.21.2.112:0/969796050 shutdown_connections 2021-02-10T11:44:38.644 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:38.642 7fce7f027700 1 --2- 172.21.2.112:0/969796050 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fce58018730 0x7fce5801ac10 unknown :-1 s=CLOSED pgs=1200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:38.644 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:38.642 7fce7f027700 1 --2- 172.21.2.112:0/969796050 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fce780bfb30 0x7fce780bff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:38.644 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:38.642 7fce7f027700 1 --2- 172.21.2.112:0/969796050 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fce780c0550 0x7fce780c0950 unknown :-1 s=CLOSED pgs=2629 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:38.644 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:38.642 7fce7f027700 1 --2- 172.21.2.112:0/969796050 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fce781a3550 0x7fce781a3950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:38.644 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:38.642 7fce7f027700 1 -- 172.21.2.112:0/969796050 >> 172.21.2.112:0/969796050 conn(0x7fce780a0290 msgr2=0x7fce7808e560 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:38.644 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:38.642 7fce7f027700 1 -- 172.21.2.112:0/969796050 shutdown_connections 2021-02-10T11:44:38.644 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:38.642 7fce7f027700 1 -- 172.21.2.112:0/969796050 wait complete. 2021-02-10T11:44:38.652 INFO:tasks.cephfs.cephfs_test_case:['0', '1', '2', '3'] 2021-02-10T11:44:38.652 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth ls --format=json-pretty 2021-02-10T11:44:39.082 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.063 7fc4e72e4700 1 Processor -- start 2021-02-10T11:44:39.082 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.063 7fc4e72e4700 1 -- start start 2021-02-10T11:44:39.082 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.064 7fc4e72e4700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fc4e01b0870 0x7fc4e00f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:39.082 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.064 7fc4e72e4700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc4e00efe00 0x7fc4e00f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:39.082 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.064 7fc4e72e4700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4e01b0c70 con 0x7fc4e01b0470 2021-02-10T11:44:39.082 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.064 7fc4e72e4700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4e00f0b50 con 0x7fc4e01b0870 2021-02-10T11:44:39.082 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.064 7fc4e72e4700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4e00f0820 con 0x7fc4e00efe00 2021-02-10T11:44:39.082 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.065 7fc4e48ff700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fc4e01b0870 0x7fc4e00f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:39.082 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.065 7fc4e5901700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc4e00efe00 0x7fc4e00f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:39.083 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.065 7fc4e5100700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7fc4e01b0470 0x7fc4e00ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:51038/0 (socket says 172.21.2.112:51038) 2021-02-10T11:44:39.083 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.065 7fc4e5100700 1 -- 172.21.2.112:0/1965519281 learned_addr learned my addr 172.21.2.112:0/1965519281 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:44:39.083 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.065 7fc4e48ff700 1 -- 172.21.2.112:0/1965519281 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc4e00efe00 msgr2=0x7fc4e00f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:39.083 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.065 7fc4e48ff700 1 --2- 172.21.2.112:0/1965519281 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc4e00efe00 0x7fc4e00f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:39.083 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.065 7fc4e48ff700 1 -- 172.21.2.112:0/1965519281 >> v1:172.21.2.120:6789/0 conn(0x7fc4e01b0470 legacy=0x7fc4e00ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:39.083 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.065 7fc4e48ff700 1 -- 172.21.2.112:0/1965519281 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4e00e52e0 con 0x7fc4e01b0870 2021-02-10T11:44:39.083 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.065 7fc4e5901700 1 --2- 172.21.2.112:0/1965519281 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc4e00efe00 0x7fc4e00f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:44:39.083 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.065 7fc4e48ff700 1 --2- 172.21.2.112:0/1965519281 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc4e01b0870 0x7fc4e00f20e0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 rx=0x7fc4dc006650 tx=0x7fc4dc009a80).ready entity=mon.1 client_cookie=d8960b5f633496c2 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:39.083 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.066 7fc4d3fff700 1 -- 172.21.2.112:0/1965519281 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc4dc0009b0 con 0x7fc4e01b0870 2021-02-10T11:44:39.083 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.066 7fc4d3fff700 1 -- 172.21.2.112:0/1965519281 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4dc006390 con 0x7fc4e01b0870 2021-02-10T11:44:39.083 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.066 7fc4d3fff700 1 -- 172.21.2.112:0/1965519281 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc4dc00c0c0 con 0x7fc4e01b0870 2021-02-10T11:44:39.083 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.066 7fc4e72e4700 1 -- 172.21.2.112:0/1965519281 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc4e01b0870 msgr2=0x7fc4e00f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:39.083 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.066 7fc4e72e4700 1 --2- 172.21.2.112:0/1965519281 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc4e01b0870 0x7fc4e00f20e0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 rx=0x7fc4dc006650 tx=0x7fc4dc009a80).stop 2021-02-10T11:44:39.084 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.066 7fc4e72e4700 1 -- 172.21.2.112:0/1965519281 shutdown_connections 2021-02-10T11:44:39.084 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.066 7fc4e72e4700 1 --2- 172.21.2.112:0/1965519281 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc4e01b0870 0x7fc4e00f20e0 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:39.084 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.066 7fc4e72e4700 1 --2- 172.21.2.112:0/1965519281 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc4e00efe00 0x7fc4e00f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:39.084 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.066 7fc4e72e4700 1 -- 172.21.2.112:0/1965519281 >> 172.21.2.112:0/1965519281 conn(0x7fc4e00ce3e0 msgr2=0x7fc4e00e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:39.084 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.067 7fc4e72e4700 1 -- 172.21.2.112:0/1965519281 shutdown_connections 2021-02-10T11:44:39.084 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.067 7fc4e72e4700 1 -- 172.21.2.112:0/1965519281 wait complete. 2021-02-10T11:44:39.085 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.081 7fc4e72e4700 1 Processor -- start 2021-02-10T11:44:39.085 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.081 7fc4e72e4700 1 -- start start 2021-02-10T11:44:39.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.081 7fc4e72e4700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc4e00980c0 0x7fc4e00b4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:39.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.081 7fc4e72e4700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc4e00b5380 0x7fc4e0090870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:39.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.081 7fc4e72e4700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc4e0090e90 0x7fc4e00b65e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:39.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.081 7fc4e72e4700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4e00918d0 con 0x7fc4e00b5380 2021-02-10T11:44:39.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.081 7fc4e72e4700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4e0091390 con 0x7fc4e0090e90 2021-02-10T11:44:39.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.081 7fc4e72e4700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4e00b6c00 con 0x7fc4e00980c0 2021-02-10T11:44:39.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.082 7fc4e48ff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc4e00b5380 0x7fc4e0090870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:39.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.082 7fc4e48ff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc4e00b5380 0x7fc4e0090870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:58026/0 (socket says 172.21.2.112:58026) 2021-02-10T11:44:39.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.082 7fc4e48ff700 1 -- 172.21.2.112:0/716699234 learned_addr learned my addr 172.21.2.112:0/716699234 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:44:39.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.082 7fc4e48ff700 1 -- 172.21.2.112:0/716699234 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc4e00980c0 msgr2=0x7fc4e00b4d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:39.087 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.082 7fc4e48ff700 1 --2- 172.21.2.112:0/716699234 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc4e00980c0 0x7fc4e00b4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:39.087 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.082 7fc4e48ff700 1 -- 172.21.2.112:0/716699234 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc4e0090e90 msgr2=0x7fc4e00b65e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:39.087 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.082 7fc4e48ff700 1 --2- 172.21.2.112:0/716699234 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc4e0090e90 0x7fc4e00b65e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:39.087 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.082 7fc4e48ff700 1 -- 172.21.2.112:0/716699234 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4e00b6e20 con 0x7fc4e00b5380 2021-02-10T11:44:39.087 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.082 7fc4e48ff700 1 --2- 172.21.2.112:0/716699234 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc4e00b5380 0x7fc4e0090870 secure :-1 s=READY pgs=2630 cs=0 l=1 rev1=1 rx=0x7fc4dc009630 tx=0x7fc4dc00b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:39.087 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.082 7fc4d1ffb700 1 -- 172.21.2.112:0/716699234 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc4dc0093a0 con 0x7fc4e00b5380 2021-02-10T11:44:39.087 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.082 7fc4d1ffb700 1 -- 172.21.2.112:0/716699234 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4dc00bd70 con 0x7fc4e00b5380 2021-02-10T11:44:39.087 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.082 7fc4e72e4700 1 -- 172.21.2.112:0/716699234 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4e00b6e20 con 0x7fc4e00b5380 2021-02-10T11:44:39.087 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.082 7fc4d1ffb700 1 -- 172.21.2.112:0/716699234 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc4dc005560 con 0x7fc4e00b5380 2021-02-10T11:44:39.087 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.082 7fc4e72e4700 1 -- 172.21.2.112:0/716699234 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4e00b6e20 con 0x7fc4e00b5380 2021-02-10T11:44:39.087 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.083 7fc4d1ffb700 1 -- 172.21.2.112:0/716699234 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc4dc005770 con 0x7fc4e00b5380 2021-02-10T11:44:39.087 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.083 7fc4d1ffb700 1 --2- 172.21.2.112:0/716699234 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc4c0018730 0x7fc4c001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:39.088 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.083 7fc4d1ffb700 1 -- 172.21.2.112:0/716699234 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 4753+0+0 (secure 0 0 0) 0x7fc4dc0322e0 con 0x7fc4e00b5380 2021-02-10T11:44:39.088 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.083 7fc4e5100700 1 --2- 172.21.2.112:0/716699234 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc4c0018730 0x7fc4c001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:39.088 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.084 7fc4e5100700 1 --2- 172.21.2.112:0/716699234 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc4c0018730 0x7fc4c001ac10 crc :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:39.114 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.113 7fc4e72e4700 1 -- 172.21.2.112:0/716699234 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4e0105a40 con 0x7fc4e00b5380 2021-02-10T11:44:39.117 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.116 7fc4d1ffb700 1 -- 172.21.2.112:0/716699234 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc4dc005560 con 0x7fc4e00b5380 2021-02-10T11:44:39.250 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.248 7fc4e72e4700 1 -- 172.21.2.112:0/716699234 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fc4e00f0b50 con 0x7fc4e00b5380 2021-02-10T11:44:39.250 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.249 7fc4d1ffb700 1 -- 172.21.2.112:0/716699234 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v86) v1 ==== 80+0+4513 (secure 0 0 0) 0x7fc4dc005770 con 0x7fc4e00b5380 2021-02-10T11:44:39.250 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:44:39.251 INFO:teuthology.orchestra.run.gibba012.stdout:{ 2021-02-10T11:44:39.251 INFO:teuthology.orchestra.run.gibba012.stdout: "auth_dump": [ 2021-02-10T11:44:39.251 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:44:39.251 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "mds.a", 2021-02-10T11:44:39.251 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCKvyNgPkV4AxAAEmVNaghdccP6lWd8hvQnxw==", 2021-02-10T11:44:39.251 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:44:39.251 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:44:39.251 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:44:39.251 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:44:39.251 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:44:39.252 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:44:39.252 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:44:39.252 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:44:39.252 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "mds.b", 2021-02-10T11:44:39.252 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCKvyNguxONLhAAHIpry+R89MAwdhFJyEhVCw==", 2021-02-10T11:44:39.252 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:44:39.252 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:44:39.252 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:44:39.252 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:44:39.252 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:44:39.252 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:44:39.252 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:44:39.252 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:44:39.253 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.0", 2021-02-10T11:44:39.253 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCSvyNgXcKPLhAAP/Bj2F+06DCtPCzQDNq+cQ==", 2021-02-10T11:44:39.253 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:44:39.253 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:44:39.253 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:44:39.253 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:44:39.253 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:44:39.253 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:44:39.253 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:44:39.253 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.1", 2021-02-10T11:44:39.253 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCVvyNghfbcLhAA66PrRwR2M7o6VTF5UhU/Fg==", 2021-02-10T11:44:39.253 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:44:39.253 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:44:39.253 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:44:39.254 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:44:39.254 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:44:39.254 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:44:39.254 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:44:39.254 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.2", 2021-02-10T11:44:39.254 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCYvyNgwiPqLhAA66BVkRsgaUlGpAsQ15Nf8Q==", 2021-02-10T11:44:39.254 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:44:39.254 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:44:39.254 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:44:39.254 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:44:39.254 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:44:39.254 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:44:39.255 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:44:39.255 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.3", 2021-02-10T11:44:39.255 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCbvyNgqBZrLxAAERPe/1L3SXAEzI3d9ReHHg==", 2021-02-10T11:44:39.255 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:44:39.255 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:44:39.255 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:44:39.255 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:44:39.255 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:44:39.255 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:44:39.255 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:44:39.255 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.4", 2021-02-10T11:44:39.255 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCkvyNgDQsxIhAAHmVjVYnx20N4SDmKqvbcuA==", 2021-02-10T11:44:39.255 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:44:39.255 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:44:39.256 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:44:39.256 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:44:39.256 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:44:39.256 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:44:39.256 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:44:39.256 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.5", 2021-02-10T11:44:39.256 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCnvyNg8ITFIhAADd4aKybAV7btoGPx3kCg3A==", 2021-02-10T11:44:39.256 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:44:39.256 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:44:39.256 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:44:39.256 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:44:39.256 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:44:39.256 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:44:39.256 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:44:39.257 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.6", 2021-02-10T11:44:39.257 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCqvyNgxOhDIxAAzXixEa/nlDc1xXrg0aeLwg==", 2021-02-10T11:44:39.257 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:44:39.257 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:44:39.257 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:44:39.257 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:44:39.257 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:44:39.257 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:44:39.257 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:44:39.257 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.7", 2021-02-10T11:44:39.257 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCtvyNgxSqGIxAACxJQlwP7DpiVVCZHA3zn1A==", 2021-02-10T11:44:39.257 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:44:39.257 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:44:39.258 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:44:39.258 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:44:39.258 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:44:39.258 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:44:39.258 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:44:39.258 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.0", 2021-02-10T11:44:39.258 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCLvyNgDkNiHRAAygU8Za4PBusTqCgwAYWTiQ==", 2021-02-10T11:44:39.258 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:44:39.258 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:44:39.258 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow r", 2021-02-10T11:44:39.258 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:44:39.258 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:44:39.259 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:44:39.259 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:44:39.259 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.1", 2021-02-10T11:44:39.259 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCLvyNgxmLqMxAA1MlSDJ9LH9u4f2sL7mm8/Q==", 2021-02-10T11:44:39.259 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:44:39.259 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:44:39.259 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow r", 2021-02-10T11:44:39.259 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:44:39.259 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:44:39.259 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:44:39.259 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:44:39.259 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.2", 2021-02-10T11:44:39.259 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCMvyNgp8veDhAAniSVHYPtwwTog/pjbaRQmw==", 2021-02-10T11:44:39.260 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:44:39.260 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:44:39.260 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow r", 2021-02-10T11:44:39.260 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:44:39.260 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:44:39.260 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:44:39.260 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:44:39.260 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.3", 2021-02-10T11:44:39.260 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCMvyNgQEVoJRAApHcWdm1sldIxu+heyD4HdQ==", 2021-02-10T11:44:39.260 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:44:39.260 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:44:39.260 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow r", 2021-02-10T11:44:39.261 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:44:39.261 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:44:39.261 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:44:39.261 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:44:39.261 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.admin", 2021-02-10T11:44:39.261 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCGvyNg7jjJBRAAtot+ypUZOiTo1GbY+oVhVw==", 2021-02-10T11:44:39.261 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:44:39.261 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow *", 2021-02-10T11:44:39.261 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:44:39.261 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:44:39.261 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:44:39.261 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:44:39.261 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:44:39.262 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:44:39.262 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "mgr.x", 2021-02-10T11:44:39.262 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCJvyNgTPVwKBAAjduiBE8+RSXBv82tdPecWg==", 2021-02-10T11:44:39.262 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:44:39.262 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow *", 2021-02-10T11:44:39.262 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow profile mgr", 2021-02-10T11:44:39.262 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:44:39.262 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:44:39.262 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:44:39.262 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:44:39.262 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "mgr.y", 2021-02-10T11:44:39.262 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCJvyNgQ1tTDhAASwBssDy+Rz31ujw8fNlWow==", 2021-02-10T11:44:39.262 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:44:39.263 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow *", 2021-02-10T11:44:39.263 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow profile mgr", 2021-02-10T11:44:39.263 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:44:39.263 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:44:39.263 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:44:39.263 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:44:39.263 INFO:teuthology.orchestra.run.gibba012.stdout:} 2021-02-10T11:44:39.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.251 7fc4e72e4700 1 -- 172.21.2.112:0/716699234 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc4c0018730 msgr2=0x7fc4c001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:39.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.251 7fc4e72e4700 1 --2- 172.21.2.112:0/716699234 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc4c0018730 0x7fc4c001ac10 crc :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:39.265 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.251 7fc4e72e4700 1 -- 172.21.2.112:0/716699234 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc4e00b5380 msgr2=0x7fc4e0090870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:39.265 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.251 7fc4e72e4700 1 --2- 172.21.2.112:0/716699234 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc4e00b5380 0x7fc4e0090870 secure :-1 s=READY pgs=2630 cs=0 l=1 rev1=1 rx=0x7fc4dc009630 tx=0x7fc4dc00b980).stop 2021-02-10T11:44:39.265 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.251 7fc4e72e4700 1 -- 172.21.2.112:0/716699234 shutdown_connections 2021-02-10T11:44:39.265 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.251 7fc4e72e4700 1 --2- 172.21.2.112:0/716699234 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc4c0018730 0x7fc4c001ac10 unknown :-1 s=CLOSED pgs=1201 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:39.265 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.251 7fc4e72e4700 1 --2- 172.21.2.112:0/716699234 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc4e0090e90 0x7fc4e00b65e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:39.265 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.251 7fc4e72e4700 1 --2- 172.21.2.112:0/716699234 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc4e00b5380 0x7fc4e0090870 unknown :-1 s=CLOSED pgs=2630 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:39.265 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.251 7fc4e72e4700 1 --2- 172.21.2.112:0/716699234 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc4e00980c0 0x7fc4e00b4d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:39.265 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.251 7fc4e72e4700 1 -- 172.21.2.112:0/716699234 >> 172.21.2.112:0/716699234 conn(0x7fc4e00c2c70 msgr2=0x7fc4e00c3ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:39.265 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.251 7fc4e72e4700 1 -- 172.21.2.112:0/716699234 shutdown_connections 2021-02-10T11:44:39.265 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.251 7fc4e72e4700 1 -- 172.21.2.112:0/716699234 wait complete. 2021-02-10T11:44:39.266 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:44:39.695 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.676 7f5fbbe0b700 1 Processor -- start 2021-02-10T11:44:39.695 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.676 7f5fbbe0b700 1 -- start start 2021-02-10T11:44:39.695 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.677 7f5fbbe0b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5fb40d0340 0x7f5fb40ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:39.695 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.677 7f5fbbe0b700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fb40f0b10 con 0x7f5fb40d1870 2021-02-10T11:44:39.695 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.677 7f5fbbe0b700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fb41b0be0 con 0x7f5fb40d1c70 2021-02-10T11:44:39.695 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.677 7f5fbbe0b700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fb40e5ed0 con 0x7f5fb40d0340 2021-02-10T11:44:39.695 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.677 7f5fb9c27700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f5fb40d1870 0x7f5fb41b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:57494/0 (socket says 172.21.2.112:57494) 2021-02-10T11:44:39.695 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.677 7f5fb9c27700 1 -- 172.21.2.112:0/3660520828 learned_addr learned my addr 172.21.2.112:0/3660520828 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:44:39.695 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.677 7f5fb8c25700 1 -- 172.21.2.112:0/3660520828 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2876729545 0 0) 0x7f5fa8001180 con 0x7f5fb40d1870 2021-02-10T11:44:39.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.677 7f5fb8c25700 1 -- 172.21.2.112:0/3660520828 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fa0002fe0 con 0x7f5fb40d1870 2021-02-10T11:44:39.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.677 7f5fb9426700 1 --2- 172.21.2.112:0/3660520828 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5fb40d0340 0x7f5fb40ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:39.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.678 7f5fb8c25700 1 -- 172.21.2.112:0/3660520828 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 865803321 0 0) 0x7f5fa8001180 con 0x7f5fb40d1870 2021-02-10T11:44:39.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.678 7f5fb8c25700 1 -- 172.21.2.112:0/3660520828 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5fb40d0340 msgr2=0x7f5fb40ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:39.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.678 7f5fb8c25700 1 --2- 172.21.2.112:0/3660520828 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5fb40d0340 0x7f5fb40ef640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:39.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.678 7f5fb8c25700 1 -- 172.21.2.112:0/3660520828 >> v1:172.21.2.120:6789/0 conn(0x7f5fb40d1c70 legacy=0x7f5fb40efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:39.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.678 7f5fb8c25700 1 -- 172.21.2.112:0/3660520828 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fb40f0450 con 0x7f5fb40d1870 2021-02-10T11:44:39.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.678 7f5fb8c25700 1 -- 172.21.2.112:0/3660520828 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f5fa8001680 con 0x7f5fb40d1870 2021-02-10T11:44:39.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.678 7f5fb8c25700 1 -- 172.21.2.112:0/3660520828 >> v1:172.21.2.112:6789/0 conn(0x7f5fb40d1870 legacy=0x7f5fb41b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:39.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.678 7f5fb8c25700 1 --2- 172.21.2.112:0/3660520828 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5fa0003ae0 0x7f5fa0005f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:39.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.678 7f5fb8c25700 1 --2- 172.21.2.112:0/3660520828 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5fa0006790 0x7f5fa0008cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:39.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.678 7f5fb8c25700 1 --2- 172.21.2.112:0/3660520828 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5fa00095b0 0x7f5fa000ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:39.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.678 7f5fb8c25700 1 -- 172.21.2.112:0/3660520828 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fa000c2d0 con 0x7f5fa0003ae0 2021-02-10T11:44:39.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.678 7f5fb8c25700 1 -- 172.21.2.112:0/3660520828 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fa000c4f0 con 0x7f5fa0006790 2021-02-10T11:44:39.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.678 7f5fb8c25700 1 -- 172.21.2.112:0/3660520828 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fa000c710 con 0x7f5fa00095b0 2021-02-10T11:44:39.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.678 7f5fb9c27700 1 --2- 172.21.2.112:0/3660520828 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5fa0003ae0 0x7f5fa0005f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:39.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.678 7f5fb9426700 1 --2- 172.21.2.112:0/3660520828 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5fa0006790 0x7f5fa0008cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:39.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.678 7f5fb9c27700 1 -- 172.21.2.112:0/3660520828 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5fa00095b0 msgr2=0x7f5fa000ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:39.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.678 7f5fb9c27700 1 --2- 172.21.2.112:0/3660520828 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5fa00095b0 0x7f5fa000ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:39.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.678 7f5fb9c27700 1 -- 172.21.2.112:0/3660520828 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5fa0006790 msgr2=0x7f5fa0008cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:39.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.678 7f5fb9c27700 1 --2- 172.21.2.112:0/3660520828 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5fa0006790 0x7f5fa0008cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:39.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.678 7f5fb9c27700 1 -- 172.21.2.112:0/3660520828 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fa0050980 con 0x7f5fa0003ae0 2021-02-10T11:44:39.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.678 7f5fb9c27700 1 --2- 172.21.2.112:0/3660520828 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5fa0003ae0 0x7f5fa0005f70 secure :-1 s=READY pgs=2632 cs=0 l=1 rev1=1 rx=0x7f5fa80014e0 tx=0x7f5fa8007ca0).ready entity=mon.0 client_cookie=73c0de292770f1d4 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:39.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.679 7f5fb8c25700 1 -- 172.21.2.112:0/3660520828 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5fa80070c0 con 0x7f5fa0003ae0 2021-02-10T11:44:39.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.679 7f5fb8c25700 1 -- 172.21.2.112:0/3660520828 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fa80043a0 con 0x7f5fa0003ae0 2021-02-10T11:44:39.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.679 7f5fb8c25700 1 -- 172.21.2.112:0/3660520828 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5fa8004820 con 0x7f5fa0003ae0 2021-02-10T11:44:39.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.679 7f5fbbe0b700 1 -- 172.21.2.112:0/3660520828 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5fa0003ae0 msgr2=0x7f5fa0005f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:39.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.679 7f5fbbe0b700 1 --2- 172.21.2.112:0/3660520828 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5fa0003ae0 0x7f5fa0005f70 secure :-1 s=READY pgs=2632 cs=0 l=1 rev1=1 rx=0x7f5fa80014e0 tx=0x7f5fa8007ca0).stop 2021-02-10T11:44:39.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.679 7f5fba428700 1 -- 172.21.2.112:0/3660520828 reap_dead start 2021-02-10T11:44:39.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.679 7f5fbbe0b700 1 -- 172.21.2.112:0/3660520828 shutdown_connections 2021-02-10T11:44:39.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.679 7f5fbbe0b700 1 -- 172.21.2.112:0/3660520828 >> 172.21.2.112:0/3660520828 conn(0x7f5fb40ce450 msgr2=0x7f5fb40d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:39.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.679 7f5fbbe0b700 1 -- 172.21.2.112:0/3660520828 shutdown_connections 2021-02-10T11:44:39.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.679 7f5fbbe0b700 1 -- 172.21.2.112:0/3660520828 wait complete. 2021-02-10T11:44:39.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.694 7f5fbbe0b700 1 Processor -- start 2021-02-10T11:44:39.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.694 7f5fbbe0b700 1 -- start start 2021-02-10T11:44:39.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.694 7f5fbbe0b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5fb4195650 0x7f5fb40b5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:39.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.694 7f5fbbe0b700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5fb40b5b80 0x7f5fb41a2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:39.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.694 7f5fbbe0b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5fb40acfa0 0x7f5fb41a3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:39.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.694 7f5fbbe0b700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fb40b6010 con 0x7f5fb4195650 2021-02-10T11:44:39.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.694 7f5fbbe0b700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fb41a3a60 con 0x7f5fb40b5b80 2021-02-10T11:44:39.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.694 7f5fbbe0b700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fb41a3c40 con 0x7f5fb40acfa0 2021-02-10T11:44:39.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.695 7f5fb9c27700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5fb4195650 0x7f5fb40b5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:39.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.695 7f5fb9c27700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5fb4195650 0x7f5fb40b5560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:58046/0 (socket says 172.21.2.112:58046) 2021-02-10T11:44:39.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.695 7f5fb9c27700 1 -- 172.21.2.112:0/3431093111 learned_addr learned my addr 172.21.2.112:0/3431093111 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:44:39.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.695 7f5fb9c27700 1 -- 172.21.2.112:0/3431093111 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5fb40acfa0 msgr2=0x7f5fb41a3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:39.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.695 7f5fb9c27700 1 --2- 172.21.2.112:0/3431093111 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5fb40acfa0 0x7f5fb41a3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:39.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.695 7f5fb9c27700 1 -- 172.21.2.112:0/3431093111 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5fb40b5b80 msgr2=0x7f5fb41a2e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:39.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.695 7f5fb9c27700 1 --2- 172.21.2.112:0/3431093111 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5fb40b5b80 0x7f5fb41a2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:39.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.695 7f5fb9c27700 1 -- 172.21.2.112:0/3431093111 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fb40bcae0 con 0x7f5fb4195650 2021-02-10T11:44:39.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.695 7f5fb9c27700 1 --2- 172.21.2.112:0/3431093111 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5fb4195650 0x7f5fb40b5560 secure :-1 s=READY pgs=2633 cs=0 l=1 rev1=1 rx=0x7f5fa8001b30 tx=0x7f5fa800a9a0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:39.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.695 7f5fa67fc700 1 -- 172.21.2.112:0/3431093111 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5fa800ad90 con 0x7f5fb4195650 2021-02-10T11:44:39.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.695 7f5fa67fc700 1 -- 172.21.2.112:0/3431093111 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fa8001b70 con 0x7f5fb4195650 2021-02-10T11:44:39.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.695 7f5fa67fc700 1 -- 172.21.2.112:0/3431093111 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5fa80043a0 con 0x7f5fb4195650 2021-02-10T11:44:39.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.695 7f5fbbe0b700 1 -- 172.21.2.112:0/3431093111 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fb40bcae0 con 0x7f5fb4195650 2021-02-10T11:44:39.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.695 7f5fbbe0b700 1 -- 172.21.2.112:0/3431093111 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fb40bcae0 con 0x7f5fb4195650 2021-02-10T11:44:39.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.696 7f5fa67fc700 1 -- 172.21.2.112:0/3431093111 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5fa80043a0 con 0x7f5fb4195650 2021-02-10T11:44:39.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.696 7f5fa67fc700 1 --2- 172.21.2.112:0/3431093111 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5f9c0186c0 0x7f5f9c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:39.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.696 7f5fa67fc700 1 -- 172.21.2.112:0/3431093111 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f5fa8032280 con 0x7f5fb4195650 2021-02-10T11:44:39.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.696 7f5fb9426700 1 --2- 172.21.2.112:0/3431093111 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5f9c0186c0 0x7f5f9c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:39.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.697 7f5fb9426700 1 --2- 172.21.2.112:0/3431093111 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5f9c0186c0 0x7f5f9c01abc0 crc :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:39.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.727 7f5fbbe0b700 1 -- 172.21.2.112:0/3431093111 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fb40b6010 con 0x7f5fb4195650 2021-02-10T11:44:39.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.730 7f5fa67fc700 1 -- 172.21.2.112:0/3431093111 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5fa8030d60 con 0x7f5fb4195650 2021-02-10T11:44:39.856 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.855 7f5fbbe0b700 1 -- 172.21.2.112:0/3431093111 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5fb40b6010 con 0x7f5fb4195650 2021-02-10T11:44:39.857 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.855 7f5fa67fc700 1 -- 172.21.2.112:0/3431093111 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v147) v1 ==== 74+0+11154 (secure 0 0 0) 0x7f5fa801f750 con 0x7f5fb4195650 2021-02-10T11:44:39.857 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:44:39.857 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":147,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:44:38.611175","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":31,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":139,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":139,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:44:39.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.857 7f5fbbe0b700 1 -- 172.21.2.112:0/3431093111 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5f9c0186c0 msgr2=0x7f5f9c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:39.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.857 7f5fbbe0b700 1 --2- 172.21.2.112:0/3431093111 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5f9c0186c0 0x7f5f9c01abc0 crc :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:39.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.857 7f5fbbe0b700 1 -- 172.21.2.112:0/3431093111 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5fb4195650 msgr2=0x7f5fb40b5560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:39.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.857 7f5fbbe0b700 1 --2- 172.21.2.112:0/3431093111 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5fb4195650 0x7f5fb40b5560 secure :-1 s=READY pgs=2633 cs=0 l=1 rev1=1 rx=0x7f5fa8001b30 tx=0x7f5fa800a9a0).stop 2021-02-10T11:44:39.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.857 7f5fbbe0b700 1 -- 172.21.2.112:0/3431093111 shutdown_connections 2021-02-10T11:44:39.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.857 7f5fbbe0b700 1 --2- 172.21.2.112:0/3431093111 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5f9c0186c0 0x7f5f9c01abc0 unknown :-1 s=CLOSED pgs=1202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:39.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.857 7f5fbbe0b700 1 --2- 172.21.2.112:0/3431093111 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5fb40b5b80 0x7f5fb41a2e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:39.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.857 7f5fbbe0b700 1 --2- 172.21.2.112:0/3431093111 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5fb40acfa0 0x7f5fb41a3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:39.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.857 7f5fbbe0b700 1 --2- 172.21.2.112:0/3431093111 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5fb4195650 0x7f5fb40b5560 unknown :-1 s=CLOSED pgs=2633 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:39.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.858 7f5fbbe0b700 1 -- 172.21.2.112:0/3431093111 >> 172.21.2.112:0/3431093111 conn(0x7f5fb40b3040 msgr2=0x7f5fb40c3250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:39.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.858 7f5fbbe0b700 1 -- 172.21.2.112:0/3431093111 shutdown_connections 2021-02-10T11:44:39.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:39.858 7f5fbbe0b700 1 -- 172.21.2.112:0/3431093111 wait complete. 2021-02-10T11:44:39.867 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:13:57.508758', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2021-02-10T11:44:39.868 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:44:40.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.282 7f3031a58700 1 Processor -- start 2021-02-10T11:44:40.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.282 7f3031a58700 1 -- start start 2021-02-10T11:44:40.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.282 7f3031a58700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f302c0d1c70 0x7f302c0cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:40.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.282 7f3031a58700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f302c0d0170 con 0x7f302c0d1870 2021-02-10T11:44:40.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.282 7f3031a58700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f302c0fe1c0 con 0x7f302c0cf580 2021-02-10T11:44:40.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.282 7f3031a58700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f302c1b0cb0 con 0x7f302c0d1c70 2021-02-10T11:44:40.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.283 7f302b7fe700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f302c0d1870 0x7f302c1b04c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:57514/0 (socket says 172.21.2.112:57514) 2021-02-10T11:44:40.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.283 7f302b7fe700 1 -- 172.21.2.112:0/2584750912 learned_addr learned my addr 172.21.2.112:0/2584750912 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:44:40.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.283 7f302a7fc700 1 -- 172.21.2.112:0/2584750912 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 551317063 0 0) 0x7f3014001180 con 0x7f302c0d1870 2021-02-10T11:44:40.302 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.283 7f302a7fc700 1 -- 172.21.2.112:0/2584750912 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3018002fe0 con 0x7f302c0d1870 2021-02-10T11:44:40.302 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.283 7f302affd700 1 --2- 172.21.2.112:0/2584750912 >> v2:172.21.2.120:3300/0 conn(0x7f302c0d1c70 0x7f302c0cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:40.302 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.283 7f302a7fc700 1 -- 172.21.2.112:0/2584750912 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2774111240 0 0) 0x7f3014001180 con 0x7f302c0d1870 2021-02-10T11:44:40.302 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.283 7f302a7fc700 1 -- 172.21.2.112:0/2584750912 >> v2:172.21.2.120:3300/0 conn(0x7f302c0d1c70 msgr2=0x7f302c0cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:40.302 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.283 7f302a7fc700 1 --2- 172.21.2.112:0/2584750912 >> v2:172.21.2.120:3300/0 conn(0x7f302c0d1c70 0x7f302c0cef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:40.302 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.283 7f302a7fc700 1 -- 172.21.2.112:0/2584750912 >> v1:172.21.2.120:6789/0 conn(0x7f302c0cf580 legacy=0x7f302c0cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:40.302 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.283 7f302a7fc700 1 -- 172.21.2.112:0/2584750912 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f302c109a60 con 0x7f302c0d1870 2021-02-10T11:44:40.302 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.283 7f302a7fc700 1 -- 172.21.2.112:0/2584750912 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f3014001680 con 0x7f302c0d1870 2021-02-10T11:44:40.302 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.283 7f302a7fc700 1 -- 172.21.2.112:0/2584750912 >> v1:172.21.2.112:6789/0 conn(0x7f302c0d1870 legacy=0x7f302c1b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:40.302 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.283 7f302a7fc700 1 --2- 172.21.2.112:0/2584750912 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3018003b10 0x7f3018005f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:40.302 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.284 7f302a7fc700 1 --2- 172.21.2.112:0/2584750912 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3018006820 0x7f3018008d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:40.302 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.284 7f302a7fc700 1 --2- 172.21.2.112:0/2584750912 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3018009620 0x7f301800bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:40.303 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.284 7f302a7fc700 1 -- 172.21.2.112:0/2584750912 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f301800c390 con 0x7f3018009620 2021-02-10T11:44:40.303 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.284 7f302a7fc700 1 -- 172.21.2.112:0/2584750912 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f301800c5b0 con 0x7f3018006820 2021-02-10T11:44:40.303 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.284 7f302a7fc700 1 -- 172.21.2.112:0/2584750912 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f301800c7d0 con 0x7f3018003b10 2021-02-10T11:44:40.303 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.284 7f302affd700 1 --2- 172.21.2.112:0/2584750912 >> v2:172.21.2.120:3300/0 conn(0x7f302c0d1c70 0x7f302c0cef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:44:40.303 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.284 7f302bfff700 1 --2- 172.21.2.112:0/2584750912 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3018009620 0x7f301800bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:40.303 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.284 7f302b7fe700 1 --2- 172.21.2.112:0/2584750912 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3018003b10 0x7f3018005f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:40.303 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.284 7f302bfff700 1 -- 172.21.2.112:0/2584750912 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3018003b10 msgr2=0x7f3018005f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:40.303 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.284 7f302bfff700 1 --2- 172.21.2.112:0/2584750912 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3018003b10 0x7f3018005f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:40.303 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.284 7f302bfff700 1 -- 172.21.2.112:0/2584750912 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3018006820 msgr2=0x7f3018008d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:40.303 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.284 7f302bfff700 1 --2- 172.21.2.112:0/2584750912 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3018006820 0x7f3018008d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:40.303 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.284 7f302bfff700 1 -- 172.21.2.112:0/2584750912 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f301800ca70 con 0x7f3018009620 2021-02-10T11:44:40.303 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.284 7f302bfff700 1 -- 172.21.2.112:0/2584750912 reap_dead start 2021-02-10T11:44:40.303 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.284 7f302bfff700 1 --2- 172.21.2.112:0/2584750912 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3018009620 0x7f301800bb40 secure :-1 s=READY pgs=2635 cs=0 l=1 rev1=1 rx=0x7f302404ae50 tx=0x7f302404de70).ready entity=mon.0 client_cookie=693fc0bf8c116a0f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:40.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.284 7f302a7fc700 1 -- 172.21.2.112:0/2584750912 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f302404f300 con 0x7f3018009620 2021-02-10T11:44:40.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.284 7f302a7fc700 1 -- 172.21.2.112:0/2584750912 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f302404f540 con 0x7f3018009620 2021-02-10T11:44:40.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.284 7f302a7fc700 1 -- 172.21.2.112:0/2584750912 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f302404fa60 con 0x7f3018009620 2021-02-10T11:44:40.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.284 7f3031a58700 1 -- 172.21.2.112:0/2584750912 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3018009620 msgr2=0x7f301800bb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:40.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.284 7f3031a58700 1 --2- 172.21.2.112:0/2584750912 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3018009620 0x7f301800bb40 secure :-1 s=READY pgs=2635 cs=0 l=1 rev1=1 rx=0x7f302404ae50 tx=0x7f302404de70).stop 2021-02-10T11:44:40.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.285 7f3031a58700 1 -- 172.21.2.112:0/2584750912 shutdown_connections 2021-02-10T11:44:40.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.285 7f3031a58700 1 --2- 172.21.2.112:0/2584750912 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3018009620 0x7f301800bb40 unknown :-1 s=CLOSED pgs=2635 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:40.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.285 7f3031a58700 1 -- 172.21.2.112:0/2584750912 >> 172.21.2.112:0/2584750912 conn(0x7f302c0ce420 msgr2=0x7f302c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:40.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.285 7f3031a58700 1 -- 172.21.2.112:0/2584750912 shutdown_connections 2021-02-10T11:44:40.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.285 7f3031a58700 1 -- 172.21.2.112:0/2584750912 wait complete. 2021-02-10T11:44:40.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.300 7f3031a58700 1 Processor -- start 2021-02-10T11:44:40.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.300 7f3031a58700 1 -- start start 2021-02-10T11:44:40.306 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.300 7f3031a58700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f302c085bc0 0x7f302c0863e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:40.306 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.300 7f3031a58700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f302c0954c0 0x7f302c096300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:40.306 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.300 7f3031a58700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f302c0958c0 0x7f302c096920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:40.306 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.300 7f3031a58700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f302c095d80 con 0x7f302c0954c0 2021-02-10T11:44:40.306 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.300 7f3031a58700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f302c0bfa60 con 0x7f302c0958c0 2021-02-10T11:44:40.306 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.300 7f3031a58700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f302c085fc0 con 0x7f302c085bc0 2021-02-10T11:44:40.306 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.301 7f302affd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f302c0954c0 0x7f302c096300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:40.306 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.301 7f302affd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f302c0954c0 0x7f302c096300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:58060/0 (socket says 172.21.2.112:58060) 2021-02-10T11:44:40.306 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.301 7f302affd700 1 -- 172.21.2.112:0/2409549765 learned_addr learned my addr 172.21.2.112:0/2409549765 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:44:40.306 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.301 7f302affd700 1 -- 172.21.2.112:0/2409549765 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f302c085bc0 msgr2=0x7f302c0863e0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:44:40.306 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.301 7f302affd700 1 --2- 172.21.2.112:0/2409549765 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f302c085bc0 0x7f302c0863e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:40.306 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.301 7f302affd700 1 -- 172.21.2.112:0/2409549765 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f302c0958c0 msgr2=0x7f302c096920 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:44:40.307 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.301 7f302affd700 1 --2- 172.21.2.112:0/2409549765 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f302c0958c0 0x7f302c096920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:40.307 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.301 7f302affd700 1 -- 172.21.2.112:0/2409549765 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f302c086c10 con 0x7f302c0954c0 2021-02-10T11:44:40.307 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.301 7f302affd700 1 --2- 172.21.2.112:0/2409549765 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f302c0954c0 0x7f302c096300 secure :-1 s=READY pgs=2636 cs=0 l=1 rev1=1 rx=0x7f301c006ee0 tx=0x7f301c00b9b0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:40.307 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.301 7f3023fff700 1 -- 172.21.2.112:0/2409549765 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f301c00bda0 con 0x7f302c0954c0 2021-02-10T11:44:40.307 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.301 7f3023fff700 1 -- 172.21.2.112:0/2409549765 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f301c0052e0 con 0x7f302c0954c0 2021-02-10T11:44:40.307 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.301 7f3031a58700 1 -- 172.21.2.112:0/2409549765 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f302c086c10 con 0x7f302c0954c0 2021-02-10T11:44:40.307 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.301 7f3031a58700 1 -- 172.21.2.112:0/2409549765 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f302c097230 con 0x7f302c0954c0 2021-02-10T11:44:40.307 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.301 7f3023fff700 1 -- 172.21.2.112:0/2409549765 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f301c005740 con 0x7f302c0954c0 2021-02-10T11:44:40.307 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.302 7f3023fff700 1 -- 172.21.2.112:0/2409549765 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f301c0054c0 con 0x7f302c0954c0 2021-02-10T11:44:40.307 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.302 7f3023fff700 1 --2- 172.21.2.112:0/2409549765 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f30100186d0 0x7f301001abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:40.307 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.302 7f3023fff700 1 -- 172.21.2.112:0/2409549765 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f301c032480 con 0x7f302c0954c0 2021-02-10T11:44:40.307 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.303 7f302b7fe700 1 --2- 172.21.2.112:0/2409549765 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f30100186d0 0x7f301001abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:40.307 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.303 7f302b7fe700 1 --2- 172.21.2.112:0/2409549765 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f30100186d0 0x7f301001abd0 crc :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:40.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.332 7f3031a58700 1 -- 172.21.2.112:0/2409549765 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f302c0fe1c0 con 0x7f302c0954c0 2021-02-10T11:44:40.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.335 7f3023fff700 1 -- 172.21.2.112:0/2409549765 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f301c0054f0 con 0x7f302c0954c0 2021-02-10T11:44:40.463 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.462 7f3031a58700 1 -- 172.21.2.112:0/2409549765 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f302c0fe1c0 con 0x7f302c0954c0 2021-02-10T11:44:40.464 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.462 7f3023fff700 1 -- 172.21.2.112:0/2409549765 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v147) v1 ==== 92+0+10 (secure 0 0 0) 0x7f301c0052e0 con 0x7f302c0954c0 2021-02-10T11:44:40.464 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:44:40.466 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.464 7f3031a58700 1 -- 172.21.2.112:0/2409549765 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f30100186d0 msgr2=0x7f301001abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:40.466 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.464 7f3031a58700 1 --2- 172.21.2.112:0/2409549765 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f30100186d0 0x7f301001abd0 crc :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:40.466 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.464 7f3031a58700 1 -- 172.21.2.112:0/2409549765 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f302c0954c0 msgr2=0x7f302c096300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:40.466 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.464 7f3031a58700 1 --2- 172.21.2.112:0/2409549765 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f302c0954c0 0x7f302c096300 secure :-1 s=READY pgs=2636 cs=0 l=1 rev1=1 rx=0x7f301c006ee0 tx=0x7f301c00b9b0).stop 2021-02-10T11:44:40.466 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.464 7f3031a58700 1 -- 172.21.2.112:0/2409549765 shutdown_connections 2021-02-10T11:44:40.466 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.464 7f3031a58700 1 --2- 172.21.2.112:0/2409549765 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f30100186d0 0x7f301001abd0 unknown :-1 s=CLOSED pgs=1203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:40.466 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.464 7f3031a58700 1 --2- 172.21.2.112:0/2409549765 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f302c0958c0 0x7f302c096920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:40.466 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.464 7f3031a58700 1 --2- 172.21.2.112:0/2409549765 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f302c0954c0 0x7f302c096300 unknown :-1 s=CLOSED pgs=2636 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:40.466 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.464 7f3031a58700 1 --2- 172.21.2.112:0/2409549765 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f302c085bc0 0x7f302c0863e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:40.466 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.464 7f3031a58700 1 -- 172.21.2.112:0/2409549765 >> 172.21.2.112:0/2409549765 conn(0x7f302c0bef00 msgr2=0x7f302c09d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:40.466 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.465 7f3031a58700 1 -- 172.21.2.112:0/2409549765 shutdown_connections 2021-02-10T11:44:40.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.465 7f3031a58700 1 -- 172.21.2.112:0/2409549765 wait complete. 2021-02-10T11:44:40.474 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --format=json-pretty osd lspools 2021-02-10T11:44:40.902 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.885 7f02732f8700 1 Processor -- start 2021-02-10T11:44:40.903 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.886 7f02732f8700 1 -- start start 2021-02-10T11:44:40.903 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.886 7f02732f8700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f026c0d1870 0x7f026c1b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:40.903 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.886 7f02732f8700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f026c1b0a10 0x7f026c0cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:40.903 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.886 7f02732f8700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f026c0d1c70 0x7f026c0cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:40.903 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.886 7f02732f8700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f026c0e5ed0 con 0x7f026c1b0a10 2021-02-10T11:44:40.903 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.886 7f02732f8700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f026c0f0b10 con 0x7f026c0d1870 2021-02-10T11:44:40.903 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.886 7f02732f8700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f026c0cff70 con 0x7f026c0d1c70 2021-02-10T11:44:40.903 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.886 7f0270913700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f026c1b0a10 0x7f026c0cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:40.903 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.886 7f0270913700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f026c1b0a10 0x7f026c0cf330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:58072/0 (socket says 172.21.2.112:58072) 2021-02-10T11:44:40.903 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.886 7f0270913700 1 -- 172.21.2.112:0/3784491072 learned_addr learned my addr 172.21.2.112:0/3784491072 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:44:40.903 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.886 7f0271915700 1 --2- 172.21.2.112:0/3784491072 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f026c0d1c70 0x7f026c0cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:40.904 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.886 7f0271114700 1 --2- 172.21.2.112:0/3784491072 >> v2:172.21.2.120:3300/0 conn(0x7f026c0d1870 0x7f026c1b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:40.904 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.887 7f0270913700 1 -- 172.21.2.112:0/3784491072 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f026c0d1c70 msgr2=0x7f026c0cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:40.904 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.887 7f0270913700 1 --2- 172.21.2.112:0/3784491072 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f026c0d1c70 0x7f026c0cf950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:40.904 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.887 7f0270913700 1 -- 172.21.2.112:0/3784491072 >> v2:172.21.2.120:3300/0 conn(0x7f026c0d1870 msgr2=0x7f026c1b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:40.904 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.887 7f0270913700 1 --2- 172.21.2.112:0/3784491072 >> v2:172.21.2.120:3300/0 conn(0x7f026c0d1870 0x7f026c1b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:40.904 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.887 7f0270913700 1 -- 172.21.2.112:0/3784491072 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f026c0d0150 con 0x7f026c1b0a10 2021-02-10T11:44:40.904 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.887 7f0270913700 1 --2- 172.21.2.112:0/3784491072 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f026c1b0a10 0x7f026c0cf330 secure :-1 s=READY pgs=2637 cs=0 l=1 rev1=1 rx=0x7f025c006700 tx=0x7f025c009b80).ready entity=mon.0 client_cookie=2180054db1b8d690 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:40.904 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.887 7f0263fff700 1 -- 172.21.2.112:0/3784491072 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f025c00c0c0 con 0x7f026c1b0a10 2021-02-10T11:44:40.904 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.887 7f0263fff700 1 -- 172.21.2.112:0/3784491072 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f025c00c440 con 0x7f026c1b0a10 2021-02-10T11:44:40.904 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.887 7f0263fff700 1 -- 172.21.2.112:0/3784491072 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f025c00c8a0 con 0x7f026c1b0a10 2021-02-10T11:44:40.904 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.887 7f02732f8700 1 -- 172.21.2.112:0/3784491072 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f026c1b0a10 msgr2=0x7f026c0cf330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:40.904 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.887 7f02732f8700 1 --2- 172.21.2.112:0/3784491072 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f026c1b0a10 0x7f026c0cf330 secure :-1 s=READY pgs=2637 cs=0 l=1 rev1=1 rx=0x7f025c006700 tx=0x7f025c009b80).stop 2021-02-10T11:44:40.904 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.887 7f02732f8700 1 -- 172.21.2.112:0/3784491072 shutdown_connections 2021-02-10T11:44:40.905 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.887 7f02732f8700 1 --2- 172.21.2.112:0/3784491072 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f026c0d1c70 0x7f026c0cf950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:40.905 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.887 7f02732f8700 1 --2- 172.21.2.112:0/3784491072 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f026c1b0a10 0x7f026c0cf330 unknown :-1 s=CLOSED pgs=2637 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:40.905 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.887 7f02732f8700 1 --2- 172.21.2.112:0/3784491072 >> v2:172.21.2.120:3300/0 conn(0x7f026c0d1870 0x7f026c1b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:40.905 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.887 7f02732f8700 1 -- 172.21.2.112:0/3784491072 >> 172.21.2.112:0/3784491072 conn(0x7f026c0ce450 msgr2=0x7f026c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:40.905 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.888 7f02732f8700 1 -- 172.21.2.112:0/3784491072 shutdown_connections 2021-02-10T11:44:40.905 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.888 7f02732f8700 1 -- 172.21.2.112:0/3784491072 wait complete. 2021-02-10T11:44:40.907 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.902 7f02732f8700 1 Processor -- start 2021-02-10T11:44:40.907 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.902 7f02732f8700 1 -- start start 2021-02-10T11:44:40.907 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.902 7f02732f8700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f026c195650 0x7f026c0b5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:40.907 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.902 7f02732f8700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f026c0b5b80 0x7f026c1a2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:40.907 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.902 7f02732f8700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f026c0acfa0 0x7f026c1a3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:40.907 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.902 7f02732f8700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f026c0b6010 con 0x7f026c0b5b80 2021-02-10T11:44:40.907 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.902 7f02732f8700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f026c1a3a60 con 0x7f026c195650 2021-02-10T11:44:40.907 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.902 7f02732f8700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f026c1a3c40 con 0x7f026c0acfa0 2021-02-10T11:44:40.907 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.903 7f0270913700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f026c0b5b80 0x7f026c1a2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:40.907 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.903 7f0270913700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f026c0b5b80 0x7f026c1a2e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:58078/0 (socket says 172.21.2.112:58078) 2021-02-10T11:44:40.907 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.903 7f0270913700 1 -- 172.21.2.112:0/626154807 learned_addr learned my addr 172.21.2.112:0/626154807 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:44:40.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.903 7f0271114700 1 --2- 172.21.2.112:0/626154807 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f026c195650 0x7f026c0b5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:40.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.903 7f0271915700 1 --2- 172.21.2.112:0/626154807 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f026c0acfa0 0x7f026c1a3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:40.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.903 7f0270913700 1 -- 172.21.2.112:0/626154807 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f026c0acfa0 msgr2=0x7f026c1a3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:40.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.903 7f0270913700 1 --2- 172.21.2.112:0/626154807 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f026c0acfa0 0x7f026c1a3440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:40.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.903 7f0270913700 1 -- 172.21.2.112:0/626154807 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f026c195650 msgr2=0x7f026c0b5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:40.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.903 7f0270913700 1 --2- 172.21.2.112:0/626154807 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f026c195650 0x7f026c0b5560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:40.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.903 7f0270913700 1 -- 172.21.2.112:0/626154807 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f026c0bcae0 con 0x7f026c0b5b80 2021-02-10T11:44:40.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.903 7f0270913700 1 --2- 172.21.2.112:0/626154807 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f026c0b5b80 0x7f026c1a2e20 secure :-1 s=READY pgs=2638 cs=0 l=1 rev1=1 rx=0x7f025c00b5e0 tx=0x7f025c00b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:40.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.903 7f0261ffb700 1 -- 172.21.2.112:0/626154807 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f025c00bd70 con 0x7f026c0b5b80 2021-02-10T11:44:40.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.903 7f0261ffb700 1 -- 172.21.2.112:0/626154807 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f025c002030 con 0x7f026c0b5b80 2021-02-10T11:44:40.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.903 7f02732f8700 1 -- 172.21.2.112:0/626154807 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f026c0bcae0 con 0x7f026c0b5b80 2021-02-10T11:44:40.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.903 7f0261ffb700 1 -- 172.21.2.112:0/626154807 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f025c002550 con 0x7f026c0b5b80 2021-02-10T11:44:40.909 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.903 7f02732f8700 1 -- 172.21.2.112:0/626154807 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f026c0bcae0 con 0x7f026c0b5b80 2021-02-10T11:44:40.909 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.904 7f0261ffb700 1 -- 172.21.2.112:0/626154807 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f025c000c90 con 0x7f026c0b5b80 2021-02-10T11:44:40.909 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.904 7f0261ffb700 1 --2- 172.21.2.112:0/626154807 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f024c0186c0 0x7f024c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:40.909 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.904 7f0261ffb700 1 -- 172.21.2.112:0/626154807 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f025c0324a0 con 0x7f026c0b5b80 2021-02-10T11:44:40.909 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.904 7f0271114700 1 --2- 172.21.2.112:0/626154807 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f024c0186c0 0x7f024c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:40.909 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.905 7f0271114700 1 --2- 172.21.2.112:0/626154807 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f024c0186c0 0x7f024c01abc0 crc :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:40.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.936 7f02732f8700 1 -- 172.21.2.112:0/626154807 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f026c1a3c40 con 0x7f026c0b5b80 2021-02-10T11:44:40.940 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:40.939 7f0261ffb700 1 -- 172.21.2.112:0/626154807 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f025c01f510 con 0x7f026c0b5b80 2021-02-10T11:44:41.063 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:41.062 7f02732f8700 1 -- 172.21.2.112:0/626154807 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f026c1a3c40 con 0x7f026c0b5b80 2021-02-10T11:44:41.063 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:41.062 7f0261ffb700 1 -- 172.21.2.112:0/626154807 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v147) v1 ==== 84+0+64 (secure 0 0 0) 0x7f025c00bd70 con 0x7f026c0b5b80 2021-02-10T11:44:41.064 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:44:41.064 INFO:teuthology.orchestra.run.gibba012.stdout:[ 2021-02-10T11:44:41.064 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:44:41.064 INFO:teuthology.orchestra.run.gibba012.stdout: "poolnum": 1, 2021-02-10T11:44:41.064 INFO:teuthology.orchestra.run.gibba012.stdout: "poolname": "rbd" 2021-02-10T11:44:41.064 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:44:41.064 INFO:teuthology.orchestra.run.gibba012.stdout:] 2021-02-10T11:44:41.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:41.064 7f02732f8700 1 -- 172.21.2.112:0/626154807 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f024c0186c0 msgr2=0x7f024c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:41.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:41.064 7f02732f8700 1 --2- 172.21.2.112:0/626154807 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f024c0186c0 0x7f024c01abc0 crc :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:41.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:41.064 7f02732f8700 1 -- 172.21.2.112:0/626154807 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f026c0b5b80 msgr2=0x7f026c1a2e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:41.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:41.064 7f02732f8700 1 --2- 172.21.2.112:0/626154807 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f026c0b5b80 0x7f026c1a2e20 secure :-1 s=READY pgs=2638 cs=0 l=1 rev1=1 rx=0x7f025c00b5e0 tx=0x7f025c00b980).stop 2021-02-10T11:44:41.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:41.064 7f02732f8700 1 -- 172.21.2.112:0/626154807 shutdown_connections 2021-02-10T11:44:41.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:41.064 7f02732f8700 1 --2- 172.21.2.112:0/626154807 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f024c0186c0 0x7f024c01abc0 unknown :-1 s=CLOSED pgs=1204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:41.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:41.064 7f02732f8700 1 --2- 172.21.2.112:0/626154807 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f026c0acfa0 0x7f026c1a3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:41.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:41.064 7f02732f8700 1 --2- 172.21.2.112:0/626154807 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f026c0b5b80 0x7f026c1a2e20 unknown :-1 s=CLOSED pgs=2638 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:41.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:41.064 7f02732f8700 1 --2- 172.21.2.112:0/626154807 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f026c195650 0x7f026c0b5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:41.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:41.064 7f02732f8700 1 -- 172.21.2.112:0/626154807 >> 172.21.2.112:0/626154807 conn(0x7f026c0b3040 msgr2=0x7f026c0c3250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:41.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:41.064 7f02732f8700 1 -- 172.21.2.112:0/626154807 shutdown_connections 2021-02-10T11:44:41.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:41.064 7f02732f8700 1 -- 172.21.2.112:0/626154807 wait complete. 2021-02-10T11:44:41.074 INFO:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2021-02-10T11:44:41.074 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_metadata 8 2021-02-10T11:44:41.505 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:41.488 7fd05295d700 1 Processor -- start 2021-02-10T11:44:41.505 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:41.488 7fd05295d700 1 -- start start 2021-02-10T11:44:41.505 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:41.489 7fd05295d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fd04c0d1870 0x7fd04c1b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:41.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:41.489 7fd05295d700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fd04c0cff70 0x7fd04c109130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:41.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:41.489 7fd05295d700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd04c0fe1c0 con 0x7fd04c1b0ae0 2021-02-10T11:44:41.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:41.489 7fd05295d700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd04c109ad0 con 0x7fd04c0d1870 2021-02-10T11:44:41.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:41.489 7fd05295d700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd04c109750 con 0x7fd04c0cff70 2021-02-10T11:44:41.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:41.489 7fd04bfff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fd04c0d1870 0x7fd04c1b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:41.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:41.489 7fd04bfff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fd04c0d1870 0x7fd04c1b04c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:58082/0 (socket says 172.21.2.112:58082) 2021-02-10T11:44:41.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:41.489 7fd04bfff700 1 -- 172.21.2.112:0/3426555547 learned_addr learned my addr 172.21.2.112:0/3426555547 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:44:41.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:41.489 7fd04bfff700 1 -- 172.21.2.112:0/3426555547 >> v2:172.21.2.120:3300/0 conn(0x7fd04c0cff70 msgr2=0x7fd04c109130 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:44:41.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:41.489 7fd04bfff700 1 --2- 172.21.2.112:0/3426555547 >> v2:172.21.2.120:3300/0 conn(0x7fd04c0cff70 0x7fd04c109130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:41.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:41.489 7fd04bfff700 1 -- 172.21.2.112:0/3426555547 >> v1:172.21.2.120:6789/0 conn(0x7fd04c1b0ae0 legacy=0x7fd04c108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:41.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:41.489 7fd04bfff700 1 -- 172.21.2.112:0/3426555547 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd04c088070 con 0x7fd04c0d1870 2021-02-10T11:44:41.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:41.489 7fd04bfff700 1 --2- 172.21.2.112:0/3426555547 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd04c0d1870 0x7fd04c1b04c0 secure :-1 s=READY pgs=2639 cs=0 l=1 rev1=1 rx=0x7fd03c006700 tx=0x7fd03c009b80).ready entity=mon.0 client_cookie=45c8ddd1f595f02 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:41.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:41.490 7fd04affd700 1 -- 172.21.2.112:0/3426555547 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd03c00c0c0 con 0x7fd04c0d1870 2021-02-10T11:44:41.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:41.490 7fd04affd700 1 -- 172.21.2.112:0/3426555547 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd03c00c440 con 0x7fd04c0d1870 2021-02-10T11:44:41.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:41.490 7fd04affd700 1 -- 172.21.2.112:0/3426555547 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd03c00c920 con 0x7fd04c0d1870 2021-02-10T11:44:41.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:41.490 7fd05295d700 1 -- 172.21.2.112:0/3426555547 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd04c0d1870 msgr2=0x7fd04c1b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:41.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:41.490 7fd05295d700 1 --2- 172.21.2.112:0/3426555547 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd04c0d1870 0x7fd04c1b04c0 secure :-1 s=READY pgs=2639 cs=0 l=1 rev1=1 rx=0x7fd03c006700 tx=0x7fd03c009b80).stop 2021-02-10T11:44:41.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:41.490 7fd05295d700 1 -- 172.21.2.112:0/3426555547 shutdown_connections 2021-02-10T11:44:41.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:41.490 7fd05295d700 1 --2- 172.21.2.112:0/3426555547 >> v2:172.21.2.120:3300/0 conn(0x7fd04c0cff70 0x7fd04c109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:41.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:41.490 7fd05295d700 1 --2- 172.21.2.112:0/3426555547 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd04c0d1870 0x7fd04c1b04c0 unknown :-1 s=CLOSED pgs=2639 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:41.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:41.490 7fd05295d700 1 -- 172.21.2.112:0/3426555547 >> 172.21.2.112:0/3426555547 conn(0x7fd04c0ce420 msgr2=0x7fd04c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:41.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:41.490 7fd05295d700 1 -- 172.21.2.112:0/3426555547 shutdown_connections 2021-02-10T11:44:41.508 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:41.490 7fd05295d700 1 -- 172.21.2.112:0/3426555547 wait complete. 2021-02-10T11:44:41.509 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:41.505 7fd05295d700 1 Processor -- start 2021-02-10T11:44:41.509 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:41.505 7fd05295d700 1 -- start start 2021-02-10T11:44:41.509 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:41.505 7fd05295d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd04c0c3820 0x7fd04c084cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:41.509 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:41.505 7fd05295d700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd04c0a0290 0x7fd04c0852d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:41.509 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:41.505 7fd05295d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd04c086270 0x7fd04c086670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:41.509 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:41.505 7fd05295d700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd04c0c0700 con 0x7fd04c0c3820 2021-02-10T11:44:41.509 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:41.505 7fd05295d700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd04c0858f0 con 0x7fd04c0a0290 2021-02-10T11:44:41.510 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:41.505 7fd05295d700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd04c085b10 con 0x7fd04c086270 2021-02-10T11:44:41.510 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:41.505 7fd04bfff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd04c0c3820 0x7fd04c084cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:41.510 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:41.505 7fd04bfff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd04c0c3820 0x7fd04c084cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:58088/0 (socket says 172.21.2.112:58088) 2021-02-10T11:44:41.510 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:41.505 7fd04bfff700 1 -- 172.21.2.112:0/1666519227 learned_addr learned my addr 172.21.2.112:0/1666519227 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:44:41.510 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:41.506 7fd04bfff700 1 -- 172.21.2.112:0/1666519227 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd04c086270 msgr2=0x7fd04c086670 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:44:41.510 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:41.506 7fd04bfff700 1 --2- 172.21.2.112:0/1666519227 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd04c086270 0x7fd04c086670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:41.510 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:41.506 7fd04bfff700 1 -- 172.21.2.112:0/1666519227 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd04c0a0290 msgr2=0x7fd04c0852d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:44:41.510 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:41.506 7fd04bfff700 1 --2- 172.21.2.112:0/1666519227 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd04c0a0290 0x7fd04c0852d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:41.510 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:41.506 7fd04bfff700 1 -- 172.21.2.112:0/1666519227 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd04c086c90 con 0x7fd04c0c3820 2021-02-10T11:44:41.510 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:41.506 7fd04bfff700 1 --2- 172.21.2.112:0/1666519227 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd04c0c3820 0x7fd04c084cb0 secure :-1 s=READY pgs=2640 cs=0 l=1 rev1=1 rx=0x7fd03c00b5e0 tx=0x7fd03c00b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:41.510 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:41.506 7fd048ff9700 1 -- 172.21.2.112:0/1666519227 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd03c00bd70 con 0x7fd04c0c3820 2021-02-10T11:44:41.510 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:41.506 7fd048ff9700 1 -- 172.21.2.112:0/1666519227 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd03c002030 con 0x7fd04c0c3820 2021-02-10T11:44:41.510 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:41.506 7fd048ff9700 1 -- 172.21.2.112:0/1666519227 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd03c0024d0 con 0x7fd04c0c3820 2021-02-10T11:44:41.511 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:41.506 7fd05295d700 1 -- 172.21.2.112:0/1666519227 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd04c086c90 con 0x7fd04c0c3820 2021-02-10T11:44:41.511 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:41.506 7fd05295d700 1 -- 172.21.2.112:0/1666519227 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd04c1a2f00 con 0x7fd04c0c3820 2021-02-10T11:44:41.511 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:41.507 7fd048ff9700 1 -- 172.21.2.112:0/1666519227 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd03c000c90 con 0x7fd04c0c3820 2021-02-10T11:44:41.511 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:41.507 7fd048ff9700 1 --2- 172.21.2.112:0/1666519227 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd02c0186c0 0x7fd02c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:41.511 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:41.507 7fd048ff9700 1 -- 172.21.2.112:0/1666519227 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 4753+0+0 (secure 0 0 0) 0x7fd03c032250 con 0x7fd04c0c3820 2021-02-10T11:44:41.511 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:41.508 7fd04b7fe700 1 --2- 172.21.2.112:0/1666519227 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd02c0186c0 0x7fd02c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:41.511 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:41.509 7fd04b7fe700 1 --2- 172.21.2.112:0/1666519227 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd02c0186c0 0x7fd02c01abc0 crc :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:41.540 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:41.539 7fd05295d700 1 -- 172.21.2.112:0/1666519227 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd04c082a70 con 0x7fd04c0c3820 2021-02-10T11:44:41.542 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:41.541 7fd048ff9700 1 -- 172.21.2.112:0/1666519227 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd03c030db0 con 0x7fd04c0c3820 2021-02-10T11:44:41.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:41.669 7fd05295d700 1 -- 172.21.2.112:0/1666519227 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"} v 0) v1 -- 0x7fd04c082a70 con 0x7fd04c0c3820 2021-02-10T11:44:42.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:42.088 7fd048ff9700 1 -- 172.21.2.112:0/1666519227 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"}]=0 pool 'cephfs_metadata' created v148) v1 ==== 133+0+0 (secure 0 0 0) 0x7fd03c000c90 con 0x7fd04c0c3820 2021-02-10T11:44:42.136 INFO:teuthology.orchestra.run.gibba012.stderr:pool 'cephfs_metadata' created 2021-02-10T11:44:42.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:42.136 7fd05295d700 1 -- 172.21.2.112:0/1666519227 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd02c0186c0 msgr2=0x7fd02c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:42.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:42.136 7fd05295d700 1 --2- 172.21.2.112:0/1666519227 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd02c0186c0 0x7fd02c01abc0 crc :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:42.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:42.136 7fd05295d700 1 -- 172.21.2.112:0/1666519227 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd04c0c3820 msgr2=0x7fd04c084cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:42.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:42.136 7fd05295d700 1 --2- 172.21.2.112:0/1666519227 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd04c0c3820 0x7fd04c084cb0 secure :-1 s=READY pgs=2640 cs=0 l=1 rev1=1 rx=0x7fd03c00b5e0 tx=0x7fd03c00b980).stop 2021-02-10T11:44:42.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:42.137 7fd05295d700 1 -- 172.21.2.112:0/1666519227 shutdown_connections 2021-02-10T11:44:42.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:42.137 7fd05295d700 1 --2- 172.21.2.112:0/1666519227 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd02c0186c0 0x7fd02c01abc0 unknown :-1 s=CLOSED pgs=1205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:42.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:42.137 7fd05295d700 1 --2- 172.21.2.112:0/1666519227 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd04c0a0290 0x7fd04c0852d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:42.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:42.137 7fd05295d700 1 --2- 172.21.2.112:0/1666519227 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd04c086270 0x7fd04c086670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:42.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:42.137 7fd05295d700 1 --2- 172.21.2.112:0/1666519227 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd04c0c3820 0x7fd04c084cb0 unknown :-1 s=CLOSED pgs=2640 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:42.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:42.137 7fd05295d700 1 -- 172.21.2.112:0/1666519227 >> 172.21.2.112:0/1666519227 conn(0x7fd04c0872c0 msgr2=0x7fd04c095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:42.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:42.137 7fd05295d700 1 -- 172.21.2.112:0/1666519227 shutdown_connections 2021-02-10T11:44:42.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:42.137 7fd05295d700 1 -- 172.21.2.112:0/1666519227 wait complete. 2021-02-10T11:44:42.151 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_data 8 2021-02-10T11:44:42.582 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:42.563 7f4f5fd87700 1 Processor -- start 2021-02-10T11:44:42.582 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:42.564 7f4f5fd87700 1 -- start start 2021-02-10T11:44:42.582 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:42.564 7f4f5fd87700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f4f5810aff0 0x7f4f580cf750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:42.582 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:42.564 7f4f5fd87700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f580fe1c0 con 0x7f4f580d1870 2021-02-10T11:44:42.583 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:42.564 7f4f5fd87700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f581b0cb0 con 0x7f4f580d1c70 2021-02-10T11:44:42.583 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:42.564 7f4f5fd87700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f58109a60 con 0x7f4f5810aff0 2021-02-10T11:44:42.583 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:42.564 7f4f5dba3700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f4f580d1870 0x7f4f581b04c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:57556/0 (socket says 172.21.2.112:57556) 2021-02-10T11:44:42.583 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:42.564 7f4f5dba3700 1 -- 172.21.2.112:0/1965747064 learned_addr learned my addr 172.21.2.112:0/1965747064 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:44:42.583 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:42.565 7f4f5cba1700 1 -- 172.21.2.112:0/1965747064 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 993872105 0 0) 0x7f4f4c0010b0 con 0x7f4f580d1870 2021-02-10T11:44:42.583 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:42.565 7f4f5cba1700 1 -- 172.21.2.112:0/1965747064 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f44002fe0 con 0x7f4f580d1870 2021-02-10T11:44:42.583 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:42.565 7f4f5e3a4700 1 --2- 172.21.2.112:0/1965747064 >> v2:172.21.2.120:3300/0 conn(0x7f4f5810aff0 0x7f4f580cf750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:42.583 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:42.565 7f4f5cba1700 1 -- 172.21.2.112:0/1965747064 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3179608357 0 0) 0x7f4f4c0010b0 con 0x7f4f580d1870 2021-02-10T11:44:42.583 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:42.565 7f4f5cba1700 1 -- 172.21.2.112:0/1965747064 >> v2:172.21.2.120:3300/0 conn(0x7f4f5810aff0 msgr2=0x7f4f580cf750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:42.583 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:42.565 7f4f5cba1700 1 --2- 172.21.2.112:0/1965747064 >> v2:172.21.2.120:3300/0 conn(0x7f4f5810aff0 0x7f4f580cf750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:42.583 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:42.565 7f4f5cba1700 1 -- 172.21.2.112:0/1965747064 >> v1:172.21.2.120:6789/0 conn(0x7f4f580d1c70 legacy=0x7f4f580cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:42.583 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:42.565 7f4f5cba1700 1 -- 172.21.2.112:0/1965747064 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f5810b3f0 con 0x7f4f580d1870 2021-02-10T11:44:42.583 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:42.565 7f4f5cba1700 1 -- 172.21.2.112:0/1965747064 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f4f4c0015b0 con 0x7f4f580d1870 2021-02-10T11:44:42.584 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:42.565 7f4f5cba1700 1 -- 172.21.2.112:0/1965747064 >> v1:172.21.2.112:6789/0 conn(0x7f4f580d1870 legacy=0x7f4f581b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:42.584 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:42.565 7f4f5cba1700 1 --2- 172.21.2.112:0/1965747064 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4f44003a70 0x7f4f44005f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:42.584 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:42.565 7f4f5cba1700 1 --2- 172.21.2.112:0/1965747064 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4f440067a0 0x7f4f44008ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:42.584 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:42.565 7f4f5cba1700 1 --2- 172.21.2.112:0/1965747064 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4f440095a0 0x7f4f4400bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:42.584 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:42.565 7f4f5cba1700 1 -- 172.21.2.112:0/1965747064 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f4400c310 con 0x7f4f44003a70 2021-02-10T11:44:42.584 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:42.565 7f4f5cba1700 1 -- 172.21.2.112:0/1965747064 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f4400c530 con 0x7f4f440095a0 2021-02-10T11:44:42.584 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:42.565 7f4f5cba1700 1 -- 172.21.2.112:0/1965747064 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f4400c750 con 0x7f4f440067a0 2021-02-10T11:44:42.584 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:42.565 7f4f5dba3700 1 --2- 172.21.2.112:0/1965747064 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4f44003a70 0x7f4f44005f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:42.584 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:42.565 7f4f5d3a2700 1 --2- 172.21.2.112:0/1965747064 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4f440067a0 0x7f4f44008ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:42.584 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:42.565 7f4f5dba3700 1 -- 172.21.2.112:0/1965747064 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4f440067a0 msgr2=0x7f4f44008ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:42.584 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:42.565 7f4f5dba3700 1 --2- 172.21.2.112:0/1965747064 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4f440067a0 0x7f4f44008ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:42.584 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:42.565 7f4f5dba3700 1 -- 172.21.2.112:0/1965747064 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4f440095a0 msgr2=0x7f4f4400bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:42.584 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:42.565 7f4f5dba3700 1 --2- 172.21.2.112:0/1965747064 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4f440095a0 0x7f4f4400bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:42.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:42.565 7f4f5dba3700 1 -- 172.21.2.112:0/1965747064 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f4400c9f0 con 0x7f4f44003a70 2021-02-10T11:44:42.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:42.565 7f4f5d3a2700 1 --2- 172.21.2.112:0/1965747064 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4f440067a0 0x7f4f44008ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:44:42.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:42.565 7f4f5e3a4700 1 -- 172.21.2.112:0/1965747064 reap_dead start 2021-02-10T11:44:42.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:42.566 7f4f5dba3700 1 --2- 172.21.2.112:0/1965747064 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4f44003a70 0x7f4f44005f00 secure :-1 s=READY pgs=2642 cs=0 l=1 rev1=1 rx=0x7f4f4c000b40 tx=0x7f4f4c04bd10).ready entity=mon.0 client_cookie=21f8af8913b67ebf server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:42.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:42.566 7f4f5cba1700 1 -- 172.21.2.112:0/1965747064 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4f4c004410 con 0x7f4f44003a70 2021-02-10T11:44:42.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:42.566 7f4f5cba1700 1 -- 172.21.2.112:0/1965747064 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f4c004690 con 0x7f4f44003a70 2021-02-10T11:44:42.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:42.566 7f4f5cba1700 1 -- 172.21.2.112:0/1965747064 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4f4c004bb0 con 0x7f4f44003a70 2021-02-10T11:44:42.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:42.566 7f4f5fd87700 1 -- 172.21.2.112:0/1965747064 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4f44003a70 msgr2=0x7f4f44005f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:42.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:42.566 7f4f5fd87700 1 --2- 172.21.2.112:0/1965747064 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4f44003a70 0x7f4f44005f00 secure :-1 s=READY pgs=2642 cs=0 l=1 rev1=1 rx=0x7f4f4c000b40 tx=0x7f4f4c04bd10).stop 2021-02-10T11:44:42.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:42.566 7f4f5fd87700 1 -- 172.21.2.112:0/1965747064 shutdown_connections 2021-02-10T11:44:42.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:42.566 7f4f5fd87700 1 --2- 172.21.2.112:0/1965747064 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4f44003a70 0x7f4f44005f00 unknown :-1 s=CLOSED pgs=2642 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:42.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:42.566 7f4f5fd87700 1 -- 172.21.2.112:0/1965747064 >> 172.21.2.112:0/1965747064 conn(0x7f4f580ce420 msgr2=0x7f4f580d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:42.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:42.567 7f4f5fd87700 1 -- 172.21.2.112:0/1965747064 shutdown_connections 2021-02-10T11:44:42.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:42.567 7f4f5fd87700 1 -- 172.21.2.112:0/1965747064 wait complete. 2021-02-10T11:44:42.586 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:42.581 7f4f5fd87700 1 Processor -- start 2021-02-10T11:44:42.586 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:42.581 7f4f5fd87700 1 -- start start 2021-02-10T11:44:42.587 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:42.582 7f4f5fd87700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4f5809bcc0 0x7f4f5809c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:42.587 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:42.582 7f4f5fd87700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4f580a0820 0x7f4f580a0c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:42.587 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:42.582 7f4f5fd87700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4f580a1240 0x7f4f580858d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:42.587 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:42.582 7f4f5fd87700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f580a66b0 con 0x7f4f580a0820 2021-02-10T11:44:42.587 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:42.582 7f4f5fd87700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f581a49b0 con 0x7f4f5809bcc0 2021-02-10T11:44:42.587 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:42.582 7f4f5fd87700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f581a4b90 con 0x7f4f580a1240 2021-02-10T11:44:42.587 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:42.582 7f4f5d3a2700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4f580a0820 0x7f4f580a0c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:42.587 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:42.582 7f4f5d3a2700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4f580a0820 0x7f4f580a0c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:58110/0 (socket says 172.21.2.112:58110) 2021-02-10T11:44:42.587 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:42.582 7f4f5d3a2700 1 -- 172.21.2.112:0/2392358363 learned_addr learned my addr 172.21.2.112:0/2392358363 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:44:42.588 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:42.582 7f4f5d3a2700 1 -- 172.21.2.112:0/2392358363 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4f580a1240 msgr2=0x7f4f580858d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:42.588 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:42.582 7f4f5d3a2700 1 --2- 172.21.2.112:0/2392358363 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4f580a1240 0x7f4f580858d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:42.588 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:42.582 7f4f5d3a2700 1 -- 172.21.2.112:0/2392358363 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4f5809bcc0 msgr2=0x7f4f5809c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:42.588 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:42.582 7f4f5d3a2700 1 --2- 172.21.2.112:0/2392358363 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4f5809bcc0 0x7f4f5809c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:42.588 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:42.582 7f4f5d3a2700 1 -- 172.21.2.112:0/2392358363 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f580a1640 con 0x7f4f580a0820 2021-02-10T11:44:42.588 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:42.582 7f4f5d3a2700 1 --2- 172.21.2.112:0/2392358363 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4f580a0820 0x7f4f580a0c20 secure :-1 s=READY pgs=2643 cs=0 l=1 rev1=1 rx=0x7f4f54006f50 tx=0x7f4f5400b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:42.588 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:42.583 7f4f4a7fc700 1 -- 172.21.2.112:0/2392358363 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4f54006860 con 0x7f4f580a0820 2021-02-10T11:44:42.588 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:42.583 7f4f4a7fc700 1 -- 172.21.2.112:0/2392358363 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f5400bd80 con 0x7f4f580a0820 2021-02-10T11:44:42.588 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:42.583 7f4f4a7fc700 1 -- 172.21.2.112:0/2392358363 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4f54005330 con 0x7f4f580a0820 2021-02-10T11:44:42.588 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:42.583 7f4f5fd87700 1 -- 172.21.2.112:0/2392358363 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f580a1640 con 0x7f4f580a0820 2021-02-10T11:44:42.588 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:42.583 7f4f5fd87700 1 -- 172.21.2.112:0/2392358363 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f580a1640 con 0x7f4f580a0820 2021-02-10T11:44:42.588 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:42.583 7f4f4a7fc700 1 -- 172.21.2.112:0/2392358363 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4f5400a000 con 0x7f4f580a0820 2021-02-10T11:44:42.588 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:42.583 7f4f4a7fc700 1 --2- 172.21.2.112:0/2392358363 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4f400186c0 0x7f4f4001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:42.589 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:42.583 7f4f4a7fc700 1 -- 172.21.2.112:0/2392358363 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5107+0+0 (secure 0 0 0) 0x7f4f54032380 con 0x7f4f580a0820 2021-02-10T11:44:42.589 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:42.584 7f4f5dba3700 1 --2- 172.21.2.112:0/2392358363 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4f400186c0 0x7f4f4001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:42.589 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:42.584 7f4f5dba3700 1 --2- 172.21.2.112:0/2392358363 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4f400186c0 0x7f4f4001abc0 crc :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:42.615 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:42.614 7f4f5fd87700 1 -- 172.21.2.112:0/2392358363 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f580fe1c0 con 0x7f4f580a0820 2021-02-10T11:44:42.617 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:42.617 7f4f4a7fc700 1 -- 172.21.2.112:0/2392358363 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4f54030d00 con 0x7f4f580a0820 2021-02-10T11:44:42.747 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:42.745 7f4f5fd87700 1 -- 172.21.2.112:0/2392358363 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"} v 0) v1 -- 0x7f4f58086d30 con 0x7f4f580a0820 2021-02-10T11:44:43.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:43.091 7f4f4a7fc700 1 -- 172.21.2.112:0/2392358363 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"}]=0 pool 'cephfs_data' created v149) v1 ==== 125+0+0 (secure 0 0 0) 0x7f4f5400a000 con 0x7f4f580a0820 2021-02-10T11:44:43.113 INFO:teuthology.orchestra.run.gibba012.stderr:pool 'cephfs_data' created 2021-02-10T11:44:43.114 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:43.113 7f4f5fd87700 1 -- 172.21.2.112:0/2392358363 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4f400186c0 msgr2=0x7f4f4001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:43.115 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:43.113 7f4f5fd87700 1 --2- 172.21.2.112:0/2392358363 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4f400186c0 0x7f4f4001abc0 crc :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:43.115 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:43.113 7f4f5fd87700 1 -- 172.21.2.112:0/2392358363 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4f580a0820 msgr2=0x7f4f580a0c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:43.115 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:43.113 7f4f5fd87700 1 --2- 172.21.2.112:0/2392358363 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4f580a0820 0x7f4f580a0c20 secure :-1 s=READY pgs=2643 cs=0 l=1 rev1=1 rx=0x7f4f54006f50 tx=0x7f4f5400b990).stop 2021-02-10T11:44:43.115 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:43.113 7f4f5fd87700 1 -- 172.21.2.112:0/2392358363 shutdown_connections 2021-02-10T11:44:43.115 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:43.113 7f4f5fd87700 1 --2- 172.21.2.112:0/2392358363 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4f400186c0 0x7f4f4001abc0 unknown :-1 s=CLOSED pgs=1206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:43.115 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:43.113 7f4f5fd87700 1 --2- 172.21.2.112:0/2392358363 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4f580a1240 0x7f4f580858d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:43.115 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:43.113 7f4f5fd87700 1 --2- 172.21.2.112:0/2392358363 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4f580a0820 0x7f4f580a0c20 unknown :-1 s=CLOSED pgs=2643 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:43.115 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:43.113 7f4f5fd87700 1 --2- 172.21.2.112:0/2392358363 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4f5809bcc0 0x7f4f5809c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:43.115 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:43.113 7f4f5fd87700 1 -- 172.21.2.112:0/2392358363 >> 172.21.2.112:0/2392358363 conn(0x7f4f580bef00 msgr2=0x7f4f5809d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:43.115 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:43.113 7f4f5fd87700 1 -- 172.21.2.112:0/2392358363 shutdown_connections 2021-02-10T11:44:43.115 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:43.114 7f4f5fd87700 1 -- 172.21.2.112:0/2392358363 wait complete. 2021-02-10T11:44:43.123 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2021-02-10T11:44:43.555 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:43.538 7f290586e700 1 Processor -- start 2021-02-10T11:44:43.555 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:43.538 7f290586e700 1 -- start start 2021-02-10T11:44:43.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:43.538 7f290586e700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f29000d1880 0x7f29001b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:43.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:43.538 7f290586e700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f29001b0af0 0x7f2900108950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:43.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:43.538 7f290586e700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29000fe1d0 con 0x7f29000d1c80 2021-02-10T11:44:43.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:43.538 7f290586e700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2900109a70 con 0x7f29000d1880 2021-02-10T11:44:43.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:43.538 7f290586e700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2900109760 con 0x7f29001b0af0 2021-02-10T11:44:43.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:43.539 7f28ff574700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f29000d1880 0x7f29001b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:43.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:43.539 7f28ff574700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f29000d1880 0x7f29001b04d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:58116/0 (socket says 172.21.2.112:58116) 2021-02-10T11:44:43.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:43.539 7f28ff574700 1 -- 172.21.2.112:0/3949686498 learned_addr learned my addr 172.21.2.112:0/3949686498 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:44:43.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:43.539 7f28ffd75700 1 --1- 172.21.2.112:0/3949686498 >> v1:172.21.2.112:6789/0 conn(0x7f29000d1c80 0x7f2900108f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:57578/0 (socket says 172.21.2.112:57578) 2021-02-10T11:44:43.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:43.539 7f28ff574700 1 -- 172.21.2.112:0/3949686498 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f29001b0af0 msgr2=0x7f2900108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:43.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:43.539 7f28ff574700 1 --2- 172.21.2.112:0/3949686498 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f29001b0af0 0x7f2900108950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:43.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:43.539 7f28ff574700 1 -- 172.21.2.112:0/3949686498 >> v1:172.21.2.112:6789/0 conn(0x7f29000d1c80 legacy=0x7f2900108f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:43.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:43.539 7f28ff574700 1 -- 172.21.2.112:0/3949686498 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2900088070 con 0x7f29000d1880 2021-02-10T11:44:43.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:43.539 7f28ff574700 1 --2- 172.21.2.112:0/3949686498 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f29000d1880 0x7f29001b04d0 secure :-1 s=READY pgs=2645 cs=0 l=1 rev1=1 rx=0x7f28f0006700 tx=0x7f28f0009b80).ready entity=mon.0 client_cookie=1eca4582680a4075 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:43.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:43.539 7f28fe572700 1 -- 172.21.2.112:0/3949686498 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f28f000c0c0 con 0x7f29000d1880 2021-02-10T11:44:43.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:43.539 7f28fe572700 1 -- 172.21.2.112:0/3949686498 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28f000c440 con 0x7f29000d1880 2021-02-10T11:44:43.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:43.539 7f28fe572700 1 -- 172.21.2.112:0/3949686498 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f28f000c960 con 0x7f29000d1880 2021-02-10T11:44:43.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:43.539 7f290586e700 1 -- 172.21.2.112:0/3949686498 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f29000d1880 msgr2=0x7f29001b04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:43.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:43.539 7f290586e700 1 --2- 172.21.2.112:0/3949686498 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f29000d1880 0x7f29001b04d0 secure :-1 s=READY pgs=2645 cs=0 l=1 rev1=1 rx=0x7f28f0006700 tx=0x7f28f0009b80).stop 2021-02-10T11:44:43.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:43.540 7f290586e700 1 -- 172.21.2.112:0/3949686498 shutdown_connections 2021-02-10T11:44:43.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:43.540 7f290586e700 1 --2- 172.21.2.112:0/3949686498 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f29001b0af0 0x7f2900108950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:43.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:43.540 7f290586e700 1 --2- 172.21.2.112:0/3949686498 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f29000d1880 0x7f29001b04d0 unknown :-1 s=CLOSED pgs=2645 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:43.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:43.540 7f290586e700 1 -- 172.21.2.112:0/3949686498 >> 172.21.2.112:0/3949686498 conn(0x7f29000ce430 msgr2=0x7f29000d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:43.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:43.540 7f290586e700 1 -- 172.21.2.112:0/3949686498 shutdown_connections 2021-02-10T11:44:43.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:43.540 7f290586e700 1 -- 172.21.2.112:0/3949686498 wait complete. 2021-02-10T11:44:43.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:43.555 7f290586e700 1 Processor -- start 2021-02-10T11:44:43.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:43.555 7f290586e700 1 -- start start 2021-02-10T11:44:43.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:43.555 7f290586e700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2900190df0 0x7f29001b7ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:43.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:43.555 7f290586e700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f290019ba20 0x7f29001b87c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:43.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:43.555 7f290586e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f290019be20 0x7f290018f830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:43.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:43.555 7f290586e700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29001b8240 con 0x7f290019be20 2021-02-10T11:44:43.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:43.555 7f290586e700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29001911f0 con 0x7f290019ba20 2021-02-10T11:44:43.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:43.555 7f290586e700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29001908a0 con 0x7f2900190df0 2021-02-10T11:44:43.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:43.555 7f28ffd75700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f290019be20 0x7f290018f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:43.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:43.555 7f28ffd75700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f290019be20 0x7f290018f830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:58122/0 (socket says 172.21.2.112:58122) 2021-02-10T11:44:43.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:43.555 7f28ffd75700 1 -- 172.21.2.112:0/1195252101 learned_addr learned my addr 172.21.2.112:0/1195252101 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:44:43.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:43.556 7f28ffd75700 1 -- 172.21.2.112:0/1195252101 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2900190df0 msgr2=0x7f29001b7ac0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:44:43.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:43.556 7f28ffd75700 1 --2- 172.21.2.112:0/1195252101 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2900190df0 0x7f29001b7ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:43.563 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:43.556 7f28ffd75700 1 -- 172.21.2.112:0/1195252101 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f290019ba20 msgr2=0x7f29001b87c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:43.563 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:43.556 7f28ffd75700 1 --2- 172.21.2.112:0/1195252101 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f290019ba20 0x7f29001b87c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:43.563 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:43.556 7f28ffd75700 1 -- 172.21.2.112:0/1195252101 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29001b8de0 con 0x7f290019be20 2021-02-10T11:44:43.563 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:43.556 7f28ffd75700 1 --2- 172.21.2.112:0/1195252101 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f290019be20 0x7f290018f830 secure :-1 s=READY pgs=2646 cs=0 l=1 rev1=1 rx=0x7f28f8009960 tx=0x7f28f800bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:43.563 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:43.556 7f28e7fff700 1 -- 172.21.2.112:0/1195252101 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f28f8006030 con 0x7f290019be20 2021-02-10T11:44:43.563 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:43.556 7f28e7fff700 1 -- 172.21.2.112:0/1195252101 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28f8006270 con 0x7f290019be20 2021-02-10T11:44:43.563 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:43.556 7f28e7fff700 1 -- 172.21.2.112:0/1195252101 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f28f8004720 con 0x7f290019be20 2021-02-10T11:44:43.564 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:43.556 7f290586e700 1 -- 172.21.2.112:0/1195252101 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29001b8de0 con 0x7f290019be20 2021-02-10T11:44:43.564 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:43.556 7f290586e700 1 -- 172.21.2.112:0/1195252101 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f290018fe50 con 0x7f290019be20 2021-02-10T11:44:43.564 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:43.557 7f28e7fff700 1 -- 172.21.2.112:0/1195252101 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f28f8006450 con 0x7f290019be20 2021-02-10T11:44:43.564 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:43.557 7f28e7fff700 1 --2- 172.21.2.112:0/1195252101 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f28e00186c0 0x7f28e001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:43.564 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:43.557 7f28e7fff700 1 -- 172.21.2.112:0/1195252101 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5457+0+0 (secure 0 0 0) 0x7f28f80328e0 con 0x7f290019be20 2021-02-10T11:44:43.564 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:43.557 7f28ff574700 1 --2- 172.21.2.112:0/1195252101 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f28e00186c0 0x7f28e001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:43.564 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:43.557 7f28ff574700 1 --2- 172.21.2.112:0/1195252101 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f28e00186c0 0x7f28e001abc0 crc :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:43.589 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:43.588 7f290586e700 1 -- 172.21.2.112:0/1195252101 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2900082aa0 con 0x7f290019be20 2021-02-10T11:44:43.592 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:43.591 7f28e7fff700 1 -- 172.21.2.112:0/1195252101 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f28f8000ec0 con 0x7f290019be20 2021-02-10T11:44:43.721 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:43.720 7f290586e700 1 -- 172.21.2.112:0/1195252101 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"} v 0) v1 -- 0x7f2900082aa0 con 0x7f290019be20 2021-02-10T11:44:43.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:43.723 7f28e7fff700 1 -- 172.21.2.112:0/1195252101 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"}]=0 new fs with metadata pool 32 and data pool 33 v127) v1 ==== 189+0+0 (secure 0 0 0) 0x7f28f8006450 con 0x7f290019be20 2021-02-10T11:44:43.728 INFO:teuthology.orchestra.run.gibba012.stderr:new fs with metadata pool 32 and data pool 33 2021-02-10T11:44:43.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:43.729 7f290586e700 1 -- 172.21.2.112:0/1195252101 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f28e00186c0 msgr2=0x7f28e001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:43.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:43.729 7f290586e700 1 --2- 172.21.2.112:0/1195252101 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f28e00186c0 0x7f28e001abc0 crc :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:43.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:43.729 7f290586e700 1 -- 172.21.2.112:0/1195252101 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f290019be20 msgr2=0x7f290018f830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:43.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:43.729 7f290586e700 1 --2- 172.21.2.112:0/1195252101 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f290019be20 0x7f290018f830 secure :-1 s=READY pgs=2646 cs=0 l=1 rev1=1 rx=0x7f28f8009960 tx=0x7f28f800bcf0).stop 2021-02-10T11:44:43.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:43.729 7f290586e700 1 -- 172.21.2.112:0/1195252101 shutdown_connections 2021-02-10T11:44:43.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:43.729 7f290586e700 1 --2- 172.21.2.112:0/1195252101 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f28e00186c0 0x7f28e001abc0 unknown :-1 s=CLOSED pgs=1209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:43.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:43.729 7f290586e700 1 --2- 172.21.2.112:0/1195252101 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f290019ba20 0x7f29001b87c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:43.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:43.729 7f290586e700 1 --2- 172.21.2.112:0/1195252101 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f290019be20 0x7f290018f830 unknown :-1 s=CLOSED pgs=2646 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:43.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:43.729 7f290586e700 1 --2- 172.21.2.112:0/1195252101 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2900190df0 0x7f29001b7ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:43.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:43.729 7f290586e700 1 -- 172.21.2.112:0/1195252101 >> 172.21.2.112:0/1195252101 conn(0x7f290019f940 msgr2=0x7f290018c290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:43.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:43.730 7f290586e700 1 -- 172.21.2.112:0/1195252101 shutdown_connections 2021-02-10T11:44:43.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:43.730 7f290586e700 1 -- 172.21.2.112:0/1195252101 wait complete. 2021-02-10T11:44:43.740 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:44:44.169 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.150 7fd36fdfb700 1 Processor -- start 2021-02-10T11:44:44.170 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.150 7fd36fdfb700 1 -- start start 2021-02-10T11:44:44.170 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.151 7fd36fdfb700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fd3680d0340 0x7fd3680ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:44.170 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.151 7fd36fdfb700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3680f0b10 con 0x7fd3680d1870 2021-02-10T11:44:44.170 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.151 7fd36fdfb700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3681b0be0 con 0x7fd3680d1c70 2021-02-10T11:44:44.170 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.151 7fd36fdfb700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3680e5ed0 con 0x7fd3680d0340 2021-02-10T11:44:44.170 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.151 7fd36d416700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fd3680d0340 0x7fd3680ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:44.170 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.151 7fd36d416700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fd3680d0340 0x7fd3680ef640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:58130/0 (socket says 172.21.2.112:58130) 2021-02-10T11:44:44.170 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.151 7fd36dc17700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fd3680d1870 0x7fd3681b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:57592/0 (socket says 172.21.2.112:57592) 2021-02-10T11:44:44.170 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.151 7fd36dc17700 1 -- 172.21.2.112:0/1768130831 learned_addr learned my addr 172.21.2.112:0/1768130831 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:44:44.170 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.151 7fd36d416700 1 -- 172.21.2.112:0/1768130831 >> v1:172.21.2.120:6789/0 conn(0x7fd3680d1c70 legacy=0x7fd3680efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:44.170 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.151 7fd36d416700 1 -- 172.21.2.112:0/1768130831 >> v1:172.21.2.112:6789/0 conn(0x7fd3680d1870 legacy=0x7fd3681b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:44.170 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.152 7fd36d416700 1 -- 172.21.2.112:0/1768130831 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3680f0450 con 0x7fd3680d0340 2021-02-10T11:44:44.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.151 7fd36cc15700 1 -- 172.21.2.112:0/1768130831 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1939472648 0 0) 0x7fd364001180 con 0x7fd3680d1870 2021-02-10T11:44:44.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.152 7fd36d416700 1 --2- 172.21.2.112:0/1768130831 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd3680d0340 0x7fd3680ef640 secure :-1 s=READY pgs=2648 cs=0 l=1 rev1=1 rx=0x7fd358006650 tx=0x7fd358009a80).ready entity=mon.0 client_cookie=9feb471b1f681008 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:44.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.152 7fd36cc15700 1 -- 172.21.2.112:0/1768130831 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd35800c0c0 con 0x7fd3680d0340 2021-02-10T11:44:44.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.152 7fd36cc15700 1 -- 172.21.2.112:0/1768130831 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd35800c340 con 0x7fd3680d0340 2021-02-10T11:44:44.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.152 7fd36cc15700 1 -- 172.21.2.112:0/1768130831 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd35800c860 con 0x7fd3680d0340 2021-02-10T11:44:44.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.152 7fd36fdfb700 1 -- 172.21.2.112:0/1768130831 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd3680d0340 msgr2=0x7fd3680ef640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:44.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.152 7fd36fdfb700 1 --2- 172.21.2.112:0/1768130831 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd3680d0340 0x7fd3680ef640 secure :-1 s=READY pgs=2648 cs=0 l=1 rev1=1 rx=0x7fd358006650 tx=0x7fd358009a80).stop 2021-02-10T11:44:44.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.153 7fd36fdfb700 1 -- 172.21.2.112:0/1768130831 shutdown_connections 2021-02-10T11:44:44.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.153 7fd36fdfb700 1 --2- 172.21.2.112:0/1768130831 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd3680d0340 0x7fd3680ef640 unknown :-1 s=CLOSED pgs=2648 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:44.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.153 7fd36fdfb700 1 -- 172.21.2.112:0/1768130831 >> 172.21.2.112:0/1768130831 conn(0x7fd3680ce450 msgr2=0x7fd3680d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:44.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.153 7fd36fdfb700 1 -- 172.21.2.112:0/1768130831 shutdown_connections 2021-02-10T11:44:44.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.153 7fd36fdfb700 1 -- 172.21.2.112:0/1768130831 wait complete. 2021-02-10T11:44:44.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.167 7fd36fdfb700 1 Processor -- start 2021-02-10T11:44:44.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.167 7fd36fdfb700 1 -- start start 2021-02-10T11:44:44.172 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.168 7fd36fdfb700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd36818f0a0 0x7fd368196ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:44.172 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.168 7fd36fdfb700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd3681970d0 0x7fd3681974d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:44.172 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.168 7fd36fdfb700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd368094650 0x7fd368094a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:44.172 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.168 7fd36fdfb700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd368095bd0 con 0x7fd36818f0a0 2021-02-10T11:44:44.172 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.168 7fd36fdfb700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3680ad7c0 con 0x7fd368094650 2021-02-10T11:44:44.172 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.168 7fd36fdfb700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd36818ec70 con 0x7fd3681970d0 2021-02-10T11:44:44.172 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.168 7fd36dc17700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd36818f0a0 0x7fd368196ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:44.172 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.168 7fd36dc17700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd36818f0a0 0x7fd368196ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:58136/0 (socket says 172.21.2.112:58136) 2021-02-10T11:44:44.172 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.168 7fd36dc17700 1 -- 172.21.2.112:0/4238941341 learned_addr learned my addr 172.21.2.112:0/4238941341 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:44:44.173 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.168 7fd36dc17700 1 -- 172.21.2.112:0/4238941341 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd3681970d0 msgr2=0x7fd3681974d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:44:44.173 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.168 7fd36dc17700 1 --2- 172.21.2.112:0/4238941341 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd3681970d0 0x7fd3681974d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:44.173 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.168 7fd36dc17700 1 -- 172.21.2.112:0/4238941341 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd368094650 msgr2=0x7fd368094a50 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:44:44.174 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.168 7fd36dc17700 1 --2- 172.21.2.112:0/4238941341 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd368094650 0x7fd368094a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:44.174 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.168 7fd36dc17700 1 -- 172.21.2.112:0/4238941341 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd368095070 con 0x7fd36818f0a0 2021-02-10T11:44:44.174 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.169 7fd36dc17700 1 --2- 172.21.2.112:0/4238941341 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd36818f0a0 0x7fd368196ab0 secure :-1 s=READY pgs=2649 cs=0 l=1 rev1=1 rx=0x7fd364001ea0 tx=0x7fd36400a880).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:44.174 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.169 7fd35e7fc700 1 -- 172.21.2.112:0/4238941341 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd3640070d0 con 0x7fd36818f0a0 2021-02-10T11:44:44.174 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.169 7fd35e7fc700 1 -- 172.21.2.112:0/4238941341 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd36400ac70 con 0x7fd36818f0a0 2021-02-10T11:44:44.174 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.169 7fd35e7fc700 1 -- 172.21.2.112:0/4238941341 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd364004620 con 0x7fd36818f0a0 2021-02-10T11:44:44.174 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.169 7fd36fdfb700 1 -- 172.21.2.112:0/4238941341 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd368095070 con 0x7fd36818f0a0 2021-02-10T11:44:44.174 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.169 7fd36fdfb700 1 -- 172.21.2.112:0/4238941341 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd368095070 con 0x7fd36818f0a0 2021-02-10T11:44:44.174 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.169 7fd35e7fc700 1 -- 172.21.2.112:0/4238941341 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd3640043a0 con 0x7fd36818f0a0 2021-02-10T11:44:44.174 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.170 7fd35e7fc700 1 --2- 172.21.2.112:0/4238941341 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd350018730 0x7fd35001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:44.174 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.170 7fd35e7fc700 1 -- 172.21.2.112:0/4238941341 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fd364032a00 con 0x7fd36818f0a0 2021-02-10T11:44:44.174 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.171 7fd36d416700 1 --2- 172.21.2.112:0/4238941341 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd350018730 0x7fd35001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:44.175 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.171 7fd36d416700 1 --2- 172.21.2.112:0/4238941341 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd350018730 0x7fd35001ac10 crc :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:44.202 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.201 7fd36fdfb700 1 -- 172.21.2.112:0/4238941341 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3680ad790 con 0x7fd36818f0a0 2021-02-10T11:44:44.204 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.203 7fd35e7fc700 1 -- 172.21.2.112:0/4238941341 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd364001160 con 0x7fd36818f0a0 2021-02-10T11:44:44.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.328 7fd36fdfb700 1 -- 172.21.2.112:0/4238941341 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd3680ad790 con 0x7fd36818f0a0 2021-02-10T11:44:44.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.329 7fd35e7fc700 1 -- 172.21.2.112:0/4238941341 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v150) v1 ==== 74+0+14020 (secure 0 0 0) 0x7fd3640043a0 con 0x7fd36818f0a0 2021-02-10T11:44:44.331 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:44:44.331 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":150,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:44:43.721211","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":32,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:44:41.688683","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"150","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":33,"pool_name":"cephfs_data","create_time":"2021-02-10 11:44:42.764257","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"150","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:44:44.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.331 7fd36fdfb700 1 -- 172.21.2.112:0/4238941341 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd350018730 msgr2=0x7fd35001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:44.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.331 7fd36fdfb700 1 --2- 172.21.2.112:0/4238941341 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd350018730 0x7fd35001ac10 crc :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:44.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.331 7fd36fdfb700 1 -- 172.21.2.112:0/4238941341 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd36818f0a0 msgr2=0x7fd368196ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:44.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.331 7fd36fdfb700 1 --2- 172.21.2.112:0/4238941341 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd36818f0a0 0x7fd368196ab0 secure :-1 s=READY pgs=2649 cs=0 l=1 rev1=1 rx=0x7fd364001ea0 tx=0x7fd36400a880).stop 2021-02-10T11:44:44.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.331 7fd36fdfb700 1 -- 172.21.2.112:0/4238941341 shutdown_connections 2021-02-10T11:44:44.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.331 7fd36fdfb700 1 --2- 172.21.2.112:0/4238941341 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd350018730 0x7fd35001ac10 unknown :-1 s=CLOSED pgs=1210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:44.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.331 7fd36fdfb700 1 --2- 172.21.2.112:0/4238941341 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd368094650 0x7fd368094a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:44.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.331 7fd36fdfb700 1 --2- 172.21.2.112:0/4238941341 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd3681970d0 0x7fd3681974d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:44.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.331 7fd36fdfb700 1 --2- 172.21.2.112:0/4238941341 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd36818f0a0 0x7fd368196ab0 unknown :-1 s=CLOSED pgs=2649 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:44.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.331 7fd36fdfb700 1 -- 172.21.2.112:0/4238941341 >> 172.21.2.112:0/4238941341 conn(0x7fd3681a38c0 msgr2=0x7fd3681a2f00 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:44.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.331 7fd36fdfb700 1 -- 172.21.2.112:0/4238941341 shutdown_connections 2021-02-10T11:44:44.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.331 7fd36fdfb700 1 -- 172.21.2.112:0/4238941341 wait complete. 2021-02-10T11:44:44.341 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:44:44.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.753 7fa0e7e16700 1 Processor -- start 2021-02-10T11:44:44.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.753 7fa0e7e16700 1 -- start start 2021-02-10T11:44:44.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.753 7fa0e7e16700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa0e00d1870 0x7fa0e01b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:44.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.753 7fa0e7e16700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0e00f0b10 con 0x7fa0e00f0650 2021-02-10T11:44:44.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.753 7fa0e7e16700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0e00d0310 con 0x7fa0e01b0a10 2021-02-10T11:44:44.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.753 7fa0e7e16700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0e00e5ed0 con 0x7fa0e00d1870 2021-02-10T11:44:44.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.753 7fa0e6433700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fa0e00f0650 0x7fa0e00cfb20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:57606/0 (socket says 172.21.2.112:57606) 2021-02-10T11:44:44.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.753 7fa0e6433700 1 -- 172.21.2.112:0/3173067646 learned_addr learned my addr 172.21.2.112:0/3173067646 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:44:44.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.754 7fa0e5c32700 1 --2- 172.21.2.112:0/3173067646 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa0e00d1870 0x7fa0e01b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:44.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.754 7fa0e4c30700 1 -- 172.21.2.112:0/3173067646 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 911687868 0 0) 0x7fa0d8001500 con 0x7fa0e00f0650 2021-02-10T11:44:44.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.754 7fa0e4c30700 1 -- 172.21.2.112:0/3173067646 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0cc002fe0 con 0x7fa0e00f0650 2021-02-10T11:44:44.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.754 7fa0e4c30700 1 -- 172.21.2.112:0/3173067646 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3782793130 0 0) 0x7fa0d4000ff0 con 0x7fa0e01b0a10 2021-02-10T11:44:44.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.754 7fa0e5c32700 1 -- 172.21.2.112:0/3173067646 >> v1:172.21.2.120:6789/0 conn(0x7fa0e01b0a10 legacy=0x7fa0e00cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:44.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.754 7fa0e5c32700 1 -- 172.21.2.112:0/3173067646 >> v1:172.21.2.112:6789/0 conn(0x7fa0e00f0650 legacy=0x7fa0e00cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:44.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.754 7fa0e5c32700 1 -- 172.21.2.112:0/3173067646 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0e00d0530 con 0x7fa0e00d1870 2021-02-10T11:44:44.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.754 7fa0e5c32700 1 --2- 172.21.2.112:0/3173067646 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa0e00d1870 0x7fa0e01b03f0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7fa0dc006650 tx=0x7fa0dc009a80).ready entity=mon.2 client_cookie=e7451754b4ecbb49 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:44.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.754 7fa0e4c30700 1 -- 172.21.2.112:0/3173067646 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa0dc00c0c0 con 0x7fa0e00d1870 2021-02-10T11:44:44.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.754 7fa0e4c30700 1 -- 172.21.2.112:0/3173067646 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0dc00c340 con 0x7fa0e00d1870 2021-02-10T11:44:44.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.754 7fa0e4c30700 1 -- 172.21.2.112:0/3173067646 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa0dc00c860 con 0x7fa0e00d1870 2021-02-10T11:44:44.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.754 7fa0e7e16700 1 -- 172.21.2.112:0/3173067646 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa0e00d1870 msgr2=0x7fa0e01b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:44.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.754 7fa0e7e16700 1 --2- 172.21.2.112:0/3173067646 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa0e00d1870 0x7fa0e01b03f0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7fa0dc006650 tx=0x7fa0dc009a80).stop 2021-02-10T11:44:44.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.755 7fa0e7e16700 1 -- 172.21.2.112:0/3173067646 shutdown_connections 2021-02-10T11:44:44.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.755 7fa0e7e16700 1 --2- 172.21.2.112:0/3173067646 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa0e00d1870 0x7fa0e01b03f0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:44.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.755 7fa0e7e16700 1 -- 172.21.2.112:0/3173067646 >> 172.21.2.112:0/3173067646 conn(0x7fa0e00ce450 msgr2=0x7fa0e00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:44.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.755 7fa0e7e16700 1 -- 172.21.2.112:0/3173067646 shutdown_connections 2021-02-10T11:44:44.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.755 7fa0e7e16700 1 -- 172.21.2.112:0/3173067646 wait complete. 2021-02-10T11:44:44.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.770 7fa0e7e16700 1 Processor -- start 2021-02-10T11:44:44.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.770 7fa0e7e16700 1 -- start start 2021-02-10T11:44:44.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.771 7fa0e7e16700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa0e01971e0 0x7fa0e018eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:44.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.771 7fa0e7e16700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa0e01b8180 0x7fa0e01b8580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:44.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.771 7fa0e7e16700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa0e01b8ba0 0x7fa0e00b17b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:44.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.771 7fa0e7e16700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0e01b9750 con 0x7fa0e01971e0 2021-02-10T11:44:44.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.771 7fa0e7e16700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0e018e930 con 0x7fa0e01b8180 2021-02-10T11:44:44.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.771 7fa0e7e16700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0e00b1dd0 con 0x7fa0e01b8ba0 2021-02-10T11:44:44.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.771 7fa0e5c32700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa0e01971e0 0x7fa0e018eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:44.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.771 7fa0e5c32700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa0e01971e0 0x7fa0e018eda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:58148/0 (socket says 172.21.2.112:58148) 2021-02-10T11:44:44.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.771 7fa0e5c32700 1 -- 172.21.2.112:0/1148009743 learned_addr learned my addr 172.21.2.112:0/1148009743 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:44:44.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.771 7fa0e5431700 1 --2- 172.21.2.112:0/1148009743 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa0e01b8180 0x7fa0e01b8580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:44.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.771 7fa0e5c32700 1 -- 172.21.2.112:0/1148009743 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa0e01b8ba0 msgr2=0x7fa0e00b17b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:44.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.771 7fa0e5c32700 1 --2- 172.21.2.112:0/1148009743 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa0e01b8ba0 0x7fa0e00b17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:44.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.771 7fa0e5c32700 1 -- 172.21.2.112:0/1148009743 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa0e01b8180 msgr2=0x7fa0e01b8580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:44.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.771 7fa0e5c32700 1 --2- 172.21.2.112:0/1148009743 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa0e01b8180 0x7fa0e01b8580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:44.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.771 7fa0e5c32700 1 -- 172.21.2.112:0/1148009743 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0e00b1ff0 con 0x7fa0e01971e0 2021-02-10T11:44:44.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.771 7fa0e5431700 1 --2- 172.21.2.112:0/1148009743 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa0e01b8180 0x7fa0e01b8580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:44:44.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.771 7fa0e5c32700 1 --2- 172.21.2.112:0/1148009743 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa0e01971e0 0x7fa0e018eda0 secure :-1 s=READY pgs=2651 cs=0 l=1 rev1=1 rx=0x7fa0dc009630 tx=0x7fa0dc00b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:44.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.771 7fa0d27fc700 1 -- 172.21.2.112:0/1148009743 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa0dc0093a0 con 0x7fa0e01971e0 2021-02-10T11:44:44.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.771 7fa0d27fc700 1 -- 172.21.2.112:0/1148009743 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0dc00bd70 con 0x7fa0e01971e0 2021-02-10T11:44:44.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.771 7fa0e7e16700 1 -- 172.21.2.112:0/1148009743 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0e00b1ff0 con 0x7fa0e01971e0 2021-02-10T11:44:44.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.771 7fa0d27fc700 1 -- 172.21.2.112:0/1148009743 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa0dc005560 con 0x7fa0e01971e0 2021-02-10T11:44:44.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.771 7fa0e7e16700 1 -- 172.21.2.112:0/1148009743 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0e00b1ff0 con 0x7fa0e01971e0 2021-02-10T11:44:44.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.772 7fa0d27fc700 1 -- 172.21.2.112:0/1148009743 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa0dc005770 con 0x7fa0e01971e0 2021-02-10T11:44:44.777 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.772 7fa0d27fc700 1 --2- 172.21.2.112:0/1148009743 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa0c8018730 0x7fa0c801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:44.777 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.772 7fa0d27fc700 1 -- 172.21.2.112:0/1148009743 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fa0dc032650 con 0x7fa0e01971e0 2021-02-10T11:44:44.777 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.772 7fa0e5431700 1 --2- 172.21.2.112:0/1148009743 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa0c8018730 0x7fa0c801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:44.777 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.773 7fa0e5431700 1 --2- 172.21.2.112:0/1148009743 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa0c8018730 0x7fa0c801ac10 crc :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:44.804 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.803 7fa0e7e16700 1 -- 172.21.2.112:0/1148009743 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0e00ee060 con 0x7fa0e01971e0 2021-02-10T11:44:44.807 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.806 7fa0d27fc700 1 -- 172.21.2.112:0/1148009743 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa0dc005560 con 0x7fa0e01971e0 2021-02-10T11:44:44.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.935 7fa0e7e16700 1 -- 172.21.2.112:0/1148009743 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa0e00ee060 con 0x7fa0e01971e0 2021-02-10T11:44:44.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.935 7fa0d27fc700 1 -- 172.21.2.112:0/1148009743 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v151) v1 ==== 74+0+14007 (secure 0 0 0) 0x7fa0dc005770 con 0x7fa0e01971e0 2021-02-10T11:44:44.937 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:44:44.937 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":151,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:44:44.723715","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":32,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:44:41.688683","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"150","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":33,"pool_name":"cephfs_data","create_time":"2021-02-10 11:44:42.764257","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"151","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:44:44.938 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.937 7fa0e7e16700 1 -- 172.21.2.112:0/1148009743 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa0c8018730 msgr2=0x7fa0c801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:44.939 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.937 7fa0e7e16700 1 --2- 172.21.2.112:0/1148009743 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa0c8018730 0x7fa0c801ac10 crc :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:44.939 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.937 7fa0e7e16700 1 -- 172.21.2.112:0/1148009743 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa0e01971e0 msgr2=0x7fa0e018eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:44.939 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.937 7fa0e7e16700 1 --2- 172.21.2.112:0/1148009743 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa0e01971e0 0x7fa0e018eda0 secure :-1 s=READY pgs=2651 cs=0 l=1 rev1=1 rx=0x7fa0dc009630 tx=0x7fa0dc00b980).stop 2021-02-10T11:44:44.939 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.937 7fa0e7e16700 1 -- 172.21.2.112:0/1148009743 shutdown_connections 2021-02-10T11:44:44.939 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.937 7fa0e7e16700 1 --2- 172.21.2.112:0/1148009743 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa0c8018730 0x7fa0c801ac10 unknown :-1 s=CLOSED pgs=1211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:44.939 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.937 7fa0e7e16700 1 --2- 172.21.2.112:0/1148009743 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa0e01b8180 0x7fa0e01b8580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:44.939 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.937 7fa0e7e16700 1 --2- 172.21.2.112:0/1148009743 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa0e01b8ba0 0x7fa0e00b17b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:44.939 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.937 7fa0e7e16700 1 --2- 172.21.2.112:0/1148009743 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa0e01971e0 0x7fa0e018eda0 unknown :-1 s=CLOSED pgs=2651 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:44.939 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.937 7fa0e7e16700 1 -- 172.21.2.112:0/1148009743 >> 172.21.2.112:0/1148009743 conn(0x7fa0e019f8f0 msgr2=0x7fa0e018c260 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:44.939 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.937 7fa0e7e16700 1 -- 172.21.2.112:0/1148009743 shutdown_connections 2021-02-10T11:44:44.939 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:44.937 7fa0e7e16700 1 -- 172.21.2.112:0/1148009743 wait complete. 2021-02-10T11:44:44.947 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2021-02-10T11:44:45.379 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:45.362 7f80b35cc700 1 Processor -- start 2021-02-10T11:44:45.380 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:45.362 7f80b35cc700 1 -- start start 2021-02-10T11:44:45.380 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:45.362 7f80b35cc700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f80ac0d1870 0x7f80ac1b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:45.380 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:45.362 7f80b35cc700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f80ac1b0ae0 0x7f80ac108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:45.380 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:45.363 7f80b35cc700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80ac109a60 con 0x7f80ac0d1c70 2021-02-10T11:44:45.380 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:45.363 7f80b35cc700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80ac0e5ef0 con 0x7f80ac1b0ae0 2021-02-10T11:44:45.380 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:45.363 7f80b35cc700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80ac109750 con 0x7f80ac0d1870 2021-02-10T11:44:45.380 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:45.363 7f80b0be7700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f80ac1b0ae0 0x7f80ac108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:45.380 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:45.363 7f80b0be7700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f80ac1b0ae0 0x7f80ac108940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:58156/0 (socket says 172.21.2.112:58156) 2021-02-10T11:44:45.380 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:45.363 7f80b0be7700 1 -- 172.21.2.112:0/351295072 learned_addr learned my addr 172.21.2.112:0/351295072 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:44:45.380 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:45.363 7f80b0be7700 1 -- 172.21.2.112:0/351295072 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f80ac0d1870 msgr2=0x7f80ac1b04c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:44:45.380 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:45.363 7f80b0be7700 1 --2- 172.21.2.112:0/351295072 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f80ac0d1870 0x7f80ac1b04c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:45.381 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:45.363 7f80b0be7700 1 -- 172.21.2.112:0/351295072 >> v1:172.21.2.120:6789/0 conn(0x7f80ac0d1c70 legacy=0x7f80ac108f60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:44:45.381 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:45.363 7f80b0be7700 1 -- 172.21.2.112:0/351295072 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80ac088070 con 0x7f80ac1b0ae0 2021-02-10T11:44:45.381 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:45.363 7f80b0be7700 1 --2- 172.21.2.112:0/351295072 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f80ac1b0ae0 0x7f80ac108940 secure :-1 s=READY pgs=2652 cs=0 l=1 rev1=1 rx=0x7f80a8006700 tx=0x7f80a8009b00).ready entity=mon.0 client_cookie=dfeb33d815883c42 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:45.381 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:45.363 7f809ffff700 1 -- 172.21.2.112:0/351295072 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f80a800c0c0 con 0x7f80ac1b0ae0 2021-02-10T11:44:45.381 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:45.363 7f809ffff700 1 -- 172.21.2.112:0/351295072 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80a800c340 con 0x7f80ac1b0ae0 2021-02-10T11:44:45.381 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:45.363 7f809ffff700 1 -- 172.21.2.112:0/351295072 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f80a800c820 con 0x7f80ac1b0ae0 2021-02-10T11:44:45.381 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:45.364 7f80b35cc700 1 -- 172.21.2.112:0/351295072 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f80ac1b0ae0 msgr2=0x7f80ac108940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:45.381 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:45.364 7f80b35cc700 1 --2- 172.21.2.112:0/351295072 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f80ac1b0ae0 0x7f80ac108940 secure :-1 s=READY pgs=2652 cs=0 l=1 rev1=1 rx=0x7f80a8006700 tx=0x7f80a8009b00).stop 2021-02-10T11:44:45.381 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:45.364 7f80b35cc700 1 -- 172.21.2.112:0/351295072 shutdown_connections 2021-02-10T11:44:45.381 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:45.364 7f80b35cc700 1 --2- 172.21.2.112:0/351295072 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f80ac1b0ae0 0x7f80ac108940 unknown :-1 s=CLOSED pgs=2652 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:45.381 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:45.364 7f80b35cc700 1 --2- 172.21.2.112:0/351295072 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f80ac0d1870 0x7f80ac1b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:45.381 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:45.364 7f80b35cc700 1 -- 172.21.2.112:0/351295072 >> 172.21.2.112:0/351295072 conn(0x7f80ac0ce420 msgr2=0x7f80ac0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:45.382 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:45.364 7f80b35cc700 1 -- 172.21.2.112:0/351295072 shutdown_connections 2021-02-10T11:44:45.382 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:45.364 7f80b35cc700 1 -- 172.21.2.112:0/351295072 wait complete. 2021-02-10T11:44:45.383 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:45.379 7f80b35cc700 1 Processor -- start 2021-02-10T11:44:45.383 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:45.379 7f80b35cc700 1 -- start start 2021-02-10T11:44:45.383 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:45.379 7f80b35cc700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f80ac0c3820 0x7f80ac084cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:45.384 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:45.379 7f80b35cc700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f80ac0a0290 0x7f80ac0852d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:45.384 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:45.379 7f80b35cc700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f80ac086270 0x7f80ac086670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:45.384 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:45.379 7f80b35cc700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80ac0c0700 con 0x7f80ac0c3820 2021-02-10T11:44:45.384 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:45.379 7f80b35cc700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80ac0858f0 con 0x7f80ac0a0290 2021-02-10T11:44:45.384 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:45.379 7f80b35cc700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80ac085b10 con 0x7f80ac086270 2021-02-10T11:44:45.384 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:45.379 7f80b13e8700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f80ac0c3820 0x7f80ac084cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:45.384 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:45.379 7f80b13e8700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f80ac0c3820 0x7f80ac084cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:58158/0 (socket says 172.21.2.112:58158) 2021-02-10T11:44:45.384 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:45.379 7f80b13e8700 1 -- 172.21.2.112:0/214694665 learned_addr learned my addr 172.21.2.112:0/214694665 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:44:45.384 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:45.380 7f80b13e8700 1 -- 172.21.2.112:0/214694665 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f80ac086270 msgr2=0x7f80ac086670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:45.384 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:45.380 7f80b13e8700 1 --2- 172.21.2.112:0/214694665 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f80ac086270 0x7f80ac086670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:45.384 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:45.380 7f80b13e8700 1 -- 172.21.2.112:0/214694665 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f80ac0a0290 msgr2=0x7f80ac0852d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:45.384 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:45.380 7f80b13e8700 1 --2- 172.21.2.112:0/214694665 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f80ac0a0290 0x7f80ac0852d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:45.384 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:45.380 7f80b13e8700 1 -- 172.21.2.112:0/214694665 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80ac086c90 con 0x7f80ac0c3820 2021-02-10T11:44:45.385 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:45.380 7f80b13e8700 1 --2- 172.21.2.112:0/214694665 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f80ac0c3820 0x7f80ac084cb0 secure :-1 s=READY pgs=2653 cs=0 l=1 rev1=1 rx=0x7f80a000b5c0 tx=0x7f80a000b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:45.385 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:45.380 7f809dffb700 1 -- 172.21.2.112:0/214694665 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f80a000bd50 con 0x7f80ac0c3820 2021-02-10T11:44:45.385 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:45.380 7f809dffb700 1 -- 172.21.2.112:0/214694665 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80a0006030 con 0x7f80ac0c3820 2021-02-10T11:44:45.385 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:45.380 7f809dffb700 1 -- 172.21.2.112:0/214694665 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f80a00054d0 con 0x7f80ac0c3820 2021-02-10T11:44:45.385 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:45.380 7f80b35cc700 1 -- 172.21.2.112:0/214694665 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80ac086c90 con 0x7f80ac0c3820 2021-02-10T11:44:45.385 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:45.380 7f80b35cc700 1 -- 172.21.2.112:0/214694665 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80ac1a2f00 con 0x7f80ac0c3820 2021-02-10T11:44:45.385 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:45.381 7f809dffb700 1 -- 172.21.2.112:0/214694665 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f80a0006210 con 0x7f80ac0c3820 2021-02-10T11:44:45.385 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:45.381 7f809dffb700 1 --2- 172.21.2.112:0/214694665 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f808c0186c0 0x7f808c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:45.385 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:45.381 7f809dffb700 1 -- 172.21.2.112:0/214694665 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f80a0032680 con 0x7f80ac0c3820 2021-02-10T11:44:45.385 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:45.381 7f80b0be7700 1 --2- 172.21.2.112:0/214694665 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f808c0186c0 0x7f808c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:45.385 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:45.382 7f80b0be7700 1 --2- 172.21.2.112:0/214694665 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f808c0186c0 0x7f808c01abc0 crc :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:45.413 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:45.412 7f80b35cc700 1 -- 172.21.2.112:0/214694665 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80ac082a70 con 0x7f80ac0c3820 2021-02-10T11:44:45.416 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:45.415 7f809dffb700 1 -- 172.21.2.112:0/214694665 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f80a0000960 con 0x7f80ac0c3820 2021-02-10T11:44:45.553 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:45.552 7f80b35cc700 1 -- 172.21.2.112:0/214694665 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"} v 0) v1 -- 0x7f80ac082a70 con 0x7f80ac0c3820 2021-02-10T11:44:45.733 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:45.732 7f809dffb700 1 -- 172.21.2.112:0/214694665 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"}]=0 v130) v1 ==== 118+0+0 (secure 0 0 0) 0x7f80a0006030 con 0x7f80ac0c3820 2021-02-10T11:44:45.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:45.769 7f80b35cc700 1 -- 172.21.2.112:0/214694665 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f808c0186c0 msgr2=0x7f808c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:45.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:45.769 7f80b35cc700 1 --2- 172.21.2.112:0/214694665 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f808c0186c0 0x7f808c01abc0 crc :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:45.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:45.769 7f80b35cc700 1 -- 172.21.2.112:0/214694665 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f80ac0c3820 msgr2=0x7f80ac084cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:45.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:45.769 7f80b35cc700 1 --2- 172.21.2.112:0/214694665 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f80ac0c3820 0x7f80ac084cb0 secure :-1 s=READY pgs=2653 cs=0 l=1 rev1=1 rx=0x7f80a000b5c0 tx=0x7f80a000b960).stop 2021-02-10T11:44:45.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:45.770 7f80b35cc700 1 -- 172.21.2.112:0/214694665 shutdown_connections 2021-02-10T11:44:45.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:45.770 7f80b35cc700 1 --2- 172.21.2.112:0/214694665 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f808c0186c0 0x7f808c01abc0 unknown :-1 s=CLOSED pgs=1212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:45.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:45.770 7f80b35cc700 1 --2- 172.21.2.112:0/214694665 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f80ac0a0290 0x7f80ac0852d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:45.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:45.770 7f80b35cc700 1 --2- 172.21.2.112:0/214694665 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f80ac086270 0x7f80ac086670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:45.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:45.770 7f80b35cc700 1 --2- 172.21.2.112:0/214694665 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f80ac0c3820 0x7f80ac084cb0 unknown :-1 s=CLOSED pgs=2653 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:45.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:45.770 7f80b35cc700 1 -- 172.21.2.112:0/214694665 >> 172.21.2.112:0/214694665 conn(0x7f80ac0872c0 msgr2=0x7f80ac095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:45.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:45.770 7f80b35cc700 1 -- 172.21.2.112:0/214694665 shutdown_connections 2021-02-10T11:44:45.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:45.770 7f80b35cc700 1 -- 172.21.2.112:0/214694665 wait complete. 2021-02-10T11:44:45.780 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:44:46.208 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.190 7f0ee63f1700 1 Processor -- start 2021-02-10T11:44:46.208 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.190 7f0ee63f1700 1 -- start start 2021-02-10T11:44:46.209 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.190 7f0ee63f1700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f0ee00d1870 0x7f0ee01b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:46.209 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.190 7f0ee63f1700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0ee01b0a10 0x7f0ee00cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:46.209 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.191 7f0ee63f1700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ee00f0b10 con 0x7f0ee00d1c70 2021-02-10T11:44:46.209 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.191 7f0ee63f1700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ee00e5ed0 con 0x7f0ee00d1870 2021-02-10T11:44:46.209 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.191 7f0ee63f1700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ee00f0650 con 0x7f0ee01b0a10 2021-02-10T11:44:46.209 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.191 7f0edffff700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f0ee00d1870 0x7f0ee01b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:46.209 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.191 7f0edf7fe700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0ee01b0a10 0x7f0ee00cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:46.209 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.191 7f0ee4a0e700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7f0ee00d1c70 0x7f0ee00cf950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:51188/0 (socket says 172.21.2.112:51188) 2021-02-10T11:44:46.209 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.191 7f0ee4a0e700 1 -- 172.21.2.112:0/1598898917 learned_addr learned my addr 172.21.2.112:0/1598898917 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:44:46.209 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.192 7f0edeffd700 1 -- 172.21.2.112:0/1598898917 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2191674549 0 0) 0x7f0ed8001510 con 0x7f0ee00d1c70 2021-02-10T11:44:46.209 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.192 7f0edeffd700 1 -- 172.21.2.112:0/1598898917 --> v1:172.21.2.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ec8002fe0 con 0x7f0ee00d1c70 2021-02-10T11:44:46.209 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.192 7f0edffff700 1 -- 172.21.2.112:0/1598898917 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0ee01b0a10 msgr2=0x7f0ee00cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:46.209 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.192 7f0edffff700 1 --2- 172.21.2.112:0/1598898917 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0ee01b0a10 0x7f0ee00cf330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:46.210 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.192 7f0edffff700 1 -- 172.21.2.112:0/1598898917 >> v1:172.21.2.120:6789/0 conn(0x7f0ee00d1c70 legacy=0x7f0ee00cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:46.210 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.192 7f0edffff700 1 -- 172.21.2.112:0/1598898917 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ee00f0870 con 0x7f0ee00d1870 2021-02-10T11:44:46.210 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.192 7f0edffff700 1 --2- 172.21.2.112:0/1598898917 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0ee00d1870 0x7f0ee01b03f0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 rx=0x7f0ed0006650 tx=0x7f0ed0009a80).ready entity=mon.1 client_cookie=7cae8c87dc5fff73 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:46.210 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.193 7f0edeffd700 1 -- 172.21.2.112:0/1598898917 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0ed00009b0 con 0x7f0ee00d1870 2021-02-10T11:44:46.210 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.193 7f0edeffd700 1 -- 172.21.2.112:0/1598898917 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ed000c0c0 con 0x7f0ee00d1870 2021-02-10T11:44:46.210 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.193 7f0edeffd700 1 -- 172.21.2.112:0/1598898917 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0ed000c520 con 0x7f0ee00d1870 2021-02-10T11:44:46.210 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.193 7f0ee63f1700 1 -- 172.21.2.112:0/1598898917 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0ee00d1870 msgr2=0x7f0ee01b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:46.210 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.193 7f0ee63f1700 1 --2- 172.21.2.112:0/1598898917 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0ee00d1870 0x7f0ee01b03f0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 rx=0x7f0ed0006650 tx=0x7f0ed0009a80).stop 2021-02-10T11:44:46.210 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.193 7f0ee63f1700 1 -- 172.21.2.112:0/1598898917 shutdown_connections 2021-02-10T11:44:46.210 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.193 7f0ee63f1700 1 --2- 172.21.2.112:0/1598898917 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0ee01b0a10 0x7f0ee00cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:46.210 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.193 7f0ee63f1700 1 --2- 172.21.2.112:0/1598898917 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0ee00d1870 0x7f0ee01b03f0 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:46.210 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.193 7f0ee63f1700 1 -- 172.21.2.112:0/1598898917 >> 172.21.2.112:0/1598898917 conn(0x7f0ee00ce450 msgr2=0x7f0ee00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:46.211 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.193 7f0ee63f1700 1 -- 172.21.2.112:0/1598898917 shutdown_connections 2021-02-10T11:44:46.211 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.193 7f0ee63f1700 1 -- 172.21.2.112:0/1598898917 wait complete. 2021-02-10T11:44:46.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.207 7f0ee63f1700 1 Processor -- start 2021-02-10T11:44:46.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.208 7f0ee63f1700 1 -- start start 2021-02-10T11:44:46.213 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.208 7f0ee63f1700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0ee0195650 0x7f0ee00b5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:46.213 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.208 7f0ee63f1700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0ee00b5b80 0x7f0ee01a2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:46.213 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.208 7f0ee63f1700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0ee00acfa0 0x7f0ee01a3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:46.213 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.208 7f0ee63f1700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ee00b6010 con 0x7f0ee0195650 2021-02-10T11:44:46.213 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.208 7f0ee63f1700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ee01a3a60 con 0x7f0ee00b5b80 2021-02-10T11:44:46.213 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.208 7f0ee63f1700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ee01a3c40 con 0x7f0ee00acfa0 2021-02-10T11:44:46.213 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.208 7f0edffff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0ee0195650 0x7f0ee00b5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:46.213 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.208 7f0edffff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0ee0195650 0x7f0ee00b5560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:58172/0 (socket says 172.21.2.112:58172) 2021-02-10T11:44:46.213 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.208 7f0edffff700 1 -- 172.21.2.112:0/6431407 learned_addr learned my addr 172.21.2.112:0/6431407 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:44:46.213 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.209 7f0ee4a0e700 1 --2- 172.21.2.112:0/6431407 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0ee00acfa0 0x7f0ee01a3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:46.213 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.209 7f0edf7fe700 1 --2- 172.21.2.112:0/6431407 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0ee00b5b80 0x7f0ee01a2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:46.213 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.209 7f0edffff700 1 -- 172.21.2.112:0/6431407 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0ee00acfa0 msgr2=0x7f0ee01a3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:46.213 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.209 7f0edffff700 1 --2- 172.21.2.112:0/6431407 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0ee00acfa0 0x7f0ee01a3440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:46.214 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.209 7f0edffff700 1 -- 172.21.2.112:0/6431407 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0ee00b5b80 msgr2=0x7f0ee01a2e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:46.214 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.209 7f0edffff700 1 --2- 172.21.2.112:0/6431407 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0ee00b5b80 0x7f0ee01a2e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:46.214 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.209 7f0edffff700 1 -- 172.21.2.112:0/6431407 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ee00bcae0 con 0x7f0ee0195650 2021-02-10T11:44:46.214 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.209 7f0edf7fe700 1 --2- 172.21.2.112:0/6431407 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0ee00b5b80 0x7f0ee01a2e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:44:46.214 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.209 7f0edffff700 1 --2- 172.21.2.112:0/6431407 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0ee0195650 0x7f0ee00b5560 secure :-1 s=READY pgs=2654 cs=0 l=1 rev1=1 rx=0x7f0ed0009650 tx=0x7f0ed000b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:46.214 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.209 7f0edcff9700 1 -- 172.21.2.112:0/6431407 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0ed0009360 con 0x7f0ee0195650 2021-02-10T11:44:46.214 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.209 7f0edcff9700 1 -- 172.21.2.112:0/6431407 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ed000bde0 con 0x7f0ee0195650 2021-02-10T11:44:46.214 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.209 7f0edcff9700 1 -- 172.21.2.112:0/6431407 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0ed00052e0 con 0x7f0ee0195650 2021-02-10T11:44:46.214 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.209 7f0ee63f1700 1 -- 172.21.2.112:0/6431407 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ee00bcae0 con 0x7f0ee0195650 2021-02-10T11:44:46.214 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.209 7f0ee63f1700 1 -- 172.21.2.112:0/6431407 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ee00bcae0 con 0x7f0ee0195650 2021-02-10T11:44:46.214 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.210 7f0edcff9700 1 -- 172.21.2.112:0/6431407 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0ed0006030 con 0x7f0ee0195650 2021-02-10T11:44:46.214 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.210 7f0edcff9700 1 --2- 172.21.2.112:0/6431407 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0ec0018700 0x7f0ec001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:46.214 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.210 7f0edcff9700 1 -- 172.21.2.112:0/6431407 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f0ed0032630 con 0x7f0ee0195650 2021-02-10T11:44:46.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.210 7f0edf7fe700 1 --2- 172.21.2.112:0/6431407 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0ec0018700 0x7f0ec001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:46.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.210 7f0edf7fe700 1 --2- 172.21.2.112:0/6431407 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0ec0018700 0x7f0ec001abc0 crc :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:46.242 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.240 7f0ee63f1700 1 -- 172.21.2.112:0/6431407 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ee00b6010 con 0x7f0ee0195650 2021-02-10T11:44:46.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.243 7f0edcff9700 1 -- 172.21.2.112:0/6431407 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0ed00009c0 con 0x7f0ee0195650 2021-02-10T11:44:46.373 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.371 7f0ee63f1700 1 -- 172.21.2.112:0/6431407 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0ee00b6010 con 0x7f0ee0195650 2021-02-10T11:44:46.373 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.372 7f0edcff9700 1 -- 172.21.2.112:0/6431407 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 130 v130) v1 ==== 95+0+2108 (secure 0 0 0) 0x7f0ed000bd50 con 0x7f0ee0195650 2021-02-10T11:44:46.374 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:44:46.374 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":130,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":12006,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/1015001314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":1015001314},{"type":"v1","addr":"172.21.2.112:6833","nonce":1015001314}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":126}],"filesystems":[{"mdsmap":{"epoch":130,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:44:43.721296","modified":"2021-02-10 11:44:45.730948","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7268},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7268":{"gid":7268,"name":"b","rank":0,"incarnation":127,"state":"up:active","state_seq":3,"addr":"172.21.2.120:6835/3799929082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":3799929082},{"type":"v1","addr":"172.21.2.120:6835","nonce":3799929082}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":16}]} 2021-02-10T11:44:46.374 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 130 2021-02-10T11:44:46.375 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.374 7f0ee63f1700 1 -- 172.21.2.112:0/6431407 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0ec0018700 msgr2=0x7f0ec001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:46.375 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.374 7f0ee63f1700 1 --2- 172.21.2.112:0/6431407 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0ec0018700 0x7f0ec001abc0 crc :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:46.375 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.374 7f0ee63f1700 1 -- 172.21.2.112:0/6431407 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0ee0195650 msgr2=0x7f0ee00b5560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:46.375 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.374 7f0ee63f1700 1 --2- 172.21.2.112:0/6431407 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0ee0195650 0x7f0ee00b5560 secure :-1 s=READY pgs=2654 cs=0 l=1 rev1=1 rx=0x7f0ed0009650 tx=0x7f0ed000b960).stop 2021-02-10T11:44:46.376 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.374 7f0ee63f1700 1 -- 172.21.2.112:0/6431407 shutdown_connections 2021-02-10T11:44:46.376 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.374 7f0ee63f1700 1 --2- 172.21.2.112:0/6431407 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0ec0018700 0x7f0ec001abc0 unknown :-1 s=CLOSED pgs=1213 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:46.376 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.374 7f0ee63f1700 1 --2- 172.21.2.112:0/6431407 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0ee00b5b80 0x7f0ee01a2e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:46.376 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.374 7f0ee63f1700 1 --2- 172.21.2.112:0/6431407 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0ee00acfa0 0x7f0ee01a3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:46.376 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.374 7f0ee63f1700 1 --2- 172.21.2.112:0/6431407 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0ee0195650 0x7f0ee00b5560 unknown :-1 s=CLOSED pgs=2654 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:46.376 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.374 7f0ee63f1700 1 -- 172.21.2.112:0/6431407 >> 172.21.2.112:0/6431407 conn(0x7f0ee00b3040 msgr2=0x7f0ee00c3250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:46.376 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.374 7f0ee63f1700 1 -- 172.21.2.112:0/6431407 shutdown_connections 2021-02-10T11:44:46.376 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.374 7f0ee63f1700 1 -- 172.21.2.112:0/6431407 wait complete. 2021-02-10T11:44:46.384 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:44:46.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.795 7facf2f98700 1 Processor -- start 2021-02-10T11:44:46.844 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.795 7facf2f98700 1 -- start start 2021-02-10T11:44:46.844 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.796 7facf2f98700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7facec0d0340 0x7facec0ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:46.844 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.796 7facf2f98700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7facec0d1c70 0x7facec0efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:46.844 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.796 7facf2f98700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facec0f0b10 con 0x7facec0d1870 2021-02-10T11:44:46.844 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.796 7facf2f98700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facec0e5ed0 con 0x7facec0d0340 2021-02-10T11:44:46.845 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.796 7facf2f98700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facec1b0be0 con 0x7facec0d1c70 2021-02-10T11:44:46.845 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.796 7facebfff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7facec0d0340 0x7facec0ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:46.845 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.796 7facebfff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7facec0d0340 0x7facec0ef640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:58180/0 (socket says 172.21.2.112:58180) 2021-02-10T11:44:46.845 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.796 7facebfff700 1 -- 172.21.2.112:0/2159376088 learned_addr learned my addr 172.21.2.112:0/2159376088 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:44:46.845 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.796 7facebfff700 1 -- 172.21.2.112:0/2159376088 >> v2:172.21.2.120:3300/0 conn(0x7facec0d1c70 msgr2=0x7facec0efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:46.845 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.796 7faceb7fe700 1 -- 172.21.2.112:0/2159376088 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3259048789 0 0) 0x7facdc000ff0 con 0x7facec0d1870 2021-02-10T11:44:46.845 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.797 7facebfff700 1 --2- 172.21.2.112:0/2159376088 >> v2:172.21.2.120:3300/0 conn(0x7facec0d1c70 0x7facec0efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:46.845 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.797 7facebfff700 1 -- 172.21.2.112:0/2159376088 >> v1:172.21.2.112:6789/0 conn(0x7facec0d1870 legacy=0x7facec1b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:46.845 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.797 7facebfff700 1 -- 172.21.2.112:0/2159376088 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facec0f20a0 con 0x7facec0d0340 2021-02-10T11:44:46.845 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.797 7facebfff700 1 --2- 172.21.2.112:0/2159376088 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7facec0d0340 0x7facec0ef640 secure :-1 s=READY pgs=2656 cs=0 l=1 rev1=1 rx=0x7facd4006700 tx=0x7facd4009b00).ready entity=mon.0 client_cookie=7e3f63736e38a3a1 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:46.845 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.797 7faceb7fe700 1 -- 172.21.2.112:0/2159376088 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7facd400c0c0 con 0x7facec0d0340 2021-02-10T11:44:46.845 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.797 7faceb7fe700 1 -- 172.21.2.112:0/2159376088 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facd400c340 con 0x7facec0d0340 2021-02-10T11:44:46.845 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.797 7faceb7fe700 1 -- 172.21.2.112:0/2159376088 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7facd400c820 con 0x7facec0d0340 2021-02-10T11:44:46.846 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.797 7facf2f98700 1 -- 172.21.2.112:0/2159376088 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7facec0d0340 msgr2=0x7facec0ef640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:46.846 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.797 7facf2f98700 1 --2- 172.21.2.112:0/2159376088 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7facec0d0340 0x7facec0ef640 secure :-1 s=READY pgs=2656 cs=0 l=1 rev1=1 rx=0x7facd4006700 tx=0x7facd4009b00).stop 2021-02-10T11:44:46.846 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.797 7facf2f98700 1 -- 172.21.2.112:0/2159376088 shutdown_connections 2021-02-10T11:44:46.846 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.797 7facf2f98700 1 --2- 172.21.2.112:0/2159376088 >> v2:172.21.2.120:3300/0 conn(0x7facec0d1c70 0x7facec0efc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:46.846 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.797 7facf2f98700 1 --2- 172.21.2.112:0/2159376088 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7facec0d0340 0x7facec0ef640 unknown :-1 s=CLOSED pgs=2656 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:46.846 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.797 7facf2f98700 1 -- 172.21.2.112:0/2159376088 >> 172.21.2.112:0/2159376088 conn(0x7facec0ce450 msgr2=0x7facec0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:46.846 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.798 7facf2f98700 1 -- 172.21.2.112:0/2159376088 shutdown_connections 2021-02-10T11:44:46.846 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.798 7facf2f98700 1 -- 172.21.2.112:0/2159376088 wait complete. 2021-02-10T11:44:46.848 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.812 7facf2f98700 1 Processor -- start 2021-02-10T11:44:46.848 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.812 7facf2f98700 1 -- start start 2021-02-10T11:44:46.848 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.812 7facf2f98700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7facec1b7cf0 0x7facec19cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:46.848 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.812 7facf2f98700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7facec1b80f0 0x7facec1b84f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:46.848 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.813 7facf2f98700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7facec19da20 0x7facec19de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:46.848 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.813 7facf2f98700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facec0a77c0 con 0x7facec1b7cf0 2021-02-10T11:44:46.848 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.813 7facf2f98700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facec19d4a0 con 0x7facec19da20 2021-02-10T11:44:46.848 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.813 7facf2f98700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facec0954f0 con 0x7facec1b80f0 2021-02-10T11:44:46.848 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.813 7facf0db4700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7facec1b7cf0 0x7facec19cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:46.849 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.813 7facf0db4700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7facec1b7cf0 0x7facec19cdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:58186/0 (socket says 172.21.2.112:58186) 2021-02-10T11:44:46.849 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.813 7facf0db4700 1 -- 172.21.2.112:0/3106709745 learned_addr learned my addr 172.21.2.112:0/3106709745 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:44:46.849 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.813 7facebfff700 1 --2- 172.21.2.112:0/3106709745 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7facec1b80f0 0x7facec1b84f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:46.849 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.813 7facf15b5700 1 --2- 172.21.2.112:0/3106709745 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7facec19da20 0x7facec19de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:46.849 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.813 7facf0db4700 1 -- 172.21.2.112:0/3106709745 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7facec1b80f0 msgr2=0x7facec1b84f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:46.849 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.813 7facf0db4700 1 --2- 172.21.2.112:0/3106709745 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7facec1b80f0 0x7facec1b84f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:46.849 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.813 7facf0db4700 1 -- 172.21.2.112:0/3106709745 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7facec19da20 msgr2=0x7facec19de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:46.849 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.813 7facf0db4700 1 --2- 172.21.2.112:0/3106709745 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7facec19da20 0x7facec19de20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:46.849 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.813 7facf0db4700 1 -- 172.21.2.112:0/3106709745 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facec1b8d60 con 0x7facec1b7cf0 2021-02-10T11:44:46.849 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.813 7facf0db4700 1 --2- 172.21.2.112:0/3106709745 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7facec1b7cf0 0x7facec19cdb0 secure :-1 s=READY pgs=2657 cs=0 l=1 rev1=1 rx=0x7facdc006ed0 tx=0x7facdc00b8e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:46.849 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.814 7face97fa700 1 -- 172.21.2.112:0/3106709745 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7facdc006020 con 0x7facec1b7cf0 2021-02-10T11:44:46.849 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.814 7face97fa700 1 -- 172.21.2.112:0/3106709745 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facdc006320 con 0x7facec1b7cf0 2021-02-10T11:44:46.849 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.814 7face97fa700 1 -- 172.21.2.112:0/3106709745 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7facdc005730 con 0x7facec1b7cf0 2021-02-10T11:44:46.849 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.814 7facf2f98700 1 -- 172.21.2.112:0/3106709745 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facec1b8d60 con 0x7facec1b7cf0 2021-02-10T11:44:46.850 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.814 7facf2f98700 1 -- 172.21.2.112:0/3106709745 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facec19e730 con 0x7facec1b7cf0 2021-02-10T11:44:46.850 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.814 7face97fa700 1 -- 172.21.2.112:0/3106709745 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7facdc0054b0 con 0x7facec1b7cf0 2021-02-10T11:44:46.850 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.814 7face97fa700 1 --2- 172.21.2.112:0/3106709745 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7facd00186c0 0x7facd001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:46.850 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.814 7face97fa700 1 -- 172.21.2.112:0/3106709745 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 5573+0+0 (secure 0 0 0) 0x7facdc032430 con 0x7facec1b7cf0 2021-02-10T11:44:46.850 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.815 7facebfff700 1 --2- 172.21.2.112:0/3106709745 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7facd00186c0 0x7facd001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:46.850 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.815 7facebfff700 1 --2- 172.21.2.112:0/3106709745 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7facd00186c0 0x7facd001abc0 crc :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:46.850 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.845 7facf2f98700 1 -- 172.21.2.112:0/3106709745 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facec0ebc00 con 0x7facec1b7cf0 2021-02-10T11:44:46.851 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.848 7face97fa700 1 -- 172.21.2.112:0/3106709745 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7facdc030d60 con 0x7facec1b7cf0 2021-02-10T11:44:46.974 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.973 7facf2f98700 1 -- 172.21.2.112:0/3106709745 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7facec0f0b10 con 0x7facec1b7cf0 2021-02-10T11:44:46.975 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.974 7face97fa700 1 -- 172.21.2.112:0/3106709745 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v151) v1 ==== 74+0+14007 (secure 0 0 0) 0x7facdc0054b0 con 0x7facec1b7cf0 2021-02-10T11:44:46.975 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:44:46.975 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":151,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:44:44.723715","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":32,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:44:41.688683","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"150","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":33,"pool_name":"cephfs_data","create_time":"2021-02-10 11:44:42.764257","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"151","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:44:46.976 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.975 7facf2f98700 1 -- 172.21.2.112:0/3106709745 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7facd00186c0 msgr2=0x7facd001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:46.976 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.975 7facf2f98700 1 --2- 172.21.2.112:0/3106709745 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7facd00186c0 0x7facd001abc0 crc :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:46.976 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.975 7facf2f98700 1 -- 172.21.2.112:0/3106709745 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7facec1b7cf0 msgr2=0x7facec19cdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:46.977 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.975 7facf2f98700 1 --2- 172.21.2.112:0/3106709745 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7facec1b7cf0 0x7facec19cdb0 secure :-1 s=READY pgs=2657 cs=0 l=1 rev1=1 rx=0x7facdc006ed0 tx=0x7facdc00b8e0).stop 2021-02-10T11:44:46.977 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.976 7facf2f98700 1 -- 172.21.2.112:0/3106709745 shutdown_connections 2021-02-10T11:44:46.977 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.976 7facf2f98700 1 --2- 172.21.2.112:0/3106709745 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7facd00186c0 0x7facd001abc0 unknown :-1 s=CLOSED pgs=1214 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:46.977 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.976 7facf2f98700 1 --2- 172.21.2.112:0/3106709745 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7facec19da20 0x7facec19de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:46.977 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.976 7facf2f98700 1 --2- 172.21.2.112:0/3106709745 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7facec1b80f0 0x7facec1b84f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:46.977 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.976 7facf2f98700 1 --2- 172.21.2.112:0/3106709745 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7facec1b7cf0 0x7facec19cdb0 unknown :-1 s=CLOSED pgs=2657 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:46.977 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.976 7facf2f98700 1 -- 172.21.2.112:0/3106709745 >> 172.21.2.112:0/3106709745 conn(0x7facec0ed810 msgr2=0x7facec094980 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:46.977 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.976 7facf2f98700 1 -- 172.21.2.112:0/3106709745 shutdown_connections 2021-02-10T11:44:46.978 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:46.976 7facf2f98700 1 -- 172.21.2.112:0/3106709745 wait complete. 2021-02-10T11:44:46.986 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:44:47.422 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:47.401 7f13a289b700 1 Processor -- start 2021-02-10T11:44:47.422 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:47.401 7f13a289b700 1 -- start start 2021-02-10T11:44:47.422 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:47.401 7f13a289b700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f139c0d1880 0x7f139c1b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:47.423 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:47.401 7f13a289b700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f139c0cff80 0x7f139c109140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:47.423 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:47.401 7f13a289b700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f139c0fe1d0 con 0x7f139c1b0af0 2021-02-10T11:44:47.423 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:47.401 7f13a289b700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f139c109ae0 con 0x7f139c0cff80 2021-02-10T11:44:47.423 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:47.401 7f13a289b700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f139c109760 con 0x7f139c0d1880 2021-02-10T11:44:47.423 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:47.401 7f13a0eb8700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f139c0cff80 0x7f139c109140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:47.424 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:47.401 7f13a0eb8700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f139c0cff80 0x7f139c109140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:58194/0 (socket says 172.21.2.112:58194) 2021-02-10T11:44:47.424 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:47.401 7f139b7fe700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7f139c1b0af0 0x7f139c108950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:51214/0 (socket says 172.21.2.112:51214) 2021-02-10T11:44:47.424 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:47.401 7f13a0eb8700 1 -- 172.21.2.112:0/559731764 learned_addr learned my addr 172.21.2.112:0/559731764 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:44:47.424 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:47.401 7f139bfff700 1 --2- 172.21.2.112:0/559731764 >> v2:172.21.2.120:3300/0 conn(0x7f139c0d1880 0x7f139c1b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:47.424 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:47.402 7f13a0eb8700 1 -- 172.21.2.112:0/559731764 >> v2:172.21.2.120:3300/0 conn(0x7f139c0d1880 msgr2=0x7f139c1b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:47.425 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:47.402 7f13a0eb8700 1 --2- 172.21.2.112:0/559731764 >> v2:172.21.2.120:3300/0 conn(0x7f139c0d1880 0x7f139c1b04d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:47.425 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:47.402 7f13a0eb8700 1 -- 172.21.2.112:0/559731764 >> v1:172.21.2.120:6789/0 conn(0x7f139c1b0af0 legacy=0x7f139c108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:47.425 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:47.402 7f13a0eb8700 1 -- 172.21.2.112:0/559731764 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f139c088070 con 0x7f139c0cff80 2021-02-10T11:44:47.425 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:47.402 7f139bfff700 1 --2- 172.21.2.112:0/559731764 >> v2:172.21.2.120:3300/0 conn(0x7f139c0d1880 0x7f139c1b04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:44:47.425 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:47.402 7f13a0eb8700 1 --2- 172.21.2.112:0/559731764 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f139c0cff80 0x7f139c109140 secure :-1 s=READY pgs=2658 cs=0 l=1 rev1=1 rx=0x7f1394006b80 tx=0x7f1394009b90).ready entity=mon.0 client_cookie=7688ee34d57bbc27 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:47.426 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:47.402 7f139affd700 1 -- 172.21.2.112:0/559731764 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f13940022e0 con 0x7f139c0cff80 2021-02-10T11:44:47.426 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:47.402 7f139affd700 1 -- 172.21.2.112:0/559731764 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1394002620 con 0x7f139c0cff80 2021-02-10T11:44:47.426 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:47.402 7f139affd700 1 -- 172.21.2.112:0/559731764 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1394002b00 con 0x7f139c0cff80 2021-02-10T11:44:47.426 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:47.402 7f13a289b700 1 -- 172.21.2.112:0/559731764 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f139c0cff80 msgr2=0x7f139c109140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:47.426 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:47.402 7f13a289b700 1 --2- 172.21.2.112:0/559731764 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f139c0cff80 0x7f139c109140 secure :-1 s=READY pgs=2658 cs=0 l=1 rev1=1 rx=0x7f1394006b80 tx=0x7f1394009b90).stop 2021-02-10T11:44:47.426 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:47.402 7f13a289b700 1 -- 172.21.2.112:0/559731764 shutdown_connections 2021-02-10T11:44:47.427 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:47.402 7f13a289b700 1 --2- 172.21.2.112:0/559731764 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f139c0cff80 0x7f139c109140 unknown :-1 s=CLOSED pgs=2658 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:47.427 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:47.402 7f13a289b700 1 --2- 172.21.2.112:0/559731764 >> v2:172.21.2.120:3300/0 conn(0x7f139c0d1880 0x7f139c1b04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:47.427 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:47.402 7f13a289b700 1 -- 172.21.2.112:0/559731764 >> 172.21.2.112:0/559731764 conn(0x7f139c0ce430 msgr2=0x7f139c0d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:47.427 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:47.403 7f13a289b700 1 -- 172.21.2.112:0/559731764 shutdown_connections 2021-02-10T11:44:47.427 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:47.403 7f13a289b700 1 -- 172.21.2.112:0/559731764 wait complete. 2021-02-10T11:44:47.428 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:47.417 7f13a289b700 1 Processor -- start 2021-02-10T11:44:47.428 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:47.417 7f13a289b700 1 -- start start 2021-02-10T11:44:47.428 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:47.418 7f13a289b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f139c1a4f80 0x7f139c0adac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:47.428 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:47.418 7f13a289b700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f139c0ae0e0 0x7f139c199a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:47.428 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:47.418 7f13a289b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f139c19a080 0x7f139c19a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:47.428 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:47.418 7f13a289b700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f139c19ab20 con 0x7f139c1a4f80 2021-02-10T11:44:47.429 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:47.418 7f13a289b700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f139c19ad40 con 0x7f139c0ae0e0 2021-02-10T11:44:47.429 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:47.418 7f13a289b700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f139c19d310 con 0x7f139c19a080 2021-02-10T11:44:47.429 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:47.418 7f139bfff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f139c1a4f80 0x7f139c0adac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:47.429 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:47.418 7f139bfff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f139c1a4f80 0x7f139c0adac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:58200/0 (socket says 172.21.2.112:58200) 2021-02-10T11:44:47.429 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:47.418 7f139bfff700 1 -- 172.21.2.112:0/2270201846 learned_addr learned my addr 172.21.2.112:0/2270201846 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:44:47.429 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:47.418 7f13a0eb8700 1 --2- 172.21.2.112:0/2270201846 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f139c19a080 0x7f139c19a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:47.430 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:47.418 7f139b7fe700 1 --2- 172.21.2.112:0/2270201846 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f139c0ae0e0 0x7f139c199a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:47.430 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:47.418 7f139bfff700 1 -- 172.21.2.112:0/2270201846 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f139c19a080 msgr2=0x7f139c19a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:47.430 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:47.418 7f139bfff700 1 --2- 172.21.2.112:0/2270201846 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f139c19a080 0x7f139c19a480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:47.430 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:47.418 7f139bfff700 1 -- 172.21.2.112:0/2270201846 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f139c0ae0e0 msgr2=0x7f139c199a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:47.430 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:47.418 7f139bfff700 1 --2- 172.21.2.112:0/2270201846 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f139c0ae0e0 0x7f139c199a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:47.430 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:47.418 7f139bfff700 1 -- 172.21.2.112:0/2270201846 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f139c19d500 con 0x7f139c1a4f80 2021-02-10T11:44:47.431 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:47.418 7f139b7fe700 1 --2- 172.21.2.112:0/2270201846 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f139c0ae0e0 0x7f139c199a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:44:47.431 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:47.419 7f139bfff700 1 --2- 172.21.2.112:0/2270201846 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f139c1a4f80 0x7f139c0adac0 secure :-1 s=READY pgs=2659 cs=0 l=1 rev1=1 rx=0x7f138c0099a0 tx=0x7f138c00bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:47.431 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:47.419 7f1398ff9700 1 -- 172.21.2.112:0/2270201846 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f138c00c2f0 con 0x7f139c1a4f80 2021-02-10T11:44:47.431 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:47.419 7f1398ff9700 1 -- 172.21.2.112:0/2270201846 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f138c00c670 con 0x7f139c1a4f80 2021-02-10T11:44:47.431 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:47.419 7f13a289b700 1 -- 172.21.2.112:0/2270201846 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f139c19d500 con 0x7f139c1a4f80 2021-02-10T11:44:47.432 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:47.419 7f1398ff9700 1 -- 172.21.2.112:0/2270201846 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f138c00cb90 con 0x7f139c1a4f80 2021-02-10T11:44:47.432 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:47.419 7f13a289b700 1 -- 172.21.2.112:0/2270201846 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f139c19d500 con 0x7f139c1a4f80 2021-02-10T11:44:47.432 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:47.419 7f1398ff9700 1 -- 172.21.2.112:0/2270201846 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f138c00c2f0 con 0x7f139c1a4f80 2021-02-10T11:44:47.432 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:47.420 7f1398ff9700 1 --2- 172.21.2.112:0/2270201846 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f13800186c0 0x7f138001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:47.432 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:47.420 7f1398ff9700 1 -- 172.21.2.112:0/2270201846 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f138c032b10 con 0x7f139c1a4f80 2021-02-10T11:44:47.432 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:47.420 7f139b7fe700 1 --2- 172.21.2.112:0/2270201846 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f13800186c0 0x7f138001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:47.433 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:47.420 7f139b7fe700 1 --2- 172.21.2.112:0/2270201846 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f13800186c0 0x7f138001abc0 crc :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:47.451 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:47.450 7f13a289b700 1 -- 172.21.2.112:0/2270201846 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f139c082990 con 0x7f139c1a4f80 2021-02-10T11:44:47.454 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:47.453 7f1398ff9700 1 -- 172.21.2.112:0/2270201846 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f138c00be30 con 0x7f139c1a4f80 2021-02-10T11:44:47.583 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:47.582 7f13a289b700 1 -- 172.21.2.112:0/2270201846 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f139c082990 con 0x7f139c1a4f80 2021-02-10T11:44:47.587 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:47.586 7f1398ff9700 1 -- 172.21.2.112:0/2270201846 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v87) v1 ==== 184+0+0 (secure 0 0 0) 0x7f138c00c2f0 con 0x7f139c1a4f80 2021-02-10T11:44:47.590 INFO:teuthology.orchestra.run.gibba012.stderr:updated caps for client.0 2021-02-10T11:44:47.592 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:47.591 7f13a289b700 1 -- 172.21.2.112:0/2270201846 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f13800186c0 msgr2=0x7f138001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:47.592 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:47.591 7f13a289b700 1 --2- 172.21.2.112:0/2270201846 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f13800186c0 0x7f138001abc0 crc :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:47.592 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:47.591 7f13a289b700 1 -- 172.21.2.112:0/2270201846 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f139c1a4f80 msgr2=0x7f139c0adac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:47.592 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:47.591 7f13a289b700 1 --2- 172.21.2.112:0/2270201846 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f139c1a4f80 0x7f139c0adac0 secure :-1 s=READY pgs=2659 cs=0 l=1 rev1=1 rx=0x7f138c0099a0 tx=0x7f138c00bcf0).stop 2021-02-10T11:44:47.592 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:47.591 7f13a289b700 1 -- 172.21.2.112:0/2270201846 shutdown_connections 2021-02-10T11:44:47.593 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:47.591 7f13a289b700 1 --2- 172.21.2.112:0/2270201846 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f13800186c0 0x7f138001abc0 unknown :-1 s=CLOSED pgs=1215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:47.593 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:47.591 7f13a289b700 1 --2- 172.21.2.112:0/2270201846 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f139c0ae0e0 0x7f139c199a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:47.593 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:47.591 7f13a289b700 1 --2- 172.21.2.112:0/2270201846 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f139c19a080 0x7f139c19a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:47.593 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:47.591 7f13a289b700 1 --2- 172.21.2.112:0/2270201846 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f139c1a4f80 0x7f139c0adac0 unknown :-1 s=CLOSED pgs=2659 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:47.593 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:47.591 7f13a289b700 1 -- 172.21.2.112:0/2270201846 >> 172.21.2.112:0/2270201846 conn(0x7f139c09a140 msgr2=0x7f139c090b40 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:47.593 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:47.591 7f13a289b700 1 -- 172.21.2.112:0/2270201846 shutdown_connections 2021-02-10T11:44:47.594 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:47.591 7f13a289b700 1 -- 172.21.2.112:0/2270201846 wait complete. 2021-02-10T11:44:47.601 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:44:48.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.023 7fe7452a1700 1 Processor -- start 2021-02-10T11:44:48.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.023 7fe7452a1700 1 -- start start 2021-02-10T11:44:48.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.023 7fe7452a1700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fe7400d1860 0x7fe7401b04b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:48.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.023 7fe7452a1700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe7401b0ad0 0x7fe740108930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:48.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.023 7fe7452a1700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fe7400d1c60 0x7fe740108f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:48.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.023 7fe7452a1700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7400fe1b0 con 0x7fe7400d1860 2021-02-10T11:44:48.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.023 7fe7452a1700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe740109a50 con 0x7fe7400d1c60 2021-02-10T11:44:48.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.023 7fe7452a1700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe740088070 con 0x7fe7401b0ad0 2021-02-10T11:44:48.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.024 7fe73f07d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fe7400d1860 0x7fe7401b04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:48.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.024 7fe73f07d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fe7400d1860 0x7fe7401b04b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:58208/0 (socket says 172.21.2.112:58208) 2021-02-10T11:44:48.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.024 7fe73f07d700 1 -- 172.21.2.112:0/4220400198 learned_addr learned my addr 172.21.2.112:0/4220400198 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:44:48.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.024 7fe73f87e700 1 --2- 172.21.2.112:0/4220400198 >> v2:172.21.2.120:3300/0 conn(0x7fe7400d1c60 0x7fe740108f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:48.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.024 7fe73e87c700 1 --2- 172.21.2.112:0/4220400198 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe7401b0ad0 0x7fe740108930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:48.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.024 7fe73f07d700 1 -- 172.21.2.112:0/4220400198 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe7401b0ad0 msgr2=0x7fe740108930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:48.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.024 7fe73f07d700 1 --2- 172.21.2.112:0/4220400198 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe7401b0ad0 0x7fe740108930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:48.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.024 7fe73f07d700 1 -- 172.21.2.112:0/4220400198 >> v2:172.21.2.120:3300/0 conn(0x7fe7400d1c60 msgr2=0x7fe740108f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:48.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.024 7fe73f07d700 1 --2- 172.21.2.112:0/4220400198 >> v2:172.21.2.120:3300/0 conn(0x7fe7400d1c60 0x7fe740108f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:48.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.024 7fe73f07d700 1 -- 172.21.2.112:0/4220400198 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe740109570 con 0x7fe7400d1860 2021-02-10T11:44:48.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.024 7fe73f07d700 1 --2- 172.21.2.112:0/4220400198 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe7400d1860 0x7fe7401b04b0 secure :-1 s=READY pgs=2660 cs=0 l=1 rev1=1 rx=0x7fe728006700 tx=0x7fe728009b80).ready entity=mon.0 client_cookie=457e2a3826142a11 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:48.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.024 7fe73e07b700 1 -- 172.21.2.112:0/4220400198 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe72800c0c0 con 0x7fe7400d1860 2021-02-10T11:44:48.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.024 7fe73e07b700 1 -- 172.21.2.112:0/4220400198 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe72800c340 con 0x7fe7400d1860 2021-02-10T11:44:48.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.024 7fe73e07b700 1 -- 172.21.2.112:0/4220400198 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe72800c860 con 0x7fe7400d1860 2021-02-10T11:44:48.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.024 7fe7452a1700 1 -- 172.21.2.112:0/4220400198 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe7400d1860 msgr2=0x7fe7401b04b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:48.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.024 7fe7452a1700 1 --2- 172.21.2.112:0/4220400198 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe7400d1860 0x7fe7401b04b0 secure :-1 s=READY pgs=2660 cs=0 l=1 rev1=1 rx=0x7fe728006700 tx=0x7fe728009b80).stop 2021-02-10T11:44:48.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.025 7fe7452a1700 1 -- 172.21.2.112:0/4220400198 shutdown_connections 2021-02-10T11:44:48.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.025 7fe7452a1700 1 --2- 172.21.2.112:0/4220400198 >> v2:172.21.2.120:3300/0 conn(0x7fe7400d1c60 0x7fe740108f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:48.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.025 7fe7452a1700 1 --2- 172.21.2.112:0/4220400198 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe7401b0ad0 0x7fe740108930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:48.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.025 7fe7452a1700 1 --2- 172.21.2.112:0/4220400198 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe7400d1860 0x7fe7401b04b0 unknown :-1 s=CLOSED pgs=2660 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:48.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.025 7fe7452a1700 1 -- 172.21.2.112:0/4220400198 >> 172.21.2.112:0/4220400198 conn(0x7fe7400ce410 msgr2=0x7fe7400d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:48.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.025 7fe7452a1700 1 -- 172.21.2.112:0/4220400198 shutdown_connections 2021-02-10T11:44:48.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.025 7fe7452a1700 1 -- 172.21.2.112:0/4220400198 wait complete. 2021-02-10T11:44:48.045 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.040 7fe7452a1700 1 Processor -- start 2021-02-10T11:44:48.045 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.040 7fe7452a1700 1 -- start start 2021-02-10T11:44:48.045 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.040 7fe7452a1700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe7400b5d30 0x7fe7400c2910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:48.045 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.040 7fe7452a1700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe7400ae630 0x7fe7400aea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:48.045 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.040 7fe7452a1700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe7400bee80 0x7fe7400bf280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:48.045 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.040 7fe7452a1700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7400c2f30 con 0x7fe7400ae630 2021-02-10T11:44:48.045 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.040 7fe7452a1700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7400c2390 con 0x7fe7400bee80 2021-02-10T11:44:48.045 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.040 7fe7452a1700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7400b6130 con 0x7fe7400b5d30 2021-02-10T11:44:48.045 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.040 7fe73e87c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe7400ae630 0x7fe7400aea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:48.045 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.040 7fe73e87c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe7400ae630 0x7fe7400aea30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:58214/0 (socket says 172.21.2.112:58214) 2021-02-10T11:44:48.045 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.040 7fe73e87c700 1 -- 172.21.2.112:0/193517235 learned_addr learned my addr 172.21.2.112:0/193517235 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:44:48.045 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.040 7fe73f87e700 1 --2- 172.21.2.112:0/193517235 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe7400bee80 0x7fe7400bf280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:48.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.040 7fe73f07d700 1 --2- 172.21.2.112:0/193517235 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe7400b5d30 0x7fe7400c2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:48.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.041 7fe73e87c700 1 -- 172.21.2.112:0/193517235 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe7400b5d30 msgr2=0x7fe7400c2910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:48.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.041 7fe73e87c700 1 --2- 172.21.2.112:0/193517235 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe7400b5d30 0x7fe7400c2910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:48.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.041 7fe73e87c700 1 -- 172.21.2.112:0/193517235 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe7400bee80 msgr2=0x7fe7400bf280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:48.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.041 7fe73e87c700 1 --2- 172.21.2.112:0/193517235 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe7400bee80 0x7fe7400bf280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:48.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.041 7fe73e87c700 1 -- 172.21.2.112:0/193517235 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7400bf8a0 con 0x7fe7400ae630 2021-02-10T11:44:48.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.041 7fe73e87c700 1 --2- 172.21.2.112:0/193517235 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe7400ae630 0x7fe7400aea30 secure :-1 s=READY pgs=2661 cs=0 l=1 rev1=1 rx=0x7fe730006f60 tx=0x7fe73000b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:48.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.041 7fe737fff700 1 -- 172.21.2.112:0/193517235 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe73000bd60 con 0x7fe7400ae630 2021-02-10T11:44:48.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.041 7fe737fff700 1 -- 172.21.2.112:0/193517235 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe730000c90 con 0x7fe7400ae630 2021-02-10T11:44:48.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.041 7fe737fff700 1 -- 172.21.2.112:0/193517235 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe7300022b0 con 0x7fe7400ae630 2021-02-10T11:44:48.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.041 7fe7452a1700 1 -- 172.21.2.112:0/193517235 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7400bf8a0 con 0x7fe7400ae630 2021-02-10T11:44:48.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.041 7fe7452a1700 1 -- 172.21.2.112:0/193517235 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7400bfe30 con 0x7fe7400ae630 2021-02-10T11:44:48.047 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.042 7fe737fff700 1 -- 172.21.2.112:0/193517235 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe730006b00 con 0x7fe7400ae630 2021-02-10T11:44:48.047 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.042 7fe737fff700 1 --2- 172.21.2.112:0/193517235 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe7240186c0 0x7fe72401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:48.047 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.042 7fe737fff700 1 -- 172.21.2.112:0/193517235 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fe730032570 con 0x7fe7400ae630 2021-02-10T11:44:48.047 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.042 7fe73f07d700 1 --2- 172.21.2.112:0/193517235 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe7240186c0 0x7fe72401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:48.047 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.043 7fe73f07d700 1 --2- 172.21.2.112:0/193517235 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe7240186c0 0x7fe72401abc0 crc :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:48.074 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.073 7fe7452a1700 1 -- 172.21.2.112:0/193517235 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7400fe1b0 con 0x7fe7400ae630 2021-02-10T11:44:48.077 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.076 7fe737fff700 1 -- 172.21.2.112:0/193517235 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe730030d20 con 0x7fe7400ae630 2021-02-10T11:44:48.207 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.206 7fe7452a1700 1 -- 172.21.2.112:0/193517235 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fe7400fe1b0 con 0x7fe7400ae630 2021-02-10T11:44:48.211 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.210 7fe737fff700 1 -- 172.21.2.112:0/193517235 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v88) v1 ==== 184+0+0 (secure 0 0 0) 0x7fe730006b00 con 0x7fe7400ae630 2021-02-10T11:44:48.214 INFO:teuthology.orchestra.run.gibba012.stderr:updated caps for client.1 2021-02-10T11:44:48.216 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.215 7fe7452a1700 1 -- 172.21.2.112:0/193517235 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe7240186c0 msgr2=0x7fe72401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:48.216 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.215 7fe7452a1700 1 --2- 172.21.2.112:0/193517235 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe7240186c0 0x7fe72401abc0 crc :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:48.216 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.215 7fe7452a1700 1 -- 172.21.2.112:0/193517235 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe7400ae630 msgr2=0x7fe7400aea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:48.216 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.215 7fe7452a1700 1 --2- 172.21.2.112:0/193517235 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe7400ae630 0x7fe7400aea30 secure :-1 s=READY pgs=2661 cs=0 l=1 rev1=1 rx=0x7fe730006f60 tx=0x7fe73000b970).stop 2021-02-10T11:44:48.216 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.215 7fe7452a1700 1 -- 172.21.2.112:0/193517235 shutdown_connections 2021-02-10T11:44:48.216 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.215 7fe7452a1700 1 --2- 172.21.2.112:0/193517235 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe7240186c0 0x7fe72401abc0 unknown :-1 s=CLOSED pgs=1216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:48.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.215 7fe7452a1700 1 --2- 172.21.2.112:0/193517235 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe7400bee80 0x7fe7400bf280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:48.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.215 7fe7452a1700 1 --2- 172.21.2.112:0/193517235 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe7400ae630 0x7fe7400aea30 unknown :-1 s=CLOSED pgs=2661 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:48.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.215 7fe7452a1700 1 --2- 172.21.2.112:0/193517235 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe7400b5d30 0x7fe7400c2910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:48.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.215 7fe7452a1700 1 -- 172.21.2.112:0/193517235 >> 172.21.2.112:0/193517235 conn(0x7fe7400874f0 msgr2=0x7fe74009e240 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:48.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.215 7fe7452a1700 1 -- 172.21.2.112:0/193517235 shutdown_connections 2021-02-10T11:44:48.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.215 7fe7452a1700 1 -- 172.21.2.112:0/193517235 wait complete. 2021-02-10T11:44:48.224 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.2 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:44:48.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.637 7f4a60e5c700 1 Processor -- start 2021-02-10T11:44:48.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.637 7f4a60e5c700 1 -- start start 2021-02-10T11:44:48.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.638 7f4a60e5c700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f4a5c0d1c80 0x7f4a5c0cef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:48.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.638 7f4a60e5c700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f4a5c0cf590 0x7f4a5c0cf990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:48.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.638 7f4a60e5c700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a5c0fe1d0 con 0x7f4a5c0d1880 2021-02-10T11:44:48.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.638 7f4a60e5c700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a5c1b0cc0 con 0x7f4a5c0d1c80 2021-02-10T11:44:48.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.638 7f4a60e5c700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a5c109a70 con 0x7f4a5c0cf590 2021-02-10T11:44:48.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.638 7f4a5a312700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f4a5c0d1c80 0x7f4a5c0cef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:48.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.638 7f4a5a312700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f4a5c0d1c80 0x7f4a5c0cef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:58222/0 (socket says 172.21.2.112:58222) 2021-02-10T11:44:48.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.638 7f4a5a312700 1 -- 172.21.2.112:0/233175872 learned_addr learned my addr 172.21.2.112:0/233175872 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:44:48.657 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.638 7f4a5a312700 1 -- 172.21.2.112:0/233175872 >> v2:172.21.2.120:3300/0 conn(0x7f4a5c0cf590 msgr2=0x7f4a5c0cf990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:48.657 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.638 7f4a5a312700 1 --2- 172.21.2.112:0/233175872 >> v2:172.21.2.120:3300/0 conn(0x7f4a5c0cf590 0x7f4a5c0cf990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:48.657 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.638 7f4a5a312700 1 -- 172.21.2.112:0/233175872 >> v1:172.21.2.120:6789/0 conn(0x7f4a5c0d1880 legacy=0x7f4a5c1b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:48.657 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.638 7f4a5a312700 1 -- 172.21.2.112:0/233175872 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a5c088070 con 0x7f4a5c0d1c80 2021-02-10T11:44:48.657 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.639 7f4a5a312700 1 --2- 172.21.2.112:0/233175872 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4a5c0d1c80 0x7f4a5c0cef70 secure :-1 s=READY pgs=2662 cs=0 l=1 rev1=1 rx=0x7f4a50006700 tx=0x7f4a50009b80).ready entity=mon.0 client_cookie=95948d7ec15b48b0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:48.657 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.639 7f4a59b11700 1 -- 172.21.2.112:0/233175872 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4a5000c0c0 con 0x7f4a5c0d1c80 2021-02-10T11:44:48.657 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.639 7f4a59b11700 1 -- 172.21.2.112:0/233175872 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a5000c440 con 0x7f4a5c0d1c80 2021-02-10T11:44:48.657 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.639 7f4a59b11700 1 -- 172.21.2.112:0/233175872 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4a5000c920 con 0x7f4a5c0d1c80 2021-02-10T11:44:48.657 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.639 7f4a60e5c700 1 -- 172.21.2.112:0/233175872 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4a5c0d1c80 msgr2=0x7f4a5c0cef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:48.657 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.639 7f4a60e5c700 1 --2- 172.21.2.112:0/233175872 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4a5c0d1c80 0x7f4a5c0cef70 secure :-1 s=READY pgs=2662 cs=0 l=1 rev1=1 rx=0x7f4a50006700 tx=0x7f4a50009b80).stop 2021-02-10T11:44:48.657 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.640 7f4a60e5c700 1 -- 172.21.2.112:0/233175872 shutdown_connections 2021-02-10T11:44:48.657 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.640 7f4a60e5c700 1 --2- 172.21.2.112:0/233175872 >> v2:172.21.2.120:3300/0 conn(0x7f4a5c0cf590 0x7f4a5c0cf990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:48.657 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.640 7f4a60e5c700 1 --2- 172.21.2.112:0/233175872 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4a5c0d1c80 0x7f4a5c0cef70 unknown :-1 s=CLOSED pgs=2662 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:48.658 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.640 7f4a60e5c700 1 -- 172.21.2.112:0/233175872 >> 172.21.2.112:0/233175872 conn(0x7f4a5c0ce430 msgr2=0x7f4a5c0d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:48.658 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.640 7f4a60e5c700 1 -- 172.21.2.112:0/233175872 shutdown_connections 2021-02-10T11:44:48.658 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.640 7f4a60e5c700 1 -- 172.21.2.112:0/233175872 wait complete. 2021-02-10T11:44:48.658 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.655 7f4a60e5c700 1 Processor -- start 2021-02-10T11:44:48.658 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.655 7f4a60e5c700 1 -- start start 2021-02-10T11:44:48.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.655 7f4a60e5c700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4a5c096170 0x7f4a5c1a7810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:48.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.655 7f4a60e5c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4a5c1a7e30 0x7f4a5c1a8230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:48.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.655 7f4a60e5c700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4a5c1ab260 0x7f4a5c1ab660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:48.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.655 7f4a60e5c700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a5c096570 con 0x7f4a5c1a7e30 2021-02-10T11:44:48.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.655 7f4a60e5c700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a5c1a8850 con 0x7f4a5c1ab260 2021-02-10T11:44:48.660 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.655 7f4a60e5c700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a5c1abc80 con 0x7f4a5c096170 2021-02-10T11:44:48.660 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.655 7f4a5a312700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4a5c1a7e30 0x7f4a5c1a8230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:48.660 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.655 7f4a5a312700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4a5c1a7e30 0x7f4a5c1a8230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:58224/0 (socket says 172.21.2.112:58224) 2021-02-10T11:44:48.660 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.655 7f4a5a312700 1 -- 172.21.2.112:0/2437528174 learned_addr learned my addr 172.21.2.112:0/2437528174 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:44:48.660 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.655 7f4a5a312700 1 -- 172.21.2.112:0/2437528174 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4a5c096170 msgr2=0x7f4a5c1a7810 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:44:48.660 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.655 7f4a5a312700 1 --2- 172.21.2.112:0/2437528174 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4a5c096170 0x7f4a5c1a7810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:48.660 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.655 7f4a5a312700 1 -- 172.21.2.112:0/2437528174 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4a5c1ab260 msgr2=0x7f4a5c1ab660 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:44:48.660 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.655 7f4a5a312700 1 --2- 172.21.2.112:0/2437528174 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4a5c1ab260 0x7f4a5c1ab660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:48.660 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.656 7f4a5a312700 1 -- 172.21.2.112:0/2437528174 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a5c1abea0 con 0x7f4a5c1a7e30 2021-02-10T11:44:48.661 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.656 7f4a5a312700 1 --2- 172.21.2.112:0/2437528174 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4a5c1a7e30 0x7f4a5c1a8230 secure :-1 s=READY pgs=2663 cs=0 l=1 rev1=1 rx=0x7f4a5000b580 tx=0x7f4a5000b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:48.661 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.656 7f4a4b7fe700 1 -- 172.21.2.112:0/2437528174 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4a5000bd10 con 0x7f4a5c1a7e30 2021-02-10T11:44:48.661 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.656 7f4a4b7fe700 1 -- 172.21.2.112:0/2437528174 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a50002030 con 0x7f4a5c1a7e30 2021-02-10T11:44:48.661 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.656 7f4a60e5c700 1 -- 172.21.2.112:0/2437528174 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a5c1abea0 con 0x7f4a5c1a7e30 2021-02-10T11:44:48.661 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.656 7f4a4b7fe700 1 -- 172.21.2.112:0/2437528174 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4a50002400 con 0x7f4a5c1a7e30 2021-02-10T11:44:48.661 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.656 7f4a60e5c700 1 -- 172.21.2.112:0/2437528174 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a5c0826b0 con 0x7f4a5c1a7e30 2021-02-10T11:44:48.661 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.656 7f4a4b7fe700 1 -- 172.21.2.112:0/2437528174 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4a50000c90 con 0x7f4a5c1a7e30 2021-02-10T11:44:48.661 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.657 7f4a4b7fe700 1 --2- 172.21.2.112:0/2437528174 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4a40018730 0x7f4a4001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:48.661 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.657 7f4a4b7fe700 1 -- 172.21.2.112:0/2437528174 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f4a50032630 con 0x7f4a5c1a7e30 2021-02-10T11:44:48.661 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.657 7f4a5ab13700 1 --2- 172.21.2.112:0/2437528174 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4a40018730 0x7f4a4001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:48.661 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.658 7f4a5ab13700 1 --2- 172.21.2.112:0/2437528174 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4a40018730 0x7f4a4001ac10 crc :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:48.689 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.688 7f4a60e5c700 1 -- 172.21.2.112:0/2437528174 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a5c1aad30 con 0x7f4a5c1a7e30 2021-02-10T11:44:48.691 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.690 7f4a4b7fe700 1 -- 172.21.2.112:0/2437528174 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4a500009b0 con 0x7f4a5c1a7e30 2021-02-10T11:44:48.823 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.822 7f4a60e5c700 1 -- 172.21.2.112:0/2437528174 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f4a5c1aad30 con 0x7f4a5c1a7e30 2021-02-10T11:44:48.825 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.824 7f4a4b7fe700 1 -- 172.21.2.112:0/2437528174 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.2 v89) v1 ==== 184+0+0 (secure 0 0 0) 0x7f4a50000c90 con 0x7f4a5c1a7e30 2021-02-10T11:44:48.826 INFO:teuthology.orchestra.run.gibba012.stderr:updated caps for client.2 2021-02-10T11:44:48.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.826 7f4a60e5c700 1 -- 172.21.2.112:0/2437528174 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4a40018730 msgr2=0x7f4a4001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:48.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.826 7f4a60e5c700 1 --2- 172.21.2.112:0/2437528174 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4a40018730 0x7f4a4001ac10 crc :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:48.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.826 7f4a60e5c700 1 -- 172.21.2.112:0/2437528174 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4a5c1a7e30 msgr2=0x7f4a5c1a8230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:48.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.826 7f4a60e5c700 1 --2- 172.21.2.112:0/2437528174 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4a5c1a7e30 0x7f4a5c1a8230 secure :-1 s=READY pgs=2663 cs=0 l=1 rev1=1 rx=0x7f4a5000b580 tx=0x7f4a5000b920).stop 2021-02-10T11:44:48.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.826 7f4a60e5c700 1 -- 172.21.2.112:0/2437528174 shutdown_connections 2021-02-10T11:44:48.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.826 7f4a60e5c700 1 --2- 172.21.2.112:0/2437528174 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4a40018730 0x7f4a4001ac10 unknown :-1 s=CLOSED pgs=1217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:48.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.826 7f4a60e5c700 1 --2- 172.21.2.112:0/2437528174 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4a5c1ab260 0x7f4a5c1ab660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:48.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.826 7f4a60e5c700 1 --2- 172.21.2.112:0/2437528174 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4a5c1a7e30 0x7f4a5c1a8230 unknown :-1 s=CLOSED pgs=2663 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:48.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.826 7f4a60e5c700 1 --2- 172.21.2.112:0/2437528174 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4a5c096170 0x7f4a5c1a7810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:48.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.826 7f4a60e5c700 1 -- 172.21.2.112:0/2437528174 >> 172.21.2.112:0/2437528174 conn(0x7f4a5c19e660 msgr2=0x7f4a5c082bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:48.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.826 7f4a60e5c700 1 -- 172.21.2.112:0/2437528174 shutdown_connections 2021-02-10T11:44:48.829 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:48.826 7f4a60e5c700 1 -- 172.21.2.112:0/2437528174 wait complete. 2021-02-10T11:44:48.836 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.3 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:44:49.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.250 7f14d7556700 1 Processor -- start 2021-02-10T11:44:49.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.251 7f14d7556700 1 -- start start 2021-02-10T11:44:49.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.251 7f14d7556700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f14d00d1860 0x7f14d01b04b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:49.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.251 7f14d7556700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14d00fe1b0 con 0x7f14d00cff60 2021-02-10T11:44:49.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.251 7f14d7556700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14d0109a50 con 0x7f14d01b0ad0 2021-02-10T11:44:49.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.251 7f14d7556700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14d0088070 con 0x7f14d00d1860 2021-02-10T11:44:49.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.251 7f14d5b73700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f14d00cff60 0x7f14d0109120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:57692/0 (socket says 172.21.2.112:57692) 2021-02-10T11:44:49.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.251 7f14d5b73700 1 -- 172.21.2.112:0/3899859453 learned_addr learned my addr 172.21.2.112:0/3899859453 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:44:49.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.252 7f14c7fff700 1 -- 172.21.2.112:0/3899859453 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2749073334 0 0) 0x7f14c8001500 con 0x7f14d00cff60 2021-02-10T11:44:49.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.252 7f14c7fff700 1 -- 172.21.2.112:0/3899859453 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14bc002fe0 con 0x7f14d00cff60 2021-02-10T11:44:49.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.252 7f14d5372700 1 --2- 172.21.2.112:0/3899859453 >> v2:172.21.2.120:3300/0 conn(0x7f14d00d1860 0x7f14d01b04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:49.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.252 7f14c7fff700 1 -- 172.21.2.112:0/3899859453 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2816762845 0 0) 0x7f14c8001500 con 0x7f14d00cff60 2021-02-10T11:44:49.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.252 7f14c7fff700 1 -- 172.21.2.112:0/3899859453 >> v2:172.21.2.120:3300/0 conn(0x7f14d00d1860 msgr2=0x7f14d01b04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:49.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.252 7f14c7fff700 1 --2- 172.21.2.112:0/3899859453 >> v2:172.21.2.120:3300/0 conn(0x7f14d00d1860 0x7f14d01b04b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:49.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.252 7f14c7fff700 1 -- 172.21.2.112:0/3899859453 >> v1:172.21.2.120:6789/0 conn(0x7f14d01b0ad0 legacy=0x7f14d0108930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:49.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.252 7f14c7fff700 1 -- 172.21.2.112:0/3899859453 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14d0088260 con 0x7f14d00cff60 2021-02-10T11:44:49.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.252 7f14c7fff700 1 -- 172.21.2.112:0/3899859453 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f14c8001a00 con 0x7f14d00cff60 2021-02-10T11:44:49.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.252 7f14c7fff700 1 -- 172.21.2.112:0/3899859453 >> v1:172.21.2.112:6789/0 conn(0x7f14d00cff60 legacy=0x7f14d0109120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:49.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.252 7f14c7fff700 1 --2- 172.21.2.112:0/3899859453 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f14bc003ac0 0x7f14bc005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:49.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.252 7f14c7fff700 1 --2- 172.21.2.112:0/3899859453 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f14bc006790 0x7f14bc008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:49.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.252 7f14c7fff700 1 --2- 172.21.2.112:0/3899859453 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f14bc009590 0x7f14bc00bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:49.272 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.252 7f14c7fff700 1 -- 172.21.2.112:0/3899859453 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14bc00c300 con 0x7f14bc006790 2021-02-10T11:44:49.272 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.252 7f14c7fff700 1 -- 172.21.2.112:0/3899859453 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14bc00c520 con 0x7f14bc009590 2021-02-10T11:44:49.272 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.252 7f14c7fff700 1 -- 172.21.2.112:0/3899859453 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14bc050750 con 0x7f14bc003ac0 2021-02-10T11:44:49.272 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.252 7f14d4b71700 1 --2- 172.21.2.112:0/3899859453 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f14bc006790 0x7f14bc008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:49.272 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.253 7f14d5b73700 1 --2- 172.21.2.112:0/3899859453 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f14bc009590 0x7f14bc00bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:49.272 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.253 7f14d4b71700 1 -- 172.21.2.112:0/3899859453 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f14bc003ac0 msgr2=0x7f14bc005ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:49.272 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.253 7f14d4b71700 1 --2- 172.21.2.112:0/3899859453 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f14bc003ac0 0x7f14bc005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:49.272 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.253 7f14d4b71700 1 -- 172.21.2.112:0/3899859453 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f14bc009590 msgr2=0x7f14bc00bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:49.272 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.253 7f14d4b71700 1 --2- 172.21.2.112:0/3899859453 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f14bc009590 0x7f14bc00bab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:49.272 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.253 7f14d4b71700 1 -- 172.21.2.112:0/3899859453 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14bc0509b0 con 0x7f14bc006790 2021-02-10T11:44:49.272 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.253 7f14d5b73700 1 -- 172.21.2.112:0/3899859453 reap_dead start 2021-02-10T11:44:49.272 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.253 7f14d4b71700 1 --2- 172.21.2.112:0/3899859453 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f14bc006790 0x7f14bc008c90 secure :-1 s=READY pgs=2665 cs=0 l=1 rev1=1 rx=0x7f14c0006ac0 tx=0x7f14c0009af0).ready entity=mon.0 client_cookie=77ba80ad0bc37c10 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:49.272 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.253 7f14c7fff700 1 -- 172.21.2.112:0/3899859453 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f14c0007470 con 0x7f14bc006790 2021-02-10T11:44:49.273 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.253 7f14c7fff700 1 -- 172.21.2.112:0/3899859453 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14c0007770 con 0x7f14bc006790 2021-02-10T11:44:49.273 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.253 7f14c7fff700 1 -- 172.21.2.112:0/3899859453 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f14c0007c90 con 0x7f14bc006790 2021-02-10T11:44:49.273 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.253 7f14d7556700 1 -- 172.21.2.112:0/3899859453 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f14bc006790 msgr2=0x7f14bc008c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:49.273 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.253 7f14d7556700 1 --2- 172.21.2.112:0/3899859453 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f14bc006790 0x7f14bc008c90 secure :-1 s=READY pgs=2665 cs=0 l=1 rev1=1 rx=0x7f14c0006ac0 tx=0x7f14c0009af0).stop 2021-02-10T11:44:49.273 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.254 7f14d7556700 1 -- 172.21.2.112:0/3899859453 shutdown_connections 2021-02-10T11:44:49.273 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.254 7f14d7556700 1 --2- 172.21.2.112:0/3899859453 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f14bc006790 0x7f14bc008c90 unknown :-1 s=CLOSED pgs=2665 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:49.273 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.254 7f14d7556700 1 -- 172.21.2.112:0/3899859453 >> 172.21.2.112:0/3899859453 conn(0x7f14d00ce410 msgr2=0x7f14d00d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:49.273 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.254 7f14d7556700 1 -- 172.21.2.112:0/3899859453 shutdown_connections 2021-02-10T11:44:49.273 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.254 7f14d7556700 1 -- 172.21.2.112:0/3899859453 wait complete. 2021-02-10T11:44:49.273 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.269 7f14d7556700 1 Processor -- start 2021-02-10T11:44:49.273 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.269 7f14d7556700 1 -- start start 2021-02-10T11:44:49.274 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.269 7f14d7556700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f14d00b5fb0 0x7f14d00a81f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:49.275 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.269 7f14d7556700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f14d00a8810 0x7f14d00a2540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:49.275 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.269 7f14d7556700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f14d00a8c10 0x7f14d00a2b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:49.275 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.269 7f14d7556700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14d00b63b0 con 0x7f14d00b5fb0 2021-02-10T11:44:49.275 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.269 7f14d7556700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14d00a9210 con 0x7f14d00a8c10 2021-02-10T11:44:49.275 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.269 7f14d7556700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14d00b2650 con 0x7f14d00a8810 2021-02-10T11:44:49.275 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.269 7f14d5372700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f14d00b5fb0 0x7f14d00a81f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:49.275 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.269 7f14d5372700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f14d00b5fb0 0x7f14d00a81f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:58244/0 (socket says 172.21.2.112:58244) 2021-02-10T11:44:49.275 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.269 7f14d5372700 1 -- 172.21.2.112:0/928029003 learned_addr learned my addr 172.21.2.112:0/928029003 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:44:49.275 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.270 7f14d5372700 1 -- 172.21.2.112:0/928029003 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f14d00a8810 msgr2=0x7f14d00a2540 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:44:49.275 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.270 7f14d5372700 1 --2- 172.21.2.112:0/928029003 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f14d00a8810 0x7f14d00a2540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:49.275 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.270 7f14d5372700 1 -- 172.21.2.112:0/928029003 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f14d00a8c10 msgr2=0x7f14d00a2b60 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:44:49.275 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.270 7f14d5372700 1 --2- 172.21.2.112:0/928029003 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f14d00a8c10 0x7f14d00a2b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:49.275 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.270 7f14d5372700 1 -- 172.21.2.112:0/928029003 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14d00a3180 con 0x7f14d00b5fb0 2021-02-10T11:44:49.276 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.270 7f14d5372700 1 --2- 172.21.2.112:0/928029003 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f14d00b5fb0 0x7f14d00a81f0 secure :-1 s=READY pgs=2666 cs=0 l=1 rev1=1 rx=0x7f14cc006f40 tx=0x7f14cc00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:49.276 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.270 7f14c5ffb700 1 -- 172.21.2.112:0/928029003 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f14cc006bb0 con 0x7f14d00b5fb0 2021-02-10T11:44:49.276 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.270 7f14c5ffb700 1 -- 172.21.2.112:0/928029003 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14cc0063c0 con 0x7f14d00b5fb0 2021-02-10T11:44:49.276 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.270 7f14c5ffb700 1 -- 172.21.2.112:0/928029003 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f14cc0052e0 con 0x7f14d00b5fb0 2021-02-10T11:44:49.276 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.270 7f14d7556700 1 -- 172.21.2.112:0/928029003 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14d00a3180 con 0x7f14d00b5fb0 2021-02-10T11:44:49.276 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.270 7f14d7556700 1 -- 172.21.2.112:0/928029003 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14d00a3710 con 0x7f14d00b5fb0 2021-02-10T11:44:49.276 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.271 7f14c5ffb700 1 -- 172.21.2.112:0/928029003 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f14cc006b50 con 0x7f14d00b5fb0 2021-02-10T11:44:49.276 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.271 7f14c5ffb700 1 --2- 172.21.2.112:0/928029003 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f14b00186c0 0x7f14b001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:49.276 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.271 7f14c5ffb700 1 -- 172.21.2.112:0/928029003 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f14cc032570 con 0x7f14d00b5fb0 2021-02-10T11:44:49.276 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.271 7f14d4b71700 1 --2- 172.21.2.112:0/928029003 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f14b00186c0 0x7f14b001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:49.276 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.272 7f14d4b71700 1 --2- 172.21.2.112:0/928029003 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f14b00186c0 0x7f14b001abc0 crc :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:49.303 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.301 7f14d7556700 1 -- 172.21.2.112:0/928029003 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14d00fe1b0 con 0x7f14d00b5fb0 2021-02-10T11:44:49.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.304 7f14c5ffb700 1 -- 172.21.2.112:0/928029003 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f14cc022ec0 con 0x7f14d00b5fb0 2021-02-10T11:44:49.436 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.435 7f14d7556700 1 -- 172.21.2.112:0/928029003 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f14d00b63b0 con 0x7f14d00b5fb0 2021-02-10T11:44:49.439 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.438 7f14c5ffb700 1 -- 172.21.2.112:0/928029003 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.3 v90) v1 ==== 184+0+0 (secure 0 0 0) 0x7f14cc006b50 con 0x7f14d00b5fb0 2021-02-10T11:44:49.443 INFO:teuthology.orchestra.run.gibba012.stderr:updated caps for client.3 2021-02-10T11:44:49.444 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.443 7f14d7556700 1 -- 172.21.2.112:0/928029003 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f14b00186c0 msgr2=0x7f14b001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:49.444 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.443 7f14d7556700 1 --2- 172.21.2.112:0/928029003 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f14b00186c0 0x7f14b001abc0 crc :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:49.444 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.443 7f14d7556700 1 -- 172.21.2.112:0/928029003 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f14d00b5fb0 msgr2=0x7f14d00a81f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:49.445 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.443 7f14d7556700 1 --2- 172.21.2.112:0/928029003 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f14d00b5fb0 0x7f14d00a81f0 secure :-1 s=READY pgs=2666 cs=0 l=1 rev1=1 rx=0x7f14cc006f40 tx=0x7f14cc00b990).stop 2021-02-10T11:44:49.445 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.444 7f14d7556700 1 -- 172.21.2.112:0/928029003 shutdown_connections 2021-02-10T11:44:49.445 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.444 7f14d7556700 1 --2- 172.21.2.112:0/928029003 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f14b00186c0 0x7f14b001abc0 unknown :-1 s=CLOSED pgs=1218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:49.445 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.444 7f14d7556700 1 --2- 172.21.2.112:0/928029003 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f14d00a8c10 0x7f14d00a2b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:49.445 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.444 7f14d7556700 1 --2- 172.21.2.112:0/928029003 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f14d00a8810 0x7f14d00a2540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:49.445 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.444 7f14d7556700 1 --2- 172.21.2.112:0/928029003 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f14d00b5fb0 0x7f14d00a81f0 unknown :-1 s=CLOSED pgs=2666 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:49.445 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.444 7f14d7556700 1 -- 172.21.2.112:0/928029003 >> 172.21.2.112:0/928029003 conn(0x7f14d009e660 msgr2=0x7f14d00a9550 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:49.446 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.444 7f14d7556700 1 -- 172.21.2.112:0/928029003 shutdown_connections 2021-02-10T11:44:49.446 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.444 7f14d7556700 1 -- 172.21.2.112:0/928029003 wait complete. 2021-02-10T11:44:49.453 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:44:49.880 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.862 7f5a86bd0700 1 Processor -- start 2021-02-10T11:44:49.880 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.862 7f5a86bd0700 1 -- start start 2021-02-10T11:44:49.880 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.862 7f5a86bd0700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f5a800d1870 0x7f5a801b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:49.880 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.862 7f5a86bd0700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5a801b0a10 0x7f5a800cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:49.880 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.862 7f5a86bd0700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a800f0b10 con 0x7f5a800d1c70 2021-02-10T11:44:49.880 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.862 7f5a86bd0700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a800e5ed0 con 0x7f5a800d1870 2021-02-10T11:44:49.881 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.862 7f5a86bd0700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a800f0650 con 0x7f5a801b0a10 2021-02-10T11:44:49.881 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.863 7f5a7ffff700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5a801b0a10 0x7f5a800cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:49.881 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.863 7f5a849ec700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f5a800d1870 0x7f5a801b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:49.881 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.863 7f5a851ed700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7f5a800d1c70 0x7f5a800cf950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:51270/0 (socket says 172.21.2.112:51270) 2021-02-10T11:44:49.881 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.863 7f5a851ed700 1 -- 172.21.2.112:0/3518286140 learned_addr learned my addr 172.21.2.112:0/3518286140 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:44:49.881 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.863 7f5a7f7fe700 1 -- 172.21.2.112:0/3518286140 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3912686806 0 0) 0x7f5a78001440 con 0x7f5a800d1c70 2021-02-10T11:44:49.881 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.863 7f5a7f7fe700 1 -- 172.21.2.112:0/3518286140 --> v1:172.21.2.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a6c002fe0 con 0x7f5a800d1c70 2021-02-10T11:44:49.881 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.864 7f5a849ec700 1 -- 172.21.2.112:0/3518286140 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5a801b0a10 msgr2=0x7f5a800cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:49.881 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.864 7f5a849ec700 1 --2- 172.21.2.112:0/3518286140 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5a801b0a10 0x7f5a800cf330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:49.881 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.864 7f5a849ec700 1 -- 172.21.2.112:0/3518286140 >> v1:172.21.2.120:6789/0 conn(0x7f5a800d1c70 legacy=0x7f5a800cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:49.881 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.864 7f5a849ec700 1 -- 172.21.2.112:0/3518286140 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a800f0870 con 0x7f5a800d1870 2021-02-10T11:44:49.881 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.864 7f5a849ec700 1 --2- 172.21.2.112:0/3518286140 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5a800d1870 0x7f5a801b03f0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 rx=0x7f5a68006650 tx=0x7f5a68009a80).ready entity=mon.1 client_cookie=d3fc728675975850 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:49.881 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.864 7f5a7f7fe700 1 -- 172.21.2.112:0/3518286140 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5a6800c0c0 con 0x7f5a800d1870 2021-02-10T11:44:49.882 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.864 7f5a7f7fe700 1 -- 172.21.2.112:0/3518286140 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a6800c340 con 0x7f5a800d1870 2021-02-10T11:44:49.882 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.864 7f5a7f7fe700 1 -- 172.21.2.112:0/3518286140 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5a680009b0 con 0x7f5a800d1870 2021-02-10T11:44:49.882 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.864 7f5a86bd0700 1 -- 172.21.2.112:0/3518286140 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5a800d1870 msgr2=0x7f5a801b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:49.882 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.864 7f5a86bd0700 1 --2- 172.21.2.112:0/3518286140 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5a800d1870 0x7f5a801b03f0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 rx=0x7f5a68006650 tx=0x7f5a68009a80).stop 2021-02-10T11:44:49.882 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.865 7f5a86bd0700 1 -- 172.21.2.112:0/3518286140 shutdown_connections 2021-02-10T11:44:49.882 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.865 7f5a86bd0700 1 --2- 172.21.2.112:0/3518286140 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5a801b0a10 0x7f5a800cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:49.882 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.865 7f5a86bd0700 1 --2- 172.21.2.112:0/3518286140 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5a800d1870 0x7f5a801b03f0 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:49.882 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.865 7f5a86bd0700 1 -- 172.21.2.112:0/3518286140 >> 172.21.2.112:0/3518286140 conn(0x7f5a800ce450 msgr2=0x7f5a800d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:49.882 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.865 7f5a86bd0700 1 -- 172.21.2.112:0/3518286140 shutdown_connections 2021-02-10T11:44:49.882 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.865 7f5a86bd0700 1 -- 172.21.2.112:0/3518286140 wait complete. 2021-02-10T11:44:49.884 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.879 7f5a86bd0700 1 Processor -- start 2021-02-10T11:44:49.884 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.879 7f5a86bd0700 1 -- start start 2021-02-10T11:44:49.884 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.880 7f5a86bd0700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5a800ada90 0x7f5a800ade90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:49.884 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.880 7f5a86bd0700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5a80099d20 0x7f5a8009a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:49.884 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.880 7f5a86bd0700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5a8009a740 0x7f5a80097bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:49.884 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.880 7f5a86bd0700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a800997a0 con 0x7f5a80099d20 2021-02-10T11:44:49.884 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.880 7f5a86bd0700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a8018f0c0 con 0x7f5a8009a740 2021-02-10T11:44:49.884 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.880 7f5a86bd0700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a8009ab40 con 0x7f5a800ada90 2021-02-10T11:44:49.884 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.880 7f5a7ffff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5a80099d20 0x7f5a8009a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:49.884 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.880 7f5a7ffff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5a80099d20 0x7f5a8009a120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:58256/0 (socket says 172.21.2.112:58256) 2021-02-10T11:44:49.885 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.880 7f5a7ffff700 1 -- 172.21.2.112:0/2265466291 learned_addr learned my addr 172.21.2.112:0/2265466291 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:44:49.885 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.880 7f5a7ffff700 1 -- 172.21.2.112:0/2265466291 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5a800ada90 msgr2=0x7f5a800ade90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:49.885 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.880 7f5a7ffff700 1 --2- 172.21.2.112:0/2265466291 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5a800ada90 0x7f5a800ade90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:49.885 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.880 7f5a7ffff700 1 -- 172.21.2.112:0/2265466291 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5a8009a740 msgr2=0x7f5a80097bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:49.885 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.880 7f5a7ffff700 1 --2- 172.21.2.112:0/2265466291 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5a8009a740 0x7f5a80097bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:49.885 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.880 7f5a7ffff700 1 -- 172.21.2.112:0/2265466291 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a8009ad60 con 0x7f5a80099d20 2021-02-10T11:44:49.885 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.880 7f5a7ffff700 1 --2- 172.21.2.112:0/2265466291 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5a80099d20 0x7f5a8009a120 secure :-1 s=READY pgs=2667 cs=0 l=1 rev1=1 rx=0x7f5a70006f60 tx=0x7f5a7000b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:49.885 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.881 7f5a7d7fa700 1 -- 172.21.2.112:0/2265466291 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5a7000bd60 con 0x7f5a80099d20 2021-02-10T11:44:49.885 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.881 7f5a7d7fa700 1 -- 172.21.2.112:0/2265466291 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a70000c90 con 0x7f5a80099d20 2021-02-10T11:44:49.885 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.881 7f5a86bd0700 1 -- 172.21.2.112:0/2265466291 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a8009ad60 con 0x7f5a80099d20 2021-02-10T11:44:49.885 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.881 7f5a7d7fa700 1 -- 172.21.2.112:0/2265466291 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5a700022b0 con 0x7f5a80099d20 2021-02-10T11:44:49.885 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.881 7f5a86bd0700 1 -- 172.21.2.112:0/2265466291 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a8009ad60 con 0x7f5a80099d20 2021-02-10T11:44:49.885 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.881 7f5a7d7fa700 1 -- 172.21.2.112:0/2265466291 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5a70006b00 con 0x7f5a80099d20 2021-02-10T11:44:49.885 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.881 7f5a7d7fa700 1 --2- 172.21.2.112:0/2265466291 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5a640186d0 0x7f5a6401abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:49.886 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.881 7f5a7d7fa700 1 -- 172.21.2.112:0/2265466291 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f5a700327c0 con 0x7f5a80099d20 2021-02-10T11:44:49.886 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.881 7f5a849ec700 1 --2- 172.21.2.112:0/2265466291 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5a640186d0 0x7f5a6401abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:49.886 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.882 7f5a849ec700 1 --2- 172.21.2.112:0/2265466291 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5a640186d0 0x7f5a6401abd0 crc :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:49.913 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.912 7f5a86bd0700 1 -- 172.21.2.112:0/2265466291 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a800ebc00 con 0x7f5a80099d20 2021-02-10T11:44:49.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:49.914 7f5a7d7fa700 1 -- 172.21.2.112:0/2265466291 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5a70000c90 con 0x7f5a80099d20 2021-02-10T11:44:50.048 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:50.047 7f5a86bd0700 1 -- 172.21.2.112:0/2265466291 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5a800ebc00 con 0x7f5a80099d20 2021-02-10T11:44:50.048 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:50.047 7f5a7d7fa700 1 -- 172.21.2.112:0/2265466291 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 130 v130) v1 ==== 95+0+2108 (secure 0 0 0) 0x7f5a7000bd60 con 0x7f5a80099d20 2021-02-10T11:44:50.049 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:44:50.049 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":130,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":12006,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/1015001314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":1015001314},{"type":"v1","addr":"172.21.2.112:6833","nonce":1015001314}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":126}],"filesystems":[{"mdsmap":{"epoch":130,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:44:43.721296","modified":"2021-02-10 11:44:45.730948","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7268},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7268":{"gid":7268,"name":"b","rank":0,"incarnation":127,"state":"up:active","state_seq":3,"addr":"172.21.2.120:6835/3799929082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":3799929082},{"type":"v1","addr":"172.21.2.120:6835","nonce":3799929082}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":16}]} 2021-02-10T11:44:50.049 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 130 2021-02-10T11:44:50.050 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:50.049 7f5a86bd0700 1 -- 172.21.2.112:0/2265466291 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5a640186d0 msgr2=0x7f5a6401abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:50.051 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:50.049 7f5a86bd0700 1 --2- 172.21.2.112:0/2265466291 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5a640186d0 0x7f5a6401abd0 crc :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:50.051 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:50.049 7f5a86bd0700 1 -- 172.21.2.112:0/2265466291 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5a80099d20 msgr2=0x7f5a8009a120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:50.051 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:50.049 7f5a86bd0700 1 --2- 172.21.2.112:0/2265466291 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5a80099d20 0x7f5a8009a120 secure :-1 s=READY pgs=2667 cs=0 l=1 rev1=1 rx=0x7f5a70006f60 tx=0x7f5a7000b970).stop 2021-02-10T11:44:50.051 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:50.049 7f5a86bd0700 1 -- 172.21.2.112:0/2265466291 shutdown_connections 2021-02-10T11:44:50.051 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:50.049 7f5a86bd0700 1 --2- 172.21.2.112:0/2265466291 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5a640186d0 0x7f5a6401abd0 unknown :-1 s=CLOSED pgs=1219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:50.051 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:50.049 7f5a86bd0700 1 --2- 172.21.2.112:0/2265466291 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5a8009a740 0x7f5a80097bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:50.051 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:50.049 7f5a86bd0700 1 --2- 172.21.2.112:0/2265466291 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5a80099d20 0x7f5a8009a120 unknown :-1 s=CLOSED pgs=2667 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:50.051 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:50.049 7f5a86bd0700 1 --2- 172.21.2.112:0/2265466291 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5a800ada90 0x7f5a800ade90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:50.051 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:50.049 7f5a86bd0700 1 -- 172.21.2.112:0/2265466291 >> 172.21.2.112:0/2265466291 conn(0x7f5a800f0550 msgr2=0x7f5a80191730 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:50.051 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:50.049 7f5a86bd0700 1 -- 172.21.2.112:0/2265466291 shutdown_connections 2021-02-10T11:44:50.051 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:50.049 7f5a86bd0700 1 -- 172.21.2.112:0/2265466291 wait complete. 2021-02-10T11:44:50.261 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 130, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:44:43.721296', 'modified': '2021-02-10 11:44:45.730948', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7268}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7268': {'gid': 7268, 'name': 'b', 'rank': 0, 'incarnation': 127, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.120:6835/3799929082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.120:6834', 'nonce': 3799929082}, {'type': 'v1', 'addr': '172.21.2.120:6835', 'nonce': 3799929082}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [33], 'metadata_pool': 32, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:44:50.261 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:44:50.262 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:44:50.659 INFO:teuthology.orchestra.run.gibba020.stdout:{ 2021-02-10T11:44:50.659 INFO:teuthology.orchestra.run.gibba020.stdout: "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", 2021-02-10T11:44:50.659 INFO:teuthology.orchestra.run.gibba020.stdout: "whoami": 0, 2021-02-10T11:44:50.659 INFO:teuthology.orchestra.run.gibba020.stdout: "id": 7268, 2021-02-10T11:44:50.659 INFO:teuthology.orchestra.run.gibba020.stdout: "want_state": "up:active", 2021-02-10T11:44:50.659 INFO:teuthology.orchestra.run.gibba020.stdout: "state": "up:active", 2021-02-10T11:44:50.659 INFO:teuthology.orchestra.run.gibba020.stdout: "rank_uptime": 6.9342364349999999, 2021-02-10T11:44:50.659 INFO:teuthology.orchestra.run.gibba020.stdout: "mdsmap_epoch": 130, 2021-02-10T11:44:50.659 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch": 151, 2021-02-10T11:44:50.659 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch_barrier": 151, 2021-02-10T11:44:50.660 INFO:teuthology.orchestra.run.gibba020.stdout: "uptime": 12.577867488000001 2021-02-10T11:44:50.660 INFO:teuthology.orchestra.run.gibba020.stdout:} 2021-02-10T11:44:50.663 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", "whoami": 0, "id": 7268, "want_state": "up:active", "state": "up:active", "rank_uptime": 6.9342364349999999, "mdsmap_epoch": 130, "osdmap_epoch": 151, "osdmap_epoch_barrier": 151, "uptime": 12.577867488000001 } 2021-02-10T11:44:50.663 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:44:51.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.073 7f1e6b609700 1 Processor -- start 2021-02-10T11:44:51.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.073 7f1e6b609700 1 -- start start 2021-02-10T11:44:51.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.074 7f1e6b609700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f1e640d0340 0x7f1e640ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:51.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.074 7f1e6b609700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e640f0b10 con 0x7f1e640d1870 2021-02-10T11:44:51.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.074 7f1e6b609700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e641b0be0 con 0x7f1e640d1c70 2021-02-10T11:44:51.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.074 7f1e6b609700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e640e5ed0 con 0x7f1e640d0340 2021-02-10T11:44:51.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.074 7f1e69425700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f1e640d1870 0x7f1e641b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:57724/0 (socket says 172.21.2.112:57724) 2021-02-10T11:44:51.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.074 7f1e69425700 1 -- 172.21.2.112:0/2666855863 learned_addr learned my addr 172.21.2.112:0/2666855863 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:44:51.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.074 7f1e57fff700 1 -- 172.21.2.112:0/2666855863 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1393395563 0 0) 0x7f1e60001180 con 0x7f1e640d1870 2021-02-10T11:44:51.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.074 7f1e57fff700 1 -- 172.21.2.112:0/2666855863 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e50002fe0 con 0x7f1e640d1870 2021-02-10T11:44:51.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.075 7f1e68c24700 1 --2- 172.21.2.112:0/2666855863 >> v2:172.21.2.120:3300/0 conn(0x7f1e640d0340 0x7f1e640ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:51.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.075 7f1e57fff700 1 -- 172.21.2.112:0/2666855863 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1982394833 0 0) 0x7f1e60001180 con 0x7f1e640d1870 2021-02-10T11:44:51.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.075 7f1e57fff700 1 -- 172.21.2.112:0/2666855863 >> v2:172.21.2.120:3300/0 conn(0x7f1e640d0340 msgr2=0x7f1e640ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:51.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.075 7f1e57fff700 1 --2- 172.21.2.112:0/2666855863 >> v2:172.21.2.120:3300/0 conn(0x7f1e640d0340 0x7f1e640ef640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:51.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.075 7f1e57fff700 1 -- 172.21.2.112:0/2666855863 >> v1:172.21.2.120:6789/0 conn(0x7f1e640d1c70 legacy=0x7f1e640efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:51.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.075 7f1e57fff700 1 -- 172.21.2.112:0/2666855863 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e640f0450 con 0x7f1e640d1870 2021-02-10T11:44:51.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.075 7f1e57fff700 1 -- 172.21.2.112:0/2666855863 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f1e60001680 con 0x7f1e640d1870 2021-02-10T11:44:51.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.075 7f1e57fff700 1 -- 172.21.2.112:0/2666855863 >> v1:172.21.2.112:6789/0 conn(0x7f1e640d1870 legacy=0x7f1e641b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:51.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.075 7f1e57fff700 1 --2- 172.21.2.112:0/2666855863 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1e50003b90 0x7f1e50006020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:51.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.075 7f1e57fff700 1 --2- 172.21.2.112:0/2666855863 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1e50006830 0x7f1e50008ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:51.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.075 7f1e57fff700 1 --2- 172.21.2.112:0/2666855863 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1e500095e0 0x7f1e5000bae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:51.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.075 7f1e57fff700 1 -- 172.21.2.112:0/2666855863 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e5000c310 con 0x7f1e50006830 2021-02-10T11:44:51.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.075 7f1e57fff700 1 -- 172.21.2.112:0/2666855863 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e5000c530 con 0x7f1e50003b90 2021-02-10T11:44:51.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.075 7f1e57fff700 1 -- 172.21.2.112:0/2666855863 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e5000c750 con 0x7f1e500095e0 2021-02-10T11:44:51.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.075 7f1e68c24700 1 --2- 172.21.2.112:0/2666855863 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1e50006830 0x7f1e50008ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:51.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.075 7f1e69425700 1 --2- 172.21.2.112:0/2666855863 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1e50003b90 0x7f1e50006020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:51.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.075 7f1e69c26700 1 --2- 172.21.2.112:0/2666855863 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1e500095e0 0x7f1e5000bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:51.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.075 7f1e68c24700 1 -- 172.21.2.112:0/2666855863 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1e500095e0 msgr2=0x7f1e5000bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:51.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.075 7f1e68c24700 1 --2- 172.21.2.112:0/2666855863 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1e500095e0 0x7f1e5000bae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:51.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.075 7f1e68c24700 1 -- 172.21.2.112:0/2666855863 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1e50003b90 msgr2=0x7f1e50006020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:51.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.075 7f1e68c24700 1 --2- 172.21.2.112:0/2666855863 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1e50003b90 0x7f1e50006020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:51.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.075 7f1e68c24700 1 -- 172.21.2.112:0/2666855863 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e5000c9f0 con 0x7f1e50006830 2021-02-10T11:44:51.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.075 7f1e69c26700 1 -- 172.21.2.112:0/2666855863 reap_dead start 2021-02-10T11:44:51.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.076 7f1e68c24700 1 --2- 172.21.2.112:0/2666855863 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1e50006830 0x7f1e50008ce0 secure :-1 s=READY pgs=2669 cs=0 l=1 rev1=1 rx=0x7f1e58006a40 tx=0x7f1e58006820).ready entity=mon.0 client_cookie=98548eafcd77b6ac server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:51.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.076 7f1e57fff700 1 -- 172.21.2.112:0/2666855863 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1e58000c90 con 0x7f1e50006830 2021-02-10T11:44:51.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.076 7f1e57fff700 1 -- 172.21.2.112:0/2666855863 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e58002030 con 0x7f1e50006830 2021-02-10T11:44:51.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.076 7f1e57fff700 1 -- 172.21.2.112:0/2666855863 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1e58002550 con 0x7f1e50006830 2021-02-10T11:44:51.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.076 7f1e6b609700 1 -- 172.21.2.112:0/2666855863 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1e50006830 msgr2=0x7f1e50008ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:51.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.076 7f1e6b609700 1 --2- 172.21.2.112:0/2666855863 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1e50006830 0x7f1e50008ce0 secure :-1 s=READY pgs=2669 cs=0 l=1 rev1=1 rx=0x7f1e58006a40 tx=0x7f1e58006820).stop 2021-02-10T11:44:51.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.076 7f1e6b609700 1 -- 172.21.2.112:0/2666855863 shutdown_connections 2021-02-10T11:44:51.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.076 7f1e6b609700 1 --2- 172.21.2.112:0/2666855863 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1e50006830 0x7f1e50008ce0 unknown :-1 s=CLOSED pgs=2669 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:51.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.076 7f1e6b609700 1 -- 172.21.2.112:0/2666855863 >> 172.21.2.112:0/2666855863 conn(0x7f1e640ce450 msgr2=0x7f1e640d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:51.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.077 7f1e6b609700 1 -- 172.21.2.112:0/2666855863 shutdown_connections 2021-02-10T11:44:51.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.077 7f1e6b609700 1 -- 172.21.2.112:0/2666855863 wait complete. 2021-02-10T11:44:51.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.091 7f1e6b609700 1 Processor -- start 2021-02-10T11:44:51.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.091 7f1e6b609700 1 -- start start 2021-02-10T11:44:51.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.092 7f1e6b609700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1e641a4a60 0x7f1e640c1c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:51.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.092 7f1e6b609700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1e640c2290 0x7f1e640c2690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:51.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.092 7f1e6b609700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1e640c2cb0 0x7f1e640bfd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:51.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.092 7f1e6b609700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e640f05b0 con 0x7f1e640c2cb0 2021-02-10T11:44:51.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.092 7f1e6b609700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e640f07d0 con 0x7f1e641a4a60 2021-02-10T11:44:51.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.092 7f1e6b609700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e640c0340 con 0x7f1e640c2290 2021-02-10T11:44:51.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.092 7f1e69c26700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1e640c2cb0 0x7f1e640bfd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:51.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.092 7f1e69c26700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1e640c2cb0 0x7f1e640bfd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:58276/0 (socket says 172.21.2.112:58276) 2021-02-10T11:44:51.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.092 7f1e69c26700 1 -- 172.21.2.112:0/211780008 learned_addr learned my addr 172.21.2.112:0/211780008 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:44:51.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.092 7f1e69c26700 1 -- 172.21.2.112:0/211780008 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1e640c2290 msgr2=0x7f1e640c2690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:51.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.092 7f1e69c26700 1 --2- 172.21.2.112:0/211780008 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1e640c2290 0x7f1e640c2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:51.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.092 7f1e69c26700 1 -- 172.21.2.112:0/211780008 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1e641a4a60 msgr2=0x7f1e640c1c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:51.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.092 7f1e69c26700 1 --2- 172.21.2.112:0/211780008 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1e641a4a60 0x7f1e640c1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:51.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.092 7f1e69c26700 1 -- 172.21.2.112:0/211780008 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e640c0560 con 0x7f1e640c2cb0 2021-02-10T11:44:51.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.092 7f1e69c26700 1 --2- 172.21.2.112:0/211780008 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1e640c2cb0 0x7f1e640bfd20 secure :-1 s=READY pgs=2670 cs=0 l=1 rev1=1 rx=0x7f1e5c006fc0 tx=0x7f1e5c00bd20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:51.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.093 7f1e55ffb700 1 -- 172.21.2.112:0/211780008 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1e5c005910 con 0x7f1e640c2cb0 2021-02-10T11:44:51.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.093 7f1e55ffb700 1 -- 172.21.2.112:0/211780008 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e5c006b70 con 0x7f1e640c2cb0 2021-02-10T11:44:51.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.093 7f1e55ffb700 1 -- 172.21.2.112:0/211780008 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1e5c006030 con 0x7f1e640c2cb0 2021-02-10T11:44:51.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.093 7f1e6b609700 1 -- 172.21.2.112:0/211780008 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e640c0560 con 0x7f1e640c2cb0 2021-02-10T11:44:51.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.093 7f1e6b609700 1 -- 172.21.2.112:0/211780008 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e640c0af0 con 0x7f1e640c2cb0 2021-02-10T11:44:51.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.093 7f1e55ffb700 1 -- 172.21.2.112:0/211780008 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1e5c005910 con 0x7f1e640c2cb0 2021-02-10T11:44:51.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.093 7f1e55ffb700 1 --2- 172.21.2.112:0/211780008 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1e440186c0 0x7f1e4401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:51.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.093 7f1e55ffb700 1 -- 172.21.2.112:0/211780008 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f1e5c0328a0 con 0x7f1e640c2cb0 2021-02-10T11:44:51.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.094 7f1e69425700 1 --2- 172.21.2.112:0/211780008 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1e440186c0 0x7f1e4401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:51.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.095 7f1e69425700 1 --2- 172.21.2.112:0/211780008 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1e440186c0 0x7f1e4401abc0 crc :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:51.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.124 7f1e6b609700 1 -- 172.21.2.112:0/211780008 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e640f0b10 con 0x7f1e640c2cb0 2021-02-10T11:44:51.127 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.126 7f1e55ffb700 1 -- 172.21.2.112:0/211780008 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1e5c0056b0 con 0x7f1e640c2cb0 2021-02-10T11:44:51.253 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.252 7f1e6b609700 1 -- 172.21.2.112:0/211780008 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1e640f0b10 con 0x7f1e640c2cb0 2021-02-10T11:44:51.254 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.253 7f1e55ffb700 1 -- 172.21.2.112:0/211780008 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v151) v1 ==== 74+0+14007 (secure 0 0 0) 0x7f1e5c005910 con 0x7f1e640c2cb0 2021-02-10T11:44:51.254 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:44:51.254 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":151,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:44:44.723715","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":32,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:44:41.688683","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"150","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":33,"pool_name":"cephfs_data","create_time":"2021-02-10 11:44:42.764257","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"151","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:44:51.255 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.254 7f1e6b609700 1 -- 172.21.2.112:0/211780008 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1e440186c0 msgr2=0x7f1e4401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:51.255 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.254 7f1e6b609700 1 --2- 172.21.2.112:0/211780008 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1e440186c0 0x7f1e4401abc0 crc :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:51.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.254 7f1e6b609700 1 -- 172.21.2.112:0/211780008 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1e640c2cb0 msgr2=0x7f1e640bfd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:51.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.254 7f1e6b609700 1 --2- 172.21.2.112:0/211780008 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1e640c2cb0 0x7f1e640bfd20 secure :-1 s=READY pgs=2670 cs=0 l=1 rev1=1 rx=0x7f1e5c006fc0 tx=0x7f1e5c00bd20).stop 2021-02-10T11:44:51.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.255 7f1e6b609700 1 -- 172.21.2.112:0/211780008 shutdown_connections 2021-02-10T11:44:51.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.255 7f1e6b609700 1 --2- 172.21.2.112:0/211780008 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f1e440186c0 0x7f1e4401abc0 unknown :-1 s=CLOSED pgs=1220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:51.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.255 7f1e6b609700 1 --2- 172.21.2.112:0/211780008 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1e640c2cb0 0x7f1e640bfd20 unknown :-1 s=CLOSED pgs=2670 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:51.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.255 7f1e6b609700 1 --2- 172.21.2.112:0/211780008 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1e640c2290 0x7f1e640c2690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:51.257 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.255 7f1e6b609700 1 --2- 172.21.2.112:0/211780008 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1e641a4a60 0x7f1e640c1c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:51.257 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.255 7f1e6b609700 1 -- 172.21.2.112:0/211780008 >> 172.21.2.112:0/211780008 conn(0x7f1e640ebe30 msgr2=0x7f1e640bf200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:51.257 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.255 7f1e6b609700 1 -- 172.21.2.112:0/211780008 shutdown_connections 2021-02-10T11:44:51.257 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.255 7f1e6b609700 1 -- 172.21.2.112:0/211780008 wait complete. 2021-02-10T11:44:51.264 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:13:57.508758', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 32, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:44:41.688683', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '150', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 33, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:44:42.764257', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '151', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:44:51.265 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:44:51.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.679 7fe6e0823700 1 Processor -- start 2021-02-10T11:44:51.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.679 7fe6e0823700 1 -- start start 2021-02-10T11:44:51.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.680 7fe6e0823700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe6dc0d1c70 0x7fe6dc0cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:51.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.680 7fe6e0823700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fe6dc0cf580 0x7fe6dc0cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:51.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.680 7fe6e0823700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6dc0fe1c0 con 0x7fe6dc0d1870 2021-02-10T11:44:51.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.680 7fe6e0823700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6dc1b0cb0 con 0x7fe6dc0cf580 2021-02-10T11:44:51.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.680 7fe6e0823700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6dc109a60 con 0x7fe6dc0d1c70 2021-02-10T11:44:51.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.680 7fe6da61c700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fe6dc0d1870 0x7fe6dc1b04c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:57742/0 (socket says 172.21.2.112:57742) 2021-02-10T11:44:51.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.680 7fe6da61c700 1 -- 172.21.2.112:0/849522634 learned_addr learned my addr 172.21.2.112:0/849522634 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:44:51.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.680 7fe6d961a700 1 -- 172.21.2.112:0/849522634 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 270880587 0 0) 0x7fe6cc0010b0 con 0x7fe6dc0d1870 2021-02-10T11:44:51.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.681 7fe6d961a700 1 -- 172.21.2.112:0/849522634 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6c8002fe0 con 0x7fe6dc0d1870 2021-02-10T11:44:51.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.681 7fe6dae1d700 1 --2- 172.21.2.112:0/849522634 >> v2:172.21.2.120:3300/0 conn(0x7fe6dc0cf580 0x7fe6dc0cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:51.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.681 7fe6d9e1b700 1 --2- 172.21.2.112:0/849522634 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe6dc0d1c70 0x7fe6dc0cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:51.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.681 7fe6d961a700 1 -- 172.21.2.112:0/849522634 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1626876362 0 0) 0x7fe6cc0010b0 con 0x7fe6dc0d1870 2021-02-10T11:44:51.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.681 7fe6d961a700 1 -- 172.21.2.112:0/849522634 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe6dc0d1c70 msgr2=0x7fe6dc0cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:51.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.681 7fe6d961a700 1 --2- 172.21.2.112:0/849522634 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe6dc0d1c70 0x7fe6dc0cef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:51.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.681 7fe6d961a700 1 -- 172.21.2.112:0/849522634 >> v2:172.21.2.120:3300/0 conn(0x7fe6dc0cf580 msgr2=0x7fe6dc0cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:51.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.681 7fe6d961a700 1 --2- 172.21.2.112:0/849522634 >> v2:172.21.2.120:3300/0 conn(0x7fe6dc0cf580 0x7fe6dc0cf980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:51.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.681 7fe6d961a700 1 -- 172.21.2.112:0/849522634 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6dc088070 con 0x7fe6dc0d1870 2021-02-10T11:44:51.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.681 7fe6d961a700 1 -- 172.21.2.112:0/849522634 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7fe6cc0015b0 con 0x7fe6dc0d1870 2021-02-10T11:44:51.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.681 7fe6d961a700 1 -- 172.21.2.112:0/849522634 >> v1:172.21.2.112:6789/0 conn(0x7fe6dc0d1870 legacy=0x7fe6dc1b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:51.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.681 7fe6d961a700 1 --2- 172.21.2.112:0/849522634 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe6c8003b10 0x7fe6c8006010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:51.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.681 7fe6d961a700 1 --2- 172.21.2.112:0/849522634 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe6c804a870 0x7fe6c804cca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:51.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.681 7fe6d961a700 1 --2- 172.21.2.112:0/849522634 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe6c804d5a0 0x7fe6c804fa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:51.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.681 7fe6d961a700 1 -- 172.21.2.112:0/849522634 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6c80502e0 con 0x7fe6c804a870 2021-02-10T11:44:51.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.681 7fe6d961a700 1 -- 172.21.2.112:0/849522634 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6c8050500 con 0x7fe6c8003b10 2021-02-10T11:44:51.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.681 7fe6d961a700 1 -- 172.21.2.112:0/849522634 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6c8050720 con 0x7fe6c804d5a0 2021-02-10T11:44:51.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.681 7fe6d9e1b700 1 --2- 172.21.2.112:0/849522634 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe6c804a870 0x7fe6c804cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:51.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.681 7fe6da61c700 1 --2- 172.21.2.112:0/849522634 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe6c8003b10 0x7fe6c8006010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:51.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.681 7fe6dae1d700 1 --2- 172.21.2.112:0/849522634 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe6c804d5a0 0x7fe6c804fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:51.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.681 7fe6d9e1b700 1 -- 172.21.2.112:0/849522634 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe6c804d5a0 msgr2=0x7fe6c804fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:51.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.681 7fe6d9e1b700 1 --2- 172.21.2.112:0/849522634 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe6c804d5a0 0x7fe6c804fa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:51.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.681 7fe6d9e1b700 1 -- 172.21.2.112:0/849522634 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe6c8003b10 msgr2=0x7fe6c8006010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:51.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.681 7fe6d9e1b700 1 --2- 172.21.2.112:0/849522634 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe6c8003b10 0x7fe6c8006010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:51.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.681 7fe6d9e1b700 1 -- 172.21.2.112:0/849522634 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6c80509c0 con 0x7fe6c804a870 2021-02-10T11:44:51.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.681 7fe6dae1d700 1 --2- 172.21.2.112:0/849522634 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe6c804d5a0 0x7fe6c804fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:44:51.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.682 7fe6d9e1b700 1 --2- 172.21.2.112:0/849522634 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe6c804a870 0x7fe6c804cca0 secure :-1 s=READY pgs=2672 cs=0 l=1 rev1=1 rx=0x7fe6c4009310 tx=0x7fe6c4009a90).ready entity=mon.0 client_cookie=53d8eca63dcef83c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:51.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.682 7fe6d961a700 1 -- 172.21.2.112:0/849522634 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe6c400c0c0 con 0x7fe6c804a870 2021-02-10T11:44:51.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.682 7fe6d961a700 1 -- 172.21.2.112:0/849522634 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6c4006400 con 0x7fe6c804a870 2021-02-10T11:44:51.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.682 7fe6d961a700 1 -- 172.21.2.112:0/849522634 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe6c400c340 con 0x7fe6c804a870 2021-02-10T11:44:51.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.682 7fe6e0823700 1 -- 172.21.2.112:0/849522634 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe6c804a870 msgr2=0x7fe6c804cca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:51.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.682 7fe6e0823700 1 --2- 172.21.2.112:0/849522634 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe6c804a870 0x7fe6c804cca0 secure :-1 s=READY pgs=2672 cs=0 l=1 rev1=1 rx=0x7fe6c4009310 tx=0x7fe6c4009a90).stop 2021-02-10T11:44:51.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.682 7fe6dae1d700 1 -- 172.21.2.112:0/849522634 reap_dead start 2021-02-10T11:44:51.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.682 7fe6e0823700 1 -- 172.21.2.112:0/849522634 shutdown_connections 2021-02-10T11:44:51.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.683 7fe6e0823700 1 -- 172.21.2.112:0/849522634 >> 172.21.2.112:0/849522634 conn(0x7fe6dc0ce420 msgr2=0x7fe6dc0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:51.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.683 7fe6e0823700 1 -- 172.21.2.112:0/849522634 shutdown_connections 2021-02-10T11:44:51.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.683 7fe6e0823700 1 -- 172.21.2.112:0/849522634 wait complete. 2021-02-10T11:44:51.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.697 7fe6e0823700 1 Processor -- start 2021-02-10T11:44:51.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.697 7fe6e0823700 1 -- start start 2021-02-10T11:44:51.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.698 7fe6e0823700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe6dc1a79a0 0x7fe6dc1a7da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:51.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.698 7fe6e0823700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe6dc0bdb40 0x7fe6dc0bdf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:51.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.698 7fe6e0823700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe6dc082ca0 0x7fe6dc0830a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:51.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.698 7fe6e0823700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6dc19d990 con 0x7fe6dc0bdb40 2021-02-10T11:44:51.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.698 7fe6e0823700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6dc19dbb0 con 0x7fe6dc082ca0 2021-02-10T11:44:51.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.698 7fe6e0823700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6dc0836c0 con 0x7fe6dc1a79a0 2021-02-10T11:44:51.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.698 7fe6dae1d700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe6dc082ca0 0x7fe6dc0830a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:51.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.698 7fe6d9e1b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe6dc0bdb40 0x7fe6dc0bdf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:51.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.698 7fe6dae1d700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe6dc082ca0 0x7fe6dc0830a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3300/0 says I am v2:172.21.2.112:60572/0 (socket says 172.21.2.112:60572) 2021-02-10T11:44:51.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.698 7fe6dae1d700 1 -- 172.21.2.112:0/614837527 learned_addr learned my addr 172.21.2.112:0/614837527 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:44:51.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.698 7fe6da61c700 1 --2- 172.21.2.112:0/614837527 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe6dc1a79a0 0x7fe6dc1a7da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:51.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.698 7fe6da61c700 1 -- 172.21.2.112:0/614837527 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe6dc082ca0 msgr2=0x7fe6dc0830a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:51.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.698 7fe6da61c700 1 --2- 172.21.2.112:0/614837527 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe6dc082ca0 0x7fe6dc0830a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:51.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.698 7fe6da61c700 1 -- 172.21.2.112:0/614837527 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe6dc0bdb40 msgr2=0x7fe6dc0bdf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:51.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.698 7fe6da61c700 1 --2- 172.21.2.112:0/614837527 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe6dc0bdb40 0x7fe6dc0bdf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:51.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.698 7fe6da61c700 1 -- 172.21.2.112:0/614837527 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6dc0838e0 con 0x7fe6dc1a79a0 2021-02-10T11:44:51.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.698 7fe6d9e1b700 1 --2- 172.21.2.112:0/614837527 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe6dc0bdb40 0x7fe6dc0bdf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:44:51.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.698 7fe6da61c700 1 --2- 172.21.2.112:0/614837527 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe6dc1a79a0 0x7fe6dc1a7da0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0x7fe6cc007580 tx=0x7fe6cc00a870).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:51.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.699 7fe6d2ffd700 1 -- 172.21.2.112:0/614837527 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe6cc000ef0 con 0x7fe6dc1a79a0 2021-02-10T11:44:51.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.699 7fe6d2ffd700 1 -- 172.21.2.112:0/614837527 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6cc0009e0 con 0x7fe6dc1a79a0 2021-02-10T11:44:51.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.699 7fe6d2ffd700 1 -- 172.21.2.112:0/614837527 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe6cc004350 con 0x7fe6dc1a79a0 2021-02-10T11:44:51.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.699 7fe6e0823700 1 -- 172.21.2.112:0/614837527 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6dc0838e0 con 0x7fe6dc1a79a0 2021-02-10T11:44:51.707 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.699 7fe6e0823700 1 -- 172.21.2.112:0/614837527 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6dc0838e0 con 0x7fe6dc1a79a0 2021-02-10T11:44:51.707 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.699 7fe6d2ffd700 1 -- 172.21.2.112:0/614837527 <== mon.2 v2:172.21.2.120:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe6cc004140 con 0x7fe6dc1a79a0 2021-02-10T11:44:51.707 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.699 7fe6d2ffd700 1 --2- 172.21.2.112:0/614837527 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe6c00186c0 0x7fe6c001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:51.707 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.700 7fe6d2ffd700 1 -- 172.21.2.112:0/614837527 <== mon.2 v2:172.21.2.120:3301/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fe6cc032420 con 0x7fe6dc1a79a0 2021-02-10T11:44:51.707 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.700 7fe6d9e1b700 1 --2- 172.21.2.112:0/614837527 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe6c00186c0 0x7fe6c001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:51.707 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.700 7fe6d9e1b700 1 --2- 172.21.2.112:0/614837527 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe6c00186c0 0x7fe6c001abc0 crc :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:51.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.730 7fe6e0823700 1 -- 172.21.2.112:0/614837527 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6dc082a70 con 0x7fe6dc1a79a0 2021-02-10T11:44:51.734 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.733 7fe6d2ffd700 1 -- 172.21.2.112:0/614837527 <== mon.2 v2:172.21.2.120:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe6cc02cf00 con 0x7fe6dc1a79a0 2021-02-10T11:44:51.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.862 7fe6e0823700 1 -- 172.21.2.112:0/614837527 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7fe6dc082a70 con 0x7fe6dc1a79a0 2021-02-10T11:44:51.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.864 7fe6d2ffd700 1 -- 172.21.2.112:0/614837527 <== mon.2 v2:172.21.2.120:3301/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v151) v1 ==== 92+0+10 (secure 0 0 0) 0x7fe6cc02cf00 con 0x7fe6dc1a79a0 2021-02-10T11:44:51.866 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:44:51.868 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.867 7fe6e0823700 1 -- 172.21.2.112:0/614837527 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe6c00186c0 msgr2=0x7fe6c001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:51.868 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.867 7fe6e0823700 1 --2- 172.21.2.112:0/614837527 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe6c00186c0 0x7fe6c001abc0 crc :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:51.868 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.867 7fe6e0823700 1 -- 172.21.2.112:0/614837527 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe6dc1a79a0 msgr2=0x7fe6dc1a7da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:51.868 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.867 7fe6e0823700 1 --2- 172.21.2.112:0/614837527 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe6dc1a79a0 0x7fe6dc1a7da0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0x7fe6cc007580 tx=0x7fe6cc00a870).stop 2021-02-10T11:44:51.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.867 7fe6e0823700 1 -- 172.21.2.112:0/614837527 shutdown_connections 2021-02-10T11:44:51.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.867 7fe6e0823700 1 --2- 172.21.2.112:0/614837527 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe6c00186c0 0x7fe6c001abc0 unknown :-1 s=CLOSED pgs=1221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:51.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.867 7fe6e0823700 1 --2- 172.21.2.112:0/614837527 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe6dc082ca0 0x7fe6dc0830a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:51.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.867 7fe6e0823700 1 --2- 172.21.2.112:0/614837527 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe6dc0bdb40 0x7fe6dc0bdf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:51.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.867 7fe6e0823700 1 --2- 172.21.2.112:0/614837527 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe6dc1a79a0 0x7fe6dc1a7da0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:51.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.867 7fe6e0823700 1 -- 172.21.2.112:0/614837527 >> 172.21.2.112:0/614837527 conn(0x7fe6dc0bef00 msgr2=0x7fe6dc09f200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:51.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.867 7fe6e0823700 1 -- 172.21.2.112:0/614837527 shutdown_connections 2021-02-10T11:44:51.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:51.867 7fe6e0823700 1 -- 172.21.2.112:0/614837527 wait complete. 2021-02-10T11:44:51.876 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:44:52.308 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.290 7f7b5f51f700 1 Processor -- start 2021-02-10T11:44:52.308 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.290 7f7b5f51f700 1 -- start start 2021-02-10T11:44:52.308 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.291 7f7b5f51f700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7b580d1870 0x7f7b581b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:52.308 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.291 7f7b5f51f700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f7b581b0ae0 0x7f7b58108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:52.308 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.291 7f7b5f51f700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b58109a60 con 0x7f7b580d1c70 2021-02-10T11:44:52.308 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.291 7f7b5f51f700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b580e5ef0 con 0x7f7b581b0ae0 2021-02-10T11:44:52.308 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.291 7f7b5f51f700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b58109750 con 0x7f7b580d1870 2021-02-10T11:44:52.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.291 7f7b5cb3a700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f7b581b0ae0 0x7f7b58108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:52.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.291 7f7b5cb3a700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f7b581b0ae0 0x7f7b58108940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:58300/0 (socket says 172.21.2.112:58300) 2021-02-10T11:44:52.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.291 7f7b5cb3a700 1 -- 172.21.2.112:0/2755237639 learned_addr learned my addr 172.21.2.112:0/2755237639 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:44:52.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.291 7f7b5cb3a700 1 -- 172.21.2.112:0/2755237639 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7b580d1870 msgr2=0x7f7b581b04c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:44:52.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.291 7f7b5cb3a700 1 --2- 172.21.2.112:0/2755237639 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7b580d1870 0x7f7b581b04c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:52.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.291 7f7b5cb3a700 1 -- 172.21.2.112:0/2755237639 >> v1:172.21.2.120:6789/0 conn(0x7f7b580d1c70 legacy=0x7f7b58108f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:52.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.291 7f7b5cb3a700 1 -- 172.21.2.112:0/2755237639 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b58088070 con 0x7f7b581b0ae0 2021-02-10T11:44:52.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.291 7f7b5cb3a700 1 --2- 172.21.2.112:0/2755237639 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7b581b0ae0 0x7f7b58108940 secure :-1 s=READY pgs=2673 cs=0 l=1 rev1=1 rx=0x7f7b4c006700 tx=0x7f7b4c009b00).ready entity=mon.0 client_cookie=90c906bbb89044c1 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:52.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.292 7f7b4bfff700 1 -- 172.21.2.112:0/2755237639 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7b4c00c0c0 con 0x7f7b581b0ae0 2021-02-10T11:44:52.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.292 7f7b4bfff700 1 -- 172.21.2.112:0/2755237639 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b4c00c340 con 0x7f7b581b0ae0 2021-02-10T11:44:52.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.292 7f7b4bfff700 1 -- 172.21.2.112:0/2755237639 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7b4c00c820 con 0x7f7b581b0ae0 2021-02-10T11:44:52.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.292 7f7b5f51f700 1 -- 172.21.2.112:0/2755237639 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7b581b0ae0 msgr2=0x7f7b58108940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:52.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.292 7f7b5f51f700 1 --2- 172.21.2.112:0/2755237639 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7b581b0ae0 0x7f7b58108940 secure :-1 s=READY pgs=2673 cs=0 l=1 rev1=1 rx=0x7f7b4c006700 tx=0x7f7b4c009b00).stop 2021-02-10T11:44:52.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.292 7f7b5f51f700 1 -- 172.21.2.112:0/2755237639 shutdown_connections 2021-02-10T11:44:52.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.292 7f7b5f51f700 1 --2- 172.21.2.112:0/2755237639 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7b581b0ae0 0x7f7b58108940 unknown :-1 s=CLOSED pgs=2673 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:52.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.292 7f7b5f51f700 1 --2- 172.21.2.112:0/2755237639 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7b580d1870 0x7f7b581b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:52.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.292 7f7b5f51f700 1 -- 172.21.2.112:0/2755237639 >> 172.21.2.112:0/2755237639 conn(0x7f7b580ce420 msgr2=0x7f7b580d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:52.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.293 7f7b5f51f700 1 -- 172.21.2.112:0/2755237639 shutdown_connections 2021-02-10T11:44:52.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.293 7f7b5f51f700 1 -- 172.21.2.112:0/2755237639 wait complete. 2021-02-10T11:44:52.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.307 7f7b5f51f700 1 Processor -- start 2021-02-10T11:44:52.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.307 7f7b5f51f700 1 -- start start 2021-02-10T11:44:52.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.307 7f7b5f51f700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7b580c3820 0x7f7b580a7950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:52.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.307 7f7b5f51f700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7b581a44c0 0x7f7b581a48c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:52.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.307 7f7b5f51f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7b580863d0 0x7f7b580867d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:52.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.307 7f7b5f51f700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b58085ed0 con 0x7f7b580863d0 2021-02-10T11:44:52.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.307 7f7b5f51f700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b580c0740 con 0x7f7b581a44c0 2021-02-10T11:44:52.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.307 7f7b5f51f700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b58086df0 con 0x7f7b580c3820 2021-02-10T11:44:52.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.307 7f7b5db3c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7b580863d0 0x7f7b580867d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:52.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.307 7f7b5db3c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7b580863d0 0x7f7b580867d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:58308/0 (socket says 172.21.2.112:58308) 2021-02-10T11:44:52.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.307 7f7b5db3c700 1 -- 172.21.2.112:0/1320030140 learned_addr learned my addr 172.21.2.112:0/1320030140 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:44:52.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.308 7f7b5db3c700 1 -- 172.21.2.112:0/1320030140 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7b580c3820 msgr2=0x7f7b580a7950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:52.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.308 7f7b5db3c700 1 --2- 172.21.2.112:0/1320030140 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7b580c3820 0x7f7b580a7950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:52.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.308 7f7b5db3c700 1 -- 172.21.2.112:0/1320030140 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7b581a44c0 msgr2=0x7f7b581a48c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:52.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.308 7f7b5db3c700 1 --2- 172.21.2.112:0/1320030140 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7b581a44c0 0x7f7b581a48c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:52.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.308 7f7b5db3c700 1 -- 172.21.2.112:0/1320030140 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b58087010 con 0x7f7b580863d0 2021-02-10T11:44:52.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.308 7f7b5db3c700 1 --2- 172.21.2.112:0/1320030140 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7b580863d0 0x7f7b580867d0 secure :-1 s=READY pgs=2674 cs=0 l=1 rev1=1 rx=0x7f7b50009940 tx=0x7f7b5000bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:52.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.308 7f7b49ffb700 1 -- 172.21.2.112:0/1320030140 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7b50000d30 con 0x7f7b580863d0 2021-02-10T11:44:52.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.308 7f7b49ffb700 1 -- 172.21.2.112:0/1320030140 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b5000c2f0 con 0x7f7b580863d0 2021-02-10T11:44:52.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.308 7f7b5f51f700 1 -- 172.21.2.112:0/1320030140 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b58087010 con 0x7f7b580863d0 2021-02-10T11:44:52.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.308 7f7b5f51f700 1 -- 172.21.2.112:0/1320030140 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b58083000 con 0x7f7b580863d0 2021-02-10T11:44:52.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.308 7f7b49ffb700 1 -- 172.21.2.112:0/1320030140 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7b5000c810 con 0x7f7b580863d0 2021-02-10T11:44:52.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.309 7f7b49ffb700 1 -- 172.21.2.112:0/1320030140 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7b50001220 con 0x7f7b580863d0 2021-02-10T11:44:52.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.309 7f7b49ffb700 1 --2- 172.21.2.112:0/1320030140 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7b380186c0 0x7f7b3801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:52.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.309 7f7b49ffb700 1 -- 172.21.2.112:0/1320030140 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f7b50032b20 con 0x7f7b580863d0 2021-02-10T11:44:52.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.309 7f7b5d33b700 1 --2- 172.21.2.112:0/1320030140 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7b380186c0 0x7f7b3801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:52.314 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.310 7f7b5d33b700 1 --2- 172.21.2.112:0/1320030140 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7b380186c0 0x7f7b3801abc0 crc :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:52.342 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.341 7f7b5f51f700 1 -- 172.21.2.112:0/1320030140 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b58085e50 con 0x7f7b580863d0 2021-02-10T11:44:52.344 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.343 7f7b49ffb700 1 -- 172.21.2.112:0/1320030140 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7b50000cc0 con 0x7f7b580863d0 2021-02-10T11:44:52.473 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.471 7f7b5f51f700 1 -- 172.21.2.112:0/1320030140 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f7b580fe1c0 con 0x7f7b580863d0 2021-02-10T11:44:52.473 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.472 7f7b49ffb700 1 -- 172.21.2.112:0/1320030140 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v151) v1 ==== 104+0+10 (secure 0 0 0) 0x7f7b5001f860 con 0x7f7b580863d0 2021-02-10T11:44:52.474 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:44:52.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.474 7f7b5f51f700 1 -- 172.21.2.112:0/1320030140 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7b380186c0 msgr2=0x7f7b3801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:52.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.474 7f7b5f51f700 1 --2- 172.21.2.112:0/1320030140 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7b380186c0 0x7f7b3801abc0 crc :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:52.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.474 7f7b5f51f700 1 -- 172.21.2.112:0/1320030140 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7b580863d0 msgr2=0x7f7b580867d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:52.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.474 7f7b5f51f700 1 --2- 172.21.2.112:0/1320030140 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7b580863d0 0x7f7b580867d0 secure :-1 s=READY pgs=2674 cs=0 l=1 rev1=1 rx=0x7f7b50009940 tx=0x7f7b5000bcf0).stop 2021-02-10T11:44:52.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.474 7f7b5f51f700 1 -- 172.21.2.112:0/1320030140 shutdown_connections 2021-02-10T11:44:52.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.474 7f7b5f51f700 1 --2- 172.21.2.112:0/1320030140 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7b380186c0 0x7f7b3801abc0 unknown :-1 s=CLOSED pgs=1222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:52.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.474 7f7b5f51f700 1 --2- 172.21.2.112:0/1320030140 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7b581a44c0 0x7f7b581a48c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:52.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.474 7f7b5f51f700 1 --2- 172.21.2.112:0/1320030140 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7b580863d0 0x7f7b580867d0 unknown :-1 s=CLOSED pgs=2674 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:52.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.474 7f7b5f51f700 1 --2- 172.21.2.112:0/1320030140 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7b580c3820 0x7f7b580a7950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:52.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.474 7f7b5f51f700 1 -- 172.21.2.112:0/1320030140 >> 172.21.2.112:0/1320030140 conn(0x7f7b580872c0 msgr2=0x7f7b58095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:52.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.474 7f7b5f51f700 1 -- 172.21.2.112:0/1320030140 shutdown_connections 2021-02-10T11:44:52.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.474 7f7b5f51f700 1 -- 172.21.2.112:0/1320030140 wait complete. 2021-02-10T11:44:52.484 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:44:52.914 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.897 7fca3a8be700 1 Processor -- start 2021-02-10T11:44:52.914 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.897 7fca3a8be700 1 -- start start 2021-02-10T11:44:52.914 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.897 7fca3a8be700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fca340d1c70 0x7fca340cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:52.914 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.897 7fca3a8be700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fca340cf580 0x7fca340cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:52.914 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.897 7fca3a8be700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca340fe1c0 con 0x7fca340d1870 2021-02-10T11:44:52.914 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.897 7fca3a8be700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca341b0cb0 con 0x7fca340cf580 2021-02-10T11:44:52.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.897 7fca3a8be700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca34109a60 con 0x7fca340d1c70 2021-02-10T11:44:52.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.898 7fca38edb700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fca340cf580 0x7fca340cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:52.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.898 7fca38edb700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fca340cf580 0x7fca340cf980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:58314/0 (socket says 172.21.2.112:58314) 2021-02-10T11:44:52.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.898 7fca38edb700 1 -- 172.21.2.112:0/1103014534 learned_addr learned my addr 172.21.2.112:0/1103014534 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:44:52.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.898 7fca38edb700 1 -- 172.21.2.112:0/1103014534 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fca340d1c70 msgr2=0x7fca340cef60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:44:52.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.898 7fca38edb700 1 --2- 172.21.2.112:0/1103014534 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fca340d1c70 0x7fca340cef60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:52.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.898 7fca38edb700 1 -- 172.21.2.112:0/1103014534 >> v1:172.21.2.120:6789/0 conn(0x7fca340d1870 legacy=0x7fca341b04c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:44:52.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.898 7fca38edb700 1 -- 172.21.2.112:0/1103014534 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca34088070 con 0x7fca340cf580 2021-02-10T11:44:52.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.898 7fca38edb700 1 --2- 172.21.2.112:0/1103014534 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fca340cf580 0x7fca340cf980 secure :-1 s=READY pgs=2675 cs=0 l=1 rev1=1 rx=0x7fca2c006f00 tx=0x7fca2c009f10).ready entity=mon.0 client_cookie=f00b1dca93790465 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:52.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.898 7fca32ffd700 1 -- 172.21.2.112:0/1103014534 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fca2c00b3b0 con 0x7fca340cf580 2021-02-10T11:44:52.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.898 7fca32ffd700 1 -- 172.21.2.112:0/1103014534 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca2c00b630 con 0x7fca340cf580 2021-02-10T11:44:52.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.898 7fca32ffd700 1 -- 172.21.2.112:0/1103014534 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fca2c00bb50 con 0x7fca340cf580 2021-02-10T11:44:52.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.898 7fca3a8be700 1 -- 172.21.2.112:0/1103014534 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fca340cf580 msgr2=0x7fca340cf980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:52.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.898 7fca3a8be700 1 --2- 172.21.2.112:0/1103014534 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fca340cf580 0x7fca340cf980 secure :-1 s=READY pgs=2675 cs=0 l=1 rev1=1 rx=0x7fca2c006f00 tx=0x7fca2c009f10).stop 2021-02-10T11:44:52.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.899 7fca3a8be700 1 -- 172.21.2.112:0/1103014534 shutdown_connections 2021-02-10T11:44:52.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.899 7fca3a8be700 1 --2- 172.21.2.112:0/1103014534 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fca340cf580 0x7fca340cf980 unknown :-1 s=CLOSED pgs=2675 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:52.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.899 7fca3a8be700 1 --2- 172.21.2.112:0/1103014534 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fca340d1c70 0x7fca340cef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:52.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.899 7fca3a8be700 1 -- 172.21.2.112:0/1103014534 >> 172.21.2.112:0/1103014534 conn(0x7fca340ce420 msgr2=0x7fca340d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:52.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.899 7fca3a8be700 1 -- 172.21.2.112:0/1103014534 shutdown_connections 2021-02-10T11:44:52.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.899 7fca3a8be700 1 -- 172.21.2.112:0/1103014534 wait complete. 2021-02-10T11:44:52.918 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.913 7fca3a8be700 1 Processor -- start 2021-02-10T11:44:52.918 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.913 7fca3a8be700 1 -- start start 2021-02-10T11:44:52.918 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.913 7fca3a8be700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fca340c3820 0x7fca340a7950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:52.918 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.913 7fca3a8be700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fca341a44c0 0x7fca341a48c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:52.918 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.913 7fca3a8be700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fca340863d0 0x7fca340867d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:52.918 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.913 7fca3a8be700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca34085ed0 con 0x7fca341a44c0 2021-02-10T11:44:52.918 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.913 7fca3a8be700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca340c0740 con 0x7fca340c3820 2021-02-10T11:44:52.919 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.913 7fca3a8be700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca34086df0 con 0x7fca340863d0 2021-02-10T11:44:52.919 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.914 7fca337fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fca341a44c0 0x7fca341a48c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:52.919 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.914 7fca337fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fca341a44c0 0x7fca341a48c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:58316/0 (socket says 172.21.2.112:58316) 2021-02-10T11:44:52.919 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.914 7fca337fe700 1 -- 172.21.2.112:0/3421519366 learned_addr learned my addr 172.21.2.112:0/3421519366 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:44:52.919 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.914 7fca33fff700 1 --2- 172.21.2.112:0/3421519366 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fca340c3820 0x7fca340a7950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:52.919 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.914 7fca38edb700 1 --2- 172.21.2.112:0/3421519366 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fca340863d0 0x7fca340867d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:52.919 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.914 7fca337fe700 1 -- 172.21.2.112:0/3421519366 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fca340863d0 msgr2=0x7fca340867d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:52.919 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.914 7fca337fe700 1 --2- 172.21.2.112:0/3421519366 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fca340863d0 0x7fca340867d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:52.919 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.914 7fca337fe700 1 -- 172.21.2.112:0/3421519366 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fca340c3820 msgr2=0x7fca340a7950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:52.919 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.914 7fca337fe700 1 --2- 172.21.2.112:0/3421519366 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fca340c3820 0x7fca340a7950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:52.919 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.914 7fca337fe700 1 -- 172.21.2.112:0/3421519366 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca34087010 con 0x7fca341a44c0 2021-02-10T11:44:52.919 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.914 7fca337fe700 1 --2- 172.21.2.112:0/3421519366 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fca341a44c0 0x7fca341a48c0 secure :-1 s=READY pgs=2676 cs=0 l=1 rev1=1 rx=0x7fca2400b5c0 tx=0x7fca2400b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:52.919 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.914 7fca30ff9700 1 -- 172.21.2.112:0/3421519366 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fca2400bd50 con 0x7fca341a44c0 2021-02-10T11:44:52.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.914 7fca30ff9700 1 -- 172.21.2.112:0/3421519366 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca24006030 con 0x7fca341a44c0 2021-02-10T11:44:52.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.914 7fca3a8be700 1 -- 172.21.2.112:0/3421519366 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca34087010 con 0x7fca341a44c0 2021-02-10T11:44:52.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.914 7fca30ff9700 1 -- 172.21.2.112:0/3421519366 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fca240054d0 con 0x7fca341a44c0 2021-02-10T11:44:52.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.914 7fca3a8be700 1 -- 172.21.2.112:0/3421519366 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca34087010 con 0x7fca341a44c0 2021-02-10T11:44:52.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.915 7fca30ff9700 1 -- 172.21.2.112:0/3421519366 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fca24006210 con 0x7fca341a44c0 2021-02-10T11:44:52.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.915 7fca30ff9700 1 --2- 172.21.2.112:0/3421519366 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fca18018730 0x7fca1801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:52.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.915 7fca30ff9700 1 -- 172.21.2.112:0/3421519366 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fca24032610 con 0x7fca341a44c0 2021-02-10T11:44:52.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.915 7fca33fff700 1 --2- 172.21.2.112:0/3421519366 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fca18018730 0x7fca1801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:52.920 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.916 7fca33fff700 1 --2- 172.21.2.112:0/3421519366 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fca18018730 0x7fca1801ac10 crc :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:52.948 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.947 7fca3a8be700 1 -- 172.21.2.112:0/3421519366 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca340fe1c0 con 0x7fca341a44c0 2021-02-10T11:44:52.950 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:52.949 7fca30ff9700 1 -- 172.21.2.112:0/3421519366 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fca24006b80 con 0x7fca341a44c0 2021-02-10T11:44:53.077 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:53.076 7fca3a8be700 1 -- 172.21.2.112:0/3421519366 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7fca340fe1c0 con 0x7fca341a44c0 2021-02-10T11:44:53.078 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:53.076 7fca30ff9700 1 -- 172.21.2.112:0/3421519366 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v151) v1 ==== 100+0+10 (secure 0 0 0) 0x7fca2400bd50 con 0x7fca341a44c0 2021-02-10T11:44:53.078 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:44:53.079 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:53.078 7fca3a8be700 1 -- 172.21.2.112:0/3421519366 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fca18018730 msgr2=0x7fca1801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:53.080 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:53.078 7fca3a8be700 1 --2- 172.21.2.112:0/3421519366 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fca18018730 0x7fca1801ac10 crc :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:53.080 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:53.078 7fca3a8be700 1 -- 172.21.2.112:0/3421519366 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fca341a44c0 msgr2=0x7fca341a48c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:53.080 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:53.078 7fca3a8be700 1 --2- 172.21.2.112:0/3421519366 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fca341a44c0 0x7fca341a48c0 secure :-1 s=READY pgs=2676 cs=0 l=1 rev1=1 rx=0x7fca2400b5c0 tx=0x7fca2400b960).stop 2021-02-10T11:44:53.080 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:53.078 7fca3a8be700 1 -- 172.21.2.112:0/3421519366 shutdown_connections 2021-02-10T11:44:53.080 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:53.078 7fca3a8be700 1 --2- 172.21.2.112:0/3421519366 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fca18018730 0x7fca1801ac10 unknown :-1 s=CLOSED pgs=1223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:53.080 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:53.078 7fca3a8be700 1 --2- 172.21.2.112:0/3421519366 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fca340863d0 0x7fca340867d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:53.080 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:53.078 7fca3a8be700 1 --2- 172.21.2.112:0/3421519366 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fca341a44c0 0x7fca341a48c0 unknown :-1 s=CLOSED pgs=2676 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:53.080 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:53.078 7fca3a8be700 1 --2- 172.21.2.112:0/3421519366 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fca340c3820 0x7fca340a7950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:53.080 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:53.078 7fca3a8be700 1 -- 172.21.2.112:0/3421519366 >> 172.21.2.112:0/3421519366 conn(0x7fca340872c0 msgr2=0x7fca34095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:53.081 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:53.079 7fca3a8be700 1 -- 172.21.2.112:0/3421519366 shutdown_connections 2021-02-10T11:44:53.081 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:53.079 7fca3a8be700 1 -- 172.21.2.112:0/3421519366 wait complete. 2021-02-10T11:44:53.088 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:44:53.089 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:44:53.517 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:53.498 7fa611782700 1 Processor -- start 2021-02-10T11:44:53.517 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:53.498 7fa611782700 1 -- start start 2021-02-10T11:44:53.517 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:53.499 7fa611782700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fa60c0d0340 0x7fa60c0ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:53.517 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:53.499 7fa611782700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fa60c0d1c70 0x7fa60c0efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:53.517 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:53.499 7fa611782700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa60c0f0b10 con 0x7fa60c0d1870 2021-02-10T11:44:53.517 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:53.499 7fa611782700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa60c0e5ed0 con 0x7fa60c0d0340 2021-02-10T11:44:53.517 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:53.499 7fa611782700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa60c1b0be0 con 0x7fa60c0d1c70 2021-02-10T11:44:53.517 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:53.499 7fa60ad73700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fa60c0d0340 0x7fa60c0ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:53.518 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:53.499 7fa60ad73700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fa60c0d0340 0x7fa60c0ef640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:58324/0 (socket says 172.21.2.112:58324) 2021-02-10T11:44:53.518 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:53.499 7fa60ad73700 1 -- 172.21.2.112:0/520290021 learned_addr learned my addr 172.21.2.112:0/520290021 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:44:53.518 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:53.499 7fa60bd75700 1 --2- 172.21.2.112:0/520290021 >> v2:172.21.2.120:3300/0 conn(0x7fa60c0d1c70 0x7fa60c0efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:53.518 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:53.499 7fa60ad73700 1 -- 172.21.2.112:0/520290021 >> v2:172.21.2.120:3300/0 conn(0x7fa60c0d1c70 msgr2=0x7fa60c0efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:53.518 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:53.499 7fa60ad73700 1 --2- 172.21.2.112:0/520290021 >> v2:172.21.2.120:3300/0 conn(0x7fa60c0d1c70 0x7fa60c0efc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:53.518 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:53.499 7fa60ad73700 1 -- 172.21.2.112:0/520290021 >> v1:172.21.2.120:6789/0 conn(0x7fa60c0d1870 legacy=0x7fa60c1b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:53.518 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:53.499 7fa60ad73700 1 -- 172.21.2.112:0/520290021 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa60c0f20a0 con 0x7fa60c0d0340 2021-02-10T11:44:53.518 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:53.499 7fa60ad73700 1 --2- 172.21.2.112:0/520290021 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa60c0d0340 0x7fa60c0ef640 secure :-1 s=READY pgs=2677 cs=0 l=1 rev1=1 rx=0x7fa5fc006700 tx=0x7fa5fc009b80).ready entity=mon.0 client_cookie=f2ef63ddb9514380 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:53.518 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:53.500 7fa60a572700 1 -- 172.21.2.112:0/520290021 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa5fc00c0c0 con 0x7fa60c0d0340 2021-02-10T11:44:53.518 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:53.500 7fa60a572700 1 -- 172.21.2.112:0/520290021 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5fc00c340 con 0x7fa60c0d0340 2021-02-10T11:44:53.518 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:53.500 7fa60a572700 1 -- 172.21.2.112:0/520290021 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa5fc00c860 con 0x7fa60c0d0340 2021-02-10T11:44:53.518 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:53.500 7fa611782700 1 -- 172.21.2.112:0/520290021 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa60c0d0340 msgr2=0x7fa60c0ef640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:53.518 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:53.500 7fa611782700 1 --2- 172.21.2.112:0/520290021 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa60c0d0340 0x7fa60c0ef640 secure :-1 s=READY pgs=2677 cs=0 l=1 rev1=1 rx=0x7fa5fc006700 tx=0x7fa5fc009b80).stop 2021-02-10T11:44:53.519 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:53.500 7fa611782700 1 -- 172.21.2.112:0/520290021 shutdown_connections 2021-02-10T11:44:53.519 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:53.500 7fa611782700 1 --2- 172.21.2.112:0/520290021 >> v2:172.21.2.120:3300/0 conn(0x7fa60c0d1c70 0x7fa60c0efc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:53.519 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:53.500 7fa611782700 1 --2- 172.21.2.112:0/520290021 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa60c0d0340 0x7fa60c0ef640 unknown :-1 s=CLOSED pgs=2677 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:53.519 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:53.500 7fa611782700 1 -- 172.21.2.112:0/520290021 >> 172.21.2.112:0/520290021 conn(0x7fa60c0ce450 msgr2=0x7fa60c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:53.519 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:53.500 7fa611782700 1 -- 172.21.2.112:0/520290021 shutdown_connections 2021-02-10T11:44:53.519 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:53.500 7fa611782700 1 -- 172.21.2.112:0/520290021 wait complete. 2021-02-10T11:44:53.520 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:53.515 7fa611782700 1 Processor -- start 2021-02-10T11:44:53.520 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:53.515 7fa611782700 1 -- start start 2021-02-10T11:44:53.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:53.515 7fa611782700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa60c093f10 0x7fa60c094310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:53.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:53.516 7fa611782700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa60c0ae9e0 0x7fa60c0aede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:53.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:53.516 7fa611782700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa60c092d90 0x7fa60c093190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:53.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:53.516 7fa611782700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa60c19ebc0 con 0x7fa60c092d90 2021-02-10T11:44:53.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:53.516 7fa611782700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa60c1b8760 con 0x7fa60c0ae9e0 2021-02-10T11:44:53.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:53.516 7fa611782700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa60c0937b0 con 0x7fa60c093f10 2021-02-10T11:44:53.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:53.516 7fa60bd75700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa60c092d90 0x7fa60c093190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:53.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:53.516 7fa60bd75700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa60c092d90 0x7fa60c093190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:58334/0 (socket says 172.21.2.112:58334) 2021-02-10T11:44:53.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:53.516 7fa60bd75700 1 -- 172.21.2.112:0/3613244725 learned_addr learned my addr 172.21.2.112:0/3613244725 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:44:53.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:53.516 7fa60bd75700 1 -- 172.21.2.112:0/3613244725 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa60c093f10 msgr2=0x7fa60c094310 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:44:53.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:53.516 7fa60bd75700 1 --2- 172.21.2.112:0/3613244725 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa60c093f10 0x7fa60c094310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:53.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:53.516 7fa60bd75700 1 -- 172.21.2.112:0/3613244725 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa60c0ae9e0 msgr2=0x7fa60c0aede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:53.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:53.516 7fa60bd75700 1 --2- 172.21.2.112:0/3613244725 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa60c0ae9e0 0x7fa60c0aede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:53.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:53.516 7fa60bd75700 1 -- 172.21.2.112:0/3613244725 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa60c0939d0 con 0x7fa60c092d90 2021-02-10T11:44:53.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:53.516 7fa60bd75700 1 --2- 172.21.2.112:0/3613244725 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa60c092d90 0x7fa60c093190 secure :-1 s=READY pgs=2678 cs=0 l=1 rev1=1 rx=0x7fa60400b970 tx=0x7fa60400bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:53.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:53.516 7fa603fff700 1 -- 172.21.2.112:0/3613244725 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa604000f90 con 0x7fa60c092d90 2021-02-10T11:44:53.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:53.517 7fa603fff700 1 -- 172.21.2.112:0/3613244725 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa604006aa0 con 0x7fa60c092d90 2021-02-10T11:44:53.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:53.517 7fa603fff700 1 -- 172.21.2.112:0/3613244725 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa604001680 con 0x7fa60c092d90 2021-02-10T11:44:53.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:53.517 7fa611782700 1 -- 172.21.2.112:0/3613244725 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa60c0939d0 con 0x7fa60c092d90 2021-02-10T11:44:53.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:53.517 7fa611782700 1 -- 172.21.2.112:0/3613244725 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa60c189cd0 con 0x7fa60c092d90 2021-02-10T11:44:53.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:53.517 7fa603fff700 1 -- 172.21.2.112:0/3613244725 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa6040011f0 con 0x7fa60c092d90 2021-02-10T11:44:53.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:53.517 7fa603fff700 1 --2- 172.21.2.112:0/3613244725 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa5f00186c0 0x7fa5f001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:53.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:53.517 7fa603fff700 1 -- 172.21.2.112:0/3613244725 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fa6040328f0 con 0x7fa60c092d90 2021-02-10T11:44:53.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:53.518 7fa60b574700 1 --2- 172.21.2.112:0/3613244725 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa5f00186c0 0x7fa5f001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:53.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:53.519 7fa60b574700 1 --2- 172.21.2.112:0/3613244725 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa5f00186c0 0x7fa5f001abc0 crc :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:53.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:53.549 7fa611782700 1 -- 172.21.2.112:0/3613244725 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa60c0ebc60 con 0x7fa60c092d90 2021-02-10T11:44:53.554 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:53.552 7fa603fff700 1 -- 172.21.2.112:0/3613244725 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa604000e30 con 0x7fa60c092d90 2021-02-10T11:44:53.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:53.681 7fa611782700 1 -- 172.21.2.112:0/3613244725 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa60c0f0b10 con 0x7fa60c092d90 2021-02-10T11:44:53.684 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:53.681 7fa603fff700 1 -- 172.21.2.112:0/3613244725 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 130 v130) v1 ==== 95+0+2108 (secure 0 0 0) 0x7fa6040011f0 con 0x7fa60c092d90 2021-02-10T11:44:53.684 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:44:53.684 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":130,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":12006,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/1015001314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":1015001314},{"type":"v1","addr":"172.21.2.112:6833","nonce":1015001314}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":126}],"filesystems":[{"mdsmap":{"epoch":130,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:44:43.721296","modified":"2021-02-10 11:44:45.730948","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7268},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7268":{"gid":7268,"name":"b","rank":0,"incarnation":127,"state":"up:active","state_seq":3,"addr":"172.21.2.120:6835/3799929082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":3799929082},{"type":"v1","addr":"172.21.2.120:6835","nonce":3799929082}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":16}]} 2021-02-10T11:44:53.685 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 130 2021-02-10T11:44:53.686 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:53.683 7fa611782700 1 -- 172.21.2.112:0/3613244725 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa5f00186c0 msgr2=0x7fa5f001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:53.686 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:53.683 7fa611782700 1 --2- 172.21.2.112:0/3613244725 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa5f00186c0 0x7fa5f001abc0 crc :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:53.686 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:53.683 7fa611782700 1 -- 172.21.2.112:0/3613244725 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa60c092d90 msgr2=0x7fa60c093190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:53.686 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:53.683 7fa611782700 1 --2- 172.21.2.112:0/3613244725 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa60c092d90 0x7fa60c093190 secure :-1 s=READY pgs=2678 cs=0 l=1 rev1=1 rx=0x7fa60400b970 tx=0x7fa60400bd10).stop 2021-02-10T11:44:53.686 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:53.684 7fa611782700 1 -- 172.21.2.112:0/3613244725 shutdown_connections 2021-02-10T11:44:53.686 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:53.684 7fa611782700 1 --2- 172.21.2.112:0/3613244725 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa5f00186c0 0x7fa5f001abc0 unknown :-1 s=CLOSED pgs=1224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:53.686 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:53.684 7fa611782700 1 --2- 172.21.2.112:0/3613244725 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa60c0ae9e0 0x7fa60c0aede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:53.686 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:53.684 7fa611782700 1 --2- 172.21.2.112:0/3613244725 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa60c092d90 0x7fa60c093190 unknown :-1 s=CLOSED pgs=2678 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:53.686 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:53.684 7fa611782700 1 --2- 172.21.2.112:0/3613244725 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa60c093f10 0x7fa60c094310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:53.686 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:53.684 7fa611782700 1 -- 172.21.2.112:0/3613244725 >> 172.21.2.112:0/3613244725 conn(0x7fa60c19f8a0 msgr2=0x7fa60c094980 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:53.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:53.684 7fa611782700 1 -- 172.21.2.112:0/3613244725 shutdown_connections 2021-02-10T11:44:53.687 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:53.684 7fa611782700 1 -- 172.21.2.112:0/3613244725 wait complete. 2021-02-10T11:44:53.896 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 130, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:44:43.721296', 'modified': '2021-02-10 11:44:45.730948', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7268}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7268': {'gid': 7268, 'name': 'b', 'rank': 0, 'incarnation': 127, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.120:6835/3799929082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.120:6834', 'nonce': 3799929082}, {'type': 'v1', 'addr': '172.21.2.120:6835', 'nonce': 3799929082}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [33], 'metadata_pool': 32, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:44:53.896 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:44:53.896 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:44:54.295 INFO:teuthology.orchestra.run.gibba020.stdout:{ 2021-02-10T11:44:54.295 INFO:teuthology.orchestra.run.gibba020.stdout: "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", 2021-02-10T11:44:54.295 INFO:teuthology.orchestra.run.gibba020.stdout: "whoami": 0, 2021-02-10T11:44:54.296 INFO:teuthology.orchestra.run.gibba020.stdout: "id": 7268, 2021-02-10T11:44:54.296 INFO:teuthology.orchestra.run.gibba020.stdout: "want_state": "up:active", 2021-02-10T11:44:54.296 INFO:teuthology.orchestra.run.gibba020.stdout: "state": "up:active", 2021-02-10T11:44:54.296 INFO:teuthology.orchestra.run.gibba020.stdout: "rank_uptime": 10.570772960999999, 2021-02-10T11:44:54.296 INFO:teuthology.orchestra.run.gibba020.stdout: "mdsmap_epoch": 130, 2021-02-10T11:44:54.296 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch": 151, 2021-02-10T11:44:54.296 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch_barrier": 151, 2021-02-10T11:44:54.296 INFO:teuthology.orchestra.run.gibba020.stdout: "uptime": 16.214403539999999 2021-02-10T11:44:54.296 INFO:teuthology.orchestra.run.gibba020.stdout:} 2021-02-10T11:44:54.299 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", "whoami": 0, "id": 7268, "want_state": "up:active", "state": "up:active", "rank_uptime": 10.570772960999999, "mdsmap_epoch": 130, "osdmap_epoch": 151, "osdmap_epoch_barrier": 151, "uptime": 16.214403539999999 } 2021-02-10T11:44:54.299 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:44:54.300 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2021-02-10T11:44:54.300 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@gibba012.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2021-02-10T11:44:54.300 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:44:54.300 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:44:54.636 DEBUG:teuthology.orchestra.run.gibba012:> sudo modprobe fuse 2021-02-10T11:44:54.981 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:44:54.982 DEBUG:teuthology.orchestra.run.gibba012:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:44:55.169 DEBUG:teuthology.orchestra.run.gibba012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:44:55.173 DEBUG:teuthology.orchestra.run.gibba020:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:44:55.326 INFO:teuthology.orchestra.run.gibba012.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:44:55.326 INFO:teuthology.orchestra.run.gibba012.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:44:55.327 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:44:55.327 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:44:55.327 DEBUG:teuthology.orchestra.run.gibba012:> ls /sys/fs/fuse/connections 2021-02-10T11:44:55.662 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2021-02-10T11:44:55.663 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:44:55.666 DEBUG:teuthology.orchestra.run.gibba012:> sudo modprobe fuse 2021-02-10T11:44:56.011 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:44:56.011 DEBUG:teuthology.orchestra.run.gibba012:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:44:56.077 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba012.stderr:ceph-fuse[176523]: starting ceph client 2021-02-10T11:44:56.077 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba012.stderr:2021-02-10 11:44:56.074 7f18955c6f80 -1 init, newargv = 0x55ac23d65960 newargc=9 2021-02-10T11:44:56.082 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba012.stderr:ceph-fuse[176523]: starting fuse 2021-02-10T11:44:56.352 INFO:teuthology.orchestra.run.gibba012.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:44:56.353 INFO:teuthology.orchestra.run.gibba012.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:44:56.353 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:44:56.354 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:44:56.354 DEBUG:teuthology.orchestra.run.gibba012:> ls /sys/fs/fuse/connections 2021-02-10T11:44:56.689 INFO:teuthology.orchestra.run.gibba012.stdout:42 2021-02-10T11:44:56.690 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2021-02-10T11:44:56.690 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:44:56.690 DEBUG:teuthology.orchestra.run.gibba012:> sudo python3 -c ' 2021-02-10T11:44:56.690 DEBUG:teuthology.orchestra.run.gibba012:> import glob 2021-02-10T11:44:56.690 DEBUG:teuthology.orchestra.run.gibba012:> import re 2021-02-10T11:44:56.690 DEBUG:teuthology.orchestra.run.gibba012:> import os 2021-02-10T11:44:56.690 DEBUG:teuthology.orchestra.run.gibba012:> import subprocess 2021-02-10T11:44:56.690 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:44:56.690 DEBUG:teuthology.orchestra.run.gibba012:> def find_socket(client_name): 2021-02-10T11:44:56.691 DEBUG:teuthology.orchestra.run.gibba012:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2021-02-10T11:44:56.691 DEBUG:teuthology.orchestra.run.gibba012:> files = glob.glob(asok_path) 2021-02-10T11:44:56.691 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:44:56.691 DEBUG:teuthology.orchestra.run.gibba012:> # Given a non-glob path, it better be there 2021-02-10T11:44:56.691 DEBUG:teuthology.orchestra.run.gibba012:> if "*" not in asok_path: 2021-02-10T11:44:56.691 DEBUG:teuthology.orchestra.run.gibba012:> assert(len(files) == 1) 2021-02-10T11:44:56.691 DEBUG:teuthology.orchestra.run.gibba012:> return files[0] 2021-02-10T11:44:56.691 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:44:56.691 DEBUG:teuthology.orchestra.run.gibba012:> for f in files: 2021-02-10T11:44:56.691 DEBUG:teuthology.orchestra.run.gibba012:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:44:56.691 DEBUG:teuthology.orchestra.run.gibba012:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:44:56.691 DEBUG:teuthology.orchestra.run.gibba012:> return f 2021-02-10T11:44:56.691 DEBUG:teuthology.orchestra.run.gibba012:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:44:56.691 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:44:56.691 DEBUG:teuthology.orchestra.run.gibba012:> print(find_socket("client.0")) 2021-02-10T11:44:56.692 DEBUG:teuthology.orchestra.run.gibba012:> ' 2021-02-10T11:44:57.053 INFO:teuthology.orchestra.run.gibba012.stdout:/var/run/ceph/ceph-client.0.176523.asok 2021-02-10T11:44:57.055 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.176523.asok 2021-02-10T11:44:57.055 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:44:57.056 DEBUG:teuthology.orchestra.run.gibba012:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.176523.asok status 2021-02-10T11:44:57.452 INFO:teuthology.orchestra.run.gibba012.stdout:{ 2021-02-10T11:44:57.452 INFO:teuthology.orchestra.run.gibba012.stdout: "metadata": { 2021-02-10T11:44:57.452 INFO:teuthology.orchestra.run.gibba012.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:44:57.452 INFO:teuthology.orchestra.run.gibba012.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:44:57.452 INFO:teuthology.orchestra.run.gibba012.stdout: "entity_id": "0", 2021-02-10T11:44:57.452 INFO:teuthology.orchestra.run.gibba012.stdout: "hostname": "gibba012", 2021-02-10T11:44:57.452 INFO:teuthology.orchestra.run.gibba012.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2021-02-10T11:44:57.453 INFO:teuthology.orchestra.run.gibba012.stdout: "pid": "176523", 2021-02-10T11:44:57.453 INFO:teuthology.orchestra.run.gibba012.stdout: "root": "/" 2021-02-10T11:44:57.453 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:44:57.453 INFO:teuthology.orchestra.run.gibba012.stdout: "dentry_count": 0, 2021-02-10T11:44:57.453 INFO:teuthology.orchestra.run.gibba012.stdout: "dentry_pinned_count": 0, 2021-02-10T11:44:57.453 INFO:teuthology.orchestra.run.gibba012.stdout: "id": 12150, 2021-02-10T11:44:57.453 INFO:teuthology.orchestra.run.gibba012.stdout: "inst": { 2021-02-10T11:44:57.453 INFO:teuthology.orchestra.run.gibba012.stdout: "name": { 2021-02-10T11:44:57.453 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "client", 2021-02-10T11:44:57.453 INFO:teuthology.orchestra.run.gibba012.stdout: "num": 12150 2021-02-10T11:44:57.453 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:44:57.453 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": { 2021-02-10T11:44:57.453 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:44:57.454 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:0", 2021-02-10T11:44:57.454 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 3586593357 2021-02-10T11:44:57.454 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:44:57.454 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:44:57.454 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": { 2021-02-10T11:44:57.454 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:44:57.454 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:0", 2021-02-10T11:44:57.454 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 3586593357 2021-02-10T11:44:57.454 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:44:57.454 INFO:teuthology.orchestra.run.gibba012.stdout: "inst_str": "client.12150 172.21.2.112:0/3586593357", 2021-02-10T11:44:57.454 INFO:teuthology.orchestra.run.gibba012.stdout: "addr_str": "172.21.2.112:0/3586593357", 2021-02-10T11:44:57.454 INFO:teuthology.orchestra.run.gibba012.stdout: "inode_count": 1, 2021-02-10T11:44:57.455 INFO:teuthology.orchestra.run.gibba012.stdout: "mds_epoch": 130, 2021-02-10T11:44:57.455 INFO:teuthology.orchestra.run.gibba012.stdout: "osd_epoch": 151, 2021-02-10T11:44:57.455 INFO:teuthology.orchestra.run.gibba012.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:44:57.455 INFO:teuthology.orchestra.run.gibba012.stdout: "blacklisted": false 2021-02-10T11:44:57.455 INFO:teuthology.orchestra.run.gibba012.stdout:} 2021-02-10T11:44:57.456 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:44:57.457 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:44:57.457 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:44:57.822 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:44:57.823 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:44:57.823 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:44:57.823 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:44:58.170 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:44:58.597 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:58.579 7f473ba9d700 1 Processor -- start 2021-02-10T11:44:58.598 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:58.579 7f473ba9d700 1 -- start start 2021-02-10T11:44:58.598 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:58.580 7f473ba9d700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f47340d1870 0x7f47341b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:58.598 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:58.580 7f473ba9d700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47340f0b10 con 0x7f47340f0650 2021-02-10T11:44:58.598 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:58.580 7f473ba9d700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47340d0310 con 0x7f47341b0a10 2021-02-10T11:44:58.598 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:58.580 7f473ba9d700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47340d0530 con 0x7f47340d1870 2021-02-10T11:44:58.598 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:58.580 7f473a0ba700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f47340f0650 0x7f47340cfb20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:57820/0 (socket says 172.21.2.112:57820) 2021-02-10T11:44:58.598 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:58.580 7f473a0ba700 1 -- 172.21.2.112:0/3797190949 learned_addr learned my addr 172.21.2.112:0/3797190949 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:44:58.598 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:58.580 7f47398b9700 1 --2- 172.21.2.112:0/3797190949 >> v2:172.21.2.120:3300/0 conn(0x7f47340d1870 0x7f47341b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:58.598 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:58.580 7f47388b7700 1 -- 172.21.2.112:0/3797190949 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 80235819 0 0) 0x7f472c001500 con 0x7f47340f0650 2021-02-10T11:44:58.598 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:58.580 7f47388b7700 1 -- 172.21.2.112:0/3797190949 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4720002fe0 con 0x7f47340f0650 2021-02-10T11:44:58.598 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:58.581 7f47398b9700 1 -- 172.21.2.112:0/3797190949 >> v1:172.21.2.120:6789/0 conn(0x7f47341b0a10 legacy=0x7f47340cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:58.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:58.581 7f47398b9700 1 -- 172.21.2.112:0/3797190949 >> v1:172.21.2.112:6789/0 conn(0x7f47340f0650 legacy=0x7f47340cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:58.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:58.581 7f47398b9700 1 -- 172.21.2.112:0/3797190949 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47340d1c70 con 0x7f47340d1870 2021-02-10T11:44:58.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:58.581 7f47398b9700 1 --2- 172.21.2.112:0/3797190949 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f47340d1870 0x7f47341b03f0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 rx=0x7f4730006650 tx=0x7f4730009a80).ready entity=mon.1 client_cookie=3216da29ac3f032c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:58.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:58.581 7f47388b7700 1 -- 172.21.2.112:0/3797190949 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f473000c0c0 con 0x7f47340d1870 2021-02-10T11:44:58.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:58.581 7f47388b7700 1 -- 172.21.2.112:0/3797190949 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f473000c340 con 0x7f47340d1870 2021-02-10T11:44:58.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:58.581 7f47388b7700 1 -- 172.21.2.112:0/3797190949 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f473000c860 con 0x7f47340d1870 2021-02-10T11:44:58.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:58.581 7f473ba9d700 1 -- 172.21.2.112:0/3797190949 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f47340d1870 msgr2=0x7f47341b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:58.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:58.581 7f473ba9d700 1 --2- 172.21.2.112:0/3797190949 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f47340d1870 0x7f47341b03f0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 rx=0x7f4730006650 tx=0x7f4730009a80).stop 2021-02-10T11:44:58.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:58.582 7f473ba9d700 1 -- 172.21.2.112:0/3797190949 shutdown_connections 2021-02-10T11:44:58.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:58.582 7f473ba9d700 1 --2- 172.21.2.112:0/3797190949 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f47340d1870 0x7f47341b03f0 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:58.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:58.582 7f473ba9d700 1 -- 172.21.2.112:0/3797190949 >> 172.21.2.112:0/3797190949 conn(0x7f47340ce450 msgr2=0x7f47340d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:58.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:58.582 7f473ba9d700 1 -- 172.21.2.112:0/3797190949 shutdown_connections 2021-02-10T11:44:58.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:58.582 7f473ba9d700 1 -- 172.21.2.112:0/3797190949 wait complete. 2021-02-10T11:44:58.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:58.597 7f473ba9d700 1 Processor -- start 2021-02-10T11:44:58.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:58.597 7f473ba9d700 1 -- start start 2021-02-10T11:44:58.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:58.597 7f473ba9d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f473418a540 0x7f4734197480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:58.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:58.597 7f473ba9d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f473418aa50 0x7f47340b77d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:58.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:58.597 7f473ba9d700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f47340b7df0 0x7f47340b81f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:58.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:58.597 7f473ba9d700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47341a79d0 con 0x7f473418a540 2021-02-10T11:44:58.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:58.597 7f473ba9d700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47341a9e20 con 0x7f47340b7df0 2021-02-10T11:44:58.602 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:58.597 7f473ba9d700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47341aa780 con 0x7f473418aa50 2021-02-10T11:44:58.602 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:58.597 7f47398b9700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f473418a540 0x7f4734197480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:58.602 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:58.597 7f47398b9700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f473418a540 0x7f4734197480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:58362/0 (socket says 172.21.2.112:58362) 2021-02-10T11:44:58.602 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:58.597 7f47398b9700 1 -- 172.21.2.112:0/929404510 learned_addr learned my addr 172.21.2.112:0/929404510 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:44:58.602 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:58.598 7f47390b8700 1 --2- 172.21.2.112:0/929404510 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f473418aa50 0x7f47340b77d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:58.602 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:58.598 7f473a0ba700 1 --2- 172.21.2.112:0/929404510 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f47340b7df0 0x7f47340b81f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:58.602 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:58.598 7f47398b9700 1 -- 172.21.2.112:0/929404510 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f473418aa50 msgr2=0x7f47340b77d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:58.602 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:58.598 7f47398b9700 1 --2- 172.21.2.112:0/929404510 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f473418aa50 0x7f47340b77d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:58.602 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:58.598 7f47398b9700 1 -- 172.21.2.112:0/929404510 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f47340b7df0 msgr2=0x7f47340b81f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:58.602 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:58.598 7f47398b9700 1 --2- 172.21.2.112:0/929404510 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f47340b7df0 0x7f47340b81f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:58.602 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:58.598 7f47398b9700 1 -- 172.21.2.112:0/929404510 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47341aa9a0 con 0x7f473418a540 2021-02-10T11:44:58.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:58.598 7f47398b9700 1 --2- 172.21.2.112:0/929404510 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f473418a540 0x7f4734197480 secure :-1 s=READY pgs=2683 cs=0 l=1 rev1=1 rx=0x7f47300096a0 tx=0x7f473000b9b0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:58.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:58.598 7f47267fc700 1 -- 172.21.2.112:0/929404510 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f47300093a0 con 0x7f473418a540 2021-02-10T11:44:58.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:58.598 7f47267fc700 1 -- 172.21.2.112:0/929404510 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47300067f0 con 0x7f473418a540 2021-02-10T11:44:58.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:58.598 7f473ba9d700 1 -- 172.21.2.112:0/929404510 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47341aa9a0 con 0x7f473418a540 2021-02-10T11:44:58.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:58.598 7f47267fc700 1 -- 172.21.2.112:0/929404510 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4730002220 con 0x7f473418a540 2021-02-10T11:44:58.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:58.599 7f473ba9d700 1 -- 172.21.2.112:0/929404510 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47341aa9a0 con 0x7f473418a540 2021-02-10T11:44:58.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:58.599 7f47267fc700 1 -- 172.21.2.112:0/929404510 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f47300052e0 con 0x7f473418a540 2021-02-10T11:44:58.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:58.599 7f47267fc700 1 --2- 172.21.2.112:0/929404510 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f471c018730 0x7f471c01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:58.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:58.599 7f47267fc700 1 -- 172.21.2.112:0/929404510 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f4730032600 con 0x7f473418a540 2021-02-10T11:44:58.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:58.600 7f47390b8700 1 --2- 172.21.2.112:0/929404510 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f471c018730 0x7f471c01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:58.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:58.600 7f47390b8700 1 --2- 172.21.2.112:0/929404510 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f471c018730 0x7f471c01ac10 crc :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:58.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:58.629 7f473ba9d700 1 -- 172.21.2.112:0/929404510 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47341aa250 con 0x7f473418a540 2021-02-10T11:44:58.633 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:58.632 7f47267fc700 1 -- 172.21.2.112:0/929404510 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4730000c90 con 0x7f473418a540 2021-02-10T11:44:58.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:58.756 7f473ba9d700 1 -- 172.21.2.112:0/929404510 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f47341aa250 con 0x7f473418a540 2021-02-10T11:44:58.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:58.757 7f47267fc700 1 -- 172.21.2.112:0/929404510 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v151) v1 ==== 74+0+14007 (secure 0 0 0) 0x7f47300052e0 con 0x7f473418a540 2021-02-10T11:44:58.759 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:44:58.759 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":151,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:44:44.723715","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":32,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:44:41.688683","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"150","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":33,"pool_name":"cephfs_data","create_time":"2021-02-10 11:44:42.764257","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"151","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:44:58.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:58.759 7f473ba9d700 1 -- 172.21.2.112:0/929404510 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f471c018730 msgr2=0x7f471c01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:58.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:58.759 7f473ba9d700 1 --2- 172.21.2.112:0/929404510 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f471c018730 0x7f471c01ac10 crc :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:58.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:58.759 7f473ba9d700 1 -- 172.21.2.112:0/929404510 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f473418a540 msgr2=0x7f4734197480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:58.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:58.759 7f473ba9d700 1 --2- 172.21.2.112:0/929404510 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f473418a540 0x7f4734197480 secure :-1 s=READY pgs=2683 cs=0 l=1 rev1=1 rx=0x7f47300096a0 tx=0x7f473000b9b0).stop 2021-02-10T11:44:58.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:58.759 7f473ba9d700 1 -- 172.21.2.112:0/929404510 shutdown_connections 2021-02-10T11:44:58.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:58.759 7f473ba9d700 1 --2- 172.21.2.112:0/929404510 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f471c018730 0x7f471c01ac10 unknown :-1 s=CLOSED pgs=1225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:58.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:58.759 7f473ba9d700 1 --2- 172.21.2.112:0/929404510 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f47340b7df0 0x7f47340b81f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:58.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:58.759 7f473ba9d700 1 --2- 172.21.2.112:0/929404510 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f473418aa50 0x7f47340b77d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:58.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:58.759 7f473ba9d700 1 --2- 172.21.2.112:0/929404510 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f473418a540 0x7f4734197480 unknown :-1 s=CLOSED pgs=2683 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:58.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:58.759 7f473ba9d700 1 -- 172.21.2.112:0/929404510 >> 172.21.2.112:0/929404510 conn(0x7f47340efc70 msgr2=0x7f47340ef410 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:58.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:58.759 7f473ba9d700 1 -- 172.21.2.112:0/929404510 shutdown_connections 2021-02-10T11:44:58.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:58.759 7f473ba9d700 1 -- 172.21.2.112:0/929404510 wait complete. 2021-02-10T11:44:58.769 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:13:57.508758', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 32, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:44:41.688683', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '150', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 33, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:44:42.764257', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '151', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:44:58.769 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:44:59.201 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.183 7fcfa009b700 1 Processor -- start 2021-02-10T11:44:59.202 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.183 7fcfa009b700 1 -- start start 2021-02-10T11:44:59.202 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.184 7fcfa009b700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fcf980d1c70 0x7fcf980cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:59.202 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.184 7fcfa009b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcf980cf580 0x7fcf980cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:59.202 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.184 7fcfa009b700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf980fe1c0 con 0x7fcf980d1870 2021-02-10T11:44:59.202 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.184 7fcfa009b700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf981b0cb0 con 0x7fcf980d1c70 2021-02-10T11:44:59.202 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.184 7fcfa009b700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf98109a60 con 0x7fcf980cf580 2021-02-10T11:44:59.202 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.184 7fcf9d6b6700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fcf980d1c70 0x7fcf980cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:59.202 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.184 7fcf9d6b6700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fcf980d1c70 0x7fcf980cef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3300/0 says I am v2:172.21.2.112:60650/0 (socket says 172.21.2.112:60650) 2021-02-10T11:44:59.202 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.184 7fcf9d6b6700 1 -- 172.21.2.112:0/87383176 learned_addr learned my addr 172.21.2.112:0/87383176 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:44:59.202 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.184 7fcf9deb7700 1 --1- 172.21.2.112:0/87383176 >> v1:172.21.2.120:6789/0 conn(0x7fcf980d1870 0x7fcf981b04c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:51390/0 (socket says 172.21.2.112:51390) 2021-02-10T11:44:59.203 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.184 7fcf9e6b8700 1 --2- 172.21.2.112:0/87383176 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcf980cf580 0x7fcf980cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:59.203 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.185 7fcf9d6b6700 1 -- 172.21.2.112:0/87383176 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcf980cf580 msgr2=0x7fcf980cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:59.203 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.185 7fcf9d6b6700 1 --2- 172.21.2.112:0/87383176 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcf980cf580 0x7fcf980cf980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:59.203 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.185 7fcf9d6b6700 1 -- 172.21.2.112:0/87383176 >> v1:172.21.2.120:6789/0 conn(0x7fcf980d1870 legacy=0x7fcf981b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:59.203 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.185 7fcf9d6b6700 1 -- 172.21.2.112:0/87383176 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf98088070 con 0x7fcf980d1c70 2021-02-10T11:44:59.203 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.185 7fcf9d6b6700 1 --2- 172.21.2.112:0/87383176 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcf980d1c70 0x7fcf980cef60 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 rx=0x7fcf94006700 tx=0x7fcf94009b80).ready entity=mon.1 client_cookie=b92a178f903c091a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:59.203 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.185 7fcf9ceb5700 1 -- 172.21.2.112:0/87383176 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcf9400c0c0 con 0x7fcf980d1c70 2021-02-10T11:44:59.203 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.185 7fcf9ceb5700 1 -- 172.21.2.112:0/87383176 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf9400c340 con 0x7fcf980d1c70 2021-02-10T11:44:59.203 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.185 7fcf9ceb5700 1 -- 172.21.2.112:0/87383176 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcf9400c860 con 0x7fcf980d1c70 2021-02-10T11:44:59.203 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.185 7fcfa009b700 1 -- 172.21.2.112:0/87383176 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcf980d1c70 msgr2=0x7fcf980cef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:59.203 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.185 7fcfa009b700 1 --2- 172.21.2.112:0/87383176 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcf980d1c70 0x7fcf980cef60 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 rx=0x7fcf94006700 tx=0x7fcf94009b80).stop 2021-02-10T11:44:59.203 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.186 7fcfa009b700 1 -- 172.21.2.112:0/87383176 shutdown_connections 2021-02-10T11:44:59.203 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.186 7fcfa009b700 1 --2- 172.21.2.112:0/87383176 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcf980d1c70 0x7fcf980cef60 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:59.204 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.186 7fcfa009b700 1 --2- 172.21.2.112:0/87383176 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcf980cf580 0x7fcf980cf980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:59.204 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.186 7fcfa009b700 1 -- 172.21.2.112:0/87383176 >> 172.21.2.112:0/87383176 conn(0x7fcf980ce420 msgr2=0x7fcf980d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:59.204 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.186 7fcfa009b700 1 -- 172.21.2.112:0/87383176 shutdown_connections 2021-02-10T11:44:59.204 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.186 7fcfa009b700 1 -- 172.21.2.112:0/87383176 wait complete. 2021-02-10T11:44:59.205 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.201 7fcfa009b700 1 Processor -- start 2021-02-10T11:44:59.205 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.201 7fcfa009b700 1 -- start start 2021-02-10T11:44:59.206 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.201 7fcfa009b700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcf980b04d0 0x7fcf981a7a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:59.206 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.201 7fcfa009b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcf981a8090 0x7fcf980c23a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:59.206 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.201 7fcfa009b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcf9808d680 0x7fcf980c29c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:59.206 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.201 7fcfa009b700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf980b0a00 con 0x7fcf981a8090 2021-02-10T11:44:59.206 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.201 7fcfa009b700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf980c2fe0 con 0x7fcf980b04d0 2021-02-10T11:44:59.206 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.201 7fcfa009b700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf980c3200 con 0x7fcf9808d680 2021-02-10T11:44:59.206 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.201 7fcf9d6b6700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcf981a8090 0x7fcf980c23a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:59.206 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.201 7fcf9d6b6700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcf981a8090 0x7fcf980c23a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:58374/0 (socket says 172.21.2.112:58374) 2021-02-10T11:44:59.206 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.201 7fcf9d6b6700 1 -- 172.21.2.112:0/447292705 learned_addr learned my addr 172.21.2.112:0/447292705 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:44:59.206 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.201 7fcf9e6b8700 1 --2- 172.21.2.112:0/447292705 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcf9808d680 0x7fcf980c29c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:59.206 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.201 7fcf9deb7700 1 --2- 172.21.2.112:0/447292705 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcf980b04d0 0x7fcf981a7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:59.206 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.202 7fcf9d6b6700 1 -- 172.21.2.112:0/447292705 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcf9808d680 msgr2=0x7fcf980c29c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:59.207 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.202 7fcf9d6b6700 1 --2- 172.21.2.112:0/447292705 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcf9808d680 0x7fcf980c29c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:59.207 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.202 7fcf9d6b6700 1 -- 172.21.2.112:0/447292705 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcf980b04d0 msgr2=0x7fcf981a7a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:59.207 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.202 7fcf9d6b6700 1 --2- 172.21.2.112:0/447292705 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcf980b04d0 0x7fcf981a7a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:59.207 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.202 7fcf9d6b6700 1 -- 172.21.2.112:0/447292705 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf9808e080 con 0x7fcf981a8090 2021-02-10T11:44:59.207 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.202 7fcf9d6b6700 1 --2- 172.21.2.112:0/447292705 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcf981a8090 0x7fcf980c23a0 secure :-1 s=READY pgs=2684 cs=0 l=1 rev1=1 rx=0x7fcf9400b5e0 tx=0x7fcf9400b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:59.207 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.202 7fcf8a7fc700 1 -- 172.21.2.112:0/447292705 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcf9400bd70 con 0x7fcf981a8090 2021-02-10T11:44:59.207 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.202 7fcf8a7fc700 1 -- 172.21.2.112:0/447292705 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf94002030 con 0x7fcf981a8090 2021-02-10T11:44:59.207 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.202 7fcf8a7fc700 1 -- 172.21.2.112:0/447292705 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcf940024d0 con 0x7fcf981a8090 2021-02-10T11:44:59.207 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.202 7fcfa009b700 1 -- 172.21.2.112:0/447292705 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf9808e080 con 0x7fcf981a8090 2021-02-10T11:44:59.207 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.202 7fcfa009b700 1 -- 172.21.2.112:0/447292705 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf9808e080 con 0x7fcf981a8090 2021-02-10T11:44:59.207 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.203 7fcf8a7fc700 1 -- 172.21.2.112:0/447292705 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fcf94000c90 con 0x7fcf981a8090 2021-02-10T11:44:59.207 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.203 7fcf8a7fc700 1 --2- 172.21.2.112:0/447292705 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcf80018730 0x7fcf8001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:59.207 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.203 7fcf8a7fc700 1 -- 172.21.2.112:0/447292705 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fcf94032610 con 0x7fcf981a8090 2021-02-10T11:44:59.207 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.203 7fcf9deb7700 1 --2- 172.21.2.112:0/447292705 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcf80018730 0x7fcf8001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:59.208 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.204 7fcf9deb7700 1 --2- 172.21.2.112:0/447292705 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcf80018730 0x7fcf8001abc0 crc :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:59.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.234 7fcfa009b700 1 -- 172.21.2.112:0/447292705 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf98082a70 con 0x7fcf981a8090 2021-02-10T11:44:59.237 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.236 7fcf8a7fc700 1 -- 172.21.2.112:0/447292705 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fcf9401f4d0 con 0x7fcf981a8090 2021-02-10T11:44:59.367 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.365 7fcfa009b700 1 -- 172.21.2.112:0/447292705 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7fcf98082a70 con 0x7fcf981a8090 2021-02-10T11:44:59.367 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.366 7fcf8a7fc700 1 -- 172.21.2.112:0/447292705 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v151) v1 ==== 92+0+10 (secure 0 0 0) 0x7fcf9400bd70 con 0x7fcf981a8090 2021-02-10T11:44:59.367 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:44:59.369 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.368 7fcfa009b700 1 -- 172.21.2.112:0/447292705 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcf80018730 msgr2=0x7fcf8001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:59.369 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.368 7fcfa009b700 1 --2- 172.21.2.112:0/447292705 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcf80018730 0x7fcf8001abc0 crc :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:59.369 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.368 7fcfa009b700 1 -- 172.21.2.112:0/447292705 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcf981a8090 msgr2=0x7fcf980c23a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:59.369 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.368 7fcfa009b700 1 --2- 172.21.2.112:0/447292705 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcf981a8090 0x7fcf980c23a0 secure :-1 s=READY pgs=2684 cs=0 l=1 rev1=1 rx=0x7fcf9400b5e0 tx=0x7fcf9400b980).stop 2021-02-10T11:44:59.369 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.368 7fcfa009b700 1 -- 172.21.2.112:0/447292705 shutdown_connections 2021-02-10T11:44:59.369 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.368 7fcfa009b700 1 --2- 172.21.2.112:0/447292705 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcf80018730 0x7fcf8001abc0 unknown :-1 s=CLOSED pgs=1226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:59.369 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.368 7fcfa009b700 1 --2- 172.21.2.112:0/447292705 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcf9808d680 0x7fcf980c29c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:59.370 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.368 7fcfa009b700 1 --2- 172.21.2.112:0/447292705 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcf981a8090 0x7fcf980c23a0 unknown :-1 s=CLOSED pgs=2684 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:59.370 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.368 7fcfa009b700 1 --2- 172.21.2.112:0/447292705 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcf980b04d0 0x7fcf981a7a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:59.370 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.368 7fcfa009b700 1 -- 172.21.2.112:0/447292705 >> 172.21.2.112:0/447292705 conn(0x7fcf980be530 msgr2=0x7fcf9809f1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:59.370 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.368 7fcfa009b700 1 -- 172.21.2.112:0/447292705 shutdown_connections 2021-02-10T11:44:59.370 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.368 7fcfa009b700 1 -- 172.21.2.112:0/447292705 wait complete. 2021-02-10T11:44:59.377 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:44:59.808 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.791 7f09a580e700 1 Processor -- start 2021-02-10T11:44:59.809 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.791 7f09a580e700 1 -- start start 2021-02-10T11:44:59.809 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.792 7f09a580e700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f09a00d1870 0x7f09a01b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:59.809 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.792 7f09a580e700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f09a00cff70 0x7f09a0109130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:59.809 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.792 7f09a580e700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09a00fe1c0 con 0x7f09a01b0ae0 2021-02-10T11:44:59.809 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.792 7f09a580e700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09a0109ad0 con 0x7f09a00cff70 2021-02-10T11:44:59.809 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.792 7f09a580e700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09a0109750 con 0x7f09a00d1870 2021-02-10T11:44:59.809 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.792 7f099fd75700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f09a00cff70 0x7f09a0109130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:59.809 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.792 7f099fd75700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f09a00cff70 0x7f09a0109130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:58382/0 (socket says 172.21.2.112:58382) 2021-02-10T11:44:59.809 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.792 7f099fd75700 1 -- 172.21.2.112:0/3727466943 learned_addr learned my addr 172.21.2.112:0/3727466943 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:44:59.810 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.792 7f099fd75700 1 -- 172.21.2.112:0/3727466943 >> v2:172.21.2.120:3300/0 conn(0x7f09a00d1870 msgr2=0x7f09a01b04c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:44:59.810 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.792 7f099fd75700 1 --2- 172.21.2.112:0/3727466943 >> v2:172.21.2.120:3300/0 conn(0x7f09a00d1870 0x7f09a01b04c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:59.810 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.792 7f099fd75700 1 -- 172.21.2.112:0/3727466943 >> v1:172.21.2.120:6789/0 conn(0x7f09a01b0ae0 legacy=0x7f09a0108940 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:44:59.810 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.792 7f099fd75700 1 -- 172.21.2.112:0/3727466943 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09a0088070 con 0x7f09a00cff70 2021-02-10T11:44:59.810 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.792 7f099fd75700 1 --2- 172.21.2.112:0/3727466943 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f09a00cff70 0x7f09a0109130 secure :-1 s=READY pgs=2685 cs=0 l=1 rev1=1 rx=0x7f0998006b80 tx=0x7f0998009b90).ready entity=mon.0 client_cookie=bb03dbf188acf554 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:59.810 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.792 7f099e572700 1 -- 172.21.2.112:0/3727466943 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f09980022e0 con 0x7f09a00cff70 2021-02-10T11:44:59.810 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.793 7f099e572700 1 -- 172.21.2.112:0/3727466943 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0998002620 con 0x7f09a00cff70 2021-02-10T11:44:59.810 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.793 7f099e572700 1 -- 172.21.2.112:0/3727466943 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0998002b00 con 0x7f09a00cff70 2021-02-10T11:44:59.810 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.793 7f09a580e700 1 -- 172.21.2.112:0/3727466943 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f09a00cff70 msgr2=0x7f09a0109130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:59.810 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.793 7f09a580e700 1 --2- 172.21.2.112:0/3727466943 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f09a00cff70 0x7f09a0109130 secure :-1 s=READY pgs=2685 cs=0 l=1 rev1=1 rx=0x7f0998006b80 tx=0x7f0998009b90).stop 2021-02-10T11:44:59.810 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.793 7f09a580e700 1 -- 172.21.2.112:0/3727466943 shutdown_connections 2021-02-10T11:44:59.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.793 7f09a580e700 1 --2- 172.21.2.112:0/3727466943 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f09a00cff70 0x7f09a0109130 unknown :-1 s=CLOSED pgs=2685 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:59.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.793 7f09a580e700 1 --2- 172.21.2.112:0/3727466943 >> v2:172.21.2.120:3300/0 conn(0x7f09a00d1870 0x7f09a01b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:59.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.793 7f09a580e700 1 -- 172.21.2.112:0/3727466943 >> 172.21.2.112:0/3727466943 conn(0x7f09a00ce420 msgr2=0x7f09a00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:59.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.793 7f09a580e700 1 -- 172.21.2.112:0/3727466943 shutdown_connections 2021-02-10T11:44:59.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.793 7f09a580e700 1 -- 172.21.2.112:0/3727466943 wait complete. 2021-02-10T11:44:59.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.808 7f09a580e700 1 Processor -- start 2021-02-10T11:44:59.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.808 7f09a580e700 1 -- start start 2021-02-10T11:44:59.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.808 7f09a580e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f09a00c3820 0x7f09a0084cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:59.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.808 7f09a580e700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f09a00a0290 0x7f09a00852d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:59.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.808 7f09a580e700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f09a0086270 0x7f09a0086670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:59.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.808 7f09a580e700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09a00c0700 con 0x7f09a00c3820 2021-02-10T11:44:59.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.808 7f09a580e700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09a00858f0 con 0x7f09a00a0290 2021-02-10T11:44:59.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.808 7f09a580e700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09a0085b10 con 0x7f09a0086270 2021-02-10T11:44:59.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.808 7f099f574700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f09a00c3820 0x7f09a0084cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:59.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.808 7f099f574700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f09a00c3820 0x7f09a0084cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:58384/0 (socket says 172.21.2.112:58384) 2021-02-10T11:44:59.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.808 7f099f574700 1 -- 172.21.2.112:0/220698452 learned_addr learned my addr 172.21.2.112:0/220698452 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:44:59.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.809 7f099fd75700 1 --2- 172.21.2.112:0/220698452 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f09a0086270 0x7f09a0086670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:59.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.809 7f099ed73700 1 --2- 172.21.2.112:0/220698452 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f09a00a0290 0x7f09a00852d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:59.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.809 7f099f574700 1 -- 172.21.2.112:0/220698452 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f09a0086270 msgr2=0x7f09a0086670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:59.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.809 7f099f574700 1 --2- 172.21.2.112:0/220698452 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f09a0086270 0x7f09a0086670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:59.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.809 7f099f574700 1 -- 172.21.2.112:0/220698452 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f09a00a0290 msgr2=0x7f09a00852d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:59.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.809 7f099f574700 1 --2- 172.21.2.112:0/220698452 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f09a00a0290 0x7f09a00852d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:59.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.809 7f099f574700 1 -- 172.21.2.112:0/220698452 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09a0086c90 con 0x7f09a00c3820 2021-02-10T11:44:59.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.809 7f099ed73700 1 --2- 172.21.2.112:0/220698452 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f09a00a0290 0x7f09a00852d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:44:59.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.809 7f099f574700 1 --2- 172.21.2.112:0/220698452 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f09a00c3820 0x7f09a0084cb0 secure :-1 s=READY pgs=2686 cs=0 l=1 rev1=1 rx=0x7f099000b5c0 tx=0x7f099000b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:59.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.809 7f0997fff700 1 -- 172.21.2.112:0/220698452 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0990000c40 con 0x7f09a00c3820 2021-02-10T11:44:59.815 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.809 7f0997fff700 1 -- 172.21.2.112:0/220698452 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f099000bd90 con 0x7f09a00c3820 2021-02-10T11:44:59.815 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.809 7f0997fff700 1 -- 172.21.2.112:0/220698452 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0990006220 con 0x7f09a00c3820 2021-02-10T11:44:59.815 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.809 7f09a580e700 1 -- 172.21.2.112:0/220698452 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09a0086c90 con 0x7f09a00c3820 2021-02-10T11:44:59.815 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.809 7f09a580e700 1 -- 172.21.2.112:0/220698452 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09a01a2f00 con 0x7f09a00c3820 2021-02-10T11:44:59.815 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.810 7f0997fff700 1 -- 172.21.2.112:0/220698452 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0990006030 con 0x7f09a00c3820 2021-02-10T11:44:59.815 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.810 7f0997fff700 1 --2- 172.21.2.112:0/220698452 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f09840186c0 0x7f098401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:59.815 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.810 7f0997fff700 1 -- 172.21.2.112:0/220698452 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f09900325c0 con 0x7f09a00c3820 2021-02-10T11:44:59.815 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.811 7f099ed73700 1 --2- 172.21.2.112:0/220698452 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f09840186c0 0x7f098401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:59.815 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.811 7f099ed73700 1 --2- 172.21.2.112:0/220698452 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f09840186c0 0x7f098401abc0 crc :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:59.843 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.842 7f09a580e700 1 -- 172.21.2.112:0/220698452 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09a0082a70 con 0x7f09a00c3820 2021-02-10T11:44:59.845 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.844 7f0997fff700 1 -- 172.21.2.112:0/220698452 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f099002cf40 con 0x7f09a00c3820 2021-02-10T11:44:59.973 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.972 7f09a580e700 1 -- 172.21.2.112:0/220698452 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f09a0082a70 con 0x7f09a00c3820 2021-02-10T11:44:59.974 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.972 7f0997fff700 1 -- 172.21.2.112:0/220698452 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v151) v1 ==== 104+0+10 (secure 0 0 0) 0x7f099002cf40 con 0x7f09a00c3820 2021-02-10T11:44:59.974 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:44:59.976 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.974 7f09a580e700 1 -- 172.21.2.112:0/220698452 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f09840186c0 msgr2=0x7f098401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:59.976 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.974 7f09a580e700 1 --2- 172.21.2.112:0/220698452 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f09840186c0 0x7f098401abc0 crc :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:59.976 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.974 7f09a580e700 1 -- 172.21.2.112:0/220698452 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f09a00c3820 msgr2=0x7f09a0084cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:59.976 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.974 7f09a580e700 1 --2- 172.21.2.112:0/220698452 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f09a00c3820 0x7f09a0084cb0 secure :-1 s=READY pgs=2686 cs=0 l=1 rev1=1 rx=0x7f099000b5c0 tx=0x7f099000b960).stop 2021-02-10T11:44:59.976 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.974 7f09a580e700 1 -- 172.21.2.112:0/220698452 shutdown_connections 2021-02-10T11:44:59.976 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.974 7f09a580e700 1 --2- 172.21.2.112:0/220698452 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f09840186c0 0x7f098401abc0 unknown :-1 s=CLOSED pgs=1227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:59.976 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.974 7f09a580e700 1 --2- 172.21.2.112:0/220698452 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f09a00a0290 0x7f09a00852d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:59.976 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.974 7f09a580e700 1 --2- 172.21.2.112:0/220698452 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f09a0086270 0x7f09a0086670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:59.976 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.974 7f09a580e700 1 --2- 172.21.2.112:0/220698452 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f09a00c3820 0x7f09a0084cb0 unknown :-1 s=CLOSED pgs=2686 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:59.976 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.974 7f09a580e700 1 -- 172.21.2.112:0/220698452 >> 172.21.2.112:0/220698452 conn(0x7f09a00872c0 msgr2=0x7f09a0095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:59.977 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.975 7f09a580e700 1 -- 172.21.2.112:0/220698452 shutdown_connections 2021-02-10T11:44:59.977 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:44:59.975 7f09a580e700 1 -- 172.21.2.112:0/220698452 wait complete. 2021-02-10T11:44:59.984 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:45:00.417 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:00.399 7f68ccf08700 1 Processor -- start 2021-02-10T11:45:00.417 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:00.399 7f68ccf08700 1 -- start start 2021-02-10T11:45:00.417 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:00.400 7f68ccf08700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f68c810afe0 0x7f68c80cf740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:00.418 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:00.400 7f68ccf08700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68c80fe1b0 con 0x7f68c80d1860 2021-02-10T11:45:00.418 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:00.400 7f68ccf08700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68c81b0ca0 con 0x7f68c80d1c60 2021-02-10T11:45:00.418 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:00.400 7f68ccf08700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68c8109a50 con 0x7f68c810afe0 2021-02-10T11:45:00.418 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:00.400 7f68c7314700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f68c810afe0 0x7f68c80cf740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:00.418 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:00.400 7f68c6b13700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f68c80d1860 0x7f68c81b04b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:57852/0 (socket says 172.21.2.112:57852) 2021-02-10T11:45:00.418 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:00.400 7f68c6b13700 1 -- 172.21.2.112:0/55984235 learned_addr learned my addr 172.21.2.112:0/55984235 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:45:00.418 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:00.400 7f68c7314700 1 -- 172.21.2.112:0/55984235 >> v1:172.21.2.120:6789/0 conn(0x7f68c80d1c60 legacy=0x7f68c80cef50 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:45:00.418 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:00.400 7f68c7314700 1 -- 172.21.2.112:0/55984235 >> v1:172.21.2.112:6789/0 conn(0x7f68c80d1860 legacy=0x7f68c81b04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:00.418 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:00.400 7f68c7314700 1 -- 172.21.2.112:0/55984235 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68c810b3e0 con 0x7f68c810afe0 2021-02-10T11:45:00.418 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:00.400 7f68c7314700 1 --2- 172.21.2.112:0/55984235 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f68c810afe0 0x7f68c80cf740 secure :-1 s=READY pgs=2688 cs=0 l=1 rev1=1 rx=0x7f68c0006e30 tx=0x7f68c0009e60).ready entity=mon.0 client_cookie=2f3234cad587d764 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:00.418 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:00.401 7f68c5b11700 1 -- 172.21.2.112:0/55984235 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f68c0006a50 con 0x7f68c810afe0 2021-02-10T11:45:00.419 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:00.401 7f68c5b11700 1 -- 172.21.2.112:0/55984235 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68c000b300 con 0x7f68c810afe0 2021-02-10T11:45:00.419 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:00.401 7f68c5b11700 1 -- 172.21.2.112:0/55984235 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f68c000b750 con 0x7f68c810afe0 2021-02-10T11:45:00.419 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:00.401 7f68ccf08700 1 -- 172.21.2.112:0/55984235 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f68c810afe0 msgr2=0x7f68c80cf740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:00.419 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:00.401 7f68ccf08700 1 --2- 172.21.2.112:0/55984235 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f68c810afe0 0x7f68c80cf740 secure :-1 s=READY pgs=2688 cs=0 l=1 rev1=1 rx=0x7f68c0006e30 tx=0x7f68c0009e60).stop 2021-02-10T11:45:00.419 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:00.401 7f68ccf08700 1 -- 172.21.2.112:0/55984235 shutdown_connections 2021-02-10T11:45:00.419 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:00.401 7f68ccf08700 1 --2- 172.21.2.112:0/55984235 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f68c810afe0 0x7f68c80cf740 unknown :-1 s=CLOSED pgs=2688 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:00.419 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:00.401 7f68ccf08700 1 -- 172.21.2.112:0/55984235 >> 172.21.2.112:0/55984235 conn(0x7f68c80ce410 msgr2=0x7f68c80d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:00.419 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:00.401 7f68ccf08700 1 -- 172.21.2.112:0/55984235 shutdown_connections 2021-02-10T11:45:00.419 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:00.401 7f68ccf08700 1 -- 172.21.2.112:0/55984235 wait complete. 2021-02-10T11:45:00.419 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:00.416 7f68ccf08700 1 Processor -- start 2021-02-10T11:45:00.419 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:00.416 7f68ccf08700 1 -- start start 2021-02-10T11:45:00.421 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:00.416 7f68ccf08700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f68c80c3500 0x7f68c80a2e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:00.421 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:00.416 7f68ccf08700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f68c80b79a0 0x7f68c80b7da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:00.421 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:00.416 7f68ccf08700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f68c80b9e40 0x7f68c80ba240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:00.421 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:00.416 7f68ccf08700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68c81065b0 con 0x7f68c80c3500 2021-02-10T11:45:00.421 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:00.416 7f68ccf08700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68c80b70f0 con 0x7f68c80b9e40 2021-02-10T11:45:00.421 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:00.416 7f68ccf08700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68c80a3450 con 0x7f68c80b79a0 2021-02-10T11:45:00.421 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:00.417 7f68c6b13700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f68c80c3500 0x7f68c80a2e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:00.421 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:00.417 7f68c6b13700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f68c80c3500 0x7f68c80a2e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:58396/0 (socket says 172.21.2.112:58396) 2021-02-10T11:45:00.421 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:00.417 7f68c6b13700 1 -- 172.21.2.112:0/2366845388 learned_addr learned my addr 172.21.2.112:0/2366845388 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:45:00.421 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:00.417 7f68c6b13700 1 -- 172.21.2.112:0/2366845388 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f68c80b79a0 msgr2=0x7f68c80b7da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:00.421 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:00.417 7f68c6b13700 1 --2- 172.21.2.112:0/2366845388 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f68c80b79a0 0x7f68c80b7da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:00.422 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:00.417 7f68c7314700 1 --2- 172.21.2.112:0/2366845388 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f68c80b9e40 0x7f68c80ba240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:00.422 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:00.417 7f68c6b13700 1 -- 172.21.2.112:0/2366845388 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f68c80b9e40 msgr2=0x7f68c80ba240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:00.422 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:00.417 7f68c6b13700 1 --2- 172.21.2.112:0/2366845388 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f68c80b9e40 0x7f68c80ba240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:00.422 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:00.417 7f68c6b13700 1 -- 172.21.2.112:0/2366845388 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68c80b83c0 con 0x7f68c80c3500 2021-02-10T11:45:00.422 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:00.417 7f68c6b13700 1 --2- 172.21.2.112:0/2366845388 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f68c80c3500 0x7f68c80a2e30 secure :-1 s=READY pgs=2689 cs=0 l=1 rev1=1 rx=0x7f68b8001410 tx=0x7f68b800a950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:00.422 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:00.417 7f68bf7fe700 1 -- 172.21.2.112:0/2366845388 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f68b800ad40 con 0x7f68c80c3500 2021-02-10T11:45:00.422 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:00.417 7f68bf7fe700 1 -- 172.21.2.112:0/2366845388 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68b8000c90 con 0x7f68c80c3500 2021-02-10T11:45:00.422 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:00.417 7f68ccf08700 1 -- 172.21.2.112:0/2366845388 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68c80b83c0 con 0x7f68c80c3500 2021-02-10T11:45:00.422 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:00.417 7f68ccf08700 1 -- 172.21.2.112:0/2366845388 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68c80b83c0 con 0x7f68c80c3500 2021-02-10T11:45:00.422 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:00.417 7f68bf7fe700 1 -- 172.21.2.112:0/2366845388 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f68b8004590 con 0x7f68c80c3500 2021-02-10T11:45:00.422 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:00.418 7f68bf7fe700 1 -- 172.21.2.112:0/2366845388 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f68b80043a0 con 0x7f68c80c3500 2021-02-10T11:45:00.422 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:00.418 7f68bf7fe700 1 --2- 172.21.2.112:0/2366845388 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f68ac018730 0x7f68ac01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:00.422 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:00.418 7f68bf7fe700 1 -- 172.21.2.112:0/2366845388 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f68b8032830 con 0x7f68c80c3500 2021-02-10T11:45:00.423 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:00.419 7f68c6312700 1 --2- 172.21.2.112:0/2366845388 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f68ac018730 0x7f68ac01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:00.423 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:00.419 7f68c6312700 1 --2- 172.21.2.112:0/2366845388 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f68ac018730 0x7f68ac01ac10 crc :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:00.451 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:00.450 7f68ccf08700 1 -- 172.21.2.112:0/2366845388 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68c80b70f0 con 0x7f68c80c3500 2021-02-10T11:45:00.454 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:00.452 7f68bf7fe700 1 -- 172.21.2.112:0/2366845388 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f68b8004850 con 0x7f68c80c3500 2021-02-10T11:45:00.582 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:00.580 7f68ccf08700 1 -- 172.21.2.112:0/2366845388 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f68c80b70f0 con 0x7f68c80c3500 2021-02-10T11:45:00.582 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:00.581 7f68bf7fe700 1 -- 172.21.2.112:0/2366845388 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v151) v1 ==== 100+0+10 (secure 0 0 0) 0x7f68b800ad40 con 0x7f68c80c3500 2021-02-10T11:45:00.584 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:45:00.584 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:00.583 7f68ccf08700 1 -- 172.21.2.112:0/2366845388 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f68ac018730 msgr2=0x7f68ac01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:00.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:00.583 7f68ccf08700 1 --2- 172.21.2.112:0/2366845388 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f68ac018730 0x7f68ac01ac10 crc :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:00.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:00.583 7f68ccf08700 1 -- 172.21.2.112:0/2366845388 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f68c80c3500 msgr2=0x7f68c80a2e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:00.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:00.583 7f68ccf08700 1 --2- 172.21.2.112:0/2366845388 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f68c80c3500 0x7f68c80a2e30 secure :-1 s=READY pgs=2689 cs=0 l=1 rev1=1 rx=0x7f68b8001410 tx=0x7f68b800a950).stop 2021-02-10T11:45:00.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:00.583 7f68ccf08700 1 -- 172.21.2.112:0/2366845388 shutdown_connections 2021-02-10T11:45:00.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:00.583 7f68ccf08700 1 --2- 172.21.2.112:0/2366845388 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f68ac018730 0x7f68ac01ac10 unknown :-1 s=CLOSED pgs=1228 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:00.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:00.583 7f68ccf08700 1 --2- 172.21.2.112:0/2366845388 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f68c80b9e40 0x7f68c80ba240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:00.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:00.583 7f68ccf08700 1 --2- 172.21.2.112:0/2366845388 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f68c80b79a0 0x7f68c80b7da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:00.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:00.583 7f68ccf08700 1 --2- 172.21.2.112:0/2366845388 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f68c80c3500 0x7f68c80a2e30 unknown :-1 s=CLOSED pgs=2689 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:00.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:00.583 7f68ccf08700 1 -- 172.21.2.112:0/2366845388 >> 172.21.2.112:0/2366845388 conn(0x7f68c8080ba0 msgr2=0x7f68c80ffca0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:00.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:00.583 7f68ccf08700 1 -- 172.21.2.112:0/2366845388 shutdown_connections 2021-02-10T11:45:00.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:00.583 7f68ccf08700 1 -- 172.21.2.112:0/2366845388 wait complete. 2021-02-10T11:45:00.593 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:45:00.593 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:45:01.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:01.003 7f5337a38700 1 Processor -- start 2021-02-10T11:45:01.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:01.003 7f5337a38700 1 -- start start 2021-02-10T11:45:01.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:01.004 7f5337a38700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f53300d1c70 0x7f53300f20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:01.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:01.004 7f5337a38700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53300f0b10 con 0x7f53300d1870 2021-02-10T11:45:01.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:01.004 7f5337a38700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53301b0be0 con 0x7f53300d0340 2021-02-10T11:45:01.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:01.004 7f5337a38700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53300e5ed0 con 0x7f53300d1c70 2021-02-10T11:45:01.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:01.004 7f5336055700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f53300d1c70 0x7f53300f20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:01.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:01.004 7f5336055700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f53300d1c70 0x7f53300f20a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:58408/0 (socket says 172.21.2.112:58408) 2021-02-10T11:45:01.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:01.004 7f5336055700 1 -- 172.21.2.112:0/2850284701 learned_addr learned my addr 172.21.2.112:0/2850284701 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:45:01.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:01.004 7f5336055700 1 -- 172.21.2.112:0/2850284701 >> v1:172.21.2.120:6789/0 conn(0x7f53300d0340 legacy=0x7f53300ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:01.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:01.004 7f5336055700 1 -- 172.21.2.112:0/2850284701 >> v1:172.21.2.112:6789/0 conn(0x7f53300d1870 legacy=0x7f53301b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:01.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:01.004 7f5336055700 1 -- 172.21.2.112:0/2850284701 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53300efe30 con 0x7f53300d1c70 2021-02-10T11:45:01.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:01.004 7f5336055700 1 --2- 172.21.2.112:0/2850284701 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f53300d1c70 0x7f53300f20a0 secure :-1 s=READY pgs=2691 cs=0 l=1 rev1=1 rx=0x7f5328006f00 tx=0x7f5328009f10).ready entity=mon.0 client_cookie=cf51d8864f78cf87 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:01.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:01.004 7f5334852700 1 -- 172.21.2.112:0/2850284701 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f532800b3b0 con 0x7f53300d1c70 2021-02-10T11:45:01.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:01.004 7f5334852700 1 -- 172.21.2.112:0/2850284701 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f532800b630 con 0x7f53300d1c70 2021-02-10T11:45:01.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:01.004 7f5334852700 1 -- 172.21.2.112:0/2850284701 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f532800bb10 con 0x7f53300d1c70 2021-02-10T11:45:01.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:01.004 7f5337a38700 1 -- 172.21.2.112:0/2850284701 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f53300d1c70 msgr2=0x7f53300f20a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:01.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:01.004 7f5337a38700 1 --2- 172.21.2.112:0/2850284701 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f53300d1c70 0x7f53300f20a0 secure :-1 s=READY pgs=2691 cs=0 l=1 rev1=1 rx=0x7f5328006f00 tx=0x7f5328009f10).stop 2021-02-10T11:45:01.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:01.005 7f5337a38700 1 -- 172.21.2.112:0/2850284701 shutdown_connections 2021-02-10T11:45:01.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:01.005 7f5337a38700 1 --2- 172.21.2.112:0/2850284701 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f53300d1c70 0x7f53300f20a0 unknown :-1 s=CLOSED pgs=2691 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:01.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:01.005 7f5337a38700 1 -- 172.21.2.112:0/2850284701 >> 172.21.2.112:0/2850284701 conn(0x7f53300ce450 msgr2=0x7f53300d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:01.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:01.005 7f5337a38700 1 -- 172.21.2.112:0/2850284701 shutdown_connections 2021-02-10T11:45:01.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:01.005 7f5337a38700 1 -- 172.21.2.112:0/2850284701 wait complete. 2021-02-10T11:45:01.024 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:01.020 7f5337a38700 1 Processor -- start 2021-02-10T11:45:01.024 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:01.020 7f5337a38700 1 -- start start 2021-02-10T11:45:01.024 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:01.020 7f5337a38700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f53300af040 0x7f53300af440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:01.024 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:01.020 7f5337a38700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f53300afa60 0x7f53300ada20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:01.024 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:01.020 7f5337a38700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f53300afe60 0x7f53300ae040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:01.024 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:01.020 7f5337a38700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53300923c0 con 0x7f53300afe60 2021-02-10T11:45:01.025 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:01.020 7f5337a38700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53300b0410 con 0x7f53300afa60 2021-02-10T11:45:01.025 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:01.020 7f5337a38700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5330091e40 con 0x7f53300af040 2021-02-10T11:45:01.025 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:01.020 7f5336055700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f53300afe60 0x7f53300ae040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:01.025 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:01.020 7f5336055700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f53300afe60 0x7f53300ae040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:58412/0 (socket says 172.21.2.112:58412) 2021-02-10T11:45:01.025 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:01.020 7f5336055700 1 -- 172.21.2.112:0/2819746990 learned_addr learned my addr 172.21.2.112:0/2819746990 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:45:01.025 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:01.020 7f5335854700 1 --2- 172.21.2.112:0/2819746990 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f53300af040 0x7f53300af440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:01.025 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:01.020 7f5335053700 1 --2- 172.21.2.112:0/2819746990 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f53300afa60 0x7f53300ada20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:01.025 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:01.021 7f5336055700 1 -- 172.21.2.112:0/2819746990 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f53300af040 msgr2=0x7f53300af440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:01.025 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:01.021 7f5336055700 1 --2- 172.21.2.112:0/2819746990 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f53300af040 0x7f53300af440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:01.025 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:01.021 7f5336055700 1 -- 172.21.2.112:0/2819746990 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f53300afa60 msgr2=0x7f53300ada20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:01.025 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:01.021 7f5336055700 1 --2- 172.21.2.112:0/2819746990 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f53300afa60 0x7f53300ada20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:01.025 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:01.021 7f5336055700 1 -- 172.21.2.112:0/2819746990 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f533008f6d0 con 0x7f53300afe60 2021-02-10T11:45:01.025 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:01.021 7f5336055700 1 --2- 172.21.2.112:0/2819746990 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f53300afe60 0x7f53300ae040 secure :-1 s=READY pgs=2692 cs=0 l=1 rev1=1 rx=0x7f532800b9b0 tx=0x7f532800bd50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:01.026 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:01.021 7f53227fc700 1 -- 172.21.2.112:0/2819746990 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5328005820 con 0x7f53300afe60 2021-02-10T11:45:01.026 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:01.021 7f53227fc700 1 -- 172.21.2.112:0/2819746990 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5328005a60 con 0x7f53300afe60 2021-02-10T11:45:01.026 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:01.021 7f53227fc700 1 -- 172.21.2.112:0/2819746990 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5328006030 con 0x7f53300afe60 2021-02-10T11:45:01.026 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:01.021 7f5337a38700 1 -- 172.21.2.112:0/2819746990 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f533008f6d0 con 0x7f53300afe60 2021-02-10T11:45:01.026 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:01.021 7f5337a38700 1 -- 172.21.2.112:0/2819746990 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f533008f6d0 con 0x7f53300afe60 2021-02-10T11:45:01.026 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:01.022 7f53227fc700 1 -- 172.21.2.112:0/2819746990 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5328005c40 con 0x7f53300afe60 2021-02-10T11:45:01.027 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:01.022 7f53227fc700 1 --2- 172.21.2.112:0/2819746990 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f53180186c0 0x7f531801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:01.027 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:01.022 7f53227fc700 1 -- 172.21.2.112:0/2819746990 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f53280328d0 con 0x7f53300afe60 2021-02-10T11:45:01.027 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:01.022 7f5335854700 1 --2- 172.21.2.112:0/2819746990 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f53180186c0 0x7f531801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:01.027 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:01.022 7f5335854700 1 --2- 172.21.2.112:0/2819746990 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f53180186c0 0x7f531801abc0 crc :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:01.055 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:01.053 7f5337a38700 1 -- 172.21.2.112:0/2819746990 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53300e7df0 con 0x7f53300afe60 2021-02-10T11:45:01.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:01.056 7f53227fc700 1 -- 172.21.2.112:0/2819746990 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5328000f90 con 0x7f53300afe60 2021-02-10T11:45:01.185 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:01.183 7f5337a38700 1 -- 172.21.2.112:0/2819746990 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f53300ce450 con 0x7f53300afe60 2021-02-10T11:45:01.185 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:01.184 7f53227fc700 1 -- 172.21.2.112:0/2819746990 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 130 v130) v1 ==== 95+0+2108 (secure 0 0 0) 0x7f5328005c20 con 0x7f53300afe60 2021-02-10T11:45:01.186 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:45:01.186 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":130,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":12006,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/1015001314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":1015001314},{"type":"v1","addr":"172.21.2.112:6833","nonce":1015001314}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":126}],"filesystems":[{"mdsmap":{"epoch":130,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:44:43.721296","modified":"2021-02-10 11:44:45.730948","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7268},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7268":{"gid":7268,"name":"b","rank":0,"incarnation":127,"state":"up:active","state_seq":3,"addr":"172.21.2.120:6835/3799929082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":3799929082},{"type":"v1","addr":"172.21.2.120:6835","nonce":3799929082}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":16}]} 2021-02-10T11:45:01.186 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 130 2021-02-10T11:45:01.187 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:01.185 7f5337a38700 1 -- 172.21.2.112:0/2819746990 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f53180186c0 msgr2=0x7f531801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:01.187 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:01.185 7f5337a38700 1 --2- 172.21.2.112:0/2819746990 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f53180186c0 0x7f531801abc0 crc :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:01.187 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:01.185 7f5337a38700 1 -- 172.21.2.112:0/2819746990 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f53300afe60 msgr2=0x7f53300ae040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:01.187 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:01.185 7f5337a38700 1 --2- 172.21.2.112:0/2819746990 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f53300afe60 0x7f53300ae040 secure :-1 s=READY pgs=2692 cs=0 l=1 rev1=1 rx=0x7f532800b9b0 tx=0x7f532800bd50).stop 2021-02-10T11:45:01.187 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:01.186 7f5337a38700 1 -- 172.21.2.112:0/2819746990 shutdown_connections 2021-02-10T11:45:01.187 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:01.186 7f5337a38700 1 --2- 172.21.2.112:0/2819746990 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f53180186c0 0x7f531801abc0 unknown :-1 s=CLOSED pgs=1229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:01.187 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:01.186 7f5337a38700 1 --2- 172.21.2.112:0/2819746990 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f53300afa60 0x7f53300ada20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:01.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:01.186 7f5337a38700 1 --2- 172.21.2.112:0/2819746990 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f53300afe60 0x7f53300ae040 unknown :-1 s=CLOSED pgs=2692 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:01.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:01.186 7f5337a38700 1 --2- 172.21.2.112:0/2819746990 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f53300af040 0x7f53300af440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:01.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:01.186 7f5337a38700 1 -- 172.21.2.112:0/2819746990 >> 172.21.2.112:0/2819746990 conn(0x7f53300a0030 msgr2=0x7f53300f22b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:01.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:01.186 7f5337a38700 1 -- 172.21.2.112:0/2819746990 shutdown_connections 2021-02-10T11:45:01.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:01.186 7f5337a38700 1 -- 172.21.2.112:0/2819746990 wait complete. 2021-02-10T11:45:01.397 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 130, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:44:43.721296', 'modified': '2021-02-10 11:44:45.730948', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7268}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7268': {'gid': 7268, 'name': 'b', 'rank': 0, 'incarnation': 127, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.120:6835/3799929082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.120:6834', 'nonce': 3799929082}, {'type': 'v1', 'addr': '172.21.2.120:6835', 'nonce': 3799929082}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [33], 'metadata_pool': 32, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:45:01.397 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:45:01.397 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:45:01.796 INFO:teuthology.orchestra.run.gibba020.stdout:{ 2021-02-10T11:45:01.797 INFO:teuthology.orchestra.run.gibba020.stdout: "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", 2021-02-10T11:45:01.797 INFO:teuthology.orchestra.run.gibba020.stdout: "whoami": 0, 2021-02-10T11:45:01.797 INFO:teuthology.orchestra.run.gibba020.stdout: "id": 7268, 2021-02-10T11:45:01.797 INFO:teuthology.orchestra.run.gibba020.stdout: "want_state": "up:active", 2021-02-10T11:45:01.797 INFO:teuthology.orchestra.run.gibba020.stdout: "state": "up:active", 2021-02-10T11:45:01.797 INFO:teuthology.orchestra.run.gibba020.stdout: "rank_uptime": 18.071936871999998, 2021-02-10T11:45:01.797 INFO:teuthology.orchestra.run.gibba020.stdout: "mdsmap_epoch": 130, 2021-02-10T11:45:01.797 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch": 151, 2021-02-10T11:45:01.797 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch_barrier": 151, 2021-02-10T11:45:01.797 INFO:teuthology.orchestra.run.gibba020.stdout: "uptime": 23.715568727000001 2021-02-10T11:45:01.797 INFO:teuthology.orchestra.run.gibba020.stdout:} 2021-02-10T11:45:01.800 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", "whoami": 0, "id": 7268, "want_state": "up:active", "state": "up:active", "rank_uptime": 18.071936871999998, "mdsmap_epoch": 130, "osdmap_epoch": 151, "osdmap_epoch_barrier": 151, "uptime": 23.715568727000001 } 2021-02-10T11:45:01.801 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:45:01.801 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2021-02-10T11:45:01.801 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.1 at ubuntu@gibba012.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2021-02-10T11:45:01.801 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:45:01.801 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:45:02.135 DEBUG:teuthology.orchestra.run.gibba012:> sudo modprobe fuse 2021-02-10T11:45:02.479 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:45:02.479 DEBUG:teuthology.orchestra.run.gibba012:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:45:02.823 INFO:teuthology.orchestra.run.gibba012.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:45:02.824 INFO:teuthology.orchestra.run.gibba012.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:45:02.824 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:45:02.825 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:45:02.825 DEBUG:teuthology.orchestra.run.gibba012:> ls /sys/fs/fuse/connections 2021-02-10T11:45:03.158 INFO:teuthology.orchestra.run.gibba012.stdout:42 2021-02-10T11:45:03.158 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [42] 2021-02-10T11:45:03.158 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.1 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:45:03.161 DEBUG:teuthology.orchestra.run.gibba012:> sudo modprobe fuse 2021-02-10T11:45:03.506 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:45:03.506 DEBUG:teuthology.orchestra.run.gibba012:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:45:03.572 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba012.stderr:ceph-fuse[177164]: starting ceph client 2021-02-10T11:45:03.573 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba012.stderr:2021-02-10 11:45:03.571 7f54ead6ff80 -1 init, newargv = 0x55d86f054210 newargc=9 2021-02-10T11:45:03.577 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba012.stderr:ceph-fuse[177164]: starting fuse 2021-02-10T11:45:03.849 INFO:teuthology.orchestra.run.gibba012.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:45:03.850 INFO:teuthology.orchestra.run.gibba012.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:45:03.850 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:45:03.850 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:45:03.851 DEBUG:teuthology.orchestra.run.gibba012:> ls /sys/fs/fuse/connections 2021-02-10T11:45:04.185 INFO:teuthology.orchestra.run.gibba012.stdout:42 2021-02-10T11:45:04.185 INFO:teuthology.orchestra.run.gibba012.stdout:43 2021-02-10T11:45:04.186 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42, 43] 2021-02-10T11:45:04.186 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:45:04.186 DEBUG:teuthology.orchestra.run.gibba012:> sudo python3 -c ' 2021-02-10T11:45:04.186 DEBUG:teuthology.orchestra.run.gibba012:> import glob 2021-02-10T11:45:04.186 DEBUG:teuthology.orchestra.run.gibba012:> import re 2021-02-10T11:45:04.186 DEBUG:teuthology.orchestra.run.gibba012:> import os 2021-02-10T11:45:04.186 DEBUG:teuthology.orchestra.run.gibba012:> import subprocess 2021-02-10T11:45:04.187 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:45:04.187 DEBUG:teuthology.orchestra.run.gibba012:> def find_socket(client_name): 2021-02-10T11:45:04.187 DEBUG:teuthology.orchestra.run.gibba012:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2021-02-10T11:45:04.187 DEBUG:teuthology.orchestra.run.gibba012:> files = glob.glob(asok_path) 2021-02-10T11:45:04.187 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:45:04.187 DEBUG:teuthology.orchestra.run.gibba012:> # Given a non-glob path, it better be there 2021-02-10T11:45:04.187 DEBUG:teuthology.orchestra.run.gibba012:> if "*" not in asok_path: 2021-02-10T11:45:04.187 DEBUG:teuthology.orchestra.run.gibba012:> assert(len(files) == 1) 2021-02-10T11:45:04.187 DEBUG:teuthology.orchestra.run.gibba012:> return files[0] 2021-02-10T11:45:04.187 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:45:04.187 DEBUG:teuthology.orchestra.run.gibba012:> for f in files: 2021-02-10T11:45:04.187 DEBUG:teuthology.orchestra.run.gibba012:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:45:04.187 DEBUG:teuthology.orchestra.run.gibba012:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:45:04.187 DEBUG:teuthology.orchestra.run.gibba012:> return f 2021-02-10T11:45:04.187 DEBUG:teuthology.orchestra.run.gibba012:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:45:04.188 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:45:04.188 DEBUG:teuthology.orchestra.run.gibba012:> print(find_socket("client.1")) 2021-02-10T11:45:04.188 DEBUG:teuthology.orchestra.run.gibba012:> ' 2021-02-10T11:45:04.549 INFO:teuthology.orchestra.run.gibba012.stdout:/var/run/ceph/ceph-client.1.177164.asok 2021-02-10T11:45:04.552 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.177164.asok 2021-02-10T11:45:04.552 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:45:04.552 DEBUG:teuthology.orchestra.run.gibba012:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.177164.asok status 2021-02-10T11:45:04.943 INFO:teuthology.orchestra.run.gibba012.stdout:{ 2021-02-10T11:45:04.943 INFO:teuthology.orchestra.run.gibba012.stdout: "metadata": { 2021-02-10T11:45:04.943 INFO:teuthology.orchestra.run.gibba012.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:45:04.943 INFO:teuthology.orchestra.run.gibba012.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:45:04.943 INFO:teuthology.orchestra.run.gibba012.stdout: "entity_id": "1", 2021-02-10T11:45:04.943 INFO:teuthology.orchestra.run.gibba012.stdout: "hostname": "gibba012", 2021-02-10T11:45:04.943 INFO:teuthology.orchestra.run.gibba012.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2021-02-10T11:45:04.943 INFO:teuthology.orchestra.run.gibba012.stdout: "pid": "177164", 2021-02-10T11:45:04.943 INFO:teuthology.orchestra.run.gibba012.stdout: "root": "/" 2021-02-10T11:45:04.944 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:45:04.944 INFO:teuthology.orchestra.run.gibba012.stdout: "dentry_count": 0, 2021-02-10T11:45:04.944 INFO:teuthology.orchestra.run.gibba012.stdout: "dentry_pinned_count": 0, 2021-02-10T11:45:04.944 INFO:teuthology.orchestra.run.gibba012.stdout: "id": 12186, 2021-02-10T11:45:04.944 INFO:teuthology.orchestra.run.gibba012.stdout: "inst": { 2021-02-10T11:45:04.944 INFO:teuthology.orchestra.run.gibba012.stdout: "name": { 2021-02-10T11:45:04.944 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "client", 2021-02-10T11:45:04.944 INFO:teuthology.orchestra.run.gibba012.stdout: "num": 12186 2021-02-10T11:45:04.944 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:45:04.944 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": { 2021-02-10T11:45:04.944 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:45:04.944 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:0", 2021-02-10T11:45:04.944 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 3183913259 2021-02-10T11:45:04.945 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:45:04.945 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:45:04.945 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": { 2021-02-10T11:45:04.945 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:45:04.945 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:0", 2021-02-10T11:45:04.945 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 3183913259 2021-02-10T11:45:04.945 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:45:04.945 INFO:teuthology.orchestra.run.gibba012.stdout: "inst_str": "client.12186 172.21.2.112:0/3183913259", 2021-02-10T11:45:04.945 INFO:teuthology.orchestra.run.gibba012.stdout: "addr_str": "172.21.2.112:0/3183913259", 2021-02-10T11:45:04.945 INFO:teuthology.orchestra.run.gibba012.stdout: "inode_count": 1, 2021-02-10T11:45:04.945 INFO:teuthology.orchestra.run.gibba012.stdout: "mds_epoch": 130, 2021-02-10T11:45:04.945 INFO:teuthology.orchestra.run.gibba012.stdout: "osd_epoch": 151, 2021-02-10T11:45:04.945 INFO:teuthology.orchestra.run.gibba012.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:45:04.945 INFO:teuthology.orchestra.run.gibba012.stdout: "blacklisted": false 2021-02-10T11:45:04.946 INFO:teuthology.orchestra.run.gibba012.stdout:} 2021-02-10T11:45:04.947 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:45:04.947 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:45:04.947 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:45:05.283 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:45:05.283 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:45:05.284 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:45:05.284 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:45:05.630 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2021-02-10T11:45:06.063 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.044 7f520215a700 1 Processor -- start 2021-02-10T11:45:06.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.044 7f520215a700 1 -- start start 2021-02-10T11:45:06.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.044 7f520215a700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f51fc0d1c80 0x7f51fc0cef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:06.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.044 7f520215a700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51fc0d0180 con 0x7f51fc0d1880 2021-02-10T11:45:06.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.044 7f520215a700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51fc0fe1d0 con 0x7f51fc0cf590 2021-02-10T11:45:06.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.044 7f520215a700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51fc1b0cc0 con 0x7f51fc0d1c80 2021-02-10T11:45:06.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.045 7f51fb7fe700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f51fc0d1880 0x7f51fc1b04d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:57894/0 (socket says 172.21.2.112:57894) 2021-02-10T11:45:06.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.045 7f51fb7fe700 1 -- 172.21.2.112:0/2282653058 learned_addr learned my addr 172.21.2.112:0/2282653058 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:45:06.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.045 7f51fa7fc700 1 -- 172.21.2.112:0/2282653058 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3201855732 0 0) 0x7f51ec001180 con 0x7f51fc0d1880 2021-02-10T11:45:06.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.045 7f51faffd700 1 --2- 172.21.2.112:0/2282653058 >> v2:172.21.2.120:3300/0 conn(0x7f51fc0d1c80 0x7f51fc0cef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:06.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.045 7f51fa7fc700 1 -- 172.21.2.112:0/2282653058 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51e4002fe0 con 0x7f51fc0d1880 2021-02-10T11:45:06.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.045 7f51fa7fc700 1 -- 172.21.2.112:0/2282653058 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 655827260 0 0) 0x7f51f4001400 con 0x7f51fc0cf590 2021-02-10T11:45:06.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.045 7f51fa7fc700 1 -- 172.21.2.112:0/2282653058 --> v1:172.21.2.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51e4003210 con 0x7f51fc0cf590 2021-02-10T11:45:06.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.045 7f51fa7fc700 1 -- 172.21.2.112:0/2282653058 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 597881052 0 0) 0x7f51ec001180 con 0x7f51fc0d1880 2021-02-10T11:45:06.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.046 7f51fa7fc700 1 -- 172.21.2.112:0/2282653058 >> v2:172.21.2.120:3300/0 conn(0x7f51fc0d1c80 msgr2=0x7f51fc0cef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:06.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.046 7f51fa7fc700 1 --2- 172.21.2.112:0/2282653058 >> v2:172.21.2.120:3300/0 conn(0x7f51fc0d1c80 0x7f51fc0cef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:06.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.046 7f51fa7fc700 1 -- 172.21.2.112:0/2282653058 >> v1:172.21.2.120:6789/0 conn(0x7f51fc0cf590 legacy=0x7f51fc0cf990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:06.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.046 7f51fa7fc700 1 -- 172.21.2.112:0/2282653058 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51fc109a70 con 0x7f51fc0d1880 2021-02-10T11:45:06.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.046 7f51fa7fc700 1 -- 172.21.2.112:0/2282653058 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f51ec001680 con 0x7f51fc0d1880 2021-02-10T11:45:06.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.046 7f51fa7fc700 1 -- 172.21.2.112:0/2282653058 >> v1:172.21.2.112:6789/0 conn(0x7f51fc0d1880 legacy=0x7f51fc1b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:06.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.046 7f51fa7fc700 1 --2- 172.21.2.112:0/2282653058 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f51e40038e0 0x7f51e4003ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:06.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.046 7f51fa7fc700 1 --2- 172.21.2.112:0/2282653058 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f51e40067e0 0x7f51e4008ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:06.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.046 7f51fa7fc700 1 --2- 172.21.2.112:0/2282653058 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f51e4009580 0x7f51e400baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:06.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.046 7f51fa7fc700 1 -- 172.21.2.112:0/2282653058 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51e400c2f0 con 0x7f51e40067e0 2021-02-10T11:45:06.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.046 7f51fa7fc700 1 -- 172.21.2.112:0/2282653058 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51e40504e0 con 0x7f51e40038e0 2021-02-10T11:45:06.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.046 7f51fa7fc700 1 -- 172.21.2.112:0/2282653058 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51e4050700 con 0x7f51e4009580 2021-02-10T11:45:06.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.046 7f51fb7fe700 1 --2- 172.21.2.112:0/2282653058 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f51e40038e0 0x7f51e4003ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:06.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.046 7f51faffd700 1 --2- 172.21.2.112:0/2282653058 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f51e40067e0 0x7f51e4008ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:06.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.046 7f51fbfff700 1 --2- 172.21.2.112:0/2282653058 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f51e4009580 0x7f51e400baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:06.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.046 7f51faffd700 1 -- 172.21.2.112:0/2282653058 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f51e4009580 msgr2=0x7f51e400baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:06.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.046 7f51faffd700 1 --2- 172.21.2.112:0/2282653058 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f51e4009580 0x7f51e400baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:06.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.046 7f51faffd700 1 -- 172.21.2.112:0/2282653058 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f51e40038e0 msgr2=0x7f51e4003ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:06.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.046 7f51faffd700 1 --2- 172.21.2.112:0/2282653058 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f51e40038e0 0x7f51e4003ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:06.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.046 7f51faffd700 1 -- 172.21.2.112:0/2282653058 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51e40509a0 con 0x7f51e40067e0 2021-02-10T11:45:06.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.046 7f51fbfff700 1 -- 172.21.2.112:0/2282653058 reap_dead start 2021-02-10T11:45:06.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.046 7f51faffd700 1 --2- 172.21.2.112:0/2282653058 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f51e40067e0 0x7f51e4008ca0 secure :-1 s=READY pgs=2695 cs=0 l=1 rev1=1 rx=0x7f51f000c8f0 tx=0x7f51f0006c20).ready entity=mon.0 client_cookie=31cf6a5c8962caa1 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:06.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.046 7f51fa7fc700 1 -- 172.21.2.112:0/2282653058 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f51f00090d0 con 0x7f51e40067e0 2021-02-10T11:45:06.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.046 7f51fa7fc700 1 -- 172.21.2.112:0/2282653058 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51f0009450 con 0x7f51e40067e0 2021-02-10T11:45:06.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.047 7f51fa7fc700 1 -- 172.21.2.112:0/2282653058 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f51f0009930 con 0x7f51e40067e0 2021-02-10T11:45:06.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.047 7f520215a700 1 -- 172.21.2.112:0/2282653058 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f51e40067e0 msgr2=0x7f51e4008ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:06.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.047 7f520215a700 1 --2- 172.21.2.112:0/2282653058 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f51e40067e0 0x7f51e4008ca0 secure :-1 s=READY pgs=2695 cs=0 l=1 rev1=1 rx=0x7f51f000c8f0 tx=0x7f51f0006c20).stop 2021-02-10T11:45:06.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.047 7f520215a700 1 -- 172.21.2.112:0/2282653058 shutdown_connections 2021-02-10T11:45:06.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.047 7f520215a700 1 --2- 172.21.2.112:0/2282653058 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f51e40067e0 0x7f51e4008ca0 unknown :-1 s=CLOSED pgs=2695 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:06.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.047 7f520215a700 1 -- 172.21.2.112:0/2282653058 >> 172.21.2.112:0/2282653058 conn(0x7f51fc0ce430 msgr2=0x7f51fc0d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:06.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.047 7f520215a700 1 -- 172.21.2.112:0/2282653058 shutdown_connections 2021-02-10T11:45:06.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.047 7f520215a700 1 -- 172.21.2.112:0/2282653058 wait complete. 2021-02-10T11:45:06.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.062 7f520215a700 1 Processor -- start 2021-02-10T11:45:06.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.062 7f520215a700 1 -- start start 2021-02-10T11:45:06.069 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.062 7f520215a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f51fc188400 0x7f51fc188800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:06.069 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.062 7f520215a700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f51fc188e20 0x7f51fc189220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:06.069 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.062 7f520215a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f51fc09cbc0 0x7f51fc09cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:06.069 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.062 7f520215a700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51fc187e80 con 0x7f51fc188400 2021-02-10T11:45:06.069 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.062 7f520215a700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51fc0b27c0 con 0x7f51fc188e20 2021-02-10T11:45:06.069 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.062 7f520215a700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51fc0b29a0 con 0x7f51fc09cbc0 2021-02-10T11:45:06.069 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.063 7f51fb7fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f51fc188400 0x7f51fc188800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:06.069 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.063 7f51fb7fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f51fc188400 0x7f51fc188800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:58444/0 (socket says 172.21.2.112:58444) 2021-02-10T11:45:06.069 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.063 7f51fb7fe700 1 -- 172.21.2.112:0/2325192013 learned_addr learned my addr 172.21.2.112:0/2325192013 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:45:06.069 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.063 7f51fb7fe700 1 -- 172.21.2.112:0/2325192013 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f51fc09cbc0 msgr2=0x7f51fc09cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:06.069 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.063 7f51fb7fe700 1 --2- 172.21.2.112:0/2325192013 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f51fc09cbc0 0x7f51fc09cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:06.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.063 7f51fb7fe700 1 -- 172.21.2.112:0/2325192013 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f51fc188e20 msgr2=0x7f51fc189220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:06.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.063 7f51fb7fe700 1 --2- 172.21.2.112:0/2325192013 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f51fc188e20 0x7f51fc189220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:06.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.063 7f51fb7fe700 1 -- 172.21.2.112:0/2325192013 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51fc09d5e0 con 0x7f51fc188400 2021-02-10T11:45:06.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.063 7f51fb7fe700 1 --2- 172.21.2.112:0/2325192013 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f51fc188400 0x7f51fc188800 secure :-1 s=READY pgs=2696 cs=0 l=1 rev1=1 rx=0x7f51ec001170 tx=0x7f51ec00a8f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:06.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.063 7f51e3fff700 1 -- 172.21.2.112:0/2325192013 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f51ec001c10 con 0x7f51fc188400 2021-02-10T11:45:06.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.063 7f51e3fff700 1 -- 172.21.2.112:0/2325192013 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51ec007310 con 0x7f51fc188400 2021-02-10T11:45:06.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.063 7f51e3fff700 1 -- 172.21.2.112:0/2325192013 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f51ec0043a0 con 0x7f51fc188400 2021-02-10T11:45:06.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.063 7f520215a700 1 -- 172.21.2.112:0/2325192013 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51fc09d5e0 con 0x7f51fc188400 2021-02-10T11:45:06.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.063 7f520215a700 1 -- 172.21.2.112:0/2325192013 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51fc09d5e0 con 0x7f51fc188400 2021-02-10T11:45:06.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.064 7f51e3fff700 1 -- 172.21.2.112:0/2325192013 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f51ec008000 con 0x7f51fc188400 2021-02-10T11:45:06.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.064 7f51e3fff700 1 --2- 172.21.2.112:0/2325192013 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f51dc018730 0x7f51dc01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:06.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.064 7f51e3fff700 1 -- 172.21.2.112:0/2325192013 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f51ec032620 con 0x7f51fc188400 2021-02-10T11:45:06.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.064 7f51faffd700 1 --2- 172.21.2.112:0/2325192013 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f51dc018730 0x7f51dc01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:06.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.065 7f51faffd700 1 --2- 172.21.2.112:0/2325192013 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f51dc018730 0x7f51dc01ac10 crc :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:06.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.094 7f520215a700 1 -- 172.21.2.112:0/2325192013 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51fc09db70 con 0x7f51fc188400 2021-02-10T11:45:06.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.097 7f51e3fff700 1 -- 172.21.2.112:0/2325192013 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f51ec007070 con 0x7f51fc188400 2021-02-10T11:45:06.237 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.236 7f520215a700 1 -- 172.21.2.112:0/2325192013 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f51fc082a70 con 0x7f51fc188400 2021-02-10T11:45:06.291 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.290 7f51e3fff700 1 -- 172.21.2.112:0/2325192013 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v131) v1 ==== 136+0+0 (secure 0 0 0) 0x7f51ec007280 con 0x7f51fc188400 2021-02-10T11:45:06.303 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.302 7f520215a700 1 -- 172.21.2.112:0/2325192013 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f51dc018730 msgr2=0x7f51dc01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:06.303 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.302 7f520215a700 1 --2- 172.21.2.112:0/2325192013 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f51dc018730 0x7f51dc01ac10 crc :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:06.303 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.302 7f520215a700 1 -- 172.21.2.112:0/2325192013 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f51fc188400 msgr2=0x7f51fc188800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:06.303 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.302 7f520215a700 1 --2- 172.21.2.112:0/2325192013 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f51fc188400 0x7f51fc188800 secure :-1 s=READY pgs=2696 cs=0 l=1 rev1=1 rx=0x7f51ec001170 tx=0x7f51ec00a8f0).stop 2021-02-10T11:45:06.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.302 7f520215a700 1 -- 172.21.2.112:0/2325192013 shutdown_connections 2021-02-10T11:45:06.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.302 7f520215a700 1 --2- 172.21.2.112:0/2325192013 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f51dc018730 0x7f51dc01ac10 unknown :-1 s=CLOSED pgs=1230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:06.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.302 7f520215a700 1 --2- 172.21.2.112:0/2325192013 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f51fc188e20 0x7f51fc189220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:06.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.302 7f520215a700 1 --2- 172.21.2.112:0/2325192013 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f51fc09cbc0 0x7f51fc09cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:06.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.302 7f520215a700 1 --2- 172.21.2.112:0/2325192013 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f51fc188400 0x7f51fc188800 unknown :-1 s=CLOSED pgs=2696 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:06.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.302 7f520215a700 1 -- 172.21.2.112:0/2325192013 >> 172.21.2.112:0/2325192013 conn(0x7f51fc0c2cd0 msgr2=0x7f51fc0b5950 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:06.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.302 7f520215a700 1 -- 172.21.2.112:0/2325192013 shutdown_connections 2021-02-10T11:45:06.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.302 7f520215a700 1 -- 172.21.2.112:0/2325192013 wait complete. 2021-02-10T11:45:06.312 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs volume ls 2021-02-10T11:45:06.741 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.723 7fd96909e700 1 Processor -- start 2021-02-10T11:45:06.741 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.723 7fd96909e700 1 -- start start 2021-02-10T11:45:06.741 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.723 7fd96909e700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fd9640d1870 0x7fd9641b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:06.741 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.723 7fd96909e700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fd9641b0a10 0x7fd9640cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:06.741 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.723 7fd96909e700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9640f0b10 con 0x7fd9640d1c70 2021-02-10T11:45:06.741 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.723 7fd96909e700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9640e5ed0 con 0x7fd9640d1870 2021-02-10T11:45:06.741 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.723 7fd96909e700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9640f0650 con 0x7fd9641b0a10 2021-02-10T11:45:06.741 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.724 7fd962d9d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fd9640d1870 0x7fd9641b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:06.742 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.724 7fd962d9d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fd9640d1870 0x7fd9641b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:58452/0 (socket says 172.21.2.112:58452) 2021-02-10T11:45:06.742 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.724 7fd962d9d700 1 -- 172.21.2.112:0/202904639 learned_addr learned my addr 172.21.2.112:0/202904639 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:45:06.742 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.724 7fd96259c700 1 --2- 172.21.2.112:0/202904639 >> v2:172.21.2.120:3300/0 conn(0x7fd9641b0a10 0x7fd9640cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:06.742 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.724 7fd962d9d700 1 -- 172.21.2.112:0/202904639 >> v2:172.21.2.120:3300/0 conn(0x7fd9641b0a10 msgr2=0x7fd9640cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:06.742 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.724 7fd962d9d700 1 --2- 172.21.2.112:0/202904639 >> v2:172.21.2.120:3300/0 conn(0x7fd9641b0a10 0x7fd9640cf330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:06.742 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.724 7fd962d9d700 1 -- 172.21.2.112:0/202904639 >> v1:172.21.2.120:6789/0 conn(0x7fd9640d1c70 legacy=0x7fd9640cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:06.742 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.724 7fd962d9d700 1 -- 172.21.2.112:0/202904639 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9640f0840 con 0x7fd9640d1870 2021-02-10T11:45:06.742 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.724 7fd96259c700 1 --2- 172.21.2.112:0/202904639 >> v2:172.21.2.120:3300/0 conn(0x7fd9641b0a10 0x7fd9640cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:45:06.742 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.724 7fd962d9d700 1 --2- 172.21.2.112:0/202904639 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd9640d1870 0x7fd9641b03f0 secure :-1 s=READY pgs=2697 cs=0 l=1 rev1=1 rx=0x7fd94c006700 tx=0x7fd94c009b80).ready entity=mon.0 client_cookie=8c30dc3d86d57d9f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:06.742 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.724 7fd961d9b700 1 -- 172.21.2.112:0/202904639 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd94c00c0c0 con 0x7fd9640d1870 2021-02-10T11:45:06.742 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.725 7fd961d9b700 1 -- 172.21.2.112:0/202904639 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd94c00c340 con 0x7fd9640d1870 2021-02-10T11:45:06.742 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.725 7fd961d9b700 1 -- 172.21.2.112:0/202904639 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd94c00c820 con 0x7fd9640d1870 2021-02-10T11:45:06.743 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.725 7fd96909e700 1 -- 172.21.2.112:0/202904639 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd9640d1870 msgr2=0x7fd9641b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:06.743 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.725 7fd96909e700 1 --2- 172.21.2.112:0/202904639 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd9640d1870 0x7fd9641b03f0 secure :-1 s=READY pgs=2697 cs=0 l=1 rev1=1 rx=0x7fd94c006700 tx=0x7fd94c009b80).stop 2021-02-10T11:45:06.743 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.725 7fd96909e700 1 -- 172.21.2.112:0/202904639 shutdown_connections 2021-02-10T11:45:06.743 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.725 7fd96909e700 1 --2- 172.21.2.112:0/202904639 >> v2:172.21.2.120:3300/0 conn(0x7fd9641b0a10 0x7fd9640cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:06.743 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.725 7fd96909e700 1 --2- 172.21.2.112:0/202904639 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd9640d1870 0x7fd9641b03f0 unknown :-1 s=CLOSED pgs=2697 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:06.743 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.725 7fd96909e700 1 -- 172.21.2.112:0/202904639 >> 172.21.2.112:0/202904639 conn(0x7fd9640ce450 msgr2=0x7fd9640d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:06.743 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.725 7fd96909e700 1 -- 172.21.2.112:0/202904639 shutdown_connections 2021-02-10T11:45:06.743 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.725 7fd96909e700 1 -- 172.21.2.112:0/202904639 wait complete. 2021-02-10T11:45:06.745 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.740 7fd96909e700 1 Processor -- start 2021-02-10T11:45:06.745 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.740 7fd96909e700 1 -- start start 2021-02-10T11:45:06.745 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.740 7fd96909e700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd9640b7510 0x7fd9640b7910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:06.745 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.740 7fd96909e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd96409d530 0x7fd96409d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:06.745 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.740 7fd96909e700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd96409df50 0x7fd96409eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:06.745 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.740 7fd96909e700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9640b8140 con 0x7fd96409d530 2021-02-10T11:45:06.746 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.740 7fd96909e700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd96409e750 con 0x7fd9640b7510 2021-02-10T11:45:06.746 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.740 7fd96909e700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd96409e350 con 0x7fd96409df50 2021-02-10T11:45:06.746 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.740 7fd96259c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd96409d530 0x7fd96409d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:06.746 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.741 7fd96259c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd96409d530 0x7fd96409d930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:58458/0 (socket says 172.21.2.112:58458) 2021-02-10T11:45:06.746 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.741 7fd96259c700 1 -- 172.21.2.112:0/796408460 learned_addr learned my addr 172.21.2.112:0/796408460 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:45:06.746 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.741 7fd96359e700 1 --2- 172.21.2.112:0/796408460 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd96409df50 0x7fd96409eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:06.746 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.741 7fd962d9d700 1 --2- 172.21.2.112:0/796408460 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd9640b7510 0x7fd9640b7910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:06.746 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.741 7fd96259c700 1 -- 172.21.2.112:0/796408460 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd96409df50 msgr2=0x7fd96409eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:06.746 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.741 7fd96259c700 1 --2- 172.21.2.112:0/796408460 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd96409df50 0x7fd96409eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:06.746 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.741 7fd96259c700 1 -- 172.21.2.112:0/796408460 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd9640b7510 msgr2=0x7fd9640b7910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:06.746 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.741 7fd96259c700 1 --2- 172.21.2.112:0/796408460 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd9640b7510 0x7fd9640b7910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:06.746 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.741 7fd96259c700 1 -- 172.21.2.112:0/796408460 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd96409f2c0 con 0x7fd96409d530 2021-02-10T11:45:06.746 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.741 7fd96259c700 1 --2- 172.21.2.112:0/796408460 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd96409d530 0x7fd96409d930 secure :-1 s=READY pgs=2698 cs=0 l=1 rev1=1 rx=0x7fd954006f40 tx=0x7fd95400b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:06.747 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.741 7fd95b7fe700 1 -- 172.21.2.112:0/796408460 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd95400bd80 con 0x7fd96409d530 2021-02-10T11:45:06.747 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.741 7fd95b7fe700 1 -- 172.21.2.112:0/796408460 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd954000c90 con 0x7fd96409d530 2021-02-10T11:45:06.747 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.741 7fd96909e700 1 -- 172.21.2.112:0/796408460 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd96409f2c0 con 0x7fd96409d530 2021-02-10T11:45:06.747 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.741 7fd95b7fe700 1 -- 172.21.2.112:0/796408460 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd954002220 con 0x7fd96409d530 2021-02-10T11:45:06.747 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.741 7fd96909e700 1 -- 172.21.2.112:0/796408460 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd96409f850 con 0x7fd96409d530 2021-02-10T11:45:06.747 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.742 7fd95b7fe700 1 -- 172.21.2.112:0/796408460 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd954006b00 con 0x7fd96409d530 2021-02-10T11:45:06.747 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.742 7fd95b7fe700 1 --2- 172.21.2.112:0/796408460 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd9480186c0 0x7fd94801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:06.747 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.742 7fd95b7fe700 1 -- 172.21.2.112:0/796408460 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fd954032400 con 0x7fd96409d530 2021-02-10T11:45:06.747 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.742 7fd962d9d700 1 --2- 172.21.2.112:0/796408460 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd9480186c0 0x7fd94801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:06.747 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.743 7fd962d9d700 1 --2- 172.21.2.112:0/796408460 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd9480186c0 0x7fd94801abc0 crc :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:06.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.773 7fd96909e700 1 -- 172.21.2.112:0/796408460 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9640ebc50 con 0x7fd96409d530 2021-02-10T11:45:06.777 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.776 7fd95b7fe700 1 -- 172.21.2.112:0/796408460 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd954000940 con 0x7fd96409d530 2021-02-10T11:45:06.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.910 7fd96909e700 1 -- 172.21.2.112:0/796408460 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs volume ls", "target": ["mgr", ""]}) v1 -- 0x7fd96409f850 con 0x7fd9480186c0 2021-02-10T11:45:06.913 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.911 7fd95b7fe700 1 -- 172.21.2.112:0/796408460 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+41 (crc 0 0 0) 0x7fd94c006430 con 0x7fd9480186c0 2021-02-10T11:45:06.914 INFO:teuthology.orchestra.run.gibba012.stdout:[ 2021-02-10T11:45:06.914 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:45:06.915 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "cephfs" 2021-02-10T11:45:06.915 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:45:06.915 INFO:teuthology.orchestra.run.gibba012.stdout:] 2021-02-10T11:45:06.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.915 7fd96909e700 1 -- 172.21.2.112:0/796408460 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd9480186c0 msgr2=0x7fd94801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:06.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.915 7fd96909e700 1 --2- 172.21.2.112:0/796408460 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd9480186c0 0x7fd94801abc0 crc :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:06.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.915 7fd96909e700 1 -- 172.21.2.112:0/796408460 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd96409d530 msgr2=0x7fd96409d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:06.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.915 7fd96909e700 1 --2- 172.21.2.112:0/796408460 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd96409d530 0x7fd96409d930 secure :-1 s=READY pgs=2698 cs=0 l=1 rev1=1 rx=0x7fd954006f40 tx=0x7fd95400b990).stop 2021-02-10T11:45:06.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.915 7fd96909e700 1 -- 172.21.2.112:0/796408460 shutdown_connections 2021-02-10T11:45:06.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.915 7fd96909e700 1 --2- 172.21.2.112:0/796408460 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd9480186c0 0x7fd94801abc0 unknown :-1 s=CLOSED pgs=1231 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:06.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.915 7fd96909e700 1 --2- 172.21.2.112:0/796408460 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd96409df50 0x7fd96409eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:06.917 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.915 7fd96909e700 1 --2- 172.21.2.112:0/796408460 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd96409d530 0x7fd96409d930 unknown :-1 s=CLOSED pgs=2698 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:06.917 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.915 7fd96909e700 1 --2- 172.21.2.112:0/796408460 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd9640b7510 0x7fd9640b7910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:06.917 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.915 7fd96909e700 1 -- 172.21.2.112:0/796408460 >> 172.21.2.112:0/796408460 conn(0x7fd9640b83b0 msgr2=0x7fd9640a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:06.917 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.915 7fd96909e700 1 -- 172.21.2.112:0/796408460 shutdown_connections 2021-02-10T11:45:06.917 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:06.915 7fd96909e700 1 -- 172.21.2.112:0/796408460 wait complete. 2021-02-10T11:45:06.925 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:45:07.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.341 7f20c6b10700 1 Processor -- start 2021-02-10T11:45:07.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.341 7f20c6b10700 1 -- start start 2021-02-10T11:45:07.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.341 7f20c6b10700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f20c010aff0 0x7f20c00cf750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:07.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.341 7f20c6b10700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20c00fe1c0 con 0x7f20c00d1c70 2021-02-10T11:45:07.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.341 7f20c6b10700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20c01b0cb0 con 0x7f20c00d1870 2021-02-10T11:45:07.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.341 7f20c6b10700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20c0109a60 con 0x7f20c010aff0 2021-02-10T11:45:07.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.342 7f20bffff700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f20c00d1c70 0x7f20c00cef60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:57930/0 (socket says 172.21.2.112:57930) 2021-02-10T11:45:07.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.342 7f20bffff700 1 -- 172.21.2.112:0/649117279 learned_addr learned my addr 172.21.2.112:0/649117279 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:45:07.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.342 7f20c512d700 1 --2- 172.21.2.112:0/649117279 >> v2:172.21.2.112:3300/0 conn(0x7f20c010aff0 0x7f20c00cf750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:07.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.342 7f20c512d700 1 -- 172.21.2.112:0/649117279 >> v1:172.21.2.120:6789/0 conn(0x7f20c00d1870 legacy=0x7f20c01b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:07.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.342 7f20c512d700 1 -- 172.21.2.112:0/649117279 >> v1:172.21.2.112:6789/0 conn(0x7f20c00d1c70 legacy=0x7f20c00cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:07.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.342 7f20c512d700 1 -- 172.21.2.112:0/649117279 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20c010b3f0 con 0x7f20c010aff0 2021-02-10T11:45:07.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.342 7f20c512d700 1 --2- 172.21.2.112:0/649117279 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f20c010aff0 0x7f20c00cf750 secure :-1 s=READY pgs=2700 cs=0 l=1 rev1=1 rx=0x7f20b8006e30 tx=0x7f20b8009e60).ready entity=mon.0 client_cookie=f745e57fbea8a6c7 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:07.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.342 7f20bf7fe700 1 -- 172.21.2.112:0/649117279 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f20b800b300 con 0x7f20c010aff0 2021-02-10T11:45:07.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.342 7f20bf7fe700 1 -- 172.21.2.112:0/649117279 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20b800b580 con 0x7f20c010aff0 2021-02-10T11:45:07.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.342 7f20bf7fe700 1 -- 172.21.2.112:0/649117279 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f20b800ba60 con 0x7f20c010aff0 2021-02-10T11:45:07.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.342 7f20c6b10700 1 -- 172.21.2.112:0/649117279 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f20c010aff0 msgr2=0x7f20c00cf750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:07.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.342 7f20c6b10700 1 --2- 172.21.2.112:0/649117279 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f20c010aff0 0x7f20c00cf750 secure :-1 s=READY pgs=2700 cs=0 l=1 rev1=1 rx=0x7f20b8006e30 tx=0x7f20b8009e60).stop 2021-02-10T11:45:07.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.343 7f20c6b10700 1 -- 172.21.2.112:0/649117279 shutdown_connections 2021-02-10T11:45:07.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.343 7f20c6b10700 1 --2- 172.21.2.112:0/649117279 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f20c010aff0 0x7f20c00cf750 unknown :-1 s=CLOSED pgs=2700 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:07.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.343 7f20c6b10700 1 -- 172.21.2.112:0/649117279 >> 172.21.2.112:0/649117279 conn(0x7f20c00ce420 msgr2=0x7f20c00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:07.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.343 7f20c6b10700 1 -- 172.21.2.112:0/649117279 shutdown_connections 2021-02-10T11:45:07.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.343 7f20c6b10700 1 -- 172.21.2.112:0/649117279 wait complete. 2021-02-10T11:45:07.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.358 7f20c6b10700 1 Processor -- start 2021-02-10T11:45:07.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.358 7f20c6b10700 1 -- start start 2021-02-10T11:45:07.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.358 7f20c6b10700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f20c00c0740 0x7f20c01c5580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:07.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.358 7f20c6b10700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f20c01c5ba0 0x7f20c00a5c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:07.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.358 7f20c6b10700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f20c00b6e60 0x7f20c00a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:07.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.358 7f20c6b10700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20c00a6880 con 0x7f20c00b6e60 2021-02-10T11:45:07.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.358 7f20c6b10700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20c00a6aa0 con 0x7f20c00c0740 2021-02-10T11:45:07.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.358 7f20c6b10700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20c00a6cc0 con 0x7f20c01c5ba0 2021-02-10T11:45:07.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.358 7f20c512d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f20c00b6e60 0x7f20c00a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:07.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.358 7f20c512d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f20c00b6e60 0x7f20c00a6260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:58470/0 (socket says 172.21.2.112:58470) 2021-02-10T11:45:07.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.358 7f20c512d700 1 -- 172.21.2.112:0/349360734 learned_addr learned my addr 172.21.2.112:0/349360734 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:45:07.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.359 7f20c512d700 1 -- 172.21.2.112:0/349360734 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f20c01c5ba0 msgr2=0x7f20c00a5c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:07.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.359 7f20c512d700 1 --2- 172.21.2.112:0/349360734 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f20c01c5ba0 0x7f20c00a5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:07.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.359 7f20c512d700 1 -- 172.21.2.112:0/349360734 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f20c00c0740 msgr2=0x7f20c01c5580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:07.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.359 7f20c512d700 1 --2- 172.21.2.112:0/349360734 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f20c00c0740 0x7f20c01c5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:07.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.359 7f20c512d700 1 -- 172.21.2.112:0/349360734 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20c00a6ee0 con 0x7f20c00b6e60 2021-02-10T11:45:07.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.359 7f20c512d700 1 --2- 172.21.2.112:0/349360734 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f20c00b6e60 0x7f20c00a6260 secure :-1 s=READY pgs=2701 cs=0 l=1 rev1=1 rx=0x7f20b8009a00 tx=0x7f20b800bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:07.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.359 7f20bd7fa700 1 -- 172.21.2.112:0/349360734 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f20b8005820 con 0x7f20c00b6e60 2021-02-10T11:45:07.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.359 7f20bd7fa700 1 -- 172.21.2.112:0/349360734 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20b8006d80 con 0x7f20c00b6e60 2021-02-10T11:45:07.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.359 7f20c6b10700 1 -- 172.21.2.112:0/349360734 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20c00a6ee0 con 0x7f20c00b6e60 2021-02-10T11:45:07.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.359 7f20c6b10700 1 -- 172.21.2.112:0/349360734 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20c00b5650 con 0x7f20c00b6e60 2021-02-10T11:45:07.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.359 7f20bd7fa700 1 -- 172.21.2.112:0/349360734 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f20b8005c90 con 0x7f20c00b6e60 2021-02-10T11:45:07.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.360 7f20bd7fa700 1 -- 172.21.2.112:0/349360734 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f20b8005820 con 0x7f20c00b6e60 2021-02-10T11:45:07.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.360 7f20bd7fa700 1 --2- 172.21.2.112:0/349360734 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f20a40186c0 0x7f20a401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:07.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.360 7f20bd7fa700 1 -- 172.21.2.112:0/349360734 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f20b8032910 con 0x7f20c00b6e60 2021-02-10T11:45:07.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.360 7f20c492c700 1 --2- 172.21.2.112:0/349360734 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f20a40186c0 0x7f20a401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:07.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.361 7f20c492c700 1 --2- 172.21.2.112:0/349360734 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f20a40186c0 0x7f20a401abc0 crc :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:07.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.391 7f20c6b10700 1 -- 172.21.2.112:0/349360734 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20c00807d0 con 0x7f20c00b6e60 2021-02-10T11:45:07.394 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.393 7f20bd7fa700 1 -- 172.21.2.112:0/349360734 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f20b8028ee0 con 0x7f20c00b6e60 2021-02-10T11:45:07.545 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.544 7f20c6b10700 1 -- 172.21.2.112:0/349360734 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f20c0088070 con 0x7f20c00b6e60 2021-02-10T11:45:07.546 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.545 7f20bd7fa700 1 -- 172.21.2.112:0/349360734 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v38)=0 v38) v1 ==== 122+0+0 (secure 0 0 0) 0x7f20b8028ee0 con 0x7f20c00b6e60 2021-02-10T11:45:07.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.546 7f20c6b10700 1 -- 172.21.2.112:0/349360734 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f20a40186c0 msgr2=0x7f20a401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:07.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.546 7f20c6b10700 1 --2- 172.21.2.112:0/349360734 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f20a40186c0 0x7f20a401abc0 crc :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:07.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.546 7f20c6b10700 1 -- 172.21.2.112:0/349360734 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f20c00b6e60 msgr2=0x7f20c00a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:07.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.546 7f20c6b10700 1 --2- 172.21.2.112:0/349360734 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f20c00b6e60 0x7f20c00a6260 secure :-1 s=READY pgs=2701 cs=0 l=1 rev1=1 rx=0x7f20b8009a00 tx=0x7f20b800bd40).stop 2021-02-10T11:45:07.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.547 7f20c6b10700 1 -- 172.21.2.112:0/349360734 shutdown_connections 2021-02-10T11:45:07.549 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.547 7f20c6b10700 1 --2- 172.21.2.112:0/349360734 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f20a40186c0 0x7f20a401abc0 unknown :-1 s=CLOSED pgs=1232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:07.549 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.547 7f20c6b10700 1 --2- 172.21.2.112:0/349360734 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f20c00b6e60 0x7f20c00a6260 unknown :-1 s=CLOSED pgs=2701 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:07.549 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.547 7f20c6b10700 1 --2- 172.21.2.112:0/349360734 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f20c01c5ba0 0x7f20c00a5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:07.549 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.547 7f20c6b10700 1 --2- 172.21.2.112:0/349360734 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f20c00c0740 0x7f20c01c5580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:07.549 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.547 7f20c6b10700 1 -- 172.21.2.112:0/349360734 >> 172.21.2.112:0/349360734 conn(0x7f20c00c4120 msgr2=0x7f20c00ffd60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:07.549 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.547 7f20c6b10700 1 -- 172.21.2.112:0/349360734 shutdown_connections 2021-02-10T11:45:07.549 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.547 7f20c6b10700 1 -- 172.21.2.112:0/349360734 wait complete. 2021-02-10T11:45:07.556 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs non_existent_subvolume 2021-02-10T11:45:07.989 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.970 7fa2b7d21700 1 Processor -- start 2021-02-10T11:45:07.989 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.971 7fa2b7d21700 1 -- start start 2021-02-10T11:45:07.989 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.971 7fa2b7d21700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa2b00d1c70 0x7fa2b00cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:07.989 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.971 7fa2b7d21700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fa2b00cf580 0x7fa2b00cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:07.989 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.971 7fa2b7d21700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2b00fe1c0 con 0x7fa2b00d1870 2021-02-10T11:45:07.989 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.971 7fa2b7d21700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2b01b0cb0 con 0x7fa2b00cf580 2021-02-10T11:45:07.989 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.971 7fa2b7d21700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2b0109a60 con 0x7fa2b00d1c70 2021-02-10T11:45:07.989 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.971 7fa2b5b3d700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fa2b00d1870 0x7fa2b01b04c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:57940/0 (socket says 172.21.2.112:57940) 2021-02-10T11:45:07.990 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.971 7fa2b5b3d700 1 -- 172.21.2.112:0/329027233 learned_addr learned my addr 172.21.2.112:0/329027233 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:45:07.990 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.971 7fa2b633e700 1 --2- 172.21.2.112:0/329027233 >> v2:172.21.2.120:3300/0 conn(0x7fa2b00cf580 0x7fa2b00cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:07.990 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.971 7fa2b533c700 1 --2- 172.21.2.112:0/329027233 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa2b00d1c70 0x7fa2b00cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:07.990 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.972 7fa2b633e700 1 -- 172.21.2.112:0/329027233 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa2b00d1c70 msgr2=0x7fa2b00cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:07.990 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.972 7fa2b633e700 1 --2- 172.21.2.112:0/329027233 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa2b00d1c70 0x7fa2b00cef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:07.990 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.972 7fa2b633e700 1 -- 172.21.2.112:0/329027233 >> v1:172.21.2.112:6789/0 conn(0x7fa2b00d1870 legacy=0x7fa2b01b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:07.990 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.972 7fa2b4b3b700 1 -- 172.21.2.112:0/329027233 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2721427145 0 0) 0x7fa2ac000ce0 con 0x7fa2b00d1870 2021-02-10T11:45:07.990 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.972 7fa2b633e700 1 -- 172.21.2.112:0/329027233 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2b0088070 con 0x7fa2b00cf580 2021-02-10T11:45:07.990 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.972 7fa2b633e700 1 --2- 172.21.2.112:0/329027233 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa2b00cf580 0x7fa2b00cf980 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 rx=0x7fa2a8006e50 tx=0x7fa2a8009e70).ready entity=mon.1 client_cookie=d2346fac1850b3d3 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:07.990 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.972 7fa2b4b3b700 1 -- 172.21.2.112:0/329027233 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa2a800b300 con 0x7fa2b00cf580 2021-02-10T11:45:07.990 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.972 7fa2b4b3b700 1 -- 172.21.2.112:0/329027233 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2a8003030 con 0x7fa2b00cf580 2021-02-10T11:45:07.990 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.972 7fa2b4b3b700 1 -- 172.21.2.112:0/329027233 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa2a800b8c0 con 0x7fa2b00cf580 2021-02-10T11:45:07.991 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.972 7fa2b7d21700 1 -- 172.21.2.112:0/329027233 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa2b00cf580 msgr2=0x7fa2b00cf980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:07.991 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.972 7fa2b7d21700 1 --2- 172.21.2.112:0/329027233 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa2b00cf580 0x7fa2b00cf980 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 rx=0x7fa2a8006e50 tx=0x7fa2a8009e70).stop 2021-02-10T11:45:07.991 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.973 7fa2b7d21700 1 -- 172.21.2.112:0/329027233 shutdown_connections 2021-02-10T11:45:07.991 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.973 7fa2b7d21700 1 --2- 172.21.2.112:0/329027233 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa2b00cf580 0x7fa2b00cf980 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:07.991 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.973 7fa2b7d21700 1 --2- 172.21.2.112:0/329027233 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa2b00d1c70 0x7fa2b00cef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:07.991 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.973 7fa2b7d21700 1 -- 172.21.2.112:0/329027233 >> 172.21.2.112:0/329027233 conn(0x7fa2b00ce420 msgr2=0x7fa2b00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:07.991 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.973 7fa2b7d21700 1 -- 172.21.2.112:0/329027233 shutdown_connections 2021-02-10T11:45:07.991 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.973 7fa2b7d21700 1 -- 172.21.2.112:0/329027233 wait complete. 2021-02-10T11:45:07.991 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.987 7fa2b7d21700 1 Processor -- start 2021-02-10T11:45:07.991 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.987 7fa2b7d21700 1 -- start start 2021-02-10T11:45:07.993 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.988 7fa2b7d21700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa2b0085bc0 0x7fa2b00863e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:07.993 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.988 7fa2b7d21700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa2b00954c0 0x7fa2b0096300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:07.993 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.988 7fa2b7d21700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa2b00958c0 0x7fa2b0096920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:07.993 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.988 7fa2b7d21700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2b0095d80 con 0x7fa2b00954c0 2021-02-10T11:45:07.993 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.988 7fa2b7d21700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2b00bfa60 con 0x7fa2b0085bc0 2021-02-10T11:45:07.993 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.988 7fa2b7d21700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2b0085fc0 con 0x7fa2b00958c0 2021-02-10T11:45:07.993 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.988 7fa2b533c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa2b00954c0 0x7fa2b0096300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:07.993 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.988 7fa2b533c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa2b00954c0 0x7fa2b0096300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:58484/0 (socket says 172.21.2.112:58484) 2021-02-10T11:45:07.993 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.988 7fa2b533c700 1 -- 172.21.2.112:0/1768304094 learned_addr learned my addr 172.21.2.112:0/1768304094 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:45:07.993 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.988 7fa2b633e700 1 --2- 172.21.2.112:0/1768304094 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa2b00958c0 0x7fa2b0096920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:07.994 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.988 7fa2b5b3d700 1 --2- 172.21.2.112:0/1768304094 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa2b0085bc0 0x7fa2b00863e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:07.994 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.989 7fa2b533c700 1 -- 172.21.2.112:0/1768304094 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa2b00958c0 msgr2=0x7fa2b0096920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:07.994 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.989 7fa2b533c700 1 --2- 172.21.2.112:0/1768304094 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa2b00958c0 0x7fa2b0096920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:07.994 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.989 7fa2b533c700 1 -- 172.21.2.112:0/1768304094 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa2b0085bc0 msgr2=0x7fa2b00863e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:07.994 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.989 7fa2b533c700 1 --2- 172.21.2.112:0/1768304094 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa2b0085bc0 0x7fa2b00863e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:07.994 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.989 7fa2b533c700 1 -- 172.21.2.112:0/1768304094 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2b0086c10 con 0x7fa2b00954c0 2021-02-10T11:45:07.994 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.989 7fa2b533c700 1 --2- 172.21.2.112:0/1768304094 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa2b00954c0 0x7fa2b0096300 secure :-1 s=READY pgs=2703 cs=0 l=1 rev1=1 rx=0x7fa2a4006f60 tx=0x7fa2a400b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:07.994 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.989 7fa2a27fc700 1 -- 172.21.2.112:0/1768304094 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa2a400bd60 con 0x7fa2b00954c0 2021-02-10T11:45:07.994 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.989 7fa2a27fc700 1 -- 172.21.2.112:0/1768304094 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2a4000c90 con 0x7fa2b00954c0 2021-02-10T11:45:07.994 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.989 7fa2a27fc700 1 -- 172.21.2.112:0/1768304094 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa2a40022b0 con 0x7fa2b00954c0 2021-02-10T11:45:07.994 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.989 7fa2b7d21700 1 -- 172.21.2.112:0/1768304094 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2b0086c10 con 0x7fa2b00954c0 2021-02-10T11:45:07.994 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.989 7fa2b7d21700 1 -- 172.21.2.112:0/1768304094 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2b0086c10 con 0x7fa2b00954c0 2021-02-10T11:45:07.995 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.990 7fa2a27fc700 1 -- 172.21.2.112:0/1768304094 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa2a4006b00 con 0x7fa2b00954c0 2021-02-10T11:45:07.995 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.990 7fa2a27fc700 1 --2- 172.21.2.112:0/1768304094 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa2980186c0 0x7fa29801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:07.995 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.990 7fa2a27fc700 1 -- 172.21.2.112:0/1768304094 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fa2a4032560 con 0x7fa2b00954c0 2021-02-10T11:45:07.995 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.990 7fa2b5b3d700 1 --2- 172.21.2.112:0/1768304094 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa2980186c0 0x7fa29801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:07.995 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:07.991 7fa2b5b3d700 1 --2- 172.21.2.112:0/1768304094 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa2980186c0 0x7fa29801abc0 crc :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:08.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:08.021 7fa2b7d21700 1 -- 172.21.2.112:0/1768304094 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2b00fe1c0 con 0x7fa2b00954c0 2021-02-10T11:45:08.024 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:08.023 7fa2a27fc700 1 -- 172.21.2.112:0/1768304094 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa2a402f070 con 0x7fa2b00954c0 2021-02-10T11:45:08.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:08.150 7fa2b7d21700 1 -- 172.21.2.112:0/1768304094 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "non_existent_subvolume", "target": ["mgr", ""]}) v1 -- 0x7fa2b00fe1c0 con 0x7fa2980186c0 2021-02-10T11:45:08.156 INFO:tasks.ceph.mgr.x.gibba020.stderr:2021-02-10 11:45:08.154 7f6382eee700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:45:08.156 INFO:tasks.ceph.mgr.x.gibba020.stderr:2021-02-10 11:45:08.154 7f6382eee700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:45:08.156 INFO:tasks.ceph.mgr.x.gibba020.stderr:2021-02-10 11:45:08.154 7f6382eee700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:45:08.156 INFO:tasks.ceph.mgr.x.gibba020.stderr:2021-02-10 11:45:08.154 7f6382eee700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:45:08.156 INFO:tasks.ceph.mgr.x.gibba020.stderr:2021-02-10 11:45:08.154 7f6382eee700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:45:08.160 INFO:tasks.ceph.mgr.x.gibba020.stderr:2021-02-10 11:45:08.159 7f6382eee700 -1 mgr.server reply reply (2) No such file or directory subvolume 'non_existent_subvolume' does not exist 2021-02-10T11:45:08.161 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:08.159 7fa2a27fc700 1 -- 172.21.2.112:0/1768304094 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: -2 subvolume 'non_existent_subvolume' does not exist) v1 ==== 57+0+0 (crc 0 0 0) 0x7fa2ac007cc0 con 0x7fa2980186c0 2021-02-10T11:45:08.167 INFO:teuthology.orchestra.run.gibba012.stderr:Error ENOENT: subvolume 'non_existent_subvolume' does not exist 2021-02-10T11:45:08.169 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:08.167 7fa2b7d21700 1 -- 172.21.2.112:0/1768304094 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa2980186c0 msgr2=0x7fa29801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:08.169 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:08.167 7fa2b7d21700 1 --2- 172.21.2.112:0/1768304094 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa2980186c0 0x7fa29801abc0 crc :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:08.169 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:08.167 7fa2b7d21700 1 -- 172.21.2.112:0/1768304094 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa2b00954c0 msgr2=0x7fa2b0096300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:08.169 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:08.167 7fa2b7d21700 1 --2- 172.21.2.112:0/1768304094 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa2b00954c0 0x7fa2b0096300 secure :-1 s=READY pgs=2703 cs=0 l=1 rev1=1 rx=0x7fa2a4006f60 tx=0x7fa2a400b970).stop 2021-02-10T11:45:08.169 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:08.167 7fa2b7d21700 1 -- 172.21.2.112:0/1768304094 shutdown_connections 2021-02-10T11:45:08.169 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:08.167 7fa2b7d21700 1 --2- 172.21.2.112:0/1768304094 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa2980186c0 0x7fa29801abc0 unknown :-1 s=CLOSED pgs=1233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:08.169 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:08.167 7fa2b7d21700 1 --2- 172.21.2.112:0/1768304094 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa2b00958c0 0x7fa2b0096920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:08.170 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:08.167 7fa2b7d21700 1 --2- 172.21.2.112:0/1768304094 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa2b00954c0 0x7fa2b0096300 unknown :-1 s=CLOSED pgs=2703 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:08.170 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:08.167 7fa2b7d21700 1 --2- 172.21.2.112:0/1768304094 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa2b0085bc0 0x7fa2b00863e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:08.170 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:08.167 7fa2b7d21700 1 -- 172.21.2.112:0/1768304094 >> 172.21.2.112:0/1768304094 conn(0x7fa2b00bef00 msgr2=0x7fa2b009d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:08.170 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:08.167 7fa2b7d21700 1 -- 172.21.2.112:0/1768304094 shutdown_connections 2021-02-10T11:45:08.170 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:08.167 7fa2b7d21700 1 -- 172.21.2.112:0/1768304094 wait complete. 2021-02-10T11:45:08.177 DEBUG:teuthology.orchestra.run:got remote process result: 2 2021-02-10T11:45:08.177 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:45:08.177 DEBUG:teuthology.orchestra.run.gibba012:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:45:08.523 DEBUG:teuthology.orchestra.run.gibba020:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:45:08.867 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:45:08.867 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:45:08.868 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:45:08.868 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:45:09.204 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:45:09.204 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:45:09.204 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba012.front.sepia.ceph.com... 2021-02-10T11:45:09.204 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:45:09.204 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:45:09.559 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba012.stderr:ceph-fuse[176523]: fuse finished with error 0 and tester_r 0 2021-02-10T11:45:09.560 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:45:09.560 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:45:09.560 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:45:09.895 INFO:teuthology.orchestra.run.gibba012.stdout:ext2/ext3 2021-02-10T11:45:09.895 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:45:09.895 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:45:09.895 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:45:10.230 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:45:10.230 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:45:10.230 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:45:10.565 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:45:10.566 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:45:10.566 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba012.front.sepia.ceph.com... 2021-02-10T11:45:10.566 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:45:10.566 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:45:10.919 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba012.stderr:ceph-fuse[177164]: fuse finished with error 0 and tester_r 0 2021-02-10T11:45:10.920 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:45:10.920 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:45:10.920 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:45:11.256 INFO:teuthology.orchestra.run.gibba012.stdout:ext2/ext3 2021-02-10T11:45:11.256 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:45:11.256 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:45:11.256 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:45:11.591 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:45:11.591 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:45:11.591 DEBUG:teuthology.orchestra.run.gibba020:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:45:11.927 INFO:teuthology.orchestra.run.gibba020.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:45:11.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:45:11.927 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:45:11.927 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:45:11.927 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:45:12.261 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:45:12.262 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:45:12.262 DEBUG:teuthology.orchestra.run.gibba020:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:45:12.596 INFO:teuthology.orchestra.run.gibba020.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:45:12.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:45:12.596 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:45:12.597 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:45:12.597 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:45:12.932 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config rm mon mon_allow_pool_delete 2021-02-10T11:45:13.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.345 7f5f78866700 1 Processor -- start 2021-02-10T11:45:13.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.345 7f5f78866700 1 -- start start 2021-02-10T11:45:13.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.346 7f5f78866700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f5f740d0340 0x7f5f740ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:13.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.346 7f5f78866700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5f740d1c70 0x7f5f740efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:13.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.346 7f5f78866700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f740f0b10 con 0x7f5f740d1870 2021-02-10T11:45:13.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.346 7f5f78866700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f740e5ed0 con 0x7f5f740d0340 2021-02-10T11:45:13.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.346 7f5f78866700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f741b0be0 con 0x7f5f740d1c70 2021-02-10T11:45:13.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.346 7f5f71e1b700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f5f740d0340 0x7f5f740ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:13.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.346 7f5f71e1b700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f5f740d0340 0x7f5f740ef640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:58492/0 (socket says 172.21.2.112:58492) 2021-02-10T11:45:13.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.346 7f5f71e1b700 1 -- 172.21.2.112:0/2755612730 learned_addr learned my addr 172.21.2.112:0/2755612730 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:45:13.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.346 7f5f71e1b700 1 -- 172.21.2.112:0/2755612730 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5f740d1c70 msgr2=0x7f5f740efc60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:45:13.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.346 7f5f71e1b700 1 --2- 172.21.2.112:0/2755612730 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5f740d1c70 0x7f5f740efc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:13.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.346 7f5f71e1b700 1 -- 172.21.2.112:0/2755612730 >> v1:172.21.2.120:6789/0 conn(0x7f5f740d1870 legacy=0x7f5f741b03f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:45:13.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.346 7f5f71e1b700 1 -- 172.21.2.112:0/2755612730 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f740f20a0 con 0x7f5f740d0340 2021-02-10T11:45:13.365 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.347 7f5f71e1b700 1 --2- 172.21.2.112:0/2755612730 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5f740d0340 0x7f5f740ef640 secure :-1 s=READY pgs=2704 cs=0 l=1 rev1=1 rx=0x7f5f68006700 tx=0x7f5f68009b80).ready entity=mon.0 client_cookie=4d532b066910194b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:13.365 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.347 7f5f7161a700 1 -- 172.21.2.112:0/2755612730 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5f6800c0c0 con 0x7f5f740d0340 2021-02-10T11:45:13.365 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.347 7f5f7161a700 1 -- 172.21.2.112:0/2755612730 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f6800c340 con 0x7f5f740d0340 2021-02-10T11:45:13.365 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.347 7f5f7161a700 1 -- 172.21.2.112:0/2755612730 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5f6800c860 con 0x7f5f740d0340 2021-02-10T11:45:13.365 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.347 7f5f78866700 1 -- 172.21.2.112:0/2755612730 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5f740d0340 msgr2=0x7f5f740ef640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:13.365 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.347 7f5f78866700 1 --2- 172.21.2.112:0/2755612730 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5f740d0340 0x7f5f740ef640 secure :-1 s=READY pgs=2704 cs=0 l=1 rev1=1 rx=0x7f5f68006700 tx=0x7f5f68009b80).stop 2021-02-10T11:45:13.365 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.348 7f5f78866700 1 -- 172.21.2.112:0/2755612730 shutdown_connections 2021-02-10T11:45:13.365 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.348 7f5f78866700 1 --2- 172.21.2.112:0/2755612730 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5f740d0340 0x7f5f740ef640 unknown :-1 s=CLOSED pgs=2704 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:13.365 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.348 7f5f78866700 1 --2- 172.21.2.112:0/2755612730 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5f740d1c70 0x7f5f740efc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:13.365 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.348 7f5f78866700 1 -- 172.21.2.112:0/2755612730 >> 172.21.2.112:0/2755612730 conn(0x7f5f740ce450 msgr2=0x7f5f740d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:13.365 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.348 7f5f78866700 1 -- 172.21.2.112:0/2755612730 shutdown_connections 2021-02-10T11:45:13.365 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.348 7f5f78866700 1 -- 172.21.2.112:0/2755612730 wait complete. 2021-02-10T11:45:13.367 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.362 7f5f78866700 1 Processor -- start 2021-02-10T11:45:13.367 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.362 7f5f78866700 1 -- start start 2021-02-10T11:45:13.367 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.363 7f5f78866700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5f741a4950 0x7f5f740f16f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:13.367 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.363 7f5f78866700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5f740f1d10 0x7f5f740f2110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:13.367 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.363 7f5f78866700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5f740f2730 0x7f5f740ef4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:13.368 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.363 7f5f78866700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f7409ebd0 con 0x7f5f740f2730 2021-02-10T11:45:13.368 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.363 7f5f78866700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f740efaf0 con 0x7f5f740f1d10 2021-02-10T11:45:13.368 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.363 7f5f78866700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f740efd10 con 0x7f5f741a4950 2021-02-10T11:45:13.368 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.363 7f5f72e1d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5f740f2730 0x7f5f740ef4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:13.368 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.363 7f5f72e1d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5f740f2730 0x7f5f740ef4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:58500/0 (socket says 172.21.2.112:58500) 2021-02-10T11:45:13.368 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.363 7f5f72e1d700 1 -- 172.21.2.112:0/2400694723 learned_addr learned my addr 172.21.2.112:0/2400694723 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:45:13.368 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.363 7f5f72e1d700 1 -- 172.21.2.112:0/2400694723 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5f741a4950 msgr2=0x7f5f740f16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:13.368 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.363 7f5f72e1d700 1 --2- 172.21.2.112:0/2400694723 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5f741a4950 0x7f5f740f16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:13.368 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.363 7f5f72e1d700 1 -- 172.21.2.112:0/2400694723 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5f740f1d10 msgr2=0x7f5f740f2110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:13.368 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.363 7f5f72e1d700 1 --2- 172.21.2.112:0/2400694723 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5f740f1d10 0x7f5f740f2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:13.368 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.363 7f5f72e1d700 1 -- 172.21.2.112:0/2400694723 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f740eff30 con 0x7f5f740f2730 2021-02-10T11:45:13.368 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.363 7f5f72e1d700 1 --2- 172.21.2.112:0/2400694723 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5f740f2730 0x7f5f740ef4d0 secure :-1 s=READY pgs=2705 cs=0 l=1 rev1=1 rx=0x7f5f6c009960 tx=0x7f5f6c00bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:13.368 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.364 7f5f62ffd700 1 -- 172.21.2.112:0/2400694723 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5f6c003030 con 0x7f5f740f2730 2021-02-10T11:45:13.369 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.364 7f5f62ffd700 1 -- 172.21.2.112:0/2400694723 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f6c001310 con 0x7f5f740f2730 2021-02-10T11:45:13.369 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.364 7f5f62ffd700 1 -- 172.21.2.112:0/2400694723 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5f6c001830 con 0x7f5f740f2730 2021-02-10T11:45:13.369 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.364 7f5f78866700 1 -- 172.21.2.112:0/2400694723 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f740eff30 con 0x7f5f740f2730 2021-02-10T11:45:13.369 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.364 7f5f78866700 1 -- 172.21.2.112:0/2400694723 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f740f04c0 con 0x7f5f740f2730 2021-02-10T11:45:13.369 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.365 7f5f62ffd700 1 -- 172.21.2.112:0/2400694723 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5f6c0014f0 con 0x7f5f740f2730 2021-02-10T11:45:13.369 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.365 7f5f62ffd700 1 --2- 172.21.2.112:0/2400694723 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5f580186c0 0x7f5f5801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:13.369 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.365 7f5f62ffd700 1 -- 172.21.2.112:0/2400694723 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f5f6c032930 con 0x7f5f740f2730 2021-02-10T11:45:13.369 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.365 7f5f7261c700 1 --2- 172.21.2.112:0/2400694723 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5f580186c0 0x7f5f5801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:13.369 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.366 7f5f7261c700 1 --2- 172.21.2.112:0/2400694723 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5f580186c0 0x7f5f5801abc0 crc :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:13.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.396 7f5f78866700 1 -- 172.21.2.112:0/2400694723 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f740f11c0 con 0x7f5f740f2730 2021-02-10T11:45:13.400 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.398 7f5f62ffd700 1 -- 172.21.2.112:0/2400694723 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5f6c001ee0 con 0x7f5f740f2730 2021-02-10T11:45:13.549 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.547 7f5f78866700 1 -- 172.21.2.112:0/2400694723 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f5f740f11c0 con 0x7f5f740f2730 2021-02-10T11:45:13.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.551 7f5f62ffd700 1 -- 172.21.2.112:0/2400694723 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v39) v1 ==== 104+0+0 (secure 0 0 0) 0x7f5f6c000f40 con 0x7f5f740f2730 2021-02-10T11:45:13.554 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.552 7f5f78866700 1 -- 172.21.2.112:0/2400694723 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5f580186c0 msgr2=0x7f5f5801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:13.554 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.552 7f5f78866700 1 --2- 172.21.2.112:0/2400694723 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5f580186c0 0x7f5f5801abc0 crc :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:13.554 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.552 7f5f78866700 1 -- 172.21.2.112:0/2400694723 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5f740f2730 msgr2=0x7f5f740ef4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:13.554 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.552 7f5f78866700 1 --2- 172.21.2.112:0/2400694723 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5f740f2730 0x7f5f740ef4d0 secure :-1 s=READY pgs=2705 cs=0 l=1 rev1=1 rx=0x7f5f6c009960 tx=0x7f5f6c00bcf0).stop 2021-02-10T11:45:13.554 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.552 7f5f78866700 1 -- 172.21.2.112:0/2400694723 shutdown_connections 2021-02-10T11:45:13.555 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.552 7f5f78866700 1 --2- 172.21.2.112:0/2400694723 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5f580186c0 0x7f5f5801abc0 unknown :-1 s=CLOSED pgs=1234 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:13.555 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.552 7f5f78866700 1 --2- 172.21.2.112:0/2400694723 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5f740f1d10 0x7f5f740f2110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:13.555 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.552 7f5f78866700 1 --2- 172.21.2.112:0/2400694723 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5f740f2730 0x7f5f740ef4d0 unknown :-1 s=CLOSED pgs=2705 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:13.555 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.552 7f5f78866700 1 --2- 172.21.2.112:0/2400694723 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5f741a4950 0x7f5f740f16f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:13.555 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.552 7f5f78866700 1 -- 172.21.2.112:0/2400694723 >> 172.21.2.112:0/2400694723 conn(0x7f5f740ba800 msgr2=0x7f5f7409f7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:13.555 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.552 7f5f78866700 1 -- 172.21.2.112:0/2400694723 shutdown_connections 2021-02-10T11:45:13.555 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.552 7f5f78866700 1 -- 172.21.2.112:0/2400694723 wait complete. 2021-02-10T11:45:13.562 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestVolumes.test_nonexistent_subvolume_rm' 2021-02-10T11:45:13.990 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.972 7fadf0e65700 1 Processor -- start 2021-02-10T11:45:13.990 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.972 7fadf0e65700 1 -- start start 2021-02-10T11:45:13.990 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.972 7fadf0e65700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fadec1b0470 0x7fadec1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:13.990 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.972 7fadf0e65700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fadec0ef610 0x7fadec0efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:13.990 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.972 7fadf0e65700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadec0f0b50 con 0x7fadec0f0030 2021-02-10T11:45:13.990 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.972 7fadf0e65700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fadec0e52e0 con 0x7fadec0ef610 2021-02-10T11:45:13.990 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.972 7fadf0e65700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadec0e5490 con 0x7fadec1b0470 2021-02-10T11:45:13.990 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.972 7fadea312700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fadec0ef610 0x7fadec0efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:13.990 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.972 7fadea312700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fadec0ef610 0x7fadec0efa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:58506/0 (socket says 172.21.2.112:58506) 2021-02-10T11:45:13.991 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.972 7fadea312700 1 -- 172.21.2.112:0/3383302731 learned_addr learned my addr 172.21.2.112:0/3383302731 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:45:13.991 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.972 7fadeab13700 1 --2- 172.21.2.112:0/3383302731 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fadec1b0470 0x7fadec1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:13.991 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.973 7fadea312700 1 -- 172.21.2.112:0/3383302731 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fadec1b0470 msgr2=0x7fadec1b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:13.991 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.973 7fadea312700 1 --2- 172.21.2.112:0/3383302731 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fadec1b0470 0x7fadec1b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:13.991 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.973 7fadea312700 1 -- 172.21.2.112:0/3383302731 >> v1:172.21.2.120:6789/0 conn(0x7fadec0f0030 legacy=0x7fadec0f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:13.991 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.973 7fadea312700 1 -- 172.21.2.112:0/3383302731 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadec0f0430 con 0x7fadec0ef610 2021-02-10T11:45:13.991 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.973 7fadea312700 1 --2- 172.21.2.112:0/3383302731 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fadec0ef610 0x7fadec0efa10 secure :-1 s=READY pgs=2706 cs=0 l=1 rev1=1 rx=0x7fade0006700 tx=0x7fade0009b00).ready entity=mon.0 client_cookie=8f9c28cbec068dd2 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:13.991 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.973 7fade9b11700 1 -- 172.21.2.112:0/3383302731 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fade000c0c0 con 0x7fadec0ef610 2021-02-10T11:45:13.991 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.973 7fade9b11700 1 -- 172.21.2.112:0/3383302731 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fade000c400 con 0x7fadec0ef610 2021-02-10T11:45:13.991 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.973 7fade9b11700 1 -- 172.21.2.112:0/3383302731 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fade000c8e0 con 0x7fadec0ef610 2021-02-10T11:45:13.991 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.973 7fadf0e65700 1 -- 172.21.2.112:0/3383302731 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fadec0ef610 msgr2=0x7fadec0efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:13.991 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.973 7fadf0e65700 1 --2- 172.21.2.112:0/3383302731 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fadec0ef610 0x7fadec0efa10 secure :-1 s=READY pgs=2706 cs=0 l=1 rev1=1 rx=0x7fade0006700 tx=0x7fade0009b00).stop 2021-02-10T11:45:13.992 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.974 7fadf0e65700 1 -- 172.21.2.112:0/3383302731 shutdown_connections 2021-02-10T11:45:13.992 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.974 7fadf0e65700 1 --2- 172.21.2.112:0/3383302731 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fadec0ef610 0x7fadec0efa10 unknown :-1 s=CLOSED pgs=2706 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:13.992 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.974 7fadf0e65700 1 --2- 172.21.2.112:0/3383302731 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fadec1b0470 0x7fadec1b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:13.992 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.974 7fadf0e65700 1 -- 172.21.2.112:0/3383302731 >> 172.21.2.112:0/3383302731 conn(0x7fadec0ce3e0 msgr2=0x7fadec0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:13.992 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.974 7fadf0e65700 1 -- 172.21.2.112:0/3383302731 shutdown_connections 2021-02-10T11:45:13.992 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.974 7fadf0e65700 1 -- 172.21.2.112:0/3383302731 wait complete. 2021-02-10T11:45:13.993 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.989 7fadf0e65700 1 Processor -- start 2021-02-10T11:45:13.993 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.989 7fadf0e65700 1 -- start start 2021-02-10T11:45:13.994 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.989 7fadf0e65700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fadec0980c0 0x7fadec0b4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:13.994 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.989 7fadf0e65700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fadec0b5380 0x7fadec090870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:13.994 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.989 7fadf0e65700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fadec090e90 0x7fadec0b65e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:13.994 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.989 7fadf0e65700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadec0918d0 con 0x7fadec090e90 2021-02-10T11:45:13.994 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.989 7fadf0e65700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadec091390 con 0x7fadec0980c0 2021-02-10T11:45:13.994 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.989 7fadf0e65700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadec0b6c00 con 0x7fadec0b5380 2021-02-10T11:45:13.994 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.989 7fadeb314700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fadec090e90 0x7fadec0b65e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:13.994 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.989 7fadeb314700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fadec090e90 0x7fadec0b65e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:58514/0 (socket says 172.21.2.112:58514) 2021-02-10T11:45:13.994 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.989 7fadeb314700 1 -- 172.21.2.112:0/3975751534 learned_addr learned my addr 172.21.2.112:0/3975751534 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:45:13.994 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.990 7fadeb314700 1 -- 172.21.2.112:0/3975751534 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fadec0b5380 msgr2=0x7fadec090870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:13.994 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.990 7fadeb314700 1 --2- 172.21.2.112:0/3975751534 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fadec0b5380 0x7fadec090870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:13.994 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.990 7fadeb314700 1 -- 172.21.2.112:0/3975751534 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fadec0980c0 msgr2=0x7fadec0b4d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:13.995 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.990 7fadeb314700 1 --2- 172.21.2.112:0/3975751534 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fadec0980c0 0x7fadec0b4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:13.995 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.990 7fadeb314700 1 -- 172.21.2.112:0/3975751534 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadec0b6e20 con 0x7fadec090e90 2021-02-10T11:45:13.995 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.990 7fadeb314700 1 --2- 172.21.2.112:0/3975751534 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fadec090e90 0x7fadec0b65e0 secure :-1 s=READY pgs=2707 cs=0 l=1 rev1=1 rx=0x7faddc006ed0 tx=0x7faddc00b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:13.995 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.990 7faddb7fe700 1 -- 172.21.2.112:0/3975751534 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faddc00bd50 con 0x7fadec090e90 2021-02-10T11:45:13.995 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.990 7faddb7fe700 1 -- 172.21.2.112:0/3975751534 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faddc006020 con 0x7fadec090e90 2021-02-10T11:45:13.995 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.990 7faddb7fe700 1 -- 172.21.2.112:0/3975751534 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faddc0054b0 con 0x7fadec090e90 2021-02-10T11:45:13.995 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.990 7fadf0e65700 1 -- 172.21.2.112:0/3975751534 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadec0b6e20 con 0x7fadec090e90 2021-02-10T11:45:13.995 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.990 7fadf0e65700 1 -- 172.21.2.112:0/3975751534 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadec0b6e20 con 0x7fadec090e90 2021-02-10T11:45:13.995 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.991 7faddb7fe700 1 -- 172.21.2.112:0/3975751534 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7faddc006200 con 0x7fadec090e90 2021-02-10T11:45:13.995 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.991 7faddb7fe700 1 --2- 172.21.2.112:0/3975751534 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fadd00186c0 0x7fadd001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:13.995 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.991 7faddb7fe700 1 -- 172.21.2.112:0/3975751534 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 5573+0+0 (secure 0 0 0) 0x7faddc0327c0 con 0x7fadec090e90 2021-02-10T11:45:13.995 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.991 7fadeab13700 1 --2- 172.21.2.112:0/3975751534 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fadd00186c0 0x7fadd001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:13.995 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:13.992 7fadeab13700 1 --2- 172.21.2.112:0/3975751534 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fadd00186c0 0x7fadd001abc0 crc :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:14.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:14.022 7fadf0e65700 1 -- 172.21.2.112:0/3975751534 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadec105a40 con 0x7fadec090e90 2021-02-10T11:45:14.026 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:14.025 7faddb7fe700 1 -- 172.21.2.112:0/3975751534 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7faddc006000 con 0x7fadec090e90 2021-02-10T11:45:14.158 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:14.156 7fadf0e65700 1 -- 172.21.2.112:0/3975751534 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_nonexistent_subvolume_rm"]} v 0) v1 -- 0x7fadec0f0b50 con 0x7fadec090e90 2021-02-10T11:45:14.609 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:14.608 7faddb7fe700 1 -- 172.21.2.112:0/3975751534 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_nonexistent_subvolume_rm"]}]=0 v743) v1 ==== 146+0+0 (secure 0 0 0) 0x7faddc006210 con 0x7fadec090e90 2021-02-10T11:45:14.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:14.626 7fadf0e65700 1 -- 172.21.2.112:0/3975751534 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fadd00186c0 msgr2=0x7fadd001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:14.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:14.626 7fadf0e65700 1 --2- 172.21.2.112:0/3975751534 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fadd00186c0 0x7fadd001abc0 crc :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:14.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:14.626 7fadf0e65700 1 -- 172.21.2.112:0/3975751534 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fadec090e90 msgr2=0x7fadec0b65e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:14.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:14.626 7fadf0e65700 1 --2- 172.21.2.112:0/3975751534 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fadec090e90 0x7fadec0b65e0 secure :-1 s=READY pgs=2707 cs=0 l=1 rev1=1 rx=0x7faddc006ed0 tx=0x7faddc00b960).stop 2021-02-10T11:45:14.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:14.626 7fadf0e65700 1 -- 172.21.2.112:0/3975751534 shutdown_connections 2021-02-10T11:45:14.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:14.626 7fadf0e65700 1 --2- 172.21.2.112:0/3975751534 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fadd00186c0 0x7fadd001abc0 unknown :-1 s=CLOSED pgs=1235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:14.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:14.626 7fadf0e65700 1 --2- 172.21.2.112:0/3975751534 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fadec090e90 0x7fadec0b65e0 unknown :-1 s=CLOSED pgs=2707 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:14.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:14.626 7fadf0e65700 1 --2- 172.21.2.112:0/3975751534 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fadec0b5380 0x7fadec090870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:14.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:14.626 7fadf0e65700 1 --2- 172.21.2.112:0/3975751534 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fadec0980c0 0x7fadec0b4d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:14.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:14.626 7fadf0e65700 1 -- 172.21.2.112:0/3975751534 >> 172.21.2.112:0/3975751534 conn(0x7fadec0c2c70 msgr2=0x7fadec0c3ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:14.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:14.626 7fadf0e65700 1 -- 172.21.2.112:0/3975751534 shutdown_connections 2021-02-10T11:45:14.629 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:14.626 7fadf0e65700 1 -- 172.21.2.112:0/3975751534 wait complete. 2021-02-10T11:45:14.636 INFO:tasks.cephfs_test_runner:test_nonexistent_subvolume_rm (tasks.cephfs.test_volumes.TestVolumes) ... ok 2021-02-10T11:45:14.636 INFO:tasks.cephfs_test_runner:Starting test: test_nonexistent_subvolume_snapshot_rm (tasks.cephfs.test_volumes.TestVolumes) 2021-02-10T11:45:14.637 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestVolumes.test_nonexistent_subvolume_snapshot_rm' 2021-02-10T11:45:15.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:15.047 7ff19306a700 1 Processor -- start 2021-02-10T11:45:15.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:15.047 7ff19306a700 1 -- start start 2021-02-10T11:45:15.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:15.047 7ff19306a700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7ff18c0d0340 0x7ff18c0ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:15.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:15.047 7ff19306a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff18c0d1c70 0x7ff18c0efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:15.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:15.047 7ff19306a700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff18c0f0b10 con 0x7ff18c0d1870 2021-02-10T11:45:15.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:15.047 7ff19306a700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff18c0e5ed0 con 0x7ff18c0d0340 2021-02-10T11:45:15.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:15.047 7ff19306a700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff18c1b0be0 con 0x7ff18c0d1c70 2021-02-10T11:45:15.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:15.047 7ff190e86700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7ff18c0d1870 0x7ff18c1b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:57980/0 (socket says 172.21.2.112:57980) 2021-02-10T11:45:15.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:15.047 7ff190e86700 1 -- 172.21.2.112:0/563875705 learned_addr learned my addr 172.21.2.112:0/563875705 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:45:15.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:15.048 7ff18bfff700 1 --2- 172.21.2.112:0/563875705 >> v2:172.21.2.120:3300/0 conn(0x7ff18c0d0340 0x7ff18c0ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:15.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:15.048 7ff191687700 1 --2- 172.21.2.112:0/563875705 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff18c0d1c70 0x7ff18c0efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:15.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:15.048 7ff18b7fe700 1 -- 172.21.2.112:0/563875705 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1800221915 0 0) 0x7ff174001180 con 0x7ff18c0d1870 2021-02-10T11:45:15.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:15.048 7ff18b7fe700 1 -- 172.21.2.112:0/563875705 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff178002fe0 con 0x7ff18c0d1870 2021-02-10T11:45:15.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:15.048 7ff18bfff700 1 -- 172.21.2.112:0/563875705 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff18c0d1c70 msgr2=0x7ff18c0efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:15.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:15.048 7ff18bfff700 1 --2- 172.21.2.112:0/563875705 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff18c0d1c70 0x7ff18c0efc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:15.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:15.048 7ff18bfff700 1 -- 172.21.2.112:0/563875705 >> v1:172.21.2.112:6789/0 conn(0x7ff18c0d1870 legacy=0x7ff18c1b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:15.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:15.048 7ff18b7fe700 1 -- 172.21.2.112:0/563875705 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3911709090 0 0) 0x7ff174001180 con 0x7ff18c0d1870 2021-02-10T11:45:15.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:15.048 7ff18bfff700 1 -- 172.21.2.112:0/563875705 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff18c0f20a0 con 0x7ff18c0d0340 2021-02-10T11:45:15.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:15.048 7ff191687700 1 --2- 172.21.2.112:0/563875705 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff18c0d1c70 0x7ff18c0efc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:45:15.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:15.049 7ff18bfff700 1 --2- 172.21.2.112:0/563875705 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff18c0d0340 0x7ff18c0ef640 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 rx=0x7ff17c0069d0 tx=0x7ff17c009e40).ready entity=mon.1 client_cookie=2681a560791ada31 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:15.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:15.049 7ff18b7fe700 1 -- 172.21.2.112:0/563875705 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff17c00b510 con 0x7ff18c0d0340 2021-02-10T11:45:15.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:15.049 7ff18b7fe700 1 -- 172.21.2.112:0/563875705 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff17c00b790 con 0x7ff18c0d0340 2021-02-10T11:45:15.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:15.049 7ff18b7fe700 1 -- 172.21.2.112:0/563875705 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff17c00bcb0 con 0x7ff18c0d0340 2021-02-10T11:45:15.068 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:15.049 7ff19306a700 1 -- 172.21.2.112:0/563875705 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff18c0d0340 msgr2=0x7ff18c0ef640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:15.068 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:15.049 7ff19306a700 1 --2- 172.21.2.112:0/563875705 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff18c0d0340 0x7ff18c0ef640 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 rx=0x7ff17c0069d0 tx=0x7ff17c009e40).stop 2021-02-10T11:45:15.068 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:15.050 7ff19306a700 1 -- 172.21.2.112:0/563875705 shutdown_connections 2021-02-10T11:45:15.068 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:15.050 7ff19306a700 1 --2- 172.21.2.112:0/563875705 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff18c0d1c70 0x7ff18c0efc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:15.068 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:15.050 7ff19306a700 1 --2- 172.21.2.112:0/563875705 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff18c0d0340 0x7ff18c0ef640 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:15.068 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:15.050 7ff19306a700 1 -- 172.21.2.112:0/563875705 >> 172.21.2.112:0/563875705 conn(0x7ff18c0ce450 msgr2=0x7ff18c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:15.068 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:15.050 7ff19306a700 1 -- 172.21.2.112:0/563875705 shutdown_connections 2021-02-10T11:45:15.068 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:15.050 7ff19306a700 1 -- 172.21.2.112:0/563875705 wait complete. 2021-02-10T11:45:15.069 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:15.064 7ff19306a700 1 Processor -- start 2021-02-10T11:45:15.069 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:15.064 7ff19306a700 1 -- start start 2021-02-10T11:45:15.069 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:15.065 7ff19306a700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff18c195650 0x7ff18c0b5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:15.069 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:15.065 7ff19306a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff18c0b5b80 0x7ff18c1a2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:15.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:15.065 7ff19306a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff18c0acfa0 0x7ff18c1a3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:15.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:15.065 7ff19306a700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff18c0b6010 con 0x7ff18c0b5b80 2021-02-10T11:45:15.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:15.065 7ff19306a700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff18c1a3a60 con 0x7ff18c195650 2021-02-10T11:45:15.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:15.065 7ff19306a700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff18c1a3c40 con 0x7ff18c0acfa0 2021-02-10T11:45:15.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:15.065 7ff18bfff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff18c0b5b80 0x7ff18c1a2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:15.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:15.065 7ff18bfff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff18c0b5b80 0x7ff18c1a2e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:58524/0 (socket says 172.21.2.112:58524) 2021-02-10T11:45:15.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:15.065 7ff18bfff700 1 -- 172.21.2.112:0/2573594421 learned_addr learned my addr 172.21.2.112:0/2573594421 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:45:15.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:15.066 7ff18bfff700 1 -- 172.21.2.112:0/2573594421 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff18c0acfa0 msgr2=0x7ff18c1a3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:15.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:15.066 7ff18bfff700 1 --2- 172.21.2.112:0/2573594421 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff18c0acfa0 0x7ff18c1a3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:15.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:15.066 7ff18bfff700 1 -- 172.21.2.112:0/2573594421 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff18c195650 msgr2=0x7ff18c0b5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:15.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:15.066 7ff18bfff700 1 --2- 172.21.2.112:0/2573594421 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff18c195650 0x7ff18c0b5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:15.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:15.066 7ff18bfff700 1 -- 172.21.2.112:0/2573594421 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff18c0bcae0 con 0x7ff18c0b5b80 2021-02-10T11:45:15.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:15.066 7ff18bfff700 1 --2- 172.21.2.112:0/2573594421 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff18c0b5b80 0x7ff18c1a2e20 secure :-1 s=READY pgs=2709 cs=0 l=1 rev1=1 rx=0x7ff17c009f40 tx=0x7ff17c00bd50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:15.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:15.066 7ff1897fa700 1 -- 172.21.2.112:0/2573594421 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff17c006ca0 con 0x7ff18c0b5b80 2021-02-10T11:45:15.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:15.066 7ff1897fa700 1 -- 172.21.2.112:0/2573594421 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff17c006240 con 0x7ff18c0b5b80 2021-02-10T11:45:15.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:15.066 7ff1897fa700 1 -- 172.21.2.112:0/2573594421 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff17c005660 con 0x7ff18c0b5b80 2021-02-10T11:45:15.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:15.066 7ff19306a700 1 -- 172.21.2.112:0/2573594421 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff18c0bcae0 con 0x7ff18c0b5b80 2021-02-10T11:45:15.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:15.066 7ff19306a700 1 -- 172.21.2.112:0/2573594421 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff18c0bd070 con 0x7ff18c0b5b80 2021-02-10T11:45:15.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:15.067 7ff1897fa700 1 -- 172.21.2.112:0/2573594421 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff17c006ca0 con 0x7ff18c0b5b80 2021-02-10T11:45:15.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:15.067 7ff1897fa700 1 --2- 172.21.2.112:0/2573594421 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff1700186c0 0x7ff17001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:15.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:15.067 7ff1897fa700 1 -- 172.21.2.112:0/2573594421 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 5573+0+0 (secure 0 0 0) 0x7ff17c0328a0 con 0x7ff18c0b5b80 2021-02-10T11:45:15.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:15.067 7ff190e86700 1 --2- 172.21.2.112:0/2573594421 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff1700186c0 0x7ff17001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:15.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:15.068 7ff190e86700 1 --2- 172.21.2.112:0/2573594421 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff1700186c0 0x7ff17001abc0 crc :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:15.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:15.097 7ff19306a700 1 -- 172.21.2.112:0/2573594421 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff18c0f0ab0 con 0x7ff18c0b5b80 2021-02-10T11:45:15.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:15.100 7ff1897fa700 1 -- 172.21.2.112:0/2573594421 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff17c000d30 con 0x7ff18c0b5b80 2021-02-10T11:45:15.233 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:15.231 7ff19306a700 1 -- 172.21.2.112:0/2573594421 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_nonexistent_subvolume_snapshot_rm"]} v 0) v1 -- 0x7ff18c0f0ab0 con 0x7ff18c0b5b80 2021-02-10T11:45:15.613 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:15.612 7ff1897fa700 1 -- 172.21.2.112:0/2573594421 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_nonexistent_subvolume_snapshot_rm"]}]=0 v744) v1 ==== 158+0+0 (secure 0 0 0) 0x7ff17c006ca0 con 0x7ff18c0b5b80 2021-02-10T11:45:15.652 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:15.650 7ff19306a700 1 -- 172.21.2.112:0/2573594421 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff1700186c0 msgr2=0x7ff17001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:15.652 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:15.650 7ff19306a700 1 --2- 172.21.2.112:0/2573594421 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff1700186c0 0x7ff17001abc0 crc :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:15.652 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:15.650 7ff19306a700 1 -- 172.21.2.112:0/2573594421 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff18c0b5b80 msgr2=0x7ff18c1a2e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:15.652 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:15.650 7ff19306a700 1 --2- 172.21.2.112:0/2573594421 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff18c0b5b80 0x7ff18c1a2e20 secure :-1 s=READY pgs=2709 cs=0 l=1 rev1=1 rx=0x7ff17c009f40 tx=0x7ff17c00bd50).stop 2021-02-10T11:45:15.652 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:15.651 7ff19306a700 1 -- 172.21.2.112:0/2573594421 shutdown_connections 2021-02-10T11:45:15.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:15.651 7ff19306a700 1 --2- 172.21.2.112:0/2573594421 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff1700186c0 0x7ff17001abc0 unknown :-1 s=CLOSED pgs=1236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:15.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:15.651 7ff19306a700 1 --2- 172.21.2.112:0/2573594421 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff18c0acfa0 0x7ff18c1a3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:15.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:15.651 7ff19306a700 1 --2- 172.21.2.112:0/2573594421 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff18c0b5b80 0x7ff18c1a2e20 unknown :-1 s=CLOSED pgs=2709 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:15.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:15.651 7ff19306a700 1 --2- 172.21.2.112:0/2573594421 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff18c195650 0x7ff18c0b5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:15.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:15.651 7ff19306a700 1 -- 172.21.2.112:0/2573594421 >> 172.21.2.112:0/2573594421 conn(0x7ff18c0b3040 msgr2=0x7ff18c0c3250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:15.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:15.651 7ff19306a700 1 -- 172.21.2.112:0/2573594421 shutdown_connections 2021-02-10T11:45:15.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:15.651 7ff19306a700 1 -- 172.21.2.112:0/2573594421 wait complete. 2021-02-10T11:45:15.660 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:45:16.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:16.076 7fa5b37dd700 1 Processor -- start 2021-02-10T11:45:16.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:16.076 7fa5b37dd700 1 -- start start 2021-02-10T11:45:16.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:16.077 7fa5b37dd700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fa5ac0d1870 0x7fa5ac1b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:16.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:16.077 7fa5b37dd700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fa5ac1b0ae0 0x7fa5ac108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:16.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:16.077 7fa5b37dd700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5ac0fe1c0 con 0x7fa5ac0d1c70 2021-02-10T11:45:16.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:16.077 7fa5b37dd700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5ac109a60 con 0x7fa5ac1b0ae0 2021-02-10T11:45:16.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:16.077 7fa5b37dd700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5ac109750 con 0x7fa5ac0d1870 2021-02-10T11:45:16.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:16.077 7fa5b0df8700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fa5ac1b0ae0 0x7fa5ac108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:16.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:16.077 7fa5b1dfa700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fa5ac0d1c70 0x7fa5ac108f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:57992/0 (socket says 172.21.2.112:57992) 2021-02-10T11:45:16.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:16.077 7fa5b1dfa700 1 -- 172.21.2.112:0/2714014384 learned_addr learned my addr 172.21.2.112:0/2714014384 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:45:16.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:16.077 7fa5b15f9700 1 --2- 172.21.2.112:0/2714014384 >> v2:172.21.2.120:3300/0 conn(0x7fa5ac0d1870 0x7fa5ac1b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:16.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:16.077 7fa5b0df8700 1 -- 172.21.2.112:0/2714014384 >> v2:172.21.2.120:3300/0 conn(0x7fa5ac0d1870 msgr2=0x7fa5ac1b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:16.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:16.077 7fa5b0df8700 1 --2- 172.21.2.112:0/2714014384 >> v2:172.21.2.120:3300/0 conn(0x7fa5ac0d1870 0x7fa5ac1b04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:16.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:16.077 7fa5b0df8700 1 -- 172.21.2.112:0/2714014384 >> v1:172.21.2.112:6789/0 conn(0x7fa5ac0d1c70 legacy=0x7fa5ac108f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:16.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:16.077 7fa5b0df8700 1 -- 172.21.2.112:0/2714014384 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5ac088070 con 0x7fa5ac1b0ae0 2021-02-10T11:45:16.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:16.077 7fa5b15f9700 1 --2- 172.21.2.112:0/2714014384 >> v2:172.21.2.120:3300/0 conn(0x7fa5ac0d1870 0x7fa5ac1b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:45:16.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:16.077 7fa5b0df8700 1 --2- 172.21.2.112:0/2714014384 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa5ac1b0ae0 0x7fa5ac108940 secure :-1 s=READY pgs=2711 cs=0 l=1 rev1=1 rx=0x7fa59c006650 tx=0x7fa59c009a80).ready entity=mon.0 client_cookie=88342e4b12c819ef server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:16.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:16.077 7fa5a3fff700 1 -- 172.21.2.112:0/2714014384 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa59c0009b0 con 0x7fa5ac1b0ae0 2021-02-10T11:45:16.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:16.077 7fa5a3fff700 1 -- 172.21.2.112:0/2714014384 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa59c00c0c0 con 0x7fa5ac1b0ae0 2021-02-10T11:45:16.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:16.077 7fa5a3fff700 1 -- 172.21.2.112:0/2714014384 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa59c00c5e0 con 0x7fa5ac1b0ae0 2021-02-10T11:45:16.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:16.077 7fa5b37dd700 1 -- 172.21.2.112:0/2714014384 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa5ac1b0ae0 msgr2=0x7fa5ac108940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:16.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:16.077 7fa5b37dd700 1 --2- 172.21.2.112:0/2714014384 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa5ac1b0ae0 0x7fa5ac108940 secure :-1 s=READY pgs=2711 cs=0 l=1 rev1=1 rx=0x7fa59c006650 tx=0x7fa59c009a80).stop 2021-02-10T11:45:16.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:16.078 7fa5b37dd700 1 -- 172.21.2.112:0/2714014384 shutdown_connections 2021-02-10T11:45:16.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:16.078 7fa5b37dd700 1 --2- 172.21.2.112:0/2714014384 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa5ac1b0ae0 0x7fa5ac108940 unknown :-1 s=CLOSED pgs=2711 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:16.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:16.078 7fa5b37dd700 1 --2- 172.21.2.112:0/2714014384 >> v2:172.21.2.120:3300/0 conn(0x7fa5ac0d1870 0x7fa5ac1b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:16.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:16.078 7fa5b37dd700 1 -- 172.21.2.112:0/2714014384 >> 172.21.2.112:0/2714014384 conn(0x7fa5ac0ce420 msgr2=0x7fa5ac0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:16.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:16.078 7fa5b37dd700 1 -- 172.21.2.112:0/2714014384 shutdown_connections 2021-02-10T11:45:16.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:16.078 7fa5b37dd700 1 -- 172.21.2.112:0/2714014384 wait complete. 2021-02-10T11:45:16.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:16.092 7fa5b37dd700 1 Processor -- start 2021-02-10T11:45:16.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:16.092 7fa5b37dd700 1 -- start start 2021-02-10T11:45:16.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:16.093 7fa5b37dd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa5ac1a7b70 0x7fa5ac1a7f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:16.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:16.093 7fa5b37dd700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa5ac0ae250 0x7fa5ac0ae650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:16.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:16.093 7fa5b37dd700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa5ac0aec70 0x7fa5ac0af070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:16.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:16.093 7fa5b37dd700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5ac0adcd0 con 0x7fa5ac1a7b70 2021-02-10T11:45:16.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:16.093 7fa5b37dd700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5ac0b7e00 con 0x7fa5ac0ae250 2021-02-10T11:45:16.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:16.093 7fa5b37dd700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5ac0ac4b0 con 0x7fa5ac0aec70 2021-02-10T11:45:16.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:16.093 7fa5b15f9700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa5ac1a7b70 0x7fa5ac1a7f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:16.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:16.093 7fa5b15f9700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa5ac1a7b70 0x7fa5ac1a7f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:58536/0 (socket says 172.21.2.112:58536) 2021-02-10T11:45:16.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:16.093 7fa5b15f9700 1 -- 172.21.2.112:0/3287030180 learned_addr learned my addr 172.21.2.112:0/3287030180 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:45:16.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:16.093 7fa5b15f9700 1 -- 172.21.2.112:0/3287030180 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa5ac0aec70 msgr2=0x7fa5ac0af070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:16.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:16.093 7fa5b15f9700 1 --2- 172.21.2.112:0/3287030180 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa5ac0aec70 0x7fa5ac0af070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:16.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:16.093 7fa5b0df8700 1 --2- 172.21.2.112:0/3287030180 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa5ac0ae250 0x7fa5ac0ae650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:16.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:16.093 7fa5b15f9700 1 -- 172.21.2.112:0/3287030180 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa5ac0ae250 msgr2=0x7fa5ac0ae650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:16.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:16.093 7fa5b15f9700 1 --2- 172.21.2.112:0/3287030180 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa5ac0ae250 0x7fa5ac0ae650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:16.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:16.094 7fa5b15f9700 1 -- 172.21.2.112:0/3287030180 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5ac0ac6d0 con 0x7fa5ac1a7b70 2021-02-10T11:45:16.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:16.094 7fa5b15f9700 1 --2- 172.21.2.112:0/3287030180 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa5ac1a7b70 0x7fa5ac1a7f70 secure :-1 s=READY pgs=2712 cs=0 l=1 rev1=1 rx=0x7fa5a800b580 tx=0x7fa5a800b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:16.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:16.094 7fa5a1ffb700 1 -- 172.21.2.112:0/3287030180 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa5a800bd10 con 0x7fa5ac1a7b70 2021-02-10T11:45:16.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:16.094 7fa5a1ffb700 1 -- 172.21.2.112:0/3287030180 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5a8000c90 con 0x7fa5ac1a7b70 2021-02-10T11:45:16.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:16.094 7fa5a1ffb700 1 -- 172.21.2.112:0/3287030180 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa5a80022b0 con 0x7fa5ac1a7b70 2021-02-10T11:45:16.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:16.094 7fa5b37dd700 1 -- 172.21.2.112:0/3287030180 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5ac0ac6d0 con 0x7fa5ac1a7b70 2021-02-10T11:45:16.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:16.094 7fa5b37dd700 1 -- 172.21.2.112:0/3287030180 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5ac0acc60 con 0x7fa5ac1a7b70 2021-02-10T11:45:16.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:16.095 7fa5a1ffb700 1 -- 172.21.2.112:0/3287030180 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa5a800a000 con 0x7fa5ac1a7b70 2021-02-10T11:45:16.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:16.095 7fa5a1ffb700 1 --2- 172.21.2.112:0/3287030180 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa58c0186c0 0x7fa58c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:16.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:16.095 7fa5a1ffb700 1 -- 172.21.2.112:0/3287030180 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fa5a8032530 con 0x7fa5ac1a7b70 2021-02-10T11:45:16.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:16.095 7fa5b0df8700 1 --2- 172.21.2.112:0/3287030180 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa58c0186c0 0x7fa58c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:16.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:16.096 7fa5b0df8700 1 --2- 172.21.2.112:0/3287030180 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa58c0186c0 0x7fa58c01abc0 crc :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:16.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:16.126 7fa5b37dd700 1 -- 172.21.2.112:0/3287030180 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5ac082a70 con 0x7fa5ac1a7b70 2021-02-10T11:45:16.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:16.129 7fa5a1ffb700 1 -- 172.21.2.112:0/3287030180 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa5a801ce40 con 0x7fa5ac1a7b70 2021-02-10T11:45:16.281 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:16.279 7fa5b37dd700 1 -- 172.21.2.112:0/3287030180 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fa5ac082a70 con 0x7fa5ac1a7b70 2021-02-10T11:45:16.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:16.283 7fa5a1ffb700 1 -- 172.21.2.112:0/3287030180 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v40)=0 v40) v1 ==== 122+0+0 (secure 0 0 0) 0x7fa5a801ce40 con 0x7fa5ac1a7b70 2021-02-10T11:45:16.290 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:16.288 7fa5b37dd700 1 -- 172.21.2.112:0/3287030180 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa58c0186c0 msgr2=0x7fa58c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:16.290 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:16.288 7fa5b37dd700 1 --2- 172.21.2.112:0/3287030180 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa58c0186c0 0x7fa58c01abc0 crc :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:16.290 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:16.288 7fa5b37dd700 1 -- 172.21.2.112:0/3287030180 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa5ac1a7b70 msgr2=0x7fa5ac1a7f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:16.291 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:16.288 7fa5b37dd700 1 --2- 172.21.2.112:0/3287030180 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa5ac1a7b70 0x7fa5ac1a7f70 secure :-1 s=READY pgs=2712 cs=0 l=1 rev1=1 rx=0x7fa5a800b580 tx=0x7fa5a800b920).stop 2021-02-10T11:45:16.291 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:16.289 7fa5b37dd700 1 -- 172.21.2.112:0/3287030180 shutdown_connections 2021-02-10T11:45:16.291 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:16.289 7fa5b37dd700 1 --2- 172.21.2.112:0/3287030180 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa58c0186c0 0x7fa58c01abc0 unknown :-1 s=CLOSED pgs=1237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:16.291 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:16.289 7fa5b37dd700 1 --2- 172.21.2.112:0/3287030180 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa5ac0ae250 0x7fa5ac0ae650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:16.291 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:16.289 7fa5b37dd700 1 --2- 172.21.2.112:0/3287030180 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa5ac0aec70 0x7fa5ac0af070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:16.291 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:16.289 7fa5b37dd700 1 --2- 172.21.2.112:0/3287030180 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa5ac1a7b70 0x7fa5ac1a7f70 unknown :-1 s=CLOSED pgs=2712 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:16.291 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:16.289 7fa5b37dd700 1 -- 172.21.2.112:0/3287030180 >> 172.21.2.112:0/3287030180 conn(0x7fa5ac0a7410 msgr2=0x7fa5ac0946e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:16.291 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:16.289 7fa5b37dd700 1 -- 172.21.2.112:0/3287030180 shutdown_connections 2021-02-10T11:45:16.291 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:16.289 7fa5b37dd700 1 -- 172.21.2.112:0/3287030180 wait complete. 2021-02-10T11:45:16.299 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:45:16.299 DEBUG:teuthology.orchestra.run.gibba012:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:45:16.645 DEBUG:teuthology.orchestra.run.gibba020:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:45:16.991 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:45:16.991 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:45:16.991 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:45:16.992 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:45:17.326 INFO:teuthology.orchestra.run.gibba012.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.0’: No such file or directory 2021-02-10T11:45:17.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:45:17.326 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2021-02-10T11:45:17.326 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:45:17.327 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:45:17.327 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:45:17.660 INFO:teuthology.orchestra.run.gibba012.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: No such file or directory 2021-02-10T11:45:17.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:45:17.661 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2021-02-10T11:45:17.661 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:45:17.661 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:45:17.661 DEBUG:teuthology.orchestra.run.gibba020:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:45:17.994 INFO:teuthology.orchestra.run.gibba020.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:45:17.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:45:17.995 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:45:17.995 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:45:17.995 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:45:17.995 DEBUG:teuthology.orchestra.run.gibba020:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:45:18.329 INFO:teuthology.orchestra.run.gibba020.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:45:18.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:45:18.330 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:45:18.330 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json-pretty 2021-02-10T11:45:18.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:18.741 7fb15553d700 1 Processor -- start 2021-02-10T11:45:18.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:18.741 7fb15553d700 1 -- start start 2021-02-10T11:45:18.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:18.741 7fb15553d700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fb1500d0340 0x7fb1500ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:18.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:18.741 7fb15553d700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1500f0b10 con 0x7fb1500d1c70 2021-02-10T11:45:18.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:18.741 7fb15553d700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1501b0be0 con 0x7fb1500d1870 2021-02-10T11:45:18.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:18.741 7fb15553d700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1500e5ed0 con 0x7fb1500d0340 2021-02-10T11:45:18.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:18.742 7fb14fac0700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fb1500d1c70 0x7fb1500efc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:58008/0 (socket says 172.21.2.112:58008) 2021-02-10T11:45:18.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:18.742 7fb14fac0700 1 -- 172.21.2.112:0/534756763 learned_addr learned my addr 172.21.2.112:0/534756763 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:45:18.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:18.742 7fb14eabe700 1 --2- 172.21.2.112:0/534756763 >> v2:172.21.2.120:3300/0 conn(0x7fb1500d0340 0x7fb1500ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:18.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:18.742 7fb14e2bd700 1 -- 172.21.2.112:0/534756763 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3504586211 0 0) 0x7fb148001500 con 0x7fb1500d1c70 2021-02-10T11:45:18.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:18.742 7fb14e2bd700 1 -- 172.21.2.112:0/534756763 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb138002fe0 con 0x7fb1500d1c70 2021-02-10T11:45:18.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:18.742 7fb14eabe700 1 -- 172.21.2.112:0/534756763 >> v1:172.21.2.120:6789/0 conn(0x7fb1500d1870 legacy=0x7fb1501b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:18.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:18.742 7fb14eabe700 1 -- 172.21.2.112:0/534756763 >> v1:172.21.2.112:6789/0 conn(0x7fb1500d1c70 legacy=0x7fb1500efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:18.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:18.742 7fb14e2bd700 1 -- 172.21.2.112:0/534756763 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2823572015 0 0) 0x7fb148001460 con 0x7fb1500d1c70 2021-02-10T11:45:18.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:18.742 7fb14eabe700 1 -- 172.21.2.112:0/534756763 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1500f0450 con 0x7fb1500d0340 2021-02-10T11:45:18.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:18.743 7fb14eabe700 1 --2- 172.21.2.112:0/534756763 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb1500d0340 0x7fb1500ef640 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 rx=0x7fb144006650 tx=0x7fb144009a80).ready entity=mon.1 client_cookie=587167ca98e6015a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:18.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:18.743 7fb14e2bd700 1 -- 172.21.2.112:0/534756763 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb14400c0c0 con 0x7fb1500d0340 2021-02-10T11:45:18.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:18.743 7fb14e2bd700 1 -- 172.21.2.112:0/534756763 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb14400c340 con 0x7fb1500d0340 2021-02-10T11:45:18.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:18.743 7fb14e2bd700 1 -- 172.21.2.112:0/534756763 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb14400c860 con 0x7fb1500d0340 2021-02-10T11:45:18.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:18.743 7fb15553d700 1 -- 172.21.2.112:0/534756763 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb1500d0340 msgr2=0x7fb1500ef640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:18.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:18.743 7fb15553d700 1 --2- 172.21.2.112:0/534756763 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb1500d0340 0x7fb1500ef640 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 rx=0x7fb144006650 tx=0x7fb144009a80).stop 2021-02-10T11:45:18.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:18.744 7fb15553d700 1 -- 172.21.2.112:0/534756763 shutdown_connections 2021-02-10T11:45:18.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:18.744 7fb15553d700 1 --2- 172.21.2.112:0/534756763 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb1500d0340 0x7fb1500ef640 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:18.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:18.744 7fb15553d700 1 -- 172.21.2.112:0/534756763 >> 172.21.2.112:0/534756763 conn(0x7fb1500ce450 msgr2=0x7fb1500d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:18.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:18.744 7fb15553d700 1 -- 172.21.2.112:0/534756763 shutdown_connections 2021-02-10T11:45:18.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:18.744 7fb15553d700 1 -- 172.21.2.112:0/534756763 wait complete. 2021-02-10T11:45:18.763 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:18.759 7fb15553d700 1 Processor -- start 2021-02-10T11:45:18.763 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:18.759 7fb15553d700 1 -- start start 2021-02-10T11:45:18.763 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:18.759 7fb15553d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb1501b7cf0 0x7fb15019cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:18.763 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:18.759 7fb15553d700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb1501b80f0 0x7fb1501b84f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:18.764 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:18.759 7fb15553d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb15019da20 0x7fb15019de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:18.764 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:18.759 7fb15553d700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1500a77c0 con 0x7fb15019da20 2021-02-10T11:45:18.764 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:18.759 7fb15553d700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb15019d4a0 con 0x7fb1501b80f0 2021-02-10T11:45:18.764 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:18.759 7fb15553d700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1500954f0 con 0x7fb1501b7cf0 2021-02-10T11:45:18.764 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:18.759 7fb14fac0700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb15019da20 0x7fb15019de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:18.764 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:18.759 7fb14fac0700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb15019da20 0x7fb15019de20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:58552/0 (socket says 172.21.2.112:58552) 2021-02-10T11:45:18.764 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:18.759 7fb14fac0700 1 -- 172.21.2.112:0/1913256502 learned_addr learned my addr 172.21.2.112:0/1913256502 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:45:18.764 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:18.759 7fb14eabe700 1 --2- 172.21.2.112:0/1913256502 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb1501b80f0 0x7fb1501b84f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:18.764 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:18.759 7fb14f2bf700 1 --2- 172.21.2.112:0/1913256502 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb1501b7cf0 0x7fb15019cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:18.764 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:18.760 7fb14fac0700 1 -- 172.21.2.112:0/1913256502 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb1501b7cf0 msgr2=0x7fb15019cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:18.764 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:18.760 7fb14fac0700 1 --2- 172.21.2.112:0/1913256502 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb1501b7cf0 0x7fb15019cdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:18.764 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:18.760 7fb14fac0700 1 -- 172.21.2.112:0/1913256502 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb1501b80f0 msgr2=0x7fb1501b84f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:18.764 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:18.760 7fb14fac0700 1 --2- 172.21.2.112:0/1913256502 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb1501b80f0 0x7fb1501b84f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:18.765 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:18.760 7fb14fac0700 1 -- 172.21.2.112:0/1913256502 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1501b8d60 con 0x7fb15019da20 2021-02-10T11:45:18.765 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:18.760 7fb14eabe700 1 --2- 172.21.2.112:0/1913256502 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb1501b80f0 0x7fb1501b84f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:45:18.765 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:18.760 7fb14f2bf700 1 --2- 172.21.2.112:0/1913256502 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb1501b7cf0 0x7fb15019cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:45:18.765 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:18.760 7fb14fac0700 1 --2- 172.21.2.112:0/1913256502 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb15019da20 0x7fb15019de20 secure :-1 s=READY pgs=2714 cs=0 l=1 rev1=1 rx=0x7fb148009980 tx=0x7fb14800bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:18.765 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:18.760 7fb137fff700 1 -- 172.21.2.112:0/1913256502 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb148006030 con 0x7fb15019da20 2021-02-10T11:45:18.765 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:18.760 7fb137fff700 1 -- 172.21.2.112:0/1913256502 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1480062b0 con 0x7fb15019da20 2021-02-10T11:45:18.765 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:18.760 7fb15553d700 1 -- 172.21.2.112:0/1913256502 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1501b8d60 con 0x7fb15019da20 2021-02-10T11:45:18.765 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:18.760 7fb15553d700 1 -- 172.21.2.112:0/1913256502 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb15019e730 con 0x7fb15019da20 2021-02-10T11:45:18.765 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:18.760 7fb137fff700 1 -- 172.21.2.112:0/1913256502 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb148004720 con 0x7fb15019da20 2021-02-10T11:45:18.765 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:18.761 7fb137fff700 1 -- 172.21.2.112:0/1913256502 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb148006490 con 0x7fb15019da20 2021-02-10T11:45:18.765 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:18.761 7fb137fff700 1 --2- 172.21.2.112:0/1913256502 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb1300186c0 0x7fb13001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:18.765 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:18.761 7fb137fff700 1 -- 172.21.2.112:0/1913256502 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fb148032960 con 0x7fb15019da20 2021-02-10T11:45:18.765 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:18.761 7fb14f2bf700 1 --2- 172.21.2.112:0/1913256502 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb1300186c0 0x7fb13001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:18.765 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:18.761 7fb14f2bf700 1 --2- 172.21.2.112:0/1913256502 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb1300186c0 0x7fb13001abc0 crc :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:18.793 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:18.791 7fb15553d700 1 -- 172.21.2.112:0/1913256502 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1500ebc00 con 0x7fb15019da20 2021-02-10T11:45:18.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:18.794 7fb137fff700 1 -- 172.21.2.112:0/1913256502 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb148000f40 con 0x7fb15019da20 2021-02-10T11:45:18.921 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:18.919 7fb15553d700 1 -- 172.21.2.112:0/1913256502 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7fb1500f0b10 con 0x7fb15019da20 2021-02-10T11:45:18.921 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:18.920 7fb137fff700 1 -- 172.21.2.112:0/1913256502 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v151) v1 ==== 81+0+30374 (secure 0 0 0) 0x7fb1480062b0 con 0x7fb15019da20 2021-02-10T11:45:18.922 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:45:18.922 INFO:teuthology.orchestra.run.gibba012.stdout:{ 2021-02-10T11:45:18.922 INFO:teuthology.orchestra.run.gibba012.stdout: "epoch": 151, 2021-02-10T11:45:18.922 INFO:teuthology.orchestra.run.gibba012.stdout: "fsid": "f14f004e-0699-4584-ad81-3754258493de", 2021-02-10T11:45:18.922 INFO:teuthology.orchestra.run.gibba012.stdout: "created": "2021-02-10 11:13:26.501424", 2021-02-10T11:45:18.923 INFO:teuthology.orchestra.run.gibba012.stdout: "modified": "2021-02-10 11:44:44.723715", 2021-02-10T11:45:18.923 INFO:teuthology.orchestra.run.gibba012.stdout: "last_up_change": "2021-02-10 11:13:52.780316", 2021-02-10T11:45:18.923 INFO:teuthology.orchestra.run.gibba012.stdout: "last_in_change": "2021-02-10 11:13:52.780316", 2021-02-10T11:45:18.923 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-02-10T11:45:18.923 INFO:teuthology.orchestra.run.gibba012.stdout: "flags_num": 5799936, 2021-02-10T11:45:18.923 INFO:teuthology.orchestra.run.gibba012.stdout: "flags_set": [ 2021-02-10T11:45:18.923 INFO:teuthology.orchestra.run.gibba012.stdout: "pglog_hardlimit", 2021-02-10T11:45:18.923 INFO:teuthology.orchestra.run.gibba012.stdout: "purged_snapdirs", 2021-02-10T11:45:18.923 INFO:teuthology.orchestra.run.gibba012.stdout: "recovery_deletes", 2021-02-10T11:45:18.923 INFO:teuthology.orchestra.run.gibba012.stdout: "sortbitwise" 2021-02-10T11:45:18.923 INFO:teuthology.orchestra.run.gibba012.stdout: ], 2021-02-10T11:45:18.923 INFO:teuthology.orchestra.run.gibba012.stdout: "crush_version": 4, 2021-02-10T11:45:18.924 INFO:teuthology.orchestra.run.gibba012.stdout: "full_ratio": 0.94999998807907104, 2021-02-10T11:45:18.924 INFO:teuthology.orchestra.run.gibba012.stdout: "backfillfull_ratio": 0.89999997615814209, 2021-02-10T11:45:18.924 INFO:teuthology.orchestra.run.gibba012.stdout: "nearfull_ratio": 0.85000002384185791, 2021-02-10T11:45:18.924 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_snapshot": "", 2021-02-10T11:45:18.924 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_max": 33, 2021-02-10T11:45:18.924 INFO:teuthology.orchestra.run.gibba012.stdout: "max_osd": 8, 2021-02-10T11:45:18.924 INFO:teuthology.orchestra.run.gibba012.stdout: "require_min_compat_client": "jewel", 2021-02-10T11:45:18.924 INFO:teuthology.orchestra.run.gibba012.stdout: "min_compat_client": "jewel", 2021-02-10T11:45:18.924 INFO:teuthology.orchestra.run.gibba012.stdout: "require_osd_release": "nautilus", 2021-02-10T11:45:18.924 INFO:teuthology.orchestra.run.gibba012.stdout: "pools": [ 2021-02-10T11:45:18.924 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:45:18.925 INFO:teuthology.orchestra.run.gibba012.stdout: "pool": 1, 2021-02-10T11:45:18.925 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_name": "rbd", 2021-02-10T11:45:18.925 INFO:teuthology.orchestra.run.gibba012.stdout: "create_time": "2021-02-10 11:13:57.508758", 2021-02-10T11:45:18.925 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:45:18.925 INFO:teuthology.orchestra.run.gibba012.stdout: "flags_names": "hashpspool", 2021-02-10T11:45:18.925 INFO:teuthology.orchestra.run.gibba012.stdout: "type": 1, 2021-02-10T11:45:18.925 INFO:teuthology.orchestra.run.gibba012.stdout: "size": 2, 2021-02-10T11:45:18.925 INFO:teuthology.orchestra.run.gibba012.stdout: "min_size": 1, 2021-02-10T11:45:18.925 INFO:teuthology.orchestra.run.gibba012.stdout: "crush_rule": 0, 2021-02-10T11:45:18.925 INFO:teuthology.orchestra.run.gibba012.stdout: "object_hash": 2, 2021-02-10T11:45:18.925 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:45:18.925 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num": 8, 2021-02-10T11:45:18.925 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num": 8, 2021-02-10T11:45:18.926 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num_target": 8, 2021-02-10T11:45:18.926 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_target": 8, 2021-02-10T11:45:18.926 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_pending": 8, 2021-02-10T11:45:18.926 INFO:teuthology.orchestra.run.gibba012.stdout: "last_pg_merge_meta": { 2021-02-10T11:45:18.926 INFO:teuthology.orchestra.run.gibba012.stdout: "source_pgid": "0.0", 2021-02-10T11:45:18.926 INFO:teuthology.orchestra.run.gibba012.stdout: "ready_epoch": 0, 2021-02-10T11:45:18.926 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_started": 0, 2021-02-10T11:45:18.926 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_clean": 0, 2021-02-10T11:45:18.926 INFO:teuthology.orchestra.run.gibba012.stdout: "source_version": "0'0", 2021-02-10T11:45:18.927 INFO:teuthology.orchestra.run.gibba012.stdout: "target_version": "0'0" 2021-02-10T11:45:18.927 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:45:18.927 INFO:teuthology.orchestra.run.gibba012.stdout: "last_change": "16", 2021-02-10T11:45:18.927 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend": "0", 2021-02-10T11:45:18.927 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:45:18.927 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:45:18.927 INFO:teuthology.orchestra.run.gibba012.stdout: "auid": 0, 2021-02-10T11:45:18.927 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:45:18.927 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_seq": 0, 2021-02-10T11:45:18.927 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_epoch": 0, 2021-02-10T11:45:18.927 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_snaps": [], 2021-02-10T11:45:18.927 INFO:teuthology.orchestra.run.gibba012.stdout: "removed_snaps": "[]", 2021-02-10T11:45:18.928 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_bytes": 0, 2021-02-10T11:45:18.928 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_objects": 0, 2021-02-10T11:45:18.928 INFO:teuthology.orchestra.run.gibba012.stdout: "tiers": [], 2021-02-10T11:45:18.928 INFO:teuthology.orchestra.run.gibba012.stdout: "tier_of": -1, 2021-02-10T11:45:18.928 INFO:teuthology.orchestra.run.gibba012.stdout: "read_tier": -1, 2021-02-10T11:45:18.928 INFO:teuthology.orchestra.run.gibba012.stdout: "write_tier": -1, 2021-02-10T11:45:18.928 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_mode": "none", 2021-02-10T11:45:18.928 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_bytes": 0, 2021-02-10T11:45:18.928 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_objects": 0, 2021-02-10T11:45:18.928 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:45:18.928 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:45:18.928 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:45:18.929 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_flush_age": 0, 2021-02-10T11:45:18.929 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_evict_age": 0, 2021-02-10T11:45:18.929 INFO:teuthology.orchestra.run.gibba012.stdout: "erasure_code_profile": "", 2021-02-10T11:45:18.929 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_params": { 2021-02-10T11:45:18.929 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "none" 2021-02-10T11:45:18.929 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:45:18.929 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_period": 0, 2021-02-10T11:45:18.929 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_count": 0, 2021-02-10T11:45:18.929 INFO:teuthology.orchestra.run.gibba012.stdout: "use_gmt_hitset": true, 2021-02-10T11:45:18.929 INFO:teuthology.orchestra.run.gibba012.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:45:18.929 INFO:teuthology.orchestra.run.gibba012.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:45:18.929 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:45:18.930 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:45:18.930 INFO:teuthology.orchestra.run.gibba012.stdout: "grade_table": [], 2021-02-10T11:45:18.930 INFO:teuthology.orchestra.run.gibba012.stdout: "stripe_width": 0, 2021-02-10T11:45:18.930 INFO:teuthology.orchestra.run.gibba012.stdout: "expected_num_objects": 0, 2021-02-10T11:45:18.930 INFO:teuthology.orchestra.run.gibba012.stdout: "fast_read": false, 2021-02-10T11:45:18.930 INFO:teuthology.orchestra.run.gibba012.stdout: "options": {}, 2021-02-10T11:45:18.930 INFO:teuthology.orchestra.run.gibba012.stdout: "application_metadata": { 2021-02-10T11:45:18.930 INFO:teuthology.orchestra.run.gibba012.stdout: "rbd": {} 2021-02-10T11:45:18.930 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:45:18.930 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:45:18.930 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:45:18.930 INFO:teuthology.orchestra.run.gibba012.stdout: "pool": 32, 2021-02-10T11:45:18.930 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_name": "cephfs_metadata", 2021-02-10T11:45:18.931 INFO:teuthology.orchestra.run.gibba012.stdout: "create_time": "2021-02-10 11:44:41.688683", 2021-02-10T11:45:18.931 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:45:18.931 INFO:teuthology.orchestra.run.gibba012.stdout: "flags_names": "hashpspool", 2021-02-10T11:45:18.931 INFO:teuthology.orchestra.run.gibba012.stdout: "type": 1, 2021-02-10T11:45:18.931 INFO:teuthology.orchestra.run.gibba012.stdout: "size": 2, 2021-02-10T11:45:18.931 INFO:teuthology.orchestra.run.gibba012.stdout: "min_size": 1, 2021-02-10T11:45:18.931 INFO:teuthology.orchestra.run.gibba012.stdout: "crush_rule": 0, 2021-02-10T11:45:18.931 INFO:teuthology.orchestra.run.gibba012.stdout: "object_hash": 2, 2021-02-10T11:45:18.931 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:45:18.931 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num": 8, 2021-02-10T11:45:18.931 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num": 8, 2021-02-10T11:45:18.931 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num_target": 8, 2021-02-10T11:45:18.931 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_target": 8, 2021-02-10T11:45:18.932 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_pending": 8, 2021-02-10T11:45:18.932 INFO:teuthology.orchestra.run.gibba012.stdout: "last_pg_merge_meta": { 2021-02-10T11:45:18.932 INFO:teuthology.orchestra.run.gibba012.stdout: "source_pgid": "0.0", 2021-02-10T11:45:18.932 INFO:teuthology.orchestra.run.gibba012.stdout: "ready_epoch": 0, 2021-02-10T11:45:18.932 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_started": 0, 2021-02-10T11:45:18.932 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_clean": 0, 2021-02-10T11:45:18.932 INFO:teuthology.orchestra.run.gibba012.stdout: "source_version": "0'0", 2021-02-10T11:45:18.932 INFO:teuthology.orchestra.run.gibba012.stdout: "target_version": "0'0" 2021-02-10T11:45:18.932 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:45:18.932 INFO:teuthology.orchestra.run.gibba012.stdout: "last_change": "150", 2021-02-10T11:45:18.932 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend": "0", 2021-02-10T11:45:18.933 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:45:18.933 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:45:18.933 INFO:teuthology.orchestra.run.gibba012.stdout: "auid": 0, 2021-02-10T11:45:18.933 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:45:18.933 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_seq": 0, 2021-02-10T11:45:18.933 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_epoch": 0, 2021-02-10T11:45:18.933 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_snaps": [], 2021-02-10T11:45:18.933 INFO:teuthology.orchestra.run.gibba012.stdout: "removed_snaps": "[]", 2021-02-10T11:45:18.933 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_bytes": 0, 2021-02-10T11:45:18.933 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_objects": 0, 2021-02-10T11:45:18.933 INFO:teuthology.orchestra.run.gibba012.stdout: "tiers": [], 2021-02-10T11:45:18.933 INFO:teuthology.orchestra.run.gibba012.stdout: "tier_of": -1, 2021-02-10T11:45:18.933 INFO:teuthology.orchestra.run.gibba012.stdout: "read_tier": -1, 2021-02-10T11:45:18.934 INFO:teuthology.orchestra.run.gibba012.stdout: "write_tier": -1, 2021-02-10T11:45:18.934 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_mode": "none", 2021-02-10T11:45:18.934 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_bytes": 0, 2021-02-10T11:45:18.934 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_objects": 0, 2021-02-10T11:45:18.934 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:45:18.934 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:45:18.934 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:45:18.934 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_flush_age": 0, 2021-02-10T11:45:18.935 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_evict_age": 0, 2021-02-10T11:45:18.935 INFO:teuthology.orchestra.run.gibba012.stdout: "erasure_code_profile": "", 2021-02-10T11:45:18.935 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_params": { 2021-02-10T11:45:18.935 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "none" 2021-02-10T11:45:18.935 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:45:18.935 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_period": 0, 2021-02-10T11:45:18.935 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_count": 0, 2021-02-10T11:45:18.935 INFO:teuthology.orchestra.run.gibba012.stdout: "use_gmt_hitset": true, 2021-02-10T11:45:18.935 INFO:teuthology.orchestra.run.gibba012.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:45:18.935 INFO:teuthology.orchestra.run.gibba012.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:45:18.935 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:45:18.935 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:45:18.935 INFO:teuthology.orchestra.run.gibba012.stdout: "grade_table": [], 2021-02-10T11:45:18.936 INFO:teuthology.orchestra.run.gibba012.stdout: "stripe_width": 0, 2021-02-10T11:45:18.936 INFO:teuthology.orchestra.run.gibba012.stdout: "expected_num_objects": 0, 2021-02-10T11:45:18.936 INFO:teuthology.orchestra.run.gibba012.stdout: "fast_read": false, 2021-02-10T11:45:18.936 INFO:teuthology.orchestra.run.gibba012.stdout: "options": { 2021-02-10T11:45:18.936 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_autoscale_bias": 4, 2021-02-10T11:45:18.936 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_min": 16, 2021-02-10T11:45:18.936 INFO:teuthology.orchestra.run.gibba012.stdout: "recovery_priority": 5 2021-02-10T11:45:18.936 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:45:18.936 INFO:teuthology.orchestra.run.gibba012.stdout: "application_metadata": { 2021-02-10T11:45:18.936 INFO:teuthology.orchestra.run.gibba012.stdout: "cephfs": { 2021-02-10T11:45:18.936 INFO:teuthology.orchestra.run.gibba012.stdout: "metadata": "cephfs" 2021-02-10T11:45:18.936 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:45:18.936 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:45:18.937 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:45:18.937 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:45:18.937 INFO:teuthology.orchestra.run.gibba012.stdout: "pool": 33, 2021-02-10T11:45:18.937 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_name": "cephfs_data", 2021-02-10T11:45:18.937 INFO:teuthology.orchestra.run.gibba012.stdout: "create_time": "2021-02-10 11:44:42.764257", 2021-02-10T11:45:18.937 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:45:18.937 INFO:teuthology.orchestra.run.gibba012.stdout: "flags_names": "hashpspool", 2021-02-10T11:45:18.937 INFO:teuthology.orchestra.run.gibba012.stdout: "type": 1, 2021-02-10T11:45:18.937 INFO:teuthology.orchestra.run.gibba012.stdout: "size": 2, 2021-02-10T11:45:18.937 INFO:teuthology.orchestra.run.gibba012.stdout: "min_size": 1, 2021-02-10T11:45:18.937 INFO:teuthology.orchestra.run.gibba012.stdout: "crush_rule": 0, 2021-02-10T11:45:18.937 INFO:teuthology.orchestra.run.gibba012.stdout: "object_hash": 2, 2021-02-10T11:45:18.937 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:45:18.937 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num": 8, 2021-02-10T11:45:18.938 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num": 8, 2021-02-10T11:45:18.938 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num_target": 8, 2021-02-10T11:45:18.938 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_target": 8, 2021-02-10T11:45:18.938 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_pending": 8, 2021-02-10T11:45:18.938 INFO:teuthology.orchestra.run.gibba012.stdout: "last_pg_merge_meta": { 2021-02-10T11:45:18.938 INFO:teuthology.orchestra.run.gibba012.stdout: "source_pgid": "0.0", 2021-02-10T11:45:18.938 INFO:teuthology.orchestra.run.gibba012.stdout: "ready_epoch": 0, 2021-02-10T11:45:18.938 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_started": 0, 2021-02-10T11:45:18.938 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_clean": 0, 2021-02-10T11:45:18.938 INFO:teuthology.orchestra.run.gibba012.stdout: "source_version": "0'0", 2021-02-10T11:45:18.938 INFO:teuthology.orchestra.run.gibba012.stdout: "target_version": "0'0" 2021-02-10T11:45:18.938 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:45:18.939 INFO:teuthology.orchestra.run.gibba012.stdout: "last_change": "151", 2021-02-10T11:45:18.939 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend": "0", 2021-02-10T11:45:18.939 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:45:18.939 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:45:18.939 INFO:teuthology.orchestra.run.gibba012.stdout: "auid": 0, 2021-02-10T11:45:18.939 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:45:18.939 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_seq": 0, 2021-02-10T11:45:18.939 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_epoch": 0, 2021-02-10T11:45:18.939 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_snaps": [], 2021-02-10T11:45:18.939 INFO:teuthology.orchestra.run.gibba012.stdout: "removed_snaps": "[]", 2021-02-10T11:45:18.939 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_bytes": 0, 2021-02-10T11:45:18.939 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_objects": 0, 2021-02-10T11:45:18.939 INFO:teuthology.orchestra.run.gibba012.stdout: "tiers": [], 2021-02-10T11:45:18.940 INFO:teuthology.orchestra.run.gibba012.stdout: "tier_of": -1, 2021-02-10T11:45:18.940 INFO:teuthology.orchestra.run.gibba012.stdout: "read_tier": -1, 2021-02-10T11:45:18.940 INFO:teuthology.orchestra.run.gibba012.stdout: "write_tier": -1, 2021-02-10T11:45:18.940 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_mode": "none", 2021-02-10T11:45:18.940 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_bytes": 0, 2021-02-10T11:45:18.940 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_objects": 0, 2021-02-10T11:45:18.940 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:45:18.940 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:45:18.940 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:45:18.940 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_flush_age": 0, 2021-02-10T11:45:18.940 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_evict_age": 0, 2021-02-10T11:45:18.940 INFO:teuthology.orchestra.run.gibba012.stdout: "erasure_code_profile": "", 2021-02-10T11:45:18.940 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_params": { 2021-02-10T11:45:18.941 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "none" 2021-02-10T11:45:18.941 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:45:18.941 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_period": 0, 2021-02-10T11:45:18.941 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_count": 0, 2021-02-10T11:45:18.941 INFO:teuthology.orchestra.run.gibba012.stdout: "use_gmt_hitset": true, 2021-02-10T11:45:18.941 INFO:teuthology.orchestra.run.gibba012.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:45:18.941 INFO:teuthology.orchestra.run.gibba012.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:45:18.941 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:45:18.941 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:45:18.941 INFO:teuthology.orchestra.run.gibba012.stdout: "grade_table": [], 2021-02-10T11:45:18.941 INFO:teuthology.orchestra.run.gibba012.stdout: "stripe_width": 0, 2021-02-10T11:45:18.941 INFO:teuthology.orchestra.run.gibba012.stdout: "expected_num_objects": 0, 2021-02-10T11:45:18.942 INFO:teuthology.orchestra.run.gibba012.stdout: "fast_read": false, 2021-02-10T11:45:18.942 INFO:teuthology.orchestra.run.gibba012.stdout: "options": {}, 2021-02-10T11:45:18.942 INFO:teuthology.orchestra.run.gibba012.stdout: "application_metadata": { 2021-02-10T11:45:18.942 INFO:teuthology.orchestra.run.gibba012.stdout: "cephfs": { 2021-02-10T11:45:18.942 INFO:teuthology.orchestra.run.gibba012.stdout: "data": "cephfs" 2021-02-10T11:45:18.942 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:45:18.942 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:45:18.942 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:45:18.943 INFO:teuthology.orchestra.run.gibba012.stdout: ], 2021-02-10T11:45:18.943 INFO:teuthology.orchestra.run.gibba012.stdout: "osds": [ 2021-02-10T11:45:18.943 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:45:18.943 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 0, 2021-02-10T11:45:18.943 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "95444b8f-6370-4a1d-ae5f-df62651b77da", 2021-02-10T11:45:18.943 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:45:18.943 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:45:18.943 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:45:18.943 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:45:18.943 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:45:18.943 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:45:18.944 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:45:18.944 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 148, 2021-02-10T11:45:18.944 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:45:18.944 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:45:18.944 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:45:18.944 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:45:18.944 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:45:18.944 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:45:18.944 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6817", 2021-02-10T11:45:18.944 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:45:18.944 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:45:18.944 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:45:18.945 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:45:18.945 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6818", 2021-02-10T11:45:18.945 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:45:18.945 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:45:18.945 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:45:18.945 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:45:18.945 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:45:18.945 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:45:18.945 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:45:18.946 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:45:18.946 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6820", 2021-02-10T11:45:18.946 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:45:18.946 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:45:18.946 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:45:18.946 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:45:18.946 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6822", 2021-02-10T11:45:18.946 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:45:18.946 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:45:18.946 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:45:18.946 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:45:18.946 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:45:18.947 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:45:18.947 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:45:18.947 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:45:18.947 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6828", 2021-02-10T11:45:18.947 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:45:18.947 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:45:18.947 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:45:18.947 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:45:18.947 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6830", 2021-02-10T11:45:18.947 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:45:18.947 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:45:18.947 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:45:18.947 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:45:18.947 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:45:18.948 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:45:18.948 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:45:18.948 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:45:18.948 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6824", 2021-02-10T11:45:18.948 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:45:18.948 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:45:18.948 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:45:18.948 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:45:18.948 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6826", 2021-02-10T11:45:18.948 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:45:18.948 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:45:18.948 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:45:18.948 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:45:18.949 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.112:6818/36030", 2021-02-10T11:45:18.949 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.112:6822/36030", 2021-02-10T11:45:18.949 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.112:6830/36030", 2021-02-10T11:45:18.949 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.112:6826/36030", 2021-02-10T11:45:18.949 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:45:18.949 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:45:18.949 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:45:18.949 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:45:18.949 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:45:18.949 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:45:18.949 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 1, 2021-02-10T11:45:18.950 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "a23e217e-0d8e-486e-be07-712adfabd262", 2021-02-10T11:45:18.950 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:45:18.950 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:45:18.950 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:45:18.950 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:45:18.950 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:45:18.950 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:45:18.950 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:45:18.950 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 141, 2021-02-10T11:45:18.951 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:45:18.951 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:45:18.951 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:45:18.951 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:45:18.951 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:45:18.951 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:45:18.951 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6800", 2021-02-10T11:45:18.951 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:45:18.951 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:45:18.951 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:45:18.951 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:45:18.951 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6802", 2021-02-10T11:45:18.952 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:45:18.952 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:45:18.952 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:45:18.952 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:45:18.952 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:45:18.952 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:45:18.952 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:45:18.952 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:45:18.952 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6804", 2021-02-10T11:45:18.952 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:45:18.953 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:45:18.953 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:45:18.953 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:45:18.953 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6806", 2021-02-10T11:45:18.953 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:45:18.953 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:45:18.953 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:45:18.953 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:45:18.953 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:45:18.953 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:45:18.953 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:45:18.953 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:45:18.953 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6812", 2021-02-10T11:45:18.954 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:45:18.954 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:45:18.954 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:45:18.954 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:45:18.954 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6814", 2021-02-10T11:45:18.954 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:45:18.954 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:45:18.954 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:45:18.954 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:45:18.954 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:45:18.954 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:45:18.954 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:45:18.954 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:45:18.955 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6808", 2021-02-10T11:45:18.955 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:45:18.955 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:45:18.955 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:45:18.955 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:45:18.955 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6810", 2021-02-10T11:45:18.955 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:45:18.955 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:45:18.955 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:45:18.955 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:45:18.956 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.112:6802/36029", 2021-02-10T11:45:18.956 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.112:6806/36029", 2021-02-10T11:45:18.956 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.112:6814/36029", 2021-02-10T11:45:18.956 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.112:6810/36029", 2021-02-10T11:45:18.956 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:45:18.956 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:45:18.956 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:45:18.956 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:45:18.956 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:45:18.956 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:45:18.956 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 2, 2021-02-10T11:45:18.956 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "026afa8a-c0f1-4b35-9a76-53c00fe0c2fc", 2021-02-10T11:45:18.956 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:45:18.957 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:45:18.957 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:45:18.957 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:45:18.957 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:45:18.957 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:45:18.957 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:45:18.957 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 149, 2021-02-10T11:45:18.957 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:45:18.957 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:45:18.957 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:45:18.958 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:45:18.958 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:45:18.958 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:45:18.958 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6801", 2021-02-10T11:45:18.958 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:45:18.958 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:45:18.958 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:45:18.958 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:45:18.958 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6803", 2021-02-10T11:45:18.958 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:45:18.959 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:45:18.959 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:45:18.959 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:45:18.959 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:45:18.959 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:45:18.959 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:45:18.959 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:45:18.959 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6805", 2021-02-10T11:45:18.959 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:45:18.959 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:45:18.959 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:45:18.959 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:45:18.959 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6807", 2021-02-10T11:45:18.960 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:45:18.960 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:45:18.960 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:45:18.960 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:45:18.960 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:45:18.960 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:45:18.960 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:45:18.960 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:45:18.960 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6813", 2021-02-10T11:45:18.960 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:45:18.960 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:45:18.960 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:45:18.960 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:45:18.960 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6815", 2021-02-10T11:45:18.961 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:45:18.961 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:45:18.961 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:45:18.961 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:45:18.961 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:45:18.961 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:45:18.961 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:45:18.961 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:45:18.961 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6809", 2021-02-10T11:45:18.961 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:45:18.961 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:45:18.961 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:45:18.961 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:45:18.962 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6811", 2021-02-10T11:45:18.962 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:45:18.962 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:45:18.962 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:45:18.962 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:45:18.962 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.112:6803/36031", 2021-02-10T11:45:18.962 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.112:6807/36031", 2021-02-10T11:45:18.962 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.112:6815/36031", 2021-02-10T11:45:18.962 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.112:6811/36031", 2021-02-10T11:45:18.962 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:45:18.962 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:45:18.962 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:45:18.962 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:45:18.963 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:45:18.963 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:45:18.963 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 3, 2021-02-10T11:45:18.963 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "58795682-e00d-4173-bb84-f7075ca89a11", 2021-02-10T11:45:18.963 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:45:18.963 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:45:18.963 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:45:18.963 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:45:18.963 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:45:18.963 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:45:18.963 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:45:18.963 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 149, 2021-02-10T11:45:18.963 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:45:18.964 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:45:18.964 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:45:18.964 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:45:18.964 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:45:18.964 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:45:18.964 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6816", 2021-02-10T11:45:18.964 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:45:18.964 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:45:18.964 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:45:18.964 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:45:18.964 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6819", 2021-02-10T11:45:18.964 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:45:18.964 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:45:18.964 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:45:18.965 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:45:18.965 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:45:18.965 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:45:18.965 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:45:18.965 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:45:18.965 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6821", 2021-02-10T11:45:18.965 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:45:18.965 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:45:18.965 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:45:18.966 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:45:18.966 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6823", 2021-02-10T11:45:18.966 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:45:18.966 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:45:18.966 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:45:18.966 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:45:18.966 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:45:18.966 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:45:18.966 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:45:18.966 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:45:18.966 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6829", 2021-02-10T11:45:18.966 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:45:18.967 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:45:18.967 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:45:18.967 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:45:18.967 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6831", 2021-02-10T11:45:18.967 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:45:18.967 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:45:18.967 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:45:18.967 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:45:18.967 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:45:18.967 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:45:18.967 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:45:18.967 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:45:18.967 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6825", 2021-02-10T11:45:18.968 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:45:18.968 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:45:18.968 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:45:18.968 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:45:18.968 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6827", 2021-02-10T11:45:18.968 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:45:18.968 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:45:18.968 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:45:18.968 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:45:18.968 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.112:6819/36028", 2021-02-10T11:45:18.968 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.112:6823/36028", 2021-02-10T11:45:18.968 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.112:6831/36028", 2021-02-10T11:45:18.969 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.112:6827/36028", 2021-02-10T11:45:18.969 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:45:18.969 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:45:18.969 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:45:18.969 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:45:18.969 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:45:18.969 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:45:18.969 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 4, 2021-02-10T11:45:18.969 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "6763f8d6-53dc-4e45-bdf6-a6eabcf81e39", 2021-02-10T11:45:18.969 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:45:18.969 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:45:18.969 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:45:18.969 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:45:18.970 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:45:18.970 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:45:18.970 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:45:18.970 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 141, 2021-02-10T11:45:18.970 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:45:18.970 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:45:18.970 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:45:18.970 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:45:18.970 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:45:18.970 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:45:18.970 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6810", 2021-02-10T11:45:18.970 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:45:18.970 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:45:18.971 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:45:18.971 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:45:18.971 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6813", 2021-02-10T11:45:18.971 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:45:18.971 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:45:18.971 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:45:18.971 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:45:18.971 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:45:18.971 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:45:18.971 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:45:18.971 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:45:18.971 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6816", 2021-02-10T11:45:18.971 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:45:18.971 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:45:18.972 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:45:18.972 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:45:18.972 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6819", 2021-02-10T11:45:18.972 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:45:18.972 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:45:18.972 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:45:18.972 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:45:18.972 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:45:18.972 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:45:18.972 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:45:18.972 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:45:18.972 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6825", 2021-02-10T11:45:18.972 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:45:18.973 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:45:18.973 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:45:18.973 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:45:18.973 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6827", 2021-02-10T11:45:18.973 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:45:18.973 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:45:18.973 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:45:18.973 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:45:18.973 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:45:18.974 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:45:18.974 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:45:18.974 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:45:18.974 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6821", 2021-02-10T11:45:18.974 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:45:18.974 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:45:18.974 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:45:18.974 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:45:18.974 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6823", 2021-02-10T11:45:18.974 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:45:18.974 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:45:18.975 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:45:18.975 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:45:18.975 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.120:6813/36388", 2021-02-10T11:45:18.975 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.120:6819/36388", 2021-02-10T11:45:18.975 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.120:6827/36388", 2021-02-10T11:45:18.975 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.120:6823/36388", 2021-02-10T11:45:18.975 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:45:18.975 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:45:18.975 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:45:18.975 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:45:18.975 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:45:18.975 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:45:18.976 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 5, 2021-02-10T11:45:18.976 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "5d0e4dd6-0cb0-4bf8-8314-626acaa1b544", 2021-02-10T11:45:18.976 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:45:18.976 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:45:18.976 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:45:18.976 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:45:18.976 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:45:18.976 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:45:18.976 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:45:18.976 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 149, 2021-02-10T11:45:18.976 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:45:18.976 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:45:18.976 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:45:18.977 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:45:18.977 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:45:18.977 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:45:18.977 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6804", 2021-02-10T11:45:18.977 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:45:18.977 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:45:18.977 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:45:18.977 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:45:18.977 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6806", 2021-02-10T11:45:18.977 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:45:18.977 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:45:18.977 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:45:18.977 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:45:18.978 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:45:18.978 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:45:18.978 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:45:18.978 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:45:18.978 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6809", 2021-02-10T11:45:18.978 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:45:18.978 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:45:18.978 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:45:18.978 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:45:18.978 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6812", 2021-02-10T11:45:18.978 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:45:18.978 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:45:18.978 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:45:18.979 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:45:18.979 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:45:18.979 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:45:18.979 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:45:18.979 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:45:18.979 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6820", 2021-02-10T11:45:18.979 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:45:18.979 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:45:18.979 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:45:18.979 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:45:18.979 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6822", 2021-02-10T11:45:18.979 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:45:18.979 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:45:18.980 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:45:18.980 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:45:18.980 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:45:18.980 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:45:18.980 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:45:18.980 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:45:18.980 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6815", 2021-02-10T11:45:18.980 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:45:18.980 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:45:18.980 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:45:18.980 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:45:18.980 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6818", 2021-02-10T11:45:18.980 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:45:18.981 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:45:18.981 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:45:18.981 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:45:18.981 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.120:6806/36387", 2021-02-10T11:45:18.981 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.120:6812/36387", 2021-02-10T11:45:18.981 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.120:6822/36387", 2021-02-10T11:45:18.981 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.120:6818/36387", 2021-02-10T11:45:18.981 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:45:18.982 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:45:18.982 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:45:18.982 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:45:18.982 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:45:18.982 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:45:18.982 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 6, 2021-02-10T11:45:18.982 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "b010e187-4541-4b00-80dd-f948f6ab8f15", 2021-02-10T11:45:18.982 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:45:18.982 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:45:18.982 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:45:18.982 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:45:18.982 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:45:18.983 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:45:18.983 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:45:18.983 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 148, 2021-02-10T11:45:18.983 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:45:18.983 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:45:18.983 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:45:18.983 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:45:18.983 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:45:18.983 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:45:18.983 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6824", 2021-02-10T11:45:18.983 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:45:18.983 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:45:18.983 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:45:18.983 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:45:18.984 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6826", 2021-02-10T11:45:18.984 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:45:18.984 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:45:18.984 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:45:18.984 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:45:18.984 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:45:18.984 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:45:18.984 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:45:18.984 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:45:18.984 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6828", 2021-02-10T11:45:18.984 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:45:18.984 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:45:18.984 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:45:18.985 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:45:18.985 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6829", 2021-02-10T11:45:18.985 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:45:18.985 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:45:18.985 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:45:18.985 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:45:18.985 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:45:18.985 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:45:18.985 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:45:18.985 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:45:18.986 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6832", 2021-02-10T11:45:18.986 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:45:18.986 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:45:18.986 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:45:18.986 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:45:18.986 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6833", 2021-02-10T11:45:18.986 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:45:18.986 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:45:18.986 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:45:18.986 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:45:18.986 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:45:18.986 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:45:18.987 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:45:18.987 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:45:18.987 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6830", 2021-02-10T11:45:18.987 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:45:18.987 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:45:18.987 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:45:18.987 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:45:18.987 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6831", 2021-02-10T11:45:18.987 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:45:18.987 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:45:18.987 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:45:18.988 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:45:18.988 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.120:6826/36385", 2021-02-10T11:45:18.988 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.120:6829/36385", 2021-02-10T11:45:18.988 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.120:6833/36385", 2021-02-10T11:45:18.988 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.120:6831/36385", 2021-02-10T11:45:18.988 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:45:18.988 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:45:18.988 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:45:18.988 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:45:18.988 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:45:18.988 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:45:18.989 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 7, 2021-02-10T11:45:18.989 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "671e2772-33b7-497e-8e44-2178bd850bcf", 2021-02-10T11:45:18.989 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:45:18.989 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:45:18.989 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:45:18.989 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:45:18.989 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:45:18.989 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:45:18.989 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:45:18.989 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 140, 2021-02-10T11:45:18.990 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:45:18.990 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:45:18.990 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:45:18.990 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:45:18.990 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:45:18.990 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:45:18.990 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6802", 2021-02-10T11:45:18.990 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:45:18.990 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:45:18.990 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:45:18.990 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:45:18.990 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6803", 2021-02-10T11:45:18.991 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:45:18.991 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:45:18.991 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:45:18.991 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:45:18.991 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:45:18.991 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:45:18.991 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:45:18.991 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:45:18.991 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6805", 2021-02-10T11:45:18.991 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:45:18.991 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:45:18.991 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:45:18.991 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:45:18.992 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6807", 2021-02-10T11:45:18.992 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:45:18.992 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:45:18.992 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:45:18.992 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:45:18.992 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:45:18.992 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:45:18.992 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:45:18.992 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:45:18.992 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6814", 2021-02-10T11:45:18.992 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:45:18.992 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:45:18.992 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:45:18.993 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:45:18.993 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6817", 2021-02-10T11:45:18.993 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:45:18.993 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:45:18.993 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:45:18.993 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:45:18.993 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:45:18.993 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:45:18.993 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:45:18.993 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:45:18.993 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6808", 2021-02-10T11:45:18.993 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:45:18.993 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:45:18.993 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:45:18.994 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:45:18.994 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6811", 2021-02-10T11:45:18.994 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:45:18.994 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:45:18.994 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:45:18.994 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:45:18.994 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.120:6803/36386", 2021-02-10T11:45:18.994 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.120:6807/36386", 2021-02-10T11:45:18.994 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.120:6817/36386", 2021-02-10T11:45:18.994 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.120:6811/36386", 2021-02-10T11:45:18.995 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:45:18.995 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:45:18.995 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:45:18.995 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:45:18.995 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:45:18.995 INFO:teuthology.orchestra.run.gibba012.stdout: ], 2021-02-10T11:45:18.995 INFO:teuthology.orchestra.run.gibba012.stdout: "osd_xinfo": [ 2021-02-10T11:45:18.995 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:45:18.995 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 0, 2021-02-10T11:45:18.995 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:45:18.995 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:45:18.995 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:45:18.995 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:45:18.996 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:45:18.996 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:45:18.996 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:45:18.996 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 1, 2021-02-10T11:45:18.996 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:45:18.996 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:45:18.996 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:45:18.996 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:45:18.996 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:45:18.996 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:45:18.997 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:45:18.997 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 2, 2021-02-10T11:45:18.997 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:45:18.997 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:45:18.997 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:45:18.997 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:45:18.997 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:45:18.997 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:45:18.997 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:45:18.997 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 3, 2021-02-10T11:45:18.997 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:45:18.997 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:45:18.998 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:45:18.998 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:45:18.998 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:45:18.998 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:45:18.998 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:45:18.998 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 4, 2021-02-10T11:45:18.998 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:45:18.998 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:45:18.998 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:45:18.998 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:45:18.998 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:45:18.998 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:45:18.998 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:45:18.999 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 5, 2021-02-10T11:45:18.999 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:45:18.999 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:45:18.999 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:45:18.999 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:45:18.999 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:45:18.999 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:45:18.999 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:45:18.999 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 6, 2021-02-10T11:45:18.999 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:45:18.999 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:45:18.999 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:45:18.999 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:45:19.000 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:45:19.000 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:45:19.000 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:45:19.000 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 7, 2021-02-10T11:45:19.000 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:45:19.000 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:45:19.000 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:45:19.000 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:45:19.000 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:45:19.000 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:45:19.000 INFO:teuthology.orchestra.run.gibba012.stdout: ], 2021-02-10T11:45:19.000 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_upmap": [], 2021-02-10T11:45:19.000 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_upmap_items": [], 2021-02-10T11:45:19.000 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_temp": [], 2021-02-10T11:45:19.001 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_temp": [], 2021-02-10T11:45:19.001 INFO:teuthology.orchestra.run.gibba012.stdout: "blacklist": {}, 2021-02-10T11:45:19.001 INFO:teuthology.orchestra.run.gibba012.stdout: "erasure_code_profiles": { 2021-02-10T11:45:19.001 INFO:teuthology.orchestra.run.gibba012.stdout: "default": { 2021-02-10T11:45:19.001 INFO:teuthology.orchestra.run.gibba012.stdout: "crush-failure-domain": "osd", 2021-02-10T11:45:19.001 INFO:teuthology.orchestra.run.gibba012.stdout: "k": "2", 2021-02-10T11:45:19.001 INFO:teuthology.orchestra.run.gibba012.stdout: "m": "1", 2021-02-10T11:45:19.001 INFO:teuthology.orchestra.run.gibba012.stdout: "plugin": "jerasure", 2021-02-10T11:45:19.001 INFO:teuthology.orchestra.run.gibba012.stdout: "ruleset-failure-domain": "osd", 2021-02-10T11:45:19.001 INFO:teuthology.orchestra.run.gibba012.stdout: "technique": "reed_sol_van" 2021-02-10T11:45:19.001 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:45:19.001 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:45:19.002 INFO:teuthology.orchestra.run.gibba012.stdout: "removed_snaps_queue": [], 2021-02-10T11:45:19.002 INFO:teuthology.orchestra.run.gibba012.stdout: "new_removed_snaps": [], 2021-02-10T11:45:19.002 INFO:teuthology.orchestra.run.gibba012.stdout: "new_purged_snaps": [], 2021-02-10T11:45:19.002 INFO:teuthology.orchestra.run.gibba012.stdout: "crush_node_flags": {}, 2021-02-10T11:45:19.002 INFO:teuthology.orchestra.run.gibba012.stdout: "device_class_flags": {} 2021-02-10T11:45:19.002 INFO:teuthology.orchestra.run.gibba012.stdout:} 2021-02-10T11:45:19.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:18.921 7fb15553d700 1 -- 172.21.2.112:0/1913256502 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb1300186c0 msgr2=0x7fb13001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:19.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:18.921 7fb15553d700 1 --2- 172.21.2.112:0/1913256502 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb1300186c0 0x7fb13001abc0 crc :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:19.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:18.921 7fb15553d700 1 -- 172.21.2.112:0/1913256502 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb15019da20 msgr2=0x7fb15019de20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:19.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:18.921 7fb15553d700 1 --2- 172.21.2.112:0/1913256502 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb15019da20 0x7fb15019de20 secure :-1 s=READY pgs=2714 cs=0 l=1 rev1=1 rx=0x7fb148009980 tx=0x7fb14800bcd0).stop 2021-02-10T11:45:19.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:18.922 7fb15553d700 1 -- 172.21.2.112:0/1913256502 shutdown_connections 2021-02-10T11:45:19.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:18.922 7fb15553d700 1 --2- 172.21.2.112:0/1913256502 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb1300186c0 0x7fb13001abc0 unknown :-1 s=CLOSED pgs=1238 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:19.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:18.922 7fb15553d700 1 --2- 172.21.2.112:0/1913256502 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb1501b80f0 0x7fb1501b84f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:19.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:18.922 7fb15553d700 1 --2- 172.21.2.112:0/1913256502 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb15019da20 0x7fb15019de20 unknown :-1 s=CLOSED pgs=2714 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:19.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:18.922 7fb15553d700 1 --2- 172.21.2.112:0/1913256502 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb1501b7cf0 0x7fb15019cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:19.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:18.922 7fb15553d700 1 -- 172.21.2.112:0/1913256502 >> 172.21.2.112:0/1913256502 conn(0x7fb1500ed810 msgr2=0x7fb150094980 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:19.005 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:18.922 7fb15553d700 1 -- 172.21.2.112:0/1913256502 shutdown_connections 2021-02-10T11:45:19.005 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:18.922 7fb15553d700 1 -- 172.21.2.112:0/1913256502 wait complete. 2021-02-10T11:45:19.005 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:45:19.434 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:19.415 7fe708b6b700 1 Processor -- start 2021-02-10T11:45:19.434 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:19.415 7fe708b6b700 1 -- start start 2021-02-10T11:45:19.435 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:19.416 7fe708b6b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe7040d0340 0x7fe7040ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:19.435 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:19.416 7fe708b6b700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fe7040d1c70 0x7fe7040efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:19.435 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:19.416 7fe708b6b700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7040f0b10 con 0x7fe7040d1870 2021-02-10T11:45:19.435 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:19.416 7fe708b6b700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7040e5ed0 con 0x7fe7040d1c70 2021-02-10T11:45:19.435 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:19.416 7fe708b6b700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7041b0be0 con 0x7fe7040d0340 2021-02-10T11:45:19.435 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:19.416 7fe70305f700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fe7040d1c70 0x7fe7040efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:19.435 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:19.416 7fe70305f700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fe7040d1c70 0x7fe7040efc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3300/0 says I am v2:172.21.2.112:60840/0 (socket says 172.21.2.112:60840) 2021-02-10T11:45:19.435 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:19.416 7fe70305f700 1 -- 172.21.2.112:0/634166301 learned_addr learned my addr 172.21.2.112:0/634166301 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:45:19.435 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:19.416 7fe70205d700 1 --2- 172.21.2.112:0/634166301 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe7040d0340 0x7fe7040ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:19.435 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:19.417 7fe70305f700 1 -- 172.21.2.112:0/634166301 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe7040d0340 msgr2=0x7fe7040ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:19.435 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:19.417 7fe70305f700 1 --2- 172.21.2.112:0/634166301 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe7040d0340 0x7fe7040ef640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:19.436 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:19.417 7fe70305f700 1 -- 172.21.2.112:0/634166301 >> v1:172.21.2.120:6789/0 conn(0x7fe7040d1870 legacy=0x7fe7041b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:19.436 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:19.417 7fe70305f700 1 -- 172.21.2.112:0/634166301 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7040f20a0 con 0x7fe7040d1c70 2021-02-10T11:45:19.436 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:19.417 7fe70205d700 1 --2- 172.21.2.112:0/634166301 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe7040d0340 0x7fe7040ef640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:45:19.436 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:19.417 7fe70305f700 1 --2- 172.21.2.112:0/634166301 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe7040d1c70 0x7fe7040efc60 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 rx=0x7fe6fc006aa0 tx=0x7fe6fc009f20).ready entity=mon.1 client_cookie=f2ef7d5cc6649f36 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:19.436 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:19.417 7fe70185c700 1 -- 172.21.2.112:0/634166301 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe6fc00b5c0 con 0x7fe7040d1c70 2021-02-10T11:45:19.436 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:19.417 7fe70185c700 1 -- 172.21.2.112:0/634166301 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6fc003030 con 0x7fe7040d1c70 2021-02-10T11:45:19.436 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:19.417 7fe70185c700 1 -- 172.21.2.112:0/634166301 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe6fc00b9f0 con 0x7fe7040d1c70 2021-02-10T11:45:19.436 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:19.418 7fe708b6b700 1 -- 172.21.2.112:0/634166301 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe7040d1c70 msgr2=0x7fe7040efc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:19.436 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:19.418 7fe708b6b700 1 --2- 172.21.2.112:0/634166301 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe7040d1c70 0x7fe7040efc60 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 rx=0x7fe6fc006aa0 tx=0x7fe6fc009f20).stop 2021-02-10T11:45:19.436 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:19.418 7fe708b6b700 1 -- 172.21.2.112:0/634166301 shutdown_connections 2021-02-10T11:45:19.436 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:19.418 7fe708b6b700 1 --2- 172.21.2.112:0/634166301 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe7040d1c70 0x7fe7040efc60 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:19.437 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:19.418 7fe708b6b700 1 --2- 172.21.2.112:0/634166301 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe7040d0340 0x7fe7040ef640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:19.437 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:19.418 7fe708b6b700 1 -- 172.21.2.112:0/634166301 >> 172.21.2.112:0/634166301 conn(0x7fe7040ce450 msgr2=0x7fe7040d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:19.437 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:19.418 7fe708b6b700 1 -- 172.21.2.112:0/634166301 shutdown_connections 2021-02-10T11:45:19.437 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:19.418 7fe708b6b700 1 -- 172.21.2.112:0/634166301 wait complete. 2021-02-10T11:45:19.437 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:19.432 7fe708b6b700 1 Processor -- start 2021-02-10T11:45:19.437 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:19.433 7fe708b6b700 1 -- start start 2021-02-10T11:45:19.438 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:19.433 7fe708b6b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe70418d870 0x7fe7040b19b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:19.438 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:19.433 7fe708b6b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe7040b1fd0 0x7fe704099ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:19.439 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:19.433 7fe708b6b700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe7040b23d0 0x7fe70409a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:19.439 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:19.433 7fe708b6b700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe70418dc70 con 0x7fe7040b1fd0 2021-02-10T11:45:19.439 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:19.433 7fe708b6b700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe704199e60 con 0x7fe7040b23d0 2021-02-10T11:45:19.439 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:19.433 7fe708b6b700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe70409ab30 con 0x7fe70418d870 2021-02-10T11:45:19.439 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:19.433 7fe70205d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe7040b1fd0 0x7fe704099ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:19.439 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:19.433 7fe70205d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe7040b1fd0 0x7fe704099ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:58562/0 (socket says 172.21.2.112:58562) 2021-02-10T11:45:19.439 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:19.433 7fe70205d700 1 -- 172.21.2.112:0/2248545833 learned_addr learned my addr 172.21.2.112:0/2248545833 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:45:19.439 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:19.434 7fe70205d700 1 -- 172.21.2.112:0/2248545833 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe70418d870 msgr2=0x7fe7040b19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:19.439 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:19.434 7fe70205d700 1 --2- 172.21.2.112:0/2248545833 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe70418d870 0x7fe7040b19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:19.439 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:19.434 7fe70205d700 1 -- 172.21.2.112:0/2248545833 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe7040b23d0 msgr2=0x7fe70409a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:19.439 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:19.434 7fe70205d700 1 --2- 172.21.2.112:0/2248545833 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe7040b23d0 0x7fe70409a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:19.440 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:19.434 7fe70205d700 1 -- 172.21.2.112:0/2248545833 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe70409ad50 con 0x7fe7040b1fd0 2021-02-10T11:45:19.440 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:19.434 7fe70205d700 1 --2- 172.21.2.112:0/2248545833 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe7040b1fd0 0x7fe704099ef0 secure :-1 s=READY pgs=2715 cs=0 l=1 rev1=1 rx=0x7fe6f8006f40 tx=0x7fe6f800b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:19.440 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:19.434 7fe6f37fe700 1 -- 172.21.2.112:0/2248545833 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe6f800bd80 con 0x7fe7040b1fd0 2021-02-10T11:45:19.440 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:19.434 7fe6f37fe700 1 -- 172.21.2.112:0/2248545833 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6f8000c90 con 0x7fe7040b1fd0 2021-02-10T11:45:19.440 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:19.434 7fe708b6b700 1 -- 172.21.2.112:0/2248545833 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe70409ad50 con 0x7fe7040b1fd0 2021-02-10T11:45:19.440 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:19.434 7fe708b6b700 1 -- 172.21.2.112:0/2248545833 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe70418e3f0 con 0x7fe7040b1fd0 2021-02-10T11:45:19.440 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:19.434 7fe6f37fe700 1 -- 172.21.2.112:0/2248545833 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe6f80022b0 con 0x7fe7040b1fd0 2021-02-10T11:45:19.440 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:19.435 7fe6f37fe700 1 -- 172.21.2.112:0/2248545833 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe6f800a000 con 0x7fe7040b1fd0 2021-02-10T11:45:19.440 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:19.435 7fe6f37fe700 1 --2- 172.21.2.112:0/2248545833 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe6e80186c0 0x7fe6e801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:19.440 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:19.435 7fe6f37fe700 1 -- 172.21.2.112:0/2248545833 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fe6f80325a0 con 0x7fe7040b1fd0 2021-02-10T11:45:19.440 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:19.435 7fe70285e700 1 --2- 172.21.2.112:0/2248545833 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe6e80186c0 0x7fe6e801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:19.440 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:19.435 7fe70285e700 1 --2- 172.21.2.112:0/2248545833 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe6e80186c0 0x7fe6e801abc0 crc :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:19.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:19.466 7fe708b6b700 1 -- 172.21.2.112:0/2248545833 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7040999c0 con 0x7fe7040b1fd0 2021-02-10T11:45:19.469 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:19.468 7fe6f37fe700 1 -- 172.21.2.112:0/2248545833 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe6f8000990 con 0x7fe7040b1fd0 2021-02-10T11:45:19.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:19.597 7fe708b6b700 1 -- 172.21.2.112:0/2248545833 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe7040999c0 con 0x7fe7040b1fd0 2021-02-10T11:45:19.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:19.597 7fe6f37fe700 1 -- 172.21.2.112:0/2248545833 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 131 v131) v1 ==== 95+0+2108 (secure 0 0 0) 0x7fe6f800bd80 con 0x7fe7040b1fd0 2021-02-10T11:45:19.600 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:45:19.600 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":131,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":12006,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/1015001314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":1015001314},{"type":"v1","addr":"172.21.2.112:6833","nonce":1015001314}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":126}],"filesystems":[{"mdsmap":{"epoch":130,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:44:43.721296","modified":"2021-02-10 11:44:45.730948","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7268},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7268":{"gid":7268,"name":"b","rank":0,"incarnation":127,"state":"up:active","state_seq":3,"addr":"172.21.2.120:6835/3799929082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":3799929082},{"type":"v1","addr":"172.21.2.120:6835","nonce":3799929082}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":16}]} 2021-02-10T11:45:19.600 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 131 2021-02-10T11:45:19.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:19.599 7fe708b6b700 1 -- 172.21.2.112:0/2248545833 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe6e80186c0 msgr2=0x7fe6e801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:19.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:19.599 7fe708b6b700 1 --2- 172.21.2.112:0/2248545833 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe6e80186c0 0x7fe6e801abc0 crc :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:19.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:19.599 7fe708b6b700 1 -- 172.21.2.112:0/2248545833 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe7040b1fd0 msgr2=0x7fe704099ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:19.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:19.599 7fe708b6b700 1 --2- 172.21.2.112:0/2248545833 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe7040b1fd0 0x7fe704099ef0 secure :-1 s=READY pgs=2715 cs=0 l=1 rev1=1 rx=0x7fe6f8006f40 tx=0x7fe6f800b990).stop 2021-02-10T11:45:19.602 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:19.599 7fe708b6b700 1 -- 172.21.2.112:0/2248545833 shutdown_connections 2021-02-10T11:45:19.602 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:19.599 7fe708b6b700 1 --2- 172.21.2.112:0/2248545833 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe6e80186c0 0x7fe6e801abc0 unknown :-1 s=CLOSED pgs=1239 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:19.602 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:19.599 7fe708b6b700 1 --2- 172.21.2.112:0/2248545833 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe7040b23d0 0x7fe70409a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:19.602 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:19.599 7fe708b6b700 1 --2- 172.21.2.112:0/2248545833 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe7040b1fd0 0x7fe704099ef0 unknown :-1 s=CLOSED pgs=2715 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:19.602 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:19.599 7fe708b6b700 1 --2- 172.21.2.112:0/2248545833 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe70418d870 0x7fe7040b19b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:19.602 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:19.599 7fe708b6b700 1 -- 172.21.2.112:0/2248545833 >> 172.21.2.112:0/2248545833 conn(0x7fe7040f0550 msgr2=0x7fe704191730 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:19.602 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:19.600 7fe708b6b700 1 -- 172.21.2.112:0/2248545833 shutdown_connections 2021-02-10T11:45:19.602 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:19.600 7fe708b6b700 1 -- 172.21.2.112:0/2248545833 wait complete. 2021-02-10T11:45:19.610 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs fail cephfs 2021-02-10T11:45:20.038 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.020 7f274476f700 1 Processor -- start 2021-02-10T11:45:20.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.020 7f274476f700 1 -- start start 2021-02-10T11:45:20.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.021 7f274476f700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f273c0d1870 0x7f273c1b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:20.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.021 7f274476f700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f273c1b0a10 0x7f273c0cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:20.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.021 7f274476f700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f273c0f0b10 con 0x7f273c0d1c70 2021-02-10T11:45:20.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.021 7f274476f700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f273c0e5ed0 con 0x7f273c1b0a10 2021-02-10T11:45:20.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.021 7f274476f700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f273c0f0650 con 0x7f273c0d1870 2021-02-10T11:45:20.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.021 7f2741d8a700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f273c1b0a10 0x7f273c0cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:20.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.021 7f2742d8c700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f273c0d1c70 0x7f273c0cf950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:58032/0 (socket says 172.21.2.112:58032) 2021-02-10T11:45:20.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.021 7f2741d8a700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f273c1b0a10 0x7f273c0cf330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:58570/0 (socket says 172.21.2.112:58570) 2021-02-10T11:45:20.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.021 7f2741d8a700 1 -- 172.21.2.112:0/3875215072 learned_addr learned my addr 172.21.2.112:0/3875215072 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:45:20.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.021 7f2741d8a700 1 -- 172.21.2.112:0/3875215072 >> v2:172.21.2.120:3300/0 conn(0x7f273c0d1870 msgr2=0x7f273c1b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:20.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.021 7f2741d8a700 1 --2- 172.21.2.112:0/3875215072 >> v2:172.21.2.120:3300/0 conn(0x7f273c0d1870 0x7f273c1b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:20.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.021 7f2741d8a700 1 -- 172.21.2.112:0/3875215072 >> v1:172.21.2.112:6789/0 conn(0x7f273c0d1c70 legacy=0x7f273c0cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:20.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.021 7f2741d8a700 1 -- 172.21.2.112:0/3875215072 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f273c0f0840 con 0x7f273c1b0a10 2021-02-10T11:45:20.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.021 7f2741d8a700 1 --2- 172.21.2.112:0/3875215072 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f273c1b0a10 0x7f273c0cf330 secure :-1 s=READY pgs=2717 cs=0 l=1 rev1=1 rx=0x7f2738006700 tx=0x7f2738009b00).ready entity=mon.0 client_cookie=998213beefb6db37 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:20.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.022 7f2741589700 1 -- 172.21.2.112:0/3875215072 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f273800c0c0 con 0x7f273c1b0a10 2021-02-10T11:45:20.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.022 7f2741589700 1 -- 172.21.2.112:0/3875215072 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f273800c340 con 0x7f273c1b0a10 2021-02-10T11:45:20.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.022 7f2741589700 1 -- 172.21.2.112:0/3875215072 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f273800c820 con 0x7f273c1b0a10 2021-02-10T11:45:20.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.022 7f274476f700 1 -- 172.21.2.112:0/3875215072 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f273c1b0a10 msgr2=0x7f273c0cf330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:20.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.022 7f274476f700 1 --2- 172.21.2.112:0/3875215072 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f273c1b0a10 0x7f273c0cf330 secure :-1 s=READY pgs=2717 cs=0 l=1 rev1=1 rx=0x7f2738006700 tx=0x7f2738009b00).stop 2021-02-10T11:45:20.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.022 7f274476f700 1 -- 172.21.2.112:0/3875215072 shutdown_connections 2021-02-10T11:45:20.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.022 7f274476f700 1 --2- 172.21.2.112:0/3875215072 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f273c1b0a10 0x7f273c0cf330 unknown :-1 s=CLOSED pgs=2717 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:20.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.022 7f274476f700 1 --2- 172.21.2.112:0/3875215072 >> v2:172.21.2.120:3300/0 conn(0x7f273c0d1870 0x7f273c1b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:20.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.023 7f274476f700 1 -- 172.21.2.112:0/3875215072 >> 172.21.2.112:0/3875215072 conn(0x7f273c0ce450 msgr2=0x7f273c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:20.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.023 7f274476f700 1 -- 172.21.2.112:0/3875215072 shutdown_connections 2021-02-10T11:45:20.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.023 7f274476f700 1 -- 172.21.2.112:0/3875215072 wait complete. 2021-02-10T11:45:20.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.037 7f274476f700 1 Processor -- start 2021-02-10T11:45:20.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.037 7f274476f700 1 -- start start 2021-02-10T11:45:20.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.038 7f274476f700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f273c1908b0 0x7f273c1a4440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:20.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.038 7f274476f700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f273c0bd920 0x7f273c0bdd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:20.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.038 7f274476f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f273c08d510 0x7f273c08d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:20.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.038 7f274476f700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f273c08cf90 con 0x7f273c08d510 2021-02-10T11:45:20.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.038 7f274476f700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f273c0be340 con 0x7f273c0bd920 2021-02-10T11:45:20.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.038 7f274476f700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f273c0be680 con 0x7f273c1908b0 2021-02-10T11:45:20.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.038 7f2742d8c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f273c08d510 0x7f273c08d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:20.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.038 7f2742d8c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f273c08d510 0x7f273c08d910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:58578/0 (socket says 172.21.2.112:58578) 2021-02-10T11:45:20.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.038 7f2742d8c700 1 -- 172.21.2.112:0/1438166310 learned_addr learned my addr 172.21.2.112:0/1438166310 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:45:20.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.038 7f2742d8c700 1 -- 172.21.2.112:0/1438166310 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f273c1908b0 msgr2=0x7f273c1a4440 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:45:20.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.038 7f2742d8c700 1 --2- 172.21.2.112:0/1438166310 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f273c1908b0 0x7f273c1a4440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:20.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.038 7f2742d8c700 1 -- 172.21.2.112:0/1438166310 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f273c0bd920 msgr2=0x7f273c0bdd20 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:45:20.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.038 7f2742d8c700 1 --2- 172.21.2.112:0/1438166310 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f273c0bd920 0x7f273c0bdd20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:20.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.038 7f2742d8c700 1 -- 172.21.2.112:0/1438166310 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f273c1a4a60 con 0x7f273c08d510 2021-02-10T11:45:20.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.038 7f2742d8c700 1 --2- 172.21.2.112:0/1438166310 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f273c08d510 0x7f273c08d910 secure :-1 s=READY pgs=2718 cs=0 l=1 rev1=1 rx=0x7f2734001b70 tx=0x7f273400ac50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:20.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.039 7f272effd700 1 -- 172.21.2.112:0/1438166310 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2734004930 con 0x7f273c08d510 2021-02-10T11:45:20.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.039 7f272effd700 1 -- 172.21.2.112:0/1438166310 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2734004c30 con 0x7f273c08d510 2021-02-10T11:45:20.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.039 7f274476f700 1 -- 172.21.2.112:0/1438166310 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f273c1a4a60 con 0x7f273c08d510 2021-02-10T11:45:20.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.039 7f274476f700 1 -- 172.21.2.112:0/1438166310 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f273c08df30 con 0x7f273c08d510 2021-02-10T11:45:20.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.039 7f272effd700 1 -- 172.21.2.112:0/1438166310 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f27340072a0 con 0x7f273c08d510 2021-02-10T11:45:20.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.039 7f272effd700 1 -- 172.21.2.112:0/1438166310 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2734004720 con 0x7f273c08d510 2021-02-10T11:45:20.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.039 7f272effd700 1 --2- 172.21.2.112:0/1438166310 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2724018730 0x7f272401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:20.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.039 7f272effd700 1 -- 172.21.2.112:0/1438166310 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f2734032b10 con 0x7f273c08d510 2021-02-10T11:45:20.045 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.040 7f274258b700 1 --2- 172.21.2.112:0/1438166310 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2724018730 0x7f272401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:20.045 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.040 7f274258b700 1 --2- 172.21.2.112:0/1438166310 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2724018730 0x7f272401ac10 crc :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:20.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.070 7f274476f700 1 -- 172.21.2.112:0/1438166310 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f273c0ebc50 con 0x7f273c08d510 2021-02-10T11:45:20.074 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.073 7f272effd700 1 -- 172.21.2.112:0/1438166310 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2734004c10 con 0x7f273c08d510 2021-02-10T11:45:20.203 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.201 7f274476f700 1 -- 172.21.2.112:0/1438166310 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f273c0f0b10 con 0x7f273c08d510 2021-02-10T11:45:20.257 INFO:tasks.ceph.mon.a.gibba012.stderr:2021-02-10 11:45:20.255 7fc3fe6e2700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-02-10T11:45:20.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.260 7f272effd700 1 -- 172.21.2.112:0/1438166310 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v132) v1 ==== 161+0+0 (secure 0 0 0) 0x7f2734004c10 con 0x7f273c08d510 2021-02-10T11:45:20.267 INFO:teuthology.orchestra.run.gibba012.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2021-02-10T11:45:20.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.267 7f274476f700 1 -- 172.21.2.112:0/1438166310 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2724018730 msgr2=0x7f272401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:20.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.267 7f274476f700 1 --2- 172.21.2.112:0/1438166310 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2724018730 0x7f272401ac10 crc :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:20.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.267 7f274476f700 1 -- 172.21.2.112:0/1438166310 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f273c08d510 msgr2=0x7f273c08d910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:20.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.267 7f274476f700 1 --2- 172.21.2.112:0/1438166310 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f273c08d510 0x7f273c08d910 secure :-1 s=READY pgs=2718 cs=0 l=1 rev1=1 rx=0x7f2734001b70 tx=0x7f273400ac50).stop 2021-02-10T11:45:20.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.267 7f274476f700 1 -- 172.21.2.112:0/1438166310 shutdown_connections 2021-02-10T11:45:20.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.267 7f274476f700 1 --2- 172.21.2.112:0/1438166310 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2724018730 0x7f272401ac10 unknown :-1 s=CLOSED pgs=1240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:20.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.267 7f274476f700 1 --2- 172.21.2.112:0/1438166310 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f273c0bd920 0x7f273c0bdd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:20.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.267 7f274476f700 1 --2- 172.21.2.112:0/1438166310 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f273c08d510 0x7f273c08d910 unknown :-1 s=CLOSED pgs=2718 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:20.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.267 7f274476f700 1 --2- 172.21.2.112:0/1438166310 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f273c1908b0 0x7f273c1a4440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:20.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.268 7f274476f700 1 -- 172.21.2.112:0/1438166310 >> 172.21.2.112:0/1438166310 conn(0x7f273c08f030 msgr2=0x7f273c0b5e80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:20.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.268 7f274476f700 1 -- 172.21.2.112:0/1438166310 shutdown_connections 2021-02-10T11:45:20.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.268 7f274476f700 1 -- 172.21.2.112:0/1438166310 wait complete. 2021-02-10T11:45:20.278 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:45:20.305 INFO:tasks.ceph.mds.b.gibba020.stdout:starting mds.b at 2021-02-10T11:45:20.708 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.689 7ff64a0c5700 1 Processor -- start 2021-02-10T11:45:20.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.689 7ff64a0c5700 1 -- start start 2021-02-10T11:45:20.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.690 7ff64a0c5700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff6440d1c70 0x7ff6440f20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:20.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.690 7ff64a0c5700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6440f0b10 con 0x7ff6440d0340 2021-02-10T11:45:20.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.690 7ff64a0c5700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6441b0be0 con 0x7ff6440d1870 2021-02-10T11:45:20.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.690 7ff64a0c5700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6440e5ed0 con 0x7ff6440d1c70 2021-02-10T11:45:20.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.690 7ff642ffd700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7ff6440d0340 0x7ff6440ef640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:58046/0 (socket says 172.21.2.112:58046) 2021-02-10T11:45:20.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.690 7ff642ffd700 1 -- 172.21.2.112:0/1769463538 learned_addr learned my addr 172.21.2.112:0/1769463538 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:45:20.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.690 7ff6427fc700 1 -- 172.21.2.112:0/1769463538 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3473084024 0 0) 0x7ff6340010b0 con 0x7ff6440d0340 2021-02-10T11:45:20.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.690 7ff6427fc700 1 -- 172.21.2.112:0/1769463538 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff630002fe0 con 0x7ff6440d0340 2021-02-10T11:45:20.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.691 7ff6427fc700 1 -- 172.21.2.112:0/1769463538 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 626924987 0 0) 0x7ff6340010b0 con 0x7ff6440d0340 2021-02-10T11:45:20.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.691 7ff6427fc700 1 -- 172.21.2.112:0/1769463538 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff6440d1c70 msgr2=0x7ff6440f20a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:20.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.691 7ff643fff700 1 --2- 172.21.2.112:0/1769463538 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff6440d1c70 0x7ff6440f20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:20.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.691 7ff6427fc700 1 --2- 172.21.2.112:0/1769463538 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff6440d1c70 0x7ff6440f20a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:20.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.691 7ff6427fc700 1 -- 172.21.2.112:0/1769463538 >> v1:172.21.2.120:6789/0 conn(0x7ff6440d1870 legacy=0x7ff6441b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:20.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.691 7ff6427fc700 1 -- 172.21.2.112:0/1769463538 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6440efe30 con 0x7ff6440d0340 2021-02-10T11:45:20.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.691 7ff6427fc700 1 -- 172.21.2.112:0/1769463538 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7ff6340015b0 con 0x7ff6440d0340 2021-02-10T11:45:20.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.691 7ff6427fc700 1 -- 172.21.2.112:0/1769463538 >> v1:172.21.2.112:6789/0 conn(0x7ff6440d0340 legacy=0x7ff6440ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:20.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.691 7ff6427fc700 1 --2- 172.21.2.112:0/1769463538 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff630003b10 0x7ff630006010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:20.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.691 7ff6427fc700 1 --2- 172.21.2.112:0/1769463538 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff630006820 0x7ff630008d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:20.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.691 7ff6427fc700 1 --2- 172.21.2.112:0/1769463538 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff630009640 0x7ff63000bb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:20.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.691 7ff6427fc700 1 -- 172.21.2.112:0/1769463538 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff63000c360 con 0x7ff630006820 2021-02-10T11:45:20.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.691 7ff6427fc700 1 -- 172.21.2.112:0/1769463538 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff63000c580 con 0x7ff630003b10 2021-02-10T11:45:20.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.691 7ff6427fc700 1 -- 172.21.2.112:0/1769463538 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff63000c7a0 con 0x7ff630009640 2021-02-10T11:45:20.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.691 7ff643fff700 1 --2- 172.21.2.112:0/1769463538 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff6440d1c70 0x7ff6440f20a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:45:20.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.691 7ff642ffd700 1 --2- 172.21.2.112:0/1769463538 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff630006820 0x7ff630008d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:20.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.691 7ff643fff700 1 --2- 172.21.2.112:0/1769463538 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff630009640 0x7ff63000bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:20.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.691 7ff642ffd700 1 -- 172.21.2.112:0/1769463538 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff630009640 msgr2=0x7ff63000bb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:20.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.691 7ff642ffd700 1 --2- 172.21.2.112:0/1769463538 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff630009640 0x7ff63000bb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:20.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.691 7ff642ffd700 1 -- 172.21.2.112:0/1769463538 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff630003b10 msgr2=0x7ff630006010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:20.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.691 7ff6437fe700 1 --2- 172.21.2.112:0/1769463538 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff630003b10 0x7ff630006010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:20.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.691 7ff642ffd700 1 --2- 172.21.2.112:0/1769463538 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff630003b10 0x7ff630006010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:20.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.691 7ff642ffd700 1 -- 172.21.2.112:0/1769463538 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff63000ca40 con 0x7ff630006820 2021-02-10T11:45:20.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.691 7ff6437fe700 1 --2- 172.21.2.112:0/1769463538 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff630003b10 0x7ff630006010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:45:20.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.691 7ff642ffd700 1 --2- 172.21.2.112:0/1769463538 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff630006820 0x7ff630008d40 secure :-1 s=READY pgs=2722 cs=0 l=1 rev1=1 rx=0x7ff6340017d0 tx=0x7ff634007cb0).ready entity=mon.0 client_cookie=e3899856aebe0809 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:20.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.691 7ff6427fc700 1 -- 172.21.2.112:0/1769463538 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff634007df0 con 0x7ff630006820 2021-02-10T11:45:20.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.691 7ff6427fc700 1 -- 172.21.2.112:0/1769463538 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff63404e320 con 0x7ff630006820 2021-02-10T11:45:20.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.691 7ff6427fc700 1 -- 172.21.2.112:0/1769463538 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff63404e840 con 0x7ff630006820 2021-02-10T11:45:20.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.692 7ff64a0c5700 1 -- 172.21.2.112:0/1769463538 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff630006820 msgr2=0x7ff630008d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:20.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.692 7ff64a0c5700 1 --2- 172.21.2.112:0/1769463538 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff630006820 0x7ff630008d40 secure :-1 s=READY pgs=2722 cs=0 l=1 rev1=1 rx=0x7ff6340017d0 tx=0x7ff634007cb0).stop 2021-02-10T11:45:20.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.692 7ff643fff700 1 -- 172.21.2.112:0/1769463538 reap_dead start 2021-02-10T11:45:20.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.692 7ff64a0c5700 1 -- 172.21.2.112:0/1769463538 shutdown_connections 2021-02-10T11:45:20.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.692 7ff64a0c5700 1 -- 172.21.2.112:0/1769463538 >> 172.21.2.112:0/1769463538 conn(0x7ff6440ce450 msgr2=0x7ff6440d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:20.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.692 7ff64a0c5700 1 -- 172.21.2.112:0/1769463538 shutdown_connections 2021-02-10T11:45:20.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.692 7ff64a0c5700 1 -- 172.21.2.112:0/1769463538 wait complete. 2021-02-10T11:45:20.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.707 7ff64a0c5700 1 Processor -- start 2021-02-10T11:45:20.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.707 7ff64a0c5700 1 -- start start 2021-02-10T11:45:20.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.707 7ff64a0c5700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff644195650 0x7ff6440b5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:20.714 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.707 7ff64a0c5700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff6440b5b80 0x7ff6441a2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:20.714 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.707 7ff64a0c5700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff6440acfa0 0x7ff6441a3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:20.714 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.707 7ff64a0c5700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6440b6010 con 0x7ff644195650 2021-02-10T11:45:20.714 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.707 7ff64a0c5700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6441a3a60 con 0x7ff6440acfa0 2021-02-10T11:45:20.714 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.707 7ff64a0c5700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6441a3c40 con 0x7ff6440b5b80 2021-02-10T11:45:20.714 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.707 7ff6437fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff644195650 0x7ff6440b5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:20.714 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.707 7ff6437fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff644195650 0x7ff6440b5560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:58596/0 (socket says 172.21.2.112:58596) 2021-02-10T11:45:20.714 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.707 7ff6437fe700 1 -- 172.21.2.112:0/2548634032 learned_addr learned my addr 172.21.2.112:0/2548634032 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:45:20.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.708 7ff6437fe700 1 -- 172.21.2.112:0/2548634032 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff6440b5b80 msgr2=0x7ff6441a2e20 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:45:20.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.708 7ff6437fe700 1 --2- 172.21.2.112:0/2548634032 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff6440b5b80 0x7ff6441a2e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:20.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.708 7ff6437fe700 1 -- 172.21.2.112:0/2548634032 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff6440acfa0 msgr2=0x7ff6441a3440 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:45:20.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.708 7ff6437fe700 1 --2- 172.21.2.112:0/2548634032 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff6440acfa0 0x7ff6441a3440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:20.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.708 7ff6437fe700 1 -- 172.21.2.112:0/2548634032 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6440bcae0 con 0x7ff644195650 2021-02-10T11:45:20.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.708 7ff6437fe700 1 --2- 172.21.2.112:0/2548634032 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff644195650 0x7ff6440b5560 secure :-1 s=READY pgs=2723 cs=0 l=1 rev1=1 rx=0x7ff62c006f10 tx=0x7ff62c00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:20.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.708 7ff63bfff700 1 -- 172.21.2.112:0/2548634032 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff62c006be0 con 0x7ff644195650 2021-02-10T11:45:20.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.708 7ff63bfff700 1 -- 172.21.2.112:0/2548634032 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff62c0063e0 con 0x7ff644195650 2021-02-10T11:45:20.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.708 7ff63bfff700 1 -- 172.21.2.112:0/2548634032 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff62c0052e0 con 0x7ff644195650 2021-02-10T11:45:20.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.708 7ff64a0c5700 1 -- 172.21.2.112:0/2548634032 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6440bcae0 con 0x7ff644195650 2021-02-10T11:45:20.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.708 7ff64a0c5700 1 -- 172.21.2.112:0/2548634032 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6440bcae0 con 0x7ff644195650 2021-02-10T11:45:20.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.708 7ff63bfff700 1 -- 172.21.2.112:0/2548634032 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff62c009fe0 con 0x7ff644195650 2021-02-10T11:45:20.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.709 7ff63bfff700 1 --2- 172.21.2.112:0/2548634032 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff628018730 0x7ff62801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:20.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.709 7ff63bfff700 1 -- 172.21.2.112:0/2548634032 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 5659+0+0 (secure 0 0 0) 0x7ff62c032690 con 0x7ff644195650 2021-02-10T11:45:20.716 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.709 7ff642ffd700 1 --2- 172.21.2.112:0/2548634032 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff628018730 0x7ff62801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:20.716 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.710 7ff642ffd700 1 --2- 172.21.2.112:0/2548634032 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff628018730 0x7ff62801ac10 crc :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:20.741 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.739 7ff64a0c5700 1 -- 172.21.2.112:0/2548634032 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6440b6010 con 0x7ff644195650 2021-02-10T11:45:20.743 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.742 7ff63bfff700 1 -- 172.21.2.112:0/2548634032 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff62c0063c0 con 0x7ff644195650 2021-02-10T11:45:20.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.870 7ff64a0c5700 1 -- 172.21.2.112:0/2548634032 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff6440b6010 con 0x7ff644195650 2021-02-10T11:45:20.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.871 7ff63bfff700 1 -- 172.21.2.112:0/2548634032 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 132 v132) v1 ==== 95+0+1763 (secure 0 0 0) 0x7ff62c0063c0 con 0x7ff644195650 2021-02-10T11:45:20.873 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 132 2021-02-10T11:45:20.873 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:45:20.873 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":132,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":12006,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/1015001314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":1015001314},{"type":"v1","addr":"172.21.2.112:6833","nonce":1015001314}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":126}],"filesystems":[{"mdsmap":{"epoch":132,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:44:43.721296","modified":"2021-02-10 11:45:20.256632","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":152,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":16}]} 2021-02-10T11:45:20.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.873 7ff64a0c5700 1 -- 172.21.2.112:0/2548634032 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff628018730 msgr2=0x7ff62801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:20.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.873 7ff64a0c5700 1 --2- 172.21.2.112:0/2548634032 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff628018730 0x7ff62801ac10 crc :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:20.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.873 7ff64a0c5700 1 -- 172.21.2.112:0/2548634032 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff644195650 msgr2=0x7ff6440b5560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:20.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.873 7ff64a0c5700 1 --2- 172.21.2.112:0/2548634032 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff644195650 0x7ff6440b5560 secure :-1 s=READY pgs=2723 cs=0 l=1 rev1=1 rx=0x7ff62c006f10 tx=0x7ff62c00b990).stop 2021-02-10T11:45:20.875 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.873 7ff64a0c5700 1 -- 172.21.2.112:0/2548634032 shutdown_connections 2021-02-10T11:45:20.875 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.873 7ff64a0c5700 1 --2- 172.21.2.112:0/2548634032 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff628018730 0x7ff62801ac10 unknown :-1 s=CLOSED pgs=1241 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:20.875 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.873 7ff64a0c5700 1 --2- 172.21.2.112:0/2548634032 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff6440acfa0 0x7ff6441a3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:20.875 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.873 7ff64a0c5700 1 --2- 172.21.2.112:0/2548634032 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff6440b5b80 0x7ff6441a2e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:20.875 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.873 7ff64a0c5700 1 --2- 172.21.2.112:0/2548634032 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff644195650 0x7ff6440b5560 unknown :-1 s=CLOSED pgs=2723 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:20.875 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.873 7ff64a0c5700 1 -- 172.21.2.112:0/2548634032 >> 172.21.2.112:0/2548634032 conn(0x7ff6440b3040 msgr2=0x7ff6440c3250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:20.875 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.873 7ff64a0c5700 1 -- 172.21.2.112:0/2548634032 shutdown_connections 2021-02-10T11:45:20.875 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:20.873 7ff64a0c5700 1 -- 172.21.2.112:0/2548634032 wait complete. 2021-02-10T11:45:20.883 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2021-02-10T11:45:21.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:21.298 7f30b2deb700 1 Processor -- start 2021-02-10T11:45:21.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:21.299 7f30b2deb700 1 -- start start 2021-02-10T11:45:21.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:21.299 7f30b2deb700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f30ac0d1c70 0x7f30ac0f20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:21.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:21.299 7f30b2deb700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30ac0f0b10 con 0x7f30ac0d1870 2021-02-10T11:45:21.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:21.299 7f30b2deb700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30ac1b0be0 con 0x7f30ac0d0340 2021-02-10T11:45:21.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:21.299 7f30b2deb700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30ac0e5ed0 con 0x7f30ac0d1c70 2021-02-10T11:45:21.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:21.299 7f30b0c07700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f30ac0d1870 0x7f30ac1b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:58066/0 (socket says 172.21.2.112:58066) 2021-02-10T11:45:21.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:21.299 7f30b0c07700 1 -- 172.21.2.112:0/4009165843 learned_addr learned my addr 172.21.2.112:0/4009165843 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:45:21.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:21.299 7f30ab7fe700 1 -- 172.21.2.112:0/4009165843 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3127010336 0 0) 0x7f30940009b0 con 0x7f30ac0d1870 2021-02-10T11:45:21.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:21.300 7f30ab7fe700 1 -- 172.21.2.112:0/4009165843 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3098002fe0 con 0x7f30ac0d1870 2021-02-10T11:45:21.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:21.300 7f30ab7fe700 1 -- 172.21.2.112:0/4009165843 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3208862797 0 0) 0x7f3094001010 con 0x7f30ac0d1870 2021-02-10T11:45:21.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:21.300 7f30ab7fe700 1 -- 172.21.2.112:0/4009165843 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f30ac0d1c70 msgr2=0x7f30ac0f20a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:21.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:21.300 7f30ab7fe700 1 --2- 172.21.2.112:0/4009165843 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f30ac0d1c70 0x7f30ac0f20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:21.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:21.300 7f30ab7fe700 1 -- 172.21.2.112:0/4009165843 >> v1:172.21.2.120:6789/0 conn(0x7f30ac0d0340 legacy=0x7f30ac0ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:21.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:21.300 7f30ab7fe700 1 -- 172.21.2.112:0/4009165843 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30ac0efe30 con 0x7f30ac0d1870 2021-02-10T11:45:21.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:21.300 7f30ab7fe700 1 -- 172.21.2.112:0/4009165843 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f3094001590 con 0x7f30ac0d1870 2021-02-10T11:45:21.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:21.300 7f30ab7fe700 1 -- 172.21.2.112:0/4009165843 >> v1:172.21.2.112:6789/0 conn(0x7f30ac0d1870 legacy=0x7f30ac1b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:21.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:21.300 7f30ab7fe700 1 --2- 172.21.2.112:0/4009165843 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3098003a60 0x7f3098005e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:21.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:21.300 7f30ab7fe700 1 --2- 172.21.2.112:0/4009165843 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3098006730 0x7f3098008c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:21.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:21.300 7f30ab7fe700 1 --2- 172.21.2.112:0/4009165843 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f30980094f0 0x7f309800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:21.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:21.300 7f30ab7fe700 1 -- 172.21.2.112:0/4009165843 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f309800c2c0 con 0x7f30980094f0 2021-02-10T11:45:21.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:21.300 7f30ab7fe700 1 -- 172.21.2.112:0/4009165843 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f309800c4e0 con 0x7f3098006730 2021-02-10T11:45:21.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:21.300 7f30ab7fe700 1 -- 172.21.2.112:0/4009165843 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f309800c700 con 0x7f3098003a60 2021-02-10T11:45:21.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:21.300 7f30b1408700 1 --2- 172.21.2.112:0/4009165843 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f30980094f0 0x7f309800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:21.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:21.300 7f30b0c07700 1 --2- 172.21.2.112:0/4009165843 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3098003a60 0x7f3098005e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:21.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:21.300 7f30abfff700 1 --2- 172.21.2.112:0/4009165843 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3098006730 0x7f3098008c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:21.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:21.300 7f30b1408700 1 -- 172.21.2.112:0/4009165843 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3098003a60 msgr2=0x7f3098005e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:21.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:21.300 7f30b1408700 1 --2- 172.21.2.112:0/4009165843 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3098003a60 0x7f3098005e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:21.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:21.300 7f30b1408700 1 -- 172.21.2.112:0/4009165843 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3098006730 msgr2=0x7f3098008c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:21.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:21.300 7f30b1408700 1 --2- 172.21.2.112:0/4009165843 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3098006730 0x7f3098008c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:21.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:21.300 7f30b1408700 1 -- 172.21.2.112:0/4009165843 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f309800c9a0 con 0x7f30980094f0 2021-02-10T11:45:21.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:21.300 7f30b1408700 1 --2- 172.21.2.112:0/4009165843 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f30980094f0 0x7f309800ba10 secure :-1 s=READY pgs=2725 cs=0 l=1 rev1=1 rx=0x7f30a4006e40 tx=0x7f30a4009e30).ready entity=mon.0 client_cookie=1e615e221b513e4a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:21.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:21.301 7f30ab7fe700 1 -- 172.21.2.112:0/4009165843 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f30a4006a10 con 0x7f30980094f0 2021-02-10T11:45:21.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:21.301 7f30ab7fe700 1 -- 172.21.2.112:0/4009165843 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30a400b300 con 0x7f30980094f0 2021-02-10T11:45:21.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:21.301 7f30ab7fe700 1 -- 172.21.2.112:0/4009165843 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f30a400b820 con 0x7f30980094f0 2021-02-10T11:45:21.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:21.301 7f30b2deb700 1 -- 172.21.2.112:0/4009165843 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f30980094f0 msgr2=0x7f309800ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:21.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:21.301 7f30b2deb700 1 --2- 172.21.2.112:0/4009165843 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f30980094f0 0x7f309800ba10 secure :-1 s=READY pgs=2725 cs=0 l=1 rev1=1 rx=0x7f30a4006e40 tx=0x7f30a4009e30).stop 2021-02-10T11:45:21.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:21.301 7f30b1408700 1 -- 172.21.2.112:0/4009165843 reap_dead start 2021-02-10T11:45:21.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:21.301 7f30b2deb700 1 -- 172.21.2.112:0/4009165843 shutdown_connections 2021-02-10T11:45:21.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:21.301 7f30b2deb700 1 -- 172.21.2.112:0/4009165843 >> 172.21.2.112:0/4009165843 conn(0x7f30ac0ce450 msgr2=0x7f30ac0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:21.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:21.301 7f30b2deb700 1 -- 172.21.2.112:0/4009165843 shutdown_connections 2021-02-10T11:45:21.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:21.301 7f30b2deb700 1 -- 172.21.2.112:0/4009165843 wait complete. 2021-02-10T11:45:21.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:21.316 7f30b2deb700 1 Processor -- start 2021-02-10T11:45:21.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:21.316 7f30b2deb700 1 -- start start 2021-02-10T11:45:21.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:21.316 7f30b2deb700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f30ac094310 0x7f30ac0ebe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:21.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:21.316 7f30b2deb700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f30ac094710 0x7f30ac0bca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:21.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:21.316 7f30b2deb700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f30ac0ec490 0x7f30ac0bd050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:21.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:21.316 7f30b2deb700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30ac093d90 con 0x7f30ac0ec490 2021-02-10T11:45:21.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:21.316 7f30b2deb700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30ac0bd670 con 0x7f30ac094710 2021-02-10T11:45:21.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:21.316 7f30b2deb700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30ac0939f0 con 0x7f30ac094310 2021-02-10T11:45:21.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:21.316 7f30b1408700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f30ac0ec490 0x7f30ac0bd050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:21.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:21.316 7f30b1408700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f30ac0ec490 0x7f30ac0bd050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:58614/0 (socket says 172.21.2.112:58614) 2021-02-10T11:45:21.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:21.316 7f30b1408700 1 -- 172.21.2.112:0/3253270566 learned_addr learned my addr 172.21.2.112:0/3253270566 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:45:21.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:21.317 7f30b1408700 1 -- 172.21.2.112:0/3253270566 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f30ac094310 msgr2=0x7f30ac0ebe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:21.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:21.317 7f30b1408700 1 --2- 172.21.2.112:0/3253270566 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f30ac094310 0x7f30ac0ebe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:21.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:21.317 7f30b1408700 1 -- 172.21.2.112:0/3253270566 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f30ac094710 msgr2=0x7f30ac0bca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:21.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:21.317 7f30b1408700 1 --2- 172.21.2.112:0/3253270566 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f30ac094710 0x7f30ac0bca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:21.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:21.317 7f30b1408700 1 -- 172.21.2.112:0/3253270566 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30ac0bda80 con 0x7f30ac0ec490 2021-02-10T11:45:21.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:21.317 7f30b1408700 1 --2- 172.21.2.112:0/3253270566 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f30ac0ec490 0x7f30ac0bd050 secure :-1 s=READY pgs=2726 cs=0 l=1 rev1=1 rx=0x7f30a4009970 tx=0x7f30a400bc40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:21.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:21.317 7f30a97fa700 1 -- 172.21.2.112:0/3253270566 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f30a40093d0 con 0x7f30ac0ec490 2021-02-10T11:45:21.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:21.317 7f30a97fa700 1 -- 172.21.2.112:0/3253270566 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30a4002030 con 0x7f30ac0ec490 2021-02-10T11:45:21.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:21.317 7f30a97fa700 1 -- 172.21.2.112:0/3253270566 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f30a40024d0 con 0x7f30ac0ec490 2021-02-10T11:45:21.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:21.317 7f30b2deb700 1 -- 172.21.2.112:0/3253270566 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30ac0bda80 con 0x7f30ac0ec490 2021-02-10T11:45:21.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:21.317 7f30b2deb700 1 -- 172.21.2.112:0/3253270566 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30ac1a76d0 con 0x7f30ac0ec490 2021-02-10T11:45:21.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:21.318 7f30a97fa700 1 -- 172.21.2.112:0/3253270566 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f30a4006020 con 0x7f30ac0ec490 2021-02-10T11:45:21.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:21.318 7f30a97fa700 1 --2- 172.21.2.112:0/3253270566 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f30900186c0 0x7f309001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:21.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:21.318 7f30a97fa700 1 -- 172.21.2.112:0/3253270566 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 5659+0+0 (secure 0 0 0) 0x7f30a4032930 con 0x7f30ac0ec490 2021-02-10T11:45:21.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:21.318 7f30b0c07700 1 --2- 172.21.2.112:0/3253270566 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f30900186c0 0x7f309001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:21.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:21.319 7f30b0c07700 1 --2- 172.21.2.112:0/3253270566 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f30900186c0 0x7f309001abc0 crc :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:21.351 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:21.349 7f30b2deb700 1 -- 172.21.2.112:0/3253270566 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30ac0ebc40 con 0x7f30ac0ec490 2021-02-10T11:45:21.353 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:21.352 7f30a97fa700 1 -- 172.21.2.112:0/3253270566 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f30a40093d0 con 0x7f30ac0ec490 2021-02-10T11:45:21.482 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:21.480 7f30b2deb700 1 -- 172.21.2.112:0/3253270566 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f30ac0f0b10 con 0x7f30ac0ec490 2021-02-10T11:45:22.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:22.268 7f30a97fa700 1 -- 172.21.2.112:0/3253270566 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v133) v1 ==== 104+0+0 (secure 0 0 0) 0x7f30a4006000 con 0x7f30ac0ec490 2021-02-10T11:45:22.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:22.302 7f30b2deb700 1 -- 172.21.2.112:0/3253270566 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f30900186c0 msgr2=0x7f309001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:22.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:22.302 7f30b2deb700 1 --2- 172.21.2.112:0/3253270566 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f30900186c0 0x7f309001abc0 crc :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:22.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:22.302 7f30b2deb700 1 -- 172.21.2.112:0/3253270566 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f30ac0ec490 msgr2=0x7f30ac0bd050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:22.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:22.302 7f30b2deb700 1 --2- 172.21.2.112:0/3253270566 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f30ac0ec490 0x7f30ac0bd050 secure :-1 s=READY pgs=2726 cs=0 l=1 rev1=1 rx=0x7f30a4009970 tx=0x7f30a400bc40).stop 2021-02-10T11:45:22.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:22.303 7f30b2deb700 1 -- 172.21.2.112:0/3253270566 shutdown_connections 2021-02-10T11:45:22.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:22.303 7f30b2deb700 1 --2- 172.21.2.112:0/3253270566 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f30900186c0 0x7f309001abc0 unknown :-1 s=CLOSED pgs=1242 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:22.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:22.303 7f30b2deb700 1 --2- 172.21.2.112:0/3253270566 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f30ac094710 0x7f30ac0bca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:22.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:22.303 7f30b2deb700 1 --2- 172.21.2.112:0/3253270566 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f30ac0ec490 0x7f30ac0bd050 unknown :-1 s=CLOSED pgs=2726 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:22.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:22.303 7f30b2deb700 1 --2- 172.21.2.112:0/3253270566 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f30ac094310 0x7f30ac0ebe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:22.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:22.303 7f30b2deb700 1 -- 172.21.2.112:0/3253270566 >> 172.21.2.112:0/3253270566 conn(0x7f30ac08fa10 msgr2=0x7f30ac0bfa60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:22.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:22.303 7f30b2deb700 1 -- 172.21.2.112:0/3253270566 shutdown_connections 2021-02-10T11:45:22.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:22.303 7f30b2deb700 1 -- 172.21.2.112:0/3253270566 wait complete. 2021-02-10T11:45:22.313 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2021-02-10T11:45:22.746 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:22.728 7f7a08842700 1 Processor -- start 2021-02-10T11:45:22.746 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:22.728 7f7a08842700 1 -- start start 2021-02-10T11:45:22.746 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:22.728 7f7a08842700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f7a040d1880 0x7f7a041b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:22.746 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:22.728 7f7a08842700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f7a041b0af0 0x7f7a04108950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:22.746 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:22.728 7f7a08842700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a040fe1d0 con 0x7f7a040d1c80 2021-02-10T11:45:22.746 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:22.728 7f7a08842700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a04109a70 con 0x7f7a040d1880 2021-02-10T11:45:22.746 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:22.728 7f7a08842700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a04109760 con 0x7f7a041b0af0 2021-02-10T11:45:22.746 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:22.728 7f7a0261c700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f7a040d1880 0x7f7a041b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:22.747 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:22.728 7f7a0261c700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f7a040d1880 0x7f7a041b04d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:58622/0 (socket says 172.21.2.112:58622) 2021-02-10T11:45:22.747 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:22.728 7f7a0261c700 1 -- 172.21.2.112:0/117367728 learned_addr learned my addr 172.21.2.112:0/117367728 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:45:22.747 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:22.729 7f7a0261c700 1 -- 172.21.2.112:0/117367728 >> v2:172.21.2.120:3300/0 conn(0x7f7a041b0af0 msgr2=0x7f7a04108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:22.747 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:22.729 7f7a0261c700 1 --2- 172.21.2.112:0/117367728 >> v2:172.21.2.120:3300/0 conn(0x7f7a041b0af0 0x7f7a04108950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:22.747 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:22.729 7f7a0261c700 1 -- 172.21.2.112:0/117367728 >> v1:172.21.2.120:6789/0 conn(0x7f7a040d1c80 legacy=0x7f7a04108f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:22.747 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:22.729 7f7a0261c700 1 -- 172.21.2.112:0/117367728 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a04088070 con 0x7f7a040d1880 2021-02-10T11:45:22.747 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:22.729 7f7a0261c700 1 --2- 172.21.2.112:0/117367728 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7a040d1880 0x7f7a041b04d0 secure :-1 s=READY pgs=2727 cs=0 l=1 rev1=1 rx=0x7f79f4006700 tx=0x7f79f4009b80).ready entity=mon.0 client_cookie=42021c7d29317d70 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:22.747 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:22.729 7f7a0161a700 1 -- 172.21.2.112:0/117367728 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f79f4000c90 con 0x7f7a040d1880 2021-02-10T11:45:22.747 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:22.729 7f7a0161a700 1 -- 172.21.2.112:0/117367728 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79f400c0c0 con 0x7f7a040d1880 2021-02-10T11:45:22.747 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:22.729 7f7a0161a700 1 -- 172.21.2.112:0/117367728 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f79f400c5e0 con 0x7f7a040d1880 2021-02-10T11:45:22.747 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:22.729 7f7a08842700 1 -- 172.21.2.112:0/117367728 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7a040d1880 msgr2=0x7f7a041b04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:22.747 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:22.729 7f7a08842700 1 --2- 172.21.2.112:0/117367728 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7a040d1880 0x7f7a041b04d0 secure :-1 s=READY pgs=2727 cs=0 l=1 rev1=1 rx=0x7f79f4006700 tx=0x7f79f4009b80).stop 2021-02-10T11:45:22.747 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:22.730 7f7a08842700 1 -- 172.21.2.112:0/117367728 shutdown_connections 2021-02-10T11:45:22.748 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:22.730 7f7a08842700 1 --2- 172.21.2.112:0/117367728 >> v2:172.21.2.120:3300/0 conn(0x7f7a041b0af0 0x7f7a04108950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:22.748 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:22.730 7f7a08842700 1 --2- 172.21.2.112:0/117367728 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7a040d1880 0x7f7a041b04d0 unknown :-1 s=CLOSED pgs=2727 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:22.748 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:22.730 7f7a08842700 1 -- 172.21.2.112:0/117367728 >> 172.21.2.112:0/117367728 conn(0x7f7a040ce430 msgr2=0x7f7a040d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:22.748 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:22.730 7f7a08842700 1 -- 172.21.2.112:0/117367728 shutdown_connections 2021-02-10T11:45:22.748 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:22.730 7f7a08842700 1 -- 172.21.2.112:0/117367728 wait complete. 2021-02-10T11:45:22.749 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:22.745 7f7a08842700 1 Processor -- start 2021-02-10T11:45:22.749 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:22.745 7f7a08842700 1 -- start start 2021-02-10T11:45:22.749 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:22.745 7f7a08842700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7a0419e5b0 0x7f7a041afc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:22.749 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:22.745 7f7a08842700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7a0419e9b0 0x7f7a041b0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:22.749 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:22.745 7f7a08842700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7a041b0250 0x7f7a041b13e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:22.749 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:22.745 7f7a08842700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a04195bd0 con 0x7f7a0419e5b0 2021-02-10T11:45:22.750 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:22.745 7f7a08842700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a041b0840 con 0x7f7a041b0250 2021-02-10T11:45:22.750 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:22.745 7f7a08842700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a0419eed0 con 0x7f7a0419e9b0 2021-02-10T11:45:22.750 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:22.745 7f7a0261c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7a0419e5b0 0x7f7a041afc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:22.750 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:22.745 7f7a0261c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7a0419e5b0 0x7f7a041afc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:58624/0 (socket says 172.21.2.112:58624) 2021-02-10T11:45:22.750 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:22.745 7f7a0261c700 1 -- 172.21.2.112:0/327947417 learned_addr learned my addr 172.21.2.112:0/327947417 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:45:22.750 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:22.746 7f7a0261c700 1 -- 172.21.2.112:0/327947417 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7a0419e9b0 msgr2=0x7f7a041b0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:22.750 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:22.746 7f7a0261c700 1 --2- 172.21.2.112:0/327947417 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7a0419e9b0 0x7f7a041b0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:22.750 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:22.746 7f7a0261c700 1 -- 172.21.2.112:0/327947417 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7a041b0250 msgr2=0x7f7a041b13e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:22.750 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:22.746 7f7a0261c700 1 --2- 172.21.2.112:0/327947417 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7a041b0250 0x7f7a041b13e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:22.750 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:22.746 7f7a0261c700 1 -- 172.21.2.112:0/327947417 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a0419f0b0 con 0x7f7a0419e5b0 2021-02-10T11:45:22.750 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:22.746 7f7a0261c700 1 --2- 172.21.2.112:0/327947417 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7a0419e5b0 0x7f7a041afc30 secure :-1 s=READY pgs=2728 cs=0 l=1 rev1=1 rx=0x7f79f4006f00 tx=0x7f79f400b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:22.751 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:22.746 7f79faffd700 1 -- 172.21.2.112:0/327947417 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f79f400bd80 con 0x7f7a0419e5b0 2021-02-10T11:45:22.751 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:22.746 7f79faffd700 1 -- 172.21.2.112:0/327947417 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79f4002030 con 0x7f7a0419e5b0 2021-02-10T11:45:22.751 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:22.746 7f79faffd700 1 -- 172.21.2.112:0/327947417 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f79f4002550 con 0x7f7a0419e5b0 2021-02-10T11:45:22.751 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:22.746 7f7a08842700 1 -- 172.21.2.112:0/327947417 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a0419f0b0 con 0x7f7a0419e5b0 2021-02-10T11:45:22.751 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:22.746 7f7a08842700 1 -- 172.21.2.112:0/327947417 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a0419f0b0 con 0x7f7a0419e5b0 2021-02-10T11:45:22.751 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:22.747 7f79faffd700 1 -- 172.21.2.112:0/327947417 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f79f4000c90 con 0x7f7a0419e5b0 2021-02-10T11:45:22.751 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:22.747 7f79faffd700 1 --2- 172.21.2.112:0/327947417 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f79e80186c0 0x7f79e801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:22.751 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:22.747 7f79faffd700 1 -- 172.21.2.112:0/327947417 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 5659+0+0 (secure 0 0 0) 0x7f79f4032490 con 0x7f7a0419e5b0 2021-02-10T11:45:22.751 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:22.747 7f7a01e1b700 1 --2- 172.21.2.112:0/327947417 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f79e80186c0 0x7f79e801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:22.751 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:22.748 7f7a01e1b700 1 --2- 172.21.2.112:0/327947417 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f79e80186c0 0x7f79e801abc0 crc :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:22.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:22.778 7f7a08842700 1 -- 172.21.2.112:0/327947417 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a04082aa0 con 0x7f7a0419e5b0 2021-02-10T11:45:22.782 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:22.781 7f79faffd700 1 -- 172.21.2.112:0/327947417 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f79f402a9d0 con 0x7f7a0419e5b0 2021-02-10T11:45:22.910 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:22.909 7f7a08842700 1 -- 172.21.2.112:0/327947417 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f7a04195bd0 con 0x7f7a0419e5b0 2021-02-10T11:45:23.273 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:23.272 7f79faffd700 1 -- 172.21.2.112:0/327947417 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v153) v1 ==== 185+0+0 (secure 0 0 0) 0x7f79f402a9d0 con 0x7f7a0419e5b0 2021-02-10T11:45:23.275 INFO:teuthology.orchestra.run.gibba012.stderr:pool 'cephfs_metadata' removed 2021-02-10T11:45:23.277 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:23.276 7f7a08842700 1 -- 172.21.2.112:0/327947417 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f79e80186c0 msgr2=0x7f79e801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:23.277 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:23.276 7f7a08842700 1 --2- 172.21.2.112:0/327947417 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f79e80186c0 0x7f79e801abc0 crc :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:23.277 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:23.276 7f7a08842700 1 -- 172.21.2.112:0/327947417 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7a0419e5b0 msgr2=0x7f7a041afc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:23.278 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:23.276 7f7a08842700 1 --2- 172.21.2.112:0/327947417 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7a0419e5b0 0x7f7a041afc30 secure :-1 s=READY pgs=2728 cs=0 l=1 rev1=1 rx=0x7f79f4006f00 tx=0x7f79f400b990).stop 2021-02-10T11:45:23.278 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:23.276 7f7a08842700 1 -- 172.21.2.112:0/327947417 shutdown_connections 2021-02-10T11:45:23.278 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:23.276 7f7a08842700 1 --2- 172.21.2.112:0/327947417 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f79e80186c0 0x7f79e801abc0 unknown :-1 s=CLOSED pgs=1243 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:23.278 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:23.276 7f7a08842700 1 --2- 172.21.2.112:0/327947417 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7a041b0250 0x7f7a041b13e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:23.278 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:23.276 7f7a08842700 1 --2- 172.21.2.112:0/327947417 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7a0419e9b0 0x7f7a041b0dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:23.278 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:23.276 7f7a08842700 1 --2- 172.21.2.112:0/327947417 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7a0419e5b0 0x7f7a041afc30 unknown :-1 s=CLOSED pgs=2728 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:23.278 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:23.276 7f7a08842700 1 -- 172.21.2.112:0/327947417 >> 172.21.2.112:0/327947417 conn(0x7f7a0419f940 msgr2=0x7f7a0418c290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:23.278 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:23.276 7f7a08842700 1 -- 172.21.2.112:0/327947417 shutdown_connections 2021-02-10T11:45:23.279 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:23.276 7f7a08842700 1 -- 172.21.2.112:0/327947417 wait complete. 2021-02-10T11:45:23.286 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2021-02-10T11:45:23.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:23.701 7f270a0dc700 1 Processor -- start 2021-02-10T11:45:23.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:23.701 7f270a0dc700 1 -- start start 2021-02-10T11:45:23.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:23.702 7f270a0dc700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f27040d1880 0x7f27041b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:23.721 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:23.702 7f270a0dc700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27040fe1d0 con 0x7f27040cff80 2021-02-10T11:45:23.721 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:23.702 7f270a0dc700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2704109a70 con 0x7f27041b0af0 2021-02-10T11:45:23.721 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:23.702 7f270a0dc700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2704088070 con 0x7f27040d1880 2021-02-10T11:45:23.721 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:23.702 7f2703fff700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f27040cff80 0x7f2704109140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:58094/0 (socket says 172.21.2.112:58094) 2021-02-10T11:45:23.721 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:23.702 7f2703fff700 1 -- 172.21.2.112:0/621926204 learned_addr learned my addr 172.21.2.112:0/621926204 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:45:23.721 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:23.702 7f27027fc700 1 -- 172.21.2.112:0/621926204 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1263536051 0 0) 0x7f26fc001500 con 0x7f27040cff80 2021-02-10T11:45:23.721 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:23.702 7f27027fc700 1 -- 172.21.2.112:0/621926204 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26f0002fe0 con 0x7f27040cff80 2021-02-10T11:45:23.721 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:23.702 7f27037fe700 1 --2- 172.21.2.112:0/621926204 >> v2:172.21.2.120:3300/0 conn(0x7f27040d1880 0x7f27041b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:23.721 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:23.702 7f27027fc700 1 -- 172.21.2.112:0/621926204 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4283507923 0 0) 0x7f26fc001460 con 0x7f27040cff80 2021-02-10T11:45:23.721 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:23.703 7f27027fc700 1 -- 172.21.2.112:0/621926204 >> v2:172.21.2.120:3300/0 conn(0x7f27040d1880 msgr2=0x7f27041b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:23.721 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:23.703 7f27027fc700 1 --2- 172.21.2.112:0/621926204 >> v2:172.21.2.120:3300/0 conn(0x7f27040d1880 0x7f27041b04d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:23.721 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:23.703 7f27027fc700 1 -- 172.21.2.112:0/621926204 >> v1:172.21.2.120:6789/0 conn(0x7f27041b0af0 legacy=0x7f2704108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:23.721 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:23.703 7f27027fc700 1 -- 172.21.2.112:0/621926204 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2704088260 con 0x7f27040cff80 2021-02-10T11:45:23.721 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:23.703 7f27027fc700 1 -- 172.21.2.112:0/621926204 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f26fc001980 con 0x7f27040cff80 2021-02-10T11:45:23.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:23.703 7f27027fc700 1 -- 172.21.2.112:0/621926204 >> v1:172.21.2.112:6789/0 conn(0x7f27040cff80 legacy=0x7f2704109140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:23.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:23.703 7f27027fc700 1 --2- 172.21.2.112:0/621926204 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f26f0003b70 0x7f26f0005f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:23.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:23.703 7f27027fc700 1 --2- 172.21.2.112:0/621926204 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f26f0006800 0x7f26f0008d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:23.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:23.703 7f27027fc700 1 --2- 172.21.2.112:0/621926204 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f26f0009620 0x7f26f000bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:23.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:23.703 7f27027fc700 1 -- 172.21.2.112:0/621926204 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26f000c390 con 0x7f26f0003b70 2021-02-10T11:45:23.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:23.703 7f27027fc700 1 -- 172.21.2.112:0/621926204 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26f000c5b0 con 0x7f26f0009620 2021-02-10T11:45:23.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:23.703 7f27027fc700 1 -- 172.21.2.112:0/621926204 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26f000c7d0 con 0x7f26f0006800 2021-02-10T11:45:23.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:23.703 7f27037fe700 1 --2- 172.21.2.112:0/621926204 >> v2:172.21.2.120:3300/0 conn(0x7f27040d1880 0x7f27041b04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:45:23.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:23.703 7f27037fe700 1 --2- 172.21.2.112:0/621926204 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f26f0003b70 0x7f26f0005f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:23.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:23.703 7f2702ffd700 1 --2- 172.21.2.112:0/621926204 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f26f0006800 0x7f26f0008d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:23.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:23.703 7f27037fe700 1 -- 172.21.2.112:0/621926204 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f26f0006800 msgr2=0x7f26f0008d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:23.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:23.703 7f2703fff700 1 --2- 172.21.2.112:0/621926204 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f26f0009620 0x7f26f000bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:23.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:23.703 7f27037fe700 1 --2- 172.21.2.112:0/621926204 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f26f0006800 0x7f26f0008d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:23.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:23.703 7f27037fe700 1 -- 172.21.2.112:0/621926204 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f26f0009620 msgr2=0x7f26f000bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:23.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:23.703 7f27037fe700 1 --2- 172.21.2.112:0/621926204 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f26f0009620 0x7f26f000bb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:23.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:23.703 7f27037fe700 1 -- 172.21.2.112:0/621926204 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26f000ca70 con 0x7f26f0003b70 2021-02-10T11:45:23.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:23.703 7f2703fff700 1 --2- 172.21.2.112:0/621926204 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f26f0009620 0x7f26f000bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:45:23.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:23.703 7f2703fff700 1 -- 172.21.2.112:0/621926204 reap_dead start 2021-02-10T11:45:23.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:23.703 7f27037fe700 1 --2- 172.21.2.112:0/621926204 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f26f0003b70 0x7f26f0005f80 secure :-1 s=READY pgs=2730 cs=0 l=1 rev1=1 rx=0x7f26ec04d9c0 tx=0x7f26ec04dd60).ready entity=mon.0 client_cookie=b48fe0905183a1ac server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:23.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:23.703 7f27027fc700 1 -- 172.21.2.112:0/621926204 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f26ec000c90 con 0x7f26f0003b70 2021-02-10T11:45:23.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:23.703 7f27027fc700 1 -- 172.21.2.112:0/621926204 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26ec002030 con 0x7f26f0003b70 2021-02-10T11:45:23.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:23.703 7f27027fc700 1 -- 172.21.2.112:0/621926204 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f26ec002510 con 0x7f26f0003b70 2021-02-10T11:45:23.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:23.703 7f270a0dc700 1 -- 172.21.2.112:0/621926204 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f26f0003b70 msgr2=0x7f26f0005f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:23.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:23.703 7f270a0dc700 1 --2- 172.21.2.112:0/621926204 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f26f0003b70 0x7f26f0005f80 secure :-1 s=READY pgs=2730 cs=0 l=1 rev1=1 rx=0x7f26ec04d9c0 tx=0x7f26ec04dd60).stop 2021-02-10T11:45:23.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:23.704 7f270a0dc700 1 -- 172.21.2.112:0/621926204 shutdown_connections 2021-02-10T11:45:23.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:23.704 7f270a0dc700 1 --2- 172.21.2.112:0/621926204 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f26f0003b70 0x7f26f0005f80 unknown :-1 s=CLOSED pgs=2730 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:23.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:23.704 7f270a0dc700 1 -- 172.21.2.112:0/621926204 >> 172.21.2.112:0/621926204 conn(0x7f27040ce430 msgr2=0x7f27040d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:23.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:23.704 7f270a0dc700 1 -- 172.21.2.112:0/621926204 shutdown_connections 2021-02-10T11:45:23.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:23.704 7f270a0dc700 1 -- 172.21.2.112:0/621926204 wait complete. 2021-02-10T11:45:23.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:23.719 7f270a0dc700 1 Processor -- start 2021-02-10T11:45:23.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:23.719 7f270a0dc700 1 -- start start 2021-02-10T11:45:23.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:23.719 7f270a0dc700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2704188400 0x7f2704188800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:23.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:23.719 7f270a0dc700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2704188e20 0x7f2704189220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:23.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:23.719 7f270a0dc700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f270409cbc0 0x7f270409cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:23.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:23.719 7f270a0dc700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2704187e80 con 0x7f2704188e20 2021-02-10T11:45:23.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:23.719 7f270a0dc700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27040b27c0 con 0x7f2704188400 2021-02-10T11:45:23.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:23.719 7f270a0dc700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27040b29a0 con 0x7f270409cbc0 2021-02-10T11:45:23.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:23.720 7f2702ffd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2704188e20 0x7f2704189220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:23.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:23.720 7f2702ffd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2704188e20 0x7f2704189220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:58644/0 (socket says 172.21.2.112:58644) 2021-02-10T11:45:23.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:23.720 7f2702ffd700 1 -- 172.21.2.112:0/427136091 learned_addr learned my addr 172.21.2.112:0/427136091 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:45:23.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:23.720 7f2702ffd700 1 -- 172.21.2.112:0/427136091 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f270409cbc0 msgr2=0x7f270409cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:23.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:23.720 7f2702ffd700 1 --2- 172.21.2.112:0/427136091 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f270409cbc0 0x7f270409cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:23.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:23.720 7f2702ffd700 1 -- 172.21.2.112:0/427136091 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2704188400 msgr2=0x7f2704188800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:23.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:23.720 7f2702ffd700 1 --2- 172.21.2.112:0/427136091 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2704188400 0x7f2704188800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:23.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:23.720 7f2702ffd700 1 -- 172.21.2.112:0/427136091 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f270409d5e0 con 0x7f2704188e20 2021-02-10T11:45:23.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:23.720 7f2702ffd700 1 --2- 172.21.2.112:0/427136091 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2704188e20 0x7f2704189220 secure :-1 s=READY pgs=2731 cs=0 l=1 rev1=1 rx=0x7f26f400b650 tx=0x7f26f400b9f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:23.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:23.720 7f26fbfff700 1 -- 172.21.2.112:0/427136091 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f26f400bde0 con 0x7f2704188e20 2021-02-10T11:45:23.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:23.720 7f26fbfff700 1 -- 172.21.2.112:0/427136091 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26f4006e20 con 0x7f2704188e20 2021-02-10T11:45:23.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:23.720 7f26fbfff700 1 -- 172.21.2.112:0/427136091 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f26f40072e0 con 0x7f2704188e20 2021-02-10T11:45:23.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:23.720 7f270a0dc700 1 -- 172.21.2.112:0/427136091 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f270409d5e0 con 0x7f2704188e20 2021-02-10T11:45:23.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:23.720 7f270a0dc700 1 -- 172.21.2.112:0/427136091 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f270409db70 con 0x7f2704188e20 2021-02-10T11:45:23.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:23.721 7f26fbfff700 1 -- 172.21.2.112:0/427136091 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f26f4000d30 con 0x7f2704188e20 2021-02-10T11:45:23.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:23.721 7f26fbfff700 1 --2- 172.21.2.112:0/427136091 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f26e80186c0 0x7f26e801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:23.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:23.721 7f26fbfff700 1 -- 172.21.2.112:0/427136091 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 5221+0+0 (secure 0 0 0) 0x7f26f40323f0 con 0x7f2704188e20 2021-02-10T11:45:23.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:23.721 7f27037fe700 1 --2- 172.21.2.112:0/427136091 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f26e80186c0 0x7f26e801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:23.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:23.722 7f27037fe700 1 --2- 172.21.2.112:0/427136091 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f26e80186c0 0x7f26e801abc0 crc :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:23.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:23.751 7f270a0dc700 1 -- 172.21.2.112:0/427136091 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2704082a70 con 0x7f2704188e20 2021-02-10T11:45:23.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:23.754 7f26fbfff700 1 -- 172.21.2.112:0/427136091 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f26f402ced0 con 0x7f2704188e20 2021-02-10T11:45:23.888 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:23.887 7f270a0dc700 1 -- 172.21.2.112:0/427136091 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f2704082a70 con 0x7f2704188e20 2021-02-10T11:45:24.279 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:24.277 7f26fbfff700 1 -- 172.21.2.112:0/427136091 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v154) v1 ==== 173+0+0 (secure 0 0 0) 0x7f26f402ced0 con 0x7f2704188e20 2021-02-10T11:45:24.303 INFO:teuthology.orchestra.run.gibba012.stderr:pool 'cephfs_data' removed 2021-02-10T11:45:24.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:24.303 7f270a0dc700 1 -- 172.21.2.112:0/427136091 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f26e80186c0 msgr2=0x7f26e801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:24.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:24.303 7f270a0dc700 1 --2- 172.21.2.112:0/427136091 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f26e80186c0 0x7f26e801abc0 crc :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:24.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:24.303 7f270a0dc700 1 -- 172.21.2.112:0/427136091 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2704188e20 msgr2=0x7f2704189220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:24.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:24.303 7f270a0dc700 1 --2- 172.21.2.112:0/427136091 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2704188e20 0x7f2704189220 secure :-1 s=READY pgs=2731 cs=0 l=1 rev1=1 rx=0x7f26f400b650 tx=0x7f26f400b9f0).stop 2021-02-10T11:45:24.306 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:24.304 7f270a0dc700 1 -- 172.21.2.112:0/427136091 shutdown_connections 2021-02-10T11:45:24.306 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:24.304 7f270a0dc700 1 --2- 172.21.2.112:0/427136091 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f26e80186c0 0x7f26e801abc0 unknown :-1 s=CLOSED pgs=1244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:24.306 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:24.304 7f270a0dc700 1 --2- 172.21.2.112:0/427136091 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f270409cbc0 0x7f270409cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:24.306 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:24.304 7f270a0dc700 1 --2- 172.21.2.112:0/427136091 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2704188e20 0x7f2704189220 unknown :-1 s=CLOSED pgs=2731 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:24.306 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:24.304 7f270a0dc700 1 --2- 172.21.2.112:0/427136091 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2704188400 0x7f2704188800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:24.307 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:24.304 7f270a0dc700 1 -- 172.21.2.112:0/427136091 >> 172.21.2.112:0/427136091 conn(0x7f27040c2cd0 msgr2=0x7f27040b5950 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:24.307 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:24.304 7f270a0dc700 1 -- 172.21.2.112:0/427136091 shutdown_connections 2021-02-10T11:45:24.307 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:24.304 7f270a0dc700 1 -- 172.21.2.112:0/427136091 wait complete. 2021-02-10T11:45:24.314 INFO:tasks.ceph.mds.a:Restarting daemon 2021-02-10T11:45:24.315 INFO:tasks.ceph.mds.a:Stopping old one... 2021-02-10T11:45:24.315 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2021-02-10T11:45:24.315 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:45:24.315 INFO:tasks.ceph.mds.b:Restarting daemon 2021-02-10T11:45:24.315 INFO:tasks.ceph.mds.b:Stopping old one... 2021-02-10T11:45:24.316 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2021-02-10T11:45:24.316 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:45:24.321 INFO:tasks.ceph.mds.a:Stopped 2021-02-10T11:45:24.321 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2021-02-10T11:45:24.322 INFO:tasks.ceph.mds.b:Stopped 2021-02-10T11:45:24.322 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2021-02-10T11:45:24.363 INFO:tasks.ceph.mds.a:Started 2021-02-10T11:45:24.363 DEBUG:teuthology.parallel:result is None 2021-02-10T11:45:24.364 INFO:tasks.ceph.mds.b:Started 2021-02-10T11:45:24.364 DEBUG:teuthology.parallel:result is None 2021-02-10T11:45:24.364 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd blacklist clear 2021-02-10T11:45:24.766 INFO:tasks.ceph.mds.b.gibba020.stdout:starting mds.b at 2021-02-10T11:45:24.771 INFO:tasks.ceph.mds.a.gibba012.stdout:starting mds.a at 2021-02-10T11:45:24.790 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:24.772 7fc16017b700 1 Processor -- start 2021-02-10T11:45:24.790 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:24.772 7fc16017b700 1 -- start start 2021-02-10T11:45:24.790 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:24.773 7fc16017b700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fc1580d1870 0x7fc1581b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:24.791 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:24.773 7fc16017b700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fc1581b0a10 0x7fc1580cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:24.791 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:24.773 7fc16017b700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1580f0b10 con 0x7fc1580d1c70 2021-02-10T11:45:24.791 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:24.773 7fc16017b700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1580e5ed0 con 0x7fc1581b0a10 2021-02-10T11:45:24.791 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:24.773 7fc16017b700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1580f0650 con 0x7fc1580d1870 2021-02-10T11:45:24.791 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:24.773 7fc15e798700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fc1580d1c70 0x7fc1580cf950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:58124/0 (socket says 172.21.2.112:58124) 2021-02-10T11:45:24.791 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:24.773 7fc15e798700 1 -- 172.21.2.112:0/2915447172 learned_addr learned my addr 172.21.2.112:0/2915447172 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:45:24.791 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:24.773 7fc15d796700 1 --2- 172.21.2.112:0/2915447172 >> v2:172.21.2.112:3300/0 conn(0x7fc1581b0a10 0x7fc1580cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:24.791 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:24.773 7fc15df97700 1 --2- 172.21.2.112:0/2915447172 >> v2:172.21.2.120:3300/0 conn(0x7fc1580d1870 0x7fc1581b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:24.791 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:24.773 7fc15cf95700 1 -- 172.21.2.112:0/2915447172 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2091825367 0 0) 0x7fc148001180 con 0x7fc1580d1c70 2021-02-10T11:45:24.791 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:24.773 7fc15cf95700 1 -- 172.21.2.112:0/2915447172 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc140002fe0 con 0x7fc1580d1c70 2021-02-10T11:45:24.791 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:24.773 7fc15d796700 1 -- 172.21.2.112:0/2915447172 >> v2:172.21.2.120:3300/0 conn(0x7fc1580d1870 msgr2=0x7fc1581b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:24.791 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:24.773 7fc15d796700 1 --2- 172.21.2.112:0/2915447172 >> v2:172.21.2.120:3300/0 conn(0x7fc1580d1870 0x7fc1581b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:24.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:24.773 7fc15d796700 1 -- 172.21.2.112:0/2915447172 >> v1:172.21.2.112:6789/0 conn(0x7fc1580d1c70 legacy=0x7fc1580cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:24.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:24.774 7fc15d796700 1 -- 172.21.2.112:0/2915447172 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1580f0840 con 0x7fc1581b0a10 2021-02-10T11:45:24.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:24.774 7fc15d796700 1 --2- 172.21.2.112:0/2915447172 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc1581b0a10 0x7fc1580cf330 secure :-1 s=READY pgs=2736 cs=0 l=1 rev1=1 rx=0x7fc14c006650 tx=0x7fc14c009a80).ready entity=mon.0 client_cookie=31cdfee8cefda0bd server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:24.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:24.774 7fc15cf95700 1 -- 172.21.2.112:0/2915447172 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc14c00c0c0 con 0x7fc1581b0a10 2021-02-10T11:45:24.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:24.774 7fc15cf95700 1 -- 172.21.2.112:0/2915447172 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc14c00c340 con 0x7fc1581b0a10 2021-02-10T11:45:24.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:24.774 7fc15cf95700 1 -- 172.21.2.112:0/2915447172 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc14c00c860 con 0x7fc1581b0a10 2021-02-10T11:45:24.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:24.774 7fc16017b700 1 -- 172.21.2.112:0/2915447172 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc1581b0a10 msgr2=0x7fc1580cf330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:24.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:24.774 7fc16017b700 1 --2- 172.21.2.112:0/2915447172 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc1581b0a10 0x7fc1580cf330 secure :-1 s=READY pgs=2736 cs=0 l=1 rev1=1 rx=0x7fc14c006650 tx=0x7fc14c009a80).stop 2021-02-10T11:45:24.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:24.774 7fc16017b700 1 -- 172.21.2.112:0/2915447172 shutdown_connections 2021-02-10T11:45:24.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:24.774 7fc16017b700 1 --2- 172.21.2.112:0/2915447172 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc1581b0a10 0x7fc1580cf330 unknown :-1 s=CLOSED pgs=2736 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:24.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:24.774 7fc16017b700 1 --2- 172.21.2.112:0/2915447172 >> v2:172.21.2.120:3300/0 conn(0x7fc1580d1870 0x7fc1581b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:24.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:24.774 7fc16017b700 1 -- 172.21.2.112:0/2915447172 >> 172.21.2.112:0/2915447172 conn(0x7fc1580ce450 msgr2=0x7fc1580d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:24.793 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:24.775 7fc16017b700 1 -- 172.21.2.112:0/2915447172 shutdown_connections 2021-02-10T11:45:24.793 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:24.775 7fc16017b700 1 -- 172.21.2.112:0/2915447172 wait complete. 2021-02-10T11:45:24.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:24.789 7fc16017b700 1 Processor -- start 2021-02-10T11:45:24.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:24.789 7fc16017b700 1 -- start start 2021-02-10T11:45:24.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:24.790 7fc16017b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc1580ebe80 0x7fc1580ec280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:24.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:24.790 7fc16017b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc1580c29e0 0x7fc1580c2de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:24.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:24.790 7fc16017b700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc1580c3400 0x7fc1580ed050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:24.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:24.790 7fc16017b700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1580ecad0 con 0x7fc1580ebe80 2021-02-10T11:45:24.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:24.790 7fc16017b700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1580c3800 con 0x7fc1580c3400 2021-02-10T11:45:24.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:24.790 7fc16017b700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1580c3a20 con 0x7fc1580c29e0 2021-02-10T11:45:24.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:24.790 7fc15df97700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc1580ebe80 0x7fc1580ec280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:24.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:24.790 7fc15df97700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc1580ebe80 0x7fc1580ec280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:58666/0 (socket says 172.21.2.112:58666) 2021-02-10T11:45:24.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:24.790 7fc15df97700 1 -- 172.21.2.112:0/1025150219 learned_addr learned my addr 172.21.2.112:0/1025150219 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:45:24.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:24.790 7fc15d796700 1 --2- 172.21.2.112:0/1025150219 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc1580c29e0 0x7fc1580c2de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:24.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:24.790 7fc15e798700 1 --2- 172.21.2.112:0/1025150219 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc1580c3400 0x7fc1580ed050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:24.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:24.790 7fc15d796700 1 -- 172.21.2.112:0/1025150219 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc1580c3400 msgr2=0x7fc1580ed050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:24.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:24.790 7fc15d796700 1 --2- 172.21.2.112:0/1025150219 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc1580c3400 0x7fc1580ed050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:24.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:24.790 7fc15d796700 1 -- 172.21.2.112:0/1025150219 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc1580ebe80 msgr2=0x7fc1580ec280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:24.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:24.790 7fc15d796700 1 --2- 172.21.2.112:0/1025150219 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc1580ebe80 0x7fc1580ec280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:24.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:24.790 7fc15d796700 1 -- 172.21.2.112:0/1025150219 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1580ed670 con 0x7fc1580c29e0 2021-02-10T11:45:24.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:24.790 7fc15e798700 1 --2- 172.21.2.112:0/1025150219 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc1580c3400 0x7fc1580ed050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:45:24.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:24.791 7fc15d796700 1 --2- 172.21.2.112:0/1025150219 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc1580c29e0 0x7fc1580c2de0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0x7fc14c0091e0 tx=0x7fc14c00b890).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:24.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:24.791 7fc1567fc700 1 -- 172.21.2.112:0/1025150219 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc14c006030 con 0x7fc1580c29e0 2021-02-10T11:45:24.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:24.791 7fc1567fc700 1 -- 172.21.2.112:0/1025150219 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc14c00bc80 con 0x7fc1580c29e0 2021-02-10T11:45:24.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:24.791 7fc1567fc700 1 -- 172.21.2.112:0/1025150219 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc14c0054d0 con 0x7fc1580c29e0 2021-02-10T11:45:24.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:24.791 7fc16017b700 1 -- 172.21.2.112:0/1025150219 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1580ed670 con 0x7fc1580c29e0 2021-02-10T11:45:24.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:24.791 7fc16017b700 1 -- 172.21.2.112:0/1025150219 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1580ed670 con 0x7fc1580c29e0 2021-02-10T11:45:24.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:24.791 7fc1567fc700 1 -- 172.21.2.112:0/1025150219 <== mon.2 v2:172.21.2.120:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc14c0052e0 con 0x7fc1580c29e0 2021-02-10T11:45:24.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:24.791 7fc1567fc700 1 --2- 172.21.2.112:0/1025150219 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc1380186c0 0x7fc13801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:24.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:24.791 7fc1567fc700 1 -- 172.21.2.112:0/1025150219 <== mon.2 v2:172.21.2.120:3301/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 4839+0+0 (secure 0 0 0) 0x7fc14c0322e0 con 0x7fc1580c29e0 2021-02-10T11:45:24.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:24.792 7fc15df97700 1 --2- 172.21.2.112:0/1025150219 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc1380186c0 0x7fc13801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:24.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:24.792 7fc15df97700 1 --2- 172.21.2.112:0/1025150219 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc1380186c0 0x7fc13801abc0 crc :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:24.824 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:24.822 7fc16017b700 1 -- 172.21.2.112:0/1025150219 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1580ebc50 con 0x7fc1580c29e0 2021-02-10T11:45:24.826 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:24.825 7fc1567fc700 1 -- 172.21.2.112:0/1025150219 <== mon.2 v2:172.21.2.120:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc14c0067a0 con 0x7fc1580c29e0 2021-02-10T11:45:24.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:24.964 7fc16017b700 1 -- 172.21.2.112:0/1025150219 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7fc1580f0b10 con 0x7fc1580c29e0 2021-02-10T11:45:25.315 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:25.314 7fc1567fc700 1 -- 172.21.2.112:0/1025150219 <== mon.2 v2:172.21.2.120:3301/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v155) v1 ==== 97+0+0 (secure 0 0 0) 0x7fc14c0052e0 con 0x7fc1580c29e0 2021-02-10T11:45:25.331 INFO:teuthology.orchestra.run.gibba012.stderr: removed all blacklist entries 2021-02-10T11:45:25.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:25.332 7fc16017b700 1 -- 172.21.2.112:0/1025150219 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc1380186c0 msgr2=0x7fc13801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:25.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:25.332 7fc16017b700 1 --2- 172.21.2.112:0/1025150219 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc1380186c0 0x7fc13801abc0 crc :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:25.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:25.332 7fc16017b700 1 -- 172.21.2.112:0/1025150219 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc1580c29e0 msgr2=0x7fc1580c2de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:25.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:25.332 7fc16017b700 1 --2- 172.21.2.112:0/1025150219 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc1580c29e0 0x7fc1580c2de0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0x7fc14c0091e0 tx=0x7fc14c00b890).stop 2021-02-10T11:45:25.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:25.332 7fc16017b700 1 -- 172.21.2.112:0/1025150219 shutdown_connections 2021-02-10T11:45:25.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:25.332 7fc16017b700 1 --2- 172.21.2.112:0/1025150219 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc1380186c0 0x7fc13801abc0 unknown :-1 s=CLOSED pgs=1245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:25.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:25.332 7fc16017b700 1 --2- 172.21.2.112:0/1025150219 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc1580c3400 0x7fc1580ed050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:25.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:25.332 7fc16017b700 1 --2- 172.21.2.112:0/1025150219 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc1580c29e0 0x7fc1580c2de0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:25.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:25.332 7fc16017b700 1 --2- 172.21.2.112:0/1025150219 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc1580ebe80 0x7fc1580ec280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:25.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:25.332 7fc16017b700 1 -- 172.21.2.112:0/1025150219 >> 172.21.2.112:0/1025150219 conn(0x7fc1580b9870 msgr2=0x7fc158094c40 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:25.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:25.332 7fc16017b700 1 -- 172.21.2.112:0/1025150219 shutdown_connections 2021-02-10T11:45:25.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:25.332 7fc16017b700 1 -- 172.21.2.112:0/1025150219 wait complete. 2021-02-10T11:45:25.342 INFO:tasks.cephfs.cephfs_test_case:['0', '1', '2', '3'] 2021-02-10T11:45:25.342 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth ls --format=json-pretty 2021-02-10T11:45:25.531 DEBUG:teuthology.orchestra.run.gibba012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:45:25.534 DEBUG:teuthology.orchestra.run.gibba020:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:45:25.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:25.753 7fcc62e8a700 1 Processor -- start 2021-02-10T11:45:25.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:25.753 7fcc62e8a700 1 -- start start 2021-02-10T11:45:25.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:25.754 7fcc62e8a700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fcc5c0f05f0 0x7fcc5c0f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:25.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:25.754 7fcc62e8a700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc5c1b0c70 con 0x7fcc5c1b0870 2021-02-10T11:45:25.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:25.754 7fcc62e8a700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc5c0f0b50 con 0x7fcc5c1b0470 2021-02-10T11:45:25.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:25.754 7fcc62e8a700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc5c0e5ea0 con 0x7fcc5c0f05f0 2021-02-10T11:45:25.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:25.754 7fcc614a7700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fcc5c0f05f0 0x7fcc5c0f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:25.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:25.754 7fcc614a7700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fcc5c0f05f0 0x7fcc5c0f20e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:58674/0 (socket says 172.21.2.112:58674) 2021-02-10T11:45:25.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:25.754 7fcc614a7700 1 -- 172.21.2.112:0/1031002976 learned_addr learned my addr 172.21.2.112:0/1031002976 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:45:25.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:25.754 7fcc614a7700 1 -- 172.21.2.112:0/1031002976 >> v1:172.21.2.120:6789/0 conn(0x7fcc5c1b0470 legacy=0x7fcc5c0ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:25.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:25.754 7fcc614a7700 1 -- 172.21.2.112:0/1031002976 >> v1:172.21.2.112:6789/0 conn(0x7fcc5c1b0870 legacy=0x7fcc5c0efe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:25.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:25.754 7fcc614a7700 1 -- 172.21.2.112:0/1031002976 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc5c0e52e0 con 0x7fcc5c0f05f0 2021-02-10T11:45:25.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:25.754 7fcc614a7700 1 --2- 172.21.2.112:0/1031002976 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcc5c0f05f0 0x7fcc5c0f20e0 secure :-1 s=READY pgs=2738 cs=0 l=1 rev1=1 rx=0x7fcc54006f00 tx=0x7fcc54009f10).ready entity=mon.0 client_cookie=9f8c4dd0ea62e09d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:25.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:25.754 7fcc5b7fe700 1 -- 172.21.2.112:0/1031002976 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcc5400b3b0 con 0x7fcc5c0f05f0 2021-02-10T11:45:25.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:25.754 7fcc5b7fe700 1 -- 172.21.2.112:0/1031002976 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc5400b730 con 0x7fcc5c0f05f0 2021-02-10T11:45:25.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:25.754 7fcc5b7fe700 1 -- 172.21.2.112:0/1031002976 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcc5400bc10 con 0x7fcc5c0f05f0 2021-02-10T11:45:25.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:25.754 7fcc62e8a700 1 -- 172.21.2.112:0/1031002976 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcc5c0f05f0 msgr2=0x7fcc5c0f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:25.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:25.754 7fcc62e8a700 1 --2- 172.21.2.112:0/1031002976 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcc5c0f05f0 0x7fcc5c0f20e0 secure :-1 s=READY pgs=2738 cs=0 l=1 rev1=1 rx=0x7fcc54006f00 tx=0x7fcc54009f10).stop 2021-02-10T11:45:25.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:25.755 7fcc62e8a700 1 -- 172.21.2.112:0/1031002976 shutdown_connections 2021-02-10T11:45:25.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:25.755 7fcc62e8a700 1 --2- 172.21.2.112:0/1031002976 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcc5c0f05f0 0x7fcc5c0f20e0 unknown :-1 s=CLOSED pgs=2738 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:25.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:25.755 7fcc62e8a700 1 -- 172.21.2.112:0/1031002976 >> 172.21.2.112:0/1031002976 conn(0x7fcc5c0ce3e0 msgr2=0x7fcc5c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:25.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:25.755 7fcc62e8a700 1 -- 172.21.2.112:0/1031002976 shutdown_connections 2021-02-10T11:45:25.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:25.755 7fcc62e8a700 1 -- 172.21.2.112:0/1031002976 wait complete. 2021-02-10T11:45:25.774 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:25.770 7fcc62e8a700 1 Processor -- start 2021-02-10T11:45:25.774 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:25.770 7fcc62e8a700 1 -- start start 2021-02-10T11:45:25.774 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:25.770 7fcc62e8a700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcc5c097170 0x7fcc5c0a71e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:25.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:25.771 7fcc62e8a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcc5c0961b0 0x7fcc5c0965b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:25.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:25.771 7fcc62e8a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcc5c096bd0 0x7fcc5c0a7e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:25.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:25.771 7fcc62e8a700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc5c0a8470 con 0x7fcc5c0961b0 2021-02-10T11:45:25.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:25.771 7fcc62e8a700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc5c0a8650 con 0x7fcc5c097170 2021-02-10T11:45:25.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:25.771 7fcc62e8a700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc5c0a8870 con 0x7fcc5c096bd0 2021-02-10T11:45:25.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:25.771 7fcc5bfff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcc5c0961b0 0x7fcc5c0965b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:25.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:25.771 7fcc5bfff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcc5c0961b0 0x7fcc5c0965b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:58680/0 (socket says 172.21.2.112:58680) 2021-02-10T11:45:25.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:25.771 7fcc5bfff700 1 -- 172.21.2.112:0/614273958 learned_addr learned my addr 172.21.2.112:0/614273958 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:45:25.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:25.771 7fcc5bfff700 1 -- 172.21.2.112:0/614273958 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcc5c096bd0 msgr2=0x7fcc5c0a7e50 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:45:25.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:25.771 7fcc5bfff700 1 --2- 172.21.2.112:0/614273958 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcc5c096bd0 0x7fcc5c0a7e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:25.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:25.771 7fcc5bfff700 1 -- 172.21.2.112:0/614273958 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcc5c097170 msgr2=0x7fcc5c0a71e0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:45:25.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:25.771 7fcc5bfff700 1 --2- 172.21.2.112:0/614273958 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcc5c097170 0x7fcc5c0a71e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:25.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:25.771 7fcc5bfff700 1 -- 172.21.2.112:0/614273958 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc5c0a8a90 con 0x7fcc5c0961b0 2021-02-10T11:45:25.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:25.771 7fcc5bfff700 1 --2- 172.21.2.112:0/614273958 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcc5c0961b0 0x7fcc5c0965b0 secure :-1 s=READY pgs=2739 cs=0 l=1 rev1=1 rx=0x7fcc4c006ed0 tx=0x7fcc4c00b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:25.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:25.771 7fcc597fa700 1 -- 172.21.2.112:0/614273958 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcc4c006020 con 0x7fcc5c0961b0 2021-02-10T11:45:25.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:25.771 7fcc597fa700 1 -- 172.21.2.112:0/614273958 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc4c006320 con 0x7fcc5c0961b0 2021-02-10T11:45:25.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:25.771 7fcc597fa700 1 -- 172.21.2.112:0/614273958 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcc4c005730 con 0x7fcc5c0961b0 2021-02-10T11:45:25.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:25.771 7fcc62e8a700 1 -- 172.21.2.112:0/614273958 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc5c0a8a90 con 0x7fcc5c0961b0 2021-02-10T11:45:25.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:25.771 7fcc62e8a700 1 -- 172.21.2.112:0/614273958 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc5c0a9020 con 0x7fcc5c0961b0 2021-02-10T11:45:25.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:25.772 7fcc597fa700 1 -- 172.21.2.112:0/614273958 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fcc4c006020 con 0x7fcc5c0961b0 2021-02-10T11:45:25.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:25.772 7fcc597fa700 1 --2- 172.21.2.112:0/614273958 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcc400186c0 0x7fcc4001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:25.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:25.772 7fcc597fa700 1 -- 172.21.2.112:0/614273958 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 4753+0+0 (secure 0 0 0) 0x7fcc4c0324d0 con 0x7fcc5c0961b0 2021-02-10T11:45:25.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:25.772 7fcc60ca6700 1 --2- 172.21.2.112:0/614273958 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcc400186c0 0x7fcc4001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:25.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:25.773 7fcc60ca6700 1 --2- 172.21.2.112:0/614273958 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcc400186c0 0x7fcc4001abc0 crc :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:25.805 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:25.804 7fcc62e8a700 1 -- 172.21.2.112:0/614273958 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc5c0a7920 con 0x7fcc5c0961b0 2021-02-10T11:45:25.808 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:25.806 7fcc597fa700 1 -- 172.21.2.112:0/614273958 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fcc4c0059d0 con 0x7fcc5c0961b0 2021-02-10T11:45:25.943 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:25.941 7fcc62e8a700 1 -- 172.21.2.112:0/614273958 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fcc5c0a7920 con 0x7fcc5c0961b0 2021-02-10T11:45:25.943 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:25.942 7fcc597fa700 1 -- 172.21.2.112:0/614273958 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v90) v1 ==== 80+0+4513 (secure 0 0 0) 0x7fcc4c006020 con 0x7fcc5c0961b0 2021-02-10T11:45:25.944 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:45:25.944 INFO:teuthology.orchestra.run.gibba012.stdout:{ 2021-02-10T11:45:25.944 INFO:teuthology.orchestra.run.gibba012.stdout: "auth_dump": [ 2021-02-10T11:45:25.944 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:45:25.944 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "mds.a", 2021-02-10T11:45:25.944 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCKvyNgPkV4AxAAEmVNaghdccP6lWd8hvQnxw==", 2021-02-10T11:45:25.944 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:45:25.944 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:45:25.944 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:45:25.944 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:45:25.945 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:45:25.945 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:45:25.945 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:45:25.945 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:45:25.945 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "mds.b", 2021-02-10T11:45:25.945 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCKvyNguxONLhAAHIpry+R89MAwdhFJyEhVCw==", 2021-02-10T11:45:25.945 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:45:25.945 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:45:25.945 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:45:25.945 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:45:25.945 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:45:25.945 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:45:25.945 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:45:25.946 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:45:25.946 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.0", 2021-02-10T11:45:25.946 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCSvyNgXcKPLhAAP/Bj2F+06DCtPCzQDNq+cQ==", 2021-02-10T11:45:25.946 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:45:25.946 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:45:25.946 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:45:25.946 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:45:25.946 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:45:25.946 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:45:25.946 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:45:25.946 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.1", 2021-02-10T11:45:25.946 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCVvyNghfbcLhAA66PrRwR2M7o6VTF5UhU/Fg==", 2021-02-10T11:45:25.947 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:45:25.947 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:45:25.947 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:45:25.947 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:45:25.947 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:45:25.947 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:45:25.947 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:45:25.947 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.2", 2021-02-10T11:45:25.947 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCYvyNgwiPqLhAA66BVkRsgaUlGpAsQ15Nf8Q==", 2021-02-10T11:45:25.947 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:45:25.947 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:45:25.947 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:45:25.948 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:45:25.948 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:45:25.948 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:45:25.948 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:45:25.948 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.3", 2021-02-10T11:45:25.948 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCbvyNgqBZrLxAAERPe/1L3SXAEzI3d9ReHHg==", 2021-02-10T11:45:25.948 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:45:25.948 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:45:25.948 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:45:25.948 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:45:25.948 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:45:25.948 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:45:25.949 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:45:25.949 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.4", 2021-02-10T11:45:25.949 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCkvyNgDQsxIhAAHmVjVYnx20N4SDmKqvbcuA==", 2021-02-10T11:45:25.949 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:45:25.949 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:45:25.949 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:45:25.949 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:45:25.949 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:45:25.949 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:45:25.949 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:45:25.949 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.5", 2021-02-10T11:45:25.950 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCnvyNg8ITFIhAADd4aKybAV7btoGPx3kCg3A==", 2021-02-10T11:45:25.950 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:45:25.950 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:45:25.950 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:45:25.950 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:45:25.950 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:45:25.950 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:45:25.950 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:45:25.950 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.6", 2021-02-10T11:45:25.950 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCqvyNgxOhDIxAAzXixEa/nlDc1xXrg0aeLwg==", 2021-02-10T11:45:25.951 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:45:25.951 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:45:25.951 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:45:25.951 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:45:25.951 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:45:25.951 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:45:25.951 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:45:25.951 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.7", 2021-02-10T11:45:25.951 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCtvyNgxSqGIxAACxJQlwP7DpiVVCZHA3zn1A==", 2021-02-10T11:45:25.951 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:45:25.951 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:45:25.951 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:45:25.951 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:45:25.952 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:45:25.952 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:45:25.952 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:45:25.952 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.0", 2021-02-10T11:45:25.952 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCLvyNgDkNiHRAAygU8Za4PBusTqCgwAYWTiQ==", 2021-02-10T11:45:25.952 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:45:25.952 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:45:25.952 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow r", 2021-02-10T11:45:25.952 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:45:25.952 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:45:25.952 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:45:25.952 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:45:25.952 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.1", 2021-02-10T11:45:25.953 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCLvyNgxmLqMxAA1MlSDJ9LH9u4f2sL7mm8/Q==", 2021-02-10T11:45:25.953 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:45:25.953 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:45:25.953 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow r", 2021-02-10T11:45:25.953 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:45:25.953 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:45:25.953 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:45:25.953 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:45:25.953 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.2", 2021-02-10T11:45:25.953 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCMvyNgp8veDhAAniSVHYPtwwTog/pjbaRQmw==", 2021-02-10T11:45:25.953 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:45:25.953 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:45:25.953 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow r", 2021-02-10T11:45:25.954 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:45:25.954 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:45:25.954 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:45:25.954 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:45:25.954 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.3", 2021-02-10T11:45:25.954 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCMvyNgQEVoJRAApHcWdm1sldIxu+heyD4HdQ==", 2021-02-10T11:45:25.954 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:45:25.954 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:45:25.954 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow r", 2021-02-10T11:45:25.954 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:45:25.954 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:45:25.954 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:45:25.954 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:45:25.955 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.admin", 2021-02-10T11:45:25.955 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCGvyNg7jjJBRAAtot+ypUZOiTo1GbY+oVhVw==", 2021-02-10T11:45:25.955 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:45:25.955 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow *", 2021-02-10T11:45:25.955 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:45:25.955 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:45:25.955 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:45:25.955 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:45:25.955 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:45:25.955 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:45:25.955 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "mgr.x", 2021-02-10T11:45:25.955 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCJvyNgTPVwKBAAjduiBE8+RSXBv82tdPecWg==", 2021-02-10T11:45:25.955 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:45:25.955 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow *", 2021-02-10T11:45:25.956 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow profile mgr", 2021-02-10T11:45:25.956 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:45:25.956 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:45:25.956 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:45:25.956 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:45:25.956 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "mgr.y", 2021-02-10T11:45:25.956 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCJvyNgQ1tTDhAASwBssDy+Rz31ujw8fNlWow==", 2021-02-10T11:45:25.956 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:45:25.956 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow *", 2021-02-10T11:45:25.956 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow profile mgr", 2021-02-10T11:45:25.956 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:45:25.956 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:45:25.956 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:45:25.956 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:45:25.957 INFO:teuthology.orchestra.run.gibba012.stdout:} 2021-02-10T11:45:25.958 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:25.943 7fcc62e8a700 1 -- 172.21.2.112:0/614273958 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcc400186c0 msgr2=0x7fcc4001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:25.958 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:25.943 7fcc62e8a700 1 --2- 172.21.2.112:0/614273958 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcc400186c0 0x7fcc4001abc0 crc :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:25.958 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:25.943 7fcc62e8a700 1 -- 172.21.2.112:0/614273958 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcc5c0961b0 msgr2=0x7fcc5c0965b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:25.958 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:25.943 7fcc62e8a700 1 --2- 172.21.2.112:0/614273958 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcc5c0961b0 0x7fcc5c0965b0 secure :-1 s=READY pgs=2739 cs=0 l=1 rev1=1 rx=0x7fcc4c006ed0 tx=0x7fcc4c00b960).stop 2021-02-10T11:45:25.958 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:25.944 7fcc62e8a700 1 -- 172.21.2.112:0/614273958 shutdown_connections 2021-02-10T11:45:25.958 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:25.944 7fcc62e8a700 1 --2- 172.21.2.112:0/614273958 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcc400186c0 0x7fcc4001abc0 unknown :-1 s=CLOSED pgs=1246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:25.958 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:25.944 7fcc62e8a700 1 --2- 172.21.2.112:0/614273958 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcc5c096bd0 0x7fcc5c0a7e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:25.958 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:25.944 7fcc62e8a700 1 --2- 172.21.2.112:0/614273958 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcc5c0961b0 0x7fcc5c0965b0 unknown :-1 s=CLOSED pgs=2739 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:25.958 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:25.944 7fcc62e8a700 1 --2- 172.21.2.112:0/614273958 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcc5c097170 0x7fcc5c0a71e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:25.958 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:25.944 7fcc62e8a700 1 -- 172.21.2.112:0/614273958 >> 172.21.2.112:0/614273958 conn(0x7fcc5c0c3bd0 msgr2=0x7fcc5c089210 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:25.958 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:25.944 7fcc62e8a700 1 -- 172.21.2.112:0/614273958 shutdown_connections 2021-02-10T11:45:25.958 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:25.944 7fcc62e8a700 1 -- 172.21.2.112:0/614273958 wait complete. 2021-02-10T11:45:25.959 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:45:26.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.369 7fc2367a0700 1 Processor -- start 2021-02-10T11:45:26.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.369 7fc2367a0700 1 -- start start 2021-02-10T11:45:26.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.369 7fc2367a0700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc2300d1870 0x7fc2301b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:26.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.369 7fc2367a0700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fc2301b0a10 0x7fc2300cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:26.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.369 7fc2367a0700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2300f0b10 con 0x7fc2300d1c70 2021-02-10T11:45:26.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.369 7fc2367a0700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2300fe230 con 0x7fc2301b0a10 2021-02-10T11:45:26.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.369 7fc2367a0700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2300e5ed0 con 0x7fc2300d1870 2021-02-10T11:45:26.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.370 7fc22f7fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fc2301b0a10 0x7fc2300cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:26.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.370 7fc22f7fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fc2301b0a10 0x7fc2300cf330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:58684/0 (socket says 172.21.2.112:58684) 2021-02-10T11:45:26.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.370 7fc22f7fe700 1 -- 172.21.2.112:0/2988988147 learned_addr learned my addr 172.21.2.112:0/2988988147 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:45:26.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.370 7fc234dbd700 1 --1- 172.21.2.112:0/2988988147 >> v1:172.21.2.112:6789/0 conn(0x7fc2300d1c70 0x7fc2300cf950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:58146/0 (socket says 172.21.2.112:58146) 2021-02-10T11:45:26.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.370 7fc22f7fe700 1 -- 172.21.2.112:0/2988988147 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc2300d1870 msgr2=0x7fc2301b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:26.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.370 7fc22f7fe700 1 --2- 172.21.2.112:0/2988988147 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc2300d1870 0x7fc2301b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:26.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.370 7fc22f7fe700 1 -- 172.21.2.112:0/2988988147 >> v1:172.21.2.112:6789/0 conn(0x7fc2300d1c70 legacy=0x7fc2300cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:26.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.370 7fc22f7fe700 1 -- 172.21.2.112:0/2988988147 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2300f0650 con 0x7fc2301b0a10 2021-02-10T11:45:26.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.370 7fc22f7fe700 1 --2- 172.21.2.112:0/2988988147 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc2301b0a10 0x7fc2300cf330 secure :-1 s=READY pgs=2741 cs=0 l=1 rev1=1 rx=0x7fc218006700 tx=0x7fc218009b80).ready entity=mon.0 client_cookie=a3828be791569c54 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:26.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.370 7fc22effd700 1 -- 172.21.2.112:0/2988988147 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc21800c0c0 con 0x7fc2301b0a10 2021-02-10T11:45:26.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.370 7fc22effd700 1 -- 172.21.2.112:0/2988988147 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc21800c340 con 0x7fc2301b0a10 2021-02-10T11:45:26.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.370 7fc22effd700 1 -- 172.21.2.112:0/2988988147 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc21800c860 con 0x7fc2301b0a10 2021-02-10T11:45:26.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.370 7fc2367a0700 1 -- 172.21.2.112:0/2988988147 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc2301b0a10 msgr2=0x7fc2300cf330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:26.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.370 7fc2367a0700 1 --2- 172.21.2.112:0/2988988147 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc2301b0a10 0x7fc2300cf330 secure :-1 s=READY pgs=2741 cs=0 l=1 rev1=1 rx=0x7fc218006700 tx=0x7fc218009b80).stop 2021-02-10T11:45:26.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.371 7fc2367a0700 1 -- 172.21.2.112:0/2988988147 shutdown_connections 2021-02-10T11:45:26.389 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.371 7fc2367a0700 1 --2- 172.21.2.112:0/2988988147 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc2301b0a10 0x7fc2300cf330 unknown :-1 s=CLOSED pgs=2741 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:26.389 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.371 7fc2367a0700 1 --2- 172.21.2.112:0/2988988147 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc2300d1870 0x7fc2301b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:26.389 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.371 7fc2367a0700 1 -- 172.21.2.112:0/2988988147 >> 172.21.2.112:0/2988988147 conn(0x7fc2300ce450 msgr2=0x7fc2300d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:26.389 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.371 7fc2367a0700 1 -- 172.21.2.112:0/2988988147 shutdown_connections 2021-02-10T11:45:26.389 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.371 7fc2367a0700 1 -- 172.21.2.112:0/2988988147 wait complete. 2021-02-10T11:45:26.391 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.386 7fc2367a0700 1 Processor -- start 2021-02-10T11:45:26.391 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.386 7fc2367a0700 1 -- start start 2021-02-10T11:45:26.391 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.386 7fc2367a0700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc2301b3460 0x7fc2301aaf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:26.391 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.386 7fc2367a0700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc23019b740 0x7fc2301ab560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:26.391 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.386 7fc2367a0700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc2300b7a60 0x7fc2300b7e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:26.391 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.386 7fc2367a0700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2301abc00 con 0x7fc2301b3460 2021-02-10T11:45:26.391 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.386 7fc2367a0700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2300b8480 con 0x7fc23019b740 2021-02-10T11:45:26.391 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.386 7fc2367a0700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2300b86a0 con 0x7fc2300b7a60 2021-02-10T11:45:26.391 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.386 7fc22ffff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc2301b3460 0x7fc2301aaf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:26.391 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.386 7fc22ffff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc2301b3460 0x7fc2301aaf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:58692/0 (socket says 172.21.2.112:58692) 2021-02-10T11:45:26.391 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.386 7fc22ffff700 1 -- 172.21.2.112:0/3354011012 learned_addr learned my addr 172.21.2.112:0/3354011012 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:45:26.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.387 7fc22ffff700 1 -- 172.21.2.112:0/3354011012 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc2300b7a60 msgr2=0x7fc2300b7e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:26.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.387 7fc22ffff700 1 --2- 172.21.2.112:0/3354011012 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc2300b7a60 0x7fc2300b7e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:26.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.387 7fc22ffff700 1 -- 172.21.2.112:0/3354011012 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc23019b740 msgr2=0x7fc2301ab560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:26.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.387 7fc22ffff700 1 --2- 172.21.2.112:0/3354011012 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc23019b740 0x7fc2301ab560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:26.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.387 7fc22ffff700 1 -- 172.21.2.112:0/3354011012 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2301ac9b0 con 0x7fc2301b3460 2021-02-10T11:45:26.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.387 7fc22ffff700 1 --2- 172.21.2.112:0/3354011012 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc2301b3460 0x7fc2301aaf40 secure :-1 s=READY pgs=2742 cs=0 l=1 rev1=1 rx=0x7fc220009980 tx=0x7fc22000bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:26.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.387 7fc22cff9700 1 -- 172.21.2.112:0/3354011012 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc22000be10 con 0x7fc2301b3460 2021-02-10T11:45:26.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.387 7fc22cff9700 1 -- 172.21.2.112:0/3354011012 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc22000c2f0 con 0x7fc2301b3460 2021-02-10T11:45:26.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.387 7fc22cff9700 1 -- 172.21.2.112:0/3354011012 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc22000c7d0 con 0x7fc2301b3460 2021-02-10T11:45:26.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.387 7fc2367a0700 1 -- 172.21.2.112:0/3354011012 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2301ac9b0 con 0x7fc2301b3460 2021-02-10T11:45:26.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.387 7fc2367a0700 1 -- 172.21.2.112:0/3354011012 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2301acf40 con 0x7fc2301b3460 2021-02-10T11:45:26.393 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.388 7fc22cff9700 1 -- 172.21.2.112:0/3354011012 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc220001010 con 0x7fc2301b3460 2021-02-10T11:45:26.393 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.388 7fc22cff9700 1 --2- 172.21.2.112:0/3354011012 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc2140186c0 0x7fc21401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:26.393 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.388 7fc22cff9700 1 -- 172.21.2.112:0/3354011012 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 4753+0+0 (secure 0 0 0) 0x7fc2200325c0 con 0x7fc2301b3460 2021-02-10T11:45:26.393 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.388 7fc22f7fe700 1 --2- 172.21.2.112:0/3354011012 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc2140186c0 0x7fc21401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:26.393 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.388 7fc22f7fe700 1 --2- 172.21.2.112:0/3354011012 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc2140186c0 0x7fc21401abc0 crc :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:26.420 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.419 7fc2367a0700 1 -- 172.21.2.112:0/3354011012 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2300ee060 con 0x7fc2301b3460 2021-02-10T11:45:26.423 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.421 7fc22cff9700 1 -- 172.21.2.112:0/3354011012 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc220000d30 con 0x7fc2301b3460 2021-02-10T11:45:26.549 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.548 7fc2367a0700 1 -- 172.21.2.112:0/3354011012 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc2300ce450 con 0x7fc2301b3460 2021-02-10T11:45:26.550 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.548 7fc22cff9700 1 -- 172.21.2.112:0/3354011012 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v155) v1 ==== 74+0+11154 (secure 0 0 0) 0x7fc220001010 con 0x7fc2301b3460 2021-02-10T11:45:26.550 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:45:26.551 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":155,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:45:25.310287","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:45:26.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.550 7fc2367a0700 1 -- 172.21.2.112:0/3354011012 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc2140186c0 msgr2=0x7fc21401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:26.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.550 7fc2367a0700 1 --2- 172.21.2.112:0/3354011012 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc2140186c0 0x7fc21401abc0 crc :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:26.553 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.550 7fc2367a0700 1 -- 172.21.2.112:0/3354011012 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc2301b3460 msgr2=0x7fc2301aaf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:26.553 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.550 7fc2367a0700 1 --2- 172.21.2.112:0/3354011012 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc2301b3460 0x7fc2301aaf40 secure :-1 s=READY pgs=2742 cs=0 l=1 rev1=1 rx=0x7fc220009980 tx=0x7fc22000bcd0).stop 2021-02-10T11:45:26.553 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.551 7fc2367a0700 1 -- 172.21.2.112:0/3354011012 shutdown_connections 2021-02-10T11:45:26.553 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.551 7fc2367a0700 1 --2- 172.21.2.112:0/3354011012 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc2140186c0 0x7fc21401abc0 unknown :-1 s=CLOSED pgs=1247 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:26.553 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.551 7fc2367a0700 1 --2- 172.21.2.112:0/3354011012 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc23019b740 0x7fc2301ab560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:26.553 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.551 7fc2367a0700 1 --2- 172.21.2.112:0/3354011012 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc2300b7a60 0x7fc2300b7e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:26.553 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.551 7fc2367a0700 1 --2- 172.21.2.112:0/3354011012 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc2301b3460 0x7fc2301aaf40 unknown :-1 s=CLOSED pgs=2742 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:26.553 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.551 7fc2367a0700 1 -- 172.21.2.112:0/3354011012 >> 172.21.2.112:0/3354011012 conn(0x7fc23019c5f0 msgr2=0x7fc2301b38a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:26.553 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.551 7fc2367a0700 1 -- 172.21.2.112:0/3354011012 shutdown_connections 2021-02-10T11:45:26.553 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.551 7fc2367a0700 1 -- 172.21.2.112:0/3354011012 wait complete. 2021-02-10T11:45:26.561 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:13:57.508758', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2021-02-10T11:45:26.561 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:45:26.995 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.977 7f2cef203700 1 Processor -- start 2021-02-10T11:45:26.995 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.977 7f2cef203700 1 -- start start 2021-02-10T11:45:26.995 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.977 7f2cef203700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f2ce80d1c70 0x7f2ce80cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:26.996 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.978 7f2cef203700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2ce80cf580 0x7f2ce80cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:26.996 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.978 7f2cef203700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ce80fe1c0 con 0x7f2ce80d1870 2021-02-10T11:45:26.996 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.978 7f2cef203700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ce81b0cb0 con 0x7f2ce80d1c70 2021-02-10T11:45:26.996 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.978 7f2cef203700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ce8109a60 con 0x7f2ce80cf580 2021-02-10T11:45:26.996 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.978 7f2ced01f700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f2ce80d1870 0x7f2ce81b04c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:58158/0 (socket says 172.21.2.112:58158) 2021-02-10T11:45:26.996 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.978 7f2ced01f700 1 -- 172.21.2.112:0/3868156074 learned_addr learned my addr 172.21.2.112:0/3868156074 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:45:26.996 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.978 7f2cec81e700 1 --2- 172.21.2.112:0/3868156074 >> v2:172.21.2.120:3300/0 conn(0x7f2ce80d1c70 0x7f2ce80cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:26.996 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.978 7f2ced820700 1 --2- 172.21.2.112:0/3868156074 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2ce80cf580 0x7f2ce80cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:26.996 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.978 7f2cdffff700 1 -- 172.21.2.112:0/3868156074 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3101084365 0 0) 0x7f2ce4001180 con 0x7f2ce80d1870 2021-02-10T11:45:26.996 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.978 7f2cdffff700 1 -- 172.21.2.112:0/3868156074 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2cd4002fe0 con 0x7f2ce80d1870 2021-02-10T11:45:26.996 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.978 7f2cec81e700 1 -- 172.21.2.112:0/3868156074 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2ce80cf580 msgr2=0x7f2ce80cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:26.996 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.978 7f2cec81e700 1 --2- 172.21.2.112:0/3868156074 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2ce80cf580 0x7f2ce80cf980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:26.996 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.978 7f2cec81e700 1 -- 172.21.2.112:0/3868156074 >> v1:172.21.2.112:6789/0 conn(0x7f2ce80d1870 legacy=0x7f2ce81b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:26.997 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.978 7f2cec81e700 1 -- 172.21.2.112:0/3868156074 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ce8088070 con 0x7f2ce80d1c70 2021-02-10T11:45:26.997 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.978 7f2cec81e700 1 --2- 172.21.2.112:0/3868156074 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2ce80d1c70 0x7f2ce80cef60 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 rx=0x7f2cd8006650 tx=0x7f2cd8009a80).ready entity=mon.1 client_cookie=8e8d99e62ab18c40 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:26.997 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.979 7f2cdffff700 1 -- 172.21.2.112:0/3868156074 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2cd80009b0 con 0x7f2ce80d1c70 2021-02-10T11:45:26.997 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.979 7f2cdffff700 1 -- 172.21.2.112:0/3868156074 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cd800c0c0 con 0x7f2ce80d1c70 2021-02-10T11:45:26.997 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.979 7f2cef203700 1 -- 172.21.2.112:0/3868156074 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2ce80d1c70 msgr2=0x7f2ce80cef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:26.997 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.979 7f2cef203700 1 --2- 172.21.2.112:0/3868156074 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2ce80d1c70 0x7f2ce80cef60 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 rx=0x7f2cd8006650 tx=0x7f2cd8009a80).stop 2021-02-10T11:45:26.997 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.979 7f2cef203700 1 -- 172.21.2.112:0/3868156074 shutdown_connections 2021-02-10T11:45:26.997 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.979 7f2cef203700 1 --2- 172.21.2.112:0/3868156074 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2ce80cf580 0x7f2ce80cf980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:26.997 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.979 7f2cef203700 1 --2- 172.21.2.112:0/3868156074 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2ce80d1c70 0x7f2ce80cef60 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:26.997 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.979 7f2cef203700 1 -- 172.21.2.112:0/3868156074 >> 172.21.2.112:0/3868156074 conn(0x7f2ce80ce420 msgr2=0x7f2ce80d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:26.997 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.979 7f2cef203700 1 -- 172.21.2.112:0/3868156074 shutdown_connections 2021-02-10T11:45:26.997 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.979 7f2cef203700 1 -- 172.21.2.112:0/3868156074 wait complete. 2021-02-10T11:45:26.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.994 7f2cef203700 1 Processor -- start 2021-02-10T11:45:26.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.994 7f2cef203700 1 -- start start 2021-02-10T11:45:26.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.994 7f2cef203700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2ce80b04d0 0x7f2ce81a7a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:26.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.994 7f2cef203700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2ce81a8090 0x7f2ce80c23a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:26.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.994 7f2cef203700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2ce808d680 0x7f2ce80c29c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:26.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.994 7f2cef203700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ce80b0a00 con 0x7f2ce81a8090 2021-02-10T11:45:26.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.994 7f2cef203700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ce80c2fe0 con 0x7f2ce80b04d0 2021-02-10T11:45:26.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.994 7f2cef203700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ce80c3200 con 0x7f2ce808d680 2021-02-10T11:45:27.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.995 7f2cec81e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2ce81a8090 0x7f2ce80c23a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:27.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.995 7f2cec81e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2ce81a8090 0x7f2ce80c23a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:58704/0 (socket says 172.21.2.112:58704) 2021-02-10T11:45:27.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.995 7f2cec81e700 1 -- 172.21.2.112:0/1182409259 learned_addr learned my addr 172.21.2.112:0/1182409259 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:45:27.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.995 7f2ced01f700 1 --2- 172.21.2.112:0/1182409259 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2ce80b04d0 0x7f2ce81a7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:27.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.995 7f2ced820700 1 --2- 172.21.2.112:0/1182409259 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2ce808d680 0x7f2ce80c29c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:27.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.995 7f2cec81e700 1 -- 172.21.2.112:0/1182409259 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2ce808d680 msgr2=0x7f2ce80c29c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:27.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.995 7f2cec81e700 1 --2- 172.21.2.112:0/1182409259 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2ce808d680 0x7f2ce80c29c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:27.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.995 7f2cec81e700 1 -- 172.21.2.112:0/1182409259 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2ce80b04d0 msgr2=0x7f2ce81a7a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:27.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.995 7f2cec81e700 1 --2- 172.21.2.112:0/1182409259 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2ce80b04d0 0x7f2ce81a7a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:27.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.995 7f2cec81e700 1 -- 172.21.2.112:0/1182409259 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ce808e080 con 0x7f2ce81a8090 2021-02-10T11:45:27.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.995 7f2cec81e700 1 --2- 172.21.2.112:0/1182409259 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2ce81a8090 0x7f2ce80c23a0 secure :-1 s=READY pgs=2744 cs=0 l=1 rev1=1 rx=0x7f2cd8009650 tx=0x7f2cd800b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:27.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.995 7f2cddffb700 1 -- 172.21.2.112:0/1182409259 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2cd8009300 con 0x7f2ce81a8090 2021-02-10T11:45:27.001 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.995 7f2cddffb700 1 -- 172.21.2.112:0/1182409259 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cd8006030 con 0x7f2ce81a8090 2021-02-10T11:45:27.001 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.995 7f2cddffb700 1 -- 172.21.2.112:0/1182409259 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2cd8005560 con 0x7f2ce81a8090 2021-02-10T11:45:27.001 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.995 7f2cef203700 1 -- 172.21.2.112:0/1182409259 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ce808e080 con 0x7f2ce81a8090 2021-02-10T11:45:27.001 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.996 7f2cef203700 1 -- 172.21.2.112:0/1182409259 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ce808e080 con 0x7f2ce81a8090 2021-02-10T11:45:27.001 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.996 7f2cddffb700 1 -- 172.21.2.112:0/1182409259 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2cd8009300 con 0x7f2ce81a8090 2021-02-10T11:45:27.001 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.996 7f2cddffb700 1 --2- 172.21.2.112:0/1182409259 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2cc80186d0 0x7f2cc801abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:27.001 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.996 7f2cddffb700 1 -- 172.21.2.112:0/1182409259 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f2cd8032310 con 0x7f2ce81a8090 2021-02-10T11:45:27.001 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.997 7f2ced01f700 1 --2- 172.21.2.112:0/1182409259 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2cc80186d0 0x7f2cc801abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:27.001 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:26.997 7f2ced01f700 1 --2- 172.21.2.112:0/1182409259 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2cc80186d0 0x7f2cc801abd0 crc :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:27.029 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:27.027 7f2cef203700 1 -- 172.21.2.112:0/1182409259 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ce8082a70 con 0x7f2ce81a8090 2021-02-10T11:45:27.031 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:27.030 7f2cddffb700 1 -- 172.21.2.112:0/1182409259 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2cd8000940 con 0x7f2ce81a8090 2021-02-10T11:45:27.161 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:27.159 7f2cef203700 1 -- 172.21.2.112:0/1182409259 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f2ce8082a70 con 0x7f2ce81a8090 2021-02-10T11:45:27.161 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:27.160 7f2cddffb700 1 -- 172.21.2.112:0/1182409259 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v155) v1 ==== 92+0+10 (secure 0 0 0) 0x7f2cd800bd50 con 0x7f2ce81a8090 2021-02-10T11:45:27.162 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:45:27.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:27.162 7f2cef203700 1 -- 172.21.2.112:0/1182409259 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2cc80186d0 msgr2=0x7f2cc801abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:27.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:27.162 7f2cef203700 1 --2- 172.21.2.112:0/1182409259 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2cc80186d0 0x7f2cc801abd0 crc :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:27.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:27.162 7f2cef203700 1 -- 172.21.2.112:0/1182409259 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2ce81a8090 msgr2=0x7f2ce80c23a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:27.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:27.162 7f2cef203700 1 --2- 172.21.2.112:0/1182409259 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2ce81a8090 0x7f2ce80c23a0 secure :-1 s=READY pgs=2744 cs=0 l=1 rev1=1 rx=0x7f2cd8009650 tx=0x7f2cd800b960).stop 2021-02-10T11:45:27.164 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:27.162 7f2cef203700 1 -- 172.21.2.112:0/1182409259 shutdown_connections 2021-02-10T11:45:27.164 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:27.162 7f2cef203700 1 --2- 172.21.2.112:0/1182409259 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2cc80186d0 0x7f2cc801abd0 unknown :-1 s=CLOSED pgs=1248 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:27.164 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:27.162 7f2cef203700 1 --2- 172.21.2.112:0/1182409259 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2ce808d680 0x7f2ce80c29c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:27.164 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:27.162 7f2cef203700 1 --2- 172.21.2.112:0/1182409259 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2ce81a8090 0x7f2ce80c23a0 unknown :-1 s=CLOSED pgs=2744 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:27.164 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:27.162 7f2cef203700 1 --2- 172.21.2.112:0/1182409259 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2ce80b04d0 0x7f2ce81a7a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:27.164 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:27.162 7f2cef203700 1 -- 172.21.2.112:0/1182409259 >> 172.21.2.112:0/1182409259 conn(0x7f2ce80be530 msgr2=0x7f2ce809f1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:27.164 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:27.162 7f2cef203700 1 -- 172.21.2.112:0/1182409259 shutdown_connections 2021-02-10T11:45:27.164 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:27.162 7f2cef203700 1 -- 172.21.2.112:0/1182409259 wait complete. 2021-02-10T11:45:27.172 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --format=json-pretty osd lspools 2021-02-10T11:45:27.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:27.583 7f7394e05700 1 Processor -- start 2021-02-10T11:45:27.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:27.583 7f7394e05700 1 -- start start 2021-02-10T11:45:27.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:27.583 7f7394e05700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f73900d0340 0x7f73900ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:27.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:27.583 7f7394e05700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f73900d1c70 0x7f73900efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:27.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:27.583 7f7394e05700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73900f0b10 con 0x7f73900d1870 2021-02-10T11:45:27.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:27.583 7f7394e05700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73900e5ed0 con 0x7f73900d1c70 2021-02-10T11:45:27.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:27.583 7f7394e05700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73901b0be0 con 0x7f73900d0340 2021-02-10T11:45:27.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:27.585 7f738f314700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f73900d1c70 0x7f73900efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:27.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:27.585 7f738e312700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f73900d0340 0x7f73900ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:27.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:27.585 7f738eb13700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7f73900d1870 0x7f73901b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:51730/0 (socket says 172.21.2.112:51730) 2021-02-10T11:45:27.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:27.585 7f738eb13700 1 -- 172.21.2.112:0/2695799507 learned_addr learned my addr 172.21.2.112:0/2695799507 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:45:27.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:27.585 7f738f314700 1 -- 172.21.2.112:0/2695799507 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f73900d0340 msgr2=0x7f73900ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:27.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:27.585 7f738f314700 1 --2- 172.21.2.112:0/2695799507 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f73900d0340 0x7f73900ef640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:27.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:27.585 7f738db11700 1 -- 172.21.2.112:0/2695799507 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1287265916 0 0) 0x7f73780010c0 con 0x7f73900d1870 2021-02-10T11:45:27.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:27.585 7f738f314700 1 -- 172.21.2.112:0/2695799507 >> v1:172.21.2.120:6789/0 conn(0x7f73900d1870 legacy=0x7f73901b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:27.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:27.585 7f738f314700 1 -- 172.21.2.112:0/2695799507 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73900f20a0 con 0x7f73900d1c70 2021-02-10T11:45:27.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:27.585 7f738e312700 1 --2- 172.21.2.112:0/2695799507 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f73900d0340 0x7f73900ef640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:45:27.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:27.586 7f738f314700 1 --2- 172.21.2.112:0/2695799507 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f73900d1c70 0x7f73900efc60 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 rx=0x7f7388006e50 tx=0x7f7388009e70).ready entity=mon.1 client_cookie=d5a4ac9f05f2ae98 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:27.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:27.586 7f738db11700 1 -- 172.21.2.112:0/2695799507 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f738800b300 con 0x7f73900d1c70 2021-02-10T11:45:27.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:27.586 7f738db11700 1 -- 172.21.2.112:0/2695799507 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7388003030 con 0x7f73900d1c70 2021-02-10T11:45:27.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:27.586 7f7394e05700 1 -- 172.21.2.112:0/2695799507 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f73900d1c70 msgr2=0x7f73900efc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:27.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:27.586 7f7394e05700 1 --2- 172.21.2.112:0/2695799507 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f73900d1c70 0x7f73900efc60 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 rx=0x7f7388006e50 tx=0x7f7388009e70).stop 2021-02-10T11:45:27.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:27.586 7f738db11700 1 -- 172.21.2.112:0/2695799507 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7388000f90 con 0x7f73900d1c70 2021-02-10T11:45:27.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:27.587 7f7394e05700 1 -- 172.21.2.112:0/2695799507 shutdown_connections 2021-02-10T11:45:27.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:27.587 7f7394e05700 1 --2- 172.21.2.112:0/2695799507 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f73900d1c70 0x7f73900efc60 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:27.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:27.587 7f7394e05700 1 --2- 172.21.2.112:0/2695799507 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f73900d0340 0x7f73900ef640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:27.606 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:27.587 7f7394e05700 1 -- 172.21.2.112:0/2695799507 >> 172.21.2.112:0/2695799507 conn(0x7f73900ce450 msgr2=0x7f73900d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:27.606 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:27.587 7f7394e05700 1 -- 172.21.2.112:0/2695799507 shutdown_connections 2021-02-10T11:45:27.606 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:27.587 7f7394e05700 1 -- 172.21.2.112:0/2695799507 wait complete. 2021-02-10T11:45:27.608 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:27.602 7f7394e05700 1 Processor -- start 2021-02-10T11:45:27.608 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:27.602 7f7394e05700 1 -- start start 2021-02-10T11:45:27.608 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:27.602 7f7394e05700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f73900ada90 0x7f73900ade90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:27.608 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:27.602 7f7394e05700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7390099d20 0x7f739009a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:27.608 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:27.602 7f7394e05700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f739009a740 0x7f7390097bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:27.608 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:27.602 7f7394e05700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73900997a0 con 0x7f7390099d20 2021-02-10T11:45:27.608 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:27.602 7f7394e05700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f739018f0c0 con 0x7f73900ada90 2021-02-10T11:45:27.608 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:27.602 7f7394e05700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f739009ab40 con 0x7f739009a740 2021-02-10T11:45:27.608 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:27.603 7f738e312700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7390099d20 0x7f739009a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:27.608 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:27.603 7f738e312700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7390099d20 0x7f739009a120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:58716/0 (socket says 172.21.2.112:58716) 2021-02-10T11:45:27.609 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:27.603 7f738e312700 1 -- 172.21.2.112:0/498446162 learned_addr learned my addr 172.21.2.112:0/498446162 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:45:27.609 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:27.603 7f738f314700 1 --2- 172.21.2.112:0/498446162 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f739009a740 0x7f7390097bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:27.609 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:27.603 7f738eb13700 1 --2- 172.21.2.112:0/498446162 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f73900ada90 0x7f73900ade90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:27.609 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:27.603 7f738f314700 1 -- 172.21.2.112:0/498446162 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f73900ada90 msgr2=0x7f73900ade90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:27.609 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:27.603 7f738f314700 1 --2- 172.21.2.112:0/498446162 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f73900ada90 0x7f73900ade90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:27.609 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:27.603 7f738f314700 1 -- 172.21.2.112:0/498446162 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7390099d20 msgr2=0x7f739009a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:27.609 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:27.603 7f738f314700 1 --2- 172.21.2.112:0/498446162 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7390099d20 0x7f739009a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:27.609 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:27.603 7f738f314700 1 -- 172.21.2.112:0/498446162 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f739009ad60 con 0x7f739009a740 2021-02-10T11:45:27.609 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:27.603 7f738f314700 1 --2- 172.21.2.112:0/498446162 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f739009a740 0x7f7390097bc0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7f7388009960 tx=0x7f738800bc90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:27.609 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:27.604 7f73877fe700 1 -- 172.21.2.112:0/498446162 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f738800c2f0 con 0x7f739009a740 2021-02-10T11:45:27.609 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:27.604 7f73877fe700 1 -- 172.21.2.112:0/498446162 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73880069f0 con 0x7f739009a740 2021-02-10T11:45:27.609 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:27.604 7f7394e05700 1 -- 172.21.2.112:0/498446162 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f739009ad60 con 0x7f739009a740 2021-02-10T11:45:27.610 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:27.604 7f73877fe700 1 -- 172.21.2.112:0/498446162 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f738800c830 con 0x7f739009a740 2021-02-10T11:45:27.610 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:27.604 7f7394e05700 1 -- 172.21.2.112:0/498446162 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f739009ad60 con 0x7f739009a740 2021-02-10T11:45:27.610 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:27.604 7f73877fe700 1 -- 172.21.2.112:0/498446162 <== mon.2 v2:172.21.2.120:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7388001030 con 0x7f739009a740 2021-02-10T11:45:27.610 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:27.604 7f73877fe700 1 --2- 172.21.2.112:0/498446162 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f73740186c0 0x7f737401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:27.610 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:27.604 7f73877fe700 1 -- 172.21.2.112:0/498446162 <== mon.2 v2:172.21.2.120:3301/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f73880324d0 con 0x7f739009a740 2021-02-10T11:45:27.610 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:27.605 7f738eb13700 1 --2- 172.21.2.112:0/498446162 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f73740186c0 0x7f737401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:27.610 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:27.605 7f738eb13700 1 --2- 172.21.2.112:0/498446162 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f73740186c0 0x7f737401abc0 crc :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:27.635 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:27.633 7f7394e05700 1 -- 172.21.2.112:0/498446162 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73900ebc00 con 0x7f739009a740 2021-02-10T11:45:27.637 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:27.636 7f73877fe700 1 -- 172.21.2.112:0/498446162 <== mon.2 v2:172.21.2.120:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f738800a960 con 0x7f739009a740 2021-02-10T11:45:27.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:27.759 7f7394e05700 1 -- 172.21.2.112:0/498446162 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f73900f0b10 con 0x7f739009a740 2021-02-10T11:45:27.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:27.759 7f73877fe700 1 -- 172.21.2.112:0/498446162 <== mon.2 v2:172.21.2.120:3301/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v155) v1 ==== 84+0+64 (secure 0 0 0) 0x7f738800a960 con 0x7f739009a740 2021-02-10T11:45:27.761 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:45:27.761 INFO:teuthology.orchestra.run.gibba012.stdout:[ 2021-02-10T11:45:27.762 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:45:27.762 INFO:teuthology.orchestra.run.gibba012.stdout: "poolnum": 1, 2021-02-10T11:45:27.762 INFO:teuthology.orchestra.run.gibba012.stdout: "poolname": "rbd" 2021-02-10T11:45:27.762 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:45:27.762 INFO:teuthology.orchestra.run.gibba012.stdout:] 2021-02-10T11:45:27.763 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:27.761 7f7394e05700 1 -- 172.21.2.112:0/498446162 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f73740186c0 msgr2=0x7f737401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:27.763 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:27.761 7f7394e05700 1 --2- 172.21.2.112:0/498446162 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f73740186c0 0x7f737401abc0 crc :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:27.763 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:27.761 7f7394e05700 1 -- 172.21.2.112:0/498446162 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f739009a740 msgr2=0x7f7390097bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:27.763 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:27.761 7f7394e05700 1 --2- 172.21.2.112:0/498446162 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f739009a740 0x7f7390097bc0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7f7388009960 tx=0x7f738800bc90).stop 2021-02-10T11:45:27.763 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:27.761 7f7394e05700 1 -- 172.21.2.112:0/498446162 shutdown_connections 2021-02-10T11:45:27.763 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:27.761 7f7394e05700 1 --2- 172.21.2.112:0/498446162 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f73740186c0 0x7f737401abc0 unknown :-1 s=CLOSED pgs=1249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:27.763 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:27.761 7f7394e05700 1 --2- 172.21.2.112:0/498446162 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f739009a740 0x7f7390097bc0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:27.763 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:27.761 7f7394e05700 1 --2- 172.21.2.112:0/498446162 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7390099d20 0x7f739009a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:27.763 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:27.761 7f7394e05700 1 --2- 172.21.2.112:0/498446162 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f73900ada90 0x7f73900ade90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:27.764 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:27.761 7f7394e05700 1 -- 172.21.2.112:0/498446162 >> 172.21.2.112:0/498446162 conn(0x7f73900f0550 msgr2=0x7f7390191730 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:27.764 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:27.761 7f7394e05700 1 -- 172.21.2.112:0/498446162 shutdown_connections 2021-02-10T11:45:27.764 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:27.761 7f7394e05700 1 -- 172.21.2.112:0/498446162 wait complete. 2021-02-10T11:45:27.771 INFO:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2021-02-10T11:45:27.772 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_metadata 8 2021-02-10T11:45:28.205 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.186 7f50b4ba6700 1 Processor -- start 2021-02-10T11:45:28.205 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.186 7f50b4ba6700 1 -- start start 2021-02-10T11:45:28.205 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.187 7f50b4ba6700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f50b00d1c60 0x7f50b00cef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:28.205 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.187 7f50b4ba6700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f50b00cf570 0x7f50b00cf970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:28.206 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.187 7f50b4ba6700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50b00fe1b0 con 0x7f50b00d1860 2021-02-10T11:45:28.206 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.187 7f50b4ba6700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50b01b0ca0 con 0x7f50b00d1c60 2021-02-10T11:45:28.206 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.187 7f50b4ba6700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50b0109a50 con 0x7f50b00cf570 2021-02-10T11:45:28.206 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.187 7f50ae05d700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f50b00d1c60 0x7f50b00cef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:28.206 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.187 7f50ae05d700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f50b00d1c60 0x7f50b00cef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3300/0 says I am v2:172.21.2.112:32770/0 (socket says 172.21.2.112:32770) 2021-02-10T11:45:28.206 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.187 7f50ae05d700 1 -- 172.21.2.112:0/1006309113 learned_addr learned my addr 172.21.2.112:0/1006309113 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:45:28.206 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.187 7f50af05f700 1 --2- 172.21.2.112:0/1006309113 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f50b00cf570 0x7f50b00cf970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:28.206 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.187 7f50ae05d700 1 -- 172.21.2.112:0/1006309113 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f50b00cf570 msgr2=0x7f50b00cf970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:28.206 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.187 7f50ae05d700 1 --2- 172.21.2.112:0/1006309113 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f50b00cf570 0x7f50b00cf970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:28.207 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.187 7f50ae05d700 1 -- 172.21.2.112:0/1006309113 >> v1:172.21.2.120:6789/0 conn(0x7f50b00d1860 legacy=0x7f50b01b04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:28.207 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.187 7f50ae05d700 1 -- 172.21.2.112:0/1006309113 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50b0088070 con 0x7f50b00d1c60 2021-02-10T11:45:28.207 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.187 7f50af05f700 1 --2- 172.21.2.112:0/1006309113 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f50b00cf570 0x7f50b00cf970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:45:28.207 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.187 7f50ae05d700 1 --2- 172.21.2.112:0/1006309113 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f50b00d1c60 0x7f50b00cef50 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 rx=0x7f50a4006700 tx=0x7f50a4009b80).ready entity=mon.1 client_cookie=2c71579ed89a61d9 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:28.207 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.188 7f50ad85c700 1 -- 172.21.2.112:0/1006309113 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f50a4006340 con 0x7f50b00d1c60 2021-02-10T11:45:28.207 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.188 7f50ad85c700 1 -- 172.21.2.112:0/1006309113 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50a400c0c0 con 0x7f50b00d1c60 2021-02-10T11:45:28.207 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.188 7f50ad85c700 1 -- 172.21.2.112:0/1006309113 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f50a400c510 con 0x7f50b00d1c60 2021-02-10T11:45:28.207 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.188 7f50b4ba6700 1 -- 172.21.2.112:0/1006309113 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f50b00d1c60 msgr2=0x7f50b00cef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:28.207 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.188 7f50b4ba6700 1 --2- 172.21.2.112:0/1006309113 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f50b00d1c60 0x7f50b00cef50 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 rx=0x7f50a4006700 tx=0x7f50a4009b80).stop 2021-02-10T11:45:28.207 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.189 7f50b4ba6700 1 -- 172.21.2.112:0/1006309113 shutdown_connections 2021-02-10T11:45:28.207 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.189 7f50b4ba6700 1 --2- 172.21.2.112:0/1006309113 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f50b00d1c60 0x7f50b00cef50 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:28.208 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.189 7f50b4ba6700 1 --2- 172.21.2.112:0/1006309113 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f50b00cf570 0x7f50b00cf970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:28.208 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.189 7f50b4ba6700 1 -- 172.21.2.112:0/1006309113 >> 172.21.2.112:0/1006309113 conn(0x7f50b00ce410 msgr2=0x7f50b00d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:28.208 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.189 7f50b4ba6700 1 -- 172.21.2.112:0/1006309113 shutdown_connections 2021-02-10T11:45:28.208 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.189 7f50b4ba6700 1 -- 172.21.2.112:0/1006309113 wait complete. 2021-02-10T11:45:28.211 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.204 7f50b4ba6700 1 Processor -- start 2021-02-10T11:45:28.211 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.204 7f50b4ba6700 1 -- start start 2021-02-10T11:45:28.211 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.204 7f50b4ba6700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f50b0085270 0x7f50b0085670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:28.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.204 7f50b4ba6700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f50b0085c90 0x7f50b0086090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:28.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.204 7f50b4ba6700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f50b00c2740 0x7f50b00c2b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:28.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.204 7f50b4ba6700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50b0084cf0 con 0x7f50b0085270 2021-02-10T11:45:28.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.204 7f50b4ba6700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50b00be1b0 con 0x7f50b00c2740 2021-02-10T11:45:28.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.204 7f50b4ba6700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50b00be390 con 0x7f50b0085c90 2021-02-10T11:45:28.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.204 7f50ae85e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f50b0085270 0x7f50b0085670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:28.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.204 7f50ae85e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f50b0085270 0x7f50b0085670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:58726/0 (socket says 172.21.2.112:58726) 2021-02-10T11:45:28.213 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.204 7f50ae85e700 1 -- 172.21.2.112:0/2147719619 learned_addr learned my addr 172.21.2.112:0/2147719619 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:45:28.213 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.205 7f50ae85e700 1 -- 172.21.2.112:0/2147719619 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f50b0085c90 msgr2=0x7f50b0086090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:28.213 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.205 7f50ae85e700 1 --2- 172.21.2.112:0/2147719619 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f50b0085c90 0x7f50b0086090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:28.213 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.205 7f50ae85e700 1 -- 172.21.2.112:0/2147719619 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f50b00c2740 msgr2=0x7f50b00c2b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:28.213 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.205 7f50ae85e700 1 --2- 172.21.2.112:0/2147719619 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f50b00c2740 0x7f50b00c2b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:28.213 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.205 7f50ae85e700 1 -- 172.21.2.112:0/2147719619 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50b00c3160 con 0x7f50b0085270 2021-02-10T11:45:28.214 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.205 7f50ae85e700 1 --2- 172.21.2.112:0/2147719619 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f50b0085270 0x7f50b0085670 secure :-1 s=READY pgs=2745 cs=0 l=1 rev1=1 rx=0x7f50a000b5b0 tx=0x7f50a000b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:28.214 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.205 7f509f7fe700 1 -- 172.21.2.112:0/2147719619 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f50a000bd40 con 0x7f50b0085270 2021-02-10T11:45:28.214 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.205 7f509f7fe700 1 -- 172.21.2.112:0/2147719619 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50a0006240 con 0x7f50b0085270 2021-02-10T11:45:28.214 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.205 7f509f7fe700 1 -- 172.21.2.112:0/2147719619 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f50a0005720 con 0x7f50b0085270 2021-02-10T11:45:28.214 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.205 7f50b4ba6700 1 -- 172.21.2.112:0/2147719619 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50b00c3160 con 0x7f50b0085270 2021-02-10T11:45:28.214 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.205 7f50b4ba6700 1 -- 172.21.2.112:0/2147719619 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50b00c36c0 con 0x7f50b0085270 2021-02-10T11:45:28.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.206 7f509f7fe700 1 -- 172.21.2.112:0/2147719619 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f50a00054a0 con 0x7f50b0085270 2021-02-10T11:45:28.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.206 7f509f7fe700 1 --2- 172.21.2.112:0/2147719619 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5094018730 0x7f509401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:28.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.206 7f509f7fe700 1 -- 172.21.2.112:0/2147719619 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f50a00324f0 con 0x7f50b0085270 2021-02-10T11:45:28.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.206 7f50ae05d700 1 --2- 172.21.2.112:0/2147719619 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5094018730 0x7f509401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:28.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.207 7f50ae05d700 1 --2- 172.21.2.112:0/2147719619 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5094018730 0x7f509401ac10 crc :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:28.238 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.236 7f50b4ba6700 1 -- 172.21.2.112:0/2147719619 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50b0082a20 con 0x7f50b0085270 2021-02-10T11:45:28.240 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.239 7f509f7fe700 1 -- 172.21.2.112:0/2147719619 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f50a0000940 con 0x7f50b0085270 2021-02-10T11:45:28.369 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.367 7f50b4ba6700 1 -- 172.21.2.112:0/2147719619 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"} v 0) v1 -- 0x7f50b0082a20 con 0x7f50b0085270 2021-02-10T11:45:28.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.499 7f509f7fe700 1 -- 172.21.2.112:0/2147719619 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"}]=0 pool 'cephfs_metadata' created v156) v1 ==== 133+0+0 (secure 0 0 0) 0x7f50a00054a0 con 0x7f50b0085270 2021-02-10T11:45:28.533 INFO:teuthology.orchestra.run.gibba012.stderr:pool 'cephfs_metadata' created 2021-02-10T11:45:28.535 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.533 7f50b4ba6700 1 -- 172.21.2.112:0/2147719619 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5094018730 msgr2=0x7f509401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:28.536 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.533 7f50b4ba6700 1 --2- 172.21.2.112:0/2147719619 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5094018730 0x7f509401ac10 crc :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:28.536 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.533 7f50b4ba6700 1 -- 172.21.2.112:0/2147719619 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f50b0085270 msgr2=0x7f50b0085670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:28.536 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.533 7f50b4ba6700 1 --2- 172.21.2.112:0/2147719619 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f50b0085270 0x7f50b0085670 secure :-1 s=READY pgs=2745 cs=0 l=1 rev1=1 rx=0x7f50a000b5b0 tx=0x7f50a000b950).stop 2021-02-10T11:45:28.536 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.533 7f50b4ba6700 1 -- 172.21.2.112:0/2147719619 shutdown_connections 2021-02-10T11:45:28.536 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.533 7f50b4ba6700 1 --2- 172.21.2.112:0/2147719619 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5094018730 0x7f509401ac10 unknown :-1 s=CLOSED pgs=1250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:28.536 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.533 7f50b4ba6700 1 --2- 172.21.2.112:0/2147719619 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f50b00c2740 0x7f50b00c2b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:28.536 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.533 7f50b4ba6700 1 --2- 172.21.2.112:0/2147719619 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f50b0085c90 0x7f50b0086090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:28.536 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.533 7f50b4ba6700 1 --2- 172.21.2.112:0/2147719619 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f50b0085270 0x7f50b0085670 unknown :-1 s=CLOSED pgs=2745 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:28.537 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.533 7f50b4ba6700 1 -- 172.21.2.112:0/2147719619 >> 172.21.2.112:0/2147719619 conn(0x7f50b00b5f10 msgr2=0x7f50b009f200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:28.537 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.533 7f50b4ba6700 1 -- 172.21.2.112:0/2147719619 shutdown_connections 2021-02-10T11:45:28.537 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.533 7f50b4ba6700 1 -- 172.21.2.112:0/2147719619 wait complete. 2021-02-10T11:45:28.544 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_data 8 2021-02-10T11:45:28.977 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.958 7fe6e70a4700 1 Processor -- start 2021-02-10T11:45:28.977 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.958 7fe6e70a4700 1 -- start start 2021-02-10T11:45:28.977 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.958 7fe6e70a4700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fe6e010aff0 0x7fe6e00cf750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:28.977 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.958 7fe6e70a4700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6e00fe1c0 con 0x7fe6e00d1870 2021-02-10T11:45:28.977 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.958 7fe6e70a4700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6e01b0cb0 con 0x7fe6e00d1c70 2021-02-10T11:45:28.977 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.958 7fe6e70a4700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6e0109a60 con 0x7fe6e010aff0 2021-02-10T11:45:28.977 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.958 7fe6e4ec0700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fe6e00d1870 0x7fe6e01b04c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:58196/0 (socket says 172.21.2.112:58196) 2021-02-10T11:45:28.977 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.958 7fe6e4ec0700 1 -- 172.21.2.112:0/3035107407 learned_addr learned my addr 172.21.2.112:0/3035107407 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:45:28.978 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.959 7fe6d77fe700 1 -- 172.21.2.112:0/3035107407 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3958278609 0 0) 0x7fe6d0000ce0 con 0x7fe6e00d1870 2021-02-10T11:45:28.978 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.959 7fe6d77fe700 1 -- 172.21.2.112:0/3035107407 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6c8002fe0 con 0x7fe6e00d1870 2021-02-10T11:45:28.978 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.959 7fe6e56c1700 1 --2- 172.21.2.112:0/3035107407 >> v2:172.21.2.120:3300/0 conn(0x7fe6e010aff0 0x7fe6e00cf750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:28.978 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.959 7fe6d77fe700 1 -- 172.21.2.112:0/3035107407 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 661536631 0 0) 0x7fe6d00010e0 con 0x7fe6e00d1870 2021-02-10T11:45:28.978 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.959 7fe6d77fe700 1 -- 172.21.2.112:0/3035107407 >> v2:172.21.2.120:3300/0 conn(0x7fe6e010aff0 msgr2=0x7fe6e00cf750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:28.978 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.959 7fe6d77fe700 1 --2- 172.21.2.112:0/3035107407 >> v2:172.21.2.120:3300/0 conn(0x7fe6e010aff0 0x7fe6e00cf750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:28.978 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.959 7fe6d77fe700 1 -- 172.21.2.112:0/3035107407 >> v1:172.21.2.120:6789/0 conn(0x7fe6e00d1c70 legacy=0x7fe6e00cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:28.978 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.959 7fe6d77fe700 1 -- 172.21.2.112:0/3035107407 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6e010b3f0 con 0x7fe6e00d1870 2021-02-10T11:45:28.978 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.959 7fe6d77fe700 1 -- 172.21.2.112:0/3035107407 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7fe6d0001600 con 0x7fe6e00d1870 2021-02-10T11:45:28.979 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.959 7fe6d77fe700 1 -- 172.21.2.112:0/3035107407 >> v1:172.21.2.112:6789/0 conn(0x7fe6e00d1870 legacy=0x7fe6e01b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:28.979 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.959 7fe6d77fe700 1 --2- 172.21.2.112:0/3035107407 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe6c8003b10 0x7fe6c8005f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:28.979 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.959 7fe6d77fe700 1 --2- 172.21.2.112:0/3035107407 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe6c80067a0 0x7fe6c8008ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:28.979 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.959 7fe6d77fe700 1 --2- 172.21.2.112:0/3035107407 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe6c8009580 0x7fe6c800baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:28.979 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.959 7fe6d77fe700 1 -- 172.21.2.112:0/3035107407 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6c800c2f0 con 0x7fe6c8009580 2021-02-10T11:45:28.979 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.959 7fe6d77fe700 1 -- 172.21.2.112:0/3035107407 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6c800c510 con 0x7fe6c80067a0 2021-02-10T11:45:28.979 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.959 7fe6d77fe700 1 -- 172.21.2.112:0/3035107407 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6c800c730 con 0x7fe6c8003b10 2021-02-10T11:45:28.979 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.959 7fe6e56c1700 1 --2- 172.21.2.112:0/3035107407 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe6c8009580 0x7fe6c800baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:28.979 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.960 7fe6e4ec0700 1 --2- 172.21.2.112:0/3035107407 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe6c8003b10 0x7fe6c8005f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:28.979 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.960 7fe6d7fff700 1 --2- 172.21.2.112:0/3035107407 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe6c80067a0 0x7fe6c8008ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:28.979 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.960 7fe6e56c1700 1 -- 172.21.2.112:0/3035107407 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe6c8003b10 msgr2=0x7fe6c8005f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:28.979 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.960 7fe6e56c1700 1 --2- 172.21.2.112:0/3035107407 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe6c8003b10 0x7fe6c8005f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:28.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.960 7fe6e56c1700 1 -- 172.21.2.112:0/3035107407 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe6c80067a0 msgr2=0x7fe6c8008ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:28.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.960 7fe6e56c1700 1 --2- 172.21.2.112:0/3035107407 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe6c80067a0 0x7fe6c8008ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:28.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.960 7fe6e56c1700 1 -- 172.21.2.112:0/3035107407 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6c800c9d0 con 0x7fe6c8009580 2021-02-10T11:45:28.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.960 7fe6e56c1700 1 -- 172.21.2.112:0/3035107407 reap_dead start 2021-02-10T11:45:28.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.960 7fe6e56c1700 1 --2- 172.21.2.112:0/3035107407 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe6c8009580 0x7fe6c800baa0 secure :-1 s=READY pgs=2747 cs=0 l=1 rev1=1 rx=0x7fe6d804dc80 tx=0x7fe6d804f300).ready entity=mon.0 client_cookie=8cd6f061b325bcb1 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:28.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.960 7fe6d77fe700 1 -- 172.21.2.112:0/3035107407 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe6d8006cb0 con 0x7fe6c8009580 2021-02-10T11:45:28.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.960 7fe6d77fe700 1 -- 172.21.2.112:0/3035107407 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6d8003030 con 0x7fe6c8009580 2021-02-10T11:45:28.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.960 7fe6d77fe700 1 -- 172.21.2.112:0/3035107407 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe6d8006cb0 con 0x7fe6c8009580 2021-02-10T11:45:28.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.960 7fe6e70a4700 1 -- 172.21.2.112:0/3035107407 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe6c8009580 msgr2=0x7fe6c800baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:28.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.960 7fe6e70a4700 1 --2- 172.21.2.112:0/3035107407 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe6c8009580 0x7fe6c800baa0 secure :-1 s=READY pgs=2747 cs=0 l=1 rev1=1 rx=0x7fe6d804dc80 tx=0x7fe6d804f300).stop 2021-02-10T11:45:28.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.961 7fe6e70a4700 1 -- 172.21.2.112:0/3035107407 shutdown_connections 2021-02-10T11:45:28.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.961 7fe6e70a4700 1 --2- 172.21.2.112:0/3035107407 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe6c8009580 0x7fe6c800baa0 unknown :-1 s=CLOSED pgs=2747 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:28.981 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.961 7fe6e70a4700 1 -- 172.21.2.112:0/3035107407 >> 172.21.2.112:0/3035107407 conn(0x7fe6e00ce420 msgr2=0x7fe6e00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:28.981 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.961 7fe6e70a4700 1 -- 172.21.2.112:0/3035107407 shutdown_connections 2021-02-10T11:45:28.981 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.961 7fe6e70a4700 1 -- 172.21.2.112:0/3035107407 wait complete. 2021-02-10T11:45:28.981 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.975 7fe6e70a4700 1 Processor -- start 2021-02-10T11:45:28.981 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.975 7fe6e70a4700 1 -- start start 2021-02-10T11:45:28.983 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.976 7fe6e70a4700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe6e00926d0 0x7fe6e00b6820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:28.983 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.976 7fe6e70a4700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe6e00c3840 0x7fe6e0093270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:28.983 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.976 7fe6e70a4700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe6e00c3c40 0x7fe6e0093890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:28.983 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.976 7fe6e70a4700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6e0092ad0 con 0x7fe6e00c3c40 2021-02-10T11:45:28.983 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.976 7fe6e70a4700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6e0092cf0 con 0x7fe6e00926d0 2021-02-10T11:45:28.983 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.976 7fe6e70a4700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6e00b6f80 con 0x7fe6e00c3840 2021-02-10T11:45:28.983 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.976 7fe6e56c1700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe6e00c3c40 0x7fe6e0093890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:28.983 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.976 7fe6e56c1700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe6e00c3c40 0x7fe6e0093890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:58746/0 (socket says 172.21.2.112:58746) 2021-02-10T11:45:28.983 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.976 7fe6e56c1700 1 -- 172.21.2.112:0/2304425386 learned_addr learned my addr 172.21.2.112:0/2304425386 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:45:28.984 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.976 7fe6e56c1700 1 -- 172.21.2.112:0/2304425386 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe6e00c3840 msgr2=0x7fe6e0093270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:28.984 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.976 7fe6d7fff700 1 --2- 172.21.2.112:0/2304425386 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe6e00c3840 0x7fe6e0093270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:28.984 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.976 7fe6e56c1700 1 --2- 172.21.2.112:0/2304425386 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe6e00c3840 0x7fe6e0093270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:28.984 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.976 7fe6e56c1700 1 -- 172.21.2.112:0/2304425386 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe6e00926d0 msgr2=0x7fe6e00b6820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:28.984 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.976 7fe6e56c1700 1 --2- 172.21.2.112:0/2304425386 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe6e00926d0 0x7fe6e00b6820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:28.984 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.976 7fe6e56c1700 1 -- 172.21.2.112:0/2304425386 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6e0093eb0 con 0x7fe6e00c3c40 2021-02-10T11:45:28.984 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.976 7fe6d7fff700 1 --2- 172.21.2.112:0/2304425386 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe6e00c3840 0x7fe6e0093270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:45:28.984 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.976 7fe6e56c1700 1 --2- 172.21.2.112:0/2304425386 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe6e00c3c40 0x7fe6e0093890 secure :-1 s=READY pgs=2748 cs=0 l=1 rev1=1 rx=0x7fe6d8050910 tx=0x7fe6d8050cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:28.984 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.977 7fe6d57fa700 1 -- 172.21.2.112:0/2304425386 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe6d804b510 con 0x7fe6e00c3c40 2021-02-10T11:45:28.984 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.977 7fe6d57fa700 1 -- 172.21.2.112:0/2304425386 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6d804d1f0 con 0x7fe6e00c3c40 2021-02-10T11:45:28.984 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.977 7fe6e70a4700 1 -- 172.21.2.112:0/2304425386 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6e0093eb0 con 0x7fe6e00c3c40 2021-02-10T11:45:28.984 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.977 7fe6d57fa700 1 -- 172.21.2.112:0/2304425386 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe6d804b9d0 con 0x7fe6e00c3c40 2021-02-10T11:45:28.985 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.977 7fe6e70a4700 1 -- 172.21.2.112:0/2304425386 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6e0093eb0 con 0x7fe6e00c3c40 2021-02-10T11:45:28.985 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.977 7fe6d57fa700 1 -- 172.21.2.112:0/2304425386 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe6d80056b0 con 0x7fe6e00c3c40 2021-02-10T11:45:28.985 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.977 7fe6d57fa700 1 --2- 172.21.2.112:0/2304425386 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe6c00186c0 0x7fe6c001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:28.985 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.977 7fe6d57fa700 1 -- 172.21.2.112:0/2304425386 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(158..158 src has 1..158) v4 ==== 5107+0+0 (secure 0 0 0) 0x7fe6d8076710 con 0x7fe6e00c3c40 2021-02-10T11:45:28.985 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.978 7fe6e4ec0700 1 --2- 172.21.2.112:0/2304425386 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe6c00186c0 0x7fe6c001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:28.985 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:28.978 7fe6e4ec0700 1 --2- 172.21.2.112:0/2304425386 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe6c00186c0 0x7fe6c001abc0 crc :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:29.009 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:29.008 7fe6e70a4700 1 -- 172.21.2.112:0/2304425386 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6e0082a70 con 0x7fe6e00c3c40 2021-02-10T11:45:29.012 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:29.010 7fe6d57fa700 1 -- 172.21.2.112:0/2304425386 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe6d8071f50 con 0x7fe6e00c3c40 2021-02-10T11:45:29.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:29.139 7fe6e70a4700 1 -- 172.21.2.112:0/2304425386 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"} v 0) v1 -- 0x7fe6e0082a70 con 0x7fe6e00c3c40 2021-02-10T11:45:29.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:29.778 7fe6d57fa700 1 -- 172.21.2.112:0/2304425386 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"}]=0 pool 'cephfs_data' created v159) v1 ==== 125+0+0 (secure 0 0 0) 0x7fe6d8071f50 con 0x7fe6e00c3c40 2021-02-10T11:45:29.808 INFO:teuthology.orchestra.run.gibba012.stderr:pool 'cephfs_data' created 2021-02-10T11:45:29.810 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:29.808 7fe6e70a4700 1 -- 172.21.2.112:0/2304425386 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe6c00186c0 msgr2=0x7fe6c001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:29.810 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:29.808 7fe6e70a4700 1 --2- 172.21.2.112:0/2304425386 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe6c00186c0 0x7fe6c001abc0 crc :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:29.810 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:29.808 7fe6e70a4700 1 -- 172.21.2.112:0/2304425386 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe6e00c3c40 msgr2=0x7fe6e0093890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:29.810 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:29.808 7fe6e70a4700 1 --2- 172.21.2.112:0/2304425386 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe6e00c3c40 0x7fe6e0093890 secure :-1 s=READY pgs=2748 cs=0 l=1 rev1=1 rx=0x7fe6d8050910 tx=0x7fe6d8050cb0).stop 2021-02-10T11:45:29.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:29.809 7fe6e70a4700 1 -- 172.21.2.112:0/2304425386 shutdown_connections 2021-02-10T11:45:29.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:29.809 7fe6e70a4700 1 --2- 172.21.2.112:0/2304425386 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe6c00186c0 0x7fe6c001abc0 unknown :-1 s=CLOSED pgs=1251 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:29.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:29.809 7fe6e70a4700 1 --2- 172.21.2.112:0/2304425386 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe6e00c3c40 0x7fe6e0093890 unknown :-1 s=CLOSED pgs=2748 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:29.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:29.809 7fe6e70a4700 1 --2- 172.21.2.112:0/2304425386 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe6e00c3840 0x7fe6e0093270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:29.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:29.809 7fe6e70a4700 1 --2- 172.21.2.112:0/2304425386 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe6e00926d0 0x7fe6e00b6820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:29.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:29.809 7fe6e70a4700 1 -- 172.21.2.112:0/2304425386 >> 172.21.2.112:0/2304425386 conn(0x7fe6e009ffd0 msgr2=0x7fe6e0087030 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:29.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:29.809 7fe6e70a4700 1 -- 172.21.2.112:0/2304425386 shutdown_connections 2021-02-10T11:45:29.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:29.809 7fe6e70a4700 1 -- 172.21.2.112:0/2304425386 wait complete. 2021-02-10T11:45:29.819 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2021-02-10T11:45:30.252 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.233 7fa142547700 1 Processor -- start 2021-02-10T11:45:30.253 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.233 7fa142547700 1 -- start start 2021-02-10T11:45:30.253 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.234 7fa142547700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa13c0d1c80 0x7fa13c0cef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:30.253 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.234 7fa142547700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa13c0d0180 con 0x7fa13c0d1880 2021-02-10T11:45:30.253 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.234 7fa142547700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa13c0fe1d0 con 0x7fa13c0cf590 2021-02-10T11:45:30.253 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.234 7fa142547700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa13c1b0cc0 con 0x7fa13c0d1c80 2021-02-10T11:45:30.254 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.234 7fa13bfff700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fa13c0d1880 0x7fa13c1b04d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:58216/0 (socket says 172.21.2.112:58216) 2021-02-10T11:45:30.254 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.234 7fa13bfff700 1 -- 172.21.2.112:0/1516598719 learned_addr learned my addr 172.21.2.112:0/1516598719 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:45:30.254 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.234 7fa13b7fe700 1 --2- 172.21.2.112:0/1516598719 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa13c0d1c80 0x7fa13c0cef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:30.254 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.234 7fa13affd700 1 -- 172.21.2.112:0/1516598719 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 231736014 0 0) 0x7fa12c000ce0 con 0x7fa13c0d1880 2021-02-10T11:45:30.254 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.234 7fa13affd700 1 -- 172.21.2.112:0/1516598719 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa124002fe0 con 0x7fa13c0d1880 2021-02-10T11:45:30.255 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.234 7fa13affd700 1 -- 172.21.2.112:0/1516598719 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2633976272 0 0) 0x7fa134000d30 con 0x7fa13c0cf590 2021-02-10T11:45:30.255 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.234 7fa13affd700 1 -- 172.21.2.112:0/1516598719 --> v1:172.21.2.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa124003210 con 0x7fa13c0cf590 2021-02-10T11:45:30.255 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.235 7fa13affd700 1 -- 172.21.2.112:0/1516598719 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 776582814 0 0) 0x7fa12c0010e0 con 0x7fa13c0d1880 2021-02-10T11:45:30.255 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.235 7fa13b7fe700 1 -- 172.21.2.112:0/1516598719 >> v1:172.21.2.120:6789/0 conn(0x7fa13c0cf590 legacy=0x7fa13c0cf990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:30.255 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.235 7fa13b7fe700 1 -- 172.21.2.112:0/1516598719 >> v1:172.21.2.112:6789/0 conn(0x7fa13c0d1880 legacy=0x7fa13c1b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:30.255 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.235 7fa13b7fe700 1 -- 172.21.2.112:0/1516598719 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa13c109a70 con 0x7fa13c0d1c80 2021-02-10T11:45:30.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.235 7fa13b7fe700 1 --2- 172.21.2.112:0/1516598719 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa13c0d1c80 0x7fa13c0cef70 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 rx=0x7fa130006650 tx=0x7fa130009a80).ready entity=mon.2 client_cookie=a1eb62424154e05b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:30.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.235 7fa13affd700 1 -- 172.21.2.112:0/1516598719 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa1300009b0 con 0x7fa13c0d1c80 2021-02-10T11:45:30.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.235 7fa13affd700 1 -- 172.21.2.112:0/1516598719 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa13000c0c0 con 0x7fa13c0d1c80 2021-02-10T11:45:30.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.235 7fa13affd700 1 -- 172.21.2.112:0/1516598719 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa1300009b0 con 0x7fa13c0d1c80 2021-02-10T11:45:30.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.235 7fa142547700 1 -- 172.21.2.112:0/1516598719 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa13c0d1c80 msgr2=0x7fa13c0cef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:30.257 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.235 7fa142547700 1 --2- 172.21.2.112:0/1516598719 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa13c0d1c80 0x7fa13c0cef70 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 rx=0x7fa130006650 tx=0x7fa130009a80).stop 2021-02-10T11:45:30.257 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.236 7fa142547700 1 -- 172.21.2.112:0/1516598719 shutdown_connections 2021-02-10T11:45:30.257 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.236 7fa142547700 1 --2- 172.21.2.112:0/1516598719 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa13c0d1c80 0x7fa13c0cef70 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:30.257 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.236 7fa142547700 1 -- 172.21.2.112:0/1516598719 >> 172.21.2.112:0/1516598719 conn(0x7fa13c0ce430 msgr2=0x7fa13c0d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:30.257 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.236 7fa142547700 1 -- 172.21.2.112:0/1516598719 shutdown_connections 2021-02-10T11:45:30.257 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.236 7fa142547700 1 -- 172.21.2.112:0/1516598719 wait complete. 2021-02-10T11:45:30.258 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.250 7fa142547700 1 Processor -- start 2021-02-10T11:45:30.258 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.251 7fa142547700 1 -- start start 2021-02-10T11:45:30.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.251 7fa142547700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa13c1a7a70 0x7fa13c1a7e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:30.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.251 7fa142547700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa13c0bf730 0x7fa13c0bfb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:30.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.251 7fa142547700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa13c0c0150 0x7fa13c0909c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:30.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.251 7fa142547700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa13c090fe0 con 0x7fa13c0bf730 2021-02-10T11:45:30.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.251 7fa142547700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa13c091200 con 0x7fa13c1a7a70 2021-02-10T11:45:30.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.251 7fa142547700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa13c091420 con 0x7fa13c0c0150 2021-02-10T11:45:30.262 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.251 7fa13b7fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa13c0bf730 0x7fa13c0bfb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:30.262 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.251 7fa13b7fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa13c0bf730 0x7fa13c0bfb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:58760/0 (socket says 172.21.2.112:58760) 2021-02-10T11:45:30.262 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.251 7fa13b7fe700 1 -- 172.21.2.112:0/2479305183 learned_addr learned my addr 172.21.2.112:0/2479305183 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:45:30.262 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.251 7fa13bfff700 1 --2- 172.21.2.112:0/2479305183 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa13c1a7a70 0x7fa13c1a7e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:30.262 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.251 7fa140b64700 1 --2- 172.21.2.112:0/2479305183 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa13c0c0150 0x7fa13c0909c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:30.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.252 7fa13b7fe700 1 -- 172.21.2.112:0/2479305183 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa13c0c0150 msgr2=0x7fa13c0909c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:30.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.252 7fa13b7fe700 1 --2- 172.21.2.112:0/2479305183 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa13c0c0150 0x7fa13c0909c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:30.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.252 7fa13b7fe700 1 -- 172.21.2.112:0/2479305183 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa13c1a7a70 msgr2=0x7fa13c1a7e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:30.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.252 7fa13b7fe700 1 --2- 172.21.2.112:0/2479305183 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa13c1a7a70 0x7fa13c1a7e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:30.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.252 7fa13b7fe700 1 -- 172.21.2.112:0/2479305183 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa13c091640 con 0x7fa13c0bf730 2021-02-10T11:45:30.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.252 7fa13b7fe700 1 --2- 172.21.2.112:0/2479305183 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa13c0bf730 0x7fa13c0bfb30 secure :-1 s=READY pgs=2750 cs=0 l=1 rev1=1 rx=0x7fa13000b660 tx=0x7fa13000ba00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:30.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.252 7fa138ff9700 1 -- 172.21.2.112:0/2479305183 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa1300009b0 con 0x7fa13c0bf730 2021-02-10T11:45:30.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.252 7fa138ff9700 1 -- 172.21.2.112:0/2479305183 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa130006760 con 0x7fa13c0bf730 2021-02-10T11:45:30.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.252 7fa138ff9700 1 -- 172.21.2.112:0/2479305183 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa130002030 con 0x7fa13c0bf730 2021-02-10T11:45:30.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.252 7fa142547700 1 -- 172.21.2.112:0/2479305183 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa13c091640 con 0x7fa13c0bf730 2021-02-10T11:45:30.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.252 7fa142547700 1 -- 172.21.2.112:0/2479305183 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa13c091640 con 0x7fa13c0bf730 2021-02-10T11:45:30.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.253 7fa138ff9700 1 -- 172.21.2.112:0/2479305183 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa130000c90 con 0x7fa13c0bf730 2021-02-10T11:45:30.265 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.253 7fa138ff9700 1 --2- 172.21.2.112:0/2479305183 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa11c0186c0 0x7fa11c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:30.265 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.253 7fa138ff9700 1 -- 172.21.2.112:0/2479305183 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(159..159 src has 1..159) v4 ==== 5457+0+0 (secure 0 0 0) 0x7fa1300324e0 con 0x7fa13c0bf730 2021-02-10T11:45:30.265 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.253 7fa13bfff700 1 --2- 172.21.2.112:0/2479305183 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa11c0186c0 0x7fa11c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:30.265 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.254 7fa13bfff700 1 --2- 172.21.2.112:0/2479305183 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa11c0186c0 0x7fa11c01abc0 crc :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:30.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.284 7fa142547700 1 -- 172.21.2.112:0/2479305183 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa13c0fe1d0 con 0x7fa13c0bf730 2021-02-10T11:45:30.288 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.286 7fa138ff9700 1 -- 172.21.2.112:0/2479305183 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa130006470 con 0x7fa13c0bf730 2021-02-10T11:45:30.417 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.415 7fa142547700 1 -- 172.21.2.112:0/2479305183 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"} v 0) v1 -- 0x7fa13c0fe1d0 con 0x7fa13c0bf730 2021-02-10T11:45:30.420 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.419 7fa138ff9700 1 -- 172.21.2.112:0/2479305183 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"}]=0 new fs with metadata pool 34 and data pool 35 v136) v1 ==== 189+0+0 (secure 0 0 0) 0x7fa130000c90 con 0x7fa13c0bf730 2021-02-10T11:45:30.424 INFO:teuthology.orchestra.run.gibba012.stderr:new fs with metadata pool 34 and data pool 35 2021-02-10T11:45:30.426 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.424 7fa142547700 1 -- 172.21.2.112:0/2479305183 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa11c0186c0 msgr2=0x7fa11c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:30.426 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.424 7fa142547700 1 --2- 172.21.2.112:0/2479305183 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa11c0186c0 0x7fa11c01abc0 crc :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:30.426 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.424 7fa142547700 1 -- 172.21.2.112:0/2479305183 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa13c0bf730 msgr2=0x7fa13c0bfb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:30.426 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.424 7fa142547700 1 --2- 172.21.2.112:0/2479305183 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa13c0bf730 0x7fa13c0bfb30 secure :-1 s=READY pgs=2750 cs=0 l=1 rev1=1 rx=0x7fa13000b660 tx=0x7fa13000ba00).stop 2021-02-10T11:45:30.426 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.424 7fa142547700 1 -- 172.21.2.112:0/2479305183 shutdown_connections 2021-02-10T11:45:30.426 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.424 7fa142547700 1 --2- 172.21.2.112:0/2479305183 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa11c0186c0 0x7fa11c01abc0 unknown :-1 s=CLOSED pgs=1254 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:30.426 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.424 7fa142547700 1 --2- 172.21.2.112:0/2479305183 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa13c0c0150 0x7fa13c0909c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:30.426 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.424 7fa142547700 1 --2- 172.21.2.112:0/2479305183 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa13c0bf730 0x7fa13c0bfb30 unknown :-1 s=CLOSED pgs=2750 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:30.426 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.424 7fa142547700 1 --2- 172.21.2.112:0/2479305183 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa13c1a7a70 0x7fa13c1a7e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:30.426 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.424 7fa142547700 1 -- 172.21.2.112:0/2479305183 >> 172.21.2.112:0/2479305183 conn(0x7fa13c0a69d0 msgr2=0x7fa13c0b5950 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:30.427 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.424 7fa142547700 1 -- 172.21.2.112:0/2479305183 shutdown_connections 2021-02-10T11:45:30.427 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.424 7fa142547700 1 -- 172.21.2.112:0/2479305183 wait complete. 2021-02-10T11:45:30.435 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:45:30.862 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.843 7fc1fddf5700 1 Processor -- start 2021-02-10T11:45:30.862 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.843 7fc1fddf5700 1 -- start start 2021-02-10T11:45:30.862 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.844 7fc1fddf5700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fc1f80d0340 0x7fc1f80ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:30.862 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.844 7fc1fddf5700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc1f80d1c70 0x7fc1f80efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:30.862 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.844 7fc1fddf5700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1f80f0b10 con 0x7fc1f80d1870 2021-02-10T11:45:30.862 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.844 7fc1fddf5700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1f80e5ed0 con 0x7fc1f80d0340 2021-02-10T11:45:30.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.844 7fc1fddf5700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1f81b0be0 con 0x7fc1f80d1c70 2021-02-10T11:45:30.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.844 7fc1f6ffd700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fc1f80d0340 0x7fc1f80ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:30.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.844 7fc1f77fe700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7fc1f80d1870 0x7fc1f81b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:51788/0 (socket says 172.21.2.112:51788) 2021-02-10T11:45:30.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.844 7fc1f7fff700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc1f80d1c70 0x7fc1f80efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:30.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.844 7fc1f77fe700 1 -- 172.21.2.112:0/2861384180 learned_addr learned my addr 172.21.2.112:0/2861384180 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:45:30.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.845 7fc1f7fff700 1 -- 172.21.2.112:0/2861384180 >> v2:172.21.2.120:3300/0 conn(0x7fc1f80d0340 msgr2=0x7fc1f80ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:30.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.845 7fc1f7fff700 1 --2- 172.21.2.112:0/2861384180 >> v2:172.21.2.120:3300/0 conn(0x7fc1f80d0340 0x7fc1f80ef640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:30.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.845 7fc1f7fff700 1 -- 172.21.2.112:0/2861384180 >> v1:172.21.2.120:6789/0 conn(0x7fc1f80d1870 legacy=0x7fc1f81b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:30.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.845 7fc1f7fff700 1 -- 172.21.2.112:0/2861384180 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1f80f20a0 con 0x7fc1f80d1c70 2021-02-10T11:45:30.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.845 7fc1f67fc700 1 -- 172.21.2.112:0/2861384180 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 215686582 0 0) 0x7fc1e8000c60 con 0x7fc1f80d1870 2021-02-10T11:45:30.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.845 7fc1f7fff700 1 --2- 172.21.2.112:0/2861384180 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc1f80d1c70 0x7fc1f80efc60 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7fc1f0006e50 tx=0x7fc1f0009df0).ready entity=mon.2 client_cookie=5d07b299d49fd48a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:30.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.846 7fc1f67fc700 1 -- 172.21.2.112:0/2861384180 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc1f000b510 con 0x7fc1f80d1c70 2021-02-10T11:45:30.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.846 7fc1f67fc700 1 -- 172.21.2.112:0/2861384180 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1f0003030 con 0x7fc1f80d1c70 2021-02-10T11:45:30.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.846 7fc1f67fc700 1 -- 172.21.2.112:0/2861384180 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc1f000b7d0 con 0x7fc1f80d1c70 2021-02-10T11:45:30.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.846 7fc1fddf5700 1 -- 172.21.2.112:0/2861384180 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc1f80d1c70 msgr2=0x7fc1f80efc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:30.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.846 7fc1fddf5700 1 --2- 172.21.2.112:0/2861384180 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc1f80d1c70 0x7fc1f80efc60 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7fc1f0006e50 tx=0x7fc1f0009df0).stop 2021-02-10T11:45:30.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.846 7fc1fddf5700 1 -- 172.21.2.112:0/2861384180 shutdown_connections 2021-02-10T11:45:30.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.846 7fc1fddf5700 1 --2- 172.21.2.112:0/2861384180 >> v2:172.21.2.120:3300/0 conn(0x7fc1f80d0340 0x7fc1f80ef640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:30.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.846 7fc1fddf5700 1 --2- 172.21.2.112:0/2861384180 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc1f80d1c70 0x7fc1f80efc60 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:30.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.846 7fc1fddf5700 1 -- 172.21.2.112:0/2861384180 >> 172.21.2.112:0/2861384180 conn(0x7fc1f80ce450 msgr2=0x7fc1f80d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:30.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.846 7fc1fddf5700 1 -- 172.21.2.112:0/2861384180 shutdown_connections 2021-02-10T11:45:30.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.846 7fc1fddf5700 1 -- 172.21.2.112:0/2861384180 wait complete. 2021-02-10T11:45:30.866 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.861 7fc1fddf5700 1 Processor -- start 2021-02-10T11:45:30.866 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.861 7fc1fddf5700 1 -- start start 2021-02-10T11:45:30.866 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.861 7fc1fddf5700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc1f80beeb0 0x7fc1f80bf2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:30.866 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.861 7fc1fddf5700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc1f818ef70 0x7fc1f818f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:30.866 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.861 7fc1fddf5700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc1f818f990 0x7fc1f80c01d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:30.866 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.861 7fc1fddf5700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1f80bfc50 con 0x7fc1f818ef70 2021-02-10T11:45:30.867 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.861 7fc1fddf5700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1f818fd90 con 0x7fc1f818f990 2021-02-10T11:45:30.867 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.861 7fc1fddf5700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1f80bf8d0 con 0x7fc1f80beeb0 2021-02-10T11:45:30.867 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.861 7fc1f6ffd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc1f818ef70 0x7fc1f818f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:30.867 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.861 7fc1f6ffd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc1f818ef70 0x7fc1f818f370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:58774/0 (socket says 172.21.2.112:58774) 2021-02-10T11:45:30.867 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.861 7fc1f6ffd700 1 -- 172.21.2.112:0/2020808761 learned_addr learned my addr 172.21.2.112:0/2020808761 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:45:30.867 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.862 7fc1f7fff700 1 --2- 172.21.2.112:0/2020808761 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc1f818f990 0x7fc1f80c01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:30.867 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.862 7fc1f77fe700 1 --2- 172.21.2.112:0/2020808761 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc1f80beeb0 0x7fc1f80bf2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:30.867 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.862 7fc1f6ffd700 1 -- 172.21.2.112:0/2020808761 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc1f80beeb0 msgr2=0x7fc1f80bf2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:30.867 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.862 7fc1f6ffd700 1 --2- 172.21.2.112:0/2020808761 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc1f80beeb0 0x7fc1f80bf2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:30.867 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.862 7fc1f6ffd700 1 -- 172.21.2.112:0/2020808761 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc1f818f990 msgr2=0x7fc1f80c01d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:30.867 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.862 7fc1f6ffd700 1 --2- 172.21.2.112:0/2020808761 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc1f818f990 0x7fc1f80c01d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:30.867 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.862 7fc1f6ffd700 1 -- 172.21.2.112:0/2020808761 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1f80c07f0 con 0x7fc1f818ef70 2021-02-10T11:45:30.867 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.862 7fc1f7fff700 1 --2- 172.21.2.112:0/2020808761 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc1f818f990 0x7fc1f80c01d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:45:30.868 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.862 7fc1f6ffd700 1 --2- 172.21.2.112:0/2020808761 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc1f818ef70 0x7fc1f818f370 secure :-1 s=READY pgs=2751 cs=0 l=1 rev1=1 rx=0x7fc1ec006f60 tx=0x7fc1ec00b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:30.868 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.862 7fc1dffff700 1 -- 172.21.2.112:0/2020808761 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc1ec0009b0 con 0x7fc1f818ef70 2021-02-10T11:45:30.868 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.862 7fc1dffff700 1 -- 172.21.2.112:0/2020808761 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1ec00bd60 con 0x7fc1f818ef70 2021-02-10T11:45:30.868 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.862 7fc1dffff700 1 -- 172.21.2.112:0/2020808761 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc1ec002030 con 0x7fc1f818ef70 2021-02-10T11:45:30.868 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.862 7fc1fddf5700 1 -- 172.21.2.112:0/2020808761 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1f80c07f0 con 0x7fc1f818ef70 2021-02-10T11:45:30.868 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.862 7fc1fddf5700 1 -- 172.21.2.112:0/2020808761 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1f80c07f0 con 0x7fc1f818ef70 2021-02-10T11:45:30.868 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.863 7fc1dffff700 1 -- 172.21.2.112:0/2020808761 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc1ec000c90 con 0x7fc1f818ef70 2021-02-10T11:45:30.868 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.863 7fc1dffff700 1 --2- 172.21.2.112:0/2020808761 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc1d8018730 0x7fc1d801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:30.868 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.863 7fc1dffff700 1 -- 172.21.2.112:0/2020808761 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(160..160 src has 1..160) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fc1ec032680 con 0x7fc1f818ef70 2021-02-10T11:45:30.868 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.863 7fc1f77fe700 1 --2- 172.21.2.112:0/2020808761 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc1d8018730 0x7fc1d801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:30.868 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.864 7fc1f77fe700 1 --2- 172.21.2.112:0/2020808761 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc1d8018730 0x7fc1d801ac10 crc :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:30.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.894 7fc1fddf5700 1 -- 172.21.2.112:0/2020808761 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1f80ebc60 con 0x7fc1f818ef70 2021-02-10T11:45:30.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:30.896 7fc1dffff700 1 -- 172.21.2.112:0/2020808761 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc1ec006b00 con 0x7fc1f818ef70 2021-02-10T11:45:31.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:31.022 7fc1fddf5700 1 -- 172.21.2.112:0/2020808761 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc1f80ebc60 con 0x7fc1f818ef70 2021-02-10T11:45:31.024 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:31.022 7fc1dffff700 1 -- 172.21.2.112:0/2020808761 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v160) v1 ==== 74+0+14020 (secure 0 0 0) 0x7fc1ec000c90 con 0x7fc1f818ef70 2021-02-10T11:45:31.024 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:45:31.025 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":160,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:45:30.416633","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":35,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":34,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:45:28.386296","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"160","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":35,"pool_name":"cephfs_data","create_time":"2021-02-10 11:45:29.158381","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"160","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:45:31.025 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:31.024 7fc1fddf5700 1 -- 172.21.2.112:0/2020808761 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc1d8018730 msgr2=0x7fc1d801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:31.025 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:31.024 7fc1fddf5700 1 --2- 172.21.2.112:0/2020808761 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc1d8018730 0x7fc1d801ac10 crc :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:31.025 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:31.024 7fc1fddf5700 1 -- 172.21.2.112:0/2020808761 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc1f818ef70 msgr2=0x7fc1f818f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:31.026 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:31.024 7fc1fddf5700 1 --2- 172.21.2.112:0/2020808761 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc1f818ef70 0x7fc1f818f370 secure :-1 s=READY pgs=2751 cs=0 l=1 rev1=1 rx=0x7fc1ec006f60 tx=0x7fc1ec00b970).stop 2021-02-10T11:45:31.026 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:31.024 7fc1fddf5700 1 -- 172.21.2.112:0/2020808761 shutdown_connections 2021-02-10T11:45:31.026 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:31.024 7fc1fddf5700 1 --2- 172.21.2.112:0/2020808761 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc1d8018730 0x7fc1d801ac10 unknown :-1 s=CLOSED pgs=1255 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:31.026 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:31.024 7fc1fddf5700 1 --2- 172.21.2.112:0/2020808761 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc1f818f990 0x7fc1f80c01d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:31.026 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:31.024 7fc1fddf5700 1 --2- 172.21.2.112:0/2020808761 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc1f818ef70 0x7fc1f818f370 unknown :-1 s=CLOSED pgs=2751 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:31.026 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:31.024 7fc1fddf5700 1 --2- 172.21.2.112:0/2020808761 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc1f80beeb0 0x7fc1f80bf2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:31.026 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:31.024 7fc1fddf5700 1 -- 172.21.2.112:0/2020808761 >> 172.21.2.112:0/2020808761 conn(0x7fc1f80c2130 msgr2=0x7fc1f809e3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:31.026 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:31.024 7fc1fddf5700 1 -- 172.21.2.112:0/2020808761 shutdown_connections 2021-02-10T11:45:31.027 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:31.024 7fc1fddf5700 1 -- 172.21.2.112:0/2020808761 wait complete. 2021-02-10T11:45:31.035 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:45:31.464 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:31.445 7f0e503b2700 1 Processor -- start 2021-02-10T11:45:31.464 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:31.445 7f0e503b2700 1 -- start start 2021-02-10T11:45:31.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:31.445 7f0e503b2700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f0e480d1870 0x7f0e481b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:31.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:31.445 7f0e503b2700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0e481b0a10 0x7f0e480cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:31.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:31.445 7f0e503b2700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e480f0b10 con 0x7f0e480d1c70 2021-02-10T11:45:31.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:31.445 7f0e503b2700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e480e5ed0 con 0x7f0e480d1870 2021-02-10T11:45:31.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:31.445 7f0e503b2700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e480f0650 con 0x7f0e481b0a10 2021-02-10T11:45:31.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:31.446 7f0e4e9cf700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f0e480d1c70 0x7f0e480cf950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:58244/0 (socket says 172.21.2.112:58244) 2021-02-10T11:45:31.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:31.446 7f0e4e9cf700 1 -- 172.21.2.112:0/3502031901 learned_addr learned my addr 172.21.2.112:0/3502031901 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:45:31.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:31.446 7f0e4d9cd700 1 --2- 172.21.2.112:0/3502031901 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0e481b0a10 0x7f0e480cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:31.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:31.446 7f0e4e1ce700 1 --2- 172.21.2.112:0/3502031901 >> v2:172.21.2.120:3300/0 conn(0x7f0e480d1870 0x7f0e481b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:31.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:31.446 7f0e4d1cc700 1 -- 172.21.2.112:0/3502031901 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2308134838 0 0) 0x7f0e3c0009b0 con 0x7f0e480d1c70 2021-02-10T11:45:31.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:31.446 7f0e4d1cc700 1 -- 172.21.2.112:0/3502031901 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e34002fe0 con 0x7f0e480d1c70 2021-02-10T11:45:31.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:31.446 7f0e4d9cd700 1 -- 172.21.2.112:0/3502031901 >> v2:172.21.2.120:3300/0 conn(0x7f0e480d1870 msgr2=0x7f0e481b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:31.465 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:31.446 7f0e4d9cd700 1 --2- 172.21.2.112:0/3502031901 >> v2:172.21.2.120:3300/0 conn(0x7f0e480d1870 0x7f0e481b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:31.466 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:31.446 7f0e4d9cd700 1 -- 172.21.2.112:0/3502031901 >> v1:172.21.2.112:6789/0 conn(0x7f0e480d1c70 legacy=0x7f0e480cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:31.466 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:31.446 7f0e4d9cd700 1 -- 172.21.2.112:0/3502031901 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e480f0870 con 0x7f0e481b0a10 2021-02-10T11:45:31.466 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:31.446 7f0e4d9cd700 1 --2- 172.21.2.112:0/3502031901 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0e481b0a10 0x7f0e480cf330 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7f0e400069d0 tx=0x7f0e40009e40).ready entity=mon.2 client_cookie=a03ea282df4672c8 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:31.466 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:31.447 7f0e4d1cc700 1 -- 172.21.2.112:0/3502031901 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0e4000b510 con 0x7f0e481b0a10 2021-02-10T11:45:31.466 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:31.447 7f0e4d1cc700 1 -- 172.21.2.112:0/3502031901 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e4000b890 con 0x7f0e481b0a10 2021-02-10T11:45:31.466 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:31.447 7f0e4d1cc700 1 -- 172.21.2.112:0/3502031901 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0e4000bdb0 con 0x7f0e481b0a10 2021-02-10T11:45:31.466 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:31.447 7f0e503b2700 1 -- 172.21.2.112:0/3502031901 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0e481b0a10 msgr2=0x7f0e480cf330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:31.466 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:31.447 7f0e503b2700 1 --2- 172.21.2.112:0/3502031901 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0e481b0a10 0x7f0e480cf330 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7f0e400069d0 tx=0x7f0e40009e40).stop 2021-02-10T11:45:31.466 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:31.447 7f0e503b2700 1 -- 172.21.2.112:0/3502031901 shutdown_connections 2021-02-10T11:45:31.466 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:31.447 7f0e503b2700 1 --2- 172.21.2.112:0/3502031901 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0e481b0a10 0x7f0e480cf330 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:31.466 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:31.447 7f0e503b2700 1 --2- 172.21.2.112:0/3502031901 >> v2:172.21.2.120:3300/0 conn(0x7f0e480d1870 0x7f0e481b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:31.466 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:31.447 7f0e503b2700 1 -- 172.21.2.112:0/3502031901 >> 172.21.2.112:0/3502031901 conn(0x7f0e480ce450 msgr2=0x7f0e480d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:31.466 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:31.447 7f0e503b2700 1 -- 172.21.2.112:0/3502031901 shutdown_connections 2021-02-10T11:45:31.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:31.448 7f0e503b2700 1 -- 172.21.2.112:0/3502031901 wait complete. 2021-02-10T11:45:31.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:31.462 7f0e503b2700 1 Processor -- start 2021-02-10T11:45:31.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:31.462 7f0e503b2700 1 -- start start 2021-02-10T11:45:31.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:31.463 7f0e503b2700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0e481942b0 0x7f0e480b3550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:31.468 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:31.463 7f0e503b2700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0e48195070 0x7f0e48195470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:31.469 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:31.463 7f0e503b2700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0e480bb8e0 0x7f0e480bbce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:31.469 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:31.463 7f0e503b2700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e481946b0 con 0x7f0e480bb8e0 2021-02-10T11:45:31.469 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:31.463 7f0e503b2700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e480b3b70 con 0x7f0e481942b0 2021-02-10T11:45:31.469 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:31.463 7f0e503b2700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e480b3d90 con 0x7f0e48195070 2021-02-10T11:45:31.469 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:31.463 7f0e4e9cf700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0e480bb8e0 0x7f0e480bbce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:31.469 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:31.463 7f0e4e9cf700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0e480bb8e0 0x7f0e480bbce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:58790/0 (socket says 172.21.2.112:58790) 2021-02-10T11:45:31.469 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:31.463 7f0e4e9cf700 1 -- 172.21.2.112:0/1343409380 learned_addr learned my addr 172.21.2.112:0/1343409380 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:45:31.469 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:31.463 7f0e4e1ce700 1 --2- 172.21.2.112:0/1343409380 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0e481942b0 0x7f0e480b3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:31.469 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:31.463 7f0e4e9cf700 1 -- 172.21.2.112:0/1343409380 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0e48195070 msgr2=0x7f0e48195470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:31.469 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:31.463 7f0e4e9cf700 1 --2- 172.21.2.112:0/1343409380 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0e48195070 0x7f0e48195470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:31.469 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:31.463 7f0e4e9cf700 1 -- 172.21.2.112:0/1343409380 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0e481942b0 msgr2=0x7f0e480b3550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:31.469 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:31.463 7f0e4e9cf700 1 --2- 172.21.2.112:0/1343409380 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0e481942b0 0x7f0e480b3550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:31.469 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:31.463 7f0e4e9cf700 1 -- 172.21.2.112:0/1343409380 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e481a49e0 con 0x7f0e480bb8e0 2021-02-10T11:45:31.470 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:31.463 7f0e4e9cf700 1 --2- 172.21.2.112:0/1343409380 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0e480bb8e0 0x7f0e480bbce0 secure :-1 s=READY pgs=2753 cs=0 l=1 rev1=1 rx=0x7f0e3c0014b0 tx=0x7f0e3c00a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:31.470 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:31.463 7f0e3affd700 1 -- 172.21.2.112:0/1343409380 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0e3c00acc0 con 0x7f0e480bb8e0 2021-02-10T11:45:31.470 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:31.463 7f0e3affd700 1 -- 172.21.2.112:0/1343409380 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e3c007030 con 0x7f0e480bb8e0 2021-02-10T11:45:31.470 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:31.463 7f0e3affd700 1 -- 172.21.2.112:0/1343409380 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0e3c004620 con 0x7f0e480bb8e0 2021-02-10T11:45:31.470 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:31.463 7f0e503b2700 1 -- 172.21.2.112:0/1343409380 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e481a49e0 con 0x7f0e480bb8e0 2021-02-10T11:45:31.470 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:31.463 7f0e503b2700 1 -- 172.21.2.112:0/1343409380 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e480bc5f0 con 0x7f0e480bb8e0 2021-02-10T11:45:31.470 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:31.464 7f0e3affd700 1 -- 172.21.2.112:0/1343409380 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0e3c0043a0 con 0x7f0e480bb8e0 2021-02-10T11:45:31.470 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:31.464 7f0e3affd700 1 --2- 172.21.2.112:0/1343409380 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0e300186c0 0x7f0e3001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:31.470 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:31.464 7f0e3affd700 1 -- 172.21.2.112:0/1343409380 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f0e3c032780 con 0x7f0e480bb8e0 2021-02-10T11:45:31.470 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:31.464 7f0e4e1ce700 1 --2- 172.21.2.112:0/1343409380 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0e300186c0 0x7f0e3001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:31.470 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:31.465 7f0e4e1ce700 1 --2- 172.21.2.112:0/1343409380 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0e300186c0 0x7f0e3001abc0 crc :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:31.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:31.495 7f0e503b2700 1 -- 172.21.2.112:0/1343409380 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e480bc5f0 con 0x7f0e480bb8e0 2021-02-10T11:45:31.500 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:31.498 7f0e3affd700 1 -- 172.21.2.112:0/1343409380 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0e3c007010 con 0x7f0e480bb8e0 2021-02-10T11:45:31.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:31.623 7f0e503b2700 1 -- 172.21.2.112:0/1343409380 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0e480bc5f0 con 0x7f0e480bb8e0 2021-02-10T11:45:31.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:31.623 7f0e3affd700 1 -- 172.21.2.112:0/1343409380 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v161) v1 ==== 74+0+14007 (secure 0 0 0) 0x7f0e3c007010 con 0x7f0e480bb8e0 2021-02-10T11:45:31.627 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:45:31.627 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":161,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:45:31.418524","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":35,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":34,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:45:28.386296","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"160","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":35,"pool_name":"cephfs_data","create_time":"2021-02-10 11:45:29.158381","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"161","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:45:31.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:31.625 7f0e503b2700 1 -- 172.21.2.112:0/1343409380 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0e300186c0 msgr2=0x7f0e3001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:31.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:31.625 7f0e503b2700 1 --2- 172.21.2.112:0/1343409380 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0e300186c0 0x7f0e3001abc0 crc :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:31.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:31.625 7f0e503b2700 1 -- 172.21.2.112:0/1343409380 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0e480bb8e0 msgr2=0x7f0e480bbce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:31.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:31.625 7f0e503b2700 1 --2- 172.21.2.112:0/1343409380 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0e480bb8e0 0x7f0e480bbce0 secure :-1 s=READY pgs=2753 cs=0 l=1 rev1=1 rx=0x7f0e3c0014b0 tx=0x7f0e3c00a8d0).stop 2021-02-10T11:45:31.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:31.625 7f0e503b2700 1 -- 172.21.2.112:0/1343409380 shutdown_connections 2021-02-10T11:45:31.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:31.625 7f0e503b2700 1 --2- 172.21.2.112:0/1343409380 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0e300186c0 0x7f0e3001abc0 unknown :-1 s=CLOSED pgs=1256 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:31.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:31.625 7f0e503b2700 1 --2- 172.21.2.112:0/1343409380 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0e480bb8e0 0x7f0e480bbce0 unknown :-1 s=CLOSED pgs=2753 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:31.629 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:31.625 7f0e503b2700 1 --2- 172.21.2.112:0/1343409380 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0e48195070 0x7f0e48195470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:31.629 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:31.625 7f0e503b2700 1 --2- 172.21.2.112:0/1343409380 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0e481942b0 0x7f0e480b3550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:31.629 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:31.625 7f0e503b2700 1 -- 172.21.2.112:0/1343409380 >> 172.21.2.112:0/1343409380 conn(0x7f0e480b3040 msgr2=0x7f0e480c3250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:31.629 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:31.625 7f0e503b2700 1 -- 172.21.2.112:0/1343409380 shutdown_connections 2021-02-10T11:45:31.629 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:31.625 7f0e503b2700 1 -- 172.21.2.112:0/1343409380 wait complete. 2021-02-10T11:45:31.637 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2021-02-10T11:45:32.068 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:32.049 7fd56e870700 1 Processor -- start 2021-02-10T11:45:32.069 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:32.049 7fd56e870700 1 -- start start 2021-02-10T11:45:32.069 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:32.049 7fd56e870700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd5680d1c70 0x7fd5680cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:32.069 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:32.049 7fd56e870700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fd5680cf580 0x7fd5680cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:32.069 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:32.049 7fd56e870700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5680fe1c0 con 0x7fd5680d1870 2021-02-10T11:45:32.069 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:32.049 7fd56e870700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5681b0cb0 con 0x7fd5680cf580 2021-02-10T11:45:32.069 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:32.049 7fd56e870700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd568109a60 con 0x7fd5680d1c70 2021-02-10T11:45:32.069 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:32.050 7fd567fff700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fd5680d1870 0x7fd5681b04c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:58258/0 (socket says 172.21.2.112:58258) 2021-02-10T11:45:32.069 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:32.050 7fd567fff700 1 -- 172.21.2.112:0/2459970844 learned_addr learned my addr 172.21.2.112:0/2459970844 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:45:32.069 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:32.050 7fd56ce8d700 1 --2- 172.21.2.112:0/2459970844 >> v2:172.21.2.120:3300/0 conn(0x7fd5680cf580 0x7fd5680cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:32.069 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:32.050 7fd5677fe700 1 --2- 172.21.2.112:0/2459970844 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd5680d1c70 0x7fd5680cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:32.069 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:32.050 7fd566ffd700 1 -- 172.21.2.112:0/2459970844 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2642899647 0 0) 0x7fd558001180 con 0x7fd5680d1870 2021-02-10T11:45:32.069 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:32.050 7fd566ffd700 1 -- 172.21.2.112:0/2459970844 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd550002fe0 con 0x7fd5680d1870 2021-02-10T11:45:32.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:32.050 7fd56ce8d700 1 -- 172.21.2.112:0/2459970844 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd5680d1c70 msgr2=0x7fd5680cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:32.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:32.050 7fd56ce8d700 1 --2- 172.21.2.112:0/2459970844 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd5680d1c70 0x7fd5680cef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:32.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:32.050 7fd56ce8d700 1 -- 172.21.2.112:0/2459970844 >> v1:172.21.2.112:6789/0 conn(0x7fd5680d1870 legacy=0x7fd5681b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:32.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:32.050 7fd56ce8d700 1 -- 172.21.2.112:0/2459970844 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd568088070 con 0x7fd5680cf580 2021-02-10T11:45:32.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:32.050 7fd56ce8d700 1 --2- 172.21.2.112:0/2459970844 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd5680cf580 0x7fd5680cf980 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 rx=0x7fd560006e50 tx=0x7fd560009e70).ready entity=mon.1 client_cookie=ccf88d8170458eac server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:32.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:32.051 7fd566ffd700 1 -- 172.21.2.112:0/2459970844 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd560006a20 con 0x7fd5680cf580 2021-02-10T11:45:32.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:32.051 7fd566ffd700 1 -- 172.21.2.112:0/2459970844 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd560006790 con 0x7fd5680cf580 2021-02-10T11:45:32.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:32.051 7fd566ffd700 1 -- 172.21.2.112:0/2459970844 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd56000b700 con 0x7fd5680cf580 2021-02-10T11:45:32.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:32.051 7fd56e870700 1 -- 172.21.2.112:0/2459970844 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd5680cf580 msgr2=0x7fd5680cf980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:32.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:32.051 7fd56e870700 1 --2- 172.21.2.112:0/2459970844 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd5680cf580 0x7fd5680cf980 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 rx=0x7fd560006e50 tx=0x7fd560009e70).stop 2021-02-10T11:45:32.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:32.051 7fd56e870700 1 -- 172.21.2.112:0/2459970844 shutdown_connections 2021-02-10T11:45:32.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:32.051 7fd56e870700 1 --2- 172.21.2.112:0/2459970844 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd5680cf580 0x7fd5680cf980 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:32.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:32.051 7fd56e870700 1 --2- 172.21.2.112:0/2459970844 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd5680d1c70 0x7fd5680cef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:32.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:32.051 7fd56e870700 1 -- 172.21.2.112:0/2459970844 >> 172.21.2.112:0/2459970844 conn(0x7fd5680ce420 msgr2=0x7fd5680d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:32.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:32.051 7fd56e870700 1 -- 172.21.2.112:0/2459970844 shutdown_connections 2021-02-10T11:45:32.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:32.051 7fd56e870700 1 -- 172.21.2.112:0/2459970844 wait complete. 2021-02-10T11:45:32.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:32.066 7fd56e870700 1 Processor -- start 2021-02-10T11:45:32.073 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:32.066 7fd56e870700 1 -- start start 2021-02-10T11:45:32.073 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:32.066 7fd56e870700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd56809bcc0 0x7fd56809c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:32.073 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:32.066 7fd56e870700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd5680a0820 0x7fd5680a0c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:32.073 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:32.066 7fd56e870700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd5680a1240 0x7fd5680858d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:32.073 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:32.066 7fd56e870700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5680a66b0 con 0x7fd5680a0820 2021-02-10T11:45:32.073 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:32.066 7fd56e870700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5681a49b0 con 0x7fd56809bcc0 2021-02-10T11:45:32.073 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:32.066 7fd56e870700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5681a4b90 con 0x7fd5680a1240 2021-02-10T11:45:32.073 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:32.067 7fd5677fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd5680a0820 0x7fd5680a0c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:32.073 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:32.067 7fd5677fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd5680a0820 0x7fd5680a0c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:58804/0 (socket says 172.21.2.112:58804) 2021-02-10T11:45:32.073 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:32.067 7fd5677fe700 1 -- 172.21.2.112:0/3169168041 learned_addr learned my addr 172.21.2.112:0/3169168041 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:45:32.074 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:32.067 7fd567fff700 1 --2- 172.21.2.112:0/3169168041 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd56809bcc0 0x7fd56809c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:32.074 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:32.067 7fd56ce8d700 1 --2- 172.21.2.112:0/3169168041 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd5680a1240 0x7fd5680858d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:32.074 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:32.067 7fd5677fe700 1 -- 172.21.2.112:0/3169168041 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd5680a1240 msgr2=0x7fd5680858d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:32.074 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:32.067 7fd5677fe700 1 --2- 172.21.2.112:0/3169168041 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd5680a1240 0x7fd5680858d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:32.074 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:32.067 7fd5677fe700 1 -- 172.21.2.112:0/3169168041 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd56809bcc0 msgr2=0x7fd56809c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:32.074 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:32.067 7fd5677fe700 1 --2- 172.21.2.112:0/3169168041 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd56809bcc0 0x7fd56809c0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:32.074 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:32.067 7fd5677fe700 1 -- 172.21.2.112:0/3169168041 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5680a1640 con 0x7fd5680a0820 2021-02-10T11:45:32.074 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:32.067 7fd5677fe700 1 --2- 172.21.2.112:0/3169168041 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd5680a0820 0x7fd5680a0c20 secure :-1 s=READY pgs=2755 cs=0 l=1 rev1=1 rx=0x7fd55c006f60 tx=0x7fd55c00b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:32.074 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:32.067 7fd564ff9700 1 -- 172.21.2.112:0/3169168041 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd55c0009b0 con 0x7fd5680a0820 2021-02-10T11:45:32.074 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:32.067 7fd564ff9700 1 -- 172.21.2.112:0/3169168041 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd55c00bd60 con 0x7fd5680a0820 2021-02-10T11:45:32.074 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:32.068 7fd56e870700 1 -- 172.21.2.112:0/3169168041 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5680a1640 con 0x7fd5680a0820 2021-02-10T11:45:32.074 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:32.068 7fd564ff9700 1 -- 172.21.2.112:0/3169168041 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd55c002030 con 0x7fd5680a0820 2021-02-10T11:45:32.075 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:32.068 7fd56e870700 1 -- 172.21.2.112:0/3169168041 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5680861e0 con 0x7fd5680a0820 2021-02-10T11:45:32.075 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:32.068 7fd564ff9700 1 -- 172.21.2.112:0/3169168041 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd55c000c90 con 0x7fd5680a0820 2021-02-10T11:45:32.075 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:32.068 7fd564ff9700 1 --2- 172.21.2.112:0/3169168041 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd5480186c0 0x7fd54801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:32.075 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:32.068 7fd564ff9700 1 -- 172.21.2.112:0/3169168041 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fd55c032620 con 0x7fd5680a0820 2021-02-10T11:45:32.075 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:32.069 7fd567fff700 1 --2- 172.21.2.112:0/3169168041 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd5480186c0 0x7fd54801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:32.075 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:32.069 7fd567fff700 1 --2- 172.21.2.112:0/3169168041 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd5480186c0 0x7fd54801abc0 crc :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:32.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:32.099 7fd56e870700 1 -- 172.21.2.112:0/3169168041 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5680fe1c0 con 0x7fd5680a0820 2021-02-10T11:45:32.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:32.102 7fd564ff9700 1 -- 172.21.2.112:0/3169168041 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd55c006b00 con 0x7fd5680a0820 2021-02-10T11:45:32.242 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:32.239 7fd56e870700 1 -- 172.21.2.112:0/3169168041 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"} v 0) v1 -- 0x7fd568086d30 con 0x7fd5680a0820 2021-02-10T11:45:32.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:32.787 7fd564ff9700 1 -- 172.21.2.112:0/3169168041 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"}]=0 v139) v1 ==== 118+0+0 (secure 0 0 0) 0x7fd55c000c90 con 0x7fd5680a0820 2021-02-10T11:45:32.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:32.809 7fd56e870700 1 -- 172.21.2.112:0/3169168041 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd5480186c0 msgr2=0x7fd54801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:32.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:32.809 7fd56e870700 1 --2- 172.21.2.112:0/3169168041 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd5480186c0 0x7fd54801abc0 crc :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:32.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:32.809 7fd56e870700 1 -- 172.21.2.112:0/3169168041 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd5680a0820 msgr2=0x7fd5680a0c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:32.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:32.809 7fd56e870700 1 --2- 172.21.2.112:0/3169168041 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd5680a0820 0x7fd5680a0c20 secure :-1 s=READY pgs=2755 cs=0 l=1 rev1=1 rx=0x7fd55c006f60 tx=0x7fd55c00b970).stop 2021-02-10T11:45:32.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:32.809 7fd56e870700 1 -- 172.21.2.112:0/3169168041 shutdown_connections 2021-02-10T11:45:32.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:32.809 7fd56e870700 1 --2- 172.21.2.112:0/3169168041 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd5480186c0 0x7fd54801abc0 unknown :-1 s=CLOSED pgs=1257 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:32.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:32.809 7fd56e870700 1 --2- 172.21.2.112:0/3169168041 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd5680a1240 0x7fd5680858d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:32.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:32.809 7fd56e870700 1 --2- 172.21.2.112:0/3169168041 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd5680a0820 0x7fd5680a0c20 unknown :-1 s=CLOSED pgs=2755 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:32.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:32.809 7fd56e870700 1 --2- 172.21.2.112:0/3169168041 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd56809bcc0 0x7fd56809c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:32.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:32.809 7fd56e870700 1 -- 172.21.2.112:0/3169168041 >> 172.21.2.112:0/3169168041 conn(0x7fd5680bef00 msgr2=0x7fd56809d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:32.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:32.809 7fd56e870700 1 -- 172.21.2.112:0/3169168041 shutdown_connections 2021-02-10T11:45:32.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:32.809 7fd56e870700 1 -- 172.21.2.112:0/3169168041 wait complete. 2021-02-10T11:45:32.820 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:45:33.250 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.228 7f6b05966700 1 Processor -- start 2021-02-10T11:45:33.250 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.228 7f6b05966700 1 -- start start 2021-02-10T11:45:33.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.229 7f6b05966700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f6b000d1c70 0x7f6b000f20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:33.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.229 7f6b05966700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b000f0b10 con 0x7f6b000d1870 2021-02-10T11:45:33.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.229 7f6b05966700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b001b0be0 con 0x7f6b000d0340 2021-02-10T11:45:33.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.229 7f6b05966700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b000e5ed0 con 0x7f6b000d1c70 2021-02-10T11:45:33.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.229 7f6aff574700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f6b000d1870 0x7f6b001b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:58272/0 (socket says 172.21.2.112:58272) 2021-02-10T11:45:33.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.229 7f6aff574700 1 -- 172.21.2.112:0/2217081139 learned_addr learned my addr 172.21.2.112:0/2217081139 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:45:33.251 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.229 7f6afe572700 1 -- 172.21.2.112:0/2217081139 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2075900044 0 0) 0x7f6ae8001060 con 0x7f6b000d1870 2021-02-10T11:45:33.252 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.229 7f6afe572700 1 -- 172.21.2.112:0/2217081139 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6aec002fe0 con 0x7f6b000d1870 2021-02-10T11:45:33.252 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.229 7f6affd75700 1 --2- 172.21.2.112:0/2217081139 >> v2:172.21.2.120:3300/0 conn(0x7f6b000d1c70 0x7f6b000f20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:33.252 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.230 7f6afe572700 1 -- 172.21.2.112:0/2217081139 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2878121761 0 0) 0x7f6ae8001500 con 0x7f6b000d1870 2021-02-10T11:45:33.252 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.230 7f6afe572700 1 -- 172.21.2.112:0/2217081139 >> v2:172.21.2.120:3300/0 conn(0x7f6b000d1c70 msgr2=0x7f6b000f20a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:33.252 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.230 7f6afe572700 1 --2- 172.21.2.112:0/2217081139 >> v2:172.21.2.120:3300/0 conn(0x7f6b000d1c70 0x7f6b000f20a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:33.252 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.230 7f6afe572700 1 -- 172.21.2.112:0/2217081139 >> v1:172.21.2.120:6789/0 conn(0x7f6b000d0340 legacy=0x7f6b000ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:33.253 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.230 7f6afe572700 1 -- 172.21.2.112:0/2217081139 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b000efe30 con 0x7f6b000d1870 2021-02-10T11:45:33.253 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.230 7f6afe572700 1 -- 172.21.2.112:0/2217081139 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f6ae8001a00 con 0x7f6b000d1870 2021-02-10T11:45:33.253 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.230 7f6afe572700 1 -- 172.21.2.112:0/2217081139 >> v1:172.21.2.112:6789/0 conn(0x7f6b000d1870 legacy=0x7f6b001b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:33.253 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.230 7f6afe572700 1 --2- 172.21.2.112:0/2217081139 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6aec003aa0 0x7f6aec005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:33.253 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.230 7f6afe572700 1 --2- 172.21.2.112:0/2217081139 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6aec006790 0x7f6aec008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:33.253 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.230 7f6afe572700 1 --2- 172.21.2.112:0/2217081139 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6aec009590 0x7f6aec00bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:33.253 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.230 7f6afe572700 1 -- 172.21.2.112:0/2217081139 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6aec00c300 con 0x7f6aec006790 2021-02-10T11:45:33.253 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.230 7f6afe572700 1 -- 172.21.2.112:0/2217081139 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6aec00c520 con 0x7f6aec003aa0 2021-02-10T11:45:33.254 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.230 7f6afe572700 1 -- 172.21.2.112:0/2217081139 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6aec00c740 con 0x7f6aec009590 2021-02-10T11:45:33.254 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.230 7f6afed73700 1 --2- 172.21.2.112:0/2217081139 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6aec006790 0x7f6aec008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:33.254 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.230 7f6aff574700 1 --2- 172.21.2.112:0/2217081139 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6aec003aa0 0x7f6aec005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:33.254 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.230 7f6affd75700 1 --2- 172.21.2.112:0/2217081139 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6aec009590 0x7f6aec00bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:33.254 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.230 7f6afed73700 1 -- 172.21.2.112:0/2217081139 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6aec009590 msgr2=0x7f6aec00bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:33.254 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.230 7f6afed73700 1 --2- 172.21.2.112:0/2217081139 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6aec009590 0x7f6aec00bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:33.254 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.230 7f6afed73700 1 -- 172.21.2.112:0/2217081139 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6aec003aa0 msgr2=0x7f6aec005ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:33.254 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.230 7f6afed73700 1 --2- 172.21.2.112:0/2217081139 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6aec003aa0 0x7f6aec005ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:33.255 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.230 7f6afed73700 1 -- 172.21.2.112:0/2217081139 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6aec050980 con 0x7f6aec006790 2021-02-10T11:45:33.255 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.230 7f6affd75700 1 --2- 172.21.2.112:0/2217081139 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6aec009590 0x7f6aec00bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:45:33.255 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.230 7f6affd75700 1 -- 172.21.2.112:0/2217081139 reap_dead start 2021-02-10T11:45:33.255 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.230 7f6afed73700 1 --2- 172.21.2.112:0/2217081139 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6aec006790 0x7f6aec008c90 secure :-1 s=READY pgs=2757 cs=0 l=1 rev1=1 rx=0x7f6af0006ab0 tx=0x7f6af0009b40).ready entity=mon.0 client_cookie=9f59c063fd6737e7 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:33.255 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.230 7f6afe572700 1 -- 172.21.2.112:0/2217081139 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6af000b2e0 con 0x7f6aec006790 2021-02-10T11:45:33.255 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.230 7f6afe572700 1 -- 172.21.2.112:0/2217081139 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6af000b520 con 0x7f6aec006790 2021-02-10T11:45:33.255 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.230 7f6afe572700 1 -- 172.21.2.112:0/2217081139 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6af0000960 con 0x7f6aec006790 2021-02-10T11:45:33.255 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.230 7f6b05966700 1 -- 172.21.2.112:0/2217081139 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6aec006790 msgr2=0x7f6aec008c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:33.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.230 7f6b05966700 1 --2- 172.21.2.112:0/2217081139 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6aec006790 0x7f6aec008c90 secure :-1 s=READY pgs=2757 cs=0 l=1 rev1=1 rx=0x7f6af0006ab0 tx=0x7f6af0009b40).stop 2021-02-10T11:45:33.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.231 7f6b05966700 1 -- 172.21.2.112:0/2217081139 shutdown_connections 2021-02-10T11:45:33.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.231 7f6b05966700 1 --2- 172.21.2.112:0/2217081139 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6aec006790 0x7f6aec008c90 unknown :-1 s=CLOSED pgs=2757 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:33.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.231 7f6b05966700 1 -- 172.21.2.112:0/2217081139 >> 172.21.2.112:0/2217081139 conn(0x7f6b000ce450 msgr2=0x7f6b000d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:33.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.231 7f6b05966700 1 -- 172.21.2.112:0/2217081139 shutdown_connections 2021-02-10T11:45:33.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.231 7f6b05966700 1 -- 172.21.2.112:0/2217081139 wait complete. 2021-02-10T11:45:33.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.246 7f6b05966700 1 Processor -- start 2021-02-10T11:45:33.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.246 7f6b05966700 1 -- start start 2021-02-10T11:45:33.257 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.246 7f6b05966700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6b00195650 0x7f6b000b5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:33.257 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.246 7f6b05966700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6b000b5b80 0x7f6b001a2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:33.257 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.246 7f6b05966700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6b000acfa0 0x7f6b001a3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:33.257 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.246 7f6b05966700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b000b6010 con 0x7f6b000b5b80 2021-02-10T11:45:33.257 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.246 7f6b05966700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b001a3a60 con 0x7f6b00195650 2021-02-10T11:45:33.257 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.246 7f6b05966700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b001a3c40 con 0x7f6b000acfa0 2021-02-10T11:45:33.257 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.246 7f6afed73700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6b000b5b80 0x7f6b001a2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:33.257 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.246 7f6afed73700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6b000b5b80 0x7f6b001a2e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:58824/0 (socket says 172.21.2.112:58824) 2021-02-10T11:45:33.257 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.246 7f6afed73700 1 -- 172.21.2.112:0/2361824607 learned_addr learned my addr 172.21.2.112:0/2361824607 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:45:33.258 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.247 7f6aff574700 1 --2- 172.21.2.112:0/2361824607 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6b00195650 0x7f6b000b5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:33.258 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.247 7f6affd75700 1 --2- 172.21.2.112:0/2361824607 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6b000acfa0 0x7f6b001a3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:33.258 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.247 7f6afed73700 1 -- 172.21.2.112:0/2361824607 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6b000acfa0 msgr2=0x7f6b001a3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:33.258 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.247 7f6afed73700 1 --2- 172.21.2.112:0/2361824607 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6b000acfa0 0x7f6b001a3440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:33.258 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.247 7f6afed73700 1 -- 172.21.2.112:0/2361824607 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6b00195650 msgr2=0x7f6b000b5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:33.258 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.247 7f6afed73700 1 --2- 172.21.2.112:0/2361824607 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6b00195650 0x7f6b000b5560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:33.258 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.247 7f6afed73700 1 -- 172.21.2.112:0/2361824607 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b000bcae0 con 0x7f6b000b5b80 2021-02-10T11:45:33.258 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.247 7f6afed73700 1 --2- 172.21.2.112:0/2361824607 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6b000b5b80 0x7f6b001a2e20 secure :-1 s=READY pgs=2758 cs=0 l=1 rev1=1 rx=0x7f6af00093a0 tx=0x7f6af000b9c0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:33.258 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.247 7f6af7fff700 1 -- 172.21.2.112:0/2361824607 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6af000bdb0 con 0x7f6b000b5b80 2021-02-10T11:45:33.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.247 7f6af7fff700 1 -- 172.21.2.112:0/2361824607 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6af0000a10 con 0x7f6b000b5b80 2021-02-10T11:45:33.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.247 7f6af7fff700 1 -- 172.21.2.112:0/2361824607 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6af00054a0 con 0x7f6b000b5b80 2021-02-10T11:45:33.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.247 7f6b05966700 1 -- 172.21.2.112:0/2361824607 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b000bcae0 con 0x7f6b000b5b80 2021-02-10T11:45:33.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.247 7f6b05966700 1 -- 172.21.2.112:0/2361824607 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b000bcae0 con 0x7f6b000b5b80 2021-02-10T11:45:33.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.248 7f6af7fff700 1 -- 172.21.2.112:0/2361824607 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6af0006750 con 0x7f6b000b5b80 2021-02-10T11:45:33.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.248 7f6af7fff700 1 --2- 172.21.2.112:0/2361824607 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6ae40186c0 0x7f6ae401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:33.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.248 7f6af7fff700 1 -- 172.21.2.112:0/2361824607 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f6af0032580 con 0x7f6b000b5b80 2021-02-10T11:45:33.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.248 7f6aff574700 1 --2- 172.21.2.112:0/2361824607 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6ae40186c0 0x7f6ae401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:33.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.249 7f6aff574700 1 --2- 172.21.2.112:0/2361824607 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6ae40186c0 0x7f6ae401abc0 crc :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:33.282 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.279 7f6b05966700 1 -- 172.21.2.112:0/2361824607 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b001a3c40 con 0x7f6b000b5b80 2021-02-10T11:45:33.284 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.282 7f6af7fff700 1 -- 172.21.2.112:0/2361824607 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6af0025fb0 con 0x7f6b000b5b80 2021-02-10T11:45:33.414 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.412 7f6b05966700 1 -- 172.21.2.112:0/2361824607 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6b001a3c40 con 0x7f6b000b5b80 2021-02-10T11:45:33.415 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.412 7f6af7fff700 1 -- 172.21.2.112:0/2361824607 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 139 v139) v1 ==== 95+0+2105 (secure 0 0 0) 0x7f6af0025fb0 con 0x7f6b000b5b80 2021-02-10T11:45:33.415 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:45:33.415 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":139,"default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":12306,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/2635651262","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":2635651262},{"type":"v1","addr":"172.21.2.112:6833","nonce":2635651262}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":135}],"filesystems":[{"mdsmap":{"epoch":139,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:45:30.416714","modified":"2021-02-10 11:45:32.783510","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8920},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8920":{"gid":8920,"name":"b","rank":0,"incarnation":136,"state":"up:active","state_seq":3,"addr":"172.21.2.120:6835/555399099","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":555399099},{"type":"v1","addr":"172.21.2.120:6835","nonce":555399099}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[35],"metadata_pool":34,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":17}]} 2021-02-10T11:45:33.416 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 139 2021-02-10T11:45:33.417 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.414 7f6b05966700 1 -- 172.21.2.112:0/2361824607 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6ae40186c0 msgr2=0x7f6ae401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:33.417 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.414 7f6b05966700 1 --2- 172.21.2.112:0/2361824607 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6ae40186c0 0x7f6ae401abc0 crc :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:33.417 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.414 7f6b05966700 1 -- 172.21.2.112:0/2361824607 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6b000b5b80 msgr2=0x7f6b001a2e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:33.417 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.414 7f6b05966700 1 --2- 172.21.2.112:0/2361824607 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6b000b5b80 0x7f6b001a2e20 secure :-1 s=READY pgs=2758 cs=0 l=1 rev1=1 rx=0x7f6af00093a0 tx=0x7f6af000b9c0).stop 2021-02-10T11:45:33.417 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.414 7f6b05966700 1 -- 172.21.2.112:0/2361824607 shutdown_connections 2021-02-10T11:45:33.417 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.414 7f6b05966700 1 --2- 172.21.2.112:0/2361824607 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6ae40186c0 0x7f6ae401abc0 unknown :-1 s=CLOSED pgs=1258 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:33.417 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.414 7f6b05966700 1 --2- 172.21.2.112:0/2361824607 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6b000acfa0 0x7f6b001a3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:33.417 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.414 7f6b05966700 1 --2- 172.21.2.112:0/2361824607 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6b000b5b80 0x7f6b001a2e20 unknown :-1 s=CLOSED pgs=2758 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:33.417 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.414 7f6b05966700 1 --2- 172.21.2.112:0/2361824607 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6b00195650 0x7f6b000b5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:33.417 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.414 7f6b05966700 1 -- 172.21.2.112:0/2361824607 >> 172.21.2.112:0/2361824607 conn(0x7f6b000b3040 msgr2=0x7f6b000c3250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:33.418 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.414 7f6b05966700 1 -- 172.21.2.112:0/2361824607 shutdown_connections 2021-02-10T11:45:33.418 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.414 7f6b05966700 1 -- 172.21.2.112:0/2361824607 wait complete. 2021-02-10T11:45:33.426 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:45:33.856 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.836 7fdf6b387700 1 Processor -- start 2021-02-10T11:45:33.857 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.837 7fdf6b387700 1 -- start start 2021-02-10T11:45:33.857 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.837 7fdf6b387700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fdf640d0340 0x7fdf640ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:33.857 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.837 7fdf6b387700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdf640d1c70 0x7fdf640efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:33.857 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.837 7fdf6b387700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf640f0b10 con 0x7fdf640d1870 2021-02-10T11:45:33.857 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.837 7fdf6b387700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf640e5ed0 con 0x7fdf640d0340 2021-02-10T11:45:33.857 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.837 7fdf6b387700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf641b0be0 con 0x7fdf640d1c70 2021-02-10T11:45:33.857 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.838 7fdf689a2700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fdf640d0340 0x7fdf640ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:33.857 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.838 7fdf699a4700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdf640d1c70 0x7fdf640efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:33.857 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.838 7fdf691a3700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7fdf640d1870 0x7fdf641b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:51848/0 (socket says 172.21.2.112:51848) 2021-02-10T11:45:33.857 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.838 7fdf691a3700 1 -- 172.21.2.112:0/1417761089 learned_addr learned my addr 172.21.2.112:0/1417761089 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:45:33.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.838 7fdf689a2700 1 -- 172.21.2.112:0/1417761089 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdf640d1c70 msgr2=0x7fdf640efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:33.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.838 7fdf689a2700 1 --2- 172.21.2.112:0/1417761089 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdf640d1c70 0x7fdf640efc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:33.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.838 7fdf689a2700 1 -- 172.21.2.112:0/1417761089 >> v1:172.21.2.120:6789/0 conn(0x7fdf640d1870 legacy=0x7fdf641b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:33.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.838 7fdf689a2700 1 -- 172.21.2.112:0/1417761089 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf640f20a0 con 0x7fdf640d0340 2021-02-10T11:45:33.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.838 7fdf689a2700 1 --2- 172.21.2.112:0/1417761089 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fdf640d0340 0x7fdf640ef640 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 rx=0x7fdf54006650 tx=0x7fdf54009a80).ready entity=mon.1 client_cookie=5106db686e433f6b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:33.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.839 7fdf5bfff700 1 -- 172.21.2.112:0/1417761089 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdf5400c0c0 con 0x7fdf640d0340 2021-02-10T11:45:33.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.839 7fdf5bfff700 1 -- 172.21.2.112:0/1417761089 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf5400c340 con 0x7fdf640d0340 2021-02-10T11:45:33.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.839 7fdf5bfff700 1 -- 172.21.2.112:0/1417761089 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdf5400c860 con 0x7fdf640d0340 2021-02-10T11:45:33.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.839 7fdf6b387700 1 -- 172.21.2.112:0/1417761089 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fdf640d0340 msgr2=0x7fdf640ef640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:33.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.839 7fdf6b387700 1 --2- 172.21.2.112:0/1417761089 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fdf640d0340 0x7fdf640ef640 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 rx=0x7fdf54006650 tx=0x7fdf54009a80).stop 2021-02-10T11:45:33.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.839 7fdf6b387700 1 -- 172.21.2.112:0/1417761089 shutdown_connections 2021-02-10T11:45:33.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.839 7fdf6b387700 1 --2- 172.21.2.112:0/1417761089 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fdf640d0340 0x7fdf640ef640 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:33.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.839 7fdf6b387700 1 --2- 172.21.2.112:0/1417761089 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdf640d1c70 0x7fdf640efc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:33.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.839 7fdf6b387700 1 -- 172.21.2.112:0/1417761089 >> 172.21.2.112:0/1417761089 conn(0x7fdf640ce450 msgr2=0x7fdf640d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:33.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.840 7fdf6b387700 1 -- 172.21.2.112:0/1417761089 shutdown_connections 2021-02-10T11:45:33.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.840 7fdf6b387700 1 -- 172.21.2.112:0/1417761089 wait complete. 2021-02-10T11:45:33.860 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.854 7fdf6b387700 1 Processor -- start 2021-02-10T11:45:33.860 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.854 7fdf6b387700 1 -- start start 2021-02-10T11:45:33.860 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.855 7fdf6b387700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdf640ada90 0x7fdf640ade90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:33.860 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.855 7fdf6b387700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdf64099d20 0x7fdf6409a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:33.860 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.855 7fdf6b387700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fdf6409a740 0x7fdf64097bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:33.861 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.855 7fdf6b387700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf640997a0 con 0x7fdf64099d20 2021-02-10T11:45:33.861 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.855 7fdf6b387700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf6418f0c0 con 0x7fdf6409a740 2021-02-10T11:45:33.861 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.855 7fdf6b387700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf6409ab40 con 0x7fdf640ada90 2021-02-10T11:45:33.861 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.855 7fdf689a2700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdf64099d20 0x7fdf6409a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:33.861 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.855 7fdf689a2700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdf64099d20 0x7fdf6409a120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:58834/0 (socket says 172.21.2.112:58834) 2021-02-10T11:45:33.861 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.855 7fdf689a2700 1 -- 172.21.2.112:0/1918503818 learned_addr learned my addr 172.21.2.112:0/1918503818 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:45:33.861 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.855 7fdf689a2700 1 -- 172.21.2.112:0/1918503818 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdf640ada90 msgr2=0x7fdf640ade90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:33.861 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.855 7fdf689a2700 1 --2- 172.21.2.112:0/1918503818 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdf640ada90 0x7fdf640ade90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:33.861 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.855 7fdf689a2700 1 -- 172.21.2.112:0/1918503818 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fdf6409a740 msgr2=0x7fdf64097bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:33.861 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.855 7fdf689a2700 1 --2- 172.21.2.112:0/1918503818 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fdf6409a740 0x7fdf64097bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:33.861 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.855 7fdf689a2700 1 -- 172.21.2.112:0/1918503818 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf6409ad60 con 0x7fdf64099d20 2021-02-10T11:45:33.861 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.855 7fdf689a2700 1 --2- 172.21.2.112:0/1918503818 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdf64099d20 0x7fdf6409a120 secure :-1 s=READY pgs=2759 cs=0 l=1 rev1=1 rx=0x7fdf54009630 tx=0x7fdf5400b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:33.861 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.856 7fdf59ffb700 1 -- 172.21.2.112:0/1918503818 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdf5400bd70 con 0x7fdf64099d20 2021-02-10T11:45:33.862 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.856 7fdf59ffb700 1 -- 172.21.2.112:0/1918503818 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf54000960 con 0x7fdf64099d20 2021-02-10T11:45:33.862 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.856 7fdf59ffb700 1 -- 172.21.2.112:0/1918503818 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdf54005560 con 0x7fdf64099d20 2021-02-10T11:45:33.862 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.856 7fdf6b387700 1 -- 172.21.2.112:0/1918503818 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf6409ad60 con 0x7fdf64099d20 2021-02-10T11:45:33.862 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.856 7fdf6b387700 1 -- 172.21.2.112:0/1918503818 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf640984d0 con 0x7fdf64099d20 2021-02-10T11:45:33.862 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.856 7fdf59ffb700 1 -- 172.21.2.112:0/1918503818 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fdf5400a000 con 0x7fdf64099d20 2021-02-10T11:45:33.862 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.856 7fdf59ffb700 1 --2- 172.21.2.112:0/1918503818 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fdf440186c0 0x7fdf4401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:33.862 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.856 7fdf59ffb700 1 -- 172.21.2.112:0/1918503818 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fdf54032580 con 0x7fdf64099d20 2021-02-10T11:45:33.862 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.857 7fdf691a3700 1 --2- 172.21.2.112:0/1918503818 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fdf440186c0 0x7fdf4401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:33.862 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.857 7fdf691a3700 1 --2- 172.21.2.112:0/1918503818 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fdf440186c0 0x7fdf4401abc0 crc :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:33.889 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.887 7fdf6b387700 1 -- 172.21.2.112:0/1918503818 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf640ebc00 con 0x7fdf64099d20 2021-02-10T11:45:33.892 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:33.889 7fdf59ffb700 1 -- 172.21.2.112:0/1918503818 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fdf54009340 con 0x7fdf64099d20 2021-02-10T11:45:34.017 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:34.014 7fdf6b387700 1 -- 172.21.2.112:0/1918503818 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdf640f0b10 con 0x7fdf64099d20 2021-02-10T11:45:34.017 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:34.015 7fdf59ffb700 1 -- 172.21.2.112:0/1918503818 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v161) v1 ==== 74+0+14007 (secure 0 0 0) 0x7fdf5400a000 con 0x7fdf64099d20 2021-02-10T11:45:34.018 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:45:34.018 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":161,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:45:31.418524","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":35,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":34,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:45:28.386296","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"160","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":35,"pool_name":"cephfs_data","create_time":"2021-02-10 11:45:29.158381","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"161","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:45:34.019 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:34.016 7fdf6b387700 1 -- 172.21.2.112:0/1918503818 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fdf440186c0 msgr2=0x7fdf4401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:34.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:34.016 7fdf6b387700 1 --2- 172.21.2.112:0/1918503818 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fdf440186c0 0x7fdf4401abc0 crc :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:34.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:34.016 7fdf6b387700 1 -- 172.21.2.112:0/1918503818 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdf64099d20 msgr2=0x7fdf6409a120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:34.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:34.016 7fdf6b387700 1 --2- 172.21.2.112:0/1918503818 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdf64099d20 0x7fdf6409a120 secure :-1 s=READY pgs=2759 cs=0 l=1 rev1=1 rx=0x7fdf54009630 tx=0x7fdf5400b980).stop 2021-02-10T11:45:34.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:34.017 7fdf6b387700 1 -- 172.21.2.112:0/1918503818 shutdown_connections 2021-02-10T11:45:34.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:34.017 7fdf6b387700 1 --2- 172.21.2.112:0/1918503818 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fdf440186c0 0x7fdf4401abc0 unknown :-1 s=CLOSED pgs=1259 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:34.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:34.017 7fdf6b387700 1 --2- 172.21.2.112:0/1918503818 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fdf6409a740 0x7fdf64097bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:34.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:34.017 7fdf6b387700 1 --2- 172.21.2.112:0/1918503818 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fdf64099d20 0x7fdf6409a120 unknown :-1 s=CLOSED pgs=2759 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:34.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:34.017 7fdf6b387700 1 --2- 172.21.2.112:0/1918503818 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fdf640ada90 0x7fdf640ade90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:34.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:34.017 7fdf6b387700 1 -- 172.21.2.112:0/1918503818 >> 172.21.2.112:0/1918503818 conn(0x7fdf640f0550 msgr2=0x7fdf64191730 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:34.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:34.017 7fdf6b387700 1 -- 172.21.2.112:0/1918503818 shutdown_connections 2021-02-10T11:45:34.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:34.017 7fdf6b387700 1 -- 172.21.2.112:0/1918503818 wait complete. 2021-02-10T11:45:34.028 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:45:34.469 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:34.450 7f2d17148700 1 Processor -- start 2021-02-10T11:45:34.469 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:34.450 7f2d17148700 1 -- start start 2021-02-10T11:45:34.470 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:34.450 7f2d17148700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f2d100d1880 0x7f2d101b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:34.470 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:34.450 7f2d17148700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f2d101b0af0 0x7f2d10108950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:34.470 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:34.450 7f2d17148700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d100fe1d0 con 0x7f2d100d1c80 2021-02-10T11:45:34.470 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:34.450 7f2d17148700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d10109a70 con 0x7f2d100d1880 2021-02-10T11:45:34.470 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:34.450 7f2d17148700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d10109760 con 0x7f2d101b0af0 2021-02-10T11:45:34.470 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:34.450 7f2d15765700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f2d100d1c80 0x7f2d10108f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:58308/0 (socket says 172.21.2.112:58308) 2021-02-10T11:45:34.470 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:34.450 7f2d15765700 1 -- 172.21.2.112:0/1126646101 learned_addr learned my addr 172.21.2.112:0/1126646101 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:45:34.470 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:34.450 7f2d14f64700 1 --2- 172.21.2.112:0/1126646101 >> v2:172.21.2.112:3300/0 conn(0x7f2d100d1880 0x7f2d101b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:34.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:34.451 7f2d0f7fe700 1 -- 172.21.2.112:0/1126646101 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2297872428 0 0) 0x7f2d08001500 con 0x7f2d100d1c80 2021-02-10T11:45:34.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:34.451 7f2d0f7fe700 1 -- 172.21.2.112:0/1126646101 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2cf4002fe0 con 0x7f2d100d1c80 2021-02-10T11:45:34.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:34.451 7f2d14f64700 1 -- 172.21.2.112:0/1126646101 >> v2:172.21.2.120:3300/0 conn(0x7f2d101b0af0 msgr2=0x7f2d10108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:34.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:34.451 7f2d14f64700 1 --2- 172.21.2.112:0/1126646101 >> v2:172.21.2.120:3300/0 conn(0x7f2d101b0af0 0x7f2d10108950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:34.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:34.451 7f2d14f64700 1 -- 172.21.2.112:0/1126646101 >> v1:172.21.2.112:6789/0 conn(0x7f2d100d1c80 legacy=0x7f2d10108f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:34.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:34.451 7f2d14f64700 1 -- 172.21.2.112:0/1126646101 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d10088070 con 0x7f2d100d1880 2021-02-10T11:45:34.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:34.451 7f2d14f64700 1 --2- 172.21.2.112:0/1126646101 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2d100d1880 0x7f2d101b04d0 secure :-1 s=READY pgs=2761 cs=0 l=1 rev1=1 rx=0x7f2d00006650 tx=0x7f2d00009a80).ready entity=mon.0 client_cookie=3655fb5cb47bfaeb server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:34.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:34.452 7f2d0f7fe700 1 -- 172.21.2.112:0/1126646101 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2d0000c0c0 con 0x7f2d100d1880 2021-02-10T11:45:34.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:34.452 7f2d0f7fe700 1 -- 172.21.2.112:0/1126646101 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d0000c440 con 0x7f2d100d1880 2021-02-10T11:45:34.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:34.452 7f2d0f7fe700 1 -- 172.21.2.112:0/1126646101 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2d0000c960 con 0x7f2d100d1880 2021-02-10T11:45:34.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:34.452 7f2d17148700 1 -- 172.21.2.112:0/1126646101 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2d100d1880 msgr2=0x7f2d101b04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:34.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:34.452 7f2d17148700 1 --2- 172.21.2.112:0/1126646101 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2d100d1880 0x7f2d101b04d0 secure :-1 s=READY pgs=2761 cs=0 l=1 rev1=1 rx=0x7f2d00006650 tx=0x7f2d00009a80).stop 2021-02-10T11:45:34.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:34.452 7f2d17148700 1 -- 172.21.2.112:0/1126646101 shutdown_connections 2021-02-10T11:45:34.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:34.452 7f2d17148700 1 --2- 172.21.2.112:0/1126646101 >> v2:172.21.2.120:3300/0 conn(0x7f2d101b0af0 0x7f2d10108950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:34.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:34.452 7f2d17148700 1 --2- 172.21.2.112:0/1126646101 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2d100d1880 0x7f2d101b04d0 unknown :-1 s=CLOSED pgs=2761 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:34.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:34.452 7f2d17148700 1 -- 172.21.2.112:0/1126646101 >> 172.21.2.112:0/1126646101 conn(0x7f2d100ce430 msgr2=0x7f2d100d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:34.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:34.452 7f2d17148700 1 -- 172.21.2.112:0/1126646101 shutdown_connections 2021-02-10T11:45:34.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:34.452 7f2d17148700 1 -- 172.21.2.112:0/1126646101 wait complete. 2021-02-10T11:45:34.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:34.467 7f2d17148700 1 Processor -- start 2021-02-10T11:45:34.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:34.467 7f2d17148700 1 -- start start 2021-02-10T11:45:34.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:34.468 7f2d17148700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2d100abd30 0x7f2d10082bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:34.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:34.468 7f2d17148700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2d100831d0 0x7f2d1008a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:34.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:34.468 7f2d17148700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2d100ac130 0x7f2d1008aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:34.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:34.468 7f2d17148700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d10083700 con 0x7f2d100831d0 2021-02-10T11:45:34.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:34.468 7f2d17148700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d1008b0c0 con 0x7f2d100ac130 2021-02-10T11:45:34.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:34.468 7f2d17148700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d1008b2a0 con 0x7f2d100abd30 2021-02-10T11:45:34.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:34.468 7f2d0ffff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2d100831d0 0x7f2d1008a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:34.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:34.468 7f2d0ffff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2d100831d0 0x7f2d1008a480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:58850/0 (socket says 172.21.2.112:58850) 2021-02-10T11:45:34.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:34.468 7f2d0ffff700 1 -- 172.21.2.112:0/1266649699 learned_addr learned my addr 172.21.2.112:0/1266649699 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:45:34.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:34.468 7f2d15765700 1 --2- 172.21.2.112:0/1266649699 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2d100ac130 0x7f2d1008aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:34.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:34.468 7f2d14f64700 1 --2- 172.21.2.112:0/1266649699 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2d100abd30 0x7f2d10082bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:34.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:34.468 7f2d0ffff700 1 -- 172.21.2.112:0/1266649699 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2d100abd30 msgr2=0x7f2d10082bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:34.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:34.468 7f2d0ffff700 1 --2- 172.21.2.112:0/1266649699 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2d100abd30 0x7f2d10082bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:34.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:34.468 7f2d0ffff700 1 -- 172.21.2.112:0/1266649699 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2d100ac130 msgr2=0x7f2d1008aaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:34.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:34.468 7f2d0ffff700 1 --2- 172.21.2.112:0/1266649699 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2d100ac130 0x7f2d1008aaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:34.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:34.468 7f2d0ffff700 1 -- 172.21.2.112:0/1266649699 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d1008b5f0 con 0x7f2d100831d0 2021-02-10T11:45:34.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:34.468 7f2d14f64700 1 --2- 172.21.2.112:0/1266649699 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2d100abd30 0x7f2d10082bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:45:34.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:34.468 7f2d15765700 1 --2- 172.21.2.112:0/1266649699 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2d100ac130 0x7f2d1008aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:45:34.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:34.468 7f2d0ffff700 1 --2- 172.21.2.112:0/1266649699 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2d100831d0 0x7f2d1008a480 secure :-1 s=READY pgs=2762 cs=0 l=1 rev1=1 rx=0x7f2cfc006f00 tx=0x7f2cfc00b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:34.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:34.469 7f2d0d7fa700 1 -- 172.21.2.112:0/1266649699 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2cfc006350 con 0x7f2d100831d0 2021-02-10T11:45:34.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:34.469 7f2d0d7fa700 1 -- 172.21.2.112:0/1266649699 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cfc00bd30 con 0x7f2d100831d0 2021-02-10T11:45:34.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:34.469 7f2d17148700 1 -- 172.21.2.112:0/1266649699 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d1008b5f0 con 0x7f2d100831d0 2021-02-10T11:45:34.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:34.469 7f2d0d7fa700 1 -- 172.21.2.112:0/1266649699 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2cfc002020 con 0x7f2d100831d0 2021-02-10T11:45:34.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:34.469 7f2d17148700 1 -- 172.21.2.112:0/1266649699 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d1008bb50 con 0x7f2d100831d0 2021-02-10T11:45:34.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:34.469 7f2d0d7fa700 1 -- 172.21.2.112:0/1266649699 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2cfc005310 con 0x7f2d100831d0 2021-02-10T11:45:34.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:34.469 7f2d0d7fa700 1 --2- 172.21.2.112:0/1266649699 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2cec018730 0x7f2cec01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:34.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:34.469 7f2d0d7fa700 1 -- 172.21.2.112:0/1266649699 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f2cfc032610 con 0x7f2d100831d0 2021-02-10T11:45:34.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:34.470 7f2d14f64700 1 --2- 172.21.2.112:0/1266649699 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2cec018730 0x7f2cec01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:34.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:34.470 7f2d14f64700 1 --2- 172.21.2.112:0/1266649699 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2cec018730 0x7f2cec01ac10 crc :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:34.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:34.500 7f2d17148700 1 -- 172.21.2.112:0/1266649699 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d10082980 con 0x7f2d100831d0 2021-02-10T11:45:34.505 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:34.503 7f2d0d7fa700 1 -- 172.21.2.112:0/1266649699 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2cfc006b90 con 0x7f2d100831d0 2021-02-10T11:45:34.637 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:34.634 7f2d17148700 1 -- 172.21.2.112:0/1266649699 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f2d10082980 con 0x7f2d100831d0 2021-02-10T11:45:34.640 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:34.637 7f2d0d7fa700 1 -- 172.21.2.112:0/1266649699 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v91) v1 ==== 184+0+0 (secure 0 0 0) 0x7f2cfc01f4d0 con 0x7f2d100831d0 2021-02-10T11:45:34.644 INFO:teuthology.orchestra.run.gibba012.stderr:updated caps for client.0 2021-02-10T11:45:34.646 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:34.643 7f2d17148700 1 -- 172.21.2.112:0/1266649699 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2cec018730 msgr2=0x7f2cec01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:34.646 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:34.643 7f2d17148700 1 --2- 172.21.2.112:0/1266649699 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2cec018730 0x7f2cec01ac10 crc :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:34.646 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:34.643 7f2d17148700 1 -- 172.21.2.112:0/1266649699 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2d100831d0 msgr2=0x7f2d1008a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:34.646 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:34.643 7f2d17148700 1 --2- 172.21.2.112:0/1266649699 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2d100831d0 0x7f2d1008a480 secure :-1 s=READY pgs=2762 cs=0 l=1 rev1=1 rx=0x7f2cfc006f00 tx=0x7f2cfc00b940).stop 2021-02-10T11:45:34.646 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:34.643 7f2d17148700 1 -- 172.21.2.112:0/1266649699 shutdown_connections 2021-02-10T11:45:34.646 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:34.643 7f2d17148700 1 --2- 172.21.2.112:0/1266649699 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2cec018730 0x7f2cec01ac10 unknown :-1 s=CLOSED pgs=1260 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:34.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:34.643 7f2d17148700 1 --2- 172.21.2.112:0/1266649699 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2d100ac130 0x7f2d1008aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:34.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:34.643 7f2d17148700 1 --2- 172.21.2.112:0/1266649699 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2d100831d0 0x7f2d1008a480 unknown :-1 s=CLOSED pgs=2762 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:34.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:34.643 7f2d17148700 1 --2- 172.21.2.112:0/1266649699 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2d100abd30 0x7f2d10082bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:34.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:34.643 7f2d17148700 1 -- 172.21.2.112:0/1266649699 >> 172.21.2.112:0/1266649699 conn(0x7f2d100afc60 msgr2=0x7f2d100b50d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:34.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:34.644 7f2d17148700 1 -- 172.21.2.112:0/1266649699 shutdown_connections 2021-02-10T11:45:34.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:34.644 7f2d17148700 1 -- 172.21.2.112:0/1266649699 wait complete. 2021-02-10T11:45:34.654 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:45:35.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.075 7fddf9c15700 1 Processor -- start 2021-02-10T11:45:35.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.075 7fddf9c15700 1 -- start start 2021-02-10T11:45:35.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.075 7fddf9c15700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fddf40d1880 0x7fddf41b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:35.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.075 7fddf9c15700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fddf41b0af0 0x7fddf4108950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:35.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.075 7fddf9c15700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddf4109a70 con 0x7fddf40d1c80 2021-02-10T11:45:35.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.075 7fddf9c15700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fddf40e5f00 con 0x7fddf41b0af0 2021-02-10T11:45:35.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.075 7fddf9c15700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddf4109760 con 0x7fddf40d1880 2021-02-10T11:45:35.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.076 7fddf2ffd700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fddf41b0af0 0x7fddf4108950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:35.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.076 7fddf2ffd700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fddf41b0af0 0x7fddf4108950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:58856/0 (socket says 172.21.2.112:58856) 2021-02-10T11:45:35.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.076 7fddf2ffd700 1 -- 172.21.2.112:0/1339765418 learned_addr learned my addr 172.21.2.112:0/1339765418 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:45:35.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.076 7fddf2ffd700 1 -- 172.21.2.112:0/1339765418 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fddf40d1880 msgr2=0x7fddf41b04d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:45:35.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.076 7fddf2ffd700 1 --2- 172.21.2.112:0/1339765418 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fddf40d1880 0x7fddf41b04d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:35.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.076 7fddf2ffd700 1 -- 172.21.2.112:0/1339765418 >> v1:172.21.2.112:6789/0 conn(0x7fddf40d1c80 legacy=0x7fddf4108f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:35.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.076 7fddf2ffd700 1 -- 172.21.2.112:0/1339765418 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddf4088070 con 0x7fddf41b0af0 2021-02-10T11:45:35.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.076 7fddf2ffd700 1 --2- 172.21.2.112:0/1339765418 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fddf41b0af0 0x7fddf4108950 secure :-1 s=READY pgs=2764 cs=0 l=1 rev1=1 rx=0x7fdde4006700 tx=0x7fdde4009b00).ready entity=mon.0 client_cookie=9be7dbba36785ee0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:35.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.076 7fddf27fc700 1 -- 172.21.2.112:0/1339765418 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdde400c0c0 con 0x7fddf41b0af0 2021-02-10T11:45:35.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.076 7fddf27fc700 1 -- 172.21.2.112:0/1339765418 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdde400c340 con 0x7fddf41b0af0 2021-02-10T11:45:35.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.076 7fddf27fc700 1 -- 172.21.2.112:0/1339765418 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdde400c820 con 0x7fddf41b0af0 2021-02-10T11:45:35.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.076 7fddf9c15700 1 -- 172.21.2.112:0/1339765418 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fddf41b0af0 msgr2=0x7fddf4108950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:35.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.076 7fddf9c15700 1 --2- 172.21.2.112:0/1339765418 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fddf41b0af0 0x7fddf4108950 secure :-1 s=READY pgs=2764 cs=0 l=1 rev1=1 rx=0x7fdde4006700 tx=0x7fdde4009b00).stop 2021-02-10T11:45:35.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.078 7fddf9c15700 1 -- 172.21.2.112:0/1339765418 shutdown_connections 2021-02-10T11:45:35.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.078 7fddf9c15700 1 --2- 172.21.2.112:0/1339765418 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fddf41b0af0 0x7fddf4108950 unknown :-1 s=CLOSED pgs=2764 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:35.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.078 7fddf9c15700 1 --2- 172.21.2.112:0/1339765418 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fddf40d1880 0x7fddf41b04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:35.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.078 7fddf9c15700 1 -- 172.21.2.112:0/1339765418 >> 172.21.2.112:0/1339765418 conn(0x7fddf40ce430 msgr2=0x7fddf40d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:35.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.078 7fddf9c15700 1 -- 172.21.2.112:0/1339765418 shutdown_connections 2021-02-10T11:45:35.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.078 7fddf9c15700 1 -- 172.21.2.112:0/1339765418 wait complete. 2021-02-10T11:45:35.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.092 7fddf9c15700 1 Processor -- start 2021-02-10T11:45:35.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.092 7fddf9c15700 1 -- start start 2021-02-10T11:45:35.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.092 7fddf9c15700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fddf41a68f0 0x7fddf419fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:35.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.092 7fddf9c15700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fddf41a5e00 0x7fddf41a02c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:35.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.092 7fddf9c15700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fddf41a08e0 0x7fddf41aae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:35.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.092 7fddf9c15700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddf41a6370 con 0x7fddf41a68f0 2021-02-10T11:45:35.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.092 7fddf9c15700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddf41a3960 con 0x7fddf41a08e0 2021-02-10T11:45:35.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.092 7fddf9c15700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddf41ab440 con 0x7fddf41a5e00 2021-02-10T11:45:35.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.092 7fddf37fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fddf41a68f0 0x7fddf419fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:35.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.092 7fddf37fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fddf41a68f0 0x7fddf419fca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:58864/0 (socket says 172.21.2.112:58864) 2021-02-10T11:45:35.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.092 7fddf37fe700 1 -- 172.21.2.112:0/1293313549 learned_addr learned my addr 172.21.2.112:0/1293313549 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:45:35.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.093 7fddf37fe700 1 -- 172.21.2.112:0/1293313549 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fddf41a5e00 msgr2=0x7fddf41a02c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:35.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.093 7fddf37fe700 1 --2- 172.21.2.112:0/1293313549 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fddf41a5e00 0x7fddf41a02c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:35.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.093 7fddf37fe700 1 -- 172.21.2.112:0/1293313549 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fddf41a08e0 msgr2=0x7fddf41aae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:35.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.093 7fddf37fe700 1 --2- 172.21.2.112:0/1293313549 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fddf41a08e0 0x7fddf41aae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:35.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.093 7fddf37fe700 1 -- 172.21.2.112:0/1293313549 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddf41ab660 con 0x7fddf41a68f0 2021-02-10T11:45:35.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.093 7fddf37fe700 1 --2- 172.21.2.112:0/1293313549 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fddf41a68f0 0x7fddf419fca0 secure :-1 s=READY pgs=2765 cs=0 l=1 rev1=1 rx=0x7fdddc00b5c0 tx=0x7fdddc00b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:35.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.093 7fddebfff700 1 -- 172.21.2.112:0/1293313549 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdddc000c40 con 0x7fddf41a68f0 2021-02-10T11:45:35.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.093 7fddebfff700 1 -- 172.21.2.112:0/1293313549 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdddc00bd50 con 0x7fddf41a68f0 2021-02-10T11:45:35.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.093 7fddebfff700 1 -- 172.21.2.112:0/1293313549 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdddc0054d0 con 0x7fddf41a68f0 2021-02-10T11:45:35.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.093 7fddf9c15700 1 -- 172.21.2.112:0/1293313549 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddf41ab660 con 0x7fddf41a68f0 2021-02-10T11:45:35.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.093 7fddf9c15700 1 -- 172.21.2.112:0/1293313549 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddf41abbf0 con 0x7fddf41a68f0 2021-02-10T11:45:35.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.094 7fddebfff700 1 -- 172.21.2.112:0/1293313549 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fdddc006030 con 0x7fddf41a68f0 2021-02-10T11:45:35.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.094 7fddebfff700 1 --2- 172.21.2.112:0/1293313549 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fddd80186c0 0x7fddd801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:35.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.094 7fddebfff700 1 -- 172.21.2.112:0/1293313549 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fdddc0325b0 con 0x7fddf41a68f0 2021-02-10T11:45:35.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.094 7fddf2ffd700 1 --2- 172.21.2.112:0/1293313549 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fddd80186c0 0x7fddd801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:35.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.095 7fddf2ffd700 1 --2- 172.21.2.112:0/1293313549 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fddd80186c0 0x7fddd801abc0 crc :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:35.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.125 7fddf9c15700 1 -- 172.21.2.112:0/1293313549 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddf40829a0 con 0x7fddf41a68f0 2021-02-10T11:45:35.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.128 7fddebfff700 1 -- 172.21.2.112:0/1293313549 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fdddc0009b0 con 0x7fddf41a68f0 2021-02-10T11:45:35.262 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.259 7fddf9c15700 1 -- 172.21.2.112:0/1293313549 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fddf40829a0 con 0x7fddf41a68f0 2021-02-10T11:45:35.265 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.263 7fddebfff700 1 -- 172.21.2.112:0/1293313549 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v92) v1 ==== 184+0+0 (secure 0 0 0) 0x7fdddc006030 con 0x7fddf41a68f0 2021-02-10T11:45:35.269 INFO:teuthology.orchestra.run.gibba012.stderr:updated caps for client.1 2021-02-10T11:45:35.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.268 7fddf9c15700 1 -- 172.21.2.112:0/1293313549 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fddd80186c0 msgr2=0x7fddd801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:35.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.268 7fddf9c15700 1 --2- 172.21.2.112:0/1293313549 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fddd80186c0 0x7fddd801abc0 crc :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:35.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.268 7fddf9c15700 1 -- 172.21.2.112:0/1293313549 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fddf41a68f0 msgr2=0x7fddf419fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:35.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.268 7fddf9c15700 1 --2- 172.21.2.112:0/1293313549 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fddf41a68f0 0x7fddf419fca0 secure :-1 s=READY pgs=2765 cs=0 l=1 rev1=1 rx=0x7fdddc00b5c0 tx=0x7fdddc00b960).stop 2021-02-10T11:45:35.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.268 7fddf9c15700 1 -- 172.21.2.112:0/1293313549 shutdown_connections 2021-02-10T11:45:35.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.268 7fddf9c15700 1 --2- 172.21.2.112:0/1293313549 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fddd80186c0 0x7fddd801abc0 unknown :-1 s=CLOSED pgs=1261 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:35.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.268 7fddf9c15700 1 --2- 172.21.2.112:0/1293313549 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fddf41a08e0 0x7fddf41aae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:35.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.268 7fddf9c15700 1 --2- 172.21.2.112:0/1293313549 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fddf41a5e00 0x7fddf41a02c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:35.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.268 7fddf9c15700 1 --2- 172.21.2.112:0/1293313549 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fddf41a68f0 0x7fddf419fca0 unknown :-1 s=CLOSED pgs=2765 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:35.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.268 7fddf9c15700 1 -- 172.21.2.112:0/1293313549 >> 172.21.2.112:0/1293313549 conn(0x7fddf419f880 msgr2=0x7fddf41a71a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:35.272 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.269 7fddf9c15700 1 -- 172.21.2.112:0/1293313549 shutdown_connections 2021-02-10T11:45:35.272 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.269 7fddf9c15700 1 -- 172.21.2.112:0/1293313549 wait complete. 2021-02-10T11:45:35.280 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.2 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:45:35.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.703 7f36b6cfe700 1 Processor -- start 2021-02-10T11:45:35.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.703 7f36b6cfe700 1 -- start start 2021-02-10T11:45:35.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.704 7f36b6cfe700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f36b00d1880 0x7f36b01b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:35.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.704 7f36b6cfe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f36b00cff80 0x7f36b0109140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:35.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.704 7f36b6cfe700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36b0109a70 con 0x7f36b01b0af0 2021-02-10T11:45:35.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.704 7f36b6cfe700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36b0109760 con 0x7f36b00cff80 2021-02-10T11:45:35.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.704 7f36b6cfe700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36b0088070 con 0x7f36b00d1880 2021-02-10T11:45:35.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.704 7f36b531b700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f36b00cff80 0x7f36b0109140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:35.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.704 7f36b531b700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f36b00cff80 0x7f36b0109140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:58872/0 (socket says 172.21.2.112:58872) 2021-02-10T11:45:35.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.704 7f36b531b700 1 -- 172.21.2.112:0/2859824022 learned_addr learned my addr 172.21.2.112:0/2859824022 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:45:35.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.704 7f36b531b700 1 -- 172.21.2.112:0/2859824022 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f36b00d1880 msgr2=0x7f36b01b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:35.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.704 7f36b531b700 1 --2- 172.21.2.112:0/2859824022 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f36b00d1880 0x7f36b01b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:35.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.704 7f36b531b700 1 -- 172.21.2.112:0/2859824022 >> v1:172.21.2.120:6789/0 conn(0x7f36b01b0af0 legacy=0x7f36b0108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:35.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.704 7f36b531b700 1 -- 172.21.2.112:0/2859824022 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36b0088260 con 0x7f36b00cff80 2021-02-10T11:45:35.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.704 7f36b531b700 1 --2- 172.21.2.112:0/2859824022 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f36b00cff80 0x7f36b0109140 secure :-1 s=READY pgs=2766 cs=0 l=1 rev1=1 rx=0x7f36a8006b80 tx=0x7f36a8009b10).ready entity=mon.0 client_cookie=703ab23bfa2df4b5 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:35.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.705 7f36af7fe700 1 -- 172.21.2.112:0/2859824022 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f36a80022e0 con 0x7f36b00cff80 2021-02-10T11:45:35.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.705 7f36af7fe700 1 -- 172.21.2.112:0/2859824022 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36a8002620 con 0x7f36b00cff80 2021-02-10T11:45:35.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.705 7f36af7fe700 1 -- 172.21.2.112:0/2859824022 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f36a8002b40 con 0x7f36b00cff80 2021-02-10T11:45:35.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.705 7f36b6cfe700 1 -- 172.21.2.112:0/2859824022 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f36b00cff80 msgr2=0x7f36b0109140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:35.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.705 7f36b6cfe700 1 --2- 172.21.2.112:0/2859824022 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f36b00cff80 0x7f36b0109140 secure :-1 s=READY pgs=2766 cs=0 l=1 rev1=1 rx=0x7f36a8006b80 tx=0x7f36a8009b10).stop 2021-02-10T11:45:35.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.705 7f36b6cfe700 1 -- 172.21.2.112:0/2859824022 shutdown_connections 2021-02-10T11:45:35.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.705 7f36b6cfe700 1 --2- 172.21.2.112:0/2859824022 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f36b00cff80 0x7f36b0109140 unknown :-1 s=CLOSED pgs=2766 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:35.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.705 7f36b6cfe700 1 --2- 172.21.2.112:0/2859824022 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f36b00d1880 0x7f36b01b04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:35.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.705 7f36b6cfe700 1 -- 172.21.2.112:0/2859824022 >> 172.21.2.112:0/2859824022 conn(0x7f36b00ce430 msgr2=0x7f36b00d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:35.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.705 7f36b6cfe700 1 -- 172.21.2.112:0/2859824022 shutdown_connections 2021-02-10T11:45:35.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.705 7f36b6cfe700 1 -- 172.21.2.112:0/2859824022 wait complete. 2021-02-10T11:45:35.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.720 7f36b6cfe700 1 Processor -- start 2021-02-10T11:45:35.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.720 7f36b6cfe700 1 -- start start 2021-02-10T11:45:35.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.720 7f36b6cfe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f36b0086810 0x7f36b019fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:35.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.720 7f36b6cfe700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f36b00b04d0 0x7f36b0097450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:35.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.720 7f36b6cfe700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f36b00b08d0 0x7f36b0097a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:35.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.720 7f36b6cfe700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36b01a02c0 con 0x7f36b0086810 2021-02-10T11:45:35.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.720 7f36b6cfe700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36b0098090 con 0x7f36b00b08d0 2021-02-10T11:45:35.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.720 7f36b6cfe700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36b00982b0 con 0x7f36b00b04d0 2021-02-10T11:45:35.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.720 7f36b4b1a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f36b0086810 0x7f36b019fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:35.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.720 7f36b4b1a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f36b0086810 0x7f36b019fca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:58876/0 (socket says 172.21.2.112:58876) 2021-02-10T11:45:35.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.720 7f36b4b1a700 1 -- 172.21.2.112:0/590919935 learned_addr learned my addr 172.21.2.112:0/590919935 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:45:35.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.721 7f36b4b1a700 1 -- 172.21.2.112:0/590919935 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f36b00b04d0 msgr2=0x7f36b0097450 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:45:35.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.721 7f36b4b1a700 1 --2- 172.21.2.112:0/590919935 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f36b00b04d0 0x7f36b0097450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:35.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.721 7f36b4b1a700 1 -- 172.21.2.112:0/590919935 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f36b00b08d0 msgr2=0x7f36b0097a70 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:45:35.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.721 7f36b4b1a700 1 --2- 172.21.2.112:0/590919935 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f36b00b08d0 0x7f36b0097a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:35.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.721 7f36b4b1a700 1 -- 172.21.2.112:0/590919935 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36b00984d0 con 0x7f36b0086810 2021-02-10T11:45:35.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.721 7f36b4b1a700 1 --2- 172.21.2.112:0/590919935 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f36b0086810 0x7f36b019fca0 secure :-1 s=READY pgs=2767 cs=0 l=1 rev1=1 rx=0x7f369800b5b0 tx=0x7f369800b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:35.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.721 7f36ad7fa700 1 -- 172.21.2.112:0/590919935 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f369800bd40 con 0x7f36b0086810 2021-02-10T11:45:35.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.721 7f36ad7fa700 1 -- 172.21.2.112:0/590919935 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3698002080 con 0x7f36b0086810 2021-02-10T11:45:35.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.721 7f36ad7fa700 1 -- 172.21.2.112:0/590919935 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3698002520 con 0x7f36b0086810 2021-02-10T11:45:35.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.721 7f36b6cfe700 1 -- 172.21.2.112:0/590919935 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36b00984d0 con 0x7f36b0086810 2021-02-10T11:45:35.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.721 7f36b6cfe700 1 -- 172.21.2.112:0/590919935 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36b00984d0 con 0x7f36b0086810 2021-02-10T11:45:35.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.722 7f36ad7fa700 1 -- 172.21.2.112:0/590919935 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3698000c90 con 0x7f36b0086810 2021-02-10T11:45:35.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.722 7f36ad7fa700 1 --2- 172.21.2.112:0/590919935 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f36940186c0 0x7f369401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:35.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.722 7f36ad7fa700 1 -- 172.21.2.112:0/590919935 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f36980327d0 con 0x7f36b0086810 2021-02-10T11:45:35.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.724 7f36affff700 1 --2- 172.21.2.112:0/590919935 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f36940186c0 0x7f369401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:35.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.725 7f36affff700 1 --2- 172.21.2.112:0/590919935 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f36940186c0 0x7f369401abc0 crc :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:35.757 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.754 7f36b6cfe700 1 -- 172.21.2.112:0/590919935 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36b00fe1d0 con 0x7f36b0086810 2021-02-10T11:45:35.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.756 7f36ad7fa700 1 -- 172.21.2.112:0/590919935 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f369801f510 con 0x7f36b0086810 2021-02-10T11:45:35.892 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.889 7f36b6cfe700 1 -- 172.21.2.112:0/590919935 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f36b00fe1d0 con 0x7f36b0086810 2021-02-10T11:45:35.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.892 7f36ad7fa700 1 -- 172.21.2.112:0/590919935 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.2 v93) v1 ==== 184+0+0 (secure 0 0 0) 0x7f369801f720 con 0x7f36b0086810 2021-02-10T11:45:35.899 INFO:teuthology.orchestra.run.gibba012.stderr:updated caps for client.2 2021-02-10T11:45:35.900 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.897 7f36b6cfe700 1 -- 172.21.2.112:0/590919935 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f36940186c0 msgr2=0x7f369401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:35.900 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.897 7f36b6cfe700 1 --2- 172.21.2.112:0/590919935 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f36940186c0 0x7f369401abc0 crc :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:35.901 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.897 7f36b6cfe700 1 -- 172.21.2.112:0/590919935 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f36b0086810 msgr2=0x7f36b019fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:35.901 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.897 7f36b6cfe700 1 --2- 172.21.2.112:0/590919935 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f36b0086810 0x7f36b019fca0 secure :-1 s=READY pgs=2767 cs=0 l=1 rev1=1 rx=0x7f369800b5b0 tx=0x7f369800b950).stop 2021-02-10T11:45:35.901 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.899 7f36b6cfe700 1 -- 172.21.2.112:0/590919935 shutdown_connections 2021-02-10T11:45:35.901 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.899 7f36b6cfe700 1 --2- 172.21.2.112:0/590919935 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f36940186c0 0x7f369401abc0 unknown :-1 s=CLOSED pgs=1262 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:35.901 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.899 7f36b6cfe700 1 --2- 172.21.2.112:0/590919935 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f36b00b08d0 0x7f36b0097a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:35.901 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.899 7f36b6cfe700 1 --2- 172.21.2.112:0/590919935 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f36b00b04d0 0x7f36b0097450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:35.901 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.899 7f36b6cfe700 1 --2- 172.21.2.112:0/590919935 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f36b0086810 0x7f36b019fca0 unknown :-1 s=CLOSED pgs=2767 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:35.901 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.899 7f36b6cfe700 1 -- 172.21.2.112:0/590919935 >> 172.21.2.112:0/590919935 conn(0x7f36b019f880 msgr2=0x7f36b008f890 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:35.901 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.899 7f36b6cfe700 1 -- 172.21.2.112:0/590919935 shutdown_connections 2021-02-10T11:45:35.901 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:35.899 7f36b6cfe700 1 -- 172.21.2.112:0/590919935 wait complete. 2021-02-10T11:45:35.909 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.3 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:45:36.349 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.330 7fe9d0e88700 1 Processor -- start 2021-02-10T11:45:36.349 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.330 7fe9d0e88700 1 -- start start 2021-02-10T11:45:36.349 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.331 7fe9d0e88700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fe9cc0d1880 0x7fe9cc1b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:36.349 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.331 7fe9d0e88700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe9cc0cff80 0x7fe9cc109140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:36.349 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.331 7fe9d0e88700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9cc109a70 con 0x7fe9cc1b0af0 2021-02-10T11:45:36.349 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.331 7fe9d0e88700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9cc109760 con 0x7fe9cc0d1880 2021-02-10T11:45:36.349 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.331 7fe9d0e88700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9cc088070 con 0x7fe9cc0cff80 2021-02-10T11:45:36.349 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.331 7fe9cab13700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fe9cc0d1880 0x7fe9cc1b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:36.349 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.331 7fe9cb314700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe9cc0cff80 0x7fe9cc109140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:36.350 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.331 7fe9ca312700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7fe9cc1b0af0 0x7fe9cc108950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:51904/0 (socket says 172.21.2.112:51904) 2021-02-10T11:45:36.350 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.331 7fe9cab13700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fe9cc0d1880 0x7fe9cc1b04d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3300/0 says I am v2:172.21.2.112:32930/0 (socket says 172.21.2.112:32930) 2021-02-10T11:45:36.350 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.331 7fe9ca312700 1 -- 172.21.2.112:0/1401431745 learned_addr learned my addr 172.21.2.112:0/1401431745 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:45:36.350 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.332 7fe9cab13700 1 -- 172.21.2.112:0/1401431745 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe9cc0cff80 msgr2=0x7fe9cc109140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:36.350 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.332 7fe9cab13700 1 --2- 172.21.2.112:0/1401431745 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe9cc0cff80 0x7fe9cc109140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:36.350 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.332 7fe9cab13700 1 -- 172.21.2.112:0/1401431745 >> v1:172.21.2.120:6789/0 conn(0x7fe9cc1b0af0 legacy=0x7fe9cc108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:36.350 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.332 7fe9cab13700 1 -- 172.21.2.112:0/1401431745 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9cc0d1c80 con 0x7fe9cc0d1880 2021-02-10T11:45:36.350 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.332 7fe9c9b11700 1 -- 172.21.2.112:0/1401431745 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 742976422 0 0) 0x7fe9b40009b0 con 0x7fe9cc1b0af0 2021-02-10T11:45:36.350 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.332 7fe9cb314700 1 --2- 172.21.2.112:0/1401431745 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe9cc0cff80 0x7fe9cc109140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:45:36.350 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.332 7fe9cab13700 1 --2- 172.21.2.112:0/1401431745 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe9cc0d1880 0x7fe9cc1b04d0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 rx=0x7fe9bc006650 tx=0x7fe9bc009a80).ready entity=mon.1 client_cookie=cd0823206e572bb2 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:36.350 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.332 7fe9c9b11700 1 -- 172.21.2.112:0/1401431745 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe9bc00c0c0 con 0x7fe9cc0d1880 2021-02-10T11:45:36.350 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.332 7fe9c9b11700 1 -- 172.21.2.112:0/1401431745 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9bc00c0c0 con 0x7fe9cc0d1880 2021-02-10T11:45:36.350 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.332 7fe9c9b11700 1 -- 172.21.2.112:0/1401431745 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe9bc00c340 con 0x7fe9cc0d1880 2021-02-10T11:45:36.351 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.332 7fe9d0e88700 1 -- 172.21.2.112:0/1401431745 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe9cc0d1880 msgr2=0x7fe9cc1b04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:36.351 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.332 7fe9d0e88700 1 --2- 172.21.2.112:0/1401431745 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe9cc0d1880 0x7fe9cc1b04d0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 rx=0x7fe9bc006650 tx=0x7fe9bc009a80).stop 2021-02-10T11:45:36.351 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.333 7fe9d0e88700 1 -- 172.21.2.112:0/1401431745 shutdown_connections 2021-02-10T11:45:36.351 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.333 7fe9d0e88700 1 --2- 172.21.2.112:0/1401431745 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe9cc0cff80 0x7fe9cc109140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:36.351 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.333 7fe9d0e88700 1 --2- 172.21.2.112:0/1401431745 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe9cc0d1880 0x7fe9cc1b04d0 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:36.351 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.333 7fe9d0e88700 1 -- 172.21.2.112:0/1401431745 >> 172.21.2.112:0/1401431745 conn(0x7fe9cc0ce430 msgr2=0x7fe9cc0d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:36.351 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.333 7fe9d0e88700 1 -- 172.21.2.112:0/1401431745 shutdown_connections 2021-02-10T11:45:36.351 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.333 7fe9d0e88700 1 -- 172.21.2.112:0/1401431745 wait complete. 2021-02-10T11:45:36.353 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.347 7fe9d0e88700 1 Processor -- start 2021-02-10T11:45:36.353 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.348 7fe9d0e88700 1 -- start start 2021-02-10T11:45:36.353 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.348 7fe9d0e88700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe9cc1a0580 0x7fe9cc086fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:36.353 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.348 7fe9d0e88700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe9cc0b66b0 0x7fe9cc0b6ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:36.353 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.348 7fe9d0e88700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe9cc082bb0 0x7fe9cc082fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:36.353 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.348 7fe9d0e88700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9cc0875d0 con 0x7fe9cc082bb0 2021-02-10T11:45:36.353 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.348 7fe9d0e88700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9cc0835d0 con 0x7fe9cc1a0580 2021-02-10T11:45:36.353 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.348 7fe9d0e88700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9cc0837f0 con 0x7fe9cc0b66b0 2021-02-10T11:45:36.353 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.348 7fe9cb314700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe9cc082bb0 0x7fe9cc082fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:36.353 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.348 7fe9cb314700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe9cc082bb0 0x7fe9cc082fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:58892/0 (socket says 172.21.2.112:58892) 2021-02-10T11:45:36.354 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.348 7fe9cb314700 1 -- 172.21.2.112:0/1988216375 learned_addr learned my addr 172.21.2.112:0/1988216375 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:45:36.354 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.348 7fe9cb314700 1 -- 172.21.2.112:0/1988216375 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe9cc0b66b0 msgr2=0x7fe9cc0b6ab0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:45:36.354 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.348 7fe9cb314700 1 --2- 172.21.2.112:0/1988216375 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe9cc0b66b0 0x7fe9cc0b6ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:36.354 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.348 7fe9cb314700 1 -- 172.21.2.112:0/1988216375 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe9cc1a0580 msgr2=0x7fe9cc086fb0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:45:36.354 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.348 7fe9cb314700 1 --2- 172.21.2.112:0/1988216375 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe9cc1a0580 0x7fe9cc086fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:36.354 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.348 7fe9cb314700 1 -- 172.21.2.112:0/1988216375 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9cc083a10 con 0x7fe9cc082bb0 2021-02-10T11:45:36.354 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.349 7fe9cb314700 1 --2- 172.21.2.112:0/1988216375 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe9cc082bb0 0x7fe9cc082fb0 secure :-1 s=READY pgs=2768 cs=0 l=1 rev1=1 rx=0x7fe9c400b970 tx=0x7fe9c400bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:36.354 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.349 7fe9c37fe700 1 -- 172.21.2.112:0/1988216375 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe9c4000f90 con 0x7fe9cc082bb0 2021-02-10T11:45:36.354 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.349 7fe9c37fe700 1 -- 172.21.2.112:0/1988216375 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9c4006790 con 0x7fe9cc082bb0 2021-02-10T11:45:36.354 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.349 7fe9d0e88700 1 -- 172.21.2.112:0/1988216375 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9cc083a10 con 0x7fe9cc082bb0 2021-02-10T11:45:36.354 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.349 7fe9d0e88700 1 -- 172.21.2.112:0/1988216375 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9cc083a10 con 0x7fe9cc082bb0 2021-02-10T11:45:36.354 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.349 7fe9c37fe700 1 -- 172.21.2.112:0/1988216375 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe9c40015f0 con 0x7fe9cc082bb0 2021-02-10T11:45:36.355 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.349 7fe9c37fe700 1 -- 172.21.2.112:0/1988216375 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe9c4000df0 con 0x7fe9cc082bb0 2021-02-10T11:45:36.355 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.350 7fe9c37fe700 1 --2- 172.21.2.112:0/1988216375 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe9b0018730 0x7fe9b001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:36.355 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.350 7fe9c37fe700 1 -- 172.21.2.112:0/1988216375 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fe9c4032b40 con 0x7fe9cc082bb0 2021-02-10T11:45:36.355 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.350 7fe9cab13700 1 --2- 172.21.2.112:0/1988216375 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe9b0018730 0x7fe9b001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:36.355 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.351 7fe9cab13700 1 --2- 172.21.2.112:0/1988216375 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe9b0018730 0x7fe9b001ac10 crc :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:36.382 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.380 7fe9d0e88700 1 -- 172.21.2.112:0/1988216375 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9cc0875d0 con 0x7fe9cc082bb0 2021-02-10T11:45:36.385 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.383 7fe9c37fe700 1 -- 172.21.2.112:0/1988216375 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe9c4000df0 con 0x7fe9cc082bb0 2021-02-10T11:45:36.515 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.514 7fe9d0e88700 1 -- 172.21.2.112:0/1988216375 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fe9cc0875d0 con 0x7fe9cc082bb0 2021-02-10T11:45:36.519 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.517 7fe9c37fe700 1 -- 172.21.2.112:0/1988216375 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.3 v94) v1 ==== 184+0+0 (secure 0 0 0) 0x7fe9c40066f0 con 0x7fe9cc082bb0 2021-02-10T11:45:36.522 INFO:teuthology.orchestra.run.gibba012.stderr:updated caps for client.3 2021-02-10T11:45:36.524 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.522 7fe9d0e88700 1 -- 172.21.2.112:0/1988216375 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe9b0018730 msgr2=0x7fe9b001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:36.524 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.522 7fe9d0e88700 1 --2- 172.21.2.112:0/1988216375 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe9b0018730 0x7fe9b001ac10 crc :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:36.524 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.522 7fe9d0e88700 1 -- 172.21.2.112:0/1988216375 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe9cc082bb0 msgr2=0x7fe9cc082fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:36.524 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.522 7fe9d0e88700 1 --2- 172.21.2.112:0/1988216375 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe9cc082bb0 0x7fe9cc082fb0 secure :-1 s=READY pgs=2768 cs=0 l=1 rev1=1 rx=0x7fe9c400b970 tx=0x7fe9c400bd10).stop 2021-02-10T11:45:36.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.522 7fe9d0e88700 1 -- 172.21.2.112:0/1988216375 shutdown_connections 2021-02-10T11:45:36.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.522 7fe9d0e88700 1 --2- 172.21.2.112:0/1988216375 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe9b0018730 0x7fe9b001ac10 unknown :-1 s=CLOSED pgs=1263 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:36.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.522 7fe9d0e88700 1 --2- 172.21.2.112:0/1988216375 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe9cc082bb0 0x7fe9cc082fb0 unknown :-1 s=CLOSED pgs=2768 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:36.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.522 7fe9d0e88700 1 --2- 172.21.2.112:0/1988216375 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe9cc0b66b0 0x7fe9cc0b6ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:36.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.522 7fe9d0e88700 1 --2- 172.21.2.112:0/1988216375 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe9cc1a0580 0x7fe9cc086fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:36.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.522 7fe9d0e88700 1 -- 172.21.2.112:0/1988216375 >> 172.21.2.112:0/1988216375 conn(0x7fe9cc0afc60 msgr2=0x7fe9cc0b50d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:36.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.523 7fe9d0e88700 1 -- 172.21.2.112:0/1988216375 shutdown_connections 2021-02-10T11:45:36.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.523 7fe9d0e88700 1 -- 172.21.2.112:0/1988216375 wait complete. 2021-02-10T11:45:36.532 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:45:36.962 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.943 7f00e057b700 1 Processor -- start 2021-02-10T11:45:36.962 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.943 7f00e057b700 1 -- start start 2021-02-10T11:45:36.962 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.944 7f00e057b700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f00d80d1870 0x7f00d81b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:36.962 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.944 7f00e057b700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f00d81b0a10 0x7f00d80cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:36.962 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.944 7f00e057b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f00d80d1c70 0x7f00d80cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:36.962 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.944 7f00e057b700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00d80e5ed0 con 0x7f00d80d1870 2021-02-10T11:45:36.962 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.944 7f00e057b700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00d80f0b10 con 0x7f00d81b0a10 2021-02-10T11:45:36.962 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.944 7f00e057b700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00d80cff70 con 0x7f00d80d1c70 2021-02-10T11:45:36.963 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.944 7f00de397700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f00d80d1870 0x7f00d81b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:36.963 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.944 7f00de397700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f00d80d1870 0x7f00d81b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:58896/0 (socket says 172.21.2.112:58896) 2021-02-10T11:45:36.963 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.944 7f00de397700 1 -- 172.21.2.112:0/3755174816 learned_addr learned my addr 172.21.2.112:0/3755174816 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:45:36.963 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.944 7f00de397700 1 -- 172.21.2.112:0/3755174816 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f00d80d1c70 msgr2=0x7f00d80cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:36.963 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.944 7f00ddb96700 1 --2- 172.21.2.112:0/3755174816 >> v2:172.21.2.120:3300/0 conn(0x7f00d81b0a10 0x7f00d80cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:36.963 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.944 7f00de397700 1 --2- 172.21.2.112:0/3755174816 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f00d80d1c70 0x7f00d80cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:36.963 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.944 7f00de397700 1 -- 172.21.2.112:0/3755174816 >> v2:172.21.2.120:3300/0 conn(0x7f00d81b0a10 msgr2=0x7f00d80cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:36.963 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.944 7f00de397700 1 --2- 172.21.2.112:0/3755174816 >> v2:172.21.2.120:3300/0 conn(0x7f00d81b0a10 0x7f00d80cf330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:36.963 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.944 7f00de397700 1 -- 172.21.2.112:0/3755174816 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00d80d0150 con 0x7f00d80d1870 2021-02-10T11:45:36.963 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.944 7f00ddb96700 1 --2- 172.21.2.112:0/3755174816 >> v2:172.21.2.120:3300/0 conn(0x7f00d81b0a10 0x7f00d80cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:45:36.963 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.945 7f00de397700 1 --2- 172.21.2.112:0/3755174816 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f00d80d1870 0x7f00d81b03f0 secure :-1 s=READY pgs=2769 cs=0 l=1 rev1=1 rx=0x7f00d4006700 tx=0x7f00d4009b00).ready entity=mon.0 client_cookie=4043f78b5b52f7b2 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:36.963 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.945 7f00dd395700 1 -- 172.21.2.112:0/3755174816 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f00d400c0c0 con 0x7f00d80d1870 2021-02-10T11:45:36.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.945 7f00dd395700 1 -- 172.21.2.112:0/3755174816 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00d400c340 con 0x7f00d80d1870 2021-02-10T11:45:36.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.945 7f00dd395700 1 -- 172.21.2.112:0/3755174816 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f00d400c820 con 0x7f00d80d1870 2021-02-10T11:45:36.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.945 7f00e057b700 1 -- 172.21.2.112:0/3755174816 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f00d80d1870 msgr2=0x7f00d81b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:36.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.945 7f00e057b700 1 --2- 172.21.2.112:0/3755174816 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f00d80d1870 0x7f00d81b03f0 secure :-1 s=READY pgs=2769 cs=0 l=1 rev1=1 rx=0x7f00d4006700 tx=0x7f00d4009b00).stop 2021-02-10T11:45:36.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.946 7f00e057b700 1 -- 172.21.2.112:0/3755174816 shutdown_connections 2021-02-10T11:45:36.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.946 7f00e057b700 1 --2- 172.21.2.112:0/3755174816 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f00d80d1c70 0x7f00d80cf950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:36.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.946 7f00e057b700 1 --2- 172.21.2.112:0/3755174816 >> v2:172.21.2.120:3300/0 conn(0x7f00d81b0a10 0x7f00d80cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:36.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.946 7f00e057b700 1 --2- 172.21.2.112:0/3755174816 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f00d80d1870 0x7f00d81b03f0 unknown :-1 s=CLOSED pgs=2769 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:36.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.946 7f00e057b700 1 -- 172.21.2.112:0/3755174816 >> 172.21.2.112:0/3755174816 conn(0x7f00d80ce450 msgr2=0x7f00d80d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:36.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.946 7f00e057b700 1 -- 172.21.2.112:0/3755174816 shutdown_connections 2021-02-10T11:45:36.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.946 7f00e057b700 1 -- 172.21.2.112:0/3755174816 wait complete. 2021-02-10T11:45:36.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.960 7f00e057b700 1 Processor -- start 2021-02-10T11:45:36.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.960 7f00e057b700 1 -- start start 2021-02-10T11:45:36.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.961 7f00e057b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f00d80970b0 0x7f00d80ebe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:36.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.961 7f00e057b700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f00d8098810 0x7f00d8098c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:36.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.961 7f00e057b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f00d8099230 0x7f00d80b18c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:36.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.961 7f00e057b700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00d80ec6d0 con 0x7f00d80970b0 2021-02-10T11:45:36.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.961 7f00e057b700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00d80974b0 con 0x7f00d8098810 2021-02-10T11:45:36.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.961 7f00e057b700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00d80b1ee0 con 0x7f00d8099230 2021-02-10T11:45:36.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.961 7f00de397700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f00d80970b0 0x7f00d80ebe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:36.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.961 7f00de397700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f00d80970b0 0x7f00d80ebe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:58900/0 (socket says 172.21.2.112:58900) 2021-02-10T11:45:36.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.961 7f00de397700 1 -- 172.21.2.112:0/3412355644 learned_addr learned my addr 172.21.2.112:0/3412355644 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:45:36.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.961 7f00de397700 1 -- 172.21.2.112:0/3412355644 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f00d8099230 msgr2=0x7f00d80b18c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:36.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.961 7f00de397700 1 --2- 172.21.2.112:0/3412355644 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f00d8099230 0x7f00d80b18c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:36.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.961 7f00de397700 1 -- 172.21.2.112:0/3412355644 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f00d8098810 msgr2=0x7f00d8098c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:36.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.961 7f00de397700 1 --2- 172.21.2.112:0/3412355644 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f00d8098810 0x7f00d8098c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:36.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.961 7f00de397700 1 -- 172.21.2.112:0/3412355644 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00d80b2100 con 0x7f00d80970b0 2021-02-10T11:45:36.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.962 7f00de397700 1 --2- 172.21.2.112:0/3412355644 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f00d80970b0 0x7f00d80ebe30 secure :-1 s=READY pgs=2770 cs=0 l=1 rev1=1 rx=0x7f00d4006f20 tx=0x7f00d400b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:36.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.962 7f00caffd700 1 -- 172.21.2.112:0/3412355644 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f00d4000c90 con 0x7f00d80970b0 2021-02-10T11:45:36.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.962 7f00caffd700 1 -- 172.21.2.112:0/3412355644 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00d4009100 con 0x7f00d80970b0 2021-02-10T11:45:36.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.962 7f00caffd700 1 -- 172.21.2.112:0/3412355644 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f00d4002220 con 0x7f00d80970b0 2021-02-10T11:45:36.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.962 7f00e057b700 1 -- 172.21.2.112:0/3412355644 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00d80b2100 con 0x7f00d80970b0 2021-02-10T11:45:36.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.962 7f00e057b700 1 -- 172.21.2.112:0/3412355644 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00d80b2690 con 0x7f00d80970b0 2021-02-10T11:45:36.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.963 7f00caffd700 1 -- 172.21.2.112:0/3412355644 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f00d4000c90 con 0x7f00d80970b0 2021-02-10T11:45:36.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.963 7f00caffd700 1 --2- 172.21.2.112:0/3412355644 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f00c00186c0 0x7f00c001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:36.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.963 7f00caffd700 1 -- 172.21.2.112:0/3412355644 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f00d4032510 con 0x7f00d80970b0 2021-02-10T11:45:36.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.963 7f00ddb96700 1 --2- 172.21.2.112:0/3412355644 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f00c00186c0 0x7f00c001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:36.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.964 7f00ddb96700 1 --2- 172.21.2.112:0/3412355644 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f00c00186c0 0x7f00c001abc0 crc :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:36.995 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.994 7f00e057b700 1 -- 172.21.2.112:0/3412355644 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00d80ebc00 con 0x7f00d80970b0 2021-02-10T11:45:36.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:36.996 7f00caffd700 1 -- 172.21.2.112:0/3412355644 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f00d4020fb0 con 0x7f00d80970b0 2021-02-10T11:45:37.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:37.123 7f00e057b700 1 -- 172.21.2.112:0/3412355644 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f00d80ebc00 con 0x7f00d80970b0 2021-02-10T11:45:37.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:37.124 7f00caffd700 1 -- 172.21.2.112:0/3412355644 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 139 v139) v1 ==== 95+0+2105 (secure 0 0 0) 0x7f00d4020fb0 con 0x7f00d80970b0 2021-02-10T11:45:37.126 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:45:37.126 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":139,"default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":12306,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/2635651262","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":2635651262},{"type":"v1","addr":"172.21.2.112:6833","nonce":2635651262}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":135}],"filesystems":[{"mdsmap":{"epoch":139,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:45:30.416714","modified":"2021-02-10 11:45:32.783510","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8920},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8920":{"gid":8920,"name":"b","rank":0,"incarnation":136,"state":"up:active","state_seq":3,"addr":"172.21.2.120:6835/555399099","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":555399099},{"type":"v1","addr":"172.21.2.120:6835","nonce":555399099}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[35],"metadata_pool":34,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":17}]} 2021-02-10T11:45:37.127 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 139 2021-02-10T11:45:37.127 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:37.126 7f00e057b700 1 -- 172.21.2.112:0/3412355644 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f00c00186c0 msgr2=0x7f00c001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:37.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:37.126 7f00e057b700 1 --2- 172.21.2.112:0/3412355644 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f00c00186c0 0x7f00c001abc0 crc :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:37.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:37.126 7f00e057b700 1 -- 172.21.2.112:0/3412355644 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f00d80970b0 msgr2=0x7f00d80ebe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:37.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:37.126 7f00e057b700 1 --2- 172.21.2.112:0/3412355644 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f00d80970b0 0x7f00d80ebe30 secure :-1 s=READY pgs=2770 cs=0 l=1 rev1=1 rx=0x7f00d4006f20 tx=0x7f00d400b990).stop 2021-02-10T11:45:37.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:37.126 7f00e057b700 1 -- 172.21.2.112:0/3412355644 shutdown_connections 2021-02-10T11:45:37.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:37.126 7f00e057b700 1 --2- 172.21.2.112:0/3412355644 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f00c00186c0 0x7f00c001abc0 unknown :-1 s=CLOSED pgs=1264 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:37.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:37.126 7f00e057b700 1 --2- 172.21.2.112:0/3412355644 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f00d8098810 0x7f00d8098c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:37.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:37.126 7f00e057b700 1 --2- 172.21.2.112:0/3412355644 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f00d8099230 0x7f00d80b18c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:37.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:37.126 7f00e057b700 1 --2- 172.21.2.112:0/3412355644 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f00d80970b0 0x7f00d80ebe30 unknown :-1 s=CLOSED pgs=2770 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:37.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:37.126 7f00e057b700 1 -- 172.21.2.112:0/3412355644 >> 172.21.2.112:0/3412355644 conn(0x7f00d819a730 msgr2=0x7f00d819e480 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:37.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:37.126 7f00e057b700 1 -- 172.21.2.112:0/3412355644 shutdown_connections 2021-02-10T11:45:37.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:37.126 7f00e057b700 1 -- 172.21.2.112:0/3412355644 wait complete. 2021-02-10T11:45:37.338 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 139, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:45:30.416714', 'modified': '2021-02-10 11:45:32.783510', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8920}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8920': {'gid': 8920, 'name': 'b', 'rank': 0, 'incarnation': 136, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.120:6835/555399099', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.120:6834', 'nonce': 555399099}, {'type': 'v1', 'addr': '172.21.2.120:6835', 'nonce': 555399099}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [35], 'metadata_pool': 34, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:45:37.338 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:45:37.339 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:45:37.737 INFO:teuthology.orchestra.run.gibba020.stdout:{ 2021-02-10T11:45:37.737 INFO:teuthology.orchestra.run.gibba020.stdout: "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", 2021-02-10T11:45:37.737 INFO:teuthology.orchestra.run.gibba020.stdout: "whoami": 0, 2021-02-10T11:45:37.737 INFO:teuthology.orchestra.run.gibba020.stdout: "id": 8920, 2021-02-10T11:45:37.737 INFO:teuthology.orchestra.run.gibba020.stdout: "want_state": "up:active", 2021-02-10T11:45:37.737 INFO:teuthology.orchestra.run.gibba020.stdout: "state": "up:active", 2021-02-10T11:45:37.737 INFO:teuthology.orchestra.run.gibba020.stdout: "rank_uptime": 7.3167225849999999, 2021-02-10T11:45:37.737 INFO:teuthology.orchestra.run.gibba020.stdout: "mdsmap_epoch": 139, 2021-02-10T11:45:37.738 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch": 161, 2021-02-10T11:45:37.738 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch_barrier": 161, 2021-02-10T11:45:37.738 INFO:teuthology.orchestra.run.gibba020.stdout: "uptime": 12.969042758000001 2021-02-10T11:45:37.738 INFO:teuthology.orchestra.run.gibba020.stdout:} 2021-02-10T11:45:37.741 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", "whoami": 0, "id": 8920, "want_state": "up:active", "state": "up:active", "rank_uptime": 7.3167225849999999, "mdsmap_epoch": 139, "osdmap_epoch": 161, "osdmap_epoch_barrier": 161, "uptime": 12.969042758000001 } 2021-02-10T11:45:37.741 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:45:38.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.152 7f531ea49700 1 Processor -- start 2021-02-10T11:45:38.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.152 7f531ea49700 1 -- start start 2021-02-10T11:45:38.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.153 7f531ea49700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f53180d0340 0x7f53180ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:38.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.153 7f531ea49700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f53180d1c70 0x7f53180efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:38.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.153 7f531ea49700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53180f0b10 con 0x7f53180d1870 2021-02-10T11:45:38.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.153 7f531ea49700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53180e5ed0 con 0x7f53180d1c70 2021-02-10T11:45:38.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.153 7f531ea49700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53181b0be0 con 0x7f53180d0340 2021-02-10T11:45:38.172 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.153 7f531c865700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f53180d1870 0x7f53181b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:58370/0 (socket says 172.21.2.112:58370) 2021-02-10T11:45:38.172 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.153 7f531c865700 1 -- 172.21.2.112:0/87258296 learned_addr learned my addr 172.21.2.112:0/87258296 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:45:38.172 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.153 7f531d066700 1 --2- 172.21.2.112:0/87258296 >> v2:172.21.2.112:3300/0 conn(0x7f53180d1c70 0x7f53180efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:38.172 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.153 7f5317fff700 1 --2- 172.21.2.112:0/87258296 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f53180d0340 0x7f53180ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:38.172 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.154 7f531d066700 1 -- 172.21.2.112:0/87258296 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f53180d0340 msgr2=0x7f53180ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:38.172 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.154 7f531d066700 1 --2- 172.21.2.112:0/87258296 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f53180d0340 0x7f53180ef640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:38.172 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.154 7f531d066700 1 -- 172.21.2.112:0/87258296 >> v1:172.21.2.112:6789/0 conn(0x7f53180d1870 legacy=0x7f53181b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:38.172 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.154 7f531d066700 1 -- 172.21.2.112:0/87258296 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53180f20a0 con 0x7f53180d1c70 2021-02-10T11:45:38.172 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.154 7f5317fff700 1 --2- 172.21.2.112:0/87258296 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f53180d0340 0x7f53180ef640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:45:38.172 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.154 7f531d066700 1 --2- 172.21.2.112:0/87258296 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f53180d1c70 0x7f53180efc60 secure :-1 s=READY pgs=2772 cs=0 l=1 rev1=1 rx=0x7f5310006ab0 tx=0x7f5310009b40).ready entity=mon.0 client_cookie=f21356f6df1b52d8 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:38.172 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.154 7f53177fe700 1 -- 172.21.2.112:0/87258296 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f53100022e0 con 0x7f53180d1c70 2021-02-10T11:45:38.172 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.154 7f53177fe700 1 -- 172.21.2.112:0/87258296 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5310002660 con 0x7f53180d1c70 2021-02-10T11:45:38.173 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.154 7f53177fe700 1 -- 172.21.2.112:0/87258296 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5310002ac0 con 0x7f53180d1c70 2021-02-10T11:45:38.173 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.154 7f531ea49700 1 -- 172.21.2.112:0/87258296 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f53180d1c70 msgr2=0x7f53180efc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:38.173 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.154 7f531ea49700 1 --2- 172.21.2.112:0/87258296 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f53180d1c70 0x7f53180efc60 secure :-1 s=READY pgs=2772 cs=0 l=1 rev1=1 rx=0x7f5310006ab0 tx=0x7f5310009b40).stop 2021-02-10T11:45:38.173 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.155 7f531ea49700 1 -- 172.21.2.112:0/87258296 shutdown_connections 2021-02-10T11:45:38.173 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.155 7f531ea49700 1 --2- 172.21.2.112:0/87258296 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f53180d1c70 0x7f53180efc60 unknown :-1 s=CLOSED pgs=2772 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:38.173 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.155 7f531ea49700 1 --2- 172.21.2.112:0/87258296 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f53180d0340 0x7f53180ef640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:38.173 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.155 7f531ea49700 1 -- 172.21.2.112:0/87258296 >> 172.21.2.112:0/87258296 conn(0x7f53180ce450 msgr2=0x7f53180d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:38.173 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.155 7f531ea49700 1 -- 172.21.2.112:0/87258296 shutdown_connections 2021-02-10T11:45:38.173 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.155 7f531ea49700 1 -- 172.21.2.112:0/87258296 wait complete. 2021-02-10T11:45:38.175 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.169 7f531ea49700 1 Processor -- start 2021-02-10T11:45:38.175 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.170 7f531ea49700 1 -- start start 2021-02-10T11:45:38.175 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.170 7f531ea49700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f53181b7cf0 0x7f531819cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:38.175 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.170 7f531ea49700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f53181b80f0 0x7f53181b84f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:38.175 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.170 7f531ea49700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f531819da20 0x7f531819de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:38.175 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.170 7f531ea49700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53180a77c0 con 0x7f53181b80f0 2021-02-10T11:45:38.175 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.170 7f531ea49700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f531819d4a0 con 0x7f531819da20 2021-02-10T11:45:38.176 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.170 7f531ea49700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53180954f0 con 0x7f53181b7cf0 2021-02-10T11:45:38.176 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.170 7f5317fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f53181b80f0 0x7f53181b84f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:38.176 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.170 7f5317fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f53181b80f0 0x7f53181b84f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:58916/0 (socket says 172.21.2.112:58916) 2021-02-10T11:45:38.176 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.170 7f5317fff700 1 -- 172.21.2.112:0/2759076402 learned_addr learned my addr 172.21.2.112:0/2759076402 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:45:38.176 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.171 7f5317fff700 1 -- 172.21.2.112:0/2759076402 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f53181b7cf0 msgr2=0x7f531819cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:38.176 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.171 7f5317fff700 1 --2- 172.21.2.112:0/2759076402 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f53181b7cf0 0x7f531819cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:38.176 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.171 7f5317fff700 1 -- 172.21.2.112:0/2759076402 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f531819da20 msgr2=0x7f531819de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:38.176 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.171 7f5317fff700 1 --2- 172.21.2.112:0/2759076402 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f531819da20 0x7f531819de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:38.176 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.171 7f5317fff700 1 -- 172.21.2.112:0/2759076402 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53181b8d60 con 0x7f53181b80f0 2021-02-10T11:45:38.176 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.171 7f5317fff700 1 --2- 172.21.2.112:0/2759076402 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f53181b80f0 0x7f53181b84f0 secure :-1 s=READY pgs=2773 cs=0 l=1 rev1=1 rx=0x7f53080099a0 tx=0x7f530800bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:38.176 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.171 7f53157fa700 1 -- 172.21.2.112:0/2759076402 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f530800c2f0 con 0x7f53181b80f0 2021-02-10T11:45:38.177 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.171 7f53157fa700 1 -- 172.21.2.112:0/2759076402 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f530800c670 con 0x7f53181b80f0 2021-02-10T11:45:38.177 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.171 7f53157fa700 1 -- 172.21.2.112:0/2759076402 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f530800cad0 con 0x7f53181b80f0 2021-02-10T11:45:38.177 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.171 7f531ea49700 1 -- 172.21.2.112:0/2759076402 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53181b8d60 con 0x7f53181b80f0 2021-02-10T11:45:38.177 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.171 7f531ea49700 1 -- 172.21.2.112:0/2759076402 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f531819e730 con 0x7f53181b80f0 2021-02-10T11:45:38.177 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.172 7f53157fa700 1 -- 172.21.2.112:0/2759076402 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f530800c2f0 con 0x7f53181b80f0 2021-02-10T11:45:38.177 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.172 7f53157fa700 1 --2- 172.21.2.112:0/2759076402 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f52fc0186c0 0x7f52fc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:38.177 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.172 7f53157fa700 1 -- 172.21.2.112:0/2759076402 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f5308032920 con 0x7f53181b80f0 2021-02-10T11:45:38.177 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.172 7f531c865700 1 --2- 172.21.2.112:0/2759076402 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f52fc0186c0 0x7f52fc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:38.177 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.173 7f531c865700 1 --2- 172.21.2.112:0/2759076402 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f52fc0186c0 0x7f52fc01abc0 crc :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:38.205 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.203 7f531ea49700 1 -- 172.21.2.112:0/2759076402 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53180ebc00 con 0x7f53181b80f0 2021-02-10T11:45:38.207 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.206 7f53157fa700 1 -- 172.21.2.112:0/2759076402 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5308000ce0 con 0x7f53181b80f0 2021-02-10T11:45:38.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.330 7f531ea49700 1 -- 172.21.2.112:0/2759076402 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f53180f0b10 con 0x7f53181b80f0 2021-02-10T11:45:38.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.331 7f53157fa700 1 -- 172.21.2.112:0/2759076402 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v161) v1 ==== 74+0+14007 (secure 0 0 0) 0x7f530800c2f0 con 0x7f53181b80f0 2021-02-10T11:45:38.335 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:45:38.335 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":161,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:45:31.418524","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":35,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":34,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:45:28.386296","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"160","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":35,"pool_name":"cephfs_data","create_time":"2021-02-10 11:45:29.158381","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"161","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:45:38.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.333 7f531ea49700 1 -- 172.21.2.112:0/2759076402 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f52fc0186c0 msgr2=0x7f52fc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:38.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.333 7f531ea49700 1 --2- 172.21.2.112:0/2759076402 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f52fc0186c0 0x7f52fc01abc0 crc :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:38.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.333 7f531ea49700 1 -- 172.21.2.112:0/2759076402 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f53181b80f0 msgr2=0x7f53181b84f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:38.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.333 7f531ea49700 1 --2- 172.21.2.112:0/2759076402 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f53181b80f0 0x7f53181b84f0 secure :-1 s=READY pgs=2773 cs=0 l=1 rev1=1 rx=0x7f53080099a0 tx=0x7f530800bcf0).stop 2021-02-10T11:45:38.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.333 7f531ea49700 1 -- 172.21.2.112:0/2759076402 shutdown_connections 2021-02-10T11:45:38.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.333 7f531ea49700 1 --2- 172.21.2.112:0/2759076402 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f52fc0186c0 0x7f52fc01abc0 unknown :-1 s=CLOSED pgs=1265 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:38.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.333 7f531ea49700 1 --2- 172.21.2.112:0/2759076402 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f531819da20 0x7f531819de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:38.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.333 7f531ea49700 1 --2- 172.21.2.112:0/2759076402 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f53181b80f0 0x7f53181b84f0 unknown :-1 s=CLOSED pgs=2773 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:38.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.333 7f531ea49700 1 --2- 172.21.2.112:0/2759076402 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f53181b7cf0 0x7f531819cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:38.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.333 7f531ea49700 1 -- 172.21.2.112:0/2759076402 >> 172.21.2.112:0/2759076402 conn(0x7f53180ed810 msgr2=0x7f5318094980 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:38.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.333 7f531ea49700 1 -- 172.21.2.112:0/2759076402 shutdown_connections 2021-02-10T11:45:38.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.333 7f531ea49700 1 -- 172.21.2.112:0/2759076402 wait complete. 2021-02-10T11:45:38.344 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:13:57.508758', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 34, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:45:28.386296', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '160', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 35, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:45:29.158381', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '161', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:45:38.345 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:45:38.778 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.760 7fbe35922700 1 Processor -- start 2021-02-10T11:45:38.778 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.760 7fbe35922700 1 -- start start 2021-02-10T11:45:38.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.760 7fbe35922700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fbe300d1c70 0x7fbe300cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:38.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.760 7fbe35922700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fbe300cf580 0x7fbe300cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:38.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.760 7fbe35922700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe300fe1c0 con 0x7fbe300d1870 2021-02-10T11:45:38.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.760 7fbe35922700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe301b0cb0 con 0x7fbe300d1c70 2021-02-10T11:45:38.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.760 7fbe35922700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe30109a60 con 0x7fbe300cf580 2021-02-10T11:45:38.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.760 7fbe2ed73700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fbe300d1c70 0x7fbe300cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:38.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.761 7fbe2ed73700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fbe300d1c70 0x7fbe300cef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:58922/0 (socket says 172.21.2.112:58922) 2021-02-10T11:45:38.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.761 7fbe2ed73700 1 -- 172.21.2.112:0/1230075491 learned_addr learned my addr 172.21.2.112:0/1230075491 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:45:38.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.761 7fbe2ed73700 1 -- 172.21.2.112:0/1230075491 >> v2:172.21.2.120:3300/0 conn(0x7fbe300cf580 msgr2=0x7fbe300cf980 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:45:38.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.761 7fbe2ed73700 1 --2- 172.21.2.112:0/1230075491 >> v2:172.21.2.120:3300/0 conn(0x7fbe300cf580 0x7fbe300cf980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:38.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.761 7fbe2ed73700 1 -- 172.21.2.112:0/1230075491 >> v1:172.21.2.120:6789/0 conn(0x7fbe300d1870 legacy=0x7fbe301b04c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:45:38.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.761 7fbe2ed73700 1 -- 172.21.2.112:0/1230075491 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe30088070 con 0x7fbe300d1c70 2021-02-10T11:45:38.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.761 7fbe2ed73700 1 --2- 172.21.2.112:0/1230075491 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbe300d1c70 0x7fbe300cef60 secure :-1 s=READY pgs=2774 cs=0 l=1 rev1=1 rx=0x7fbe18006700 tx=0x7fbe18009b80).ready entity=mon.0 client_cookie=8c9521ba73608b28 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:38.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.761 7fbe2e572700 1 -- 172.21.2.112:0/1230075491 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbe1800c0c0 con 0x7fbe300d1c70 2021-02-10T11:45:38.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.761 7fbe2e572700 1 -- 172.21.2.112:0/1230075491 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe1800c340 con 0x7fbe300d1c70 2021-02-10T11:45:38.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.761 7fbe2e572700 1 -- 172.21.2.112:0/1230075491 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbe1800c860 con 0x7fbe300d1c70 2021-02-10T11:45:38.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.761 7fbe35922700 1 -- 172.21.2.112:0/1230075491 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbe300d1c70 msgr2=0x7fbe300cef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:38.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.761 7fbe35922700 1 --2- 172.21.2.112:0/1230075491 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbe300d1c70 0x7fbe300cef60 secure :-1 s=READY pgs=2774 cs=0 l=1 rev1=1 rx=0x7fbe18006700 tx=0x7fbe18009b80).stop 2021-02-10T11:45:38.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.762 7fbe35922700 1 -- 172.21.2.112:0/1230075491 shutdown_connections 2021-02-10T11:45:38.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.762 7fbe35922700 1 --2- 172.21.2.112:0/1230075491 >> v2:172.21.2.120:3300/0 conn(0x7fbe300cf580 0x7fbe300cf980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:38.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.762 7fbe35922700 1 --2- 172.21.2.112:0/1230075491 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbe300d1c70 0x7fbe300cef60 unknown :-1 s=CLOSED pgs=2774 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:38.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.762 7fbe35922700 1 -- 172.21.2.112:0/1230075491 >> 172.21.2.112:0/1230075491 conn(0x7fbe300ce420 msgr2=0x7fbe300d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:38.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.762 7fbe35922700 1 -- 172.21.2.112:0/1230075491 shutdown_connections 2021-02-10T11:45:38.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.762 7fbe35922700 1 -- 172.21.2.112:0/1230075491 wait complete. 2021-02-10T11:45:38.782 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.777 7fbe35922700 1 Processor -- start 2021-02-10T11:45:38.782 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.777 7fbe35922700 1 -- start start 2021-02-10T11:45:38.782 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.777 7fbe35922700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbe300c3820 0x7fbe30084cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:38.782 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.777 7fbe35922700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fbe300a0290 0x7fbe300852d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:38.782 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.777 7fbe35922700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbe30086270 0x7fbe30086670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:38.782 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.777 7fbe35922700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe300c0700 con 0x7fbe30086270 2021-02-10T11:45:38.782 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.777 7fbe35922700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe300858f0 con 0x7fbe300a0290 2021-02-10T11:45:38.782 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.777 7fbe35922700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe30085b10 con 0x7fbe300c3820 2021-02-10T11:45:38.782 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.778 7fbe2fd75700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbe30086270 0x7fbe30086670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:38.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.778 7fbe2fd75700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbe30086270 0x7fbe30086670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:58932/0 (socket says 172.21.2.112:58932) 2021-02-10T11:45:38.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.778 7fbe2fd75700 1 -- 172.21.2.112:0/1762687335 learned_addr learned my addr 172.21.2.112:0/1762687335 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:45:38.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.778 7fbe2ed73700 1 --2- 172.21.2.112:0/1762687335 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fbe300a0290 0x7fbe300852d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:38.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.778 7fbe2f574700 1 --2- 172.21.2.112:0/1762687335 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbe300c3820 0x7fbe30084cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:38.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.778 7fbe2fd75700 1 -- 172.21.2.112:0/1762687335 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbe300c3820 msgr2=0x7fbe30084cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:38.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.778 7fbe2fd75700 1 --2- 172.21.2.112:0/1762687335 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbe300c3820 0x7fbe30084cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:38.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.778 7fbe2fd75700 1 -- 172.21.2.112:0/1762687335 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fbe300a0290 msgr2=0x7fbe300852d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:38.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.778 7fbe2fd75700 1 --2- 172.21.2.112:0/1762687335 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fbe300a0290 0x7fbe300852d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:38.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.778 7fbe2fd75700 1 -- 172.21.2.112:0/1762687335 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe30086c90 con 0x7fbe30086270 2021-02-10T11:45:38.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.778 7fbe2fd75700 1 --2- 172.21.2.112:0/1762687335 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbe30086270 0x7fbe30086670 secure :-1 s=READY pgs=2775 cs=0 l=1 rev1=1 rx=0x7fbe28009960 tx=0x7fbe2800bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:38.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.778 7fbe27fff700 1 -- 172.21.2.112:0/1762687335 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbe28003030 con 0x7fbe30086270 2021-02-10T11:45:38.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.778 7fbe27fff700 1 -- 172.21.2.112:0/1762687335 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe28001310 con 0x7fbe30086270 2021-02-10T11:45:38.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.778 7fbe35922700 1 -- 172.21.2.112:0/1762687335 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe30086c90 con 0x7fbe30086270 2021-02-10T11:45:38.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.778 7fbe35922700 1 -- 172.21.2.112:0/1762687335 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe301a2f00 con 0x7fbe30086270 2021-02-10T11:45:38.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.778 7fbe27fff700 1 -- 172.21.2.112:0/1762687335 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbe280017f0 con 0x7fbe30086270 2021-02-10T11:45:38.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.779 7fbe27fff700 1 -- 172.21.2.112:0/1762687335 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fbe280014f0 con 0x7fbe30086270 2021-02-10T11:45:38.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.779 7fbe27fff700 1 --2- 172.21.2.112:0/1762687335 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fbe140186d0 0x7fbe1401abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:38.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.779 7fbe27fff700 1 -- 172.21.2.112:0/1762687335 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fbe28032ba0 con 0x7fbe30086270 2021-02-10T11:45:38.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.779 7fbe2f574700 1 --2- 172.21.2.112:0/1762687335 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fbe140186d0 0x7fbe1401abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:38.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.780 7fbe2f574700 1 --2- 172.21.2.112:0/1762687335 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fbe140186d0 0x7fbe1401abd0 crc :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:38.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.811 7fbe35922700 1 -- 172.21.2.112:0/1762687335 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe30082a70 con 0x7fbe30086270 2021-02-10T11:45:38.815 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.813 7fbe27fff700 1 -- 172.21.2.112:0/1762687335 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fbe28000f20 con 0x7fbe30086270 2021-02-10T11:45:38.944 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.942 7fbe35922700 1 -- 172.21.2.112:0/1762687335 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7fbe30082a70 con 0x7fbe30086270 2021-02-10T11:45:38.945 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.943 7fbe27fff700 1 -- 172.21.2.112:0/1762687335 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v161) v1 ==== 92+0+10 (secure 0 0 0) 0x7fbe28001310 con 0x7fbe30086270 2021-02-10T11:45:38.945 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:45:38.946 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.945 7fbe35922700 1 -- 172.21.2.112:0/1762687335 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fbe140186d0 msgr2=0x7fbe1401abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:38.947 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.945 7fbe35922700 1 --2- 172.21.2.112:0/1762687335 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fbe140186d0 0x7fbe1401abd0 crc :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:38.947 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.945 7fbe35922700 1 -- 172.21.2.112:0/1762687335 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbe30086270 msgr2=0x7fbe30086670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:38.947 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.945 7fbe35922700 1 --2- 172.21.2.112:0/1762687335 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbe30086270 0x7fbe30086670 secure :-1 s=READY pgs=2775 cs=0 l=1 rev1=1 rx=0x7fbe28009960 tx=0x7fbe2800bcf0).stop 2021-02-10T11:45:38.947 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.945 7fbe35922700 1 -- 172.21.2.112:0/1762687335 shutdown_connections 2021-02-10T11:45:38.947 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.945 7fbe35922700 1 --2- 172.21.2.112:0/1762687335 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fbe140186d0 0x7fbe1401abd0 unknown :-1 s=CLOSED pgs=1266 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:38.947 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.945 7fbe35922700 1 --2- 172.21.2.112:0/1762687335 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fbe300a0290 0x7fbe300852d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:38.947 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.945 7fbe35922700 1 --2- 172.21.2.112:0/1762687335 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbe30086270 0x7fbe30086670 unknown :-1 s=CLOSED pgs=2775 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:38.947 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.945 7fbe35922700 1 --2- 172.21.2.112:0/1762687335 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbe300c3820 0x7fbe30084cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:38.947 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.945 7fbe35922700 1 -- 172.21.2.112:0/1762687335 >> 172.21.2.112:0/1762687335 conn(0x7fbe300872c0 msgr2=0x7fbe30095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:38.948 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.945 7fbe35922700 1 -- 172.21.2.112:0/1762687335 shutdown_connections 2021-02-10T11:45:38.948 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:38.945 7fbe35922700 1 -- 172.21.2.112:0/1762687335 wait complete. 2021-02-10T11:45:38.955 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:45:39.391 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.372 7f96b4486700 1 Processor -- start 2021-02-10T11:45:39.391 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.372 7f96b4486700 1 -- start start 2021-02-10T11:45:39.391 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.373 7f96b4486700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f96ac10aff0 0x7f96ac0cf750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:39.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.373 7f96b4486700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96ac0fe1c0 con 0x7f96ac0d1870 2021-02-10T11:45:39.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.373 7f96b4486700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96ac1b0cb0 con 0x7f96ac0d1c70 2021-02-10T11:45:39.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.373 7f96b4486700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96ac109a60 con 0x7f96ac10aff0 2021-02-10T11:45:39.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.373 7f96b22a2700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f96ac0d1870 0x7f96ac1b04c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:58398/0 (socket says 172.21.2.112:58398) 2021-02-10T11:45:39.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.373 7f96b22a2700 1 -- 172.21.2.112:0/1756463847 learned_addr learned my addr 172.21.2.112:0/1756463847 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:45:39.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.373 7f96b12a0700 1 -- 172.21.2.112:0/1756463847 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1552360212 0 0) 0x7f96a00010b0 con 0x7f96ac0d1870 2021-02-10T11:45:39.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.373 7f96b12a0700 1 -- 172.21.2.112:0/1756463847 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9698002fe0 con 0x7f96ac0d1870 2021-02-10T11:45:39.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.373 7f96b2aa3700 1 --2- 172.21.2.112:0/1756463847 >> v2:172.21.2.120:3300/0 conn(0x7f96ac10aff0 0x7f96ac0cf750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:39.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.374 7f96b12a0700 1 -- 172.21.2.112:0/1756463847 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 71646890 0 0) 0x7f96a00010b0 con 0x7f96ac0d1870 2021-02-10T11:45:39.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.374 7f96b12a0700 1 -- 172.21.2.112:0/1756463847 >> v2:172.21.2.120:3300/0 conn(0x7f96ac10aff0 msgr2=0x7f96ac0cf750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:39.393 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.374 7f96b12a0700 1 --2- 172.21.2.112:0/1756463847 >> v2:172.21.2.120:3300/0 conn(0x7f96ac10aff0 0x7f96ac0cf750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:39.393 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.374 7f96b12a0700 1 -- 172.21.2.112:0/1756463847 >> v1:172.21.2.120:6789/0 conn(0x7f96ac0d1c70 legacy=0x7f96ac0cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:39.393 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.374 7f96b12a0700 1 -- 172.21.2.112:0/1756463847 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96ac10b3f0 con 0x7f96ac0d1870 2021-02-10T11:45:39.393 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.374 7f96b12a0700 1 -- 172.21.2.112:0/1756463847 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f96a00015b0 con 0x7f96ac0d1870 2021-02-10T11:45:39.393 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.374 7f96b12a0700 1 -- 172.21.2.112:0/1756463847 >> v1:172.21.2.112:6789/0 conn(0x7f96ac0d1870 legacy=0x7f96ac1b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:39.393 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.374 7f96b12a0700 1 --2- 172.21.2.112:0/1756463847 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9698003b10 0x7f9698006010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:39.393 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.374 7f96b12a0700 1 --2- 172.21.2.112:0/1756463847 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f96980067a0 0x7f9698008cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:39.393 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.374 7f96b12a0700 1 --2- 172.21.2.112:0/1756463847 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f96980095c0 0x7f969800bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:39.393 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.374 7f96b12a0700 1 -- 172.21.2.112:0/1756463847 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f969800c2f0 con 0x7f96980095c0 2021-02-10T11:45:39.394 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.374 7f96b12a0700 1 -- 172.21.2.112:0/1756463847 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f969800c510 con 0x7f96980067a0 2021-02-10T11:45:39.394 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.374 7f96b12a0700 1 -- 172.21.2.112:0/1756463847 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f969800c730 con 0x7f9698003b10 2021-02-10T11:45:39.394 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.374 7f96b2aa3700 1 --2- 172.21.2.112:0/1756463847 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f96980095c0 0x7f969800bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:39.394 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.374 7f96b1aa1700 1 --2- 172.21.2.112:0/1756463847 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f96980067a0 0x7f9698008cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:39.394 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.374 7f96b22a2700 1 --2- 172.21.2.112:0/1756463847 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9698003b10 0x7f9698006010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:39.394 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.374 7f96b2aa3700 1 -- 172.21.2.112:0/1756463847 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9698003b10 msgr2=0x7f9698006010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:39.394 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.374 7f96b2aa3700 1 --2- 172.21.2.112:0/1756463847 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9698003b10 0x7f9698006010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:39.394 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.374 7f96b2aa3700 1 -- 172.21.2.112:0/1756463847 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f96980067a0 msgr2=0x7f9698008cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:39.394 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.374 7f96b2aa3700 1 --2- 172.21.2.112:0/1756463847 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f96980067a0 0x7f9698008cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:39.394 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.374 7f96b2aa3700 1 -- 172.21.2.112:0/1756463847 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96980509a0 con 0x7f96980095c0 2021-02-10T11:45:39.394 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.374 7f96b2aa3700 1 -- 172.21.2.112:0/1756463847 reap_dead start 2021-02-10T11:45:39.395 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.374 7f96b2aa3700 1 --2- 172.21.2.112:0/1756463847 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f96980095c0 0x7f969800bac0 secure :-1 s=READY pgs=2777 cs=0 l=1 rev1=1 rx=0x7f96a4006a40 tx=0x7f96a4009f40).ready entity=mon.0 client_cookie=e52134aab99035f3 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:39.395 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.374 7f96b12a0700 1 -- 172.21.2.112:0/1756463847 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f96a4006ab0 con 0x7f96980095c0 2021-02-10T11:45:39.395 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.374 7f96b12a0700 1 -- 172.21.2.112:0/1756463847 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96a4007070 con 0x7f96980095c0 2021-02-10T11:45:39.395 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.374 7f96b12a0700 1 -- 172.21.2.112:0/1756463847 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f96a4007590 con 0x7f96980095c0 2021-02-10T11:45:39.395 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.374 7f96b4486700 1 -- 172.21.2.112:0/1756463847 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f96980095c0 msgr2=0x7f969800bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:39.395 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.374 7f96b4486700 1 --2- 172.21.2.112:0/1756463847 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f96980095c0 0x7f969800bac0 secure :-1 s=READY pgs=2777 cs=0 l=1 rev1=1 rx=0x7f96a4006a40 tx=0x7f96a4009f40).stop 2021-02-10T11:45:39.395 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.375 7f96b4486700 1 -- 172.21.2.112:0/1756463847 shutdown_connections 2021-02-10T11:45:39.395 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.375 7f96b4486700 1 --2- 172.21.2.112:0/1756463847 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f96980095c0 0x7f969800bac0 unknown :-1 s=CLOSED pgs=2777 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:39.395 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.375 7f96b4486700 1 -- 172.21.2.112:0/1756463847 >> 172.21.2.112:0/1756463847 conn(0x7f96ac0ce420 msgr2=0x7f96ac0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:39.395 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.375 7f96b4486700 1 -- 172.21.2.112:0/1756463847 shutdown_connections 2021-02-10T11:45:39.395 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.375 7f96b4486700 1 -- 172.21.2.112:0/1756463847 wait complete. 2021-02-10T11:45:39.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.389 7f96b4486700 1 Processor -- start 2021-02-10T11:45:39.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.389 7f96b4486700 1 -- start start 2021-02-10T11:45:39.398 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.390 7f96b4486700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f96ac09bcc0 0x7f96ac09c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:39.398 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.390 7f96b4486700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f96ac0a0820 0x7f96ac0a0c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:39.398 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.390 7f96b4486700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f96ac0a1240 0x7f96ac0858d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:39.398 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.390 7f96b4486700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96ac0a66b0 con 0x7f96ac09bcc0 2021-02-10T11:45:39.398 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.390 7f96b4486700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96ac1a49b0 con 0x7f96ac0a0820 2021-02-10T11:45:39.398 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.390 7f96b4486700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96ac1a4b90 con 0x7f96ac0a1240 2021-02-10T11:45:39.398 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.390 7f96b22a2700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f96ac09bcc0 0x7f96ac09c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:39.399 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.390 7f96b22a2700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f96ac09bcc0 0x7f96ac09c0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:58948/0 (socket says 172.21.2.112:58948) 2021-02-10T11:45:39.399 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.390 7f96b22a2700 1 -- 172.21.2.112:0/3275238777 learned_addr learned my addr 172.21.2.112:0/3275238777 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:45:39.399 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.390 7f96b22a2700 1 -- 172.21.2.112:0/3275238777 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f96ac0a1240 msgr2=0x7f96ac0858d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:39.399 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.390 7f96b22a2700 1 --2- 172.21.2.112:0/3275238777 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f96ac0a1240 0x7f96ac0858d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:39.399 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.390 7f96b22a2700 1 -- 172.21.2.112:0/3275238777 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f96ac0a0820 msgr2=0x7f96ac0a0c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:39.399 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.390 7f96b22a2700 1 --2- 172.21.2.112:0/3275238777 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f96ac0a0820 0x7f96ac0a0c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:39.399 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.390 7f96b22a2700 1 -- 172.21.2.112:0/3275238777 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96ac0a1640 con 0x7f96ac09bcc0 2021-02-10T11:45:39.399 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.390 7f96b22a2700 1 --2- 172.21.2.112:0/3275238777 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f96ac09bcc0 0x7f96ac09c0c0 secure :-1 s=READY pgs=2778 cs=0 l=1 rev1=1 rx=0x7f96a0001c00 tx=0x7f96a000a8e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:39.399 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.391 7f969effd700 1 -- 172.21.2.112:0/3275238777 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f96a00009b0 con 0x7f96ac09bcc0 2021-02-10T11:45:39.399 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.391 7f969effd700 1 -- 172.21.2.112:0/3275238777 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96a000acd0 con 0x7f96ac09bcc0 2021-02-10T11:45:39.399 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.391 7f969effd700 1 -- 172.21.2.112:0/3275238777 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f96a0004540 con 0x7f96ac09bcc0 2021-02-10T11:45:39.400 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.391 7f96b4486700 1 -- 172.21.2.112:0/3275238777 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96ac0a1640 con 0x7f96ac09bcc0 2021-02-10T11:45:39.400 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.391 7f96b4486700 1 -- 172.21.2.112:0/3275238777 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96ac0861e0 con 0x7f96ac09bcc0 2021-02-10T11:45:39.400 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.391 7f969effd700 1 -- 172.21.2.112:0/3275238777 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f96a0004140 con 0x7f96ac09bcc0 2021-02-10T11:45:39.400 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.392 7f969effd700 1 --2- 172.21.2.112:0/3275238777 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f96940186c0 0x7f969401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:39.400 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.392 7f969effd700 1 -- 172.21.2.112:0/3275238777 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f96a0032590 con 0x7f96ac09bcc0 2021-02-10T11:45:39.400 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.392 7f96b1aa1700 1 --2- 172.21.2.112:0/3275238777 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f96940186c0 0x7f969401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:39.400 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.393 7f96b1aa1700 1 --2- 172.21.2.112:0/3275238777 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f96940186c0 0x7f969401abc0 crc :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:39.424 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.423 7f96b4486700 1 -- 172.21.2.112:0/3275238777 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96ac086a70 con 0x7f96ac09bcc0 2021-02-10T11:45:39.427 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.425 7f969effd700 1 -- 172.21.2.112:0/3275238777 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f96a0000940 con 0x7f96ac09bcc0 2021-02-10T11:45:39.555 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.553 7f96b4486700 1 -- 172.21.2.112:0/3275238777 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f96ac086a70 con 0x7f96ac09bcc0 2021-02-10T11:45:39.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.554 7f969effd700 1 -- 172.21.2.112:0/3275238777 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v161) v1 ==== 104+0+10 (secure 0 0 0) 0x7f96a001f520 con 0x7f96ac09bcc0 2021-02-10T11:45:39.556 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:45:39.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.556 7f96b4486700 1 -- 172.21.2.112:0/3275238777 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f96940186c0 msgr2=0x7f969401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:39.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.556 7f96b4486700 1 --2- 172.21.2.112:0/3275238777 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f96940186c0 0x7f969401abc0 crc :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:39.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.556 7f96b4486700 1 -- 172.21.2.112:0/3275238777 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f96ac09bcc0 msgr2=0x7f96ac09c0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:39.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.556 7f96b4486700 1 --2- 172.21.2.112:0/3275238777 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f96ac09bcc0 0x7f96ac09c0c0 secure :-1 s=READY pgs=2778 cs=0 l=1 rev1=1 rx=0x7f96a0001c00 tx=0x7f96a000a8e0).stop 2021-02-10T11:45:39.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.556 7f96b4486700 1 -- 172.21.2.112:0/3275238777 shutdown_connections 2021-02-10T11:45:39.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.556 7f96b4486700 1 --2- 172.21.2.112:0/3275238777 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f96940186c0 0x7f969401abc0 unknown :-1 s=CLOSED pgs=1267 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:39.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.556 7f96b4486700 1 --2- 172.21.2.112:0/3275238777 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f96ac0a0820 0x7f96ac0a0c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:39.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.556 7f96b4486700 1 --2- 172.21.2.112:0/3275238777 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f96ac0a1240 0x7f96ac0858d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:39.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.556 7f96b4486700 1 --2- 172.21.2.112:0/3275238777 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f96ac09bcc0 0x7f96ac09c0c0 unknown :-1 s=CLOSED pgs=2778 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:39.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.556 7f96b4486700 1 -- 172.21.2.112:0/3275238777 >> 172.21.2.112:0/3275238777 conn(0x7f96ac0bef00 msgr2=0x7f96ac09d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:39.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.556 7f96b4486700 1 -- 172.21.2.112:0/3275238777 shutdown_connections 2021-02-10T11:45:39.559 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.556 7f96b4486700 1 -- 172.21.2.112:0/3275238777 wait complete. 2021-02-10T11:45:39.566 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:45:39.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.981 7fd651ef0700 1 Processor -- start 2021-02-10T11:45:39.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.981 7fd651ef0700 1 -- start start 2021-02-10T11:45:39.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.981 7fd651ef0700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fd64c10afe0 0x7fd64c0cf740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:39.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.981 7fd651ef0700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd64c0fe1b0 con 0x7fd64c0d1c60 2021-02-10T11:45:40.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.981 7fd651ef0700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd64c1b0ca0 con 0x7fd64c0d1860 2021-02-10T11:45:40.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.981 7fd651ef0700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd64c109a50 con 0x7fd64c10afe0 2021-02-10T11:45:40.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.982 7fd64bfff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fd64c10afe0 0x7fd64c0cf740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:40.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.982 7fd64bfff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fd64c10afe0 0x7fd64c0cf740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:58960/0 (socket says 172.21.2.112:58960) 2021-02-10T11:45:40.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.982 7fd64bfff700 1 -- 172.21.2.112:0/3548863307 learned_addr learned my addr 172.21.2.112:0/3548863307 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:45:40.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.982 7fd64bfff700 1 -- 172.21.2.112:0/3548863307 >> v1:172.21.2.120:6789/0 conn(0x7fd64c0d1860 legacy=0x7fd64c1b04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:40.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.982 7fd64bfff700 1 -- 172.21.2.112:0/3548863307 >> v1:172.21.2.112:6789/0 conn(0x7fd64c0d1c60 legacy=0x7fd64c0cef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:40.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.982 7fd64bfff700 1 -- 172.21.2.112:0/3548863307 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd64c10b3e0 con 0x7fd64c10afe0 2021-02-10T11:45:40.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.982 7fd64bfff700 1 --2- 172.21.2.112:0/3548863307 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd64c10afe0 0x7fd64c0cf740 secure :-1 s=READY pgs=2780 cs=0 l=1 rev1=1 rx=0x7fd644006f00 tx=0x7fd644009f10).ready entity=mon.0 client_cookie=cdf1cb6bc300334f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:40.001 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.982 7fd64a7fc700 1 -- 172.21.2.112:0/3548863307 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd64400b3b0 con 0x7fd64c10afe0 2021-02-10T11:45:40.001 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.982 7fd64a7fc700 1 -- 172.21.2.112:0/3548863307 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd64400b730 con 0x7fd64c10afe0 2021-02-10T11:45:40.001 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.982 7fd64a7fc700 1 -- 172.21.2.112:0/3548863307 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd64400bc50 con 0x7fd64c10afe0 2021-02-10T11:45:40.001 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.982 7fd651ef0700 1 -- 172.21.2.112:0/3548863307 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd64c10afe0 msgr2=0x7fd64c0cf740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:40.001 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.982 7fd651ef0700 1 --2- 172.21.2.112:0/3548863307 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd64c10afe0 0x7fd64c0cf740 secure :-1 s=READY pgs=2780 cs=0 l=1 rev1=1 rx=0x7fd644006f00 tx=0x7fd644009f10).stop 2021-02-10T11:45:40.001 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.983 7fd651ef0700 1 -- 172.21.2.112:0/3548863307 shutdown_connections 2021-02-10T11:45:40.001 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.983 7fd651ef0700 1 --2- 172.21.2.112:0/3548863307 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd64c10afe0 0x7fd64c0cf740 unknown :-1 s=CLOSED pgs=2780 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:40.001 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.983 7fd651ef0700 1 -- 172.21.2.112:0/3548863307 >> 172.21.2.112:0/3548863307 conn(0x7fd64c0ce410 msgr2=0x7fd64c0d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:40.001 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.983 7fd651ef0700 1 -- 172.21.2.112:0/3548863307 shutdown_connections 2021-02-10T11:45:40.001 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.983 7fd651ef0700 1 -- 172.21.2.112:0/3548863307 wait complete. 2021-02-10T11:45:40.001 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.998 7fd651ef0700 1 Processor -- start 2021-02-10T11:45:40.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.998 7fd651ef0700 1 -- start start 2021-02-10T11:45:40.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.998 7fd651ef0700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd64c0c3500 0x7fd64c0a2e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:40.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.998 7fd651ef0700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd64c0b79a0 0x7fd64c0b7da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:40.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.998 7fd651ef0700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd64c0b9e40 0x7fd64c0ba240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:40.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.998 7fd651ef0700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd64c1065b0 con 0x7fd64c0b9e40 2021-02-10T11:45:40.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.998 7fd651ef0700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd64c0b70f0 con 0x7fd64c0c3500 2021-02-10T11:45:40.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.998 7fd651ef0700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd64c0a3450 con 0x7fd64c0b79a0 2021-02-10T11:45:40.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.998 7fd64bfff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd64c0b9e40 0x7fd64c0ba240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:40.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.998 7fd64bfff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd64c0b9e40 0x7fd64c0ba240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:58966/0 (socket says 172.21.2.112:58966) 2021-02-10T11:45:40.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.998 7fd64bfff700 1 -- 172.21.2.112:0/3220214404 learned_addr learned my addr 172.21.2.112:0/3220214404 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:45:40.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.999 7fd64bfff700 1 -- 172.21.2.112:0/3220214404 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd64c0b79a0 msgr2=0x7fd64c0b7da0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:45:40.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.999 7fd64bfff700 1 --2- 172.21.2.112:0/3220214404 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd64c0b79a0 0x7fd64c0b7da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:40.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.999 7fd64bfff700 1 -- 172.21.2.112:0/3220214404 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd64c0c3500 msgr2=0x7fd64c0a2e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:40.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.999 7fd64bfff700 1 --2- 172.21.2.112:0/3220214404 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd64c0c3500 0x7fd64c0a2e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:40.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.999 7fd64bfff700 1 -- 172.21.2.112:0/3220214404 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd64c0b83c0 con 0x7fd64c0b9e40 2021-02-10T11:45:40.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.999 7fd64bfff700 1 --2- 172.21.2.112:0/3220214404 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd64c0b9e40 0x7fd64c0ba240 secure :-1 s=READY pgs=2781 cs=0 l=1 rev1=1 rx=0x7fd64400b9b0 tx=0x7fd64400bd50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:40.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.999 7fd643fff700 1 -- 172.21.2.112:0/3220214404 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd644005820 con 0x7fd64c0b9e40 2021-02-10T11:45:40.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.999 7fd643fff700 1 -- 172.21.2.112:0/3220214404 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd644005aa0 con 0x7fd64c0b9e40 2021-02-10T11:45:40.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.999 7fd643fff700 1 -- 172.21.2.112:0/3220214404 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd644006030 con 0x7fd64c0b9e40 2021-02-10T11:45:40.005 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.999 7fd651ef0700 1 -- 172.21.2.112:0/3220214404 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd64c0b83c0 con 0x7fd64c0b9e40 2021-02-10T11:45:40.005 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:39.999 7fd651ef0700 1 -- 172.21.2.112:0/3220214404 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd64c0bab50 con 0x7fd64c0b9e40 2021-02-10T11:45:40.005 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:40.000 7fd643fff700 1 -- 172.21.2.112:0/3220214404 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd644006030 con 0x7fd64c0b9e40 2021-02-10T11:45:40.005 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:40.000 7fd643fff700 1 --2- 172.21.2.112:0/3220214404 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd6300186c0 0x7fd63001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:40.005 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:40.000 7fd643fff700 1 -- 172.21.2.112:0/3220214404 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fd644032940 con 0x7fd64c0b9e40 2021-02-10T11:45:40.005 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:40.000 7fd64b7fe700 1 --2- 172.21.2.112:0/3220214404 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd6300186c0 0x7fd63001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:40.005 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:40.001 7fd64b7fe700 1 --2- 172.21.2.112:0/3220214404 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd6300186c0 0x7fd63001abc0 crc :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:40.033 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:40.031 7fd651ef0700 1 -- 172.21.2.112:0/3220214404 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd64c0b70f0 con 0x7fd64c0b9e40 2021-02-10T11:45:40.036 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:40.034 7fd643fff700 1 -- 172.21.2.112:0/3220214404 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd644005c80 con 0x7fd64c0b9e40 2021-02-10T11:45:40.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:40.161 7fd651ef0700 1 -- 172.21.2.112:0/3220214404 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7fd64c0b70f0 con 0x7fd64c0b9e40 2021-02-10T11:45:40.164 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:40.162 7fd643fff700 1 -- 172.21.2.112:0/3220214404 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v161) v1 ==== 100+0+10 (secure 0 0 0) 0x7fd644006030 con 0x7fd64c0b9e40 2021-02-10T11:45:40.164 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:45:40.165 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:40.164 7fd651ef0700 1 -- 172.21.2.112:0/3220214404 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd6300186c0 msgr2=0x7fd63001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:40.166 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:40.164 7fd651ef0700 1 --2- 172.21.2.112:0/3220214404 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd6300186c0 0x7fd63001abc0 crc :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:40.166 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:40.164 7fd651ef0700 1 -- 172.21.2.112:0/3220214404 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd64c0b9e40 msgr2=0x7fd64c0ba240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:40.166 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:40.164 7fd651ef0700 1 --2- 172.21.2.112:0/3220214404 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd64c0b9e40 0x7fd64c0ba240 secure :-1 s=READY pgs=2781 cs=0 l=1 rev1=1 rx=0x7fd64400b9b0 tx=0x7fd64400bd50).stop 2021-02-10T11:45:40.166 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:40.164 7fd651ef0700 1 -- 172.21.2.112:0/3220214404 shutdown_connections 2021-02-10T11:45:40.166 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:40.164 7fd651ef0700 1 --2- 172.21.2.112:0/3220214404 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd6300186c0 0x7fd63001abc0 unknown :-1 s=CLOSED pgs=1268 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:40.166 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:40.164 7fd651ef0700 1 --2- 172.21.2.112:0/3220214404 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd64c0b9e40 0x7fd64c0ba240 unknown :-1 s=CLOSED pgs=2781 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:40.166 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:40.164 7fd651ef0700 1 --2- 172.21.2.112:0/3220214404 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd64c0b79a0 0x7fd64c0b7da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:40.166 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:40.164 7fd651ef0700 1 --2- 172.21.2.112:0/3220214404 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd64c0c3500 0x7fd64c0a2e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:40.166 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:40.164 7fd651ef0700 1 -- 172.21.2.112:0/3220214404 >> 172.21.2.112:0/3220214404 conn(0x7fd64c080ba0 msgr2=0x7fd64c0ffca0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:40.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:40.164 7fd651ef0700 1 -- 172.21.2.112:0/3220214404 shutdown_connections 2021-02-10T11:45:40.167 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:40.164 7fd651ef0700 1 -- 172.21.2.112:0/3220214404 wait complete. 2021-02-10T11:45:40.174 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:45:40.175 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:45:40.602 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:40.584 7f3ea8535700 1 Processor -- start 2021-02-10T11:45:40.602 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:40.584 7f3ea8535700 1 -- start start 2021-02-10T11:45:40.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:40.585 7f3ea8535700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f3ea00d1870 0x7f3ea01b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:40.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:40.585 7f3ea8535700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3ea01b0a10 0x7f3ea00cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:40.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:40.585 7f3ea8535700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ea00f0b10 con 0x7f3ea00d1c70 2021-02-10T11:45:40.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:40.585 7f3ea8535700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ea00e5ed0 con 0x7f3ea00d1870 2021-02-10T11:45:40.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:40.585 7f3ea8535700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ea00f0650 con 0x7f3ea01b0a10 2021-02-10T11:45:40.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:40.585 7f3ea6351700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f3ea00d1870 0x7f3ea01b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:40.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:40.585 7f3ea6351700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f3ea00d1870 0x7f3ea01b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:58970/0 (socket says 172.21.2.112:58970) 2021-02-10T11:45:40.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:40.585 7f3ea6351700 1 -- 172.21.2.112:0/2664134591 learned_addr learned my addr 172.21.2.112:0/2664134591 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:45:40.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:40.585 7f3ea5b50700 1 --2- 172.21.2.112:0/2664134591 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3ea01b0a10 0x7f3ea00cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:40.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:40.585 7f3ea6351700 1 -- 172.21.2.112:0/2664134591 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3ea01b0a10 msgr2=0x7f3ea00cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:40.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:40.585 7f3ea6351700 1 --2- 172.21.2.112:0/2664134591 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3ea01b0a10 0x7f3ea00cf330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:40.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:40.585 7f3ea6351700 1 -- 172.21.2.112:0/2664134591 >> v1:172.21.2.120:6789/0 conn(0x7f3ea00d1c70 legacy=0x7f3ea00cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:40.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:40.585 7f3ea6351700 1 -- 172.21.2.112:0/2664134591 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ea00f0870 con 0x7f3ea00d1870 2021-02-10T11:45:40.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:40.585 7f3ea6351700 1 --2- 172.21.2.112:0/2664134591 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3ea00d1870 0x7f3ea01b03f0 secure :-1 s=READY pgs=2782 cs=0 l=1 rev1=1 rx=0x7f3e94006700 tx=0x7f3e94009b80).ready entity=mon.0 client_cookie=b229c5205ff560d6 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:40.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:40.586 7f3ea534f700 1 -- 172.21.2.112:0/2664134591 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3e9400c0c0 con 0x7f3ea00d1870 2021-02-10T11:45:40.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:40.586 7f3ea534f700 1 -- 172.21.2.112:0/2664134591 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e9400c440 con 0x7f3ea00d1870 2021-02-10T11:45:40.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:40.586 7f3ea534f700 1 -- 172.21.2.112:0/2664134591 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3e9400c960 con 0x7f3ea00d1870 2021-02-10T11:45:40.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:40.586 7f3ea8535700 1 -- 172.21.2.112:0/2664134591 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3ea00d1870 msgr2=0x7f3ea01b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:40.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:40.586 7f3ea8535700 1 --2- 172.21.2.112:0/2664134591 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3ea00d1870 0x7f3ea01b03f0 secure :-1 s=READY pgs=2782 cs=0 l=1 rev1=1 rx=0x7f3e94006700 tx=0x7f3e94009b80).stop 2021-02-10T11:45:40.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:40.586 7f3ea8535700 1 -- 172.21.2.112:0/2664134591 shutdown_connections 2021-02-10T11:45:40.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:40.586 7f3ea8535700 1 --2- 172.21.2.112:0/2664134591 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3ea01b0a10 0x7f3ea00cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:40.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:40.586 7f3ea8535700 1 --2- 172.21.2.112:0/2664134591 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3ea00d1870 0x7f3ea01b03f0 unknown :-1 s=CLOSED pgs=2782 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:40.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:40.586 7f3ea8535700 1 -- 172.21.2.112:0/2664134591 >> 172.21.2.112:0/2664134591 conn(0x7f3ea00ce450 msgr2=0x7f3ea00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:40.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:40.586 7f3ea8535700 1 -- 172.21.2.112:0/2664134591 shutdown_connections 2021-02-10T11:45:40.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:40.586 7f3ea8535700 1 -- 172.21.2.112:0/2664134591 wait complete. 2021-02-10T11:45:40.606 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:40.601 7f3ea8535700 1 Processor -- start 2021-02-10T11:45:40.606 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:40.601 7f3ea8535700 1 -- start start 2021-02-10T11:45:40.606 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:40.601 7f3ea8535700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3ea01988e0 0x7f3ea01a0f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:40.607 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:40.601 7f3ea8535700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3ea01a1570 0x7f3ea00b1680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:40.607 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:40.601 7f3ea8535700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3ea00b1ca0 0x7f3ea00b20a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:40.607 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:40.601 7f3ea8535700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ea01990c0 con 0x7f3ea01988e0 2021-02-10T11:45:40.607 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:40.601 7f3ea8535700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ea0198ce0 con 0x7f3ea01a1570 2021-02-10T11:45:40.607 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:40.601 7f3ea8535700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ea0198320 con 0x7f3ea00b1ca0 2021-02-10T11:45:40.607 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:40.601 7f3ea6351700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3ea01988e0 0x7f3ea01a0f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:40.607 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:40.601 7f3ea6351700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3ea01988e0 0x7f3ea01a0f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:58972/0 (socket says 172.21.2.112:58972) 2021-02-10T11:45:40.607 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:40.601 7f3ea6351700 1 -- 172.21.2.112:0/3156492031 learned_addr learned my addr 172.21.2.112:0/3156492031 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:45:40.607 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:40.602 7f3ea6351700 1 -- 172.21.2.112:0/3156492031 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3ea00b1ca0 msgr2=0x7f3ea00b20a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:45:40.607 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:40.602 7f3ea6351700 1 --2- 172.21.2.112:0/3156492031 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3ea00b1ca0 0x7f3ea00b20a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:40.607 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:40.602 7f3ea6351700 1 -- 172.21.2.112:0/3156492031 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3ea01a1570 msgr2=0x7f3ea00b1680 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:45:40.607 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:40.602 7f3ea6351700 1 --2- 172.21.2.112:0/3156492031 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3ea01a1570 0x7f3ea00b1680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:40.608 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:40.602 7f3ea6351700 1 -- 172.21.2.112:0/3156492031 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ea01a1f20 con 0x7f3ea01988e0 2021-02-10T11:45:40.608 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:40.602 7f3ea6351700 1 --2- 172.21.2.112:0/3156492031 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3ea01988e0 0x7f3ea01a0f50 secure :-1 s=READY pgs=2783 cs=0 l=1 rev1=1 rx=0x7f3e9400b5e0 tx=0x7f3e9400b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:40.608 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:40.602 7f3e92ffd700 1 -- 172.21.2.112:0/3156492031 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3e9400bd70 con 0x7f3ea01988e0 2021-02-10T11:45:40.608 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:40.602 7f3e92ffd700 1 -- 172.21.2.112:0/3156492031 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e94002030 con 0x7f3ea01988e0 2021-02-10T11:45:40.608 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:40.602 7f3ea8535700 1 -- 172.21.2.112:0/3156492031 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ea01a1f20 con 0x7f3ea01988e0 2021-02-10T11:45:40.608 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:40.602 7f3ea8535700 1 -- 172.21.2.112:0/3156492031 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ea01a24b0 con 0x7f3ea01988e0 2021-02-10T11:45:40.608 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:40.602 7f3e92ffd700 1 -- 172.21.2.112:0/3156492031 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3e94002550 con 0x7f3ea01988e0 2021-02-10T11:45:40.608 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:40.603 7f3e92ffd700 1 -- 172.21.2.112:0/3156492031 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3e94000c90 con 0x7f3ea01988e0 2021-02-10T11:45:40.608 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:40.603 7f3e92ffd700 1 --2- 172.21.2.112:0/3156492031 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3e88018730 0x7f3e8801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:40.608 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:40.603 7f3e92ffd700 1 -- 172.21.2.112:0/3156492031 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f3e94032810 con 0x7f3ea01988e0 2021-02-10T11:45:40.608 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:40.603 7f3ea5b50700 1 --2- 172.21.2.112:0/3156492031 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3e88018730 0x7f3e8801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:40.608 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:40.604 7f3ea5b50700 1 --2- 172.21.2.112:0/3156492031 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3e88018730 0x7f3e8801ac10 crc :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:40.636 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:40.635 7f3ea8535700 1 -- 172.21.2.112:0/3156492031 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ea01990a0 con 0x7f3ea01988e0 2021-02-10T11:45:40.639 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:40.637 7f3e92ffd700 1 -- 172.21.2.112:0/3156492031 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3e94000940 con 0x7f3ea01988e0 2021-02-10T11:45:40.769 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:40.767 7f3ea8535700 1 -- 172.21.2.112:0/3156492031 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3ea00ebcd0 con 0x7f3ea01988e0 2021-02-10T11:45:40.769 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:40.767 7f3e92ffd700 1 -- 172.21.2.112:0/3156492031 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 139 v139) v1 ==== 95+0+2105 (secure 0 0 0) 0x7f3e94000c90 con 0x7f3ea01988e0 2021-02-10T11:45:40.770 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 139 2021-02-10T11:45:40.770 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:45:40.770 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":139,"default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":12306,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/2635651262","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":2635651262},{"type":"v1","addr":"172.21.2.112:6833","nonce":2635651262}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":135}],"filesystems":[{"mdsmap":{"epoch":139,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:45:30.416714","modified":"2021-02-10 11:45:32.783510","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8920},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8920":{"gid":8920,"name":"b","rank":0,"incarnation":136,"state":"up:active","state_seq":3,"addr":"172.21.2.120:6835/555399099","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":555399099},{"type":"v1","addr":"172.21.2.120:6835","nonce":555399099}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[35],"metadata_pool":34,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":17}]} 2021-02-10T11:45:40.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:40.770 7f3ea8535700 1 -- 172.21.2.112:0/3156492031 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3e88018730 msgr2=0x7f3e8801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:40.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:40.770 7f3ea8535700 1 --2- 172.21.2.112:0/3156492031 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3e88018730 0x7f3e8801ac10 crc :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:40.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:40.770 7f3ea8535700 1 -- 172.21.2.112:0/3156492031 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3ea01988e0 msgr2=0x7f3ea01a0f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:40.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:40.770 7f3ea8535700 1 --2- 172.21.2.112:0/3156492031 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3ea01988e0 0x7f3ea01a0f50 secure :-1 s=READY pgs=2783 cs=0 l=1 rev1=1 rx=0x7f3e9400b5e0 tx=0x7f3e9400b980).stop 2021-02-10T11:45:40.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:40.770 7f3ea8535700 1 -- 172.21.2.112:0/3156492031 shutdown_connections 2021-02-10T11:45:40.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:40.770 7f3ea8535700 1 --2- 172.21.2.112:0/3156492031 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3e88018730 0x7f3e8801ac10 unknown :-1 s=CLOSED pgs=1269 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:40.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:40.770 7f3ea8535700 1 --2- 172.21.2.112:0/3156492031 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3ea01a1570 0x7f3ea00b1680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:40.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:40.770 7f3ea8535700 1 --2- 172.21.2.112:0/3156492031 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3ea00b1ca0 0x7f3ea00b20a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:40.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:40.770 7f3ea8535700 1 --2- 172.21.2.112:0/3156492031 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3ea01988e0 0x7f3ea01a0f50 unknown :-1 s=CLOSED pgs=2783 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:40.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:40.770 7f3ea8535700 1 -- 172.21.2.112:0/3156492031 >> 172.21.2.112:0/3156492031 conn(0x7f3ea01b9b90 msgr2=0x7f3ea018c260 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:40.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:40.770 7f3ea8535700 1 -- 172.21.2.112:0/3156492031 shutdown_connections 2021-02-10T11:45:40.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:40.770 7f3ea8535700 1 -- 172.21.2.112:0/3156492031 wait complete. 2021-02-10T11:45:40.981 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 139, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:45:30.416714', 'modified': '2021-02-10 11:45:32.783510', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8920}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8920': {'gid': 8920, 'name': 'b', 'rank': 0, 'incarnation': 136, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.120:6835/555399099', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.120:6834', 'nonce': 555399099}, {'type': 'v1', 'addr': '172.21.2.120:6835', 'nonce': 555399099}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [35], 'metadata_pool': 34, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:45:40.982 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:45:40.982 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:45:41.381 INFO:teuthology.orchestra.run.gibba020.stdout:{ 2021-02-10T11:45:41.381 INFO:teuthology.orchestra.run.gibba020.stdout: "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", 2021-02-10T11:45:41.381 INFO:teuthology.orchestra.run.gibba020.stdout: "whoami": 0, 2021-02-10T11:45:41.381 INFO:teuthology.orchestra.run.gibba020.stdout: "id": 8920, 2021-02-10T11:45:41.381 INFO:teuthology.orchestra.run.gibba020.stdout: "want_state": "up:active", 2021-02-10T11:45:41.381 INFO:teuthology.orchestra.run.gibba020.stdout: "state": "up:active", 2021-02-10T11:45:41.381 INFO:teuthology.orchestra.run.gibba020.stdout: "rank_uptime": 10.960520242999999, 2021-02-10T11:45:41.381 INFO:teuthology.orchestra.run.gibba020.stdout: "mdsmap_epoch": 139, 2021-02-10T11:45:41.381 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch": 161, 2021-02-10T11:45:41.382 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch_barrier": 161, 2021-02-10T11:45:41.382 INFO:teuthology.orchestra.run.gibba020.stdout: "uptime": 16.612841617000001 2021-02-10T11:45:41.382 INFO:teuthology.orchestra.run.gibba020.stdout:} 2021-02-10T11:45:41.385 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", "whoami": 0, "id": 8920, "want_state": "up:active", "state": "up:active", "rank_uptime": 10.960520242999999, "mdsmap_epoch": 139, "osdmap_epoch": 161, "osdmap_epoch_barrier": 161, "uptime": 16.612841617000001 } 2021-02-10T11:45:41.385 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:45:41.385 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2021-02-10T11:45:41.385 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@gibba012.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2021-02-10T11:45:41.385 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:45:41.385 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:45:41.722 DEBUG:teuthology.orchestra.run.gibba012:> sudo modprobe fuse 2021-02-10T11:45:42.067 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:45:42.067 DEBUG:teuthology.orchestra.run.gibba012:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:45:42.410 INFO:teuthology.orchestra.run.gibba012.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:45:42.410 INFO:teuthology.orchestra.run.gibba012.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:45:42.411 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:45:42.411 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:45:42.411 DEBUG:teuthology.orchestra.run.gibba012:> ls /sys/fs/fuse/connections 2021-02-10T11:45:42.747 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2021-02-10T11:45:42.747 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:45:42.750 DEBUG:teuthology.orchestra.run.gibba012:> sudo modprobe fuse 2021-02-10T11:45:43.095 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:45:43.095 DEBUG:teuthology.orchestra.run.gibba012:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:45:43.161 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba012.stderr:ceph-fuse[180092]: starting ceph client 2021-02-10T11:45:43.162 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba012.stderr:2021-02-10 11:45:43.160 7f6a1625df80 -1 init, newargv = 0x55bb02d6ad60 newargc=9 2021-02-10T11:45:43.167 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba012.stderr:ceph-fuse[180092]: starting fuse 2021-02-10T11:45:43.439 INFO:teuthology.orchestra.run.gibba012.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:45:43.439 INFO:teuthology.orchestra.run.gibba012.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:45:43.440 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:45:43.440 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:45:43.440 DEBUG:teuthology.orchestra.run.gibba012:> ls /sys/fs/fuse/connections 2021-02-10T11:45:43.774 INFO:teuthology.orchestra.run.gibba012.stdout:42 2021-02-10T11:45:43.774 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2021-02-10T11:45:43.774 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:45:43.775 DEBUG:teuthology.orchestra.run.gibba012:> sudo python3 -c ' 2021-02-10T11:45:43.775 DEBUG:teuthology.orchestra.run.gibba012:> import glob 2021-02-10T11:45:43.775 DEBUG:teuthology.orchestra.run.gibba012:> import re 2021-02-10T11:45:43.775 DEBUG:teuthology.orchestra.run.gibba012:> import os 2021-02-10T11:45:43.775 DEBUG:teuthology.orchestra.run.gibba012:> import subprocess 2021-02-10T11:45:43.775 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:45:43.775 DEBUG:teuthology.orchestra.run.gibba012:> def find_socket(client_name): 2021-02-10T11:45:43.775 DEBUG:teuthology.orchestra.run.gibba012:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2021-02-10T11:45:43.775 DEBUG:teuthology.orchestra.run.gibba012:> files = glob.glob(asok_path) 2021-02-10T11:45:43.775 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:45:43.775 DEBUG:teuthology.orchestra.run.gibba012:> # Given a non-glob path, it better be there 2021-02-10T11:45:43.775 DEBUG:teuthology.orchestra.run.gibba012:> if "*" not in asok_path: 2021-02-10T11:45:43.775 DEBUG:teuthology.orchestra.run.gibba012:> assert(len(files) == 1) 2021-02-10T11:45:43.776 DEBUG:teuthology.orchestra.run.gibba012:> return files[0] 2021-02-10T11:45:43.776 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:45:43.776 DEBUG:teuthology.orchestra.run.gibba012:> for f in files: 2021-02-10T11:45:43.776 DEBUG:teuthology.orchestra.run.gibba012:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:45:43.776 DEBUG:teuthology.orchestra.run.gibba012:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:45:43.776 DEBUG:teuthology.orchestra.run.gibba012:> return f 2021-02-10T11:45:43.776 DEBUG:teuthology.orchestra.run.gibba012:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:45:43.776 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:45:43.776 DEBUG:teuthology.orchestra.run.gibba012:> print(find_socket("client.0")) 2021-02-10T11:45:43.776 DEBUG:teuthology.orchestra.run.gibba012:> ' 2021-02-10T11:45:44.138 INFO:teuthology.orchestra.run.gibba012.stdout:/var/run/ceph/ceph-client.0.180092.asok 2021-02-10T11:45:44.141 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.180092.asok 2021-02-10T11:45:44.141 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:45:44.141 DEBUG:teuthology.orchestra.run.gibba012:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.180092.asok status 2021-02-10T11:45:44.531 INFO:teuthology.orchestra.run.gibba012.stdout:{ 2021-02-10T11:45:44.532 INFO:teuthology.orchestra.run.gibba012.stdout: "metadata": { 2021-02-10T11:45:44.532 INFO:teuthology.orchestra.run.gibba012.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:45:44.532 INFO:teuthology.orchestra.run.gibba012.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:45:44.532 INFO:teuthology.orchestra.run.gibba012.stdout: "entity_id": "0", 2021-02-10T11:45:44.532 INFO:teuthology.orchestra.run.gibba012.stdout: "hostname": "gibba012", 2021-02-10T11:45:44.532 INFO:teuthology.orchestra.run.gibba012.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2021-02-10T11:45:44.532 INFO:teuthology.orchestra.run.gibba012.stdout: "pid": "180092", 2021-02-10T11:45:44.532 INFO:teuthology.orchestra.run.gibba012.stdout: "root": "/" 2021-02-10T11:45:44.532 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:45:44.532 INFO:teuthology.orchestra.run.gibba012.stdout: "dentry_count": 0, 2021-02-10T11:45:44.533 INFO:teuthology.orchestra.run.gibba012.stdout: "dentry_pinned_count": 0, 2021-02-10T11:45:44.533 INFO:teuthology.orchestra.run.gibba012.stdout: "id": 12456, 2021-02-10T11:45:44.533 INFO:teuthology.orchestra.run.gibba012.stdout: "inst": { 2021-02-10T11:45:44.533 INFO:teuthology.orchestra.run.gibba012.stdout: "name": { 2021-02-10T11:45:44.533 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "client", 2021-02-10T11:45:44.533 INFO:teuthology.orchestra.run.gibba012.stdout: "num": 12456 2021-02-10T11:45:44.533 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:45:44.533 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": { 2021-02-10T11:45:44.533 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:45:44.533 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:0", 2021-02-10T11:45:44.533 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 1449730010 2021-02-10T11:45:44.533 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:45:44.533 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:45:44.534 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": { 2021-02-10T11:45:44.534 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:45:44.534 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:0", 2021-02-10T11:45:44.534 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 1449730010 2021-02-10T11:45:44.534 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:45:44.534 INFO:teuthology.orchestra.run.gibba012.stdout: "inst_str": "client.12456 172.21.2.112:0/1449730010", 2021-02-10T11:45:44.534 INFO:teuthology.orchestra.run.gibba012.stdout: "addr_str": "172.21.2.112:0/1449730010", 2021-02-10T11:45:44.535 INFO:teuthology.orchestra.run.gibba012.stdout: "inode_count": 1, 2021-02-10T11:45:44.535 INFO:teuthology.orchestra.run.gibba012.stdout: "mds_epoch": 139, 2021-02-10T11:45:44.535 INFO:teuthology.orchestra.run.gibba012.stdout: "osd_epoch": 161, 2021-02-10T11:45:44.535 INFO:teuthology.orchestra.run.gibba012.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:45:44.535 INFO:teuthology.orchestra.run.gibba012.stdout: "blacklisted": false 2021-02-10T11:45:44.535 INFO:teuthology.orchestra.run.gibba012.stdout:} 2021-02-10T11:45:44.536 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:45:44.536 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:45:44.536 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:45:44.872 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:45:44.873 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:45:44.873 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:45:44.873 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:45:45.220 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:45:45.649 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:45.631 7faf4e305700 1 Processor -- start 2021-02-10T11:45:45.650 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:45.631 7faf4e305700 1 -- start start 2021-02-10T11:45:45.650 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:45.631 7faf4e305700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7faf480d0340 0x7faf480ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:45.650 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:45.631 7faf4e305700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7faf480d1c70 0x7faf480efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:45.650 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:45.631 7faf4e305700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf480f0b10 con 0x7faf480d1870 2021-02-10T11:45:45.650 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:45.631 7faf4e305700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf480e5ed0 con 0x7faf480d1c70 2021-02-10T11:45:45.650 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:45.631 7faf4e305700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf481b0be0 con 0x7faf480d0340 2021-02-10T11:45:45.650 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:45.632 7faf4c922700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7faf480d1c70 0x7faf480efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:45.650 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:45.632 7faf4c922700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7faf480d1c70 0x7faf480efc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:58998/0 (socket says 172.21.2.112:58998) 2021-02-10T11:45:45.650 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:45.632 7faf4c922700 1 -- 172.21.2.112:0/2358723908 learned_addr learned my addr 172.21.2.112:0/2358723908 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:45:45.650 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:45.632 7faf4c922700 1 -- 172.21.2.112:0/2358723908 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7faf480d0340 msgr2=0x7faf480ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:45.651 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:45.632 7faf4c922700 1 --2- 172.21.2.112:0/2358723908 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7faf480d0340 0x7faf480ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:45.651 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:45.632 7faf4c922700 1 -- 172.21.2.112:0/2358723908 >> v1:172.21.2.112:6789/0 conn(0x7faf480d1870 legacy=0x7faf481b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:45.651 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:45.632 7faf46ffd700 1 -- 172.21.2.112:0/2358723908 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 465011289 0 0) 0x7faf38000ce0 con 0x7faf480d1870 2021-02-10T11:45:45.651 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:45.632 7faf4c922700 1 -- 172.21.2.112:0/2358723908 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf480f20a0 con 0x7faf480d1c70 2021-02-10T11:45:45.651 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:45.632 7faf4c922700 1 --2- 172.21.2.112:0/2358723908 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faf480d1c70 0x7faf480efc60 secure :-1 s=READY pgs=2787 cs=0 l=1 rev1=1 rx=0x7faf40006f00 tx=0x7faf40009e90).ready entity=mon.0 client_cookie=fe39d9c5972d1c25 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:45.651 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:45.633 7faf46ffd700 1 -- 172.21.2.112:0/2358723908 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faf4000b300 con 0x7faf480d1c70 2021-02-10T11:45:45.651 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:45.633 7faf46ffd700 1 -- 172.21.2.112:0/2358723908 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf4000b580 con 0x7faf480d1c70 2021-02-10T11:45:45.651 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:45.633 7faf46ffd700 1 -- 172.21.2.112:0/2358723908 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faf4000b9d0 con 0x7faf480d1c70 2021-02-10T11:45:45.651 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:45.633 7faf4e305700 1 -- 172.21.2.112:0/2358723908 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faf480d1c70 msgr2=0x7faf480efc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:45.651 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:45.633 7faf4e305700 1 --2- 172.21.2.112:0/2358723908 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faf480d1c70 0x7faf480efc60 secure :-1 s=READY pgs=2787 cs=0 l=1 rev1=1 rx=0x7faf40006f00 tx=0x7faf40009e90).stop 2021-02-10T11:45:45.651 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:45.634 7faf4e305700 1 -- 172.21.2.112:0/2358723908 shutdown_connections 2021-02-10T11:45:45.651 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:45.634 7faf4e305700 1 --2- 172.21.2.112:0/2358723908 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faf480d1c70 0x7faf480efc60 unknown :-1 s=CLOSED pgs=2787 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:45.651 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:45.634 7faf4e305700 1 --2- 172.21.2.112:0/2358723908 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7faf480d0340 0x7faf480ef640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:45.652 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:45.634 7faf4e305700 1 -- 172.21.2.112:0/2358723908 >> 172.21.2.112:0/2358723908 conn(0x7faf480ce450 msgr2=0x7faf480d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:45.652 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:45.634 7faf4e305700 1 -- 172.21.2.112:0/2358723908 shutdown_connections 2021-02-10T11:45:45.652 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:45.634 7faf4e305700 1 -- 172.21.2.112:0/2358723908 wait complete. 2021-02-10T11:45:45.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:45.648 7faf4e305700 1 Processor -- start 2021-02-10T11:45:45.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:45.648 7faf4e305700 1 -- start start 2021-02-10T11:45:45.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:45.649 7faf4e305700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faf481b8ad0 0x7faf48187e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:45.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:45.649 7faf4e305700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7faf481b8ed0 0x7faf480b1af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:45.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:45.649 7faf4e305700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7faf480b2110 0x7faf481b7a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:45.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:45.649 7faf4e305700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf48188470 con 0x7faf481b8ad0 2021-02-10T11:45:45.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:45.649 7faf4e305700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf4818be10 con 0x7faf481b8ed0 2021-02-10T11:45:45.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:45.649 7faf4e305700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf4818c030 con 0x7faf480b2110 2021-02-10T11:45:45.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:45.649 7faf47fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faf481b8ad0 0x7faf48187e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:45.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:45.649 7faf47fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faf481b8ad0 0x7faf48187e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:59000/0 (socket says 172.21.2.112:59000) 2021-02-10T11:45:45.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:45.649 7faf47fff700 1 -- 172.21.2.112:0/2018282396 learned_addr learned my addr 172.21.2.112:0/2018282396 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:45:45.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:45.649 7faf47fff700 1 -- 172.21.2.112:0/2018282396 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7faf480b2110 msgr2=0x7faf481b7a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:45.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:45.649 7faf47fff700 1 --2- 172.21.2.112:0/2018282396 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7faf480b2110 0x7faf481b7a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:45.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:45.649 7faf47fff700 1 -- 172.21.2.112:0/2018282396 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7faf481b8ed0 msgr2=0x7faf480b1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:45.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:45.649 7faf47fff700 1 --2- 172.21.2.112:0/2018282396 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7faf481b8ed0 0x7faf480b1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:45.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:45.649 7faf47fff700 1 -- 172.21.2.112:0/2018282396 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf480b2510 con 0x7faf481b8ad0 2021-02-10T11:45:45.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:45.649 7faf47fff700 1 --2- 172.21.2.112:0/2018282396 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faf481b8ad0 0x7faf48187e50 secure :-1 s=READY pgs=2788 cs=0 l=1 rev1=1 rx=0x7faf38006f20 tx=0x7faf3800b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:45.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:45.650 7faf44ff9700 1 -- 172.21.2.112:0/2018282396 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faf3800bd80 con 0x7faf481b8ad0 2021-02-10T11:45:45.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:45.650 7faf44ff9700 1 -- 172.21.2.112:0/2018282396 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf38000ce0 con 0x7faf481b8ad0 2021-02-10T11:45:45.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:45.650 7faf44ff9700 1 -- 172.21.2.112:0/2018282396 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faf38001590 con 0x7faf481b8ad0 2021-02-10T11:45:45.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:45.650 7faf4e305700 1 -- 172.21.2.112:0/2018282396 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf480b2510 con 0x7faf481b8ad0 2021-02-10T11:45:45.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:45.650 7faf4e305700 1 -- 172.21.2.112:0/2018282396 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf480b2510 con 0x7faf481b8ad0 2021-02-10T11:45:45.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:45.650 7faf44ff9700 1 -- 172.21.2.112:0/2018282396 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7faf38001100 con 0x7faf481b8ad0 2021-02-10T11:45:45.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:45.651 7faf44ff9700 1 --2- 172.21.2.112:0/2018282396 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7faf280186c0 0x7faf2801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:45.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:45.651 7faf44ff9700 1 -- 172.21.2.112:0/2018282396 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 5573+0+0 (secure 0 0 0) 0x7faf38032820 con 0x7faf481b8ad0 2021-02-10T11:45:45.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:45.651 7faf477fe700 1 --2- 172.21.2.112:0/2018282396 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7faf280186c0 0x7faf2801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:45.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:45.652 7faf477fe700 1 --2- 172.21.2.112:0/2018282396 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7faf280186c0 0x7faf2801abc0 crc :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:45.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:45.682 7faf4e305700 1 -- 172.21.2.112:0/2018282396 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf480ebc00 con 0x7faf481b8ad0 2021-02-10T11:45:45.686 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:45.684 7faf44ff9700 1 -- 172.21.2.112:0/2018282396 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7faf38000c80 con 0x7faf481b8ad0 2021-02-10T11:45:45.810 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:45.809 7faf4e305700 1 -- 172.21.2.112:0/2018282396 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faf480ebc00 con 0x7faf481b8ad0 2021-02-10T11:45:45.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:45.809 7faf44ff9700 1 -- 172.21.2.112:0/2018282396 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v161) v1 ==== 74+0+14007 (secure 0 0 0) 0x7faf38000e90 con 0x7faf481b8ad0 2021-02-10T11:45:45.811 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:45:45.812 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":161,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:45:31.418524","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":35,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":34,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:45:28.386296","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"160","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":35,"pool_name":"cephfs_data","create_time":"2021-02-10 11:45:29.158381","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"161","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:45:45.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:45.811 7faf4e305700 1 -- 172.21.2.112:0/2018282396 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7faf280186c0 msgr2=0x7faf2801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:45.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:45.811 7faf4e305700 1 --2- 172.21.2.112:0/2018282396 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7faf280186c0 0x7faf2801abc0 crc :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:45.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:45.811 7faf4e305700 1 -- 172.21.2.112:0/2018282396 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faf481b8ad0 msgr2=0x7faf48187e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:45.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:45.811 7faf4e305700 1 --2- 172.21.2.112:0/2018282396 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faf481b8ad0 0x7faf48187e50 secure :-1 s=READY pgs=2788 cs=0 l=1 rev1=1 rx=0x7faf38006f20 tx=0x7faf3800b990).stop 2021-02-10T11:45:45.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:45.811 7faf4e305700 1 -- 172.21.2.112:0/2018282396 shutdown_connections 2021-02-10T11:45:45.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:45.811 7faf4e305700 1 --2- 172.21.2.112:0/2018282396 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7faf280186c0 0x7faf2801abc0 unknown :-1 s=CLOSED pgs=1270 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:45.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:45.811 7faf4e305700 1 --2- 172.21.2.112:0/2018282396 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7faf481b8ed0 0x7faf480b1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:45.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:45.811 7faf4e305700 1 --2- 172.21.2.112:0/2018282396 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7faf480b2110 0x7faf481b7a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:45.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:45.811 7faf4e305700 1 --2- 172.21.2.112:0/2018282396 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faf481b8ad0 0x7faf48187e50 unknown :-1 s=CLOSED pgs=2788 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:45.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:45.811 7faf4e305700 1 -- 172.21.2.112:0/2018282396 >> 172.21.2.112:0/2018282396 conn(0x7faf480ed810 msgr2=0x7faf48094980 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:45.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:45.811 7faf4e305700 1 -- 172.21.2.112:0/2018282396 shutdown_connections 2021-02-10T11:45:45.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:45.811 7faf4e305700 1 -- 172.21.2.112:0/2018282396 wait complete. 2021-02-10T11:45:45.821 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:13:57.508758', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 34, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:45:28.386296', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '160', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 35, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:45:29.158381', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '161', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:45:45.822 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:45:46.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.237 7f8291cbd700 1 Processor -- start 2021-02-10T11:45:46.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.237 7f8291cbd700 1 -- start start 2021-02-10T11:45:46.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.238 7f8291cbd700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f828c0d1870 0x7f828c1b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:46.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.238 7f8291cbd700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f828c1b0ae0 0x7f828c108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:46.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.238 7f8291cbd700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f828c0fe1c0 con 0x7f828c0d1c70 2021-02-10T11:45:46.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.238 7f8291cbd700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f828c109a60 con 0x7f828c0d1870 2021-02-10T11:45:46.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.238 7f8291cbd700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f828c109750 con 0x7f828c1b0ae0 2021-02-10T11:45:46.257 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.238 7f828b7fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f828c0d1870 0x7f828c1b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:46.257 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.238 7f828b7fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f828c0d1870 0x7f828c1b04c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:59008/0 (socket says 172.21.2.112:59008) 2021-02-10T11:45:46.257 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.238 7f828b7fe700 1 -- 172.21.2.112:0/3931384628 learned_addr learned my addr 172.21.2.112:0/3931384628 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:45:46.257 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.238 7f828b7fe700 1 -- 172.21.2.112:0/3931384628 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f828c1b0ae0 msgr2=0x7f828c108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:46.257 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.238 7f828b7fe700 1 --2- 172.21.2.112:0/3931384628 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f828c1b0ae0 0x7f828c108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:46.257 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.238 7f828b7fe700 1 -- 172.21.2.112:0/3931384628 >> v1:172.21.2.112:6789/0 conn(0x7f828c0d1c70 legacy=0x7f828c108f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:46.257 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.238 7f828b7fe700 1 -- 172.21.2.112:0/3931384628 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f828c088070 con 0x7f828c0d1870 2021-02-10T11:45:46.257 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.239 7f828b7fe700 1 --2- 172.21.2.112:0/3931384628 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f828c0d1870 0x7f828c1b04c0 secure :-1 s=READY pgs=2790 cs=0 l=1 rev1=1 rx=0x7f827c006700 tx=0x7f827c009b00).ready entity=mon.0 client_cookie=2555323163503ec1 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:46.257 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.239 7f828a7fc700 1 -- 172.21.2.112:0/3931384628 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f827c006340 con 0x7f828c0d1870 2021-02-10T11:45:46.257 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.239 7f828a7fc700 1 -- 172.21.2.112:0/3931384628 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f827c00c0c0 con 0x7f828c0d1870 2021-02-10T11:45:46.257 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.239 7f828a7fc700 1 -- 172.21.2.112:0/3931384628 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f827c00c5e0 con 0x7f828c0d1870 2021-02-10T11:45:46.257 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.239 7f8291cbd700 1 -- 172.21.2.112:0/3931384628 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f828c0d1870 msgr2=0x7f828c1b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:46.257 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.239 7f8291cbd700 1 --2- 172.21.2.112:0/3931384628 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f828c0d1870 0x7f828c1b04c0 secure :-1 s=READY pgs=2790 cs=0 l=1 rev1=1 rx=0x7f827c006700 tx=0x7f827c009b00).stop 2021-02-10T11:45:46.258 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.240 7f8291cbd700 1 -- 172.21.2.112:0/3931384628 shutdown_connections 2021-02-10T11:45:46.258 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.240 7f8291cbd700 1 --2- 172.21.2.112:0/3931384628 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f828c1b0ae0 0x7f828c108940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:46.258 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.240 7f8291cbd700 1 --2- 172.21.2.112:0/3931384628 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f828c0d1870 0x7f828c1b04c0 unknown :-1 s=CLOSED pgs=2790 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:46.258 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.240 7f8291cbd700 1 -- 172.21.2.112:0/3931384628 >> 172.21.2.112:0/3931384628 conn(0x7f828c0ce420 msgr2=0x7f828c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:46.258 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.240 7f8291cbd700 1 -- 172.21.2.112:0/3931384628 shutdown_connections 2021-02-10T11:45:46.258 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.240 7f8291cbd700 1 -- 172.21.2.112:0/3931384628 wait complete. 2021-02-10T11:45:46.258 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.254 7f8291cbd700 1 Processor -- start 2021-02-10T11:45:46.258 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.254 7f8291cbd700 1 -- start start 2021-02-10T11:45:46.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.255 7f8291cbd700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f828c0b9c90 0x7f828c09ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:46.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.255 7f8291cbd700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f828c09f320 0x7f828c1a74c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:46.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.255 7f8291cbd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f828c097250 0x7f828c1a7ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:46.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.255 7f8291cbd700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f828c09f890 con 0x7f828c097250 2021-02-10T11:45:46.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.255 7f8291cbd700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f828c0b9830 con 0x7f828c0b9c90 2021-02-10T11:45:46.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.255 7f8291cbd700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f828c1a8100 con 0x7f828c09f320 2021-02-10T11:45:46.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.255 7f828bfff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f828c097250 0x7f828c1a7ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:46.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.255 7f828bfff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f828c097250 0x7f828c1a7ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:59018/0 (socket says 172.21.2.112:59018) 2021-02-10T11:45:46.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.255 7f828bfff700 1 -- 172.21.2.112:0/383242124 learned_addr learned my addr 172.21.2.112:0/383242124 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:45:46.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.255 7f828bfff700 1 -- 172.21.2.112:0/383242124 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f828c09f320 msgr2=0x7f828c1a74c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:46.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.255 7f828bfff700 1 --2- 172.21.2.112:0/383242124 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f828c09f320 0x7f828c1a74c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:46.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.255 7f828bfff700 1 -- 172.21.2.112:0/383242124 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f828c0b9c90 msgr2=0x7f828c09ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:46.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.255 7f828bfff700 1 --2- 172.21.2.112:0/383242124 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f828c0b9c90 0x7f828c09ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:46.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.255 7f828bfff700 1 -- 172.21.2.112:0/383242124 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f828c09fd90 con 0x7f828c097250 2021-02-10T11:45:46.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.255 7f828bfff700 1 --2- 172.21.2.112:0/383242124 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f828c097250 0x7f828c1a7ae0 secure :-1 s=READY pgs=2791 cs=0 l=1 rev1=1 rx=0x7f8284009980 tx=0x7f828400bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:46.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.256 7f8273fff700 1 -- 172.21.2.112:0/383242124 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8284005a60 con 0x7f828c097250 2021-02-10T11:45:46.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.256 7f8273fff700 1 -- 172.21.2.112:0/383242124 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8284005ce0 con 0x7f828c097250 2021-02-10T11:45:46.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.256 7f8291cbd700 1 -- 172.21.2.112:0/383242124 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f828c09fd90 con 0x7f828c097250 2021-02-10T11:45:46.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.256 7f8273fff700 1 -- 172.21.2.112:0/383242124 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f82840062b0 con 0x7f828c097250 2021-02-10T11:45:46.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.256 7f8291cbd700 1 -- 172.21.2.112:0/383242124 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f828c09fd90 con 0x7f828c097250 2021-02-10T11:45:46.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.256 7f8273fff700 1 -- 172.21.2.112:0/383242124 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8284005870 con 0x7f828c097250 2021-02-10T11:45:46.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.256 7f8273fff700 1 --2- 172.21.2.112:0/383242124 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f826c018730 0x7f826c01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:46.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.256 7f8273fff700 1 -- 172.21.2.112:0/383242124 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f8284032a30 con 0x7f828c097250 2021-02-10T11:45:46.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.256 7f828b7fe700 1 --2- 172.21.2.112:0/383242124 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f826c018730 0x7f826c01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:46.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.257 7f828b7fe700 1 --2- 172.21.2.112:0/383242124 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f826c018730 0x7f826c01ac10 crc :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:46.289 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.287 7f8291cbd700 1 -- 172.21.2.112:0/383242124 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f828c082780 con 0x7f828c097250 2021-02-10T11:45:46.292 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.290 7f8273fff700 1 -- 172.21.2.112:0/383242124 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f828400be10 con 0x7f828c097250 2021-02-10T11:45:46.421 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.420 7f8291cbd700 1 -- 172.21.2.112:0/383242124 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f828c082780 con 0x7f828c097250 2021-02-10T11:45:46.422 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.420 7f8273fff700 1 -- 172.21.2.112:0/383242124 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v161) v1 ==== 92+0+10 (secure 0 0 0) 0x7f8284005ce0 con 0x7f828c097250 2021-02-10T11:45:46.422 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:45:46.424 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.422 7f8291cbd700 1 -- 172.21.2.112:0/383242124 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f826c018730 msgr2=0x7f826c01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:46.424 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.422 7f8291cbd700 1 --2- 172.21.2.112:0/383242124 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f826c018730 0x7f826c01ac10 crc :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:46.424 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.422 7f8291cbd700 1 -- 172.21.2.112:0/383242124 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f828c097250 msgr2=0x7f828c1a7ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:46.424 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.422 7f8291cbd700 1 --2- 172.21.2.112:0/383242124 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f828c097250 0x7f828c1a7ae0 secure :-1 s=READY pgs=2791 cs=0 l=1 rev1=1 rx=0x7f8284009980 tx=0x7f828400bcd0).stop 2021-02-10T11:45:46.424 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.422 7f8291cbd700 1 -- 172.21.2.112:0/383242124 shutdown_connections 2021-02-10T11:45:46.424 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.422 7f8291cbd700 1 --2- 172.21.2.112:0/383242124 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f826c018730 0x7f826c01ac10 unknown :-1 s=CLOSED pgs=1271 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:46.424 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.422 7f8291cbd700 1 --2- 172.21.2.112:0/383242124 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f828c097250 0x7f828c1a7ae0 unknown :-1 s=CLOSED pgs=2791 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:46.424 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.422 7f8291cbd700 1 --2- 172.21.2.112:0/383242124 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f828c09f320 0x7f828c1a74c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:46.424 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.422 7f8291cbd700 1 --2- 172.21.2.112:0/383242124 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f828c0b9c90 0x7f828c09ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:46.424 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.422 7f8291cbd700 1 -- 172.21.2.112:0/383242124 >> 172.21.2.112:0/383242124 conn(0x7f828c098430 msgr2=0x7f828c0830d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:46.425 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.422 7f8291cbd700 1 -- 172.21.2.112:0/383242124 shutdown_connections 2021-02-10T11:45:46.425 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.422 7f8291cbd700 1 -- 172.21.2.112:0/383242124 wait complete. 2021-02-10T11:45:46.432 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:45:46.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.846 7f2b150a4700 1 Processor -- start 2021-02-10T11:45:46.865 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.846 7f2b150a4700 1 -- start start 2021-02-10T11:45:46.865 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.846 7f2b150a4700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2b100d1870 0x7f2b101b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:46.865 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.846 7f2b150a4700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f2b101b0ae0 0x7f2b10108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:46.865 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.846 7f2b150a4700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b10109a60 con 0x7f2b100d1c70 2021-02-10T11:45:46.865 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.846 7f2b150a4700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b100e5ef0 con 0x7f2b101b0ae0 2021-02-10T11:45:46.865 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.846 7f2b150a4700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b10109750 con 0x7f2b100d1870 2021-02-10T11:45:46.865 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.847 7f2b0f59e700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7f2b100d1c70 0x7f2b10108f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:52044/0 (socket says 172.21.2.112:52044) 2021-02-10T11:45:46.865 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.847 7f2b0f59e700 1 -- 172.21.2.112:0/3907329818 learned_addr learned my addr 172.21.2.112:0/3907329818 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:45:46.865 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.847 7f2b0e59c700 1 --2- 172.21.2.112:0/3907329818 >> v2:172.21.2.120:3300/0 conn(0x7f2b101b0ae0 0x7f2b10108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:46.865 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.847 7f2b0ed9d700 1 --2- 172.21.2.112:0/3907329818 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2b100d1870 0x7f2b101b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:46.866 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.847 7f2b0ed9d700 1 -- 172.21.2.112:0/3907329818 >> v2:172.21.2.120:3300/0 conn(0x7f2b101b0ae0 msgr2=0x7f2b10108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:46.866 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.847 7f2b0ed9d700 1 --2- 172.21.2.112:0/3907329818 >> v2:172.21.2.120:3300/0 conn(0x7f2b101b0ae0 0x7f2b10108940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:46.866 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.847 7f2b0ed9d700 1 -- 172.21.2.112:0/3907329818 >> v1:172.21.2.120:6789/0 conn(0x7f2b100d1c70 legacy=0x7f2b10108f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:46.866 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.847 7f2b0ed9d700 1 -- 172.21.2.112:0/3907329818 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b10088070 con 0x7f2b100d1870 2021-02-10T11:45:46.866 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.847 7f2b0e59c700 1 --2- 172.21.2.112:0/3907329818 >> v2:172.21.2.120:3300/0 conn(0x7f2b101b0ae0 0x7f2b10108940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:45:46.866 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.847 7f2b0ed9d700 1 --2- 172.21.2.112:0/3907329818 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2b100d1870 0x7f2b101b04c0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 rx=0x7f2af8006650 tx=0x7f2af8009a80).ready entity=mon.2 client_cookie=38e3d5c60840dae3 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:46.866 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.848 7f2b0dd9b700 1 -- 172.21.2.112:0/3907329818 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2af80009b0 con 0x7f2b100d1870 2021-02-10T11:45:46.866 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.848 7f2b0dd9b700 1 -- 172.21.2.112:0/3907329818 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2af800c0c0 con 0x7f2b100d1870 2021-02-10T11:45:46.866 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.848 7f2b0dd9b700 1 -- 172.21.2.112:0/3907329818 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2af800c5e0 con 0x7f2b100d1870 2021-02-10T11:45:46.866 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.848 7f2b150a4700 1 -- 172.21.2.112:0/3907329818 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2b100d1870 msgr2=0x7f2b101b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:46.866 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.848 7f2b150a4700 1 --2- 172.21.2.112:0/3907329818 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2b100d1870 0x7f2b101b04c0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 rx=0x7f2af8006650 tx=0x7f2af8009a80).stop 2021-02-10T11:45:46.866 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.849 7f2b150a4700 1 -- 172.21.2.112:0/3907329818 shutdown_connections 2021-02-10T11:45:46.867 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.849 7f2b150a4700 1 --2- 172.21.2.112:0/3907329818 >> v2:172.21.2.120:3300/0 conn(0x7f2b101b0ae0 0x7f2b10108940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:46.867 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.849 7f2b150a4700 1 --2- 172.21.2.112:0/3907329818 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2b100d1870 0x7f2b101b04c0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:46.867 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.849 7f2b150a4700 1 -- 172.21.2.112:0/3907329818 >> 172.21.2.112:0/3907329818 conn(0x7f2b100ce420 msgr2=0x7f2b100d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:46.867 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.849 7f2b150a4700 1 -- 172.21.2.112:0/3907329818 shutdown_connections 2021-02-10T11:45:46.867 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.849 7f2b150a4700 1 -- 172.21.2.112:0/3907329818 wait complete. 2021-02-10T11:45:46.868 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.863 7f2b150a4700 1 Processor -- start 2021-02-10T11:45:46.868 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.863 7f2b150a4700 1 -- start start 2021-02-10T11:45:46.868 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.864 7f2b150a4700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2b100abd60 0x7f2b100ac160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:46.868 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.864 7f2b150a4700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2b100ac780 0x7f2b100acb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:46.868 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.864 7f2b150a4700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2b10083910 0x7f2b10083d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:46.868 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.864 7f2b150a4700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b10087180 con 0x7f2b100ac780 2021-02-10T11:45:46.868 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.864 7f2b150a4700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b1018fcd0 con 0x7f2b10083910 2021-02-10T11:45:46.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.864 7f2b150a4700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b10084330 con 0x7f2b100abd60 2021-02-10T11:45:46.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.864 7f2b0e59c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2b100ac780 0x7f2b100acb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:46.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.864 7f2b0f59e700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2b10083910 0x7f2b10083d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:46.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.864 7f2b0e59c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2b100ac780 0x7f2b100acb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:59030/0 (socket says 172.21.2.112:59030) 2021-02-10T11:45:46.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.864 7f2b0e59c700 1 -- 172.21.2.112:0/3785827465 learned_addr learned my addr 172.21.2.112:0/3785827465 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:45:46.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.864 7f2b0ed9d700 1 --2- 172.21.2.112:0/3785827465 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2b100abd60 0x7f2b100ac160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:46.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.864 7f2b0e59c700 1 -- 172.21.2.112:0/3785827465 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2b100abd60 msgr2=0x7f2b100ac160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:46.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.864 7f2b0e59c700 1 --2- 172.21.2.112:0/3785827465 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2b100abd60 0x7f2b100ac160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:46.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.864 7f2b0e59c700 1 -- 172.21.2.112:0/3785827465 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2b10083910 msgr2=0x7f2b10083d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:46.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.864 7f2b0e59c700 1 --2- 172.21.2.112:0/3785827465 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2b10083910 0x7f2b10083d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:46.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.864 7f2b0e59c700 1 -- 172.21.2.112:0/3785827465 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b10084550 con 0x7f2b100ac780 2021-02-10T11:45:46.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.864 7f2b0e59c700 1 --2- 172.21.2.112:0/3785827465 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2b100ac780 0x7f2b100acb80 secure :-1 s=READY pgs=2792 cs=0 l=1 rev1=1 rx=0x7f2b0000b580 tx=0x7f2b0000b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:46.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.866 7f2b077fe700 1 -- 172.21.2.112:0/3785827465 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2b0000bd10 con 0x7f2b100ac780 2021-02-10T11:45:46.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.866 7f2b077fe700 1 -- 172.21.2.112:0/3785827465 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b00000c90 con 0x7f2b100ac780 2021-02-10T11:45:46.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.866 7f2b077fe700 1 -- 172.21.2.112:0/3785827465 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2b00002220 con 0x7f2b100ac780 2021-02-10T11:45:46.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.866 7f2b150a4700 1 -- 172.21.2.112:0/3785827465 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b10084550 con 0x7f2b100ac780 2021-02-10T11:45:46.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.866 7f2b150a4700 1 -- 172.21.2.112:0/3785827465 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b10084ae0 con 0x7f2b100ac780 2021-02-10T11:45:46.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.867 7f2b077fe700 1 -- 172.21.2.112:0/3785827465 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2b00006b00 con 0x7f2b100ac780 2021-02-10T11:45:46.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.867 7f2b077fe700 1 --2- 172.21.2.112:0/3785827465 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2af40186c0 0x7f2af401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:46.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.867 7f2b077fe700 1 -- 172.21.2.112:0/3785827465 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f2b00032600 con 0x7f2b100ac780 2021-02-10T11:45:46.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.867 7f2b0ed9d700 1 --2- 172.21.2.112:0/3785827465 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2af40186c0 0x7f2af401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:46.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.868 7f2b0ed9d700 1 --2- 172.21.2.112:0/3785827465 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2af40186c0 0x7f2af401abc0 crc :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:46.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.896 7f2b150a4700 1 -- 172.21.2.112:0/3785827465 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b10082980 con 0x7f2b100ac780 2021-02-10T11:45:46.900 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:46.898 7f2b077fe700 1 -- 172.21.2.112:0/3785827465 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2b0000bd10 con 0x7f2b100ac780 2021-02-10T11:45:47.028 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:47.026 7f2b150a4700 1 -- 172.21.2.112:0/3785827465 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f2b10109a60 con 0x7f2b100ac780 2021-02-10T11:45:47.028 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:47.026 7f2b077fe700 1 -- 172.21.2.112:0/3785827465 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v161) v1 ==== 104+0+10 (secure 0 0 0) 0x7f2b00000c90 con 0x7f2b100ac780 2021-02-10T11:45:47.029 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:45:47.030 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:47.028 7f2b150a4700 1 -- 172.21.2.112:0/3785827465 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2af40186c0 msgr2=0x7f2af401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:47.031 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:47.028 7f2b150a4700 1 --2- 172.21.2.112:0/3785827465 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2af40186c0 0x7f2af401abc0 crc :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:47.031 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:47.028 7f2b150a4700 1 -- 172.21.2.112:0/3785827465 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2b100ac780 msgr2=0x7f2b100acb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:47.031 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:47.028 7f2b150a4700 1 --2- 172.21.2.112:0/3785827465 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2b100ac780 0x7f2b100acb80 secure :-1 s=READY pgs=2792 cs=0 l=1 rev1=1 rx=0x7f2b0000b580 tx=0x7f2b0000b920).stop 2021-02-10T11:45:47.031 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:47.028 7f2b150a4700 1 -- 172.21.2.112:0/3785827465 shutdown_connections 2021-02-10T11:45:47.031 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:47.028 7f2b150a4700 1 --2- 172.21.2.112:0/3785827465 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2af40186c0 0x7f2af401abc0 unknown :-1 s=CLOSED pgs=1272 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:47.031 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:47.028 7f2b150a4700 1 --2- 172.21.2.112:0/3785827465 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2b10083910 0x7f2b10083d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:47.031 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:47.028 7f2b150a4700 1 --2- 172.21.2.112:0/3785827465 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2b100ac780 0x7f2b100acb80 unknown :-1 s=CLOSED pgs=2792 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:47.031 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:47.028 7f2b150a4700 1 --2- 172.21.2.112:0/3785827465 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2b100abd60 0x7f2b100ac160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:47.031 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:47.028 7f2b150a4700 1 -- 172.21.2.112:0/3785827465 >> 172.21.2.112:0/3785827465 conn(0x7f2b100a2b30 msgr2=0x7f2b1009ce60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:47.031 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:47.029 7f2b150a4700 1 -- 172.21.2.112:0/3785827465 shutdown_connections 2021-02-10T11:45:47.031 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:47.029 7f2b150a4700 1 -- 172.21.2.112:0/3785827465 wait complete. 2021-02-10T11:45:47.039 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:45:47.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:47.452 7f56b94da700 1 Processor -- start 2021-02-10T11:45:47.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:47.452 7f56b94da700 1 -- start start 2021-02-10T11:45:47.473 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:47.453 7f56b94da700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f56b40d1870 0x7f56b41b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:47.473 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:47.453 7f56b94da700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f56b41b0ae0 0x7f56b4108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:47.473 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:47.453 7f56b94da700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56b4109a60 con 0x7f56b40d1c70 2021-02-10T11:45:47.473 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:47.453 7f56b94da700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56b40e5ef0 con 0x7f56b41b0ae0 2021-02-10T11:45:47.473 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:47.453 7f56b94da700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56b4109750 con 0x7f56b40d1870 2021-02-10T11:45:47.473 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:47.453 7f56b2abe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f56b41b0ae0 0x7f56b4108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:47.473 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:47.453 7f56b3ac0700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f56b40d1c70 0x7f56b4108f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:58498/0 (socket says 172.21.2.112:58498) 2021-02-10T11:45:47.473 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:47.453 7f56b3ac0700 1 -- 172.21.2.112:0/2685251942 learned_addr learned my addr 172.21.2.112:0/2685251942 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:45:47.473 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:47.453 7f56b32bf700 1 --2- 172.21.2.112:0/2685251942 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f56b40d1870 0x7f56b41b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:47.473 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:47.453 7f56b2abe700 1 -- 172.21.2.112:0/2685251942 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f56b40d1870 msgr2=0x7f56b41b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:47.473 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:47.453 7f56b2abe700 1 --2- 172.21.2.112:0/2685251942 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f56b40d1870 0x7f56b41b04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:47.473 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:47.453 7f56b2abe700 1 -- 172.21.2.112:0/2685251942 >> v1:172.21.2.112:6789/0 conn(0x7f56b40d1c70 legacy=0x7f56b4108f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:47.473 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:47.453 7f56b2abe700 1 -- 172.21.2.112:0/2685251942 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56b4088070 con 0x7f56b41b0ae0 2021-02-10T11:45:47.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:47.454 7f56b2abe700 1 --2- 172.21.2.112:0/2685251942 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f56b41b0ae0 0x7f56b4108940 secure :-1 s=READY pgs=2794 cs=0 l=1 rev1=1 rx=0x7f56a8006650 tx=0x7f56a8009a00).ready entity=mon.0 client_cookie=d8f81d8f2edc6907 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:47.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:47.454 7f56b22bd700 1 -- 172.21.2.112:0/2685251942 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f56a80009b0 con 0x7f56b41b0ae0 2021-02-10T11:45:47.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:47.454 7f56b22bd700 1 -- 172.21.2.112:0/2685251942 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56a800c0c0 con 0x7f56b41b0ae0 2021-02-10T11:45:47.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:47.454 7f56b22bd700 1 -- 172.21.2.112:0/2685251942 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f56a800c5a0 con 0x7f56b41b0ae0 2021-02-10T11:45:47.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:47.454 7f56b94da700 1 -- 172.21.2.112:0/2685251942 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f56b41b0ae0 msgr2=0x7f56b4108940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:47.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:47.454 7f56b94da700 1 --2- 172.21.2.112:0/2685251942 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f56b41b0ae0 0x7f56b4108940 secure :-1 s=READY pgs=2794 cs=0 l=1 rev1=1 rx=0x7f56a8006650 tx=0x7f56a8009a00).stop 2021-02-10T11:45:47.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:47.455 7f56b94da700 1 -- 172.21.2.112:0/2685251942 shutdown_connections 2021-02-10T11:45:47.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:47.455 7f56b94da700 1 --2- 172.21.2.112:0/2685251942 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f56b41b0ae0 0x7f56b4108940 unknown :-1 s=CLOSED pgs=2794 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:47.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:47.455 7f56b94da700 1 --2- 172.21.2.112:0/2685251942 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f56b40d1870 0x7f56b41b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:47.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:47.455 7f56b94da700 1 -- 172.21.2.112:0/2685251942 >> 172.21.2.112:0/2685251942 conn(0x7f56b40ce420 msgr2=0x7f56b40d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:47.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:47.455 7f56b94da700 1 -- 172.21.2.112:0/2685251942 shutdown_connections 2021-02-10T11:45:47.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:47.455 7f56b94da700 1 -- 172.21.2.112:0/2685251942 wait complete. 2021-02-10T11:45:47.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:47.469 7f56b94da700 1 Processor -- start 2021-02-10T11:45:47.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:47.470 7f56b94da700 1 -- start start 2021-02-10T11:45:47.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:47.470 7f56b94da700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f56b40a2580 0x7f56b40a1cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:47.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:47.470 7f56b94da700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f56b4094c40 0x7f56b41a75e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:47.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:47.470 7f56b94da700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f56b4095040 0x7f56b41a7c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:47.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:47.470 7f56b94da700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56b4188180 con 0x7f56b4094c40 2021-02-10T11:45:47.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:47.470 7f56b94da700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56b4187e70 con 0x7f56b40a2580 2021-02-10T11:45:47.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:47.470 7f56b94da700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56b41a8220 con 0x7f56b4095040 2021-02-10T11:45:47.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:47.470 7f56b2abe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f56b4094c40 0x7f56b41a75e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:47.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:47.470 7f56b2abe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f56b4094c40 0x7f56b41a75e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:59044/0 (socket says 172.21.2.112:59044) 2021-02-10T11:45:47.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:47.470 7f56b2abe700 1 -- 172.21.2.112:0/1165939121 learned_addr learned my addr 172.21.2.112:0/1165939121 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:45:47.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:47.471 7f56b2abe700 1 -- 172.21.2.112:0/1165939121 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f56b4095040 msgr2=0x7f56b41a7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:47.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:47.471 7f56b2abe700 1 --2- 172.21.2.112:0/1165939121 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f56b4095040 0x7f56b41a7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:47.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:47.471 7f56b2abe700 1 -- 172.21.2.112:0/1165939121 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f56b40a2580 msgr2=0x7f56b40a1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:47.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:47.471 7f56b2abe700 1 --2- 172.21.2.112:0/1165939121 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f56b40a2580 0x7f56b40a1cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:47.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:47.471 7f56b2abe700 1 -- 172.21.2.112:0/1165939121 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56b4188700 con 0x7f56b4094c40 2021-02-10T11:45:47.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:47.471 7f56b2abe700 1 --2- 172.21.2.112:0/1165939121 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f56b4094c40 0x7f56b41a75e0 secure :-1 s=READY pgs=2795 cs=0 l=1 rev1=1 rx=0x7f56a800b640 tx=0x7f56a800b9e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:47.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:47.471 7f569bfff700 1 -- 172.21.2.112:0/1165939121 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f56a800bdd0 con 0x7f56b4094c40 2021-02-10T11:45:47.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:47.471 7f569bfff700 1 -- 172.21.2.112:0/1165939121 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56a8002030 con 0x7f56b4094c40 2021-02-10T11:45:47.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:47.471 7f569bfff700 1 -- 172.21.2.112:0/1165939121 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f56a8002550 con 0x7f56b4094c40 2021-02-10T11:45:47.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:47.471 7f56b94da700 1 -- 172.21.2.112:0/1165939121 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56b4188700 con 0x7f56b4094c40 2021-02-10T11:45:47.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:47.471 7f56b94da700 1 -- 172.21.2.112:0/1165939121 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56b4188c90 con 0x7f56b4094c40 2021-02-10T11:45:47.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:47.472 7f569bfff700 1 -- 172.21.2.112:0/1165939121 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f56a8006a10 con 0x7f56b4094c40 2021-02-10T11:45:47.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:47.472 7f569bfff700 1 --2- 172.21.2.112:0/1165939121 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f56940186c0 0x7f569401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:47.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:47.472 7f569bfff700 1 -- 172.21.2.112:0/1165939121 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f56a80325c0 con 0x7f56b4094c40 2021-02-10T11:45:47.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:47.472 7f56b32bf700 1 --2- 172.21.2.112:0/1165939121 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f56940186c0 0x7f569401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:47.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:47.473 7f56b32bf700 1 --2- 172.21.2.112:0/1165939121 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f56940186c0 0x7f569401abc0 crc :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:47.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:47.503 7f56b94da700 1 -- 172.21.2.112:0/1165939121 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56b4082980 con 0x7f56b4094c40 2021-02-10T11:45:47.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:47.505 7f569bfff700 1 -- 172.21.2.112:0/1165939121 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f56a80009b0 con 0x7f56b4094c40 2021-02-10T11:45:47.636 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:47.634 7f56b94da700 1 -- 172.21.2.112:0/1165939121 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f56b4109a60 con 0x7f56b4094c40 2021-02-10T11:45:47.636 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:47.635 7f569bfff700 1 -- 172.21.2.112:0/1165939121 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v161) v1 ==== 100+0+10 (secure 0 0 0) 0x7f56a800bdd0 con 0x7f56b4094c40 2021-02-10T11:45:47.637 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:45:47.639 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:47.636 7f56b94da700 1 -- 172.21.2.112:0/1165939121 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f56940186c0 msgr2=0x7f569401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:47.639 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:47.636 7f56b94da700 1 --2- 172.21.2.112:0/1165939121 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f56940186c0 0x7f569401abc0 crc :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:47.639 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:47.636 7f56b94da700 1 -- 172.21.2.112:0/1165939121 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f56b4094c40 msgr2=0x7f56b41a75e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:47.639 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:47.636 7f56b94da700 1 --2- 172.21.2.112:0/1165939121 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f56b4094c40 0x7f56b41a75e0 secure :-1 s=READY pgs=2795 cs=0 l=1 rev1=1 rx=0x7f56a800b640 tx=0x7f56a800b9e0).stop 2021-02-10T11:45:47.639 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:47.636 7f56b94da700 1 -- 172.21.2.112:0/1165939121 shutdown_connections 2021-02-10T11:45:47.639 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:47.636 7f56b94da700 1 --2- 172.21.2.112:0/1165939121 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f56940186c0 0x7f569401abc0 unknown :-1 s=CLOSED pgs=1273 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:47.639 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:47.636 7f56b94da700 1 --2- 172.21.2.112:0/1165939121 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f56b4095040 0x7f56b41a7c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:47.639 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:47.636 7f56b94da700 1 --2- 172.21.2.112:0/1165939121 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f56b4094c40 0x7f56b41a75e0 unknown :-1 s=CLOSED pgs=2795 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:47.639 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:47.636 7f56b94da700 1 --2- 172.21.2.112:0/1165939121 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f56b40a2580 0x7f56b40a1cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:47.639 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:47.636 7f56b94da700 1 -- 172.21.2.112:0/1165939121 >> 172.21.2.112:0/1165939121 conn(0x7f56b40a2b30 msgr2=0x7f56b409ce60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:47.640 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:47.637 7f56b94da700 1 -- 172.21.2.112:0/1165939121 shutdown_connections 2021-02-10T11:45:47.640 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:47.637 7f56b94da700 1 -- 172.21.2.112:0/1165939121 wait complete. 2021-02-10T11:45:47.647 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:45:47.647 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:45:48.076 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:48.058 7fd030f76700 1 Processor -- start 2021-02-10T11:45:48.077 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:48.058 7fd030f76700 1 -- start start 2021-02-10T11:45:48.077 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:48.058 7fd030f76700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd02c0d0340 0x7fd02c0ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:48.077 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:48.058 7fd030f76700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fd02c0d1c70 0x7fd02c0efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:48.077 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:48.058 7fd030f76700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd02c0f0b10 con 0x7fd02c0d1870 2021-02-10T11:45:48.077 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:48.058 7fd030f76700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd02c0e5ed0 con 0x7fd02c0d1c70 2021-02-10T11:45:48.077 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:48.058 7fd030f76700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd02c1b0be0 con 0x7fd02c0d0340 2021-02-10T11:45:48.077 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:48.058 7fd02ab13700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fd02c0d1870 0x7fd02c1b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:58512/0 (socket says 172.21.2.112:58512) 2021-02-10T11:45:48.077 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:48.058 7fd02b314700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fd02c0d1c70 0x7fd02c0efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:48.077 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:48.058 7fd02ab13700 1 -- 172.21.2.112:0/398737336 learned_addr learned my addr 172.21.2.112:0/398737336 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:45:48.077 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:48.058 7fd02a312700 1 --2- 172.21.2.112:0/398737336 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd02c0d0340 0x7fd02c0ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:48.077 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:48.059 7fd029b11700 1 -- 172.21.2.112:0/398737336 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3806440594 0 0) 0x7fd01c000ce0 con 0x7fd02c0d1870 2021-02-10T11:45:48.077 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:48.059 7fd029b11700 1 -- 172.21.2.112:0/398737336 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd018002fe0 con 0x7fd02c0d1870 2021-02-10T11:45:48.078 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:48.059 7fd02b314700 1 -- 172.21.2.112:0/398737336 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd02c0d0340 msgr2=0x7fd02c0ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:48.078 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:48.059 7fd02b314700 1 --2- 172.21.2.112:0/398737336 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd02c0d0340 0x7fd02c0ef640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:48.078 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:48.059 7fd02b314700 1 -- 172.21.2.112:0/398737336 >> v1:172.21.2.112:6789/0 conn(0x7fd02c0d1870 legacy=0x7fd02c1b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:48.078 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:48.059 7fd02b314700 1 -- 172.21.2.112:0/398737336 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd02c0f20a0 con 0x7fd02c0d1c70 2021-02-10T11:45:48.078 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:48.059 7fd02a312700 1 --2- 172.21.2.112:0/398737336 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd02c0d0340 0x7fd02c0ef640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:45:48.078 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:48.059 7fd02b314700 1 --2- 172.21.2.112:0/398737336 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd02c0d1c70 0x7fd02c0efc60 secure :-1 s=READY pgs=2797 cs=0 l=1 rev1=1 rx=0x7fd024006e30 tx=0x7fd024009e60).ready entity=mon.0 client_cookie=7355780d93255af3 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:48.078 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:48.059 7fd029b11700 1 -- 172.21.2.112:0/398737336 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd024006a50 con 0x7fd02c0d1c70 2021-02-10T11:45:48.078 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:48.059 7fd029b11700 1 -- 172.21.2.112:0/398737336 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd02400b300 con 0x7fd02c0d1c70 2021-02-10T11:45:48.078 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:48.060 7fd029b11700 1 -- 172.21.2.112:0/398737336 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd02400b820 con 0x7fd02c0d1c70 2021-02-10T11:45:48.078 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:48.060 7fd030f76700 1 -- 172.21.2.112:0/398737336 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd02c0d1c70 msgr2=0x7fd02c0efc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:48.078 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:48.060 7fd030f76700 1 --2- 172.21.2.112:0/398737336 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd02c0d1c70 0x7fd02c0efc60 secure :-1 s=READY pgs=2797 cs=0 l=1 rev1=1 rx=0x7fd024006e30 tx=0x7fd024009e60).stop 2021-02-10T11:45:48.078 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:48.060 7fd030f76700 1 -- 172.21.2.112:0/398737336 shutdown_connections 2021-02-10T11:45:48.079 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:48.060 7fd030f76700 1 --2- 172.21.2.112:0/398737336 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd02c0d1c70 0x7fd02c0efc60 unknown :-1 s=CLOSED pgs=2797 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:48.079 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:48.060 7fd030f76700 1 --2- 172.21.2.112:0/398737336 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd02c0d0340 0x7fd02c0ef640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:48.079 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:48.060 7fd030f76700 1 -- 172.21.2.112:0/398737336 >> 172.21.2.112:0/398737336 conn(0x7fd02c0ce450 msgr2=0x7fd02c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:48.079 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:48.060 7fd030f76700 1 -- 172.21.2.112:0/398737336 shutdown_connections 2021-02-10T11:45:48.079 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:48.060 7fd030f76700 1 -- 172.21.2.112:0/398737336 wait complete. 2021-02-10T11:45:48.080 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:48.075 7fd030f76700 1 Processor -- start 2021-02-10T11:45:48.081 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:48.075 7fd030f76700 1 -- start start 2021-02-10T11:45:48.081 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:48.075 7fd030f76700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd02c195650 0x7fd02c0b5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:48.081 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:48.075 7fd030f76700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd02c0b5b80 0x7fd02c1a2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:48.081 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:48.075 7fd030f76700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd02c0acfa0 0x7fd02c1a3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:48.081 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:48.076 7fd030f76700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd02c0b6010 con 0x7fd02c195650 2021-02-10T11:45:48.081 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:48.076 7fd030f76700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd02c1a3a60 con 0x7fd02c0b5b80 2021-02-10T11:45:48.081 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:48.076 7fd030f76700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd02c1a3c40 con 0x7fd02c0acfa0 2021-02-10T11:45:48.081 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:48.076 7fd02ab13700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd02c195650 0x7fd02c0b5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:48.081 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:48.076 7fd02ab13700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd02c195650 0x7fd02c0b5560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:59060/0 (socket says 172.21.2.112:59060) 2021-02-10T11:45:48.081 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:48.076 7fd02ab13700 1 -- 172.21.2.112:0/1419475342 learned_addr learned my addr 172.21.2.112:0/1419475342 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:45:48.081 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:48.076 7fd02b314700 1 --2- 172.21.2.112:0/1419475342 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd02c0acfa0 0x7fd02c1a3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:48.081 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:48.076 7fd02a312700 1 --2- 172.21.2.112:0/1419475342 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd02c0b5b80 0x7fd02c1a2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:48.082 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:48.076 7fd02ab13700 1 -- 172.21.2.112:0/1419475342 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd02c0acfa0 msgr2=0x7fd02c1a3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:48.082 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:48.076 7fd02ab13700 1 --2- 172.21.2.112:0/1419475342 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd02c0acfa0 0x7fd02c1a3440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:48.082 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:48.076 7fd02ab13700 1 -- 172.21.2.112:0/1419475342 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd02c0b5b80 msgr2=0x7fd02c1a2e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:48.082 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:48.076 7fd02ab13700 1 --2- 172.21.2.112:0/1419475342 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd02c0b5b80 0x7fd02c1a2e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:48.082 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:48.076 7fd02ab13700 1 -- 172.21.2.112:0/1419475342 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd02c0bcae0 con 0x7fd02c195650 2021-02-10T11:45:48.082 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:48.076 7fd02ab13700 1 --2- 172.21.2.112:0/1419475342 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd02c195650 0x7fd02c0b5560 secure :-1 s=READY pgs=2798 cs=0 l=1 rev1=1 rx=0x7fd01c001490 tx=0x7fd01c00a950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:48.082 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:48.077 7fd0237fe700 1 -- 172.21.2.112:0/1419475342 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd01c00ad40 con 0x7fd02c195650 2021-02-10T11:45:48.082 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:48.077 7fd0237fe700 1 -- 172.21.2.112:0/1419475342 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd01c000ce0 con 0x7fd02c195650 2021-02-10T11:45:48.082 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:48.077 7fd0237fe700 1 -- 172.21.2.112:0/1419475342 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd01c004590 con 0x7fd02c195650 2021-02-10T11:45:48.082 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:48.077 7fd030f76700 1 -- 172.21.2.112:0/1419475342 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd02c0bcae0 con 0x7fd02c195650 2021-02-10T11:45:48.082 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:48.077 7fd030f76700 1 -- 172.21.2.112:0/1419475342 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd02c0bcae0 con 0x7fd02c195650 2021-02-10T11:45:48.082 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:48.077 7fd0237fe700 1 -- 172.21.2.112:0/1419475342 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd01c0043a0 con 0x7fd02c195650 2021-02-10T11:45:48.082 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:48.077 7fd0237fe700 1 --2- 172.21.2.112:0/1419475342 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd0100186c0 0x7fd01001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:48.083 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:48.077 7fd0237fe700 1 -- 172.21.2.112:0/1419475342 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fd01c032540 con 0x7fd02c195650 2021-02-10T11:45:48.083 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:48.078 7fd02a312700 1 --2- 172.21.2.112:0/1419475342 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd0100186c0 0x7fd01001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:48.083 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:48.078 7fd02a312700 1 --2- 172.21.2.112:0/1419475342 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd0100186c0 0x7fd01001abc0 crc :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:48.110 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:48.108 7fd030f76700 1 -- 172.21.2.112:0/1419475342 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd02c0b6010 con 0x7fd02c195650 2021-02-10T11:45:48.112 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:48.111 7fd0237fe700 1 -- 172.21.2.112:0/1419475342 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd01c007050 con 0x7fd02c195650 2021-02-10T11:45:48.240 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:48.238 7fd030f76700 1 -- 172.21.2.112:0/1419475342 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd02c0b6010 con 0x7fd02c195650 2021-02-10T11:45:48.241 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:48.239 7fd0237fe700 1 -- 172.21.2.112:0/1419475342 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 139 v139) v1 ==== 95+0+2105 (secure 0 0 0) 0x7fd01c00ad40 con 0x7fd02c195650 2021-02-10T11:45:48.241 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:45:48.241 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":139,"default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":12306,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/2635651262","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":2635651262},{"type":"v1","addr":"172.21.2.112:6833","nonce":2635651262}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":135}],"filesystems":[{"mdsmap":{"epoch":139,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:45:30.416714","modified":"2021-02-10 11:45:32.783510","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8920},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8920":{"gid":8920,"name":"b","rank":0,"incarnation":136,"state":"up:active","state_seq":3,"addr":"172.21.2.120:6835/555399099","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":555399099},{"type":"v1","addr":"172.21.2.120:6835","nonce":555399099}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[35],"metadata_pool":34,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":17}]} 2021-02-10T11:45:48.241 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 139 2021-02-10T11:45:48.242 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:48.241 7fd030f76700 1 -- 172.21.2.112:0/1419475342 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd0100186c0 msgr2=0x7fd01001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:48.242 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:48.241 7fd030f76700 1 --2- 172.21.2.112:0/1419475342 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd0100186c0 0x7fd01001abc0 crc :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:48.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:48.241 7fd030f76700 1 -- 172.21.2.112:0/1419475342 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd02c195650 msgr2=0x7fd02c0b5560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:48.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:48.241 7fd030f76700 1 --2- 172.21.2.112:0/1419475342 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd02c195650 0x7fd02c0b5560 secure :-1 s=READY pgs=2798 cs=0 l=1 rev1=1 rx=0x7fd01c001490 tx=0x7fd01c00a950).stop 2021-02-10T11:45:48.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:48.241 7fd030f76700 1 -- 172.21.2.112:0/1419475342 shutdown_connections 2021-02-10T11:45:48.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:48.241 7fd030f76700 1 --2- 172.21.2.112:0/1419475342 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd0100186c0 0x7fd01001abc0 unknown :-1 s=CLOSED pgs=1274 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:48.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:48.241 7fd030f76700 1 --2- 172.21.2.112:0/1419475342 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd02c0b5b80 0x7fd02c1a2e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:48.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:48.241 7fd030f76700 1 --2- 172.21.2.112:0/1419475342 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd02c0acfa0 0x7fd02c1a3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:48.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:48.241 7fd030f76700 1 --2- 172.21.2.112:0/1419475342 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd02c195650 0x7fd02c0b5560 unknown :-1 s=CLOSED pgs=2798 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:48.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:48.241 7fd030f76700 1 -- 172.21.2.112:0/1419475342 >> 172.21.2.112:0/1419475342 conn(0x7fd02c0b3040 msgr2=0x7fd02c0c3250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:48.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:48.241 7fd030f76700 1 -- 172.21.2.112:0/1419475342 shutdown_connections 2021-02-10T11:45:48.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:48.241 7fd030f76700 1 -- 172.21.2.112:0/1419475342 wait complete. 2021-02-10T11:45:48.452 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 139, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:45:30.416714', 'modified': '2021-02-10 11:45:32.783510', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8920}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8920': {'gid': 8920, 'name': 'b', 'rank': 0, 'incarnation': 136, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.120:6835/555399099', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.120:6834', 'nonce': 555399099}, {'type': 'v1', 'addr': '172.21.2.120:6835', 'nonce': 555399099}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [35], 'metadata_pool': 34, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:45:48.453 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:45:48.453 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:45:48.850 INFO:teuthology.orchestra.run.gibba020.stdout:{ 2021-02-10T11:45:48.851 INFO:teuthology.orchestra.run.gibba020.stdout: "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", 2021-02-10T11:45:48.851 INFO:teuthology.orchestra.run.gibba020.stdout: "whoami": 0, 2021-02-10T11:45:48.851 INFO:teuthology.orchestra.run.gibba020.stdout: "id": 8920, 2021-02-10T11:45:48.851 INFO:teuthology.orchestra.run.gibba020.stdout: "want_state": "up:active", 2021-02-10T11:45:48.851 INFO:teuthology.orchestra.run.gibba020.stdout: "state": "up:active", 2021-02-10T11:45:48.851 INFO:teuthology.orchestra.run.gibba020.stdout: "rank_uptime": 18.430014009000001, 2021-02-10T11:45:48.851 INFO:teuthology.orchestra.run.gibba020.stdout: "mdsmap_epoch": 139, 2021-02-10T11:45:48.851 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch": 161, 2021-02-10T11:45:48.851 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch_barrier": 161, 2021-02-10T11:45:48.851 INFO:teuthology.orchestra.run.gibba020.stdout: "uptime": 24.082335164 2021-02-10T11:45:48.851 INFO:teuthology.orchestra.run.gibba020.stdout:} 2021-02-10T11:45:48.854 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", "whoami": 0, "id": 8920, "want_state": "up:active", "state": "up:active", "rank_uptime": 18.430014009000001, "mdsmap_epoch": 139, "osdmap_epoch": 161, "osdmap_epoch_barrier": 161, "uptime": 24.082335164 } 2021-02-10T11:45:48.855 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:45:48.855 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2021-02-10T11:45:48.855 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.1 at ubuntu@gibba012.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2021-02-10T11:45:48.855 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:45:48.855 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:45:49.193 DEBUG:teuthology.orchestra.run.gibba012:> sudo modprobe fuse 2021-02-10T11:45:49.537 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:45:49.537 DEBUG:teuthology.orchestra.run.gibba012:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:45:49.882 INFO:teuthology.orchestra.run.gibba012.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:45:49.882 INFO:teuthology.orchestra.run.gibba012.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:45:49.883 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:45:49.883 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:45:49.883 DEBUG:teuthology.orchestra.run.gibba012:> ls /sys/fs/fuse/connections 2021-02-10T11:45:50.219 INFO:teuthology.orchestra.run.gibba012.stdout:42 2021-02-10T11:45:50.219 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [42] 2021-02-10T11:45:50.219 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.1 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:45:50.222 DEBUG:teuthology.orchestra.run.gibba012:> sudo modprobe fuse 2021-02-10T11:45:50.568 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:45:50.568 DEBUG:teuthology.orchestra.run.gibba012:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:45:50.634 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba012.stderr:ceph-fuse[180732]: starting ceph client 2021-02-10T11:45:50.634 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba012.stderr:2021-02-10 11:45:50.632 7fefd8a13f80 -1 init, newargv = 0x563a6cb451f0 newargc=9 2021-02-10T11:45:50.640 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba012.stderr:ceph-fuse[180732]: starting fuse 2021-02-10T11:45:50.910 INFO:teuthology.orchestra.run.gibba012.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:45:50.910 INFO:teuthology.orchestra.run.gibba012.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:45:50.911 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:45:50.911 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:45:50.912 DEBUG:teuthology.orchestra.run.gibba012:> ls /sys/fs/fuse/connections 2021-02-10T11:45:51.247 INFO:teuthology.orchestra.run.gibba012.stdout:42 2021-02-10T11:45:51.247 INFO:teuthology.orchestra.run.gibba012.stdout:43 2021-02-10T11:45:51.247 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42, 43] 2021-02-10T11:45:51.247 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:45:51.248 DEBUG:teuthology.orchestra.run.gibba012:> sudo python3 -c ' 2021-02-10T11:45:51.248 DEBUG:teuthology.orchestra.run.gibba012:> import glob 2021-02-10T11:45:51.248 DEBUG:teuthology.orchestra.run.gibba012:> import re 2021-02-10T11:45:51.248 DEBUG:teuthology.orchestra.run.gibba012:> import os 2021-02-10T11:45:51.248 DEBUG:teuthology.orchestra.run.gibba012:> import subprocess 2021-02-10T11:45:51.248 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:45:51.248 DEBUG:teuthology.orchestra.run.gibba012:> def find_socket(client_name): 2021-02-10T11:45:51.248 DEBUG:teuthology.orchestra.run.gibba012:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2021-02-10T11:45:51.248 DEBUG:teuthology.orchestra.run.gibba012:> files = glob.glob(asok_path) 2021-02-10T11:45:51.248 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:45:51.248 DEBUG:teuthology.orchestra.run.gibba012:> # Given a non-glob path, it better be there 2021-02-10T11:45:51.248 DEBUG:teuthology.orchestra.run.gibba012:> if "*" not in asok_path: 2021-02-10T11:45:51.248 DEBUG:teuthology.orchestra.run.gibba012:> assert(len(files) == 1) 2021-02-10T11:45:51.249 DEBUG:teuthology.orchestra.run.gibba012:> return files[0] 2021-02-10T11:45:51.249 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:45:51.249 DEBUG:teuthology.orchestra.run.gibba012:> for f in files: 2021-02-10T11:45:51.249 DEBUG:teuthology.orchestra.run.gibba012:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:45:51.249 DEBUG:teuthology.orchestra.run.gibba012:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:45:51.249 DEBUG:teuthology.orchestra.run.gibba012:> return f 2021-02-10T11:45:51.249 DEBUG:teuthology.orchestra.run.gibba012:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:45:51.249 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:45:51.249 DEBUG:teuthology.orchestra.run.gibba012:> print(find_socket("client.1")) 2021-02-10T11:45:51.249 DEBUG:teuthology.orchestra.run.gibba012:> ' 2021-02-10T11:45:51.611 INFO:teuthology.orchestra.run.gibba012.stdout:/var/run/ceph/ceph-client.1.180732.asok 2021-02-10T11:45:51.614 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.180732.asok 2021-02-10T11:45:51.614 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:45:51.614 DEBUG:teuthology.orchestra.run.gibba012:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.180732.asok status 2021-02-10T11:45:52.004 INFO:teuthology.orchestra.run.gibba012.stdout:{ 2021-02-10T11:45:52.005 INFO:teuthology.orchestra.run.gibba012.stdout: "metadata": { 2021-02-10T11:45:52.005 INFO:teuthology.orchestra.run.gibba012.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:45:52.005 INFO:teuthology.orchestra.run.gibba012.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:45:52.005 INFO:teuthology.orchestra.run.gibba012.stdout: "entity_id": "1", 2021-02-10T11:45:52.005 INFO:teuthology.orchestra.run.gibba012.stdout: "hostname": "gibba012", 2021-02-10T11:45:52.005 INFO:teuthology.orchestra.run.gibba012.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2021-02-10T11:45:52.005 INFO:teuthology.orchestra.run.gibba012.stdout: "pid": "180732", 2021-02-10T11:45:52.005 INFO:teuthology.orchestra.run.gibba012.stdout: "root": "/" 2021-02-10T11:45:52.005 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:45:52.005 INFO:teuthology.orchestra.run.gibba012.stdout: "dentry_count": 0, 2021-02-10T11:45:52.005 INFO:teuthology.orchestra.run.gibba012.stdout: "dentry_pinned_count": 0, 2021-02-10T11:45:52.005 INFO:teuthology.orchestra.run.gibba012.stdout: "id": 12501, 2021-02-10T11:45:52.005 INFO:teuthology.orchestra.run.gibba012.stdout: "inst": { 2021-02-10T11:45:52.006 INFO:teuthology.orchestra.run.gibba012.stdout: "name": { 2021-02-10T11:45:52.006 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "client", 2021-02-10T11:45:52.006 INFO:teuthology.orchestra.run.gibba012.stdout: "num": 12501 2021-02-10T11:45:52.006 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:45:52.006 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": { 2021-02-10T11:45:52.006 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:45:52.006 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:0", 2021-02-10T11:45:52.006 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 2922630648 2021-02-10T11:45:52.006 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:45:52.006 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:45:52.006 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": { 2021-02-10T11:45:52.006 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:45:52.006 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:0", 2021-02-10T11:45:52.007 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 2922630648 2021-02-10T11:45:52.007 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:45:52.007 INFO:teuthology.orchestra.run.gibba012.stdout: "inst_str": "client.12501 172.21.2.112:0/2922630648", 2021-02-10T11:45:52.007 INFO:teuthology.orchestra.run.gibba012.stdout: "addr_str": "172.21.2.112:0/2922630648", 2021-02-10T11:45:52.007 INFO:teuthology.orchestra.run.gibba012.stdout: "inode_count": 1, 2021-02-10T11:45:52.007 INFO:teuthology.orchestra.run.gibba012.stdout: "mds_epoch": 139, 2021-02-10T11:45:52.007 INFO:teuthology.orchestra.run.gibba012.stdout: "osd_epoch": 161, 2021-02-10T11:45:52.007 INFO:teuthology.orchestra.run.gibba012.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:45:52.007 INFO:teuthology.orchestra.run.gibba012.stdout: "blacklisted": false 2021-02-10T11:45:52.007 INFO:teuthology.orchestra.run.gibba012.stdout:} 2021-02-10T11:45:52.008 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:45:52.009 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:45:52.009 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:45:52.344 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:45:52.344 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:45:52.345 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:45:52.345 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:45:52.691 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2021-02-10T11:45:53.124 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.105 7f0ab7062700 1 Processor -- start 2021-02-10T11:45:53.124 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.105 7f0ab7062700 1 -- start start 2021-02-10T11:45:53.124 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.106 7f0ab7062700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0ab00d1880 0x7f0ab01b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:53.124 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.106 7f0ab7062700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f0ab00cff80 0x7f0ab0109140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:53.124 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.106 7f0ab7062700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ab0109a70 con 0x7f0ab01b0af0 2021-02-10T11:45:53.124 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.106 7f0ab7062700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ab0109760 con 0x7f0ab00cff80 2021-02-10T11:45:53.124 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.106 7f0ab7062700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ab0088070 con 0x7f0ab00d1880 2021-02-10T11:45:53.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.106 7f0aaffff700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f0ab01b0af0 0x7f0ab0108950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:58538/0 (socket says 172.21.2.112:58538) 2021-02-10T11:45:53.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.106 7f0aaffff700 1 -- 172.21.2.112:0/2784580681 learned_addr learned my addr 172.21.2.112:0/2784580681 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:45:53.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.106 7f0ab567f700 1 --2- 172.21.2.112:0/2784580681 >> v2:172.21.2.112:3300/0 conn(0x7f0ab00cff80 0x7f0ab0109140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:53.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.106 7f0ab4e7e700 1 --2- 172.21.2.112:0/2784580681 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0ab00d1880 0x7f0ab01b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:53.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.106 7f0ab567f700 1 -- 172.21.2.112:0/2784580681 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0ab00d1880 msgr2=0x7f0ab01b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:53.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.106 7f0ab567f700 1 --2- 172.21.2.112:0/2784580681 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0ab00d1880 0x7f0ab01b04d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:53.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.106 7f0ab567f700 1 -- 172.21.2.112:0/2784580681 >> v1:172.21.2.112:6789/0 conn(0x7f0ab01b0af0 legacy=0x7f0ab0108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:53.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.106 7f0aaf7fe700 1 -- 172.21.2.112:0/2784580681 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1277891125 0 0) 0x7f0aa0001060 con 0x7f0ab01b0af0 2021-02-10T11:45:53.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.106 7f0ab567f700 1 -- 172.21.2.112:0/2784580681 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ab0088260 con 0x7f0ab00cff80 2021-02-10T11:45:53.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.107 7f0ab567f700 1 --2- 172.21.2.112:0/2784580681 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0ab00cff80 0x7f0ab0109140 secure :-1 s=READY pgs=2802 cs=0 l=1 rev1=1 rx=0x7f0aa8006ab0 tx=0x7f0aa8009b40).ready entity=mon.0 client_cookie=6c0576fe0579d4e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:53.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.107 7f0aaf7fe700 1 -- 172.21.2.112:0/2784580681 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0aa80022e0 con 0x7f0ab00cff80 2021-02-10T11:45:53.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.107 7f0aaf7fe700 1 -- 172.21.2.112:0/2784580681 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0aa8002560 con 0x7f0ab00cff80 2021-02-10T11:45:53.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.107 7f0aaf7fe700 1 -- 172.21.2.112:0/2784580681 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0aa80029f0 con 0x7f0ab00cff80 2021-02-10T11:45:53.126 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.107 7f0ab7062700 1 -- 172.21.2.112:0/2784580681 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0ab00cff80 msgr2=0x7f0ab0109140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:53.126 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.107 7f0ab7062700 1 --2- 172.21.2.112:0/2784580681 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0ab00cff80 0x7f0ab0109140 secure :-1 s=READY pgs=2802 cs=0 l=1 rev1=1 rx=0x7f0aa8006ab0 tx=0x7f0aa8009b40).stop 2021-02-10T11:45:53.126 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.108 7f0ab7062700 1 -- 172.21.2.112:0/2784580681 shutdown_connections 2021-02-10T11:45:53.126 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.108 7f0ab7062700 1 --2- 172.21.2.112:0/2784580681 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0ab00cff80 0x7f0ab0109140 unknown :-1 s=CLOSED pgs=2802 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:53.126 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.108 7f0ab7062700 1 --2- 172.21.2.112:0/2784580681 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0ab00d1880 0x7f0ab01b04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:53.126 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.108 7f0ab7062700 1 -- 172.21.2.112:0/2784580681 >> 172.21.2.112:0/2784580681 conn(0x7f0ab00ce430 msgr2=0x7f0ab00d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:53.126 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.108 7f0ab7062700 1 -- 172.21.2.112:0/2784580681 shutdown_connections 2021-02-10T11:45:53.126 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.108 7f0ab7062700 1 -- 172.21.2.112:0/2784580681 wait complete. 2021-02-10T11:45:53.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.122 7f0ab7062700 1 Processor -- start 2021-02-10T11:45:53.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.123 7f0ab7062700 1 -- start start 2021-02-10T11:45:53.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.123 7f0ab7062700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0ab00b1f10 0x7f0ab00b2310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:53.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.123 7f0ab7062700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0ab0083160 0x7f0ab0083560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:53.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.123 7f0ab7062700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0ab0083b80 0x7f0ab0083f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:53.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.123 7f0ab7062700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ab009a820 con 0x7f0ab0083b80 2021-02-10T11:45:53.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.123 7f0ab7062700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ab009acc0 con 0x7f0ab0083160 2021-02-10T11:45:53.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.123 7f0ab7062700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ab01af2d0 con 0x7f0ab00b1f10 2021-02-10T11:45:53.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.123 7f0ab567f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0ab0083b80 0x7f0ab0083f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:53.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.123 7f0ab567f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0ab0083b80 0x7f0ab0083f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:59086/0 (socket says 172.21.2.112:59086) 2021-02-10T11:45:53.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.123 7f0ab567f700 1 -- 172.21.2.112:0/2863479387 learned_addr learned my addr 172.21.2.112:0/2863479387 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:45:53.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.123 7f0ab4e7e700 1 --2- 172.21.2.112:0/2863479387 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0ab00b1f10 0x7f0ab00b2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:53.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.123 7f0aaffff700 1 --2- 172.21.2.112:0/2863479387 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0ab0083160 0x7f0ab0083560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:53.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.124 7f0ab567f700 1 -- 172.21.2.112:0/2863479387 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0ab00b1f10 msgr2=0x7f0ab00b2310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:53.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.124 7f0ab567f700 1 --2- 172.21.2.112:0/2863479387 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0ab00b1f10 0x7f0ab00b2310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:53.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.124 7f0ab567f700 1 -- 172.21.2.112:0/2863479387 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0ab0083160 msgr2=0x7f0ab0083560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:53.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.124 7f0ab567f700 1 --2- 172.21.2.112:0/2863479387 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0ab0083160 0x7f0ab0083560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:53.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.124 7f0ab567f700 1 -- 172.21.2.112:0/2863479387 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ab01af4f0 con 0x7f0ab0083b80 2021-02-10T11:45:53.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.124 7f0ab567f700 1 --2- 172.21.2.112:0/2863479387 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0ab0083b80 0x7f0ab0083f80 secure :-1 s=READY pgs=2803 cs=0 l=1 rev1=1 rx=0x7f0aa800b5e0 tx=0x7f0aa800b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:53.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.124 7f0aad7fa700 1 -- 172.21.2.112:0/2863479387 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0aa8009070 con 0x7f0ab0083b80 2021-02-10T11:45:53.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.124 7f0aad7fa700 1 -- 172.21.2.112:0/2863479387 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0aa8002280 con 0x7f0ab0083b80 2021-02-10T11:45:53.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.124 7f0aad7fa700 1 -- 172.21.2.112:0/2863479387 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0aa8002760 con 0x7f0ab0083b80 2021-02-10T11:45:53.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.124 7f0ab7062700 1 -- 172.21.2.112:0/2863479387 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ab01af4f0 con 0x7f0ab0083b80 2021-02-10T11:45:53.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.124 7f0ab7062700 1 -- 172.21.2.112:0/2863479387 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ab01af4f0 con 0x7f0ab0083b80 2021-02-10T11:45:53.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.125 7f0aad7fa700 1 -- 172.21.2.112:0/2863479387 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0aa8002460 con 0x7f0ab0083b80 2021-02-10T11:45:53.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.125 7f0aad7fa700 1 --2- 172.21.2.112:0/2863479387 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0a940186c0 0x7f0a9401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:53.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.125 7f0aad7fa700 1 -- 172.21.2.112:0/2863479387 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f0aa8032820 con 0x7f0ab0083b80 2021-02-10T11:45:53.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.125 7f0ab4e7e700 1 --2- 172.21.2.112:0/2863479387 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0a940186c0 0x7f0a9401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:53.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.126 7f0ab4e7e700 1 --2- 172.21.2.112:0/2863479387 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0a940186c0 0x7f0a9401abc0 crc :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:53.157 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.156 7f0ab7062700 1 -- 172.21.2.112:0/2863479387 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ab0109a70 con 0x7f0ab0083b80 2021-02-10T11:45:53.160 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.158 7f0aad7fa700 1 -- 172.21.2.112:0/2863479387 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0aa8002070 con 0x7f0ab0083b80 2021-02-10T11:45:53.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.293 7f0ab7062700 1 -- 172.21.2.112:0/2863479387 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f0ab0109a70 con 0x7f0ab0083b80 2021-02-10T11:45:53.349 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.347 7f0aad7fa700 1 -- 172.21.2.112:0/2863479387 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v140) v1 ==== 136+0+0 (secure 0 0 0) 0x7f0aa8000ef0 con 0x7f0ab0083b80 2021-02-10T11:45:53.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.359 7f0ab7062700 1 -- 172.21.2.112:0/2863479387 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0a940186c0 msgr2=0x7f0a9401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:53.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.359 7f0ab7062700 1 --2- 172.21.2.112:0/2863479387 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0a940186c0 0x7f0a9401abc0 crc :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:53.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.359 7f0ab7062700 1 -- 172.21.2.112:0/2863479387 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0ab0083b80 msgr2=0x7f0ab0083f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:53.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.359 7f0ab7062700 1 --2- 172.21.2.112:0/2863479387 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0ab0083b80 0x7f0ab0083f80 secure :-1 s=READY pgs=2803 cs=0 l=1 rev1=1 rx=0x7f0aa800b5e0 tx=0x7f0aa800b980).stop 2021-02-10T11:45:53.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.359 7f0ab7062700 1 -- 172.21.2.112:0/2863479387 shutdown_connections 2021-02-10T11:45:53.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.359 7f0ab7062700 1 --2- 172.21.2.112:0/2863479387 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0a940186c0 0x7f0a9401abc0 unknown :-1 s=CLOSED pgs=1275 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:53.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.359 7f0ab7062700 1 --2- 172.21.2.112:0/2863479387 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0ab0083160 0x7f0ab0083560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:53.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.359 7f0ab7062700 1 --2- 172.21.2.112:0/2863479387 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0ab0083b80 0x7f0ab0083f80 unknown :-1 s=CLOSED pgs=2803 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:53.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.359 7f0ab7062700 1 --2- 172.21.2.112:0/2863479387 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0ab00b1f10 0x7f0ab00b2310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:53.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.359 7f0ab7062700 1 -- 172.21.2.112:0/2863479387 >> 172.21.2.112:0/2863479387 conn(0x7f0ab0087390 msgr2=0x7f0ab0090c90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:53.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.359 7f0ab7062700 1 -- 172.21.2.112:0/2863479387 shutdown_connections 2021-02-10T11:45:53.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.360 7f0ab7062700 1 -- 172.21.2.112:0/2863479387 wait complete. 2021-02-10T11:45:53.369 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs volume ls 2021-02-10T11:45:53.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.777 7f6798c1b700 1 Processor -- start 2021-02-10T11:45:53.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.777 7f6798c1b700 1 -- start start 2021-02-10T11:45:53.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.778 7f6798c1b700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f67940d1870 0x7f67941b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:53.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.778 7f6798c1b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f67940f0650 0x7f67940cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:53.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.778 7f6798c1b700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67940f0b10 con 0x7f67941b0a10 2021-02-10T11:45:53.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.778 7f6798c1b700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67940e5ed0 con 0x7f67940d1870 2021-02-10T11:45:53.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.778 7f6798c1b700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67940d1c70 con 0x7f67940f0650 2021-02-10T11:45:53.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.778 7f679205d700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f67941b0a10 0x7f67940cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:58554/0 (socket says 172.21.2.112:58554) 2021-02-10T11:45:53.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.778 7f679205d700 1 -- 172.21.2.112:0/58135632 learned_addr learned my addr 172.21.2.112:0/58135632 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:45:53.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.778 7f679185c700 1 -- 172.21.2.112:0/58135632 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1573740522 0 0) 0x7f67840010b0 con 0x7f67941b0a10 2021-02-10T11:45:53.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.779 7f679185c700 1 -- 172.21.2.112:0/58135632 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6780002fe0 con 0x7f67941b0a10 2021-02-10T11:45:53.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.779 7f679285e700 1 --2- 172.21.2.112:0/58135632 >> v2:172.21.2.120:3300/0 conn(0x7f67940d1870 0x7f67941b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:53.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.779 7f679305f700 1 --2- 172.21.2.112:0/58135632 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f67940f0650 0x7f67940cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:53.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.779 7f679185c700 1 -- 172.21.2.112:0/58135632 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 564950673 0 0) 0x7f67840010b0 con 0x7f67941b0a10 2021-02-10T11:45:53.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.779 7f679185c700 1 -- 172.21.2.112:0/58135632 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f67940f0650 msgr2=0x7f67940cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:53.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.779 7f679185c700 1 --2- 172.21.2.112:0/58135632 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f67940f0650 0x7f67940cfb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:53.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.779 7f679185c700 1 -- 172.21.2.112:0/58135632 >> v2:172.21.2.120:3300/0 conn(0x7f67940d1870 msgr2=0x7f67941b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:53.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.779 7f679185c700 1 --2- 172.21.2.112:0/58135632 >> v2:172.21.2.120:3300/0 conn(0x7f67940d1870 0x7f67941b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:53.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.779 7f679185c700 1 -- 172.21.2.112:0/58135632 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67940d1e90 con 0x7f67941b0a10 2021-02-10T11:45:53.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.779 7f679185c700 1 -- 172.21.2.112:0/58135632 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f67840015b0 con 0x7f67941b0a10 2021-02-10T11:45:53.800 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.779 7f679185c700 1 -- 172.21.2.112:0/58135632 >> v1:172.21.2.112:6789/0 conn(0x7f67941b0a10 legacy=0x7f67940cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:53.800 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.779 7f679185c700 1 --2- 172.21.2.112:0/58135632 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6780003b10 0x7f6780006000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:53.800 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.779 7f679185c700 1 --2- 172.21.2.112:0/58135632 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6780006820 0x7f6780008d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:53.800 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.779 7f679185c700 1 --2- 172.21.2.112:0/58135632 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f678004d5d0 0x7f678004fad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:53.800 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.779 7f679185c700 1 -- 172.21.2.112:0/58135632 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6780050380 con 0x7f678004d5d0 2021-02-10T11:45:53.800 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.779 7f679185c700 1 -- 172.21.2.112:0/58135632 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67800505a0 con 0x7f6780006820 2021-02-10T11:45:53.800 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.779 7f679185c700 1 -- 172.21.2.112:0/58135632 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67800507c0 con 0x7f6780003b10 2021-02-10T11:45:53.800 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.779 7f679285e700 1 --2- 172.21.2.112:0/58135632 >> v2:172.21.2.120:3300/0 conn(0x7f67940d1870 0x7f67941b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:45:53.801 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.779 7f679305f700 1 --2- 172.21.2.112:0/58135632 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f678004d5d0 0x7f678004fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:53.801 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.779 7f679205d700 1 --2- 172.21.2.112:0/58135632 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6780006820 0x7f6780008d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:53.801 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.779 7f679305f700 1 -- 172.21.2.112:0/58135632 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6780003b10 msgr2=0x7f6780006000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:53.801 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.779 7f679305f700 1 --2- 172.21.2.112:0/58135632 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6780003b10 0x7f6780006000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:53.801 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.779 7f679305f700 1 -- 172.21.2.112:0/58135632 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6780006820 msgr2=0x7f6780008d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:53.801 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.779 7f679305f700 1 --2- 172.21.2.112:0/58135632 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6780006820 0x7f6780008d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:53.802 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.779 7f679305f700 1 -- 172.21.2.112:0/58135632 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6780050a60 con 0x7f678004d5d0 2021-02-10T11:45:53.802 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.780 7f679205d700 1 --2- 172.21.2.112:0/58135632 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6780006820 0x7f6780008d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:45:53.802 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.780 7f679305f700 1 --2- 172.21.2.112:0/58135632 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f678004d5d0 0x7f678004fad0 secure :-1 s=READY pgs=2805 cs=0 l=1 rev1=1 rx=0x7f678c006fc0 tx=0x7f678c009e70).ready entity=mon.0 client_cookie=ab9bbe444d6488df server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:53.802 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.780 7f679185c700 1 -- 172.21.2.112:0/58135632 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f678c006a50 con 0x7f678004d5d0 2021-02-10T11:45:53.802 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.780 7f679185c700 1 -- 172.21.2.112:0/58135632 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f678c00b300 con 0x7f678004d5d0 2021-02-10T11:45:53.803 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.780 7f679185c700 1 -- 172.21.2.112:0/58135632 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f678c00b7a0 con 0x7f678004d5d0 2021-02-10T11:45:53.803 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.780 7f6798c1b700 1 -- 172.21.2.112:0/58135632 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f678004d5d0 msgr2=0x7f678004fad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:53.803 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.780 7f6798c1b700 1 --2- 172.21.2.112:0/58135632 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f678004d5d0 0x7f678004fad0 secure :-1 s=READY pgs=2805 cs=0 l=1 rev1=1 rx=0x7f678c006fc0 tx=0x7f678c009e70).stop 2021-02-10T11:45:53.803 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.780 7f679305f700 1 -- 172.21.2.112:0/58135632 reap_dead start 2021-02-10T11:45:53.803 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.780 7f6798c1b700 1 -- 172.21.2.112:0/58135632 shutdown_connections 2021-02-10T11:45:53.803 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.780 7f6798c1b700 1 -- 172.21.2.112:0/58135632 >> 172.21.2.112:0/58135632 conn(0x7f67940ce450 msgr2=0x7f67940d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:53.804 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.780 7f6798c1b700 1 -- 172.21.2.112:0/58135632 shutdown_connections 2021-02-10T11:45:53.804 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.780 7f6798c1b700 1 -- 172.21.2.112:0/58135632 wait complete. 2021-02-10T11:45:53.804 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.795 7f6798c1b700 1 Processor -- start 2021-02-10T11:45:53.804 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.795 7f6798c1b700 1 -- start start 2021-02-10T11:45:53.804 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.796 7f6798c1b700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f679408e8f0 0x7f679408ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:53.804 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.796 7f6798c1b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f67940ebe80 0x7f67940ec280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:53.804 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.796 7f6798c1b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f67940ec8a0 0x7f67940ed650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:53.805 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.796 7f6798c1b700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67940a0a50 con 0x7f67940ebe80 2021-02-10T11:45:53.805 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.796 7f6798c1b700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f679408f310 con 0x7f679408e8f0 2021-02-10T11:45:53.805 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.796 7f6798c1b700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f679408f530 con 0x7f67940ec8a0 2021-02-10T11:45:53.805 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.796 7f679205d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f67940ebe80 0x7f67940ec280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:53.805 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.796 7f679205d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f67940ebe80 0x7f67940ec280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:59104/0 (socket says 172.21.2.112:59104) 2021-02-10T11:45:53.805 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.796 7f679205d700 1 -- 172.21.2.112:0/2416998639 learned_addr learned my addr 172.21.2.112:0/2416998639 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:45:53.808 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.796 7f679285e700 1 --2- 172.21.2.112:0/2416998639 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f679408e8f0 0x7f679408ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:53.808 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.796 7f679205d700 1 -- 172.21.2.112:0/2416998639 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f67940ec8a0 msgr2=0x7f67940ed650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:53.808 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.796 7f679205d700 1 --2- 172.21.2.112:0/2416998639 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f67940ec8a0 0x7f67940ed650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:53.808 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.796 7f679205d700 1 -- 172.21.2.112:0/2416998639 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f679408e8f0 msgr2=0x7f679408ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:53.808 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.796 7f679205d700 1 --2- 172.21.2.112:0/2416998639 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f679408e8f0 0x7f679408ecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:53.808 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.796 7f679205d700 1 -- 172.21.2.112:0/2416998639 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67940ecca0 con 0x7f67940ebe80 2021-02-10T11:45:53.809 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.796 7f679285e700 1 --2- 172.21.2.112:0/2416998639 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f679408e8f0 0x7f679408ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:45:53.809 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.796 7f679205d700 1 --2- 172.21.2.112:0/2416998639 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f67940ebe80 0x7f67940ec280 secure :-1 s=READY pgs=2806 cs=0 l=1 rev1=1 rx=0x7f6784001c10 tx=0x7f678400a8f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:53.809 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.796 7f678b7fe700 1 -- 172.21.2.112:0/2416998639 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f67840012c0 con 0x7f67940ebe80 2021-02-10T11:45:53.809 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.796 7f678b7fe700 1 -- 172.21.2.112:0/2416998639 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6784000c90 con 0x7f67940ebe80 2021-02-10T11:45:53.809 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.796 7f678b7fe700 1 -- 172.21.2.112:0/2416998639 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6784004350 con 0x7f67940ebe80 2021-02-10T11:45:53.809 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.796 7f6798c1b700 1 -- 172.21.2.112:0/2416998639 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67940ecca0 con 0x7f67940ebe80 2021-02-10T11:45:53.809 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.796 7f6798c1b700 1 -- 172.21.2.112:0/2416998639 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67940ecca0 con 0x7f67940ebe80 2021-02-10T11:45:53.809 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.797 7f678b7fe700 1 -- 172.21.2.112:0/2416998639 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6784004140 con 0x7f67940ebe80 2021-02-10T11:45:53.809 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.797 7f678b7fe700 1 --2- 172.21.2.112:0/2416998639 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f67780186c0 0x7f677801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:53.809 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.797 7f678b7fe700 1 -- 172.21.2.112:0/2416998639 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f6784032400 con 0x7f67940ebe80 2021-02-10T11:45:53.810 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.797 7f679285e700 1 --2- 172.21.2.112:0/2416998639 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f67780186c0 0x7f677801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:53.810 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.798 7f679285e700 1 --2- 172.21.2.112:0/2416998639 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f67780186c0 0x7f677801abc0 crc :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:53.830 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.828 7f6798c1b700 1 -- 172.21.2.112:0/2416998639 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67940ebc50 con 0x7f67940ebe80 2021-02-10T11:45:53.832 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.830 7f678b7fe700 1 -- 172.21.2.112:0/2416998639 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6784021dc0 con 0x7f67940ebe80 2021-02-10T11:45:53.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.965 7f6798c1b700 1 -- 172.21.2.112:0/2416998639 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs volume ls", "target": ["mgr", ""]}) v1 -- 0x7f67940ebc50 con 0x7f67780186c0 2021-02-10T11:45:53.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.966 7f678b7fe700 1 -- 172.21.2.112:0/2416998639 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+41 (crc 0 0 0) 0x7f677c000940 con 0x7f67780186c0 2021-02-10T11:45:53.970 INFO:teuthology.orchestra.run.gibba012.stdout:[ 2021-02-10T11:45:53.970 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:45:53.970 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "cephfs" 2021-02-10T11:45:53.970 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:45:53.970 INFO:teuthology.orchestra.run.gibba012.stdout:] 2021-02-10T11:45:53.972 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.970 7f6798c1b700 1 -- 172.21.2.112:0/2416998639 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f67780186c0 msgr2=0x7f677801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:53.972 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.970 7f6798c1b700 1 --2- 172.21.2.112:0/2416998639 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f67780186c0 0x7f677801abc0 crc :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:53.972 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.970 7f6798c1b700 1 -- 172.21.2.112:0/2416998639 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f67940ebe80 msgr2=0x7f67940ec280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:53.972 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.970 7f6798c1b700 1 --2- 172.21.2.112:0/2416998639 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f67940ebe80 0x7f67940ec280 secure :-1 s=READY pgs=2806 cs=0 l=1 rev1=1 rx=0x7f6784001c10 tx=0x7f678400a8f0).stop 2021-02-10T11:45:53.972 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.970 7f6798c1b700 1 -- 172.21.2.112:0/2416998639 shutdown_connections 2021-02-10T11:45:53.972 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.970 7f6798c1b700 1 --2- 172.21.2.112:0/2416998639 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f67780186c0 0x7f677801abc0 unknown :-1 s=CLOSED pgs=1276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:53.972 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.970 7f6798c1b700 1 --2- 172.21.2.112:0/2416998639 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f67940ec8a0 0x7f67940ed650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:53.972 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.970 7f6798c1b700 1 --2- 172.21.2.112:0/2416998639 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f67940ebe80 0x7f67940ec280 unknown :-1 s=CLOSED pgs=2806 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:53.972 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.970 7f6798c1b700 1 --2- 172.21.2.112:0/2416998639 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f679408e8f0 0x7f679408ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:53.973 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.970 7f6798c1b700 1 -- 172.21.2.112:0/2416998639 >> 172.21.2.112:0/2416998639 conn(0x7f6794094d90 msgr2=0x7f679408a420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:53.973 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.970 7f6798c1b700 1 -- 172.21.2.112:0/2416998639 shutdown_connections 2021-02-10T11:45:53.973 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:53.970 7f6798c1b700 1 -- 172.21.2.112:0/2416998639 wait complete. 2021-02-10T11:45:53.980 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:45:54.411 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:54.393 7f662ef79700 1 Processor -- start 2021-02-10T11:45:54.411 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:54.393 7f662ef79700 1 -- start start 2021-02-10T11:45:54.411 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:54.393 7f662ef79700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f66280d1860 0x7f66281b04b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:54.411 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:54.393 7f662ef79700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f66281b0ad0 0x7f6628108930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:54.412 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:54.393 7f662ef79700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66280fe1b0 con 0x7f66280d1c60 2021-02-10T11:45:54.412 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:54.393 7f662ef79700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6628109a50 con 0x7f66281b0ad0 2021-02-10T11:45:54.412 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:54.393 7f662ef79700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6628109740 con 0x7f66280d1860 2021-02-10T11:45:54.412 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:54.393 7f662d596700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f66280d1c60 0x7f6628108f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:58576/0 (socket says 172.21.2.112:58576) 2021-02-10T11:45:54.412 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:54.393 7f662d596700 1 -- 172.21.2.112:0/1421622910 learned_addr learned my addr 172.21.2.112:0/1421622910 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:45:54.412 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:54.393 7f6627fff700 1 --2- 172.21.2.112:0/1421622910 >> v2:172.21.2.112:3300/0 conn(0x7f66281b0ad0 0x7f6628108930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:54.412 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:54.394 7f662cd95700 1 --2- 172.21.2.112:0/1421622910 >> v2:172.21.2.120:3300/0 conn(0x7f66280d1860 0x7f66281b04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:54.412 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:54.394 7f66277fe700 1 -- 172.21.2.112:0/1421622910 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 945187286 0 0) 0x7f6620001500 con 0x7f66280d1c60 2021-02-10T11:45:54.412 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:54.394 7f66277fe700 1 -- 172.21.2.112:0/1421622910 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6614002fe0 con 0x7f66280d1c60 2021-02-10T11:45:54.412 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:54.394 7f6627fff700 1 -- 172.21.2.112:0/1421622910 >> v2:172.21.2.120:3300/0 conn(0x7f66280d1860 msgr2=0x7f66281b04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:54.412 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:54.394 7f6627fff700 1 --2- 172.21.2.112:0/1421622910 >> v2:172.21.2.120:3300/0 conn(0x7f66280d1860 0x7f66281b04b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:54.412 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:54.394 7f6627fff700 1 -- 172.21.2.112:0/1421622910 >> v1:172.21.2.112:6789/0 conn(0x7f66280d1c60 legacy=0x7f6628108f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:54.413 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:54.394 7f6627fff700 1 -- 172.21.2.112:0/1421622910 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6628088070 con 0x7f66281b0ad0 2021-02-10T11:45:54.413 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:54.394 7f6627fff700 1 --2- 172.21.2.112:0/1421622910 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f66281b0ad0 0x7f6628108930 secure :-1 s=READY pgs=2808 cs=0 l=1 rev1=1 rx=0x7f6618006650 tx=0x7f6618009a80).ready entity=mon.0 client_cookie=6f00a2a9696d7b9a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:54.413 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:54.394 7f66277fe700 1 -- 172.21.2.112:0/1421622910 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f66180009b0 con 0x7f66281b0ad0 2021-02-10T11:45:54.413 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:54.394 7f66277fe700 1 -- 172.21.2.112:0/1421622910 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f661800c0c0 con 0x7f66281b0ad0 2021-02-10T11:45:54.413 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:54.394 7f66277fe700 1 -- 172.21.2.112:0/1421622910 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f661800c5e0 con 0x7f66281b0ad0 2021-02-10T11:45:54.413 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:54.394 7f662ef79700 1 -- 172.21.2.112:0/1421622910 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f66281b0ad0 msgr2=0x7f6628108930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:54.413 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:54.394 7f662ef79700 1 --2- 172.21.2.112:0/1421622910 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f66281b0ad0 0x7f6628108930 secure :-1 s=READY pgs=2808 cs=0 l=1 rev1=1 rx=0x7f6618006650 tx=0x7f6618009a80).stop 2021-02-10T11:45:54.413 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:54.395 7f662ef79700 1 -- 172.21.2.112:0/1421622910 shutdown_connections 2021-02-10T11:45:54.413 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:54.395 7f662ef79700 1 --2- 172.21.2.112:0/1421622910 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f66281b0ad0 0x7f6628108930 unknown :-1 s=CLOSED pgs=2808 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:54.413 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:54.395 7f662ef79700 1 --2- 172.21.2.112:0/1421622910 >> v2:172.21.2.120:3300/0 conn(0x7f66280d1860 0x7f66281b04b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:54.413 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:54.395 7f662ef79700 1 -- 172.21.2.112:0/1421622910 >> 172.21.2.112:0/1421622910 conn(0x7f66280ce410 msgr2=0x7f66280d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:54.413 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:54.395 7f662ef79700 1 -- 172.21.2.112:0/1421622910 shutdown_connections 2021-02-10T11:45:54.413 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:54.395 7f662ef79700 1 -- 172.21.2.112:0/1421622910 wait complete. 2021-02-10T11:45:54.415 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:54.410 7f662ef79700 1 Processor -- start 2021-02-10T11:45:54.415 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:54.410 7f662ef79700 1 -- start start 2021-02-10T11:45:54.415 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:54.410 7f662ef79700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6628085270 0x7f6628085670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:54.415 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:54.410 7f662ef79700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6628085c90 0x7f6628086090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:54.415 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:54.410 7f662ef79700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f66280c2740 0x7f66280c2b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:54.415 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:54.410 7f662ef79700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6628084cf0 con 0x7f6628085270 2021-02-10T11:45:54.415 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:54.410 7f662ef79700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66280be1b0 con 0x7f66280c2740 2021-02-10T11:45:54.415 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:54.410 7f662ef79700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66280be390 con 0x7f6628085c90 2021-02-10T11:45:54.415 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:54.410 7f662cd95700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6628085270 0x7f6628085670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:54.416 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:54.410 7f662cd95700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6628085270 0x7f6628085670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:59116/0 (socket says 172.21.2.112:59116) 2021-02-10T11:45:54.416 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:54.410 7f662cd95700 1 -- 172.21.2.112:0/798684000 learned_addr learned my addr 172.21.2.112:0/798684000 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:45:54.416 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:54.411 7f662cd95700 1 -- 172.21.2.112:0/798684000 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6628085c90 msgr2=0x7f6628086090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:54.416 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:54.411 7f662cd95700 1 --2- 172.21.2.112:0/798684000 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6628085c90 0x7f6628086090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:54.416 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:54.411 7f662cd95700 1 -- 172.21.2.112:0/798684000 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f66280c2740 msgr2=0x7f66280c2b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:54.416 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:54.411 7f662cd95700 1 --2- 172.21.2.112:0/798684000 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f66280c2740 0x7f66280c2b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:54.416 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:54.411 7f662cd95700 1 -- 172.21.2.112:0/798684000 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66280c3160 con 0x7f6628085270 2021-02-10T11:45:54.416 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:54.411 7f662cd95700 1 --2- 172.21.2.112:0/798684000 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6628085270 0x7f6628085670 secure :-1 s=READY pgs=2809 cs=0 l=1 rev1=1 rx=0x7f6610006f00 tx=0x7f661000b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:54.416 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:54.411 7f66257fa700 1 -- 172.21.2.112:0/798684000 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f661000bd30 con 0x7f6628085270 2021-02-10T11:45:54.416 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:54.411 7f66257fa700 1 -- 172.21.2.112:0/798684000 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6610002030 con 0x7f6628085270 2021-02-10T11:45:54.416 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:54.411 7f66257fa700 1 -- 172.21.2.112:0/798684000 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6610002550 con 0x7f6628085270 2021-02-10T11:45:54.416 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:54.411 7f662ef79700 1 -- 172.21.2.112:0/798684000 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66280c3160 con 0x7f6628085270 2021-02-10T11:45:54.416 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:54.411 7f662ef79700 1 -- 172.21.2.112:0/798684000 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66280c36c0 con 0x7f6628085270 2021-02-10T11:45:54.417 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:54.412 7f66257fa700 1 -- 172.21.2.112:0/798684000 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6610000c90 con 0x7f6628085270 2021-02-10T11:45:54.417 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:54.412 7f66257fa700 1 --2- 172.21.2.112:0/798684000 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f660c0186c0 0x7f660c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:54.417 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:54.412 7f66257fa700 1 -- 172.21.2.112:0/798684000 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f66100325a0 con 0x7f6628085270 2021-02-10T11:45:54.417 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:54.412 7f6627fff700 1 --2- 172.21.2.112:0/798684000 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f660c0186c0 0x7f660c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:54.417 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:54.413 7f6627fff700 1 --2- 172.21.2.112:0/798684000 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f660c0186c0 0x7f660c01abc0 crc :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:54.445 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:54.443 7f662ef79700 1 -- 172.21.2.112:0/798684000 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6628082a20 con 0x7f6628085270 2021-02-10T11:45:54.447 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:54.446 7f66257fa700 1 -- 172.21.2.112:0/798684000 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6610000940 con 0x7f6628085270 2021-02-10T11:45:54.595 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:54.593 7f662ef79700 1 -- 172.21.2.112:0/798684000 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f6628082a20 con 0x7f6628085270 2021-02-10T11:45:54.596 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:54.594 7f66257fa700 1 -- 172.21.2.112:0/798684000 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v40)=0 v40) v1 ==== 122+0+0 (secure 0 0 0) 0x7f6610000c90 con 0x7f6628085270 2021-02-10T11:45:54.598 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:54.596 7f662ef79700 1 -- 172.21.2.112:0/798684000 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f660c0186c0 msgr2=0x7f660c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:54.598 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:54.596 7f662ef79700 1 --2- 172.21.2.112:0/798684000 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f660c0186c0 0x7f660c01abc0 crc :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:54.598 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:54.596 7f662ef79700 1 -- 172.21.2.112:0/798684000 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6628085270 msgr2=0x7f6628085670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:54.598 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:54.596 7f662ef79700 1 --2- 172.21.2.112:0/798684000 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6628085270 0x7f6628085670 secure :-1 s=READY pgs=2809 cs=0 l=1 rev1=1 rx=0x7f6610006f00 tx=0x7f661000b940).stop 2021-02-10T11:45:54.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:54.596 7f662ef79700 1 -- 172.21.2.112:0/798684000 shutdown_connections 2021-02-10T11:45:54.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:54.596 7f662ef79700 1 --2- 172.21.2.112:0/798684000 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f660c0186c0 0x7f660c01abc0 unknown :-1 s=CLOSED pgs=1277 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:54.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:54.596 7f662ef79700 1 --2- 172.21.2.112:0/798684000 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f66280c2740 0x7f66280c2b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:54.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:54.596 7f662ef79700 1 --2- 172.21.2.112:0/798684000 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6628085c90 0x7f6628086090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:54.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:54.596 7f662ef79700 1 --2- 172.21.2.112:0/798684000 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6628085270 0x7f6628085670 unknown :-1 s=CLOSED pgs=2809 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:54.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:54.596 7f662ef79700 1 -- 172.21.2.112:0/798684000 >> 172.21.2.112:0/798684000 conn(0x7f66280b5f10 msgr2=0x7f662809f200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:54.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:54.596 7f662ef79700 1 -- 172.21.2.112:0/798684000 shutdown_connections 2021-02-10T11:45:54.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:54.596 7f662ef79700 1 -- 172.21.2.112:0/798684000 wait complete. 2021-02-10T11:45:54.606 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000486841 2021-02-10T11:45:55.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.020 7fed1dda8700 1 Processor -- start 2021-02-10T11:45:55.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.021 7fed1dda8700 1 -- start start 2021-02-10T11:45:55.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.021 7fed1dda8700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fed180d1860 0x7fed181b04b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:55.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.021 7fed1dda8700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fed181b0ad0 0x7fed18108930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:55.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.021 7fed1dda8700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fed180d1c60 0x7fed18108f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:55.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.021 7fed1dda8700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed180e5ee0 con 0x7fed180d1c60 2021-02-10T11:45:55.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.021 7fed1dda8700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed18109a50 con 0x7fed181b0ad0 2021-02-10T11:45:55.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.021 7fed1dda8700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed18088070 con 0x7fed180d1860 2021-02-10T11:45:55.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.021 7fed17fff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fed180d1c60 0x7fed18108f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:55.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.021 7fed17fff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fed180d1c60 0x7fed18108f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:59128/0 (socket says 172.21.2.112:59128) 2021-02-10T11:45:55.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.021 7fed17fff700 1 -- 172.21.2.112:0/695966352 learned_addr learned my addr 172.21.2.112:0/695966352 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:45:55.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.022 7fed17fff700 1 -- 172.21.2.112:0/695966352 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fed180d1860 msgr2=0x7fed181b04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:55.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.022 7fed17fff700 1 --2- 172.21.2.112:0/695966352 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fed180d1860 0x7fed181b04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:55.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.022 7fed17fff700 1 -- 172.21.2.112:0/695966352 >> v2:172.21.2.120:3300/0 conn(0x7fed181b0ad0 msgr2=0x7fed18108930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:55.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.022 7fed17fff700 1 --2- 172.21.2.112:0/695966352 >> v2:172.21.2.120:3300/0 conn(0x7fed181b0ad0 0x7fed18108930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:55.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.022 7fed17fff700 1 -- 172.21.2.112:0/695966352 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed18109570 con 0x7fed180d1c60 2021-02-10T11:45:55.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.022 7fed17fff700 1 --2- 172.21.2.112:0/695966352 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fed180d1c60 0x7fed18108f50 secure :-1 s=READY pgs=2810 cs=0 l=1 rev1=1 rx=0x7fed10006f00 tx=0x7fed10009f10).ready entity=mon.0 client_cookie=dadfdea5d3a8befc server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:55.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.022 7fed167fc700 1 -- 172.21.2.112:0/695966352 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fed1000b3b0 con 0x7fed180d1c60 2021-02-10T11:45:55.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.022 7fed167fc700 1 -- 172.21.2.112:0/695966352 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed1000b630 con 0x7fed180d1c60 2021-02-10T11:45:55.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.022 7fed167fc700 1 -- 172.21.2.112:0/695966352 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fed1000bb50 con 0x7fed180d1c60 2021-02-10T11:45:55.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.022 7fed1dda8700 1 -- 172.21.2.112:0/695966352 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fed180d1c60 msgr2=0x7fed18108f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:55.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.022 7fed1dda8700 1 --2- 172.21.2.112:0/695966352 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fed180d1c60 0x7fed18108f50 secure :-1 s=READY pgs=2810 cs=0 l=1 rev1=1 rx=0x7fed10006f00 tx=0x7fed10009f10).stop 2021-02-10T11:45:55.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.023 7fed1dda8700 1 -- 172.21.2.112:0/695966352 shutdown_connections 2021-02-10T11:45:55.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.023 7fed1dda8700 1 --2- 172.21.2.112:0/695966352 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fed180d1c60 0x7fed18108f50 unknown :-1 s=CLOSED pgs=2810 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:55.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.023 7fed1dda8700 1 --2- 172.21.2.112:0/695966352 >> v2:172.21.2.120:3300/0 conn(0x7fed181b0ad0 0x7fed18108930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:55.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.023 7fed1dda8700 1 --2- 172.21.2.112:0/695966352 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fed180d1860 0x7fed181b04b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:55.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.023 7fed1dda8700 1 -- 172.21.2.112:0/695966352 >> 172.21.2.112:0/695966352 conn(0x7fed180ce410 msgr2=0x7fed180d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:55.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.023 7fed1dda8700 1 -- 172.21.2.112:0/695966352 shutdown_connections 2021-02-10T11:45:55.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.023 7fed1dda8700 1 -- 172.21.2.112:0/695966352 wait complete. 2021-02-10T11:45:55.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.037 7fed1dda8700 1 Processor -- start 2021-02-10T11:45:55.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.038 7fed1dda8700 1 -- start start 2021-02-10T11:45:55.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.038 7fed1dda8700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fed181a79e0 0x7fed181a7de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:55.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.038 7fed1dda8700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fed180ba0d0 0x7fed180ba4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:55.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.038 7fed1dda8700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fed180baaf0 0x7fed180baef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:55.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.038 7fed1dda8700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed18086d30 con 0x7fed180baaf0 2021-02-10T11:45:55.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.038 7fed1dda8700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed18082b50 con 0x7fed180ba0d0 2021-02-10T11:45:55.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.038 7fed1dda8700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed18082d70 con 0x7fed181a79e0 2021-02-10T11:45:55.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.038 7fed17fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fed180baaf0 0x7fed180baef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:55.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.038 7fed17fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fed180baaf0 0x7fed180baef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:59132/0 (socket says 172.21.2.112:59132) 2021-02-10T11:45:55.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.038 7fed17fff700 1 -- 172.21.2.112:0/4046410220 learned_addr learned my addr 172.21.2.112:0/4046410220 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:45:55.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.039 7fed17fff700 1 -- 172.21.2.112:0/4046410220 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fed181a79e0 msgr2=0x7fed181a7de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:55.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.039 7fed17fff700 1 --2- 172.21.2.112:0/4046410220 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fed181a79e0 0x7fed181a7de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:55.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.039 7fed17fff700 1 -- 172.21.2.112:0/4046410220 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fed180ba0d0 msgr2=0x7fed180ba4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:55.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.039 7fed17fff700 1 --2- 172.21.2.112:0/4046410220 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fed180ba0d0 0x7fed180ba4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:55.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.039 7fed17fff700 1 -- 172.21.2.112:0/4046410220 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed18082f90 con 0x7fed180baaf0 2021-02-10T11:45:55.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.039 7fed17fff700 1 --2- 172.21.2.112:0/4046410220 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fed180baaf0 0x7fed180baef0 secure :-1 s=READY pgs=2811 cs=0 l=1 rev1=1 rx=0x7fed100099f0 tx=0x7fed1000bc40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:55.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.039 7fed0ffff700 1 -- 172.21.2.112:0/4046410220 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fed10000cc0 con 0x7fed180baaf0 2021-02-10T11:45:55.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.039 7fed0ffff700 1 -- 172.21.2.112:0/4046410220 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed1000bd80 con 0x7fed180baaf0 2021-02-10T11:45:55.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.039 7fed0ffff700 1 -- 172.21.2.112:0/4046410220 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fed10005aa0 con 0x7fed180baaf0 2021-02-10T11:45:55.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.039 7fed1dda8700 1 -- 172.21.2.112:0/4046410220 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed18082f90 con 0x7fed180baaf0 2021-02-10T11:45:55.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.039 7fed1dda8700 1 -- 172.21.2.112:0/4046410220 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed18083520 con 0x7fed180baaf0 2021-02-10T11:45:55.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.040 7fed0ffff700 1 -- 172.21.2.112:0/4046410220 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fed10005a30 con 0x7fed180baaf0 2021-02-10T11:45:55.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.040 7fed0ffff700 1 --2- 172.21.2.112:0/4046410220 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fecfc0186c0 0x7fecfc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:55.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.040 7fed0ffff700 1 -- 172.21.2.112:0/4046410220 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fed10032980 con 0x7fed180baaf0 2021-02-10T11:45:55.045 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.041 7fed177fe700 1 --2- 172.21.2.112:0/4046410220 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fecfc0186c0 0x7fecfc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:55.045 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.041 7fed177fe700 1 --2- 172.21.2.112:0/4046410220 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fecfc0186c0 0x7fecfc01abc0 crc :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:55.073 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.071 7fed1dda8700 1 -- 172.21.2.112:0/4046410220 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed18109a50 con 0x7fed180baaf0 2021-02-10T11:45:55.075 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.074 7fed0ffff700 1 -- 172.21.2.112:0/4046410220 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fed10005820 con 0x7fed180baaf0 2021-02-10T11:45:55.201 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.199 7fed1dda8700 1 -- 172.21.2.112:0/4046410220 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000486841", "target": ["mgr", ""]}) v1 -- 0x7fed18083520 con 0x7fecfc0186c0 2021-02-10T11:45:55.206 INFO:tasks.ceph.mgr.x.gibba020.stderr:2021-02-10 11:45:55.204 7f6382eee700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:45:55.206 INFO:tasks.ceph.mgr.x.gibba020.stderr:2021-02-10 11:45:55.204 7f6382eee700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:45:55.206 INFO:tasks.ceph.mgr.x.gibba020.stderr:2021-02-10 11:45:55.204 7f6382eee700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:45:55.206 INFO:tasks.ceph.mgr.x.gibba020.stderr:2021-02-10 11:45:55.204 7f6382eee700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:45:55.206 INFO:tasks.ceph.mgr.x.gibba020.stderr:2021-02-10 11:45:55.204 7f6382eee700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:45:55.236 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.234 7fed0ffff700 1 -- 172.21.2.112:0/4046410220 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fed080009b0 con 0x7fecfc0186c0 2021-02-10T11:45:55.267 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.265 7fed1dda8700 1 -- 172.21.2.112:0/4046410220 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fecfc0186c0 msgr2=0x7fecfc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:55.267 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.265 7fed1dda8700 1 --2- 172.21.2.112:0/4046410220 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fecfc0186c0 0x7fecfc01abc0 crc :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:55.267 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.265 7fed1dda8700 1 -- 172.21.2.112:0/4046410220 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fed180baaf0 msgr2=0x7fed180baef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:55.267 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.265 7fed1dda8700 1 --2- 172.21.2.112:0/4046410220 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fed180baaf0 0x7fed180baef0 secure :-1 s=READY pgs=2811 cs=0 l=1 rev1=1 rx=0x7fed100099f0 tx=0x7fed1000bc40).stop 2021-02-10T11:45:55.267 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.265 7fed1dda8700 1 -- 172.21.2.112:0/4046410220 shutdown_connections 2021-02-10T11:45:55.267 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.265 7fed1dda8700 1 --2- 172.21.2.112:0/4046410220 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fecfc0186c0 0x7fecfc01abc0 unknown :-1 s=CLOSED pgs=1278 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:55.267 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.265 7fed1dda8700 1 --2- 172.21.2.112:0/4046410220 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fed180ba0d0 0x7fed180ba4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:55.267 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.265 7fed1dda8700 1 --2- 172.21.2.112:0/4046410220 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fed180baaf0 0x7fed180baef0 unknown :-1 s=CLOSED pgs=2811 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:55.268 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.265 7fed1dda8700 1 --2- 172.21.2.112:0/4046410220 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fed181a79e0 0x7fed181a7de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:55.268 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.265 7fed1dda8700 1 -- 172.21.2.112:0/4046410220 >> 172.21.2.112:0/4046410220 conn(0x7fed180aecd0 msgr2=0x7fed1809c9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:55.268 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.265 7fed1dda8700 1 -- 172.21.2.112:0/4046410220 shutdown_connections 2021-02-10T11:45:55.268 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.265 7fed1dda8700 1 -- 172.21.2.112:0/4046410220 wait complete. 2021-02-10T11:45:55.275 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume snapshot create cephfs subvolume_0000000000486841 snapshot_0000000000014182 2021-02-10T11:45:55.708 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.689 7f76292e4700 1 Processor -- start 2021-02-10T11:45:55.708 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.689 7f76292e4700 1 -- start start 2021-02-10T11:45:55.708 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.690 7f76292e4700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f76240d1880 0x7f76241b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:55.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.690 7f76292e4700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f76241b0af0 0x7f7624108950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:55.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.690 7f76292e4700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76240fe1d0 con 0x7f76240d1c80 2021-02-10T11:45:55.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.690 7f76292e4700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7624109a70 con 0x7f76241b0af0 2021-02-10T11:45:55.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.690 7f76292e4700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7624109760 con 0x7f76240d1880 2021-02-10T11:45:55.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.690 7f762387e700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f76240d1c80 0x7f7624108f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:58600/0 (socket says 172.21.2.112:58600) 2021-02-10T11:45:55.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.690 7f762387e700 1 -- 172.21.2.112:0/1816763040 learned_addr learned my addr 172.21.2.112:0/1816763040 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:45:55.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.690 7f762287c700 1 --2- 172.21.2.112:0/1816763040 >> v2:172.21.2.112:3300/0 conn(0x7f76241b0af0 0x7f7624108950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:55.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.690 7f762307d700 1 --2- 172.21.2.112:0/1816763040 >> v2:172.21.2.120:3300/0 conn(0x7f76240d1880 0x7f76241b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:55.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.691 7f762287c700 1 -- 172.21.2.112:0/1816763040 >> v2:172.21.2.120:3300/0 conn(0x7f76240d1880 msgr2=0x7f76241b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:55.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.691 7f762287c700 1 --2- 172.21.2.112:0/1816763040 >> v2:172.21.2.120:3300/0 conn(0x7f76240d1880 0x7f76241b04d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:55.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.691 7f762287c700 1 -- 172.21.2.112:0/1816763040 >> v1:172.21.2.112:6789/0 conn(0x7f76240d1c80 legacy=0x7f7624108f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:55.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.691 7f762287c700 1 -- 172.21.2.112:0/1816763040 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7624088070 con 0x7f76241b0af0 2021-02-10T11:45:55.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.691 7f762307d700 1 --2- 172.21.2.112:0/1816763040 >> v2:172.21.2.120:3300/0 conn(0x7f76240d1880 0x7f76241b04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:45:55.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.691 7f762287c700 1 --2- 172.21.2.112:0/1816763040 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f76241b0af0 0x7f7624108950 secure :-1 s=READY pgs=2813 cs=0 l=1 rev1=1 rx=0x7f7614006650 tx=0x7f7614009a80).ready entity=mon.0 client_cookie=d51ff961f7dea76e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:55.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.691 7f762207b700 1 -- 172.21.2.112:0/1816763040 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f761400c0c0 con 0x7f76241b0af0 2021-02-10T11:45:55.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.691 7f762207b700 1 -- 172.21.2.112:0/1816763040 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f761400c300 con 0x7f76241b0af0 2021-02-10T11:45:55.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.691 7f762207b700 1 -- 172.21.2.112:0/1816763040 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f761400c750 con 0x7f76241b0af0 2021-02-10T11:45:55.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.691 7f76292e4700 1 -- 172.21.2.112:0/1816763040 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f76241b0af0 msgr2=0x7f7624108950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:55.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.691 7f76292e4700 1 --2- 172.21.2.112:0/1816763040 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f76241b0af0 0x7f7624108950 secure :-1 s=READY pgs=2813 cs=0 l=1 rev1=1 rx=0x7f7614006650 tx=0x7f7614009a80).stop 2021-02-10T11:45:55.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.692 7f76292e4700 1 -- 172.21.2.112:0/1816763040 shutdown_connections 2021-02-10T11:45:55.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.692 7f76292e4700 1 --2- 172.21.2.112:0/1816763040 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f76241b0af0 0x7f7624108950 unknown :-1 s=CLOSED pgs=2813 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:55.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.692 7f76292e4700 1 --2- 172.21.2.112:0/1816763040 >> v2:172.21.2.120:3300/0 conn(0x7f76240d1880 0x7f76241b04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:55.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.692 7f76292e4700 1 -- 172.21.2.112:0/1816763040 >> 172.21.2.112:0/1816763040 conn(0x7f76240ce430 msgr2=0x7f76240d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:55.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.692 7f76292e4700 1 -- 172.21.2.112:0/1816763040 shutdown_connections 2021-02-10T11:45:55.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.692 7f76292e4700 1 -- 172.21.2.112:0/1816763040 wait complete. 2021-02-10T11:45:55.716 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.707 7f76292e4700 1 Processor -- start 2021-02-10T11:45:55.716 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.707 7f76292e4700 1 -- start start 2021-02-10T11:45:55.716 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.707 7f76292e4700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f76241a5af0 0x7f76240b17a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:55.716 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.707 7f76292e4700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f76241a6030 0x7f76240b1dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:55.717 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.707 7f76292e4700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f76240b23e0 0x7f7624099e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:55.717 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.707 7f76292e4700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f762418e0e0 con 0x7f76240b23e0 2021-02-10T11:45:55.717 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.707 7f76292e4700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f762409a470 con 0x7f76241a6030 2021-02-10T11:45:55.717 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.707 7f76292e4700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f762409a690 con 0x7f76241a5af0 2021-02-10T11:45:55.717 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.707 7f762387e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f76240b23e0 0x7f7624099e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:55.718 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.707 7f762387e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f76240b23e0 0x7f7624099e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:59148/0 (socket says 172.21.2.112:59148) 2021-02-10T11:45:55.718 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.707 7f762387e700 1 -- 172.21.2.112:0/3559660533 learned_addr learned my addr 172.21.2.112:0/3559660533 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:45:55.718 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.708 7f762387e700 1 -- 172.21.2.112:0/3559660533 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f76241a5af0 msgr2=0x7f76240b17a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:45:55.718 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.708 7f762387e700 1 --2- 172.21.2.112:0/3559660533 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f76241a5af0 0x7f76240b17a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:55.718 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.708 7f762387e700 1 -- 172.21.2.112:0/3559660533 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f76241a6030 msgr2=0x7f76240b1dc0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:45:55.718 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.708 7f762387e700 1 --2- 172.21.2.112:0/3559660533 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f76241a6030 0x7f76240b1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:55.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.708 7f762387e700 1 -- 172.21.2.112:0/3559660533 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f762409a8b0 con 0x7f76240b23e0 2021-02-10T11:45:55.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.708 7f762387e700 1 --2- 172.21.2.112:0/3559660533 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f76240b23e0 0x7f7624099e50 secure :-1 s=READY pgs=2814 cs=0 l=1 rev1=1 rx=0x7f761c001bf0 tx=0x7f761c00ac50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:55.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.708 7f761bfff700 1 -- 172.21.2.112:0/3559660533 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f761c0070d0 con 0x7f76240b23e0 2021-02-10T11:45:55.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.708 7f761bfff700 1 -- 172.21.2.112:0/3559660533 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f761c007350 con 0x7f76240b23e0 2021-02-10T11:45:55.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.708 7f761bfff700 1 -- 172.21.2.112:0/3559660533 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f761c004720 con 0x7f76240b23e0 2021-02-10T11:45:55.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.708 7f76292e4700 1 -- 172.21.2.112:0/3559660533 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f762409a8b0 con 0x7f76240b23e0 2021-02-10T11:45:55.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.708 7f76292e4700 1 -- 172.21.2.112:0/3559660533 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f762418cc10 con 0x7f76240b23e0 2021-02-10T11:45:55.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.709 7f761bfff700 1 -- 172.21.2.112:0/3559660533 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f761c007530 con 0x7f76240b23e0 2021-02-10T11:45:55.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.709 7f761bfff700 1 --2- 172.21.2.112:0/3559660533 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f76080186c0 0x7f760801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:55.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.709 7f761bfff700 1 -- 172.21.2.112:0/3559660533 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f761c032760 con 0x7f76240b23e0 2021-02-10T11:45:55.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.709 7f762307d700 1 --2- 172.21.2.112:0/3559660533 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f76080186c0 0x7f760801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:55.721 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.710 7f762307d700 1 --2- 172.21.2.112:0/3559660533 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f76080186c0 0x7f760801abc0 crc :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:55.742 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.740 7f76292e4700 1 -- 172.21.2.112:0/3559660533 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7624082a70 con 0x7f76240b23e0 2021-02-10T11:45:55.744 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.742 7f761bfff700 1 -- 172.21.2.112:0/3559660533 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f761c0014e0 con 0x7f76240b23e0 2021-02-10T11:45:55.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.871 7f76292e4700 1 -- 172.21.2.112:0/3559660533 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume snapshot create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000486841", "target": ["mgr", ""], "snap_name": "snapshot_0000000000014182"}) v1 -- 0x7f762409ae40 con 0x7f76080186c0 2021-02-10T11:45:55.882 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.880 7f761bfff700 1 -- 172.21.2.112:0/3559660533 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f760c0009b0 con 0x7f76080186c0 2021-02-10T11:45:55.889 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.887 7f76292e4700 1 -- 172.21.2.112:0/3559660533 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f76080186c0 msgr2=0x7f760801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:55.889 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.887 7f76292e4700 1 --2- 172.21.2.112:0/3559660533 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f76080186c0 0x7f760801abc0 crc :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:55.890 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.887 7f76292e4700 1 -- 172.21.2.112:0/3559660533 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f76240b23e0 msgr2=0x7f7624099e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:55.890 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.887 7f76292e4700 1 --2- 172.21.2.112:0/3559660533 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f76240b23e0 0x7f7624099e50 secure :-1 s=READY pgs=2814 cs=0 l=1 rev1=1 rx=0x7f761c001bf0 tx=0x7f761c00ac50).stop 2021-02-10T11:45:55.890 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.887 7f76292e4700 1 -- 172.21.2.112:0/3559660533 shutdown_connections 2021-02-10T11:45:55.890 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.887 7f76292e4700 1 --2- 172.21.2.112:0/3559660533 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f76080186c0 0x7f760801abc0 unknown :-1 s=CLOSED pgs=1279 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:55.890 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.887 7f76292e4700 1 --2- 172.21.2.112:0/3559660533 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f76241a6030 0x7f76240b1dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:55.890 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.887 7f76292e4700 1 --2- 172.21.2.112:0/3559660533 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f76240b23e0 0x7f7624099e50 unknown :-1 s=CLOSED pgs=2814 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:55.890 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.887 7f76292e4700 1 --2- 172.21.2.112:0/3559660533 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f76241a5af0 0x7f76240b17a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:55.890 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.887 7f76292e4700 1 -- 172.21.2.112:0/3559660533 >> 172.21.2.112:0/3559660533 conn(0x7f76240873b0 msgr2=0x7f76241b2b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:55.891 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.888 7f76292e4700 1 -- 172.21.2.112:0/3559660533 shutdown_connections 2021-02-10T11:45:55.891 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:55.888 7f76292e4700 1 -- 172.21.2.112:0/3559660533 wait complete. 2021-02-10T11:45:55.898 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume snapshot rm cephfs subvolume_0000000000486841 snapshot_0000000000014182 2021-02-10T11:45:55.932 DEBUG:teuthology.orchestra.run.gibba012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:45:55.944 DEBUG:teuthology.orchestra.run.gibba020:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:45:56.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.313 7f6b4d0b7700 1 Processor -- start 2021-02-10T11:45:56.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.313 7f6b4d0b7700 1 -- start start 2021-02-10T11:45:56.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.314 7f6b4d0b7700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f6b4810b000 0x7f6b480cf760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:56.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.314 7f6b4d0b7700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b480fe1d0 con 0x7f6b480d1c80 2021-02-10T11:45:56.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.314 7f6b4d0b7700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b481b0cc0 con 0x7f6b480d1880 2021-02-10T11:45:56.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.314 7f6b4d0b7700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b48109a70 con 0x7f6b4810b000 2021-02-10T11:45:56.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.314 7f6b4659c700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f6b480d1c80 0x7f6b480cef70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:58616/0 (socket says 172.21.2.112:58616) 2021-02-10T11:45:56.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.314 7f6b4659c700 1 -- 172.21.2.112:0/1845635470 learned_addr learned my addr 172.21.2.112:0/1845635470 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:45:56.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.314 7f6b4759e700 1 --2- 172.21.2.112:0/1845635470 >> v2:172.21.2.120:3300/0 conn(0x7f6b4810b000 0x7f6b480cf760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:56.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.314 7f6b45d9b700 1 -- 172.21.2.112:0/1845635470 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 311421445 0 0) 0x7f6b300010b0 con 0x7f6b480d1c80 2021-02-10T11:45:56.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.314 7f6b45d9b700 1 -- 172.21.2.112:0/1845635470 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b34002fe0 con 0x7f6b480d1c80 2021-02-10T11:45:56.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.314 7f6b4759e700 1 -- 172.21.2.112:0/1845635470 >> v1:172.21.2.120:6789/0 conn(0x7f6b480d1880 legacy=0x7f6b481b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:56.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.314 7f6b4759e700 1 -- 172.21.2.112:0/1845635470 >> v1:172.21.2.112:6789/0 conn(0x7f6b480d1c80 legacy=0x7f6b480cef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:56.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.314 7f6b4759e700 1 -- 172.21.2.112:0/1845635470 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b4810b400 con 0x7f6b4810b000 2021-02-10T11:45:56.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.315 7f6b4759e700 1 --2- 172.21.2.112:0/1845635470 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6b4810b000 0x7f6b480cf760 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 rx=0x7f6b40006e50 tx=0x7f6b40009e70).ready entity=mon.1 client_cookie=14c4664181196a63 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:56.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.315 7f6b45d9b700 1 -- 172.21.2.112:0/1845635470 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6b40006a20 con 0x7f6b4810b000 2021-02-10T11:45:56.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.315 7f6b45d9b700 1 -- 172.21.2.112:0/1845635470 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b40006790 con 0x7f6b4810b000 2021-02-10T11:45:56.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.315 7f6b45d9b700 1 -- 172.21.2.112:0/1845635470 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6b4000b700 con 0x7f6b4810b000 2021-02-10T11:45:56.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.315 7f6b4d0b7700 1 -- 172.21.2.112:0/1845635470 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6b4810b000 msgr2=0x7f6b480cf760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:56.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.315 7f6b4d0b7700 1 --2- 172.21.2.112:0/1845635470 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6b4810b000 0x7f6b480cf760 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 rx=0x7f6b40006e50 tx=0x7f6b40009e70).stop 2021-02-10T11:45:56.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.315 7f6b4d0b7700 1 -- 172.21.2.112:0/1845635470 shutdown_connections 2021-02-10T11:45:56.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.315 7f6b4d0b7700 1 --2- 172.21.2.112:0/1845635470 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6b4810b000 0x7f6b480cf760 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:56.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.315 7f6b4d0b7700 1 -- 172.21.2.112:0/1845635470 >> 172.21.2.112:0/1845635470 conn(0x7f6b480ce430 msgr2=0x7f6b480d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:56.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.316 7f6b4d0b7700 1 -- 172.21.2.112:0/1845635470 shutdown_connections 2021-02-10T11:45:56.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.316 7f6b4d0b7700 1 -- 172.21.2.112:0/1845635470 wait complete. 2021-02-10T11:45:56.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.330 7f6b4d0b7700 1 Processor -- start 2021-02-10T11:45:56.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.330 7f6b4d0b7700 1 -- start start 2021-02-10T11:45:56.339 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.330 7f6b4d0b7700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6b48187e80 0x7f6b48188280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:56.339 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.330 7f6b4d0b7700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6b481888a0 0x7f6b48189490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:56.339 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.330 7f6b4d0b7700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6b48189ab0 0x7f6b48189eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:56.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.331 7f6b4d0b7700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b48188ca0 con 0x7f6b48187e80 2021-02-10T11:45:56.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.331 7f6b4d0b7700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b48188f10 con 0x7f6b481888a0 2021-02-10T11:45:56.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.331 7f6b4d0b7700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b48099520 con 0x7f6b48189ab0 2021-02-10T11:45:56.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.331 7f6b46d9d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6b48187e80 0x7f6b48188280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:56.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.331 7f6b4659c700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6b481888a0 0x7f6b48189490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:56.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.331 7f6b46d9d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6b48187e80 0x7f6b48188280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:59158/0 (socket says 172.21.2.112:59158) 2021-02-10T11:45:56.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.331 7f6b46d9d700 1 -- 172.21.2.112:0/3119694464 learned_addr learned my addr 172.21.2.112:0/3119694464 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:45:56.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.331 7f6b4659c700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6b481888a0 0x7f6b48189490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3300/0 says I am v2:172.21.2.112:33206/0 (socket says 172.21.2.112:33206) 2021-02-10T11:45:56.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.331 7f6b4759e700 1 --2- 172.21.2.112:0/3119694464 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6b48189ab0 0x7f6b48189eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:56.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.331 7f6b46d9d700 1 -- 172.21.2.112:0/3119694464 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6b48189ab0 msgr2=0x7f6b48189eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:56.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.331 7f6b46d9d700 1 --2- 172.21.2.112:0/3119694464 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6b48189ab0 0x7f6b48189eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:56.342 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.331 7f6b46d9d700 1 -- 172.21.2.112:0/3119694464 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6b481888a0 msgr2=0x7f6b48189490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:56.342 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.331 7f6b46d9d700 1 --2- 172.21.2.112:0/3119694464 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6b481888a0 0x7f6b48189490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:56.342 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.331 7f6b46d9d700 1 -- 172.21.2.112:0/3119694464 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b480999c0 con 0x7f6b48187e80 2021-02-10T11:45:56.342 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.331 7f6b46d9d700 1 --2- 172.21.2.112:0/3119694464 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6b48187e80 0x7f6b48188280 secure :-1 s=READY pgs=2816 cs=0 l=1 rev1=1 rx=0x7f6b3800b5b0 tx=0x7f6b3800b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:56.342 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.332 7f6b3f7fe700 1 -- 172.21.2.112:0/3119694464 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6b3800bd40 con 0x7f6b48187e80 2021-02-10T11:45:56.342 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.332 7f6b3f7fe700 1 -- 172.21.2.112:0/3119694464 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b38000c60 con 0x7f6b48187e80 2021-02-10T11:45:56.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.332 7f6b3f7fe700 1 -- 172.21.2.112:0/3119694464 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6b38006210 con 0x7f6b48187e80 2021-02-10T11:45:56.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.332 7f6b4d0b7700 1 -- 172.21.2.112:0/3119694464 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b480999c0 con 0x7f6b48187e80 2021-02-10T11:45:56.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.332 7f6b4d0b7700 1 -- 172.21.2.112:0/3119694464 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b4818a7c0 con 0x7f6b48187e80 2021-02-10T11:45:56.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.332 7f6b3f7fe700 1 -- 172.21.2.112:0/3119694464 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6b38006020 con 0x7f6b48187e80 2021-02-10T11:45:56.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.333 7f6b3f7fe700 1 --2- 172.21.2.112:0/3119694464 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6b2c0186c0 0x7f6b2c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:56.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.333 7f6b3f7fe700 1 -- 172.21.2.112:0/3119694464 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f6b38032580 con 0x7f6b48187e80 2021-02-10T11:45:56.344 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.333 7f6b4659c700 1 --2- 172.21.2.112:0/3119694464 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6b2c0186c0 0x7f6b2c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:56.344 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.333 7f6b4659c700 1 --2- 172.21.2.112:0/3119694464 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6b2c0186c0 0x7f6b2c01abc0 crc :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:56.366 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.364 7f6b4d0b7700 1 -- 172.21.2.112:0/3119694464 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b48195bd0 con 0x7f6b48187e80 2021-02-10T11:45:56.368 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.366 7f6b3f7fe700 1 -- 172.21.2.112:0/3119694464 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6b38000940 con 0x7f6b48187e80 2021-02-10T11:45:56.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.494 7f6b4d0b7700 1 -- 172.21.2.112:0/3119694464 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume snapshot rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000486841", "target": ["mgr", ""], "snap_name": "snapshot_0000000000014182"}) v1 -- 0x7f6b48188ca0 con 0x7f6b2c0186c0 2021-02-10T11:45:56.509 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.507 7f6b3f7fe700 1 -- 172.21.2.112:0/3119694464 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f6b300009b0 con 0x7f6b2c0186c0 2021-02-10T11:45:56.514 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.512 7f6b4d0b7700 1 -- 172.21.2.112:0/3119694464 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6b2c0186c0 msgr2=0x7f6b2c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:56.515 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.512 7f6b4d0b7700 1 --2- 172.21.2.112:0/3119694464 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6b2c0186c0 0x7f6b2c01abc0 crc :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:56.515 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.512 7f6b4d0b7700 1 -- 172.21.2.112:0/3119694464 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6b48187e80 msgr2=0x7f6b48188280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:56.515 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.512 7f6b4d0b7700 1 --2- 172.21.2.112:0/3119694464 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6b48187e80 0x7f6b48188280 secure :-1 s=READY pgs=2816 cs=0 l=1 rev1=1 rx=0x7f6b3800b5b0 tx=0x7f6b3800b950).stop 2021-02-10T11:45:56.515 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.512 7f6b4d0b7700 1 -- 172.21.2.112:0/3119694464 shutdown_connections 2021-02-10T11:45:56.515 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.512 7f6b4d0b7700 1 --2- 172.21.2.112:0/3119694464 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6b2c0186c0 0x7f6b2c01abc0 unknown :-1 s=CLOSED pgs=1280 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:56.515 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.512 7f6b4d0b7700 1 --2- 172.21.2.112:0/3119694464 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6b481888a0 0x7f6b48189490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:56.515 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.512 7f6b4d0b7700 1 --2- 172.21.2.112:0/3119694464 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6b48189ab0 0x7f6b48189eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:56.515 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.512 7f6b4d0b7700 1 --2- 172.21.2.112:0/3119694464 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6b48187e80 0x7f6b48188280 unknown :-1 s=CLOSED pgs=2816 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:56.515 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.512 7f6b4d0b7700 1 -- 172.21.2.112:0/3119694464 >> 172.21.2.112:0/3119694464 conn(0x7f6b4819f940 msgr2=0x7f6b4818c290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:56.515 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.513 7f6b4d0b7700 1 -- 172.21.2.112:0/3119694464 shutdown_connections 2021-02-10T11:45:56.515 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.513 7f6b4d0b7700 1 -- 172.21.2.112:0/3119694464 wait complete. 2021-02-10T11:45:56.523 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume snapshot rm cephfs subvolume_0000000000486841 snapshot_0000000000014182 2021-02-10T11:45:56.956 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.936 7f08b89a9700 1 Processor -- start 2021-02-10T11:45:56.957 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.936 7f08b89a9700 1 -- start start 2021-02-10T11:45:56.957 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.937 7f08b89a9700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f08b40d1880 0x7f08b41b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:56.957 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.937 7f08b89a9700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f08b40cff80 0x7f08b4109140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:56.957 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.937 7f08b89a9700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08b4109a70 con 0x7f08b41b0af0 2021-02-10T11:45:56.957 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.937 7f08b89a9700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08b4109760 con 0x7f08b40d1880 2021-02-10T11:45:56.957 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.937 7f08b89a9700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08b4088070 con 0x7f08b40cff80 2021-02-10T11:45:56.957 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.937 7f08b1e1b700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f08b41b0af0 0x7f08b4108950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:58628/0 (socket says 172.21.2.112:58628) 2021-02-10T11:45:56.957 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.937 7f08b1e1b700 1 -- 172.21.2.112:0/1044709887 learned_addr learned my addr 172.21.2.112:0/1044709887 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:45:56.957 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.937 7f08b161a700 1 -- 172.21.2.112:0/1044709887 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1166789392 0 0) 0x7f08a80010b0 con 0x7f08b41b0af0 2021-02-10T11:45:56.957 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.937 7f08b161a700 1 -- 172.21.2.112:0/1044709887 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f089c002fe0 con 0x7f08b41b0af0 2021-02-10T11:45:56.957 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.937 7f08b2e1d700 1 --2- 172.21.2.112:0/1044709887 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f08b40cff80 0x7f08b4109140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:56.958 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.937 7f08b261c700 1 --2- 172.21.2.112:0/1044709887 >> v2:172.21.2.120:3300/0 conn(0x7f08b40d1880 0x7f08b41b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:56.958 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.938 7f08b161a700 1 -- 172.21.2.112:0/1044709887 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3979957087 0 0) 0x7f08a80010b0 con 0x7f08b41b0af0 2021-02-10T11:45:56.958 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.938 7f08b161a700 1 -- 172.21.2.112:0/1044709887 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f08b40cff80 msgr2=0x7f08b4109140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:56.958 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.938 7f08b161a700 1 --2- 172.21.2.112:0/1044709887 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f08b40cff80 0x7f08b4109140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:56.958 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.938 7f08b161a700 1 -- 172.21.2.112:0/1044709887 >> v2:172.21.2.120:3300/0 conn(0x7f08b40d1880 msgr2=0x7f08b41b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:56.958 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.938 7f08b161a700 1 --2- 172.21.2.112:0/1044709887 >> v2:172.21.2.120:3300/0 conn(0x7f08b40d1880 0x7f08b41b04d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:56.958 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.938 7f08b161a700 1 -- 172.21.2.112:0/1044709887 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08b40d1c80 con 0x7f08b41b0af0 2021-02-10T11:45:56.958 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.938 7f08b161a700 1 -- 172.21.2.112:0/1044709887 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f08a80015b0 con 0x7f08b41b0af0 2021-02-10T11:45:56.958 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.938 7f08b161a700 1 -- 172.21.2.112:0/1044709887 >> v1:172.21.2.112:6789/0 conn(0x7f08b41b0af0 legacy=0x7f08b4108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:56.958 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.938 7f08b161a700 1 --2- 172.21.2.112:0/1044709887 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f089c003ac0 0x7f089c005ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:56.958 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.938 7f08b161a700 1 --2- 172.21.2.112:0/1044709887 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f089c04a760 0x7f089c04cc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:56.958 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.938 7f08b161a700 1 --2- 172.21.2.112:0/1044709887 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f089c04d520 0x7f089c04fa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:56.959 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.938 7f08b161a700 1 -- 172.21.2.112:0/1044709887 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f089c0502f0 con 0x7f089c04a760 2021-02-10T11:45:56.959 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.938 7f08b161a700 1 -- 172.21.2.112:0/1044709887 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f089c050510 con 0x7f089c04d520 2021-02-10T11:45:56.959 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.938 7f08b161a700 1 -- 172.21.2.112:0/1044709887 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f089c050730 con 0x7f089c003ac0 2021-02-10T11:45:56.959 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.938 7f08b1e1b700 1 --2- 172.21.2.112:0/1044709887 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f089c04a760 0x7f089c04cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:56.959 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.938 7f08b261c700 1 --2- 172.21.2.112:0/1044709887 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f089c003ac0 0x7f089c005ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:56.959 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.938 7f08b2e1d700 1 --2- 172.21.2.112:0/1044709887 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f089c04d520 0x7f089c04fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:56.959 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.939 7f08b1e1b700 1 -- 172.21.2.112:0/1044709887 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f089c003ac0 msgr2=0x7f089c005ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:56.959 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.939 7f08b1e1b700 1 --2- 172.21.2.112:0/1044709887 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f089c003ac0 0x7f089c005ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:56.959 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.939 7f08b1e1b700 1 -- 172.21.2.112:0/1044709887 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f089c04d520 msgr2=0x7f089c04fa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:56.959 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.939 7f08b1e1b700 1 --2- 172.21.2.112:0/1044709887 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f089c04d520 0x7f089c04fa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:56.959 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.939 7f08b1e1b700 1 -- 172.21.2.112:0/1044709887 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f089c0509d0 con 0x7f089c04a760 2021-02-10T11:45:56.959 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.939 7f08b1e1b700 1 --2- 172.21.2.112:0/1044709887 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f089c04a760 0x7f089c04cc80 secure :-1 s=READY pgs=2818 cs=0 l=1 rev1=1 rx=0x7f08a80017d0 tx=0x7f08a8007d30).ready entity=mon.0 client_cookie=fb426d719f38ccd2 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:56.959 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.939 7f08b161a700 1 -- 172.21.2.112:0/1044709887 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f08a80013a0 con 0x7f089c04a760 2021-02-10T11:45:56.960 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.939 7f08b161a700 1 -- 172.21.2.112:0/1044709887 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08a800a310 con 0x7f089c04a760 2021-02-10T11:45:56.960 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.939 7f08b161a700 1 -- 172.21.2.112:0/1044709887 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f08a800a7f0 con 0x7f089c04a760 2021-02-10T11:45:56.960 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.939 7f08b89a9700 1 -- 172.21.2.112:0/1044709887 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f089c04a760 msgr2=0x7f089c04cc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:56.960 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.939 7f08b89a9700 1 --2- 172.21.2.112:0/1044709887 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f089c04a760 0x7f089c04cc80 secure :-1 s=READY pgs=2818 cs=0 l=1 rev1=1 rx=0x7f08a80017d0 tx=0x7f08a8007d30).stop 2021-02-10T11:45:56.960 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.939 7f08b2e1d700 1 -- 172.21.2.112:0/1044709887 reap_dead start 2021-02-10T11:45:56.960 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.940 7f08b89a9700 1 -- 172.21.2.112:0/1044709887 shutdown_connections 2021-02-10T11:45:56.960 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.940 7f08b89a9700 1 -- 172.21.2.112:0/1044709887 >> 172.21.2.112:0/1044709887 conn(0x7f08b40ce430 msgr2=0x7f08b40d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:56.960 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.940 7f08b89a9700 1 -- 172.21.2.112:0/1044709887 shutdown_connections 2021-02-10T11:45:56.960 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.940 7f08b89a9700 1 -- 172.21.2.112:0/1044709887 wait complete. 2021-02-10T11:45:56.960 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.954 7f08b89a9700 1 Processor -- start 2021-02-10T11:45:56.960 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.955 7f08b89a9700 1 -- start start 2021-02-10T11:45:56.961 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.955 7f08b89a9700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f08b41a3570 0x7f08b41a3970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:56.962 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.955 7f08b89a9700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f08b409c9f0 0x7f08b409cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:56.962 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.955 7f08b89a9700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f08b409d410 0x7f08b4098170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:56.962 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.955 7f08b89a9700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08b409c470 con 0x7f08b41a3570 2021-02-10T11:45:56.962 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.955 7f08b89a9700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08b40b2840 con 0x7f08b409d410 2021-02-10T11:45:56.962 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.955 7f08b89a9700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08b40b2a20 con 0x7f08b409c9f0 2021-02-10T11:45:56.962 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.955 7f08b261c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f08b41a3570 0x7f08b41a3970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:56.962 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.955 7f08b261c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f08b41a3570 0x7f08b41a3970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:59176/0 (socket says 172.21.2.112:59176) 2021-02-10T11:45:56.962 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.955 7f08b261c700 1 -- 172.21.2.112:0/2123743531 learned_addr learned my addr 172.21.2.112:0/2123743531 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:45:56.962 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.956 7f08b261c700 1 -- 172.21.2.112:0/2123743531 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f08b409c9f0 msgr2=0x7f08b409cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:56.962 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.956 7f08b261c700 1 --2- 172.21.2.112:0/2123743531 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f08b409c9f0 0x7f08b409cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:56.962 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.956 7f08b261c700 1 -- 172.21.2.112:0/2123743531 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f08b409d410 msgr2=0x7f08b4098170 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:45:56.962 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.956 7f08b261c700 1 --2- 172.21.2.112:0/2123743531 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f08b409d410 0x7f08b4098170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:56.963 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.956 7f08b261c700 1 -- 172.21.2.112:0/2123743531 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08b409d880 con 0x7f08b41a3570 2021-02-10T11:45:56.963 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.956 7f08b261c700 1 --2- 172.21.2.112:0/2123743531 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f08b41a3570 0x7f08b41a3970 secure :-1 s=READY pgs=2819 cs=0 l=1 rev1=1 rx=0x7f08a4006f60 tx=0x7f08a400b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:56.963 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.956 7f08a2ffd700 1 -- 172.21.2.112:0/2123743531 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f08a40009b0 con 0x7f08b41a3570 2021-02-10T11:45:56.963 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.956 7f08a2ffd700 1 -- 172.21.2.112:0/2123743531 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08a400bd60 con 0x7f08b41a3570 2021-02-10T11:45:56.963 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.956 7f08a2ffd700 1 -- 172.21.2.112:0/2123743531 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f08a4002220 con 0x7f08b41a3570 2021-02-10T11:45:56.963 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.956 7f08b89a9700 1 -- 172.21.2.112:0/2123743531 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08b409d880 con 0x7f08b41a3570 2021-02-10T11:45:56.963 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.956 7f08b89a9700 1 -- 172.21.2.112:0/2123743531 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08b4098a80 con 0x7f08b41a3570 2021-02-10T11:45:56.963 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.957 7f08a2ffd700 1 -- 172.21.2.112:0/2123743531 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f08a4006b00 con 0x7f08b41a3570 2021-02-10T11:45:56.963 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.957 7f08a2ffd700 1 --2- 172.21.2.112:0/2123743531 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f08980186c0 0x7f089801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:56.963 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.957 7f08a2ffd700 1 -- 172.21.2.112:0/2123743531 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f08a4032540 con 0x7f08b41a3570 2021-02-10T11:45:56.963 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.957 7f08b1e1b700 1 --2- 172.21.2.112:0/2123743531 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f08980186c0 0x7f089801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:56.963 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.957 7f08b1e1b700 1 --2- 172.21.2.112:0/2123743531 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f08980186c0 0x7f089801abc0 crc :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:56.989 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.987 7f08b89a9700 1 -- 172.21.2.112:0/2123743531 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08b4082a70 con 0x7f08b41a3570 2021-02-10T11:45:56.991 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:56.989 7f08a2ffd700 1 -- 172.21.2.112:0/2123743531 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f08a4000940 con 0x7f08b41a3570 2021-02-10T11:45:57.119 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:57.117 7f08b89a9700 1 -- 172.21.2.112:0/2123743531 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume snapshot rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000486841", "target": ["mgr", ""], "snap_name": "snapshot_0000000000014182"}) v1 -- 0x7f08b409c470 con 0x7f08980186c0 2021-02-10T11:45:57.123 INFO:tasks.ceph.mgr.x.gibba020.stderr:2021-02-10 11:45:57.120 7f6382eee700 -1 mgr.server reply reply (2) No such file or directory snapshot 'snapshot_0000000000014182' does not exist 2021-02-10T11:45:57.123 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:57.121 7f08a2ffd700 1 -- 172.21.2.112:0/2123743531 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: -2 snapshot 'snapshot_0000000000014182' does not exist) v1 ==== 59+0+0 (crc 0 0 0) 0x7f08a8007610 con 0x7f08980186c0 2021-02-10T11:45:57.127 INFO:teuthology.orchestra.run.gibba012.stderr:Error ENOENT: snapshot 'snapshot_0000000000014182' does not exist 2021-02-10T11:45:57.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:57.127 7f08b89a9700 1 -- 172.21.2.112:0/2123743531 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f08980186c0 msgr2=0x7f089801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:57.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:57.127 7f08b89a9700 1 --2- 172.21.2.112:0/2123743531 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f08980186c0 0x7f089801abc0 crc :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:57.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:57.127 7f08b89a9700 1 -- 172.21.2.112:0/2123743531 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f08b41a3570 msgr2=0x7f08b41a3970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:57.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:57.127 7f08b89a9700 1 --2- 172.21.2.112:0/2123743531 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f08b41a3570 0x7f08b41a3970 secure :-1 s=READY pgs=2819 cs=0 l=1 rev1=1 rx=0x7f08a4006f60 tx=0x7f08a400b970).stop 2021-02-10T11:45:57.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:57.127 7f08b89a9700 1 -- 172.21.2.112:0/2123743531 shutdown_connections 2021-02-10T11:45:57.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:57.127 7f08b89a9700 1 --2- 172.21.2.112:0/2123743531 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f08980186c0 0x7f089801abc0 unknown :-1 s=CLOSED pgs=1281 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:57.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:57.127 7f08b89a9700 1 --2- 172.21.2.112:0/2123743531 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f08b409d410 0x7f08b4098170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:57.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:57.127 7f08b89a9700 1 --2- 172.21.2.112:0/2123743531 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f08b409c9f0 0x7f08b409cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:57.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:57.127 7f08b89a9700 1 --2- 172.21.2.112:0/2123743531 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f08b41a3570 0x7f08b41a3970 unknown :-1 s=CLOSED pgs=2819 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:57.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:57.127 7f08b89a9700 1 -- 172.21.2.112:0/2123743531 >> 172.21.2.112:0/2123743531 conn(0x7f08b40c2cd0 msgr2=0x7f08b40b5950 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:57.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:57.127 7f08b89a9700 1 -- 172.21.2.112:0/2123743531 shutdown_connections 2021-02-10T11:45:57.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:57.127 7f08b89a9700 1 -- 172.21.2.112:0/2123743531 wait complete. 2021-02-10T11:45:57.137 DEBUG:teuthology.orchestra.run:got remote process result: 2 2021-02-10T11:45:57.137 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000486841 2021-02-10T11:45:57.570 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:57.551 7f5b0e82a700 1 Processor -- start 2021-02-10T11:45:57.570 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:57.552 7f5b0e82a700 1 -- start start 2021-02-10T11:45:57.570 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:57.552 7f5b0e82a700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f5b080d1870 0x7f5b081b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:57.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:57.552 7f5b0e82a700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f5b080cff70 0x7f5b08109130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:57.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:57.552 7f5b0e82a700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b080fe1c0 con 0x7f5b081b0ae0 2021-02-10T11:45:57.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:57.552 7f5b0e82a700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b08109ad0 con 0x7f5b080d1870 2021-02-10T11:45:57.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:57.552 7f5b0e82a700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b08109750 con 0x7f5b080cff70 2021-02-10T11:45:57.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:57.552 7f5b07fff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f5b080d1870 0x7f5b081b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:57.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:57.552 7f5b07fff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f5b080d1870 0x7f5b081b04c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:59184/0 (socket says 172.21.2.112:59184) 2021-02-10T11:45:57.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:57.552 7f5b07fff700 1 -- 172.21.2.112:0/2047311022 learned_addr learned my addr 172.21.2.112:0/2047311022 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:45:57.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:57.553 7f5b07fff700 1 -- 172.21.2.112:0/2047311022 >> v2:172.21.2.120:3300/0 conn(0x7f5b080cff70 msgr2=0x7f5b08109130 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:45:57.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:57.553 7f5b07fff700 1 --2- 172.21.2.112:0/2047311022 >> v2:172.21.2.120:3300/0 conn(0x7f5b080cff70 0x7f5b08109130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:57.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:57.553 7f5b07fff700 1 -- 172.21.2.112:0/2047311022 >> v1:172.21.2.120:6789/0 conn(0x7f5b081b0ae0 legacy=0x7f5b08108940 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:45:57.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:57.553 7f5b07fff700 1 -- 172.21.2.112:0/2047311022 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b08088070 con 0x7f5b080d1870 2021-02-10T11:45:57.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:57.553 7f5b07fff700 1 --2- 172.21.2.112:0/2047311022 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5b080d1870 0x7f5b081b04c0 secure :-1 s=READY pgs=2820 cs=0 l=1 rev1=1 rx=0x7f5af8006700 tx=0x7f5af8009b80).ready entity=mon.0 client_cookie=433dda7c16b0ee1 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:57.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:57.553 7f5b06ffd700 1 -- 172.21.2.112:0/2047311022 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5af800c0c0 con 0x7f5b080d1870 2021-02-10T11:45:57.572 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:57.553 7f5b06ffd700 1 -- 172.21.2.112:0/2047311022 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5af800c340 con 0x7f5b080d1870 2021-02-10T11:45:57.572 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:57.553 7f5b06ffd700 1 -- 172.21.2.112:0/2047311022 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5af800c860 con 0x7f5b080d1870 2021-02-10T11:45:57.572 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:57.553 7f5b0e82a700 1 -- 172.21.2.112:0/2047311022 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5b080d1870 msgr2=0x7f5b081b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:57.572 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:57.553 7f5b0e82a700 1 --2- 172.21.2.112:0/2047311022 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5b080d1870 0x7f5b081b04c0 secure :-1 s=READY pgs=2820 cs=0 l=1 rev1=1 rx=0x7f5af8006700 tx=0x7f5af8009b80).stop 2021-02-10T11:45:57.572 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:57.554 7f5b0e82a700 1 -- 172.21.2.112:0/2047311022 shutdown_connections 2021-02-10T11:45:57.572 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:57.554 7f5b0e82a700 1 --2- 172.21.2.112:0/2047311022 >> v2:172.21.2.120:3300/0 conn(0x7f5b080cff70 0x7f5b08109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:57.572 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:57.554 7f5b0e82a700 1 --2- 172.21.2.112:0/2047311022 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5b080d1870 0x7f5b081b04c0 unknown :-1 s=CLOSED pgs=2820 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:57.572 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:57.554 7f5b0e82a700 1 -- 172.21.2.112:0/2047311022 >> 172.21.2.112:0/2047311022 conn(0x7f5b080ce420 msgr2=0x7f5b080d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:57.572 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:57.554 7f5b0e82a700 1 -- 172.21.2.112:0/2047311022 shutdown_connections 2021-02-10T11:45:57.572 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:57.554 7f5b0e82a700 1 -- 172.21.2.112:0/2047311022 wait complete. 2021-02-10T11:45:57.574 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:57.569 7f5b0e82a700 1 Processor -- start 2021-02-10T11:45:57.574 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:57.569 7f5b0e82a700 1 -- start start 2021-02-10T11:45:57.574 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:57.569 7f5b0e82a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5b080c3820 0x7f5b08084cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:57.574 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:57.569 7f5b0e82a700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5b080a0290 0x7f5b080852d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:57.574 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:57.569 7f5b0e82a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5b08086270 0x7f5b08086670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:57.574 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:57.569 7f5b0e82a700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b080c0700 con 0x7f5b08086270 2021-02-10T11:45:57.574 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:57.569 7f5b0e82a700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b080858f0 con 0x7f5b080a0290 2021-02-10T11:45:57.574 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:57.569 7f5b0e82a700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b08085b10 con 0x7f5b080c3820 2021-02-10T11:45:57.574 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:57.569 7f5b0ce47700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5b08086270 0x7f5b08086670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:57.574 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:57.569 7f5b0ce47700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5b08086270 0x7f5b08086670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:59192/0 (socket says 172.21.2.112:59192) 2021-02-10T11:45:57.574 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:57.569 7f5b0ce47700 1 -- 172.21.2.112:0/214950692 learned_addr learned my addr 172.21.2.112:0/214950692 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:45:57.575 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:57.570 7f5b0ce47700 1 -- 172.21.2.112:0/214950692 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5b080c3820 msgr2=0x7f5b08084cb0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:45:57.575 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:57.570 7f5b0ce47700 1 --2- 172.21.2.112:0/214950692 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5b080c3820 0x7f5b08084cb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:57.575 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:57.570 7f5b0ce47700 1 -- 172.21.2.112:0/214950692 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5b080a0290 msgr2=0x7f5b080852d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:45:57.575 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:57.570 7f5b0ce47700 1 --2- 172.21.2.112:0/214950692 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5b080a0290 0x7f5b080852d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:57.575 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:57.570 7f5b0ce47700 1 -- 172.21.2.112:0/214950692 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b08086c90 con 0x7f5b08086270 2021-02-10T11:45:57.575 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:57.570 7f5b0ce47700 1 --2- 172.21.2.112:0/214950692 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5b08086270 0x7f5b08086670 secure :-1 s=READY pgs=2821 cs=0 l=1 rev1=1 rx=0x7f5b00009960 tx=0x7f5b0000bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:57.575 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:57.570 7f5b04ff9700 1 -- 172.21.2.112:0/214950692 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5b00003030 con 0x7f5b08086270 2021-02-10T11:45:57.575 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:57.570 7f5b04ff9700 1 -- 172.21.2.112:0/214950692 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b00001310 con 0x7f5b08086270 2021-02-10T11:45:57.575 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:57.570 7f5b04ff9700 1 -- 172.21.2.112:0/214950692 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5b00001830 con 0x7f5b08086270 2021-02-10T11:45:57.575 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:57.570 7f5b0e82a700 1 -- 172.21.2.112:0/214950692 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b08086c90 con 0x7f5b08086270 2021-02-10T11:45:57.575 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:57.570 7f5b0e82a700 1 -- 172.21.2.112:0/214950692 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b081a2f00 con 0x7f5b08086270 2021-02-10T11:45:57.575 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:57.571 7f5b04ff9700 1 -- 172.21.2.112:0/214950692 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5b000014f0 con 0x7f5b08086270 2021-02-10T11:45:57.575 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:57.571 7f5b04ff9700 1 --2- 172.21.2.112:0/214950692 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5ae80186c0 0x7f5ae801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:57.576 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:57.571 7f5b04ff9700 1 -- 172.21.2.112:0/214950692 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f5b00032960 con 0x7f5b08086270 2021-02-10T11:45:57.576 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:57.572 7f5b07fff700 1 --2- 172.21.2.112:0/214950692 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5ae80186c0 0x7f5ae801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:57.576 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:57.572 7f5b07fff700 1 --2- 172.21.2.112:0/214950692 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5ae80186c0 0x7f5ae801abc0 crc :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:57.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:57.601 7f5b0e82a700 1 -- 172.21.2.112:0/214950692 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b08082a70 con 0x7f5b08086270 2021-02-10T11:45:57.606 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:57.604 7f5b04ff9700 1 -- 172.21.2.112:0/214950692 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5b00000f40 con 0x7f5b08086270 2021-02-10T11:45:57.735 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:57.733 7f5b0e82a700 1 -- 172.21.2.112:0/214950692 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000486841", "target": ["mgr", ""]}) v1 -- 0x7f5b080c0700 con 0x7f5ae80186c0 2021-02-10T11:45:57.744 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:57.742 7f5b04ff9700 1 -- 172.21.2.112:0/214950692 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f5af8006500 con 0x7f5ae80186c0 2021-02-10T11:45:57.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:57.749 7f5b0e82a700 1 -- 172.21.2.112:0/214950692 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5ae80186c0 msgr2=0x7f5ae801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:57.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:57.749 7f5b0e82a700 1 --2- 172.21.2.112:0/214950692 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5ae80186c0 0x7f5ae801abc0 crc :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:57.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:57.750 7f5b0e82a700 1 -- 172.21.2.112:0/214950692 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5b08086270 msgr2=0x7f5b08086670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:57.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:57.750 7f5b0e82a700 1 --2- 172.21.2.112:0/214950692 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5b08086270 0x7f5b08086670 secure :-1 s=READY pgs=2821 cs=0 l=1 rev1=1 rx=0x7f5b00009960 tx=0x7f5b0000bcf0).stop 2021-02-10T11:45:57.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:57.750 7f5b0e82a700 1 -- 172.21.2.112:0/214950692 shutdown_connections 2021-02-10T11:45:57.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:57.750 7f5b0e82a700 1 --2- 172.21.2.112:0/214950692 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5ae80186c0 0x7f5ae801abc0 unknown :-1 s=CLOSED pgs=1282 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:57.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:57.750 7f5b0e82a700 1 --2- 172.21.2.112:0/214950692 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5b080a0290 0x7f5b080852d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:57.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:57.750 7f5b0e82a700 1 --2- 172.21.2.112:0/214950692 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5b08086270 0x7f5b08086670 unknown :-1 s=CLOSED pgs=2821 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:57.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:57.750 7f5b0e82a700 1 --2- 172.21.2.112:0/214950692 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5b080c3820 0x7f5b08084cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:57.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:57.750 7f5b0e82a700 1 -- 172.21.2.112:0/214950692 >> 172.21.2.112:0/214950692 conn(0x7f5b080872c0 msgr2=0x7f5b08095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:57.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:57.750 7f5b0e82a700 1 -- 172.21.2.112:0/214950692 shutdown_connections 2021-02-10T11:45:57.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:45:57.750 7f5b0e82a700 1 -- 172.21.2.112:0/214950692 wait complete. 2021-02-10T11:45:57.760 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo getfattr --only-values -n ceph.dir.entries /home/ubuntu/cephtest/mnt.0/./volumes/_deleting 2021-02-10T11:45:58.110 INFO:teuthology.orchestra.run.gibba012.stderr:getfattr: Removing leading '/' from absolute path names 2021-02-10T11:45:58.111 INFO:teuthology.orchestra.run.gibba012.stdout:0 2021-02-10T11:45:58.111 DEBUG:tasks.cephfs.mount:Directory ./volumes/_deleting seen empty from 0 after 0s 2021-02-10T11:45:58.111 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:45:58.111 DEBUG:teuthology.orchestra.run.gibba012:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:45:58.457 DEBUG:teuthology.orchestra.run.gibba020:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:45:58.801 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:45:58.801 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:45:58.801 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:45:58.801 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:45:59.137 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:45:59.137 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:45:59.137 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba012.front.sepia.ceph.com... 2021-02-10T11:45:59.137 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:45:59.137 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:45:59.493 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba012.stderr:ceph-fuse[180092]: fuse finished with error 0 and tester_r 0 2021-02-10T11:45:59.494 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:45:59.495 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:45:59.495 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:45:59.830 INFO:teuthology.orchestra.run.gibba012.stdout:ext2/ext3 2021-02-10T11:45:59.830 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:45:59.830 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:45:59.831 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:46:00.166 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:46:00.167 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:46:00.167 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:46:00.503 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:46:00.503 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:46:00.503 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba012.front.sepia.ceph.com... 2021-02-10T11:46:00.504 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:46:00.504 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:46:00.857 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba012.stderr:ceph-fuse[180732]: fuse finished with error 0 and tester_r 0 2021-02-10T11:46:00.859 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:46:00.859 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:46:00.859 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:46:01.194 INFO:teuthology.orchestra.run.gibba012.stdout:ext2/ext3 2021-02-10T11:46:01.194 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:46:01.195 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:46:01.195 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:46:01.530 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:46:01.530 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:46:01.530 DEBUG:teuthology.orchestra.run.gibba020:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:46:01.865 INFO:teuthology.orchestra.run.gibba020.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:46:01.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:46:01.865 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:46:01.865 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:46:01.865 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:46:02.199 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:46:02.199 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:46:02.199 DEBUG:teuthology.orchestra.run.gibba020:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:46:02.533 INFO:teuthology.orchestra.run.gibba020.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:46:02.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:46:02.534 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:46:02.534 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:46:02.534 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:46:02.868 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config rm mon mon_allow_pool_delete 2021-02-10T11:46:03.303 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.283 7fd5e5a31700 1 Processor -- start 2021-02-10T11:46:03.303 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.283 7fd5e5a31700 1 -- start start 2021-02-10T11:46:03.303 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.283 7fd5e5a31700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fd5e00d1870 0x7fd5e01b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:03.303 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.283 7fd5e5a31700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd5e01b0a10 0x7fd5e00cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:03.303 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.283 7fd5e5a31700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5e00f0b10 con 0x7fd5e00d1c70 2021-02-10T11:46:03.303 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.283 7fd5e5a31700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5e00e5ed0 con 0x7fd5e00d1870 2021-02-10T11:46:03.303 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.283 7fd5e5a31700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5e00f0650 con 0x7fd5e01b0a10 2021-02-10T11:46:03.303 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.284 7fd5dffff700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fd5e00d1c70 0x7fd5e00cf950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:58658/0 (socket says 172.21.2.112:58658) 2021-02-10T11:46:03.303 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.284 7fd5dffff700 1 -- 172.21.2.112:0/3672363407 learned_addr learned my addr 172.21.2.112:0/3672363407 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:46:03.303 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.284 7fd5de7fc700 1 -- 172.21.2.112:0/3672363407 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2623376866 0 0) 0x7fd5d8001500 con 0x7fd5e00d1c70 2021-02-10T11:46:03.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.284 7fd5de7fc700 1 -- 172.21.2.112:0/3672363407 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5cc002fe0 con 0x7fd5e00d1c70 2021-02-10T11:46:03.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.284 7fd5df7fe700 1 --2- 172.21.2.112:0/3672363407 >> v2:172.21.2.120:3300/0 conn(0x7fd5e00d1870 0x7fd5e01b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:03.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.284 7fd5deffd700 1 --2- 172.21.2.112:0/3672363407 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd5e01b0a10 0x7fd5e00cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:03.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.284 7fd5de7fc700 1 -- 172.21.2.112:0/3672363407 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 689467848 0 0) 0x7fd5d8001460 con 0x7fd5e00d1c70 2021-02-10T11:46:03.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.284 7fd5de7fc700 1 -- 172.21.2.112:0/3672363407 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd5e01b0a10 msgr2=0x7fd5e00cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:46:03.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.284 7fd5de7fc700 1 --2- 172.21.2.112:0/3672363407 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd5e01b0a10 0x7fd5e00cf330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:03.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.284 7fd5de7fc700 1 -- 172.21.2.112:0/3672363407 >> v2:172.21.2.120:3300/0 conn(0x7fd5e00d1870 msgr2=0x7fd5e01b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:46:03.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.284 7fd5de7fc700 1 --2- 172.21.2.112:0/3672363407 >> v2:172.21.2.120:3300/0 conn(0x7fd5e00d1870 0x7fd5e01b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:03.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.284 7fd5de7fc700 1 -- 172.21.2.112:0/3672363407 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5e00f0870 con 0x7fd5e00d1c70 2021-02-10T11:46:03.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.284 7fd5de7fc700 1 -- 172.21.2.112:0/3672363407 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7fd5d8001980 con 0x7fd5e00d1c70 2021-02-10T11:46:03.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.284 7fd5de7fc700 1 -- 172.21.2.112:0/3672363407 >> v1:172.21.2.112:6789/0 conn(0x7fd5e00d1c70 legacy=0x7fd5e00cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:03.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.284 7fd5de7fc700 1 --2- 172.21.2.112:0/3672363407 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd5cc003b90 0x7fd5cc006000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:03.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.284 7fd5de7fc700 1 --2- 172.21.2.112:0/3672363407 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd5cc006880 0x7fd5cc008da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:03.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.284 7fd5de7fc700 1 --2- 172.21.2.112:0/3672363407 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd5cc009640 0x7fd5cc00bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:03.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.284 7fd5de7fc700 1 -- 172.21.2.112:0/3672363407 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5cc00c410 con 0x7fd5cc006880 2021-02-10T11:46:03.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.285 7fd5de7fc700 1 -- 172.21.2.112:0/3672363407 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5cc050600 con 0x7fd5cc009640 2021-02-10T11:46:03.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.285 7fd5de7fc700 1 -- 172.21.2.112:0/3672363407 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5cc050820 con 0x7fd5cc003b90 2021-02-10T11:46:03.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.285 7fd5deffd700 1 --2- 172.21.2.112:0/3672363407 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd5e01b0a10 0x7fd5e00cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:46:03.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.285 7fd5df7fe700 1 --2- 172.21.2.112:0/3672363407 >> v2:172.21.2.120:3300/0 conn(0x7fd5e00d1870 0x7fd5e01b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:46:03.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.285 7fd5deffd700 1 --2- 172.21.2.112:0/3672363407 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd5cc006880 0x7fd5cc008da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:03.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.285 7fd5df7fe700 1 --2- 172.21.2.112:0/3672363407 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd5cc003b90 0x7fd5cc006000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:03.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.285 7fd5dffff700 1 --2- 172.21.2.112:0/3672363407 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd5cc009640 0x7fd5cc00bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:03.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.285 7fd5deffd700 1 -- 172.21.2.112:0/3672363407 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd5cc003b90 msgr2=0x7fd5cc006000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:46:03.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.285 7fd5deffd700 1 --2- 172.21.2.112:0/3672363407 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd5cc003b90 0x7fd5cc006000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:03.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.285 7fd5deffd700 1 -- 172.21.2.112:0/3672363407 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd5cc009640 msgr2=0x7fd5cc00bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:46:03.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.285 7fd5deffd700 1 --2- 172.21.2.112:0/3672363407 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd5cc009640 0x7fd5cc00bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:03.306 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.285 7fd5deffd700 1 -- 172.21.2.112:0/3672363407 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5cc050ac0 con 0x7fd5cc006880 2021-02-10T11:46:03.306 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.285 7fd5df7fe700 1 --2- 172.21.2.112:0/3672363407 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd5cc003b90 0x7fd5cc006000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:46:03.306 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.285 7fd5dffff700 1 --2- 172.21.2.112:0/3672363407 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd5cc009640 0x7fd5cc00bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:46:03.306 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.285 7fd5deffd700 1 --2- 172.21.2.112:0/3672363407 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd5cc006880 0x7fd5cc008da0 secure :-1 s=READY pgs=2823 cs=0 l=1 rev1=1 rx=0x7fd5d0006890 tx=0x7fd5d0009aa0).ready entity=mon.0 client_cookie=89bbb7105ef2b518 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:03.306 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.285 7fd5de7fc700 1 -- 172.21.2.112:0/3672363407 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd5d000c0c0 con 0x7fd5cc006880 2021-02-10T11:46:03.306 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.285 7fd5de7fc700 1 -- 172.21.2.112:0/3672363407 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5d000c3c0 con 0x7fd5cc006880 2021-02-10T11:46:03.306 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.285 7fd5de7fc700 1 -- 172.21.2.112:0/3672363407 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd5d000c820 con 0x7fd5cc006880 2021-02-10T11:46:03.306 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.285 7fd5e5a31700 1 -- 172.21.2.112:0/3672363407 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd5cc006880 msgr2=0x7fd5cc008da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:03.306 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.285 7fd5e5a31700 1 --2- 172.21.2.112:0/3672363407 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd5cc006880 0x7fd5cc008da0 secure :-1 s=READY pgs=2823 cs=0 l=1 rev1=1 rx=0x7fd5d0006890 tx=0x7fd5d0009aa0).stop 2021-02-10T11:46:03.306 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.285 7fd5dffff700 1 -- 172.21.2.112:0/3672363407 reap_dead start 2021-02-10T11:46:03.306 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.286 7fd5e5a31700 1 -- 172.21.2.112:0/3672363407 shutdown_connections 2021-02-10T11:46:03.306 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.286 7fd5e5a31700 1 -- 172.21.2.112:0/3672363407 >> 172.21.2.112:0/3672363407 conn(0x7fd5e00ce450 msgr2=0x7fd5e00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:03.307 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.286 7fd5e5a31700 1 -- 172.21.2.112:0/3672363407 shutdown_connections 2021-02-10T11:46:03.307 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.286 7fd5e5a31700 1 -- 172.21.2.112:0/3672363407 wait complete. 2021-02-10T11:46:03.307 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.301 7fd5e5a31700 1 Processor -- start 2021-02-10T11:46:03.307 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.301 7fd5e5a31700 1 -- start start 2021-02-10T11:46:03.308 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.301 7fd5e5a31700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd5e0094310 0x7fd5e00ebe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:03.308 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.301 7fd5e5a31700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd5e0094710 0x7fd5e00bca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:03.308 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.301 7fd5e5a31700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd5e00ec490 0x7fd5e00bd050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:03.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.301 7fd5e5a31700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5e0093d90 con 0x7fd5e0094710 2021-02-10T11:46:03.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.301 7fd5e5a31700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5e00bd670 con 0x7fd5e0094310 2021-02-10T11:46:03.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.301 7fd5e5a31700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5e00939f0 con 0x7fd5e00ec490 2021-02-10T11:46:03.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.301 7fd5deffd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd5e0094710 0x7fd5e00bca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:03.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.301 7fd5deffd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd5e0094710 0x7fd5e00bca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:59208/0 (socket says 172.21.2.112:59208) 2021-02-10T11:46:03.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.301 7fd5deffd700 1 -- 172.21.2.112:0/587980567 learned_addr learned my addr 172.21.2.112:0/587980567 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:46:03.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.302 7fd5dffff700 1 --2- 172.21.2.112:0/587980567 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd5e00ec490 0x7fd5e00bd050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:03.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.302 7fd5df7fe700 1 --2- 172.21.2.112:0/587980567 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd5e0094310 0x7fd5e00ebe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:03.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.302 7fd5deffd700 1 -- 172.21.2.112:0/587980567 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd5e00ec490 msgr2=0x7fd5e00bd050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:03.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.302 7fd5deffd700 1 --2- 172.21.2.112:0/587980567 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd5e00ec490 0x7fd5e00bd050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:03.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.302 7fd5deffd700 1 -- 172.21.2.112:0/587980567 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd5e0094310 msgr2=0x7fd5e00ebe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:03.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.302 7fd5deffd700 1 --2- 172.21.2.112:0/587980567 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd5e0094310 0x7fd5e00ebe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:03.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.302 7fd5deffd700 1 -- 172.21.2.112:0/587980567 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5e00bda80 con 0x7fd5e0094710 2021-02-10T11:46:03.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.302 7fd5df7fe700 1 --2- 172.21.2.112:0/587980567 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd5e0094310 0x7fd5e00ebe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:46:03.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.302 7fd5deffd700 1 --2- 172.21.2.112:0/587980567 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd5e0094710 0x7fd5e00bca30 secure :-1 s=READY pgs=2824 cs=0 l=1 rev1=1 rx=0x7fd5d0006f40 tx=0x7fd5d000b910).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:03.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.302 7fd5d7fff700 1 -- 172.21.2.112:0/587980567 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd5d000bd00 con 0x7fd5e0094710 2021-02-10T11:46:03.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.302 7fd5d7fff700 1 -- 172.21.2.112:0/587980567 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5d0006ba0 con 0x7fd5e0094710 2021-02-10T11:46:03.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.302 7fd5d7fff700 1 -- 172.21.2.112:0/587980567 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd5d00022b0 con 0x7fd5e0094710 2021-02-10T11:46:03.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.302 7fd5e5a31700 1 -- 172.21.2.112:0/587980567 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5e00bda80 con 0x7fd5e0094710 2021-02-10T11:46:03.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.302 7fd5e5a31700 1 -- 172.21.2.112:0/587980567 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5e00bda80 con 0x7fd5e0094710 2021-02-10T11:46:03.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.303 7fd5d7fff700 1 -- 172.21.2.112:0/587980567 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd5d000a000 con 0x7fd5e0094710 2021-02-10T11:46:03.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.303 7fd5d7fff700 1 --2- 172.21.2.112:0/587980567 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd5c40186c0 0x7fd5c401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:03.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.303 7fd5d7fff700 1 -- 172.21.2.112:0/587980567 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 5645+0+0 (secure 0 0 0) 0x7fd5d0032620 con 0x7fd5e0094710 2021-02-10T11:46:03.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.303 7fd5df7fe700 1 --2- 172.21.2.112:0/587980567 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd5c40186c0 0x7fd5c401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:03.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.304 7fd5df7fe700 1 --2- 172.21.2.112:0/587980567 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd5c40186c0 0x7fd5c401abc0 crc :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:03.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.333 7fd5e5a31700 1 -- 172.21.2.112:0/587980567 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5e00ebc40 con 0x7fd5e0094710 2021-02-10T11:46:03.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.336 7fd5d7fff700 1 -- 172.21.2.112:0/587980567 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd5d001f500 con 0x7fd5e0094710 2021-02-10T11:46:03.486 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.484 7fd5e5a31700 1 -- 172.21.2.112:0/587980567 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fd5e00f0b10 con 0x7fd5e0094710 2021-02-10T11:46:03.489 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.488 7fd5d7fff700 1 -- 172.21.2.112:0/587980567 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v41) v1 ==== 104+0+0 (secure 0 0 0) 0x7fd5d000a000 con 0x7fd5e0094710 2021-02-10T11:46:03.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.492 7fd5e5a31700 1 -- 172.21.2.112:0/587980567 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd5c40186c0 msgr2=0x7fd5c401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:03.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.492 7fd5e5a31700 1 --2- 172.21.2.112:0/587980567 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd5c40186c0 0x7fd5c401abc0 crc :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:03.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.492 7fd5e5a31700 1 -- 172.21.2.112:0/587980567 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd5e0094710 msgr2=0x7fd5e00bca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:03.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.492 7fd5e5a31700 1 --2- 172.21.2.112:0/587980567 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd5e0094710 0x7fd5e00bca30 secure :-1 s=READY pgs=2824 cs=0 l=1 rev1=1 rx=0x7fd5d0006f40 tx=0x7fd5d000b910).stop 2021-02-10T11:46:03.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.492 7fd5e5a31700 1 -- 172.21.2.112:0/587980567 shutdown_connections 2021-02-10T11:46:03.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.492 7fd5e5a31700 1 --2- 172.21.2.112:0/587980567 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd5c40186c0 0x7fd5c401abc0 unknown :-1 s=CLOSED pgs=1283 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:03.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.492 7fd5e5a31700 1 --2- 172.21.2.112:0/587980567 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd5e00ec490 0x7fd5e00bd050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:03.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.492 7fd5e5a31700 1 --2- 172.21.2.112:0/587980567 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd5e0094710 0x7fd5e00bca30 unknown :-1 s=CLOSED pgs=2824 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:03.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.492 7fd5e5a31700 1 --2- 172.21.2.112:0/587980567 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd5e0094310 0x7fd5e00ebe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:03.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.492 7fd5e5a31700 1 -- 172.21.2.112:0/587980567 >> 172.21.2.112:0/587980567 conn(0x7fd5e008fa10 msgr2=0x7fd5e00bfa60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:03.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.493 7fd5e5a31700 1 -- 172.21.2.112:0/587980567 shutdown_connections 2021-02-10T11:46:03.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.493 7fd5e5a31700 1 -- 172.21.2.112:0/587980567 wait complete. 2021-02-10T11:46:03.503 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestVolumes.test_nonexistent_subvolume_snapshot_rm' 2021-02-10T11:46:03.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.914 7fcb24963700 1 Processor -- start 2021-02-10T11:46:03.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.914 7fcb24963700 1 -- start start 2021-02-10T11:46:03.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.915 7fcb24963700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fcb200d1870 0x7fcb201b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:03.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.915 7fcb24963700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb200f0b10 con 0x7fcb200f0650 2021-02-10T11:46:03.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.915 7fcb24963700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb200d0310 con 0x7fcb201b0a10 2021-02-10T11:46:03.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.915 7fcb24963700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb200d0530 con 0x7fcb200d1870 2021-02-10T11:46:03.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.915 7fcb1ee1d700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fcb200f0650 0x7fcb200cfb20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:58676/0 (socket says 172.21.2.112:58676) 2021-02-10T11:46:03.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.915 7fcb1ee1d700 1 -- 172.21.2.112:0/4152816742 learned_addr learned my addr 172.21.2.112:0/4152816742 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:46:03.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.915 7fcb1d61a700 1 -- 172.21.2.112:0/4152816742 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2591150721 0 0) 0x7fcb18001500 con 0x7fcb200f0650 2021-02-10T11:46:03.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.915 7fcb1e61c700 1 --2- 172.21.2.112:0/4152816742 >> v2:172.21.2.120:3300/0 conn(0x7fcb200d1870 0x7fcb201b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:03.935 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.915 7fcb1d61a700 1 -- 172.21.2.112:0/4152816742 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb0c002fe0 con 0x7fcb200f0650 2021-02-10T11:46:03.935 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.915 7fcb1d61a700 1 -- 172.21.2.112:0/4152816742 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2201949940 0 0) 0x7fcb18001460 con 0x7fcb200f0650 2021-02-10T11:46:03.935 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.915 7fcb1d61a700 1 -- 172.21.2.112:0/4152816742 >> v2:172.21.2.120:3300/0 conn(0x7fcb200d1870 msgr2=0x7fcb201b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:46:03.935 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.915 7fcb1d61a700 1 --2- 172.21.2.112:0/4152816742 >> v2:172.21.2.120:3300/0 conn(0x7fcb200d1870 0x7fcb201b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:03.935 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.915 7fcb1d61a700 1 -- 172.21.2.112:0/4152816742 >> v1:172.21.2.120:6789/0 conn(0x7fcb201b0a10 legacy=0x7fcb200cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:03.935 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.915 7fcb1d61a700 1 -- 172.21.2.112:0/4152816742 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb200d1c70 con 0x7fcb200f0650 2021-02-10T11:46:03.935 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.915 7fcb1d61a700 1 -- 172.21.2.112:0/4152816742 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7fcb18001980 con 0x7fcb200f0650 2021-02-10T11:46:03.935 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.915 7fcb1d61a700 1 -- 172.21.2.112:0/4152816742 >> v1:172.21.2.112:6789/0 conn(0x7fcb200f0650 legacy=0x7fcb200cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:03.935 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.915 7fcb1d61a700 1 --2- 172.21.2.112:0/4152816742 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcb0c003a40 0x7fcb0c005ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:03.935 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.915 7fcb1d61a700 1 --2- 172.21.2.112:0/4152816742 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcb0c006770 0x7fcb0c008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:03.935 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.915 7fcb1d61a700 1 --2- 172.21.2.112:0/4152816742 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcb0c009590 0x7fcb0c00bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:03.935 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.915 7fcb1d61a700 1 -- 172.21.2.112:0/4152816742 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb0c00c300 con 0x7fcb0c009590 2021-02-10T11:46:03.935 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.915 7fcb1d61a700 1 -- 172.21.2.112:0/4152816742 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb0c00c520 con 0x7fcb0c006770 2021-02-10T11:46:03.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.916 7fcb1d61a700 1 -- 172.21.2.112:0/4152816742 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb0c050750 con 0x7fcb0c003a40 2021-02-10T11:46:03.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.916 7fcb1ee1d700 1 --2- 172.21.2.112:0/4152816742 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcb0c009590 0x7fcb0c00bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:03.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.916 7fcb1de1b700 1 --2- 172.21.2.112:0/4152816742 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcb0c006770 0x7fcb0c008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:03.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.916 7fcb1e61c700 1 --2- 172.21.2.112:0/4152816742 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcb0c003a40 0x7fcb0c005ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:03.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.916 7fcb1ee1d700 1 -- 172.21.2.112:0/4152816742 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcb0c003a40 msgr2=0x7fcb0c005ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:46:03.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.916 7fcb1ee1d700 1 --2- 172.21.2.112:0/4152816742 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcb0c003a40 0x7fcb0c005ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:03.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.916 7fcb1ee1d700 1 -- 172.21.2.112:0/4152816742 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcb0c006770 msgr2=0x7fcb0c008c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:46:03.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.916 7fcb1ee1d700 1 --2- 172.21.2.112:0/4152816742 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcb0c006770 0x7fcb0c008c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:03.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.916 7fcb1ee1d700 1 -- 172.21.2.112:0/4152816742 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb0c0509b0 con 0x7fcb0c009590 2021-02-10T11:46:03.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.916 7fcb1de1b700 1 --2- 172.21.2.112:0/4152816742 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcb0c006770 0x7fcb0c008c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:46:03.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.916 7fcb1ee1d700 1 -- 172.21.2.112:0/4152816742 reap_dead start 2021-02-10T11:46:03.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.916 7fcb1ee1d700 1 --2- 172.21.2.112:0/4152816742 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcb0c009590 0x7fcb0c00bab0 secure :-1 s=READY pgs=2826 cs=0 l=1 rev1=1 rx=0x7fcb18001130 tx=0x7fcb1800a060).ready entity=mon.0 client_cookie=941208f287cd1106 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:03.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.916 7fcb1d61a700 1 -- 172.21.2.112:0/4152816742 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcb180070c0 con 0x7fcb0c009590 2021-02-10T11:46:03.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.916 7fcb1d61a700 1 -- 172.21.2.112:0/4152816742 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb18007440 con 0x7fcb0c009590 2021-02-10T11:46:03.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.916 7fcb1d61a700 1 -- 172.21.2.112:0/4152816742 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcb180049a0 con 0x7fcb0c009590 2021-02-10T11:46:03.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.916 7fcb24963700 1 -- 172.21.2.112:0/4152816742 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcb0c009590 msgr2=0x7fcb0c00bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:03.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.916 7fcb24963700 1 --2- 172.21.2.112:0/4152816742 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcb0c009590 0x7fcb0c00bab0 secure :-1 s=READY pgs=2826 cs=0 l=1 rev1=1 rx=0x7fcb18001130 tx=0x7fcb1800a060).stop 2021-02-10T11:46:03.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.917 7fcb24963700 1 -- 172.21.2.112:0/4152816742 shutdown_connections 2021-02-10T11:46:03.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.917 7fcb24963700 1 --2- 172.21.2.112:0/4152816742 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcb0c009590 0x7fcb0c00bab0 unknown :-1 s=CLOSED pgs=2826 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:03.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.917 7fcb24963700 1 -- 172.21.2.112:0/4152816742 >> 172.21.2.112:0/4152816742 conn(0x7fcb200ce450 msgr2=0x7fcb200d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:03.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.917 7fcb24963700 1 -- 172.21.2.112:0/4152816742 shutdown_connections 2021-02-10T11:46:03.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.917 7fcb24963700 1 -- 172.21.2.112:0/4152816742 wait complete. 2021-02-10T11:46:03.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.932 7fcb24963700 1 Processor -- start 2021-02-10T11:46:03.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.932 7fcb24963700 1 -- start start 2021-02-10T11:46:03.939 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.932 7fcb24963700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcb20195650 0x7fcb200b5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:03.939 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.932 7fcb24963700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcb200b5b80 0x7fcb201a2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:03.939 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.932 7fcb24963700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcb200acfa0 0x7fcb201a3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:03.939 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.932 7fcb24963700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb200b6010 con 0x7fcb20195650 2021-02-10T11:46:03.939 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.932 7fcb24963700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb201a3a60 con 0x7fcb200b5b80 2021-02-10T11:46:03.939 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.932 7fcb24963700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb201a3c40 con 0x7fcb200acfa0 2021-02-10T11:46:03.939 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.932 7fcb1e61c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcb20195650 0x7fcb200b5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:03.939 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.932 7fcb1e61c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcb20195650 0x7fcb200b5560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:59228/0 (socket says 172.21.2.112:59228) 2021-02-10T11:46:03.939 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.932 7fcb1e61c700 1 -- 172.21.2.112:0/2719042909 learned_addr learned my addr 172.21.2.112:0/2719042909 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:46:03.939 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.933 7fcb1e61c700 1 -- 172.21.2.112:0/2719042909 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcb200acfa0 msgr2=0x7fcb201a3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:03.940 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.933 7fcb1e61c700 1 --2- 172.21.2.112:0/2719042909 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcb200acfa0 0x7fcb201a3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:03.940 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.933 7fcb1e61c700 1 -- 172.21.2.112:0/2719042909 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcb200b5b80 msgr2=0x7fcb201a2e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:03.940 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.933 7fcb1e61c700 1 --2- 172.21.2.112:0/2719042909 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcb200b5b80 0x7fcb201a2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:03.940 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.933 7fcb1e61c700 1 -- 172.21.2.112:0/2719042909 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb200bcae0 con 0x7fcb20195650 2021-02-10T11:46:03.940 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.933 7fcb1e61c700 1 --2- 172.21.2.112:0/2719042909 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcb20195650 0x7fcb200b5560 secure :-1 s=READY pgs=2827 cs=0 l=1 rev1=1 rx=0x7fcb10006f60 tx=0x7fcb1000b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:03.940 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.933 7fcb16ffd700 1 -- 172.21.2.112:0/2719042909 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcb1000bd60 con 0x7fcb20195650 2021-02-10T11:46:03.940 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.933 7fcb16ffd700 1 -- 172.21.2.112:0/2719042909 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb10000c90 con 0x7fcb20195650 2021-02-10T11:46:03.940 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.933 7fcb16ffd700 1 -- 172.21.2.112:0/2719042909 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcb100022b0 con 0x7fcb20195650 2021-02-10T11:46:03.940 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.933 7fcb24963700 1 -- 172.21.2.112:0/2719042909 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb200bcae0 con 0x7fcb20195650 2021-02-10T11:46:03.940 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.933 7fcb24963700 1 -- 172.21.2.112:0/2719042909 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb200bd070 con 0x7fcb20195650 2021-02-10T11:46:03.940 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.934 7fcb16ffd700 1 -- 172.21.2.112:0/2719042909 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fcb10006b00 con 0x7fcb20195650 2021-02-10T11:46:03.941 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.934 7fcb16ffd700 1 --2- 172.21.2.112:0/2719042909 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcb040186c0 0x7fcb0401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:03.941 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.934 7fcb16ffd700 1 -- 172.21.2.112:0/2719042909 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 5645+0+0 (secure 0 0 0) 0x7fcb10032590 con 0x7fcb20195650 2021-02-10T11:46:03.941 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.934 7fcb1de1b700 1 --2- 172.21.2.112:0/2719042909 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcb040186c0 0x7fcb0401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:03.941 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.935 7fcb1de1b700 1 --2- 172.21.2.112:0/2719042909 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcb040186c0 0x7fcb0401abc0 crc :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:03.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.965 7fcb24963700 1 -- 172.21.2.112:0/2719042909 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb200bcaa0 con 0x7fcb20195650 2021-02-10T11:46:03.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:03.967 7fcb16ffd700 1 -- 172.21.2.112:0/2719042909 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fcb1002d220 con 0x7fcb20195650 2021-02-10T11:46:04.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:04.100 7fcb24963700 1 -- 172.21.2.112:0/2719042909 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_nonexistent_subvolume_snapshot_rm"]} v 0) v1 -- 0x7fcb200bcaa0 con 0x7fcb20195650 2021-02-10T11:46:04.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:04.675 7fcb16ffd700 1 -- 172.21.2.112:0/2719042909 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_nonexistent_subvolume_snapshot_rm"]}]=0 v778) v1 ==== 155+0+0 (secure 0 0 0) 0x7fcb1002d220 con 0x7fcb20195650 2021-02-10T11:46:04.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:04.720 7fcb24963700 1 -- 172.21.2.112:0/2719042909 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcb040186c0 msgr2=0x7fcb0401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:04.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:04.720 7fcb24963700 1 --2- 172.21.2.112:0/2719042909 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcb040186c0 0x7fcb0401abc0 crc :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:04.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:04.720 7fcb24963700 1 -- 172.21.2.112:0/2719042909 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcb20195650 msgr2=0x7fcb200b5560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:04.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:04.720 7fcb24963700 1 --2- 172.21.2.112:0/2719042909 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcb20195650 0x7fcb200b5560 secure :-1 s=READY pgs=2827 cs=0 l=1 rev1=1 rx=0x7fcb10006f60 tx=0x7fcb1000b970).stop 2021-02-10T11:46:04.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:04.720 7fcb24963700 1 -- 172.21.2.112:0/2719042909 shutdown_connections 2021-02-10T11:46:04.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:04.720 7fcb24963700 1 --2- 172.21.2.112:0/2719042909 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcb040186c0 0x7fcb0401abc0 unknown :-1 s=CLOSED pgs=1284 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:04.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:04.720 7fcb24963700 1 --2- 172.21.2.112:0/2719042909 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcb200b5b80 0x7fcb201a2e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:04.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:04.720 7fcb24963700 1 --2- 172.21.2.112:0/2719042909 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcb200acfa0 0x7fcb201a3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:04.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:04.720 7fcb24963700 1 --2- 172.21.2.112:0/2719042909 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcb20195650 0x7fcb200b5560 unknown :-1 s=CLOSED pgs=2827 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:04.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:04.720 7fcb24963700 1 -- 172.21.2.112:0/2719042909 >> 172.21.2.112:0/2719042909 conn(0x7fcb200b3040 msgr2=0x7fcb200c3250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:04.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:04.720 7fcb24963700 1 -- 172.21.2.112:0/2719042909 shutdown_connections 2021-02-10T11:46:04.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:04.720 7fcb24963700 1 -- 172.21.2.112:0/2719042909 wait complete. 2021-02-10T11:46:04.730 INFO:tasks.cephfs_test_runner:test_nonexistent_subvolume_snapshot_rm (tasks.cephfs.test_volumes.TestVolumes) ... ok 2021-02-10T11:46:04.730 INFO:tasks.cephfs_test_runner:Starting test: test_recover_auth_metadata_during_authorize (tasks.cephfs.test_volumes.TestVolumes) 2021-02-10T11:46:04.731 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestVolumes.test_recover_auth_metadata_during_authorize' 2021-02-10T11:46:05.159 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:05.140 7f5239222700 1 Processor -- start 2021-02-10T11:46:05.159 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:05.140 7f5239222700 1 -- start start 2021-02-10T11:46:05.159 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:05.141 7f5239222700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f52340d0340 0x7f52340ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:05.160 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:05.141 7f5239222700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52340f0b10 con 0x7f52340d1870 2021-02-10T11:46:05.160 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:05.141 7f5239222700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52341b0be0 con 0x7f52340d1c70 2021-02-10T11:46:05.160 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:05.141 7f5239222700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52340e5ed0 con 0x7f52340d0340 2021-02-10T11:46:05.160 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:05.141 7f5232d9d700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f52340d1870 0x7f52341b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:58694/0 (socket says 172.21.2.112:58694) 2021-02-10T11:46:05.160 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:05.141 7f5232d9d700 1 -- 172.21.2.112:0/1278719468 learned_addr learned my addr 172.21.2.112:0/1278719468 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:46:05.160 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:05.141 7f523259c700 1 --2- 172.21.2.112:0/1278719468 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f52340d0340 0x7f52340ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:05.160 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:05.141 7f5231d9b700 1 -- 172.21.2.112:0/1278719468 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2856195108 0 0) 0x7f5224001180 con 0x7f52340d1870 2021-02-10T11:46:05.160 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:05.141 7f5231d9b700 1 -- 172.21.2.112:0/1278719468 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f521c002fe0 con 0x7f52340d1870 2021-02-10T11:46:05.160 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:05.142 7f5231d9b700 1 -- 172.21.2.112:0/1278719468 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 910462923 0 0) 0x7f522c001400 con 0x7f52340d1c70 2021-02-10T11:46:05.160 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:05.142 7f523259c700 1 -- 172.21.2.112:0/1278719468 >> v1:172.21.2.120:6789/0 conn(0x7f52340d1c70 legacy=0x7f52340efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:05.160 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:05.142 7f523259c700 1 -- 172.21.2.112:0/1278719468 >> v1:172.21.2.112:6789/0 conn(0x7f52340d1870 legacy=0x7f52341b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:05.160 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:05.142 7f523259c700 1 -- 172.21.2.112:0/1278719468 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52340f0450 con 0x7f52340d0340 2021-02-10T11:46:05.160 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:05.142 7f523259c700 1 --2- 172.21.2.112:0/1278719468 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f52340d0340 0x7f52340ef640 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0x7f5228006650 tx=0x7f5228009a80).ready entity=mon.2 client_cookie=8e253f63263658f7 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:05.161 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:05.142 7f5231d9b700 1 -- 172.21.2.112:0/1278719468 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f522800c0c0 con 0x7f52340d0340 2021-02-10T11:46:05.161 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:05.142 7f5231d9b700 1 -- 172.21.2.112:0/1278719468 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f522800c340 con 0x7f52340d0340 2021-02-10T11:46:05.161 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:05.142 7f5231d9b700 1 -- 172.21.2.112:0/1278719468 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f522800c860 con 0x7f52340d0340 2021-02-10T11:46:05.161 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:05.142 7f5239222700 1 -- 172.21.2.112:0/1278719468 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f52340d0340 msgr2=0x7f52340ef640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:05.161 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:05.142 7f5239222700 1 --2- 172.21.2.112:0/1278719468 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f52340d0340 0x7f52340ef640 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0x7f5228006650 tx=0x7f5228009a80).stop 2021-02-10T11:46:05.161 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:05.143 7f5239222700 1 -- 172.21.2.112:0/1278719468 shutdown_connections 2021-02-10T11:46:05.161 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:05.143 7f5239222700 1 --2- 172.21.2.112:0/1278719468 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f52340d0340 0x7f52340ef640 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:05.161 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:05.143 7f5239222700 1 -- 172.21.2.112:0/1278719468 >> 172.21.2.112:0/1278719468 conn(0x7f52340ce450 msgr2=0x7f52340d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:05.161 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:05.143 7f5239222700 1 -- 172.21.2.112:0/1278719468 shutdown_connections 2021-02-10T11:46:05.161 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:05.143 7f5239222700 1 -- 172.21.2.112:0/1278719468 wait complete. 2021-02-10T11:46:05.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:05.158 7f5239222700 1 Processor -- start 2021-02-10T11:46:05.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:05.158 7f5239222700 1 -- start start 2021-02-10T11:46:05.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:05.158 7f5239222700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f52341b7cf0 0x7f523419cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:05.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:05.158 7f5239222700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f52341b80f0 0x7f52341b84f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:05.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:05.158 7f5239222700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f523419da20 0x7f523419de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:05.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:05.158 7f5239222700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52340a77c0 con 0x7f52341b7cf0 2021-02-10T11:46:05.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:05.158 7f5239222700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f523419d4a0 con 0x7f52341b80f0 2021-02-10T11:46:05.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:05.158 7f5239222700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52340954f0 con 0x7f523419da20 2021-02-10T11:46:05.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:05.158 7f5232d9d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f52341b7cf0 0x7f523419cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:05.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:05.158 7f5232d9d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f52341b7cf0 0x7f523419cdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:59236/0 (socket says 172.21.2.112:59236) 2021-02-10T11:46:05.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:05.158 7f5232d9d700 1 -- 172.21.2.112:0/137108995 learned_addr learned my addr 172.21.2.112:0/137108995 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:46:05.164 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:05.159 7f5232d9d700 1 -- 172.21.2.112:0/137108995 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f523419da20 msgr2=0x7f523419de20 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:46:05.164 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:05.159 7f5232d9d700 1 --2- 172.21.2.112:0/137108995 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f523419da20 0x7f523419de20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:05.164 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:05.159 7f5232d9d700 1 -- 172.21.2.112:0/137108995 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f52341b80f0 msgr2=0x7f52341b84f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:05.164 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:05.159 7f5232d9d700 1 --2- 172.21.2.112:0/137108995 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f52341b80f0 0x7f52341b84f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:05.164 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:05.159 7f5232d9d700 1 -- 172.21.2.112:0/137108995 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52341b8d60 con 0x7f52341b7cf0 2021-02-10T11:46:05.164 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:05.159 7f5232d9d700 1 --2- 172.21.2.112:0/137108995 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f52341b7cf0 0x7f523419cdb0 secure :-1 s=READY pgs=2829 cs=0 l=1 rev1=1 rx=0x7f5224000b80 tx=0x7f522400a980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:05.164 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:05.159 7f52237fe700 1 -- 172.21.2.112:0/137108995 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5224001490 con 0x7f52341b7cf0 2021-02-10T11:46:05.164 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:05.159 7f52237fe700 1 -- 172.21.2.112:0/137108995 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f522400ad70 con 0x7f52341b7cf0 2021-02-10T11:46:05.164 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:05.159 7f52237fe700 1 -- 172.21.2.112:0/137108995 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f52240045b0 con 0x7f52341b7cf0 2021-02-10T11:46:05.164 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:05.159 7f5239222700 1 -- 172.21.2.112:0/137108995 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52341b8d60 con 0x7f52341b7cf0 2021-02-10T11:46:05.164 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:05.159 7f5239222700 1 -- 172.21.2.112:0/137108995 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f523419e730 con 0x7f52341b7cf0 2021-02-10T11:46:05.164 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:05.160 7f52237fe700 1 -- 172.21.2.112:0/137108995 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f52240043a0 con 0x7f52341b7cf0 2021-02-10T11:46:05.164 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:05.160 7f52237fe700 1 --2- 172.21.2.112:0/137108995 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f52180186c0 0x7f521801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:05.164 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:05.160 7f52237fe700 1 -- 172.21.2.112:0/137108995 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 5645+0+0 (secure 0 0 0) 0x7f52240325c0 con 0x7f52341b7cf0 2021-02-10T11:46:05.165 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:05.160 7f523259c700 1 --2- 172.21.2.112:0/137108995 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f52180186c0 0x7f521801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:05.165 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:05.161 7f523259c700 1 --2- 172.21.2.112:0/137108995 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f52180186c0 0x7f521801abc0 crc :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:05.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:05.190 7f5239222700 1 -- 172.21.2.112:0/137108995 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52340ebc00 con 0x7f52341b7cf0 2021-02-10T11:46:05.195 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:05.193 7f52237fe700 1 -- 172.21.2.112:0/137108995 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5224025fe0 con 0x7f52341b7cf0 2021-02-10T11:46:05.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:05.326 7f5239222700 1 -- 172.21.2.112:0/137108995 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_recover_auth_metadata_during_authorize"]} v 0) v1 -- 0x7f52340f0b10 con 0x7f52341b7cf0 2021-02-10T11:46:05.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:05.679 7f52237fe700 1 -- 172.21.2.112:0/137108995 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_recover_auth_metadata_during_authorize"]}]=0 v779) v1 ==== 163+0+0 (secure 0 0 0) 0x7f5224025fe0 con 0x7f52341b7cf0 2021-02-10T11:46:05.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:05.694 7f5239222700 1 -- 172.21.2.112:0/137108995 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f52180186c0 msgr2=0x7f521801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:05.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:05.694 7f5239222700 1 --2- 172.21.2.112:0/137108995 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f52180186c0 0x7f521801abc0 crc :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:05.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:05.694 7f5239222700 1 -- 172.21.2.112:0/137108995 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f52341b7cf0 msgr2=0x7f523419cdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:05.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:05.694 7f5239222700 1 --2- 172.21.2.112:0/137108995 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f52341b7cf0 0x7f523419cdb0 secure :-1 s=READY pgs=2829 cs=0 l=1 rev1=1 rx=0x7f5224000b80 tx=0x7f522400a980).stop 2021-02-10T11:46:05.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:05.694 7f5239222700 1 -- 172.21.2.112:0/137108995 shutdown_connections 2021-02-10T11:46:05.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:05.694 7f5239222700 1 --2- 172.21.2.112:0/137108995 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f52180186c0 0x7f521801abc0 unknown :-1 s=CLOSED pgs=1285 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:05.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:05.694 7f5239222700 1 --2- 172.21.2.112:0/137108995 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f52341b80f0 0x7f52341b84f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:05.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:05.694 7f5239222700 1 --2- 172.21.2.112:0/137108995 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f523419da20 0x7f523419de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:05.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:05.694 7f5239222700 1 --2- 172.21.2.112:0/137108995 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f52341b7cf0 0x7f523419cdb0 unknown :-1 s=CLOSED pgs=2829 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:05.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:05.694 7f5239222700 1 -- 172.21.2.112:0/137108995 >> 172.21.2.112:0/137108995 conn(0x7f52340ed810 msgr2=0x7f5234094980 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:05.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:05.694 7f5239222700 1 -- 172.21.2.112:0/137108995 shutdown_connections 2021-02-10T11:46:05.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:05.694 7f5239222700 1 -- 172.21.2.112:0/137108995 wait complete. 2021-02-10T11:46:05.705 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:46:06.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:06.119 7fee09707700 1 Processor -- start 2021-02-10T11:46:06.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:06.119 7fee09707700 1 -- start start 2021-02-10T11:46:06.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:06.120 7fee09707700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fee040d1c70 0x7fee040cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:06.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:06.120 7fee09707700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee040d0170 con 0x7fee040d1870 2021-02-10T11:46:06.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:06.120 7fee09707700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee040fe1c0 con 0x7fee040cf580 2021-02-10T11:46:06.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:06.120 7fee09707700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee041b0cb0 con 0x7fee040d1c70 2021-02-10T11:46:06.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:06.120 7fee032bf700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fee040d1870 0x7fee041b04c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:58706/0 (socket says 172.21.2.112:58706) 2021-02-10T11:46:06.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:06.120 7fee032bf700 1 -- 172.21.2.112:0/1195445129 learned_addr learned my addr 172.21.2.112:0/1195445129 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:46:06.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:06.121 7fee022bd700 1 -- 172.21.2.112:0/1195445129 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1980961640 0 0) 0x7fedec001180 con 0x7fee040d1870 2021-02-10T11:46:06.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:06.121 7fee022bd700 1 -- 172.21.2.112:0/1195445129 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fedf0002fe0 con 0x7fee040d1870 2021-02-10T11:46:06.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:06.121 7fee02abe700 1 --2- 172.21.2.112:0/1195445129 >> v2:172.21.2.120:3300/0 conn(0x7fee040d1c70 0x7fee040cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:06.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:06.121 7fee022bd700 1 -- 172.21.2.112:0/1195445129 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 762562487 0 0) 0x7fedec001180 con 0x7fee040d1870 2021-02-10T11:46:06.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:06.121 7fee022bd700 1 -- 172.21.2.112:0/1195445129 >> v2:172.21.2.120:3300/0 conn(0x7fee040d1c70 msgr2=0x7fee040cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:46:06.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:06.121 7fee022bd700 1 --2- 172.21.2.112:0/1195445129 >> v2:172.21.2.120:3300/0 conn(0x7fee040d1c70 0x7fee040cef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:06.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:06.121 7fee022bd700 1 -- 172.21.2.112:0/1195445129 >> v1:172.21.2.120:6789/0 conn(0x7fee040cf580 legacy=0x7fee040cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:06.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:06.121 7fee022bd700 1 -- 172.21.2.112:0/1195445129 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee04109a60 con 0x7fee040d1870 2021-02-10T11:46:06.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:06.121 7fee022bd700 1 -- 172.21.2.112:0/1195445129 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7fedec001680 con 0x7fee040d1870 2021-02-10T11:46:06.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:06.121 7fee022bd700 1 -- 172.21.2.112:0/1195445129 >> v1:172.21.2.112:6789/0 conn(0x7fee040d1870 legacy=0x7fee041b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:06.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:06.121 7fee022bd700 1 --2- 172.21.2.112:0/1195445129 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fedf0003b10 0x7fedf0005f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:06.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:06.121 7fee022bd700 1 --2- 172.21.2.112:0/1195445129 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fedf00067c0 0x7fedf0008c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:06.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:06.121 7fee022bd700 1 --2- 172.21.2.112:0/1195445129 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fedf0009580 0x7fedf000baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:06.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:06.121 7fee022bd700 1 -- 172.21.2.112:0/1195445129 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedf000c2f0 con 0x7fedf0003b10 2021-02-10T11:46:06.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:06.121 7fee022bd700 1 -- 172.21.2.112:0/1195445129 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedf000c510 con 0x7fedf0009580 2021-02-10T11:46:06.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:06.121 7fee022bd700 1 -- 172.21.2.112:0/1195445129 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedf000c730 con 0x7fedf00067c0 2021-02-10T11:46:06.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:06.121 7fee032bf700 1 --2- 172.21.2.112:0/1195445129 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fedf0003b10 0x7fedf0005f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:06.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:06.121 7fee03ac0700 1 --2- 172.21.2.112:0/1195445129 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fedf0009580 0x7fedf000baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:06.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:06.122 7fee02abe700 1 --2- 172.21.2.112:0/1195445129 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fedf00067c0 0x7fedf0008c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:06.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:06.122 7fee032bf700 1 -- 172.21.2.112:0/1195445129 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fedf00067c0 msgr2=0x7fedf0008c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:46:06.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:06.122 7fee032bf700 1 --2- 172.21.2.112:0/1195445129 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fedf00067c0 0x7fedf0008c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:06.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:06.122 7fee032bf700 1 -- 172.21.2.112:0/1195445129 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fedf0009580 msgr2=0x7fedf000baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:46:06.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:06.122 7fee032bf700 1 --2- 172.21.2.112:0/1195445129 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fedf0009580 0x7fedf000baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:06.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:06.122 7fee032bf700 1 -- 172.21.2.112:0/1195445129 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedf00509a0 con 0x7fedf0003b10 2021-02-10T11:46:06.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:06.122 7fee03ac0700 1 -- 172.21.2.112:0/1195445129 reap_dead start 2021-02-10T11:46:06.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:06.122 7fee032bf700 1 --2- 172.21.2.112:0/1195445129 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fedf0003b10 0x7fedf0005f20 secure :-1 s=READY pgs=2831 cs=0 l=1 rev1=1 rx=0x7fedec000b60 tx=0x7fedec007d20).ready entity=mon.0 client_cookie=9e435440b3960bac server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:06.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:06.122 7fee022bd700 1 -- 172.21.2.112:0/1195445129 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fedec0070c0 con 0x7fedf0003b10 2021-02-10T11:46:06.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:06.122 7fee022bd700 1 -- 172.21.2.112:0/1195445129 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedec0043a0 con 0x7fedf0003b10 2021-02-10T11:46:06.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:06.122 7fee022bd700 1 -- 172.21.2.112:0/1195445129 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fedec004840 con 0x7fedf0003b10 2021-02-10T11:46:06.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:06.122 7fee09707700 1 -- 172.21.2.112:0/1195445129 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fedf0003b10 msgr2=0x7fedf0005f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:06.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:06.122 7fee09707700 1 --2- 172.21.2.112:0/1195445129 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fedf0003b10 0x7fedf0005f20 secure :-1 s=READY pgs=2831 cs=0 l=1 rev1=1 rx=0x7fedec000b60 tx=0x7fedec007d20).stop 2021-02-10T11:46:06.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:06.122 7fee09707700 1 -- 172.21.2.112:0/1195445129 shutdown_connections 2021-02-10T11:46:06.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:06.122 7fee09707700 1 --2- 172.21.2.112:0/1195445129 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fedf0003b10 0x7fedf0005f20 unknown :-1 s=CLOSED pgs=2831 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:06.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:06.122 7fee09707700 1 -- 172.21.2.112:0/1195445129 >> 172.21.2.112:0/1195445129 conn(0x7fee040ce420 msgr2=0x7fee040d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:06.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:06.122 7fee09707700 1 -- 172.21.2.112:0/1195445129 shutdown_connections 2021-02-10T11:46:06.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:06.122 7fee09707700 1 -- 172.21.2.112:0/1195445129 wait complete. 2021-02-10T11:46:06.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:06.137 7fee09707700 1 Processor -- start 2021-02-10T11:46:06.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:06.137 7fee09707700 1 -- start start 2021-02-10T11:46:06.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:06.137 7fee09707700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fee041a79a0 0x7fee041a7da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:06.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:06.138 7fee09707700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fee040bdb40 0x7fee040bdf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:06.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:06.138 7fee09707700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fee04082ca0 0x7fee040830a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:06.145 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:06.138 7fee09707700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee0419d990 con 0x7fee040bdb40 2021-02-10T11:46:06.145 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:06.138 7fee09707700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee0419dbb0 con 0x7fee041a79a0 2021-02-10T11:46:06.145 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:06.138 7fee09707700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee040836c0 con 0x7fee04082ca0 2021-02-10T11:46:06.145 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:06.138 7fee02abe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fee040bdb40 0x7fee040bdf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:06.145 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:06.138 7fee02abe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fee040bdb40 0x7fee040bdf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:59258/0 (socket says 172.21.2.112:59258) 2021-02-10T11:46:06.145 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:06.138 7fee02abe700 1 -- 172.21.2.112:0/3382911244 learned_addr learned my addr 172.21.2.112:0/3382911244 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:46:06.145 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:06.138 7fee02abe700 1 -- 172.21.2.112:0/3382911244 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fee04082ca0 msgr2=0x7fee040830a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:06.145 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:06.138 7fee02abe700 1 --2- 172.21.2.112:0/3382911244 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fee04082ca0 0x7fee040830a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:06.145 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:06.138 7fee02abe700 1 -- 172.21.2.112:0/3382911244 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fee041a79a0 msgr2=0x7fee041a7da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:06.145 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:06.138 7fee02abe700 1 --2- 172.21.2.112:0/3382911244 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fee041a79a0 0x7fee041a7da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:06.145 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:06.138 7fee02abe700 1 -- 172.21.2.112:0/3382911244 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee040838e0 con 0x7fee040bdb40 2021-02-10T11:46:06.145 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:06.138 7fee02abe700 1 --2- 172.21.2.112:0/3382911244 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fee040bdb40 0x7fee040bdf40 secure :-1 s=READY pgs=2832 cs=0 l=1 rev1=1 rx=0x7fedf400b560 tx=0x7fedf400b900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:06.145 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:06.138 7fedfbfff700 1 -- 172.21.2.112:0/3382911244 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fedf400bcf0 con 0x7fee040bdb40 2021-02-10T11:46:06.146 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:06.139 7fedfbfff700 1 -- 172.21.2.112:0/3382911244 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedf4000c90 con 0x7fee040bdb40 2021-02-10T11:46:06.146 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:06.139 7fedfbfff700 1 -- 172.21.2.112:0/3382911244 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fedf40022b0 con 0x7fee040bdb40 2021-02-10T11:46:06.146 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:06.139 7fee09707700 1 -- 172.21.2.112:0/3382911244 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee040838e0 con 0x7fee040bdb40 2021-02-10T11:46:06.146 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:06.139 7fee09707700 1 -- 172.21.2.112:0/3382911244 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee040838e0 con 0x7fee040bdb40 2021-02-10T11:46:06.146 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:06.139 7fedfbfff700 1 -- 172.21.2.112:0/3382911244 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fedf4006b00 con 0x7fee040bdb40 2021-02-10T11:46:06.147 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:06.139 7fedfbfff700 1 --2- 172.21.2.112:0/3382911244 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fede80186d0 0x7fede801abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:06.147 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:06.139 7fedfbfff700 1 -- 172.21.2.112:0/3382911244 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 5645+0+0 (secure 0 0 0) 0x7fedf4032650 con 0x7fee040bdb40 2021-02-10T11:46:06.147 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:06.140 7fee032bf700 1 --2- 172.21.2.112:0/3382911244 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fede80186d0 0x7fede801abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:06.147 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:06.140 7fee032bf700 1 --2- 172.21.2.112:0/3382911244 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fede80186d0 0x7fede801abd0 crc :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:06.172 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:06.170 7fee09707700 1 -- 172.21.2.112:0/3382911244 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee04082a70 con 0x7fee040bdb40 2021-02-10T11:46:06.174 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:06.172 7fedfbfff700 1 -- 172.21.2.112:0/3382911244 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fedf400bcf0 con 0x7fee040bdb40 2021-02-10T11:46:06.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:06.320 7fee09707700 1 -- 172.21.2.112:0/3382911244 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fee04082a70 con 0x7fee040bdb40 2021-02-10T11:46:06.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:06.323 7fedfbfff700 1 -- 172.21.2.112:0/3382911244 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v42)=0 v42) v1 ==== 122+0+0 (secure 0 0 0) 0x7fedf4006b00 con 0x7fee040bdb40 2021-02-10T11:46:06.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:06.329 7fee09707700 1 -- 172.21.2.112:0/3382911244 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fede80186d0 msgr2=0x7fede801abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:06.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:06.329 7fee09707700 1 --2- 172.21.2.112:0/3382911244 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fede80186d0 0x7fede801abd0 crc :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:06.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:06.329 7fee09707700 1 -- 172.21.2.112:0/3382911244 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fee040bdb40 msgr2=0x7fee040bdf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:06.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:06.329 7fee09707700 1 --2- 172.21.2.112:0/3382911244 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fee040bdb40 0x7fee040bdf40 secure :-1 s=READY pgs=2832 cs=0 l=1 rev1=1 rx=0x7fedf400b560 tx=0x7fedf400b900).stop 2021-02-10T11:46:06.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:06.329 7fee09707700 1 -- 172.21.2.112:0/3382911244 shutdown_connections 2021-02-10T11:46:06.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:06.329 7fee09707700 1 --2- 172.21.2.112:0/3382911244 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fede80186d0 0x7fede801abd0 unknown :-1 s=CLOSED pgs=1286 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:06.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:06.329 7fee09707700 1 --2- 172.21.2.112:0/3382911244 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fee04082ca0 0x7fee040830a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:06.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:06.329 7fee09707700 1 --2- 172.21.2.112:0/3382911244 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fee040bdb40 0x7fee040bdf40 unknown :-1 s=CLOSED pgs=2832 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:06.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:06.329 7fee09707700 1 --2- 172.21.2.112:0/3382911244 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fee041a79a0 0x7fee041a7da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:06.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:06.329 7fee09707700 1 -- 172.21.2.112:0/3382911244 >> 172.21.2.112:0/3382911244 conn(0x7fee040bef00 msgr2=0x7fee0409f200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:06.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:06.329 7fee09707700 1 -- 172.21.2.112:0/3382911244 shutdown_connections 2021-02-10T11:46:06.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:06.329 7fee09707700 1 -- 172.21.2.112:0/3382911244 wait complete. 2021-02-10T11:46:06.339 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:46:06.339 DEBUG:teuthology.orchestra.run.gibba012:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:46:06.687 DEBUG:teuthology.orchestra.run.gibba020:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:46:07.031 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:46:07.031 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:46:07.031 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:46:07.031 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:46:07.365 INFO:teuthology.orchestra.run.gibba012.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.0’: No such file or directory 2021-02-10T11:46:07.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:46:07.365 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2021-02-10T11:46:07.366 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:46:07.366 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:46:07.366 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:46:07.701 INFO:teuthology.orchestra.run.gibba012.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: No such file or directory 2021-02-10T11:46:07.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:46:07.702 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2021-02-10T11:46:07.702 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:46:07.702 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:46:07.702 DEBUG:teuthology.orchestra.run.gibba020:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:46:08.038 INFO:teuthology.orchestra.run.gibba020.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:46:08.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:46:08.038 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:46:08.038 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:46:08.039 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:46:08.039 DEBUG:teuthology.orchestra.run.gibba020:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:46:08.375 INFO:teuthology.orchestra.run.gibba020.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:46:08.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:46:08.376 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:46:08.376 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json-pretty 2021-02-10T11:46:08.806 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:08.787 7fa6b1596700 1 Processor -- start 2021-02-10T11:46:08.806 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:08.787 7fa6b1596700 1 -- start start 2021-02-10T11:46:08.807 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:08.787 7fa6b1596700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fa6ac0d0340 0x7fa6ac0ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:08.807 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:08.787 7fa6b1596700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa6ac0d1c70 0x7fa6ac0efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:08.807 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:08.787 7fa6b1596700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6ac0f0b10 con 0x7fa6ac0d1870 2021-02-10T11:46:08.807 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:08.787 7fa6b1596700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6ac0e5ed0 con 0x7fa6ac0d0340 2021-02-10T11:46:08.807 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:08.787 7fa6b1596700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6ac1b0be0 con 0x7fa6ac0d1c70 2021-02-10T11:46:08.807 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:08.788 7fa6abac0700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa6ac0d1c70 0x7fa6ac0efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:08.807 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:08.788 7fa6aaabe700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fa6ac0d0340 0x7fa6ac0ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:08.807 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:08.788 7fa6ab2bf700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7fa6ac0d1870 0x7fa6ac1b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:52284/0 (socket says 172.21.2.112:52284) 2021-02-10T11:46:08.807 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:08.788 7fa6ab2bf700 1 -- 172.21.2.112:0/3487860743 learned_addr learned my addr 172.21.2.112:0/3487860743 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:46:08.807 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:08.789 7fa6abac0700 1 -- 172.21.2.112:0/3487860743 >> v2:172.21.2.120:3300/0 conn(0x7fa6ac0d0340 msgr2=0x7fa6ac0ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:46:08.807 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:08.789 7fa6abac0700 1 --2- 172.21.2.112:0/3487860743 >> v2:172.21.2.120:3300/0 conn(0x7fa6ac0d0340 0x7fa6ac0ef640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:08.807 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:08.789 7fa6abac0700 1 -- 172.21.2.112:0/3487860743 >> v1:172.21.2.120:6789/0 conn(0x7fa6ac0d1870 legacy=0x7fa6ac1b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:08.808 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:08.789 7fa6abac0700 1 -- 172.21.2.112:0/3487860743 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6ac0f20a0 con 0x7fa6ac0d1c70 2021-02-10T11:46:08.808 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:08.789 7fa6aaabe700 1 --2- 172.21.2.112:0/3487860743 >> v2:172.21.2.120:3300/0 conn(0x7fa6ac0d0340 0x7fa6ac0ef640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:46:08.808 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:08.789 7fa6abac0700 1 --2- 172.21.2.112:0/3487860743 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa6ac0d1c70 0x7fa6ac0efc60 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7fa6a4006e50 tx=0x7fa6a4009e70).ready entity=mon.2 client_cookie=9e32058f535077cb server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:08.808 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:08.789 7fa6aa2bd700 1 -- 172.21.2.112:0/3487860743 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa6a4006aa0 con 0x7fa6ac0d1c70 2021-02-10T11:46:08.808 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:08.789 7fa6aa2bd700 1 -- 172.21.2.112:0/3487860743 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6a4006790 con 0x7fa6ac0d1c70 2021-02-10T11:46:08.808 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:08.789 7fa6aa2bd700 1 -- 172.21.2.112:0/3487860743 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa6a400b700 con 0x7fa6ac0d1c70 2021-02-10T11:46:08.808 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:08.789 7fa6b1596700 1 -- 172.21.2.112:0/3487860743 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa6ac0d1c70 msgr2=0x7fa6ac0efc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:08.808 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:08.789 7fa6b1596700 1 --2- 172.21.2.112:0/3487860743 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa6ac0d1c70 0x7fa6ac0efc60 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7fa6a4006e50 tx=0x7fa6a4009e70).stop 2021-02-10T11:46:08.808 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:08.790 7fa6b1596700 1 -- 172.21.2.112:0/3487860743 shutdown_connections 2021-02-10T11:46:08.808 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:08.790 7fa6b1596700 1 --2- 172.21.2.112:0/3487860743 >> v2:172.21.2.120:3300/0 conn(0x7fa6ac0d0340 0x7fa6ac0ef640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:08.808 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:08.790 7fa6b1596700 1 --2- 172.21.2.112:0/3487860743 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa6ac0d1c70 0x7fa6ac0efc60 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:08.808 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:08.790 7fa6b1596700 1 -- 172.21.2.112:0/3487860743 >> 172.21.2.112:0/3487860743 conn(0x7fa6ac0ce450 msgr2=0x7fa6ac0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:08.809 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:08.790 7fa6b1596700 1 -- 172.21.2.112:0/3487860743 shutdown_connections 2021-02-10T11:46:08.809 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:08.790 7fa6b1596700 1 -- 172.21.2.112:0/3487860743 wait complete. 2021-02-10T11:46:08.810 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:08.805 7fa6b1596700 1 Processor -- start 2021-02-10T11:46:08.810 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:08.805 7fa6b1596700 1 -- start start 2021-02-10T11:46:08.810 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:08.805 7fa6b1596700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa6ac0ada90 0x7fa6ac0ade90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:08.810 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:08.805 7fa6b1596700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa6ac099d20 0x7fa6ac09a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:08.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:08.805 7fa6b1596700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa6ac09a740 0x7fa6ac097bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:08.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:08.805 7fa6b1596700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6ac0997a0 con 0x7fa6ac099d20 2021-02-10T11:46:08.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:08.805 7fa6b1596700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6ac18f0c0 con 0x7fa6ac09a740 2021-02-10T11:46:08.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:08.805 7fa6b1596700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6ac09ab40 con 0x7fa6ac0ada90 2021-02-10T11:46:08.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:08.805 7fa6aaabe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa6ac099d20 0x7fa6ac09a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:08.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:08.805 7fa6aaabe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa6ac099d20 0x7fa6ac09a120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:59270/0 (socket says 172.21.2.112:59270) 2021-02-10T11:46:08.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:08.805 7fa6aaabe700 1 -- 172.21.2.112:0/374230041 learned_addr learned my addr 172.21.2.112:0/374230041 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:46:08.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:08.806 7fa6abac0700 1 --2- 172.21.2.112:0/374230041 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa6ac09a740 0x7fa6ac097bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:08.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:08.806 7fa6aaabe700 1 -- 172.21.2.112:0/374230041 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa6ac0ada90 msgr2=0x7fa6ac0ade90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:08.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:08.806 7fa6aaabe700 1 --2- 172.21.2.112:0/374230041 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa6ac0ada90 0x7fa6ac0ade90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:08.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:08.806 7fa6aaabe700 1 -- 172.21.2.112:0/374230041 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa6ac09a740 msgr2=0x7fa6ac097bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:08.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:08.806 7fa6aaabe700 1 --2- 172.21.2.112:0/374230041 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa6ac09a740 0x7fa6ac097bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:08.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:08.806 7fa6aaabe700 1 -- 172.21.2.112:0/374230041 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6ac09ad60 con 0x7fa6ac099d20 2021-02-10T11:46:08.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:08.806 7fa6aaabe700 1 --2- 172.21.2.112:0/374230041 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa6ac099d20 0x7fa6ac09a120 secure :-1 s=READY pgs=2833 cs=0 l=1 rev1=1 rx=0x7fa698006f40 tx=0x7fa69800b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:08.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:08.806 7fa6a3fff700 1 -- 172.21.2.112:0/374230041 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa69800bd80 con 0x7fa6ac099d20 2021-02-10T11:46:08.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:08.806 7fa6a3fff700 1 -- 172.21.2.112:0/374230041 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa698000c90 con 0x7fa6ac099d20 2021-02-10T11:46:08.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:08.806 7fa6a3fff700 1 -- 172.21.2.112:0/374230041 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa698002220 con 0x7fa6ac099d20 2021-02-10T11:46:08.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:08.806 7fa6b1596700 1 -- 172.21.2.112:0/374230041 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6ac09ad60 con 0x7fa6ac099d20 2021-02-10T11:46:08.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:08.806 7fa6b1596700 1 -- 172.21.2.112:0/374230041 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6ac0984d0 con 0x7fa6ac099d20 2021-02-10T11:46:08.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:08.807 7fa6a3fff700 1 -- 172.21.2.112:0/374230041 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa69800a000 con 0x7fa6ac099d20 2021-02-10T11:46:08.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:08.807 7fa6a3fff700 1 --2- 172.21.2.112:0/374230041 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa6880186c0 0x7fa68801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:08.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:08.807 7fa6a3fff700 1 -- 172.21.2.112:0/374230041 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(163..163 src has 1..163) v4 ==== 5617+0+0 (secure 0 0 0) 0x7fa6980325a0 con 0x7fa6ac099d20 2021-02-10T11:46:08.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:08.807 7fa6ab2bf700 1 --2- 172.21.2.112:0/374230041 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa6880186c0 0x7fa68801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:08.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:08.808 7fa6ab2bf700 1 --2- 172.21.2.112:0/374230041 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa6880186c0 0x7fa68801abc0 crc :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:08.839 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:08.837 7fa6b1596700 1 -- 172.21.2.112:0/374230041 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6ac0ebc00 con 0x7fa6ac099d20 2021-02-10T11:46:08.841 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:08.839 7fa6a3fff700 1 -- 172.21.2.112:0/374230041 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa698000960 con 0x7fa6ac099d20 2021-02-10T11:46:08.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:08.964 7fa6b1596700 1 -- 172.21.2.112:0/374230041 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7fa6ac0f0b10 con 0x7fa6ac099d20 2021-02-10T11:46:08.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:08.965 7fa6a3fff700 1 -- 172.21.2.112:0/374230041 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v163) v1 ==== 81+0+30586 (secure 0 0 0) 0x7fa698030d20 con 0x7fa6ac099d20 2021-02-10T11:46:08.968 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:46:08.968 INFO:teuthology.orchestra.run.gibba012.stdout:{ 2021-02-10T11:46:08.968 INFO:teuthology.orchestra.run.gibba012.stdout: "epoch": 163, 2021-02-10T11:46:08.968 INFO:teuthology.orchestra.run.gibba012.stdout: "fsid": "f14f004e-0699-4584-ad81-3754258493de", 2021-02-10T11:46:08.968 INFO:teuthology.orchestra.run.gibba012.stdout: "created": "2021-02-10 11:13:26.501424", 2021-02-10T11:46:08.968 INFO:teuthology.orchestra.run.gibba012.stdout: "modified": "2021-02-10 11:46:06.494252", 2021-02-10T11:46:08.968 INFO:teuthology.orchestra.run.gibba012.stdout: "last_up_change": "2021-02-10 11:13:52.780316", 2021-02-10T11:46:08.968 INFO:teuthology.orchestra.run.gibba012.stdout: "last_in_change": "2021-02-10 11:13:52.780316", 2021-02-10T11:46:08.968 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-02-10T11:46:08.968 INFO:teuthology.orchestra.run.gibba012.stdout: "flags_num": 5799936, 2021-02-10T11:46:08.968 INFO:teuthology.orchestra.run.gibba012.stdout: "flags_set": [ 2021-02-10T11:46:08.969 INFO:teuthology.orchestra.run.gibba012.stdout: "pglog_hardlimit", 2021-02-10T11:46:08.969 INFO:teuthology.orchestra.run.gibba012.stdout: "purged_snapdirs", 2021-02-10T11:46:08.969 INFO:teuthology.orchestra.run.gibba012.stdout: "recovery_deletes", 2021-02-10T11:46:08.969 INFO:teuthology.orchestra.run.gibba012.stdout: "sortbitwise" 2021-02-10T11:46:08.969 INFO:teuthology.orchestra.run.gibba012.stdout: ], 2021-02-10T11:46:08.969 INFO:teuthology.orchestra.run.gibba012.stdout: "crush_version": 4, 2021-02-10T11:46:08.969 INFO:teuthology.orchestra.run.gibba012.stdout: "full_ratio": 0.94999998807907104, 2021-02-10T11:46:08.969 INFO:teuthology.orchestra.run.gibba012.stdout: "backfillfull_ratio": 0.89999997615814209, 2021-02-10T11:46:08.969 INFO:teuthology.orchestra.run.gibba012.stdout: "nearfull_ratio": 0.85000002384185791, 2021-02-10T11:46:08.969 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_snapshot": "", 2021-02-10T11:46:08.969 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_max": 35, 2021-02-10T11:46:08.969 INFO:teuthology.orchestra.run.gibba012.stdout: "max_osd": 8, 2021-02-10T11:46:08.969 INFO:teuthology.orchestra.run.gibba012.stdout: "require_min_compat_client": "jewel", 2021-02-10T11:46:08.970 INFO:teuthology.orchestra.run.gibba012.stdout: "min_compat_client": "jewel", 2021-02-10T11:46:08.970 INFO:teuthology.orchestra.run.gibba012.stdout: "require_osd_release": "nautilus", 2021-02-10T11:46:08.970 INFO:teuthology.orchestra.run.gibba012.stdout: "pools": [ 2021-02-10T11:46:08.970 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:46:08.970 INFO:teuthology.orchestra.run.gibba012.stdout: "pool": 1, 2021-02-10T11:46:08.970 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_name": "rbd", 2021-02-10T11:46:08.970 INFO:teuthology.orchestra.run.gibba012.stdout: "create_time": "2021-02-10 11:13:57.508758", 2021-02-10T11:46:08.970 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:46:08.970 INFO:teuthology.orchestra.run.gibba012.stdout: "flags_names": "hashpspool", 2021-02-10T11:46:08.970 INFO:teuthology.orchestra.run.gibba012.stdout: "type": 1, 2021-02-10T11:46:08.970 INFO:teuthology.orchestra.run.gibba012.stdout: "size": 2, 2021-02-10T11:46:08.970 INFO:teuthology.orchestra.run.gibba012.stdout: "min_size": 1, 2021-02-10T11:46:08.970 INFO:teuthology.orchestra.run.gibba012.stdout: "crush_rule": 0, 2021-02-10T11:46:08.971 INFO:teuthology.orchestra.run.gibba012.stdout: "object_hash": 2, 2021-02-10T11:46:08.971 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:46:08.971 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num": 8, 2021-02-10T11:46:08.971 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num": 8, 2021-02-10T11:46:08.971 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num_target": 8, 2021-02-10T11:46:08.971 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_target": 8, 2021-02-10T11:46:08.971 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_pending": 8, 2021-02-10T11:46:08.971 INFO:teuthology.orchestra.run.gibba012.stdout: "last_pg_merge_meta": { 2021-02-10T11:46:08.971 INFO:teuthology.orchestra.run.gibba012.stdout: "source_pgid": "0.0", 2021-02-10T11:46:08.971 INFO:teuthology.orchestra.run.gibba012.stdout: "ready_epoch": 0, 2021-02-10T11:46:08.971 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_started": 0, 2021-02-10T11:46:08.971 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_clean": 0, 2021-02-10T11:46:08.971 INFO:teuthology.orchestra.run.gibba012.stdout: "source_version": "0'0", 2021-02-10T11:46:08.972 INFO:teuthology.orchestra.run.gibba012.stdout: "target_version": "0'0" 2021-02-10T11:46:08.972 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:46:08.972 INFO:teuthology.orchestra.run.gibba012.stdout: "last_change": "16", 2021-02-10T11:46:08.972 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend": "0", 2021-02-10T11:46:08.972 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:46:08.972 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:46:08.972 INFO:teuthology.orchestra.run.gibba012.stdout: "auid": 0, 2021-02-10T11:46:08.972 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:46:08.972 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_seq": 0, 2021-02-10T11:46:08.972 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_epoch": 0, 2021-02-10T11:46:08.972 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_snaps": [], 2021-02-10T11:46:08.972 INFO:teuthology.orchestra.run.gibba012.stdout: "removed_snaps": "[]", 2021-02-10T11:46:08.972 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_bytes": 0, 2021-02-10T11:46:08.973 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_objects": 0, 2021-02-10T11:46:08.973 INFO:teuthology.orchestra.run.gibba012.stdout: "tiers": [], 2021-02-10T11:46:08.973 INFO:teuthology.orchestra.run.gibba012.stdout: "tier_of": -1, 2021-02-10T11:46:08.973 INFO:teuthology.orchestra.run.gibba012.stdout: "read_tier": -1, 2021-02-10T11:46:08.973 INFO:teuthology.orchestra.run.gibba012.stdout: "write_tier": -1, 2021-02-10T11:46:08.973 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_mode": "none", 2021-02-10T11:46:08.973 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_bytes": 0, 2021-02-10T11:46:08.973 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_objects": 0, 2021-02-10T11:46:08.973 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:46:08.973 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:46:08.973 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:46:08.973 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_flush_age": 0, 2021-02-10T11:46:08.973 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_evict_age": 0, 2021-02-10T11:46:08.974 INFO:teuthology.orchestra.run.gibba012.stdout: "erasure_code_profile": "", 2021-02-10T11:46:08.974 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_params": { 2021-02-10T11:46:08.974 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "none" 2021-02-10T11:46:08.974 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:46:08.974 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_period": 0, 2021-02-10T11:46:08.974 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_count": 0, 2021-02-10T11:46:08.974 INFO:teuthology.orchestra.run.gibba012.stdout: "use_gmt_hitset": true, 2021-02-10T11:46:08.974 INFO:teuthology.orchestra.run.gibba012.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:46:08.974 INFO:teuthology.orchestra.run.gibba012.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:46:08.974 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:46:08.974 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:46:08.974 INFO:teuthology.orchestra.run.gibba012.stdout: "grade_table": [], 2021-02-10T11:46:08.974 INFO:teuthology.orchestra.run.gibba012.stdout: "stripe_width": 0, 2021-02-10T11:46:08.975 INFO:teuthology.orchestra.run.gibba012.stdout: "expected_num_objects": 0, 2021-02-10T11:46:08.975 INFO:teuthology.orchestra.run.gibba012.stdout: "fast_read": false, 2021-02-10T11:46:08.975 INFO:teuthology.orchestra.run.gibba012.stdout: "options": {}, 2021-02-10T11:46:08.975 INFO:teuthology.orchestra.run.gibba012.stdout: "application_metadata": { 2021-02-10T11:46:08.975 INFO:teuthology.orchestra.run.gibba012.stdout: "rbd": {} 2021-02-10T11:46:08.975 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:46:08.975 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:46:08.975 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:46:08.975 INFO:teuthology.orchestra.run.gibba012.stdout: "pool": 34, 2021-02-10T11:46:08.975 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_name": "cephfs_metadata", 2021-02-10T11:46:08.975 INFO:teuthology.orchestra.run.gibba012.stdout: "create_time": "2021-02-10 11:45:28.386296", 2021-02-10T11:46:08.975 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:46:08.976 INFO:teuthology.orchestra.run.gibba012.stdout: "flags_names": "hashpspool", 2021-02-10T11:46:08.976 INFO:teuthology.orchestra.run.gibba012.stdout: "type": 1, 2021-02-10T11:46:08.976 INFO:teuthology.orchestra.run.gibba012.stdout: "size": 2, 2021-02-10T11:46:08.976 INFO:teuthology.orchestra.run.gibba012.stdout: "min_size": 1, 2021-02-10T11:46:08.976 INFO:teuthology.orchestra.run.gibba012.stdout: "crush_rule": 0, 2021-02-10T11:46:08.976 INFO:teuthology.orchestra.run.gibba012.stdout: "object_hash": 2, 2021-02-10T11:46:08.976 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:46:08.976 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num": 8, 2021-02-10T11:46:08.976 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num": 8, 2021-02-10T11:46:08.976 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num_target": 8, 2021-02-10T11:46:08.976 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_target": 8, 2021-02-10T11:46:08.976 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_pending": 8, 2021-02-10T11:46:08.976 INFO:teuthology.orchestra.run.gibba012.stdout: "last_pg_merge_meta": { 2021-02-10T11:46:08.977 INFO:teuthology.orchestra.run.gibba012.stdout: "source_pgid": "0.0", 2021-02-10T11:46:08.977 INFO:teuthology.orchestra.run.gibba012.stdout: "ready_epoch": 0, 2021-02-10T11:46:08.977 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_started": 0, 2021-02-10T11:46:08.977 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_clean": 0, 2021-02-10T11:46:08.977 INFO:teuthology.orchestra.run.gibba012.stdout: "source_version": "0'0", 2021-02-10T11:46:08.977 INFO:teuthology.orchestra.run.gibba012.stdout: "target_version": "0'0" 2021-02-10T11:46:08.977 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:46:08.977 INFO:teuthology.orchestra.run.gibba012.stdout: "last_change": "160", 2021-02-10T11:46:08.977 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend": "0", 2021-02-10T11:46:08.977 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:46:08.977 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:46:08.977 INFO:teuthology.orchestra.run.gibba012.stdout: "auid": 0, 2021-02-10T11:46:08.977 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:46:08.977 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_seq": 0, 2021-02-10T11:46:08.978 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_epoch": 0, 2021-02-10T11:46:08.978 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_snaps": [], 2021-02-10T11:46:08.978 INFO:teuthology.orchestra.run.gibba012.stdout: "removed_snaps": "[]", 2021-02-10T11:46:08.978 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_bytes": 0, 2021-02-10T11:46:08.978 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_objects": 0, 2021-02-10T11:46:08.978 INFO:teuthology.orchestra.run.gibba012.stdout: "tiers": [], 2021-02-10T11:46:08.978 INFO:teuthology.orchestra.run.gibba012.stdout: "tier_of": -1, 2021-02-10T11:46:08.978 INFO:teuthology.orchestra.run.gibba012.stdout: "read_tier": -1, 2021-02-10T11:46:08.978 INFO:teuthology.orchestra.run.gibba012.stdout: "write_tier": -1, 2021-02-10T11:46:08.978 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_mode": "none", 2021-02-10T11:46:08.978 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_bytes": 0, 2021-02-10T11:46:08.978 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_objects": 0, 2021-02-10T11:46:08.979 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:46:08.979 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:46:08.979 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:46:08.979 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_flush_age": 0, 2021-02-10T11:46:08.979 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_evict_age": 0, 2021-02-10T11:46:08.979 INFO:teuthology.orchestra.run.gibba012.stdout: "erasure_code_profile": "", 2021-02-10T11:46:08.979 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_params": { 2021-02-10T11:46:08.979 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "none" 2021-02-10T11:46:08.979 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:46:08.979 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_period": 0, 2021-02-10T11:46:08.979 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_count": 0, 2021-02-10T11:46:08.979 INFO:teuthology.orchestra.run.gibba012.stdout: "use_gmt_hitset": true, 2021-02-10T11:46:08.979 INFO:teuthology.orchestra.run.gibba012.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:46:08.979 INFO:teuthology.orchestra.run.gibba012.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:46:08.980 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:46:08.980 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:46:08.980 INFO:teuthology.orchestra.run.gibba012.stdout: "grade_table": [], 2021-02-10T11:46:08.980 INFO:teuthology.orchestra.run.gibba012.stdout: "stripe_width": 0, 2021-02-10T11:46:08.980 INFO:teuthology.orchestra.run.gibba012.stdout: "expected_num_objects": 0, 2021-02-10T11:46:08.980 INFO:teuthology.orchestra.run.gibba012.stdout: "fast_read": false, 2021-02-10T11:46:08.980 INFO:teuthology.orchestra.run.gibba012.stdout: "options": { 2021-02-10T11:46:08.980 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_autoscale_bias": 4, 2021-02-10T11:46:08.980 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_min": 16, 2021-02-10T11:46:08.980 INFO:teuthology.orchestra.run.gibba012.stdout: "recovery_priority": 5 2021-02-10T11:46:08.980 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:46:08.981 INFO:teuthology.orchestra.run.gibba012.stdout: "application_metadata": { 2021-02-10T11:46:08.981 INFO:teuthology.orchestra.run.gibba012.stdout: "cephfs": { 2021-02-10T11:46:08.981 INFO:teuthology.orchestra.run.gibba012.stdout: "metadata": "cephfs" 2021-02-10T11:46:08.981 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:46:08.981 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:46:08.981 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:46:08.981 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:46:08.981 INFO:teuthology.orchestra.run.gibba012.stdout: "pool": 35, 2021-02-10T11:46:08.981 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_name": "cephfs_data", 2021-02-10T11:46:08.981 INFO:teuthology.orchestra.run.gibba012.stdout: "create_time": "2021-02-10 11:45:29.158381", 2021-02-10T11:46:08.981 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 8193, 2021-02-10T11:46:08.981 INFO:teuthology.orchestra.run.gibba012.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2021-02-10T11:46:08.981 INFO:teuthology.orchestra.run.gibba012.stdout: "type": 1, 2021-02-10T11:46:08.982 INFO:teuthology.orchestra.run.gibba012.stdout: "size": 2, 2021-02-10T11:46:08.982 INFO:teuthology.orchestra.run.gibba012.stdout: "min_size": 1, 2021-02-10T11:46:08.982 INFO:teuthology.orchestra.run.gibba012.stdout: "crush_rule": 0, 2021-02-10T11:46:08.982 INFO:teuthology.orchestra.run.gibba012.stdout: "object_hash": 2, 2021-02-10T11:46:08.982 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:46:08.982 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num": 8, 2021-02-10T11:46:08.982 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num": 8, 2021-02-10T11:46:08.982 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num_target": 8, 2021-02-10T11:46:08.982 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_target": 8, 2021-02-10T11:46:08.982 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_pending": 8, 2021-02-10T11:46:08.982 INFO:teuthology.orchestra.run.gibba012.stdout: "last_pg_merge_meta": { 2021-02-10T11:46:08.982 INFO:teuthology.orchestra.run.gibba012.stdout: "source_pgid": "0.0", 2021-02-10T11:46:08.982 INFO:teuthology.orchestra.run.gibba012.stdout: "ready_epoch": 0, 2021-02-10T11:46:08.983 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_started": 0, 2021-02-10T11:46:08.983 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_clean": 0, 2021-02-10T11:46:08.984 INFO:teuthology.orchestra.run.gibba012.stdout: "source_version": "0'0", 2021-02-10T11:46:08.984 INFO:teuthology.orchestra.run.gibba012.stdout: "target_version": "0'0" 2021-02-10T11:46:08.984 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:46:08.984 INFO:teuthology.orchestra.run.gibba012.stdout: "last_change": "162", 2021-02-10T11:46:08.984 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend": "0", 2021-02-10T11:46:08.984 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:46:08.984 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:46:08.984 INFO:teuthology.orchestra.run.gibba012.stdout: "auid": 0, 2021-02-10T11:46:08.984 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:46:08.984 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_seq": 3, 2021-02-10T11:46:08.984 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_epoch": 162, 2021-02-10T11:46:08.985 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_snaps": [], 2021-02-10T11:46:08.985 INFO:teuthology.orchestra.run.gibba012.stdout: "removed_snaps": "[2~2]", 2021-02-10T11:46:08.985 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_bytes": 0, 2021-02-10T11:46:08.985 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_objects": 0, 2021-02-10T11:46:08.985 INFO:teuthology.orchestra.run.gibba012.stdout: "tiers": [], 2021-02-10T11:46:08.985 INFO:teuthology.orchestra.run.gibba012.stdout: "tier_of": -1, 2021-02-10T11:46:08.985 INFO:teuthology.orchestra.run.gibba012.stdout: "read_tier": -1, 2021-02-10T11:46:08.985 INFO:teuthology.orchestra.run.gibba012.stdout: "write_tier": -1, 2021-02-10T11:46:08.985 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_mode": "none", 2021-02-10T11:46:08.985 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_bytes": 0, 2021-02-10T11:46:08.985 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_objects": 0, 2021-02-10T11:46:08.985 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:46:08.985 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:46:08.985 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:46:08.986 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_flush_age": 0, 2021-02-10T11:46:08.986 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_evict_age": 0, 2021-02-10T11:46:08.986 INFO:teuthology.orchestra.run.gibba012.stdout: "erasure_code_profile": "", 2021-02-10T11:46:08.986 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_params": { 2021-02-10T11:46:08.986 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "none" 2021-02-10T11:46:08.986 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:46:08.986 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_period": 0, 2021-02-10T11:46:08.986 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_count": 0, 2021-02-10T11:46:08.986 INFO:teuthology.orchestra.run.gibba012.stdout: "use_gmt_hitset": true, 2021-02-10T11:46:08.986 INFO:teuthology.orchestra.run.gibba012.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:46:08.986 INFO:teuthology.orchestra.run.gibba012.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:46:08.986 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:46:08.987 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:46:08.987 INFO:teuthology.orchestra.run.gibba012.stdout: "grade_table": [], 2021-02-10T11:46:08.987 INFO:teuthology.orchestra.run.gibba012.stdout: "stripe_width": 0, 2021-02-10T11:46:08.987 INFO:teuthology.orchestra.run.gibba012.stdout: "expected_num_objects": 0, 2021-02-10T11:46:08.987 INFO:teuthology.orchestra.run.gibba012.stdout: "fast_read": false, 2021-02-10T11:46:08.987 INFO:teuthology.orchestra.run.gibba012.stdout: "options": {}, 2021-02-10T11:46:08.987 INFO:teuthology.orchestra.run.gibba012.stdout: "application_metadata": { 2021-02-10T11:46:08.987 INFO:teuthology.orchestra.run.gibba012.stdout: "cephfs": { 2021-02-10T11:46:08.987 INFO:teuthology.orchestra.run.gibba012.stdout: "data": "cephfs" 2021-02-10T11:46:08.987 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:46:08.987 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:46:08.987 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:46:08.987 INFO:teuthology.orchestra.run.gibba012.stdout: ], 2021-02-10T11:46:08.988 INFO:teuthology.orchestra.run.gibba012.stdout: "osds": [ 2021-02-10T11:46:08.988 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:46:08.988 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 0, 2021-02-10T11:46:08.988 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "95444b8f-6370-4a1d-ae5f-df62651b77da", 2021-02-10T11:46:08.988 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:46:08.988 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:46:08.988 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:46:08.988 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:46:08.988 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:46:08.988 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:46:08.989 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:46:08.989 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 156, 2021-02-10T11:46:08.989 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:46:08.989 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:46:08.989 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:46:08.989 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:46:08.989 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:46:08.989 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:46:08.989 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6817", 2021-02-10T11:46:08.989 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:46:08.989 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:46:08.989 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:46:08.989 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:46:08.990 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6818", 2021-02-10T11:46:08.990 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:46:08.990 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:46:08.990 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:46:08.990 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:46:08.990 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:46:08.990 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:46:08.990 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:46:08.990 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:46:08.990 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6820", 2021-02-10T11:46:08.990 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:46:08.990 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:46:08.991 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:46:08.991 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:46:08.991 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6822", 2021-02-10T11:46:08.991 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:46:08.991 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:46:08.991 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:46:08.991 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:46:08.991 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:46:08.991 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:46:08.991 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:46:08.991 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:46:08.991 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6828", 2021-02-10T11:46:08.991 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:46:08.992 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:46:08.992 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:46:08.992 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:46:08.992 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6830", 2021-02-10T11:46:08.992 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:46:08.992 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:46:08.992 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:46:08.992 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:46:08.992 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:46:08.992 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:46:08.992 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:46:08.992 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:46:08.992 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6824", 2021-02-10T11:46:08.993 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:46:08.993 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:46:08.993 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:46:08.993 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:46:08.993 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6826", 2021-02-10T11:46:08.993 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:46:08.993 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:46:08.993 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:46:08.993 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:46:08.993 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.112:6818/36030", 2021-02-10T11:46:08.993 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.112:6822/36030", 2021-02-10T11:46:08.993 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.112:6830/36030", 2021-02-10T11:46:08.993 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.112:6826/36030", 2021-02-10T11:46:08.994 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:46:08.994 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:46:08.994 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:46:08.994 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:46:08.994 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:46:08.994 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:46:08.994 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 1, 2021-02-10T11:46:08.994 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "a23e217e-0d8e-486e-be07-712adfabd262", 2021-02-10T11:46:08.994 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:46:08.994 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:46:08.994 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:46:08.994 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:46:08.995 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:46:08.995 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:46:08.995 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:46:08.995 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 141, 2021-02-10T11:46:08.995 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:46:08.995 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:46:08.995 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:46:08.995 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:46:08.995 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:46:08.995 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:46:08.995 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6800", 2021-02-10T11:46:08.995 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:46:08.995 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:46:08.996 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:46:08.996 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:46:08.996 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6802", 2021-02-10T11:46:08.996 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:46:08.996 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:46:08.996 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:46:08.996 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:46:08.996 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:46:08.996 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:46:08.996 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:46:08.996 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:46:08.996 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6804", 2021-02-10T11:46:08.997 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:46:08.997 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:46:08.997 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:46:08.997 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:46:08.997 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6806", 2021-02-10T11:46:08.997 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:46:08.997 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:46:08.997 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:46:08.997 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:46:08.997 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:46:08.997 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:46:08.997 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:46:08.997 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:46:08.997 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6812", 2021-02-10T11:46:08.998 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:46:08.998 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:46:08.998 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:46:08.998 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:46:08.998 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6814", 2021-02-10T11:46:08.998 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:46:08.998 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:46:08.998 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:46:08.998 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:46:08.998 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:46:08.998 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:46:08.998 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:46:08.999 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:46:08.999 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6808", 2021-02-10T11:46:08.999 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:46:08.999 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:46:08.999 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:46:08.999 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:46:08.999 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6810", 2021-02-10T11:46:08.999 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:46:08.999 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:46:08.999 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:46:08.999 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:46:08.999 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.112:6802/36029", 2021-02-10T11:46:08.999 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.112:6806/36029", 2021-02-10T11:46:09.000 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.112:6814/36029", 2021-02-10T11:46:09.000 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.112:6810/36029", 2021-02-10T11:46:09.000 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:46:09.000 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:46:09.000 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:46:09.000 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:46:09.000 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:46:09.000 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:46:09.000 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 2, 2021-02-10T11:46:09.000 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "026afa8a-c0f1-4b35-9a76-53c00fe0c2fc", 2021-02-10T11:46:09.000 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:46:09.000 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:46:09.000 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:46:09.001 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:46:09.001 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:46:09.001 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:46:09.001 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:46:09.001 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 159, 2021-02-10T11:46:09.001 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:46:09.001 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:46:09.001 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:46:09.001 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:46:09.001 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:46:09.001 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:46:09.001 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6801", 2021-02-10T11:46:09.001 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:46:09.002 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:46:09.002 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:46:09.002 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:46:09.002 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6803", 2021-02-10T11:46:09.002 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:46:09.002 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:46:09.002 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:46:09.002 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:46:09.002 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:46:09.002 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:46:09.002 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:46:09.002 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:46:09.002 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6805", 2021-02-10T11:46:09.003 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:46:09.003 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:46:09.003 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:46:09.003 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:46:09.003 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6807", 2021-02-10T11:46:09.003 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:46:09.003 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:46:09.003 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:46:09.003 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:46:09.003 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:46:09.003 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:46:09.003 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:46:09.003 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:46:09.003 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6813", 2021-02-10T11:46:09.004 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:46:09.004 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:46:09.004 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:46:09.004 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:46:09.004 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6815", 2021-02-10T11:46:09.004 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:46:09.004 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:46:09.004 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:46:09.004 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:46:09.004 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:46:09.004 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:46:09.004 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:46:09.004 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:46:09.005 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6809", 2021-02-10T11:46:09.005 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:46:09.005 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:46:09.005 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:46:09.005 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:46:09.005 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6811", 2021-02-10T11:46:09.005 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:46:09.005 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:46:09.005 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:46:09.005 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:46:09.005 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.112:6803/36031", 2021-02-10T11:46:09.005 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.112:6807/36031", 2021-02-10T11:46:09.005 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.112:6815/36031", 2021-02-10T11:46:09.005 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.112:6811/36031", 2021-02-10T11:46:09.006 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:46:09.006 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:46:09.006 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:46:09.006 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:46:09.006 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:46:09.006 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:46:09.006 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 3, 2021-02-10T11:46:09.006 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "58795682-e00d-4173-bb84-f7075ca89a11", 2021-02-10T11:46:09.006 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:46:09.006 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:46:09.006 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:46:09.006 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:46:09.007 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:46:09.007 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:46:09.007 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:46:09.007 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 159, 2021-02-10T11:46:09.007 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:46:09.007 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:46:09.007 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:46:09.007 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:46:09.007 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:46:09.007 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:46:09.007 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6816", 2021-02-10T11:46:09.007 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:46:09.007 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:46:09.008 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:46:09.008 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:46:09.008 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6819", 2021-02-10T11:46:09.008 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:46:09.008 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:46:09.008 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:46:09.008 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:46:09.008 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:46:09.008 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:46:09.008 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:46:09.008 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:46:09.008 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6821", 2021-02-10T11:46:09.008 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:46:09.009 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:46:09.009 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:46:09.009 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:46:09.009 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6823", 2021-02-10T11:46:09.009 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:46:09.009 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:46:09.009 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:46:09.009 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:46:09.009 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:46:09.009 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:46:09.009 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:46:09.009 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:46:09.009 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6829", 2021-02-10T11:46:09.010 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:46:09.010 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:46:09.010 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:46:09.010 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:46:09.010 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6831", 2021-02-10T11:46:09.010 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:46:09.010 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:46:09.010 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:46:09.010 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:46:09.010 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:46:09.010 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:46:09.010 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:46:09.011 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:46:09.011 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6825", 2021-02-10T11:46:09.011 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:46:09.011 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:46:09.011 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:46:09.011 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:46:09.011 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6827", 2021-02-10T11:46:09.011 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:46:09.011 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:46:09.011 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:46:09.011 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:46:09.011 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.112:6819/36028", 2021-02-10T11:46:09.011 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.112:6823/36028", 2021-02-10T11:46:09.011 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.112:6831/36028", 2021-02-10T11:46:09.012 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.112:6827/36028", 2021-02-10T11:46:09.012 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:46:09.012 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:46:09.012 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:46:09.012 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:46:09.012 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:46:09.012 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:46:09.012 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 4, 2021-02-10T11:46:09.012 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "6763f8d6-53dc-4e45-bdf6-a6eabcf81e39", 2021-02-10T11:46:09.012 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:46:09.012 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:46:09.012 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:46:09.012 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:46:09.013 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:46:09.013 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:46:09.013 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:46:09.013 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 159, 2021-02-10T11:46:09.013 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:46:09.013 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:46:09.013 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:46:09.013 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:46:09.013 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:46:09.013 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:46:09.013 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6810", 2021-02-10T11:46:09.013 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:46:09.013 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:46:09.014 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:46:09.014 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:46:09.014 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6813", 2021-02-10T11:46:09.014 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:46:09.014 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:46:09.014 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:46:09.014 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:46:09.014 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:46:09.014 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:46:09.014 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:46:09.014 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:46:09.014 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6816", 2021-02-10T11:46:09.014 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:46:09.015 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:46:09.015 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:46:09.015 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:46:09.015 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6819", 2021-02-10T11:46:09.015 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:46:09.015 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:46:09.015 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:46:09.015 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:46:09.015 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:46:09.015 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:46:09.015 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:46:09.015 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:46:09.016 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6825", 2021-02-10T11:46:09.016 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:46:09.016 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:46:09.016 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:46:09.016 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:46:09.016 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6827", 2021-02-10T11:46:09.016 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:46:09.016 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:46:09.016 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:46:09.016 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:46:09.016 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:46:09.016 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:46:09.016 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:46:09.017 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:46:09.017 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6821", 2021-02-10T11:46:09.017 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:46:09.017 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:46:09.017 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:46:09.017 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:46:09.017 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6823", 2021-02-10T11:46:09.017 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:46:09.017 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:46:09.017 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:46:09.017 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:46:09.017 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.120:6813/36388", 2021-02-10T11:46:09.017 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.120:6819/36388", 2021-02-10T11:46:09.017 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.120:6827/36388", 2021-02-10T11:46:09.018 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.120:6823/36388", 2021-02-10T11:46:09.018 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:46:09.018 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:46:09.018 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:46:09.018 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:46:09.018 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:46:09.018 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:46:09.018 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 5, 2021-02-10T11:46:09.018 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "5d0e4dd6-0cb0-4bf8-8314-626acaa1b544", 2021-02-10T11:46:09.018 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:46:09.018 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:46:09.018 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:46:09.019 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:46:09.019 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:46:09.019 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:46:09.019 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:46:09.019 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 159, 2021-02-10T11:46:09.019 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:46:09.019 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:46:09.019 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:46:09.019 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:46:09.019 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:46:09.019 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:46:09.019 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6804", 2021-02-10T11:46:09.019 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:46:09.020 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:46:09.020 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:46:09.020 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:46:09.020 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6806", 2021-02-10T11:46:09.020 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:46:09.020 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:46:09.020 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:46:09.020 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:46:09.020 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:46:09.020 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:46:09.020 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:46:09.020 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:46:09.020 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6809", 2021-02-10T11:46:09.021 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:46:09.021 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:46:09.021 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:46:09.021 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:46:09.021 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6812", 2021-02-10T11:46:09.021 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:46:09.021 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:46:09.021 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:46:09.021 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:46:09.021 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:46:09.021 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:46:09.021 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:46:09.022 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:46:09.022 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6820", 2021-02-10T11:46:09.022 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:46:09.022 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:46:09.022 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:46:09.022 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:46:09.022 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6822", 2021-02-10T11:46:09.022 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:46:09.022 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:46:09.022 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:46:09.022 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:46:09.022 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:46:09.022 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:46:09.023 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:46:09.023 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:46:09.023 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6815", 2021-02-10T11:46:09.023 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:46:09.023 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:46:09.023 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:46:09.023 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:46:09.023 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6818", 2021-02-10T11:46:09.023 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:46:09.023 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:46:09.023 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:46:09.023 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:46:09.023 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.120:6806/36387", 2021-02-10T11:46:09.024 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.120:6812/36387", 2021-02-10T11:46:09.024 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.120:6822/36387", 2021-02-10T11:46:09.024 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.120:6818/36387", 2021-02-10T11:46:09.024 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:46:09.024 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:46:09.024 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:46:09.024 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:46:09.024 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:46:09.024 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:46:09.024 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 6, 2021-02-10T11:46:09.024 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "b010e187-4541-4b00-80dd-f948f6ab8f15", 2021-02-10T11:46:09.024 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:46:09.024 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:46:09.024 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:46:09.025 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:46:09.025 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:46:09.025 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:46:09.025 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:46:09.025 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 159, 2021-02-10T11:46:09.025 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:46:09.025 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:46:09.025 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:46:09.025 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:46:09.025 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:46:09.025 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:46:09.025 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6824", 2021-02-10T11:46:09.025 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:46:09.026 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:46:09.026 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:46:09.026 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:46:09.026 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6826", 2021-02-10T11:46:09.026 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:46:09.026 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:46:09.026 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:46:09.026 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:46:09.026 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:46:09.026 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:46:09.026 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:46:09.026 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:46:09.027 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6828", 2021-02-10T11:46:09.027 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:46:09.027 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:46:09.027 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:46:09.027 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:46:09.027 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6829", 2021-02-10T11:46:09.027 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:46:09.027 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:46:09.027 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:46:09.027 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:46:09.027 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:46:09.027 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:46:09.027 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:46:09.028 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:46:09.028 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6832", 2021-02-10T11:46:09.028 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:46:09.028 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:46:09.028 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:46:09.028 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:46:09.028 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6833", 2021-02-10T11:46:09.028 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:46:09.028 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:46:09.028 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:46:09.028 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:46:09.028 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:46:09.028 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:46:09.029 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:46:09.029 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:46:09.029 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6830", 2021-02-10T11:46:09.029 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:46:09.029 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:46:09.029 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:46:09.029 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:46:09.029 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6831", 2021-02-10T11:46:09.029 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:46:09.029 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:46:09.029 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:46:09.029 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:46:09.029 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.120:6826/36385", 2021-02-10T11:46:09.030 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.120:6829/36385", 2021-02-10T11:46:09.030 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.120:6833/36385", 2021-02-10T11:46:09.030 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.120:6831/36385", 2021-02-10T11:46:09.030 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:46:09.030 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:46:09.030 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:46:09.030 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:46:09.030 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:46:09.030 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:46:09.030 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 7, 2021-02-10T11:46:09.030 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "671e2772-33b7-497e-8e44-2178bd850bcf", 2021-02-10T11:46:09.030 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:46:09.030 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:46:09.031 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:46:09.031 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:46:09.031 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:46:09.031 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:46:09.031 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:46:09.031 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 156, 2021-02-10T11:46:09.031 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:46:09.031 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:46:09.031 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:46:09.031 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:46:09.031 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:46:09.031 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:46:09.031 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6802", 2021-02-10T11:46:09.032 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:46:09.032 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:46:09.032 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:46:09.032 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:46:09.032 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6803", 2021-02-10T11:46:09.032 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:46:09.032 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:46:09.032 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:46:09.032 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:46:09.032 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:46:09.032 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:46:09.032 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:46:09.032 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:46:09.033 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6805", 2021-02-10T11:46:09.033 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:46:09.033 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:46:09.033 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:46:09.033 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:46:09.033 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6807", 2021-02-10T11:46:09.033 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:46:09.033 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:46:09.033 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:46:09.033 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:46:09.033 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:46:09.033 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:46:09.033 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:46:09.034 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:46:09.034 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6814", 2021-02-10T11:46:09.034 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:46:09.034 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:46:09.034 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:46:09.034 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:46:09.034 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6817", 2021-02-10T11:46:09.034 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:46:09.034 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:46:09.034 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:46:09.034 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:46:09.034 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:46:09.034 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:46:09.035 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:46:09.035 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:46:09.035 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6808", 2021-02-10T11:46:09.035 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:46:09.035 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:46:09.035 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:46:09.035 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:46:09.035 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6811", 2021-02-10T11:46:09.035 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:46:09.035 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:46:09.035 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:46:09.035 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:46:09.035 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.120:6803/36386", 2021-02-10T11:46:09.036 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.120:6807/36386", 2021-02-10T11:46:09.036 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.120:6817/36386", 2021-02-10T11:46:09.036 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.120:6811/36386", 2021-02-10T11:46:09.036 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:46:09.036 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:46:09.036 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:46:09.036 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:46:09.036 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:46:09.036 INFO:teuthology.orchestra.run.gibba012.stdout: ], 2021-02-10T11:46:09.036 INFO:teuthology.orchestra.run.gibba012.stdout: "osd_xinfo": [ 2021-02-10T11:46:09.036 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:46:09.036 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 0, 2021-02-10T11:46:09.036 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:46:09.036 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:46:09.037 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:46:09.037 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:46:09.037 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:46:09.037 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:46:09.037 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:46:09.037 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 1, 2021-02-10T11:46:09.037 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:46:09.037 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:46:09.037 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:46:09.037 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:46:09.037 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:46:09.037 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:46:09.037 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:46:09.038 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 2, 2021-02-10T11:46:09.038 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:46:09.038 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:46:09.038 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:46:09.038 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:46:09.038 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:46:09.038 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:46:09.038 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:46:09.038 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 3, 2021-02-10T11:46:09.038 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:46:09.038 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:46:09.038 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:46:09.039 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:46:09.039 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:46:09.039 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:46:09.039 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:46:09.039 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 4, 2021-02-10T11:46:09.039 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:46:09.039 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:46:09.039 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:46:09.039 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:46:09.039 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:46:09.039 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:46:09.039 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:46:09.039 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 5, 2021-02-10T11:46:09.039 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:46:09.040 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:46:09.040 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:46:09.040 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:46:09.040 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:46:09.040 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:46:09.040 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:46:09.040 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 6, 2021-02-10T11:46:09.040 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:46:09.040 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:46:09.040 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:46:09.040 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:46:09.040 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:46:09.040 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:46:09.041 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:46:09.041 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 7, 2021-02-10T11:46:09.041 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:46:09.041 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:46:09.041 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:46:09.041 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:46:09.041 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:46:09.041 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:46:09.041 INFO:teuthology.orchestra.run.gibba012.stdout: ], 2021-02-10T11:46:09.041 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_upmap": [], 2021-02-10T11:46:09.041 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_upmap_items": [], 2021-02-10T11:46:09.041 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_temp": [], 2021-02-10T11:46:09.041 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_temp": [], 2021-02-10T11:46:09.042 INFO:teuthology.orchestra.run.gibba012.stdout: "blacklist": {}, 2021-02-10T11:46:09.042 INFO:teuthology.orchestra.run.gibba012.stdout: "erasure_code_profiles": { 2021-02-10T11:46:09.042 INFO:teuthology.orchestra.run.gibba012.stdout: "default": { 2021-02-10T11:46:09.042 INFO:teuthology.orchestra.run.gibba012.stdout: "crush-failure-domain": "osd", 2021-02-10T11:46:09.042 INFO:teuthology.orchestra.run.gibba012.stdout: "k": "2", 2021-02-10T11:46:09.042 INFO:teuthology.orchestra.run.gibba012.stdout: "m": "1", 2021-02-10T11:46:09.042 INFO:teuthology.orchestra.run.gibba012.stdout: "plugin": "jerasure", 2021-02-10T11:46:09.042 INFO:teuthology.orchestra.run.gibba012.stdout: "ruleset-failure-domain": "osd", 2021-02-10T11:46:09.042 INFO:teuthology.orchestra.run.gibba012.stdout: "technique": "reed_sol_van" 2021-02-10T11:46:09.042 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:46:09.042 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:46:09.043 INFO:teuthology.orchestra.run.gibba012.stdout: "removed_snaps_queue": [], 2021-02-10T11:46:09.043 INFO:teuthology.orchestra.run.gibba012.stdout: "new_removed_snaps": [], 2021-02-10T11:46:09.043 INFO:teuthology.orchestra.run.gibba012.stdout: "new_purged_snaps": [ 2021-02-10T11:46:09.043 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:46:09.043 INFO:teuthology.orchestra.run.gibba012.stdout: "pool": 35, 2021-02-10T11:46:09.043 INFO:teuthology.orchestra.run.gibba012.stdout: "snaps": [ 2021-02-10T11:46:09.043 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:46:09.043 INFO:teuthology.orchestra.run.gibba012.stdout: "begin": 2, 2021-02-10T11:46:09.043 INFO:teuthology.orchestra.run.gibba012.stdout: "length": 2 2021-02-10T11:46:09.043 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:46:09.043 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:46:09.043 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:46:09.043 INFO:teuthology.orchestra.run.gibba012.stdout: ], 2021-02-10T11:46:09.044 INFO:teuthology.orchestra.run.gibba012.stdout: "crush_node_flags": {}, 2021-02-10T11:46:09.044 INFO:teuthology.orchestra.run.gibba012.stdout: "device_class_flags": {} 2021-02-10T11:46:09.044 INFO:teuthology.orchestra.run.gibba012.stdout:} 2021-02-10T11:46:09.045 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:08.966 7fa6b1596700 1 -- 172.21.2.112:0/374230041 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa6880186c0 msgr2=0x7fa68801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:09.045 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:08.966 7fa6b1596700 1 --2- 172.21.2.112:0/374230041 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa6880186c0 0x7fa68801abc0 crc :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:09.045 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:08.966 7fa6b1596700 1 -- 172.21.2.112:0/374230041 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa6ac099d20 msgr2=0x7fa6ac09a120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:09.045 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:08.966 7fa6b1596700 1 --2- 172.21.2.112:0/374230041 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa6ac099d20 0x7fa6ac09a120 secure :-1 s=READY pgs=2833 cs=0 l=1 rev1=1 rx=0x7fa698006f40 tx=0x7fa69800b990).stop 2021-02-10T11:46:09.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:08.967 7fa6b1596700 1 -- 172.21.2.112:0/374230041 shutdown_connections 2021-02-10T11:46:09.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:08.967 7fa6b1596700 1 --2- 172.21.2.112:0/374230041 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa6880186c0 0x7fa68801abc0 unknown :-1 s=CLOSED pgs=1287 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:09.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:08.967 7fa6b1596700 1 --2- 172.21.2.112:0/374230041 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa6ac09a740 0x7fa6ac097bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:09.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:08.967 7fa6b1596700 1 --2- 172.21.2.112:0/374230041 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa6ac099d20 0x7fa6ac09a120 unknown :-1 s=CLOSED pgs=2833 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:09.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:08.967 7fa6b1596700 1 --2- 172.21.2.112:0/374230041 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa6ac0ada90 0x7fa6ac0ade90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:09.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:08.967 7fa6b1596700 1 -- 172.21.2.112:0/374230041 >> 172.21.2.112:0/374230041 conn(0x7fa6ac0f0550 msgr2=0x7fa6ac191730 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:09.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:08.967 7fa6b1596700 1 -- 172.21.2.112:0/374230041 shutdown_connections 2021-02-10T11:46:09.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:08.967 7fa6b1596700 1 -- 172.21.2.112:0/374230041 wait complete. 2021-02-10T11:46:09.047 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:46:09.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:09.458 7f70aaa4d700 1 Processor -- start 2021-02-10T11:46:09.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:09.458 7f70aaa4d700 1 -- start start 2021-02-10T11:46:09.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:09.458 7f70aaa4d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f70a40d0340 0x7f70a40ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:09.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:09.458 7f70aaa4d700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f70a40d1c70 0x7f70a40efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:09.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:09.458 7f70aaa4d700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70a40f0b10 con 0x7f70a40d1870 2021-02-10T11:46:09.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:09.458 7f70aaa4d700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70a40e5ed0 con 0x7f70a40d0340 2021-02-10T11:46:09.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:09.458 7f70aaa4d700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70a41b0be0 con 0x7f70a40d1c70 2021-02-10T11:46:09.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:09.458 7f70a3fff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f70a40d0340 0x7f70a40ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:09.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:09.458 7f70a3fff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f70a40d0340 0x7f70a40ef640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:59280/0 (socket says 172.21.2.112:59280) 2021-02-10T11:46:09.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:09.458 7f70a8869700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f70a40d1870 0x7f70a41b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:58740/0 (socket says 172.21.2.112:58740) 2021-02-10T11:46:09.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:09.458 7f70a3fff700 1 -- 172.21.2.112:0/3329009016 learned_addr learned my addr 172.21.2.112:0/3329009016 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:46:09.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:09.458 7f70a906a700 1 --2- 172.21.2.112:0/3329009016 >> v2:172.21.2.120:3300/0 conn(0x7f70a40d1c70 0x7f70a40efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:09.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:09.459 7f70a37fe700 1 -- 172.21.2.112:0/3329009016 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2965990508 0 0) 0x7f709c001060 con 0x7f70a40d1870 2021-02-10T11:46:09.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:09.459 7f70a37fe700 1 -- 172.21.2.112:0/3329009016 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7088002fe0 con 0x7f70a40d1870 2021-02-10T11:46:09.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:09.459 7f70a3fff700 1 -- 172.21.2.112:0/3329009016 >> v2:172.21.2.120:3300/0 conn(0x7f70a40d1c70 msgr2=0x7f70a40efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:46:09.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:09.459 7f70a3fff700 1 --2- 172.21.2.112:0/3329009016 >> v2:172.21.2.120:3300/0 conn(0x7f70a40d1c70 0x7f70a40efc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:09.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:09.459 7f70a3fff700 1 -- 172.21.2.112:0/3329009016 >> v1:172.21.2.112:6789/0 conn(0x7f70a40d1870 legacy=0x7f70a41b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:09.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:09.459 7f70a3fff700 1 -- 172.21.2.112:0/3329009016 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70a40f20a0 con 0x7f70a40d0340 2021-02-10T11:46:09.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:09.459 7f70a3fff700 1 --2- 172.21.2.112:0/3329009016 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f70a40d0340 0x7f70a40ef640 secure :-1 s=READY pgs=2835 cs=0 l=1 rev1=1 rx=0x7f7090006700 tx=0x7f7090009b80).ready entity=mon.0 client_cookie=3068a5a88b05d129 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:09.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:09.459 7f70a37fe700 1 -- 172.21.2.112:0/3329009016 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f709000c0c0 con 0x7f70a40d0340 2021-02-10T11:46:09.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:09.459 7f70a37fe700 1 -- 172.21.2.112:0/3329009016 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f709000c340 con 0x7f70a40d0340 2021-02-10T11:46:09.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:09.459 7f70a37fe700 1 -- 172.21.2.112:0/3329009016 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f709000c820 con 0x7f70a40d0340 2021-02-10T11:46:09.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:09.459 7f70aaa4d700 1 -- 172.21.2.112:0/3329009016 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f70a40d0340 msgr2=0x7f70a40ef640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:09.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:09.459 7f70aaa4d700 1 --2- 172.21.2.112:0/3329009016 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f70a40d0340 0x7f70a40ef640 secure :-1 s=READY pgs=2835 cs=0 l=1 rev1=1 rx=0x7f7090006700 tx=0x7f7090009b80).stop 2021-02-10T11:46:09.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:09.459 7f70aaa4d700 1 -- 172.21.2.112:0/3329009016 shutdown_connections 2021-02-10T11:46:09.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:09.459 7f70aaa4d700 1 --2- 172.21.2.112:0/3329009016 >> v2:172.21.2.120:3300/0 conn(0x7f70a40d1c70 0x7f70a40efc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:09.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:09.459 7f70aaa4d700 1 --2- 172.21.2.112:0/3329009016 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f70a40d0340 0x7f70a40ef640 unknown :-1 s=CLOSED pgs=2835 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:09.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:09.459 7f70aaa4d700 1 -- 172.21.2.112:0/3329009016 >> 172.21.2.112:0/3329009016 conn(0x7f70a40ce450 msgr2=0x7f70a40d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:09.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:09.460 7f70aaa4d700 1 -- 172.21.2.112:0/3329009016 shutdown_connections 2021-02-10T11:46:09.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:09.460 7f70aaa4d700 1 -- 172.21.2.112:0/3329009016 wait complete. 2021-02-10T11:46:09.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:09.474 7f70aaa4d700 1 Processor -- start 2021-02-10T11:46:09.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:09.474 7f70aaa4d700 1 -- start start 2021-02-10T11:46:09.485 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:09.475 7f70aaa4d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f70a4195650 0x7f70a40b5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:09.485 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:09.475 7f70aaa4d700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f70a40b5b80 0x7f70a41a2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:09.486 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:09.475 7f70aaa4d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f70a40acfa0 0x7f70a41a3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:09.486 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:09.475 7f70aaa4d700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70a40b6010 con 0x7f70a4195650 2021-02-10T11:46:09.486 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:09.475 7f70aaa4d700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70a41a3a60 con 0x7f70a40b5b80 2021-02-10T11:46:09.486 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:09.475 7f70aaa4d700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70a41a3c40 con 0x7f70a40acfa0 2021-02-10T11:46:09.486 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:09.475 7f70a8869700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f70a4195650 0x7f70a40b5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:09.486 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:09.475 7f70a8869700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f70a4195650 0x7f70a40b5560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:59284/0 (socket says 172.21.2.112:59284) 2021-02-10T11:46:09.487 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:09.475 7f70a8869700 1 -- 172.21.2.112:0/3893800133 learned_addr learned my addr 172.21.2.112:0/3893800133 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:46:09.487 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:09.475 7f70a8869700 1 -- 172.21.2.112:0/3893800133 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f70a40acfa0 msgr2=0x7f70a41a3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:09.487 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:09.475 7f70a8869700 1 --2- 172.21.2.112:0/3893800133 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f70a40acfa0 0x7f70a41a3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:09.487 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:09.475 7f70a8869700 1 -- 172.21.2.112:0/3893800133 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f70a40b5b80 msgr2=0x7f70a41a2e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:09.487 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:09.475 7f70a8869700 1 --2- 172.21.2.112:0/3893800133 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f70a40b5b80 0x7f70a41a2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:09.487 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:09.475 7f70a8869700 1 -- 172.21.2.112:0/3893800133 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70a40bcae0 con 0x7f70a4195650 2021-02-10T11:46:09.488 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:09.475 7f70a8869700 1 --2- 172.21.2.112:0/3893800133 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f70a4195650 0x7f70a40b5560 secure :-1 s=READY pgs=2836 cs=0 l=1 rev1=1 rx=0x7f709c007f90 tx=0x7f709c00ac50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:09.488 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:09.476 7f70a17fa700 1 -- 172.21.2.112:0/3893800133 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f709c007d00 con 0x7f70a4195650 2021-02-10T11:46:09.488 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:09.476 7f70a17fa700 1 -- 172.21.2.112:0/3893800133 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f709c001270 con 0x7f70a4195650 2021-02-10T11:46:09.488 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:09.476 7f70a17fa700 1 -- 172.21.2.112:0/3893800133 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f709c004c70 con 0x7f70a4195650 2021-02-10T11:46:09.488 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:09.476 7f70aaa4d700 1 -- 172.21.2.112:0/3893800133 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70a40bcae0 con 0x7f70a4195650 2021-02-10T11:46:09.488 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:09.476 7f70aaa4d700 1 -- 172.21.2.112:0/3893800133 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70a40bcae0 con 0x7f70a4195650 2021-02-10T11:46:09.489 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:09.477 7f70a17fa700 1 -- 172.21.2.112:0/3893800133 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f709c007230 con 0x7f70a4195650 2021-02-10T11:46:09.489 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:09.477 7f70a17fa700 1 --2- 172.21.2.112:0/3893800133 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f70800186c0 0x7f708001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:09.489 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:09.477 7f70a17fa700 1 -- 172.21.2.112:0/3893800133 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(163..163 src has 1..163) v4 ==== 5617+0+0 (secure 0 0 0) 0x7f709c0328e0 con 0x7f70a4195650 2021-02-10T11:46:09.489 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:09.477 7f70a3fff700 1 --2- 172.21.2.112:0/3893800133 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f70800186c0 0x7f708001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:09.489 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:09.478 7f70a3fff700 1 --2- 172.21.2.112:0/3893800133 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f70800186c0 0x7f708001abc0 crc :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:09.510 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:09.508 7f70aaa4d700 1 -- 172.21.2.112:0/3893800133 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70a40b6010 con 0x7f70a4195650 2021-02-10T11:46:09.513 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:09.511 7f70a17fa700 1 -- 172.21.2.112:0/3893800133 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f709c006f20 con 0x7f70a4195650 2021-02-10T11:46:09.642 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:09.640 7f70aaa4d700 1 -- 172.21.2.112:0/3893800133 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f70a40b6010 con 0x7f70a4195650 2021-02-10T11:46:09.643 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:09.640 7f70a17fa700 1 -- 172.21.2.112:0/3893800133 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 140 v140) v1 ==== 95+0+2105 (secure 0 0 0) 0x7f709c007130 con 0x7f70a4195650 2021-02-10T11:46:09.643 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:46:09.644 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":140,"default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":12306,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/2635651262","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":2635651262},{"type":"v1","addr":"172.21.2.112:6833","nonce":2635651262}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":135}],"filesystems":[{"mdsmap":{"epoch":139,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:45:30.416714","modified":"2021-02-10 11:45:32.783510","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8920},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8920":{"gid":8920,"name":"b","rank":0,"incarnation":136,"state":"up:active","state_seq":3,"addr":"172.21.2.120:6835/555399099","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":555399099},{"type":"v1","addr":"172.21.2.120:6835","nonce":555399099}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[35],"metadata_pool":34,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":17}]} 2021-02-10T11:46:09.644 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 140 2021-02-10T11:46:09.645 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:09.642 7f70aaa4d700 1 -- 172.21.2.112:0/3893800133 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f70800186c0 msgr2=0x7f708001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:09.645 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:09.642 7f70aaa4d700 1 --2- 172.21.2.112:0/3893800133 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f70800186c0 0x7f708001abc0 crc :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:09.645 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:09.642 7f70aaa4d700 1 -- 172.21.2.112:0/3893800133 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f70a4195650 msgr2=0x7f70a40b5560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:09.645 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:09.642 7f70aaa4d700 1 --2- 172.21.2.112:0/3893800133 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f70a4195650 0x7f70a40b5560 secure :-1 s=READY pgs=2836 cs=0 l=1 rev1=1 rx=0x7f709c007f90 tx=0x7f709c00ac50).stop 2021-02-10T11:46:09.646 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:09.642 7f70aaa4d700 1 -- 172.21.2.112:0/3893800133 shutdown_connections 2021-02-10T11:46:09.646 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:09.642 7f70aaa4d700 1 --2- 172.21.2.112:0/3893800133 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f70800186c0 0x7f708001abc0 unknown :-1 s=CLOSED pgs=1288 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:09.646 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:09.642 7f70aaa4d700 1 --2- 172.21.2.112:0/3893800133 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f70a40b5b80 0x7f70a41a2e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:09.646 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:09.642 7f70aaa4d700 1 --2- 172.21.2.112:0/3893800133 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f70a40acfa0 0x7f70a41a3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:09.646 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:09.642 7f70aaa4d700 1 --2- 172.21.2.112:0/3893800133 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f70a4195650 0x7f70a40b5560 unknown :-1 s=CLOSED pgs=2836 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:09.646 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:09.642 7f70aaa4d700 1 -- 172.21.2.112:0/3893800133 >> 172.21.2.112:0/3893800133 conn(0x7f70a40b3040 msgr2=0x7f70a40c3250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:09.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:09.643 7f70aaa4d700 1 -- 172.21.2.112:0/3893800133 shutdown_connections 2021-02-10T11:46:09.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:09.643 7f70aaa4d700 1 -- 172.21.2.112:0/3893800133 wait complete. 2021-02-10T11:46:09.654 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs fail cephfs 2021-02-10T11:46:10.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.070 7fddd779e700 1 Processor -- start 2021-02-10T11:46:10.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.070 7fddd779e700 1 -- start start 2021-02-10T11:46:10.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.071 7fddd779e700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fddd00d0340 0x7fddd00ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:10.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.071 7fddd779e700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fddd00d1c70 0x7fddd00efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:10.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.071 7fddd779e700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddd00f0b10 con 0x7fddd00d1870 2021-02-10T11:46:10.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.071 7fddd779e700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fddd00e5ed0 con 0x7fddd00d1c70 2021-02-10T11:46:10.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.071 7fddd779e700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddd01b0be0 con 0x7fddd00d0340 2021-02-10T11:46:10.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.072 7fddd5dbb700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fddd00d1c70 0x7fddd00efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:10.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.072 7fddd4db9700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fddd00d0340 0x7fddd00ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:10.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.072 7fddd55ba700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7fddd00d1870 0x7fddd01b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:52312/0 (socket says 172.21.2.112:52312) 2021-02-10T11:46:10.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.072 7fddd55ba700 1 -- 172.21.2.112:0/2800376161 learned_addr learned my addr 172.21.2.112:0/2800376161 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:46:10.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.073 7fddd4db9700 1 -- 172.21.2.112:0/2800376161 >> v2:172.21.2.120:3300/0 conn(0x7fddd00d1c70 msgr2=0x7fddd00efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:46:10.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.073 7fddd4db9700 1 --2- 172.21.2.112:0/2800376161 >> v2:172.21.2.120:3300/0 conn(0x7fddd00d1c70 0x7fddd00efc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:10.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.073 7fddd4db9700 1 -- 172.21.2.112:0/2800376161 >> v1:172.21.2.120:6789/0 conn(0x7fddd00d1870 legacy=0x7fddd01b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:10.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.073 7fddd4db9700 1 -- 172.21.2.112:0/2800376161 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddd00f20a0 con 0x7fddd00d0340 2021-02-10T11:46:10.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.073 7fddd5dbb700 1 --2- 172.21.2.112:0/2800376161 >> v2:172.21.2.120:3300/0 conn(0x7fddd00d1c70 0x7fddd00efc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:46:10.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.073 7fddd4db9700 1 --2- 172.21.2.112:0/2800376161 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fddd00d0340 0x7fddd00ef640 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 rx=0x7fddcc006650 tx=0x7fddcc009a80).ready entity=mon.2 client_cookie=900180b2654df10 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:10.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.075 7fddc3fff700 1 -- 172.21.2.112:0/2800376161 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fddcc0009b0 con 0x7fddd00d0340 2021-02-10T11:46:10.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.075 7fddc3fff700 1 -- 172.21.2.112:0/2800376161 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddcc00c0c0 con 0x7fddd00d0340 2021-02-10T11:46:10.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.075 7fddc3fff700 1 -- 172.21.2.112:0/2800376161 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fddcc00c5e0 con 0x7fddd00d0340 2021-02-10T11:46:10.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.075 7fddd779e700 1 -- 172.21.2.112:0/2800376161 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fddd00d0340 msgr2=0x7fddd00ef640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:10.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.075 7fddd779e700 1 --2- 172.21.2.112:0/2800376161 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fddd00d0340 0x7fddd00ef640 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 rx=0x7fddcc006650 tx=0x7fddcc009a80).stop 2021-02-10T11:46:10.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.075 7fddd779e700 1 -- 172.21.2.112:0/2800376161 shutdown_connections 2021-02-10T11:46:10.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.075 7fddd779e700 1 --2- 172.21.2.112:0/2800376161 >> v2:172.21.2.120:3300/0 conn(0x7fddd00d1c70 0x7fddd00efc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:10.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.075 7fddd779e700 1 --2- 172.21.2.112:0/2800376161 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fddd00d0340 0x7fddd00ef640 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:10.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.075 7fddd779e700 1 -- 172.21.2.112:0/2800376161 >> 172.21.2.112:0/2800376161 conn(0x7fddd00ce450 msgr2=0x7fddd00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:10.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.075 7fddd779e700 1 -- 172.21.2.112:0/2800376161 shutdown_connections 2021-02-10T11:46:10.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.075 7fddd779e700 1 -- 172.21.2.112:0/2800376161 wait complete. 2021-02-10T11:46:10.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.089 7fddd779e700 1 Processor -- start 2021-02-10T11:46:10.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.089 7fddd779e700 1 -- start start 2021-02-10T11:46:10.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.089 7fddd779e700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fddd01a06a0 0x7fddd009f380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:10.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.089 7fddd779e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fddd009f9a0 0x7fddd008b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:10.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.089 7fddd779e700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fddd008bb80 0x7fddd008bf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:10.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.089 7fddd779e700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddd008c780 con 0x7fddd009f9a0 2021-02-10T11:46:10.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.089 7fddd779e700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddd009fda0 con 0x7fddd008bb80 2021-02-10T11:46:10.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.089 7fddd779e700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddd008cd00 con 0x7fddd01a06a0 2021-02-10T11:46:10.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.090 7fddd4db9700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fddd009f9a0 0x7fddd008b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:10.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.090 7fddd4db9700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fddd009f9a0 0x7fddd008b560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:59298/0 (socket says 172.21.2.112:59298) 2021-02-10T11:46:10.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.090 7fddd4db9700 1 -- 172.21.2.112:0/285356081 learned_addr learned my addr 172.21.2.112:0/285356081 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:46:10.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.090 7fddd4db9700 1 -- 172.21.2.112:0/285356081 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fddd01a06a0 msgr2=0x7fddd009f380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:10.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.090 7fddd4db9700 1 --2- 172.21.2.112:0/285356081 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fddd01a06a0 0x7fddd009f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:10.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.090 7fddd4db9700 1 -- 172.21.2.112:0/285356081 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fddd008bb80 msgr2=0x7fddd008bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:10.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.090 7fddd4db9700 1 --2- 172.21.2.112:0/285356081 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fddd008bb80 0x7fddd008bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:10.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.090 7fddd4db9700 1 -- 172.21.2.112:0/285356081 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddd008cf20 con 0x7fddd009f9a0 2021-02-10T11:46:10.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.090 7fddd4db9700 1 --2- 172.21.2.112:0/285356081 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fddd009f9a0 0x7fddd008b560 secure :-1 s=READY pgs=2837 cs=0 l=1 rev1=1 rx=0x7fddcc009630 tx=0x7fddcc00b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:10.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.090 7fddc1ffb700 1 -- 172.21.2.112:0/285356081 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fddcc0093a0 con 0x7fddd009f9a0 2021-02-10T11:46:10.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.090 7fddc1ffb700 1 -- 172.21.2.112:0/285356081 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddcc00bd70 con 0x7fddd009f9a0 2021-02-10T11:46:10.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.090 7fddc1ffb700 1 -- 172.21.2.112:0/285356081 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fddcc005560 con 0x7fddd009f9a0 2021-02-10T11:46:10.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.090 7fddd779e700 1 -- 172.21.2.112:0/285356081 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddd008cf20 con 0x7fddd009f9a0 2021-02-10T11:46:10.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.091 7fddd779e700 1 -- 172.21.2.112:0/285356081 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddd008d4b0 con 0x7fddd009f9a0 2021-02-10T11:46:10.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.091 7fddc1ffb700 1 -- 172.21.2.112:0/285356081 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fddcc00a000 con 0x7fddd009f9a0 2021-02-10T11:46:10.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.091 7fddc1ffb700 1 --2- 172.21.2.112:0/285356081 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fddb00186c0 0x7fddb001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:10.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.091 7fddc1ffb700 1 -- 172.21.2.112:0/285356081 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(163..163 src has 1..163) v4 ==== 5617+0+0 (secure 0 0 0) 0x7fddcc0325b0 con 0x7fddd009f9a0 2021-02-10T11:46:10.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.092 7fddd55ba700 1 --2- 172.21.2.112:0/285356081 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fddb00186c0 0x7fddb001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:10.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.092 7fddd55ba700 1 --2- 172.21.2.112:0/285356081 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fddb00186c0 0x7fddb001abc0 crc :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:10.123 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.120 7fddd779e700 1 -- 172.21.2.112:0/285356081 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddd00ebc50 con 0x7fddd009f9a0 2021-02-10T11:46:10.126 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.123 7fddc1ffb700 1 -- 172.21.2.112:0/285356081 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fddcc009340 con 0x7fddd009f9a0 2021-02-10T11:46:10.254 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.251 7fddd779e700 1 -- 172.21.2.112:0/285356081 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fddd00f0b10 con 0x7fddd009f9a0 2021-02-10T11:46:10.267 INFO:tasks.ceph.mon.a.gibba012.stderr:2021-02-10 11:46:10.264 7fc3fe6e2700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-02-10T11:46:10.272 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.269 7fddc1ffb700 1 -- 172.21.2.112:0/285356081 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v141) v1 ==== 161+0+0 (secure 0 0 0) 0x7fddcc00a000 con 0x7fddd009f9a0 2021-02-10T11:46:10.285 INFO:teuthology.orchestra.run.gibba012.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2021-02-10T11:46:10.287 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.284 7fddd779e700 1 -- 172.21.2.112:0/285356081 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fddb00186c0 msgr2=0x7fddb001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:10.287 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.284 7fddd779e700 1 --2- 172.21.2.112:0/285356081 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fddb00186c0 0x7fddb001abc0 crc :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:10.287 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.284 7fddd779e700 1 -- 172.21.2.112:0/285356081 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fddd009f9a0 msgr2=0x7fddd008b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:10.287 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.284 7fddd779e700 1 --2- 172.21.2.112:0/285356081 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fddd009f9a0 0x7fddd008b560 secure :-1 s=READY pgs=2837 cs=0 l=1 rev1=1 rx=0x7fddcc009630 tx=0x7fddcc00b980).stop 2021-02-10T11:46:10.288 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.284 7fddd779e700 1 -- 172.21.2.112:0/285356081 shutdown_connections 2021-02-10T11:46:10.288 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.284 7fddd779e700 1 --2- 172.21.2.112:0/285356081 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fddb00186c0 0x7fddb001abc0 unknown :-1 s=CLOSED pgs=1289 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:10.288 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.284 7fddd779e700 1 --2- 172.21.2.112:0/285356081 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fddd008bb80 0x7fddd008bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:10.288 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.284 7fddd779e700 1 --2- 172.21.2.112:0/285356081 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fddd009f9a0 0x7fddd008b560 unknown :-1 s=CLOSED pgs=2837 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:10.288 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.284 7fddd779e700 1 --2- 172.21.2.112:0/285356081 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fddd01a06a0 0x7fddd009f380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:10.288 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.284 7fddd779e700 1 -- 172.21.2.112:0/285356081 >> 172.21.2.112:0/285356081 conn(0x7fddd00b83b0 msgr2=0x7fddd00a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:10.288 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.285 7fddd779e700 1 -- 172.21.2.112:0/285356081 shutdown_connections 2021-02-10T11:46:10.289 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.285 7fddd779e700 1 -- 172.21.2.112:0/285356081 wait complete. 2021-02-10T11:46:10.296 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:46:10.324 INFO:tasks.ceph.mds.b.gibba020.stdout:starting mds.b at 2021-02-10T11:46:10.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.705 7fe717777700 1 Processor -- start 2021-02-10T11:46:10.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.705 7fe717777700 1 -- start start 2021-02-10T11:46:10.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.706 7fe717777700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fe7100d1870 0x7fe7101b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:10.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.706 7fe717777700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fe7101b0a10 0x7fe7100cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:10.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.706 7fe717777700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7100f0b10 con 0x7fe7100d1c70 2021-02-10T11:46:10.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.706 7fe717777700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7100e5ed0 con 0x7fe7100d1870 2021-02-10T11:46:10.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.706 7fe717777700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7100f0650 con 0x7fe7101b0a10 2021-02-10T11:46:10.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.706 7fe715593700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fe7100d1870 0x7fe7101b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:10.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.706 7fe715593700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fe7100d1870 0x7fe7101b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:59306/0 (socket says 172.21.2.112:59306) 2021-02-10T11:46:10.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.706 7fe715593700 1 -- 172.21.2.112:0/3528025388 learned_addr learned my addr 172.21.2.112:0/3528025388 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:46:10.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.706 7fe715593700 1 -- 172.21.2.112:0/3528025388 >> v2:172.21.2.120:3300/0 conn(0x7fe7101b0a10 msgr2=0x7fe7100cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:46:10.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.706 7fe715593700 1 --2- 172.21.2.112:0/3528025388 >> v2:172.21.2.120:3300/0 conn(0x7fe7101b0a10 0x7fe7100cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:10.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.706 7fe715593700 1 -- 172.21.2.112:0/3528025388 >> v1:172.21.2.120:6789/0 conn(0x7fe7100d1c70 legacy=0x7fe7100cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:46:10.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.706 7fe715593700 1 -- 172.21.2.112:0/3528025388 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7100f0840 con 0x7fe7100d1870 2021-02-10T11:46:10.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.706 7fe715593700 1 --2- 172.21.2.112:0/3528025388 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe7100d1870 0x7fe7101b03f0 secure :-1 s=READY pgs=2839 cs=0 l=1 rev1=1 rx=0x7fe704006700 tx=0x7fe704009b80).ready entity=mon.0 client_cookie=d1721476eb7636ba server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:10.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.707 7fe703fff700 1 -- 172.21.2.112:0/3528025388 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe70400c0c0 con 0x7fe7100d1870 2021-02-10T11:46:10.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.707 7fe703fff700 1 -- 172.21.2.112:0/3528025388 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe70400c440 con 0x7fe7100d1870 2021-02-10T11:46:10.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.707 7fe703fff700 1 -- 172.21.2.112:0/3528025388 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe70400c8a0 con 0x7fe7100d1870 2021-02-10T11:46:10.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.707 7fe717777700 1 -- 172.21.2.112:0/3528025388 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe7100d1870 msgr2=0x7fe7101b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:10.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.707 7fe717777700 1 --2- 172.21.2.112:0/3528025388 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe7100d1870 0x7fe7101b03f0 secure :-1 s=READY pgs=2839 cs=0 l=1 rev1=1 rx=0x7fe704006700 tx=0x7fe704009b80).stop 2021-02-10T11:46:10.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.707 7fe717777700 1 -- 172.21.2.112:0/3528025388 shutdown_connections 2021-02-10T11:46:10.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.707 7fe717777700 1 --2- 172.21.2.112:0/3528025388 >> v2:172.21.2.120:3300/0 conn(0x7fe7101b0a10 0x7fe7100cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:10.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.707 7fe717777700 1 --2- 172.21.2.112:0/3528025388 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe7100d1870 0x7fe7101b03f0 unknown :-1 s=CLOSED pgs=2839 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:10.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.707 7fe717777700 1 -- 172.21.2.112:0/3528025388 >> 172.21.2.112:0/3528025388 conn(0x7fe7100ce450 msgr2=0x7fe7100d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:10.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.708 7fe717777700 1 -- 172.21.2.112:0/3528025388 shutdown_connections 2021-02-10T11:46:10.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.708 7fe717777700 1 -- 172.21.2.112:0/3528025388 wait complete. 2021-02-10T11:46:10.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.722 7fe717777700 1 Processor -- start 2021-02-10T11:46:10.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.722 7fe717777700 1 -- start start 2021-02-10T11:46:10.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.723 7fe717777700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe71018a540 0x7fe710197480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:10.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.723 7fe717777700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe71018aa50 0x7fe7100b77d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:10.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.723 7fe717777700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe7100b7df0 0x7fe7100b81f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:10.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.723 7fe717777700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7101a79d0 con 0x7fe71018a540 2021-02-10T11:46:10.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.723 7fe717777700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7101a9e20 con 0x7fe7100b7df0 2021-02-10T11:46:10.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.723 7fe717777700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7101aa780 con 0x7fe71018aa50 2021-02-10T11:46:10.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.723 7fe715593700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe71018a540 0x7fe710197480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:10.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.723 7fe715593700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe71018a540 0x7fe710197480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:59308/0 (socket says 172.21.2.112:59308) 2021-02-10T11:46:10.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.723 7fe715593700 1 -- 172.21.2.112:0/4176282853 learned_addr learned my addr 172.21.2.112:0/4176282853 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:46:10.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.723 7fe715d94700 1 --2- 172.21.2.112:0/4176282853 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe7100b7df0 0x7fe7100b81f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:10.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.723 7fe715593700 1 -- 172.21.2.112:0/4176282853 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe71018aa50 msgr2=0x7fe7100b77d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:10.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.723 7fe715593700 1 --2- 172.21.2.112:0/4176282853 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe71018aa50 0x7fe7100b77d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:10.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.723 7fe715593700 1 -- 172.21.2.112:0/4176282853 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe7100b7df0 msgr2=0x7fe7100b81f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:10.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.723 7fe715593700 1 --2- 172.21.2.112:0/4176282853 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe7100b7df0 0x7fe7100b81f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:10.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.723 7fe715593700 1 -- 172.21.2.112:0/4176282853 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7101aa9a0 con 0x7fe71018a540 2021-02-10T11:46:10.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.723 7fe715593700 1 --2- 172.21.2.112:0/4176282853 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe71018a540 0x7fe710197480 secure :-1 s=READY pgs=2840 cs=0 l=1 rev1=1 rx=0x7fe70400b5e0 tx=0x7fe70400b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:10.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.723 7fe701ffb700 1 -- 172.21.2.112:0/4176282853 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe70400bd70 con 0x7fe71018a540 2021-02-10T11:46:10.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.723 7fe701ffb700 1 -- 172.21.2.112:0/4176282853 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe704002030 con 0x7fe71018a540 2021-02-10T11:46:10.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.723 7fe717777700 1 -- 172.21.2.112:0/4176282853 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7101aa9a0 con 0x7fe71018a540 2021-02-10T11:46:10.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.724 7fe717777700 1 -- 172.21.2.112:0/4176282853 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7101aa9a0 con 0x7fe71018a540 2021-02-10T11:46:10.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.724 7fe701ffb700 1 -- 172.21.2.112:0/4176282853 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe704002550 con 0x7fe71018a540 2021-02-10T11:46:10.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.724 7fe701ffb700 1 -- 172.21.2.112:0/4176282853 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe704000c90 con 0x7fe71018a540 2021-02-10T11:46:10.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.724 7fe701ffb700 1 --2- 172.21.2.112:0/4176282853 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe6f00186c0 0x7fe6f001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:10.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.724 7fe701ffb700 1 -- 172.21.2.112:0/4176282853 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(164..164 src has 1..164) v4 ==== 5675+0+0 (secure 0 0 0) 0x7fe7040326a0 con 0x7fe71018a540 2021-02-10T11:46:10.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.725 7fe714d92700 1 --2- 172.21.2.112:0/4176282853 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe6f00186c0 0x7fe6f001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:10.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.725 7fe714d92700 1 --2- 172.21.2.112:0/4176282853 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe6f00186c0 0x7fe6f001abc0 crc :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:10.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.756 7fe717777700 1 -- 172.21.2.112:0/4176282853 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7100ce450 con 0x7fe71018a540 2021-02-10T11:46:10.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.758 7fe701ffb700 1 -- 172.21.2.112:0/4176282853 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe70401f500 con 0x7fe71018a540 2021-02-10T11:46:10.891 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.887 7fe717777700 1 -- 172.21.2.112:0/4176282853 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe7100ce450 con 0x7fe71018a540 2021-02-10T11:46:10.891 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.887 7fe701ffb700 1 -- 172.21.2.112:0/4176282853 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 141 v141) v1 ==== 95+0+1763 (secure 0 0 0) 0x7fe70400bd70 con 0x7fe71018a540 2021-02-10T11:46:10.891 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 141 2021-02-10T11:46:10.892 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:46:10.892 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":141,"default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":12306,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/2635651262","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":2635651262},{"type":"v1","addr":"172.21.2.112:6833","nonce":2635651262}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":135}],"filesystems":[{"mdsmap":{"epoch":141,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:45:30.416714","modified":"2021-02-10 11:46:10.266351","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":164,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[35],"metadata_pool":34,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":17}]} 2021-02-10T11:46:10.892 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.889 7fe717777700 1 -- 172.21.2.112:0/4176282853 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe6f00186c0 msgr2=0x7fe6f001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:10.892 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.889 7fe717777700 1 --2- 172.21.2.112:0/4176282853 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe6f00186c0 0x7fe6f001abc0 crc :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:10.892 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.889 7fe717777700 1 -- 172.21.2.112:0/4176282853 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe71018a540 msgr2=0x7fe710197480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:10.892 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.889 7fe717777700 1 --2- 172.21.2.112:0/4176282853 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe71018a540 0x7fe710197480 secure :-1 s=READY pgs=2840 cs=0 l=1 rev1=1 rx=0x7fe70400b5e0 tx=0x7fe70400b980).stop 2021-02-10T11:46:10.893 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.889 7fe717777700 1 -- 172.21.2.112:0/4176282853 shutdown_connections 2021-02-10T11:46:10.893 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.889 7fe717777700 1 --2- 172.21.2.112:0/4176282853 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe6f00186c0 0x7fe6f001abc0 unknown :-1 s=CLOSED pgs=1290 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:10.893 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.889 7fe717777700 1 --2- 172.21.2.112:0/4176282853 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe7100b7df0 0x7fe7100b81f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:10.893 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.889 7fe717777700 1 --2- 172.21.2.112:0/4176282853 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe71018aa50 0x7fe7100b77d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:10.893 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.889 7fe717777700 1 --2- 172.21.2.112:0/4176282853 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe71018a540 0x7fe710197480 unknown :-1 s=CLOSED pgs=2840 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:10.893 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.889 7fe717777700 1 -- 172.21.2.112:0/4176282853 >> 172.21.2.112:0/4176282853 conn(0x7fe7100efc70 msgr2=0x7fe7100ef410 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:10.893 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.889 7fe717777700 1 -- 172.21.2.112:0/4176282853 shutdown_connections 2021-02-10T11:46:10.893 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:10.889 7fe717777700 1 -- 172.21.2.112:0/4176282853 wait complete. 2021-02-10T11:46:10.901 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2021-02-10T11:46:11.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:11.314 7f503616a700 1 Processor -- start 2021-02-10T11:46:11.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:11.314 7f503616a700 1 -- start start 2021-02-10T11:46:11.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:11.314 7f503616a700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f50300d1870 0x7f50301b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:11.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:11.314 7f503616a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f50300f0650 0x7f50300cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:11.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:11.314 7f503616a700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50300f0b10 con 0x7f50301b0a10 2021-02-10T11:46:11.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:11.314 7f503616a700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50300e5ed0 con 0x7f50300d1870 2021-02-10T11:46:11.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:11.314 7f503616a700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50300d1c70 con 0x7f50300f0650 2021-02-10T11:46:11.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:11.315 7f502f7fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f50300d1870 0x7f50301b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:11.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:11.315 7f502f7fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f50300d1870 0x7f50301b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:59316/0 (socket says 172.21.2.112:59316) 2021-02-10T11:46:11.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:11.315 7f502f7fe700 1 -- 172.21.2.112:0/3998045016 learned_addr learned my addr 172.21.2.112:0/3998045016 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:46:11.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:11.315 7f502ffff700 1 --2- 172.21.2.112:0/3998045016 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f50300f0650 0x7f50300cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:11.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:11.315 7f502f7fe700 1 -- 172.21.2.112:0/3998045016 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f50300f0650 msgr2=0x7f50300cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:46:11.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:11.315 7f502f7fe700 1 --2- 172.21.2.112:0/3998045016 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f50300f0650 0x7f50300cfb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:11.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:11.315 7f502f7fe700 1 -- 172.21.2.112:0/3998045016 >> v1:172.21.2.120:6789/0 conn(0x7f50301b0a10 legacy=0x7f50300cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:11.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:11.315 7f502f7fe700 1 -- 172.21.2.112:0/3998045016 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50300d1e90 con 0x7f50300d1870 2021-02-10T11:46:11.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:11.315 7f502f7fe700 1 --2- 172.21.2.112:0/3998045016 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f50300d1870 0x7f50301b03f0 secure :-1 s=READY pgs=2841 cs=0 l=1 rev1=1 rx=0x7f5018006700 tx=0x7f5018009b80).ready entity=mon.0 client_cookie=7176556759ea630c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:11.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:11.315 7f502e7fc700 1 -- 172.21.2.112:0/3998045016 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f501800c0c0 con 0x7f50300d1870 2021-02-10T11:46:11.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:11.315 7f502e7fc700 1 -- 172.21.2.112:0/3998045016 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f501800c340 con 0x7f50300d1870 2021-02-10T11:46:11.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:11.315 7f502e7fc700 1 -- 172.21.2.112:0/3998045016 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f501800c860 con 0x7f50300d1870 2021-02-10T11:46:11.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:11.315 7f503616a700 1 -- 172.21.2.112:0/3998045016 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f50300d1870 msgr2=0x7f50301b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:11.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:11.315 7f503616a700 1 --2- 172.21.2.112:0/3998045016 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f50300d1870 0x7f50301b03f0 secure :-1 s=READY pgs=2841 cs=0 l=1 rev1=1 rx=0x7f5018006700 tx=0x7f5018009b80).stop 2021-02-10T11:46:11.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:11.316 7f503616a700 1 -- 172.21.2.112:0/3998045016 shutdown_connections 2021-02-10T11:46:11.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:11.316 7f503616a700 1 --2- 172.21.2.112:0/3998045016 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f50300f0650 0x7f50300cfb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:11.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:11.316 7f503616a700 1 --2- 172.21.2.112:0/3998045016 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f50300d1870 0x7f50301b03f0 unknown :-1 s=CLOSED pgs=2841 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:11.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:11.316 7f503616a700 1 -- 172.21.2.112:0/3998045016 >> 172.21.2.112:0/3998045016 conn(0x7f50300ce450 msgr2=0x7f50300d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:11.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:11.316 7f503616a700 1 -- 172.21.2.112:0/3998045016 shutdown_connections 2021-02-10T11:46:11.335 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:11.316 7f503616a700 1 -- 172.21.2.112:0/3998045016 wait complete. 2021-02-10T11:46:11.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:11.330 7f503616a700 1 Processor -- start 2021-02-10T11:46:11.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:11.331 7f503616a700 1 -- start start 2021-02-10T11:46:11.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:11.331 7f503616a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f50301a4a80 0x7f50300906b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:11.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:11.331 7f503616a700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f50300c30f0 0x7f50300c34f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:11.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:11.331 7f503616a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f50300ef5a0 0x7f50300ef9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:11.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:11.331 7f503616a700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50300c3b10 con 0x7f50300ef5a0 2021-02-10T11:46:11.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:11.331 7f503616a700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50300ef020 con 0x7f50300c30f0 2021-02-10T11:46:11.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:11.331 7f503616a700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50300effc0 con 0x7f50301a4a80 2021-02-10T11:46:11.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:11.331 7f502ffff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f50300ef5a0 0x7f50300ef9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:11.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:11.331 7f502f7fe700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f50301a4a80 0x7f50300906b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:11.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:11.331 7f502f7fe700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f50301a4a80 0x7f50300906b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3301/0 says I am v2:172.21.2.112:59002/0 (socket says 172.21.2.112:59002) 2021-02-10T11:46:11.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:11.331 7f502f7fe700 1 -- 172.21.2.112:0/3001458245 learned_addr learned my addr 172.21.2.112:0/3001458245 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:46:11.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:11.331 7f502ffff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f50300ef5a0 0x7f50300ef9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:59324/0 (socket says 172.21.2.112:59324) 2021-02-10T11:46:11.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:11.331 7f502effd700 1 --2- 172.21.2.112:0/3001458245 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f50300c30f0 0x7f50300c34f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:11.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:11.332 7f502ffff700 1 -- 172.21.2.112:0/3001458245 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f50301a4a80 msgr2=0x7f50300906b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:11.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:11.332 7f502ffff700 1 --2- 172.21.2.112:0/3001458245 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f50301a4a80 0x7f50300906b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:11.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:11.332 7f502ffff700 1 -- 172.21.2.112:0/3001458245 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f50300c30f0 msgr2=0x7f50300c34f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:11.339 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:11.332 7f502ffff700 1 --2- 172.21.2.112:0/3001458245 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f50300c30f0 0x7f50300c34f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:11.339 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:11.332 7f502ffff700 1 -- 172.21.2.112:0/3001458245 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50300ebe70 con 0x7f50300ef5a0 2021-02-10T11:46:11.339 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:11.332 7f502f7fe700 1 --2- 172.21.2.112:0/3001458245 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f50301a4a80 0x7f50300906b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:46:11.339 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:11.332 7f502ffff700 1 --2- 172.21.2.112:0/3001458245 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f50300ef5a0 0x7f50300ef9a0 secure :-1 s=READY pgs=2842 cs=0 l=1 rev1=1 rx=0x7f502800b8a0 tx=0x7f502800bc40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:11.339 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:11.332 7f5027fff700 1 -- 172.21.2.112:0/3001458245 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5028000f90 con 0x7f50300ef5a0 2021-02-10T11:46:11.339 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:11.332 7f5027fff700 1 -- 172.21.2.112:0/3001458245 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50280066f0 con 0x7f50300ef5a0 2021-02-10T11:46:11.339 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:11.332 7f5027fff700 1 -- 172.21.2.112:0/3001458245 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f502800c4e0 con 0x7f50300ef5a0 2021-02-10T11:46:11.339 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:11.332 7f503616a700 1 -- 172.21.2.112:0/3001458245 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50300ebe70 con 0x7f50300ef5a0 2021-02-10T11:46:11.339 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:11.332 7f503616a700 1 -- 172.21.2.112:0/3001458245 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50300ec400 con 0x7f50300ef5a0 2021-02-10T11:46:11.339 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:11.332 7f5027fff700 1 -- 172.21.2.112:0/3001458245 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f50280011f0 con 0x7f50300ef5a0 2021-02-10T11:46:11.339 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:11.332 7f5027fff700 1 --2- 172.21.2.112:0/3001458245 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f50140186c0 0x7f501401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:11.339 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:11.332 7f5027fff700 1 -- 172.21.2.112:0/3001458245 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(164..164 src has 1..164) v4 ==== 5675+0+0 (secure 0 0 0) 0x7f50280066f0 con 0x7f50300ef5a0 2021-02-10T11:46:11.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:11.333 7f502f7fe700 1 --2- 172.21.2.112:0/3001458245 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f50140186c0 0x7f501401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:11.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:11.333 7f502f7fe700 1 --2- 172.21.2.112:0/3001458245 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f50140186c0 0x7f501401abc0 crc :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:11.368 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:11.364 7f503616a700 1 -- 172.21.2.112:0/3001458245 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50300f0b10 con 0x7f50300ef5a0 2021-02-10T11:46:11.370 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:11.367 7f5027fff700 1 -- 172.21.2.112:0/3001458245 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5028000f90 con 0x7f50300ef5a0 2021-02-10T11:46:11.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:11.494 7f503616a700 1 -- 172.21.2.112:0/3001458245 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f50300f0b10 con 0x7f50300ef5a0 2021-02-10T11:46:12.281 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:12.278 7f5027fff700 1 -- 172.21.2.112:0/3001458245 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v142) v1 ==== 104+0+0 (secure 0 0 0) 0x7f50280066f0 con 0x7f50300ef5a0 2021-02-10T11:46:12.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:12.316 7f503616a700 1 -- 172.21.2.112:0/3001458245 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f50140186c0 msgr2=0x7f501401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:12.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:12.316 7f503616a700 1 --2- 172.21.2.112:0/3001458245 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f50140186c0 0x7f501401abc0 crc :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:12.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:12.316 7f503616a700 1 -- 172.21.2.112:0/3001458245 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f50300ef5a0 msgr2=0x7f50300ef9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:12.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:12.316 7f503616a700 1 --2- 172.21.2.112:0/3001458245 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f50300ef5a0 0x7f50300ef9a0 secure :-1 s=READY pgs=2842 cs=0 l=1 rev1=1 rx=0x7f502800b8a0 tx=0x7f502800bc40).stop 2021-02-10T11:46:12.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:12.316 7f503616a700 1 -- 172.21.2.112:0/3001458245 shutdown_connections 2021-02-10T11:46:12.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:12.316 7f503616a700 1 --2- 172.21.2.112:0/3001458245 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f50140186c0 0x7f501401abc0 unknown :-1 s=CLOSED pgs=1291 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:12.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:12.316 7f503616a700 1 --2- 172.21.2.112:0/3001458245 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f50300c30f0 0x7f50300c34f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:12.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:12.316 7f503616a700 1 --2- 172.21.2.112:0/3001458245 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f50300ef5a0 0x7f50300ef9a0 unknown :-1 s=CLOSED pgs=2842 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:12.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:12.316 7f503616a700 1 --2- 172.21.2.112:0/3001458245 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f50301a4a80 0x7f50300906b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:12.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:12.316 7f503616a700 1 -- 172.21.2.112:0/3001458245 >> 172.21.2.112:0/3001458245 conn(0x7f50300a6bb0 msgr2=0x7f5030095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:12.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:12.316 7f503616a700 1 -- 172.21.2.112:0/3001458245 shutdown_connections 2021-02-10T11:46:12.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:12.316 7f503616a700 1 -- 172.21.2.112:0/3001458245 wait complete. 2021-02-10T11:46:12.327 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2021-02-10T11:46:12.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:12.739 7fcd73ca9700 1 Processor -- start 2021-02-10T11:46:12.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:12.739 7fcd73ca9700 1 -- start start 2021-02-10T11:46:12.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:12.740 7fcd73ca9700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fcd6c10b000 0x7fcd6c0cf760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:12.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:12.740 7fcd73ca9700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd6c0fe1d0 con 0x7fcd6c0d1c80 2021-02-10T11:46:12.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:12.740 7fcd73ca9700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd6c1b0cc0 con 0x7fcd6c0d1880 2021-02-10T11:46:12.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:12.740 7fcd73ca9700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd6c109a70 con 0x7fcd6c10b000 2021-02-10T11:46:12.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:12.740 7fcd712c4700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fcd6c0d1c80 0x7fcd6c0cef70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:58790/0 (socket says 172.21.2.112:58790) 2021-02-10T11:46:12.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:12.740 7fcd712c4700 1 -- 172.21.2.112:0/653237500 learned_addr learned my addr 172.21.2.112:0/653237500 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:46:12.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:12.740 7fcd722c6700 1 --2- 172.21.2.112:0/653237500 >> v2:172.21.2.120:3300/0 conn(0x7fcd6c10b000 0x7fcd6c0cf760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:12.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:12.740 7fcd70ac3700 1 -- 172.21.2.112:0/653237500 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1222555868 0 0) 0x7fcd5c0009b0 con 0x7fcd6c0d1c80 2021-02-10T11:46:12.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:12.740 7fcd70ac3700 1 -- 172.21.2.112:0/653237500 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd58002fe0 con 0x7fcd6c0d1c80 2021-02-10T11:46:12.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:12.740 7fcd70ac3700 1 -- 172.21.2.112:0/653237500 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 266741619 0 0) 0x7fcd5c0010b0 con 0x7fcd6c0d1c80 2021-02-10T11:46:12.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:12.740 7fcd70ac3700 1 -- 172.21.2.112:0/653237500 >> v2:172.21.2.120:3300/0 conn(0x7fcd6c10b000 msgr2=0x7fcd6c0cf760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:46:12.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:12.740 7fcd70ac3700 1 --2- 172.21.2.112:0/653237500 >> v2:172.21.2.120:3300/0 conn(0x7fcd6c10b000 0x7fcd6c0cf760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:12.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:12.740 7fcd70ac3700 1 -- 172.21.2.112:0/653237500 >> v1:172.21.2.120:6789/0 conn(0x7fcd6c0d1880 legacy=0x7fcd6c1b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:12.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:12.740 7fcd70ac3700 1 -- 172.21.2.112:0/653237500 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd6c10b400 con 0x7fcd6c0d1c80 2021-02-10T11:46:12.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:12.741 7fcd70ac3700 1 -- 172.21.2.112:0/653237500 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7fcd5c0015b0 con 0x7fcd6c0d1c80 2021-02-10T11:46:12.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:12.741 7fcd70ac3700 1 -- 172.21.2.112:0/653237500 >> v1:172.21.2.112:6789/0 conn(0x7fcd6c0d1c80 legacy=0x7fcd6c0cef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:12.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:12.741 7fcd70ac3700 1 --2- 172.21.2.112:0/653237500 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcd58003ac0 0x7fcd58005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:12.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:12.741 7fcd70ac3700 1 --2- 172.21.2.112:0/653237500 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcd58006790 0x7fcd58008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:12.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:12.741 7fcd70ac3700 1 --2- 172.21.2.112:0/653237500 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcd58009590 0x7fcd5800bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:12.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:12.741 7fcd70ac3700 1 -- 172.21.2.112:0/653237500 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd5800c300 con 0x7fcd58009590 2021-02-10T11:46:12.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:12.741 7fcd70ac3700 1 -- 172.21.2.112:0/653237500 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd5800c520 con 0x7fcd58003ac0 2021-02-10T11:46:12.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:12.741 7fcd70ac3700 1 -- 172.21.2.112:0/653237500 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd5800c740 con 0x7fcd58006790 2021-02-10T11:46:12.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:12.741 7fcd722c6700 1 --2- 172.21.2.112:0/653237500 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcd58009590 0x7fcd5800bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:12.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:12.741 7fcd71ac5700 1 --2- 172.21.2.112:0/653237500 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcd58003ac0 0x7fcd58005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:12.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:12.741 7fcd712c4700 1 --2- 172.21.2.112:0/653237500 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcd58006790 0x7fcd58008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:12.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:12.741 7fcd722c6700 1 -- 172.21.2.112:0/653237500 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcd58006790 msgr2=0x7fcd58008c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:46:12.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:12.741 7fcd722c6700 1 --2- 172.21.2.112:0/653237500 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcd58006790 0x7fcd58008c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:12.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:12.741 7fcd722c6700 1 -- 172.21.2.112:0/653237500 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcd58003ac0 msgr2=0x7fcd58005ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:46:12.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:12.741 7fcd722c6700 1 --2- 172.21.2.112:0/653237500 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcd58003ac0 0x7fcd58005ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:12.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:12.741 7fcd722c6700 1 -- 172.21.2.112:0/653237500 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd580509b0 con 0x7fcd58009590 2021-02-10T11:46:12.763 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:12.741 7fcd71ac5700 1 --2- 172.21.2.112:0/653237500 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcd58003ac0 0x7fcd58005ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:46:12.763 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:12.741 7fcd722c6700 1 -- 172.21.2.112:0/653237500 reap_dead start 2021-02-10T11:46:12.763 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:12.741 7fcd722c6700 1 --2- 172.21.2.112:0/653237500 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcd58009590 0x7fcd5800bab0 secure :-1 s=READY pgs=2844 cs=0 l=1 rev1=1 rx=0x7fcd64006a40 tx=0x7fcd6400c0c0).ready entity=mon.0 client_cookie=c94f1a31ab4845ca server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:12.763 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:12.742 7fcd70ac3700 1 -- 172.21.2.112:0/653237500 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcd64006a80 con 0x7fcd58009590 2021-02-10T11:46:12.763 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:12.742 7fcd70ac3700 1 -- 172.21.2.112:0/653237500 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd64006a80 con 0x7fcd58009590 2021-02-10T11:46:12.763 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:12.742 7fcd70ac3700 1 -- 172.21.2.112:0/653237500 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcd64007070 con 0x7fcd58009590 2021-02-10T11:46:12.763 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:12.742 7fcd73ca9700 1 -- 172.21.2.112:0/653237500 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcd58009590 msgr2=0x7fcd5800bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:12.763 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:12.742 7fcd73ca9700 1 --2- 172.21.2.112:0/653237500 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcd58009590 0x7fcd5800bab0 secure :-1 s=READY pgs=2844 cs=0 l=1 rev1=1 rx=0x7fcd64006a40 tx=0x7fcd6400c0c0).stop 2021-02-10T11:46:12.763 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:12.742 7fcd73ca9700 1 -- 172.21.2.112:0/653237500 shutdown_connections 2021-02-10T11:46:12.763 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:12.742 7fcd73ca9700 1 --2- 172.21.2.112:0/653237500 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcd58009590 0x7fcd5800bab0 unknown :-1 s=CLOSED pgs=2844 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:12.763 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:12.742 7fcd73ca9700 1 -- 172.21.2.112:0/653237500 >> 172.21.2.112:0/653237500 conn(0x7fcd6c0ce430 msgr2=0x7fcd6c0d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:12.763 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:12.742 7fcd73ca9700 1 -- 172.21.2.112:0/653237500 shutdown_connections 2021-02-10T11:46:12.763 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:12.742 7fcd73ca9700 1 -- 172.21.2.112:0/653237500 wait complete. 2021-02-10T11:46:12.764 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:12.757 7fcd73ca9700 1 Processor -- start 2021-02-10T11:46:12.764 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:12.757 7fcd73ca9700 1 -- start start 2021-02-10T11:46:12.766 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:12.757 7fcd73ca9700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcd6c188400 0x7fcd6c188800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:12.766 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:12.757 7fcd73ca9700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcd6c188e20 0x7fcd6c189220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:12.767 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:12.757 7fcd73ca9700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcd6c09cbc0 0x7fcd6c09cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:12.767 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:12.757 7fcd73ca9700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd6c187e80 con 0x7fcd6c188e20 2021-02-10T11:46:12.767 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:12.757 7fcd73ca9700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd6c0b27c0 con 0x7fcd6c188400 2021-02-10T11:46:12.767 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:12.757 7fcd73ca9700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd6c0b29a0 con 0x7fcd6c09cbc0 2021-02-10T11:46:12.767 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:12.757 7fcd712c4700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcd6c188e20 0x7fcd6c189220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:12.767 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:12.757 7fcd712c4700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcd6c188e20 0x7fcd6c189220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:59340/0 (socket says 172.21.2.112:59340) 2021-02-10T11:46:12.767 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:12.757 7fcd712c4700 1 -- 172.21.2.112:0/217593347 learned_addr learned my addr 172.21.2.112:0/217593347 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:46:12.767 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:12.758 7fcd71ac5700 1 --2- 172.21.2.112:0/217593347 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcd6c188400 0x7fcd6c188800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:12.767 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:12.758 7fcd722c6700 1 --2- 172.21.2.112:0/217593347 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcd6c09cbc0 0x7fcd6c09cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:12.767 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:12.758 7fcd712c4700 1 -- 172.21.2.112:0/217593347 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcd6c09cbc0 msgr2=0x7fcd6c09cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:12.767 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:12.758 7fcd712c4700 1 --2- 172.21.2.112:0/217593347 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcd6c09cbc0 0x7fcd6c09cfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:12.767 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:12.758 7fcd712c4700 1 -- 172.21.2.112:0/217593347 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcd6c188400 msgr2=0x7fcd6c188800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:12.768 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:12.758 7fcd712c4700 1 --2- 172.21.2.112:0/217593347 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcd6c188400 0x7fcd6c188800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:12.768 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:12.758 7fcd712c4700 1 -- 172.21.2.112:0/217593347 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd6c09d5e0 con 0x7fcd6c188e20 2021-02-10T11:46:12.768 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:12.758 7fcd722c6700 1 --2- 172.21.2.112:0/217593347 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcd6c09cbc0 0x7fcd6c09cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:46:12.768 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:12.758 7fcd712c4700 1 --2- 172.21.2.112:0/217593347 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcd6c188e20 0x7fcd6c189220 secure :-1 s=READY pgs=2845 cs=0 l=1 rev1=1 rx=0x7fcd5c001c60 tx=0x7fcd5c00a8f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:12.768 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:12.758 7fcd627fc700 1 -- 172.21.2.112:0/217593347 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcd5c00ace0 con 0x7fcd6c188e20 2021-02-10T11:46:12.768 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:12.758 7fcd627fc700 1 -- 172.21.2.112:0/217593347 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd5c004350 con 0x7fcd6c188e20 2021-02-10T11:46:12.768 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:12.758 7fcd73ca9700 1 -- 172.21.2.112:0/217593347 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd6c09d5e0 con 0x7fcd6c188e20 2021-02-10T11:46:12.768 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:12.758 7fcd627fc700 1 -- 172.21.2.112:0/217593347 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcd5c004870 con 0x7fcd6c188e20 2021-02-10T11:46:12.768 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:12.758 7fcd73ca9700 1 -- 172.21.2.112:0/217593347 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd6c09d5e0 con 0x7fcd6c188e20 2021-02-10T11:46:12.768 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:12.759 7fcd627fc700 1 -- 172.21.2.112:0/217593347 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fcd5c004530 con 0x7fcd6c188e20 2021-02-10T11:46:12.768 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:12.759 7fcd627fc700 1 --2- 172.21.2.112:0/217593347 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcd540186d0 0x7fcd5401abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:12.768 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:12.759 7fcd627fc700 1 -- 172.21.2.112:0/217593347 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(164..164 src has 1..164) v4 ==== 5675+0+0 (secure 0 0 0) 0x7fcd5c0327f0 con 0x7fcd6c188e20 2021-02-10T11:46:12.768 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:12.759 7fcd71ac5700 1 --2- 172.21.2.112:0/217593347 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcd540186d0 0x7fcd5401abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:12.768 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:12.759 7fcd71ac5700 1 --2- 172.21.2.112:0/217593347 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcd540186d0 0x7fcd5401abd0 crc :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:12.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:12.789 7fcd73ca9700 1 -- 172.21.2.112:0/217593347 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd6c082a70 con 0x7fcd6c188e20 2021-02-10T11:46:12.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:12.792 7fcd627fc700 1 -- 172.21.2.112:0/217593347 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fcd5c004140 con 0x7fcd6c188e20 2021-02-10T11:46:12.923 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:12.920 7fcd73ca9700 1 -- 172.21.2.112:0/217593347 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fcd6c082a70 con 0x7fcd6c188e20 2021-02-10T11:46:13.285 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:13.282 7fcd627fc700 1 -- 172.21.2.112:0/217593347 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v165) v1 ==== 185+0+0 (secure 0 0 0) 0x7fcd5c004480 con 0x7fcd6c188e20 2021-02-10T11:46:13.290 INFO:teuthology.orchestra.run.gibba012.stderr:pool 'cephfs_metadata' removed 2021-02-10T11:46:13.292 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:13.289 7fcd73ca9700 1 -- 172.21.2.112:0/217593347 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcd540186d0 msgr2=0x7fcd5401abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:13.292 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:13.289 7fcd73ca9700 1 --2- 172.21.2.112:0/217593347 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcd540186d0 0x7fcd5401abd0 crc :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:13.292 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:13.289 7fcd73ca9700 1 -- 172.21.2.112:0/217593347 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcd6c188e20 msgr2=0x7fcd6c189220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:13.293 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:13.289 7fcd73ca9700 1 --2- 172.21.2.112:0/217593347 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcd6c188e20 0x7fcd6c189220 secure :-1 s=READY pgs=2845 cs=0 l=1 rev1=1 rx=0x7fcd5c001c60 tx=0x7fcd5c00a8f0).stop 2021-02-10T11:46:13.293 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:13.289 7fcd73ca9700 1 -- 172.21.2.112:0/217593347 shutdown_connections 2021-02-10T11:46:13.293 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:13.289 7fcd73ca9700 1 --2- 172.21.2.112:0/217593347 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcd540186d0 0x7fcd5401abd0 unknown :-1 s=CLOSED pgs=1292 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:13.293 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:13.289 7fcd73ca9700 1 --2- 172.21.2.112:0/217593347 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcd6c09cbc0 0x7fcd6c09cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:13.293 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:13.289 7fcd73ca9700 1 --2- 172.21.2.112:0/217593347 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcd6c188e20 0x7fcd6c189220 unknown :-1 s=CLOSED pgs=2845 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:13.293 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:13.289 7fcd73ca9700 1 --2- 172.21.2.112:0/217593347 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcd6c188400 0x7fcd6c188800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:13.293 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:13.289 7fcd73ca9700 1 -- 172.21.2.112:0/217593347 >> 172.21.2.112:0/217593347 conn(0x7fcd6c0c2cd0 msgr2=0x7fcd6c0b5950 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:13.293 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:13.289 7fcd73ca9700 1 -- 172.21.2.112:0/217593347 shutdown_connections 2021-02-10T11:46:13.293 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:13.289 7fcd73ca9700 1 -- 172.21.2.112:0/217593347 wait complete. 2021-02-10T11:46:13.301 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2021-02-10T11:46:13.734 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:13.714 7f962cc0b700 1 Processor -- start 2021-02-10T11:46:13.735 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:13.714 7f962cc0b700 1 -- start start 2021-02-10T11:46:13.735 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:13.715 7f962cc0b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f96280d1870 0x7f96281b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:13.735 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:13.715 7f962cc0b700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f96280cff70 0x7f9628109130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:13.735 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:13.715 7f962cc0b700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9628109a60 con 0x7f96281b0ae0 2021-02-10T11:46:13.735 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:13.715 7f962cc0b700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9628109750 con 0x7f96280cff70 2021-02-10T11:46:13.735 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:13.715 7f962cc0b700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9628088070 con 0x7f96280d1870 2021-02-10T11:46:13.735 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:13.715 7f962705f700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f96280cff70 0x7f9628109130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:13.735 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:13.715 7f962705f700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f96280cff70 0x7f9628109130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3300/0 says I am v2:172.21.2.112:33398/0 (socket says 172.21.2.112:33398) 2021-02-10T11:46:13.735 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:13.715 7f962705f700 1 -- 172.21.2.112:0/1770740125 learned_addr learned my addr 172.21.2.112:0/1770740125 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:46:13.735 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:13.715 7f962605d700 1 --1- 172.21.2.112:0/1770740125 >> v1:172.21.2.120:6789/0 conn(0x7f96281b0ae0 0x7f9628108940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:52366/0 (socket says 172.21.2.112:52366) 2021-02-10T11:46:13.735 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:13.715 7f962685e700 1 --2- 172.21.2.112:0/1770740125 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f96280d1870 0x7f96281b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:13.736 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:13.716 7f962705f700 1 -- 172.21.2.112:0/1770740125 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f96280d1870 msgr2=0x7f96281b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:46:13.736 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:13.716 7f962705f700 1 --2- 172.21.2.112:0/1770740125 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f96280d1870 0x7f96281b04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:13.736 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:13.716 7f962705f700 1 -- 172.21.2.112:0/1770740125 >> v1:172.21.2.120:6789/0 conn(0x7f96281b0ae0 legacy=0x7f9628108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:13.736 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:13.716 7f962705f700 1 -- 172.21.2.112:0/1770740125 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9628088260 con 0x7f96280cff70 2021-02-10T11:46:13.736 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:13.716 7f962705f700 1 --2- 172.21.2.112:0/1770740125 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f96280cff70 0x7f9628109130 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 rx=0x7f9620006aa0 tx=0x7f9620009f20).ready entity=mon.1 client_cookie=7894deb188fef7bd server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:13.736 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:13.716 7f962585c700 1 -- 172.21.2.112:0/1770740125 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f962000b5c0 con 0x7f96280cff70 2021-02-10T11:46:13.736 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:13.716 7f962585c700 1 -- 172.21.2.112:0/1770740125 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9620003030 con 0x7f96280cff70 2021-02-10T11:46:13.736 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:13.716 7f962585c700 1 -- 172.21.2.112:0/1770740125 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f962000b9f0 con 0x7f96280cff70 2021-02-10T11:46:13.736 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:13.716 7f962cc0b700 1 -- 172.21.2.112:0/1770740125 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f96280cff70 msgr2=0x7f9628109130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:13.736 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:13.716 7f962cc0b700 1 --2- 172.21.2.112:0/1770740125 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f96280cff70 0x7f9628109130 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 rx=0x7f9620006aa0 tx=0x7f9620009f20).stop 2021-02-10T11:46:13.736 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:13.717 7f962cc0b700 1 -- 172.21.2.112:0/1770740125 shutdown_connections 2021-02-10T11:46:13.736 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:13.717 7f962cc0b700 1 --2- 172.21.2.112:0/1770740125 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f96280cff70 0x7f9628109130 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:13.736 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:13.717 7f962cc0b700 1 --2- 172.21.2.112:0/1770740125 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f96280d1870 0x7f96281b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:13.737 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:13.717 7f962cc0b700 1 -- 172.21.2.112:0/1770740125 >> 172.21.2.112:0/1770740125 conn(0x7f96280ce420 msgr2=0x7f96280d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:13.737 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:13.717 7f962cc0b700 1 -- 172.21.2.112:0/1770740125 shutdown_connections 2021-02-10T11:46:13.737 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:13.717 7f962cc0b700 1 -- 172.21.2.112:0/1770740125 wait complete. 2021-02-10T11:46:13.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:13.732 7f962cc0b700 1 Processor -- start 2021-02-10T11:46:13.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:13.732 7f962cc0b700 1 -- start start 2021-02-10T11:46:13.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:13.732 7f962cc0b700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f96281b6ee0 0x7f96281aceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:13.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:13.732 7f962cc0b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f962818df20 0x7f962818e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:13.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:13.732 7f962cc0b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f962818e940 0x7f96280ad2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:13.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:13.732 7f962cc0b700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f962818ed40 con 0x7f962818df20 2021-02-10T11:46:13.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:13.732 7f962cc0b700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96281b72e0 con 0x7f96281b6ee0 2021-02-10T11:46:13.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:13.732 7f962cc0b700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96280ad910 con 0x7f962818e940 2021-02-10T11:46:13.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:13.732 7f962605d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f962818df20 0x7f962818e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:13.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:13.732 7f962685e700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f96281b6ee0 0x7f96281aceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:13.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:13.733 7f962705f700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f962818e940 0x7f96280ad2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:13.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:13.733 7f962605d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f962818df20 0x7f962818e320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:59354/0 (socket says 172.21.2.112:59354) 2021-02-10T11:46:13.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:13.733 7f962685e700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f96281b6ee0 0x7f96281aceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3300/0 says I am v2:172.21.2.112:33404/0 (socket says 172.21.2.112:33404) 2021-02-10T11:46:13.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:13.733 7f962605d700 1 -- 172.21.2.112:0/2497711961 learned_addr learned my addr 172.21.2.112:0/2497711961 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:46:13.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:13.733 7f962705f700 1 -- 172.21.2.112:0/2497711961 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f96281b6ee0 msgr2=0x7f96281aceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:13.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:13.733 7f962705f700 1 --2- 172.21.2.112:0/2497711961 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f96281b6ee0 0x7f96281aceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:13.740 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:13.733 7f962705f700 1 -- 172.21.2.112:0/2497711961 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f962818df20 msgr2=0x7f962818e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:13.740 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:13.733 7f962705f700 1 --2- 172.21.2.112:0/2497711961 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f962818df20 0x7f962818e320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:13.740 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:13.733 7f962705f700 1 -- 172.21.2.112:0/2497711961 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96280adb30 con 0x7f962818e940 2021-02-10T11:46:13.740 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:13.733 7f962705f700 1 --2- 172.21.2.112:0/2497711961 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f962818e940 0x7f96280ad2f0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7f9620009410 tx=0x7f962000bc10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:13.740 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:13.733 7f96177fe700 1 -- 172.21.2.112:0/2497711961 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9620009450 con 0x7f962818e940 2021-02-10T11:46:13.740 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:13.733 7f962cc0b700 1 -- 172.21.2.112:0/2497711961 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96280adb30 con 0x7f962818e940 2021-02-10T11:46:13.740 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:13.733 7f96177fe700 1 -- 172.21.2.112:0/2497711961 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9620006bb0 con 0x7f962818e940 2021-02-10T11:46:13.740 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:13.733 7f96177fe700 1 -- 172.21.2.112:0/2497711961 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f96200022b0 con 0x7f962818e940 2021-02-10T11:46:13.740 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:13.733 7f962cc0b700 1 -- 172.21.2.112:0/2497711961 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96280adb30 con 0x7f962818e940 2021-02-10T11:46:13.740 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:13.734 7f96177fe700 1 -- 172.21.2.112:0/2497711961 <== mon.2 v2:172.21.2.120:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9620009fb0 con 0x7f962818e940 2021-02-10T11:46:13.740 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:13.734 7f96177fe700 1 --2- 172.21.2.112:0/2497711961 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f960c0186c0 0x7f960c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:13.740 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:13.734 7f96177fe700 1 -- 172.21.2.112:0/2497711961 <== mon.2 v2:172.21.2.120:3301/0 5 ==== osd_map(165..165 src has 1..165) v4 ==== 5237+0+0 (secure 0 0 0) 0x7f96200326c0 con 0x7f962818e940 2021-02-10T11:46:13.740 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:13.734 7f962685e700 1 --2- 172.21.2.112:0/2497711961 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f960c0186c0 0x7f960c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:13.740 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:13.735 7f962685e700 1 --2- 172.21.2.112:0/2497711961 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f960c0186c0 0x7f960c01abc0 crc :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:13.768 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:13.765 7f962cc0b700 1 -- 172.21.2.112:0/2497711961 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f962818ed40 con 0x7f962818e940 2021-02-10T11:46:13.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:13.769 7f96177fe700 1 -- 172.21.2.112:0/2497711961 <== mon.2 v2:172.21.2.120:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9620000df0 con 0x7f962818e940 2021-02-10T11:46:13.901 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:13.898 7f962cc0b700 1 -- 172.21.2.112:0/2497711961 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_command({"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f962818ed40 con 0x7f962818e940 2021-02-10T11:46:14.292 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:14.288 7f96177fe700 1 -- 172.21.2.112:0/2497711961 <== mon.2 v2:172.21.2.120:3301/0 7 ==== mon_command_ack([{"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v166) v1 ==== 173+0+0 (secure 0 0 0) 0x7f962000a310 con 0x7f962818e940 2021-02-10T11:46:14.318 INFO:teuthology.orchestra.run.gibba012.stderr:pool 'cephfs_data' removed 2021-02-10T11:46:14.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:14.317 7f962cc0b700 1 -- 172.21.2.112:0/2497711961 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f960c0186c0 msgr2=0x7f960c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:14.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:14.317 7f962cc0b700 1 --2- 172.21.2.112:0/2497711961 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f960c0186c0 0x7f960c01abc0 crc :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:14.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:14.317 7f962cc0b700 1 -- 172.21.2.112:0/2497711961 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f962818e940 msgr2=0x7f96280ad2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:14.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:14.317 7f962cc0b700 1 --2- 172.21.2.112:0/2497711961 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f962818e940 0x7f96280ad2f0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7f9620009410 tx=0x7f962000bc10).stop 2021-02-10T11:46:14.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:14.317 7f962cc0b700 1 -- 172.21.2.112:0/2497711961 shutdown_connections 2021-02-10T11:46:14.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:14.317 7f962cc0b700 1 --2- 172.21.2.112:0/2497711961 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f960c0186c0 0x7f960c01abc0 unknown :-1 s=CLOSED pgs=1293 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:14.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:14.317 7f962cc0b700 1 --2- 172.21.2.112:0/2497711961 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f962818e940 0x7f96280ad2f0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:14.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:14.317 7f962cc0b700 1 --2- 172.21.2.112:0/2497711961 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f962818df20 0x7f962818e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:14.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:14.317 7f962cc0b700 1 --2- 172.21.2.112:0/2497711961 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f96281b6ee0 0x7f96281aceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:14.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:14.317 7f962cc0b700 1 -- 172.21.2.112:0/2497711961 >> 172.21.2.112:0/2497711961 conn(0x7f9628084160 msgr2=0x7f962818c280 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:14.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:14.317 7f962cc0b700 1 -- 172.21.2.112:0/2497711961 shutdown_connections 2021-02-10T11:46:14.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:14.317 7f962cc0b700 1 -- 172.21.2.112:0/2497711961 wait complete. 2021-02-10T11:46:14.329 INFO:tasks.ceph.mds.a:Restarting daemon 2021-02-10T11:46:14.329 INFO:tasks.ceph.mds.a:Stopping old one... 2021-02-10T11:46:14.329 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2021-02-10T11:46:14.329 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:46:14.329 INFO:tasks.ceph.mds.b:Restarting daemon 2021-02-10T11:46:14.330 INFO:tasks.ceph.mds.b:Stopping old one... 2021-02-10T11:46:14.330 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2021-02-10T11:46:14.330 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:46:14.335 INFO:tasks.ceph.mds.a:Stopped 2021-02-10T11:46:14.335 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2021-02-10T11:46:14.336 INFO:tasks.ceph.mds.b:Stopped 2021-02-10T11:46:14.336 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2021-02-10T11:46:14.377 INFO:tasks.ceph.mds.a:Started 2021-02-10T11:46:14.378 DEBUG:teuthology.parallel:result is None 2021-02-10T11:46:14.378 INFO:tasks.ceph.mds.b:Started 2021-02-10T11:46:14.378 DEBUG:teuthology.parallel:result is None 2021-02-10T11:46:14.379 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd blacklist clear 2021-02-10T11:46:14.781 INFO:tasks.ceph.mds.b.gibba020.stdout:starting mds.b at 2021-02-10T11:46:14.785 INFO:tasks.ceph.mds.a.gibba012.stdout:starting mds.a at 2021-02-10T11:46:14.806 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:14.786 7f0b614c4700 1 Processor -- start 2021-02-10T11:46:14.806 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:14.786 7f0b614c4700 1 -- start start 2021-02-10T11:46:14.806 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:14.786 7f0b614c4700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f0b5c0d1870 0x7f0b5c1b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:14.806 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:14.786 7f0b614c4700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f0b5c0f0650 0x7f0b5c0cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:14.806 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:14.786 7f0b614c4700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b5c0f0b10 con 0x7f0b5c1b0a10 2021-02-10T11:46:14.806 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:14.786 7f0b614c4700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b5c0e5ed0 con 0x7f0b5c0d1870 2021-02-10T11:46:14.806 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:14.786 7f0b614c4700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b5c0d1c70 con 0x7f0b5c0f0650 2021-02-10T11:46:14.806 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:14.786 7f0b5b2bf700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f0b5c0d1870 0x7f0b5c1b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:14.806 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:14.786 7f0b5b2bf700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f0b5c0d1870 0x7f0b5c1b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:59372/0 (socket says 172.21.2.112:59372) 2021-02-10T11:46:14.806 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:14.786 7f0b5b2bf700 1 -- 172.21.2.112:0/1018576881 learned_addr learned my addr 172.21.2.112:0/1018576881 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:46:14.806 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:14.787 7f0b5bac0700 1 --2- 172.21.2.112:0/1018576881 >> v2:172.21.2.120:3300/0 conn(0x7f0b5c0f0650 0x7f0b5c0cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:14.807 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:14.787 7f0b5b2bf700 1 -- 172.21.2.112:0/1018576881 >> v2:172.21.2.120:3300/0 conn(0x7f0b5c0f0650 msgr2=0x7f0b5c0cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:46:14.807 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:14.787 7f0b5b2bf700 1 --2- 172.21.2.112:0/1018576881 >> v2:172.21.2.120:3300/0 conn(0x7f0b5c0f0650 0x7f0b5c0cfb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:14.807 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:14.787 7f0b5b2bf700 1 -- 172.21.2.112:0/1018576881 >> v1:172.21.2.112:6789/0 conn(0x7f0b5c1b0a10 legacy=0x7f0b5c0cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:14.807 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:14.787 7f0b5b2bf700 1 -- 172.21.2.112:0/1018576881 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b5c0d1e90 con 0x7f0b5c0d1870 2021-02-10T11:46:14.807 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:14.787 7f0b5aabe700 1 -- 172.21.2.112:0/1018576881 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3301222992 0 0) 0x7f0b44000ff0 con 0x7f0b5c1b0a10 2021-02-10T11:46:14.807 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:14.787 7f0b5b2bf700 1 --2- 172.21.2.112:0/1018576881 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0b5c0d1870 0x7f0b5c1b03f0 secure :-1 s=READY pgs=2851 cs=0 l=1 rev1=1 rx=0x7f0b4c006700 tx=0x7f0b4c009b80).ready entity=mon.0 client_cookie=efd2738c83fa143f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:14.807 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:14.787 7f0b5aabe700 1 -- 172.21.2.112:0/1018576881 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0b4c00c0c0 con 0x7f0b5c0d1870 2021-02-10T11:46:14.807 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:14.787 7f0b5aabe700 1 -- 172.21.2.112:0/1018576881 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b4c00c440 con 0x7f0b5c0d1870 2021-02-10T11:46:14.807 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:14.787 7f0b5aabe700 1 -- 172.21.2.112:0/1018576881 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0b4c00c920 con 0x7f0b5c0d1870 2021-02-10T11:46:14.807 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:14.787 7f0b614c4700 1 -- 172.21.2.112:0/1018576881 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0b5c0d1870 msgr2=0x7f0b5c1b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:14.807 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:14.787 7f0b614c4700 1 --2- 172.21.2.112:0/1018576881 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0b5c0d1870 0x7f0b5c1b03f0 secure :-1 s=READY pgs=2851 cs=0 l=1 rev1=1 rx=0x7f0b4c006700 tx=0x7f0b4c009b80).stop 2021-02-10T11:46:14.807 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:14.787 7f0b614c4700 1 -- 172.21.2.112:0/1018576881 shutdown_connections 2021-02-10T11:46:14.807 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:14.787 7f0b614c4700 1 --2- 172.21.2.112:0/1018576881 >> v2:172.21.2.120:3300/0 conn(0x7f0b5c0f0650 0x7f0b5c0cfb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:14.808 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:14.787 7f0b614c4700 1 --2- 172.21.2.112:0/1018576881 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0b5c0d1870 0x7f0b5c1b03f0 unknown :-1 s=CLOSED pgs=2851 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:14.808 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:14.787 7f0b614c4700 1 -- 172.21.2.112:0/1018576881 >> 172.21.2.112:0/1018576881 conn(0x7f0b5c0ce450 msgr2=0x7f0b5c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:14.808 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:14.787 7f0b614c4700 1 -- 172.21.2.112:0/1018576881 shutdown_connections 2021-02-10T11:46:14.808 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:14.787 7f0b614c4700 1 -- 172.21.2.112:0/1018576881 wait complete. 2021-02-10T11:46:14.808 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:14.802 7f0b614c4700 1 Processor -- start 2021-02-10T11:46:14.808 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:14.802 7f0b614c4700 1 -- start start 2021-02-10T11:46:14.808 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:14.802 7f0b614c4700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0b5c0b7510 0x7f0b5c0b7910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:14.808 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:14.802 7f0b614c4700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0b5c09d530 0x7f0b5c09d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:14.808 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:14.802 7f0b614c4700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0b5c09df50 0x7f0b5c09eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:14.808 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:14.802 7f0b614c4700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b5c0b8140 con 0x7f0b5c09df50 2021-02-10T11:46:14.808 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:14.802 7f0b614c4700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b5c09e750 con 0x7f0b5c09d530 2021-02-10T11:46:14.808 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:14.802 7f0b614c4700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b5c09e350 con 0x7f0b5c0b7510 2021-02-10T11:46:14.808 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:14.803 7f0b5bac0700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0b5c09df50 0x7f0b5c09eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:14.809 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:14.803 7f0b5bac0700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0b5c09df50 0x7f0b5c09eca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:59380/0 (socket says 172.21.2.112:59380) 2021-02-10T11:46:14.809 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:14.803 7f0b5bac0700 1 -- 172.21.2.112:0/2178272520 learned_addr learned my addr 172.21.2.112:0/2178272520 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:46:14.810 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:14.803 7f0b5bac0700 1 -- 172.21.2.112:0/2178272520 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0b5c0b7510 msgr2=0x7f0b5c0b7910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:14.810 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:14.803 7f0b5bac0700 1 --2- 172.21.2.112:0/2178272520 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0b5c0b7510 0x7f0b5c0b7910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:14.810 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:14.803 7f0b53fff700 1 --2- 172.21.2.112:0/2178272520 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0b5c09d530 0x7f0b5c09d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:14.810 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:14.803 7f0b5bac0700 1 -- 172.21.2.112:0/2178272520 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0b5c09d530 msgr2=0x7f0b5c09d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:14.810 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:14.803 7f0b5bac0700 1 --2- 172.21.2.112:0/2178272520 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0b5c09d530 0x7f0b5c09d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:14.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:14.803 7f0b5bac0700 1 -- 172.21.2.112:0/2178272520 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b5c09f2c0 con 0x7f0b5c09df50 2021-02-10T11:46:14.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:14.803 7f0b53fff700 1 --2- 172.21.2.112:0/2178272520 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0b5c09d530 0x7f0b5c09d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:46:14.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:14.803 7f0b5bac0700 1 --2- 172.21.2.112:0/2178272520 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0b5c09df50 0x7f0b5c09eca0 secure :-1 s=READY pgs=2852 cs=0 l=1 rev1=1 rx=0x7f0b540099a0 tx=0x7f0b5400bc70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:14.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:14.803 7f0b58aba700 1 -- 172.21.2.112:0/2178272520 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0b5400bdb0 con 0x7f0b5c09df50 2021-02-10T11:46:14.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:14.803 7f0b58aba700 1 -- 172.21.2.112:0/2178272520 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b54003030 con 0x7f0b5c09df50 2021-02-10T11:46:14.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:14.803 7f0b58aba700 1 -- 172.21.2.112:0/2178272520 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0b540013e0 con 0x7f0b5c09df50 2021-02-10T11:46:14.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:14.803 7f0b614c4700 1 -- 172.21.2.112:0/2178272520 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b5c09f2c0 con 0x7f0b5c09df50 2021-02-10T11:46:14.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:14.804 7f0b614c4700 1 -- 172.21.2.112:0/2178272520 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b5c09f850 con 0x7f0b5c09df50 2021-02-10T11:46:14.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:14.804 7f0b58aba700 1 -- 172.21.2.112:0/2178272520 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0b540011f0 con 0x7f0b5c09df50 2021-02-10T11:46:14.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:14.804 7f0b58aba700 1 --2- 172.21.2.112:0/2178272520 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0b400186c0 0x7f0b4001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:14.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:14.804 7f0b58aba700 1 -- 172.21.2.112:0/2178272520 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(166..166 src has 1..166) v4 ==== 4839+0+0 (secure 0 0 0) 0x7f0b540324b0 con 0x7f0b5c09df50 2021-02-10T11:46:14.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:14.804 7f0b5b2bf700 1 --2- 172.21.2.112:0/2178272520 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0b400186c0 0x7f0b4001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:14.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:14.805 7f0b5b2bf700 1 --2- 172.21.2.112:0/2178272520 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0b400186c0 0x7f0b4001abc0 crc :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:14.838 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:14.836 7f0b614c4700 1 -- 172.21.2.112:0/2178272520 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b5c0ebc50 con 0x7f0b5c09df50 2021-02-10T11:46:14.841 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:14.839 7f0b58aba700 1 -- 172.21.2.112:0/2178272520 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0b54000f90 con 0x7f0b5c09df50 2021-02-10T11:46:14.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:14.977 7f0b614c4700 1 -- 172.21.2.112:0/2178272520 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7f0b5c0f0b10 con 0x7f0b5c09df50 2021-02-10T11:46:15.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:15.331 7f0b58aba700 1 -- 172.21.2.112:0/2178272520 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v167) v1 ==== 97+0+0 (secure 0 0 0) 0x7f0b5401f870 con 0x7f0b5c09df50 2021-02-10T11:46:15.347 INFO:teuthology.orchestra.run.gibba012.stderr: removed all blacklist entries 2021-02-10T11:46:15.349 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:15.347 7f0b614c4700 1 -- 172.21.2.112:0/2178272520 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0b400186c0 msgr2=0x7f0b4001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:15.349 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:15.347 7f0b614c4700 1 --2- 172.21.2.112:0/2178272520 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0b400186c0 0x7f0b4001abc0 crc :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:15.349 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:15.347 7f0b614c4700 1 -- 172.21.2.112:0/2178272520 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0b5c09df50 msgr2=0x7f0b5c09eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:15.349 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:15.347 7f0b614c4700 1 --2- 172.21.2.112:0/2178272520 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0b5c09df50 0x7f0b5c09eca0 secure :-1 s=READY pgs=2852 cs=0 l=1 rev1=1 rx=0x7f0b540099a0 tx=0x7f0b5400bc70).stop 2021-02-10T11:46:15.350 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:15.347 7f0b614c4700 1 -- 172.21.2.112:0/2178272520 shutdown_connections 2021-02-10T11:46:15.350 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:15.347 7f0b614c4700 1 --2- 172.21.2.112:0/2178272520 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0b400186c0 0x7f0b4001abc0 unknown :-1 s=CLOSED pgs=1294 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:15.350 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:15.347 7f0b614c4700 1 --2- 172.21.2.112:0/2178272520 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0b5c09d530 0x7f0b5c09d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:15.350 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:15.347 7f0b614c4700 1 --2- 172.21.2.112:0/2178272520 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0b5c09df50 0x7f0b5c09eca0 unknown :-1 s=CLOSED pgs=2852 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:15.350 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:15.347 7f0b614c4700 1 --2- 172.21.2.112:0/2178272520 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0b5c0b7510 0x7f0b5c0b7910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:15.350 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:15.347 7f0b614c4700 1 -- 172.21.2.112:0/2178272520 >> 172.21.2.112:0/2178272520 conn(0x7f0b5c0b83b0 msgr2=0x7f0b5c0a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:15.350 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:15.347 7f0b614c4700 1 -- 172.21.2.112:0/2178272520 shutdown_connections 2021-02-10T11:46:15.350 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:15.347 7f0b614c4700 1 -- 172.21.2.112:0/2178272520 wait complete. 2021-02-10T11:46:15.357 INFO:tasks.cephfs.cephfs_test_case:['0', '1', '2', '3'] 2021-02-10T11:46:15.358 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth ls --format=json-pretty 2021-02-10T11:46:15.788 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:15.768 7f028a523700 1 Processor -- start 2021-02-10T11:46:15.788 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:15.768 7f028a523700 1 -- start start 2021-02-10T11:46:15.788 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:15.769 7f028a523700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f02840d1870 0x7f02841b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:15.788 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:15.769 7f028a523700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02840f0b10 con 0x7f02840f0650 2021-02-10T11:46:15.788 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:15.769 7f028a523700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02840d0310 con 0x7f02841b0a10 2021-02-10T11:46:15.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:15.769 7f028a523700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02840d0530 con 0x7f02840d1870 2021-02-10T11:46:15.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:15.769 7f0288b40700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f02840f0650 0x7f02840cfb20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:58850/0 (socket says 172.21.2.112:58850) 2021-02-10T11:46:15.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:15.769 7f0288b40700 1 -- 172.21.2.112:0/374985561 learned_addr learned my addr 172.21.2.112:0/374985561 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:46:15.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:15.769 7f0283fff700 1 --2- 172.21.2.112:0/374985561 >> v2:172.21.2.120:3300/0 conn(0x7f02840d1870 0x7f02841b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:15.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:15.769 7f0282ffd700 1 -- 172.21.2.112:0/374985561 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 875233762 0 0) 0x7f027c000f90 con 0x7f02840f0650 2021-02-10T11:46:15.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:15.769 7f0282ffd700 1 -- 172.21.2.112:0/374985561 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0268002fe0 con 0x7f02840f0650 2021-02-10T11:46:15.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:15.769 7f0282ffd700 1 -- 172.21.2.112:0/374985561 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3830066429 0 0) 0x7f0270000c60 con 0x7f02841b0a10 2021-02-10T11:46:15.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:15.769 7f0282ffd700 1 -- 172.21.2.112:0/374985561 --> v1:172.21.2.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0268003210 con 0x7f02841b0a10 2021-02-10T11:46:15.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:15.770 7f0282ffd700 1 -- 172.21.2.112:0/374985561 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2880516997 0 0) 0x7f027c001460 con 0x7f02840f0650 2021-02-10T11:46:15.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:15.770 7f0282ffd700 1 -- 172.21.2.112:0/374985561 >> v2:172.21.2.120:3300/0 conn(0x7f02840d1870 msgr2=0x7f02841b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:46:15.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:15.770 7f0282ffd700 1 --2- 172.21.2.112:0/374985561 >> v2:172.21.2.120:3300/0 conn(0x7f02840d1870 0x7f02841b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:15.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:15.770 7f0282ffd700 1 -- 172.21.2.112:0/374985561 >> v1:172.21.2.120:6789/0 conn(0x7f02841b0a10 legacy=0x7f02840cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:15.790 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:15.770 7f0282ffd700 1 -- 172.21.2.112:0/374985561 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02840d1c70 con 0x7f02840f0650 2021-02-10T11:46:15.790 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:15.770 7f0282ffd700 1 -- 172.21.2.112:0/374985561 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f027c001980 con 0x7f02840f0650 2021-02-10T11:46:15.790 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:15.770 7f0282ffd700 1 -- 172.21.2.112:0/374985561 >> v1:172.21.2.112:6789/0 conn(0x7f02840f0650 legacy=0x7f02840cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:15.790 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:15.770 7f0282ffd700 1 --2- 172.21.2.112:0/374985561 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0268003b90 0x7f0268005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:15.790 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:15.770 7f0282ffd700 1 --2- 172.21.2.112:0/374985561 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f02680067c0 0x7f0268008cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:15.790 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:15.770 7f0282ffd700 1 --2- 172.21.2.112:0/374985561 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f02680095c0 0x7f026800bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:15.790 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:15.770 7f0282ffd700 1 -- 172.21.2.112:0/374985561 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f026800c310 con 0x7f02680067c0 2021-02-10T11:46:15.790 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:15.770 7f0282ffd700 1 -- 172.21.2.112:0/374985561 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0268050500 con 0x7f02680095c0 2021-02-10T11:46:15.791 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:15.770 7f0282ffd700 1 -- 172.21.2.112:0/374985561 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0268050720 con 0x7f0268003b90 2021-02-10T11:46:15.791 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:15.770 7f02837fe700 1 --2- 172.21.2.112:0/374985561 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f02680067c0 0x7f0268008cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:15.791 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:15.770 7f0283fff700 1 --2- 172.21.2.112:0/374985561 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0268003b90 0x7f0268005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:15.791 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:15.770 7f0288b40700 1 --2- 172.21.2.112:0/374985561 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f02680095c0 0x7f026800bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:15.791 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:15.770 7f02837fe700 1 -- 172.21.2.112:0/374985561 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0268003b90 msgr2=0x7f0268005fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:46:15.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:15.770 7f02837fe700 1 --2- 172.21.2.112:0/374985561 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0268003b90 0x7f0268005fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:15.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:15.770 7f02837fe700 1 -- 172.21.2.112:0/374985561 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f02680095c0 msgr2=0x7f026800bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:46:15.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:15.770 7f02837fe700 1 --2- 172.21.2.112:0/374985561 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f02680095c0 0x7f026800bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:15.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:15.770 7f02837fe700 1 -- 172.21.2.112:0/374985561 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02680509c0 con 0x7f02680067c0 2021-02-10T11:46:15.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:15.770 7f0288b40700 1 -- 172.21.2.112:0/374985561 reap_dead start 2021-02-10T11:46:15.793 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:15.770 7f02837fe700 1 --2- 172.21.2.112:0/374985561 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f02680067c0 0x7f0268008cc0 secure :-1 s=READY pgs=2854 cs=0 l=1 rev1=1 rx=0x7f0270001040 tx=0x7f0270009a20).ready entity=mon.0 client_cookie=d7ca0398c03fc310 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:15.793 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:15.770 7f0282ffd700 1 -- 172.21.2.112:0/374985561 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0270000c10 con 0x7f02680067c0 2021-02-10T11:46:15.793 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:15.770 7f0282ffd700 1 -- 172.21.2.112:0/374985561 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02700066e0 con 0x7f02680067c0 2021-02-10T11:46:15.793 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:15.770 7f0282ffd700 1 -- 172.21.2.112:0/374985561 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0270001480 con 0x7f02680067c0 2021-02-10T11:46:15.793 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:15.770 7f028a523700 1 -- 172.21.2.112:0/374985561 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f02680067c0 msgr2=0x7f0268008cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:15.793 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:15.770 7f028a523700 1 --2- 172.21.2.112:0/374985561 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f02680067c0 0x7f0268008cc0 secure :-1 s=READY pgs=2854 cs=0 l=1 rev1=1 rx=0x7f0270001040 tx=0x7f0270009a20).stop 2021-02-10T11:46:15.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:15.771 7f028a523700 1 -- 172.21.2.112:0/374985561 shutdown_connections 2021-02-10T11:46:15.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:15.771 7f028a523700 1 --2- 172.21.2.112:0/374985561 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f02680067c0 0x7f0268008cc0 unknown :-1 s=CLOSED pgs=2854 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:15.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:15.771 7f028a523700 1 -- 172.21.2.112:0/374985561 >> 172.21.2.112:0/374985561 conn(0x7f02840ce450 msgr2=0x7f02840d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:15.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:15.771 7f028a523700 1 -- 172.21.2.112:0/374985561 shutdown_connections 2021-02-10T11:46:15.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:15.771 7f028a523700 1 -- 172.21.2.112:0/374985561 wait complete. 2021-02-10T11:46:15.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:15.786 7f028a523700 1 Processor -- start 2021-02-10T11:46:15.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:15.786 7f028a523700 1 -- start start 2021-02-10T11:46:15.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:15.786 7f028a523700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f02840f0280 0x7f02840a2310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:15.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:15.786 7f028a523700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f02840f0680 0x7f02840a2930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:15.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:15.786 7f028a523700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f02840ec180 0x7f02840ec580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:15.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:15.786 7f028a523700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02840efd00 con 0x7f02840ec180 2021-02-10T11:46:15.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:15.786 7f028a523700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02840ecba0 con 0x7f02840f0280 2021-02-10T11:46:15.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:15.786 7f028a523700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02840ecd80 con 0x7f02840f0680 2021-02-10T11:46:15.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:15.786 7f0288b40700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f02840ec180 0x7f02840ec580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:15.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:15.786 7f0288b40700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f02840ec180 0x7f02840ec580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:59396/0 (socket says 172.21.2.112:59396) 2021-02-10T11:46:15.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:15.786 7f0288b40700 1 -- 172.21.2.112:0/444140513 learned_addr learned my addr 172.21.2.112:0/444140513 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:46:15.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:15.787 7f0283fff700 1 --2- 172.21.2.112:0/444140513 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f02840f0280 0x7f02840a2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:15.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:15.787 7f02837fe700 1 --2- 172.21.2.112:0/444140513 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f02840f0680 0x7f02840a2930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:15.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:15.787 7f0288b40700 1 -- 172.21.2.112:0/444140513 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f02840f0680 msgr2=0x7f02840a2930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:15.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:15.787 7f0288b40700 1 --2- 172.21.2.112:0/444140513 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f02840f0680 0x7f02840a2930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:15.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:15.787 7f0288b40700 1 -- 172.21.2.112:0/444140513 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f02840f0280 msgr2=0x7f02840a2310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:15.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:15.787 7f0288b40700 1 --2- 172.21.2.112:0/444140513 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f02840f0280 0x7f02840a2310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:15.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:15.787 7f0288b40700 1 -- 172.21.2.112:0/444140513 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02840ecfa0 con 0x7f02840ec180 2021-02-10T11:46:15.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:15.787 7f0288b40700 1 --2- 172.21.2.112:0/444140513 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f02840ec180 0x7f02840ec580 secure :-1 s=READY pgs=2855 cs=0 l=1 rev1=1 rx=0x7f027c007a00 tx=0x7f027c00acc0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:15.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:15.787 7f0280ff9700 1 -- 172.21.2.112:0/444140513 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f027c004930 con 0x7f02840ec180 2021-02-10T11:46:15.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:15.787 7f0280ff9700 1 -- 172.21.2.112:0/444140513 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f027c004bb0 con 0x7f02840ec180 2021-02-10T11:46:15.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:15.787 7f0280ff9700 1 -- 172.21.2.112:0/444140513 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f027c007020 con 0x7f02840ec180 2021-02-10T11:46:15.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:15.787 7f028a523700 1 -- 172.21.2.112:0/444140513 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02840ecfa0 con 0x7f02840ec180 2021-02-10T11:46:15.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:15.787 7f028a523700 1 -- 172.21.2.112:0/444140513 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02840ecfa0 con 0x7f02840ec180 2021-02-10T11:46:15.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:15.788 7f0280ff9700 1 -- 172.21.2.112:0/444140513 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f027c004720 con 0x7f02840ec180 2021-02-10T11:46:15.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:15.788 7f0280ff9700 1 --2- 172.21.2.112:0/444140513 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f02600186c0 0x7f026001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:15.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:15.788 7f0280ff9700 1 -- 172.21.2.112:0/444140513 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(167..167 src has 1..167) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f027c032620 con 0x7f02840ec180 2021-02-10T11:46:15.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:15.788 7f0283fff700 1 --2- 172.21.2.112:0/444140513 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f02600186c0 0x7f026001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:15.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:15.789 7f0283fff700 1 --2- 172.21.2.112:0/444140513 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f02600186c0 0x7f026001abc0 crc :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:15.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:15.818 7f028a523700 1 -- 172.21.2.112:0/444140513 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02840ebc60 con 0x7f02840ec180 2021-02-10T11:46:15.822 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:15.820 7f0280ff9700 1 -- 172.21.2.112:0/444140513 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f027c0074f0 con 0x7f02840ec180 2021-02-10T11:46:15.956 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:15.954 7f028a523700 1 -- 172.21.2.112:0/444140513 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f02840f0b10 con 0x7f02840ec180 2021-02-10T11:46:15.957 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:15.955 7f0280ff9700 1 -- 172.21.2.112:0/444140513 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v94) v1 ==== 80+0+4513 (secure 0 0 0) 0x7f027c004720 con 0x7f02840ec180 2021-02-10T11:46:15.957 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:46:15.958 INFO:teuthology.orchestra.run.gibba012.stdout:{ 2021-02-10T11:46:15.958 INFO:teuthology.orchestra.run.gibba012.stdout: "auth_dump": [ 2021-02-10T11:46:15.958 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:46:15.958 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "mds.a", 2021-02-10T11:46:15.958 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCKvyNgPkV4AxAAEmVNaghdccP6lWd8hvQnxw==", 2021-02-10T11:46:15.958 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:46:15.958 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:46:15.958 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:46:15.958 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:46:15.958 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:46:15.958 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:46:15.959 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:46:15.959 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:46:15.959 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "mds.b", 2021-02-10T11:46:15.959 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCKvyNguxONLhAAHIpry+R89MAwdhFJyEhVCw==", 2021-02-10T11:46:15.959 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:46:15.959 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:46:15.959 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:46:15.959 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:46:15.959 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:46:15.959 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:46:15.959 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:46:15.959 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:46:15.959 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.0", 2021-02-10T11:46:15.960 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCSvyNgXcKPLhAAP/Bj2F+06DCtPCzQDNq+cQ==", 2021-02-10T11:46:15.960 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:46:15.960 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:46:15.960 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:46:15.960 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:46:15.960 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:46:15.960 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:46:15.960 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:46:15.960 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.1", 2021-02-10T11:46:15.960 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCVvyNghfbcLhAA66PrRwR2M7o6VTF5UhU/Fg==", 2021-02-10T11:46:15.960 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:46:15.960 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:46:15.960 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:46:15.961 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:46:15.961 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:46:15.961 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:46:15.961 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:46:15.961 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.2", 2021-02-10T11:46:15.961 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCYvyNgwiPqLhAA66BVkRsgaUlGpAsQ15Nf8Q==", 2021-02-10T11:46:15.961 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:46:15.961 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:46:15.961 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:46:15.961 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:46:15.961 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:46:15.961 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:46:15.961 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:46:15.961 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.3", 2021-02-10T11:46:15.962 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCbvyNgqBZrLxAAERPe/1L3SXAEzI3d9ReHHg==", 2021-02-10T11:46:15.962 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:46:15.962 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:46:15.962 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:46:15.962 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:46:15.962 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:46:15.962 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:46:15.962 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:46:15.962 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.4", 2021-02-10T11:46:15.962 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCkvyNgDQsxIhAAHmVjVYnx20N4SDmKqvbcuA==", 2021-02-10T11:46:15.962 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:46:15.963 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:46:15.963 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:46:15.963 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:46:15.963 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:46:15.963 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:46:15.963 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:46:15.963 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.5", 2021-02-10T11:46:15.963 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCnvyNg8ITFIhAADd4aKybAV7btoGPx3kCg3A==", 2021-02-10T11:46:15.963 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:46:15.963 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:46:15.963 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:46:15.963 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:46:15.963 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:46:15.963 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:46:15.964 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:46:15.964 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.6", 2021-02-10T11:46:15.964 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCqvyNgxOhDIxAAzXixEa/nlDc1xXrg0aeLwg==", 2021-02-10T11:46:15.964 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:46:15.964 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:46:15.964 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:46:15.964 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:46:15.964 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:46:15.964 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:46:15.964 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:46:15.964 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.7", 2021-02-10T11:46:15.964 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCtvyNgxSqGIxAACxJQlwP7DpiVVCZHA3zn1A==", 2021-02-10T11:46:15.965 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:46:15.965 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:46:15.965 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:46:15.965 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:46:15.965 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:46:15.965 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:46:15.965 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:46:15.965 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.0", 2021-02-10T11:46:15.965 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCLvyNgDkNiHRAAygU8Za4PBusTqCgwAYWTiQ==", 2021-02-10T11:46:15.965 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:46:15.965 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:46:15.965 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow r", 2021-02-10T11:46:15.965 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:46:15.966 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:46:15.966 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:46:15.966 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:46:15.966 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.1", 2021-02-10T11:46:15.966 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCLvyNgxmLqMxAA1MlSDJ9LH9u4f2sL7mm8/Q==", 2021-02-10T11:46:15.966 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:46:15.966 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:46:15.966 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow r", 2021-02-10T11:46:15.966 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:46:15.966 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:46:15.966 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:46:15.966 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:46:15.966 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.2", 2021-02-10T11:46:15.967 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCMvyNgp8veDhAAniSVHYPtwwTog/pjbaRQmw==", 2021-02-10T11:46:15.967 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:46:15.967 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:46:15.967 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow r", 2021-02-10T11:46:15.967 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:46:15.967 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:46:15.967 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:46:15.967 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:46:15.967 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.3", 2021-02-10T11:46:15.967 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCMvyNgQEVoJRAApHcWdm1sldIxu+heyD4HdQ==", 2021-02-10T11:46:15.967 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:46:15.967 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:46:15.967 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow r", 2021-02-10T11:46:15.967 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:46:15.968 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:46:15.968 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:46:15.968 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:46:15.968 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.admin", 2021-02-10T11:46:15.968 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCGvyNg7jjJBRAAtot+ypUZOiTo1GbY+oVhVw==", 2021-02-10T11:46:15.968 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:46:15.968 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow *", 2021-02-10T11:46:15.968 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:46:15.968 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:46:15.968 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:46:15.968 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:46:15.968 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:46:15.968 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:46:15.969 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "mgr.x", 2021-02-10T11:46:15.969 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCJvyNgTPVwKBAAjduiBE8+RSXBv82tdPecWg==", 2021-02-10T11:46:15.969 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:46:15.969 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow *", 2021-02-10T11:46:15.969 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow profile mgr", 2021-02-10T11:46:15.969 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:46:15.969 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:46:15.969 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:46:15.969 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:46:15.969 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "mgr.y", 2021-02-10T11:46:15.969 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCJvyNgQ1tTDhAASwBssDy+Rz31ujw8fNlWow==", 2021-02-10T11:46:15.969 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:46:15.969 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow *", 2021-02-10T11:46:15.970 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow profile mgr", 2021-02-10T11:46:15.970 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:46:15.970 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:46:15.970 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:46:15.970 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:46:15.970 INFO:teuthology.orchestra.run.gibba012.stdout:} 2021-02-10T11:46:15.971 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:15.957 7f028a523700 1 -- 172.21.2.112:0/444140513 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f02600186c0 msgr2=0x7f026001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:15.971 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:15.957 7f028a523700 1 --2- 172.21.2.112:0/444140513 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f02600186c0 0x7f026001abc0 crc :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:15.971 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:15.957 7f028a523700 1 -- 172.21.2.112:0/444140513 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f02840ec180 msgr2=0x7f02840ec580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:15.971 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:15.957 7f028a523700 1 --2- 172.21.2.112:0/444140513 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f02840ec180 0x7f02840ec580 secure :-1 s=READY pgs=2855 cs=0 l=1 rev1=1 rx=0x7f027c007a00 tx=0x7f027c00acc0).stop 2021-02-10T11:46:15.971 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:15.957 7f028a523700 1 -- 172.21.2.112:0/444140513 shutdown_connections 2021-02-10T11:46:15.971 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:15.957 7f028a523700 1 --2- 172.21.2.112:0/444140513 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f02600186c0 0x7f026001abc0 unknown :-1 s=CLOSED pgs=1295 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:15.971 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:15.957 7f028a523700 1 --2- 172.21.2.112:0/444140513 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f02840ec180 0x7f02840ec580 unknown :-1 s=CLOSED pgs=2855 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:15.971 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:15.957 7f028a523700 1 --2- 172.21.2.112:0/444140513 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f02840f0680 0x7f02840a2930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:15.972 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:15.957 7f028a523700 1 --2- 172.21.2.112:0/444140513 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f02840f0280 0x7f02840a2310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:15.972 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:15.957 7f028a523700 1 -- 172.21.2.112:0/444140513 >> 172.21.2.112:0/444140513 conn(0x7f02840c2130 msgr2=0x7f028409e3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:15.972 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:15.957 7f028a523700 1 -- 172.21.2.112:0/444140513 shutdown_connections 2021-02-10T11:46:15.972 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:15.957 7f028a523700 1 -- 172.21.2.112:0/444140513 wait complete. 2021-02-10T11:46:15.972 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:46:16.404 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:16.385 7f0caa752700 1 Processor -- start 2021-02-10T11:46:16.404 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:16.385 7f0caa752700 1 -- start start 2021-02-10T11:46:16.404 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:16.386 7f0caa752700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f0ca40d1870 0x7f0ca41b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:16.404 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:16.386 7f0caa752700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f0ca41b0a10 0x7f0ca40cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:16.405 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:16.386 7f0caa752700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ca40f0b10 con 0x7f0ca40d1c70 2021-02-10T11:46:16.405 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:16.386 7f0caa752700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ca40e5ed0 con 0x7f0ca41b0a10 2021-02-10T11:46:16.405 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:16.386 7f0caa752700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ca40f0650 con 0x7f0ca40d1870 2021-02-10T11:46:16.405 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:16.386 7f0ca37fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f0ca41b0a10 0x7f0ca40cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:16.405 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:16.386 7f0ca8d6f700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7f0ca40d1c70 0x7f0ca40cf950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:52424/0 (socket says 172.21.2.112:52424) 2021-02-10T11:46:16.405 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:16.386 7f0ca8d6f700 1 -- 172.21.2.112:0/58044175 learned_addr learned my addr 172.21.2.112:0/58044175 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:46:16.405 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:16.386 7f0ca3fff700 1 --2- 172.21.2.112:0/58044175 >> v2:172.21.2.120:3300/0 conn(0x7f0ca40d1870 0x7f0ca41b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:16.405 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:16.387 7f0ca2ffd700 1 -- 172.21.2.112:0/58044175 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4266794073 0 0) 0x7f0c9c001440 con 0x7f0ca40d1c70 2021-02-10T11:46:16.405 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:16.387 7f0ca2ffd700 1 -- 172.21.2.112:0/58044175 --> v1:172.21.2.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c8c002fe0 con 0x7f0ca40d1c70 2021-02-10T11:46:16.405 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:16.387 7f0ca37fe700 1 -- 172.21.2.112:0/58044175 >> v2:172.21.2.120:3300/0 conn(0x7f0ca40d1870 msgr2=0x7f0ca41b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:46:16.405 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:16.387 7f0ca37fe700 1 --2- 172.21.2.112:0/58044175 >> v2:172.21.2.120:3300/0 conn(0x7f0ca40d1870 0x7f0ca41b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:16.405 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:16.387 7f0ca37fe700 1 -- 172.21.2.112:0/58044175 >> v1:172.21.2.120:6789/0 conn(0x7f0ca40d1c70 legacy=0x7f0ca40cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:16.405 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:16.387 7f0ca37fe700 1 -- 172.21.2.112:0/58044175 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ca40f0840 con 0x7f0ca41b0a10 2021-02-10T11:46:16.406 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:16.387 7f0ca3fff700 1 --2- 172.21.2.112:0/58044175 >> v2:172.21.2.120:3300/0 conn(0x7f0ca40d1870 0x7f0ca41b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:46:16.406 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:16.387 7f0ca37fe700 1 --2- 172.21.2.112:0/58044175 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0ca41b0a10 0x7f0ca40cf330 secure :-1 s=READY pgs=2856 cs=0 l=1 rev1=1 rx=0x7f0c98006650 tx=0x7f0c98009a80).ready entity=mon.0 client_cookie=89b612797a17dc6c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:16.406 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:16.387 7f0ca2ffd700 1 -- 172.21.2.112:0/58044175 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0c98006390 con 0x7f0ca41b0a10 2021-02-10T11:46:16.406 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:16.387 7f0ca2ffd700 1 -- 172.21.2.112:0/58044175 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c980009b0 con 0x7f0ca41b0a10 2021-02-10T11:46:16.406 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:16.387 7f0ca2ffd700 1 -- 172.21.2.112:0/58044175 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0c9800c2b0 con 0x7f0ca41b0a10 2021-02-10T11:46:16.406 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:16.387 7f0caa752700 1 -- 172.21.2.112:0/58044175 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0ca41b0a10 msgr2=0x7f0ca40cf330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:16.406 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:16.387 7f0caa752700 1 --2- 172.21.2.112:0/58044175 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0ca41b0a10 0x7f0ca40cf330 secure :-1 s=READY pgs=2856 cs=0 l=1 rev1=1 rx=0x7f0c98006650 tx=0x7f0c98009a80).stop 2021-02-10T11:46:16.406 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:16.388 7f0caa752700 1 -- 172.21.2.112:0/58044175 shutdown_connections 2021-02-10T11:46:16.406 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:16.388 7f0caa752700 1 --2- 172.21.2.112:0/58044175 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0ca41b0a10 0x7f0ca40cf330 unknown :-1 s=CLOSED pgs=2856 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:16.406 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:16.388 7f0caa752700 1 --2- 172.21.2.112:0/58044175 >> v2:172.21.2.120:3300/0 conn(0x7f0ca40d1870 0x7f0ca41b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:16.406 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:16.388 7f0caa752700 1 -- 172.21.2.112:0/58044175 >> 172.21.2.112:0/58044175 conn(0x7f0ca40ce450 msgr2=0x7f0ca40d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:16.407 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:16.388 7f0caa752700 1 -- 172.21.2.112:0/58044175 shutdown_connections 2021-02-10T11:46:16.407 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:16.388 7f0caa752700 1 -- 172.21.2.112:0/58044175 wait complete. 2021-02-10T11:46:16.407 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:16.402 7f0caa752700 1 Processor -- start 2021-02-10T11:46:16.408 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:16.402 7f0caa752700 1 -- start start 2021-02-10T11:46:16.408 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:16.403 7f0caa752700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0ca41b0190 0x7f0ca40ab750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:16.408 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:16.403 7f0caa752700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0ca40abd70 0x7f0ca41a0b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:16.408 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:16.403 7f0caa752700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0ca41a1150 0x7f0ca418b830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:16.408 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:16.403 7f0caa752700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ca41ad730 con 0x7f0ca41a1150 2021-02-10T11:46:16.409 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:16.403 7f0caa752700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ca41ad910 con 0x7f0ca41b0190 2021-02-10T11:46:16.409 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:16.403 7f0caa752700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ca41a1550 con 0x7f0ca40abd70 2021-02-10T11:46:16.409 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:16.403 7f0ca3fff700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0ca41b0190 0x7f0ca40ab750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:16.409 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:16.403 7f0ca8d6f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0ca41a1150 0x7f0ca418b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:16.409 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:16.403 7f0ca8d6f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0ca41a1150 0x7f0ca418b830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:59408/0 (socket says 172.21.2.112:59408) 2021-02-10T11:46:16.409 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:16.403 7f0ca8d6f700 1 -- 172.21.2.112:0/596435338 learned_addr learned my addr 172.21.2.112:0/596435338 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:46:16.409 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:16.403 7f0ca37fe700 1 --2- 172.21.2.112:0/596435338 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0ca40abd70 0x7f0ca41a0b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:16.409 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:16.403 7f0ca8d6f700 1 -- 172.21.2.112:0/596435338 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0ca40abd70 msgr2=0x7f0ca41a0b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:16.409 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:16.403 7f0ca8d6f700 1 --2- 172.21.2.112:0/596435338 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0ca40abd70 0x7f0ca41a0b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:16.409 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:16.403 7f0ca8d6f700 1 -- 172.21.2.112:0/596435338 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0ca41b0190 msgr2=0x7f0ca40ab750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:16.409 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:16.403 7f0ca8d6f700 1 --2- 172.21.2.112:0/596435338 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0ca41b0190 0x7f0ca40ab750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:16.409 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:16.403 7f0ca8d6f700 1 -- 172.21.2.112:0/596435338 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ca41a1770 con 0x7f0ca41a1150 2021-02-10T11:46:16.409 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:16.403 7f0ca3fff700 1 --2- 172.21.2.112:0/596435338 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0ca41b0190 0x7f0ca40ab750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:46:16.410 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:16.404 7f0ca8d6f700 1 --2- 172.21.2.112:0/596435338 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0ca41a1150 0x7f0ca418b830 secure :-1 s=READY pgs=2857 cs=0 l=1 rev1=1 rx=0x7f0c9c00a940 tx=0x7f0c9c00ace0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:16.410 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:16.404 7f0ca0ff9700 1 -- 172.21.2.112:0/596435338 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0c9c0048b0 con 0x7f0ca41a1150 2021-02-10T11:46:16.410 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:16.404 7f0ca0ff9700 1 -- 172.21.2.112:0/596435338 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c9c004b70 con 0x7f0ca41a1150 2021-02-10T11:46:16.410 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:16.404 7f0ca0ff9700 1 -- 172.21.2.112:0/596435338 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0c9c0070d0 con 0x7f0ca41a1150 2021-02-10T11:46:16.410 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:16.404 7f0caa752700 1 -- 172.21.2.112:0/596435338 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ca41a1770 con 0x7f0ca41a1150 2021-02-10T11:46:16.410 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:16.404 7f0caa752700 1 -- 172.21.2.112:0/596435338 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ca41a1770 con 0x7f0ca41a1150 2021-02-10T11:46:16.410 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:16.404 7f0ca0ff9700 1 -- 172.21.2.112:0/596435338 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0c9c000de0 con 0x7f0ca41a1150 2021-02-10T11:46:16.410 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:16.404 7f0ca0ff9700 1 --2- 172.21.2.112:0/596435338 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0c840186c0 0x7f0c8401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:16.410 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:16.405 7f0ca0ff9700 1 -- 172.21.2.112:0/596435338 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(167..167 src has 1..167) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f0c9c032600 con 0x7f0ca41a1150 2021-02-10T11:46:16.410 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:16.405 7f0ca3fff700 1 --2- 172.21.2.112:0/596435338 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0c840186c0 0x7f0c8401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:16.410 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:16.405 7f0ca3fff700 1 --2- 172.21.2.112:0/596435338 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0c840186c0 0x7f0c8401abc0 crc :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:16.438 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:16.436 7f0caa752700 1 -- 172.21.2.112:0/596435338 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ca40ebc00 con 0x7f0ca41a1150 2021-02-10T11:46:16.440 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:16.438 7f0ca0ff9700 1 -- 172.21.2.112:0/596435338 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0c9c004d50 con 0x7f0ca41a1150 2021-02-10T11:46:16.566 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:16.564 7f0caa752700 1 -- 172.21.2.112:0/596435338 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0ca40ebc00 con 0x7f0ca41a1150 2021-02-10T11:46:16.566 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:16.564 7f0ca0ff9700 1 -- 172.21.2.112:0/596435338 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v167) v1 ==== 74+0+11154 (secure 0 0 0) 0x7f0c9c000de0 con 0x7f0ca41a1150 2021-02-10T11:46:16.567 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:46:16.567 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":167,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:46:15.328768","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":35,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:46:16.569 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:16.566 7f0caa752700 1 -- 172.21.2.112:0/596435338 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0c840186c0 msgr2=0x7f0c8401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:16.569 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:16.566 7f0caa752700 1 --2- 172.21.2.112:0/596435338 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0c840186c0 0x7f0c8401abc0 crc :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:16.569 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:16.566 7f0caa752700 1 -- 172.21.2.112:0/596435338 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0ca41a1150 msgr2=0x7f0ca418b830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:16.569 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:16.566 7f0caa752700 1 --2- 172.21.2.112:0/596435338 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0ca41a1150 0x7f0ca418b830 secure :-1 s=READY pgs=2857 cs=0 l=1 rev1=1 rx=0x7f0c9c00a940 tx=0x7f0c9c00ace0).stop 2021-02-10T11:46:16.569 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:16.566 7f0caa752700 1 -- 172.21.2.112:0/596435338 shutdown_connections 2021-02-10T11:46:16.569 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:16.566 7f0caa752700 1 --2- 172.21.2.112:0/596435338 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0c840186c0 0x7f0c8401abc0 unknown :-1 s=CLOSED pgs=1296 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:16.569 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:16.566 7f0caa752700 1 --2- 172.21.2.112:0/596435338 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0ca41a1150 0x7f0ca418b830 unknown :-1 s=CLOSED pgs=2857 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:16.569 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:16.566 7f0caa752700 1 --2- 172.21.2.112:0/596435338 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0ca40abd70 0x7f0ca41a0b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:16.569 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:16.566 7f0caa752700 1 --2- 172.21.2.112:0/596435338 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0ca41b0190 0x7f0ca40ab750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:16.569 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:16.567 7f0caa752700 1 -- 172.21.2.112:0/596435338 >> 172.21.2.112:0/596435338 conn(0x7f0ca419d660 msgr2=0x7f0ca41b0810 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:16.570 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:16.567 7f0caa752700 1 -- 172.21.2.112:0/596435338 shutdown_connections 2021-02-10T11:46:16.570 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:16.567 7f0caa752700 1 -- 172.21.2.112:0/596435338 wait complete. 2021-02-10T11:46:16.577 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:13:57.508758', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2021-02-10T11:46:16.578 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:46:17.010 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:16.991 7f4d3d823700 1 Processor -- start 2021-02-10T11:46:17.010 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:16.991 7f4d3d823700 1 -- start start 2021-02-10T11:46:17.010 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:16.991 7f4d3d823700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f4d380d1860 0x7f4d381b04b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:17.010 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:16.991 7f4d3d823700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4d381b0ad0 0x7f4d38108930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:17.010 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:16.991 7f4d3d823700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f4d380d1c60 0x7f4d38108f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:17.010 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:16.991 7f4d3d823700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d380fe1b0 con 0x7f4d380d1860 2021-02-10T11:46:17.010 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:16.991 7f4d3d823700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d38109a50 con 0x7f4d380d1c60 2021-02-10T11:46:17.011 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:16.991 7f4d3d823700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d38088070 con 0x7f4d381b0ad0 2021-02-10T11:46:17.011 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:16.992 7f4d37574700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f4d380d1860 0x7f4d381b04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:17.011 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:16.992 7f4d37574700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f4d380d1860 0x7f4d381b04b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:59414/0 (socket says 172.21.2.112:59414) 2021-02-10T11:46:17.011 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:16.992 7f4d37574700 1 -- 172.21.2.112:0/3322893355 learned_addr learned my addr 172.21.2.112:0/3322893355 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:46:17.011 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:16.992 7f4d37d75700 1 --2- 172.21.2.112:0/3322893355 >> v2:172.21.2.120:3300/0 conn(0x7f4d380d1c60 0x7f4d38108f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:17.011 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:16.992 7f4d36d73700 1 --2- 172.21.2.112:0/3322893355 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4d381b0ad0 0x7f4d38108930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:17.011 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:16.992 7f4d37574700 1 -- 172.21.2.112:0/3322893355 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4d381b0ad0 msgr2=0x7f4d38108930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:46:17.011 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:16.992 7f4d37574700 1 --2- 172.21.2.112:0/3322893355 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4d381b0ad0 0x7f4d38108930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:17.011 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:16.992 7f4d37574700 1 -- 172.21.2.112:0/3322893355 >> v2:172.21.2.120:3300/0 conn(0x7f4d380d1c60 msgr2=0x7f4d38108f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:46:17.011 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:16.992 7f4d37574700 1 --2- 172.21.2.112:0/3322893355 >> v2:172.21.2.120:3300/0 conn(0x7f4d380d1c60 0x7f4d38108f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:17.011 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:16.992 7f4d37574700 1 -- 172.21.2.112:0/3322893355 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d38109570 con 0x7f4d380d1860 2021-02-10T11:46:17.011 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:16.992 7f4d37574700 1 --2- 172.21.2.112:0/3322893355 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4d380d1860 0x7f4d381b04b0 secure :-1 s=READY pgs=2858 cs=0 l=1 rev1=1 rx=0x7f4d20006700 tx=0x7f4d20009b00).ready entity=mon.0 client_cookie=ce48f1f7f4eb3dc6 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:17.011 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:16.992 7f4d36572700 1 -- 172.21.2.112:0/3322893355 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4d2000c0c0 con 0x7f4d380d1860 2021-02-10T11:46:17.012 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:16.992 7f4d36572700 1 -- 172.21.2.112:0/3322893355 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d2000c400 con 0x7f4d380d1860 2021-02-10T11:46:17.012 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:16.992 7f4d36572700 1 -- 172.21.2.112:0/3322893355 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4d2000c920 con 0x7f4d380d1860 2021-02-10T11:46:17.012 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:16.992 7f4d3d823700 1 -- 172.21.2.112:0/3322893355 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4d380d1860 msgr2=0x7f4d381b04b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:17.012 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:16.992 7f4d3d823700 1 --2- 172.21.2.112:0/3322893355 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4d380d1860 0x7f4d381b04b0 secure :-1 s=READY pgs=2858 cs=0 l=1 rev1=1 rx=0x7f4d20006700 tx=0x7f4d20009b00).stop 2021-02-10T11:46:17.012 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:16.993 7f4d3d823700 1 -- 172.21.2.112:0/3322893355 shutdown_connections 2021-02-10T11:46:17.012 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:16.993 7f4d3d823700 1 --2- 172.21.2.112:0/3322893355 >> v2:172.21.2.120:3300/0 conn(0x7f4d380d1c60 0x7f4d38108f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:17.012 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:16.993 7f4d3d823700 1 --2- 172.21.2.112:0/3322893355 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4d381b0ad0 0x7f4d38108930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:17.012 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:16.993 7f4d3d823700 1 --2- 172.21.2.112:0/3322893355 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4d380d1860 0x7f4d381b04b0 unknown :-1 s=CLOSED pgs=2858 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:17.012 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:16.993 7f4d3d823700 1 -- 172.21.2.112:0/3322893355 >> 172.21.2.112:0/3322893355 conn(0x7f4d380ce410 msgr2=0x7f4d380d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:17.012 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:16.993 7f4d3d823700 1 -- 172.21.2.112:0/3322893355 shutdown_connections 2021-02-10T11:46:17.012 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:16.993 7f4d3d823700 1 -- 172.21.2.112:0/3322893355 wait complete. 2021-02-10T11:46:17.012 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:17.008 7f4d3d823700 1 Processor -- start 2021-02-10T11:46:17.014 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:17.008 7f4d3d823700 1 -- start start 2021-02-10T11:46:17.014 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:17.008 7f4d3d823700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4d380badd0 0x7f4d3809eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:17.014 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:17.008 7f4d3d823700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4d380adf50 0x7f4d3809f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:17.014 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:17.008 7f4d3d823700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4d381a7470 0x7f4d381a7870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:17.014 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:17.008 7f4d3d823700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d3809e5a0 con 0x7f4d381a7470 2021-02-10T11:46:17.014 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:17.008 7f4d3d823700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d3809f6e0 con 0x7f4d380adf50 2021-02-10T11:46:17.014 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:17.008 7f4d3d823700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d380bb220 con 0x7f4d380badd0 2021-02-10T11:46:17.014 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:17.009 7f4d37d75700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4d381a7470 0x7f4d381a7870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:17.015 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:17.009 7f4d37d75700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4d381a7470 0x7f4d381a7870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:59420/0 (socket says 172.21.2.112:59420) 2021-02-10T11:46:17.015 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:17.009 7f4d37d75700 1 -- 172.21.2.112:0/3320085772 learned_addr learned my addr 172.21.2.112:0/3320085772 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:46:17.015 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:17.009 7f4d36d73700 1 --2- 172.21.2.112:0/3320085772 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4d380adf50 0x7f4d3809f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:17.015 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:17.009 7f4d37d75700 1 -- 172.21.2.112:0/3320085772 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4d380badd0 msgr2=0x7f4d3809eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:17.015 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:17.009 7f4d37d75700 1 --2- 172.21.2.112:0/3320085772 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4d380badd0 0x7f4d3809eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:17.015 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:17.009 7f4d37d75700 1 -- 172.21.2.112:0/3320085772 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4d380adf50 msgr2=0x7f4d3809f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:17.015 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:17.009 7f4d37d75700 1 --2- 172.21.2.112:0/3320085772 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4d380adf50 0x7f4d3809f0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:17.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:17.009 7f4d37d75700 1 -- 172.21.2.112:0/3320085772 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d381a7e90 con 0x7f4d381a7470 2021-02-10T11:46:17.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:17.009 7f4d36d73700 1 --2- 172.21.2.112:0/3320085772 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4d380adf50 0x7f4d3809f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:46:17.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:17.009 7f4d37d75700 1 --2- 172.21.2.112:0/3320085772 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4d381a7470 0x7f4d381a7870 secure :-1 s=READY pgs=2859 cs=0 l=1 rev1=1 rx=0x7f4d3000b970 tx=0x7f4d3000bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:17.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:17.010 7f4d2ffff700 1 -- 172.21.2.112:0/3320085772 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4d30000f90 con 0x7f4d381a7470 2021-02-10T11:46:17.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:17.010 7f4d2ffff700 1 -- 172.21.2.112:0/3320085772 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d30001400 con 0x7f4d381a7470 2021-02-10T11:46:17.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:17.010 7f4d3d823700 1 -- 172.21.2.112:0/3320085772 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d381a7e90 con 0x7f4d381a7470 2021-02-10T11:46:17.017 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:17.010 7f4d3d823700 1 -- 172.21.2.112:0/3320085772 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d3809d1f0 con 0x7f4d381a7470 2021-02-10T11:46:17.017 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:17.010 7f4d2ffff700 1 -- 172.21.2.112:0/3320085772 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4d30001920 con 0x7f4d381a7470 2021-02-10T11:46:17.017 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:17.010 7f4d2ffff700 1 -- 172.21.2.112:0/3320085772 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4d300015e0 con 0x7f4d381a7470 2021-02-10T11:46:17.017 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:17.010 7f4d2ffff700 1 --2- 172.21.2.112:0/3320085772 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4d1c0186c0 0x7f4d1c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:17.017 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:17.010 7f4d2ffff700 1 -- 172.21.2.112:0/3320085772 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(167..167 src has 1..167) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f4d30032600 con 0x7f4d381a7470 2021-02-10T11:46:17.017 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:17.011 7f4d37574700 1 --2- 172.21.2.112:0/3320085772 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4d1c0186c0 0x7f4d1c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:17.017 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:17.011 7f4d37574700 1 --2- 172.21.2.112:0/3320085772 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4d1c0186c0 0x7f4d1c01abc0 crc :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:17.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:17.041 7f4d3d823700 1 -- 172.21.2.112:0/3320085772 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d38082a20 con 0x7f4d381a7470 2021-02-10T11:46:17.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:17.044 7f4d2ffff700 1 -- 172.21.2.112:0/3320085772 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4d30000e30 con 0x7f4d381a7470 2021-02-10T11:46:17.174 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:17.172 7f4d3d823700 1 -- 172.21.2.112:0/3320085772 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f4d3809e520 con 0x7f4d381a7470 2021-02-10T11:46:17.174 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:17.172 7f4d2ffff700 1 -- 172.21.2.112:0/3320085772 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v167) v1 ==== 92+0+10 (secure 0 0 0) 0x7f4d300015e0 con 0x7f4d381a7470 2021-02-10T11:46:17.175 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:46:17.176 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:17.174 7f4d3d823700 1 -- 172.21.2.112:0/3320085772 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4d1c0186c0 msgr2=0x7f4d1c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:17.176 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:17.174 7f4d3d823700 1 --2- 172.21.2.112:0/3320085772 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4d1c0186c0 0x7f4d1c01abc0 crc :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:17.176 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:17.174 7f4d3d823700 1 -- 172.21.2.112:0/3320085772 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4d381a7470 msgr2=0x7f4d381a7870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:17.176 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:17.174 7f4d3d823700 1 --2- 172.21.2.112:0/3320085772 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4d381a7470 0x7f4d381a7870 secure :-1 s=READY pgs=2859 cs=0 l=1 rev1=1 rx=0x7f4d3000b970 tx=0x7f4d3000bd10).stop 2021-02-10T11:46:17.176 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:17.174 7f4d3d823700 1 -- 172.21.2.112:0/3320085772 shutdown_connections 2021-02-10T11:46:17.176 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:17.174 7f4d3d823700 1 --2- 172.21.2.112:0/3320085772 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4d1c0186c0 0x7f4d1c01abc0 unknown :-1 s=CLOSED pgs=1297 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:17.176 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:17.174 7f4d3d823700 1 --2- 172.21.2.112:0/3320085772 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4d380adf50 0x7f4d3809f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:17.176 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:17.174 7f4d3d823700 1 --2- 172.21.2.112:0/3320085772 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4d381a7470 0x7f4d381a7870 unknown :-1 s=CLOSED pgs=2859 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:17.176 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:17.174 7f4d3d823700 1 --2- 172.21.2.112:0/3320085772 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4d380badd0 0x7f4d3809eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:17.177 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:17.174 7f4d3d823700 1 -- 172.21.2.112:0/3320085772 >> 172.21.2.112:0/3320085772 conn(0x7f4d380bee90 msgr2=0x7f4d380bb880 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:17.177 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:17.174 7f4d3d823700 1 -- 172.21.2.112:0/3320085772 shutdown_connections 2021-02-10T11:46:17.177 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:17.174 7f4d3d823700 1 -- 172.21.2.112:0/3320085772 wait complete. 2021-02-10T11:46:17.185 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --format=json-pretty osd lspools 2021-02-10T11:46:17.614 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:17.595 7f957868c700 1 Processor -- start 2021-02-10T11:46:17.615 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:17.595 7f957868c700 1 -- start start 2021-02-10T11:46:17.615 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:17.596 7f957868c700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f95700d0340 0x7f95700ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:17.615 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:17.596 7f957868c700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f95700d1c70 0x7f95700efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:17.615 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:17.596 7f957868c700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95700f0b10 con 0x7f95700d1870 2021-02-10T11:46:17.615 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:17.596 7f957868c700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95700e5ed0 con 0x7f95700d0340 2021-02-10T11:46:17.615 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:17.596 7f957868c700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95701b0be0 con 0x7f95700d1c70 2021-02-10T11:46:17.615 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:17.596 7f9575ca7700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f95700d0340 0x7f95700ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:17.615 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:17.596 7f95764a8700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f95700d1870 0x7f95701b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:58886/0 (socket says 172.21.2.112:58886) 2021-02-10T11:46:17.615 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:17.596 7f9576ca9700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f95700d1c70 0x7f95700efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:17.615 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:17.596 7f95764a8700 1 -- 172.21.2.112:0/20574551 learned_addr learned my addr 172.21.2.112:0/20574551 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:46:17.615 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:17.596 7f9575ca7700 1 -- 172.21.2.112:0/20574551 >> v2:172.21.2.120:3300/0 conn(0x7f95700d1c70 msgr2=0x7f95700efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:46:17.616 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:17.596 7f9575ca7700 1 --2- 172.21.2.112:0/20574551 >> v2:172.21.2.120:3300/0 conn(0x7f95700d1c70 0x7f95700efc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:17.616 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:17.596 7f9575ca7700 1 -- 172.21.2.112:0/20574551 >> v1:172.21.2.112:6789/0 conn(0x7f95700d1870 legacy=0x7f95701b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:17.616 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:17.596 7f9575ca7700 1 -- 172.21.2.112:0/20574551 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95700f20a0 con 0x7f95700d0340 2021-02-10T11:46:17.616 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:17.597 7f9575ca7700 1 --2- 172.21.2.112:0/20574551 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f95700d0340 0x7f95700ef640 secure :-1 s=READY pgs=2861 cs=0 l=1 rev1=1 rx=0x7f9564006650 tx=0x7f9564009a80).ready entity=mon.0 client_cookie=e36d9cb5d4e3e862 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:17.616 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:17.597 7f95754a6700 1 -- 172.21.2.112:0/20574551 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f956400c0c0 con 0x7f95700d0340 2021-02-10T11:46:17.616 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:17.597 7f95754a6700 1 -- 172.21.2.112:0/20574551 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f956400c340 con 0x7f95700d0340 2021-02-10T11:46:17.616 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:17.597 7f95754a6700 1 -- 172.21.2.112:0/20574551 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f956400c860 con 0x7f95700d0340 2021-02-10T11:46:17.616 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:17.597 7f957868c700 1 -- 172.21.2.112:0/20574551 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f95700d0340 msgr2=0x7f95700ef640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:17.616 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:17.597 7f957868c700 1 --2- 172.21.2.112:0/20574551 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f95700d0340 0x7f95700ef640 secure :-1 s=READY pgs=2861 cs=0 l=1 rev1=1 rx=0x7f9564006650 tx=0x7f9564009a80).stop 2021-02-10T11:46:17.616 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:17.598 7f957868c700 1 -- 172.21.2.112:0/20574551 shutdown_connections 2021-02-10T11:46:17.616 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:17.598 7f957868c700 1 --2- 172.21.2.112:0/20574551 >> v2:172.21.2.120:3300/0 conn(0x7f95700d1c70 0x7f95700efc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:17.616 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:17.598 7f957868c700 1 --2- 172.21.2.112:0/20574551 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f95700d0340 0x7f95700ef640 unknown :-1 s=CLOSED pgs=2861 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:17.616 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:17.598 7f957868c700 1 -- 172.21.2.112:0/20574551 >> 172.21.2.112:0/20574551 conn(0x7f95700ce450 msgr2=0x7f95700d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:17.617 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:17.598 7f957868c700 1 -- 172.21.2.112:0/20574551 shutdown_connections 2021-02-10T11:46:17.617 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:17.598 7f957868c700 1 -- 172.21.2.112:0/20574551 wait complete. 2021-02-10T11:46:17.618 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:17.613 7f957868c700 1 Processor -- start 2021-02-10T11:46:17.619 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:17.613 7f957868c700 1 -- start start 2021-02-10T11:46:17.619 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:17.613 7f957868c700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f95700970b0 0x7f95700ebe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:17.619 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:17.613 7f957868c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9570098810 0x7f9570098c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:17.619 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:17.613 7f957868c700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9570099230 0x7f95700b18c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:17.619 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:17.613 7f957868c700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95700ec6d0 con 0x7f9570098810 2021-02-10T11:46:17.619 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:17.613 7f957868c700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95700974b0 con 0x7f9570099230 2021-02-10T11:46:17.619 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:17.613 7f957868c700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95700b1ee0 con 0x7f95700970b0 2021-02-10T11:46:17.619 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:17.613 7f9575ca7700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9570098810 0x7f9570098c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:17.619 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:17.613 7f9575ca7700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9570098810 0x7f9570098c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:59432/0 (socket says 172.21.2.112:59432) 2021-02-10T11:46:17.619 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:17.613 7f9575ca7700 1 -- 172.21.2.112:0/1463123394 learned_addr learned my addr 172.21.2.112:0/1463123394 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:46:17.619 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:17.613 7f95764a8700 1 --2- 172.21.2.112:0/1463123394 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f95700970b0 0x7f95700ebe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:17.619 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:17.614 7f9576ca9700 1 --2- 172.21.2.112:0/1463123394 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9570099230 0x7f95700b18c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:17.619 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:17.614 7f9575ca7700 1 -- 172.21.2.112:0/1463123394 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f95700970b0 msgr2=0x7f95700ebe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:17.620 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:17.614 7f9575ca7700 1 --2- 172.21.2.112:0/1463123394 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f95700970b0 0x7f95700ebe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:17.620 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:17.614 7f9575ca7700 1 -- 172.21.2.112:0/1463123394 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9570099230 msgr2=0x7f95700b18c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:17.620 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:17.614 7f9575ca7700 1 --2- 172.21.2.112:0/1463123394 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9570099230 0x7f95700b18c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:17.620 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:17.614 7f9575ca7700 1 -- 172.21.2.112:0/1463123394 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95700b2100 con 0x7f9570098810 2021-02-10T11:46:17.620 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:17.614 7f9575ca7700 1 --2- 172.21.2.112:0/1463123394 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9570098810 0x7f9570098c10 secure :-1 s=READY pgs=2862 cs=0 l=1 rev1=1 rx=0x7f9564009630 tx=0x7f956400b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:17.620 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:17.614 7f9562ffd700 1 -- 172.21.2.112:0/1463123394 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f95640093a0 con 0x7f9570098810 2021-02-10T11:46:17.620 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:17.614 7f9562ffd700 1 -- 172.21.2.112:0/1463123394 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f956400bd70 con 0x7f9570098810 2021-02-10T11:46:17.620 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:17.614 7f957868c700 1 -- 172.21.2.112:0/1463123394 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95700b2100 con 0x7f9570098810 2021-02-10T11:46:17.620 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:17.614 7f957868c700 1 -- 172.21.2.112:0/1463123394 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95700b2690 con 0x7f9570098810 2021-02-10T11:46:17.620 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:17.614 7f9562ffd700 1 -- 172.21.2.112:0/1463123394 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f95640054d0 con 0x7f9570098810 2021-02-10T11:46:17.620 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:17.615 7f9562ffd700 1 -- 172.21.2.112:0/1463123394 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f956400a000 con 0x7f9570098810 2021-02-10T11:46:17.620 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:17.615 7f9562ffd700 1 --2- 172.21.2.112:0/1463123394 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9558018730 0x7f955801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:17.620 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:17.615 7f9562ffd700 1 -- 172.21.2.112:0/1463123394 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(167..167 src has 1..167) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f95640324a0 con 0x7f9570098810 2021-02-10T11:46:17.621 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:17.615 7f95764a8700 1 --2- 172.21.2.112:0/1463123394 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9558018730 0x7f955801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:17.621 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:17.616 7f95764a8700 1 --2- 172.21.2.112:0/1463123394 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9558018730 0x7f955801ac10 crc :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:17.648 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:17.646 7f957868c700 1 -- 172.21.2.112:0/1463123394 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95700ebc00 con 0x7f9570098810 2021-02-10T11:46:17.650 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:17.648 7f9562ffd700 1 -- 172.21.2.112:0/1463123394 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9564009340 con 0x7f9570098810 2021-02-10T11:46:17.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:17.771 7f957868c700 1 -- 172.21.2.112:0/1463123394 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f95700f0b10 con 0x7f9570098810 2021-02-10T11:46:17.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:17.771 7f9562ffd700 1 -- 172.21.2.112:0/1463123394 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v167) v1 ==== 84+0+64 (secure 0 0 0) 0x7f956400bd70 con 0x7f9570098810 2021-02-10T11:46:17.774 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:46:17.774 INFO:teuthology.orchestra.run.gibba012.stdout:[ 2021-02-10T11:46:17.774 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:46:17.774 INFO:teuthology.orchestra.run.gibba012.stdout: "poolnum": 1, 2021-02-10T11:46:17.774 INFO:teuthology.orchestra.run.gibba012.stdout: "poolname": "rbd" 2021-02-10T11:46:17.774 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:46:17.774 INFO:teuthology.orchestra.run.gibba012.stdout:] 2021-02-10T11:46:17.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:17.773 7f957868c700 1 -- 172.21.2.112:0/1463123394 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9558018730 msgr2=0x7f955801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:17.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:17.773 7f957868c700 1 --2- 172.21.2.112:0/1463123394 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9558018730 0x7f955801ac10 crc :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:17.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:17.773 7f957868c700 1 -- 172.21.2.112:0/1463123394 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9570098810 msgr2=0x7f9570098c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:17.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:17.773 7f957868c700 1 --2- 172.21.2.112:0/1463123394 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9570098810 0x7f9570098c10 secure :-1 s=READY pgs=2862 cs=0 l=1 rev1=1 rx=0x7f9564009630 tx=0x7f956400b980).stop 2021-02-10T11:46:17.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:17.773 7f957868c700 1 -- 172.21.2.112:0/1463123394 shutdown_connections 2021-02-10T11:46:17.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:17.773 7f957868c700 1 --2- 172.21.2.112:0/1463123394 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9558018730 0x7f955801ac10 unknown :-1 s=CLOSED pgs=1298 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:17.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:17.773 7f957868c700 1 --2- 172.21.2.112:0/1463123394 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9570099230 0x7f95700b18c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:17.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:17.773 7f957868c700 1 --2- 172.21.2.112:0/1463123394 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9570098810 0x7f9570098c10 unknown :-1 s=CLOSED pgs=2862 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:17.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:17.773 7f957868c700 1 --2- 172.21.2.112:0/1463123394 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f95700970b0 0x7f95700ebe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:17.777 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:17.773 7f957868c700 1 -- 172.21.2.112:0/1463123394 >> 172.21.2.112:0/1463123394 conn(0x7f957019a730 msgr2=0x7f957019e480 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:17.777 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:17.774 7f957868c700 1 -- 172.21.2.112:0/1463123394 shutdown_connections 2021-02-10T11:46:17.777 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:17.774 7f957868c700 1 -- 172.21.2.112:0/1463123394 wait complete. 2021-02-10T11:46:17.784 INFO:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2021-02-10T11:46:17.784 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_metadata 8 2021-02-10T11:46:18.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:18.195 7f9b76db9700 1 Processor -- start 2021-02-10T11:46:18.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:18.196 7f9b76db9700 1 -- start start 2021-02-10T11:46:18.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:18.196 7f9b76db9700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f9b700d1860 0x7f9b701b04b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:18.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:18.196 7f9b76db9700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b700fe1b0 con 0x7f9b701b0ad0 2021-02-10T11:46:18.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:18.196 7f9b76db9700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b70109a50 con 0x7f9b700cff60 2021-02-10T11:46:18.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:18.196 7f9b76db9700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b70088070 con 0x7f9b700d1860 2021-02-10T11:46:18.216 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:18.196 7f9b6ffff700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f9b701b0ad0 0x7f9b70108930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:58900/0 (socket says 172.21.2.112:58900) 2021-02-10T11:46:18.216 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:18.196 7f9b6ffff700 1 -- 172.21.2.112:0/438964933 learned_addr learned my addr 172.21.2.112:0/438964933 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:46:18.216 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:18.196 7f9b6f7fe700 1 -- 172.21.2.112:0/438964933 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4006527732 0 0) 0x7f9b60001060 con 0x7f9b701b0ad0 2021-02-10T11:46:18.216 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:18.196 7f9b6f7fe700 1 -- 172.21.2.112:0/438964933 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b5c002fe0 con 0x7f9b701b0ad0 2021-02-10T11:46:18.216 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:18.197 7f9b6f7fe700 1 -- 172.21.2.112:0/438964933 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 509926788 0 0) 0x7f9b60001500 con 0x7f9b701b0ad0 2021-02-10T11:46:18.216 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:18.197 7f9b6f7fe700 1 -- 172.21.2.112:0/438964933 >> v2:172.21.2.120:3300/0 conn(0x7f9b700d1860 msgr2=0x7f9b701b04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:46:18.216 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:18.197 7f9b6f7fe700 1 --2- 172.21.2.112:0/438964933 >> v2:172.21.2.120:3300/0 conn(0x7f9b700d1860 0x7f9b701b04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:18.216 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:18.197 7f9b6f7fe700 1 -- 172.21.2.112:0/438964933 >> v1:172.21.2.120:6789/0 conn(0x7f9b700cff60 legacy=0x7f9b70109120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:46:18.216 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:18.197 7f9b6f7fe700 1 -- 172.21.2.112:0/438964933 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b70088260 con 0x7f9b701b0ad0 2021-02-10T11:46:18.216 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:18.197 7f9b6f7fe700 1 -- 172.21.2.112:0/438964933 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f9b60001a00 con 0x7f9b701b0ad0 2021-02-10T11:46:18.216 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:18.197 7f9b6f7fe700 1 -- 172.21.2.112:0/438964933 >> v1:172.21.2.112:6789/0 conn(0x7f9b701b0ad0 legacy=0x7f9b70108930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:18.216 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:18.197 7f9b6f7fe700 1 --2- 172.21.2.112:0/438964933 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9b5c003af0 0x7f9b5c005f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:18.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:18.197 7f9b6f7fe700 1 --2- 172.21.2.112:0/438964933 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9b5c006780 0x7f9b5c008c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:18.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:18.197 7f9b6f7fe700 1 --2- 172.21.2.112:0/438964933 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9b5c009580 0x7f9b5c00baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:18.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:18.197 7f9b6f7fe700 1 -- 172.21.2.112:0/438964933 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b5c00c2f0 con 0x7f9b5c003af0 2021-02-10T11:46:18.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:18.197 7f9b6f7fe700 1 -- 172.21.2.112:0/438964933 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b5c00c510 con 0x7f9b5c006780 2021-02-10T11:46:18.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:18.197 7f9b6f7fe700 1 -- 172.21.2.112:0/438964933 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b5c00c730 con 0x7f9b5c009580 2021-02-10T11:46:18.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:18.197 7f9b6ffff700 1 --2- 172.21.2.112:0/438964933 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9b5c006780 0x7f9b5c008c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:18.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:18.197 7f9b74bd5700 1 --2- 172.21.2.112:0/438964933 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9b5c003af0 0x7f9b5c005f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:18.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:18.197 7f9b753d6700 1 --2- 172.21.2.112:0/438964933 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9b5c009580 0x7f9b5c00baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:18.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:18.197 7f9b74bd5700 1 -- 172.21.2.112:0/438964933 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9b5c009580 msgr2=0x7f9b5c00baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:46:18.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:18.197 7f9b74bd5700 1 --2- 172.21.2.112:0/438964933 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9b5c009580 0x7f9b5c00baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:18.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:18.197 7f9b74bd5700 1 -- 172.21.2.112:0/438964933 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9b5c006780 msgr2=0x7f9b5c008c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:46:18.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:18.197 7f9b74bd5700 1 --2- 172.21.2.112:0/438964933 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9b5c006780 0x7f9b5c008c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:18.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:18.197 7f9b74bd5700 1 -- 172.21.2.112:0/438964933 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b5c00c9d0 con 0x7f9b5c003af0 2021-02-10T11:46:18.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:18.197 7f9b753d6700 1 --2- 172.21.2.112:0/438964933 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9b5c009580 0x7f9b5c00baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:46:18.218 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:18.198 7f9b753d6700 1 -- 172.21.2.112:0/438964933 reap_dead start 2021-02-10T11:46:18.218 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:18.198 7f9b74bd5700 1 --2- 172.21.2.112:0/438964933 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9b5c003af0 0x7f9b5c005f00 secure :-1 s=READY pgs=2864 cs=0 l=1 rev1=1 rx=0x7f9b58006bf0 tx=0x7f9b58009c50).ready entity=mon.0 client_cookie=816e0a31b908971d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:18.218 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:18.198 7f9b6f7fe700 1 -- 172.21.2.112:0/438964933 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9b58000c90 con 0x7f9b5c003af0 2021-02-10T11:46:18.218 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:18.198 7f9b6f7fe700 1 -- 172.21.2.112:0/438964933 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b58002030 con 0x7f9b5c003af0 2021-02-10T11:46:18.218 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:18.198 7f9b6f7fe700 1 -- 172.21.2.112:0/438964933 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9b58002490 con 0x7f9b5c003af0 2021-02-10T11:46:18.218 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:18.198 7f9b76db9700 1 -- 172.21.2.112:0/438964933 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9b5c003af0 msgr2=0x7f9b5c005f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:18.218 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:18.198 7f9b76db9700 1 --2- 172.21.2.112:0/438964933 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9b5c003af0 0x7f9b5c005f00 secure :-1 s=READY pgs=2864 cs=0 l=1 rev1=1 rx=0x7f9b58006bf0 tx=0x7f9b58009c50).stop 2021-02-10T11:46:18.218 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:18.198 7f9b76db9700 1 -- 172.21.2.112:0/438964933 shutdown_connections 2021-02-10T11:46:18.218 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:18.198 7f9b76db9700 1 --2- 172.21.2.112:0/438964933 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9b5c003af0 0x7f9b5c005f00 unknown :-1 s=CLOSED pgs=2864 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:18.218 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:18.198 7f9b76db9700 1 -- 172.21.2.112:0/438964933 >> 172.21.2.112:0/438964933 conn(0x7f9b700ce410 msgr2=0x7f9b700d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:18.218 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:18.198 7f9b76db9700 1 -- 172.21.2.112:0/438964933 shutdown_connections 2021-02-10T11:46:18.218 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:18.198 7f9b76db9700 1 -- 172.21.2.112:0/438964933 wait complete. 2021-02-10T11:46:18.219 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:18.213 7f9b76db9700 1 Processor -- start 2021-02-10T11:46:18.219 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:18.213 7f9b76db9700 1 -- start start 2021-02-10T11:46:18.220 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:18.213 7f9b76db9700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9b700adf50 0x7f9b700ae350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:18.220 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:18.213 7f9b76db9700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9b700ae970 0x7f9b700aed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:18.220 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:18.213 7f9b76db9700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9b700982c0 0x7f9b700986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:18.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:18.213 7f9b76db9700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b7009f450 con 0x7f9b700ae970 2021-02-10T11:46:18.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:18.213 7f9b76db9700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b70098ce0 con 0x7f9b700adf50 2021-02-10T11:46:18.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:18.213 7f9b76db9700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b70098ed0 con 0x7f9b700982c0 2021-02-10T11:46:18.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:18.214 7f9b74bd5700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9b700adf50 0x7f9b700ae350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:18.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:18.214 7f9b6ffff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9b700ae970 0x7f9b700aed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:18.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:18.214 7f9b753d6700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9b700982c0 0x7f9b700986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:18.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:18.214 7f9b74bd5700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9b700adf50 0x7f9b700ae350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3300/0 says I am v2:172.21.2.112:33498/0 (socket says 172.21.2.112:33498) 2021-02-10T11:46:18.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:18.214 7f9b74bd5700 1 -- 172.21.2.112:0/210784628 learned_addr learned my addr 172.21.2.112:0/210784628 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:46:18.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:18.214 7f9b753d6700 1 -- 172.21.2.112:0/210784628 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9b700adf50 msgr2=0x7f9b700ae350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:18.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:18.214 7f9b753d6700 1 --2- 172.21.2.112:0/210784628 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9b700adf50 0x7f9b700ae350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:18.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:18.214 7f9b753d6700 1 -- 172.21.2.112:0/210784628 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9b700ae970 msgr2=0x7f9b700aed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:18.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:18.214 7f9b753d6700 1 --2- 172.21.2.112:0/210784628 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9b700ae970 0x7f9b700aed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:18.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:18.214 7f9b753d6700 1 -- 172.21.2.112:0/210784628 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b700990f0 con 0x7f9b700982c0 2021-02-10T11:46:18.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:18.214 7f9b753d6700 1 --2- 172.21.2.112:0/210784628 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9b700982c0 0x7f9b700986c0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 rx=0x7f9b68009540 tx=0x7f9b6800b890).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:18.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:18.214 7f9b6d7fa700 1 -- 172.21.2.112:0/210784628 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9b6800bc80 con 0x7f9b700982c0 2021-02-10T11:46:18.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:18.214 7f9b6d7fa700 1 -- 172.21.2.112:0/210784628 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b68005330 con 0x7f9b700982c0 2021-02-10T11:46:18.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:18.214 7f9b76db9700 1 -- 172.21.2.112:0/210784628 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b700990f0 con 0x7f9b700982c0 2021-02-10T11:46:18.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:18.214 7f9b6d7fa700 1 -- 172.21.2.112:0/210784628 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9b68005810 con 0x7f9b700982c0 2021-02-10T11:46:18.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:18.214 7f9b76db9700 1 -- 172.21.2.112:0/210784628 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b700990f0 con 0x7f9b700982c0 2021-02-10T11:46:18.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:18.215 7f9b6d7fa700 1 -- 172.21.2.112:0/210784628 <== mon.2 v2:172.21.2.120:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9b68000a50 con 0x7f9b700982c0 2021-02-10T11:46:18.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:18.215 7f9b6d7fa700 1 --2- 172.21.2.112:0/210784628 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9b540186c0 0x7f9b5401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:18.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:18.215 7f9b6d7fa700 1 -- 172.21.2.112:0/210784628 <== mon.2 v2:172.21.2.120:3301/0 5 ==== osd_map(167..167 src has 1..167) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f9b680323a0 con 0x7f9b700982c0 2021-02-10T11:46:18.223 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:18.215 7f9b74bd5700 1 --2- 172.21.2.112:0/210784628 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9b540186c0 0x7f9b5401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:18.223 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:18.216 7f9b74bd5700 1 --2- 172.21.2.112:0/210784628 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9b540186c0 0x7f9b5401abc0 crc :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:18.248 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:18.246 7f9b76db9700 1 -- 172.21.2.112:0/210784628 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b70082a20 con 0x7f9b700982c0 2021-02-10T11:46:18.250 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:18.248 7f9b6d7fa700 1 -- 172.21.2.112:0/210784628 <== mon.2 v2:172.21.2.120:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9b6800bc80 con 0x7f9b700982c0 2021-02-10T11:46:18.378 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:18.375 7f9b76db9700 1 -- 172.21.2.112:0/210784628 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"} v 0) v1 -- 0x7f9b70082a20 con 0x7f9b700982c0 2021-02-10T11:46:18.788 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:18.786 7f9b6d7fa700 1 -- 172.21.2.112:0/210784628 <== mon.2 v2:172.21.2.120:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"}]=0 pool 'cephfs_metadata' created v168) v1 ==== 133+0+0 (secure 0 0 0) 0x7f9b68000a50 con 0x7f9b700982c0 2021-02-10T11:46:18.792 INFO:teuthology.orchestra.run.gibba012.stderr:pool 'cephfs_metadata' created 2021-02-10T11:46:18.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:18.792 7f9b76db9700 1 -- 172.21.2.112:0/210784628 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9b540186c0 msgr2=0x7f9b5401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:18.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:18.792 7f9b76db9700 1 --2- 172.21.2.112:0/210784628 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9b540186c0 0x7f9b5401abc0 crc :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:18.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:18.792 7f9b76db9700 1 -- 172.21.2.112:0/210784628 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9b700982c0 msgr2=0x7f9b700986c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:18.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:18.792 7f9b76db9700 1 --2- 172.21.2.112:0/210784628 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9b700982c0 0x7f9b700986c0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 rx=0x7f9b68009540 tx=0x7f9b6800b890).stop 2021-02-10T11:46:18.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:18.792 7f9b76db9700 1 -- 172.21.2.112:0/210784628 shutdown_connections 2021-02-10T11:46:18.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:18.792 7f9b76db9700 1 --2- 172.21.2.112:0/210784628 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9b540186c0 0x7f9b5401abc0 unknown :-1 s=CLOSED pgs=1299 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:18.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:18.792 7f9b76db9700 1 --2- 172.21.2.112:0/210784628 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9b700982c0 0x7f9b700986c0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:18.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:18.792 7f9b76db9700 1 --2- 172.21.2.112:0/210784628 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9b700ae970 0x7f9b700aed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:18.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:18.792 7f9b76db9700 1 --2- 172.21.2.112:0/210784628 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9b700adf50 0x7f9b700ae350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:18.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:18.792 7f9b76db9700 1 -- 172.21.2.112:0/210784628 >> 172.21.2.112:0/210784628 conn(0x7f9b700bee90 msgr2=0x7f9b700bb880 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:18.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:18.792 7f9b76db9700 1 -- 172.21.2.112:0/210784628 shutdown_connections 2021-02-10T11:46:18.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:18.792 7f9b76db9700 1 -- 172.21.2.112:0/210784628 wait complete. 2021-02-10T11:46:18.803 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_data 8 2021-02-10T11:46:19.233 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:19.215 7f7edbe5d700 1 Processor -- start 2021-02-10T11:46:19.233 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:19.215 7f7edbe5d700 1 -- start start 2021-02-10T11:46:19.234 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:19.215 7f7edbe5d700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f7ed40d1870 0x7f7ed41b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:19.234 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:19.215 7f7edbe5d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f7ed40cff70 0x7f7ed4109130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:19.234 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:19.215 7f7edbe5d700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ed40fe1c0 con 0x7f7ed41b0ae0 2021-02-10T11:46:19.234 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:19.215 7f7edbe5d700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ed4109ad0 con 0x7f7ed40cff70 2021-02-10T11:46:19.234 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:19.215 7f7edbe5d700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ed4109750 con 0x7f7ed40d1870 2021-02-10T11:46:19.234 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:19.215 7f7eda47a700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f7ed40cff70 0x7f7ed4109130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:19.234 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:19.216 7f7eda47a700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f7ed40cff70 0x7f7ed4109130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:59458/0 (socket says 172.21.2.112:59458) 2021-02-10T11:46:19.234 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:19.216 7f7eda47a700 1 -- 172.21.2.112:0/2346613263 learned_addr learned my addr 172.21.2.112:0/2346613263 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:46:19.234 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:19.216 7f7eda47a700 1 -- 172.21.2.112:0/2346613263 >> v2:172.21.2.120:3300/0 conn(0x7f7ed40d1870 msgr2=0x7f7ed41b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:46:19.234 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:19.216 7f7eda47a700 1 --2- 172.21.2.112:0/2346613263 >> v2:172.21.2.120:3300/0 conn(0x7f7ed40d1870 0x7f7ed41b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:19.234 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:19.216 7f7eda47a700 1 -- 172.21.2.112:0/2346613263 >> v1:172.21.2.120:6789/0 conn(0x7f7ed41b0ae0 legacy=0x7f7ed4108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:46:19.234 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:19.216 7f7eda47a700 1 -- 172.21.2.112:0/2346613263 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ed4088070 con 0x7f7ed40cff70 2021-02-10T11:46:19.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:19.216 7f7eda47a700 1 --2- 172.21.2.112:0/2346613263 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7ed40cff70 0x7f7ed4109130 secure :-1 s=READY pgs=2865 cs=0 l=1 rev1=1 rx=0x7f7ecc006f00 tx=0x7f7ecc009f10).ready entity=mon.0 client_cookie=bf5479328872f331 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:19.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:19.216 7f7ed8c77700 1 -- 172.21.2.112:0/2346613263 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7ecc00b3b0 con 0x7f7ed40cff70 2021-02-10T11:46:19.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:19.216 7f7ed8c77700 1 -- 172.21.2.112:0/2346613263 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ecc00b730 con 0x7f7ed40cff70 2021-02-10T11:46:19.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:19.216 7f7ed8c77700 1 -- 172.21.2.112:0/2346613263 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7ecc00bb90 con 0x7f7ed40cff70 2021-02-10T11:46:19.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:19.216 7f7edbe5d700 1 -- 172.21.2.112:0/2346613263 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7ed40cff70 msgr2=0x7f7ed4109130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:19.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:19.216 7f7edbe5d700 1 --2- 172.21.2.112:0/2346613263 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7ed40cff70 0x7f7ed4109130 secure :-1 s=READY pgs=2865 cs=0 l=1 rev1=1 rx=0x7f7ecc006f00 tx=0x7f7ecc009f10).stop 2021-02-10T11:46:19.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:19.217 7f7edbe5d700 1 -- 172.21.2.112:0/2346613263 shutdown_connections 2021-02-10T11:46:19.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:19.217 7f7edbe5d700 1 --2- 172.21.2.112:0/2346613263 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7ed40cff70 0x7f7ed4109130 unknown :-1 s=CLOSED pgs=2865 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:19.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:19.217 7f7edbe5d700 1 --2- 172.21.2.112:0/2346613263 >> v2:172.21.2.120:3300/0 conn(0x7f7ed40d1870 0x7f7ed41b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:19.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:19.217 7f7edbe5d700 1 -- 172.21.2.112:0/2346613263 >> 172.21.2.112:0/2346613263 conn(0x7f7ed40ce420 msgr2=0x7f7ed40d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:19.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:19.217 7f7edbe5d700 1 -- 172.21.2.112:0/2346613263 shutdown_connections 2021-02-10T11:46:19.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:19.217 7f7edbe5d700 1 -- 172.21.2.112:0/2346613263 wait complete. 2021-02-10T11:46:19.237 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:19.232 7f7edbe5d700 1 Processor -- start 2021-02-10T11:46:19.237 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:19.232 7f7edbe5d700 1 -- start start 2021-02-10T11:46:19.237 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:19.232 7f7edbe5d700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7ed40c3820 0x7f7ed40a7950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:19.237 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:19.232 7f7edbe5d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7ed41a44c0 0x7f7ed41a48c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:19.237 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:19.232 7f7edbe5d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7ed40863d0 0x7f7ed40867d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:19.237 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:19.232 7f7edbe5d700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ed4085ed0 con 0x7f7ed40863d0 2021-02-10T11:46:19.237 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:19.232 7f7edbe5d700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ed40c0740 con 0x7f7ed40c3820 2021-02-10T11:46:19.237 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:19.232 7f7edbe5d700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ed4086df0 con 0x7f7ed41a44c0 2021-02-10T11:46:19.237 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:19.232 7f7eda47a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7ed40863d0 0x7f7ed40867d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:19.237 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:19.232 7f7eda47a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7ed40863d0 0x7f7ed40867d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:59464/0 (socket says 172.21.2.112:59464) 2021-02-10T11:46:19.237 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:19.232 7f7eda47a700 1 -- 172.21.2.112:0/2591288355 learned_addr learned my addr 172.21.2.112:0/2591288355 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:46:19.237 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:19.233 7f7eda47a700 1 -- 172.21.2.112:0/2591288355 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7ed41a44c0 msgr2=0x7f7ed41a48c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:46:19.238 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:19.233 7f7eda47a700 1 --2- 172.21.2.112:0/2591288355 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7ed41a44c0 0x7f7ed41a48c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:19.238 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:19.233 7f7eda47a700 1 -- 172.21.2.112:0/2591288355 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7ed40c3820 msgr2=0x7f7ed40a7950 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:46:19.238 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:19.233 7f7eda47a700 1 --2- 172.21.2.112:0/2591288355 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7ed40c3820 0x7f7ed40a7950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:19.238 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:19.233 7f7eda47a700 1 -- 172.21.2.112:0/2591288355 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ed4087010 con 0x7f7ed40863d0 2021-02-10T11:46:19.238 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:19.233 7f7eda47a700 1 --2- 172.21.2.112:0/2591288355 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7ed40863d0 0x7f7ed40867d0 secure :-1 s=READY pgs=2866 cs=0 l=1 rev1=1 rx=0x7f7ecc009390 tx=0x7f7ecc00bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:19.238 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:19.233 7f7ec67fc700 1 -- 172.21.2.112:0/2591288355 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7ecc003030 con 0x7f7ed40863d0 2021-02-10T11:46:19.238 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:19.233 7f7ec67fc700 1 -- 172.21.2.112:0/2591288355 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ecc0068e0 con 0x7f7ed40863d0 2021-02-10T11:46:19.238 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:19.233 7f7ec67fc700 1 -- 172.21.2.112:0/2591288355 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7ecc00c570 con 0x7f7ed40863d0 2021-02-10T11:46:19.238 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:19.233 7f7edbe5d700 1 -- 172.21.2.112:0/2591288355 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ed4087010 con 0x7f7ed40863d0 2021-02-10T11:46:19.238 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:19.233 7f7edbe5d700 1 -- 172.21.2.112:0/2591288355 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ed4087010 con 0x7f7ed40863d0 2021-02-10T11:46:19.238 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:19.234 7f7ec67fc700 1 -- 172.21.2.112:0/2591288355 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7ecc005820 con 0x7f7ed40863d0 2021-02-10T11:46:19.238 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:19.234 7f7ec67fc700 1 --2- 172.21.2.112:0/2591288355 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7ebc0186c0 0x7f7ebc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:19.238 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:19.234 7f7ec67fc700 1 -- 172.21.2.112:0/2591288355 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(169..169 src has 1..169) v4 ==== 5107+0+0 (secure 0 0 0) 0x7f7ecc0326e0 con 0x7f7ed40863d0 2021-02-10T11:46:19.239 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:19.234 7f7ed9c79700 1 --2- 172.21.2.112:0/2591288355 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7ebc0186c0 0x7f7ebc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:19.239 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:19.235 7f7ed9c79700 1 --2- 172.21.2.112:0/2591288355 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7ebc0186c0 0x7f7ebc01abc0 crc :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:19.267 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:19.265 7f7edbe5d700 1 -- 172.21.2.112:0/2591288355 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ed40fe1c0 con 0x7f7ed40863d0 2021-02-10T11:46:19.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:19.268 7f7ec67fc700 1 -- 172.21.2.112:0/2591288355 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7ecc006c80 con 0x7f7ed40863d0 2021-02-10T11:46:19.398 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:19.396 7f7edbe5d700 1 -- 172.21.2.112:0/2591288355 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"} v 0) v1 -- 0x7f7ed40fe1c0 con 0x7f7ed40863d0 2021-02-10T11:46:19.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:19.794 7f7ec67fc700 1 -- 172.21.2.112:0/2591288355 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"}]=0 pool 'cephfs_data' created v170) v1 ==== 125+0+0 (secure 0 0 0) 0x7f7ecc005820 con 0x7f7ed40863d0 2021-02-10T11:46:19.814 INFO:teuthology.orchestra.run.gibba012.stderr:pool 'cephfs_data' created 2021-02-10T11:46:19.816 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:19.813 7f7edbe5d700 1 -- 172.21.2.112:0/2591288355 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7ebc0186c0 msgr2=0x7f7ebc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:19.816 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:19.813 7f7edbe5d700 1 --2- 172.21.2.112:0/2591288355 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7ebc0186c0 0x7f7ebc01abc0 crc :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:19.816 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:19.813 7f7edbe5d700 1 -- 172.21.2.112:0/2591288355 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7ed40863d0 msgr2=0x7f7ed40867d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:19.816 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:19.813 7f7edbe5d700 1 --2- 172.21.2.112:0/2591288355 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7ed40863d0 0x7f7ed40867d0 secure :-1 s=READY pgs=2866 cs=0 l=1 rev1=1 rx=0x7f7ecc009390 tx=0x7f7ecc00bd40).stop 2021-02-10T11:46:19.816 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:19.814 7f7edbe5d700 1 -- 172.21.2.112:0/2591288355 shutdown_connections 2021-02-10T11:46:19.816 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:19.814 7f7edbe5d700 1 --2- 172.21.2.112:0/2591288355 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7ebc0186c0 0x7f7ebc01abc0 unknown :-1 s=CLOSED pgs=1300 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:19.816 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:19.814 7f7edbe5d700 1 --2- 172.21.2.112:0/2591288355 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7ed40863d0 0x7f7ed40867d0 unknown :-1 s=CLOSED pgs=2866 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:19.816 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:19.814 7f7edbe5d700 1 --2- 172.21.2.112:0/2591288355 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7ed41a44c0 0x7f7ed41a48c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:19.816 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:19.814 7f7edbe5d700 1 --2- 172.21.2.112:0/2591288355 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7ed40c3820 0x7f7ed40a7950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:19.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:19.814 7f7edbe5d700 1 -- 172.21.2.112:0/2591288355 >> 172.21.2.112:0/2591288355 conn(0x7f7ed40872c0 msgr2=0x7f7ed4095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:19.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:19.814 7f7edbe5d700 1 -- 172.21.2.112:0/2591288355 shutdown_connections 2021-02-10T11:46:19.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:19.814 7f7edbe5d700 1 -- 172.21.2.112:0/2591288355 wait complete. 2021-02-10T11:46:19.824 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2021-02-10T11:46:20.257 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.237 7ff1c83c4700 1 Processor -- start 2021-02-10T11:46:20.258 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.237 7ff1c83c4700 1 -- start start 2021-02-10T11:46:20.258 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.238 7ff1c83c4700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7ff1c00d1880 0x7ff1c01b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:20.258 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.238 7ff1c83c4700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1c00fe1d0 con 0x7ff1c00cff80 2021-02-10T11:46:20.258 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.238 7ff1c83c4700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1c0109a70 con 0x7ff1c01b0af0 2021-02-10T11:46:20.258 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.238 7ff1c83c4700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1c0088070 con 0x7ff1c00d1880 2021-02-10T11:46:20.258 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.238 7ff1c69e1700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7ff1c00cff80 0x7ff1c0109140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:58934/0 (socket says 172.21.2.112:58934) 2021-02-10T11:46:20.258 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.238 7ff1c69e1700 1 -- 172.21.2.112:0/2864291938 learned_addr learned my addr 172.21.2.112:0/2864291938 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:46:20.258 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.238 7ff1c51de700 1 -- 172.21.2.112:0/2864291938 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3213599138 0 0) 0x7ff1b8001500 con 0x7ff1c00cff80 2021-02-10T11:46:20.258 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.238 7ff1c51de700 1 -- 172.21.2.112:0/2864291938 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1ac002fe0 con 0x7ff1c00cff80 2021-02-10T11:46:20.258 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.238 7ff1c61e0700 1 --2- 172.21.2.112:0/2864291938 >> v2:172.21.2.120:3300/0 conn(0x7ff1c00d1880 0x7ff1c01b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:20.258 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.239 7ff1c51de700 1 -- 172.21.2.112:0/2864291938 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 38230043 0 0) 0x7ff1b8001460 con 0x7ff1c00cff80 2021-02-10T11:46:20.258 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.239 7ff1c51de700 1 -- 172.21.2.112:0/2864291938 >> v2:172.21.2.120:3300/0 conn(0x7ff1c00d1880 msgr2=0x7ff1c01b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:46:20.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.239 7ff1c51de700 1 --2- 172.21.2.112:0/2864291938 >> v2:172.21.2.120:3300/0 conn(0x7ff1c00d1880 0x7ff1c01b04d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:20.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.239 7ff1c51de700 1 -- 172.21.2.112:0/2864291938 >> v1:172.21.2.120:6789/0 conn(0x7ff1c01b0af0 legacy=0x7ff1c0108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:46:20.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.239 7ff1c51de700 1 -- 172.21.2.112:0/2864291938 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1c0088260 con 0x7ff1c00cff80 2021-02-10T11:46:20.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.239 7ff1c51de700 1 -- 172.21.2.112:0/2864291938 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7ff1b8001980 con 0x7ff1c00cff80 2021-02-10T11:46:20.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.239 7ff1c51de700 1 -- 172.21.2.112:0/2864291938 >> v1:172.21.2.112:6789/0 conn(0x7ff1c00cff80 legacy=0x7ff1c0109140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:20.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.239 7ff1c51de700 1 --2- 172.21.2.112:0/2864291938 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff1ac003aa0 0x7ff1ac005ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:20.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.239 7ff1c51de700 1 --2- 172.21.2.112:0/2864291938 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff1ac006770 0x7ff1ac008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:20.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.239 7ff1c51de700 1 --2- 172.21.2.112:0/2864291938 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff1ac009590 0x7ff1ac00bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:20.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.239 7ff1c51de700 1 -- 172.21.2.112:0/2864291938 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1ac00c300 con 0x7ff1ac006770 2021-02-10T11:46:20.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.239 7ff1c51de700 1 -- 172.21.2.112:0/2864291938 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1ac00c520 con 0x7ff1ac009590 2021-02-10T11:46:20.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.239 7ff1c51de700 1 -- 172.21.2.112:0/2864291938 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1ac00c740 con 0x7ff1ac003aa0 2021-02-10T11:46:20.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.239 7ff1c59df700 1 --2- 172.21.2.112:0/2864291938 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff1ac006770 0x7ff1ac008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:20.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.239 7ff1c69e1700 1 --2- 172.21.2.112:0/2864291938 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff1ac009590 0x7ff1ac00bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:20.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.239 7ff1c61e0700 1 --2- 172.21.2.112:0/2864291938 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff1ac003aa0 0x7ff1ac005ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:20.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.239 7ff1c59df700 1 -- 172.21.2.112:0/2864291938 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff1ac003aa0 msgr2=0x7ff1ac005ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:46:20.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.239 7ff1c59df700 1 --2- 172.21.2.112:0/2864291938 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff1ac003aa0 0x7ff1ac005ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:20.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.239 7ff1c59df700 1 -- 172.21.2.112:0/2864291938 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff1ac009590 msgr2=0x7ff1ac00bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:46:20.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.239 7ff1c59df700 1 --2- 172.21.2.112:0/2864291938 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff1ac009590 0x7ff1ac00bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:20.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.239 7ff1c59df700 1 -- 172.21.2.112:0/2864291938 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1ac00c9e0 con 0x7ff1ac006770 2021-02-10T11:46:20.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.239 7ff1c69e1700 1 -- 172.21.2.112:0/2864291938 reap_dead start 2021-02-10T11:46:20.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.240 7ff1c59df700 1 --2- 172.21.2.112:0/2864291938 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff1ac006770 0x7ff1ac008c90 secure :-1 s=READY pgs=2868 cs=0 l=1 rev1=1 rx=0x7ff1bc04aac0 tx=0x7ff1bc04daf0).ready entity=mon.0 client_cookie=6347a6cc5d12490c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:20.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.240 7ff1c51de700 1 -- 172.21.2.112:0/2864291938 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff1bc04a6b0 con 0x7ff1ac006770 2021-02-10T11:46:20.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.240 7ff1c51de700 1 -- 172.21.2.112:0/2864291938 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1bc000e20 con 0x7ff1ac006770 2021-02-10T11:46:20.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.240 7ff1c51de700 1 -- 172.21.2.112:0/2864291938 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff1bc04f560 con 0x7ff1ac006770 2021-02-10T11:46:20.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.240 7ff1c83c4700 1 -- 172.21.2.112:0/2864291938 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff1ac006770 msgr2=0x7ff1ac008c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:20.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.240 7ff1c83c4700 1 --2- 172.21.2.112:0/2864291938 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff1ac006770 0x7ff1ac008c90 secure :-1 s=READY pgs=2868 cs=0 l=1 rev1=1 rx=0x7ff1bc04aac0 tx=0x7ff1bc04daf0).stop 2021-02-10T11:46:20.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.240 7ff1c83c4700 1 -- 172.21.2.112:0/2864291938 shutdown_connections 2021-02-10T11:46:20.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.240 7ff1c83c4700 1 --2- 172.21.2.112:0/2864291938 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff1ac006770 0x7ff1ac008c90 unknown :-1 s=CLOSED pgs=2868 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:20.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.240 7ff1c83c4700 1 -- 172.21.2.112:0/2864291938 >> 172.21.2.112:0/2864291938 conn(0x7ff1c00ce430 msgr2=0x7ff1c00d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:20.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.240 7ff1c83c4700 1 -- 172.21.2.112:0/2864291938 shutdown_connections 2021-02-10T11:46:20.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.240 7ff1c83c4700 1 -- 172.21.2.112:0/2864291938 wait complete. 2021-02-10T11:46:20.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.255 7ff1c83c4700 1 Processor -- start 2021-02-10T11:46:20.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.255 7ff1c83c4700 1 -- start start 2021-02-10T11:46:20.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.255 7ff1c83c4700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff1c01a3570 0x7ff1c01a3970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:20.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.255 7ff1c83c4700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff1c009c9f0 0x7ff1c009cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:20.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.255 7ff1c83c4700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff1c009d410 0x7ff1c0098170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:20.262 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.255 7ff1c83c4700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1c009c470 con 0x7ff1c009d410 2021-02-10T11:46:20.262 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.255 7ff1c83c4700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1c00b2840 con 0x7ff1c01a3570 2021-02-10T11:46:20.262 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.255 7ff1c83c4700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1c00b2a20 con 0x7ff1c009c9f0 2021-02-10T11:46:20.262 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.255 7ff1c69e1700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff1c009d410 0x7ff1c0098170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:20.262 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.255 7ff1c69e1700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff1c009d410 0x7ff1c0098170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:59486/0 (socket says 172.21.2.112:59486) 2021-02-10T11:46:20.262 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.255 7ff1c69e1700 1 -- 172.21.2.112:0/824395096 learned_addr learned my addr 172.21.2.112:0/824395096 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:46:20.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.256 7ff1c69e1700 1 -- 172.21.2.112:0/824395096 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff1c009c9f0 msgr2=0x7ff1c009cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:20.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.256 7ff1c69e1700 1 --2- 172.21.2.112:0/824395096 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff1c009c9f0 0x7ff1c009cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:20.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.256 7ff1c69e1700 1 -- 172.21.2.112:0/824395096 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff1c01a3570 msgr2=0x7ff1c01a3970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:20.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.256 7ff1c69e1700 1 --2- 172.21.2.112:0/824395096 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff1c01a3570 0x7ff1c01a3970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:20.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.256 7ff1c69e1700 1 -- 172.21.2.112:0/824395096 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1c009d880 con 0x7ff1c009d410 2021-02-10T11:46:20.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.256 7ff1c69e1700 1 --2- 172.21.2.112:0/824395096 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff1c009d410 0x7ff1c0098170 secure :-1 s=READY pgs=2869 cs=0 l=1 rev1=1 rx=0x7ff1b800a900 tx=0x7ff1b800aca0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:20.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.256 7ff1b2ffd700 1 -- 172.21.2.112:0/824395096 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff1b80070d0 con 0x7ff1c009d410 2021-02-10T11:46:20.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.256 7ff1b2ffd700 1 -- 172.21.2.112:0/824395096 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1b80073d0 con 0x7ff1c009d410 2021-02-10T11:46:20.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.256 7ff1c83c4700 1 -- 172.21.2.112:0/824395096 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1c009d880 con 0x7ff1c009d410 2021-02-10T11:46:20.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.256 7ff1c83c4700 1 -- 172.21.2.112:0/824395096 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1c0098a80 con 0x7ff1c009d410 2021-02-10T11:46:20.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.256 7ff1b2ffd700 1 -- 172.21.2.112:0/824395096 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff1b8004b20 con 0x7ff1c009d410 2021-02-10T11:46:20.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.257 7ff1b2ffd700 1 -- 172.21.2.112:0/824395096 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff1b8001cb0 con 0x7ff1c009d410 2021-02-10T11:46:20.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.257 7ff1b2ffd700 1 --2- 172.21.2.112:0/824395096 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff1a80186c0 0x7ff1a801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:20.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.257 7ff1b2ffd700 1 -- 172.21.2.112:0/824395096 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 5457+0+0 (secure 0 0 0) 0x7ff1b80328f0 con 0x7ff1c009d410 2021-02-10T11:46:20.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.257 7ff1c61e0700 1 --2- 172.21.2.112:0/824395096 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff1a80186c0 0x7ff1a801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:20.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.258 7ff1c61e0700 1 --2- 172.21.2.112:0/824395096 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff1a80186c0 0x7ff1a801abc0 crc :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:20.290 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.288 7ff1c83c4700 1 -- 172.21.2.112:0/824395096 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1c0082a70 con 0x7ff1c009d410 2021-02-10T11:46:20.293 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.291 7ff1b2ffd700 1 -- 172.21.2.112:0/824395096 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff1b8001fb0 con 0x7ff1c009d410 2021-02-10T11:46:20.422 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.420 7ff1c83c4700 1 -- 172.21.2.112:0/824395096 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"} v 0) v1 -- 0x7ff1c0082a70 con 0x7ff1c009d410 2021-02-10T11:46:20.425 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.423 7ff1b2ffd700 1 -- 172.21.2.112:0/824395096 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"}]=0 new fs with metadata pool 36 and data pool 37 v145) v1 ==== 189+0+0 (secure 0 0 0) 0x7ff1b80071b0 con 0x7ff1c009d410 2021-02-10T11:46:20.430 INFO:teuthology.orchestra.run.gibba012.stderr:new fs with metadata pool 36 and data pool 37 2021-02-10T11:46:20.433 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.430 7ff1c83c4700 1 -- 172.21.2.112:0/824395096 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff1a80186c0 msgr2=0x7ff1a801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:20.433 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.430 7ff1c83c4700 1 --2- 172.21.2.112:0/824395096 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff1a80186c0 0x7ff1a801abc0 crc :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:20.433 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.430 7ff1c83c4700 1 -- 172.21.2.112:0/824395096 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff1c009d410 msgr2=0x7ff1c0098170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:20.433 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.430 7ff1c83c4700 1 --2- 172.21.2.112:0/824395096 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff1c009d410 0x7ff1c0098170 secure :-1 s=READY pgs=2869 cs=0 l=1 rev1=1 rx=0x7ff1b800a900 tx=0x7ff1b800aca0).stop 2021-02-10T11:46:20.433 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.430 7ff1c83c4700 1 -- 172.21.2.112:0/824395096 shutdown_connections 2021-02-10T11:46:20.433 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.430 7ff1c83c4700 1 --2- 172.21.2.112:0/824395096 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff1a80186c0 0x7ff1a801abc0 unknown :-1 s=CLOSED pgs=1303 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:20.433 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.430 7ff1c83c4700 1 --2- 172.21.2.112:0/824395096 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff1c009d410 0x7ff1c0098170 unknown :-1 s=CLOSED pgs=2869 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:20.433 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.430 7ff1c83c4700 1 --2- 172.21.2.112:0/824395096 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff1c009c9f0 0x7ff1c009cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:20.433 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.430 7ff1c83c4700 1 --2- 172.21.2.112:0/824395096 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff1c01a3570 0x7ff1c01a3970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:20.433 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.430 7ff1c83c4700 1 -- 172.21.2.112:0/824395096 >> 172.21.2.112:0/824395096 conn(0x7ff1c00c2cd0 msgr2=0x7ff1c00b5950 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:20.433 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.430 7ff1c83c4700 1 -- 172.21.2.112:0/824395096 shutdown_connections 2021-02-10T11:46:20.433 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.430 7ff1c83c4700 1 -- 172.21.2.112:0/824395096 wait complete. 2021-02-10T11:46:20.441 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:46:20.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.849 7f5a14287700 1 Processor -- start 2021-02-10T11:46:20.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.849 7f5a14287700 1 -- start start 2021-02-10T11:46:20.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.850 7f5a14287700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f5a0c0d1870 0x7f5a0c1b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:20.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.850 7f5a14287700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a0c0f0b10 con 0x7f5a0c1b0a10 2021-02-10T11:46:20.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.850 7f5a14287700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a0c0d0310 con 0x7f5a0c0f0650 2021-02-10T11:46:20.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.850 7f5a14287700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a0c0d0530 con 0x7f5a0c0d1870 2021-02-10T11:46:20.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.850 7f5a120a3700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f5a0c0d1870 0x7f5a0c1b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:20.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.850 7f5a118a2700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f5a0c1b0a10 0x7f5a0c0cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:58952/0 (socket says 172.21.2.112:58952) 2021-02-10T11:46:20.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.850 7f5a118a2700 1 -- 172.21.2.112:0/3035281750 learned_addr learned my addr 172.21.2.112:0/3035281750 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:46:20.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.850 7f5a120a3700 1 -- 172.21.2.112:0/3035281750 >> v1:172.21.2.120:6789/0 conn(0x7f5a0c0f0650 legacy=0x7f5a0c0cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:46:20.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.850 7f5a120a3700 1 -- 172.21.2.112:0/3035281750 >> v1:172.21.2.112:6789/0 conn(0x7f5a0c1b0a10 legacy=0x7f5a0c0cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:20.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.851 7f5a120a3700 1 -- 172.21.2.112:0/3035281750 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a0c0d1c70 con 0x7f5a0c0d1870 2021-02-10T11:46:20.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.851 7f5a120a3700 1 --2- 172.21.2.112:0/3035281750 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5a0c0d1870 0x7f5a0c1b03f0 secure :-1 s=READY pgs=2871 cs=0 l=1 rev1=1 rx=0x7f5a08006650 tx=0x7f5a08009a00).ready entity=mon.0 client_cookie=f86122cb80e6023c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:20.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.851 7f5a110a1700 1 -- 172.21.2.112:0/3035281750 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5a080009b0 con 0x7f5a0c0d1870 2021-02-10T11:46:20.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.851 7f5a110a1700 1 -- 172.21.2.112:0/3035281750 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a0800c0c0 con 0x7f5a0c0d1870 2021-02-10T11:46:20.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.851 7f5a110a1700 1 -- 172.21.2.112:0/3035281750 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5a0800c5a0 con 0x7f5a0c0d1870 2021-02-10T11:46:20.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.851 7f5a14287700 1 -- 172.21.2.112:0/3035281750 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5a0c0d1870 msgr2=0x7f5a0c1b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:20.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.851 7f5a14287700 1 --2- 172.21.2.112:0/3035281750 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5a0c0d1870 0x7f5a0c1b03f0 secure :-1 s=READY pgs=2871 cs=0 l=1 rev1=1 rx=0x7f5a08006650 tx=0x7f5a08009a00).stop 2021-02-10T11:46:20.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.852 7f5a14287700 1 -- 172.21.2.112:0/3035281750 shutdown_connections 2021-02-10T11:46:20.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.852 7f5a14287700 1 --2- 172.21.2.112:0/3035281750 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5a0c0d1870 0x7f5a0c1b03f0 unknown :-1 s=CLOSED pgs=2871 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:20.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.852 7f5a14287700 1 -- 172.21.2.112:0/3035281750 >> 172.21.2.112:0/3035281750 conn(0x7f5a0c0ce450 msgr2=0x7f5a0c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:20.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.852 7f5a14287700 1 -- 172.21.2.112:0/3035281750 shutdown_connections 2021-02-10T11:46:20.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.852 7f5a14287700 1 -- 172.21.2.112:0/3035281750 wait complete. 2021-02-10T11:46:20.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.866 7f5a14287700 1 Processor -- start 2021-02-10T11:46:20.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.866 7f5a14287700 1 -- start start 2021-02-10T11:46:20.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.867 7f5a14287700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5a0c094020 0x7f5a0c093730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:20.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.867 7f5a14287700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5a0c09ea70 0x7f5a0c09ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:20.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.867 7f5a14287700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5a0c1b0770 0x7f5a0c1b0b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:20.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.867 7f5a14287700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a0c09f4e0 con 0x7f5a0c1b0770 2021-02-10T11:46:20.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.867 7f5a14287700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a0c0ab6b0 con 0x7f5a0c094020 2021-02-10T11:46:20.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.867 7f5a14287700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a0c1b1190 con 0x7f5a0c09ea70 2021-02-10T11:46:20.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.867 7f5a128a4700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5a0c1b0770 0x7f5a0c1b0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:20.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.867 7f5a128a4700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5a0c1b0770 0x7f5a0c1b0b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:59496/0 (socket says 172.21.2.112:59496) 2021-02-10T11:46:20.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.867 7f5a128a4700 1 -- 172.21.2.112:0/178197743 learned_addr learned my addr 172.21.2.112:0/178197743 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:46:20.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.867 7f5a128a4700 1 -- 172.21.2.112:0/178197743 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5a0c09ea70 msgr2=0x7f5a0c09ee70 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:46:20.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.867 7f5a128a4700 1 --2- 172.21.2.112:0/178197743 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5a0c09ea70 0x7f5a0c09ee70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:20.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.867 7f5a128a4700 1 -- 172.21.2.112:0/178197743 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5a0c094020 msgr2=0x7f5a0c093730 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:46:20.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.867 7f5a128a4700 1 --2- 172.21.2.112:0/178197743 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5a0c094020 0x7f5a0c093730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:20.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.867 7f5a128a4700 1 -- 172.21.2.112:0/178197743 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a0c1b13b0 con 0x7f5a0c1b0770 2021-02-10T11:46:20.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.868 7f5a128a4700 1 --2- 172.21.2.112:0/178197743 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5a0c1b0770 0x7f5a0c1b0b70 secure :-1 s=READY pgs=2872 cs=0 l=1 rev1=1 rx=0x7f5a04009940 tx=0x7f5a0400bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:20.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.868 7f5a02ffd700 1 -- 172.21.2.112:0/178197743 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5a04000f90 con 0x7f5a0c1b0770 2021-02-10T11:46:20.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.868 7f5a02ffd700 1 -- 172.21.2.112:0/178197743 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a0400c2f0 con 0x7f5a0c1b0770 2021-02-10T11:46:20.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.868 7f5a14287700 1 -- 172.21.2.112:0/178197743 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a0c1b13b0 con 0x7f5a0c1b0770 2021-02-10T11:46:20.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.868 7f5a14287700 1 -- 172.21.2.112:0/178197743 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a0c1b13b0 con 0x7f5a0c1b0770 2021-02-10T11:46:20.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.868 7f5a02ffd700 1 -- 172.21.2.112:0/178197743 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5a0400c810 con 0x7f5a0c1b0770 2021-02-10T11:46:20.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.868 7f5a02ffd700 1 -- 172.21.2.112:0/178197743 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5a04001220 con 0x7f5a0c1b0770 2021-02-10T11:46:20.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.869 7f5a02ffd700 1 --2- 172.21.2.112:0/178197743 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f59f4018730 0x7f59f401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:20.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.869 7f5a02ffd700 1 -- 172.21.2.112:0/178197743 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f5a04032b80 con 0x7f5a0c1b0770 2021-02-10T11:46:20.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.869 7f5a120a3700 1 --2- 172.21.2.112:0/178197743 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f59f4018730 0x7f59f401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:20.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.869 7f5a120a3700 1 --2- 172.21.2.112:0/178197743 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f59f4018730 0x7f59f401ac10 crc :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:20.902 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.899 7f5a14287700 1 -- 172.21.2.112:0/178197743 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a0c0e7e20 con 0x7f5a0c1b0770 2021-02-10T11:46:20.904 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:20.902 7f5a02ffd700 1 -- 172.21.2.112:0/178197743 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5a0400be30 con 0x7f5a0c1b0770 2021-02-10T11:46:21.029 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:21.027 7f5a14287700 1 -- 172.21.2.112:0/178197743 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5a0c0d1ee0 con 0x7f5a0c1b0770 2021-02-10T11:46:21.030 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:21.027 7f5a02ffd700 1 -- 172.21.2.112:0/178197743 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v171) v1 ==== 74+0+14020 (secure 0 0 0) 0x7f5a0401f8c0 con 0x7f5a0c1b0770 2021-02-10T11:46:21.030 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:46:21.031 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":171,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:46:20.421786","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":37,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":36,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:46:18.395022","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"171","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":37,"pool_name":"cephfs_data","create_time":"2021-02-10 11:46:19.415292","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"171","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":168,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:46:21.032 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:21.029 7f5a14287700 1 -- 172.21.2.112:0/178197743 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f59f4018730 msgr2=0x7f59f401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:21.032 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:21.029 7f5a14287700 1 --2- 172.21.2.112:0/178197743 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f59f4018730 0x7f59f401ac10 crc :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:21.032 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:21.029 7f5a14287700 1 -- 172.21.2.112:0/178197743 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5a0c1b0770 msgr2=0x7f5a0c1b0b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:21.032 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:21.029 7f5a14287700 1 --2- 172.21.2.112:0/178197743 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5a0c1b0770 0x7f5a0c1b0b70 secure :-1 s=READY pgs=2872 cs=0 l=1 rev1=1 rx=0x7f5a04009940 tx=0x7f5a0400bcf0).stop 2021-02-10T11:46:21.032 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:21.029 7f5a14287700 1 -- 172.21.2.112:0/178197743 shutdown_connections 2021-02-10T11:46:21.032 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:21.029 7f5a14287700 1 --2- 172.21.2.112:0/178197743 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f59f4018730 0x7f59f401ac10 unknown :-1 s=CLOSED pgs=1304 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:21.032 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:21.029 7f5a14287700 1 --2- 172.21.2.112:0/178197743 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5a0c1b0770 0x7f5a0c1b0b70 unknown :-1 s=CLOSED pgs=2872 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:21.032 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:21.029 7f5a14287700 1 --2- 172.21.2.112:0/178197743 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5a0c09ea70 0x7f5a0c09ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:21.032 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:21.029 7f5a14287700 1 --2- 172.21.2.112:0/178197743 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5a0c094020 0x7f5a0c093730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:21.032 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:21.029 7f5a14287700 1 -- 172.21.2.112:0/178197743 >> 172.21.2.112:0/178197743 conn(0x7f5a0c0a0030 msgr2=0x7f5a0c0ea250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:21.032 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:21.030 7f5a14287700 1 -- 172.21.2.112:0/178197743 shutdown_connections 2021-02-10T11:46:21.033 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:21.030 7f5a14287700 1 -- 172.21.2.112:0/178197743 wait complete. 2021-02-10T11:46:21.041 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:46:21.470 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:21.451 7fc242056700 1 Processor -- start 2021-02-10T11:46:21.470 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:21.452 7fc242056700 1 -- start start 2021-02-10T11:46:21.470 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:21.452 7fc242056700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fc23c0d1c70 0x7fc23c0f20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:21.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:21.452 7fc242056700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc23c0f0b10 con 0x7fc23c0d1870 2021-02-10T11:46:21.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:21.452 7fc242056700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc23c1b0be0 con 0x7fc23c0d0340 2021-02-10T11:46:21.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:21.452 7fc242056700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc23c0e5ed0 con 0x7fc23c0d1c70 2021-02-10T11:46:21.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:21.452 7fc23bfff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fc23c0d1c70 0x7fc23c0f20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:21.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:21.452 7fc23b7fe700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fc23c0d1870 0x7fc23c1b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:58964/0 (socket says 172.21.2.112:58964) 2021-02-10T11:46:21.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:21.452 7fc23b7fe700 1 -- 172.21.2.112:0/1785354832 learned_addr learned my addr 172.21.2.112:0/1785354832 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:46:21.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:21.452 7fc23bfff700 1 -- 172.21.2.112:0/1785354832 >> v1:172.21.2.120:6789/0 conn(0x7fc23c0d0340 legacy=0x7fc23c0ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:21.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:21.452 7fc23bfff700 1 -- 172.21.2.112:0/1785354832 >> v1:172.21.2.112:6789/0 conn(0x7fc23c0d1870 legacy=0x7fc23c1b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:21.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:21.453 7fc23bfff700 1 -- 172.21.2.112:0/1785354832 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc23c0efe30 con 0x7fc23c0d1c70 2021-02-10T11:46:21.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:21.453 7fc23bfff700 1 --2- 172.21.2.112:0/1785354832 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc23c0d1c70 0x7fc23c0f20a0 secure :-1 s=READY pgs=2874 cs=0 l=1 rev1=1 rx=0x7fc22c006ab0 tx=0x7fc22c009b40).ready entity=mon.0 client_cookie=2562541b9f770c18 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:21.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:21.453 7fc23a7fc700 1 -- 172.21.2.112:0/1785354832 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc22c0022e0 con 0x7fc23c0d1c70 2021-02-10T11:46:21.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:21.453 7fc23a7fc700 1 -- 172.21.2.112:0/1785354832 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc22c002560 con 0x7fc23c0d1c70 2021-02-10T11:46:21.471 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:21.453 7fc23a7fc700 1 -- 172.21.2.112:0/1785354832 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc22c0029f0 con 0x7fc23c0d1c70 2021-02-10T11:46:21.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:21.453 7fc242056700 1 -- 172.21.2.112:0/1785354832 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc23c0d1c70 msgr2=0x7fc23c0f20a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:21.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:21.453 7fc242056700 1 --2- 172.21.2.112:0/1785354832 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc23c0d1c70 0x7fc23c0f20a0 secure :-1 s=READY pgs=2874 cs=0 l=1 rev1=1 rx=0x7fc22c006ab0 tx=0x7fc22c009b40).stop 2021-02-10T11:46:21.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:21.453 7fc242056700 1 -- 172.21.2.112:0/1785354832 shutdown_connections 2021-02-10T11:46:21.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:21.453 7fc242056700 1 --2- 172.21.2.112:0/1785354832 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc23c0d1c70 0x7fc23c0f20a0 unknown :-1 s=CLOSED pgs=2874 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:21.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:21.453 7fc242056700 1 -- 172.21.2.112:0/1785354832 >> 172.21.2.112:0/1785354832 conn(0x7fc23c0ce450 msgr2=0x7fc23c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:21.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:21.453 7fc242056700 1 -- 172.21.2.112:0/1785354832 shutdown_connections 2021-02-10T11:46:21.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:21.453 7fc242056700 1 -- 172.21.2.112:0/1785354832 wait complete. 2021-02-10T11:46:21.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:21.468 7fc242056700 1 Processor -- start 2021-02-10T11:46:21.472 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:21.468 7fc242056700 1 -- start start 2021-02-10T11:46:21.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:21.468 7fc242056700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc23c199960 0x7fc23c0bb6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:21.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:21.468 7fc242056700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc23c0bbce0 0x7fc23c0bc0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:21.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:21.469 7fc242056700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc23c09ca90 0x7fc23c09ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:21.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:21.469 7fc242056700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc23c09c590 con 0x7fc23c0bbce0 2021-02-10T11:46:21.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:21.469 7fc242056700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc23c09d4b0 con 0x7fc23c09ca90 2021-02-10T11:46:21.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:21.469 7fc242056700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc23c09d6d0 con 0x7fc23c199960 2021-02-10T11:46:21.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:21.469 7fc23affd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc23c0bbce0 0x7fc23c0bc0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:21.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:21.469 7fc23bfff700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc23c09ca90 0x7fc23c09ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:21.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:21.469 7fc23affd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc23c0bbce0 0x7fc23c0bc0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:59506/0 (socket says 172.21.2.112:59506) 2021-02-10T11:46:21.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:21.469 7fc23affd700 1 -- 172.21.2.112:0/1142863933 learned_addr learned my addr 172.21.2.112:0/1142863933 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:46:21.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:21.469 7fc23b7fe700 1 --2- 172.21.2.112:0/1142863933 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc23c199960 0x7fc23c0bb6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:21.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:21.469 7fc23bfff700 1 -- 172.21.2.112:0/1142863933 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc23c199960 msgr2=0x7fc23c0bb6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:21.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:21.469 7fc23bfff700 1 --2- 172.21.2.112:0/1142863933 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc23c199960 0x7fc23c0bb6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:21.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:21.469 7fc23bfff700 1 -- 172.21.2.112:0/1142863933 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc23c0bbce0 msgr2=0x7fc23c0bc0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:21.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:21.469 7fc23bfff700 1 --2- 172.21.2.112:0/1142863933 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc23c0bbce0 0x7fc23c0bc0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:21.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:21.469 7fc23bfff700 1 -- 172.21.2.112:0/1142863933 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc23c09d8f0 con 0x7fc23c09ca90 2021-02-10T11:46:21.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:21.469 7fc23bfff700 1 --2- 172.21.2.112:0/1142863933 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc23c09ca90 0x7fc23c09ce90 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 rx=0x7fc22c009690 tx=0x7fc22c00b930).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:21.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:21.469 7fc223fff700 1 -- 172.21.2.112:0/1142863933 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc22c002030 con 0x7fc23c09ca90 2021-02-10T11:46:21.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:21.469 7fc223fff700 1 -- 172.21.2.112:0/1142863933 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc22c0022f0 con 0x7fc23c09ca90 2021-02-10T11:46:21.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:21.469 7fc242056700 1 -- 172.21.2.112:0/1142863933 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc23c09d8f0 con 0x7fc23c09ca90 2021-02-10T11:46:21.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:21.469 7fc223fff700 1 -- 172.21.2.112:0/1142863933 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc22c0027d0 con 0x7fc23c09ca90 2021-02-10T11:46:21.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:21.470 7fc242056700 1 -- 172.21.2.112:0/1142863933 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc23c09d8f0 con 0x7fc23c09ca90 2021-02-10T11:46:21.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:21.470 7fc223fff700 1 -- 172.21.2.112:0/1142863933 <== mon.1 v2:172.21.2.120:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc22c0024d0 con 0x7fc23c09ca90 2021-02-10T11:46:21.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:21.470 7fc223fff700 1 --2- 172.21.2.112:0/1142863933 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc21c0186c0 0x7fc21c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:21.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:21.470 7fc223fff700 1 -- 172.21.2.112:0/1142863933 <== mon.1 v2:172.21.2.120:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fc22c0326d0 con 0x7fc23c09ca90 2021-02-10T11:46:21.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:21.470 7fc23b7fe700 1 --2- 172.21.2.112:0/1142863933 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc21c0186c0 0x7fc21c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:21.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:21.471 7fc23b7fe700 1 --2- 172.21.2.112:0/1142863933 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc21c0186c0 0x7fc21c01abc0 crc :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:21.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:21.502 7fc242056700 1 -- 172.21.2.112:0/1142863933 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc23c0e7df0 con 0x7fc23c09ca90 2021-02-10T11:46:21.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:21.504 7fc223fff700 1 -- 172.21.2.112:0/1142863933 <== mon.1 v2:172.21.2.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc22c002500 con 0x7fc23c09ca90 2021-02-10T11:46:21.632 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:21.630 7fc242056700 1 -- 172.21.2.112:0/1142863933 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc23c0e7df0 con 0x7fc23c09ca90 2021-02-10T11:46:21.633 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:21.631 7fc223fff700 1 -- 172.21.2.112:0/1142863933 <== mon.1 v2:172.21.2.120:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v172) v1 ==== 74+0+14007 (secure 0 0 0) 0x7fc22c0022d0 con 0x7fc23c09ca90 2021-02-10T11:46:21.635 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:46:21.635 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":172,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:46:21.423567","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":37,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":36,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:46:18.395022","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"171","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":37,"pool_name":"cephfs_data","create_time":"2021-02-10 11:46:19.415292","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":168,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:46:21.636 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:21.634 7fc242056700 1 -- 172.21.2.112:0/1142863933 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc21c0186c0 msgr2=0x7fc21c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:21.637 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:21.634 7fc242056700 1 --2- 172.21.2.112:0/1142863933 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc21c0186c0 0x7fc21c01abc0 crc :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:21.637 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:21.634 7fc242056700 1 -- 172.21.2.112:0/1142863933 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc23c09ca90 msgr2=0x7fc23c09ce90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:21.637 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:21.634 7fc242056700 1 --2- 172.21.2.112:0/1142863933 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc23c09ca90 0x7fc23c09ce90 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 rx=0x7fc22c009690 tx=0x7fc22c00b930).stop 2021-02-10T11:46:21.637 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:21.634 7fc242056700 1 -- 172.21.2.112:0/1142863933 shutdown_connections 2021-02-10T11:46:21.637 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:21.634 7fc242056700 1 --2- 172.21.2.112:0/1142863933 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc21c0186c0 0x7fc21c01abc0 unknown :-1 s=CLOSED pgs=1305 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:21.637 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:21.634 7fc242056700 1 --2- 172.21.2.112:0/1142863933 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc23c09ca90 0x7fc23c09ce90 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:21.637 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:21.634 7fc242056700 1 --2- 172.21.2.112:0/1142863933 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc23c0bbce0 0x7fc23c0bc0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:21.637 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:21.634 7fc242056700 1 --2- 172.21.2.112:0/1142863933 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc23c199960 0x7fc23c0bb6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:21.637 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:21.634 7fc242056700 1 -- 172.21.2.112:0/1142863933 >> 172.21.2.112:0/1142863933 conn(0x7fc23c19d610 msgr2=0x7fc23c0f22b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:21.637 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:21.634 7fc242056700 1 -- 172.21.2.112:0/1142863933 shutdown_connections 2021-02-10T11:46:21.637 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:21.634 7fc242056700 1 -- 172.21.2.112:0/1142863933 wait complete. 2021-02-10T11:46:21.645 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2021-02-10T11:46:22.080 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.060 7f7074bd4700 1 Processor -- start 2021-02-10T11:46:22.081 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.060 7f7074bd4700 1 -- start start 2021-02-10T11:46:22.081 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.060 7f7074bd4700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f70700d1c70 0x7f70700cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:22.081 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.060 7f7074bd4700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f70700cf580 0x7f70700cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:22.081 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.060 7f7074bd4700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70700fe1c0 con 0x7f70700d1870 2021-02-10T11:46:22.081 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.060 7f7074bd4700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70701b0cb0 con 0x7f70700cf580 2021-02-10T11:46:22.081 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.060 7f7074bd4700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7070109a60 con 0x7f70700d1c70 2021-02-10T11:46:22.081 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.062 7f706f05f700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f70700cf580 0x7f70700cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:22.081 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.062 7f706e05d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f70700d1c70 0x7f70700cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:22.081 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.062 7f706e85e700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7f70700d1870 0x7f70701b04c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:52534/0 (socket says 172.21.2.112:52534) 2021-02-10T11:46:22.081 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.062 7f706e85e700 1 -- 172.21.2.112:0/3293751273 learned_addr learned my addr 172.21.2.112:0/3293751273 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:46:22.081 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.062 7f706e05d700 1 -- 172.21.2.112:0/3293751273 >> v2:172.21.2.120:3300/0 conn(0x7f70700cf580 msgr2=0x7f70700cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:46:22.082 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.062 7f706e05d700 1 --2- 172.21.2.112:0/3293751273 >> v2:172.21.2.120:3300/0 conn(0x7f70700cf580 0x7f70700cf980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:22.082 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.062 7f706e05d700 1 -- 172.21.2.112:0/3293751273 >> v1:172.21.2.120:6789/0 conn(0x7f70700d1870 legacy=0x7f70701b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:22.082 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.062 7f706e05d700 1 -- 172.21.2.112:0/3293751273 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7070088070 con 0x7f70700d1c70 2021-02-10T11:46:22.082 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.062 7f706f05f700 1 --2- 172.21.2.112:0/3293751273 >> v2:172.21.2.120:3300/0 conn(0x7f70700cf580 0x7f70700cf980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:46:22.082 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.062 7f706e05d700 1 --2- 172.21.2.112:0/3293751273 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f70700d1c70 0x7f70700cef60 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7f7060006650 tx=0x7f7060009a80).ready entity=mon.2 client_cookie=815d0cb102668e67 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:22.082 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.063 7f706d85c700 1 -- 172.21.2.112:0/3293751273 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f70600009b0 con 0x7f70700d1c70 2021-02-10T11:46:22.082 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.063 7f706d85c700 1 -- 172.21.2.112:0/3293751273 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f706000c0c0 con 0x7f70700d1c70 2021-02-10T11:46:22.082 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.063 7f706d85c700 1 -- 172.21.2.112:0/3293751273 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f706000c5e0 con 0x7f70700d1c70 2021-02-10T11:46:22.082 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.063 7f7074bd4700 1 -- 172.21.2.112:0/3293751273 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f70700d1c70 msgr2=0x7f70700cef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:22.082 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.063 7f7074bd4700 1 --2- 172.21.2.112:0/3293751273 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f70700d1c70 0x7f70700cef60 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7f7060006650 tx=0x7f7060009a80).stop 2021-02-10T11:46:22.083 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.064 7f7074bd4700 1 -- 172.21.2.112:0/3293751273 shutdown_connections 2021-02-10T11:46:22.083 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.064 7f7074bd4700 1 --2- 172.21.2.112:0/3293751273 >> v2:172.21.2.120:3300/0 conn(0x7f70700cf580 0x7f70700cf980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:22.083 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.064 7f7074bd4700 1 --2- 172.21.2.112:0/3293751273 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f70700d1c70 0x7f70700cef60 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:22.083 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.064 7f7074bd4700 1 -- 172.21.2.112:0/3293751273 >> 172.21.2.112:0/3293751273 conn(0x7f70700ce420 msgr2=0x7f70700d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:22.083 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.064 7f7074bd4700 1 -- 172.21.2.112:0/3293751273 shutdown_connections 2021-02-10T11:46:22.083 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.064 7f7074bd4700 1 -- 172.21.2.112:0/3293751273 wait complete. 2021-02-10T11:46:22.085 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.079 7f7074bd4700 1 Processor -- start 2021-02-10T11:46:22.085 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.079 7f7074bd4700 1 -- start start 2021-02-10T11:46:22.085 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.079 7f7074bd4700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7070191b10 0x7f70700ae640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:22.085 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.079 7f7074bd4700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7070191f10 0x7f707019ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:22.085 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.079 7f7074bd4700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f70701a05f0 0x7f7070082ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:22.085 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.079 7f7074bd4700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7070192440 con 0x7f7070191b10 2021-02-10T11:46:22.085 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.079 7f7074bd4700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70701916e0 con 0x7f7070191f10 2021-02-10T11:46:22.085 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.079 7f7074bd4700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7070191900 con 0x7f70701a05f0 2021-02-10T11:46:22.085 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.079 7f706e05d700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7070191f10 0x7f707019ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:22.085 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.079 7f706e85e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7070191b10 0x7f70700ae640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:22.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.079 7f706e05d700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7070191f10 0x7f707019ffd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3300/0 says I am v2:172.21.2.112:33568/0 (socket says 172.21.2.112:33568) 2021-02-10T11:46:22.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.079 7f706e05d700 1 -- 172.21.2.112:0/1237441482 learned_addr learned my addr 172.21.2.112:0/1237441482 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:46:22.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.079 7f706e85e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7070191b10 0x7f70700ae640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:59522/0 (socket says 172.21.2.112:59522) 2021-02-10T11:46:22.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.080 7f706f05f700 1 --2- 172.21.2.112:0/1237441482 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f70701a05f0 0x7f7070082ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:22.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.080 7f706e05d700 1 -- 172.21.2.112:0/1237441482 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f70701a05f0 msgr2=0x7f7070082ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:22.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.080 7f706e05d700 1 --2- 172.21.2.112:0/1237441482 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f70701a05f0 0x7f7070082ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:22.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.080 7f706e05d700 1 -- 172.21.2.112:0/1237441482 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7070191b10 msgr2=0x7f70700ae640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:22.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.080 7f706e05d700 1 --2- 172.21.2.112:0/1237441482 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7070191b10 0x7f70700ae640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:22.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.080 7f706e05d700 1 -- 172.21.2.112:0/1237441482 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70700834f0 con 0x7f7070191f10 2021-02-10T11:46:22.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.080 7f706f05f700 1 --2- 172.21.2.112:0/1237441482 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f70701a05f0 0x7f7070082ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:46:22.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.080 7f706e05d700 1 --2- 172.21.2.112:0/1237441482 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7070191f10 0x7f707019ffd0 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 rx=0x7f7060009630 tx=0x7f706000b970).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:22.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.081 7f70677fe700 1 -- 172.21.2.112:0/1237441482 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f706000bd60 con 0x7f7070191f10 2021-02-10T11:46:22.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.081 7f70677fe700 1 -- 172.21.2.112:0/1237441482 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7060006740 con 0x7f7070191f10 2021-02-10T11:46:22.087 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.081 7f70677fe700 1 -- 172.21.2.112:0/1237441482 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7060002270 con 0x7f7070191f10 2021-02-10T11:46:22.087 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.081 7f7074bd4700 1 -- 172.21.2.112:0/1237441482 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70700834f0 con 0x7f7070191f10 2021-02-10T11:46:22.087 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.081 7f7074bd4700 1 -- 172.21.2.112:0/1237441482 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7070083a80 con 0x7f7070191f10 2021-02-10T11:46:22.087 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.081 7f70677fe700 1 -- 172.21.2.112:0/1237441482 <== mon.1 v2:172.21.2.120:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7060000c90 con 0x7f7070191f10 2021-02-10T11:46:22.087 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.081 7f70677fe700 1 --2- 172.21.2.112:0/1237441482 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f70540186c0 0x7f705401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:22.087 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.081 7f70677fe700 1 -- 172.21.2.112:0/1237441482 <== mon.1 v2:172.21.2.120:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f7060032570 con 0x7f7070191f10 2021-02-10T11:46:22.087 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.082 7f706e85e700 1 --2- 172.21.2.112:0/1237441482 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f70540186c0 0x7f705401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:22.087 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.083 7f706e85e700 1 --2- 172.21.2.112:0/1237441482 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f70540186c0 0x7f705401abc0 crc :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:22.113 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.111 7f7074bd4700 1 -- 172.21.2.112:0/1237441482 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7070191900 con 0x7f7070191f10 2021-02-10T11:46:22.116 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.114 7f70677fe700 1 -- 172.21.2.112:0/1237441482 <== mon.1 v2:172.21.2.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7060015730 con 0x7f7070191f10 2021-02-10T11:46:22.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.254 7f7074bd4700 1 -- 172.21.2.112:0/1237441482 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_command({"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"} v 0) v1 -- 0x7f7070191900 con 0x7f7070191f10 2021-02-10T11:46:22.434 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.431 7f70677fe700 1 -- 172.21.2.112:0/1237441482 <== mon.1 v2:172.21.2.120:3300/0 7 ==== mon_command_ack([{"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"}]=0 v148) v1 ==== 118+0+0 (secure 0 0 0) 0x7f7060015730 con 0x7f7070191f10 2021-02-10T11:46:22.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.471 7f7074bd4700 1 -- 172.21.2.112:0/1237441482 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f70540186c0 msgr2=0x7f705401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:22.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.471 7f7074bd4700 1 --2- 172.21.2.112:0/1237441482 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f70540186c0 0x7f705401abc0 crc :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:22.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.471 7f7074bd4700 1 -- 172.21.2.112:0/1237441482 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7070191f10 msgr2=0x7f707019ffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:22.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.471 7f7074bd4700 1 --2- 172.21.2.112:0/1237441482 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7070191f10 0x7f707019ffd0 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 rx=0x7f7060009630 tx=0x7f706000b970).stop 2021-02-10T11:46:22.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.471 7f7074bd4700 1 -- 172.21.2.112:0/1237441482 shutdown_connections 2021-02-10T11:46:22.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.471 7f7074bd4700 1 --2- 172.21.2.112:0/1237441482 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f70540186c0 0x7f705401abc0 unknown :-1 s=CLOSED pgs=1306 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:22.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.471 7f7074bd4700 1 --2- 172.21.2.112:0/1237441482 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7070191f10 0x7f707019ffd0 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:22.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.471 7f7074bd4700 1 --2- 172.21.2.112:0/1237441482 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f70701a05f0 0x7f7070082ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:22.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.471 7f7074bd4700 1 --2- 172.21.2.112:0/1237441482 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7070191b10 0x7f70700ae640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:22.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.471 7f7074bd4700 1 -- 172.21.2.112:0/1237441482 >> 172.21.2.112:0/1237441482 conn(0x7f70700a02d0 msgr2=0x7f707009f200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:22.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.472 7f7074bd4700 1 -- 172.21.2.112:0/1237441482 shutdown_connections 2021-02-10T11:46:22.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.472 7f7074bd4700 1 -- 172.21.2.112:0/1237441482 wait complete. 2021-02-10T11:46:22.482 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:46:22.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.892 7f1307935700 1 Processor -- start 2021-02-10T11:46:22.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.892 7f1307935700 1 -- start start 2021-02-10T11:46:22.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.893 7f1307935700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f13001b0470 0x7f13001b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:22.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.893 7f1307935700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13000f0b50 con 0x7f13000f20e0 2021-02-10T11:46:22.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.893 7f1307935700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13000f24e0 con 0x7f13000ef610 2021-02-10T11:46:22.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.893 7f1307935700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13000e5ea0 con 0x7f13001b0470 2021-02-10T11:46:22.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.893 7f1305751700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f13001b0470 0x7f13001b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:22.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.893 7f1305751700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f13001b0470 0x7f13001b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:59528/0 (socket says 172.21.2.112:59528) 2021-02-10T11:46:22.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.893 7f1305751700 1 -- 172.21.2.112:0/3984030223 learned_addr learned my addr 172.21.2.112:0/3984030223 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:46:22.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.893 7f1305751700 1 -- 172.21.2.112:0/3984030223 >> v1:172.21.2.120:6789/0 conn(0x7f13000ef610 legacy=0x7f13000efa10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:46:22.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.893 7f1305751700 1 -- 172.21.2.112:0/3984030223 >> v1:172.21.2.112:6789/0 conn(0x7f13000f20e0 legacy=0x7f13000f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:22.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.893 7f1305751700 1 -- 172.21.2.112:0/3984030223 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13000e52e0 con 0x7f13001b0470 2021-02-10T11:46:22.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.894 7f1305751700 1 --2- 172.21.2.112:0/3984030223 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f13001b0470 0x7f13001b0870 secure :-1 s=READY pgs=2876 cs=0 l=1 rev1=1 rx=0x7f12f4006700 tx=0x7f12f4009b80).ready entity=mon.0 client_cookie=8b6bf31f8a9fc029 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:22.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.894 7f12f3fff700 1 -- 172.21.2.112:0/3984030223 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f12f400c0c0 con 0x7f13001b0470 2021-02-10T11:46:22.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.894 7f12f3fff700 1 -- 172.21.2.112:0/3984030223 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12f400c340 con 0x7f13001b0470 2021-02-10T11:46:22.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.894 7f12f3fff700 1 -- 172.21.2.112:0/3984030223 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f12f400c860 con 0x7f13001b0470 2021-02-10T11:46:22.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.894 7f1307935700 1 -- 172.21.2.112:0/3984030223 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f13001b0470 msgr2=0x7f13001b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:22.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.894 7f1307935700 1 --2- 172.21.2.112:0/3984030223 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f13001b0470 0x7f13001b0870 secure :-1 s=READY pgs=2876 cs=0 l=1 rev1=1 rx=0x7f12f4006700 tx=0x7f12f4009b80).stop 2021-02-10T11:46:22.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.894 7f1307935700 1 -- 172.21.2.112:0/3984030223 shutdown_connections 2021-02-10T11:46:22.913 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.894 7f1307935700 1 --2- 172.21.2.112:0/3984030223 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f13001b0470 0x7f13001b0870 unknown :-1 s=CLOSED pgs=2876 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:22.913 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.894 7f1307935700 1 -- 172.21.2.112:0/3984030223 >> 172.21.2.112:0/3984030223 conn(0x7f13000ce3e0 msgr2=0x7f13000e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:22.913 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.895 7f1307935700 1 -- 172.21.2.112:0/3984030223 shutdown_connections 2021-02-10T11:46:22.913 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.895 7f1307935700 1 -- 172.21.2.112:0/3984030223 wait complete. 2021-02-10T11:46:22.914 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.909 7f1307935700 1 Processor -- start 2021-02-10T11:46:22.914 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.909 7f1307935700 1 -- start start 2021-02-10T11:46:22.914 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.910 7f1307935700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1300092e30 0x7f13001c6320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:22.914 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.910 7f1307935700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f13000964a0 0x7f13000968a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:22.914 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.910 7f1307935700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1300096ec0 0x7f1300084f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:22.914 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.910 7f1307935700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13000932b0 con 0x7f1300092e30 2021-02-10T11:46:22.914 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.910 7f1307935700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13001b5f70 con 0x7f13000964a0 2021-02-10T11:46:22.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.910 7f1307935700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13001b6150 con 0x7f1300096ec0 2021-02-10T11:46:22.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.910 7f1305751700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1300092e30 0x7f13001c6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:22.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.910 7f1305751700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1300092e30 0x7f13001c6320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:59530/0 (socket says 172.21.2.112:59530) 2021-02-10T11:46:22.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.910 7f1305751700 1 -- 172.21.2.112:0/4236999281 learned_addr learned my addr 172.21.2.112:0/4236999281 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:46:22.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.910 7f1305751700 1 -- 172.21.2.112:0/4236999281 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1300096ec0 msgr2=0x7f1300084f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:22.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.910 7f1305751700 1 --2- 172.21.2.112:0/4236999281 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1300096ec0 0x7f1300084f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:22.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.910 7f1305751700 1 -- 172.21.2.112:0/4236999281 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f13000964a0 msgr2=0x7f13000968a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:22.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.910 7f1305751700 1 --2- 172.21.2.112:0/4236999281 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f13000964a0 0x7f13000968a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:22.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.910 7f1305751700 1 -- 172.21.2.112:0/4236999281 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13001c6970 con 0x7f1300092e30 2021-02-10T11:46:22.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.910 7f1305751700 1 --2- 172.21.2.112:0/4236999281 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1300092e30 0x7f13001c6320 secure :-1 s=READY pgs=2877 cs=0 l=1 rev1=1 rx=0x7f12f400b5e0 tx=0x7f12f400b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:22.915 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.911 7f12f1ffb700 1 -- 172.21.2.112:0/4236999281 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f12f400bd70 con 0x7f1300092e30 2021-02-10T11:46:22.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.911 7f1307935700 1 -- 172.21.2.112:0/4236999281 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13001c6970 con 0x7f1300092e30 2021-02-10T11:46:22.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.911 7f12f1ffb700 1 -- 172.21.2.112:0/4236999281 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12f400bd70 con 0x7f1300092e30 2021-02-10T11:46:22.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.911 7f12f1ffb700 1 -- 172.21.2.112:0/4236999281 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f12f4002030 con 0x7f1300092e30 2021-02-10T11:46:22.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.911 7f1307935700 1 -- 172.21.2.112:0/4236999281 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1300085580 con 0x7f1300092e30 2021-02-10T11:46:22.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.911 7f12f1ffb700 1 -- 172.21.2.112:0/4236999281 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f12f4000c90 con 0x7f1300092e30 2021-02-10T11:46:22.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.911 7f12f1ffb700 1 --2- 172.21.2.112:0/4236999281 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f12e00186c0 0x7f12e001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:22.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.911 7f12f1ffb700 1 -- 172.21.2.112:0/4236999281 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f12f4032450 con 0x7f1300092e30 2021-02-10T11:46:22.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.912 7f1304f50700 1 --2- 172.21.2.112:0/4236999281 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f12e00186c0 0x7f12e001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:22.917 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.912 7f1304f50700 1 --2- 172.21.2.112:0/4236999281 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f12e00186c0 0x7f12e001abc0 crc :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:22.945 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.943 7f1307935700 1 -- 172.21.2.112:0/4236999281 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f130007ed50 con 0x7f1300092e30 2021-02-10T11:46:22.947 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:22.945 7f12f1ffb700 1 -- 172.21.2.112:0/4236999281 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f12f40009b0 con 0x7f1300092e30 2021-02-10T11:46:23.075 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:23.073 7f1307935700 1 -- 172.21.2.112:0/4236999281 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f13000ce3e0 con 0x7f1300092e30 2021-02-10T11:46:23.076 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:23.074 7f12f1ffb700 1 -- 172.21.2.112:0/4236999281 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 148 v148) v1 ==== 95+0+2111 (secure 0 0 0) 0x7f12f4000c90 con 0x7f1300092e30 2021-02-10T11:46:23.076 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:46:23.076 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":148,"default_fscid":18,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":12642,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/4212630012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":4212630012},{"type":"v1","addr":"172.21.2.112:6833","nonce":4212630012}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":144}],"filesystems":[{"mdsmap":{"epoch":148,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:46:20.421874","modified":"2021-02-10 11:46:22.427977","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12639},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12639":{"gid":12639,"name":"b","rank":0,"incarnation":145,"state":"up:active","state_seq":3,"addr":"172.21.2.120:6835/1654642147","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":1654642147},{"type":"v1","addr":"172.21.2.120:6835","nonce":1654642147}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[37],"metadata_pool":36,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":18}]} 2021-02-10T11:46:23.077 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 148 2021-02-10T11:46:23.077 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:23.075 7f1307935700 1 -- 172.21.2.112:0/4236999281 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f12e00186c0 msgr2=0x7f12e001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:23.077 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:23.075 7f1307935700 1 --2- 172.21.2.112:0/4236999281 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f12e00186c0 0x7f12e001abc0 crc :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:23.077 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:23.075 7f1307935700 1 -- 172.21.2.112:0/4236999281 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1300092e30 msgr2=0x7f13001c6320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:23.077 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:23.075 7f1307935700 1 --2- 172.21.2.112:0/4236999281 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1300092e30 0x7f13001c6320 secure :-1 s=READY pgs=2877 cs=0 l=1 rev1=1 rx=0x7f12f400b5e0 tx=0x7f12f400b980).stop 2021-02-10T11:46:23.078 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:23.075 7f1307935700 1 -- 172.21.2.112:0/4236999281 shutdown_connections 2021-02-10T11:46:23.078 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:23.075 7f1307935700 1 --2- 172.21.2.112:0/4236999281 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f12e00186c0 0x7f12e001abc0 unknown :-1 s=CLOSED pgs=1307 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:23.078 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:23.075 7f1307935700 1 --2- 172.21.2.112:0/4236999281 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f13000964a0 0x7f13000968a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:23.078 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:23.076 7f1307935700 1 --2- 172.21.2.112:0/4236999281 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1300096ec0 0x7f1300084f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:23.078 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:23.076 7f1307935700 1 --2- 172.21.2.112:0/4236999281 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1300092e30 0x7f13001c6320 unknown :-1 s=CLOSED pgs=2877 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:23.078 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:23.076 7f1307935700 1 -- 172.21.2.112:0/4236999281 >> 172.21.2.112:0/4236999281 conn(0x7f13000c3bd0 msgr2=0x7f1300089210 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:23.078 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:23.076 7f1307935700 1 -- 172.21.2.112:0/4236999281 shutdown_connections 2021-02-10T11:46:23.079 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:23.076 7f1307935700 1 -- 172.21.2.112:0/4236999281 wait complete. 2021-02-10T11:46:23.086 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:46:23.516 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:23.497 7f2a7196a700 1 Processor -- start 2021-02-10T11:46:23.516 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:23.497 7f2a7196a700 1 -- start start 2021-02-10T11:46:23.516 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:23.498 7f2a7196a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2a6c0d1870 0x7f2a6c1b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:23.517 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:23.498 7f2a7196a700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a6c0f0b10 con 0x7f2a6c0f0650 2021-02-10T11:46:23.517 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:23.498 7f2a7196a700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a6c0d0310 con 0x7f2a6c1b0a10 2021-02-10T11:46:23.517 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:23.498 7f2a7196a700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a6c0e5ed0 con 0x7f2a6c0d1870 2021-02-10T11:46:23.517 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:23.498 7f2a6b574700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2a6c0d1870 0x7f2a6c1b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:23.517 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:23.498 7f2a6b574700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2a6c0d1870 0x7f2a6c1b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3301/0 says I am v2:172.21.2.112:59220/0 (socket says 172.21.2.112:59220) 2021-02-10T11:46:23.517 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:23.498 7f2a6b574700 1 -- 172.21.2.112:0/1554760497 learned_addr learned my addr 172.21.2.112:0/1554760497 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:46:23.517 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:23.498 7f2a6bd75700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f2a6c0f0650 0x7f2a6c0cfb20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:59004/0 (socket says 172.21.2.112:59004) 2021-02-10T11:46:23.517 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:23.499 7f2a6a572700 1 -- 172.21.2.112:0/1554760497 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4628807 0 0) 0x7f2a5c000c10 con 0x7f2a6c0f0650 2021-02-10T11:46:23.517 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:23.499 7f2a6b574700 1 -- 172.21.2.112:0/1554760497 >> v1:172.21.2.120:6789/0 conn(0x7f2a6c1b0a10 legacy=0x7f2a6c0cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:23.517 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:23.499 7f2a6b574700 1 -- 172.21.2.112:0/1554760497 >> v1:172.21.2.112:6789/0 conn(0x7f2a6c0f0650 legacy=0x7f2a6c0cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:23.517 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:23.499 7f2a6b574700 1 -- 172.21.2.112:0/1554760497 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a6c0d0530 con 0x7f2a6c0d1870 2021-02-10T11:46:23.517 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:23.499 7f2a6b574700 1 --2- 172.21.2.112:0/1554760497 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2a6c0d1870 0x7f2a6c1b03f0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0x7f2a64006a80 tx=0x7f2a64009f00).ready entity=mon.2 client_cookie=33f8084ef1f38c8 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:23.518 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:23.499 7f2a6a572700 1 -- 172.21.2.112:0/1554760497 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2a6400b5c0 con 0x7f2a6c0d1870 2021-02-10T11:46:23.518 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:23.499 7f2a6a572700 1 -- 172.21.2.112:0/1554760497 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a6400b800 con 0x7f2a6c0d1870 2021-02-10T11:46:23.518 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:23.499 7f2a6a572700 1 -- 172.21.2.112:0/1554760497 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2a6400bd20 con 0x7f2a6c0d1870 2021-02-10T11:46:23.518 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:23.499 7f2a7196a700 1 -- 172.21.2.112:0/1554760497 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2a6c0d1870 msgr2=0x7f2a6c1b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:23.518 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:23.499 7f2a7196a700 1 --2- 172.21.2.112:0/1554760497 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2a6c0d1870 0x7f2a6c1b03f0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0x7f2a64006a80 tx=0x7f2a64009f00).stop 2021-02-10T11:46:23.518 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:23.500 7f2a7196a700 1 -- 172.21.2.112:0/1554760497 shutdown_connections 2021-02-10T11:46:23.518 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:23.500 7f2a7196a700 1 --2- 172.21.2.112:0/1554760497 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2a6c0d1870 0x7f2a6c1b03f0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:23.518 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:23.500 7f2a7196a700 1 -- 172.21.2.112:0/1554760497 >> 172.21.2.112:0/1554760497 conn(0x7f2a6c0ce450 msgr2=0x7f2a6c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:23.518 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:23.500 7f2a7196a700 1 -- 172.21.2.112:0/1554760497 shutdown_connections 2021-02-10T11:46:23.518 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:23.500 7f2a7196a700 1 -- 172.21.2.112:0/1554760497 wait complete. 2021-02-10T11:46:23.520 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:23.514 7f2a7196a700 1 Processor -- start 2021-02-10T11:46:23.520 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:23.515 7f2a7196a700 1 -- start start 2021-02-10T11:46:23.520 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:23.515 7f2a7196a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2a6c18a540 0x7f2a6c197480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:23.520 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:23.515 7f2a7196a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2a6c18aa50 0x7f2a6c0b77d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:23.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:23.515 7f2a7196a700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2a6c0b7df0 0x7f2a6c0b81f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:23.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:23.515 7f2a7196a700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a6c1a79d0 con 0x7f2a6c18aa50 2021-02-10T11:46:23.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:23.515 7f2a7196a700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a6c1a9e20 con 0x7f2a6c0b7df0 2021-02-10T11:46:23.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:23.515 7f2a7196a700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a6c1aa780 con 0x7f2a6c18a540 2021-02-10T11:46:23.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:23.515 7f2a6ad73700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2a6c18aa50 0x7f2a6c0b77d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:23.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:23.515 7f2a6b574700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2a6c18a540 0x7f2a6c197480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:23.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:23.515 7f2a6ad73700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2a6c18aa50 0x7f2a6c0b77d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:59546/0 (socket says 172.21.2.112:59546) 2021-02-10T11:46:23.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:23.515 7f2a6ad73700 1 -- 172.21.2.112:0/2110026676 learned_addr learned my addr 172.21.2.112:0/2110026676 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:46:23.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:23.515 7f2a6bd75700 1 --2- 172.21.2.112:0/2110026676 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2a6c0b7df0 0x7f2a6c0b81f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:23.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:23.516 7f2a6ad73700 1 -- 172.21.2.112:0/2110026676 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2a6c18a540 msgr2=0x7f2a6c197480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:23.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:23.516 7f2a6ad73700 1 --2- 172.21.2.112:0/2110026676 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2a6c18a540 0x7f2a6c197480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:23.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:23.516 7f2a6ad73700 1 -- 172.21.2.112:0/2110026676 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2a6c0b7df0 msgr2=0x7f2a6c0b81f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:23.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:23.516 7f2a6ad73700 1 --2- 172.21.2.112:0/2110026676 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2a6c0b7df0 0x7f2a6c0b81f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:23.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:23.516 7f2a6ad73700 1 -- 172.21.2.112:0/2110026676 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a6c1aa9a0 con 0x7f2a6c18aa50 2021-02-10T11:46:23.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:23.516 7f2a6ad73700 1 --2- 172.21.2.112:0/2110026676 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2a6c18aa50 0x7f2a6c0b77d0 secure :-1 s=READY pgs=2879 cs=0 l=1 rev1=1 rx=0x7f2a60003010 tx=0x7f2a6000a8a0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:23.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:23.516 7f2a53fff700 1 -- 172.21.2.112:0/2110026676 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2a6000ac90 con 0x7f2a6c18aa50 2021-02-10T11:46:23.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:23.516 7f2a53fff700 1 -- 172.21.2.112:0/2110026676 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a6000c0d0 con 0x7f2a6c18aa50 2021-02-10T11:46:23.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:23.516 7f2a53fff700 1 -- 172.21.2.112:0/2110026676 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2a6000c5f0 con 0x7f2a6c18aa50 2021-02-10T11:46:23.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:23.516 7f2a7196a700 1 -- 172.21.2.112:0/2110026676 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a6c1aa9a0 con 0x7f2a6c18aa50 2021-02-10T11:46:23.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:23.516 7f2a7196a700 1 -- 172.21.2.112:0/2110026676 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a6c1aaf30 con 0x7f2a6c18aa50 2021-02-10T11:46:23.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:23.517 7f2a53fff700 1 -- 172.21.2.112:0/2110026676 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2a60008030 con 0x7f2a6c18aa50 2021-02-10T11:46:23.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:23.517 7f2a53fff700 1 --2- 172.21.2.112:0/2110026676 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2a4c0186c0 0x7f2a4c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:23.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:23.517 7f2a53fff700 1 -- 172.21.2.112:0/2110026676 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f2a60032440 con 0x7f2a6c18aa50 2021-02-10T11:46:23.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:23.517 7f2a6b574700 1 --2- 172.21.2.112:0/2110026676 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2a4c0186c0 0x7f2a4c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:23.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:23.517 7f2a6b574700 1 --2- 172.21.2.112:0/2110026676 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2a4c0186c0 0x7f2a4c01abc0 crc :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:23.550 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:23.548 7f2a7196a700 1 -- 172.21.2.112:0/2110026676 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a6c1aa250 con 0x7f2a6c18aa50 2021-02-10T11:46:23.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:23.550 7f2a53fff700 1 -- 172.21.2.112:0/2110026676 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2a6002df20 con 0x7f2a6c18aa50 2021-02-10T11:46:23.679 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:23.676 7f2a7196a700 1 -- 172.21.2.112:0/2110026676 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2a6c1aa250 con 0x7f2a6c18aa50 2021-02-10T11:46:23.679 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:23.677 7f2a53fff700 1 -- 172.21.2.112:0/2110026676 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v172) v1 ==== 74+0+14007 (secure 0 0 0) 0x7f2a6002df20 con 0x7f2a6c18aa50 2021-02-10T11:46:23.680 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:46:23.680 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":172,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:46:21.423567","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":37,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":36,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:46:18.395022","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"171","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":37,"pool_name":"cephfs_data","create_time":"2021-02-10 11:46:19.415292","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":168,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:46:23.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:23.679 7f2a7196a700 1 -- 172.21.2.112:0/2110026676 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2a4c0186c0 msgr2=0x7f2a4c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:23.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:23.679 7f2a7196a700 1 --2- 172.21.2.112:0/2110026676 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2a4c0186c0 0x7f2a4c01abc0 crc :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:23.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:23.679 7f2a7196a700 1 -- 172.21.2.112:0/2110026676 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2a6c18aa50 msgr2=0x7f2a6c0b77d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:23.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:23.679 7f2a7196a700 1 --2- 172.21.2.112:0/2110026676 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2a6c18aa50 0x7f2a6c0b77d0 secure :-1 s=READY pgs=2879 cs=0 l=1 rev1=1 rx=0x7f2a60003010 tx=0x7f2a6000a8a0).stop 2021-02-10T11:46:23.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:23.679 7f2a7196a700 1 -- 172.21.2.112:0/2110026676 shutdown_connections 2021-02-10T11:46:23.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:23.679 7f2a7196a700 1 --2- 172.21.2.112:0/2110026676 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2a4c0186c0 0x7f2a4c01abc0 unknown :-1 s=CLOSED pgs=1308 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:23.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:23.679 7f2a7196a700 1 --2- 172.21.2.112:0/2110026676 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2a6c0b7df0 0x7f2a6c0b81f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:23.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:23.679 7f2a7196a700 1 --2- 172.21.2.112:0/2110026676 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2a6c18aa50 0x7f2a6c0b77d0 unknown :-1 s=CLOSED pgs=2879 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:23.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:23.679 7f2a7196a700 1 --2- 172.21.2.112:0/2110026676 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2a6c18a540 0x7f2a6c197480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:23.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:23.679 7f2a7196a700 1 -- 172.21.2.112:0/2110026676 >> 172.21.2.112:0/2110026676 conn(0x7f2a6c0efc70 msgr2=0x7f2a6c0ef410 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:23.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:23.679 7f2a7196a700 1 -- 172.21.2.112:0/2110026676 shutdown_connections 2021-02-10T11:46:23.683 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:23.680 7f2a7196a700 1 -- 172.21.2.112:0/2110026676 wait complete. 2021-02-10T11:46:23.690 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:46:24.133 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.114 7fca0fdd1700 1 Processor -- start 2021-02-10T11:46:24.133 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.114 7fca0fdd1700 1 -- start start 2021-02-10T11:46:24.133 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.115 7fca0fdd1700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fca080d1880 0x7fca081b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:24.134 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.115 7fca0fdd1700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fca080cff80 0x7fca08109140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:24.134 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.115 7fca0fdd1700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca08109a70 con 0x7fca081b0af0 2021-02-10T11:46:24.134 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.115 7fca0fdd1700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca08109760 con 0x7fca080d1880 2021-02-10T11:46:24.134 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.115 7fca0fdd1700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca08088070 con 0x7fca080cff80 2021-02-10T11:46:24.134 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.115 7fca0dbed700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fca080d1880 0x7fca081b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:24.134 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.115 7fca0d3ec700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fca081b0af0 0x7fca08108950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:59018/0 (socket says 172.21.2.112:59018) 2021-02-10T11:46:24.134 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.115 7fca0dbed700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fca080d1880 0x7fca081b04d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:59552/0 (socket says 172.21.2.112:59552) 2021-02-10T11:46:24.134 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.115 7fca0d3ec700 1 -- 172.21.2.112:0/3845183449 learned_addr learned my addr 172.21.2.112:0/3845183449 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:46:24.134 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.115 7fca0cbeb700 1 -- 172.21.2.112:0/3845183449 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 490140238 0 0) 0x7fc9f80009b0 con 0x7fca081b0af0 2021-02-10T11:46:24.135 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.115 7fca0cbeb700 1 -- 172.21.2.112:0/3845183449 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9f4002fe0 con 0x7fca081b0af0 2021-02-10T11:46:24.135 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.115 7fca0e3ee700 1 --2- 172.21.2.112:0/3845183449 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fca080cff80 0x7fca08109140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:24.135 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.115 7fca0dbed700 1 -- 172.21.2.112:0/3845183449 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fca080cff80 msgr2=0x7fca08109140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:46:24.135 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.115 7fca0dbed700 1 --2- 172.21.2.112:0/3845183449 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fca080cff80 0x7fca08109140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:24.135 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.115 7fca0dbed700 1 -- 172.21.2.112:0/3845183449 >> v1:172.21.2.112:6789/0 conn(0x7fca081b0af0 legacy=0x7fca08108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:24.135 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.115 7fca0dbed700 1 -- 172.21.2.112:0/3845183449 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca080d1c80 con 0x7fca080d1880 2021-02-10T11:46:24.135 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.115 7fca0dbed700 1 --2- 172.21.2.112:0/3845183449 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fca080d1880 0x7fca081b04d0 secure :-1 s=READY pgs=2881 cs=0 l=1 rev1=1 rx=0x7fca04006650 tx=0x7fca04009a80).ready entity=mon.0 client_cookie=2af388ff0980c62 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:24.135 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.116 7fca0cbeb700 1 -- 172.21.2.112:0/3845183449 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fca040009b0 con 0x7fca080d1880 2021-02-10T11:46:24.135 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.116 7fca0cbeb700 1 -- 172.21.2.112:0/3845183449 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca0400c0c0 con 0x7fca080d1880 2021-02-10T11:46:24.135 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.116 7fca0cbeb700 1 -- 172.21.2.112:0/3845183449 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fca0400c510 con 0x7fca080d1880 2021-02-10T11:46:24.136 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.116 7fca0fdd1700 1 -- 172.21.2.112:0/3845183449 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fca080d1880 msgr2=0x7fca081b04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:24.136 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.116 7fca0fdd1700 1 --2- 172.21.2.112:0/3845183449 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fca080d1880 0x7fca081b04d0 secure :-1 s=READY pgs=2881 cs=0 l=1 rev1=1 rx=0x7fca04006650 tx=0x7fca04009a80).stop 2021-02-10T11:46:24.136 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.116 7fca0fdd1700 1 -- 172.21.2.112:0/3845183449 shutdown_connections 2021-02-10T11:46:24.136 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.116 7fca0fdd1700 1 --2- 172.21.2.112:0/3845183449 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fca080cff80 0x7fca08109140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:24.136 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.116 7fca0fdd1700 1 --2- 172.21.2.112:0/3845183449 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fca080d1880 0x7fca081b04d0 unknown :-1 s=CLOSED pgs=2881 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:24.136 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.116 7fca0fdd1700 1 -- 172.21.2.112:0/3845183449 >> 172.21.2.112:0/3845183449 conn(0x7fca080ce430 msgr2=0x7fca080d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:24.136 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.117 7fca0fdd1700 1 -- 172.21.2.112:0/3845183449 shutdown_connections 2021-02-10T11:46:24.136 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.117 7fca0fdd1700 1 -- 172.21.2.112:0/3845183449 wait complete. 2021-02-10T11:46:24.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.131 7fca0fdd1700 1 Processor -- start 2021-02-10T11:46:24.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.131 7fca0fdd1700 1 -- start start 2021-02-10T11:46:24.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.132 7fca0fdd1700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fca0819b340 0x7fca080ad9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:24.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.132 7fca0fdd1700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fca080adfc0 0x7fca081a4a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:24.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.132 7fca0fdd1700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fca081a5080 0x7fca081a5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:24.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.132 7fca0fdd1700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca0819b810 con 0x7fca0819b340 2021-02-10T11:46:24.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.132 7fca0fdd1700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca081a5aa0 con 0x7fca080adfc0 2021-02-10T11:46:24.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.132 7fca0fdd1700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca081a5cc0 con 0x7fca081a5080 2021-02-10T11:46:24.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.132 7fca0dbed700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fca0819b340 0x7fca080ad9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:24.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.132 7fca0dbed700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fca0819b340 0x7fca080ad9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:59560/0 (socket says 172.21.2.112:59560) 2021-02-10T11:46:24.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.132 7fca0dbed700 1 -- 172.21.2.112:0/298616858 learned_addr learned my addr 172.21.2.112:0/298616858 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:46:24.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.132 7fca0d3ec700 1 --2- 172.21.2.112:0/298616858 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fca080adfc0 0x7fca081a4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:24.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.132 7fca0e3ee700 1 --2- 172.21.2.112:0/298616858 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fca081a5080 0x7fca081a5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:24.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.132 7fca0dbed700 1 -- 172.21.2.112:0/298616858 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fca081a5080 msgr2=0x7fca081a5480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:24.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.132 7fca0dbed700 1 --2- 172.21.2.112:0/298616858 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fca081a5080 0x7fca081a5480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:24.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.132 7fca0dbed700 1 -- 172.21.2.112:0/298616858 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fca080adfc0 msgr2=0x7fca081a4a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:24.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.132 7fca0dbed700 1 --2- 172.21.2.112:0/298616858 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fca080adfc0 0x7fca081a4a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:24.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.132 7fca0dbed700 1 -- 172.21.2.112:0/298616858 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca081a5ee0 con 0x7fca0819b340 2021-02-10T11:46:24.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.133 7fca0dbed700 1 --2- 172.21.2.112:0/298616858 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fca0819b340 0x7fca080ad9a0 secure :-1 s=READY pgs=2882 cs=0 l=1 rev1=1 rx=0x7fca04009630 tx=0x7fca0400b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:24.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.133 7fc9fe7fc700 1 -- 172.21.2.112:0/298616858 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fca040093a0 con 0x7fca0819b340 2021-02-10T11:46:24.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.133 7fc9fe7fc700 1 -- 172.21.2.112:0/298616858 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca0400bd70 con 0x7fca0819b340 2021-02-10T11:46:24.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.133 7fc9fe7fc700 1 -- 172.21.2.112:0/298616858 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fca04005560 con 0x7fca0819b340 2021-02-10T11:46:24.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.133 7fca0fdd1700 1 -- 172.21.2.112:0/298616858 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca081a5ee0 con 0x7fca0819b340 2021-02-10T11:46:24.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.133 7fca0fdd1700 1 -- 172.21.2.112:0/298616858 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca081a2fb0 con 0x7fca0819b340 2021-02-10T11:46:24.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.133 7fc9fe7fc700 1 -- 172.21.2.112:0/298616858 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fca04008240 con 0x7fca0819b340 2021-02-10T11:46:24.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.133 7fc9fe7fc700 1 --2- 172.21.2.112:0/298616858 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc9f00186c0 0x7fc9f001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:24.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.133 7fc9fe7fc700 1 -- 172.21.2.112:0/298616858 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fca040327d0 con 0x7fca0819b340 2021-02-10T11:46:24.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.134 7fca0d3ec700 1 --2- 172.21.2.112:0/298616858 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc9f00186c0 0x7fc9f001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:24.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.134 7fca0d3ec700 1 --2- 172.21.2.112:0/298616858 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc9f00186c0 0x7fc9f001abc0 crc :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:24.166 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.164 7fca0fdd1700 1 -- 172.21.2.112:0/298616858 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca08082980 con 0x7fca0819b340 2021-02-10T11:46:24.168 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.166 7fc9fe7fc700 1 -- 172.21.2.112:0/298616858 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fca04008030 con 0x7fca0819b340 2021-02-10T11:46:24.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.296 7fca0fdd1700 1 -- 172.21.2.112:0/298616858 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fca08082980 con 0x7fca0819b340 2021-02-10T11:46:24.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.299 7fc9fe7fc700 1 -- 172.21.2.112:0/298616858 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v95) v1 ==== 184+0+0 (secure 0 0 0) 0x7fca04008240 con 0x7fca0819b340 2021-02-10T11:46:24.306 INFO:teuthology.orchestra.run.gibba012.stderr:updated caps for client.0 2021-02-10T11:46:24.307 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.305 7fca0fdd1700 1 -- 172.21.2.112:0/298616858 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc9f00186c0 msgr2=0x7fc9f001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:24.307 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.305 7fca0fdd1700 1 --2- 172.21.2.112:0/298616858 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc9f00186c0 0x7fc9f001abc0 crc :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:24.307 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.305 7fca0fdd1700 1 -- 172.21.2.112:0/298616858 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fca0819b340 msgr2=0x7fca080ad9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:24.307 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.305 7fca0fdd1700 1 --2- 172.21.2.112:0/298616858 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fca0819b340 0x7fca080ad9a0 secure :-1 s=READY pgs=2882 cs=0 l=1 rev1=1 rx=0x7fca04009630 tx=0x7fca0400b980).stop 2021-02-10T11:46:24.308 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.305 7fca0fdd1700 1 -- 172.21.2.112:0/298616858 shutdown_connections 2021-02-10T11:46:24.308 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.305 7fca0fdd1700 1 --2- 172.21.2.112:0/298616858 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc9f00186c0 0x7fc9f001abc0 unknown :-1 s=CLOSED pgs=1309 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:24.308 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.305 7fca0fdd1700 1 --2- 172.21.2.112:0/298616858 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fca080adfc0 0x7fca081a4a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:24.308 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.305 7fca0fdd1700 1 --2- 172.21.2.112:0/298616858 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fca081a5080 0x7fca081a5480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:24.308 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.305 7fca0fdd1700 1 --2- 172.21.2.112:0/298616858 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fca0819b340 0x7fca080ad9a0 unknown :-1 s=CLOSED pgs=2882 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:24.308 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.305 7fca0fdd1700 1 -- 172.21.2.112:0/298616858 >> 172.21.2.112:0/298616858 conn(0x7fca081b63c0 msgr2=0x7fca08082bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:24.308 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.305 7fca0fdd1700 1 -- 172.21.2.112:0/298616858 shutdown_connections 2021-02-10T11:46:24.308 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.305 7fca0fdd1700 1 -- 172.21.2.112:0/298616858 wait complete. 2021-02-10T11:46:24.316 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:46:24.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.738 7fe1fcbea700 1 Processor -- start 2021-02-10T11:46:24.757 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.738 7fe1fcbea700 1 -- start start 2021-02-10T11:46:24.757 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.738 7fe1fcbea700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe1f80d1c80 0x7fe1f80cef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:24.757 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.738 7fe1fcbea700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fe1f80cf590 0x7fe1f80cf990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:24.757 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.738 7fe1fcbea700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1f80fe1d0 con 0x7fe1f80d1880 2021-02-10T11:46:24.757 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.738 7fe1fcbea700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1f81b0cc0 con 0x7fe1f80cf590 2021-02-10T11:46:24.757 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.738 7fe1fcbea700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1f8109a70 con 0x7fe1f80d1c80 2021-02-10T11:46:24.757 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.739 7fe1f705f700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fe1f80cf590 0x7fe1f80cf990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:24.757 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.739 7fe1f685e700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fe1f80d1880 0x7fe1f81b04d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:59028/0 (socket says 172.21.2.112:59028) 2021-02-10T11:46:24.757 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.739 7fe1f685e700 1 -- 172.21.2.112:0/208830296 learned_addr learned my addr 172.21.2.112:0/208830296 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:46:24.757 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.739 7fe1f605d700 1 --2- 172.21.2.112:0/208830296 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe1f80d1c80 0x7fe1f80cef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:24.757 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.739 7fe1f705f700 1 -- 172.21.2.112:0/208830296 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe1f80d1c80 msgr2=0x7fe1f80cef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:46:24.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.739 7fe1f705f700 1 --2- 172.21.2.112:0/208830296 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe1f80d1c80 0x7fe1f80cef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:24.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.739 7fe1f705f700 1 -- 172.21.2.112:0/208830296 >> v1:172.21.2.112:6789/0 conn(0x7fe1f80d1880 legacy=0x7fe1f81b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:24.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.739 7fe1f705f700 1 -- 172.21.2.112:0/208830296 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1f8088070 con 0x7fe1f80cf590 2021-02-10T11:46:24.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.739 7fe1f585c700 1 -- 172.21.2.112:0/208830296 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1019511964 0 0) 0x7fe1e80009b0 con 0x7fe1f80d1880 2021-02-10T11:46:24.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.739 7fe1f705f700 1 --2- 172.21.2.112:0/208830296 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe1f80cf590 0x7fe1f80cf990 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 rx=0x7fe1f0006e50 tx=0x7fe1f0009df0).ready entity=mon.1 client_cookie=1c13938da22aaf80 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:24.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.739 7fe1f585c700 1 -- 172.21.2.112:0/208830296 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe1f000b510 con 0x7fe1f80cf590 2021-02-10T11:46:24.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.739 7fe1f585c700 1 -- 172.21.2.112:0/208830296 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1f0003030 con 0x7fe1f80cf590 2021-02-10T11:46:24.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.740 7fe1fcbea700 1 -- 172.21.2.112:0/208830296 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe1f80cf590 msgr2=0x7fe1f80cf990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:24.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.740 7fe1f585c700 1 -- 172.21.2.112:0/208830296 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe1f0000f90 con 0x7fe1f80cf590 2021-02-10T11:46:24.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.740 7fe1fcbea700 1 --2- 172.21.2.112:0/208830296 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe1f80cf590 0x7fe1f80cf990 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 rx=0x7fe1f0006e50 tx=0x7fe1f0009df0).stop 2021-02-10T11:46:24.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.740 7fe1fcbea700 1 -- 172.21.2.112:0/208830296 shutdown_connections 2021-02-10T11:46:24.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.740 7fe1fcbea700 1 --2- 172.21.2.112:0/208830296 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe1f80cf590 0x7fe1f80cf990 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:24.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.740 7fe1fcbea700 1 --2- 172.21.2.112:0/208830296 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe1f80d1c80 0x7fe1f80cef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:24.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.740 7fe1fcbea700 1 -- 172.21.2.112:0/208830296 >> 172.21.2.112:0/208830296 conn(0x7fe1f80ce430 msgr2=0x7fe1f80d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:24.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.740 7fe1fcbea700 1 -- 172.21.2.112:0/208830296 shutdown_connections 2021-02-10T11:46:24.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.740 7fe1fcbea700 1 -- 172.21.2.112:0/208830296 wait complete. 2021-02-10T11:46:24.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.755 7fe1fcbea700 1 Processor -- start 2021-02-10T11:46:24.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.755 7fe1fcbea700 1 -- start start 2021-02-10T11:46:24.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.755 7fe1fcbea700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe1f80abd30 0x7fe1f8082bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:24.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.755 7fe1fcbea700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe1f80831d0 0x7fe1f808a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:24.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.755 7fe1fcbea700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe1f80ac130 0x7fe1f808aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:24.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.755 7fe1fcbea700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1f8083700 con 0x7fe1f80831d0 2021-02-10T11:46:24.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.755 7fe1fcbea700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1f808b0c0 con 0x7fe1f80ac130 2021-02-10T11:46:24.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.755 7fe1fcbea700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1f808b2a0 con 0x7fe1f80abd30 2021-02-10T11:46:24.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.755 7fe1f605d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe1f80831d0 0x7fe1f808a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:24.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.755 7fe1f605d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe1f80831d0 0x7fe1f808a480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:59570/0 (socket says 172.21.2.112:59570) 2021-02-10T11:46:24.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.755 7fe1f605d700 1 -- 172.21.2.112:0/1726419888 learned_addr learned my addr 172.21.2.112:0/1726419888 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:46:24.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.756 7fe1f605d700 1 -- 172.21.2.112:0/1726419888 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe1f80abd30 msgr2=0x7fe1f8082bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:24.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.756 7fe1f605d700 1 --2- 172.21.2.112:0/1726419888 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe1f80abd30 0x7fe1f8082bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:24.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.756 7fe1f605d700 1 -- 172.21.2.112:0/1726419888 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe1f80ac130 msgr2=0x7fe1f808aaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:24.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.756 7fe1f605d700 1 --2- 172.21.2.112:0/1726419888 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe1f80ac130 0x7fe1f808aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:24.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.756 7fe1f605d700 1 -- 172.21.2.112:0/1726419888 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1f808b5f0 con 0x7fe1f80831d0 2021-02-10T11:46:24.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.756 7fe1f605d700 1 --2- 172.21.2.112:0/1726419888 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe1f80831d0 0x7fe1f808a480 secure :-1 s=READY pgs=2884 cs=0 l=1 rev1=1 rx=0x7fe1ec006f60 tx=0x7fe1ec00b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:24.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.756 7fe1e77fe700 1 -- 172.21.2.112:0/1726419888 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe1ec00bd60 con 0x7fe1f80831d0 2021-02-10T11:46:24.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.756 7fe1e77fe700 1 -- 172.21.2.112:0/1726419888 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1ec000c90 con 0x7fe1f80831d0 2021-02-10T11:46:24.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.756 7fe1fcbea700 1 -- 172.21.2.112:0/1726419888 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1f808b5f0 con 0x7fe1f80831d0 2021-02-10T11:46:24.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.756 7fe1e77fe700 1 -- 172.21.2.112:0/1726419888 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe1ec0022b0 con 0x7fe1f80831d0 2021-02-10T11:46:24.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.756 7fe1fcbea700 1 -- 172.21.2.112:0/1726419888 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1f808bb50 con 0x7fe1f80831d0 2021-02-10T11:46:24.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.757 7fe1e77fe700 1 -- 172.21.2.112:0/1726419888 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe1ec006b00 con 0x7fe1f80831d0 2021-02-10T11:46:24.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.757 7fe1e77fe700 1 --2- 172.21.2.112:0/1726419888 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe1dc0186c0 0x7fe1dc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:24.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.757 7fe1e77fe700 1 -- 172.21.2.112:0/1726419888 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fe1ec032400 con 0x7fe1f80831d0 2021-02-10T11:46:24.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.757 7fe1f685e700 1 --2- 172.21.2.112:0/1726419888 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe1dc0186c0 0x7fe1dc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:24.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.758 7fe1f685e700 1 --2- 172.21.2.112:0/1726419888 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe1dc0186c0 0x7fe1dc01abc0 crc :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:24.790 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.788 7fe1fcbea700 1 -- 172.21.2.112:0/1726419888 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1f8082980 con 0x7fe1f80831d0 2021-02-10T11:46:24.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.790 7fe1e77fe700 1 -- 172.21.2.112:0/1726419888 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe1ec030650 con 0x7fe1f80831d0 2021-02-10T11:46:24.923 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.921 7fe1fcbea700 1 -- 172.21.2.112:0/1726419888 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fe1f8082980 con 0x7fe1f80831d0 2021-02-10T11:46:24.927 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.924 7fe1e77fe700 1 -- 172.21.2.112:0/1726419888 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v96) v1 ==== 184+0+0 (secure 0 0 0) 0x7fe1ec030650 con 0x7fe1f80831d0 2021-02-10T11:46:24.930 INFO:teuthology.orchestra.run.gibba012.stderr:updated caps for client.1 2021-02-10T11:46:24.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.929 7fe1fcbea700 1 -- 172.21.2.112:0/1726419888 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe1dc0186c0 msgr2=0x7fe1dc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:24.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.929 7fe1fcbea700 1 --2- 172.21.2.112:0/1726419888 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe1dc0186c0 0x7fe1dc01abc0 crc :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:24.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.929 7fe1fcbea700 1 -- 172.21.2.112:0/1726419888 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe1f80831d0 msgr2=0x7fe1f808a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:24.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.929 7fe1fcbea700 1 --2- 172.21.2.112:0/1726419888 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe1f80831d0 0x7fe1f808a480 secure :-1 s=READY pgs=2884 cs=0 l=1 rev1=1 rx=0x7fe1ec006f60 tx=0x7fe1ec00b970).stop 2021-02-10T11:46:24.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.930 7fe1fcbea700 1 -- 172.21.2.112:0/1726419888 shutdown_connections 2021-02-10T11:46:24.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.930 7fe1fcbea700 1 --2- 172.21.2.112:0/1726419888 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe1dc0186c0 0x7fe1dc01abc0 unknown :-1 s=CLOSED pgs=1310 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:24.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.930 7fe1fcbea700 1 --2- 172.21.2.112:0/1726419888 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe1f80ac130 0x7fe1f808aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:24.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.930 7fe1fcbea700 1 --2- 172.21.2.112:0/1726419888 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe1f80831d0 0x7fe1f808a480 unknown :-1 s=CLOSED pgs=2884 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:24.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.930 7fe1fcbea700 1 --2- 172.21.2.112:0/1726419888 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe1f80abd30 0x7fe1f8082bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:24.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.930 7fe1fcbea700 1 -- 172.21.2.112:0/1726419888 >> 172.21.2.112:0/1726419888 conn(0x7fe1f80afc60 msgr2=0x7fe1f80b50d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:24.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.930 7fe1fcbea700 1 -- 172.21.2.112:0/1726419888 shutdown_connections 2021-02-10T11:46:24.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:24.930 7fe1fcbea700 1 -- 172.21.2.112:0/1726419888 wait complete. 2021-02-10T11:46:24.940 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.2 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:46:25.381 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.363 7fa5ce7de700 1 Processor -- start 2021-02-10T11:46:25.381 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.363 7fa5ce7de700 1 -- start start 2021-02-10T11:46:25.382 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.363 7fa5ce7de700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fa5c80d1c80 0x7fa5c80cef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:25.382 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.363 7fa5ce7de700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fa5c80cf590 0x7fa5c80cf990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:25.382 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.363 7fa5ce7de700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5c80fe1d0 con 0x7fa5c80d1880 2021-02-10T11:46:25.382 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.363 7fa5ce7de700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5c81b0cc0 con 0x7fa5c80d1c80 2021-02-10T11:46:25.382 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.363 7fa5ce7de700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5c8109a70 con 0x7fa5c80cf590 2021-02-10T11:46:25.382 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.363 7fa5c77fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fa5c80d1c80 0x7fa5c80cef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:25.382 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.363 7fa5c77fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fa5c80d1c80 0x7fa5c80cef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:59578/0 (socket says 172.21.2.112:59578) 2021-02-10T11:46:25.382 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.363 7fa5c77fe700 1 -- 172.21.2.112:0/3549208591 learned_addr learned my addr 172.21.2.112:0/3549208591 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:46:25.382 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.364 7fa5c77fe700 1 -- 172.21.2.112:0/3549208591 >> v2:172.21.2.120:3300/0 conn(0x7fa5c80cf590 msgr2=0x7fa5c80cf990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:46:25.382 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.364 7fa5c77fe700 1 --2- 172.21.2.112:0/3549208591 >> v2:172.21.2.120:3300/0 conn(0x7fa5c80cf590 0x7fa5c80cf990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:25.382 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.364 7fa5c77fe700 1 -- 172.21.2.112:0/3549208591 >> v1:172.21.2.120:6789/0 conn(0x7fa5c80d1880 legacy=0x7fa5c81b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:46:25.382 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.364 7fa5c77fe700 1 -- 172.21.2.112:0/3549208591 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5c8088070 con 0x7fa5c80d1c80 2021-02-10T11:46:25.383 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.364 7fa5c77fe700 1 --2- 172.21.2.112:0/3549208591 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa5c80d1c80 0x7fa5c80cef70 secure :-1 s=READY pgs=2885 cs=0 l=1 rev1=1 rx=0x7fa5bc006700 tx=0x7fa5bc009b80).ready entity=mon.0 client_cookie=62f4af8fd092dbba server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:25.383 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.364 7fa5c6ffd700 1 -- 172.21.2.112:0/3549208591 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa5bc00c0c0 con 0x7fa5c80d1c80 2021-02-10T11:46:25.383 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.364 7fa5c6ffd700 1 -- 172.21.2.112:0/3549208591 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5bc00c340 con 0x7fa5c80d1c80 2021-02-10T11:46:25.383 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.364 7fa5c6ffd700 1 -- 172.21.2.112:0/3549208591 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa5bc00c860 con 0x7fa5c80d1c80 2021-02-10T11:46:25.383 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.364 7fa5ce7de700 1 -- 172.21.2.112:0/3549208591 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa5c80d1c80 msgr2=0x7fa5c80cef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:25.383 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.364 7fa5ce7de700 1 --2- 172.21.2.112:0/3549208591 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa5c80d1c80 0x7fa5c80cef70 secure :-1 s=READY pgs=2885 cs=0 l=1 rev1=1 rx=0x7fa5bc006700 tx=0x7fa5bc009b80).stop 2021-02-10T11:46:25.383 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.365 7fa5ce7de700 1 -- 172.21.2.112:0/3549208591 shutdown_connections 2021-02-10T11:46:25.383 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.365 7fa5ce7de700 1 --2- 172.21.2.112:0/3549208591 >> v2:172.21.2.120:3300/0 conn(0x7fa5c80cf590 0x7fa5c80cf990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:25.383 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.365 7fa5ce7de700 1 --2- 172.21.2.112:0/3549208591 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa5c80d1c80 0x7fa5c80cef70 unknown :-1 s=CLOSED pgs=2885 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:25.383 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.365 7fa5ce7de700 1 -- 172.21.2.112:0/3549208591 >> 172.21.2.112:0/3549208591 conn(0x7fa5c80ce430 msgr2=0x7fa5c80d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:25.383 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.365 7fa5ce7de700 1 -- 172.21.2.112:0/3549208591 shutdown_connections 2021-02-10T11:46:25.383 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.365 7fa5ce7de700 1 -- 172.21.2.112:0/3549208591 wait complete. 2021-02-10T11:46:25.385 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.379 7fa5ce7de700 1 Processor -- start 2021-02-10T11:46:25.385 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.380 7fa5ce7de700 1 -- start start 2021-02-10T11:46:25.385 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.380 7fa5ce7de700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa5c8086520 0x7fa5c81a1cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:25.385 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.380 7fa5ce7de700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa5c8086920 0x7fa5c8187e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:25.385 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.380 7fa5ce7de700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa5c81a22d0 0x7fa5c81884a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:25.385 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.380 7fa5ce7de700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5c8099600 con 0x7fa5c8086920 2021-02-10T11:46:25.385 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.380 7fa5ce7de700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5c80999f0 con 0x7fa5c81a22d0 2021-02-10T11:46:25.385 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.380 7fa5ce7de700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5c8188ac0 con 0x7fa5c8086520 2021-02-10T11:46:25.385 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.380 7fa5c77fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa5c8086920 0x7fa5c8187e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:25.385 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.380 7fa5c77fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa5c8086920 0x7fa5c8187e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:59582/0 (socket says 172.21.2.112:59582) 2021-02-10T11:46:25.385 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.380 7fa5c77fe700 1 -- 172.21.2.112:0/1301063871 learned_addr learned my addr 172.21.2.112:0/1301063871 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:46:25.385 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.381 7fa5c77fe700 1 -- 172.21.2.112:0/1301063871 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa5c8086520 msgr2=0x7fa5c81a1cb0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:46:25.386 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.381 7fa5c77fe700 1 --2- 172.21.2.112:0/1301063871 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa5c8086520 0x7fa5c81a1cb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:25.386 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.381 7fa5c77fe700 1 -- 172.21.2.112:0/1301063871 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa5c81a22d0 msgr2=0x7fa5c81884a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:25.386 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.381 7fa5c77fe700 1 --2- 172.21.2.112:0/1301063871 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa5c81a22d0 0x7fa5c81884a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:25.386 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.381 7fa5c77fe700 1 -- 172.21.2.112:0/1301063871 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5c8188ce0 con 0x7fa5c8086920 2021-02-10T11:46:25.386 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.381 7fa5c77fe700 1 --2- 172.21.2.112:0/1301063871 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa5c8086920 0x7fa5c8187e80 secure :-1 s=READY pgs=2886 cs=0 l=1 rev1=1 rx=0x7fa5bc00b5e0 tx=0x7fa5bc00b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:25.386 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.381 7fa5c4ff9700 1 -- 172.21.2.112:0/1301063871 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa5bc00bd70 con 0x7fa5c8086920 2021-02-10T11:46:25.386 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.381 7fa5c4ff9700 1 -- 172.21.2.112:0/1301063871 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5bc002030 con 0x7fa5c8086920 2021-02-10T11:46:25.386 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.381 7fa5c4ff9700 1 -- 172.21.2.112:0/1301063871 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa5bc002550 con 0x7fa5c8086920 2021-02-10T11:46:25.386 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.381 7fa5ce7de700 1 -- 172.21.2.112:0/1301063871 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5c8188ce0 con 0x7fa5c8086920 2021-02-10T11:46:25.386 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.381 7fa5ce7de700 1 -- 172.21.2.112:0/1301063871 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5c8189270 con 0x7fa5c8086920 2021-02-10T11:46:25.386 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.382 7fa5c4ff9700 1 -- 172.21.2.112:0/1301063871 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa5bc000c90 con 0x7fa5c8086920 2021-02-10T11:46:25.386 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.382 7fa5c4ff9700 1 --2- 172.21.2.112:0/1301063871 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa5a80186c0 0x7fa5a801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:25.386 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.382 7fa5c4ff9700 1 -- 172.21.2.112:0/1301063871 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fa5bc0327e0 con 0x7fa5c8086920 2021-02-10T11:46:25.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.383 7fa5c7fff700 1 --2- 172.21.2.112:0/1301063871 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa5a80186c0 0x7fa5a801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:25.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.383 7fa5c7fff700 1 --2- 172.21.2.112:0/1301063871 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa5a80186c0 0x7fa5a801abc0 crc :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:25.415 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.413 7fa5ce7de700 1 -- 172.21.2.112:0/1301063871 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5c80fe1d0 con 0x7fa5c8086920 2021-02-10T11:46:25.417 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.415 7fa5c4ff9700 1 -- 172.21.2.112:0/1301063871 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa5bc0009b0 con 0x7fa5c8086920 2021-02-10T11:46:25.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.546 7fa5ce7de700 1 -- 172.21.2.112:0/1301063871 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fa5c80fe1d0 con 0x7fa5c8086920 2021-02-10T11:46:25.551 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.549 7fa5c4ff9700 1 -- 172.21.2.112:0/1301063871 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.2 v97) v1 ==== 184+0+0 (secure 0 0 0) 0x7fa5bc01f520 con 0x7fa5c8086920 2021-02-10T11:46:25.556 INFO:teuthology.orchestra.run.gibba012.stderr:updated caps for client.2 2021-02-10T11:46:25.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.555 7fa5ce7de700 1 -- 172.21.2.112:0/1301063871 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa5a80186c0 msgr2=0x7fa5a801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:25.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.555 7fa5ce7de700 1 --2- 172.21.2.112:0/1301063871 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa5a80186c0 0x7fa5a801abc0 crc :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:25.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.555 7fa5ce7de700 1 -- 172.21.2.112:0/1301063871 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa5c8086920 msgr2=0x7fa5c8187e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:25.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.555 7fa5ce7de700 1 --2- 172.21.2.112:0/1301063871 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa5c8086920 0x7fa5c8187e80 secure :-1 s=READY pgs=2886 cs=0 l=1 rev1=1 rx=0x7fa5bc00b5e0 tx=0x7fa5bc00b980).stop 2021-02-10T11:46:25.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.555 7fa5ce7de700 1 -- 172.21.2.112:0/1301063871 shutdown_connections 2021-02-10T11:46:25.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.555 7fa5ce7de700 1 --2- 172.21.2.112:0/1301063871 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa5a80186c0 0x7fa5a801abc0 unknown :-1 s=CLOSED pgs=1311 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:25.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.555 7fa5ce7de700 1 --2- 172.21.2.112:0/1301063871 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa5c81a22d0 0x7fa5c81884a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:25.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.555 7fa5ce7de700 1 --2- 172.21.2.112:0/1301063871 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa5c8086920 0x7fa5c8187e80 unknown :-1 s=CLOSED pgs=2886 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:25.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.555 7fa5ce7de700 1 --2- 172.21.2.112:0/1301063871 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa5c8086520 0x7fa5c81a1cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:25.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.555 7fa5ce7de700 1 -- 172.21.2.112:0/1301063871 >> 172.21.2.112:0/1301063871 conn(0x7fa5c819f880 msgr2=0x7fa5c808f890 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:25.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.555 7fa5ce7de700 1 -- 172.21.2.112:0/1301063871 shutdown_connections 2021-02-10T11:46:25.558 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.555 7fa5ce7de700 1 -- 172.21.2.112:0/1301063871 wait complete. 2021-02-10T11:46:25.566 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.3 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:46:25.997 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.977 7f4d8f81c700 1 Processor -- start 2021-02-10T11:46:25.997 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.978 7f4d8f81c700 1 -- start start 2021-02-10T11:46:25.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.978 7f4d8f81c700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4d880d1860 0x7f4d881b04b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:25.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.978 7f4d8f81c700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f4d880cff60 0x7f4d88109120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:25.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.978 7f4d8f81c700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d88109a50 con 0x7f4d881b0ad0 2021-02-10T11:46:25.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.978 7f4d8f81c700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d88109740 con 0x7f4d880cff60 2021-02-10T11:46:25.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.978 7f4d8f81c700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d88088070 con 0x7f4d880d1860 2021-02-10T11:46:25.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.978 7f4d8ce37700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f4d881b0ad0 0x7f4d88108930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:59052/0 (socket says 172.21.2.112:59052) 2021-02-10T11:46:25.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.978 7f4d8ce37700 1 -- 172.21.2.112:0/291109740 learned_addr learned my addr 172.21.2.112:0/291109740 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:46:25.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.979 7f4d7bfff700 1 -- 172.21.2.112:0/291109740 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4214129535 0 0) 0x7f4d840009b0 con 0x7f4d881b0ad0 2021-02-10T11:46:25.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.979 7f4d7bfff700 1 -- 172.21.2.112:0/291109740 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d74002fe0 con 0x7f4d881b0ad0 2021-02-10T11:46:25.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.979 7f4d8de39700 1 --2- 172.21.2.112:0/291109740 >> v2:172.21.2.120:3300/0 conn(0x7f4d880cff60 0x7f4d88109120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:25.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.979 7f4d8d638700 1 --2- 172.21.2.112:0/291109740 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4d880d1860 0x7f4d881b04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:25.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.979 7f4d7bfff700 1 -- 172.21.2.112:0/291109740 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3166976717 0 0) 0x7f4d840010b0 con 0x7f4d881b0ad0 2021-02-10T11:46:25.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.979 7f4d7bfff700 1 -- 172.21.2.112:0/291109740 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4d880d1860 msgr2=0x7f4d881b04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:46:25.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.979 7f4d7bfff700 1 --2- 172.21.2.112:0/291109740 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4d880d1860 0x7f4d881b04b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:25.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.979 7f4d7bfff700 1 -- 172.21.2.112:0/291109740 >> v2:172.21.2.120:3300/0 conn(0x7f4d880cff60 msgr2=0x7f4d88109120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:46:25.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.979 7f4d7bfff700 1 --2- 172.21.2.112:0/291109740 >> v2:172.21.2.120:3300/0 conn(0x7f4d880cff60 0x7f4d88109120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:25.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.979 7f4d7bfff700 1 -- 172.21.2.112:0/291109740 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d88088260 con 0x7f4d881b0ad0 2021-02-10T11:46:25.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.979 7f4d7bfff700 1 -- 172.21.2.112:0/291109740 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f4d840015b0 con 0x7f4d881b0ad0 2021-02-10T11:46:25.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.979 7f4d8d638700 1 --2- 172.21.2.112:0/291109740 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4d880d1860 0x7f4d881b04b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:46:25.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.979 7f4d7bfff700 1 -- 172.21.2.112:0/291109740 >> v1:172.21.2.112:6789/0 conn(0x7f4d881b0ad0 legacy=0x7f4d88108930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:25.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.979 7f4d7bfff700 1 --2- 172.21.2.112:0/291109740 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4d74003a60 0x7f4d74005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:25.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.979 7f4d7bfff700 1 --2- 172.21.2.112:0/291109740 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4d7404a7a0 0x7f4d7404cc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:25.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.979 7f4d7bfff700 1 --2- 172.21.2.112:0/291109740 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4d7404d510 0x7f4d7404fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:25.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.979 7f4d7bfff700 1 -- 172.21.2.112:0/291109740 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d740502e0 con 0x7f4d7404d510 2021-02-10T11:46:25.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.979 7f4d7bfff700 1 -- 172.21.2.112:0/291109740 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d74050500 con 0x7f4d74003a60 2021-02-10T11:46:26.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.979 7f4d7bfff700 1 -- 172.21.2.112:0/291109740 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d74050720 con 0x7f4d7404a7a0 2021-02-10T11:46:26.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.979 7f4d8de39700 1 --2- 172.21.2.112:0/291109740 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4d7404d510 0x7f4d7404fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:26.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.979 7f4d8d638700 1 --2- 172.21.2.112:0/291109740 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4d74003a60 0x7f4d74005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:26.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.980 7f4d8ce37700 1 --2- 172.21.2.112:0/291109740 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4d7404a7a0 0x7f4d7404cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:26.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.980 7f4d8de39700 1 -- 172.21.2.112:0/291109740 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4d7404a7a0 msgr2=0x7f4d7404cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:46:26.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.980 7f4d8de39700 1 --2- 172.21.2.112:0/291109740 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4d7404a7a0 0x7f4d7404cc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:26.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.980 7f4d8de39700 1 -- 172.21.2.112:0/291109740 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4d74003a60 msgr2=0x7f4d74005ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:46:26.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.980 7f4d8de39700 1 --2- 172.21.2.112:0/291109740 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4d74003a60 0x7f4d74005ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:26.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.980 7f4d8de39700 1 -- 172.21.2.112:0/291109740 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d740509c0 con 0x7f4d7404d510 2021-02-10T11:46:26.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.980 7f4d8de39700 1 --2- 172.21.2.112:0/291109740 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4d7404d510 0x7f4d7404fa30 secure :-1 s=READY pgs=2888 cs=0 l=1 rev1=1 rx=0x7f4d80006a40 tx=0x7f4d8000c0c0).ready entity=mon.0 client_cookie=2823968550503dea server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:26.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.980 7f4d7bfff700 1 -- 172.21.2.112:0/291109740 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4d8000c4b0 con 0x7f4d7404d510 2021-02-10T11:46:26.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.980 7f4d7bfff700 1 -- 172.21.2.112:0/291109740 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d8000c6f0 con 0x7f4d7404d510 2021-02-10T11:46:26.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.980 7f4d7bfff700 1 -- 172.21.2.112:0/291109740 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4d8000cc10 con 0x7f4d7404d510 2021-02-10T11:46:26.001 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.980 7f4d8f81c700 1 -- 172.21.2.112:0/291109740 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4d7404d510 msgr2=0x7f4d7404fa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:26.001 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.980 7f4d8f81c700 1 --2- 172.21.2.112:0/291109740 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4d7404d510 0x7f4d7404fa30 secure :-1 s=READY pgs=2888 cs=0 l=1 rev1=1 rx=0x7f4d80006a40 tx=0x7f4d8000c0c0).stop 2021-02-10T11:46:26.001 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.980 7f4d8de39700 1 -- 172.21.2.112:0/291109740 reap_dead start 2021-02-10T11:46:26.001 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.981 7f4d8f81c700 1 -- 172.21.2.112:0/291109740 shutdown_connections 2021-02-10T11:46:26.001 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.981 7f4d8f81c700 1 -- 172.21.2.112:0/291109740 >> 172.21.2.112:0/291109740 conn(0x7f4d880ce410 msgr2=0x7f4d880d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:26.001 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.981 7f4d8f81c700 1 -- 172.21.2.112:0/291109740 shutdown_connections 2021-02-10T11:46:26.001 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.981 7f4d8f81c700 1 -- 172.21.2.112:0/291109740 wait complete. 2021-02-10T11:46:26.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.995 7f4d8f81c700 1 Processor -- start 2021-02-10T11:46:26.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.995 7f4d8f81c700 1 -- start start 2021-02-10T11:46:26.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.996 7f4d8f81c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4d880b5150 0x7f4d880c0b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:26.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.996 7f4d8f81c700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4d880c1160 0x7f4d880c1560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:26.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.996 7f4d8f81c700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4d880c1b80 0x7f4d880bee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:26.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.996 7f4d8f81c700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d880c05c0 con 0x7f4d880b5150 2021-02-10T11:46:26.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.996 7f4d8f81c700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d880b5550 con 0x7f4d880c1160 2021-02-10T11:46:26.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.996 7f4d8f81c700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d8808bf00 con 0x7f4d880c1b80 2021-02-10T11:46:26.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.996 7f4d8d638700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4d880b5150 0x7f4d880c0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:26.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.996 7f4d8d638700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4d880b5150 0x7f4d880c0b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:59600/0 (socket says 172.21.2.112:59600) 2021-02-10T11:46:26.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.996 7f4d8d638700 1 -- 172.21.2.112:0/2954082182 learned_addr learned my addr 172.21.2.112:0/2954082182 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:46:26.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.997 7f4d8ce37700 1 --2- 172.21.2.112:0/2954082182 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4d880c1160 0x7f4d880c1560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:26.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.997 7f4d8de39700 1 --2- 172.21.2.112:0/2954082182 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4d880c1b80 0x7f4d880bee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:26.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.997 7f4d8d638700 1 -- 172.21.2.112:0/2954082182 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4d880c1b80 msgr2=0x7f4d880bee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:26.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.997 7f4d8d638700 1 --2- 172.21.2.112:0/2954082182 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4d880c1b80 0x7f4d880bee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:26.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.997 7f4d8d638700 1 -- 172.21.2.112:0/2954082182 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4d880c1160 msgr2=0x7f4d880c1560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:26.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.997 7f4d8d638700 1 --2- 172.21.2.112:0/2954082182 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4d880c1160 0x7f4d880c1560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:26.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.997 7f4d8d638700 1 -- 172.21.2.112:0/2954082182 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d880bf480 con 0x7f4d880b5150 2021-02-10T11:46:26.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.997 7f4d8d638700 1 --2- 172.21.2.112:0/2954082182 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4d880b5150 0x7f4d880c0b40 secure :-1 s=READY pgs=2889 cs=0 l=1 rev1=1 rx=0x7f4d7c00b610 tx=0x7f4d7c00b9b0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:26.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.997 7f4d79ffb700 1 -- 172.21.2.112:0/2954082182 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4d7c00bda0 con 0x7f4d880b5150 2021-02-10T11:46:26.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.997 7f4d79ffb700 1 -- 172.21.2.112:0/2954082182 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d7c000c90 con 0x7f4d880b5150 2021-02-10T11:46:26.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.997 7f4d79ffb700 1 -- 172.21.2.112:0/2954082182 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4d7c0022b0 con 0x7f4d880b5150 2021-02-10T11:46:26.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.997 7f4d8f81c700 1 -- 172.21.2.112:0/2954082182 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d880bf480 con 0x7f4d880b5150 2021-02-10T11:46:26.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.997 7f4d8f81c700 1 -- 172.21.2.112:0/2954082182 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d880bfa10 con 0x7f4d880b5150 2021-02-10T11:46:26.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.998 7f4d79ffb700 1 -- 172.21.2.112:0/2954082182 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4d7c009fe0 con 0x7f4d880b5150 2021-02-10T11:46:26.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.998 7f4d79ffb700 1 --2- 172.21.2.112:0/2954082182 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4d680186c0 0x7f4d6801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:26.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.998 7f4d79ffb700 1 -- 172.21.2.112:0/2954082182 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f4d7c0323f0 con 0x7f4d880b5150 2021-02-10T11:46:26.005 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.998 7f4d8ce37700 1 --2- 172.21.2.112:0/2954082182 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4d680186c0 0x7f4d6801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:26.005 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:25.998 7f4d8ce37700 1 --2- 172.21.2.112:0/2954082182 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4d680186c0 0x7f4d6801abc0 crc :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:26.030 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:26.028 7f4d8f81c700 1 -- 172.21.2.112:0/2954082182 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d88109a50 con 0x7f4d880b5150 2021-02-10T11:46:26.033 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:26.031 7f4d79ffb700 1 -- 172.21.2.112:0/2954082182 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4d7c025fd0 con 0x7f4d880b5150 2021-02-10T11:46:26.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:26.160 7f4d8f81c700 1 -- 172.21.2.112:0/2954082182 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f4d88109a50 con 0x7f4d880b5150 2021-02-10T11:46:26.166 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:26.164 7f4d79ffb700 1 -- 172.21.2.112:0/2954082182 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.3 v98) v1 ==== 184+0+0 (secure 0 0 0) 0x7f4d7c009fe0 con 0x7f4d880b5150 2021-02-10T11:46:26.169 INFO:teuthology.orchestra.run.gibba012.stderr:updated caps for client.3 2021-02-10T11:46:26.170 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:26.168 7f4d8f81c700 1 -- 172.21.2.112:0/2954082182 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4d680186c0 msgr2=0x7f4d6801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:26.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:26.168 7f4d8f81c700 1 --2- 172.21.2.112:0/2954082182 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4d680186c0 0x7f4d6801abc0 crc :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:26.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:26.168 7f4d8f81c700 1 -- 172.21.2.112:0/2954082182 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4d880b5150 msgr2=0x7f4d880c0b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:26.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:26.168 7f4d8f81c700 1 --2- 172.21.2.112:0/2954082182 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4d880b5150 0x7f4d880c0b40 secure :-1 s=READY pgs=2889 cs=0 l=1 rev1=1 rx=0x7f4d7c00b610 tx=0x7f4d7c00b9b0).stop 2021-02-10T11:46:26.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:26.169 7f4d8f81c700 1 -- 172.21.2.112:0/2954082182 shutdown_connections 2021-02-10T11:46:26.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:26.169 7f4d8f81c700 1 --2- 172.21.2.112:0/2954082182 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4d680186c0 0x7f4d6801abc0 unknown :-1 s=CLOSED pgs=1312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:26.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:26.169 7f4d8f81c700 1 --2- 172.21.2.112:0/2954082182 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4d880c1160 0x7f4d880c1560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:26.171 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:26.169 7f4d8f81c700 1 --2- 172.21.2.112:0/2954082182 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4d880c1b80 0x7f4d880bee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:26.172 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:26.169 7f4d8f81c700 1 --2- 172.21.2.112:0/2954082182 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4d880b5150 0x7f4d880c0b40 unknown :-1 s=CLOSED pgs=2889 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:26.172 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:26.169 7f4d8f81c700 1 -- 172.21.2.112:0/2954082182 >> 172.21.2.112:0/2954082182 conn(0x7f4d8809dbe0 msgr2=0x7f4d88086f40 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:26.172 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:26.169 7f4d8f81c700 1 -- 172.21.2.112:0/2954082182 shutdown_connections 2021-02-10T11:46:26.172 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:26.169 7f4d8f81c700 1 -- 172.21.2.112:0/2954082182 wait complete. 2021-02-10T11:46:26.179 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:46:26.310 DEBUG:teuthology.orchestra.run.gibba012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:46:26.313 DEBUG:teuthology.orchestra.run.gibba020:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:46:26.607 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:26.588 7f00c49a4700 1 Processor -- start 2021-02-10T11:46:26.607 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:26.588 7f00c49a4700 1 -- start start 2021-02-10T11:46:26.607 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:26.589 7f00c49a4700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f00c00d0340 0x7f00c00ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:26.607 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:26.589 7f00c49a4700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00c00f0b10 con 0x7f00c00d1c70 2021-02-10T11:46:26.607 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:26.589 7f00c49a4700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00c01b0be0 con 0x7f00c00d1870 2021-02-10T11:46:26.607 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:26.589 7f00c49a4700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00c00e5ed0 con 0x7f00c00d0340 2021-02-10T11:46:26.607 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:26.589 7f00bde1b700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f00c00d0340 0x7f00c00ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:26.607 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:26.589 7f00bee1d700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f00c00d1c70 0x7f00c00efc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:59074/0 (socket says 172.21.2.112:59074) 2021-02-10T11:46:26.608 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:26.589 7f00bee1d700 1 -- 172.21.2.112:0/953312881 learned_addr learned my addr 172.21.2.112:0/953312881 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:46:26.608 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:26.589 7f00bde1b700 1 -- 172.21.2.112:0/953312881 >> v1:172.21.2.120:6789/0 conn(0x7f00c00d1870 legacy=0x7f00c01b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:46:26.608 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:26.589 7f00bde1b700 1 -- 172.21.2.112:0/953312881 >> v1:172.21.2.112:6789/0 conn(0x7f00c00d1c70 legacy=0x7f00c00efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:26.608 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:26.589 7f00bde1b700 1 -- 172.21.2.112:0/953312881 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00c00f0450 con 0x7f00c00d0340 2021-02-10T11:46:26.608 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:26.589 7f00bde1b700 1 --2- 172.21.2.112:0/953312881 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f00c00d0340 0x7f00c00ef640 secure :-1 s=READY pgs=2891 cs=0 l=1 rev1=1 rx=0x7f00b0006650 tx=0x7f00b0009a80).ready entity=mon.0 client_cookie=c0e215aef275e647 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:26.608 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:26.589 7f00bd61a700 1 -- 172.21.2.112:0/953312881 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f00b000c0c0 con 0x7f00c00d0340 2021-02-10T11:46:26.608 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:26.589 7f00bd61a700 1 -- 172.21.2.112:0/953312881 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00b000c400 con 0x7f00c00d0340 2021-02-10T11:46:26.608 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:26.589 7f00bd61a700 1 -- 172.21.2.112:0/953312881 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f00b000c850 con 0x7f00c00d0340 2021-02-10T11:46:26.608 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:26.589 7f00c49a4700 1 -- 172.21.2.112:0/953312881 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f00c00d0340 msgr2=0x7f00c00ef640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:26.608 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:26.589 7f00c49a4700 1 --2- 172.21.2.112:0/953312881 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f00c00d0340 0x7f00c00ef640 secure :-1 s=READY pgs=2891 cs=0 l=1 rev1=1 rx=0x7f00b0006650 tx=0x7f00b0009a80).stop 2021-02-10T11:46:26.608 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:26.590 7f00c49a4700 1 -- 172.21.2.112:0/953312881 shutdown_connections 2021-02-10T11:46:26.608 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:26.590 7f00c49a4700 1 --2- 172.21.2.112:0/953312881 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f00c00d0340 0x7f00c00ef640 unknown :-1 s=CLOSED pgs=2891 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:26.609 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:26.590 7f00c49a4700 1 -- 172.21.2.112:0/953312881 >> 172.21.2.112:0/953312881 conn(0x7f00c00ce450 msgr2=0x7f00c00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:26.609 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:26.590 7f00c49a4700 1 -- 172.21.2.112:0/953312881 shutdown_connections 2021-02-10T11:46:26.609 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:26.590 7f00c49a4700 1 -- 172.21.2.112:0/953312881 wait complete. 2021-02-10T11:46:26.610 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:26.605 7f00c49a4700 1 Processor -- start 2021-02-10T11:46:26.610 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:26.605 7f00c49a4700 1 -- start start 2021-02-10T11:46:26.610 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:26.605 7f00c49a4700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f00c00b32d0 0x7f00c00b11e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:26.611 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:26.605 7f00c49a4700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f00c00b1800 0x7f00c00aaa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:26.611 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:26.605 7f00c49a4700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f00c00ab090 0x7f00c00ab490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:26.611 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:26.605 7f00c49a4700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00c00b1d10 con 0x7f00c00b32d0 2021-02-10T11:46:26.611 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:26.605 7f00c49a4700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00c00b1f30 con 0x7f00c00ab090 2021-02-10T11:46:26.611 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:26.605 7f00c49a4700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00c00b2150 con 0x7f00c00b1800 2021-02-10T11:46:26.611 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:26.606 7f00be61c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f00c00b32d0 0x7f00c00b11e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:26.611 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:26.606 7f00be61c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f00c00b32d0 0x7f00c00b11e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:59616/0 (socket says 172.21.2.112:59616) 2021-02-10T11:46:26.611 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:26.606 7f00be61c700 1 -- 172.21.2.112:0/3914119144 learned_addr learned my addr 172.21.2.112:0/3914119144 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:46:26.611 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:26.606 7f00be61c700 1 -- 172.21.2.112:0/3914119144 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f00c00b1800 msgr2=0x7f00c00aaa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:26.611 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:26.606 7f00be61c700 1 --2- 172.21.2.112:0/3914119144 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f00c00b1800 0x7f00c00aaa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:26.611 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:26.606 7f00be61c700 1 -- 172.21.2.112:0/3914119144 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f00c00ab090 msgr2=0x7f00c00ab490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:26.611 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:26.606 7f00be61c700 1 --2- 172.21.2.112:0/3914119144 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f00c00ab090 0x7f00c00ab490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:26.611 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:26.606 7f00be61c700 1 -- 172.21.2.112:0/3914119144 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00c00b2370 con 0x7f00c00b32d0 2021-02-10T11:46:26.611 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:26.606 7f00be61c700 1 --2- 172.21.2.112:0/3914119144 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f00c00b32d0 0x7f00c00b11e0 secure :-1 s=READY pgs=2892 cs=0 l=1 rev1=1 rx=0x7f00a800b530 tx=0x7f00a800b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:26.612 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:26.606 7f00b6ffd700 1 -- 172.21.2.112:0/3914119144 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f00a800bcc0 con 0x7f00c00b32d0 2021-02-10T11:46:26.612 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:26.606 7f00b6ffd700 1 -- 172.21.2.112:0/3914119144 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00a8006020 con 0x7f00c00b32d0 2021-02-10T11:46:26.612 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:26.606 7f00c49a4700 1 -- 172.21.2.112:0/3914119144 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00c00b2370 con 0x7f00c00b32d0 2021-02-10T11:46:26.612 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:26.607 7f00c49a4700 1 -- 172.21.2.112:0/3914119144 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00c00b2370 con 0x7f00c00b32d0 2021-02-10T11:46:26.612 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:26.607 7f00b6ffd700 1 -- 172.21.2.112:0/3914119144 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f00a80054a0 con 0x7f00c00b32d0 2021-02-10T11:46:26.612 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:26.607 7f00b6ffd700 1 -- 172.21.2.112:0/3914119144 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f00a80054a0 con 0x7f00c00b32d0 2021-02-10T11:46:26.612 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:26.607 7f00b6ffd700 1 --2- 172.21.2.112:0/3914119144 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f00a4018730 0x7f00a401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:26.612 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:26.607 7f00b6ffd700 1 -- 172.21.2.112:0/3914119144 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f00a8032800 con 0x7f00c00b32d0 2021-02-10T11:46:26.612 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:26.607 7f00bde1b700 1 --2- 172.21.2.112:0/3914119144 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f00a4018730 0x7f00a401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:26.612 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:26.608 7f00bde1b700 1 --2- 172.21.2.112:0/3914119144 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f00a4018730 0x7f00a401ac10 crc :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:26.640 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:26.638 7f00c49a4700 1 -- 172.21.2.112:0/3914119144 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00c00ffb10 con 0x7f00c00b32d0 2021-02-10T11:46:26.643 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:26.641 7f00b6ffd700 1 -- 172.21.2.112:0/3914119144 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f00a8005f20 con 0x7f00c00b32d0 2021-02-10T11:46:26.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:26.774 7f00c49a4700 1 -- 172.21.2.112:0/3914119144 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f00c00ffb10 con 0x7f00c00b32d0 2021-02-10T11:46:26.777 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:26.774 7f00b6ffd700 1 -- 172.21.2.112:0/3914119144 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 148 v148) v1 ==== 95+0+2111 (secure 0 0 0) 0x7f00a80054a0 con 0x7f00c00b32d0 2021-02-10T11:46:26.777 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:46:26.777 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":148,"default_fscid":18,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":12642,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/4212630012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":4212630012},{"type":"v1","addr":"172.21.2.112:6833","nonce":4212630012}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":144}],"filesystems":[{"mdsmap":{"epoch":148,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:46:20.421874","modified":"2021-02-10 11:46:22.427977","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12639},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12639":{"gid":12639,"name":"b","rank":0,"incarnation":145,"state":"up:active","state_seq":3,"addr":"172.21.2.120:6835/1654642147","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":1654642147},{"type":"v1","addr":"172.21.2.120:6835","nonce":1654642147}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[37],"metadata_pool":36,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":18}]} 2021-02-10T11:46:26.777 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 148 2021-02-10T11:46:26.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:26.776 7f00c49a4700 1 -- 172.21.2.112:0/3914119144 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f00a4018730 msgr2=0x7f00a401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:26.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:26.776 7f00c49a4700 1 --2- 172.21.2.112:0/3914119144 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f00a4018730 0x7f00a401ac10 crc :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:26.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:26.776 7f00c49a4700 1 -- 172.21.2.112:0/3914119144 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f00c00b32d0 msgr2=0x7f00c00b11e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:26.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:26.776 7f00c49a4700 1 --2- 172.21.2.112:0/3914119144 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f00c00b32d0 0x7f00c00b11e0 secure :-1 s=READY pgs=2892 cs=0 l=1 rev1=1 rx=0x7f00a800b530 tx=0x7f00a800b8d0).stop 2021-02-10T11:46:26.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:26.777 7f00c49a4700 1 -- 172.21.2.112:0/3914119144 shutdown_connections 2021-02-10T11:46:26.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:26.777 7f00c49a4700 1 --2- 172.21.2.112:0/3914119144 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f00a4018730 0x7f00a401ac10 unknown :-1 s=CLOSED pgs=1313 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:26.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:26.777 7f00c49a4700 1 --2- 172.21.2.112:0/3914119144 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f00c00ab090 0x7f00c00ab490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:26.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:26.777 7f00c49a4700 1 --2- 172.21.2.112:0/3914119144 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f00c00b1800 0x7f00c00aaa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:26.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:26.777 7f00c49a4700 1 --2- 172.21.2.112:0/3914119144 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f00c00b32d0 0x7f00c00b11e0 unknown :-1 s=CLOSED pgs=2892 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:26.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:26.777 7f00c49a4700 1 -- 172.21.2.112:0/3914119144 >> 172.21.2.112:0/3914119144 conn(0x7f00c010a820 msgr2=0x7f00c0101c00 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:26.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:26.777 7f00c49a4700 1 -- 172.21.2.112:0/3914119144 shutdown_connections 2021-02-10T11:46:26.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:26.777 7f00c49a4700 1 -- 172.21.2.112:0/3914119144 wait complete. 2021-02-10T11:46:26.989 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 148, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:46:20.421874', 'modified': '2021-02-10 11:46:22.427977', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 12639}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_12639': {'gid': 12639, 'name': 'b', 'rank': 0, 'incarnation': 145, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.120:6835/1654642147', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.120:6834', 'nonce': 1654642147}, {'type': 'v1', 'addr': '172.21.2.120:6835', 'nonce': 1654642147}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [37], 'metadata_pool': 36, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:46:26.989 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:46:26.989 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:46:27.389 INFO:teuthology.orchestra.run.gibba020.stdout:{ 2021-02-10T11:46:27.390 INFO:teuthology.orchestra.run.gibba020.stdout: "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", 2021-02-10T11:46:27.390 INFO:teuthology.orchestra.run.gibba020.stdout: "whoami": 0, 2021-02-10T11:46:27.390 INFO:teuthology.orchestra.run.gibba020.stdout: "id": 12639, 2021-02-10T11:46:27.390 INFO:teuthology.orchestra.run.gibba020.stdout: "want_state": "up:active", 2021-02-10T11:46:27.390 INFO:teuthology.orchestra.run.gibba020.stdout: "state": "up:active", 2021-02-10T11:46:27.391 INFO:teuthology.orchestra.run.gibba020.stdout: "rank_uptime": 6.9644243570000004, 2021-02-10T11:46:27.391 INFO:teuthology.orchestra.run.gibba020.stdout: "mdsmap_epoch": 148, 2021-02-10T11:46:27.391 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch": 172, 2021-02-10T11:46:27.391 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch_barrier": 172, 2021-02-10T11:46:27.391 INFO:teuthology.orchestra.run.gibba020.stdout: "uptime": 12.606808986000001 2021-02-10T11:46:27.391 INFO:teuthology.orchestra.run.gibba020.stdout:} 2021-02-10T11:46:27.393 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", "whoami": 0, "id": 12639, "want_state": "up:active", "state": "up:active", "rank_uptime": 6.9644243570000004, "mdsmap_epoch": 148, "osdmap_epoch": 172, "osdmap_epoch_barrier": 172, "uptime": 12.606808986000001 } 2021-02-10T11:46:27.394 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:46:27.823 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:27.804 7f894359d700 1 Processor -- start 2021-02-10T11:46:27.823 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:27.804 7f894359d700 1 -- start start 2021-02-10T11:46:27.824 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:27.805 7f894359d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f893c0d1870 0x7f893c1b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:27.824 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:27.805 7f894359d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f893c0f0650 0x7f893c0cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:27.824 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:27.805 7f894359d700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f893c0f0b10 con 0x7f893c1b0a10 2021-02-10T11:46:27.824 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:27.805 7f894359d700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f893c0fe230 con 0x7f893c0f0650 2021-02-10T11:46:27.824 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:27.805 7f894359d700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f893c0e5ed0 con 0x7f893c0d1870 2021-02-10T11:46:27.824 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:27.805 7f8941bba700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f893c0f0650 0x7f893c0cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:27.824 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:27.805 7f8941bba700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f893c0f0650 0x7f893c0cfb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:59622/0 (socket says 172.21.2.112:59622) 2021-02-10T11:46:27.824 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:27.805 7f8941bba700 1 -- 172.21.2.112:0/3677660121 learned_addr learned my addr 172.21.2.112:0/3677660121 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:46:27.824 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:27.805 7f8941bba700 1 -- 172.21.2.112:0/3677660121 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f893c0d1870 msgr2=0x7f893c1b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:46:27.824 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:27.805 7f8941bba700 1 --2- 172.21.2.112:0/3677660121 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f893c0d1870 0x7f893c1b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:27.824 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:27.805 7f8941bba700 1 -- 172.21.2.112:0/3677660121 >> v1:172.21.2.120:6789/0 conn(0x7f893c1b0a10 legacy=0x7f893c0cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:46:27.824 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:27.806 7f8941bba700 1 -- 172.21.2.112:0/3677660121 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f893c0f20a0 con 0x7f893c0f0650 2021-02-10T11:46:27.825 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:27.806 7f8941bba700 1 --2- 172.21.2.112:0/3677660121 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f893c0f0650 0x7f893c0cfb20 secure :-1 s=READY pgs=2893 cs=0 l=1 rev1=1 rx=0x7f8934006f00 tx=0x7f8934009f10).ready entity=mon.0 client_cookie=a286addde5e94e6d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:27.825 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:27.806 7f892ffff700 1 -- 172.21.2.112:0/3677660121 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f893400b3b0 con 0x7f893c0f0650 2021-02-10T11:46:27.825 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:27.806 7f892ffff700 1 -- 172.21.2.112:0/3677660121 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f893400b670 con 0x7f893c0f0650 2021-02-10T11:46:27.825 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:27.806 7f892ffff700 1 -- 172.21.2.112:0/3677660121 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f893400bb90 con 0x7f893c0f0650 2021-02-10T11:46:27.825 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:27.806 7f894359d700 1 -- 172.21.2.112:0/3677660121 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f893c0f0650 msgr2=0x7f893c0cfb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:27.825 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:27.806 7f894359d700 1 --2- 172.21.2.112:0/3677660121 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f893c0f0650 0x7f893c0cfb20 secure :-1 s=READY pgs=2893 cs=0 l=1 rev1=1 rx=0x7f8934006f00 tx=0x7f8934009f10).stop 2021-02-10T11:46:27.825 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:27.807 7f894359d700 1 -- 172.21.2.112:0/3677660121 shutdown_connections 2021-02-10T11:46:27.825 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:27.807 7f894359d700 1 --2- 172.21.2.112:0/3677660121 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f893c0f0650 0x7f893c0cfb20 unknown :-1 s=CLOSED pgs=2893 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:27.825 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:27.807 7f894359d700 1 --2- 172.21.2.112:0/3677660121 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f893c0d1870 0x7f893c1b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:27.825 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:27.807 7f894359d700 1 -- 172.21.2.112:0/3677660121 >> 172.21.2.112:0/3677660121 conn(0x7f893c0ce450 msgr2=0x7f893c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:27.825 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:27.807 7f894359d700 1 -- 172.21.2.112:0/3677660121 shutdown_connections 2021-02-10T11:46:27.825 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:27.807 7f894359d700 1 -- 172.21.2.112:0/3677660121 wait complete. 2021-02-10T11:46:27.827 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:27.821 7f894359d700 1 Processor -- start 2021-02-10T11:46:27.827 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:27.821 7f894359d700 1 -- start start 2021-02-10T11:46:27.827 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:27.822 7f894359d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f893c0b0ad0 0x7f893c091f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:27.827 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:27.822 7f894359d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f893c1a8850 0x7f893c1a8c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:27.827 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:27.822 7f894359d700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f893c1a6fa0 0x7f893c1a73a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:27.827 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:27.822 7f894359d700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f893c1abfe0 con 0x7f893c0b0ad0 2021-02-10T11:46:27.827 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:27.822 7f894359d700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f893c1a9c40 con 0x7f893c1a6fa0 2021-02-10T11:46:27.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:27.822 7f894359d700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f893c1a9730 con 0x7f893c1a8850 2021-02-10T11:46:27.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:27.822 7f89413b9700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f893c0b0ad0 0x7f893c091f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:27.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:27.822 7f89413b9700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f893c0b0ad0 0x7f893c091f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:59626/0 (socket says 172.21.2.112:59626) 2021-02-10T11:46:27.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:27.822 7f89413b9700 1 -- 172.21.2.112:0/682792545 learned_addr learned my addr 172.21.2.112:0/682792545 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:46:27.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:27.823 7f8941bba700 1 --2- 172.21.2.112:0/682792545 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f893c1a6fa0 0x7f893c1a73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:27.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:27.823 7f8940bb8700 1 --2- 172.21.2.112:0/682792545 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f893c1a8850 0x7f893c1a8c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:27.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:27.823 7f89413b9700 1 -- 172.21.2.112:0/682792545 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f893c1a8850 msgr2=0x7f893c1a8c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:27.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:27.823 7f89413b9700 1 --2- 172.21.2.112:0/682792545 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f893c1a8850 0x7f893c1a8c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:27.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:27.823 7f89413b9700 1 -- 172.21.2.112:0/682792545 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f893c1a6fa0 msgr2=0x7f893c1a73a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:27.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:27.823 7f89413b9700 1 --2- 172.21.2.112:0/682792545 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f893c1a6fa0 0x7f893c1a73a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:27.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:27.823 7f89413b9700 1 -- 172.21.2.112:0/682792545 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f893c1a9270 con 0x7f893c0b0ad0 2021-02-10T11:46:27.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:27.823 7f8941bba700 1 --2- 172.21.2.112:0/682792545 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f893c1a6fa0 0x7f893c1a73a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:46:27.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:27.823 7f8940bb8700 1 --2- 172.21.2.112:0/682792545 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f893c1a8850 0x7f893c1a8c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:46:27.829 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:27.823 7f89413b9700 1 --2- 172.21.2.112:0/682792545 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f893c0b0ad0 0x7f893c091f30 secure :-1 s=READY pgs=2894 cs=0 l=1 rev1=1 rx=0x7f8938006f00 tx=0x7f893800b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:27.829 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:27.823 7f892dffb700 1 -- 172.21.2.112:0/682792545 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8938006350 con 0x7f893c0b0ad0 2021-02-10T11:46:27.829 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:27.823 7f892dffb700 1 -- 172.21.2.112:0/682792545 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8938006b90 con 0x7f893c0b0ad0 2021-02-10T11:46:27.829 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:27.823 7f892dffb700 1 -- 172.21.2.112:0/682792545 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8938000c90 con 0x7f893c0b0ad0 2021-02-10T11:46:27.829 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:27.823 7f894359d700 1 -- 172.21.2.112:0/682792545 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f893c1a9270 con 0x7f893c0b0ad0 2021-02-10T11:46:27.829 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:27.823 7f894359d700 1 -- 172.21.2.112:0/682792545 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f893c1a7cb0 con 0x7f893c0b0ad0 2021-02-10T11:46:27.829 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:27.824 7f892dffb700 1 -- 172.21.2.112:0/682792545 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8938007000 con 0x7f893c0b0ad0 2021-02-10T11:46:27.829 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:27.824 7f892dffb700 1 --2- 172.21.2.112:0/682792545 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f891c0186c0 0x7f891c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:27.829 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:27.824 7f892dffb700 1 -- 172.21.2.112:0/682792545 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f8938032580 con 0x7f893c0b0ad0 2021-02-10T11:46:27.829 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:27.824 7f8940bb8700 1 --2- 172.21.2.112:0/682792545 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f891c0186c0 0x7f891c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:27.829 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:27.825 7f8940bb8700 1 --2- 172.21.2.112:0/682792545 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f891c0186c0 0x7f891c01abc0 crc :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:27.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:27.855 7f894359d700 1 -- 172.21.2.112:0/682792545 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f893c1a9710 con 0x7f893c0b0ad0 2021-02-10T11:46:27.860 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:27.858 7f892dffb700 1 -- 172.21.2.112:0/682792545 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f89380009b0 con 0x7f893c0b0ad0 2021-02-10T11:46:27.985 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:27.982 7f894359d700 1 -- 172.21.2.112:0/682792545 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f893c1a9710 con 0x7f893c0b0ad0 2021-02-10T11:46:27.985 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:27.983 7f892dffb700 1 -- 172.21.2.112:0/682792545 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v172) v1 ==== 74+0+14007 (secure 0 0 0) 0x7f8938007f00 con 0x7f893c0b0ad0 2021-02-10T11:46:27.986 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:46:27.986 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":172,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:46:21.423567","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":37,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":36,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:46:18.395022","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"171","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":37,"pool_name":"cephfs_data","create_time":"2021-02-10 11:46:19.415292","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":168,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:46:27.986 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:27.984 7f894359d700 1 -- 172.21.2.112:0/682792545 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f891c0186c0 msgr2=0x7f891c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:27.987 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:27.984 7f894359d700 1 --2- 172.21.2.112:0/682792545 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f891c0186c0 0x7f891c01abc0 crc :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:27.987 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:27.984 7f894359d700 1 -- 172.21.2.112:0/682792545 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f893c0b0ad0 msgr2=0x7f893c091f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:27.987 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:27.984 7f894359d700 1 --2- 172.21.2.112:0/682792545 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f893c0b0ad0 0x7f893c091f30 secure :-1 s=READY pgs=2894 cs=0 l=1 rev1=1 rx=0x7f8938006f00 tx=0x7f893800b940).stop 2021-02-10T11:46:27.987 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:27.985 7f894359d700 1 -- 172.21.2.112:0/682792545 shutdown_connections 2021-02-10T11:46:27.987 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:27.985 7f894359d700 1 --2- 172.21.2.112:0/682792545 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f891c0186c0 0x7f891c01abc0 unknown :-1 s=CLOSED pgs=1314 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:27.988 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:27.985 7f894359d700 1 --2- 172.21.2.112:0/682792545 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f893c1a6fa0 0x7f893c1a73a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:27.988 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:27.985 7f894359d700 1 --2- 172.21.2.112:0/682792545 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f893c1a8850 0x7f893c1a8c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:27.988 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:27.985 7f894359d700 1 --2- 172.21.2.112:0/682792545 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f893c0b0ad0 0x7f893c091f30 unknown :-1 s=CLOSED pgs=2894 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:27.988 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:27.985 7f894359d700 1 -- 172.21.2.112:0/682792545 >> 172.21.2.112:0/682792545 conn(0x7f893c19f8f0 msgr2=0x7f893c1a5f80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:27.988 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:27.985 7f894359d700 1 -- 172.21.2.112:0/682792545 shutdown_connections 2021-02-10T11:46:27.988 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:27.985 7f894359d700 1 -- 172.21.2.112:0/682792545 wait complete. 2021-02-10T11:46:27.996 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:13:57.508758', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 36, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:46:18.395022', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '171', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 37, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:46:19.415292', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '172', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:46:27.996 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:46:28.428 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:28.409 7fb8bb90f700 1 Processor -- start 2021-02-10T11:46:28.428 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:28.409 7fb8bb90f700 1 -- start start 2021-02-10T11:46:28.428 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:28.410 7fb8bb90f700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fb8b40d1870 0x7fb8b41b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:28.429 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:28.410 7fb8bb90f700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8b40fe1c0 con 0x7fb8b41b0ae0 2021-02-10T11:46:28.429 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:28.410 7fb8bb90f700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8b4109a60 con 0x7fb8b40cff70 2021-02-10T11:46:28.429 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:28.410 7fb8bb90f700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8b4088070 con 0x7fb8b40d1870 2021-02-10T11:46:28.429 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:28.410 7fb8b972b700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fb8b40d1870 0x7fb8b41b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:28.429 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:28.410 7fb8b8f2a700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fb8b41b0ae0 0x7fb8b4108940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:59096/0 (socket says 172.21.2.112:59096) 2021-02-10T11:46:28.429 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:28.410 7fb8b8f2a700 1 -- 172.21.2.112:0/854906766 learned_addr learned my addr 172.21.2.112:0/854906766 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:46:28.429 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:28.410 7fb8a7fff700 1 -- 172.21.2.112:0/854906766 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2830226892 0 0) 0x7fb8a80009b0 con 0x7fb8b41b0ae0 2021-02-10T11:46:28.429 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:28.410 7fb8a7fff700 1 -- 172.21.2.112:0/854906766 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8a0002fe0 con 0x7fb8b41b0ae0 2021-02-10T11:46:28.430 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:28.411 7fb8b972b700 1 -- 172.21.2.112:0/854906766 >> v1:172.21.2.120:6789/0 conn(0x7fb8b40cff70 legacy=0x7fb8b4109130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:28.430 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:28.411 7fb8b972b700 1 -- 172.21.2.112:0/854906766 >> v1:172.21.2.112:6789/0 conn(0x7fb8b41b0ae0 legacy=0x7fb8b4108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:28.430 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:28.411 7fb8b972b700 1 -- 172.21.2.112:0/854906766 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8b4088260 con 0x7fb8b40d1870 2021-02-10T11:46:28.430 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:28.411 7fb8b972b700 1 --2- 172.21.2.112:0/854906766 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb8b40d1870 0x7fb8b41b04c0 secure :-1 s=READY pgs=2896 cs=0 l=1 rev1=1 rx=0x7fb8b0006650 tx=0x7fb8b0009a00).ready entity=mon.0 client_cookie=27c504f5507d79e9 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:28.430 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:28.411 7fb8a7fff700 1 -- 172.21.2.112:0/854906766 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb8b0009df0 con 0x7fb8b40d1870 2021-02-10T11:46:28.430 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:28.411 7fb8a7fff700 1 -- 172.21.2.112:0/854906766 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8b00009b0 con 0x7fb8b40d1870 2021-02-10T11:46:28.430 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:28.411 7fb8a7fff700 1 -- 172.21.2.112:0/854906766 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb8b000c2b0 con 0x7fb8b40d1870 2021-02-10T11:46:28.430 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:28.411 7fb8bb90f700 1 -- 172.21.2.112:0/854906766 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb8b40d1870 msgr2=0x7fb8b41b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:28.430 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:28.411 7fb8bb90f700 1 --2- 172.21.2.112:0/854906766 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb8b40d1870 0x7fb8b41b04c0 secure :-1 s=READY pgs=2896 cs=0 l=1 rev1=1 rx=0x7fb8b0006650 tx=0x7fb8b0009a00).stop 2021-02-10T11:46:28.430 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:28.412 7fb8bb90f700 1 -- 172.21.2.112:0/854906766 shutdown_connections 2021-02-10T11:46:28.430 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:28.412 7fb8bb90f700 1 --2- 172.21.2.112:0/854906766 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb8b40d1870 0x7fb8b41b04c0 unknown :-1 s=CLOSED pgs=2896 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:28.431 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:28.412 7fb8bb90f700 1 -- 172.21.2.112:0/854906766 >> 172.21.2.112:0/854906766 conn(0x7fb8b40ce420 msgr2=0x7fb8b40d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:28.431 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:28.412 7fb8bb90f700 1 -- 172.21.2.112:0/854906766 shutdown_connections 2021-02-10T11:46:28.431 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:28.412 7fb8bb90f700 1 -- 172.21.2.112:0/854906766 wait complete. 2021-02-10T11:46:28.432 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:28.426 7fb8bb90f700 1 Processor -- start 2021-02-10T11:46:28.432 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:28.426 7fb8bb90f700 1 -- start start 2021-02-10T11:46:28.432 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:28.427 7fb8bb90f700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb8b41a0020 0x7fb8b41a0420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:28.432 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:28.427 7fb8bb90f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb8b4091330 0x7fb8b41917e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:28.433 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:28.427 7fb8bb90f700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb8b4191e00 0x7fb8b4092d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:28.433 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:28.427 7fb8bb90f700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8b4192200 con 0x7fb8b4091330 2021-02-10T11:46:28.433 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:28.427 7fb8bb90f700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8b4092790 con 0x7fb8b4191e00 2021-02-10T11:46:28.433 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:28.427 7fb8bb90f700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8b4096180 con 0x7fb8b41a0020 2021-02-10T11:46:28.433 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:28.427 7fb8b8f2a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb8b4091330 0x7fb8b41917e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:28.433 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:28.427 7fb8b8f2a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb8b4091330 0x7fb8b41917e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:59638/0 (socket says 172.21.2.112:59638) 2021-02-10T11:46:28.433 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:28.427 7fb8b8f2a700 1 -- 172.21.2.112:0/125596298 learned_addr learned my addr 172.21.2.112:0/125596298 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:46:28.433 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:28.427 7fb8b8f2a700 1 -- 172.21.2.112:0/125596298 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb8b41a0020 msgr2=0x7fb8b41a0420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:28.433 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:28.427 7fb8b8f2a700 1 --2- 172.21.2.112:0/125596298 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb8b41a0020 0x7fb8b41a0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:28.433 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:28.427 7fb8b8f2a700 1 -- 172.21.2.112:0/125596298 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb8b4191e00 msgr2=0x7fb8b4092d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:28.433 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:28.427 7fb8b8f2a700 1 --2- 172.21.2.112:0/125596298 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb8b4191e00 0x7fb8b4092d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:28.434 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:28.427 7fb8b8f2a700 1 -- 172.21.2.112:0/125596298 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8b4093330 con 0x7fb8b4091330 2021-02-10T11:46:28.434 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:28.428 7fb8b8f2a700 1 --2- 172.21.2.112:0/125596298 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb8b4091330 0x7fb8b41917e0 secure :-1 s=READY pgs=2897 cs=0 l=1 rev1=1 rx=0x7fb8a8001c60 tx=0x7fb8a800a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:28.434 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:28.428 7fb8a5ffb700 1 -- 172.21.2.112:0/125596298 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb8a800acc0 con 0x7fb8b4091330 2021-02-10T11:46:28.434 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:28.428 7fb8a5ffb700 1 -- 172.21.2.112:0/125596298 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8a8004140 con 0x7fb8b4091330 2021-02-10T11:46:28.434 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:28.428 7fb8a5ffb700 1 -- 172.21.2.112:0/125596298 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb8a8004620 con 0x7fb8b4091330 2021-02-10T11:46:28.434 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:28.428 7fb8bb90f700 1 -- 172.21.2.112:0/125596298 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8b4093330 con 0x7fb8b4091330 2021-02-10T11:46:28.434 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:28.428 7fb8bb90f700 1 -- 172.21.2.112:0/125596298 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8b40938c0 con 0x7fb8b4091330 2021-02-10T11:46:28.434 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:28.428 7fb8a5ffb700 1 -- 172.21.2.112:0/125596298 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb8a8004320 con 0x7fb8b4091330 2021-02-10T11:46:28.434 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:28.428 7fb8a5ffb700 1 --2- 172.21.2.112:0/125596298 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb8940186c0 0x7fb89401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:28.434 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:28.428 7fb8a5ffb700 1 -- 172.21.2.112:0/125596298 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fb8a8032590 con 0x7fb8b4091330 2021-02-10T11:46:28.435 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:28.429 7fb8b972b700 1 --2- 172.21.2.112:0/125596298 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb8940186c0 0x7fb89401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:28.435 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:28.429 7fb8b972b700 1 --2- 172.21.2.112:0/125596298 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb8940186c0 0x7fb89401abc0 crc :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:28.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:28.460 7fb8bb90f700 1 -- 172.21.2.112:0/125596298 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8b4082a70 con 0x7fb8b4091330 2021-02-10T11:46:28.464 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:28.462 7fb8a5ffb700 1 -- 172.21.2.112:0/125596298 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb8a8000940 con 0x7fb8b4091330 2021-02-10T11:46:28.593 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:28.591 7fb8bb90f700 1 -- 172.21.2.112:0/125596298 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7fb8b4082a70 con 0x7fb8b4091330 2021-02-10T11:46:28.593 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:28.591 7fb8a5ffb700 1 -- 172.21.2.112:0/125596298 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v172) v1 ==== 92+0+10 (secure 0 0 0) 0x7fb8a8004320 con 0x7fb8b4091330 2021-02-10T11:46:28.594 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:46:28.595 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:28.593 7fb8bb90f700 1 -- 172.21.2.112:0/125596298 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb8940186c0 msgr2=0x7fb89401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:28.595 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:28.593 7fb8bb90f700 1 --2- 172.21.2.112:0/125596298 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb8940186c0 0x7fb89401abc0 crc :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:28.595 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:28.593 7fb8bb90f700 1 -- 172.21.2.112:0/125596298 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb8b4091330 msgr2=0x7fb8b41917e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:28.595 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:28.593 7fb8bb90f700 1 --2- 172.21.2.112:0/125596298 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb8b4091330 0x7fb8b41917e0 secure :-1 s=READY pgs=2897 cs=0 l=1 rev1=1 rx=0x7fb8a8001c60 tx=0x7fb8a800a8d0).stop 2021-02-10T11:46:28.596 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:28.593 7fb8bb90f700 1 -- 172.21.2.112:0/125596298 shutdown_connections 2021-02-10T11:46:28.596 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:28.593 7fb8bb90f700 1 --2- 172.21.2.112:0/125596298 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb8940186c0 0x7fb89401abc0 unknown :-1 s=CLOSED pgs=1315 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:28.596 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:28.593 7fb8bb90f700 1 --2- 172.21.2.112:0/125596298 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb8b4191e00 0x7fb8b4092d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:28.596 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:28.593 7fb8bb90f700 1 --2- 172.21.2.112:0/125596298 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb8b4091330 0x7fb8b41917e0 unknown :-1 s=CLOSED pgs=2897 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:28.596 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:28.593 7fb8bb90f700 1 --2- 172.21.2.112:0/125596298 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb8b41a0020 0x7fb8b41a0420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:28.596 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:28.593 7fb8bb90f700 1 -- 172.21.2.112:0/125596298 >> 172.21.2.112:0/125596298 conn(0x7fb8b40bc0c0 msgr2=0x7fb8b4082ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:28.596 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:28.594 7fb8bb90f700 1 -- 172.21.2.112:0/125596298 shutdown_connections 2021-02-10T11:46:28.597 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:28.594 7fb8bb90f700 1 -- 172.21.2.112:0/125596298 wait complete. 2021-02-10T11:46:28.604 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:46:29.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.018 7f5104d47700 1 Processor -- start 2021-02-10T11:46:29.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.018 7f5104d47700 1 -- start start 2021-02-10T11:46:29.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.018 7f5104d47700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f51000d1870 0x7f51001b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:29.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.018 7f5104d47700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51000fe1c0 con 0x7f51000cff70 2021-02-10T11:46:29.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.018 7f5104d47700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5100109a60 con 0x7f51001b0ae0 2021-02-10T11:46:29.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.018 7f5104d47700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5100088070 con 0x7f51000d1870 2021-02-10T11:46:29.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.019 7f50ff314700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f51000cff70 0x7f5100109130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:59110/0 (socket says 172.21.2.112:59110) 2021-02-10T11:46:29.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.019 7f50ff314700 1 -- 172.21.2.112:0/2409306470 learned_addr learned my addr 172.21.2.112:0/2409306470 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:46:29.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.019 7f50fdb11700 1 -- 172.21.2.112:0/2409306470 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1681709744 0 0) 0x7f50f8001180 con 0x7f51000cff70 2021-02-10T11:46:29.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.019 7f50fdb11700 1 -- 172.21.2.112:0/2409306470 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50ec002fe0 con 0x7f51000cff70 2021-02-10T11:46:29.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.019 7f50feb13700 1 --2- 172.21.2.112:0/2409306470 >> v2:172.21.2.120:3300/0 conn(0x7f51000d1870 0x7f51001b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:29.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.019 7f50fdb11700 1 -- 172.21.2.112:0/2409306470 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1908201331 0 0) 0x7f50f80010e0 con 0x7f51000cff70 2021-02-10T11:46:29.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.019 7f50fdb11700 1 -- 172.21.2.112:0/2409306470 >> v2:172.21.2.120:3300/0 conn(0x7f51000d1870 msgr2=0x7f51001b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:46:29.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.019 7f50fdb11700 1 --2- 172.21.2.112:0/2409306470 >> v2:172.21.2.120:3300/0 conn(0x7f51000d1870 0x7f51001b04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:29.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.019 7f50fdb11700 1 -- 172.21.2.112:0/2409306470 >> v1:172.21.2.120:6789/0 conn(0x7f51001b0ae0 legacy=0x7f5100108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:29.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.019 7f50fdb11700 1 -- 172.21.2.112:0/2409306470 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5100088260 con 0x7f51000cff70 2021-02-10T11:46:29.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.019 7f50fdb11700 1 -- 172.21.2.112:0/2409306470 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f50f8001600 con 0x7f51000cff70 2021-02-10T11:46:29.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.019 7f50fdb11700 1 -- 172.21.2.112:0/2409306470 >> v1:172.21.2.112:6789/0 conn(0x7f51000cff70 legacy=0x7f5100109130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:29.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.019 7f50fdb11700 1 --2- 172.21.2.112:0/2409306470 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f50ec003af0 0x7f50ec006010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:29.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.019 7f50fdb11700 1 --2- 172.21.2.112:0/2409306470 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f50ec0068b0 0x7f50ec008dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:29.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.019 7f50fdb11700 1 --2- 172.21.2.112:0/2409306470 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f50ec0096d0 0x7f50ec00bb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:29.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.019 7f50fdb11700 1 -- 172.21.2.112:0/2409306470 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50ec00c3d0 con 0x7f50ec003af0 2021-02-10T11:46:29.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.019 7f50fdb11700 1 -- 172.21.2.112:0/2409306470 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50ec00c5f0 con 0x7f50ec0068b0 2021-02-10T11:46:29.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.019 7f50fdb11700 1 -- 172.21.2.112:0/2409306470 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50ec00c810 con 0x7f50ec0096d0 2021-02-10T11:46:29.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.019 7f50feb13700 1 --2- 172.21.2.112:0/2409306470 >> v2:172.21.2.120:3300/0 conn(0x7f51000d1870 0x7f51001b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:46:29.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.020 7f50feb13700 1 --2- 172.21.2.112:0/2409306470 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f50ec003af0 0x7f50ec006010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:29.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.020 7f50ff314700 1 --2- 172.21.2.112:0/2409306470 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f50ec0096d0 0x7f50ec00bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:29.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.020 7f50fe312700 1 --2- 172.21.2.112:0/2409306470 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f50ec0068b0 0x7f50ec008dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:29.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.020 7f50feb13700 1 -- 172.21.2.112:0/2409306470 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f50ec0096d0 msgr2=0x7f50ec00bb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:46:29.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.020 7f50feb13700 1 --2- 172.21.2.112:0/2409306470 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f50ec0096d0 0x7f50ec00bb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:29.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.020 7f50feb13700 1 -- 172.21.2.112:0/2409306470 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f50ec0068b0 msgr2=0x7f50ec008dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:46:29.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.020 7f50feb13700 1 --2- 172.21.2.112:0/2409306470 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f50ec0068b0 0x7f50ec008dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:29.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.020 7f50feb13700 1 -- 172.21.2.112:0/2409306470 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50ec00cab0 con 0x7f50ec003af0 2021-02-10T11:46:29.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.020 7f50ff314700 1 -- 172.21.2.112:0/2409306470 reap_dead start 2021-02-10T11:46:29.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.020 7f50feb13700 1 --2- 172.21.2.112:0/2409306470 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f50ec003af0 0x7f50ec006010 secure :-1 s=READY pgs=2899 cs=0 l=1 rev1=1 rx=0x7f50f004d9c0 tx=0x7f50f004dd60).ready entity=mon.0 client_cookie=b8afc806562df003 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:29.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.020 7f50fdb11700 1 -- 172.21.2.112:0/2409306470 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f50f0000c90 con 0x7f50ec003af0 2021-02-10T11:46:29.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.020 7f50fdb11700 1 -- 172.21.2.112:0/2409306470 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50f00067d0 con 0x7f50ec003af0 2021-02-10T11:46:29.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.020 7f50fdb11700 1 -- 172.21.2.112:0/2409306470 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f50f00022b0 con 0x7f50ec003af0 2021-02-10T11:46:29.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.020 7f5104d47700 1 -- 172.21.2.112:0/2409306470 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f50ec003af0 msgr2=0x7f50ec006010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:29.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.020 7f5104d47700 1 --2- 172.21.2.112:0/2409306470 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f50ec003af0 0x7f50ec006010 secure :-1 s=READY pgs=2899 cs=0 l=1 rev1=1 rx=0x7f50f004d9c0 tx=0x7f50f004dd60).stop 2021-02-10T11:46:29.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.021 7f5104d47700 1 -- 172.21.2.112:0/2409306470 shutdown_connections 2021-02-10T11:46:29.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.021 7f5104d47700 1 --2- 172.21.2.112:0/2409306470 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f50ec003af0 0x7f50ec006010 unknown :-1 s=CLOSED pgs=2899 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:29.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.021 7f5104d47700 1 -- 172.21.2.112:0/2409306470 >> 172.21.2.112:0/2409306470 conn(0x7f51000ce420 msgr2=0x7f51000d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:29.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.021 7f5104d47700 1 -- 172.21.2.112:0/2409306470 shutdown_connections 2021-02-10T11:46:29.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.021 7f5104d47700 1 -- 172.21.2.112:0/2409306470 wait complete. 2021-02-10T11:46:29.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.037 7f5104d47700 1 Processor -- start 2021-02-10T11:46:29.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.037 7f5104d47700 1 -- start start 2021-02-10T11:46:29.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.037 7f5104d47700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5100085bc0 0x7f51000863e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:29.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.037 7f5104d47700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f51000954c0 0x7f5100096300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:29.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.037 7f5104d47700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f51000958c0 0x7f5100096920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:29.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.037 7f5104d47700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5100095d80 con 0x7f5100085bc0 2021-02-10T11:46:29.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.037 7f5104d47700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51000bfa60 con 0x7f51000954c0 2021-02-10T11:46:29.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.037 7f5104d47700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5100085fc0 con 0x7f51000958c0 2021-02-10T11:46:29.045 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.037 7f50feb13700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5100085bc0 0x7f51000863e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:29.045 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.037 7f50feb13700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5100085bc0 0x7f51000863e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:59656/0 (socket says 172.21.2.112:59656) 2021-02-10T11:46:29.045 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.037 7f50feb13700 1 -- 172.21.2.112:0/470877919 learned_addr learned my addr 172.21.2.112:0/470877919 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:46:29.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.038 7f50feb13700 1 -- 172.21.2.112:0/470877919 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f51000958c0 msgr2=0x7f5100096920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:29.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.038 7f50feb13700 1 --2- 172.21.2.112:0/470877919 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f51000958c0 0x7f5100096920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:29.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.038 7f50feb13700 1 -- 172.21.2.112:0/470877919 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f51000954c0 msgr2=0x7f5100096300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:29.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.038 7f50feb13700 1 --2- 172.21.2.112:0/470877919 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f51000954c0 0x7f5100096300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:29.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.038 7f50feb13700 1 -- 172.21.2.112:0/470877919 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5100086c10 con 0x7f5100085bc0 2021-02-10T11:46:29.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.038 7f50feb13700 1 --2- 172.21.2.112:0/470877919 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5100085bc0 0x7f51000863e0 secure :-1 s=READY pgs=2900 cs=0 l=1 rev1=1 rx=0x7f50f004f590 tx=0x7f50f004f930).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:29.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.038 7f50f77fe700 1 -- 172.21.2.112:0/470877919 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f50f004fd20 con 0x7f5100085bc0 2021-02-10T11:46:29.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.038 7f50f77fe700 1 -- 172.21.2.112:0/470877919 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50f00052e0 con 0x7f5100085bc0 2021-02-10T11:46:29.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.038 7f50f77fe700 1 -- 172.21.2.112:0/470877919 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f50f0005740 con 0x7f5100085bc0 2021-02-10T11:46:29.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.038 7f5104d47700 1 -- 172.21.2.112:0/470877919 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5100086c10 con 0x7f5100085bc0 2021-02-10T11:46:29.047 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.038 7f5104d47700 1 -- 172.21.2.112:0/470877919 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5100086c10 con 0x7f5100085bc0 2021-02-10T11:46:29.047 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.039 7f50f77fe700 1 -- 172.21.2.112:0/470877919 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f50f0006030 con 0x7f5100085bc0 2021-02-10T11:46:29.047 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.039 7f50f77fe700 1 --2- 172.21.2.112:0/470877919 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f50e40186d0 0x7f50e401abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:29.047 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.039 7f50f77fe700 1 -- 172.21.2.112:0/470877919 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f50f0076680 con 0x7f5100085bc0 2021-02-10T11:46:29.047 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.039 7f50fe312700 1 --2- 172.21.2.112:0/470877919 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f50e40186d0 0x7f50e401abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:29.047 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.040 7f50fe312700 1 --2- 172.21.2.112:0/470877919 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f50e40186d0 0x7f50e401abd0 crc :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:29.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.068 7f5104d47700 1 -- 172.21.2.112:0/470877919 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51000fe1c0 con 0x7f5100085bc0 2021-02-10T11:46:29.073 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.071 7f50f77fe700 1 -- 172.21.2.112:0/470877919 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f50f004d030 con 0x7f5100085bc0 2021-02-10T11:46:29.201 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.199 7f5104d47700 1 -- 172.21.2.112:0/470877919 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f51000fe1c0 con 0x7f5100085bc0 2021-02-10T11:46:29.201 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.199 7f50f77fe700 1 -- 172.21.2.112:0/470877919 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v172) v1 ==== 104+0+10 (secure 0 0 0) 0x7f50f004fd20 con 0x7f5100085bc0 2021-02-10T11:46:29.202 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:46:29.204 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.201 7f5104d47700 1 -- 172.21.2.112:0/470877919 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f50e40186d0 msgr2=0x7f50e401abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:29.204 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.201 7f5104d47700 1 --2- 172.21.2.112:0/470877919 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f50e40186d0 0x7f50e401abd0 crc :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:29.204 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.201 7f5104d47700 1 -- 172.21.2.112:0/470877919 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5100085bc0 msgr2=0x7f51000863e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:29.204 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.201 7f5104d47700 1 --2- 172.21.2.112:0/470877919 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5100085bc0 0x7f51000863e0 secure :-1 s=READY pgs=2900 cs=0 l=1 rev1=1 rx=0x7f50f004f590 tx=0x7f50f004f930).stop 2021-02-10T11:46:29.204 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.201 7f5104d47700 1 -- 172.21.2.112:0/470877919 shutdown_connections 2021-02-10T11:46:29.204 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.201 7f5104d47700 1 --2- 172.21.2.112:0/470877919 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f50e40186d0 0x7f50e401abd0 unknown :-1 s=CLOSED pgs=1316 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:29.204 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.201 7f5104d47700 1 --2- 172.21.2.112:0/470877919 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f51000954c0 0x7f5100096300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:29.204 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.201 7f5104d47700 1 --2- 172.21.2.112:0/470877919 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f51000958c0 0x7f5100096920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:29.204 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.201 7f5104d47700 1 --2- 172.21.2.112:0/470877919 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5100085bc0 0x7f51000863e0 unknown :-1 s=CLOSED pgs=2900 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:29.205 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.202 7f5104d47700 1 -- 172.21.2.112:0/470877919 >> 172.21.2.112:0/470877919 conn(0x7f51000bef00 msgr2=0x7f510009d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:29.205 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.202 7f5104d47700 1 -- 172.21.2.112:0/470877919 shutdown_connections 2021-02-10T11:46:29.205 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.202 7f5104d47700 1 -- 172.21.2.112:0/470877919 wait complete. 2021-02-10T11:46:29.212 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:46:29.645 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.625 7f69f99cf700 1 Processor -- start 2021-02-10T11:46:29.645 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.625 7f69f99cf700 1 -- start start 2021-02-10T11:46:29.645 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.626 7f69f99cf700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f69f40d1870 0x7f69f41b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:29.645 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.626 7f69f99cf700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69f4109a60 con 0x7f69f41b0ae0 2021-02-10T11:46:29.645 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.626 7f69f99cf700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69f4088070 con 0x7f69f40cff70 2021-02-10T11:46:29.646 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.626 7f69f99cf700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69f40e5ef0 con 0x7f69f40d1870 2021-02-10T11:46:29.646 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.626 7f69f2d73700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f69f41b0ae0 0x7f69f4108940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:59126/0 (socket says 172.21.2.112:59126) 2021-02-10T11:46:29.646 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.626 7f69f2d73700 1 -- 172.21.2.112:0/3069616698 learned_addr learned my addr 172.21.2.112:0/3069616698 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:46:29.646 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.627 7f69f2572700 1 -- 172.21.2.112:0/3069616698 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 97956118 0 0) 0x7f69e40009b0 con 0x7f69f41b0ae0 2021-02-10T11:46:29.646 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.627 7f69f2572700 1 -- 172.21.2.112:0/3069616698 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69e0002fe0 con 0x7f69f41b0ae0 2021-02-10T11:46:29.646 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.627 7f69f3574700 1 --2- 172.21.2.112:0/3069616698 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f69f40d1870 0x7f69f41b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:29.646 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.627 7f69f2572700 1 -- 172.21.2.112:0/3069616698 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3046814662 0 0) 0x7f69e4001010 con 0x7f69f41b0ae0 2021-02-10T11:46:29.646 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.627 7f69f2572700 1 -- 172.21.2.112:0/3069616698 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f69f40d1870 msgr2=0x7f69f41b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:46:29.646 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.627 7f69f2572700 1 --2- 172.21.2.112:0/3069616698 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f69f40d1870 0x7f69f41b04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:29.646 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.627 7f69f2572700 1 -- 172.21.2.112:0/3069616698 >> v1:172.21.2.120:6789/0 conn(0x7f69f40cff70 legacy=0x7f69f4109130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:29.646 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.627 7f69f2572700 1 -- 172.21.2.112:0/3069616698 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69f40d1c70 con 0x7f69f41b0ae0 2021-02-10T11:46:29.646 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.627 7f69f2572700 1 -- 172.21.2.112:0/3069616698 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f69e4001590 con 0x7f69f41b0ae0 2021-02-10T11:46:29.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.627 7f69f2572700 1 -- 172.21.2.112:0/3069616698 >> v1:172.21.2.112:6789/0 conn(0x7f69f41b0ae0 legacy=0x7f69f4108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:29.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.627 7f69f2572700 1 --2- 172.21.2.112:0/3069616698 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f69e0003a60 0x7f69e0005f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:29.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.627 7f69f2572700 1 --2- 172.21.2.112:0/3069616698 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f69e0006780 0x7f69e0008c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:29.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.627 7f69f2572700 1 --2- 172.21.2.112:0/3069616698 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f69e00094f0 0x7f69e000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:29.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.627 7f69f2572700 1 -- 172.21.2.112:0/3069616698 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69e000c2c0 con 0x7f69e00094f0 2021-02-10T11:46:29.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.627 7f69f2572700 1 -- 172.21.2.112:0/3069616698 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69e000c4e0 con 0x7f69e0003a60 2021-02-10T11:46:29.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.627 7f69f2572700 1 -- 172.21.2.112:0/3069616698 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69e0050710 con 0x7f69e0006780 2021-02-10T11:46:29.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.627 7f69f3574700 1 --2- 172.21.2.112:0/3069616698 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f69e0003a60 0x7f69e0005f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:29.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.627 7f69f3d75700 1 --2- 172.21.2.112:0/3069616698 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f69e00094f0 0x7f69e000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:29.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.627 7f69f2d73700 1 --2- 172.21.2.112:0/3069616698 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f69e0006780 0x7f69e0008c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:29.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.628 7f69f3d75700 1 -- 172.21.2.112:0/3069616698 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f69e0006780 msgr2=0x7f69e0008c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:46:29.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.628 7f69f3d75700 1 --2- 172.21.2.112:0/3069616698 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f69e0006780 0x7f69e0008c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:29.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.628 7f69f3d75700 1 -- 172.21.2.112:0/3069616698 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f69e0003a60 msgr2=0x7f69e0005f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:46:29.648 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.628 7f69f3d75700 1 --2- 172.21.2.112:0/3069616698 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f69e0003a60 0x7f69e0005f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:29.648 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.628 7f69f3d75700 1 -- 172.21.2.112:0/3069616698 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69e0050940 con 0x7f69e00094f0 2021-02-10T11:46:29.648 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.628 7f69f3d75700 1 --2- 172.21.2.112:0/3069616698 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f69e00094f0 0x7f69e000ba10 secure :-1 s=READY pgs=2902 cs=0 l=1 rev1=1 rx=0x7f69ec006e40 tx=0x7f69ec009e30).ready entity=mon.0 client_cookie=8b28c3e49608e7d6 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:29.648 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.628 7f69f2572700 1 -- 172.21.2.112:0/3069616698 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f69ec00b300 con 0x7f69e00094f0 2021-02-10T11:46:29.648 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.628 7f69f2572700 1 -- 172.21.2.112:0/3069616698 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69ec006a60 con 0x7f69e00094f0 2021-02-10T11:46:29.648 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.628 7f69f2572700 1 -- 172.21.2.112:0/3069616698 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f69ec00b7f0 con 0x7f69e00094f0 2021-02-10T11:46:29.648 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.628 7f69f99cf700 1 -- 172.21.2.112:0/3069616698 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f69e00094f0 msgr2=0x7f69e000ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:29.648 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.628 7f69f99cf700 1 --2- 172.21.2.112:0/3069616698 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f69e00094f0 0x7f69e000ba10 secure :-1 s=READY pgs=2902 cs=0 l=1 rev1=1 rx=0x7f69ec006e40 tx=0x7f69ec009e30).stop 2021-02-10T11:46:29.648 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.628 7f69f3d75700 1 -- 172.21.2.112:0/3069616698 reap_dead start 2021-02-10T11:46:29.648 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.628 7f69f99cf700 1 -- 172.21.2.112:0/3069616698 shutdown_connections 2021-02-10T11:46:29.648 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.628 7f69f99cf700 1 -- 172.21.2.112:0/3069616698 >> 172.21.2.112:0/3069616698 conn(0x7f69f40ce420 msgr2=0x7f69f40d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:29.649 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.628 7f69f99cf700 1 -- 172.21.2.112:0/3069616698 shutdown_connections 2021-02-10T11:46:29.649 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.628 7f69f99cf700 1 -- 172.21.2.112:0/3069616698 wait complete. 2021-02-10T11:46:29.649 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.643 7f69f99cf700 1 Processor -- start 2021-02-10T11:46:29.649 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.643 7f69f99cf700 1 -- start start 2021-02-10T11:46:29.651 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.643 7f69f99cf700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f69f4085bc0 0x7f69f40863e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:29.651 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.643 7f69f99cf700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f69f40954c0 0x7f69f4096300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:29.651 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.643 7f69f99cf700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f69f40958c0 0x7f69f4096920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:29.651 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.643 7f69f99cf700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69f4095d80 con 0x7f69f40954c0 2021-02-10T11:46:29.651 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.643 7f69f99cf700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69f40bfa60 con 0x7f69f4085bc0 2021-02-10T11:46:29.651 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.643 7f69f99cf700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69f4085fc0 con 0x7f69f40958c0 2021-02-10T11:46:29.651 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.644 7f69f2d73700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f69f40954c0 0x7f69f4096300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:29.651 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.644 7f69f2d73700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f69f40954c0 0x7f69f4096300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:59672/0 (socket says 172.21.2.112:59672) 2021-02-10T11:46:29.651 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.644 7f69f2d73700 1 -- 172.21.2.112:0/1709151018 learned_addr learned my addr 172.21.2.112:0/1709151018 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:46:29.652 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.644 7f69f3574700 1 --2- 172.21.2.112:0/1709151018 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f69f4085bc0 0x7f69f40863e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:29.652 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.644 7f69f3d75700 1 --2- 172.21.2.112:0/1709151018 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f69f40958c0 0x7f69f4096920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:29.652 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.644 7f69f2d73700 1 -- 172.21.2.112:0/1709151018 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f69f40958c0 msgr2=0x7f69f4096920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:29.652 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.644 7f69f2d73700 1 --2- 172.21.2.112:0/1709151018 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f69f40958c0 0x7f69f4096920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:29.652 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.644 7f69f2d73700 1 -- 172.21.2.112:0/1709151018 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f69f4085bc0 msgr2=0x7f69f40863e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:29.652 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.644 7f69f2d73700 1 --2- 172.21.2.112:0/1709151018 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f69f4085bc0 0x7f69f40863e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:29.652 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.644 7f69f2d73700 1 -- 172.21.2.112:0/1709151018 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69f4086c10 con 0x7f69f40954c0 2021-02-10T11:46:29.652 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.644 7f69f3574700 1 --2- 172.21.2.112:0/1709151018 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f69f4085bc0 0x7f69f40863e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:46:29.652 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.644 7f69f3d75700 1 --2- 172.21.2.112:0/1709151018 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f69f40958c0 0x7f69f4096920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:46:29.652 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.644 7f69f2d73700 1 --2- 172.21.2.112:0/1709151018 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f69f40954c0 0x7f69f4096300 secure :-1 s=READY pgs=2903 cs=0 l=1 rev1=1 rx=0x7f69e4001bf0 tx=0x7f69e400a900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:29.652 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.644 7f69ebfff700 1 -- 172.21.2.112:0/1709151018 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f69e400acf0 con 0x7f69f40954c0 2021-02-10T11:46:29.652 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.645 7f69ebfff700 1 -- 172.21.2.112:0/1709151018 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69e4004140 con 0x7f69f40954c0 2021-02-10T11:46:29.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.645 7f69ebfff700 1 -- 172.21.2.112:0/1709151018 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f69e4004510 con 0x7f69f40954c0 2021-02-10T11:46:29.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.645 7f69f99cf700 1 -- 172.21.2.112:0/1709151018 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69f4086c10 con 0x7f69f40954c0 2021-02-10T11:46:29.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.645 7f69f99cf700 1 -- 172.21.2.112:0/1709151018 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69f4097230 con 0x7f69f40954c0 2021-02-10T11:46:29.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.645 7f69ebfff700 1 -- 172.21.2.112:0/1709151018 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f69e4007f30 con 0x7f69f40954c0 2021-02-10T11:46:29.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.645 7f69ebfff700 1 --2- 172.21.2.112:0/1709151018 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f69d80186c0 0x7f69d801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:29.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.645 7f69ebfff700 1 -- 172.21.2.112:0/1709151018 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f69e40323b0 con 0x7f69f40954c0 2021-02-10T11:46:29.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.646 7f69f3574700 1 --2- 172.21.2.112:0/1709151018 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f69d80186c0 0x7f69d801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:29.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.646 7f69f3574700 1 --2- 172.21.2.112:0/1709151018 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f69d80186c0 0x7f69d801abc0 crc :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:29.678 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.676 7f69f99cf700 1 -- 172.21.2.112:0/1709151018 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69f40fe1c0 con 0x7f69f40954c0 2021-02-10T11:46:29.681 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.679 7f69ebfff700 1 -- 172.21.2.112:0/1709151018 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f69e401be60 con 0x7f69f40954c0 2021-02-10T11:46:29.807 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.805 7f69f99cf700 1 -- 172.21.2.112:0/1709151018 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f69f40fe1c0 con 0x7f69f40954c0 2021-02-10T11:46:29.808 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.805 7f69ebfff700 1 -- 172.21.2.112:0/1709151018 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v172) v1 ==== 100+0+10 (secure 0 0 0) 0x7f69e401be60 con 0x7f69f40954c0 2021-02-10T11:46:29.808 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:46:29.810 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.807 7f69f99cf700 1 -- 172.21.2.112:0/1709151018 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f69d80186c0 msgr2=0x7f69d801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:29.810 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.807 7f69f99cf700 1 --2- 172.21.2.112:0/1709151018 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f69d80186c0 0x7f69d801abc0 crc :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:29.810 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.807 7f69f99cf700 1 -- 172.21.2.112:0/1709151018 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f69f40954c0 msgr2=0x7f69f4096300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:29.810 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.807 7f69f99cf700 1 --2- 172.21.2.112:0/1709151018 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f69f40954c0 0x7f69f4096300 secure :-1 s=READY pgs=2903 cs=0 l=1 rev1=1 rx=0x7f69e4001bf0 tx=0x7f69e400a900).stop 2021-02-10T11:46:29.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.807 7f69f99cf700 1 -- 172.21.2.112:0/1709151018 shutdown_connections 2021-02-10T11:46:29.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.807 7f69f99cf700 1 --2- 172.21.2.112:0/1709151018 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f69d80186c0 0x7f69d801abc0 unknown :-1 s=CLOSED pgs=1317 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:29.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.808 7f69f99cf700 1 --2- 172.21.2.112:0/1709151018 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f69f40958c0 0x7f69f4096920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:29.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.808 7f69f99cf700 1 --2- 172.21.2.112:0/1709151018 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f69f40954c0 0x7f69f4096300 unknown :-1 s=CLOSED pgs=2903 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:29.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.808 7f69f99cf700 1 --2- 172.21.2.112:0/1709151018 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f69f4085bc0 0x7f69f40863e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:29.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.808 7f69f99cf700 1 -- 172.21.2.112:0/1709151018 >> 172.21.2.112:0/1709151018 conn(0x7f69f40bef00 msgr2=0x7f69f409d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:29.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.808 7f69f99cf700 1 -- 172.21.2.112:0/1709151018 shutdown_connections 2021-02-10T11:46:29.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:29.808 7f69f99cf700 1 -- 172.21.2.112:0/1709151018 wait complete. 2021-02-10T11:46:29.818 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:46:29.818 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:46:30.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:30.228 7fc70684d700 1 Processor -- start 2021-02-10T11:46:30.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:30.228 7fc70684d700 1 -- start start 2021-02-10T11:46:30.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:30.229 7fc70684d700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fc7000d1870 0x7fc7001b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:30.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:30.229 7fc70684d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fc7001b0a10 0x7fc7000cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:30.248 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:30.229 7fc70684d700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7000f0b10 con 0x7fc7000d1c70 2021-02-10T11:46:30.248 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:30.229 7fc70684d700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7000e5ed0 con 0x7fc7001b0a10 2021-02-10T11:46:30.248 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:30.229 7fc70684d700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7000f0650 con 0x7fc7000d1870 2021-02-10T11:46:30.248 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:30.229 7fc6ff7fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fc7001b0a10 0x7fc7000cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:30.248 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:30.229 7fc704e6a700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fc7000d1c70 0x7fc7000cf950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:59142/0 (socket says 172.21.2.112:59142) 2021-02-10T11:46:30.248 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:30.229 7fc704e6a700 1 -- 172.21.2.112:0/4036974094 learned_addr learned my addr 172.21.2.112:0/4036974094 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:46:30.248 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:30.229 7fc6fffff700 1 --2- 172.21.2.112:0/4036974094 >> v2:172.21.2.120:3300/0 conn(0x7fc7000d1870 0x7fc7001b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:30.248 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:30.229 7fc6ff7fe700 1 -- 172.21.2.112:0/4036974094 >> v2:172.21.2.120:3300/0 conn(0x7fc7000d1870 msgr2=0x7fc7001b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:46:30.248 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:30.229 7fc6ff7fe700 1 --2- 172.21.2.112:0/4036974094 >> v2:172.21.2.120:3300/0 conn(0x7fc7000d1870 0x7fc7001b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:30.248 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:30.229 7fc6ff7fe700 1 -- 172.21.2.112:0/4036974094 >> v1:172.21.2.112:6789/0 conn(0x7fc7000d1c70 legacy=0x7fc7000cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:30.248 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:30.230 7fc6ff7fe700 1 -- 172.21.2.112:0/4036974094 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7000f0840 con 0x7fc7001b0a10 2021-02-10T11:46:30.248 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:30.229 7fc6feffd700 1 -- 172.21.2.112:0/4036974094 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4132190835 0 0) 0x7fc6f8001500 con 0x7fc7000d1c70 2021-02-10T11:46:30.249 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:30.230 7fc6ff7fe700 1 --2- 172.21.2.112:0/4036974094 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc7001b0a10 0x7fc7000cf330 secure :-1 s=READY pgs=2905 cs=0 l=1 rev1=1 rx=0x7fc6f4006650 tx=0x7fc6f4009a00).ready entity=mon.0 client_cookie=205cf893ee13f9a1 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:30.249 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:30.230 7fc6feffd700 1 -- 172.21.2.112:0/4036974094 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc6f4009df0 con 0x7fc7001b0a10 2021-02-10T11:46:30.249 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:30.230 7fc6feffd700 1 -- 172.21.2.112:0/4036974094 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6f40009b0 con 0x7fc7001b0a10 2021-02-10T11:46:30.249 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:30.230 7fc6feffd700 1 -- 172.21.2.112:0/4036974094 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc6f400c2b0 con 0x7fc7001b0a10 2021-02-10T11:46:30.249 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:30.230 7fc70684d700 1 -- 172.21.2.112:0/4036974094 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc7001b0a10 msgr2=0x7fc7000cf330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:30.249 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:30.230 7fc70684d700 1 --2- 172.21.2.112:0/4036974094 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc7001b0a10 0x7fc7000cf330 secure :-1 s=READY pgs=2905 cs=0 l=1 rev1=1 rx=0x7fc6f4006650 tx=0x7fc6f4009a00).stop 2021-02-10T11:46:30.249 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:30.231 7fc70684d700 1 -- 172.21.2.112:0/4036974094 shutdown_connections 2021-02-10T11:46:30.249 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:30.231 7fc70684d700 1 --2- 172.21.2.112:0/4036974094 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc7001b0a10 0x7fc7000cf330 unknown :-1 s=CLOSED pgs=2905 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:30.249 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:30.231 7fc70684d700 1 --2- 172.21.2.112:0/4036974094 >> v2:172.21.2.120:3300/0 conn(0x7fc7000d1870 0x7fc7001b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:30.249 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:30.231 7fc70684d700 1 -- 172.21.2.112:0/4036974094 >> 172.21.2.112:0/4036974094 conn(0x7fc7000ce450 msgr2=0x7fc7000d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:30.249 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:30.231 7fc70684d700 1 -- 172.21.2.112:0/4036974094 shutdown_connections 2021-02-10T11:46:30.249 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:30.231 7fc70684d700 1 -- 172.21.2.112:0/4036974094 wait complete. 2021-02-10T11:46:30.252 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:30.245 7fc70684d700 1 Processor -- start 2021-02-10T11:46:30.252 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:30.246 7fc70684d700 1 -- start start 2021-02-10T11:46:30.252 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:30.246 7fc70684d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc700199110 0x7fc700199510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:30.252 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:30.246 7fc70684d700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc7001b1c50 0x7fc70019a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:30.253 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:30.246 7fc70684d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc70019aae0 0x7fc70019aee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:30.253 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:30.246 7fc70684d700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc700199f40 con 0x7fc70019aae0 2021-02-10T11:46:30.253 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:30.246 7fc70684d700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc70018f160 con 0x7fc7001b1c50 2021-02-10T11:46:30.253 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:30.246 7fc70684d700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc70018ecc0 con 0x7fc700199110 2021-02-10T11:46:30.253 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:30.246 7fc704e6a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc70019aae0 0x7fc70019aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:30.253 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:30.246 7fc704e6a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc70019aae0 0x7fc70019aee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:59684/0 (socket says 172.21.2.112:59684) 2021-02-10T11:46:30.253 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:30.246 7fc704e6a700 1 -- 172.21.2.112:0/1552751773 learned_addr learned my addr 172.21.2.112:0/1552751773 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:46:30.253 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:30.247 7fc6ff7fe700 1 --2- 172.21.2.112:0/1552751773 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc7001b1c50 0x7fc70019a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:30.253 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:30.247 7fc704e6a700 1 -- 172.21.2.112:0/1552751773 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc700199110 msgr2=0x7fc700199510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:30.253 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:30.247 7fc6fffff700 1 --2- 172.21.2.112:0/1552751773 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc700199110 0x7fc700199510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:30.253 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:30.247 7fc704e6a700 1 --2- 172.21.2.112:0/1552751773 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc700199110 0x7fc700199510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:30.254 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:30.247 7fc704e6a700 1 -- 172.21.2.112:0/1552751773 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc7001b1c50 msgr2=0x7fc70019a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:30.254 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:30.247 7fc704e6a700 1 --2- 172.21.2.112:0/1552751773 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc7001b1c50 0x7fc70019a4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:30.254 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:30.247 7fc704e6a700 1 -- 172.21.2.112:0/1552751773 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc700199b30 con 0x7fc70019aae0 2021-02-10T11:46:30.254 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:30.247 7fc6ff7fe700 1 --2- 172.21.2.112:0/1552751773 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc7001b1c50 0x7fc70019a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:46:30.254 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:30.247 7fc704e6a700 1 --2- 172.21.2.112:0/1552751773 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc70019aae0 0x7fc70019aee0 secure :-1 s=READY pgs=2906 cs=0 l=1 rev1=1 rx=0x7fc6f8001c10 tx=0x7fc6f800ac60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:30.254 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:30.247 7fc6fcff9700 1 -- 172.21.2.112:0/1552751773 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc6f8000f90 con 0x7fc70019aae0 2021-02-10T11:46:30.254 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:30.247 7fc6fcff9700 1 -- 172.21.2.112:0/1552751773 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6f80070d0 con 0x7fc70019aae0 2021-02-10T11:46:30.255 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:30.247 7fc70684d700 1 -- 172.21.2.112:0/1552751773 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc700199b30 con 0x7fc70019aae0 2021-02-10T11:46:30.255 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:30.247 7fc70684d700 1 -- 172.21.2.112:0/1552751773 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc700199b30 con 0x7fc70019aae0 2021-02-10T11:46:30.255 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:30.247 7fc6fcff9700 1 -- 172.21.2.112:0/1552751773 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc6f8004720 con 0x7fc70019aae0 2021-02-10T11:46:30.255 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:30.248 7fc6fcff9700 1 -- 172.21.2.112:0/1552751773 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc6f8004720 con 0x7fc70019aae0 2021-02-10T11:46:30.255 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:30.248 7fc6fcff9700 1 --2- 172.21.2.112:0/1552751773 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc6e00186c0 0x7fc6e001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:30.255 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:30.248 7fc6fcff9700 1 -- 172.21.2.112:0/1552751773 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fc6f8032890 con 0x7fc70019aae0 2021-02-10T11:46:30.255 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:30.248 7fc6fffff700 1 --2- 172.21.2.112:0/1552751773 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc6e00186c0 0x7fc6e001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:30.255 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:30.249 7fc6fffff700 1 --2- 172.21.2.112:0/1552751773 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc6e00186c0 0x7fc6e001abc0 crc :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:30.282 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:30.279 7fc70684d700 1 -- 172.21.2.112:0/1552751773 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7000ebc00 con 0x7fc70019aae0 2021-02-10T11:46:30.283 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:30.281 7fc6fcff9700 1 -- 172.21.2.112:0/1552751773 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc6f8000cc0 con 0x7fc70019aae0 2021-02-10T11:46:30.411 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:30.409 7fc70684d700 1 -- 172.21.2.112:0/1552751773 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc7000f0b10 con 0x7fc70019aae0 2021-02-10T11:46:30.412 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:30.409 7fc6fcff9700 1 -- 172.21.2.112:0/1552751773 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 148 v148) v1 ==== 95+0+2111 (secure 0 0 0) 0x7fc6f8004720 con 0x7fc70019aae0 2021-02-10T11:46:30.412 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:46:30.412 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":148,"default_fscid":18,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":12642,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/4212630012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":4212630012},{"type":"v1","addr":"172.21.2.112:6833","nonce":4212630012}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":144}],"filesystems":[{"mdsmap":{"epoch":148,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:46:20.421874","modified":"2021-02-10 11:46:22.427977","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12639},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12639":{"gid":12639,"name":"b","rank":0,"incarnation":145,"state":"up:active","state_seq":3,"addr":"172.21.2.120:6835/1654642147","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":1654642147},{"type":"v1","addr":"172.21.2.120:6835","nonce":1654642147}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[37],"metadata_pool":36,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":18}]} 2021-02-10T11:46:30.412 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 148 2021-02-10T11:46:30.413 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:30.411 7fc70684d700 1 -- 172.21.2.112:0/1552751773 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc6e00186c0 msgr2=0x7fc6e001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:30.414 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:30.411 7fc70684d700 1 --2- 172.21.2.112:0/1552751773 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc6e00186c0 0x7fc6e001abc0 crc :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:30.414 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:30.411 7fc70684d700 1 -- 172.21.2.112:0/1552751773 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc70019aae0 msgr2=0x7fc70019aee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:30.414 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:30.411 7fc70684d700 1 --2- 172.21.2.112:0/1552751773 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc70019aae0 0x7fc70019aee0 secure :-1 s=READY pgs=2906 cs=0 l=1 rev1=1 rx=0x7fc6f8001c10 tx=0x7fc6f800ac60).stop 2021-02-10T11:46:30.414 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:30.411 7fc70684d700 1 -- 172.21.2.112:0/1552751773 shutdown_connections 2021-02-10T11:46:30.414 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:30.411 7fc70684d700 1 --2- 172.21.2.112:0/1552751773 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc6e00186c0 0x7fc6e001abc0 unknown :-1 s=CLOSED pgs=1318 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:30.414 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:30.411 7fc70684d700 1 --2- 172.21.2.112:0/1552751773 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc7001b1c50 0x7fc70019a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:30.414 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:30.411 7fc70684d700 1 --2- 172.21.2.112:0/1552751773 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc70019aae0 0x7fc70019aee0 unknown :-1 s=CLOSED pgs=2906 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:30.414 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:30.411 7fc70684d700 1 --2- 172.21.2.112:0/1552751773 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc700199110 0x7fc700199510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:30.414 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:30.411 7fc70684d700 1 -- 172.21.2.112:0/1552751773 >> 172.21.2.112:0/1552751773 conn(0x7fc70019f880 msgr2=0x7fc700198620 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:30.414 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:30.412 7fc70684d700 1 -- 172.21.2.112:0/1552751773 shutdown_connections 2021-02-10T11:46:30.415 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:30.412 7fc70684d700 1 -- 172.21.2.112:0/1552751773 wait complete. 2021-02-10T11:46:30.624 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 148, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:46:20.421874', 'modified': '2021-02-10 11:46:22.427977', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 12639}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_12639': {'gid': 12639, 'name': 'b', 'rank': 0, 'incarnation': 145, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.120:6835/1654642147', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.120:6834', 'nonce': 1654642147}, {'type': 'v1', 'addr': '172.21.2.120:6835', 'nonce': 1654642147}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [37], 'metadata_pool': 36, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:46:30.624 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:46:30.624 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:46:31.024 INFO:teuthology.orchestra.run.gibba020.stdout:{ 2021-02-10T11:46:31.024 INFO:teuthology.orchestra.run.gibba020.stdout: "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", 2021-02-10T11:46:31.024 INFO:teuthology.orchestra.run.gibba020.stdout: "whoami": 0, 2021-02-10T11:46:31.024 INFO:teuthology.orchestra.run.gibba020.stdout: "id": 12639, 2021-02-10T11:46:31.024 INFO:teuthology.orchestra.run.gibba020.stdout: "want_state": "up:active", 2021-02-10T11:46:31.024 INFO:teuthology.orchestra.run.gibba020.stdout: "state": "up:active", 2021-02-10T11:46:31.024 INFO:teuthology.orchestra.run.gibba020.stdout: "rank_uptime": 10.598835502, 2021-02-10T11:46:31.024 INFO:teuthology.orchestra.run.gibba020.stdout: "mdsmap_epoch": 148, 2021-02-10T11:46:31.024 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch": 172, 2021-02-10T11:46:31.024 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch_barrier": 172, 2021-02-10T11:46:31.025 INFO:teuthology.orchestra.run.gibba020.stdout: "uptime": 16.241222220000001 2021-02-10T11:46:31.025 INFO:teuthology.orchestra.run.gibba020.stdout:} 2021-02-10T11:46:31.028 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", "whoami": 0, "id": 12639, "want_state": "up:active", "state": "up:active", "rank_uptime": 10.598835502, "mdsmap_epoch": 148, "osdmap_epoch": 172, "osdmap_epoch_barrier": 172, "uptime": 16.241222220000001 } 2021-02-10T11:46:31.028 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:46:31.028 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2021-02-10T11:46:31.028 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@gibba012.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2021-02-10T11:46:31.029 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:46:31.029 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:46:31.364 DEBUG:teuthology.orchestra.run.gibba012:> sudo modprobe fuse 2021-02-10T11:46:31.708 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:46:31.708 DEBUG:teuthology.orchestra.run.gibba012:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:46:32.052 INFO:teuthology.orchestra.run.gibba012.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:46:32.052 INFO:teuthology.orchestra.run.gibba012.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:46:32.052 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:46:32.053 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:46:32.053 DEBUG:teuthology.orchestra.run.gibba012:> ls /sys/fs/fuse/connections 2021-02-10T11:46:32.387 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2021-02-10T11:46:32.387 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:46:32.390 DEBUG:teuthology.orchestra.run.gibba012:> sudo modprobe fuse 2021-02-10T11:46:32.736 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:46:32.736 DEBUG:teuthology.orchestra.run.gibba012:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:46:32.803 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba012.stderr:ceph-fuse[183940]: starting ceph client 2021-02-10T11:46:32.803 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba012.stderr:2021-02-10 11:46:32.800 7f3adb8e7f80 -1 init, newargv = 0x55cdb12f51a0 newargc=9 2021-02-10T11:46:32.807 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba012.stderr:ceph-fuse[183940]: starting fuse 2021-02-10T11:46:33.079 INFO:teuthology.orchestra.run.gibba012.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:46:33.079 INFO:teuthology.orchestra.run.gibba012.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:46:33.080 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:46:33.080 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:46:33.081 DEBUG:teuthology.orchestra.run.gibba012:> ls /sys/fs/fuse/connections 2021-02-10T11:46:33.415 INFO:teuthology.orchestra.run.gibba012.stdout:42 2021-02-10T11:46:33.416 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2021-02-10T11:46:33.416 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:46:33.416 DEBUG:teuthology.orchestra.run.gibba012:> sudo python3 -c ' 2021-02-10T11:46:33.416 DEBUG:teuthology.orchestra.run.gibba012:> import glob 2021-02-10T11:46:33.416 DEBUG:teuthology.orchestra.run.gibba012:> import re 2021-02-10T11:46:33.416 DEBUG:teuthology.orchestra.run.gibba012:> import os 2021-02-10T11:46:33.416 DEBUG:teuthology.orchestra.run.gibba012:> import subprocess 2021-02-10T11:46:33.417 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:46:33.417 DEBUG:teuthology.orchestra.run.gibba012:> def find_socket(client_name): 2021-02-10T11:46:33.417 DEBUG:teuthology.orchestra.run.gibba012:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2021-02-10T11:46:33.417 DEBUG:teuthology.orchestra.run.gibba012:> files = glob.glob(asok_path) 2021-02-10T11:46:33.417 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:46:33.417 DEBUG:teuthology.orchestra.run.gibba012:> # Given a non-glob path, it better be there 2021-02-10T11:46:33.417 DEBUG:teuthology.orchestra.run.gibba012:> if "*" not in asok_path: 2021-02-10T11:46:33.417 DEBUG:teuthology.orchestra.run.gibba012:> assert(len(files) == 1) 2021-02-10T11:46:33.417 DEBUG:teuthology.orchestra.run.gibba012:> return files[0] 2021-02-10T11:46:33.417 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:46:33.417 DEBUG:teuthology.orchestra.run.gibba012:> for f in files: 2021-02-10T11:46:33.417 DEBUG:teuthology.orchestra.run.gibba012:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:46:33.417 DEBUG:teuthology.orchestra.run.gibba012:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:46:33.417 DEBUG:teuthology.orchestra.run.gibba012:> return f 2021-02-10T11:46:33.418 DEBUG:teuthology.orchestra.run.gibba012:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:46:33.418 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:46:33.418 DEBUG:teuthology.orchestra.run.gibba012:> print(find_socket("client.0")) 2021-02-10T11:46:33.418 DEBUG:teuthology.orchestra.run.gibba012:> ' 2021-02-10T11:46:33.780 INFO:teuthology.orchestra.run.gibba012.stdout:/var/run/ceph/ceph-client.0.183940.asok 2021-02-10T11:46:33.783 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.183940.asok 2021-02-10T11:46:33.783 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:46:33.783 DEBUG:teuthology.orchestra.run.gibba012:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.183940.asok status 2021-02-10T11:46:34.173 INFO:teuthology.orchestra.run.gibba012.stdout:{ 2021-02-10T11:46:34.174 INFO:teuthology.orchestra.run.gibba012.stdout: "metadata": { 2021-02-10T11:46:34.174 INFO:teuthology.orchestra.run.gibba012.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:46:34.174 INFO:teuthology.orchestra.run.gibba012.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:46:34.174 INFO:teuthology.orchestra.run.gibba012.stdout: "entity_id": "0", 2021-02-10T11:46:34.174 INFO:teuthology.orchestra.run.gibba012.stdout: "hostname": "gibba012", 2021-02-10T11:46:34.174 INFO:teuthology.orchestra.run.gibba012.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2021-02-10T11:46:34.174 INFO:teuthology.orchestra.run.gibba012.stdout: "pid": "183940", 2021-02-10T11:46:34.174 INFO:teuthology.orchestra.run.gibba012.stdout: "root": "/" 2021-02-10T11:46:34.174 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:46:34.174 INFO:teuthology.orchestra.run.gibba012.stdout: "dentry_count": 0, 2021-02-10T11:46:34.174 INFO:teuthology.orchestra.run.gibba012.stdout: "dentry_pinned_count": 0, 2021-02-10T11:46:34.174 INFO:teuthology.orchestra.run.gibba012.stdout: "id": 12810, 2021-02-10T11:46:34.175 INFO:teuthology.orchestra.run.gibba012.stdout: "inst": { 2021-02-10T11:46:34.175 INFO:teuthology.orchestra.run.gibba012.stdout: "name": { 2021-02-10T11:46:34.175 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "client", 2021-02-10T11:46:34.175 INFO:teuthology.orchestra.run.gibba012.stdout: "num": 12810 2021-02-10T11:46:34.175 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:46:34.175 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": { 2021-02-10T11:46:34.175 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:46:34.175 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:0", 2021-02-10T11:46:34.175 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 2262171188 2021-02-10T11:46:34.175 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:46:34.175 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:46:34.175 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": { 2021-02-10T11:46:34.175 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:46:34.176 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:0", 2021-02-10T11:46:34.176 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 2262171188 2021-02-10T11:46:34.176 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:46:34.176 INFO:teuthology.orchestra.run.gibba012.stdout: "inst_str": "client.12810 172.21.2.112:0/2262171188", 2021-02-10T11:46:34.176 INFO:teuthology.orchestra.run.gibba012.stdout: "addr_str": "172.21.2.112:0/2262171188", 2021-02-10T11:46:34.176 INFO:teuthology.orchestra.run.gibba012.stdout: "inode_count": 1, 2021-02-10T11:46:34.176 INFO:teuthology.orchestra.run.gibba012.stdout: "mds_epoch": 148, 2021-02-10T11:46:34.176 INFO:teuthology.orchestra.run.gibba012.stdout: "osd_epoch": 172, 2021-02-10T11:46:34.176 INFO:teuthology.orchestra.run.gibba012.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:46:34.176 INFO:teuthology.orchestra.run.gibba012.stdout: "blacklisted": false 2021-02-10T11:46:34.176 INFO:teuthology.orchestra.run.gibba012.stdout:} 2021-02-10T11:46:34.178 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:46:34.178 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:46:34.178 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:46:34.514 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:46:34.515 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:46:34.515 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:46:34.515 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:46:34.860 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:46:35.289 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.270 7f3d96227700 1 Processor -- start 2021-02-10T11:46:35.289 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.270 7f3d96227700 1 -- start start 2021-02-10T11:46:35.289 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.270 7f3d96227700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f3d900d1870 0x7f3d901b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:35.289 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.270 7f3d96227700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f3d901b0a10 0x7f3d900cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:35.289 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.270 7f3d96227700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d900f0b10 con 0x7f3d900d1c70 2021-02-10T11:46:35.290 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.270 7f3d96227700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d900e5ed0 con 0x7f3d900d1870 2021-02-10T11:46:35.290 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.270 7f3d96227700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d900f0650 con 0x7f3d901b0a10 2021-02-10T11:46:35.290 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.270 7f3d94844700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f3d900d1c70 0x7f3d900cf950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:59164/0 (socket says 172.21.2.112:59164) 2021-02-10T11:46:35.290 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.270 7f3d94844700 1 -- 172.21.2.112:0/1657013434 learned_addr learned my addr 172.21.2.112:0/1657013434 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:46:35.290 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.270 7f3d8ffff700 1 --2- 172.21.2.112:0/1657013434 >> v2:172.21.2.112:3300/0 conn(0x7f3d900d1870 0x7f3d901b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:35.290 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.271 7f3d8effd700 1 -- 172.21.2.112:0/1657013434 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1922996825 0 0) 0x7f3d88001500 con 0x7f3d900d1c70 2021-02-10T11:46:35.290 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.271 7f3d8effd700 1 -- 172.21.2.112:0/1657013434 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d78002fe0 con 0x7f3d900d1c70 2021-02-10T11:46:35.290 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.271 7f3d8ffff700 1 -- 172.21.2.112:0/1657013434 >> v2:172.21.2.120:3300/0 conn(0x7f3d901b0a10 msgr2=0x7f3d900cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:46:35.290 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.271 7f3d8f7fe700 1 --2- 172.21.2.112:0/1657013434 >> v2:172.21.2.120:3300/0 conn(0x7f3d901b0a10 0x7f3d900cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:35.290 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.271 7f3d8ffff700 1 --2- 172.21.2.112:0/1657013434 >> v2:172.21.2.120:3300/0 conn(0x7f3d901b0a10 0x7f3d900cf330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:35.290 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.271 7f3d8ffff700 1 -- 172.21.2.112:0/1657013434 >> v1:172.21.2.112:6789/0 conn(0x7f3d900d1c70 legacy=0x7f3d900cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:35.290 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.271 7f3d8ffff700 1 -- 172.21.2.112:0/1657013434 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d900f0840 con 0x7f3d900d1870 2021-02-10T11:46:35.290 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.271 7f3d8f7fe700 1 --2- 172.21.2.112:0/1657013434 >> v2:172.21.2.120:3300/0 conn(0x7f3d901b0a10 0x7f3d900cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:46:35.291 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.271 7f3d8ffff700 1 --2- 172.21.2.112:0/1657013434 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3d900d1870 0x7f3d901b03f0 secure :-1 s=READY pgs=2910 cs=0 l=1 rev1=1 rx=0x7f3d80006650 tx=0x7f3d80009a80).ready entity=mon.0 client_cookie=7363b627e10ade2b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:35.291 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.271 7f3d8effd700 1 -- 172.21.2.112:0/1657013434 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3d80006390 con 0x7f3d900d1870 2021-02-10T11:46:35.291 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.271 7f3d8effd700 1 -- 172.21.2.112:0/1657013434 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d800009b0 con 0x7f3d900d1870 2021-02-10T11:46:35.291 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.271 7f3d8effd700 1 -- 172.21.2.112:0/1657013434 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3d8000c2b0 con 0x7f3d900d1870 2021-02-10T11:46:35.291 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.271 7f3d96227700 1 -- 172.21.2.112:0/1657013434 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3d900d1870 msgr2=0x7f3d901b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:35.291 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.271 7f3d96227700 1 --2- 172.21.2.112:0/1657013434 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3d900d1870 0x7f3d901b03f0 secure :-1 s=READY pgs=2910 cs=0 l=1 rev1=1 rx=0x7f3d80006650 tx=0x7f3d80009a80).stop 2021-02-10T11:46:35.291 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.272 7f3d96227700 1 -- 172.21.2.112:0/1657013434 shutdown_connections 2021-02-10T11:46:35.291 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.272 7f3d96227700 1 --2- 172.21.2.112:0/1657013434 >> v2:172.21.2.120:3300/0 conn(0x7f3d901b0a10 0x7f3d900cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:35.291 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.272 7f3d96227700 1 --2- 172.21.2.112:0/1657013434 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3d900d1870 0x7f3d901b03f0 unknown :-1 s=CLOSED pgs=2910 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:35.291 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.272 7f3d96227700 1 -- 172.21.2.112:0/1657013434 >> 172.21.2.112:0/1657013434 conn(0x7f3d900ce450 msgr2=0x7f3d900d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:35.291 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.272 7f3d96227700 1 -- 172.21.2.112:0/1657013434 shutdown_connections 2021-02-10T11:46:35.291 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.272 7f3d96227700 1 -- 172.21.2.112:0/1657013434 wait complete. 2021-02-10T11:46:35.291 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.287 7f3d96227700 1 Processor -- start 2021-02-10T11:46:35.291 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.287 7f3d96227700 1 -- start start 2021-02-10T11:46:35.293 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.287 7f3d96227700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3d901942b0 0x7f3d900b3550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:35.293 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.287 7f3d96227700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3d90195070 0x7f3d90195470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:35.293 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.287 7f3d96227700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3d900bb8e0 0x7f3d900bbce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:35.293 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.287 7f3d96227700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d901946b0 con 0x7f3d90195070 2021-02-10T11:46:35.293 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.287 7f3d96227700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d900b3b70 con 0x7f3d900bb8e0 2021-02-10T11:46:35.293 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.287 7f3d96227700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d900b3d90 con 0x7f3d901942b0 2021-02-10T11:46:35.293 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.288 7f3d8f7fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3d90195070 0x7f3d90195470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:35.293 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.288 7f3d8f7fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3d90195070 0x7f3d90195470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:59708/0 (socket says 172.21.2.112:59708) 2021-02-10T11:46:35.293 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.288 7f3d8f7fe700 1 -- 172.21.2.112:0/216023800 learned_addr learned my addr 172.21.2.112:0/216023800 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:46:35.293 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.288 7f3d94844700 1 --2- 172.21.2.112:0/216023800 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3d900bb8e0 0x7f3d900bbce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:35.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.288 7f3d8ffff700 1 --2- 172.21.2.112:0/216023800 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3d901942b0 0x7f3d900b3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:35.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.288 7f3d8f7fe700 1 -- 172.21.2.112:0/216023800 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3d901942b0 msgr2=0x7f3d900b3550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:35.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.288 7f3d8f7fe700 1 --2- 172.21.2.112:0/216023800 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3d901942b0 0x7f3d900b3550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:35.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.288 7f3d8f7fe700 1 -- 172.21.2.112:0/216023800 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3d900bb8e0 msgr2=0x7f3d900bbce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:35.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.288 7f3d8f7fe700 1 --2- 172.21.2.112:0/216023800 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3d900bb8e0 0x7f3d900bbce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:35.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.288 7f3d8f7fe700 1 -- 172.21.2.112:0/216023800 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d901a49e0 con 0x7f3d90195070 2021-02-10T11:46:35.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.288 7f3d8f7fe700 1 --2- 172.21.2.112:0/216023800 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3d90195070 0x7f3d90195470 secure :-1 s=READY pgs=2911 cs=0 l=1 rev1=1 rx=0x7f3d84006f20 tx=0x7f3d8400b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:35.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.288 7f3d8cff9700 1 -- 172.21.2.112:0/216023800 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3d8400bd80 con 0x7f3d90195070 2021-02-10T11:46:35.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.288 7f3d8cff9700 1 -- 172.21.2.112:0/216023800 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d84000c90 con 0x7f3d90195070 2021-02-10T11:46:35.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.289 7f3d8cff9700 1 -- 172.21.2.112:0/216023800 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3d840022b0 con 0x7f3d90195070 2021-02-10T11:46:35.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.289 7f3d96227700 1 -- 172.21.2.112:0/216023800 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d901a49e0 con 0x7f3d90195070 2021-02-10T11:46:35.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.289 7f3d96227700 1 -- 172.21.2.112:0/216023800 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d900bc5f0 con 0x7f3d90195070 2021-02-10T11:46:35.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.289 7f3d8cff9700 1 -- 172.21.2.112:0/216023800 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3d8400a000 con 0x7f3d90195070 2021-02-10T11:46:35.295 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.289 7f3d8cff9700 1 --2- 172.21.2.112:0/216023800 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3d700186d0 0x7f3d7001abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:35.295 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.289 7f3d8cff9700 1 -- 172.21.2.112:0/216023800 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f3d84032610 con 0x7f3d90195070 2021-02-10T11:46:35.295 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.290 7f3d8ffff700 1 --2- 172.21.2.112:0/216023800 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3d700186d0 0x7f3d7001abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:35.295 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.290 7f3d8ffff700 1 --2- 172.21.2.112:0/216023800 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3d700186d0 0x7f3d7001abd0 crc :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:35.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.320 7f3d96227700 1 -- 172.21.2.112:0/216023800 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d900bc5f0 con 0x7f3d90195070 2021-02-10T11:46:35.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.323 7f3d8cff9700 1 -- 172.21.2.112:0/216023800 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3d84000940 con 0x7f3d90195070 2021-02-10T11:46:35.449 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.447 7f3d96227700 1 -- 172.21.2.112:0/216023800 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3d900bc5f0 con 0x7f3d90195070 2021-02-10T11:46:35.450 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.448 7f3d8cff9700 1 -- 172.21.2.112:0/216023800 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v172) v1 ==== 74+0+14007 (secure 0 0 0) 0x7f3d84000c90 con 0x7f3d90195070 2021-02-10T11:46:35.451 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:46:35.451 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":172,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:46:21.423567","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":37,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":36,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:46:18.395022","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"171","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":37,"pool_name":"cephfs_data","create_time":"2021-02-10 11:46:19.415292","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":168,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:46:35.452 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.449 7f3d96227700 1 -- 172.21.2.112:0/216023800 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3d700186d0 msgr2=0x7f3d7001abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:35.452 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.449 7f3d96227700 1 --2- 172.21.2.112:0/216023800 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3d700186d0 0x7f3d7001abd0 crc :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:35.452 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.449 7f3d96227700 1 -- 172.21.2.112:0/216023800 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3d90195070 msgr2=0x7f3d90195470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:35.452 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.449 7f3d96227700 1 --2- 172.21.2.112:0/216023800 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3d90195070 0x7f3d90195470 secure :-1 s=READY pgs=2911 cs=0 l=1 rev1=1 rx=0x7f3d84006f20 tx=0x7f3d8400b990).stop 2021-02-10T11:46:35.452 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.449 7f3d96227700 1 -- 172.21.2.112:0/216023800 shutdown_connections 2021-02-10T11:46:35.452 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.449 7f3d96227700 1 --2- 172.21.2.112:0/216023800 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3d700186d0 0x7f3d7001abd0 unknown :-1 s=CLOSED pgs=1319 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:35.453 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.449 7f3d96227700 1 --2- 172.21.2.112:0/216023800 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3d900bb8e0 0x7f3d900bbce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:35.453 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.449 7f3d96227700 1 --2- 172.21.2.112:0/216023800 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3d90195070 0x7f3d90195470 unknown :-1 s=CLOSED pgs=2911 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:35.453 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.449 7f3d96227700 1 --2- 172.21.2.112:0/216023800 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3d901942b0 0x7f3d900b3550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:35.453 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.449 7f3d96227700 1 -- 172.21.2.112:0/216023800 >> 172.21.2.112:0/216023800 conn(0x7f3d900b3040 msgr2=0x7f3d900c3250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:35.453 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.450 7f3d96227700 1 -- 172.21.2.112:0/216023800 shutdown_connections 2021-02-10T11:46:35.453 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.450 7f3d96227700 1 -- 172.21.2.112:0/216023800 wait complete. 2021-02-10T11:46:35.460 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:13:57.508758', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 36, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:46:18.395022', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '171', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 37, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:46:19.415292', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '172', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:46:35.461 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:46:35.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.874 7f9894b1a700 1 Processor -- start 2021-02-10T11:46:35.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.874 7f9894b1a700 1 -- start start 2021-02-10T11:46:35.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.875 7f9894b1a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f98900d1c70 0x7f98900cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:35.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.875 7f9894b1a700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f98900cf580 0x7f98900cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:35.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.875 7f9894b1a700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98900fe1c0 con 0x7f98900d1870 2021-02-10T11:46:35.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.875 7f9894b1a700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98901b0cb0 con 0x7f98900cf580 2021-02-10T11:46:35.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.875 7f9894b1a700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9890109a60 con 0x7f98900d1c70 2021-02-10T11:46:35.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.875 7f988e85e700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f98900d1870 0x7f98901b04c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:59178/0 (socket says 172.21.2.112:59178) 2021-02-10T11:46:35.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.875 7f988e85e700 1 -- 172.21.2.112:0/2829594593 learned_addr learned my addr 172.21.2.112:0/2829594593 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:46:35.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.875 7f988d85c700 1 -- 172.21.2.112:0/2829594593 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1845327997 0 0) 0x7f9880001180 con 0x7f98900d1870 2021-02-10T11:46:35.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.875 7f988d85c700 1 -- 172.21.2.112:0/2829594593 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f987c002fe0 con 0x7f98900d1870 2021-02-10T11:46:35.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.875 7f988f05f700 1 --2- 172.21.2.112:0/2829594593 >> v2:172.21.2.120:3300/0 conn(0x7f98900cf580 0x7f98900cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:35.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.876 7f988e05d700 1 --2- 172.21.2.112:0/2829594593 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f98900d1c70 0x7f98900cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:35.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.876 7f988d85c700 1 -- 172.21.2.112:0/2829594593 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1535736111 0 0) 0x7f9880001180 con 0x7f98900d1870 2021-02-10T11:46:35.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.876 7f988d85c700 1 -- 172.21.2.112:0/2829594593 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f98900d1c70 msgr2=0x7f98900cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:46:35.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.876 7f988d85c700 1 --2- 172.21.2.112:0/2829594593 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f98900d1c70 0x7f98900cef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:35.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.876 7f988d85c700 1 -- 172.21.2.112:0/2829594593 >> v2:172.21.2.120:3300/0 conn(0x7f98900cf580 msgr2=0x7f98900cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:46:35.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.876 7f988d85c700 1 --2- 172.21.2.112:0/2829594593 >> v2:172.21.2.120:3300/0 conn(0x7f98900cf580 0x7f98900cf980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:35.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.876 7f988d85c700 1 -- 172.21.2.112:0/2829594593 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9890088070 con 0x7f98900d1870 2021-02-10T11:46:35.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.876 7f988d85c700 1 -- 172.21.2.112:0/2829594593 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f9880001680 con 0x7f98900d1870 2021-02-10T11:46:35.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.876 7f988d85c700 1 -- 172.21.2.112:0/2829594593 >> v1:172.21.2.112:6789/0 conn(0x7f98900d1870 legacy=0x7f98901b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:35.897 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.876 7f988d85c700 1 --2- 172.21.2.112:0/2829594593 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f987c003a60 0x7f987c005f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:35.897 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.876 7f988d85c700 1 --2- 172.21.2.112:0/2829594593 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f987c04a7a0 0x7f987c04cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:35.897 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.876 7f988d85c700 1 --2- 172.21.2.112:0/2829594593 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f987c04d500 0x7f987c04fa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:35.897 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.876 7f988d85c700 1 -- 172.21.2.112:0/2829594593 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f987c0502d0 con 0x7f987c04a7a0 2021-02-10T11:46:35.897 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.876 7f988d85c700 1 -- 172.21.2.112:0/2829594593 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f987c0504f0 con 0x7f987c04d500 2021-02-10T11:46:35.897 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.876 7f988d85c700 1 -- 172.21.2.112:0/2829594593 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f987c050710 con 0x7f987c003a60 2021-02-10T11:46:35.897 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.876 7f988e85e700 1 --2- 172.21.2.112:0/2829594593 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f987c003a60 0x7f987c005f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:35.897 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.876 7f988e05d700 1 --2- 172.21.2.112:0/2829594593 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f987c04a7a0 0x7f987c04cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:35.897 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.876 7f988f05f700 1 --2- 172.21.2.112:0/2829594593 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f987c04d500 0x7f987c04fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:35.897 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.877 7f988e05d700 1 -- 172.21.2.112:0/2829594593 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f987c003a60 msgr2=0x7f987c005f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:46:35.897 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.877 7f988e05d700 1 --2- 172.21.2.112:0/2829594593 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f987c003a60 0x7f987c005f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:35.897 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.877 7f988e05d700 1 -- 172.21.2.112:0/2829594593 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f987c04d500 msgr2=0x7f987c04fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:46:35.897 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.877 7f988e05d700 1 --2- 172.21.2.112:0/2829594593 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f987c04d500 0x7f987c04fa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:35.897 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.877 7f988e05d700 1 -- 172.21.2.112:0/2829594593 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f987c0509b0 con 0x7f987c04a7a0 2021-02-10T11:46:35.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.877 7f988e05d700 1 --2- 172.21.2.112:0/2829594593 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f987c04a7a0 0x7f987c04cc60 secure :-1 s=READY pgs=2913 cs=0 l=1 rev1=1 rx=0x7f9878009310 tx=0x7f9878009a90).ready entity=mon.0 client_cookie=c79a5f6ab1767184 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:35.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.877 7f988d85c700 1 -- 172.21.2.112:0/2829594593 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f987800c0c0 con 0x7f987c04a7a0 2021-02-10T11:46:35.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.877 7f988d85c700 1 -- 172.21.2.112:0/2829594593 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9878006400 con 0x7f987c04a7a0 2021-02-10T11:46:35.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.877 7f988d85c700 1 -- 172.21.2.112:0/2829594593 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f987800c340 con 0x7f987c04a7a0 2021-02-10T11:46:35.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.877 7f9894b1a700 1 -- 172.21.2.112:0/2829594593 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f987c04a7a0 msgr2=0x7f987c04cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:35.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.877 7f9894b1a700 1 --2- 172.21.2.112:0/2829594593 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f987c04a7a0 0x7f987c04cc60 secure :-1 s=READY pgs=2913 cs=0 l=1 rev1=1 rx=0x7f9878009310 tx=0x7f9878009a90).stop 2021-02-10T11:46:35.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.877 7f988f05f700 1 -- 172.21.2.112:0/2829594593 reap_dead start 2021-02-10T11:46:35.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.878 7f9894b1a700 1 -- 172.21.2.112:0/2829594593 shutdown_connections 2021-02-10T11:46:35.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.878 7f9894b1a700 1 -- 172.21.2.112:0/2829594593 >> 172.21.2.112:0/2829594593 conn(0x7f98900ce420 msgr2=0x7f98900d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:35.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.878 7f9894b1a700 1 -- 172.21.2.112:0/2829594593 shutdown_connections 2021-02-10T11:46:35.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.878 7f9894b1a700 1 -- 172.21.2.112:0/2829594593 wait complete. 2021-02-10T11:46:35.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.892 7f9894b1a700 1 Processor -- start 2021-02-10T11:46:35.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.893 7f9894b1a700 1 -- start start 2021-02-10T11:46:35.900 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.893 7f9894b1a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f989009bcc0 0x7f989009c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:35.900 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.893 7f9894b1a700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f98900a0820 0x7f98900a0c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:35.900 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.893 7f9894b1a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f98900a1240 0x7f98900858d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:35.900 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.893 7f9894b1a700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98900a66b0 con 0x7f98900a1240 2021-02-10T11:46:35.900 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.893 7f9894b1a700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98901a49b0 con 0x7f98900a0820 2021-02-10T11:46:35.900 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.893 7f9894b1a700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98901a4b90 con 0x7f989009bcc0 2021-02-10T11:46:35.900 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.893 7f988f05f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f98900a1240 0x7f98900858d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:35.900 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.893 7f988f05f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f98900a1240 0x7f98900858d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:59730/0 (socket says 172.21.2.112:59730) 2021-02-10T11:46:35.900 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.893 7f988f05f700 1 -- 172.21.2.112:0/76029001 learned_addr learned my addr 172.21.2.112:0/76029001 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:46:35.900 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.893 7f988e85e700 1 --2- 172.21.2.112:0/76029001 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f989009bcc0 0x7f989009c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:35.901 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.893 7f988e05d700 1 --2- 172.21.2.112:0/76029001 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f98900a0820 0x7f98900a0c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:35.901 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.894 7f988f05f700 1 -- 172.21.2.112:0/76029001 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f989009bcc0 msgr2=0x7f989009c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:35.901 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.894 7f988f05f700 1 --2- 172.21.2.112:0/76029001 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f989009bcc0 0x7f989009c0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:35.901 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.894 7f988f05f700 1 -- 172.21.2.112:0/76029001 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f98900a0820 msgr2=0x7f98900a0c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:35.901 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.894 7f988f05f700 1 --2- 172.21.2.112:0/76029001 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f98900a0820 0x7f98900a0c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:35.901 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.894 7f988f05f700 1 -- 172.21.2.112:0/76029001 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98900a1640 con 0x7f98900a1240 2021-02-10T11:46:35.901 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.894 7f988f05f700 1 --2- 172.21.2.112:0/76029001 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f98900a1240 0x7f98900858d0 secure :-1 s=READY pgs=2914 cs=0 l=1 rev1=1 rx=0x7f9888006fa0 tx=0x7f988800bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:35.901 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.894 7f98877fe700 1 -- 172.21.2.112:0/76029001 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9888000df0 con 0x7f98900a1240 2021-02-10T11:46:35.901 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.894 7f98877fe700 1 -- 172.21.2.112:0/76029001 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f988800c2f0 con 0x7f98900a1240 2021-02-10T11:46:35.901 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.894 7f98877fe700 1 -- 172.21.2.112:0/76029001 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f988800c750 con 0x7f98900a1240 2021-02-10T11:46:35.901 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.894 7f9894b1a700 1 -- 172.21.2.112:0/76029001 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98900a1640 con 0x7f98900a1240 2021-02-10T11:46:35.901 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.894 7f9894b1a700 1 -- 172.21.2.112:0/76029001 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98900861e0 con 0x7f98900a1240 2021-02-10T11:46:35.901 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.895 7f98877fe700 1 -- 172.21.2.112:0/76029001 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9888000df0 con 0x7f98900a1240 2021-02-10T11:46:35.901 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.895 7f98877fe700 1 --2- 172.21.2.112:0/76029001 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f98740186c0 0x7f987401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:35.902 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.895 7f98877fe700 1 -- 172.21.2.112:0/76029001 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f9888032970 con 0x7f98900a1240 2021-02-10T11:46:35.902 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.895 7f988e85e700 1 --2- 172.21.2.112:0/76029001 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f98740186c0 0x7f987401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:35.902 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.896 7f988e85e700 1 --2- 172.21.2.112:0/76029001 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f98740186c0 0x7f987401abc0 crc :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:35.927 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.925 7f9894b1a700 1 -- 172.21.2.112:0/76029001 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9890086a70 con 0x7f98900a1240 2021-02-10T11:46:35.929 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:35.927 7f98877fe700 1 -- 172.21.2.112:0/76029001 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9888022000 con 0x7f98900a1240 2021-02-10T11:46:36.059 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:36.056 7f9894b1a700 1 -- 172.21.2.112:0/76029001 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f9890086a70 con 0x7f98900a1240 2021-02-10T11:46:36.059 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:36.057 7f98877fe700 1 -- 172.21.2.112:0/76029001 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v172) v1 ==== 92+0+10 (secure 0 0 0) 0x7f9888022000 con 0x7f98900a1240 2021-02-10T11:46:36.059 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:46:36.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:36.059 7f9894b1a700 1 -- 172.21.2.112:0/76029001 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f98740186c0 msgr2=0x7f987401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:36.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:36.059 7f9894b1a700 1 --2- 172.21.2.112:0/76029001 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f98740186c0 0x7f987401abc0 crc :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:36.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:36.059 7f9894b1a700 1 -- 172.21.2.112:0/76029001 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f98900a1240 msgr2=0x7f98900858d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:36.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:36.059 7f9894b1a700 1 --2- 172.21.2.112:0/76029001 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f98900a1240 0x7f98900858d0 secure :-1 s=READY pgs=2914 cs=0 l=1 rev1=1 rx=0x7f9888006fa0 tx=0x7f988800bcf0).stop 2021-02-10T11:46:36.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:36.059 7f9894b1a700 1 -- 172.21.2.112:0/76029001 shutdown_connections 2021-02-10T11:46:36.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:36.059 7f9894b1a700 1 --2- 172.21.2.112:0/76029001 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f98740186c0 0x7f987401abc0 unknown :-1 s=CLOSED pgs=1320 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:36.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:36.059 7f9894b1a700 1 --2- 172.21.2.112:0/76029001 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f98900a0820 0x7f98900a0c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:36.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:36.059 7f9894b1a700 1 --2- 172.21.2.112:0/76029001 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f98900a1240 0x7f98900858d0 unknown :-1 s=CLOSED pgs=2914 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:36.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:36.059 7f9894b1a700 1 --2- 172.21.2.112:0/76029001 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f989009bcc0 0x7f989009c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:36.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:36.059 7f9894b1a700 1 -- 172.21.2.112:0/76029001 >> 172.21.2.112:0/76029001 conn(0x7f98900bef00 msgr2=0x7f989009d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:36.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:36.059 7f9894b1a700 1 -- 172.21.2.112:0/76029001 shutdown_connections 2021-02-10T11:46:36.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:36.059 7f9894b1a700 1 -- 172.21.2.112:0/76029001 wait complete. 2021-02-10T11:46:36.070 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:46:36.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:36.482 7f2915399700 1 Processor -- start 2021-02-10T11:46:36.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:36.483 7f2915399700 1 -- start start 2021-02-10T11:46:36.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:36.483 7f2915399700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f29100d1870 0x7f29101b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:36.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:36.483 7f2915399700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f29101b0ae0 0x7f2910108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:36.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:36.483 7f2915399700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f29100d1c70 0x7f2910108f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:36.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:36.483 7f2915399700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29100fe1c0 con 0x7f29100d1870 2021-02-10T11:46:36.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:36.483 7f2915399700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2910109a60 con 0x7f29101b0ae0 2021-02-10T11:46:36.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:36.483 7f2915399700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2910088070 con 0x7f29100d1c70 2021-02-10T11:46:36.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:36.483 7f290f07d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f29100d1870 0x7f29101b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:36.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:36.483 7f290f07d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f29100d1870 0x7f29101b04c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:59736/0 (socket says 172.21.2.112:59736) 2021-02-10T11:46:36.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:36.483 7f290f07d700 1 -- 172.21.2.112:0/2468606964 learned_addr learned my addr 172.21.2.112:0/2468606964 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:46:36.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:36.484 7f290f07d700 1 -- 172.21.2.112:0/2468606964 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f29100d1c70 msgr2=0x7f2910108f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:46:36.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:36.484 7f290f07d700 1 --2- 172.21.2.112:0/2468606964 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f29100d1c70 0x7f2910108f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:36.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:36.484 7f290f07d700 1 -- 172.21.2.112:0/2468606964 >> v2:172.21.2.120:3300/0 conn(0x7f29101b0ae0 msgr2=0x7f2910108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:46:36.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:36.484 7f290f07d700 1 --2- 172.21.2.112:0/2468606964 >> v2:172.21.2.120:3300/0 conn(0x7f29101b0ae0 0x7f2910108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:36.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:36.484 7f290f07d700 1 -- 172.21.2.112:0/2468606964 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2910109580 con 0x7f29100d1870 2021-02-10T11:46:36.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:36.484 7f290f07d700 1 --2- 172.21.2.112:0/2468606964 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f29100d1870 0x7f29101b04c0 secure :-1 s=READY pgs=2915 cs=0 l=1 rev1=1 rx=0x7f2900006700 tx=0x7f2900009b80).ready entity=mon.0 client_cookie=c265f7e55f572f16 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:36.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:36.484 7f290e07b700 1 -- 172.21.2.112:0/2468606964 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f290000c0c0 con 0x7f29100d1870 2021-02-10T11:46:36.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:36.484 7f290e07b700 1 -- 172.21.2.112:0/2468606964 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f290000c400 con 0x7f29100d1870 2021-02-10T11:46:36.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:36.484 7f290e07b700 1 -- 172.21.2.112:0/2468606964 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f290000c920 con 0x7f29100d1870 2021-02-10T11:46:36.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:36.484 7f2915399700 1 -- 172.21.2.112:0/2468606964 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f29100d1870 msgr2=0x7f29101b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:36.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:36.484 7f2915399700 1 --2- 172.21.2.112:0/2468606964 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f29100d1870 0x7f29101b04c0 secure :-1 s=READY pgs=2915 cs=0 l=1 rev1=1 rx=0x7f2900006700 tx=0x7f2900009b80).stop 2021-02-10T11:46:36.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:36.485 7f2915399700 1 -- 172.21.2.112:0/2468606964 shutdown_connections 2021-02-10T11:46:36.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:36.485 7f2915399700 1 --2- 172.21.2.112:0/2468606964 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f29100d1c70 0x7f2910108f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:36.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:36.485 7f2915399700 1 --2- 172.21.2.112:0/2468606964 >> v2:172.21.2.120:3300/0 conn(0x7f29101b0ae0 0x7f2910108940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:36.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:36.485 7f2915399700 1 --2- 172.21.2.112:0/2468606964 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f29100d1870 0x7f29101b04c0 unknown :-1 s=CLOSED pgs=2915 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:36.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:36.485 7f2915399700 1 -- 172.21.2.112:0/2468606964 >> 172.21.2.112:0/2468606964 conn(0x7f29100ce420 msgr2=0x7f29100d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:36.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:36.485 7f2915399700 1 -- 172.21.2.112:0/2468606964 shutdown_connections 2021-02-10T11:46:36.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:36.485 7f2915399700 1 -- 172.21.2.112:0/2468606964 wait complete. 2021-02-10T11:46:36.505 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:36.500 7f2915399700 1 Processor -- start 2021-02-10T11:46:36.505 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:36.500 7f2915399700 1 -- start start 2021-02-10T11:46:36.505 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:36.500 7f2915399700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f29100c2370 0x7f291018f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:36.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:36.500 7f2915399700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2910084cb0 0x7f29101a7470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:36.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:36.500 7f2915399700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f29101a7a90 0x7f2910085700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:36.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:36.500 7f2915399700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29100be840 con 0x7f2910084cb0 2021-02-10T11:46:36.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:36.500 7f2915399700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29100c29a0 con 0x7f29100c2370 2021-02-10T11:46:36.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:36.500 7f2915399700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29101a7e90 con 0x7f29101a7a90 2021-02-10T11:46:36.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:36.501 7f290e87c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2910084cb0 0x7f29101a7470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:36.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:36.501 7f290e87c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2910084cb0 0x7f29101a7470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:59740/0 (socket says 172.21.2.112:59740) 2021-02-10T11:46:36.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:36.501 7f290e87c700 1 -- 172.21.2.112:0/1777361149 learned_addr learned my addr 172.21.2.112:0/1777361149 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:46:36.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:36.501 7f290f07d700 1 --2- 172.21.2.112:0/1777361149 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f29100c2370 0x7f291018f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:36.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:36.501 7f290f87e700 1 --2- 172.21.2.112:0/1777361149 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f29101a7a90 0x7f2910085700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:36.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:36.501 7f290e87c700 1 -- 172.21.2.112:0/1777361149 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f29101a7a90 msgr2=0x7f2910085700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:36.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:36.501 7f290e87c700 1 --2- 172.21.2.112:0/1777361149 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f29101a7a90 0x7f2910085700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:36.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:36.501 7f290e87c700 1 -- 172.21.2.112:0/1777361149 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f29100c2370 msgr2=0x7f291018f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:36.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:36.501 7f290e87c700 1 --2- 172.21.2.112:0/1777361149 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f29100c2370 0x7f291018f870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:36.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:36.501 7f290e87c700 1 -- 172.21.2.112:0/1777361149 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29101a80b0 con 0x7f2910084cb0 2021-02-10T11:46:36.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:36.501 7f290e87c700 1 --2- 172.21.2.112:0/1777361149 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2910084cb0 0x7f29101a7470 secure :-1 s=READY pgs=2916 cs=0 l=1 rev1=1 rx=0x7f28f8006f00 tx=0x7f28f800b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:36.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:36.501 7f2907fff700 1 -- 172.21.2.112:0/1777361149 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f28f8006350 con 0x7f2910084cb0 2021-02-10T11:46:36.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:36.502 7f2907fff700 1 -- 172.21.2.112:0/1777361149 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28f800bd30 con 0x7f2910084cb0 2021-02-10T11:46:36.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:36.502 7f2907fff700 1 -- 172.21.2.112:0/1777361149 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f28f8002020 con 0x7f2910084cb0 2021-02-10T11:46:36.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:36.502 7f2915399700 1 -- 172.21.2.112:0/1777361149 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29101a80b0 con 0x7f2910084cb0 2021-02-10T11:46:36.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:36.502 7f2915399700 1 -- 172.21.2.112:0/1777361149 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29101a80b0 con 0x7f2910084cb0 2021-02-10T11:46:36.508 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:36.504 7f2907fff700 1 -- 172.21.2.112:0/1777361149 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f28f8005310 con 0x7f2910084cb0 2021-02-10T11:46:36.508 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:36.504 7f2907fff700 1 --2- 172.21.2.112:0/1777361149 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f28f40186c0 0x7f28f401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:36.508 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:36.504 7f2907fff700 1 -- 172.21.2.112:0/1777361149 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f28f80323a0 con 0x7f2910084cb0 2021-02-10T11:46:36.508 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:36.504 7f290f07d700 1 --2- 172.21.2.112:0/1777361149 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f28f40186c0 0x7f28f401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:36.508 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:36.505 7f290f07d700 1 --2- 172.21.2.112:0/1777361149 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f28f40186c0 0x7f28f401abc0 crc :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:36.535 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:36.533 7f2915399700 1 -- 172.21.2.112:0/1777361149 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29100be840 con 0x7f2910084cb0 2021-02-10T11:46:36.538 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:36.536 7f2907fff700 1 -- 172.21.2.112:0/1777361149 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f28f8006b90 con 0x7f2910084cb0 2021-02-10T11:46:36.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:36.665 7f2915399700 1 -- 172.21.2.112:0/1777361149 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f29100be840 con 0x7f2910084cb0 2021-02-10T11:46:36.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:36.665 7f2907fff700 1 -- 172.21.2.112:0/1777361149 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v172) v1 ==== 104+0+10 (secure 0 0 0) 0x7f28f800bd30 con 0x7f2910084cb0 2021-02-10T11:46:36.668 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:46:36.669 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:36.667 7f2915399700 1 -- 172.21.2.112:0/1777361149 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f28f40186c0 msgr2=0x7f28f401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:36.670 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:36.667 7f2915399700 1 --2- 172.21.2.112:0/1777361149 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f28f40186c0 0x7f28f401abc0 crc :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:36.670 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:36.667 7f2915399700 1 -- 172.21.2.112:0/1777361149 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2910084cb0 msgr2=0x7f29101a7470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:36.670 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:36.667 7f2915399700 1 --2- 172.21.2.112:0/1777361149 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2910084cb0 0x7f29101a7470 secure :-1 s=READY pgs=2916 cs=0 l=1 rev1=1 rx=0x7f28f8006f00 tx=0x7f28f800b940).stop 2021-02-10T11:46:36.670 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:36.667 7f2915399700 1 -- 172.21.2.112:0/1777361149 shutdown_connections 2021-02-10T11:46:36.670 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:36.667 7f2915399700 1 --2- 172.21.2.112:0/1777361149 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f28f40186c0 0x7f28f401abc0 unknown :-1 s=CLOSED pgs=1321 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:36.670 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:36.667 7f2915399700 1 --2- 172.21.2.112:0/1777361149 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f29101a7a90 0x7f2910085700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:36.670 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:36.667 7f2915399700 1 --2- 172.21.2.112:0/1777361149 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2910084cb0 0x7f29101a7470 unknown :-1 s=CLOSED pgs=2916 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:36.670 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:36.667 7f2915399700 1 --2- 172.21.2.112:0/1777361149 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f29100c2370 0x7f291018f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:36.670 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:36.667 7f2915399700 1 -- 172.21.2.112:0/1777361149 >> 172.21.2.112:0/1777361149 conn(0x7f29100b5dc0 msgr2=0x7f29100c2d90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:36.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:36.668 7f2915399700 1 -- 172.21.2.112:0/1777361149 shutdown_connections 2021-02-10T11:46:36.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:36.668 7f2915399700 1 -- 172.21.2.112:0/1777361149 wait complete. 2021-02-10T11:46:36.678 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:46:37.110 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.090 7efde6aca700 1 Processor -- start 2021-02-10T11:46:37.110 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.091 7efde6aca700 1 -- start start 2021-02-10T11:46:37.110 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.091 7efde6aca700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efde00d1c60 0x7efde00cef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:37.110 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.091 7efde6aca700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7efde00cf570 0x7efde00cf970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:37.110 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.091 7efde6aca700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efde00fe1b0 con 0x7efde00d1860 2021-02-10T11:46:37.110 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.091 7efde6aca700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efde01b0ca0 con 0x7efde00cf570 2021-02-10T11:46:37.110 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.091 7efde6aca700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efde0109a50 con 0x7efde00d1c60 2021-02-10T11:46:37.111 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.091 7efde50e7700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7efde00cf570 0x7efde00cf970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:37.111 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.091 7efde48e6700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7efde00d1860 0x7efde01b04b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:59208/0 (socket says 172.21.2.112:59208) 2021-02-10T11:46:37.111 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.091 7efde48e6700 1 -- 172.21.2.112:0/2604132619 learned_addr learned my addr 172.21.2.112:0/2604132619 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:46:37.111 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.092 7efddffff700 1 --2- 172.21.2.112:0/2604132619 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efde00d1c60 0x7efde00cef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:37.111 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.092 7efde50e7700 1 -- 172.21.2.112:0/2604132619 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efde00d1c60 msgr2=0x7efde00cef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:46:37.111 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.092 7efde50e7700 1 --2- 172.21.2.112:0/2604132619 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efde00d1c60 0x7efde00cef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:37.111 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.092 7efde50e7700 1 -- 172.21.2.112:0/2604132619 >> v1:172.21.2.112:6789/0 conn(0x7efde00d1860 legacy=0x7efde01b04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:37.111 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.092 7efde50e7700 1 -- 172.21.2.112:0/2604132619 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efde0088070 con 0x7efde00cf570 2021-02-10T11:46:37.111 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.092 7efddffff700 1 --2- 172.21.2.112:0/2604132619 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efde00d1c60 0x7efde00cef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:46:37.111 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.092 7efde50e7700 1 --2- 172.21.2.112:0/2604132619 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efde00cf570 0x7efde00cf970 secure :-1 s=READY pgs=2918 cs=0 l=1 rev1=1 rx=0x7efdd8006e30 tx=0x7efdd8009e60).ready entity=mon.0 client_cookie=b48462e56ce2cd32 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:37.111 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.092 7efddf7fe700 1 -- 172.21.2.112:0/2604132619 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efdd8006a50 con 0x7efde00cf570 2021-02-10T11:46:37.111 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.092 7efddf7fe700 1 -- 172.21.2.112:0/2604132619 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdd800b300 con 0x7efde00cf570 2021-02-10T11:46:37.112 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.092 7efddf7fe700 1 -- 172.21.2.112:0/2604132619 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efdd800b820 con 0x7efde00cf570 2021-02-10T11:46:37.112 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.092 7efde6aca700 1 -- 172.21.2.112:0/2604132619 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efde00cf570 msgr2=0x7efde00cf970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:37.112 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.092 7efde6aca700 1 --2- 172.21.2.112:0/2604132619 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efde00cf570 0x7efde00cf970 secure :-1 s=READY pgs=2918 cs=0 l=1 rev1=1 rx=0x7efdd8006e30 tx=0x7efdd8009e60).stop 2021-02-10T11:46:37.112 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.093 7efde6aca700 1 -- 172.21.2.112:0/2604132619 shutdown_connections 2021-02-10T11:46:37.112 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.093 7efde6aca700 1 --2- 172.21.2.112:0/2604132619 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efde00cf570 0x7efde00cf970 unknown :-1 s=CLOSED pgs=2918 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:37.112 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.093 7efde6aca700 1 --2- 172.21.2.112:0/2604132619 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efde00d1c60 0x7efde00cef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:37.112 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.093 7efde6aca700 1 -- 172.21.2.112:0/2604132619 >> 172.21.2.112:0/2604132619 conn(0x7efde00ce410 msgr2=0x7efde00d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:37.112 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.093 7efde6aca700 1 -- 172.21.2.112:0/2604132619 shutdown_connections 2021-02-10T11:46:37.112 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.093 7efde6aca700 1 -- 172.21.2.112:0/2604132619 wait complete. 2021-02-10T11:46:37.113 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.108 7efde6aca700 1 Processor -- start 2021-02-10T11:46:37.114 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.108 7efde6aca700 1 -- start start 2021-02-10T11:46:37.114 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.108 7efde6aca700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7efde01a05e0 0x7efde018b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:37.114 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.108 7efde6aca700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efde00b8780 0x7efde00bfa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:37.114 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.108 7efde6aca700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efde00c0030 0x7efde0189e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:37.114 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.108 7efde6aca700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efde00b82a0 con 0x7efde00c0030 2021-02-10T11:46:37.114 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.108 7efde6aca700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efde00c0430 con 0x7efde01a05e0 2021-02-10T11:46:37.114 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.108 7efde6aca700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efde00c0650 con 0x7efde00b8780 2021-02-10T11:46:37.114 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.108 7efde50e7700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efde00c0030 0x7efde0189e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:37.114 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.108 7efde50e7700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efde00c0030 0x7efde0189e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:59752/0 (socket says 172.21.2.112:59752) 2021-02-10T11:46:37.114 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.108 7efde50e7700 1 -- 172.21.2.112:0/754356762 learned_addr learned my addr 172.21.2.112:0/754356762 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:46:37.114 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.109 7efde50e7700 1 -- 172.21.2.112:0/754356762 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efde00b8780 msgr2=0x7efde00bfa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:37.114 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.109 7efde50e7700 1 --2- 172.21.2.112:0/754356762 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efde00b8780 0x7efde00bfa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:37.115 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.109 7efde50e7700 1 -- 172.21.2.112:0/754356762 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7efde01a05e0 msgr2=0x7efde018b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:37.115 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.109 7efde50e7700 1 --2- 172.21.2.112:0/754356762 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7efde01a05e0 0x7efde018b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:37.115 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.109 7efde50e7700 1 -- 172.21.2.112:0/754356762 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efde018a490 con 0x7efde00c0030 2021-02-10T11:46:37.115 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.109 7efde50e7700 1 --2- 172.21.2.112:0/754356762 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efde00c0030 0x7efde0189e70 secure :-1 s=READY pgs=2919 cs=0 l=1 rev1=1 rx=0x7efdd8001070 tx=0x7efdd800bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:37.115 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.109 7efddd7fa700 1 -- 172.21.2.112:0/754356762 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efdd8001270 con 0x7efde00c0030 2021-02-10T11:46:37.115 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.109 7efddd7fa700 1 -- 172.21.2.112:0/754356762 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdd80014f0 con 0x7efde00c0030 2021-02-10T11:46:37.115 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.109 7efddd7fa700 1 -- 172.21.2.112:0/754356762 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efdd80019d0 con 0x7efde00c0030 2021-02-10T11:46:37.115 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.109 7efde6aca700 1 -- 172.21.2.112:0/754356762 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efde018a490 con 0x7efde00c0030 2021-02-10T11:46:37.115 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.109 7efde6aca700 1 -- 172.21.2.112:0/754356762 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efde018aa20 con 0x7efde00c0030 2021-02-10T11:46:37.115 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.110 7efddd7fa700 1 -- 172.21.2.112:0/754356762 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7efdd80016d0 con 0x7efde00c0030 2021-02-10T11:46:37.115 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.110 7efddd7fa700 1 --2- 172.21.2.112:0/754356762 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7efdc40186c0 0x7efdc401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:37.115 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.110 7efddd7fa700 1 -- 172.21.2.112:0/754356762 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 5573+0+0 (secure 0 0 0) 0x7efdd8032930 con 0x7efde00c0030 2021-02-10T11:46:37.115 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.110 7efde48e6700 1 --2- 172.21.2.112:0/754356762 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7efdc40186c0 0x7efdc401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:37.116 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.111 7efde48e6700 1 --2- 172.21.2.112:0/754356762 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7efdc40186c0 0x7efdc401abc0 crc :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:37.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.141 7efde6aca700 1 -- 172.21.2.112:0/754356762 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efde0082a20 con 0x7efde00c0030 2021-02-10T11:46:37.145 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.143 7efddd7fa700 1 -- 172.21.2.112:0/754356762 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7efdd80093b0 con 0x7efde00c0030 2021-02-10T11:46:37.273 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.271 7efde6aca700 1 -- 172.21.2.112:0/754356762 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7efde0082a20 con 0x7efde00c0030 2021-02-10T11:46:37.274 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.271 7efddd7fa700 1 -- 172.21.2.112:0/754356762 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v172) v1 ==== 100+0+10 (secure 0 0 0) 0x7efdd80016d0 con 0x7efde00c0030 2021-02-10T11:46:37.274 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:46:37.275 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.273 7efde6aca700 1 -- 172.21.2.112:0/754356762 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7efdc40186c0 msgr2=0x7efdc401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:37.275 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.273 7efde6aca700 1 --2- 172.21.2.112:0/754356762 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7efdc40186c0 0x7efdc401abc0 crc :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:37.275 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.273 7efde6aca700 1 -- 172.21.2.112:0/754356762 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efde00c0030 msgr2=0x7efde0189e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:37.275 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.273 7efde6aca700 1 --2- 172.21.2.112:0/754356762 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efde00c0030 0x7efde0189e70 secure :-1 s=READY pgs=2919 cs=0 l=1 rev1=1 rx=0x7efdd8001070 tx=0x7efdd800bd40).stop 2021-02-10T11:46:37.276 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.273 7efde6aca700 1 -- 172.21.2.112:0/754356762 shutdown_connections 2021-02-10T11:46:37.276 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.273 7efde6aca700 1 --2- 172.21.2.112:0/754356762 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7efdc40186c0 0x7efdc401abc0 unknown :-1 s=CLOSED pgs=1322 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:37.276 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.273 7efde6aca700 1 --2- 172.21.2.112:0/754356762 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efde00c0030 0x7efde0189e70 unknown :-1 s=CLOSED pgs=2919 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:37.276 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.273 7efde6aca700 1 --2- 172.21.2.112:0/754356762 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efde00b8780 0x7efde00bfa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:37.276 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.273 7efde6aca700 1 --2- 172.21.2.112:0/754356762 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7efde01a05e0 0x7efde018b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:37.276 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.273 7efde6aca700 1 -- 172.21.2.112:0/754356762 >> 172.21.2.112:0/754356762 conn(0x7efde00a6a80 msgr2=0x7efde00ae110 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:37.277 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.273 7efde6aca700 1 -- 172.21.2.112:0/754356762 shutdown_connections 2021-02-10T11:46:37.277 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.273 7efde6aca700 1 -- 172.21.2.112:0/754356762 wait complete. 2021-02-10T11:46:37.284 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:46:37.284 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:46:37.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.693 7fcdf13d5700 1 Processor -- start 2021-02-10T11:46:37.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.693 7fcdf13d5700 1 -- start start 2021-02-10T11:46:37.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.693 7fcdf13d5700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fcdec0d1870 0x7fcdec1b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:37.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.693 7fcdf13d5700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdec0f0b10 con 0x7fcdec0f0650 2021-02-10T11:46:37.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.693 7fcdf13d5700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdec0d0310 con 0x7fcdec1b0a10 2021-02-10T11:46:37.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.693 7fcdf13d5700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcdec0d0530 con 0x7fcdec0d1870 2021-02-10T11:46:37.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.693 7fcdeb87e700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fcdec0f0650 0x7fcdec0cfb20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:59220/0 (socket says 172.21.2.112:59220) 2021-02-10T11:46:37.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.693 7fcdeb87e700 1 -- 172.21.2.112:0/334291050 learned_addr learned my addr 172.21.2.112:0/334291050 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:46:37.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.693 7fcdeb07d700 1 --2- 172.21.2.112:0/334291050 >> v2:172.21.2.112:3300/0 conn(0x7fcdec0d1870 0x7fcdec1b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:37.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.694 7fcdea07b700 1 -- 172.21.2.112:0/334291050 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1800661422 0 0) 0x7fcde4001500 con 0x7fcdec0f0650 2021-02-10T11:46:37.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.694 7fcdea07b700 1 -- 172.21.2.112:0/334291050 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcdd8002fe0 con 0x7fcdec0f0650 2021-02-10T11:46:37.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.694 7fcdeb07d700 1 -- 172.21.2.112:0/334291050 >> v1:172.21.2.120:6789/0 conn(0x7fcdec1b0a10 legacy=0x7fcdec0cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:46:37.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.694 7fcdeb07d700 1 -- 172.21.2.112:0/334291050 >> v1:172.21.2.112:6789/0 conn(0x7fcdec0f0650 legacy=0x7fcdec0cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:37.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.694 7fcdeb07d700 1 -- 172.21.2.112:0/334291050 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdec0d1c70 con 0x7fcdec0d1870 2021-02-10T11:46:37.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.694 7fcdeb07d700 1 --2- 172.21.2.112:0/334291050 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcdec0d1870 0x7fcdec1b03f0 secure :-1 s=READY pgs=2921 cs=0 l=1 rev1=1 rx=0x7fcddc006650 tx=0x7fcddc009a80).ready entity=mon.0 client_cookie=7869adf0e292ad5 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:37.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.694 7fcdea07b700 1 -- 172.21.2.112:0/334291050 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcddc00c0c0 con 0x7fcdec0d1870 2021-02-10T11:46:37.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.694 7fcdea07b700 1 -- 172.21.2.112:0/334291050 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcddc00c340 con 0x7fcdec0d1870 2021-02-10T11:46:37.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.694 7fcdea07b700 1 -- 172.21.2.112:0/334291050 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcddc00c860 con 0x7fcdec0d1870 2021-02-10T11:46:37.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.694 7fcdf13d5700 1 -- 172.21.2.112:0/334291050 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcdec0d1870 msgr2=0x7fcdec1b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:37.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.694 7fcdf13d5700 1 --2- 172.21.2.112:0/334291050 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcdec0d1870 0x7fcdec1b03f0 secure :-1 s=READY pgs=2921 cs=0 l=1 rev1=1 rx=0x7fcddc006650 tx=0x7fcddc009a80).stop 2021-02-10T11:46:37.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.695 7fcdf13d5700 1 -- 172.21.2.112:0/334291050 shutdown_connections 2021-02-10T11:46:37.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.695 7fcdf13d5700 1 --2- 172.21.2.112:0/334291050 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcdec0d1870 0x7fcdec1b03f0 unknown :-1 s=CLOSED pgs=2921 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:37.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.695 7fcdf13d5700 1 -- 172.21.2.112:0/334291050 >> 172.21.2.112:0/334291050 conn(0x7fcdec0ce450 msgr2=0x7fcdec0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:37.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.695 7fcdf13d5700 1 -- 172.21.2.112:0/334291050 shutdown_connections 2021-02-10T11:46:37.714 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.695 7fcdf13d5700 1 -- 172.21.2.112:0/334291050 wait complete. 2021-02-10T11:46:37.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.710 7fcdf13d5700 1 Processor -- start 2021-02-10T11:46:37.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.710 7fcdf13d5700 1 -- start start 2021-02-10T11:46:37.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.710 7fcdf13d5700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcdec099d50 0x7fcdec1b31b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:37.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.710 7fcdf13d5700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcdec187e50 0x7fcdec188250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:37.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.710 7fcdf13d5700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcdec1b12a0 0x7fcdec1b16a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:37.716 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.710 7fcdf13d5700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdec188870 con 0x7fcdec1b12a0 2021-02-10T11:46:37.716 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.710 7fcdf13d5700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdec1b37d0 con 0x7fcdec099d50 2021-02-10T11:46:37.716 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.710 7fcdf13d5700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdec1b1cc0 con 0x7fcdec187e50 2021-02-10T11:46:37.716 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.710 7fcdeb87e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcdec1b12a0 0x7fcdec1b16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:37.716 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.710 7fcdeb87e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcdec1b12a0 0x7fcdec1b16a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:59762/0 (socket says 172.21.2.112:59762) 2021-02-10T11:46:37.716 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.710 7fcdeb87e700 1 -- 172.21.2.112:0/1873552699 learned_addr learned my addr 172.21.2.112:0/1873552699 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:46:37.716 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.711 7fcdeb87e700 1 -- 172.21.2.112:0/1873552699 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcdec187e50 msgr2=0x7fcdec188250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:37.716 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.711 7fcdeb87e700 1 --2- 172.21.2.112:0/1873552699 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcdec187e50 0x7fcdec188250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:37.716 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.711 7fcdeb87e700 1 -- 172.21.2.112:0/1873552699 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcdec099d50 msgr2=0x7fcdec1b31b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:37.716 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.711 7fcdeb87e700 1 --2- 172.21.2.112:0/1873552699 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcdec099d50 0x7fcdec1b31b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:37.717 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.711 7fcdeb87e700 1 -- 172.21.2.112:0/1873552699 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdec1b1ee0 con 0x7fcdec1b12a0 2021-02-10T11:46:37.717 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.711 7fcdeb87e700 1 --2- 172.21.2.112:0/1873552699 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcdec1b12a0 0x7fcdec1b16a0 secure :-1 s=READY pgs=2922 cs=0 l=1 rev1=1 rx=0x7fcde400a8f0 tx=0x7fcde400ac90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:37.717 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.711 7fcde3fff700 1 -- 172.21.2.112:0/1873552699 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcde40070d0 con 0x7fcdec1b12a0 2021-02-10T11:46:37.717 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.711 7fcde3fff700 1 -- 172.21.2.112:0/1873552699 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcde4007310 con 0x7fcdec1b12a0 2021-02-10T11:46:37.717 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.711 7fcde3fff700 1 -- 172.21.2.112:0/1873552699 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcde40049a0 con 0x7fcdec1b12a0 2021-02-10T11:46:37.717 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.711 7fcdf13d5700 1 -- 172.21.2.112:0/1873552699 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdec1b1ee0 con 0x7fcdec1b12a0 2021-02-10T11:46:37.717 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.711 7fcdf13d5700 1 -- 172.21.2.112:0/1873552699 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdec1b2470 con 0x7fcdec1b12a0 2021-02-10T11:46:37.717 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.712 7fcde3fff700 1 -- 172.21.2.112:0/1873552699 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fcde4004720 con 0x7fcdec1b12a0 2021-02-10T11:46:37.717 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.712 7fcde3fff700 1 --2- 172.21.2.112:0/1873552699 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcdd00186c0 0x7fcdd001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:37.717 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.712 7fcde3fff700 1 -- 172.21.2.112:0/1873552699 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fcde40328e0 con 0x7fcdec1b12a0 2021-02-10T11:46:37.717 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.712 7fcdeb07d700 1 --2- 172.21.2.112:0/1873552699 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcdd00186c0 0x7fcdd001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:37.717 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.713 7fcdeb07d700 1 --2- 172.21.2.112:0/1873552699 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcdd00186c0 0x7fcdd001abc0 crc :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:37.745 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.743 7fcdf13d5700 1 -- 172.21.2.112:0/1873552699 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdec09a1e0 con 0x7fcdec1b12a0 2021-02-10T11:46:37.748 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.746 7fcde3fff700 1 -- 172.21.2.112:0/1873552699 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fcde4000d30 con 0x7fcdec1b12a0 2021-02-10T11:46:37.876 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.874 7fcdf13d5700 1 -- 172.21.2.112:0/1873552699 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcdec09a1e0 con 0x7fcdec1b12a0 2021-02-10T11:46:37.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.874 7fcde3fff700 1 -- 172.21.2.112:0/1873552699 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 148 v148) v1 ==== 95+0+2111 (secure 0 0 0) 0x7fcde4000d30 con 0x7fcdec1b12a0 2021-02-10T11:46:37.878 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 148 2021-02-10T11:46:37.878 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:46:37.878 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":148,"default_fscid":18,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":12642,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/4212630012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":4212630012},{"type":"v1","addr":"172.21.2.112:6833","nonce":4212630012}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":144}],"filesystems":[{"mdsmap":{"epoch":148,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:46:20.421874","modified":"2021-02-10 11:46:22.427977","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12639},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12639":{"gid":12639,"name":"b","rank":0,"incarnation":145,"state":"up:active","state_seq":3,"addr":"172.21.2.120:6835/1654642147","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":1654642147},{"type":"v1","addr":"172.21.2.120:6835","nonce":1654642147}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[37],"metadata_pool":36,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":18}]} 2021-02-10T11:46:37.879 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.876 7fcdf13d5700 1 -- 172.21.2.112:0/1873552699 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcdd00186c0 msgr2=0x7fcdd001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:37.879 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.876 7fcdf13d5700 1 --2- 172.21.2.112:0/1873552699 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcdd00186c0 0x7fcdd001abc0 crc :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:37.879 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.876 7fcdf13d5700 1 -- 172.21.2.112:0/1873552699 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcdec1b12a0 msgr2=0x7fcdec1b16a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:37.879 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.876 7fcdf13d5700 1 --2- 172.21.2.112:0/1873552699 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcdec1b12a0 0x7fcdec1b16a0 secure :-1 s=READY pgs=2922 cs=0 l=1 rev1=1 rx=0x7fcde400a8f0 tx=0x7fcde400ac90).stop 2021-02-10T11:46:37.879 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.876 7fcdf13d5700 1 -- 172.21.2.112:0/1873552699 shutdown_connections 2021-02-10T11:46:37.879 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.876 7fcdf13d5700 1 --2- 172.21.2.112:0/1873552699 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcdd00186c0 0x7fcdd001abc0 unknown :-1 s=CLOSED pgs=1323 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:37.880 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.876 7fcdf13d5700 1 --2- 172.21.2.112:0/1873552699 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcdec1b12a0 0x7fcdec1b16a0 unknown :-1 s=CLOSED pgs=2922 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:37.880 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.876 7fcdf13d5700 1 --2- 172.21.2.112:0/1873552699 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcdec187e50 0x7fcdec188250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:37.880 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.876 7fcdf13d5700 1 --2- 172.21.2.112:0/1873552699 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcdec099d50 0x7fcdec1b31b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:37.880 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.876 7fcdf13d5700 1 -- 172.21.2.112:0/1873552699 >> 172.21.2.112:0/1873552699 conn(0x7fcdec19f8c0 msgr2=0x7fcdec1a5fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:37.880 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.877 7fcdf13d5700 1 -- 172.21.2.112:0/1873552699 shutdown_connections 2021-02-10T11:46:37.880 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:37.877 7fcdf13d5700 1 -- 172.21.2.112:0/1873552699 wait complete. 2021-02-10T11:46:38.088 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 148, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:46:20.421874', 'modified': '2021-02-10 11:46:22.427977', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 12639}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_12639': {'gid': 12639, 'name': 'b', 'rank': 0, 'incarnation': 145, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.120:6835/1654642147', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.120:6834', 'nonce': 1654642147}, {'type': 'v1', 'addr': '172.21.2.120:6835', 'nonce': 1654642147}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [37], 'metadata_pool': 36, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:46:38.088 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:46:38.088 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:46:38.485 INFO:teuthology.orchestra.run.gibba020.stdout:{ 2021-02-10T11:46:38.486 INFO:teuthology.orchestra.run.gibba020.stdout: "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", 2021-02-10T11:46:38.486 INFO:teuthology.orchestra.run.gibba020.stdout: "whoami": 0, 2021-02-10T11:46:38.486 INFO:teuthology.orchestra.run.gibba020.stdout: "id": 12639, 2021-02-10T11:46:38.486 INFO:teuthology.orchestra.run.gibba020.stdout: "want_state": "up:active", 2021-02-10T11:46:38.486 INFO:teuthology.orchestra.run.gibba020.stdout: "state": "up:active", 2021-02-10T11:46:38.486 INFO:teuthology.orchestra.run.gibba020.stdout: "rank_uptime": 18.06062605, 2021-02-10T11:46:38.486 INFO:teuthology.orchestra.run.gibba020.stdout: "mdsmap_epoch": 148, 2021-02-10T11:46:38.486 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch": 172, 2021-02-10T11:46:38.486 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch_barrier": 172, 2021-02-10T11:46:38.486 INFO:teuthology.orchestra.run.gibba020.stdout: "uptime": 23.703012597000001 2021-02-10T11:46:38.486 INFO:teuthology.orchestra.run.gibba020.stdout:} 2021-02-10T11:46:38.490 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", "whoami": 0, "id": 12639, "want_state": "up:active", "state": "up:active", "rank_uptime": 18.06062605, "mdsmap_epoch": 148, "osdmap_epoch": 172, "osdmap_epoch_barrier": 172, "uptime": 23.703012597000001 } 2021-02-10T11:46:38.490 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:46:38.490 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2021-02-10T11:46:38.490 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.1 at ubuntu@gibba012.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2021-02-10T11:46:38.490 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:46:38.491 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:46:38.827 DEBUG:teuthology.orchestra.run.gibba012:> sudo modprobe fuse 2021-02-10T11:46:39.172 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:46:39.173 DEBUG:teuthology.orchestra.run.gibba012:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:46:39.515 INFO:teuthology.orchestra.run.gibba012.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:46:39.515 INFO:teuthology.orchestra.run.gibba012.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:46:39.516 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:46:39.516 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:46:39.516 DEBUG:teuthology.orchestra.run.gibba012:> ls /sys/fs/fuse/connections 2021-02-10T11:46:39.853 INFO:teuthology.orchestra.run.gibba012.stdout:42 2021-02-10T11:46:39.853 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [42] 2021-02-10T11:46:39.854 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.1 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:46:39.857 DEBUG:teuthology.orchestra.run.gibba012:> sudo modprobe fuse 2021-02-10T11:46:40.200 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:46:40.200 DEBUG:teuthology.orchestra.run.gibba012:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:46:40.267 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba012.stderr:ceph-fuse[184581]: starting ceph client 2021-02-10T11:46:40.267 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba012.stderr:2021-02-10 11:46:40.264 7f0b045d5f80 -1 init, newargv = 0x56148ffb4b90 newargc=9 2021-02-10T11:46:40.272 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba012.stderr:ceph-fuse[184581]: starting fuse 2021-02-10T11:46:40.543 INFO:teuthology.orchestra.run.gibba012.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:46:40.543 INFO:teuthology.orchestra.run.gibba012.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:46:40.544 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:46:40.544 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:46:40.545 DEBUG:teuthology.orchestra.run.gibba012:> ls /sys/fs/fuse/connections 2021-02-10T11:46:40.893 INFO:teuthology.orchestra.run.gibba012.stdout:42 2021-02-10T11:46:40.893 INFO:teuthology.orchestra.run.gibba012.stdout:43 2021-02-10T11:46:40.893 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42, 43] 2021-02-10T11:46:40.893 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:46:40.893 DEBUG:teuthology.orchestra.run.gibba012:> sudo python3 -c ' 2021-02-10T11:46:40.893 DEBUG:teuthology.orchestra.run.gibba012:> import glob 2021-02-10T11:46:40.893 DEBUG:teuthology.orchestra.run.gibba012:> import re 2021-02-10T11:46:40.894 DEBUG:teuthology.orchestra.run.gibba012:> import os 2021-02-10T11:46:40.894 DEBUG:teuthology.orchestra.run.gibba012:> import subprocess 2021-02-10T11:46:40.894 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:46:40.894 DEBUG:teuthology.orchestra.run.gibba012:> def find_socket(client_name): 2021-02-10T11:46:40.894 DEBUG:teuthology.orchestra.run.gibba012:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2021-02-10T11:46:40.894 DEBUG:teuthology.orchestra.run.gibba012:> files = glob.glob(asok_path) 2021-02-10T11:46:40.894 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:46:40.894 DEBUG:teuthology.orchestra.run.gibba012:> # Given a non-glob path, it better be there 2021-02-10T11:46:40.894 DEBUG:teuthology.orchestra.run.gibba012:> if "*" not in asok_path: 2021-02-10T11:46:40.894 DEBUG:teuthology.orchestra.run.gibba012:> assert(len(files) == 1) 2021-02-10T11:46:40.894 DEBUG:teuthology.orchestra.run.gibba012:> return files[0] 2021-02-10T11:46:40.894 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:46:40.894 DEBUG:teuthology.orchestra.run.gibba012:> for f in files: 2021-02-10T11:46:40.894 DEBUG:teuthology.orchestra.run.gibba012:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:46:40.895 DEBUG:teuthology.orchestra.run.gibba012:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:46:40.895 DEBUG:teuthology.orchestra.run.gibba012:> return f 2021-02-10T11:46:40.895 DEBUG:teuthology.orchestra.run.gibba012:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:46:40.895 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:46:40.895 DEBUG:teuthology.orchestra.run.gibba012:> print(find_socket("client.1")) 2021-02-10T11:46:40.895 DEBUG:teuthology.orchestra.run.gibba012:> ' 2021-02-10T11:46:41.256 INFO:teuthology.orchestra.run.gibba012.stdout:/var/run/ceph/ceph-client.1.184581.asok 2021-02-10T11:46:41.259 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.184581.asok 2021-02-10T11:46:41.259 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:46:41.259 DEBUG:teuthology.orchestra.run.gibba012:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.184581.asok status 2021-02-10T11:46:41.649 INFO:teuthology.orchestra.run.gibba012.stdout:{ 2021-02-10T11:46:41.649 INFO:teuthology.orchestra.run.gibba012.stdout: "metadata": { 2021-02-10T11:46:41.649 INFO:teuthology.orchestra.run.gibba012.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:46:41.649 INFO:teuthology.orchestra.run.gibba012.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:46:41.649 INFO:teuthology.orchestra.run.gibba012.stdout: "entity_id": "1", 2021-02-10T11:46:41.649 INFO:teuthology.orchestra.run.gibba012.stdout: "hostname": "gibba012", 2021-02-10T11:46:41.649 INFO:teuthology.orchestra.run.gibba012.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2021-02-10T11:46:41.649 INFO:teuthology.orchestra.run.gibba012.stdout: "pid": "184581", 2021-02-10T11:46:41.650 INFO:teuthology.orchestra.run.gibba012.stdout: "root": "/" 2021-02-10T11:46:41.650 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:46:41.650 INFO:teuthology.orchestra.run.gibba012.stdout: "dentry_count": 0, 2021-02-10T11:46:41.650 INFO:teuthology.orchestra.run.gibba012.stdout: "dentry_pinned_count": 0, 2021-02-10T11:46:41.650 INFO:teuthology.orchestra.run.gibba012.stdout: "id": 12855, 2021-02-10T11:46:41.650 INFO:teuthology.orchestra.run.gibba012.stdout: "inst": { 2021-02-10T11:46:41.650 INFO:teuthology.orchestra.run.gibba012.stdout: "name": { 2021-02-10T11:46:41.650 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "client", 2021-02-10T11:46:41.650 INFO:teuthology.orchestra.run.gibba012.stdout: "num": 12855 2021-02-10T11:46:41.650 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:46:41.650 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": { 2021-02-10T11:46:41.650 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:46:41.651 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:0", 2021-02-10T11:46:41.651 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 1689181349 2021-02-10T11:46:41.651 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:46:41.651 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:46:41.651 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": { 2021-02-10T11:46:41.651 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:46:41.651 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:0", 2021-02-10T11:46:41.651 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 1689181349 2021-02-10T11:46:41.651 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:46:41.651 INFO:teuthology.orchestra.run.gibba012.stdout: "inst_str": "client.12855 172.21.2.112:0/1689181349", 2021-02-10T11:46:41.651 INFO:teuthology.orchestra.run.gibba012.stdout: "addr_str": "172.21.2.112:0/1689181349", 2021-02-10T11:46:41.651 INFO:teuthology.orchestra.run.gibba012.stdout: "inode_count": 1, 2021-02-10T11:46:41.651 INFO:teuthology.orchestra.run.gibba012.stdout: "mds_epoch": 148, 2021-02-10T11:46:41.652 INFO:teuthology.orchestra.run.gibba012.stdout: "osd_epoch": 172, 2021-02-10T11:46:41.652 INFO:teuthology.orchestra.run.gibba012.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:46:41.652 INFO:teuthology.orchestra.run.gibba012.stdout: "blacklisted": false 2021-02-10T11:46:41.652 INFO:teuthology.orchestra.run.gibba012.stdout:} 2021-02-10T11:46:41.653 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:46:41.653 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:46:41.653 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:46:41.989 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:46:41.989 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:46:41.990 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:46:41.990 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:46:42.336 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2021-02-10T11:46:42.769 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:42.750 7fc4a87cd700 1 Processor -- start 2021-02-10T11:46:42.769 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:42.750 7fc4a87cd700 1 -- start start 2021-02-10T11:46:42.769 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:42.750 7fc4a87cd700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc4a00d1880 0x7fc4a01b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:42.769 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:42.750 7fc4a87cd700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fc4a00cff80 0x7fc4a0109140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:42.769 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:42.750 7fc4a87cd700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4a0109a70 con 0x7fc4a01b0af0 2021-02-10T11:46:42.769 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:42.750 7fc4a87cd700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4a0109760 con 0x7fc4a00cff80 2021-02-10T11:46:42.769 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:42.750 7fc4a87cd700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4a0088070 con 0x7fc4a00d1880 2021-02-10T11:46:42.770 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:42.751 7fc4a6dea700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fc4a00cff80 0x7fc4a0109140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:42.770 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:42.751 7fc4a6dea700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fc4a00cff80 0x7fc4a0109140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:59784/0 (socket says 172.21.2.112:59784) 2021-02-10T11:46:42.770 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:42.751 7fc4a6dea700 1 -- 172.21.2.112:0/3842083849 learned_addr learned my addr 172.21.2.112:0/3842083849 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:46:42.770 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:42.751 7fc4a6dea700 1 -- 172.21.2.112:0/3842083849 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc4a00d1880 msgr2=0x7fc4a01b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:46:42.770 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:42.751 7fc4a6dea700 1 --2- 172.21.2.112:0/3842083849 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc4a00d1880 0x7fc4a01b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:42.770 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:42.751 7fc4a6dea700 1 -- 172.21.2.112:0/3842083849 >> v1:172.21.2.120:6789/0 conn(0x7fc4a01b0af0 legacy=0x7fc4a0108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:46:42.770 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:42.751 7fc4a6dea700 1 -- 172.21.2.112:0/3842083849 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4a0088260 con 0x7fc4a00cff80 2021-02-10T11:46:42.770 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:42.751 7fc4a6dea700 1 --2- 172.21.2.112:0/3842083849 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc4a00cff80 0x7fc4a0109140 secure :-1 s=READY pgs=2925 cs=0 l=1 rev1=1 rx=0x7fc498006f00 tx=0x7fc498009f10).ready entity=mon.0 client_cookie=2b169ed687d25d00 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:42.770 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:42.751 7fc4a55e7700 1 -- 172.21.2.112:0/3842083849 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc49800b3b0 con 0x7fc4a00cff80 2021-02-10T11:46:42.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:42.751 7fc4a55e7700 1 -- 172.21.2.112:0/3842083849 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc49800b730 con 0x7fc4a00cff80 2021-02-10T11:46:42.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:42.751 7fc4a55e7700 1 -- 172.21.2.112:0/3842083849 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc49800bc50 con 0x7fc4a00cff80 2021-02-10T11:46:42.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:42.751 7fc4a87cd700 1 -- 172.21.2.112:0/3842083849 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc4a00cff80 msgr2=0x7fc4a0109140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:42.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:42.751 7fc4a87cd700 1 --2- 172.21.2.112:0/3842083849 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc4a00cff80 0x7fc4a0109140 secure :-1 s=READY pgs=2925 cs=0 l=1 rev1=1 rx=0x7fc498006f00 tx=0x7fc498009f10).stop 2021-02-10T11:46:42.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:42.752 7fc4a87cd700 1 -- 172.21.2.112:0/3842083849 shutdown_connections 2021-02-10T11:46:42.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:42.752 7fc4a87cd700 1 --2- 172.21.2.112:0/3842083849 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc4a00cff80 0x7fc4a0109140 unknown :-1 s=CLOSED pgs=2925 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:42.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:42.752 7fc4a87cd700 1 --2- 172.21.2.112:0/3842083849 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc4a00d1880 0x7fc4a01b04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:42.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:42.752 7fc4a87cd700 1 -- 172.21.2.112:0/3842083849 >> 172.21.2.112:0/3842083849 conn(0x7fc4a00ce430 msgr2=0x7fc4a00d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:42.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:42.752 7fc4a87cd700 1 -- 172.21.2.112:0/3842083849 shutdown_connections 2021-02-10T11:46:42.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:42.752 7fc4a87cd700 1 -- 172.21.2.112:0/3842083849 wait complete. 2021-02-10T11:46:42.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:42.767 7fc4a87cd700 1 Processor -- start 2021-02-10T11:46:42.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:42.767 7fc4a87cd700 1 -- start start 2021-02-10T11:46:42.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:42.767 7fc4a87cd700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc4a019e5b0 0x7fc4a01afc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:42.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:42.767 7fc4a87cd700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc4a019e9b0 0x7fc4a01b0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:42.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:42.767 7fc4a87cd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc4a01b0250 0x7fc4a01b13e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:42.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:42.767 7fc4a87cd700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4a0195bd0 con 0x7fc4a01b0250 2021-02-10T11:46:42.774 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:42.767 7fc4a87cd700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4a01b0840 con 0x7fc4a019e9b0 2021-02-10T11:46:42.774 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:42.767 7fc4a87cd700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4a019eed0 con 0x7fc4a019e5b0 2021-02-10T11:46:42.774 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:42.767 7fc4a6dea700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc4a01b0250 0x7fc4a01b13e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:42.774 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:42.767 7fc4a6dea700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc4a01b0250 0x7fc4a01b13e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:59790/0 (socket says 172.21.2.112:59790) 2021-02-10T11:46:42.774 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:42.767 7fc4a6dea700 1 -- 172.21.2.112:0/670701768 learned_addr learned my addr 172.21.2.112:0/670701768 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:46:42.774 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:42.768 7fc4a6dea700 1 -- 172.21.2.112:0/670701768 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc4a019e5b0 msgr2=0x7fc4a01afc30 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:46:42.774 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:42.768 7fc4a6dea700 1 --2- 172.21.2.112:0/670701768 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc4a019e5b0 0x7fc4a01afc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:42.774 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:42.768 7fc4a6dea700 1 -- 172.21.2.112:0/670701768 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc4a019e9b0 msgr2=0x7fc4a01b0dc0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:46:42.774 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:42.768 7fc4a6dea700 1 --2- 172.21.2.112:0/670701768 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc4a019e9b0 0x7fc4a01b0dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:42.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:42.768 7fc4a6dea700 1 -- 172.21.2.112:0/670701768 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4a019f0b0 con 0x7fc4a01b0250 2021-02-10T11:46:42.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:42.768 7fc4a6dea700 1 --2- 172.21.2.112:0/670701768 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc4a01b0250 0x7fc4a01b13e0 secure :-1 s=READY pgs=2926 cs=0 l=1 rev1=1 rx=0x7fc498009390 tx=0x7fc49800bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:42.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:42.768 7fc496ffd700 1 -- 172.21.2.112:0/670701768 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc498009450 con 0x7fc4a01b0250 2021-02-10T11:46:42.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:42.768 7fc496ffd700 1 -- 172.21.2.112:0/670701768 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc498003030 con 0x7fc4a01b0250 2021-02-10T11:46:42.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:42.768 7fc496ffd700 1 -- 172.21.2.112:0/670701768 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc498001480 con 0x7fc4a01b0250 2021-02-10T11:46:42.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:42.768 7fc4a87cd700 1 -- 172.21.2.112:0/670701768 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4a019f0b0 con 0x7fc4a01b0250 2021-02-10T11:46:42.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:42.768 7fc4a87cd700 1 -- 172.21.2.112:0/670701768 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4a01b1cf0 con 0x7fc4a01b0250 2021-02-10T11:46:42.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:42.769 7fc496ffd700 1 -- 172.21.2.112:0/670701768 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc498001270 con 0x7fc4a01b0250 2021-02-10T11:46:42.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:42.769 7fc496ffd700 1 --2- 172.21.2.112:0/670701768 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc4880186c0 0x7fc48801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:42.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:42.769 7fc496ffd700 1 -- 172.21.2.112:0/670701768 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fc4980328a0 con 0x7fc4a01b0250 2021-02-10T11:46:42.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:42.769 7fc4a65e9700 1 --2- 172.21.2.112:0/670701768 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc4880186c0 0x7fc48801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:42.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:42.769 7fc4a65e9700 1 --2- 172.21.2.112:0/670701768 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc4880186c0 0x7fc48801abc0 crc :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:42.803 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:42.800 7fc4a87cd700 1 -- 172.21.2.112:0/670701768 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4a01b0840 con 0x7fc4a01b0250 2021-02-10T11:46:42.805 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:42.803 7fc496ffd700 1 -- 172.21.2.112:0/670701768 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc49801e590 con 0x7fc4a01b0250 2021-02-10T11:46:42.941 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:42.939 7fc4a87cd700 1 -- 172.21.2.112:0/670701768 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fc4a01b0840 con 0x7fc4a01b0250 2021-02-10T11:46:43.282 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:43.279 7fc496ffd700 1 -- 172.21.2.112:0/670701768 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v149) v1 ==== 136+0+0 (secure 0 0 0) 0x7fc49801e590 con 0x7fc4a01b0250 2021-02-10T11:46:43.311 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:43.308 7fc4a87cd700 1 -- 172.21.2.112:0/670701768 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc4880186c0 msgr2=0x7fc48801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:43.311 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:43.308 7fc4a87cd700 1 --2- 172.21.2.112:0/670701768 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc4880186c0 0x7fc48801abc0 crc :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:43.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:43.308 7fc4a87cd700 1 -- 172.21.2.112:0/670701768 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc4a01b0250 msgr2=0x7fc4a01b13e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:43.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:43.308 7fc4a87cd700 1 --2- 172.21.2.112:0/670701768 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc4a01b0250 0x7fc4a01b13e0 secure :-1 s=READY pgs=2926 cs=0 l=1 rev1=1 rx=0x7fc498009390 tx=0x7fc49800bd40).stop 2021-02-10T11:46:43.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:43.308 7fc4a87cd700 1 -- 172.21.2.112:0/670701768 shutdown_connections 2021-02-10T11:46:43.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:43.308 7fc4a87cd700 1 --2- 172.21.2.112:0/670701768 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc4880186c0 0x7fc48801abc0 unknown :-1 s=CLOSED pgs=1324 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:43.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:43.308 7fc4a87cd700 1 --2- 172.21.2.112:0/670701768 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc4a019e9b0 0x7fc4a01b0dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:43.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:43.308 7fc4a87cd700 1 --2- 172.21.2.112:0/670701768 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc4a01b0250 0x7fc4a01b13e0 unknown :-1 s=CLOSED pgs=2926 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:43.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:43.308 7fc4a87cd700 1 --2- 172.21.2.112:0/670701768 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc4a019e5b0 0x7fc4a01afc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:43.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:43.308 7fc4a87cd700 1 -- 172.21.2.112:0/670701768 >> 172.21.2.112:0/670701768 conn(0x7fc4a019f940 msgr2=0x7fc4a018c290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:43.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:43.308 7fc4a87cd700 1 -- 172.21.2.112:0/670701768 shutdown_connections 2021-02-10T11:46:43.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:43.308 7fc4a87cd700 1 -- 172.21.2.112:0/670701768 wait complete. 2021-02-10T11:46:43.319 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs volume ls 2021-02-10T11:46:43.747 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:43.728 7feeddb88700 1 Processor -- start 2021-02-10T11:46:43.747 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:43.728 7feeddb88700 1 -- start start 2021-02-10T11:46:43.747 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:43.728 7feeddb88700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7feed80d1870 0x7feed81b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:43.747 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:43.728 7feeddb88700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7feed80f0650 0x7feed80cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:43.747 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:43.728 7feeddb88700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feed80f0b10 con 0x7feed81b0a10 2021-02-10T11:46:43.747 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:43.728 7feeddb88700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feed80e5ed0 con 0x7feed80f0650 2021-02-10T11:46:43.747 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:43.728 7feeddb88700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feed80d1c70 con 0x7feed80d1870 2021-02-10T11:46:43.747 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:43.729 7feed7fff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7feed80f0650 0x7feed80cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:43.747 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:43.729 7feed7fff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7feed80f0650 0x7feed80cfb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:59794/0 (socket says 172.21.2.112:59794) 2021-02-10T11:46:43.748 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:43.729 7feed7fff700 1 -- 172.21.2.112:0/3488787879 learned_addr learned my addr 172.21.2.112:0/3488787879 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:46:43.748 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:43.729 7feed6ffd700 1 --1- 172.21.2.112:0/3488787879 >> v1:172.21.2.112:6789/0 conn(0x7feed81b0a10 0x7feed80cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:59256/0 (socket says 172.21.2.112:59256) 2021-02-10T11:46:43.748 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:43.729 7feed77fe700 1 --2- 172.21.2.112:0/3488787879 >> v2:172.21.2.120:3300/0 conn(0x7feed80d1870 0x7feed81b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:43.748 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:43.729 7feed7fff700 1 -- 172.21.2.112:0/3488787879 >> v2:172.21.2.120:3300/0 conn(0x7feed80d1870 msgr2=0x7feed81b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:46:43.748 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:43.729 7feed7fff700 1 --2- 172.21.2.112:0/3488787879 >> v2:172.21.2.120:3300/0 conn(0x7feed80d1870 0x7feed81b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:43.748 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:43.729 7feed7fff700 1 -- 172.21.2.112:0/3488787879 >> v1:172.21.2.112:6789/0 conn(0x7feed81b0a10 legacy=0x7feed80cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:43.748 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:43.729 7feed7fff700 1 -- 172.21.2.112:0/3488787879 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feed80d1e90 con 0x7feed80f0650 2021-02-10T11:46:43.748 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:43.729 7feed77fe700 1 --2- 172.21.2.112:0/3488787879 >> v2:172.21.2.120:3300/0 conn(0x7feed80d1870 0x7feed81b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:46:43.748 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:43.729 7feed7fff700 1 --2- 172.21.2.112:0/3488787879 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7feed80f0650 0x7feed80cfb20 secure :-1 s=READY pgs=2928 cs=0 l=1 rev1=1 rx=0x7feed0006f00 tx=0x7feed0009f10).ready entity=mon.0 client_cookie=ec50e4ab34695ef7 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:43.748 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:43.729 7feed67fc700 1 -- 172.21.2.112:0/3488787879 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feed000b3b0 con 0x7feed80f0650 2021-02-10T11:46:43.748 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:43.730 7feed67fc700 1 -- 172.21.2.112:0/3488787879 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feed000b670 con 0x7feed80f0650 2021-02-10T11:46:43.748 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:43.730 7feed67fc700 1 -- 172.21.2.112:0/3488787879 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feed000bb50 con 0x7feed80f0650 2021-02-10T11:46:43.748 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:43.730 7feeddb88700 1 -- 172.21.2.112:0/3488787879 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7feed80f0650 msgr2=0x7feed80cfb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:43.748 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:43.730 7feeddb88700 1 --2- 172.21.2.112:0/3488787879 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7feed80f0650 0x7feed80cfb20 secure :-1 s=READY pgs=2928 cs=0 l=1 rev1=1 rx=0x7feed0006f00 tx=0x7feed0009f10).stop 2021-02-10T11:46:43.749 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:43.730 7feeddb88700 1 -- 172.21.2.112:0/3488787879 shutdown_connections 2021-02-10T11:46:43.749 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:43.730 7feeddb88700 1 --2- 172.21.2.112:0/3488787879 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7feed80f0650 0x7feed80cfb20 unknown :-1 s=CLOSED pgs=2928 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:43.749 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:43.730 7feeddb88700 1 --2- 172.21.2.112:0/3488787879 >> v2:172.21.2.120:3300/0 conn(0x7feed80d1870 0x7feed81b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:43.749 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:43.730 7feeddb88700 1 -- 172.21.2.112:0/3488787879 >> 172.21.2.112:0/3488787879 conn(0x7feed80ce450 msgr2=0x7feed80d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:43.749 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:43.730 7feeddb88700 1 -- 172.21.2.112:0/3488787879 shutdown_connections 2021-02-10T11:46:43.749 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:43.730 7feeddb88700 1 -- 172.21.2.112:0/3488787879 wait complete. 2021-02-10T11:46:43.751 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:43.745 7feeddb88700 1 Processor -- start 2021-02-10T11:46:43.751 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:43.745 7feeddb88700 1 -- start start 2021-02-10T11:46:43.751 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:43.745 7feeddb88700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7feed818b550 0x7feed818b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:43.751 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:43.745 7feeddb88700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7feed81a3530 0x7feed81a3930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:43.751 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:43.745 7feeddb88700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7feed80c1540 0x7feed80c1940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:43.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:43.745 7feeddb88700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feed80b5560 con 0x7feed80c1540 2021-02-10T11:46:43.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:43.745 7feeddb88700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feed80c1f60 con 0x7feed818b550 2021-02-10T11:46:43.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:43.745 7feeddb88700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feed80c2180 con 0x7feed81a3530 2021-02-10T11:46:43.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:43.745 7feed7fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7feed80c1540 0x7feed80c1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:43.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:43.745 7feed7fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7feed80c1540 0x7feed80c1940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:59800/0 (socket says 172.21.2.112:59800) 2021-02-10T11:46:43.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:43.746 7feed7fff700 1 -- 172.21.2.112:0/2346521767 learned_addr learned my addr 172.21.2.112:0/2346521767 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:46:43.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:43.746 7feed6ffd700 1 --2- 172.21.2.112:0/2346521767 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7feed81a3530 0x7feed81a3930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:43.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:43.746 7feed77fe700 1 --2- 172.21.2.112:0/2346521767 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7feed818b550 0x7feed818b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:43.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:43.746 7feed7fff700 1 -- 172.21.2.112:0/2346521767 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7feed81a3530 msgr2=0x7feed81a3930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:43.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:43.746 7feed7fff700 1 --2- 172.21.2.112:0/2346521767 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7feed81a3530 0x7feed81a3930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:43.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:43.746 7feed7fff700 1 -- 172.21.2.112:0/2346521767 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7feed818b550 msgr2=0x7feed818b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:43.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:43.746 7feed7fff700 1 --2- 172.21.2.112:0/2346521767 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7feed818b550 0x7feed818b950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:43.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:43.746 7feed7fff700 1 -- 172.21.2.112:0/2346521767 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feed80c23a0 con 0x7feed80c1540 2021-02-10T11:46:43.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:43.746 7feed6ffd700 1 --2- 172.21.2.112:0/2346521767 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7feed81a3530 0x7feed81a3930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:46:43.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:43.746 7feed77fe700 1 --2- 172.21.2.112:0/2346521767 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7feed818b550 0x7feed818b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:46:43.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:43.746 7feed7fff700 1 --2- 172.21.2.112:0/2346521767 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7feed80c1540 0x7feed80c1940 secure :-1 s=READY pgs=2929 cs=0 l=1 rev1=1 rx=0x7feed000b9b0 tx=0x7feed000bd50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:43.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:43.746 7feecffff700 1 -- 172.21.2.112:0/2346521767 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feed0009450 con 0x7feed80c1540 2021-02-10T11:46:43.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:43.747 7feecffff700 1 -- 172.21.2.112:0/2346521767 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feed000c2f0 con 0x7feed80c1540 2021-02-10T11:46:43.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:43.747 7feecffff700 1 -- 172.21.2.112:0/2346521767 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feed000c750 con 0x7feed80c1540 2021-02-10T11:46:43.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:43.747 7feeddb88700 1 -- 172.21.2.112:0/2346521767 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feed80c23a0 con 0x7feed80c1540 2021-02-10T11:46:43.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:43.747 7feeddb88700 1 -- 172.21.2.112:0/2346521767 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feed80c23a0 con 0x7feed80c1540 2021-02-10T11:46:43.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:43.747 7feecffff700 1 -- 172.21.2.112:0/2346521767 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7feed0009450 con 0x7feed80c1540 2021-02-10T11:46:43.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:43.747 7feecffff700 1 --2- 172.21.2.112:0/2346521767 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7feebc018700 0x7feebc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:43.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:43.747 7feecffff700 1 -- 172.21.2.112:0/2346521767 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 5573+0+0 (secure 0 0 0) 0x7feed0032b80 con 0x7feed80c1540 2021-02-10T11:46:43.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:43.748 7feed77fe700 1 --2- 172.21.2.112:0/2346521767 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7feebc018700 0x7feebc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:43.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:43.748 7feed77fe700 1 --2- 172.21.2.112:0/2346521767 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7feebc018700 0x7feebc01abc0 crc :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:43.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:43.778 7feeddb88700 1 -- 172.21.2.112:0/2346521767 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feed80ebc90 con 0x7feed80c1540 2021-02-10T11:46:43.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:43.781 7feecffff700 1 -- 172.21.2.112:0/2346521767 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7feed0000f90 con 0x7feed80c1540 2021-02-10T11:46:43.917 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:43.915 7feeddb88700 1 -- 172.21.2.112:0/2346521767 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs volume ls", "target": ["mgr", ""]}) v1 -- 0x7feed80f0b10 con 0x7feebc018700 2021-02-10T11:46:43.919 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:43.916 7feecffff700 1 -- 172.21.2.112:0/2346521767 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+41 (crc 0 0 0) 0x7feec8006ad0 con 0x7feebc018700 2021-02-10T11:46:43.920 INFO:teuthology.orchestra.run.gibba012.stdout:[ 2021-02-10T11:46:43.921 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:46:43.921 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "cephfs" 2021-02-10T11:46:43.921 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:46:43.921 INFO:teuthology.orchestra.run.gibba012.stdout:] 2021-02-10T11:46:43.922 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:43.919 7feeddb88700 1 -- 172.21.2.112:0/2346521767 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7feebc018700 msgr2=0x7feebc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:43.922 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:43.919 7feeddb88700 1 --2- 172.21.2.112:0/2346521767 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7feebc018700 0x7feebc01abc0 crc :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:43.922 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:43.919 7feeddb88700 1 -- 172.21.2.112:0/2346521767 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7feed80c1540 msgr2=0x7feed80c1940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:43.922 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:43.919 7feeddb88700 1 --2- 172.21.2.112:0/2346521767 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7feed80c1540 0x7feed80c1940 secure :-1 s=READY pgs=2929 cs=0 l=1 rev1=1 rx=0x7feed000b9b0 tx=0x7feed000bd50).stop 2021-02-10T11:46:43.923 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:43.919 7feeddb88700 1 -- 172.21.2.112:0/2346521767 shutdown_connections 2021-02-10T11:46:43.923 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:43.919 7feeddb88700 1 --2- 172.21.2.112:0/2346521767 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7feebc018700 0x7feebc01abc0 unknown :-1 s=CLOSED pgs=1325 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:43.923 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:43.919 7feeddb88700 1 --2- 172.21.2.112:0/2346521767 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7feed80c1540 0x7feed80c1940 unknown :-1 s=CLOSED pgs=2929 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:43.923 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:43.919 7feeddb88700 1 --2- 172.21.2.112:0/2346521767 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7feed81a3530 0x7feed81a3930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:43.923 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:43.919 7feeddb88700 1 --2- 172.21.2.112:0/2346521767 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7feed818b550 0x7feed818b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:43.923 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:43.919 7feeddb88700 1 -- 172.21.2.112:0/2346521767 >> 172.21.2.112:0/2346521767 conn(0x7feed80a6870 msgr2=0x7feed808a420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:43.923 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:43.920 7feeddb88700 1 -- 172.21.2.112:0/2346521767 shutdown_connections 2021-02-10T11:46:43.923 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:43.920 7feeddb88700 1 -- 172.21.2.112:0/2346521767 wait complete. 2021-02-10T11:46:43.933 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:46:44.366 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.347 7fcccc602700 1 Processor -- start 2021-02-10T11:46:44.366 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.347 7fcccc602700 1 -- start start 2021-02-10T11:46:44.366 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.348 7fcccc602700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fccc40d1c70 0x7fccc40cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:44.367 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.348 7fcccc602700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fccc40cf580 0x7fccc40cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:44.367 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.348 7fcccc602700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccc40fe1c0 con 0x7fccc40d1870 2021-02-10T11:46:44.367 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.348 7fcccc602700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccc41b0cb0 con 0x7fccc40d1c70 2021-02-10T11:46:44.367 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.348 7fcccc602700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccc4109a60 con 0x7fccc40cf580 2021-02-10T11:46:44.367 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.348 7fccc9c1d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fccc40d1c70 0x7fccc40cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:44.367 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.348 7fccc9c1d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fccc40d1c70 0x7fccc40cef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:59804/0 (socket says 172.21.2.112:59804) 2021-02-10T11:46:44.368 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.348 7fccc9c1d700 1 -- 172.21.2.112:0/1827870116 learned_addr learned my addr 172.21.2.112:0/1827870116 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:46:44.368 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.348 7fccc9c1d700 1 -- 172.21.2.112:0/1827870116 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fccc40cf580 msgr2=0x7fccc40cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:46:44.368 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.348 7fccc9c1d700 1 --2- 172.21.2.112:0/1827870116 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fccc40cf580 0x7fccc40cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:44.368 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.348 7fccc9c1d700 1 -- 172.21.2.112:0/1827870116 >> v1:172.21.2.120:6789/0 conn(0x7fccc40d1870 legacy=0x7fccc41b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:46:44.368 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.348 7fccc9c1d700 1 -- 172.21.2.112:0/1827870116 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccc4088070 con 0x7fccc40d1c70 2021-02-10T11:46:44.369 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.349 7fccc9c1d700 1 --2- 172.21.2.112:0/1827870116 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fccc40d1c70 0x7fccc40cef60 secure :-1 s=READY pgs=2930 cs=0 l=1 rev1=1 rx=0x7fccb8006700 tx=0x7fccb8009b00).ready entity=mon.0 client_cookie=a7e2963d7a639fad server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:44.369 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.349 7fccc941c700 1 -- 172.21.2.112:0/1827870116 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fccb800c0c0 con 0x7fccc40d1c70 2021-02-10T11:46:44.369 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.349 7fccc941c700 1 -- 172.21.2.112:0/1827870116 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccb800c340 con 0x7fccc40d1c70 2021-02-10T11:46:44.369 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.349 7fccc941c700 1 -- 172.21.2.112:0/1827870116 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fccb800c820 con 0x7fccc40d1c70 2021-02-10T11:46:44.369 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.349 7fcccc602700 1 -- 172.21.2.112:0/1827870116 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fccc40d1c70 msgr2=0x7fccc40cef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:44.369 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.349 7fcccc602700 1 --2- 172.21.2.112:0/1827870116 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fccc40d1c70 0x7fccc40cef60 secure :-1 s=READY pgs=2930 cs=0 l=1 rev1=1 rx=0x7fccb8006700 tx=0x7fccb8009b00).stop 2021-02-10T11:46:44.370 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.349 7fcccc602700 1 -- 172.21.2.112:0/1827870116 shutdown_connections 2021-02-10T11:46:44.370 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.350 7fcccc602700 1 --2- 172.21.2.112:0/1827870116 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fccc40d1c70 0x7fccc40cef60 unknown :-1 s=CLOSED pgs=2930 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:44.370 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.350 7fcccc602700 1 --2- 172.21.2.112:0/1827870116 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fccc40cf580 0x7fccc40cf980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:44.370 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.350 7fcccc602700 1 -- 172.21.2.112:0/1827870116 >> 172.21.2.112:0/1827870116 conn(0x7fccc40ce420 msgr2=0x7fccc40d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:44.370 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.350 7fcccc602700 1 -- 172.21.2.112:0/1827870116 shutdown_connections 2021-02-10T11:46:44.370 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.350 7fcccc602700 1 -- 172.21.2.112:0/1827870116 wait complete. 2021-02-10T11:46:44.373 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.364 7fcccc602700 1 Processor -- start 2021-02-10T11:46:44.373 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.364 7fcccc602700 1 -- start start 2021-02-10T11:46:44.374 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.365 7fcccc602700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fccc40c3820 0x7fccc4084cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:44.374 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.365 7fcccc602700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fccc40a0290 0x7fccc40852d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:44.374 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.365 7fcccc602700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fccc4086270 0x7fccc4086670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:44.374 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.365 7fcccc602700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccc40c0700 con 0x7fccc40c3820 2021-02-10T11:46:44.375 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.365 7fcccc602700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccc40858f0 con 0x7fccc4086270 2021-02-10T11:46:44.375 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.365 7fcccc602700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccc4085b10 con 0x7fccc40a0290 2021-02-10T11:46:44.375 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.365 7fccca41e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fccc40c3820 0x7fccc4084cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:44.375 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.365 7fccca41e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fccc40c3820 0x7fccc4084cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:59808/0 (socket says 172.21.2.112:59808) 2021-02-10T11:46:44.375 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.365 7fccca41e700 1 -- 172.21.2.112:0/3000508879 learned_addr learned my addr 172.21.2.112:0/3000508879 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:46:44.375 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.365 7fcccac1f700 1 --2- 172.21.2.112:0/3000508879 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fccc4086270 0x7fccc4086670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:44.376 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.365 7fccc9c1d700 1 --2- 172.21.2.112:0/3000508879 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fccc40a0290 0x7fccc40852d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:44.376 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.365 7fccca41e700 1 -- 172.21.2.112:0/3000508879 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fccc40a0290 msgr2=0x7fccc40852d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:44.376 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.365 7fccca41e700 1 --2- 172.21.2.112:0/3000508879 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fccc40a0290 0x7fccc40852d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:44.376 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.365 7fccca41e700 1 -- 172.21.2.112:0/3000508879 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fccc4086270 msgr2=0x7fccc4086670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:44.376 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.365 7fccca41e700 1 --2- 172.21.2.112:0/3000508879 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fccc4086270 0x7fccc4086670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:44.376 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.365 7fccca41e700 1 -- 172.21.2.112:0/3000508879 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccc4086c90 con 0x7fccc40c3820 2021-02-10T11:46:44.377 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.365 7fccc9c1d700 1 --2- 172.21.2.112:0/3000508879 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fccc40a0290 0x7fccc40852d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:46:44.377 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.365 7fccca41e700 1 --2- 172.21.2.112:0/3000508879 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fccc40c3820 0x7fccc4084cb0 secure :-1 s=READY pgs=2931 cs=0 l=1 rev1=1 rx=0x7fccc000b5b0 tx=0x7fccc000b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:44.377 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.365 7fccb6ffd700 1 -- 172.21.2.112:0/3000508879 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fccc000bd40 con 0x7fccc40c3820 2021-02-10T11:46:44.377 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.365 7fccb6ffd700 1 -- 172.21.2.112:0/3000508879 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccc0006030 con 0x7fccc40c3820 2021-02-10T11:46:44.377 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.365 7fcccc602700 1 -- 172.21.2.112:0/3000508879 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccc4086c90 con 0x7fccc40c3820 2021-02-10T11:46:44.377 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.365 7fccb6ffd700 1 -- 172.21.2.112:0/3000508879 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fccc00054a0 con 0x7fccc40c3820 2021-02-10T11:46:44.378 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.365 7fcccc602700 1 -- 172.21.2.112:0/3000508879 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccc41a2f00 con 0x7fccc40c3820 2021-02-10T11:46:44.378 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.366 7fccb6ffd700 1 -- 172.21.2.112:0/3000508879 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fccc0006210 con 0x7fccc40c3820 2021-02-10T11:46:44.378 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.366 7fccb6ffd700 1 --2- 172.21.2.112:0/3000508879 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fccac0186d0 0x7fccac01abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:44.378 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.366 7fccb6ffd700 1 -- 172.21.2.112:0/3000508879 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fccc0032640 con 0x7fccc40c3820 2021-02-10T11:46:44.378 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.366 7fccc9c1d700 1 --2- 172.21.2.112:0/3000508879 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fccac0186d0 0x7fccac01abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:44.378 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.367 7fccc9c1d700 1 --2- 172.21.2.112:0/3000508879 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fccac0186d0 0x7fccac01abd0 crc :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:44.400 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.398 7fcccc602700 1 -- 172.21.2.112:0/3000508879 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccc4082a70 con 0x7fccc40c3820 2021-02-10T11:46:44.403 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.400 7fccb6ffd700 1 -- 172.21.2.112:0/3000508879 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fccc0000c10 con 0x7fccc40c3820 2021-02-10T11:46:44.553 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.550 7fcccc602700 1 -- 172.21.2.112:0/3000508879 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fccc4082a70 con 0x7fccc40c3820 2021-02-10T11:46:44.553 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.550 7fccb6ffd700 1 -- 172.21.2.112:0/3000508879 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v42)=0 v42) v1 ==== 122+0+0 (secure 0 0 0) 0x7fccc0006030 con 0x7fccc40c3820 2021-02-10T11:46:44.555 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.553 7fcccc602700 1 -- 172.21.2.112:0/3000508879 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fccac0186d0 msgr2=0x7fccac01abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:44.555 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.553 7fcccc602700 1 --2- 172.21.2.112:0/3000508879 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fccac0186d0 0x7fccac01abd0 crc :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:44.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.553 7fcccc602700 1 -- 172.21.2.112:0/3000508879 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fccc40c3820 msgr2=0x7fccc4084cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:44.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.553 7fcccc602700 1 --2- 172.21.2.112:0/3000508879 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fccc40c3820 0x7fccc4084cb0 secure :-1 s=READY pgs=2931 cs=0 l=1 rev1=1 rx=0x7fccc000b5b0 tx=0x7fccc000b950).stop 2021-02-10T11:46:44.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.553 7fcccc602700 1 -- 172.21.2.112:0/3000508879 shutdown_connections 2021-02-10T11:46:44.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.553 7fcccc602700 1 --2- 172.21.2.112:0/3000508879 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fccac0186d0 0x7fccac01abd0 unknown :-1 s=CLOSED pgs=1326 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:44.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.553 7fcccc602700 1 --2- 172.21.2.112:0/3000508879 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fccc4086270 0x7fccc4086670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:44.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.553 7fcccc602700 1 --2- 172.21.2.112:0/3000508879 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fccc40a0290 0x7fccc40852d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:44.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.553 7fcccc602700 1 --2- 172.21.2.112:0/3000508879 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fccc40c3820 0x7fccc4084cb0 unknown :-1 s=CLOSED pgs=2931 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:44.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.553 7fcccc602700 1 -- 172.21.2.112:0/3000508879 >> 172.21.2.112:0/3000508879 conn(0x7fccc40872c0 msgr2=0x7fccc4095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:44.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.553 7fcccc602700 1 -- 172.21.2.112:0/3000508879 shutdown_connections 2021-02-10T11:46:44.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.553 7fcccc602700 1 -- 172.21.2.112:0/3000508879 wait complete. 2021-02-10T11:46:44.563 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolumegroup create cephfs group_0000000000707517 2021-02-10T11:46:44.997 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.978 7f10290e3700 1 Processor -- start 2021-02-10T11:46:44.997 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.978 7f10290e3700 1 -- start start 2021-02-10T11:46:44.997 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.979 7f10290e3700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f10240d1870 0x7f10241b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:44.997 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.979 7f10290e3700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f10241b0ae0 0x7f1024108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:44.997 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.979 7f10290e3700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10240fe1c0 con 0x7f10240d1c70 2021-02-10T11:46:44.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.979 7f10290e3700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1024109a60 con 0x7f10241b0ae0 2021-02-10T11:46:44.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.979 7f10290e3700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1024109750 con 0x7f10240d1870 2021-02-10T11:46:44.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.979 7f102259c700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f10241b0ae0 0x7f1024108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:44.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.979 7f102259c700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f10241b0ae0 0x7f1024108940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:59818/0 (socket says 172.21.2.112:59818) 2021-02-10T11:46:44.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.979 7f102259c700 1 -- 172.21.2.112:0/3291239636 learned_addr learned my addr 172.21.2.112:0/3291239636 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:46:44.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.979 7f102359e700 1 --1- 172.21.2.112:0/3291239636 >> v1:172.21.2.112:6789/0 conn(0x7f10240d1c70 0x7f1024108f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:59282/0 (socket says 172.21.2.112:59282) 2021-02-10T11:46:44.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.979 7f102259c700 1 -- 172.21.2.112:0/3291239636 >> v2:172.21.2.120:3300/0 conn(0x7f10240d1870 msgr2=0x7f10241b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:46:44.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.979 7f1022d9d700 1 --2- 172.21.2.112:0/3291239636 >> v2:172.21.2.120:3300/0 conn(0x7f10240d1870 0x7f10241b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:44.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.979 7f102259c700 1 --2- 172.21.2.112:0/3291239636 >> v2:172.21.2.120:3300/0 conn(0x7f10240d1870 0x7f10241b04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:44.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.979 7f102259c700 1 -- 172.21.2.112:0/3291239636 >> v1:172.21.2.112:6789/0 conn(0x7f10240d1c70 legacy=0x7f1024108f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:44.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.979 7f102259c700 1 -- 172.21.2.112:0/3291239636 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1024088070 con 0x7f10241b0ae0 2021-02-10T11:46:44.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.979 7f1022d9d700 1 --2- 172.21.2.112:0/3291239636 >> v2:172.21.2.120:3300/0 conn(0x7f10240d1870 0x7f10241b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:46:44.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.979 7f102259c700 1 --2- 172.21.2.112:0/3291239636 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f10241b0ae0 0x7f1024108940 secure :-1 s=READY pgs=2933 cs=0 l=1 rev1=1 rx=0x7f1014006700 tx=0x7f1014009b80).ready entity=mon.0 client_cookie=6811538072be97a3 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:44.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.980 7f1021d9b700 1 -- 172.21.2.112:0/3291239636 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f101400c0c0 con 0x7f10241b0ae0 2021-02-10T11:46:44.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.980 7f1021d9b700 1 -- 172.21.2.112:0/3291239636 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f101400c340 con 0x7f10241b0ae0 2021-02-10T11:46:44.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.980 7f1021d9b700 1 -- 172.21.2.112:0/3291239636 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f101400c860 con 0x7f10241b0ae0 2021-02-10T11:46:44.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.980 7f10290e3700 1 -- 172.21.2.112:0/3291239636 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f10241b0ae0 msgr2=0x7f1024108940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:44.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.980 7f10290e3700 1 --2- 172.21.2.112:0/3291239636 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f10241b0ae0 0x7f1024108940 secure :-1 s=READY pgs=2933 cs=0 l=1 rev1=1 rx=0x7f1014006700 tx=0x7f1014009b80).stop 2021-02-10T11:46:44.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.980 7f10290e3700 1 -- 172.21.2.112:0/3291239636 shutdown_connections 2021-02-10T11:46:44.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.980 7f10290e3700 1 --2- 172.21.2.112:0/3291239636 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f10241b0ae0 0x7f1024108940 unknown :-1 s=CLOSED pgs=2933 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:44.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.980 7f10290e3700 1 --2- 172.21.2.112:0/3291239636 >> v2:172.21.2.120:3300/0 conn(0x7f10240d1870 0x7f10241b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:44.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.980 7f10290e3700 1 -- 172.21.2.112:0/3291239636 >> 172.21.2.112:0/3291239636 conn(0x7f10240ce420 msgr2=0x7f10240d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:44.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.980 7f10290e3700 1 -- 172.21.2.112:0/3291239636 shutdown_connections 2021-02-10T11:46:44.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.980 7f10290e3700 1 -- 172.21.2.112:0/3291239636 wait complete. 2021-02-10T11:46:45.001 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.995 7f10290e3700 1 Processor -- start 2021-02-10T11:46:45.001 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.995 7f10290e3700 1 -- start start 2021-02-10T11:46:45.001 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.995 7f10290e3700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f10240c3820 0x7f1024084cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:45.001 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.995 7f10290e3700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f10240a0290 0x7f10240852d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:45.001 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.995 7f10290e3700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1024086270 0x7f1024086670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:45.001 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.995 7f10290e3700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10240c0700 con 0x7f10240a0290 2021-02-10T11:46:45.001 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.995 7f10290e3700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10240858f0 con 0x7f10240c3820 2021-02-10T11:46:45.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.995 7f10290e3700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1024085b10 con 0x7f1024086270 2021-02-10T11:46:45.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.996 7f102259c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f10240a0290 0x7f10240852d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:45.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.996 7f102259c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f10240a0290 0x7f10240852d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:59826/0 (socket says 172.21.2.112:59826) 2021-02-10T11:46:45.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.996 7f102259c700 1 -- 172.21.2.112:0/3298537688 learned_addr learned my addr 172.21.2.112:0/3298537688 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:46:45.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.996 7f102259c700 1 -- 172.21.2.112:0/3298537688 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1024086270 msgr2=0x7f1024086670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:45.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.996 7f102259c700 1 --2- 172.21.2.112:0/3298537688 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1024086270 0x7f1024086670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:45.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.996 7f102259c700 1 -- 172.21.2.112:0/3298537688 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f10240c3820 msgr2=0x7f1024084cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:45.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.996 7f102259c700 1 --2- 172.21.2.112:0/3298537688 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f10240c3820 0x7f1024084cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:45.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.996 7f102259c700 1 -- 172.21.2.112:0/3298537688 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1024086c90 con 0x7f10240a0290 2021-02-10T11:46:45.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.996 7f102259c700 1 --2- 172.21.2.112:0/3298537688 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f10240a0290 0x7f10240852d0 secure :-1 s=READY pgs=2934 cs=0 l=1 rev1=1 rx=0x7f101400b5e0 tx=0x7f101400b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:45.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.996 7f101b7fe700 1 -- 172.21.2.112:0/3298537688 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f101400bd70 con 0x7f10240a0290 2021-02-10T11:46:45.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.996 7f101b7fe700 1 -- 172.21.2.112:0/3298537688 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1014002030 con 0x7f10240a0290 2021-02-10T11:46:45.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.996 7f10290e3700 1 -- 172.21.2.112:0/3298537688 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1024086c90 con 0x7f10240a0290 2021-02-10T11:46:45.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.996 7f101b7fe700 1 -- 172.21.2.112:0/3298537688 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1014002550 con 0x7f10240a0290 2021-02-10T11:46:45.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.997 7f10290e3700 1 -- 172.21.2.112:0/3298537688 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1024086c90 con 0x7f10240a0290 2021-02-10T11:46:45.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.997 7f101b7fe700 1 -- 172.21.2.112:0/3298537688 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1014002210 con 0x7f10240a0290 2021-02-10T11:46:45.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.997 7f101b7fe700 1 --2- 172.21.2.112:0/3298537688 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f10080186c0 0x7f100801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:45.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.997 7f101b7fe700 1 -- 172.21.2.112:0/3298537688 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f10140327d0 con 0x7f10240a0290 2021-02-10T11:46:45.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.998 7f1022d9d700 1 --2- 172.21.2.112:0/3298537688 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f10080186c0 0x7f100801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:45.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:44.998 7f1022d9d700 1 --2- 172.21.2.112:0/3298537688 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f10080186c0 0x7f100801abc0 crc :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:45.031 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:45.028 7f10290e3700 1 -- 172.21.2.112:0/3298537688 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1024082a70 con 0x7f10240a0290 2021-02-10T11:46:45.033 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:45.031 7f101b7fe700 1 -- 172.21.2.112:0/3298537688 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1014002280 con 0x7f10240a0290 2021-02-10T11:46:45.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:45.160 7f10290e3700 1 -- 172.21.2.112:0/3298537688 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "target": ["mgr", ""], "group_name": "group_0000000000707517"}) v1 -- 0x7f1024082a70 con 0x7f10080186c0 2021-02-10T11:46:45.168 INFO:tasks.ceph.mgr.x.gibba020.stderr:2021-02-10 11:46:45.165 7f6382eee700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:46:45.168 INFO:tasks.ceph.mgr.x.gibba020.stderr:2021-02-10 11:46:45.165 7f6382eee700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:46:45.168 INFO:tasks.ceph.mgr.x.gibba020.stderr:2021-02-10 11:46:45.165 7f6382eee700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:46:45.168 INFO:tasks.ceph.mgr.x.gibba020.stderr:2021-02-10 11:46:45.165 7f6382eee700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:46:45.168 INFO:tasks.ceph.mgr.x.gibba020.stderr:2021-02-10 11:46:45.165 7f6382eee700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:46:45.182 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:45.180 7f101b7fe700 1 -- 172.21.2.112:0/3298537688 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f100c0077a0 con 0x7f10080186c0 2021-02-10T11:46:45.196 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:45.194 7f10290e3700 1 -- 172.21.2.112:0/3298537688 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f10080186c0 msgr2=0x7f100801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:45.197 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:45.194 7f10290e3700 1 --2- 172.21.2.112:0/3298537688 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f10080186c0 0x7f100801abc0 crc :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:45.197 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:45.194 7f10290e3700 1 -- 172.21.2.112:0/3298537688 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f10240a0290 msgr2=0x7f10240852d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:45.197 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:45.194 7f10290e3700 1 --2- 172.21.2.112:0/3298537688 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f10240a0290 0x7f10240852d0 secure :-1 s=READY pgs=2934 cs=0 l=1 rev1=1 rx=0x7f101400b5e0 tx=0x7f101400b980).stop 2021-02-10T11:46:45.197 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:45.194 7f10290e3700 1 -- 172.21.2.112:0/3298537688 shutdown_connections 2021-02-10T11:46:45.197 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:45.194 7f10290e3700 1 --2- 172.21.2.112:0/3298537688 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f10080186c0 0x7f100801abc0 unknown :-1 s=CLOSED pgs=1327 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:45.197 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:45.194 7f10290e3700 1 --2- 172.21.2.112:0/3298537688 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1024086270 0x7f1024086670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:45.197 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:45.194 7f10290e3700 1 --2- 172.21.2.112:0/3298537688 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f10240a0290 0x7f10240852d0 unknown :-1 s=CLOSED pgs=2934 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:45.197 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:45.194 7f10290e3700 1 --2- 172.21.2.112:0/3298537688 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f10240c3820 0x7f1024084cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:45.197 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:45.194 7f10290e3700 1 -- 172.21.2.112:0/3298537688 >> 172.21.2.112:0/3298537688 conn(0x7f10240872c0 msgr2=0x7f1024095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:45.197 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:45.194 7f10290e3700 1 -- 172.21.2.112:0/3298537688 shutdown_connections 2021-02-10T11:46:45.197 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:45.194 7f10290e3700 1 -- 172.21.2.112:0/3298537688 wait complete. 2021-02-10T11:46:45.205 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000544322 --group_name group_0000000000707517 2021-02-10T11:46:45.642 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:45.619 7f56dd6bf700 1 Processor -- start 2021-02-10T11:46:45.643 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:45.619 7f56dd6bf700 1 -- start start 2021-02-10T11:46:45.643 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:45.619 7f56dd6bf700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f56d80d1c70 0x7f56d80cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:45.643 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:45.619 7f56dd6bf700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f56d80cf580 0x7f56d80cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:45.644 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:45.619 7f56dd6bf700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56d80fe1c0 con 0x7f56d80d1870 2021-02-10T11:46:45.644 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:45.619 7f56dd6bf700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56d81b0cb0 con 0x7f56d80d1c70 2021-02-10T11:46:45.644 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:45.619 7f56dd6bf700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56d8109a60 con 0x7f56d80cf580 2021-02-10T11:46:45.644 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:45.620 7f56d6abe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f56d80d1c70 0x7f56d80cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:45.644 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:45.620 7f56d6abe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f56d80d1c70 0x7f56d80cef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:59832/0 (socket says 172.21.2.112:59832) 2021-02-10T11:46:45.644 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:45.620 7f56d6abe700 1 -- 172.21.2.112:0/1483030375 learned_addr learned my addr 172.21.2.112:0/1483030375 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:46:45.645 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:45.620 7f56d7ac0700 1 --2- 172.21.2.112:0/1483030375 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f56d80cf580 0x7f56d80cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:45.645 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:45.620 7f56d6abe700 1 -- 172.21.2.112:0/1483030375 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f56d80cf580 msgr2=0x7f56d80cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:46:45.645 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:45.620 7f56d6abe700 1 --2- 172.21.2.112:0/1483030375 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f56d80cf580 0x7f56d80cf980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:45.645 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:45.620 7f56d6abe700 1 -- 172.21.2.112:0/1483030375 >> v1:172.21.2.120:6789/0 conn(0x7f56d80d1870 legacy=0x7f56d81b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:45.645 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:45.620 7f56d6abe700 1 -- 172.21.2.112:0/1483030375 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56d8088070 con 0x7f56d80d1c70 2021-02-10T11:46:45.645 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:45.620 7f56d6abe700 1 --2- 172.21.2.112:0/1483030375 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f56d80d1c70 0x7f56d80cef60 secure :-1 s=READY pgs=2935 cs=0 l=1 rev1=1 rx=0x7f56c8006700 tx=0x7f56c8009b80).ready entity=mon.0 client_cookie=d8ce540840a95b2 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:45.645 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:45.620 7f56d62bd700 1 -- 172.21.2.112:0/1483030375 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f56c800c0c0 con 0x7f56d80d1c70 2021-02-10T11:46:45.646 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:45.620 7f56d62bd700 1 -- 172.21.2.112:0/1483030375 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56c800c340 con 0x7f56d80d1c70 2021-02-10T11:46:45.646 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:45.620 7f56d62bd700 1 -- 172.21.2.112:0/1483030375 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f56c800c820 con 0x7f56d80d1c70 2021-02-10T11:46:45.646 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:45.620 7f56dd6bf700 1 -- 172.21.2.112:0/1483030375 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f56d80d1c70 msgr2=0x7f56d80cef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:45.646 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:45.620 7f56dd6bf700 1 --2- 172.21.2.112:0/1483030375 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f56d80d1c70 0x7f56d80cef60 secure :-1 s=READY pgs=2935 cs=0 l=1 rev1=1 rx=0x7f56c8006700 tx=0x7f56c8009b80).stop 2021-02-10T11:46:45.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:45.621 7f56dd6bf700 1 -- 172.21.2.112:0/1483030375 shutdown_connections 2021-02-10T11:46:45.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:45.621 7f56dd6bf700 1 --2- 172.21.2.112:0/1483030375 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f56d80d1c70 0x7f56d80cef60 unknown :-1 s=CLOSED pgs=2935 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:45.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:45.621 7f56dd6bf700 1 --2- 172.21.2.112:0/1483030375 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f56d80cf580 0x7f56d80cf980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:45.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:45.621 7f56dd6bf700 1 -- 172.21.2.112:0/1483030375 >> 172.21.2.112:0/1483030375 conn(0x7f56d80ce420 msgr2=0x7f56d80d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:45.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:45.621 7f56dd6bf700 1 -- 172.21.2.112:0/1483030375 shutdown_connections 2021-02-10T11:46:45.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:45.621 7f56dd6bf700 1 -- 172.21.2.112:0/1483030375 wait complete. 2021-02-10T11:46:45.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:45.636 7f56dd6bf700 1 Processor -- start 2021-02-10T11:46:45.648 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:45.636 7f56dd6bf700 1 -- start start 2021-02-10T11:46:45.648 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:45.636 7f56dd6bf700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f56d81a05f0 0x7f56d80a8f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:45.650 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:45.636 7f56dd6bf700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f56d80aba50 0x7f56d80b3090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:45.650 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:45.636 7f56dd6bf700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f56d80b36b0 0x7f56d80b42a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:45.650 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:45.636 7f56dd6bf700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56d80b3d20 con 0x7f56d80b36b0 2021-02-10T11:46:45.650 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:45.636 7f56dd6bf700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56d80abe50 con 0x7f56d81a05f0 2021-02-10T11:46:45.650 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:45.636 7f56dd6bf700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56d80b23c0 con 0x7f56d80aba50 2021-02-10T11:46:45.650 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:45.636 7f56d7ac0700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f56d80b36b0 0x7f56d80b42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:45.651 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:45.636 7f56d72bf700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f56d81a05f0 0x7f56d80a8f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:45.651 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:45.637 7f56d7ac0700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f56d80b36b0 0x7f56d80b42a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:59840/0 (socket says 172.21.2.112:59840) 2021-02-10T11:46:45.651 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:45.637 7f56d7ac0700 1 -- 172.21.2.112:0/2276504440 learned_addr learned my addr 172.21.2.112:0/2276504440 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:46:45.651 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:45.637 7f56d72bf700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f56d81a05f0 0x7f56d80a8f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3300/0 says I am v2:172.21.2.112:33884/0 (socket says 172.21.2.112:33884) 2021-02-10T11:46:45.651 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:45.637 7f56d6abe700 1 --2- 172.21.2.112:0/2276504440 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f56d80aba50 0x7f56d80b3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:45.652 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:45.637 7f56d7ac0700 1 -- 172.21.2.112:0/2276504440 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f56d80aba50 msgr2=0x7f56d80b3090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:45.652 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:45.637 7f56d7ac0700 1 --2- 172.21.2.112:0/2276504440 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f56d80aba50 0x7f56d80b3090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:45.652 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:45.637 7f56d7ac0700 1 -- 172.21.2.112:0/2276504440 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f56d81a05f0 msgr2=0x7f56d80a8f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:45.652 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:45.637 7f56d7ac0700 1 --2- 172.21.2.112:0/2276504440 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f56d81a05f0 0x7f56d80a8f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:45.652 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:45.637 7f56d7ac0700 1 -- 172.21.2.112:0/2276504440 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56d80b48c0 con 0x7f56d80b36b0 2021-02-10T11:46:45.652 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:45.637 7f56d72bf700 1 --2- 172.21.2.112:0/2276504440 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f56d81a05f0 0x7f56d80a8f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:46:45.652 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:45.637 7f56d7ac0700 1 --2- 172.21.2.112:0/2276504440 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f56d80b36b0 0x7f56d80b42a0 secure :-1 s=READY pgs=2936 cs=0 l=1 rev1=1 rx=0x7f56d00099a0 tx=0x7f56d000bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:45.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:45.637 7f56cffff700 1 -- 172.21.2.112:0/2276504440 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f56d00066f0 con 0x7f56d80b36b0 2021-02-10T11:46:45.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:45.637 7f56cffff700 1 -- 172.21.2.112:0/2276504440 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56d0003030 con 0x7f56d80b36b0 2021-02-10T11:46:45.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:45.637 7f56cffff700 1 -- 172.21.2.112:0/2276504440 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f56d00013e0 con 0x7f56d80b36b0 2021-02-10T11:46:45.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:45.638 7f56dd6bf700 1 -- 172.21.2.112:0/2276504440 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56d80b48c0 con 0x7f56d80b36b0 2021-02-10T11:46:45.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:45.638 7f56dd6bf700 1 -- 172.21.2.112:0/2276504440 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56d80b48c0 con 0x7f56d80b36b0 2021-02-10T11:46:45.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:45.638 7f56cffff700 1 -- 172.21.2.112:0/2276504440 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f56d00011f0 con 0x7f56d80b36b0 2021-02-10T11:46:45.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:45.638 7f56cffff700 1 --2- 172.21.2.112:0/2276504440 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f56bc0186c0 0x7f56bc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:45.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:45.638 7f56cffff700 1 -- 172.21.2.112:0/2276504440 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f56d00327a0 con 0x7f56d80b36b0 2021-02-10T11:46:45.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:45.639 7f56d72bf700 1 --2- 172.21.2.112:0/2276504440 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f56bc0186c0 0x7f56bc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:45.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:45.639 7f56d72bf700 1 --2- 172.21.2.112:0/2276504440 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f56bc0186c0 0x7f56bc01abc0 crc :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:45.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:45.669 7f56dd6bf700 1 -- 172.21.2.112:0/2276504440 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56d80825d0 con 0x7f56d80b36b0 2021-02-10T11:46:45.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:45.672 7f56cffff700 1 -- 172.21.2.112:0/2276504440 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f56d0020ef0 con 0x7f56d80b36b0 2021-02-10T11:46:45.804 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:45.801 7f56dd6bf700 1 -- 172.21.2.112:0/2276504440 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000544322", "target": ["mgr", ""], "group_name": "group_0000000000707517"}) v1 -- 0x7f56d80825d0 con 0x7f56bc0186c0 2021-02-10T11:46:45.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:45.826 7f56cffff700 1 -- 172.21.2.112:0/2276504440 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f56c0000960 con 0x7f56bc0186c0 2021-02-10T11:46:45.838 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:45.834 7f56dd6bf700 1 -- 172.21.2.112:0/2276504440 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f56bc0186c0 msgr2=0x7f56bc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:45.839 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:45.834 7f56dd6bf700 1 --2- 172.21.2.112:0/2276504440 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f56bc0186c0 0x7f56bc01abc0 crc :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:45.839 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:45.834 7f56dd6bf700 1 -- 172.21.2.112:0/2276504440 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f56d80b36b0 msgr2=0x7f56d80b42a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:45.839 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:45.834 7f56dd6bf700 1 --2- 172.21.2.112:0/2276504440 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f56d80b36b0 0x7f56d80b42a0 secure :-1 s=READY pgs=2936 cs=0 l=1 rev1=1 rx=0x7f56d00099a0 tx=0x7f56d000bcf0).stop 2021-02-10T11:46:45.839 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:45.834 7f56dd6bf700 1 -- 172.21.2.112:0/2276504440 shutdown_connections 2021-02-10T11:46:45.839 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:45.834 7f56dd6bf700 1 --2- 172.21.2.112:0/2276504440 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f56bc0186c0 0x7f56bc01abc0 unknown :-1 s=CLOSED pgs=1328 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:45.839 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:45.834 7f56dd6bf700 1 --2- 172.21.2.112:0/2276504440 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f56d80b36b0 0x7f56d80b42a0 unknown :-1 s=CLOSED pgs=2936 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:45.839 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:45.834 7f56dd6bf700 1 --2- 172.21.2.112:0/2276504440 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f56d80aba50 0x7f56d80b3090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:45.839 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:45.834 7f56dd6bf700 1 --2- 172.21.2.112:0/2276504440 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f56d81a05f0 0x7f56d80a8f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:45.839 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:45.834 7f56dd6bf700 1 -- 172.21.2.112:0/2276504440 >> 172.21.2.112:0/2276504440 conn(0x7f56d80bc070 msgr2=0x7f56d80ba770 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:45.839 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:45.834 7f56dd6bf700 1 -- 172.21.2.112:0/2276504440 shutdown_connections 2021-02-10T11:46:45.839 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:45.835 7f56dd6bf700 1 -- 172.21.2.112:0/2276504440 wait complete. 2021-02-10T11:46:45.845 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume authorize cephfs subvolume_0000000000544322 guest1 --group_name group_0000000000707517 --tenant_id tenant1 2021-02-10T11:46:46.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:46.275 7fd7ca9b5700 1 Processor -- start 2021-02-10T11:46:46.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:46.276 7fd7ca9b5700 1 -- start start 2021-02-10T11:46:46.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:46.276 7fd7ca9b5700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fd7c40f06d0 0x7fd7c40f1100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:46.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:46.276 7fd7ca9b5700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7c40f0b80 con 0x7fd7c40d18d0 2021-02-10T11:46:46.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:46.276 7fd7ca9b5700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7c40d1cd0 con 0x7fd7c40cffd0 2021-02-10T11:46:46.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:46.276 7fd7ca9b5700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7c40d1ef0 con 0x7fd7c40f06d0 2021-02-10T11:46:46.295 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:46.276 7fd7c8fd2700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fd7c40f06d0 0x7fd7c40f1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:46.295 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:46.276 7fd7c8fd2700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fd7c40f06d0 0x7fd7c40f1100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:59848/0 (socket says 172.21.2.112:59848) 2021-02-10T11:46:46.295 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:46.276 7fd7c8fd2700 1 -- 172.21.2.112:0/4133198160 learned_addr learned my addr 172.21.2.112:0/4133198160 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:46:46.295 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:46.277 7fd7c8fd2700 1 -- 172.21.2.112:0/4133198160 >> v1:172.21.2.120:6789/0 conn(0x7fd7c40cffd0 legacy=0x7fd7c41b3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:46:46.295 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:46.277 7fd7c8fd2700 1 -- 172.21.2.112:0/4133198160 >> v1:172.21.2.112:6789/0 conn(0x7fd7c40d18d0 legacy=0x7fd7c41b3280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:46.295 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:46.277 7fd7c8fd2700 1 -- 172.21.2.112:0/4133198160 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7c40d20e0 con 0x7fd7c40f06d0 2021-02-10T11:46:46.295 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:46.277 7fd7c8fd2700 1 --2- 172.21.2.112:0/4133198160 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd7c40f06d0 0x7fd7c40f1100 secure :-1 s=READY pgs=2938 cs=0 l=1 rev1=1 rx=0x7fd7bc006f00 tx=0x7fd7bc009f10).ready entity=mon.0 client_cookie=a289134ea083741a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:46.295 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:46.277 7fd7c2ffd700 1 -- 172.21.2.112:0/4133198160 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd7bc00b3b0 con 0x7fd7c40f06d0 2021-02-10T11:46:46.295 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:46.277 7fd7c2ffd700 1 -- 172.21.2.112:0/4133198160 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7bc00b630 con 0x7fd7c40f06d0 2021-02-10T11:46:46.295 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:46.277 7fd7c2ffd700 1 -- 172.21.2.112:0/4133198160 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd7bc00bb50 con 0x7fd7c40f06d0 2021-02-10T11:46:46.295 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:46.277 7fd7ca9b5700 1 -- 172.21.2.112:0/4133198160 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd7c40f06d0 msgr2=0x7fd7c40f1100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:46.295 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:46.277 7fd7ca9b5700 1 --2- 172.21.2.112:0/4133198160 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd7c40f06d0 0x7fd7c40f1100 secure :-1 s=READY pgs=2938 cs=0 l=1 rev1=1 rx=0x7fd7bc006f00 tx=0x7fd7bc009f10).stop 2021-02-10T11:46:46.295 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:46.277 7fd7ca9b5700 1 -- 172.21.2.112:0/4133198160 shutdown_connections 2021-02-10T11:46:46.296 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:46.277 7fd7ca9b5700 1 --2- 172.21.2.112:0/4133198160 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd7c40f06d0 0x7fd7c40f1100 unknown :-1 s=CLOSED pgs=2938 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:46.296 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:46.277 7fd7ca9b5700 1 -- 172.21.2.112:0/4133198160 >> 172.21.2.112:0/4133198160 conn(0x7fd7c40ce480 msgr2=0x7fd7c40d0840 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:46.296 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:46.277 7fd7ca9b5700 1 -- 172.21.2.112:0/4133198160 shutdown_connections 2021-02-10T11:46:46.296 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:46.277 7fd7ca9b5700 1 -- 172.21.2.112:0/4133198160 wait complete. 2021-02-10T11:46:46.297 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:46.292 7fd7ca9b5700 1 Processor -- start 2021-02-10T11:46:46.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:46.292 7fd7ca9b5700 1 -- start start 2021-02-10T11:46:46.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:46.292 7fd7ca9b5700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd7c41b2050 0x7fd7c409b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:46.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:46.292 7fd7ca9b5700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd7c41b2730 0x7fd7c408c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:46.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:46.292 7fd7ca9b5700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd7c41b2b30 0x7fd7c408c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:46.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:46.292 7fd7ca9b5700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7c408d660 con 0x7fd7c41b2730 2021-02-10T11:46:46.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:46.292 7fd7ca9b5700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7c409c0f0 con 0x7fd7c41b2050 2021-02-10T11:46:46.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:46.292 7fd7ca9b5700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7c41b2450 con 0x7fd7c41b2b30 2021-02-10T11:46:46.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:46.293 7fd7c37fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd7c41b2730 0x7fd7c408c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:46.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:46.293 7fd7c37fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd7c41b2730 0x7fd7c408c2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:59854/0 (socket says 172.21.2.112:59854) 2021-02-10T11:46:46.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:46.293 7fd7c37fe700 1 -- 172.21.2.112:0/2824917320 learned_addr learned my addr 172.21.2.112:0/2824917320 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:46:46.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:46.293 7fd7c3fff700 1 --2- 172.21.2.112:0/2824917320 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd7c41b2050 0x7fd7c409b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:46.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:46.293 7fd7c8fd2700 1 --2- 172.21.2.112:0/2824917320 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd7c41b2b30 0x7fd7c408c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:46.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:46.293 7fd7c3fff700 1 -- 172.21.2.112:0/2824917320 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd7c41b2b30 msgr2=0x7fd7c408c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:46.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:46.293 7fd7c3fff700 1 --2- 172.21.2.112:0/2824917320 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd7c41b2b30 0x7fd7c408c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:46.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:46.293 7fd7c3fff700 1 -- 172.21.2.112:0/2824917320 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd7c41b2730 msgr2=0x7fd7c408c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:46.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:46.293 7fd7c3fff700 1 --2- 172.21.2.112:0/2824917320 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd7c41b2730 0x7fd7c408c2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:46.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:46.293 7fd7c3fff700 1 -- 172.21.2.112:0/2824917320 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7c40c3d10 con 0x7fd7c41b2050 2021-02-10T11:46:46.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:46.293 7fd7c37fe700 1 --2- 172.21.2.112:0/2824917320 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd7c41b2730 0x7fd7c408c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:46:46.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:46.293 7fd7c3fff700 1 --2- 172.21.2.112:0/2824917320 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd7c41b2050 0x7fd7c409b9c0 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 rx=0x7fd7b4000d50 tx=0x7fd7b400b8e0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:46.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:46.294 7fd7c0ff9700 1 -- 172.21.2.112:0/2824917320 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd7b400bcd0 con 0x7fd7c41b2050 2021-02-10T11:46:46.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:46.294 7fd7c0ff9700 1 -- 172.21.2.112:0/2824917320 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7b4006240 con 0x7fd7c41b2050 2021-02-10T11:46:46.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:46.294 7fd7c0ff9700 1 -- 172.21.2.112:0/2824917320 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd7b4005730 con 0x7fd7c41b2050 2021-02-10T11:46:46.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:46.294 7fd7ca9b5700 1 -- 172.21.2.112:0/2824917320 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7c40c3d10 con 0x7fd7c41b2050 2021-02-10T11:46:46.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:46.294 7fd7ca9b5700 1 -- 172.21.2.112:0/2824917320 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7c408d1e0 con 0x7fd7c41b2050 2021-02-10T11:46:46.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:46.294 7fd7c0ff9700 1 -- 172.21.2.112:0/2824917320 <== mon.1 v2:172.21.2.120:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd7b4005c00 con 0x7fd7c41b2050 2021-02-10T11:46:46.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:46.294 7fd7c0ff9700 1 --2- 172.21.2.112:0/2824917320 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd7a40186c0 0x7fd7a401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:46.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:46.294 7fd7c0ff9700 1 -- 172.21.2.112:0/2824917320 <== mon.1 v2:172.21.2.120:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fd7b40324c0 con 0x7fd7c41b2050 2021-02-10T11:46:46.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:46.295 7fd7c37fe700 1 --2- 172.21.2.112:0/2824917320 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd7a40186c0 0x7fd7a401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:46.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:46.295 7fd7c37fe700 1 --2- 172.21.2.112:0/2824917320 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd7a40186c0 0x7fd7a401abc0 crc :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:46.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:46.325 7fd7ca9b5700 1 -- 172.21.2.112:0/2824917320 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7c410a280 con 0x7fd7c41b2050 2021-02-10T11:46:46.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:46.329 7fd7c0ff9700 1 -- 172.21.2.112:0/2824917320 <== mon.1 v2:172.21.2.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd7b4016820 con 0x7fd7c41b2050 2021-02-10T11:46:46.460 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:46.457 7fd7ca9b5700 1 -- 172.21.2.112:0/2824917320 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume authorize", "vol_name": "cephfs", "sub_name": "subvolume_0000000000544322", "target": ["mgr", ""], "tenant_id": "tenant1", "auth_id": "guest1", "group_name": "group_0000000000707517"}) v1 -- 0x7fd7c408d1e0 con 0x7fd7a40186c0 2021-02-10T11:46:46.465 INFO:tasks.ceph.mgr.x.gibba020.stderr:Creating meta for ID guest1 with tenant tenant1 2021-02-10T11:46:46.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:46.494 7fd7c0ff9700 1 -- 172.21.2.112:0/2824917320 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+41 (crc 0 0 0) 0x7fd7b8000e20 con 0x7fd7a40186c0 2021-02-10T11:46:46.524 INFO:teuthology.orchestra.run.gibba012.stdout:AQCmxyNgrzdXHBAAacZLqUPv1dFaPAGJVBErLw== 2021-02-10T11:46:46.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:46.523 7fd7ca9b5700 1 -- 172.21.2.112:0/2824917320 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd7a40186c0 msgr2=0x7fd7a401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:46.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:46.523 7fd7ca9b5700 1 --2- 172.21.2.112:0/2824917320 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd7a40186c0 0x7fd7a401abc0 crc :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:46.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:46.523 7fd7ca9b5700 1 -- 172.21.2.112:0/2824917320 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd7c41b2050 msgr2=0x7fd7c409b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:46.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:46.523 7fd7ca9b5700 1 --2- 172.21.2.112:0/2824917320 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd7c41b2050 0x7fd7c409b9c0 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 rx=0x7fd7b4000d50 tx=0x7fd7b400b8e0).stop 2021-02-10T11:46:46.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:46.523 7fd7ca9b5700 1 -- 172.21.2.112:0/2824917320 shutdown_connections 2021-02-10T11:46:46.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:46.523 7fd7ca9b5700 1 --2- 172.21.2.112:0/2824917320 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd7a40186c0 0x7fd7a401abc0 unknown :-1 s=CLOSED pgs=1329 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:46.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:46.523 7fd7ca9b5700 1 --2- 172.21.2.112:0/2824917320 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd7c41b2b30 0x7fd7c408c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:46.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:46.523 7fd7ca9b5700 1 --2- 172.21.2.112:0/2824917320 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd7c41b2730 0x7fd7c408c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:46.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:46.523 7fd7ca9b5700 1 --2- 172.21.2.112:0/2824917320 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd7c41b2050 0x7fd7c409b9c0 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:46.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:46.523 7fd7ca9b5700 1 -- 172.21.2.112:0/2824917320 >> 172.21.2.112:0/2824917320 conn(0x7fd7c409ca50 msgr2=0x7fd7c408af90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:46.528 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:46.524 7fd7ca9b5700 1 -- 172.21.2.112:0/2824917320 shutdown_connections 2021-02-10T11:46:46.528 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:46.524 7fd7ca9b5700 1 -- 172.21.2.112:0/2824917320 wait complete. 2021-02-10T11:46:46.534 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.1 && sudo ls volumes 2021-02-10T11:46:46.883 INFO:teuthology.orchestra.run.gibba012.stdout:group_0000000000707517 2021-02-10T11:46:46.883 INFO:teuthology.orchestra.run.gibba012.stdout:_group_0000000000707517:subvolume_0000000000544322.meta 2021-02-10T11:46:46.884 INFO:teuthology.orchestra.run.gibba012.stdout:$guest1.meta 2021-02-10T11:46:46.885 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo sudo cat '/home/ubuntu/cephtest/mnt.0/volumes/$guest1.meta' 2021-02-10T11:46:47.250 INFO:teuthology.orchestra.run.gibba012.stdout:{"compat_version": 1, "tenant_id": "tenant1", "version": 5, "dirty": false, "subvolumes": {"group_0000000000707517/subvolume_0000000000544322": {"access_level": "rw", "dirty": false}}} 2021-02-10T11:46:47.250 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.1 && sudo sed -i s/false/true/g 'volumes/$guest1.meta' 2021-02-10T11:46:47.615 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume authorize cephfs subvolume_0000000000544322 guest1 --group_name group_0000000000707517 --tenant_id tenant1 2021-02-10T11:46:48.055 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:48.036 7f190b626700 1 Processor -- start 2021-02-10T11:46:48.056 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:48.037 7f190b626700 1 -- start start 2021-02-10T11:46:48.056 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:48.037 7f190b626700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f19040e53a0 0x7f19041b0550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:48.056 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:48.037 7f190b626700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f19041b0b70 0x7f19041089d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:48.056 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:48.037 7f190b626700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1904109ac0 con 0x7f1904108ff0 2021-02-10T11:46:48.056 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:48.037 7f190b626700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19040e5f80 con 0x7f19041b0b70 2021-02-10T11:46:48.056 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:48.037 7f190b626700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19040e57a0 con 0x7f19040e53a0 2021-02-10T11:46:48.056 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:48.037 7f1908c41700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f19041b0b70 0x7f19041089d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:48.056 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:48.037 7f1908c41700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f19041b0b70 0x7f19041089d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:59864/0 (socket says 172.21.2.112:59864) 2021-02-10T11:46:48.056 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:48.037 7f1908c41700 1 -- 172.21.2.112:0/2312196655 learned_addr learned my addr 172.21.2.112:0/2312196655 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:46:48.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:48.038 7f1908c41700 1 -- 172.21.2.112:0/2312196655 >> v2:172.21.2.120:3300/0 conn(0x7f19040e53a0 msgr2=0x7f19041b0550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:46:48.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:48.038 7f1908c41700 1 --2- 172.21.2.112:0/2312196655 >> v2:172.21.2.120:3300/0 conn(0x7f19040e53a0 0x7f19041b0550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:48.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:48.038 7f1908c41700 1 -- 172.21.2.112:0/2312196655 >> v1:172.21.2.112:6789/0 conn(0x7f1904108ff0 legacy=0x7f190410a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:48.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:48.038 7f1908c41700 1 -- 172.21.2.112:0/2312196655 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1904088070 con 0x7f19041b0b70 2021-02-10T11:46:48.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:48.038 7f1908c41700 1 --2- 172.21.2.112:0/2312196655 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f19041b0b70 0x7f19041089d0 secure :-1 s=READY pgs=2940 cs=0 l=1 rev1=1 rx=0x7f18f4006700 tx=0x7f18f4009b00).ready entity=mon.0 client_cookie=71e73345bc136151 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:48.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:48.038 7f18fbfff700 1 -- 172.21.2.112:0/2312196655 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f18f400c0c0 con 0x7f19041b0b70 2021-02-10T11:46:48.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:48.038 7f18fbfff700 1 -- 172.21.2.112:0/2312196655 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18f400c340 con 0x7f19041b0b70 2021-02-10T11:46:48.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:48.038 7f18fbfff700 1 -- 172.21.2.112:0/2312196655 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f18f400c820 con 0x7f19041b0b70 2021-02-10T11:46:48.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:48.038 7f190b626700 1 -- 172.21.2.112:0/2312196655 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f19041b0b70 msgr2=0x7f19041089d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:48.058 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:48.038 7f190b626700 1 --2- 172.21.2.112:0/2312196655 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f19041b0b70 0x7f19041089d0 secure :-1 s=READY pgs=2940 cs=0 l=1 rev1=1 rx=0x7f18f4006700 tx=0x7f18f4009b00).stop 2021-02-10T11:46:48.058 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:48.039 7f190b626700 1 -- 172.21.2.112:0/2312196655 shutdown_connections 2021-02-10T11:46:48.058 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:48.039 7f190b626700 1 --2- 172.21.2.112:0/2312196655 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f19041b0b70 0x7f19041089d0 unknown :-1 s=CLOSED pgs=2940 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:48.058 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:48.039 7f190b626700 1 --2- 172.21.2.112:0/2312196655 >> v2:172.21.2.120:3300/0 conn(0x7f19040e53a0 0x7f19041b0550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:48.058 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:48.039 7f190b626700 1 -- 172.21.2.112:0/2312196655 >> 172.21.2.112:0/2312196655 conn(0x7f19040ce4b0 msgr2=0x7f19040e45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:48.058 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:48.039 7f190b626700 1 -- 172.21.2.112:0/2312196655 shutdown_connections 2021-02-10T11:46:48.058 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:48.039 7f190b626700 1 -- 172.21.2.112:0/2312196655 wait complete. 2021-02-10T11:46:48.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:48.053 7f190b626700 1 Processor -- start 2021-02-10T11:46:48.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:48.053 7f190b626700 1 -- start start 2021-02-10T11:46:48.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:48.054 7f190b626700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f19041a8820 0x7f19041a8c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:48.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:48.054 7f190b626700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f190408c6d0 0x7f190408cad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:48.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:48.054 7f190b626700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f190408d0f0 0x7f190408d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:48.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:48.054 7f190b626700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19041a93a0 con 0x7f19041a8820 2021-02-10T11:46:48.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:48.054 7f190b626700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19041a9920 con 0x7f190408c6d0 2021-02-10T11:46:48.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:48.054 7f190b626700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19041a9b00 con 0x7f190408d0f0 2021-02-10T11:46:48.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:48.054 7f1909442700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f19041a8820 0x7f19041a8c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:48.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:48.054 7f1909442700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f19041a8820 0x7f19041a8c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:59868/0 (socket says 172.21.2.112:59868) 2021-02-10T11:46:48.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:48.054 7f1909442700 1 -- 172.21.2.112:0/1983200861 learned_addr learned my addr 172.21.2.112:0/1983200861 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:46:48.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:48.055 7f1909442700 1 -- 172.21.2.112:0/1983200861 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f190408d0f0 msgr2=0x7f190408d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:48.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:48.055 7f1909442700 1 --2- 172.21.2.112:0/1983200861 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f190408d0f0 0x7f190408d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:48.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:48.055 7f1909442700 1 -- 172.21.2.112:0/1983200861 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f190408c6d0 msgr2=0x7f190408cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:48.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:48.055 7f1909442700 1 --2- 172.21.2.112:0/1983200861 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f190408c6d0 0x7f190408cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:48.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:48.055 7f1909442700 1 -- 172.21.2.112:0/1983200861 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19041a9d20 con 0x7f19041a8820 2021-02-10T11:46:48.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:48.055 7f1909442700 1 --2- 172.21.2.112:0/1983200861 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f19041a8820 0x7f19041a8c20 secure :-1 s=READY pgs=2941 cs=0 l=1 rev1=1 rx=0x7f190000b530 tx=0x7f190000b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:48.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:48.055 7f18f9ffb700 1 -- 172.21.2.112:0/1983200861 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1900006350 con 0x7f19041a8820 2021-02-10T11:46:48.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:48.055 7f18f9ffb700 1 -- 172.21.2.112:0/1983200861 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1900006c20 con 0x7f19041a8820 2021-02-10T11:46:48.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:48.055 7f18f9ffb700 1 -- 172.21.2.112:0/1983200861 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1900000c90 con 0x7f19041a8820 2021-02-10T11:46:48.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:48.055 7f190b626700 1 -- 172.21.2.112:0/1983200861 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19041a9d20 con 0x7f19041a8820 2021-02-10T11:46:48.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:48.055 7f190b626700 1 -- 172.21.2.112:0/1983200861 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19041aa2b0 con 0x7f19041a8820 2021-02-10T11:46:48.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:48.056 7f18f9ffb700 1 -- 172.21.2.112:0/1983200861 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1900000c90 con 0x7f19041a8820 2021-02-10T11:46:48.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:48.056 7f18f9ffb700 1 --2- 172.21.2.112:0/1983200861 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f18e40186c0 0x7f18e401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:48.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:48.056 7f18f9ffb700 1 -- 172.21.2.112:0/1983200861 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f1900006b90 con 0x7f19041a8820 2021-02-10T11:46:48.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:48.056 7f1908c41700 1 --2- 172.21.2.112:0/1983200861 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f18e40186c0 0x7f18e401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:48.062 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:48.057 7f1908c41700 1 --2- 172.21.2.112:0/1983200861 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f18e40186c0 0x7f18e401abc0 crc :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:48.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:48.087 7f190b626700 1 -- 172.21.2.112:0/1983200861 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1904103810 con 0x7f19041a8820 2021-02-10T11:46:48.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:48.090 7f18f9ffb700 1 -- 172.21.2.112:0/1983200861 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1900006350 con 0x7f19041a8820 2021-02-10T11:46:48.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:48.215 7f190b626700 1 -- 172.21.2.112:0/1983200861 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume authorize", "vol_name": "cephfs", "sub_name": "subvolume_0000000000544322", "target": ["mgr", ""], "tenant_id": "tenant1", "auth_id": "guest1", "group_name": "group_0000000000707517"}) v1 -- 0x7f1904111fa0 con 0x7f18e40186c0 2021-02-10T11:46:48.280 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:48.278 7f18f9ffb700 1 -- 172.21.2.112:0/1983200861 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+41 (crc 0 0 0) 0x7f18f4009550 con 0x7f18e40186c0 2021-02-10T11:46:48.281 INFO:teuthology.orchestra.run.gibba012.stdout:AQCmxyNgrzdXHBAAacZLqUPv1dFaPAGJVBErLw== 2021-02-10T11:46:48.283 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:48.280 7f190b626700 1 -- 172.21.2.112:0/1983200861 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f18e40186c0 msgr2=0x7f18e401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:48.283 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:48.280 7f190b626700 1 --2- 172.21.2.112:0/1983200861 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f18e40186c0 0x7f18e401abc0 crc :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:48.283 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:48.280 7f190b626700 1 -- 172.21.2.112:0/1983200861 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f19041a8820 msgr2=0x7f19041a8c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:48.283 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:48.280 7f190b626700 1 --2- 172.21.2.112:0/1983200861 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f19041a8820 0x7f19041a8c20 secure :-1 s=READY pgs=2941 cs=0 l=1 rev1=1 rx=0x7f190000b530 tx=0x7f190000b8d0).stop 2021-02-10T11:46:48.283 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:48.280 7f190b626700 1 -- 172.21.2.112:0/1983200861 shutdown_connections 2021-02-10T11:46:48.283 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:48.280 7f190b626700 1 --2- 172.21.2.112:0/1983200861 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f18e40186c0 0x7f18e401abc0 unknown :-1 s=CLOSED pgs=1330 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:48.283 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:48.280 7f190b626700 1 --2- 172.21.2.112:0/1983200861 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f190408c6d0 0x7f190408cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:48.283 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:48.280 7f190b626700 1 --2- 172.21.2.112:0/1983200861 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f190408d0f0 0x7f190408d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:48.283 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:48.280 7f190b626700 1 --2- 172.21.2.112:0/1983200861 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f19041a8820 0x7f19041a8c20 unknown :-1 s=CLOSED pgs=2941 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:48.284 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:48.280 7f190b626700 1 -- 172.21.2.112:0/1983200861 >> 172.21.2.112:0/1983200861 conn(0x7f190418cb60 msgr2=0x7f1904199f40 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:48.284 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:48.280 7f190b626700 1 -- 172.21.2.112:0/1983200861 shutdown_connections 2021-02-10T11:46:48.284 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:48.280 7f190b626700 1 -- 172.21.2.112:0/1983200861 wait complete. 2021-02-10T11:46:48.291 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo sudo cat '/home/ubuntu/cephtest/mnt.0/volumes/$guest1.meta' 2021-02-10T11:46:48.651 INFO:teuthology.orchestra.run.gibba012.stdout:{"tenant_id": "tenant1", "version": 5, "compat_version": 1, "subvolumes": {"group_0000000000707517/subvolume_0000000000544322": {"access_level": "rw", "dirty": false}}, "dirty": false} 2021-02-10T11:46:48.652 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume deauthorize cephfs subvolume_0000000000544322 guest1 --group_name group_0000000000707517 2021-02-10T11:46:49.084 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:49.066 7fe438680700 1 Processor -- start 2021-02-10T11:46:49.084 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:49.066 7fe438680700 1 -- start start 2021-02-10T11:46:49.085 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:49.066 7fe438680700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fe4300eb220 0x7fe4300eb620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:49.085 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:49.066 7fe438680700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fe43010aad0 0x7fe43010aed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:49.085 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:49.066 7fe438680700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe43010b4f0 con 0x7fe4300dce80 2021-02-10T11:46:49.085 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:49.066 7fe438680700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe43010a550 con 0x7fe43010aad0 2021-02-10T11:46:49.085 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:49.066 7fe438680700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4300881d0 con 0x7fe4300eb220 2021-02-10T11:46:49.085 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:49.066 7fe436c9d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fe43010aad0 0x7fe43010aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:49.085 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:49.066 7fe436c9d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fe43010aad0 0x7fe43010aed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:59882/0 (socket says 172.21.2.112:59882) 2021-02-10T11:46:49.085 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:49.066 7fe436c9d700 1 -- 172.21.2.112:0/119661398 learned_addr learned my addr 172.21.2.112:0/119661398 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:46:49.085 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:49.067 7fe436c9d700 1 -- 172.21.2.112:0/119661398 >> v2:172.21.2.120:3300/0 conn(0x7fe4300eb220 msgr2=0x7fe4300eb620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:46:49.085 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:49.067 7fe436c9d700 1 --2- 172.21.2.112:0/119661398 >> v2:172.21.2.120:3300/0 conn(0x7fe4300eb220 0x7fe4300eb620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:49.085 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:49.067 7fe436c9d700 1 -- 172.21.2.112:0/119661398 >> v1:172.21.2.112:6789/0 conn(0x7fe4300dce80 legacy=0x7fe4300d14f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:49.085 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:49.067 7fe436c9d700 1 -- 172.21.2.112:0/119661398 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4300d1f30 con 0x7fe43010aad0 2021-02-10T11:46:49.085 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:49.067 7fe436c9d700 1 --2- 172.21.2.112:0/119661398 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe43010aad0 0x7fe43010aed0 secure :-1 s=READY pgs=2943 cs=0 l=1 rev1=1 rx=0x7fe428006f00 tx=0x7fe428009f10).ready entity=mon.0 client_cookie=7138824224ff2975 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:49.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:49.067 7fe43549a700 1 -- 172.21.2.112:0/119661398 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe42800b3b0 con 0x7fe43010aad0 2021-02-10T11:46:49.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:49.067 7fe43549a700 1 -- 172.21.2.112:0/119661398 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe42800b730 con 0x7fe43010aad0 2021-02-10T11:46:49.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:49.067 7fe43549a700 1 -- 172.21.2.112:0/119661398 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe42800bb90 con 0x7fe43010aad0 2021-02-10T11:46:49.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:49.067 7fe438680700 1 -- 172.21.2.112:0/119661398 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe43010aad0 msgr2=0x7fe43010aed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:49.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:49.067 7fe438680700 1 --2- 172.21.2.112:0/119661398 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe43010aad0 0x7fe43010aed0 secure :-1 s=READY pgs=2943 cs=0 l=1 rev1=1 rx=0x7fe428006f00 tx=0x7fe428009f10).stop 2021-02-10T11:46:49.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:49.068 7fe438680700 1 -- 172.21.2.112:0/119661398 shutdown_connections 2021-02-10T11:46:49.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:49.068 7fe438680700 1 --2- 172.21.2.112:0/119661398 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe43010aad0 0x7fe43010aed0 unknown :-1 s=CLOSED pgs=2943 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:49.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:49.068 7fe438680700 1 --2- 172.21.2.112:0/119661398 >> v2:172.21.2.120:3300/0 conn(0x7fe4300eb220 0x7fe4300eb620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:49.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:49.068 7fe438680700 1 -- 172.21.2.112:0/119661398 >> 172.21.2.112:0/119661398 conn(0x7fe4300ea630 msgr2=0x7fe4300d0800 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:49.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:49.068 7fe438680700 1 -- 172.21.2.112:0/119661398 shutdown_connections 2021-02-10T11:46:49.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:49.068 7fe438680700 1 -- 172.21.2.112:0/119661398 wait complete. 2021-02-10T11:46:49.088 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:49.082 7fe438680700 1 Processor -- start 2021-02-10T11:46:49.088 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:49.082 7fe438680700 1 -- start start 2021-02-10T11:46:49.088 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:49.083 7fe438680700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe4300ac9c0 0x7fe430104f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:49.088 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:49.083 7fe438680700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe430092490 0x7fe430092890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:49.088 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:49.083 7fe438680700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe430092eb0 0x7fe4300932b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:49.088 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:49.083 7fe438680700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe430091f90 con 0x7fe430092490 2021-02-10T11:46:49.088 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:49.083 7fe438680700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe430105560 con 0x7fe430092eb0 2021-02-10T11:46:49.088 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:49.083 7fe438680700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe430090900 con 0x7fe4300ac9c0 2021-02-10T11:46:49.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:49.083 7fe435c9b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe430092490 0x7fe430092890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:49.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:49.083 7fe435c9b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe430092490 0x7fe430092890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:59888/0 (socket says 172.21.2.112:59888) 2021-02-10T11:46:49.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:49.083 7fe435c9b700 1 -- 172.21.2.112:0/2984251106 learned_addr learned my addr 172.21.2.112:0/2984251106 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:46:49.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:49.083 7fe435c9b700 1 -- 172.21.2.112:0/2984251106 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe4300ac9c0 msgr2=0x7fe430104f00 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:46:49.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:49.083 7fe435c9b700 1 --2- 172.21.2.112:0/2984251106 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe4300ac9c0 0x7fe430104f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:49.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:49.083 7fe435c9b700 1 -- 172.21.2.112:0/2984251106 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe430092eb0 msgr2=0x7fe4300932b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:46:49.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:49.083 7fe435c9b700 1 --2- 172.21.2.112:0/2984251106 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe430092eb0 0x7fe4300932b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:49.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:49.083 7fe435c9b700 1 -- 172.21.2.112:0/2984251106 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe430090b20 con 0x7fe430092490 2021-02-10T11:46:49.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:49.083 7fe435c9b700 1 --2- 172.21.2.112:0/2984251106 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe430092490 0x7fe430092890 secure :-1 s=READY pgs=2944 cs=0 l=1 rev1=1 rx=0x7fe42c00b530 tx=0x7fe42c00b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:49.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:49.083 7fe422ffd700 1 -- 172.21.2.112:0/2984251106 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe42c00bcc0 con 0x7fe430092490 2021-02-10T11:46:49.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:49.084 7fe422ffd700 1 -- 172.21.2.112:0/2984251106 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe42c000c90 con 0x7fe430092490 2021-02-10T11:46:49.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:49.084 7fe422ffd700 1 -- 172.21.2.112:0/2984251106 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe42c0022a0 con 0x7fe430092490 2021-02-10T11:46:49.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:49.084 7fe438680700 1 -- 172.21.2.112:0/2984251106 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe430090b20 con 0x7fe430092490 2021-02-10T11:46:49.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:49.084 7fe438680700 1 -- 172.21.2.112:0/2984251106 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe430090b20 con 0x7fe430092490 2021-02-10T11:46:49.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:49.084 7fe422ffd700 1 -- 172.21.2.112:0/2984251106 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe42c005310 con 0x7fe430092490 2021-02-10T11:46:49.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:49.084 7fe422ffd700 1 --2- 172.21.2.112:0/2984251106 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe4180186d0 0x7fe41801abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:49.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:49.084 7fe422ffd700 1 -- 172.21.2.112:0/2984251106 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fe42c032610 con 0x7fe430092490 2021-02-10T11:46:49.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:49.085 7fe43649c700 1 --2- 172.21.2.112:0/2984251106 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe4180186d0 0x7fe41801abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:49.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:49.085 7fe43649c700 1 --2- 172.21.2.112:0/2984251106 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe4180186d0 0x7fe41801abd0 crc :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:49.118 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:49.116 7fe438680700 1 -- 172.21.2.112:0/2984251106 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe43010a550 con 0x7fe430092490 2021-02-10T11:46:49.121 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:49.118 7fe422ffd700 1 -- 172.21.2.112:0/2984251106 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe42c006b90 con 0x7fe430092490 2021-02-10T11:46:49.245 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:49.243 7fe438680700 1 -- 172.21.2.112:0/2984251106 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume deauthorize", "vol_name": "cephfs", "sub_name": "subvolume_0000000000544322", "target": ["mgr", ""], "auth_id": "guest1", "group_name": "group_0000000000707517"}) v1 -- 0x7fe43010a550 con 0x7fe4180186d0 2021-02-10T11:46:49.284 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:49.282 7fe422ffd700 1 -- 172.21.2.112:0/2984251106 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fe424006db0 con 0x7fe4180186d0 2021-02-10T11:46:49.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:49.309 7fe438680700 1 -- 172.21.2.112:0/2984251106 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe4180186d0 msgr2=0x7fe41801abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:49.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:49.309 7fe438680700 1 --2- 172.21.2.112:0/2984251106 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe4180186d0 0x7fe41801abd0 crc :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:49.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:49.309 7fe438680700 1 -- 172.21.2.112:0/2984251106 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe430092490 msgr2=0x7fe430092890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:49.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:49.309 7fe438680700 1 --2- 172.21.2.112:0/2984251106 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe430092490 0x7fe430092890 secure :-1 s=READY pgs=2944 cs=0 l=1 rev1=1 rx=0x7fe42c00b530 tx=0x7fe42c00b8d0).stop 2021-02-10T11:46:49.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:49.309 7fe438680700 1 -- 172.21.2.112:0/2984251106 shutdown_connections 2021-02-10T11:46:49.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:49.309 7fe438680700 1 --2- 172.21.2.112:0/2984251106 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe4180186d0 0x7fe41801abd0 unknown :-1 s=CLOSED pgs=1331 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:49.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:49.309 7fe438680700 1 --2- 172.21.2.112:0/2984251106 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe430092eb0 0x7fe4300932b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:49.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:49.309 7fe438680700 1 --2- 172.21.2.112:0/2984251106 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe430092490 0x7fe430092890 unknown :-1 s=CLOSED pgs=2944 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:49.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:49.309 7fe438680700 1 --2- 172.21.2.112:0/2984251106 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe4300ac9c0 0x7fe430104f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:49.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:49.309 7fe438680700 1 -- 172.21.2.112:0/2984251106 >> 172.21.2.112:0/2984251106 conn(0x7fe4301a8260 msgr2=0x7fe4300fac60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:49.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:49.309 7fe438680700 1 -- 172.21.2.112:0/2984251106 shutdown_connections 2021-02-10T11:46:49.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:49.309 7fe438680700 1 -- 172.21.2.112:0/2984251106 wait complete. 2021-02-10T11:46:49.320 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:46:49.320 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:46:49.320 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:46:49.654 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:46:49.655 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:46:49.655 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba012.front.sepia.ceph.com... 2021-02-10T11:46:49.655 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:46:49.655 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:46:50.007 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba012.stderr:ceph-fuse[184581]: fuse finished with error 0 and tester_r 0 2021-02-10T11:46:50.008 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:46:50.009 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:46:50.009 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:46:50.345 INFO:teuthology.orchestra.run.gibba012.stdout:ext2/ext3 2021-02-10T11:46:50.345 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:46:50.345 INFO:teuthology.orchestra.run:waiting for 900 2021-02-10T11:46:50.346 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:46:50.346 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:46:50.681 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth rm client.guest1 2021-02-10T11:46:51.109 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.089 7ff63faf8700 1 Processor -- start 2021-02-10T11:46:51.110 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.089 7ff63faf8700 1 -- start start 2021-02-10T11:46:51.110 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.090 7ff63faf8700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7ff6380d1c60 0x7ff6380f2090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:51.110 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.090 7ff63faf8700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6380f0b00 con 0x7ff6380d0330 2021-02-10T11:46:51.110 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.090 7ff63faf8700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6381b0bd0 con 0x7ff6380d1860 2021-02-10T11:46:51.110 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.090 7ff63faf8700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6380e5ec0 con 0x7ff6380d1c60 2021-02-10T11:46:51.110 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.090 7ff63d113700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7ff6380d0330 0x7ff6380ef630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:59354/0 (socket says 172.21.2.112:59354) 2021-02-10T11:46:51.110 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.090 7ff63d113700 1 -- 172.21.2.112:0/613524345 learned_addr learned my addr 172.21.2.112:0/613524345 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:46:51.110 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.090 7ff63e115700 1 --2- 172.21.2.112:0/613524345 >> v2:172.21.2.112:3300/0 conn(0x7ff6380d1c60 0x7ff6380f2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:51.110 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.090 7ff63e115700 1 -- 172.21.2.112:0/613524345 >> v1:172.21.2.120:6789/0 conn(0x7ff6380d1860 legacy=0x7ff6381b03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:46:51.110 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.090 7ff63e115700 1 -- 172.21.2.112:0/613524345 >> v1:172.21.2.112:6789/0 conn(0x7ff6380d0330 legacy=0x7ff6380ef630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:51.110 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.090 7ff63c912700 1 -- 172.21.2.112:0/613524345 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1366311226 0 0) 0x7ff634001180 con 0x7ff6380d0330 2021-02-10T11:46:51.110 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.090 7ff63e115700 1 -- 172.21.2.112:0/613524345 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6380efe20 con 0x7ff6380d1c60 2021-02-10T11:46:51.110 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.091 7ff63e115700 1 --2- 172.21.2.112:0/613524345 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff6380d1c60 0x7ff6380f2090 secure :-1 s=READY pgs=2946 cs=0 l=1 rev1=1 rx=0x7ff630006e30 tx=0x7ff630009de0).ready entity=mon.0 client_cookie=5903ce292d9a3667 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:51.111 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.091 7ff63c912700 1 -- 172.21.2.112:0/613524345 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff63000b300 con 0x7ff6380d1c60 2021-02-10T11:46:51.111 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.091 7ff63c912700 1 -- 172.21.2.112:0/613524345 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff63000b580 con 0x7ff6380d1c60 2021-02-10T11:46:51.111 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.091 7ff63c912700 1 -- 172.21.2.112:0/613524345 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff63000baa0 con 0x7ff6380d1c60 2021-02-10T11:46:51.111 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.091 7ff63faf8700 1 -- 172.21.2.112:0/613524345 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff6380d1c60 msgr2=0x7ff6380f2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:51.111 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.091 7ff63faf8700 1 --2- 172.21.2.112:0/613524345 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff6380d1c60 0x7ff6380f2090 secure :-1 s=READY pgs=2946 cs=0 l=1 rev1=1 rx=0x7ff630006e30 tx=0x7ff630009de0).stop 2021-02-10T11:46:51.111 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.091 7ff63faf8700 1 -- 172.21.2.112:0/613524345 shutdown_connections 2021-02-10T11:46:51.111 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.091 7ff63faf8700 1 --2- 172.21.2.112:0/613524345 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff6380d1c60 0x7ff6380f2090 unknown :-1 s=CLOSED pgs=2946 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:51.111 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.091 7ff63faf8700 1 -- 172.21.2.112:0/613524345 >> 172.21.2.112:0/613524345 conn(0x7ff6380ce440 msgr2=0x7ff6380d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:51.111 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.092 7ff63faf8700 1 -- 172.21.2.112:0/613524345 shutdown_connections 2021-02-10T11:46:51.111 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.092 7ff63faf8700 1 -- 172.21.2.112:0/613524345 wait complete. 2021-02-10T11:46:51.112 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.106 7ff63faf8700 1 Processor -- start 2021-02-10T11:46:51.113 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.106 7ff63faf8700 1 -- start start 2021-02-10T11:46:51.113 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.107 7ff63faf8700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff6380bfa20 0x7ff638193380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:51.113 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.107 7ff63faf8700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff6381939a0 0x7ff638193da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:51.113 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.107 7ff63faf8700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff6380c0440 0x7ff6380c0840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:51.113 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.107 7ff63faf8700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6380bfec0 con 0x7ff6380bfa20 2021-02-10T11:46:51.113 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.107 7ff63faf8700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6380a0380 con 0x7ff6381939a0 2021-02-10T11:46:51.113 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.107 7ff63faf8700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6380a05a0 con 0x7ff6380c0440 2021-02-10T11:46:51.113 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.107 7ff63d914700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff6380bfa20 0x7ff638193380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:51.113 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.107 7ff63d914700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff6380bfa20 0x7ff638193380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:59898/0 (socket says 172.21.2.112:59898) 2021-02-10T11:46:51.113 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.107 7ff63d914700 1 -- 172.21.2.112:0/1861892122 learned_addr learned my addr 172.21.2.112:0/1861892122 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:46:51.113 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.108 7ff63d914700 1 -- 172.21.2.112:0/1861892122 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff6380c0440 msgr2=0x7ff6380c0840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:51.113 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.108 7ff63d914700 1 --2- 172.21.2.112:0/1861892122 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff6380c0440 0x7ff6380c0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:51.113 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.108 7ff63d914700 1 -- 172.21.2.112:0/1861892122 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff6381939a0 msgr2=0x7ff638193da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:51.114 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.108 7ff63d914700 1 --2- 172.21.2.112:0/1861892122 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff6381939a0 0x7ff638193da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:51.114 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.108 7ff63d914700 1 -- 172.21.2.112:0/1861892122 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6380c0e60 con 0x7ff6380bfa20 2021-02-10T11:46:51.114 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.108 7ff63d914700 1 --2- 172.21.2.112:0/1861892122 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff6380bfa20 0x7ff638193380 secure :-1 s=READY pgs=2947 cs=0 l=1 rev1=1 rx=0x7ff62c00b5b0 tx=0x7ff62c00b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:51.114 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.108 7ff62a7fc700 1 -- 172.21.2.112:0/1861892122 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff62c00bd40 con 0x7ff6380bfa20 2021-02-10T11:46:51.114 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.108 7ff62a7fc700 1 -- 172.21.2.112:0/1861892122 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff62c006020 con 0x7ff6380bfa20 2021-02-10T11:46:51.114 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.108 7ff62a7fc700 1 -- 172.21.2.112:0/1861892122 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff62c0054a0 con 0x7ff6380bfa20 2021-02-10T11:46:51.114 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.108 7ff63faf8700 1 -- 172.21.2.112:0/1861892122 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6380c0e60 con 0x7ff6380bfa20 2021-02-10T11:46:51.114 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.108 7ff63faf8700 1 -- 172.21.2.112:0/1861892122 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6380c0e60 con 0x7ff6380bfa20 2021-02-10T11:46:51.114 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.109 7ff62a7fc700 1 -- 172.21.2.112:0/1861892122 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff62c006200 con 0x7ff6380bfa20 2021-02-10T11:46:51.114 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.109 7ff62a7fc700 1 --2- 172.21.2.112:0/1861892122 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff6200186c0 0x7ff62001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:51.114 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.109 7ff62a7fc700 1 -- 172.21.2.112:0/1861892122 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 5573+0+0 (secure 0 0 0) 0x7ff62c032540 con 0x7ff6380bfa20 2021-02-10T11:46:51.115 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.110 7ff63d113700 1 --2- 172.21.2.112:0/1861892122 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff6200186c0 0x7ff62001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:51.115 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.110 7ff63d113700 1 --2- 172.21.2.112:0/1861892122 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff6200186c0 0x7ff62001abc0 crc :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:51.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.140 7ff63faf8700 1 -- 172.21.2.112:0/1861892122 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff63819d360 con 0x7ff6380bfa20 2021-02-10T11:46:51.146 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.142 7ff62a7fc700 1 -- 172.21.2.112:0/1861892122 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff62c02ca90 con 0x7ff6380bfa20 2021-02-10T11:46:51.275 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.270 7ff63faf8700 1 -- 172.21.2.112:0/1861892122 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "auth rm", "entity": "client.guest1"} v 0) v1 -- 0x7ff63819d360 con 0x7ff6380bfa20 2021-02-10T11:46:51.275 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.271 7ff62a7fc700 1 -- 172.21.2.112:0/1861892122 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth rm", "entity": "client.guest1"}]=0 entity client.guest1 does not exist v101) v1 ==== 117+0+0 (secure 0 0 0) 0x7ff62c006200 con 0x7ff6380bfa20 2021-02-10T11:46:51.276 INFO:teuthology.orchestra.run.gibba012.stderr:entity client.guest1 does not exist 2021-02-10T11:46:51.276 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.273 7ff63faf8700 1 -- 172.21.2.112:0/1861892122 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff6200186c0 msgr2=0x7ff62001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:51.276 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.273 7ff63faf8700 1 --2- 172.21.2.112:0/1861892122 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff6200186c0 0x7ff62001abc0 crc :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:51.276 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.273 7ff63faf8700 1 -- 172.21.2.112:0/1861892122 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff6380bfa20 msgr2=0x7ff638193380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:51.276 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.273 7ff63faf8700 1 --2- 172.21.2.112:0/1861892122 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff6380bfa20 0x7ff638193380 secure :-1 s=READY pgs=2947 cs=0 l=1 rev1=1 rx=0x7ff62c00b5b0 tx=0x7ff62c00b950).stop 2021-02-10T11:46:51.276 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.273 7ff63faf8700 1 -- 172.21.2.112:0/1861892122 shutdown_connections 2021-02-10T11:46:51.276 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.273 7ff63faf8700 1 --2- 172.21.2.112:0/1861892122 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff6200186c0 0x7ff62001abc0 unknown :-1 s=CLOSED pgs=1332 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:51.277 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.273 7ff63faf8700 1 --2- 172.21.2.112:0/1861892122 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff6381939a0 0x7ff638193da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:51.277 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.273 7ff63faf8700 1 --2- 172.21.2.112:0/1861892122 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff6380c0440 0x7ff6380c0840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:51.277 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.273 7ff63faf8700 1 --2- 172.21.2.112:0/1861892122 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff6380bfa20 0x7ff638193380 unknown :-1 s=CLOSED pgs=2947 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:51.277 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.273 7ff63faf8700 1 -- 172.21.2.112:0/1861892122 >> 172.21.2.112:0/1861892122 conn(0x7ff638190ae0 msgr2=0x7ff6380bf070 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:51.277 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.273 7ff63faf8700 1 -- 172.21.2.112:0/1861892122 shutdown_connections 2021-02-10T11:46:51.277 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.273 7ff63faf8700 1 -- 172.21.2.112:0/1861892122 wait complete. 2021-02-10T11:46:51.285 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000544322 --group_name group_0000000000707517 2021-02-10T11:46:51.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.698 7f455ee50700 1 Processor -- start 2021-02-10T11:46:51.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.698 7f455ee50700 1 -- start start 2021-02-10T11:46:51.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.698 7f455ee50700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f45580d1c80 0x7f45580cef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:51.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.698 7f455ee50700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f45580cf590 0x7f45580cf990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:51.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.698 7f455ee50700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45580fe1d0 con 0x7f45580d1880 2021-02-10T11:46:51.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.698 7f455ee50700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45581b0cc0 con 0x7f45580d1c80 2021-02-10T11:46:51.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.698 7f455ee50700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4558109a70 con 0x7f45580cf590 2021-02-10T11:46:51.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.699 7f455d46d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f45580cf590 0x7f45580cf990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:51.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.699 7f455d46d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f45580cf590 0x7f45580cf990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3301/0 says I am v2:172.21.2.112:59592/0 (socket says 172.21.2.112:59592) 2021-02-10T11:46:51.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.699 7f455d46d700 1 -- 172.21.2.112:0/1834758698 learned_addr learned my addr 172.21.2.112:0/1834758698 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:46:51.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.699 7f4557fff700 1 --2- 172.21.2.112:0/1834758698 >> v2:172.21.2.120:3300/0 conn(0x7f45580d1c80 0x7f45580cef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:51.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.700 7f455d46d700 1 -- 172.21.2.112:0/1834758698 >> v2:172.21.2.120:3300/0 conn(0x7f45580d1c80 msgr2=0x7f45580cef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:46:51.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.700 7f455d46d700 1 --2- 172.21.2.112:0/1834758698 >> v2:172.21.2.120:3300/0 conn(0x7f45580d1c80 0x7f45580cef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:51.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.700 7f455d46d700 1 -- 172.21.2.112:0/1834758698 >> v1:172.21.2.120:6789/0 conn(0x7f45580d1880 legacy=0x7f45581b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:51.721 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.700 7f455d46d700 1 -- 172.21.2.112:0/1834758698 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4558088070 con 0x7f45580cf590 2021-02-10T11:46:51.721 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.700 7f455d46d700 1 --2- 172.21.2.112:0/1834758698 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f45580cf590 0x7f45580cf990 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7f4550006aa0 tx=0x7f4550009f20).ready entity=mon.2 client_cookie=5ee4a168ecb0cb49 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:51.721 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.700 7f45577fe700 1 -- 172.21.2.112:0/1834758698 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f455000b5c0 con 0x7f45580cf590 2021-02-10T11:46:51.721 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.700 7f45577fe700 1 -- 172.21.2.112:0/1834758698 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4550003030 con 0x7f45580cf590 2021-02-10T11:46:51.721 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.700 7f45577fe700 1 -- 172.21.2.112:0/1834758698 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f455000bab0 con 0x7f45580cf590 2021-02-10T11:46:51.721 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.700 7f455ee50700 1 -- 172.21.2.112:0/1834758698 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f45580cf590 msgr2=0x7f45580cf990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:51.721 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.700 7f455ee50700 1 --2- 172.21.2.112:0/1834758698 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f45580cf590 0x7f45580cf990 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7f4550006aa0 tx=0x7f4550009f20).stop 2021-02-10T11:46:51.721 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.701 7f455ee50700 1 -- 172.21.2.112:0/1834758698 shutdown_connections 2021-02-10T11:46:51.721 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.701 7f455ee50700 1 --2- 172.21.2.112:0/1834758698 >> v2:172.21.2.120:3300/0 conn(0x7f45580d1c80 0x7f45580cef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:51.721 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.701 7f455ee50700 1 --2- 172.21.2.112:0/1834758698 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f45580cf590 0x7f45580cf990 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:51.721 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.701 7f455ee50700 1 -- 172.21.2.112:0/1834758698 >> 172.21.2.112:0/1834758698 conn(0x7f45580ce430 msgr2=0x7f45580d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:51.721 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.701 7f455ee50700 1 -- 172.21.2.112:0/1834758698 shutdown_connections 2021-02-10T11:46:51.721 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.701 7f455ee50700 1 -- 172.21.2.112:0/1834758698 wait complete. 2021-02-10T11:46:51.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.716 7f455ee50700 1 Processor -- start 2021-02-10T11:46:51.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.716 7f455ee50700 1 -- start start 2021-02-10T11:46:51.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.716 7f455ee50700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4558095ea0 0x7f45580963b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:51.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.716 7f455ee50700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4558196b40 0x7f45580ae8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:51.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.716 7f455ee50700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4558196f40 0x7f4558197a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:51.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.716 7f455ee50700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4558197570 con 0x7f4558196f40 2021-02-10T11:46:51.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.716 7f455ee50700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4558196760 con 0x7f4558196b40 2021-02-10T11:46:51.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.716 7f455ee50700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45580aef10 con 0x7f4558095ea0 2021-02-10T11:46:51.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.716 7f455d46d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4558196f40 0x7f4558197a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:51.723 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.716 7f455d46d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4558196f40 0x7f4558197a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:59916/0 (socket says 172.21.2.112:59916) 2021-02-10T11:46:51.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.716 7f455d46d700 1 -- 172.21.2.112:0/1280708664 learned_addr learned my addr 172.21.2.112:0/1280708664 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:46:51.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.717 7f455d46d700 1 -- 172.21.2.112:0/1280708664 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4558095ea0 msgr2=0x7f45580963b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:46:51.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.717 7f455d46d700 1 --2- 172.21.2.112:0/1280708664 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4558095ea0 0x7f45580963b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:51.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.717 7f455d46d700 1 -- 172.21.2.112:0/1280708664 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4558196b40 msgr2=0x7f45580ae8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:51.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.717 7f455d46d700 1 --2- 172.21.2.112:0/1280708664 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4558196b40 0x7f45580ae8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:51.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.717 7f455d46d700 1 -- 172.21.2.112:0/1280708664 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4558198090 con 0x7f4558196f40 2021-02-10T11:46:51.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.717 7f455d46d700 1 --2- 172.21.2.112:0/1280708664 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4558196f40 0x7f4558197a70 secure :-1 s=READY pgs=2948 cs=0 l=1 rev1=1 rx=0x7f455000b8f0 tx=0x7f455000bc90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:51.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.717 7f45557fa700 1 -- 172.21.2.112:0/1280708664 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f455000bdd0 con 0x7f4558196f40 2021-02-10T11:46:51.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.717 7f45557fa700 1 -- 172.21.2.112:0/1280708664 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f455000c2f0 con 0x7f4558196f40 2021-02-10T11:46:51.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.717 7f45557fa700 1 -- 172.21.2.112:0/1280708664 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f455000c810 con 0x7f4558196f40 2021-02-10T11:46:51.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.717 7f455ee50700 1 -- 172.21.2.112:0/1280708664 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4558198090 con 0x7f4558196f40 2021-02-10T11:46:51.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.717 7f455ee50700 1 -- 172.21.2.112:0/1280708664 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4558198620 con 0x7f4558196f40 2021-02-10T11:46:51.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.718 7f45557fa700 1 -- 172.21.2.112:0/1280708664 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4550009fb0 con 0x7f4558196f40 2021-02-10T11:46:51.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.718 7f45557fa700 1 --2- 172.21.2.112:0/1280708664 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f453c0186c0 0x7f453c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:51.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.718 7f45557fa700 1 -- 172.21.2.112:0/1280708664 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f4550032920 con 0x7f4558196f40 2021-02-10T11:46:51.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.718 7f455cc6c700 1 --2- 172.21.2.112:0/1280708664 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f453c0186c0 0x7f453c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:51.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.719 7f455cc6c700 1 --2- 172.21.2.112:0/1280708664 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f453c0186c0 0x7f453c01abc0 crc :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:51.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.748 7f455ee50700 1 -- 172.21.2.112:0/1280708664 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4558082a70 con 0x7f4558196f40 2021-02-10T11:46:51.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.751 7f45557fa700 1 -- 172.21.2.112:0/1280708664 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f455001dfb0 con 0x7f4558196f40 2021-02-10T11:46:51.880 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.877 7f455ee50700 1 -- 172.21.2.112:0/1280708664 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000544322", "target": ["mgr", ""], "group_name": "group_0000000000707517"}) v1 -- 0x7f4558198620 con 0x7f453c0186c0 2021-02-10T11:46:51.890 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.888 7f45557fa700 1 -- 172.21.2.112:0/1280708664 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f4540006db0 con 0x7f453c0186c0 2021-02-10T11:46:51.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.894 7f455ee50700 1 -- 172.21.2.112:0/1280708664 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f453c0186c0 msgr2=0x7f453c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:51.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.894 7f455ee50700 1 --2- 172.21.2.112:0/1280708664 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f453c0186c0 0x7f453c01abc0 crc :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:51.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.894 7f455ee50700 1 -- 172.21.2.112:0/1280708664 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4558196f40 msgr2=0x7f4558197a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:51.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.894 7f455ee50700 1 --2- 172.21.2.112:0/1280708664 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4558196f40 0x7f4558197a70 secure :-1 s=READY pgs=2948 cs=0 l=1 rev1=1 rx=0x7f455000b8f0 tx=0x7f455000bc90).stop 2021-02-10T11:46:51.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.895 7f455ee50700 1 -- 172.21.2.112:0/1280708664 shutdown_connections 2021-02-10T11:46:51.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.895 7f455ee50700 1 --2- 172.21.2.112:0/1280708664 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f453c0186c0 0x7f453c01abc0 unknown :-1 s=CLOSED pgs=1333 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:51.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.895 7f455ee50700 1 --2- 172.21.2.112:0/1280708664 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4558196b40 0x7f45580ae8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:51.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.895 7f455ee50700 1 --2- 172.21.2.112:0/1280708664 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4558196f40 0x7f4558197a70 unknown :-1 s=CLOSED pgs=2948 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:51.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.895 7f455ee50700 1 --2- 172.21.2.112:0/1280708664 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4558095ea0 0x7f45580963b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:51.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.895 7f455ee50700 1 -- 172.21.2.112:0/1280708664 >> 172.21.2.112:0/1280708664 conn(0x7f455819a370 msgr2=0x7f45581ae590 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:51.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.895 7f455ee50700 1 -- 172.21.2.112:0/1280708664 shutdown_connections 2021-02-10T11:46:51.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:51.895 7f455ee50700 1 -- 172.21.2.112:0/1280708664 wait complete. 2021-02-10T11:46:51.907 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolumegroup rm cephfs group_0000000000707517 2021-02-10T11:46:52.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:52.319 7f5a9e2d0700 1 Processor -- start 2021-02-10T11:46:52.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:52.319 7f5a9e2d0700 1 -- start start 2021-02-10T11:46:52.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:52.319 7f5a9e2d0700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f5a980d1860 0x7f5a981b04b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:52.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:52.319 7f5a9e2d0700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f5a981b0ad0 0x7f5a98108930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:52.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:52.319 7f5a9e2d0700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a980fe1b0 con 0x7f5a980d1c60 2021-02-10T11:46:52.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:52.319 7f5a9e2d0700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a98109a50 con 0x7f5a981b0ad0 2021-02-10T11:46:52.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:52.319 7f5a9e2d0700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a98109740 con 0x7f5a980d1860 2021-02-10T11:46:52.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:52.319 7f5a977fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f5a981b0ad0 0x7f5a98108930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:52.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:52.319 7f5a977fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f5a981b0ad0 0x7f5a98108930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:59924/0 (socket says 172.21.2.112:59924) 2021-02-10T11:46:52.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:52.319 7f5a977fe700 1 -- 172.21.2.112:0/3972117198 learned_addr learned my addr 172.21.2.112:0/3972117198 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:46:52.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:52.320 7f5a977fe700 1 -- 172.21.2.112:0/3972117198 >> v2:172.21.2.120:3300/0 conn(0x7f5a980d1860 msgr2=0x7f5a981b04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:46:52.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:52.320 7f5a977fe700 1 --2- 172.21.2.112:0/3972117198 >> v2:172.21.2.120:3300/0 conn(0x7f5a980d1860 0x7f5a981b04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:52.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:52.320 7f5a977fe700 1 -- 172.21.2.112:0/3972117198 >> v1:172.21.2.120:6789/0 conn(0x7f5a980d1c60 legacy=0x7f5a98108f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:46:52.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:52.320 7f5a977fe700 1 -- 172.21.2.112:0/3972117198 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a98088070 con 0x7f5a981b0ad0 2021-02-10T11:46:52.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:52.320 7f5a977fe700 1 --2- 172.21.2.112:0/3972117198 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5a981b0ad0 0x7f5a98108930 secure :-1 s=READY pgs=2949 cs=0 l=1 rev1=1 rx=0x7f5a8c006700 tx=0x7f5a8c009b80).ready entity=mon.0 client_cookie=ad97c05cd54ff123 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:52.342 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:52.320 7f5a96ffd700 1 -- 172.21.2.112:0/3972117198 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5a8c00c0c0 con 0x7f5a981b0ad0 2021-02-10T11:46:52.342 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:52.320 7f5a96ffd700 1 -- 172.21.2.112:0/3972117198 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a8c00c440 con 0x7f5a981b0ad0 2021-02-10T11:46:52.342 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:52.320 7f5a96ffd700 1 -- 172.21.2.112:0/3972117198 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5a8c00c960 con 0x7f5a981b0ad0 2021-02-10T11:46:52.342 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:52.320 7f5a9e2d0700 1 -- 172.21.2.112:0/3972117198 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5a981b0ad0 msgr2=0x7f5a98108930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:52.342 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:52.320 7f5a9e2d0700 1 --2- 172.21.2.112:0/3972117198 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5a981b0ad0 0x7f5a98108930 secure :-1 s=READY pgs=2949 cs=0 l=1 rev1=1 rx=0x7f5a8c006700 tx=0x7f5a8c009b80).stop 2021-02-10T11:46:52.342 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:52.321 7f5a9e2d0700 1 -- 172.21.2.112:0/3972117198 shutdown_connections 2021-02-10T11:46:52.342 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:52.321 7f5a9e2d0700 1 --2- 172.21.2.112:0/3972117198 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5a981b0ad0 0x7f5a98108930 unknown :-1 s=CLOSED pgs=2949 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:52.342 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:52.321 7f5a9e2d0700 1 --2- 172.21.2.112:0/3972117198 >> v2:172.21.2.120:3300/0 conn(0x7f5a980d1860 0x7f5a981b04b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:52.342 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:52.321 7f5a9e2d0700 1 -- 172.21.2.112:0/3972117198 >> 172.21.2.112:0/3972117198 conn(0x7f5a980ce410 msgr2=0x7f5a980d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:52.342 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:52.321 7f5a9e2d0700 1 -- 172.21.2.112:0/3972117198 shutdown_connections 2021-02-10T11:46:52.342 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:52.321 7f5a9e2d0700 1 -- 172.21.2.112:0/3972117198 wait complete. 2021-02-10T11:46:52.342 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:52.336 7f5a9e2d0700 1 Processor -- start 2021-02-10T11:46:52.342 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:52.336 7f5a9e2d0700 1 -- start start 2021-02-10T11:46:52.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:52.336 7f5a9e2d0700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5a98192fe0 0x7f5a98096810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:52.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:52.336 7f5a9e2d0700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5a98096e30 0x7f5a981a7380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:52.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:52.336 7f5a9e2d0700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5a981a79a0 0x7f5a980aa050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:52.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:52.336 7f5a9e2d0700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a980a9ad0 con 0x7f5a98096e30 2021-02-10T11:46:52.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:52.336 7f5a9e2d0700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a98097590 con 0x7f5a981a79a0 2021-02-10T11:46:52.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:52.336 7f5a9e2d0700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a98097230 con 0x7f5a98192fe0 2021-02-10T11:46:52.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:52.336 7f5a977fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5a98096e30 0x7f5a981a7380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:52.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:52.337 7f5a977fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5a98096e30 0x7f5a981a7380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:59926/0 (socket says 172.21.2.112:59926) 2021-02-10T11:46:52.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:52.337 7f5a977fe700 1 -- 172.21.2.112:0/4132631774 learned_addr learned my addr 172.21.2.112:0/4132631774 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:46:52.344 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:52.337 7f5a977fe700 1 -- 172.21.2.112:0/4132631774 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5a98192fe0 msgr2=0x7f5a98096810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:52.344 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:52.337 7f5a977fe700 1 --2- 172.21.2.112:0/4132631774 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5a98192fe0 0x7f5a98096810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:52.344 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:52.337 7f5a977fe700 1 -- 172.21.2.112:0/4132631774 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5a981a79a0 msgr2=0x7f5a980aa050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:52.345 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:52.337 7f5a977fe700 1 --2- 172.21.2.112:0/4132631774 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5a981a79a0 0x7f5a980aa050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:52.345 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:52.337 7f5a977fe700 1 -- 172.21.2.112:0/4132631774 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a981a7da0 con 0x7f5a98096e30 2021-02-10T11:46:52.345 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:52.337 7f5a977fe700 1 --2- 172.21.2.112:0/4132631774 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5a98096e30 0x7f5a981a7380 secure :-1 s=READY pgs=2950 cs=0 l=1 rev1=1 rx=0x7f5a8c00b580 tx=0x7f5a8c00b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:52.345 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:52.337 7f5a94ff9700 1 -- 172.21.2.112:0/4132631774 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5a8c00bd10 con 0x7f5a98096e30 2021-02-10T11:46:52.345 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:52.337 7f5a94ff9700 1 -- 172.21.2.112:0/4132631774 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a8c002030 con 0x7f5a98096e30 2021-02-10T11:46:52.345 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:52.337 7f5a94ff9700 1 -- 172.21.2.112:0/4132631774 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5a8c002480 con 0x7f5a98096e30 2021-02-10T11:46:52.345 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:52.337 7f5a9e2d0700 1 -- 172.21.2.112:0/4132631774 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a981a7da0 con 0x7f5a98096e30 2021-02-10T11:46:52.345 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:52.337 7f5a9e2d0700 1 -- 172.21.2.112:0/4132631774 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a981a7da0 con 0x7f5a98096e30 2021-02-10T11:46:52.345 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:52.338 7f5a94ff9700 1 -- 172.21.2.112:0/4132631774 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5a8c000c90 con 0x7f5a98096e30 2021-02-10T11:46:52.345 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:52.338 7f5a94ff9700 1 --2- 172.21.2.112:0/4132631774 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5a780186c0 0x7f5a7801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:52.345 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:52.338 7f5a94ff9700 1 -- 172.21.2.112:0/4132631774 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f5a8c032680 con 0x7f5a98096e30 2021-02-10T11:46:52.345 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:52.339 7f5a97fff700 1 --2- 172.21.2.112:0/4132631774 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5a780186c0 0x7f5a7801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:52.345 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:52.339 7f5a97fff700 1 --2- 172.21.2.112:0/4132631774 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5a780186c0 0x7f5a7801abc0 crc :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:52.373 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:52.369 7f5a9e2d0700 1 -- 172.21.2.112:0/4132631774 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a98097570 con 0x7f5a98096e30 2021-02-10T11:46:52.375 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:52.372 7f5a94ff9700 1 -- 172.21.2.112:0/4132631774 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5a8c01f540 con 0x7f5a98096e30 2021-02-10T11:46:52.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:52.499 7f5a9e2d0700 1 -- 172.21.2.112:0/4132631774 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolumegroup rm", "vol_name": "cephfs", "target": ["mgr", ""], "group_name": "group_0000000000707517"}) v1 -- 0x7f5a98097570 con 0x7f5a780186c0 2021-02-10T11:46:52.506 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:52.504 7f5a94ff9700 1 -- 172.21.2.112:0/4132631774 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f5a880009b0 con 0x7f5a780186c0 2021-02-10T11:46:52.512 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:52.508 7f5a9e2d0700 1 -- 172.21.2.112:0/4132631774 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5a780186c0 msgr2=0x7f5a7801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:52.512 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:52.508 7f5a9e2d0700 1 --2- 172.21.2.112:0/4132631774 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5a780186c0 0x7f5a7801abc0 crc :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:52.512 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:52.508 7f5a9e2d0700 1 -- 172.21.2.112:0/4132631774 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5a98096e30 msgr2=0x7f5a981a7380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:52.512 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:52.508 7f5a9e2d0700 1 --2- 172.21.2.112:0/4132631774 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5a98096e30 0x7f5a981a7380 secure :-1 s=READY pgs=2950 cs=0 l=1 rev1=1 rx=0x7f5a8c00b580 tx=0x7f5a8c00b920).stop 2021-02-10T11:46:52.512 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:52.508 7f5a9e2d0700 1 -- 172.21.2.112:0/4132631774 shutdown_connections 2021-02-10T11:46:52.512 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:52.508 7f5a9e2d0700 1 --2- 172.21.2.112:0/4132631774 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5a780186c0 0x7f5a7801abc0 unknown :-1 s=CLOSED pgs=1334 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:52.512 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:52.508 7f5a9e2d0700 1 --2- 172.21.2.112:0/4132631774 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5a981a79a0 0x7f5a980aa050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:52.512 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:52.508 7f5a9e2d0700 1 --2- 172.21.2.112:0/4132631774 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5a98096e30 0x7f5a981a7380 unknown :-1 s=CLOSED pgs=2950 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:52.512 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:52.508 7f5a9e2d0700 1 --2- 172.21.2.112:0/4132631774 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5a98192fe0 0x7f5a98096810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:52.512 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:52.508 7f5a9e2d0700 1 -- 172.21.2.112:0/4132631774 >> 172.21.2.112:0/4132631774 conn(0x7f5a981935d0 msgr2=0x7f5a98086470 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:52.513 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:52.508 7f5a9e2d0700 1 -- 172.21.2.112:0/4132631774 shutdown_connections 2021-02-10T11:46:52.513 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:52.508 7f5a9e2d0700 1 -- 172.21.2.112:0/4132631774 wait complete. 2021-02-10T11:46:52.520 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:46:52.520 DEBUG:teuthology.orchestra.run.gibba012:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:46:52.866 DEBUG:teuthology.orchestra.run.gibba020:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:46:53.211 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:46:53.211 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:46:53.211 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:46:53.211 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:46:53.548 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:46:53.548 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:46:53.548 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba012.front.sepia.ceph.com... 2021-02-10T11:46:53.549 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:46:53.549 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:46:53.901 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba012.stderr:ceph-fuse[183940]: fuse finished with error 0 and tester_r 0 2021-02-10T11:46:53.903 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:46:53.903 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:46:53.903 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:46:54.237 INFO:teuthology.orchestra.run.gibba012.stdout:ext2/ext3 2021-02-10T11:46:54.237 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:46:54.237 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:46:54.237 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:46:54.573 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:46:54.574 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:46:54.574 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:46:54.908 INFO:teuthology.orchestra.run.gibba012.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: No such file or directory 2021-02-10T11:46:54.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:46:54.909 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2021-02-10T11:46:54.909 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:46:54.909 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:46:55.245 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:46:55.245 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:46:55.245 DEBUG:teuthology.orchestra.run.gibba020:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:46:55.579 INFO:teuthology.orchestra.run.gibba020.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:46:55.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:46:55.580 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:46:55.580 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:46:55.580 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:46:55.915 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:46:55.916 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:46:55.916 DEBUG:teuthology.orchestra.run.gibba020:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:46:56.249 INFO:teuthology.orchestra.run.gibba020.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:46:56.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:46:56.250 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:46:56.250 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:46:56.250 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:46:56.587 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config rm mon mon_allow_pool_delete 2021-02-10T11:46:56.727 DEBUG:teuthology.orchestra.run.gibba012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:46:56.731 DEBUG:teuthology.orchestra.run.gibba020:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:46:57.019 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.000 7fa4020f7700 1 Processor -- start 2021-02-10T11:46:57.019 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.000 7fa4020f7700 1 -- start start 2021-02-10T11:46:57.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.001 7fa4020f7700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fa3fc0d1870 0x7fa3fc1b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:57.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.001 7fa4020f7700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fa3fc0f0650 0x7fa3fc0cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:57.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.001 7fa4020f7700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3fc0f0b10 con 0x7fa3fc1b0a10 2021-02-10T11:46:57.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.001 7fa4020f7700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3fc0e5ed0 con 0x7fa3fc0d1870 2021-02-10T11:46:57.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.001 7fa4020f7700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3fc0d1c70 con 0x7fa3fc0f0650 2021-02-10T11:46:57.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.001 7fa3fb7fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fa3fc0d1870 0x7fa3fc1b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:57.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.001 7fa3faffd700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fa3fc1b0a10 0x7fa3fc0cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:59396/0 (socket says 172.21.2.112:59396) 2021-02-10T11:46:57.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.001 7fa3faffd700 1 -- 172.21.2.112:0/1074026442 learned_addr learned my addr 172.21.2.112:0/1074026442 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:46:57.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.001 7fa3fb7fe700 1 -- 172.21.2.112:0/1074026442 >> v2:172.21.2.120:3300/0 conn(0x7fa3fc0f0650 msgr2=0x7fa3fc0cfb20 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:46:57.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.001 7fa3fb7fe700 1 --2- 172.21.2.112:0/1074026442 >> v2:172.21.2.120:3300/0 conn(0x7fa3fc0f0650 0x7fa3fc0cfb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:57.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.001 7fa3fb7fe700 1 -- 172.21.2.112:0/1074026442 >> v1:172.21.2.112:6789/0 conn(0x7fa3fc1b0a10 legacy=0x7fa3fc0cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:57.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.001 7fa3fb7fe700 1 -- 172.21.2.112:0/1074026442 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3fc0d1e90 con 0x7fa3fc0d1870 2021-02-10T11:46:57.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.002 7fa3fb7fe700 1 --2- 172.21.2.112:0/1074026442 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa3fc0d1870 0x7fa3fc1b03f0 secure :-1 s=READY pgs=2952 cs=0 l=1 rev1=1 rx=0x7fa3e4006650 tx=0x7fa3e4009a80).ready entity=mon.0 client_cookie=75e977a00db53571 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:57.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.002 7fa3fa7fc700 1 -- 172.21.2.112:0/1074026442 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa3e40009b0 con 0x7fa3fc0d1870 2021-02-10T11:46:57.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.002 7fa3fa7fc700 1 -- 172.21.2.112:0/1074026442 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3e400c0c0 con 0x7fa3fc0d1870 2021-02-10T11:46:57.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.002 7fa3fa7fc700 1 -- 172.21.2.112:0/1074026442 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa3e400c520 con 0x7fa3fc0d1870 2021-02-10T11:46:57.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.002 7fa4020f7700 1 -- 172.21.2.112:0/1074026442 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa3fc0d1870 msgr2=0x7fa3fc1b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:57.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.002 7fa4020f7700 1 --2- 172.21.2.112:0/1074026442 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa3fc0d1870 0x7fa3fc1b03f0 secure :-1 s=READY pgs=2952 cs=0 l=1 rev1=1 rx=0x7fa3e4006650 tx=0x7fa3e4009a80).stop 2021-02-10T11:46:57.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.002 7fa4020f7700 1 -- 172.21.2.112:0/1074026442 shutdown_connections 2021-02-10T11:46:57.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.002 7fa4020f7700 1 --2- 172.21.2.112:0/1074026442 >> v2:172.21.2.120:3300/0 conn(0x7fa3fc0f0650 0x7fa3fc0cfb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:57.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.002 7fa4020f7700 1 --2- 172.21.2.112:0/1074026442 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa3fc0d1870 0x7fa3fc1b03f0 unknown :-1 s=CLOSED pgs=2952 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:57.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.002 7fa4020f7700 1 -- 172.21.2.112:0/1074026442 >> 172.21.2.112:0/1074026442 conn(0x7fa3fc0ce450 msgr2=0x7fa3fc0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:57.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.003 7fa4020f7700 1 -- 172.21.2.112:0/1074026442 shutdown_connections 2021-02-10T11:46:57.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.003 7fa4020f7700 1 -- 172.21.2.112:0/1074026442 wait complete. 2021-02-10T11:46:57.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.017 7fa4020f7700 1 Processor -- start 2021-02-10T11:46:57.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.017 7fa4020f7700 1 -- start start 2021-02-10T11:46:57.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.018 7fa4020f7700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa3fc0a2fb0 0x7fa3fc0a33b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:57.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.018 7fa4020f7700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa3fc0a39d0 0x7fa3fc0a4860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:57.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.018 7fa4020f7700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa3fc0a3dd0 0x7fa3fc0a4e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:57.024 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.018 7fa4020f7700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3fc0a2620 con 0x7fa3fc0a39d0 2021-02-10T11:46:57.024 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.018 7fa4020f7700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3fc0a2840 con 0x7fa3fc0a3dd0 2021-02-10T11:46:57.024 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.018 7fa4020f7700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3fc0a2b80 con 0x7fa3fc0a2fb0 2021-02-10T11:46:57.024 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.018 7fa3faffd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa3fc0a39d0 0x7fa3fc0a4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:57.024 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.018 7fa3faffd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa3fc0a39d0 0x7fa3fc0a4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:59940/0 (socket says 172.21.2.112:59940) 2021-02-10T11:46:57.024 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.018 7fa3faffd700 1 -- 172.21.2.112:0/520087304 learned_addr learned my addr 172.21.2.112:0/520087304 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:46:57.024 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.018 7fa3faffd700 1 -- 172.21.2.112:0/520087304 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa3fc0a2fb0 msgr2=0x7fa3fc0a33b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:57.024 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.018 7fa3faffd700 1 --2- 172.21.2.112:0/520087304 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa3fc0a2fb0 0x7fa3fc0a33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:57.024 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.018 7fa3faffd700 1 -- 172.21.2.112:0/520087304 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa3fc0a3dd0 msgr2=0x7fa3fc0a4e80 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:46:57.024 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.018 7fa3faffd700 1 --2- 172.21.2.112:0/520087304 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa3fc0a3dd0 0x7fa3fc0a4e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:57.024 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.018 7fa3faffd700 1 -- 172.21.2.112:0/520087304 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3fc0a54a0 con 0x7fa3fc0a39d0 2021-02-10T11:46:57.025 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.018 7fa3faffd700 1 --2- 172.21.2.112:0/520087304 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa3fc0a39d0 0x7fa3fc0a4860 secure :-1 s=READY pgs=2953 cs=0 l=1 rev1=1 rx=0x7fa3ec001c60 tx=0x7fa3ec00a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:57.025 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.019 7fa3f3fff700 1 -- 172.21.2.112:0/520087304 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa3ec00acc0 con 0x7fa3fc0a39d0 2021-02-10T11:46:57.025 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.019 7fa3f3fff700 1 -- 172.21.2.112:0/520087304 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3ec004140 con 0x7fa3fc0a39d0 2021-02-10T11:46:57.025 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.019 7fa4020f7700 1 -- 172.21.2.112:0/520087304 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3fc0a54a0 con 0x7fa3fc0a39d0 2021-02-10T11:46:57.025 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.019 7fa3f3fff700 1 -- 172.21.2.112:0/520087304 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa3ec004620 con 0x7fa3fc0a39d0 2021-02-10T11:46:57.025 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.019 7fa4020f7700 1 -- 172.21.2.112:0/520087304 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3fc0a54a0 con 0x7fa3fc0a39d0 2021-02-10T11:46:57.025 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.019 7fa3f3fff700 1 -- 172.21.2.112:0/520087304 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa3ec000dd0 con 0x7fa3fc0a39d0 2021-02-10T11:46:57.025 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.019 7fa3f3fff700 1 --2- 172.21.2.112:0/520087304 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa3e0018730 0x7fa3e001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:57.025 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.019 7fa3f3fff700 1 -- 172.21.2.112:0/520087304 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fa3ec0325f0 con 0x7fa3fc0a39d0 2021-02-10T11:46:57.025 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.020 7fa3fb7fe700 1 --2- 172.21.2.112:0/520087304 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa3e0018730 0x7fa3e001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:57.025 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.020 7fa3fb7fe700 1 --2- 172.21.2.112:0/520087304 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa3e0018730 0x7fa3e001ac10 crc :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:57.053 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.050 7fa4020f7700 1 -- 172.21.2.112:0/520087304 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3fc0ee060 con 0x7fa3fc0a39d0 2021-02-10T11:46:57.055 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.053 7fa3f3fff700 1 -- 172.21.2.112:0/520087304 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa3ec0009b0 con 0x7fa3fc0a39d0 2021-02-10T11:46:57.207 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.205 7fa4020f7700 1 -- 172.21.2.112:0/520087304 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fa3fc0a5d20 con 0x7fa3fc0a39d0 2021-02-10T11:46:57.211 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.209 7fa3f3fff700 1 -- 172.21.2.112:0/520087304 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v43) v1 ==== 104+0+0 (secure 0 0 0) 0x7fa3ec000db0 con 0x7fa3fc0a39d0 2021-02-10T11:46:57.216 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.214 7fa4020f7700 1 -- 172.21.2.112:0/520087304 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa3e0018730 msgr2=0x7fa3e001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:57.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.214 7fa4020f7700 1 --2- 172.21.2.112:0/520087304 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa3e0018730 0x7fa3e001ac10 crc :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:57.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.214 7fa4020f7700 1 -- 172.21.2.112:0/520087304 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa3fc0a39d0 msgr2=0x7fa3fc0a4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:57.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.214 7fa4020f7700 1 --2- 172.21.2.112:0/520087304 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa3fc0a39d0 0x7fa3fc0a4860 secure :-1 s=READY pgs=2953 cs=0 l=1 rev1=1 rx=0x7fa3ec001c60 tx=0x7fa3ec00a8d0).stop 2021-02-10T11:46:57.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.214 7fa4020f7700 1 -- 172.21.2.112:0/520087304 shutdown_connections 2021-02-10T11:46:57.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.214 7fa4020f7700 1 --2- 172.21.2.112:0/520087304 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa3e0018730 0x7fa3e001ac10 unknown :-1 s=CLOSED pgs=1335 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:57.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.214 7fa4020f7700 1 --2- 172.21.2.112:0/520087304 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa3fc0a3dd0 0x7fa3fc0a4e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:57.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.214 7fa4020f7700 1 --2- 172.21.2.112:0/520087304 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa3fc0a39d0 0x7fa3fc0a4860 unknown :-1 s=CLOSED pgs=2953 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:57.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.214 7fa4020f7700 1 --2- 172.21.2.112:0/520087304 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa3fc0a2fb0 0x7fa3fc0a33b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:57.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.214 7fa4020f7700 1 -- 172.21.2.112:0/520087304 >> 172.21.2.112:0/520087304 conn(0x7fa3fc08f030 msgr2=0x7fa3fc0b5e80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:57.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.214 7fa4020f7700 1 -- 172.21.2.112:0/520087304 shutdown_connections 2021-02-10T11:46:57.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.214 7fa4020f7700 1 -- 172.21.2.112:0/520087304 wait complete. 2021-02-10T11:46:57.226 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestVolumes.test_recover_auth_metadata_during_authorize' 2021-02-10T11:46:57.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.635 7f7445a0d700 1 Processor -- start 2021-02-10T11:46:57.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.635 7f7445a0d700 1 -- start start 2021-02-10T11:46:57.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.635 7f7445a0d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f74400d0340 0x7f74400ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:57.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.635 7f7445a0d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f74400d1c70 0x7f74400efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:57.654 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.635 7f7445a0d700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74400f0b10 con 0x7f74400d1870 2021-02-10T11:46:57.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.635 7f7445a0d700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74400e5ed0 con 0x7f74400d0340 2021-02-10T11:46:57.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.635 7f7445a0d700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74401b0be0 con 0x7f74400d1c70 2021-02-10T11:46:57.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.636 7f743effd700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f74400d0340 0x7f74400ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:57.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.636 7f743effd700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f74400d0340 0x7f74400ef640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:59950/0 (socket says 172.21.2.112:59950) 2021-02-10T11:46:57.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.636 7f743effd700 1 -- 172.21.2.112:0/88826534 learned_addr learned my addr 172.21.2.112:0/88826534 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:46:57.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.636 7f743effd700 1 -- 172.21.2.112:0/88826534 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f74400d1c70 msgr2=0x7f74400efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:46:57.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.636 7f743effd700 1 --2- 172.21.2.112:0/88826534 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f74400d1c70 0x7f74400efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:57.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.636 7f743effd700 1 -- 172.21.2.112:0/88826534 >> v1:172.21.2.120:6789/0 conn(0x7f74400d1870 legacy=0x7f74401b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:46:57.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.636 7f743effd700 1 -- 172.21.2.112:0/88826534 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74400f20a0 con 0x7f74400d0340 2021-02-10T11:46:57.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.636 7f743effd700 1 --2- 172.21.2.112:0/88826534 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f74400d0340 0x7f74400ef640 secure :-1 s=READY pgs=2954 cs=0 l=1 rev1=1 rx=0x7f7434006700 tx=0x7f7434009b80).ready entity=mon.0 client_cookie=a2fe1b812589aaab server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:57.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.636 7f743e7fc700 1 -- 172.21.2.112:0/88826534 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f743400c0c0 con 0x7f74400d0340 2021-02-10T11:46:57.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.637 7f743e7fc700 1 -- 172.21.2.112:0/88826534 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f743400c340 con 0x7f74400d0340 2021-02-10T11:46:57.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.637 7f743e7fc700 1 -- 172.21.2.112:0/88826534 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f743400c7a0 con 0x7f74400d0340 2021-02-10T11:46:57.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.637 7f7445a0d700 1 -- 172.21.2.112:0/88826534 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f74400d0340 msgr2=0x7f74400ef640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:57.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.637 7f7445a0d700 1 --2- 172.21.2.112:0/88826534 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f74400d0340 0x7f74400ef640 secure :-1 s=READY pgs=2954 cs=0 l=1 rev1=1 rx=0x7f7434006700 tx=0x7f7434009b80).stop 2021-02-10T11:46:57.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.637 7f7445a0d700 1 -- 172.21.2.112:0/88826534 shutdown_connections 2021-02-10T11:46:57.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.637 7f7445a0d700 1 --2- 172.21.2.112:0/88826534 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f74400d0340 0x7f74400ef640 unknown :-1 s=CLOSED pgs=2954 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:57.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.637 7f7445a0d700 1 --2- 172.21.2.112:0/88826534 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f74400d1c70 0x7f74400efc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:57.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.637 7f7445a0d700 1 -- 172.21.2.112:0/88826534 >> 172.21.2.112:0/88826534 conn(0x7f74400ce450 msgr2=0x7f74400d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:57.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.638 7f7445a0d700 1 -- 172.21.2.112:0/88826534 shutdown_connections 2021-02-10T11:46:57.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.638 7f7445a0d700 1 -- 172.21.2.112:0/88826534 wait complete. 2021-02-10T11:46:57.657 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.652 7f7445a0d700 1 Processor -- start 2021-02-10T11:46:57.658 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.652 7f7445a0d700 1 -- start start 2021-02-10T11:46:57.658 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.652 7f7445a0d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f744018a540 0x7f7440197480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:57.658 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.652 7f7445a0d700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f744018aa50 0x7f74400b77d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:57.658 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.652 7f7445a0d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f74400b7df0 0x7f74400b81f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:57.658 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.652 7f7445a0d700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74401a79d0 con 0x7f744018a540 2021-02-10T11:46:57.658 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.652 7f7445a0d700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74401a9e20 con 0x7f744018aa50 2021-02-10T11:46:57.658 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.652 7f7445a0d700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74401aa780 con 0x7f74400b7df0 2021-02-10T11:46:57.658 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.652 7f743f7fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f744018a540 0x7f7440197480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:57.658 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.652 7f743f7fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f744018a540 0x7f7440197480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:59954/0 (socket says 172.21.2.112:59954) 2021-02-10T11:46:57.658 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.652 7f743f7fe700 1 -- 172.21.2.112:0/711520250 learned_addr learned my addr 172.21.2.112:0/711520250 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:46:57.658 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.653 7f743f7fe700 1 -- 172.21.2.112:0/711520250 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f74400b7df0 msgr2=0x7f74400b81f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:57.658 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.653 7f743f7fe700 1 --2- 172.21.2.112:0/711520250 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f74400b7df0 0x7f74400b81f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:57.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.653 7f743f7fe700 1 -- 172.21.2.112:0/711520250 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f744018aa50 msgr2=0x7f74400b77d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:57.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.653 7f743f7fe700 1 --2- 172.21.2.112:0/711520250 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f744018aa50 0x7f74400b77d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:57.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.653 7f743f7fe700 1 -- 172.21.2.112:0/711520250 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74401aa9a0 con 0x7f744018a540 2021-02-10T11:46:57.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.653 7f743f7fe700 1 --2- 172.21.2.112:0/711520250 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f744018a540 0x7f7440197480 secure :-1 s=READY pgs=2955 cs=0 l=1 rev1=1 rx=0x7f743000b5b0 tx=0x7f743000b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:57.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.653 7f7427fff700 1 -- 172.21.2.112:0/711520250 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f743000bd40 con 0x7f744018a540 2021-02-10T11:46:57.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.653 7f7427fff700 1 -- 172.21.2.112:0/711520250 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7430002080 con 0x7f744018a540 2021-02-10T11:46:57.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.653 7f7427fff700 1 -- 172.21.2.112:0/711520250 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7430002510 con 0x7f744018a540 2021-02-10T11:46:57.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.653 7f7445a0d700 1 -- 172.21.2.112:0/711520250 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74401aa9a0 con 0x7f744018a540 2021-02-10T11:46:57.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.653 7f7445a0d700 1 -- 172.21.2.112:0/711520250 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74401aaf30 con 0x7f744018a540 2021-02-10T11:46:57.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.654 7f7427fff700 1 -- 172.21.2.112:0/711520250 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7430002260 con 0x7f744018a540 2021-02-10T11:46:57.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.654 7f7427fff700 1 --2- 172.21.2.112:0/711520250 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f74200186c0 0x7f742001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:57.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.654 7f7427fff700 1 -- 172.21.2.112:0/711520250 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f7430032580 con 0x7f744018a540 2021-02-10T11:46:57.659 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.655 7f743effd700 1 --2- 172.21.2.112:0/711520250 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f74200186c0 0x7f742001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:57.660 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.655 7f743effd700 1 --2- 172.21.2.112:0/711520250 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f74200186c0 0x7f742001abc0 crc :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:57.688 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.685 7f7445a0d700 1 -- 172.21.2.112:0/711520250 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74401aa250 con 0x7f744018a540 2021-02-10T11:46:57.690 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.688 7f7427fff700 1 -- 172.21.2.112:0/711520250 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7430020fa0 con 0x7f744018a540 2021-02-10T11:46:57.823 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:57.820 7f7445a0d700 1 -- 172.21.2.112:0/711520250 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_recover_auth_metadata_during_authorize"]} v 0) v1 -- 0x7f74401aa250 con 0x7f744018a540 2021-02-10T11:46:58.288 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:58.286 7f7427fff700 1 -- 172.21.2.112:0/711520250 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_recover_auth_metadata_during_authorize"]}]=0 v816) v1 ==== 160+0+0 (secure 0 0 0) 0x7f7430020fa0 con 0x7f744018a540 2021-02-10T11:46:58.290 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:58.288 7f7445a0d700 1 -- 172.21.2.112:0/711520250 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f74200186c0 msgr2=0x7f742001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:58.291 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:58.288 7f7445a0d700 1 --2- 172.21.2.112:0/711520250 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f74200186c0 0x7f742001abc0 crc :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:58.291 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:58.288 7f7445a0d700 1 -- 172.21.2.112:0/711520250 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f744018a540 msgr2=0x7f7440197480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:58.291 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:58.288 7f7445a0d700 1 --2- 172.21.2.112:0/711520250 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f744018a540 0x7f7440197480 secure :-1 s=READY pgs=2955 cs=0 l=1 rev1=1 rx=0x7f743000b5b0 tx=0x7f743000b950).stop 2021-02-10T11:46:58.291 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:58.288 7f7445a0d700 1 -- 172.21.2.112:0/711520250 shutdown_connections 2021-02-10T11:46:58.291 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:58.288 7f7445a0d700 1 --2- 172.21.2.112:0/711520250 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f74200186c0 0x7f742001abc0 unknown :-1 s=CLOSED pgs=1336 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:58.291 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:58.288 7f7445a0d700 1 --2- 172.21.2.112:0/711520250 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f744018aa50 0x7f74400b77d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:58.292 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:58.288 7f7445a0d700 1 --2- 172.21.2.112:0/711520250 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f74400b7df0 0x7f74400b81f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:58.292 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:58.288 7f7445a0d700 1 --2- 172.21.2.112:0/711520250 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f744018a540 0x7f7440197480 unknown :-1 s=CLOSED pgs=2955 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:58.292 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:58.288 7f7445a0d700 1 -- 172.21.2.112:0/711520250 >> 172.21.2.112:0/711520250 conn(0x7f74400efc70 msgr2=0x7f74400ef410 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:58.292 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:58.289 7f7445a0d700 1 -- 172.21.2.112:0/711520250 shutdown_connections 2021-02-10T11:46:58.292 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:58.289 7f7445a0d700 1 -- 172.21.2.112:0/711520250 wait complete. 2021-02-10T11:46:58.299 INFO:tasks.cephfs_test_runner:test_recover_auth_metadata_during_authorize (tasks.cephfs.test_volumes.TestVolumes) ... ok 2021-02-10T11:46:58.299 INFO:tasks.cephfs_test_runner:Starting test: test_recover_auth_metadata_during_deauthorize (tasks.cephfs.test_volumes.TestVolumes) 2021-02-10T11:46:58.300 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestVolumes.test_recover_auth_metadata_during_deauthorize' 2021-02-10T11:46:58.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:58.708 7f97c33dd700 1 Processor -- start 2021-02-10T11:46:58.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:58.708 7f97c33dd700 1 -- start start 2021-02-10T11:46:58.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:58.709 7f97c33dd700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f97bc0d1870 0x7f97bc1b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:58.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:58.709 7f97c33dd700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f97bc0f0650 0x7f97bc0cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:58.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:58.709 7f97c33dd700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97bc0f0b10 con 0x7f97bc1b0a10 2021-02-10T11:46:58.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:58.709 7f97c33dd700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97bc0e5ed0 con 0x7f97bc0d1870 2021-02-10T11:46:58.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:58.709 7f97c33dd700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97bc0d1c70 con 0x7f97bc0f0650 2021-02-10T11:46:58.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:58.709 7f97c11f9700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f97bc0d1870 0x7f97bc1b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:58.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:58.709 7f97c11f9700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f97bc0d1870 0x7f97bc1b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:59962/0 (socket says 172.21.2.112:59962) 2021-02-10T11:46:58.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:58.709 7f97c11f9700 1 -- 172.21.2.112:0/658427274 learned_addr learned my addr 172.21.2.112:0/658427274 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:46:58.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:58.709 7f97c11f9700 1 -- 172.21.2.112:0/658427274 >> v2:172.21.2.120:3300/0 conn(0x7f97bc0f0650 msgr2=0x7f97bc0cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:46:58.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:58.709 7f97c11f9700 1 --2- 172.21.2.112:0/658427274 >> v2:172.21.2.120:3300/0 conn(0x7f97bc0f0650 0x7f97bc0cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:58.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:58.709 7f97c11f9700 1 -- 172.21.2.112:0/658427274 >> v1:172.21.2.120:6789/0 conn(0x7f97bc1b0a10 legacy=0x7f97bc0cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:46:58.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:58.709 7f97c11f9700 1 -- 172.21.2.112:0/658427274 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97bc0d1e90 con 0x7f97bc0d1870 2021-02-10T11:46:58.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:58.709 7f97c11f9700 1 --2- 172.21.2.112:0/658427274 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f97bc0d1870 0x7f97bc1b03f0 secure :-1 s=READY pgs=2956 cs=0 l=1 rev1=1 rx=0x7f97b0006700 tx=0x7f97b0009b80).ready entity=mon.0 client_cookie=6899b2e0e2f88b94 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:58.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:58.710 7f97affff700 1 -- 172.21.2.112:0/658427274 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f97b000c0c0 con 0x7f97bc0d1870 2021-02-10T11:46:58.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:58.710 7f97affff700 1 -- 172.21.2.112:0/658427274 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97b000c340 con 0x7f97bc0d1870 2021-02-10T11:46:58.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:58.710 7f97affff700 1 -- 172.21.2.112:0/658427274 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f97b000c860 con 0x7f97bc0d1870 2021-02-10T11:46:58.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:58.710 7f97c33dd700 1 -- 172.21.2.112:0/658427274 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f97bc0d1870 msgr2=0x7f97bc1b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:58.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:58.710 7f97c33dd700 1 --2- 172.21.2.112:0/658427274 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f97bc0d1870 0x7f97bc1b03f0 secure :-1 s=READY pgs=2956 cs=0 l=1 rev1=1 rx=0x7f97b0006700 tx=0x7f97b0009b80).stop 2021-02-10T11:46:58.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:58.710 7f97c33dd700 1 -- 172.21.2.112:0/658427274 shutdown_connections 2021-02-10T11:46:58.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:58.710 7f97c33dd700 1 --2- 172.21.2.112:0/658427274 >> v2:172.21.2.120:3300/0 conn(0x7f97bc0f0650 0x7f97bc0cfb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:58.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:58.710 7f97c33dd700 1 --2- 172.21.2.112:0/658427274 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f97bc0d1870 0x7f97bc1b03f0 unknown :-1 s=CLOSED pgs=2956 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:58.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:58.710 7f97c33dd700 1 -- 172.21.2.112:0/658427274 >> 172.21.2.112:0/658427274 conn(0x7f97bc0ce450 msgr2=0x7f97bc0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:58.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:58.711 7f97c33dd700 1 -- 172.21.2.112:0/658427274 shutdown_connections 2021-02-10T11:46:58.729 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:58.711 7f97c33dd700 1 -- 172.21.2.112:0/658427274 wait complete. 2021-02-10T11:46:58.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:58.725 7f97c33dd700 1 Processor -- start 2021-02-10T11:46:58.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:58.725 7f97c33dd700 1 -- start start 2021-02-10T11:46:58.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:58.726 7f97c33dd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f97bc18a540 0x7f97bc197480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:58.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:58.726 7f97c33dd700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f97bc18aa50 0x7f97bc0b77d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:58.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:58.726 7f97c33dd700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f97bc0b7df0 0x7f97bc0b81f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:58.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:58.726 7f97c33dd700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97bc1a79d0 con 0x7f97bc18a540 2021-02-10T11:46:58.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:58.726 7f97c33dd700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97bc1a9e20 con 0x7f97bc18aa50 2021-02-10T11:46:58.731 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:58.726 7f97c33dd700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97bc1aa780 con 0x7f97bc0b7df0 2021-02-10T11:46:58.732 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:58.726 7f97c11f9700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f97bc18a540 0x7f97bc197480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:58.732 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:58.726 7f97c11f9700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f97bc18a540 0x7f97bc197480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:59966/0 (socket says 172.21.2.112:59966) 2021-02-10T11:46:58.732 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:58.726 7f97c11f9700 1 -- 172.21.2.112:0/2006378763 learned_addr learned my addr 172.21.2.112:0/2006378763 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:46:58.732 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:58.726 7f97c11f9700 1 -- 172.21.2.112:0/2006378763 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f97bc0b7df0 msgr2=0x7f97bc0b81f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:58.732 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:58.726 7f97c11f9700 1 --2- 172.21.2.112:0/2006378763 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f97bc0b7df0 0x7f97bc0b81f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:58.732 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:58.726 7f97c11f9700 1 -- 172.21.2.112:0/2006378763 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f97bc18aa50 msgr2=0x7f97bc0b77d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:58.732 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:58.726 7f97c11f9700 1 --2- 172.21.2.112:0/2006378763 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f97bc18aa50 0x7f97bc0b77d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:58.732 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:58.726 7f97c11f9700 1 -- 172.21.2.112:0/2006378763 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97bc1aa9a0 con 0x7f97bc18a540 2021-02-10T11:46:58.732 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:58.726 7f97c11f9700 1 --2- 172.21.2.112:0/2006378763 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f97bc18a540 0x7f97bc197480 secure :-1 s=READY pgs=2957 cs=0 l=1 rev1=1 rx=0x7f97b000b5e0 tx=0x7f97b000b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:58.732 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:58.727 7f97adffb700 1 -- 172.21.2.112:0/2006378763 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f97b000bd70 con 0x7f97bc18a540 2021-02-10T11:46:58.732 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:58.727 7f97adffb700 1 -- 172.21.2.112:0/2006378763 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97b0002030 con 0x7f97bc18a540 2021-02-10T11:46:58.732 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:58.727 7f97c33dd700 1 -- 172.21.2.112:0/2006378763 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97bc1aa9a0 con 0x7f97bc18a540 2021-02-10T11:46:58.732 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:58.727 7f97c33dd700 1 -- 172.21.2.112:0/2006378763 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97bc1aaf30 con 0x7f97bc18a540 2021-02-10T11:46:58.732 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:58.727 7f97adffb700 1 -- 172.21.2.112:0/2006378763 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f97b00009b0 con 0x7f97bc18a540 2021-02-10T11:46:58.733 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:58.727 7f97adffb700 1 -- 172.21.2.112:0/2006378763 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f97b00024e0 con 0x7f97bc18a540 2021-02-10T11:46:58.733 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:58.727 7f97adffb700 1 --2- 172.21.2.112:0/2006378763 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f979c018730 0x7f979c01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:58.733 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:58.727 7f97adffb700 1 -- 172.21.2.112:0/2006378763 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f97b00327c0 con 0x7f97bc18a540 2021-02-10T11:46:58.733 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:58.728 7f97c09f8700 1 --2- 172.21.2.112:0/2006378763 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f979c018730 0x7f979c01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:58.733 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:58.728 7f97c09f8700 1 --2- 172.21.2.112:0/2006378763 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f979c018730 0x7f979c01ac10 crc :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:58.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:58.759 7f97c33dd700 1 -- 172.21.2.112:0/2006378763 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97bc1aa250 con 0x7f97bc18a540 2021-02-10T11:46:58.764 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:58.761 7f97adffb700 1 -- 172.21.2.112:0/2006378763 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f97b00022d0 con 0x7f97bc18a540 2021-02-10T11:46:58.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:58.894 7f97c33dd700 1 -- 172.21.2.112:0/2006378763 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_recover_auth_metadata_during_deauthorize"]} v 0) v1 -- 0x7f97bc1aa250 con 0x7f97bc18a540 2021-02-10T11:46:59.293 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:59.290 7f97adffb700 1 -- 172.21.2.112:0/2006378763 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_recover_auth_metadata_during_deauthorize"]}]=0 v817) v1 ==== 165+0+0 (secure 0 0 0) 0x7f97b00024e0 con 0x7f97bc18a540 2021-02-10T11:46:59.314 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:59.312 7f97c33dd700 1 -- 172.21.2.112:0/2006378763 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f979c018730 msgr2=0x7f979c01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:59.314 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:59.312 7f97c33dd700 1 --2- 172.21.2.112:0/2006378763 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f979c018730 0x7f979c01ac10 crc :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:59.315 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:59.312 7f97c33dd700 1 -- 172.21.2.112:0/2006378763 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f97bc18a540 msgr2=0x7f97bc197480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:59.315 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:59.312 7f97c33dd700 1 --2- 172.21.2.112:0/2006378763 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f97bc18a540 0x7f97bc197480 secure :-1 s=READY pgs=2957 cs=0 l=1 rev1=1 rx=0x7f97b000b5e0 tx=0x7f97b000b980).stop 2021-02-10T11:46:59.315 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:59.312 7f97c33dd700 1 -- 172.21.2.112:0/2006378763 shutdown_connections 2021-02-10T11:46:59.315 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:59.312 7f97c33dd700 1 --2- 172.21.2.112:0/2006378763 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f979c018730 0x7f979c01ac10 unknown :-1 s=CLOSED pgs=1337 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:59.315 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:59.312 7f97c33dd700 1 --2- 172.21.2.112:0/2006378763 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f97bc18aa50 0x7f97bc0b77d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:59.315 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:59.312 7f97c33dd700 1 --2- 172.21.2.112:0/2006378763 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f97bc0b7df0 0x7f97bc0b81f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:59.315 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:59.312 7f97c33dd700 1 --2- 172.21.2.112:0/2006378763 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f97bc18a540 0x7f97bc197480 unknown :-1 s=CLOSED pgs=2957 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:59.315 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:59.312 7f97c33dd700 1 -- 172.21.2.112:0/2006378763 >> 172.21.2.112:0/2006378763 conn(0x7f97bc0efc70 msgr2=0x7f97bc0ef410 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:59.316 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:59.312 7f97c33dd700 1 -- 172.21.2.112:0/2006378763 shutdown_connections 2021-02-10T11:46:59.316 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:59.312 7f97c33dd700 1 -- 172.21.2.112:0/2006378763 wait complete. 2021-02-10T11:46:59.323 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:46:59.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:59.736 7fab328b7700 1 Processor -- start 2021-02-10T11:46:59.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:59.736 7fab328b7700 1 -- start start 2021-02-10T11:46:59.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:59.737 7fab328b7700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fab2c0d1c70 0x7fab2c0cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:59.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:59.737 7fab328b7700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fab2c0cf580 0x7fab2c0cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:59.757 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:59.737 7fab328b7700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab2c0fe1c0 con 0x7fab2c0d1870 2021-02-10T11:46:59.757 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:59.737 7fab328b7700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab2c1b0cb0 con 0x7fab2c0d1c70 2021-02-10T11:46:59.757 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:59.737 7fab328b7700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab2c109a60 con 0x7fab2c0cf580 2021-02-10T11:46:59.757 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:59.738 7fab30ed4700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fab2c0cf580 0x7fab2c0cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:59.757 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:59.738 7fab2b7fe700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fab2c0d1c70 0x7fab2c0cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:59.757 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:59.738 7fab2bfff700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7fab2c0d1870 0x7fab2c1b04c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:52994/0 (socket says 172.21.2.112:52994) 2021-02-10T11:46:59.757 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:59.738 7fab2bfff700 1 -- 172.21.2.112:0/1819892535 learned_addr learned my addr 172.21.2.112:0/1819892535 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:46:59.757 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:59.738 7fab2affd700 1 -- 172.21.2.112:0/1819892535 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3345823395 0 0) 0x7fab1c0010c0 con 0x7fab2c0d1870 2021-02-10T11:46:59.757 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:59.738 7fab2affd700 1 -- 172.21.2.112:0/1819892535 --> v1:172.21.2.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab14002fe0 con 0x7fab2c0d1870 2021-02-10T11:46:59.757 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:59.738 7fab30ed4700 1 -- 172.21.2.112:0/1819892535 >> v2:172.21.2.120:3300/0 conn(0x7fab2c0d1c70 msgr2=0x7fab2c0cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:46:59.757 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:59.738 7fab30ed4700 1 --2- 172.21.2.112:0/1819892535 >> v2:172.21.2.120:3300/0 conn(0x7fab2c0d1c70 0x7fab2c0cef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:59.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:59.738 7fab30ed4700 1 -- 172.21.2.112:0/1819892535 >> v1:172.21.2.120:6789/0 conn(0x7fab2c0d1870 legacy=0x7fab2c1b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:59.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:59.738 7fab30ed4700 1 -- 172.21.2.112:0/1819892535 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab2c088070 con 0x7fab2c0cf580 2021-02-10T11:46:59.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:59.738 7fab2b7fe700 1 --2- 172.21.2.112:0/1819892535 >> v2:172.21.2.120:3300/0 conn(0x7fab2c0d1c70 0x7fab2c0cef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:46:59.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:59.739 7fab30ed4700 1 --2- 172.21.2.112:0/1819892535 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fab2c0cf580 0x7fab2c0cf980 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7fab24006e50 tx=0x7fab24009e70).ready entity=mon.2 client_cookie=1735f39b514156e9 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:59.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:59.739 7fab2affd700 1 -- 172.21.2.112:0/1819892535 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fab24006a20 con 0x7fab2c0cf580 2021-02-10T11:46:59.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:59.739 7fab2affd700 1 -- 172.21.2.112:0/1819892535 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab24006790 con 0x7fab2c0cf580 2021-02-10T11:46:59.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:59.739 7fab2affd700 1 -- 172.21.2.112:0/1819892535 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fab2400b700 con 0x7fab2c0cf580 2021-02-10T11:46:59.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:59.739 7fab328b7700 1 -- 172.21.2.112:0/1819892535 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fab2c0cf580 msgr2=0x7fab2c0cf980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:59.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:59.739 7fab328b7700 1 --2- 172.21.2.112:0/1819892535 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fab2c0cf580 0x7fab2c0cf980 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7fab24006e50 tx=0x7fab24009e70).stop 2021-02-10T11:46:59.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:59.739 7fab328b7700 1 -- 172.21.2.112:0/1819892535 shutdown_connections 2021-02-10T11:46:59.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:59.739 7fab328b7700 1 --2- 172.21.2.112:0/1819892535 >> v2:172.21.2.120:3300/0 conn(0x7fab2c0d1c70 0x7fab2c0cef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:59.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:59.739 7fab328b7700 1 --2- 172.21.2.112:0/1819892535 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fab2c0cf580 0x7fab2c0cf980 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:59.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:59.739 7fab328b7700 1 -- 172.21.2.112:0/1819892535 >> 172.21.2.112:0/1819892535 conn(0x7fab2c0ce420 msgr2=0x7fab2c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:59.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:59.740 7fab328b7700 1 -- 172.21.2.112:0/1819892535 shutdown_connections 2021-02-10T11:46:59.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:59.740 7fab328b7700 1 -- 172.21.2.112:0/1819892535 wait complete. 2021-02-10T11:46:59.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:59.754 7fab328b7700 1 Processor -- start 2021-02-10T11:46:59.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:59.754 7fab328b7700 1 -- start start 2021-02-10T11:46:59.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:59.754 7fab328b7700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fab2c1a79a0 0x7fab2c1a7da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:59.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:59.755 7fab328b7700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fab2c0bdb40 0x7fab2c0bdf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:59.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:59.755 7fab328b7700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fab2c082ca0 0x7fab2c0830a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:59.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:59.755 7fab328b7700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab2c19d990 con 0x7fab2c0bdb40 2021-02-10T11:46:59.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:59.755 7fab328b7700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab2c19dbb0 con 0x7fab2c1a79a0 2021-02-10T11:46:59.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:59.755 7fab328b7700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab2c0836c0 con 0x7fab2c082ca0 2021-02-10T11:46:59.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:59.755 7fab2b7fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fab2c0bdb40 0x7fab2c0bdf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:59.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:59.755 7fab2b7fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fab2c0bdb40 0x7fab2c0bdf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:59984/0 (socket says 172.21.2.112:59984) 2021-02-10T11:46:59.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:59.755 7fab2b7fe700 1 -- 172.21.2.112:0/2055898290 learned_addr learned my addr 172.21.2.112:0/2055898290 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:46:59.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:59.755 7fab2bfff700 1 --2- 172.21.2.112:0/2055898290 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fab2c1a79a0 0x7fab2c1a7da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:59.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:59.755 7fab30ed4700 1 --2- 172.21.2.112:0/2055898290 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fab2c082ca0 0x7fab2c0830a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:59.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:59.755 7fab2b7fe700 1 -- 172.21.2.112:0/2055898290 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fab2c082ca0 msgr2=0x7fab2c0830a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:59.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:59.755 7fab2b7fe700 1 --2- 172.21.2.112:0/2055898290 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fab2c082ca0 0x7fab2c0830a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:59.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:59.755 7fab2b7fe700 1 -- 172.21.2.112:0/2055898290 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fab2c1a79a0 msgr2=0x7fab2c1a7da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:59.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:59.755 7fab2b7fe700 1 --2- 172.21.2.112:0/2055898290 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fab2c1a79a0 0x7fab2c1a7da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:59.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:59.755 7fab2b7fe700 1 -- 172.21.2.112:0/2055898290 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab2c0838e0 con 0x7fab2c0bdb40 2021-02-10T11:46:59.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:59.756 7fab2b7fe700 1 --2- 172.21.2.112:0/2055898290 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fab2c0bdb40 0x7fab2c0bdf40 secure :-1 s=READY pgs=2958 cs=0 l=1 rev1=1 rx=0x7fab20006f40 tx=0x7fab2000b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:59.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:59.756 7fab28ff9700 1 -- 172.21.2.112:0/2055898290 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fab20006bd0 con 0x7fab2c0bdb40 2021-02-10T11:46:59.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:59.756 7fab28ff9700 1 -- 172.21.2.112:0/2055898290 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab2000bd80 con 0x7fab2c0bdb40 2021-02-10T11:46:59.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:59.756 7fab28ff9700 1 -- 172.21.2.112:0/2055898290 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fab200022b0 con 0x7fab2c0bdb40 2021-02-10T11:46:59.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:59.756 7fab328b7700 1 -- 172.21.2.112:0/2055898290 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab2c0838e0 con 0x7fab2c0bdb40 2021-02-10T11:46:59.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:59.756 7fab328b7700 1 -- 172.21.2.112:0/2055898290 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab2c084420 con 0x7fab2c0bdb40 2021-02-10T11:46:59.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:59.756 7fab28ff9700 1 -- 172.21.2.112:0/2055898290 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fab20000c90 con 0x7fab2c0bdb40 2021-02-10T11:46:59.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:59.756 7fab28ff9700 1 --2- 172.21.2.112:0/2055898290 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fab0c0186c0 0x7fab0c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:59.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:59.756 7fab28ff9700 1 -- 172.21.2.112:0/2055898290 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fab200323e0 con 0x7fab2c0bdb40 2021-02-10T11:46:59.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:59.757 7fab2bfff700 1 --2- 172.21.2.112:0/2055898290 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fab0c0186c0 0x7fab0c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:59.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:59.757 7fab2bfff700 1 --2- 172.21.2.112:0/2055898290 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fab0c0186c0 0x7fab0c01abc0 crc :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:59.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:59.786 7fab328b7700 1 -- 172.21.2.112:0/2055898290 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab2c082a70 con 0x7fab2c0bdb40 2021-02-10T11:46:59.791 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:59.789 7fab28ff9700 1 -- 172.21.2.112:0/2055898290 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fab200009b0 con 0x7fab2c0bdb40 2021-02-10T11:46:59.940 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:59.937 7fab328b7700 1 -- 172.21.2.112:0/2055898290 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fab2c0838e0 con 0x7fab2c0bdb40 2021-02-10T11:46:59.943 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:59.941 7fab28ff9700 1 -- 172.21.2.112:0/2055898290 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v44)=0 v44) v1 ==== 122+0+0 (secure 0 0 0) 0x7fab20000c90 con 0x7fab2c0bdb40 2021-02-10T11:46:59.948 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:59.945 7fab328b7700 1 -- 172.21.2.112:0/2055898290 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fab0c0186c0 msgr2=0x7fab0c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:59.948 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:59.945 7fab328b7700 1 --2- 172.21.2.112:0/2055898290 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fab0c0186c0 0x7fab0c01abc0 crc :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:59.948 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:59.945 7fab328b7700 1 -- 172.21.2.112:0/2055898290 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fab2c0bdb40 msgr2=0x7fab2c0bdf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:59.949 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:59.945 7fab328b7700 1 --2- 172.21.2.112:0/2055898290 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fab2c0bdb40 0x7fab2c0bdf40 secure :-1 s=READY pgs=2958 cs=0 l=1 rev1=1 rx=0x7fab20006f40 tx=0x7fab2000b990).stop 2021-02-10T11:46:59.949 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:59.946 7fab328b7700 1 -- 172.21.2.112:0/2055898290 shutdown_connections 2021-02-10T11:46:59.949 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:59.946 7fab328b7700 1 --2- 172.21.2.112:0/2055898290 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fab0c0186c0 0x7fab0c01abc0 unknown :-1 s=CLOSED pgs=1338 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:59.949 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:59.946 7fab328b7700 1 --2- 172.21.2.112:0/2055898290 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fab2c082ca0 0x7fab2c0830a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:59.949 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:59.946 7fab328b7700 1 --2- 172.21.2.112:0/2055898290 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fab2c0bdb40 0x7fab2c0bdf40 unknown :-1 s=CLOSED pgs=2958 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:59.949 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:59.946 7fab328b7700 1 --2- 172.21.2.112:0/2055898290 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fab2c1a79a0 0x7fab2c1a7da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:59.949 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:59.946 7fab328b7700 1 -- 172.21.2.112:0/2055898290 >> 172.21.2.112:0/2055898290 conn(0x7fab2c0bef00 msgr2=0x7fab2c09f200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:59.949 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:59.946 7fab328b7700 1 -- 172.21.2.112:0/2055898290 shutdown_connections 2021-02-10T11:46:59.949 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:46:59.946 7fab328b7700 1 -- 172.21.2.112:0/2055898290 wait complete. 2021-02-10T11:46:59.957 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:46:59.958 DEBUG:teuthology.orchestra.run.gibba012:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:47:00.306 DEBUG:teuthology.orchestra.run.gibba020:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:47:00.651 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:47:00.652 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:47:00.652 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:47:00.652 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:47:00.989 INFO:teuthology.orchestra.run.gibba012.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.0’: No such file or directory 2021-02-10T11:47:00.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:47:00.990 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2021-02-10T11:47:00.990 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:47:00.990 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:47:00.990 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:47:01.324 INFO:teuthology.orchestra.run.gibba012.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: No such file or directory 2021-02-10T11:47:01.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:47:01.325 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2021-02-10T11:47:01.325 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:47:01.325 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:47:01.325 DEBUG:teuthology.orchestra.run.gibba020:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:47:01.660 INFO:teuthology.orchestra.run.gibba020.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:47:01.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:47:01.660 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:47:01.660 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:47:01.661 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:47:01.661 DEBUG:teuthology.orchestra.run.gibba020:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:47:01.995 INFO:teuthology.orchestra.run.gibba020.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:47:01.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:47:01.996 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:47:01.996 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json-pretty 2021-02-10T11:47:02.424 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:02.405 7f8753d32700 1 Processor -- start 2021-02-10T11:47:02.424 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:02.405 7f8753d32700 1 -- start start 2021-02-10T11:47:02.424 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:02.405 7f8753d32700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f874c0d1870 0x7f874c1b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:02.424 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:02.405 7f8753d32700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f874c1b0a10 0x7f874c0cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:02.424 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:02.405 7f8753d32700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f874c0f0b10 con 0x7f874c0d1c70 2021-02-10T11:47:02.424 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:02.405 7f8753d32700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f874c0e5ed0 con 0x7f874c0d1870 2021-02-10T11:47:02.424 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:02.405 7f8753d32700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f874c0f0650 con 0x7f874c1b0a10 2021-02-10T11:47:02.425 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:02.405 7f875234f700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f874c0d1c70 0x7f874c0cf950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:59450/0 (socket says 172.21.2.112:59450) 2021-02-10T11:47:02.425 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:02.405 7f875234f700 1 -- 172.21.2.112:0/342697882 learned_addr learned my addr 172.21.2.112:0/342697882 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:47:02.425 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:02.406 7f8751b4e700 1 --2- 172.21.2.112:0/342697882 >> v2:172.21.2.112:3300/0 conn(0x7f874c0d1870 0x7f874c1b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:02.425 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:02.406 7f875134d700 1 --2- 172.21.2.112:0/342697882 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f874c1b0a10 0x7f874c0cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:02.425 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:02.406 7f8751b4e700 1 -- 172.21.2.112:0/342697882 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f874c1b0a10 msgr2=0x7f874c0cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:02.425 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:02.406 7f8751b4e700 1 --2- 172.21.2.112:0/342697882 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f874c1b0a10 0x7f874c0cf330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:02.425 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:02.406 7f8751b4e700 1 -- 172.21.2.112:0/342697882 >> v1:172.21.2.112:6789/0 conn(0x7f874c0d1c70 legacy=0x7f874c0cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:02.425 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:02.406 7f8750b4c700 1 -- 172.21.2.112:0/342697882 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3366054134 0 0) 0x7f8744001500 con 0x7f874c0d1c70 2021-02-10T11:47:02.425 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:02.406 7f8751b4e700 1 -- 172.21.2.112:0/342697882 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f874c0f0870 con 0x7f874c0d1870 2021-02-10T11:47:02.425 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:02.406 7f8751b4e700 1 --2- 172.21.2.112:0/342697882 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f874c0d1870 0x7f874c1b03f0 secure :-1 s=READY pgs=2960 cs=0 l=1 rev1=1 rx=0x7f8748006650 tx=0x7f8748009a00).ready entity=mon.0 client_cookie=e7428c4267065213 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:02.425 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:02.406 7f8750b4c700 1 -- 172.21.2.112:0/342697882 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f87480009b0 con 0x7f874c0d1870 2021-02-10T11:47:02.425 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:02.406 7f8750b4c700 1 -- 172.21.2.112:0/342697882 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f874800c0c0 con 0x7f874c0d1870 2021-02-10T11:47:02.425 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:02.406 7f8750b4c700 1 -- 172.21.2.112:0/342697882 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f874800c5a0 con 0x7f874c0d1870 2021-02-10T11:47:02.426 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:02.406 7f8753d32700 1 -- 172.21.2.112:0/342697882 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f874c0d1870 msgr2=0x7f874c1b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:02.426 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:02.406 7f8753d32700 1 --2- 172.21.2.112:0/342697882 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f874c0d1870 0x7f874c1b03f0 secure :-1 s=READY pgs=2960 cs=0 l=1 rev1=1 rx=0x7f8748006650 tx=0x7f8748009a00).stop 2021-02-10T11:47:02.426 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:02.407 7f8753d32700 1 -- 172.21.2.112:0/342697882 shutdown_connections 2021-02-10T11:47:02.426 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:02.407 7f8753d32700 1 --2- 172.21.2.112:0/342697882 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f874c1b0a10 0x7f874c0cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:02.426 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:02.407 7f8753d32700 1 --2- 172.21.2.112:0/342697882 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f874c0d1870 0x7f874c1b03f0 unknown :-1 s=CLOSED pgs=2960 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:02.426 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:02.407 7f8753d32700 1 -- 172.21.2.112:0/342697882 >> 172.21.2.112:0/342697882 conn(0x7f874c0ce450 msgr2=0x7f874c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:02.426 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:02.407 7f8753d32700 1 -- 172.21.2.112:0/342697882 shutdown_connections 2021-02-10T11:47:02.426 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:02.407 7f8753d32700 1 -- 172.21.2.112:0/342697882 wait complete. 2021-02-10T11:47:02.428 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:02.422 7f8753d32700 1 Processor -- start 2021-02-10T11:47:02.428 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:02.422 7f8753d32700 1 -- start start 2021-02-10T11:47:02.428 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:02.422 7f8753d32700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f874c0ed0f0 0x7f874c19ff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:02.428 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:02.422 7f8753d32700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f874c0ed6f0 0x7f874c08e780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:02.428 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:02.422 7f8753d32700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f874c08eda0 0x7f874c0a9af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:02.428 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:02.422 7f8753d32700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f874c0b7af0 con 0x7f874c08eda0 2021-02-10T11:47:02.428 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:02.422 7f8753d32700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f874c1a05a0 con 0x7f874c0ed6f0 2021-02-10T11:47:02.428 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:02.422 7f8753d32700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f874c0aa110 con 0x7f874c0ed0f0 2021-02-10T11:47:02.428 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:02.422 7f875234f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f874c08eda0 0x7f874c0a9af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:02.428 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:02.422 7f875234f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f874c08eda0 0x7f874c0a9af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:59992/0 (socket says 172.21.2.112:59992) 2021-02-10T11:47:02.428 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:02.422 7f875234f700 1 -- 172.21.2.112:0/2121954913 learned_addr learned my addr 172.21.2.112:0/2121954913 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:47:02.429 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:02.422 7f8751b4e700 1 --2- 172.21.2.112:0/2121954913 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f874c0ed0f0 0x7f874c19ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:02.429 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:02.422 7f875134d700 1 --2- 172.21.2.112:0/2121954913 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f874c0ed6f0 0x7f874c08e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:02.429 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:02.423 7f875234f700 1 -- 172.21.2.112:0/2121954913 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f874c0ed0f0 msgr2=0x7f874c19ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:02.429 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:02.423 7f875234f700 1 --2- 172.21.2.112:0/2121954913 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f874c0ed0f0 0x7f874c19ff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:02.429 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:02.423 7f875234f700 1 -- 172.21.2.112:0/2121954913 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f874c0ed6f0 msgr2=0x7f874c08e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:02.429 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:02.423 7f875234f700 1 --2- 172.21.2.112:0/2121954913 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f874c0ed6f0 0x7f874c08e780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:02.429 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:02.423 7f875234f700 1 -- 172.21.2.112:0/2121954913 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f874c0aa330 con 0x7f874c08eda0 2021-02-10T11:47:02.429 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:02.423 7f8751b4e700 1 --2- 172.21.2.112:0/2121954913 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f874c0ed0f0 0x7f874c19ff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:47:02.429 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:02.423 7f875234f700 1 --2- 172.21.2.112:0/2121954913 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f874c08eda0 0x7f874c0a9af0 secure :-1 s=READY pgs=2961 cs=0 l=1 rev1=1 rx=0x7f8744001c10 tx=0x7f874400ac60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:02.429 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:02.423 7f873e7fc700 1 -- 172.21.2.112:0/2121954913 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f87440070d0 con 0x7f874c08eda0 2021-02-10T11:47:02.429 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:02.423 7f873e7fc700 1 -- 172.21.2.112:0/2121954913 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8744007450 con 0x7f874c08eda0 2021-02-10T11:47:02.429 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:02.423 7f8753d32700 1 -- 172.21.2.112:0/2121954913 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f874c0aa330 con 0x7f874c08eda0 2021-02-10T11:47:02.429 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:02.423 7f873e7fc700 1 -- 172.21.2.112:0/2121954913 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f87440049a0 con 0x7f874c08eda0 2021-02-10T11:47:02.430 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:02.423 7f8753d32700 1 -- 172.21.2.112:0/2121954913 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f874c0aa330 con 0x7f874c08eda0 2021-02-10T11:47:02.430 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:02.423 7f873e7fc700 1 -- 172.21.2.112:0/2121954913 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8744004720 con 0x7f874c08eda0 2021-02-10T11:47:02.430 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:02.424 7f873e7fc700 1 --2- 172.21.2.112:0/2121954913 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8734018730 0x7f873401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:02.430 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:02.424 7f873e7fc700 1 -- 172.21.2.112:0/2121954913 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f8744032b20 con 0x7f874c08eda0 2021-02-10T11:47:02.430 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:02.424 7f8751b4e700 1 --2- 172.21.2.112:0/2121954913 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8734018730 0x7f873401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:02.430 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:02.424 7f8751b4e700 1 --2- 172.21.2.112:0/2121954913 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8734018730 0x7f873401ac10 crc :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:02.457 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:02.455 7f8753d32700 1 -- 172.21.2.112:0/2121954913 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f874c0ebc00 con 0x7f874c08eda0 2021-02-10T11:47:02.460 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:02.457 7f873e7fc700 1 -- 172.21.2.112:0/2121954913 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8744000cc0 con 0x7f874c08eda0 2021-02-10T11:47:02.584 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:02.581 7f8753d32700 1 -- 172.21.2.112:0/2121954913 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f874c0f0b10 con 0x7f874c08eda0 2021-02-10T11:47:02.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:02.582 7f873e7fc700 1 -- 172.21.2.112:0/2121954913 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v172) v1 ==== 81+0+30374 (secure 0 0 0) 0x7f8744004720 con 0x7f874c08eda0 2021-02-10T11:47:02.586 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:47:02.586 INFO:teuthology.orchestra.run.gibba012.stdout:{ 2021-02-10T11:47:02.586 INFO:teuthology.orchestra.run.gibba012.stdout: "epoch": 172, 2021-02-10T11:47:02.586 INFO:teuthology.orchestra.run.gibba012.stdout: "fsid": "f14f004e-0699-4584-ad81-3754258493de", 2021-02-10T11:47:02.586 INFO:teuthology.orchestra.run.gibba012.stdout: "created": "2021-02-10 11:13:26.501424", 2021-02-10T11:47:02.587 INFO:teuthology.orchestra.run.gibba012.stdout: "modified": "2021-02-10 11:46:21.423567", 2021-02-10T11:47:02.587 INFO:teuthology.orchestra.run.gibba012.stdout: "last_up_change": "2021-02-10 11:13:52.780316", 2021-02-10T11:47:02.587 INFO:teuthology.orchestra.run.gibba012.stdout: "last_in_change": "2021-02-10 11:13:52.780316", 2021-02-10T11:47:02.587 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-02-10T11:47:02.587 INFO:teuthology.orchestra.run.gibba012.stdout: "flags_num": 5799936, 2021-02-10T11:47:02.588 INFO:teuthology.orchestra.run.gibba012.stdout: "flags_set": [ 2021-02-10T11:47:02.588 INFO:teuthology.orchestra.run.gibba012.stdout: "pglog_hardlimit", 2021-02-10T11:47:02.588 INFO:teuthology.orchestra.run.gibba012.stdout: "purged_snapdirs", 2021-02-10T11:47:02.588 INFO:teuthology.orchestra.run.gibba012.stdout: "recovery_deletes", 2021-02-10T11:47:02.588 INFO:teuthology.orchestra.run.gibba012.stdout: "sortbitwise" 2021-02-10T11:47:02.588 INFO:teuthology.orchestra.run.gibba012.stdout: ], 2021-02-10T11:47:02.589 INFO:teuthology.orchestra.run.gibba012.stdout: "crush_version": 4, 2021-02-10T11:47:02.589 INFO:teuthology.orchestra.run.gibba012.stdout: "full_ratio": 0.94999998807907104, 2021-02-10T11:47:02.589 INFO:teuthology.orchestra.run.gibba012.stdout: "backfillfull_ratio": 0.89999997615814209, 2021-02-10T11:47:02.589 INFO:teuthology.orchestra.run.gibba012.stdout: "nearfull_ratio": 0.85000002384185791, 2021-02-10T11:47:02.589 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_snapshot": "", 2021-02-10T11:47:02.590 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_max": 37, 2021-02-10T11:47:02.590 INFO:teuthology.orchestra.run.gibba012.stdout: "max_osd": 8, 2021-02-10T11:47:02.590 INFO:teuthology.orchestra.run.gibba012.stdout: "require_min_compat_client": "jewel", 2021-02-10T11:47:02.590 INFO:teuthology.orchestra.run.gibba012.stdout: "min_compat_client": "jewel", 2021-02-10T11:47:02.590 INFO:teuthology.orchestra.run.gibba012.stdout: "require_osd_release": "nautilus", 2021-02-10T11:47:02.590 INFO:teuthology.orchestra.run.gibba012.stdout: "pools": [ 2021-02-10T11:47:02.591 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:02.591 INFO:teuthology.orchestra.run.gibba012.stdout: "pool": 1, 2021-02-10T11:47:02.591 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_name": "rbd", 2021-02-10T11:47:02.591 INFO:teuthology.orchestra.run.gibba012.stdout: "create_time": "2021-02-10 11:13:57.508758", 2021-02-10T11:47:02.591 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:47:02.591 INFO:teuthology.orchestra.run.gibba012.stdout: "flags_names": "hashpspool", 2021-02-10T11:47:02.592 INFO:teuthology.orchestra.run.gibba012.stdout: "type": 1, 2021-02-10T11:47:02.592 INFO:teuthology.orchestra.run.gibba012.stdout: "size": 2, 2021-02-10T11:47:02.592 INFO:teuthology.orchestra.run.gibba012.stdout: "min_size": 1, 2021-02-10T11:47:02.592 INFO:teuthology.orchestra.run.gibba012.stdout: "crush_rule": 0, 2021-02-10T11:47:02.592 INFO:teuthology.orchestra.run.gibba012.stdout: "object_hash": 2, 2021-02-10T11:47:02.592 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:47:02.593 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num": 8, 2021-02-10T11:47:02.593 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num": 8, 2021-02-10T11:47:02.593 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num_target": 8, 2021-02-10T11:47:02.593 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_target": 8, 2021-02-10T11:47:02.593 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_pending": 8, 2021-02-10T11:47:02.594 INFO:teuthology.orchestra.run.gibba012.stdout: "last_pg_merge_meta": { 2021-02-10T11:47:02.594 INFO:teuthology.orchestra.run.gibba012.stdout: "source_pgid": "0.0", 2021-02-10T11:47:02.594 INFO:teuthology.orchestra.run.gibba012.stdout: "ready_epoch": 0, 2021-02-10T11:47:02.594 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_started": 0, 2021-02-10T11:47:02.594 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_clean": 0, 2021-02-10T11:47:02.595 INFO:teuthology.orchestra.run.gibba012.stdout: "source_version": "0'0", 2021-02-10T11:47:02.595 INFO:teuthology.orchestra.run.gibba012.stdout: "target_version": "0'0" 2021-02-10T11:47:02.595 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:02.595 INFO:teuthology.orchestra.run.gibba012.stdout: "last_change": "16", 2021-02-10T11:47:02.595 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend": "0", 2021-02-10T11:47:02.595 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:47:02.596 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:47:02.596 INFO:teuthology.orchestra.run.gibba012.stdout: "auid": 0, 2021-02-10T11:47:02.596 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:47:02.596 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_seq": 0, 2021-02-10T11:47:02.596 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_epoch": 0, 2021-02-10T11:47:02.596 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_snaps": [], 2021-02-10T11:47:02.597 INFO:teuthology.orchestra.run.gibba012.stdout: "removed_snaps": "[]", 2021-02-10T11:47:02.597 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_bytes": 0, 2021-02-10T11:47:02.597 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_objects": 0, 2021-02-10T11:47:02.597 INFO:teuthology.orchestra.run.gibba012.stdout: "tiers": [], 2021-02-10T11:47:02.597 INFO:teuthology.orchestra.run.gibba012.stdout: "tier_of": -1, 2021-02-10T11:47:02.597 INFO:teuthology.orchestra.run.gibba012.stdout: "read_tier": -1, 2021-02-10T11:47:02.597 INFO:teuthology.orchestra.run.gibba012.stdout: "write_tier": -1, 2021-02-10T11:47:02.598 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_mode": "none", 2021-02-10T11:47:02.598 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_bytes": 0, 2021-02-10T11:47:02.598 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_objects": 0, 2021-02-10T11:47:02.598 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:47:02.598 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:47:02.598 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:47:02.599 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_flush_age": 0, 2021-02-10T11:47:02.599 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_evict_age": 0, 2021-02-10T11:47:02.599 INFO:teuthology.orchestra.run.gibba012.stdout: "erasure_code_profile": "", 2021-02-10T11:47:02.599 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_params": { 2021-02-10T11:47:02.599 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "none" 2021-02-10T11:47:02.599 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:02.600 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_period": 0, 2021-02-10T11:47:02.600 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_count": 0, 2021-02-10T11:47:02.600 INFO:teuthology.orchestra.run.gibba012.stdout: "use_gmt_hitset": true, 2021-02-10T11:47:02.600 INFO:teuthology.orchestra.run.gibba012.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:47:02.600 INFO:teuthology.orchestra.run.gibba012.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:47:02.600 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:47:02.600 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:47:02.601 INFO:teuthology.orchestra.run.gibba012.stdout: "grade_table": [], 2021-02-10T11:47:02.601 INFO:teuthology.orchestra.run.gibba012.stdout: "stripe_width": 0, 2021-02-10T11:47:02.601 INFO:teuthology.orchestra.run.gibba012.stdout: "expected_num_objects": 0, 2021-02-10T11:47:02.601 INFO:teuthology.orchestra.run.gibba012.stdout: "fast_read": false, 2021-02-10T11:47:02.601 INFO:teuthology.orchestra.run.gibba012.stdout: "options": {}, 2021-02-10T11:47:02.601 INFO:teuthology.orchestra.run.gibba012.stdout: "application_metadata": { 2021-02-10T11:47:02.601 INFO:teuthology.orchestra.run.gibba012.stdout: "rbd": {} 2021-02-10T11:47:02.601 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:47:02.602 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:02.602 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:02.602 INFO:teuthology.orchestra.run.gibba012.stdout: "pool": 36, 2021-02-10T11:47:02.602 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_name": "cephfs_metadata", 2021-02-10T11:47:02.602 INFO:teuthology.orchestra.run.gibba012.stdout: "create_time": "2021-02-10 11:46:18.395022", 2021-02-10T11:47:02.602 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:47:02.602 INFO:teuthology.orchestra.run.gibba012.stdout: "flags_names": "hashpspool", 2021-02-10T11:47:02.602 INFO:teuthology.orchestra.run.gibba012.stdout: "type": 1, 2021-02-10T11:47:02.603 INFO:teuthology.orchestra.run.gibba012.stdout: "size": 2, 2021-02-10T11:47:02.603 INFO:teuthology.orchestra.run.gibba012.stdout: "min_size": 1, 2021-02-10T11:47:02.603 INFO:teuthology.orchestra.run.gibba012.stdout: "crush_rule": 0, 2021-02-10T11:47:02.603 INFO:teuthology.orchestra.run.gibba012.stdout: "object_hash": 2, 2021-02-10T11:47:02.603 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:47:02.603 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num": 8, 2021-02-10T11:47:02.603 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num": 8, 2021-02-10T11:47:02.603 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num_target": 8, 2021-02-10T11:47:02.604 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_target": 8, 2021-02-10T11:47:02.604 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_pending": 8, 2021-02-10T11:47:02.604 INFO:teuthology.orchestra.run.gibba012.stdout: "last_pg_merge_meta": { 2021-02-10T11:47:02.604 INFO:teuthology.orchestra.run.gibba012.stdout: "source_pgid": "0.0", 2021-02-10T11:47:02.604 INFO:teuthology.orchestra.run.gibba012.stdout: "ready_epoch": 0, 2021-02-10T11:47:02.604 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_started": 0, 2021-02-10T11:47:02.604 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_clean": 0, 2021-02-10T11:47:02.604 INFO:teuthology.orchestra.run.gibba012.stdout: "source_version": "0'0", 2021-02-10T11:47:02.605 INFO:teuthology.orchestra.run.gibba012.stdout: "target_version": "0'0" 2021-02-10T11:47:02.605 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:02.605 INFO:teuthology.orchestra.run.gibba012.stdout: "last_change": "171", 2021-02-10T11:47:02.605 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend": "0", 2021-02-10T11:47:02.605 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:47:02.605 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:47:02.605 INFO:teuthology.orchestra.run.gibba012.stdout: "auid": 0, 2021-02-10T11:47:02.605 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:47:02.605 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_seq": 0, 2021-02-10T11:47:02.605 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_epoch": 0, 2021-02-10T11:47:02.605 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_snaps": [], 2021-02-10T11:47:02.605 INFO:teuthology.orchestra.run.gibba012.stdout: "removed_snaps": "[]", 2021-02-10T11:47:02.606 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_bytes": 0, 2021-02-10T11:47:02.606 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_objects": 0, 2021-02-10T11:47:02.606 INFO:teuthology.orchestra.run.gibba012.stdout: "tiers": [], 2021-02-10T11:47:02.606 INFO:teuthology.orchestra.run.gibba012.stdout: "tier_of": -1, 2021-02-10T11:47:02.606 INFO:teuthology.orchestra.run.gibba012.stdout: "read_tier": -1, 2021-02-10T11:47:02.606 INFO:teuthology.orchestra.run.gibba012.stdout: "write_tier": -1, 2021-02-10T11:47:02.606 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_mode": "none", 2021-02-10T11:47:02.606 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_bytes": 0, 2021-02-10T11:47:02.606 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_objects": 0, 2021-02-10T11:47:02.606 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:47:02.606 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:47:02.606 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:47:02.606 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_flush_age": 0, 2021-02-10T11:47:02.607 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_evict_age": 0, 2021-02-10T11:47:02.607 INFO:teuthology.orchestra.run.gibba012.stdout: "erasure_code_profile": "", 2021-02-10T11:47:02.607 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_params": { 2021-02-10T11:47:02.607 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "none" 2021-02-10T11:47:02.607 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:02.607 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_period": 0, 2021-02-10T11:47:02.607 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_count": 0, 2021-02-10T11:47:02.607 INFO:teuthology.orchestra.run.gibba012.stdout: "use_gmt_hitset": true, 2021-02-10T11:47:02.607 INFO:teuthology.orchestra.run.gibba012.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:47:02.607 INFO:teuthology.orchestra.run.gibba012.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:47:02.607 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:47:02.607 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:47:02.607 INFO:teuthology.orchestra.run.gibba012.stdout: "grade_table": [], 2021-02-10T11:47:02.608 INFO:teuthology.orchestra.run.gibba012.stdout: "stripe_width": 0, 2021-02-10T11:47:02.608 INFO:teuthology.orchestra.run.gibba012.stdout: "expected_num_objects": 0, 2021-02-10T11:47:02.608 INFO:teuthology.orchestra.run.gibba012.stdout: "fast_read": false, 2021-02-10T11:47:02.608 INFO:teuthology.orchestra.run.gibba012.stdout: "options": { 2021-02-10T11:47:02.608 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_autoscale_bias": 4, 2021-02-10T11:47:02.608 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_min": 16, 2021-02-10T11:47:02.608 INFO:teuthology.orchestra.run.gibba012.stdout: "recovery_priority": 5 2021-02-10T11:47:02.608 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:02.608 INFO:teuthology.orchestra.run.gibba012.stdout: "application_metadata": { 2021-02-10T11:47:02.608 INFO:teuthology.orchestra.run.gibba012.stdout: "cephfs": { 2021-02-10T11:47:02.609 INFO:teuthology.orchestra.run.gibba012.stdout: "metadata": "cephfs" 2021-02-10T11:47:02.609 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:47:02.609 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:47:02.609 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:02.609 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:02.609 INFO:teuthology.orchestra.run.gibba012.stdout: "pool": 37, 2021-02-10T11:47:02.610 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_name": "cephfs_data", 2021-02-10T11:47:02.610 INFO:teuthology.orchestra.run.gibba012.stdout: "create_time": "2021-02-10 11:46:19.415292", 2021-02-10T11:47:02.610 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:47:02.610 INFO:teuthology.orchestra.run.gibba012.stdout: "flags_names": "hashpspool", 2021-02-10T11:47:02.610 INFO:teuthology.orchestra.run.gibba012.stdout: "type": 1, 2021-02-10T11:47:02.611 INFO:teuthology.orchestra.run.gibba012.stdout: "size": 2, 2021-02-10T11:47:02.611 INFO:teuthology.orchestra.run.gibba012.stdout: "min_size": 1, 2021-02-10T11:47:02.611 INFO:teuthology.orchestra.run.gibba012.stdout: "crush_rule": 0, 2021-02-10T11:47:02.611 INFO:teuthology.orchestra.run.gibba012.stdout: "object_hash": 2, 2021-02-10T11:47:02.611 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:47:02.612 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num": 8, 2021-02-10T11:47:02.612 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num": 8, 2021-02-10T11:47:02.612 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num_target": 8, 2021-02-10T11:47:02.612 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_target": 8, 2021-02-10T11:47:02.612 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_pending": 8, 2021-02-10T11:47:02.612 INFO:teuthology.orchestra.run.gibba012.stdout: "last_pg_merge_meta": { 2021-02-10T11:47:02.613 INFO:teuthology.orchestra.run.gibba012.stdout: "source_pgid": "0.0", 2021-02-10T11:47:02.613 INFO:teuthology.orchestra.run.gibba012.stdout: "ready_epoch": 0, 2021-02-10T11:47:02.613 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_started": 0, 2021-02-10T11:47:02.613 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_clean": 0, 2021-02-10T11:47:02.613 INFO:teuthology.orchestra.run.gibba012.stdout: "source_version": "0'0", 2021-02-10T11:47:02.613 INFO:teuthology.orchestra.run.gibba012.stdout: "target_version": "0'0" 2021-02-10T11:47:02.614 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:02.614 INFO:teuthology.orchestra.run.gibba012.stdout: "last_change": "172", 2021-02-10T11:47:02.614 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend": "0", 2021-02-10T11:47:02.614 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:47:02.614 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:47:02.615 INFO:teuthology.orchestra.run.gibba012.stdout: "auid": 0, 2021-02-10T11:47:02.615 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:47:02.615 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_seq": 0, 2021-02-10T11:47:02.615 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_epoch": 0, 2021-02-10T11:47:02.615 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_snaps": [], 2021-02-10T11:47:02.615 INFO:teuthology.orchestra.run.gibba012.stdout: "removed_snaps": "[]", 2021-02-10T11:47:02.616 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_bytes": 0, 2021-02-10T11:47:02.616 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_objects": 0, 2021-02-10T11:47:02.616 INFO:teuthology.orchestra.run.gibba012.stdout: "tiers": [], 2021-02-10T11:47:02.616 INFO:teuthology.orchestra.run.gibba012.stdout: "tier_of": -1, 2021-02-10T11:47:02.616 INFO:teuthology.orchestra.run.gibba012.stdout: "read_tier": -1, 2021-02-10T11:47:02.616 INFO:teuthology.orchestra.run.gibba012.stdout: "write_tier": -1, 2021-02-10T11:47:02.617 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_mode": "none", 2021-02-10T11:47:02.617 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_bytes": 0, 2021-02-10T11:47:02.617 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_objects": 0, 2021-02-10T11:47:02.617 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:47:02.617 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:47:02.618 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:47:02.618 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_flush_age": 0, 2021-02-10T11:47:02.618 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_evict_age": 0, 2021-02-10T11:47:02.618 INFO:teuthology.orchestra.run.gibba012.stdout: "erasure_code_profile": "", 2021-02-10T11:47:02.618 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_params": { 2021-02-10T11:47:02.618 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "none" 2021-02-10T11:47:02.619 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:02.619 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_period": 0, 2021-02-10T11:47:02.619 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_count": 0, 2021-02-10T11:47:02.619 INFO:teuthology.orchestra.run.gibba012.stdout: "use_gmt_hitset": true, 2021-02-10T11:47:02.619 INFO:teuthology.orchestra.run.gibba012.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:47:02.620 INFO:teuthology.orchestra.run.gibba012.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:47:02.620 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:47:02.620 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:47:02.620 INFO:teuthology.orchestra.run.gibba012.stdout: "grade_table": [], 2021-02-10T11:47:02.620 INFO:teuthology.orchestra.run.gibba012.stdout: "stripe_width": 0, 2021-02-10T11:47:02.620 INFO:teuthology.orchestra.run.gibba012.stdout: "expected_num_objects": 0, 2021-02-10T11:47:02.621 INFO:teuthology.orchestra.run.gibba012.stdout: "fast_read": false, 2021-02-10T11:47:02.621 INFO:teuthology.orchestra.run.gibba012.stdout: "options": {}, 2021-02-10T11:47:02.621 INFO:teuthology.orchestra.run.gibba012.stdout: "application_metadata": { 2021-02-10T11:47:02.621 INFO:teuthology.orchestra.run.gibba012.stdout: "cephfs": { 2021-02-10T11:47:02.621 INFO:teuthology.orchestra.run.gibba012.stdout: "data": "cephfs" 2021-02-10T11:47:02.621 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:47:02.622 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:47:02.622 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:47:02.622 INFO:teuthology.orchestra.run.gibba012.stdout: ], 2021-02-10T11:47:02.622 INFO:teuthology.orchestra.run.gibba012.stdout: "osds": [ 2021-02-10T11:47:02.622 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:02.623 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 0, 2021-02-10T11:47:02.623 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "95444b8f-6370-4a1d-ae5f-df62651b77da", 2021-02-10T11:47:02.623 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:47:02.623 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:47:02.623 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:47:02.623 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:47:02.624 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:47:02.624 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:47:02.624 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:47:02.624 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 170, 2021-02-10T11:47:02.624 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:47:02.625 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:47:02.625 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:47:02.625 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:47:02.625 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:02.625 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:47:02.625 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6817", 2021-02-10T11:47:02.626 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:47:02.626 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:02.626 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:02.626 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:47:02.626 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6818", 2021-02-10T11:47:02.627 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:47:02.627 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:47:02.627 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:47:02.627 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:02.627 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:47:02.627 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:47:02.628 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:02.628 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:47:02.628 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6820", 2021-02-10T11:47:02.628 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:47:02.628 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:02.629 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:02.629 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:47:02.629 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6822", 2021-02-10T11:47:02.629 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:47:02.629 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:47:02.629 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:47:02.630 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:02.630 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:47:02.630 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:47:02.630 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:02.630 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:47:02.631 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6828", 2021-02-10T11:47:02.631 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:47:02.631 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:02.631 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:02.631 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:47:02.631 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6830", 2021-02-10T11:47:02.632 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:47:02.632 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:47:02.632 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:47:02.632 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:02.632 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:47:02.633 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:47:02.633 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:02.633 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:47:02.633 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6824", 2021-02-10T11:47:02.633 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:47:02.633 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:02.634 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:02.634 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:47:02.634 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6826", 2021-02-10T11:47:02.634 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:47:02.634 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:47:02.635 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:47:02.635 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:02.635 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.112:6818/36030", 2021-02-10T11:47:02.635 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.112:6822/36030", 2021-02-10T11:47:02.635 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.112:6830/36030", 2021-02-10T11:47:02.635 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.112:6826/36030", 2021-02-10T11:47:02.636 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:47:02.636 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:47:02.636 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:47:02.636 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:47:02.636 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:02.636 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:02.637 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 1, 2021-02-10T11:47:02.637 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "a23e217e-0d8e-486e-be07-712adfabd262", 2021-02-10T11:47:02.637 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:47:02.637 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:47:02.637 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:47:02.637 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:47:02.638 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:47:02.638 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:47:02.638 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:47:02.638 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 170, 2021-02-10T11:47:02.638 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:47:02.639 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:47:02.639 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:47:02.639 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:47:02.639 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:02.639 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:47:02.639 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6800", 2021-02-10T11:47:02.640 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:47:02.640 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:02.640 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:02.640 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:47:02.640 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6802", 2021-02-10T11:47:02.640 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:47:02.641 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:47:02.641 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:47:02.641 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:02.641 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:47:02.641 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:47:02.642 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:02.642 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:47:02.642 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6804", 2021-02-10T11:47:02.642 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:47:02.642 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:02.643 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:02.643 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:47:02.643 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6806", 2021-02-10T11:47:02.643 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:47:02.643 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:47:02.643 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:47:02.644 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:02.644 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:47:02.644 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:47:02.644 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:02.644 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:47:02.644 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6812", 2021-02-10T11:47:02.645 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:47:02.645 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:02.645 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:02.645 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:47:02.645 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6814", 2021-02-10T11:47:02.645 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:47:02.645 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:47:02.645 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:47:02.645 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:02.645 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:47:02.645 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:47:02.646 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:02.646 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:47:02.646 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6808", 2021-02-10T11:47:02.646 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:47:02.646 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:02.646 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:02.646 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:47:02.646 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6810", 2021-02-10T11:47:02.646 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:47:02.646 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:47:02.646 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:47:02.646 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:02.646 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.112:6802/36029", 2021-02-10T11:47:02.647 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.112:6806/36029", 2021-02-10T11:47:02.647 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.112:6814/36029", 2021-02-10T11:47:02.647 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.112:6810/36029", 2021-02-10T11:47:02.647 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:47:02.647 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:47:02.647 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:47:02.647 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:47:02.647 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:02.647 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:02.647 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 2, 2021-02-10T11:47:02.647 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "026afa8a-c0f1-4b35-9a76-53c00fe0c2fc", 2021-02-10T11:47:02.647 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:47:02.647 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:47:02.647 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:47:02.648 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:47:02.648 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:47:02.648 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:47:02.648 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:47:02.648 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 170, 2021-02-10T11:47:02.648 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:47:02.648 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:47:02.648 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:47:02.648 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:47:02.648 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:02.648 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:47:02.648 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6801", 2021-02-10T11:47:02.649 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:47:02.649 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:02.649 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:02.649 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:47:02.649 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6803", 2021-02-10T11:47:02.649 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:47:02.649 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:47:02.649 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:47:02.649 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:02.649 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:47:02.649 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:47:02.649 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:02.649 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:47:02.650 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6805", 2021-02-10T11:47:02.650 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:47:02.650 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:02.650 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:02.650 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:47:02.651 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6807", 2021-02-10T11:47:02.651 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:47:02.651 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:47:02.651 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:47:02.651 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:02.651 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:47:02.651 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:47:02.651 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:02.651 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:47:02.652 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6813", 2021-02-10T11:47:02.652 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:47:02.652 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:02.652 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:02.652 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:47:02.652 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6815", 2021-02-10T11:47:02.652 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:47:02.653 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:47:02.653 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:47:02.653 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:02.653 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:47:02.653 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:47:02.653 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:02.653 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:47:02.653 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6809", 2021-02-10T11:47:02.653 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:47:02.653 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:02.653 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:02.653 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:47:02.653 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6811", 2021-02-10T11:47:02.654 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:47:02.654 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:47:02.654 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:47:02.654 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:02.654 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.112:6803/36031", 2021-02-10T11:47:02.654 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.112:6807/36031", 2021-02-10T11:47:02.654 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.112:6815/36031", 2021-02-10T11:47:02.654 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.112:6811/36031", 2021-02-10T11:47:02.654 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:47:02.654 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:47:02.654 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:47:02.654 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:47:02.654 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:02.655 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:02.655 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 3, 2021-02-10T11:47:02.655 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "58795682-e00d-4173-bb84-f7075ca89a11", 2021-02-10T11:47:02.655 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:47:02.655 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:47:02.655 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:47:02.655 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:47:02.655 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:47:02.655 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:47:02.655 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:47:02.655 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 170, 2021-02-10T11:47:02.655 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:47:02.655 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:47:02.656 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:47:02.656 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:47:02.656 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:02.656 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:47:02.656 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6816", 2021-02-10T11:47:02.656 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:47:02.656 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:02.656 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:02.656 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:47:02.656 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6819", 2021-02-10T11:47:02.657 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:47:02.657 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:47:02.657 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:47:02.657 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:02.657 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:47:02.657 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:47:02.657 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:02.657 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:47:02.657 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6821", 2021-02-10T11:47:02.657 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:47:02.657 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:02.657 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:02.658 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:47:02.658 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6823", 2021-02-10T11:47:02.658 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:47:02.658 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:47:02.658 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:47:02.658 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:02.658 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:47:02.658 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:47:02.658 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:02.659 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:47:02.659 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6829", 2021-02-10T11:47:02.659 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:47:02.659 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:02.659 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:02.659 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:47:02.659 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6831", 2021-02-10T11:47:02.659 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:47:02.659 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:47:02.659 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:47:02.659 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:02.659 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:47:02.660 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:47:02.660 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:02.660 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:47:02.660 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6825", 2021-02-10T11:47:02.660 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:47:02.660 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:02.660 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:02.660 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:47:02.660 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6827", 2021-02-10T11:47:02.660 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:47:02.660 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:47:02.660 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:47:02.661 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:02.661 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.112:6819/36028", 2021-02-10T11:47:02.661 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.112:6823/36028", 2021-02-10T11:47:02.661 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.112:6831/36028", 2021-02-10T11:47:02.661 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.112:6827/36028", 2021-02-10T11:47:02.661 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:47:02.661 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:47:02.661 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:47:02.661 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:47:02.661 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:02.661 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:02.661 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 4, 2021-02-10T11:47:02.661 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "6763f8d6-53dc-4e45-bdf6-a6eabcf81e39", 2021-02-10T11:47:02.662 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:47:02.662 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:47:02.662 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:47:02.662 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:47:02.662 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:47:02.662 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:47:02.663 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:47:02.663 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 170, 2021-02-10T11:47:02.663 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:47:02.663 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:47:02.663 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:47:02.663 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:47:02.663 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:02.663 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:47:02.663 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6810", 2021-02-10T11:47:02.663 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:47:02.664 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:02.664 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:02.664 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:47:02.664 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6813", 2021-02-10T11:47:02.664 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:47:02.664 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:47:02.664 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:47:02.664 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:02.664 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:47:02.664 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:47:02.664 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:02.664 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:47:02.664 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6816", 2021-02-10T11:47:02.665 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:47:02.665 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:02.665 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:02.665 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:47:02.665 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6819", 2021-02-10T11:47:02.665 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:47:02.665 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:47:02.665 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:47:02.665 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:02.665 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:47:02.665 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:47:02.665 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:02.665 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:47:02.665 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6825", 2021-02-10T11:47:02.666 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:47:02.666 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:02.666 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:02.666 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:47:02.666 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6827", 2021-02-10T11:47:02.666 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:47:02.666 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:47:02.666 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:47:02.666 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:02.666 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:47:02.666 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:47:02.666 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:02.667 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:47:02.667 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6821", 2021-02-10T11:47:02.667 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:47:02.667 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:02.667 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:02.667 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:47:02.667 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6823", 2021-02-10T11:47:02.667 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:47:02.667 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:47:02.667 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:47:02.667 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:02.667 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.120:6813/36388", 2021-02-10T11:47:02.668 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.120:6819/36388", 2021-02-10T11:47:02.668 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.120:6827/36388", 2021-02-10T11:47:02.668 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.120:6823/36388", 2021-02-10T11:47:02.668 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:47:02.668 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:47:02.668 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:47:02.668 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:47:02.668 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:02.668 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:02.668 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 5, 2021-02-10T11:47:02.668 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "5d0e4dd6-0cb0-4bf8-8314-626acaa1b544", 2021-02-10T11:47:02.668 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:47:02.669 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:47:02.669 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:47:02.669 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:47:02.669 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:47:02.669 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:47:02.669 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:47:02.669 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 170, 2021-02-10T11:47:02.669 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:47:02.669 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:47:02.669 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:47:02.669 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:47:02.669 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:02.670 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:47:02.670 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6804", 2021-02-10T11:47:02.670 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:47:02.670 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:02.670 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:02.670 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:47:02.670 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6806", 2021-02-10T11:47:02.670 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:47:02.670 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:47:02.670 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:47:02.670 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:02.671 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:47:02.671 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:47:02.671 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:02.671 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:47:02.671 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6809", 2021-02-10T11:47:02.671 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:47:02.671 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:02.671 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:02.671 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:47:02.671 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6812", 2021-02-10T11:47:02.671 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:47:02.671 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:47:02.671 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:47:02.672 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:02.672 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:47:02.672 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:47:02.672 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:02.672 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:47:02.672 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6820", 2021-02-10T11:47:02.672 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:47:02.672 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:02.672 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:02.672 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:47:02.672 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6822", 2021-02-10T11:47:02.672 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:47:02.673 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:47:02.673 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:47:02.673 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:02.673 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:47:02.673 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:47:02.673 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:02.673 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:47:02.673 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6815", 2021-02-10T11:47:02.673 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:47:02.673 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:02.673 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:02.674 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:47:02.674 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6818", 2021-02-10T11:47:02.674 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:47:02.674 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:47:02.674 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:47:02.674 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:02.674 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.120:6806/36387", 2021-02-10T11:47:02.674 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.120:6812/36387", 2021-02-10T11:47:02.674 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.120:6822/36387", 2021-02-10T11:47:02.675 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.120:6818/36387", 2021-02-10T11:47:02.675 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:47:02.675 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:47:02.675 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:47:02.675 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:47:02.675 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:02.675 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:02.675 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 6, 2021-02-10T11:47:02.675 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "b010e187-4541-4b00-80dd-f948f6ab8f15", 2021-02-10T11:47:02.675 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:47:02.675 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:47:02.675 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:47:02.676 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:47:02.676 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:47:02.676 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:47:02.676 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:47:02.676 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 170, 2021-02-10T11:47:02.676 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:47:02.676 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:47:02.676 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:47:02.676 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:47:02.676 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:02.676 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:47:02.677 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6824", 2021-02-10T11:47:02.677 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:47:02.677 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:02.677 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:02.677 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:47:02.677 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6826", 2021-02-10T11:47:02.677 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:47:02.677 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:47:02.677 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:47:02.677 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:02.677 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:47:02.677 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:47:02.677 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:02.678 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:47:02.678 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6828", 2021-02-10T11:47:02.678 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:47:02.678 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:02.678 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:02.678 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:47:02.678 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6829", 2021-02-10T11:47:02.678 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:47:02.678 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:47:02.678 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:47:02.678 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:02.678 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:47:02.679 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:47:02.679 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:02.679 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:47:02.679 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6832", 2021-02-10T11:47:02.679 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:47:02.679 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:02.679 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:02.679 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:47:02.679 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6833", 2021-02-10T11:47:02.679 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:47:02.679 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:47:02.679 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:47:02.679 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:02.680 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:47:02.680 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:47:02.680 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:02.680 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:47:02.680 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6830", 2021-02-10T11:47:02.680 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:47:02.680 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:02.680 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:02.680 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:47:02.680 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6831", 2021-02-10T11:47:02.680 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:47:02.680 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:47:02.680 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:47:02.681 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:02.681 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.120:6826/36385", 2021-02-10T11:47:02.681 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.120:6829/36385", 2021-02-10T11:47:02.681 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.120:6833/36385", 2021-02-10T11:47:02.681 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.120:6831/36385", 2021-02-10T11:47:02.681 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:47:02.681 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:47:02.681 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:47:02.681 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:47:02.681 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:02.681 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:02.681 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 7, 2021-02-10T11:47:02.681 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "671e2772-33b7-497e-8e44-2178bd850bcf", 2021-02-10T11:47:02.682 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:47:02.682 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:47:02.682 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:47:02.682 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:47:02.682 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:47:02.682 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:47:02.682 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:47:02.682 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 168, 2021-02-10T11:47:02.682 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:47:02.682 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:47:02.682 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:47:02.682 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:47:02.682 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:02.683 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:47:02.683 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6802", 2021-02-10T11:47:02.683 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:47:02.683 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:02.683 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:02.683 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:47:02.683 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6803", 2021-02-10T11:47:02.683 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:47:02.683 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:47:02.683 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:47:02.683 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:02.683 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:47:02.683 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:47:02.684 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:02.684 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:47:02.684 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6805", 2021-02-10T11:47:02.684 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:47:02.684 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:02.684 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:02.684 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:47:02.684 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6807", 2021-02-10T11:47:02.684 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:47:02.684 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:47:02.685 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:47:02.685 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:02.685 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:47:02.685 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:47:02.685 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:02.685 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:47:02.685 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6814", 2021-02-10T11:47:02.685 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:47:02.685 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:02.685 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:02.685 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:47:02.686 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6817", 2021-02-10T11:47:02.686 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:47:02.686 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:47:02.686 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:47:02.686 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:02.686 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:47:02.686 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:47:02.686 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:02.686 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:47:02.687 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6808", 2021-02-10T11:47:02.687 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:47:02.687 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:02.687 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:02.687 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:47:02.687 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6811", 2021-02-10T11:47:02.687 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:47:02.687 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:47:02.687 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:47:02.687 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:02.687 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.120:6803/36386", 2021-02-10T11:47:02.687 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.120:6807/36386", 2021-02-10T11:47:02.687 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.120:6817/36386", 2021-02-10T11:47:02.688 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.120:6811/36386", 2021-02-10T11:47:02.688 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:47:02.688 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:47:02.688 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:47:02.688 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:47:02.688 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:47:02.688 INFO:teuthology.orchestra.run.gibba012.stdout: ], 2021-02-10T11:47:02.688 INFO:teuthology.orchestra.run.gibba012.stdout: "osd_xinfo": [ 2021-02-10T11:47:02.688 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:02.688 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 0, 2021-02-10T11:47:02.689 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:47:02.689 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:47:02.689 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:47:02.689 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:47:02.689 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:47:02.689 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:02.689 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:02.689 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 1, 2021-02-10T11:47:02.689 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:47:02.689 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:47:02.690 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:47:02.690 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:47:02.690 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:47:02.690 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:02.690 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:02.690 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 2, 2021-02-10T11:47:02.690 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:47:02.690 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:47:02.691 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:47:02.691 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:47:02.691 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:47:02.691 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:02.691 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:02.691 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 3, 2021-02-10T11:47:02.691 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:47:02.691 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:47:02.691 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:47:02.691 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:47:02.691 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:47:02.692 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:02.692 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:02.692 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 4, 2021-02-10T11:47:02.692 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:47:02.692 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:47:02.692 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:47:02.692 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:47:02.692 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:47:02.692 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:02.692 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:02.692 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 5, 2021-02-10T11:47:02.692 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:47:02.692 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:47:02.693 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:47:02.693 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:47:02.693 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:47:02.693 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:02.693 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:02.693 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 6, 2021-02-10T11:47:02.693 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:47:02.693 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:47:02.693 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:47:02.693 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:47:02.693 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:47:02.693 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:02.693 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:02.694 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 7, 2021-02-10T11:47:02.694 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:47:02.694 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:47:02.694 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:47:02.694 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:47:02.694 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:47:02.694 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:47:02.694 INFO:teuthology.orchestra.run.gibba012.stdout: ], 2021-02-10T11:47:02.694 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_upmap": [], 2021-02-10T11:47:02.694 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_upmap_items": [], 2021-02-10T11:47:02.694 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_temp": [], 2021-02-10T11:47:02.695 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_temp": [], 2021-02-10T11:47:02.695 INFO:teuthology.orchestra.run.gibba012.stdout: "blacklist": {}, 2021-02-10T11:47:02.695 INFO:teuthology.orchestra.run.gibba012.stdout: "erasure_code_profiles": { 2021-02-10T11:47:02.695 INFO:teuthology.orchestra.run.gibba012.stdout: "default": { 2021-02-10T11:47:02.695 INFO:teuthology.orchestra.run.gibba012.stdout: "crush-failure-domain": "osd", 2021-02-10T11:47:02.695 INFO:teuthology.orchestra.run.gibba012.stdout: "k": "2", 2021-02-10T11:47:02.695 INFO:teuthology.orchestra.run.gibba012.stdout: "m": "1", 2021-02-10T11:47:02.695 INFO:teuthology.orchestra.run.gibba012.stdout: "plugin": "jerasure", 2021-02-10T11:47:02.696 INFO:teuthology.orchestra.run.gibba012.stdout: "ruleset-failure-domain": "osd", 2021-02-10T11:47:02.696 INFO:teuthology.orchestra.run.gibba012.stdout: "technique": "reed_sol_van" 2021-02-10T11:47:02.696 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:47:02.696 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:02.696 INFO:teuthology.orchestra.run.gibba012.stdout: "removed_snaps_queue": [], 2021-02-10T11:47:02.696 INFO:teuthology.orchestra.run.gibba012.stdout: "new_removed_snaps": [], 2021-02-10T11:47:02.696 INFO:teuthology.orchestra.run.gibba012.stdout: "new_purged_snaps": [], 2021-02-10T11:47:02.696 INFO:teuthology.orchestra.run.gibba012.stdout: "crush_node_flags": {}, 2021-02-10T11:47:02.696 INFO:teuthology.orchestra.run.gibba012.stdout: "device_class_flags": {} 2021-02-10T11:47:02.696 INFO:teuthology.orchestra.run.gibba012.stdout:} 2021-02-10T11:47:02.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:02.584 7f8753d32700 1 -- 172.21.2.112:0/2121954913 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8734018730 msgr2=0x7f873401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:02.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:02.584 7f8753d32700 1 --2- 172.21.2.112:0/2121954913 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8734018730 0x7f873401ac10 crc :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:02.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:02.584 7f8753d32700 1 -- 172.21.2.112:0/2121954913 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f874c08eda0 msgr2=0x7f874c0a9af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:02.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:02.584 7f8753d32700 1 --2- 172.21.2.112:0/2121954913 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f874c08eda0 0x7f874c0a9af0 secure :-1 s=READY pgs=2961 cs=0 l=1 rev1=1 rx=0x7f8744001c10 tx=0x7f874400ac60).stop 2021-02-10T11:47:02.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:02.584 7f8753d32700 1 -- 172.21.2.112:0/2121954913 shutdown_connections 2021-02-10T11:47:02.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:02.584 7f8753d32700 1 --2- 172.21.2.112:0/2121954913 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8734018730 0x7f873401ac10 unknown :-1 s=CLOSED pgs=1339 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:02.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:02.584 7f8753d32700 1 --2- 172.21.2.112:0/2121954913 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f874c0ed6f0 0x7f874c08e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:02.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:02.584 7f8753d32700 1 --2- 172.21.2.112:0/2121954913 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f874c08eda0 0x7f874c0a9af0 unknown :-1 s=CLOSED pgs=2961 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:02.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:02.584 7f8753d32700 1 --2- 172.21.2.112:0/2121954913 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f874c0ed0f0 0x7f874c19ff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:02.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:02.584 7f8753d32700 1 -- 172.21.2.112:0/2121954913 >> 172.21.2.112:0/2121954913 conn(0x7f874c0b7e70 msgr2=0x7f874c0c3850 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:02.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:02.584 7f8753d32700 1 -- 172.21.2.112:0/2121954913 shutdown_connections 2021-02-10T11:47:02.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:02.584 7f8753d32700 1 -- 172.21.2.112:0/2121954913 wait complete. 2021-02-10T11:47:02.699 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:47:03.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.109 7ff2538bf700 1 Processor -- start 2021-02-10T11:47:03.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.109 7ff2538bf700 1 -- start start 2021-02-10T11:47:03.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.109 7ff2538bf700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7ff24c0d0340 0x7ff24c0ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:03.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.109 7ff2538bf700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7ff24c0d1c70 0x7ff24c0efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:03.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.109 7ff2538bf700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff24c0f0b10 con 0x7ff24c0d1870 2021-02-10T11:47:03.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.110 7ff2538bf700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff24c0e5ed0 con 0x7ff24c0d0340 2021-02-10T11:47:03.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.110 7ff2538bf700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff24c1b0be0 con 0x7ff24c0d1c70 2021-02-10T11:47:03.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.110 7ff250eda700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7ff24c0d0340 0x7ff24c0ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:03.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.110 7ff2516db700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7ff24c0d1870 0x7ff24c1b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:59462/0 (socket says 172.21.2.112:59462) 2021-02-10T11:47:03.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.110 7ff2516db700 1 -- 172.21.2.112:0/2509866122 learned_addr learned my addr 172.21.2.112:0/2509866122 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:47:03.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.110 7ff251edc700 1 --2- 172.21.2.112:0/2509866122 >> v2:172.21.2.120:3300/0 conn(0x7ff24c0d1c70 0x7ff24c0efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:03.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.110 7ff250eda700 1 -- 172.21.2.112:0/2509866122 >> v2:172.21.2.120:3300/0 conn(0x7ff24c0d1c70 msgr2=0x7ff24c0efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:03.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.110 7ff250eda700 1 --2- 172.21.2.112:0/2509866122 >> v2:172.21.2.120:3300/0 conn(0x7ff24c0d1c70 0x7ff24c0efc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:03.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.110 7ff250eda700 1 -- 172.21.2.112:0/2509866122 >> v1:172.21.2.112:6789/0 conn(0x7ff24c0d1870 legacy=0x7ff24c1b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:03.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.110 7ff250eda700 1 -- 172.21.2.112:0/2509866122 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff24c0f20a0 con 0x7ff24c0d0340 2021-02-10T11:47:03.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.110 7ff250eda700 1 --2- 172.21.2.112:0/2509866122 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff24c0d0340 0x7ff24c0ef640 secure :-1 s=READY pgs=2963 cs=0 l=1 rev1=1 rx=0x7ff248006650 tx=0x7ff248009a00).ready entity=mon.0 client_cookie=27706eb46f27096e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:03.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.111 7ff23ffff700 1 -- 172.21.2.112:0/2509866122 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff2480009b0 con 0x7ff24c0d0340 2021-02-10T11:47:03.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.111 7ff23ffff700 1 -- 172.21.2.112:0/2509866122 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff24800c0c0 con 0x7ff24c0d0340 2021-02-10T11:47:03.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.111 7ff23ffff700 1 -- 172.21.2.112:0/2509866122 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff24800c510 con 0x7ff24c0d0340 2021-02-10T11:47:03.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.111 7ff2538bf700 1 -- 172.21.2.112:0/2509866122 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff24c0d0340 msgr2=0x7ff24c0ef640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:03.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.111 7ff2538bf700 1 --2- 172.21.2.112:0/2509866122 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff24c0d0340 0x7ff24c0ef640 secure :-1 s=READY pgs=2963 cs=0 l=1 rev1=1 rx=0x7ff248006650 tx=0x7ff248009a00).stop 2021-02-10T11:47:03.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.111 7ff2538bf700 1 -- 172.21.2.112:0/2509866122 shutdown_connections 2021-02-10T11:47:03.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.111 7ff2538bf700 1 --2- 172.21.2.112:0/2509866122 >> v2:172.21.2.120:3300/0 conn(0x7ff24c0d1c70 0x7ff24c0efc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:03.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.111 7ff2538bf700 1 --2- 172.21.2.112:0/2509866122 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff24c0d0340 0x7ff24c0ef640 unknown :-1 s=CLOSED pgs=2963 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:03.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.111 7ff2538bf700 1 -- 172.21.2.112:0/2509866122 >> 172.21.2.112:0/2509866122 conn(0x7ff24c0ce450 msgr2=0x7ff24c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:03.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.112 7ff2538bf700 1 -- 172.21.2.112:0/2509866122 shutdown_connections 2021-02-10T11:47:03.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.112 7ff2538bf700 1 -- 172.21.2.112:0/2509866122 wait complete. 2021-02-10T11:47:03.132 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.126 7ff2538bf700 1 Processor -- start 2021-02-10T11:47:03.132 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.126 7ff2538bf700 1 -- start start 2021-02-10T11:47:03.132 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.127 7ff2538bf700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff24c195650 0x7ff24c0b5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:03.132 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.127 7ff2538bf700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff24c0b5b80 0x7ff24c1a2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:03.132 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.127 7ff2538bf700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff24c0acfa0 0x7ff24c1a3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:03.132 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.127 7ff2538bf700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff24c0b6010 con 0x7ff24c0b5b80 2021-02-10T11:47:03.132 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.127 7ff2538bf700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff24c1a3a60 con 0x7ff24c195650 2021-02-10T11:47:03.132 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.127 7ff2538bf700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff24c1a3c40 con 0x7ff24c0acfa0 2021-02-10T11:47:03.132 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.127 7ff250eda700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff24c0b5b80 0x7ff24c1a2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:03.132 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.127 7ff250eda700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff24c0b5b80 0x7ff24c1a2e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:60010/0 (socket says 172.21.2.112:60010) 2021-02-10T11:47:03.132 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.127 7ff250eda700 1 -- 172.21.2.112:0/2364005667 learned_addr learned my addr 172.21.2.112:0/2364005667 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:47:03.133 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.127 7ff250eda700 1 -- 172.21.2.112:0/2364005667 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff24c0acfa0 msgr2=0x7ff24c1a3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:03.133 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.127 7ff250eda700 1 --2- 172.21.2.112:0/2364005667 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff24c0acfa0 0x7ff24c1a3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:03.133 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.127 7ff250eda700 1 -- 172.21.2.112:0/2364005667 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff24c195650 msgr2=0x7ff24c0b5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:03.133 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.127 7ff250eda700 1 --2- 172.21.2.112:0/2364005667 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff24c195650 0x7ff24c0b5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:03.133 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.127 7ff250eda700 1 -- 172.21.2.112:0/2364005667 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff24c0bcae0 con 0x7ff24c0b5b80 2021-02-10T11:47:03.133 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.127 7ff250eda700 1 --2- 172.21.2.112:0/2364005667 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff24c0b5b80 0x7ff24c1a2e20 secure :-1 s=READY pgs=2964 cs=0 l=1 rev1=1 rx=0x7ff24800b590 tx=0x7ff24800b930).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:03.133 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.128 7ff23dffb700 1 -- 172.21.2.112:0/2364005667 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff24800bd20 con 0x7ff24c0b5b80 2021-02-10T11:47:03.133 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.128 7ff23dffb700 1 -- 172.21.2.112:0/2364005667 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff248000c90 con 0x7ff24c0b5b80 2021-02-10T11:47:03.133 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.128 7ff23dffb700 1 -- 172.21.2.112:0/2364005667 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff2480022b0 con 0x7ff24c0b5b80 2021-02-10T11:47:03.133 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.128 7ff2538bf700 1 -- 172.21.2.112:0/2364005667 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff24c0bcae0 con 0x7ff24c0b5b80 2021-02-10T11:47:03.133 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.128 7ff2538bf700 1 -- 172.21.2.112:0/2364005667 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff24c0bd070 con 0x7ff24c0b5b80 2021-02-10T11:47:03.133 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.128 7ff23dffb700 1 -- 172.21.2.112:0/2364005667 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff2480052e0 con 0x7ff24c0b5b80 2021-02-10T11:47:03.133 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.128 7ff23dffb700 1 --2- 172.21.2.112:0/2364005667 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff22c018730 0x7ff22c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:03.133 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.128 7ff23dffb700 1 -- 172.21.2.112:0/2364005667 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 5573+0+0 (secure 0 0 0) 0x7ff2480327a0 con 0x7ff24c0b5b80 2021-02-10T11:47:03.134 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.129 7ff2516db700 1 --2- 172.21.2.112:0/2364005667 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff22c018730 0x7ff22c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:03.134 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.129 7ff2516db700 1 --2- 172.21.2.112:0/2364005667 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff22c018730 0x7ff22c01abc0 crc :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:03.161 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.159 7ff2538bf700 1 -- 172.21.2.112:0/2364005667 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff24c1a3c40 con 0x7ff24c0b5b80 2021-02-10T11:47:03.164 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.161 7ff23dffb700 1 -- 172.21.2.112:0/2364005667 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff248006ae0 con 0x7ff24c0b5b80 2021-02-10T11:47:03.297 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.295 7ff2538bf700 1 -- 172.21.2.112:0/2364005667 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff24c1a3c40 con 0x7ff24c0b5b80 2021-02-10T11:47:03.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.295 7ff23dffb700 1 -- 172.21.2.112:0/2364005667 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 149 v149) v1 ==== 95+0+2111 (secure 0 0 0) 0x7ff2480052e0 con 0x7ff24c0b5b80 2021-02-10T11:47:03.298 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:47:03.298 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":149,"default_fscid":18,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":12642,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/4212630012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":4212630012},{"type":"v1","addr":"172.21.2.112:6833","nonce":4212630012}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":144}],"filesystems":[{"mdsmap":{"epoch":148,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:46:20.421874","modified":"2021-02-10 11:46:22.427977","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12639},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12639":{"gid":12639,"name":"b","rank":0,"incarnation":145,"state":"up:active","state_seq":3,"addr":"172.21.2.120:6835/1654642147","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":1654642147},{"type":"v1","addr":"172.21.2.120:6835","nonce":1654642147}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[37],"metadata_pool":36,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":18}]} 2021-02-10T11:47:03.302 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 149 2021-02-10T11:47:03.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.297 7ff2538bf700 1 -- 172.21.2.112:0/2364005667 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff22c018730 msgr2=0x7ff22c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:03.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.297 7ff2538bf700 1 --2- 172.21.2.112:0/2364005667 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff22c018730 0x7ff22c01abc0 crc :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:03.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.297 7ff2538bf700 1 -- 172.21.2.112:0/2364005667 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff24c0b5b80 msgr2=0x7ff24c1a2e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:03.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.297 7ff2538bf700 1 --2- 172.21.2.112:0/2364005667 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff24c0b5b80 0x7ff24c1a2e20 secure :-1 s=READY pgs=2964 cs=0 l=1 rev1=1 rx=0x7ff24800b590 tx=0x7ff24800b930).stop 2021-02-10T11:47:03.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.298 7ff2538bf700 1 -- 172.21.2.112:0/2364005667 shutdown_connections 2021-02-10T11:47:03.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.298 7ff2538bf700 1 --2- 172.21.2.112:0/2364005667 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff22c018730 0x7ff22c01abc0 unknown :-1 s=CLOSED pgs=1340 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:03.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.298 7ff2538bf700 1 --2- 172.21.2.112:0/2364005667 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff24c0acfa0 0x7ff24c1a3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:03.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.298 7ff2538bf700 1 --2- 172.21.2.112:0/2364005667 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff24c0b5b80 0x7ff24c1a2e20 unknown :-1 s=CLOSED pgs=2964 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:03.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.298 7ff2538bf700 1 --2- 172.21.2.112:0/2364005667 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff24c195650 0x7ff24c0b5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:03.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.298 7ff2538bf700 1 -- 172.21.2.112:0/2364005667 >> 172.21.2.112:0/2364005667 conn(0x7ff24c0b3040 msgr2=0x7ff24c0c3250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:03.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.298 7ff2538bf700 1 -- 172.21.2.112:0/2364005667 shutdown_connections 2021-02-10T11:47:03.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.298 7ff2538bf700 1 -- 172.21.2.112:0/2364005667 wait complete. 2021-02-10T11:47:03.309 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs fail cephfs 2021-02-10T11:47:03.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.719 7fcb17d91700 1 Processor -- start 2021-02-10T11:47:03.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.719 7fcb17d91700 1 -- start start 2021-02-10T11:47:03.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.720 7fcb17d91700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fcb100d0340 0x7fcb100ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:03.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.720 7fcb17d91700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fcb100d1c70 0x7fcb100efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:03.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.720 7fcb17d91700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb100f0b10 con 0x7fcb100d1870 2021-02-10T11:47:03.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.720 7fcb17d91700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb100e5ed0 con 0x7fcb100d1c70 2021-02-10T11:47:03.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.720 7fcb17d91700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb101b0be0 con 0x7fcb100d0340 2021-02-10T11:47:03.740 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.720 7fcb153ac700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fcb100d0340 0x7fcb100ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:03.740 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.720 7fcb163ae700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fcb100d1c70 0x7fcb100efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:03.740 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.720 7fcb163ae700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fcb100d1c70 0x7fcb100efc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:60018/0 (socket says 172.21.2.112:60018) 2021-02-10T11:47:03.740 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.720 7fcb163ae700 1 -- 172.21.2.112:0/377760207 learned_addr learned my addr 172.21.2.112:0/377760207 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:47:03.740 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.720 7fcb153ac700 1 --2- 172.21.2.112:0/377760207 >> v2:172.21.2.120:3300/0 conn(0x7fcb100d0340 0x7fcb100ef640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3300/0 says I am v2:172.21.2.112:34064/0 (socket says 172.21.2.112:34064) 2021-02-10T11:47:03.740 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.720 7fcb163ae700 1 -- 172.21.2.112:0/377760207 >> v2:172.21.2.120:3300/0 conn(0x7fcb100d0340 msgr2=0x7fcb100ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:03.740 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.720 7fcb163ae700 1 --2- 172.21.2.112:0/377760207 >> v2:172.21.2.120:3300/0 conn(0x7fcb100d0340 0x7fcb100ef640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:03.740 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.720 7fcb163ae700 1 -- 172.21.2.112:0/377760207 >> v1:172.21.2.120:6789/0 conn(0x7fcb100d1870 legacy=0x7fcb101b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:03.740 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.720 7fcb163ae700 1 -- 172.21.2.112:0/377760207 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb100f20a0 con 0x7fcb100d1c70 2021-02-10T11:47:03.740 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.720 7fcb153ac700 1 --2- 172.21.2.112:0/377760207 >> v2:172.21.2.120:3300/0 conn(0x7fcb100d0340 0x7fcb100ef640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:47:03.740 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.721 7fcb163ae700 1 --2- 172.21.2.112:0/377760207 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcb100d1c70 0x7fcb100efc60 secure :-1 s=READY pgs=2965 cs=0 l=1 rev1=1 rx=0x7fcb08006f00 tx=0x7fcb08009f10).ready entity=mon.0 client_cookie=1da8bcb5eb7dc2 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:03.741 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.721 7fcb14bab700 1 -- 172.21.2.112:0/377760207 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcb0800b3b0 con 0x7fcb100d1c70 2021-02-10T11:47:03.741 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.721 7fcb14bab700 1 -- 172.21.2.112:0/377760207 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb0800b730 con 0x7fcb100d1c70 2021-02-10T11:47:03.741 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.721 7fcb14bab700 1 -- 172.21.2.112:0/377760207 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcb0800bc50 con 0x7fcb100d1c70 2021-02-10T11:47:03.741 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.721 7fcb17d91700 1 -- 172.21.2.112:0/377760207 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcb100d1c70 msgr2=0x7fcb100efc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:03.741 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.721 7fcb17d91700 1 --2- 172.21.2.112:0/377760207 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcb100d1c70 0x7fcb100efc60 secure :-1 s=READY pgs=2965 cs=0 l=1 rev1=1 rx=0x7fcb08006f00 tx=0x7fcb08009f10).stop 2021-02-10T11:47:03.741 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.722 7fcb17d91700 1 -- 172.21.2.112:0/377760207 shutdown_connections 2021-02-10T11:47:03.741 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.722 7fcb17d91700 1 --2- 172.21.2.112:0/377760207 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcb100d1c70 0x7fcb100efc60 unknown :-1 s=CLOSED pgs=2965 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:03.741 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.722 7fcb17d91700 1 --2- 172.21.2.112:0/377760207 >> v2:172.21.2.120:3300/0 conn(0x7fcb100d0340 0x7fcb100ef640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:03.741 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.722 7fcb17d91700 1 -- 172.21.2.112:0/377760207 >> 172.21.2.112:0/377760207 conn(0x7fcb100ce450 msgr2=0x7fcb100d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:03.741 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.722 7fcb17d91700 1 -- 172.21.2.112:0/377760207 shutdown_connections 2021-02-10T11:47:03.741 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.722 7fcb17d91700 1 -- 172.21.2.112:0/377760207 wait complete. 2021-02-10T11:47:03.743 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.737 7fcb17d91700 1 Processor -- start 2021-02-10T11:47:03.743 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.737 7fcb17d91700 1 -- start start 2021-02-10T11:47:03.743 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.737 7fcb17d91700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcb1018b550 0x7fcb1018b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:03.743 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.737 7fcb17d91700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcb101a3530 0x7fcb101a3930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:03.743 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.737 7fcb17d91700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcb100c1540 0x7fcb100c1940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:03.743 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.737 7fcb17d91700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb100b5560 con 0x7fcb101a3530 2021-02-10T11:47:03.744 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.737 7fcb17d91700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb100c1f60 con 0x7fcb100c1540 2021-02-10T11:47:03.744 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.737 7fcb17d91700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb100c2180 con 0x7fcb1018b550 2021-02-10T11:47:03.744 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.737 7fcb153ac700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcb101a3530 0x7fcb101a3930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:03.744 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.737 7fcb153ac700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcb101a3530 0x7fcb101a3930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:60022/0 (socket says 172.21.2.112:60022) 2021-02-10T11:47:03.744 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.737 7fcb153ac700 1 -- 172.21.2.112:0/2346628519 learned_addr learned my addr 172.21.2.112:0/2346628519 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:47:03.744 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.737 7fcb15bad700 1 --2- 172.21.2.112:0/2346628519 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcb1018b550 0x7fcb1018b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:03.744 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.737 7fcb163ae700 1 --2- 172.21.2.112:0/2346628519 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcb100c1540 0x7fcb100c1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:03.744 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.738 7fcb153ac700 1 -- 172.21.2.112:0/2346628519 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcb1018b550 msgr2=0x7fcb1018b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:03.744 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.738 7fcb153ac700 1 --2- 172.21.2.112:0/2346628519 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcb1018b550 0x7fcb1018b950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:03.744 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.738 7fcb153ac700 1 -- 172.21.2.112:0/2346628519 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcb100c1540 msgr2=0x7fcb100c1940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:03.744 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.738 7fcb153ac700 1 --2- 172.21.2.112:0/2346628519 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcb100c1540 0x7fcb100c1940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:03.744 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.738 7fcb153ac700 1 -- 172.21.2.112:0/2346628519 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb100c23a0 con 0x7fcb101a3530 2021-02-10T11:47:03.744 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.738 7fcb153ac700 1 --2- 172.21.2.112:0/2346628519 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcb101a3530 0x7fcb101a3930 secure :-1 s=READY pgs=2966 cs=0 l=1 rev1=1 rx=0x7fcb0c006f40 tx=0x7fcb0c00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:03.745 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.738 7fcb027fc700 1 -- 172.21.2.112:0/2346628519 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcb0c00bd80 con 0x7fcb101a3530 2021-02-10T11:47:03.745 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.738 7fcb027fc700 1 -- 172.21.2.112:0/2346628519 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb0c000c90 con 0x7fcb101a3530 2021-02-10T11:47:03.745 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.738 7fcb027fc700 1 -- 172.21.2.112:0/2346628519 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcb0c002220 con 0x7fcb101a3530 2021-02-10T11:47:03.745 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.738 7fcb17d91700 1 -- 172.21.2.112:0/2346628519 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb100c23a0 con 0x7fcb101a3530 2021-02-10T11:47:03.745 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.738 7fcb17d91700 1 -- 172.21.2.112:0/2346628519 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb100c23a0 con 0x7fcb101a3530 2021-02-10T11:47:03.745 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.739 7fcb027fc700 1 -- 172.21.2.112:0/2346628519 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fcb0c00a000 con 0x7fcb101a3530 2021-02-10T11:47:03.745 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.739 7fcb027fc700 1 --2- 172.21.2.112:0/2346628519 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcaf80186c0 0x7fcaf801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:03.745 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.739 7fcb027fc700 1 -- 172.21.2.112:0/2346628519 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fcb0c0323e0 con 0x7fcb101a3530 2021-02-10T11:47:03.745 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.739 7fcb15bad700 1 --2- 172.21.2.112:0/2346628519 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcaf80186c0 0x7fcaf801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:03.745 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.740 7fcb15bad700 1 --2- 172.21.2.112:0/2346628519 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcaf80186c0 0x7fcaf801abc0 crc :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:03.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.769 7fcb17d91700 1 -- 172.21.2.112:0/2346628519 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb100ebc90 con 0x7fcb101a3530 2021-02-10T11:47:03.774 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.772 7fcb027fc700 1 -- 172.21.2.112:0/2346628519 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fcb0c006b00 con 0x7fcb101a3530 2021-02-10T11:47:03.904 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:03.901 7fcb17d91700 1 -- 172.21.2.112:0/2346628519 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fcb100f0b10 con 0x7fcb101a3530 2021-02-10T11:47:04.286 INFO:tasks.ceph.mon.a.gibba012.stderr:2021-02-10 11:47:04.284 7fc3fe6e2700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-02-10T11:47:04.289 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:04.287 7fcb027fc700 1 -- 172.21.2.112:0/2346628519 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v150) v1 ==== 161+0+0 (secure 0 0 0) 0x7fcb0c00a000 con 0x7fcb101a3530 2021-02-10T11:47:04.320 INFO:teuthology.orchestra.run.gibba012.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2021-02-10T11:47:04.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:04.319 7fcb17d91700 1 -- 172.21.2.112:0/2346628519 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcaf80186c0 msgr2=0x7fcaf801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:04.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:04.319 7fcb17d91700 1 --2- 172.21.2.112:0/2346628519 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcaf80186c0 0x7fcaf801abc0 crc :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:04.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:04.319 7fcb17d91700 1 -- 172.21.2.112:0/2346628519 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcb101a3530 msgr2=0x7fcb101a3930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:04.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:04.319 7fcb17d91700 1 --2- 172.21.2.112:0/2346628519 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcb101a3530 0x7fcb101a3930 secure :-1 s=READY pgs=2966 cs=0 l=1 rev1=1 rx=0x7fcb0c006f40 tx=0x7fcb0c00b990).stop 2021-02-10T11:47:04.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:04.320 7fcb17d91700 1 -- 172.21.2.112:0/2346628519 shutdown_connections 2021-02-10T11:47:04.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:04.320 7fcb17d91700 1 --2- 172.21.2.112:0/2346628519 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fcaf80186c0 0x7fcaf801abc0 unknown :-1 s=CLOSED pgs=1341 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:04.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:04.320 7fcb17d91700 1 --2- 172.21.2.112:0/2346628519 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fcb100c1540 0x7fcb100c1940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:04.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:04.320 7fcb17d91700 1 --2- 172.21.2.112:0/2346628519 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fcb101a3530 0x7fcb101a3930 unknown :-1 s=CLOSED pgs=2966 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:04.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:04.320 7fcb17d91700 1 --2- 172.21.2.112:0/2346628519 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fcb1018b550 0x7fcb1018b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:04.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:04.320 7fcb17d91700 1 -- 172.21.2.112:0/2346628519 >> 172.21.2.112:0/2346628519 conn(0x7fcb100a6870 msgr2=0x7fcb1008a420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:04.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:04.320 7fcb17d91700 1 -- 172.21.2.112:0/2346628519 shutdown_connections 2021-02-10T11:47:04.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:04.320 7fcb17d91700 1 -- 172.21.2.112:0/2346628519 wait complete. 2021-02-10T11:47:04.331 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:47:04.341 INFO:tasks.ceph.mds.b.gibba020.stdout:starting mds.b at 2021-02-10T11:47:04.763 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:04.742 7f2d90021700 1 Processor -- start 2021-02-10T11:47:04.764 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:04.743 7f2d90021700 1 -- start start 2021-02-10T11:47:04.764 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:04.743 7f2d90021700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2d880d0340 0x7f2d880ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:04.764 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:04.743 7f2d90021700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f2d880d1c70 0x7f2d880efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:04.764 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:04.743 7f2d90021700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d880f0b10 con 0x7f2d880d1870 2021-02-10T11:47:04.764 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:04.743 7f2d90021700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d880e5ed0 con 0x7f2d880d1c70 2021-02-10T11:47:04.764 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:04.743 7f2d90021700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d881b0be0 con 0x7f2d880d0340 2021-02-10T11:47:04.764 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:04.743 7f2d8de3d700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f2d880d1870 0x7f2d881b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:59490/0 (socket says 172.21.2.112:59490) 2021-02-10T11:47:04.764 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:04.743 7f2d8de3d700 1 -- 172.21.2.112:0/1258044951 learned_addr learned my addr 172.21.2.112:0/1258044951 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:47:04.764 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:04.744 7f2d8ce3b700 1 -- 172.21.2.112:0/1258044951 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1766232722 0 0) 0x7f2d84000ce0 con 0x7f2d880d1870 2021-02-10T11:47:04.764 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:04.744 7f2d8ce3b700 1 -- 172.21.2.112:0/1258044951 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d74002fe0 con 0x7f2d880d1870 2021-02-10T11:47:04.764 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:04.744 7f2d8d63c700 1 --2- 172.21.2.112:0/1258044951 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2d880d0340 0x7f2d880ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:04.764 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:04.744 7f2d8e63e700 1 --2- 172.21.2.112:0/1258044951 >> v2:172.21.2.120:3300/0 conn(0x7f2d880d1c70 0x7f2d880efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:04.765 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:04.744 7f2d8ce3b700 1 -- 172.21.2.112:0/1258044951 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3899806360 0 0) 0x7f2d840010e0 con 0x7f2d880d1870 2021-02-10T11:47:04.765 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:04.744 7f2d8ce3b700 1 -- 172.21.2.112:0/1258044951 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2d880d0340 msgr2=0x7f2d880ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:04.765 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:04.744 7f2d8ce3b700 1 --2- 172.21.2.112:0/1258044951 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2d880d0340 0x7f2d880ef640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:04.765 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:04.744 7f2d8ce3b700 1 -- 172.21.2.112:0/1258044951 >> v2:172.21.2.120:3300/0 conn(0x7f2d880d1c70 msgr2=0x7f2d880efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:04.765 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:04.744 7f2d8ce3b700 1 --2- 172.21.2.112:0/1258044951 >> v2:172.21.2.120:3300/0 conn(0x7f2d880d1c70 0x7f2d880efc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:04.765 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:04.744 7f2d8ce3b700 1 -- 172.21.2.112:0/1258044951 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d880f20a0 con 0x7f2d880d1870 2021-02-10T11:47:04.765 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:04.744 7f2d8ce3b700 1 -- 172.21.2.112:0/1258044951 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f2d84001600 con 0x7f2d880d1870 2021-02-10T11:47:04.765 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:04.744 7f2d8ce3b700 1 -- 172.21.2.112:0/1258044951 >> v1:172.21.2.112:6789/0 conn(0x7f2d880d1870 legacy=0x7f2d881b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:04.765 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:04.744 7f2d8ce3b700 1 --2- 172.21.2.112:0/1258044951 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2d74003a60 0x7f2d74005e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:04.765 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:04.744 7f2d8ce3b700 1 --2- 172.21.2.112:0/1258044951 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2d74006730 0x7f2d74008c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:04.765 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:04.744 7f2d8ce3b700 1 --2- 172.21.2.112:0/1258044951 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2d740094f0 0x7f2d7400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:04.765 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:04.744 7f2d8ce3b700 1 -- 172.21.2.112:0/1258044951 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d7400c2c0 con 0x7f2d74006730 2021-02-10T11:47:04.766 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:04.744 7f2d8ce3b700 1 -- 172.21.2.112:0/1258044951 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d740504b0 con 0x7f2d740094f0 2021-02-10T11:47:04.766 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:04.744 7f2d8ce3b700 1 -- 172.21.2.112:0/1258044951 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d740506d0 con 0x7f2d74003a60 2021-02-10T11:47:04.766 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:04.744 7f2d8d63c700 1 --2- 172.21.2.112:0/1258044951 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2d74006730 0x7f2d74008c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:04.766 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:04.745 7f2d8de3d700 1 --2- 172.21.2.112:0/1258044951 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2d74003a60 0x7f2d74005e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:04.766 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:04.745 7f2d8e63e700 1 --2- 172.21.2.112:0/1258044951 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2d740094f0 0x7f2d7400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:04.766 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:04.745 7f2d8d63c700 1 -- 172.21.2.112:0/1258044951 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2d74003a60 msgr2=0x7f2d74005e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:04.766 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:04.745 7f2d8d63c700 1 --2- 172.21.2.112:0/1258044951 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2d74003a60 0x7f2d74005e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:04.766 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:04.745 7f2d8d63c700 1 -- 172.21.2.112:0/1258044951 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2d740094f0 msgr2=0x7f2d7400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:04.766 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:04.745 7f2d8d63c700 1 --2- 172.21.2.112:0/1258044951 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2d740094f0 0x7f2d7400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:04.766 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:04.745 7f2d8d63c700 1 -- 172.21.2.112:0/1258044951 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d74050970 con 0x7f2d74006730 2021-02-10T11:47:04.766 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:04.745 7f2d8de3d700 1 --2- 172.21.2.112:0/1258044951 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2d74003a60 0x7f2d74005e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:47:04.766 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:04.745 7f2d8d63c700 1 --2- 172.21.2.112:0/1258044951 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2d74006730 0x7f2d74008c50 secure :-1 s=READY pgs=2969 cs=0 l=1 rev1=1 rx=0x7f2d7c006ad0 tx=0x7f2d7c009b40).ready entity=mon.0 client_cookie=28684bd08cf24884 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:04.767 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:04.745 7f2d8ce3b700 1 -- 172.21.2.112:0/1258044951 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2d7c0066d0 con 0x7f2d74006730 2021-02-10T11:47:04.767 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:04.745 7f2d8ce3b700 1 -- 172.21.2.112:0/1258044951 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d7c00c0c0 con 0x7f2d74006730 2021-02-10T11:47:04.767 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:04.745 7f2d8ce3b700 1 -- 172.21.2.112:0/1258044951 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2d7c00c5e0 con 0x7f2d74006730 2021-02-10T11:47:04.767 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:04.745 7f2d90021700 1 -- 172.21.2.112:0/1258044951 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2d74006730 msgr2=0x7f2d74008c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:04.767 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:04.745 7f2d90021700 1 --2- 172.21.2.112:0/1258044951 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2d74006730 0x7f2d74008c50 secure :-1 s=READY pgs=2969 cs=0 l=1 rev1=1 rx=0x7f2d7c006ad0 tx=0x7f2d7c009b40).stop 2021-02-10T11:47:04.767 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:04.745 7f2d8e63e700 1 -- 172.21.2.112:0/1258044951 reap_dead start 2021-02-10T11:47:04.767 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:04.746 7f2d90021700 1 -- 172.21.2.112:0/1258044951 shutdown_connections 2021-02-10T11:47:04.767 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:04.746 7f2d90021700 1 -- 172.21.2.112:0/1258044951 >> 172.21.2.112:0/1258044951 conn(0x7f2d880ce450 msgr2=0x7f2d880d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:04.767 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:04.746 7f2d90021700 1 -- 172.21.2.112:0/1258044951 shutdown_connections 2021-02-10T11:47:04.767 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:04.746 7f2d90021700 1 -- 172.21.2.112:0/1258044951 wait complete. 2021-02-10T11:47:04.767 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:04.761 7f2d90021700 1 Processor -- start 2021-02-10T11:47:04.767 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:04.761 7f2d90021700 1 -- start start 2021-02-10T11:47:04.769 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:04.761 7f2d90021700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2d88195650 0x7f2d880b5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:04.769 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:04.761 7f2d90021700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2d880b5b80 0x7f2d881a2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:04.769 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:04.761 7f2d90021700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2d880acfa0 0x7f2d881a3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:04.769 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:04.761 7f2d90021700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d880b6010 con 0x7f2d880b5b80 2021-02-10T11:47:04.769 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:04.761 7f2d90021700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d881a3a60 con 0x7f2d88195650 2021-02-10T11:47:04.769 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:04.761 7f2d90021700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d881a3c40 con 0x7f2d880acfa0 2021-02-10T11:47:04.769 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:04.761 7f2d8d63c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2d880b5b80 0x7f2d881a2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:04.769 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:04.762 7f2d8d63c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2d880b5b80 0x7f2d881a2e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:60042/0 (socket says 172.21.2.112:60042) 2021-02-10T11:47:04.769 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:04.762 7f2d8d63c700 1 -- 172.21.2.112:0/2056677905 learned_addr learned my addr 172.21.2.112:0/2056677905 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:47:04.769 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:04.762 7f2d8d63c700 1 -- 172.21.2.112:0/2056677905 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2d880acfa0 msgr2=0x7f2d881a3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:04.770 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:04.762 7f2d8d63c700 1 --2- 172.21.2.112:0/2056677905 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2d880acfa0 0x7f2d881a3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:04.770 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:04.762 7f2d8d63c700 1 -- 172.21.2.112:0/2056677905 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2d88195650 msgr2=0x7f2d880b5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:04.770 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:04.762 7f2d8d63c700 1 --2- 172.21.2.112:0/2056677905 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2d88195650 0x7f2d880b5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:04.770 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:04.762 7f2d8d63c700 1 -- 172.21.2.112:0/2056677905 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d880bcae0 con 0x7f2d880b5b80 2021-02-10T11:47:04.770 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:04.762 7f2d8d63c700 1 --2- 172.21.2.112:0/2056677905 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2d880b5b80 0x7f2d881a2e20 secure :-1 s=READY pgs=2970 cs=0 l=1 rev1=1 rx=0x7f2d7c00a520 tx=0x7f2d7c00a8c0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:04.770 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:04.762 7f2d7a7fc700 1 -- 172.21.2.112:0/2056677905 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2d7c0009b0 con 0x7f2d880b5b80 2021-02-10T11:47:04.770 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:04.762 7f2d7a7fc700 1 -- 172.21.2.112:0/2056677905 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d7c00ad80 con 0x7f2d880b5b80 2021-02-10T11:47:04.770 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:04.762 7f2d7a7fc700 1 -- 172.21.2.112:0/2056677905 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2d7c00b260 con 0x7f2d880b5b80 2021-02-10T11:47:04.770 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:04.762 7f2d90021700 1 -- 172.21.2.112:0/2056677905 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d880bcae0 con 0x7f2d880b5b80 2021-02-10T11:47:04.770 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:04.762 7f2d90021700 1 -- 172.21.2.112:0/2056677905 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d880bcae0 con 0x7f2d880b5b80 2021-02-10T11:47:04.770 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:04.763 7f2d7a7fc700 1 -- 172.21.2.112:0/2056677905 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2d7c000ea0 con 0x7f2d880b5b80 2021-02-10T11:47:04.770 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:04.763 7f2d7a7fc700 1 --2- 172.21.2.112:0/2056677905 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2d700186c0 0x7f2d7001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:04.770 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:04.763 7f2d7a7fc700 1 -- 172.21.2.112:0/2056677905 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(173..173 src has 1..173) v4 ==== 5659+0+0 (secure 0 0 0) 0x7f2d7c032410 con 0x7f2d880b5b80 2021-02-10T11:47:04.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:04.763 7f2d8de3d700 1 --2- 172.21.2.112:0/2056677905 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2d700186c0 0x7f2d7001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:04.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:04.764 7f2d8de3d700 1 --2- 172.21.2.112:0/2056677905 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2d700186c0 0x7f2d7001abc0 crc :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:04.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:04.793 7f2d90021700 1 -- 172.21.2.112:0/2056677905 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d880b6010 con 0x7f2d880b5b80 2021-02-10T11:47:04.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:04.796 7f2d7a7fc700 1 -- 172.21.2.112:0/2056677905 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2d7c007d60 con 0x7f2d880b5b80 2021-02-10T11:47:04.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:04.928 7f2d90021700 1 -- 172.21.2.112:0/2056677905 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2d880b6010 con 0x7f2d880b5b80 2021-02-10T11:47:04.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:04.928 7f2d7a7fc700 1 -- 172.21.2.112:0/2056677905 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 150 v150) v1 ==== 95+0+1763 (secure 0 0 0) 0x7f2d7c000ea0 con 0x7f2d880b5b80 2021-02-10T11:47:04.931 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:47:04.932 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":150,"default_fscid":18,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":12642,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/4212630012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":4212630012},{"type":"v1","addr":"172.21.2.112:6833","nonce":4212630012}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":144}],"filesystems":[{"mdsmap":{"epoch":150,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:46:20.421874","modified":"2021-02-10 11:47:04.285054","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":173,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[37],"metadata_pool":36,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":18}]} 2021-02-10T11:47:04.932 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 150 2021-02-10T11:47:04.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:04.930 7f2d90021700 1 -- 172.21.2.112:0/2056677905 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2d700186c0 msgr2=0x7f2d7001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:04.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:04.930 7f2d90021700 1 --2- 172.21.2.112:0/2056677905 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2d700186c0 0x7f2d7001abc0 crc :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:04.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:04.930 7f2d90021700 1 -- 172.21.2.112:0/2056677905 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2d880b5b80 msgr2=0x7f2d881a2e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:04.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:04.930 7f2d90021700 1 --2- 172.21.2.112:0/2056677905 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2d880b5b80 0x7f2d881a2e20 secure :-1 s=READY pgs=2970 cs=0 l=1 rev1=1 rx=0x7f2d7c00a520 tx=0x7f2d7c00a8c0).stop 2021-02-10T11:47:04.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:04.931 7f2d90021700 1 -- 172.21.2.112:0/2056677905 shutdown_connections 2021-02-10T11:47:04.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:04.931 7f2d90021700 1 --2- 172.21.2.112:0/2056677905 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2d700186c0 0x7f2d7001abc0 unknown :-1 s=CLOSED pgs=1342 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:04.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:04.931 7f2d90021700 1 --2- 172.21.2.112:0/2056677905 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2d880acfa0 0x7f2d881a3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:04.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:04.931 7f2d90021700 1 --2- 172.21.2.112:0/2056677905 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2d880b5b80 0x7f2d881a2e20 unknown :-1 s=CLOSED pgs=2970 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:04.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:04.931 7f2d90021700 1 --2- 172.21.2.112:0/2056677905 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2d88195650 0x7f2d880b5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:04.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:04.931 7f2d90021700 1 -- 172.21.2.112:0/2056677905 >> 172.21.2.112:0/2056677905 conn(0x7f2d880b3040 msgr2=0x7f2d880c3250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:04.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:04.931 7f2d90021700 1 -- 172.21.2.112:0/2056677905 shutdown_connections 2021-02-10T11:47:04.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:04.931 7f2d90021700 1 -- 172.21.2.112:0/2056677905 wait complete. 2021-02-10T11:47:04.942 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2021-02-10T11:47:05.375 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:05.356 7f84adb50700 1 Processor -- start 2021-02-10T11:47:05.376 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:05.356 7f84adb50700 1 -- start start 2021-02-10T11:47:05.376 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:05.357 7f84adb50700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f84a80d1870 0x7f84a81b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:05.376 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:05.357 7f84adb50700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f84a81b0a10 0x7f84a80cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:05.376 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:05.357 7f84adb50700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84a80f0b10 con 0x7f84a80d1c70 2021-02-10T11:47:05.376 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:05.357 7f84adb50700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84a80e5ed0 con 0x7f84a80d1870 2021-02-10T11:47:05.376 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:05.357 7f84adb50700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84a80f0650 con 0x7f84a81b0a10 2021-02-10T11:47:05.376 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:05.357 7f84a77fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f84a80d1870 0x7f84a81b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:05.376 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:05.357 7f84a77fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f84a80d1870 0x7f84a81b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:60048/0 (socket says 172.21.2.112:60048) 2021-02-10T11:47:05.376 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:05.357 7f84a7fff700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f84a80d1c70 0x7f84a80cf950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:59510/0 (socket says 172.21.2.112:59510) 2021-02-10T11:47:05.376 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:05.357 7f84a7fff700 1 -- 172.21.2.112:0/1482598327 learned_addr learned my addr 172.21.2.112:0/1482598327 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:47:05.376 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:05.357 7f84a6ffd700 1 --2- 172.21.2.112:0/1482598327 >> v2:172.21.2.120:3300/0 conn(0x7f84a81b0a10 0x7f84a80cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:05.376 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:05.358 7f84a77fe700 1 -- 172.21.2.112:0/1482598327 >> v2:172.21.2.120:3300/0 conn(0x7f84a81b0a10 msgr2=0x7f84a80cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:05.376 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:05.358 7f84a77fe700 1 --2- 172.21.2.112:0/1482598327 >> v2:172.21.2.120:3300/0 conn(0x7f84a81b0a10 0x7f84a80cf330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:05.377 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:05.358 7f84a77fe700 1 -- 172.21.2.112:0/1482598327 >> v1:172.21.2.112:6789/0 conn(0x7f84a80d1c70 legacy=0x7f84a80cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:05.377 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:05.358 7f84a77fe700 1 -- 172.21.2.112:0/1482598327 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84a80f0840 con 0x7f84a80d1870 2021-02-10T11:47:05.377 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:05.358 7f84a6ffd700 1 --2- 172.21.2.112:0/1482598327 >> v2:172.21.2.120:3300/0 conn(0x7f84a81b0a10 0x7f84a80cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:47:05.377 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:05.358 7f84a77fe700 1 --2- 172.21.2.112:0/1482598327 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f84a80d1870 0x7f84a81b03f0 secure :-1 s=READY pgs=2972 cs=0 l=1 rev1=1 rx=0x7f8490006650 tx=0x7f8490009a80).ready entity=mon.0 client_cookie=3f1c77b9cf7f8dcd server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:05.377 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:05.358 7f84a67fc700 1 -- 172.21.2.112:0/1482598327 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f849000c0c0 con 0x7f84a80d1870 2021-02-10T11:47:05.377 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:05.358 7f84a67fc700 1 -- 172.21.2.112:0/1482598327 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f849000c340 con 0x7f84a80d1870 2021-02-10T11:47:05.377 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:05.358 7f84a67fc700 1 -- 172.21.2.112:0/1482598327 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f849000c790 con 0x7f84a80d1870 2021-02-10T11:47:05.377 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:05.358 7f84adb50700 1 -- 172.21.2.112:0/1482598327 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f84a80d1870 msgr2=0x7f84a81b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:05.377 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:05.358 7f84adb50700 1 --2- 172.21.2.112:0/1482598327 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f84a80d1870 0x7f84a81b03f0 secure :-1 s=READY pgs=2972 cs=0 l=1 rev1=1 rx=0x7f8490006650 tx=0x7f8490009a80).stop 2021-02-10T11:47:05.377 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:05.359 7f84adb50700 1 -- 172.21.2.112:0/1482598327 shutdown_connections 2021-02-10T11:47:05.377 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:05.359 7f84adb50700 1 --2- 172.21.2.112:0/1482598327 >> v2:172.21.2.120:3300/0 conn(0x7f84a81b0a10 0x7f84a80cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:05.377 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:05.359 7f84adb50700 1 --2- 172.21.2.112:0/1482598327 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f84a80d1870 0x7f84a81b03f0 unknown :-1 s=CLOSED pgs=2972 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:05.378 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:05.359 7f84adb50700 1 -- 172.21.2.112:0/1482598327 >> 172.21.2.112:0/1482598327 conn(0x7f84a80ce450 msgr2=0x7f84a80d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:05.378 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:05.359 7f84adb50700 1 -- 172.21.2.112:0/1482598327 shutdown_connections 2021-02-10T11:47:05.378 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:05.359 7f84adb50700 1 -- 172.21.2.112:0/1482598327 wait complete. 2021-02-10T11:47:05.380 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:05.373 7f84adb50700 1 Processor -- start 2021-02-10T11:47:05.380 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:05.373 7f84adb50700 1 -- start start 2021-02-10T11:47:05.380 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:05.374 7f84adb50700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f84a8094310 0x7f84a80ebe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:05.380 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:05.374 7f84adb50700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f84a8094710 0x7f84a80bca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:05.380 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:05.374 7f84adb50700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f84a80ec490 0x7f84a80bd050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:05.380 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:05.374 7f84adb50700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84a8093d90 con 0x7f84a8094710 2021-02-10T11:47:05.380 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:05.374 7f84adb50700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84a80bd670 con 0x7f84a8094310 2021-02-10T11:47:05.380 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:05.374 7f84adb50700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84a80939f0 con 0x7f84a80ec490 2021-02-10T11:47:05.380 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:05.374 7f84a6ffd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f84a8094710 0x7f84a80bca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:05.380 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:05.374 7f84a6ffd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f84a8094710 0x7f84a80bca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:60052/0 (socket says 172.21.2.112:60052) 2021-02-10T11:47:05.380 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:05.374 7f84a6ffd700 1 -- 172.21.2.112:0/1628231404 learned_addr learned my addr 172.21.2.112:0/1628231404 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:47:05.380 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:05.374 7f84a77fe700 1 --2- 172.21.2.112:0/1628231404 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f84a8094310 0x7f84a80ebe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:05.381 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:05.374 7f84a7fff700 1 --2- 172.21.2.112:0/1628231404 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f84a80ec490 0x7f84a80bd050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:05.381 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:05.374 7f84a6ffd700 1 -- 172.21.2.112:0/1628231404 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f84a80ec490 msgr2=0x7f84a80bd050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:05.381 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:05.374 7f84a6ffd700 1 --2- 172.21.2.112:0/1628231404 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f84a80ec490 0x7f84a80bd050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:05.381 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:05.374 7f84a6ffd700 1 -- 172.21.2.112:0/1628231404 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f84a8094310 msgr2=0x7f84a80ebe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:05.381 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:05.374 7f84a6ffd700 1 --2- 172.21.2.112:0/1628231404 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f84a8094310 0x7f84a80ebe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:05.381 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:05.374 7f84a6ffd700 1 -- 172.21.2.112:0/1628231404 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84a80bda80 con 0x7f84a8094710 2021-02-10T11:47:05.381 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:05.374 7f84a7fff700 1 --2- 172.21.2.112:0/1628231404 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f84a80ec490 0x7f84a80bd050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:47:05.381 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:05.374 7f84a77fe700 1 --2- 172.21.2.112:0/1628231404 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f84a8094310 0x7f84a80ebe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:47:05.381 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:05.375 7f84a6ffd700 1 --2- 172.21.2.112:0/1628231404 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f84a8094710 0x7f84a80bca30 secure :-1 s=READY pgs=2973 cs=0 l=1 rev1=1 rx=0x7f8498006f20 tx=0x7f849800b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:05.381 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:05.375 7f849ffff700 1 -- 172.21.2.112:0/1628231404 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f849800bd80 con 0x7f84a8094710 2021-02-10T11:47:05.381 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:05.375 7f849ffff700 1 -- 172.21.2.112:0/1628231404 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8498000c90 con 0x7f84a8094710 2021-02-10T11:47:05.381 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:05.375 7f84adb50700 1 -- 172.21.2.112:0/1628231404 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84a80bda80 con 0x7f84a8094710 2021-02-10T11:47:05.381 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:05.375 7f84adb50700 1 -- 172.21.2.112:0/1628231404 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84a81a76d0 con 0x7f84a8094710 2021-02-10T11:47:05.382 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:05.375 7f849ffff700 1 -- 172.21.2.112:0/1628231404 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f84980022b0 con 0x7f84a8094710 2021-02-10T11:47:05.382 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:05.375 7f849ffff700 1 -- 172.21.2.112:0/1628231404 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f849800a000 con 0x7f84a8094710 2021-02-10T11:47:05.382 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:05.376 7f849ffff700 1 --2- 172.21.2.112:0/1628231404 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f848c0186c0 0x7f848c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:05.382 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:05.376 7f849ffff700 1 -- 172.21.2.112:0/1628231404 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(173..173 src has 1..173) v4 ==== 5659+0+0 (secure 0 0 0) 0x7f8498032570 con 0x7f84a8094710 2021-02-10T11:47:05.382 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:05.376 7f84a77fe700 1 --2- 172.21.2.112:0/1628231404 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f848c0186c0 0x7f848c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:05.382 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:05.376 7f84a77fe700 1 --2- 172.21.2.112:0/1628231404 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f848c0186c0 0x7f848c01abc0 crc :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:05.409 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:05.407 7f84adb50700 1 -- 172.21.2.112:0/1628231404 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84a80ebc40 con 0x7f84a8094710 2021-02-10T11:47:05.412 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:05.409 7f849ffff700 1 -- 172.21.2.112:0/1628231404 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f849802e1c0 con 0x7f84a8094710 2021-02-10T11:47:05.541 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:05.538 7f84adb50700 1 -- 172.21.2.112:0/1628231404 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f84a80f0b10 con 0x7f84a8094710 2021-02-10T11:47:06.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:06.296 7f849ffff700 1 -- 172.21.2.112:0/1628231404 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v151) v1 ==== 104+0+0 (secure 0 0 0) 0x7f849802e1c0 con 0x7f84a8094710 2021-02-10T11:47:06.311 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:06.309 7f84adb50700 1 -- 172.21.2.112:0/1628231404 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f848c0186c0 msgr2=0x7f848c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:06.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:06.309 7f84adb50700 1 --2- 172.21.2.112:0/1628231404 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f848c0186c0 0x7f848c01abc0 crc :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:06.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:06.309 7f84adb50700 1 -- 172.21.2.112:0/1628231404 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f84a8094710 msgr2=0x7f84a80bca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:06.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:06.309 7f84adb50700 1 --2- 172.21.2.112:0/1628231404 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f84a8094710 0x7f84a80bca30 secure :-1 s=READY pgs=2973 cs=0 l=1 rev1=1 rx=0x7f8498006f20 tx=0x7f849800b990).stop 2021-02-10T11:47:06.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:06.309 7f84adb50700 1 -- 172.21.2.112:0/1628231404 shutdown_connections 2021-02-10T11:47:06.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:06.309 7f84adb50700 1 --2- 172.21.2.112:0/1628231404 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f848c0186c0 0x7f848c01abc0 unknown :-1 s=CLOSED pgs=1343 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:06.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:06.309 7f84adb50700 1 --2- 172.21.2.112:0/1628231404 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f84a80ec490 0x7f84a80bd050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:06.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:06.309 7f84adb50700 1 --2- 172.21.2.112:0/1628231404 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f84a8094710 0x7f84a80bca30 unknown :-1 s=CLOSED pgs=2973 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:06.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:06.309 7f84adb50700 1 --2- 172.21.2.112:0/1628231404 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f84a8094310 0x7f84a80ebe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:06.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:06.309 7f84adb50700 1 -- 172.21.2.112:0/1628231404 >> 172.21.2.112:0/1628231404 conn(0x7f84a808fa10 msgr2=0x7f84a80bfa60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:06.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:06.309 7f84adb50700 1 -- 172.21.2.112:0/1628231404 shutdown_connections 2021-02-10T11:47:06.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:06.309 7f84adb50700 1 -- 172.21.2.112:0/1628231404 wait complete. 2021-02-10T11:47:06.320 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2021-02-10T11:47:06.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:06.734 7fb7af3cc700 1 Processor -- start 2021-02-10T11:47:06.754 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:06.734 7fb7af3cc700 1 -- start start 2021-02-10T11:47:06.754 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:06.735 7fb7af3cc700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fb7a80d1880 0x7fb7a81b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:06.754 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:06.735 7fb7af3cc700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fb7a80cff80 0x7fb7a8109140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:06.754 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:06.735 7fb7af3cc700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7a80fe1d0 con 0x7fb7a81b0af0 2021-02-10T11:47:06.754 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:06.735 7fb7af3cc700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7a8109ae0 con 0x7fb7a80cff80 2021-02-10T11:47:06.754 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:06.735 7fb7af3cc700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7a8109760 con 0x7fb7a80d1880 2021-02-10T11:47:06.754 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:06.735 7fb7ad9e9700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fb7a80cff80 0x7fb7a8109140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:06.754 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:06.735 7fb7ac9e7700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fb7a81b0af0 0x7fb7a8108950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:59524/0 (socket says 172.21.2.112:59524) 2021-02-10T11:47:06.754 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:06.735 7fb7ad9e9700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fb7a80cff80 0x7fb7a8109140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:60060/0 (socket says 172.21.2.112:60060) 2021-02-10T11:47:06.754 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:06.735 7fb7ac9e7700 1 -- 172.21.2.112:0/1517616502 learned_addr learned my addr 172.21.2.112:0/1517616502 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:47:06.754 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:06.736 7fb7ad9e9700 1 -- 172.21.2.112:0/1517616502 >> v2:172.21.2.120:3300/0 conn(0x7fb7a80d1880 msgr2=0x7fb7a81b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:06.754 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:06.736 7fb7ad9e9700 1 --2- 172.21.2.112:0/1517616502 >> v2:172.21.2.120:3300/0 conn(0x7fb7a80d1880 0x7fb7a81b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:06.754 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:06.736 7fb7ad9e9700 1 -- 172.21.2.112:0/1517616502 >> v1:172.21.2.112:6789/0 conn(0x7fb7a81b0af0 legacy=0x7fb7a8108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:06.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:06.736 7fb7ad9e9700 1 -- 172.21.2.112:0/1517616502 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7a8088070 con 0x7fb7a80cff80 2021-02-10T11:47:06.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:06.736 7fb7ad9e9700 1 --2- 172.21.2.112:0/1517616502 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb7a80cff80 0x7fb7a8109140 secure :-1 s=READY pgs=2975 cs=0 l=1 rev1=1 rx=0x7fb7a0006ab0 tx=0x7fb7a0009b40).ready entity=mon.0 client_cookie=f432543357a67e02 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:06.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:06.736 7fb7a7fff700 1 -- 172.21.2.112:0/1517616502 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb7a00022e0 con 0x7fb7a80cff80 2021-02-10T11:47:06.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:06.736 7fb7a7fff700 1 -- 172.21.2.112:0/1517616502 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7a0002560 con 0x7fb7a80cff80 2021-02-10T11:47:06.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:06.736 7fb7a7fff700 1 -- 172.21.2.112:0/1517616502 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb7a00029f0 con 0x7fb7a80cff80 2021-02-10T11:47:06.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:06.736 7fb7af3cc700 1 -- 172.21.2.112:0/1517616502 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb7a80cff80 msgr2=0x7fb7a8109140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:06.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:06.736 7fb7af3cc700 1 --2- 172.21.2.112:0/1517616502 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb7a80cff80 0x7fb7a8109140 secure :-1 s=READY pgs=2975 cs=0 l=1 rev1=1 rx=0x7fb7a0006ab0 tx=0x7fb7a0009b40).stop 2021-02-10T11:47:06.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:06.737 7fb7af3cc700 1 -- 172.21.2.112:0/1517616502 shutdown_connections 2021-02-10T11:47:06.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:06.737 7fb7af3cc700 1 --2- 172.21.2.112:0/1517616502 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb7a80cff80 0x7fb7a8109140 unknown :-1 s=CLOSED pgs=2975 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:06.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:06.737 7fb7af3cc700 1 --2- 172.21.2.112:0/1517616502 >> v2:172.21.2.120:3300/0 conn(0x7fb7a80d1880 0x7fb7a81b04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:06.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:06.737 7fb7af3cc700 1 -- 172.21.2.112:0/1517616502 >> 172.21.2.112:0/1517616502 conn(0x7fb7a80ce430 msgr2=0x7fb7a80d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:06.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:06.737 7fb7af3cc700 1 -- 172.21.2.112:0/1517616502 shutdown_connections 2021-02-10T11:47:06.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:06.737 7fb7af3cc700 1 -- 172.21.2.112:0/1517616502 wait complete. 2021-02-10T11:47:06.757 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:06.751 7fb7af3cc700 1 Processor -- start 2021-02-10T11:47:06.757 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:06.751 7fb7af3cc700 1 -- start start 2021-02-10T11:47:06.757 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:06.752 7fb7af3cc700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb7a81b58d0 0x7fb7a80aa4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:06.757 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:06.752 7fb7af3cc700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb7a80aaaf0 0x7fb7a81b80a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:06.757 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:06.752 7fb7af3cc700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb7a81b86c0 0x7fb7a819de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:06.757 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:06.752 7fb7af3cc700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7a81b8ac0 con 0x7fb7a81b86c0 2021-02-10T11:47:06.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:06.752 7fb7af3cc700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7a81b8e20 con 0x7fb7a80aaaf0 2021-02-10T11:47:06.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:06.752 7fb7af3cc700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7a81b5cd0 con 0x7fb7a81b58d0 2021-02-10T11:47:06.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:06.752 7fb7ad9e9700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb7a81b86c0 0x7fb7a819de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:06.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:06.752 7fb7ad9e9700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb7a81b86c0 0x7fb7a819de70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:60070/0 (socket says 172.21.2.112:60070) 2021-02-10T11:47:06.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:06.752 7fb7ad9e9700 1 -- 172.21.2.112:0/2591162289 learned_addr learned my addr 172.21.2.112:0/2591162289 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:47:06.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:06.752 7fb7ad9e9700 1 -- 172.21.2.112:0/2591162289 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb7a81b58d0 msgr2=0x7fb7a80aa4d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:47:06.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:06.752 7fb7ad9e9700 1 --2- 172.21.2.112:0/2591162289 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb7a81b58d0 0x7fb7a80aa4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:06.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:06.752 7fb7ad9e9700 1 -- 172.21.2.112:0/2591162289 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb7a80aaaf0 msgr2=0x7fb7a81b80a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:47:06.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:06.752 7fb7ad9e9700 1 --2- 172.21.2.112:0/2591162289 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb7a80aaaf0 0x7fb7a81b80a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:06.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:06.752 7fb7ad9e9700 1 -- 172.21.2.112:0/2591162289 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7a8095710 con 0x7fb7a81b86c0 2021-02-10T11:47:06.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:06.752 7fb7ad9e9700 1 --2- 172.21.2.112:0/2591162289 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb7a81b86c0 0x7fb7a819de70 secure :-1 s=READY pgs=2976 cs=0 l=1 rev1=1 rx=0x7fb7a000b5e0 tx=0x7fb7a000b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:06.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:06.753 7fb7a5ffb700 1 -- 172.21.2.112:0/2591162289 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb7a0002280 con 0x7fb7a81b86c0 2021-02-10T11:47:06.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:06.753 7fb7a5ffb700 1 -- 172.21.2.112:0/2591162289 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7a0009070 con 0x7fb7a81b86c0 2021-02-10T11:47:06.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:06.753 7fb7a5ffb700 1 -- 172.21.2.112:0/2591162289 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb7a00027c0 con 0x7fb7a81b86c0 2021-02-10T11:47:06.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:06.753 7fb7af3cc700 1 -- 172.21.2.112:0/2591162289 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7a8095710 con 0x7fb7a81b86c0 2021-02-10T11:47:06.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:06.753 7fb7af3cc700 1 -- 172.21.2.112:0/2591162289 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7a819e780 con 0x7fb7a81b86c0 2021-02-10T11:47:06.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:06.753 7fb7a5ffb700 1 -- 172.21.2.112:0/2591162289 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb7a0006a20 con 0x7fb7a81b86c0 2021-02-10T11:47:06.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:06.754 7fb7a5ffb700 1 --2- 172.21.2.112:0/2591162289 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb7880186c0 0x7fb78801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:06.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:06.754 7fb7a5ffb700 1 -- 172.21.2.112:0/2591162289 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(173..173 src has 1..173) v4 ==== 5659+0+0 (secure 0 0 0) 0x7fb7a00325a0 con 0x7fb7a81b86c0 2021-02-10T11:47:06.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:06.754 7fb7ad1e8700 1 --2- 172.21.2.112:0/2591162289 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb7880186c0 0x7fb78801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:06.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:06.756 7fb7ad1e8700 1 --2- 172.21.2.112:0/2591162289 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb7880186c0 0x7fb78801abc0 crc :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:06.788 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:06.785 7fb7af3cc700 1 -- 172.21.2.112:0/2591162289 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7a8082aa0 con 0x7fb7a81b86c0 2021-02-10T11:47:06.790 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:06.788 7fb7a5ffb700 1 -- 172.21.2.112:0/2591162289 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb7a0030d00 con 0x7fb7a81b86c0 2021-02-10T11:47:06.917 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:06.914 7fb7af3cc700 1 -- 172.21.2.112:0/2591162289 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fb7a8082aa0 con 0x7fb7a81b86c0 2021-02-10T11:47:07.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:07.303 7fb7a5ffb700 1 -- 172.21.2.112:0/2591162289 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v174) v1 ==== 185+0+0 (secure 0 0 0) 0x7fb7a00056b0 con 0x7fb7a81b86c0 2021-02-10T11:47:07.335 INFO:teuthology.orchestra.run.gibba012.stderr:pool 'cephfs_metadata' removed 2021-02-10T11:47:07.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:07.333 7fb7af3cc700 1 -- 172.21.2.112:0/2591162289 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb7880186c0 msgr2=0x7fb78801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:07.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:07.333 7fb7af3cc700 1 --2- 172.21.2.112:0/2591162289 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb7880186c0 0x7fb78801abc0 crc :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:07.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:07.333 7fb7af3cc700 1 -- 172.21.2.112:0/2591162289 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb7a81b86c0 msgr2=0x7fb7a819de70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:07.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:07.333 7fb7af3cc700 1 --2- 172.21.2.112:0/2591162289 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb7a81b86c0 0x7fb7a819de70 secure :-1 s=READY pgs=2976 cs=0 l=1 rev1=1 rx=0x7fb7a000b5e0 tx=0x7fb7a000b980).stop 2021-02-10T11:47:07.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:07.334 7fb7af3cc700 1 -- 172.21.2.112:0/2591162289 shutdown_connections 2021-02-10T11:47:07.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:07.334 7fb7af3cc700 1 --2- 172.21.2.112:0/2591162289 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb7880186c0 0x7fb78801abc0 unknown :-1 s=CLOSED pgs=1344 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:07.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:07.334 7fb7af3cc700 1 --2- 172.21.2.112:0/2591162289 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb7a80aaaf0 0x7fb7a81b80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:07.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:07.334 7fb7af3cc700 1 --2- 172.21.2.112:0/2591162289 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb7a81b86c0 0x7fb7a819de70 unknown :-1 s=CLOSED pgs=2976 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:07.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:07.334 7fb7af3cc700 1 --2- 172.21.2.112:0/2591162289 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb7a81b58d0 0x7fb7a80aa4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:07.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:07.334 7fb7af3cc700 1 -- 172.21.2.112:0/2591162289 >> 172.21.2.112:0/2591162289 conn(0x7fb7a819f940 msgr2=0x7fb7a818c290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:07.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:07.334 7fb7af3cc700 1 -- 172.21.2.112:0/2591162289 shutdown_connections 2021-02-10T11:47:07.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:07.334 7fb7af3cc700 1 -- 172.21.2.112:0/2591162289 wait complete. 2021-02-10T11:47:07.345 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2021-02-10T11:47:07.777 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:07.758 7fb289136700 1 Processor -- start 2021-02-10T11:47:07.777 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:07.758 7fb289136700 1 -- start start 2021-02-10T11:47:07.777 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:07.759 7fb289136700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fb28410b000 0x7fb2840cf760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:07.777 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:07.759 7fb289136700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2840fe1d0 con 0x7fb2840d1880 2021-02-10T11:47:07.777 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:07.759 7fb289136700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2841b0cc0 con 0x7fb2840d1c80 2021-02-10T11:47:07.777 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:07.759 7fb289136700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb284109a70 con 0x7fb28410b000 2021-02-10T11:47:07.778 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:07.759 7fb28359e700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fb28410b000 0x7fb2840cf760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:07.778 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:07.759 7fb28359e700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fb28410b000 0x7fb2840cf760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:60078/0 (socket says 172.21.2.112:60078) 2021-02-10T11:47:07.778 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:07.759 7fb28359e700 1 -- 172.21.2.112:0/266520155 learned_addr learned my addr 172.21.2.112:0/266520155 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:47:07.778 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:07.759 7fb28359e700 1 -- 172.21.2.112:0/266520155 >> v1:172.21.2.120:6789/0 conn(0x7fb2840d1c80 legacy=0x7fb2840cef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:07.778 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:07.759 7fb28359e700 1 -- 172.21.2.112:0/266520155 >> v1:172.21.2.112:6789/0 conn(0x7fb2840d1880 legacy=0x7fb2841b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:07.778 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:07.759 7fb28359e700 1 -- 172.21.2.112:0/266520155 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb28410b400 con 0x7fb28410b000 2021-02-10T11:47:07.778 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:07.759 7fb28359e700 1 --2- 172.21.2.112:0/266520155 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb28410b000 0x7fb2840cf760 secure :-1 s=READY pgs=2978 cs=0 l=1 rev1=1 rx=0x7fb27c006f00 tx=0x7fb27c009f10).ready entity=mon.0 client_cookie=ab70c9e3c5ec834e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:07.778 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:07.760 7fb281d9b700 1 -- 172.21.2.112:0/266520155 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb27c00b3b0 con 0x7fb28410b000 2021-02-10T11:47:07.778 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:07.760 7fb281d9b700 1 -- 172.21.2.112:0/266520155 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb27c00b730 con 0x7fb28410b000 2021-02-10T11:47:07.778 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:07.760 7fb281d9b700 1 -- 172.21.2.112:0/266520155 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb27c00bc10 con 0x7fb28410b000 2021-02-10T11:47:07.778 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:07.760 7fb289136700 1 -- 172.21.2.112:0/266520155 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb28410b000 msgr2=0x7fb2840cf760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:07.778 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:07.760 7fb289136700 1 --2- 172.21.2.112:0/266520155 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb28410b000 0x7fb2840cf760 secure :-1 s=READY pgs=2978 cs=0 l=1 rev1=1 rx=0x7fb27c006f00 tx=0x7fb27c009f10).stop 2021-02-10T11:47:07.778 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:07.760 7fb289136700 1 -- 172.21.2.112:0/266520155 shutdown_connections 2021-02-10T11:47:07.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:07.760 7fb289136700 1 --2- 172.21.2.112:0/266520155 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb28410b000 0x7fb2840cf760 unknown :-1 s=CLOSED pgs=2978 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:07.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:07.760 7fb289136700 1 -- 172.21.2.112:0/266520155 >> 172.21.2.112:0/266520155 conn(0x7fb2840ce430 msgr2=0x7fb2840d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:07.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:07.760 7fb289136700 1 -- 172.21.2.112:0/266520155 shutdown_connections 2021-02-10T11:47:07.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:07.760 7fb289136700 1 -- 172.21.2.112:0/266520155 wait complete. 2021-02-10T11:47:07.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:07.775 7fb289136700 1 Processor -- start 2021-02-10T11:47:07.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:07.775 7fb289136700 1 -- start start 2021-02-10T11:47:07.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:07.775 7fb289136700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb284096300 0x7fb284096700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:07.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:07.775 7fb289136700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb2840b1990 0x7fb2840b1d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:07.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:07.775 7fb289136700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb2840b23b0 0x7fb284101100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:07.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:07.775 7fb289136700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2840b2830 con 0x7fb2840b1990 2021-02-10T11:47:07.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:07.775 7fb289136700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2840b2a50 con 0x7fb284096300 2021-02-10T11:47:07.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:07.775 7fb289136700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb284101720 con 0x7fb2840b23b0 2021-02-10T11:47:07.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:07.775 7fb28259c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb2840b1990 0x7fb2840b1d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:07.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:07.775 7fb28259c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb2840b1990 0x7fb2840b1d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:60082/0 (socket says 172.21.2.112:60082) 2021-02-10T11:47:07.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:07.775 7fb282d9d700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb284096300 0x7fb284096700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:07.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:07.775 7fb28259c700 1 -- 172.21.2.112:0/3939859139 learned_addr learned my addr 172.21.2.112:0/3939859139 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:47:07.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:07.775 7fb28359e700 1 --2- 172.21.2.112:0/3939859139 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb2840b23b0 0x7fb284101100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:07.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:07.776 7fb28259c700 1 -- 172.21.2.112:0/3939859139 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb2840b23b0 msgr2=0x7fb284101100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:07.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:07.776 7fb28259c700 1 --2- 172.21.2.112:0/3939859139 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb2840b23b0 0x7fb284101100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:07.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:07.776 7fb28259c700 1 -- 172.21.2.112:0/3939859139 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb284096300 msgr2=0x7fb284096700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:07.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:07.776 7fb28259c700 1 --2- 172.21.2.112:0/3939859139 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb284096300 0x7fb284096700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:07.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:07.776 7fb28259c700 1 -- 172.21.2.112:0/3939859139 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb284101940 con 0x7fb2840b1990 2021-02-10T11:47:07.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:07.776 7fb28259c700 1 --2- 172.21.2.112:0/3939859139 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb2840b1990 0x7fb2840b1d90 secure :-1 s=READY pgs=2979 cs=0 l=1 rev1=1 rx=0x7fb26c006ed0 tx=0x7fb26c00b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:07.782 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:07.776 7fb27b7fe700 1 -- 172.21.2.112:0/3939859139 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb26c006020 con 0x7fb2840b1990 2021-02-10T11:47:07.782 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:07.776 7fb27b7fe700 1 -- 172.21.2.112:0/3939859139 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb26c006360 con 0x7fb2840b1990 2021-02-10T11:47:07.782 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:07.776 7fb27b7fe700 1 -- 172.21.2.112:0/3939859139 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb26c005730 con 0x7fb2840b1990 2021-02-10T11:47:07.782 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:07.776 7fb289136700 1 -- 172.21.2.112:0/3939859139 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb284101940 con 0x7fb2840b1990 2021-02-10T11:47:07.782 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:07.776 7fb289136700 1 -- 172.21.2.112:0/3939859139 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb284101ed0 con 0x7fb2840b1990 2021-02-10T11:47:07.782 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:07.777 7fb27b7fe700 1 -- 172.21.2.112:0/3939859139 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb26c006020 con 0x7fb2840b1990 2021-02-10T11:47:07.782 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:07.777 7fb27b7fe700 1 --2- 172.21.2.112:0/3939859139 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb2680186c0 0x7fb26801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:07.782 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:07.777 7fb27b7fe700 1 -- 172.21.2.112:0/3939859139 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(174..174 src has 1..174) v4 ==== 5221+0+0 (secure 0 0 0) 0x7fb26c0324a0 con 0x7fb2840b1990 2021-02-10T11:47:07.782 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:07.777 7fb282d9d700 1 --2- 172.21.2.112:0/3939859139 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb2680186c0 0x7fb26801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:07.782 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:07.777 7fb282d9d700 1 --2- 172.21.2.112:0/3939859139 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb2680186c0 0x7fb26801abc0 crc :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:07.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:07.809 7fb289136700 1 -- 172.21.2.112:0/3939859139 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb284102470 con 0x7fb2840b1990 2021-02-10T11:47:07.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:07.811 7fb27b7fe700 1 -- 172.21.2.112:0/3939859139 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb26c030db0 con 0x7fb2840b1990 2021-02-10T11:47:07.943 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:07.941 7fb289136700 1 -- 172.21.2.112:0/3939859139 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fb284102470 con 0x7fb2840b1990 2021-02-10T11:47:08.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:08.307 7fb27b7fe700 1 -- 172.21.2.112:0/3939859139 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v175) v1 ==== 173+0+0 (secure 0 0 0) 0x7fb26c005fe0 con 0x7fb2840b1990 2021-02-10T11:47:08.310 INFO:teuthology.orchestra.run.gibba012.stderr:pool 'cephfs_data' removed 2021-02-10T11:47:08.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:08.310 7fb289136700 1 -- 172.21.2.112:0/3939859139 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb2680186c0 msgr2=0x7fb26801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:08.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:08.310 7fb289136700 1 --2- 172.21.2.112:0/3939859139 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb2680186c0 0x7fb26801abc0 crc :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:08.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:08.310 7fb289136700 1 -- 172.21.2.112:0/3939859139 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb2840b1990 msgr2=0x7fb2840b1d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:08.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:08.310 7fb289136700 1 --2- 172.21.2.112:0/3939859139 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb2840b1990 0x7fb2840b1d90 secure :-1 s=READY pgs=2979 cs=0 l=1 rev1=1 rx=0x7fb26c006ed0 tx=0x7fb26c00b960).stop 2021-02-10T11:47:08.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:08.310 7fb289136700 1 -- 172.21.2.112:0/3939859139 shutdown_connections 2021-02-10T11:47:08.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:08.310 7fb289136700 1 --2- 172.21.2.112:0/3939859139 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb2680186c0 0x7fb26801abc0 unknown :-1 s=CLOSED pgs=1345 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:08.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:08.310 7fb289136700 1 --2- 172.21.2.112:0/3939859139 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb2840b23b0 0x7fb284101100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:08.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:08.310 7fb289136700 1 --2- 172.21.2.112:0/3939859139 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb2840b1990 0x7fb2840b1d90 unknown :-1 s=CLOSED pgs=2979 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:08.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:08.310 7fb289136700 1 --2- 172.21.2.112:0/3939859139 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb284096300 0x7fb284096700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:08.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:08.310 7fb289136700 1 -- 172.21.2.112:0/3939859139 >> 172.21.2.112:0/3939859139 conn(0x7fb28409fb30 msgr2=0x7fb2840bb050 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:08.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:08.310 7fb289136700 1 -- 172.21.2.112:0/3939859139 shutdown_connections 2021-02-10T11:47:08.314 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:08.310 7fb289136700 1 -- 172.21.2.112:0/3939859139 wait complete. 2021-02-10T11:47:08.321 INFO:tasks.ceph.mds.a:Restarting daemon 2021-02-10T11:47:08.321 INFO:tasks.ceph.mds.a:Stopping old one... 2021-02-10T11:47:08.321 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2021-02-10T11:47:08.321 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:47:08.321 INFO:tasks.ceph.mds.b:Restarting daemon 2021-02-10T11:47:08.321 INFO:tasks.ceph.mds.b:Stopping old one... 2021-02-10T11:47:08.322 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2021-02-10T11:47:08.322 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:47:08.327 INFO:tasks.ceph.mds.a:Stopped 2021-02-10T11:47:08.327 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2021-02-10T11:47:08.328 INFO:tasks.ceph.mds.b:Stopped 2021-02-10T11:47:08.328 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2021-02-10T11:47:08.369 INFO:tasks.ceph.mds.a:Started 2021-02-10T11:47:08.369 DEBUG:teuthology.parallel:result is None 2021-02-10T11:47:08.371 INFO:tasks.ceph.mds.b:Started 2021-02-10T11:47:08.372 DEBUG:teuthology.parallel:result is None 2021-02-10T11:47:08.372 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd blacklist clear 2021-02-10T11:47:08.777 INFO:tasks.ceph.mds.a.gibba012.stdout:starting mds.a at 2021-02-10T11:47:08.778 INFO:tasks.ceph.mds.b.gibba020.stdout:starting mds.b at 2021-02-10T11:47:08.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:08.777 7f1693bb2700 1 Processor -- start 2021-02-10T11:47:08.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:08.778 7f1693bb2700 1 -- start start 2021-02-10T11:47:08.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:08.778 7f1693bb2700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f168c0d0340 0x7f168c0ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:08.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:08.778 7f1693bb2700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f168c0f0b10 con 0x7f168c0d1c70 2021-02-10T11:47:08.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:08.778 7f1693bb2700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f168c1b0be0 con 0x7f168c0d1870 2021-02-10T11:47:08.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:08.778 7f1693bb2700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f168c0e5ed0 con 0x7f168c0d0340 2021-02-10T11:47:08.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:08.778 7f16921cf700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f168c0d1c70 0x7f168c0efc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:59564/0 (socket says 172.21.2.112:59564) 2021-02-10T11:47:08.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:08.778 7f16921cf700 1 -- 172.21.2.112:0/1146273253 learned_addr learned my addr 172.21.2.112:0/1146273253 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:47:08.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:08.778 7f16911cd700 1 --2- 172.21.2.112:0/1146273253 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f168c0d0340 0x7f168c0ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:08.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:08.778 7f16909cc700 1 -- 172.21.2.112:0/1146273253 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1901494741 0 0) 0x7f1684000d30 con 0x7f168c0d1c70 2021-02-10T11:47:08.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:08.778 7f16909cc700 1 -- 172.21.2.112:0/1146273253 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1678002fe0 con 0x7f168c0d1c70 2021-02-10T11:47:08.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:08.778 7f16909cc700 1 -- 172.21.2.112:0/1146273253 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2551613004 0 0) 0x7f16800009b0 con 0x7f168c0d1870 2021-02-10T11:47:08.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:08.778 7f16909cc700 1 -- 172.21.2.112:0/1146273253 --> v1:172.21.2.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1678003210 con 0x7f168c0d1870 2021-02-10T11:47:08.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:08.778 7f16909cc700 1 -- 172.21.2.112:0/1146273253 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 845623844 0 0) 0x7f1684001460 con 0x7f168c0d1c70 2021-02-10T11:47:08.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:08.778 7f16909cc700 1 -- 172.21.2.112:0/1146273253 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f168c0d0340 msgr2=0x7f168c0ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:08.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:08.778 7f16909cc700 1 --2- 172.21.2.112:0/1146273253 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f168c0d0340 0x7f168c0ef640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:08.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:08.778 7f16909cc700 1 -- 172.21.2.112:0/1146273253 >> v1:172.21.2.120:6789/0 conn(0x7f168c0d1870 legacy=0x7f168c1b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:08.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:08.778 7f16909cc700 1 -- 172.21.2.112:0/1146273253 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f168c0d2fe0 con 0x7f168c0d1c70 2021-02-10T11:47:08.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:08.778 7f16909cc700 1 -- 172.21.2.112:0/1146273253 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f1684001980 con 0x7f168c0d1c70 2021-02-10T11:47:08.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:08.778 7f16909cc700 1 -- 172.21.2.112:0/1146273253 >> v1:172.21.2.112:6789/0 conn(0x7f168c0d1c70 legacy=0x7f168c0efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:08.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:08.778 7f16909cc700 1 --2- 172.21.2.112:0/1146273253 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1678003b90 0x7f1678005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:08.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:08.778 7f16909cc700 1 --2- 172.21.2.112:0/1146273253 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1678006770 0x7f1678008c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:08.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:08.778 7f16909cc700 1 --2- 172.21.2.112:0/1146273253 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1678009500 0x7f167800ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:08.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:08.779 7f16909cc700 1 -- 172.21.2.112:0/1146273253 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16780502e0 con 0x7f1678006770 2021-02-10T11:47:08.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:08.779 7f16909cc700 1 -- 172.21.2.112:0/1146273253 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16780504c0 con 0x7f1678009500 2021-02-10T11:47:08.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:08.779 7f16909cc700 1 -- 172.21.2.112:0/1146273253 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16780506e0 con 0x7f1678003b90 2021-02-10T11:47:08.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:08.779 7f16911cd700 1 --2- 172.21.2.112:0/1146273253 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1678006770 0x7f1678008c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:08.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:08.779 7f16919ce700 1 --2- 172.21.2.112:0/1146273253 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1678003b90 0x7f1678005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:08.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:08.779 7f16911cd700 1 -- 172.21.2.112:0/1146273253 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1678003b90 msgr2=0x7f1678005fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:08.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:08.779 7f16911cd700 1 --2- 172.21.2.112:0/1146273253 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1678003b90 0x7f1678005fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:08.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:08.779 7f16911cd700 1 -- 172.21.2.112:0/1146273253 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1678009500 msgr2=0x7f167800ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:08.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:08.779 7f16911cd700 1 --2- 172.21.2.112:0/1146273253 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f1678009500 0x7f167800ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:08.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:08.779 7f16911cd700 1 -- 172.21.2.112:0/1146273253 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1678050980 con 0x7f1678006770 2021-02-10T11:47:08.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:08.779 7f16911cd700 1 --2- 172.21.2.112:0/1146273253 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1678006770 0x7f1678008c60 secure :-1 s=READY pgs=2985 cs=0 l=1 rev1=1 rx=0x7f1688009310 tx=0x7f1688009a10).ready entity=mon.0 client_cookie=86a0949d1679a205 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:08.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:08.779 7f16909cc700 1 -- 172.21.2.112:0/1146273253 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f16880009b0 con 0x7f1678006770 2021-02-10T11:47:08.800 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:08.779 7f16909cc700 1 -- 172.21.2.112:0/1146273253 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1688006400 con 0x7f1678006770 2021-02-10T11:47:08.800 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:08.779 7f16909cc700 1 -- 172.21.2.112:0/1146273253 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f16880009b0 con 0x7f1678006770 2021-02-10T11:47:08.800 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:08.779 7f1693bb2700 1 -- 172.21.2.112:0/1146273253 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1678006770 msgr2=0x7f1678008c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:08.800 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:08.779 7f1693bb2700 1 --2- 172.21.2.112:0/1146273253 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f1678006770 0x7f1678008c60 secure :-1 s=READY pgs=2985 cs=0 l=1 rev1=1 rx=0x7f1688009310 tx=0x7f1688009a10).stop 2021-02-10T11:47:08.800 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:08.779 7f16921cf700 1 -- 172.21.2.112:0/1146273253 reap_dead start 2021-02-10T11:47:08.800 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:08.779 7f1693bb2700 1 -- 172.21.2.112:0/1146273253 shutdown_connections 2021-02-10T11:47:08.800 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:08.779 7f1693bb2700 1 -- 172.21.2.112:0/1146273253 >> 172.21.2.112:0/1146273253 conn(0x7f168c0ce450 msgr2=0x7f168c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:08.800 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:08.780 7f1693bb2700 1 -- 172.21.2.112:0/1146273253 shutdown_connections 2021-02-10T11:47:08.800 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:08.780 7f1693bb2700 1 -- 172.21.2.112:0/1146273253 wait complete. 2021-02-10T11:47:08.800 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:08.794 7f1693bb2700 1 Processor -- start 2021-02-10T11:47:08.800 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:08.794 7f1693bb2700 1 -- start start 2021-02-10T11:47:08.802 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:08.795 7f1693bb2700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f168c19ffa0 0x7f168c0eda10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:08.802 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:08.795 7f1693bb2700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f168c1a03a0 0x7f168c0ee030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:08.802 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:08.795 7f1693bb2700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f168c0ee650 0x7f168c0ebe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:08.802 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:08.795 7f1693bb2700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f168c0ed490 con 0x7f168c0ee650 2021-02-10T11:47:08.802 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:08.795 7f1693bb2700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f168c0eea50 con 0x7f168c19ffa0 2021-02-10T11:47:08.802 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:08.795 7f1693bb2700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f168c0eec30 con 0x7f168c1a03a0 2021-02-10T11:47:08.802 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:08.795 7f16921cf700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f168c0ee650 0x7f168c0ebe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:08.802 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:08.795 7f16921cf700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f168c0ee650 0x7f168c0ebe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:60116/0 (socket says 172.21.2.112:60116) 2021-02-10T11:47:08.802 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:08.795 7f16921cf700 1 -- 172.21.2.112:0/2936418860 learned_addr learned my addr 172.21.2.112:0/2936418860 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:47:08.803 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:08.795 7f16911cd700 1 --2- 172.21.2.112:0/2936418860 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f168c1a03a0 0x7f168c0ee030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:08.803 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:08.795 7f16919ce700 1 --2- 172.21.2.112:0/2936418860 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f168c19ffa0 0x7f168c0eda10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:08.803 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:08.795 7f16911cd700 1 -- 172.21.2.112:0/2936418860 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f168c19ffa0 msgr2=0x7f168c0eda10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:08.803 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:08.795 7f16911cd700 1 --2- 172.21.2.112:0/2936418860 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f168c19ffa0 0x7f168c0eda10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:08.803 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:08.795 7f16911cd700 1 -- 172.21.2.112:0/2936418860 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f168c0ee650 msgr2=0x7f168c0ebe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:08.803 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:08.795 7f16911cd700 1 --2- 172.21.2.112:0/2936418860 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f168c0ee650 0x7f168c0ebe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:08.803 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:08.795 7f16911cd700 1 -- 172.21.2.112:0/2936418860 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f168c0ec4a0 con 0x7f168c1a03a0 2021-02-10T11:47:08.803 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:08.795 7f16921cf700 1 --2- 172.21.2.112:0/2936418860 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f168c0ee650 0x7f168c0ebe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:47:08.803 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:08.796 7f16911cd700 1 --2- 172.21.2.112:0/2936418860 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f168c1a03a0 0x7f168c0ee030 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7f168800afc0 tx=0x7f168800c840).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:08.803 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:08.796 7f167e7fc700 1 -- 172.21.2.112:0/2936418860 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1688000940 con 0x7f168c1a03a0 2021-02-10T11:47:08.803 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:08.796 7f167e7fc700 1 -- 172.21.2.112:0/2936418860 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1688000c90 con 0x7f168c1a03a0 2021-02-10T11:47:08.803 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:08.796 7f167e7fc700 1 -- 172.21.2.112:0/2936418860 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f16880020d0 con 0x7f168c1a03a0 2021-02-10T11:47:08.803 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:08.796 7f1693bb2700 1 -- 172.21.2.112:0/2936418860 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f168c0ec4a0 con 0x7f168c1a03a0 2021-02-10T11:47:08.804 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:08.796 7f1693bb2700 1 -- 172.21.2.112:0/2936418860 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f168c0ec4a0 con 0x7f168c1a03a0 2021-02-10T11:47:08.804 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:08.796 7f167e7fc700 1 -- 172.21.2.112:0/2936418860 <== mon.2 v2:172.21.2.120:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1688000940 con 0x7f168c1a03a0 2021-02-10T11:47:08.804 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:08.796 7f167e7fc700 1 --2- 172.21.2.112:0/2936418860 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f16740186c0 0x7f167401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:08.804 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:08.796 7f167e7fc700 1 -- 172.21.2.112:0/2936418860 <== mon.2 v2:172.21.2.120:3301/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 4839+0+0 (secure 0 0 0) 0x7f16880321b0 con 0x7f168c1a03a0 2021-02-10T11:47:08.804 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:08.797 7f16919ce700 1 --2- 172.21.2.112:0/2936418860 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f16740186c0 0x7f167401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:08.804 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:08.797 7f16919ce700 1 --2- 172.21.2.112:0/2936418860 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f16740186c0 0x7f167401abc0 crc :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:08.829 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:08.827 7f1693bb2700 1 -- 172.21.2.112:0/2936418860 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f168c0ebc50 con 0x7f168c1a03a0 2021-02-10T11:47:08.832 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:08.829 7f167e7fc700 1 -- 172.21.2.112:0/2936418860 <== mon.2 v2:172.21.2.120:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1688026e70 con 0x7f168c1a03a0 2021-02-10T11:47:08.974 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:08.971 7f1693bb2700 1 -- 172.21.2.112:0/2936418860 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7f168c0ebc50 con 0x7f168c1a03a0 2021-02-10T11:47:09.315 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:09.313 7f167e7fc700 1 -- 172.21.2.112:0/2936418860 <== mon.2 v2:172.21.2.120:3301/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v176) v1 ==== 97+0+0 (secure 0 0 0) 0x7f1688026e70 con 0x7f168c1a03a0 2021-02-10T11:47:09.340 INFO:teuthology.orchestra.run.gibba012.stderr: removed all blacklist entries 2021-02-10T11:47:09.342 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:09.339 7f1693bb2700 1 -- 172.21.2.112:0/2936418860 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f16740186c0 msgr2=0x7f167401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:09.342 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:09.339 7f1693bb2700 1 --2- 172.21.2.112:0/2936418860 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f16740186c0 0x7f167401abc0 crc :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:09.342 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:09.340 7f1693bb2700 1 -- 172.21.2.112:0/2936418860 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f168c1a03a0 msgr2=0x7f168c0ee030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:09.342 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:09.340 7f1693bb2700 1 --2- 172.21.2.112:0/2936418860 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f168c1a03a0 0x7f168c0ee030 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7f168800afc0 tx=0x7f168800c840).stop 2021-02-10T11:47:09.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:09.340 7f1693bb2700 1 -- 172.21.2.112:0/2936418860 shutdown_connections 2021-02-10T11:47:09.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:09.340 7f1693bb2700 1 --2- 172.21.2.112:0/2936418860 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f16740186c0 0x7f167401abc0 unknown :-1 s=CLOSED pgs=1346 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:09.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:09.340 7f1693bb2700 1 --2- 172.21.2.112:0/2936418860 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f168c0ee650 0x7f168c0ebe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:09.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:09.340 7f1693bb2700 1 --2- 172.21.2.112:0/2936418860 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f168c1a03a0 0x7f168c0ee030 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:09.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:09.340 7f1693bb2700 1 --2- 172.21.2.112:0/2936418860 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f168c19ffa0 0x7f168c0eda10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:09.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:09.340 7f1693bb2700 1 -- 172.21.2.112:0/2936418860 >> 172.21.2.112:0/2936418860 conn(0x7f168c094d90 msgr2=0x7f168c08a420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:09.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:09.340 7f1693bb2700 1 -- 172.21.2.112:0/2936418860 shutdown_connections 2021-02-10T11:47:09.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:09.340 7f1693bb2700 1 -- 172.21.2.112:0/2936418860 wait complete. 2021-02-10T11:47:09.355 INFO:tasks.cephfs.cephfs_test_case:['0', '1', '2', '3'] 2021-02-10T11:47:09.355 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth ls --format=json-pretty 2021-02-10T11:47:09.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:09.766 7f73cdd3e700 1 Processor -- start 2021-02-10T11:47:09.786 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:09.766 7f73cdd3e700 1 -- start start 2021-02-10T11:47:09.786 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:09.766 7f73cdd3e700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f73c80d0340 0x7f73c80ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:09.786 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:09.766 7f73cdd3e700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73c80f0b10 con 0x7f73c80d1870 2021-02-10T11:47:09.786 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:09.766 7f73cdd3e700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73c81b0be0 con 0x7f73c80d1c70 2021-02-10T11:47:09.786 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:09.766 7f73cdd3e700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73c80e5ed0 con 0x7f73c80d0340 2021-02-10T11:47:09.786 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:09.766 7f73c77fe700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f73c80d1870 0x7f73c81b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:59582/0 (socket says 172.21.2.112:59582) 2021-02-10T11:47:09.786 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:09.766 7f73c77fe700 1 -- 172.21.2.112:0/168158485 learned_addr learned my addr 172.21.2.112:0/168158485 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:47:09.786 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:09.767 7f73c67fc700 1 -- 172.21.2.112:0/168158485 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 887396867 0 0) 0x7f73b80010b0 con 0x7f73c80d1870 2021-02-10T11:47:09.786 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:09.767 7f73c67fc700 1 -- 172.21.2.112:0/168158485 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73b4002fe0 con 0x7f73c80d1870 2021-02-10T11:47:09.786 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:09.767 7f73c6ffd700 1 --2- 172.21.2.112:0/168158485 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f73c80d0340 0x7f73c80ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:09.786 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:09.767 7f73c67fc700 1 -- 172.21.2.112:0/168158485 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4029299705 0 0) 0x7f73b80010b0 con 0x7f73c80d1870 2021-02-10T11:47:09.787 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:09.767 7f73c67fc700 1 -- 172.21.2.112:0/168158485 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f73c80d0340 msgr2=0x7f73c80ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:09.787 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:09.767 7f73c67fc700 1 --2- 172.21.2.112:0/168158485 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f73c80d0340 0x7f73c80ef640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:09.787 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:09.767 7f73c67fc700 1 -- 172.21.2.112:0/168158485 >> v1:172.21.2.120:6789/0 conn(0x7f73c80d1c70 legacy=0x7f73c80efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:09.787 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:09.767 7f73c67fc700 1 -- 172.21.2.112:0/168158485 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73c80f0450 con 0x7f73c80d1870 2021-02-10T11:47:09.787 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:09.767 7f73c67fc700 1 -- 172.21.2.112:0/168158485 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f73b80015b0 con 0x7f73c80d1870 2021-02-10T11:47:09.787 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:09.767 7f73c67fc700 1 -- 172.21.2.112:0/168158485 >> v1:172.21.2.112:6789/0 conn(0x7f73c80d1870 legacy=0x7f73c81b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:09.787 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:09.767 7f73c67fc700 1 --2- 172.21.2.112:0/168158485 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f73b4003a70 0x7f73b4005f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:09.787 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:09.767 7f73c67fc700 1 --2- 172.21.2.112:0/168158485 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f73b4006810 0x7f73b4008d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:09.787 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:09.767 7f73c67fc700 1 --2- 172.21.2.112:0/168158485 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f73b4009630 0x7f73b400bb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:09.787 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:09.767 7f73c67fc700 1 -- 172.21.2.112:0/168158485 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73b400c2f0 con 0x7f73b4003a70 2021-02-10T11:47:09.787 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:09.767 7f73c67fc700 1 -- 172.21.2.112:0/168158485 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73b400c510 con 0x7f73b4006810 2021-02-10T11:47:09.787 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:09.767 7f73c67fc700 1 -- 172.21.2.112:0/168158485 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73b400c730 con 0x7f73b4009630 2021-02-10T11:47:09.788 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:09.767 7f73c77fe700 1 --2- 172.21.2.112:0/168158485 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f73b4003a70 0x7f73b4005f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:09.788 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:09.767 7f73c6ffd700 1 --2- 172.21.2.112:0/168158485 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f73b4006810 0x7f73b4008d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:09.788 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:09.767 7f73c77fe700 1 -- 172.21.2.112:0/168158485 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f73b4009630 msgr2=0x7f73b400bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:09.788 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:09.767 7f73c77fe700 1 --2- 172.21.2.112:0/168158485 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f73b4009630 0x7f73b400bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:09.788 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:09.767 7f73c77fe700 1 -- 172.21.2.112:0/168158485 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f73b4006810 msgr2=0x7f73b4008d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:09.788 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:09.767 7f73c77fe700 1 --2- 172.21.2.112:0/168158485 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f73b4006810 0x7f73b4008d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:09.788 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:09.767 7f73c77fe700 1 -- 172.21.2.112:0/168158485 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73b400c9d0 con 0x7f73b4003a70 2021-02-10T11:47:09.788 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:09.767 7f73c6ffd700 1 --2- 172.21.2.112:0/168158485 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f73b4006810 0x7f73b4008d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:47:09.788 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:09.767 7f73c77fe700 1 --2- 172.21.2.112:0/168158485 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f73b4003a70 0x7f73b4005f70 secure :-1 s=READY pgs=2987 cs=0 l=1 rev1=1 rx=0x7f73b8007560 tx=0x7f73b8007d10).ready entity=mon.0 client_cookie=4dd3eab970601c8d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:09.788 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:09.768 7f73c67fc700 1 -- 172.21.2.112:0/168158485 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f73b80070d0 con 0x7f73b4003a70 2021-02-10T11:47:09.788 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:09.768 7f73c67fc700 1 -- 172.21.2.112:0/168158485 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73b8001400 con 0x7f73b4003a70 2021-02-10T11:47:09.788 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:09.768 7f73c67fc700 1 -- 172.21.2.112:0/168158485 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f73b804e510 con 0x7f73b4003a70 2021-02-10T11:47:09.788 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:09.768 7f73cdd3e700 1 -- 172.21.2.112:0/168158485 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f73b4003a70 msgr2=0x7f73b4005f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:09.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:09.768 7f73cdd3e700 1 --2- 172.21.2.112:0/168158485 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f73b4003a70 0x7f73b4005f70 secure :-1 s=READY pgs=2987 cs=0 l=1 rev1=1 rx=0x7f73b8007560 tx=0x7f73b8007d10).stop 2021-02-10T11:47:09.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:09.768 7f73c7fff700 1 -- 172.21.2.112:0/168158485 reap_dead start 2021-02-10T11:47:09.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:09.768 7f73cdd3e700 1 -- 172.21.2.112:0/168158485 shutdown_connections 2021-02-10T11:47:09.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:09.768 7f73cdd3e700 1 -- 172.21.2.112:0/168158485 >> 172.21.2.112:0/168158485 conn(0x7f73c80ce450 msgr2=0x7f73c80d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:09.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:09.768 7f73cdd3e700 1 -- 172.21.2.112:0/168158485 shutdown_connections 2021-02-10T11:47:09.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:09.768 7f73cdd3e700 1 -- 172.21.2.112:0/168158485 wait complete. 2021-02-10T11:47:09.790 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:09.783 7f73cdd3e700 1 Processor -- start 2021-02-10T11:47:09.790 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:09.783 7f73cdd3e700 1 -- start start 2021-02-10T11:47:09.791 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:09.784 7f73cdd3e700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f73c8195650 0x7f73c80b5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:09.791 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:09.784 7f73cdd3e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f73c80b5b80 0x7f73c81a2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:09.791 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:09.784 7f73cdd3e700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f73c80acfa0 0x7f73c81a3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:09.791 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:09.784 7f73cdd3e700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73c80b6010 con 0x7f73c80b5b80 2021-02-10T11:47:09.791 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:09.784 7f73cdd3e700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73c81a3a60 con 0x7f73c8195650 2021-02-10T11:47:09.791 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:09.784 7f73cdd3e700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73c81a3c40 con 0x7f73c80acfa0 2021-02-10T11:47:09.791 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:09.784 7f73c6ffd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f73c80b5b80 0x7f73c81a2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:09.791 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:09.784 7f73c6ffd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f73c80b5b80 0x7f73c81a2e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:60130/0 (socket says 172.21.2.112:60130) 2021-02-10T11:47:09.791 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:09.784 7f73c6ffd700 1 -- 172.21.2.112:0/988330081 learned_addr learned my addr 172.21.2.112:0/988330081 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:47:09.791 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:09.784 7f73c6ffd700 1 -- 172.21.2.112:0/988330081 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f73c80acfa0 msgr2=0x7f73c81a3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:09.791 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:09.784 7f73c6ffd700 1 --2- 172.21.2.112:0/988330081 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f73c80acfa0 0x7f73c81a3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:09.791 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:09.784 7f73c6ffd700 1 -- 172.21.2.112:0/988330081 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f73c8195650 msgr2=0x7f73c80b5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:09.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:09.784 7f73c6ffd700 1 --2- 172.21.2.112:0/988330081 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f73c8195650 0x7f73c80b5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:09.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:09.784 7f73c6ffd700 1 -- 172.21.2.112:0/988330081 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73c80bcae0 con 0x7f73c80b5b80 2021-02-10T11:47:09.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:09.784 7f73c6ffd700 1 --2- 172.21.2.112:0/988330081 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f73c80b5b80 0x7f73c81a2e20 secure :-1 s=READY pgs=2988 cs=0 l=1 rev1=1 rx=0x7f73b0006f10 tx=0x7f73b000b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:09.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:09.784 7f73bffff700 1 -- 172.21.2.112:0/988330081 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f73b000bd80 con 0x7f73c80b5b80 2021-02-10T11:47:09.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:09.784 7f73bffff700 1 -- 172.21.2.112:0/988330081 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73b0000c90 con 0x7f73c80b5b80 2021-02-10T11:47:09.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:09.784 7f73bffff700 1 -- 172.21.2.112:0/988330081 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f73b00022b0 con 0x7f73c80b5b80 2021-02-10T11:47:09.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:09.784 7f73cdd3e700 1 -- 172.21.2.112:0/988330081 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73c80bcae0 con 0x7f73c80b5b80 2021-02-10T11:47:09.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:09.785 7f73cdd3e700 1 -- 172.21.2.112:0/988330081 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73c80bcae0 con 0x7f73c80b5b80 2021-02-10T11:47:09.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:09.785 7f73bffff700 1 -- 172.21.2.112:0/988330081 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f73b0009fe0 con 0x7f73c80b5b80 2021-02-10T11:47:09.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:09.785 7f73bffff700 1 --2- 172.21.2.112:0/988330081 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f73ac018700 0x7f73ac01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:09.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:09.785 7f73bffff700 1 -- 172.21.2.112:0/988330081 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(176..176 src has 1..176) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f73b00322f0 con 0x7f73c80b5b80 2021-02-10T11:47:09.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:09.786 7f73c77fe700 1 --2- 172.21.2.112:0/988330081 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f73ac018700 0x7f73ac01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:09.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:09.786 7f73c77fe700 1 --2- 172.21.2.112:0/988330081 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f73ac018700 0x7f73ac01abc0 crc :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:09.819 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:09.816 7f73cdd3e700 1 -- 172.21.2.112:0/988330081 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73c80b6010 con 0x7f73c80b5b80 2021-02-10T11:47:09.821 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:09.819 7f73bffff700 1 -- 172.21.2.112:0/988330081 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f73b0006ba0 con 0x7f73c80b5b80 2021-02-10T11:47:09.953 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:09.950 7f73cdd3e700 1 -- 172.21.2.112:0/988330081 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f73c80b6010 con 0x7f73c80b5b80 2021-02-10T11:47:09.953 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:09.951 7f73bffff700 1 -- 172.21.2.112:0/988330081 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v101) v1 ==== 80+0+4513 (secure 0 0 0) 0x7f73b000bd80 con 0x7f73c80b5b80 2021-02-10T11:47:09.954 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:47:09.954 INFO:teuthology.orchestra.run.gibba012.stdout:{ 2021-02-10T11:47:09.954 INFO:teuthology.orchestra.run.gibba012.stdout: "auth_dump": [ 2021-02-10T11:47:09.954 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:09.954 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "mds.a", 2021-02-10T11:47:09.954 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCKvyNgPkV4AxAAEmVNaghdccP6lWd8hvQnxw==", 2021-02-10T11:47:09.955 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:47:09.955 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:47:09.955 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:47:09.955 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:47:09.955 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:47:09.955 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:47:09.955 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:09.955 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:09.955 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "mds.b", 2021-02-10T11:47:09.955 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCKvyNguxONLhAAHIpry+R89MAwdhFJyEhVCw==", 2021-02-10T11:47:09.955 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:47:09.955 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:47:09.955 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:47:09.955 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:47:09.956 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:47:09.956 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:47:09.956 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:09.956 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:09.956 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.0", 2021-02-10T11:47:09.956 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCSvyNgXcKPLhAAP/Bj2F+06DCtPCzQDNq+cQ==", 2021-02-10T11:47:09.956 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:47:09.956 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:47:09.956 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:47:09.956 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:47:09.956 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:47:09.956 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:09.956 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:09.957 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.1", 2021-02-10T11:47:09.957 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCVvyNghfbcLhAA66PrRwR2M7o6VTF5UhU/Fg==", 2021-02-10T11:47:09.957 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:47:09.957 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:47:09.957 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:47:09.957 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:47:09.957 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:47:09.957 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:09.957 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:09.957 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.2", 2021-02-10T11:47:09.957 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCYvyNgwiPqLhAA66BVkRsgaUlGpAsQ15Nf8Q==", 2021-02-10T11:47:09.957 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:47:09.957 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:47:09.958 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:47:09.958 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:47:09.958 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:47:09.958 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:09.958 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:09.958 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.3", 2021-02-10T11:47:09.958 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCbvyNgqBZrLxAAERPe/1L3SXAEzI3d9ReHHg==", 2021-02-10T11:47:09.958 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:47:09.958 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:47:09.958 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:47:09.958 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:47:09.958 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:47:09.958 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:09.959 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:09.959 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.4", 2021-02-10T11:47:09.959 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCkvyNgDQsxIhAAHmVjVYnx20N4SDmKqvbcuA==", 2021-02-10T11:47:09.959 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:47:09.959 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:47:09.959 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:47:09.959 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:47:09.959 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:47:09.959 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:09.959 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:09.959 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.5", 2021-02-10T11:47:09.959 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCnvyNg8ITFIhAADd4aKybAV7btoGPx3kCg3A==", 2021-02-10T11:47:09.959 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:47:09.959 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:47:09.960 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:47:09.960 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:47:09.960 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:47:09.960 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:09.960 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:09.960 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.6", 2021-02-10T11:47:09.960 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCqvyNgxOhDIxAAzXixEa/nlDc1xXrg0aeLwg==", 2021-02-10T11:47:09.960 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:47:09.960 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:47:09.960 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:47:09.960 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:47:09.960 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:47:09.960 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:09.961 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:09.961 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.7", 2021-02-10T11:47:09.961 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCtvyNgxSqGIxAACxJQlwP7DpiVVCZHA3zn1A==", 2021-02-10T11:47:09.961 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:47:09.961 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:47:09.961 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:47:09.961 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:47:09.961 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:47:09.961 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:09.961 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:09.961 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.0", 2021-02-10T11:47:09.961 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCLvyNgDkNiHRAAygU8Za4PBusTqCgwAYWTiQ==", 2021-02-10T11:47:09.961 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:47:09.961 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:47:09.962 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow r", 2021-02-10T11:47:09.962 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:47:09.962 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:47:09.962 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:09.962 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:09.962 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.1", 2021-02-10T11:47:09.962 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCLvyNgxmLqMxAA1MlSDJ9LH9u4f2sL7mm8/Q==", 2021-02-10T11:47:09.962 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:47:09.962 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:47:09.962 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow r", 2021-02-10T11:47:09.962 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:47:09.962 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:47:09.962 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:09.963 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:09.963 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.2", 2021-02-10T11:47:09.963 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCMvyNgp8veDhAAniSVHYPtwwTog/pjbaRQmw==", 2021-02-10T11:47:09.963 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:47:09.963 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:47:09.963 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow r", 2021-02-10T11:47:09.963 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:47:09.963 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:47:09.963 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:09.963 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:09.963 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.3", 2021-02-10T11:47:09.963 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCMvyNgQEVoJRAApHcWdm1sldIxu+heyD4HdQ==", 2021-02-10T11:47:09.963 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:47:09.964 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:47:09.964 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow r", 2021-02-10T11:47:09.964 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:47:09.964 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:47:09.964 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:09.964 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:09.964 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.admin", 2021-02-10T11:47:09.964 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCGvyNg7jjJBRAAtot+ypUZOiTo1GbY+oVhVw==", 2021-02-10T11:47:09.964 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:47:09.964 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow *", 2021-02-10T11:47:09.964 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:47:09.964 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:47:09.964 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:47:09.965 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:47:09.965 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:09.965 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:09.965 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "mgr.x", 2021-02-10T11:47:09.965 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCJvyNgTPVwKBAAjduiBE8+RSXBv82tdPecWg==", 2021-02-10T11:47:09.965 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:47:09.965 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow *", 2021-02-10T11:47:09.965 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow profile mgr", 2021-02-10T11:47:09.965 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:47:09.965 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:47:09.965 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:09.965 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:09.965 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "mgr.y", 2021-02-10T11:47:09.965 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCJvyNgQ1tTDhAASwBssDy+Rz31ujw8fNlWow==", 2021-02-10T11:47:09.966 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:47:09.966 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow *", 2021-02-10T11:47:09.966 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow profile mgr", 2021-02-10T11:47:09.966 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:47:09.966 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:47:09.966 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:47:09.966 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:47:09.966 INFO:teuthology.orchestra.run.gibba012.stdout:} 2021-02-10T11:47:09.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:09.953 7f73cdd3e700 1 -- 172.21.2.112:0/988330081 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f73ac018700 msgr2=0x7f73ac01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:09.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:09.953 7f73cdd3e700 1 --2- 172.21.2.112:0/988330081 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f73ac018700 0x7f73ac01abc0 crc :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:09.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:09.953 7f73cdd3e700 1 -- 172.21.2.112:0/988330081 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f73c80b5b80 msgr2=0x7f73c81a2e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:09.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:09.953 7f73cdd3e700 1 --2- 172.21.2.112:0/988330081 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f73c80b5b80 0x7f73c81a2e20 secure :-1 s=READY pgs=2988 cs=0 l=1 rev1=1 rx=0x7f73b0006f10 tx=0x7f73b000b990).stop 2021-02-10T11:47:09.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:09.953 7f73cdd3e700 1 -- 172.21.2.112:0/988330081 shutdown_connections 2021-02-10T11:47:09.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:09.953 7f73cdd3e700 1 --2- 172.21.2.112:0/988330081 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f73ac018700 0x7f73ac01abc0 unknown :-1 s=CLOSED pgs=1347 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:09.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:09.953 7f73cdd3e700 1 --2- 172.21.2.112:0/988330081 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f73c80acfa0 0x7f73c81a3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:09.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:09.953 7f73cdd3e700 1 --2- 172.21.2.112:0/988330081 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f73c80b5b80 0x7f73c81a2e20 unknown :-1 s=CLOSED pgs=2988 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:09.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:09.953 7f73cdd3e700 1 --2- 172.21.2.112:0/988330081 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f73c8195650 0x7f73c80b5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:09.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:09.953 7f73cdd3e700 1 -- 172.21.2.112:0/988330081 >> 172.21.2.112:0/988330081 conn(0x7f73c80b3040 msgr2=0x7f73c80c3250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:09.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:09.953 7f73cdd3e700 1 -- 172.21.2.112:0/988330081 shutdown_connections 2021-02-10T11:47:09.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:09.953 7f73cdd3e700 1 -- 172.21.2.112:0/988330081 wait complete. 2021-02-10T11:47:09.969 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:47:10.399 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.380 7f52399dd700 1 Processor -- start 2021-02-10T11:47:10.399 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.380 7f52399dd700 1 -- start start 2021-02-10T11:47:10.399 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.380 7f52399dd700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f52340d0340 0x7f52340ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:10.399 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.380 7f52399dd700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f52340d1c70 0x7f52340efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:10.399 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.380 7f52399dd700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52340f0b10 con 0x7f52340d1870 2021-02-10T11:47:10.399 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.380 7f52399dd700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52340e5ed0 con 0x7f52340d0340 2021-02-10T11:47:10.399 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.380 7f52399dd700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52341b0be0 con 0x7f52340d1c70 2021-02-10T11:47:10.399 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.380 7f5232d73700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f52340d0340 0x7f52340ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:10.399 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.380 7f5232d73700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f52340d0340 0x7f52340ef640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:60138/0 (socket says 172.21.2.112:60138) 2021-02-10T11:47:10.400 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.381 7f5232d73700 1 -- 172.21.2.112:0/457589173 learned_addr learned my addr 172.21.2.112:0/457589173 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:47:10.400 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.381 7f5232d73700 1 -- 172.21.2.112:0/457589173 >> v2:172.21.2.120:3300/0 conn(0x7f52340d1c70 msgr2=0x7f52340efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:10.400 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.381 7f5232d73700 1 --2- 172.21.2.112:0/457589173 >> v2:172.21.2.120:3300/0 conn(0x7f52340d1c70 0x7f52340efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:10.400 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.381 7f5232d73700 1 -- 172.21.2.112:0/457589173 >> v1:172.21.2.120:6789/0 conn(0x7f52340d1870 legacy=0x7f52341b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:10.400 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.381 7f5232d73700 1 -- 172.21.2.112:0/457589173 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52340f20a0 con 0x7f52340d0340 2021-02-10T11:47:10.400 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.381 7f5232d73700 1 --2- 172.21.2.112:0/457589173 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f52340d0340 0x7f52340ef640 secure :-1 s=READY pgs=2989 cs=0 l=1 rev1=1 rx=0x7f5224006700 tx=0x7f5224009b80).ready entity=mon.0 client_cookie=5e78deb9a8eaf745 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:10.400 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.381 7f5232572700 1 -- 172.21.2.112:0/457589173 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f522400c0c0 con 0x7f52340d0340 2021-02-10T11:47:10.400 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.381 7f5232572700 1 -- 172.21.2.112:0/457589173 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f522400c3c0 con 0x7f52340d0340 2021-02-10T11:47:10.400 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.381 7f5232572700 1 -- 172.21.2.112:0/457589173 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f522400c8e0 con 0x7f52340d0340 2021-02-10T11:47:10.400 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.381 7f52399dd700 1 -- 172.21.2.112:0/457589173 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f52340d0340 msgr2=0x7f52340ef640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:10.400 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.381 7f52399dd700 1 --2- 172.21.2.112:0/457589173 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f52340d0340 0x7f52340ef640 secure :-1 s=READY pgs=2989 cs=0 l=1 rev1=1 rx=0x7f5224006700 tx=0x7f5224009b80).stop 2021-02-10T11:47:10.400 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.382 7f52399dd700 1 -- 172.21.2.112:0/457589173 shutdown_connections 2021-02-10T11:47:10.400 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.382 7f52399dd700 1 --2- 172.21.2.112:0/457589173 >> v2:172.21.2.120:3300/0 conn(0x7f52340d1c70 0x7f52340efc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:10.401 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.382 7f52399dd700 1 --2- 172.21.2.112:0/457589173 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f52340d0340 0x7f52340ef640 unknown :-1 s=CLOSED pgs=2989 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:10.401 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.382 7f52399dd700 1 -- 172.21.2.112:0/457589173 >> 172.21.2.112:0/457589173 conn(0x7f52340ce450 msgr2=0x7f52340d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:10.401 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.382 7f52399dd700 1 -- 172.21.2.112:0/457589173 shutdown_connections 2021-02-10T11:47:10.401 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.382 7f52399dd700 1 -- 172.21.2.112:0/457589173 wait complete. 2021-02-10T11:47:10.402 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.397 7f52399dd700 1 Processor -- start 2021-02-10T11:47:10.402 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.397 7f52399dd700 1 -- start start 2021-02-10T11:47:10.402 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.397 7f52399dd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f52340b4e00 0x7f52340a0fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:10.403 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.397 7f52399dd700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f52340a3a60 0x7f52340a3e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:10.403 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.397 7f52399dd700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5234094750 0x7f5234094b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:10.403 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.397 7f52399dd700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52340a4550 con 0x7f52340b4e00 2021-02-10T11:47:10.403 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.397 7f52399dd700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f523418bf30 con 0x7f5234094750 2021-02-10T11:47:10.403 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.397 7f52399dd700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f523418ba60 con 0x7f52340a3a60 2021-02-10T11:47:10.403 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.397 7f5233574700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f52340b4e00 0x7f52340a0fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:10.403 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.397 7f5233574700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f52340b4e00 0x7f52340a0fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:60144/0 (socket says 172.21.2.112:60144) 2021-02-10T11:47:10.403 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.397 7f5233574700 1 -- 172.21.2.112:0/1497173858 learned_addr learned my addr 172.21.2.112:0/1497173858 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:47:10.403 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.398 7f5232d73700 1 --2- 172.21.2.112:0/1497173858 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f52340a3a60 0x7f52340a3e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:10.403 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.398 7f5233d75700 1 --2- 172.21.2.112:0/1497173858 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5234094750 0x7f5234094b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:10.403 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.398 7f5233574700 1 -- 172.21.2.112:0/1497173858 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f52340a3a60 msgr2=0x7f52340a3e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:10.403 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.398 7f5233574700 1 --2- 172.21.2.112:0/1497173858 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f52340a3a60 0x7f52340a3e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:10.403 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.398 7f5233574700 1 -- 172.21.2.112:0/1497173858 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5234094750 msgr2=0x7f5234094b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:10.404 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.398 7f5233574700 1 --2- 172.21.2.112:0/1497173858 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5234094750 0x7f5234094b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:10.404 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.398 7f5233574700 1 -- 172.21.2.112:0/1497173858 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52340a15c0 con 0x7f52340b4e00 2021-02-10T11:47:10.404 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.398 7f5232d73700 1 --2- 172.21.2.112:0/1497173858 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f52340a3a60 0x7f52340a3e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:47:10.404 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.398 7f5233574700 1 --2- 172.21.2.112:0/1497173858 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f52340b4e00 0x7f52340a0fa0 secure :-1 s=READY pgs=2990 cs=0 l=1 rev1=1 rx=0x7f521c00b5b0 tx=0x7f521c00b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:10.404 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.398 7f522bfff700 1 -- 172.21.2.112:0/1497173858 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f521c00bd40 con 0x7f52340b4e00 2021-02-10T11:47:10.404 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.398 7f522bfff700 1 -- 172.21.2.112:0/1497173858 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f521c000ce0 con 0x7f52340b4e00 2021-02-10T11:47:10.404 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.398 7f522bfff700 1 -- 172.21.2.112:0/1497173858 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f521c0054a0 con 0x7f52340b4e00 2021-02-10T11:47:10.404 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.398 7f52399dd700 1 -- 172.21.2.112:0/1497173858 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52340a15c0 con 0x7f52340b4e00 2021-02-10T11:47:10.404 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.398 7f52399dd700 1 -- 172.21.2.112:0/1497173858 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5234095460 con 0x7f52340b4e00 2021-02-10T11:47:10.404 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.399 7f522bfff700 1 -- 172.21.2.112:0/1497173858 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f521c006030 con 0x7f52340b4e00 2021-02-10T11:47:10.404 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.399 7f522bfff700 1 --2- 172.21.2.112:0/1497173858 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f52180186c0 0x7f521801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:10.404 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.399 7f522bfff700 1 -- 172.21.2.112:0/1497173858 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(176..176 src has 1..176) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f521c0320d0 con 0x7f52340b4e00 2021-02-10T11:47:10.404 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.399 7f5232d73700 1 --2- 172.21.2.112:0/1497173858 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f52180186c0 0x7f521801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:10.405 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.400 7f5232d73700 1 --2- 172.21.2.112:0/1497173858 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f52180186c0 0x7f521801abc0 crc :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:10.433 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.430 7f52399dd700 1 -- 172.21.2.112:0/1497173858 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52340ee0e0 con 0x7f52340b4e00 2021-02-10T11:47:10.435 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.433 7f522bfff700 1 -- 172.21.2.112:0/1497173858 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f521c000940 con 0x7f52340b4e00 2021-02-10T11:47:10.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.558 7f52399dd700 1 -- 172.21.2.112:0/1497173858 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f52340ce450 con 0x7f52340b4e00 2021-02-10T11:47:10.562 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.559 7f522bfff700 1 -- 172.21.2.112:0/1497173858 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v176) v1 ==== 74+0+11154 (secure 0 0 0) 0x7f521c006030 con 0x7f52340b4e00 2021-02-10T11:47:10.562 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:47:10.562 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":176,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:47:09.307232","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":37,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":168,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:47:10.564 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.561 7f52399dd700 1 -- 172.21.2.112:0/1497173858 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f52180186c0 msgr2=0x7f521801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:10.564 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.561 7f52399dd700 1 --2- 172.21.2.112:0/1497173858 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f52180186c0 0x7f521801abc0 crc :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:10.564 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.561 7f52399dd700 1 -- 172.21.2.112:0/1497173858 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f52340b4e00 msgr2=0x7f52340a0fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:10.564 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.561 7f52399dd700 1 --2- 172.21.2.112:0/1497173858 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f52340b4e00 0x7f52340a0fa0 secure :-1 s=READY pgs=2990 cs=0 l=1 rev1=1 rx=0x7f521c00b5b0 tx=0x7f521c00b950).stop 2021-02-10T11:47:10.564 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.561 7f52399dd700 1 -- 172.21.2.112:0/1497173858 shutdown_connections 2021-02-10T11:47:10.564 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.561 7f52399dd700 1 --2- 172.21.2.112:0/1497173858 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f52180186c0 0x7f521801abc0 unknown :-1 s=CLOSED pgs=1348 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:10.564 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.561 7f52399dd700 1 --2- 172.21.2.112:0/1497173858 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5234094750 0x7f5234094b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:10.564 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.561 7f52399dd700 1 --2- 172.21.2.112:0/1497173858 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f52340a3a60 0x7f52340a3e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:10.565 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.561 7f52399dd700 1 --2- 172.21.2.112:0/1497173858 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f52340b4e00 0x7f52340a0fa0 unknown :-1 s=CLOSED pgs=2990 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:10.565 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.561 7f52399dd700 1 -- 172.21.2.112:0/1497173858 >> 172.21.2.112:0/1497173858 conn(0x7f523418cab0 msgr2=0x7f5234092c60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:10.565 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.561 7f52399dd700 1 -- 172.21.2.112:0/1497173858 shutdown_connections 2021-02-10T11:47:10.565 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.561 7f52399dd700 1 -- 172.21.2.112:0/1497173858 wait complete. 2021-02-10T11:47:10.572 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:13:57.508758', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2021-02-10T11:47:10.573 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:47:11.007 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.987 7f48d6169700 1 Processor -- start 2021-02-10T11:47:11.007 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.987 7f48d6169700 1 -- start start 2021-02-10T11:47:11.007 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.988 7f48d6169700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f48d010aff0 0x7f48d00cf750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:11.007 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.988 7f48d6169700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48d00fe1c0 con 0x7f48d00d1870 2021-02-10T11:47:11.007 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.988 7f48d6169700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48d01b0cb0 con 0x7f48d00d1c70 2021-02-10T11:47:11.008 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.988 7f48d6169700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48d0109a60 con 0x7f48d010aff0 2021-02-10T11:47:11.008 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.988 7f48cf7fe700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f48d00d1870 0x7f48d01b04c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:59612/0 (socket says 172.21.2.112:59612) 2021-02-10T11:47:11.008 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.988 7f48cf7fe700 1 -- 172.21.2.112:0/664525582 learned_addr learned my addr 172.21.2.112:0/664525582 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:47:11.008 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.988 7f48ce7fc700 1 -- 172.21.2.112:0/664525582 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2088702353 0 0) 0x7f48c00010b0 con 0x7f48d00d1870 2021-02-10T11:47:11.008 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.988 7f48ce7fc700 1 -- 172.21.2.112:0/664525582 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48bc002fe0 con 0x7f48d00d1870 2021-02-10T11:47:11.008 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.989 7f48ce7fc700 1 -- 172.21.2.112:0/664525582 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 710988716 0 0) 0x7f48c00010b0 con 0x7f48d00d1870 2021-02-10T11:47:11.008 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.989 7f48ce7fc700 1 -- 172.21.2.112:0/664525582 >> v2:172.21.2.120:3300/0 conn(0x7f48d010aff0 msgr2=0x7f48d00cf750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:11.008 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.989 7f48cffff700 1 --2- 172.21.2.112:0/664525582 >> v2:172.21.2.120:3300/0 conn(0x7f48d010aff0 0x7f48d00cf750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:11.008 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.989 7f48ce7fc700 1 --2- 172.21.2.112:0/664525582 >> v2:172.21.2.120:3300/0 conn(0x7f48d010aff0 0x7f48d00cf750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:11.008 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.989 7f48ce7fc700 1 -- 172.21.2.112:0/664525582 >> v1:172.21.2.120:6789/0 conn(0x7f48d00d1c70 legacy=0x7f48d00cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:11.008 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.989 7f48ce7fc700 1 -- 172.21.2.112:0/664525582 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48d010b3f0 con 0x7f48d00d1870 2021-02-10T11:47:11.008 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.989 7f48ce7fc700 1 -- 172.21.2.112:0/664525582 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f48c00015b0 con 0x7f48d00d1870 2021-02-10T11:47:11.008 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.989 7f48cffff700 1 --2- 172.21.2.112:0/664525582 >> v2:172.21.2.120:3300/0 conn(0x7f48d010aff0 0x7f48d00cf750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:47:11.009 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.989 7f48ce7fc700 1 -- 172.21.2.112:0/664525582 >> v1:172.21.2.112:6789/0 conn(0x7f48d00d1870 legacy=0x7f48d01b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:11.009 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.989 7f48ce7fc700 1 --2- 172.21.2.112:0/664525582 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f48bc003a70 0x7f48bc005f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:11.009 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.989 7f48ce7fc700 1 --2- 172.21.2.112:0/664525582 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f48bc0067a0 0x7f48bc008ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:11.009 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.989 7f48ce7fc700 1 --2- 172.21.2.112:0/664525582 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f48bc0095a0 0x7f48bc00bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:11.009 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.989 7f48ce7fc700 1 -- 172.21.2.112:0/664525582 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48bc00c310 con 0x7f48bc0067a0 2021-02-10T11:47:11.009 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.989 7f48ce7fc700 1 -- 172.21.2.112:0/664525582 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48bc00c530 con 0x7f48bc003a70 2021-02-10T11:47:11.009 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.989 7f48ce7fc700 1 -- 172.21.2.112:0/664525582 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48bc00c750 con 0x7f48bc0095a0 2021-02-10T11:47:11.009 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.989 7f48ceffd700 1 --2- 172.21.2.112:0/664525582 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f48bc0067a0 0x7f48bc008ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:11.009 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.989 7f48cffff700 1 --2- 172.21.2.112:0/664525582 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f48bc0095a0 0x7f48bc00bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:11.009 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.989 7f48cf7fe700 1 --2- 172.21.2.112:0/664525582 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f48bc003a70 0x7f48bc005f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:11.009 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.989 7f48ceffd700 1 -- 172.21.2.112:0/664525582 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f48bc0095a0 msgr2=0x7f48bc00bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:11.009 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.989 7f48ceffd700 1 --2- 172.21.2.112:0/664525582 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f48bc0095a0 0x7f48bc00bac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:11.009 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.989 7f48ceffd700 1 -- 172.21.2.112:0/664525582 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f48bc003a70 msgr2=0x7f48bc005f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:11.009 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.989 7f48ceffd700 1 --2- 172.21.2.112:0/664525582 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f48bc003a70 0x7f48bc005f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:11.010 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.989 7f48ceffd700 1 -- 172.21.2.112:0/664525582 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48bc00c9f0 con 0x7f48bc0067a0 2021-02-10T11:47:11.010 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.989 7f48cffff700 1 -- 172.21.2.112:0/664525582 reap_dead start 2021-02-10T11:47:11.010 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.989 7f48cf7fe700 1 --2- 172.21.2.112:0/664525582 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f48bc003a70 0x7f48bc005f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:47:11.010 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.989 7f48ceffd700 1 --2- 172.21.2.112:0/664525582 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f48bc0067a0 0x7f48bc008ca0 secure :-1 s=READY pgs=2992 cs=0 l=1 rev1=1 rx=0x7f48b8006ab0 tx=0x7f48b804daa0).ready entity=mon.0 client_cookie=2da4d6958b7539d5 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:11.010 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.989 7f48ce7fc700 1 -- 172.21.2.112:0/664525582 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f48b80500c0 con 0x7f48bc0067a0 2021-02-10T11:47:11.010 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.989 7f48ce7fc700 1 -- 172.21.2.112:0/664525582 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48b80066f0 con 0x7f48bc0067a0 2021-02-10T11:47:11.010 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.989 7f48ce7fc700 1 -- 172.21.2.112:0/664525582 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f48b80505c0 con 0x7f48bc0067a0 2021-02-10T11:47:11.010 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.989 7f48d6169700 1 -- 172.21.2.112:0/664525582 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f48bc0067a0 msgr2=0x7f48bc008ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:11.010 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.989 7f48d6169700 1 --2- 172.21.2.112:0/664525582 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f48bc0067a0 0x7f48bc008ca0 secure :-1 s=READY pgs=2992 cs=0 l=1 rev1=1 rx=0x7f48b8006ab0 tx=0x7f48b804daa0).stop 2021-02-10T11:47:11.010 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.990 7f48d6169700 1 -- 172.21.2.112:0/664525582 shutdown_connections 2021-02-10T11:47:11.010 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.990 7f48d6169700 1 --2- 172.21.2.112:0/664525582 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f48bc0067a0 0x7f48bc008ca0 unknown :-1 s=CLOSED pgs=2992 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:11.010 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.990 7f48d6169700 1 -- 172.21.2.112:0/664525582 >> 172.21.2.112:0/664525582 conn(0x7f48d00ce420 msgr2=0x7f48d00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:11.010 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.990 7f48d6169700 1 -- 172.21.2.112:0/664525582 shutdown_connections 2021-02-10T11:47:11.011 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:10.990 7f48d6169700 1 -- 172.21.2.112:0/664525582 wait complete. 2021-02-10T11:47:11.012 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:11.005 7f48d6169700 1 Processor -- start 2021-02-10T11:47:11.012 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:11.005 7f48d6169700 1 -- start start 2021-02-10T11:47:11.012 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:11.005 7f48d6169700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f48d009bcc0 0x7f48d009c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:11.012 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:11.005 7f48d6169700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f48d00a0820 0x7f48d00a0c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:11.012 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:11.005 7f48d6169700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f48d00a1240 0x7f48d00858d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:11.012 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:11.005 7f48d6169700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48d00a66b0 con 0x7f48d00a0820 2021-02-10T11:47:11.012 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:11.005 7f48d6169700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48d01a49b0 con 0x7f48d00a1240 2021-02-10T11:47:11.012 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:11.005 7f48d6169700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48d01a4b90 con 0x7f48d009bcc0 2021-02-10T11:47:11.013 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:11.005 7f48ceffd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f48d00a0820 0x7f48d00a0c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:11.013 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:11.006 7f48ceffd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f48d00a0820 0x7f48d00a0c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:60164/0 (socket says 172.21.2.112:60164) 2021-02-10T11:47:11.013 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:11.006 7f48ceffd700 1 -- 172.21.2.112:0/4209755232 learned_addr learned my addr 172.21.2.112:0/4209755232 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:47:11.013 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:11.006 7f48cffff700 1 --2- 172.21.2.112:0/4209755232 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f48d00a1240 0x7f48d00858d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:11.013 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:11.006 7f48cf7fe700 1 --2- 172.21.2.112:0/4209755232 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f48d009bcc0 0x7f48d009c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:11.013 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:11.006 7f48ceffd700 1 -- 172.21.2.112:0/4209755232 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f48d009bcc0 msgr2=0x7f48d009c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:11.013 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:11.006 7f48ceffd700 1 --2- 172.21.2.112:0/4209755232 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f48d009bcc0 0x7f48d009c0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:11.013 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:11.006 7f48ceffd700 1 -- 172.21.2.112:0/4209755232 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f48d00a1240 msgr2=0x7f48d00858d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:11.013 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:11.006 7f48ceffd700 1 --2- 172.21.2.112:0/4209755232 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f48d00a1240 0x7f48d00858d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:11.013 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:11.006 7f48ceffd700 1 -- 172.21.2.112:0/4209755232 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48d00a1640 con 0x7f48d00a0820 2021-02-10T11:47:11.013 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:11.006 7f48ceffd700 1 --2- 172.21.2.112:0/4209755232 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f48d00a0820 0x7f48d00a0c20 secure :-1 s=READY pgs=2993 cs=0 l=1 rev1=1 rx=0x7f48b80065d0 tx=0x7f48b8050970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:11.013 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:11.006 7f48c7fff700 1 -- 172.21.2.112:0/4209755232 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f48b8000de0 con 0x7f48d00a0820 2021-02-10T11:47:11.013 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:11.006 7f48c7fff700 1 -- 172.21.2.112:0/4209755232 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48b8050d60 con 0x7f48d00a0820 2021-02-10T11:47:11.013 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:11.006 7f48c7fff700 1 -- 172.21.2.112:0/4209755232 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f48b804e430 con 0x7f48d00a0820 2021-02-10T11:47:11.014 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:11.006 7f48d6169700 1 -- 172.21.2.112:0/4209755232 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48d00a1640 con 0x7f48d00a0820 2021-02-10T11:47:11.014 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:11.006 7f48d6169700 1 -- 172.21.2.112:0/4209755232 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48d00861e0 con 0x7f48d00a0820 2021-02-10T11:47:11.014 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:11.007 7f48c7fff700 1 -- 172.21.2.112:0/4209755232 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f48b804e030 con 0x7f48d00a0820 2021-02-10T11:47:11.014 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:11.007 7f48c7fff700 1 --2- 172.21.2.112:0/4209755232 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f48b40186c0 0x7f48b401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:11.014 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:11.007 7f48c7fff700 1 -- 172.21.2.112:0/4209755232 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(176..176 src has 1..176) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f48b8076490 con 0x7f48d00a0820 2021-02-10T11:47:11.014 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:11.007 7f48cf7fe700 1 --2- 172.21.2.112:0/4209755232 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f48b40186c0 0x7f48b401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:11.014 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:11.008 7f48cf7fe700 1 --2- 172.21.2.112:0/4209755232 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f48b40186c0 0x7f48b401abc0 crc :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:11.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:11.037 7f48d6169700 1 -- 172.21.2.112:0/4209755232 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48d0086a70 con 0x7f48d00a0820 2021-02-10T11:47:11.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:11.040 7f48c7fff700 1 -- 172.21.2.112:0/4209755232 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f48b8063500 con 0x7f48d00a0820 2021-02-10T11:47:11.172 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:11.169 7f48d6169700 1 -- 172.21.2.112:0/4209755232 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f48d0086a70 con 0x7f48d00a0820 2021-02-10T11:47:11.172 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:11.169 7f48c7fff700 1 -- 172.21.2.112:0/4209755232 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v176) v1 ==== 92+0+10 (secure 0 0 0) 0x7f48b8063500 con 0x7f48d00a0820 2021-02-10T11:47:11.172 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:47:11.174 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:11.171 7f48d6169700 1 -- 172.21.2.112:0/4209755232 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f48b40186c0 msgr2=0x7f48b401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:11.174 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:11.171 7f48d6169700 1 --2- 172.21.2.112:0/4209755232 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f48b40186c0 0x7f48b401abc0 crc :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:11.174 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:11.171 7f48d6169700 1 -- 172.21.2.112:0/4209755232 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f48d00a0820 msgr2=0x7f48d00a0c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:11.174 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:11.171 7f48d6169700 1 --2- 172.21.2.112:0/4209755232 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f48d00a0820 0x7f48d00a0c20 secure :-1 s=READY pgs=2993 cs=0 l=1 rev1=1 rx=0x7f48b80065d0 tx=0x7f48b8050970).stop 2021-02-10T11:47:11.174 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:11.171 7f48d6169700 1 -- 172.21.2.112:0/4209755232 shutdown_connections 2021-02-10T11:47:11.174 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:11.171 7f48d6169700 1 --2- 172.21.2.112:0/4209755232 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f48b40186c0 0x7f48b401abc0 unknown :-1 s=CLOSED pgs=1349 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:11.174 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:11.171 7f48d6169700 1 --2- 172.21.2.112:0/4209755232 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f48d00a1240 0x7f48d00858d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:11.174 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:11.171 7f48d6169700 1 --2- 172.21.2.112:0/4209755232 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f48d00a0820 0x7f48d00a0c20 unknown :-1 s=CLOSED pgs=2993 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:11.174 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:11.171 7f48d6169700 1 --2- 172.21.2.112:0/4209755232 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f48d009bcc0 0x7f48d009c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:11.175 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:11.171 7f48d6169700 1 -- 172.21.2.112:0/4209755232 >> 172.21.2.112:0/4209755232 conn(0x7f48d00bef00 msgr2=0x7f48d009d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:11.175 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:11.172 7f48d6169700 1 -- 172.21.2.112:0/4209755232 shutdown_connections 2021-02-10T11:47:11.175 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:11.172 7f48d6169700 1 -- 172.21.2.112:0/4209755232 wait complete. 2021-02-10T11:47:11.183 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --format=json-pretty osd lspools 2021-02-10T11:47:11.611 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:11.592 7f10831ad700 1 Processor -- start 2021-02-10T11:47:11.612 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:11.592 7f10831ad700 1 -- start start 2021-02-10T11:47:11.612 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:11.592 7f10831ad700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f107c0d1870 0x7f107c1b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:11.612 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:11.592 7f10831ad700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f107c0f0650 0x7f107c0cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:11.612 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:11.592 7f10831ad700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f107c0f0b10 con 0x7f107c1b0a10 2021-02-10T11:47:11.612 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:11.592 7f10831ad700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f107c0e5ed0 con 0x7f107c0d1870 2021-02-10T11:47:11.612 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:11.592 7f10831ad700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f107c0d1c70 con 0x7f107c0f0650 2021-02-10T11:47:11.612 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:11.593 7f1080fc9700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f107c0d1870 0x7f107c1b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:11.612 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:11.593 7f1080fc9700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f107c0d1870 0x7f107c1b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:60172/0 (socket says 172.21.2.112:60172) 2021-02-10T11:47:11.612 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:11.593 7f1080fc9700 1 -- 172.21.2.112:0/1536638069 learned_addr learned my addr 172.21.2.112:0/1536638069 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:47:11.612 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:11.593 7f10817ca700 1 --2- 172.21.2.112:0/1536638069 >> v2:172.21.2.120:3300/0 conn(0x7f107c0f0650 0x7f107c0cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:11.613 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:11.593 7f1080fc9700 1 -- 172.21.2.112:0/1536638069 >> v2:172.21.2.120:3300/0 conn(0x7f107c0f0650 msgr2=0x7f107c0cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:11.613 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:11.593 7f1080fc9700 1 --2- 172.21.2.112:0/1536638069 >> v2:172.21.2.120:3300/0 conn(0x7f107c0f0650 0x7f107c0cfb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:11.613 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:11.593 7f1080fc9700 1 -- 172.21.2.112:0/1536638069 >> v1:172.21.2.120:6789/0 conn(0x7f107c1b0a10 legacy=0x7f107c0cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:11.613 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:11.593 7f1080fc9700 1 -- 172.21.2.112:0/1536638069 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f107c0d1e90 con 0x7f107c0d1870 2021-02-10T11:47:11.613 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:11.593 7f1080fc9700 1 --2- 172.21.2.112:0/1536638069 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f107c0d1870 0x7f107c1b03f0 secure :-1 s=READY pgs=2994 cs=0 l=1 rev1=1 rx=0x7f1064006700 tx=0x7f1064009b00).ready entity=mon.0 client_cookie=ae448ded74744f19 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:11.613 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:11.593 7f107b7fe700 1 -- 172.21.2.112:0/1536638069 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f106400c0c0 con 0x7f107c0d1870 2021-02-10T11:47:11.613 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:11.593 7f107b7fe700 1 -- 172.21.2.112:0/1536638069 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f106400c400 con 0x7f107c0d1870 2021-02-10T11:47:11.613 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:11.593 7f107b7fe700 1 -- 172.21.2.112:0/1536638069 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f106400c920 con 0x7f107c0d1870 2021-02-10T11:47:11.613 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:11.593 7f10831ad700 1 -- 172.21.2.112:0/1536638069 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f107c0d1870 msgr2=0x7f107c1b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:11.613 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:11.593 7f10831ad700 1 --2- 172.21.2.112:0/1536638069 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f107c0d1870 0x7f107c1b03f0 secure :-1 s=READY pgs=2994 cs=0 l=1 rev1=1 rx=0x7f1064006700 tx=0x7f1064009b00).stop 2021-02-10T11:47:11.613 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:11.594 7f10831ad700 1 -- 172.21.2.112:0/1536638069 shutdown_connections 2021-02-10T11:47:11.613 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:11.594 7f10831ad700 1 --2- 172.21.2.112:0/1536638069 >> v2:172.21.2.120:3300/0 conn(0x7f107c0f0650 0x7f107c0cfb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:11.613 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:11.594 7f10831ad700 1 --2- 172.21.2.112:0/1536638069 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f107c0d1870 0x7f107c1b03f0 unknown :-1 s=CLOSED pgs=2994 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:11.614 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:11.594 7f10831ad700 1 -- 172.21.2.112:0/1536638069 >> 172.21.2.112:0/1536638069 conn(0x7f107c0ce450 msgr2=0x7f107c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:11.614 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:11.594 7f10831ad700 1 -- 172.21.2.112:0/1536638069 shutdown_connections 2021-02-10T11:47:11.614 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:11.594 7f10831ad700 1 -- 172.21.2.112:0/1536638069 wait complete. 2021-02-10T11:47:11.614 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:11.609 7f10831ad700 1 Processor -- start 2021-02-10T11:47:11.614 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:11.609 7f10831ad700 1 -- start start 2021-02-10T11:47:11.614 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:11.609 7f10831ad700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f107c190ae0 0x7f107c090c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:11.614 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:11.609 7f10831ad700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f107c0912a0 0x7f107c0ecb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:11.614 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:11.609 7f10831ad700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f107c0ed130 0x7f107c0ed530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:11.614 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:11.609 7f10831ad700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f107c08e460 con 0x7f107c0ed130 2021-02-10T11:47:11.614 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:11.609 7f10831ad700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f107c1880e0 con 0x7f107c190ae0 2021-02-10T11:47:11.614 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:11.609 7f10831ad700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f107c1882c0 con 0x7f107c0912a0 2021-02-10T11:47:11.616 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:11.609 7f10817ca700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f107c0ed130 0x7f107c0ed530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:11.616 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:11.609 7f10817ca700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f107c0ed130 0x7f107c0ed530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:60176/0 (socket says 172.21.2.112:60176) 2021-02-10T11:47:11.616 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:11.609 7f10817ca700 1 -- 172.21.2.112:0/1279530664 learned_addr learned my addr 172.21.2.112:0/1279530664 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:47:11.616 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:11.609 7f1080fc9700 1 --2- 172.21.2.112:0/1279530664 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f107c190ae0 0x7f107c090c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:11.616 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:11.609 7f107bfff700 1 --2- 172.21.2.112:0/1279530664 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f107c0912a0 0x7f107c0ecb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:11.616 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:11.610 7f10817ca700 1 -- 172.21.2.112:0/1279530664 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f107c0912a0 msgr2=0x7f107c0ecb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:11.616 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:11.610 7f10817ca700 1 --2- 172.21.2.112:0/1279530664 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f107c0912a0 0x7f107c0ecb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:11.616 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:11.610 7f10817ca700 1 -- 172.21.2.112:0/1279530664 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f107c190ae0 msgr2=0x7f107c090c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:11.616 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:11.610 7f10817ca700 1 --2- 172.21.2.112:0/1279530664 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f107c190ae0 0x7f107c090c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:11.616 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:11.610 7f10817ca700 1 -- 172.21.2.112:0/1279530664 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f107c1884e0 con 0x7f107c0ed130 2021-02-10T11:47:11.616 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:11.610 7f10817ca700 1 --2- 172.21.2.112:0/1279530664 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f107c0ed130 0x7f107c0ed530 secure :-1 s=READY pgs=2995 cs=0 l=1 rev1=1 rx=0x7f1074009980 tx=0x7f107400bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:11.617 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:11.610 7f10797fa700 1 -- 172.21.2.112:0/1279530664 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f107400c2f0 con 0x7f107c0ed130 2021-02-10T11:47:11.617 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:11.610 7f10797fa700 1 -- 172.21.2.112:0/1279530664 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f107400c530 con 0x7f107c0ed130 2021-02-10T11:47:11.617 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:11.610 7f10797fa700 1 -- 172.21.2.112:0/1279530664 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f107400ca10 con 0x7f107c0ed130 2021-02-10T11:47:11.617 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:11.610 7f10831ad700 1 -- 172.21.2.112:0/1279530664 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f107c1884e0 con 0x7f107c0ed130 2021-02-10T11:47:11.617 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:11.610 7f10831ad700 1 -- 172.21.2.112:0/1279530664 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f107c188a70 con 0x7f107c0ed130 2021-02-10T11:47:11.617 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:11.611 7f10797fa700 1 -- 172.21.2.112:0/1279530664 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f10740011f0 con 0x7f107c0ed130 2021-02-10T11:47:11.617 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:11.611 7f10797fa700 1 --2- 172.21.2.112:0/1279530664 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f10600186c0 0x7f106001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:11.617 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:11.611 7f10797fa700 1 -- 172.21.2.112:0/1279530664 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(176..176 src has 1..176) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f10740325f0 con 0x7f107c0ed130 2021-02-10T11:47:11.617 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:11.611 7f1080fc9700 1 --2- 172.21.2.112:0/1279530664 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f10600186c0 0x7f106001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:11.617 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:11.611 7f1080fc9700 1 --2- 172.21.2.112:0/1279530664 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f10600186c0 0x7f106001abc0 crc :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:11.645 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:11.642 7f10831ad700 1 -- 172.21.2.112:0/1279530664 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f107c0ebc00 con 0x7f107c0ed130 2021-02-10T11:47:11.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:11.645 7f10797fa700 1 -- 172.21.2.112:0/1279530664 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f107400be10 con 0x7f107c0ed130 2021-02-10T11:47:11.770 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:11.767 7f10831ad700 1 -- 172.21.2.112:0/1279530664 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f107c0f0b10 con 0x7f107c0ed130 2021-02-10T11:47:11.770 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:11.768 7f10797fa700 1 -- 172.21.2.112:0/1279530664 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v176) v1 ==== 84+0+64 (secure 0 0 0) 0x7f10740011f0 con 0x7f107c0ed130 2021-02-10T11:47:11.771 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:47:11.771 INFO:teuthology.orchestra.run.gibba012.stdout:[ 2021-02-10T11:47:11.771 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:11.771 INFO:teuthology.orchestra.run.gibba012.stdout: "poolnum": 1, 2021-02-10T11:47:11.771 INFO:teuthology.orchestra.run.gibba012.stdout: "poolname": "rbd" 2021-02-10T11:47:11.771 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:47:11.771 INFO:teuthology.orchestra.run.gibba012.stdout:] 2021-02-10T11:47:11.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:11.769 7f10831ad700 1 -- 172.21.2.112:0/1279530664 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f10600186c0 msgr2=0x7f106001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:11.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:11.769 7f10831ad700 1 --2- 172.21.2.112:0/1279530664 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f10600186c0 0x7f106001abc0 crc :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:11.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:11.769 7f10831ad700 1 -- 172.21.2.112:0/1279530664 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f107c0ed130 msgr2=0x7f107c0ed530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:11.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:11.769 7f10831ad700 1 --2- 172.21.2.112:0/1279530664 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f107c0ed130 0x7f107c0ed530 secure :-1 s=READY pgs=2995 cs=0 l=1 rev1=1 rx=0x7f1074009980 tx=0x7f107400bcd0).stop 2021-02-10T11:47:11.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:11.770 7f10831ad700 1 -- 172.21.2.112:0/1279530664 shutdown_connections 2021-02-10T11:47:11.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:11.770 7f10831ad700 1 --2- 172.21.2.112:0/1279530664 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f10600186c0 0x7f106001abc0 unknown :-1 s=CLOSED pgs=1350 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:11.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:11.770 7f10831ad700 1 --2- 172.21.2.112:0/1279530664 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f107c0ed130 0x7f107c0ed530 unknown :-1 s=CLOSED pgs=2995 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:11.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:11.770 7f10831ad700 1 --2- 172.21.2.112:0/1279530664 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f107c0912a0 0x7f107c0ecb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:11.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:11.770 7f10831ad700 1 --2- 172.21.2.112:0/1279530664 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f107c190ae0 0x7f107c090c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:11.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:11.770 7f10831ad700 1 -- 172.21.2.112:0/1279530664 >> 172.21.2.112:0/1279530664 conn(0x7f107c0b7e70 msgr2=0x7f107c0c3850 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:11.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:11.770 7f10831ad700 1 -- 172.21.2.112:0/1279530664 shutdown_connections 2021-02-10T11:47:11.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:11.770 7f10831ad700 1 -- 172.21.2.112:0/1279530664 wait complete. 2021-02-10T11:47:11.781 INFO:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2021-02-10T11:47:11.781 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_metadata 8 2021-02-10T11:47:12.214 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:12.194 7f834af66700 1 Processor -- start 2021-02-10T11:47:12.214 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:12.194 7f834af66700 1 -- start start 2021-02-10T11:47:12.214 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:12.195 7f834af66700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f83440d1870 0x7f83441b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:12.214 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:12.195 7f834af66700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f83441b0ae0 0x7f8344108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:12.214 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:12.195 7f834af66700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83440fe1c0 con 0x7f83440d1c70 2021-02-10T11:47:12.214 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:12.195 7f834af66700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8344109a60 con 0x7f83440d1870 2021-02-10T11:47:12.214 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:12.195 7f834af66700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8344109750 con 0x7f83441b0ae0 2021-02-10T11:47:12.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:12.195 7f8348d82700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f83440d1870 0x7f83441b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:12.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:12.195 7f8348d82700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f83440d1870 0x7f83441b04c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:60186/0 (socket says 172.21.2.112:60186) 2021-02-10T11:47:12.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:12.195 7f8348d82700 1 -- 172.21.2.112:0/4040021481 learned_addr learned my addr 172.21.2.112:0/4040021481 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:47:12.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:12.195 7f8348d82700 1 -- 172.21.2.112:0/4040021481 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f83441b0ae0 msgr2=0x7f8344108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:12.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:12.195 7f8348d82700 1 --2- 172.21.2.112:0/4040021481 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f83441b0ae0 0x7f8344108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:12.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:12.195 7f8348d82700 1 -- 172.21.2.112:0/4040021481 >> v1:172.21.2.120:6789/0 conn(0x7f83440d1c70 legacy=0x7f8344108f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:12.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:12.195 7f8348d82700 1 -- 172.21.2.112:0/4040021481 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8344088070 con 0x7f83440d1870 2021-02-10T11:47:12.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:12.196 7f8348d82700 1 --2- 172.21.2.112:0/4040021481 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f83440d1870 0x7f83441b04c0 secure :-1 s=READY pgs=2996 cs=0 l=1 rev1=1 rx=0x7f832c006700 tx=0x7f832c009b80).ready entity=mon.0 client_cookie=d9565b1883974e01 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:12.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:12.196 7f83437fe700 1 -- 172.21.2.112:0/4040021481 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f832c00c0c0 con 0x7f83440d1870 2021-02-10T11:47:12.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:12.196 7f83437fe700 1 -- 172.21.2.112:0/4040021481 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f832c00c440 con 0x7f83440d1870 2021-02-10T11:47:12.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:12.196 7f83437fe700 1 -- 172.21.2.112:0/4040021481 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f832c00c920 con 0x7f83440d1870 2021-02-10T11:47:12.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:12.196 7f834af66700 1 -- 172.21.2.112:0/4040021481 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f83440d1870 msgr2=0x7f83441b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:12.216 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:12.196 7f834af66700 1 --2- 172.21.2.112:0/4040021481 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f83440d1870 0x7f83441b04c0 secure :-1 s=READY pgs=2996 cs=0 l=1 rev1=1 rx=0x7f832c006700 tx=0x7f832c009b80).stop 2021-02-10T11:47:12.216 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:12.196 7f834af66700 1 -- 172.21.2.112:0/4040021481 shutdown_connections 2021-02-10T11:47:12.216 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:12.196 7f834af66700 1 --2- 172.21.2.112:0/4040021481 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f83441b0ae0 0x7f8344108940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:12.216 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:12.196 7f834af66700 1 --2- 172.21.2.112:0/4040021481 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f83440d1870 0x7f83441b04c0 unknown :-1 s=CLOSED pgs=2996 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:12.216 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:12.196 7f834af66700 1 -- 172.21.2.112:0/4040021481 >> 172.21.2.112:0/4040021481 conn(0x7f83440ce420 msgr2=0x7f83440d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:12.216 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:12.197 7f834af66700 1 -- 172.21.2.112:0/4040021481 shutdown_connections 2021-02-10T11:47:12.216 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:12.197 7f834af66700 1 -- 172.21.2.112:0/4040021481 wait complete. 2021-02-10T11:47:12.218 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:12.212 7f834af66700 1 Processor -- start 2021-02-10T11:47:12.218 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:12.212 7f834af66700 1 -- start start 2021-02-10T11:47:12.218 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:12.212 7f834af66700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f83440c3820 0x7f8344084cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:12.218 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:12.212 7f834af66700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f83440a0290 0x7f83440852d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:12.218 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:12.212 7f834af66700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8344086270 0x7f8344086670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:12.218 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:12.212 7f834af66700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83440c0700 con 0x7f8344086270 2021-02-10T11:47:12.218 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:12.212 7f834af66700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83440858f0 con 0x7f83440c3820 2021-02-10T11:47:12.218 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:12.212 7f834af66700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8344085b10 con 0x7f83440a0290 2021-02-10T11:47:12.218 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:12.213 7f8349583700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8344086270 0x7f8344086670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:12.218 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:12.213 7f8349583700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8344086270 0x7f8344086670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:60190/0 (socket says 172.21.2.112:60190) 2021-02-10T11:47:12.219 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:12.213 7f8349583700 1 -- 172.21.2.112:0/1412509287 learned_addr learned my addr 172.21.2.112:0/1412509287 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:47:12.219 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:12.213 7f8348d82700 1 --2- 172.21.2.112:0/1412509287 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f83440c3820 0x7f8344084cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:12.219 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:12.213 7f8343fff700 1 --2- 172.21.2.112:0/1412509287 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f83440a0290 0x7f83440852d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:12.219 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:12.213 7f8349583700 1 -- 172.21.2.112:0/1412509287 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f83440a0290 msgr2=0x7f83440852d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:12.219 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:12.213 7f8349583700 1 --2- 172.21.2.112:0/1412509287 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f83440a0290 0x7f83440852d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:12.219 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:12.213 7f8349583700 1 -- 172.21.2.112:0/1412509287 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f83440c3820 msgr2=0x7f8344084cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:12.219 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:12.213 7f8349583700 1 --2- 172.21.2.112:0/1412509287 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f83440c3820 0x7f8344084cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:12.219 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:12.213 7f8349583700 1 -- 172.21.2.112:0/1412509287 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8344086c90 con 0x7f8344086270 2021-02-10T11:47:12.219 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:12.213 7f8349583700 1 --2- 172.21.2.112:0/1412509287 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8344086270 0x7f8344086670 secure :-1 s=READY pgs=2997 cs=0 l=1 rev1=1 rx=0x7f833c006ed0 tx=0x7f833c00b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:12.219 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:12.213 7f83417fa700 1 -- 172.21.2.112:0/1412509287 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f833c00bd50 con 0x7f8344086270 2021-02-10T11:47:12.219 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:12.213 7f83417fa700 1 -- 172.21.2.112:0/1412509287 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f833c006020 con 0x7f8344086270 2021-02-10T11:47:12.219 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:12.213 7f83417fa700 1 -- 172.21.2.112:0/1412509287 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f833c0054b0 con 0x7f8344086270 2021-02-10T11:47:12.220 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:12.213 7f834af66700 1 -- 172.21.2.112:0/1412509287 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8344086c90 con 0x7f8344086270 2021-02-10T11:47:12.220 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:12.213 7f834af66700 1 -- 172.21.2.112:0/1412509287 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83441a2f00 con 0x7f8344086270 2021-02-10T11:47:12.220 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:12.214 7f83417fa700 1 -- 172.21.2.112:0/1412509287 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f833c006200 con 0x7f8344086270 2021-02-10T11:47:12.220 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:12.214 7f83417fa700 1 --2- 172.21.2.112:0/1412509287 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8328018730 0x7f832801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:12.220 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:12.214 7f83417fa700 1 -- 172.21.2.112:0/1412509287 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(176..176 src has 1..176) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f833c032300 con 0x7f8344086270 2021-02-10T11:47:12.220 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:12.214 7f8348d82700 1 --2- 172.21.2.112:0/1412509287 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8328018730 0x7f832801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:12.220 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:12.215 7f8348d82700 1 --2- 172.21.2.112:0/1412509287 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8328018730 0x7f832801ac10 crc :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:12.247 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:12.244 7f834af66700 1 -- 172.21.2.112:0/1412509287 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8344082a70 con 0x7f8344086270 2021-02-10T11:47:12.250 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:12.247 7f83417fa700 1 -- 172.21.2.112:0/1412509287 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f833c000bc0 con 0x7f8344086270 2021-02-10T11:47:12.379 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:12.377 7f834af66700 1 -- 172.21.2.112:0/1412509287 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"} v 0) v1 -- 0x7f8344082a70 con 0x7f8344086270 2021-02-10T11:47:12.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:12.782 7f83417fa700 1 -- 172.21.2.112:0/1412509287 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"}]=0 pool 'cephfs_metadata' created v177) v1 ==== 133+0+0 (secure 0 0 0) 0x7f833c006020 con 0x7f8344086270 2021-02-10T11:47:12.795 INFO:teuthology.orchestra.run.gibba012.stderr:pool 'cephfs_metadata' created 2021-02-10T11:47:12.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:12.794 7f834af66700 1 -- 172.21.2.112:0/1412509287 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8328018730 msgr2=0x7f832801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:12.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:12.794 7f834af66700 1 --2- 172.21.2.112:0/1412509287 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8328018730 0x7f832801ac10 crc :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:12.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:12.794 7f834af66700 1 -- 172.21.2.112:0/1412509287 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8344086270 msgr2=0x7f8344086670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:12.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:12.794 7f834af66700 1 --2- 172.21.2.112:0/1412509287 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8344086270 0x7f8344086670 secure :-1 s=READY pgs=2997 cs=0 l=1 rev1=1 rx=0x7f833c006ed0 tx=0x7f833c00b960).stop 2021-02-10T11:47:12.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:12.795 7f834af66700 1 -- 172.21.2.112:0/1412509287 shutdown_connections 2021-02-10T11:47:12.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:12.795 7f834af66700 1 --2- 172.21.2.112:0/1412509287 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8328018730 0x7f832801ac10 unknown :-1 s=CLOSED pgs=1351 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:12.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:12.795 7f834af66700 1 --2- 172.21.2.112:0/1412509287 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8344086270 0x7f8344086670 unknown :-1 s=CLOSED pgs=2997 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:12.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:12.795 7f834af66700 1 --2- 172.21.2.112:0/1412509287 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f83440a0290 0x7f83440852d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:12.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:12.795 7f834af66700 1 --2- 172.21.2.112:0/1412509287 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f83440c3820 0x7f8344084cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:12.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:12.795 7f834af66700 1 -- 172.21.2.112:0/1412509287 >> 172.21.2.112:0/1412509287 conn(0x7f83440872c0 msgr2=0x7f8344095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:12.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:12.795 7f834af66700 1 -- 172.21.2.112:0/1412509287 shutdown_connections 2021-02-10T11:47:12.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:12.795 7f834af66700 1 -- 172.21.2.112:0/1412509287 wait complete. 2021-02-10T11:47:12.805 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_data 8 2021-02-10T11:47:13.238 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:13.219 7f414588b700 1 Processor -- start 2021-02-10T11:47:13.238 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:13.219 7f414588b700 1 -- start start 2021-02-10T11:47:13.238 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:13.219 7f414588b700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f41400d1c70 0x7f41400cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:13.238 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:13.219 7f414588b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f41400cf580 0x7f41400cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:13.238 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:13.219 7f414588b700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41400fe1c0 con 0x7f41400d1870 2021-02-10T11:47:13.238 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:13.219 7f414588b700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41401b0cb0 con 0x7f41400d1c70 2021-02-10T11:47:13.238 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:13.219 7f414588b700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4140109a60 con 0x7f41400cf580 2021-02-10T11:47:13.238 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:13.219 7f413ed73700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f41400d1c70 0x7f41400cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:13.238 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:13.219 7f413f574700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f41400d1870 0x7f41401b04c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:59660/0 (socket says 172.21.2.112:59660) 2021-02-10T11:47:13.238 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:13.219 7f413f574700 1 -- 172.21.2.112:0/806569108 learned_addr learned my addr 172.21.2.112:0/806569108 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:47:13.239 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:13.220 7f413ed73700 1 -- 172.21.2.112:0/806569108 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f41400cf580 msgr2=0x7f41400cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:13.239 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:13.220 7f413ed73700 1 --2- 172.21.2.112:0/806569108 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f41400cf580 0x7f41400cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:13.239 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:13.220 7f413ed73700 1 -- 172.21.2.112:0/806569108 >> v1:172.21.2.112:6789/0 conn(0x7f41400d1870 legacy=0x7f41401b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:13.239 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:13.220 7f413ed73700 1 -- 172.21.2.112:0/806569108 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4140088070 con 0x7f41400d1c70 2021-02-10T11:47:13.239 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:13.220 7f413ed73700 1 --2- 172.21.2.112:0/806569108 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f41400d1c70 0x7f41400cef60 secure :-1 s=READY pgs=2999 cs=0 l=1 rev1=1 rx=0x7f4130006650 tx=0x7f4130009a00).ready entity=mon.0 client_cookie=44d13085f04de5ec server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:13.239 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:13.220 7f413e572700 1 -- 172.21.2.112:0/806569108 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4130009df0 con 0x7f41400d1c70 2021-02-10T11:47:13.239 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:13.220 7f413e572700 1 -- 172.21.2.112:0/806569108 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41300009b0 con 0x7f41400d1c70 2021-02-10T11:47:13.239 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:13.220 7f413e572700 1 -- 172.21.2.112:0/806569108 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f413000c2b0 con 0x7f41400d1c70 2021-02-10T11:47:13.239 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:13.220 7f414588b700 1 -- 172.21.2.112:0/806569108 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f41400d1c70 msgr2=0x7f41400cef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:13.239 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:13.220 7f414588b700 1 --2- 172.21.2.112:0/806569108 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f41400d1c70 0x7f41400cef60 secure :-1 s=READY pgs=2999 cs=0 l=1 rev1=1 rx=0x7f4130006650 tx=0x7f4130009a00).stop 2021-02-10T11:47:13.239 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:13.221 7f414588b700 1 -- 172.21.2.112:0/806569108 shutdown_connections 2021-02-10T11:47:13.239 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:13.221 7f414588b700 1 --2- 172.21.2.112:0/806569108 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f41400cf580 0x7f41400cf980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:13.239 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:13.221 7f414588b700 1 --2- 172.21.2.112:0/806569108 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f41400d1c70 0x7f41400cef60 unknown :-1 s=CLOSED pgs=2999 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:13.240 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:13.221 7f414588b700 1 -- 172.21.2.112:0/806569108 >> 172.21.2.112:0/806569108 conn(0x7f41400ce420 msgr2=0x7f41400d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:13.240 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:13.221 7f414588b700 1 -- 172.21.2.112:0/806569108 shutdown_connections 2021-02-10T11:47:13.240 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:13.221 7f414588b700 1 -- 172.21.2.112:0/806569108 wait complete. 2021-02-10T11:47:13.241 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:13.235 7f414588b700 1 Processor -- start 2021-02-10T11:47:13.242 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:13.236 7f414588b700 1 -- start start 2021-02-10T11:47:13.242 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:13.236 7f414588b700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f41400c3820 0x7f41400a7950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:13.242 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:13.236 7f414588b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f41401a44c0 0x7f41401a48c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:13.242 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:13.236 7f414588b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f41400863d0 0x7f41400867d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:13.242 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:13.236 7f414588b700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4140085ed0 con 0x7f41400863d0 2021-02-10T11:47:13.242 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:13.236 7f414588b700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41400c0740 con 0x7f41400c3820 2021-02-10T11:47:13.242 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:13.236 7f414588b700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4140086df0 con 0x7f41401a44c0 2021-02-10T11:47:13.242 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:13.236 7f413fd75700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f41400863d0 0x7f41400867d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:13.242 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:13.236 7f413fd75700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f41400863d0 0x7f41400867d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:60204/0 (socket says 172.21.2.112:60204) 2021-02-10T11:47:13.242 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:13.236 7f413fd75700 1 -- 172.21.2.112:0/3030619136 learned_addr learned my addr 172.21.2.112:0/3030619136 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:47:13.242 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:13.236 7f413fd75700 1 -- 172.21.2.112:0/3030619136 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f41401a44c0 msgr2=0x7f41401a48c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:13.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:13.236 7f413fd75700 1 --2- 172.21.2.112:0/3030619136 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f41401a44c0 0x7f41401a48c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:13.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:13.236 7f413fd75700 1 -- 172.21.2.112:0/3030619136 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f41400c3820 msgr2=0x7f41400a7950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:13.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:13.236 7f413fd75700 1 --2- 172.21.2.112:0/3030619136 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f41400c3820 0x7f41400a7950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:13.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:13.236 7f413fd75700 1 -- 172.21.2.112:0/3030619136 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4140087010 con 0x7f41400863d0 2021-02-10T11:47:13.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:13.237 7f413fd75700 1 --2- 172.21.2.112:0/3030619136 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f41400863d0 0x7f41400867d0 secure :-1 s=READY pgs=3000 cs=0 l=1 rev1=1 rx=0x7f4138009980 tx=0x7f413800bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:13.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:13.237 7f4137fff700 1 -- 172.21.2.112:0/3030619136 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4138005870 con 0x7f41400863d0 2021-02-10T11:47:13.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:13.237 7f4137fff700 1 -- 172.21.2.112:0/3030619136 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4138005b30 con 0x7f41400863d0 2021-02-10T11:47:13.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:13.237 7f4137fff700 1 -- 172.21.2.112:0/3030619136 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4138006030 con 0x7f41400863d0 2021-02-10T11:47:13.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:13.237 7f414588b700 1 -- 172.21.2.112:0/3030619136 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4140087010 con 0x7f41400863d0 2021-02-10T11:47:13.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:13.237 7f414588b700 1 -- 172.21.2.112:0/3030619136 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4140083000 con 0x7f41400863d0 2021-02-10T11:47:13.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:13.237 7f4137fff700 1 -- 172.21.2.112:0/3030619136 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4138005d10 con 0x7f41400863d0 2021-02-10T11:47:13.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:13.237 7f4137fff700 1 --2- 172.21.2.112:0/3030619136 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f41240186c0 0x7f412401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:13.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:13.237 7f4137fff700 1 -- 172.21.2.112:0/3030619136 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(177..177 src has 1..177) v4 ==== 5107+0+0 (secure 0 0 0) 0x7f41380325d0 con 0x7f41400863d0 2021-02-10T11:47:13.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:13.238 7f413f574700 1 --2- 172.21.2.112:0/3030619136 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f41240186c0 0x7f412401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:13.244 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:13.238 7f413f574700 1 --2- 172.21.2.112:0/3030619136 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f41240186c0 0x7f412401abc0 crc :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:13.272 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:13.269 7f414588b700 1 -- 172.21.2.112:0/3030619136 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41400fe1c0 con 0x7f41400863d0 2021-02-10T11:47:13.275 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:13.272 7f4137fff700 1 -- 172.21.2.112:0/3030619136 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4138028ef0 con 0x7f41400863d0 2021-02-10T11:47:13.404 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:13.401 7f414588b700 1 -- 172.21.2.112:0/3030619136 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"} v 0) v1 -- 0x7f41400fe1c0 con 0x7f41400863d0 2021-02-10T11:47:13.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:13.789 7f4137fff700 1 -- 172.21.2.112:0/3030619136 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"}]=0 pool 'cephfs_data' created v178) v1 ==== 125+0+0 (secure 0 0 0) 0x7f4138006f20 con 0x7f41400863d0 2021-02-10T11:47:13.820 INFO:teuthology.orchestra.run.gibba012.stderr:pool 'cephfs_data' created 2021-02-10T11:47:13.822 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:13.819 7f414588b700 1 -- 172.21.2.112:0/3030619136 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f41240186c0 msgr2=0x7f412401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:13.822 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:13.819 7f414588b700 1 --2- 172.21.2.112:0/3030619136 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f41240186c0 0x7f412401abc0 crc :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:13.822 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:13.819 7f414588b700 1 -- 172.21.2.112:0/3030619136 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f41400863d0 msgr2=0x7f41400867d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:13.822 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:13.819 7f414588b700 1 --2- 172.21.2.112:0/3030619136 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f41400863d0 0x7f41400867d0 secure :-1 s=READY pgs=3000 cs=0 l=1 rev1=1 rx=0x7f4138009980 tx=0x7f413800bcd0).stop 2021-02-10T11:47:13.822 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:13.819 7f414588b700 1 -- 172.21.2.112:0/3030619136 shutdown_connections 2021-02-10T11:47:13.822 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:13.819 7f414588b700 1 --2- 172.21.2.112:0/3030619136 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f41240186c0 0x7f412401abc0 unknown :-1 s=CLOSED pgs=1352 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:13.822 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:13.819 7f414588b700 1 --2- 172.21.2.112:0/3030619136 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f41400863d0 0x7f41400867d0 unknown :-1 s=CLOSED pgs=3000 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:13.822 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:13.819 7f414588b700 1 --2- 172.21.2.112:0/3030619136 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f41401a44c0 0x7f41401a48c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:13.822 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:13.819 7f414588b700 1 --2- 172.21.2.112:0/3030619136 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f41400c3820 0x7f41400a7950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:13.822 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:13.819 7f414588b700 1 -- 172.21.2.112:0/3030619136 >> 172.21.2.112:0/3030619136 conn(0x7f41400872c0 msgr2=0x7f4140095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:13.823 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:13.819 7f414588b700 1 -- 172.21.2.112:0/3030619136 shutdown_connections 2021-02-10T11:47:13.823 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:13.819 7f414588b700 1 -- 172.21.2.112:0/3030619136 wait complete. 2021-02-10T11:47:13.830 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2021-02-10T11:47:14.262 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.243 7faa23b00700 1 Processor -- start 2021-02-10T11:47:14.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.243 7faa23b00700 1 -- start start 2021-02-10T11:47:14.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.244 7faa23b00700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7faa1c0d1880 0x7faa1c1b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:14.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.244 7faa23b00700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7faa1c0cff80 0x7faa1c109140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:14.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.244 7faa23b00700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa1c109a70 con 0x7faa1c1b0af0 2021-02-10T11:47:14.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.244 7faa23b00700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa1c109760 con 0x7faa1c0cff80 2021-02-10T11:47:14.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.244 7faa23b00700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa1c088070 con 0x7faa1c0d1880 2021-02-10T11:47:14.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.244 7faa2211d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7faa1c0cff80 0x7faa1c109140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:14.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.244 7faa2211d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7faa1c0cff80 0x7faa1c109140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:60216/0 (socket says 172.21.2.112:60216) 2021-02-10T11:47:14.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.244 7faa2111b700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7faa1c1b0af0 0x7faa1c108950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:59676/0 (socket says 172.21.2.112:59676) 2021-02-10T11:47:14.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.244 7faa2211d700 1 -- 172.21.2.112:0/366570986 learned_addr learned my addr 172.21.2.112:0/366570986 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:47:14.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.245 7faa2211d700 1 -- 172.21.2.112:0/366570986 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7faa1c0d1880 msgr2=0x7faa1c1b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:14.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.245 7faa2211d700 1 --2- 172.21.2.112:0/366570986 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7faa1c0d1880 0x7faa1c1b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:14.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.245 7faa2211d700 1 -- 172.21.2.112:0/366570986 >> v1:172.21.2.112:6789/0 conn(0x7faa1c1b0af0 legacy=0x7faa1c108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:14.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.245 7faa2211d700 1 -- 172.21.2.112:0/366570986 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa1c088260 con 0x7faa1c0cff80 2021-02-10T11:47:14.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.245 7faa2211d700 1 --2- 172.21.2.112:0/366570986 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faa1c0cff80 0x7faa1c109140 secure :-1 s=READY pgs=3002 cs=0 l=1 rev1=1 rx=0x7faa14006b80 tx=0x7faa14009b90).ready entity=mon.0 client_cookie=f3ca839e19e7b1e0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:14.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.245 7faa2091a700 1 -- 172.21.2.112:0/366570986 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faa140022e0 con 0x7faa1c0cff80 2021-02-10T11:47:14.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.245 7faa2091a700 1 -- 172.21.2.112:0/366570986 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa14002560 con 0x7faa1c0cff80 2021-02-10T11:47:14.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.245 7faa2091a700 1 -- 172.21.2.112:0/366570986 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faa14002a80 con 0x7faa1c0cff80 2021-02-10T11:47:14.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.245 7faa23b00700 1 -- 172.21.2.112:0/366570986 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faa1c0cff80 msgr2=0x7faa1c109140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:14.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.245 7faa23b00700 1 --2- 172.21.2.112:0/366570986 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faa1c0cff80 0x7faa1c109140 secure :-1 s=READY pgs=3002 cs=0 l=1 rev1=1 rx=0x7faa14006b80 tx=0x7faa14009b90).stop 2021-02-10T11:47:14.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.246 7faa23b00700 1 -- 172.21.2.112:0/366570986 shutdown_connections 2021-02-10T11:47:14.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.246 7faa23b00700 1 --2- 172.21.2.112:0/366570986 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faa1c0cff80 0x7faa1c109140 unknown :-1 s=CLOSED pgs=3002 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:14.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.246 7faa23b00700 1 --2- 172.21.2.112:0/366570986 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7faa1c0d1880 0x7faa1c1b04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:14.265 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.246 7faa23b00700 1 -- 172.21.2.112:0/366570986 >> 172.21.2.112:0/366570986 conn(0x7faa1c0ce430 msgr2=0x7faa1c0d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:14.265 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.246 7faa23b00700 1 -- 172.21.2.112:0/366570986 shutdown_connections 2021-02-10T11:47:14.265 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.246 7faa23b00700 1 -- 172.21.2.112:0/366570986 wait complete. 2021-02-10T11:47:14.266 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.260 7faa23b00700 1 Processor -- start 2021-02-10T11:47:14.267 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.260 7faa23b00700 1 -- start start 2021-02-10T11:47:14.267 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.261 7faa23b00700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7faa1c096710 0x7faa1c096f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:14.267 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.261 7faa23b00700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7faa1c19c360 0x7faa1c0ad520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:14.267 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.261 7faa23b00700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faa1c0adb40 0x7faa1c19cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:14.267 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.261 7faa23b00700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa1c19c970 con 0x7faa1c0adb40 2021-02-10T11:47:14.267 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.261 7faa23b00700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa1c096b10 con 0x7faa1c096710 2021-02-10T11:47:14.267 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.261 7faa23b00700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa1c082c60 con 0x7faa1c19c360 2021-02-10T11:47:14.267 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.261 7faa2211d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faa1c0adb40 0x7faa1c19cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:14.267 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.261 7faa2211d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faa1c0adb40 0x7faa1c19cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:60220/0 (socket says 172.21.2.112:60220) 2021-02-10T11:47:14.267 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.261 7faa2211d700 1 -- 172.21.2.112:0/2541403276 learned_addr learned my addr 172.21.2.112:0/2541403276 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:47:14.267 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.261 7faa2211d700 1 -- 172.21.2.112:0/2541403276 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7faa1c19c360 msgr2=0x7faa1c0ad520 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:47:14.267 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.261 7faa2211d700 1 --2- 172.21.2.112:0/2541403276 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7faa1c19c360 0x7faa1c0ad520 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:14.267 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.261 7faa2211d700 1 -- 172.21.2.112:0/2541403276 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7faa1c096710 msgr2=0x7faa1c096f70 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:47:14.268 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.261 7faa2211d700 1 --2- 172.21.2.112:0/2541403276 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7faa1c096710 0x7faa1c096f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:14.268 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.261 7faa2211d700 1 -- 172.21.2.112:0/2541403276 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa1c097590 con 0x7faa1c0adb40 2021-02-10T11:47:14.268 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.261 7faa2211d700 1 --2- 172.21.2.112:0/2541403276 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faa1c0adb40 0x7faa1c19cef0 secure :-1 s=READY pgs=3003 cs=0 l=1 rev1=1 rx=0x7faa14009640 tx=0x7faa1400b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:14.268 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.261 7faa127fc700 1 -- 172.21.2.112:0/2541403276 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faa1400bd50 con 0x7faa1c0adb40 2021-02-10T11:47:14.268 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.261 7faa127fc700 1 -- 172.21.2.112:0/2541403276 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa14006240 con 0x7faa1c0adb40 2021-02-10T11:47:14.268 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.262 7faa127fc700 1 -- 172.21.2.112:0/2541403276 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faa140054a0 con 0x7faa1c0adb40 2021-02-10T11:47:14.268 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.262 7faa23b00700 1 -- 172.21.2.112:0/2541403276 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa1c097590 con 0x7faa1c0adb40 2021-02-10T11:47:14.268 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.262 7faa23b00700 1 -- 172.21.2.112:0/2541403276 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa1c097590 con 0x7faa1c0adb40 2021-02-10T11:47:14.268 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.262 7faa127fc700 1 -- 172.21.2.112:0/2541403276 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7faa14006420 con 0x7faa1c0adb40 2021-02-10T11:47:14.268 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.262 7faa127fc700 1 --2- 172.21.2.112:0/2541403276 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7faa040186c0 0x7faa0401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:14.268 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.262 7faa127fc700 1 -- 172.21.2.112:0/2541403276 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(178..178 src has 1..178) v4 ==== 5457+0+0 (secure 0 0 0) 0x7faa14032560 con 0x7faa1c0adb40 2021-02-10T11:47:14.268 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.263 7faa2191c700 1 --2- 172.21.2.112:0/2541403276 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7faa040186c0 0x7faa0401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:14.268 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.263 7faa2191c700 1 --2- 172.21.2.112:0/2541403276 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7faa040186c0 0x7faa0401abc0 crc :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:14.296 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.294 7faa23b00700 1 -- 172.21.2.112:0/2541403276 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa1c109a70 con 0x7faa1c0adb40 2021-02-10T11:47:14.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.296 7faa127fc700 1 -- 172.21.2.112:0/2541403276 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7faa14006030 con 0x7faa1c0adb40 2021-02-10T11:47:14.427 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.424 7faa23b00700 1 -- 172.21.2.112:0/2541403276 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"} v 0) v1 -- 0x7faa1c109a70 con 0x7faa1c0adb40 2021-02-10T11:47:14.430 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.427 7faa127fc700 1 -- 172.21.2.112:0/2541403276 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"}]=0 new fs with metadata pool 38 and data pool 39 v153) v1 ==== 189+0+0 (secure 0 0 0) 0x7faa14006420 con 0x7faa1c0adb40 2021-02-10T11:47:14.434 INFO:teuthology.orchestra.run.gibba012.stderr:new fs with metadata pool 38 and data pool 39 2021-02-10T11:47:14.436 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.433 7faa23b00700 1 -- 172.21.2.112:0/2541403276 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7faa040186c0 msgr2=0x7faa0401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:14.436 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.433 7faa23b00700 1 --2- 172.21.2.112:0/2541403276 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7faa040186c0 0x7faa0401abc0 crc :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:14.436 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.433 7faa23b00700 1 -- 172.21.2.112:0/2541403276 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faa1c0adb40 msgr2=0x7faa1c19cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:14.436 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.433 7faa23b00700 1 --2- 172.21.2.112:0/2541403276 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faa1c0adb40 0x7faa1c19cef0 secure :-1 s=READY pgs=3003 cs=0 l=1 rev1=1 rx=0x7faa14009640 tx=0x7faa1400b960).stop 2021-02-10T11:47:14.436 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.433 7faa23b00700 1 -- 172.21.2.112:0/2541403276 shutdown_connections 2021-02-10T11:47:14.436 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.433 7faa23b00700 1 --2- 172.21.2.112:0/2541403276 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7faa040186c0 0x7faa0401abc0 unknown :-1 s=CLOSED pgs=1355 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:14.437 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.433 7faa23b00700 1 --2- 172.21.2.112:0/2541403276 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faa1c0adb40 0x7faa1c19cef0 unknown :-1 s=CLOSED pgs=3003 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:14.437 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.433 7faa23b00700 1 --2- 172.21.2.112:0/2541403276 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7faa1c19c360 0x7faa1c0ad520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:14.437 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.433 7faa23b00700 1 --2- 172.21.2.112:0/2541403276 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7faa1c096710 0x7faa1c096f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:14.437 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.433 7faa23b00700 1 -- 172.21.2.112:0/2541403276 >> 172.21.2.112:0/2541403276 conn(0x7faa1c19e990 msgr2=0x7faa1c1918e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:14.437 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.433 7faa23b00700 1 -- 172.21.2.112:0/2541403276 shutdown_connections 2021-02-10T11:47:14.437 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.433 7faa23b00700 1 -- 172.21.2.112:0/2541403276 wait complete. 2021-02-10T11:47:14.444 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:47:14.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.853 7fd345ede700 1 Processor -- start 2021-02-10T11:47:14.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.853 7fd345ede700 1 -- start start 2021-02-10T11:47:14.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.854 7fd345ede700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fd3400d1c70 0x7fd3400f20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:14.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.854 7fd345ede700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3400f0b10 con 0x7fd3400d0340 2021-02-10T11:47:14.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.854 7fd345ede700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3401b0be0 con 0x7fd3400d1870 2021-02-10T11:47:14.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.854 7fd345ede700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3400e5ed0 con 0x7fd3400d1c70 2021-02-10T11:47:14.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.854 7fd33effd700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fd3400d0340 0x7fd3400ef640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:59690/0 (socket says 172.21.2.112:59690) 2021-02-10T11:47:14.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.854 7fd33effd700 1 -- 172.21.2.112:0/4099084037 learned_addr learned my addr 172.21.2.112:0/4099084037 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:47:14.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.854 7fd33e7fc700 1 -- 172.21.2.112:0/4099084037 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1468216595 0 0) 0x7fd3300010b0 con 0x7fd3400d0340 2021-02-10T11:47:14.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.854 7fd33e7fc700 1 -- 172.21.2.112:0/4099084037 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd32c002fe0 con 0x7fd3400d0340 2021-02-10T11:47:14.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.855 7fd33e7fc700 1 -- 172.21.2.112:0/4099084037 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1085230262 0 0) 0x7fd3300010b0 con 0x7fd3400d0340 2021-02-10T11:47:14.874 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.855 7fd33e7fc700 1 -- 172.21.2.112:0/4099084037 >> v2:172.21.2.120:3300/0 conn(0x7fd3400d1c70 msgr2=0x7fd3400f20a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:14.875 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.855 7fd33e7fc700 1 --2- 172.21.2.112:0/4099084037 >> v2:172.21.2.120:3300/0 conn(0x7fd3400d1c70 0x7fd3400f20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:14.875 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.855 7fd33e7fc700 1 -- 172.21.2.112:0/4099084037 >> v1:172.21.2.120:6789/0 conn(0x7fd3400d1870 legacy=0x7fd3401b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:14.875 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.855 7fd33e7fc700 1 -- 172.21.2.112:0/4099084037 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3400efe30 con 0x7fd3400d0340 2021-02-10T11:47:14.875 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.855 7fd33e7fc700 1 -- 172.21.2.112:0/4099084037 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7fd3300015b0 con 0x7fd3400d0340 2021-02-10T11:47:14.875 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.855 7fd33e7fc700 1 -- 172.21.2.112:0/4099084037 >> v1:172.21.2.112:6789/0 conn(0x7fd3400d0340 legacy=0x7fd3400ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:14.875 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.855 7fd33e7fc700 1 --2- 172.21.2.112:0/4099084037 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd32c003aa0 0x7fd32c005ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:14.875 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.855 7fd33e7fc700 1 --2- 172.21.2.112:0/4099084037 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd32c006770 0x7fd32c008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:14.875 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.855 7fd33e7fc700 1 --2- 172.21.2.112:0/4099084037 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd32c009590 0x7fd32c00bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:14.875 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.855 7fd33e7fc700 1 -- 172.21.2.112:0/4099084037 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd32c00c300 con 0x7fd32c003aa0 2021-02-10T11:47:14.875 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.855 7fd33e7fc700 1 -- 172.21.2.112:0/4099084037 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd32c00c520 con 0x7fd32c009590 2021-02-10T11:47:14.875 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.855 7fd33e7fc700 1 -- 172.21.2.112:0/4099084037 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd32c00c740 con 0x7fd32c006770 2021-02-10T11:47:14.875 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.855 7fd33f7fe700 1 --2- 172.21.2.112:0/4099084037 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd32c003aa0 0x7fd32c005ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:14.876 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.855 7fd33f7fe700 1 -- 172.21.2.112:0/4099084037 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd32c006770 msgr2=0x7fd32c008c90 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:47:14.876 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.855 7fd33f7fe700 1 --2- 172.21.2.112:0/4099084037 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd32c006770 0x7fd32c008c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:14.876 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.855 7fd33f7fe700 1 -- 172.21.2.112:0/4099084037 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd32c009590 msgr2=0x7fd32c00bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:14.876 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.855 7fd33f7fe700 1 --2- 172.21.2.112:0/4099084037 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd32c009590 0x7fd32c00bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:14.876 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.855 7fd33f7fe700 1 -- 172.21.2.112:0/4099084037 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd32c00c9e0 con 0x7fd32c003aa0 2021-02-10T11:47:14.876 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.855 7fd33ffff700 1 -- 172.21.2.112:0/4099084037 reap_dead start 2021-02-10T11:47:14.876 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.855 7fd33f7fe700 1 --2- 172.21.2.112:0/4099084037 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd32c003aa0 0x7fd32c005ed0 secure :-1 s=READY pgs=3005 cs=0 l=1 rev1=1 rx=0x7fd328006ab0 tx=0x7fd32804fbc0).ready entity=mon.0 client_cookie=5d99fa0a59770d25 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:14.876 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.855 7fd33e7fc700 1 -- 172.21.2.112:0/4099084037 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd328002030 con 0x7fd32c003aa0 2021-02-10T11:47:14.876 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.855 7fd33e7fc700 1 -- 172.21.2.112:0/4099084037 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd328002330 con 0x7fd32c003aa0 2021-02-10T11:47:14.876 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.855 7fd33e7fc700 1 -- 172.21.2.112:0/4099084037 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd328002850 con 0x7fd32c003aa0 2021-02-10T11:47:14.876 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.856 7fd345ede700 1 -- 172.21.2.112:0/4099084037 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd32c003aa0 msgr2=0x7fd32c005ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:14.876 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.856 7fd345ede700 1 --2- 172.21.2.112:0/4099084037 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd32c003aa0 0x7fd32c005ed0 secure :-1 s=READY pgs=3005 cs=0 l=1 rev1=1 rx=0x7fd328006ab0 tx=0x7fd32804fbc0).stop 2021-02-10T11:47:14.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.856 7fd345ede700 1 -- 172.21.2.112:0/4099084037 shutdown_connections 2021-02-10T11:47:14.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.856 7fd345ede700 1 --2- 172.21.2.112:0/4099084037 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd32c003aa0 0x7fd32c005ed0 unknown :-1 s=CLOSED pgs=3005 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:14.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.856 7fd345ede700 1 -- 172.21.2.112:0/4099084037 >> 172.21.2.112:0/4099084037 conn(0x7fd3400ce450 msgr2=0x7fd3400d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:14.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.856 7fd345ede700 1 -- 172.21.2.112:0/4099084037 shutdown_connections 2021-02-10T11:47:14.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.856 7fd345ede700 1 -- 172.21.2.112:0/4099084037 wait complete. 2021-02-10T11:47:14.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.871 7fd345ede700 1 Processor -- start 2021-02-10T11:47:14.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.871 7fd345ede700 1 -- start start 2021-02-10T11:47:14.878 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.871 7fd345ede700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd3401942b0 0x7fd3400b3550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:14.878 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.871 7fd345ede700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd340195070 0x7fd340195470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:14.879 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.871 7fd345ede700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd3400bb8e0 0x7fd3400bbce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:14.879 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.871 7fd345ede700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3401946b0 con 0x7fd340195070 2021-02-10T11:47:14.879 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.871 7fd345ede700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3400b3b70 con 0x7fd3400bb8e0 2021-02-10T11:47:14.879 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.871 7fd345ede700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3400b3d90 con 0x7fd3401942b0 2021-02-10T11:47:14.879 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.871 7fd33effd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd340195070 0x7fd340195470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:14.879 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.871 7fd33effd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd340195070 0x7fd340195470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:60242/0 (socket says 172.21.2.112:60242) 2021-02-10T11:47:14.879 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.871 7fd33effd700 1 -- 172.21.2.112:0/2609378475 learned_addr learned my addr 172.21.2.112:0/2609378475 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:47:14.879 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.872 7fd33effd700 1 -- 172.21.2.112:0/2609378475 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd3401942b0 msgr2=0x7fd3400b3550 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:47:14.879 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.872 7fd33effd700 1 --2- 172.21.2.112:0/2609378475 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd3401942b0 0x7fd3400b3550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:14.879 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.872 7fd33effd700 1 -- 172.21.2.112:0/2609378475 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd3400bb8e0 msgr2=0x7fd3400bbce0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:47:14.879 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.872 7fd33effd700 1 --2- 172.21.2.112:0/2609378475 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd3400bb8e0 0x7fd3400bbce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:14.879 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.872 7fd33effd700 1 -- 172.21.2.112:0/2609378475 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3401a49e0 con 0x7fd340195070 2021-02-10T11:47:14.879 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.872 7fd33effd700 1 --2- 172.21.2.112:0/2609378475 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd340195070 0x7fd340195470 secure :-1 s=READY pgs=3006 cs=0 l=1 rev1=1 rx=0x7fd33000a590 tx=0x7fd33000a930).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:14.880 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.872 7fd337fff700 1 -- 172.21.2.112:0/2609378475 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd33000adb0 con 0x7fd340195070 2021-02-10T11:47:14.880 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.872 7fd337fff700 1 -- 172.21.2.112:0/2609378475 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3300018a0 con 0x7fd340195070 2021-02-10T11:47:14.880 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.872 7fd345ede700 1 -- 172.21.2.112:0/2609378475 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3401a49e0 con 0x7fd340195070 2021-02-10T11:47:14.880 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.872 7fd345ede700 1 -- 172.21.2.112:0/2609378475 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3400bc5f0 con 0x7fd340195070 2021-02-10T11:47:14.880 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.872 7fd337fff700 1 -- 172.21.2.112:0/2609378475 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd330004330 con 0x7fd340195070 2021-02-10T11:47:14.880 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.873 7fd337fff700 1 -- 172.21.2.112:0/2609378475 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd330004140 con 0x7fd340195070 2021-02-10T11:47:14.880 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.873 7fd337fff700 1 --2- 172.21.2.112:0/2609378475 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd3240186c0 0x7fd32401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:14.880 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.873 7fd337fff700 1 -- 172.21.2.112:0/2609378475 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fd330032760 con 0x7fd340195070 2021-02-10T11:47:14.880 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.873 7fd33f7fe700 1 --2- 172.21.2.112:0/2609378475 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd3240186c0 0x7fd32401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:14.880 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.874 7fd33f7fe700 1 --2- 172.21.2.112:0/2609378475 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd3240186c0 0x7fd32401abc0 crc :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:14.907 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.904 7fd345ede700 1 -- 172.21.2.112:0/2609378475 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3400bc5f0 con 0x7fd340195070 2021-02-10T11:47:14.909 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:14.907 7fd337fff700 1 -- 172.21.2.112:0/2609378475 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd330004140 con 0x7fd340195070 2021-02-10T11:47:15.036 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:15.033 7fd345ede700 1 -- 172.21.2.112:0/2609378475 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd3400bc5f0 con 0x7fd340195070 2021-02-10T11:47:15.036 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:15.034 7fd337fff700 1 -- 172.21.2.112:0/2609378475 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v179) v1 ==== 74+0+14020 (secure 0 0 0) 0x7fd330001960 con 0x7fd340195070 2021-02-10T11:47:15.037 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:47:15.037 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":179,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:47:14.425624","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":39,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":38,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:47:12.395866","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"179","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":39,"pool_name":"cephfs_data","create_time":"2021-02-10 11:47:13.420625","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"179","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:47:15.038 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:15.035 7fd345ede700 1 -- 172.21.2.112:0/2609378475 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd3240186c0 msgr2=0x7fd32401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:15.038 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:15.035 7fd345ede700 1 --2- 172.21.2.112:0/2609378475 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd3240186c0 0x7fd32401abc0 crc :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:15.038 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:15.035 7fd345ede700 1 -- 172.21.2.112:0/2609378475 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd340195070 msgr2=0x7fd340195470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:15.038 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:15.035 7fd345ede700 1 --2- 172.21.2.112:0/2609378475 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd340195070 0x7fd340195470 secure :-1 s=READY pgs=3006 cs=0 l=1 rev1=1 rx=0x7fd33000a590 tx=0x7fd33000a930).stop 2021-02-10T11:47:15.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:15.036 7fd345ede700 1 -- 172.21.2.112:0/2609378475 shutdown_connections 2021-02-10T11:47:15.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:15.036 7fd345ede700 1 --2- 172.21.2.112:0/2609378475 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd3240186c0 0x7fd32401abc0 unknown :-1 s=CLOSED pgs=1356 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:15.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:15.036 7fd345ede700 1 --2- 172.21.2.112:0/2609378475 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd3400bb8e0 0x7fd3400bbce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:15.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:15.036 7fd345ede700 1 --2- 172.21.2.112:0/2609378475 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd340195070 0x7fd340195470 unknown :-1 s=CLOSED pgs=3006 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:15.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:15.036 7fd345ede700 1 --2- 172.21.2.112:0/2609378475 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd3401942b0 0x7fd3400b3550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:15.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:15.036 7fd345ede700 1 -- 172.21.2.112:0/2609378475 >> 172.21.2.112:0/2609378475 conn(0x7fd3400b3040 msgr2=0x7fd3400c3250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:15.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:15.036 7fd345ede700 1 -- 172.21.2.112:0/2609378475 shutdown_connections 2021-02-10T11:47:15.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:15.036 7fd345ede700 1 -- 172.21.2.112:0/2609378475 wait complete. 2021-02-10T11:47:15.047 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:47:15.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:15.459 7f88bab52700 1 Processor -- start 2021-02-10T11:47:15.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:15.459 7f88bab52700 1 -- start start 2021-02-10T11:47:15.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:15.460 7f88bab52700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f88b40d1870 0x7f88b41b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:15.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:15.460 7f88bab52700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f88b40f0650 0x7f88b40cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:15.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:15.460 7f88bab52700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88b40f0b10 con 0x7f88b41b0a10 2021-02-10T11:47:15.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:15.460 7f88bab52700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88b40e5ed0 con 0x7f88b40d1870 2021-02-10T11:47:15.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:15.460 7f88bab52700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88b40d1c70 con 0x7f88b40f0650 2021-02-10T11:47:15.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:15.460 7f88b3fff700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f88b41b0a10 0x7f88b40cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:59710/0 (socket says 172.21.2.112:59710) 2021-02-10T11:47:15.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:15.460 7f88b3fff700 1 -- 172.21.2.112:0/1168693816 learned_addr learned my addr 172.21.2.112:0/1168693816 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:47:15.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:15.460 7f88b896e700 1 --2- 172.21.2.112:0/1168693816 >> v2:172.21.2.112:3300/0 conn(0x7f88b40d1870 0x7f88b41b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:15.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:15.460 7f88b916f700 1 --2- 172.21.2.112:0/1168693816 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f88b40f0650 0x7f88b40cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:15.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:15.460 7f88b37fe700 1 -- 172.21.2.112:0/1168693816 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3399724819 0 0) 0x7f88a4000ce0 con 0x7f88b41b0a10 2021-02-10T11:47:15.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:15.460 7f88b37fe700 1 -- 172.21.2.112:0/1168693816 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88a0002fe0 con 0x7f88b41b0a10 2021-02-10T11:47:15.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:15.460 7f88b916f700 1 -- 172.21.2.112:0/1168693816 >> v2:172.21.2.112:3300/0 conn(0x7f88b40d1870 msgr2=0x7f88b41b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:15.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:15.460 7f88b916f700 1 --2- 172.21.2.112:0/1168693816 >> v2:172.21.2.112:3300/0 conn(0x7f88b40d1870 0x7f88b41b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:15.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:15.460 7f88b916f700 1 -- 172.21.2.112:0/1168693816 >> v1:172.21.2.112:6789/0 conn(0x7f88b41b0a10 legacy=0x7f88b40cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:15.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:15.460 7f88b916f700 1 -- 172.21.2.112:0/1168693816 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88b40d1e90 con 0x7f88b40f0650 2021-02-10T11:47:15.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:15.460 7f88b896e700 1 --2- 172.21.2.112:0/1168693816 >> v2:172.21.2.112:3300/0 conn(0x7f88b40d1870 0x7f88b41b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:47:15.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:15.460 7f88b916f700 1 --2- 172.21.2.112:0/1168693816 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f88b40f0650 0x7f88b40cfb20 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 rx=0x7f88ac006e50 tx=0x7f88ac009e70).ready entity=mon.2 client_cookie=218262bd6837e74d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:15.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:15.460 7f88b37fe700 1 -- 172.21.2.112:0/1168693816 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f88ac000f90 con 0x7f88b40f0650 2021-02-10T11:47:15.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:15.460 7f88b37fe700 1 -- 172.21.2.112:0/1168693816 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88ac0066f0 con 0x7f88b40f0650 2021-02-10T11:47:15.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:15.460 7f88b37fe700 1 -- 172.21.2.112:0/1168693816 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f88ac00b790 con 0x7f88b40f0650 2021-02-10T11:47:15.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:15.461 7f88bab52700 1 -- 172.21.2.112:0/1168693816 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f88b40f0650 msgr2=0x7f88b40cfb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:15.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:15.461 7f88bab52700 1 --2- 172.21.2.112:0/1168693816 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f88b40f0650 0x7f88b40cfb20 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 rx=0x7f88ac006e50 tx=0x7f88ac009e70).stop 2021-02-10T11:47:15.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:15.461 7f88bab52700 1 -- 172.21.2.112:0/1168693816 shutdown_connections 2021-02-10T11:47:15.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:15.461 7f88bab52700 1 --2- 172.21.2.112:0/1168693816 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f88b40f0650 0x7f88b40cfb20 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:15.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:15.461 7f88bab52700 1 --2- 172.21.2.112:0/1168693816 >> v2:172.21.2.112:3300/0 conn(0x7f88b40d1870 0x7f88b41b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:15.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:15.461 7f88bab52700 1 -- 172.21.2.112:0/1168693816 >> 172.21.2.112:0/1168693816 conn(0x7f88b40ce450 msgr2=0x7f88b40d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:15.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:15.461 7f88bab52700 1 -- 172.21.2.112:0/1168693816 shutdown_connections 2021-02-10T11:47:15.481 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:15.461 7f88bab52700 1 -- 172.21.2.112:0/1168693816 wait complete. 2021-02-10T11:47:15.482 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:15.476 7f88bab52700 1 Processor -- start 2021-02-10T11:47:15.482 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:15.476 7f88bab52700 1 -- start start 2021-02-10T11:47:15.482 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:15.476 7f88bab52700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f88b40ada90 0x7f88b40ade90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:15.482 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:15.476 7f88bab52700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f88b4099d20 0x7f88b409a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:15.482 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:15.476 7f88bab52700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f88b409a740 0x7f88b4097bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:15.482 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:15.476 7f88bab52700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88b40997a0 con 0x7f88b4099d20 2021-02-10T11:47:15.483 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:15.476 7f88bab52700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88b418f0c0 con 0x7f88b409a740 2021-02-10T11:47:15.483 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:15.476 7f88bab52700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88b409ab40 con 0x7f88b40ada90 2021-02-10T11:47:15.483 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:15.476 7f88b3fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f88b4099d20 0x7f88b409a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:15.483 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:15.476 7f88b3fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f88b4099d20 0x7f88b409a120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:60252/0 (socket says 172.21.2.112:60252) 2021-02-10T11:47:15.483 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:15.476 7f88b3fff700 1 -- 172.21.2.112:0/3356846990 learned_addr learned my addr 172.21.2.112:0/3356846990 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:47:15.483 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:15.476 7f88b916f700 1 --2- 172.21.2.112:0/3356846990 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f88b409a740 0x7f88b4097bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:15.483 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:15.476 7f88b896e700 1 --2- 172.21.2.112:0/3356846990 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f88b40ada90 0x7f88b40ade90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:15.483 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:15.477 7f88b3fff700 1 -- 172.21.2.112:0/3356846990 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f88b40ada90 msgr2=0x7f88b40ade90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:15.483 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:15.477 7f88b3fff700 1 --2- 172.21.2.112:0/3356846990 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f88b40ada90 0x7f88b40ade90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:15.483 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:15.477 7f88b3fff700 1 -- 172.21.2.112:0/3356846990 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f88b409a740 msgr2=0x7f88b4097bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:15.483 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:15.477 7f88b3fff700 1 --2- 172.21.2.112:0/3356846990 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f88b409a740 0x7f88b4097bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:15.483 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:15.477 7f88b3fff700 1 -- 172.21.2.112:0/3356846990 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88b409ad60 con 0x7f88b4099d20 2021-02-10T11:47:15.483 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:15.477 7f88b896e700 1 --2- 172.21.2.112:0/3356846990 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f88b40ada90 0x7f88b40ade90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:47:15.484 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:15.477 7f88b3fff700 1 --2- 172.21.2.112:0/3356846990 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f88b4099d20 0x7f88b409a120 secure :-1 s=READY pgs=3008 cs=0 l=1 rev1=1 rx=0x7f88a4006f00 tx=0x7f88a400b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:15.484 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:15.477 7f88b17fa700 1 -- 172.21.2.112:0/3356846990 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f88a400bd30 con 0x7f88b4099d20 2021-02-10T11:47:15.484 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:15.477 7f88b17fa700 1 -- 172.21.2.112:0/3356846990 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88a4006170 con 0x7f88b4099d20 2021-02-10T11:47:15.484 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:15.477 7f88b17fa700 1 -- 172.21.2.112:0/3356846990 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f88a4004590 con 0x7f88b4099d20 2021-02-10T11:47:15.484 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:15.477 7f88bab52700 1 -- 172.21.2.112:0/3356846990 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88b409ad60 con 0x7f88b4099d20 2021-02-10T11:47:15.484 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:15.477 7f88bab52700 1 -- 172.21.2.112:0/3356846990 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88b40984d0 con 0x7f88b4099d20 2021-02-10T11:47:15.484 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:15.477 7f88b17fa700 1 -- 172.21.2.112:0/3356846990 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f88a4009fb0 con 0x7f88b4099d20 2021-02-10T11:47:15.484 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:15.477 7f88b17fa700 1 --2- 172.21.2.112:0/3356846990 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f88980186c0 0x7f889801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:15.484 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:15.478 7f88b17fa700 1 -- 172.21.2.112:0/3356846990 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f88a4032370 con 0x7f88b4099d20 2021-02-10T11:47:15.484 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:15.478 7f88b896e700 1 --2- 172.21.2.112:0/3356846990 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f88980186c0 0x7f889801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:15.484 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:15.478 7f88b896e700 1 --2- 172.21.2.112:0/3356846990 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f88980186c0 0x7f889801abc0 crc :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:15.512 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:15.509 7f88bab52700 1 -- 172.21.2.112:0/3356846990 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88b40ebc00 con 0x7f88b4099d20 2021-02-10T11:47:15.514 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:15.512 7f88b17fa700 1 -- 172.21.2.112:0/3356846990 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f88a40043a0 con 0x7f88b4099d20 2021-02-10T11:47:15.642 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:15.640 7f88bab52700 1 -- 172.21.2.112:0/3356846990 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f88b40f0b10 con 0x7f88b4099d20 2021-02-10T11:47:15.643 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:15.640 7f88b17fa700 1 -- 172.21.2.112:0/3356846990 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v180) v1 ==== 74+0+14007 (secure 0 0 0) 0x7f88a4009fb0 con 0x7f88b4099d20 2021-02-10T11:47:15.643 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:47:15.643 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":180,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:47:15.427174","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":39,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":38,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:47:12.395866","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"179","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":39,"pool_name":"cephfs_data","create_time":"2021-02-10 11:47:13.420625","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"180","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:47:15.645 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:15.642 7f88bab52700 1 -- 172.21.2.112:0/3356846990 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f88980186c0 msgr2=0x7f889801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:15.645 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:15.642 7f88bab52700 1 --2- 172.21.2.112:0/3356846990 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f88980186c0 0x7f889801abc0 crc :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:15.645 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:15.642 7f88bab52700 1 -- 172.21.2.112:0/3356846990 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f88b4099d20 msgr2=0x7f88b409a120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:15.645 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:15.642 7f88bab52700 1 --2- 172.21.2.112:0/3356846990 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f88b4099d20 0x7f88b409a120 secure :-1 s=READY pgs=3008 cs=0 l=1 rev1=1 rx=0x7f88a4006f00 tx=0x7f88a400b940).stop 2021-02-10T11:47:15.645 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:15.642 7f88bab52700 1 -- 172.21.2.112:0/3356846990 shutdown_connections 2021-02-10T11:47:15.645 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:15.642 7f88bab52700 1 --2- 172.21.2.112:0/3356846990 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f88980186c0 0x7f889801abc0 unknown :-1 s=CLOSED pgs=1357 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:15.645 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:15.642 7f88bab52700 1 --2- 172.21.2.112:0/3356846990 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f88b409a740 0x7f88b4097bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:15.645 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:15.642 7f88bab52700 1 --2- 172.21.2.112:0/3356846990 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f88b4099d20 0x7f88b409a120 unknown :-1 s=CLOSED pgs=3008 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:15.645 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:15.642 7f88bab52700 1 --2- 172.21.2.112:0/3356846990 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f88b40ada90 0x7f88b40ade90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:15.645 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:15.642 7f88bab52700 1 -- 172.21.2.112:0/3356846990 >> 172.21.2.112:0/3356846990 conn(0x7f88b40f0550 msgr2=0x7f88b4191730 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:15.646 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:15.642 7f88bab52700 1 -- 172.21.2.112:0/3356846990 shutdown_connections 2021-02-10T11:47:15.646 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:15.642 7f88bab52700 1 -- 172.21.2.112:0/3356846990 wait complete. 2021-02-10T11:47:15.654 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2021-02-10T11:47:16.085 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.067 7f795e998700 1 Processor -- start 2021-02-10T11:47:16.085 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.067 7f795e998700 1 -- start start 2021-02-10T11:47:16.085 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.067 7f795e998700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f79580d1870 0x7f79581b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:16.085 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.067 7f795e998700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f79580cff70 0x7f7958109130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:16.085 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.067 7f795e998700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7958109a60 con 0x7f79581b0ae0 2021-02-10T11:47:16.085 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.067 7f795e998700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7958109750 con 0x7f79580cff70 2021-02-10T11:47:16.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.067 7f795e998700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7958088070 con 0x7f79580d1870 2021-02-10T11:47:16.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.067 7f795cfb5700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f79580cff70 0x7f7958109130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:16.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.067 7f795cfb5700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f79580cff70 0x7f7958109130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:60260/0 (socket says 172.21.2.112:60260) 2021-02-10T11:47:16.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.067 7f795cfb5700 1 -- 172.21.2.112:0/4207082237 learned_addr learned my addr 172.21.2.112:0/4207082237 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:47:16.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.068 7f795cfb5700 1 -- 172.21.2.112:0/4207082237 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f79580d1870 msgr2=0x7f79581b04c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:47:16.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.068 7f795cfb5700 1 --2- 172.21.2.112:0/4207082237 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f79580d1870 0x7f79581b04c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:16.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.068 7f795cfb5700 1 -- 172.21.2.112:0/4207082237 >> v1:172.21.2.120:6789/0 conn(0x7f79581b0ae0 legacy=0x7f7958108940 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:47:16.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.068 7f795cfb5700 1 -- 172.21.2.112:0/4207082237 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7958088260 con 0x7f79580cff70 2021-02-10T11:47:16.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.068 7f795cfb5700 1 --2- 172.21.2.112:0/4207082237 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f79580cff70 0x7f7958109130 secure :-1 s=READY pgs=3009 cs=0 l=1 rev1=1 rx=0x7f7950006f00 tx=0x7f7950009f10).ready entity=mon.0 client_cookie=bee04951f25cf8af server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:16.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.068 7f7956ffd700 1 -- 172.21.2.112:0/4207082237 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f795000b3b0 con 0x7f79580cff70 2021-02-10T11:47:16.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.068 7f7956ffd700 1 -- 172.21.2.112:0/4207082237 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f795000b630 con 0x7f79580cff70 2021-02-10T11:47:16.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.068 7f7956ffd700 1 -- 172.21.2.112:0/4207082237 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f795000bb50 con 0x7f79580cff70 2021-02-10T11:47:16.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.068 7f795e998700 1 -- 172.21.2.112:0/4207082237 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f79580cff70 msgr2=0x7f7958109130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:16.087 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.068 7f795e998700 1 --2- 172.21.2.112:0/4207082237 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f79580cff70 0x7f7958109130 secure :-1 s=READY pgs=3009 cs=0 l=1 rev1=1 rx=0x7f7950006f00 tx=0x7f7950009f10).stop 2021-02-10T11:47:16.087 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.068 7f795e998700 1 -- 172.21.2.112:0/4207082237 shutdown_connections 2021-02-10T11:47:16.087 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.068 7f795e998700 1 --2- 172.21.2.112:0/4207082237 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f79580cff70 0x7f7958109130 unknown :-1 s=CLOSED pgs=3009 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:16.087 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.068 7f795e998700 1 --2- 172.21.2.112:0/4207082237 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f79580d1870 0x7f79581b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:16.087 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.068 7f795e998700 1 -- 172.21.2.112:0/4207082237 >> 172.21.2.112:0/4207082237 conn(0x7f79580ce420 msgr2=0x7f79580d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:16.087 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.068 7f795e998700 1 -- 172.21.2.112:0/4207082237 shutdown_connections 2021-02-10T11:47:16.087 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.069 7f795e998700 1 -- 172.21.2.112:0/4207082237 wait complete. 2021-02-10T11:47:16.088 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.083 7f795e998700 1 Processor -- start 2021-02-10T11:47:16.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.083 7f795e998700 1 -- start start 2021-02-10T11:47:16.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.083 7f795e998700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f79580c3820 0x7f7958084cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:16.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.083 7f795e998700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f79580a0290 0x7f79580852d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:16.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.083 7f795e998700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7958086270 0x7f7958086670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:16.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.083 7f795e998700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79580c0700 con 0x7f7958086270 2021-02-10T11:47:16.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.083 7f795e998700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79580858f0 con 0x7f79580a0290 2021-02-10T11:47:16.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.083 7f795e998700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7958085b10 con 0x7f79580c3820 2021-02-10T11:47:16.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.083 7f795cfb5700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7958086270 0x7f7958086670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:16.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.083 7f795cfb5700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7958086270 0x7f7958086670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:60268/0 (socket says 172.21.2.112:60268) 2021-02-10T11:47:16.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.083 7f795cfb5700 1 -- 172.21.2.112:0/358554289 learned_addr learned my addr 172.21.2.112:0/358554289 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:47:16.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.084 7f795cfb5700 1 -- 172.21.2.112:0/358554289 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f79580c3820 msgr2=0x7f7958084cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:16.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.084 7f795cfb5700 1 --2- 172.21.2.112:0/358554289 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f79580c3820 0x7f7958084cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:16.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.084 7f795cfb5700 1 -- 172.21.2.112:0/358554289 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f79580a0290 msgr2=0x7f79580852d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:16.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.084 7f795cfb5700 1 --2- 172.21.2.112:0/358554289 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f79580a0290 0x7f79580852d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:16.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.084 7f795cfb5700 1 -- 172.21.2.112:0/358554289 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7958086c90 con 0x7f7958086270 2021-02-10T11:47:16.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.084 7f795cfb5700 1 --2- 172.21.2.112:0/358554289 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7958086270 0x7f7958086670 secure :-1 s=READY pgs=3010 cs=0 l=1 rev1=1 rx=0x7f7950009390 tx=0x7f795000bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:16.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.084 7f7954ff9700 1 -- 172.21.2.112:0/358554289 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7950009450 con 0x7f7958086270 2021-02-10T11:47:16.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.084 7f7954ff9700 1 -- 172.21.2.112:0/358554289 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79500068e0 con 0x7f7958086270 2021-02-10T11:47:16.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.084 7f795e998700 1 -- 172.21.2.112:0/358554289 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7958086c90 con 0x7f7958086270 2021-02-10T11:47:16.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.084 7f795e998700 1 -- 172.21.2.112:0/358554289 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79581a2f00 con 0x7f7958086270 2021-02-10T11:47:16.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.084 7f7954ff9700 1 -- 172.21.2.112:0/358554289 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f795000c4e0 con 0x7f7958086270 2021-02-10T11:47:16.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.085 7f7954ff9700 1 -- 172.21.2.112:0/358554289 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7950001270 con 0x7f7958086270 2021-02-10T11:47:16.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.085 7f7954ff9700 1 --2- 172.21.2.112:0/358554289 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7938018730 0x7f793801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:16.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.085 7f7954ff9700 1 -- 172.21.2.112:0/358554289 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f79500329f0 con 0x7f7958086270 2021-02-10T11:47:16.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.085 7f795e998700 1 -- 172.21.2.112:0/358554289 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7958082a70 con 0x7f7958086270 2021-02-10T11:47:16.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.085 7f7957fff700 1 --2- 172.21.2.112:0/358554289 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7938018730 0x7f793801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:16.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.086 7f7957fff700 1 --2- 172.21.2.112:0/358554289 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7938018730 0x7f793801abc0 crc :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:16.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.088 7f7954ff9700 1 -- 172.21.2.112:0/358554289 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7950003080 con 0x7f7958086270 2021-02-10T11:47:16.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.223 7f795e998700 1 -- 172.21.2.112:0/358554289 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"} v 0) v1 -- 0x7f7958082a70 con 0x7f7958086270 2021-02-10T11:47:16.441 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.438 7f7954ff9700 1 -- 172.21.2.112:0/358554289 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"}]=0 v156) v1 ==== 118+0+0 (secure 0 0 0) 0x7f7950001270 con 0x7f7958086270 2021-02-10T11:47:16.443 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.440 7f795e998700 1 -- 172.21.2.112:0/358554289 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7938018730 msgr2=0x7f793801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:16.443 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.440 7f795e998700 1 --2- 172.21.2.112:0/358554289 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7938018730 0x7f793801abc0 crc :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:16.443 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.440 7f795e998700 1 -- 172.21.2.112:0/358554289 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7958086270 msgr2=0x7f7958086670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:16.443 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.440 7f795e998700 1 --2- 172.21.2.112:0/358554289 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7958086270 0x7f7958086670 secure :-1 s=READY pgs=3010 cs=0 l=1 rev1=1 rx=0x7f7950009390 tx=0x7f795000bd40).stop 2021-02-10T11:47:16.444 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.441 7f795e998700 1 -- 172.21.2.112:0/358554289 shutdown_connections 2021-02-10T11:47:16.444 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.441 7f795e998700 1 --2- 172.21.2.112:0/358554289 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f7938018730 0x7f793801abc0 unknown :-1 s=CLOSED pgs=1358 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:16.444 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.441 7f795e998700 1 --2- 172.21.2.112:0/358554289 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f79580a0290 0x7f79580852d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:16.444 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.441 7f795e998700 1 --2- 172.21.2.112:0/358554289 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7958086270 0x7f7958086670 unknown :-1 s=CLOSED pgs=3010 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:16.444 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.441 7f795e998700 1 --2- 172.21.2.112:0/358554289 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f79580c3820 0x7f7958084cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:16.444 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.441 7f795e998700 1 -- 172.21.2.112:0/358554289 >> 172.21.2.112:0/358554289 conn(0x7f79580872c0 msgr2=0x7f7958095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:16.444 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.441 7f795e998700 1 -- 172.21.2.112:0/358554289 shutdown_connections 2021-02-10T11:47:16.444 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.441 7f795e998700 1 -- 172.21.2.112:0/358554289 wait complete. 2021-02-10T11:47:16.452 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:47:16.882 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.863 7fbe015ea700 1 Processor -- start 2021-02-10T11:47:16.882 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.863 7fbe015ea700 1 -- start start 2021-02-10T11:47:16.882 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.863 7fbe015ea700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbdfc0d0340 0x7fbdfc0ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:16.882 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.863 7fbe015ea700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fbdfc0d1c70 0x7fbdfc0efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:16.882 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.863 7fbe015ea700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdfc0f0b10 con 0x7fbdfc0d1870 2021-02-10T11:47:16.882 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.863 7fbe015ea700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbdfc0e5ed0 con 0x7fbdfc0d1c70 2021-02-10T11:47:16.882 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.863 7fbe015ea700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdfc1b0be0 con 0x7fbdfc0d0340 2021-02-10T11:47:16.882 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.863 7fbdfbac0700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fbdfc0d1c70 0x7fbdfc0efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:16.882 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.863 7fbdfbac0700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fbdfc0d1c70 0x7fbdfc0efc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:60276/0 (socket says 172.21.2.112:60276) 2021-02-10T11:47:16.882 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.863 7fbdfbac0700 1 -- 172.21.2.112:0/1775048459 learned_addr learned my addr 172.21.2.112:0/1775048459 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:47:16.883 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.864 7fbdfbac0700 1 -- 172.21.2.112:0/1775048459 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbdfc0d0340 msgr2=0x7fbdfc0ef640 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:47:16.883 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.864 7fbdfbac0700 1 --2- 172.21.2.112:0/1775048459 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbdfc0d0340 0x7fbdfc0ef640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:16.883 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.864 7fbdfbac0700 1 -- 172.21.2.112:0/1775048459 >> v1:172.21.2.112:6789/0 conn(0x7fbdfc0d1870 legacy=0x7fbdfc1b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:16.883 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.864 7fbdfbac0700 1 -- 172.21.2.112:0/1775048459 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdfc0f20a0 con 0x7fbdfc0d1c70 2021-02-10T11:47:16.883 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.864 7fbdfbac0700 1 --2- 172.21.2.112:0/1775048459 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbdfc0d1c70 0x7fbdfc0efc60 secure :-1 s=READY pgs=3012 cs=0 l=1 rev1=1 rx=0x7fbdf4006f00 tx=0x7fbdf4009f10).ready entity=mon.0 client_cookie=ebf75dd3ca881dd0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:16.883 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.864 7fbdfa2bd700 1 -- 172.21.2.112:0/1775048459 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbdf400b3b0 con 0x7fbdfc0d1c70 2021-02-10T11:47:16.883 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.864 7fbdfa2bd700 1 -- 172.21.2.112:0/1775048459 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdf400b630 con 0x7fbdfc0d1c70 2021-02-10T11:47:16.883 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.864 7fbdfa2bd700 1 -- 172.21.2.112:0/1775048459 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbdf400bb10 con 0x7fbdfc0d1c70 2021-02-10T11:47:16.883 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.864 7fbe015ea700 1 -- 172.21.2.112:0/1775048459 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbdfc0d1c70 msgr2=0x7fbdfc0efc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:16.883 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.864 7fbe015ea700 1 --2- 172.21.2.112:0/1775048459 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbdfc0d1c70 0x7fbdfc0efc60 secure :-1 s=READY pgs=3012 cs=0 l=1 rev1=1 rx=0x7fbdf4006f00 tx=0x7fbdf4009f10).stop 2021-02-10T11:47:16.883 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.865 7fbe015ea700 1 -- 172.21.2.112:0/1775048459 shutdown_connections 2021-02-10T11:47:16.883 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.865 7fbe015ea700 1 --2- 172.21.2.112:0/1775048459 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbdfc0d1c70 0x7fbdfc0efc60 unknown :-1 s=CLOSED pgs=3012 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:16.883 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.865 7fbe015ea700 1 --2- 172.21.2.112:0/1775048459 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbdfc0d0340 0x7fbdfc0ef640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:16.884 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.865 7fbe015ea700 1 -- 172.21.2.112:0/1775048459 >> 172.21.2.112:0/1775048459 conn(0x7fbdfc0ce450 msgr2=0x7fbdfc0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:16.884 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.865 7fbe015ea700 1 -- 172.21.2.112:0/1775048459 shutdown_connections 2021-02-10T11:47:16.884 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.865 7fbe015ea700 1 -- 172.21.2.112:0/1775048459 wait complete. 2021-02-10T11:47:16.885 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.879 7fbe015ea700 1 Processor -- start 2021-02-10T11:47:16.885 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.880 7fbe015ea700 1 -- start start 2021-02-10T11:47:16.885 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.880 7fbe015ea700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fbdfc19ede0 0x7fbdfc19f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:16.885 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.880 7fbe015ea700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbdfc1aca40 0x7fbdfc1ace40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:16.885 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.880 7fbe015ea700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbdfc19d3c0 0x7fbdfc19d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:16.886 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.880 7fbe015ea700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdfc1ad460 con 0x7fbdfc19d3c0 2021-02-10T11:47:16.886 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.880 7fbe015ea700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdfc19fca0 con 0x7fbdfc19ede0 2021-02-10T11:47:16.886 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.880 7fbe015ea700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdfc19dde0 con 0x7fbdfc1aca40 2021-02-10T11:47:16.886 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.880 7fbdfbac0700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbdfc19d3c0 0x7fbdfc19d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:16.886 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.880 7fbdfbac0700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbdfc19d3c0 0x7fbdfc19d7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:60282/0 (socket says 172.21.2.112:60282) 2021-02-10T11:47:16.886 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.880 7fbdfbac0700 1 -- 172.21.2.112:0/1240289284 learned_addr learned my addr 172.21.2.112:0/1240289284 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:47:16.886 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.880 7fbdfbac0700 1 -- 172.21.2.112:0/1240289284 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbdfc1aca40 msgr2=0x7fbdfc1ace40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:16.886 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.880 7fbdfbac0700 1 --2- 172.21.2.112:0/1240289284 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbdfc1aca40 0x7fbdfc1ace40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:16.886 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.880 7fbdfbac0700 1 -- 172.21.2.112:0/1240289284 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fbdfc19ede0 msgr2=0x7fbdfc19f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:16.886 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.880 7fbdfbac0700 1 --2- 172.21.2.112:0/1240289284 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fbdfc19ede0 0x7fbdfc19f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:16.886 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.880 7fbdfbac0700 1 -- 172.21.2.112:0/1240289284 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdfc19e000 con 0x7fbdfc19d3c0 2021-02-10T11:47:16.886 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.881 7fbdfbac0700 1 --2- 172.21.2.112:0/1240289284 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbdfc19d3c0 0x7fbdfc19d7c0 secure :-1 s=READY pgs=3013 cs=0 l=1 rev1=1 rx=0x7fbdf4009390 tx=0x7fbdf400bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:16.886 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.881 7fbdf3fff700 1 -- 172.21.2.112:0/1240289284 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbdf4003030 con 0x7fbdfc19d3c0 2021-02-10T11:47:16.887 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.881 7fbdf3fff700 1 -- 172.21.2.112:0/1240289284 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdf40068e0 con 0x7fbdfc19d3c0 2021-02-10T11:47:16.887 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.881 7fbdf3fff700 1 -- 172.21.2.112:0/1240289284 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbdf400c4e0 con 0x7fbdfc19d3c0 2021-02-10T11:47:16.887 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.881 7fbe015ea700 1 -- 172.21.2.112:0/1240289284 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdfc19e000 con 0x7fbdfc19d3c0 2021-02-10T11:47:16.887 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.881 7fbe015ea700 1 -- 172.21.2.112:0/1240289284 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdfc19e000 con 0x7fbdfc19d3c0 2021-02-10T11:47:16.887 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.882 7fbdf3fff700 1 -- 172.21.2.112:0/1240289284 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fbdf4001270 con 0x7fbdfc19d3c0 2021-02-10T11:47:16.887 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.882 7fbdf3fff700 1 --2- 172.21.2.112:0/1240289284 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fbde00186d0 0x7fbde001abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:16.887 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.882 7fbdf3fff700 1 -- 172.21.2.112:0/1240289284 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fbdf40329a0 con 0x7fbdfc19d3c0 2021-02-10T11:47:16.887 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.882 7fbdfb2bf700 1 --2- 172.21.2.112:0/1240289284 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fbde00186d0 0x7fbde001abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:16.887 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.883 7fbdfb2bf700 1 --2- 172.21.2.112:0/1240289284 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fbde00186d0 0x7fbde001abd0 crc :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:16.916 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.913 7fbe015ea700 1 -- 172.21.2.112:0/1240289284 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdfc19e8b0 con 0x7fbdfc19d3c0 2021-02-10T11:47:16.918 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:16.916 7fbdf3fff700 1 -- 172.21.2.112:0/1240289284 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fbdf4006c80 con 0x7fbdfc19d3c0 2021-02-10T11:47:17.049 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:17.047 7fbe015ea700 1 -- 172.21.2.112:0/1240289284 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbdfc19e8b0 con 0x7fbdfc19d3c0 2021-02-10T11:47:17.050 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:17.047 7fbdf3fff700 1 -- 172.21.2.112:0/1240289284 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 156 v156) v1 ==== 95+0+2108 (secure 0 0 0) 0x7fbdf4001270 con 0x7fbdfc19d3c0 2021-02-10T11:47:17.050 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:47:17.050 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":156,"default_fscid":19,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":13038,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/4134422613","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":4134422613},{"type":"v1","addr":"172.21.2.112:6833","nonce":4134422613}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":152}],"filesystems":[{"mdsmap":{"epoch":156,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:47:14.425711","modified":"2021-02-10 11:47:16.434766","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9307},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9307":{"gid":9307,"name":"b","rank":0,"incarnation":153,"state":"up:active","state_seq":3,"addr":"172.21.2.120:6835/2625088905","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":2625088905},{"type":"v1","addr":"172.21.2.120:6835","nonce":2625088905}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[39],"metadata_pool":38,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":19}]} 2021-02-10T11:47:17.051 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 156 2021-02-10T11:47:17.051 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:17.049 7fbe015ea700 1 -- 172.21.2.112:0/1240289284 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fbde00186d0 msgr2=0x7fbde001abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:17.051 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:17.049 7fbe015ea700 1 --2- 172.21.2.112:0/1240289284 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fbde00186d0 0x7fbde001abd0 crc :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:17.052 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:17.049 7fbe015ea700 1 -- 172.21.2.112:0/1240289284 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbdfc19d3c0 msgr2=0x7fbdfc19d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:17.052 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:17.049 7fbe015ea700 1 --2- 172.21.2.112:0/1240289284 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbdfc19d3c0 0x7fbdfc19d7c0 secure :-1 s=READY pgs=3013 cs=0 l=1 rev1=1 rx=0x7fbdf4009390 tx=0x7fbdf400bd40).stop 2021-02-10T11:47:17.052 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:17.049 7fbe015ea700 1 -- 172.21.2.112:0/1240289284 shutdown_connections 2021-02-10T11:47:17.052 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:17.049 7fbe015ea700 1 --2- 172.21.2.112:0/1240289284 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fbde00186d0 0x7fbde001abd0 unknown :-1 s=CLOSED pgs=1359 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:17.052 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:17.049 7fbe015ea700 1 --2- 172.21.2.112:0/1240289284 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbdfc19d3c0 0x7fbdfc19d7c0 unknown :-1 s=CLOSED pgs=3013 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:17.052 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:17.049 7fbe015ea700 1 --2- 172.21.2.112:0/1240289284 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbdfc1aca40 0x7fbdfc1ace40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:17.052 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:17.049 7fbe015ea700 1 --2- 172.21.2.112:0/1240289284 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fbdfc19ede0 0x7fbdfc19f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:17.052 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:17.049 7fbe015ea700 1 -- 172.21.2.112:0/1240289284 >> 172.21.2.112:0/1240289284 conn(0x7fbdfc19f880 msgr2=0x7fbdfc094ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:17.052 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:17.049 7fbe015ea700 1 -- 172.21.2.112:0/1240289284 shutdown_connections 2021-02-10T11:47:17.053 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:17.049 7fbe015ea700 1 -- 172.21.2.112:0/1240289284 wait complete. 2021-02-10T11:47:17.060 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:47:17.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:17.470 7fe18f1e8700 1 Processor -- start 2021-02-10T11:47:17.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:17.470 7fe18f1e8700 1 -- start start 2021-02-10T11:47:17.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:17.471 7fe18f1e8700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fe1880d1870 0x7fe1881b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:17.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:17.471 7fe18f1e8700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fe1880f0650 0x7fe1880cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:17.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:17.471 7fe18f1e8700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1880f0b10 con 0x7fe1881b0a10 2021-02-10T11:47:17.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:17.471 7fe18f1e8700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1880e5ed0 con 0x7fe1880f0650 2021-02-10T11:47:17.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:17.471 7fe18f1e8700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1880d1c70 con 0x7fe1880d1870 2021-02-10T11:47:17.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:17.471 7fe18d805700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fe1880f0650 0x7fe1880cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:17.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:17.471 7fe18d805700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fe1880f0650 0x7fe1880cfb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:60286/0 (socket says 172.21.2.112:60286) 2021-02-10T11:47:17.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:17.471 7fe18d805700 1 -- 172.21.2.112:0/3410716848 learned_addr learned my addr 172.21.2.112:0/3410716848 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:47:17.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:17.471 7fe18c803700 1 --1- 172.21.2.112:0/3410716848 >> v1:172.21.2.112:6789/0 conn(0x7fe1881b0a10 0x7fe1880cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:59748/0 (socket says 172.21.2.112:59748) 2021-02-10T11:47:17.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:17.471 7fe18d004700 1 --2- 172.21.2.112:0/3410716848 >> v2:172.21.2.120:3300/0 conn(0x7fe1880d1870 0x7fe1881b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:17.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:17.471 7fe17ffff700 1 -- 172.21.2.112:0/3410716848 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1977774197 0 0) 0x7fe1780009b0 con 0x7fe1881b0a10 2021-02-10T11:47:17.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:17.471 7fe17ffff700 1 -- 172.21.2.112:0/3410716848 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe174002fe0 con 0x7fe1881b0a10 2021-02-10T11:47:17.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:17.472 7fe18d805700 1 -- 172.21.2.112:0/3410716848 >> v2:172.21.2.120:3300/0 conn(0x7fe1880d1870 msgr2=0x7fe1881b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:17.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:17.472 7fe18d805700 1 --2- 172.21.2.112:0/3410716848 >> v2:172.21.2.120:3300/0 conn(0x7fe1880d1870 0x7fe1881b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:17.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:17.472 7fe18d805700 1 -- 172.21.2.112:0/3410716848 >> v1:172.21.2.112:6789/0 conn(0x7fe1881b0a10 legacy=0x7fe1880cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:17.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:17.472 7fe18d805700 1 -- 172.21.2.112:0/3410716848 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1880d1e90 con 0x7fe1880f0650 2021-02-10T11:47:17.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:17.472 7fe18d004700 1 --2- 172.21.2.112:0/3410716848 >> v2:172.21.2.120:3300/0 conn(0x7fe1880d1870 0x7fe1881b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:47:17.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:17.472 7fe18d805700 1 --2- 172.21.2.112:0/3410716848 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe1880f0650 0x7fe1880cfb20 secure :-1 s=READY pgs=3015 cs=0 l=1 rev1=1 rx=0x7fe180006f00 tx=0x7fe180009f10).ready entity=mon.0 client_cookie=90b6ab974ea110d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:17.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:17.472 7fe17ffff700 1 -- 172.21.2.112:0/3410716848 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe18000b3b0 con 0x7fe1880f0650 2021-02-10T11:47:17.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:17.472 7fe17ffff700 1 -- 172.21.2.112:0/3410716848 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe18000b630 con 0x7fe1880f0650 2021-02-10T11:47:17.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:17.472 7fe17ffff700 1 -- 172.21.2.112:0/3410716848 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe18000bb50 con 0x7fe1880f0650 2021-02-10T11:47:17.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:17.472 7fe18f1e8700 1 -- 172.21.2.112:0/3410716848 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe1880f0650 msgr2=0x7fe1880cfb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:17.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:17.472 7fe18f1e8700 1 --2- 172.21.2.112:0/3410716848 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe1880f0650 0x7fe1880cfb20 secure :-1 s=READY pgs=3015 cs=0 l=1 rev1=1 rx=0x7fe180006f00 tx=0x7fe180009f10).stop 2021-02-10T11:47:17.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:17.473 7fe18f1e8700 1 -- 172.21.2.112:0/3410716848 shutdown_connections 2021-02-10T11:47:17.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:17.473 7fe18f1e8700 1 --2- 172.21.2.112:0/3410716848 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe1880f0650 0x7fe1880cfb20 unknown :-1 s=CLOSED pgs=3015 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:17.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:17.473 7fe18f1e8700 1 --2- 172.21.2.112:0/3410716848 >> v2:172.21.2.120:3300/0 conn(0x7fe1880d1870 0x7fe1881b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:17.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:17.473 7fe18f1e8700 1 -- 172.21.2.112:0/3410716848 >> 172.21.2.112:0/3410716848 conn(0x7fe1880ce450 msgr2=0x7fe1880d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:17.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:17.473 7fe18f1e8700 1 -- 172.21.2.112:0/3410716848 shutdown_connections 2021-02-10T11:47:17.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:17.473 7fe18f1e8700 1 -- 172.21.2.112:0/3410716848 wait complete. 2021-02-10T11:47:17.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:17.487 7fe18f1e8700 1 Processor -- start 2021-02-10T11:47:17.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:17.487 7fe18f1e8700 1 -- start start 2021-02-10T11:47:17.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:17.488 7fe18f1e8700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe1880beeb0 0x7fe1880bf2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:17.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:17.488 7fe18f1e8700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe18818ef70 0x7fe18818f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:17.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:17.488 7fe18f1e8700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe18818f990 0x7fe1880c01d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:17.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:17.488 7fe18f1e8700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1880bfc50 con 0x7fe18818ef70 2021-02-10T11:47:17.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:17.488 7fe18f1e8700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe18818fd90 con 0x7fe1880beeb0 2021-02-10T11:47:17.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:17.488 7fe18f1e8700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1880bf8d0 con 0x7fe18818f990 2021-02-10T11:47:17.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:17.488 7fe18c803700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe18818ef70 0x7fe18818f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:17.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:17.488 7fe18c803700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe18818ef70 0x7fe18818f370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:60292/0 (socket says 172.21.2.112:60292) 2021-02-10T11:47:17.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:17.488 7fe18c803700 1 -- 172.21.2.112:0/2728579120 learned_addr learned my addr 172.21.2.112:0/2728579120 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:47:17.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:17.488 7fe18c803700 1 -- 172.21.2.112:0/2728579120 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe18818f990 msgr2=0x7fe1880c01d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:17.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:17.488 7fe18c803700 1 --2- 172.21.2.112:0/2728579120 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe18818f990 0x7fe1880c01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:17.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:17.488 7fe18c803700 1 -- 172.21.2.112:0/2728579120 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe1880beeb0 msgr2=0x7fe1880bf2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:17.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:17.488 7fe18c803700 1 --2- 172.21.2.112:0/2728579120 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe1880beeb0 0x7fe1880bf2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:17.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:17.488 7fe18c803700 1 -- 172.21.2.112:0/2728579120 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1880c07f0 con 0x7fe18818ef70 2021-02-10T11:47:17.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:17.489 7fe18c803700 1 --2- 172.21.2.112:0/2728579120 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe18818ef70 0x7fe18818f370 secure :-1 s=READY pgs=3016 cs=0 l=1 rev1=1 rx=0x7fe17800a4c0 tx=0x7fe17800a860).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:17.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:17.489 7fe17dffb700 1 -- 172.21.2.112:0/2728579120 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe17800ac50 con 0x7fe18818ef70 2021-02-10T11:47:17.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:17.489 7fe17dffb700 1 -- 172.21.2.112:0/2728579120 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe178004350 con 0x7fe18818ef70 2021-02-10T11:47:17.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:17.489 7fe17dffb700 1 -- 172.21.2.112:0/2728579120 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe1780047b0 con 0x7fe18818ef70 2021-02-10T11:47:17.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:17.489 7fe18f1e8700 1 -- 172.21.2.112:0/2728579120 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1880c07f0 con 0x7fe18818ef70 2021-02-10T11:47:17.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:17.489 7fe18f1e8700 1 -- 172.21.2.112:0/2728579120 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1880c0d80 con 0x7fe18818ef70 2021-02-10T11:47:17.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:17.489 7fe17dffb700 1 -- 172.21.2.112:0/2728579120 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe178004530 con 0x7fe18818ef70 2021-02-10T11:47:17.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:17.490 7fe17dffb700 1 --2- 172.21.2.112:0/2728579120 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe1680186c0 0x7fe16801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:17.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:17.490 7fe17dffb700 1 -- 172.21.2.112:0/2728579120 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fe1780323d0 con 0x7fe18818ef70 2021-02-10T11:47:17.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:17.490 7fe18d004700 1 --2- 172.21.2.112:0/2728579120 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe1680186c0 0x7fe16801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:17.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:17.490 7fe18d004700 1 --2- 172.21.2.112:0/2728579120 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe1680186c0 0x7fe16801abc0 crc :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:17.524 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:17.521 7fe18f1e8700 1 -- 172.21.2.112:0/2728579120 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1880ebc60 con 0x7fe18818ef70 2021-02-10T11:47:17.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:17.524 7fe17dffb700 1 -- 172.21.2.112:0/2728579120 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe178000960 con 0x7fe18818ef70 2021-02-10T11:47:17.652 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:17.649 7fe18f1e8700 1 -- 172.21.2.112:0/2728579120 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe1880f0b10 con 0x7fe18818ef70 2021-02-10T11:47:17.653 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:17.650 7fe17dffb700 1 -- 172.21.2.112:0/2728579120 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v180) v1 ==== 74+0+14007 (secure 0 0 0) 0x7fe178004420 con 0x7fe18818ef70 2021-02-10T11:47:17.653 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:47:17.653 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":180,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:47:15.427174","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":39,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":38,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:47:12.395866","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"179","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":39,"pool_name":"cephfs_data","create_time":"2021-02-10 11:47:13.420625","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"180","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:47:17.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:17.651 7fe18f1e8700 1 -- 172.21.2.112:0/2728579120 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe1680186c0 msgr2=0x7fe16801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:17.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:17.651 7fe18f1e8700 1 --2- 172.21.2.112:0/2728579120 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe1680186c0 0x7fe16801abc0 crc :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:17.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:17.651 7fe18f1e8700 1 -- 172.21.2.112:0/2728579120 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe18818ef70 msgr2=0x7fe18818f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:17.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:17.651 7fe18f1e8700 1 --2- 172.21.2.112:0/2728579120 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe18818ef70 0x7fe18818f370 secure :-1 s=READY pgs=3016 cs=0 l=1 rev1=1 rx=0x7fe17800a4c0 tx=0x7fe17800a860).stop 2021-02-10T11:47:17.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:17.651 7fe18f1e8700 1 -- 172.21.2.112:0/2728579120 shutdown_connections 2021-02-10T11:47:17.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:17.651 7fe18f1e8700 1 --2- 172.21.2.112:0/2728579120 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe1680186c0 0x7fe16801abc0 unknown :-1 s=CLOSED pgs=1360 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:17.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:17.651 7fe18f1e8700 1 --2- 172.21.2.112:0/2728579120 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe18818f990 0x7fe1880c01d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:17.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:17.651 7fe18f1e8700 1 --2- 172.21.2.112:0/2728579120 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe18818ef70 0x7fe18818f370 unknown :-1 s=CLOSED pgs=3016 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:17.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:17.651 7fe18f1e8700 1 --2- 172.21.2.112:0/2728579120 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe1880beeb0 0x7fe1880bf2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:17.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:17.651 7fe18f1e8700 1 -- 172.21.2.112:0/2728579120 >> 172.21.2.112:0/2728579120 conn(0x7fe1880c2130 msgr2=0x7fe18809e3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:17.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:17.652 7fe18f1e8700 1 -- 172.21.2.112:0/2728579120 shutdown_connections 2021-02-10T11:47:17.656 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:17.652 7fe18f1e8700 1 -- 172.21.2.112:0/2728579120 wait complete. 2021-02-10T11:47:17.664 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:47:18.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.087 7f03e2f93700 1 Processor -- start 2021-02-10T11:47:18.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.087 7f03e2f93700 1 -- start start 2021-02-10T11:47:18.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.087 7f03e2f93700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f03dc0d1880 0x7f03dc1b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:18.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.087 7f03e2f93700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f03dc0cff80 0x7f03dc109140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:18.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.087 7f03e2f93700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03dc109a70 con 0x7f03dc1b0af0 2021-02-10T11:47:18.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.087 7f03e2f93700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03dc109760 con 0x7f03dc0d1880 2021-02-10T11:47:18.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.087 7f03e2f93700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03dc088070 con 0x7f03dc0cff80 2021-02-10T11:47:18.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.087 7f03e0daf700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f03dc0d1880 0x7f03dc1b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:18.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.087 7f03e0daf700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f03dc0d1880 0x7f03dc1b04d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:60300/0 (socket says 172.21.2.112:60300) 2021-02-10T11:47:18.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.087 7f03e0daf700 1 -- 172.21.2.112:0/2362735684 learned_addr learned my addr 172.21.2.112:0/2362735684 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:47:18.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.088 7f03e0daf700 1 -- 172.21.2.112:0/2362735684 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f03dc0cff80 msgr2=0x7f03dc109140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:18.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.088 7f03e0daf700 1 --2- 172.21.2.112:0/2362735684 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f03dc0cff80 0x7f03dc109140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:18.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.088 7f03e0daf700 1 -- 172.21.2.112:0/2362735684 >> v1:172.21.2.120:6789/0 conn(0x7f03dc1b0af0 legacy=0x7f03dc108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:18.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.088 7f03e0daf700 1 -- 172.21.2.112:0/2362735684 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03dc0d1c80 con 0x7f03dc0d1880 2021-02-10T11:47:18.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.088 7f03e0daf700 1 --2- 172.21.2.112:0/2362735684 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f03dc0d1880 0x7f03dc1b04d0 secure :-1 s=READY pgs=3017 cs=0 l=1 rev1=1 rx=0x7f03c4006700 tx=0x7f03c4009b00).ready entity=mon.0 client_cookie=e571208a365dc360 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:18.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.088 7f03db7fe700 1 -- 172.21.2.112:0/2362735684 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f03c40009b0 con 0x7f03dc0d1880 2021-02-10T11:47:18.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.088 7f03db7fe700 1 -- 172.21.2.112:0/2362735684 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03c400c0c0 con 0x7f03dc0d1880 2021-02-10T11:47:18.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.088 7f03db7fe700 1 -- 172.21.2.112:0/2362735684 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f03c400c5c0 con 0x7f03dc0d1880 2021-02-10T11:47:18.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.088 7f03e2f93700 1 -- 172.21.2.112:0/2362735684 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f03dc0d1880 msgr2=0x7f03dc1b04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:18.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.088 7f03e2f93700 1 --2- 172.21.2.112:0/2362735684 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f03dc0d1880 0x7f03dc1b04d0 secure :-1 s=READY pgs=3017 cs=0 l=1 rev1=1 rx=0x7f03c4006700 tx=0x7f03c4009b00).stop 2021-02-10T11:47:18.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.089 7f03e2f93700 1 -- 172.21.2.112:0/2362735684 shutdown_connections 2021-02-10T11:47:18.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.089 7f03e2f93700 1 --2- 172.21.2.112:0/2362735684 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f03dc0cff80 0x7f03dc109140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:18.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.089 7f03e2f93700 1 --2- 172.21.2.112:0/2362735684 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f03dc0d1880 0x7f03dc1b04d0 unknown :-1 s=CLOSED pgs=3017 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:18.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.089 7f03e2f93700 1 -- 172.21.2.112:0/2362735684 >> 172.21.2.112:0/2362735684 conn(0x7f03dc0ce430 msgr2=0x7f03dc0d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:18.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.089 7f03e2f93700 1 -- 172.21.2.112:0/2362735684 shutdown_connections 2021-02-10T11:47:18.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.089 7f03e2f93700 1 -- 172.21.2.112:0/2362735684 wait complete. 2021-02-10T11:47:18.109 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.103 7f03e2f93700 1 Processor -- start 2021-02-10T11:47:18.109 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.103 7f03e2f93700 1 -- start start 2021-02-10T11:47:18.109 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.104 7f03e2f93700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f03dc086810 0x7f03dc19fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:18.109 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.104 7f03e2f93700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f03dc0b04d0 0x7f03dc097450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:18.110 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.104 7f03e2f93700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f03dc0b08d0 0x7f03dc097a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:18.110 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.104 7f03e2f93700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03dc1a02c0 con 0x7f03dc0b04d0 2021-02-10T11:47:18.110 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.104 7f03e2f93700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03dc098090 con 0x7f03dc0b08d0 2021-02-10T11:47:18.110 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.104 7f03e2f93700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03dc0982b0 con 0x7f03dc086810 2021-02-10T11:47:18.110 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.104 7f03dbfff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f03dc0b04d0 0x7f03dc097450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:18.110 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.104 7f03dbfff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f03dc0b04d0 0x7f03dc097450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:60304/0 (socket says 172.21.2.112:60304) 2021-02-10T11:47:18.110 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.104 7f03dbfff700 1 -- 172.21.2.112:0/2943043399 learned_addr learned my addr 172.21.2.112:0/2943043399 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:47:18.110 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.104 7f03dbfff700 1 -- 172.21.2.112:0/2943043399 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f03dc086810 msgr2=0x7f03dc19fca0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:47:18.110 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.104 7f03dbfff700 1 --2- 172.21.2.112:0/2943043399 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f03dc086810 0x7f03dc19fca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:18.110 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.104 7f03dbfff700 1 -- 172.21.2.112:0/2943043399 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f03dc0b08d0 msgr2=0x7f03dc097a70 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:47:18.110 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.104 7f03dbfff700 1 --2- 172.21.2.112:0/2943043399 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f03dc0b08d0 0x7f03dc097a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:18.110 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.104 7f03dbfff700 1 -- 172.21.2.112:0/2943043399 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03dc0984d0 con 0x7f03dc0b04d0 2021-02-10T11:47:18.110 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.104 7f03dbfff700 1 --2- 172.21.2.112:0/2943043399 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f03dc0b04d0 0x7f03dc097450 secure :-1 s=READY pgs=3018 cs=0 l=1 rev1=1 rx=0x7f03cc00b5b0 tx=0x7f03cc00b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:18.111 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.104 7f03d97fa700 1 -- 172.21.2.112:0/2943043399 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f03cc00bd40 con 0x7f03dc0b04d0 2021-02-10T11:47:18.111 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.104 7f03d97fa700 1 -- 172.21.2.112:0/2943043399 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03cc0009b0 con 0x7f03dc0b04d0 2021-02-10T11:47:18.111 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.104 7f03d97fa700 1 -- 172.21.2.112:0/2943043399 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f03cc006210 con 0x7f03dc0b04d0 2021-02-10T11:47:18.111 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.104 7f03e2f93700 1 -- 172.21.2.112:0/2943043399 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03dc0984d0 con 0x7f03dc0b04d0 2021-02-10T11:47:18.111 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.105 7f03e2f93700 1 -- 172.21.2.112:0/2943043399 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03dc0984d0 con 0x7f03dc0b04d0 2021-02-10T11:47:18.111 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.105 7f03d97fa700 1 -- 172.21.2.112:0/2943043399 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f03cc006020 con 0x7f03dc0b04d0 2021-02-10T11:47:18.111 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.105 7f03d97fa700 1 --2- 172.21.2.112:0/2943043399 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f03c00186d0 0x7f03c001abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:18.111 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.105 7f03d97fa700 1 -- 172.21.2.112:0/2943043399 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f03cc0327c0 con 0x7f03dc0b04d0 2021-02-10T11:47:18.111 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.106 7f03e0daf700 1 --2- 172.21.2.112:0/2943043399 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f03c00186d0 0x7f03c001abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:18.111 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.107 7f03e0daf700 1 --2- 172.21.2.112:0/2943043399 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f03c00186d0 0x7f03c001abd0 crc :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:18.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.137 7f03e2f93700 1 -- 172.21.2.112:0/2943043399 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03dc0fe1d0 con 0x7f03dc0b04d0 2021-02-10T11:47:18.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.139 7f03d97fa700 1 -- 172.21.2.112:0/2943043399 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f03cc00bd40 con 0x7f03dc0b04d0 2021-02-10T11:47:18.274 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.271 7f03e2f93700 1 -- 172.21.2.112:0/2943043399 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f03dc0fe1d0 con 0x7f03dc0b04d0 2021-02-10T11:47:18.277 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.275 7f03d97fa700 1 -- 172.21.2.112:0/2943043399 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v102) v1 ==== 184+0+0 (secure 0 0 0) 0x7f03cc006020 con 0x7f03dc0b04d0 2021-02-10T11:47:18.281 INFO:teuthology.orchestra.run.gibba012.stderr:updated caps for client.0 2021-02-10T11:47:18.283 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.280 7f03e2f93700 1 -- 172.21.2.112:0/2943043399 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f03c00186d0 msgr2=0x7f03c001abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:18.283 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.280 7f03e2f93700 1 --2- 172.21.2.112:0/2943043399 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f03c00186d0 0x7f03c001abd0 crc :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:18.283 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.280 7f03e2f93700 1 -- 172.21.2.112:0/2943043399 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f03dc0b04d0 msgr2=0x7f03dc097450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:18.283 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.280 7f03e2f93700 1 --2- 172.21.2.112:0/2943043399 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f03dc0b04d0 0x7f03dc097450 secure :-1 s=READY pgs=3018 cs=0 l=1 rev1=1 rx=0x7f03cc00b5b0 tx=0x7f03cc00b950).stop 2021-02-10T11:47:18.283 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.280 7f03e2f93700 1 -- 172.21.2.112:0/2943043399 shutdown_connections 2021-02-10T11:47:18.283 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.280 7f03e2f93700 1 --2- 172.21.2.112:0/2943043399 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f03c00186d0 0x7f03c001abd0 unknown :-1 s=CLOSED pgs=1361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:18.283 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.280 7f03e2f93700 1 --2- 172.21.2.112:0/2943043399 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f03dc0b08d0 0x7f03dc097a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:18.283 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.280 7f03e2f93700 1 --2- 172.21.2.112:0/2943043399 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f03dc0b04d0 0x7f03dc097450 unknown :-1 s=CLOSED pgs=3018 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:18.284 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.280 7f03e2f93700 1 --2- 172.21.2.112:0/2943043399 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f03dc086810 0x7f03dc19fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:18.284 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.280 7f03e2f93700 1 -- 172.21.2.112:0/2943043399 >> 172.21.2.112:0/2943043399 conn(0x7f03dc19f880 msgr2=0x7f03dc08f890 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:18.284 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.280 7f03e2f93700 1 -- 172.21.2.112:0/2943043399 shutdown_connections 2021-02-10T11:47:18.284 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.280 7f03e2f93700 1 -- 172.21.2.112:0/2943043399 wait complete. 2021-02-10T11:47:18.292 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:47:18.733 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.713 7f126ebb9700 1 Processor -- start 2021-02-10T11:47:18.733 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.713 7f126ebb9700 1 -- start start 2021-02-10T11:47:18.734 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.714 7f126ebb9700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f12680d1880 0x7f12681b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:18.734 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.714 7f126ebb9700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f12681b0af0 0x7f1268108950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:18.734 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.714 7f126ebb9700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1268109a70 con 0x7f12680d1c80 2021-02-10T11:47:18.734 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.714 7f126ebb9700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12680e5f00 con 0x7f12681b0af0 2021-02-10T11:47:18.734 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.714 7f126ebb9700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1268109760 con 0x7f12680d1880 2021-02-10T11:47:18.734 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.714 7f1267fff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f12681b0af0 0x7f1268108950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:18.734 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.714 7f1267fff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f12681b0af0 0x7f1268108950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:60314/0 (socket says 172.21.2.112:60314) 2021-02-10T11:47:18.734 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.714 7f1267fff700 1 -- 172.21.2.112:0/985900716 learned_addr learned my addr 172.21.2.112:0/985900716 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:47:18.734 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.714 7f126d1d6700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f12680d1c80 0x7f1268108f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:59774/0 (socket says 172.21.2.112:59774) 2021-02-10T11:47:18.734 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.715 7f1267fff700 1 -- 172.21.2.112:0/985900716 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f12680d1880 msgr2=0x7f12681b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:18.734 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.715 7f1267fff700 1 --2- 172.21.2.112:0/985900716 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f12680d1880 0x7f12681b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:18.734 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.715 7f1267fff700 1 -- 172.21.2.112:0/985900716 >> v1:172.21.2.112:6789/0 conn(0x7f12680d1c80 legacy=0x7f1268108f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:18.735 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.715 7f1267fff700 1 -- 172.21.2.112:0/985900716 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1268088070 con 0x7f12681b0af0 2021-02-10T11:47:18.735 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.715 7f1267fff700 1 --2- 172.21.2.112:0/985900716 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f12681b0af0 0x7f1268108950 secure :-1 s=READY pgs=3020 cs=0 l=1 rev1=1 rx=0x7f1258006a80 tx=0x7f1258009f00).ready entity=mon.0 client_cookie=eb0a9725dc6be637 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:18.735 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.715 7f12677fe700 1 -- 172.21.2.112:0/985900716 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f125800b5c0 con 0x7f12681b0af0 2021-02-10T11:47:18.735 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.715 7f12677fe700 1 -- 172.21.2.112:0/985900716 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f125800b840 con 0x7f12681b0af0 2021-02-10T11:47:18.735 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.715 7f12677fe700 1 -- 172.21.2.112:0/985900716 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f125800bca0 con 0x7f12681b0af0 2021-02-10T11:47:18.735 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.715 7f126ebb9700 1 -- 172.21.2.112:0/985900716 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f12681b0af0 msgr2=0x7f1268108950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:18.735 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.715 7f126ebb9700 1 --2- 172.21.2.112:0/985900716 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f12681b0af0 0x7f1268108950 secure :-1 s=READY pgs=3020 cs=0 l=1 rev1=1 rx=0x7f1258006a80 tx=0x7f1258009f00).stop 2021-02-10T11:47:18.735 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.716 7f126ebb9700 1 -- 172.21.2.112:0/985900716 shutdown_connections 2021-02-10T11:47:18.735 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.716 7f126ebb9700 1 --2- 172.21.2.112:0/985900716 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f12681b0af0 0x7f1268108950 unknown :-1 s=CLOSED pgs=3020 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:18.735 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.716 7f126ebb9700 1 --2- 172.21.2.112:0/985900716 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f12680d1880 0x7f12681b04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:18.735 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.716 7f126ebb9700 1 -- 172.21.2.112:0/985900716 >> 172.21.2.112:0/985900716 conn(0x7f12680ce430 msgr2=0x7f12680d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:18.735 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.716 7f126ebb9700 1 -- 172.21.2.112:0/985900716 shutdown_connections 2021-02-10T11:47:18.735 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.716 7f126ebb9700 1 -- 172.21.2.112:0/985900716 wait complete. 2021-02-10T11:47:18.737 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.731 7f126ebb9700 1 Processor -- start 2021-02-10T11:47:18.737 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.731 7f126ebb9700 1 -- start start 2021-02-10T11:47:18.737 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.731 7f126ebb9700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f12681b9470 0x7f12681ae6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:18.737 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.731 7f126ebb9700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f12681aecf0 0x7f12680957f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:18.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.731 7f126ebb9700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f12681af0f0 0x7f1268095e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:18.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.731 7f126ebb9700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1268096d70 con 0x7f12681aecf0 2021-02-10T11:47:18.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.731 7f126ebb9700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12680a7780 con 0x7f12681af0f0 2021-02-10T11:47:18.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.731 7f126ebb9700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12681b9870 con 0x7f12681b9470 2021-02-10T11:47:18.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.732 7f1267fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f12681aecf0 0x7f12680957f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:18.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.732 7f1267fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f12681aecf0 0x7f12680957f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:60318/0 (socket says 172.21.2.112:60318) 2021-02-10T11:47:18.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.732 7f1267fff700 1 -- 172.21.2.112:0/1356913363 learned_addr learned my addr 172.21.2.112:0/1356913363 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:47:18.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.732 7f1267fff700 1 -- 172.21.2.112:0/1356913363 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f12681b9470 msgr2=0x7f12681ae6d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:47:18.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.732 7f1267fff700 1 --2- 172.21.2.112:0/1356913363 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f12681b9470 0x7f12681ae6d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:18.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.732 7f1267fff700 1 -- 172.21.2.112:0/1356913363 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f12681af0f0 msgr2=0x7f1268095e10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:47:18.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.732 7f1267fff700 1 --2- 172.21.2.112:0/1356913363 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f12681af0f0 0x7f1268095e10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:18.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.732 7f1267fff700 1 -- 172.21.2.112:0/1356913363 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f126819e6b0 con 0x7f12681aecf0 2021-02-10T11:47:18.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.732 7f1267fff700 1 --2- 172.21.2.112:0/1356913363 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f12681aecf0 0x7f12680957f0 secure :-1 s=READY pgs=3021 cs=0 l=1 rev1=1 rx=0x7f1258009450 tx=0x7f125800bd20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:18.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.732 7f12657fa700 1 -- 172.21.2.112:0/1356913363 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f125800c2f0 con 0x7f12681aecf0 2021-02-10T11:47:18.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.732 7f12657fa700 1 -- 172.21.2.112:0/1356913363 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f125800be60 con 0x7f12681aecf0 2021-02-10T11:47:18.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.732 7f12657fa700 1 -- 172.21.2.112:0/1356913363 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f125800c6e0 con 0x7f12681aecf0 2021-02-10T11:47:18.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.732 7f126ebb9700 1 -- 172.21.2.112:0/1356913363 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f126819e6b0 con 0x7f12681aecf0 2021-02-10T11:47:18.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.733 7f126ebb9700 1 -- 172.21.2.112:0/1356913363 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f126819e6b0 con 0x7f12681aecf0 2021-02-10T11:47:18.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.733 7f12657fa700 1 -- 172.21.2.112:0/1356913363 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1258005660 con 0x7f12681aecf0 2021-02-10T11:47:18.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.733 7f12657fa700 1 --2- 172.21.2.112:0/1356913363 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f124c0186c0 0x7f124c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:18.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.733 7f12657fa700 1 -- 172.21.2.112:0/1356913363 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f1258032b20 con 0x7f12681aecf0 2021-02-10T11:47:18.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.734 7f126c9d5700 1 --2- 172.21.2.112:0/1356913363 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f124c0186c0 0x7f124c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:18.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.734 7f126c9d5700 1 --2- 172.21.2.112:0/1356913363 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f124c0186c0 0x7f124c01abc0 crc :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:18.766 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.763 7f126ebb9700 1 -- 172.21.2.112:0/1356913363 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12680829a0 con 0x7f12681aecf0 2021-02-10T11:47:18.769 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.766 7f12657fa700 1 -- 172.21.2.112:0/1356913363 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f125801f860 con 0x7f12681aecf0 2021-02-10T11:47:18.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.896 7f126ebb9700 1 -- 172.21.2.112:0/1356913363 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f12680a7780 con 0x7f12681aecf0 2021-02-10T11:47:18.902 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.899 7f12657fa700 1 -- 172.21.2.112:0/1356913363 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v103) v1 ==== 184+0+0 (secure 0 0 0) 0x7f1258005660 con 0x7f12681aecf0 2021-02-10T11:47:18.906 INFO:teuthology.orchestra.run.gibba012.stderr:updated caps for client.1 2021-02-10T11:47:18.907 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.904 7f126ebb9700 1 -- 172.21.2.112:0/1356913363 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f124c0186c0 msgr2=0x7f124c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:18.907 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.904 7f126ebb9700 1 --2- 172.21.2.112:0/1356913363 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f124c0186c0 0x7f124c01abc0 crc :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:18.907 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.904 7f126ebb9700 1 -- 172.21.2.112:0/1356913363 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f12681aecf0 msgr2=0x7f12680957f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:18.907 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.904 7f126ebb9700 1 --2- 172.21.2.112:0/1356913363 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f12681aecf0 0x7f12680957f0 secure :-1 s=READY pgs=3021 cs=0 l=1 rev1=1 rx=0x7f1258009450 tx=0x7f125800bd20).stop 2021-02-10T11:47:18.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.905 7f126ebb9700 1 -- 172.21.2.112:0/1356913363 shutdown_connections 2021-02-10T11:47:18.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.905 7f126ebb9700 1 --2- 172.21.2.112:0/1356913363 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f124c0186c0 0x7f124c01abc0 unknown :-1 s=CLOSED pgs=1362 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:18.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.905 7f126ebb9700 1 --2- 172.21.2.112:0/1356913363 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f12681af0f0 0x7f1268095e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:18.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.905 7f126ebb9700 1 --2- 172.21.2.112:0/1356913363 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f12681aecf0 0x7f12680957f0 unknown :-1 s=CLOSED pgs=3021 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:18.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.905 7f126ebb9700 1 --2- 172.21.2.112:0/1356913363 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f12681b9470 0x7f12681ae6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:18.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.905 7f126ebb9700 1 -- 172.21.2.112:0/1356913363 >> 172.21.2.112:0/1356913363 conn(0x7f12680845b0 msgr2=0x7f126818c290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:18.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.905 7f126ebb9700 1 -- 172.21.2.112:0/1356913363 shutdown_connections 2021-02-10T11:47:18.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:18.905 7f126ebb9700 1 -- 172.21.2.112:0/1356913363 wait complete. 2021-02-10T11:47:18.916 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.2 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:47:19.348 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.328 7f5cef02b700 1 Processor -- start 2021-02-10T11:47:19.348 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.328 7f5cef02b700 1 -- start start 2021-02-10T11:47:19.348 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.328 7f5cef02b700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f5ce810b000 0x7f5ce80cf760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:19.348 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.329 7f5cef02b700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ce80fe1d0 con 0x7f5ce80d1880 2021-02-10T11:47:19.349 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.329 7f5cef02b700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ce81b0cc0 con 0x7f5ce80d1c80 2021-02-10T11:47:19.349 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.329 7f5cef02b700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ce8109a70 con 0x7f5ce810b000 2021-02-10T11:47:19.349 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.329 7f5cece47700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f5ce80d1880 0x7f5ce81b04d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:59786/0 (socket says 172.21.2.112:59786) 2021-02-10T11:47:19.349 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.329 7f5cece47700 1 -- 172.21.2.112:0/1687699535 learned_addr learned my addr 172.21.2.112:0/1687699535 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:47:19.349 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.329 7f5ced648700 1 --2- 172.21.2.112:0/1687699535 >> v2:172.21.2.120:3300/0 conn(0x7f5ce810b000 0x7f5ce80cf760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:19.349 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.329 7f5ce77fe700 1 -- 172.21.2.112:0/1687699535 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 517248897 0 0) 0x7f5cd0001180 con 0x7f5ce80d1880 2021-02-10T11:47:19.349 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.329 7f5ce77fe700 1 -- 172.21.2.112:0/1687699535 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5cd4002fe0 con 0x7f5ce80d1880 2021-02-10T11:47:19.349 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.330 7f5ced648700 1 -- 172.21.2.112:0/1687699535 >> v1:172.21.2.120:6789/0 conn(0x7f5ce80d1c80 legacy=0x7f5ce80cef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:19.349 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.330 7f5ced648700 1 -- 172.21.2.112:0/1687699535 >> v1:172.21.2.112:6789/0 conn(0x7f5ce80d1880 legacy=0x7f5ce81b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:19.349 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.330 7f5ced648700 1 -- 172.21.2.112:0/1687699535 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ce810b400 con 0x7f5ce810b000 2021-02-10T11:47:19.349 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.330 7f5ced648700 1 --2- 172.21.2.112:0/1687699535 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5ce810b000 0x7f5ce80cf760 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 rx=0x7f5ce0006e50 tx=0x7f5ce0009e70).ready entity=mon.1 client_cookie=8a5c6c56ca29edf8 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:19.349 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.330 7f5ce77fe700 1 -- 172.21.2.112:0/1687699535 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5ce0006a20 con 0x7f5ce810b000 2021-02-10T11:47:19.350 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.330 7f5ce77fe700 1 -- 172.21.2.112:0/1687699535 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ce0006790 con 0x7f5ce810b000 2021-02-10T11:47:19.350 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.330 7f5ce77fe700 1 -- 172.21.2.112:0/1687699535 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5ce000b700 con 0x7f5ce810b000 2021-02-10T11:47:19.350 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.330 7f5cef02b700 1 -- 172.21.2.112:0/1687699535 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5ce810b000 msgr2=0x7f5ce80cf760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:19.350 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.330 7f5cef02b700 1 --2- 172.21.2.112:0/1687699535 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5ce810b000 0x7f5ce80cf760 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 rx=0x7f5ce0006e50 tx=0x7f5ce0009e70).stop 2021-02-10T11:47:19.350 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.331 7f5cef02b700 1 -- 172.21.2.112:0/1687699535 shutdown_connections 2021-02-10T11:47:19.350 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.331 7f5cef02b700 1 --2- 172.21.2.112:0/1687699535 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5ce810b000 0x7f5ce80cf760 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:19.350 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.331 7f5cef02b700 1 -- 172.21.2.112:0/1687699535 >> 172.21.2.112:0/1687699535 conn(0x7f5ce80ce430 msgr2=0x7f5ce80d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:19.350 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.331 7f5cef02b700 1 -- 172.21.2.112:0/1687699535 shutdown_connections 2021-02-10T11:47:19.350 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.331 7f5cef02b700 1 -- 172.21.2.112:0/1687699535 wait complete. 2021-02-10T11:47:19.352 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.346 7f5cef02b700 1 Processor -- start 2021-02-10T11:47:19.352 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.346 7f5cef02b700 1 -- start start 2021-02-10T11:47:19.352 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.346 7f5cef02b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5ce8086810 0x7f5ce819fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:19.352 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.346 7f5cef02b700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5ce80b04d0 0x7f5ce8097450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:19.352 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.346 7f5cef02b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5ce80b08d0 0x7f5ce8097a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:19.352 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.346 7f5cef02b700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ce81a02c0 con 0x7f5ce80b08d0 2021-02-10T11:47:19.352 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.346 7f5cef02b700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ce8098090 con 0x7f5ce80b04d0 2021-02-10T11:47:19.352 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.346 7f5cef02b700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ce80982b0 con 0x7f5ce8086810 2021-02-10T11:47:19.352 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.347 7f5ced648700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5ce80b08d0 0x7f5ce8097a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:19.352 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.347 7f5ced648700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5ce80b08d0 0x7f5ce8097a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:60330/0 (socket says 172.21.2.112:60330) 2021-02-10T11:47:19.352 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.347 7f5ced648700 1 -- 172.21.2.112:0/3436884833 learned_addr learned my addr 172.21.2.112:0/3436884833 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:47:19.352 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.347 7f5ced648700 1 -- 172.21.2.112:0/3436884833 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5ce8086810 msgr2=0x7f5ce819fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:19.353 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.347 7f5ced648700 1 --2- 172.21.2.112:0/3436884833 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5ce8086810 0x7f5ce819fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:19.353 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.347 7f5ced648700 1 -- 172.21.2.112:0/3436884833 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5ce80b04d0 msgr2=0x7f5ce8097450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:19.353 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.347 7f5ced648700 1 --2- 172.21.2.112:0/3436884833 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5ce80b04d0 0x7f5ce8097450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:19.353 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.347 7f5ced648700 1 -- 172.21.2.112:0/3436884833 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ce80984d0 con 0x7f5ce80b08d0 2021-02-10T11:47:19.353 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.347 7f5ced648700 1 --2- 172.21.2.112:0/3436884833 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5ce80b08d0 0x7f5ce8097a70 secure :-1 s=READY pgs=3023 cs=0 l=1 rev1=1 rx=0x7f5ce00099d0 tx=0x7f5ce000bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:19.353 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.347 7f5ce57fa700 1 -- 172.21.2.112:0/3436884833 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5ce00082e0 con 0x7f5ce80b08d0 2021-02-10T11:47:19.353 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.348 7f5ce57fa700 1 -- 172.21.2.112:0/3436884833 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ce0008560 con 0x7f5ce80b08d0 2021-02-10T11:47:19.353 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.348 7f5ce57fa700 1 -- 172.21.2.112:0/3436884833 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5ce0008930 con 0x7f5ce80b08d0 2021-02-10T11:47:19.353 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.348 7f5cef02b700 1 -- 172.21.2.112:0/3436884833 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ce80984d0 con 0x7f5ce80b08d0 2021-02-10T11:47:19.353 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.348 7f5cef02b700 1 -- 172.21.2.112:0/3436884833 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ce8098b80 con 0x7f5ce80b08d0 2021-02-10T11:47:19.353 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.348 7f5ce57fa700 1 -- 172.21.2.112:0/3436884833 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5ce0008740 con 0x7f5ce80b08d0 2021-02-10T11:47:19.353 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.348 7f5ce57fa700 1 --2- 172.21.2.112:0/3436884833 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5ccc0186c0 0x7f5ccc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:19.353 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.348 7f5ce57fa700 1 -- 172.21.2.112:0/3436884833 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f5ce00327c0 con 0x7f5ce80b08d0 2021-02-10T11:47:19.354 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.349 7f5cece47700 1 --2- 172.21.2.112:0/3436884833 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5ccc0186c0 0x7f5ccc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:19.354 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.349 7f5cece47700 1 --2- 172.21.2.112:0/3436884833 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5ccc0186c0 0x7f5ccc01abc0 crc :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:19.381 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.378 7f5cef02b700 1 -- 172.21.2.112:0/3436884833 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ce80fe1d0 con 0x7f5ce80b08d0 2021-02-10T11:47:19.383 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.381 7f5ce57fa700 1 -- 172.21.2.112:0/3436884833 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5ce00093d0 con 0x7f5ce80b08d0 2021-02-10T11:47:19.513 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.510 7f5cef02b700 1 -- 172.21.2.112:0/3436884833 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f5ce80fe1d0 con 0x7f5ce80b08d0 2021-02-10T11:47:19.516 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.513 7f5ce57fa700 1 -- 172.21.2.112:0/3436884833 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.2 v104) v1 ==== 184+0+0 (secure 0 0 0) 0x7f5ce0008620 con 0x7f5ce80b08d0 2021-02-10T11:47:19.520 INFO:teuthology.orchestra.run.gibba012.stderr:updated caps for client.2 2021-02-10T11:47:19.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.519 7f5cef02b700 1 -- 172.21.2.112:0/3436884833 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5ccc0186c0 msgr2=0x7f5ccc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:19.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.519 7f5cef02b700 1 --2- 172.21.2.112:0/3436884833 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5ccc0186c0 0x7f5ccc01abc0 crc :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:19.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.519 7f5cef02b700 1 -- 172.21.2.112:0/3436884833 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5ce80b08d0 msgr2=0x7f5ce8097a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:19.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.519 7f5cef02b700 1 --2- 172.21.2.112:0/3436884833 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5ce80b08d0 0x7f5ce8097a70 secure :-1 s=READY pgs=3023 cs=0 l=1 rev1=1 rx=0x7f5ce00099d0 tx=0x7f5ce000bd40).stop 2021-02-10T11:47:19.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.519 7f5cef02b700 1 -- 172.21.2.112:0/3436884833 shutdown_connections 2021-02-10T11:47:19.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.519 7f5cef02b700 1 --2- 172.21.2.112:0/3436884833 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5ccc0186c0 0x7f5ccc01abc0 unknown :-1 s=CLOSED pgs=1363 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:19.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.519 7f5cef02b700 1 --2- 172.21.2.112:0/3436884833 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5ce80b04d0 0x7f5ce8097450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:19.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.519 7f5cef02b700 1 --2- 172.21.2.112:0/3436884833 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5ce80b08d0 0x7f5ce8097a70 unknown :-1 s=CLOSED pgs=3023 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:19.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.519 7f5cef02b700 1 --2- 172.21.2.112:0/3436884833 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5ce8086810 0x7f5ce819fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:19.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.519 7f5cef02b700 1 -- 172.21.2.112:0/3436884833 >> 172.21.2.112:0/3436884833 conn(0x7f5ce819f880 msgr2=0x7f5ce808f890 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:19.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.519 7f5cef02b700 1 -- 172.21.2.112:0/3436884833 shutdown_connections 2021-02-10T11:47:19.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.519 7f5cef02b700 1 -- 172.21.2.112:0/3436884833 wait complete. 2021-02-10T11:47:19.531 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.3 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:47:19.973 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.952 7f4f4c21d700 1 Processor -- start 2021-02-10T11:47:19.973 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.952 7f4f4c21d700 1 -- start start 2021-02-10T11:47:19.973 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.953 7f4f4c21d700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f4f440d1880 0x7f4f441b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:19.973 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.953 7f4f4c21d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4f440cff80 0x7f4f44109140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:19.973 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.953 7f4f4c21d700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f44109a70 con 0x7f4f441b0af0 2021-02-10T11:47:19.973 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.953 7f4f4c21d700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f44109760 con 0x7f4f440d1880 2021-02-10T11:47:19.973 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.953 7f4f4c21d700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f44088070 con 0x7f4f440cff80 2021-02-10T11:47:19.973 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.953 7f4f49838700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f4f441b0af0 0x7f4f44108950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:59800/0 (socket says 172.21.2.112:59800) 2021-02-10T11:47:19.974 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.953 7f4f49838700 1 -- 172.21.2.112:0/685118922 learned_addr learned my addr 172.21.2.112:0/685118922 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:47:19.974 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.953 7f4f49037700 1 -- 172.21.2.112:0/685118922 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2529767604 0 0) 0x7f4f380010b0 con 0x7f4f441b0af0 2021-02-10T11:47:19.974 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.953 7f4f49037700 1 -- 172.21.2.112:0/685118922 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f30002fe0 con 0x7f4f441b0af0 2021-02-10T11:47:19.974 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.954 7f4f4a83a700 1 --2- 172.21.2.112:0/685118922 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4f440cff80 0x7f4f44109140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:19.974 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.954 7f4f4a039700 1 --2- 172.21.2.112:0/685118922 >> v2:172.21.2.120:3300/0 conn(0x7f4f440d1880 0x7f4f441b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:19.974 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.954 7f4f49037700 1 -- 172.21.2.112:0/685118922 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 739383213 0 0) 0x7f4f380010b0 con 0x7f4f441b0af0 2021-02-10T11:47:19.974 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.954 7f4f49037700 1 -- 172.21.2.112:0/685118922 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4f440cff80 msgr2=0x7f4f44109140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:19.974 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.954 7f4f49037700 1 --2- 172.21.2.112:0/685118922 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4f440cff80 0x7f4f44109140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:19.974 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.954 7f4f49037700 1 -- 172.21.2.112:0/685118922 >> v2:172.21.2.120:3300/0 conn(0x7f4f440d1880 msgr2=0x7f4f441b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:19.974 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.954 7f4f49037700 1 --2- 172.21.2.112:0/685118922 >> v2:172.21.2.120:3300/0 conn(0x7f4f440d1880 0x7f4f441b04d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:19.974 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.954 7f4f49037700 1 -- 172.21.2.112:0/685118922 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f440d1c80 con 0x7f4f441b0af0 2021-02-10T11:47:19.974 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.954 7f4f49037700 1 -- 172.21.2.112:0/685118922 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f4f380015b0 con 0x7f4f441b0af0 2021-02-10T11:47:19.974 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.954 7f4f49037700 1 -- 172.21.2.112:0/685118922 >> v1:172.21.2.112:6789/0 conn(0x7f4f441b0af0 legacy=0x7f4f44108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:19.975 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.954 7f4f49037700 1 --2- 172.21.2.112:0/685118922 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4f30003a60 0x7f4f30005f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:19.975 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.954 7f4f49037700 1 --2- 172.21.2.112:0/685118922 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4f30006780 0x7f4f30008ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:19.975 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.954 7f4f49037700 1 --2- 172.21.2.112:0/685118922 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4f300095a0 0x7f4f3000ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:19.975 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.954 7f4f49037700 1 -- 172.21.2.112:0/685118922 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f300502d0 con 0x7f4f30003a60 2021-02-10T11:47:19.975 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.954 7f4f49037700 1 -- 172.21.2.112:0/685118922 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f30050480 con 0x7f4f30006780 2021-02-10T11:47:19.975 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.954 7f4f49037700 1 -- 172.21.2.112:0/685118922 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f300506a0 con 0x7f4f300095a0 2021-02-10T11:47:19.975 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.954 7f4f4a039700 1 --2- 172.21.2.112:0/685118922 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4f30003a60 0x7f4f30005f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:19.975 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.954 7f4f4a83a700 1 --2- 172.21.2.112:0/685118922 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4f300095a0 0x7f4f3000ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:19.975 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.954 7f4f49838700 1 --2- 172.21.2.112:0/685118922 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4f30006780 0x7f4f30008ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:19.975 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.954 7f4f4a039700 1 -- 172.21.2.112:0/685118922 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4f300095a0 msgr2=0x7f4f3000ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:19.975 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.954 7f4f4a039700 1 --2- 172.21.2.112:0/685118922 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4f300095a0 0x7f4f3000ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:19.976 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.954 7f4f4a039700 1 -- 172.21.2.112:0/685118922 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4f30006780 msgr2=0x7f4f30008ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:19.976 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.954 7f4f4a039700 1 --2- 172.21.2.112:0/685118922 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4f30006780 0x7f4f30008ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:19.976 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.954 7f4f4a039700 1 -- 172.21.2.112:0/685118922 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f30050940 con 0x7f4f30003a60 2021-02-10T11:47:19.976 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.954 7f4f4a039700 1 --2- 172.21.2.112:0/685118922 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4f30003a60 0x7f4f30005f60 secure :-1 s=READY pgs=3025 cs=0 l=1 rev1=1 rx=0x7f4f40006360 tx=0x7f4f40009c30).ready entity=mon.0 client_cookie=5432e64c62911494 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:19.976 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.955 7f4f49037700 1 -- 172.21.2.112:0/685118922 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4f400009b0 con 0x7f4f30003a60 2021-02-10T11:47:19.976 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.955 7f4f49037700 1 -- 172.21.2.112:0/685118922 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f40000d20 con 0x7f4f30003a60 2021-02-10T11:47:19.976 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.955 7f4f49037700 1 -- 172.21.2.112:0/685118922 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4f40002220 con 0x7f4f30003a60 2021-02-10T11:47:19.976 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.955 7f4f4c21d700 1 -- 172.21.2.112:0/685118922 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4f30003a60 msgr2=0x7f4f30005f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:19.976 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.955 7f4f4c21d700 1 --2- 172.21.2.112:0/685118922 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4f30003a60 0x7f4f30005f60 secure :-1 s=READY pgs=3025 cs=0 l=1 rev1=1 rx=0x7f4f40006360 tx=0x7f4f40009c30).stop 2021-02-10T11:47:19.976 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.955 7f4f4a83a700 1 -- 172.21.2.112:0/685118922 reap_dead start 2021-02-10T11:47:19.976 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.955 7f4f4c21d700 1 -- 172.21.2.112:0/685118922 shutdown_connections 2021-02-10T11:47:19.976 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.955 7f4f4c21d700 1 -- 172.21.2.112:0/685118922 >> 172.21.2.112:0/685118922 conn(0x7f4f440ce430 msgr2=0x7f4f440d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:19.976 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.955 7f4f4c21d700 1 -- 172.21.2.112:0/685118922 shutdown_connections 2021-02-10T11:47:19.976 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.955 7f4f4c21d700 1 -- 172.21.2.112:0/685118922 wait complete. 2021-02-10T11:47:19.977 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.970 7f4f4c21d700 1 Processor -- start 2021-02-10T11:47:19.977 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.970 7f4f4c21d700 1 -- start start 2021-02-10T11:47:19.978 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.971 7f4f4c21d700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4f441a0580 0x7f4f44086fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:19.978 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.971 7f4f4c21d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4f440b66b0 0x7f4f440b6ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:19.978 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.971 7f4f4c21d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4f44082bb0 0x7f4f44082fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:19.978 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.971 7f4f4c21d700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f440875d0 con 0x7f4f440b66b0 2021-02-10T11:47:19.978 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.971 7f4f4c21d700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f440835d0 con 0x7f4f441a0580 2021-02-10T11:47:19.978 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.971 7f4f4c21d700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f440837f0 con 0x7f4f44082bb0 2021-02-10T11:47:19.978 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.971 7f4f49838700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4f440b66b0 0x7f4f440b6ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:19.979 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.971 7f4f49838700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4f440b66b0 0x7f4f440b6ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:60352/0 (socket says 172.21.2.112:60352) 2021-02-10T11:47:19.979 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.971 7f4f49838700 1 -- 172.21.2.112:0/4293412387 learned_addr learned my addr 172.21.2.112:0/4293412387 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:47:19.979 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.971 7f4f49838700 1 -- 172.21.2.112:0/4293412387 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4f44082bb0 msgr2=0x7f4f44082fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:19.979 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.971 7f4f49838700 1 --2- 172.21.2.112:0/4293412387 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4f44082bb0 0x7f4f44082fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:19.979 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.971 7f4f49838700 1 -- 172.21.2.112:0/4293412387 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4f441a0580 msgr2=0x7f4f44086fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:19.979 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.971 7f4f49838700 1 --2- 172.21.2.112:0/4293412387 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4f441a0580 0x7f4f44086fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:19.979 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.971 7f4f49838700 1 -- 172.21.2.112:0/4293412387 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f44083a10 con 0x7f4f440b66b0 2021-02-10T11:47:19.979 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.971 7f4f49838700 1 --2- 172.21.2.112:0/4293412387 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4f440b66b0 0x7f4f440b6ab0 secure :-1 s=READY pgs=3026 cs=0 l=1 rev1=1 rx=0x7f4f3800a580 tx=0x7f4f3800a920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:19.979 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.972 7f4f36ffd700 1 -- 172.21.2.112:0/4293412387 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4f38000e20 con 0x7f4f440b66b0 2021-02-10T11:47:19.979 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.972 7f4f36ffd700 1 -- 172.21.2.112:0/4293412387 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f38004350 con 0x7f4f440b66b0 2021-02-10T11:47:19.979 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.972 7f4f36ffd700 1 -- 172.21.2.112:0/4293412387 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4f380047f0 con 0x7f4f440b66b0 2021-02-10T11:47:19.979 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.972 7f4f4c21d700 1 -- 172.21.2.112:0/4293412387 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f44083a10 con 0x7f4f440b66b0 2021-02-10T11:47:19.979 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.972 7f4f4c21d700 1 -- 172.21.2.112:0/4293412387 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f44083fa0 con 0x7f4f440b66b0 2021-02-10T11:47:19.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.972 7f4f36ffd700 1 -- 172.21.2.112:0/4293412387 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4f38000c90 con 0x7f4f440b66b0 2021-02-10T11:47:19.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.972 7f4f36ffd700 1 --2- 172.21.2.112:0/4293412387 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4f2c0186c0 0x7f4f2c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:19.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.972 7f4f36ffd700 1 -- 172.21.2.112:0/4293412387 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f4f38032770 con 0x7f4f440b66b0 2021-02-10T11:47:19.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.973 7f4f4a039700 1 --2- 172.21.2.112:0/4293412387 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4f2c0186c0 0x7f4f2c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:19.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:19.973 7f4f4a039700 1 --2- 172.21.2.112:0/4293412387 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4f2c0186c0 0x7f4f2c01abc0 crc :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:20.006 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:20.003 7f4f4c21d700 1 -- 172.21.2.112:0/4293412387 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f44082980 con 0x7f4f440b66b0 2021-02-10T11:47:20.008 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:20.005 7f4f36ffd700 1 -- 172.21.2.112:0/4293412387 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4f3801f4b0 con 0x7f4f440b66b0 2021-02-10T11:47:20.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:20.136 7f4f4c21d700 1 -- 172.21.2.112:0/4293412387 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f4f44082980 con 0x7f4f440b66b0 2021-02-10T11:47:20.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:20.139 7f4f36ffd700 1 -- 172.21.2.112:0/4293412387 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.3 v105) v1 ==== 184+0+0 (secure 0 0 0) 0x7f4f38000c90 con 0x7f4f440b66b0 2021-02-10T11:47:20.142 INFO:teuthology.orchestra.run.gibba012.stderr:updated caps for client.3 2021-02-10T11:47:20.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:20.141 7f4f4c21d700 1 -- 172.21.2.112:0/4293412387 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4f2c0186c0 msgr2=0x7f4f2c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:20.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:20.141 7f4f4c21d700 1 --2- 172.21.2.112:0/4293412387 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4f2c0186c0 0x7f4f2c01abc0 crc :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:20.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:20.141 7f4f4c21d700 1 -- 172.21.2.112:0/4293412387 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4f440b66b0 msgr2=0x7f4f440b6ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:20.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:20.141 7f4f4c21d700 1 --2- 172.21.2.112:0/4293412387 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4f440b66b0 0x7f4f440b6ab0 secure :-1 s=READY pgs=3026 cs=0 l=1 rev1=1 rx=0x7f4f3800a580 tx=0x7f4f3800a920).stop 2021-02-10T11:47:20.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:20.141 7f4f4c21d700 1 -- 172.21.2.112:0/4293412387 shutdown_connections 2021-02-10T11:47:20.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:20.141 7f4f4c21d700 1 --2- 172.21.2.112:0/4293412387 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4f2c0186c0 0x7f4f2c01abc0 unknown :-1 s=CLOSED pgs=1364 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:20.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:20.141 7f4f4c21d700 1 --2- 172.21.2.112:0/4293412387 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4f44082bb0 0x7f4f44082fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:20.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:20.141 7f4f4c21d700 1 --2- 172.21.2.112:0/4293412387 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4f440b66b0 0x7f4f440b6ab0 unknown :-1 s=CLOSED pgs=3026 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:20.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:20.141 7f4f4c21d700 1 --2- 172.21.2.112:0/4293412387 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4f441a0580 0x7f4f44086fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:20.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:20.141 7f4f4c21d700 1 -- 172.21.2.112:0/4293412387 >> 172.21.2.112:0/4293412387 conn(0x7f4f440afc60 msgr2=0x7f4f440b50d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:20.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:20.141 7f4f4c21d700 1 -- 172.21.2.112:0/4293412387 shutdown_connections 2021-02-10T11:47:20.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:20.141 7f4f4c21d700 1 -- 172.21.2.112:0/4293412387 wait complete. 2021-02-10T11:47:20.152 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:47:20.580 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:20.561 7f98135bf700 1 Processor -- start 2021-02-10T11:47:20.580 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:20.561 7f98135bf700 1 -- start start 2021-02-10T11:47:20.580 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:20.561 7f98135bf700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f980c0d0340 0x7f980c0ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:20.580 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:20.561 7f98135bf700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f980c0d1c70 0x7f980c0efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:20.580 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:20.561 7f98135bf700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f980c0f0b10 con 0x7f980c0d1870 2021-02-10T11:47:20.580 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:20.561 7f98135bf700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f980c0e5ed0 con 0x7f980c0d1c70 2021-02-10T11:47:20.580 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:20.561 7f98135bf700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f980c1b0be0 con 0x7f980c0d0340 2021-02-10T11:47:20.580 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:20.561 7f9811bdc700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f980c0d1c70 0x7f980c0efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:20.580 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:20.561 7f98113db700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f980c0d1870 0x7f980c1b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:59818/0 (socket says 172.21.2.112:59818) 2021-02-10T11:47:20.580 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:20.562 7f9811bdc700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f980c0d1c70 0x7f980c0efc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:60360/0 (socket says 172.21.2.112:60360) 2021-02-10T11:47:20.581 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:20.562 7f98113db700 1 -- 172.21.2.112:0/3112116494 learned_addr learned my addr 172.21.2.112:0/3112116494 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:47:20.581 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:20.562 7f9810bda700 1 --2- 172.21.2.112:0/3112116494 >> v2:172.21.2.120:3300/0 conn(0x7f980c0d0340 0x7f980c0ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:20.581 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:20.562 7f9811bdc700 1 -- 172.21.2.112:0/3112116494 >> v2:172.21.2.120:3300/0 conn(0x7f980c0d0340 msgr2=0x7f980c0ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:20.581 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:20.562 7f9811bdc700 1 --2- 172.21.2.112:0/3112116494 >> v2:172.21.2.120:3300/0 conn(0x7f980c0d0340 0x7f980c0ef640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:20.581 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:20.562 7f9811bdc700 1 -- 172.21.2.112:0/3112116494 >> v1:172.21.2.112:6789/0 conn(0x7f980c0d1870 legacy=0x7f980c1b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:20.581 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:20.562 7f9811bdc700 1 -- 172.21.2.112:0/3112116494 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f980c0f20a0 con 0x7f980c0d1c70 2021-02-10T11:47:20.581 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:20.562 7f9811bdc700 1 --2- 172.21.2.112:0/3112116494 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f980c0d1c70 0x7f980c0efc60 secure :-1 s=READY pgs=3028 cs=0 l=1 rev1=1 rx=0x7f9804006e30 tx=0x7f9804009e60).ready entity=mon.0 client_cookie=90beccba79ecfd6d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:20.581 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:20.562 7f97fffff700 1 -- 172.21.2.112:0/3112116494 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9804006a50 con 0x7f980c0d1c70 2021-02-10T11:47:20.581 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:20.562 7f97fffff700 1 -- 172.21.2.112:0/3112116494 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f980400b300 con 0x7f980c0d1c70 2021-02-10T11:47:20.581 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:20.562 7f97fffff700 1 -- 172.21.2.112:0/3112116494 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f980400b820 con 0x7f980c0d1c70 2021-02-10T11:47:20.581 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:20.562 7f98135bf700 1 -- 172.21.2.112:0/3112116494 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f980c0d1c70 msgr2=0x7f980c0efc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:20.581 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:20.562 7f98135bf700 1 --2- 172.21.2.112:0/3112116494 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f980c0d1c70 0x7f980c0efc60 secure :-1 s=READY pgs=3028 cs=0 l=1 rev1=1 rx=0x7f9804006e30 tx=0x7f9804009e60).stop 2021-02-10T11:47:20.581 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:20.563 7f98135bf700 1 -- 172.21.2.112:0/3112116494 shutdown_connections 2021-02-10T11:47:20.581 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:20.563 7f98135bf700 1 --2- 172.21.2.112:0/3112116494 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f980c0d1c70 0x7f980c0efc60 unknown :-1 s=CLOSED pgs=3028 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:20.582 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:20.563 7f98135bf700 1 --2- 172.21.2.112:0/3112116494 >> v2:172.21.2.120:3300/0 conn(0x7f980c0d0340 0x7f980c0ef640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:20.582 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:20.563 7f98135bf700 1 -- 172.21.2.112:0/3112116494 >> 172.21.2.112:0/3112116494 conn(0x7f980c0ce450 msgr2=0x7f980c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:20.582 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:20.563 7f98135bf700 1 -- 172.21.2.112:0/3112116494 shutdown_connections 2021-02-10T11:47:20.582 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:20.563 7f98135bf700 1 -- 172.21.2.112:0/3112116494 wait complete. 2021-02-10T11:47:20.583 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:20.577 7f98135bf700 1 Processor -- start 2021-02-10T11:47:20.584 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:20.577 7f98135bf700 1 -- start start 2021-02-10T11:47:20.584 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:20.578 7f98135bf700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f980c0ada90 0x7f980c0ade90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:20.584 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:20.578 7f98135bf700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f980c099d20 0x7f980c09a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:20.584 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:20.578 7f98135bf700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f980c09a740 0x7f980c097bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:20.584 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:20.578 7f98135bf700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f980c0997a0 con 0x7f980c099d20 2021-02-10T11:47:20.584 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:20.578 7f98135bf700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f980c18f0c0 con 0x7f980c0ada90 2021-02-10T11:47:20.584 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:20.578 7f98135bf700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f980c09ab40 con 0x7f980c09a740 2021-02-10T11:47:20.584 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:20.578 7f9810bda700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f980c099d20 0x7f980c09a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:20.584 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:20.578 7f9810bda700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f980c099d20 0x7f980c09a120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:60362/0 (socket says 172.21.2.112:60362) 2021-02-10T11:47:20.584 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:20.578 7f9810bda700 1 -- 172.21.2.112:0/4011735553 learned_addr learned my addr 172.21.2.112:0/4011735553 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:47:20.584 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:20.578 7f9811bdc700 1 --2- 172.21.2.112:0/4011735553 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f980c09a740 0x7f980c097bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:20.584 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:20.578 7f98113db700 1 --2- 172.21.2.112:0/4011735553 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f980c0ada90 0x7f980c0ade90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:20.584 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:20.579 7f9811bdc700 1 -- 172.21.2.112:0/4011735553 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f980c0ada90 msgr2=0x7f980c0ade90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:20.584 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:20.579 7f9811bdc700 1 --2- 172.21.2.112:0/4011735553 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f980c0ada90 0x7f980c0ade90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:20.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:20.579 7f9811bdc700 1 -- 172.21.2.112:0/4011735553 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f980c099d20 msgr2=0x7f980c09a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:20.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:20.579 7f9811bdc700 1 --2- 172.21.2.112:0/4011735553 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f980c099d20 0x7f980c09a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:20.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:20.579 7f9811bdc700 1 -- 172.21.2.112:0/4011735553 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f980c09ad60 con 0x7f980c09a740 2021-02-10T11:47:20.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:20.579 7f9810bda700 1 --2- 172.21.2.112:0/4011735553 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f980c099d20 0x7f980c09a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:47:20.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:20.579 7f9811bdc700 1 --2- 172.21.2.112:0/4011735553 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f980c09a740 0x7f980c097bc0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7f980400b8f0 tx=0x7f980400bc90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:20.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:20.579 7f97fdffb700 1 -- 172.21.2.112:0/4011735553 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f980400bdd0 con 0x7f980c09a740 2021-02-10T11:47:20.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:20.579 7f97fdffb700 1 -- 172.21.2.112:0/4011735553 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f980400bdd0 con 0x7f980c09a740 2021-02-10T11:47:20.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:20.579 7f97fdffb700 1 -- 172.21.2.112:0/4011735553 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9804005820 con 0x7f980c09a740 2021-02-10T11:47:20.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:20.579 7f98135bf700 1 -- 172.21.2.112:0/4011735553 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f980c09ad60 con 0x7f980c09a740 2021-02-10T11:47:20.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:20.579 7f98135bf700 1 -- 172.21.2.112:0/4011735553 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f980c0984d0 con 0x7f980c09a740 2021-02-10T11:47:20.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:20.580 7f97fdffb700 1 -- 172.21.2.112:0/4011735553 <== mon.2 v2:172.21.2.120:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9804005a30 con 0x7f980c09a740 2021-02-10T11:47:20.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:20.580 7f97fdffb700 1 --2- 172.21.2.112:0/4011735553 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f97ec0186c0 0x7f97ec01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:20.586 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:20.580 7f97fdffb700 1 -- 172.21.2.112:0/4011735553 <== mon.2 v2:172.21.2.120:3301/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f98040328a0 con 0x7f980c09a740 2021-02-10T11:47:20.586 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:20.580 7f98113db700 1 --2- 172.21.2.112:0/4011735553 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f97ec0186c0 0x7f97ec01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:20.586 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:20.580 7f98113db700 1 --2- 172.21.2.112:0/4011735553 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f97ec0186c0 0x7f97ec01abc0 crc :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:20.614 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:20.611 7f98135bf700 1 -- 172.21.2.112:0/4011735553 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f980c0ebc00 con 0x7f980c09a740 2021-02-10T11:47:20.616 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:20.614 7f97fdffb700 1 -- 172.21.2.112:0/4011735553 <== mon.2 v2:172.21.2.120:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9804005820 con 0x7f980c09a740 2021-02-10T11:47:20.747 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:20.744 7f98135bf700 1 -- 172.21.2.112:0/4011735553 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f980c0f0b10 con 0x7f980c09a740 2021-02-10T11:47:20.748 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:20.746 7f97fdffb700 1 -- 172.21.2.112:0/4011735553 <== mon.2 v2:172.21.2.120:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 156 v156) v1 ==== 95+0+2108 (secure 0 0 0) 0x7f9804005a30 con 0x7f980c09a740 2021-02-10T11:47:20.750 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:47:20.750 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":156,"default_fscid":19,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":13038,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/4134422613","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":4134422613},{"type":"v1","addr":"172.21.2.112:6833","nonce":4134422613}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":152}],"filesystems":[{"mdsmap":{"epoch":156,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:47:14.425711","modified":"2021-02-10 11:47:16.434766","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9307},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9307":{"gid":9307,"name":"b","rank":0,"incarnation":153,"state":"up:active","state_seq":3,"addr":"172.21.2.120:6835/2625088905","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":2625088905},{"type":"v1","addr":"172.21.2.120:6835","nonce":2625088905}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[39],"metadata_pool":38,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":19}]} 2021-02-10T11:47:20.751 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 156 2021-02-10T11:47:20.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:20.749 7f98135bf700 1 -- 172.21.2.112:0/4011735553 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f97ec0186c0 msgr2=0x7f97ec01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:20.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:20.749 7f98135bf700 1 --2- 172.21.2.112:0/4011735553 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f97ec0186c0 0x7f97ec01abc0 crc :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:20.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:20.749 7f98135bf700 1 -- 172.21.2.112:0/4011735553 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f980c09a740 msgr2=0x7f980c097bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:20.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:20.749 7f98135bf700 1 --2- 172.21.2.112:0/4011735553 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f980c09a740 0x7f980c097bc0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7f980400b8f0 tx=0x7f980400bc90).stop 2021-02-10T11:47:20.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:20.749 7f98135bf700 1 -- 172.21.2.112:0/4011735553 shutdown_connections 2021-02-10T11:47:20.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:20.749 7f98135bf700 1 --2- 172.21.2.112:0/4011735553 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f97ec0186c0 0x7f97ec01abc0 unknown :-1 s=CLOSED pgs=1365 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:20.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:20.749 7f98135bf700 1 --2- 172.21.2.112:0/4011735553 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f980c09a740 0x7f980c097bc0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:20.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:20.749 7f98135bf700 1 --2- 172.21.2.112:0/4011735553 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f980c099d20 0x7f980c09a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:20.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:20.749 7f98135bf700 1 --2- 172.21.2.112:0/4011735553 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f980c0ada90 0x7f980c0ade90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:20.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:20.749 7f98135bf700 1 -- 172.21.2.112:0/4011735553 >> 172.21.2.112:0/4011735553 conn(0x7f980c0f0550 msgr2=0x7f980c191730 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:20.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:20.749 7f98135bf700 1 -- 172.21.2.112:0/4011735553 shutdown_connections 2021-02-10T11:47:20.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:20.749 7f98135bf700 1 -- 172.21.2.112:0/4011735553 wait complete. 2021-02-10T11:47:20.962 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 156, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:47:14.425711', 'modified': '2021-02-10 11:47:16.434766', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9307}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9307': {'gid': 9307, 'name': 'b', 'rank': 0, 'incarnation': 153, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.120:6835/2625088905', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.120:6834', 'nonce': 2625088905}, {'type': 'v1', 'addr': '172.21.2.120:6835', 'nonce': 2625088905}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [39], 'metadata_pool': 38, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:47:20.963 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:47:20.963 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:47:21.363 INFO:teuthology.orchestra.run.gibba020.stdout:{ 2021-02-10T11:47:21.363 INFO:teuthology.orchestra.run.gibba020.stdout: "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", 2021-02-10T11:47:21.363 INFO:teuthology.orchestra.run.gibba020.stdout: "whoami": 0, 2021-02-10T11:47:21.363 INFO:teuthology.orchestra.run.gibba020.stdout: "id": 9307, 2021-02-10T11:47:21.363 INFO:teuthology.orchestra.run.gibba020.stdout: "want_state": "up:active", 2021-02-10T11:47:21.363 INFO:teuthology.orchestra.run.gibba020.stdout: "state": "up:active", 2021-02-10T11:47:21.363 INFO:teuthology.orchestra.run.gibba020.stdout: "rank_uptime": 6.9326413730000001, 2021-02-10T11:47:21.363 INFO:teuthology.orchestra.run.gibba020.stdout: "mdsmap_epoch": 156, 2021-02-10T11:47:21.363 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch": 180, 2021-02-10T11:47:21.363 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch_barrier": 180, 2021-02-10T11:47:21.364 INFO:teuthology.orchestra.run.gibba020.stdout: "uptime": 12.582700996 2021-02-10T11:47:21.364 INFO:teuthology.orchestra.run.gibba020.stdout:} 2021-02-10T11:47:21.367 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", "whoami": 0, "id": 9307, "want_state": "up:active", "state": "up:active", "rank_uptime": 6.9326413730000001, "mdsmap_epoch": 156, "osdmap_epoch": 180, "osdmap_epoch_barrier": 180, "uptime": 12.582700996 } 2021-02-10T11:47:21.367 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:47:21.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:21.778 7f615272c700 1 Processor -- start 2021-02-10T11:47:21.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:21.778 7f615272c700 1 -- start start 2021-02-10T11:47:21.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:21.778 7f615272c700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f614c0d0340 0x7f614c0ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:21.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:21.778 7f615272c700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f614c0f0b10 con 0x7f614c0d1870 2021-02-10T11:47:21.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:21.778 7f615272c700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f614c1b0be0 con 0x7f614c0d1c70 2021-02-10T11:47:21.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:21.778 7f615272c700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f614c0e5ed0 con 0x7f614c0d0340 2021-02-10T11:47:21.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:21.779 7f614b7fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f614c0d0340 0x7f614c0ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:21.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:21.779 7f614b7fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f614c0d0340 0x7f614c0ef640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:60372/0 (socket says 172.21.2.112:60372) 2021-02-10T11:47:21.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:21.779 7f614b7fe700 1 -- 172.21.2.112:0/926674759 learned_addr learned my addr 172.21.2.112:0/926674759 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:47:21.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:21.779 7f614b7fe700 1 -- 172.21.2.112:0/926674759 >> v1:172.21.2.120:6789/0 conn(0x7f614c0d1c70 legacy=0x7f614c0efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:21.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:21.779 7f614b7fe700 1 -- 172.21.2.112:0/926674759 >> v1:172.21.2.112:6789/0 conn(0x7f614c0d1870 legacy=0x7f614c1b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:21.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:21.779 7f614b7fe700 1 -- 172.21.2.112:0/926674759 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f614c0f0450 con 0x7f614c0d0340 2021-02-10T11:47:21.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:21.779 7f614b7fe700 1 --2- 172.21.2.112:0/926674759 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f614c0d0340 0x7f614c0ef640 secure :-1 s=READY pgs=3030 cs=0 l=1 rev1=1 rx=0x7f6140006700 tx=0x7f6140009b80).ready entity=mon.0 client_cookie=6c1a125316eb7c1f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:21.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:21.780 7f614affd700 1 -- 172.21.2.112:0/926674759 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f614000c0c0 con 0x7f614c0d0340 2021-02-10T11:47:21.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:21.780 7f614affd700 1 -- 172.21.2.112:0/926674759 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f614000c440 con 0x7f614c0d0340 2021-02-10T11:47:21.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:21.780 7f614affd700 1 -- 172.21.2.112:0/926674759 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f614000c960 con 0x7f614c0d0340 2021-02-10T11:47:21.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:21.780 7f615272c700 1 -- 172.21.2.112:0/926674759 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f614c0d0340 msgr2=0x7f614c0ef640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:21.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:21.780 7f615272c700 1 --2- 172.21.2.112:0/926674759 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f614c0d0340 0x7f614c0ef640 secure :-1 s=READY pgs=3030 cs=0 l=1 rev1=1 rx=0x7f6140006700 tx=0x7f6140009b80).stop 2021-02-10T11:47:21.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:21.780 7f615272c700 1 -- 172.21.2.112:0/926674759 shutdown_connections 2021-02-10T11:47:21.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:21.780 7f615272c700 1 --2- 172.21.2.112:0/926674759 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f614c0d0340 0x7f614c0ef640 unknown :-1 s=CLOSED pgs=3030 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:21.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:21.780 7f615272c700 1 -- 172.21.2.112:0/926674759 >> 172.21.2.112:0/926674759 conn(0x7f614c0ce450 msgr2=0x7f614c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:21.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:21.780 7f615272c700 1 -- 172.21.2.112:0/926674759 shutdown_connections 2021-02-10T11:47:21.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:21.780 7f615272c700 1 -- 172.21.2.112:0/926674759 wait complete. 2021-02-10T11:47:21.800 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:21.795 7f615272c700 1 Processor -- start 2021-02-10T11:47:21.801 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:21.795 7f615272c700 1 -- start start 2021-02-10T11:47:21.801 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:21.795 7f615272c700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f614c103520 0x7f614c0c3640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:21.801 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:21.795 7f615272c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f614c103920 0x7f614c0b32d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:21.801 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:21.795 7f615272c700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f614c08a450 0x7f614c08a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:21.801 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:21.795 7f615272c700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f614c0c3c60 con 0x7f614c103920 2021-02-10T11:47:21.801 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:21.795 7f615272c700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f614c0b38f0 con 0x7f614c08a450 2021-02-10T11:47:21.801 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:21.795 7f615272c700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f614c0c0be0 con 0x7f614c103520 2021-02-10T11:47:21.801 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:21.795 7f614b7fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f614c103920 0x7f614c0b32d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:21.801 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:21.795 7f614b7fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f614c103920 0x7f614c0b32d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:60376/0 (socket says 172.21.2.112:60376) 2021-02-10T11:47:21.801 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:21.795 7f614b7fe700 1 -- 172.21.2.112:0/3242957871 learned_addr learned my addr 172.21.2.112:0/3242957871 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:47:21.801 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:21.796 7f614b7fe700 1 -- 172.21.2.112:0/3242957871 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f614c103520 msgr2=0x7f614c0c3640 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:47:21.801 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:21.796 7f614b7fe700 1 --2- 172.21.2.112:0/3242957871 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f614c103520 0x7f614c0c3640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:21.802 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:21.796 7f614b7fe700 1 -- 172.21.2.112:0/3242957871 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f614c08a450 msgr2=0x7f614c08a850 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:47:21.802 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:21.796 7f614b7fe700 1 --2- 172.21.2.112:0/3242957871 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f614c08a450 0x7f614c08a850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:21.802 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:21.796 7f614b7fe700 1 -- 172.21.2.112:0/3242957871 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f614c08ae70 con 0x7f614c103920 2021-02-10T11:47:21.802 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:21.796 7f614b7fe700 1 --2- 172.21.2.112:0/3242957871 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f614c103920 0x7f614c0b32d0 secure :-1 s=READY pgs=3031 cs=0 l=1 rev1=1 rx=0x7f614000b650 tx=0x7f614000b9f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:21.802 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:21.796 7f6148ff9700 1 -- 172.21.2.112:0/3242957871 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f614000bde0 con 0x7f614c103920 2021-02-10T11:47:21.802 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:21.796 7f6148ff9700 1 -- 172.21.2.112:0/3242957871 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6140000c90 con 0x7f614c103920 2021-02-10T11:47:21.802 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:21.796 7f6148ff9700 1 -- 172.21.2.112:0/3242957871 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f61400022b0 con 0x7f614c103920 2021-02-10T11:47:21.802 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:21.796 7f615272c700 1 -- 172.21.2.112:0/3242957871 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f614c08ae70 con 0x7f614c103920 2021-02-10T11:47:21.802 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:21.796 7f615272c700 1 -- 172.21.2.112:0/3242957871 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f614c08b400 con 0x7f614c103920 2021-02-10T11:47:21.802 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:21.797 7f6148ff9700 1 -- 172.21.2.112:0/3242957871 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6140009f90 con 0x7f614c103920 2021-02-10T11:47:21.803 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:21.797 7f6148ff9700 1 --2- 172.21.2.112:0/3242957871 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f612c0186c0 0x7f612c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:21.803 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:21.797 7f6148ff9700 1 -- 172.21.2.112:0/3242957871 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f6140032590 con 0x7f614c103920 2021-02-10T11:47:21.803 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:21.798 7f614bfff700 1 --2- 172.21.2.112:0/3242957871 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f612c0186c0 0x7f612c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:21.803 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:21.799 7f614bfff700 1 --2- 172.21.2.112:0/3242957871 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f612c0186c0 0x7f612c01abc0 crc :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:21.831 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:21.828 7f615272c700 1 -- 172.21.2.112:0/3242957871 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f614c0b38f0 con 0x7f614c103920 2021-02-10T11:47:21.833 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:21.831 7f6148ff9700 1 -- 172.21.2.112:0/3242957871 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6140025e90 con 0x7f614c103920 2021-02-10T11:47:21.959 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:21.956 7f615272c700 1 -- 172.21.2.112:0/3242957871 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f614c0b38f0 con 0x7f614c103920 2021-02-10T11:47:21.959 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:21.957 7f6148ff9700 1 -- 172.21.2.112:0/3242957871 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v180) v1 ==== 74+0+14007 (secure 0 0 0) 0x7f6140025e90 con 0x7f614c103920 2021-02-10T11:47:21.960 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:47:21.960 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":180,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:47:15.427174","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":39,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":38,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:47:12.395866","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"179","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":39,"pool_name":"cephfs_data","create_time":"2021-02-10 11:47:13.420625","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"180","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:47:21.961 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:21.958 7f615272c700 1 -- 172.21.2.112:0/3242957871 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f612c0186c0 msgr2=0x7f612c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:21.961 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:21.958 7f615272c700 1 --2- 172.21.2.112:0/3242957871 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f612c0186c0 0x7f612c01abc0 crc :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:21.961 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:21.958 7f615272c700 1 -- 172.21.2.112:0/3242957871 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f614c103920 msgr2=0x7f614c0b32d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:21.961 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:21.958 7f615272c700 1 --2- 172.21.2.112:0/3242957871 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f614c103920 0x7f614c0b32d0 secure :-1 s=READY pgs=3031 cs=0 l=1 rev1=1 rx=0x7f614000b650 tx=0x7f614000b9f0).stop 2021-02-10T11:47:21.961 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:21.958 7f615272c700 1 -- 172.21.2.112:0/3242957871 shutdown_connections 2021-02-10T11:47:21.961 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:21.958 7f615272c700 1 --2- 172.21.2.112:0/3242957871 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f612c0186c0 0x7f612c01abc0 unknown :-1 s=CLOSED pgs=1366 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:21.961 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:21.958 7f615272c700 1 --2- 172.21.2.112:0/3242957871 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f614c08a450 0x7f614c08a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:21.961 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:21.958 7f615272c700 1 --2- 172.21.2.112:0/3242957871 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f614c103920 0x7f614c0b32d0 unknown :-1 s=CLOSED pgs=3031 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:21.961 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:21.958 7f615272c700 1 --2- 172.21.2.112:0/3242957871 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f614c103520 0x7f614c0c3640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:21.962 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:21.958 7f615272c700 1 -- 172.21.2.112:0/3242957871 >> 172.21.2.112:0/3242957871 conn(0x7f614c10a820 msgr2=0x7f614c108150 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:21.962 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:21.959 7f615272c700 1 -- 172.21.2.112:0/3242957871 shutdown_connections 2021-02-10T11:47:21.962 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:21.959 7f615272c700 1 -- 172.21.2.112:0/3242957871 wait complete. 2021-02-10T11:47:21.970 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:13:57.508758', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 38, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:47:12.395866', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '179', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 39, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:47:13.420625', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '180', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:47:21.971 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:47:22.404 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:22.384 7f41779a9700 1 Processor -- start 2021-02-10T11:47:22.404 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:22.385 7f41779a9700 1 -- start start 2021-02-10T11:47:22.404 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:22.385 7f41779a9700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f41700d1c70 0x7f41700cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:22.404 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:22.385 7f41779a9700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f41700cf580 0x7f41700cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:22.404 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:22.385 7f41779a9700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41700fe1c0 con 0x7f41700d1870 2021-02-10T11:47:22.404 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:22.385 7f41779a9700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41701b0cb0 con 0x7f41700d1c70 2021-02-10T11:47:22.404 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:22.385 7f41779a9700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4170109a60 con 0x7f41700cf580 2021-02-10T11:47:22.404 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:22.385 7f41757c5700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7f41700d1870 0x7f41701b04c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:53404/0 (socket says 172.21.2.112:53404) 2021-02-10T11:47:22.404 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:22.385 7f41757c5700 1 -- 172.21.2.112:0/1979522882 learned_addr learned my addr 172.21.2.112:0/1979522882 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:47:22.404 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:22.385 7f4174fc4700 1 --2- 172.21.2.112:0/1979522882 >> v2:172.21.2.120:3300/0 conn(0x7f41700d1c70 0x7f41700cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:22.405 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:22.385 7f4175fc6700 1 --2- 172.21.2.112:0/1979522882 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f41700cf580 0x7f41700cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:22.405 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:22.386 7f4163fff700 1 -- 172.21.2.112:0/1979522882 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3505528007 0 0) 0x7f416c0010b0 con 0x7f41700d1870 2021-02-10T11:47:22.405 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:22.386 7f4175fc6700 1 -- 172.21.2.112:0/1979522882 >> v2:172.21.2.120:3300/0 conn(0x7f41700d1c70 msgr2=0x7f41700cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:22.405 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:22.386 7f4175fc6700 1 --2- 172.21.2.112:0/1979522882 >> v2:172.21.2.120:3300/0 conn(0x7f41700d1c70 0x7f41700cef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:22.405 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:22.386 7f4175fc6700 1 -- 172.21.2.112:0/1979522882 >> v1:172.21.2.120:6789/0 conn(0x7f41700d1870 legacy=0x7f41701b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:22.405 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:22.386 7f4175fc6700 1 -- 172.21.2.112:0/1979522882 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4170088070 con 0x7f41700cf580 2021-02-10T11:47:22.405 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:22.386 7f4174fc4700 1 --2- 172.21.2.112:0/1979522882 >> v2:172.21.2.120:3300/0 conn(0x7f41700d1c70 0x7f41700cef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:47:22.405 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:22.386 7f4175fc6700 1 --2- 172.21.2.112:0/1979522882 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f41700cf580 0x7f41700cf980 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0x7f4168006e50 tx=0x7f4168009e70).ready entity=mon.2 client_cookie=45a5a0e9e3394e08 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:22.405 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:22.386 7f4163fff700 1 -- 172.21.2.112:0/1979522882 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4168006a20 con 0x7f41700cf580 2021-02-10T11:47:22.405 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:22.386 7f4163fff700 1 -- 172.21.2.112:0/1979522882 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4168006790 con 0x7f41700cf580 2021-02-10T11:47:22.405 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:22.386 7f4163fff700 1 -- 172.21.2.112:0/1979522882 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f416800b700 con 0x7f41700cf580 2021-02-10T11:47:22.405 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:22.386 7f41779a9700 1 -- 172.21.2.112:0/1979522882 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f41700cf580 msgr2=0x7f41700cf980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:22.406 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:22.386 7f41779a9700 1 --2- 172.21.2.112:0/1979522882 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f41700cf580 0x7f41700cf980 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0x7f4168006e50 tx=0x7f4168009e70).stop 2021-02-10T11:47:22.406 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:22.387 7f41779a9700 1 -- 172.21.2.112:0/1979522882 shutdown_connections 2021-02-10T11:47:22.406 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:22.387 7f41779a9700 1 --2- 172.21.2.112:0/1979522882 >> v2:172.21.2.120:3300/0 conn(0x7f41700d1c70 0x7f41700cef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:22.406 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:22.387 7f41779a9700 1 --2- 172.21.2.112:0/1979522882 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f41700cf580 0x7f41700cf980 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:22.406 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:22.387 7f41779a9700 1 -- 172.21.2.112:0/1979522882 >> 172.21.2.112:0/1979522882 conn(0x7f41700ce420 msgr2=0x7f41700d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:22.406 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:22.387 7f41779a9700 1 -- 172.21.2.112:0/1979522882 shutdown_connections 2021-02-10T11:47:22.406 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:22.387 7f41779a9700 1 -- 172.21.2.112:0/1979522882 wait complete. 2021-02-10T11:47:22.407 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:22.401 7f41779a9700 1 Processor -- start 2021-02-10T11:47:22.408 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:22.401 7f41779a9700 1 -- start start 2021-02-10T11:47:22.408 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:22.402 7f41779a9700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f417009bcc0 0x7f417009c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:22.408 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:22.402 7f41779a9700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f41700a0820 0x7f41700a0c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:22.408 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:22.402 7f41779a9700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f41700a1240 0x7f41700858d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:22.408 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:22.402 7f41779a9700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41700a66b0 con 0x7f417009bcc0 2021-02-10T11:47:22.408 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:22.402 7f41779a9700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41701a49b0 con 0x7f41700a0820 2021-02-10T11:47:22.408 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:22.402 7f41779a9700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41701a4b90 con 0x7f41700a1240 2021-02-10T11:47:22.408 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:22.402 7f41757c5700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f417009bcc0 0x7f417009c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:22.408 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:22.402 7f41757c5700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f417009bcc0 0x7f417009c0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:60390/0 (socket says 172.21.2.112:60390) 2021-02-10T11:47:22.408 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:22.402 7f41757c5700 1 -- 172.21.2.112:0/1673393935 learned_addr learned my addr 172.21.2.112:0/1673393935 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:47:22.408 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:22.402 7f4175fc6700 1 --2- 172.21.2.112:0/1673393935 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f41700a1240 0x7f41700858d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:22.409 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:22.402 7f4174fc4700 1 --2- 172.21.2.112:0/1673393935 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f41700a0820 0x7f41700a0c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:22.409 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:22.403 7f41757c5700 1 -- 172.21.2.112:0/1673393935 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f41700a1240 msgr2=0x7f41700858d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:22.409 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:22.403 7f41757c5700 1 --2- 172.21.2.112:0/1673393935 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f41700a1240 0x7f41700858d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:22.409 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:22.403 7f41757c5700 1 -- 172.21.2.112:0/1673393935 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f41700a0820 msgr2=0x7f41700a0c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:22.409 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:22.403 7f41757c5700 1 --2- 172.21.2.112:0/1673393935 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f41700a0820 0x7f41700a0c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:22.409 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:22.403 7f41757c5700 1 -- 172.21.2.112:0/1673393935 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41700a1640 con 0x7f417009bcc0 2021-02-10T11:47:22.409 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:22.403 7f41757c5700 1 --2- 172.21.2.112:0/1673393935 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f417009bcc0 0x7f417009c0c0 secure :-1 s=READY pgs=3032 cs=0 l=1 rev1=1 rx=0x7f416c001c10 tx=0x7f416c00a8f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:22.409 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:22.403 7f4161ffb700 1 -- 172.21.2.112:0/1673393935 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f416c0009b0 con 0x7f417009bcc0 2021-02-10T11:47:22.409 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:22.403 7f4161ffb700 1 -- 172.21.2.112:0/1673393935 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f416c0012c0 con 0x7f417009bcc0 2021-02-10T11:47:22.409 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:22.403 7f4161ffb700 1 -- 172.21.2.112:0/1673393935 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f416c004350 con 0x7f417009bcc0 2021-02-10T11:47:22.409 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:22.403 7f41779a9700 1 -- 172.21.2.112:0/1673393935 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41700a1640 con 0x7f417009bcc0 2021-02-10T11:47:22.409 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:22.403 7f41779a9700 1 -- 172.21.2.112:0/1673393935 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41700a1640 con 0x7f417009bcc0 2021-02-10T11:47:22.409 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:22.404 7f4161ffb700 1 -- 172.21.2.112:0/1673393935 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f416c007fd0 con 0x7f417009bcc0 2021-02-10T11:47:22.410 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:22.404 7f4161ffb700 1 --2- 172.21.2.112:0/1673393935 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f41500186c0 0x7f415001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:22.410 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:22.404 7f4161ffb700 1 -- 172.21.2.112:0/1673393935 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f416c032520 con 0x7f417009bcc0 2021-02-10T11:47:22.410 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:22.404 7f4174fc4700 1 --2- 172.21.2.112:0/1673393935 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f41500186c0 0x7f415001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:22.410 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:22.404 7f4174fc4700 1 --2- 172.21.2.112:0/1673393935 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f41500186c0 0x7f415001abc0 crc :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:22.437 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:22.435 7f41779a9700 1 -- 172.21.2.112:0/1673393935 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41700fe1c0 con 0x7f417009bcc0 2021-02-10T11:47:22.440 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:22.437 7f4161ffb700 1 -- 172.21.2.112:0/1673393935 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f416c004140 con 0x7f417009bcc0 2021-02-10T11:47:22.569 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:22.566 7f41779a9700 1 -- 172.21.2.112:0/1673393935 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f41700fe1c0 con 0x7f417009bcc0 2021-02-10T11:47:22.570 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:22.567 7f4161ffb700 1 -- 172.21.2.112:0/1673393935 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v180) v1 ==== 92+0+10 (secure 0 0 0) 0x7f416c007fd0 con 0x7f417009bcc0 2021-02-10T11:47:22.570 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:47:22.571 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:22.569 7f41779a9700 1 -- 172.21.2.112:0/1673393935 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f41500186c0 msgr2=0x7f415001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:22.572 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:22.569 7f41779a9700 1 --2- 172.21.2.112:0/1673393935 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f41500186c0 0x7f415001abc0 crc :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:22.572 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:22.569 7f41779a9700 1 -- 172.21.2.112:0/1673393935 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f417009bcc0 msgr2=0x7f417009c0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:22.572 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:22.569 7f41779a9700 1 --2- 172.21.2.112:0/1673393935 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f417009bcc0 0x7f417009c0c0 secure :-1 s=READY pgs=3032 cs=0 l=1 rev1=1 rx=0x7f416c001c10 tx=0x7f416c00a8f0).stop 2021-02-10T11:47:22.572 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:22.569 7f41779a9700 1 -- 172.21.2.112:0/1673393935 shutdown_connections 2021-02-10T11:47:22.572 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:22.569 7f41779a9700 1 --2- 172.21.2.112:0/1673393935 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f41500186c0 0x7f415001abc0 unknown :-1 s=CLOSED pgs=1367 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:22.572 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:22.569 7f41779a9700 1 --2- 172.21.2.112:0/1673393935 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f41700a0820 0x7f41700a0c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:22.573 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:22.569 7f41779a9700 1 --2- 172.21.2.112:0/1673393935 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f41700a1240 0x7f41700858d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:22.573 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:22.569 7f41779a9700 1 --2- 172.21.2.112:0/1673393935 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f417009bcc0 0x7f417009c0c0 unknown :-1 s=CLOSED pgs=3032 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:22.573 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:22.569 7f41779a9700 1 -- 172.21.2.112:0/1673393935 >> 172.21.2.112:0/1673393935 conn(0x7f41700bef00 msgr2=0x7f417009d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:22.573 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:22.569 7f41779a9700 1 -- 172.21.2.112:0/1673393935 shutdown_connections 2021-02-10T11:47:22.573 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:22.569 7f41779a9700 1 -- 172.21.2.112:0/1673393935 wait complete. 2021-02-10T11:47:22.581 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:47:23.013 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:22.994 7f997cc7f700 1 Processor -- start 2021-02-10T11:47:23.014 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:22.994 7f997cc7f700 1 -- start start 2021-02-10T11:47:23.014 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:22.995 7f997cc7f700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f99780d1c60 0x7f99780cef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:23.014 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:22.995 7f997cc7f700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f99780cf570 0x7f99780cf970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:23.014 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:22.995 7f997cc7f700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99780fe1b0 con 0x7f99780d1860 2021-02-10T11:47:23.014 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:22.995 7f997cc7f700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99781b0ca0 con 0x7f99780cf570 2021-02-10T11:47:23.014 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:22.995 7f997cc7f700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9978109a50 con 0x7f99780d1c60 2021-02-10T11:47:23.014 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:22.995 7f997685e700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f99780d1860 0x7f99781b04b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:59858/0 (socket says 172.21.2.112:59858) 2021-02-10T11:47:23.014 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:22.995 7f997685e700 1 -- 172.21.2.112:0/1358390360 learned_addr learned my addr 172.21.2.112:0/1358390360 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:47:23.014 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:22.995 7f997705f700 1 --2- 172.21.2.112:0/1358390360 >> v2:172.21.2.112:3300/0 conn(0x7f99780cf570 0x7f99780cf970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:23.014 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:22.995 7f997705f700 1 -- 172.21.2.112:0/1358390360 >> v2:172.21.2.120:3300/0 conn(0x7f99780d1c60 msgr2=0x7f99780cef50 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:47:23.014 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:22.995 7f997705f700 1 --2- 172.21.2.112:0/1358390360 >> v2:172.21.2.120:3300/0 conn(0x7f99780d1c60 0x7f99780cef50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:23.015 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:22.995 7f997705f700 1 -- 172.21.2.112:0/1358390360 >> v1:172.21.2.112:6789/0 conn(0x7f99780d1860 legacy=0x7f99781b04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:23.015 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:22.995 7f997585c700 1 -- 172.21.2.112:0/1358390360 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 947924013 0 0) 0x7f9968001180 con 0x7f99780d1860 2021-02-10T11:47:23.015 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:22.995 7f997705f700 1 -- 172.21.2.112:0/1358390360 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9978088070 con 0x7f99780cf570 2021-02-10T11:47:23.015 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:22.995 7f997705f700 1 --2- 172.21.2.112:0/1358390360 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f99780cf570 0x7f99780cf970 secure :-1 s=READY pgs=3034 cs=0 l=1 rev1=1 rx=0x7f9970006e30 tx=0x7f9970009e60).ready entity=mon.0 client_cookie=d16f0da8de1ad52b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:23.015 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:22.996 7f997585c700 1 -- 172.21.2.112:0/1358390360 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9970006a50 con 0x7f99780cf570 2021-02-10T11:47:23.015 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:22.996 7f997585c700 1 -- 172.21.2.112:0/1358390360 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f997000b300 con 0x7f99780cf570 2021-02-10T11:47:23.015 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:22.996 7f997585c700 1 -- 172.21.2.112:0/1358390360 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f997000b7e0 con 0x7f99780cf570 2021-02-10T11:47:23.015 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:22.996 7f997cc7f700 1 -- 172.21.2.112:0/1358390360 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f99780cf570 msgr2=0x7f99780cf970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:23.015 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:22.996 7f997cc7f700 1 --2- 172.21.2.112:0/1358390360 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f99780cf570 0x7f99780cf970 secure :-1 s=READY pgs=3034 cs=0 l=1 rev1=1 rx=0x7f9970006e30 tx=0x7f9970009e60).stop 2021-02-10T11:47:23.015 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:22.996 7f997cc7f700 1 -- 172.21.2.112:0/1358390360 shutdown_connections 2021-02-10T11:47:23.015 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:22.996 7f997cc7f700 1 --2- 172.21.2.112:0/1358390360 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f99780cf570 0x7f99780cf970 unknown :-1 s=CLOSED pgs=3034 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:23.015 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:22.996 7f997cc7f700 1 --2- 172.21.2.112:0/1358390360 >> v2:172.21.2.120:3300/0 conn(0x7f99780d1c60 0x7f99780cef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:23.015 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:22.996 7f997cc7f700 1 -- 172.21.2.112:0/1358390360 >> 172.21.2.112:0/1358390360 conn(0x7f99780ce410 msgr2=0x7f99780d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:23.015 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:22.996 7f997cc7f700 1 -- 172.21.2.112:0/1358390360 shutdown_connections 2021-02-10T11:47:23.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:22.997 7f997cc7f700 1 -- 172.21.2.112:0/1358390360 wait complete. 2021-02-10T11:47:23.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:23.011 7f997cc7f700 1 Processor -- start 2021-02-10T11:47:23.016 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:23.011 7f997cc7f700 1 -- start start 2021-02-10T11:47:23.018 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:23.011 7f997cc7f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f99781a05e0 0x7f997818b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:23.018 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:23.011 7f997cc7f700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f99780b8780 0x7f99780bfa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:23.018 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:23.011 7f997cc7f700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f99780c0030 0x7f9978189e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:23.018 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:23.011 7f997cc7f700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99780b82a0 con 0x7f99781a05e0 2021-02-10T11:47:23.018 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:23.011 7f997cc7f700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99780c0430 con 0x7f99780c0030 2021-02-10T11:47:23.018 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:23.011 7f997cc7f700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99780c0650 con 0x7f99780b8780 2021-02-10T11:47:23.018 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:23.011 7f997685e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f99781a05e0 0x7f997818b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:23.018 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:23.011 7f997685e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f99781a05e0 0x7f997818b970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:60406/0 (socket says 172.21.2.112:60406) 2021-02-10T11:47:23.018 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:23.011 7f997685e700 1 -- 172.21.2.112:0/644991009 learned_addr learned my addr 172.21.2.112:0/644991009 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:47:23.018 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:23.012 7f997685e700 1 -- 172.21.2.112:0/644991009 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f99780b8780 msgr2=0x7f99780bfa10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:47:23.018 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:23.012 7f997685e700 1 --2- 172.21.2.112:0/644991009 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f99780b8780 0x7f99780bfa10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:23.019 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:23.012 7f997685e700 1 -- 172.21.2.112:0/644991009 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f99780c0030 msgr2=0x7f9978189e70 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:47:23.019 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:23.012 7f997685e700 1 --2- 172.21.2.112:0/644991009 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f99780c0030 0x7f9978189e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:23.019 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:23.012 7f997685e700 1 -- 172.21.2.112:0/644991009 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f997818a490 con 0x7f99781a05e0 2021-02-10T11:47:23.019 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:23.012 7f997685e700 1 --2- 172.21.2.112:0/644991009 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f99781a05e0 0x7f997818b970 secure :-1 s=READY pgs=3035 cs=0 l=1 rev1=1 rx=0x7f996800a570 tx=0x7f996800a910).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:23.019 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:23.012 7f996f7fe700 1 -- 172.21.2.112:0/644991009 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f99680070d0 con 0x7f99781a05e0 2021-02-10T11:47:23.019 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:23.012 7f996f7fe700 1 -- 172.21.2.112:0/644991009 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f996800ad00 con 0x7f99781a05e0 2021-02-10T11:47:23.019 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:23.012 7f997cc7f700 1 -- 172.21.2.112:0/644991009 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f997818a490 con 0x7f99781a05e0 2021-02-10T11:47:23.019 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:23.012 7f996f7fe700 1 -- 172.21.2.112:0/644991009 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9968004590 con 0x7f99781a05e0 2021-02-10T11:47:23.019 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:23.012 7f997cc7f700 1 -- 172.21.2.112:0/644991009 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f997818a490 con 0x7f99781a05e0 2021-02-10T11:47:23.019 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:23.012 7f996f7fe700 1 -- 172.21.2.112:0/644991009 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9968007f60 con 0x7f99781a05e0 2021-02-10T11:47:23.019 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:23.013 7f996f7fe700 1 --2- 172.21.2.112:0/644991009 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f995c018730 0x7f995c01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:23.019 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:23.013 7f996f7fe700 1 -- 172.21.2.112:0/644991009 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f9968032770 con 0x7f99781a05e0 2021-02-10T11:47:23.019 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:23.013 7f997605d700 1 --2- 172.21.2.112:0/644991009 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f995c018730 0x7f995c01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:23.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:23.013 7f997605d700 1 --2- 172.21.2.112:0/644991009 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f995c018730 0x7f995c01ac10 crc :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:23.048 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:23.045 7f997cc7f700 1 -- 172.21.2.112:0/644991009 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9978082a20 con 0x7f99781a05e0 2021-02-10T11:47:23.050 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:23.047 7f996f7fe700 1 -- 172.21.2.112:0/644991009 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9968000c90 con 0x7f99781a05e0 2021-02-10T11:47:23.176 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:23.174 7f997cc7f700 1 -- 172.21.2.112:0/644991009 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f9978082a20 con 0x7f99781a05e0 2021-02-10T11:47:23.177 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:23.174 7f996f7fe700 1 -- 172.21.2.112:0/644991009 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v180) v1 ==== 104+0+10 (secure 0 0 0) 0x7f996800ad00 con 0x7f99781a05e0 2021-02-10T11:47:23.177 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:47:23.179 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:23.176 7f997cc7f700 1 -- 172.21.2.112:0/644991009 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f995c018730 msgr2=0x7f995c01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:23.179 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:23.176 7f997cc7f700 1 --2- 172.21.2.112:0/644991009 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f995c018730 0x7f995c01ac10 crc :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:23.179 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:23.176 7f997cc7f700 1 -- 172.21.2.112:0/644991009 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f99781a05e0 msgr2=0x7f997818b970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:23.180 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:23.176 7f997cc7f700 1 --2- 172.21.2.112:0/644991009 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f99781a05e0 0x7f997818b970 secure :-1 s=READY pgs=3035 cs=0 l=1 rev1=1 rx=0x7f996800a570 tx=0x7f996800a910).stop 2021-02-10T11:47:23.180 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:23.176 7f997cc7f700 1 -- 172.21.2.112:0/644991009 shutdown_connections 2021-02-10T11:47:23.180 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:23.176 7f997cc7f700 1 --2- 172.21.2.112:0/644991009 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f995c018730 0x7f995c01ac10 unknown :-1 s=CLOSED pgs=1368 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:23.180 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:23.176 7f997cc7f700 1 --2- 172.21.2.112:0/644991009 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f99780c0030 0x7f9978189e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:23.180 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:23.176 7f997cc7f700 1 --2- 172.21.2.112:0/644991009 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f99780b8780 0x7f99780bfa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:23.180 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:23.176 7f997cc7f700 1 --2- 172.21.2.112:0/644991009 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f99781a05e0 0x7f997818b970 unknown :-1 s=CLOSED pgs=3035 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:23.180 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:23.176 7f997cc7f700 1 -- 172.21.2.112:0/644991009 >> 172.21.2.112:0/644991009 conn(0x7f99780a6a80 msgr2=0x7f99780ae110 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:23.180 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:23.176 7f997cc7f700 1 -- 172.21.2.112:0/644991009 shutdown_connections 2021-02-10T11:47:23.180 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:23.176 7f997cc7f700 1 -- 172.21.2.112:0/644991009 wait complete. 2021-02-10T11:47:23.187 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:47:23.621 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:23.602 7f9e0b0b0700 1 Processor -- start 2021-02-10T11:47:23.621 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:23.602 7f9e0b0b0700 1 -- start start 2021-02-10T11:47:23.621 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:23.602 7f9e0b0b0700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f9e040d1870 0x7f9e041b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:23.621 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:23.602 7f9e0b0b0700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f9e041b0ae0 0x7f9e04108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:23.621 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:23.602 7f9e0b0b0700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e040fe1c0 con 0x7f9e040d1c70 2021-02-10T11:47:23.621 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:23.602 7f9e0b0b0700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e04109a60 con 0x7f9e041b0ae0 2021-02-10T11:47:23.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:23.602 7f9e0b0b0700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e04109750 con 0x7f9e040d1870 2021-02-10T11:47:23.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:23.602 7f9e03fff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f9e041b0ae0 0x7f9e04108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:23.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:23.602 7f9e03fff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f9e041b0ae0 0x7f9e04108940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:60412/0 (socket says 172.21.2.112:60412) 2021-02-10T11:47:23.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:23.602 7f9e03fff700 1 -- 172.21.2.112:0/1346394655 learned_addr learned my addr 172.21.2.112:0/1346394655 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:47:23.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:23.603 7f9e037fe700 1 -- 172.21.2.112:0/1346394655 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2753215993 0 0) 0x7f9dfc000f90 con 0x7f9e040d1c70 2021-02-10T11:47:23.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:23.603 7f9e037fe700 1 -- 172.21.2.112:0/1346394655 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9df0002fe0 con 0x7f9e040d1c70 2021-02-10T11:47:23.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:23.603 7f9e08ecc700 1 --2- 172.21.2.112:0/1346394655 >> v2:172.21.2.120:3300/0 conn(0x7f9e040d1870 0x7f9e041b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:23.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:23.603 7f9e03fff700 1 -- 172.21.2.112:0/1346394655 >> v2:172.21.2.120:3300/0 conn(0x7f9e040d1870 msgr2=0x7f9e041b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:23.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:23.603 7f9e03fff700 1 --2- 172.21.2.112:0/1346394655 >> v2:172.21.2.120:3300/0 conn(0x7f9e040d1870 0x7f9e041b04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:23.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:23.603 7f9e03fff700 1 -- 172.21.2.112:0/1346394655 >> v1:172.21.2.112:6789/0 conn(0x7f9e040d1c70 legacy=0x7f9e04108f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:23.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:23.603 7f9e03fff700 1 -- 172.21.2.112:0/1346394655 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e04088070 con 0x7f9e041b0ae0 2021-02-10T11:47:23.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:23.603 7f9e08ecc700 1 --2- 172.21.2.112:0/1346394655 >> v2:172.21.2.120:3300/0 conn(0x7f9e040d1870 0x7f9e041b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:47:23.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:23.603 7f9e03fff700 1 --2- 172.21.2.112:0/1346394655 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9e041b0ae0 0x7f9e04108940 secure :-1 s=READY pgs=3037 cs=0 l=1 rev1=1 rx=0x7f9df4006700 tx=0x7f9df4009b80).ready entity=mon.0 client_cookie=78f723adeeb41e9f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:23.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:23.603 7f9e037fe700 1 -- 172.21.2.112:0/1346394655 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9df4006340 con 0x7f9e041b0ae0 2021-02-10T11:47:23.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:23.603 7f9e037fe700 1 -- 172.21.2.112:0/1346394655 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9df400c0c0 con 0x7f9e041b0ae0 2021-02-10T11:47:23.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:23.603 7f9e037fe700 1 -- 172.21.2.112:0/1346394655 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9df400c5e0 con 0x7f9e041b0ae0 2021-02-10T11:47:23.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:23.604 7f9e0b0b0700 1 -- 172.21.2.112:0/1346394655 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9e041b0ae0 msgr2=0x7f9e04108940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:23.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:23.604 7f9e0b0b0700 1 --2- 172.21.2.112:0/1346394655 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9e041b0ae0 0x7f9e04108940 secure :-1 s=READY pgs=3037 cs=0 l=1 rev1=1 rx=0x7f9df4006700 tx=0x7f9df4009b80).stop 2021-02-10T11:47:23.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:23.604 7f9e0b0b0700 1 -- 172.21.2.112:0/1346394655 shutdown_connections 2021-02-10T11:47:23.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:23.604 7f9e0b0b0700 1 --2- 172.21.2.112:0/1346394655 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9e041b0ae0 0x7f9e04108940 unknown :-1 s=CLOSED pgs=3037 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:23.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:23.604 7f9e0b0b0700 1 --2- 172.21.2.112:0/1346394655 >> v2:172.21.2.120:3300/0 conn(0x7f9e040d1870 0x7f9e041b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:23.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:23.604 7f9e0b0b0700 1 -- 172.21.2.112:0/1346394655 >> 172.21.2.112:0/1346394655 conn(0x7f9e040ce420 msgr2=0x7f9e040d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:23.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:23.604 7f9e0b0b0700 1 -- 172.21.2.112:0/1346394655 shutdown_connections 2021-02-10T11:47:23.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:23.604 7f9e0b0b0700 1 -- 172.21.2.112:0/1346394655 wait complete. 2021-02-10T11:47:23.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:23.619 7f9e0b0b0700 1 Processor -- start 2021-02-10T11:47:23.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:23.619 7f9e0b0b0700 1 -- start start 2021-02-10T11:47:23.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:23.619 7f9e0b0b0700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9e04085bc0 0x7f9e040863e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:23.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:23.619 7f9e0b0b0700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9e040954c0 0x7f9e04096300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:23.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:23.619 7f9e0b0b0700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9e040958c0 0x7f9e04096920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:23.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:23.619 7f9e0b0b0700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e04095d80 con 0x7f9e04085bc0 2021-02-10T11:47:23.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:23.619 7f9e0b0b0700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e040bfa60 con 0x7f9e040958c0 2021-02-10T11:47:23.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:23.619 7f9e0b0b0700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e04085fc0 con 0x7f9e040954c0 2021-02-10T11:47:23.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:23.619 7f9e08ecc700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9e04085bc0 0x7f9e040863e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:23.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:23.619 7f9e08ecc700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9e04085bc0 0x7f9e040863e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:60418/0 (socket says 172.21.2.112:60418) 2021-02-10T11:47:23.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:23.619 7f9e08ecc700 1 -- 172.21.2.112:0/1201116977 learned_addr learned my addr 172.21.2.112:0/1201116977 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:47:23.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:23.620 7f9e08ecc700 1 -- 172.21.2.112:0/1201116977 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9e040954c0 msgr2=0x7f9e04096300 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:47:23.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:23.620 7f9e08ecc700 1 --2- 172.21.2.112:0/1201116977 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9e040954c0 0x7f9e04096300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:23.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:23.620 7f9e08ecc700 1 -- 172.21.2.112:0/1201116977 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9e040958c0 msgr2=0x7f9e04096920 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:47:23.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:23.620 7f9e08ecc700 1 --2- 172.21.2.112:0/1201116977 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9e040958c0 0x7f9e04096920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:23.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:23.620 7f9e08ecc700 1 -- 172.21.2.112:0/1201116977 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e04086c10 con 0x7f9e04085bc0 2021-02-10T11:47:23.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:23.620 7f9e08ecc700 1 --2- 172.21.2.112:0/1201116977 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9e04085bc0 0x7f9e040863e0 secure :-1 s=READY pgs=3038 cs=0 l=1 rev1=1 rx=0x7f9dec006f20 tx=0x7f9dec00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:23.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:23.620 7f9e017fa700 1 -- 172.21.2.112:0/1201116977 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9dec00bd80 con 0x7f9e04085bc0 2021-02-10T11:47:23.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:23.620 7f9e017fa700 1 -- 172.21.2.112:0/1201116977 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dec000c90 con 0x7f9e04085bc0 2021-02-10T11:47:23.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:23.620 7f9e0b0b0700 1 -- 172.21.2.112:0/1201116977 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e04086c10 con 0x7f9e04085bc0 2021-02-10T11:47:23.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:23.620 7f9e0b0b0700 1 -- 172.21.2.112:0/1201116977 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e04097230 con 0x7f9e04085bc0 2021-02-10T11:47:23.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:23.620 7f9e017fa700 1 -- 172.21.2.112:0/1201116977 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9dec002220 con 0x7f9e04085bc0 2021-02-10T11:47:23.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:23.621 7f9e017fa700 1 -- 172.21.2.112:0/1201116977 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9dec00a000 con 0x7f9e04085bc0 2021-02-10T11:47:23.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:23.621 7f9e017fa700 1 --2- 172.21.2.112:0/1201116977 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9de80186c0 0x7f9de801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:23.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:23.621 7f9e017fa700 1 -- 172.21.2.112:0/1201116977 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f9dec0327c0 con 0x7f9e04085bc0 2021-02-10T11:47:23.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:23.621 7f9e03fff700 1 --2- 172.21.2.112:0/1201116977 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9de80186c0 0x7f9de801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:23.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:23.622 7f9e03fff700 1 --2- 172.21.2.112:0/1201116977 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9de80186c0 0x7f9de801abc0 crc :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:23.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:23.652 7f9e0b0b0700 1 -- 172.21.2.112:0/1201116977 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e040fe1c0 con 0x7f9e04085bc0 2021-02-10T11:47:23.658 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:23.655 7f9e017fa700 1 -- 172.21.2.112:0/1201116977 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9dec000960 con 0x7f9e04085bc0 2021-02-10T11:47:23.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:23.782 7f9e0b0b0700 1 -- 172.21.2.112:0/1201116977 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f9e040fe1c0 con 0x7f9e04085bc0 2021-02-10T11:47:23.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:23.782 7f9e017fa700 1 -- 172.21.2.112:0/1201116977 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v180) v1 ==== 100+0+10 (secure 0 0 0) 0x7f9dec00bd80 con 0x7f9e04085bc0 2021-02-10T11:47:23.786 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:47:23.787 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:23.784 7f9e0b0b0700 1 -- 172.21.2.112:0/1201116977 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9de80186c0 msgr2=0x7f9de801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:23.787 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:23.784 7f9e0b0b0700 1 --2- 172.21.2.112:0/1201116977 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9de80186c0 0x7f9de801abc0 crc :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:23.787 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:23.784 7f9e0b0b0700 1 -- 172.21.2.112:0/1201116977 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9e04085bc0 msgr2=0x7f9e040863e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:23.787 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:23.784 7f9e0b0b0700 1 --2- 172.21.2.112:0/1201116977 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9e04085bc0 0x7f9e040863e0 secure :-1 s=READY pgs=3038 cs=0 l=1 rev1=1 rx=0x7f9dec006f20 tx=0x7f9dec00b990).stop 2021-02-10T11:47:23.788 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:23.785 7f9e0b0b0700 1 -- 172.21.2.112:0/1201116977 shutdown_connections 2021-02-10T11:47:23.788 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:23.785 7f9e0b0b0700 1 --2- 172.21.2.112:0/1201116977 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9de80186c0 0x7f9de801abc0 unknown :-1 s=CLOSED pgs=1369 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:23.788 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:23.785 7f9e0b0b0700 1 --2- 172.21.2.112:0/1201116977 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9e040958c0 0x7f9e04096920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:23.788 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:23.785 7f9e0b0b0700 1 --2- 172.21.2.112:0/1201116977 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9e040954c0 0x7f9e04096300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:23.788 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:23.785 7f9e0b0b0700 1 --2- 172.21.2.112:0/1201116977 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9e04085bc0 0x7f9e040863e0 unknown :-1 s=CLOSED pgs=3038 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:23.788 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:23.785 7f9e0b0b0700 1 -- 172.21.2.112:0/1201116977 >> 172.21.2.112:0/1201116977 conn(0x7f9e040bef00 msgr2=0x7f9e0409d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:23.788 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:23.785 7f9e0b0b0700 1 -- 172.21.2.112:0/1201116977 shutdown_connections 2021-02-10T11:47:23.788 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:23.785 7f9e0b0b0700 1 -- 172.21.2.112:0/1201116977 wait complete. 2021-02-10T11:47:23.796 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:47:23.796 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:47:24.224 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:24.205 7f4d4952a700 1 Processor -- start 2021-02-10T11:47:24.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:24.205 7f4d4952a700 1 -- start start 2021-02-10T11:47:24.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:24.205 7f4d4952a700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f4d440d1870 0x7f4d441b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:24.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:24.205 7f4d4952a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4d441b0a10 0x7f4d440cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:24.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:24.205 7f4d4952a700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d440f0b10 con 0x7f4d440d1c70 2021-02-10T11:47:24.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:24.205 7f4d4952a700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d440e5ed0 con 0x7f4d440d1870 2021-02-10T11:47:24.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:24.205 7f4d4952a700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d440f0650 con 0x7f4d441b0a10 2021-02-10T11:47:24.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:24.205 7f4d43ac0700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f4d440d1c70 0x7f4d440cf950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:59886/0 (socket says 172.21.2.112:59886) 2021-02-10T11:47:24.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:24.205 7f4d43ac0700 1 -- 172.21.2.112:0/2047673950 learned_addr learned my addr 172.21.2.112:0/2047673950 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:47:24.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:24.206 7f4d422bd700 1 -- 172.21.2.112:0/2047673950 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3816414756 0 0) 0x7f4d3c001500 con 0x7f4d440d1c70 2021-02-10T11:47:24.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:24.206 7f4d422bd700 1 -- 172.21.2.112:0/2047673950 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d30002fe0 con 0x7f4d440d1c70 2021-02-10T11:47:24.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:24.206 7f4d422bd700 1 -- 172.21.2.112:0/2047673950 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1126264092 0 0) 0x7f4d3c001500 con 0x7f4d440d1c70 2021-02-10T11:47:24.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:24.206 7f4d422bd700 1 -- 172.21.2.112:0/2047673950 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4d441b0a10 msgr2=0x7f4d440cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:24.227 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:24.206 7f4d422bd700 1 --2- 172.21.2.112:0/2047673950 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4d441b0a10 0x7f4d440cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:24.227 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:24.206 7f4d422bd700 1 -- 172.21.2.112:0/2047673950 >> v2:172.21.2.120:3300/0 conn(0x7f4d440d1870 msgr2=0x7f4d441b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:24.227 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:24.206 7f4d422bd700 1 --2- 172.21.2.112:0/2047673950 >> v2:172.21.2.120:3300/0 conn(0x7f4d440d1870 0x7f4d441b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:24.227 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:24.206 7f4d422bd700 1 -- 172.21.2.112:0/2047673950 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d440f0870 con 0x7f4d440d1c70 2021-02-10T11:47:24.227 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:24.206 7f4d422bd700 1 -- 172.21.2.112:0/2047673950 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f4d3c001a00 con 0x7f4d440d1c70 2021-02-10T11:47:24.228 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:24.206 7f4d422bd700 1 -- 172.21.2.112:0/2047673950 >> v1:172.21.2.112:6789/0 conn(0x7f4d440d1c70 legacy=0x7f4d440cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:24.228 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:24.206 7f4d422bd700 1 --2- 172.21.2.112:0/2047673950 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4d30003b90 0x7f4d30006040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:24.228 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:24.206 7f4d422bd700 1 --2- 172.21.2.112:0/2047673950 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4d300067c0 0x7f4d30008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:24.228 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:24.206 7f4d422bd700 1 --2- 172.21.2.112:0/2047673950 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4d30009510 0x7f4d3000ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:24.228 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:24.206 7f4d422bd700 1 -- 172.21.2.112:0/2047673950 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d3000c2e0 con 0x7f4d30009510 2021-02-10T11:47:24.228 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:24.206 7f4d422bd700 1 -- 172.21.2.112:0/2047673950 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d3000c500 con 0x7f4d30003b90 2021-02-10T11:47:24.228 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:24.206 7f4d422bd700 1 -- 172.21.2.112:0/2047673950 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d3000c720 con 0x7f4d300067c0 2021-02-10T11:47:24.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:24.206 7f4d43ac0700 1 --2- 172.21.2.112:0/2047673950 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4d30009510 0x7f4d3000ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:24.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:24.207 7f4d432bf700 1 --2- 172.21.2.112:0/2047673950 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4d30003b90 0x7f4d30006040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:24.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:24.207 7f4d42abe700 1 --2- 172.21.2.112:0/2047673950 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4d300067c0 0x7f4d30008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:24.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:24.207 7f4d43ac0700 1 -- 172.21.2.112:0/2047673950 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4d300067c0 msgr2=0x7f4d30008c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:24.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:24.207 7f4d43ac0700 1 --2- 172.21.2.112:0/2047673950 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4d300067c0 0x7f4d30008c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:24.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:24.207 7f4d43ac0700 1 -- 172.21.2.112:0/2047673950 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4d30003b90 msgr2=0x7f4d30006040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:24.230 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:24.207 7f4d43ac0700 1 --2- 172.21.2.112:0/2047673950 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4d30003b90 0x7f4d30006040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:24.230 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:24.207 7f4d43ac0700 1 -- 172.21.2.112:0/2047673950 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d3000c9c0 con 0x7f4d30009510 2021-02-10T11:47:24.230 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:24.207 7f4d432bf700 1 --2- 172.21.2.112:0/2047673950 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4d30003b90 0x7f4d30006040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:47:24.230 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:24.207 7f4d43ac0700 1 --2- 172.21.2.112:0/2047673950 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4d30009510 0x7f4d3000ba30 secure :-1 s=READY pgs=3040 cs=0 l=1 rev1=1 rx=0x7f4d3c007ce0 tx=0x7f4d3c04e320).ready entity=mon.0 client_cookie=d1493e0df9f08136 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:24.230 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:24.207 7f4d422bd700 1 -- 172.21.2.112:0/2047673950 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4d3c002000 con 0x7f4d30009510 2021-02-10T11:47:24.230 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:24.207 7f4d422bd700 1 -- 172.21.2.112:0/2047673950 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d3c0070c0 con 0x7f4d30009510 2021-02-10T11:47:24.230 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:24.207 7f4d422bd700 1 -- 172.21.2.112:0/2047673950 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4d3c0073b0 con 0x7f4d30009510 2021-02-10T11:47:24.230 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:24.207 7f4d4952a700 1 -- 172.21.2.112:0/2047673950 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4d30009510 msgr2=0x7f4d3000ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:24.230 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:24.207 7f4d4952a700 1 --2- 172.21.2.112:0/2047673950 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4d30009510 0x7f4d3000ba30 secure :-1 s=READY pgs=3040 cs=0 l=1 rev1=1 rx=0x7f4d3c007ce0 tx=0x7f4d3c04e320).stop 2021-02-10T11:47:24.230 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:24.207 7f4d43ac0700 1 -- 172.21.2.112:0/2047673950 reap_dead start 2021-02-10T11:47:24.230 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:24.208 7f4d4952a700 1 -- 172.21.2.112:0/2047673950 shutdown_connections 2021-02-10T11:47:24.231 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:24.208 7f4d4952a700 1 -- 172.21.2.112:0/2047673950 >> 172.21.2.112:0/2047673950 conn(0x7f4d440ce450 msgr2=0x7f4d440d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:24.231 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:24.208 7f4d4952a700 1 -- 172.21.2.112:0/2047673950 shutdown_connections 2021-02-10T11:47:24.231 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:24.208 7f4d4952a700 1 -- 172.21.2.112:0/2047673950 wait complete. 2021-02-10T11:47:24.233 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:24.222 7f4d4952a700 1 Processor -- start 2021-02-10T11:47:24.233 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:24.222 7f4d4952a700 1 -- start start 2021-02-10T11:47:24.233 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:24.223 7f4d4952a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4d44195650 0x7f4d440b5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:24.233 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:24.223 7f4d4952a700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4d440b5b80 0x7f4d441a2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:24.233 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:24.223 7f4d4952a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4d440acfa0 0x7f4d441a3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:24.234 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:24.223 7f4d4952a700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d440b6010 con 0x7f4d44195650 2021-02-10T11:47:24.234 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:24.223 7f4d4952a700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d441a3a60 con 0x7f4d440b5b80 2021-02-10T11:47:24.234 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:24.223 7f4d4952a700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d441a3c40 con 0x7f4d440acfa0 2021-02-10T11:47:24.234 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:24.223 7f4d432bf700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4d44195650 0x7f4d440b5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:24.234 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:24.223 7f4d432bf700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4d44195650 0x7f4d440b5560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:60436/0 (socket says 172.21.2.112:60436) 2021-02-10T11:47:24.234 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:24.223 7f4d432bf700 1 -- 172.21.2.112:0/3163909978 learned_addr learned my addr 172.21.2.112:0/3163909978 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:47:24.234 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:24.223 7f4d432bf700 1 -- 172.21.2.112:0/3163909978 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4d440acfa0 msgr2=0x7f4d441a3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:24.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:24.223 7f4d432bf700 1 --2- 172.21.2.112:0/3163909978 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4d440acfa0 0x7f4d441a3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:24.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:24.223 7f4d432bf700 1 -- 172.21.2.112:0/3163909978 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4d440b5b80 msgr2=0x7f4d441a2e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:24.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:24.223 7f4d432bf700 1 --2- 172.21.2.112:0/3163909978 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4d440b5b80 0x7f4d441a2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:24.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:24.223 7f4d432bf700 1 -- 172.21.2.112:0/3163909978 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d440bcae0 con 0x7f4d44195650 2021-02-10T11:47:24.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:24.224 7f4d432bf700 1 --2- 172.21.2.112:0/3163909978 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4d44195650 0x7f4d440b5560 secure :-1 s=READY pgs=3041 cs=0 l=1 rev1=1 rx=0x7f4d2c00b5c0 tx=0x7f4d2c00b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:24.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:24.224 7f4d3bfff700 1 -- 172.21.2.112:0/3163909978 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4d2c0009b0 con 0x7f4d44195650 2021-02-10T11:47:24.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:24.224 7f4d3bfff700 1 -- 172.21.2.112:0/3163909978 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d2c00bde0 con 0x7f4d44195650 2021-02-10T11:47:24.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:24.224 7f4d3bfff700 1 -- 172.21.2.112:0/3163909978 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4d2c006220 con 0x7f4d44195650 2021-02-10T11:47:24.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:24.224 7f4d4952a700 1 -- 172.21.2.112:0/3163909978 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d440bcae0 con 0x7f4d44195650 2021-02-10T11:47:24.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:24.224 7f4d4952a700 1 -- 172.21.2.112:0/3163909978 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d440bd070 con 0x7f4d44195650 2021-02-10T11:47:24.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:24.225 7f4d3bfff700 1 -- 172.21.2.112:0/3163909978 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4d2c006030 con 0x7f4d44195650 2021-02-10T11:47:24.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:24.225 7f4d3bfff700 1 --2- 172.21.2.112:0/3163909978 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4d280186c0 0x7f4d2801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:24.236 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:24.225 7f4d3bfff700 1 -- 172.21.2.112:0/3163909978 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f4d2c0323e0 con 0x7f4d44195650 2021-02-10T11:47:24.236 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:24.225 7f4d42abe700 1 --2- 172.21.2.112:0/3163909978 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4d280186c0 0x7f4d2801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:24.236 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:24.226 7f4d42abe700 1 --2- 172.21.2.112:0/3163909978 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4d280186c0 0x7f4d2801abc0 crc :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:24.258 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:24.255 7f4d4952a700 1 -- 172.21.2.112:0/3163909978 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d440f0ab0 con 0x7f4d44195650 2021-02-10T11:47:24.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:24.258 7f4d3bfff700 1 -- 172.21.2.112:0/3163909978 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4d2c0195d0 con 0x7f4d44195650 2021-02-10T11:47:24.389 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:24.386 7f4d4952a700 1 -- 172.21.2.112:0/3163909978 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4d440f0ab0 con 0x7f4d44195650 2021-02-10T11:47:24.389 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:24.386 7f4d3bfff700 1 -- 172.21.2.112:0/3163909978 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 156 v156) v1 ==== 95+0+2108 (secure 0 0 0) 0x7f4d2c0195d0 con 0x7f4d44195650 2021-02-10T11:47:24.390 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:47:24.390 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":156,"default_fscid":19,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":13038,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/4134422613","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":4134422613},{"type":"v1","addr":"172.21.2.112:6833","nonce":4134422613}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":152}],"filesystems":[{"mdsmap":{"epoch":156,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:47:14.425711","modified":"2021-02-10 11:47:16.434766","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9307},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9307":{"gid":9307,"name":"b","rank":0,"incarnation":153,"state":"up:active","state_seq":3,"addr":"172.21.2.120:6835/2625088905","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":2625088905},{"type":"v1","addr":"172.21.2.120:6835","nonce":2625088905}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[39],"metadata_pool":38,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":19}]} 2021-02-10T11:47:24.390 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 156 2021-02-10T11:47:24.391 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:24.388 7f4d4952a700 1 -- 172.21.2.112:0/3163909978 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4d280186c0 msgr2=0x7f4d2801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:24.391 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:24.388 7f4d4952a700 1 --2- 172.21.2.112:0/3163909978 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4d280186c0 0x7f4d2801abc0 crc :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:24.391 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:24.388 7f4d4952a700 1 -- 172.21.2.112:0/3163909978 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4d44195650 msgr2=0x7f4d440b5560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:24.391 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:24.388 7f4d4952a700 1 --2- 172.21.2.112:0/3163909978 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4d44195650 0x7f4d440b5560 secure :-1 s=READY pgs=3041 cs=0 l=1 rev1=1 rx=0x7f4d2c00b5c0 tx=0x7f4d2c00b960).stop 2021-02-10T11:47:24.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:24.388 7f4d4952a700 1 -- 172.21.2.112:0/3163909978 shutdown_connections 2021-02-10T11:47:24.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:24.388 7f4d4952a700 1 --2- 172.21.2.112:0/3163909978 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4d280186c0 0x7f4d2801abc0 unknown :-1 s=CLOSED pgs=1370 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:24.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:24.388 7f4d4952a700 1 --2- 172.21.2.112:0/3163909978 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4d440b5b80 0x7f4d441a2e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:24.393 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:24.388 7f4d4952a700 1 --2- 172.21.2.112:0/3163909978 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4d440acfa0 0x7f4d441a3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:24.393 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:24.388 7f4d4952a700 1 --2- 172.21.2.112:0/3163909978 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4d44195650 0x7f4d440b5560 unknown :-1 s=CLOSED pgs=3041 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:24.393 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:24.388 7f4d4952a700 1 -- 172.21.2.112:0/3163909978 >> 172.21.2.112:0/3163909978 conn(0x7f4d440b3040 msgr2=0x7f4d440c3250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:24.393 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:24.389 7f4d4952a700 1 -- 172.21.2.112:0/3163909978 shutdown_connections 2021-02-10T11:47:24.393 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:24.389 7f4d4952a700 1 -- 172.21.2.112:0/3163909978 wait complete. 2021-02-10T11:47:24.601 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 156, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:47:14.425711', 'modified': '2021-02-10 11:47:16.434766', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9307}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9307': {'gid': 9307, 'name': 'b', 'rank': 0, 'incarnation': 153, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.120:6835/2625088905', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.120:6834', 'nonce': 2625088905}, {'type': 'v1', 'addr': '172.21.2.120:6835', 'nonce': 2625088905}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [39], 'metadata_pool': 38, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:47:24.601 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:47:24.602 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:47:24.999 INFO:teuthology.orchestra.run.gibba020.stdout:{ 2021-02-10T11:47:25.000 INFO:teuthology.orchestra.run.gibba020.stdout: "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", 2021-02-10T11:47:25.000 INFO:teuthology.orchestra.run.gibba020.stdout: "whoami": 0, 2021-02-10T11:47:25.000 INFO:teuthology.orchestra.run.gibba020.stdout: "id": 9307, 2021-02-10T11:47:25.000 INFO:teuthology.orchestra.run.gibba020.stdout: "want_state": "up:active", 2021-02-10T11:47:25.000 INFO:teuthology.orchestra.run.gibba020.stdout: "state": "up:active", 2021-02-10T11:47:25.000 INFO:teuthology.orchestra.run.gibba020.stdout: "rank_uptime": 10.569524275999999, 2021-02-10T11:47:25.000 INFO:teuthology.orchestra.run.gibba020.stdout: "mdsmap_epoch": 156, 2021-02-10T11:47:25.000 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch": 180, 2021-02-10T11:47:25.000 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch_barrier": 180, 2021-02-10T11:47:25.000 INFO:teuthology.orchestra.run.gibba020.stdout: "uptime": 16.219584967999999 2021-02-10T11:47:25.000 INFO:teuthology.orchestra.run.gibba020.stdout:} 2021-02-10T11:47:25.003 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", "whoami": 0, "id": 9307, "want_state": "up:active", "state": "up:active", "rank_uptime": 10.569524275999999, "mdsmap_epoch": 156, "osdmap_epoch": 180, "osdmap_epoch_barrier": 180, "uptime": 16.219584967999999 } 2021-02-10T11:47:25.004 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:47:25.004 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2021-02-10T11:47:25.004 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@gibba012.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2021-02-10T11:47:25.004 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:47:25.004 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:47:25.339 DEBUG:teuthology.orchestra.run.gibba012:> sudo modprobe fuse 2021-02-10T11:47:25.684 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:47:25.684 DEBUG:teuthology.orchestra.run.gibba012:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:47:26.028 INFO:teuthology.orchestra.run.gibba012.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:47:26.029 INFO:teuthology.orchestra.run.gibba012.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:47:26.029 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:47:26.029 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:47:26.030 DEBUG:teuthology.orchestra.run.gibba012:> ls /sys/fs/fuse/connections 2021-02-10T11:47:26.365 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2021-02-10T11:47:26.365 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:47:26.367 DEBUG:teuthology.orchestra.run.gibba012:> sudo modprobe fuse 2021-02-10T11:47:26.712 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:47:26.712 DEBUG:teuthology.orchestra.run.gibba012:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:47:26.778 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba012.stderr:ceph-fuse[188076]: starting ceph client 2021-02-10T11:47:26.779 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba012.stderr:2021-02-10 11:47:26.776 7f2f91a17f80 -1 init, newargv = 0x55dc566e7230 newargc=9 2021-02-10T11:47:26.784 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba012.stderr:ceph-fuse[188076]: starting fuse 2021-02-10T11:47:27.055 INFO:teuthology.orchestra.run.gibba012.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:47:27.055 INFO:teuthology.orchestra.run.gibba012.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:47:27.055 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:47:27.056 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:47:27.056 DEBUG:teuthology.orchestra.run.gibba012:> ls /sys/fs/fuse/connections 2021-02-10T11:47:27.107 DEBUG:teuthology.orchestra.run.gibba012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:47:27.111 DEBUG:teuthology.orchestra.run.gibba020:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:47:27.391 INFO:teuthology.orchestra.run.gibba012.stdout:42 2021-02-10T11:47:27.392 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2021-02-10T11:47:27.392 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:47:27.392 DEBUG:teuthology.orchestra.run.gibba012:> sudo python3 -c ' 2021-02-10T11:47:27.392 DEBUG:teuthology.orchestra.run.gibba012:> import glob 2021-02-10T11:47:27.392 DEBUG:teuthology.orchestra.run.gibba012:> import re 2021-02-10T11:47:27.392 DEBUG:teuthology.orchestra.run.gibba012:> import os 2021-02-10T11:47:27.392 DEBUG:teuthology.orchestra.run.gibba012:> import subprocess 2021-02-10T11:47:27.392 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:47:27.392 DEBUG:teuthology.orchestra.run.gibba012:> def find_socket(client_name): 2021-02-10T11:47:27.393 DEBUG:teuthology.orchestra.run.gibba012:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2021-02-10T11:47:27.393 DEBUG:teuthology.orchestra.run.gibba012:> files = glob.glob(asok_path) 2021-02-10T11:47:27.393 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:47:27.393 DEBUG:teuthology.orchestra.run.gibba012:> # Given a non-glob path, it better be there 2021-02-10T11:47:27.393 DEBUG:teuthology.orchestra.run.gibba012:> if "*" not in asok_path: 2021-02-10T11:47:27.393 DEBUG:teuthology.orchestra.run.gibba012:> assert(len(files) == 1) 2021-02-10T11:47:27.393 DEBUG:teuthology.orchestra.run.gibba012:> return files[0] 2021-02-10T11:47:27.393 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:47:27.393 DEBUG:teuthology.orchestra.run.gibba012:> for f in files: 2021-02-10T11:47:27.393 DEBUG:teuthology.orchestra.run.gibba012:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:47:27.393 DEBUG:teuthology.orchestra.run.gibba012:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:47:27.393 DEBUG:teuthology.orchestra.run.gibba012:> return f 2021-02-10T11:47:27.393 DEBUG:teuthology.orchestra.run.gibba012:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:47:27.394 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:47:27.394 DEBUG:teuthology.orchestra.run.gibba012:> print(find_socket("client.0")) 2021-02-10T11:47:27.394 DEBUG:teuthology.orchestra.run.gibba012:> ' 2021-02-10T11:47:27.757 INFO:teuthology.orchestra.run.gibba012.stdout:/var/run/ceph/ceph-client.0.188076.asok 2021-02-10T11:47:27.760 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.188076.asok 2021-02-10T11:47:27.760 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:47:27.760 DEBUG:teuthology.orchestra.run.gibba012:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.188076.asok status 2021-02-10T11:47:28.151 INFO:teuthology.orchestra.run.gibba012.stdout:{ 2021-02-10T11:47:28.151 INFO:teuthology.orchestra.run.gibba012.stdout: "metadata": { 2021-02-10T11:47:28.151 INFO:teuthology.orchestra.run.gibba012.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:47:28.152 INFO:teuthology.orchestra.run.gibba012.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:47:28.152 INFO:teuthology.orchestra.run.gibba012.stdout: "entity_id": "0", 2021-02-10T11:47:28.152 INFO:teuthology.orchestra.run.gibba012.stdout: "hostname": "gibba012", 2021-02-10T11:47:28.152 INFO:teuthology.orchestra.run.gibba012.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2021-02-10T11:47:28.152 INFO:teuthology.orchestra.run.gibba012.stdout: "pid": "188076", 2021-02-10T11:47:28.152 INFO:teuthology.orchestra.run.gibba012.stdout: "root": "/" 2021-02-10T11:47:28.152 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:28.152 INFO:teuthology.orchestra.run.gibba012.stdout: "dentry_count": 0, 2021-02-10T11:47:28.152 INFO:teuthology.orchestra.run.gibba012.stdout: "dentry_pinned_count": 0, 2021-02-10T11:47:28.152 INFO:teuthology.orchestra.run.gibba012.stdout: "id": 13212, 2021-02-10T11:47:28.152 INFO:teuthology.orchestra.run.gibba012.stdout: "inst": { 2021-02-10T11:47:28.152 INFO:teuthology.orchestra.run.gibba012.stdout: "name": { 2021-02-10T11:47:28.153 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "client", 2021-02-10T11:47:28.153 INFO:teuthology.orchestra.run.gibba012.stdout: "num": 13212 2021-02-10T11:47:28.153 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:28.153 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": { 2021-02-10T11:47:28.153 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:47:28.153 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:0", 2021-02-10T11:47:28.153 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 3017669136 2021-02-10T11:47:28.153 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:47:28.153 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:28.153 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": { 2021-02-10T11:47:28.153 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:47:28.153 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:0", 2021-02-10T11:47:28.153 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 3017669136 2021-02-10T11:47:28.154 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:28.154 INFO:teuthology.orchestra.run.gibba012.stdout: "inst_str": "client.13212 172.21.2.112:0/3017669136", 2021-02-10T11:47:28.154 INFO:teuthology.orchestra.run.gibba012.stdout: "addr_str": "172.21.2.112:0/3017669136", 2021-02-10T11:47:28.154 INFO:teuthology.orchestra.run.gibba012.stdout: "inode_count": 1, 2021-02-10T11:47:28.154 INFO:teuthology.orchestra.run.gibba012.stdout: "mds_epoch": 156, 2021-02-10T11:47:28.154 INFO:teuthology.orchestra.run.gibba012.stdout: "osd_epoch": 180, 2021-02-10T11:47:28.154 INFO:teuthology.orchestra.run.gibba012.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:47:28.154 INFO:teuthology.orchestra.run.gibba012.stdout: "blacklisted": false 2021-02-10T11:47:28.154 INFO:teuthology.orchestra.run.gibba012.stdout:} 2021-02-10T11:47:28.155 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:47:28.155 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:47:28.156 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:47:28.491 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:47:28.491 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:47:28.491 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:47:28.491 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:47:28.838 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:47:29.268 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.248 7f4ba058f700 1 Processor -- start 2021-02-10T11:47:29.268 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.248 7f4ba058f700 1 -- start start 2021-02-10T11:47:29.268 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.249 7f4ba058f700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f4b980d1870 0x7f4b981b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:29.268 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.249 7f4ba058f700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4b980f0650 0x7f4b980cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:29.268 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.249 7f4ba058f700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b980f0b10 con 0x7f4b981b0a10 2021-02-10T11:47:29.268 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.249 7f4ba058f700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b980e5ed0 con 0x7f4b980d1870 2021-02-10T11:47:29.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.249 7f4ba058f700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b980d1c70 con 0x7f4b980f0650 2021-02-10T11:47:29.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.249 7f4b9e3ab700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f4b980d1870 0x7f4b981b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:29.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.249 7f4b9e3ab700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f4b980d1870 0x7f4b981b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:60456/0 (socket says 172.21.2.112:60456) 2021-02-10T11:47:29.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.249 7f4b9e3ab700 1 -- 172.21.2.112:0/3236242425 learned_addr learned my addr 172.21.2.112:0/3236242425 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:47:29.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.250 7f4b9e3ab700 1 -- 172.21.2.112:0/3236242425 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4b980f0650 msgr2=0x7f4b980cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:29.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.250 7f4b9e3ab700 1 --2- 172.21.2.112:0/3236242425 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4b980f0650 0x7f4b980cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:29.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.250 7f4b9e3ab700 1 -- 172.21.2.112:0/3236242425 >> v1:172.21.2.120:6789/0 conn(0x7f4b981b0a10 legacy=0x7f4b980cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:29.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.250 7f4b9e3ab700 1 -- 172.21.2.112:0/3236242425 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b980d1e90 con 0x7f4b980d1870 2021-02-10T11:47:29.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.250 7f4b9e3ab700 1 --2- 172.21.2.112:0/3236242425 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4b980d1870 0x7f4b981b03f0 secure :-1 s=READY pgs=3044 cs=0 l=1 rev1=1 rx=0x7f4b94006700 tx=0x7f4b94009b80).ready entity=mon.0 client_cookie=5cccb9b216356ae3 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:29.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.250 7f4b9d3a9700 1 -- 172.21.2.112:0/3236242425 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4b9400c0c0 con 0x7f4b980d1870 2021-02-10T11:47:29.269 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.250 7f4b9d3a9700 1 -- 172.21.2.112:0/3236242425 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b9400c340 con 0x7f4b980d1870 2021-02-10T11:47:29.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.250 7f4b9d3a9700 1 -- 172.21.2.112:0/3236242425 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4b9400c860 con 0x7f4b980d1870 2021-02-10T11:47:29.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.250 7f4ba058f700 1 -- 172.21.2.112:0/3236242425 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4b980d1870 msgr2=0x7f4b981b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:29.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.250 7f4ba058f700 1 --2- 172.21.2.112:0/3236242425 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4b980d1870 0x7f4b981b03f0 secure :-1 s=READY pgs=3044 cs=0 l=1 rev1=1 rx=0x7f4b94006700 tx=0x7f4b94009b80).stop 2021-02-10T11:47:29.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.251 7f4ba058f700 1 -- 172.21.2.112:0/3236242425 shutdown_connections 2021-02-10T11:47:29.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.251 7f4ba058f700 1 --2- 172.21.2.112:0/3236242425 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4b980f0650 0x7f4b980cfb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:29.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.251 7f4ba058f700 1 --2- 172.21.2.112:0/3236242425 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4b980d1870 0x7f4b981b03f0 unknown :-1 s=CLOSED pgs=3044 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:29.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.251 7f4ba058f700 1 -- 172.21.2.112:0/3236242425 >> 172.21.2.112:0/3236242425 conn(0x7f4b980ce450 msgr2=0x7f4b980d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:29.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.251 7f4ba058f700 1 -- 172.21.2.112:0/3236242425 shutdown_connections 2021-02-10T11:47:29.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.251 7f4ba058f700 1 -- 172.21.2.112:0/3236242425 wait complete. 2021-02-10T11:47:29.272 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.265 7f4ba058f700 1 Processor -- start 2021-02-10T11:47:29.272 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.265 7f4ba058f700 1 -- start start 2021-02-10T11:47:29.272 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.266 7f4ba058f700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4b980b0ad0 0x7f4b98091f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:29.272 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.266 7f4ba058f700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4b981a8850 0x7f4b981a8c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:29.272 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.266 7f4ba058f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4b981a6fa0 0x7f4b981a73a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:29.272 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.266 7f4ba058f700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b981abfe0 con 0x7f4b981a6fa0 2021-02-10T11:47:29.272 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.266 7f4ba058f700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b981a9c40 con 0x7f4b980b0ad0 2021-02-10T11:47:29.272 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.266 7f4ba058f700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b981a9730 con 0x7f4b981a8850 2021-02-10T11:47:29.273 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.266 7f4b9ebac700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4b981a6fa0 0x7f4b981a73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:29.273 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.266 7f4b9ebac700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4b981a6fa0 0x7f4b981a73a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:60462/0 (socket says 172.21.2.112:60462) 2021-02-10T11:47:29.273 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.266 7f4b9ebac700 1 -- 172.21.2.112:0/2986541455 learned_addr learned my addr 172.21.2.112:0/2986541455 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:47:29.273 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.266 7f4b9ebac700 1 -- 172.21.2.112:0/2986541455 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4b981a8850 msgr2=0x7f4b981a8c50 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:47:29.273 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.266 7f4b9ebac700 1 --2- 172.21.2.112:0/2986541455 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4b981a8850 0x7f4b981a8c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:29.273 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.266 7f4b9ebac700 1 -- 172.21.2.112:0/2986541455 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4b980b0ad0 msgr2=0x7f4b98091f30 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:47:29.273 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.266 7f4b9ebac700 1 --2- 172.21.2.112:0/2986541455 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4b980b0ad0 0x7f4b98091f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:29.273 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.266 7f4b9ebac700 1 -- 172.21.2.112:0/2986541455 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b981a9270 con 0x7f4b981a6fa0 2021-02-10T11:47:29.273 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.266 7f4b9ebac700 1 --2- 172.21.2.112:0/2986541455 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4b981a6fa0 0x7f4b981a73a0 secure :-1 s=READY pgs=3045 cs=0 l=1 rev1=1 rx=0x7f4b90009980 tx=0x7f4b9000bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:29.273 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.267 7f4b8effd700 1 -- 172.21.2.112:0/2986541455 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4b90005870 con 0x7f4b981a6fa0 2021-02-10T11:47:29.273 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.267 7f4b8effd700 1 -- 172.21.2.112:0/2986541455 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b90005b30 con 0x7f4b981a6fa0 2021-02-10T11:47:29.274 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.267 7f4ba058f700 1 -- 172.21.2.112:0/2986541455 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b981a9270 con 0x7f4b981a6fa0 2021-02-10T11:47:29.274 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.267 7f4b8effd700 1 -- 172.21.2.112:0/2986541455 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4b90006030 con 0x7f4b981a6fa0 2021-02-10T11:47:29.274 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.267 7f4ba058f700 1 -- 172.21.2.112:0/2986541455 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b981a9270 con 0x7f4b981a6fa0 2021-02-10T11:47:29.274 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.267 7f4b8effd700 1 -- 172.21.2.112:0/2986541455 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4b90005d10 con 0x7f4b981a6fa0 2021-02-10T11:47:29.274 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.267 7f4b8effd700 1 --2- 172.21.2.112:0/2986541455 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4b800186c0 0x7f4b8001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:29.274 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.267 7f4b8effd700 1 -- 172.21.2.112:0/2986541455 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f4b90032960 con 0x7f4b981a6fa0 2021-02-10T11:47:29.274 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.268 7f4b9e3ab700 1 --2- 172.21.2.112:0/2986541455 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4b800186c0 0x7f4b8001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:29.274 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.268 7f4b9e3ab700 1 --2- 172.21.2.112:0/2986541455 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4b800186c0 0x7f4b8001abc0 crc :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:29.302 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.299 7f4ba058f700 1 -- 172.21.2.112:0/2986541455 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b981abfe0 con 0x7f4b981a6fa0 2021-02-10T11:47:29.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.301 7f4b8effd700 1 -- 172.21.2.112:0/2986541455 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4b9000be10 con 0x7f4b981a6fa0 2021-02-10T11:47:29.427 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.424 7f4ba058f700 1 -- 172.21.2.112:0/2986541455 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4b981abfe0 con 0x7f4b981a6fa0 2021-02-10T11:47:29.428 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.425 7f4b8effd700 1 -- 172.21.2.112:0/2986541455 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v180) v1 ==== 74+0+14007 (secure 0 0 0) 0x7f4b90006f20 con 0x7f4b981a6fa0 2021-02-10T11:47:29.428 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:47:29.428 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":180,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:47:15.427174","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":39,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":38,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:47:12.395866","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"179","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":39,"pool_name":"cephfs_data","create_time":"2021-02-10 11:47:13.420625","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"180","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:47:29.430 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.427 7f4ba058f700 1 -- 172.21.2.112:0/2986541455 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4b800186c0 msgr2=0x7f4b8001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:29.430 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.427 7f4ba058f700 1 --2- 172.21.2.112:0/2986541455 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4b800186c0 0x7f4b8001abc0 crc :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:29.430 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.427 7f4ba058f700 1 -- 172.21.2.112:0/2986541455 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4b981a6fa0 msgr2=0x7f4b981a73a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:29.430 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.427 7f4ba058f700 1 --2- 172.21.2.112:0/2986541455 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4b981a6fa0 0x7f4b981a73a0 secure :-1 s=READY pgs=3045 cs=0 l=1 rev1=1 rx=0x7f4b90009980 tx=0x7f4b9000bcd0).stop 2021-02-10T11:47:29.430 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.427 7f4ba058f700 1 -- 172.21.2.112:0/2986541455 shutdown_connections 2021-02-10T11:47:29.430 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.427 7f4ba058f700 1 --2- 172.21.2.112:0/2986541455 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4b800186c0 0x7f4b8001abc0 unknown :-1 s=CLOSED pgs=1371 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:29.430 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.427 7f4ba058f700 1 --2- 172.21.2.112:0/2986541455 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4b981a6fa0 0x7f4b981a73a0 unknown :-1 s=CLOSED pgs=3045 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:29.430 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.427 7f4ba058f700 1 --2- 172.21.2.112:0/2986541455 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4b981a8850 0x7f4b981a8c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:29.430 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.427 7f4ba058f700 1 --2- 172.21.2.112:0/2986541455 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4b980b0ad0 0x7f4b98091f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:29.430 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.427 7f4ba058f700 1 -- 172.21.2.112:0/2986541455 >> 172.21.2.112:0/2986541455 conn(0x7f4b9819f8f0 msgr2=0x7f4b981a5f80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:29.430 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.427 7f4ba058f700 1 -- 172.21.2.112:0/2986541455 shutdown_connections 2021-02-10T11:47:29.431 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.427 7f4ba058f700 1 -- 172.21.2.112:0/2986541455 wait complete. 2021-02-10T11:47:29.438 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:13:57.508758', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 38, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:47:12.395866', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '179', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 39, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:47:13.420625', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '180', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:47:29.439 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:47:29.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.852 7f0b4157f700 1 Processor -- start 2021-02-10T11:47:29.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.852 7f0b4157f700 1 -- start start 2021-02-10T11:47:29.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.853 7f0b4157f700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f0b3c0d1c70 0x7f0b3c0cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:29.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.853 7f0b4157f700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f0b3c0cf580 0x7f0b3c0cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:29.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.853 7f0b4157f700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b3c0fe1c0 con 0x7f0b3c0d1870 2021-02-10T11:47:29.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.853 7f0b4157f700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b3c1b0cb0 con 0x7f0b3c0d1c70 2021-02-10T11:47:29.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.853 7f0b4157f700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b3c109a60 con 0x7f0b3c0cf580 2021-02-10T11:47:29.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.853 7f0b3aabe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f0b3c0d1c70 0x7f0b3c0cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:29.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.853 7f0b3b2bf700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f0b3c0d1870 0x7f0b3c1b04c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:59932/0 (socket says 172.21.2.112:59932) 2021-02-10T11:47:29.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.853 7f0b3b2bf700 1 -- 172.21.2.112:0/3234762045 learned_addr learned my addr 172.21.2.112:0/3234762045 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:47:29.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.853 7f0b3aabe700 1 -- 172.21.2.112:0/3234762045 >> v2:172.21.2.120:3300/0 conn(0x7f0b3c0cf580 msgr2=0x7f0b3c0cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:29.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.853 7f0b3aabe700 1 --2- 172.21.2.112:0/3234762045 >> v2:172.21.2.120:3300/0 conn(0x7f0b3c0cf580 0x7f0b3c0cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:29.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.853 7f0b3aabe700 1 -- 172.21.2.112:0/3234762045 >> v1:172.21.2.112:6789/0 conn(0x7f0b3c0d1870 legacy=0x7f0b3c1b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:29.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.853 7f0b3aabe700 1 -- 172.21.2.112:0/3234762045 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b3c088070 con 0x7f0b3c0d1c70 2021-02-10T11:47:29.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.853 7f0b3aabe700 1 --2- 172.21.2.112:0/3234762045 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0b3c0d1c70 0x7f0b3c0cef60 secure :-1 s=READY pgs=3047 cs=0 l=1 rev1=1 rx=0x7f0b2c006650 tx=0x7f0b2c009a80).ready entity=mon.0 client_cookie=640626740f76ed6 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:29.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.854 7f0b3a2bd700 1 -- 172.21.2.112:0/3234762045 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0b2c0009b0 con 0x7f0b3c0d1c70 2021-02-10T11:47:29.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.854 7f0b3a2bd700 1 -- 172.21.2.112:0/3234762045 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b2c00c0c0 con 0x7f0b3c0d1c70 2021-02-10T11:47:29.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.854 7f0b3a2bd700 1 -- 172.21.2.112:0/3234762045 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0b2c00c5e0 con 0x7f0b3c0d1c70 2021-02-10T11:47:29.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.854 7f0b4157f700 1 -- 172.21.2.112:0/3234762045 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0b3c0d1c70 msgr2=0x7f0b3c0cef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:29.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.854 7f0b4157f700 1 --2- 172.21.2.112:0/3234762045 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0b3c0d1c70 0x7f0b3c0cef60 secure :-1 s=READY pgs=3047 cs=0 l=1 rev1=1 rx=0x7f0b2c006650 tx=0x7f0b2c009a80).stop 2021-02-10T11:47:29.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.854 7f0b4157f700 1 -- 172.21.2.112:0/3234762045 shutdown_connections 2021-02-10T11:47:29.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.854 7f0b4157f700 1 --2- 172.21.2.112:0/3234762045 >> v2:172.21.2.120:3300/0 conn(0x7f0b3c0cf580 0x7f0b3c0cf980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:29.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.854 7f0b4157f700 1 --2- 172.21.2.112:0/3234762045 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0b3c0d1c70 0x7f0b3c0cef60 unknown :-1 s=CLOSED pgs=3047 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:29.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.854 7f0b4157f700 1 -- 172.21.2.112:0/3234762045 >> 172.21.2.112:0/3234762045 conn(0x7f0b3c0ce420 msgr2=0x7f0b3c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:29.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.854 7f0b4157f700 1 -- 172.21.2.112:0/3234762045 shutdown_connections 2021-02-10T11:47:29.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.854 7f0b4157f700 1 -- 172.21.2.112:0/3234762045 wait complete. 2021-02-10T11:47:29.875 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.869 7f0b4157f700 1 Processor -- start 2021-02-10T11:47:29.875 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.869 7f0b4157f700 1 -- start start 2021-02-10T11:47:29.875 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.869 7f0b4157f700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0b3c0c3820 0x7f0b3c084cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:29.875 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.869 7f0b4157f700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0b3c0a0290 0x7f0b3c0852d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:29.875 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.869 7f0b4157f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0b3c086270 0x7f0b3c086670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:29.875 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.869 7f0b4157f700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b3c0c0700 con 0x7f0b3c086270 2021-02-10T11:47:29.875 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.869 7f0b4157f700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b3c0858f0 con 0x7f0b3c0c3820 2021-02-10T11:47:29.875 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.869 7f0b4157f700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b3c085b10 con 0x7f0b3c0a0290 2021-02-10T11:47:29.875 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.870 7f0b3bac0700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0b3c086270 0x7f0b3c086670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:29.875 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.870 7f0b3bac0700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0b3c086270 0x7f0b3c086670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:60480/0 (socket says 172.21.2.112:60480) 2021-02-10T11:47:29.876 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.870 7f0b3bac0700 1 -- 172.21.2.112:0/2022173035 learned_addr learned my addr 172.21.2.112:0/2022173035 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:47:29.876 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.870 7f0b3aabe700 1 --2- 172.21.2.112:0/2022173035 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0b3c0a0290 0x7f0b3c0852d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:29.876 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.870 7f0b3bac0700 1 -- 172.21.2.112:0/2022173035 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0b3c0a0290 msgr2=0x7f0b3c0852d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:29.876 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.870 7f0b3bac0700 1 --2- 172.21.2.112:0/2022173035 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0b3c0a0290 0x7f0b3c0852d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:29.876 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.870 7f0b3bac0700 1 -- 172.21.2.112:0/2022173035 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0b3c0c3820 msgr2=0x7f0b3c084cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:29.876 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.870 7f0b3bac0700 1 --2- 172.21.2.112:0/2022173035 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0b3c0c3820 0x7f0b3c084cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:29.876 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.870 7f0b3bac0700 1 -- 172.21.2.112:0/2022173035 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b3c086c90 con 0x7f0b3c086270 2021-02-10T11:47:29.876 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.870 7f0b3bac0700 1 --2- 172.21.2.112:0/2022173035 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0b3c086270 0x7f0b3c086670 secure :-1 s=READY pgs=3048 cs=0 l=1 rev1=1 rx=0x7f0b34009980 tx=0x7f0b3400bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:29.876 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.870 7f0b33fff700 1 -- 172.21.2.112:0/2022173035 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0b34005af0 con 0x7f0b3c086270 2021-02-10T11:47:29.876 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.870 7f0b33fff700 1 -- 172.21.2.112:0/2022173035 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b34005d70 con 0x7f0b3c086270 2021-02-10T11:47:29.876 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.870 7f0b33fff700 1 -- 172.21.2.112:0/2022173035 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0b340062b0 con 0x7f0b3c086270 2021-02-10T11:47:29.876 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.870 7f0b4157f700 1 -- 172.21.2.112:0/2022173035 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b3c086c90 con 0x7f0b3c086270 2021-02-10T11:47:29.876 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.871 7f0b4157f700 1 -- 172.21.2.112:0/2022173035 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b3c1a2f00 con 0x7f0b3c086270 2021-02-10T11:47:29.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.871 7f0b33fff700 1 -- 172.21.2.112:0/2022173035 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0b34006030 con 0x7f0b3c086270 2021-02-10T11:47:29.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.871 7f0b33fff700 1 --2- 172.21.2.112:0/2022173035 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0b200186c0 0x7f0b2001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:29.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.871 7f0b33fff700 1 -- 172.21.2.112:0/2022173035 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f0b34032900 con 0x7f0b3c086270 2021-02-10T11:47:29.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.872 7f0b3b2bf700 1 --2- 172.21.2.112:0/2022173035 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0b200186c0 0x7f0b2001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:29.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.872 7f0b3b2bf700 1 --2- 172.21.2.112:0/2022173035 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0b200186c0 0x7f0b2001abc0 crc :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:29.906 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.903 7f0b4157f700 1 -- 172.21.2.112:0/2022173035 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b3c082a70 con 0x7f0b3c086270 2021-02-10T11:47:29.909 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:29.906 7f0b33fff700 1 -- 172.21.2.112:0/2022173035 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0b34006570 con 0x7f0b3c086270 2021-02-10T11:47:30.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:30.036 7f0b4157f700 1 -- 172.21.2.112:0/2022173035 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f0b3c082a70 con 0x7f0b3c086270 2021-02-10T11:47:30.039 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:30.037 7f0b33fff700 1 -- 172.21.2.112:0/2022173035 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v180) v1 ==== 92+0+10 (secure 0 0 0) 0x7f0b34006030 con 0x7f0b3c086270 2021-02-10T11:47:30.040 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:47:30.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:30.038 7f0b4157f700 1 -- 172.21.2.112:0/2022173035 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0b200186c0 msgr2=0x7f0b2001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:30.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:30.038 7f0b4157f700 1 --2- 172.21.2.112:0/2022173035 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0b200186c0 0x7f0b2001abc0 crc :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:30.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:30.038 7f0b4157f700 1 -- 172.21.2.112:0/2022173035 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0b3c086270 msgr2=0x7f0b3c086670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:30.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:30.038 7f0b4157f700 1 --2- 172.21.2.112:0/2022173035 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0b3c086270 0x7f0b3c086670 secure :-1 s=READY pgs=3048 cs=0 l=1 rev1=1 rx=0x7f0b34009980 tx=0x7f0b3400bcd0).stop 2021-02-10T11:47:30.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:30.039 7f0b4157f700 1 -- 172.21.2.112:0/2022173035 shutdown_connections 2021-02-10T11:47:30.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:30.039 7f0b4157f700 1 --2- 172.21.2.112:0/2022173035 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0b200186c0 0x7f0b2001abc0 unknown :-1 s=CLOSED pgs=1372 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:30.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:30.039 7f0b4157f700 1 --2- 172.21.2.112:0/2022173035 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0b3c086270 0x7f0b3c086670 unknown :-1 s=CLOSED pgs=3048 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:30.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:30.039 7f0b4157f700 1 --2- 172.21.2.112:0/2022173035 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0b3c0a0290 0x7f0b3c0852d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:30.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:30.039 7f0b4157f700 1 --2- 172.21.2.112:0/2022173035 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0b3c0c3820 0x7f0b3c084cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:30.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:30.039 7f0b4157f700 1 -- 172.21.2.112:0/2022173035 >> 172.21.2.112:0/2022173035 conn(0x7f0b3c0872c0 msgr2=0x7f0b3c095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:30.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:30.039 7f0b4157f700 1 -- 172.21.2.112:0/2022173035 shutdown_connections 2021-02-10T11:47:30.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:30.039 7f0b4157f700 1 -- 172.21.2.112:0/2022173035 wait complete. 2021-02-10T11:47:30.050 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:47:30.484 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:30.463 7f5994e75700 1 Processor -- start 2021-02-10T11:47:30.484 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:30.463 7f5994e75700 1 -- start start 2021-02-10T11:47:30.484 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:30.464 7f5994e75700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f59900d1c70 0x7f59900cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:30.484 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:30.464 7f5994e75700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59900d0170 con 0x7f59900cf580 2021-02-10T11:47:30.484 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:30.464 7f5994e75700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59900fe1c0 con 0x7f59900d1870 2021-02-10T11:47:30.484 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:30.464 7f5994e75700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59901b0cb0 con 0x7f59900d1c70 2021-02-10T11:47:30.484 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:30.464 7f598f314700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f59900cf580 0x7f59900cf980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:59950/0 (socket says 172.21.2.112:59950) 2021-02-10T11:47:30.484 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:30.464 7f598f314700 1 -- 172.21.2.112:0/2746869461 learned_addr learned my addr 172.21.2.112:0/2746869461 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:47:30.485 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:30.464 7f598db11700 1 -- 172.21.2.112:0/2746869461 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1162839939 0 0) 0x7f5988001500 con 0x7f59900cf580 2021-02-10T11:47:30.485 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:30.464 7f598db11700 1 -- 172.21.2.112:0/2746869461 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f597c002fe0 con 0x7f59900cf580 2021-02-10T11:47:30.485 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:30.464 7f598db11700 1 -- 172.21.2.112:0/2746869461 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2505341824 0 0) 0x7f5988001460 con 0x7f59900cf580 2021-02-10T11:47:30.485 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:30.464 7f598e312700 1 --2- 172.21.2.112:0/2746869461 >> v2:172.21.2.120:3300/0 conn(0x7f59900d1c70 0x7f59900cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:30.485 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:30.464 7f598db11700 1 -- 172.21.2.112:0/2746869461 >> v2:172.21.2.120:3300/0 conn(0x7f59900d1c70 msgr2=0x7f59900cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:30.485 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:30.464 7f598db11700 1 --2- 172.21.2.112:0/2746869461 >> v2:172.21.2.120:3300/0 conn(0x7f59900d1c70 0x7f59900cef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:30.485 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:30.464 7f598db11700 1 -- 172.21.2.112:0/2746869461 >> v1:172.21.2.120:6789/0 conn(0x7f59900d1870 legacy=0x7f59901b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:30.485 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:30.464 7f598db11700 1 -- 172.21.2.112:0/2746869461 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5990109a60 con 0x7f59900cf580 2021-02-10T11:47:30.485 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:30.464 7f598db11700 1 -- 172.21.2.112:0/2746869461 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f5988001980 con 0x7f59900cf580 2021-02-10T11:47:30.485 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:30.465 7f598db11700 1 -- 172.21.2.112:0/2746869461 >> v1:172.21.2.112:6789/0 conn(0x7f59900cf580 legacy=0x7f59900cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:30.485 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:30.465 7f598db11700 1 --2- 172.21.2.112:0/2746869461 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f597c003ae0 0x7f597c005f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:30.485 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:30.465 7f598db11700 1 --2- 172.21.2.112:0/2746869461 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f597c006830 0x7f597c008d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:30.486 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:30.465 7f598db11700 1 --2- 172.21.2.112:0/2746869461 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f597c009630 0x7f597c00bb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:30.486 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:30.465 7f598db11700 1 -- 172.21.2.112:0/2746869461 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f597c00c3a0 con 0x7f597c003ae0 2021-02-10T11:47:30.486 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:30.465 7f598db11700 1 -- 172.21.2.112:0/2746869461 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f597c00c5c0 con 0x7f597c006830 2021-02-10T11:47:30.486 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:30.465 7f598db11700 1 -- 172.21.2.112:0/2746869461 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f597c00c7e0 con 0x7f597c009630 2021-02-10T11:47:30.486 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:30.465 7f598e312700 1 --2- 172.21.2.112:0/2746869461 >> v2:172.21.2.120:3300/0 conn(0x7f59900d1c70 0x7f59900cef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:47:30.486 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:30.465 7f598eb13700 1 --2- 172.21.2.112:0/2746869461 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f597c003ae0 0x7f597c005f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:30.486 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:30.465 7f598f314700 1 --2- 172.21.2.112:0/2746869461 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f597c009630 0x7f597c00bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:30.486 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:30.465 7f598e312700 1 --2- 172.21.2.112:0/2746869461 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f597c006830 0x7f597c008d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:30.486 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:30.465 7f598f314700 1 -- 172.21.2.112:0/2746869461 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f597c006830 msgr2=0x7f597c008d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:30.486 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:30.465 7f598f314700 1 --2- 172.21.2.112:0/2746869461 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f597c006830 0x7f597c008d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:30.486 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:30.465 7f598f314700 1 -- 172.21.2.112:0/2746869461 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f597c003ae0 msgr2=0x7f597c005f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:30.486 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:30.465 7f598f314700 1 --2- 172.21.2.112:0/2746869461 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f597c003ae0 0x7f597c005f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:30.486 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:30.465 7f598f314700 1 -- 172.21.2.112:0/2746869461 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f597c00ca80 con 0x7f597c009630 2021-02-10T11:47:30.487 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:30.465 7f598f314700 1 -- 172.21.2.112:0/2746869461 reap_dead start 2021-02-10T11:47:30.487 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:30.465 7f598f314700 1 --2- 172.21.2.112:0/2746869461 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f597c009630 0x7f597c00bb50 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7f5988001070 tx=0x7f598800a060).ready entity=mon.2 client_cookie=d945e96d14abafdd server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:30.487 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:30.466 7f598db11700 1 -- 172.21.2.112:0/2746869461 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5988001a10 con 0x7f597c009630 2021-02-10T11:47:30.487 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:30.466 7f598db11700 1 -- 172.21.2.112:0/2746869461 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5988007150 con 0x7f597c009630 2021-02-10T11:47:30.487 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:30.466 7f598db11700 1 -- 172.21.2.112:0/2746869461 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5988001a10 con 0x7f597c009630 2021-02-10T11:47:30.487 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:30.466 7f5994e75700 1 -- 172.21.2.112:0/2746869461 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f597c009630 msgr2=0x7f597c00bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:30.487 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:30.466 7f5994e75700 1 --2- 172.21.2.112:0/2746869461 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f597c009630 0x7f597c00bb50 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7f5988001070 tx=0x7f598800a060).stop 2021-02-10T11:47:30.487 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:30.466 7f5994e75700 1 -- 172.21.2.112:0/2746869461 shutdown_connections 2021-02-10T11:47:30.487 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:30.466 7f5994e75700 1 --2- 172.21.2.112:0/2746869461 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f597c009630 0x7f597c00bb50 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:30.487 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:30.466 7f5994e75700 1 -- 172.21.2.112:0/2746869461 >> 172.21.2.112:0/2746869461 conn(0x7f59900ce420 msgr2=0x7f59900d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:30.487 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:30.466 7f5994e75700 1 -- 172.21.2.112:0/2746869461 shutdown_connections 2021-02-10T11:47:30.487 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:30.467 7f5994e75700 1 -- 172.21.2.112:0/2746869461 wait complete. 2021-02-10T11:47:30.489 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:30.481 7f5994e75700 1 Processor -- start 2021-02-10T11:47:30.489 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:30.482 7f5994e75700 1 -- start start 2021-02-10T11:47:30.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:30.482 7f5994e75700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f59900b22c0 0x7f59900b26c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:30.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:30.482 7f5994e75700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f59900a6df0 0x7f59900a71f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:30.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:30.482 7f5994e75700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f59900a5cd0 0x7f59900a60d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:30.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:30.482 7f5994e75700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59900c23f0 con 0x7f59900b22c0 2021-02-10T11:47:30.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:30.482 7f5994e75700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59900a66f0 con 0x7f59900a5cd0 2021-02-10T11:47:30.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:30.482 7f5994e75700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59900a68e0 con 0x7f59900a6df0 2021-02-10T11:47:30.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:30.482 7f598f314700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f59900a5cd0 0x7f59900a60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:30.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:30.482 7f598e312700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f59900a6df0 0x7f59900a71f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:30.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:30.482 7f598f314700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f59900a5cd0 0x7f59900a60d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3300/0 says I am v2:172.21.2.112:34544/0 (socket says 172.21.2.112:34544) 2021-02-10T11:47:30.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:30.482 7f598f314700 1 -- 172.21.2.112:0/3950910978 learned_addr learned my addr 172.21.2.112:0/3950910978 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:47:30.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:30.482 7f598eb13700 1 --2- 172.21.2.112:0/3950910978 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f59900b22c0 0x7f59900b26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:30.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:30.483 7f598eb13700 1 -- 172.21.2.112:0/3950910978 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f59900a6df0 msgr2=0x7f59900a71f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:30.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:30.483 7f598eb13700 1 --2- 172.21.2.112:0/3950910978 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f59900a6df0 0x7f59900a71f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:30.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:30.483 7f598eb13700 1 -- 172.21.2.112:0/3950910978 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f59900a5cd0 msgr2=0x7f59900a60d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:30.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:30.483 7f598eb13700 1 --2- 172.21.2.112:0/3950910978 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f59900a5cd0 0x7f59900a60d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:30.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:30.483 7f598eb13700 1 -- 172.21.2.112:0/3950910978 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5990083980 con 0x7f59900b22c0 2021-02-10T11:47:30.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:30.483 7f598f314700 1 --2- 172.21.2.112:0/3950910978 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f59900a5cd0 0x7f59900a60d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:47:30.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:30.483 7f598eb13700 1 --2- 172.21.2.112:0/3950910978 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f59900b22c0 0x7f59900b26c0 secure :-1 s=READY pgs=3050 cs=0 l=1 rev1=1 rx=0x7f597804e540 tx=0x7f5978050860).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:30.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:30.484 7f59877fe700 1 -- 172.21.2.112:0/3950910978 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5978050c50 con 0x7f59900b22c0 2021-02-10T11:47:30.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:30.484 7f59877fe700 1 -- 172.21.2.112:0/3950910978 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5978002080 con 0x7f59900b22c0 2021-02-10T11:47:30.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:30.484 7f59877fe700 1 -- 172.21.2.112:0/3950910978 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f59780024d0 con 0x7f59900b22c0 2021-02-10T11:47:30.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:30.484 7f5994e75700 1 -- 172.21.2.112:0/3950910978 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5990083980 con 0x7f59900b22c0 2021-02-10T11:47:30.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:30.484 7f5994e75700 1 -- 172.21.2.112:0/3950910978 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5990083980 con 0x7f59900b22c0 2021-02-10T11:47:30.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:30.485 7f59877fe700 1 -- 172.21.2.112:0/3950910978 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5978000c60 con 0x7f59900b22c0 2021-02-10T11:47:30.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:30.485 7f59877fe700 1 --2- 172.21.2.112:0/3950910978 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f59740186c0 0x7f597401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:30.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:30.485 7f59877fe700 1 -- 172.21.2.112:0/3950910978 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f5978076700 con 0x7f59900b22c0 2021-02-10T11:47:30.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:30.485 7f598e312700 1 --2- 172.21.2.112:0/3950910978 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f59740186c0 0x7f597401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:30.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:30.485 7f598e312700 1 --2- 172.21.2.112:0/3950910978 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f59740186c0 0x7f597401abc0 crc :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:30.516 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:30.513 7f5994e75700 1 -- 172.21.2.112:0/3950910978 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5990082a70 con 0x7f59900b22c0 2021-02-10T11:47:30.519 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:30.516 7f59877fe700 1 -- 172.21.2.112:0/3950910978 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5978006a50 con 0x7f59900b22c0 2021-02-10T11:47:30.648 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:30.645 7f5994e75700 1 -- 172.21.2.112:0/3950910978 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f5990082a70 con 0x7f59900b22c0 2021-02-10T11:47:30.649 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:30.646 7f59877fe700 1 -- 172.21.2.112:0/3950910978 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v180) v1 ==== 104+0+10 (secure 0 0 0) 0x7f5978050e00 con 0x7f59900b22c0 2021-02-10T11:47:30.649 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:47:30.651 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:30.648 7f5994e75700 1 -- 172.21.2.112:0/3950910978 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f59740186c0 msgr2=0x7f597401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:30.651 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:30.648 7f5994e75700 1 --2- 172.21.2.112:0/3950910978 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f59740186c0 0x7f597401abc0 crc :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:30.651 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:30.648 7f5994e75700 1 -- 172.21.2.112:0/3950910978 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f59900b22c0 msgr2=0x7f59900b26c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:30.651 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:30.648 7f5994e75700 1 --2- 172.21.2.112:0/3950910978 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f59900b22c0 0x7f59900b26c0 secure :-1 s=READY pgs=3050 cs=0 l=1 rev1=1 rx=0x7f597804e540 tx=0x7f5978050860).stop 2021-02-10T11:47:30.651 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:30.648 7f5994e75700 1 -- 172.21.2.112:0/3950910978 shutdown_connections 2021-02-10T11:47:30.651 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:30.648 7f5994e75700 1 --2- 172.21.2.112:0/3950910978 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f59740186c0 0x7f597401abc0 unknown :-1 s=CLOSED pgs=1373 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:30.651 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:30.648 7f5994e75700 1 --2- 172.21.2.112:0/3950910978 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f59900a5cd0 0x7f59900a60d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:30.651 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:30.648 7f5994e75700 1 --2- 172.21.2.112:0/3950910978 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f59900a6df0 0x7f59900a71f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:30.651 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:30.648 7f5994e75700 1 --2- 172.21.2.112:0/3950910978 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f59900b22c0 0x7f59900b26c0 unknown :-1 s=CLOSED pgs=3050 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:30.652 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:30.648 7f5994e75700 1 -- 172.21.2.112:0/3950910978 >> 172.21.2.112:0/3950910978 conn(0x7f599009ffd0 msgr2=0x7f5990082ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:30.652 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:30.648 7f5994e75700 1 -- 172.21.2.112:0/3950910978 shutdown_connections 2021-02-10T11:47:30.652 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:30.648 7f5994e75700 1 -- 172.21.2.112:0/3950910978 wait complete. 2021-02-10T11:47:30.659 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:47:31.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.071 7f7367242700 1 Processor -- start 2021-02-10T11:47:31.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.071 7f7367242700 1 -- start start 2021-02-10T11:47:31.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.071 7f7367242700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f73600d1870 0x7f73601b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:31.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.071 7f7367242700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f73600cff70 0x7f7360109130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:31.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.071 7f7367242700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73600fe1c0 con 0x7f73601b0ae0 2021-02-10T11:47:31.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.071 7f7367242700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7360109ad0 con 0x7f73600cff70 2021-02-10T11:47:31.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.071 7f7367242700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7360109750 con 0x7f73600d1870 2021-02-10T11:47:31.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.072 7f736585f700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f73600cff70 0x7f7360109130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:31.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.072 7f736585f700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f73600cff70 0x7f7360109130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:60504/0 (socket says 172.21.2.112:60504) 2021-02-10T11:47:31.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.072 7f736485d700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f73601b0ae0 0x7f7360108940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:59966/0 (socket says 172.21.2.112:59966) 2021-02-10T11:47:31.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.072 7f736485d700 1 -- 172.21.2.112:0/24987145 learned_addr learned my addr 172.21.2.112:0/24987145 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:47:31.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.072 7f736505e700 1 --2- 172.21.2.112:0/24987145 >> v2:172.21.2.120:3300/0 conn(0x7f73600d1870 0x7f73601b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:31.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.072 7f736585f700 1 -- 172.21.2.112:0/24987145 >> v2:172.21.2.120:3300/0 conn(0x7f73600d1870 msgr2=0x7f73601b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:31.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.072 7f736585f700 1 --2- 172.21.2.112:0/24987145 >> v2:172.21.2.120:3300/0 conn(0x7f73600d1870 0x7f73601b04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:31.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.072 7f736585f700 1 -- 172.21.2.112:0/24987145 >> v1:172.21.2.112:6789/0 conn(0x7f73601b0ae0 legacy=0x7f7360108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:31.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.072 7f736585f700 1 -- 172.21.2.112:0/24987145 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7360088070 con 0x7f73600cff70 2021-02-10T11:47:31.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.072 7f736585f700 1 --2- 172.21.2.112:0/24987145 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f73600cff70 0x7f7360109130 secure :-1 s=READY pgs=3052 cs=0 l=1 rev1=1 rx=0x7f7358006e30 tx=0x7f7358009e60).ready entity=mon.0 client_cookie=4fe3889b86ae67fe server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:31.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.072 7f7353fff700 1 -- 172.21.2.112:0/24987145 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f735800b300 con 0x7f73600cff70 2021-02-10T11:47:31.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.072 7f7353fff700 1 -- 172.21.2.112:0/24987145 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7358000d30 con 0x7f73600cff70 2021-02-10T11:47:31.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.072 7f7353fff700 1 -- 172.21.2.112:0/24987145 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f735800b800 con 0x7f73600cff70 2021-02-10T11:47:31.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.072 7f7367242700 1 -- 172.21.2.112:0/24987145 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f73600cff70 msgr2=0x7f7360109130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:31.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.072 7f7367242700 1 --2- 172.21.2.112:0/24987145 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f73600cff70 0x7f7360109130 secure :-1 s=READY pgs=3052 cs=0 l=1 rev1=1 rx=0x7f7358006e30 tx=0x7f7358009e60).stop 2021-02-10T11:47:31.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.073 7f7367242700 1 -- 172.21.2.112:0/24987145 shutdown_connections 2021-02-10T11:47:31.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.073 7f7367242700 1 --2- 172.21.2.112:0/24987145 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f73600cff70 0x7f7360109130 unknown :-1 s=CLOSED pgs=3052 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:31.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.073 7f7367242700 1 --2- 172.21.2.112:0/24987145 >> v2:172.21.2.120:3300/0 conn(0x7f73600d1870 0x7f73601b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:31.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.073 7f7367242700 1 -- 172.21.2.112:0/24987145 >> 172.21.2.112:0/24987145 conn(0x7f73600ce420 msgr2=0x7f73600d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:31.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.073 7f7367242700 1 -- 172.21.2.112:0/24987145 shutdown_connections 2021-02-10T11:47:31.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.073 7f7367242700 1 -- 172.21.2.112:0/24987145 wait complete. 2021-02-10T11:47:31.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.088 7f7367242700 1 Processor -- start 2021-02-10T11:47:31.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.088 7f7367242700 1 -- start start 2021-02-10T11:47:31.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.089 7f7367242700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7360191b10 0x7f73600ae640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:31.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.089 7f7367242700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7360191f10 0x7f736019ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:31.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.089 7f7367242700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f73601a05f0 0x7f7360082ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:31.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.089 7f7367242700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7360192440 con 0x7f7360191b10 2021-02-10T11:47:31.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.089 7f7367242700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73601916e0 con 0x7f73601a05f0 2021-02-10T11:47:31.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.089 7f7367242700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7360191900 con 0x7f7360191f10 2021-02-10T11:47:31.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.089 7f736585f700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f73601a05f0 0x7f7360082ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:31.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.089 7f736505e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7360191b10 0x7f73600ae640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:31.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.089 7f736585f700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f73601a05f0 0x7f7360082ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3300/0 says I am v2:172.21.2.112:34556/0 (socket says 172.21.2.112:34556) 2021-02-10T11:47:31.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.089 7f736585f700 1 -- 172.21.2.112:0/4135172671 learned_addr learned my addr 172.21.2.112:0/4135172671 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:47:31.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.089 7f736485d700 1 --2- 172.21.2.112:0/4135172671 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7360191f10 0x7f736019ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:31.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.089 7f736585f700 1 -- 172.21.2.112:0/4135172671 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7360191f10 msgr2=0x7f736019ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:31.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.089 7f736585f700 1 --2- 172.21.2.112:0/4135172671 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7360191f10 0x7f736019ffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:31.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.089 7f736585f700 1 -- 172.21.2.112:0/4135172671 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7360191b10 msgr2=0x7f73600ae640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:31.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.089 7f736585f700 1 --2- 172.21.2.112:0/4135172671 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7360191b10 0x7f73600ae640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:31.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.089 7f736585f700 1 -- 172.21.2.112:0/4135172671 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73600834f0 con 0x7f73601a05f0 2021-02-10T11:47:31.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.089 7f736585f700 1 --2- 172.21.2.112:0/4135172671 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f73601a05f0 0x7f7360082ed0 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 rx=0x7f735800b960 tx=0x7f735800bd00).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:31.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.090 7f7351ffb700 1 -- 172.21.2.112:0/4135172671 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7358009110 con 0x7f73601a05f0 2021-02-10T11:47:31.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.090 7f7351ffb700 1 -- 172.21.2.112:0/4135172671 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7358003030 con 0x7f73601a05f0 2021-02-10T11:47:31.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.090 7f7351ffb700 1 -- 172.21.2.112:0/4135172671 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7358001460 con 0x7f73601a05f0 2021-02-10T11:47:31.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.090 7f7367242700 1 -- 172.21.2.112:0/4135172671 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73600834f0 con 0x7f73601a05f0 2021-02-10T11:47:31.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.090 7f7367242700 1 -- 172.21.2.112:0/4135172671 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7360083a80 con 0x7f73601a05f0 2021-02-10T11:47:31.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.090 7f7351ffb700 1 -- 172.21.2.112:0/4135172671 <== mon.1 v2:172.21.2.120:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7358001270 con 0x7f73601a05f0 2021-02-10T11:47:31.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.090 7f7351ffb700 1 --2- 172.21.2.112:0/4135172671 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f73400186c0 0x7f734001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:31.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.090 7f7351ffb700 1 -- 172.21.2.112:0/4135172671 <== mon.1 v2:172.21.2.120:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f7358032960 con 0x7f73601a05f0 2021-02-10T11:47:31.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.091 7f736505e700 1 --2- 172.21.2.112:0/4135172671 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f73400186c0 0x7f734001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:31.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.091 7f736505e700 1 --2- 172.21.2.112:0/4135172671 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f73400186c0 0x7f734001abc0 crc :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:31.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.121 7f7367242700 1 -- 172.21.2.112:0/4135172671 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73600fe1c0 con 0x7f73601a05f0 2021-02-10T11:47:31.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.125 7f7351ffb700 1 -- 172.21.2.112:0/4135172671 <== mon.1 v2:172.21.2.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7358009110 con 0x7f73601a05f0 2021-02-10T11:47:31.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.255 7f7367242700 1 -- 172.21.2.112:0/4135172671 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f73600fe1c0 con 0x7f73601a05f0 2021-02-10T11:47:31.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.256 7f7351ffb700 1 -- 172.21.2.112:0/4135172671 <== mon.1 v2:172.21.2.120:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v180) v1 ==== 100+0+10 (secure 0 0 0) 0x7f7358001270 con 0x7f73601a05f0 2021-02-10T11:47:31.260 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:47:31.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.257 7f7367242700 1 -- 172.21.2.112:0/4135172671 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f73400186c0 msgr2=0x7f734001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:31.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.257 7f7367242700 1 --2- 172.21.2.112:0/4135172671 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f73400186c0 0x7f734001abc0 crc :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:31.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.257 7f7367242700 1 -- 172.21.2.112:0/4135172671 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f73601a05f0 msgr2=0x7f7360082ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:31.262 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.257 7f7367242700 1 --2- 172.21.2.112:0/4135172671 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f73601a05f0 0x7f7360082ed0 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 rx=0x7f735800b960 tx=0x7f735800bd00).stop 2021-02-10T11:47:31.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.258 7f7367242700 1 -- 172.21.2.112:0/4135172671 shutdown_connections 2021-02-10T11:47:31.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.258 7f7367242700 1 --2- 172.21.2.112:0/4135172671 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f73400186c0 0x7f734001abc0 unknown :-1 s=CLOSED pgs=1374 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:31.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.258 7f7367242700 1 --2- 172.21.2.112:0/4135172671 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f73601a05f0 0x7f7360082ed0 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:31.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.258 7f7367242700 1 --2- 172.21.2.112:0/4135172671 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7360191f10 0x7f736019ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:31.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.258 7f7367242700 1 --2- 172.21.2.112:0/4135172671 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7360191b10 0x7f73600ae640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:31.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.258 7f7367242700 1 -- 172.21.2.112:0/4135172671 >> 172.21.2.112:0/4135172671 conn(0x7f73600a02d0 msgr2=0x7f736009f200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:31.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.258 7f7367242700 1 -- 172.21.2.112:0/4135172671 shutdown_connections 2021-02-10T11:47:31.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.258 7f7367242700 1 -- 172.21.2.112:0/4135172671 wait complete. 2021-02-10T11:47:31.270 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:47:31.270 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:47:31.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.679 7f42f8827700 1 Processor -- start 2021-02-10T11:47:31.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.679 7f42f8827700 1 -- start start 2021-02-10T11:47:31.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.680 7f42f8827700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f42f40d1870 0x7f42f41b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:31.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.680 7f42f8827700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f42f41b0a10 0x7f42f40cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:31.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.680 7f42f8827700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42f40f0b10 con 0x7f42f40d1c70 2021-02-10T11:47:31.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.680 7f42f8827700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42f40e5ed0 con 0x7f42f40d1870 2021-02-10T11:47:31.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.680 7f42f8827700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42f40f0650 con 0x7f42f41b0a10 2021-02-10T11:47:31.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.680 7f42f261c700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f42f40d1870 0x7f42f41b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:31.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.680 7f42f261c700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f42f40d1870 0x7f42f41b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:60516/0 (socket says 172.21.2.112:60516) 2021-02-10T11:47:31.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.680 7f42f261c700 1 -- 172.21.2.112:0/4027248155 learned_addr learned my addr 172.21.2.112:0/4027248155 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:47:31.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.680 7f42f1e1b700 1 --2- 172.21.2.112:0/4027248155 >> v2:172.21.2.120:3300/0 conn(0x7f42f41b0a10 0x7f42f40cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:31.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.680 7f42f261c700 1 -- 172.21.2.112:0/4027248155 >> v2:172.21.2.120:3300/0 conn(0x7f42f41b0a10 msgr2=0x7f42f40cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:31.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.681 7f42f261c700 1 --2- 172.21.2.112:0/4027248155 >> v2:172.21.2.120:3300/0 conn(0x7f42f41b0a10 0x7f42f40cf330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:31.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.681 7f42f261c700 1 -- 172.21.2.112:0/4027248155 >> v1:172.21.2.120:6789/0 conn(0x7f42f40d1c70 legacy=0x7f42f40cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:31.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.681 7f42f261c700 1 -- 172.21.2.112:0/4027248155 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42f40f0840 con 0x7f42f40d1870 2021-02-10T11:47:31.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.681 7f42f261c700 1 --2- 172.21.2.112:0/4027248155 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f42f40d1870 0x7f42f41b03f0 secure :-1 s=READY pgs=3053 cs=0 l=1 rev1=1 rx=0x7f42e4006700 tx=0x7f42e4009b80).ready entity=mon.0 client_cookie=fabfa06a0a7e6ffb server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:31.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.681 7f42f161a700 1 -- 172.21.2.112:0/4027248155 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f42e400c0c0 con 0x7f42f40d1870 2021-02-10T11:47:31.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.681 7f42f161a700 1 -- 172.21.2.112:0/4027248155 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42e400c340 con 0x7f42f40d1870 2021-02-10T11:47:31.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.681 7f42f161a700 1 -- 172.21.2.112:0/4027248155 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f42e400c820 con 0x7f42f40d1870 2021-02-10T11:47:31.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.681 7f42f8827700 1 -- 172.21.2.112:0/4027248155 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f42f40d1870 msgr2=0x7f42f41b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:31.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.681 7f42f8827700 1 --2- 172.21.2.112:0/4027248155 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f42f40d1870 0x7f42f41b03f0 secure :-1 s=READY pgs=3053 cs=0 l=1 rev1=1 rx=0x7f42e4006700 tx=0x7f42e4009b80).stop 2021-02-10T11:47:31.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.682 7f42f8827700 1 -- 172.21.2.112:0/4027248155 shutdown_connections 2021-02-10T11:47:31.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.682 7f42f8827700 1 --2- 172.21.2.112:0/4027248155 >> v2:172.21.2.120:3300/0 conn(0x7f42f41b0a10 0x7f42f40cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:31.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.682 7f42f8827700 1 --2- 172.21.2.112:0/4027248155 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f42f40d1870 0x7f42f41b03f0 unknown :-1 s=CLOSED pgs=3053 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:31.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.682 7f42f8827700 1 -- 172.21.2.112:0/4027248155 >> 172.21.2.112:0/4027248155 conn(0x7f42f40ce450 msgr2=0x7f42f40d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:31.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.682 7f42f8827700 1 -- 172.21.2.112:0/4027248155 shutdown_connections 2021-02-10T11:47:31.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.682 7f42f8827700 1 -- 172.21.2.112:0/4027248155 wait complete. 2021-02-10T11:47:31.704 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.696 7f42f8827700 1 Processor -- start 2021-02-10T11:47:31.704 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.696 7f42f8827700 1 -- start start 2021-02-10T11:47:31.704 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.697 7f42f8827700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f42f408e360 0x7f42f40b7730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:31.704 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.697 7f42f8827700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f42f40a9600 0x7f42f40a9a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:31.704 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.697 7f42f8827700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f42f40aa0a0 0x7f42f40ecb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:31.704 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.697 7f42f8827700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42f408dde0 con 0x7f42f40a9600 2021-02-10T11:47:31.704 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.697 7f42f8827700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42f40ed130 con 0x7f42f40aa0a0 2021-02-10T11:47:31.704 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.697 7f42f8827700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42f40ed310 con 0x7f42f408e360 2021-02-10T11:47:31.704 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.697 7f42f1e1b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f42f40a9600 0x7f42f40a9a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:31.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.697 7f42f1e1b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f42f40a9600 0x7f42f40a9a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:60522/0 (socket says 172.21.2.112:60522) 2021-02-10T11:47:31.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.697 7f42f1e1b700 1 -- 172.21.2.112:0/921036119 learned_addr learned my addr 172.21.2.112:0/921036119 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:47:31.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.697 7f42f261c700 1 --2- 172.21.2.112:0/921036119 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f42f408e360 0x7f42f40b7730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:31.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.697 7f42f2e1d700 1 --2- 172.21.2.112:0/921036119 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f42f40aa0a0 0x7f42f40ecb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:31.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.697 7f42f1e1b700 1 -- 172.21.2.112:0/921036119 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f42f408e360 msgr2=0x7f42f40b7730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:31.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.697 7f42f1e1b700 1 --2- 172.21.2.112:0/921036119 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f42f408e360 0x7f42f40b7730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:31.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.697 7f42f1e1b700 1 -- 172.21.2.112:0/921036119 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f42f40aa0a0 msgr2=0x7f42f40ecb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:31.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.697 7f42f1e1b700 1 --2- 172.21.2.112:0/921036119 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f42f40aa0a0 0x7f42f40ecb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:31.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.697 7f42f1e1b700 1 -- 172.21.2.112:0/921036119 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42f40ed530 con 0x7f42f40a9600 2021-02-10T11:47:31.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.697 7f42f261c700 1 --2- 172.21.2.112:0/921036119 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f42f408e360 0x7f42f40b7730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:47:31.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.698 7f42f1e1b700 1 --2- 172.21.2.112:0/921036119 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f42f40a9600 0x7f42f40a9a80 secure :-1 s=READY pgs=3054 cs=0 l=1 rev1=1 rx=0x7f42dc006f60 tx=0x7f42dc00b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:31.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.698 7f42eaffd700 1 -- 172.21.2.112:0/921036119 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f42dc00bd60 con 0x7f42f40a9600 2021-02-10T11:47:31.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.698 7f42eaffd700 1 -- 172.21.2.112:0/921036119 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42dc000c90 con 0x7f42f40a9600 2021-02-10T11:47:31.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.698 7f42eaffd700 1 -- 172.21.2.112:0/921036119 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f42dc0022b0 con 0x7f42f40a9600 2021-02-10T11:47:31.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.698 7f42f8827700 1 -- 172.21.2.112:0/921036119 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42f40ed530 con 0x7f42f40a9600 2021-02-10T11:47:31.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.698 7f42f8827700 1 -- 172.21.2.112:0/921036119 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42f408e780 con 0x7f42f40a9600 2021-02-10T11:47:31.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.699 7f42eaffd700 1 -- 172.21.2.112:0/921036119 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f42dc006b00 con 0x7f42f40a9600 2021-02-10T11:47:31.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.699 7f42eaffd700 1 --2- 172.21.2.112:0/921036119 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f42d80186c0 0x7f42d801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:31.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.699 7f42eaffd700 1 -- 172.21.2.112:0/921036119 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f42dc032570 con 0x7f42f40a9600 2021-02-10T11:47:31.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.699 7f42f261c700 1 --2- 172.21.2.112:0/921036119 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f42d80186c0 0x7f42d801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:31.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.700 7f42f261c700 1 --2- 172.21.2.112:0/921036119 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f42d80186c0 0x7f42d801abc0 crc :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:31.733 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.730 7f42f8827700 1 -- 172.21.2.112:0/921036119 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42f40ebc00 con 0x7f42f40a9600 2021-02-10T11:47:31.736 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.732 7f42eaffd700 1 -- 172.21.2.112:0/921036119 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f42dc000940 con 0x7f42f40a9600 2021-02-10T11:47:31.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.860 7f42f8827700 1 -- 172.21.2.112:0/921036119 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f42f40f0b10 con 0x7f42f40a9600 2021-02-10T11:47:31.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.860 7f42eaffd700 1 -- 172.21.2.112:0/921036119 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 156 v156) v1 ==== 95+0+2108 (secure 0 0 0) 0x7f42dc000940 con 0x7f42f40a9600 2021-02-10T11:47:31.865 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:47:31.865 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":156,"default_fscid":19,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":13038,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/4134422613","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":4134422613},{"type":"v1","addr":"172.21.2.112:6833","nonce":4134422613}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":152}],"filesystems":[{"mdsmap":{"epoch":156,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:47:14.425711","modified":"2021-02-10 11:47:16.434766","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9307},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9307":{"gid":9307,"name":"b","rank":0,"incarnation":153,"state":"up:active","state_seq":3,"addr":"172.21.2.120:6835/2625088905","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":2625088905},{"type":"v1","addr":"172.21.2.120:6835","nonce":2625088905}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[39],"metadata_pool":38,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":19}]} 2021-02-10T11:47:31.865 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 156 2021-02-10T11:47:31.866 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.862 7f42f8827700 1 -- 172.21.2.112:0/921036119 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f42d80186c0 msgr2=0x7f42d801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:31.866 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.862 7f42f8827700 1 --2- 172.21.2.112:0/921036119 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f42d80186c0 0x7f42d801abc0 crc :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:31.867 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.862 7f42f8827700 1 -- 172.21.2.112:0/921036119 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f42f40a9600 msgr2=0x7f42f40a9a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:31.867 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.862 7f42f8827700 1 --2- 172.21.2.112:0/921036119 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f42f40a9600 0x7f42f40a9a80 secure :-1 s=READY pgs=3054 cs=0 l=1 rev1=1 rx=0x7f42dc006f60 tx=0x7f42dc00b970).stop 2021-02-10T11:47:31.867 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.862 7f42f8827700 1 -- 172.21.2.112:0/921036119 shutdown_connections 2021-02-10T11:47:31.867 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.862 7f42f8827700 1 --2- 172.21.2.112:0/921036119 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f42d80186c0 0x7f42d801abc0 unknown :-1 s=CLOSED pgs=1375 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:31.867 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.863 7f42f8827700 1 --2- 172.21.2.112:0/921036119 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f42f40aa0a0 0x7f42f40ecb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:31.867 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.863 7f42f8827700 1 --2- 172.21.2.112:0/921036119 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f42f40a9600 0x7f42f40a9a80 unknown :-1 s=CLOSED pgs=3054 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:31.867 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.863 7f42f8827700 1 --2- 172.21.2.112:0/921036119 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f42f408e360 0x7f42f40b7730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:31.867 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.863 7f42f8827700 1 -- 172.21.2.112:0/921036119 >> 172.21.2.112:0/921036119 conn(0x7f42f40b7e70 msgr2=0x7f42f40c3850 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:31.867 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.863 7f42f8827700 1 -- 172.21.2.112:0/921036119 shutdown_connections 2021-02-10T11:47:31.867 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:31.863 7f42f8827700 1 -- 172.21.2.112:0/921036119 wait complete. 2021-02-10T11:47:32.076 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 156, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:47:14.425711', 'modified': '2021-02-10 11:47:16.434766', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9307}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9307': {'gid': 9307, 'name': 'b', 'rank': 0, 'incarnation': 153, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.120:6835/2625088905', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.120:6834', 'nonce': 2625088905}, {'type': 'v1', 'addr': '172.21.2.120:6835', 'nonce': 2625088905}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [39], 'metadata_pool': 38, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:47:32.077 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:47:32.077 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:47:32.479 INFO:teuthology.orchestra.run.gibba020.stdout:{ 2021-02-10T11:47:32.479 INFO:teuthology.orchestra.run.gibba020.stdout: "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", 2021-02-10T11:47:32.480 INFO:teuthology.orchestra.run.gibba020.stdout: "whoami": 0, 2021-02-10T11:47:32.480 INFO:teuthology.orchestra.run.gibba020.stdout: "id": 9307, 2021-02-10T11:47:32.480 INFO:teuthology.orchestra.run.gibba020.stdout: "want_state": "up:active", 2021-02-10T11:47:32.480 INFO:teuthology.orchestra.run.gibba020.stdout: "state": "up:active", 2021-02-10T11:47:32.480 INFO:teuthology.orchestra.run.gibba020.stdout: "rank_uptime": 18.049058999, 2021-02-10T11:47:32.480 INFO:teuthology.orchestra.run.gibba020.stdout: "mdsmap_epoch": 156, 2021-02-10T11:47:32.480 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch": 180, 2021-02-10T11:47:32.481 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch_barrier": 180, 2021-02-10T11:47:32.481 INFO:teuthology.orchestra.run.gibba020.stdout: "uptime": 23.699120668999999 2021-02-10T11:47:32.481 INFO:teuthology.orchestra.run.gibba020.stdout:} 2021-02-10T11:47:32.483 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", "whoami": 0, "id": 9307, "want_state": "up:active", "state": "up:active", "rank_uptime": 18.049058999, "mdsmap_epoch": 156, "osdmap_epoch": 180, "osdmap_epoch_barrier": 180, "uptime": 23.699120668999999 } 2021-02-10T11:47:32.484 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:47:32.484 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2021-02-10T11:47:32.484 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.1 at ubuntu@gibba012.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2021-02-10T11:47:32.484 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:47:32.484 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:47:32.826 DEBUG:teuthology.orchestra.run.gibba012:> sudo modprobe fuse 2021-02-10T11:47:33.170 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:47:33.171 DEBUG:teuthology.orchestra.run.gibba012:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:47:33.514 INFO:teuthology.orchestra.run.gibba012.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:47:33.514 INFO:teuthology.orchestra.run.gibba012.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:47:33.515 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:47:33.515 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:47:33.515 DEBUG:teuthology.orchestra.run.gibba012:> ls /sys/fs/fuse/connections 2021-02-10T11:47:33.851 INFO:teuthology.orchestra.run.gibba012.stdout:42 2021-02-10T11:47:33.851 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [42] 2021-02-10T11:47:33.852 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.1 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:47:33.854 DEBUG:teuthology.orchestra.run.gibba012:> sudo modprobe fuse 2021-02-10T11:47:34.198 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:47:34.198 DEBUG:teuthology.orchestra.run.gibba012:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:47:34.266 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba012.stderr:ceph-fuse[188743]: starting ceph client 2021-02-10T11:47:34.266 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba012.stderr:2021-02-10 11:47:34.262 7fd578cddf80 -1 init, newargv = 0x55e104aea1f0 newargc=9 2021-02-10T11:47:34.272 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba012.stderr:ceph-fuse[188743]: starting fuse 2021-02-10T11:47:34.540 INFO:teuthology.orchestra.run.gibba012.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:47:34.541 INFO:teuthology.orchestra.run.gibba012.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:47:34.541 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:47:34.542 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:47:34.542 DEBUG:teuthology.orchestra.run.gibba012:> ls /sys/fs/fuse/connections 2021-02-10T11:47:34.878 INFO:teuthology.orchestra.run.gibba012.stdout:42 2021-02-10T11:47:34.878 INFO:teuthology.orchestra.run.gibba012.stdout:43 2021-02-10T11:47:34.878 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42, 43] 2021-02-10T11:47:34.879 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:47:34.879 DEBUG:teuthology.orchestra.run.gibba012:> sudo python3 -c ' 2021-02-10T11:47:34.879 DEBUG:teuthology.orchestra.run.gibba012:> import glob 2021-02-10T11:47:34.879 DEBUG:teuthology.orchestra.run.gibba012:> import re 2021-02-10T11:47:34.879 DEBUG:teuthology.orchestra.run.gibba012:> import os 2021-02-10T11:47:34.879 DEBUG:teuthology.orchestra.run.gibba012:> import subprocess 2021-02-10T11:47:34.879 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:47:34.879 DEBUG:teuthology.orchestra.run.gibba012:> def find_socket(client_name): 2021-02-10T11:47:34.879 DEBUG:teuthology.orchestra.run.gibba012:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2021-02-10T11:47:34.879 DEBUG:teuthology.orchestra.run.gibba012:> files = glob.glob(asok_path) 2021-02-10T11:47:34.879 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:47:34.879 DEBUG:teuthology.orchestra.run.gibba012:> # Given a non-glob path, it better be there 2021-02-10T11:47:34.879 DEBUG:teuthology.orchestra.run.gibba012:> if "*" not in asok_path: 2021-02-10T11:47:34.880 DEBUG:teuthology.orchestra.run.gibba012:> assert(len(files) == 1) 2021-02-10T11:47:34.880 DEBUG:teuthology.orchestra.run.gibba012:> return files[0] 2021-02-10T11:47:34.880 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:47:34.880 DEBUG:teuthology.orchestra.run.gibba012:> for f in files: 2021-02-10T11:47:34.880 DEBUG:teuthology.orchestra.run.gibba012:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:47:34.880 DEBUG:teuthology.orchestra.run.gibba012:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:47:34.880 DEBUG:teuthology.orchestra.run.gibba012:> return f 2021-02-10T11:47:34.880 DEBUG:teuthology.orchestra.run.gibba012:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:47:34.880 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:47:34.880 DEBUG:teuthology.orchestra.run.gibba012:> print(find_socket("client.1")) 2021-02-10T11:47:34.880 DEBUG:teuthology.orchestra.run.gibba012:> ' 2021-02-10T11:47:35.242 INFO:teuthology.orchestra.run.gibba012.stdout:/var/run/ceph/ceph-client.1.188743.asok 2021-02-10T11:47:35.245 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.188743.asok 2021-02-10T11:47:35.245 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:47:35.245 DEBUG:teuthology.orchestra.run.gibba012:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.188743.asok status 2021-02-10T11:47:35.639 INFO:teuthology.orchestra.run.gibba012.stdout:{ 2021-02-10T11:47:35.639 INFO:teuthology.orchestra.run.gibba012.stdout: "metadata": { 2021-02-10T11:47:35.639 INFO:teuthology.orchestra.run.gibba012.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:47:35.639 INFO:teuthology.orchestra.run.gibba012.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:47:35.639 INFO:teuthology.orchestra.run.gibba012.stdout: "entity_id": "1", 2021-02-10T11:47:35.639 INFO:teuthology.orchestra.run.gibba012.stdout: "hostname": "gibba012", 2021-02-10T11:47:35.639 INFO:teuthology.orchestra.run.gibba012.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2021-02-10T11:47:35.640 INFO:teuthology.orchestra.run.gibba012.stdout: "pid": "188743", 2021-02-10T11:47:35.640 INFO:teuthology.orchestra.run.gibba012.stdout: "root": "/" 2021-02-10T11:47:35.640 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:35.640 INFO:teuthology.orchestra.run.gibba012.stdout: "dentry_count": 0, 2021-02-10T11:47:35.640 INFO:teuthology.orchestra.run.gibba012.stdout: "dentry_pinned_count": 0, 2021-02-10T11:47:35.640 INFO:teuthology.orchestra.run.gibba012.stdout: "id": 13254, 2021-02-10T11:47:35.640 INFO:teuthology.orchestra.run.gibba012.stdout: "inst": { 2021-02-10T11:47:35.640 INFO:teuthology.orchestra.run.gibba012.stdout: "name": { 2021-02-10T11:47:35.640 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "client", 2021-02-10T11:47:35.640 INFO:teuthology.orchestra.run.gibba012.stdout: "num": 13254 2021-02-10T11:47:35.640 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:35.640 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": { 2021-02-10T11:47:35.640 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:47:35.641 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:0", 2021-02-10T11:47:35.641 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 614751487 2021-02-10T11:47:35.641 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:47:35.641 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:35.641 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": { 2021-02-10T11:47:35.641 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:47:35.641 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:0", 2021-02-10T11:47:35.641 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 614751487 2021-02-10T11:47:35.641 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:35.641 INFO:teuthology.orchestra.run.gibba012.stdout: "inst_str": "client.13254 172.21.2.112:0/614751487", 2021-02-10T11:47:35.641 INFO:teuthology.orchestra.run.gibba012.stdout: "addr_str": "172.21.2.112:0/614751487", 2021-02-10T11:47:35.641 INFO:teuthology.orchestra.run.gibba012.stdout: "inode_count": 1, 2021-02-10T11:47:35.641 INFO:teuthology.orchestra.run.gibba012.stdout: "mds_epoch": 156, 2021-02-10T11:47:35.641 INFO:teuthology.orchestra.run.gibba012.stdout: "osd_epoch": 180, 2021-02-10T11:47:35.642 INFO:teuthology.orchestra.run.gibba012.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:47:35.642 INFO:teuthology.orchestra.run.gibba012.stdout: "blacklisted": false 2021-02-10T11:47:35.642 INFO:teuthology.orchestra.run.gibba012.stdout:} 2021-02-10T11:47:35.643 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:47:35.643 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:47:35.643 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:47:35.981 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:47:35.981 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:47:35.981 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:47:35.981 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:47:36.328 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2021-02-10T11:47:36.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:36.741 7fc0f8a96700 1 Processor -- start 2021-02-10T11:47:36.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:36.741 7fc0f8a96700 1 -- start start 2021-02-10T11:47:36.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:36.742 7fc0f8a96700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fc0f40d1880 0x7fc0f41b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:36.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:36.742 7fc0f8a96700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fc0f41b0af0 0x7fc0f4108950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:36.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:36.742 7fc0f8a96700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0f40fe1d0 con 0x7fc0f40d1c80 2021-02-10T11:47:36.761 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:36.742 7fc0f8a96700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0f4109a70 con 0x7fc0f40d1880 2021-02-10T11:47:36.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:36.742 7fc0f8a96700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0f4109760 con 0x7fc0f41b0af0 2021-02-10T11:47:36.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:36.742 7fc0f305f700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fc0f40d1c80 0x7fc0f4108f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:60012/0 (socket says 172.21.2.112:60012) 2021-02-10T11:47:36.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:36.742 7fc0f305f700 1 -- 172.21.2.112:0/370637069 learned_addr learned my addr 172.21.2.112:0/370637069 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:47:36.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:36.742 7fc0f285e700 1 --2- 172.21.2.112:0/370637069 >> v2:172.21.2.112:3300/0 conn(0x7fc0f40d1880 0x7fc0f41b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:36.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:36.742 7fc0f285e700 1 -- 172.21.2.112:0/370637069 >> v2:172.21.2.120:3300/0 conn(0x7fc0f41b0af0 msgr2=0x7fc0f4108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:36.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:36.742 7fc0f285e700 1 --2- 172.21.2.112:0/370637069 >> v2:172.21.2.120:3300/0 conn(0x7fc0f41b0af0 0x7fc0f4108950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:36.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:36.742 7fc0f285e700 1 -- 172.21.2.112:0/370637069 >> v1:172.21.2.112:6789/0 conn(0x7fc0f40d1c80 legacy=0x7fc0f4108f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:36.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:36.742 7fc0f185c700 1 -- 172.21.2.112:0/370637069 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1146512687 0 0) 0x7fc0ec001500 con 0x7fc0f40d1c80 2021-02-10T11:47:36.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:36.743 7fc0f285e700 1 -- 172.21.2.112:0/370637069 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0f4088070 con 0x7fc0f40d1880 2021-02-10T11:47:36.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:36.743 7fc0f285e700 1 --2- 172.21.2.112:0/370637069 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc0f40d1880 0x7fc0f41b04d0 secure :-1 s=READY pgs=3058 cs=0 l=1 rev1=1 rx=0x7fc0e4006650 tx=0x7fc0e4009a00).ready entity=mon.0 client_cookie=6c0bc7894286ce9 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:36.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:36.743 7fc0f185c700 1 -- 172.21.2.112:0/370637069 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc0e40009b0 con 0x7fc0f40d1880 2021-02-10T11:47:36.763 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:36.743 7fc0f185c700 1 -- 172.21.2.112:0/370637069 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0e400c0c0 con 0x7fc0f40d1880 2021-02-10T11:47:36.763 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:36.743 7fc0f185c700 1 -- 172.21.2.112:0/370637069 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc0e400c5a0 con 0x7fc0f40d1880 2021-02-10T11:47:36.763 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:36.743 7fc0f8a96700 1 -- 172.21.2.112:0/370637069 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc0f40d1880 msgr2=0x7fc0f41b04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:36.763 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:36.743 7fc0f8a96700 1 --2- 172.21.2.112:0/370637069 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc0f40d1880 0x7fc0f41b04d0 secure :-1 s=READY pgs=3058 cs=0 l=1 rev1=1 rx=0x7fc0e4006650 tx=0x7fc0e4009a00).stop 2021-02-10T11:47:36.763 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:36.744 7fc0f8a96700 1 -- 172.21.2.112:0/370637069 shutdown_connections 2021-02-10T11:47:36.763 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:36.744 7fc0f8a96700 1 --2- 172.21.2.112:0/370637069 >> v2:172.21.2.120:3300/0 conn(0x7fc0f41b0af0 0x7fc0f4108950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:36.763 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:36.744 7fc0f8a96700 1 --2- 172.21.2.112:0/370637069 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc0f40d1880 0x7fc0f41b04d0 unknown :-1 s=CLOSED pgs=3058 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:36.763 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:36.744 7fc0f8a96700 1 -- 172.21.2.112:0/370637069 >> 172.21.2.112:0/370637069 conn(0x7fc0f40ce430 msgr2=0x7fc0f40d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:36.763 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:36.744 7fc0f8a96700 1 -- 172.21.2.112:0/370637069 shutdown_connections 2021-02-10T11:47:36.763 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:36.744 7fc0f8a96700 1 -- 172.21.2.112:0/370637069 wait complete. 2021-02-10T11:47:36.765 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:36.758 7fc0f8a96700 1 Processor -- start 2021-02-10T11:47:36.765 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:36.759 7fc0f8a96700 1 -- start start 2021-02-10T11:47:36.765 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:36.759 7fc0f8a96700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc0f4190df0 0x7fc0f41b7ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:36.765 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:36.759 7fc0f8a96700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc0f419ba20 0x7fc0f41b87c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:36.765 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:36.759 7fc0f8a96700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc0f419be20 0x7fc0f418f830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:36.765 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:36.759 7fc0f8a96700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0f41b8240 con 0x7fc0f4190df0 2021-02-10T11:47:36.765 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:36.759 7fc0f8a96700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0f41911f0 con 0x7fc0f419ba20 2021-02-10T11:47:36.765 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:36.759 7fc0f8a96700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0f41908a0 con 0x7fc0f419be20 2021-02-10T11:47:36.766 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:36.759 7fc0f285e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc0f4190df0 0x7fc0f41b7ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:36.766 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:36.759 7fc0f285e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc0f4190df0 0x7fc0f41b7ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:60554/0 (socket says 172.21.2.112:60554) 2021-02-10T11:47:36.766 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:36.759 7fc0f285e700 1 -- 172.21.2.112:0/2922066508 learned_addr learned my addr 172.21.2.112:0/2922066508 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:47:36.766 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:36.759 7fc0f285e700 1 -- 172.21.2.112:0/2922066508 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc0f419be20 msgr2=0x7fc0f418f830 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:47:36.766 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:36.759 7fc0f285e700 1 --2- 172.21.2.112:0/2922066508 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc0f419be20 0x7fc0f418f830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:36.766 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:36.759 7fc0f285e700 1 -- 172.21.2.112:0/2922066508 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc0f419ba20 msgr2=0x7fc0f41b87c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:47:36.766 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:36.759 7fc0f285e700 1 --2- 172.21.2.112:0/2922066508 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc0f419ba20 0x7fc0f41b87c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:36.766 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:36.759 7fc0f285e700 1 -- 172.21.2.112:0/2922066508 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0f41b8de0 con 0x7fc0f4190df0 2021-02-10T11:47:36.767 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:36.760 7fc0f285e700 1 --2- 172.21.2.112:0/2922066508 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc0f4190df0 0x7fc0f41b7ac0 secure :-1 s=READY pgs=3059 cs=0 l=1 rev1=1 rx=0x7fc0e400b590 tx=0x7fc0e400b930).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:36.767 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:36.760 7fc0e37fe700 1 -- 172.21.2.112:0/2922066508 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc0e400bd20 con 0x7fc0f4190df0 2021-02-10T11:47:36.767 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:36.760 7fc0e37fe700 1 -- 172.21.2.112:0/2922066508 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0e4000c90 con 0x7fc0f4190df0 2021-02-10T11:47:36.767 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:36.760 7fc0e37fe700 1 -- 172.21.2.112:0/2922066508 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc0e40022b0 con 0x7fc0f4190df0 2021-02-10T11:47:36.767 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:36.760 7fc0f8a96700 1 -- 172.21.2.112:0/2922066508 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0f41b8de0 con 0x7fc0f4190df0 2021-02-10T11:47:36.767 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:36.760 7fc0f8a96700 1 -- 172.21.2.112:0/2922066508 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0f41b8de0 con 0x7fc0f4190df0 2021-02-10T11:47:36.767 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:36.760 7fc0e37fe700 1 -- 172.21.2.112:0/2922066508 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc0e40052e0 con 0x7fc0f4190df0 2021-02-10T11:47:36.767 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:36.760 7fc0e37fe700 1 --2- 172.21.2.112:0/2922066508 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc0d80186c0 0x7fc0d801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:36.767 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:36.760 7fc0e37fe700 1 -- 172.21.2.112:0/2922066508 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fc0e40323e0 con 0x7fc0f4190df0 2021-02-10T11:47:36.767 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:36.761 7fc0f205d700 1 --2- 172.21.2.112:0/2922066508 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc0d80186c0 0x7fc0d801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:36.767 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:36.762 7fc0f205d700 1 --2- 172.21.2.112:0/2922066508 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc0d80186c0 0x7fc0d801abc0 crc :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:36.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:36.792 7fc0f8a96700 1 -- 172.21.2.112:0/2922066508 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0f4082aa0 con 0x7fc0f4190df0 2021-02-10T11:47:36.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:36.795 7fc0e37fe700 1 -- 172.21.2.112:0/2922066508 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc0e4006ac0 con 0x7fc0f4190df0 2021-02-10T11:47:36.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:36.930 7fc0f8a96700 1 -- 172.21.2.112:0/2922066508 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fc0f4082aa0 con 0x7fc0f4190df0 2021-02-10T11:47:37.295 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:37.292 7fc0e37fe700 1 -- 172.21.2.112:0/2922066508 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v157) v1 ==== 136+0+0 (secure 0 0 0) 0x7fc0e40052e0 con 0x7fc0f4190df0 2021-02-10T11:47:37.302 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:37.299 7fc0f8a96700 1 -- 172.21.2.112:0/2922066508 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc0d80186c0 msgr2=0x7fc0d801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:37.302 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:37.299 7fc0f8a96700 1 --2- 172.21.2.112:0/2922066508 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc0d80186c0 0x7fc0d801abc0 crc :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:37.302 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:37.299 7fc0f8a96700 1 -- 172.21.2.112:0/2922066508 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc0f4190df0 msgr2=0x7fc0f41b7ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:37.302 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:37.299 7fc0f8a96700 1 --2- 172.21.2.112:0/2922066508 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc0f4190df0 0x7fc0f41b7ac0 secure :-1 s=READY pgs=3059 cs=0 l=1 rev1=1 rx=0x7fc0e400b590 tx=0x7fc0e400b930).stop 2021-02-10T11:47:37.303 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:37.299 7fc0f8a96700 1 -- 172.21.2.112:0/2922066508 shutdown_connections 2021-02-10T11:47:37.303 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:37.299 7fc0f8a96700 1 --2- 172.21.2.112:0/2922066508 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc0d80186c0 0x7fc0d801abc0 unknown :-1 s=CLOSED pgs=1376 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:37.303 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:37.299 7fc0f8a96700 1 --2- 172.21.2.112:0/2922066508 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc0f419ba20 0x7fc0f41b87c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:37.303 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:37.299 7fc0f8a96700 1 --2- 172.21.2.112:0/2922066508 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc0f419be20 0x7fc0f418f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:37.303 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:37.299 7fc0f8a96700 1 --2- 172.21.2.112:0/2922066508 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc0f4190df0 0x7fc0f41b7ac0 unknown :-1 s=CLOSED pgs=3059 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:37.303 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:37.299 7fc0f8a96700 1 -- 172.21.2.112:0/2922066508 >> 172.21.2.112:0/2922066508 conn(0x7fc0f419f940 msgr2=0x7fc0f418c290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:37.303 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:37.299 7fc0f8a96700 1 -- 172.21.2.112:0/2922066508 shutdown_connections 2021-02-10T11:47:37.303 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:37.299 7fc0f8a96700 1 -- 172.21.2.112:0/2922066508 wait complete. 2021-02-10T11:47:37.310 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs volume ls 2021-02-10T11:47:37.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:37.719 7f20c9600700 1 Processor -- start 2021-02-10T11:47:37.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:37.719 7f20c9600700 1 -- start start 2021-02-10T11:47:37.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:37.719 7f20c9600700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f20c40d1860 0x7f20c41b03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:37.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:37.719 7f20c9600700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f20c41b0a00 0x7f20c40cf320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:37.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:37.719 7f20c9600700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f20c40d1c60 0x7f20c40cf940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:37.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:37.719 7f20c9600700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20c40fe1f0 con 0x7f20c40d1c60 2021-02-10T11:47:37.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:37.719 7f20c9600700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20c40e5f00 con 0x7f20c40d1860 2021-02-10T11:47:37.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:37.719 7f20c9600700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20c40f0b40 con 0x7f20c41b0a00 2021-02-10T11:47:37.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:37.719 7f20c3ac0700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f20c40d1c60 0x7f20c40cf940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:37.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:37.719 7f20c3ac0700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f20c40d1c60 0x7f20c40cf940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:60562/0 (socket says 172.21.2.112:60562) 2021-02-10T11:47:37.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:37.719 7f20c3ac0700 1 -- 172.21.2.112:0/3489832592 learned_addr learned my addr 172.21.2.112:0/3489832592 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:47:37.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:37.720 7f20c3ac0700 1 -- 172.21.2.112:0/3489832592 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f20c41b0a00 msgr2=0x7f20c40cf320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:37.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:37.720 7f20c3ac0700 1 --2- 172.21.2.112:0/3489832592 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f20c41b0a00 0x7f20c40cf320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:37.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:37.720 7f20c3ac0700 1 -- 172.21.2.112:0/3489832592 >> v2:172.21.2.120:3300/0 conn(0x7f20c40d1860 msgr2=0x7f20c41b03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:37.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:37.720 7f20c3ac0700 1 --2- 172.21.2.112:0/3489832592 >> v2:172.21.2.120:3300/0 conn(0x7f20c40d1860 0x7f20c41b03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:37.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:37.720 7f20c3ac0700 1 -- 172.21.2.112:0/3489832592 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20c40cff60 con 0x7f20c40d1c60 2021-02-10T11:47:37.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:37.720 7f20c3ac0700 1 --2- 172.21.2.112:0/3489832592 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f20c40d1c60 0x7f20c40cf940 secure :-1 s=READY pgs=3060 cs=0 l=1 rev1=1 rx=0x7f20bc006b80 tx=0x7f20bc009b90).ready entity=mon.0 client_cookie=4ac0d155c6dc967d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:37.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:37.720 7f20c22bd700 1 -- 172.21.2.112:0/3489832592 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f20bc0022e0 con 0x7f20c40d1c60 2021-02-10T11:47:37.740 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:37.720 7f20c22bd700 1 -- 172.21.2.112:0/3489832592 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20bc002620 con 0x7f20c40d1c60 2021-02-10T11:47:37.740 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:37.720 7f20c22bd700 1 -- 172.21.2.112:0/3489832592 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f20bc002b00 con 0x7f20c40d1c60 2021-02-10T11:47:37.740 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:37.720 7f20c9600700 1 -- 172.21.2.112:0/3489832592 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f20c40d1c60 msgr2=0x7f20c40cf940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:37.740 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:37.720 7f20c9600700 1 --2- 172.21.2.112:0/3489832592 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f20c40d1c60 0x7f20c40cf940 secure :-1 s=READY pgs=3060 cs=0 l=1 rev1=1 rx=0x7f20bc006b80 tx=0x7f20bc009b90).stop 2021-02-10T11:47:37.740 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:37.721 7f20c9600700 1 -- 172.21.2.112:0/3489832592 shutdown_connections 2021-02-10T11:47:37.740 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:37.721 7f20c9600700 1 --2- 172.21.2.112:0/3489832592 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f20c40d1c60 0x7f20c40cf940 unknown :-1 s=CLOSED pgs=3060 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:37.740 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:37.721 7f20c9600700 1 --2- 172.21.2.112:0/3489832592 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f20c41b0a00 0x7f20c40cf320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:37.740 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:37.721 7f20c9600700 1 --2- 172.21.2.112:0/3489832592 >> v2:172.21.2.120:3300/0 conn(0x7f20c40d1860 0x7f20c41b03e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:37.740 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:37.721 7f20c9600700 1 -- 172.21.2.112:0/3489832592 >> 172.21.2.112:0/3489832592 conn(0x7f20c40ce440 msgr2=0x7f20c40d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:37.740 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:37.721 7f20c9600700 1 -- 172.21.2.112:0/3489832592 shutdown_connections 2021-02-10T11:47:37.740 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:37.721 7f20c9600700 1 -- 172.21.2.112:0/3489832592 wait complete. 2021-02-10T11:47:37.742 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:37.736 7f20c9600700 1 Processor -- start 2021-02-10T11:47:37.742 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:37.736 7f20c9600700 1 -- start start 2021-02-10T11:47:37.742 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:37.736 7f20c9600700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f20c40acbb0 0x7f20c40ac030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:37.742 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:37.736 7f20c9600700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f20c40ac650 0x7f20c40ad080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:37.742 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:37.736 7f20c9600700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f20c40ad6a0 0x7f20c40f19b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:37.742 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:37.736 7f20c9600700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20c40f1430 con 0x7f20c40ac650 2021-02-10T11:47:37.742 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:37.736 7f20c9600700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20c40adaa0 con 0x7f20c40acbb0 2021-02-10T11:47:37.743 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:37.736 7f20c9600700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20c40f1fd0 con 0x7f20c40ad6a0 2021-02-10T11:47:37.743 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:37.736 7f20c2abe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f20c40ac650 0x7f20c40ad080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:37.743 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:37.736 7f20c2abe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f20c40ac650 0x7f20c40ad080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:60566/0 (socket says 172.21.2.112:60566) 2021-02-10T11:47:37.743 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:37.737 7f20c2abe700 1 -- 172.21.2.112:0/3234998209 learned_addr learned my addr 172.21.2.112:0/3234998209 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:47:37.743 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:37.737 7f20c32bf700 1 --2- 172.21.2.112:0/3234998209 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f20c40acbb0 0x7f20c40ac030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:37.743 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:37.737 7f20c3ac0700 1 --2- 172.21.2.112:0/3234998209 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f20c40ad6a0 0x7f20c40f19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:37.743 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:37.737 7f20c2abe700 1 -- 172.21.2.112:0/3234998209 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f20c40ad6a0 msgr2=0x7f20c40f19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:37.743 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:37.737 7f20c2abe700 1 --2- 172.21.2.112:0/3234998209 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f20c40ad6a0 0x7f20c40f19b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:37.743 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:37.737 7f20c2abe700 1 -- 172.21.2.112:0/3234998209 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f20c40acbb0 msgr2=0x7f20c40ac030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:37.743 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:37.737 7f20c2abe700 1 --2- 172.21.2.112:0/3234998209 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f20c40acbb0 0x7f20c40ac030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:37.743 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:37.737 7f20c2abe700 1 -- 172.21.2.112:0/3234998209 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20c40f21f0 con 0x7f20c40ac650 2021-02-10T11:47:37.743 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:37.737 7f20c3ac0700 1 --2- 172.21.2.112:0/3234998209 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f20c40ad6a0 0x7f20c40f19b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:47:37.744 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:37.737 7f20c2abe700 1 --2- 172.21.2.112:0/3234998209 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f20c40ac650 0x7f20c40ad080 secure :-1 s=READY pgs=3061 cs=0 l=1 rev1=1 rx=0x7f20b4006f00 tx=0x7f20b400b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:37.744 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:37.737 7f20bbfff700 1 -- 172.21.2.112:0/3234998209 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f20b4006b90 con 0x7f20c40ac650 2021-02-10T11:47:37.744 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:37.737 7f20bbfff700 1 -- 172.21.2.112:0/3234998209 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20b400bd30 con 0x7f20c40ac650 2021-02-10T11:47:37.744 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:37.737 7f20bbfff700 1 -- 172.21.2.112:0/3234998209 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f20b40020d0 con 0x7f20c40ac650 2021-02-10T11:47:37.744 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:37.737 7f20c9600700 1 -- 172.21.2.112:0/3234998209 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20c40f21f0 con 0x7f20c40ac650 2021-02-10T11:47:37.744 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:37.738 7f20c9600700 1 -- 172.21.2.112:0/3234998209 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20c40f2780 con 0x7f20c40ac650 2021-02-10T11:47:37.744 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:37.738 7f20bbfff700 1 -- 172.21.2.112:0/3234998209 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f20b4005310 con 0x7f20c40ac650 2021-02-10T11:47:37.744 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:37.738 7f20bbfff700 1 --2- 172.21.2.112:0/3234998209 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f209c0186c0 0x7f209c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:37.744 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:37.738 7f20bbfff700 1 -- 172.21.2.112:0/3234998209 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f20b4032680 con 0x7f20c40ac650 2021-02-10T11:47:37.744 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:37.739 7f20c32bf700 1 --2- 172.21.2.112:0/3234998209 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f209c0186c0 0x7f209c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:37.744 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:37.739 7f20c32bf700 1 --2- 172.21.2.112:0/3234998209 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f209c0186c0 0x7f209c01abc0 crc :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:37.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:37.769 7f20c9600700 1 -- 172.21.2.112:0/3234998209 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20c40ee060 con 0x7f20c40ac650 2021-02-10T11:47:37.774 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:37.771 7f20bbfff700 1 -- 172.21.2.112:0/3234998209 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f20b4006b90 con 0x7f20c40ac650 2021-02-10T11:47:37.907 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:37.904 7f20c9600700 1 -- 172.21.2.112:0/3234998209 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs volume ls", "target": ["mgr", ""]}) v1 -- 0x7f20c40f2780 con 0x7f209c0186c0 2021-02-10T11:47:37.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:37.905 7f20bbfff700 1 -- 172.21.2.112:0/3234998209 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+41 (crc 0 0 0) 0x7f20ac006970 con 0x7f209c0186c0 2021-02-10T11:47:37.910 INFO:teuthology.orchestra.run.gibba012.stdout:[ 2021-02-10T11:47:37.910 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:37.910 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "cephfs" 2021-02-10T11:47:37.910 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:47:37.911 INFO:teuthology.orchestra.run.gibba012.stdout:] 2021-02-10T11:47:37.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:37.908 7f20c9600700 1 -- 172.21.2.112:0/3234998209 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f209c0186c0 msgr2=0x7f209c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:37.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:37.908 7f20c9600700 1 --2- 172.21.2.112:0/3234998209 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f209c0186c0 0x7f209c01abc0 crc :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:37.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:37.908 7f20c9600700 1 -- 172.21.2.112:0/3234998209 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f20c40ac650 msgr2=0x7f20c40ad080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:37.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:37.908 7f20c9600700 1 --2- 172.21.2.112:0/3234998209 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f20c40ac650 0x7f20c40ad080 secure :-1 s=READY pgs=3061 cs=0 l=1 rev1=1 rx=0x7f20b4006f00 tx=0x7f20b400b940).stop 2021-02-10T11:47:37.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:37.909 7f20c9600700 1 -- 172.21.2.112:0/3234998209 shutdown_connections 2021-02-10T11:47:37.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:37.909 7f20c9600700 1 --2- 172.21.2.112:0/3234998209 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f209c0186c0 0x7f209c01abc0 unknown :-1 s=CLOSED pgs=1377 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:37.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:37.909 7f20c9600700 1 --2- 172.21.2.112:0/3234998209 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f20c40ad6a0 0x7f20c40f19b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:37.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:37.909 7f20c9600700 1 --2- 172.21.2.112:0/3234998209 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f20c40ac650 0x7f20c40ad080 unknown :-1 s=CLOSED pgs=3061 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:37.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:37.909 7f20c9600700 1 --2- 172.21.2.112:0/3234998209 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f20c40acbb0 0x7f20c40ac030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:37.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:37.909 7f20c9600700 1 -- 172.21.2.112:0/3234998209 >> 172.21.2.112:0/3234998209 conn(0x7f20c40bea70 msgr2=0x7f20c4095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:37.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:37.909 7f20c9600700 1 -- 172.21.2.112:0/3234998209 shutdown_connections 2021-02-10T11:47:37.913 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:37.909 7f20c9600700 1 -- 172.21.2.112:0/3234998209 wait complete. 2021-02-10T11:47:37.920 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:47:38.354 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.333 7fce70e01700 1 Processor -- start 2021-02-10T11:47:38.354 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.333 7fce70e01700 1 -- start start 2021-02-10T11:47:38.354 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.334 7fce70e01700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fce6c0d1c70 0x7fce6c0cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:38.354 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.334 7fce70e01700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce6c0d0170 con 0x7fce6c0cf580 2021-02-10T11:47:38.354 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.334 7fce70e01700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce6c0fe1c0 con 0x7fce6c0d1870 2021-02-10T11:47:38.354 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.334 7fce70e01700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce6c1b0cb0 con 0x7fce6c0d1c70 2021-02-10T11:47:38.354 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.334 7fce6b314700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fce6c0cf580 0x7fce6c0cf980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:60036/0 (socket says 172.21.2.112:60036) 2021-02-10T11:47:38.354 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.334 7fce6b314700 1 -- 172.21.2.112:0/349390583 learned_addr learned my addr 172.21.2.112:0/349390583 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:47:38.354 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.335 7fce6a312700 1 --2- 172.21.2.112:0/349390583 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fce6c0d1c70 0x7fce6c0cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:38.354 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.335 7fce69b11700 1 -- 172.21.2.112:0/349390583 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2775373303 0 0) 0x7fce64001500 con 0x7fce6c0cf580 2021-02-10T11:47:38.355 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.335 7fce69b11700 1 -- 172.21.2.112:0/349390583 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce58002fe0 con 0x7fce6c0cf580 2021-02-10T11:47:38.355 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.335 7fce6a312700 1 -- 172.21.2.112:0/349390583 >> v1:172.21.2.120:6789/0 conn(0x7fce6c0d1870 legacy=0x7fce6c1b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:38.355 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.335 7fce6a312700 1 -- 172.21.2.112:0/349390583 >> v1:172.21.2.112:6789/0 conn(0x7fce6c0cf580 legacy=0x7fce6c0cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:38.355 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.335 7fce69b11700 1 -- 172.21.2.112:0/349390583 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2063039283 0 0) 0x7fce64001500 con 0x7fce6c0cf580 2021-02-10T11:47:38.355 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.335 7fce6a312700 1 -- 172.21.2.112:0/349390583 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce6c109a60 con 0x7fce6c0d1c70 2021-02-10T11:47:38.355 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.335 7fce6a312700 1 --2- 172.21.2.112:0/349390583 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fce6c0d1c70 0x7fce6c0cef60 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0x7fce54006650 tx=0x7fce54009a00).ready entity=mon.2 client_cookie=1b315d951f28eb43 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:38.355 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.336 7fce69b11700 1 -- 172.21.2.112:0/349390583 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fce54009df0 con 0x7fce6c0d1c70 2021-02-10T11:47:38.355 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.336 7fce69b11700 1 -- 172.21.2.112:0/349390583 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce540009b0 con 0x7fce6c0d1c70 2021-02-10T11:47:38.355 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.336 7fce69b11700 1 -- 172.21.2.112:0/349390583 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fce5400c2b0 con 0x7fce6c0d1c70 2021-02-10T11:47:38.355 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.336 7fce70e01700 1 -- 172.21.2.112:0/349390583 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fce6c0d1c70 msgr2=0x7fce6c0cef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:38.355 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.336 7fce70e01700 1 --2- 172.21.2.112:0/349390583 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fce6c0d1c70 0x7fce6c0cef60 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0x7fce54006650 tx=0x7fce54009a00).stop 2021-02-10T11:47:38.355 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.337 7fce70e01700 1 -- 172.21.2.112:0/349390583 shutdown_connections 2021-02-10T11:47:38.355 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.337 7fce70e01700 1 --2- 172.21.2.112:0/349390583 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fce6c0d1c70 0x7fce6c0cef60 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:38.355 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.337 7fce70e01700 1 -- 172.21.2.112:0/349390583 >> 172.21.2.112:0/349390583 conn(0x7fce6c0ce420 msgr2=0x7fce6c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:38.356 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.337 7fce70e01700 1 -- 172.21.2.112:0/349390583 shutdown_connections 2021-02-10T11:47:38.356 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.337 7fce70e01700 1 -- 172.21.2.112:0/349390583 wait complete. 2021-02-10T11:47:38.357 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.351 7fce70e01700 1 Processor -- start 2021-02-10T11:47:38.357 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.351 7fce70e01700 1 -- start start 2021-02-10T11:47:38.357 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.352 7fce70e01700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fce6c0c2370 0x7fce6c18f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:38.357 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.352 7fce70e01700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fce6c084cb0 0x7fce6c1a7470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:38.357 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.352 7fce70e01700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fce6c1a7a90 0x7fce6c085700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:38.357 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.352 7fce70e01700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce6c0be840 con 0x7fce6c0c2370 2021-02-10T11:47:38.358 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.352 7fce70e01700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce6c0c29a0 con 0x7fce6c1a7a90 2021-02-10T11:47:38.358 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.352 7fce70e01700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce6c1a7e90 con 0x7fce6c084cb0 2021-02-10T11:47:38.358 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.352 7fce6ab13700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fce6c0c2370 0x7fce6c18f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:38.358 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.352 7fce6ab13700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fce6c0c2370 0x7fce6c18f870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:60576/0 (socket says 172.21.2.112:60576) 2021-02-10T11:47:38.358 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.352 7fce6ab13700 1 -- 172.21.2.112:0/1625136458 learned_addr learned my addr 172.21.2.112:0/1625136458 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:47:38.358 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.352 7fce6ab13700 1 -- 172.21.2.112:0/1625136458 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fce6c084cb0 msgr2=0x7fce6c1a7470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:38.358 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.352 7fce6ab13700 1 --2- 172.21.2.112:0/1625136458 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fce6c084cb0 0x7fce6c1a7470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:38.358 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.352 7fce6ab13700 1 -- 172.21.2.112:0/1625136458 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fce6c1a7a90 msgr2=0x7fce6c085700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:38.358 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.352 7fce6ab13700 1 --2- 172.21.2.112:0/1625136458 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fce6c1a7a90 0x7fce6c085700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:38.358 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.352 7fce6ab13700 1 -- 172.21.2.112:0/1625136458 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce6c1a80b0 con 0x7fce6c0c2370 2021-02-10T11:47:38.358 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.353 7fce6ab13700 1 --2- 172.21.2.112:0/1625136458 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fce6c0c2370 0x7fce6c18f870 secure :-1 s=READY pgs=3063 cs=0 l=1 rev1=1 rx=0x7fce5c00b530 tx=0x7fce5c00b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:38.358 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.353 7fce637fe700 1 -- 172.21.2.112:0/1625136458 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fce5c00bcc0 con 0x7fce6c0c2370 2021-02-10T11:47:38.358 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.353 7fce637fe700 1 -- 172.21.2.112:0/1625136458 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce5c006020 con 0x7fce6c0c2370 2021-02-10T11:47:38.358 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.353 7fce637fe700 1 -- 172.21.2.112:0/1625136458 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fce5c0054a0 con 0x7fce6c0c2370 2021-02-10T11:47:38.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.353 7fce70e01700 1 -- 172.21.2.112:0/1625136458 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce6c1a80b0 con 0x7fce6c0c2370 2021-02-10T11:47:38.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.353 7fce70e01700 1 -- 172.21.2.112:0/1625136458 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce6c086010 con 0x7fce6c0c2370 2021-02-10T11:47:38.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.353 7fce637fe700 1 -- 172.21.2.112:0/1625136458 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fce5c006200 con 0x7fce6c0c2370 2021-02-10T11:47:38.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.354 7fce637fe700 1 --2- 172.21.2.112:0/1625136458 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fce500186c0 0x7fce5001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:38.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.354 7fce637fe700 1 -- 172.21.2.112:0/1625136458 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fce5c0325e0 con 0x7fce6c0c2370 2021-02-10T11:47:38.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.354 7fce6a312700 1 --2- 172.21.2.112:0/1625136458 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fce500186c0 0x7fce5001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:38.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.354 7fce6a312700 1 --2- 172.21.2.112:0/1625136458 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fce500186c0 0x7fce5001abc0 crc :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:38.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.384 7fce70e01700 1 -- 172.21.2.112:0/1625136458 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce6c082a70 con 0x7fce6c0c2370 2021-02-10T11:47:38.389 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.387 7fce637fe700 1 -- 172.21.2.112:0/1625136458 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fce5c030d90 con 0x7fce6c0c2370 2021-02-10T11:47:38.540 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.537 7fce70e01700 1 -- 172.21.2.112:0/1625136458 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fce6c082a70 con 0x7fce6c0c2370 2021-02-10T11:47:38.541 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.538 7fce637fe700 1 -- 172.21.2.112:0/1625136458 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v44)=0 v44) v1 ==== 122+0+0 (secure 0 0 0) 0x7fce5c0061e0 con 0x7fce6c0c2370 2021-02-10T11:47:38.543 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.539 7fce70e01700 1 -- 172.21.2.112:0/1625136458 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fce500186c0 msgr2=0x7fce5001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:38.543 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.539 7fce70e01700 1 --2- 172.21.2.112:0/1625136458 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fce500186c0 0x7fce5001abc0 crc :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:38.543 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.539 7fce70e01700 1 -- 172.21.2.112:0/1625136458 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fce6c0c2370 msgr2=0x7fce6c18f870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:38.543 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.539 7fce70e01700 1 --2- 172.21.2.112:0/1625136458 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fce6c0c2370 0x7fce6c18f870 secure :-1 s=READY pgs=3063 cs=0 l=1 rev1=1 rx=0x7fce5c00b530 tx=0x7fce5c00b8d0).stop 2021-02-10T11:47:38.543 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.540 7fce70e01700 1 -- 172.21.2.112:0/1625136458 shutdown_connections 2021-02-10T11:47:38.543 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.540 7fce70e01700 1 --2- 172.21.2.112:0/1625136458 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fce500186c0 0x7fce5001abc0 unknown :-1 s=CLOSED pgs=1378 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:38.544 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.540 7fce70e01700 1 --2- 172.21.2.112:0/1625136458 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fce6c1a7a90 0x7fce6c085700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:38.544 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.540 7fce70e01700 1 --2- 172.21.2.112:0/1625136458 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fce6c084cb0 0x7fce6c1a7470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:38.544 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.540 7fce70e01700 1 --2- 172.21.2.112:0/1625136458 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fce6c0c2370 0x7fce6c18f870 unknown :-1 s=CLOSED pgs=3063 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:38.544 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.540 7fce70e01700 1 -- 172.21.2.112:0/1625136458 >> 172.21.2.112:0/1625136458 conn(0x7fce6c0b5dc0 msgr2=0x7fce6c0c2d90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:38.544 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.540 7fce70e01700 1 -- 172.21.2.112:0/1625136458 shutdown_connections 2021-02-10T11:47:38.544 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.540 7fce70e01700 1 -- 172.21.2.112:0/1625136458 wait complete. 2021-02-10T11:47:38.551 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolumegroup create cephfs group_0000000000232095 2021-02-10T11:47:38.983 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.964 7fa637c4b700 1 Processor -- start 2021-02-10T11:47:38.983 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.964 7fa637c4b700 1 -- start start 2021-02-10T11:47:38.983 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.964 7fa637c4b700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fa6300d1c70 0x7fa6300cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:38.983 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.964 7fa637c4b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa6300cf580 0x7fa6300cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:38.984 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.964 7fa637c4b700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6300fe1c0 con 0x7fa6300d1870 2021-02-10T11:47:38.984 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.964 7fa637c4b700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6301b0cb0 con 0x7fa6300d1c70 2021-02-10T11:47:38.984 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.964 7fa637c4b700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa630109a60 con 0x7fa6300cf580 2021-02-10T11:47:38.984 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.964 7fa635a67700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fa6300d1870 0x7fa6301b04c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:60044/0 (socket says 172.21.2.112:60044) 2021-02-10T11:47:38.984 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.964 7fa635a67700 1 -- 172.21.2.112:0/3660913163 learned_addr learned my addr 172.21.2.112:0/3660913163 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:47:38.984 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.965 7fa635266700 1 --2- 172.21.2.112:0/3660913163 >> v2:172.21.2.120:3300/0 conn(0x7fa6300d1c70 0x7fa6300cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:38.984 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.965 7fa636268700 1 --2- 172.21.2.112:0/3660913163 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa6300cf580 0x7fa6300cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:38.984 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.965 7fa634a65700 1 -- 172.21.2.112:0/3660913163 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1374135649 0 0) 0x7fa62c001180 con 0x7fa6300d1870 2021-02-10T11:47:38.984 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.965 7fa634a65700 1 -- 172.21.2.112:0/3660913163 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa61c002fe0 con 0x7fa6300d1870 2021-02-10T11:47:38.984 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.965 7fa636268700 1 -- 172.21.2.112:0/3660913163 >> v2:172.21.2.120:3300/0 conn(0x7fa6300d1c70 msgr2=0x7fa6300cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:38.984 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.965 7fa636268700 1 --2- 172.21.2.112:0/3660913163 >> v2:172.21.2.120:3300/0 conn(0x7fa6300d1c70 0x7fa6300cef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:38.985 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.965 7fa636268700 1 -- 172.21.2.112:0/3660913163 >> v1:172.21.2.112:6789/0 conn(0x7fa6300d1870 legacy=0x7fa6301b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:38.985 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.965 7fa636268700 1 -- 172.21.2.112:0/3660913163 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa630088070 con 0x7fa6300cf580 2021-02-10T11:47:38.985 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.965 7fa636268700 1 --2- 172.21.2.112:0/3660913163 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa6300cf580 0x7fa6300cf980 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7fa628006e50 tx=0x7fa628009e70).ready entity=mon.2 client_cookie=90c56dd5e048a1e7 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:38.985 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.965 7fa634a65700 1 -- 172.21.2.112:0/3660913163 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa62800b510 con 0x7fa6300cf580 2021-02-10T11:47:38.985 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.965 7fa634a65700 1 -- 172.21.2.112:0/3660913163 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa628006790 con 0x7fa6300cf580 2021-02-10T11:47:38.985 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.965 7fa634a65700 1 -- 172.21.2.112:0/3660913163 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa62800b940 con 0x7fa6300cf580 2021-02-10T11:47:38.985 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.965 7fa637c4b700 1 -- 172.21.2.112:0/3660913163 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa6300cf580 msgr2=0x7fa6300cf980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:38.985 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.965 7fa637c4b700 1 --2- 172.21.2.112:0/3660913163 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa6300cf580 0x7fa6300cf980 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7fa628006e50 tx=0x7fa628009e70).stop 2021-02-10T11:47:38.985 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.966 7fa637c4b700 1 -- 172.21.2.112:0/3660913163 shutdown_connections 2021-02-10T11:47:38.985 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.966 7fa637c4b700 1 --2- 172.21.2.112:0/3660913163 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa6300cf580 0x7fa6300cf980 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:38.985 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.966 7fa637c4b700 1 --2- 172.21.2.112:0/3660913163 >> v2:172.21.2.120:3300/0 conn(0x7fa6300d1c70 0x7fa6300cef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:38.985 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.966 7fa637c4b700 1 -- 172.21.2.112:0/3660913163 >> 172.21.2.112:0/3660913163 conn(0x7fa6300ce420 msgr2=0x7fa6300d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:38.985 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.966 7fa637c4b700 1 -- 172.21.2.112:0/3660913163 shutdown_connections 2021-02-10T11:47:38.986 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.966 7fa637c4b700 1 -- 172.21.2.112:0/3660913163 wait complete. 2021-02-10T11:47:38.987 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.981 7fa637c4b700 1 Processor -- start 2021-02-10T11:47:38.987 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.981 7fa637c4b700 1 -- start start 2021-02-10T11:47:38.987 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.981 7fa637c4b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa63009bcc0 0x7fa63009c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:38.987 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.981 7fa637c4b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa6300a0820 0x7fa6300a0c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:38.988 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.981 7fa637c4b700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa6300a1240 0x7fa6300858d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:38.988 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.981 7fa637c4b700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6300a66b0 con 0x7fa6300a0820 2021-02-10T11:47:38.988 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.981 7fa637c4b700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6301a49b0 con 0x7fa6300a1240 2021-02-10T11:47:38.988 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.981 7fa637c4b700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6301a4b90 con 0x7fa63009bcc0 2021-02-10T11:47:38.988 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.981 7fa635266700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa6300a0820 0x7fa6300a0c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:38.988 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.981 7fa635266700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa6300a0820 0x7fa6300a0c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:60588/0 (socket says 172.21.2.112:60588) 2021-02-10T11:47:38.988 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.981 7fa635266700 1 -- 172.21.2.112:0/797872992 learned_addr learned my addr 172.21.2.112:0/797872992 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:47:38.988 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.981 7fa635a67700 1 --2- 172.21.2.112:0/797872992 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa63009bcc0 0x7fa63009c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:38.988 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.981 7fa636268700 1 --2- 172.21.2.112:0/797872992 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa6300a1240 0x7fa6300858d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:38.988 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.982 7fa635266700 1 -- 172.21.2.112:0/797872992 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa63009bcc0 msgr2=0x7fa63009c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:38.988 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.982 7fa635266700 1 --2- 172.21.2.112:0/797872992 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa63009bcc0 0x7fa63009c0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:38.988 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.982 7fa635266700 1 -- 172.21.2.112:0/797872992 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa6300a1240 msgr2=0x7fa6300858d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:38.988 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.982 7fa635266700 1 --2- 172.21.2.112:0/797872992 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa6300a1240 0x7fa6300858d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:38.989 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.982 7fa635266700 1 -- 172.21.2.112:0/797872992 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6300a1640 con 0x7fa6300a0820 2021-02-10T11:47:38.989 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.982 7fa636268700 1 --2- 172.21.2.112:0/797872992 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa6300a1240 0x7fa6300858d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:47:38.989 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.982 7fa635266700 1 --2- 172.21.2.112:0/797872992 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa6300a0820 0x7fa6300a0c20 secure :-1 s=READY pgs=3065 cs=0 l=1 rev1=1 rx=0x7fa62400b560 tx=0x7fa62400b900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:38.989 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.982 7fa6227fc700 1 -- 172.21.2.112:0/797872992 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa6240009b0 con 0x7fa6300a0820 2021-02-10T11:47:38.989 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.982 7fa6227fc700 1 -- 172.21.2.112:0/797872992 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa62400bcf0 con 0x7fa6300a0820 2021-02-10T11:47:38.989 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.982 7fa6227fc700 1 -- 172.21.2.112:0/797872992 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa624002030 con 0x7fa6300a0820 2021-02-10T11:47:38.989 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.982 7fa637c4b700 1 -- 172.21.2.112:0/797872992 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6300a1640 con 0x7fa6300a0820 2021-02-10T11:47:38.989 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.982 7fa637c4b700 1 -- 172.21.2.112:0/797872992 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6300a1640 con 0x7fa6300a0820 2021-02-10T11:47:38.989 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.983 7fa6227fc700 1 -- 172.21.2.112:0/797872992 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa6240052e0 con 0x7fa6300a0820 2021-02-10T11:47:38.989 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.983 7fa6227fc700 1 --2- 172.21.2.112:0/797872992 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa618018730 0x7fa61801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:38.989 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.983 7fa6227fc700 1 -- 172.21.2.112:0/797872992 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fa624032660 con 0x7fa6300a0820 2021-02-10T11:47:38.989 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.983 7fa635a67700 1 --2- 172.21.2.112:0/797872992 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa618018730 0x7fa61801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:38.989 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:38.983 7fa635a67700 1 --2- 172.21.2.112:0/797872992 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa618018730 0x7fa61801ac10 crc :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:39.017 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:39.014 7fa637c4b700 1 -- 172.21.2.112:0/797872992 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6300fe1c0 con 0x7fa6300a0820 2021-02-10T11:47:39.019 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:39.017 7fa6227fc700 1 -- 172.21.2.112:0/797872992 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa624006b00 con 0x7fa6300a0820 2021-02-10T11:47:39.147 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:39.144 7fa637c4b700 1 -- 172.21.2.112:0/797872992 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "target": ["mgr", ""], "group_name": "group_0000000000232095"}) v1 -- 0x7fa6300fe1c0 con 0x7fa618018730 2021-02-10T11:47:39.150 INFO:tasks.ceph.mgr.x.gibba020.stderr:2021-02-10 11:47:39.147 7f6382eee700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:47:39.150 INFO:tasks.ceph.mgr.x.gibba020.stderr:2021-02-10 11:47:39.147 7f6382eee700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:47:39.150 INFO:tasks.ceph.mgr.x.gibba020.stderr:2021-02-10 11:47:39.147 7f6382eee700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:47:39.151 INFO:tasks.ceph.mgr.x.gibba020.stderr:2021-02-10 11:47:39.147 7f6382eee700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:47:39.151 INFO:tasks.ceph.mgr.x.gibba020.stderr:2021-02-10 11:47:39.147 7f6382eee700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:47:39.164 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:39.161 7fa6227fc700 1 -- 172.21.2.112:0/797872992 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fa62c000ce0 con 0x7fa618018730 2021-02-10T11:47:39.180 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:39.178 7fa637c4b700 1 -- 172.21.2.112:0/797872992 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa618018730 msgr2=0x7fa61801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:39.181 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:39.178 7fa637c4b700 1 --2- 172.21.2.112:0/797872992 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa618018730 0x7fa61801ac10 crc :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:39.181 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:39.178 7fa637c4b700 1 -- 172.21.2.112:0/797872992 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa6300a0820 msgr2=0x7fa6300a0c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:39.181 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:39.178 7fa637c4b700 1 --2- 172.21.2.112:0/797872992 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa6300a0820 0x7fa6300a0c20 secure :-1 s=READY pgs=3065 cs=0 l=1 rev1=1 rx=0x7fa62400b560 tx=0x7fa62400b900).stop 2021-02-10T11:47:39.181 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:39.178 7fa637c4b700 1 -- 172.21.2.112:0/797872992 shutdown_connections 2021-02-10T11:47:39.181 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:39.178 7fa637c4b700 1 --2- 172.21.2.112:0/797872992 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa618018730 0x7fa61801ac10 unknown :-1 s=CLOSED pgs=1379 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:39.181 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:39.178 7fa637c4b700 1 --2- 172.21.2.112:0/797872992 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa6300a1240 0x7fa6300858d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:39.181 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:39.178 7fa637c4b700 1 --2- 172.21.2.112:0/797872992 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa6300a0820 0x7fa6300a0c20 unknown :-1 s=CLOSED pgs=3065 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:39.182 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:39.178 7fa637c4b700 1 --2- 172.21.2.112:0/797872992 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa63009bcc0 0x7fa63009c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:39.182 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:39.178 7fa637c4b700 1 -- 172.21.2.112:0/797872992 >> 172.21.2.112:0/797872992 conn(0x7fa6300bef00 msgr2=0x7fa63009d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:39.182 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:39.178 7fa637c4b700 1 -- 172.21.2.112:0/797872992 shutdown_connections 2021-02-10T11:47:39.182 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:39.178 7fa637c4b700 1 -- 172.21.2.112:0/797872992 wait complete. 2021-02-10T11:47:39.189 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000120407 --group_name group_0000000000232095 2021-02-10T11:47:39.621 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:39.602 7fccdb3fb700 1 Processor -- start 2021-02-10T11:47:39.621 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:39.602 7fccdb3fb700 1 -- start start 2021-02-10T11:47:39.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:39.603 7fccdb3fb700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fccd40d1880 0x7fccd41b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:39.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:39.603 7fccdb3fb700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fccd41b0af0 0x7fccd4108950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:39.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:39.603 7fccdb3fb700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fccd40d1c80 0x7fccd4108f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:39.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:39.603 7fccdb3fb700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccd40e5f00 con 0x7fccd41b0af0 2021-02-10T11:47:39.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:39.603 7fccdb3fb700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccd4109a70 con 0x7fccd40d1c80 2021-02-10T11:47:39.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:39.603 7fccdb3fb700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccd4088070 con 0x7fccd40d1880 2021-02-10T11:47:39.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:39.603 7fccd8a16700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fccd41b0af0 0x7fccd4108950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:39.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:39.603 7fccd8a16700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fccd41b0af0 0x7fccd4108950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:60596/0 (socket says 172.21.2.112:60596) 2021-02-10T11:47:39.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:39.603 7fccd8a16700 1 -- 172.21.2.112:0/2987412231 learned_addr learned my addr 172.21.2.112:0/2987412231 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:47:39.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:39.603 7fccd9217700 1 --2- 172.21.2.112:0/2987412231 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fccd40d1880 0x7fccd41b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:39.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:39.603 7fccd9a18700 1 --2- 172.21.2.112:0/2987412231 >> v2:172.21.2.120:3300/0 conn(0x7fccd40d1c80 0x7fccd4108f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:39.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:39.603 7fccd8a16700 1 -- 172.21.2.112:0/2987412231 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fccd40d1880 msgr2=0x7fccd41b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:39.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:39.603 7fccd8a16700 1 --2- 172.21.2.112:0/2987412231 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fccd40d1880 0x7fccd41b04d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:39.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:39.603 7fccd8a16700 1 -- 172.21.2.112:0/2987412231 >> v2:172.21.2.120:3300/0 conn(0x7fccd40d1c80 msgr2=0x7fccd4108f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:39.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:39.603 7fccd8a16700 1 --2- 172.21.2.112:0/2987412231 >> v2:172.21.2.120:3300/0 conn(0x7fccd40d1c80 0x7fccd4108f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:39.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:39.603 7fccd8a16700 1 -- 172.21.2.112:0/2987412231 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccd4109590 con 0x7fccd41b0af0 2021-02-10T11:47:39.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:39.603 7fccd8a16700 1 --2- 172.21.2.112:0/2987412231 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fccd41b0af0 0x7fccd4108950 secure :-1 s=READY pgs=3066 cs=0 l=1 rev1=1 rx=0x7fccc8006700 tx=0x7fccc8009b80).ready entity=mon.0 client_cookie=a38e9bb9c0eba514 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:39.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:39.603 7fccc7fff700 1 -- 172.21.2.112:0/2987412231 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fccc800c0c0 con 0x7fccd41b0af0 2021-02-10T11:47:39.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:39.603 7fccc7fff700 1 -- 172.21.2.112:0/2987412231 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccc800c340 con 0x7fccd41b0af0 2021-02-10T11:47:39.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:39.603 7fccc7fff700 1 -- 172.21.2.112:0/2987412231 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fccc800c860 con 0x7fccd41b0af0 2021-02-10T11:47:39.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:39.604 7fccdb3fb700 1 -- 172.21.2.112:0/2987412231 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fccd41b0af0 msgr2=0x7fccd4108950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:39.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:39.604 7fccdb3fb700 1 --2- 172.21.2.112:0/2987412231 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fccd41b0af0 0x7fccd4108950 secure :-1 s=READY pgs=3066 cs=0 l=1 rev1=1 rx=0x7fccc8006700 tx=0x7fccc8009b80).stop 2021-02-10T11:47:39.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:39.604 7fccdb3fb700 1 -- 172.21.2.112:0/2987412231 shutdown_connections 2021-02-10T11:47:39.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:39.604 7fccdb3fb700 1 --2- 172.21.2.112:0/2987412231 >> v2:172.21.2.120:3300/0 conn(0x7fccd40d1c80 0x7fccd4108f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:39.624 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:39.604 7fccdb3fb700 1 --2- 172.21.2.112:0/2987412231 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fccd41b0af0 0x7fccd4108950 unknown :-1 s=CLOSED pgs=3066 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:39.624 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:39.604 7fccdb3fb700 1 --2- 172.21.2.112:0/2987412231 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fccd40d1880 0x7fccd41b04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:39.624 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:39.604 7fccdb3fb700 1 -- 172.21.2.112:0/2987412231 >> 172.21.2.112:0/2987412231 conn(0x7fccd40ce430 msgr2=0x7fccd40d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:39.624 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:39.604 7fccdb3fb700 1 -- 172.21.2.112:0/2987412231 shutdown_connections 2021-02-10T11:47:39.624 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:39.604 7fccdb3fb700 1 -- 172.21.2.112:0/2987412231 wait complete. 2021-02-10T11:47:39.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:39.619 7fccdb3fb700 1 Processor -- start 2021-02-10T11:47:39.625 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:39.619 7fccdb3fb700 1 -- start start 2021-02-10T11:47:39.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:39.619 7fccdb3fb700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fccd409ab20 0x7fccd409af20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:39.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:39.619 7fccdb3fb700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fccd40c0d10 0x7fccd40c1110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:39.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:39.619 7fccdb3fb700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fccd40c1730 0x7fccd409bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:39.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:39.619 7fccdb3fb700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccd409b540 con 0x7fccd40c0d10 2021-02-10T11:47:39.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:39.619 7fccdb3fb700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccd40c1b30 con 0x7fccd40c1730 2021-02-10T11:47:39.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:39.619 7fccdb3fb700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccd40c1d50 con 0x7fccd409ab20 2021-02-10T11:47:39.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:39.619 7fccd8a16700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fccd40c0d10 0x7fccd40c1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:39.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:39.620 7fccd8a16700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fccd40c0d10 0x7fccd40c1110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:60600/0 (socket says 172.21.2.112:60600) 2021-02-10T11:47:39.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:39.620 7fccd8a16700 1 -- 172.21.2.112:0/669406608 learned_addr learned my addr 172.21.2.112:0/669406608 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:47:39.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:39.620 7fccd9a18700 1 --2- 172.21.2.112:0/669406608 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fccd40c1730 0x7fccd409bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:39.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:39.620 7fccd9217700 1 --2- 172.21.2.112:0/669406608 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fccd409ab20 0x7fccd409af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:39.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:39.620 7fccd8a16700 1 -- 172.21.2.112:0/669406608 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fccd409ab20 msgr2=0x7fccd409af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:39.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:39.620 7fccd8a16700 1 --2- 172.21.2.112:0/669406608 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fccd409ab20 0x7fccd409af20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:39.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:39.620 7fccd8a16700 1 -- 172.21.2.112:0/669406608 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fccd40c1730 msgr2=0x7fccd409bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:39.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:39.620 7fccd8a16700 1 --2- 172.21.2.112:0/669406608 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fccd40c1730 0x7fccd409bde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:39.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:39.620 7fccd8a16700 1 -- 172.21.2.112:0/669406608 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccd409c400 con 0x7fccd40c0d10 2021-02-10T11:47:39.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:39.620 7fccd8a16700 1 --2- 172.21.2.112:0/669406608 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fccd40c0d10 0x7fccd40c1110 secure :-1 s=READY pgs=3067 cs=0 l=1 rev1=1 rx=0x7fccc800b5e0 tx=0x7fccc800b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:39.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:39.620 7fccc5ffb700 1 -- 172.21.2.112:0/669406608 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fccc800bd70 con 0x7fccd40c0d10 2021-02-10T11:47:39.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:39.620 7fccc5ffb700 1 -- 172.21.2.112:0/669406608 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccc8002030 con 0x7fccd40c0d10 2021-02-10T11:47:39.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:39.620 7fccc5ffb700 1 -- 172.21.2.112:0/669406608 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fccc80024d0 con 0x7fccd40c0d10 2021-02-10T11:47:39.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:39.620 7fccdb3fb700 1 -- 172.21.2.112:0/669406608 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccd409c400 con 0x7fccd40c0d10 2021-02-10T11:47:39.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:39.620 7fccdb3fb700 1 -- 172.21.2.112:0/669406608 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccd409c400 con 0x7fccd40c0d10 2021-02-10T11:47:39.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:39.621 7fccc5ffb700 1 -- 172.21.2.112:0/669406608 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fccc8000c90 con 0x7fccd40c0d10 2021-02-10T11:47:39.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:39.621 7fccc5ffb700 1 --2- 172.21.2.112:0/669406608 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fccb40186c0 0x7fccb401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:39.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:39.621 7fccc5ffb700 1 -- 172.21.2.112:0/669406608 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fccc8032580 con 0x7fccd40c0d10 2021-02-10T11:47:39.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:39.621 7fccd9217700 1 --2- 172.21.2.112:0/669406608 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fccb40186c0 0x7fccb401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:39.628 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:39.622 7fccd9217700 1 --2- 172.21.2.112:0/669406608 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fccb40186c0 0x7fccb401abc0 crc :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:39.655 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:39.652 7fccdb3fb700 1 -- 172.21.2.112:0/669406608 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccd4082a50 con 0x7fccd40c0d10 2021-02-10T11:47:39.658 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:39.655 7fccc5ffb700 1 -- 172.21.2.112:0/669406608 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fccc8030d30 con 0x7fccd40c0d10 2021-02-10T11:47:39.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:39.780 7fccdb3fb700 1 -- 172.21.2.112:0/669406608 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000120407", "target": ["mgr", ""], "group_name": "group_0000000000232095"}) v1 -- 0x7fccd4109a70 con 0x7fccb40186c0 2021-02-10T11:47:39.806 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:39.803 7fccc5ffb700 1 -- 172.21.2.112:0/669406608 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fccd00009b0 con 0x7fccb40186c0 2021-02-10T11:47:39.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:39.814 7fccdb3fb700 1 -- 172.21.2.112:0/669406608 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fccb40186c0 msgr2=0x7fccb401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:39.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:39.814 7fccdb3fb700 1 --2- 172.21.2.112:0/669406608 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fccb40186c0 0x7fccb401abc0 crc :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:39.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:39.814 7fccdb3fb700 1 -- 172.21.2.112:0/669406608 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fccd40c0d10 msgr2=0x7fccd40c1110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:39.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:39.814 7fccdb3fb700 1 --2- 172.21.2.112:0/669406608 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fccd40c0d10 0x7fccd40c1110 secure :-1 s=READY pgs=3067 cs=0 l=1 rev1=1 rx=0x7fccc800b5e0 tx=0x7fccc800b980).stop 2021-02-10T11:47:39.818 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:39.814 7fccdb3fb700 1 -- 172.21.2.112:0/669406608 shutdown_connections 2021-02-10T11:47:39.818 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:39.814 7fccdb3fb700 1 --2- 172.21.2.112:0/669406608 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fccb40186c0 0x7fccb401abc0 unknown :-1 s=CLOSED pgs=1380 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:39.818 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:39.814 7fccdb3fb700 1 --2- 172.21.2.112:0/669406608 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fccd40c1730 0x7fccd409bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:39.818 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:39.814 7fccdb3fb700 1 --2- 172.21.2.112:0/669406608 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fccd40c0d10 0x7fccd40c1110 unknown :-1 s=CLOSED pgs=3067 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:39.818 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:39.814 7fccdb3fb700 1 --2- 172.21.2.112:0/669406608 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fccd409ab20 0x7fccd409af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:39.818 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:39.814 7fccdb3fb700 1 -- 172.21.2.112:0/669406608 >> 172.21.2.112:0/669406608 conn(0x7fccd40c20f0 msgr2=0x7fccd409e410 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:39.818 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:39.815 7fccdb3fb700 1 -- 172.21.2.112:0/669406608 shutdown_connections 2021-02-10T11:47:39.819 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:39.815 7fccdb3fb700 1 -- 172.21.2.112:0/669406608 wait complete. 2021-02-10T11:47:39.825 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000120408 --group_name group_0000000000232095 2021-02-10T11:47:40.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.239 7f9afa012700 1 Processor -- start 2021-02-10T11:47:40.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.239 7f9afa012700 1 -- start start 2021-02-10T11:47:40.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.239 7f9afa012700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9af40d1c80 0x7f9af40cef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:40.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.239 7f9afa012700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9af40d0180 con 0x7f9af40d1880 2021-02-10T11:47:40.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.239 7f9afa012700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9af40fe1d0 con 0x7f9af40cf590 2021-02-10T11:47:40.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.239 7f9afa012700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9af41b0cc0 con 0x7f9af40d1c80 2021-02-10T11:47:40.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.240 7f9af37fe700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f9af40d1880 0x7f9af41b04d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:60068/0 (socket says 172.21.2.112:60068) 2021-02-10T11:47:40.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.240 7f9af37fe700 1 -- 172.21.2.112:0/1463241444 learned_addr learned my addr 172.21.2.112:0/1463241444 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:47:40.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.240 7f9af2ffd700 1 --2- 172.21.2.112:0/1463241444 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9af40d1c80 0x7f9af40cef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:40.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.240 7f9af27fc700 1 -- 172.21.2.112:0/1463241444 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4055232490 0 0) 0x7f9ae40010b0 con 0x7f9af40d1880 2021-02-10T11:47:40.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.240 7f9af27fc700 1 -- 172.21.2.112:0/1463241444 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9adc002fe0 con 0x7f9af40d1880 2021-02-10T11:47:40.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.240 7f9af27fc700 1 -- 172.21.2.112:0/1463241444 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4146347631 0 0) 0x7f9aec001400 con 0x7f9af40cf590 2021-02-10T11:47:40.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.240 7f9af27fc700 1 -- 172.21.2.112:0/1463241444 --> v1:172.21.2.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9adc003210 con 0x7f9af40cf590 2021-02-10T11:47:40.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.241 7f9af2ffd700 1 -- 172.21.2.112:0/1463241444 >> v1:172.21.2.120:6789/0 conn(0x7f9af40cf590 legacy=0x7f9af40cf990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:40.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.241 7f9af2ffd700 1 -- 172.21.2.112:0/1463241444 >> v1:172.21.2.112:6789/0 conn(0x7f9af40d1880 legacy=0x7f9af41b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:40.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.241 7f9af27fc700 1 -- 172.21.2.112:0/1463241444 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3899623676 0 0) 0x7f9ae40010b0 con 0x7f9af40d1880 2021-02-10T11:47:40.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.241 7f9af2ffd700 1 -- 172.21.2.112:0/1463241444 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9af4109a70 con 0x7f9af40d1c80 2021-02-10T11:47:40.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.241 7f9af2ffd700 1 --2- 172.21.2.112:0/1463241444 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9af40d1c80 0x7f9af40cef70 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7f9ae8006650 tx=0x7f9ae8009a00).ready entity=mon.2 client_cookie=9513aeb4aa194222 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:40.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.241 7f9af27fc700 1 -- 172.21.2.112:0/1463241444 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9ae8009df0 con 0x7f9af40d1c80 2021-02-10T11:47:40.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.241 7f9af27fc700 1 -- 172.21.2.112:0/1463241444 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ae80009b0 con 0x7f9af40d1c80 2021-02-10T11:47:40.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.241 7f9af27fc700 1 -- 172.21.2.112:0/1463241444 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9ae800c2b0 con 0x7f9af40d1c80 2021-02-10T11:47:40.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.241 7f9afa012700 1 -- 172.21.2.112:0/1463241444 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9af40d1c80 msgr2=0x7f9af40cef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:40.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.241 7f9afa012700 1 --2- 172.21.2.112:0/1463241444 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9af40d1c80 0x7f9af40cef70 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7f9ae8006650 tx=0x7f9ae8009a00).stop 2021-02-10T11:47:40.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.242 7f9afa012700 1 -- 172.21.2.112:0/1463241444 shutdown_connections 2021-02-10T11:47:40.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.242 7f9afa012700 1 --2- 172.21.2.112:0/1463241444 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9af40d1c80 0x7f9af40cef70 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:40.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.242 7f9afa012700 1 -- 172.21.2.112:0/1463241444 >> 172.21.2.112:0/1463241444 conn(0x7f9af40ce430 msgr2=0x7f9af40d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:40.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.242 7f9afa012700 1 -- 172.21.2.112:0/1463241444 shutdown_connections 2021-02-10T11:47:40.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.242 7f9afa012700 1 -- 172.21.2.112:0/1463241444 wait complete. 2021-02-10T11:47:40.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.256 7f9afa012700 1 Processor -- start 2021-02-10T11:47:40.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.257 7f9afa012700 1 -- start start 2021-02-10T11:47:40.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.257 7f9afa012700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9af4188400 0x7f9af4188800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:40.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.257 7f9afa012700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9af4188e20 0x7f9af4189220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:40.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.257 7f9afa012700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9af409cbc0 0x7f9af409cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:40.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.257 7f9afa012700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9af4187e80 con 0x7f9af4188400 2021-02-10T11:47:40.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.257 7f9afa012700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9af40b27c0 con 0x7f9af409cbc0 2021-02-10T11:47:40.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.257 7f9afa012700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9af40b29a0 con 0x7f9af4188e20 2021-02-10T11:47:40.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.257 7f9af37fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9af4188400 0x7f9af4188800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:40.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.257 7f9af37fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9af4188400 0x7f9af4188800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:60614/0 (socket says 172.21.2.112:60614) 2021-02-10T11:47:40.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.257 7f9af37fe700 1 -- 172.21.2.112:0/2644701333 learned_addr learned my addr 172.21.2.112:0/2644701333 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:47:40.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.258 7f9af37fe700 1 -- 172.21.2.112:0/2644701333 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9af4188e20 msgr2=0x7f9af4189220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:40.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.258 7f9af37fe700 1 --2- 172.21.2.112:0/2644701333 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9af4188e20 0x7f9af4189220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:40.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.258 7f9af37fe700 1 -- 172.21.2.112:0/2644701333 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9af409cbc0 msgr2=0x7f9af409cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:40.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.258 7f9af37fe700 1 --2- 172.21.2.112:0/2644701333 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9af409cbc0 0x7f9af409cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:40.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.258 7f9af37fe700 1 -- 172.21.2.112:0/2644701333 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9af409d5e0 con 0x7f9af4188400 2021-02-10T11:47:40.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.258 7f9af37fe700 1 --2- 172.21.2.112:0/2644701333 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9af4188400 0x7f9af4188800 secure :-1 s=READY pgs=3069 cs=0 l=1 rev1=1 rx=0x7f9ae4001c60 tx=0x7f9ae400a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:40.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.258 7f9adbfff700 1 -- 172.21.2.112:0/2644701333 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9ae400acc0 con 0x7f9af4188400 2021-02-10T11:47:40.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.258 7f9adbfff700 1 -- 172.21.2.112:0/2644701333 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ae4004140 con 0x7f9af4188400 2021-02-10T11:47:40.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.258 7f9adbfff700 1 -- 172.21.2.112:0/2644701333 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9ae4004660 con 0x7f9af4188400 2021-02-10T11:47:40.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.258 7f9afa012700 1 -- 172.21.2.112:0/2644701333 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9af409d5e0 con 0x7f9af4188400 2021-02-10T11:47:40.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.258 7f9afa012700 1 -- 172.21.2.112:0/2644701333 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9af409db70 con 0x7f9af4188400 2021-02-10T11:47:40.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.259 7f9adbfff700 1 -- 172.21.2.112:0/2644701333 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9ae4004320 con 0x7f9af4188400 2021-02-10T11:47:40.265 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.259 7f9adbfff700 1 --2- 172.21.2.112:0/2644701333 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9ad40186c0 0x7f9ad401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:40.265 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.259 7f9adbfff700 1 -- 172.21.2.112:0/2644701333 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f9ae4032550 con 0x7f9af4188400 2021-02-10T11:47:40.265 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.259 7f9af2ffd700 1 --2- 172.21.2.112:0/2644701333 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9ad40186c0 0x7f9ad401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:40.265 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.260 7f9af2ffd700 1 --2- 172.21.2.112:0/2644701333 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9ad40186c0 0x7f9ad401abc0 crc :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:40.292 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.289 7f9afa012700 1 -- 172.21.2.112:0/2644701333 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9af4082a70 con 0x7f9af4188400 2021-02-10T11:47:40.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.292 7f9adbfff700 1 -- 172.21.2.112:0/2644701333 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9ae4000940 con 0x7f9af4188400 2021-02-10T11:47:40.423 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.420 7f9afa012700 1 -- 172.21.2.112:0/2644701333 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000120408", "target": ["mgr", ""], "group_name": "group_0000000000232095"}) v1 -- 0x7f9af40b27c0 con 0x7f9ad40186c0 2021-02-10T11:47:40.441 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.439 7f9adbfff700 1 -- 172.21.2.112:0/2644701333 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f9ae80092d0 con 0x7f9ad40186c0 2021-02-10T11:47:40.456 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.453 7f9afa012700 1 -- 172.21.2.112:0/2644701333 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9ad40186c0 msgr2=0x7f9ad401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:40.456 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.453 7f9afa012700 1 --2- 172.21.2.112:0/2644701333 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9ad40186c0 0x7f9ad401abc0 crc :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:40.456 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.453 7f9afa012700 1 -- 172.21.2.112:0/2644701333 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9af4188400 msgr2=0x7f9af4188800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:40.456 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.453 7f9afa012700 1 --2- 172.21.2.112:0/2644701333 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9af4188400 0x7f9af4188800 secure :-1 s=READY pgs=3069 cs=0 l=1 rev1=1 rx=0x7f9ae4001c60 tx=0x7f9ae400a8d0).stop 2021-02-10T11:47:40.456 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.453 7f9afa012700 1 -- 172.21.2.112:0/2644701333 shutdown_connections 2021-02-10T11:47:40.457 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.453 7f9afa012700 1 --2- 172.21.2.112:0/2644701333 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9ad40186c0 0x7f9ad401abc0 unknown :-1 s=CLOSED pgs=1381 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:40.457 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.453 7f9afa012700 1 --2- 172.21.2.112:0/2644701333 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9af409cbc0 0x7f9af409cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:40.457 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.453 7f9afa012700 1 --2- 172.21.2.112:0/2644701333 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9af4188e20 0x7f9af4189220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:40.457 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.453 7f9afa012700 1 --2- 172.21.2.112:0/2644701333 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9af4188400 0x7f9af4188800 unknown :-1 s=CLOSED pgs=3069 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:40.457 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.453 7f9afa012700 1 -- 172.21.2.112:0/2644701333 >> 172.21.2.112:0/2644701333 conn(0x7f9af40c2cd0 msgr2=0x7f9af40b5950 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:40.457 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.454 7f9afa012700 1 -- 172.21.2.112:0/2644701333 shutdown_connections 2021-02-10T11:47:40.457 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.454 7f9afa012700 1 -- 172.21.2.112:0/2644701333 wait complete. 2021-02-10T11:47:40.465 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume authorize cephfs subvolume_0000000000120407 guest1 --group_name group_0000000000232095 --tenant_id tenant1 2021-02-10T11:47:40.906 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.886 7fd36399e700 1 Processor -- start 2021-02-10T11:47:40.906 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.886 7fd36399e700 1 -- start start 2021-02-10T11:47:40.906 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.887 7fd36399e700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd35c0e53a0 0x7fd35c1b0550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:40.907 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.887 7fd36399e700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd35c109ac0 con 0x7fd35c10b050 2021-02-10T11:47:40.907 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.887 7fd36399e700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd35c0e57a0 con 0x7fd35c1b0b70 2021-02-10T11:47:40.907 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.887 7fd36399e700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd35c0fe2b0 con 0x7fd35c0e53a0 2021-02-10T11:47:40.907 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.887 7fd361fbb700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fd35c10b050 0x7fd35c1091c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:60082/0 (socket says 172.21.2.112:60082) 2021-02-10T11:47:40.907 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.887 7fd361fbb700 1 -- 172.21.2.112:0/3030911722 learned_addr learned my addr 172.21.2.112:0/3030911722 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:47:40.907 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.887 7fd3617ba700 1 --2- 172.21.2.112:0/3030911722 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd35c0e53a0 0x7fd35c1b0550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:40.907 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.887 7fd34ffff700 1 -- 172.21.2.112:0/3030911722 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2218675210 0 0) 0x7fd354001500 con 0x7fd35c10b050 2021-02-10T11:47:40.907 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.888 7fd34ffff700 1 -- 172.21.2.112:0/3030911722 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd348002fe0 con 0x7fd35c10b050 2021-02-10T11:47:40.907 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.888 7fd3617ba700 1 -- 172.21.2.112:0/3030911722 >> v1:172.21.2.120:6789/0 conn(0x7fd35c1b0b70 legacy=0x7fd35c1089d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:40.907 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.888 7fd3617ba700 1 -- 172.21.2.112:0/3030911722 >> v1:172.21.2.112:6789/0 conn(0x7fd35c10b050 legacy=0x7fd35c1091c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:40.907 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.888 7fd3617ba700 1 -- 172.21.2.112:0/3030911722 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd35c0e5f80 con 0x7fd35c0e53a0 2021-02-10T11:47:40.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.888 7fd3617ba700 1 --2- 172.21.2.112:0/3030911722 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd35c0e53a0 0x7fd35c1b0550 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7fd350006650 tx=0x7fd350009a80).ready entity=mon.2 client_cookie=9b67a246f401a879 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:40.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.888 7fd34ffff700 1 -- 172.21.2.112:0/3030911722 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd3500009b0 con 0x7fd35c0e53a0 2021-02-10T11:47:40.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.888 7fd34ffff700 1 -- 172.21.2.112:0/3030911722 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd35000c0c0 con 0x7fd35c0e53a0 2021-02-10T11:47:40.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.888 7fd34ffff700 1 -- 172.21.2.112:0/3030911722 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd35000c520 con 0x7fd35c0e53a0 2021-02-10T11:47:40.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.888 7fd36399e700 1 -- 172.21.2.112:0/3030911722 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd35c0e53a0 msgr2=0x7fd35c1b0550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:40.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.888 7fd36399e700 1 --2- 172.21.2.112:0/3030911722 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd35c0e53a0 0x7fd35c1b0550 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7fd350006650 tx=0x7fd350009a80).stop 2021-02-10T11:47:40.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.889 7fd36399e700 1 -- 172.21.2.112:0/3030911722 shutdown_connections 2021-02-10T11:47:40.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.889 7fd36399e700 1 --2- 172.21.2.112:0/3030911722 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd35c0e53a0 0x7fd35c1b0550 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:40.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.889 7fd36399e700 1 -- 172.21.2.112:0/3030911722 >> 172.21.2.112:0/3030911722 conn(0x7fd35c0ce4b0 msgr2=0x7fd35c0e45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:40.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.889 7fd36399e700 1 -- 172.21.2.112:0/3030911722 shutdown_connections 2021-02-10T11:47:40.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.889 7fd36399e700 1 -- 172.21.2.112:0/3030911722 wait complete. 2021-02-10T11:47:40.910 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.904 7fd36399e700 1 Processor -- start 2021-02-10T11:47:40.910 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.904 7fd36399e700 1 -- start start 2021-02-10T11:47:40.910 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.904 7fd36399e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd35c1a8820 0x7fd35c1a8c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:40.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.904 7fd36399e700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd35c08c6d0 0x7fd35c08cad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:40.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.904 7fd36399e700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd35c08d0f0 0x7fd35c08d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:40.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.904 7fd36399e700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd35c1a93a0 con 0x7fd35c1a8820 2021-02-10T11:47:40.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.904 7fd36399e700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd35c1a9920 con 0x7fd35c08c6d0 2021-02-10T11:47:40.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.904 7fd36399e700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd35c1a9b00 con 0x7fd35c08d0f0 2021-02-10T11:47:40.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.905 7fd3617ba700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd35c1a8820 0x7fd35c1a8c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:40.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.905 7fd3617ba700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd35c1a8820 0x7fd35c1a8c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:60628/0 (socket says 172.21.2.112:60628) 2021-02-10T11:47:40.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.905 7fd3617ba700 1 -- 172.21.2.112:0/4155068273 learned_addr learned my addr 172.21.2.112:0/4155068273 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:47:40.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.905 7fd360fb9700 1 --2- 172.21.2.112:0/4155068273 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd35c08c6d0 0x7fd35c08cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:40.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.905 7fd361fbb700 1 --2- 172.21.2.112:0/4155068273 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd35c08d0f0 0x7fd35c08d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:40.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.905 7fd3617ba700 1 -- 172.21.2.112:0/4155068273 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd35c08d0f0 msgr2=0x7fd35c08d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:40.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.905 7fd3617ba700 1 --2- 172.21.2.112:0/4155068273 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd35c08d0f0 0x7fd35c08d4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:40.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.905 7fd3617ba700 1 -- 172.21.2.112:0/4155068273 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd35c08c6d0 msgr2=0x7fd35c08cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:40.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.905 7fd3617ba700 1 --2- 172.21.2.112:0/4155068273 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd35c08c6d0 0x7fd35c08cad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:40.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.905 7fd3617ba700 1 -- 172.21.2.112:0/4155068273 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd35c1a9d20 con 0x7fd35c1a8820 2021-02-10T11:47:40.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.905 7fd3617ba700 1 --2- 172.21.2.112:0/4155068273 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd35c1a8820 0x7fd35c1a8c20 secure :-1 s=READY pgs=3071 cs=0 l=1 rev1=1 rx=0x7fd350009630 tx=0x7fd35000b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:40.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.905 7fd34dffb700 1 -- 172.21.2.112:0/4155068273 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd3500093a0 con 0x7fd35c1a8820 2021-02-10T11:47:40.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.905 7fd34dffb700 1 -- 172.21.2.112:0/4155068273 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd35000bd70 con 0x7fd35c1a8820 2021-02-10T11:47:40.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.905 7fd36399e700 1 -- 172.21.2.112:0/4155068273 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd35c1a9d20 con 0x7fd35c1a8820 2021-02-10T11:47:40.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.905 7fd34dffb700 1 -- 172.21.2.112:0/4155068273 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd350005560 con 0x7fd35c1a8820 2021-02-10T11:47:40.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.905 7fd36399e700 1 -- 172.21.2.112:0/4155068273 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd35c1aa2b0 con 0x7fd35c1a8820 2021-02-10T11:47:40.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.906 7fd34dffb700 1 -- 172.21.2.112:0/4155068273 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd350007020 con 0x7fd35c1a8820 2021-02-10T11:47:40.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.906 7fd34dffb700 1 --2- 172.21.2.112:0/4155068273 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd33c0186c0 0x7fd33c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:40.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.906 7fd34dffb700 1 -- 172.21.2.112:0/4155068273 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fd350032410 con 0x7fd35c1a8820 2021-02-10T11:47:40.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.907 7fd360fb9700 1 --2- 172.21.2.112:0/4155068273 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd33c0186c0 0x7fd33c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:40.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.907 7fd360fb9700 1 --2- 172.21.2.112:0/4155068273 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd33c0186c0 0x7fd33c01abc0 crc :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:40.940 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.937 7fd36399e700 1 -- 172.21.2.112:0/4155068273 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd35c103810 con 0x7fd35c1a8820 2021-02-10T11:47:40.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:40.940 7fd34dffb700 1 -- 172.21.2.112:0/4155068273 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd350009340 con 0x7fd35c1a8820 2021-02-10T11:47:41.069 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:41.066 7fd36399e700 1 -- 172.21.2.112:0/4155068273 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume authorize", "vol_name": "cephfs", "sub_name": "subvolume_0000000000120407", "target": ["mgr", ""], "tenant_id": "tenant1", "auth_id": "guest1", "group_name": "group_0000000000232095"}) v1 -- 0x7fd35c1a93a0 con 0x7fd33c0186c0 2021-02-10T11:47:41.074 INFO:tasks.ceph.mgr.x.gibba020.stderr:Creating meta for ID guest1 with tenant tenant1 2021-02-10T11:47:41.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:41.100 7fd34dffb700 1 -- 172.21.2.112:0/4155068273 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+41 (crc 0 0 0) 0x7fd358006a90 con 0x7fd33c0186c0 2021-02-10T11:47:41.133 INFO:teuthology.orchestra.run.gibba012.stdout:AQDdxyNgq7oCBRAAHDCX1bF9z3tQDQZlkq3y3w== 2021-02-10T11:47:41.135 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:41.131 7fd36399e700 1 -- 172.21.2.112:0/4155068273 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd33c0186c0 msgr2=0x7fd33c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:41.135 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:41.131 7fd36399e700 1 --2- 172.21.2.112:0/4155068273 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd33c0186c0 0x7fd33c01abc0 crc :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:41.135 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:41.131 7fd36399e700 1 -- 172.21.2.112:0/4155068273 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd35c1a8820 msgr2=0x7fd35c1a8c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:41.135 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:41.132 7fd36399e700 1 --2- 172.21.2.112:0/4155068273 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd35c1a8820 0x7fd35c1a8c20 secure :-1 s=READY pgs=3071 cs=0 l=1 rev1=1 rx=0x7fd350009630 tx=0x7fd35000b980).stop 2021-02-10T11:47:41.135 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:41.132 7fd36399e700 1 -- 172.21.2.112:0/4155068273 shutdown_connections 2021-02-10T11:47:41.135 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:41.132 7fd36399e700 1 --2- 172.21.2.112:0/4155068273 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd33c0186c0 0x7fd33c01abc0 unknown :-1 s=CLOSED pgs=1382 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:41.135 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:41.132 7fd36399e700 1 --2- 172.21.2.112:0/4155068273 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd35c08c6d0 0x7fd35c08cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:41.135 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:41.132 7fd36399e700 1 --2- 172.21.2.112:0/4155068273 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd35c08d0f0 0x7fd35c08d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:41.135 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:41.132 7fd36399e700 1 --2- 172.21.2.112:0/4155068273 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd35c1a8820 0x7fd35c1a8c20 unknown :-1 s=CLOSED pgs=3071 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:41.135 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:41.132 7fd36399e700 1 -- 172.21.2.112:0/4155068273 >> 172.21.2.112:0/4155068273 conn(0x7fd35c18cb60 msgr2=0x7fd35c199f40 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:41.135 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:41.132 7fd36399e700 1 -- 172.21.2.112:0/4155068273 shutdown_connections 2021-02-10T11:47:41.136 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:41.132 7fd36399e700 1 -- 172.21.2.112:0/4155068273 wait complete. 2021-02-10T11:47:41.143 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.1 && sudo ls volumes 2021-02-10T11:47:41.492 INFO:teuthology.orchestra.run.gibba012.stdout:group_0000000000232095 2021-02-10T11:47:41.493 INFO:teuthology.orchestra.run.gibba012.stdout:_group_0000000000232095:subvolume_0000000000120407.meta 2021-02-10T11:47:41.493 INFO:teuthology.orchestra.run.gibba012.stdout:_group_0000000000232095:subvolume_0000000000120408.meta 2021-02-10T11:47:41.493 INFO:teuthology.orchestra.run.gibba012.stdout:$guest1.meta 2021-02-10T11:47:41.494 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo sudo cat '/home/ubuntu/cephtest/mnt.0/volumes/$guest1.meta' 2021-02-10T11:47:41.857 INFO:teuthology.orchestra.run.gibba012.stdout:{"compat_version": 1, "tenant_id": "tenant1", "version": 5, "dirty": false, "subvolumes": {"group_0000000000232095/subvolume_0000000000120407": {"access_level": "rw", "dirty": false}}} 2021-02-10T11:47:41.857 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume authorize cephfs subvolume_0000000000120408 guest1 --group_name group_0000000000232095 --tenant_id tenant1 2021-02-10T11:47:42.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:42.279 7fd9163d0700 1 Processor -- start 2021-02-10T11:47:42.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:42.279 7fd9163d0700 1 -- start start 2021-02-10T11:47:42.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:42.279 7fd9163d0700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fd9100e53a0 0x7fd9101b0550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:42.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:42.279 7fd9163d0700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fd9101b0b70 0x7fd9101089d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:42.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:42.279 7fd9163d0700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd910109ac0 con 0x7fd910108ff0 2021-02-10T11:47:42.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:42.279 7fd9163d0700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9100e5f80 con 0x7fd9100e53a0 2021-02-10T11:47:42.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:42.279 7fd9163d0700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9100e57a0 con 0x7fd9101b0b70 2021-02-10T11:47:42.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:42.280 7fd90ffff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fd9100e53a0 0x7fd9101b0550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:42.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:42.280 7fd90ffff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fd9100e53a0 0x7fd9101b0550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:60636/0 (socket says 172.21.2.112:60636) 2021-02-10T11:47:42.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:42.280 7fd90ffff700 1 -- 172.21.2.112:0/2235577514 learned_addr learned my addr 172.21.2.112:0/2235577514 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:47:42.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:42.280 7fd90ffff700 1 -- 172.21.2.112:0/2235577514 >> v2:172.21.2.120:3300/0 conn(0x7fd9101b0b70 msgr2=0x7fd9101089d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:42.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:42.280 7fd90ffff700 1 --2- 172.21.2.112:0/2235577514 >> v2:172.21.2.120:3300/0 conn(0x7fd9101b0b70 0x7fd9101089d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:42.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:42.280 7fd90ffff700 1 -- 172.21.2.112:0/2235577514 >> v1:172.21.2.120:6789/0 conn(0x7fd910108ff0 legacy=0x7fd91010a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:42.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:42.280 7fd90ffff700 1 -- 172.21.2.112:0/2235577514 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd910088070 con 0x7fd9100e53a0 2021-02-10T11:47:42.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:42.280 7fd90ffff700 1 --2- 172.21.2.112:0/2235577514 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd9100e53a0 0x7fd9101b0550 secure :-1 s=READY pgs=3072 cs=0 l=1 rev1=1 rx=0x7fd900006700 tx=0x7fd900009b80).ready entity=mon.0 client_cookie=d312e76fafe85dba server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:42.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:42.280 7fd90effd700 1 -- 172.21.2.112:0/2235577514 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd90000c0c0 con 0x7fd9100e53a0 2021-02-10T11:47:42.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:42.280 7fd90effd700 1 -- 172.21.2.112:0/2235577514 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd90000c340 con 0x7fd9100e53a0 2021-02-10T11:47:42.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:42.280 7fd90effd700 1 -- 172.21.2.112:0/2235577514 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd90000c860 con 0x7fd9100e53a0 2021-02-10T11:47:42.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:42.280 7fd9163d0700 1 -- 172.21.2.112:0/2235577514 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd9100e53a0 msgr2=0x7fd9101b0550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:42.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:42.280 7fd9163d0700 1 --2- 172.21.2.112:0/2235577514 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd9100e53a0 0x7fd9101b0550 secure :-1 s=READY pgs=3072 cs=0 l=1 rev1=1 rx=0x7fd900006700 tx=0x7fd900009b80).stop 2021-02-10T11:47:42.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:42.281 7fd9163d0700 1 -- 172.21.2.112:0/2235577514 shutdown_connections 2021-02-10T11:47:42.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:42.281 7fd9163d0700 1 --2- 172.21.2.112:0/2235577514 >> v2:172.21.2.120:3300/0 conn(0x7fd9101b0b70 0x7fd9101089d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:42.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:42.281 7fd9163d0700 1 --2- 172.21.2.112:0/2235577514 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd9100e53a0 0x7fd9101b0550 unknown :-1 s=CLOSED pgs=3072 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:42.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:42.281 7fd9163d0700 1 -- 172.21.2.112:0/2235577514 >> 172.21.2.112:0/2235577514 conn(0x7fd9100ce4b0 msgr2=0x7fd9100e45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:42.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:42.281 7fd9163d0700 1 -- 172.21.2.112:0/2235577514 shutdown_connections 2021-02-10T11:47:42.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:42.281 7fd9163d0700 1 -- 172.21.2.112:0/2235577514 wait complete. 2021-02-10T11:47:42.302 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:42.296 7fd9163d0700 1 Processor -- start 2021-02-10T11:47:42.302 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:42.296 7fd9163d0700 1 -- start start 2021-02-10T11:47:42.303 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:42.296 7fd9163d0700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd9101a8820 0x7fd9101a8c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:42.303 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:42.296 7fd9163d0700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd91008c6d0 0x7fd91008cad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:42.303 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:42.296 7fd9163d0700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd91008d0f0 0x7fd91008d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:42.303 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:42.296 7fd9163d0700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9101a93a0 con 0x7fd91008c6d0 2021-02-10T11:47:42.303 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:42.296 7fd9163d0700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9101a9920 con 0x7fd91008d0f0 2021-02-10T11:47:42.303 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:42.296 7fd9163d0700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9101a9b00 con 0x7fd9101a8820 2021-02-10T11:47:42.303 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:42.296 7fd90f7fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd91008c6d0 0x7fd91008cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:42.303 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:42.296 7fd90f7fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd91008c6d0 0x7fd91008cad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:60644/0 (socket says 172.21.2.112:60644) 2021-02-10T11:47:42.303 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:42.296 7fd90f7fe700 1 -- 172.21.2.112:0/1006534507 learned_addr learned my addr 172.21.2.112:0/1006534507 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:47:42.303 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:42.297 7fd90ffff700 1 --2- 172.21.2.112:0/1006534507 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd9101a8820 0x7fd9101a8c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:42.303 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:42.297 7fd9149ed700 1 --2- 172.21.2.112:0/1006534507 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd91008d0f0 0x7fd91008d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:42.303 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:42.297 7fd90f7fe700 1 -- 172.21.2.112:0/1006534507 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd9101a8820 msgr2=0x7fd9101a8c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:42.303 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:42.297 7fd90f7fe700 1 --2- 172.21.2.112:0/1006534507 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd9101a8820 0x7fd9101a8c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:42.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:42.297 7fd90f7fe700 1 -- 172.21.2.112:0/1006534507 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd91008d0f0 msgr2=0x7fd91008d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:42.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:42.297 7fd90f7fe700 1 --2- 172.21.2.112:0/1006534507 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd91008d0f0 0x7fd91008d4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:42.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:42.297 7fd90f7fe700 1 -- 172.21.2.112:0/1006534507 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9101a9d20 con 0x7fd91008c6d0 2021-02-10T11:47:42.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:42.297 7fd90ffff700 1 --2- 172.21.2.112:0/1006534507 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd9101a8820 0x7fd9101a8c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:47:42.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:42.297 7fd90f7fe700 1 --2- 172.21.2.112:0/1006534507 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd91008c6d0 0x7fd91008cad0 secure :-1 s=READY pgs=3073 cs=0 l=1 rev1=1 rx=0x7fd904006f00 tx=0x7fd90400b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:42.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:42.297 7fd90cff9700 1 -- 172.21.2.112:0/1006534507 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd9040009b0 con 0x7fd91008c6d0 2021-02-10T11:47:42.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:42.297 7fd90cff9700 1 -- 172.21.2.112:0/1006534507 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd904006350 con 0x7fd91008c6d0 2021-02-10T11:47:42.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:42.297 7fd90cff9700 1 -- 172.21.2.112:0/1006534507 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd904000c90 con 0x7fd91008c6d0 2021-02-10T11:47:42.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:42.297 7fd9163d0700 1 -- 172.21.2.112:0/1006534507 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9101a9d20 con 0x7fd91008c6d0 2021-02-10T11:47:42.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:42.297 7fd9163d0700 1 -- 172.21.2.112:0/1006534507 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9101a9d20 con 0x7fd91008c6d0 2021-02-10T11:47:42.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:42.298 7fd90cff9700 1 -- 172.21.2.112:0/1006534507 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd904005310 con 0x7fd91008c6d0 2021-02-10T11:47:42.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:42.298 7fd90cff9700 1 --2- 172.21.2.112:0/1006534507 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd8f00186d0 0x7fd8f001abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:42.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:42.298 7fd90cff9700 1 -- 172.21.2.112:0/1006534507 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fd9040325f0 con 0x7fd91008c6d0 2021-02-10T11:47:42.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:42.298 7fd90ffff700 1 --2- 172.21.2.112:0/1006534507 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd8f00186d0 0x7fd8f001abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:42.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:42.299 7fd90ffff700 1 --2- 172.21.2.112:0/1006534507 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd8f00186d0 0x7fd8f001abd0 crc :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:42.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:42.329 7fd9163d0700 1 -- 172.21.2.112:0/1006534507 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9101a98c0 con 0x7fd91008c6d0 2021-02-10T11:47:42.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:42.332 7fd90cff9700 1 -- 172.21.2.112:0/1006534507 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd904006b90 con 0x7fd91008c6d0 2021-02-10T11:47:42.460 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:42.457 7fd9163d0700 1 -- 172.21.2.112:0/1006534507 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume authorize", "vol_name": "cephfs", "sub_name": "subvolume_0000000000120408", "target": ["mgr", ""], "tenant_id": "tenant1", "auth_id": "guest1", "group_name": "group_0000000000232095"}) v1 -- 0x7fd9101a98c0 con 0x7fd8f00186d0 2021-02-10T11:47:42.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:42.492 7fd90cff9700 1 -- 172.21.2.112:0/1006534507 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+41 (crc 0 0 0) 0x7fd9000090c0 con 0x7fd8f00186d0 2021-02-10T11:47:42.523 INFO:teuthology.orchestra.run.gibba012.stdout:AQDdxyNgq7oCBRAAHDCX1bF9z3tQDQZlkq3y3w== 2021-02-10T11:47:42.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:42.522 7fd9163d0700 1 -- 172.21.2.112:0/1006534507 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd8f00186d0 msgr2=0x7fd8f001abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:42.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:42.522 7fd9163d0700 1 --2- 172.21.2.112:0/1006534507 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd8f00186d0 0x7fd8f001abd0 crc :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:42.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:42.522 7fd9163d0700 1 -- 172.21.2.112:0/1006534507 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd91008c6d0 msgr2=0x7fd91008cad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:42.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:42.522 7fd9163d0700 1 --2- 172.21.2.112:0/1006534507 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd91008c6d0 0x7fd91008cad0 secure :-1 s=READY pgs=3073 cs=0 l=1 rev1=1 rx=0x7fd904006f00 tx=0x7fd90400b940).stop 2021-02-10T11:47:42.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:42.522 7fd9163d0700 1 -- 172.21.2.112:0/1006534507 shutdown_connections 2021-02-10T11:47:42.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:42.522 7fd9163d0700 1 --2- 172.21.2.112:0/1006534507 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd8f00186d0 0x7fd8f001abd0 unknown :-1 s=CLOSED pgs=1383 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:42.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:42.522 7fd9163d0700 1 --2- 172.21.2.112:0/1006534507 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd91008d0f0 0x7fd91008d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:42.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:42.522 7fd9163d0700 1 --2- 172.21.2.112:0/1006534507 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd91008c6d0 0x7fd91008cad0 unknown :-1 s=CLOSED pgs=3073 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:42.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:42.522 7fd9163d0700 1 --2- 172.21.2.112:0/1006534507 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd9101a8820 0x7fd9101a8c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:42.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:42.522 7fd9163d0700 1 -- 172.21.2.112:0/1006534507 >> 172.21.2.112:0/1006534507 conn(0x7fd91018cb60 msgr2=0x7fd910199f40 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:42.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:42.523 7fd9163d0700 1 -- 172.21.2.112:0/1006534507 shutdown_connections 2021-02-10T11:47:42.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:42.523 7fd9163d0700 1 -- 172.21.2.112:0/1006534507 wait complete. 2021-02-10T11:47:42.533 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.1 && sudo sed -i s/false/true/g 'volumes/$guest1.meta' 2021-02-10T11:47:42.903 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume deauthorize cephfs subvolume_0000000000120408 guest1 --group_name group_0000000000232095 2021-02-10T11:47:43.336 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:43.317 7f49a232c700 1 Processor -- start 2021-02-10T11:47:43.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:43.317 7f49a232c700 1 -- start start 2021-02-10T11:47:43.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:43.317 7f49a232c700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f499c0dce80 0x7f499c0d14f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:43.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:43.317 7f49a232c700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f499c10b2c0 0x7f499c0eb620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:43.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:43.317 7f49a232c700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f499c10a550 con 0x7f499c0eb220 2021-02-10T11:47:43.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:43.317 7f49a232c700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f499c088240 con 0x7f499c0dce80 2021-02-10T11:47:43.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:43.317 7f49a232c700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f499c0d1f30 con 0x7f499c10b2c0 2021-02-10T11:47:43.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:43.317 7f499b7fe700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f499c0eb220 0x7f499c10aad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:60116/0 (socket says 172.21.2.112:60116) 2021-02-10T11:47:43.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:43.317 7f499bfff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f499c0dce80 0x7f499c0d14f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:43.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:43.317 7f499b7fe700 1 -- 172.21.2.112:0/2961877570 learned_addr learned my addr 172.21.2.112:0/2961877570 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:47:43.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:43.318 7f499bfff700 1 -- 172.21.2.112:0/2961877570 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f499c10b2c0 msgr2=0x7f499c0eb620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:43.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:43.318 7f499bfff700 1 --2- 172.21.2.112:0/2961877570 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f499c10b2c0 0x7f499c0eb620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:43.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:43.318 7f499bfff700 1 -- 172.21.2.112:0/2961877570 >> v1:172.21.2.112:6789/0 conn(0x7f499c0eb220 legacy=0x7f499c10aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:43.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:43.318 7f499bfff700 1 -- 172.21.2.112:0/2961877570 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f499c0dd280 con 0x7f499c0dce80 2021-02-10T11:47:43.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:43.318 7f499bfff700 1 --2- 172.21.2.112:0/2961877570 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f499c0dce80 0x7f499c0d14f0 secure :-1 s=READY pgs=3075 cs=0 l=1 rev1=1 rx=0x7f498c006650 tx=0x7f498c009a80).ready entity=mon.0 client_cookie=673879183c4bfb0f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:43.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:43.318 7f499affd700 1 -- 172.21.2.112:0/2961877570 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f498c00c0c0 con 0x7f499c0dce80 2021-02-10T11:47:43.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:43.318 7f499affd700 1 -- 172.21.2.112:0/2961877570 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f498c00c340 con 0x7f499c0dce80 2021-02-10T11:47:43.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:43.318 7f499affd700 1 -- 172.21.2.112:0/2961877570 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f498c00c860 con 0x7f499c0dce80 2021-02-10T11:47:43.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:43.318 7f49a232c700 1 -- 172.21.2.112:0/2961877570 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f499c0dce80 msgr2=0x7f499c0d14f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:43.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:43.318 7f49a232c700 1 --2- 172.21.2.112:0/2961877570 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f499c0dce80 0x7f499c0d14f0 secure :-1 s=READY pgs=3075 cs=0 l=1 rev1=1 rx=0x7f498c006650 tx=0x7f498c009a80).stop 2021-02-10T11:47:43.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:43.319 7f49a232c700 1 -- 172.21.2.112:0/2961877570 shutdown_connections 2021-02-10T11:47:43.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:43.319 7f49a232c700 1 --2- 172.21.2.112:0/2961877570 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f499c10b2c0 0x7f499c0eb620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:43.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:43.319 7f49a232c700 1 --2- 172.21.2.112:0/2961877570 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f499c0dce80 0x7f499c0d14f0 unknown :-1 s=CLOSED pgs=3075 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:43.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:43.319 7f49a232c700 1 -- 172.21.2.112:0/2961877570 >> 172.21.2.112:0/2961877570 conn(0x7f499c0ea630 msgr2=0x7f499c0d0800 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:43.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:43.319 7f49a232c700 1 -- 172.21.2.112:0/2961877570 shutdown_connections 2021-02-10T11:47:43.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:43.319 7f49a232c700 1 -- 172.21.2.112:0/2961877570 wait complete. 2021-02-10T11:47:43.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:43.334 7f49a232c700 1 Processor -- start 2021-02-10T11:47:43.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:43.334 7f49a232c700 1 -- start start 2021-02-10T11:47:43.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:43.334 7f49a232c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f499c0ac9e0 0x7f499c104d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:43.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:43.334 7f49a232c700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f499c092540 0x7f499c092940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:43.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:43.334 7f49a232c700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f499c092f60 0x7f499c093360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:43.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:43.334 7f49a232c700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f499c105340 con 0x7f499c0ac9e0 2021-02-10T11:47:43.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:43.334 7f49a232c700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f499c090900 con 0x7f499c092540 2021-02-10T11:47:43.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:43.334 7f49a232c700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f499c090b20 con 0x7f499c092f60 2021-02-10T11:47:43.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:43.335 7f499bfff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f499c0ac9e0 0x7f499c104d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:43.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:43.335 7f499bfff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f499c0ac9e0 0x7f499c104d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:60664/0 (socket says 172.21.2.112:60664) 2021-02-10T11:47:43.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:43.335 7f499bfff700 1 -- 172.21.2.112:0/1105585173 learned_addr learned my addr 172.21.2.112:0/1105585173 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:47:43.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:43.335 7f499bfff700 1 -- 172.21.2.112:0/1105585173 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f499c092f60 msgr2=0x7f499c093360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:43.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:43.335 7f499bfff700 1 --2- 172.21.2.112:0/1105585173 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f499c092f60 0x7f499c093360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:43.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:43.335 7f499bfff700 1 -- 172.21.2.112:0/1105585173 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f499c092540 msgr2=0x7f499c092940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:43.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:43.335 7f499bfff700 1 --2- 172.21.2.112:0/1105585173 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f499c092540 0x7f499c092940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:43.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:43.335 7f499bfff700 1 -- 172.21.2.112:0/1105585173 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f499c090d40 con 0x7f499c0ac9e0 2021-02-10T11:47:43.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:43.335 7f499bfff700 1 --2- 172.21.2.112:0/1105585173 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f499c0ac9e0 0x7f499c104d20 secure :-1 s=READY pgs=3076 cs=0 l=1 rev1=1 rx=0x7f498c009630 tx=0x7f498c00b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:43.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:43.335 7f4998ff9700 1 -- 172.21.2.112:0/1105585173 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f498c00bd70 con 0x7f499c0ac9e0 2021-02-10T11:47:43.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:43.335 7f4998ff9700 1 -- 172.21.2.112:0/1105585173 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f498c000960 con 0x7f499c0ac9e0 2021-02-10T11:47:43.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:43.335 7f49a232c700 1 -- 172.21.2.112:0/1105585173 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f499c090d40 con 0x7f499c0ac9e0 2021-02-10T11:47:43.342 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:43.335 7f49a232c700 1 -- 172.21.2.112:0/1105585173 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f499c090d40 con 0x7f499c0ac9e0 2021-02-10T11:47:43.342 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:43.336 7f4998ff9700 1 -- 172.21.2.112:0/1105585173 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f498c005560 con 0x7f499c0ac9e0 2021-02-10T11:47:43.342 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:43.336 7f4998ff9700 1 -- 172.21.2.112:0/1105585173 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f498c00a000 con 0x7f499c0ac9e0 2021-02-10T11:47:43.342 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:43.336 7f4998ff9700 1 --2- 172.21.2.112:0/1105585173 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f497c018730 0x7f497c01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:43.342 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:43.336 7f4998ff9700 1 -- 172.21.2.112:0/1105585173 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f498c032680 con 0x7f499c0ac9e0 2021-02-10T11:47:43.342 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:43.337 7f499b7fe700 1 --2- 172.21.2.112:0/1105585173 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f497c018730 0x7f497c01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:43.342 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:43.337 7f499b7fe700 1 --2- 172.21.2.112:0/1105585173 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f497c018730 0x7f497c01ac10 crc :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:43.370 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:43.367 7f49a232c700 1 -- 172.21.2.112:0/1105585173 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4974002fd0 con 0x7f499c0ac9e0 2021-02-10T11:47:43.373 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:43.370 7f4998ff9700 1 -- 172.21.2.112:0/1105585173 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f498c000940 con 0x7f499c0ac9e0 2021-02-10T11:47:43.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:43.496 7f49a232c700 1 -- 172.21.2.112:0/1105585173 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume deauthorize", "vol_name": "cephfs", "sub_name": "subvolume_0000000000120408", "target": ["mgr", ""], "auth_id": "guest1", "group_name": "group_0000000000232095"}) v1 -- 0x7f4974002f80 con 0x7f497c018730 2021-02-10T11:47:43.561 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:43.558 7f4998ff9700 1 -- 172.21.2.112:0/1105585173 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f4990000e20 con 0x7f497c018730 2021-02-10T11:47:43.564 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:43.561 7f49a232c700 1 -- 172.21.2.112:0/1105585173 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f497c018730 msgr2=0x7f497c01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:43.565 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:43.561 7f49a232c700 1 --2- 172.21.2.112:0/1105585173 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f497c018730 0x7f497c01ac10 crc :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:43.565 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:43.561 7f49a232c700 1 -- 172.21.2.112:0/1105585173 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f499c0ac9e0 msgr2=0x7f499c104d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:43.565 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:43.561 7f49a232c700 1 --2- 172.21.2.112:0/1105585173 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f499c0ac9e0 0x7f499c104d20 secure :-1 s=READY pgs=3076 cs=0 l=1 rev1=1 rx=0x7f498c009630 tx=0x7f498c00b980).stop 2021-02-10T11:47:43.565 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:43.562 7f49a232c700 1 -- 172.21.2.112:0/1105585173 shutdown_connections 2021-02-10T11:47:43.565 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:43.562 7f49a232c700 1 --2- 172.21.2.112:0/1105585173 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f497c018730 0x7f497c01ac10 unknown :-1 s=CLOSED pgs=1384 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:43.565 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:43.562 7f49a232c700 1 --2- 172.21.2.112:0/1105585173 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f499c092540 0x7f499c092940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:43.565 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:43.562 7f49a232c700 1 --2- 172.21.2.112:0/1105585173 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f499c092f60 0x7f499c093360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:43.565 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:43.562 7f49a232c700 1 --2- 172.21.2.112:0/1105585173 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f499c0ac9e0 0x7f499c104d20 unknown :-1 s=CLOSED pgs=3076 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:43.565 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:43.562 7f49a232c700 1 -- 172.21.2.112:0/1105585173 >> 172.21.2.112:0/1105585173 conn(0x7f499c1a8260 msgr2=0x7f499c0fac90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:43.565 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:43.562 7f49a232c700 1 -- 172.21.2.112:0/1105585173 shutdown_connections 2021-02-10T11:47:43.565 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:43.562 7f49a232c700 1 -- 172.21.2.112:0/1105585173 wait complete. 2021-02-10T11:47:43.573 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo sudo cat '/home/ubuntu/cephtest/mnt.0/volumes/$guest1.meta' 2021-02-10T11:47:43.935 INFO:teuthology.orchestra.run.gibba012.stdout:{"tenant_id": "tenant1", "version": 5, "dirty": false, "compat_version": 1, "subvolumes": {"group_0000000000232095/subvolume_0000000000120407": {"access_level": "rw", "dirty": false}}} 2021-02-10T11:47:43.935 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume deauthorize cephfs subvolume_0000000000120407 guest1 --group_name group_0000000000232095 2021-02-10T11:47:44.369 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:44.349 7f6fc3871700 1 Processor -- start 2021-02-10T11:47:44.369 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:44.349 7f6fc3871700 1 -- start start 2021-02-10T11:47:44.369 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:44.350 7f6fc3871700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f6fbc0dce80 0x7f6fbc0d14f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:44.369 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:44.350 7f6fc3871700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6fbc0eb220 0x7f6fbc0eb620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:44.369 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:44.350 7f6fc3871700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f6fbc10aad0 0x7f6fbc10aed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:44.370 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:44.350 7f6fc3871700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fbc10b570 con 0x7f6fbc0dce80 2021-02-10T11:47:44.370 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:44.350 7f6fc3871700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fbc10a550 con 0x7f6fbc10aad0 2021-02-10T11:47:44.370 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:44.350 7f6fc3871700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fbc0881d0 con 0x7f6fbc0eb220 2021-02-10T11:47:44.370 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:44.350 7f6fc168d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f6fbc0dce80 0x7f6fbc0d14f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:44.370 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:44.350 7f6fc168d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f6fbc0dce80 0x7f6fbc0d14f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:60670/0 (socket says 172.21.2.112:60670) 2021-02-10T11:47:44.370 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:44.350 7f6fc168d700 1 -- 172.21.2.112:0/825860410 learned_addr learned my addr 172.21.2.112:0/825860410 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:47:44.370 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:44.350 7f6fc1e8e700 1 --2- 172.21.2.112:0/825860410 >> v2:172.21.2.120:3300/0 conn(0x7f6fbc10aad0 0x7f6fbc10aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:44.370 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:44.350 7f6fc0e8c700 1 --2- 172.21.2.112:0/825860410 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6fbc0eb220 0x7f6fbc0eb620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:44.370 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:44.350 7f6fc168d700 1 -- 172.21.2.112:0/825860410 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6fbc0eb220 msgr2=0x7f6fbc0eb620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:44.370 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:44.350 7f6fc168d700 1 --2- 172.21.2.112:0/825860410 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6fbc0eb220 0x7f6fbc0eb620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:44.370 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:44.350 7f6fc168d700 1 -- 172.21.2.112:0/825860410 >> v2:172.21.2.120:3300/0 conn(0x7f6fbc10aad0 msgr2=0x7f6fbc10aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:44.370 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:44.350 7f6fc168d700 1 --2- 172.21.2.112:0/825860410 >> v2:172.21.2.120:3300/0 conn(0x7f6fbc10aad0 0x7f6fbc10aed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:44.371 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:44.350 7f6fc168d700 1 -- 172.21.2.112:0/825860410 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fbc0d1f30 con 0x7f6fbc0dce80 2021-02-10T11:47:44.371 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:44.350 7f6fc1e8e700 1 --2- 172.21.2.112:0/825860410 >> v2:172.21.2.120:3300/0 conn(0x7f6fbc10aad0 0x7f6fbc10aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:47:44.371 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:44.350 7f6fc0e8c700 1 --2- 172.21.2.112:0/825860410 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6fbc0eb220 0x7f6fbc0eb620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:47:44.371 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:44.351 7f6fc168d700 1 --2- 172.21.2.112:0/825860410 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6fbc0dce80 0x7f6fbc0d14f0 secure :-1 s=READY pgs=3077 cs=0 l=1 rev1=1 rx=0x7f6fb8006700 tx=0x7f6fb8009b80).ready entity=mon.0 client_cookie=44e96316d92c7549 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:44.371 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:44.351 7f6fb3fff700 1 -- 172.21.2.112:0/825860410 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6fb800c0c0 con 0x7f6fbc0dce80 2021-02-10T11:47:44.371 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:44.351 7f6fb3fff700 1 -- 172.21.2.112:0/825860410 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fb800c340 con 0x7f6fbc0dce80 2021-02-10T11:47:44.371 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:44.351 7f6fb3fff700 1 -- 172.21.2.112:0/825860410 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6fb800c860 con 0x7f6fbc0dce80 2021-02-10T11:47:44.371 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:44.351 7f6fc3871700 1 -- 172.21.2.112:0/825860410 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6fbc0dce80 msgr2=0x7f6fbc0d14f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:44.371 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:44.351 7f6fc3871700 1 --2- 172.21.2.112:0/825860410 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6fbc0dce80 0x7f6fbc0d14f0 secure :-1 s=READY pgs=3077 cs=0 l=1 rev1=1 rx=0x7f6fb8006700 tx=0x7f6fb8009b80).stop 2021-02-10T11:47:44.371 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:44.352 7f6fc3871700 1 -- 172.21.2.112:0/825860410 shutdown_connections 2021-02-10T11:47:44.371 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:44.352 7f6fc3871700 1 --2- 172.21.2.112:0/825860410 >> v2:172.21.2.120:3300/0 conn(0x7f6fbc10aad0 0x7f6fbc10aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:44.371 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:44.352 7f6fc3871700 1 --2- 172.21.2.112:0/825860410 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6fbc0eb220 0x7f6fbc0eb620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:44.371 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:44.352 7f6fc3871700 1 --2- 172.21.2.112:0/825860410 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6fbc0dce80 0x7f6fbc0d14f0 unknown :-1 s=CLOSED pgs=3077 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:44.371 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:44.352 7f6fc3871700 1 -- 172.21.2.112:0/825860410 >> 172.21.2.112:0/825860410 conn(0x7f6fbc0ea630 msgr2=0x7f6fbc0d0800 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:44.372 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:44.352 7f6fc3871700 1 -- 172.21.2.112:0/825860410 shutdown_connections 2021-02-10T11:47:44.372 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:44.352 7f6fc3871700 1 -- 172.21.2.112:0/825860410 wait complete. 2021-02-10T11:47:44.373 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:44.367 7f6fc3871700 1 Processor -- start 2021-02-10T11:47:44.373 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:44.367 7f6fc3871700 1 -- start start 2021-02-10T11:47:44.373 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:44.367 7f6fc3871700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6fbc1a0010 0x7f6fbc0a8400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:44.373 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:44.367 7f6fc3871700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6fbc0a8a20 0x7f6fbc095fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:44.373 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:44.367 7f6fc3871700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6fbc1a0410 0x7f6fbc0965c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:44.373 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:44.367 7f6fc3871700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fbc10a1b0 con 0x7f6fbc1a0410 2021-02-10T11:47:44.373 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:44.367 7f6fc3871700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fbc10a390 con 0x7f6fbc1a0010 2021-02-10T11:47:44.373 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:44.367 7f6fc3871700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fbc1a09d0 con 0x7f6fbc0a8a20 2021-02-10T11:47:44.374 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:44.367 7f6fc1e8e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6fbc1a0410 0x7f6fbc0965c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:44.374 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:44.367 7f6fc1e8e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6fbc1a0410 0x7f6fbc0965c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:60674/0 (socket says 172.21.2.112:60674) 2021-02-10T11:47:44.374 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:44.367 7f6fc1e8e700 1 -- 172.21.2.112:0/2975820368 learned_addr learned my addr 172.21.2.112:0/2975820368 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:47:44.374 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:44.368 7f6fc1e8e700 1 -- 172.21.2.112:0/2975820368 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6fbc0a8a20 msgr2=0x7f6fbc095fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:44.374 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:44.368 7f6fc1e8e700 1 --2- 172.21.2.112:0/2975820368 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6fbc0a8a20 0x7f6fbc095fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:44.374 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:44.368 7f6fc1e8e700 1 -- 172.21.2.112:0/2975820368 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6fbc1a0010 msgr2=0x7f6fbc0a8400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:44.374 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:44.368 7f6fc1e8e700 1 --2- 172.21.2.112:0/2975820368 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6fbc1a0010 0x7f6fbc0a8400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:44.374 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:44.368 7f6fc1e8e700 1 -- 172.21.2.112:0/2975820368 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fbc0954e0 con 0x7f6fbc1a0410 2021-02-10T11:47:44.374 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:44.368 7f6fc1e8e700 1 --2- 172.21.2.112:0/2975820368 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6fbc1a0410 0x7f6fbc0965c0 secure :-1 s=READY pgs=3078 cs=0 l=1 rev1=1 rx=0x7f6fb4009980 tx=0x7f6fb400bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:44.374 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:44.368 7f6fb1ffb700 1 -- 172.21.2.112:0/2975820368 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6fb400c2f0 con 0x7f6fbc1a0410 2021-02-10T11:47:44.374 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:44.368 7f6fb1ffb700 1 -- 172.21.2.112:0/2975820368 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fb4006790 con 0x7f6fbc1a0410 2021-02-10T11:47:44.374 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:44.368 7f6fb1ffb700 1 -- 172.21.2.112:0/2975820368 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6fb400c870 con 0x7f6fbc1a0410 2021-02-10T11:47:44.375 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:44.368 7f6fc3871700 1 -- 172.21.2.112:0/2975820368 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fbc0954e0 con 0x7f6fbc1a0410 2021-02-10T11:47:44.375 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:44.368 7f6fc3871700 1 -- 172.21.2.112:0/2975820368 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fbc096ed0 con 0x7f6fbc1a0410 2021-02-10T11:47:44.375 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:44.369 7f6fb1ffb700 1 -- 172.21.2.112:0/2975820368 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6fb40011f0 con 0x7f6fbc1a0410 2021-02-10T11:47:44.375 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:44.369 7f6fb1ffb700 1 --2- 172.21.2.112:0/2975820368 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6f9c0186c0 0x7f6f9c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:44.375 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:44.369 7f6fb1ffb700 1 -- 172.21.2.112:0/2975820368 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f6fb4032810 con 0x7f6fbc1a0410 2021-02-10T11:47:44.375 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:44.369 7f6fc168d700 1 --2- 172.21.2.112:0/2975820368 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6f9c0186c0 0x7f6f9c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:44.375 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:44.370 7f6fc168d700 1 --2- 172.21.2.112:0/2975820368 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6f9c0186c0 0x7f6f9c01abc0 crc :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:44.402 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:44.399 7f6fc3871700 1 -- 172.21.2.112:0/2975820368 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fbc095780 con 0x7f6fbc1a0410 2021-02-10T11:47:44.405 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:44.402 7f6fb1ffb700 1 -- 172.21.2.112:0/2975820368 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6fb4000f40 con 0x7f6fbc1a0410 2021-02-10T11:47:44.535 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:44.532 7f6fc3871700 1 -- 172.21.2.112:0/2975820368 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume deauthorize", "vol_name": "cephfs", "sub_name": "subvolume_0000000000120407", "target": ["mgr", ""], "auth_id": "guest1", "group_name": "group_0000000000232095"}) v1 -- 0x7f6fbc096ed0 con 0x7f6f9c0186c0 2021-02-10T11:47:44.568 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:44.565 7f6fb1ffb700 1 -- 172.21.2.112:0/2975820368 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f6fb8000990 con 0x7f6f9c0186c0 2021-02-10T11:47:44.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:44.597 7f6fc3871700 1 -- 172.21.2.112:0/2975820368 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6f9c0186c0 msgr2=0x7f6f9c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:44.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:44.597 7f6fc3871700 1 --2- 172.21.2.112:0/2975820368 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6f9c0186c0 0x7f6f9c01abc0 crc :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:44.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:44.597 7f6fc3871700 1 -- 172.21.2.112:0/2975820368 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6fbc1a0410 msgr2=0x7f6fbc0965c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:44.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:44.597 7f6fc3871700 1 --2- 172.21.2.112:0/2975820368 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6fbc1a0410 0x7f6fbc0965c0 secure :-1 s=READY pgs=3078 cs=0 l=1 rev1=1 rx=0x7f6fb4009980 tx=0x7f6fb400bcd0).stop 2021-02-10T11:47:44.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:44.598 7f6fc3871700 1 -- 172.21.2.112:0/2975820368 shutdown_connections 2021-02-10T11:47:44.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:44.598 7f6fc3871700 1 --2- 172.21.2.112:0/2975820368 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6f9c0186c0 0x7f6f9c01abc0 unknown :-1 s=CLOSED pgs=1385 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:44.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:44.598 7f6fc3871700 1 --2- 172.21.2.112:0/2975820368 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6fbc1a0410 0x7f6fbc0965c0 unknown :-1 s=CLOSED pgs=3078 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:44.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:44.598 7f6fc3871700 1 --2- 172.21.2.112:0/2975820368 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6fbc0a8a20 0x7f6fbc095fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:44.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:44.598 7f6fc3871700 1 --2- 172.21.2.112:0/2975820368 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6fbc1a0010 0x7f6fbc0a8400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:44.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:44.598 7f6fc3871700 1 -- 172.21.2.112:0/2975820368 >> 172.21.2.112:0/2975820368 conn(0x7f6fbc1aea90 msgr2=0x7f6fbc19e1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:44.601 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:44.598 7f6fc3871700 1 -- 172.21.2.112:0/2975820368 shutdown_connections 2021-02-10T11:47:44.602 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:44.598 7f6fc3871700 1 -- 172.21.2.112:0/2975820368 wait complete. 2021-02-10T11:47:44.609 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:47:44.609 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:47:44.609 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:47:44.944 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:47:44.944 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:47:44.944 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba012.front.sepia.ceph.com... 2021-02-10T11:47:44.944 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:47:44.944 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:47:45.299 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba012.stderr:ceph-fuse[188743]: fuse finished with error 0 and tester_r 0 2021-02-10T11:47:45.300 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:47:45.301 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:47:45.301 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:47:45.637 INFO:teuthology.orchestra.run.gibba012.stdout:ext2/ext3 2021-02-10T11:47:45.637 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:47:45.638 INFO:teuthology.orchestra.run:waiting for 900 2021-02-10T11:47:45.638 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:47:45.638 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:47:45.977 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth rm client.guest1 2021-02-10T11:47:46.405 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:46.386 7f3998f1d700 1 Processor -- start 2021-02-10T11:47:46.406 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:46.386 7f3998f1d700 1 -- start start 2021-02-10T11:47:46.406 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:46.387 7f3998f1d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f39940d1870 0x7f39941b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:46.406 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:46.387 7f3998f1d700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f39940f0650 0x7f39940cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:46.406 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:46.387 7f3998f1d700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39940f0b10 con 0x7f39941b0a10 2021-02-10T11:47:46.406 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:46.387 7f3998f1d700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39940e5ed0 con 0x7f39940d1870 2021-02-10T11:47:46.406 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:46.387 7f3998f1d700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39940d1c70 con 0x7f39940f0650 2021-02-10T11:47:46.406 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:46.387 7f3992b13700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f39940d1870 0x7f39941b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:46.406 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:46.387 7f3992b13700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f39940d1870 0x7f39941b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:60682/0 (socket says 172.21.2.112:60682) 2021-02-10T11:47:46.406 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:46.387 7f3992b13700 1 -- 172.21.2.112:0/1060478685 learned_addr learned my addr 172.21.2.112:0/1060478685 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:47:46.407 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:46.387 7f3992b13700 1 -- 172.21.2.112:0/1060478685 >> v2:172.21.2.120:3300/0 conn(0x7f39940f0650 msgr2=0x7f39940cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:46.407 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:46.387 7f3992b13700 1 --2- 172.21.2.112:0/1060478685 >> v2:172.21.2.120:3300/0 conn(0x7f39940f0650 0x7f39940cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:46.407 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:46.387 7f3992b13700 1 -- 172.21.2.112:0/1060478685 >> v1:172.21.2.112:6789/0 conn(0x7f39941b0a10 legacy=0x7f39940cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:46.407 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:46.387 7f3992b13700 1 -- 172.21.2.112:0/1060478685 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39940d1e90 con 0x7f39940d1870 2021-02-10T11:47:46.407 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:46.387 7f3992b13700 1 --2- 172.21.2.112:0/1060478685 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f39940d1870 0x7f39941b03f0 secure :-1 s=READY pgs=3080 cs=0 l=1 rev1=1 rx=0x7f3984006700 tx=0x7f3984009b00).ready entity=mon.0 client_cookie=f5b649464e34100e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:46.407 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:46.387 7f3991b11700 1 -- 172.21.2.112:0/1060478685 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f398400c0c0 con 0x7f39940d1870 2021-02-10T11:47:46.407 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:46.388 7f3991b11700 1 -- 172.21.2.112:0/1060478685 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f398400c340 con 0x7f39940d1870 2021-02-10T11:47:46.407 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:46.388 7f3991b11700 1 -- 172.21.2.112:0/1060478685 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f398400c820 con 0x7f39940d1870 2021-02-10T11:47:46.407 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:46.388 7f3998f1d700 1 -- 172.21.2.112:0/1060478685 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f39940d1870 msgr2=0x7f39941b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:46.407 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:46.388 7f3998f1d700 1 --2- 172.21.2.112:0/1060478685 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f39940d1870 0x7f39941b03f0 secure :-1 s=READY pgs=3080 cs=0 l=1 rev1=1 rx=0x7f3984006700 tx=0x7f3984009b00).stop 2021-02-10T11:47:46.407 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:46.388 7f3998f1d700 1 -- 172.21.2.112:0/1060478685 shutdown_connections 2021-02-10T11:47:46.407 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:46.388 7f3998f1d700 1 --2- 172.21.2.112:0/1060478685 >> v2:172.21.2.120:3300/0 conn(0x7f39940f0650 0x7f39940cfb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:46.408 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:46.388 7f3998f1d700 1 --2- 172.21.2.112:0/1060478685 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f39940d1870 0x7f39941b03f0 unknown :-1 s=CLOSED pgs=3080 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:46.408 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:46.388 7f3998f1d700 1 -- 172.21.2.112:0/1060478685 >> 172.21.2.112:0/1060478685 conn(0x7f39940ce450 msgr2=0x7f39940d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:46.408 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:46.388 7f3998f1d700 1 -- 172.21.2.112:0/1060478685 shutdown_connections 2021-02-10T11:47:46.408 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:46.388 7f3998f1d700 1 -- 172.21.2.112:0/1060478685 wait complete. 2021-02-10T11:47:46.409 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:46.403 7f3998f1d700 1 Processor -- start 2021-02-10T11:47:46.409 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:46.403 7f3998f1d700 1 -- start start 2021-02-10T11:47:46.409 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:46.403 7f3998f1d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f39940bdda0 0x7f399418b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:46.409 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:46.403 7f3998f1d700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f399418b870 0x7f39940c0030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:46.410 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:46.403 7f3998f1d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f399418bc70 0x7f39940c0650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:46.410 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:46.403 7f3998f1d700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f399419d390 con 0x7f399418bc70 2021-02-10T11:47:46.410 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:46.403 7f3998f1d700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39940bd740 con 0x7f399418b870 2021-02-10T11:47:46.410 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:46.403 7f3998f1d700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39940be1a0 con 0x7f39940bdda0 2021-02-10T11:47:46.410 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:46.403 7f3993314700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f399418bc70 0x7f39940c0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:46.410 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:46.403 7f3993314700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f399418bc70 0x7f39940c0650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:60690/0 (socket says 172.21.2.112:60690) 2021-02-10T11:47:46.410 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:46.403 7f3993314700 1 -- 172.21.2.112:0/1990198123 learned_addr learned my addr 172.21.2.112:0/1990198123 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:47:46.411 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:46.404 7f3993314700 1 -- 172.21.2.112:0/1990198123 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f39940bdda0 msgr2=0x7f399418b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:46.411 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:46.404 7f3993314700 1 --2- 172.21.2.112:0/1990198123 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f39940bdda0 0x7f399418b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:46.411 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:46.404 7f3993314700 1 -- 172.21.2.112:0/1990198123 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f399418b870 msgr2=0x7f39940c0030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:46.411 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:46.404 7f3993314700 1 --2- 172.21.2.112:0/1990198123 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f399418b870 0x7f39940c0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:46.411 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:46.404 7f3993314700 1 -- 172.21.2.112:0/1990198123 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39940c0c70 con 0x7f399418bc70 2021-02-10T11:47:46.411 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:46.404 7f3993314700 1 --2- 172.21.2.112:0/1990198123 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f399418bc70 0x7f39940c0650 secure :-1 s=READY pgs=3081 cs=0 l=1 rev1=1 rx=0x7f398c009980 tx=0x7f398c00bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:46.411 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:46.404 7f398b7fe700 1 -- 172.21.2.112:0/1990198123 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f398c005af0 con 0x7f399418bc70 2021-02-10T11:47:46.411 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:46.404 7f398b7fe700 1 -- 172.21.2.112:0/1990198123 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f398c005d70 con 0x7f399418bc70 2021-02-10T11:47:46.411 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:46.404 7f3998f1d700 1 -- 172.21.2.112:0/1990198123 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39940c0c70 con 0x7f399418bc70 2021-02-10T11:47:46.411 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:46.404 7f3998f1d700 1 -- 172.21.2.112:0/1990198123 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39940c1200 con 0x7f399418bc70 2021-02-10T11:47:46.412 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:46.404 7f398b7fe700 1 -- 172.21.2.112:0/1990198123 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f398c006220 con 0x7f399418bc70 2021-02-10T11:47:46.412 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:46.405 7f398b7fe700 1 -- 172.21.2.112:0/1990198123 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f398c006030 con 0x7f399418bc70 2021-02-10T11:47:46.412 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:46.405 7f398b7fe700 1 --2- 172.21.2.112:0/1990198123 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3978018730 0x7f397801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:46.412 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:46.405 7f398b7fe700 1 -- 172.21.2.112:0/1990198123 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f398c032b70 con 0x7f399418bc70 2021-02-10T11:47:46.412 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:46.405 7f3992b13700 1 --2- 172.21.2.112:0/1990198123 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3978018730 0x7f397801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:46.412 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:46.405 7f3992b13700 1 --2- 172.21.2.112:0/1990198123 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3978018730 0x7f397801ac10 crc :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:46.440 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:46.437 7f3998f1d700 1 -- 172.21.2.112:0/1990198123 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39940ee0b0 con 0x7f399418bc70 2021-02-10T11:47:46.442 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:46.440 7f398b7fe700 1 -- 172.21.2.112:0/1990198123 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f398c0064e0 con 0x7f399418bc70 2021-02-10T11:47:46.573 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:46.570 7f3998f1d700 1 -- 172.21.2.112:0/1990198123 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "auth rm", "entity": "client.guest1"} v 0) v1 -- 0x7f39940ce450 con 0x7f399418bc70 2021-02-10T11:47:46.574 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:46.571 7f398b7fe700 1 -- 172.21.2.112:0/1990198123 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth rm", "entity": "client.guest1"}]=0 entity client.guest1 does not exist v109) v1 ==== 117+0+0 (secure 0 0 0) 0x7f398c006030 con 0x7f399418bc70 2021-02-10T11:47:46.574 INFO:teuthology.orchestra.run.gibba012.stderr:entity client.guest1 does not exist 2021-02-10T11:47:46.576 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:46.573 7f3998f1d700 1 -- 172.21.2.112:0/1990198123 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3978018730 msgr2=0x7f397801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:46.576 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:46.573 7f3998f1d700 1 --2- 172.21.2.112:0/1990198123 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3978018730 0x7f397801ac10 crc :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:46.576 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:46.573 7f3998f1d700 1 -- 172.21.2.112:0/1990198123 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f399418bc70 msgr2=0x7f39940c0650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:46.576 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:46.573 7f3998f1d700 1 --2- 172.21.2.112:0/1990198123 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f399418bc70 0x7f39940c0650 secure :-1 s=READY pgs=3081 cs=0 l=1 rev1=1 rx=0x7f398c009980 tx=0x7f398c00bcd0).stop 2021-02-10T11:47:46.576 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:46.573 7f3998f1d700 1 -- 172.21.2.112:0/1990198123 shutdown_connections 2021-02-10T11:47:46.576 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:46.573 7f3998f1d700 1 --2- 172.21.2.112:0/1990198123 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3978018730 0x7f397801ac10 unknown :-1 s=CLOSED pgs=1386 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:46.577 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:46.573 7f3998f1d700 1 --2- 172.21.2.112:0/1990198123 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f399418b870 0x7f39940c0030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:46.577 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:46.573 7f3998f1d700 1 --2- 172.21.2.112:0/1990198123 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f399418bc70 0x7f39940c0650 unknown :-1 s=CLOSED pgs=3081 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:46.577 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:46.573 7f3998f1d700 1 --2- 172.21.2.112:0/1990198123 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f39940bdda0 0x7f399418b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:46.577 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:46.573 7f3998f1d700 1 -- 172.21.2.112:0/1990198123 >> 172.21.2.112:0/1990198123 conn(0x7f39940a6eb0 msgr2=0x7f3994098ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:46.577 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:46.573 7f3998f1d700 1 -- 172.21.2.112:0/1990198123 shutdown_connections 2021-02-10T11:47:46.577 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:46.573 7f3998f1d700 1 -- 172.21.2.112:0/1990198123 wait complete. 2021-02-10T11:47:46.585 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000120407 --group_name group_0000000000232095 2021-02-10T11:47:47.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.000 7ff4b41f0700 1 Processor -- start 2021-02-10T11:47:47.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.000 7ff4b41f0700 1 -- start start 2021-02-10T11:47:47.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.001 7ff4b41f0700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff4ac0d1c80 0x7ff4ac0cef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:47.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.001 7ff4b41f0700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4ac0d0180 con 0x7ff4ac0cf590 2021-02-10T11:47:47.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.001 7ff4b41f0700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4ac0fe1d0 con 0x7ff4ac0d1880 2021-02-10T11:47:47.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.001 7ff4b41f0700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4ac1b0cc0 con 0x7ff4ac0d1c80 2021-02-10T11:47:47.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.001 7ff4b280d700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7ff4ac0cf590 0x7ff4ac0cf990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:60162/0 (socket says 172.21.2.112:60162) 2021-02-10T11:47:47.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.001 7ff4b280d700 1 -- 172.21.2.112:0/3836992659 learned_addr learned my addr 172.21.2.112:0/3836992659 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:47:47.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.001 7ff4b100a700 1 -- 172.21.2.112:0/3836992659 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 391309336 0 0) 0x7ff4a4001500 con 0x7ff4ac0cf590 2021-02-10T11:47:47.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.001 7ff4b180b700 1 --2- 172.21.2.112:0/3836992659 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff4ac0d1c80 0x7ff4ac0cef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:47.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.001 7ff4b100a700 1 -- 172.21.2.112:0/3836992659 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff498002fe0 con 0x7ff4ac0cf590 2021-02-10T11:47:47.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.002 7ff4b100a700 1 -- 172.21.2.112:0/3836992659 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2327961703 0 0) 0x7ff4a4001500 con 0x7ff4ac0cf590 2021-02-10T11:47:47.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.002 7ff4b100a700 1 -- 172.21.2.112:0/3836992659 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff4ac0d1c80 msgr2=0x7ff4ac0cef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:47.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.002 7ff4b100a700 1 --2- 172.21.2.112:0/3836992659 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff4ac0d1c80 0x7ff4ac0cef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:47.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.002 7ff4b100a700 1 -- 172.21.2.112:0/3836992659 >> v1:172.21.2.120:6789/0 conn(0x7ff4ac0d1880 legacy=0x7ff4ac1b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:47.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.002 7ff4b100a700 1 -- 172.21.2.112:0/3836992659 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4ac109a70 con 0x7ff4ac0cf590 2021-02-10T11:47:47.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.002 7ff4b100a700 1 -- 172.21.2.112:0/3836992659 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7ff4a4001a00 con 0x7ff4ac0cf590 2021-02-10T11:47:47.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.002 7ff4b100a700 1 -- 172.21.2.112:0/3836992659 >> v1:172.21.2.112:6789/0 conn(0x7ff4ac0cf590 legacy=0x7ff4ac0cf990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:47.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.002 7ff4b100a700 1 --2- 172.21.2.112:0/3836992659 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff498003a60 0x7ff498005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:47.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.002 7ff4b100a700 1 --2- 172.21.2.112:0/3836992659 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff498006790 0x7ff498008c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:47.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.002 7ff4b100a700 1 --2- 172.21.2.112:0/3836992659 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff498009500 0x7ff49800ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:47.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.002 7ff4b100a700 1 -- 172.21.2.112:0/3836992659 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff49800c2d0 con 0x7ff498009500 2021-02-10T11:47:47.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.002 7ff4b100a700 1 -- 172.21.2.112:0/3836992659 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff49800c4f0 con 0x7ff498003a60 2021-02-10T11:47:47.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.002 7ff4b100a700 1 -- 172.21.2.112:0/3836992659 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff49800c710 con 0x7ff498006790 2021-02-10T11:47:47.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.002 7ff4b280d700 1 --2- 172.21.2.112:0/3836992659 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff498009500 0x7ff49800ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:47.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.002 7ff4b180b700 1 --2- 172.21.2.112:0/3836992659 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff498006790 0x7ff498008c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:47.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.002 7ff4b200c700 1 --2- 172.21.2.112:0/3836992659 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff498003a60 0x7ff498005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:47.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.002 7ff4b280d700 1 -- 172.21.2.112:0/3836992659 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff498006790 msgr2=0x7ff498008c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:47.024 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.002 7ff4b280d700 1 --2- 172.21.2.112:0/3836992659 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff498006790 0x7ff498008c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:47.024 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.002 7ff4b280d700 1 -- 172.21.2.112:0/3836992659 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff498003a60 msgr2=0x7ff498005ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:47.024 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.002 7ff4b280d700 1 --2- 172.21.2.112:0/3836992659 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff498003a60 0x7ff498005ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:47.024 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.002 7ff4b280d700 1 -- 172.21.2.112:0/3836992659 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff498050950 con 0x7ff498009500 2021-02-10T11:47:47.024 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.002 7ff4b280d700 1 --2- 172.21.2.112:0/3836992659 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff498009500 0x7ff49800ba20 secure :-1 s=READY pgs=3083 cs=0 l=1 rev1=1 rx=0x7ff4a40078f0 tx=0x7ff4a400a310).ready entity=mon.0 client_cookie=5b56af1442a77db7 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:47.024 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.003 7ff4b100a700 1 -- 172.21.2.112:0/3836992659 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff4a4007be0 con 0x7ff498009500 2021-02-10T11:47:47.024 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.003 7ff4b100a700 1 -- 172.21.2.112:0/3836992659 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4a40070c0 con 0x7ff498009500 2021-02-10T11:47:47.024 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.003 7ff4b100a700 1 -- 172.21.2.112:0/3836992659 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff4a4007550 con 0x7ff498009500 2021-02-10T11:47:47.024 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.003 7ff4b41f0700 1 -- 172.21.2.112:0/3836992659 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff498009500 msgr2=0x7ff49800ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:47.024 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.003 7ff4b41f0700 1 --2- 172.21.2.112:0/3836992659 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff498009500 0x7ff49800ba20 secure :-1 s=READY pgs=3083 cs=0 l=1 rev1=1 rx=0x7ff4a40078f0 tx=0x7ff4a400a310).stop 2021-02-10T11:47:47.025 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.003 7ff4b280d700 1 -- 172.21.2.112:0/3836992659 reap_dead start 2021-02-10T11:47:47.025 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.003 7ff4b41f0700 1 -- 172.21.2.112:0/3836992659 shutdown_connections 2021-02-10T11:47:47.025 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.003 7ff4b41f0700 1 -- 172.21.2.112:0/3836992659 >> 172.21.2.112:0/3836992659 conn(0x7ff4ac0ce430 msgr2=0x7ff4ac0d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:47.025 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.004 7ff4b41f0700 1 -- 172.21.2.112:0/3836992659 shutdown_connections 2021-02-10T11:47:47.025 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.004 7ff4b41f0700 1 -- 172.21.2.112:0/3836992659 wait complete. 2021-02-10T11:47:47.025 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.018 7ff4b41f0700 1 Processor -- start 2021-02-10T11:47:47.025 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.018 7ff4b41f0700 1 -- start start 2021-02-10T11:47:47.026 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.019 7ff4b41f0700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff4ac1a3570 0x7ff4ac1a3970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:47.027 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.019 7ff4b41f0700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff4ac09c9f0 0x7ff4ac09cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:47.027 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.019 7ff4b41f0700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff4ac09d410 0x7ff4ac098170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:47.027 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.019 7ff4b41f0700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4ac09c470 con 0x7ff4ac09c9f0 2021-02-10T11:47:47.027 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.019 7ff4b41f0700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4ac0b2840 con 0x7ff4ac09d410 2021-02-10T11:47:47.027 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.019 7ff4b41f0700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4ac0b2a20 con 0x7ff4ac1a3570 2021-02-10T11:47:47.027 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.019 7ff4b180b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff4ac09c9f0 0x7ff4ac09cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:47.027 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.019 7ff4b180b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff4ac09c9f0 0x7ff4ac09cdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:60710/0 (socket says 172.21.2.112:60710) 2021-02-10T11:47:47.027 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.019 7ff4b180b700 1 -- 172.21.2.112:0/3212464268 learned_addr learned my addr 172.21.2.112:0/3212464268 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:47:47.028 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.019 7ff4b180b700 1 -- 172.21.2.112:0/3212464268 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff4ac1a3570 msgr2=0x7ff4ac1a3970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:47.028 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.019 7ff4b180b700 1 --2- 172.21.2.112:0/3212464268 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff4ac1a3570 0x7ff4ac1a3970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:47.028 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.019 7ff4b180b700 1 -- 172.21.2.112:0/3212464268 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff4ac09d410 msgr2=0x7ff4ac098170 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:47:47.028 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.019 7ff4b180b700 1 --2- 172.21.2.112:0/3212464268 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff4ac09d410 0x7ff4ac098170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:47.028 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.019 7ff4b180b700 1 -- 172.21.2.112:0/3212464268 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4ac09d880 con 0x7ff4ac09c9f0 2021-02-10T11:47:47.028 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.019 7ff4b180b700 1 --2- 172.21.2.112:0/3212464268 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff4ac09c9f0 0x7ff4ac09cdf0 secure :-1 s=READY pgs=3084 cs=0 l=1 rev1=1 rx=0x7ff4a0006f60 tx=0x7ff4a000b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:47.028 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.019 7ff49effd700 1 -- 172.21.2.112:0/3212464268 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff4a000bd60 con 0x7ff4ac09c9f0 2021-02-10T11:47:47.028 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.019 7ff49effd700 1 -- 172.21.2.112:0/3212464268 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4a0000c90 con 0x7ff4ac09c9f0 2021-02-10T11:47:47.028 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.019 7ff49effd700 1 -- 172.21.2.112:0/3212464268 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff4a0002220 con 0x7ff4ac09c9f0 2021-02-10T11:47:47.028 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.019 7ff4b41f0700 1 -- 172.21.2.112:0/3212464268 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4ac09d880 con 0x7ff4ac09c9f0 2021-02-10T11:47:47.028 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.019 7ff4b41f0700 1 -- 172.21.2.112:0/3212464268 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4ac098a80 con 0x7ff4ac09c9f0 2021-02-10T11:47:47.028 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.020 7ff49effd700 1 -- 172.21.2.112:0/3212464268 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff4a0006b00 con 0x7ff4ac09c9f0 2021-02-10T11:47:47.028 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.020 7ff49effd700 1 --2- 172.21.2.112:0/3212464268 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff4940186c0 0x7ff49401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:47.029 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.020 7ff49effd700 1 -- 172.21.2.112:0/3212464268 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 5573+0+0 (secure 0 0 0) 0x7ff4a0032520 con 0x7ff4ac09c9f0 2021-02-10T11:47:47.029 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.020 7ff4b200c700 1 --2- 172.21.2.112:0/3212464268 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff4940186c0 0x7ff49401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:47.029 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.021 7ff4b200c700 1 --2- 172.21.2.112:0/3212464268 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff4940186c0 0x7ff49401abc0 crc :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:47.053 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.050 7ff4b41f0700 1 -- 172.21.2.112:0/3212464268 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4ac082a70 con 0x7ff4ac09c9f0 2021-02-10T11:47:47.056 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.053 7ff49effd700 1 -- 172.21.2.112:0/3212464268 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff4a0000940 con 0x7ff4ac09c9f0 2021-02-10T11:47:47.182 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.179 7ff4b41f0700 1 -- 172.21.2.112:0/3212464268 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000120407", "target": ["mgr", ""], "group_name": "group_0000000000232095"}) v1 -- 0x7ff4ac09c470 con 0x7ff4940186c0 2021-02-10T11:47:47.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.189 7ff49effd700 1 -- 172.21.2.112:0/3212464268 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7ff4a8000de0 con 0x7ff4940186c0 2021-02-10T11:47:47.200 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.197 7ff4b41f0700 1 -- 172.21.2.112:0/3212464268 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff4940186c0 msgr2=0x7ff49401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:47.201 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.197 7ff4b41f0700 1 --2- 172.21.2.112:0/3212464268 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff4940186c0 0x7ff49401abc0 crc :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:47.201 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.197 7ff4b41f0700 1 -- 172.21.2.112:0/3212464268 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff4ac09c9f0 msgr2=0x7ff4ac09cdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:47.201 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.197 7ff4b41f0700 1 --2- 172.21.2.112:0/3212464268 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff4ac09c9f0 0x7ff4ac09cdf0 secure :-1 s=READY pgs=3084 cs=0 l=1 rev1=1 rx=0x7ff4a0006f60 tx=0x7ff4a000b970).stop 2021-02-10T11:47:47.201 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.197 7ff4b41f0700 1 -- 172.21.2.112:0/3212464268 shutdown_connections 2021-02-10T11:47:47.201 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.197 7ff4b41f0700 1 --2- 172.21.2.112:0/3212464268 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff4940186c0 0x7ff49401abc0 unknown :-1 s=CLOSED pgs=1387 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:47.201 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.197 7ff4b41f0700 1 --2- 172.21.2.112:0/3212464268 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff4ac09d410 0x7ff4ac098170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:47.201 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.197 7ff4b41f0700 1 --2- 172.21.2.112:0/3212464268 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff4ac09c9f0 0x7ff4ac09cdf0 unknown :-1 s=CLOSED pgs=3084 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:47.201 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.197 7ff4b41f0700 1 --2- 172.21.2.112:0/3212464268 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff4ac1a3570 0x7ff4ac1a3970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:47.201 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.197 7ff4b41f0700 1 -- 172.21.2.112:0/3212464268 >> 172.21.2.112:0/3212464268 conn(0x7ff4ac0c2cd0 msgr2=0x7ff4ac0b5950 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:47.201 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.198 7ff4b41f0700 1 -- 172.21.2.112:0/3212464268 shutdown_connections 2021-02-10T11:47:47.201 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.198 7ff4b41f0700 1 -- 172.21.2.112:0/3212464268 wait complete. 2021-02-10T11:47:47.209 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000120408 --group_name group_0000000000232095 2021-02-10T11:47:47.641 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.621 7f7068465700 1 Processor -- start 2021-02-10T11:47:47.641 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.621 7f7068465700 1 -- start start 2021-02-10T11:47:47.641 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.621 7f7068465700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f706010b000 0x7f70600cf760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:47.641 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.621 7f7068465700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70600fe1d0 con 0x7f70600d1880 2021-02-10T11:47:47.641 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.621 7f7068465700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70601b0cc0 con 0x7f70600d1c80 2021-02-10T11:47:47.641 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.621 7f7068465700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7060109a70 con 0x7f706010b000 2021-02-10T11:47:47.641 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.621 7f7066a82700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f706010b000 0x7f70600cf760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:47.641 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.621 7f7066a82700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f706010b000 0x7f70600cf760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:60720/0 (socket says 172.21.2.112:60720) 2021-02-10T11:47:47.641 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.621 7f7066a82700 1 -- 172.21.2.112:0/1861859652 learned_addr learned my addr 172.21.2.112:0/1861859652 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:47:47.641 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.622 7f7066a82700 1 -- 172.21.2.112:0/1861859652 >> v1:172.21.2.120:6789/0 conn(0x7f70600d1c80 legacy=0x7f70600cef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:47.642 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.622 7f7066a82700 1 -- 172.21.2.112:0/1861859652 >> v1:172.21.2.112:6789/0 conn(0x7f70600d1880 legacy=0x7f70601b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:47.642 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.622 7f7066a82700 1 -- 172.21.2.112:0/1861859652 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f706010b400 con 0x7f706010b000 2021-02-10T11:47:47.642 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.622 7f7066a82700 1 --2- 172.21.2.112:0/1861859652 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f706010b000 0x7f70600cf760 secure :-1 s=READY pgs=3086 cs=0 l=1 rev1=1 rx=0x7f7058006f00 tx=0x7f7058009e90).ready entity=mon.0 client_cookie=ed02083b8cd098f2 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:47.642 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.622 7f706527f700 1 -- 172.21.2.112:0/1861859652 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f705800b300 con 0x7f706010b000 2021-02-10T11:47:47.642 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.622 7f706527f700 1 -- 172.21.2.112:0/1861859652 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f705800b640 con 0x7f706010b000 2021-02-10T11:47:47.642 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.622 7f706527f700 1 -- 172.21.2.112:0/1861859652 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f705800ba90 con 0x7f706010b000 2021-02-10T11:47:47.642 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.622 7f7068465700 1 -- 172.21.2.112:0/1861859652 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f706010b000 msgr2=0x7f70600cf760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:47.642 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.622 7f7068465700 1 --2- 172.21.2.112:0/1861859652 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f706010b000 0x7f70600cf760 secure :-1 s=READY pgs=3086 cs=0 l=1 rev1=1 rx=0x7f7058006f00 tx=0x7f7058009e90).stop 2021-02-10T11:47:47.642 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.623 7f7068465700 1 -- 172.21.2.112:0/1861859652 shutdown_connections 2021-02-10T11:47:47.642 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.623 7f7068465700 1 --2- 172.21.2.112:0/1861859652 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f706010b000 0x7f70600cf760 unknown :-1 s=CLOSED pgs=3086 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:47.642 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.623 7f7068465700 1 -- 172.21.2.112:0/1861859652 >> 172.21.2.112:0/1861859652 conn(0x7f70600ce430 msgr2=0x7f70600d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:47.642 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.623 7f7068465700 1 -- 172.21.2.112:0/1861859652 shutdown_connections 2021-02-10T11:47:47.642 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.623 7f7068465700 1 -- 172.21.2.112:0/1861859652 wait complete. 2021-02-10T11:47:47.643 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.638 7f7068465700 1 Processor -- start 2021-02-10T11:47:47.643 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.638 7f7068465700 1 -- start start 2021-02-10T11:47:47.644 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.638 7f7068465700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7060189e90 0x7f70600b3120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:47.644 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.638 7f7068465700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7060107730 0x7f7060107b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:47.644 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.638 7f7068465700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7060108150 0x7f706018a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:47.644 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.638 7f7068465700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f706018a4c0 con 0x7f7060107730 2021-02-10T11:47:47.644 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.638 7f7068465700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70600b3780 con 0x7f7060189e90 2021-02-10T11:47:47.644 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.638 7f7068465700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70600bd100 con 0x7f7060108150 2021-02-10T11:47:47.645 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.638 7f7065a80700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7060107730 0x7f7060107b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:47.645 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.638 7f7065a80700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7060107730 0x7f7060107b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:60726/0 (socket says 172.21.2.112:60726) 2021-02-10T11:47:47.645 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.638 7f7065a80700 1 -- 172.21.2.112:0/4170746504 learned_addr learned my addr 172.21.2.112:0/4170746504 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:47:47.646 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.639 7f7066281700 1 --2- 172.21.2.112:0/4170746504 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7060189e90 0x7f70600b3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:47.646 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.639 7f7066a82700 1 --2- 172.21.2.112:0/4170746504 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7060108150 0x7f706018a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:47.646 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.639 7f7065a80700 1 -- 172.21.2.112:0/4170746504 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7060108150 msgr2=0x7f706018a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:47.646 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.639 7f7065a80700 1 --2- 172.21.2.112:0/4170746504 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7060108150 0x7f706018a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:47.646 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.639 7f7065a80700 1 -- 172.21.2.112:0/4170746504 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7060189e90 msgr2=0x7f70600b3120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:47.646 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.639 7f7065a80700 1 --2- 172.21.2.112:0/4170746504 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7060189e90 0x7f70600b3120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:47.646 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.639 7f7065a80700 1 -- 172.21.2.112:0/4170746504 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7060108550 con 0x7f7060107730 2021-02-10T11:47:47.646 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.639 7f7066281700 1 --2- 172.21.2.112:0/4170746504 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7060189e90 0x7f70600b3120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:47:47.646 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.639 7f7066a82700 1 --2- 172.21.2.112:0/4170746504 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7060108150 0x7f706018a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:47:47.646 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.639 7f7065a80700 1 --2- 172.21.2.112:0/4170746504 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7060107730 0x7f7060107b30 secure :-1 s=READY pgs=3087 cs=0 l=1 rev1=1 rx=0x7f705400b5b0 tx=0x7f705400b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:47.646 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.639 7f7052ffd700 1 -- 172.21.2.112:0/4170746504 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f705400bd40 con 0x7f7060107730 2021-02-10T11:47:47.646 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.639 7f7052ffd700 1 -- 172.21.2.112:0/4170746504 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70540009b0 con 0x7f7060107730 2021-02-10T11:47:47.646 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.639 7f7052ffd700 1 -- 172.21.2.112:0/4170746504 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f70540062a0 con 0x7f7060107730 2021-02-10T11:47:47.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.639 7f7068465700 1 -- 172.21.2.112:0/4170746504 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7060108550 con 0x7f7060107730 2021-02-10T11:47:47.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.640 7f7068465700 1 -- 172.21.2.112:0/4170746504 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7060108550 con 0x7f7060107730 2021-02-10T11:47:47.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.640 7f7052ffd700 1 -- 172.21.2.112:0/4170746504 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7054006020 con 0x7f7060107730 2021-02-10T11:47:47.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.640 7f7052ffd700 1 --2- 172.21.2.112:0/4170746504 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f70480186c0 0x7f704801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:47.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.640 7f7052ffd700 1 -- 172.21.2.112:0/4170746504 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f70540327b0 con 0x7f7060107730 2021-02-10T11:47:47.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.641 7f7066281700 1 --2- 172.21.2.112:0/4170746504 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f70480186c0 0x7f704801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:47.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.641 7f7066281700 1 --2- 172.21.2.112:0/4170746504 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f70480186c0 0x7f704801abc0 crc :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:47.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.671 7f7068465700 1 -- 172.21.2.112:0/4170746504 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7060080340 con 0x7f7060107730 2021-02-10T11:47:47.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.673 7f7052ffd700 1 -- 172.21.2.112:0/4170746504 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f705400bd40 con 0x7f7060107730 2021-02-10T11:47:47.801 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.798 7f7068465700 1 -- 172.21.2.112:0/4170746504 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000120408", "target": ["mgr", ""], "group_name": "group_0000000000232095"}) v1 -- 0x7f7060080340 con 0x7f70480186c0 2021-02-10T11:47:47.810 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.808 7f7052ffd700 1 -- 172.21.2.112:0/4170746504 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f705c006ac0 con 0x7f70480186c0 2021-02-10T11:47:47.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.816 7f7068465700 1 -- 172.21.2.112:0/4170746504 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f70480186c0 msgr2=0x7f704801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:47.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.816 7f7068465700 1 --2- 172.21.2.112:0/4170746504 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f70480186c0 0x7f704801abc0 crc :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:47.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.816 7f7068465700 1 -- 172.21.2.112:0/4170746504 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7060107730 msgr2=0x7f7060107b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:47.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.816 7f7068465700 1 --2- 172.21.2.112:0/4170746504 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7060107730 0x7f7060107b30 secure :-1 s=READY pgs=3087 cs=0 l=1 rev1=1 rx=0x7f705400b5b0 tx=0x7f705400b950).stop 2021-02-10T11:47:47.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.817 7f7068465700 1 -- 172.21.2.112:0/4170746504 shutdown_connections 2021-02-10T11:47:47.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.817 7f7068465700 1 --2- 172.21.2.112:0/4170746504 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f70480186c0 0x7f704801abc0 unknown :-1 s=CLOSED pgs=1388 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:47.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.817 7f7068465700 1 --2- 172.21.2.112:0/4170746504 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f7060108150 0x7f706018a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:47.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.817 7f7068465700 1 --2- 172.21.2.112:0/4170746504 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7060107730 0x7f7060107b30 unknown :-1 s=CLOSED pgs=3087 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:47.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.817 7f7068465700 1 --2- 172.21.2.112:0/4170746504 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f7060189e90 0x7f70600b3120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:47.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.817 7f7068465700 1 -- 172.21.2.112:0/4170746504 >> 172.21.2.112:0/4170746504 conn(0x7f706019afc0 msgr2=0x7f70600b3b70 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:47.821 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.817 7f7068465700 1 -- 172.21.2.112:0/4170746504 shutdown_connections 2021-02-10T11:47:47.821 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:47.817 7f7068465700 1 -- 172.21.2.112:0/4170746504 wait complete. 2021-02-10T11:47:47.828 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolumegroup rm cephfs group_0000000000232095 2021-02-10T11:47:48.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:48.241 7f2ba0c2e700 1 Processor -- start 2021-02-10T11:47:48.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:48.241 7f2ba0c2e700 1 -- start start 2021-02-10T11:47:48.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:48.242 7f2ba0c2e700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f2b9c0d1870 0x7f2b9c1b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:48.262 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:48.242 7f2ba0c2e700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f2b9c0cff70 0x7f2b9c109130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:48.262 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:48.242 7f2ba0c2e700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b9c0fe1c0 con 0x7f2b9c1b0ae0 2021-02-10T11:47:48.262 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:48.242 7f2ba0c2e700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b9c109ad0 con 0x7f2b9c0cff70 2021-02-10T11:47:48.262 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:48.242 7f2ba0c2e700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b9c109750 con 0x7f2b9c0d1870 2021-02-10T11:47:48.262 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:48.242 7f2b9b05f700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f2b9c0cff70 0x7f2b9c109130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:48.262 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:48.242 7f2b9b05f700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f2b9c0cff70 0x7f2b9c109130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:60730/0 (socket says 172.21.2.112:60730) 2021-02-10T11:47:48.262 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:48.242 7f2b9b05f700 1 -- 172.21.2.112:0/806596294 learned_addr learned my addr 172.21.2.112:0/806596294 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:47:48.262 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:48.242 7f2b9a05d700 1 --1- 172.21.2.112:0/806596294 >> v1:172.21.2.120:6789/0 conn(0x7f2b9c1b0ae0 0x7f2b9c108940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:53750/0 (socket says 172.21.2.112:53750) 2021-02-10T11:47:48.262 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:48.242 7f2b9a85e700 1 --2- 172.21.2.112:0/806596294 >> v2:172.21.2.120:3300/0 conn(0x7f2b9c0d1870 0x7f2b9c1b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:48.262 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:48.242 7f2b9b05f700 1 -- 172.21.2.112:0/806596294 >> v2:172.21.2.120:3300/0 conn(0x7f2b9c0d1870 msgr2=0x7f2b9c1b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:48.262 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:48.242 7f2b9b05f700 1 --2- 172.21.2.112:0/806596294 >> v2:172.21.2.120:3300/0 conn(0x7f2b9c0d1870 0x7f2b9c1b04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:48.262 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:48.242 7f2b9b05f700 1 -- 172.21.2.112:0/806596294 >> v1:172.21.2.120:6789/0 conn(0x7f2b9c1b0ae0 legacy=0x7f2b9c108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:48.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:48.242 7f2b9b05f700 1 -- 172.21.2.112:0/806596294 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b9c088070 con 0x7f2b9c0cff70 2021-02-10T11:47:48.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:48.242 7f2b9b05f700 1 --2- 172.21.2.112:0/806596294 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2b9c0cff70 0x7f2b9c109130 secure :-1 s=READY pgs=3088 cs=0 l=1 rev1=1 rx=0x7f2b94006f00 tx=0x7f2b94009f10).ready entity=mon.0 client_cookie=2497529182fefb64 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:48.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:48.243 7f2b9985c700 1 -- 172.21.2.112:0/806596294 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2b9400b3b0 con 0x7f2b9c0cff70 2021-02-10T11:47:48.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:48.243 7f2b9985c700 1 -- 172.21.2.112:0/806596294 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b9400b730 con 0x7f2b9c0cff70 2021-02-10T11:47:48.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:48.243 7f2b9985c700 1 -- 172.21.2.112:0/806596294 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2b9400bc50 con 0x7f2b9c0cff70 2021-02-10T11:47:48.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:48.243 7f2ba0c2e700 1 -- 172.21.2.112:0/806596294 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2b9c0cff70 msgr2=0x7f2b9c109130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:48.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:48.243 7f2ba0c2e700 1 --2- 172.21.2.112:0/806596294 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2b9c0cff70 0x7f2b9c109130 secure :-1 s=READY pgs=3088 cs=0 l=1 rev1=1 rx=0x7f2b94006f00 tx=0x7f2b94009f10).stop 2021-02-10T11:47:48.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:48.244 7f2ba0c2e700 1 -- 172.21.2.112:0/806596294 shutdown_connections 2021-02-10T11:47:48.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:48.244 7f2ba0c2e700 1 --2- 172.21.2.112:0/806596294 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2b9c0cff70 0x7f2b9c109130 unknown :-1 s=CLOSED pgs=3088 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:48.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:48.244 7f2ba0c2e700 1 --2- 172.21.2.112:0/806596294 >> v2:172.21.2.120:3300/0 conn(0x7f2b9c0d1870 0x7f2b9c1b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:48.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:48.244 7f2ba0c2e700 1 -- 172.21.2.112:0/806596294 >> 172.21.2.112:0/806596294 conn(0x7f2b9c0ce420 msgr2=0x7f2b9c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:48.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:48.244 7f2ba0c2e700 1 -- 172.21.2.112:0/806596294 shutdown_connections 2021-02-10T11:47:48.263 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:48.244 7f2ba0c2e700 1 -- 172.21.2.112:0/806596294 wait complete. 2021-02-10T11:47:48.265 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:48.259 7f2ba0c2e700 1 Processor -- start 2021-02-10T11:47:48.265 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:48.259 7f2ba0c2e700 1 -- start start 2021-02-10T11:47:48.265 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:48.259 7f2ba0c2e700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2b9c191b10 0x7f2b9c0ae640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:48.265 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:48.259 7f2ba0c2e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2b9c191f10 0x7f2b9c19ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:48.265 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:48.259 7f2ba0c2e700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2b9c1a05f0 0x7f2b9c082ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:48.265 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:48.259 7f2ba0c2e700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b9c192440 con 0x7f2b9c191f10 2021-02-10T11:47:48.265 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:48.259 7f2ba0c2e700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b9c1916e0 con 0x7f2b9c1a05f0 2021-02-10T11:47:48.265 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:48.259 7f2ba0c2e700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b9c191900 con 0x7f2b9c191b10 2021-02-10T11:47:48.265 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:48.259 7f2b9b05f700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2b9c1a05f0 0x7f2b9c082ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:48.266 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:48.259 7f2b9a05d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2b9c191f10 0x7f2b9c19ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:48.266 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:48.259 7f2b9b05f700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2b9c1a05f0 0x7f2b9c082ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3300/0 says I am v2:172.21.2.112:34782/0 (socket says 172.21.2.112:34782) 2021-02-10T11:47:48.266 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:48.259 7f2b9b05f700 1 -- 172.21.2.112:0/65960468 learned_addr learned my addr 172.21.2.112:0/65960468 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:47:48.266 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:48.259 7f2b9a85e700 1 --2- 172.21.2.112:0/65960468 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2b9c191b10 0x7f2b9c0ae640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:48.266 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:48.260 7f2b9a05d700 1 -- 172.21.2.112:0/65960468 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2b9c191b10 msgr2=0x7f2b9c0ae640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:48.266 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:48.260 7f2b9a05d700 1 --2- 172.21.2.112:0/65960468 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2b9c191b10 0x7f2b9c0ae640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:48.266 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:48.260 7f2b9a05d700 1 -- 172.21.2.112:0/65960468 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2b9c1a05f0 msgr2=0x7f2b9c082ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:48.266 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:48.260 7f2b9a05d700 1 --2- 172.21.2.112:0/65960468 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2b9c1a05f0 0x7f2b9c082ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:48.266 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:48.260 7f2b9a05d700 1 -- 172.21.2.112:0/65960468 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b9c0834f0 con 0x7f2b9c191f10 2021-02-10T11:47:48.266 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:48.260 7f2b9a85e700 1 --2- 172.21.2.112:0/65960468 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2b9c191b10 0x7f2b9c0ae640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:47:48.266 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:48.260 7f2b9a05d700 1 --2- 172.21.2.112:0/65960468 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2b9c191f10 0x7f2b9c19ffd0 secure :-1 s=READY pgs=3089 cs=0 l=1 rev1=1 rx=0x7f2b9000a4d0 tx=0x7f2b9000a870).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:48.266 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:48.260 7f2b8b7fe700 1 -- 172.21.2.112:0/65960468 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2b9000ac60 con 0x7f2b9c191f10 2021-02-10T11:47:48.266 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:48.260 7f2b8b7fe700 1 -- 172.21.2.112:0/65960468 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b90004140 con 0x7f2b9c191f10 2021-02-10T11:47:48.267 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:48.260 7f2b8b7fe700 1 -- 172.21.2.112:0/65960468 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2b90004660 con 0x7f2b9c191f10 2021-02-10T11:47:48.267 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:48.260 7f2ba0c2e700 1 -- 172.21.2.112:0/65960468 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b9c0834f0 con 0x7f2b9c191f10 2021-02-10T11:47:48.267 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:48.260 7f2ba0c2e700 1 -- 172.21.2.112:0/65960468 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b9c0834f0 con 0x7f2b9c191f10 2021-02-10T11:47:48.267 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:48.261 7f2b8b7fe700 1 -- 172.21.2.112:0/65960468 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2b90000ce0 con 0x7f2b9c191f10 2021-02-10T11:47:48.267 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:48.261 7f2b8b7fe700 1 --2- 172.21.2.112:0/65960468 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2b800186c0 0x7f2b8001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:48.267 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:48.261 7f2b8b7fe700 1 -- 172.21.2.112:0/65960468 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f2b90032460 con 0x7f2b9c191f10 2021-02-10T11:47:48.267 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:48.261 7f2b9a85e700 1 --2- 172.21.2.112:0/65960468 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2b800186c0 0x7f2b8001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:48.267 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:48.262 7f2b9a85e700 1 --2- 172.21.2.112:0/65960468 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2b800186c0 0x7f2b8001abc0 crc :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:48.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:48.291 7f2ba0c2e700 1 -- 172.21.2.112:0/65960468 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b9c191900 con 0x7f2b9c191f10 2021-02-10T11:47:48.297 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:48.294 7f2b8b7fe700 1 -- 172.21.2.112:0/65960468 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2b900009b0 con 0x7f2b9c191f10 2021-02-10T11:47:48.425 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:48.422 7f2ba0c2e700 1 -- 172.21.2.112:0/65960468 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolumegroup rm", "vol_name": "cephfs", "target": ["mgr", ""], "group_name": "group_0000000000232095"}) v1 -- 0x7f2b9c1916e0 con 0x7f2b800186c0 2021-02-10T11:47:48.428 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:48.425 7f2b8b7fe700 1 -- 172.21.2.112:0/65960468 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f2b8c0009b0 con 0x7f2b800186c0 2021-02-10T11:47:48.434 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:48.431 7f2ba0c2e700 1 -- 172.21.2.112:0/65960468 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2b800186c0 msgr2=0x7f2b8001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:48.435 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:48.431 7f2ba0c2e700 1 --2- 172.21.2.112:0/65960468 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2b800186c0 0x7f2b8001abc0 crc :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:48.435 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:48.431 7f2ba0c2e700 1 -- 172.21.2.112:0/65960468 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2b9c191f10 msgr2=0x7f2b9c19ffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:48.435 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:48.431 7f2ba0c2e700 1 --2- 172.21.2.112:0/65960468 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2b9c191f10 0x7f2b9c19ffd0 secure :-1 s=READY pgs=3089 cs=0 l=1 rev1=1 rx=0x7f2b9000a4d0 tx=0x7f2b9000a870).stop 2021-02-10T11:47:48.435 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:48.432 7f2ba0c2e700 1 -- 172.21.2.112:0/65960468 shutdown_connections 2021-02-10T11:47:48.435 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:48.432 7f2ba0c2e700 1 --2- 172.21.2.112:0/65960468 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2b800186c0 0x7f2b8001abc0 unknown :-1 s=CLOSED pgs=1389 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:48.435 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:48.432 7f2ba0c2e700 1 --2- 172.21.2.112:0/65960468 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2b9c1a05f0 0x7f2b9c082ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:48.435 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:48.432 7f2ba0c2e700 1 --2- 172.21.2.112:0/65960468 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2b9c191f10 0x7f2b9c19ffd0 unknown :-1 s=CLOSED pgs=3089 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:48.435 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:48.432 7f2ba0c2e700 1 --2- 172.21.2.112:0/65960468 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2b9c191b10 0x7f2b9c0ae640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:48.435 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:48.432 7f2ba0c2e700 1 -- 172.21.2.112:0/65960468 >> 172.21.2.112:0/65960468 conn(0x7f2b9c0a02d0 msgr2=0x7f2b9c09f200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:48.436 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:48.432 7f2ba0c2e700 1 -- 172.21.2.112:0/65960468 shutdown_connections 2021-02-10T11:47:48.436 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:48.432 7f2ba0c2e700 1 -- 172.21.2.112:0/65960468 wait complete. 2021-02-10T11:47:48.443 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:47:48.443 DEBUG:teuthology.orchestra.run.gibba012:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:47:48.790 DEBUG:teuthology.orchestra.run.gibba020:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:47:49.134 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:47:49.134 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:47:49.135 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:47:49.135 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:47:49.472 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:47:49.472 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:47:49.472 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba012.front.sepia.ceph.com... 2021-02-10T11:47:49.472 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:47:49.472 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:47:49.826 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba012.stderr:ceph-fuse[188076]: fuse finished with error 0 and tester_r 0 2021-02-10T11:47:49.827 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:47:49.828 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:47:49.828 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:47:50.163 INFO:teuthology.orchestra.run.gibba012.stdout:ext2/ext3 2021-02-10T11:47:50.164 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:47:50.164 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:47:50.164 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:47:50.499 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:47:50.500 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:47:50.500 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:47:50.833 INFO:teuthology.orchestra.run.gibba012.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: No such file or directory 2021-02-10T11:47:50.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:47:50.834 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2021-02-10T11:47:50.834 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:47:50.834 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:47:51.169 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:47:51.170 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:47:51.170 DEBUG:teuthology.orchestra.run.gibba020:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:47:51.505 INFO:teuthology.orchestra.run.gibba020.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:47:51.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:47:51.506 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:47:51.506 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:47:51.506 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:47:51.841 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:47:51.841 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:47:51.841 DEBUG:teuthology.orchestra.run.gibba020:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:47:52.175 INFO:teuthology.orchestra.run.gibba020.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:47:52.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:47:52.175 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:47:52.175 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:47:52.175 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:47:52.509 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config rm mon mon_allow_pool_delete 2021-02-10T11:47:52.941 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:52.922 7fe64e39d700 1 Processor -- start 2021-02-10T11:47:52.941 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:52.922 7fe64e39d700 1 -- start start 2021-02-10T11:47:52.941 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:52.922 7fe64e39d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe6480d1870 0x7fe6481b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:52.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:52.922 7fe64e39d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fe6481b0a10 0x7fe6480cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:52.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:52.922 7fe64e39d700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6480f0b10 con 0x7fe6480d1c70 2021-02-10T11:47:52.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:52.922 7fe64e39d700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6480fe230 con 0x7fe6481b0a10 2021-02-10T11:47:52.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:52.922 7fe64e39d700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6480e5ed0 con 0x7fe6480d1870 2021-02-10T11:47:52.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:52.923 7fe6477fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fe6481b0a10 0x7fe6480cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:52.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:52.923 7fe6477fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fe6481b0a10 0x7fe6480cf330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:60740/0 (socket says 172.21.2.112:60740) 2021-02-10T11:47:52.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:52.923 7fe64c9ba700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fe6480d1c70 0x7fe6480cf950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:60202/0 (socket says 172.21.2.112:60202) 2021-02-10T11:47:52.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:52.923 7fe6477fe700 1 -- 172.21.2.112:0/2882724761 learned_addr learned my addr 172.21.2.112:0/2882724761 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:47:52.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:52.923 7fe6477fe700 1 -- 172.21.2.112:0/2882724761 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe6480d1870 msgr2=0x7fe6481b03f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:47:52.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:52.923 7fe6477fe700 1 --2- 172.21.2.112:0/2882724761 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe6480d1870 0x7fe6481b03f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:52.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:52.923 7fe6477fe700 1 -- 172.21.2.112:0/2882724761 >> v1:172.21.2.112:6789/0 conn(0x7fe6480d1c70 legacy=0x7fe6480cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:52.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:52.923 7fe6477fe700 1 -- 172.21.2.112:0/2882724761 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6480f0650 con 0x7fe6481b0a10 2021-02-10T11:47:52.943 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:52.923 7fe6477fe700 1 --2- 172.21.2.112:0/2882724761 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe6481b0a10 0x7fe6480cf330 secure :-1 s=READY pgs=3091 cs=0 l=1 rev1=1 rx=0x7fe63c006700 tx=0x7fe63c009b80).ready entity=mon.0 client_cookie=1c1a73c4a8e3a46d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:52.943 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:52.923 7fe646ffd700 1 -- 172.21.2.112:0/2882724761 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe63c00c0c0 con 0x7fe6481b0a10 2021-02-10T11:47:52.943 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:52.923 7fe646ffd700 1 -- 172.21.2.112:0/2882724761 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe63c00c380 con 0x7fe6481b0a10 2021-02-10T11:47:52.943 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:52.923 7fe646ffd700 1 -- 172.21.2.112:0/2882724761 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe63c00c7e0 con 0x7fe6481b0a10 2021-02-10T11:47:52.943 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:52.923 7fe64e39d700 1 -- 172.21.2.112:0/2882724761 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe6481b0a10 msgr2=0x7fe6480cf330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:52.943 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:52.923 7fe64e39d700 1 --2- 172.21.2.112:0/2882724761 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe6481b0a10 0x7fe6480cf330 secure :-1 s=READY pgs=3091 cs=0 l=1 rev1=1 rx=0x7fe63c006700 tx=0x7fe63c009b80).stop 2021-02-10T11:47:52.943 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:52.924 7fe64e39d700 1 -- 172.21.2.112:0/2882724761 shutdown_connections 2021-02-10T11:47:52.943 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:52.924 7fe64e39d700 1 --2- 172.21.2.112:0/2882724761 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe6481b0a10 0x7fe6480cf330 unknown :-1 s=CLOSED pgs=3091 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:52.943 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:52.924 7fe64e39d700 1 --2- 172.21.2.112:0/2882724761 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe6480d1870 0x7fe6481b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:52.943 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:52.924 7fe64e39d700 1 -- 172.21.2.112:0/2882724761 >> 172.21.2.112:0/2882724761 conn(0x7fe6480ce450 msgr2=0x7fe6480d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:52.943 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:52.924 7fe64e39d700 1 -- 172.21.2.112:0/2882724761 shutdown_connections 2021-02-10T11:47:52.943 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:52.924 7fe64e39d700 1 -- 172.21.2.112:0/2882724761 wait complete. 2021-02-10T11:47:52.945 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:52.939 7fe64e39d700 1 Processor -- start 2021-02-10T11:47:52.945 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:52.939 7fe64e39d700 1 -- start start 2021-02-10T11:47:52.945 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:52.939 7fe64e39d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe6480ee060 0x7fe6480ade80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:52.945 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:52.939 7fe64e39d700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe6480ee460 0x7fe6481a7b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:52.945 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:52.939 7fe64e39d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe6481a8180 0x7fe648094860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:52.945 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:52.939 7fe64e39d700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6480ae4a0 con 0x7fe6481a8180 2021-02-10T11:47:52.945 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:52.939 7fe64e39d700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe648094e80 con 0x7fe6480ee460 2021-02-10T11:47:52.945 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:52.939 7fe64e39d700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6480950a0 con 0x7fe6480ee060 2021-02-10T11:47:52.945 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:52.940 7fe64c9ba700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe6481a8180 0x7fe648094860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:52.945 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:52.940 7fe64c9ba700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe6481a8180 0x7fe648094860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:60748/0 (socket says 172.21.2.112:60748) 2021-02-10T11:47:52.945 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:52.940 7fe64c9ba700 1 -- 172.21.2.112:0/1511675868 learned_addr learned my addr 172.21.2.112:0/1511675868 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:47:52.946 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:52.940 7fe6477fe700 1 --2- 172.21.2.112:0/1511675868 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe6480ee460 0x7fe6481a7b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:52.946 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:52.940 7fe64c9ba700 1 -- 172.21.2.112:0/1511675868 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe6480ee060 msgr2=0x7fe6480ade80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:52.946 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:52.940 7fe647fff700 1 --2- 172.21.2.112:0/1511675868 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe6480ee060 0x7fe6480ade80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:52.946 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:52.940 7fe64c9ba700 1 --2- 172.21.2.112:0/1511675868 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe6480ee060 0x7fe6480ade80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:52.946 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:52.940 7fe64c9ba700 1 -- 172.21.2.112:0/1511675868 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe6480ee460 msgr2=0x7fe6481a7b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:52.946 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:52.940 7fe64c9ba700 1 --2- 172.21.2.112:0/1511675868 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe6480ee460 0x7fe6481a7b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:52.946 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:52.940 7fe64c9ba700 1 -- 172.21.2.112:0/1511675868 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6480952c0 con 0x7fe6481a8180 2021-02-10T11:47:52.946 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:52.940 7fe6477fe700 1 --2- 172.21.2.112:0/1511675868 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe6480ee460 0x7fe6481a7b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:47:52.946 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:52.940 7fe64c9ba700 1 --2- 172.21.2.112:0/1511675868 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe6481a8180 0x7fe648094860 secure :-1 s=READY pgs=3092 cs=0 l=1 rev1=1 rx=0x7fe640007f90 tx=0x7fe64000ac50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:52.946 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:52.940 7fe644ff9700 1 -- 172.21.2.112:0/1511675868 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe640000ec0 con 0x7fe6481a8180 2021-02-10T11:47:52.946 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:52.940 7fe644ff9700 1 -- 172.21.2.112:0/1511675868 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6400070b0 con 0x7fe6481a8180 2021-02-10T11:47:52.946 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:52.940 7fe644ff9700 1 -- 172.21.2.112:0/1511675868 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe640007540 con 0x7fe6481a8180 2021-02-10T11:47:52.946 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:52.941 7fe64e39d700 1 -- 172.21.2.112:0/1511675868 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6480952c0 con 0x7fe6481a8180 2021-02-10T11:47:52.947 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:52.941 7fe64e39d700 1 -- 172.21.2.112:0/1511675868 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe648092f20 con 0x7fe6481a8180 2021-02-10T11:47:52.947 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:52.941 7fe644ff9700 1 -- 172.21.2.112:0/1511675868 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe640007290 con 0x7fe6481a8180 2021-02-10T11:47:52.947 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:52.941 7fe644ff9700 1 --2- 172.21.2.112:0/1511675868 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe6280186c0 0x7fe62801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:52.947 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:52.941 7fe644ff9700 1 -- 172.21.2.112:0/1511675868 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fe6400328b0 con 0x7fe6481a8180 2021-02-10T11:47:52.947 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:52.942 7fe647fff700 1 --2- 172.21.2.112:0/1511675868 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe6280186c0 0x7fe62801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:52.947 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:52.942 7fe647fff700 1 --2- 172.21.2.112:0/1511675868 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe6280186c0 0x7fe62801abc0 crc :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:52.975 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:52.972 7fe64e39d700 1 -- 172.21.2.112:0/1511675868 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe648094330 con 0x7fe6481a8180 2021-02-10T11:47:52.978 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:52.975 7fe644ff9700 1 -- 172.21.2.112:0/1511675868 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe640001270 con 0x7fe6481a8180 2021-02-10T11:47:53.126 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:53.123 7fe64e39d700 1 -- 172.21.2.112:0/1511675868 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fe648094330 con 0x7fe6481a8180 2021-02-10T11:47:53.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:53.127 7fe644ff9700 1 -- 172.21.2.112:0/1511675868 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v45) v1 ==== 104+0+0 (secure 0 0 0) 0x7fe640007290 con 0x7fe6481a8180 2021-02-10T11:47:53.135 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:53.132 7fe64e39d700 1 -- 172.21.2.112:0/1511675868 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe6280186c0 msgr2=0x7fe62801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:53.136 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:53.132 7fe64e39d700 1 --2- 172.21.2.112:0/1511675868 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe6280186c0 0x7fe62801abc0 crc :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:53.136 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:53.132 7fe64e39d700 1 -- 172.21.2.112:0/1511675868 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe6481a8180 msgr2=0x7fe648094860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:53.136 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:53.132 7fe64e39d700 1 --2- 172.21.2.112:0/1511675868 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe6481a8180 0x7fe648094860 secure :-1 s=READY pgs=3092 cs=0 l=1 rev1=1 rx=0x7fe640007f90 tx=0x7fe64000ac50).stop 2021-02-10T11:47:53.136 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:53.133 7fe64e39d700 1 -- 172.21.2.112:0/1511675868 shutdown_connections 2021-02-10T11:47:53.136 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:53.133 7fe64e39d700 1 --2- 172.21.2.112:0/1511675868 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe6280186c0 0x7fe62801abc0 unknown :-1 s=CLOSED pgs=1390 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:53.136 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:53.133 7fe64e39d700 1 --2- 172.21.2.112:0/1511675868 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe6480ee460 0x7fe6481a7b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:53.136 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:53.133 7fe64e39d700 1 --2- 172.21.2.112:0/1511675868 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe6481a8180 0x7fe648094860 unknown :-1 s=CLOSED pgs=3092 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:53.136 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:53.133 7fe64e39d700 1 --2- 172.21.2.112:0/1511675868 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe6480ee060 0x7fe6480ade80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:53.136 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:53.133 7fe64e39d700 1 -- 172.21.2.112:0/1511675868 >> 172.21.2.112:0/1511675868 conn(0x7fe6480955e0 msgr2=0x7fe6480bfa80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:53.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:53.133 7fe64e39d700 1 -- 172.21.2.112:0/1511675868 shutdown_connections 2021-02-10T11:47:53.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:53.133 7fe64e39d700 1 -- 172.21.2.112:0/1511675868 wait complete. 2021-02-10T11:47:53.144 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestVolumes.test_recover_auth_metadata_during_deauthorize' 2021-02-10T11:47:53.574 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:53.553 7fab5d6ea700 1 Processor -- start 2021-02-10T11:47:53.574 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:53.553 7fab5d6ea700 1 -- start start 2021-02-10T11:47:53.574 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:53.553 7fab5d6ea700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fab580d1c70 0x7fab580f20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:53.574 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:53.553 7fab5d6ea700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab580f0b10 con 0x7fab580d0340 2021-02-10T11:47:53.574 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:53.553 7fab5d6ea700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab581b0be0 con 0x7fab580d1870 2021-02-10T11:47:53.574 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:53.553 7fab5d6ea700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab580e5ed0 con 0x7fab580d1c70 2021-02-10T11:47:53.575 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:53.553 7fab56abe700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fab580d0340 0x7fab580ef640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:60214/0 (socket says 172.21.2.112:60214) 2021-02-10T11:47:53.575 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:53.553 7fab56abe700 1 -- 172.21.2.112:0/112671263 learned_addr learned my addr 172.21.2.112:0/112671263 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:47:53.575 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:53.554 7fab562bd700 1 -- 172.21.2.112:0/112671263 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1048593167 0 0) 0x7fab48001180 con 0x7fab580d0340 2021-02-10T11:47:53.575 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:53.554 7fab562bd700 1 -- 172.21.2.112:0/112671263 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab44002fe0 con 0x7fab580d0340 2021-02-10T11:47:53.575 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:53.554 7fab57ac0700 1 --2- 172.21.2.112:0/112671263 >> v2:172.21.2.120:3300/0 conn(0x7fab580d1c70 0x7fab580f20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:53.575 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:53.554 7fab562bd700 1 -- 172.21.2.112:0/112671263 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3813869509 0 0) 0x7fab48001180 con 0x7fab580d0340 2021-02-10T11:47:53.575 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:53.554 7fab562bd700 1 -- 172.21.2.112:0/112671263 >> v2:172.21.2.120:3300/0 conn(0x7fab580d1c70 msgr2=0x7fab580f20a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:53.575 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:53.554 7fab562bd700 1 --2- 172.21.2.112:0/112671263 >> v2:172.21.2.120:3300/0 conn(0x7fab580d1c70 0x7fab580f20a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:53.575 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:53.554 7fab562bd700 1 -- 172.21.2.112:0/112671263 >> v1:172.21.2.120:6789/0 conn(0x7fab580d1870 legacy=0x7fab581b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:53.575 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:53.554 7fab562bd700 1 -- 172.21.2.112:0/112671263 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab580efe30 con 0x7fab580d0340 2021-02-10T11:47:53.575 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:53.554 7fab562bd700 1 -- 172.21.2.112:0/112671263 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7fab48001680 con 0x7fab580d0340 2021-02-10T11:47:53.575 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:53.554 7fab562bd700 1 -- 172.21.2.112:0/112671263 >> v1:172.21.2.112:6789/0 conn(0x7fab580d0340 legacy=0x7fab580ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:53.575 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:53.554 7fab562bd700 1 --2- 172.21.2.112:0/112671263 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fab44003ac0 0x7fab44005ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:53.576 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:53.554 7fab562bd700 1 --2- 172.21.2.112:0/112671263 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fab44006770 0x7fab44008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:53.576 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:53.554 7fab562bd700 1 --2- 172.21.2.112:0/112671263 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fab44009590 0x7fab4400bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:53.576 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:53.554 7fab562bd700 1 -- 172.21.2.112:0/112671263 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab4400c300 con 0x7fab44006770 2021-02-10T11:47:53.576 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:53.554 7fab562bd700 1 -- 172.21.2.112:0/112671263 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab4400c520 con 0x7fab44009590 2021-02-10T11:47:53.576 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:53.554 7fab562bd700 1 -- 172.21.2.112:0/112671263 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab4400c740 con 0x7fab44003ac0 2021-02-10T11:47:53.576 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:53.554 7fab56abe700 1 --2- 172.21.2.112:0/112671263 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fab44006770 0x7fab44008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:53.576 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:53.555 7fab572bf700 1 --2- 172.21.2.112:0/112671263 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fab44003ac0 0x7fab44005ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:53.576 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:53.555 7fab57ac0700 1 --2- 172.21.2.112:0/112671263 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fab44009590 0x7fab4400bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:53.576 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:53.555 7fab56abe700 1 -- 172.21.2.112:0/112671263 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fab44003ac0 msgr2=0x7fab44005ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:53.576 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:53.555 7fab56abe700 1 --2- 172.21.2.112:0/112671263 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fab44003ac0 0x7fab44005ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:53.576 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:53.555 7fab56abe700 1 -- 172.21.2.112:0/112671263 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fab44009590 msgr2=0x7fab4400bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:53.576 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:53.555 7fab56abe700 1 --2- 172.21.2.112:0/112671263 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fab44009590 0x7fab4400bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:53.576 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:53.555 7fab56abe700 1 -- 172.21.2.112:0/112671263 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab4400c9e0 con 0x7fab44006770 2021-02-10T11:47:53.577 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:53.555 7fab572bf700 1 --2- 172.21.2.112:0/112671263 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fab44003ac0 0x7fab44005ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:47:53.577 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:53.555 7fab57ac0700 1 -- 172.21.2.112:0/112671263 reap_dead start 2021-02-10T11:47:53.577 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:53.555 7fab56abe700 1 --2- 172.21.2.112:0/112671263 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fab44006770 0x7fab44008c90 secure :-1 s=READY pgs=3094 cs=0 l=1 rev1=1 rx=0x7fab48001d90 tx=0x7fab48007cf0).ready entity=mon.0 client_cookie=ec2482a7f16d80ec server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:53.577 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:53.555 7fab562bd700 1 -- 172.21.2.112:0/112671263 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fab48000c90 con 0x7fab44006770 2021-02-10T11:47:53.577 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:53.555 7fab562bd700 1 -- 172.21.2.112:0/112671263 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab480070d0 con 0x7fab44006770 2021-02-10T11:47:53.577 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:53.555 7fab562bd700 1 -- 172.21.2.112:0/112671263 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fab48004620 con 0x7fab44006770 2021-02-10T11:47:53.577 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:53.555 7fab5d6ea700 1 -- 172.21.2.112:0/112671263 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fab44006770 msgr2=0x7fab44008c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:53.577 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:53.555 7fab5d6ea700 1 --2- 172.21.2.112:0/112671263 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fab44006770 0x7fab44008c90 secure :-1 s=READY pgs=3094 cs=0 l=1 rev1=1 rx=0x7fab48001d90 tx=0x7fab48007cf0).stop 2021-02-10T11:47:53.577 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:53.556 7fab5d6ea700 1 -- 172.21.2.112:0/112671263 shutdown_connections 2021-02-10T11:47:53.577 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:53.556 7fab5d6ea700 1 --2- 172.21.2.112:0/112671263 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fab44006770 0x7fab44008c90 unknown :-1 s=CLOSED pgs=3094 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:53.577 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:53.556 7fab5d6ea700 1 -- 172.21.2.112:0/112671263 >> 172.21.2.112:0/112671263 conn(0x7fab580ce450 msgr2=0x7fab580d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:53.577 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:53.556 7fab5d6ea700 1 -- 172.21.2.112:0/112671263 shutdown_connections 2021-02-10T11:47:53.577 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:53.556 7fab5d6ea700 1 -- 172.21.2.112:0/112671263 wait complete. 2021-02-10T11:47:53.578 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:53.570 7fab5d6ea700 1 Processor -- start 2021-02-10T11:47:53.578 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:53.570 7fab5d6ea700 1 -- start start 2021-02-10T11:47:53.578 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:53.571 7fab5d6ea700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fab5808de50 0x7fab580b0c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:53.578 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:53.571 7fab5d6ea700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fab5808ce90 0x7fab5808d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:53.578 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:53.571 7fab5d6ea700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fab5808d8b0 0x7fab5818c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:53.578 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:53.571 7fab5d6ea700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab5818ccb0 con 0x7fab5808ce90 2021-02-10T11:47:53.578 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:53.571 7fab5d6ea700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab5818cea0 con 0x7fab5808d8b0 2021-02-10T11:47:53.578 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:53.571 7fab5d6ea700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab5818d0c0 con 0x7fab5808de50 2021-02-10T11:47:53.578 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:53.571 7fab56abe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fab5808ce90 0x7fab5808d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:53.578 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:53.571 7fab56abe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fab5808ce90 0x7fab5808d290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:60764/0 (socket says 172.21.2.112:60764) 2021-02-10T11:47:53.579 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:53.571 7fab56abe700 1 -- 172.21.2.112:0/2535723852 learned_addr learned my addr 172.21.2.112:0/2535723852 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:47:53.580 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:53.571 7fab56abe700 1 -- 172.21.2.112:0/2535723852 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fab5808de50 msgr2=0x7fab580b0c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:53.580 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:53.571 7fab56abe700 1 --2- 172.21.2.112:0/2535723852 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fab5808de50 0x7fab580b0c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:53.580 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:53.571 7fab56abe700 1 -- 172.21.2.112:0/2535723852 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fab5808d8b0 msgr2=0x7fab5818c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:53.580 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:53.571 7fab56abe700 1 --2- 172.21.2.112:0/2535723852 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fab5808d8b0 0x7fab5818c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:53.580 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:53.571 7fab56abe700 1 -- 172.21.2.112:0/2535723852 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab5818d2e0 con 0x7fab5808ce90 2021-02-10T11:47:53.580 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:53.572 7fab56abe700 1 --2- 172.21.2.112:0/2535723852 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fab5808ce90 0x7fab5808d290 secure :-1 s=READY pgs=3095 cs=0 l=1 rev1=1 rx=0x7fab4804f520 tx=0x7fab4804f8c0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:53.580 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:53.572 7fab4ffff700 1 -- 172.21.2.112:0/2535723852 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fab48001080 con 0x7fab5808ce90 2021-02-10T11:47:53.581 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:53.572 7fab4ffff700 1 -- 172.21.2.112:0/2535723852 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab4804fcb0 con 0x7fab5808ce90 2021-02-10T11:47:53.581 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:53.572 7fab5d6ea700 1 -- 172.21.2.112:0/2535723852 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab5818d2e0 con 0x7fab5808ce90 2021-02-10T11:47:53.581 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:53.572 7fab4ffff700 1 -- 172.21.2.112:0/2535723852 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fab480502f0 con 0x7fab5808ce90 2021-02-10T11:47:53.581 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:53.572 7fab5d6ea700 1 -- 172.21.2.112:0/2535723852 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab5818d2e0 con 0x7fab5808ce90 2021-02-10T11:47:53.581 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:53.572 7fab4ffff700 1 -- 172.21.2.112:0/2535723852 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fab4804c050 con 0x7fab5808ce90 2021-02-10T11:47:53.581 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:53.573 7fab4ffff700 1 --2- 172.21.2.112:0/2535723852 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fab3c018730 0x7fab3c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:53.581 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:53.573 7fab4ffff700 1 -- 172.21.2.112:0/2535723852 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fab48076610 con 0x7fab5808ce90 2021-02-10T11:47:53.581 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:53.573 7fab572bf700 1 --2- 172.21.2.112:0/2535723852 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fab3c018730 0x7fab3c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:53.581 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:53.574 7fab572bf700 1 --2- 172.21.2.112:0/2535723852 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fab3c018730 0x7fab3c01abc0 crc :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:53.606 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:53.603 7fab5d6ea700 1 -- 172.21.2.112:0/2535723852 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab580f0b10 con 0x7fab5808ce90 2021-02-10T11:47:53.609 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:53.606 7fab4ffff700 1 -- 172.21.2.112:0/2535723852 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fab48000da0 con 0x7fab5808ce90 2021-02-10T11:47:53.741 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:53.738 7fab5d6ea700 1 -- 172.21.2.112:0/2535723852 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_recover_auth_metadata_during_deauthorize"]} v 0) v1 -- 0x7fab580f0b10 con 0x7fab5808ce90 2021-02-10T11:47:54.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:54.301 7fab4ffff700 1 -- 172.21.2.112:0/2535723852 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_recover_auth_metadata_during_deauthorize"]}]=0 v855) v1 ==== 162+0+0 (secure 0 0 0) 0x7fab4804cef0 con 0x7fab5808ce90 2021-02-10T11:47:54.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:54.307 7fab5d6ea700 1 -- 172.21.2.112:0/2535723852 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fab3c018730 msgr2=0x7fab3c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:54.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:54.307 7fab5d6ea700 1 --2- 172.21.2.112:0/2535723852 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fab3c018730 0x7fab3c01abc0 crc :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:54.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:54.307 7fab5d6ea700 1 -- 172.21.2.112:0/2535723852 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fab5808ce90 msgr2=0x7fab5808d290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:54.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:54.307 7fab5d6ea700 1 --2- 172.21.2.112:0/2535723852 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fab5808ce90 0x7fab5808d290 secure :-1 s=READY pgs=3095 cs=0 l=1 rev1=1 rx=0x7fab4804f520 tx=0x7fab4804f8c0).stop 2021-02-10T11:47:54.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:54.307 7fab5d6ea700 1 -- 172.21.2.112:0/2535723852 shutdown_connections 2021-02-10T11:47:54.311 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:54.307 7fab5d6ea700 1 --2- 172.21.2.112:0/2535723852 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fab3c018730 0x7fab3c01abc0 unknown :-1 s=CLOSED pgs=1391 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:54.311 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:54.307 7fab5d6ea700 1 --2- 172.21.2.112:0/2535723852 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fab5808d8b0 0x7fab5818c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:54.311 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:54.307 7fab5d6ea700 1 --2- 172.21.2.112:0/2535723852 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fab5808ce90 0x7fab5808d290 unknown :-1 s=CLOSED pgs=3095 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:54.311 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:54.307 7fab5d6ea700 1 --2- 172.21.2.112:0/2535723852 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fab5808de50 0x7fab580b0c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:54.311 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:54.307 7fab5d6ea700 1 -- 172.21.2.112:0/2535723852 >> 172.21.2.112:0/2535723852 conn(0x7fab5809eb60 msgr2=0x7fab580a67b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:54.311 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:54.308 7fab5d6ea700 1 -- 172.21.2.112:0/2535723852 shutdown_connections 2021-02-10T11:47:54.311 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:54.308 7fab5d6ea700 1 -- 172.21.2.112:0/2535723852 wait complete. 2021-02-10T11:47:54.319 INFO:tasks.cephfs_test_runner:test_recover_auth_metadata_during_deauthorize (tasks.cephfs.test_volumes.TestVolumes) ... ok 2021-02-10T11:47:54.319 INFO:tasks.cephfs_test_runner:Starting test: test_subvolume_authorized_list (tasks.cephfs.test_volumes.TestVolumes) 2021-02-10T11:47:54.319 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestVolumes.test_subvolume_authorized_list' 2021-02-10T11:47:54.749 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:54.728 7f676cb78700 1 Processor -- start 2021-02-10T11:47:54.749 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:54.728 7f676cb78700 1 -- start start 2021-02-10T11:47:54.749 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:54.729 7f676cb78700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f67680d1870 0x7f67681b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:54.749 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:54.729 7f676cb78700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f67681b0a10 0x7f67680cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:54.749 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:54.729 7f676cb78700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67680f0b10 con 0x7f67680d1c70 2021-02-10T11:47:54.750 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:54.729 7f676cb78700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67680fe230 con 0x7f67681b0a10 2021-02-10T11:47:54.750 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:54.729 7f676cb78700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67680e5ed0 con 0x7f67680d1870 2021-02-10T11:47:54.750 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:54.730 7f676705f700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7f67680d1c70 0x7f67680cf950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:53792/0 (socket says 172.21.2.112:53792) 2021-02-10T11:47:54.750 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:54.730 7f676705f700 1 -- 172.21.2.112:0/2656205565 learned_addr learned my addr 172.21.2.112:0/2656205565 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:47:54.750 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:54.730 7f676605d700 1 --2- 172.21.2.112:0/2656205565 >> v2:172.21.2.120:3300/0 conn(0x7f67681b0a10 0x7f67680cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:54.750 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:54.730 7f676685e700 1 --2- 172.21.2.112:0/2656205565 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f67680d1870 0x7f67681b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:54.750 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:54.730 7f676585c700 1 -- 172.21.2.112:0/2656205565 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1956826127 0 0) 0x7f6760001510 con 0x7f67680d1c70 2021-02-10T11:47:54.750 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:54.730 7f676585c700 1 -- 172.21.2.112:0/2656205565 --> v1:172.21.2.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6750002fe0 con 0x7f67680d1c70 2021-02-10T11:47:54.750 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:54.730 7f676605d700 1 -- 172.21.2.112:0/2656205565 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f67680d1870 msgr2=0x7f67681b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:54.750 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:54.730 7f676605d700 1 --2- 172.21.2.112:0/2656205565 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f67680d1870 0x7f67681b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:54.750 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:54.730 7f676605d700 1 -- 172.21.2.112:0/2656205565 >> v1:172.21.2.120:6789/0 conn(0x7f67680d1c70 legacy=0x7f67680cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:54.750 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:54.730 7f676605d700 1 -- 172.21.2.112:0/2656205565 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67680f0650 con 0x7f67681b0a10 2021-02-10T11:47:54.751 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:54.730 7f676685e700 1 --2- 172.21.2.112:0/2656205565 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f67680d1870 0x7f67681b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:47:54.751 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:54.731 7f676605d700 1 --2- 172.21.2.112:0/2656205565 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f67681b0a10 0x7f67680cf330 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 rx=0x7f675c006650 tx=0x7f675c009a80).ready entity=mon.1 client_cookie=dc63cb8ec60676e4 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:54.751 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:54.731 7f676585c700 1 -- 172.21.2.112:0/2656205565 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f675c006390 con 0x7f67681b0a10 2021-02-10T11:47:54.751 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:54.731 7f676585c700 1 -- 172.21.2.112:0/2656205565 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f675c0009b0 con 0x7f67681b0a10 2021-02-10T11:47:54.751 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:54.731 7f676585c700 1 -- 172.21.2.112:0/2656205565 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f675c00c2b0 con 0x7f67681b0a10 2021-02-10T11:47:54.751 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:54.731 7f676cb78700 1 -- 172.21.2.112:0/2656205565 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f67681b0a10 msgr2=0x7f67680cf330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:54.751 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:54.731 7f676cb78700 1 --2- 172.21.2.112:0/2656205565 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f67681b0a10 0x7f67680cf330 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 rx=0x7f675c006650 tx=0x7f675c009a80).stop 2021-02-10T11:47:54.751 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:54.732 7f676cb78700 1 -- 172.21.2.112:0/2656205565 shutdown_connections 2021-02-10T11:47:54.751 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:54.732 7f676cb78700 1 --2- 172.21.2.112:0/2656205565 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f67681b0a10 0x7f67680cf330 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:54.751 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:54.732 7f676cb78700 1 --2- 172.21.2.112:0/2656205565 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f67680d1870 0x7f67681b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:54.751 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:54.732 7f676cb78700 1 -- 172.21.2.112:0/2656205565 >> 172.21.2.112:0/2656205565 conn(0x7f67680ce450 msgr2=0x7f67680d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:54.751 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:54.732 7f676cb78700 1 -- 172.21.2.112:0/2656205565 shutdown_connections 2021-02-10T11:47:54.751 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:54.732 7f676cb78700 1 -- 172.21.2.112:0/2656205565 wait complete. 2021-02-10T11:47:54.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:54.747 7f676cb78700 1 Processor -- start 2021-02-10T11:47:54.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:54.747 7f676cb78700 1 -- start start 2021-02-10T11:47:54.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:54.747 7f676cb78700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f67680c33c0 0x7f67680ec230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:54.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:54.747 7f676cb78700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f67680b5e30 0x7f676819ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:54.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:54.747 7f676cb78700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f676819f260 0x7f67680ed010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:54.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:54.747 7f676cb78700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67680c3810 con 0x7f676819f260 2021-02-10T11:47:54.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:54.747 7f676cb78700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67680eca90 con 0x7f67680b5e30 2021-02-10T11:47:54.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:54.747 7f676cb78700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67680b2790 con 0x7f67680c33c0 2021-02-10T11:47:54.754 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:54.747 7f676705f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f676819f260 0x7f67680ed010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:54.754 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:54.747 7f676705f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f676819f260 0x7f67680ed010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:60774/0 (socket says 172.21.2.112:60774) 2021-02-10T11:47:54.754 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:54.747 7f676705f700 1 -- 172.21.2.112:0/570325221 learned_addr learned my addr 172.21.2.112:0/570325221 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:47:54.754 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:54.747 7f676605d700 1 --2- 172.21.2.112:0/570325221 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f67680b5e30 0x7f676819ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:54.754 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:54.747 7f676685e700 1 --2- 172.21.2.112:0/570325221 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f67680c33c0 0x7f67680ec230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:54.754 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:54.748 7f676685e700 1 -- 172.21.2.112:0/570325221 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f67680b5e30 msgr2=0x7f676819ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:54.754 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:54.748 7f676685e700 1 --2- 172.21.2.112:0/570325221 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f67680b5e30 0x7f676819ec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:54.754 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:54.748 7f676685e700 1 -- 172.21.2.112:0/570325221 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f676819f260 msgr2=0x7f67680ed010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:54.754 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:54.748 7f676685e700 1 --2- 172.21.2.112:0/570325221 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f676819f260 0x7f67680ed010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:54.754 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:54.748 7f676685e700 1 -- 172.21.2.112:0/570325221 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f676819f660 con 0x7f67680c33c0 2021-02-10T11:47:54.754 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:54.748 7f676705f700 1 --2- 172.21.2.112:0/570325221 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f676819f260 0x7f67680ed010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:47:54.754 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:54.748 7f676685e700 1 --2- 172.21.2.112:0/570325221 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f67680c33c0 0x7f67680ec230 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7f6758009540 tx=0x7f675800b8e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:54.754 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:54.748 7f67577fe700 1 -- 172.21.2.112:0/570325221 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f675800bcd0 con 0x7f67680c33c0 2021-02-10T11:47:54.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:54.748 7f67577fe700 1 -- 172.21.2.112:0/570325221 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67580063e0 con 0x7f67680c33c0 2021-02-10T11:47:54.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:54.748 7f67577fe700 1 -- 172.21.2.112:0/570325221 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6758002030 con 0x7f67680c33c0 2021-02-10T11:47:54.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:54.748 7f676cb78700 1 -- 172.21.2.112:0/570325221 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f676819f660 con 0x7f67680c33c0 2021-02-10T11:47:54.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:54.748 7f676cb78700 1 -- 172.21.2.112:0/570325221 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f676819f660 con 0x7f67680c33c0 2021-02-10T11:47:54.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:54.749 7f67577fe700 1 -- 172.21.2.112:0/570325221 <== mon.2 v2:172.21.2.120:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6758009fe0 con 0x7f67680c33c0 2021-02-10T11:47:54.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:54.749 7f67577fe700 1 --2- 172.21.2.112:0/570325221 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f674c0186c0 0x7f674c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:54.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:54.749 7f67577fe700 1 -- 172.21.2.112:0/570325221 <== mon.2 v2:172.21.2.120:3301/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f6758032480 con 0x7f67680c33c0 2021-02-10T11:47:54.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:54.749 7f676605d700 1 --2- 172.21.2.112:0/570325221 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f674c0186c0 0x7f674c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:54.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:54.750 7f676605d700 1 --2- 172.21.2.112:0/570325221 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f674c0186c0 0x7f674c01abc0 crc :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:54.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:54.779 7f676cb78700 1 -- 172.21.2.112:0/570325221 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67680ec850 con 0x7f67680c33c0 2021-02-10T11:47:54.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:54.781 7f67577fe700 1 -- 172.21.2.112:0/570325221 <== mon.2 v2:172.21.2.120:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6758029470 con 0x7f67680c33c0 2021-02-10T11:47:54.914 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:54.911 7f676cb78700 1 -- 172.21.2.112:0/570325221 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_subvolume_authorized_list"]} v 0) v1 -- 0x7f67680ec850 con 0x7f67680c33c0 2021-02-10T11:47:55.308 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:55.305 7f67577fe700 1 -- 172.21.2.112:0/570325221 <== mon.2 v2:172.21.2.120:3301/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_subvolume_authorized_list"]}]=0 v856) v1 ==== 150+0+0 (secure 0 0 0) 0x7f6758029470 con 0x7f67680c33c0 2021-02-10T11:47:55.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:55.330 7f676cb78700 1 -- 172.21.2.112:0/570325221 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f674c0186c0 msgr2=0x7f674c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:55.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:55.330 7f676cb78700 1 --2- 172.21.2.112:0/570325221 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f674c0186c0 0x7f674c01abc0 crc :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:55.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:55.330 7f676cb78700 1 -- 172.21.2.112:0/570325221 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f67680c33c0 msgr2=0x7f67680ec230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:55.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:55.330 7f676cb78700 1 --2- 172.21.2.112:0/570325221 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f67680c33c0 0x7f67680ec230 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7f6758009540 tx=0x7f675800b8e0).stop 2021-02-10T11:47:55.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:55.330 7f676cb78700 1 -- 172.21.2.112:0/570325221 shutdown_connections 2021-02-10T11:47:55.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:55.330 7f676cb78700 1 --2- 172.21.2.112:0/570325221 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f674c0186c0 0x7f674c01abc0 unknown :-1 s=CLOSED pgs=1392 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:55.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:55.330 7f676cb78700 1 --2- 172.21.2.112:0/570325221 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f67680b5e30 0x7f676819ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:55.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:55.330 7f676cb78700 1 --2- 172.21.2.112:0/570325221 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f676819f260 0x7f67680ed010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:55.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:55.330 7f676cb78700 1 --2- 172.21.2.112:0/570325221 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f67680c33c0 0x7f67680ec230 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:55.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:55.330 7f676cb78700 1 -- 172.21.2.112:0/570325221 >> 172.21.2.112:0/570325221 conn(0x7f67680ebe30 msgr2=0x7f676808a880 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:55.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:55.330 7f676cb78700 1 -- 172.21.2.112:0/570325221 shutdown_connections 2021-02-10T11:47:55.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:55.330 7f676cb78700 1 -- 172.21.2.112:0/570325221 wait complete. 2021-02-10T11:47:55.341 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:47:55.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:55.753 7f3bf5791700 1 Processor -- start 2021-02-10T11:47:55.774 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:55.753 7f3bf5791700 1 -- start start 2021-02-10T11:47:55.774 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:55.753 7f3bf5791700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f3bf00d1c60 0x7f3bf00cef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:55.774 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:55.754 7f3bf5791700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bf00d0160 con 0x7f3bf00cf570 2021-02-10T11:47:55.774 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:55.754 7f3bf5791700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bf00fe1b0 con 0x7f3bf00d1860 2021-02-10T11:47:55.774 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:55.754 7f3bf5791700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3bf01b0ca0 con 0x7f3bf00d1c60 2021-02-10T11:47:55.774 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:55.754 7f3befd75700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f3bf00cf570 0x7f3bf00cf970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:60248/0 (socket says 172.21.2.112:60248) 2021-02-10T11:47:55.774 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:55.754 7f3befd75700 1 -- 172.21.2.112:0/1947351984 learned_addr learned my addr 172.21.2.112:0/1947351984 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:47:55.774 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:55.754 7f3bee572700 1 -- 172.21.2.112:0/1947351984 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3458607422 0 0) 0x7f3be8001500 con 0x7f3bf00cf570 2021-02-10T11:47:55.774 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:55.754 7f3bee572700 1 -- 172.21.2.112:0/1947351984 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3bdc002fe0 con 0x7f3bf00cf570 2021-02-10T11:47:55.774 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:55.754 7f3bee572700 1 -- 172.21.2.112:0/1947351984 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 396572654 0 0) 0x7f3be8001460 con 0x7f3bf00cf570 2021-02-10T11:47:55.774 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:55.754 7f3beed73700 1 --2- 172.21.2.112:0/1947351984 >> v2:172.21.2.120:3300/0 conn(0x7f3bf00d1c60 0x7f3bf00cef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:55.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:55.754 7f3bee572700 1 -- 172.21.2.112:0/1947351984 >> v2:172.21.2.120:3300/0 conn(0x7f3bf00d1c60 msgr2=0x7f3bf00cef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:55.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:55.754 7f3bee572700 1 --2- 172.21.2.112:0/1947351984 >> v2:172.21.2.120:3300/0 conn(0x7f3bf00d1c60 0x7f3bf00cef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:55.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:55.755 7f3bee572700 1 -- 172.21.2.112:0/1947351984 >> v1:172.21.2.120:6789/0 conn(0x7f3bf00d1860 legacy=0x7f3bf01b04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:55.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:55.755 7f3bee572700 1 -- 172.21.2.112:0/1947351984 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bf0109a50 con 0x7f3bf00cf570 2021-02-10T11:47:55.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:55.755 7f3bee572700 1 -- 172.21.2.112:0/1947351984 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f3be8001980 con 0x7f3bf00cf570 2021-02-10T11:47:55.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:55.755 7f3bee572700 1 -- 172.21.2.112:0/1947351984 >> v1:172.21.2.112:6789/0 conn(0x7f3bf00cf570 legacy=0x7f3bf00cf970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:55.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:55.755 7f3bee572700 1 --2- 172.21.2.112:0/1947351984 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3bdc003b20 0x7f3bdc005f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:55.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:55.755 7f3bee572700 1 --2- 172.21.2.112:0/1947351984 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3bdc006810 0x7f3bdc008d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:55.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:55.755 7f3bee572700 1 --2- 172.21.2.112:0/1947351984 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3bdc009630 0x7f3bdc00bb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:55.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:55.755 7f3bee572700 1 -- 172.21.2.112:0/1947351984 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bdc00c3a0 con 0x7f3bdc006810 2021-02-10T11:47:55.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:55.755 7f3bee572700 1 -- 172.21.2.112:0/1947351984 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bdc00c5c0 con 0x7f3bdc009630 2021-02-10T11:47:55.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:55.755 7f3bee572700 1 -- 172.21.2.112:0/1947351984 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bdc00c7e0 con 0x7f3bdc003b20 2021-02-10T11:47:55.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:55.755 7f3beed73700 1 --2- 172.21.2.112:0/1947351984 >> v2:172.21.2.120:3300/0 conn(0x7f3bf00d1c60 0x7f3bf00cef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:47:55.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:55.755 7f3beed73700 1 --2- 172.21.2.112:0/1947351984 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3bdc006810 0x7f3bdc008d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:55.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:55.755 7f3bef574700 1 --2- 172.21.2.112:0/1947351984 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3bdc003b20 0x7f3bdc005f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:55.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:55.755 7f3befd75700 1 --2- 172.21.2.112:0/1947351984 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3bdc009630 0x7f3bdc00bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:55.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:55.755 7f3beed73700 1 -- 172.21.2.112:0/1947351984 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3bdc003b20 msgr2=0x7f3bdc005f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:55.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:55.755 7f3beed73700 1 --2- 172.21.2.112:0/1947351984 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3bdc003b20 0x7f3bdc005f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:55.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:55.755 7f3beed73700 1 -- 172.21.2.112:0/1947351984 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3bdc009630 msgr2=0x7f3bdc00bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:55.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:55.755 7f3beed73700 1 --2- 172.21.2.112:0/1947351984 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3bdc009630 0x7f3bdc00bb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:55.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:55.755 7f3beed73700 1 -- 172.21.2.112:0/1947351984 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bdc00ca80 con 0x7f3bdc006810 2021-02-10T11:47:55.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:55.755 7f3befd75700 1 -- 172.21.2.112:0/1947351984 reap_dead start 2021-02-10T11:47:55.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:55.755 7f3beed73700 1 --2- 172.21.2.112:0/1947351984 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3bdc006810 0x7f3bdc008d30 secure :-1 s=READY pgs=3097 cs=0 l=1 rev1=1 rx=0x7f3bd804d8b0 tx=0x7f3bd8006460).ready entity=mon.0 client_cookie=927893950a6249db server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:55.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:55.755 7f3bee572700 1 -- 172.21.2.112:0/1947351984 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3bd80065d0 con 0x7f3bdc006810 2021-02-10T11:47:55.777 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:55.755 7f3bee572700 1 -- 172.21.2.112:0/1947351984 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bd80500c0 con 0x7f3bdc006810 2021-02-10T11:47:55.777 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:55.755 7f3bee572700 1 -- 172.21.2.112:0/1947351984 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3bd8050590 con 0x7f3bdc006810 2021-02-10T11:47:55.777 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:55.755 7f3bf5791700 1 -- 172.21.2.112:0/1947351984 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3bdc006810 msgr2=0x7f3bdc008d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:55.777 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:55.755 7f3bf5791700 1 --2- 172.21.2.112:0/1947351984 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3bdc006810 0x7f3bdc008d30 secure :-1 s=READY pgs=3097 cs=0 l=1 rev1=1 rx=0x7f3bd804d8b0 tx=0x7f3bd8006460).stop 2021-02-10T11:47:55.777 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:55.756 7f3bf5791700 1 -- 172.21.2.112:0/1947351984 shutdown_connections 2021-02-10T11:47:55.777 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:55.756 7f3bf5791700 1 --2- 172.21.2.112:0/1947351984 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3bdc006810 0x7f3bdc008d30 unknown :-1 s=CLOSED pgs=3097 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:55.777 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:55.756 7f3bf5791700 1 -- 172.21.2.112:0/1947351984 >> 172.21.2.112:0/1947351984 conn(0x7f3bf00ce410 msgr2=0x7f3bf00d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:55.777 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:55.756 7f3bf5791700 1 -- 172.21.2.112:0/1947351984 shutdown_connections 2021-02-10T11:47:55.777 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:55.756 7f3bf5791700 1 -- 172.21.2.112:0/1947351984 wait complete. 2021-02-10T11:47:55.777 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:55.770 7f3bf5791700 1 Processor -- start 2021-02-10T11:47:55.778 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:55.771 7f3bf5791700 1 -- start start 2021-02-10T11:47:55.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:55.771 7f3bf5791700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3bf00badd0 0x7f3bf009eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:55.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:55.771 7f3bf5791700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3bf00adf50 0x7f3bf009f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:55.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:55.771 7f3bf5791700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3bf01a7470 0x7f3bf01a7870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:55.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:55.771 7f3bf5791700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bf009e5a0 con 0x7f3bf00adf50 2021-02-10T11:47:55.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:55.771 7f3bf5791700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bf009f6e0 con 0x7f3bf00badd0 2021-02-10T11:47:55.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:55.771 7f3bf5791700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bf00bb220 con 0x7f3bf01a7470 2021-02-10T11:47:55.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:55.771 7f3beed73700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3bf00adf50 0x7f3bf009f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:55.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:55.771 7f3beed73700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3bf00adf50 0x7f3bf009f0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:60798/0 (socket says 172.21.2.112:60798) 2021-02-10T11:47:55.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:55.771 7f3beed73700 1 -- 172.21.2.112:0/1985020386 learned_addr learned my addr 172.21.2.112:0/1985020386 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:47:55.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:55.771 7f3bef574700 1 --2- 172.21.2.112:0/1985020386 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3bf00badd0 0x7f3bf009eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:55.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:55.771 7f3befd75700 1 --2- 172.21.2.112:0/1985020386 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3bf01a7470 0x7f3bf01a7870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:55.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:55.772 7f3beed73700 1 -- 172.21.2.112:0/1985020386 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3bf01a7470 msgr2=0x7f3bf01a7870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:55.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:55.772 7f3beed73700 1 --2- 172.21.2.112:0/1985020386 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3bf01a7470 0x7f3bf01a7870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:55.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:55.772 7f3beed73700 1 -- 172.21.2.112:0/1985020386 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3bf00badd0 msgr2=0x7f3bf009eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:55.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:55.772 7f3beed73700 1 --2- 172.21.2.112:0/1985020386 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3bf00badd0 0x7f3bf009eaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:55.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:55.772 7f3beed73700 1 -- 172.21.2.112:0/1985020386 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bf01a7e90 con 0x7f3bf00adf50 2021-02-10T11:47:55.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:55.772 7f3beed73700 1 --2- 172.21.2.112:0/1985020386 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3bf00adf50 0x7f3bf009f0c0 secure :-1 s=READY pgs=3098 cs=0 l=1 rev1=1 rx=0x7f3bd804e5c0 tx=0x7f3bd804e960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:55.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:55.772 7f3be7fff700 1 -- 172.21.2.112:0/1985020386 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3bd8000960 con 0x7f3bf00adf50 2021-02-10T11:47:55.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:55.772 7f3be7fff700 1 -- 172.21.2.112:0/1985020386 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bd8006020 con 0x7f3bf00adf50 2021-02-10T11:47:55.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:55.772 7f3be7fff700 1 -- 172.21.2.112:0/1985020386 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3bd80054d0 con 0x7f3bf00adf50 2021-02-10T11:47:55.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:55.772 7f3bf5791700 1 -- 172.21.2.112:0/1985020386 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bf01a7e90 con 0x7f3bf00adf50 2021-02-10T11:47:55.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:55.772 7f3bf5791700 1 -- 172.21.2.112:0/1985020386 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bf009d1f0 con 0x7f3bf00adf50 2021-02-10T11:47:55.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:55.773 7f3be7fff700 1 -- 172.21.2.112:0/1985020386 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3bd8000940 con 0x7f3bf00adf50 2021-02-10T11:47:55.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:55.773 7f3be7fff700 1 --2- 172.21.2.112:0/1985020386 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3bd40186c0 0x7f3bd401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:55.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:55.773 7f3be7fff700 1 -- 172.21.2.112:0/1985020386 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f3bd80765c0 con 0x7f3bf00adf50 2021-02-10T11:47:55.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:55.773 7f3bef574700 1 --2- 172.21.2.112:0/1985020386 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3bd40186c0 0x7f3bd401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:55.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:55.774 7f3bef574700 1 --2- 172.21.2.112:0/1985020386 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3bd40186c0 0x7f3bd401abc0 crc :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:55.806 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:55.803 7f3bf5791700 1 -- 172.21.2.112:0/1985020386 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bf0082a20 con 0x7f3bf00adf50 2021-02-10T11:47:55.809 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:55.806 7f3be7fff700 1 -- 172.21.2.112:0/1985020386 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3bd8074d70 con 0x7f3bf00adf50 2021-02-10T11:47:55.958 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:55.954 7f3bf5791700 1 -- 172.21.2.112:0/1985020386 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f3bf0082a20 con 0x7f3bf00adf50 2021-02-10T11:47:55.961 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:55.958 7f3be7fff700 1 -- 172.21.2.112:0/1985020386 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v46)=0 v46) v1 ==== 122+0+0 (secure 0 0 0) 0x7f3bd8000940 con 0x7f3bf00adf50 2021-02-10T11:47:55.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:55.963 7f3bf5791700 1 -- 172.21.2.112:0/1985020386 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3bd40186c0 msgr2=0x7f3bd401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:55.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:55.963 7f3bf5791700 1 --2- 172.21.2.112:0/1985020386 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3bd40186c0 0x7f3bd401abc0 crc :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:55.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:55.963 7f3bf5791700 1 -- 172.21.2.112:0/1985020386 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3bf00adf50 msgr2=0x7f3bf009f0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:55.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:55.963 7f3bf5791700 1 --2- 172.21.2.112:0/1985020386 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3bf00adf50 0x7f3bf009f0c0 secure :-1 s=READY pgs=3098 cs=0 l=1 rev1=1 rx=0x7f3bd804e5c0 tx=0x7f3bd804e960).stop 2021-02-10T11:47:55.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:55.963 7f3bf5791700 1 -- 172.21.2.112:0/1985020386 shutdown_connections 2021-02-10T11:47:55.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:55.963 7f3bf5791700 1 --2- 172.21.2.112:0/1985020386 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3bd40186c0 0x7f3bd401abc0 unknown :-1 s=CLOSED pgs=1393 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:55.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:55.963 7f3bf5791700 1 --2- 172.21.2.112:0/1985020386 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3bf01a7470 0x7f3bf01a7870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:55.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:55.963 7f3bf5791700 1 --2- 172.21.2.112:0/1985020386 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3bf00adf50 0x7f3bf009f0c0 unknown :-1 s=CLOSED pgs=3098 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:55.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:55.963 7f3bf5791700 1 --2- 172.21.2.112:0/1985020386 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3bf00badd0 0x7f3bf009eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:55.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:55.963 7f3bf5791700 1 -- 172.21.2.112:0/1985020386 >> 172.21.2.112:0/1985020386 conn(0x7f3bf00bee90 msgr2=0x7f3bf00bb880 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:55.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:55.963 7f3bf5791700 1 -- 172.21.2.112:0/1985020386 shutdown_connections 2021-02-10T11:47:55.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:55.963 7f3bf5791700 1 -- 172.21.2.112:0/1985020386 wait complete. 2021-02-10T11:47:55.975 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:47:55.975 DEBUG:teuthology.orchestra.run.gibba012:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:47:56.321 DEBUG:teuthology.orchestra.run.gibba020:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:47:56.665 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:47:56.665 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:47:56.665 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:47:56.665 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:47:57.000 INFO:teuthology.orchestra.run.gibba012.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.0’: No such file or directory 2021-02-10T11:47:57.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:47:57.000 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2021-02-10T11:47:57.001 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:47:57.001 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:47:57.001 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:47:57.336 INFO:teuthology.orchestra.run.gibba012.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: No such file or directory 2021-02-10T11:47:57.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:47:57.337 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2021-02-10T11:47:57.337 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:47:57.337 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:47:57.337 DEBUG:teuthology.orchestra.run.gibba020:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:47:57.475 DEBUG:teuthology.orchestra.run.gibba012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:47:57.478 DEBUG:teuthology.orchestra.run.gibba020:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:47:57.672 INFO:teuthology.orchestra.run.gibba020.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:47:57.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:47:57.672 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:47:57.673 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:47:57.673 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:47:57.673 DEBUG:teuthology.orchestra.run.gibba020:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:47:58.007 INFO:teuthology.orchestra.run.gibba020.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:47:58.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:47:58.008 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:47:58.008 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json-pretty 2021-02-10T11:47:58.436 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:58.417 7f473941c700 1 Processor -- start 2021-02-10T11:47:58.437 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:58.417 7f473941c700 1 -- start start 2021-02-10T11:47:58.437 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:58.418 7f473941c700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f47340d0340 0x7f47340ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:58.437 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:58.418 7f473941c700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47340f0b10 con 0x7f47340d1870 2021-02-10T11:47:58.437 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:58.418 7f473941c700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47341b0be0 con 0x7f47340d1c70 2021-02-10T11:47:58.437 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:58.418 7f473941c700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47340e5ed0 con 0x7f47340d0340 2021-02-10T11:47:58.437 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:58.418 7f473287c700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f47340d0340 0x7f47340ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:58.437 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:58.418 7f473287c700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f47340d0340 0x7f47340ef640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:60802/0 (socket says 172.21.2.112:60802) 2021-02-10T11:47:58.437 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:58.418 7f473287c700 1 -- 172.21.2.112:0/3735647090 learned_addr learned my addr 172.21.2.112:0/3735647090 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:47:58.437 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:58.418 7f473307d700 1 --1- 172.21.2.112:0/3735647090 >> v1:172.21.2.112:6789/0 conn(0x7f47340d1870 0x7f47341b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:60264/0 (socket says 172.21.2.112:60264) 2021-02-10T11:47:58.437 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:58.418 7f473287c700 1 -- 172.21.2.112:0/3735647090 >> v1:172.21.2.120:6789/0 conn(0x7f47340d1c70 legacy=0x7f47340efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:58.437 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:58.418 7f473287c700 1 -- 172.21.2.112:0/3735647090 >> v1:172.21.2.112:6789/0 conn(0x7f47340d1870 legacy=0x7f47341b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:58.437 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:58.418 7f473287c700 1 -- 172.21.2.112:0/3735647090 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47340f0450 con 0x7f47340d0340 2021-02-10T11:47:58.438 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:58.418 7f473287c700 1 --2- 172.21.2.112:0/3735647090 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f47340d0340 0x7f47340ef640 secure :-1 s=READY pgs=3100 cs=0 l=1 rev1=1 rx=0x7f472c006a80 tx=0x7f472c009e80).ready entity=mon.0 client_cookie=aa13dfc2d541f15e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:58.438 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:58.418 7f473207b700 1 -- 172.21.2.112:0/3735647090 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f472c00b510 con 0x7f47340d0340 2021-02-10T11:47:58.438 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:58.418 7f473207b700 1 -- 172.21.2.112:0/3735647090 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f472c00b7d0 con 0x7f47340d0340 2021-02-10T11:47:58.438 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:58.418 7f473207b700 1 -- 172.21.2.112:0/3735647090 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f472c00bcb0 con 0x7f47340d0340 2021-02-10T11:47:58.438 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:58.418 7f473941c700 1 -- 172.21.2.112:0/3735647090 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f47340d0340 msgr2=0x7f47340ef640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:58.438 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:58.418 7f473941c700 1 --2- 172.21.2.112:0/3735647090 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f47340d0340 0x7f47340ef640 secure :-1 s=READY pgs=3100 cs=0 l=1 rev1=1 rx=0x7f472c006a80 tx=0x7f472c009e80).stop 2021-02-10T11:47:58.438 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:58.419 7f473941c700 1 -- 172.21.2.112:0/3735647090 shutdown_connections 2021-02-10T11:47:58.438 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:58.419 7f473941c700 1 --2- 172.21.2.112:0/3735647090 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f47340d0340 0x7f47340ef640 unknown :-1 s=CLOSED pgs=3100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:58.438 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:58.419 7f473941c700 1 -- 172.21.2.112:0/3735647090 >> 172.21.2.112:0/3735647090 conn(0x7f47340ce450 msgr2=0x7f47340d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:58.438 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:58.419 7f473941c700 1 -- 172.21.2.112:0/3735647090 shutdown_connections 2021-02-10T11:47:58.438 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:58.419 7f473941c700 1 -- 172.21.2.112:0/3735647090 wait complete. 2021-02-10T11:47:58.439 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:58.433 7f473941c700 1 Processor -- start 2021-02-10T11:47:58.439 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:58.433 7f473941c700 1 -- start start 2021-02-10T11:47:58.440 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:58.434 7f473941c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f47340b32d0 0x7f47340b11e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:58.441 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:58.434 7f473941c700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f47340b1800 0x7f47340aaa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:58.441 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:58.434 7f473941c700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f47340ab090 0x7f47340ab490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:58.441 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:58.434 7f473941c700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47340b1d10 con 0x7f47340b32d0 2021-02-10T11:47:58.441 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:58.434 7f473941c700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47340b1f30 con 0x7f47340b1800 2021-02-10T11:47:58.441 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:58.434 7f473941c700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47340b2150 con 0x7f47340ab090 2021-02-10T11:47:58.441 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:58.434 7f473307d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f47340b32d0 0x7f47340b11e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:58.441 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:58.434 7f473307d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f47340b32d0 0x7f47340b11e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:60810/0 (socket says 172.21.2.112:60810) 2021-02-10T11:47:58.441 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:58.434 7f473307d700 1 -- 172.21.2.112:0/3677154412 learned_addr learned my addr 172.21.2.112:0/3677154412 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:47:58.441 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:58.434 7f473387e700 1 --2- 172.21.2.112:0/3677154412 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f47340ab090 0x7f47340ab490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:58.441 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:58.434 7f473307d700 1 -- 172.21.2.112:0/3677154412 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f47340ab090 msgr2=0x7f47340ab490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:58.441 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:58.434 7f473307d700 1 --2- 172.21.2.112:0/3677154412 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f47340ab090 0x7f47340ab490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:58.442 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:58.434 7f473307d700 1 -- 172.21.2.112:0/3677154412 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f47340b1800 msgr2=0x7f47340aaa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:58.442 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:58.434 7f473307d700 1 --2- 172.21.2.112:0/3677154412 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f47340b1800 0x7f47340aaa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:58.442 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:58.434 7f473307d700 1 -- 172.21.2.112:0/3677154412 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47340b2370 con 0x7f47340b32d0 2021-02-10T11:47:58.442 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:58.434 7f473387e700 1 --2- 172.21.2.112:0/3677154412 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f47340ab090 0x7f47340ab490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:47:58.442 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:58.435 7f473307d700 1 --2- 172.21.2.112:0/3677154412 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f47340b32d0 0x7f47340b11e0 secure :-1 s=READY pgs=3101 cs=0 l=1 rev1=1 rx=0x7f472800a590 tx=0x7f472800a930).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:58.442 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:58.435 7f471bfff700 1 -- 172.21.2.112:0/3677154412 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4728000ef0 con 0x7f47340b32d0 2021-02-10T11:47:58.442 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:58.435 7f471bfff700 1 -- 172.21.2.112:0/3677154412 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f472800ad20 con 0x7f47340b32d0 2021-02-10T11:47:58.442 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:58.435 7f471bfff700 1 -- 172.21.2.112:0/3677154412 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4728004620 con 0x7f47340b32d0 2021-02-10T11:47:58.442 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:58.435 7f473941c700 1 -- 172.21.2.112:0/3677154412 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47340b2370 con 0x7f47340b32d0 2021-02-10T11:47:58.442 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:58.435 7f473941c700 1 -- 172.21.2.112:0/3677154412 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47340b2900 con 0x7f47340b32d0 2021-02-10T11:47:58.442 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:58.436 7f471bfff700 1 -- 172.21.2.112:0/3677154412 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f47280043a0 con 0x7f47340b32d0 2021-02-10T11:47:58.442 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:58.436 7f471bfff700 1 --2- 172.21.2.112:0/3677154412 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f47140186c0 0x7f471401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:58.443 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:58.436 7f471bfff700 1 -- 172.21.2.112:0/3677154412 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f47280325a0 con 0x7f47340b32d0 2021-02-10T11:47:58.443 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:58.436 7f473287c700 1 --2- 172.21.2.112:0/3677154412 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f47140186c0 0x7f471401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:58.443 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:58.437 7f473287c700 1 --2- 172.21.2.112:0/3677154412 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f47140186c0 0x7f471401abc0 crc :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:58.470 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:58.467 7f473941c700 1 -- 172.21.2.112:0/3677154412 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47340ffb10 con 0x7f47340b32d0 2021-02-10T11:47:58.473 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:58.470 7f471bfff700 1 -- 172.21.2.112:0/3677154412 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4728027e90 con 0x7f47340b32d0 2021-02-10T11:47:58.596 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:58.593 7f473941c700 1 -- 172.21.2.112:0/3677154412 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f47340f06f0 con 0x7f47340b32d0 2021-02-10T11:47:58.597 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:58.594 7f471bfff700 1 -- 172.21.2.112:0/3677154412 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v180) v1 ==== 81+0+30374 (secure 0 0 0) 0x7f4728027e90 con 0x7f47340b32d0 2021-02-10T11:47:58.598 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:47:58.598 INFO:teuthology.orchestra.run.gibba012.stdout:{ 2021-02-10T11:47:58.598 INFO:teuthology.orchestra.run.gibba012.stdout: "epoch": 180, 2021-02-10T11:47:58.598 INFO:teuthology.orchestra.run.gibba012.stdout: "fsid": "f14f004e-0699-4584-ad81-3754258493de", 2021-02-10T11:47:58.598 INFO:teuthology.orchestra.run.gibba012.stdout: "created": "2021-02-10 11:13:26.501424", 2021-02-10T11:47:58.598 INFO:teuthology.orchestra.run.gibba012.stdout: "modified": "2021-02-10 11:47:15.427174", 2021-02-10T11:47:58.598 INFO:teuthology.orchestra.run.gibba012.stdout: "last_up_change": "2021-02-10 11:13:52.780316", 2021-02-10T11:47:58.599 INFO:teuthology.orchestra.run.gibba012.stdout: "last_in_change": "2021-02-10 11:13:52.780316", 2021-02-10T11:47:58.599 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-02-10T11:47:58.599 INFO:teuthology.orchestra.run.gibba012.stdout: "flags_num": 5799936, 2021-02-10T11:47:58.599 INFO:teuthology.orchestra.run.gibba012.stdout: "flags_set": [ 2021-02-10T11:47:58.599 INFO:teuthology.orchestra.run.gibba012.stdout: "pglog_hardlimit", 2021-02-10T11:47:58.599 INFO:teuthology.orchestra.run.gibba012.stdout: "purged_snapdirs", 2021-02-10T11:47:58.599 INFO:teuthology.orchestra.run.gibba012.stdout: "recovery_deletes", 2021-02-10T11:47:58.599 INFO:teuthology.orchestra.run.gibba012.stdout: "sortbitwise" 2021-02-10T11:47:58.599 INFO:teuthology.orchestra.run.gibba012.stdout: ], 2021-02-10T11:47:58.599 INFO:teuthology.orchestra.run.gibba012.stdout: "crush_version": 4, 2021-02-10T11:47:58.599 INFO:teuthology.orchestra.run.gibba012.stdout: "full_ratio": 0.94999998807907104, 2021-02-10T11:47:58.599 INFO:teuthology.orchestra.run.gibba012.stdout: "backfillfull_ratio": 0.89999997615814209, 2021-02-10T11:47:58.599 INFO:teuthology.orchestra.run.gibba012.stdout: "nearfull_ratio": 0.85000002384185791, 2021-02-10T11:47:58.599 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_snapshot": "", 2021-02-10T11:47:58.600 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_max": 39, 2021-02-10T11:47:58.600 INFO:teuthology.orchestra.run.gibba012.stdout: "max_osd": 8, 2021-02-10T11:47:58.600 INFO:teuthology.orchestra.run.gibba012.stdout: "require_min_compat_client": "jewel", 2021-02-10T11:47:58.600 INFO:teuthology.orchestra.run.gibba012.stdout: "min_compat_client": "jewel", 2021-02-10T11:47:58.600 INFO:teuthology.orchestra.run.gibba012.stdout: "require_osd_release": "nautilus", 2021-02-10T11:47:58.600 INFO:teuthology.orchestra.run.gibba012.stdout: "pools": [ 2021-02-10T11:47:58.600 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:58.600 INFO:teuthology.orchestra.run.gibba012.stdout: "pool": 1, 2021-02-10T11:47:58.600 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_name": "rbd", 2021-02-10T11:47:58.600 INFO:teuthology.orchestra.run.gibba012.stdout: "create_time": "2021-02-10 11:13:57.508758", 2021-02-10T11:47:58.600 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:47:58.600 INFO:teuthology.orchestra.run.gibba012.stdout: "flags_names": "hashpspool", 2021-02-10T11:47:58.601 INFO:teuthology.orchestra.run.gibba012.stdout: "type": 1, 2021-02-10T11:47:58.601 INFO:teuthology.orchestra.run.gibba012.stdout: "size": 2, 2021-02-10T11:47:58.601 INFO:teuthology.orchestra.run.gibba012.stdout: "min_size": 1, 2021-02-10T11:47:58.601 INFO:teuthology.orchestra.run.gibba012.stdout: "crush_rule": 0, 2021-02-10T11:47:58.601 INFO:teuthology.orchestra.run.gibba012.stdout: "object_hash": 2, 2021-02-10T11:47:58.601 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:47:58.601 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num": 8, 2021-02-10T11:47:58.601 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num": 8, 2021-02-10T11:47:58.601 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num_target": 8, 2021-02-10T11:47:58.601 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_target": 8, 2021-02-10T11:47:58.601 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_pending": 8, 2021-02-10T11:47:58.601 INFO:teuthology.orchestra.run.gibba012.stdout: "last_pg_merge_meta": { 2021-02-10T11:47:58.601 INFO:teuthology.orchestra.run.gibba012.stdout: "source_pgid": "0.0", 2021-02-10T11:47:58.601 INFO:teuthology.orchestra.run.gibba012.stdout: "ready_epoch": 0, 2021-02-10T11:47:58.602 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_started": 0, 2021-02-10T11:47:58.602 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_clean": 0, 2021-02-10T11:47:58.602 INFO:teuthology.orchestra.run.gibba012.stdout: "source_version": "0'0", 2021-02-10T11:47:58.602 INFO:teuthology.orchestra.run.gibba012.stdout: "target_version": "0'0" 2021-02-10T11:47:58.602 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:58.602 INFO:teuthology.orchestra.run.gibba012.stdout: "last_change": "16", 2021-02-10T11:47:58.602 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend": "0", 2021-02-10T11:47:58.602 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:47:58.602 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:47:58.602 INFO:teuthology.orchestra.run.gibba012.stdout: "auid": 0, 2021-02-10T11:47:58.602 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:47:58.602 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_seq": 0, 2021-02-10T11:47:58.602 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_epoch": 0, 2021-02-10T11:47:58.603 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_snaps": [], 2021-02-10T11:47:58.603 INFO:teuthology.orchestra.run.gibba012.stdout: "removed_snaps": "[]", 2021-02-10T11:47:58.603 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_bytes": 0, 2021-02-10T11:47:58.603 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_objects": 0, 2021-02-10T11:47:58.603 INFO:teuthology.orchestra.run.gibba012.stdout: "tiers": [], 2021-02-10T11:47:58.603 INFO:teuthology.orchestra.run.gibba012.stdout: "tier_of": -1, 2021-02-10T11:47:58.603 INFO:teuthology.orchestra.run.gibba012.stdout: "read_tier": -1, 2021-02-10T11:47:58.603 INFO:teuthology.orchestra.run.gibba012.stdout: "write_tier": -1, 2021-02-10T11:47:58.603 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_mode": "none", 2021-02-10T11:47:58.603 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_bytes": 0, 2021-02-10T11:47:58.603 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_objects": 0, 2021-02-10T11:47:58.603 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:47:58.603 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:47:58.603 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:47:58.604 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_flush_age": 0, 2021-02-10T11:47:58.604 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_evict_age": 0, 2021-02-10T11:47:58.604 INFO:teuthology.orchestra.run.gibba012.stdout: "erasure_code_profile": "", 2021-02-10T11:47:58.604 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_params": { 2021-02-10T11:47:58.604 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "none" 2021-02-10T11:47:58.604 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:58.604 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_period": 0, 2021-02-10T11:47:58.604 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_count": 0, 2021-02-10T11:47:58.604 INFO:teuthology.orchestra.run.gibba012.stdout: "use_gmt_hitset": true, 2021-02-10T11:47:58.604 INFO:teuthology.orchestra.run.gibba012.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:47:58.604 INFO:teuthology.orchestra.run.gibba012.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:47:58.604 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:47:58.604 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:47:58.605 INFO:teuthology.orchestra.run.gibba012.stdout: "grade_table": [], 2021-02-10T11:47:58.605 INFO:teuthology.orchestra.run.gibba012.stdout: "stripe_width": 0, 2021-02-10T11:47:58.605 INFO:teuthology.orchestra.run.gibba012.stdout: "expected_num_objects": 0, 2021-02-10T11:47:58.605 INFO:teuthology.orchestra.run.gibba012.stdout: "fast_read": false, 2021-02-10T11:47:58.605 INFO:teuthology.orchestra.run.gibba012.stdout: "options": {}, 2021-02-10T11:47:58.605 INFO:teuthology.orchestra.run.gibba012.stdout: "application_metadata": { 2021-02-10T11:47:58.605 INFO:teuthology.orchestra.run.gibba012.stdout: "rbd": {} 2021-02-10T11:47:58.605 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:47:58.605 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:58.605 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:58.605 INFO:teuthology.orchestra.run.gibba012.stdout: "pool": 38, 2021-02-10T11:47:58.605 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_name": "cephfs_metadata", 2021-02-10T11:47:58.605 INFO:teuthology.orchestra.run.gibba012.stdout: "create_time": "2021-02-10 11:47:12.395866", 2021-02-10T11:47:58.605 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:47:58.606 INFO:teuthology.orchestra.run.gibba012.stdout: "flags_names": "hashpspool", 2021-02-10T11:47:58.606 INFO:teuthology.orchestra.run.gibba012.stdout: "type": 1, 2021-02-10T11:47:58.606 INFO:teuthology.orchestra.run.gibba012.stdout: "size": 2, 2021-02-10T11:47:58.606 INFO:teuthology.orchestra.run.gibba012.stdout: "min_size": 1, 2021-02-10T11:47:58.606 INFO:teuthology.orchestra.run.gibba012.stdout: "crush_rule": 0, 2021-02-10T11:47:58.606 INFO:teuthology.orchestra.run.gibba012.stdout: "object_hash": 2, 2021-02-10T11:47:58.606 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:47:58.606 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num": 8, 2021-02-10T11:47:58.606 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num": 8, 2021-02-10T11:47:58.606 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num_target": 8, 2021-02-10T11:47:58.606 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_target": 8, 2021-02-10T11:47:58.606 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_pending": 8, 2021-02-10T11:47:58.606 INFO:teuthology.orchestra.run.gibba012.stdout: "last_pg_merge_meta": { 2021-02-10T11:47:58.606 INFO:teuthology.orchestra.run.gibba012.stdout: "source_pgid": "0.0", 2021-02-10T11:47:58.607 INFO:teuthology.orchestra.run.gibba012.stdout: "ready_epoch": 0, 2021-02-10T11:47:58.607 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_started": 0, 2021-02-10T11:47:58.607 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_clean": 0, 2021-02-10T11:47:58.607 INFO:teuthology.orchestra.run.gibba012.stdout: "source_version": "0'0", 2021-02-10T11:47:58.607 INFO:teuthology.orchestra.run.gibba012.stdout: "target_version": "0'0" 2021-02-10T11:47:58.607 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:58.607 INFO:teuthology.orchestra.run.gibba012.stdout: "last_change": "179", 2021-02-10T11:47:58.607 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend": "0", 2021-02-10T11:47:58.607 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:47:58.607 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:47:58.607 INFO:teuthology.orchestra.run.gibba012.stdout: "auid": 0, 2021-02-10T11:47:58.607 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:47:58.607 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_seq": 0, 2021-02-10T11:47:58.608 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_epoch": 0, 2021-02-10T11:47:58.608 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_snaps": [], 2021-02-10T11:47:58.608 INFO:teuthology.orchestra.run.gibba012.stdout: "removed_snaps": "[]", 2021-02-10T11:47:58.608 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_bytes": 0, 2021-02-10T11:47:58.608 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_objects": 0, 2021-02-10T11:47:58.608 INFO:teuthology.orchestra.run.gibba012.stdout: "tiers": [], 2021-02-10T11:47:58.608 INFO:teuthology.orchestra.run.gibba012.stdout: "tier_of": -1, 2021-02-10T11:47:58.608 INFO:teuthology.orchestra.run.gibba012.stdout: "read_tier": -1, 2021-02-10T11:47:58.608 INFO:teuthology.orchestra.run.gibba012.stdout: "write_tier": -1, 2021-02-10T11:47:58.608 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_mode": "none", 2021-02-10T11:47:58.608 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_bytes": 0, 2021-02-10T11:47:58.608 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_objects": 0, 2021-02-10T11:47:58.608 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:47:58.608 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:47:58.609 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:47:58.609 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_flush_age": 0, 2021-02-10T11:47:58.609 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_evict_age": 0, 2021-02-10T11:47:58.609 INFO:teuthology.orchestra.run.gibba012.stdout: "erasure_code_profile": "", 2021-02-10T11:47:58.609 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_params": { 2021-02-10T11:47:58.609 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "none" 2021-02-10T11:47:58.609 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:58.609 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_period": 0, 2021-02-10T11:47:58.609 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_count": 0, 2021-02-10T11:47:58.609 INFO:teuthology.orchestra.run.gibba012.stdout: "use_gmt_hitset": true, 2021-02-10T11:47:58.609 INFO:teuthology.orchestra.run.gibba012.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:47:58.609 INFO:teuthology.orchestra.run.gibba012.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:47:58.609 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:47:58.610 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:47:58.610 INFO:teuthology.orchestra.run.gibba012.stdout: "grade_table": [], 2021-02-10T11:47:58.610 INFO:teuthology.orchestra.run.gibba012.stdout: "stripe_width": 0, 2021-02-10T11:47:58.610 INFO:teuthology.orchestra.run.gibba012.stdout: "expected_num_objects": 0, 2021-02-10T11:47:58.610 INFO:teuthology.orchestra.run.gibba012.stdout: "fast_read": false, 2021-02-10T11:47:58.610 INFO:teuthology.orchestra.run.gibba012.stdout: "options": { 2021-02-10T11:47:58.610 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_autoscale_bias": 4, 2021-02-10T11:47:58.610 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_min": 16, 2021-02-10T11:47:58.610 INFO:teuthology.orchestra.run.gibba012.stdout: "recovery_priority": 5 2021-02-10T11:47:58.610 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:58.610 INFO:teuthology.orchestra.run.gibba012.stdout: "application_metadata": { 2021-02-10T11:47:58.610 INFO:teuthology.orchestra.run.gibba012.stdout: "cephfs": { 2021-02-10T11:47:58.610 INFO:teuthology.orchestra.run.gibba012.stdout: "metadata": "cephfs" 2021-02-10T11:47:58.611 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:47:58.611 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:47:58.611 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:58.611 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:58.611 INFO:teuthology.orchestra.run.gibba012.stdout: "pool": 39, 2021-02-10T11:47:58.611 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_name": "cephfs_data", 2021-02-10T11:47:58.611 INFO:teuthology.orchestra.run.gibba012.stdout: "create_time": "2021-02-10 11:47:13.420625", 2021-02-10T11:47:58.611 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:47:58.611 INFO:teuthology.orchestra.run.gibba012.stdout: "flags_names": "hashpspool", 2021-02-10T11:47:58.611 INFO:teuthology.orchestra.run.gibba012.stdout: "type": 1, 2021-02-10T11:47:58.611 INFO:teuthology.orchestra.run.gibba012.stdout: "size": 2, 2021-02-10T11:47:58.611 INFO:teuthology.orchestra.run.gibba012.stdout: "min_size": 1, 2021-02-10T11:47:58.611 INFO:teuthology.orchestra.run.gibba012.stdout: "crush_rule": 0, 2021-02-10T11:47:58.611 INFO:teuthology.orchestra.run.gibba012.stdout: "object_hash": 2, 2021-02-10T11:47:58.612 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:47:58.612 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num": 8, 2021-02-10T11:47:58.612 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num": 8, 2021-02-10T11:47:58.612 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num_target": 8, 2021-02-10T11:47:58.612 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_target": 8, 2021-02-10T11:47:58.612 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_pending": 8, 2021-02-10T11:47:58.612 INFO:teuthology.orchestra.run.gibba012.stdout: "last_pg_merge_meta": { 2021-02-10T11:47:58.612 INFO:teuthology.orchestra.run.gibba012.stdout: "source_pgid": "0.0", 2021-02-10T11:47:58.612 INFO:teuthology.orchestra.run.gibba012.stdout: "ready_epoch": 0, 2021-02-10T11:47:58.612 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_started": 0, 2021-02-10T11:47:58.612 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_clean": 0, 2021-02-10T11:47:58.612 INFO:teuthology.orchestra.run.gibba012.stdout: "source_version": "0'0", 2021-02-10T11:47:58.612 INFO:teuthology.orchestra.run.gibba012.stdout: "target_version": "0'0" 2021-02-10T11:47:58.612 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:58.613 INFO:teuthology.orchestra.run.gibba012.stdout: "last_change": "180", 2021-02-10T11:47:58.613 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend": "0", 2021-02-10T11:47:58.613 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:47:58.613 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:47:58.613 INFO:teuthology.orchestra.run.gibba012.stdout: "auid": 0, 2021-02-10T11:47:58.613 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:47:58.613 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_seq": 0, 2021-02-10T11:47:58.613 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_epoch": 0, 2021-02-10T11:47:58.613 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_snaps": [], 2021-02-10T11:47:58.613 INFO:teuthology.orchestra.run.gibba012.stdout: "removed_snaps": "[]", 2021-02-10T11:47:58.613 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_bytes": 0, 2021-02-10T11:47:58.613 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_objects": 0, 2021-02-10T11:47:58.613 INFO:teuthology.orchestra.run.gibba012.stdout: "tiers": [], 2021-02-10T11:47:58.613 INFO:teuthology.orchestra.run.gibba012.stdout: "tier_of": -1, 2021-02-10T11:47:58.614 INFO:teuthology.orchestra.run.gibba012.stdout: "read_tier": -1, 2021-02-10T11:47:58.614 INFO:teuthology.orchestra.run.gibba012.stdout: "write_tier": -1, 2021-02-10T11:47:58.614 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_mode": "none", 2021-02-10T11:47:58.614 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_bytes": 0, 2021-02-10T11:47:58.614 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_objects": 0, 2021-02-10T11:47:58.614 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:47:58.614 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:47:58.614 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:47:58.614 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_flush_age": 0, 2021-02-10T11:47:58.614 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_evict_age": 0, 2021-02-10T11:47:58.614 INFO:teuthology.orchestra.run.gibba012.stdout: "erasure_code_profile": "", 2021-02-10T11:47:58.614 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_params": { 2021-02-10T11:47:58.614 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "none" 2021-02-10T11:47:58.615 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:58.615 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_period": 0, 2021-02-10T11:47:58.615 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_count": 0, 2021-02-10T11:47:58.615 INFO:teuthology.orchestra.run.gibba012.stdout: "use_gmt_hitset": true, 2021-02-10T11:47:58.615 INFO:teuthology.orchestra.run.gibba012.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:47:58.615 INFO:teuthology.orchestra.run.gibba012.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:47:58.615 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:47:58.615 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:47:58.615 INFO:teuthology.orchestra.run.gibba012.stdout: "grade_table": [], 2021-02-10T11:47:58.615 INFO:teuthology.orchestra.run.gibba012.stdout: "stripe_width": 0, 2021-02-10T11:47:58.615 INFO:teuthology.orchestra.run.gibba012.stdout: "expected_num_objects": 0, 2021-02-10T11:47:58.615 INFO:teuthology.orchestra.run.gibba012.stdout: "fast_read": false, 2021-02-10T11:47:58.615 INFO:teuthology.orchestra.run.gibba012.stdout: "options": {}, 2021-02-10T11:47:58.616 INFO:teuthology.orchestra.run.gibba012.stdout: "application_metadata": { 2021-02-10T11:47:58.616 INFO:teuthology.orchestra.run.gibba012.stdout: "cephfs": { 2021-02-10T11:47:58.616 INFO:teuthology.orchestra.run.gibba012.stdout: "data": "cephfs" 2021-02-10T11:47:58.616 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:47:58.616 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:47:58.616 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:47:58.616 INFO:teuthology.orchestra.run.gibba012.stdout: ], 2021-02-10T11:47:58.616 INFO:teuthology.orchestra.run.gibba012.stdout: "osds": [ 2021-02-10T11:47:58.616 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:58.616 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 0, 2021-02-10T11:47:58.616 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "95444b8f-6370-4a1d-ae5f-df62651b77da", 2021-02-10T11:47:58.616 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:47:58.616 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:47:58.616 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:47:58.617 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:47:58.617 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:47:58.617 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:47:58.617 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:47:58.617 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 178, 2021-02-10T11:47:58.617 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:47:58.617 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:47:58.617 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:47:58.617 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:47:58.617 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:58.617 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:47:58.617 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6817", 2021-02-10T11:47:58.617 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:47:58.618 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:58.618 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:58.618 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:47:58.618 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6818", 2021-02-10T11:47:58.618 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:47:58.618 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:47:58.618 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:47:58.618 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:58.618 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:47:58.618 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:47:58.618 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:58.618 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:47:58.618 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6820", 2021-02-10T11:47:58.619 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:47:58.619 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:58.619 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:58.619 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:47:58.619 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6822", 2021-02-10T11:47:58.619 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:47:58.619 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:47:58.619 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:47:58.619 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:58.619 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:47:58.619 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:47:58.619 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:58.619 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:47:58.620 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6828", 2021-02-10T11:47:58.620 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:47:58.620 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:58.620 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:58.620 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:47:58.620 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6830", 2021-02-10T11:47:58.620 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:47:58.620 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:47:58.620 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:47:58.620 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:58.620 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:47:58.620 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:47:58.620 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:58.620 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:47:58.621 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6824", 2021-02-10T11:47:58.621 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:47:58.621 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:58.621 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:58.621 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:47:58.621 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6826", 2021-02-10T11:47:58.621 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:47:58.621 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:47:58.621 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:47:58.621 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:58.621 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.112:6818/36030", 2021-02-10T11:47:58.621 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.112:6822/36030", 2021-02-10T11:47:58.621 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.112:6830/36030", 2021-02-10T11:47:58.622 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.112:6826/36030", 2021-02-10T11:47:58.622 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:47:58.622 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:47:58.622 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:47:58.622 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:47:58.622 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:58.622 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:58.622 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 1, 2021-02-10T11:47:58.622 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "a23e217e-0d8e-486e-be07-712adfabd262", 2021-02-10T11:47:58.622 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:47:58.622 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:47:58.622 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:47:58.622 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:47:58.622 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:47:58.623 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:47:58.623 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:47:58.623 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 178, 2021-02-10T11:47:58.623 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:47:58.623 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:47:58.623 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:47:58.623 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:47:58.623 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:58.623 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:47:58.623 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6800", 2021-02-10T11:47:58.623 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:47:58.623 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:58.623 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:58.624 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:47:58.624 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6802", 2021-02-10T11:47:58.624 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:47:58.624 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:47:58.624 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:47:58.624 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:58.624 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:47:58.624 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:47:58.624 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:58.624 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:47:58.624 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6804", 2021-02-10T11:47:58.624 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:47:58.624 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:58.624 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:58.625 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:47:58.625 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6806", 2021-02-10T11:47:58.625 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:47:58.625 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:47:58.625 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:47:58.625 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:58.625 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:47:58.625 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:47:58.625 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:58.625 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:47:58.625 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6812", 2021-02-10T11:47:58.625 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:47:58.625 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:58.625 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:58.626 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:47:58.626 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6814", 2021-02-10T11:47:58.626 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:47:58.626 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:47:58.626 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:47:58.626 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:58.626 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:47:58.626 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:47:58.626 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:58.626 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:47:58.626 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6808", 2021-02-10T11:47:58.626 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:47:58.626 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:58.627 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:58.627 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:47:58.627 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6810", 2021-02-10T11:47:58.627 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:47:58.627 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:47:58.627 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:47:58.627 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:58.627 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.112:6802/36029", 2021-02-10T11:47:58.627 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.112:6806/36029", 2021-02-10T11:47:58.627 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.112:6814/36029", 2021-02-10T11:47:58.627 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.112:6810/36029", 2021-02-10T11:47:58.627 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:47:58.627 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:47:58.628 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:47:58.628 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:47:58.628 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:58.628 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:58.628 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 2, 2021-02-10T11:47:58.628 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "026afa8a-c0f1-4b35-9a76-53c00fe0c2fc", 2021-02-10T11:47:58.628 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:47:58.628 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:47:58.628 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:47:58.628 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:47:58.628 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:47:58.628 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:47:58.628 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:47:58.628 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 178, 2021-02-10T11:47:58.629 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:47:58.629 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:47:58.629 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:47:58.629 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:47:58.629 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:58.629 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:47:58.629 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6801", 2021-02-10T11:47:58.629 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:47:58.629 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:58.629 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:58.629 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:47:58.629 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6803", 2021-02-10T11:47:58.629 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:47:58.629 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:47:58.630 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:47:58.630 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:58.630 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:47:58.630 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:47:58.630 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:58.630 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:47:58.630 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6805", 2021-02-10T11:47:58.630 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:47:58.630 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:58.630 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:58.630 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:47:58.630 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6807", 2021-02-10T11:47:58.630 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:47:58.631 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:47:58.631 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:47:58.631 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:58.631 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:47:58.631 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:47:58.631 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:58.631 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:47:58.631 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6813", 2021-02-10T11:47:58.631 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:47:58.631 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:58.631 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:58.631 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:47:58.631 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6815", 2021-02-10T11:47:58.631 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:47:58.632 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:47:58.632 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:47:58.632 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:58.632 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:47:58.632 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:47:58.632 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:58.632 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:47:58.632 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6809", 2021-02-10T11:47:58.632 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:47:58.632 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:58.632 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:58.632 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:47:58.632 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6811", 2021-02-10T11:47:58.632 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:47:58.633 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:47:58.633 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:47:58.633 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:58.633 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.112:6803/36031", 2021-02-10T11:47:58.633 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.112:6807/36031", 2021-02-10T11:47:58.633 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.112:6815/36031", 2021-02-10T11:47:58.633 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.112:6811/36031", 2021-02-10T11:47:58.633 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:47:58.633 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:47:58.633 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:47:58.633 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:47:58.633 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:58.633 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:58.633 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 3, 2021-02-10T11:47:58.634 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "58795682-e00d-4173-bb84-f7075ca89a11", 2021-02-10T11:47:58.634 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:47:58.634 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:47:58.634 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:47:58.634 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:47:58.634 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:47:58.634 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:47:58.634 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:47:58.634 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 178, 2021-02-10T11:47:58.634 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:47:58.634 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:47:58.634 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:47:58.634 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:47:58.635 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:58.635 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:47:58.635 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6816", 2021-02-10T11:47:58.635 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:47:58.635 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:58.635 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:58.635 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:47:58.635 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6819", 2021-02-10T11:47:58.635 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:47:58.635 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:47:58.635 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:47:58.635 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:58.635 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:47:58.635 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:47:58.636 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:58.636 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:47:58.636 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6821", 2021-02-10T11:47:58.636 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:47:58.636 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:58.636 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:58.636 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:47:58.636 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6823", 2021-02-10T11:47:58.636 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:47:58.636 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:47:58.636 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:47:58.636 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:58.636 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:47:58.637 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:47:58.637 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:58.637 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:47:58.637 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6829", 2021-02-10T11:47:58.637 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:47:58.637 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:58.637 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:58.637 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:47:58.637 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6831", 2021-02-10T11:47:58.637 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:47:58.637 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:47:58.637 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:47:58.637 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:58.637 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:47:58.638 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:47:58.638 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:58.638 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:47:58.638 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6825", 2021-02-10T11:47:58.638 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:47:58.638 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:58.638 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:58.638 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:47:58.638 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6827", 2021-02-10T11:47:58.638 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:47:58.638 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:47:58.638 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:47:58.638 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:58.638 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.112:6819/36028", 2021-02-10T11:47:58.639 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.112:6823/36028", 2021-02-10T11:47:58.639 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.112:6831/36028", 2021-02-10T11:47:58.639 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.112:6827/36028", 2021-02-10T11:47:58.639 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:47:58.639 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:47:58.639 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:47:58.639 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:47:58.639 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:58.639 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:58.639 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 4, 2021-02-10T11:47:58.639 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "6763f8d6-53dc-4e45-bdf6-a6eabcf81e39", 2021-02-10T11:47:58.639 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:47:58.639 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:47:58.639 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:47:58.640 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:47:58.640 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:47:58.640 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:47:58.640 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:47:58.640 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 178, 2021-02-10T11:47:58.640 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:47:58.640 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:47:58.640 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:47:58.640 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:47:58.640 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:58.640 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:47:58.640 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6810", 2021-02-10T11:47:58.640 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:47:58.641 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:58.641 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:58.641 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:47:58.641 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6813", 2021-02-10T11:47:58.641 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:47:58.641 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:47:58.641 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:47:58.641 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:58.641 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:47:58.641 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:47:58.641 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:58.641 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:47:58.641 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6816", 2021-02-10T11:47:58.641 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:47:58.642 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:58.642 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:58.642 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:47:58.642 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6819", 2021-02-10T11:47:58.642 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:47:58.642 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:47:58.642 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:47:58.642 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:58.642 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:47:58.642 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:47:58.642 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:58.642 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:47:58.642 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6825", 2021-02-10T11:47:58.642 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:47:58.643 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:58.643 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:58.643 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:47:58.643 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6827", 2021-02-10T11:47:58.643 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:47:58.643 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:47:58.643 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:47:58.643 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:58.643 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:47:58.643 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:47:58.643 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:58.643 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:47:58.643 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6821", 2021-02-10T11:47:58.643 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:47:58.644 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:58.644 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:58.644 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:47:58.644 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6823", 2021-02-10T11:47:58.644 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:47:58.644 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:47:58.644 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:47:58.644 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:58.644 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.120:6813/36388", 2021-02-10T11:47:58.644 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.120:6819/36388", 2021-02-10T11:47:58.644 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.120:6827/36388", 2021-02-10T11:47:58.644 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.120:6823/36388", 2021-02-10T11:47:58.644 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:47:58.645 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:47:58.645 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:47:58.645 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:47:58.645 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:58.645 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:58.645 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 5, 2021-02-10T11:47:58.645 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "5d0e4dd6-0cb0-4bf8-8314-626acaa1b544", 2021-02-10T11:47:58.645 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:47:58.645 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:47:58.645 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:47:58.645 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:47:58.645 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:47:58.645 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:47:58.645 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:47:58.646 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 178, 2021-02-10T11:47:58.646 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:47:58.646 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:47:58.646 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:47:58.646 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:47:58.646 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:58.646 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:47:58.646 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6804", 2021-02-10T11:47:58.646 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:47:58.646 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:58.646 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:58.646 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:47:58.646 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6806", 2021-02-10T11:47:58.647 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:47:58.647 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:47:58.647 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:47:58.647 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:58.647 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:47:58.647 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:47:58.647 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:58.647 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:47:58.647 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6809", 2021-02-10T11:47:58.647 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:47:58.647 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:58.647 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:58.647 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:47:58.647 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6812", 2021-02-10T11:47:58.648 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:47:58.648 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:47:58.648 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:47:58.648 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:58.648 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:47:58.648 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:47:58.648 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:58.648 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:47:58.648 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6820", 2021-02-10T11:47:58.648 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:47:58.648 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:58.648 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:58.648 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:47:58.649 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6822", 2021-02-10T11:47:58.649 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:47:58.649 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:47:58.649 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:47:58.649 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:58.649 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:47:58.649 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:47:58.649 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:58.649 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:47:58.649 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6815", 2021-02-10T11:47:58.649 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:47:58.649 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:58.649 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:58.649 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:47:58.650 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6818", 2021-02-10T11:47:58.650 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:47:58.650 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:47:58.650 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:47:58.650 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:58.650 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.120:6806/36387", 2021-02-10T11:47:58.650 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.120:6812/36387", 2021-02-10T11:47:58.650 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.120:6822/36387", 2021-02-10T11:47:58.650 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.120:6818/36387", 2021-02-10T11:47:58.650 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:47:58.650 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:47:58.650 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:47:58.650 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:47:58.651 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:58.651 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:58.651 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 6, 2021-02-10T11:47:58.651 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "b010e187-4541-4b00-80dd-f948f6ab8f15", 2021-02-10T11:47:58.651 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:47:58.651 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:47:58.651 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:47:58.651 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:47:58.651 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:47:58.651 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:47:58.651 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:47:58.651 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 177, 2021-02-10T11:47:58.651 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:47:58.651 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:47:58.652 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:47:58.652 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:47:58.652 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:58.652 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:47:58.652 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6824", 2021-02-10T11:47:58.652 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:47:58.652 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:58.652 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:58.652 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:47:58.652 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6826", 2021-02-10T11:47:58.652 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:47:58.652 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:47:58.652 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:47:58.653 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:58.653 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:47:58.653 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:47:58.653 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:58.653 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:47:58.653 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6828", 2021-02-10T11:47:58.653 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:47:58.653 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:58.653 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:58.653 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:47:58.653 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6829", 2021-02-10T11:47:58.653 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:47:58.653 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:47:58.653 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:47:58.654 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:58.654 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:47:58.654 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:47:58.654 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:58.654 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:47:58.654 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6832", 2021-02-10T11:47:58.654 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:47:58.654 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:58.654 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:58.654 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:47:58.654 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6833", 2021-02-10T11:47:58.654 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:47:58.654 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:47:58.655 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:47:58.655 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:58.655 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:47:58.655 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:47:58.655 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:58.655 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:47:58.655 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6830", 2021-02-10T11:47:58.655 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:47:58.655 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:58.655 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:58.655 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:47:58.655 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6831", 2021-02-10T11:47:58.655 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:47:58.655 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:47:58.656 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:47:58.656 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:58.656 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.120:6826/36385", 2021-02-10T11:47:58.656 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.120:6829/36385", 2021-02-10T11:47:58.656 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.120:6833/36385", 2021-02-10T11:47:58.656 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.120:6831/36385", 2021-02-10T11:47:58.656 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:47:58.656 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:47:58.656 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:47:58.656 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:47:58.656 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:58.656 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:58.656 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 7, 2021-02-10T11:47:58.656 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "671e2772-33b7-497e-8e44-2178bd850bcf", 2021-02-10T11:47:58.657 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:47:58.657 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:47:58.657 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:47:58.657 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:47:58.657 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:47:58.657 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:47:58.657 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:47:58.657 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 178, 2021-02-10T11:47:58.657 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:47:58.657 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:47:58.657 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:47:58.657 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:47:58.657 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:58.657 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:47:58.658 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6802", 2021-02-10T11:47:58.658 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:47:58.658 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:58.658 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:58.658 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:47:58.658 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6803", 2021-02-10T11:47:58.658 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:47:58.658 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:47:58.658 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:47:58.658 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:58.658 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:47:58.658 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:47:58.658 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:58.659 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:47:58.659 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6805", 2021-02-10T11:47:58.659 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:47:58.659 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:58.659 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:58.659 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:47:58.659 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6807", 2021-02-10T11:47:58.659 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:47:58.659 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:47:58.659 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:47:58.659 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:58.659 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:47:58.659 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:47:58.660 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:58.660 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:47:58.660 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6814", 2021-02-10T11:47:58.660 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:47:58.660 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:58.660 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:58.660 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:47:58.660 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6817", 2021-02-10T11:47:58.660 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:47:58.660 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:47:58.660 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:47:58.660 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:58.660 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:47:58.660 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:47:58.661 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:58.661 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:47:58.661 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6808", 2021-02-10T11:47:58.661 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:47:58.661 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:58.661 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:58.661 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:47:58.661 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6811", 2021-02-10T11:47:58.661 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:47:58.661 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:47:58.661 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:47:58.661 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:58.661 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.120:6803/36386", 2021-02-10T11:47:58.661 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.120:6807/36386", 2021-02-10T11:47:58.662 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.120:6817/36386", 2021-02-10T11:47:58.662 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.120:6811/36386", 2021-02-10T11:47:58.662 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:47:58.662 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:47:58.662 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:47:58.662 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:47:58.662 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:47:58.662 INFO:teuthology.orchestra.run.gibba012.stdout: ], 2021-02-10T11:47:58.662 INFO:teuthology.orchestra.run.gibba012.stdout: "osd_xinfo": [ 2021-02-10T11:47:58.662 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:58.662 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 0, 2021-02-10T11:47:58.662 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:47:58.662 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:47:58.662 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:47:58.663 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:47:58.663 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:47:58.663 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:58.663 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:58.663 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 1, 2021-02-10T11:47:58.663 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:47:58.663 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:47:58.663 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:47:58.663 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:47:58.663 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:47:58.663 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:58.663 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:58.663 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 2, 2021-02-10T11:47:58.664 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:47:58.664 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:47:58.664 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:47:58.664 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:47:58.664 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:47:58.664 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:58.664 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:58.664 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 3, 2021-02-10T11:47:58.664 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:47:58.664 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:47:58.664 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:47:58.664 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:47:58.664 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:47:58.664 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:58.665 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:58.665 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 4, 2021-02-10T11:47:58.665 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:47:58.665 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:47:58.665 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:47:58.665 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:47:58.665 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:47:58.665 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:58.665 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:58.665 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 5, 2021-02-10T11:47:58.665 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:47:58.665 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:47:58.665 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:47:58.665 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:47:58.665 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:47:58.666 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:58.666 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:58.666 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 6, 2021-02-10T11:47:58.666 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:47:58.666 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:47:58.666 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:47:58.666 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:47:58.666 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:47:58.666 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:58.666 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:47:58.666 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 7, 2021-02-10T11:47:58.666 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:47:58.666 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:47:58.667 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:47:58.667 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:47:58.667 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:47:58.667 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:47:58.667 INFO:teuthology.orchestra.run.gibba012.stdout: ], 2021-02-10T11:47:58.667 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_upmap": [], 2021-02-10T11:47:58.667 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_upmap_items": [], 2021-02-10T11:47:58.667 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_temp": [], 2021-02-10T11:47:58.667 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_temp": [], 2021-02-10T11:47:58.667 INFO:teuthology.orchestra.run.gibba012.stdout: "blacklist": {}, 2021-02-10T11:47:58.667 INFO:teuthology.orchestra.run.gibba012.stdout: "erasure_code_profiles": { 2021-02-10T11:47:58.667 INFO:teuthology.orchestra.run.gibba012.stdout: "default": { 2021-02-10T11:47:58.667 INFO:teuthology.orchestra.run.gibba012.stdout: "crush-failure-domain": "osd", 2021-02-10T11:47:58.667 INFO:teuthology.orchestra.run.gibba012.stdout: "k": "2", 2021-02-10T11:47:58.668 INFO:teuthology.orchestra.run.gibba012.stdout: "m": "1", 2021-02-10T11:47:58.668 INFO:teuthology.orchestra.run.gibba012.stdout: "plugin": "jerasure", 2021-02-10T11:47:58.668 INFO:teuthology.orchestra.run.gibba012.stdout: "ruleset-failure-domain": "osd", 2021-02-10T11:47:58.668 INFO:teuthology.orchestra.run.gibba012.stdout: "technique": "reed_sol_van" 2021-02-10T11:47:58.668 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:47:58.668 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:47:58.668 INFO:teuthology.orchestra.run.gibba012.stdout: "removed_snaps_queue": [], 2021-02-10T11:47:58.668 INFO:teuthology.orchestra.run.gibba012.stdout: "new_removed_snaps": [], 2021-02-10T11:47:58.668 INFO:teuthology.orchestra.run.gibba012.stdout: "new_purged_snaps": [], 2021-02-10T11:47:58.668 INFO:teuthology.orchestra.run.gibba012.stdout: "crush_node_flags": {}, 2021-02-10T11:47:58.668 INFO:teuthology.orchestra.run.gibba012.stdout: "device_class_flags": {} 2021-02-10T11:47:58.668 INFO:teuthology.orchestra.run.gibba012.stdout:} 2021-02-10T11:47:58.670 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:58.595 7f473941c700 1 -- 172.21.2.112:0/3677154412 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f47140186c0 msgr2=0x7f471401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:58.670 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:58.596 7f473941c700 1 --2- 172.21.2.112:0/3677154412 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f47140186c0 0x7f471401abc0 crc :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:58.670 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:58.596 7f473941c700 1 -- 172.21.2.112:0/3677154412 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f47340b32d0 msgr2=0x7f47340b11e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:58.670 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:58.596 7f473941c700 1 --2- 172.21.2.112:0/3677154412 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f47340b32d0 0x7f47340b11e0 secure :-1 s=READY pgs=3101 cs=0 l=1 rev1=1 rx=0x7f472800a590 tx=0x7f472800a930).stop 2021-02-10T11:47:58.670 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:58.596 7f473941c700 1 -- 172.21.2.112:0/3677154412 shutdown_connections 2021-02-10T11:47:58.670 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:58.596 7f473941c700 1 --2- 172.21.2.112:0/3677154412 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f47140186c0 0x7f471401abc0 unknown :-1 s=CLOSED pgs=1394 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:58.670 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:58.596 7f473941c700 1 --2- 172.21.2.112:0/3677154412 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f47340b1800 0x7f47340aaa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:58.670 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:58.596 7f473941c700 1 --2- 172.21.2.112:0/3677154412 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f47340ab090 0x7f47340ab490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:58.670 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:58.596 7f473941c700 1 --2- 172.21.2.112:0/3677154412 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f47340b32d0 0x7f47340b11e0 unknown :-1 s=CLOSED pgs=3101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:58.670 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:58.596 7f473941c700 1 -- 172.21.2.112:0/3677154412 >> 172.21.2.112:0/3677154412 conn(0x7f473410a820 msgr2=0x7f4734101c00 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:58.670 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:58.596 7f473941c700 1 -- 172.21.2.112:0/3677154412 shutdown_connections 2021-02-10T11:47:58.670 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:58.596 7f473941c700 1 -- 172.21.2.112:0/3677154412 wait complete. 2021-02-10T11:47:58.671 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:47:59.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.082 7f883d1f9700 1 Processor -- start 2021-02-10T11:47:59.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.082 7f883d1f9700 1 -- start start 2021-02-10T11:47:59.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.082 7f883d1f9700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f88380d1870 0x7f88381b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:59.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.082 7f883d1f9700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f88380f0650 0x7f88380cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:59.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.082 7f883d1f9700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88380f0b10 con 0x7f88381b0a10 2021-02-10T11:47:59.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.082 7f883d1f9700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88380e5ed0 con 0x7f88380d1870 2021-02-10T11:47:59.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.082 7f883d1f9700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88380d1c70 con 0x7f88380f0650 2021-02-10T11:47:59.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.083 7f883759e700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f88380f0650 0x7f88380cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:59.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.083 7f8836d9d700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f88380d1870 0x7f88381b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:59.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.083 7f883659c700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7f88381b0a10 0x7f88380cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:53836/0 (socket says 172.21.2.112:53836) 2021-02-10T11:47:59.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.083 7f883759e700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f88380f0650 0x7f88380cfb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3301/0 says I am v2:172.21.2.112:60500/0 (socket says 172.21.2.112:60500) 2021-02-10T11:47:59.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.083 7f883659c700 1 -- 172.21.2.112:0/1797402938 learned_addr learned my addr 172.21.2.112:0/1797402938 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:47:59.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.083 7f8836d9d700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f88380d1870 0x7f88381b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3300/0 says I am v2:172.21.2.112:34862/0 (socket says 172.21.2.112:34862) 2021-02-10T11:47:59.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.083 7f883759e700 1 -- 172.21.2.112:0/1797402938 >> v2:172.21.2.120:3300/0 conn(0x7f88380d1870 msgr2=0x7f88381b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:59.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.083 7f883759e700 1 --2- 172.21.2.112:0/1797402938 >> v2:172.21.2.120:3300/0 conn(0x7f88380d1870 0x7f88381b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:59.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.083 7f883759e700 1 -- 172.21.2.112:0/1797402938 >> v1:172.21.2.120:6789/0 conn(0x7f88381b0a10 legacy=0x7f88380cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:59.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.083 7f883759e700 1 -- 172.21.2.112:0/1797402938 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88380d1e90 con 0x7f88380f0650 2021-02-10T11:47:59.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.083 7f883759e700 1 --2- 172.21.2.112:0/1797402938 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f88380f0650 0x7f88380cfb20 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 rx=0x7f8830006e50 tx=0x7f8830009e70).ready entity=mon.2 client_cookie=6b3af5f2aae7e5d5 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:59.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.083 7f8835d9b700 1 -- 172.21.2.112:0/1797402938 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f883000b300 con 0x7f88380f0650 2021-02-10T11:47:59.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.083 7f8835d9b700 1 -- 172.21.2.112:0/1797402938 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8830003030 con 0x7f88380f0650 2021-02-10T11:47:59.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.083 7f8835d9b700 1 -- 172.21.2.112:0/1797402938 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f883000b770 con 0x7f88380f0650 2021-02-10T11:47:59.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.084 7f883d1f9700 1 -- 172.21.2.112:0/1797402938 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f88380f0650 msgr2=0x7f88380cfb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:59.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.084 7f883d1f9700 1 --2- 172.21.2.112:0/1797402938 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f88380f0650 0x7f88380cfb20 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 rx=0x7f8830006e50 tx=0x7f8830009e70).stop 2021-02-10T11:47:59.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.084 7f883d1f9700 1 -- 172.21.2.112:0/1797402938 shutdown_connections 2021-02-10T11:47:59.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.084 7f883d1f9700 1 --2- 172.21.2.112:0/1797402938 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f88380f0650 0x7f88380cfb20 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:59.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.084 7f883d1f9700 1 --2- 172.21.2.112:0/1797402938 >> v2:172.21.2.120:3300/0 conn(0x7f88380d1870 0x7f88381b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:59.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.084 7f883d1f9700 1 -- 172.21.2.112:0/1797402938 >> 172.21.2.112:0/1797402938 conn(0x7f88380ce450 msgr2=0x7f88380d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:59.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.084 7f883d1f9700 1 -- 172.21.2.112:0/1797402938 shutdown_connections 2021-02-10T11:47:59.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.084 7f883d1f9700 1 -- 172.21.2.112:0/1797402938 wait complete. 2021-02-10T11:47:59.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.098 7f883d1f9700 1 Processor -- start 2021-02-10T11:47:59.104 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.099 7f883d1f9700 1 -- start start 2021-02-10T11:47:59.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.099 7f883d1f9700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f88380ada90 0x7f88380ade90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:59.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.099 7f883d1f9700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8838099d20 0x7f883809a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:59.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.099 7f883d1f9700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f883809a740 0x7f8838097bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:59.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.099 7f883d1f9700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88380997a0 con 0x7f883809a740 2021-02-10T11:47:59.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.099 7f883d1f9700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f883818f0c0 con 0x7f8838099d20 2021-02-10T11:47:59.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.099 7f883d1f9700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f883809ab40 con 0x7f88380ada90 2021-02-10T11:47:59.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.099 7f883759e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f883809a740 0x7f8838097bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:59.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.099 7f883759e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f883809a740 0x7f8838097bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:60824/0 (socket says 172.21.2.112:60824) 2021-02-10T11:47:59.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.099 7f883759e700 1 -- 172.21.2.112:0/1225308415 learned_addr learned my addr 172.21.2.112:0/1225308415 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:47:59.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.099 7f8836d9d700 1 --2- 172.21.2.112:0/1225308415 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f88380ada90 0x7f88380ade90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:59.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.099 7f883659c700 1 --2- 172.21.2.112:0/1225308415 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8838099d20 0x7f883809a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:59.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.100 7f883759e700 1 -- 172.21.2.112:0/1225308415 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f88380ada90 msgr2=0x7f88380ade90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:59.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.100 7f883759e700 1 --2- 172.21.2.112:0/1225308415 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f88380ada90 0x7f88380ade90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:59.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.100 7f883759e700 1 -- 172.21.2.112:0/1225308415 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8838099d20 msgr2=0x7f883809a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:59.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.100 7f883759e700 1 --2- 172.21.2.112:0/1225308415 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8838099d20 0x7f883809a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:59.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.100 7f883759e700 1 -- 172.21.2.112:0/1225308415 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f883809ad60 con 0x7f883809a740 2021-02-10T11:47:59.106 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.100 7f883759e700 1 --2- 172.21.2.112:0/1225308415 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f883809a740 0x7f8838097bc0 secure :-1 s=READY pgs=3102 cs=0 l=1 rev1=1 rx=0x7f8830009f70 tx=0x7f883000bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:59.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.100 7f882f7fe700 1 -- 172.21.2.112:0/1225308415 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8830006970 con 0x7f883809a740 2021-02-10T11:47:59.107 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.100 7f882f7fe700 1 -- 172.21.2.112:0/1225308415 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f883000c2f0 con 0x7f883809a740 2021-02-10T11:47:59.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.100 7f882f7fe700 1 -- 172.21.2.112:0/1225308415 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f883000c750 con 0x7f883809a740 2021-02-10T11:47:59.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.100 7f883d1f9700 1 -- 172.21.2.112:0/1225308415 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f883809ad60 con 0x7f883809a740 2021-02-10T11:47:59.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.100 7f883d1f9700 1 -- 172.21.2.112:0/1225308415 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88380984d0 con 0x7f883809a740 2021-02-10T11:47:59.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.101 7f882f7fe700 1 -- 172.21.2.112:0/1225308415 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8830002020 con 0x7f883809a740 2021-02-10T11:47:59.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.101 7f882f7fe700 1 --2- 172.21.2.112:0/1225308415 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f881c0186c0 0x7f881c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:59.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.101 7f882f7fe700 1 -- 172.21.2.112:0/1225308415 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f88300327a0 con 0x7f883809a740 2021-02-10T11:47:59.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.101 7f8836d9d700 1 --2- 172.21.2.112:0/1225308415 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f881c0186c0 0x7f881c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:59.108 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.101 7f8836d9d700 1 --2- 172.21.2.112:0/1225308415 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f881c0186c0 0x7f881c01abc0 crc :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:59.135 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.132 7f883d1f9700 1 -- 172.21.2.112:0/1225308415 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f883818f070 con 0x7f883809a740 2021-02-10T11:47:59.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.135 7f882f7fe700 1 -- 172.21.2.112:0/1225308415 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f88300023a0 con 0x7f883809a740 2021-02-10T11:47:59.264 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.261 7f883d1f9700 1 -- 172.21.2.112:0/1225308415 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f883818f070 con 0x7f883809a740 2021-02-10T11:47:59.265 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.262 7f882f7fe700 1 -- 172.21.2.112:0/1225308415 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 157 v157) v1 ==== 95+0+2108 (secure 0 0 0) 0x7f8830002000 con 0x7f883809a740 2021-02-10T11:47:59.265 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:47:59.266 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":157,"default_fscid":19,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":13038,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/4134422613","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":4134422613},{"type":"v1","addr":"172.21.2.112:6833","nonce":4134422613}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":152}],"filesystems":[{"mdsmap":{"epoch":156,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:47:14.425711","modified":"2021-02-10 11:47:16.434766","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9307},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9307":{"gid":9307,"name":"b","rank":0,"incarnation":153,"state":"up:active","state_seq":3,"addr":"172.21.2.120:6835/2625088905","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":2625088905},{"type":"v1","addr":"172.21.2.120:6835","nonce":2625088905}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[39],"metadata_pool":38,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":19}]} 2021-02-10T11:47:59.266 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 157 2021-02-10T11:47:59.267 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.264 7f883d1f9700 1 -- 172.21.2.112:0/1225308415 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f881c0186c0 msgr2=0x7f881c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:59.267 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.264 7f883d1f9700 1 --2- 172.21.2.112:0/1225308415 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f881c0186c0 0x7f881c01abc0 crc :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:59.267 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.264 7f883d1f9700 1 -- 172.21.2.112:0/1225308415 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f883809a740 msgr2=0x7f8838097bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:59.267 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.264 7f883d1f9700 1 --2- 172.21.2.112:0/1225308415 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f883809a740 0x7f8838097bc0 secure :-1 s=READY pgs=3102 cs=0 l=1 rev1=1 rx=0x7f8830009f70 tx=0x7f883000bcd0).stop 2021-02-10T11:47:59.267 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.264 7f883d1f9700 1 -- 172.21.2.112:0/1225308415 shutdown_connections 2021-02-10T11:47:59.267 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.264 7f883d1f9700 1 --2- 172.21.2.112:0/1225308415 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f881c0186c0 0x7f881c01abc0 unknown :-1 s=CLOSED pgs=1395 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:59.267 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.264 7f883d1f9700 1 --2- 172.21.2.112:0/1225308415 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8838099d20 0x7f883809a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:59.267 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.264 7f883d1f9700 1 --2- 172.21.2.112:0/1225308415 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f883809a740 0x7f8838097bc0 unknown :-1 s=CLOSED pgs=3102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:59.267 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.264 7f883d1f9700 1 --2- 172.21.2.112:0/1225308415 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f88380ada90 0x7f88380ade90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:59.267 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.264 7f883d1f9700 1 -- 172.21.2.112:0/1225308415 >> 172.21.2.112:0/1225308415 conn(0x7f88380f0550 msgr2=0x7f8838191730 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:59.268 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.264 7f883d1f9700 1 -- 172.21.2.112:0/1225308415 shutdown_connections 2021-02-10T11:47:59.268 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.264 7f883d1f9700 1 -- 172.21.2.112:0/1225308415 wait complete. 2021-02-10T11:47:59.276 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs fail cephfs 2021-02-10T11:47:59.707 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.687 7ffb82f79700 1 Processor -- start 2021-02-10T11:47:59.707 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.687 7ffb82f79700 1 -- start start 2021-02-10T11:47:59.707 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.687 7ffb82f79700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7ffb7c0d0340 0x7ffb7c0ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:59.707 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.687 7ffb82f79700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ffb7c0d1c70 0x7ffb7c0efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:59.707 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.687 7ffb82f79700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb7c0f0b10 con 0x7ffb7c0d1870 2021-02-10T11:47:59.707 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.687 7ffb82f79700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb7c0e5ed0 con 0x7ffb7c0d0340 2021-02-10T11:47:59.707 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.687 7ffb82f79700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb7c1b0be0 con 0x7ffb7c0d1c70 2021-02-10T11:47:59.707 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.688 7ffb80d95700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7ffb7c0d1870 0x7ffb7c1b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:60290/0 (socket says 172.21.2.112:60290) 2021-02-10T11:47:59.707 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.688 7ffb80d95700 1 -- 172.21.2.112:0/482642371 learned_addr learned my addr 172.21.2.112:0/482642371 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:47:59.707 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.688 7ffb81596700 1 --2- 172.21.2.112:0/482642371 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ffb7c0d1c70 0x7ffb7c0efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:59.708 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.688 7ffb7bfff700 1 --2- 172.21.2.112:0/482642371 >> v2:172.21.2.120:3300/0 conn(0x7ffb7c0d0340 0x7ffb7c0ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:59.708 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.688 7ffb7b7fe700 1 -- 172.21.2.112:0/482642371 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2037878020 0 0) 0x7ffb64001180 con 0x7ffb7c0d1870 2021-02-10T11:47:59.708 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.688 7ffb7b7fe700 1 -- 172.21.2.112:0/482642371 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb68002fe0 con 0x7ffb7c0d1870 2021-02-10T11:47:59.708 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.688 7ffb81596700 1 -- 172.21.2.112:0/482642371 >> v2:172.21.2.120:3300/0 conn(0x7ffb7c0d0340 msgr2=0x7ffb7c0ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:59.708 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.688 7ffb81596700 1 --2- 172.21.2.112:0/482642371 >> v2:172.21.2.120:3300/0 conn(0x7ffb7c0d0340 0x7ffb7c0ef640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:59.708 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.688 7ffb81596700 1 -- 172.21.2.112:0/482642371 >> v1:172.21.2.112:6789/0 conn(0x7ffb7c0d1870 legacy=0x7ffb7c1b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:59.708 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.688 7ffb81596700 1 -- 172.21.2.112:0/482642371 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb7c0f20a0 con 0x7ffb7c0d1c70 2021-02-10T11:47:59.708 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.688 7ffb7bfff700 1 --2- 172.21.2.112:0/482642371 >> v2:172.21.2.120:3300/0 conn(0x7ffb7c0d0340 0x7ffb7c0ef640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:47:59.708 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.689 7ffb81596700 1 --2- 172.21.2.112:0/482642371 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ffb7c0d1c70 0x7ffb7c0efc60 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7ffb74006e50 tx=0x7ffb74009e70).ready entity=mon.2 client_cookie=b67628a934f945d9 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:59.708 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.689 7ffb7b7fe700 1 -- 172.21.2.112:0/482642371 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffb7400b510 con 0x7ffb7c0d1c70 2021-02-10T11:47:59.708 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.689 7ffb7b7fe700 1 -- 172.21.2.112:0/482642371 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb74006790 con 0x7ffb7c0d1c70 2021-02-10T11:47:59.708 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.689 7ffb7b7fe700 1 -- 172.21.2.112:0/482642371 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffb7400b9d0 con 0x7ffb7c0d1c70 2021-02-10T11:47:59.708 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.689 7ffb82f79700 1 -- 172.21.2.112:0/482642371 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ffb7c0d1c70 msgr2=0x7ffb7c0efc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:59.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.689 7ffb82f79700 1 --2- 172.21.2.112:0/482642371 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ffb7c0d1c70 0x7ffb7c0efc60 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7ffb74006e50 tx=0x7ffb74009e70).stop 2021-02-10T11:47:59.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.689 7ffb82f79700 1 -- 172.21.2.112:0/482642371 shutdown_connections 2021-02-10T11:47:59.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.689 7ffb82f79700 1 --2- 172.21.2.112:0/482642371 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ffb7c0d1c70 0x7ffb7c0efc60 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:59.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.689 7ffb82f79700 1 --2- 172.21.2.112:0/482642371 >> v2:172.21.2.120:3300/0 conn(0x7ffb7c0d0340 0x7ffb7c0ef640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:59.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.689 7ffb82f79700 1 -- 172.21.2.112:0/482642371 >> 172.21.2.112:0/482642371 conn(0x7ffb7c0ce450 msgr2=0x7ffb7c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:59.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.689 7ffb82f79700 1 -- 172.21.2.112:0/482642371 shutdown_connections 2021-02-10T11:47:59.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.689 7ffb82f79700 1 -- 172.21.2.112:0/482642371 wait complete. 2021-02-10T11:47:59.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.704 7ffb82f79700 1 Processor -- start 2021-02-10T11:47:59.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.704 7ffb82f79700 1 -- start start 2021-02-10T11:47:59.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.705 7ffb82f79700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ffb7c0ebe80 0x7ffb7c0b8730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:59.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.705 7ffb82f79700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ffb7c0afa30 0x7ffb7c0afe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:59.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.705 7ffb82f79700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ffb7c0b0450 0x7ffb7c0ec8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:59.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.705 7ffb82f79700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb7c0b8d50 con 0x7ffb7c0b0450 2021-02-10T11:47:59.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.705 7ffb82f79700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb7c09df70 con 0x7ffb7c0afa30 2021-02-10T11:47:59.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.705 7ffb82f79700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb7c0b0850 con 0x7ffb7c0ebe80 2021-02-10T11:47:59.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.705 7ffb81596700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ffb7c0b0450 0x7ffb7c0ec8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:59.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.705 7ffb81596700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ffb7c0b0450 0x7ffb7c0ec8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:60832/0 (socket says 172.21.2.112:60832) 2021-02-10T11:47:59.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.705 7ffb81596700 1 -- 172.21.2.112:0/3577611660 learned_addr learned my addr 172.21.2.112:0/3577611660 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:47:59.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.705 7ffb81596700 1 -- 172.21.2.112:0/3577611660 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ffb7c0ebe80 msgr2=0x7ffb7c0b8730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:59.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.705 7ffb81596700 1 --2- 172.21.2.112:0/3577611660 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ffb7c0ebe80 0x7ffb7c0b8730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:59.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.705 7ffb81596700 1 -- 172.21.2.112:0/3577611660 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ffb7c0afa30 msgr2=0x7ffb7c0afe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:59.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.705 7ffb81596700 1 --2- 172.21.2.112:0/3577611660 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ffb7c0afa30 0x7ffb7c0afe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:59.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.705 7ffb81596700 1 -- 172.21.2.112:0/3577611660 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb7c0ecef0 con 0x7ffb7c0b0450 2021-02-10T11:47:59.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.705 7ffb81596700 1 --2- 172.21.2.112:0/3577611660 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ffb7c0b0450 0x7ffb7c0ec8d0 secure :-1 s=READY pgs=3104 cs=0 l=1 rev1=1 rx=0x7ffb740099b0 tx=0x7ffb7400bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:59.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.705 7ffb797fa700 1 -- 172.21.2.112:0/3577611660 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffb74002030 con 0x7ffb7c0b0450 2021-02-10T11:47:59.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.705 7ffb797fa700 1 -- 172.21.2.112:0/3577611660 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb74002370 con 0x7ffb7c0b0450 2021-02-10T11:47:59.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.705 7ffb82f79700 1 -- 172.21.2.112:0/3577611660 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb7c0ecef0 con 0x7ffb7c0b0450 2021-02-10T11:47:59.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.705 7ffb797fa700 1 -- 172.21.2.112:0/3577611660 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffb740027d0 con 0x7ffb7c0b0450 2021-02-10T11:47:59.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.705 7ffb82f79700 1 -- 172.21.2.112:0/3577611660 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb7c0ecef0 con 0x7ffb7c0b0450 2021-02-10T11:47:59.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.706 7ffb797fa700 1 -- 172.21.2.112:0/3577611660 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ffb74002550 con 0x7ffb7c0b0450 2021-02-10T11:47:59.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.706 7ffb797fa700 1 --2- 172.21.2.112:0/3577611660 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ffb600186d0 0x7ffb6001abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:59.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.706 7ffb797fa700 1 -- 172.21.2.112:0/3577611660 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 5573+0+0 (secure 0 0 0) 0x7ffb740329a0 con 0x7ffb7c0b0450 2021-02-10T11:47:59.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.707 7ffb80d95700 1 --2- 172.21.2.112:0/3577611660 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ffb600186d0 0x7ffb6001abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:59.712 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.707 7ffb80d95700 1 --2- 172.21.2.112:0/3577611660 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ffb600186d0 0x7ffb6001abd0 crc :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:59.740 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.737 7ffb82f79700 1 -- 172.21.2.112:0/3577611660 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb7c0b8d50 con 0x7ffb7c0b0450 2021-02-10T11:47:59.742 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.740 7ffb797fa700 1 -- 172.21.2.112:0/3577611660 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ffb7400be10 con 0x7ffb7c0b0450 2021-02-10T11:47:59.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:47:59.867 7ffb82f79700 1 -- 172.21.2.112:0/3577611660 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7ffb7c0b8d50 con 0x7ffb7c0b0450 2021-02-10T11:48:00.296 INFO:tasks.ceph.mon.a.gibba012.stderr:2021-02-10 11:48:00.293 7fc3fe6e2700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-02-10T11:48:00.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:00.297 7ffb797fa700 1 -- 172.21.2.112:0/3577611660 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v158) v1 ==== 161+0+0 (secure 0 0 0) 0x7ffb74002260 con 0x7ffb7c0b0450 2021-02-10T11:48:00.336 INFO:teuthology.orchestra.run.gibba012.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2021-02-10T11:48:00.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:00.335 7ffb82f79700 1 -- 172.21.2.112:0/3577611660 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ffb600186d0 msgr2=0x7ffb6001abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:00.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:00.335 7ffb82f79700 1 --2- 172.21.2.112:0/3577611660 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ffb600186d0 0x7ffb6001abd0 crc :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:00.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:00.335 7ffb82f79700 1 -- 172.21.2.112:0/3577611660 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ffb7c0b0450 msgr2=0x7ffb7c0ec8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:00.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:00.335 7ffb82f79700 1 --2- 172.21.2.112:0/3577611660 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ffb7c0b0450 0x7ffb7c0ec8d0 secure :-1 s=READY pgs=3104 cs=0 l=1 rev1=1 rx=0x7ffb740099b0 tx=0x7ffb7400bcd0).stop 2021-02-10T11:48:00.339 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:00.335 7ffb82f79700 1 -- 172.21.2.112:0/3577611660 shutdown_connections 2021-02-10T11:48:00.339 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:00.335 7ffb82f79700 1 --2- 172.21.2.112:0/3577611660 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ffb600186d0 0x7ffb6001abd0 unknown :-1 s=CLOSED pgs=1396 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:00.339 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:00.335 7ffb82f79700 1 --2- 172.21.2.112:0/3577611660 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ffb7c0afa30 0x7ffb7c0afe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:00.339 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:00.335 7ffb82f79700 1 --2- 172.21.2.112:0/3577611660 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ffb7c0b0450 0x7ffb7c0ec8d0 unknown :-1 s=CLOSED pgs=3104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:00.339 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:00.335 7ffb82f79700 1 --2- 172.21.2.112:0/3577611660 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ffb7c0ebe80 0x7ffb7c0b8730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:00.339 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:00.335 7ffb82f79700 1 -- 172.21.2.112:0/3577611660 >> 172.21.2.112:0/3577611660 conn(0x7ffb7c0a6b90 msgr2=0x7ffb7c0af100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:00.339 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:00.336 7ffb82f79700 1 -- 172.21.2.112:0/3577611660 shutdown_connections 2021-02-10T11:48:00.339 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:00.336 7ffb82f79700 1 -- 172.21.2.112:0/3577611660 wait complete. 2021-02-10T11:48:00.347 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:48:00.352 INFO:tasks.ceph.mds.b.gibba020.stdout:starting mds.b at 2021-02-10T11:48:00.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:00.756 7f6bcff1b700 1 Processor -- start 2021-02-10T11:48:00.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:00.756 7f6bcff1b700 1 -- start start 2021-02-10T11:48:00.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:00.756 7f6bcff1b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6bc80d1870 0x7f6bc81b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:00.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:00.756 7f6bcff1b700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f6bc81b0a10 0x7f6bc80cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:00.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:00.756 7f6bcff1b700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bc80f0b10 con 0x7f6bc80d1c70 2021-02-10T11:48:00.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:00.756 7f6bcff1b700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bc80fe230 con 0x7f6bc81b0a10 2021-02-10T11:48:00.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:00.756 7f6bcff1b700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bc80e5ed0 con 0x7f6bc80d1870 2021-02-10T11:48:00.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:00.756 7f6bcd536700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f6bc81b0a10 0x7f6bc80cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:00.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:00.756 7f6bce538700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f6bc80d1c70 0x7f6bc80cf950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:60302/0 (socket says 172.21.2.112:60302) 2021-02-10T11:48:00.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:00.756 7f6bce538700 1 -- 172.21.2.112:0/2150470341 learned_addr learned my addr 172.21.2.112:0/2150470341 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:48:00.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:00.757 7f6bcd536700 1 -- 172.21.2.112:0/2150470341 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6bc80d1870 msgr2=0x7f6bc81b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:00.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:00.757 7f6bcd536700 1 --2- 172.21.2.112:0/2150470341 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6bc80d1870 0x7f6bc81b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:00.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:00.757 7f6bcd536700 1 -- 172.21.2.112:0/2150470341 >> v1:172.21.2.112:6789/0 conn(0x7f6bc80d1c70 legacy=0x7f6bc80cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:00.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:00.757 7f6bcd536700 1 -- 172.21.2.112:0/2150470341 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bc80f0650 con 0x7f6bc81b0a10 2021-02-10T11:48:00.777 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:00.757 7f6bcd536700 1 --2- 172.21.2.112:0/2150470341 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6bc81b0a10 0x7f6bc80cf330 secure :-1 s=READY pgs=3108 cs=0 l=1 rev1=1 rx=0x7f6bc4006650 tx=0x7f6bc4009a80).ready entity=mon.0 client_cookie=94052ca6ef5ac0a6 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:00.777 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:00.757 7f6bccd35700 1 -- 172.21.2.112:0/2150470341 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6bc400c0c0 con 0x7f6bc81b0a10 2021-02-10T11:48:00.777 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:00.757 7f6bccd35700 1 -- 172.21.2.112:0/2150470341 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bc400c340 con 0x7f6bc81b0a10 2021-02-10T11:48:00.777 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:00.757 7f6bccd35700 1 -- 172.21.2.112:0/2150470341 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6bc400c860 con 0x7f6bc81b0a10 2021-02-10T11:48:00.777 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:00.757 7f6bcff1b700 1 -- 172.21.2.112:0/2150470341 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6bc81b0a10 msgr2=0x7f6bc80cf330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:00.777 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:00.757 7f6bcff1b700 1 --2- 172.21.2.112:0/2150470341 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6bc81b0a10 0x7f6bc80cf330 secure :-1 s=READY pgs=3108 cs=0 l=1 rev1=1 rx=0x7f6bc4006650 tx=0x7f6bc4009a80).stop 2021-02-10T11:48:00.777 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:00.758 7f6bcff1b700 1 -- 172.21.2.112:0/2150470341 shutdown_connections 2021-02-10T11:48:00.777 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:00.758 7f6bcff1b700 1 --2- 172.21.2.112:0/2150470341 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6bc81b0a10 0x7f6bc80cf330 unknown :-1 s=CLOSED pgs=3108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:00.777 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:00.758 7f6bcff1b700 1 --2- 172.21.2.112:0/2150470341 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6bc80d1870 0x7f6bc81b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:00.777 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:00.758 7f6bcff1b700 1 -- 172.21.2.112:0/2150470341 >> 172.21.2.112:0/2150470341 conn(0x7f6bc80ce450 msgr2=0x7f6bc80d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:00.777 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:00.758 7f6bcff1b700 1 -- 172.21.2.112:0/2150470341 shutdown_connections 2021-02-10T11:48:00.777 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:00.758 7f6bcff1b700 1 -- 172.21.2.112:0/2150470341 wait complete. 2021-02-10T11:48:00.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:00.773 7f6bcff1b700 1 Processor -- start 2021-02-10T11:48:00.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:00.773 7f6bcff1b700 1 -- start start 2021-02-10T11:48:00.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:00.773 7f6bcff1b700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6bc80b4f30 0x7f6bc818b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:00.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:00.773 7f6bcff1b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6bc8093f40 0x7f6bc8094340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:00.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:00.773 7f6bcff1b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6bc8094960 0x7f6bc8187e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:00.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:00.773 7f6bcff1b700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bc8094e90 con 0x7f6bc8094960 2021-02-10T11:48:00.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:00.773 7f6bcff1b700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bc818bff0 con 0x7f6bc80b4f30 2021-02-10T11:48:00.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:00.773 7f6bcff1b700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bc808f780 con 0x7f6bc8093f40 2021-02-10T11:48:00.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:00.773 7f6bce538700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6bc8094960 0x7f6bc8187e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:00.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:00.773 7f6bce538700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6bc8094960 0x7f6bc8187e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:60848/0 (socket says 172.21.2.112:60848) 2021-02-10T11:48:00.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:00.773 7f6bce538700 1 -- 172.21.2.112:0/1827370923 learned_addr learned my addr 172.21.2.112:0/1827370923 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:48:00.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:00.774 7f6bce538700 1 -- 172.21.2.112:0/1827370923 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6bc8093f40 msgr2=0x7f6bc8094340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:00.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:00.774 7f6bce538700 1 --2- 172.21.2.112:0/1827370923 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6bc8093f40 0x7f6bc8094340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:00.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:00.774 7f6bce538700 1 -- 172.21.2.112:0/1827370923 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6bc80b4f30 msgr2=0x7f6bc818b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:00.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:00.774 7f6bce538700 1 --2- 172.21.2.112:0/1827370923 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6bc80b4f30 0x7f6bc818b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:00.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:00.774 7f6bce538700 1 -- 172.21.2.112:0/1827370923 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bc8188470 con 0x7f6bc8094960 2021-02-10T11:48:00.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:00.774 7f6bce538700 1 --2- 172.21.2.112:0/1827370923 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6bc8094960 0x7f6bc8187e50 secure :-1 s=READY pgs=3109 cs=0 l=1 rev1=1 rx=0x7f6bc000a940 tx=0x7f6bc000ace0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:00.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:00.774 7f6bba7fc700 1 -- 172.21.2.112:0/1827370923 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6bc0001270 con 0x7f6bc8094960 2021-02-10T11:48:00.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:00.774 7f6bba7fc700 1 -- 172.21.2.112:0/1827370923 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bc00070b0 con 0x7f6bc8094960 2021-02-10T11:48:00.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:00.774 7f6bba7fc700 1 -- 172.21.2.112:0/1827370923 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6bc0007480 con 0x7f6bc8094960 2021-02-10T11:48:00.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:00.774 7f6bcff1b700 1 -- 172.21.2.112:0/1827370923 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bc8188470 con 0x7f6bc8094960 2021-02-10T11:48:00.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:00.774 7f6bcff1b700 1 -- 172.21.2.112:0/1827370923 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bc8188470 con 0x7f6bc8094960 2021-02-10T11:48:00.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:00.775 7f6bba7fc700 1 -- 172.21.2.112:0/1827370923 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6bc0007290 con 0x7f6bc8094960 2021-02-10T11:48:00.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:00.775 7f6bba7fc700 1 --2- 172.21.2.112:0/1827370923 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6bb00186d0 0x7f6bb001abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:00.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:00.775 7f6bba7fc700 1 -- 172.21.2.112:0/1827370923 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(181..181 src has 1..181) v4 ==== 5659+0+0 (secure 0 0 0) 0x7f6bc0032a20 con 0x7f6bc8094960 2021-02-10T11:48:00.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:00.775 7f6bcdd37700 1 --2- 172.21.2.112:0/1827370923 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6bb00186d0 0x7f6bb001abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:00.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:00.776 7f6bcdd37700 1 --2- 172.21.2.112:0/1827370923 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6bb00186d0 0x7f6bb001abd0 crc :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:00.809 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:00.806 7f6bcff1b700 1 -- 172.21.2.112:0/1827370923 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bc80ce450 con 0x7f6bc8094960 2021-02-10T11:48:00.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:00.809 7f6bba7fc700 1 -- 172.21.2.112:0/1827370923 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6bc0007cb0 con 0x7f6bc8094960 2021-02-10T11:48:00.941 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:00.938 7f6bcff1b700 1 -- 172.21.2.112:0/1827370923 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6bc80ce450 con 0x7f6bc8094960 2021-02-10T11:48:00.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:00.939 7f6bba7fc700 1 -- 172.21.2.112:0/1827370923 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 158 v158) v1 ==== 95+0+1763 (secure 0 0 0) 0x7f6bc00070b0 con 0x7f6bc8094960 2021-02-10T11:48:00.942 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:48:00.943 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":158,"default_fscid":19,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":13038,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/4134422613","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":4134422613},{"type":"v1","addr":"172.21.2.112:6833","nonce":4134422613}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":152}],"filesystems":[{"mdsmap":{"epoch":158,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:47:14.425711","modified":"2021-02-10 11:48:00.294299","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":181,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[39],"metadata_pool":38,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":19}]} 2021-02-10T11:48:00.943 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 158 2021-02-10T11:48:00.944 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:00.941 7f6bcff1b700 1 -- 172.21.2.112:0/1827370923 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6bb00186d0 msgr2=0x7f6bb001abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:00.944 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:00.941 7f6bcff1b700 1 --2- 172.21.2.112:0/1827370923 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6bb00186d0 0x7f6bb001abd0 crc :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:00.944 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:00.941 7f6bcff1b700 1 -- 172.21.2.112:0/1827370923 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6bc8094960 msgr2=0x7f6bc8187e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:00.944 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:00.941 7f6bcff1b700 1 --2- 172.21.2.112:0/1827370923 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6bc8094960 0x7f6bc8187e50 secure :-1 s=READY pgs=3109 cs=0 l=1 rev1=1 rx=0x7f6bc000a940 tx=0x7f6bc000ace0).stop 2021-02-10T11:48:00.944 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:00.941 7f6bcff1b700 1 -- 172.21.2.112:0/1827370923 shutdown_connections 2021-02-10T11:48:00.944 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:00.941 7f6bcff1b700 1 --2- 172.21.2.112:0/1827370923 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6bb00186d0 0x7f6bb001abd0 unknown :-1 s=CLOSED pgs=1397 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:00.944 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:00.941 7f6bcff1b700 1 --2- 172.21.2.112:0/1827370923 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6bc8094960 0x7f6bc8187e50 unknown :-1 s=CLOSED pgs=3109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:00.944 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:00.941 7f6bcff1b700 1 --2- 172.21.2.112:0/1827370923 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6bc8093f40 0x7f6bc8094340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:00.944 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:00.941 7f6bcff1b700 1 --2- 172.21.2.112:0/1827370923 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6bc80b4f30 0x7f6bc818b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:00.944 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:00.941 7f6bcff1b700 1 -- 172.21.2.112:0/1827370923 >> 172.21.2.112:0/1827370923 conn(0x7f6bc819f8f0 msgr2=0x7f6bc81a5f80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:00.945 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:00.941 7f6bcff1b700 1 -- 172.21.2.112:0/1827370923 shutdown_connections 2021-02-10T11:48:00.945 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:00.941 7f6bcff1b700 1 -- 172.21.2.112:0/1827370923 wait complete. 2021-02-10T11:48:00.952 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2021-02-10T11:48:01.386 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:01.367 7f56be0d1700 1 Processor -- start 2021-02-10T11:48:01.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:01.367 7f56be0d1700 1 -- start start 2021-02-10T11:48:01.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:01.368 7f56be0d1700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f56b80d0340 0x7f56b80ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:01.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:01.368 7f56be0d1700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f56b80d1c70 0x7f56b80efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:01.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:01.368 7f56be0d1700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56b80f0b10 con 0x7f56b80d1870 2021-02-10T11:48:01.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:01.368 7f56be0d1700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56b80e5ed0 con 0x7f56b80d1c70 2021-02-10T11:48:01.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:01.368 7f56be0d1700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56b81b0be0 con 0x7f56b80d0340 2021-02-10T11:48:01.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:01.368 7f56b7fff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f56b80d1c70 0x7f56b80efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:01.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:01.368 7f56b77fe700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f56b80d1870 0x7f56b81b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:60314/0 (socket says 172.21.2.112:60314) 2021-02-10T11:48:01.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:01.368 7f56b7fff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f56b80d1c70 0x7f56b80efc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:60856/0 (socket says 172.21.2.112:60856) 2021-02-10T11:48:01.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:01.368 7f56b77fe700 1 -- 172.21.2.112:0/2980266136 learned_addr learned my addr 172.21.2.112:0/2980266136 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:48:01.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:01.368 7f56b6ffd700 1 --2- 172.21.2.112:0/2980266136 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f56b80d0340 0x7f56b80ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:01.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:01.368 7f56b6ffd700 1 -- 172.21.2.112:0/2980266136 >> v2:172.21.2.112:3300/0 conn(0x7f56b80d1c70 msgr2=0x7f56b80efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:01.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:01.368 7f56b6ffd700 1 --2- 172.21.2.112:0/2980266136 >> v2:172.21.2.112:3300/0 conn(0x7f56b80d1c70 0x7f56b80efc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:01.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:01.368 7f56b6ffd700 1 -- 172.21.2.112:0/2980266136 >> v1:172.21.2.112:6789/0 conn(0x7f56b80d1870 legacy=0x7f56b81b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:01.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:01.368 7f56b6ffd700 1 -- 172.21.2.112:0/2980266136 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56b80f20a0 con 0x7f56b80d0340 2021-02-10T11:48:01.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:01.369 7f56b6ffd700 1 --2- 172.21.2.112:0/2980266136 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f56b80d0340 0x7f56b80ef640 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0x7f56a8006650 tx=0x7f56a8009a80).ready entity=mon.2 client_cookie=de05227ace245cc8 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:01.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:01.369 7f56b67fc700 1 -- 172.21.2.112:0/2980266136 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f56a800c0c0 con 0x7f56b80d0340 2021-02-10T11:48:01.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:01.369 7f56b67fc700 1 -- 172.21.2.112:0/2980266136 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56a800c340 con 0x7f56b80d0340 2021-02-10T11:48:01.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:01.369 7f56b67fc700 1 -- 172.21.2.112:0/2980266136 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f56a800c860 con 0x7f56b80d0340 2021-02-10T11:48:01.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:01.369 7f56be0d1700 1 -- 172.21.2.112:0/2980266136 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f56b80d0340 msgr2=0x7f56b80ef640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:01.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:01.369 7f56be0d1700 1 --2- 172.21.2.112:0/2980266136 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f56b80d0340 0x7f56b80ef640 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0x7f56a8006650 tx=0x7f56a8009a80).stop 2021-02-10T11:48:01.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:01.369 7f56be0d1700 1 -- 172.21.2.112:0/2980266136 shutdown_connections 2021-02-10T11:48:01.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:01.369 7f56be0d1700 1 --2- 172.21.2.112:0/2980266136 >> v2:172.21.2.112:3300/0 conn(0x7f56b80d1c70 0x7f56b80efc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:01.389 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:01.369 7f56be0d1700 1 --2- 172.21.2.112:0/2980266136 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f56b80d0340 0x7f56b80ef640 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:01.389 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:01.369 7f56be0d1700 1 -- 172.21.2.112:0/2980266136 >> 172.21.2.112:0/2980266136 conn(0x7f56b80ce450 msgr2=0x7f56b80d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:01.389 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:01.369 7f56be0d1700 1 -- 172.21.2.112:0/2980266136 shutdown_connections 2021-02-10T11:48:01.389 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:01.369 7f56be0d1700 1 -- 172.21.2.112:0/2980266136 wait complete. 2021-02-10T11:48:01.390 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:01.384 7f56be0d1700 1 Processor -- start 2021-02-10T11:48:01.390 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:01.384 7f56be0d1700 1 -- start start 2021-02-10T11:48:01.390 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:01.384 7f56be0d1700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f56b8090bb0 0x7f56b8090fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:01.391 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:01.384 7f56be0d1700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f56b819fb70 0x7f56b819ff70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:01.391 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:01.384 7f56be0d1700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f56b81a0590 0x7f56b80a5b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:01.391 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:01.384 7f56be0d1700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56b80915d0 con 0x7f56b8090bb0 2021-02-10T11:48:01.391 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:01.384 7f56be0d1700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56b81a0990 con 0x7f56b81a0590 2021-02-10T11:48:01.391 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:01.384 7f56be0d1700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56b80ee5b0 con 0x7f56b819fb70 2021-02-10T11:48:01.391 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:01.385 7f56b77fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f56b8090bb0 0x7f56b8090fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:01.391 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:01.385 7f56b77fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f56b8090bb0 0x7f56b8090fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:60860/0 (socket says 172.21.2.112:60860) 2021-02-10T11:48:01.391 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:01.385 7f56b77fe700 1 -- 172.21.2.112:0/4073358636 learned_addr learned my addr 172.21.2.112:0/4073358636 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:48:01.391 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:01.385 7f56b77fe700 1 -- 172.21.2.112:0/4073358636 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f56b819fb70 msgr2=0x7f56b819ff70 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:48:01.391 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:01.385 7f56b77fe700 1 --2- 172.21.2.112:0/4073358636 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f56b819fb70 0x7f56b819ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:01.391 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:01.385 7f56b77fe700 1 -- 172.21.2.112:0/4073358636 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f56b81a0590 msgr2=0x7f56b80a5b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:01.391 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:01.385 7f56b77fe700 1 --2- 172.21.2.112:0/4073358636 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f56b81a0590 0x7f56b80a5b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:01.391 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:01.385 7f56b77fe700 1 -- 172.21.2.112:0/4073358636 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56b80eea00 con 0x7f56b8090bb0 2021-02-10T11:48:01.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:01.385 7f56b77fe700 1 --2- 172.21.2.112:0/4073358636 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f56b8090bb0 0x7f56b8090fb0 secure :-1 s=READY pgs=3111 cs=0 l=1 rev1=1 rx=0x7f56a0001400 tx=0x7f56a000a900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:01.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:01.385 7f56affff700 1 -- 172.21.2.112:0/4073358636 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f56a000acf0 con 0x7f56b8090bb0 2021-02-10T11:48:01.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:01.385 7f56affff700 1 -- 172.21.2.112:0/4073358636 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56a0000c90 con 0x7f56b8090bb0 2021-02-10T11:48:01.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:01.385 7f56affff700 1 -- 172.21.2.112:0/4073358636 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f56a0004590 con 0x7f56b8090bb0 2021-02-10T11:48:01.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:01.385 7f56be0d1700 1 -- 172.21.2.112:0/4073358636 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56b80eea00 con 0x7f56b8090bb0 2021-02-10T11:48:01.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:01.385 7f56be0d1700 1 -- 172.21.2.112:0/4073358636 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56b80eea00 con 0x7f56b8090bb0 2021-02-10T11:48:01.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:01.386 7f56affff700 1 -- 172.21.2.112:0/4073358636 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f56a00043a0 con 0x7f56b8090bb0 2021-02-10T11:48:01.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:01.386 7f56affff700 1 --2- 172.21.2.112:0/4073358636 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f569c0186d0 0x7f569c01abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:01.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:01.386 7f56affff700 1 -- 172.21.2.112:0/4073358636 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(181..181 src has 1..181) v4 ==== 5659+0+0 (secure 0 0 0) 0x7f56a00326b0 con 0x7f56b8090bb0 2021-02-10T11:48:01.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:01.386 7f56b6ffd700 1 --2- 172.21.2.112:0/4073358636 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f569c0186d0 0x7f569c01abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:01.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:01.387 7f56b6ffd700 1 --2- 172.21.2.112:0/4073358636 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f569c0186d0 0x7f569c01abd0 crc :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:01.421 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:01.418 7f56be0d1700 1 -- 172.21.2.112:0/4073358636 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56b80a55f0 con 0x7f56b8090bb0 2021-02-10T11:48:01.423 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:01.421 7f56affff700 1 -- 172.21.2.112:0/4073358636 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f56a0007070 con 0x7f56b8090bb0 2021-02-10T11:48:01.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:01.549 7f56be0d1700 1 -- 172.21.2.112:0/4073358636 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f56b80a55f0 con 0x7f56b8090bb0 2021-02-10T11:48:02.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:02.309 7f56affff700 1 -- 172.21.2.112:0/4073358636 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v159) v1 ==== 104+0+0 (secure 0 0 0) 0x7f56a00043a0 con 0x7f56b8090bb0 2021-02-10T11:48:02.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:02.322 7f56be0d1700 1 -- 172.21.2.112:0/4073358636 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f569c0186d0 msgr2=0x7f569c01abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:02.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:02.322 7f56be0d1700 1 --2- 172.21.2.112:0/4073358636 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f569c0186d0 0x7f569c01abd0 crc :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:02.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:02.322 7f56be0d1700 1 -- 172.21.2.112:0/4073358636 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f56b8090bb0 msgr2=0x7f56b8090fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:02.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:02.322 7f56be0d1700 1 --2- 172.21.2.112:0/4073358636 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f56b8090bb0 0x7f56b8090fb0 secure :-1 s=READY pgs=3111 cs=0 l=1 rev1=1 rx=0x7f56a0001400 tx=0x7f56a000a900).stop 2021-02-10T11:48:02.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:02.322 7f56be0d1700 1 -- 172.21.2.112:0/4073358636 shutdown_connections 2021-02-10T11:48:02.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:02.322 7f56be0d1700 1 --2- 172.21.2.112:0/4073358636 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f569c0186d0 0x7f569c01abd0 unknown :-1 s=CLOSED pgs=1398 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:02.326 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:02.322 7f56be0d1700 1 --2- 172.21.2.112:0/4073358636 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f56b81a0590 0x7f56b80a5b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:02.326 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:02.322 7f56be0d1700 1 --2- 172.21.2.112:0/4073358636 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f56b819fb70 0x7f56b819ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:02.326 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:02.322 7f56be0d1700 1 --2- 172.21.2.112:0/4073358636 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f56b8090bb0 0x7f56b8090fb0 unknown :-1 s=CLOSED pgs=3111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:02.326 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:02.322 7f56be0d1700 1 -- 172.21.2.112:0/4073358636 >> 172.21.2.112:0/4073358636 conn(0x7f56b80bb350 msgr2=0x7f56b80af280 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:02.326 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:02.322 7f56be0d1700 1 -- 172.21.2.112:0/4073358636 shutdown_connections 2021-02-10T11:48:02.326 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:02.322 7f56be0d1700 1 -- 172.21.2.112:0/4073358636 wait complete. 2021-02-10T11:48:02.333 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2021-02-10T11:48:02.767 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:02.746 7f8aaaaca700 1 Processor -- start 2021-02-10T11:48:02.767 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:02.746 7f8aaaaca700 1 -- start start 2021-02-10T11:48:02.767 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:02.747 7f8aaaaca700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f8aa410b000 0x7f8aa40cf760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:02.767 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:02.747 7f8aaaaca700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8aa40fe1d0 con 0x7f8aa40d1c80 2021-02-10T11:48:02.767 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:02.747 7f8aaaaca700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8aa41b0cc0 con 0x7f8aa40d1880 2021-02-10T11:48:02.767 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:02.747 7f8aaaaca700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8aa4109a70 con 0x7f8aa410b000 2021-02-10T11:48:02.767 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:02.747 7f8aa3fff700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f8aa40d1c80 0x7f8aa40cef70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:60328/0 (socket says 172.21.2.112:60328) 2021-02-10T11:48:02.768 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:02.747 7f8aa3fff700 1 -- 172.21.2.112:0/2398629493 learned_addr learned my addr 172.21.2.112:0/2398629493 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:48:02.768 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:02.747 7f8aa37fe700 1 -- 172.21.2.112:0/2398629493 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3877718985 0 0) 0x7f8a94000d30 con 0x7f8aa40d1c80 2021-02-10T11:48:02.768 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:02.747 7f8aa37fe700 1 -- 172.21.2.112:0/2398629493 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a90002fe0 con 0x7f8aa40d1c80 2021-02-10T11:48:02.768 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:02.747 7f8aa90e7700 1 --2- 172.21.2.112:0/2398629493 >> v2:172.21.2.120:3300/0 conn(0x7f8aa410b000 0x7f8aa40cf760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:02.768 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:02.748 7f8aa37fe700 1 -- 172.21.2.112:0/2398629493 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3235299842 0 0) 0x7f8a94001390 con 0x7f8aa40d1c80 2021-02-10T11:48:02.768 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:02.748 7f8aa37fe700 1 -- 172.21.2.112:0/2398629493 >> v2:172.21.2.120:3300/0 conn(0x7f8aa410b000 msgr2=0x7f8aa40cf760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:02.768 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:02.748 7f8aa37fe700 1 --2- 172.21.2.112:0/2398629493 >> v2:172.21.2.120:3300/0 conn(0x7f8aa410b000 0x7f8aa40cf760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:02.768 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:02.748 7f8aa37fe700 1 -- 172.21.2.112:0/2398629493 >> v1:172.21.2.120:6789/0 conn(0x7f8aa40d1880 legacy=0x7f8aa41b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:02.768 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:02.748 7f8aa37fe700 1 -- 172.21.2.112:0/2398629493 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8aa410b400 con 0x7f8aa40d1c80 2021-02-10T11:48:02.768 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:02.748 7f8aa37fe700 1 -- 172.21.2.112:0/2398629493 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f8a94001910 con 0x7f8aa40d1c80 2021-02-10T11:48:02.768 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:02.748 7f8aa37fe700 1 -- 172.21.2.112:0/2398629493 >> v1:172.21.2.112:6789/0 conn(0x7f8aa40d1c80 legacy=0x7f8aa40cef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:02.768 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:02.748 7f8aa37fe700 1 --2- 172.21.2.112:0/2398629493 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8a90003a60 0x7f8a90005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:02.769 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:02.748 7f8aa37fe700 1 --2- 172.21.2.112:0/2398629493 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8a90006790 0x7f8a90008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:02.769 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:02.748 7f8aa37fe700 1 --2- 172.21.2.112:0/2398629493 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8a90009590 0x7f8a9000bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:02.769 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:02.748 7f8aa37fe700 1 -- 172.21.2.112:0/2398629493 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a9000c300 con 0x7f8a90006790 2021-02-10T11:48:02.769 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:02.748 7f8aa37fe700 1 -- 172.21.2.112:0/2398629493 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a9000c520 con 0x7f8a90009590 2021-02-10T11:48:02.769 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:02.748 7f8aa37fe700 1 -- 172.21.2.112:0/2398629493 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a90050750 con 0x7f8a90003a60 2021-02-10T11:48:02.769 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:02.748 7f8aa3fff700 1 --2- 172.21.2.112:0/2398629493 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8a90006790 0x7f8a90008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:02.769 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:02.748 7f8aa88e6700 1 --2- 172.21.2.112:0/2398629493 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8a90003a60 0x7f8a90005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:02.769 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:02.748 7f8aa90e7700 1 --2- 172.21.2.112:0/2398629493 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8a90009590 0x7f8a9000bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:02.769 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:02.748 7f8aa3fff700 1 -- 172.21.2.112:0/2398629493 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8a90003a60 msgr2=0x7f8a90005ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:02.769 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:02.748 7f8aa3fff700 1 --2- 172.21.2.112:0/2398629493 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8a90003a60 0x7f8a90005ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:02.769 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:02.748 7f8aa3fff700 1 -- 172.21.2.112:0/2398629493 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8a90009590 msgr2=0x7f8a9000bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:02.769 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:02.748 7f8aa3fff700 1 --2- 172.21.2.112:0/2398629493 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8a90009590 0x7f8a9000bab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:02.769 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:02.748 7f8aa90e7700 1 -- 172.21.2.112:0/2398629493 reap_dead start 2021-02-10T11:48:02.769 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:02.748 7f8aa3fff700 1 -- 172.21.2.112:0/2398629493 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a900509b0 con 0x7f8a90006790 2021-02-10T11:48:02.770 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:02.748 7f8aa88e6700 1 --2- 172.21.2.112:0/2398629493 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8a90003a60 0x7f8a90005ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:48:02.770 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:02.748 7f8aa3fff700 1 --2- 172.21.2.112:0/2398629493 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8a90006790 0x7f8a90008c90 secure :-1 s=READY pgs=3113 cs=0 l=1 rev1=1 rx=0x7f8a94000ee0 tx=0x7f8a9400a310).ready entity=mon.0 client_cookie=28295e88475597ce server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:02.770 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:02.748 7f8aa37fe700 1 -- 172.21.2.112:0/2398629493 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8a94001750 con 0x7f8a90006790 2021-02-10T11:48:02.770 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:02.748 7f8aa37fe700 1 -- 172.21.2.112:0/2398629493 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a94000d00 con 0x7f8a90006790 2021-02-10T11:48:02.770 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:02.748 7f8aa37fe700 1 -- 172.21.2.112:0/2398629493 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8a940072b0 con 0x7f8a90006790 2021-02-10T11:48:02.770 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:02.748 7f8aaaaca700 1 -- 172.21.2.112:0/2398629493 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8a90006790 msgr2=0x7f8a90008c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:02.770 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:02.749 7f8aaaaca700 1 --2- 172.21.2.112:0/2398629493 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8a90006790 0x7f8a90008c90 secure :-1 s=READY pgs=3113 cs=0 l=1 rev1=1 rx=0x7f8a94000ee0 tx=0x7f8a9400a310).stop 2021-02-10T11:48:02.770 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:02.749 7f8aaaaca700 1 -- 172.21.2.112:0/2398629493 shutdown_connections 2021-02-10T11:48:02.770 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:02.749 7f8aaaaca700 1 --2- 172.21.2.112:0/2398629493 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8a90006790 0x7f8a90008c90 unknown :-1 s=CLOSED pgs=3113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:02.770 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:02.749 7f8aaaaca700 1 -- 172.21.2.112:0/2398629493 >> 172.21.2.112:0/2398629493 conn(0x7f8aa40ce430 msgr2=0x7f8aa40d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:02.770 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:02.749 7f8aaaaca700 1 -- 172.21.2.112:0/2398629493 shutdown_connections 2021-02-10T11:48:02.770 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:02.749 7f8aaaaca700 1 -- 172.21.2.112:0/2398629493 wait complete. 2021-02-10T11:48:02.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:02.764 7f8aaaaca700 1 Processor -- start 2021-02-10T11:48:02.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:02.764 7f8aaaaca700 1 -- start start 2021-02-10T11:48:02.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:02.764 7f8aaaaca700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8aa41a3570 0x7f8aa41a3970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:02.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:02.764 7f8aaaaca700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8aa409c9f0 0x7f8aa409cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:02.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:02.764 7f8aaaaca700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8aa409d410 0x7f8aa4098170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:02.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:02.764 7f8aaaaca700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8aa409c470 con 0x7f8aa409d410 2021-02-10T11:48:02.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:02.764 7f8aaaaca700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8aa40b2840 con 0x7f8aa409c9f0 2021-02-10T11:48:02.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:02.764 7f8aaaaca700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8aa40b2a20 con 0x7f8aa41a3570 2021-02-10T11:48:02.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:02.765 7f8aa90e7700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8aa409d410 0x7f8aa4098170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:02.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:02.765 7f8aa90e7700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8aa409d410 0x7f8aa4098170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:60878/0 (socket says 172.21.2.112:60878) 2021-02-10T11:48:02.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:02.765 7f8aa90e7700 1 -- 172.21.2.112:0/3378240146 learned_addr learned my addr 172.21.2.112:0/3378240146 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:48:02.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:02.765 7f8aa90e7700 1 -- 172.21.2.112:0/3378240146 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8aa41a3570 msgr2=0x7f8aa41a3970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:02.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:02.765 7f8aa90e7700 1 --2- 172.21.2.112:0/3378240146 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8aa41a3570 0x7f8aa41a3970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:02.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:02.765 7f8aa90e7700 1 -- 172.21.2.112:0/3378240146 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8aa409c9f0 msgr2=0x7f8aa409cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:02.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:02.765 7f8aa90e7700 1 --2- 172.21.2.112:0/3378240146 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8aa409c9f0 0x7f8aa409cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:02.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:02.765 7f8aa90e7700 1 -- 172.21.2.112:0/3378240146 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8aa409d880 con 0x7f8aa409d410 2021-02-10T11:48:02.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:02.765 7f8aa90e7700 1 --2- 172.21.2.112:0/3378240146 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8aa409d410 0x7f8aa4098170 secure :-1 s=READY pgs=3114 cs=0 l=1 rev1=1 rx=0x7f8a9c00b5e0 tx=0x7f8a9c00b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:02.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:02.765 7f8aa17fa700 1 -- 172.21.2.112:0/3378240146 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8a9c00bd70 con 0x7f8aa409d410 2021-02-10T11:48:02.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:02.765 7f8aa17fa700 1 -- 172.21.2.112:0/3378240146 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a9c006730 con 0x7f8aa409d410 2021-02-10T11:48:02.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:02.765 7f8aa17fa700 1 -- 172.21.2.112:0/3378240146 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8a9c002220 con 0x7f8aa409d410 2021-02-10T11:48:02.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:02.765 7f8aaaaca700 1 -- 172.21.2.112:0/3378240146 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8aa409d880 con 0x7f8aa409d410 2021-02-10T11:48:02.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:02.765 7f8aaaaca700 1 -- 172.21.2.112:0/3378240146 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8aa409d880 con 0x7f8aa409d410 2021-02-10T11:48:02.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:02.766 7f8aa17fa700 1 -- 172.21.2.112:0/3378240146 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8a9c000a70 con 0x7f8aa409d410 2021-02-10T11:48:02.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:02.766 7f8aa17fa700 1 --2- 172.21.2.112:0/3378240146 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8a88018730 0x7f8a8801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:02.774 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:02.766 7f8aa17fa700 1 -- 172.21.2.112:0/3378240146 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(181..181 src has 1..181) v4 ==== 5659+0+0 (secure 0 0 0) 0x7f8a9c0327d0 con 0x7f8aa409d410 2021-02-10T11:48:02.774 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:02.766 7f8aa88e6700 1 --2- 172.21.2.112:0/3378240146 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8a88018730 0x7f8a8801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:02.774 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:02.767 7f8aa88e6700 1 --2- 172.21.2.112:0/3378240146 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8a88018730 0x7f8a8801ac10 crc :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:02.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:02.796 7f8aaaaca700 1 -- 172.21.2.112:0/3378240146 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8aa4082a70 con 0x7f8aa409d410 2021-02-10T11:48:02.802 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:02.799 7f8aa17fa700 1 -- 172.21.2.112:0/3378240146 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8a9c0063a0 con 0x7f8aa409d410 2021-02-10T11:48:02.928 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:02.925 7f8aaaaca700 1 -- 172.21.2.112:0/3378240146 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f8aa4082a70 con 0x7f8aa409d410 2021-02-10T11:48:03.315 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:03.312 7f8aa17fa700 1 -- 172.21.2.112:0/3378240146 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v182) v1 ==== 185+0+0 (secure 0 0 0) 0x7f8a9c006730 con 0x7f8aa409d410 2021-02-10T11:48:03.344 INFO:teuthology.orchestra.run.gibba012.stderr:pool 'cephfs_metadata' removed 2021-02-10T11:48:03.345 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:03.342 7f8aaaaca700 1 -- 172.21.2.112:0/3378240146 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8a88018730 msgr2=0x7f8a8801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:03.345 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:03.342 7f8aaaaca700 1 --2- 172.21.2.112:0/3378240146 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8a88018730 0x7f8a8801ac10 crc :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:03.346 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:03.342 7f8aaaaca700 1 -- 172.21.2.112:0/3378240146 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8aa409d410 msgr2=0x7f8aa4098170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:03.346 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:03.342 7f8aaaaca700 1 --2- 172.21.2.112:0/3378240146 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8aa409d410 0x7f8aa4098170 secure :-1 s=READY pgs=3114 cs=0 l=1 rev1=1 rx=0x7f8a9c00b5e0 tx=0x7f8a9c00b980).stop 2021-02-10T11:48:03.346 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:03.343 7f8aaaaca700 1 -- 172.21.2.112:0/3378240146 shutdown_connections 2021-02-10T11:48:03.346 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:03.343 7f8aaaaca700 1 --2- 172.21.2.112:0/3378240146 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8a88018730 0x7f8a8801ac10 unknown :-1 s=CLOSED pgs=1399 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:03.346 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:03.343 7f8aaaaca700 1 --2- 172.21.2.112:0/3378240146 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8aa409c9f0 0x7f8aa409cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:03.347 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:03.343 7f8aaaaca700 1 --2- 172.21.2.112:0/3378240146 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8aa409d410 0x7f8aa4098170 unknown :-1 s=CLOSED pgs=3114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:03.347 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:03.343 7f8aaaaca700 1 --2- 172.21.2.112:0/3378240146 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8aa41a3570 0x7f8aa41a3970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:03.347 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:03.343 7f8aaaaca700 1 -- 172.21.2.112:0/3378240146 >> 172.21.2.112:0/3378240146 conn(0x7f8aa40c2cd0 msgr2=0x7f8aa40b5950 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:03.347 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:03.343 7f8aaaaca700 1 -- 172.21.2.112:0/3378240146 shutdown_connections 2021-02-10T11:48:03.347 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:03.343 7f8aaaaca700 1 -- 172.21.2.112:0/3378240146 wait complete. 2021-02-10T11:48:03.355 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2021-02-10T11:48:03.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:03.769 7f79c6a78700 1 Processor -- start 2021-02-10T11:48:03.789 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:03.769 7f79c6a78700 1 -- start start 2021-02-10T11:48:03.790 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:03.770 7f79c6a78700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f79c00d1880 0x7f79c01b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:03.790 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:03.770 7f79c6a78700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f79c00cff80 0x7f79c0109140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:03.790 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:03.770 7f79c6a78700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79c00fe1d0 con 0x7f79c01b0af0 2021-02-10T11:48:03.790 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:03.770 7f79c6a78700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79c0109ae0 con 0x7f79c00d1880 2021-02-10T11:48:03.790 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:03.770 7f79c6a78700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79c0109760 con 0x7f79c00cff80 2021-02-10T11:48:03.790 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:03.770 7f79c4894700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f79c00d1880 0x7f79c01b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:03.790 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:03.770 7f79c4894700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f79c00d1880 0x7f79c01b04d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:60886/0 (socket says 172.21.2.112:60886) 2021-02-10T11:48:03.790 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:03.770 7f79c4894700 1 -- 172.21.2.112:0/771624894 learned_addr learned my addr 172.21.2.112:0/771624894 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:48:03.790 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:03.770 7f79bffff700 1 --1- 172.21.2.112:0/771624894 >> v1:172.21.2.112:6789/0 conn(0x7f79c01b0af0 0x7f79c0108950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:60346/0 (socket says 172.21.2.112:60346) 2021-02-10T11:48:03.790 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:03.771 7f79c4894700 1 -- 172.21.2.112:0/771624894 >> v2:172.21.2.120:3300/0 conn(0x7f79c00cff80 msgr2=0x7f79c0109140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:03.790 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:03.771 7f79c4894700 1 --2- 172.21.2.112:0/771624894 >> v2:172.21.2.120:3300/0 conn(0x7f79c00cff80 0x7f79c0109140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:03.790 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:03.771 7f79c4894700 1 -- 172.21.2.112:0/771624894 >> v1:172.21.2.112:6789/0 conn(0x7f79c01b0af0 legacy=0x7f79c0108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:03.790 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:03.771 7f79bf7fe700 1 -- 172.21.2.112:0/771624894 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 471649840 0 0) 0x7f79b0001060 con 0x7f79c01b0af0 2021-02-10T11:48:03.791 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:03.771 7f79c4894700 1 -- 172.21.2.112:0/771624894 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79c0088070 con 0x7f79c00d1880 2021-02-10T11:48:03.791 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:03.771 7f79c4894700 1 --2- 172.21.2.112:0/771624894 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f79c00d1880 0x7f79c01b04d0 secure :-1 s=READY pgs=3116 cs=0 l=1 rev1=1 rx=0x7f79a8006700 tx=0x7f79a8009b00).ready entity=mon.0 client_cookie=cf8c987070a90767 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:03.791 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:03.771 7f79bf7fe700 1 -- 172.21.2.112:0/771624894 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f79a800c0c0 con 0x7f79c00d1880 2021-02-10T11:48:03.791 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:03.771 7f79bf7fe700 1 -- 172.21.2.112:0/771624894 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79a800c400 con 0x7f79c00d1880 2021-02-10T11:48:03.791 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:03.771 7f79bf7fe700 1 -- 172.21.2.112:0/771624894 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f79a800c860 con 0x7f79c00d1880 2021-02-10T11:48:03.791 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:03.771 7f79c6a78700 1 -- 172.21.2.112:0/771624894 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f79c00d1880 msgr2=0x7f79c01b04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:03.791 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:03.771 7f79c6a78700 1 --2- 172.21.2.112:0/771624894 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f79c00d1880 0x7f79c01b04d0 secure :-1 s=READY pgs=3116 cs=0 l=1 rev1=1 rx=0x7f79a8006700 tx=0x7f79a8009b00).stop 2021-02-10T11:48:03.791 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:03.772 7f79c6a78700 1 -- 172.21.2.112:0/771624894 shutdown_connections 2021-02-10T11:48:03.791 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:03.772 7f79c6a78700 1 --2- 172.21.2.112:0/771624894 >> v2:172.21.2.120:3300/0 conn(0x7f79c00cff80 0x7f79c0109140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:03.791 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:03.772 7f79c6a78700 1 --2- 172.21.2.112:0/771624894 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f79c00d1880 0x7f79c01b04d0 unknown :-1 s=CLOSED pgs=3116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:03.791 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:03.772 7f79c6a78700 1 -- 172.21.2.112:0/771624894 >> 172.21.2.112:0/771624894 conn(0x7f79c00ce430 msgr2=0x7f79c00d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:03.791 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:03.772 7f79c6a78700 1 -- 172.21.2.112:0/771624894 shutdown_connections 2021-02-10T11:48:03.791 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:03.772 7f79c6a78700 1 -- 172.21.2.112:0/771624894 wait complete. 2021-02-10T11:48:03.793 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:03.787 7f79c6a78700 1 Processor -- start 2021-02-10T11:48:03.793 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:03.787 7f79c6a78700 1 -- start start 2021-02-10T11:48:03.793 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:03.787 7f79c6a78700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f79c0090c90 0x7f79c00ada60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:03.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:03.787 7f79c6a78700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f79c01931c0 0x7f79c00936f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:03.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:03.787 7f79c6a78700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f79c00ae080 0x7f79c0093d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:03.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:03.787 7f79c6a78700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79c0091290 con 0x7f79c01931c0 2021-02-10T11:48:03.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:03.787 7f79c6a78700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79c0094330 con 0x7f79c0090c90 2021-02-10T11:48:03.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:03.787 7f79c6a78700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79c0094550 con 0x7f79c00ae080 2021-02-10T11:48:03.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:03.787 7f79bffff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f79c01931c0 0x7f79c00936f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:03.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:03.787 7f79bffff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f79c01931c0 0x7f79c00936f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:60892/0 (socket says 172.21.2.112:60892) 2021-02-10T11:48:03.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:03.787 7f79bffff700 1 -- 172.21.2.112:0/1812756365 learned_addr learned my addr 172.21.2.112:0/1812756365 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:48:03.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:03.788 7f79c5095700 1 --2- 172.21.2.112:0/1812756365 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f79c00ae080 0x7f79c0093d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:03.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:03.788 7f79c4894700 1 --2- 172.21.2.112:0/1812756365 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f79c0090c90 0x7f79c00ada60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:03.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:03.788 7f79bffff700 1 -- 172.21.2.112:0/1812756365 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f79c00ae080 msgr2=0x7f79c0093d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:03.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:03.788 7f79bffff700 1 --2- 172.21.2.112:0/1812756365 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f79c00ae080 0x7f79c0093d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:03.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:03.788 7f79bffff700 1 -- 172.21.2.112:0/1812756365 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f79c0090c90 msgr2=0x7f79c00ada60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:03.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:03.788 7f79bffff700 1 --2- 172.21.2.112:0/1812756365 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f79c0090c90 0x7f79c00ada60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:03.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:03.788 7f79bffff700 1 -- 172.21.2.112:0/1812756365 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79c0094e60 con 0x7f79c01931c0 2021-02-10T11:48:03.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:03.788 7f79bffff700 1 --2- 172.21.2.112:0/1812756365 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f79c01931c0 0x7f79c00936f0 secure :-1 s=READY pgs=3117 cs=0 l=1 rev1=1 rx=0x7f79b0009980 tx=0x7f79b000bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:03.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:03.788 7f79bd7fa700 1 -- 172.21.2.112:0/1812756365 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f79b0006300 con 0x7f79c01931c0 2021-02-10T11:48:03.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:03.788 7f79bd7fa700 1 -- 172.21.2.112:0/1812756365 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79b0006580 con 0x7f79c01931c0 2021-02-10T11:48:03.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:03.788 7f79c6a78700 1 -- 172.21.2.112:0/1812756365 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79c0094e60 con 0x7f79c01931c0 2021-02-10T11:48:03.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:03.788 7f79bd7fa700 1 -- 172.21.2.112:0/1812756365 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f79b0004910 con 0x7f79c01931c0 2021-02-10T11:48:03.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:03.788 7f79c6a78700 1 -- 172.21.2.112:0/1812756365 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79c0094e60 con 0x7f79c01931c0 2021-02-10T11:48:03.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:03.789 7f79bd7fa700 1 -- 172.21.2.112:0/1812756365 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f79b0006020 con 0x7f79c01931c0 2021-02-10T11:48:03.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:03.789 7f79bd7fa700 1 --2- 172.21.2.112:0/1812756365 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f79a4018730 0x7f79a401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:03.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:03.789 7f79bd7fa700 1 -- 172.21.2.112:0/1812756365 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(182..182 src has 1..182) v4 ==== 5221+0+0 (secure 0 0 0) 0x7f79b0032830 con 0x7f79c01931c0 2021-02-10T11:48:03.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:03.789 7f79c4894700 1 --2- 172.21.2.112:0/1812756365 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f79a4018730 0x7f79a401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:03.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:03.790 7f79c4894700 1 --2- 172.21.2.112:0/1812756365 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f79a4018730 0x7f79a401abc0 crc :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:03.824 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:03.820 7f79c6a78700 1 -- 172.21.2.112:0/1812756365 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79c0082ab0 con 0x7f79c01931c0 2021-02-10T11:48:03.826 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:03.822 7f79bd7fa700 1 -- 172.21.2.112:0/1812756365 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f79b000be10 con 0x7f79c01931c0 2021-02-10T11:48:03.954 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:03.951 7f79c6a78700 1 -- 172.21.2.112:0/1812756365 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f79c0082ab0 con 0x7f79c01931c0 2021-02-10T11:48:04.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:04.318 7f79bd7fa700 1 -- 172.21.2.112:0/1812756365 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v183) v1 ==== 173+0+0 (secure 0 0 0) 0x7f79b0006580 con 0x7f79c01931c0 2021-02-10T11:48:04.371 INFO:teuthology.orchestra.run.gibba012.stderr:pool 'cephfs_data' removed 2021-02-10T11:48:04.372 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:04.370 7f79c6a78700 1 -- 172.21.2.112:0/1812756365 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f79a4018730 msgr2=0x7f79a401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:04.373 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:04.370 7f79c6a78700 1 --2- 172.21.2.112:0/1812756365 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f79a4018730 0x7f79a401abc0 crc :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:04.373 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:04.370 7f79c6a78700 1 -- 172.21.2.112:0/1812756365 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f79c01931c0 msgr2=0x7f79c00936f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:04.373 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:04.370 7f79c6a78700 1 --2- 172.21.2.112:0/1812756365 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f79c01931c0 0x7f79c00936f0 secure :-1 s=READY pgs=3117 cs=0 l=1 rev1=1 rx=0x7f79b0009980 tx=0x7f79b000bcd0).stop 2021-02-10T11:48:04.373 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:04.370 7f79c6a78700 1 -- 172.21.2.112:0/1812756365 shutdown_connections 2021-02-10T11:48:04.373 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:04.370 7f79c6a78700 1 --2- 172.21.2.112:0/1812756365 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f79a4018730 0x7f79a401abc0 unknown :-1 s=CLOSED pgs=1400 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:04.374 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:04.370 7f79c6a78700 1 --2- 172.21.2.112:0/1812756365 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f79c00ae080 0x7f79c0093d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:04.374 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:04.370 7f79c6a78700 1 --2- 172.21.2.112:0/1812756365 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f79c01931c0 0x7f79c00936f0 unknown :-1 s=CLOSED pgs=3117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:04.374 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:04.370 7f79c6a78700 1 --2- 172.21.2.112:0/1812756365 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f79c0090c90 0x7f79c00ada60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:04.374 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:04.370 7f79c6a78700 1 -- 172.21.2.112:0/1812756365 >> 172.21.2.112:0/1812756365 conn(0x7f79c019ebd0 msgr2=0x7f79c018c290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:04.374 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:04.370 7f79c6a78700 1 -- 172.21.2.112:0/1812756365 shutdown_connections 2021-02-10T11:48:04.374 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:04.370 7f79c6a78700 1 -- 172.21.2.112:0/1812756365 wait complete. 2021-02-10T11:48:04.382 INFO:tasks.ceph.mds.a:Restarting daemon 2021-02-10T11:48:04.382 INFO:tasks.ceph.mds.a:Stopping old one... 2021-02-10T11:48:04.382 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2021-02-10T11:48:04.382 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:48:04.382 INFO:tasks.ceph.mds.b:Restarting daemon 2021-02-10T11:48:04.382 INFO:tasks.ceph.mds.b:Stopping old one... 2021-02-10T11:48:04.383 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2021-02-10T11:48:04.383 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:48:04.388 INFO:tasks.ceph.mds.a:Stopped 2021-02-10T11:48:04.388 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2021-02-10T11:48:04.389 INFO:tasks.ceph.mds.b:Stopped 2021-02-10T11:48:04.389 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2021-02-10T11:48:04.430 INFO:tasks.ceph.mds.a:Started 2021-02-10T11:48:04.430 DEBUG:teuthology.parallel:result is None 2021-02-10T11:48:04.431 INFO:tasks.ceph.mds.b:Started 2021-02-10T11:48:04.431 DEBUG:teuthology.parallel:result is None 2021-02-10T11:48:04.431 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd blacklist clear 2021-02-10T11:48:04.835 INFO:tasks.ceph.mds.b.gibba020.stdout:starting mds.b at 2021-02-10T11:48:04.840 INFO:tasks.ceph.mds.a.gibba012.stdout:starting mds.a at 2021-02-10T11:48:04.861 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:04.841 7f5168751700 1 Processor -- start 2021-02-10T11:48:04.862 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:04.841 7f5168751700 1 -- start start 2021-02-10T11:48:04.862 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:04.842 7f5168751700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f51600d1870 0x7f51601b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:04.862 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:04.842 7f5168751700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f51601b0a10 0x7f51600cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:04.862 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:04.842 7f5168751700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51600f0b10 con 0x7f51600d1c70 2021-02-10T11:48:04.862 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:04.842 7f5168751700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51600e5ed0 con 0x7f51600d1870 2021-02-10T11:48:04.862 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:04.842 7f5168751700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51600f0650 con 0x7f51601b0a10 2021-02-10T11:48:04.862 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:04.842 7f5166d6e700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f51600d1c70 0x7f51600cf950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:60374/0 (socket says 172.21.2.112:60374) 2021-02-10T11:48:04.862 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:04.842 7f5166d6e700 1 -- 172.21.2.112:0/720443791 learned_addr learned my addr 172.21.2.112:0/720443791 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:48:04.862 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:04.842 7f516656d700 1 --2- 172.21.2.112:0/720443791 >> v2:172.21.2.112:3300/0 conn(0x7f51600d1870 0x7f51601b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:04.862 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:04.842 7f5165d6c700 1 --2- 172.21.2.112:0/720443791 >> v2:172.21.2.120:3300/0 conn(0x7f51601b0a10 0x7f51600cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:04.862 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:04.843 7f516656d700 1 -- 172.21.2.112:0/720443791 >> v2:172.21.2.120:3300/0 conn(0x7f51601b0a10 msgr2=0x7f51600cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:04.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:04.843 7f516656d700 1 --2- 172.21.2.112:0/720443791 >> v2:172.21.2.120:3300/0 conn(0x7f51601b0a10 0x7f51600cf330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:04.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:04.843 7f516656d700 1 -- 172.21.2.112:0/720443791 >> v1:172.21.2.112:6789/0 conn(0x7f51600d1c70 legacy=0x7f51600cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:04.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:04.843 7f516656d700 1 -- 172.21.2.112:0/720443791 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51600f0840 con 0x7f51600d1870 2021-02-10T11:48:04.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:04.843 7f516656d700 1 --2- 172.21.2.112:0/720443791 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f51600d1870 0x7f51601b03f0 secure :-1 s=READY pgs=3120 cs=0 l=1 rev1=1 rx=0x7f5154006650 tx=0x7f5154009a80).ready entity=mon.0 client_cookie=b9d6ff953cf22d88 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:04.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:04.843 7f516556b700 1 -- 172.21.2.112:0/720443791 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f51540009b0 con 0x7f51600d1870 2021-02-10T11:48:04.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:04.843 7f516556b700 1 -- 172.21.2.112:0/720443791 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f515400c0c0 con 0x7f51600d1870 2021-02-10T11:48:04.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:04.843 7f516556b700 1 -- 172.21.2.112:0/720443791 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f515400c520 con 0x7f51600d1870 2021-02-10T11:48:04.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:04.843 7f5168751700 1 -- 172.21.2.112:0/720443791 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f51600d1870 msgr2=0x7f51601b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:04.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:04.843 7f5168751700 1 --2- 172.21.2.112:0/720443791 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f51600d1870 0x7f51601b03f0 secure :-1 s=READY pgs=3120 cs=0 l=1 rev1=1 rx=0x7f5154006650 tx=0x7f5154009a80).stop 2021-02-10T11:48:04.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:04.844 7f5168751700 1 -- 172.21.2.112:0/720443791 shutdown_connections 2021-02-10T11:48:04.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:04.844 7f5168751700 1 --2- 172.21.2.112:0/720443791 >> v2:172.21.2.120:3300/0 conn(0x7f51601b0a10 0x7f51600cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:04.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:04.844 7f5168751700 1 --2- 172.21.2.112:0/720443791 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f51600d1870 0x7f51601b03f0 unknown :-1 s=CLOSED pgs=3120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:04.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:04.844 7f5168751700 1 -- 172.21.2.112:0/720443791 >> 172.21.2.112:0/720443791 conn(0x7f51600ce450 msgr2=0x7f51600d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:04.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:04.844 7f5168751700 1 -- 172.21.2.112:0/720443791 shutdown_connections 2021-02-10T11:48:04.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:04.844 7f5168751700 1 -- 172.21.2.112:0/720443791 wait complete. 2021-02-10T11:48:04.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:04.858 7f5168751700 1 Processor -- start 2021-02-10T11:48:04.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:04.859 7f5168751700 1 -- start start 2021-02-10T11:48:04.866 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:04.859 7f5168751700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f51600a61f0 0x7f51600a65f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:04.866 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:04.859 7f5168751700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f51600ab700 0x7f51600ac220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:04.866 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:04.859 7f5168751700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f51600a6c10 0x7f51600ac840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:04.866 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:04.859 7f5168751700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5160199fc0 con 0x7f51600a6c10 2021-02-10T11:48:04.866 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:04.859 7f5168751700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51600abca0 con 0x7f51600a61f0 2021-02-10T11:48:04.866 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:04.859 7f5168751700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51600a7240 con 0x7f51600ab700 2021-02-10T11:48:04.866 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:04.859 7f5166d6e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f51600a6c10 0x7f51600ac840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:04.866 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:04.859 7f5166d6e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f51600a6c10 0x7f51600ac840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:60918/0 (socket says 172.21.2.112:60918) 2021-02-10T11:48:04.867 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:04.859 7f5166d6e700 1 -- 172.21.2.112:0/94359514 learned_addr learned my addr 172.21.2.112:0/94359514 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:48:04.867 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:04.859 7f5165d6c700 1 --2- 172.21.2.112:0/94359514 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f51600ab700 0x7f51600ac220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:04.867 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:04.859 7f516656d700 1 --2- 172.21.2.112:0/94359514 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f51600a61f0 0x7f51600a65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:04.867 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:04.859 7f5166d6e700 1 -- 172.21.2.112:0/94359514 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f51600ab700 msgr2=0x7f51600ac220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:04.867 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:04.859 7f5166d6e700 1 --2- 172.21.2.112:0/94359514 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f51600ab700 0x7f51600ac220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:04.867 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:04.859 7f5166d6e700 1 -- 172.21.2.112:0/94359514 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f51600a61f0 msgr2=0x7f51600a65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:04.867 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:04.859 7f5166d6e700 1 --2- 172.21.2.112:0/94359514 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f51600a61f0 0x7f51600a65f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:04.867 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:04.859 7f5166d6e700 1 -- 172.21.2.112:0/94359514 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51600ace60 con 0x7f51600a6c10 2021-02-10T11:48:04.867 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:04.860 7f5166d6e700 1 --2- 172.21.2.112:0/94359514 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f51600a6c10 0x7f51600ac840 secure :-1 s=READY pgs=3121 cs=0 l=1 rev1=1 rx=0x7f515800a900 tx=0x7f515800aca0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:04.867 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:04.860 7f5152ffd700 1 -- 172.21.2.112:0/94359514 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f51580070d0 con 0x7f51600a6c10 2021-02-10T11:48:04.867 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:04.860 7f5168751700 1 -- 172.21.2.112:0/94359514 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51600ace60 con 0x7f51600a6c10 2021-02-10T11:48:04.867 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:04.860 7f5152ffd700 1 -- 172.21.2.112:0/94359514 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5158000f90 con 0x7f51600a6c10 2021-02-10T11:48:04.867 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:04.860 7f5152ffd700 1 -- 172.21.2.112:0/94359514 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5158001270 con 0x7f51600a6c10 2021-02-10T11:48:04.868 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:04.860 7f5168751700 1 -- 172.21.2.112:0/94359514 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51600ace60 con 0x7f51600a6c10 2021-02-10T11:48:04.868 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:04.860 7f5152ffd700 1 -- 172.21.2.112:0/94359514 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5158007390 con 0x7f51600a6c10 2021-02-10T11:48:04.868 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:04.860 7f5152ffd700 1 --2- 172.21.2.112:0/94359514 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5148018730 0x7f514801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:04.868 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:04.860 7f5152ffd700 1 -- 172.21.2.112:0/94359514 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(183..183 src has 1..183) v4 ==== 4839+0+0 (secure 0 0 0) 0x7f51580326a0 con 0x7f51600a6c10 2021-02-10T11:48:04.868 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:04.860 7f516656d700 1 --2- 172.21.2.112:0/94359514 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5148018730 0x7f514801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:04.868 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:04.861 7f516656d700 1 --2- 172.21.2.112:0/94359514 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5148018730 0x7f514801abc0 crc :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:04.894 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:04.891 7f5168751700 1 -- 172.21.2.112:0/94359514 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51600eb960 con 0x7f51600a6c10 2021-02-10T11:48:04.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:04.893 7f5152ffd700 1 -- 172.21.2.112:0/94359514 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5158007cb0 con 0x7f51600a6c10 2021-02-10T11:48:05.036 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:05.033 7f5168751700 1 -- 172.21.2.112:0/94359514 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7f51600eb960 con 0x7f51600a6c10 2021-02-10T11:48:05.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:05.357 7f5152ffd700 1 -- 172.21.2.112:0/94359514 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v184) v1 ==== 97+0+0 (secure 0 0 0) 0x7f51580071f0 con 0x7f51600a6c10 2021-02-10T11:48:05.403 INFO:teuthology.orchestra.run.gibba012.stderr: removed all blacklist entries 2021-02-10T11:48:05.405 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:05.401 7f5168751700 1 -- 172.21.2.112:0/94359514 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5148018730 msgr2=0x7f514801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:05.405 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:05.401 7f5168751700 1 --2- 172.21.2.112:0/94359514 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5148018730 0x7f514801abc0 crc :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:05.405 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:05.401 7f5168751700 1 -- 172.21.2.112:0/94359514 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f51600a6c10 msgr2=0x7f51600ac840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:05.406 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:05.401 7f5168751700 1 --2- 172.21.2.112:0/94359514 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f51600a6c10 0x7f51600ac840 secure :-1 s=READY pgs=3121 cs=0 l=1 rev1=1 rx=0x7f515800a900 tx=0x7f515800aca0).stop 2021-02-10T11:48:05.406 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:05.402 7f5168751700 1 -- 172.21.2.112:0/94359514 shutdown_connections 2021-02-10T11:48:05.406 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:05.402 7f5168751700 1 --2- 172.21.2.112:0/94359514 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5148018730 0x7f514801abc0 unknown :-1 s=CLOSED pgs=1401 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:05.406 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:05.402 7f5168751700 1 --2- 172.21.2.112:0/94359514 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f51600a6c10 0x7f51600ac840 unknown :-1 s=CLOSED pgs=3121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:05.406 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:05.402 7f5168751700 1 --2- 172.21.2.112:0/94359514 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f51600ab700 0x7f51600ac220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:05.406 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:05.402 7f5168751700 1 --2- 172.21.2.112:0/94359514 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f51600a61f0 0x7f51600a65f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:05.406 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:05.402 7f5168751700 1 -- 172.21.2.112:0/94359514 >> 172.21.2.112:0/94359514 conn(0x7f5160093800 msgr2=0x7f51600bdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:05.406 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:05.402 7f5168751700 1 -- 172.21.2.112:0/94359514 shutdown_connections 2021-02-10T11:48:05.406 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:05.402 7f5168751700 1 -- 172.21.2.112:0/94359514 wait complete. 2021-02-10T11:48:05.413 INFO:tasks.cephfs.cephfs_test_case:['0', '1', '2', '3'] 2021-02-10T11:48:05.414 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth ls --format=json-pretty 2021-02-10T11:48:05.843 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:05.823 7f1863267700 1 Processor -- start 2021-02-10T11:48:05.843 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:05.823 7f1863267700 1 -- start start 2021-02-10T11:48:05.843 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:05.824 7f1863267700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f185c0d1870 0x7f185c1b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:05.844 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:05.824 7f1863267700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f185c0f0650 0x7f185c0cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:05.844 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:05.824 7f1863267700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f185c0f0b10 con 0x7f185c1b0a10 2021-02-10T11:48:05.844 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:05.824 7f1863267700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f185c0e5ed0 con 0x7f185c0d1870 2021-02-10T11:48:05.844 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:05.824 7f1863267700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f185c0d1c70 con 0x7f185c0f0650 2021-02-10T11:48:05.844 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:05.824 7f1861884700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f185c0f0650 0x7f185c0cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:05.844 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:05.824 7f1861083700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f185c0d1870 0x7f185c1b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:05.844 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:05.824 7f1860882700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7f185c1b0a10 0x7f185c0cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:53944/0 (socket says 172.21.2.112:53944) 2021-02-10T11:48:05.844 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:05.824 7f1860882700 1 -- 172.21.2.112:0/3956282246 learned_addr learned my addr 172.21.2.112:0/3956282246 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:48:05.844 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:05.825 7f1861884700 1 -- 172.21.2.112:0/3956282246 >> v2:172.21.2.120:3300/0 conn(0x7f185c0d1870 msgr2=0x7f185c1b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:05.844 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:05.825 7f1861884700 1 --2- 172.21.2.112:0/3956282246 >> v2:172.21.2.120:3300/0 conn(0x7f185c0d1870 0x7f185c1b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:05.844 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:05.825 7f1861884700 1 -- 172.21.2.112:0/3956282246 >> v1:172.21.2.120:6789/0 conn(0x7f185c1b0a10 legacy=0x7f185c0cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:05.844 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:05.825 7f1861884700 1 -- 172.21.2.112:0/3956282246 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f185c0d1e90 con 0x7f185c0f0650 2021-02-10T11:48:05.845 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:05.825 7f1861083700 1 --2- 172.21.2.112:0/3956282246 >> v2:172.21.2.120:3300/0 conn(0x7f185c0d1870 0x7f185c1b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:48:05.845 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:05.825 7f1861884700 1 --2- 172.21.2.112:0/3956282246 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f185c0f0650 0x7f185c0cfb20 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0x7f1854006e50 tx=0x7f1854009e70).ready entity=mon.2 client_cookie=156ba8c0c879e26a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:05.845 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:05.825 7f1853fff700 1 -- 172.21.2.112:0/3956282246 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1854000f90 con 0x7f185c0f0650 2021-02-10T11:48:05.845 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:05.825 7f1853fff700 1 -- 172.21.2.112:0/3956282246 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18540066f0 con 0x7f185c0f0650 2021-02-10T11:48:05.845 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:05.825 7f1853fff700 1 -- 172.21.2.112:0/3956282246 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f185400b700 con 0x7f185c0f0650 2021-02-10T11:48:05.845 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:05.825 7f1863267700 1 -- 172.21.2.112:0/3956282246 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f185c0f0650 msgr2=0x7f185c0cfb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:05.845 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:05.825 7f1863267700 1 --2- 172.21.2.112:0/3956282246 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f185c0f0650 0x7f185c0cfb20 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0x7f1854006e50 tx=0x7f1854009e70).stop 2021-02-10T11:48:05.845 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:05.826 7f1863267700 1 -- 172.21.2.112:0/3956282246 shutdown_connections 2021-02-10T11:48:05.845 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:05.826 7f1863267700 1 --2- 172.21.2.112:0/3956282246 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f185c0f0650 0x7f185c0cfb20 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:05.845 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:05.826 7f1863267700 1 --2- 172.21.2.112:0/3956282246 >> v2:172.21.2.120:3300/0 conn(0x7f185c0d1870 0x7f185c1b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:05.845 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:05.826 7f1863267700 1 -- 172.21.2.112:0/3956282246 >> 172.21.2.112:0/3956282246 conn(0x7f185c0ce450 msgr2=0x7f185c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:05.845 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:05.826 7f1863267700 1 -- 172.21.2.112:0/3956282246 shutdown_connections 2021-02-10T11:48:05.845 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:05.826 7f1863267700 1 -- 172.21.2.112:0/3956282246 wait complete. 2021-02-10T11:48:05.847 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:05.841 7f1863267700 1 Processor -- start 2021-02-10T11:48:05.847 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:05.841 7f1863267700 1 -- start start 2021-02-10T11:48:05.847 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:05.841 7f1863267700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f185c0ed4e0 0x7f185c090e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:05.847 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:05.841 7f1863267700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f185c0ed8e0 0x7f185c0edce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:05.848 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:05.841 7f1863267700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f185c0ee8d0 0x7f185c0eecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:05.848 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:05.841 7f1863267700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f185c0ee350 con 0x7f185c0ee8d0 2021-02-10T11:48:05.848 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:05.841 7f1863267700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f185c08e410 con 0x7f185c0ed8e0 2021-02-10T11:48:05.848 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:05.841 7f1863267700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f185c0ef2f0 con 0x7f185c0ed4e0 2021-02-10T11:48:05.848 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:05.842 7f1861884700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f185c0ee8d0 0x7f185c0eecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:05.848 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:05.842 7f1861884700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f185c0ee8d0 0x7f185c0eecd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:60932/0 (socket says 172.21.2.112:60932) 2021-02-10T11:48:05.848 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:05.842 7f1861884700 1 -- 172.21.2.112:0/55685028 learned_addr learned my addr 172.21.2.112:0/55685028 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:48:05.848 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:05.842 7f1861884700 1 -- 172.21.2.112:0/55685028 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f185c0ed4e0 msgr2=0x7f185c090e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:05.848 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:05.842 7f1861884700 1 --2- 172.21.2.112:0/55685028 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f185c0ed4e0 0x7f185c090e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:05.848 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:05.842 7f1861884700 1 -- 172.21.2.112:0/55685028 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f185c0ed8e0 msgr2=0x7f185c0edce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:05.848 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:05.842 7f1860882700 1 --2- 172.21.2.112:0/55685028 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f185c0ed8e0 0x7f185c0edce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:05.848 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:05.842 7f1861884700 1 --2- 172.21.2.112:0/55685028 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f185c0ed8e0 0x7f185c0edce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:05.848 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:05.842 7f1861884700 1 -- 172.21.2.112:0/55685028 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f185c0ef510 con 0x7f185c0ee8d0 2021-02-10T11:48:05.849 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:05.842 7f1860882700 1 --2- 172.21.2.112:0/55685028 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f185c0ed8e0 0x7f185c0edce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:48:05.849 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:05.842 7f1861884700 1 --2- 172.21.2.112:0/55685028 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f185c0ee8d0 0x7f185c0eecd0 secure :-1 s=READY pgs=3122 cs=0 l=1 rev1=1 rx=0x7f185400b8c0 tx=0x7f185400bc60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:05.849 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:05.842 7f1851ffb700 1 -- 172.21.2.112:0/55685028 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1854002030 con 0x7f185c0ee8d0 2021-02-10T11:48:05.849 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:05.843 7f1851ffb700 1 -- 172.21.2.112:0/55685028 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1854002330 con 0x7f185c0ee8d0 2021-02-10T11:48:05.849 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:05.843 7f1851ffb700 1 -- 172.21.2.112:0/55685028 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1854002780 con 0x7f185c0ee8d0 2021-02-10T11:48:05.849 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:05.843 7f1863267700 1 -- 172.21.2.112:0/55685028 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f185c0ef510 con 0x7f185c0ee8d0 2021-02-10T11:48:05.849 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:05.843 7f1863267700 1 -- 172.21.2.112:0/55685028 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f185c0efaa0 con 0x7f185c0ee8d0 2021-02-10T11:48:05.849 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:05.843 7f1851ffb700 1 -- 172.21.2.112:0/55685028 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1854002510 con 0x7f185c0ee8d0 2021-02-10T11:48:05.849 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:05.843 7f1851ffb700 1 --2- 172.21.2.112:0/55685028 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f183c0186c0 0x7f183c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:05.849 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:05.843 7f1851ffb700 1 -- 172.21.2.112:0/55685028 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(184..184 src has 1..184) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f18540325c0 con 0x7f185c0ee8d0 2021-02-10T11:48:05.849 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:05.844 7f1861083700 1 --2- 172.21.2.112:0/55685028 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f183c0186c0 0x7f183c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:05.849 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:05.844 7f1861083700 1 --2- 172.21.2.112:0/55685028 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f183c0186c0 0x7f183c01abc0 crc :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:05.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:05.874 7f1863267700 1 -- 172.21.2.112:0/55685028 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f185c0ebc00 con 0x7f185c0ee8d0 2021-02-10T11:48:05.879 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:05.876 7f1851ffb700 1 -- 172.21.2.112:0/55685028 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f18540093d0 con 0x7f185c0ee8d0 2021-02-10T11:48:06.011 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:06.008 7f1863267700 1 -- 172.21.2.112:0/55685028 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f185c0f0b10 con 0x7f185c0ee8d0 2021-02-10T11:48:06.011 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:06.008 7f1851ffb700 1 -- 172.21.2.112:0/55685028 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v109) v1 ==== 80+0+4513 (secure 0 0 0) 0x7f1854002510 con 0x7f185c0ee8d0 2021-02-10T11:48:06.012 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:48:06.012 INFO:teuthology.orchestra.run.gibba012.stdout:{ 2021-02-10T11:48:06.012 INFO:teuthology.orchestra.run.gibba012.stdout: "auth_dump": [ 2021-02-10T11:48:06.012 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:48:06.012 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "mds.a", 2021-02-10T11:48:06.012 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCKvyNgPkV4AxAAEmVNaghdccP6lWd8hvQnxw==", 2021-02-10T11:48:06.012 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:48:06.013 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:48:06.013 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:48:06.013 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:48:06.013 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:48:06.013 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:48:06.013 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:48:06.013 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:48:06.013 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "mds.b", 2021-02-10T11:48:06.013 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCKvyNguxONLhAAHIpry+R89MAwdhFJyEhVCw==", 2021-02-10T11:48:06.014 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:48:06.014 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:48:06.014 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:48:06.014 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:48:06.015 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:48:06.015 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:48:06.015 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:48:06.015 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:48:06.015 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.0", 2021-02-10T11:48:06.015 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCSvyNgXcKPLhAAP/Bj2F+06DCtPCzQDNq+cQ==", 2021-02-10T11:48:06.015 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:48:06.015 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:48:06.015 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:48:06.015 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:48:06.015 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:48:06.015 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:48:06.015 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:48:06.016 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.1", 2021-02-10T11:48:06.016 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCVvyNghfbcLhAA66PrRwR2M7o6VTF5UhU/Fg==", 2021-02-10T11:48:06.016 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:48:06.016 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:48:06.016 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:48:06.016 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:48:06.016 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:48:06.016 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:48:06.016 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:48:06.016 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.2", 2021-02-10T11:48:06.016 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCYvyNgwiPqLhAA66BVkRsgaUlGpAsQ15Nf8Q==", 2021-02-10T11:48:06.016 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:48:06.016 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:48:06.016 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:48:06.017 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:48:06.017 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:48:06.017 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:48:06.017 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:48:06.017 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.3", 2021-02-10T11:48:06.017 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCbvyNgqBZrLxAAERPe/1L3SXAEzI3d9ReHHg==", 2021-02-10T11:48:06.017 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:48:06.017 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:48:06.017 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:48:06.017 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:48:06.017 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:48:06.017 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:48:06.017 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:48:06.018 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.4", 2021-02-10T11:48:06.018 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCkvyNgDQsxIhAAHmVjVYnx20N4SDmKqvbcuA==", 2021-02-10T11:48:06.018 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:48:06.018 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:48:06.018 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:48:06.018 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:48:06.018 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:48:06.018 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:48:06.018 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:48:06.018 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.5", 2021-02-10T11:48:06.018 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCnvyNg8ITFIhAADd4aKybAV7btoGPx3kCg3A==", 2021-02-10T11:48:06.018 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:48:06.018 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:48:06.019 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:48:06.019 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:48:06.019 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:48:06.019 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:48:06.019 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:48:06.019 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.6", 2021-02-10T11:48:06.019 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCqvyNgxOhDIxAAzXixEa/nlDc1xXrg0aeLwg==", 2021-02-10T11:48:06.019 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:48:06.019 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:48:06.019 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:48:06.019 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:48:06.019 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:48:06.019 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:48:06.019 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:48:06.020 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.7", 2021-02-10T11:48:06.020 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCtvyNgxSqGIxAACxJQlwP7DpiVVCZHA3zn1A==", 2021-02-10T11:48:06.020 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:48:06.020 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:48:06.020 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:48:06.020 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:48:06.020 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:48:06.020 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:48:06.020 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:48:06.020 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.0", 2021-02-10T11:48:06.020 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCLvyNgDkNiHRAAygU8Za4PBusTqCgwAYWTiQ==", 2021-02-10T11:48:06.021 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:48:06.021 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:48:06.021 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow r", 2021-02-10T11:48:06.021 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:48:06.021 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:48:06.021 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:48:06.021 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:48:06.021 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.1", 2021-02-10T11:48:06.021 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCLvyNgxmLqMxAA1MlSDJ9LH9u4f2sL7mm8/Q==", 2021-02-10T11:48:06.021 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:48:06.021 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:48:06.021 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow r", 2021-02-10T11:48:06.021 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:48:06.022 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:48:06.022 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:48:06.022 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:48:06.022 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.2", 2021-02-10T11:48:06.022 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCMvyNgp8veDhAAniSVHYPtwwTog/pjbaRQmw==", 2021-02-10T11:48:06.022 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:48:06.022 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:48:06.022 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow r", 2021-02-10T11:48:06.022 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:48:06.022 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:48:06.022 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:48:06.022 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:48:06.022 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.3", 2021-02-10T11:48:06.023 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCMvyNgQEVoJRAApHcWdm1sldIxu+heyD4HdQ==", 2021-02-10T11:48:06.023 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:48:06.023 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:48:06.023 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow r", 2021-02-10T11:48:06.023 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:48:06.023 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:48:06.023 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:48:06.023 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:48:06.023 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.admin", 2021-02-10T11:48:06.023 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCGvyNg7jjJBRAAtot+ypUZOiTo1GbY+oVhVw==", 2021-02-10T11:48:06.023 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:48:06.023 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow *", 2021-02-10T11:48:06.023 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:48:06.023 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:48:06.024 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:48:06.024 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:48:06.024 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:48:06.024 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:48:06.024 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "mgr.x", 2021-02-10T11:48:06.024 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCJvyNgTPVwKBAAjduiBE8+RSXBv82tdPecWg==", 2021-02-10T11:48:06.024 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:48:06.024 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow *", 2021-02-10T11:48:06.024 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow profile mgr", 2021-02-10T11:48:06.024 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:48:06.024 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:48:06.024 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:48:06.024 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:48:06.024 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "mgr.y", 2021-02-10T11:48:06.025 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCJvyNgQ1tTDhAASwBssDy+Rz31ujw8fNlWow==", 2021-02-10T11:48:06.025 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:48:06.025 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow *", 2021-02-10T11:48:06.025 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow profile mgr", 2021-02-10T11:48:06.025 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:48:06.025 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:48:06.025 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:48:06.025 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:48:06.025 INFO:teuthology.orchestra.run.gibba012.stdout:} 2021-02-10T11:48:06.026 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:06.010 7f1863267700 1 -- 172.21.2.112:0/55685028 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f183c0186c0 msgr2=0x7f183c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:06.027 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:06.010 7f1863267700 1 --2- 172.21.2.112:0/55685028 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f183c0186c0 0x7f183c01abc0 crc :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:06.027 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:06.010 7f1863267700 1 -- 172.21.2.112:0/55685028 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f185c0ee8d0 msgr2=0x7f185c0eecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:06.027 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:06.010 7f1863267700 1 --2- 172.21.2.112:0/55685028 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f185c0ee8d0 0x7f185c0eecd0 secure :-1 s=READY pgs=3122 cs=0 l=1 rev1=1 rx=0x7f185400b8c0 tx=0x7f185400bc60).stop 2021-02-10T11:48:06.027 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:06.010 7f1863267700 1 -- 172.21.2.112:0/55685028 shutdown_connections 2021-02-10T11:48:06.027 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:06.010 7f1863267700 1 --2- 172.21.2.112:0/55685028 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f183c0186c0 0x7f183c01abc0 unknown :-1 s=CLOSED pgs=1402 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:06.027 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:06.010 7f1863267700 1 --2- 172.21.2.112:0/55685028 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f185c0ed8e0 0x7f185c0edce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:06.027 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:06.010 7f1863267700 1 --2- 172.21.2.112:0/55685028 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f185c0ee8d0 0x7f185c0eecd0 unknown :-1 s=CLOSED pgs=3122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:06.027 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:06.010 7f1863267700 1 --2- 172.21.2.112:0/55685028 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f185c0ed4e0 0x7f185c090e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:06.027 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:06.010 7f1863267700 1 -- 172.21.2.112:0/55685028 >> 172.21.2.112:0/55685028 conn(0x7f185c18a400 msgr2=0x7f185c1a4440 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:06.027 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:06.011 7f1863267700 1 -- 172.21.2.112:0/55685028 shutdown_connections 2021-02-10T11:48:06.027 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:06.011 7f1863267700 1 -- 172.21.2.112:0/55685028 wait complete. 2021-02-10T11:48:06.028 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:48:06.457 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:06.437 7f114f4a2700 1 Processor -- start 2021-02-10T11:48:06.457 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:06.437 7f114f4a2700 1 -- start start 2021-02-10T11:48:06.458 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:06.438 7f114f4a2700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f11480d1870 0x7f11481b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:06.458 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:06.438 7f114f4a2700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f11481b0a10 0x7f11480cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:06.458 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:06.438 7f114f4a2700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f11480d1c70 0x7f11480cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:06.458 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:06.438 7f114f4a2700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11480fe200 con 0x7f11480d1870 2021-02-10T11:48:06.458 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:06.438 7f114f4a2700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11480e5f10 con 0x7f11480d1c70 2021-02-10T11:48:06.458 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:06.438 7f114f4a2700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11480f0b50 con 0x7f11481b0a10 2021-02-10T11:48:06.458 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:06.438 7f114d2be700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f11480d1870 0x7f11481b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:06.458 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:06.438 7f114d2be700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f11480d1870 0x7f11481b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:60936/0 (socket says 172.21.2.112:60936) 2021-02-10T11:48:06.458 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:06.438 7f114d2be700 1 -- 172.21.2.112:0/3142932447 learned_addr learned my addr 172.21.2.112:0/3142932447 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:48:06.458 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:06.438 7f114d2be700 1 -- 172.21.2.112:0/3142932447 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f11481b0a10 msgr2=0x7f11480cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:06.458 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:06.438 7f114d2be700 1 --2- 172.21.2.112:0/3142932447 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f11481b0a10 0x7f11480cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:06.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:06.438 7f114d2be700 1 -- 172.21.2.112:0/3142932447 >> v2:172.21.2.120:3300/0 conn(0x7f11480d1c70 msgr2=0x7f11480cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:06.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:06.438 7f114d2be700 1 --2- 172.21.2.112:0/3142932447 >> v2:172.21.2.120:3300/0 conn(0x7f11480d1c70 0x7f11480cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:06.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:06.438 7f114d2be700 1 -- 172.21.2.112:0/3142932447 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11480cff70 con 0x7f11480d1870 2021-02-10T11:48:06.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:06.439 7f114d2be700 1 --2- 172.21.2.112:0/3142932447 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f11480d1870 0x7f11481b03f0 secure :-1 s=READY pgs=3123 cs=0 l=1 rev1=1 rx=0x7f113c006700 tx=0x7f113c009b80).ready entity=mon.0 client_cookie=8f17f57c1f8dc82a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:06.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:06.439 7f113bfff700 1 -- 172.21.2.112:0/3142932447 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f113c00c0c0 con 0x7f11480d1870 2021-02-10T11:48:06.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:06.439 7f113bfff700 1 -- 172.21.2.112:0/3142932447 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f113c00c440 con 0x7f11480d1870 2021-02-10T11:48:06.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:06.439 7f113bfff700 1 -- 172.21.2.112:0/3142932447 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f113c00c960 con 0x7f11480d1870 2021-02-10T11:48:06.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:06.439 7f114f4a2700 1 -- 172.21.2.112:0/3142932447 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f11480d1870 msgr2=0x7f11481b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:06.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:06.439 7f114f4a2700 1 --2- 172.21.2.112:0/3142932447 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f11480d1870 0x7f11481b03f0 secure :-1 s=READY pgs=3123 cs=0 l=1 rev1=1 rx=0x7f113c006700 tx=0x7f113c009b80).stop 2021-02-10T11:48:06.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:06.440 7f114f4a2700 1 -- 172.21.2.112:0/3142932447 shutdown_connections 2021-02-10T11:48:06.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:06.440 7f114f4a2700 1 --2- 172.21.2.112:0/3142932447 >> v2:172.21.2.120:3300/0 conn(0x7f11480d1c70 0x7f11480cf950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:06.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:06.440 7f114f4a2700 1 --2- 172.21.2.112:0/3142932447 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f11481b0a10 0x7f11480cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:06.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:06.440 7f114f4a2700 1 --2- 172.21.2.112:0/3142932447 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f11480d1870 0x7f11481b03f0 unknown :-1 s=CLOSED pgs=3123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:06.460 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:06.440 7f114f4a2700 1 -- 172.21.2.112:0/3142932447 >> 172.21.2.112:0/3142932447 conn(0x7f11480ce450 msgr2=0x7f11480d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:06.460 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:06.440 7f114f4a2700 1 -- 172.21.2.112:0/3142932447 shutdown_connections 2021-02-10T11:48:06.460 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:06.440 7f114f4a2700 1 -- 172.21.2.112:0/3142932447 wait complete. 2021-02-10T11:48:06.461 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:06.455 7f114f4a2700 1 Processor -- start 2021-02-10T11:48:06.461 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:06.455 7f114f4a2700 1 -- start start 2021-02-10T11:48:06.461 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:06.455 7f114f4a2700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f114818e930 0x7f11480aa4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:06.461 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:06.455 7f114f4a2700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f11480aaaf0 0x7f1148093700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:06.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:06.455 7f114f4a2700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f114818ed30 0x7f1148093d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:06.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:06.455 7f114f4a2700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11480944e0 con 0x7f114818ed30 2021-02-10T11:48:06.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:06.455 7f114f4a2700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f114808a590 con 0x7f114818e930 2021-02-10T11:48:06.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:06.455 7f114f4a2700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f114808a770 con 0x7f11480aaaf0 2021-02-10T11:48:06.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:06.455 7f114dabf700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f114818ed30 0x7f1148093d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:06.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:06.455 7f114dabf700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f114818ed30 0x7f1148093d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:60942/0 (socket says 172.21.2.112:60942) 2021-02-10T11:48:06.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:06.455 7f114dabf700 1 -- 172.21.2.112:0/927473051 learned_addr learned my addr 172.21.2.112:0/927473051 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:48:06.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:06.456 7f114dabf700 1 -- 172.21.2.112:0/927473051 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f11480aaaf0 msgr2=0x7f1148093700 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:48:06.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:06.456 7f114dabf700 1 --2- 172.21.2.112:0/927473051 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f11480aaaf0 0x7f1148093700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:06.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:06.456 7f114dabf700 1 -- 172.21.2.112:0/927473051 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f114818e930 msgr2=0x7f11480aa4d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:48:06.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:06.456 7f114dabf700 1 --2- 172.21.2.112:0/927473051 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f114818e930 0x7f11480aa4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:06.462 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:06.456 7f114dabf700 1 -- 172.21.2.112:0/927473051 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f114808a990 con 0x7f114818ed30 2021-02-10T11:48:06.463 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:06.456 7f114dabf700 1 --2- 172.21.2.112:0/927473051 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f114818ed30 0x7f1148093d20 secure :-1 s=READY pgs=3124 cs=0 l=1 rev1=1 rx=0x7f1140009980 tx=0x7f114000bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:06.463 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:06.456 7f1139ffb700 1 -- 172.21.2.112:0/927473051 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1140005af0 con 0x7f114818ed30 2021-02-10T11:48:06.463 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:06.456 7f1139ffb700 1 -- 172.21.2.112:0/927473051 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1140005d70 con 0x7f114818ed30 2021-02-10T11:48:06.463 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:06.456 7f1139ffb700 1 -- 172.21.2.112:0/927473051 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f11400062b0 con 0x7f114818ed30 2021-02-10T11:48:06.463 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:06.456 7f114f4a2700 1 -- 172.21.2.112:0/927473051 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f114808a990 con 0x7f114818ed30 2021-02-10T11:48:06.463 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:06.456 7f114f4a2700 1 -- 172.21.2.112:0/927473051 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f114808af20 con 0x7f114818ed30 2021-02-10T11:48:06.463 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:06.457 7f1139ffb700 1 -- 172.21.2.112:0/927473051 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1140006030 con 0x7f114818ed30 2021-02-10T11:48:06.463 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:06.457 7f1139ffb700 1 --2- 172.21.2.112:0/927473051 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f11280186c0 0x7f112801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:06.463 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:06.457 7f1139ffb700 1 -- 172.21.2.112:0/927473051 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(184..184 src has 1..184) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f11400325d0 con 0x7f114818ed30 2021-02-10T11:48:06.463 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:06.457 7f114d2be700 1 --2- 172.21.2.112:0/927473051 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f11280186c0 0x7f112801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:06.463 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:06.458 7f114d2be700 1 --2- 172.21.2.112:0/927473051 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f11280186c0 0x7f112801abc0 crc :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:06.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:06.488 7f114f4a2700 1 -- 172.21.2.112:0/927473051 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11480ee060 con 0x7f114818ed30 2021-02-10T11:48:06.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:06.490 7f1139ffb700 1 -- 172.21.2.112:0/927473051 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f114002f120 con 0x7f114818ed30 2021-02-10T11:48:06.618 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:06.615 7f114f4a2700 1 -- 172.21.2.112:0/927473051 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f11480ce450 con 0x7f114818ed30 2021-02-10T11:48:06.618 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:06.615 7f1139ffb700 1 -- 172.21.2.112:0/927473051 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v184) v1 ==== 74+0+11154 (secure 0 0 0) 0x7f114002f120 con 0x7f114818ed30 2021-02-10T11:48:06.619 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:48:06.619 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":184,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:48:05.355216","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":39,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:48:06.620 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:06.617 7f114f4a2700 1 -- 172.21.2.112:0/927473051 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f11280186c0 msgr2=0x7f112801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:06.620 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:06.617 7f114f4a2700 1 --2- 172.21.2.112:0/927473051 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f11280186c0 0x7f112801abc0 crc :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:06.620 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:06.617 7f114f4a2700 1 -- 172.21.2.112:0/927473051 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f114818ed30 msgr2=0x7f1148093d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:06.620 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:06.617 7f114f4a2700 1 --2- 172.21.2.112:0/927473051 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f114818ed30 0x7f1148093d20 secure :-1 s=READY pgs=3124 cs=0 l=1 rev1=1 rx=0x7f1140009980 tx=0x7f114000bcd0).stop 2021-02-10T11:48:06.621 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:06.617 7f114f4a2700 1 -- 172.21.2.112:0/927473051 shutdown_connections 2021-02-10T11:48:06.621 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:06.617 7f114f4a2700 1 --2- 172.21.2.112:0/927473051 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f11280186c0 0x7f112801abc0 unknown :-1 s=CLOSED pgs=1403 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:06.621 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:06.617 7f114f4a2700 1 --2- 172.21.2.112:0/927473051 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f114818ed30 0x7f1148093d20 unknown :-1 s=CLOSED pgs=3124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:06.621 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:06.617 7f114f4a2700 1 --2- 172.21.2.112:0/927473051 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f11480aaaf0 0x7f1148093700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:06.621 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:06.617 7f114f4a2700 1 --2- 172.21.2.112:0/927473051 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f114818e930 0x7f11480aa4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:06.621 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:06.617 7f114f4a2700 1 -- 172.21.2.112:0/927473051 >> 172.21.2.112:0/927473051 conn(0x7f114819f8f0 msgr2=0x7f114818c260 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:06.621 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:06.618 7f114f4a2700 1 -- 172.21.2.112:0/927473051 shutdown_connections 2021-02-10T11:48:06.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:06.618 7f114f4a2700 1 -- 172.21.2.112:0/927473051 wait complete. 2021-02-10T11:48:06.629 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:13:57.508758', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2021-02-10T11:48:06.629 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:48:07.063 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.041 7f392da8f700 1 Processor -- start 2021-02-10T11:48:07.063 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.041 7f392da8f700 1 -- start start 2021-02-10T11:48:07.063 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.042 7f392da8f700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f39280d1c70 0x7f39280cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:07.063 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.042 7f392da8f700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f39280cf580 0x7f39280cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:07.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.042 7f392da8f700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39280fe1c0 con 0x7f39280d1870 2021-02-10T11:48:07.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.042 7f392da8f700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39281b0cb0 con 0x7f39280cf580 2021-02-10T11:48:07.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.042 7f392da8f700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3928109a60 con 0x7f39280d1c70 2021-02-10T11:48:07.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.043 7f3927fff700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f39280cf580 0x7f39280cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:07.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.043 7f39277fe700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7f39280d1870 0x7f39281b04c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:53966/0 (socket says 172.21.2.112:53966) 2021-02-10T11:48:07.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.043 7f3926ffd700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f39280d1c70 0x7f39280cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:07.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.043 7f39277fe700 1 -- 172.21.2.112:0/635847636 learned_addr learned my addr 172.21.2.112:0/635847636 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:48:07.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.044 7f3927fff700 1 -- 172.21.2.112:0/635847636 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f39280d1c70 msgr2=0x7f39280cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:07.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.044 7f3927fff700 1 --2- 172.21.2.112:0/635847636 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f39280d1c70 0x7f39280cef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:07.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.044 7f3927fff700 1 -- 172.21.2.112:0/635847636 >> v1:172.21.2.120:6789/0 conn(0x7f39280d1870 legacy=0x7f39281b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:07.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.044 7f3927fff700 1 -- 172.21.2.112:0/635847636 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3928088070 con 0x7f39280cf580 2021-02-10T11:48:07.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.044 7f3926ffd700 1 --2- 172.21.2.112:0/635847636 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f39280d1c70 0x7f39280cef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:48:07.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.044 7f3927fff700 1 --2- 172.21.2.112:0/635847636 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f39280cf580 0x7f39280cf980 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 rx=0x7f3920006e50 tx=0x7f3920009e70).ready entity=mon.1 client_cookie=cc06175c17ba365c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:07.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.044 7f39267fc700 1 -- 172.21.2.112:0/635847636 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3920000f90 con 0x7f39280cf580 2021-02-10T11:48:07.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.044 7f39267fc700 1 -- 172.21.2.112:0/635847636 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39200066f0 con 0x7f39280cf580 2021-02-10T11:48:07.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.044 7f39267fc700 1 -- 172.21.2.112:0/635847636 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f392000b700 con 0x7f39280cf580 2021-02-10T11:48:07.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.045 7f392da8f700 1 -- 172.21.2.112:0/635847636 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f39280cf580 msgr2=0x7f39280cf980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:07.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.045 7f392da8f700 1 --2- 172.21.2.112:0/635847636 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f39280cf580 0x7f39280cf980 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 rx=0x7f3920006e50 tx=0x7f3920009e70).stop 2021-02-10T11:48:07.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.045 7f392da8f700 1 -- 172.21.2.112:0/635847636 shutdown_connections 2021-02-10T11:48:07.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.045 7f392da8f700 1 --2- 172.21.2.112:0/635847636 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f39280cf580 0x7f39280cf980 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:07.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.045 7f392da8f700 1 --2- 172.21.2.112:0/635847636 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f39280d1c70 0x7f39280cef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:07.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.045 7f392da8f700 1 -- 172.21.2.112:0/635847636 >> 172.21.2.112:0/635847636 conn(0x7f39280ce420 msgr2=0x7f39280d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:07.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.045 7f392da8f700 1 -- 172.21.2.112:0/635847636 shutdown_connections 2021-02-10T11:48:07.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.045 7f392da8f700 1 -- 172.21.2.112:0/635847636 wait complete. 2021-02-10T11:48:07.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.060 7f392da8f700 1 Processor -- start 2021-02-10T11:48:07.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.060 7f392da8f700 1 -- start start 2021-02-10T11:48:07.069 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.060 7f392da8f700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f39280b04d0 0x7f39281a7a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:07.069 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.060 7f392da8f700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f39281a8090 0x7f39280c23a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:07.069 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.060 7f392da8f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f392808d680 0x7f39280c29c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:07.069 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.060 7f392da8f700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39280b0a00 con 0x7f392808d680 2021-02-10T11:48:07.069 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.060 7f392da8f700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39280c2fe0 con 0x7f39280b04d0 2021-02-10T11:48:07.069 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.060 7f392da8f700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39280c3200 con 0x7f39281a8090 2021-02-10T11:48:07.069 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.061 7f3927fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f392808d680 0x7f39280c29c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:07.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.061 7f3927fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f392808d680 0x7f39280c29c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:60956/0 (socket says 172.21.2.112:60956) 2021-02-10T11:48:07.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.061 7f3927fff700 1 -- 172.21.2.112:0/1118419092 learned_addr learned my addr 172.21.2.112:0/1118419092 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:48:07.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.061 7f3927fff700 1 -- 172.21.2.112:0/1118419092 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f39281a8090 msgr2=0x7f39280c23a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:07.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.061 7f3927fff700 1 --2- 172.21.2.112:0/1118419092 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f39281a8090 0x7f39280c23a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:07.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.061 7f3927fff700 1 -- 172.21.2.112:0/1118419092 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f39280b04d0 msgr2=0x7f39281a7a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:07.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.061 7f3927fff700 1 --2- 172.21.2.112:0/1118419092 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f39280b04d0 0x7f39281a7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:07.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.061 7f3927fff700 1 -- 172.21.2.112:0/1118419092 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f392808e080 con 0x7f392808d680 2021-02-10T11:48:07.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.061 7f3927fff700 1 --2- 172.21.2.112:0/1118419092 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f392808d680 0x7f39280c29c0 secure :-1 s=READY pgs=3125 cs=0 l=1 rev1=1 rx=0x7f39200099b0 tx=0x7f392000bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:07.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.061 7f390ffff700 1 -- 172.21.2.112:0/1118419092 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3920002030 con 0x7f392808d680 2021-02-10T11:48:07.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.061 7f390ffff700 1 -- 172.21.2.112:0/1118419092 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3920002270 con 0x7f392808d680 2021-02-10T11:48:07.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.061 7f392da8f700 1 -- 172.21.2.112:0/1118419092 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f392808e080 con 0x7f392808d680 2021-02-10T11:48:07.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.061 7f392da8f700 1 -- 172.21.2.112:0/1118419092 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f392808e610 con 0x7f392808d680 2021-02-10T11:48:07.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.061 7f390ffff700 1 -- 172.21.2.112:0/1118419092 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3920002700 con 0x7f392808d680 2021-02-10T11:48:07.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.062 7f390ffff700 1 -- 172.21.2.112:0/1118419092 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3920002450 con 0x7f392808d680 2021-02-10T11:48:07.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.062 7f390ffff700 1 --2- 172.21.2.112:0/1118419092 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f39080186c0 0x7f390801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:07.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.062 7f390ffff700 1 -- 172.21.2.112:0/1118419092 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(184..184 src has 1..184) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f39200325b0 con 0x7f392808d680 2021-02-10T11:48:07.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.062 7f39277fe700 1 --2- 172.21.2.112:0/1118419092 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f39080186c0 0x7f390801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:07.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.063 7f39277fe700 1 --2- 172.21.2.112:0/1118419092 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f39080186c0 0x7f390801abc0 crc :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:07.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.092 7f392da8f700 1 -- 172.21.2.112:0/1118419092 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3928082a70 con 0x7f392808d680 2021-02-10T11:48:07.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.094 7f390ffff700 1 -- 172.21.2.112:0/1118419092 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f39200093d0 con 0x7f392808d680 2021-02-10T11:48:07.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.223 7f392da8f700 1 -- 172.21.2.112:0/1118419092 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f3928082a70 con 0x7f392808d680 2021-02-10T11:48:07.227 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.223 7f390ffff700 1 -- 172.21.2.112:0/1118419092 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v184) v1 ==== 92+0+10 (secure 0 0 0) 0x7f3920002450 con 0x7f392808d680 2021-02-10T11:48:07.227 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:48:07.228 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.225 7f392da8f700 1 -- 172.21.2.112:0/1118419092 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f39080186c0 msgr2=0x7f390801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:07.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.225 7f392da8f700 1 --2- 172.21.2.112:0/1118419092 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f39080186c0 0x7f390801abc0 crc :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:07.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.225 7f392da8f700 1 -- 172.21.2.112:0/1118419092 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f392808d680 msgr2=0x7f39280c29c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:07.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.225 7f392da8f700 1 --2- 172.21.2.112:0/1118419092 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f392808d680 0x7f39280c29c0 secure :-1 s=READY pgs=3125 cs=0 l=1 rev1=1 rx=0x7f39200099b0 tx=0x7f392000bcd0).stop 2021-02-10T11:48:07.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.225 7f392da8f700 1 -- 172.21.2.112:0/1118419092 shutdown_connections 2021-02-10T11:48:07.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.225 7f392da8f700 1 --2- 172.21.2.112:0/1118419092 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f39080186c0 0x7f390801abc0 unknown :-1 s=CLOSED pgs=1404 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:07.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.225 7f392da8f700 1 --2- 172.21.2.112:0/1118419092 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f392808d680 0x7f39280c29c0 unknown :-1 s=CLOSED pgs=3125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:07.230 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.225 7f392da8f700 1 --2- 172.21.2.112:0/1118419092 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f39281a8090 0x7f39280c23a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:07.230 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.225 7f392da8f700 1 --2- 172.21.2.112:0/1118419092 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f39280b04d0 0x7f39281a7a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:07.230 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.225 7f392da8f700 1 -- 172.21.2.112:0/1118419092 >> 172.21.2.112:0/1118419092 conn(0x7f39280be530 msgr2=0x7f392809f1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:07.230 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.226 7f392da8f700 1 -- 172.21.2.112:0/1118419092 shutdown_connections 2021-02-10T11:48:07.230 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.226 7f392da8f700 1 -- 172.21.2.112:0/1118419092 wait complete. 2021-02-10T11:48:07.238 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --format=json-pretty osd lspools 2021-02-10T11:48:07.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.648 7f73230f2700 1 Processor -- start 2021-02-10T11:48:07.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.648 7f73230f2700 1 -- start start 2021-02-10T11:48:07.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.648 7f73230f2700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f731c0d1870 0x7f731c1b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:07.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.648 7f73230f2700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f731c1b0a10 0x7f731c0cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:07.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.648 7f73230f2700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f731c0f0b10 con 0x7f731c0d1c70 2021-02-10T11:48:07.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.648 7f73230f2700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f731c0fe230 con 0x7f731c1b0a10 2021-02-10T11:48:07.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.648 7f73230f2700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f731c0e5ed0 con 0x7f731c0d1870 2021-02-10T11:48:07.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.648 7f732170f700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f731c0d1c70 0x7f731c0cf950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:60424/0 (socket says 172.21.2.112:60424) 2021-02-10T11:48:07.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.648 7f732170f700 1 -- 172.21.2.112:0/4184175533 learned_addr learned my addr 172.21.2.112:0/4184175533 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:48:07.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.648 7f7313fff700 1 --2- 172.21.2.112:0/4184175533 >> v2:172.21.2.112:3300/0 conn(0x7f731c1b0a10 0x7f731c0cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:07.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.649 7f73137fe700 1 -- 172.21.2.112:0/4184175533 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2980604570 0 0) 0x7f7314000d30 con 0x7f731c0d1c70 2021-02-10T11:48:07.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.649 7f7313fff700 1 -- 172.21.2.112:0/4184175533 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f731c0d1870 msgr2=0x7f731c1b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:07.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.649 7f7313fff700 1 --2- 172.21.2.112:0/4184175533 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f731c0d1870 0x7f731c1b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:07.669 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.649 7f7313fff700 1 -- 172.21.2.112:0/4184175533 >> v1:172.21.2.112:6789/0 conn(0x7f731c0d1c70 legacy=0x7f731c0cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:07.669 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.649 7f7313fff700 1 -- 172.21.2.112:0/4184175533 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f731c0f0650 con 0x7f731c1b0a10 2021-02-10T11:48:07.669 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.649 7f7313fff700 1 --2- 172.21.2.112:0/4184175533 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f731c1b0a10 0x7f731c0cf330 secure :-1 s=READY pgs=3127 cs=0 l=1 rev1=1 rx=0x7f7318006650 tx=0x7f7318009a80).ready entity=mon.0 client_cookie=8b8be8f0eea70359 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:07.669 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.649 7f73137fe700 1 -- 172.21.2.112:0/4184175533 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f73180009b0 con 0x7f731c1b0a10 2021-02-10T11:48:07.669 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.649 7f73137fe700 1 -- 172.21.2.112:0/4184175533 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f731800c0c0 con 0x7f731c1b0a10 2021-02-10T11:48:07.669 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.649 7f73137fe700 1 -- 172.21.2.112:0/4184175533 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f731800c510 con 0x7f731c1b0a10 2021-02-10T11:48:07.669 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.649 7f73230f2700 1 -- 172.21.2.112:0/4184175533 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f731c1b0a10 msgr2=0x7f731c0cf330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:07.669 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.649 7f73230f2700 1 --2- 172.21.2.112:0/4184175533 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f731c1b0a10 0x7f731c0cf330 secure :-1 s=READY pgs=3127 cs=0 l=1 rev1=1 rx=0x7f7318006650 tx=0x7f7318009a80).stop 2021-02-10T11:48:07.669 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.650 7f73230f2700 1 -- 172.21.2.112:0/4184175533 shutdown_connections 2021-02-10T11:48:07.669 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.650 7f73230f2700 1 --2- 172.21.2.112:0/4184175533 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f731c1b0a10 0x7f731c0cf330 unknown :-1 s=CLOSED pgs=3127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:07.669 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.650 7f73230f2700 1 --2- 172.21.2.112:0/4184175533 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f731c0d1870 0x7f731c1b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:07.669 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.650 7f73230f2700 1 -- 172.21.2.112:0/4184175533 >> 172.21.2.112:0/4184175533 conn(0x7f731c0ce450 msgr2=0x7f731c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:07.669 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.650 7f73230f2700 1 -- 172.21.2.112:0/4184175533 shutdown_connections 2021-02-10T11:48:07.669 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.650 7f73230f2700 1 -- 172.21.2.112:0/4184175533 wait complete. 2021-02-10T11:48:07.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.665 7f73230f2700 1 Processor -- start 2021-02-10T11:48:07.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.665 7f73230f2700 1 -- start start 2021-02-10T11:48:07.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.665 7f73230f2700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f731c0ed0f0 0x7f731c0ab250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:07.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.665 7f73230f2700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f731c0ed5f0 0x7f731c0ab870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:07.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.665 7f73230f2700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f731c1b8770 0x7f731c1b8b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:07.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.665 7f73230f2700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f731c193430 con 0x7f731c0ed0f0 2021-02-10T11:48:07.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.665 7f73230f2700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f731c0abe90 con 0x7f731c1b8770 2021-02-10T11:48:07.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.665 7f73230f2700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f731c0ac0b0 con 0x7f731c0ed5f0 2021-02-10T11:48:07.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.665 7f7320f0e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f731c0ed0f0 0x7f731c0ab250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:07.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.665 7f7320f0e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f731c0ed0f0 0x7f731c0ab250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:60970/0 (socket says 172.21.2.112:60970) 2021-02-10T11:48:07.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.665 7f7320f0e700 1 -- 172.21.2.112:0/4221306412 learned_addr learned my addr 172.21.2.112:0/4221306412 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:48:07.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.666 7f732170f700 1 --2- 172.21.2.112:0/4221306412 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f731c1b8770 0x7f731c1b8b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:07.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.666 7f7320f0e700 1 -- 172.21.2.112:0/4221306412 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f731c0ed5f0 msgr2=0x7f731c0ab870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:07.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.666 7f7320f0e700 1 --2- 172.21.2.112:0/4221306412 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f731c0ed5f0 0x7f731c0ab870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:07.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.666 7f7320f0e700 1 -- 172.21.2.112:0/4221306412 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f731c1b8770 msgr2=0x7f731c1b8b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:07.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.666 7f7320f0e700 1 --2- 172.21.2.112:0/4221306412 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f731c1b8770 0x7f731c1b8b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:07.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.666 7f7320f0e700 1 -- 172.21.2.112:0/4221306412 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f731c1b9200 con 0x7f731c0ed0f0 2021-02-10T11:48:07.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.666 7f7320f0e700 1 --2- 172.21.2.112:0/4221306412 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f731c0ed0f0 0x7f731c0ab250 secure :-1 s=READY pgs=3128 cs=0 l=1 rev1=1 rx=0x7f730c006f00 tx=0x7f730c00b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:07.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.666 7f73117fa700 1 -- 172.21.2.112:0/4221306412 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f730c0009b0 con 0x7f731c0ed0f0 2021-02-10T11:48:07.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.666 7f73117fa700 1 -- 172.21.2.112:0/4221306412 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f730c00bdc0 con 0x7f731c0ed0f0 2021-02-10T11:48:07.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.666 7f73117fa700 1 -- 172.21.2.112:0/4221306412 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f730c002020 con 0x7f731c0ed0f0 2021-02-10T11:48:07.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.666 7f73230f2700 1 -- 172.21.2.112:0/4221306412 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f731c1b9200 con 0x7f731c0ed0f0 2021-02-10T11:48:07.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.666 7f73230f2700 1 -- 172.21.2.112:0/4221306412 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f731c1b9200 con 0x7f731c0ed0f0 2021-02-10T11:48:07.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.667 7f73117fa700 1 -- 172.21.2.112:0/4221306412 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f730c005310 con 0x7f731c0ed0f0 2021-02-10T11:48:07.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.667 7f73117fa700 1 --2- 172.21.2.112:0/4221306412 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f72fc0186c0 0x7f72fc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:07.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.667 7f73117fa700 1 -- 172.21.2.112:0/4221306412 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(184..184 src has 1..184) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f730c032490 con 0x7f731c0ed0f0 2021-02-10T11:48:07.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.667 7f7313fff700 1 --2- 172.21.2.112:0/4221306412 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f72fc0186c0 0x7f72fc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:07.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.668 7f7313fff700 1 --2- 172.21.2.112:0/4221306412 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f72fc0186c0 0x7f72fc01abc0 crc :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:07.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.698 7f73230f2700 1 -- 172.21.2.112:0/4221306412 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f731c0abe90 con 0x7f731c0ed0f0 2021-02-10T11:48:07.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.700 7f73117fa700 1 -- 172.21.2.112:0/4221306412 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f730c006350 con 0x7f731c0ed0f0 2021-02-10T11:48:07.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.825 7f73230f2700 1 -- 172.21.2.112:0/4221306412 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f731c0abe90 con 0x7f731c0ed0f0 2021-02-10T11:48:07.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.825 7f73117fa700 1 -- 172.21.2.112:0/4221306412 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v184) v1 ==== 84+0+64 (secure 0 0 0) 0x7f730c00bd30 con 0x7f731c0ed0f0 2021-02-10T11:48:07.829 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:48:07.829 INFO:teuthology.orchestra.run.gibba012.stdout:[ 2021-02-10T11:48:07.829 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:48:07.829 INFO:teuthology.orchestra.run.gibba012.stdout: "poolnum": 1, 2021-02-10T11:48:07.829 INFO:teuthology.orchestra.run.gibba012.stdout: "poolname": "rbd" 2021-02-10T11:48:07.829 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:48:07.829 INFO:teuthology.orchestra.run.gibba012.stdout:] 2021-02-10T11:48:07.830 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.827 7f73230f2700 1 -- 172.21.2.112:0/4221306412 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f72fc0186c0 msgr2=0x7f72fc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:07.830 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.827 7f73230f2700 1 --2- 172.21.2.112:0/4221306412 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f72fc0186c0 0x7f72fc01abc0 crc :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:07.830 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.827 7f73230f2700 1 -- 172.21.2.112:0/4221306412 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f731c0ed0f0 msgr2=0x7f731c0ab250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:07.830 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.827 7f73230f2700 1 --2- 172.21.2.112:0/4221306412 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f731c0ed0f0 0x7f731c0ab250 secure :-1 s=READY pgs=3128 cs=0 l=1 rev1=1 rx=0x7f730c006f00 tx=0x7f730c00b940).stop 2021-02-10T11:48:07.831 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.827 7f73230f2700 1 -- 172.21.2.112:0/4221306412 shutdown_connections 2021-02-10T11:48:07.831 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.827 7f73230f2700 1 --2- 172.21.2.112:0/4221306412 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f72fc0186c0 0x7f72fc01abc0 unknown :-1 s=CLOSED pgs=1405 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:07.831 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.827 7f73230f2700 1 --2- 172.21.2.112:0/4221306412 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f731c1b8770 0x7f731c1b8b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:07.831 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.827 7f73230f2700 1 --2- 172.21.2.112:0/4221306412 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f731c0ed5f0 0x7f731c0ab870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:07.831 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.827 7f73230f2700 1 --2- 172.21.2.112:0/4221306412 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f731c0ed0f0 0x7f731c0ab250 unknown :-1 s=CLOSED pgs=3128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:07.831 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.827 7f73230f2700 1 -- 172.21.2.112:0/4221306412 >> 172.21.2.112:0/4221306412 conn(0x7f731c19c580 msgr2=0x7f731c0916f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:07.831 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.827 7f73230f2700 1 -- 172.21.2.112:0/4221306412 shutdown_connections 2021-02-10T11:48:07.831 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:07.827 7f73230f2700 1 -- 172.21.2.112:0/4221306412 wait complete. 2021-02-10T11:48:07.839 INFO:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2021-02-10T11:48:07.839 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_metadata 8 2021-02-10T11:48:08.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:08.252 7faaec168700 1 Processor -- start 2021-02-10T11:48:08.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:08.252 7faaec168700 1 -- start start 2021-02-10T11:48:08.272 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:08.253 7faaec168700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7faae40d1860 0x7faae41b04b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:08.272 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:08.253 7faaec168700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7faae40cff60 0x7faae4109120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:08.272 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:08.253 7faaec168700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faae4109a50 con 0x7faae41b0ad0 2021-02-10T11:48:08.272 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:08.253 7faaec168700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faae4109740 con 0x7faae40cff60 2021-02-10T11:48:08.272 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:08.253 7faaec168700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faae4088070 con 0x7faae40d1860 2021-02-10T11:48:08.272 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:08.253 7faaea785700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7faae40cff60 0x7faae4109120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:08.273 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:08.253 7faaea785700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7faae40cff60 0x7faae4109120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:60974/0 (socket says 172.21.2.112:60974) 2021-02-10T11:48:08.273 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:08.253 7faaea785700 1 -- 172.21.2.112:0/1701671254 learned_addr learned my addr 172.21.2.112:0/1701671254 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:48:08.273 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:08.253 7faaea785700 1 -- 172.21.2.112:0/1701671254 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7faae40d1860 msgr2=0x7faae41b04b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:48:08.273 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:08.253 7faaea785700 1 --2- 172.21.2.112:0/1701671254 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7faae40d1860 0x7faae41b04b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:08.273 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:08.253 7faaea785700 1 -- 172.21.2.112:0/1701671254 >> v1:172.21.2.120:6789/0 conn(0x7faae41b0ad0 legacy=0x7faae4108930 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:48:08.273 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:08.253 7faaea785700 1 -- 172.21.2.112:0/1701671254 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faae4088260 con 0x7faae40cff60 2021-02-10T11:48:08.273 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:08.253 7faaea785700 1 --2- 172.21.2.112:0/1701671254 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faae40cff60 0x7faae4109120 secure :-1 s=READY pgs=3129 cs=0 l=1 rev1=1 rx=0x7faadc006f00 tx=0x7faadc009f10).ready entity=mon.0 client_cookie=edb7414d7d674ee0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:08.273 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:08.253 7faae8f82700 1 -- 172.21.2.112:0/1701671254 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faadc00b3b0 con 0x7faae40cff60 2021-02-10T11:48:08.273 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:08.254 7faae8f82700 1 -- 172.21.2.112:0/1701671254 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faadc00b630 con 0x7faae40cff60 2021-02-10T11:48:08.274 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:08.254 7faae8f82700 1 -- 172.21.2.112:0/1701671254 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faadc00bb50 con 0x7faae40cff60 2021-02-10T11:48:08.274 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:08.254 7faaec168700 1 -- 172.21.2.112:0/1701671254 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faae40cff60 msgr2=0x7faae4109120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:08.274 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:08.254 7faaec168700 1 --2- 172.21.2.112:0/1701671254 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faae40cff60 0x7faae4109120 secure :-1 s=READY pgs=3129 cs=0 l=1 rev1=1 rx=0x7faadc006f00 tx=0x7faadc009f10).stop 2021-02-10T11:48:08.274 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:08.254 7faaec168700 1 -- 172.21.2.112:0/1701671254 shutdown_connections 2021-02-10T11:48:08.274 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:08.254 7faaec168700 1 --2- 172.21.2.112:0/1701671254 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faae40cff60 0x7faae4109120 unknown :-1 s=CLOSED pgs=3129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:08.274 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:08.254 7faaec168700 1 --2- 172.21.2.112:0/1701671254 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7faae40d1860 0x7faae41b04b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:08.274 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:08.254 7faaec168700 1 -- 172.21.2.112:0/1701671254 >> 172.21.2.112:0/1701671254 conn(0x7faae40ce410 msgr2=0x7faae40d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:08.275 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:08.254 7faaec168700 1 -- 172.21.2.112:0/1701671254 shutdown_connections 2021-02-10T11:48:08.275 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:08.254 7faaec168700 1 -- 172.21.2.112:0/1701671254 wait complete. 2021-02-10T11:48:08.277 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:08.268 7faaec168700 1 Processor -- start 2021-02-10T11:48:08.277 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:08.268 7faaec168700 1 -- start start 2021-02-10T11:48:08.277 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:08.269 7faaec168700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7faae41a05e0 0x7faae418b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:08.277 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:08.269 7faaec168700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faae40b8780 0x7faae40bfa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:08.277 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:08.269 7faaec168700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7faae40c0030 0x7faae4189e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:08.277 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:08.269 7faaec168700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faae40b82a0 con 0x7faae40b8780 2021-02-10T11:48:08.277 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:08.269 7faaec168700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faae40c0430 con 0x7faae40c0030 2021-02-10T11:48:08.277 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:08.269 7faaec168700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faae40c0650 con 0x7faae41a05e0 2021-02-10T11:48:08.277 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:08.269 7faae9783700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faae40b8780 0x7faae40bfa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:08.278 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:08.269 7faae9783700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faae40b8780 0x7faae40bfa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:60978/0 (socket says 172.21.2.112:60978) 2021-02-10T11:48:08.278 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:08.269 7faae9783700 1 -- 172.21.2.112:0/1368989863 learned_addr learned my addr 172.21.2.112:0/1368989863 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:48:08.278 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:08.269 7faae9783700 1 -- 172.21.2.112:0/1368989863 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7faae41a05e0 msgr2=0x7faae418b970 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:48:08.278 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:08.269 7faae9783700 1 --2- 172.21.2.112:0/1368989863 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7faae41a05e0 0x7faae418b970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:08.278 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:08.269 7faae9783700 1 -- 172.21.2.112:0/1368989863 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7faae40c0030 msgr2=0x7faae4189e70 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:48:08.278 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:08.269 7faae9783700 1 --2- 172.21.2.112:0/1368989863 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7faae40c0030 0x7faae4189e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:08.278 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:08.269 7faae9783700 1 -- 172.21.2.112:0/1368989863 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faae418a490 con 0x7faae40b8780 2021-02-10T11:48:08.278 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:08.269 7faae9783700 1 --2- 172.21.2.112:0/1368989863 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faae40b8780 0x7faae40bfa10 secure :-1 s=READY pgs=3130 cs=0 l=1 rev1=1 rx=0x7faae000b5b0 tx=0x7faae000b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:08.278 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:08.269 7faada7fc700 1 -- 172.21.2.112:0/1368989863 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faae000bd40 con 0x7faae40b8780 2021-02-10T11:48:08.278 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:08.270 7faada7fc700 1 -- 172.21.2.112:0/1368989863 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faae0006240 con 0x7faae40b8780 2021-02-10T11:48:08.278 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:08.270 7faaec168700 1 -- 172.21.2.112:0/1368989863 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faae418a490 con 0x7faae40b8780 2021-02-10T11:48:08.279 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:08.270 7faada7fc700 1 -- 172.21.2.112:0/1368989863 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faae0005690 con 0x7faae40b8780 2021-02-10T11:48:08.279 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:08.270 7faaec168700 1 -- 172.21.2.112:0/1368989863 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faae418a490 con 0x7faae40b8780 2021-02-10T11:48:08.279 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:08.270 7faada7fc700 1 -- 172.21.2.112:0/1368989863 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7faae00054a0 con 0x7faae40b8780 2021-02-10T11:48:08.279 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:08.270 7faada7fc700 1 --2- 172.21.2.112:0/1368989863 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7faacc018730 0x7faacc01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:08.279 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:08.270 7faada7fc700 1 -- 172.21.2.112:0/1368989863 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(184..184 src has 1..184) v4 ==== 4753+0+0 (secure 0 0 0) 0x7faae0032320 con 0x7faae40b8780 2021-02-10T11:48:08.279 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:08.271 7faae9f84700 1 --2- 172.21.2.112:0/1368989863 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7faacc018730 0x7faacc01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:08.279 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:08.272 7faae9f84700 1 --2- 172.21.2.112:0/1368989863 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7faacc018730 0x7faacc01ac10 crc :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:08.306 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:08.303 7faaec168700 1 -- 172.21.2.112:0/1368989863 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faae4082a20 con 0x7faae40b8780 2021-02-10T11:48:08.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:08.305 7faada7fc700 1 -- 172.21.2.112:0/1368989863 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7faae0000c60 con 0x7faae40b8780 2021-02-10T11:48:08.438 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:08.435 7faaec168700 1 -- 172.21.2.112:0/1368989863 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"} v 0) v1 -- 0x7faae4082a20 con 0x7faae40b8780 2021-02-10T11:48:08.843 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:08.840 7faada7fc700 1 -- 172.21.2.112:0/1368989863 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"}]=0 pool 'cephfs_metadata' created v185) v1 ==== 133+0+0 (secure 0 0 0) 0x7faae00054a0 con 0x7faae40b8780 2021-02-10T11:48:08.854 INFO:teuthology.orchestra.run.gibba012.stderr:pool 'cephfs_metadata' created 2021-02-10T11:48:08.856 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:08.852 7faaec168700 1 -- 172.21.2.112:0/1368989863 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7faacc018730 msgr2=0x7faacc01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:08.856 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:08.852 7faaec168700 1 --2- 172.21.2.112:0/1368989863 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7faacc018730 0x7faacc01ac10 crc :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:08.856 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:08.852 7faaec168700 1 -- 172.21.2.112:0/1368989863 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faae40b8780 msgr2=0x7faae40bfa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:08.856 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:08.852 7faaec168700 1 --2- 172.21.2.112:0/1368989863 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faae40b8780 0x7faae40bfa10 secure :-1 s=READY pgs=3130 cs=0 l=1 rev1=1 rx=0x7faae000b5b0 tx=0x7faae000b950).stop 2021-02-10T11:48:08.856 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:08.852 7faaec168700 1 -- 172.21.2.112:0/1368989863 shutdown_connections 2021-02-10T11:48:08.856 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:08.852 7faaec168700 1 --2- 172.21.2.112:0/1368989863 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7faacc018730 0x7faacc01ac10 unknown :-1 s=CLOSED pgs=1406 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:08.856 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:08.852 7faaec168700 1 --2- 172.21.2.112:0/1368989863 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7faae40c0030 0x7faae4189e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:08.856 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:08.852 7faaec168700 1 --2- 172.21.2.112:0/1368989863 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faae40b8780 0x7faae40bfa10 unknown :-1 s=CLOSED pgs=3130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:08.856 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:08.852 7faaec168700 1 --2- 172.21.2.112:0/1368989863 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7faae41a05e0 0x7faae418b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:08.856 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:08.852 7faaec168700 1 -- 172.21.2.112:0/1368989863 >> 172.21.2.112:0/1368989863 conn(0x7faae40a6a80 msgr2=0x7faae40ae110 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:08.856 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:08.852 7faaec168700 1 -- 172.21.2.112:0/1368989863 shutdown_connections 2021-02-10T11:48:08.857 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:08.852 7faaec168700 1 -- 172.21.2.112:0/1368989863 wait complete. 2021-02-10T11:48:08.864 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_data 8 2021-02-10T11:48:09.297 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:09.274 7ff3eba40700 1 Processor -- start 2021-02-10T11:48:09.297 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:09.275 7ff3eba40700 1 -- start start 2021-02-10T11:48:09.297 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:09.275 7ff3eba40700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7ff3e40d1c70 0x7ff3e40cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:09.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:09.275 7ff3eba40700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7ff3e40cf580 0x7ff3e40cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:09.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:09.275 7ff3eba40700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3e40fe1c0 con 0x7ff3e40d1870 2021-02-10T11:48:09.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:09.275 7ff3eba40700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3e41b0cb0 con 0x7ff3e40d1c70 2021-02-10T11:48:09.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:09.275 7ff3eba40700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3e4109a60 con 0x7ff3e40cf580 2021-02-10T11:48:09.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:09.275 7ff3e905b700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7ff3e40d1c70 0x7ff3e40cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:09.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:09.275 7ff3e905b700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7ff3e40d1c70 0x7ff3e40cef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:60986/0 (socket says 172.21.2.112:60986) 2021-02-10T11:48:09.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:09.275 7ff3e905b700 1 -- 172.21.2.112:0/3534316511 learned_addr learned my addr 172.21.2.112:0/3534316511 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:48:09.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:09.276 7ff3e905b700 1 -- 172.21.2.112:0/3534316511 >> v2:172.21.2.120:3300/0 conn(0x7ff3e40cf580 msgr2=0x7ff3e40cf980 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:48:09.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:09.276 7ff3e905b700 1 --2- 172.21.2.112:0/3534316511 >> v2:172.21.2.120:3300/0 conn(0x7ff3e40cf580 0x7ff3e40cf980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:09.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:09.276 7ff3e905b700 1 -- 172.21.2.112:0/3534316511 >> v1:172.21.2.120:6789/0 conn(0x7ff3e40d1870 legacy=0x7ff3e41b04c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:48:09.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:09.276 7ff3e905b700 1 -- 172.21.2.112:0/3534316511 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3e4088070 con 0x7ff3e40d1c70 2021-02-10T11:48:09.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:09.276 7ff3e905b700 1 --2- 172.21.2.112:0/3534316511 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff3e40d1c70 0x7ff3e40cef60 secure :-1 s=READY pgs=3131 cs=0 l=1 rev1=1 rx=0x7ff3e0006700 tx=0x7ff3e0009b00).ready entity=mon.0 client_cookie=cc8059b84b067971 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:09.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:09.276 7ff3e885a700 1 -- 172.21.2.112:0/3534316511 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff3e000c0c0 con 0x7ff3e40d1c70 2021-02-10T11:48:09.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:09.276 7ff3e885a700 1 -- 172.21.2.112:0/3534316511 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3e000c340 con 0x7ff3e40d1c70 2021-02-10T11:48:09.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:09.276 7ff3e885a700 1 -- 172.21.2.112:0/3534316511 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff3e000c820 con 0x7ff3e40d1c70 2021-02-10T11:48:09.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:09.276 7ff3eba40700 1 -- 172.21.2.112:0/3534316511 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff3e40d1c70 msgr2=0x7ff3e40cef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:09.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:09.276 7ff3eba40700 1 --2- 172.21.2.112:0/3534316511 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff3e40d1c70 0x7ff3e40cef60 secure :-1 s=READY pgs=3131 cs=0 l=1 rev1=1 rx=0x7ff3e0006700 tx=0x7ff3e0009b00).stop 2021-02-10T11:48:09.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:09.277 7ff3eba40700 1 -- 172.21.2.112:0/3534316511 shutdown_connections 2021-02-10T11:48:09.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:09.277 7ff3eba40700 1 --2- 172.21.2.112:0/3534316511 >> v2:172.21.2.120:3300/0 conn(0x7ff3e40cf580 0x7ff3e40cf980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:09.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:09.277 7ff3eba40700 1 --2- 172.21.2.112:0/3534316511 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff3e40d1c70 0x7ff3e40cef60 unknown :-1 s=CLOSED pgs=3131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:09.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:09.277 7ff3eba40700 1 -- 172.21.2.112:0/3534316511 >> 172.21.2.112:0/3534316511 conn(0x7ff3e40ce420 msgr2=0x7ff3e40d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:09.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:09.277 7ff3eba40700 1 -- 172.21.2.112:0/3534316511 shutdown_connections 2021-02-10T11:48:09.302 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:09.277 7ff3eba40700 1 -- 172.21.2.112:0/3534316511 wait complete. 2021-02-10T11:48:09.302 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:09.292 7ff3eba40700 1 Processor -- start 2021-02-10T11:48:09.302 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:09.292 7ff3eba40700 1 -- start start 2021-02-10T11:48:09.302 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:09.292 7ff3eba40700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff3e40c3820 0x7ff3e40a7950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:09.302 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:09.292 7ff3eba40700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff3e41a44c0 0x7ff3e41a48c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:09.302 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:09.292 7ff3eba40700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff3e40863d0 0x7ff3e40867d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:09.303 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:09.292 7ff3eba40700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3e4085ed0 con 0x7ff3e40863d0 2021-02-10T11:48:09.303 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:09.292 7ff3eba40700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3e40c0740 con 0x7ff3e41a44c0 2021-02-10T11:48:09.303 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:09.292 7ff3eba40700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3e4086df0 con 0x7ff3e40c3820 2021-02-10T11:48:09.303 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:09.292 7ff3ea05d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff3e40863d0 0x7ff3e40867d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:09.303 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:09.292 7ff3ea05d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff3e40863d0 0x7ff3e40867d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:60992/0 (socket says 172.21.2.112:60992) 2021-02-10T11:48:09.303 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:09.292 7ff3ea05d700 1 -- 172.21.2.112:0/2171018808 learned_addr learned my addr 172.21.2.112:0/2171018808 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:48:09.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:09.292 7ff3ea05d700 1 -- 172.21.2.112:0/2171018808 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff3e40c3820 msgr2=0x7ff3e40a7950 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:48:09.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:09.292 7ff3ea05d700 1 --2- 172.21.2.112:0/2171018808 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff3e40c3820 0x7ff3e40a7950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:09.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:09.292 7ff3ea05d700 1 -- 172.21.2.112:0/2171018808 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff3e41a44c0 msgr2=0x7ff3e41a48c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:48:09.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:09.292 7ff3ea05d700 1 --2- 172.21.2.112:0/2171018808 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff3e41a44c0 0x7ff3e41a48c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:09.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:09.292 7ff3ea05d700 1 -- 172.21.2.112:0/2171018808 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3e4087010 con 0x7ff3e40863d0 2021-02-10T11:48:09.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:09.293 7ff3ea05d700 1 --2- 172.21.2.112:0/2171018808 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff3e40863d0 0x7ff3e40867d0 secure :-1 s=READY pgs=3132 cs=0 l=1 rev1=1 rx=0x7ff3dc009960 tx=0x7ff3dc00bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:09.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:09.293 7ff3d67fc700 1 -- 172.21.2.112:0/2171018808 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff3dc003030 con 0x7ff3e40863d0 2021-02-10T11:48:09.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:09.293 7ff3d67fc700 1 -- 172.21.2.112:0/2171018808 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3dc001310 con 0x7ff3e40863d0 2021-02-10T11:48:09.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:09.293 7ff3d67fc700 1 -- 172.21.2.112:0/2171018808 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff3dc001770 con 0x7ff3e40863d0 2021-02-10T11:48:09.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:09.293 7ff3eba40700 1 -- 172.21.2.112:0/2171018808 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3e4087010 con 0x7ff3e40863d0 2021-02-10T11:48:09.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:09.293 7ff3eba40700 1 -- 172.21.2.112:0/2171018808 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3e4087010 con 0x7ff3e40863d0 2021-02-10T11:48:09.306 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:09.294 7ff3d67fc700 1 -- 172.21.2.112:0/2171018808 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff3dc000d10 con 0x7ff3e40863d0 2021-02-10T11:48:09.306 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:09.294 7ff3d67fc700 1 --2- 172.21.2.112:0/2171018808 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff3cc0186c0 0x7ff3cc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:09.306 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:09.294 7ff3d67fc700 1 -- 172.21.2.112:0/2171018808 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(186..186 src has 1..186) v4 ==== 5107+0+0 (secure 0 0 0) 0x7ff3dc032730 con 0x7ff3e40863d0 2021-02-10T11:48:09.307 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:09.295 7ff3e985c700 1 --2- 172.21.2.112:0/2171018808 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff3cc0186c0 0x7ff3cc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:09.307 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:09.296 7ff3e985c700 1 --2- 172.21.2.112:0/2171018808 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff3cc0186c0 0x7ff3cc01abc0 crc :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:09.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:09.325 7ff3eba40700 1 -- 172.21.2.112:0/2171018808 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3e40fe1c0 con 0x7ff3e40863d0 2021-02-10T11:48:09.331 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:09.327 7ff3d67fc700 1 -- 172.21.2.112:0/2171018808 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff3dc006690 con 0x7ff3e40863d0 2021-02-10T11:48:09.458 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:09.455 7ff3eba40700 1 -- 172.21.2.112:0/2171018808 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"} v 0) v1 -- 0x7ff3e40fe1c0 con 0x7ff3e40863d0 2021-02-10T11:48:09.852 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:09.849 7ff3d67fc700 1 -- 172.21.2.112:0/2171018808 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"}]=0 pool 'cephfs_data' created v187) v1 ==== 125+0+0 (secure 0 0 0) 0x7ff3dc006690 con 0x7ff3e40863d0 2021-02-10T11:48:09.874 INFO:teuthology.orchestra.run.gibba012.stderr:pool 'cephfs_data' created 2021-02-10T11:48:09.876 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:09.873 7ff3eba40700 1 -- 172.21.2.112:0/2171018808 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff3cc0186c0 msgr2=0x7ff3cc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:09.876 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:09.873 7ff3eba40700 1 --2- 172.21.2.112:0/2171018808 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff3cc0186c0 0x7ff3cc01abc0 crc :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:09.876 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:09.873 7ff3eba40700 1 -- 172.21.2.112:0/2171018808 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff3e40863d0 msgr2=0x7ff3e40867d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:09.876 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:09.873 7ff3eba40700 1 --2- 172.21.2.112:0/2171018808 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff3e40863d0 0x7ff3e40867d0 secure :-1 s=READY pgs=3132 cs=0 l=1 rev1=1 rx=0x7ff3dc009960 tx=0x7ff3dc00bcf0).stop 2021-02-10T11:48:09.876 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:09.873 7ff3eba40700 1 -- 172.21.2.112:0/2171018808 shutdown_connections 2021-02-10T11:48:09.876 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:09.873 7ff3eba40700 1 --2- 172.21.2.112:0/2171018808 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff3cc0186c0 0x7ff3cc01abc0 unknown :-1 s=CLOSED pgs=1407 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:09.876 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:09.873 7ff3eba40700 1 --2- 172.21.2.112:0/2171018808 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff3e41a44c0 0x7ff3e41a48c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:09.876 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:09.873 7ff3eba40700 1 --2- 172.21.2.112:0/2171018808 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff3e40863d0 0x7ff3e40867d0 unknown :-1 s=CLOSED pgs=3132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:09.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:09.873 7ff3eba40700 1 --2- 172.21.2.112:0/2171018808 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff3e40c3820 0x7ff3e40a7950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:09.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:09.873 7ff3eba40700 1 -- 172.21.2.112:0/2171018808 >> 172.21.2.112:0/2171018808 conn(0x7ff3e40872c0 msgr2=0x7ff3e4095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:09.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:09.873 7ff3eba40700 1 -- 172.21.2.112:0/2171018808 shutdown_connections 2021-02-10T11:48:09.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:09.873 7ff3eba40700 1 -- 172.21.2.112:0/2171018808 wait complete. 2021-02-10T11:48:09.885 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2021-02-10T11:48:10.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.298 7f08a31ca700 1 Processor -- start 2021-02-10T11:48:10.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.298 7f08a31ca700 1 -- start start 2021-02-10T11:48:10.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.299 7f08a31ca700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f089c0d1c70 0x7f089c0cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:10.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.299 7f08a31ca700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f089c0d0170 con 0x7f089c0d1870 2021-02-10T11:48:10.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.299 7f08a31ca700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f089c0fe1c0 con 0x7f089c0cf580 2021-02-10T11:48:10.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.299 7f08a31ca700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f089c1b0cb0 con 0x7f089c0d1c70 2021-02-10T11:48:10.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.299 7f08a0fe6700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f089c0d1870 0x7f089c1b04c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:60464/0 (socket says 172.21.2.112:60464) 2021-02-10T11:48:10.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.299 7f08a0fe6700 1 -- 172.21.2.112:0/1223290973 learned_addr learned my addr 172.21.2.112:0/1223290973 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:48:10.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.299 7f089bfff700 1 --2- 172.21.2.112:0/1223290973 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f089c0d1c70 0x7f089c0cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:10.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.299 7f089b7fe700 1 -- 172.21.2.112:0/1223290973 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3215068630 0 0) 0x7f08840010b0 con 0x7f089c0d1870 2021-02-10T11:48:10.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.299 7f089b7fe700 1 -- 172.21.2.112:0/1223290973 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0888002fe0 con 0x7f089c0d1870 2021-02-10T11:48:10.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.299 7f089b7fe700 1 -- 172.21.2.112:0/1223290973 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4054139991 0 0) 0x7f0894001400 con 0x7f089c0cf580 2021-02-10T11:48:10.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.299 7f089b7fe700 1 -- 172.21.2.112:0/1223290973 --> v1:172.21.2.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0888003210 con 0x7f089c0cf580 2021-02-10T11:48:10.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.299 7f089b7fe700 1 -- 172.21.2.112:0/1223290973 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1647139673 0 0) 0x7f0884000e20 con 0x7f089c0d1870 2021-02-10T11:48:10.319 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.299 7f089bfff700 1 -- 172.21.2.112:0/1223290973 >> v1:172.21.2.120:6789/0 conn(0x7f089c0cf580 legacy=0x7f089c0cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:10.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.300 7f089bfff700 1 -- 172.21.2.112:0/1223290973 >> v1:172.21.2.112:6789/0 conn(0x7f089c0d1870 legacy=0x7f089c1b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:10.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.300 7f089bfff700 1 -- 172.21.2.112:0/1223290973 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f089c109a60 con 0x7f089c0d1c70 2021-02-10T11:48:10.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.300 7f089bfff700 1 --2- 172.21.2.112:0/1223290973 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f089c0d1c70 0x7f089c0cef60 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7f088c006650 tx=0x7f088c009a80).ready entity=mon.2 client_cookie=ef4a23e61bd0da12 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:10.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.300 7f089b7fe700 1 -- 172.21.2.112:0/1223290973 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f088c00c0c0 con 0x7f089c0d1c70 2021-02-10T11:48:10.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.300 7f089b7fe700 1 -- 172.21.2.112:0/1223290973 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f088c00c340 con 0x7f089c0d1c70 2021-02-10T11:48:10.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.300 7f089b7fe700 1 -- 172.21.2.112:0/1223290973 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f088c00c860 con 0x7f089c0d1c70 2021-02-10T11:48:10.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.300 7f08a31ca700 1 -- 172.21.2.112:0/1223290973 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f089c0d1c70 msgr2=0x7f089c0cef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:10.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.300 7f08a31ca700 1 --2- 172.21.2.112:0/1223290973 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f089c0d1c70 0x7f089c0cef60 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7f088c006650 tx=0x7f088c009a80).stop 2021-02-10T11:48:10.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.301 7f08a31ca700 1 -- 172.21.2.112:0/1223290973 shutdown_connections 2021-02-10T11:48:10.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.301 7f08a31ca700 1 --2- 172.21.2.112:0/1223290973 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f089c0d1c70 0x7f089c0cef60 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:10.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.301 7f08a31ca700 1 -- 172.21.2.112:0/1223290973 >> 172.21.2.112:0/1223290973 conn(0x7f089c0ce420 msgr2=0x7f089c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:10.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.301 7f08a31ca700 1 -- 172.21.2.112:0/1223290973 shutdown_connections 2021-02-10T11:48:10.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.301 7f08a31ca700 1 -- 172.21.2.112:0/1223290973 wait complete. 2021-02-10T11:48:10.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.315 7f08a31ca700 1 Processor -- start 2021-02-10T11:48:10.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.315 7f08a31ca700 1 -- start start 2021-02-10T11:48:10.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.316 7f08a31ca700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f089c19e5c0 0x7f089c099fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:10.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.316 7f08a31ca700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f089c09a5d0 0x7f089c09a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:10.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.316 7f08a31ca700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f089c0982a0 0x7f089c0986a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:10.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.316 7f08a31ca700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f089c19feb0 con 0x7f089c0982a0 2021-02-10T11:48:10.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.316 7f08a31ca700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f089c099a30 con 0x7f089c09a5d0 2021-02-10T11:48:10.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.316 7f08a31ca700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f089c19fb30 con 0x7f089c19e5c0 2021-02-10T11:48:10.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.316 7f08a17e7700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f089c0982a0 0x7f089c0986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:10.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.316 7f08a17e7700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f089c0982a0 0x7f089c0986a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:32774/0 (socket says 172.21.2.112:32774) 2021-02-10T11:48:10.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.316 7f08a17e7700 1 -- 172.21.2.112:0/2702197490 learned_addr learned my addr 172.21.2.112:0/2702197490 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:48:10.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.316 7f089bfff700 1 --2- 172.21.2.112:0/2702197490 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f089c09a5d0 0x7f089c09a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:10.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.316 7f08a0fe6700 1 --2- 172.21.2.112:0/2702197490 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f089c19e5c0 0x7f089c099fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:10.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.316 7f08a17e7700 1 -- 172.21.2.112:0/2702197490 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f089c19e5c0 msgr2=0x7f089c099fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:10.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.316 7f08a17e7700 1 --2- 172.21.2.112:0/2702197490 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f089c19e5c0 0x7f089c099fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:10.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.316 7f08a17e7700 1 -- 172.21.2.112:0/2702197490 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f089c09a5d0 msgr2=0x7f089c09a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:10.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.316 7f08a17e7700 1 --2- 172.21.2.112:0/2702197490 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f089c09a5d0 0x7f089c09a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:10.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.316 7f08a17e7700 1 -- 172.21.2.112:0/2702197490 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f089c19e9c0 con 0x7f089c0982a0 2021-02-10T11:48:10.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.317 7f08a17e7700 1 --2- 172.21.2.112:0/2702197490 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f089c0982a0 0x7f089c0986a0 secure :-1 s=READY pgs=3134 cs=0 l=1 rev1=1 rx=0x7f08940099a0 tx=0x7f089400bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:10.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.317 7f08997fa700 1 -- 172.21.2.112:0/2702197490 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f08940058a0 con 0x7f089c0982a0 2021-02-10T11:48:10.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.317 7f08997fa700 1 -- 172.21.2.112:0/2702197490 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0894003210 con 0x7f089c0982a0 2021-02-10T11:48:10.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.317 7f08997fa700 1 -- 172.21.2.112:0/2702197490 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0894006030 con 0x7f089c0982a0 2021-02-10T11:48:10.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.317 7f08a31ca700 1 -- 172.21.2.112:0/2702197490 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f089c19e9c0 con 0x7f089c0982a0 2021-02-10T11:48:10.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.317 7f08a31ca700 1 -- 172.21.2.112:0/2702197490 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f089c19e9c0 con 0x7f089c0982a0 2021-02-10T11:48:10.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.317 7f08997fa700 1 -- 172.21.2.112:0/2702197490 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0894005ba0 con 0x7f089c0982a0 2021-02-10T11:48:10.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.318 7f08997fa700 1 --2- 172.21.2.112:0/2702197490 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f08800186c0 0x7f088001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:10.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.318 7f08997fa700 1 -- 172.21.2.112:0/2702197490 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 5457+0+0 (secure 0 0 0) 0x7f08940326f0 con 0x7f089c0982a0 2021-02-10T11:48:10.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.318 7f08a0fe6700 1 --2- 172.21.2.112:0/2702197490 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f08800186c0 0x7f088001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:10.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.319 7f08a0fe6700 1 --2- 172.21.2.112:0/2702197490 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f08800186c0 0x7f088001abc0 crc :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:10.351 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.348 7f08a31ca700 1 -- 172.21.2.112:0/2702197490 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f089c0825d0 con 0x7f089c0982a0 2021-02-10T11:48:10.354 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.351 7f08997fa700 1 -- 172.21.2.112:0/2702197490 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f089400be30 con 0x7f089c0982a0 2021-02-10T11:48:10.482 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.479 7f08a31ca700 1 -- 172.21.2.112:0/2702197490 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"} v 0) v1 -- 0x7f089c0825d0 con 0x7f089c0982a0 2021-02-10T11:48:10.487 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.484 7f08997fa700 1 -- 172.21.2.112:0/2702197490 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"}]=0 new fs with metadata pool 40 and data pool 41 v162) v1 ==== 189+0+0 (secure 0 0 0) 0x7f0894006030 con 0x7f089c0982a0 2021-02-10T11:48:10.489 INFO:teuthology.orchestra.run.gibba012.stderr:new fs with metadata pool 40 and data pool 41 2021-02-10T11:48:10.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.488 7f08a31ca700 1 -- 172.21.2.112:0/2702197490 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f08800186c0 msgr2=0x7f088001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:10.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.488 7f08a31ca700 1 --2- 172.21.2.112:0/2702197490 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f08800186c0 0x7f088001abc0 crc :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:10.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.488 7f08a31ca700 1 -- 172.21.2.112:0/2702197490 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f089c0982a0 msgr2=0x7f089c0986a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:10.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.488 7f08a31ca700 1 --2- 172.21.2.112:0/2702197490 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f089c0982a0 0x7f089c0986a0 secure :-1 s=READY pgs=3134 cs=0 l=1 rev1=1 rx=0x7f08940099a0 tx=0x7f089400bcf0).stop 2021-02-10T11:48:10.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.489 7f08a31ca700 1 -- 172.21.2.112:0/2702197490 shutdown_connections 2021-02-10T11:48:10.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.489 7f08a31ca700 1 --2- 172.21.2.112:0/2702197490 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f08800186c0 0x7f088001abc0 unknown :-1 s=CLOSED pgs=1410 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:10.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.489 7f08a31ca700 1 --2- 172.21.2.112:0/2702197490 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f089c09a5d0 0x7f089c09a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:10.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.489 7f08a31ca700 1 --2- 172.21.2.112:0/2702197490 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f089c0982a0 0x7f089c0986a0 unknown :-1 s=CLOSED pgs=3134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:10.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.489 7f08a31ca700 1 --2- 172.21.2.112:0/2702197490 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f089c19e5c0 0x7f089c099fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:10.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.489 7f08a31ca700 1 -- 172.21.2.112:0/2702197490 >> 172.21.2.112:0/2702197490 conn(0x7f089c1b3560 msgr2=0x7f089c1b90e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:10.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.489 7f08a31ca700 1 -- 172.21.2.112:0/2702197490 shutdown_connections 2021-02-10T11:48:10.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.489 7f08a31ca700 1 -- 172.21.2.112:0/2702197490 wait complete. 2021-02-10T11:48:10.500 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:48:10.927 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.908 7fd57e995700 1 Processor -- start 2021-02-10T11:48:10.928 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.908 7fd57e995700 1 -- start start 2021-02-10T11:48:10.928 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.909 7fd57e995700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fd5780d1870 0x7fd5781b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:10.928 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.909 7fd57e995700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd5780f0650 0x7fd5780cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:10.928 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.909 7fd57e995700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5780f0b10 con 0x7fd5781b0a10 2021-02-10T11:48:10.928 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.909 7fd57e995700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5780e5ed0 con 0x7fd5780d1870 2021-02-10T11:48:10.928 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.909 7fd57e995700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5780d1c70 con 0x7fd5780f0650 2021-02-10T11:48:10.928 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.909 7fd577fff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fd5780d1870 0x7fd5781b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:10.928 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.909 7fd5777fe700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fd5781b0a10 0x7fd5780cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:60478/0 (socket says 172.21.2.112:60478) 2021-02-10T11:48:10.928 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.909 7fd5777fe700 1 -- 172.21.2.112:0/3421757160 learned_addr learned my addr 172.21.2.112:0/3421757160 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:48:10.928 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.909 7fd577fff700 1 -- 172.21.2.112:0/3421757160 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd5780f0650 msgr2=0x7fd5780cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:10.929 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.909 7fd577fff700 1 --2- 172.21.2.112:0/3421757160 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd5780f0650 0x7fd5780cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:10.929 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.909 7fd577fff700 1 -- 172.21.2.112:0/3421757160 >> v1:172.21.2.112:6789/0 conn(0x7fd5781b0a10 legacy=0x7fd5780cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:10.929 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.909 7fd577fff700 1 -- 172.21.2.112:0/3421757160 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5780d1e90 con 0x7fd5780d1870 2021-02-10T11:48:10.929 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.909 7fd577fff700 1 --2- 172.21.2.112:0/3421757160 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd5780d1870 0x7fd5781b03f0 secure :-1 s=READY pgs=3136 cs=0 l=1 rev1=1 rx=0x7fd568006650 tx=0x7fd568009a80).ready entity=mon.0 client_cookie=6625abba4dc582c9 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:10.929 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.910 7fd576ffd700 1 -- 172.21.2.112:0/3421757160 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd5680009b0 con 0x7fd5780d1870 2021-02-10T11:48:10.929 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.910 7fd576ffd700 1 -- 172.21.2.112:0/3421757160 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd56800c0c0 con 0x7fd5780d1870 2021-02-10T11:48:10.929 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.910 7fd576ffd700 1 -- 172.21.2.112:0/3421757160 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd56800c520 con 0x7fd5780d1870 2021-02-10T11:48:10.929 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.910 7fd57e995700 1 -- 172.21.2.112:0/3421757160 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd5780d1870 msgr2=0x7fd5781b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:10.929 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.910 7fd57e995700 1 --2- 172.21.2.112:0/3421757160 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd5780d1870 0x7fd5781b03f0 secure :-1 s=READY pgs=3136 cs=0 l=1 rev1=1 rx=0x7fd568006650 tx=0x7fd568009a80).stop 2021-02-10T11:48:10.929 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.910 7fd57e995700 1 -- 172.21.2.112:0/3421757160 shutdown_connections 2021-02-10T11:48:10.929 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.910 7fd57e995700 1 --2- 172.21.2.112:0/3421757160 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd5780f0650 0x7fd5780cfb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:10.929 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.910 7fd57e995700 1 --2- 172.21.2.112:0/3421757160 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd5780d1870 0x7fd5781b03f0 unknown :-1 s=CLOSED pgs=3136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:10.929 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.910 7fd57e995700 1 -- 172.21.2.112:0/3421757160 >> 172.21.2.112:0/3421757160 conn(0x7fd5780ce450 msgr2=0x7fd5780d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:10.930 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.910 7fd57e995700 1 -- 172.21.2.112:0/3421757160 shutdown_connections 2021-02-10T11:48:10.930 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.910 7fd57e995700 1 -- 172.21.2.112:0/3421757160 wait complete. 2021-02-10T11:48:10.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.925 7fd57e995700 1 Processor -- start 2021-02-10T11:48:10.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.925 7fd57e995700 1 -- start start 2021-02-10T11:48:10.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.925 7fd57e995700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd5781ae550 0x7fd5781924f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:10.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.925 7fd57e995700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd5781ae950 0x7fd5781a9aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:10.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.925 7fd57e995700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd5781aa0c0 0x7fd5781aa4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:10.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.925 7fd57e995700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5781a9520 con 0x7fd5781aa0c0 2021-02-10T11:48:10.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.925 7fd57e995700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd57818fee0 con 0x7fd5781ae950 2021-02-10T11:48:10.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.925 7fd57e995700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd57818f9d0 con 0x7fd5781ae550 2021-02-10T11:48:10.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.925 7fd57cfb2700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd5781aa0c0 0x7fd5781aa4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:10.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.925 7fd577fff700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd5781ae550 0x7fd5781924f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:10.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.926 7fd577fff700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd5781ae550 0x7fd5781924f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3301/0 says I am v2:172.21.2.112:60702/0 (socket says 172.21.2.112:60702) 2021-02-10T11:48:10.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.926 7fd577fff700 1 -- 172.21.2.112:0/2075268082 learned_addr learned my addr 172.21.2.112:0/2075268082 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:48:10.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.926 7fd5777fe700 1 --2- 172.21.2.112:0/2075268082 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd5781ae950 0x7fd5781a9aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:10.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.926 7fd577fff700 1 -- 172.21.2.112:0/2075268082 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd5781ae950 msgr2=0x7fd5781a9aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:10.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.926 7fd577fff700 1 --2- 172.21.2.112:0/2075268082 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd5781ae950 0x7fd5781a9aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:10.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.926 7fd577fff700 1 -- 172.21.2.112:0/2075268082 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd5781aa0c0 msgr2=0x7fd5781aa4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:10.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.926 7fd577fff700 1 --2- 172.21.2.112:0/2075268082 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd5781aa0c0 0x7fd5781aa4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:10.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.926 7fd577fff700 1 -- 172.21.2.112:0/2075268082 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5781a9170 con 0x7fd5781ae550 2021-02-10T11:48:10.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.926 7fd577fff700 1 --2- 172.21.2.112:0/2075268082 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd5781ae550 0x7fd5781924f0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0x7fd568009630 tx=0x7fd56800b970).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:10.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.926 7fd574ff9700 1 -- 172.21.2.112:0/2075268082 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd56800bd60 con 0x7fd5781ae550 2021-02-10T11:48:10.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.926 7fd57e995700 1 -- 172.21.2.112:0/2075268082 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5781a9170 con 0x7fd5781ae550 2021-02-10T11:48:10.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.926 7fd574ff9700 1 -- 172.21.2.112:0/2075268082 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd568006740 con 0x7fd5781ae550 2021-02-10T11:48:10.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.926 7fd574ff9700 1 -- 172.21.2.112:0/2075268082 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd56800bd60 con 0x7fd5781ae550 2021-02-10T11:48:10.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.926 7fd57e995700 1 -- 172.21.2.112:0/2075268082 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5781a9170 con 0x7fd5781ae550 2021-02-10T11:48:10.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.927 7fd574ff9700 1 -- 172.21.2.112:0/2075268082 <== mon.2 v2:172.21.2.120:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd56800a000 con 0x7fd5781ae550 2021-02-10T11:48:10.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.927 7fd574ff9700 1 --2- 172.21.2.112:0/2075268082 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd5580186c0 0x7fd55801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:10.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.927 7fd574ff9700 1 -- 172.21.2.112:0/2075268082 <== mon.2 v2:172.21.2.120:3301/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fd5680325b0 con 0x7fd5781ae550 2021-02-10T11:48:10.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.927 7fd5777fe700 1 --2- 172.21.2.112:0/2075268082 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd5580186c0 0x7fd55801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:10.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.928 7fd5777fe700 1 --2- 172.21.2.112:0/2075268082 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd5580186c0 0x7fd55801abc0 crc :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:10.962 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.959 7fd57e995700 1 -- 172.21.2.112:0/2075268082 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5780ee060 con 0x7fd5781ae550 2021-02-10T11:48:10.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:10.962 7fd574ff9700 1 -- 172.21.2.112:0/2075268082 <== mon.2 v2:172.21.2.120:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd568030d60 con 0x7fd5781ae550 2021-02-10T11:48:11.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:11.089 7fd57e995700 1 -- 172.21.2.112:0/2075268082 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd5780ce450 con 0x7fd5781ae550 2021-02-10T11:48:11.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:11.091 7fd574ff9700 1 -- 172.21.2.112:0/2075268082 <== mon.2 v2:172.21.2.120:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v188) v1 ==== 74+0+14020 (secure 0 0 0) 0x7fd56800a000 con 0x7fd5781ae550 2021-02-10T11:48:11.096 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:48:11.096 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":188,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:48:10.480941","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":41,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":40,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:48:08.454038","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"188","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":41,"pool_name":"cephfs_data","create_time":"2021-02-10 11:48:09.474627","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"188","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:48:11.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:11.094 7fd57e995700 1 -- 172.21.2.112:0/2075268082 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd5580186c0 msgr2=0x7fd55801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:11.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:11.094 7fd57e995700 1 --2- 172.21.2.112:0/2075268082 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd5580186c0 0x7fd55801abc0 crc :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:11.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:11.094 7fd57e995700 1 -- 172.21.2.112:0/2075268082 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd5781ae550 msgr2=0x7fd5781924f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:11.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:11.094 7fd57e995700 1 --2- 172.21.2.112:0/2075268082 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd5781ae550 0x7fd5781924f0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0x7fd568009630 tx=0x7fd56800b970).stop 2021-02-10T11:48:11.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:11.094 7fd57e995700 1 -- 172.21.2.112:0/2075268082 shutdown_connections 2021-02-10T11:48:11.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:11.094 7fd57e995700 1 --2- 172.21.2.112:0/2075268082 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd5580186c0 0x7fd55801abc0 unknown :-1 s=CLOSED pgs=1411 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:11.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:11.094 7fd57e995700 1 --2- 172.21.2.112:0/2075268082 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd5781ae950 0x7fd5781a9aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:11.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:11.094 7fd57e995700 1 --2- 172.21.2.112:0/2075268082 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd5781aa0c0 0x7fd5781aa4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:11.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:11.094 7fd57e995700 1 --2- 172.21.2.112:0/2075268082 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd5781ae550 0x7fd5781924f0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:11.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:11.094 7fd57e995700 1 -- 172.21.2.112:0/2075268082 >> 172.21.2.112:0/2075268082 conn(0x7fd5781a8c90 msgr2=0x7fd5781b38a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:11.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:11.095 7fd57e995700 1 -- 172.21.2.112:0/2075268082 shutdown_connections 2021-02-10T11:48:11.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:11.095 7fd57e995700 1 -- 172.21.2.112:0/2075268082 wait complete. 2021-02-10T11:48:11.106 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:48:11.537 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:11.517 7fa097580700 1 Processor -- start 2021-02-10T11:48:11.537 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:11.517 7fa097580700 1 -- start start 2021-02-10T11:48:11.537 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:11.518 7fa097580700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fa0900d0340 0x7fa0900ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:11.537 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:11.518 7fa097580700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fa0900d1c70 0x7fa0900efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:11.537 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:11.518 7fa097580700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0900f0b10 con 0x7fa0900d1870 2021-02-10T11:48:11.537 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:11.518 7fa097580700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0900e5ed0 con 0x7fa0900d0340 2021-02-10T11:48:11.538 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:11.518 7fa097580700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0901b0be0 con 0x7fa0900d1c70 2021-02-10T11:48:11.538 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:11.518 7fa094b9b700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fa0900d0340 0x7fa0900ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:11.538 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:11.518 7fa095b9d700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fa0900d1c70 0x7fa0900efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:11.538 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:11.518 7fa094b9b700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fa0900d0340 0x7fa0900ef640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:32794/0 (socket says 172.21.2.112:32794) 2021-02-10T11:48:11.538 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:11.518 7fa094b9b700 1 -- 172.21.2.112:0/3704861606 learned_addr learned my addr 172.21.2.112:0/3704861606 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:48:11.538 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:11.518 7fa094b9b700 1 -- 172.21.2.112:0/3704861606 >> v2:172.21.2.120:3300/0 conn(0x7fa0900d1c70 msgr2=0x7fa0900efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:11.538 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:11.518 7fa094b9b700 1 --2- 172.21.2.112:0/3704861606 >> v2:172.21.2.120:3300/0 conn(0x7fa0900d1c70 0x7fa0900efc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:11.538 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:11.518 7fa094b9b700 1 -- 172.21.2.112:0/3704861606 >> v1:172.21.2.120:6789/0 conn(0x7fa0900d1870 legacy=0x7fa0901b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:11.538 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:11.518 7fa094b9b700 1 -- 172.21.2.112:0/3704861606 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0900f20a0 con 0x7fa0900d0340 2021-02-10T11:48:11.538 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:11.518 7fa094b9b700 1 --2- 172.21.2.112:0/3704861606 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa0900d0340 0x7fa0900ef640 secure :-1 s=READY pgs=3137 cs=0 l=1 rev1=1 rx=0x7fa08c006700 tx=0x7fa08c009b00).ready entity=mon.0 client_cookie=26444ea98400240c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:11.538 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:11.519 7fa083fff700 1 -- 172.21.2.112:0/3704861606 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa08c00c0c0 con 0x7fa0900d0340 2021-02-10T11:48:11.539 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:11.519 7fa083fff700 1 -- 172.21.2.112:0/3704861606 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa08c00c400 con 0x7fa0900d0340 2021-02-10T11:48:11.539 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:11.519 7fa083fff700 1 -- 172.21.2.112:0/3704861606 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa08c00c920 con 0x7fa0900d0340 2021-02-10T11:48:11.539 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:11.519 7fa097580700 1 -- 172.21.2.112:0/3704861606 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa0900d0340 msgr2=0x7fa0900ef640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:11.539 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:11.519 7fa097580700 1 --2- 172.21.2.112:0/3704861606 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa0900d0340 0x7fa0900ef640 secure :-1 s=READY pgs=3137 cs=0 l=1 rev1=1 rx=0x7fa08c006700 tx=0x7fa08c009b00).stop 2021-02-10T11:48:11.539 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:11.519 7fa097580700 1 -- 172.21.2.112:0/3704861606 shutdown_connections 2021-02-10T11:48:11.539 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:11.519 7fa097580700 1 --2- 172.21.2.112:0/3704861606 >> v2:172.21.2.120:3300/0 conn(0x7fa0900d1c70 0x7fa0900efc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:11.539 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:11.519 7fa097580700 1 --2- 172.21.2.112:0/3704861606 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa0900d0340 0x7fa0900ef640 unknown :-1 s=CLOSED pgs=3137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:11.539 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:11.519 7fa097580700 1 -- 172.21.2.112:0/3704861606 >> 172.21.2.112:0/3704861606 conn(0x7fa0900ce450 msgr2=0x7fa0900d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:11.539 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:11.519 7fa097580700 1 -- 172.21.2.112:0/3704861606 shutdown_connections 2021-02-10T11:48:11.539 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:11.519 7fa097580700 1 -- 172.21.2.112:0/3704861606 wait complete. 2021-02-10T11:48:11.539 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:11.534 7fa097580700 1 Processor -- start 2021-02-10T11:48:11.539 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:11.534 7fa097580700 1 -- start start 2021-02-10T11:48:11.541 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:11.534 7fa097580700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa0900f0280 0x7fa0900a2310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:11.541 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:11.534 7fa097580700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa0900f0680 0x7fa0900a2930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:11.541 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:11.534 7fa097580700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa0900ec180 0x7fa0900ec580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:11.541 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:11.534 7fa097580700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0900efd00 con 0x7fa0900f0680 2021-02-10T11:48:11.541 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:11.534 7fa097580700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0900ecba0 con 0x7fa0900ec180 2021-02-10T11:48:11.541 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:11.534 7fa097580700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0900ecd80 con 0x7fa0900f0280 2021-02-10T11:48:11.541 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:11.535 7fa094b9b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa0900f0680 0x7fa0900a2930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:11.541 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:11.535 7fa094b9b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa0900f0680 0x7fa0900a2930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:32800/0 (socket says 172.21.2.112:32800) 2021-02-10T11:48:11.542 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:11.535 7fa094b9b700 1 -- 172.21.2.112:0/2529692160 learned_addr learned my addr 172.21.2.112:0/2529692160 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:48:11.542 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:11.535 7fa095b9d700 1 --2- 172.21.2.112:0/2529692160 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa0900ec180 0x7fa0900ec580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:11.542 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:11.535 7fa09539c700 1 --2- 172.21.2.112:0/2529692160 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa0900f0280 0x7fa0900a2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:11.542 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:11.535 7fa094b9b700 1 -- 172.21.2.112:0/2529692160 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa0900f0280 msgr2=0x7fa0900a2310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:11.542 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:11.535 7fa094b9b700 1 --2- 172.21.2.112:0/2529692160 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa0900f0280 0x7fa0900a2310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:11.542 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:11.535 7fa094b9b700 1 -- 172.21.2.112:0/2529692160 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa0900ec180 msgr2=0x7fa0900ec580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:11.542 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:11.535 7fa094b9b700 1 --2- 172.21.2.112:0/2529692160 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa0900ec180 0x7fa0900ec580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:11.542 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:11.535 7fa094b9b700 1 -- 172.21.2.112:0/2529692160 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0900ecfa0 con 0x7fa0900f0680 2021-02-10T11:48:11.542 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:11.535 7fa094b9b700 1 --2- 172.21.2.112:0/2529692160 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa0900f0680 0x7fa0900a2930 secure :-1 s=READY pgs=3138 cs=0 l=1 rev1=1 rx=0x7fa08c00b660 tx=0x7fa08c00ba00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:11.542 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:11.535 7fa081ffb700 1 -- 172.21.2.112:0/2529692160 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa08c009440 con 0x7fa0900f0680 2021-02-10T11:48:11.542 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:11.535 7fa081ffb700 1 -- 172.21.2.112:0/2529692160 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa08c0090c0 con 0x7fa0900f0680 2021-02-10T11:48:11.542 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:11.535 7fa081ffb700 1 -- 172.21.2.112:0/2529692160 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa08c0052e0 con 0x7fa0900f0680 2021-02-10T11:48:11.542 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:11.535 7fa097580700 1 -- 172.21.2.112:0/2529692160 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0900ecfa0 con 0x7fa0900f0680 2021-02-10T11:48:11.543 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:11.535 7fa097580700 1 -- 172.21.2.112:0/2529692160 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0900ed530 con 0x7fa0900f0680 2021-02-10T11:48:11.543 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:11.536 7fa081ffb700 1 -- 172.21.2.112:0/2529692160 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa08c00a000 con 0x7fa0900f0680 2021-02-10T11:48:11.543 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:11.536 7fa081ffb700 1 --2- 172.21.2.112:0/2529692160 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa0700186c0 0x7fa07001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:11.543 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:11.536 7fa081ffb700 1 -- 172.21.2.112:0/2529692160 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fa08c032440 con 0x7fa0900f0680 2021-02-10T11:48:11.543 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:11.537 7fa09539c700 1 --2- 172.21.2.112:0/2529692160 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa0700186c0 0x7fa07001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:11.543 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:11.537 7fa09539c700 1 --2- 172.21.2.112:0/2529692160 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa0700186c0 0x7fa07001abc0 crc :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:11.570 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:11.567 7fa097580700 1 -- 172.21.2.112:0/2529692160 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0900ebc60 con 0x7fa0900f0680 2021-02-10T11:48:11.573 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:11.570 7fa081ffb700 1 -- 172.21.2.112:0/2529692160 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa08c0093e0 con 0x7fa0900f0680 2021-02-10T11:48:11.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:11.695 7fa097580700 1 -- 172.21.2.112:0/2529692160 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa0900ebc60 con 0x7fa0900f0680 2021-02-10T11:48:11.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:11.696 7fa081ffb700 1 -- 172.21.2.112:0/2529692160 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v189) v1 ==== 74+0+14007 (secure 0 0 0) 0x7fa08c00a000 con 0x7fa0900f0680 2021-02-10T11:48:11.699 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:48:11.699 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":189,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:48:11.483156","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":41,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":40,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:48:08.454038","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"188","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":41,"pool_name":"cephfs_data","create_time":"2021-02-10 11:48:09.474627","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"189","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:48:11.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:11.697 7fa097580700 1 -- 172.21.2.112:0/2529692160 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa0700186c0 msgr2=0x7fa07001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:11.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:11.697 7fa097580700 1 --2- 172.21.2.112:0/2529692160 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa0700186c0 0x7fa07001abc0 crc :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:11.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:11.697 7fa097580700 1 -- 172.21.2.112:0/2529692160 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa0900f0680 msgr2=0x7fa0900a2930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:11.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:11.697 7fa097580700 1 --2- 172.21.2.112:0/2529692160 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa0900f0680 0x7fa0900a2930 secure :-1 s=READY pgs=3138 cs=0 l=1 rev1=1 rx=0x7fa08c00b660 tx=0x7fa08c00ba00).stop 2021-02-10T11:48:11.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:11.698 7fa097580700 1 -- 172.21.2.112:0/2529692160 shutdown_connections 2021-02-10T11:48:11.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:11.698 7fa097580700 1 --2- 172.21.2.112:0/2529692160 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa0700186c0 0x7fa07001abc0 unknown :-1 s=CLOSED pgs=1412 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:11.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:11.698 7fa097580700 1 --2- 172.21.2.112:0/2529692160 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa0900ec180 0x7fa0900ec580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:11.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:11.698 7fa097580700 1 --2- 172.21.2.112:0/2529692160 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa0900f0680 0x7fa0900a2930 unknown :-1 s=CLOSED pgs=3138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:11.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:11.698 7fa097580700 1 --2- 172.21.2.112:0/2529692160 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa0900f0280 0x7fa0900a2310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:11.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:11.698 7fa097580700 1 -- 172.21.2.112:0/2529692160 >> 172.21.2.112:0/2529692160 conn(0x7fa0900c2130 msgr2=0x7fa09009e3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:11.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:11.698 7fa097580700 1 -- 172.21.2.112:0/2529692160 shutdown_connections 2021-02-10T11:48:11.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:11.698 7fa097580700 1 -- 172.21.2.112:0/2529692160 wait complete. 2021-02-10T11:48:11.709 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2021-02-10T11:48:12.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.123 7f55766e4700 1 Processor -- start 2021-02-10T11:48:12.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.123 7f55766e4700 1 -- start start 2021-02-10T11:48:12.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.123 7f55766e4700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f557010afe0 0x7f55700cf740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:12.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.123 7f55766e4700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55700fe1b0 con 0x7f55700d1c60 2021-02-10T11:48:12.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.123 7f55766e4700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55701b0ca0 con 0x7f55700d1860 2021-02-10T11:48:12.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.123 7f55766e4700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5570109a50 con 0x7f557010afe0 2021-02-10T11:48:12.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.124 7f5574d01700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f557010afe0 0x7f55700cf740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:12.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.124 7f5574d01700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f557010afe0 0x7f55700cf740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:32810/0 (socket says 172.21.2.112:32810) 2021-02-10T11:48:12.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.124 7f556f7fe700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f55700d1c60 0x7f55700cef50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:60502/0 (socket says 172.21.2.112:60502) 2021-02-10T11:48:12.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.124 7f5574d01700 1 -- 172.21.2.112:0/1574931415 learned_addr learned my addr 172.21.2.112:0/1574931415 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:48:12.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.124 7f5574d01700 1 -- 172.21.2.112:0/1574931415 >> v1:172.21.2.120:6789/0 conn(0x7f55700d1860 legacy=0x7f55701b04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:12.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.124 7f5574d01700 1 -- 172.21.2.112:0/1574931415 >> v1:172.21.2.112:6789/0 conn(0x7f55700d1c60 legacy=0x7f55700cef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:12.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.124 7f556effd700 1 -- 172.21.2.112:0/1574931415 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1513702444 0 0) 0x7f555c001100 con 0x7f55700d1c60 2021-02-10T11:48:12.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.124 7f5574d01700 1 -- 172.21.2.112:0/1574931415 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f557010b3e0 con 0x7f557010afe0 2021-02-10T11:48:12.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.124 7f5574d01700 1 --2- 172.21.2.112:0/1574931415 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f557010afe0 0x7f55700cf740 secure :-1 s=READY pgs=3140 cs=0 l=1 rev1=1 rx=0x7f5568006f00 tx=0x7f5568009e90).ready entity=mon.0 client_cookie=edf966a369a098c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:12.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.124 7f556effd700 1 -- 172.21.2.112:0/1574931415 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f556800b300 con 0x7f557010afe0 2021-02-10T11:48:12.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.124 7f556effd700 1 -- 172.21.2.112:0/1574931415 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f556800b580 con 0x7f557010afe0 2021-02-10T11:48:12.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.124 7f556effd700 1 -- 172.21.2.112:0/1574931415 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f556800ba60 con 0x7f557010afe0 2021-02-10T11:48:12.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.125 7f55766e4700 1 -- 172.21.2.112:0/1574931415 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f557010afe0 msgr2=0x7f55700cf740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:12.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.125 7f55766e4700 1 --2- 172.21.2.112:0/1574931415 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f557010afe0 0x7f55700cf740 secure :-1 s=READY pgs=3140 cs=0 l=1 rev1=1 rx=0x7f5568006f00 tx=0x7f5568009e90).stop 2021-02-10T11:48:12.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.125 7f55766e4700 1 -- 172.21.2.112:0/1574931415 shutdown_connections 2021-02-10T11:48:12.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.125 7f55766e4700 1 --2- 172.21.2.112:0/1574931415 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f557010afe0 0x7f55700cf740 unknown :-1 s=CLOSED pgs=3140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:12.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.125 7f55766e4700 1 -- 172.21.2.112:0/1574931415 >> 172.21.2.112:0/1574931415 conn(0x7f55700ce410 msgr2=0x7f55700d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:12.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.125 7f55766e4700 1 -- 172.21.2.112:0/1574931415 shutdown_connections 2021-02-10T11:48:12.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.125 7f55766e4700 1 -- 172.21.2.112:0/1574931415 wait complete. 2021-02-10T11:48:12.146 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.140 7f55766e4700 1 Processor -- start 2021-02-10T11:48:12.146 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.140 7f55766e4700 1 -- start start 2021-02-10T11:48:12.146 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.140 7f55766e4700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f55700c17e0 0x7f55700c1be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:12.146 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.140 7f55766e4700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5570084c60 0x7f55701a7330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:12.146 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.140 7f55766e4700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f55701a7950 0x7f55701a7d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:12.146 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.140 7f55766e4700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5570085120 con 0x7f55701a7950 2021-02-10T11:48:12.146 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.140 7f55766e4700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55700c2200 con 0x7f5570084c60 2021-02-10T11:48:12.147 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.140 7f55766e4700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55700c2540 con 0x7f55700c17e0 2021-02-10T11:48:12.147 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.140 7f5574d01700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f55701a7950 0x7f55701a7d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:12.147 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.140 7f5574d01700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f55701a7950 0x7f55701a7d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:32816/0 (socket says 172.21.2.112:32816) 2021-02-10T11:48:12.147 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.140 7f5574d01700 1 -- 172.21.2.112:0/2485099269 learned_addr learned my addr 172.21.2.112:0/2485099269 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:48:12.147 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.141 7f5574d01700 1 -- 172.21.2.112:0/2485099269 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f55700c17e0 msgr2=0x7f55700c1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:12.147 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.141 7f5574d01700 1 --2- 172.21.2.112:0/2485099269 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f55700c17e0 0x7f55700c1be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:12.147 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.141 7f5574d01700 1 -- 172.21.2.112:0/2485099269 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5570084c60 msgr2=0x7f55701a7330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:12.147 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.141 7f5574d01700 1 --2- 172.21.2.112:0/2485099269 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5570084c60 0x7f55701a7330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:12.147 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.141 7f5574d01700 1 -- 172.21.2.112:0/2485099269 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55700856a0 con 0x7f55701a7950 2021-02-10T11:48:12.147 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.141 7f5574d01700 1 --2- 172.21.2.112:0/2485099269 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f55701a7950 0x7f55701a7d50 secure :-1 s=READY pgs=3141 cs=0 l=1 rev1=1 rx=0x7f556800b910 tx=0x7f556800bcb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:12.147 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.141 7f556cff9700 1 -- 172.21.2.112:0/2485099269 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f556800bdf0 con 0x7f55701a7950 2021-02-10T11:48:12.147 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.141 7f556cff9700 1 -- 172.21.2.112:0/2485099269 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5568005820 con 0x7f55701a7950 2021-02-10T11:48:12.148 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.141 7f556cff9700 1 -- 172.21.2.112:0/2485099269 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5568005d40 con 0x7f55701a7950 2021-02-10T11:48:12.148 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.141 7f55766e4700 1 -- 172.21.2.112:0/2485099269 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55700856a0 con 0x7f55701a7950 2021-02-10T11:48:12.148 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.141 7f55766e4700 1 -- 172.21.2.112:0/2485099269 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55700856a0 con 0x7f55701a7950 2021-02-10T11:48:12.148 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.142 7f556cff9700 1 -- 172.21.2.112:0/2485099269 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5568005a00 con 0x7f55701a7950 2021-02-10T11:48:12.148 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.142 7f556cff9700 1 --2- 172.21.2.112:0/2485099269 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f554c0186c0 0x7f554c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:12.148 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.142 7f556cff9700 1 -- 172.21.2.112:0/2485099269 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f5568032940 con 0x7f55701a7950 2021-02-10T11:48:12.148 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.142 7f556ffff700 1 --2- 172.21.2.112:0/2485099269 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f554c0186c0 0x7f554c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:12.148 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.143 7f556ffff700 1 --2- 172.21.2.112:0/2485099269 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f554c0186c0 0x7f554c01abc0 crc :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:12.176 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.173 7f55766e4700 1 -- 172.21.2.112:0/2485099269 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5570082a20 con 0x7f55701a7950 2021-02-10T11:48:12.179 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.176 7f556cff9700 1 -- 172.21.2.112:0/2485099269 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5568006ef0 con 0x7f55701a7950 2021-02-10T11:48:12.314 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.310 7f55766e4700 1 -- 172.21.2.112:0/2485099269 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"} v 0) v1 -- 0x7f5570082a20 con 0x7f55701a7950 2021-02-10T11:48:12.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.492 7f556cff9700 1 -- 172.21.2.112:0/2485099269 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"}]=0 v165) v1 ==== 118+0+0 (secure 0 0 0) 0x7f55680059e0 con 0x7f55701a7950 2021-02-10T11:48:12.532 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.527 7f55766e4700 1 -- 172.21.2.112:0/2485099269 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f554c0186c0 msgr2=0x7f554c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:12.532 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.527 7f55766e4700 1 --2- 172.21.2.112:0/2485099269 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f554c0186c0 0x7f554c01abc0 crc :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:12.532 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.527 7f55766e4700 1 -- 172.21.2.112:0/2485099269 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f55701a7950 msgr2=0x7f55701a7d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:12.532 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.527 7f55766e4700 1 --2- 172.21.2.112:0/2485099269 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f55701a7950 0x7f55701a7d50 secure :-1 s=READY pgs=3141 cs=0 l=1 rev1=1 rx=0x7f556800b910 tx=0x7f556800bcb0).stop 2021-02-10T11:48:12.532 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.527 7f55766e4700 1 -- 172.21.2.112:0/2485099269 shutdown_connections 2021-02-10T11:48:12.532 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.527 7f55766e4700 1 --2- 172.21.2.112:0/2485099269 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f554c0186c0 0x7f554c01abc0 unknown :-1 s=CLOSED pgs=1413 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:12.533 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.527 7f55766e4700 1 --2- 172.21.2.112:0/2485099269 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5570084c60 0x7f55701a7330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:12.533 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.527 7f55766e4700 1 --2- 172.21.2.112:0/2485099269 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f55701a7950 0x7f55701a7d50 unknown :-1 s=CLOSED pgs=3141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:12.533 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.527 7f55766e4700 1 --2- 172.21.2.112:0/2485099269 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f55700c17e0 0x7f55700c1be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:12.533 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.527 7f55766e4700 1 -- 172.21.2.112:0/2485099269 >> 172.21.2.112:0/2485099269 conn(0x7f55700b5f80 msgr2=0x7f55700be2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:12.533 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.528 7f55766e4700 1 -- 172.21.2.112:0/2485099269 shutdown_connections 2021-02-10T11:48:12.533 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.528 7f55766e4700 1 -- 172.21.2.112:0/2485099269 wait complete. 2021-02-10T11:48:12.540 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:48:12.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.950 7fe1c00a3700 1 Processor -- start 2021-02-10T11:48:12.971 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.950 7fe1c00a3700 1 -- start start 2021-02-10T11:48:12.971 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.950 7fe1c00a3700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fe1b80d1870 0x7fe1b81b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:12.971 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.950 7fe1c00a3700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fe1b81b0a10 0x7fe1b80cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:12.971 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.950 7fe1c00a3700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1b80f0b10 con 0x7fe1b80d1c70 2021-02-10T11:48:12.971 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.950 7fe1c00a3700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1b80e5ed0 con 0x7fe1b81b0a10 2021-02-10T11:48:12.971 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.950 7fe1c00a3700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1b80f0650 con 0x7fe1b80d1870 2021-02-10T11:48:12.971 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.950 7fe1bd6be700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fe1b81b0a10 0x7fe1b80cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:12.971 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.950 7fe1bd6be700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fe1b81b0a10 0x7fe1b80cf330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:32820/0 (socket says 172.21.2.112:32820) 2021-02-10T11:48:12.971 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.950 7fe1bd6be700 1 -- 172.21.2.112:0/1667156249 learned_addr learned my addr 172.21.2.112:0/1667156249 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:48:12.972 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.951 7fe1bd6be700 1 -- 172.21.2.112:0/1667156249 >> v2:172.21.2.120:3300/0 conn(0x7fe1b80d1870 msgr2=0x7fe1b81b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:12.972 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.951 7fe1bd6be700 1 --2- 172.21.2.112:0/1667156249 >> v2:172.21.2.120:3300/0 conn(0x7fe1b80d1870 0x7fe1b81b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:12.972 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.951 7fe1bd6be700 1 -- 172.21.2.112:0/1667156249 >> v1:172.21.2.120:6789/0 conn(0x7fe1b80d1c70 legacy=0x7fe1b80cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:12.972 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.951 7fe1bd6be700 1 -- 172.21.2.112:0/1667156249 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1b80f0840 con 0x7fe1b81b0a10 2021-02-10T11:48:12.972 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.951 7fe1bd6be700 1 --2- 172.21.2.112:0/1667156249 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe1b81b0a10 0x7fe1b80cf330 secure :-1 s=READY pgs=3142 cs=0 l=1 rev1=1 rx=0x7fe1ac006700 tx=0x7fe1ac009b80).ready entity=mon.0 client_cookie=f4a87ee6f1a8b0de server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:12.972 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.951 7fe1bcebd700 1 -- 172.21.2.112:0/1667156249 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe1ac00c0c0 con 0x7fe1b81b0a10 2021-02-10T11:48:12.972 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.951 7fe1bcebd700 1 -- 172.21.2.112:0/1667156249 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1ac00c380 con 0x7fe1b81b0a10 2021-02-10T11:48:12.972 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.951 7fe1bcebd700 1 -- 172.21.2.112:0/1667156249 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe1ac00c8a0 con 0x7fe1b81b0a10 2021-02-10T11:48:12.972 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.951 7fe1c00a3700 1 -- 172.21.2.112:0/1667156249 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe1b81b0a10 msgr2=0x7fe1b80cf330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:12.972 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.951 7fe1c00a3700 1 --2- 172.21.2.112:0/1667156249 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe1b81b0a10 0x7fe1b80cf330 secure :-1 s=READY pgs=3142 cs=0 l=1 rev1=1 rx=0x7fe1ac006700 tx=0x7fe1ac009b80).stop 2021-02-10T11:48:12.972 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.952 7fe1c00a3700 1 -- 172.21.2.112:0/1667156249 shutdown_connections 2021-02-10T11:48:12.972 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.952 7fe1c00a3700 1 --2- 172.21.2.112:0/1667156249 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe1b81b0a10 0x7fe1b80cf330 unknown :-1 s=CLOSED pgs=3142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:12.973 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.952 7fe1c00a3700 1 --2- 172.21.2.112:0/1667156249 >> v2:172.21.2.120:3300/0 conn(0x7fe1b80d1870 0x7fe1b81b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:12.973 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.952 7fe1c00a3700 1 -- 172.21.2.112:0/1667156249 >> 172.21.2.112:0/1667156249 conn(0x7fe1b80ce450 msgr2=0x7fe1b80d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:12.973 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.952 7fe1c00a3700 1 -- 172.21.2.112:0/1667156249 shutdown_connections 2021-02-10T11:48:12.973 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.952 7fe1c00a3700 1 -- 172.21.2.112:0/1667156249 wait complete. 2021-02-10T11:48:12.974 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.967 7fe1c00a3700 1 Processor -- start 2021-02-10T11:48:12.974 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.967 7fe1c00a3700 1 -- start start 2021-02-10T11:48:12.974 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.967 7fe1c00a3700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe1b80b0ad0 0x7fe1b8091f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:12.975 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.967 7fe1c00a3700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe1b81a8850 0x7fe1b81a8c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:12.975 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.967 7fe1c00a3700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe1b81a6fa0 0x7fe1b81a73a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:12.975 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.967 7fe1c00a3700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1b81abfe0 con 0x7fe1b81a8850 2021-02-10T11:48:12.975 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.967 7fe1c00a3700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1b81a9c40 con 0x7fe1b80b0ad0 2021-02-10T11:48:12.975 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.967 7fe1c00a3700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1b81a9730 con 0x7fe1b81a6fa0 2021-02-10T11:48:12.975 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.967 7fe1bd6be700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe1b81a8850 0x7fe1b81a8c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:12.975 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.967 7fe1bd6be700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe1b81a8850 0x7fe1b81a8c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:32822/0 (socket says 172.21.2.112:32822) 2021-02-10T11:48:12.975 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.967 7fe1bd6be700 1 -- 172.21.2.112:0/207518932 learned_addr learned my addr 172.21.2.112:0/207518932 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:48:12.975 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.968 7fe1bd6be700 1 -- 172.21.2.112:0/207518932 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe1b81a6fa0 msgr2=0x7fe1b81a73a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:12.975 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.968 7fe1bd6be700 1 --2- 172.21.2.112:0/207518932 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe1b81a6fa0 0x7fe1b81a73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:12.975 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.968 7fe1bd6be700 1 -- 172.21.2.112:0/207518932 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe1b80b0ad0 msgr2=0x7fe1b8091f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:12.975 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.968 7fe1bd6be700 1 --2- 172.21.2.112:0/207518932 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe1b80b0ad0 0x7fe1b8091f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:12.975 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.968 7fe1bd6be700 1 -- 172.21.2.112:0/207518932 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1b81a9270 con 0x7fe1b81a8850 2021-02-10T11:48:12.976 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.968 7fe1bd6be700 1 --2- 172.21.2.112:0/207518932 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe1b81a8850 0x7fe1b81a8c50 secure :-1 s=READY pgs=3143 cs=0 l=1 rev1=1 rx=0x7fe1ac00b650 tx=0x7fe1ac00b9f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:12.976 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.968 7fe1aa7fc700 1 -- 172.21.2.112:0/207518932 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe1ac00bde0 con 0x7fe1b81a8850 2021-02-10T11:48:12.976 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.968 7fe1aa7fc700 1 -- 172.21.2.112:0/207518932 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1ac002030 con 0x7fe1b81a8850 2021-02-10T11:48:12.976 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.968 7fe1aa7fc700 1 -- 172.21.2.112:0/207518932 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe1ac002490 con 0x7fe1b81a8850 2021-02-10T11:48:12.976 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.968 7fe1c00a3700 1 -- 172.21.2.112:0/207518932 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1b81a9270 con 0x7fe1b81a8850 2021-02-10T11:48:12.976 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.968 7fe1c00a3700 1 -- 172.21.2.112:0/207518932 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1b81a9270 con 0x7fe1b81a8850 2021-02-10T11:48:12.976 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.969 7fe1aa7fc700 1 -- 172.21.2.112:0/207518932 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe1ac000c90 con 0x7fe1b81a8850 2021-02-10T11:48:12.976 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.969 7fe1aa7fc700 1 --2- 172.21.2.112:0/207518932 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe1a00186c0 0x7fe1a001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:12.976 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.969 7fe1aa7fc700 1 -- 172.21.2.112:0/207518932 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fe1ac032430 con 0x7fe1b81a8850 2021-02-10T11:48:12.976 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.969 7fe1bdebf700 1 --2- 172.21.2.112:0/207518932 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe1a00186c0 0x7fe1a001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:12.976 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:12.970 7fe1bdebf700 1 --2- 172.21.2.112:0/207518932 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe1a00186c0 0x7fe1a001abc0 crc :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:13.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:13.000 7fe1c00a3700 1 -- 172.21.2.112:0/207518932 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1b80ce450 con 0x7fe1b81a8850 2021-02-10T11:48:13.007 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:13.003 7fe1aa7fc700 1 -- 172.21.2.112:0/207518932 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe1ac030d60 con 0x7fe1b81a8850 2021-02-10T11:48:13.135 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:13.131 7fe1c00a3700 1 -- 172.21.2.112:0/207518932 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe1b80ce450 con 0x7fe1b81a8850 2021-02-10T11:48:13.135 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:13.131 7fe1aa7fc700 1 -- 172.21.2.112:0/207518932 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 165 v165) v1 ==== 95+0+2107 (secure 0 0 0) 0x7fe1ac00bde0 con 0x7fe1b81a8850 2021-02-10T11:48:13.136 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:48:13.136 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":165,"default_fscid":20,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7763,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/1478301417","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":1478301417},{"type":"v1","addr":"172.21.2.112:6833","nonce":1478301417}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":161}],"filesystems":[{"mdsmap":{"epoch":165,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:48:10.481032","modified":"2021-02-10 11:48:12.490392","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7760":{"gid":7760,"name":"b","rank":0,"incarnation":162,"state":"up:active","state_seq":3,"addr":"172.21.2.120:6835/2943198956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":2943198956},{"type":"v1","addr":"172.21.2.120:6835","nonce":2943198956}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[41],"metadata_pool":40,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":20}]} 2021-02-10T11:48:13.136 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 165 2021-02-10T11:48:13.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:13.133 7fe1c00a3700 1 -- 172.21.2.112:0/207518932 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe1a00186c0 msgr2=0x7fe1a001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:13.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:13.133 7fe1c00a3700 1 --2- 172.21.2.112:0/207518932 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe1a00186c0 0x7fe1a001abc0 crc :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:13.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:13.133 7fe1c00a3700 1 -- 172.21.2.112:0/207518932 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe1b81a8850 msgr2=0x7fe1b81a8c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:13.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:13.133 7fe1c00a3700 1 --2- 172.21.2.112:0/207518932 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe1b81a8850 0x7fe1b81a8c50 secure :-1 s=READY pgs=3143 cs=0 l=1 rev1=1 rx=0x7fe1ac00b650 tx=0x7fe1ac00b9f0).stop 2021-02-10T11:48:13.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:13.133 7fe1c00a3700 1 -- 172.21.2.112:0/207518932 shutdown_connections 2021-02-10T11:48:13.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:13.133 7fe1c00a3700 1 --2- 172.21.2.112:0/207518932 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe1a00186c0 0x7fe1a001abc0 unknown :-1 s=CLOSED pgs=1414 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:13.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:13.133 7fe1c00a3700 1 --2- 172.21.2.112:0/207518932 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe1b81a6fa0 0x7fe1b81a73a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:13.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:13.133 7fe1c00a3700 1 --2- 172.21.2.112:0/207518932 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe1b81a8850 0x7fe1b81a8c50 unknown :-1 s=CLOSED pgs=3143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:13.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:13.133 7fe1c00a3700 1 --2- 172.21.2.112:0/207518932 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe1b80b0ad0 0x7fe1b8091f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:13.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:13.134 7fe1c00a3700 1 -- 172.21.2.112:0/207518932 >> 172.21.2.112:0/207518932 conn(0x7fe1b819f8f0 msgr2=0x7fe1b81a5f80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:13.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:13.134 7fe1c00a3700 1 -- 172.21.2.112:0/207518932 shutdown_connections 2021-02-10T11:48:13.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:13.134 7fe1c00a3700 1 -- 172.21.2.112:0/207518932 wait complete. 2021-02-10T11:48:13.146 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:48:13.578 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:13.556 7f4ed3b13700 1 Processor -- start 2021-02-10T11:48:13.578 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:13.556 7f4ed3b13700 1 -- start start 2021-02-10T11:48:13.578 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:13.556 7f4ed3b13700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4ecc0d1870 0x7f4ecc1b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:13.578 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:13.556 7f4ed3b13700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f4ecc1b0a10 0x7f4ecc0cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:13.578 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:13.556 7f4ed3b13700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ecc0f0b10 con 0x7f4ecc0d1c70 2021-02-10T11:48:13.578 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:13.556 7f4ed3b13700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ecc0fe230 con 0x7f4ecc1b0a10 2021-02-10T11:48:13.578 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:13.556 7f4ed3b13700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ecc0e5ed0 con 0x7f4ecc0d1870 2021-02-10T11:48:13.578 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:13.558 7f4ed112e700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f4ecc1b0a10 0x7f4ecc0cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:13.578 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:13.558 7f4ed2130700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7f4ecc0d1c70 0x7f4ecc0cf950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:54082/0 (socket says 172.21.2.112:54082) 2021-02-10T11:48:13.579 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:13.558 7f4ed192f700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4ecc0d1870 0x7f4ecc1b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:13.579 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:13.558 7f4ed2130700 1 -- 172.21.2.112:0/1901577945 learned_addr learned my addr 172.21.2.112:0/1901577945 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:48:13.579 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:13.558 7f4ed192f700 1 -- 172.21.2.112:0/1901577945 >> v2:172.21.2.120:3300/0 conn(0x7f4ecc1b0a10 msgr2=0x7f4ecc0cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:13.579 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:13.558 7f4ed192f700 1 --2- 172.21.2.112:0/1901577945 >> v2:172.21.2.120:3300/0 conn(0x7f4ecc1b0a10 0x7f4ecc0cf330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:13.579 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:13.558 7f4ed192f700 1 -- 172.21.2.112:0/1901577945 >> v1:172.21.2.120:6789/0 conn(0x7f4ecc0d1c70 legacy=0x7f4ecc0cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:13.579 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:13.558 7f4ed192f700 1 -- 172.21.2.112:0/1901577945 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ecc0f0650 con 0x7f4ecc0d1870 2021-02-10T11:48:13.579 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:13.558 7f4ed192f700 1 --2- 172.21.2.112:0/1901577945 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4ecc0d1870 0x7f4ecc1b03f0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7f4ec0006650 tx=0x7f4ec0009a80).ready entity=mon.2 client_cookie=1f129c68816b113b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:13.579 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:13.558 7f4ed092d700 1 -- 172.21.2.112:0/1901577945 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4ec00009b0 con 0x7f4ecc0d1870 2021-02-10T11:48:13.579 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:13.558 7f4ed092d700 1 -- 172.21.2.112:0/1901577945 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ec000c0c0 con 0x7f4ecc0d1870 2021-02-10T11:48:13.579 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:13.558 7f4ed092d700 1 -- 172.21.2.112:0/1901577945 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4ec000c5e0 con 0x7f4ecc0d1870 2021-02-10T11:48:13.579 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:13.559 7f4ed3b13700 1 -- 172.21.2.112:0/1901577945 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4ecc0d1870 msgr2=0x7f4ecc1b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:13.579 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:13.559 7f4ed3b13700 1 --2- 172.21.2.112:0/1901577945 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4ecc0d1870 0x7f4ecc1b03f0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7f4ec0006650 tx=0x7f4ec0009a80).stop 2021-02-10T11:48:13.579 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:13.559 7f4ed3b13700 1 -- 172.21.2.112:0/1901577945 shutdown_connections 2021-02-10T11:48:13.580 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:13.559 7f4ed3b13700 1 --2- 172.21.2.112:0/1901577945 >> v2:172.21.2.120:3300/0 conn(0x7f4ecc1b0a10 0x7f4ecc0cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:13.580 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:13.559 7f4ed3b13700 1 --2- 172.21.2.112:0/1901577945 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4ecc0d1870 0x7f4ecc1b03f0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:13.580 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:13.559 7f4ed3b13700 1 -- 172.21.2.112:0/1901577945 >> 172.21.2.112:0/1901577945 conn(0x7f4ecc0ce450 msgr2=0x7f4ecc0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:13.580 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:13.560 7f4ed3b13700 1 -- 172.21.2.112:0/1901577945 shutdown_connections 2021-02-10T11:48:13.580 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:13.560 7f4ed3b13700 1 -- 172.21.2.112:0/1901577945 wait complete. 2021-02-10T11:48:13.581 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:13.574 7f4ed3b13700 1 Processor -- start 2021-02-10T11:48:13.581 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:13.574 7f4ed3b13700 1 -- start start 2021-02-10T11:48:13.581 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:13.574 7f4ed3b13700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4ecc18d870 0x7f4ecc0b19b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:13.582 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:13.574 7f4ed3b13700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4ecc0b1fd0 0x7f4ecc099ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:13.582 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:13.574 7f4ed3b13700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4ecc0b23d0 0x7f4ecc09a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:13.582 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:13.574 7f4ed3b13700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ecc18dc70 con 0x7f4ecc18d870 2021-02-10T11:48:13.582 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:13.574 7f4ed3b13700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ecc199e60 con 0x7f4ecc0b1fd0 2021-02-10T11:48:13.582 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:13.574 7f4ed3b13700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ecc09ab30 con 0x7f4ecc0b23d0 2021-02-10T11:48:13.582 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:13.575 7f4ed192f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4ecc18d870 0x7f4ecc0b19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:13.582 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:13.575 7f4ed192f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4ecc18d870 0x7f4ecc0b19b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:32834/0 (socket says 172.21.2.112:32834) 2021-02-10T11:48:13.582 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:13.575 7f4ed192f700 1 -- 172.21.2.112:0/1052377175 learned_addr learned my addr 172.21.2.112:0/1052377175 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:48:13.582 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:13.575 7f4ed2130700 1 --2- 172.21.2.112:0/1052377175 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4ecc0b23d0 0x7f4ecc09a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:13.582 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:13.575 7f4ed112e700 1 --2- 172.21.2.112:0/1052377175 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4ecc0b1fd0 0x7f4ecc099ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:13.582 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:13.575 7f4ed192f700 1 -- 172.21.2.112:0/1052377175 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4ecc0b23d0 msgr2=0x7f4ecc09a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:13.582 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:13.575 7f4ed192f700 1 --2- 172.21.2.112:0/1052377175 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4ecc0b23d0 0x7f4ecc09a510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:13.582 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:13.575 7f4ed192f700 1 -- 172.21.2.112:0/1052377175 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4ecc0b1fd0 msgr2=0x7f4ecc099ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:13.583 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:13.575 7f4ed192f700 1 --2- 172.21.2.112:0/1052377175 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4ecc0b1fd0 0x7f4ecc099ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:13.583 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:13.575 7f4ed192f700 1 -- 172.21.2.112:0/1052377175 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ecc09ad50 con 0x7f4ecc18d870 2021-02-10T11:48:13.583 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:13.575 7f4ed192f700 1 --2- 172.21.2.112:0/1052377175 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4ecc18d870 0x7f4ecc0b19b0 secure :-1 s=READY pgs=3144 cs=0 l=1 rev1=1 rx=0x7f4ec0009630 tx=0x7f4ec000b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:13.583 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:13.575 7f4ebe7fc700 1 -- 172.21.2.112:0/1052377175 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4ec00093a0 con 0x7f4ecc18d870 2021-02-10T11:48:13.583 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:13.576 7f4ebe7fc700 1 -- 172.21.2.112:0/1052377175 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ec000bd70 con 0x7f4ecc18d870 2021-02-10T11:48:13.583 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:13.576 7f4ed3b13700 1 -- 172.21.2.112:0/1052377175 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ecc09ad50 con 0x7f4ecc18d870 2021-02-10T11:48:13.583 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:13.576 7f4ebe7fc700 1 -- 172.21.2.112:0/1052377175 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4ec00054d0 con 0x7f4ecc18d870 2021-02-10T11:48:13.583 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:13.576 7f4ed3b13700 1 -- 172.21.2.112:0/1052377175 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ecc09ad50 con 0x7f4ecc18d870 2021-02-10T11:48:13.583 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:13.576 7f4ebe7fc700 1 -- 172.21.2.112:0/1052377175 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4ec0008240 con 0x7f4ecc18d870 2021-02-10T11:48:13.583 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:13.576 7f4ebe7fc700 1 --2- 172.21.2.112:0/1052377175 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4eb4018730 0x7f4eb401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:13.583 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:13.576 7f4ebe7fc700 1 -- 172.21.2.112:0/1052377175 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f4ec0032650 con 0x7f4ecc18d870 2021-02-10T11:48:13.583 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:13.577 7f4ed112e700 1 --2- 172.21.2.112:0/1052377175 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4eb4018730 0x7f4eb401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:13.583 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:13.577 7f4ed112e700 1 --2- 172.21.2.112:0/1052377175 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4eb4018730 0x7f4eb401ac10 crc :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:13.611 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:13.606 7f4ed3b13700 1 -- 172.21.2.112:0/1052377175 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ecc0999c0 con 0x7f4ecc18d870 2021-02-10T11:48:13.613 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:13.609 7f4ebe7fc700 1 -- 172.21.2.112:0/1052377175 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4ec0007f10 con 0x7f4ecc18d870 2021-02-10T11:48:13.737 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:13.733 7f4ed3b13700 1 -- 172.21.2.112:0/1052377175 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4ecc0999c0 con 0x7f4ecc18d870 2021-02-10T11:48:13.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:13.734 7f4ebe7fc700 1 -- 172.21.2.112:0/1052377175 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v189) v1 ==== 74+0+14007 (secure 0 0 0) 0x7f4ec0008120 con 0x7f4ecc18d870 2021-02-10T11:48:13.739 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:48:13.739 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":189,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:48:11.483156","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":41,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":40,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:48:08.454038","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"188","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":41,"pool_name":"cephfs_data","create_time":"2021-02-10 11:48:09.474627","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"189","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:48:13.740 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:13.736 7f4ed3b13700 1 -- 172.21.2.112:0/1052377175 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4eb4018730 msgr2=0x7f4eb401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:13.740 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:13.736 7f4ed3b13700 1 --2- 172.21.2.112:0/1052377175 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4eb4018730 0x7f4eb401ac10 crc :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:13.740 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:13.736 7f4ed3b13700 1 -- 172.21.2.112:0/1052377175 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4ecc18d870 msgr2=0x7f4ecc0b19b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:13.740 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:13.736 7f4ed3b13700 1 --2- 172.21.2.112:0/1052377175 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4ecc18d870 0x7f4ecc0b19b0 secure :-1 s=READY pgs=3144 cs=0 l=1 rev1=1 rx=0x7f4ec0009630 tx=0x7f4ec000b980).stop 2021-02-10T11:48:13.740 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:13.736 7f4ed3b13700 1 -- 172.21.2.112:0/1052377175 shutdown_connections 2021-02-10T11:48:13.740 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:13.736 7f4ed3b13700 1 --2- 172.21.2.112:0/1052377175 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4eb4018730 0x7f4eb401ac10 unknown :-1 s=CLOSED pgs=1415 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:13.740 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:13.736 7f4ed3b13700 1 --2- 172.21.2.112:0/1052377175 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4ecc0b1fd0 0x7f4ecc099ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:13.740 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:13.736 7f4ed3b13700 1 --2- 172.21.2.112:0/1052377175 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4ecc0b23d0 0x7f4ecc09a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:13.741 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:13.736 7f4ed3b13700 1 --2- 172.21.2.112:0/1052377175 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4ecc18d870 0x7f4ecc0b19b0 unknown :-1 s=CLOSED pgs=3144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:13.741 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:13.736 7f4ed3b13700 1 -- 172.21.2.112:0/1052377175 >> 172.21.2.112:0/1052377175 conn(0x7f4ecc0f0550 msgr2=0x7f4ecc191730 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:13.741 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:13.736 7f4ed3b13700 1 -- 172.21.2.112:0/1052377175 shutdown_connections 2021-02-10T11:48:13.741 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:13.736 7f4ed3b13700 1 -- 172.21.2.112:0/1052377175 wait complete. 2021-02-10T11:48:13.749 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:48:14.191 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.169 7f2e44058700 1 Processor -- start 2021-02-10T11:48:14.191 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.169 7f2e44058700 1 -- start start 2021-02-10T11:48:14.191 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.170 7f2e44058700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2e3c0d1880 0x7f2e3c1b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:14.191 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.170 7f2e44058700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e3c109a70 con 0x7f2e3c1b0af0 2021-02-10T11:48:14.191 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.170 7f2e44058700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e3c088070 con 0x7f2e3c0cff80 2021-02-10T11:48:14.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.170 7f2e44058700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e3c0e5f00 con 0x7f2e3c0d1880 2021-02-10T11:48:14.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.170 7f2e41673700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f2e3c1b0af0 0x7f2e3c108950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:60536/0 (socket says 172.21.2.112:60536) 2021-02-10T11:48:14.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.170 7f2e41673700 1 -- 172.21.2.112:0/945262483 learned_addr learned my addr 172.21.2.112:0/945262483 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:48:14.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.170 7f2e40e72700 1 -- 172.21.2.112:0/945262483 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3214972404 0 0) 0x7f2e380010b0 con 0x7f2e3c1b0af0 2021-02-10T11:48:14.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.170 7f2e40e72700 1 -- 172.21.2.112:0/945262483 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e28002fe0 con 0x7f2e3c1b0af0 2021-02-10T11:48:14.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.171 7f2e41e74700 1 --2- 172.21.2.112:0/945262483 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2e3c0d1880 0x7f2e3c1b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:14.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.171 7f2e40e72700 1 -- 172.21.2.112:0/945262483 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2669129213 0 0) 0x7f2e380010b0 con 0x7f2e3c1b0af0 2021-02-10T11:48:14.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.171 7f2e40e72700 1 -- 172.21.2.112:0/945262483 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2e3c0d1880 msgr2=0x7f2e3c1b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:14.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.171 7f2e40e72700 1 --2- 172.21.2.112:0/945262483 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2e3c0d1880 0x7f2e3c1b04d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:14.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.171 7f2e40e72700 1 -- 172.21.2.112:0/945262483 >> v1:172.21.2.120:6789/0 conn(0x7f2e3c0cff80 legacy=0x7f2e3c109140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:14.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.171 7f2e40e72700 1 -- 172.21.2.112:0/945262483 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e3c0d1c80 con 0x7f2e3c1b0af0 2021-02-10T11:48:14.193 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.171 7f2e40e72700 1 -- 172.21.2.112:0/945262483 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f2e380015b0 con 0x7f2e3c1b0af0 2021-02-10T11:48:14.193 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.171 7f2e40e72700 1 -- 172.21.2.112:0/945262483 >> v1:172.21.2.112:6789/0 conn(0x7f2e3c1b0af0 legacy=0x7f2e3c108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:14.193 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.171 7f2e40e72700 1 --2- 172.21.2.112:0/945262483 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2e28003a70 0x7f2e28005f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:14.193 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.171 7f2e40e72700 1 --2- 172.21.2.112:0/945262483 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2e28006810 0x7f2e28008d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:14.193 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.171 7f2e40e72700 1 --2- 172.21.2.112:0/945262483 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2e28009630 0x7f2e2800bb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:14.193 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.171 7f2e40e72700 1 -- 172.21.2.112:0/945262483 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e2800c370 con 0x7f2e28006810 2021-02-10T11:48:14.193 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.171 7f2e40e72700 1 -- 172.21.2.112:0/945262483 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e2800c590 con 0x7f2e28003a70 2021-02-10T11:48:14.193 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.171 7f2e40e72700 1 -- 172.21.2.112:0/945262483 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e2800c7b0 con 0x7f2e28009630 2021-02-10T11:48:14.193 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.171 7f2e41673700 1 --2- 172.21.2.112:0/945262483 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2e28006810 0x7f2e28008d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:14.193 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.171 7f2e42675700 1 --2- 172.21.2.112:0/945262483 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2e28009630 0x7f2e2800bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:14.193 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.171 7f2e41e74700 1 --2- 172.21.2.112:0/945262483 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2e28003a70 0x7f2e28005f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:14.193 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.171 7f2e41673700 1 -- 172.21.2.112:0/945262483 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2e28009630 msgr2=0x7f2e2800bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:14.193 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.171 7f2e41673700 1 --2- 172.21.2.112:0/945262483 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2e28009630 0x7f2e2800bb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:14.193 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.171 7f2e41673700 1 -- 172.21.2.112:0/945262483 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2e28003a70 msgr2=0x7f2e28005f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:14.194 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.171 7f2e41673700 1 --2- 172.21.2.112:0/945262483 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2e28003a70 0x7f2e28005f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:14.194 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.171 7f2e41673700 1 -- 172.21.2.112:0/945262483 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e2800ca50 con 0x7f2e28006810 2021-02-10T11:48:14.194 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.171 7f2e41673700 1 --2- 172.21.2.112:0/945262483 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2e28006810 0x7f2e28008d30 secure :-1 s=READY pgs=3146 cs=0 l=1 rev1=1 rx=0x7f2e38001c60 tx=0x7f2e38007d40).ready entity=mon.0 client_cookie=34b30931d975f7cf server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:14.194 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.172 7f2e40e72700 1 -- 172.21.2.112:0/945262483 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2e380009b0 con 0x7f2e28006810 2021-02-10T11:48:14.194 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.172 7f2e40e72700 1 -- 172.21.2.112:0/945262483 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e380013f0 con 0x7f2e28006810 2021-02-10T11:48:14.194 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.172 7f2e40e72700 1 -- 172.21.2.112:0/945262483 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2e3804e510 con 0x7f2e28006810 2021-02-10T11:48:14.194 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.172 7f2e44058700 1 -- 172.21.2.112:0/945262483 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2e28006810 msgr2=0x7f2e28008d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:14.194 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.172 7f2e44058700 1 --2- 172.21.2.112:0/945262483 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2e28006810 0x7f2e28008d30 secure :-1 s=READY pgs=3146 cs=0 l=1 rev1=1 rx=0x7f2e38001c60 tx=0x7f2e38007d40).stop 2021-02-10T11:48:14.194 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.172 7f2e42675700 1 -- 172.21.2.112:0/945262483 reap_dead start 2021-02-10T11:48:14.194 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.172 7f2e44058700 1 -- 172.21.2.112:0/945262483 shutdown_connections 2021-02-10T11:48:14.194 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.172 7f2e44058700 1 -- 172.21.2.112:0/945262483 >> 172.21.2.112:0/945262483 conn(0x7f2e3c0ce430 msgr2=0x7f2e3c0d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:14.194 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.173 7f2e44058700 1 -- 172.21.2.112:0/945262483 shutdown_connections 2021-02-10T11:48:14.195 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.173 7f2e44058700 1 -- 172.21.2.112:0/945262483 wait complete. 2021-02-10T11:48:14.196 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.187 7f2e44058700 1 Processor -- start 2021-02-10T11:48:14.196 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.187 7f2e44058700 1 -- start start 2021-02-10T11:48:14.196 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.188 7f2e44058700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2e3c0abd30 0x7f2e3c082bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:14.196 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.188 7f2e44058700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2e3c0831d0 0x7f2e3c08a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:14.196 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.188 7f2e44058700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2e3c0ac130 0x7f2e3c08aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:14.196 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.188 7f2e44058700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e3c083700 con 0x7f2e3c0ac130 2021-02-10T11:48:14.197 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.188 7f2e44058700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e3c08b0c0 con 0x7f2e3c0abd30 2021-02-10T11:48:14.197 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.188 7f2e44058700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e3c08b2a0 con 0x7f2e3c0831d0 2021-02-10T11:48:14.197 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.188 7f2e42675700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2e3c0ac130 0x7f2e3c08aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:14.197 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.188 7f2e42675700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2e3c0ac130 0x7f2e3c08aaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:32850/0 (socket says 172.21.2.112:32850) 2021-02-10T11:48:14.197 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.188 7f2e42675700 1 -- 172.21.2.112:0/1486085082 learned_addr learned my addr 172.21.2.112:0/1486085082 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:48:14.197 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.188 7f2e42675700 1 -- 172.21.2.112:0/1486085082 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2e3c0831d0 msgr2=0x7f2e3c08a480 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:48:14.197 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.188 7f2e42675700 1 --2- 172.21.2.112:0/1486085082 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2e3c0831d0 0x7f2e3c08a480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:14.197 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.188 7f2e42675700 1 -- 172.21.2.112:0/1486085082 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2e3c0abd30 msgr2=0x7f2e3c082bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:48:14.197 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.188 7f2e42675700 1 --2- 172.21.2.112:0/1486085082 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2e3c0abd30 0x7f2e3c082bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:14.197 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.188 7f2e42675700 1 -- 172.21.2.112:0/1486085082 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e3c08b5f0 con 0x7f2e3c0ac130 2021-02-10T11:48:14.197 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.188 7f2e42675700 1 --2- 172.21.2.112:0/1486085082 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2e3c0ac130 0x7f2e3c08aaa0 secure :-1 s=READY pgs=3147 cs=0 l=1 rev1=1 rx=0x7f2e3400a8f0 tx=0x7f2e3400ac90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:14.197 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.188 7f2e2e7fc700 1 -- 172.21.2.112:0/1486085082 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2e3400c300 con 0x7f2e3c0ac130 2021-02-10T11:48:14.197 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.188 7f2e2e7fc700 1 -- 172.21.2.112:0/1486085082 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e3400c540 con 0x7f2e3c0ac130 2021-02-10T11:48:14.198 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.188 7f2e2e7fc700 1 -- 172.21.2.112:0/1486085082 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2e3400ca60 con 0x7f2e3c0ac130 2021-02-10T11:48:14.198 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.188 7f2e44058700 1 -- 172.21.2.112:0/1486085082 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e3c08b5f0 con 0x7f2e3c0ac130 2021-02-10T11:48:14.198 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.189 7f2e44058700 1 -- 172.21.2.112:0/1486085082 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e3c08b5f0 con 0x7f2e3c0ac130 2021-02-10T11:48:14.198 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.189 7f2e2e7fc700 1 -- 172.21.2.112:0/1486085082 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2e34001010 con 0x7f2e3c0ac130 2021-02-10T11:48:14.198 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.189 7f2e2e7fc700 1 --2- 172.21.2.112:0/1486085082 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2e24018730 0x7f2e2401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:14.198 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.189 7f2e2e7fc700 1 -- 172.21.2.112:0/1486085082 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f2e34032b30 con 0x7f2e3c0ac130 2021-02-10T11:48:14.198 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.190 7f2e41e74700 1 --2- 172.21.2.112:0/1486085082 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2e24018730 0x7f2e2401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:14.198 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.190 7f2e41e74700 1 --2- 172.21.2.112:0/1486085082 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2e24018730 0x7f2e2401ac10 crc :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:14.224 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.220 7f2e44058700 1 -- 172.21.2.112:0/1486085082 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e3c082980 con 0x7f2e3c0ac130 2021-02-10T11:48:14.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.222 7f2e2e7fc700 1 -- 172.21.2.112:0/1486085082 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2e3400c300 con 0x7f2e3c0ac130 2021-02-10T11:48:14.357 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.353 7f2e44058700 1 -- 172.21.2.112:0/1486085082 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f2e3c082980 con 0x7f2e3c0ac130 2021-02-10T11:48:14.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.357 7f2e2e7fc700 1 -- 172.21.2.112:0/1486085082 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v110) v1 ==== 184+0+0 (secure 0 0 0) 0x7f2e3401f870 con 0x7f2e3c0ac130 2021-02-10T11:48:14.364 INFO:teuthology.orchestra.run.gibba012.stderr:updated caps for client.0 2021-02-10T11:48:14.366 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.362 7f2e44058700 1 -- 172.21.2.112:0/1486085082 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2e24018730 msgr2=0x7f2e2401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:14.366 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.362 7f2e44058700 1 --2- 172.21.2.112:0/1486085082 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2e24018730 0x7f2e2401ac10 crc :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:14.366 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.362 7f2e44058700 1 -- 172.21.2.112:0/1486085082 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2e3c0ac130 msgr2=0x7f2e3c08aaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:14.366 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.362 7f2e44058700 1 --2- 172.21.2.112:0/1486085082 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2e3c0ac130 0x7f2e3c08aaa0 secure :-1 s=READY pgs=3147 cs=0 l=1 rev1=1 rx=0x7f2e3400a8f0 tx=0x7f2e3400ac90).stop 2021-02-10T11:48:14.366 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.362 7f2e44058700 1 -- 172.21.2.112:0/1486085082 shutdown_connections 2021-02-10T11:48:14.366 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.362 7f2e44058700 1 --2- 172.21.2.112:0/1486085082 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2e24018730 0x7f2e2401ac10 unknown :-1 s=CLOSED pgs=1416 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:14.367 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.362 7f2e44058700 1 --2- 172.21.2.112:0/1486085082 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2e3c0ac130 0x7f2e3c08aaa0 unknown :-1 s=CLOSED pgs=3147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:14.367 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.362 7f2e44058700 1 --2- 172.21.2.112:0/1486085082 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2e3c0831d0 0x7f2e3c08a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:14.367 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.362 7f2e44058700 1 --2- 172.21.2.112:0/1486085082 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2e3c0abd30 0x7f2e3c082bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:14.367 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.362 7f2e44058700 1 -- 172.21.2.112:0/1486085082 >> 172.21.2.112:0/1486085082 conn(0x7f2e3c0afc60 msgr2=0x7f2e3c0b50d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:14.367 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.362 7f2e44058700 1 -- 172.21.2.112:0/1486085082 shutdown_connections 2021-02-10T11:48:14.367 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.362 7f2e44058700 1 -- 172.21.2.112:0/1486085082 wait complete. 2021-02-10T11:48:14.374 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:48:14.815 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.793 7f35130b2700 1 Processor -- start 2021-02-10T11:48:14.815 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.793 7f35130b2700 1 -- start start 2021-02-10T11:48:14.815 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.794 7f35130b2700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f350c0d1c80 0x7f350c0cef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:14.815 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.794 7f35130b2700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f350c0d0180 con 0x7f350c0d1880 2021-02-10T11:48:14.815 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.794 7f35130b2700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f350c0fe1d0 con 0x7f350c0cf590 2021-02-10T11:48:14.815 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.794 7f35130b2700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f350c1b0cc0 con 0x7f350c0d1c80 2021-02-10T11:48:14.815 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.794 7f3510ece700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f350c0d1880 0x7f350c1b04d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:60552/0 (socket says 172.21.2.112:60552) 2021-02-10T11:48:14.815 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.794 7f3510ece700 1 -- 172.21.2.112:0/1172324854 learned_addr learned my addr 172.21.2.112:0/1172324854 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:48:14.815 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.794 7f350bfff700 1 --2- 172.21.2.112:0/1172324854 >> v2:172.21.2.112:3300/0 conn(0x7f350c0d1c80 0x7f350c0cef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:14.815 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.794 7f350bfff700 1 -- 172.21.2.112:0/1172324854 >> v1:172.21.2.120:6789/0 conn(0x7f350c0cf590 legacy=0x7f350c0cf990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:14.816 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.795 7f350bfff700 1 -- 172.21.2.112:0/1172324854 >> v1:172.21.2.112:6789/0 conn(0x7f350c0d1880 legacy=0x7f350c1b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:14.816 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.795 7f350b7fe700 1 -- 172.21.2.112:0/1172324854 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2271711796 0 0) 0x7f34f4000ce0 con 0x7f350c0d1880 2021-02-10T11:48:14.816 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.795 7f350bfff700 1 -- 172.21.2.112:0/1172324854 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f350c109a70 con 0x7f350c0d1c80 2021-02-10T11:48:14.816 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.795 7f350bfff700 1 --2- 172.21.2.112:0/1172324854 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f350c0d1c80 0x7f350c0cef70 secure :-1 s=READY pgs=3149 cs=0 l=1 rev1=1 rx=0x7f34fc006650 tx=0x7f34fc009a00).ready entity=mon.0 client_cookie=92dbb173df5209b7 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:14.816 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.795 7f350b7fe700 1 -- 172.21.2.112:0/1172324854 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f34fc009df0 con 0x7f350c0d1c80 2021-02-10T11:48:14.816 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.795 7f350b7fe700 1 -- 172.21.2.112:0/1172324854 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34fc0009b0 con 0x7f350c0d1c80 2021-02-10T11:48:14.816 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.795 7f350b7fe700 1 -- 172.21.2.112:0/1172324854 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f34fc00c2b0 con 0x7f350c0d1c80 2021-02-10T11:48:14.816 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.795 7f35130b2700 1 -- 172.21.2.112:0/1172324854 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f350c0d1c80 msgr2=0x7f350c0cef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:14.816 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.795 7f35130b2700 1 --2- 172.21.2.112:0/1172324854 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f350c0d1c80 0x7f350c0cef70 secure :-1 s=READY pgs=3149 cs=0 l=1 rev1=1 rx=0x7f34fc006650 tx=0x7f34fc009a00).stop 2021-02-10T11:48:14.816 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.796 7f35130b2700 1 -- 172.21.2.112:0/1172324854 shutdown_connections 2021-02-10T11:48:14.816 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.796 7f35130b2700 1 --2- 172.21.2.112:0/1172324854 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f350c0d1c80 0x7f350c0cef70 unknown :-1 s=CLOSED pgs=3149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:14.816 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.796 7f35130b2700 1 -- 172.21.2.112:0/1172324854 >> 172.21.2.112:0/1172324854 conn(0x7f350c0ce430 msgr2=0x7f350c0d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:14.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.796 7f35130b2700 1 -- 172.21.2.112:0/1172324854 shutdown_connections 2021-02-10T11:48:14.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.796 7f35130b2700 1 -- 172.21.2.112:0/1172324854 wait complete. 2021-02-10T11:48:14.819 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.811 7f35130b2700 1 Processor -- start 2021-02-10T11:48:14.819 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.811 7f35130b2700 1 -- start start 2021-02-10T11:48:14.819 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.811 7f35130b2700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f350c1aaf10 0x7f350c1ab310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:14.819 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.811 7f35130b2700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f350c1a3750 0x7f350c1a3b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:14.819 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.811 7f35130b2700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f350c1a4170 0x7f350c1a4c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:14.819 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.811 7f35130b2700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f350c1abe10 con 0x7f350c1a4170 2021-02-10T11:48:14.819 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.811 7f35130b2700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f350c0b4f10 con 0x7f350c1aaf10 2021-02-10T11:48:14.819 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.811 7f35130b2700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f350c1ab930 con 0x7f350c1a3750 2021-02-10T11:48:14.819 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.812 7f35116cf700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f350c1a4170 0x7f350c1a4c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:14.819 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.812 7f35116cf700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f350c1a4170 0x7f350c1a4c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:32864/0 (socket says 172.21.2.112:32864) 2021-02-10T11:48:14.819 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.812 7f35116cf700 1 -- 172.21.2.112:0/325061238 learned_addr learned my addr 172.21.2.112:0/325061238 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:48:14.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.812 7f350bfff700 1 --2- 172.21.2.112:0/325061238 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f350c1a3750 0x7f350c1a3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:14.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.812 7f35116cf700 1 -- 172.21.2.112:0/325061238 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f350c1a3750 msgr2=0x7f350c1a3b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:14.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.812 7f35116cf700 1 --2- 172.21.2.112:0/325061238 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f350c1a3750 0x7f350c1a3b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:14.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.812 7f35116cf700 1 -- 172.21.2.112:0/325061238 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f350c1aaf10 msgr2=0x7f350c1ab310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:14.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.812 7f35116cf700 1 --2- 172.21.2.112:0/325061238 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f350c1aaf10 0x7f350c1ab310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:14.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.812 7f35116cf700 1 -- 172.21.2.112:0/325061238 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f350c1a5260 con 0x7f350c1a4170 2021-02-10T11:48:14.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.812 7f350bfff700 1 --2- 172.21.2.112:0/325061238 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f350c1a3750 0x7f350c1a3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:48:14.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.812 7f35116cf700 1 --2- 172.21.2.112:0/325061238 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f350c1a4170 0x7f350c1a4c40 secure :-1 s=READY pgs=3150 cs=0 l=1 rev1=1 rx=0x7f3504009940 tx=0x7f350400bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:14.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.812 7f35097fa700 1 -- 172.21.2.112:0/325061238 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3504000ce0 con 0x7f350c1a4170 2021-02-10T11:48:14.821 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.812 7f35097fa700 1 -- 172.21.2.112:0/325061238 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f350400c2f0 con 0x7f350c1a4170 2021-02-10T11:48:14.821 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.812 7f35097fa700 1 -- 172.21.2.112:0/325061238 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f350400c750 con 0x7f350c1a4170 2021-02-10T11:48:14.821 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.812 7f35130b2700 1 -- 172.21.2.112:0/325061238 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f350c1a5260 con 0x7f350c1a4170 2021-02-10T11:48:14.821 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.812 7f35130b2700 1 -- 172.21.2.112:0/325061238 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f350c1a57f0 con 0x7f350c1a4170 2021-02-10T11:48:14.821 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.813 7f35097fa700 1 -- 172.21.2.112:0/325061238 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3504001220 con 0x7f350c1a4170 2021-02-10T11:48:14.821 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.813 7f35097fa700 1 --2- 172.21.2.112:0/325061238 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f34f00186c0 0x7f34f001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:14.821 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.813 7f35097fa700 1 -- 172.21.2.112:0/325061238 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f35040328d0 con 0x7f350c1a4170 2021-02-10T11:48:14.821 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.813 7f3510ece700 1 --2- 172.21.2.112:0/325061238 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f34f00186c0 0x7f34f001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:14.821 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.814 7f3510ece700 1 --2- 172.21.2.112:0/325061238 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f34f00186c0 0x7f34f001abc0 crc :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:14.848 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.844 7f35130b2700 1 -- 172.21.2.112:0/325061238 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f350c0829a0 con 0x7f350c1a4170 2021-02-10T11:48:14.850 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.846 7f35097fa700 1 -- 172.21.2.112:0/325061238 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3504000cc0 con 0x7f350c1a4170 2021-02-10T11:48:14.981 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.977 7f35130b2700 1 -- 172.21.2.112:0/325061238 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f350c0829a0 con 0x7f350c1a4170 2021-02-10T11:48:14.985 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.981 7f35097fa700 1 -- 172.21.2.112:0/325061238 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v111) v1 ==== 184+0+0 (secure 0 0 0) 0x7f350401fa20 con 0x7f350c1a4170 2021-02-10T11:48:14.988 INFO:teuthology.orchestra.run.gibba012.stderr:updated caps for client.1 2021-02-10T11:48:14.990 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.985 7f35130b2700 1 -- 172.21.2.112:0/325061238 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f34f00186c0 msgr2=0x7f34f001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:14.990 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.985 7f35130b2700 1 --2- 172.21.2.112:0/325061238 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f34f00186c0 0x7f34f001abc0 crc :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:14.990 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.985 7f35130b2700 1 -- 172.21.2.112:0/325061238 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f350c1a4170 msgr2=0x7f350c1a4c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:14.990 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.985 7f35130b2700 1 --2- 172.21.2.112:0/325061238 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f350c1a4170 0x7f350c1a4c40 secure :-1 s=READY pgs=3150 cs=0 l=1 rev1=1 rx=0x7f3504009940 tx=0x7f350400bcf0).stop 2021-02-10T11:48:14.990 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.986 7f35130b2700 1 -- 172.21.2.112:0/325061238 shutdown_connections 2021-02-10T11:48:14.990 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.986 7f35130b2700 1 --2- 172.21.2.112:0/325061238 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f34f00186c0 0x7f34f001abc0 unknown :-1 s=CLOSED pgs=1417 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:14.990 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.986 7f35130b2700 1 --2- 172.21.2.112:0/325061238 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f350c1a4170 0x7f350c1a4c40 unknown :-1 s=CLOSED pgs=3150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:14.991 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.986 7f35130b2700 1 --2- 172.21.2.112:0/325061238 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f350c1a3750 0x7f350c1a3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:14.991 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.986 7f35130b2700 1 --2- 172.21.2.112:0/325061238 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f350c1aaf10 0x7f350c1ab310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:14.991 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.986 7f35130b2700 1 -- 172.21.2.112:0/325061238 >> 172.21.2.112:0/325061238 conn(0x7f350c19f880 msgr2=0x7f350c1a71a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:14.991 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.986 7f35130b2700 1 -- 172.21.2.112:0/325061238 shutdown_connections 2021-02-10T11:48:14.991 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:14.986 7f35130b2700 1 -- 172.21.2.112:0/325061238 wait complete. 2021-02-10T11:48:15.000 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.2 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:48:15.443 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:15.421 7fad7da6f700 1 Processor -- start 2021-02-10T11:48:15.443 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:15.422 7fad7da6f700 1 -- start start 2021-02-10T11:48:15.443 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:15.422 7fad7da6f700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fad7810b000 0x7fad780cf760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:15.443 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:15.422 7fad7da6f700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad780fe1d0 con 0x7fad780d1880 2021-02-10T11:48:15.443 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:15.422 7fad7da6f700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad781b0cc0 con 0x7fad780d1c80 2021-02-10T11:48:15.443 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:15.422 7fad7da6f700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad78109a70 con 0x7fad7810b000 2021-02-10T11:48:15.443 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:15.422 7fad777fe700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fad780d1880 0x7fad781b04d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:60566/0 (socket says 172.21.2.112:60566) 2021-02-10T11:48:15.443 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:15.422 7fad777fe700 1 -- 172.21.2.112:0/2216140780 learned_addr learned my addr 172.21.2.112:0/2216140780 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:48:15.443 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:15.422 7fad77fff700 1 --2- 172.21.2.112:0/2216140780 >> v2:172.21.2.120:3300/0 conn(0x7fad7810b000 0x7fad780cf760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:15.443 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:15.423 7fad767fc700 1 -- 172.21.2.112:0/2216140780 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2024852314 0 0) 0x7fad68001180 con 0x7fad780d1880 2021-02-10T11:48:15.443 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:15.423 7fad767fc700 1 -- 172.21.2.112:0/2216140780 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad64002fe0 con 0x7fad780d1880 2021-02-10T11:48:15.444 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:15.423 7fad77fff700 1 -- 172.21.2.112:0/2216140780 >> v1:172.21.2.120:6789/0 conn(0x7fad780d1c80 legacy=0x7fad780cef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:15.444 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:15.423 7fad767fc700 1 -- 172.21.2.112:0/2216140780 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3729420101 0 0) 0x7fad60001080 con 0x7fad780d1c80 2021-02-10T11:48:15.444 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:15.423 7fad77fff700 1 -- 172.21.2.112:0/2216140780 >> v1:172.21.2.112:6789/0 conn(0x7fad780d1880 legacy=0x7fad781b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:15.444 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:15.423 7fad77fff700 1 -- 172.21.2.112:0/2216140780 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad7810b400 con 0x7fad7810b000 2021-02-10T11:48:15.444 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:15.423 7fad77fff700 1 --2- 172.21.2.112:0/2216140780 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fad7810b000 0x7fad780cf760 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 rx=0x7fad70006e50 tx=0x7fad70009e70).ready entity=mon.1 client_cookie=f80ab2ad2c8a3f38 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:15.444 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:15.423 7fad767fc700 1 -- 172.21.2.112:0/2216140780 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fad7000b510 con 0x7fad7810b000 2021-02-10T11:48:15.444 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:15.423 7fad767fc700 1 -- 172.21.2.112:0/2216140780 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad70006790 con 0x7fad7810b000 2021-02-10T11:48:15.444 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:15.423 7fad767fc700 1 -- 172.21.2.112:0/2216140780 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fad7000b940 con 0x7fad7810b000 2021-02-10T11:48:15.444 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:15.423 7fad7da6f700 1 -- 172.21.2.112:0/2216140780 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fad7810b000 msgr2=0x7fad780cf760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:15.444 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:15.423 7fad7da6f700 1 --2- 172.21.2.112:0/2216140780 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fad7810b000 0x7fad780cf760 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 rx=0x7fad70006e50 tx=0x7fad70009e70).stop 2021-02-10T11:48:15.444 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:15.424 7fad7da6f700 1 -- 172.21.2.112:0/2216140780 shutdown_connections 2021-02-10T11:48:15.444 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:15.424 7fad7da6f700 1 --2- 172.21.2.112:0/2216140780 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fad7810b000 0x7fad780cf760 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:15.444 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:15.424 7fad7da6f700 1 -- 172.21.2.112:0/2216140780 >> 172.21.2.112:0/2216140780 conn(0x7fad780ce430 msgr2=0x7fad780d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:15.445 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:15.424 7fad7da6f700 1 -- 172.21.2.112:0/2216140780 shutdown_connections 2021-02-10T11:48:15.445 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:15.424 7fad7da6f700 1 -- 172.21.2.112:0/2216140780 wait complete. 2021-02-10T11:48:15.446 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:15.439 7fad7da6f700 1 Processor -- start 2021-02-10T11:48:15.446 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:15.439 7fad7da6f700 1 -- start start 2021-02-10T11:48:15.446 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:15.439 7fad7da6f700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fad781b9470 0x7fad781ae6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:15.446 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:15.439 7fad7da6f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fad781aecf0 0x7fad780957f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:15.446 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:15.439 7fad7da6f700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fad781af0f0 0x7fad78095e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:15.446 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:15.439 7fad7da6f700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad78096d70 con 0x7fad781aecf0 2021-02-10T11:48:15.446 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:15.439 7fad7da6f700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad780a7780 con 0x7fad781af0f0 2021-02-10T11:48:15.446 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:15.439 7fad7da6f700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad781b9870 con 0x7fad781b9470 2021-02-10T11:48:15.447 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:15.440 7fad76ffd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fad781aecf0 0x7fad780957f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:15.447 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:15.440 7fad76ffd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fad781aecf0 0x7fad780957f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:32880/0 (socket says 172.21.2.112:32880) 2021-02-10T11:48:15.447 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:15.440 7fad76ffd700 1 -- 172.21.2.112:0/3270569797 learned_addr learned my addr 172.21.2.112:0/3270569797 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:48:15.447 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:15.440 7fad76ffd700 1 -- 172.21.2.112:0/3270569797 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fad781b9470 msgr2=0x7fad781ae6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:15.447 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:15.440 7fad76ffd700 1 --2- 172.21.2.112:0/3270569797 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fad781b9470 0x7fad781ae6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:15.447 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:15.440 7fad76ffd700 1 -- 172.21.2.112:0/3270569797 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fad781af0f0 msgr2=0x7fad78095e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:15.447 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:15.440 7fad76ffd700 1 --2- 172.21.2.112:0/3270569797 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fad781af0f0 0x7fad78095e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:15.447 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:15.440 7fad76ffd700 1 -- 172.21.2.112:0/3270569797 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad7819e6b0 con 0x7fad781aecf0 2021-02-10T11:48:15.447 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:15.440 7fad76ffd700 1 --2- 172.21.2.112:0/3270569797 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fad781aecf0 0x7fad780957f0 secure :-1 s=READY pgs=3152 cs=0 l=1 rev1=1 rx=0x7fad60006f30 tx=0x7fad6000b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:15.447 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:15.440 7fad6ffff700 1 -- 172.21.2.112:0/3270569797 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fad6000bd80 con 0x7fad781aecf0 2021-02-10T11:48:15.447 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:15.440 7fad6ffff700 1 -- 172.21.2.112:0/3270569797 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad60006030 con 0x7fad781aecf0 2021-02-10T11:48:15.447 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:15.440 7fad6ffff700 1 -- 172.21.2.112:0/3270569797 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fad60005540 con 0x7fad781aecf0 2021-02-10T11:48:15.448 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:15.440 7fad7da6f700 1 -- 172.21.2.112:0/3270569797 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad7819e6b0 con 0x7fad781aecf0 2021-02-10T11:48:15.448 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:15.440 7fad7da6f700 1 -- 172.21.2.112:0/3270569797 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad78096720 con 0x7fad781aecf0 2021-02-10T11:48:15.448 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:15.441 7fad6ffff700 1 -- 172.21.2.112:0/3270569797 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fad60006210 con 0x7fad781aecf0 2021-02-10T11:48:15.449 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:15.441 7fad6ffff700 1 --2- 172.21.2.112:0/3270569797 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fad5c0186c0 0x7fad5c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:15.449 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:15.441 7fad6ffff700 1 -- 172.21.2.112:0/3270569797 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fad60032560 con 0x7fad781aecf0 2021-02-10T11:48:15.449 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:15.442 7fad777fe700 1 --2- 172.21.2.112:0/3270569797 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fad5c0186c0 0x7fad5c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:15.449 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:15.442 7fad777fe700 1 --2- 172.21.2.112:0/3270569797 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fad5c0186c0 0x7fad5c01abc0 crc :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:15.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:15.472 7fad7da6f700 1 -- 172.21.2.112:0/3270569797 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad780829a0 con 0x7fad781aecf0 2021-02-10T11:48:15.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:15.474 7fad6ffff700 1 -- 172.21.2.112:0/3270569797 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fad60000ba0 con 0x7fad781aecf0 2021-02-10T11:48:15.611 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:15.607 7fad7da6f700 1 -- 172.21.2.112:0/3270569797 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fad780a7780 con 0x7fad781aecf0 2021-02-10T11:48:15.613 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:15.609 7fad6ffff700 1 -- 172.21.2.112:0/3270569797 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.2 v112) v1 ==== 184+0+0 (secure 0 0 0) 0x7fad6001f500 con 0x7fad781aecf0 2021-02-10T11:48:15.614 INFO:teuthology.orchestra.run.gibba012.stderr:updated caps for client.2 2021-02-10T11:48:15.616 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:15.611 7fad7da6f700 1 -- 172.21.2.112:0/3270569797 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fad5c0186c0 msgr2=0x7fad5c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:15.616 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:15.611 7fad7da6f700 1 --2- 172.21.2.112:0/3270569797 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fad5c0186c0 0x7fad5c01abc0 crc :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:15.616 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:15.611 7fad7da6f700 1 -- 172.21.2.112:0/3270569797 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fad781aecf0 msgr2=0x7fad780957f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:15.617 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:15.611 7fad7da6f700 1 --2- 172.21.2.112:0/3270569797 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fad781aecf0 0x7fad780957f0 secure :-1 s=READY pgs=3152 cs=0 l=1 rev1=1 rx=0x7fad60006f30 tx=0x7fad6000b990).stop 2021-02-10T11:48:15.617 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:15.611 7fad7da6f700 1 -- 172.21.2.112:0/3270569797 shutdown_connections 2021-02-10T11:48:15.617 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:15.611 7fad7da6f700 1 --2- 172.21.2.112:0/3270569797 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fad5c0186c0 0x7fad5c01abc0 unknown :-1 s=CLOSED pgs=1418 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:15.617 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:15.611 7fad7da6f700 1 --2- 172.21.2.112:0/3270569797 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fad781af0f0 0x7fad78095e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:15.617 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:15.611 7fad7da6f700 1 --2- 172.21.2.112:0/3270569797 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fad781aecf0 0x7fad780957f0 unknown :-1 s=CLOSED pgs=3152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:15.617 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:15.611 7fad7da6f700 1 --2- 172.21.2.112:0/3270569797 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fad781b9470 0x7fad781ae6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:15.617 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:15.611 7fad7da6f700 1 -- 172.21.2.112:0/3270569797 >> 172.21.2.112:0/3270569797 conn(0x7fad780845b0 msgr2=0x7fad7818c290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:15.617 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:15.612 7fad7da6f700 1 -- 172.21.2.112:0/3270569797 shutdown_connections 2021-02-10T11:48:15.617 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:15.612 7fad7da6f700 1 -- 172.21.2.112:0/3270569797 wait complete. 2021-02-10T11:48:15.624 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.3 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:48:16.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.045 7f62121c4700 1 Processor -- start 2021-02-10T11:48:16.066 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.045 7f62121c4700 1 -- start start 2021-02-10T11:48:16.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.046 7f62121c4700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f620c0d1c80 0x7f620c0cef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:16.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.046 7f62121c4700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f620c0cf590 0x7f620c0cf990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:16.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.046 7f62121c4700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f620c0fe1d0 con 0x7f620c0d1880 2021-02-10T11:48:16.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.046 7f62121c4700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f620c1b0cc0 con 0x7f620c0d1c80 2021-02-10T11:48:16.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.046 7f62121c4700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f620c109a70 con 0x7f620c0cf590 2021-02-10T11:48:16.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.046 7f620b7fe700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f620c0d1880 0x7f620c1b04d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:60580/0 (socket says 172.21.2.112:60580) 2021-02-10T11:48:16.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.046 7f620b7fe700 1 -- 172.21.2.112:0/2062997459 learned_addr learned my addr 172.21.2.112:0/2062997459 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:48:16.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.046 7f620bfff700 1 --2- 172.21.2.112:0/2062997459 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f620c0cf590 0x7f620c0cf990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:16.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.046 7f620affd700 1 --2- 172.21.2.112:0/2062997459 >> v2:172.21.2.120:3300/0 conn(0x7f620c0d1c80 0x7f620c0cef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:16.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.047 7f620a7fc700 1 -- 172.21.2.112:0/2062997459 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1762631950 0 0) 0x7f61fc001180 con 0x7f620c0d1880 2021-02-10T11:48:16.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.047 7f620affd700 1 -- 172.21.2.112:0/2062997459 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f620c0cf590 msgr2=0x7f620c0cf990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:16.067 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.047 7f620affd700 1 --2- 172.21.2.112:0/2062997459 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f620c0cf590 0x7f620c0cf990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:16.068 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.047 7f620affd700 1 -- 172.21.2.112:0/2062997459 >> v1:172.21.2.112:6789/0 conn(0x7f620c0d1880 legacy=0x7f620c1b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:16.068 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.047 7f620affd700 1 -- 172.21.2.112:0/2062997459 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f620c088070 con 0x7f620c0d1c80 2021-02-10T11:48:16.068 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.047 7f620bfff700 1 --2- 172.21.2.112:0/2062997459 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f620c0cf590 0x7f620c0cf990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:48:16.068 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.047 7f620affd700 1 --2- 172.21.2.112:0/2062997459 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f620c0d1c80 0x7f620c0cef70 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 rx=0x7f6200006650 tx=0x7f6200009a80).ready entity=mon.1 client_cookie=abff03d5b6b019d4 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:16.068 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.047 7f620a7fc700 1 -- 172.21.2.112:0/2062997459 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f62000009b0 con 0x7f620c0d1c80 2021-02-10T11:48:16.068 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.047 7f620a7fc700 1 -- 172.21.2.112:0/2062997459 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f620000c0c0 con 0x7f620c0d1c80 2021-02-10T11:48:16.068 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.047 7f620a7fc700 1 -- 172.21.2.112:0/2062997459 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f620000c5e0 con 0x7f620c0d1c80 2021-02-10T11:48:16.068 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.047 7f62121c4700 1 -- 172.21.2.112:0/2062997459 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f620c0d1c80 msgr2=0x7f620c0cef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:16.068 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.047 7f62121c4700 1 --2- 172.21.2.112:0/2062997459 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f620c0d1c80 0x7f620c0cef70 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 rx=0x7f6200006650 tx=0x7f6200009a80).stop 2021-02-10T11:48:16.068 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.048 7f62121c4700 1 -- 172.21.2.112:0/2062997459 shutdown_connections 2021-02-10T11:48:16.068 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.048 7f62121c4700 1 --2- 172.21.2.112:0/2062997459 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f620c0cf590 0x7f620c0cf990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:16.068 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.048 7f62121c4700 1 --2- 172.21.2.112:0/2062997459 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f620c0d1c80 0x7f620c0cef70 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:16.069 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.048 7f62121c4700 1 -- 172.21.2.112:0/2062997459 >> 172.21.2.112:0/2062997459 conn(0x7f620c0ce430 msgr2=0x7f620c0d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:16.069 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.048 7f62121c4700 1 -- 172.21.2.112:0/2062997459 shutdown_connections 2021-02-10T11:48:16.069 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.048 7f62121c4700 1 -- 172.21.2.112:0/2062997459 wait complete. 2021-02-10T11:48:16.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.062 7f62121c4700 1 Processor -- start 2021-02-10T11:48:16.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.063 7f62121c4700 1 -- start start 2021-02-10T11:48:16.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.063 7f62121c4700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f620c1a0580 0x7f620c086fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:16.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.063 7f62121c4700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f620c0b66b0 0x7f620c0b6ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:16.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.063 7f62121c4700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f620c082bb0 0x7f620c082fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:16.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.063 7f62121c4700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f620c0875d0 con 0x7f620c1a0580 2021-02-10T11:48:16.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.063 7f62121c4700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f620c0835d0 con 0x7f620c082bb0 2021-02-10T11:48:16.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.063 7f62121c4700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f620c0837f0 con 0x7f620c0b66b0 2021-02-10T11:48:16.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.063 7f620b7fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f620c1a0580 0x7f620c086fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:16.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.063 7f620b7fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f620c1a0580 0x7f620c086fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:32892/0 (socket says 172.21.2.112:32892) 2021-02-10T11:48:16.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.063 7f620b7fe700 1 -- 172.21.2.112:0/1204767349 learned_addr learned my addr 172.21.2.112:0/1204767349 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:48:16.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.063 7f620affd700 1 --2- 172.21.2.112:0/1204767349 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f620c0b66b0 0x7f620c0b6ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:16.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.063 7f620bfff700 1 --2- 172.21.2.112:0/1204767349 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f620c082bb0 0x7f620c082fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:16.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.063 7f620b7fe700 1 -- 172.21.2.112:0/1204767349 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f620c0b66b0 msgr2=0x7f620c0b6ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:16.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.063 7f620b7fe700 1 --2- 172.21.2.112:0/1204767349 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f620c0b66b0 0x7f620c0b6ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:16.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.063 7f620b7fe700 1 -- 172.21.2.112:0/1204767349 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f620c082bb0 msgr2=0x7f620c082fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:16.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.063 7f620b7fe700 1 --2- 172.21.2.112:0/1204767349 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f620c082bb0 0x7f620c082fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:16.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.063 7f620b7fe700 1 -- 172.21.2.112:0/1204767349 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f620c083a10 con 0x7f620c1a0580 2021-02-10T11:48:16.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.064 7f620b7fe700 1 --2- 172.21.2.112:0/1204767349 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f620c1a0580 0x7f620c086fb0 secure :-1 s=READY pgs=3154 cs=0 l=1 rev1=1 rx=0x7f61fc00a570 tx=0x7f61fc00a910).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:16.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.064 7f61f3fff700 1 -- 172.21.2.112:0/1204767349 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f61fc00ad00 con 0x7f620c1a0580 2021-02-10T11:48:16.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.064 7f61f3fff700 1 -- 172.21.2.112:0/1204767349 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61fc001180 con 0x7f620c1a0580 2021-02-10T11:48:16.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.064 7f62121c4700 1 -- 172.21.2.112:0/1204767349 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f620c083a10 con 0x7f620c1a0580 2021-02-10T11:48:16.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.064 7f61f3fff700 1 -- 172.21.2.112:0/1204767349 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f61fc004620 con 0x7f620c1a0580 2021-02-10T11:48:16.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.064 7f62121c4700 1 -- 172.21.2.112:0/1204767349 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f620c083fa0 con 0x7f620c1a0580 2021-02-10T11:48:16.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.065 7f61f3fff700 1 -- 172.21.2.112:0/1204767349 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f61fc0043a0 con 0x7f620c1a0580 2021-02-10T11:48:16.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.065 7f61f3fff700 1 --2- 172.21.2.112:0/1204767349 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f61ec0186c0 0x7f61ec01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:16.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.065 7f61f3fff700 1 -- 172.21.2.112:0/1204767349 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f61fc0323d0 con 0x7f620c1a0580 2021-02-10T11:48:16.073 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.065 7f620affd700 1 --2- 172.21.2.112:0/1204767349 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f61ec0186c0 0x7f61ec01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:16.073 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.065 7f620affd700 1 --2- 172.21.2.112:0/1204767349 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f61ec0186c0 0x7f61ec01abc0 crc :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:16.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.095 7f62121c4700 1 -- 172.21.2.112:0/1204767349 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f620c082980 con 0x7f620c1a0580 2021-02-10T11:48:16.102 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.098 7f61f3fff700 1 -- 172.21.2.112:0/1204767349 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f61fc0070d0 con 0x7f620c1a0580 2021-02-10T11:48:16.233 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.229 7f62121c4700 1 -- 172.21.2.112:0/1204767349 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f620c082980 con 0x7f620c1a0580 2021-02-10T11:48:16.237 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.233 7f61f3fff700 1 -- 172.21.2.112:0/1204767349 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.3 v113) v1 ==== 184+0+0 (secure 0 0 0) 0x7f61fc0043a0 con 0x7f620c1a0580 2021-02-10T11:48:16.240 INFO:teuthology.orchestra.run.gibba012.stderr:updated caps for client.3 2021-02-10T11:48:16.242 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.238 7f62121c4700 1 -- 172.21.2.112:0/1204767349 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f61ec0186c0 msgr2=0x7f61ec01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:16.242 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.238 7f62121c4700 1 --2- 172.21.2.112:0/1204767349 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f61ec0186c0 0x7f61ec01abc0 crc :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:16.242 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.238 7f62121c4700 1 -- 172.21.2.112:0/1204767349 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f620c1a0580 msgr2=0x7f620c086fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:16.242 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.238 7f62121c4700 1 --2- 172.21.2.112:0/1204767349 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f620c1a0580 0x7f620c086fb0 secure :-1 s=READY pgs=3154 cs=0 l=1 rev1=1 rx=0x7f61fc00a570 tx=0x7f61fc00a910).stop 2021-02-10T11:48:16.242 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.238 7f62121c4700 1 -- 172.21.2.112:0/1204767349 shutdown_connections 2021-02-10T11:48:16.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.238 7f62121c4700 1 --2- 172.21.2.112:0/1204767349 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f61ec0186c0 0x7f61ec01abc0 unknown :-1 s=CLOSED pgs=1419 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:16.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.238 7f62121c4700 1 --2- 172.21.2.112:0/1204767349 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f620c082bb0 0x7f620c082fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:16.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.238 7f62121c4700 1 --2- 172.21.2.112:0/1204767349 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f620c0b66b0 0x7f620c0b6ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:16.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.238 7f62121c4700 1 --2- 172.21.2.112:0/1204767349 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f620c1a0580 0x7f620c086fb0 unknown :-1 s=CLOSED pgs=3154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:16.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.238 7f62121c4700 1 -- 172.21.2.112:0/1204767349 >> 172.21.2.112:0/1204767349 conn(0x7f620c0afc60 msgr2=0x7f620c0b50d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:16.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.238 7f62121c4700 1 -- 172.21.2.112:0/1204767349 shutdown_connections 2021-02-10T11:48:16.243 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.238 7f62121c4700 1 -- 172.21.2.112:0/1204767349 wait complete. 2021-02-10T11:48:16.251 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:48:16.680 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.659 7f6d6a603700 1 Processor -- start 2021-02-10T11:48:16.680 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.659 7f6d6a603700 1 -- start start 2021-02-10T11:48:16.680 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.660 7f6d6a603700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6d641b0470 0x7f6d641b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:16.680 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.660 7f6d6a603700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f6d640f20e0 0x7f6d640f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:16.681 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.660 7f6d6a603700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d640f0b50 con 0x7f6d640ef610 2021-02-10T11:48:16.681 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.660 7f6d6a603700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d640f0820 con 0x7f6d640f20e0 2021-02-10T11:48:16.681 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.660 7f6d6a603700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d640f24e0 con 0x7f6d641b0470 2021-02-10T11:48:16.681 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.660 7f6d637fe700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f6d640ef610 0x7f6d640efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:60594/0 (socket says 172.21.2.112:60594) 2021-02-10T11:48:16.681 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.660 7f6d637fe700 1 -- 172.21.2.112:0/3194422364 learned_addr learned my addr 172.21.2.112:0/3194422364 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:48:16.681 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.660 7f6d68c20700 1 --2- 172.21.2.112:0/3194422364 >> v2:172.21.2.112:3300/0 conn(0x7f6d640f20e0 0x7f6d640f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:16.681 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.660 7f6d63fff700 1 --2- 172.21.2.112:0/3194422364 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6d641b0470 0x7f6d641b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:16.681 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.660 7f6d68c20700 1 -- 172.21.2.112:0/3194422364 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6d641b0470 msgr2=0x7f6d641b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:16.681 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.660 7f6d68c20700 1 --2- 172.21.2.112:0/3194422364 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6d641b0470 0x7f6d641b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:16.681 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.660 7f6d68c20700 1 -- 172.21.2.112:0/3194422364 >> v1:172.21.2.112:6789/0 conn(0x7f6d640ef610 legacy=0x7f6d640efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:16.681 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.660 7f6d68c20700 1 -- 172.21.2.112:0/3194422364 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d640e52e0 con 0x7f6d640f20e0 2021-02-10T11:48:16.681 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.660 7f6d63fff700 1 --2- 172.21.2.112:0/3194422364 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6d641b0470 0x7f6d641b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:48:16.681 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.661 7f6d68c20700 1 --2- 172.21.2.112:0/3194422364 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6d640f20e0 0x7f6d640f0200 secure :-1 s=READY pgs=3156 cs=0 l=1 rev1=1 rx=0x7f6d5c006e30 tx=0x7f6d5c009e60).ready entity=mon.0 client_cookie=d8725b0e15a9ea8e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:16.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.661 7f6d62ffd700 1 -- 172.21.2.112:0/3194422364 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6d5c00b300 con 0x7f6d640f20e0 2021-02-10T11:48:16.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.661 7f6d62ffd700 1 -- 172.21.2.112:0/3194422364 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d5c00b580 con 0x7f6d640f20e0 2021-02-10T11:48:16.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.661 7f6d62ffd700 1 -- 172.21.2.112:0/3194422364 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6d5c00ba60 con 0x7f6d640f20e0 2021-02-10T11:48:16.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.661 7f6d6a603700 1 -- 172.21.2.112:0/3194422364 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6d640f20e0 msgr2=0x7f6d640f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:16.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.661 7f6d6a603700 1 --2- 172.21.2.112:0/3194422364 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6d640f20e0 0x7f6d640f0200 secure :-1 s=READY pgs=3156 cs=0 l=1 rev1=1 rx=0x7f6d5c006e30 tx=0x7f6d5c009e60).stop 2021-02-10T11:48:16.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.661 7f6d6a603700 1 -- 172.21.2.112:0/3194422364 shutdown_connections 2021-02-10T11:48:16.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.661 7f6d6a603700 1 --2- 172.21.2.112:0/3194422364 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6d640f20e0 0x7f6d640f0200 unknown :-1 s=CLOSED pgs=3156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:16.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.661 7f6d6a603700 1 --2- 172.21.2.112:0/3194422364 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6d641b0470 0x7f6d641b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:16.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.661 7f6d6a603700 1 -- 172.21.2.112:0/3194422364 >> 172.21.2.112:0/3194422364 conn(0x7f6d640ce3e0 msgr2=0x7f6d640e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:16.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.662 7f6d6a603700 1 -- 172.21.2.112:0/3194422364 shutdown_connections 2021-02-10T11:48:16.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.662 7f6d6a603700 1 -- 172.21.2.112:0/3194422364 wait complete. 2021-02-10T11:48:16.684 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.676 7f6d6a603700 1 Processor -- start 2021-02-10T11:48:16.684 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.676 7f6d6a603700 1 -- start start 2021-02-10T11:48:16.684 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.677 7f6d6a603700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6d640aa900 0x7f6d641b0590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:16.684 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.677 7f6d6a603700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6d641b0bb0 0x7f6d641b0fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:16.684 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.677 7f6d6a603700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6d641b15d0 0x7f6d64095440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:16.684 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.677 7f6d6a603700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d64095a60 con 0x7f6d640aa900 2021-02-10T11:48:16.684 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.677 7f6d6a603700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d64095c80 con 0x7f6d641b0bb0 2021-02-10T11:48:16.684 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.677 7f6d6a603700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d64095ea0 con 0x7f6d641b15d0 2021-02-10T11:48:16.684 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.677 7f6d63fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6d640aa900 0x7f6d641b0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:16.684 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.677 7f6d63fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6d640aa900 0x7f6d641b0590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:32902/0 (socket says 172.21.2.112:32902) 2021-02-10T11:48:16.685 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.677 7f6d63fff700 1 -- 172.21.2.112:0/2814113575 learned_addr learned my addr 172.21.2.112:0/2814113575 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:48:16.685 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.677 7f6d637fe700 1 --2- 172.21.2.112:0/2814113575 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6d641b0bb0 0x7f6d641b0fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:16.685 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.677 7f6d63fff700 1 -- 172.21.2.112:0/2814113575 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6d641b15d0 msgr2=0x7f6d64095440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:16.685 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.677 7f6d63fff700 1 --2- 172.21.2.112:0/2814113575 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6d641b15d0 0x7f6d64095440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:16.685 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.677 7f6d63fff700 1 -- 172.21.2.112:0/2814113575 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6d641b0bb0 msgr2=0x7f6d641b0fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:16.685 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.677 7f6d63fff700 1 --2- 172.21.2.112:0/2814113575 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6d641b0bb0 0x7f6d641b0fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:16.685 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.677 7f6d63fff700 1 -- 172.21.2.112:0/2814113575 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d640960c0 con 0x7f6d640aa900 2021-02-10T11:48:16.685 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.678 7f6d63fff700 1 --2- 172.21.2.112:0/2814113575 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6d640aa900 0x7f6d641b0590 secure :-1 s=READY pgs=3157 cs=0 l=1 rev1=1 rx=0x7f6d54006f20 tx=0x7f6d5400b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:16.685 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.678 7f6d60ff9700 1 -- 172.21.2.112:0/2814113575 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6d54006bb0 con 0x7f6d640aa900 2021-02-10T11:48:16.685 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.678 7f6d60ff9700 1 -- 172.21.2.112:0/2814113575 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d5400bd80 con 0x7f6d640aa900 2021-02-10T11:48:16.685 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.678 7f6d6a603700 1 -- 172.21.2.112:0/2814113575 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d640960c0 con 0x7f6d640aa900 2021-02-10T11:48:16.685 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.678 7f6d60ff9700 1 -- 172.21.2.112:0/2814113575 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6d54002220 con 0x7f6d640aa900 2021-02-10T11:48:16.686 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.678 7f6d6a603700 1 -- 172.21.2.112:0/2814113575 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d64096650 con 0x7f6d640aa900 2021-02-10T11:48:16.686 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.678 7f6d60ff9700 1 -- 172.21.2.112:0/2814113575 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6d5400a000 con 0x7f6d640aa900 2021-02-10T11:48:16.686 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.679 7f6d60ff9700 1 --2- 172.21.2.112:0/2814113575 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6d44018700 0x7f6d4401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:16.686 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.679 7f6d60ff9700 1 -- 172.21.2.112:0/2814113575 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f6d54032630 con 0x7f6d640aa900 2021-02-10T11:48:16.686 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.679 7f6d637fe700 1 --2- 172.21.2.112:0/2814113575 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6d44018700 0x7f6d4401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:16.686 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.679 7f6d637fe700 1 --2- 172.21.2.112:0/2814113575 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6d44018700 0x7f6d4401abc0 crc :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:16.714 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.710 7f6d6a603700 1 -- 172.21.2.112:0/2814113575 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d640f0b50 con 0x7f6d640aa900 2021-02-10T11:48:16.716 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.713 7f6d60ff9700 1 -- 172.21.2.112:0/2814113575 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6d540009b0 con 0x7f6d640aa900 2021-02-10T11:48:16.844 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.840 7f6d6a603700 1 -- 172.21.2.112:0/2814113575 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6d640f0b50 con 0x7f6d640aa900 2021-02-10T11:48:16.845 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.840 7f6d60ff9700 1 -- 172.21.2.112:0/2814113575 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 165 v165) v1 ==== 95+0+2107 (secure 0 0 0) 0x7f6d5400bd80 con 0x7f6d640aa900 2021-02-10T11:48:16.845 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:48:16.845 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":165,"default_fscid":20,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7763,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/1478301417","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":1478301417},{"type":"v1","addr":"172.21.2.112:6833","nonce":1478301417}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":161}],"filesystems":[{"mdsmap":{"epoch":165,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:48:10.481032","modified":"2021-02-10 11:48:12.490392","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7760":{"gid":7760,"name":"b","rank":0,"incarnation":162,"state":"up:active","state_seq":3,"addr":"172.21.2.120:6835/2943198956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":2943198956},{"type":"v1","addr":"172.21.2.120:6835","nonce":2943198956}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[41],"metadata_pool":40,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":20}]} 2021-02-10T11:48:16.845 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 165 2021-02-10T11:48:16.847 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.843 7f6d6a603700 1 -- 172.21.2.112:0/2814113575 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6d44018700 msgr2=0x7f6d4401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:16.847 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.843 7f6d6a603700 1 --2- 172.21.2.112:0/2814113575 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6d44018700 0x7f6d4401abc0 crc :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:16.847 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.843 7f6d6a603700 1 -- 172.21.2.112:0/2814113575 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6d640aa900 msgr2=0x7f6d641b0590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:16.847 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.843 7f6d6a603700 1 --2- 172.21.2.112:0/2814113575 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6d640aa900 0x7f6d641b0590 secure :-1 s=READY pgs=3157 cs=0 l=1 rev1=1 rx=0x7f6d54006f20 tx=0x7f6d5400b990).stop 2021-02-10T11:48:16.847 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.843 7f6d6a603700 1 -- 172.21.2.112:0/2814113575 shutdown_connections 2021-02-10T11:48:16.847 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.843 7f6d6a603700 1 --2- 172.21.2.112:0/2814113575 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6d44018700 0x7f6d4401abc0 unknown :-1 s=CLOSED pgs=1420 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:16.847 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.843 7f6d6a603700 1 --2- 172.21.2.112:0/2814113575 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6d641b0bb0 0x7f6d641b0fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:16.847 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.843 7f6d6a603700 1 --2- 172.21.2.112:0/2814113575 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6d641b15d0 0x7f6d64095440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:16.847 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.843 7f6d6a603700 1 --2- 172.21.2.112:0/2814113575 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6d640aa900 0x7f6d641b0590 unknown :-1 s=CLOSED pgs=3157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:16.847 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.843 7f6d6a603700 1 -- 172.21.2.112:0/2814113575 >> 172.21.2.112:0/2814113575 conn(0x7f6d6419f7f0 msgr2=0x7f6d641a6740 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:16.847 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.843 7f6d6a603700 1 -- 172.21.2.112:0/2814113575 shutdown_connections 2021-02-10T11:48:16.848 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:16.843 7f6d6a603700 1 -- 172.21.2.112:0/2814113575 wait complete. 2021-02-10T11:48:17.056 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 165, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:48:10.481032', 'modified': '2021-02-10 11:48:12.490392', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7760}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7760': {'gid': 7760, 'name': 'b', 'rank': 0, 'incarnation': 162, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.120:6835/2943198956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.120:6834', 'nonce': 2943198956}, {'type': 'v1', 'addr': '172.21.2.120:6835', 'nonce': 2943198956}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [41], 'metadata_pool': 40, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:48:17.056 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:48:17.056 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:48:17.455 INFO:teuthology.orchestra.run.gibba020.stdout:{ 2021-02-10T11:48:17.455 INFO:teuthology.orchestra.run.gibba020.stdout: "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", 2021-02-10T11:48:17.455 INFO:teuthology.orchestra.run.gibba020.stdout: "whoami": 0, 2021-02-10T11:48:17.456 INFO:teuthology.orchestra.run.gibba020.stdout: "id": 7760, 2021-02-10T11:48:17.456 INFO:teuthology.orchestra.run.gibba020.stdout: "want_state": "up:active", 2021-02-10T11:48:17.456 INFO:teuthology.orchestra.run.gibba020.stdout: "state": "up:active", 2021-02-10T11:48:17.456 INFO:teuthology.orchestra.run.gibba020.stdout: "rank_uptime": 6.9688848959999996, 2021-02-10T11:48:17.456 INFO:teuthology.orchestra.run.gibba020.stdout: "mdsmap_epoch": 165, 2021-02-10T11:48:17.456 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch": 189, 2021-02-10T11:48:17.456 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch_barrier": 189, 2021-02-10T11:48:17.456 INFO:teuthology.orchestra.run.gibba020.stdout: "uptime": 12.619203898 2021-02-10T11:48:17.456 INFO:teuthology.orchestra.run.gibba020.stdout:} 2021-02-10T11:48:17.459 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", "whoami": 0, "id": 7760, "want_state": "up:active", "state": "up:active", "rank_uptime": 6.9688848959999996, "mdsmap_epoch": 165, "osdmap_epoch": 189, "osdmap_epoch_barrier": 189, "uptime": 12.619203898 } 2021-02-10T11:48:17.460 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:48:17.891 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:17.870 7f70c7472700 1 Processor -- start 2021-02-10T11:48:17.894 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:17.870 7f70c7472700 1 -- start start 2021-02-10T11:48:17.894 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:17.871 7f70c7472700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f70c00d0340 0x7f70c00ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:17.894 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:17.871 7f70c7472700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f70c00d1c70 0x7f70c00efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:17.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:17.871 7f70c7472700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70c00f0b10 con 0x7f70c00d1870 2021-02-10T11:48:17.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:17.871 7f70c7472700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70c00e5ed0 con 0x7f70c00d0340 2021-02-10T11:48:17.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:17.871 7f70c7472700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70c01b0be0 con 0x7f70c00d1c70 2021-02-10T11:48:17.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:17.872 7f70c5a8f700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f70c00d1c70 0x7f70c00efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:17.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:17.872 7f70c4a8d700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f70c00d0340 0x7f70c00ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:17.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:17.872 7f70c528e700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7f70c00d1870 0x7f70c01b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:54162/0 (socket says 172.21.2.112:54162) 2021-02-10T11:48:17.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:17.872 7f70c528e700 1 -- 172.21.2.112:0/3740245773 learned_addr learned my addr 172.21.2.112:0/3740245773 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:48:17.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:17.873 7f70c4a8d700 1 -- 172.21.2.112:0/3740245773 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f70c00d1c70 msgr2=0x7f70c00efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:17.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:17.873 7f70c4a8d700 1 --2- 172.21.2.112:0/3740245773 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f70c00d1c70 0x7f70c00efc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:17.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:17.873 7f70c4a8d700 1 -- 172.21.2.112:0/3740245773 >> v1:172.21.2.120:6789/0 conn(0x7f70c00d1870 legacy=0x7f70c01b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:17.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:17.873 7f70c4a8d700 1 -- 172.21.2.112:0/3740245773 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70c00f20a0 con 0x7f70c00d0340 2021-02-10T11:48:17.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:17.873 7f70c5a8f700 1 --2- 172.21.2.112:0/3740245773 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f70c00d1c70 0x7f70c00efc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:48:17.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:17.873 7f70c4a8d700 1 --2- 172.21.2.112:0/3740245773 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f70c00d0340 0x7f70c00ef640 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 rx=0x7f70b4006650 tx=0x7f70b4009a80).ready entity=mon.1 client_cookie=ea3f5b25a1f84bc1 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:17.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:17.873 7f70b3fff700 1 -- 172.21.2.112:0/3740245773 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f70b4006390 con 0x7f70c00d0340 2021-02-10T11:48:17.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:17.873 7f70b3fff700 1 -- 172.21.2.112:0/3740245773 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70b40009b0 con 0x7f70c00d0340 2021-02-10T11:48:17.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:17.873 7f70b3fff700 1 -- 172.21.2.112:0/3740245773 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f70b400c2b0 con 0x7f70c00d0340 2021-02-10T11:48:17.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:17.873 7f70c7472700 1 -- 172.21.2.112:0/3740245773 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f70c00d0340 msgr2=0x7f70c00ef640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:17.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:17.873 7f70c7472700 1 --2- 172.21.2.112:0/3740245773 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f70c00d0340 0x7f70c00ef640 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 rx=0x7f70b4006650 tx=0x7f70b4009a80).stop 2021-02-10T11:48:17.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:17.874 7f70c7472700 1 -- 172.21.2.112:0/3740245773 shutdown_connections 2021-02-10T11:48:17.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:17.874 7f70c7472700 1 --2- 172.21.2.112:0/3740245773 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f70c00d0340 0x7f70c00ef640 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:17.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:17.874 7f70c7472700 1 --2- 172.21.2.112:0/3740245773 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f70c00d1c70 0x7f70c00efc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:17.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:17.874 7f70c7472700 1 -- 172.21.2.112:0/3740245773 >> 172.21.2.112:0/3740245773 conn(0x7f70c00ce450 msgr2=0x7f70c00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:17.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:17.874 7f70c7472700 1 -- 172.21.2.112:0/3740245773 shutdown_connections 2021-02-10T11:48:17.896 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:17.874 7f70c7472700 1 -- 172.21.2.112:0/3740245773 wait complete. 2021-02-10T11:48:17.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:17.889 7f70c7472700 1 Processor -- start 2021-02-10T11:48:17.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:17.889 7f70c7472700 1 -- start start 2021-02-10T11:48:17.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:17.889 7f70c7472700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f70c018d870 0x7f70c00b19b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:17.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:17.889 7f70c7472700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f70c00b1fd0 0x7f70c0099ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:17.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:17.889 7f70c7472700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f70c00b23d0 0x7f70c009a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:17.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:17.889 7f70c7472700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70c018dc70 con 0x7f70c018d870 2021-02-10T11:48:17.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:17.889 7f70c7472700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70c0199e60 con 0x7f70c00b1fd0 2021-02-10T11:48:17.898 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:17.889 7f70c7472700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70c009ab30 con 0x7f70c00b23d0 2021-02-10T11:48:17.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:17.889 7f70c528e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f70c018d870 0x7f70c00b19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:17.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:17.889 7f70c528e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f70c018d870 0x7f70c00b19b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:32916/0 (socket says 172.21.2.112:32916) 2021-02-10T11:48:17.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:17.889 7f70c528e700 1 -- 172.21.2.112:0/754697772 learned_addr learned my addr 172.21.2.112:0/754697772 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:48:17.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:17.890 7f70c5a8f700 1 --2- 172.21.2.112:0/754697772 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f70c00b23d0 0x7f70c009a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:17.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:17.890 7f70c4a8d700 1 --2- 172.21.2.112:0/754697772 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f70c00b1fd0 0x7f70c0099ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:17.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:17.890 7f70c528e700 1 -- 172.21.2.112:0/754697772 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f70c00b23d0 msgr2=0x7f70c009a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:17.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:17.890 7f70c528e700 1 --2- 172.21.2.112:0/754697772 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f70c00b23d0 0x7f70c009a510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:17.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:17.890 7f70c528e700 1 -- 172.21.2.112:0/754697772 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f70c00b1fd0 msgr2=0x7f70c0099ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:17.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:17.890 7f70c528e700 1 --2- 172.21.2.112:0/754697772 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f70c00b1fd0 0x7f70c0099ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:17.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:17.890 7f70c528e700 1 -- 172.21.2.112:0/754697772 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70c009ad50 con 0x7f70c018d870 2021-02-10T11:48:17.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:17.890 7f70c528e700 1 --2- 172.21.2.112:0/754697772 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f70c018d870 0x7f70c00b19b0 secure :-1 s=READY pgs=3158 cs=0 l=1 rev1=1 rx=0x7f70bc001c60 tx=0x7f70bc00a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:17.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:17.890 7f70b1ffb700 1 -- 172.21.2.112:0/754697772 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f70bc00acc0 con 0x7f70c018d870 2021-02-10T11:48:17.899 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:17.890 7f70b1ffb700 1 -- 172.21.2.112:0/754697772 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70bc004140 con 0x7f70c018d870 2021-02-10T11:48:17.900 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:17.890 7f70b1ffb700 1 -- 172.21.2.112:0/754697772 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f70bc0045e0 con 0x7f70c018d870 2021-02-10T11:48:17.900 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:17.890 7f70c7472700 1 -- 172.21.2.112:0/754697772 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70c009ad50 con 0x7f70c018d870 2021-02-10T11:48:17.900 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:17.890 7f70c7472700 1 -- 172.21.2.112:0/754697772 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70c009ad50 con 0x7f70c018d870 2021-02-10T11:48:17.900 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:17.891 7f70b1ffb700 1 -- 172.21.2.112:0/754697772 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f70bc004320 con 0x7f70c018d870 2021-02-10T11:48:17.900 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:17.891 7f70b1ffb700 1 --2- 172.21.2.112:0/754697772 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f70a00186c0 0x7f70a001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:17.900 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:17.891 7f70b1ffb700 1 -- 172.21.2.112:0/754697772 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f70bc0323d0 con 0x7f70c018d870 2021-02-10T11:48:17.900 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:17.891 7f70c4a8d700 1 --2- 172.21.2.112:0/754697772 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f70a00186c0 0x7f70a001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:17.900 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:17.892 7f70c4a8d700 1 --2- 172.21.2.112:0/754697772 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f70a00186c0 0x7f70a001abc0 crc :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:17.924 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:17.920 7f70c7472700 1 -- 172.21.2.112:0/754697772 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70c00999c0 con 0x7f70c018d870 2021-02-10T11:48:17.926 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:17.923 7f70b1ffb700 1 -- 172.21.2.112:0/754697772 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f70bc000e20 con 0x7f70c018d870 2021-02-10T11:48:18.052 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:18.049 7f70c7472700 1 -- 172.21.2.112:0/754697772 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f70c00999c0 con 0x7f70c018d870 2021-02-10T11:48:18.053 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:18.049 7f70b1ffb700 1 -- 172.21.2.112:0/754697772 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v189) v1 ==== 74+0+14007 (secure 0 0 0) 0x7f70bc004320 con 0x7f70c018d870 2021-02-10T11:48:18.053 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:48:18.053 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":189,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:48:11.483156","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":41,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":40,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:48:08.454038","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"188","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":41,"pool_name":"cephfs_data","create_time":"2021-02-10 11:48:09.474627","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"189","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:48:18.054 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:18.051 7f70c7472700 1 -- 172.21.2.112:0/754697772 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f70a00186c0 msgr2=0x7f70a001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:18.054 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:18.051 7f70c7472700 1 --2- 172.21.2.112:0/754697772 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f70a00186c0 0x7f70a001abc0 crc :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:18.054 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:18.051 7f70c7472700 1 -- 172.21.2.112:0/754697772 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f70c018d870 msgr2=0x7f70c00b19b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:18.054 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:18.051 7f70c7472700 1 --2- 172.21.2.112:0/754697772 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f70c018d870 0x7f70c00b19b0 secure :-1 s=READY pgs=3158 cs=0 l=1 rev1=1 rx=0x7f70bc001c60 tx=0x7f70bc00a8d0).stop 2021-02-10T11:48:18.055 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:18.051 7f70c7472700 1 -- 172.21.2.112:0/754697772 shutdown_connections 2021-02-10T11:48:18.055 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:18.051 7f70c7472700 1 --2- 172.21.2.112:0/754697772 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f70a00186c0 0x7f70a001abc0 unknown :-1 s=CLOSED pgs=1421 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:18.055 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:18.051 7f70c7472700 1 --2- 172.21.2.112:0/754697772 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f70c00b1fd0 0x7f70c0099ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:18.055 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:18.051 7f70c7472700 1 --2- 172.21.2.112:0/754697772 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f70c00b23d0 0x7f70c009a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:18.055 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:18.051 7f70c7472700 1 --2- 172.21.2.112:0/754697772 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f70c018d870 0x7f70c00b19b0 unknown :-1 s=CLOSED pgs=3158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:18.055 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:18.051 7f70c7472700 1 -- 172.21.2.112:0/754697772 >> 172.21.2.112:0/754697772 conn(0x7f70c00f0550 msgr2=0x7f70c0191730 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:18.056 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:18.052 7f70c7472700 1 -- 172.21.2.112:0/754697772 shutdown_connections 2021-02-10T11:48:18.056 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:18.052 7f70c7472700 1 -- 172.21.2.112:0/754697772 wait complete. 2021-02-10T11:48:18.063 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:13:57.508758', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 40, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:48:08.454038', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '188', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 41, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:48:09.474627', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '189', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:48:18.064 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:48:18.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:18.476 7eff25ab0700 1 Processor -- start 2021-02-10T11:48:18.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:18.476 7eff25ab0700 1 -- start start 2021-02-10T11:48:18.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:18.477 7eff25ab0700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7eff200d1870 0x7eff201b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:18.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:18.477 7eff25ab0700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7eff201b0ae0 0x7eff20108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:18.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:18.477 7eff25ab0700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff200fe1c0 con 0x7eff200d1c70 2021-02-10T11:48:18.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:18.477 7eff25ab0700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff20109a60 con 0x7eff200d1870 2021-02-10T11:48:18.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:18.477 7eff25ab0700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff20109750 con 0x7eff201b0ae0 2021-02-10T11:48:18.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:18.477 7eff1f7fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7eff200d1870 0x7eff201b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:18.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:18.477 7eff1f7fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7eff200d1870 0x7eff201b04c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:32924/0 (socket says 172.21.2.112:32924) 2021-02-10T11:48:18.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:18.477 7eff1f7fe700 1 -- 172.21.2.112:0/1520287127 learned_addr learned my addr 172.21.2.112:0/1520287127 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:48:18.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:18.477 7eff1f7fe700 1 -- 172.21.2.112:0/1520287127 >> v2:172.21.2.120:3300/0 conn(0x7eff201b0ae0 msgr2=0x7eff20108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:18.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:18.477 7eff1f7fe700 1 --2- 172.21.2.112:0/1520287127 >> v2:172.21.2.120:3300/0 conn(0x7eff201b0ae0 0x7eff20108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:18.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:18.477 7eff1f7fe700 1 -- 172.21.2.112:0/1520287127 >> v1:172.21.2.120:6789/0 conn(0x7eff200d1c70 legacy=0x7eff20108f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:18.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:18.477 7eff1f7fe700 1 -- 172.21.2.112:0/1520287127 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff20088070 con 0x7eff200d1870 2021-02-10T11:48:18.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:18.478 7eff1f7fe700 1 --2- 172.21.2.112:0/1520287127 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7eff200d1870 0x7eff201b04c0 secure :-1 s=READY pgs=3159 cs=0 l=1 rev1=1 rx=0x7eff10006700 tx=0x7eff10009b00).ready entity=mon.0 client_cookie=b19462574339f723 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:18.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:18.478 7eff1e7fc700 1 -- 172.21.2.112:0/1520287127 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7eff1000c0c0 con 0x7eff200d1870 2021-02-10T11:48:18.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:18.478 7eff1e7fc700 1 -- 172.21.2.112:0/1520287127 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff1000c340 con 0x7eff200d1870 2021-02-10T11:48:18.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:18.478 7eff1e7fc700 1 -- 172.21.2.112:0/1520287127 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7eff1000c820 con 0x7eff200d1870 2021-02-10T11:48:18.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:18.478 7eff25ab0700 1 -- 172.21.2.112:0/1520287127 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7eff200d1870 msgr2=0x7eff201b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:18.498 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:18.478 7eff25ab0700 1 --2- 172.21.2.112:0/1520287127 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7eff200d1870 0x7eff201b04c0 secure :-1 s=READY pgs=3159 cs=0 l=1 rev1=1 rx=0x7eff10006700 tx=0x7eff10009b00).stop 2021-02-10T11:48:18.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:18.478 7eff25ab0700 1 -- 172.21.2.112:0/1520287127 shutdown_connections 2021-02-10T11:48:18.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:18.478 7eff25ab0700 1 --2- 172.21.2.112:0/1520287127 >> v2:172.21.2.120:3300/0 conn(0x7eff201b0ae0 0x7eff20108940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:18.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:18.478 7eff25ab0700 1 --2- 172.21.2.112:0/1520287127 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7eff200d1870 0x7eff201b04c0 unknown :-1 s=CLOSED pgs=3159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:18.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:18.478 7eff25ab0700 1 -- 172.21.2.112:0/1520287127 >> 172.21.2.112:0/1520287127 conn(0x7eff200ce420 msgr2=0x7eff200d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:18.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:18.479 7eff25ab0700 1 -- 172.21.2.112:0/1520287127 shutdown_connections 2021-02-10T11:48:18.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:18.479 7eff25ab0700 1 -- 172.21.2.112:0/1520287127 wait complete. 2021-02-10T11:48:18.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:18.493 7eff25ab0700 1 Processor -- start 2021-02-10T11:48:18.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:18.493 7eff25ab0700 1 -- start start 2021-02-10T11:48:18.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:18.494 7eff25ab0700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7eff200c2370 0x7eff2018f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:18.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:18.494 7eff25ab0700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7eff20084cb0 0x7eff201a7470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:18.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:18.494 7eff25ab0700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7eff201a7a90 0x7eff20085700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:18.501 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:18.494 7eff25ab0700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff200be840 con 0x7eff20084cb0 2021-02-10T11:48:18.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:18.494 7eff25ab0700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff200c29a0 con 0x7eff200c2370 2021-02-10T11:48:18.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:18.494 7eff25ab0700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff201a7e90 con 0x7eff201a7a90 2021-02-10T11:48:18.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:18.494 7eff1effd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7eff20084cb0 0x7eff201a7470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:18.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:18.494 7eff1effd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7eff20084cb0 0x7eff201a7470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:32932/0 (socket says 172.21.2.112:32932) 2021-02-10T11:48:18.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:18.494 7eff1effd700 1 -- 172.21.2.112:0/4015242911 learned_addr learned my addr 172.21.2.112:0/4015242911 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:48:18.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:18.494 7eff1ffff700 1 --2- 172.21.2.112:0/4015242911 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7eff201a7a90 0x7eff20085700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:18.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:18.494 7eff1f7fe700 1 --2- 172.21.2.112:0/4015242911 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7eff200c2370 0x7eff2018f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:18.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:18.494 7eff1effd700 1 -- 172.21.2.112:0/4015242911 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7eff201a7a90 msgr2=0x7eff20085700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:18.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:18.494 7eff1effd700 1 --2- 172.21.2.112:0/4015242911 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7eff201a7a90 0x7eff20085700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:18.502 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:18.494 7eff1effd700 1 -- 172.21.2.112:0/4015242911 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7eff200c2370 msgr2=0x7eff2018f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:18.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:18.494 7eff1effd700 1 --2- 172.21.2.112:0/4015242911 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7eff200c2370 0x7eff2018f870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:18.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:18.494 7eff1effd700 1 -- 172.21.2.112:0/4015242911 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff201a80b0 con 0x7eff20084cb0 2021-02-10T11:48:18.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:18.494 7eff1f7fe700 1 --2- 172.21.2.112:0/4015242911 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7eff200c2370 0x7eff2018f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:48:18.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:18.494 7eff1effd700 1 --2- 172.21.2.112:0/4015242911 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7eff20084cb0 0x7eff201a7470 secure :-1 s=READY pgs=3160 cs=0 l=1 rev1=1 rx=0x7eff08006f00 tx=0x7eff0800b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:18.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:18.494 7eff17fff700 1 -- 172.21.2.112:0/4015242911 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7eff0800bd30 con 0x7eff20084cb0 2021-02-10T11:48:18.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:18.495 7eff25ab0700 1 -- 172.21.2.112:0/4015242911 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff201a80b0 con 0x7eff20084cb0 2021-02-10T11:48:18.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:18.495 7eff17fff700 1 -- 172.21.2.112:0/4015242911 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff0800bd30 con 0x7eff20084cb0 2021-02-10T11:48:18.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:18.495 7eff17fff700 1 -- 172.21.2.112:0/4015242911 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7eff08000c90 con 0x7eff20084cb0 2021-02-10T11:48:18.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:18.495 7eff25ab0700 1 -- 172.21.2.112:0/4015242911 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff201a80b0 con 0x7eff20084cb0 2021-02-10T11:48:18.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:18.495 7eff17fff700 1 -- 172.21.2.112:0/4015242911 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7eff08000c90 con 0x7eff20084cb0 2021-02-10T11:48:18.503 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:18.495 7eff17fff700 1 --2- 172.21.2.112:0/4015242911 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7eff04018730 0x7eff0401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:18.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:18.495 7eff17fff700 1 -- 172.21.2.112:0/4015242911 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 5573+0+0 (secure 0 0 0) 0x7eff080327c0 con 0x7eff20084cb0 2021-02-10T11:48:18.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:18.496 7eff1f7fe700 1 --2- 172.21.2.112:0/4015242911 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7eff04018730 0x7eff0401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:18.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:18.496 7eff1f7fe700 1 --2- 172.21.2.112:0/4015242911 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7eff04018730 0x7eff0401ac10 crc :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:18.532 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:18.527 7eff25ab0700 1 -- 172.21.2.112:0/4015242911 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff200be840 con 0x7eff20084cb0 2021-02-10T11:48:18.532 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:18.529 7eff17fff700 1 -- 172.21.2.112:0/4015242911 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7eff08006b90 con 0x7eff20084cb0 2021-02-10T11:48:18.661 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:18.658 7eff25ab0700 1 -- 172.21.2.112:0/4015242911 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7eff200be840 con 0x7eff20084cb0 2021-02-10T11:48:18.662 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:18.658 7eff17fff700 1 -- 172.21.2.112:0/4015242911 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v189) v1 ==== 92+0+10 (secure 0 0 0) 0x7eff0800bd30 con 0x7eff20084cb0 2021-02-10T11:48:18.662 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:48:18.666 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:18.660 7eff25ab0700 1 -- 172.21.2.112:0/4015242911 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7eff04018730 msgr2=0x7eff0401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:18.666 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:18.660 7eff25ab0700 1 --2- 172.21.2.112:0/4015242911 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7eff04018730 0x7eff0401ac10 crc :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:18.666 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:18.660 7eff25ab0700 1 -- 172.21.2.112:0/4015242911 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7eff20084cb0 msgr2=0x7eff201a7470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:18.666 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:18.660 7eff25ab0700 1 --2- 172.21.2.112:0/4015242911 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7eff20084cb0 0x7eff201a7470 secure :-1 s=READY pgs=3160 cs=0 l=1 rev1=1 rx=0x7eff08006f00 tx=0x7eff0800b940).stop 2021-02-10T11:48:18.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:18.660 7eff25ab0700 1 -- 172.21.2.112:0/4015242911 shutdown_connections 2021-02-10T11:48:18.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:18.660 7eff25ab0700 1 --2- 172.21.2.112:0/4015242911 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7eff04018730 0x7eff0401ac10 unknown :-1 s=CLOSED pgs=1422 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:18.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:18.660 7eff25ab0700 1 --2- 172.21.2.112:0/4015242911 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7eff201a7a90 0x7eff20085700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:18.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:18.660 7eff25ab0700 1 --2- 172.21.2.112:0/4015242911 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7eff20084cb0 0x7eff201a7470 unknown :-1 s=CLOSED pgs=3160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:18.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:18.660 7eff25ab0700 1 --2- 172.21.2.112:0/4015242911 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7eff200c2370 0x7eff2018f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:18.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:18.660 7eff25ab0700 1 -- 172.21.2.112:0/4015242911 >> 172.21.2.112:0/4015242911 conn(0x7eff200b5dc0 msgr2=0x7eff200c2d90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:18.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:18.661 7eff25ab0700 1 -- 172.21.2.112:0/4015242911 shutdown_connections 2021-02-10T11:48:18.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:18.661 7eff25ab0700 1 -- 172.21.2.112:0/4015242911 wait complete. 2021-02-10T11:48:18.673 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:48:19.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.109 7f90838f5700 1 Processor -- start 2021-02-10T11:48:19.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.109 7f90838f5700 1 -- start start 2021-02-10T11:48:19.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.110 7f90838f5700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f907c0d1870 0x7f907c1b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:19.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.110 7f90838f5700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f907c1b0ae0 0x7f907c108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:19.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.110 7f90838f5700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f907c0d1c70 0x7f907c108f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:19.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.110 7f90838f5700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f907c0e5ef0 con 0x7f907c1b0ae0 2021-02-10T11:48:19.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.110 7f90838f5700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f907c109a60 con 0x7f907c0d1c70 2021-02-10T11:48:19.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.110 7f90838f5700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f907c088070 con 0x7f907c0d1870 2021-02-10T11:48:19.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.110 7f9080f10700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f907c1b0ae0 0x7f907c108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:19.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.110 7f9080f10700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f907c1b0ae0 0x7f907c108940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:32938/0 (socket says 172.21.2.112:32938) 2021-02-10T11:48:19.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.110 7f9080f10700 1 -- 172.21.2.112:0/2794078243 learned_addr learned my addr 172.21.2.112:0/2794078243 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:48:19.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.110 7f9080f10700 1 -- 172.21.2.112:0/2794078243 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f907c0d1870 msgr2=0x7f907c1b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:19.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.110 7f9080f10700 1 --2- 172.21.2.112:0/2794078243 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f907c0d1870 0x7f907c1b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:19.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.110 7f9080f10700 1 -- 172.21.2.112:0/2794078243 >> v2:172.21.2.120:3300/0 conn(0x7f907c0d1c70 msgr2=0x7f907c108f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:19.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.110 7f9080f10700 1 --2- 172.21.2.112:0/2794078243 >> v2:172.21.2.120:3300/0 conn(0x7f907c0d1c70 0x7f907c108f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:19.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.110 7f9080f10700 1 -- 172.21.2.112:0/2794078243 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f907c109580 con 0x7f907c1b0ae0 2021-02-10T11:48:19.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.111 7f9080f10700 1 --2- 172.21.2.112:0/2794078243 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f907c1b0ae0 0x7f907c108940 secure :-1 s=READY pgs=3161 cs=0 l=1 rev1=1 rx=0x7f9070006700 tx=0x7f9070009b00).ready entity=mon.0 client_cookie=37ff2ece44b7efee server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:19.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.111 7f906ffff700 1 -- 172.21.2.112:0/2794078243 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f907000c0c0 con 0x7f907c1b0ae0 2021-02-10T11:48:19.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.111 7f906ffff700 1 -- 172.21.2.112:0/2794078243 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f907000c340 con 0x7f907c1b0ae0 2021-02-10T11:48:19.131 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.111 7f906ffff700 1 -- 172.21.2.112:0/2794078243 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f907000c820 con 0x7f907c1b0ae0 2021-02-10T11:48:19.131 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.111 7f90838f5700 1 -- 172.21.2.112:0/2794078243 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f907c1b0ae0 msgr2=0x7f907c108940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:19.131 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.111 7f90838f5700 1 --2- 172.21.2.112:0/2794078243 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f907c1b0ae0 0x7f907c108940 secure :-1 s=READY pgs=3161 cs=0 l=1 rev1=1 rx=0x7f9070006700 tx=0x7f9070009b00).stop 2021-02-10T11:48:19.131 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.111 7f90838f5700 1 -- 172.21.2.112:0/2794078243 shutdown_connections 2021-02-10T11:48:19.131 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.111 7f90838f5700 1 --2- 172.21.2.112:0/2794078243 >> v2:172.21.2.120:3300/0 conn(0x7f907c0d1c70 0x7f907c108f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:19.131 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.111 7f90838f5700 1 --2- 172.21.2.112:0/2794078243 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f907c1b0ae0 0x7f907c108940 unknown :-1 s=CLOSED pgs=3161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:19.131 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.111 7f90838f5700 1 --2- 172.21.2.112:0/2794078243 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f907c0d1870 0x7f907c1b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:19.131 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.111 7f90838f5700 1 -- 172.21.2.112:0/2794078243 >> 172.21.2.112:0/2794078243 conn(0x7f907c0ce420 msgr2=0x7f907c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:19.131 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.112 7f90838f5700 1 -- 172.21.2.112:0/2794078243 shutdown_connections 2021-02-10T11:48:19.131 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.112 7f90838f5700 1 -- 172.21.2.112:0/2794078243 wait complete. 2021-02-10T11:48:19.133 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.126 7f90838f5700 1 Processor -- start 2021-02-10T11:48:19.133 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.126 7f90838f5700 1 -- start start 2021-02-10T11:48:19.133 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.127 7f90838f5700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f907c0a73c0 0x7f907c0a6b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:19.133 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.127 7f90838f5700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f907c0add40 0x7f907c0ae140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:19.133 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.127 7f90838f5700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f907c0ae760 0x7f907c0af180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:19.133 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.127 7f90838f5700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f907c0aec00 con 0x7f907c0ae760 2021-02-10T11:48:19.133 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.127 7f90838f5700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f907c0875d0 con 0x7f907c0add40 2021-02-10T11:48:19.134 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.127 7f90838f5700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f907c0af7a0 con 0x7f907c0a73c0 2021-02-10T11:48:19.134 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.127 7f9081f12700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f907c0ae760 0x7f907c0af180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:19.134 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.127 7f9081f12700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f907c0ae760 0x7f907c0af180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:32942/0 (socket says 172.21.2.112:32942) 2021-02-10T11:48:19.134 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.127 7f9081f12700 1 -- 172.21.2.112:0/991434868 learned_addr learned my addr 172.21.2.112:0/991434868 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:48:19.134 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.127 7f9080f10700 1 --2- 172.21.2.112:0/991434868 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f907c0add40 0x7f907c0ae140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:19.134 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.127 7f9081f12700 1 -- 172.21.2.112:0/991434868 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f907c0a73c0 msgr2=0x7f907c0a6b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:19.134 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.127 7f9081711700 1 --2- 172.21.2.112:0/991434868 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f907c0a73c0 0x7f907c0a6b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:19.134 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.127 7f9081f12700 1 --2- 172.21.2.112:0/991434868 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f907c0a73c0 0x7f907c0a6b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:19.134 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.127 7f9081f12700 1 -- 172.21.2.112:0/991434868 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f907c0add40 msgr2=0x7f907c0ae140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:19.134 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.127 7f9081f12700 1 --2- 172.21.2.112:0/991434868 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f907c0add40 0x7f907c0ae140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:19.134 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.127 7f9081f12700 1 -- 172.21.2.112:0/991434868 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f907c0af9c0 con 0x7f907c0ae760 2021-02-10T11:48:19.134 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.127 7f9080f10700 1 --2- 172.21.2.112:0/991434868 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f907c0add40 0x7f907c0ae140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:48:19.135 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.127 7f9081711700 1 --2- 172.21.2.112:0/991434868 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f907c0a73c0 0x7f907c0a6b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:48:19.135 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.128 7f9081f12700 1 --2- 172.21.2.112:0/991434868 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f907c0ae760 0x7f907c0af180 secure :-1 s=READY pgs=3162 cs=0 l=1 rev1=1 rx=0x7f9074009980 tx=0x7f907400bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:19.135 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.128 7f906dffb700 1 -- 172.21.2.112:0/991434868 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9074005a60 con 0x7f907c0ae760 2021-02-10T11:48:19.135 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.128 7f906dffb700 1 -- 172.21.2.112:0/991434868 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9074005ce0 con 0x7f907c0ae760 2021-02-10T11:48:19.135 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.128 7f906dffb700 1 -- 172.21.2.112:0/991434868 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9074006220 con 0x7f907c0ae760 2021-02-10T11:48:19.135 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.128 7f90838f5700 1 -- 172.21.2.112:0/991434868 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f907c0af9c0 con 0x7f907c0ae760 2021-02-10T11:48:19.135 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.128 7f90838f5700 1 -- 172.21.2.112:0/991434868 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f907c0af9c0 con 0x7f907c0ae760 2021-02-10T11:48:19.135 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.128 7f906dffb700 1 -- 172.21.2.112:0/991434868 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9074005870 con 0x7f907c0ae760 2021-02-10T11:48:19.135 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.129 7f906dffb700 1 --2- 172.21.2.112:0/991434868 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f905c0186c0 0x7f905c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:19.136 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.129 7f906dffb700 1 -- 172.21.2.112:0/991434868 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f9074032b30 con 0x7f907c0ae760 2021-02-10T11:48:19.136 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.129 7f9081711700 1 --2- 172.21.2.112:0/991434868 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f905c0186c0 0x7f905c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:19.136 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.130 7f9081711700 1 --2- 172.21.2.112:0/991434868 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f905c0186c0 0x7f905c01abc0 crc :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:19.164 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.160 7f90838f5700 1 -- 172.21.2.112:0/991434868 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f907c082980 con 0x7f907c0ae760 2021-02-10T11:48:19.166 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.163 7f906dffb700 1 -- 172.21.2.112:0/991434868 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f907400be10 con 0x7f907c0ae760 2021-02-10T11:48:19.293 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.288 7f90838f5700 1 -- 172.21.2.112:0/991434868 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f907c109a60 con 0x7f907c0ae760 2021-02-10T11:48:19.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.289 7f906dffb700 1 -- 172.21.2.112:0/991434868 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v189) v1 ==== 104+0+10 (secure 0 0 0) 0x7f907400be10 con 0x7f907c0ae760 2021-02-10T11:48:19.294 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:48:19.296 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.291 7f90838f5700 1 -- 172.21.2.112:0/991434868 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f905c0186c0 msgr2=0x7f905c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:19.296 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.291 7f90838f5700 1 --2- 172.21.2.112:0/991434868 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f905c0186c0 0x7f905c01abc0 crc :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:19.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.291 7f90838f5700 1 -- 172.21.2.112:0/991434868 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f907c0ae760 msgr2=0x7f907c0af180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:19.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.291 7f90838f5700 1 --2- 172.21.2.112:0/991434868 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f907c0ae760 0x7f907c0af180 secure :-1 s=READY pgs=3162 cs=0 l=1 rev1=1 rx=0x7f9074009980 tx=0x7f907400bcd0).stop 2021-02-10T11:48:19.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.291 7f90838f5700 1 -- 172.21.2.112:0/991434868 shutdown_connections 2021-02-10T11:48:19.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.291 7f90838f5700 1 --2- 172.21.2.112:0/991434868 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f905c0186c0 0x7f905c01abc0 unknown :-1 s=CLOSED pgs=1423 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:19.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.291 7f90838f5700 1 --2- 172.21.2.112:0/991434868 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f907c0add40 0x7f907c0ae140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:19.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.291 7f90838f5700 1 --2- 172.21.2.112:0/991434868 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f907c0ae760 0x7f907c0af180 unknown :-1 s=CLOSED pgs=3162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:19.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.291 7f90838f5700 1 --2- 172.21.2.112:0/991434868 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f907c0a73c0 0x7f907c0a6b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:19.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.291 7f90838f5700 1 -- 172.21.2.112:0/991434868 >> 172.21.2.112:0/991434868 conn(0x7f907c0be980 msgr2=0x7f907c092cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:19.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.291 7f90838f5700 1 -- 172.21.2.112:0/991434868 shutdown_connections 2021-02-10T11:48:19.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.291 7f90838f5700 1 -- 172.21.2.112:0/991434868 wait complete. 2021-02-10T11:48:19.303 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:48:19.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.718 7fc90014f700 1 Processor -- start 2021-02-10T11:48:19.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.718 7fc90014f700 1 -- start start 2021-02-10T11:48:19.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.718 7fc90014f700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fc8f80d1870 0x7fc8f81b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:19.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.718 7fc90014f700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fc8f81b0ae0 0x7fc8f8108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:19.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.718 7fc90014f700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc8f80d1c70 0x7fc8f8108f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:19.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.718 7fc90014f700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8f80fe1c0 con 0x7fc8f80d1870 2021-02-10T11:48:19.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.718 7fc90014f700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8f8109a60 con 0x7fc8f81b0ae0 2021-02-10T11:48:19.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.718 7fc90014f700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f8088070 con 0x7fc8f80d1c70 2021-02-10T11:48:19.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.718 7fc8fdf6b700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fc8f80d1870 0x7fc8f81b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:19.738 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.719 7fc8fdf6b700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fc8f80d1870 0x7fc8f81b04c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:32950/0 (socket says 172.21.2.112:32950) 2021-02-10T11:48:19.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.719 7fc8fdf6b700 1 -- 172.21.2.112:0/4190498664 learned_addr learned my addr 172.21.2.112:0/4190498664 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:48:19.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.719 7fc8fdf6b700 1 -- 172.21.2.112:0/4190498664 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc8f80d1c70 msgr2=0x7fc8f8108f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:19.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.719 7fc8fdf6b700 1 --2- 172.21.2.112:0/4190498664 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc8f80d1c70 0x7fc8f8108f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:19.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.719 7fc8fdf6b700 1 -- 172.21.2.112:0/4190498664 >> v2:172.21.2.120:3300/0 conn(0x7fc8f81b0ae0 msgr2=0x7fc8f8108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:19.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.719 7fc8fdf6b700 1 --2- 172.21.2.112:0/4190498664 >> v2:172.21.2.120:3300/0 conn(0x7fc8f81b0ae0 0x7fc8f8108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:19.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.719 7fc8fdf6b700 1 -- 172.21.2.112:0/4190498664 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8f8109580 con 0x7fc8f80d1870 2021-02-10T11:48:19.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.719 7fc8fdf6b700 1 --2- 172.21.2.112:0/4190498664 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc8f80d1870 0x7fc8f81b04c0 secure :-1 s=READY pgs=3163 cs=0 l=1 rev1=1 rx=0x7fc8ec006700 tx=0x7fc8ec009b80).ready entity=mon.0 client_cookie=8baf59479b61bffe server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:19.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.719 7fc8fcf69700 1 -- 172.21.2.112:0/4190498664 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc8ec00c0c0 con 0x7fc8f80d1870 2021-02-10T11:48:19.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.719 7fc8fcf69700 1 -- 172.21.2.112:0/4190498664 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8ec00c440 con 0x7fc8f80d1870 2021-02-10T11:48:19.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.719 7fc8fcf69700 1 -- 172.21.2.112:0/4190498664 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc8ec00c960 con 0x7fc8f80d1870 2021-02-10T11:48:19.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.719 7fc90014f700 1 -- 172.21.2.112:0/4190498664 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc8f80d1870 msgr2=0x7fc8f81b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:19.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.719 7fc90014f700 1 --2- 172.21.2.112:0/4190498664 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc8f80d1870 0x7fc8f81b04c0 secure :-1 s=READY pgs=3163 cs=0 l=1 rev1=1 rx=0x7fc8ec006700 tx=0x7fc8ec009b80).stop 2021-02-10T11:48:19.739 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.720 7fc90014f700 1 -- 172.21.2.112:0/4190498664 shutdown_connections 2021-02-10T11:48:19.740 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.720 7fc90014f700 1 --2- 172.21.2.112:0/4190498664 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc8f80d1c70 0x7fc8f8108f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:19.740 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.720 7fc90014f700 1 --2- 172.21.2.112:0/4190498664 >> v2:172.21.2.120:3300/0 conn(0x7fc8f81b0ae0 0x7fc8f8108940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:19.740 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.720 7fc90014f700 1 --2- 172.21.2.112:0/4190498664 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc8f80d1870 0x7fc8f81b04c0 unknown :-1 s=CLOSED pgs=3163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:19.740 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.720 7fc90014f700 1 -- 172.21.2.112:0/4190498664 >> 172.21.2.112:0/4190498664 conn(0x7fc8f80ce420 msgr2=0x7fc8f80d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:19.740 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.720 7fc90014f700 1 -- 172.21.2.112:0/4190498664 shutdown_connections 2021-02-10T11:48:19.740 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.720 7fc90014f700 1 -- 172.21.2.112:0/4190498664 wait complete. 2021-02-10T11:48:19.741 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.735 7fc90014f700 1 Processor -- start 2021-02-10T11:48:19.741 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.735 7fc90014f700 1 -- start start 2021-02-10T11:48:19.741 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.735 7fc90014f700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc8f80c3820 0x7fc8f8084cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:19.742 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.735 7fc90014f700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc8f80a0290 0x7fc8f80852d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:19.742 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.735 7fc90014f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc8f8086270 0x7fc8f8086670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:19.742 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.735 7fc90014f700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f80c0700 con 0x7fc8f8086270 2021-02-10T11:48:19.742 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.735 7fc90014f700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f80858f0 con 0x7fc8f80a0290 2021-02-10T11:48:19.742 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.735 7fc90014f700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f8085b10 con 0x7fc8f80c3820 2021-02-10T11:48:19.742 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.735 7fc8fe76c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc8f8086270 0x7fc8f8086670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:19.742 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.735 7fc8fe76c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc8f8086270 0x7fc8f8086670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:32956/0 (socket says 172.21.2.112:32956) 2021-02-10T11:48:19.742 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.735 7fc8fe76c700 1 -- 172.21.2.112:0/171564911 learned_addr learned my addr 172.21.2.112:0/171564911 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:48:19.742 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.736 7fc8fe76c700 1 -- 172.21.2.112:0/171564911 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc8f80c3820 msgr2=0x7fc8f8084cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:19.742 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.736 7fc8fe76c700 1 --2- 172.21.2.112:0/171564911 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc8f80c3820 0x7fc8f8084cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:19.742 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.736 7fc8fe76c700 1 -- 172.21.2.112:0/171564911 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc8f80a0290 msgr2=0x7fc8f80852d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:19.742 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.736 7fc8fe76c700 1 --2- 172.21.2.112:0/171564911 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc8f80a0290 0x7fc8f80852d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:19.742 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.736 7fc8fe76c700 1 -- 172.21.2.112:0/171564911 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8f8086c90 con 0x7fc8f8086270 2021-02-10T11:48:19.742 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.736 7fc8fe76c700 1 --2- 172.21.2.112:0/171564911 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc8f8086270 0x7fc8f8086670 secure :-1 s=READY pgs=3164 cs=0 l=1 rev1=1 rx=0x7fc8f0009980 tx=0x7fc8f000bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:19.743 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.736 7fc8ea7fc700 1 -- 172.21.2.112:0/171564911 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc8f0003020 con 0x7fc8f8086270 2021-02-10T11:48:19.743 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.736 7fc8ea7fc700 1 -- 172.21.2.112:0/171564911 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8f0005a80 con 0x7fc8f8086270 2021-02-10T11:48:19.743 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.736 7fc8ea7fc700 1 -- 172.21.2.112:0/171564911 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc8f0006030 con 0x7fc8f8086270 2021-02-10T11:48:19.743 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.736 7fc90014f700 1 -- 172.21.2.112:0/171564911 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8f8086c90 con 0x7fc8f8086270 2021-02-10T11:48:19.743 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.736 7fc90014f700 1 -- 172.21.2.112:0/171564911 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8f8086c90 con 0x7fc8f8086270 2021-02-10T11:48:19.743 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.737 7fc8ea7fc700 1 -- 172.21.2.112:0/171564911 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc8f0005c60 con 0x7fc8f8086270 2021-02-10T11:48:19.743 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.737 7fc8ea7fc700 1 --2- 172.21.2.112:0/171564911 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc8e00186c0 0x7fc8e001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:19.743 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.737 7fc8ea7fc700 1 -- 172.21.2.112:0/171564911 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fc8f0032750 con 0x7fc8f8086270 2021-02-10T11:48:19.743 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.737 7fc8fdf6b700 1 --2- 172.21.2.112:0/171564911 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc8e00186c0 0x7fc8e001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:19.743 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.738 7fc8fdf6b700 1 --2- 172.21.2.112:0/171564911 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc8e00186c0 0x7fc8e001abc0 crc :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:19.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.768 7fc90014f700 1 -- 172.21.2.112:0/171564911 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8f8082a70 con 0x7fc8f8086270 2021-02-10T11:48:19.774 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.771 7fc8ea7fc700 1 -- 172.21.2.112:0/171564911 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc8f0005870 con 0x7fc8f8086270 2021-02-10T11:48:19.901 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.898 7fc90014f700 1 -- 172.21.2.112:0/171564911 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7fc8f8082a70 con 0x7fc8f8086270 2021-02-10T11:48:19.902 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.898 7fc8ea7fc700 1 -- 172.21.2.112:0/171564911 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v189) v1 ==== 100+0+10 (secure 0 0 0) 0x7fc8f0005c60 con 0x7fc8f8086270 2021-02-10T11:48:19.902 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:48:19.904 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.900 7fc90014f700 1 -- 172.21.2.112:0/171564911 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc8e00186c0 msgr2=0x7fc8e001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:19.904 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.900 7fc90014f700 1 --2- 172.21.2.112:0/171564911 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc8e00186c0 0x7fc8e001abc0 crc :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:19.904 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.900 7fc90014f700 1 -- 172.21.2.112:0/171564911 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc8f8086270 msgr2=0x7fc8f8086670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:19.904 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.900 7fc90014f700 1 --2- 172.21.2.112:0/171564911 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc8f8086270 0x7fc8f8086670 secure :-1 s=READY pgs=3164 cs=0 l=1 rev1=1 rx=0x7fc8f0009980 tx=0x7fc8f000bcd0).stop 2021-02-10T11:48:19.904 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.900 7fc90014f700 1 -- 172.21.2.112:0/171564911 shutdown_connections 2021-02-10T11:48:19.904 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.900 7fc90014f700 1 --2- 172.21.2.112:0/171564911 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc8e00186c0 0x7fc8e001abc0 unknown :-1 s=CLOSED pgs=1424 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:19.904 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.900 7fc90014f700 1 --2- 172.21.2.112:0/171564911 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc8f80a0290 0x7fc8f80852d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:19.904 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.900 7fc90014f700 1 --2- 172.21.2.112:0/171564911 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc8f8086270 0x7fc8f8086670 unknown :-1 s=CLOSED pgs=3164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:19.904 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.900 7fc90014f700 1 --2- 172.21.2.112:0/171564911 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc8f80c3820 0x7fc8f8084cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:19.904 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.900 7fc90014f700 1 -- 172.21.2.112:0/171564911 >> 172.21.2.112:0/171564911 conn(0x7fc8f80872c0 msgr2=0x7fc8f8095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:19.905 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.900 7fc90014f700 1 -- 172.21.2.112:0/171564911 shutdown_connections 2021-02-10T11:48:19.905 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:19.900 7fc90014f700 1 -- 172.21.2.112:0/171564911 wait complete. 2021-02-10T11:48:19.914 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:48:19.914 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:48:20.348 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:20.323 7f085faae700 1 Processor -- start 2021-02-10T11:48:20.348 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:20.323 7f085faae700 1 -- start start 2021-02-10T11:48:20.348 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:20.324 7f085faae700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f08580d1870 0x7f08581b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:20.348 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:20.324 7f085faae700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f08581b0a10 0x7f08580cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:20.348 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:20.324 7f085faae700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08580f0b10 con 0x7f08580d1c70 2021-02-10T11:48:20.348 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:20.324 7f085faae700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08580fe230 con 0x7f08581b0a10 2021-02-10T11:48:20.348 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:20.324 7f085faae700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08580e5ed0 con 0x7f08580d1870 2021-02-10T11:48:20.348 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:20.324 7f085d0c9700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f08581b0a10 0x7f08580cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:20.348 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:20.324 7f085d0c9700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f08581b0a10 0x7f08580cf330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:32960/0 (socket says 172.21.2.112:32960) 2021-02-10T11:48:20.349 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:20.324 7f085d0c9700 1 -- 172.21.2.112:0/2958642147 learned_addr learned my addr 172.21.2.112:0/2958642147 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:48:20.349 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:20.324 7f085d8ca700 1 --2- 172.21.2.112:0/2958642147 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f08580d1870 0x7f08581b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:20.349 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:20.324 7f085d0c9700 1 -- 172.21.2.112:0/2958642147 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f08580d1870 msgr2=0x7f08581b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:20.349 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:20.324 7f085d0c9700 1 --2- 172.21.2.112:0/2958642147 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f08580d1870 0x7f08581b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:20.349 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:20.324 7f085d0c9700 1 -- 172.21.2.112:0/2958642147 >> v1:172.21.2.120:6789/0 conn(0x7f08580d1c70 legacy=0x7f08580cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:20.350 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:20.324 7f085d0c9700 1 -- 172.21.2.112:0/2958642147 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08580f0650 con 0x7f08581b0a10 2021-02-10T11:48:20.350 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:20.325 7f085d0c9700 1 --2- 172.21.2.112:0/2958642147 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f08581b0a10 0x7f08580cf330 secure :-1 s=READY pgs=3165 cs=0 l=1 rev1=1 rx=0x7f0854006700 tx=0x7f0854009b80).ready entity=mon.0 client_cookie=9f67e1373c48ef99 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:20.350 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:20.325 7f085c8c8700 1 -- 172.21.2.112:0/2958642147 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f085400c0c0 con 0x7f08581b0a10 2021-02-10T11:48:20.350 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:20.325 7f085c8c8700 1 -- 172.21.2.112:0/2958642147 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f085400c440 con 0x7f08581b0a10 2021-02-10T11:48:20.350 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:20.325 7f085c8c8700 1 -- 172.21.2.112:0/2958642147 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f085400c960 con 0x7f08581b0a10 2021-02-10T11:48:20.350 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:20.325 7f085faae700 1 -- 172.21.2.112:0/2958642147 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f08581b0a10 msgr2=0x7f08580cf330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:20.350 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:20.325 7f085faae700 1 --2- 172.21.2.112:0/2958642147 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f08581b0a10 0x7f08580cf330 secure :-1 s=READY pgs=3165 cs=0 l=1 rev1=1 rx=0x7f0854006700 tx=0x7f0854009b80).stop 2021-02-10T11:48:20.350 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:20.326 7f085faae700 1 -- 172.21.2.112:0/2958642147 shutdown_connections 2021-02-10T11:48:20.350 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:20.326 7f085faae700 1 --2- 172.21.2.112:0/2958642147 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f08581b0a10 0x7f08580cf330 unknown :-1 s=CLOSED pgs=3165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:20.351 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:20.326 7f085faae700 1 --2- 172.21.2.112:0/2958642147 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f08580d1870 0x7f08581b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:20.351 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:20.326 7f085faae700 1 -- 172.21.2.112:0/2958642147 >> 172.21.2.112:0/2958642147 conn(0x7f08580ce450 msgr2=0x7f08580d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:20.351 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:20.326 7f085faae700 1 -- 172.21.2.112:0/2958642147 shutdown_connections 2021-02-10T11:48:20.351 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:20.326 7f085faae700 1 -- 172.21.2.112:0/2958642147 wait complete. 2021-02-10T11:48:20.353 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:20.341 7f085faae700 1 Processor -- start 2021-02-10T11:48:20.353 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:20.341 7f085faae700 1 -- start start 2021-02-10T11:48:20.353 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:20.341 7f085faae700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f08580ada90 0x7f08580ade90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:20.353 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:20.341 7f085faae700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0858099d20 0x7f085809a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:20.353 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:20.341 7f085faae700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f085809a740 0x7f0858097bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:20.353 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:20.341 7f085faae700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08580997a0 con 0x7f08580ada90 2021-02-10T11:48:20.353 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:20.341 7f085faae700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f085818f0c0 con 0x7f0858099d20 2021-02-10T11:48:20.354 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:20.341 7f085faae700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f085809ab40 con 0x7f085809a740 2021-02-10T11:48:20.354 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:20.341 7f085d8ca700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f08580ada90 0x7f08580ade90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:20.354 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:20.341 7f085d8ca700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f08580ada90 0x7f08580ade90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:32966/0 (socket says 172.21.2.112:32966) 2021-02-10T11:48:20.354 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:20.341 7f085d8ca700 1 -- 172.21.2.112:0/618388234 learned_addr learned my addr 172.21.2.112:0/618388234 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:48:20.354 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:20.342 7f085d8ca700 1 -- 172.21.2.112:0/618388234 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f085809a740 msgr2=0x7f0858097bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:20.354 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:20.342 7f085d8ca700 1 --2- 172.21.2.112:0/618388234 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f085809a740 0x7f0858097bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:20.354 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:20.342 7f085d8ca700 1 -- 172.21.2.112:0/618388234 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0858099d20 msgr2=0x7f085809a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:20.354 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:20.342 7f085d8ca700 1 --2- 172.21.2.112:0/618388234 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0858099d20 0x7f085809a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:20.354 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:20.342 7f085d8ca700 1 -- 172.21.2.112:0/618388234 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f085809ad60 con 0x7f08580ada90 2021-02-10T11:48:20.354 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:20.342 7f085d8ca700 1 --2- 172.21.2.112:0/618388234 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f08580ada90 0x7f08580ade90 secure :-1 s=READY pgs=3166 cs=0 l=1 rev1=1 rx=0x7f084c006f60 tx=0x7f084c00b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:20.355 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:20.342 7f084a7fc700 1 -- 172.21.2.112:0/618388234 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f084c00bd60 con 0x7f08580ada90 2021-02-10T11:48:20.355 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:20.342 7f084a7fc700 1 -- 172.21.2.112:0/618388234 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f084c000c90 con 0x7f08580ada90 2021-02-10T11:48:20.355 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:20.342 7f084a7fc700 1 -- 172.21.2.112:0/618388234 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f084c002220 con 0x7f08580ada90 2021-02-10T11:48:20.355 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:20.342 7f085faae700 1 -- 172.21.2.112:0/618388234 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f085809ad60 con 0x7f08580ada90 2021-02-10T11:48:20.355 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:20.343 7f085faae700 1 -- 172.21.2.112:0/618388234 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f085809ad60 con 0x7f08580ada90 2021-02-10T11:48:20.355 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:20.343 7f084a7fc700 1 -- 172.21.2.112:0/618388234 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f084c0052e0 con 0x7f08580ada90 2021-02-10T11:48:20.355 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:20.343 7f084a7fc700 1 --2- 172.21.2.112:0/618388234 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f08400186c0 0x7f084001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:20.355 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:20.343 7f084a7fc700 1 -- 172.21.2.112:0/618388234 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f084c032780 con 0x7f08580ada90 2021-02-10T11:48:20.355 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:20.344 7f085d0c9700 1 --2- 172.21.2.112:0/618388234 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f08400186c0 0x7f084001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:20.355 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:20.344 7f085d0c9700 1 --2- 172.21.2.112:0/618388234 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f08400186c0 0x7f084001abc0 crc :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:20.377 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:20.374 7f085faae700 1 -- 172.21.2.112:0/618388234 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08580ebc00 con 0x7f08580ada90 2021-02-10T11:48:20.379 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:20.376 7f084a7fc700 1 -- 172.21.2.112:0/618388234 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f084c000c90 con 0x7f08580ada90 2021-02-10T11:48:20.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:20.504 7f085faae700 1 -- 172.21.2.112:0/618388234 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f08580ebc00 con 0x7f08580ada90 2021-02-10T11:48:20.508 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:20.504 7f084a7fc700 1 -- 172.21.2.112:0/618388234 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 165 v165) v1 ==== 95+0+2107 (secure 0 0 0) 0x7f084c00bd60 con 0x7f08580ada90 2021-02-10T11:48:20.508 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:48:20.508 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":165,"default_fscid":20,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7763,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/1478301417","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":1478301417},{"type":"v1","addr":"172.21.2.112:6833","nonce":1478301417}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":161}],"filesystems":[{"mdsmap":{"epoch":165,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:48:10.481032","modified":"2021-02-10 11:48:12.490392","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7760":{"gid":7760,"name":"b","rank":0,"incarnation":162,"state":"up:active","state_seq":3,"addr":"172.21.2.120:6835/2943198956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":2943198956},{"type":"v1","addr":"172.21.2.120:6835","nonce":2943198956}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[41],"metadata_pool":40,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":20}]} 2021-02-10T11:48:20.509 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 165 2021-02-10T11:48:20.510 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:20.506 7f085faae700 1 -- 172.21.2.112:0/618388234 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f08400186c0 msgr2=0x7f084001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:20.510 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:20.506 7f085faae700 1 --2- 172.21.2.112:0/618388234 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f08400186c0 0x7f084001abc0 crc :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:20.510 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:20.506 7f085faae700 1 -- 172.21.2.112:0/618388234 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f08580ada90 msgr2=0x7f08580ade90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:20.510 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:20.506 7f085faae700 1 --2- 172.21.2.112:0/618388234 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f08580ada90 0x7f08580ade90 secure :-1 s=READY pgs=3166 cs=0 l=1 rev1=1 rx=0x7f084c006f60 tx=0x7f084c00b970).stop 2021-02-10T11:48:20.510 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:20.507 7f085faae700 1 -- 172.21.2.112:0/618388234 shutdown_connections 2021-02-10T11:48:20.510 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:20.507 7f085faae700 1 --2- 172.21.2.112:0/618388234 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f08400186c0 0x7f084001abc0 unknown :-1 s=CLOSED pgs=1425 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:20.510 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:20.507 7f085faae700 1 --2- 172.21.2.112:0/618388234 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0858099d20 0x7f085809a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:20.511 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:20.507 7f085faae700 1 --2- 172.21.2.112:0/618388234 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f085809a740 0x7f0858097bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:20.511 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:20.507 7f085faae700 1 --2- 172.21.2.112:0/618388234 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f08580ada90 0x7f08580ade90 unknown :-1 s=CLOSED pgs=3166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:20.511 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:20.507 7f085faae700 1 -- 172.21.2.112:0/618388234 >> 172.21.2.112:0/618388234 conn(0x7f08580f0550 msgr2=0x7f0858191730 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:20.511 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:20.507 7f085faae700 1 -- 172.21.2.112:0/618388234 shutdown_connections 2021-02-10T11:48:20.511 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:20.507 7f085faae700 1 -- 172.21.2.112:0/618388234 wait complete. 2021-02-10T11:48:20.720 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 165, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:48:10.481032', 'modified': '2021-02-10 11:48:12.490392', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7760}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7760': {'gid': 7760, 'name': 'b', 'rank': 0, 'incarnation': 162, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.120:6835/2943198956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.120:6834', 'nonce': 2943198956}, {'type': 'v1', 'addr': '172.21.2.120:6835', 'nonce': 2943198956}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [41], 'metadata_pool': 40, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:48:20.721 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:48:20.721 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:48:21.120 INFO:teuthology.orchestra.run.gibba020.stdout:{ 2021-02-10T11:48:21.120 INFO:teuthology.orchestra.run.gibba020.stdout: "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", 2021-02-10T11:48:21.120 INFO:teuthology.orchestra.run.gibba020.stdout: "whoami": 0, 2021-02-10T11:48:21.120 INFO:teuthology.orchestra.run.gibba020.stdout: "id": 7760, 2021-02-10T11:48:21.120 INFO:teuthology.orchestra.run.gibba020.stdout: "want_state": "up:active", 2021-02-10T11:48:21.120 INFO:teuthology.orchestra.run.gibba020.stdout: "state": "up:active", 2021-02-10T11:48:21.120 INFO:teuthology.orchestra.run.gibba020.stdout: "rank_uptime": 10.633324565000001, 2021-02-10T11:48:21.120 INFO:teuthology.orchestra.run.gibba020.stdout: "mdsmap_epoch": 165, 2021-02-10T11:48:21.120 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch": 189, 2021-02-10T11:48:21.120 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch_barrier": 189, 2021-02-10T11:48:21.121 INFO:teuthology.orchestra.run.gibba020.stdout: "uptime": 16.283644478999999 2021-02-10T11:48:21.121 INFO:teuthology.orchestra.run.gibba020.stdout:} 2021-02-10T11:48:21.124 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", "whoami": 0, "id": 7760, "want_state": "up:active", "state": "up:active", "rank_uptime": 10.633324565000001, "mdsmap_epoch": 165, "osdmap_epoch": 189, "osdmap_epoch_barrier": 189, "uptime": 16.283644478999999 } 2021-02-10T11:48:21.124 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:48:21.124 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2021-02-10T11:48:21.124 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@gibba012.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2021-02-10T11:48:21.124 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:48:21.124 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:48:21.460 DEBUG:teuthology.orchestra.run.gibba012:> sudo modprobe fuse 2021-02-10T11:48:21.805 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:48:21.806 DEBUG:teuthology.orchestra.run.gibba012:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:48:22.150 INFO:teuthology.orchestra.run.gibba012.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:48:22.150 INFO:teuthology.orchestra.run.gibba012.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:48:22.150 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:48:22.151 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:48:22.151 DEBUG:teuthology.orchestra.run.gibba012:> ls /sys/fs/fuse/connections 2021-02-10T11:48:22.484 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2021-02-10T11:48:22.485 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:48:22.488 DEBUG:teuthology.orchestra.run.gibba012:> sudo modprobe fuse 2021-02-10T11:48:22.832 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:48:22.832 DEBUG:teuthology.orchestra.run.gibba012:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:48:22.900 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba012.stderr:ceph-fuse[192408]: starting ceph client 2021-02-10T11:48:22.900 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba012.stderr:2021-02-10 11:48:22.896 7fd6ba85bf80 -1 init, newargv = 0x55ee48b5c1a0 newargc=9 2021-02-10T11:48:22.905 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba012.stderr:ceph-fuse[192408]: starting fuse 2021-02-10T11:48:23.173 INFO:teuthology.orchestra.run.gibba012.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:48:23.174 INFO:teuthology.orchestra.run.gibba012.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:48:23.174 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:48:23.175 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:48:23.175 DEBUG:teuthology.orchestra.run.gibba012:> ls /sys/fs/fuse/connections 2021-02-10T11:48:23.510 INFO:teuthology.orchestra.run.gibba012.stdout:42 2021-02-10T11:48:23.510 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2021-02-10T11:48:23.511 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:48:23.511 DEBUG:teuthology.orchestra.run.gibba012:> sudo python3 -c ' 2021-02-10T11:48:23.511 DEBUG:teuthology.orchestra.run.gibba012:> import glob 2021-02-10T11:48:23.511 DEBUG:teuthology.orchestra.run.gibba012:> import re 2021-02-10T11:48:23.511 DEBUG:teuthology.orchestra.run.gibba012:> import os 2021-02-10T11:48:23.511 DEBUG:teuthology.orchestra.run.gibba012:> import subprocess 2021-02-10T11:48:23.511 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:48:23.511 DEBUG:teuthology.orchestra.run.gibba012:> def find_socket(client_name): 2021-02-10T11:48:23.511 DEBUG:teuthology.orchestra.run.gibba012:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2021-02-10T11:48:23.511 DEBUG:teuthology.orchestra.run.gibba012:> files = glob.glob(asok_path) 2021-02-10T11:48:23.511 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:48:23.511 DEBUG:teuthology.orchestra.run.gibba012:> # Given a non-glob path, it better be there 2021-02-10T11:48:23.511 DEBUG:teuthology.orchestra.run.gibba012:> if "*" not in asok_path: 2021-02-10T11:48:23.512 DEBUG:teuthology.orchestra.run.gibba012:> assert(len(files) == 1) 2021-02-10T11:48:23.512 DEBUG:teuthology.orchestra.run.gibba012:> return files[0] 2021-02-10T11:48:23.512 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:48:23.512 DEBUG:teuthology.orchestra.run.gibba012:> for f in files: 2021-02-10T11:48:23.512 DEBUG:teuthology.orchestra.run.gibba012:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:48:23.512 DEBUG:teuthology.orchestra.run.gibba012:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:48:23.512 DEBUG:teuthology.orchestra.run.gibba012:> return f 2021-02-10T11:48:23.512 DEBUG:teuthology.orchestra.run.gibba012:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:48:23.512 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:48:23.512 DEBUG:teuthology.orchestra.run.gibba012:> print(find_socket("client.0")) 2021-02-10T11:48:23.512 DEBUG:teuthology.orchestra.run.gibba012:> ' 2021-02-10T11:48:23.873 INFO:teuthology.orchestra.run.gibba012.stdout:/var/run/ceph/ceph-client.0.192408.asok 2021-02-10T11:48:23.876 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.192408.asok 2021-02-10T11:48:23.876 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:48:23.876 DEBUG:teuthology.orchestra.run.gibba012:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.192408.asok status 2021-02-10T11:48:24.267 INFO:teuthology.orchestra.run.gibba012.stdout:{ 2021-02-10T11:48:24.268 INFO:teuthology.orchestra.run.gibba012.stdout: "metadata": { 2021-02-10T11:48:24.268 INFO:teuthology.orchestra.run.gibba012.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:48:24.268 INFO:teuthology.orchestra.run.gibba012.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:48:24.268 INFO:teuthology.orchestra.run.gibba012.stdout: "entity_id": "0", 2021-02-10T11:48:24.268 INFO:teuthology.orchestra.run.gibba012.stdout: "hostname": "gibba012", 2021-02-10T11:48:24.269 INFO:teuthology.orchestra.run.gibba012.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2021-02-10T11:48:24.269 INFO:teuthology.orchestra.run.gibba012.stdout: "pid": "192408", 2021-02-10T11:48:24.269 INFO:teuthology.orchestra.run.gibba012.stdout: "root": "/" 2021-02-10T11:48:24.269 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:48:24.269 INFO:teuthology.orchestra.run.gibba012.stdout: "dentry_count": 0, 2021-02-10T11:48:24.270 INFO:teuthology.orchestra.run.gibba012.stdout: "dentry_pinned_count": 0, 2021-02-10T11:48:24.270 INFO:teuthology.orchestra.run.gibba012.stdout: "id": 13599, 2021-02-10T11:48:24.270 INFO:teuthology.orchestra.run.gibba012.stdout: "inst": { 2021-02-10T11:48:24.270 INFO:teuthology.orchestra.run.gibba012.stdout: "name": { 2021-02-10T11:48:24.270 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "client", 2021-02-10T11:48:24.271 INFO:teuthology.orchestra.run.gibba012.stdout: "num": 13599 2021-02-10T11:48:24.271 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:48:24.271 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": { 2021-02-10T11:48:24.271 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:48:24.271 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:0", 2021-02-10T11:48:24.271 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 390410113 2021-02-10T11:48:24.272 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:48:24.272 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:48:24.272 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": { 2021-02-10T11:48:24.272 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:48:24.272 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:0", 2021-02-10T11:48:24.272 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 390410113 2021-02-10T11:48:24.273 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:48:24.273 INFO:teuthology.orchestra.run.gibba012.stdout: "inst_str": "client.13599 172.21.2.112:0/390410113", 2021-02-10T11:48:24.273 INFO:teuthology.orchestra.run.gibba012.stdout: "addr_str": "172.21.2.112:0/390410113", 2021-02-10T11:48:24.273 INFO:teuthology.orchestra.run.gibba012.stdout: "inode_count": 1, 2021-02-10T11:48:24.273 INFO:teuthology.orchestra.run.gibba012.stdout: "mds_epoch": 165, 2021-02-10T11:48:24.274 INFO:teuthology.orchestra.run.gibba012.stdout: "osd_epoch": 189, 2021-02-10T11:48:24.274 INFO:teuthology.orchestra.run.gibba012.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:48:24.274 INFO:teuthology.orchestra.run.gibba012.stdout: "blacklisted": false 2021-02-10T11:48:24.274 INFO:teuthology.orchestra.run.gibba012.stdout:} 2021-02-10T11:48:24.276 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:48:24.276 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:48:24.276 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:48:24.614 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:48:24.614 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:48:24.615 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:48:24.615 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:48:24.960 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:48:25.390 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.371 7f8bbd704700 1 Processor -- start 2021-02-10T11:48:25.390 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.371 7f8bbd704700 1 -- start start 2021-02-10T11:48:25.390 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.371 7f8bbd704700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f8bb80d1870 0x7f8bb81b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:25.391 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.371 7f8bbd704700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8bb81b0a10 0x7f8bb80cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:25.391 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.371 7f8bbd704700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f8bb80d1c70 0x7f8bb80cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:25.391 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.371 7f8bbd704700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8bb80fe200 con 0x7f8bb80d1870 2021-02-10T11:48:25.391 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.371 7f8bbd704700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8bb80e5f10 con 0x7f8bb80d1c70 2021-02-10T11:48:25.391 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.371 7f8bbd704700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bb80f0b50 con 0x7f8bb81b0a10 2021-02-10T11:48:25.391 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.371 7f8bb72bf700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f8bb80d1870 0x7f8bb81b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:25.391 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.372 7f8bb72bf700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f8bb80d1870 0x7f8bb81b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:32986/0 (socket says 172.21.2.112:32986) 2021-02-10T11:48:25.391 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.372 7f8bb72bf700 1 -- 172.21.2.112:0/548829588 learned_addr learned my addr 172.21.2.112:0/548829588 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:48:25.391 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.372 7f8bb72bf700 1 -- 172.21.2.112:0/548829588 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8bb81b0a10 msgr2=0x7f8bb80cf330 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:48:25.391 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.372 7f8bb72bf700 1 --2- 172.21.2.112:0/548829588 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8bb81b0a10 0x7f8bb80cf330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:25.391 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.372 7f8bb72bf700 1 -- 172.21.2.112:0/548829588 >> v2:172.21.2.120:3300/0 conn(0x7f8bb80d1c70 msgr2=0x7f8bb80cf950 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:48:25.391 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.372 7f8bb72bf700 1 --2- 172.21.2.112:0/548829588 >> v2:172.21.2.120:3300/0 conn(0x7f8bb80d1c70 0x7f8bb80cf950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:25.391 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.372 7f8bb72bf700 1 -- 172.21.2.112:0/548829588 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bb80cff70 con 0x7f8bb80d1870 2021-02-10T11:48:25.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.372 7f8bb72bf700 1 --2- 172.21.2.112:0/548829588 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8bb80d1870 0x7f8bb81b03f0 secure :-1 s=READY pgs=3168 cs=0 l=1 rev1=1 rx=0x7f8ba8006700 tx=0x7f8ba8009b00).ready entity=mon.0 client_cookie=ec5aecbbd97066ec server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:25.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.372 7f8bb62bd700 1 -- 172.21.2.112:0/548829588 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8ba800c0c0 con 0x7f8bb80d1870 2021-02-10T11:48:25.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.372 7f8bb62bd700 1 -- 172.21.2.112:0/548829588 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ba800c400 con 0x7f8bb80d1870 2021-02-10T11:48:25.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.372 7f8bb62bd700 1 -- 172.21.2.112:0/548829588 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8ba800c860 con 0x7f8bb80d1870 2021-02-10T11:48:25.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.372 7f8bbd704700 1 -- 172.21.2.112:0/548829588 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8bb80d1870 msgr2=0x7f8bb81b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:25.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.372 7f8bbd704700 1 --2- 172.21.2.112:0/548829588 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8bb80d1870 0x7f8bb81b03f0 secure :-1 s=READY pgs=3168 cs=0 l=1 rev1=1 rx=0x7f8ba8006700 tx=0x7f8ba8009b00).stop 2021-02-10T11:48:25.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.373 7f8bbd704700 1 -- 172.21.2.112:0/548829588 shutdown_connections 2021-02-10T11:48:25.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.373 7f8bbd704700 1 --2- 172.21.2.112:0/548829588 >> v2:172.21.2.120:3300/0 conn(0x7f8bb80d1c70 0x7f8bb80cf950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:25.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.373 7f8bbd704700 1 --2- 172.21.2.112:0/548829588 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8bb81b0a10 0x7f8bb80cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:25.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.373 7f8bbd704700 1 --2- 172.21.2.112:0/548829588 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8bb80d1870 0x7f8bb81b03f0 unknown :-1 s=CLOSED pgs=3168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:25.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.373 7f8bbd704700 1 -- 172.21.2.112:0/548829588 >> 172.21.2.112:0/548829588 conn(0x7f8bb80ce450 msgr2=0x7f8bb80d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:25.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.373 7f8bbd704700 1 -- 172.21.2.112:0/548829588 shutdown_connections 2021-02-10T11:48:25.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.373 7f8bbd704700 1 -- 172.21.2.112:0/548829588 wait complete. 2021-02-10T11:48:25.394 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.387 7f8bbd704700 1 Processor -- start 2021-02-10T11:48:25.394 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.387 7f8bbd704700 1 -- start start 2021-02-10T11:48:25.394 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.388 7f8bbd704700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8bb80aeea0 0x7f8bb81b81a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:25.394 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.388 7f8bbd704700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8bb8096630 0x7f8bb81b87c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:25.394 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.388 7f8bbd704700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8bb80953f0 0x7f8bb80957f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:25.394 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.388 7f8bbd704700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bb8189550 con 0x7f8bb80953f0 2021-02-10T11:48:25.394 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.388 7f8bbd704700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bb8095e10 con 0x7f8bb80aeea0 2021-02-10T11:48:25.394 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.388 7f8bbd704700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bb8096030 con 0x7f8bb8096630 2021-02-10T11:48:25.394 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.388 7f8bb7ac0700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8bb80953f0 0x7f8bb80957f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:25.394 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.388 7f8bb7ac0700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8bb80953f0 0x7f8bb80957f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:32994/0 (socket says 172.21.2.112:32994) 2021-02-10T11:48:25.395 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.388 7f8bb7ac0700 1 -- 172.21.2.112:0/2870815224 learned_addr learned my addr 172.21.2.112:0/2870815224 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:48:25.395 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.388 7f8bb7ac0700 1 -- 172.21.2.112:0/2870815224 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8bb8096630 msgr2=0x7f8bb81b87c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:25.395 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.388 7f8bb7ac0700 1 --2- 172.21.2.112:0/2870815224 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8bb8096630 0x7f8bb81b87c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:25.395 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.388 7f8bb7ac0700 1 -- 172.21.2.112:0/2870815224 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8bb80aeea0 msgr2=0x7f8bb81b81a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:25.395 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.388 7f8bb7ac0700 1 --2- 172.21.2.112:0/2870815224 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8bb80aeea0 0x7f8bb81b81a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:25.395 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.388 7f8bb7ac0700 1 -- 172.21.2.112:0/2870815224 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bb8096250 con 0x7f8bb80953f0 2021-02-10T11:48:25.395 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.389 7f8bb7ac0700 1 --2- 172.21.2.112:0/2870815224 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8bb80953f0 0x7f8bb80957f0 secure :-1 s=READY pgs=3169 cs=0 l=1 rev1=1 rx=0x7f8bb0009960 tx=0x7f8bb000bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:25.395 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.389 7f8baffff700 1 -- 172.21.2.112:0/2870815224 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8bb0003030 con 0x7f8bb80953f0 2021-02-10T11:48:25.395 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.389 7f8baffff700 1 -- 172.21.2.112:0/2870815224 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bb0001310 con 0x7f8bb80953f0 2021-02-10T11:48:25.395 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.389 7f8baffff700 1 -- 172.21.2.112:0/2870815224 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8bb0001770 con 0x7f8bb80953f0 2021-02-10T11:48:25.395 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.389 7f8bbd704700 1 -- 172.21.2.112:0/2870815224 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bb8096250 con 0x7f8bb80953f0 2021-02-10T11:48:25.395 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.389 7f8bbd704700 1 -- 172.21.2.112:0/2870815224 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bb8096250 con 0x7f8bb80953f0 2021-02-10T11:48:25.395 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.389 7f8baffff700 1 -- 172.21.2.112:0/2870815224 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8bb0000d10 con 0x7f8bb80953f0 2021-02-10T11:48:25.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.389 7f8baffff700 1 --2- 172.21.2.112:0/2870815224 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8b9c0186c0 0x7f8b9c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:25.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.389 7f8baffff700 1 -- 172.21.2.112:0/2870815224 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f8bb0032b10 con 0x7f8bb80953f0 2021-02-10T11:48:25.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.390 7f8bb72bf700 1 --2- 172.21.2.112:0/2870815224 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8b9c0186c0 0x7f8b9c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:25.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.390 7f8bb72bf700 1 --2- 172.21.2.112:0/2870815224 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8b9c0186c0 0x7f8b9c01abc0 crc :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:25.425 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.421 7f8bbd704700 1 -- 172.21.2.112:0/2870815224 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bb80ce450 con 0x7f8bb80953f0 2021-02-10T11:48:25.427 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.424 7f8baffff700 1 -- 172.21.2.112:0/2870815224 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8bb0001310 con 0x7f8bb80953f0 2021-02-10T11:48:25.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.549 7f8bbd704700 1 -- 172.21.2.112:0/2870815224 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8bb80ce450 con 0x7f8bb80953f0 2021-02-10T11:48:25.552 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.549 7f8baffff700 1 -- 172.21.2.112:0/2870815224 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v189) v1 ==== 74+0+14007 (secure 0 0 0) 0x7f8bb0000d10 con 0x7f8bb80953f0 2021-02-10T11:48:25.553 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:48:25.553 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":189,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:48:11.483156","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":41,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":40,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:48:08.454038","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"188","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":41,"pool_name":"cephfs_data","create_time":"2021-02-10 11:48:09.474627","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"189","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:48:25.554 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.551 7f8bbd704700 1 -- 172.21.2.112:0/2870815224 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8b9c0186c0 msgr2=0x7f8b9c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:25.555 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.551 7f8bbd704700 1 --2- 172.21.2.112:0/2870815224 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8b9c0186c0 0x7f8b9c01abc0 crc :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:25.555 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.551 7f8bbd704700 1 -- 172.21.2.112:0/2870815224 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8bb80953f0 msgr2=0x7f8bb80957f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:25.555 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.551 7f8bbd704700 1 --2- 172.21.2.112:0/2870815224 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8bb80953f0 0x7f8bb80957f0 secure :-1 s=READY pgs=3169 cs=0 l=1 rev1=1 rx=0x7f8bb0009960 tx=0x7f8bb000bcf0).stop 2021-02-10T11:48:25.555 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.551 7f8bbd704700 1 -- 172.21.2.112:0/2870815224 shutdown_connections 2021-02-10T11:48:25.555 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.551 7f8bbd704700 1 --2- 172.21.2.112:0/2870815224 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8b9c0186c0 0x7f8b9c01abc0 unknown :-1 s=CLOSED pgs=1426 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:25.555 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.551 7f8bbd704700 1 --2- 172.21.2.112:0/2870815224 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8bb80953f0 0x7f8bb80957f0 unknown :-1 s=CLOSED pgs=3169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:25.555 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.551 7f8bbd704700 1 --2- 172.21.2.112:0/2870815224 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8bb8096630 0x7f8bb81b87c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:25.555 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.551 7f8bbd704700 1 --2- 172.21.2.112:0/2870815224 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8bb80aeea0 0x7f8bb81b81a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:25.555 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.551 7f8bbd704700 1 -- 172.21.2.112:0/2870815224 >> 172.21.2.112:0/2870815224 conn(0x7f8bb80efc40 msgr2=0x7f8bb81b9080 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:25.555 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.551 7f8bbd704700 1 -- 172.21.2.112:0/2870815224 shutdown_connections 2021-02-10T11:48:25.555 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.551 7f8bbd704700 1 -- 172.21.2.112:0/2870815224 wait complete. 2021-02-10T11:48:25.563 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:13:57.508758', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 40, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:48:08.454038', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '188', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 41, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:48:09.474627', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '189', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:48:25.564 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:48:25.996 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.977 7fed60ea2700 1 Processor -- start 2021-02-10T11:48:25.997 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.977 7fed60ea2700 1 -- start start 2021-02-10T11:48:25.997 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.977 7fed60ea2700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fed5c0d1860 0x7fed5c1b04b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:25.997 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.977 7fed60ea2700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fed5c0cff60 0x7fed5c109120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:25.997 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.977 7fed60ea2700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed5c109a50 con 0x7fed5c1b0ad0 2021-02-10T11:48:25.997 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.977 7fed60ea2700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed5c109740 con 0x7fed5c0cff60 2021-02-10T11:48:25.997 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.977 7fed60ea2700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed5c088070 con 0x7fed5c0d1860 2021-02-10T11:48:25.997 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.978 7fed5b314700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fed5c0cff60 0x7fed5c109120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:25.997 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.978 7fed5a312700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fed5c1b0ad0 0x7fed5c108930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:60694/0 (socket says 172.21.2.112:60694) 2021-02-10T11:48:25.997 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.978 7fed5a312700 1 -- 172.21.2.112:0/309698057 learned_addr learned my addr 172.21.2.112:0/309698057 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:48:25.997 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.978 7fed5b314700 1 -- 172.21.2.112:0/309698057 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fed5c0d1860 msgr2=0x7fed5c1b04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:25.997 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.978 7fed5b314700 1 --2- 172.21.2.112:0/309698057 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fed5c0d1860 0x7fed5c1b04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:25.997 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.978 7fed5b314700 1 -- 172.21.2.112:0/309698057 >> v1:172.21.2.112:6789/0 conn(0x7fed5c1b0ad0 legacy=0x7fed5c108930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:25.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.978 7fed5b314700 1 -- 172.21.2.112:0/309698057 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed5c088260 con 0x7fed5c0cff60 2021-02-10T11:48:25.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.978 7fed5b314700 1 --2- 172.21.2.112:0/309698057 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fed5c0cff60 0x7fed5c109120 secure :-1 s=READY pgs=3171 cs=0 l=1 rev1=1 rx=0x7fed54006e30 tx=0x7fed54009e60).ready entity=mon.0 client_cookie=d166c1479c9edc73 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:25.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.978 7fed59b11700 1 -- 172.21.2.112:0/309698057 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fed54006a50 con 0x7fed5c0cff60 2021-02-10T11:48:25.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.978 7fed59b11700 1 -- 172.21.2.112:0/309698057 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed5400b300 con 0x7fed5c0cff60 2021-02-10T11:48:25.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.978 7fed59b11700 1 -- 172.21.2.112:0/309698057 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fed5400b820 con 0x7fed5c0cff60 2021-02-10T11:48:25.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.978 7fed60ea2700 1 -- 172.21.2.112:0/309698057 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fed5c0cff60 msgr2=0x7fed5c109120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:25.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.978 7fed60ea2700 1 --2- 172.21.2.112:0/309698057 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fed5c0cff60 0x7fed5c109120 secure :-1 s=READY pgs=3171 cs=0 l=1 rev1=1 rx=0x7fed54006e30 tx=0x7fed54009e60).stop 2021-02-10T11:48:25.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.979 7fed60ea2700 1 -- 172.21.2.112:0/309698057 shutdown_connections 2021-02-10T11:48:25.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.979 7fed60ea2700 1 --2- 172.21.2.112:0/309698057 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fed5c0cff60 0x7fed5c109120 unknown :-1 s=CLOSED pgs=3171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:25.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.979 7fed60ea2700 1 --2- 172.21.2.112:0/309698057 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fed5c0d1860 0x7fed5c1b04b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:25.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.979 7fed60ea2700 1 -- 172.21.2.112:0/309698057 >> 172.21.2.112:0/309698057 conn(0x7fed5c0ce410 msgr2=0x7fed5c0d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:25.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.979 7fed60ea2700 1 -- 172.21.2.112:0/309698057 shutdown_connections 2021-02-10T11:48:25.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.979 7fed60ea2700 1 -- 172.21.2.112:0/309698057 wait complete. 2021-02-10T11:48:26.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.994 7fed60ea2700 1 Processor -- start 2021-02-10T11:48:26.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.994 7fed60ea2700 1 -- start start 2021-02-10T11:48:26.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.994 7fed60ea2700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fed5c0bfa10 0x7fed5c0b82a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:26.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.994 7fed60ea2700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fed5c1a4a70 0x7fed5c094c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:26.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.994 7fed60ea2700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fed5c0c0410 0x7fed5c0c0810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:26.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.994 7fed60ea2700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed5c0c3a50 con 0x7fed5c0bfa10 2021-02-10T11:48:26.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.994 7fed60ea2700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed5c0b88c0 con 0x7fed5c0c0410 2021-02-10T11:48:26.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.994 7fed60ea2700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed5c095260 con 0x7fed5c1a4a70 2021-02-10T11:48:26.001 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.994 7fed5ab13700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fed5c0bfa10 0x7fed5c0b82a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:26.001 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.994 7fed5ab13700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fed5c0bfa10 0x7fed5c0b82a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:33008/0 (socket says 172.21.2.112:33008) 2021-02-10T11:48:26.001 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.994 7fed5ab13700 1 -- 172.21.2.112:0/1211580767 learned_addr learned my addr 172.21.2.112:0/1211580767 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:48:26.001 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.994 7fed5a312700 1 --2- 172.21.2.112:0/1211580767 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fed5c1a4a70 0x7fed5c094c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:26.001 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.994 7fed5b314700 1 --2- 172.21.2.112:0/1211580767 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fed5c0c0410 0x7fed5c0c0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:26.001 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.995 7fed5ab13700 1 -- 172.21.2.112:0/1211580767 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fed5c1a4a70 msgr2=0x7fed5c094c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:26.001 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.995 7fed5ab13700 1 --2- 172.21.2.112:0/1211580767 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fed5c1a4a70 0x7fed5c094c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:26.001 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.995 7fed5ab13700 1 -- 172.21.2.112:0/1211580767 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fed5c0c0410 msgr2=0x7fed5c0c0810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:26.001 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.995 7fed5ab13700 1 --2- 172.21.2.112:0/1211580767 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fed5c0c0410 0x7fed5c0c0810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:26.001 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.995 7fed5ab13700 1 -- 172.21.2.112:0/1211580767 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed5c0c3540 con 0x7fed5c0bfa10 2021-02-10T11:48:26.001 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.995 7fed5ab13700 1 --2- 172.21.2.112:0/1211580767 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fed5c0bfa10 0x7fed5c0b82a0 secure :-1 s=READY pgs=3172 cs=0 l=1 rev1=1 rx=0x7fed4400b530 tx=0x7fed4400b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:26.001 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.995 7fed537fe700 1 -- 172.21.2.112:0/1211580767 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fed4400bcc0 con 0x7fed5c0bfa10 2021-02-10T11:48:26.001 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.995 7fed537fe700 1 -- 172.21.2.112:0/1211580767 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed44000c90 con 0x7fed5c0bfa10 2021-02-10T11:48:26.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.995 7fed60ea2700 1 -- 172.21.2.112:0/1211580767 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed5c0c3540 con 0x7fed5c0bfa10 2021-02-10T11:48:26.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.995 7fed60ea2700 1 -- 172.21.2.112:0/1211580767 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed5c0c3540 con 0x7fed5c0bfa10 2021-02-10T11:48:26.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.995 7fed537fe700 1 -- 172.21.2.112:0/1211580767 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fed440022a0 con 0x7fed5c0bfa10 2021-02-10T11:48:26.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.996 7fed537fe700 1 -- 172.21.2.112:0/1211580767 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fed44005310 con 0x7fed5c0bfa10 2021-02-10T11:48:26.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.996 7fed537fe700 1 --2- 172.21.2.112:0/1211580767 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fed40018730 0x7fed4001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:26.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.996 7fed537fe700 1 -- 172.21.2.112:0/1211580767 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fed44032770 con 0x7fed5c0bfa10 2021-02-10T11:48:26.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.996 7fed5a312700 1 --2- 172.21.2.112:0/1211580767 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fed40018730 0x7fed4001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:26.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:25.997 7fed5a312700 1 --2- 172.21.2.112:0/1211580767 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fed40018730 0x7fed4001ac10 crc :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:26.031 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:26.027 7fed60ea2700 1 -- 172.21.2.112:0/1211580767 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed5c0bfee0 con 0x7fed5c0bfa10 2021-02-10T11:48:26.033 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:26.030 7fed537fe700 1 -- 172.21.2.112:0/1211580767 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fed44006b90 con 0x7fed5c0bfa10 2021-02-10T11:48:26.161 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:26.158 7fed60ea2700 1 -- 172.21.2.112:0/1211580767 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7fed5c0bfee0 con 0x7fed5c0bfa10 2021-02-10T11:48:26.162 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:26.158 7fed537fe700 1 -- 172.21.2.112:0/1211580767 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v189) v1 ==== 92+0+10 (secure 0 0 0) 0x7fed44000c90 con 0x7fed5c0bfa10 2021-02-10T11:48:26.162 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:48:26.164 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:26.160 7fed60ea2700 1 -- 172.21.2.112:0/1211580767 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fed40018730 msgr2=0x7fed4001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:26.164 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:26.160 7fed60ea2700 1 --2- 172.21.2.112:0/1211580767 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fed40018730 0x7fed4001ac10 crc :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:26.164 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:26.160 7fed60ea2700 1 -- 172.21.2.112:0/1211580767 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fed5c0bfa10 msgr2=0x7fed5c0b82a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:26.164 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:26.160 7fed60ea2700 1 --2- 172.21.2.112:0/1211580767 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fed5c0bfa10 0x7fed5c0b82a0 secure :-1 s=READY pgs=3172 cs=0 l=1 rev1=1 rx=0x7fed4400b530 tx=0x7fed4400b8d0).stop 2021-02-10T11:48:26.164 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:26.161 7fed60ea2700 1 -- 172.21.2.112:0/1211580767 shutdown_connections 2021-02-10T11:48:26.164 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:26.161 7fed60ea2700 1 --2- 172.21.2.112:0/1211580767 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fed40018730 0x7fed4001ac10 unknown :-1 s=CLOSED pgs=1427 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:26.164 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:26.161 7fed60ea2700 1 --2- 172.21.2.112:0/1211580767 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fed5c0c0410 0x7fed5c0c0810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:26.165 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:26.161 7fed60ea2700 1 --2- 172.21.2.112:0/1211580767 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fed5c1a4a70 0x7fed5c094c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:26.165 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:26.161 7fed60ea2700 1 --2- 172.21.2.112:0/1211580767 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fed5c0bfa10 0x7fed5c0b82a0 unknown :-1 s=CLOSED pgs=3172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:26.165 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:26.161 7fed60ea2700 1 -- 172.21.2.112:0/1211580767 >> 172.21.2.112:0/1211580767 conn(0x7fed5c0a6a80 msgr2=0x7fed5c0ae110 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:26.165 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:26.161 7fed60ea2700 1 -- 172.21.2.112:0/1211580767 shutdown_connections 2021-02-10T11:48:26.165 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:26.161 7fed60ea2700 1 -- 172.21.2.112:0/1211580767 wait complete. 2021-02-10T11:48:26.172 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:48:26.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:26.584 7f8e1b3d7700 1 Processor -- start 2021-02-10T11:48:26.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:26.585 7f8e1b3d7700 1 -- start start 2021-02-10T11:48:26.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:26.585 7f8e1b3d7700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f8e140d1c70 0x7f8e140cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:26.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:26.585 7f8e1b3d7700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f8e140cf580 0x7f8e140cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:26.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:26.585 7f8e1b3d7700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e140fe1c0 con 0x7f8e140d1870 2021-02-10T11:48:26.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:26.585 7f8e1b3d7700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e141b0cb0 con 0x7f8e140cf580 2021-02-10T11:48:26.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:26.585 7f8e1b3d7700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e14109a60 con 0x7f8e140d1c70 2021-02-10T11:48:26.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:26.585 7f8e199f4700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f8e140cf580 0x7f8e140cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:26.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:26.585 7f8e199f4700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f8e140cf580 0x7f8e140cf980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:33016/0 (socket says 172.21.2.112:33016) 2021-02-10T11:48:26.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:26.585 7f8e199f4700 1 -- 172.21.2.112:0/279424901 learned_addr learned my addr 172.21.2.112:0/279424901 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:48:26.606 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:26.585 7f8e189f2700 1 --2- 172.21.2.112:0/279424901 >> v2:172.21.2.120:3300/0 conn(0x7f8e140d1c70 0x7f8e140cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:26.606 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:26.586 7f8e199f4700 1 -- 172.21.2.112:0/279424901 >> v2:172.21.2.120:3300/0 conn(0x7f8e140d1c70 msgr2=0x7f8e140cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:26.606 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:26.586 7f8e199f4700 1 --2- 172.21.2.112:0/279424901 >> v2:172.21.2.120:3300/0 conn(0x7f8e140d1c70 0x7f8e140cef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:26.606 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:26.586 7f8e199f4700 1 -- 172.21.2.112:0/279424901 >> v1:172.21.2.120:6789/0 conn(0x7f8e140d1870 legacy=0x7f8e141b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:26.606 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:26.586 7f8e199f4700 1 -- 172.21.2.112:0/279424901 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e14088070 con 0x7f8e140cf580 2021-02-10T11:48:26.606 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:26.586 7f8e199f4700 1 --2- 172.21.2.112:0/279424901 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8e140cf580 0x7f8e140cf980 secure :-1 s=READY pgs=3173 cs=0 l=1 rev1=1 rx=0x7f8e0c006f00 tx=0x7f8e0c009f10).ready entity=mon.0 client_cookie=fce7711389c6fc7 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:26.606 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:26.586 7f8e07fff700 1 -- 172.21.2.112:0/279424901 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8e0c00b3b0 con 0x7f8e140cf580 2021-02-10T11:48:26.606 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:26.586 7f8e07fff700 1 -- 172.21.2.112:0/279424901 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e0c00b630 con 0x7f8e140cf580 2021-02-10T11:48:26.606 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:26.586 7f8e07fff700 1 -- 172.21.2.112:0/279424901 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8e0c00ba90 con 0x7f8e140cf580 2021-02-10T11:48:26.606 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:26.586 7f8e1b3d7700 1 -- 172.21.2.112:0/279424901 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8e140cf580 msgr2=0x7f8e140cf980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:26.606 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:26.586 7f8e1b3d7700 1 --2- 172.21.2.112:0/279424901 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8e140cf580 0x7f8e140cf980 secure :-1 s=READY pgs=3173 cs=0 l=1 rev1=1 rx=0x7f8e0c006f00 tx=0x7f8e0c009f10).stop 2021-02-10T11:48:26.606 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:26.587 7f8e1b3d7700 1 -- 172.21.2.112:0/279424901 shutdown_connections 2021-02-10T11:48:26.607 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:26.587 7f8e1b3d7700 1 --2- 172.21.2.112:0/279424901 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8e140cf580 0x7f8e140cf980 unknown :-1 s=CLOSED pgs=3173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:26.607 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:26.587 7f8e1b3d7700 1 --2- 172.21.2.112:0/279424901 >> v2:172.21.2.120:3300/0 conn(0x7f8e140d1c70 0x7f8e140cef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:26.607 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:26.587 7f8e1b3d7700 1 -- 172.21.2.112:0/279424901 >> 172.21.2.112:0/279424901 conn(0x7f8e140ce420 msgr2=0x7f8e140d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:26.607 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:26.587 7f8e1b3d7700 1 -- 172.21.2.112:0/279424901 shutdown_connections 2021-02-10T11:48:26.607 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:26.587 7f8e1b3d7700 1 -- 172.21.2.112:0/279424901 wait complete. 2021-02-10T11:48:26.608 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:26.601 7f8e1b3d7700 1 Processor -- start 2021-02-10T11:48:26.609 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:26.601 7f8e1b3d7700 1 -- start start 2021-02-10T11:48:26.609 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:26.602 7f8e1b3d7700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8e140b04d0 0x7f8e141a7a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:26.609 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:26.602 7f8e1b3d7700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8e141a8090 0x7f8e140c23a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:26.609 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:26.602 7f8e1b3d7700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8e1408d680 0x7f8e140c29c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:26.609 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:26.602 7f8e1b3d7700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e140b0a00 con 0x7f8e1408d680 2021-02-10T11:48:26.609 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:26.602 7f8e1b3d7700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e140c2fe0 con 0x7f8e141a8090 2021-02-10T11:48:26.609 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:26.602 7f8e1b3d7700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e140c3200 con 0x7f8e140b04d0 2021-02-10T11:48:26.609 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:26.602 7f8e199f4700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8e1408d680 0x7f8e140c29c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:26.609 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:26.602 7f8e199f4700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8e1408d680 0x7f8e140c29c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:33022/0 (socket says 172.21.2.112:33022) 2021-02-10T11:48:26.609 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:26.602 7f8e199f4700 1 -- 172.21.2.112:0/3230683864 learned_addr learned my addr 172.21.2.112:0/3230683864 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:48:26.609 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:26.602 7f8e189f2700 1 --2- 172.21.2.112:0/3230683864 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8e141a8090 0x7f8e140c23a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:26.609 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:26.602 7f8e191f3700 1 --2- 172.21.2.112:0/3230683864 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8e140b04d0 0x7f8e141a7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:26.609 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:26.603 7f8e199f4700 1 -- 172.21.2.112:0/3230683864 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8e140b04d0 msgr2=0x7f8e141a7a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:26.610 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:26.603 7f8e199f4700 1 --2- 172.21.2.112:0/3230683864 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8e140b04d0 0x7f8e141a7a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:26.610 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:26.603 7f8e199f4700 1 -- 172.21.2.112:0/3230683864 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8e141a8090 msgr2=0x7f8e140c23a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:26.610 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:26.603 7f8e199f4700 1 --2- 172.21.2.112:0/3230683864 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8e141a8090 0x7f8e140c23a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:26.610 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:26.603 7f8e199f4700 1 -- 172.21.2.112:0/3230683864 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e1408e080 con 0x7f8e1408d680 2021-02-10T11:48:26.610 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:26.603 7f8e189f2700 1 --2- 172.21.2.112:0/3230683864 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8e141a8090 0x7f8e140c23a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:48:26.610 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:26.603 7f8e191f3700 1 --2- 172.21.2.112:0/3230683864 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8e140b04d0 0x7f8e141a7a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:48:26.610 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:26.603 7f8e199f4700 1 --2- 172.21.2.112:0/3230683864 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8e1408d680 0x7f8e140c29c0 secure :-1 s=READY pgs=3174 cs=0 l=1 rev1=1 rx=0x7f8e0c009390 tx=0x7f8e0c00bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:26.610 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:26.603 7f8e05ffb700 1 -- 172.21.2.112:0/3230683864 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8e0c003030 con 0x7f8e1408d680 2021-02-10T11:48:26.610 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:26.603 7f8e05ffb700 1 -- 172.21.2.112:0/3230683864 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e0c006c80 con 0x7f8e1408d680 2021-02-10T11:48:26.610 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:26.603 7f8e1b3d7700 1 -- 172.21.2.112:0/3230683864 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e1408e080 con 0x7f8e1408d680 2021-02-10T11:48:26.610 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:26.603 7f8e1b3d7700 1 -- 172.21.2.112:0/3230683864 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e1408e610 con 0x7f8e1408d680 2021-02-10T11:48:26.610 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:26.603 7f8e05ffb700 1 -- 172.21.2.112:0/3230683864 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8e0c001480 con 0x7f8e1408d680 2021-02-10T11:48:26.611 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:26.604 7f8e05ffb700 1 -- 172.21.2.112:0/3230683864 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8e0c001270 con 0x7f8e1408d680 2021-02-10T11:48:26.611 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:26.604 7f8e05ffb700 1 --2- 172.21.2.112:0/3230683864 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8df40186c0 0x7f8df401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:26.611 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:26.604 7f8e05ffb700 1 -- 172.21.2.112:0/3230683864 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f8e0c032b60 con 0x7f8e1408d680 2021-02-10T11:48:26.611 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:26.604 7f8e191f3700 1 --2- 172.21.2.112:0/3230683864 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8df40186c0 0x7f8df401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:26.611 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:26.605 7f8e191f3700 1 --2- 172.21.2.112:0/3230683864 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8df40186c0 0x7f8df401abc0 crc :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:26.638 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:26.635 7f8e1b3d7700 1 -- 172.21.2.112:0/3230683864 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e14082a70 con 0x7f8e1408d680 2021-02-10T11:48:26.641 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:26.637 7f8e05ffb700 1 -- 172.21.2.112:0/3230683864 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8e0c006c80 con 0x7f8e1408d680 2021-02-10T11:48:26.769 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:26.766 7f8e1b3d7700 1 -- 172.21.2.112:0/3230683864 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f8e14082a70 con 0x7f8e1408d680 2021-02-10T11:48:26.770 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:26.767 7f8e05ffb700 1 -- 172.21.2.112:0/3230683864 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v189) v1 ==== 104+0+10 (secure 0 0 0) 0x7f8e0c001270 con 0x7f8e1408d680 2021-02-10T11:48:26.770 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:48:26.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:26.768 7f8e1b3d7700 1 -- 172.21.2.112:0/3230683864 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8df40186c0 msgr2=0x7f8df401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:26.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:26.768 7f8e1b3d7700 1 --2- 172.21.2.112:0/3230683864 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8df40186c0 0x7f8df401abc0 crc :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:26.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:26.768 7f8e1b3d7700 1 -- 172.21.2.112:0/3230683864 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8e1408d680 msgr2=0x7f8e140c29c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:26.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:26.768 7f8e1b3d7700 1 --2- 172.21.2.112:0/3230683864 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8e1408d680 0x7f8e140c29c0 secure :-1 s=READY pgs=3174 cs=0 l=1 rev1=1 rx=0x7f8e0c009390 tx=0x7f8e0c00bd40).stop 2021-02-10T11:48:26.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:26.769 7f8e1b3d7700 1 -- 172.21.2.112:0/3230683864 shutdown_connections 2021-02-10T11:48:26.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:26.769 7f8e1b3d7700 1 --2- 172.21.2.112:0/3230683864 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f8df40186c0 0x7f8df401abc0 unknown :-1 s=CLOSED pgs=1428 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:26.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:26.769 7f8e1b3d7700 1 --2- 172.21.2.112:0/3230683864 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f8e141a8090 0x7f8e140c23a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:26.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:26.769 7f8e1b3d7700 1 --2- 172.21.2.112:0/3230683864 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f8e1408d680 0x7f8e140c29c0 unknown :-1 s=CLOSED pgs=3174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:26.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:26.769 7f8e1b3d7700 1 --2- 172.21.2.112:0/3230683864 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f8e140b04d0 0x7f8e141a7a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:26.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:26.769 7f8e1b3d7700 1 -- 172.21.2.112:0/3230683864 >> 172.21.2.112:0/3230683864 conn(0x7f8e140be530 msgr2=0x7f8e1409f1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:26.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:26.769 7f8e1b3d7700 1 -- 172.21.2.112:0/3230683864 shutdown_connections 2021-02-10T11:48:26.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:26.769 7f8e1b3d7700 1 -- 172.21.2.112:0/3230683864 wait complete. 2021-02-10T11:48:26.780 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:48:27.211 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.192 7f6184a50700 1 Processor -- start 2021-02-10T11:48:27.211 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.192 7f6184a50700 1 -- start start 2021-02-10T11:48:27.211 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.193 7f6184a50700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f61800d1c70 0x7f61800cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:27.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.193 7f6184a50700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f61800cf580 0x7f61800cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:27.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.193 7f6184a50700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61800fe1c0 con 0x7f61800d1870 2021-02-10T11:48:27.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.193 7f6184a50700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61801b0cb0 con 0x7f61800d1c70 2021-02-10T11:48:27.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.193 7f6184a50700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6180109a60 con 0x7f61800cf580 2021-02-10T11:48:27.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.193 7f617e05d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f61800d1c70 0x7f61800cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:27.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.193 7f617e05d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f61800d1c70 0x7f61800cef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:33026/0 (socket says 172.21.2.112:33026) 2021-02-10T11:48:27.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.193 7f617e05d700 1 -- 172.21.2.112:0/2423253692 learned_addr learned my addr 172.21.2.112:0/2423253692 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:48:27.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.193 7f617e05d700 1 -- 172.21.2.112:0/2423253692 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f61800cf580 msgr2=0x7f61800cf980 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:48:27.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.193 7f617e05d700 1 --2- 172.21.2.112:0/2423253692 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f61800cf580 0x7f61800cf980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:27.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.193 7f617e05d700 1 -- 172.21.2.112:0/2423253692 >> v1:172.21.2.120:6789/0 conn(0x7f61800d1870 legacy=0x7f61801b04c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:48:27.212 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.193 7f617e05d700 1 -- 172.21.2.112:0/2423253692 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6180088070 con 0x7f61800d1c70 2021-02-10T11:48:27.213 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.193 7f617e05d700 1 --2- 172.21.2.112:0/2423253692 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f61800d1c70 0x7f61800cef60 secure :-1 s=READY pgs=3175 cs=0 l=1 rev1=1 rx=0x7f6168006700 tx=0x7f6168009b80).ready entity=mon.0 client_cookie=475977f568409d40 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:27.213 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.193 7f617d85c700 1 -- 172.21.2.112:0/2423253692 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f616800c0c0 con 0x7f61800d1c70 2021-02-10T11:48:27.213 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.193 7f617d85c700 1 -- 172.21.2.112:0/2423253692 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f616800c400 con 0x7f61800d1c70 2021-02-10T11:48:27.213 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.193 7f617d85c700 1 -- 172.21.2.112:0/2423253692 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f616800c8e0 con 0x7f61800d1c70 2021-02-10T11:48:27.213 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.193 7f6184a50700 1 -- 172.21.2.112:0/2423253692 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f61800d1c70 msgr2=0x7f61800cef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:27.213 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.193 7f6184a50700 1 --2- 172.21.2.112:0/2423253692 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f61800d1c70 0x7f61800cef60 secure :-1 s=READY pgs=3175 cs=0 l=1 rev1=1 rx=0x7f6168006700 tx=0x7f6168009b80).stop 2021-02-10T11:48:27.213 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.194 7f6184a50700 1 -- 172.21.2.112:0/2423253692 shutdown_connections 2021-02-10T11:48:27.213 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.194 7f6184a50700 1 --2- 172.21.2.112:0/2423253692 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f61800d1c70 0x7f61800cef60 unknown :-1 s=CLOSED pgs=3175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:27.213 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.194 7f6184a50700 1 --2- 172.21.2.112:0/2423253692 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f61800cf580 0x7f61800cf980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:27.213 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.194 7f6184a50700 1 -- 172.21.2.112:0/2423253692 >> 172.21.2.112:0/2423253692 conn(0x7f61800ce420 msgr2=0x7f61800d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:27.213 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.194 7f6184a50700 1 -- 172.21.2.112:0/2423253692 shutdown_connections 2021-02-10T11:48:27.213 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.194 7f6184a50700 1 -- 172.21.2.112:0/2423253692 wait complete. 2021-02-10T11:48:27.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.208 7f6184a50700 1 Processor -- start 2021-02-10T11:48:27.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.208 7f6184a50700 1 -- start start 2021-02-10T11:48:27.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.209 7f6184a50700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f61800c3820 0x7f61800a7950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:27.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.209 7f6184a50700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f61801a44c0 0x7f61801a48c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:27.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.209 7f6184a50700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f61800863d0 0x7f61800867d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:27.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.209 7f6184a50700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6180085ed0 con 0x7f61800c3820 2021-02-10T11:48:27.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.209 7f6184a50700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61800c0740 con 0x7f61800863d0 2021-02-10T11:48:27.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.209 7f6184a50700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6180086df0 con 0x7f61801a44c0 2021-02-10T11:48:27.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.209 7f617e85e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f61800c3820 0x7f61800a7950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:27.215 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.209 7f617e85e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f61800c3820 0x7f61800a7950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:33036/0 (socket says 172.21.2.112:33036) 2021-02-10T11:48:27.216 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.209 7f617e85e700 1 -- 172.21.2.112:0/2083370780 learned_addr learned my addr 172.21.2.112:0/2083370780 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:48:27.216 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.209 7f617e85e700 1 -- 172.21.2.112:0/2083370780 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f61801a44c0 msgr2=0x7f61801a48c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:27.216 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.209 7f617e85e700 1 --2- 172.21.2.112:0/2083370780 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f61801a44c0 0x7f61801a48c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:27.216 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.209 7f617e85e700 1 -- 172.21.2.112:0/2083370780 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f61800863d0 msgr2=0x7f61800867d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:27.216 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.209 7f617e85e700 1 --2- 172.21.2.112:0/2083370780 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f61800863d0 0x7f61800867d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:27.216 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.210 7f617e85e700 1 -- 172.21.2.112:0/2083370780 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6180087010 con 0x7f61800c3820 2021-02-10T11:48:27.216 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.210 7f617e85e700 1 --2- 172.21.2.112:0/2083370780 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f61800c3820 0x7f61800a7950 secure :-1 s=READY pgs=3176 cs=0 l=1 rev1=1 rx=0x7f617000b530 tx=0x7f617000b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:27.216 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.210 7f61777fe700 1 -- 172.21.2.112:0/2083370780 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f617000bcc0 con 0x7f61800c3820 2021-02-10T11:48:27.216 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.210 7f61777fe700 1 -- 172.21.2.112:0/2083370780 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6170006020 con 0x7f61800c3820 2021-02-10T11:48:27.216 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.210 7f61777fe700 1 -- 172.21.2.112:0/2083370780 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f61700054a0 con 0x7f61800c3820 2021-02-10T11:48:27.216 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.210 7f6184a50700 1 -- 172.21.2.112:0/2083370780 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6180087010 con 0x7f61800c3820 2021-02-10T11:48:27.216 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.210 7f6184a50700 1 -- 172.21.2.112:0/2083370780 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6180087010 con 0x7f61800c3820 2021-02-10T11:48:27.216 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.211 7f61777fe700 1 -- 172.21.2.112:0/2083370780 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6170006200 con 0x7f61800c3820 2021-02-10T11:48:27.216 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.211 7f61777fe700 1 --2- 172.21.2.112:0/2083370780 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f61640186c0 0x7f616401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:27.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.211 7f61777fe700 1 -- 172.21.2.112:0/2083370780 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f61700327e0 con 0x7f61800c3820 2021-02-10T11:48:27.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.211 7f617e05d700 1 --2- 172.21.2.112:0/2083370780 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f61640186c0 0x7f616401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:27.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.212 7f617e05d700 1 --2- 172.21.2.112:0/2083370780 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f61640186c0 0x7f616401abc0 crc :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:27.246 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.243 7f6184a50700 1 -- 172.21.2.112:0/2083370780 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61800fe1c0 con 0x7f61800c3820 2021-02-10T11:48:27.248 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.245 7f61777fe700 1 -- 172.21.2.112:0/2083370780 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f617000bcc0 con 0x7f61800c3820 2021-02-10T11:48:27.378 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.375 7f6184a50700 1 -- 172.21.2.112:0/2083370780 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f61800fe1c0 con 0x7f61800c3820 2021-02-10T11:48:27.379 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.375 7f61777fe700 1 -- 172.21.2.112:0/2083370780 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v189) v1 ==== 100+0+10 (secure 0 0 0) 0x7f6170005ef0 con 0x7f61800c3820 2021-02-10T11:48:27.379 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:48:27.381 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.377 7f6184a50700 1 -- 172.21.2.112:0/2083370780 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f61640186c0 msgr2=0x7f616401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:27.381 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.377 7f6184a50700 1 --2- 172.21.2.112:0/2083370780 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f61640186c0 0x7f616401abc0 crc :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:27.381 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.377 7f6184a50700 1 -- 172.21.2.112:0/2083370780 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f61800c3820 msgr2=0x7f61800a7950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:27.381 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.377 7f6184a50700 1 --2- 172.21.2.112:0/2083370780 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f61800c3820 0x7f61800a7950 secure :-1 s=READY pgs=3176 cs=0 l=1 rev1=1 rx=0x7f617000b530 tx=0x7f617000b8d0).stop 2021-02-10T11:48:27.381 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.377 7f6184a50700 1 -- 172.21.2.112:0/2083370780 shutdown_connections 2021-02-10T11:48:27.381 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.377 7f6184a50700 1 --2- 172.21.2.112:0/2083370780 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f61640186c0 0x7f616401abc0 unknown :-1 s=CLOSED pgs=1429 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:27.381 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.377 7f6184a50700 1 --2- 172.21.2.112:0/2083370780 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f61800863d0 0x7f61800867d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:27.381 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.377 7f6184a50700 1 --2- 172.21.2.112:0/2083370780 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f61801a44c0 0x7f61801a48c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:27.381 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.377 7f6184a50700 1 --2- 172.21.2.112:0/2083370780 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f61800c3820 0x7f61800a7950 unknown :-1 s=CLOSED pgs=3176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:27.381 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.377 7f6184a50700 1 -- 172.21.2.112:0/2083370780 >> 172.21.2.112:0/2083370780 conn(0x7f61800872c0 msgr2=0x7f6180095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:27.381 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.378 7f6184a50700 1 -- 172.21.2.112:0/2083370780 shutdown_connections 2021-02-10T11:48:27.381 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.378 7f6184a50700 1 -- 172.21.2.112:0/2083370780 wait complete. 2021-02-10T11:48:27.389 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:48:27.389 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:48:27.819 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.799 7f99de48e700 1 Processor -- start 2021-02-10T11:48:27.819 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.799 7f99de48e700 1 -- start start 2021-02-10T11:48:27.819 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.800 7f99de48e700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f99d80d1870 0x7f99d81b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:27.819 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.800 7f99de48e700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f99d81b0a10 0x7f99d80cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:27.819 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.800 7f99de48e700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99d80f0b10 con 0x7f99d80d1c70 2021-02-10T11:48:27.819 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.800 7f99de48e700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99d80e5ed0 con 0x7f99d80d1870 2021-02-10T11:48:27.819 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.800 7f99de48e700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99d80f0650 con 0x7f99d81b0a10 2021-02-10T11:48:27.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.800 7f99d7fff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f99d80d1870 0x7f99d81b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:27.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.800 7f99d7fff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f99d80d1870 0x7f99d81b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:33040/0 (socket says 172.21.2.112:33040) 2021-02-10T11:48:27.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.800 7f99d7fff700 1 -- 172.21.2.112:0/3590519433 learned_addr learned my addr 172.21.2.112:0/3590519433 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:48:27.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.800 7f99d7fff700 1 -- 172.21.2.112:0/3590519433 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f99d81b0a10 msgr2=0x7f99d80cf330 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:48:27.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.800 7f99d7fff700 1 --2- 172.21.2.112:0/3590519433 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f99d81b0a10 0x7f99d80cf330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:27.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.800 7f99d7fff700 1 -- 172.21.2.112:0/3590519433 >> v1:172.21.2.120:6789/0 conn(0x7f99d80d1c70 legacy=0x7f99d80cf950 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:48:27.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.800 7f99d7fff700 1 -- 172.21.2.112:0/3590519433 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99d80f0870 con 0x7f99d80d1870 2021-02-10T11:48:27.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.800 7f99d7fff700 1 --2- 172.21.2.112:0/3590519433 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f99d80d1870 0x7f99d81b03f0 secure :-1 s=READY pgs=3177 cs=0 l=1 rev1=1 rx=0x7f99c8006700 tx=0x7f99c8009b00).ready entity=mon.0 client_cookie=a04ea3d69497a35 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:27.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.800 7f99d6ffd700 1 -- 172.21.2.112:0/3590519433 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f99c800c0c0 con 0x7f99d80d1870 2021-02-10T11:48:27.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.800 7f99d6ffd700 1 -- 172.21.2.112:0/3590519433 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99c800c340 con 0x7f99d80d1870 2021-02-10T11:48:27.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.801 7f99d6ffd700 1 -- 172.21.2.112:0/3590519433 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f99c800c820 con 0x7f99d80d1870 2021-02-10T11:48:27.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.801 7f99de48e700 1 -- 172.21.2.112:0/3590519433 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f99d80d1870 msgr2=0x7f99d81b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:27.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.801 7f99de48e700 1 --2- 172.21.2.112:0/3590519433 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f99d80d1870 0x7f99d81b03f0 secure :-1 s=READY pgs=3177 cs=0 l=1 rev1=1 rx=0x7f99c8006700 tx=0x7f99c8009b00).stop 2021-02-10T11:48:27.821 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.801 7f99de48e700 1 -- 172.21.2.112:0/3590519433 shutdown_connections 2021-02-10T11:48:27.821 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.801 7f99de48e700 1 --2- 172.21.2.112:0/3590519433 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f99d81b0a10 0x7f99d80cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:27.821 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.801 7f99de48e700 1 --2- 172.21.2.112:0/3590519433 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f99d80d1870 0x7f99d81b03f0 unknown :-1 s=CLOSED pgs=3177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:27.821 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.801 7f99de48e700 1 -- 172.21.2.112:0/3590519433 >> 172.21.2.112:0/3590519433 conn(0x7f99d80ce450 msgr2=0x7f99d80d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:27.821 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.801 7f99de48e700 1 -- 172.21.2.112:0/3590519433 shutdown_connections 2021-02-10T11:48:27.821 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.801 7f99de48e700 1 -- 172.21.2.112:0/3590519433 wait complete. 2021-02-10T11:48:27.821 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.816 7f99de48e700 1 Processor -- start 2021-02-10T11:48:27.822 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.816 7f99de48e700 1 -- start start 2021-02-10T11:48:27.823 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.816 7f99de48e700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f99d818a540 0x7f99d8197480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:27.823 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.816 7f99de48e700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f99d818aa50 0x7f99d80b77d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:27.823 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.816 7f99de48e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f99d80b7df0 0x7f99d80b81f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:27.823 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.816 7f99de48e700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99d81a79d0 con 0x7f99d80b7df0 2021-02-10T11:48:27.823 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.816 7f99de48e700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99d81a9e20 con 0x7f99d818aa50 2021-02-10T11:48:27.823 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.816 7f99de48e700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99d81aa780 con 0x7f99d818a540 2021-02-10T11:48:27.823 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.816 7f99dcaab700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f99d80b7df0 0x7f99d80b81f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:27.823 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.816 7f99dcaab700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f99d80b7df0 0x7f99d80b81f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:33048/0 (socket says 172.21.2.112:33048) 2021-02-10T11:48:27.823 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.817 7f99dcaab700 1 -- 172.21.2.112:0/1616361662 learned_addr learned my addr 172.21.2.112:0/1616361662 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:48:27.823 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.817 7f99dcaab700 1 -- 172.21.2.112:0/1616361662 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f99d818a540 msgr2=0x7f99d8197480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:27.823 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.817 7f99dcaab700 1 --2- 172.21.2.112:0/1616361662 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f99d818a540 0x7f99d8197480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:27.823 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.817 7f99dcaab700 1 -- 172.21.2.112:0/1616361662 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f99d818aa50 msgr2=0x7f99d80b77d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:27.824 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.817 7f99dcaab700 1 --2- 172.21.2.112:0/1616361662 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f99d818aa50 0x7f99d80b77d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:27.824 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.817 7f99dcaab700 1 -- 172.21.2.112:0/1616361662 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99d81aa9a0 con 0x7f99d80b7df0 2021-02-10T11:48:27.824 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.817 7f99dcaab700 1 --2- 172.21.2.112:0/1616361662 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f99d80b7df0 0x7f99d80b81f0 secure :-1 s=READY pgs=3178 cs=0 l=1 rev1=1 rx=0x7f99d0009940 tx=0x7f99d000bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:27.824 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.817 7f99d4ff9700 1 -- 172.21.2.112:0/1616361662 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f99d0000f90 con 0x7f99d80b7df0 2021-02-10T11:48:27.824 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.817 7f99d4ff9700 1 -- 172.21.2.112:0/1616361662 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99d0000d30 con 0x7f99d80b7df0 2021-02-10T11:48:27.824 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.817 7f99de48e700 1 -- 172.21.2.112:0/1616361662 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99d81aa9a0 con 0x7f99d80b7df0 2021-02-10T11:48:27.824 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.817 7f99de48e700 1 -- 172.21.2.112:0/1616361662 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99d81aaf30 con 0x7f99d80b7df0 2021-02-10T11:48:27.824 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.817 7f99d4ff9700 1 -- 172.21.2.112:0/1616361662 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f99d000c570 con 0x7f99d80b7df0 2021-02-10T11:48:27.824 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.818 7f99d4ff9700 1 -- 172.21.2.112:0/1616361662 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f99d0001220 con 0x7f99d80b7df0 2021-02-10T11:48:27.824 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.818 7f99d4ff9700 1 --2- 172.21.2.112:0/1616361662 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f99b80186c0 0x7f99b801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:27.824 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.818 7f99d4ff9700 1 -- 172.21.2.112:0/1616361662 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f99d0032b40 con 0x7f99d80b7df0 2021-02-10T11:48:27.824 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.818 7f99d7fff700 1 --2- 172.21.2.112:0/1616361662 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f99b80186c0 0x7f99b801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:27.824 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.819 7f99d7fff700 1 --2- 172.21.2.112:0/1616361662 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f99b80186c0 0x7f99b801abc0 crc :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:27.834 DEBUG:teuthology.orchestra.run.gibba012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:48:27.837 DEBUG:teuthology.orchestra.run.gibba020:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:48:27.853 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.849 7f99de48e700 1 -- 172.21.2.112:0/1616361662 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99d81aa250 con 0x7f99d80b7df0 2021-02-10T11:48:27.855 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.852 7f99d4ff9700 1 -- 172.21.2.112:0/1616361662 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f99d0000f90 con 0x7f99d80b7df0 2021-02-10T11:48:27.987 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.984 7f99de48e700 1 -- 172.21.2.112:0/1616361662 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f99d81aa250 con 0x7f99d80b7df0 2021-02-10T11:48:27.987 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.984 7f99d4ff9700 1 -- 172.21.2.112:0/1616361662 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 165 v165) v1 ==== 95+0+2107 (secure 0 0 0) 0x7f99d0000cc0 con 0x7f99d80b7df0 2021-02-10T11:48:27.988 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:48:27.988 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":165,"default_fscid":20,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7763,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/1478301417","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":1478301417},{"type":"v1","addr":"172.21.2.112:6833","nonce":1478301417}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":161}],"filesystems":[{"mdsmap":{"epoch":165,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:48:10.481032","modified":"2021-02-10 11:48:12.490392","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7760":{"gid":7760,"name":"b","rank":0,"incarnation":162,"state":"up:active","state_seq":3,"addr":"172.21.2.120:6835/2943198956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":2943198956},{"type":"v1","addr":"172.21.2.120:6835","nonce":2943198956}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[41],"metadata_pool":40,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":20}]} 2021-02-10T11:48:27.988 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 165 2021-02-10T11:48:27.990 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.986 7f99de48e700 1 -- 172.21.2.112:0/1616361662 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f99b80186c0 msgr2=0x7f99b801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:27.990 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.986 7f99de48e700 1 --2- 172.21.2.112:0/1616361662 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f99b80186c0 0x7f99b801abc0 crc :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:27.990 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.986 7f99de48e700 1 -- 172.21.2.112:0/1616361662 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f99d80b7df0 msgr2=0x7f99d80b81f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:27.990 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.986 7f99de48e700 1 --2- 172.21.2.112:0/1616361662 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f99d80b7df0 0x7f99d80b81f0 secure :-1 s=READY pgs=3178 cs=0 l=1 rev1=1 rx=0x7f99d0009940 tx=0x7f99d000bcf0).stop 2021-02-10T11:48:27.990 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.986 7f99de48e700 1 -- 172.21.2.112:0/1616361662 shutdown_connections 2021-02-10T11:48:27.990 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.986 7f99de48e700 1 --2- 172.21.2.112:0/1616361662 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f99b80186c0 0x7f99b801abc0 unknown :-1 s=CLOSED pgs=1430 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:27.990 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.986 7f99de48e700 1 --2- 172.21.2.112:0/1616361662 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f99d818aa50 0x7f99d80b77d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:27.990 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.986 7f99de48e700 1 --2- 172.21.2.112:0/1616361662 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f99d80b7df0 0x7f99d80b81f0 unknown :-1 s=CLOSED pgs=3178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:27.990 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.986 7f99de48e700 1 --2- 172.21.2.112:0/1616361662 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f99d818a540 0x7f99d8197480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:27.990 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.986 7f99de48e700 1 -- 172.21.2.112:0/1616361662 >> 172.21.2.112:0/1616361662 conn(0x7f99d80efc70 msgr2=0x7f99d80ef410 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:27.990 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.986 7f99de48e700 1 -- 172.21.2.112:0/1616361662 shutdown_connections 2021-02-10T11:48:27.991 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:27.987 7f99de48e700 1 -- 172.21.2.112:0/1616361662 wait complete. 2021-02-10T11:48:28.198 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 165, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:48:10.481032', 'modified': '2021-02-10 11:48:12.490392', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7760}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7760': {'gid': 7760, 'name': 'b', 'rank': 0, 'incarnation': 162, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.120:6835/2943198956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.120:6834', 'nonce': 2943198956}, {'type': 'v1', 'addr': '172.21.2.120:6835', 'nonce': 2943198956}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [41], 'metadata_pool': 40, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:48:28.199 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:48:28.199 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:48:28.599 INFO:teuthology.orchestra.run.gibba020.stdout:{ 2021-02-10T11:48:28.599 INFO:teuthology.orchestra.run.gibba020.stdout: "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", 2021-02-10T11:48:28.599 INFO:teuthology.orchestra.run.gibba020.stdout: "whoami": 0, 2021-02-10T11:48:28.599 INFO:teuthology.orchestra.run.gibba020.stdout: "id": 7760, 2021-02-10T11:48:28.599 INFO:teuthology.orchestra.run.gibba020.stdout: "want_state": "up:active", 2021-02-10T11:48:28.599 INFO:teuthology.orchestra.run.gibba020.stdout: "state": "up:active", 2021-02-10T11:48:28.599 INFO:teuthology.orchestra.run.gibba020.stdout: "rank_uptime": 18.112291544000001, 2021-02-10T11:48:28.599 INFO:teuthology.orchestra.run.gibba020.stdout: "mdsmap_epoch": 165, 2021-02-10T11:48:28.599 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch": 189, 2021-02-10T11:48:28.600 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch_barrier": 189, 2021-02-10T11:48:28.600 INFO:teuthology.orchestra.run.gibba020.stdout: "uptime": 23.762614956 2021-02-10T11:48:28.600 INFO:teuthology.orchestra.run.gibba020.stdout:} 2021-02-10T11:48:28.603 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", "whoami": 0, "id": 7760, "want_state": "up:active", "state": "up:active", "rank_uptime": 18.112291544000001, "mdsmap_epoch": 165, "osdmap_epoch": 189, "osdmap_epoch_barrier": 189, "uptime": 23.762614956 } 2021-02-10T11:48:28.603 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:48:28.603 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2021-02-10T11:48:28.603 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.1 at ubuntu@gibba012.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2021-02-10T11:48:28.603 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:48:28.604 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:48:28.939 DEBUG:teuthology.orchestra.run.gibba012:> sudo modprobe fuse 2021-02-10T11:48:29.284 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:48:29.284 DEBUG:teuthology.orchestra.run.gibba012:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:48:29.629 INFO:teuthology.orchestra.run.gibba012.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:48:29.629 INFO:teuthology.orchestra.run.gibba012.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:48:29.629 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:48:29.630 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:48:29.630 DEBUG:teuthology.orchestra.run.gibba012:> ls /sys/fs/fuse/connections 2021-02-10T11:48:29.964 INFO:teuthology.orchestra.run.gibba012.stdout:42 2021-02-10T11:48:29.965 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [42] 2021-02-10T11:48:29.965 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.1 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:48:29.968 DEBUG:teuthology.orchestra.run.gibba012:> sudo modprobe fuse 2021-02-10T11:48:30.311 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:48:30.311 DEBUG:teuthology.orchestra.run.gibba012:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:48:30.378 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba012.stderr:ceph-fuse[193075]: starting ceph client 2021-02-10T11:48:30.378 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba012.stderr:2021-02-10 11:48:30.374 7f4e771d3f80 -1 init, newargv = 0x55b6acde2d70 newargc=9 2021-02-10T11:48:30.382 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba012.stderr:ceph-fuse[193075]: starting fuse 2021-02-10T11:48:30.654 INFO:teuthology.orchestra.run.gibba012.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:48:30.654 INFO:teuthology.orchestra.run.gibba012.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:48:30.655 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:48:30.655 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:48:30.655 DEBUG:teuthology.orchestra.run.gibba012:> ls /sys/fs/fuse/connections 2021-02-10T11:48:30.991 INFO:teuthology.orchestra.run.gibba012.stdout:42 2021-02-10T11:48:30.991 INFO:teuthology.orchestra.run.gibba012.stdout:43 2021-02-10T11:48:30.991 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42, 43] 2021-02-10T11:48:30.991 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:48:30.992 DEBUG:teuthology.orchestra.run.gibba012:> sudo python3 -c ' 2021-02-10T11:48:30.992 DEBUG:teuthology.orchestra.run.gibba012:> import glob 2021-02-10T11:48:30.992 DEBUG:teuthology.orchestra.run.gibba012:> import re 2021-02-10T11:48:30.992 DEBUG:teuthology.orchestra.run.gibba012:> import os 2021-02-10T11:48:30.992 DEBUG:teuthology.orchestra.run.gibba012:> import subprocess 2021-02-10T11:48:30.992 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:48:30.992 DEBUG:teuthology.orchestra.run.gibba012:> def find_socket(client_name): 2021-02-10T11:48:30.992 DEBUG:teuthology.orchestra.run.gibba012:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2021-02-10T11:48:30.992 DEBUG:teuthology.orchestra.run.gibba012:> files = glob.glob(asok_path) 2021-02-10T11:48:30.992 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:48:30.992 DEBUG:teuthology.orchestra.run.gibba012:> # Given a non-glob path, it better be there 2021-02-10T11:48:30.992 DEBUG:teuthology.orchestra.run.gibba012:> if "*" not in asok_path: 2021-02-10T11:48:30.992 DEBUG:teuthology.orchestra.run.gibba012:> assert(len(files) == 1) 2021-02-10T11:48:30.993 DEBUG:teuthology.orchestra.run.gibba012:> return files[0] 2021-02-10T11:48:30.993 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:48:30.993 DEBUG:teuthology.orchestra.run.gibba012:> for f in files: 2021-02-10T11:48:30.993 DEBUG:teuthology.orchestra.run.gibba012:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:48:30.993 DEBUG:teuthology.orchestra.run.gibba012:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:48:30.993 DEBUG:teuthology.orchestra.run.gibba012:> return f 2021-02-10T11:48:30.993 DEBUG:teuthology.orchestra.run.gibba012:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:48:30.993 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:48:30.993 DEBUG:teuthology.orchestra.run.gibba012:> print(find_socket("client.1")) 2021-02-10T11:48:30.993 DEBUG:teuthology.orchestra.run.gibba012:> ' 2021-02-10T11:48:31.354 INFO:teuthology.orchestra.run.gibba012.stdout:/var/run/ceph/ceph-client.1.193075.asok 2021-02-10T11:48:31.357 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.193075.asok 2021-02-10T11:48:31.357 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:48:31.357 DEBUG:teuthology.orchestra.run.gibba012:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.193075.asok status 2021-02-10T11:48:31.747 INFO:teuthology.orchestra.run.gibba012.stdout:{ 2021-02-10T11:48:31.747 INFO:teuthology.orchestra.run.gibba012.stdout: "metadata": { 2021-02-10T11:48:31.747 INFO:teuthology.orchestra.run.gibba012.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:48:31.747 INFO:teuthology.orchestra.run.gibba012.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:48:31.747 INFO:teuthology.orchestra.run.gibba012.stdout: "entity_id": "1", 2021-02-10T11:48:31.747 INFO:teuthology.orchestra.run.gibba012.stdout: "hostname": "gibba012", 2021-02-10T11:48:31.747 INFO:teuthology.orchestra.run.gibba012.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2021-02-10T11:48:31.748 INFO:teuthology.orchestra.run.gibba012.stdout: "pid": "193075", 2021-02-10T11:48:31.748 INFO:teuthology.orchestra.run.gibba012.stdout: "root": "/" 2021-02-10T11:48:31.748 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:48:31.748 INFO:teuthology.orchestra.run.gibba012.stdout: "dentry_count": 0, 2021-02-10T11:48:31.748 INFO:teuthology.orchestra.run.gibba012.stdout: "dentry_pinned_count": 0, 2021-02-10T11:48:31.748 INFO:teuthology.orchestra.run.gibba012.stdout: "id": 13638, 2021-02-10T11:48:31.748 INFO:teuthology.orchestra.run.gibba012.stdout: "inst": { 2021-02-10T11:48:31.748 INFO:teuthology.orchestra.run.gibba012.stdout: "name": { 2021-02-10T11:48:31.748 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "client", 2021-02-10T11:48:31.748 INFO:teuthology.orchestra.run.gibba012.stdout: "num": 13638 2021-02-10T11:48:31.748 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:48:31.748 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": { 2021-02-10T11:48:31.748 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:48:31.748 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:0", 2021-02-10T11:48:31.749 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 2706771652 2021-02-10T11:48:31.749 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:48:31.749 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:48:31.749 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": { 2021-02-10T11:48:31.749 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:48:31.749 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:0", 2021-02-10T11:48:31.749 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 2706771652 2021-02-10T11:48:31.749 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:48:31.749 INFO:teuthology.orchestra.run.gibba012.stdout: "inst_str": "client.13638 172.21.2.112:0/2706771652", 2021-02-10T11:48:31.749 INFO:teuthology.orchestra.run.gibba012.stdout: "addr_str": "172.21.2.112:0/2706771652", 2021-02-10T11:48:31.749 INFO:teuthology.orchestra.run.gibba012.stdout: "inode_count": 1, 2021-02-10T11:48:31.749 INFO:teuthology.orchestra.run.gibba012.stdout: "mds_epoch": 165, 2021-02-10T11:48:31.749 INFO:teuthology.orchestra.run.gibba012.stdout: "osd_epoch": 189, 2021-02-10T11:48:31.750 INFO:teuthology.orchestra.run.gibba012.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:48:31.750 INFO:teuthology.orchestra.run.gibba012.stdout: "blacklisted": false 2021-02-10T11:48:31.750 INFO:teuthology.orchestra.run.gibba012.stdout:} 2021-02-10T11:48:31.751 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:48:31.751 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:48:31.752 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:48:32.087 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:48:32.087 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:48:32.088 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:48:32.088 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:48:32.434 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2021-02-10T11:48:32.867 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:32.847 7f43e010b700 1 Processor -- start 2021-02-10T11:48:32.867 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:32.847 7f43e010b700 1 -- start start 2021-02-10T11:48:32.867 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:32.848 7f43e010b700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f43d80d1c80 0x7f43d80cef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:32.867 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:32.848 7f43e010b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f43d80cf590 0x7f43d80cf990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:32.867 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:32.848 7f43e010b700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43d80fe1d0 con 0x7f43d80d1880 2021-02-10T11:48:32.867 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:32.848 7f43e010b700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43d81b0cc0 con 0x7f43d80d1c80 2021-02-10T11:48:32.867 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:32.848 7f43e010b700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43d8109a70 con 0x7f43d80cf590 2021-02-10T11:48:32.867 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:32.848 7f43dd726700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f43d80d1c80 0x7f43d80cef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:32.867 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:32.848 7f43ddf27700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f43d80d1880 0x7f43d81b04d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:60760/0 (socket says 172.21.2.112:60760) 2021-02-10T11:48:32.868 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:32.848 7f43ddf27700 1 -- 172.21.2.112:0/1640828523 learned_addr learned my addr 172.21.2.112:0/1640828523 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:48:32.868 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:32.848 7f43dd726700 1 -- 172.21.2.112:0/1640828523 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f43d80cf590 msgr2=0x7f43d80cf990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:32.868 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:32.848 7f43dd726700 1 --2- 172.21.2.112:0/1640828523 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f43d80cf590 0x7f43d80cf990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:32.868 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:32.848 7f43dcf25700 1 -- 172.21.2.112:0/1640828523 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3691097381 0 0) 0x7f43d40009b0 con 0x7f43d80d1880 2021-02-10T11:48:32.868 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:32.848 7f43dd726700 1 -- 172.21.2.112:0/1640828523 >> v1:172.21.2.112:6789/0 conn(0x7f43d80d1880 legacy=0x7f43d81b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:32.868 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:32.848 7f43dd726700 1 -- 172.21.2.112:0/1640828523 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43d8088070 con 0x7f43d80d1c80 2021-02-10T11:48:32.868 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:32.848 7f43dd726700 1 --2- 172.21.2.112:0/1640828523 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f43d80d1c80 0x7f43d80cef70 secure :-1 s=READY pgs=3182 cs=0 l=1 rev1=1 rx=0x7f43c8006650 tx=0x7f43c8009a80).ready entity=mon.0 client_cookie=8be7d04b109af1bb server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:32.868 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:32.849 7f43dcf25700 1 -- 172.21.2.112:0/1640828523 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f43c80009b0 con 0x7f43d80d1c80 2021-02-10T11:48:32.868 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:32.849 7f43dcf25700 1 -- 172.21.2.112:0/1640828523 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43c800c0c0 con 0x7f43d80d1c80 2021-02-10T11:48:32.868 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:32.849 7f43dcf25700 1 -- 172.21.2.112:0/1640828523 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f43c800c510 con 0x7f43d80d1c80 2021-02-10T11:48:32.868 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:32.849 7f43e010b700 1 -- 172.21.2.112:0/1640828523 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f43d80d1c80 msgr2=0x7f43d80cef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:32.868 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:32.849 7f43e010b700 1 --2- 172.21.2.112:0/1640828523 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f43d80d1c80 0x7f43d80cef70 secure :-1 s=READY pgs=3182 cs=0 l=1 rev1=1 rx=0x7f43c8006650 tx=0x7f43c8009a80).stop 2021-02-10T11:48:32.868 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:32.849 7f43e010b700 1 -- 172.21.2.112:0/1640828523 shutdown_connections 2021-02-10T11:48:32.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:32.849 7f43e010b700 1 --2- 172.21.2.112:0/1640828523 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f43d80cf590 0x7f43d80cf990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:32.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:32.849 7f43e010b700 1 --2- 172.21.2.112:0/1640828523 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f43d80d1c80 0x7f43d80cef70 unknown :-1 s=CLOSED pgs=3182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:32.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:32.849 7f43e010b700 1 -- 172.21.2.112:0/1640828523 >> 172.21.2.112:0/1640828523 conn(0x7f43d80ce430 msgr2=0x7f43d80d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:32.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:32.849 7f43e010b700 1 -- 172.21.2.112:0/1640828523 shutdown_connections 2021-02-10T11:48:32.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:32.849 7f43e010b700 1 -- 172.21.2.112:0/1640828523 wait complete. 2021-02-10T11:48:32.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:32.864 7f43e010b700 1 Processor -- start 2021-02-10T11:48:32.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:32.864 7f43e010b700 1 -- start start 2021-02-10T11:48:32.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:32.864 7f43e010b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f43d8083f90 0x7f43d81a4500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:32.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:32.864 7f43e010b700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f43d8084390 0x7f43d80b18a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:32.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:32.864 7f43e010b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f43d81b6f40 0x7f43d80b1ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:32.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:32.864 7f43e010b700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43d81a4b20 con 0x7f43d8083f90 2021-02-10T11:48:32.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:32.864 7f43e010b700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43d81b7520 con 0x7f43d8084390 2021-02-10T11:48:32.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:32.864 7f43e010b700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43d80b24e0 con 0x7f43d81b6f40 2021-02-10T11:48:32.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:32.865 7f43ddf27700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f43d8083f90 0x7f43d81a4500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:32.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:32.865 7f43ddf27700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f43d8083f90 0x7f43d81a4500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:33070/0 (socket says 172.21.2.112:33070) 2021-02-10T11:48:32.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:32.865 7f43ddf27700 1 -- 172.21.2.112:0/594863379 learned_addr learned my addr 172.21.2.112:0/594863379 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:48:32.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:32.865 7f43ddf27700 1 -- 172.21.2.112:0/594863379 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f43d81b6f40 msgr2=0x7f43d80b1ec0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:48:32.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:32.865 7f43ddf27700 1 --2- 172.21.2.112:0/594863379 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f43d81b6f40 0x7f43d80b1ec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:32.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:32.865 7f43ddf27700 1 -- 172.21.2.112:0/594863379 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f43d8084390 msgr2=0x7f43d80b18a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:48:32.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:32.865 7f43ddf27700 1 --2- 172.21.2.112:0/594863379 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f43d8084390 0x7f43d80b18a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:32.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:32.865 7f43ddf27700 1 -- 172.21.2.112:0/594863379 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43d81b7aa0 con 0x7f43d8083f90 2021-02-10T11:48:32.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:32.865 7f43ddf27700 1 --2- 172.21.2.112:0/594863379 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f43d8083f90 0x7f43d81a4500 secure :-1 s=READY pgs=3183 cs=0 l=1 rev1=1 rx=0x7f43d4001c60 tx=0x7f43d400a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:32.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:32.865 7f43ce7fc700 1 -- 172.21.2.112:0/594863379 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f43d400acc0 con 0x7f43d8083f90 2021-02-10T11:48:32.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:32.865 7f43ce7fc700 1 -- 172.21.2.112:0/594863379 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43d4004140 con 0x7f43d8083f90 2021-02-10T11:48:32.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:32.865 7f43ce7fc700 1 -- 172.21.2.112:0/594863379 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f43d4004660 con 0x7f43d8083f90 2021-02-10T11:48:32.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:32.865 7f43e010b700 1 -- 172.21.2.112:0/594863379 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43d81b7aa0 con 0x7f43d8083f90 2021-02-10T11:48:32.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:32.866 7f43e010b700 1 -- 172.21.2.112:0/594863379 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43d81b7aa0 con 0x7f43d8083f90 2021-02-10T11:48:32.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:32.866 7f43ce7fc700 1 -- 172.21.2.112:0/594863379 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f43d4004320 con 0x7f43d8083f90 2021-02-10T11:48:32.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:32.866 7f43ce7fc700 1 --2- 172.21.2.112:0/594863379 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f43c00186c0 0x7f43c001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:32.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:32.866 7f43ce7fc700 1 -- 172.21.2.112:0/594863379 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f43d40325e0 con 0x7f43d8083f90 2021-02-10T11:48:32.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:32.867 7f43dd726700 1 --2- 172.21.2.112:0/594863379 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f43c00186c0 0x7f43c001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:32.872 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:32.867 7f43dd726700 1 --2- 172.21.2.112:0/594863379 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f43c00186c0 0x7f43c001abc0 crc :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:32.901 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:32.897 7f43e010b700 1 -- 172.21.2.112:0/594863379 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43d8082aa0 con 0x7f43d8083f90 2021-02-10T11:48:32.903 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:32.900 7f43ce7fc700 1 -- 172.21.2.112:0/594863379 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f43d40009b0 con 0x7f43d8083f90 2021-02-10T11:48:33.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:33.036 7f43e010b700 1 -- 172.21.2.112:0/594863379 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f43d8082aa0 con 0x7f43d8083f90 2021-02-10T11:48:33.306 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:33.303 7f43ce7fc700 1 -- 172.21.2.112:0/594863379 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v166) v1 ==== 136+0+0 (secure 0 0 0) 0x7f43d4004140 con 0x7f43d8083f90 2021-02-10T11:48:33.358 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:33.354 7f43e010b700 1 -- 172.21.2.112:0/594863379 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f43c00186c0 msgr2=0x7f43c001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:33.358 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:33.354 7f43e010b700 1 --2- 172.21.2.112:0/594863379 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f43c00186c0 0x7f43c001abc0 crc :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:33.358 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:33.354 7f43e010b700 1 -- 172.21.2.112:0/594863379 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f43d8083f90 msgr2=0x7f43d81a4500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:33.358 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:33.354 7f43e010b700 1 --2- 172.21.2.112:0/594863379 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f43d8083f90 0x7f43d81a4500 secure :-1 s=READY pgs=3183 cs=0 l=1 rev1=1 rx=0x7f43d4001c60 tx=0x7f43d400a8d0).stop 2021-02-10T11:48:33.358 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:33.354 7f43e010b700 1 -- 172.21.2.112:0/594863379 shutdown_connections 2021-02-10T11:48:33.358 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:33.354 7f43e010b700 1 --2- 172.21.2.112:0/594863379 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f43c00186c0 0x7f43c001abc0 unknown :-1 s=CLOSED pgs=1431 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:33.358 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:33.354 7f43e010b700 1 --2- 172.21.2.112:0/594863379 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f43d8084390 0x7f43d80b18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:33.358 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:33.354 7f43e010b700 1 --2- 172.21.2.112:0/594863379 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f43d81b6f40 0x7f43d80b1ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:33.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:33.354 7f43e010b700 1 --2- 172.21.2.112:0/594863379 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f43d8083f90 0x7f43d81a4500 unknown :-1 s=CLOSED pgs=3183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:33.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:33.354 7f43e010b700 1 -- 172.21.2.112:0/594863379 >> 172.21.2.112:0/594863379 conn(0x7f43d819c650 msgr2=0x7f43d8082cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:33.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:33.354 7f43e010b700 1 -- 172.21.2.112:0/594863379 shutdown_connections 2021-02-10T11:48:33.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:33.354 7f43e010b700 1 -- 172.21.2.112:0/594863379 wait complete. 2021-02-10T11:48:33.366 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs volume ls 2021-02-10T11:48:33.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:33.773 7f853b261700 1 Processor -- start 2021-02-10T11:48:33.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:33.773 7f853b261700 1 -- start start 2021-02-10T11:48:33.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:33.774 7f853b261700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f85340d0340 0x7f85340ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:33.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:33.774 7f853b261700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f85340d1c70 0x7f85340efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:33.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:33.774 7f853b261700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85340f0b10 con 0x7f85340d1870 2021-02-10T11:48:33.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:33.774 7f853b261700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85340e5ed0 con 0x7f85340d1c70 2021-02-10T11:48:33.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:33.774 7f853b261700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85341b0be0 con 0x7f85340d0340 2021-02-10T11:48:33.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:33.775 7f853987e700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f85340d1c70 0x7f85340efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:33.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:33.775 7f853887c700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f85340d0340 0x7f85340ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:33.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:33.775 7f853907d700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7f85340d1870 0x7f85341b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:54330/0 (socket says 172.21.2.112:54330) 2021-02-10T11:48:33.795 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:33.775 7f853907d700 1 -- 172.21.2.112:0/1236159332 learned_addr learned my addr 172.21.2.112:0/1236159332 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:48:33.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:33.776 7f853887c700 1 -- 172.21.2.112:0/1236159332 >> v2:172.21.2.120:3300/0 conn(0x7f85340d1c70 msgr2=0x7f85340efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:33.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:33.776 7f853887c700 1 --2- 172.21.2.112:0/1236159332 >> v2:172.21.2.120:3300/0 conn(0x7f85340d1c70 0x7f85340efc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:33.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:33.776 7f853887c700 1 -- 172.21.2.112:0/1236159332 >> v1:172.21.2.120:6789/0 conn(0x7f85340d1870 legacy=0x7f85341b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:33.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:33.776 7f853887c700 1 -- 172.21.2.112:0/1236159332 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85340f20a0 con 0x7f85340d0340 2021-02-10T11:48:33.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:33.776 7f853987e700 1 --2- 172.21.2.112:0/1236159332 >> v2:172.21.2.120:3300/0 conn(0x7f85340d1c70 0x7f85340efc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:48:33.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:33.776 7f853887c700 1 --2- 172.21.2.112:0/1236159332 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f85340d0340 0x7f85340ef640 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7f8528006650 tx=0x7f8528009a80).ready entity=mon.2 client_cookie=2afdac1b3c664033 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:33.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:33.776 7f8527fff700 1 -- 172.21.2.112:0/1236159332 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f852800c0c0 con 0x7f85340d0340 2021-02-10T11:48:33.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:33.776 7f8527fff700 1 -- 172.21.2.112:0/1236159332 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f852800c340 con 0x7f85340d0340 2021-02-10T11:48:33.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:33.776 7f8527fff700 1 -- 172.21.2.112:0/1236159332 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f852800c860 con 0x7f85340d0340 2021-02-10T11:48:33.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:33.776 7f853b261700 1 -- 172.21.2.112:0/1236159332 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f85340d0340 msgr2=0x7f85340ef640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:33.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:33.776 7f853b261700 1 --2- 172.21.2.112:0/1236159332 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f85340d0340 0x7f85340ef640 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7f8528006650 tx=0x7f8528009a80).stop 2021-02-10T11:48:33.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:33.777 7f853b261700 1 -- 172.21.2.112:0/1236159332 shutdown_connections 2021-02-10T11:48:33.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:33.777 7f853b261700 1 --2- 172.21.2.112:0/1236159332 >> v2:172.21.2.120:3300/0 conn(0x7f85340d1c70 0x7f85340efc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:33.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:33.777 7f853b261700 1 --2- 172.21.2.112:0/1236159332 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f85340d0340 0x7f85340ef640 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:33.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:33.777 7f853b261700 1 -- 172.21.2.112:0/1236159332 >> 172.21.2.112:0/1236159332 conn(0x7f85340ce450 msgr2=0x7f85340d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:33.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:33.777 7f853b261700 1 -- 172.21.2.112:0/1236159332 shutdown_connections 2021-02-10T11:48:33.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:33.777 7f853b261700 1 -- 172.21.2.112:0/1236159332 wait complete. 2021-02-10T11:48:33.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:33.792 7f853b261700 1 Processor -- start 2021-02-10T11:48:33.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:33.792 7f853b261700 1 -- start start 2021-02-10T11:48:33.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:33.792 7f853b261700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f85341a06a0 0x7f853409f380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:33.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:33.792 7f853b261700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f853409f9a0 0x7f853408b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:33.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:33.792 7f853b261700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f853408bb80 0x7f853408bf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:33.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:33.792 7f853b261700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f853408c780 con 0x7f85341a06a0 2021-02-10T11:48:33.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:33.792 7f853b261700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f853409fda0 con 0x7f853409f9a0 2021-02-10T11:48:33.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:33.792 7f853b261700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f853408cd00 con 0x7f853408bb80 2021-02-10T11:48:33.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:33.792 7f853907d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f85341a06a0 0x7f853409f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:33.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:33.792 7f853907d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f85341a06a0 0x7f853409f380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:33086/0 (socket says 172.21.2.112:33086) 2021-02-10T11:48:33.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:33.792 7f853907d700 1 -- 172.21.2.112:0/1732214253 learned_addr learned my addr 172.21.2.112:0/1732214253 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:48:33.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:33.793 7f853907d700 1 -- 172.21.2.112:0/1732214253 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f853408bb80 msgr2=0x7f853408bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:33.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:33.793 7f853907d700 1 --2- 172.21.2.112:0/1732214253 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f853408bb80 0x7f853408bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:33.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:33.793 7f853907d700 1 -- 172.21.2.112:0/1732214253 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f853409f9a0 msgr2=0x7f853408b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:33.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:33.793 7f853907d700 1 --2- 172.21.2.112:0/1732214253 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f853409f9a0 0x7f853408b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:33.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:33.793 7f853907d700 1 -- 172.21.2.112:0/1732214253 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f853408cf20 con 0x7f85341a06a0 2021-02-10T11:48:33.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:33.793 7f853907d700 1 --2- 172.21.2.112:0/1732214253 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f85341a06a0 0x7f853409f380 secure :-1 s=READY pgs=3184 cs=0 l=1 rev1=1 rx=0x7f8530007fa0 tx=0x7f853000a950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:33.799 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:33.793 7f8525ffb700 1 -- 172.21.2.112:0/1732214253 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f85300070d0 con 0x7f85341a06a0 2021-02-10T11:48:33.800 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:33.793 7f8525ffb700 1 -- 172.21.2.112:0/1732214253 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f853000ad40 con 0x7f85341a06a0 2021-02-10T11:48:33.800 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:33.793 7f853b261700 1 -- 172.21.2.112:0/1732214253 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f853408cf20 con 0x7f85341a06a0 2021-02-10T11:48:33.800 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:33.793 7f8525ffb700 1 -- 172.21.2.112:0/1732214253 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f85300045a0 con 0x7f85341a06a0 2021-02-10T11:48:33.800 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:33.793 7f853b261700 1 -- 172.21.2.112:0/1732214253 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f853408d4b0 con 0x7f85341a06a0 2021-02-10T11:48:33.800 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:33.794 7f8525ffb700 1 -- 172.21.2.112:0/1732214253 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8530000c10 con 0x7f85341a06a0 2021-02-10T11:48:33.800 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:33.794 7f8525ffb700 1 --2- 172.21.2.112:0/1732214253 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f85140186c0 0x7f851401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:33.800 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:33.794 7f8525ffb700 1 -- 172.21.2.112:0/1732214253 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f8530032430 con 0x7f85341a06a0 2021-02-10T11:48:33.800 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:33.794 7f853887c700 1 --2- 172.21.2.112:0/1732214253 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f85140186c0 0x7f851401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:33.800 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:33.795 7f853887c700 1 --2- 172.21.2.112:0/1732214253 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f85140186c0 0x7f851401abc0 crc :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:33.827 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:33.823 7f853b261700 1 -- 172.21.2.112:0/1732214253 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85340ebc50 con 0x7f85341a06a0 2021-02-10T11:48:33.829 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:33.826 7f8525ffb700 1 -- 172.21.2.112:0/1732214253 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f853002df10 con 0x7f85341a06a0 2021-02-10T11:48:33.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:33.962 7f853b261700 1 -- 172.21.2.112:0/1732214253 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs volume ls", "target": ["mgr", ""]}) v1 -- 0x7f853408d4b0 con 0x7f85140186c0 2021-02-10T11:48:33.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:33.963 7f8525ffb700 1 -- 172.21.2.112:0/1732214253 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+41 (crc 0 0 0) 0x7f8528006040 con 0x7f85140186c0 2021-02-10T11:48:33.966 INFO:teuthology.orchestra.run.gibba012.stdout:[ 2021-02-10T11:48:33.966 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:48:33.966 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "cephfs" 2021-02-10T11:48:33.966 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:48:33.967 INFO:teuthology.orchestra.run.gibba012.stdout:] 2021-02-10T11:48:33.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:33.964 7f853b261700 1 -- 172.21.2.112:0/1732214253 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f85140186c0 msgr2=0x7f851401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:33.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:33.964 7f853b261700 1 --2- 172.21.2.112:0/1732214253 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f85140186c0 0x7f851401abc0 crc :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:33.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:33.964 7f853b261700 1 -- 172.21.2.112:0/1732214253 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f85341a06a0 msgr2=0x7f853409f380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:33.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:33.964 7f853b261700 1 --2- 172.21.2.112:0/1732214253 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f85341a06a0 0x7f853409f380 secure :-1 s=READY pgs=3184 cs=0 l=1 rev1=1 rx=0x7f8530007fa0 tx=0x7f853000a950).stop 2021-02-10T11:48:33.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:33.964 7f853b261700 1 -- 172.21.2.112:0/1732214253 shutdown_connections 2021-02-10T11:48:33.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:33.964 7f853b261700 1 --2- 172.21.2.112:0/1732214253 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f85140186c0 0x7f851401abc0 unknown :-1 s=CLOSED pgs=1432 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:33.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:33.964 7f853b261700 1 --2- 172.21.2.112:0/1732214253 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f853409f9a0 0x7f853408b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:33.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:33.964 7f853b261700 1 --2- 172.21.2.112:0/1732214253 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f853408bb80 0x7f853408bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:33.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:33.964 7f853b261700 1 --2- 172.21.2.112:0/1732214253 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f85341a06a0 0x7f853409f380 unknown :-1 s=CLOSED pgs=3184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:33.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:33.964 7f853b261700 1 -- 172.21.2.112:0/1732214253 >> 172.21.2.112:0/1732214253 conn(0x7f85340b83b0 msgr2=0x7f85340a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:33.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:33.964 7f853b261700 1 -- 172.21.2.112:0/1732214253 shutdown_connections 2021-02-10T11:48:33.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:33.964 7f853b261700 1 -- 172.21.2.112:0/1732214253 wait complete. 2021-02-10T11:48:33.976 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:48:34.409 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:34.389 7f5acffad700 1 Processor -- start 2021-02-10T11:48:34.410 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:34.390 7f5acffad700 1 -- start start 2021-02-10T11:48:34.410 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:34.390 7f5acffad700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f5ac80d1c70 0x7f5ac80cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:34.410 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:34.390 7f5acffad700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ac80d0170 con 0x7f5ac80d1870 2021-02-10T11:48:34.410 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:34.390 7f5acffad700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ac80fe1c0 con 0x7f5ac80cf580 2021-02-10T11:48:34.410 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:34.390 7f5acffad700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ac81b0cb0 con 0x7f5ac80d1c70 2021-02-10T11:48:34.410 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:34.390 7f5acd5c8700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f5ac80d1c70 0x7f5ac80cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:34.410 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:34.390 7f5acddc9700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f5ac80d1870 0x7f5ac81b04c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:60786/0 (socket says 172.21.2.112:60786) 2021-02-10T11:48:34.410 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:34.390 7f5acddc9700 1 -- 172.21.2.112:0/1934561119 learned_addr learned my addr 172.21.2.112:0/1934561119 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:48:34.410 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:34.391 7f5acd5c8700 1 -- 172.21.2.112:0/1934561119 >> v1:172.21.2.120:6789/0 conn(0x7f5ac80cf580 legacy=0x7f5ac80cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:34.411 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:34.391 7f5acd5c8700 1 -- 172.21.2.112:0/1934561119 >> v1:172.21.2.112:6789/0 conn(0x7f5ac80d1870 legacy=0x7f5ac81b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:34.411 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:34.391 7f5acd5c8700 1 -- 172.21.2.112:0/1934561119 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ac8109a60 con 0x7f5ac80d1c70 2021-02-10T11:48:34.411 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:34.391 7f5acd5c8700 1 --2- 172.21.2.112:0/1934561119 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5ac80d1c70 0x7f5ac80cef60 secure :-1 s=READY pgs=3186 cs=0 l=1 rev1=1 rx=0x7f5ac4006650 tx=0x7f5ac4009a80).ready entity=mon.0 client_cookie=80b8982fab3e9bcf server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:34.411 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:34.391 7f5accdc7700 1 -- 172.21.2.112:0/1934561119 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5ac40009b0 con 0x7f5ac80d1c70 2021-02-10T11:48:34.411 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:34.391 7f5accdc7700 1 -- 172.21.2.112:0/1934561119 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ac400c0c0 con 0x7f5ac80d1c70 2021-02-10T11:48:34.411 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:34.391 7f5accdc7700 1 -- 172.21.2.112:0/1934561119 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5ac400c5e0 con 0x7f5ac80d1c70 2021-02-10T11:48:34.411 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:34.391 7f5acffad700 1 -- 172.21.2.112:0/1934561119 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5ac80d1c70 msgr2=0x7f5ac80cef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:34.411 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:34.391 7f5acffad700 1 --2- 172.21.2.112:0/1934561119 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5ac80d1c70 0x7f5ac80cef60 secure :-1 s=READY pgs=3186 cs=0 l=1 rev1=1 rx=0x7f5ac4006650 tx=0x7f5ac4009a80).stop 2021-02-10T11:48:34.411 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:34.392 7f5acffad700 1 -- 172.21.2.112:0/1934561119 shutdown_connections 2021-02-10T11:48:34.411 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:34.392 7f5acffad700 1 --2- 172.21.2.112:0/1934561119 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5ac80d1c70 0x7f5ac80cef60 unknown :-1 s=CLOSED pgs=3186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:34.411 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:34.392 7f5acffad700 1 -- 172.21.2.112:0/1934561119 >> 172.21.2.112:0/1934561119 conn(0x7f5ac80ce420 msgr2=0x7f5ac80d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:34.411 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:34.392 7f5acffad700 1 -- 172.21.2.112:0/1934561119 shutdown_connections 2021-02-10T11:48:34.411 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:34.392 7f5acffad700 1 -- 172.21.2.112:0/1934561119 wait complete. 2021-02-10T11:48:34.413 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:34.407 7f5acffad700 1 Processor -- start 2021-02-10T11:48:34.413 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:34.407 7f5acffad700 1 -- start start 2021-02-10T11:48:34.413 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:34.407 7f5acffad700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5ac80c0740 0x7f5ac81c5580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:34.413 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:34.407 7f5acffad700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5ac81c5ba0 0x7f5ac80a5c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:34.414 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:34.407 7f5acffad700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5ac80b6e60 0x7f5ac80a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:34.414 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:34.407 7f5acffad700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ac80a6880 con 0x7f5ac80b6e60 2021-02-10T11:48:34.414 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:34.407 7f5acffad700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ac80a6aa0 con 0x7f5ac80c0740 2021-02-10T11:48:34.414 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:34.407 7f5acffad700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ac80a6cc0 con 0x7f5ac81c5ba0 2021-02-10T11:48:34.414 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:34.407 7f5ace5ca700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5ac80b6e60 0x7f5ac80a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:34.414 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:34.407 7f5ace5ca700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5ac80b6e60 0x7f5ac80a6260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:33102/0 (socket says 172.21.2.112:33102) 2021-02-10T11:48:34.414 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:34.407 7f5ace5ca700 1 -- 172.21.2.112:0/4099740162 learned_addr learned my addr 172.21.2.112:0/4099740162 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:48:34.414 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:34.408 7f5acddc9700 1 --2- 172.21.2.112:0/4099740162 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5ac80c0740 0x7f5ac81c5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:34.414 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:34.408 7f5acd5c8700 1 --2- 172.21.2.112:0/4099740162 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5ac81c5ba0 0x7f5ac80a5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:34.414 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:34.408 7f5ace5ca700 1 -- 172.21.2.112:0/4099740162 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5ac81c5ba0 msgr2=0x7f5ac80a5c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:34.414 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:34.408 7f5ace5ca700 1 --2- 172.21.2.112:0/4099740162 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5ac81c5ba0 0x7f5ac80a5c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:34.414 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:34.408 7f5ace5ca700 1 -- 172.21.2.112:0/4099740162 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5ac80c0740 msgr2=0x7f5ac81c5580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:34.415 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:34.408 7f5ace5ca700 1 --2- 172.21.2.112:0/4099740162 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5ac80c0740 0x7f5ac81c5580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:34.415 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:34.408 7f5ace5ca700 1 -- 172.21.2.112:0/4099740162 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ac80a6ee0 con 0x7f5ac80b6e60 2021-02-10T11:48:34.415 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:34.408 7f5acddc9700 1 --2- 172.21.2.112:0/4099740162 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5ac80c0740 0x7f5ac81c5580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:48:34.415 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:34.408 7f5ace5ca700 1 --2- 172.21.2.112:0/4099740162 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5ac80b6e60 0x7f5ac80a6260 secure :-1 s=READY pgs=3187 cs=0 l=1 rev1=1 rx=0x7f5ac0009940 tx=0x7f5ac000bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:34.415 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:34.408 7f5aba7fc700 1 -- 172.21.2.112:0/4099740162 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5ac0000f90 con 0x7f5ac80b6e60 2021-02-10T11:48:34.415 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:34.408 7f5aba7fc700 1 -- 172.21.2.112:0/4099740162 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ac000c2f0 con 0x7f5ac80b6e60 2021-02-10T11:48:34.415 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:34.408 7f5aba7fc700 1 -- 172.21.2.112:0/4099740162 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5ac000c750 con 0x7f5ac80b6e60 2021-02-10T11:48:34.415 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:34.408 7f5acffad700 1 -- 172.21.2.112:0/4099740162 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ac80a6ee0 con 0x7f5ac80b6e60 2021-02-10T11:48:34.415 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:34.408 7f5acffad700 1 -- 172.21.2.112:0/4099740162 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ac80a6ee0 con 0x7f5ac80b6e60 2021-02-10T11:48:34.415 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:34.409 7f5aba7fc700 1 -- 172.21.2.112:0/4099740162 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5ac0001220 con 0x7f5ac80b6e60 2021-02-10T11:48:34.415 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:34.409 7f5aba7fc700 1 --2- 172.21.2.112:0/4099740162 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5ab00186c0 0x7f5ab001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:34.415 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:34.409 7f5aba7fc700 1 -- 172.21.2.112:0/4099740162 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f5ac0032b80 con 0x7f5ac80b6e60 2021-02-10T11:48:34.415 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:34.409 7f5acddc9700 1 --2- 172.21.2.112:0/4099740162 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5ab00186c0 0x7f5ab001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:34.415 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:34.410 7f5acddc9700 1 --2- 172.21.2.112:0/4099740162 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5ab00186c0 0x7f5ab001abc0 crc :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:34.443 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:34.440 7f5acffad700 1 -- 172.21.2.112:0/4099740162 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ac80807d0 con 0x7f5ac80b6e60 2021-02-10T11:48:34.445 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:34.442 7f5aba7fc700 1 -- 172.21.2.112:0/4099740162 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5ac000be30 con 0x7f5ac80b6e60 2021-02-10T11:48:34.597 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:34.593 7f5acffad700 1 -- 172.21.2.112:0/4099740162 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f5ac8088070 con 0x7f5ac80b6e60 2021-02-10T11:48:34.597 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:34.593 7f5aba7fc700 1 -- 172.21.2.112:0/4099740162 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v46)=0 v46) v1 ==== 122+0+0 (secure 0 0 0) 0x7f5ac0001220 con 0x7f5ac80b6e60 2021-02-10T11:48:34.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:34.596 7f5acffad700 1 -- 172.21.2.112:0/4099740162 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5ab00186c0 msgr2=0x7f5ab001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:34.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:34.596 7f5acffad700 1 --2- 172.21.2.112:0/4099740162 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5ab00186c0 0x7f5ab001abc0 crc :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:34.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:34.596 7f5acffad700 1 -- 172.21.2.112:0/4099740162 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5ac80b6e60 msgr2=0x7f5ac80a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:34.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:34.596 7f5acffad700 1 --2- 172.21.2.112:0/4099740162 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5ac80b6e60 0x7f5ac80a6260 secure :-1 s=READY pgs=3187 cs=0 l=1 rev1=1 rx=0x7f5ac0009940 tx=0x7f5ac000bcf0).stop 2021-02-10T11:48:34.600 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:34.596 7f5acffad700 1 -- 172.21.2.112:0/4099740162 shutdown_connections 2021-02-10T11:48:34.600 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:34.596 7f5acffad700 1 --2- 172.21.2.112:0/4099740162 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5ab00186c0 0x7f5ab001abc0 unknown :-1 s=CLOSED pgs=1433 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:34.600 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:34.596 7f5acffad700 1 --2- 172.21.2.112:0/4099740162 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5ac80b6e60 0x7f5ac80a6260 unknown :-1 s=CLOSED pgs=3187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:34.600 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:34.596 7f5acffad700 1 --2- 172.21.2.112:0/4099740162 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5ac81c5ba0 0x7f5ac80a5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:34.600 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:34.596 7f5acffad700 1 --2- 172.21.2.112:0/4099740162 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5ac80c0740 0x7f5ac81c5580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:34.600 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:34.596 7f5acffad700 1 -- 172.21.2.112:0/4099740162 >> 172.21.2.112:0/4099740162 conn(0x7f5ac80c4120 msgr2=0x7f5ac80ffd60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:34.600 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:34.596 7f5acffad700 1 -- 172.21.2.112:0/4099740162 shutdown_connections 2021-02-10T11:48:34.600 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:34.596 7f5acffad700 1 -- 172.21.2.112:0/4099740162 wait complete. 2021-02-10T11:48:34.608 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolumegroup create cephfs group_0000000000578561 2021-02-10T11:48:35.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.021 7fda5ad55700 1 Processor -- start 2021-02-10T11:48:35.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.021 7fda5ad55700 1 -- start start 2021-02-10T11:48:35.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.021 7fda5ad55700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fda540d1870 0x7fda541b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:35.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.021 7fda5ad55700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fda541b0ae0 0x7fda54108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:35.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.021 7fda5ad55700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fda540d1c70 0x7fda54108f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:35.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.021 7fda5ad55700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda540fe1c0 con 0x7fda540d1870 2021-02-10T11:48:35.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.021 7fda5ad55700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda54109a60 con 0x7fda540d1c70 2021-02-10T11:48:35.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.021 7fda5ad55700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda54088070 con 0x7fda541b0ae0 2021-02-10T11:48:35.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.021 7fda58b71700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fda540d1870 0x7fda541b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:35.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.021 7fda58b71700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fda540d1870 0x7fda541b04c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:33106/0 (socket says 172.21.2.112:33106) 2021-02-10T11:48:35.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.021 7fda58b71700 1 -- 172.21.2.112:0/1444090276 learned_addr learned my addr 172.21.2.112:0/1444090276 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:48:35.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.021 7fda53fff700 1 --2- 172.21.2.112:0/1444090276 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fda541b0ae0 0x7fda54108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:35.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.022 7fda59372700 1 --2- 172.21.2.112:0/1444090276 >> v2:172.21.2.120:3300/0 conn(0x7fda540d1c70 0x7fda54108f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:35.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.022 7fda58b71700 1 -- 172.21.2.112:0/1444090276 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fda541b0ae0 msgr2=0x7fda54108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:35.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.022 7fda58b71700 1 --2- 172.21.2.112:0/1444090276 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fda541b0ae0 0x7fda54108940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:35.042 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.022 7fda58b71700 1 -- 172.21.2.112:0/1444090276 >> v2:172.21.2.120:3300/0 conn(0x7fda540d1c70 msgr2=0x7fda54108f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:35.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.022 7fda58b71700 1 --2- 172.21.2.112:0/1444090276 >> v2:172.21.2.120:3300/0 conn(0x7fda540d1c70 0x7fda54108f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:35.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.022 7fda58b71700 1 -- 172.21.2.112:0/1444090276 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda54109580 con 0x7fda540d1870 2021-02-10T11:48:35.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.022 7fda58b71700 1 --2- 172.21.2.112:0/1444090276 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fda540d1870 0x7fda541b04c0 secure :-1 s=READY pgs=3188 cs=0 l=1 rev1=1 rx=0x7fda3c006700 tx=0x7fda3c009b80).ready entity=mon.0 client_cookie=f5f92354171c00b9 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:35.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.022 7fda537fe700 1 -- 172.21.2.112:0/1444090276 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fda3c00c0c0 con 0x7fda540d1870 2021-02-10T11:48:35.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.022 7fda537fe700 1 -- 172.21.2.112:0/1444090276 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda3c00c400 con 0x7fda540d1870 2021-02-10T11:48:35.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.022 7fda537fe700 1 -- 172.21.2.112:0/1444090276 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fda3c00c920 con 0x7fda540d1870 2021-02-10T11:48:35.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.022 7fda5ad55700 1 -- 172.21.2.112:0/1444090276 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fda540d1870 msgr2=0x7fda541b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:35.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.022 7fda5ad55700 1 --2- 172.21.2.112:0/1444090276 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fda540d1870 0x7fda541b04c0 secure :-1 s=READY pgs=3188 cs=0 l=1 rev1=1 rx=0x7fda3c006700 tx=0x7fda3c009b80).stop 2021-02-10T11:48:35.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.023 7fda5ad55700 1 -- 172.21.2.112:0/1444090276 shutdown_connections 2021-02-10T11:48:35.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.023 7fda5ad55700 1 --2- 172.21.2.112:0/1444090276 >> v2:172.21.2.120:3300/0 conn(0x7fda540d1c70 0x7fda54108f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:35.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.023 7fda5ad55700 1 --2- 172.21.2.112:0/1444090276 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fda541b0ae0 0x7fda54108940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:35.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.023 7fda5ad55700 1 --2- 172.21.2.112:0/1444090276 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fda540d1870 0x7fda541b04c0 unknown :-1 s=CLOSED pgs=3188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:35.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.023 7fda5ad55700 1 -- 172.21.2.112:0/1444090276 >> 172.21.2.112:0/1444090276 conn(0x7fda540ce420 msgr2=0x7fda540d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:35.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.023 7fda5ad55700 1 -- 172.21.2.112:0/1444090276 shutdown_connections 2021-02-10T11:48:35.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.023 7fda5ad55700 1 -- 172.21.2.112:0/1444090276 wait complete. 2021-02-10T11:48:35.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.038 7fda5ad55700 1 Processor -- start 2021-02-10T11:48:35.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.038 7fda5ad55700 1 -- start start 2021-02-10T11:48:35.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.038 7fda5ad55700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fda540af3b0 0x7fda541a7480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:35.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.038 7fda5ad55700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fda541a7aa0 0x7fda54095de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:35.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.038 7fda5ad55700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fda541a7ea0 0x7fda54096400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:35.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.038 7fda5ad55700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda541a82a0 con 0x7fda541a7aa0 2021-02-10T11:48:35.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.038 7fda5ad55700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda540bfc80 con 0x7fda541a7ea0 2021-02-10T11:48:35.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.038 7fda5ad55700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda540bf840 con 0x7fda540af3b0 2021-02-10T11:48:35.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.039 7fda53fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fda541a7aa0 0x7fda54095de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:35.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.039 7fda53fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fda541a7aa0 0x7fda54095de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:33112/0 (socket says 172.21.2.112:33112) 2021-02-10T11:48:35.046 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.039 7fda53fff700 1 -- 172.21.2.112:0/4217735013 learned_addr learned my addr 172.21.2.112:0/4217735013 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:48:35.047 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.039 7fda59372700 1 --2- 172.21.2.112:0/4217735013 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fda541a7ea0 0x7fda54096400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:35.047 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.039 7fda58b71700 1 --2- 172.21.2.112:0/4217735013 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fda540af3b0 0x7fda541a7480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:35.047 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.039 7fda53fff700 1 -- 172.21.2.112:0/4217735013 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fda540af3b0 msgr2=0x7fda541a7480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:35.047 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.039 7fda53fff700 1 --2- 172.21.2.112:0/4217735013 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fda540af3b0 0x7fda541a7480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:35.047 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.039 7fda53fff700 1 -- 172.21.2.112:0/4217735013 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fda541a7ea0 msgr2=0x7fda54096400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:35.047 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.039 7fda53fff700 1 --2- 172.21.2.112:0/4217735013 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fda541a7ea0 0x7fda54096400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:35.047 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.039 7fda53fff700 1 -- 172.21.2.112:0/4217735013 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda54096a20 con 0x7fda541a7aa0 2021-02-10T11:48:35.047 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.039 7fda58b71700 1 --2- 172.21.2.112:0/4217735013 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fda540af3b0 0x7fda541a7480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:48:35.047 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.039 7fda59372700 1 --2- 172.21.2.112:0/4217735013 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fda541a7ea0 0x7fda54096400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:48:35.047 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.039 7fda53fff700 1 --2- 172.21.2.112:0/4217735013 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fda541a7aa0 0x7fda54095de0 secure :-1 s=READY pgs=3189 cs=0 l=1 rev1=1 rx=0x7fda440099e0 tx=0x7fda4400bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:35.047 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.039 7fda517fa700 1 -- 172.21.2.112:0/4217735013 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fda4400c2f0 con 0x7fda541a7aa0 2021-02-10T11:48:35.047 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.040 7fda517fa700 1 -- 172.21.2.112:0/4217735013 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda4400c630 con 0x7fda541a7aa0 2021-02-10T11:48:35.048 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.040 7fda5ad55700 1 -- 172.21.2.112:0/4217735013 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda54096a20 con 0x7fda541a7aa0 2021-02-10T11:48:35.048 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.040 7fda517fa700 1 -- 172.21.2.112:0/4217735013 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fda4400cb10 con 0x7fda541a7aa0 2021-02-10T11:48:35.048 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.040 7fda5ad55700 1 -- 172.21.2.112:0/4217735013 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda540c01d0 con 0x7fda541a7aa0 2021-02-10T11:48:35.048 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.040 7fda517fa700 1 -- 172.21.2.112:0/4217735013 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fda4400a020 con 0x7fda541a7aa0 2021-02-10T11:48:35.048 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.040 7fda517fa700 1 --2- 172.21.2.112:0/4217735013 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fda380186c0 0x7fda3801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:35.048 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.040 7fda517fa700 1 -- 172.21.2.112:0/4217735013 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fda44032750 con 0x7fda541a7aa0 2021-02-10T11:48:35.048 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.041 7fda58b71700 1 --2- 172.21.2.112:0/4217735013 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fda380186c0 0x7fda3801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:35.048 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.041 7fda58b71700 1 --2- 172.21.2.112:0/4217735013 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fda380186c0 0x7fda3801abc0 crc :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:35.074 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.071 7fda5ad55700 1 -- 172.21.2.112:0/4217735013 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda54082a70 con 0x7fda541a7aa0 2021-02-10T11:48:35.077 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.074 7fda517fa700 1 -- 172.21.2.112:0/4217735013 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fda44000ce0 con 0x7fda541a7aa0 2021-02-10T11:48:35.205 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.201 7fda5ad55700 1 -- 172.21.2.112:0/4217735013 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "target": ["mgr", ""], "group_name": "group_0000000000578561"}) v1 -- 0x7fda540bfa30 con 0x7fda380186c0 2021-02-10T11:48:35.210 INFO:tasks.ceph.mgr.x.gibba020.stderr:2021-02-10 11:48:35.206 7f6382eee700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:48:35.210 INFO:tasks.ceph.mgr.x.gibba020.stderr:2021-02-10 11:48:35.206 7f6382eee700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:48:35.210 INFO:tasks.ceph.mgr.x.gibba020.stderr:2021-02-10 11:48:35.206 7f6382eee700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:48:35.210 INFO:tasks.ceph.mgr.x.gibba020.stderr:2021-02-10 11:48:35.206 7f6382eee700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:48:35.210 INFO:tasks.ceph.mgr.x.gibba020.stderr:2021-02-10 11:48:35.206 7f6382eee700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:48:35.223 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.220 7fda517fa700 1 -- 172.21.2.112:0/4217735013 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fda3c0009d0 con 0x7fda380186c0 2021-02-10T11:48:35.238 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.235 7fda5ad55700 1 -- 172.21.2.112:0/4217735013 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fda380186c0 msgr2=0x7fda3801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:35.238 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.235 7fda5ad55700 1 --2- 172.21.2.112:0/4217735013 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fda380186c0 0x7fda3801abc0 crc :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:35.239 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.235 7fda5ad55700 1 -- 172.21.2.112:0/4217735013 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fda541a7aa0 msgr2=0x7fda54095de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:35.239 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.235 7fda5ad55700 1 --2- 172.21.2.112:0/4217735013 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fda541a7aa0 0x7fda54095de0 secure :-1 s=READY pgs=3189 cs=0 l=1 rev1=1 rx=0x7fda440099e0 tx=0x7fda4400bd40).stop 2021-02-10T11:48:35.239 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.235 7fda5ad55700 1 -- 172.21.2.112:0/4217735013 shutdown_connections 2021-02-10T11:48:35.239 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.235 7fda5ad55700 1 --2- 172.21.2.112:0/4217735013 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fda380186c0 0x7fda3801abc0 unknown :-1 s=CLOSED pgs=1434 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:35.239 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.235 7fda5ad55700 1 --2- 172.21.2.112:0/4217735013 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fda541a7ea0 0x7fda54096400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:35.239 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.235 7fda5ad55700 1 --2- 172.21.2.112:0/4217735013 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fda541a7aa0 0x7fda54095de0 unknown :-1 s=CLOSED pgs=3189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:35.240 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.235 7fda5ad55700 1 --2- 172.21.2.112:0/4217735013 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fda540af3b0 0x7fda541a7480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:35.240 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.235 7fda5ad55700 1 -- 172.21.2.112:0/4217735013 >> 172.21.2.112:0/4217735013 conn(0x7fda540b0090 msgr2=0x7fda540a3690 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:35.240 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.236 7fda5ad55700 1 -- 172.21.2.112:0/4217735013 shutdown_connections 2021-02-10T11:48:35.240 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.236 7fda5ad55700 1 -- 172.21.2.112:0/4217735013 wait complete. 2021-02-10T11:48:35.247 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000563415 --group_name group_0000000000578561 2021-02-10T11:48:35.680 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.660 7f9b3015b700 1 Processor -- start 2021-02-10T11:48:35.680 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.660 7f9b3015b700 1 -- start start 2021-02-10T11:48:35.680 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.661 7f9b3015b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9b280d1880 0x7f9b281b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:35.681 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.661 7f9b3015b700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f9b280cff80 0x7f9b28109140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:35.681 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.661 7f9b3015b700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b28109a70 con 0x7f9b281b0af0 2021-02-10T11:48:35.681 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.661 7f9b3015b700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b28109760 con 0x7f9b280cff80 2021-02-10T11:48:35.681 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.661 7f9b3015b700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b28088070 con 0x7f9b280d1880 2021-02-10T11:48:35.681 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.661 7f9b2e778700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f9b280cff80 0x7f9b28109140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:35.681 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.661 7f9b2e778700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f9b280cff80 0x7f9b28109140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:33120/0 (socket says 172.21.2.112:33120) 2021-02-10T11:48:35.681 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.661 7f9b2e778700 1 -- 172.21.2.112:0/4134369194 learned_addr learned my addr 172.21.2.112:0/4134369194 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:48:35.681 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.661 7f9b2df77700 1 --2- 172.21.2.112:0/4134369194 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9b280d1880 0x7f9b281b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:35.681 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.661 7f9b2e778700 1 -- 172.21.2.112:0/4134369194 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9b280d1880 msgr2=0x7f9b281b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:35.681 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.661 7f9b2e778700 1 --2- 172.21.2.112:0/4134369194 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9b280d1880 0x7f9b281b04d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:35.681 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.661 7f9b2e778700 1 -- 172.21.2.112:0/4134369194 >> v1:172.21.2.112:6789/0 conn(0x7f9b281b0af0 legacy=0x7f9b28108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:35.681 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.661 7f9b2e778700 1 -- 172.21.2.112:0/4134369194 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b28088260 con 0x7f9b280cff80 2021-02-10T11:48:35.681 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.662 7f9b2e778700 1 --2- 172.21.2.112:0/4134369194 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9b280cff80 0x7f9b28109140 secure :-1 s=READY pgs=3191 cs=0 l=1 rev1=1 rx=0x7f9b20006f00 tx=0x7f9b20009f10).ready entity=mon.0 client_cookie=e7463611c7fa54ec server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:35.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.662 7f9b2cf75700 1 -- 172.21.2.112:0/4134369194 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9b2000b3b0 con 0x7f9b280cff80 2021-02-10T11:48:35.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.662 7f9b2cf75700 1 -- 172.21.2.112:0/4134369194 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b2000b730 con 0x7f9b280cff80 2021-02-10T11:48:35.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.662 7f9b2cf75700 1 -- 172.21.2.112:0/4134369194 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9b2000bc10 con 0x7f9b280cff80 2021-02-10T11:48:35.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.662 7f9b3015b700 1 -- 172.21.2.112:0/4134369194 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9b280cff80 msgr2=0x7f9b28109140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:35.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.662 7f9b3015b700 1 --2- 172.21.2.112:0/4134369194 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9b280cff80 0x7f9b28109140 secure :-1 s=READY pgs=3191 cs=0 l=1 rev1=1 rx=0x7f9b20006f00 tx=0x7f9b20009f10).stop 2021-02-10T11:48:35.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.662 7f9b3015b700 1 -- 172.21.2.112:0/4134369194 shutdown_connections 2021-02-10T11:48:35.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.662 7f9b3015b700 1 --2- 172.21.2.112:0/4134369194 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9b280cff80 0x7f9b28109140 unknown :-1 s=CLOSED pgs=3191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:35.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.662 7f9b3015b700 1 --2- 172.21.2.112:0/4134369194 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9b280d1880 0x7f9b281b04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:35.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.662 7f9b3015b700 1 -- 172.21.2.112:0/4134369194 >> 172.21.2.112:0/4134369194 conn(0x7f9b280ce430 msgr2=0x7f9b280d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:35.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.663 7f9b3015b700 1 -- 172.21.2.112:0/4134369194 shutdown_connections 2021-02-10T11:48:35.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.663 7f9b3015b700 1 -- 172.21.2.112:0/4134369194 wait complete. 2021-02-10T11:48:35.684 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.677 7f9b3015b700 1 Processor -- start 2021-02-10T11:48:35.684 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.677 7f9b3015b700 1 -- start start 2021-02-10T11:48:35.684 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.678 7f9b3015b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9b2809ab20 0x7f9b2809af20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:35.684 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.678 7f9b3015b700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9b280c0d10 0x7f9b280c1110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:35.684 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.678 7f9b3015b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9b280c1730 0x7f9b2809bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:35.684 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.678 7f9b3015b700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b2809b540 con 0x7f9b280c1730 2021-02-10T11:48:35.685 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.678 7f9b3015b700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b280c1b30 con 0x7f9b280c0d10 2021-02-10T11:48:35.685 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.678 7f9b3015b700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b280c1d50 con 0x7f9b2809ab20 2021-02-10T11:48:35.685 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.678 7f9b2e778700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9b280c1730 0x7f9b2809bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:35.685 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.678 7f9b2e778700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9b280c1730 0x7f9b2809bde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:33130/0 (socket says 172.21.2.112:33130) 2021-02-10T11:48:35.685 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.678 7f9b2e778700 1 -- 172.21.2.112:0/750624162 learned_addr learned my addr 172.21.2.112:0/750624162 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:48:35.685 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.678 7f9b2d776700 1 --2- 172.21.2.112:0/750624162 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9b280c0d10 0x7f9b280c1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:35.685 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.678 7f9b2df77700 1 --2- 172.21.2.112:0/750624162 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9b2809ab20 0x7f9b2809af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:35.685 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.678 7f9b2e778700 1 -- 172.21.2.112:0/750624162 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9b2809ab20 msgr2=0x7f9b2809af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:35.685 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.678 7f9b2e778700 1 --2- 172.21.2.112:0/750624162 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9b2809ab20 0x7f9b2809af20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:35.685 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.678 7f9b2e778700 1 -- 172.21.2.112:0/750624162 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9b280c0d10 msgr2=0x7f9b280c1110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:35.685 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.678 7f9b2e778700 1 --2- 172.21.2.112:0/750624162 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9b280c0d10 0x7f9b280c1110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:35.685 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.678 7f9b2e778700 1 -- 172.21.2.112:0/750624162 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b2809c400 con 0x7f9b280c1730 2021-02-10T11:48:35.685 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.679 7f9b2d776700 1 --2- 172.21.2.112:0/750624162 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9b280c0d10 0x7f9b280c1110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:48:35.686 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.679 7f9b2df77700 1 --2- 172.21.2.112:0/750624162 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9b2809ab20 0x7f9b2809af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:48:35.686 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.679 7f9b2e778700 1 --2- 172.21.2.112:0/750624162 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9b280c1730 0x7f9b2809bde0 secure :-1 s=READY pgs=3192 cs=0 l=1 rev1=1 rx=0x7f9b200099f0 tx=0x7f9b2000bcc0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:35.686 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.679 7f9b1a7fc700 1 -- 172.21.2.112:0/750624162 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9b2000be00 con 0x7f9b280c1730 2021-02-10T11:48:35.686 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.679 7f9b1a7fc700 1 -- 172.21.2.112:0/750624162 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b20005820 con 0x7f9b280c1730 2021-02-10T11:48:35.686 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.679 7f9b1a7fc700 1 -- 172.21.2.112:0/750624162 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9b20005d00 con 0x7f9b280c1730 2021-02-10T11:48:35.686 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.679 7f9b3015b700 1 -- 172.21.2.112:0/750624162 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b2809c400 con 0x7f9b280c1730 2021-02-10T11:48:35.686 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.679 7f9b3015b700 1 -- 172.21.2.112:0/750624162 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b2809c400 con 0x7f9b280c1730 2021-02-10T11:48:35.686 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.680 7f9b1a7fc700 1 -- 172.21.2.112:0/750624162 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9b20005a00 con 0x7f9b280c1730 2021-02-10T11:48:35.686 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.680 7f9b1a7fc700 1 --2- 172.21.2.112:0/750624162 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9b100186c0 0x7f9b1001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:35.686 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.680 7f9b1a7fc700 1 -- 172.21.2.112:0/750624162 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f9b200328a0 con 0x7f9b280c1730 2021-02-10T11:48:35.686 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.680 7f9b2df77700 1 --2- 172.21.2.112:0/750624162 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9b100186c0 0x7f9b1001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:35.686 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.681 7f9b2df77700 1 --2- 172.21.2.112:0/750624162 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9b100186c0 0x7f9b1001abc0 crc :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:35.714 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.711 7f9b3015b700 1 -- 172.21.2.112:0/750624162 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b2809b8b0 con 0x7f9b280c1730 2021-02-10T11:48:35.716 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.713 7f9b1a7fc700 1 -- 172.21.2.112:0/750624162 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9b2000be00 con 0x7f9b280c1730 2021-02-10T11:48:35.844 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.840 7f9b3015b700 1 -- 172.21.2.112:0/750624162 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000563415", "target": ["mgr", ""], "group_name": "group_0000000000578561"}) v1 -- 0x7f9b2809b8b0 con 0x7f9b100186c0 2021-02-10T11:48:35.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.867 7f9b1a7fc700 1 -- 172.21.2.112:0/750624162 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f9b1c006b10 con 0x7f9b100186c0 2021-02-10T11:48:35.878 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.874 7f9b3015b700 1 -- 172.21.2.112:0/750624162 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9b100186c0 msgr2=0x7f9b1001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:35.878 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.874 7f9b3015b700 1 --2- 172.21.2.112:0/750624162 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9b100186c0 0x7f9b1001abc0 crc :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:35.878 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.874 7f9b3015b700 1 -- 172.21.2.112:0/750624162 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9b280c1730 msgr2=0x7f9b2809bde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:35.878 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.874 7f9b3015b700 1 --2- 172.21.2.112:0/750624162 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9b280c1730 0x7f9b2809bde0 secure :-1 s=READY pgs=3192 cs=0 l=1 rev1=1 rx=0x7f9b200099f0 tx=0x7f9b2000bcc0).stop 2021-02-10T11:48:35.878 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.874 7f9b3015b700 1 -- 172.21.2.112:0/750624162 shutdown_connections 2021-02-10T11:48:35.878 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.874 7f9b3015b700 1 --2- 172.21.2.112:0/750624162 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9b100186c0 0x7f9b1001abc0 unknown :-1 s=CLOSED pgs=1435 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:35.878 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.874 7f9b3015b700 1 --2- 172.21.2.112:0/750624162 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9b280c0d10 0x7f9b280c1110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:35.879 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.874 7f9b3015b700 1 --2- 172.21.2.112:0/750624162 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9b280c1730 0x7f9b2809bde0 unknown :-1 s=CLOSED pgs=3192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:35.879 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.874 7f9b3015b700 1 --2- 172.21.2.112:0/750624162 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9b2809ab20 0x7f9b2809af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:35.879 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.874 7f9b3015b700 1 -- 172.21.2.112:0/750624162 >> 172.21.2.112:0/750624162 conn(0x7f9b280c20f0 msgr2=0x7f9b2809e410 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:35.879 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.874 7f9b3015b700 1 -- 172.21.2.112:0/750624162 shutdown_connections 2021-02-10T11:48:35.879 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:35.875 7f9b3015b700 1 -- 172.21.2.112:0/750624162 wait complete. 2021-02-10T11:48:35.886 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume authorize cephfs subvolume_0000000000563415 alice --group_name group_0000000000578561 2021-02-10T11:48:36.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.301 7f04c2c36700 1 Processor -- start 2021-02-10T11:48:36.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.301 7f04c2c36700 1 -- start start 2021-02-10T11:48:36.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.301 7f04c2c36700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f04bc0dce80 0x7f04bc0d14f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:36.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.301 7f04c2c36700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f04bc10b2c0 0x7f04bc0eb620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:36.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.301 7f04c2c36700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04bc10a550 con 0x7f04bc0eb220 2021-02-10T11:48:36.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.301 7f04c2c36700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04bc0d1fa0 con 0x7f04bc10b2c0 2021-02-10T11:48:36.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.301 7f04c2c36700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04bc0881d0 con 0x7f04bc0dce80 2021-02-10T11:48:36.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.302 7f04b3fff700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f04bc0eb220 0x7f04bc10aad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:60832/0 (socket says 172.21.2.112:60832) 2021-02-10T11:48:36.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.302 7f04b3fff700 1 -- 172.21.2.112:0/1005276244 learned_addr learned my addr 172.21.2.112:0/1005276244 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:48:36.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.302 7f04c0a52700 1 --2- 172.21.2.112:0/1005276244 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f04bc0dce80 0x7f04bc0d14f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:36.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.302 7f04c1253700 1 --2- 172.21.2.112:0/1005276244 >> v2:172.21.2.120:3300/0 conn(0x7f04bc10b2c0 0x7f04bc0eb620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:36.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.302 7f04c0a52700 1 -- 172.21.2.112:0/1005276244 >> v2:172.21.2.120:3300/0 conn(0x7f04bc10b2c0 msgr2=0x7f04bc0eb620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:36.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.302 7f04c0a52700 1 --2- 172.21.2.112:0/1005276244 >> v2:172.21.2.120:3300/0 conn(0x7f04bc10b2c0 0x7f04bc0eb620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:36.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.302 7f04c0a52700 1 -- 172.21.2.112:0/1005276244 >> v1:172.21.2.112:6789/0 conn(0x7f04bc0eb220 legacy=0x7f04bc10aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:36.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.302 7f04b37fe700 1 -- 172.21.2.112:0/1005276244 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2503997047 0 0) 0x7f04b80009b0 con 0x7f04bc0eb220 2021-02-10T11:48:36.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.302 7f04c0a52700 1 -- 172.21.2.112:0/1005276244 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04bc0dd280 con 0x7f04bc0dce80 2021-02-10T11:48:36.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.302 7f04c0a52700 1 --2- 172.21.2.112:0/1005276244 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f04bc0dce80 0x7f04bc0d14f0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 rx=0x7f04ac006650 tx=0x7f04ac009a80).ready entity=mon.2 client_cookie=f8614f2e97a97fda server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:36.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.302 7f04b37fe700 1 -- 172.21.2.112:0/1005276244 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f04ac0009b0 con 0x7f04bc0dce80 2021-02-10T11:48:36.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.302 7f04b37fe700 1 -- 172.21.2.112:0/1005276244 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04ac006740 con 0x7f04bc0dce80 2021-02-10T11:48:36.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.302 7f04b37fe700 1 -- 172.21.2.112:0/1005276244 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f04ac00c0c0 con 0x7f04bc0dce80 2021-02-10T11:48:36.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.302 7f04c2c36700 1 -- 172.21.2.112:0/1005276244 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f04bc0dce80 msgr2=0x7f04bc0d14f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:36.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.302 7f04c2c36700 1 --2- 172.21.2.112:0/1005276244 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f04bc0dce80 0x7f04bc0d14f0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 rx=0x7f04ac006650 tx=0x7f04ac009a80).stop 2021-02-10T11:48:36.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.303 7f04c2c36700 1 -- 172.21.2.112:0/1005276244 shutdown_connections 2021-02-10T11:48:36.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.303 7f04c2c36700 1 --2- 172.21.2.112:0/1005276244 >> v2:172.21.2.120:3300/0 conn(0x7f04bc10b2c0 0x7f04bc0eb620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:36.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.303 7f04c2c36700 1 --2- 172.21.2.112:0/1005276244 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f04bc0dce80 0x7f04bc0d14f0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:36.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.303 7f04c2c36700 1 -- 172.21.2.112:0/1005276244 >> 172.21.2.112:0/1005276244 conn(0x7f04bc0ea630 msgr2=0x7f04bc0d0800 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:36.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.303 7f04c2c36700 1 -- 172.21.2.112:0/1005276244 shutdown_connections 2021-02-10T11:48:36.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.303 7f04c2c36700 1 -- 172.21.2.112:0/1005276244 wait complete. 2021-02-10T11:48:36.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.317 7f04c2c36700 1 Processor -- start 2021-02-10T11:48:36.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.317 7f04c2c36700 1 -- start start 2021-02-10T11:48:36.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.318 7f04c2c36700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f04bc0fcf20 0x7f04bc106950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:36.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.318 7f04c2c36700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f04bc09e5f0 0x7f04bc09e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:36.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.318 7f04c2c36700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f04bc09f010 0x7f04bc1a2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:36.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.318 7f04c2c36700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04bc0fd320 con 0x7f04bc09e5f0 2021-02-10T11:48:36.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.318 7f04c2c36700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04bc1a35e0 con 0x7f04bc09f010 2021-02-10T11:48:36.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.318 7f04c2c36700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04bc1a3800 con 0x7f04bc0fcf20 2021-02-10T11:48:36.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.318 7f04b3fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f04bc09e5f0 0x7f04bc09e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:36.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.318 7f04b3fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f04bc09e5f0 0x7f04bc09e9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:33142/0 (socket says 172.21.2.112:33142) 2021-02-10T11:48:36.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.318 7f04b3fff700 1 -- 172.21.2.112:0/3718200850 learned_addr learned my addr 172.21.2.112:0/3718200850 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:48:36.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.318 7f04c1253700 1 --2- 172.21.2.112:0/3718200850 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f04bc09f010 0x7f04bc1a2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:36.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.318 7f04c0a52700 1 --2- 172.21.2.112:0/3718200850 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f04bc0fcf20 0x7f04bc106950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:36.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.319 7f04c1253700 1 -- 172.21.2.112:0/3718200850 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f04bc0fcf20 msgr2=0x7f04bc106950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:36.326 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.319 7f04c1253700 1 --2- 172.21.2.112:0/3718200850 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f04bc0fcf20 0x7f04bc106950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:36.326 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.319 7f04c1253700 1 -- 172.21.2.112:0/3718200850 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f04bc09e5f0 msgr2=0x7f04bc09e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:36.326 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.319 7f04c1253700 1 --2- 172.21.2.112:0/3718200850 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f04bc09e5f0 0x7f04bc09e9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:36.326 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.319 7f04c1253700 1 -- 172.21.2.112:0/3718200850 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04bc1a3a20 con 0x7f04bc09f010 2021-02-10T11:48:36.326 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.319 7f04c1253700 1 --2- 172.21.2.112:0/3718200850 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f04bc09f010 0x7f04bc1a2fc0 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 rx=0x7f04b400b820 tx=0x7f04b400bbc0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:36.326 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.319 7f04b17fa700 1 -- 172.21.2.112:0/3718200850 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f04b4000f90 con 0x7f04bc09f010 2021-02-10T11:48:36.326 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.319 7f04b17fa700 1 -- 172.21.2.112:0/3718200850 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04b40011f0 con 0x7f04bc09f010 2021-02-10T11:48:36.326 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.319 7f04c2c36700 1 -- 172.21.2.112:0/3718200850 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04bc1a3a20 con 0x7f04bc09f010 2021-02-10T11:48:36.326 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.319 7f04b17fa700 1 -- 172.21.2.112:0/3718200850 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f04b4001670 con 0x7f04bc09f010 2021-02-10T11:48:36.326 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.319 7f04c2c36700 1 -- 172.21.2.112:0/3718200850 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04bc1a3a20 con 0x7f04bc09f010 2021-02-10T11:48:36.326 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.319 7f04b17fa700 1 -- 172.21.2.112:0/3718200850 <== mon.1 v2:172.21.2.120:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f04b4000e80 con 0x7f04bc09f010 2021-02-10T11:48:36.326 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.320 7f04b17fa700 1 --2- 172.21.2.112:0/3718200850 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f049c018730 0x7f049c01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:36.326 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.320 7f04b17fa700 1 -- 172.21.2.112:0/3718200850 <== mon.1 v2:172.21.2.120:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f04b40328d0 con 0x7f04bc09f010 2021-02-10T11:48:36.327 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.320 7f04c0a52700 1 --2- 172.21.2.112:0/3718200850 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f049c018730 0x7f049c01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:36.327 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.320 7f04c0a52700 1 --2- 172.21.2.112:0/3718200850 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f049c018730 0x7f049c01ac10 crc :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:36.355 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.351 7f04c2c36700 1 -- 172.21.2.112:0/3718200850 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04bc0f93d0 con 0x7f04bc09f010 2021-02-10T11:48:36.357 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.354 7f04b17fa700 1 -- 172.21.2.112:0/3718200850 <== mon.1 v2:172.21.2.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f04b40066f0 con 0x7f04bc09f010 2021-02-10T11:48:36.482 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.479 7f04c2c36700 1 -- 172.21.2.112:0/3718200850 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume authorize", "vol_name": "cephfs", "sub_name": "subvolume_0000000000563415", "target": ["mgr", ""], "auth_id": "alice", "group_name": "group_0000000000578561"}) v1 -- 0x7f04bc0fd320 con 0x7f049c018730 2021-02-10T11:48:36.488 INFO:tasks.ceph.mgr.x.gibba020.stderr:Creating meta for ID alice with tenant None 2021-02-10T11:48:36.517 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.514 7f04b17fa700 1 -- 172.21.2.112:0/3718200850 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f04ac006420 con 0x7f049c018730 2021-02-10T11:48:36.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.545 7f04c2c36700 1 -- 172.21.2.112:0/3718200850 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f049c018730 msgr2=0x7f049c01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:36.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.545 7f04c2c36700 1 --2- 172.21.2.112:0/3718200850 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f049c018730 0x7f049c01ac10 crc :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:36.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.545 7f04c2c36700 1 -- 172.21.2.112:0/3718200850 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f04bc09f010 msgr2=0x7f04bc1a2fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:36.548 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.545 7f04c2c36700 1 --2- 172.21.2.112:0/3718200850 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f04bc09f010 0x7f04bc1a2fc0 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 rx=0x7f04b400b820 tx=0x7f04b400bbc0).stop 2021-02-10T11:48:36.549 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.545 7f04c2c36700 1 -- 172.21.2.112:0/3718200850 shutdown_connections 2021-02-10T11:48:36.549 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.545 7f04c2c36700 1 --2- 172.21.2.112:0/3718200850 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f049c018730 0x7f049c01ac10 unknown :-1 s=CLOSED pgs=1436 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:36.549 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.545 7f04c2c36700 1 --2- 172.21.2.112:0/3718200850 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f04bc09f010 0x7f04bc1a2fc0 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:36.549 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.545 7f04c2c36700 1 --2- 172.21.2.112:0/3718200850 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f04bc09e5f0 0x7f04bc09e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:36.549 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.545 7f04c2c36700 1 --2- 172.21.2.112:0/3718200850 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f04bc0fcf20 0x7f04bc106950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:36.549 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.545 7f04c2c36700 1 -- 172.21.2.112:0/3718200850 >> 172.21.2.112:0/3718200850 conn(0x7f04bc0b1660 msgr2=0x7f04bc0a8e10 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:36.550 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.545 7f04c2c36700 1 -- 172.21.2.112:0/3718200850 shutdown_connections 2021-02-10T11:48:36.550 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.545 7f04c2c36700 1 -- 172.21.2.112:0/3718200850 wait complete. 2021-02-10T11:48:36.556 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume authorize cephfs subvolume_0000000000563415 guest1 --group_name group_0000000000578561 2021-02-10T11:48:36.989 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.969 7fca91c0e700 1 Processor -- start 2021-02-10T11:48:36.989 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.969 7fca91c0e700 1 -- start start 2021-02-10T11:48:36.989 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.969 7fca91c0e700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fca8c0dce80 0x7fca8c0d14f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:36.989 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.969 7fca91c0e700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fca8c0eb220 0x7fca8c0eb620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:36.989 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.969 7fca91c0e700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fca8c10aad0 0x7fca8c10aed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:36.989 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.969 7fca91c0e700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca8c10b570 con 0x7fca8c0dce80 2021-02-10T11:48:36.989 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.969 7fca91c0e700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca8c10a550 con 0x7fca8c10aad0 2021-02-10T11:48:36.989 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.969 7fca91c0e700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca8c0881d0 con 0x7fca8c0eb220 2021-02-10T11:48:36.989 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.969 7fca8b7fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fca8c0dce80 0x7fca8c0d14f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:36.990 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.969 7fca8b7fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fca8c0dce80 0x7fca8c0d14f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:33148/0 (socket says 172.21.2.112:33148) 2021-02-10T11:48:36.990 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.969 7fca8b7fe700 1 -- 172.21.2.112:0/1644991161 learned_addr learned my addr 172.21.2.112:0/1644991161 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:48:36.990 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.970 7fca8b7fe700 1 -- 172.21.2.112:0/1644991161 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fca8c0eb220 msgr2=0x7fca8c0eb620 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:48:36.990 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.970 7fca8b7fe700 1 --2- 172.21.2.112:0/1644991161 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fca8c0eb220 0x7fca8c0eb620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:36.990 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.970 7fca8b7fe700 1 -- 172.21.2.112:0/1644991161 >> v2:172.21.2.120:3300/0 conn(0x7fca8c10aad0 msgr2=0x7fca8c10aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:36.990 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.970 7fca8b7fe700 1 --2- 172.21.2.112:0/1644991161 >> v2:172.21.2.120:3300/0 conn(0x7fca8c10aad0 0x7fca8c10aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:36.990 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.970 7fca8b7fe700 1 -- 172.21.2.112:0/1644991161 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca8c0d1f30 con 0x7fca8c0dce80 2021-02-10T11:48:36.990 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.970 7fca8b7fe700 1 --2- 172.21.2.112:0/1644991161 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fca8c0dce80 0x7fca8c0d14f0 secure :-1 s=READY pgs=3194 cs=0 l=1 rev1=1 rx=0x7fca7c006700 tx=0x7fca7c009b80).ready entity=mon.0 client_cookie=e2a37395a768a453 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:36.990 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.970 7fca8a7fc700 1 -- 172.21.2.112:0/1644991161 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fca7c000c90 con 0x7fca8c0dce80 2021-02-10T11:48:36.990 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.970 7fca8a7fc700 1 -- 172.21.2.112:0/1644991161 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca7c00c0c0 con 0x7fca8c0dce80 2021-02-10T11:48:36.990 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.970 7fca8a7fc700 1 -- 172.21.2.112:0/1644991161 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fca7c00c5e0 con 0x7fca8c0dce80 2021-02-10T11:48:36.991 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.970 7fca91c0e700 1 -- 172.21.2.112:0/1644991161 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fca8c0dce80 msgr2=0x7fca8c0d14f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:36.991 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.970 7fca91c0e700 1 --2- 172.21.2.112:0/1644991161 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fca8c0dce80 0x7fca8c0d14f0 secure :-1 s=READY pgs=3194 cs=0 l=1 rev1=1 rx=0x7fca7c006700 tx=0x7fca7c009b80).stop 2021-02-10T11:48:36.991 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.971 7fca91c0e700 1 -- 172.21.2.112:0/1644991161 shutdown_connections 2021-02-10T11:48:36.991 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.971 7fca91c0e700 1 --2- 172.21.2.112:0/1644991161 >> v2:172.21.2.120:3300/0 conn(0x7fca8c10aad0 0x7fca8c10aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:36.991 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.971 7fca91c0e700 1 --2- 172.21.2.112:0/1644991161 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fca8c0eb220 0x7fca8c0eb620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:36.991 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.971 7fca91c0e700 1 --2- 172.21.2.112:0/1644991161 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fca8c0dce80 0x7fca8c0d14f0 unknown :-1 s=CLOSED pgs=3194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:36.991 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.971 7fca91c0e700 1 -- 172.21.2.112:0/1644991161 >> 172.21.2.112:0/1644991161 conn(0x7fca8c0ea630 msgr2=0x7fca8c0d0800 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:36.991 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.971 7fca91c0e700 1 -- 172.21.2.112:0/1644991161 shutdown_connections 2021-02-10T11:48:36.991 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.971 7fca91c0e700 1 -- 172.21.2.112:0/1644991161 wait complete. 2021-02-10T11:48:36.993 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.986 7fca91c0e700 1 Processor -- start 2021-02-10T11:48:36.993 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.986 7fca91c0e700 1 -- start start 2021-02-10T11:48:36.993 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.986 7fca91c0e700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fca8c105920 0x7fca8c104c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:36.993 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.986 7fca91c0e700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fca8c105d20 0x7fca8c103380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:36.993 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.986 7fca91c0e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fca8c105280 0x7fca8c1039a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:36.993 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.986 7fca91c0e700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca8c105680 con 0x7fca8c105280 2021-02-10T11:48:36.993 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.986 7fca91c0e700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca8c191ad0 con 0x7fca8c105d20 2021-02-10T11:48:36.993 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.986 7fca91c0e700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca8c1046e0 con 0x7fca8c105920 2021-02-10T11:48:36.993 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.987 7fca8bfff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fca8c105280 0x7fca8c1039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:36.993 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.987 7fca8bfff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fca8c105280 0x7fca8c1039a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:33156/0 (socket says 172.21.2.112:33156) 2021-02-10T11:48:36.994 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.987 7fca8bfff700 1 -- 172.21.2.112:0/1512206732 learned_addr learned my addr 172.21.2.112:0/1512206732 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:48:36.994 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.987 7fca8bfff700 1 -- 172.21.2.112:0/1512206732 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fca8c105920 msgr2=0x7fca8c104c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:36.994 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.987 7fca8bfff700 1 --2- 172.21.2.112:0/1512206732 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fca8c105920 0x7fca8c104c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:36.994 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.987 7fca8bfff700 1 -- 172.21.2.112:0/1512206732 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fca8c105d20 msgr2=0x7fca8c103380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:36.994 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.987 7fca8bfff700 1 --2- 172.21.2.112:0/1512206732 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fca8c105d20 0x7fca8c103380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:36.994 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.987 7fca8bfff700 1 -- 172.21.2.112:0/1512206732 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca8c191630 con 0x7fca8c105280 2021-02-10T11:48:36.994 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.987 7fca8bfff700 1 --2- 172.21.2.112:0/1512206732 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fca8c105280 0x7fca8c1039a0 secure :-1 s=READY pgs=3195 cs=0 l=1 rev1=1 rx=0x7fca84009980 tx=0x7fca8400bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:36.994 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.987 7fca83fff700 1 -- 172.21.2.112:0/1512206732 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fca84003020 con 0x7fca8c105280 2021-02-10T11:48:36.994 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.987 7fca83fff700 1 -- 172.21.2.112:0/1512206732 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca84005a80 con 0x7fca8c105280 2021-02-10T11:48:36.994 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.987 7fca83fff700 1 -- 172.21.2.112:0/1512206732 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fca84006030 con 0x7fca8c105280 2021-02-10T11:48:36.994 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.987 7fca91c0e700 1 -- 172.21.2.112:0/1512206732 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca8c191630 con 0x7fca8c105280 2021-02-10T11:48:36.994 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.987 7fca91c0e700 1 -- 172.21.2.112:0/1512206732 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca8c1042b0 con 0x7fca8c105280 2021-02-10T11:48:36.995 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.988 7fca83fff700 1 -- 172.21.2.112:0/1512206732 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fca84006030 con 0x7fca8c105280 2021-02-10T11:48:36.995 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.988 7fca83fff700 1 --2- 172.21.2.112:0/1512206732 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fca700186c0 0x7fca7001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:36.995 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.988 7fca83fff700 1 -- 172.21.2.112:0/1512206732 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fca840328f0 con 0x7fca8c105280 2021-02-10T11:48:36.995 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.989 7fca8b7fe700 1 --2- 172.21.2.112:0/1512206732 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fca700186c0 0x7fca7001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:36.995 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:36.989 7fca8b7fe700 1 --2- 172.21.2.112:0/1512206732 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fca700186c0 0x7fca7001abc0 crc :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:37.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:37.019 7fca91c0e700 1 -- 172.21.2.112:0/1512206732 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca8c10a550 con 0x7fca8c105280 2021-02-10T11:48:37.025 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:37.021 7fca83fff700 1 -- 172.21.2.112:0/1512206732 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fca8402cf70 con 0x7fca8c105280 2021-02-10T11:48:37.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:37.147 7fca91c0e700 1 -- 172.21.2.112:0/1512206732 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume authorize", "vol_name": "cephfs", "sub_name": "subvolume_0000000000563415", "target": ["mgr", ""], "auth_id": "guest1", "group_name": "group_0000000000578561"}) v1 -- 0x7fca8c105680 con 0x7fca700186c0 2021-02-10T11:48:37.156 INFO:tasks.ceph.mgr.x.gibba020.stderr:Creating meta for ID guest1 with tenant None 2021-02-10T11:48:37.193 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:37.190 7fca83fff700 1 -- 172.21.2.112:0/1512206732 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fca7c000940 con 0x7fca700186c0 2021-02-10T11:48:37.216 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:37.213 7fca91c0e700 1 -- 172.21.2.112:0/1512206732 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fca700186c0 msgr2=0x7fca7001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:37.216 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:37.213 7fca91c0e700 1 --2- 172.21.2.112:0/1512206732 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fca700186c0 0x7fca7001abc0 crc :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:37.216 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:37.213 7fca91c0e700 1 -- 172.21.2.112:0/1512206732 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fca8c105280 msgr2=0x7fca8c1039a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:37.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:37.213 7fca91c0e700 1 --2- 172.21.2.112:0/1512206732 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fca8c105280 0x7fca8c1039a0 secure :-1 s=READY pgs=3195 cs=0 l=1 rev1=1 rx=0x7fca84009980 tx=0x7fca8400bcd0).stop 2021-02-10T11:48:37.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:37.213 7fca91c0e700 1 -- 172.21.2.112:0/1512206732 shutdown_connections 2021-02-10T11:48:37.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:37.213 7fca91c0e700 1 --2- 172.21.2.112:0/1512206732 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fca700186c0 0x7fca7001abc0 unknown :-1 s=CLOSED pgs=1437 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:37.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:37.213 7fca91c0e700 1 --2- 172.21.2.112:0/1512206732 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fca8c105d20 0x7fca8c103380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:37.218 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:37.213 7fca91c0e700 1 --2- 172.21.2.112:0/1512206732 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fca8c105280 0x7fca8c1039a0 unknown :-1 s=CLOSED pgs=3195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:37.218 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:37.213 7fca91c0e700 1 --2- 172.21.2.112:0/1512206732 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fca8c105920 0x7fca8c104c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:37.218 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:37.213 7fca91c0e700 1 -- 172.21.2.112:0/1512206732 >> 172.21.2.112:0/1512206732 conn(0x7fca8c19b3a0 msgr2=0x7fca8c102370 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:37.218 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:37.214 7fca91c0e700 1 -- 172.21.2.112:0/1512206732 shutdown_connections 2021-02-10T11:48:37.218 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:37.214 7fca91c0e700 1 -- 172.21.2.112:0/1512206732 wait complete. 2021-02-10T11:48:37.225 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume authorize cephfs subvolume_0000000000563415 guest2 --group_name group_0000000000578561 --access_level r 2021-02-10T11:48:37.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:37.646 7f4784854700 1 Processor -- start 2021-02-10T11:48:37.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:37.646 7f4784854700 1 -- start start 2021-02-10T11:48:37.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:37.647 7f4784854700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f47800e53a0 0x7f47801b0550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:37.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:37.647 7f4784854700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f478010b050 0x7f47801091c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:37.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:37.647 7f4784854700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4780109ac0 con 0x7f47801b0b70 2021-02-10T11:48:37.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:37.647 7f4784854700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47800fe2b0 con 0x7f478010b050 2021-02-10T11:48:37.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:37.647 7f4784854700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47800e5f80 con 0x7f47800e53a0 2021-02-10T11:48:37.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:37.647 7f477e61c700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f47800e53a0 0x7f47801b0550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:37.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:37.647 7f477ee1d700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f478010b050 0x7f47801091c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:37.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:37.647 7f477de1b700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7f47801b0b70 0x7f47801089d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:54412/0 (socket says 172.21.2.112:54412) 2021-02-10T11:48:37.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:37.648 7f477de1b700 1 -- 172.21.2.112:0/2501353935 learned_addr learned my addr 172.21.2.112:0/2501353935 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:48:37.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:37.648 7f477ee1d700 1 -- 172.21.2.112:0/2501353935 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f47800e53a0 msgr2=0x7f47801b0550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:37.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:37.648 7f477ee1d700 1 --2- 172.21.2.112:0/2501353935 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f47800e53a0 0x7f47801b0550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:37.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:37.648 7f477ee1d700 1 -- 172.21.2.112:0/2501353935 >> v1:172.21.2.120:6789/0 conn(0x7f47801b0b70 legacy=0x7f47801089d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:37.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:37.648 7f477ee1d700 1 -- 172.21.2.112:0/2501353935 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47800e57a0 con 0x7f478010b050 2021-02-10T11:48:37.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:37.648 7f477ee1d700 1 --2- 172.21.2.112:0/2501353935 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f478010b050 0x7f47801091c0 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 rx=0x7f4778006e50 tx=0x7f4778009e70).ready entity=mon.1 client_cookie=5ca29570f9796b48 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:37.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:37.648 7f477d61a700 1 -- 172.21.2.112:0/2501353935 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4778006a20 con 0x7f478010b050 2021-02-10T11:48:37.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:37.649 7f477d61a700 1 -- 172.21.2.112:0/2501353935 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4778006790 con 0x7f478010b050 2021-02-10T11:48:37.669 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:37.649 7f477d61a700 1 -- 172.21.2.112:0/2501353935 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f477800b700 con 0x7f478010b050 2021-02-10T11:48:37.669 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:37.649 7f4784854700 1 -- 172.21.2.112:0/2501353935 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f478010b050 msgr2=0x7f47801091c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:37.669 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:37.649 7f4784854700 1 --2- 172.21.2.112:0/2501353935 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f478010b050 0x7f47801091c0 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 rx=0x7f4778006e50 tx=0x7f4778009e70).stop 2021-02-10T11:48:37.669 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:37.650 7f4784854700 1 -- 172.21.2.112:0/2501353935 shutdown_connections 2021-02-10T11:48:37.669 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:37.650 7f4784854700 1 --2- 172.21.2.112:0/2501353935 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f478010b050 0x7f47801091c0 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:37.669 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:37.650 7f4784854700 1 --2- 172.21.2.112:0/2501353935 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f47800e53a0 0x7f47801b0550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:37.669 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:37.650 7f4784854700 1 -- 172.21.2.112:0/2501353935 >> 172.21.2.112:0/2501353935 conn(0x7f47800ce4b0 msgr2=0x7f47800e45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:37.669 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:37.650 7f4784854700 1 -- 172.21.2.112:0/2501353935 shutdown_connections 2021-02-10T11:48:37.669 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:37.650 7f4784854700 1 -- 172.21.2.112:0/2501353935 wait complete. 2021-02-10T11:48:37.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:37.664 7f4784854700 1 Processor -- start 2021-02-10T11:48:37.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:37.664 7f4784854700 1 -- start start 2021-02-10T11:48:37.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:37.665 7f4784854700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f47800aaa90 0x7f4780095ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:37.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:37.665 7f4784854700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f47800962c0 0x7f47800966c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:37.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:37.665 7f4784854700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4780096ce0 0x7f47800908d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:37.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:37.665 7f4784854700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f478019eb80 con 0x7f47800962c0 2021-02-10T11:48:37.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:37.665 7f4784854700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4780090ef0 con 0x7f4780096ce0 2021-02-10T11:48:37.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:37.665 7f4784854700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4780091110 con 0x7f47800aaa90 2021-02-10T11:48:37.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:37.665 7f477de1b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f47800962c0 0x7f47800966c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:37.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:37.665 7f477de1b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f47800962c0 0x7f47800966c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:33168/0 (socket says 172.21.2.112:33168) 2021-02-10T11:48:37.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:37.665 7f477de1b700 1 -- 172.21.2.112:0/1482304775 learned_addr learned my addr 172.21.2.112:0/1482304775 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:48:37.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:37.665 7f477e61c700 1 --2- 172.21.2.112:0/1482304775 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f47800aaa90 0x7f4780095ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:37.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:37.665 7f477ee1d700 1 --2- 172.21.2.112:0/1482304775 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4780096ce0 0x7f47800908d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:37.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:37.665 7f477de1b700 1 -- 172.21.2.112:0/1482304775 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f47800aaa90 msgr2=0x7f4780095ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:37.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:37.665 7f477de1b700 1 --2- 172.21.2.112:0/1482304775 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f47800aaa90 0x7f4780095ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:37.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:37.665 7f477de1b700 1 -- 172.21.2.112:0/1482304775 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4780096ce0 msgr2=0x7f47800908d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:37.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:37.665 7f477de1b700 1 --2- 172.21.2.112:0/1482304775 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4780096ce0 0x7f47800908d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:37.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:37.665 7f477de1b700 1 -- 172.21.2.112:0/1482304775 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4780091330 con 0x7f47800962c0 2021-02-10T11:48:37.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:37.665 7f477de1b700 1 --2- 172.21.2.112:0/1482304775 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f47800962c0 0x7f47800966c0 secure :-1 s=READY pgs=3196 cs=0 l=1 rev1=1 rx=0x7f4774001c60 tx=0x7f477400a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:37.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:37.666 7f476effd700 1 -- 172.21.2.112:0/1482304775 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f477400acc0 con 0x7f47800962c0 2021-02-10T11:48:37.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:37.666 7f476effd700 1 -- 172.21.2.112:0/1482304775 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4774004140 con 0x7f47800962c0 2021-02-10T11:48:37.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:37.666 7f476effd700 1 -- 172.21.2.112:0/1482304775 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4774004660 con 0x7f47800962c0 2021-02-10T11:48:37.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:37.666 7f4784854700 1 -- 172.21.2.112:0/1482304775 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4780091330 con 0x7f47800962c0 2021-02-10T11:48:37.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:37.666 7f4784854700 1 -- 172.21.2.112:0/1482304775 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47800918c0 con 0x7f47800962c0 2021-02-10T11:48:37.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:37.666 7f476effd700 1 -- 172.21.2.112:0/1482304775 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4774004320 con 0x7f47800962c0 2021-02-10T11:48:37.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:37.667 7f476effd700 1 --2- 172.21.2.112:0/1482304775 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f47640186c0 0x7f476401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:37.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:37.667 7f476effd700 1 -- 172.21.2.112:0/1482304775 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f4774032570 con 0x7f47800962c0 2021-02-10T11:48:37.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:37.667 7f477e61c700 1 --2- 172.21.2.112:0/1482304775 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f47640186c0 0x7f476401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:37.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:37.667 7f477e61c700 1 --2- 172.21.2.112:0/1482304775 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f47640186c0 0x7f476401abc0 crc :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:37.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:37.697 7f4784854700 1 -- 172.21.2.112:0/1482304775 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4780103750 con 0x7f47800962c0 2021-02-10T11:48:37.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:37.700 7f476effd700 1 -- 172.21.2.112:0/1482304775 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4774000dd0 con 0x7f47800962c0 2021-02-10T11:48:37.829 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:37.825 7f4784854700 1 -- 172.21.2.112:0/1482304775 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"access_level": "r", "prefix": "fs subvolume authorize", "vol_name": "cephfs", "sub_name": "subvolume_0000000000563415", "target": ["mgr", ""], "auth_id": "guest2", "group_name": "group_0000000000578561"}) v1 -- 0x7f478019eb80 con 0x7f47640186c0 2021-02-10T11:48:37.833 INFO:tasks.ceph.mgr.x.gibba020.stderr:Creating meta for ID guest2 with tenant None 2021-02-10T11:48:37.869 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:37.866 7f476effd700 1 -- 172.21.2.112:0/1482304775 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f4770006a70 con 0x7f47640186c0 2021-02-10T11:48:37.894 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:37.891 7f4784854700 1 -- 172.21.2.112:0/1482304775 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f47640186c0 msgr2=0x7f476401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:37.894 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:37.891 7f4784854700 1 --2- 172.21.2.112:0/1482304775 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f47640186c0 0x7f476401abc0 crc :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:37.894 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:37.891 7f4784854700 1 -- 172.21.2.112:0/1482304775 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f47800962c0 msgr2=0x7f47800966c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:37.894 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:37.891 7f4784854700 1 --2- 172.21.2.112:0/1482304775 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f47800962c0 0x7f47800966c0 secure :-1 s=READY pgs=3196 cs=0 l=1 rev1=1 rx=0x7f4774001c60 tx=0x7f477400a8d0).stop 2021-02-10T11:48:37.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:37.891 7f4784854700 1 -- 172.21.2.112:0/1482304775 shutdown_connections 2021-02-10T11:48:37.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:37.891 7f4784854700 1 --2- 172.21.2.112:0/1482304775 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f47640186c0 0x7f476401abc0 unknown :-1 s=CLOSED pgs=1438 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:37.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:37.891 7f4784854700 1 --2- 172.21.2.112:0/1482304775 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4780096ce0 0x7f47800908d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:37.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:37.891 7f4784854700 1 --2- 172.21.2.112:0/1482304775 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f47800962c0 0x7f47800966c0 unknown :-1 s=CLOSED pgs=3196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:37.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:37.891 7f4784854700 1 --2- 172.21.2.112:0/1482304775 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f47800aaa90 0x7f4780095ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:37.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:37.891 7f4784854700 1 -- 172.21.2.112:0/1482304775 >> 172.21.2.112:0/1482304775 conn(0x7f478019b000 msgr2=0x7f478018c310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:37.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:37.891 7f4784854700 1 -- 172.21.2.112:0/1482304775 shutdown_connections 2021-02-10T11:48:37.895 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:37.891 7f4784854700 1 -- 172.21.2.112:0/1482304775 wait complete. 2021-02-10T11:48:37.902 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume authorized_list cephfs subvolume_0000000000563415 --group_name group_0000000000578561 2021-02-10T11:48:38.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.316 7fb6601e3700 1 Processor -- start 2021-02-10T11:48:38.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.316 7fb6601e3700 1 -- start start 2021-02-10T11:48:38.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.317 7fb6601e3700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fb6580d1870 0x7fb6581b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:38.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.317 7fb6601e3700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb6581b0ae0 0x7fb658108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:38.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.317 7fb6601e3700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6580fe1c0 con 0x7fb6580d1c70 2021-02-10T11:48:38.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.317 7fb6601e3700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb658109a60 con 0x7fb6580d1870 2021-02-10T11:48:38.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.317 7fb6601e3700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb658109750 con 0x7fb6581b0ae0 2021-02-10T11:48:38.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.317 7fb65e800700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7fb6580d1c70 0x7fb658108f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:54426/0 (socket says 172.21.2.112:54426) 2021-02-10T11:48:38.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.317 7fb65dfff700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fb6580d1870 0x7fb6581b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:38.337 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.317 7fb65d7fe700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb6581b0ae0 0x7fb658108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:38.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.317 7fb65e800700 1 -- 172.21.2.112:0/776958823 learned_addr learned my addr 172.21.2.112:0/776958823 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:48:38.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.317 7fb65cffd700 1 -- 172.21.2.112:0/776958823 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 577883320 0 0) 0x7fb650001440 con 0x7fb6580d1c70 2021-02-10T11:48:38.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.317 7fb65cffd700 1 -- 172.21.2.112:0/776958823 --> v1:172.21.2.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb644002fe0 con 0x7fb6580d1c70 2021-02-10T11:48:38.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.317 7fb65d7fe700 1 -- 172.21.2.112:0/776958823 >> v2:172.21.2.120:3300/0 conn(0x7fb6580d1870 msgr2=0x7fb6581b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:38.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.317 7fb65d7fe700 1 --2- 172.21.2.112:0/776958823 >> v2:172.21.2.120:3300/0 conn(0x7fb6580d1870 0x7fb6581b04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:38.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.317 7fb65d7fe700 1 -- 172.21.2.112:0/776958823 >> v1:172.21.2.120:6789/0 conn(0x7fb6580d1c70 legacy=0x7fb658108f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:38.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.318 7fb65d7fe700 1 -- 172.21.2.112:0/776958823 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb658088070 con 0x7fb6581b0ae0 2021-02-10T11:48:38.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.318 7fb65d7fe700 1 --2- 172.21.2.112:0/776958823 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb6581b0ae0 0x7fb658108940 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7fb648006650 tx=0x7fb648009a80).ready entity=mon.2 client_cookie=4b45939e5ca81b20 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:38.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.318 7fb65cffd700 1 -- 172.21.2.112:0/776958823 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb6480009b0 con 0x7fb6581b0ae0 2021-02-10T11:48:38.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.318 7fb65cffd700 1 -- 172.21.2.112:0/776958823 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb64800c0c0 con 0x7fb6581b0ae0 2021-02-10T11:48:38.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.318 7fb65cffd700 1 -- 172.21.2.112:0/776958823 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb64800c5e0 con 0x7fb6581b0ae0 2021-02-10T11:48:38.338 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.318 7fb6601e3700 1 -- 172.21.2.112:0/776958823 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb6581b0ae0 msgr2=0x7fb658108940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:38.339 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.318 7fb6601e3700 1 --2- 172.21.2.112:0/776958823 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb6581b0ae0 0x7fb658108940 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7fb648006650 tx=0x7fb648009a80).stop 2021-02-10T11:48:38.339 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.319 7fb6601e3700 1 -- 172.21.2.112:0/776958823 shutdown_connections 2021-02-10T11:48:38.339 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.319 7fb6601e3700 1 --2- 172.21.2.112:0/776958823 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb6581b0ae0 0x7fb658108940 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:38.339 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.319 7fb6601e3700 1 --2- 172.21.2.112:0/776958823 >> v2:172.21.2.120:3300/0 conn(0x7fb6580d1870 0x7fb6581b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:38.339 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.319 7fb6601e3700 1 -- 172.21.2.112:0/776958823 >> 172.21.2.112:0/776958823 conn(0x7fb6580ce420 msgr2=0x7fb6580d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:38.339 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.319 7fb6601e3700 1 -- 172.21.2.112:0/776958823 shutdown_connections 2021-02-10T11:48:38.339 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.319 7fb6601e3700 1 -- 172.21.2.112:0/776958823 wait complete. 2021-02-10T11:48:38.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.334 7fb6601e3700 1 Processor -- start 2021-02-10T11:48:38.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.334 7fb6601e3700 1 -- start start 2021-02-10T11:48:38.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.334 7fb6601e3700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb6581a05f0 0x7fb6580a8f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:38.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.334 7fb6601e3700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb6580aba50 0x7fb6580b3090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:38.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.334 7fb6601e3700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb6580b36b0 0x7fb6580b42a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:38.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.334 7fb6601e3700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6580b3d20 con 0x7fb6580b36b0 2021-02-10T11:48:38.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.334 7fb6601e3700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6580abe50 con 0x7fb6580aba50 2021-02-10T11:48:38.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.334 7fb6601e3700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6580b23c0 con 0x7fb6581a05f0 2021-02-10T11:48:38.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.334 7fb65e800700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb6580b36b0 0x7fb6580b42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:38.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.334 7fb65e800700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb6580b36b0 0x7fb6580b42a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:33180/0 (socket says 172.21.2.112:33180) 2021-02-10T11:48:38.342 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.334 7fb65e800700 1 -- 172.21.2.112:0/443915542 learned_addr learned my addr 172.21.2.112:0/443915542 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:48:38.342 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.335 7fb65d7fe700 1 --2- 172.21.2.112:0/443915542 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb6580aba50 0x7fb6580b3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:38.342 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.335 7fb65dfff700 1 --2- 172.21.2.112:0/443915542 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb6581a05f0 0x7fb6580a8f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:38.342 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.335 7fb65e800700 1 -- 172.21.2.112:0/443915542 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb6581a05f0 msgr2=0x7fb6580a8f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:38.342 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.335 7fb65e800700 1 --2- 172.21.2.112:0/443915542 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb6581a05f0 0x7fb6580a8f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:38.342 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.335 7fb65e800700 1 -- 172.21.2.112:0/443915542 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb6580aba50 msgr2=0x7fb6580b3090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:38.342 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.335 7fb65e800700 1 --2- 172.21.2.112:0/443915542 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb6580aba50 0x7fb6580b3090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:38.342 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.335 7fb65e800700 1 -- 172.21.2.112:0/443915542 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6580b48c0 con 0x7fb6580b36b0 2021-02-10T11:48:38.342 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.335 7fb65dfff700 1 --2- 172.21.2.112:0/443915542 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb6581a05f0 0x7fb6580a8f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:48:38.342 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.335 7fb65d7fe700 1 --2- 172.21.2.112:0/443915542 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb6580aba50 0x7fb6580b3090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:48:38.342 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.335 7fb65e800700 1 --2- 172.21.2.112:0/443915542 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb6580b36b0 0x7fb6580b42a0 secure :-1 s=READY pgs=3197 cs=0 l=1 rev1=1 rx=0x7fb65000a8f0 tx=0x7fb65000ac90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:38.342 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.335 7fb64e7fc700 1 -- 172.21.2.112:0/443915542 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb6500048b0 con 0x7fb6580b36b0 2021-02-10T11:48:38.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.335 7fb64e7fc700 1 -- 172.21.2.112:0/443915542 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb650004bb0 con 0x7fb6580b36b0 2021-02-10T11:48:38.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.335 7fb64e7fc700 1 -- 172.21.2.112:0/443915542 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb6500072a0 con 0x7fb6580b36b0 2021-02-10T11:48:38.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.336 7fb6601e3700 1 -- 172.21.2.112:0/443915542 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6580b48c0 con 0x7fb6580b36b0 2021-02-10T11:48:38.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.336 7fb6601e3700 1 -- 172.21.2.112:0/443915542 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6580b48c0 con 0x7fb6580b36b0 2021-02-10T11:48:38.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.336 7fb64e7fc700 1 -- 172.21.2.112:0/443915542 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb650000e90 con 0x7fb6580b36b0 2021-02-10T11:48:38.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.336 7fb64e7fc700 1 --2- 172.21.2.112:0/443915542 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb6400186c0 0x7fb64001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:38.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.336 7fb64e7fc700 1 -- 172.21.2.112:0/443915542 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fb650032720 con 0x7fb6580b36b0 2021-02-10T11:48:38.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.337 7fb65dfff700 1 --2- 172.21.2.112:0/443915542 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb6400186c0 0x7fb64001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:38.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.337 7fb65dfff700 1 --2- 172.21.2.112:0/443915542 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb6400186c0 0x7fb64001abc0 crc :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:38.369 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.365 7fb6601e3700 1 -- 172.21.2.112:0/443915542 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6580825d0 con 0x7fb6580b36b0 2021-02-10T11:48:38.371 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.368 7fb64e7fc700 1 -- 172.21.2.112:0/443915542 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb65002af80 con 0x7fb6580b36b0 2021-02-10T11:48:38.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.491 7fb6601e3700 1 -- 172.21.2.112:0/443915542 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume authorized_list", "vol_name": "cephfs", "sub_name": "subvolume_0000000000563415", "target": ["mgr", ""], "group_name": "group_0000000000578561"}) v1 -- 0x7fb6580825d0 con 0x7fb6400186c0 2021-02-10T11:48:38.499 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.496 7fb64e7fc700 1 -- 172.21.2.112:0/443915542 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+111 (crc 0 0 0) 0x7fb654006b10 con 0x7fb6400186c0 2021-02-10T11:48:38.502 INFO:teuthology.orchestra.run.gibba012.stdout:[ 2021-02-10T11:48:38.502 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:48:38.502 INFO:teuthology.orchestra.run.gibba012.stdout: "guest2": "r" 2021-02-10T11:48:38.502 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:48:38.502 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:48:38.502 INFO:teuthology.orchestra.run.gibba012.stdout: "alice": "rw" 2021-02-10T11:48:38.502 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:48:38.502 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:48:38.503 INFO:teuthology.orchestra.run.gibba012.stdout: "guest1": "rw" 2021-02-10T11:48:38.503 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:48:38.503 INFO:teuthology.orchestra.run.gibba012.stdout:] 2021-02-10T11:48:38.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.501 7fb6601e3700 1 -- 172.21.2.112:0/443915542 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb6400186c0 msgr2=0x7fb64001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:38.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.501 7fb6601e3700 1 --2- 172.21.2.112:0/443915542 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb6400186c0 0x7fb64001abc0 crc :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:38.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.501 7fb6601e3700 1 -- 172.21.2.112:0/443915542 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb6580b36b0 msgr2=0x7fb6580b42a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:38.504 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.501 7fb6601e3700 1 --2- 172.21.2.112:0/443915542 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb6580b36b0 0x7fb6580b42a0 secure :-1 s=READY pgs=3197 cs=0 l=1 rev1=1 rx=0x7fb65000a8f0 tx=0x7fb65000ac90).stop 2021-02-10T11:48:38.505 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.501 7fb6601e3700 1 -- 172.21.2.112:0/443915542 shutdown_connections 2021-02-10T11:48:38.505 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.501 7fb6601e3700 1 --2- 172.21.2.112:0/443915542 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb6400186c0 0x7fb64001abc0 unknown :-1 s=CLOSED pgs=1439 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:38.505 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.501 7fb6601e3700 1 --2- 172.21.2.112:0/443915542 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb6580aba50 0x7fb6580b3090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:38.505 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.501 7fb6601e3700 1 --2- 172.21.2.112:0/443915542 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb6580b36b0 0x7fb6580b42a0 unknown :-1 s=CLOSED pgs=3197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:38.505 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.501 7fb6601e3700 1 --2- 172.21.2.112:0/443915542 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb6581a05f0 0x7fb6580a8f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:38.505 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.501 7fb6601e3700 1 -- 172.21.2.112:0/443915542 >> 172.21.2.112:0/443915542 conn(0x7fb6580bc070 msgr2=0x7fb6580ba770 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:38.505 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.501 7fb6601e3700 1 -- 172.21.2.112:0/443915542 shutdown_connections 2021-02-10T11:48:38.505 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.501 7fb6601e3700 1 -- 172.21.2.112:0/443915542 wait complete. 2021-02-10T11:48:38.513 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume deauthorize cephfs subvolume_0000000000563415 alice --group_name group_0000000000578561 2021-02-10T11:48:38.947 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.926 7f9a7c33f700 1 Processor -- start 2021-02-10T11:48:38.947 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.926 7f9a7c33f700 1 -- start start 2021-02-10T11:48:38.947 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.927 7f9a7c33f700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f9a740eb220 0x7f9a740eb620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:38.947 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.927 7f9a7c33f700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f9a7410aad0 0x7f9a7410aed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:38.947 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.927 7f9a7c33f700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a7410b4f0 con 0x7f9a740dce80 2021-02-10T11:48:38.947 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.927 7f9a7c33f700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a7410a550 con 0x7f9a740eb220 2021-02-10T11:48:38.947 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.927 7f9a7c33f700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a740881d0 con 0x7f9a7410aad0 2021-02-10T11:48:38.947 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.927 7f9a7995a700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f9a740eb220 0x7f9a740eb620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:38.948 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.927 7f9a7a15b700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f9a740dce80 0x7f9a740d14f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:60880/0 (socket says 172.21.2.112:60880) 2021-02-10T11:48:38.948 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.927 7f9a7a15b700 1 -- 172.21.2.112:0/3360927099 learned_addr learned my addr 172.21.2.112:0/3360927099 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:48:38.948 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.928 7f9a7a95c700 1 --2- 172.21.2.112:0/3360927099 >> v2:172.21.2.120:3300/0 conn(0x7f9a7410aad0 0x7f9a7410aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:38.948 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.928 7f9a7995a700 1 -- 172.21.2.112:0/3360927099 >> v2:172.21.2.120:3300/0 conn(0x7f9a7410aad0 msgr2=0x7f9a7410aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:38.948 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.928 7f9a7995a700 1 --2- 172.21.2.112:0/3360927099 >> v2:172.21.2.120:3300/0 conn(0x7f9a7410aad0 0x7f9a7410aed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:38.948 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.928 7f9a7995a700 1 -- 172.21.2.112:0/3360927099 >> v1:172.21.2.112:6789/0 conn(0x7f9a740dce80 legacy=0x7f9a740d14f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:38.948 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.928 7f9a7995a700 1 -- 172.21.2.112:0/3360927099 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a740d1f30 con 0x7f9a740eb220 2021-02-10T11:48:38.948 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.928 7f9a79159700 1 -- 172.21.2.112:0/3360927099 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2449741013 0 0) 0x7f9a70000ce0 con 0x7f9a740dce80 2021-02-10T11:48:38.948 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.928 7f9a7995a700 1 --2- 172.21.2.112:0/3360927099 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9a740eb220 0x7f9a740eb620 secure :-1 s=READY pgs=3199 cs=0 l=1 rev1=1 rx=0x7f9a68006650 tx=0x7f9a68009a00).ready entity=mon.0 client_cookie=275d72e832d09abd server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:38.948 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.928 7f9a79159700 1 -- 172.21.2.112:0/3360927099 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9a680009b0 con 0x7f9a740eb220 2021-02-10T11:48:38.948 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.928 7f9a79159700 1 -- 172.21.2.112:0/3360927099 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a6800c0c0 con 0x7f9a740eb220 2021-02-10T11:48:38.948 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.928 7f9a79159700 1 -- 172.21.2.112:0/3360927099 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9a6800c5a0 con 0x7f9a740eb220 2021-02-10T11:48:38.948 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.928 7f9a7c33f700 1 -- 172.21.2.112:0/3360927099 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9a740eb220 msgr2=0x7f9a740eb620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:38.949 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.928 7f9a7c33f700 1 --2- 172.21.2.112:0/3360927099 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9a740eb220 0x7f9a740eb620 secure :-1 s=READY pgs=3199 cs=0 l=1 rev1=1 rx=0x7f9a68006650 tx=0x7f9a68009a00).stop 2021-02-10T11:48:38.949 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.929 7f9a7c33f700 1 -- 172.21.2.112:0/3360927099 shutdown_connections 2021-02-10T11:48:38.949 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.929 7f9a7c33f700 1 --2- 172.21.2.112:0/3360927099 >> v2:172.21.2.120:3300/0 conn(0x7f9a7410aad0 0x7f9a7410aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:38.949 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.929 7f9a7c33f700 1 --2- 172.21.2.112:0/3360927099 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9a740eb220 0x7f9a740eb620 unknown :-1 s=CLOSED pgs=3199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:38.949 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.929 7f9a7c33f700 1 -- 172.21.2.112:0/3360927099 >> 172.21.2.112:0/3360927099 conn(0x7f9a740ea630 msgr2=0x7f9a740d0800 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:38.949 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.929 7f9a7c33f700 1 -- 172.21.2.112:0/3360927099 shutdown_connections 2021-02-10T11:48:38.949 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.929 7f9a7c33f700 1 -- 172.21.2.112:0/3360927099 wait complete. 2021-02-10T11:48:38.949 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.944 7f9a7c33f700 1 Processor -- start 2021-02-10T11:48:38.949 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.944 7f9a7c33f700 1 -- start start 2021-02-10T11:48:38.951 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.944 7f9a7c33f700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9a740bfa20 0x7f9a740a4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:38.951 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.944 7f9a7c33f700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9a740a50c0 0x7f9a740a54c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:38.951 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.944 7f9a7c33f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9a740a5ae0 0x7f9a74097f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:38.951 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.944 7f9a7c33f700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a740bfea0 con 0x7f9a740a5ae0 2021-02-10T11:48:38.951 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.944 7f9a7c33f700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a740a4550 con 0x7f9a740bfa20 2021-02-10T11:48:38.951 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.944 7f9a7c33f700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a74099650 con 0x7f9a740a50c0 2021-02-10T11:48:38.951 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.944 7f9a7a95c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9a740a5ae0 0x7f9a74097f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:38.951 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.944 7f9a7a95c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9a740a5ae0 0x7f9a74097f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:33192/0 (socket says 172.21.2.112:33192) 2021-02-10T11:48:38.951 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.944 7f9a7a95c700 1 -- 172.21.2.112:0/170992221 learned_addr learned my addr 172.21.2.112:0/170992221 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:48:38.951 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.945 7f9a7a15b700 1 --2- 172.21.2.112:0/170992221 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9a740bfa20 0x7f9a740a4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:38.951 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.945 7f9a7995a700 1 --2- 172.21.2.112:0/170992221 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9a740a50c0 0x7f9a740a54c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:38.952 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.945 7f9a7a95c700 1 -- 172.21.2.112:0/170992221 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9a740a50c0 msgr2=0x7f9a740a54c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:38.952 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.945 7f9a7a95c700 1 --2- 172.21.2.112:0/170992221 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9a740a50c0 0x7f9a740a54c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:38.952 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.945 7f9a7a95c700 1 -- 172.21.2.112:0/170992221 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9a740bfa20 msgr2=0x7f9a740a4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:38.952 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.945 7f9a7a95c700 1 --2- 172.21.2.112:0/170992221 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9a740bfa20 0x7f9a740a4aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:38.952 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.945 7f9a7a95c700 1 -- 172.21.2.112:0/170992221 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a740985b0 con 0x7f9a740a5ae0 2021-02-10T11:48:38.952 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.945 7f9a7a95c700 1 --2- 172.21.2.112:0/170992221 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9a740a5ae0 0x7f9a74097f90 secure :-1 s=READY pgs=3200 cs=0 l=1 rev1=1 rx=0x7f9a6c0099a0 tx=0x7f9a6c00bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:38.952 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.945 7f9a66ffd700 1 -- 172.21.2.112:0/170992221 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9a6c0066f0 con 0x7f9a740a5ae0 2021-02-10T11:48:38.952 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.945 7f9a66ffd700 1 -- 172.21.2.112:0/170992221 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a6c003030 con 0x7f9a740a5ae0 2021-02-10T11:48:38.952 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.945 7f9a66ffd700 1 -- 172.21.2.112:0/170992221 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9a6c0013e0 con 0x7f9a740a5ae0 2021-02-10T11:48:38.953 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.945 7f9a7c33f700 1 -- 172.21.2.112:0/170992221 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a740985b0 con 0x7f9a740a5ae0 2021-02-10T11:48:38.953 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.946 7f9a7c33f700 1 -- 172.21.2.112:0/170992221 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a740985b0 con 0x7f9a740a5ae0 2021-02-10T11:48:38.953 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.946 7f9a66ffd700 1 -- 172.21.2.112:0/170992221 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9a6c0011f0 con 0x7f9a740a5ae0 2021-02-10T11:48:38.953 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.946 7f9a66ffd700 1 --2- 172.21.2.112:0/170992221 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9a5c018700 0x7f9a5c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:38.953 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.946 7f9a66ffd700 1 -- 172.21.2.112:0/170992221 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f9a6c0329c0 con 0x7f9a740a5ae0 2021-02-10T11:48:38.953 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.946 7f9a7a15b700 1 --2- 172.21.2.112:0/170992221 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9a5c018700 0x7f9a5c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:38.953 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.947 7f9a7a15b700 1 --2- 172.21.2.112:0/170992221 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9a5c018700 0x7f9a5c01abc0 crc :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:38.980 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.977 7f9a7c33f700 1 -- 172.21.2.112:0/170992221 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a740f93d0 con 0x7f9a740a5ae0 2021-02-10T11:48:38.983 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:38.980 7f9a66ffd700 1 -- 172.21.2.112:0/170992221 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9a6c00be30 con 0x7f9a740a5ae0 2021-02-10T11:48:39.109 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:39.106 7f9a7c33f700 1 -- 172.21.2.112:0/170992221 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume deauthorize", "vol_name": "cephfs", "sub_name": "subvolume_0000000000563415", "target": ["mgr", ""], "auth_id": "alice", "group_name": "group_0000000000578561"}) v1 -- 0x7f9a740f93d0 con 0x7f9a5c018700 2021-02-10T11:48:39.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:39.141 7f9a66ffd700 1 -- 172.21.2.112:0/170992221 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f9a70006e40 con 0x7f9a5c018700 2021-02-10T11:48:39.175 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:39.171 7f9a7c33f700 1 -- 172.21.2.112:0/170992221 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9a5c018700 msgr2=0x7f9a5c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:39.175 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:39.171 7f9a7c33f700 1 --2- 172.21.2.112:0/170992221 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9a5c018700 0x7f9a5c01abc0 crc :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:39.175 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:39.171 7f9a7c33f700 1 -- 172.21.2.112:0/170992221 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9a740a5ae0 msgr2=0x7f9a74097f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:39.175 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:39.171 7f9a7c33f700 1 --2- 172.21.2.112:0/170992221 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9a740a5ae0 0x7f9a74097f90 secure :-1 s=READY pgs=3200 cs=0 l=1 rev1=1 rx=0x7f9a6c0099a0 tx=0x7f9a6c00bcf0).stop 2021-02-10T11:48:39.175 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:39.171 7f9a7c33f700 1 -- 172.21.2.112:0/170992221 shutdown_connections 2021-02-10T11:48:39.175 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:39.171 7f9a7c33f700 1 --2- 172.21.2.112:0/170992221 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9a5c018700 0x7f9a5c01abc0 unknown :-1 s=CLOSED pgs=1440 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:39.175 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:39.171 7f9a7c33f700 1 --2- 172.21.2.112:0/170992221 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9a740a5ae0 0x7f9a74097f90 unknown :-1 s=CLOSED pgs=3200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:39.176 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:39.171 7f9a7c33f700 1 --2- 172.21.2.112:0/170992221 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9a740a50c0 0x7f9a740a54c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:39.176 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:39.171 7f9a7c33f700 1 --2- 172.21.2.112:0/170992221 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9a740bfa20 0x7f9a740a4aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:39.176 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:39.171 7f9a7c33f700 1 -- 172.21.2.112:0/170992221 >> 172.21.2.112:0/170992221 conn(0x7f9a74100d30 msgr2=0x7f9a74108b00 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:39.176 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:39.171 7f9a7c33f700 1 -- 172.21.2.112:0/170992221 shutdown_connections 2021-02-10T11:48:39.176 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:39.171 7f9a7c33f700 1 -- 172.21.2.112:0/170992221 wait complete. 2021-02-10T11:48:39.183 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume deauthorize cephfs subvolume_0000000000563415 guest1 --group_name group_0000000000578561 2021-02-10T11:48:39.616 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:39.596 7f2d80ee9700 1 Processor -- start 2021-02-10T11:48:39.616 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:39.596 7f2d80ee9700 1 -- start start 2021-02-10T11:48:39.616 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:39.596 7f2d80ee9700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2d7c0dce80 0x7f2d7c0d14f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:39.616 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:39.596 7f2d80ee9700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f2d7c10b2c0 0x7f2d7c0eb620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:39.617 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:39.596 7f2d80ee9700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d7c10a550 con 0x7f2d7c0eb220 2021-02-10T11:48:39.617 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:39.596 7f2d80ee9700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d7c0d1fa0 con 0x7f2d7c10b2c0 2021-02-10T11:48:39.617 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:39.596 7f2d80ee9700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d7c0881d0 con 0x7f2d7c0dce80 2021-02-10T11:48:39.617 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:39.597 7f2d7a312700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f2d7c0eb220 0x7f2d7c10aad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:60896/0 (socket says 172.21.2.112:60896) 2021-02-10T11:48:39.617 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:39.597 7f2d7a312700 1 -- 172.21.2.112:0/2126768791 learned_addr learned my addr 172.21.2.112:0/2126768791 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:48:39.617 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:39.597 7f2d7ab13700 1 --2- 172.21.2.112:0/2126768791 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2d7c0dce80 0x7f2d7c0d14f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:39.617 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:39.597 7f2d7b314700 1 --2- 172.21.2.112:0/2126768791 >> v2:172.21.2.120:3300/0 conn(0x7f2d7c10b2c0 0x7f2d7c0eb620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:39.617 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:39.597 7f2d79b11700 1 -- 172.21.2.112:0/2126768791 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3460938158 0 0) 0x7f2d6c0010b0 con 0x7f2d7c0eb220 2021-02-10T11:48:39.617 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:39.597 7f2d79b11700 1 -- 172.21.2.112:0/2126768791 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d68002fe0 con 0x7f2d7c0eb220 2021-02-10T11:48:39.617 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:39.597 7f2d7ab13700 1 -- 172.21.2.112:0/2126768791 >> v2:172.21.2.120:3300/0 conn(0x7f2d7c10b2c0 msgr2=0x7f2d7c0eb620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:39.617 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:39.597 7f2d7ab13700 1 --2- 172.21.2.112:0/2126768791 >> v2:172.21.2.120:3300/0 conn(0x7f2d7c10b2c0 0x7f2d7c0eb620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:39.617 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:39.597 7f2d7ab13700 1 -- 172.21.2.112:0/2126768791 >> v1:172.21.2.112:6789/0 conn(0x7f2d7c0eb220 legacy=0x7f2d7c10aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:39.617 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:39.597 7f2d7ab13700 1 -- 172.21.2.112:0/2126768791 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d7c0dd280 con 0x7f2d7c0dce80 2021-02-10T11:48:39.618 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:39.597 7f2d7ab13700 1 --2- 172.21.2.112:0/2126768791 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2d7c0dce80 0x7f2d7c0d14f0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 rx=0x7f2d64006650 tx=0x7f2d64009a80).ready entity=mon.2 client_cookie=9d9f2f2a8528cf02 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:39.618 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:39.598 7f2d79b11700 1 -- 172.21.2.112:0/2126768791 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2d640009b0 con 0x7f2d7c0dce80 2021-02-10T11:48:39.618 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:39.598 7f2d79b11700 1 -- 172.21.2.112:0/2126768791 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d6400c0c0 con 0x7f2d7c0dce80 2021-02-10T11:48:39.618 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:39.598 7f2d79b11700 1 -- 172.21.2.112:0/2126768791 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2d6400c5e0 con 0x7f2d7c0dce80 2021-02-10T11:48:39.618 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:39.598 7f2d80ee9700 1 -- 172.21.2.112:0/2126768791 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2d7c0dce80 msgr2=0x7f2d7c0d14f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:39.618 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:39.598 7f2d80ee9700 1 --2- 172.21.2.112:0/2126768791 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2d7c0dce80 0x7f2d7c0d14f0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 rx=0x7f2d64006650 tx=0x7f2d64009a80).stop 2021-02-10T11:48:39.618 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:39.598 7f2d80ee9700 1 -- 172.21.2.112:0/2126768791 shutdown_connections 2021-02-10T11:48:39.618 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:39.598 7f2d80ee9700 1 --2- 172.21.2.112:0/2126768791 >> v2:172.21.2.120:3300/0 conn(0x7f2d7c10b2c0 0x7f2d7c0eb620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:39.618 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:39.598 7f2d80ee9700 1 --2- 172.21.2.112:0/2126768791 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2d7c0dce80 0x7f2d7c0d14f0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:39.618 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:39.598 7f2d80ee9700 1 -- 172.21.2.112:0/2126768791 >> 172.21.2.112:0/2126768791 conn(0x7f2d7c0ea630 msgr2=0x7f2d7c0d0800 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:39.618 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:39.599 7f2d80ee9700 1 -- 172.21.2.112:0/2126768791 shutdown_connections 2021-02-10T11:48:39.618 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:39.599 7f2d80ee9700 1 -- 172.21.2.112:0/2126768791 wait complete. 2021-02-10T11:48:39.620 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:39.613 7f2d80ee9700 1 Processor -- start 2021-02-10T11:48:39.620 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:39.613 7f2d80ee9700 1 -- start start 2021-02-10T11:48:39.620 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:39.614 7f2d80ee9700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2d7c0fcf20 0x7f2d7c106950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:39.621 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:39.614 7f2d80ee9700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2d7c09e5f0 0x7f2d7c09e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:39.621 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:39.614 7f2d80ee9700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2d7c09f010 0x7f2d7c1a2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:39.621 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:39.614 7f2d80ee9700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d7c0fd320 con 0x7f2d7c0fcf20 2021-02-10T11:48:39.621 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:39.614 7f2d80ee9700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d7c1a35e0 con 0x7f2d7c09e5f0 2021-02-10T11:48:39.621 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:39.614 7f2d80ee9700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d7c1a3800 con 0x7f2d7c09f010 2021-02-10T11:48:39.621 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:39.614 7f2d7ab13700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2d7c0fcf20 0x7f2d7c106950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:39.621 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:39.614 7f2d7ab13700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2d7c0fcf20 0x7f2d7c106950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:33208/0 (socket says 172.21.2.112:33208) 2021-02-10T11:48:39.621 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:39.614 7f2d7ab13700 1 -- 172.21.2.112:0/100256655 learned_addr learned my addr 172.21.2.112:0/100256655 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:48:39.621 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:39.614 7f2d7b314700 1 --2- 172.21.2.112:0/100256655 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2d7c09f010 0x7f2d7c1a2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:39.621 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:39.614 7f2d7a312700 1 --2- 172.21.2.112:0/100256655 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2d7c09e5f0 0x7f2d7c09e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:39.621 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:39.614 7f2d7ab13700 1 -- 172.21.2.112:0/100256655 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2d7c09f010 msgr2=0x7f2d7c1a2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:39.621 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:39.614 7f2d7ab13700 1 --2- 172.21.2.112:0/100256655 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2d7c09f010 0x7f2d7c1a2fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:39.621 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:39.614 7f2d7ab13700 1 -- 172.21.2.112:0/100256655 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2d7c09e5f0 msgr2=0x7f2d7c09e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:39.621 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:39.614 7f2d7ab13700 1 --2- 172.21.2.112:0/100256655 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2d7c09e5f0 0x7f2d7c09e9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:39.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:39.614 7f2d7ab13700 1 -- 172.21.2.112:0/100256655 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d7c1a3a20 con 0x7f2d7c0fcf20 2021-02-10T11:48:39.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:39.614 7f2d7ab13700 1 --2- 172.21.2.112:0/100256655 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2d7c0fcf20 0x7f2d7c106950 secure :-1 s=READY pgs=3202 cs=0 l=1 rev1=1 rx=0x7f2d64009650 tx=0x7f2d6400b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:39.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:39.615 7f2d737fe700 1 -- 172.21.2.112:0/100256655 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2d64009360 con 0x7f2d7c0fcf20 2021-02-10T11:48:39.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:39.615 7f2d737fe700 1 -- 172.21.2.112:0/100256655 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d6400bde0 con 0x7f2d7c0fcf20 2021-02-10T11:48:39.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:39.615 7f2d80ee9700 1 -- 172.21.2.112:0/100256655 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d7c1a3a20 con 0x7f2d7c0fcf20 2021-02-10T11:48:39.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:39.615 7f2d737fe700 1 -- 172.21.2.112:0/100256655 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2d640052e0 con 0x7f2d7c0fcf20 2021-02-10T11:48:39.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:39.615 7f2d80ee9700 1 -- 172.21.2.112:0/100256655 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d7c1a3a20 con 0x7f2d7c0fcf20 2021-02-10T11:48:39.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:39.615 7f2d737fe700 1 -- 172.21.2.112:0/100256655 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2d64006030 con 0x7f2d7c0fcf20 2021-02-10T11:48:39.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:39.615 7f2d737fe700 1 --2- 172.21.2.112:0/100256655 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2d600186c0 0x7f2d6001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:39.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:39.615 7f2d737fe700 1 -- 172.21.2.112:0/100256655 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f2d640327b0 con 0x7f2d7c0fcf20 2021-02-10T11:48:39.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:39.616 7f2d7a312700 1 --2- 172.21.2.112:0/100256655 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2d600186c0 0x7f2d6001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:39.622 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:39.616 7f2d7a312700 1 --2- 172.21.2.112:0/100256655 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2d600186c0 0x7f2d6001abc0 crc :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:39.650 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:39.646 7f2d80ee9700 1 -- 172.21.2.112:0/100256655 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d7c0fd320 con 0x7f2d7c0fcf20 2021-02-10T11:48:39.652 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:39.649 7f2d737fe700 1 -- 172.21.2.112:0/100256655 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2d64000940 con 0x7f2d7c0fcf20 2021-02-10T11:48:39.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:39.776 7f2d80ee9700 1 -- 172.21.2.112:0/100256655 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume deauthorize", "vol_name": "cephfs", "sub_name": "subvolume_0000000000563415", "target": ["mgr", ""], "auth_id": "guest1", "group_name": "group_0000000000578561"}) v1 -- 0x7f2d7c0fd320 con 0x7f2d600186c0 2021-02-10T11:48:39.815 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:39.812 7f2d737fe700 1 -- 172.21.2.112:0/100256655 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f2d6c000ec0 con 0x7f2d600186c0 2021-02-10T11:48:39.845 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:39.841 7f2d80ee9700 1 -- 172.21.2.112:0/100256655 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2d600186c0 msgr2=0x7f2d6001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:39.845 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:39.841 7f2d80ee9700 1 --2- 172.21.2.112:0/100256655 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2d600186c0 0x7f2d6001abc0 crc :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:39.845 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:39.842 7f2d80ee9700 1 -- 172.21.2.112:0/100256655 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2d7c0fcf20 msgr2=0x7f2d7c106950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:39.846 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:39.842 7f2d80ee9700 1 --2- 172.21.2.112:0/100256655 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2d7c0fcf20 0x7f2d7c106950 secure :-1 s=READY pgs=3202 cs=0 l=1 rev1=1 rx=0x7f2d64009650 tx=0x7f2d6400b960).stop 2021-02-10T11:48:39.846 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:39.842 7f2d80ee9700 1 -- 172.21.2.112:0/100256655 shutdown_connections 2021-02-10T11:48:39.846 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:39.842 7f2d80ee9700 1 --2- 172.21.2.112:0/100256655 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2d600186c0 0x7f2d6001abc0 unknown :-1 s=CLOSED pgs=1441 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:39.846 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:39.842 7f2d80ee9700 1 --2- 172.21.2.112:0/100256655 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2d7c09e5f0 0x7f2d7c09e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:39.846 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:39.842 7f2d80ee9700 1 --2- 172.21.2.112:0/100256655 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2d7c09f010 0x7f2d7c1a2fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:39.846 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:39.842 7f2d80ee9700 1 --2- 172.21.2.112:0/100256655 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2d7c0fcf20 0x7f2d7c106950 unknown :-1 s=CLOSED pgs=3202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:39.846 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:39.842 7f2d80ee9700 1 -- 172.21.2.112:0/100256655 >> 172.21.2.112:0/100256655 conn(0x7f2d7c0b1660 msgr2=0x7f2d7c0a8e10 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:39.846 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:39.842 7f2d80ee9700 1 -- 172.21.2.112:0/100256655 shutdown_connections 2021-02-10T11:48:39.846 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:39.842 7f2d80ee9700 1 -- 172.21.2.112:0/100256655 wait complete. 2021-02-10T11:48:39.854 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume deauthorize cephfs subvolume_0000000000563415 guest2 --group_name group_0000000000578561 2021-02-10T11:48:40.287 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.267 7fa70981d700 1 Processor -- start 2021-02-10T11:48:40.287 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.267 7fa70981d700 1 -- start start 2021-02-10T11:48:40.287 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.268 7fa70981d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fa7040dce80 0x7fa7040d14f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:40.287 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.268 7fa70981d700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fa7040eb220 0x7fa7040eb620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:40.287 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.268 7fa70981d700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa70410a550 con 0x7fa70410aad0 2021-02-10T11:48:40.287 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.268 7fa70981d700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7040881d0 con 0x7fa7040dce80 2021-02-10T11:48:40.287 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.268 7fa70981d700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7040d1f30 con 0x7fa7040eb220 2021-02-10T11:48:40.287 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.268 7fa703d75700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7fa70410aad0 0x7fa70410aed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:54464/0 (socket says 172.21.2.112:54464) 2021-02-10T11:48:40.287 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.268 7fa703d75700 1 -- 172.21.2.112:0/1015476853 learned_addr learned my addr 172.21.2.112:0/1015476853 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:48:40.287 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.268 7fa703574700 1 --2- 172.21.2.112:0/1015476853 >> v2:172.21.2.112:3300/0 conn(0x7fa7040dce80 0x7fa7040d14f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:40.288 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.268 7fa702d73700 1 --2- 172.21.2.112:0/1015476853 >> v2:172.21.2.120:3300/0 conn(0x7fa7040eb220 0x7fa7040eb620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:40.288 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.268 7fa702572700 1 -- 172.21.2.112:0/1015476853 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 987009998 0 0) 0x7fa6fc000f90 con 0x7fa70410aad0 2021-02-10T11:48:40.288 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.268 7fa702572700 1 -- 172.21.2.112:0/1015476853 --> v1:172.21.2.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6f0002fe0 con 0x7fa70410aad0 2021-02-10T11:48:40.288 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.268 7fa703574700 1 -- 172.21.2.112:0/1015476853 >> v2:172.21.2.120:3300/0 conn(0x7fa7040eb220 msgr2=0x7fa7040eb620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:40.288 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.268 7fa703574700 1 --2- 172.21.2.112:0/1015476853 >> v2:172.21.2.120:3300/0 conn(0x7fa7040eb220 0x7fa7040eb620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:40.288 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.268 7fa703574700 1 -- 172.21.2.112:0/1015476853 >> v1:172.21.2.120:6789/0 conn(0x7fa70410aad0 legacy=0x7fa70410aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:40.288 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.268 7fa703574700 1 -- 172.21.2.112:0/1015476853 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7040dd280 con 0x7fa7040dce80 2021-02-10T11:48:40.288 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.268 7fa703574700 1 --2- 172.21.2.112:0/1015476853 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa7040dce80 0x7fa7040d14f0 secure :-1 s=READY pgs=3203 cs=0 l=1 rev1=1 rx=0x7fa6f4006650 tx=0x7fa6f4009a80).ready entity=mon.0 client_cookie=2c99381bcc00fb7e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:40.288 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.268 7fa702572700 1 -- 172.21.2.112:0/1015476853 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa6f400c0c0 con 0x7fa7040dce80 2021-02-10T11:48:40.288 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.269 7fa702572700 1 -- 172.21.2.112:0/1015476853 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6f400c340 con 0x7fa7040dce80 2021-02-10T11:48:40.288 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.269 7fa702572700 1 -- 172.21.2.112:0/1015476853 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa6f400c860 con 0x7fa7040dce80 2021-02-10T11:48:40.288 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.269 7fa70981d700 1 -- 172.21.2.112:0/1015476853 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa7040dce80 msgr2=0x7fa7040d14f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:40.288 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.269 7fa70981d700 1 --2- 172.21.2.112:0/1015476853 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa7040dce80 0x7fa7040d14f0 secure :-1 s=READY pgs=3203 cs=0 l=1 rev1=1 rx=0x7fa6f4006650 tx=0x7fa6f4009a80).stop 2021-02-10T11:48:40.289 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.269 7fa70981d700 1 -- 172.21.2.112:0/1015476853 shutdown_connections 2021-02-10T11:48:40.289 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.269 7fa70981d700 1 --2- 172.21.2.112:0/1015476853 >> v2:172.21.2.120:3300/0 conn(0x7fa7040eb220 0x7fa7040eb620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:40.289 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.269 7fa70981d700 1 --2- 172.21.2.112:0/1015476853 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa7040dce80 0x7fa7040d14f0 unknown :-1 s=CLOSED pgs=3203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:40.289 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.269 7fa70981d700 1 -- 172.21.2.112:0/1015476853 >> 172.21.2.112:0/1015476853 conn(0x7fa7040ea630 msgr2=0x7fa7040d0800 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:40.289 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.269 7fa70981d700 1 -- 172.21.2.112:0/1015476853 shutdown_connections 2021-02-10T11:48:40.289 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.269 7fa70981d700 1 -- 172.21.2.112:0/1015476853 wait complete. 2021-02-10T11:48:40.291 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.284 7fa70981d700 1 Processor -- start 2021-02-10T11:48:40.291 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.284 7fa70981d700 1 -- start start 2021-02-10T11:48:40.291 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.284 7fa70981d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa7040fbd10 0x7fa7040fc110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:40.291 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.284 7fa70981d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa7040fc840 0x7fa7040fcc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:40.291 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.284 7fa70981d700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa70408a800 0x7fa70408ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:40.291 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.284 7fa70981d700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa70419eb50 con 0x7fa7040fbd10 2021-02-10T11:48:40.291 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.284 7fa70981d700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7041073d0 con 0x7fa70408a800 2021-02-10T11:48:40.291 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.284 7fa70981d700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7041075c0 con 0x7fa7040fc840 2021-02-10T11:48:40.291 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.284 7fa703d75700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa70408a800 0x7fa70408ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:40.291 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.284 7fa703d75700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa70408a800 0x7fa70408ac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3300/0 says I am v2:172.21.2.112:35496/0 (socket says 172.21.2.112:35496) 2021-02-10T11:48:40.291 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.284 7fa703d75700 1 -- 172.21.2.112:0/3938288150 learned_addr learned my addr 172.21.2.112:0/3938288150 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:48:40.292 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.284 7fa702d73700 1 --2- 172.21.2.112:0/3938288150 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa7040fc840 0x7fa7040fcc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:40.292 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.284 7fa703574700 1 --2- 172.21.2.112:0/3938288150 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa7040fbd10 0x7fa7040fc110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:40.292 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.285 7fa702d73700 1 -- 172.21.2.112:0/3938288150 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa70408a800 msgr2=0x7fa70408ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:40.292 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.285 7fa702d73700 1 --2- 172.21.2.112:0/3938288150 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa70408a800 0x7fa70408ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:40.292 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.285 7fa702d73700 1 -- 172.21.2.112:0/3938288150 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa7040fbd10 msgr2=0x7fa7040fc110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:40.292 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.285 7fa702d73700 1 --2- 172.21.2.112:0/3938288150 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa7040fbd10 0x7fa7040fc110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:40.292 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.285 7fa702d73700 1 -- 172.21.2.112:0/3938288150 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa70408b220 con 0x7fa7040fc840 2021-02-10T11:48:40.292 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.285 7fa703d75700 1 --2- 172.21.2.112:0/3938288150 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa70408a800 0x7fa70408ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:48:40.292 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.285 7fa703574700 1 --2- 172.21.2.112:0/3938288150 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa7040fbd10 0x7fa7040fc110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:48:40.292 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.285 7fa702d73700 1 --2- 172.21.2.112:0/3938288150 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa7040fc840 0x7fa7040fcc40 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7fa6ec009540 tx=0x7fa6ec00b8e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:40.292 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.285 7fa6fbfff700 1 -- 172.21.2.112:0/3938288150 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa6ec0063c0 con 0x7fa7040fc840 2021-02-10T11:48:40.292 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.285 7fa6fbfff700 1 -- 172.21.2.112:0/3938288150 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6ec00bd60 con 0x7fa7040fc840 2021-02-10T11:48:40.292 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.285 7fa70981d700 1 -- 172.21.2.112:0/3938288150 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa70408b220 con 0x7fa7040fc840 2021-02-10T11:48:40.293 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.285 7fa6fbfff700 1 -- 172.21.2.112:0/3938288150 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa6ec002030 con 0x7fa7040fc840 2021-02-10T11:48:40.293 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.285 7fa70981d700 1 -- 172.21.2.112:0/3938288150 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa70408b770 con 0x7fa7040fc840 2021-02-10T11:48:40.293 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.286 7fa6fbfff700 1 -- 172.21.2.112:0/3938288150 <== mon.2 v2:172.21.2.120:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa6ec000c90 con 0x7fa7040fc840 2021-02-10T11:48:40.293 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.286 7fa6fbfff700 1 --2- 172.21.2.112:0/3938288150 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa6e80186c0 0x7fa6e801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:40.293 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.286 7fa6fbfff700 1 -- 172.21.2.112:0/3938288150 <== mon.2 v2:172.21.2.120:3301/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fa6ec0324d0 con 0x7fa7040fc840 2021-02-10T11:48:40.293 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.286 7fa703574700 1 --2- 172.21.2.112:0/3938288150 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa6e80186c0 0x7fa6e801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:40.293 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.287 7fa703574700 1 --2- 172.21.2.112:0/3938288150 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa6e80186c0 0x7fa6e801abc0 crc :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:40.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.318 7fa70981d700 1 -- 172.21.2.112:0/3938288150 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7040f9390 con 0x7fa7040fc840 2021-02-10T11:48:40.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.321 7fa6fbfff700 1 -- 172.21.2.112:0/3938288150 <== mon.2 v2:172.21.2.120:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa6ec000940 con 0x7fa7040fc840 2021-02-10T11:48:40.455 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.452 7fa70981d700 1 -- 172.21.2.112:0/3938288150 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume deauthorize", "vol_name": "cephfs", "sub_name": "subvolume_0000000000563415", "target": ["mgr", ""], "auth_id": "guest2", "group_name": "group_0000000000578561"}) v1 -- 0x7fa70419eb50 con 0x7fa6e80186c0 2021-02-10T11:48:40.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.487 7fa6fbfff700 1 -- 172.21.2.112:0/3938288150 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fa6f4009270 con 0x7fa6e80186c0 2021-02-10T11:48:40.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.517 7fa70981d700 1 -- 172.21.2.112:0/3938288150 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa6e80186c0 msgr2=0x7fa6e801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:40.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.517 7fa70981d700 1 --2- 172.21.2.112:0/3938288150 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa6e80186c0 0x7fa6e801abc0 crc :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:40.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.517 7fa70981d700 1 -- 172.21.2.112:0/3938288150 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa7040fc840 msgr2=0x7fa7040fcc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:40.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.517 7fa70981d700 1 --2- 172.21.2.112:0/3938288150 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa7040fc840 0x7fa7040fcc40 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7fa6ec009540 tx=0x7fa6ec00b8e0).stop 2021-02-10T11:48:40.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.517 7fa70981d700 1 -- 172.21.2.112:0/3938288150 shutdown_connections 2021-02-10T11:48:40.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.517 7fa70981d700 1 --2- 172.21.2.112:0/3938288150 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa6e80186c0 0x7fa6e801abc0 unknown :-1 s=CLOSED pgs=1442 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:40.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.517 7fa70981d700 1 --2- 172.21.2.112:0/3938288150 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa70408a800 0x7fa70408ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:40.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.517 7fa70981d700 1 --2- 172.21.2.112:0/3938288150 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa7040fc840 0x7fa7040fcc40 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:40.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.517 7fa70981d700 1 --2- 172.21.2.112:0/3938288150 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa7040fbd10 0x7fa7040fc110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:40.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.517 7fa70981d700 1 -- 172.21.2.112:0/3938288150 >> 172.21.2.112:0/3938288150 conn(0x7fa70419f300 msgr2=0x7fa70419c5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:40.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.518 7fa70981d700 1 -- 172.21.2.112:0/3938288150 shutdown_connections 2021-02-10T11:48:40.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.518 7fa70981d700 1 -- 172.21.2.112:0/3938288150 wait complete. 2021-02-10T11:48:40.529 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000563415 --group_name group_0000000000578561 2021-02-10T11:48:40.962 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.941 7f6ce7d59700 1 Processor -- start 2021-02-10T11:48:40.963 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.942 7f6ce7d59700 1 -- start start 2021-02-10T11:48:40.963 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.942 7f6ce7d59700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6ce00d1880 0x7f6ce01b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:40.963 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.942 7f6ce7d59700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f6ce01b0af0 0x7f6ce0108950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:40.963 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.942 7f6ce7d59700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f6ce00d1c80 0x7f6ce0108f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:40.963 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.942 7f6ce7d59700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ce00e5f00 con 0x7f6ce00d1c80 2021-02-10T11:48:40.963 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.942 7f6ce7d59700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ce0109a70 con 0x7f6ce01b0af0 2021-02-10T11:48:40.963 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.942 7f6ce7d59700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ce0088070 con 0x7f6ce00d1880 2021-02-10T11:48:40.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.942 7f6ce6376700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f6ce00d1c80 0x7f6ce0108f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:40.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.942 7f6ce6376700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f6ce00d1c80 0x7f6ce0108f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:33222/0 (socket says 172.21.2.112:33222) 2021-02-10T11:48:40.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.942 7f6ce6376700 1 -- 172.21.2.112:0/3055031523 learned_addr learned my addr 172.21.2.112:0/3055031523 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:48:40.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.943 7f6ce5b75700 1 --2- 172.21.2.112:0/3055031523 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6ce00d1880 0x7f6ce01b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:40.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.943 7f6ce6376700 1 -- 172.21.2.112:0/3055031523 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6ce00d1880 msgr2=0x7f6ce01b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:40.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.943 7f6ce6376700 1 --2- 172.21.2.112:0/3055031523 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6ce00d1880 0x7f6ce01b04d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:40.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.943 7f6ce6376700 1 -- 172.21.2.112:0/3055031523 >> v2:172.21.2.120:3300/0 conn(0x7f6ce01b0af0 msgr2=0x7f6ce0108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:40.964 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.943 7f6ce5374700 1 --2- 172.21.2.112:0/3055031523 >> v2:172.21.2.120:3300/0 conn(0x7f6ce01b0af0 0x7f6ce0108950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:40.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.943 7f6ce6376700 1 --2- 172.21.2.112:0/3055031523 >> v2:172.21.2.120:3300/0 conn(0x7f6ce01b0af0 0x7f6ce0108950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:40.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.943 7f6ce6376700 1 -- 172.21.2.112:0/3055031523 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ce0109590 con 0x7f6ce00d1c80 2021-02-10T11:48:40.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.943 7f6ce5b75700 1 --2- 172.21.2.112:0/3055031523 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6ce00d1880 0x7f6ce01b04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:48:40.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.943 7f6ce5374700 1 --2- 172.21.2.112:0/3055031523 >> v2:172.21.2.120:3300/0 conn(0x7f6ce01b0af0 0x7f6ce0108950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:48:40.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.943 7f6ce6376700 1 --2- 172.21.2.112:0/3055031523 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6ce00d1c80 0x7f6ce0108f70 secure :-1 s=READY pgs=3204 cs=0 l=1 rev1=1 rx=0x7f6cd8006f00 tx=0x7f6cd8009f10).ready entity=mon.0 client_cookie=3f89507921e03418 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:40.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.943 7f6ce4b73700 1 -- 172.21.2.112:0/3055031523 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6cd800b3b0 con 0x7f6ce00d1c80 2021-02-10T11:48:40.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.943 7f6ce4b73700 1 -- 172.21.2.112:0/3055031523 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cd8000d30 con 0x7f6ce00d1c80 2021-02-10T11:48:40.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.943 7f6ce4b73700 1 -- 172.21.2.112:0/3055031523 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6cd800b820 con 0x7f6ce00d1c80 2021-02-10T11:48:40.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.943 7f6ce7d59700 1 -- 172.21.2.112:0/3055031523 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6ce00d1c80 msgr2=0x7f6ce0108f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:40.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.943 7f6ce7d59700 1 --2- 172.21.2.112:0/3055031523 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6ce00d1c80 0x7f6ce0108f70 secure :-1 s=READY pgs=3204 cs=0 l=1 rev1=1 rx=0x7f6cd8006f00 tx=0x7f6cd8009f10).stop 2021-02-10T11:48:40.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.944 7f6ce7d59700 1 -- 172.21.2.112:0/3055031523 shutdown_connections 2021-02-10T11:48:40.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.944 7f6ce7d59700 1 --2- 172.21.2.112:0/3055031523 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6ce00d1c80 0x7f6ce0108f70 unknown :-1 s=CLOSED pgs=3204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:40.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.944 7f6ce7d59700 1 --2- 172.21.2.112:0/3055031523 >> v2:172.21.2.120:3300/0 conn(0x7f6ce01b0af0 0x7f6ce0108950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:40.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.944 7f6ce7d59700 1 --2- 172.21.2.112:0/3055031523 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6ce00d1880 0x7f6ce01b04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:40.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.944 7f6ce7d59700 1 -- 172.21.2.112:0/3055031523 >> 172.21.2.112:0/3055031523 conn(0x7f6ce00ce430 msgr2=0x7f6ce00d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:40.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.944 7f6ce7d59700 1 -- 172.21.2.112:0/3055031523 shutdown_connections 2021-02-10T11:48:40.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.944 7f6ce7d59700 1 -- 172.21.2.112:0/3055031523 wait complete. 2021-02-10T11:48:40.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.959 7f6ce7d59700 1 Processor -- start 2021-02-10T11:48:40.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.959 7f6ce7d59700 1 -- start start 2021-02-10T11:48:40.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.960 7f6ce7d59700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6ce00bbf70 0x7f6ce00a7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:40.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.960 7f6ce7d59700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6ce00a81f0 0x7f6ce00bc820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:40.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.960 7f6ce7d59700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6ce00bce40 0x7f6ce00ba730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:40.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.960 7f6ce7d59700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ce00bc490 con 0x7f6ce00bbf70 2021-02-10T11:48:40.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.960 7f6ce7d59700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ce00bad50 con 0x7f6ce00bce40 2021-02-10T11:48:40.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.960 7f6ce7d59700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ce00baf70 con 0x7f6ce00a81f0 2021-02-10T11:48:40.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.960 7f6ce5b75700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6ce00bbf70 0x7f6ce00a7bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:40.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.960 7f6ce5b75700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6ce00bbf70 0x7f6ce00a7bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:33224/0 (socket says 172.21.2.112:33224) 2021-02-10T11:48:40.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.960 7f6ce5b75700 1 -- 172.21.2.112:0/4167706392 learned_addr learned my addr 172.21.2.112:0/4167706392 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:48:40.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.960 7f6ce5374700 1 --2- 172.21.2.112:0/4167706392 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6ce00a81f0 0x7f6ce00bc820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:40.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.960 7f6ce6376700 1 --2- 172.21.2.112:0/4167706392 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6ce00bce40 0x7f6ce00ba730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:40.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.960 7f6ce5b75700 1 -- 172.21.2.112:0/4167706392 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6ce00a81f0 msgr2=0x7f6ce00bc820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:40.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.960 7f6ce5b75700 1 --2- 172.21.2.112:0/4167706392 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6ce00a81f0 0x7f6ce00bc820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:40.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.960 7f6ce5b75700 1 -- 172.21.2.112:0/4167706392 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6ce00bce40 msgr2=0x7f6ce00ba730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:40.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.960 7f6ce5b75700 1 --2- 172.21.2.112:0/4167706392 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6ce00bce40 0x7f6ce00ba730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:40.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.960 7f6ce5b75700 1 -- 172.21.2.112:0/4167706392 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ce00bb190 con 0x7f6ce00bbf70 2021-02-10T11:48:40.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.960 7f6ce5b75700 1 --2- 172.21.2.112:0/4167706392 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6ce00bbf70 0x7f6ce00a7bd0 secure :-1 s=READY pgs=3205 cs=0 l=1 rev1=1 rx=0x7f6cd4006f20 tx=0x7f6cd400b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:40.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.961 7f6cd27fc700 1 -- 172.21.2.112:0/4167706392 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6cd400bd80 con 0x7f6ce00bbf70 2021-02-10T11:48:40.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.961 7f6cd27fc700 1 -- 172.21.2.112:0/4167706392 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cd4000c90 con 0x7f6ce00bbf70 2021-02-10T11:48:40.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.961 7f6cd27fc700 1 -- 172.21.2.112:0/4167706392 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6cd40022b0 con 0x7f6ce00bbf70 2021-02-10T11:48:40.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.961 7f6ce7d59700 1 -- 172.21.2.112:0/4167706392 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ce00bb190 con 0x7f6ce00bbf70 2021-02-10T11:48:40.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.961 7f6ce7d59700 1 -- 172.21.2.112:0/4167706392 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ce00bb190 con 0x7f6ce00bbf70 2021-02-10T11:48:40.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.961 7f6cd27fc700 1 -- 172.21.2.112:0/4167706392 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6cd400a000 con 0x7f6ce00bbf70 2021-02-10T11:48:40.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.962 7f6cd27fc700 1 --2- 172.21.2.112:0/4167706392 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6cc8018730 0x7f6cc801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:40.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.962 7f6cd27fc700 1 -- 172.21.2.112:0/4167706392 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f6cd4032790 con 0x7f6ce00bbf70 2021-02-10T11:48:40.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.962 7f6ce5374700 1 --2- 172.21.2.112:0/4167706392 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6cc8018730 0x7f6cc801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:40.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.962 7f6ce5374700 1 --2- 172.21.2.112:0/4167706392 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6cc8018730 0x7f6cc801ac10 crc :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:40.995 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.992 7f6ce7d59700 1 -- 172.21.2.112:0/4167706392 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ce00a76a0 con 0x7f6ce00bbf70 2021-02-10T11:48:40.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:40.995 7f6cd27fc700 1 -- 172.21.2.112:0/4167706392 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6cd4006bb0 con 0x7f6ce00bbf70 2021-02-10T11:48:41.124 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:41.121 7f6ce7d59700 1 -- 172.21.2.112:0/4167706392 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000563415", "target": ["mgr", ""], "group_name": "group_0000000000578561"}) v1 -- 0x7f6ce00a76a0 con 0x7f6cc8018730 2021-02-10T11:48:41.134 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:41.131 7f6cd27fc700 1 -- 172.21.2.112:0/4167706392 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f6cdc006540 con 0x7f6cc8018730 2021-02-10T11:48:41.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:41.137 7f6ce7d59700 1 -- 172.21.2.112:0/4167706392 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6cc8018730 msgr2=0x7f6cc801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:41.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:41.137 7f6ce7d59700 1 --2- 172.21.2.112:0/4167706392 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6cc8018730 0x7f6cc801ac10 crc :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:41.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:41.137 7f6ce7d59700 1 -- 172.21.2.112:0/4167706392 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6ce00bbf70 msgr2=0x7f6ce00a7bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:41.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:41.137 7f6ce7d59700 1 --2- 172.21.2.112:0/4167706392 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6ce00bbf70 0x7f6ce00a7bd0 secure :-1 s=READY pgs=3205 cs=0 l=1 rev1=1 rx=0x7f6cd4006f20 tx=0x7f6cd400b990).stop 2021-02-10T11:48:41.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:41.138 7f6ce7d59700 1 -- 172.21.2.112:0/4167706392 shutdown_connections 2021-02-10T11:48:41.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:41.138 7f6ce7d59700 1 --2- 172.21.2.112:0/4167706392 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6cc8018730 0x7f6cc801ac10 unknown :-1 s=CLOSED pgs=1443 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:41.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:41.138 7f6ce7d59700 1 --2- 172.21.2.112:0/4167706392 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6ce00bce40 0x7f6ce00ba730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:41.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:41.138 7f6ce7d59700 1 --2- 172.21.2.112:0/4167706392 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6ce00a81f0 0x7f6ce00bc820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:41.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:41.138 7f6ce7d59700 1 --2- 172.21.2.112:0/4167706392 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6ce00bbf70 0x7f6ce00a7bd0 unknown :-1 s=CLOSED pgs=3205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:41.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:41.138 7f6ce7d59700 1 -- 172.21.2.112:0/4167706392 >> 172.21.2.112:0/4167706392 conn(0x7f6ce009f5c0 msgr2=0x7f6ce00b2ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:41.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:41.138 7f6ce7d59700 1 -- 172.21.2.112:0/4167706392 shutdown_connections 2021-02-10T11:48:41.142 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:41.138 7f6ce7d59700 1 -- 172.21.2.112:0/4167706392 wait complete. 2021-02-10T11:48:41.149 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolumegroup rm cephfs group_0000000000578561 2021-02-10T11:48:41.583 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:41.563 7effc80f1700 1 Processor -- start 2021-02-10T11:48:41.583 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:41.563 7effc80f1700 1 -- start start 2021-02-10T11:48:41.583 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:41.563 7effc80f1700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7effc00d1870 0x7effc01b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:41.583 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:41.563 7effc80f1700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7effc01b0ae0 0x7effc0108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:41.583 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:41.563 7effc80f1700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effc0109a60 con 0x7effc00d1c70 2021-02-10T11:48:41.583 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:41.563 7effc80f1700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7effc00e5ef0 con 0x7effc01b0ae0 2021-02-10T11:48:41.583 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:41.563 7effc80f1700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effc0109750 con 0x7effc00d1870 2021-02-10T11:48:41.583 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:41.564 7effc670e700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7effc00d1c70 0x7effc0108f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:54484/0 (socket says 172.21.2.112:54484) 2021-02-10T11:48:41.583 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:41.564 7effc670e700 1 -- 172.21.2.112:0/322446211 learned_addr learned my addr 172.21.2.112:0/322446211 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:48:41.583 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:41.564 7effc570c700 1 --2- 172.21.2.112:0/322446211 >> v2:172.21.2.120:3300/0 conn(0x7effc01b0ae0 0x7effc0108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:41.584 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:41.564 7effc5f0d700 1 --2- 172.21.2.112:0/322446211 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7effc00d1870 0x7effc01b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:41.584 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:41.564 7effc570c700 1 -- 172.21.2.112:0/322446211 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7effc00d1870 msgr2=0x7effc01b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:41.584 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:41.564 7effc570c700 1 --2- 172.21.2.112:0/322446211 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7effc00d1870 0x7effc01b04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:41.584 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:41.564 7effc570c700 1 -- 172.21.2.112:0/322446211 >> v1:172.21.2.120:6789/0 conn(0x7effc00d1c70 legacy=0x7effc0108f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:41.584 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:41.564 7effc570c700 1 -- 172.21.2.112:0/322446211 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effc0088070 con 0x7effc01b0ae0 2021-02-10T11:48:41.584 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:41.564 7effc570c700 1 --2- 172.21.2.112:0/322446211 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7effc01b0ae0 0x7effc0108940 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 rx=0x7effbc006650 tx=0x7effbc009a80).ready entity=mon.1 client_cookie=b09d209ef51f8d50 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:41.584 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:41.564 7effc4f0b700 1 -- 172.21.2.112:0/322446211 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7effbc00c0c0 con 0x7effc01b0ae0 2021-02-10T11:48:41.584 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:41.564 7effc4f0b700 1 -- 172.21.2.112:0/322446211 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effbc00c340 con 0x7effc01b0ae0 2021-02-10T11:48:41.584 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:41.564 7effc4f0b700 1 -- 172.21.2.112:0/322446211 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7effbc00c820 con 0x7effc01b0ae0 2021-02-10T11:48:41.584 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:41.565 7effc80f1700 1 -- 172.21.2.112:0/322446211 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7effc01b0ae0 msgr2=0x7effc0108940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:41.584 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:41.565 7effc80f1700 1 --2- 172.21.2.112:0/322446211 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7effc01b0ae0 0x7effc0108940 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 rx=0x7effbc006650 tx=0x7effbc009a80).stop 2021-02-10T11:48:41.584 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:41.565 7effc80f1700 1 -- 172.21.2.112:0/322446211 shutdown_connections 2021-02-10T11:48:41.584 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:41.565 7effc80f1700 1 --2- 172.21.2.112:0/322446211 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7effc01b0ae0 0x7effc0108940 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:41.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:41.565 7effc80f1700 1 --2- 172.21.2.112:0/322446211 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7effc00d1870 0x7effc01b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:41.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:41.565 7effc80f1700 1 -- 172.21.2.112:0/322446211 >> 172.21.2.112:0/322446211 conn(0x7effc00ce420 msgr2=0x7effc00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:41.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:41.565 7effc80f1700 1 -- 172.21.2.112:0/322446211 shutdown_connections 2021-02-10T11:48:41.585 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:41.565 7effc80f1700 1 -- 172.21.2.112:0/322446211 wait complete. 2021-02-10T11:48:41.586 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:41.580 7effc80f1700 1 Processor -- start 2021-02-10T11:48:41.586 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:41.580 7effc80f1700 1 -- start start 2021-02-10T11:48:41.586 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:41.580 7effc80f1700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7effc00a58c0 0x7effc00a4d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:41.587 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:41.580 7effc80f1700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7effc00b1ba0 0x7effc00b1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:41.587 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:41.580 7effc80f1700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7effc00b25c0 0x7effc00b3270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:41.587 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:41.580 7effc80f1700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effc00b2a40 con 0x7effc00a58c0 2021-02-10T11:48:41.587 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:41.580 7effc80f1700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effc00a5340 con 0x7effc00b1ba0 2021-02-10T11:48:41.587 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:41.580 7effc80f1700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effc00a5560 con 0x7effc00b25c0 2021-02-10T11:48:41.587 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:41.580 7effc5f0d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7effc00a58c0 0x7effc00a4d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:41.587 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:41.580 7effc5f0d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7effc00a58c0 0x7effc00a4d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:33234/0 (socket says 172.21.2.112:33234) 2021-02-10T11:48:41.587 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:41.580 7effc5f0d700 1 -- 172.21.2.112:0/1330560972 learned_addr learned my addr 172.21.2.112:0/1330560972 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:48:41.587 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:41.581 7effc5f0d700 1 -- 172.21.2.112:0/1330560972 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7effc00b25c0 msgr2=0x7effc00b3270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:41.587 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:41.581 7effc5f0d700 1 --2- 172.21.2.112:0/1330560972 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7effc00b25c0 0x7effc00b3270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:41.587 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:41.581 7effc5f0d700 1 -- 172.21.2.112:0/1330560972 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7effc00b1ba0 msgr2=0x7effc00b1fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:41.587 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:41.581 7effc5f0d700 1 --2- 172.21.2.112:0/1330560972 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7effc00b1ba0 0x7effc00b1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:41.587 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:41.581 7effc5f0d700 1 -- 172.21.2.112:0/1330560972 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effc00b3890 con 0x7effc00a58c0 2021-02-10T11:48:41.588 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:41.581 7effc5f0d700 1 --2- 172.21.2.112:0/1330560972 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7effc00a58c0 0x7effc00a4d20 secure :-1 s=READY pgs=3206 cs=0 l=1 rev1=1 rx=0x7effb4006f60 tx=0x7effb400b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:41.588 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:41.581 7effb27fc700 1 -- 172.21.2.112:0/1330560972 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7effb400bd60 con 0x7effc00a58c0 2021-02-10T11:48:41.588 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:41.581 7effb27fc700 1 -- 172.21.2.112:0/1330560972 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effb4000c90 con 0x7effc00a58c0 2021-02-10T11:48:41.588 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:41.581 7effb27fc700 1 -- 172.21.2.112:0/1330560972 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7effb40022b0 con 0x7effc00a58c0 2021-02-10T11:48:41.588 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:41.581 7effc80f1700 1 -- 172.21.2.112:0/1330560972 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effc00b3890 con 0x7effc00a58c0 2021-02-10T11:48:41.588 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:41.581 7effc80f1700 1 -- 172.21.2.112:0/1330560972 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effc00b3890 con 0x7effc00a58c0 2021-02-10T11:48:41.588 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:41.582 7effb27fc700 1 -- 172.21.2.112:0/1330560972 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7effb4006b00 con 0x7effc00a58c0 2021-02-10T11:48:41.588 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:41.582 7effb27fc700 1 --2- 172.21.2.112:0/1330560972 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7effa80186c0 0x7effa801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:41.588 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:41.582 7effb27fc700 1 -- 172.21.2.112:0/1330560972 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 5573+0+0 (secure 0 0 0) 0x7effb40323e0 con 0x7effc00a58c0 2021-02-10T11:48:41.588 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:41.582 7effc570c700 1 --2- 172.21.2.112:0/1330560972 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7effa80186c0 0x7effa801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:41.588 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:41.583 7effc570c700 1 --2- 172.21.2.112:0/1330560972 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7effa80186c0 0x7effa801abc0 crc :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:41.616 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:41.613 7effc80f1700 1 -- 172.21.2.112:0/1330560972 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effc0082980 con 0x7effc00a58c0 2021-02-10T11:48:41.619 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:41.616 7effb27fc700 1 -- 172.21.2.112:0/1330560972 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7effb4015ff0 con 0x7effc00a58c0 2021-02-10T11:48:41.747 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:41.743 7effc80f1700 1 -- 172.21.2.112:0/1330560972 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolumegroup rm", "vol_name": "cephfs", "target": ["mgr", ""], "group_name": "group_0000000000578561"}) v1 -- 0x7effc00b3890 con 0x7effa80186c0 2021-02-10T11:48:41.751 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:41.747 7effb27fc700 1 -- 172.21.2.112:0/1330560972 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7effbc000940 con 0x7effa80186c0 2021-02-10T11:48:41.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:41.753 7effc80f1700 1 -- 172.21.2.112:0/1330560972 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7effa80186c0 msgr2=0x7effa801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:41.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:41.753 7effc80f1700 1 --2- 172.21.2.112:0/1330560972 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7effa80186c0 0x7effa801abc0 crc :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:41.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:41.753 7effc80f1700 1 -- 172.21.2.112:0/1330560972 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7effc00a58c0 msgr2=0x7effc00a4d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:41.757 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:41.753 7effc80f1700 1 --2- 172.21.2.112:0/1330560972 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7effc00a58c0 0x7effc00a4d20 secure :-1 s=READY pgs=3206 cs=0 l=1 rev1=1 rx=0x7effb4006f60 tx=0x7effb400b970).stop 2021-02-10T11:48:41.757 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:41.753 7effc80f1700 1 -- 172.21.2.112:0/1330560972 shutdown_connections 2021-02-10T11:48:41.757 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:41.753 7effc80f1700 1 --2- 172.21.2.112:0/1330560972 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7effa80186c0 0x7effa801abc0 unknown :-1 s=CLOSED pgs=1444 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:41.757 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:41.753 7effc80f1700 1 --2- 172.21.2.112:0/1330560972 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7effc00b1ba0 0x7effc00b1fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:41.757 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:41.753 7effc80f1700 1 --2- 172.21.2.112:0/1330560972 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7effc00b25c0 0x7effc00b3270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:41.757 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:41.753 7effc80f1700 1 --2- 172.21.2.112:0/1330560972 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7effc00a58c0 0x7effc00a4d20 unknown :-1 s=CLOSED pgs=3206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:41.757 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:41.753 7effc80f1700 1 -- 172.21.2.112:0/1330560972 >> 172.21.2.112:0/1330560972 conn(0x7effc00b5dc0 msgr2=0x7effc0095960 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:41.757 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:41.753 7effc80f1700 1 -- 172.21.2.112:0/1330560972 shutdown_connections 2021-02-10T11:48:41.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:41.753 7effc80f1700 1 -- 172.21.2.112:0/1330560972 wait complete. 2021-02-10T11:48:41.765 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:48:41.765 DEBUG:teuthology.orchestra.run.gibba012:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:48:42.112 DEBUG:teuthology.orchestra.run.gibba020:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:48:42.456 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:48:42.456 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:48:42.456 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:48:42.457 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:48:42.793 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:48:42.793 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:48:42.794 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba012.front.sepia.ceph.com... 2021-02-10T11:48:42.794 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:48:42.794 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:48:43.146 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba012.stderr:ceph-fuse[192408]: fuse finished with error 0 and tester_r 0 2021-02-10T11:48:43.148 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:48:43.148 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:48:43.148 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:48:43.484 INFO:teuthology.orchestra.run.gibba012.stdout:ext2/ext3 2021-02-10T11:48:43.484 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:48:43.484 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:48:43.484 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:48:43.820 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:48:43.821 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:48:43.821 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:48:44.157 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:48:44.158 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:48:44.158 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba012.front.sepia.ceph.com... 2021-02-10T11:48:44.158 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:48:44.158 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:48:44.513 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba012.stderr:ceph-fuse[193075]: fuse finished with error 0 and tester_r 0 2021-02-10T11:48:44.515 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:48:44.515 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:48:44.515 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:48:44.851 INFO:teuthology.orchestra.run.gibba012.stdout:ext2/ext3 2021-02-10T11:48:44.851 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:48:44.851 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:48:44.852 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:48:45.187 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:48:45.187 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:48:45.187 DEBUG:teuthology.orchestra.run.gibba020:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:48:45.527 INFO:teuthology.orchestra.run.gibba020.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:48:45.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:48:45.528 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:48:45.528 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:48:45.528 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:48:45.863 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:48:45.863 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:48:45.863 DEBUG:teuthology.orchestra.run.gibba020:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:48:46.197 INFO:teuthology.orchestra.run.gibba020.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:48:46.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:48:46.197 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:48:46.198 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:48:46.198 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:48:46.531 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config rm mon mon_allow_pool_delete 2021-02-10T11:48:46.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:46.945 7fb3d8044700 1 Processor -- start 2021-02-10T11:48:46.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:46.945 7fb3d8044700 1 -- start start 2021-02-10T11:48:46.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:46.945 7fb3d8044700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fb3d00d0340 0x7fb3d00ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:46.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:46.946 7fb3d8044700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3d00f0b10 con 0x7fb3d00d1870 2021-02-10T11:48:46.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:46.946 7fb3d8044700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3d01b0be0 con 0x7fb3d00d1c70 2021-02-10T11:48:46.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:46.946 7fb3d8044700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3d00e5ed0 con 0x7fb3d00d0340 2021-02-10T11:48:46.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:46.946 7fb3d565f700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fb3d00d0340 0x7fb3d00ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:46.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:46.946 7fb3d5e60700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fb3d00d1870 0x7fb3d01b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:60936/0 (socket says 172.21.2.112:60936) 2021-02-10T11:48:46.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:46.946 7fb3d6661700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7fb3d00d1c70 0x7fb3d00efc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:54494/0 (socket says 172.21.2.112:54494) 2021-02-10T11:48:46.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:46.946 7fb3d5e60700 1 -- 172.21.2.112:0/4214858309 learned_addr learned my addr 172.21.2.112:0/4214858309 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:48:46.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:46.946 7fb3d565f700 1 -- 172.21.2.112:0/4214858309 >> v1:172.21.2.120:6789/0 conn(0x7fb3d00d1c70 legacy=0x7fb3d00efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:46.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:46.946 7fb3d565f700 1 -- 172.21.2.112:0/4214858309 >> v1:172.21.2.112:6789/0 conn(0x7fb3d00d1870 legacy=0x7fb3d01b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:46.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:46.946 7fb3d4e5e700 1 -- 172.21.2.112:0/4214858309 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3783028525 0 0) 0x7fb3cc0010b0 con 0x7fb3d00d1870 2021-02-10T11:48:46.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:46.946 7fb3d565f700 1 -- 172.21.2.112:0/4214858309 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3d00f0450 con 0x7fb3d00d0340 2021-02-10T11:48:46.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:46.946 7fb3d565f700 1 --2- 172.21.2.112:0/4214858309 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb3d00d0340 0x7fb3d00ef640 secure :-1 s=READY pgs=3208 cs=0 l=1 rev1=1 rx=0x7fb3c0006650 tx=0x7fb3c0009a80).ready entity=mon.0 client_cookie=1ae581c737c0f769 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:46.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:46.947 7fb3d4e5e700 1 -- 172.21.2.112:0/4214858309 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb3c00009b0 con 0x7fb3d00d0340 2021-02-10T11:48:46.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:46.947 7fb3d4e5e700 1 -- 172.21.2.112:0/4214858309 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3c000c0c0 con 0x7fb3d00d0340 2021-02-10T11:48:46.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:46.947 7fb3d4e5e700 1 -- 172.21.2.112:0/4214858309 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb3c000c5e0 con 0x7fb3d00d0340 2021-02-10T11:48:46.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:46.947 7fb3d8044700 1 -- 172.21.2.112:0/4214858309 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb3d00d0340 msgr2=0x7fb3d00ef640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:46.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:46.947 7fb3d8044700 1 --2- 172.21.2.112:0/4214858309 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb3d00d0340 0x7fb3d00ef640 secure :-1 s=READY pgs=3208 cs=0 l=1 rev1=1 rx=0x7fb3c0006650 tx=0x7fb3c0009a80).stop 2021-02-10T11:48:46.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:46.947 7fb3d8044700 1 -- 172.21.2.112:0/4214858309 shutdown_connections 2021-02-10T11:48:46.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:46.947 7fb3d8044700 1 --2- 172.21.2.112:0/4214858309 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb3d00d0340 0x7fb3d00ef640 unknown :-1 s=CLOSED pgs=3208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:46.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:46.947 7fb3d8044700 1 -- 172.21.2.112:0/4214858309 >> 172.21.2.112:0/4214858309 conn(0x7fb3d00ce450 msgr2=0x7fb3d00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:46.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:46.948 7fb3d8044700 1 -- 172.21.2.112:0/4214858309 shutdown_connections 2021-02-10T11:48:46.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:46.948 7fb3d8044700 1 -- 172.21.2.112:0/4214858309 wait complete. 2021-02-10T11:48:46.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:46.962 7fb3d8044700 1 Processor -- start 2021-02-10T11:48:46.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:46.962 7fb3d8044700 1 -- start start 2021-02-10T11:48:46.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:46.963 7fb3d8044700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb3d009bf50 0x7fb3d009c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:46.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:46.963 7fb3d8044700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb3d009c350 0x7fb3d00ac5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:46.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:46.963 7fb3d8044700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb3d009cfc0 0x7fb3d00acbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:46.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:46.963 7fb3d8044700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3d00ad210 con 0x7fb3d009cfc0 2021-02-10T11:48:46.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:46.963 7fb3d8044700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3d00ad400 con 0x7fb3d009c350 2021-02-10T11:48:46.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:46.963 7fb3d8044700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3d00ad620 con 0x7fb3d009bf50 2021-02-10T11:48:46.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:46.963 7fb3d6661700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb3d009cfc0 0x7fb3d00acbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:46.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:46.963 7fb3d5e60700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb3d009bf50 0x7fb3d009c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:46.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:46.963 7fb3d565f700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb3d009c350 0x7fb3d00ac5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:46.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:46.963 7fb3d6661700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb3d009cfc0 0x7fb3d00acbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:33250/0 (socket says 172.21.2.112:33250) 2021-02-10T11:48:46.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:46.963 7fb3d6661700 1 -- 172.21.2.112:0/3293199128 learned_addr learned my addr 172.21.2.112:0/3293199128 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:48:46.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:46.963 7fb3d5e60700 1 -- 172.21.2.112:0/3293199128 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb3d009c350 msgr2=0x7fb3d00ac5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:46.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:46.963 7fb3d5e60700 1 --2- 172.21.2.112:0/3293199128 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb3d009c350 0x7fb3d00ac5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:46.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:46.963 7fb3d5e60700 1 -- 172.21.2.112:0/3293199128 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb3d009cfc0 msgr2=0x7fb3d00acbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:46.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:46.963 7fb3d5e60700 1 --2- 172.21.2.112:0/3293199128 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb3d009cfc0 0x7fb3d00acbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:46.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:46.963 7fb3d5e60700 1 -- 172.21.2.112:0/3293199128 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3d00ada80 con 0x7fb3d009bf50 2021-02-10T11:48:46.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:46.964 7fb3d5e60700 1 --2- 172.21.2.112:0/3293199128 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb3d009bf50 0x7fb3d009c9a0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 rx=0x7fb3cc001c60 tx=0x7fb3cc00a870).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:46.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:46.964 7fb3c67fc700 1 -- 172.21.2.112:0/3293199128 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb3cc0009b0 con 0x7fb3d009bf50 2021-02-10T11:48:46.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:46.964 7fb3c67fc700 1 -- 172.21.2.112:0/3293199128 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3cc00ac60 con 0x7fb3d009bf50 2021-02-10T11:48:46.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:46.964 7fb3c67fc700 1 -- 172.21.2.112:0/3293199128 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb3cc0045d0 con 0x7fb3d009bf50 2021-02-10T11:48:46.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:46.964 7fb3d8044700 1 -- 172.21.2.112:0/3293199128 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3d00ada80 con 0x7fb3d009bf50 2021-02-10T11:48:46.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:46.964 7fb3d8044700 1 -- 172.21.2.112:0/3293199128 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3d00ada80 con 0x7fb3d009bf50 2021-02-10T11:48:46.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:46.965 7fb3c67fc700 1 -- 172.21.2.112:0/3293199128 <== mon.2 v2:172.21.2.120:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb3cc004140 con 0x7fb3d009bf50 2021-02-10T11:48:46.971 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:46.965 7fb3c67fc700 1 --2- 172.21.2.112:0/3293199128 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb3b80186c0 0x7fb3b801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:46.971 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:46.965 7fb3c67fc700 1 -- 172.21.2.112:0/3293199128 <== mon.2 v2:172.21.2.120:3301/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fb3cc0326b0 con 0x7fb3d009bf50 2021-02-10T11:48:46.971 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:46.965 7fb3d565f700 1 --2- 172.21.2.112:0/3293199128 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb3b80186c0 0x7fb3b801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:46.971 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:46.965 7fb3d565f700 1 --2- 172.21.2.112:0/3293199128 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb3b80186c0 0x7fb3b801abc0 crc :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:46.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:46.996 7fb3d8044700 1 -- 172.21.2.112:0/3293199128 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3d00ce450 con 0x7fb3d009bf50 2021-02-10T11:48:47.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:46.999 7fb3c67fc700 1 -- 172.21.2.112:0/3293199128 <== mon.2 v2:172.21.2.120:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb3cc000e20 con 0x7fb3d009bf50 2021-02-10T11:48:47.156 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:47.153 7fb3d8044700 1 -- 172.21.2.112:0/3293199128 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fb3d00ce450 con 0x7fb3d009bf50 2021-02-10T11:48:47.161 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:47.157 7fb3c67fc700 1 -- 172.21.2.112:0/3293199128 <== mon.2 v2:172.21.2.120:3301/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v47) v1 ==== 104+0+0 (secure 0 0 0) 0x7fb3cc004140 con 0x7fb3d009bf50 2021-02-10T11:48:47.165 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:47.161 7fb3d8044700 1 -- 172.21.2.112:0/3293199128 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb3b80186c0 msgr2=0x7fb3b801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:47.165 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:47.161 7fb3d8044700 1 --2- 172.21.2.112:0/3293199128 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb3b80186c0 0x7fb3b801abc0 crc :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:47.165 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:47.161 7fb3d8044700 1 -- 172.21.2.112:0/3293199128 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb3d009bf50 msgr2=0x7fb3d009c9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:47.165 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:47.161 7fb3d8044700 1 --2- 172.21.2.112:0/3293199128 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb3d009bf50 0x7fb3d009c9a0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 rx=0x7fb3cc001c60 tx=0x7fb3cc00a870).stop 2021-02-10T11:48:47.165 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:47.161 7fb3d8044700 1 -- 172.21.2.112:0/3293199128 shutdown_connections 2021-02-10T11:48:47.165 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:47.161 7fb3d8044700 1 --2- 172.21.2.112:0/3293199128 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb3b80186c0 0x7fb3b801abc0 unknown :-1 s=CLOSED pgs=1445 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:47.165 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:47.161 7fb3d8044700 1 --2- 172.21.2.112:0/3293199128 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb3d009c350 0x7fb3d00ac5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:47.165 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:47.161 7fb3d8044700 1 --2- 172.21.2.112:0/3293199128 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb3d009cfc0 0x7fb3d00acbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:47.166 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:47.161 7fb3d8044700 1 --2- 172.21.2.112:0/3293199128 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb3d009bf50 0x7fb3d009c9a0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:47.166 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:47.161 7fb3d8044700 1 -- 172.21.2.112:0/3293199128 >> 172.21.2.112:0/3293199128 conn(0x7fb3d00a6620 msgr2=0x7fb3d00b17a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:47.166 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:47.161 7fb3d8044700 1 -- 172.21.2.112:0/3293199128 shutdown_connections 2021-02-10T11:48:47.166 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:47.161 7fb3d8044700 1 -- 172.21.2.112:0/3293199128 wait complete. 2021-02-10T11:48:47.174 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestVolumes.test_subvolume_authorized_list' 2021-02-10T11:48:47.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:47.583 7ff387c31700 1 Processor -- start 2021-02-10T11:48:47.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:47.583 7ff387c31700 1 -- start start 2021-02-10T11:48:47.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:47.584 7ff387c31700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7ff3800d1870 0x7ff3801b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:47.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:47.584 7ff387c31700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff3800f0650 0x7ff3800cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:47.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:47.584 7ff387c31700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3800f0b10 con 0x7ff3801b0a10 2021-02-10T11:48:47.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:47.584 7ff387c31700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3800e5ed0 con 0x7ff3800d1870 2021-02-10T11:48:47.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:47.584 7ff387c31700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3800d1c70 con 0x7ff3800f0650 2021-02-10T11:48:47.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:47.584 7ff385a4d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7ff3800d1870 0x7ff3801b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:47.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:47.584 7ff385a4d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7ff3800d1870 0x7ff3801b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:33254/0 (socket says 172.21.2.112:33254) 2021-02-10T11:48:47.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:47.584 7ff385a4d700 1 -- 172.21.2.112:0/52231255 learned_addr learned my addr 172.21.2.112:0/52231255 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:48:47.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:47.584 7ff385a4d700 1 -- 172.21.2.112:0/52231255 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff3800f0650 msgr2=0x7ff3800cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:47.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:47.584 7ff385a4d700 1 --2- 172.21.2.112:0/52231255 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff3800f0650 0x7ff3800cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:47.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:47.584 7ff385a4d700 1 -- 172.21.2.112:0/52231255 >> v1:172.21.2.120:6789/0 conn(0x7ff3801b0a10 legacy=0x7ff3800cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:47.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:47.585 7ff385a4d700 1 -- 172.21.2.112:0/52231255 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3800d1e90 con 0x7ff3800d1870 2021-02-10T11:48:47.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:47.585 7ff385a4d700 1 --2- 172.21.2.112:0/52231255 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff3800d1870 0x7ff3801b03f0 secure :-1 s=READY pgs=3209 cs=0 l=1 rev1=1 rx=0x7ff37c006700 tx=0x7ff37c009b80).ready entity=mon.0 client_cookie=5cb99d39c91d867a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:47.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:47.585 7ff384a4b700 1 -- 172.21.2.112:0/52231255 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff37c00c0c0 con 0x7ff3800d1870 2021-02-10T11:48:47.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:47.585 7ff384a4b700 1 -- 172.21.2.112:0/52231255 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff37c00c340 con 0x7ff3800d1870 2021-02-10T11:48:47.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:47.585 7ff384a4b700 1 -- 172.21.2.112:0/52231255 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff37c00c860 con 0x7ff3800d1870 2021-02-10T11:48:47.605 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:47.585 7ff387c31700 1 -- 172.21.2.112:0/52231255 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff3800d1870 msgr2=0x7ff3801b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:47.606 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:47.585 7ff387c31700 1 --2- 172.21.2.112:0/52231255 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff3800d1870 0x7ff3801b03f0 secure :-1 s=READY pgs=3209 cs=0 l=1 rev1=1 rx=0x7ff37c006700 tx=0x7ff37c009b80).stop 2021-02-10T11:48:47.606 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:47.586 7ff387c31700 1 -- 172.21.2.112:0/52231255 shutdown_connections 2021-02-10T11:48:47.606 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:47.586 7ff387c31700 1 --2- 172.21.2.112:0/52231255 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff3800f0650 0x7ff3800cfb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:47.606 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:47.586 7ff387c31700 1 --2- 172.21.2.112:0/52231255 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff3800d1870 0x7ff3801b03f0 unknown :-1 s=CLOSED pgs=3209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:47.606 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:47.586 7ff387c31700 1 -- 172.21.2.112:0/52231255 >> 172.21.2.112:0/52231255 conn(0x7ff3800ce450 msgr2=0x7ff3800d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:47.606 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:47.586 7ff387c31700 1 -- 172.21.2.112:0/52231255 shutdown_connections 2021-02-10T11:48:47.606 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:47.586 7ff387c31700 1 -- 172.21.2.112:0/52231255 wait complete. 2021-02-10T11:48:47.608 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:47.601 7ff387c31700 1 Processor -- start 2021-02-10T11:48:47.608 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:47.601 7ff387c31700 1 -- start start 2021-02-10T11:48:47.608 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:47.601 7ff387c31700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff380199ab0 0x7ff3801aab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:47.608 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:47.601 7ff387c31700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff3801ab1b0 0x7ff3801ab5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:47.608 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:47.601 7ff387c31700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff38019d250 0x7ff38019d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:47.608 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:47.601 7ff387c31700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3801ac040 con 0x7ff380199ab0 2021-02-10T11:48:47.608 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:47.601 7ff387c31700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff380199eb0 con 0x7ff38019d250 2021-02-10T11:48:47.608 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:47.601 7ff387c31700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3801abbd0 con 0x7ff3801ab1b0 2021-02-10T11:48:47.609 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:47.601 7ff385a4d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff380199ab0 0x7ff3801aab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:47.609 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:47.601 7ff385a4d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff380199ab0 0x7ff3801aab90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:33260/0 (socket says 172.21.2.112:33260) 2021-02-10T11:48:47.609 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:47.601 7ff385a4d700 1 -- 172.21.2.112:0/3236946384 learned_addr learned my addr 172.21.2.112:0/3236946384 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:48:47.609 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:47.602 7ff385a4d700 1 -- 172.21.2.112:0/3236946384 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff3801ab1b0 msgr2=0x7ff3801ab5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:47.609 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:47.602 7ff385a4d700 1 --2- 172.21.2.112:0/3236946384 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff3801ab1b0 0x7ff3801ab5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:47.609 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:47.602 7ff385a4d700 1 -- 172.21.2.112:0/3236946384 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff38019d250 msgr2=0x7ff38019d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:47.609 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:47.602 7ff385a4d700 1 --2- 172.21.2.112:0/3236946384 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff38019d250 0x7ff38019d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:47.609 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:47.602 7ff385a4d700 1 -- 172.21.2.112:0/3236946384 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff38019dc70 con 0x7ff380199ab0 2021-02-10T11:48:47.609 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:47.602 7ff385a4d700 1 --2- 172.21.2.112:0/3236946384 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff380199ab0 0x7ff3801aab90 secure :-1 s=READY pgs=3210 cs=0 l=1 rev1=1 rx=0x7ff37c00b5e0 tx=0x7ff37c00b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:47.609 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:47.602 7ff3727fc700 1 -- 172.21.2.112:0/3236946384 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff37c00bd70 con 0x7ff380199ab0 2021-02-10T11:48:47.609 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:47.602 7ff3727fc700 1 -- 172.21.2.112:0/3236946384 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff37c002030 con 0x7ff380199ab0 2021-02-10T11:48:47.609 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:47.602 7ff3727fc700 1 -- 172.21.2.112:0/3236946384 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff37c002550 con 0x7ff380199ab0 2021-02-10T11:48:47.610 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:47.602 7ff387c31700 1 -- 172.21.2.112:0/3236946384 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff38019dc70 con 0x7ff380199ab0 2021-02-10T11:48:47.610 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:47.602 7ff387c31700 1 -- 172.21.2.112:0/3236946384 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff38019dc70 con 0x7ff380199ab0 2021-02-10T11:48:47.610 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:47.603 7ff3727fc700 1 -- 172.21.2.112:0/3236946384 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff37c000c90 con 0x7ff380199ab0 2021-02-10T11:48:47.610 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:47.603 7ff3727fc700 1 --2- 172.21.2.112:0/3236946384 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff3680186c0 0x7ff36801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:47.610 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:47.603 7ff3727fc700 1 -- 172.21.2.112:0/3236946384 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 5573+0+0 (secure 0 0 0) 0x7ff37c032430 con 0x7ff380199ab0 2021-02-10T11:48:47.610 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:47.603 7ff38524c700 1 --2- 172.21.2.112:0/3236946384 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff3680186c0 0x7ff36801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:47.610 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:47.604 7ff38524c700 1 --2- 172.21.2.112:0/3236946384 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff3680186c0 0x7ff36801abc0 crc :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:47.637 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:47.634 7ff387c31700 1 -- 172.21.2.112:0/3236946384 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff38019cd20 con 0x7ff380199ab0 2021-02-10T11:48:47.640 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:47.636 7ff3727fc700 1 -- 172.21.2.112:0/3236946384 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff37c030d60 con 0x7ff380199ab0 2021-02-10T11:48:47.770 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:47.767 7ff387c31700 1 -- 172.21.2.112:0/3236946384 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_subvolume_authorized_list"]} v 0) v1 -- 0x7ff38019cd20 con 0x7ff380199ab0 2021-02-10T11:48:48.315 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:48.311 7ff3727fc700 1 -- 172.21.2.112:0/3236946384 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_subvolume_authorized_list"]}]=0 v892) v1 ==== 147+0+0 (secure 0 0 0) 0x7ff37c000c90 con 0x7ff380199ab0 2021-02-10T11:48:48.342 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:48.338 7ff387c31700 1 -- 172.21.2.112:0/3236946384 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff3680186c0 msgr2=0x7ff36801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:48.342 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:48.338 7ff387c31700 1 --2- 172.21.2.112:0/3236946384 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff3680186c0 0x7ff36801abc0 crc :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:48.342 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:48.338 7ff387c31700 1 -- 172.21.2.112:0/3236946384 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff380199ab0 msgr2=0x7ff3801aab90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:48.342 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:48.338 7ff387c31700 1 --2- 172.21.2.112:0/3236946384 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff380199ab0 0x7ff3801aab90 secure :-1 s=READY pgs=3210 cs=0 l=1 rev1=1 rx=0x7ff37c00b5e0 tx=0x7ff37c00b980).stop 2021-02-10T11:48:48.342 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:48.338 7ff387c31700 1 -- 172.21.2.112:0/3236946384 shutdown_connections 2021-02-10T11:48:48.342 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:48.338 7ff387c31700 1 --2- 172.21.2.112:0/3236946384 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff3680186c0 0x7ff36801abc0 unknown :-1 s=CLOSED pgs=1446 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:48.342 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:48.338 7ff387c31700 1 --2- 172.21.2.112:0/3236946384 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff38019d250 0x7ff38019d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:48.342 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:48.338 7ff387c31700 1 --2- 172.21.2.112:0/3236946384 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff3801ab1b0 0x7ff3801ab5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:48.342 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:48.338 7ff387c31700 1 --2- 172.21.2.112:0/3236946384 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff380199ab0 0x7ff3801aab90 unknown :-1 s=CLOSED pgs=3210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:48.342 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:48.338 7ff387c31700 1 -- 172.21.2.112:0/3236946384 >> 172.21.2.112:0/3236946384 conn(0x7ff38019f8c0 msgr2=0x7ff3801a7410 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:48.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:48.338 7ff387c31700 1 -- 172.21.2.112:0/3236946384 shutdown_connections 2021-02-10T11:48:48.343 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:48.338 7ff387c31700 1 -- 172.21.2.112:0/3236946384 wait complete. 2021-02-10T11:48:48.350 INFO:tasks.cephfs_test_runner:test_subvolume_authorized_list (tasks.cephfs.test_volumes.TestVolumes) ... ok 2021-02-10T11:48:48.350 INFO:tasks.cephfs_test_runner:Starting test: test_subvolume_clone_in_progress_getpath (tasks.cephfs.test_volumes.TestVolumes) 2021-02-10T11:48:48.350 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestVolumes.test_subvolume_clone_in_progress_getpath' 2021-02-10T11:48:48.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:48.759 7fb15ded8700 1 Processor -- start 2021-02-10T11:48:48.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:48.759 7fb15ded8700 1 -- start start 2021-02-10T11:48:48.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:48.760 7fb15ded8700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb1580d1870 0x7fb1581b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:48.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:48.760 7fb15ded8700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1580f0b10 con 0x7fb1581b0a10 2021-02-10T11:48:48.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:48.760 7fb15ded8700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1580d0310 con 0x7fb1580f0650 2021-02-10T11:48:48.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:48.760 7fb15ded8700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1580e5ed0 con 0x7fb1580d1870 2021-02-10T11:48:48.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:48.760 7fb156ffd700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fb1581b0a10 0x7fb1580cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:60960/0 (socket says 172.21.2.112:60960) 2021-02-10T11:48:48.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:48.760 7fb156ffd700 1 -- 172.21.2.112:0/2854473170 learned_addr learned my addr 172.21.2.112:0/2854473170 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:48:48.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:48.760 7fb1577fe700 1 --2- 172.21.2.112:0/2854473170 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb1580d1870 0x7fb1581b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:48.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:48.760 7fb1567fc700 1 -- 172.21.2.112:0/2854473170 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 596020809 0 0) 0x7fb1400010b0 con 0x7fb1581b0a10 2021-02-10T11:48:48.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:48.760 7fb1567fc700 1 -- 172.21.2.112:0/2854473170 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb144002fe0 con 0x7fb1581b0a10 2021-02-10T11:48:48.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:48.761 7fb1567fc700 1 -- 172.21.2.112:0/2854473170 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3527552336 0 0) 0x7fb1400010b0 con 0x7fb1581b0a10 2021-02-10T11:48:48.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:48.761 7fb1567fc700 1 -- 172.21.2.112:0/2854473170 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb1580d1870 msgr2=0x7fb1581b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:48.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:48.761 7fb1567fc700 1 --2- 172.21.2.112:0/2854473170 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb1580d1870 0x7fb1581b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:48.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:48.761 7fb1567fc700 1 -- 172.21.2.112:0/2854473170 >> v1:172.21.2.120:6789/0 conn(0x7fb1580f0650 legacy=0x7fb1580cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:48.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:48.761 7fb1567fc700 1 -- 172.21.2.112:0/2854473170 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1580d0530 con 0x7fb1581b0a10 2021-02-10T11:48:48.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:48.761 7fb1567fc700 1 -- 172.21.2.112:0/2854473170 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7fb1400015b0 con 0x7fb1581b0a10 2021-02-10T11:48:48.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:48.761 7fb1567fc700 1 -- 172.21.2.112:0/2854473170 >> v1:172.21.2.112:6789/0 conn(0x7fb1581b0a10 legacy=0x7fb1580cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:48.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:48.761 7fb1567fc700 1 --2- 172.21.2.112:0/2854473170 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb144003b10 0x7fb144005ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:48.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:48.761 7fb1567fc700 1 --2- 172.21.2.112:0/2854473170 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb1440067c0 0x7fb144008ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:48.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:48.761 7fb1567fc700 1 --2- 172.21.2.112:0/2854473170 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb1440095a0 0x7fb14400ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:48.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:48.761 7fb1567fc700 1 -- 172.21.2.112:0/2854473170 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb14400c2c0 con 0x7fb144003b10 2021-02-10T11:48:48.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:48.761 7fb1567fc700 1 -- 172.21.2.112:0/2854473170 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb14400c4e0 con 0x7fb1440067c0 2021-02-10T11:48:48.781 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:48.761 7fb1567fc700 1 -- 172.21.2.112:0/2854473170 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb144050710 con 0x7fb1440095a0 2021-02-10T11:48:48.782 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:48.761 7fb1577fe700 1 --2- 172.21.2.112:0/2854473170 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb144003b10 0x7fb144005ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:48.782 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:48.761 7fb156ffd700 1 --2- 172.21.2.112:0/2854473170 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb1440067c0 0x7fb144008ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:48.782 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:48.761 7fb1577fe700 1 -- 172.21.2.112:0/2854473170 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb1440095a0 msgr2=0x7fb14400ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:48.782 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:48.761 7fb157fff700 1 --2- 172.21.2.112:0/2854473170 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb1440095a0 0x7fb14400ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:48.782 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:48.761 7fb1577fe700 1 --2- 172.21.2.112:0/2854473170 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb1440095a0 0x7fb14400ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:48.782 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:48.761 7fb1577fe700 1 -- 172.21.2.112:0/2854473170 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb1440067c0 msgr2=0x7fb144008ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:48.782 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:48.761 7fb1577fe700 1 --2- 172.21.2.112:0/2854473170 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb1440067c0 0x7fb144008ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:48.782 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:48.761 7fb1577fe700 1 -- 172.21.2.112:0/2854473170 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb144050970 con 0x7fb144003b10 2021-02-10T11:48:48.782 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:48.761 7fb157fff700 1 --2- 172.21.2.112:0/2854473170 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb1440095a0 0x7fb14400ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:48:48.782 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:48.761 7fb1577fe700 1 --2- 172.21.2.112:0/2854473170 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb144003b10 0x7fb144005ff0 secure :-1 s=READY pgs=3212 cs=0 l=1 rev1=1 rx=0x7fb148006ac0 tx=0x7fb148009ab0).ready entity=mon.0 client_cookie=54f272079e702a66 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:48.782 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:48.761 7fb1567fc700 1 -- 172.21.2.112:0/2854473170 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb14800c0c0 con 0x7fb144003b10 2021-02-10T11:48:48.782 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:48.761 7fb1567fc700 1 -- 172.21.2.112:0/2854473170 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb14800c3c0 con 0x7fb144003b10 2021-02-10T11:48:48.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:48.761 7fb1567fc700 1 -- 172.21.2.112:0/2854473170 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb14800c8a0 con 0x7fb144003b10 2021-02-10T11:48:48.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:48.761 7fb15ded8700 1 -- 172.21.2.112:0/2854473170 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb144003b10 msgr2=0x7fb144005ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:48.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:48.761 7fb15ded8700 1 --2- 172.21.2.112:0/2854473170 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb144003b10 0x7fb144005ff0 secure :-1 s=READY pgs=3212 cs=0 l=1 rev1=1 rx=0x7fb148006ac0 tx=0x7fb148009ab0).stop 2021-02-10T11:48:48.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:48.761 7fb157fff700 1 -- 172.21.2.112:0/2854473170 reap_dead start 2021-02-10T11:48:48.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:48.762 7fb15ded8700 1 -- 172.21.2.112:0/2854473170 shutdown_connections 2021-02-10T11:48:48.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:48.762 7fb15ded8700 1 -- 172.21.2.112:0/2854473170 >> 172.21.2.112:0/2854473170 conn(0x7fb1580ce450 msgr2=0x7fb1580d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:48.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:48.762 7fb15ded8700 1 -- 172.21.2.112:0/2854473170 shutdown_connections 2021-02-10T11:48:48.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:48.762 7fb15ded8700 1 -- 172.21.2.112:0/2854473170 wait complete. 2021-02-10T11:48:48.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:48.777 7fb15ded8700 1 Processor -- start 2021-02-10T11:48:48.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:48.777 7fb15ded8700 1 -- start start 2021-02-10T11:48:48.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:48.777 7fb15ded8700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb1580a2560 0x7fb1580b4fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:48.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:48.777 7fb15ded8700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb1580b55e0 0x7fb1580b59e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:48.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:48.777 7fb15ded8700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb1580ee200 0x7fb1580ee600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:48.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:48.777 7fb15ded8700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb15818fda0 con 0x7fb1580a2560 2021-02-10T11:48:48.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:48.777 7fb15ded8700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1580a20c0 con 0x7fb1580b55e0 2021-02-10T11:48:48.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:48.777 7fb15ded8700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1580b6000 con 0x7fb1580ee200 2021-02-10T11:48:48.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:48.777 7fb1577fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb1580a2560 0x7fb1580b4fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:48.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:48.777 7fb1577fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb1580a2560 0x7fb1580b4fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:33278/0 (socket says 172.21.2.112:33278) 2021-02-10T11:48:48.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:48.777 7fb1577fe700 1 -- 172.21.2.112:0/3755078682 learned_addr learned my addr 172.21.2.112:0/3755078682 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:48:48.786 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:48.778 7fb157fff700 1 --2- 172.21.2.112:0/3755078682 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb1580ee200 0x7fb1580ee600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:48.786 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:48.778 7fb156ffd700 1 --2- 172.21.2.112:0/3755078682 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb1580b55e0 0x7fb1580b59e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:48.786 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:48.778 7fb1577fe700 1 -- 172.21.2.112:0/3755078682 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb1580ee200 msgr2=0x7fb1580ee600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:48.786 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:48.778 7fb1577fe700 1 --2- 172.21.2.112:0/3755078682 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb1580ee200 0x7fb1580ee600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:48.786 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:48.778 7fb1577fe700 1 -- 172.21.2.112:0/3755078682 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb1580b55e0 msgr2=0x7fb1580b59e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:48.786 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:48.778 7fb1577fe700 1 --2- 172.21.2.112:0/3755078682 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb1580b55e0 0x7fb1580b59e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:48.786 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:48.778 7fb1577fe700 1 -- 172.21.2.112:0/3755078682 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1580eec20 con 0x7fb1580a2560 2021-02-10T11:48:48.786 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:48.778 7fb1577fe700 1 --2- 172.21.2.112:0/3755078682 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb1580a2560 0x7fb1580b4fc0 secure :-1 s=READY pgs=3213 cs=0 l=1 rev1=1 rx=0x7fb1480097a0 tx=0x7fb14800b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:48.786 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:48.778 7fb14ffff700 1 -- 172.21.2.112:0/3755078682 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb14800bd60 con 0x7fb1580a2560 2021-02-10T11:48:48.786 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:48.778 7fb14ffff700 1 -- 172.21.2.112:0/3755078682 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb148000c90 con 0x7fb1580a2560 2021-02-10T11:48:48.786 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:48.778 7fb14ffff700 1 -- 172.21.2.112:0/3755078682 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb1480022b0 con 0x7fb1580a2560 2021-02-10T11:48:48.786 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:48.778 7fb15ded8700 1 -- 172.21.2.112:0/3755078682 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1580eec20 con 0x7fb1580a2560 2021-02-10T11:48:48.787 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:48.778 7fb15ded8700 1 -- 172.21.2.112:0/3755078682 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1580eec20 con 0x7fb1580a2560 2021-02-10T11:48:48.787 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:48.779 7fb14ffff700 1 -- 172.21.2.112:0/3755078682 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb148000940 con 0x7fb1580a2560 2021-02-10T11:48:48.787 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:48.779 7fb14ffff700 1 --2- 172.21.2.112:0/3755078682 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb13c018730 0x7fb13c01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:48.787 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:48.779 7fb14ffff700 1 -- 172.21.2.112:0/3755078682 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fb148032650 con 0x7fb1580a2560 2021-02-10T11:48:48.787 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:48.779 7fb156ffd700 1 --2- 172.21.2.112:0/3755078682 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb13c018730 0x7fb13c01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:48.787 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:48.780 7fb156ffd700 1 --2- 172.21.2.112:0/3755078682 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb13c018730 0x7fb13c01ac10 crc :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:48.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:48.810 7fb15ded8700 1 -- 172.21.2.112:0/3755078682 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1580f0b10 con 0x7fb1580a2560 2021-02-10T11:48:48.816 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:48.812 7fb14ffff700 1 -- 172.21.2.112:0/3755078682 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb148000940 con 0x7fb1580a2560 2021-02-10T11:48:48.949 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:48.945 7fb15ded8700 1 -- 172.21.2.112:0/3755078682 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_subvolume_clone_in_progress_getpath"]} v 0) v1 -- 0x7fb1580f0b10 con 0x7fb1580a2560 2021-02-10T11:48:49.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:49.314 7fb14ffff700 1 -- 172.21.2.112:0/3755078682 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_subvolume_clone_in_progress_getpath"]}]=0 v893) v1 ==== 160+0+0 (secure 0 0 0) 0x7fb148000c90 con 0x7fb1580a2560 2021-02-10T11:48:49.367 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:49.363 7fb15ded8700 1 -- 172.21.2.112:0/3755078682 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb13c018730 msgr2=0x7fb13c01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:49.367 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:49.363 7fb15ded8700 1 --2- 172.21.2.112:0/3755078682 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb13c018730 0x7fb13c01ac10 crc :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:49.367 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:49.363 7fb15ded8700 1 -- 172.21.2.112:0/3755078682 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb1580a2560 msgr2=0x7fb1580b4fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:49.367 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:49.363 7fb15ded8700 1 --2- 172.21.2.112:0/3755078682 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb1580a2560 0x7fb1580b4fc0 secure :-1 s=READY pgs=3213 cs=0 l=1 rev1=1 rx=0x7fb1480097a0 tx=0x7fb14800b970).stop 2021-02-10T11:48:49.367 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:49.364 7fb15ded8700 1 -- 172.21.2.112:0/3755078682 shutdown_connections 2021-02-10T11:48:49.367 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:49.364 7fb15ded8700 1 --2- 172.21.2.112:0/3755078682 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb13c018730 0x7fb13c01ac10 unknown :-1 s=CLOSED pgs=1447 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:49.367 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:49.364 7fb15ded8700 1 --2- 172.21.2.112:0/3755078682 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb1580b55e0 0x7fb1580b59e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:49.368 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:49.364 7fb15ded8700 1 --2- 172.21.2.112:0/3755078682 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb1580ee200 0x7fb1580ee600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:49.368 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:49.364 7fb15ded8700 1 --2- 172.21.2.112:0/3755078682 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb1580a2560 0x7fb1580b4fc0 unknown :-1 s=CLOSED pgs=3213 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:49.368 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:49.364 7fb15ded8700 1 -- 172.21.2.112:0/3755078682 >> 172.21.2.112:0/3755078682 conn(0x7fb1580ac990 msgr2=0x7fb158091b30 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:49.368 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:49.364 7fb15ded8700 1 -- 172.21.2.112:0/3755078682 shutdown_connections 2021-02-10T11:48:49.368 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:49.364 7fb15ded8700 1 -- 172.21.2.112:0/3755078682 wait complete. 2021-02-10T11:48:49.375 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:48:49.808 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:49.787 7f228aa2c700 1 Processor -- start 2021-02-10T11:48:49.808 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:49.787 7f228aa2c700 1 -- start start 2021-02-10T11:48:49.809 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:49.788 7f228aa2c700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f22840d1c70 0x7f22840cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:49.809 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:49.788 7f228aa2c700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22840d0170 con 0x7f22840d1870 2021-02-10T11:48:49.809 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:49.788 7f228aa2c700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22840fe1c0 con 0x7f22840cf580 2021-02-10T11:48:49.809 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:49.788 7f228aa2c700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22841b0cb0 con 0x7f22840d1c70 2021-02-10T11:48:49.809 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:49.788 7f2288848700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f22840d1870 0x7f22841b04c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:60978/0 (socket says 172.21.2.112:60978) 2021-02-10T11:48:49.809 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:49.788 7f2288848700 1 -- 172.21.2.112:0/4250642321 learned_addr learned my addr 172.21.2.112:0/4250642321 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:48:49.809 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:49.788 7f2283fff700 1 --2- 172.21.2.112:0/4250642321 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f22840d1c70 0x7f22840cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:49.809 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:49.788 7f22837fe700 1 -- 172.21.2.112:0/4250642321 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 170244168 0 0) 0x7f226c0009b0 con 0x7f22840d1870 2021-02-10T11:48:49.809 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:49.788 7f22837fe700 1 -- 172.21.2.112:0/4250642321 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2270002fe0 con 0x7f22840d1870 2021-02-10T11:48:49.809 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:49.788 7f22837fe700 1 -- 172.21.2.112:0/4250642321 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4270028751 0 0) 0x7f226c001010 con 0x7f22840d1870 2021-02-10T11:48:49.809 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:49.788 7f22837fe700 1 -- 172.21.2.112:0/4250642321 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f22840d1c70 msgr2=0x7f22840cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:49.809 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:49.788 7f22837fe700 1 --2- 172.21.2.112:0/4250642321 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f22840d1c70 0x7f22840cef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:49.809 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:49.788 7f22837fe700 1 -- 172.21.2.112:0/4250642321 >> v1:172.21.2.120:6789/0 conn(0x7f22840cf580 legacy=0x7f22840cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:49.810 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:49.789 7f22837fe700 1 -- 172.21.2.112:0/4250642321 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2284109a60 con 0x7f22840d1870 2021-02-10T11:48:49.810 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:49.789 7f22837fe700 1 -- 172.21.2.112:0/4250642321 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f226c001530 con 0x7f22840d1870 2021-02-10T11:48:49.810 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:49.789 7f22837fe700 1 -- 172.21.2.112:0/4250642321 >> v1:172.21.2.112:6789/0 conn(0x7f22840d1870 legacy=0x7f22841b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:49.810 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:49.789 7f22837fe700 1 --2- 172.21.2.112:0/4250642321 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2270003a40 0x7f2270005e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:49.810 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:49.789 7f22837fe700 1 --2- 172.21.2.112:0/4250642321 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2270006730 0x7f2270008c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:49.810 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:49.789 7f22837fe700 1 --2- 172.21.2.112:0/4250642321 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f22700094f0 0x7f227000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:49.810 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:49.789 7f22837fe700 1 -- 172.21.2.112:0/4250642321 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f227000c2c0 con 0x7f22700094f0 2021-02-10T11:48:49.810 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:49.789 7f22837fe700 1 -- 172.21.2.112:0/4250642321 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f227000c4e0 con 0x7f2270006730 2021-02-10T11:48:49.810 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:49.789 7f22837fe700 1 -- 172.21.2.112:0/4250642321 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f227000c700 con 0x7f2270003a40 2021-02-10T11:48:49.810 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:49.789 7f2283fff700 1 --2- 172.21.2.112:0/4250642321 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2270006730 0x7f2270008c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:49.810 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:49.789 7f2289049700 1 --2- 172.21.2.112:0/4250642321 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f22700094f0 0x7f227000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:49.810 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:49.789 7f2288848700 1 --2- 172.21.2.112:0/4250642321 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2270003a40 0x7f2270005e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:49.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:49.789 7f2283fff700 1 -- 172.21.2.112:0/4250642321 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2270003a40 msgr2=0x7f2270005e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:49.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:49.789 7f2283fff700 1 --2- 172.21.2.112:0/4250642321 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2270003a40 0x7f2270005e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:49.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:49.789 7f2283fff700 1 -- 172.21.2.112:0/4250642321 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f22700094f0 msgr2=0x7f227000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:49.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:49.789 7f2283fff700 1 --2- 172.21.2.112:0/4250642321 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f22700094f0 0x7f227000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:49.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:49.789 7f2283fff700 1 -- 172.21.2.112:0/4250642321 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2270050970 con 0x7f2270006730 2021-02-10T11:48:49.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:49.789 7f2283fff700 1 --2- 172.21.2.112:0/4250642321 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2270006730 0x7f2270008c50 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 rx=0x7f2274000ee0 tx=0x7f2274009de0).ready entity=mon.1 client_cookie=2d86b40c9713231c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:49.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:49.790 7f22837fe700 1 -- 172.21.2.112:0/4250642321 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f22740067f0 con 0x7f2270006730 2021-02-10T11:48:49.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:49.790 7f22837fe700 1 -- 172.21.2.112:0/4250642321 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f227400b300 con 0x7f2270006730 2021-02-10T11:48:49.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:49.790 7f22837fe700 1 -- 172.21.2.112:0/4250642321 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f227400b7a0 con 0x7f2270006730 2021-02-10T11:48:49.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:49.790 7f228aa2c700 1 -- 172.21.2.112:0/4250642321 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2270006730 msgr2=0x7f2270008c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:49.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:49.790 7f228aa2c700 1 --2- 172.21.2.112:0/4250642321 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2270006730 0x7f2270008c50 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 rx=0x7f2274000ee0 tx=0x7f2274009de0).stop 2021-02-10T11:48:49.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:49.790 7f2289049700 1 -- 172.21.2.112:0/4250642321 reap_dead start 2021-02-10T11:48:49.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:49.790 7f228aa2c700 1 -- 172.21.2.112:0/4250642321 shutdown_connections 2021-02-10T11:48:49.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:49.790 7f228aa2c700 1 -- 172.21.2.112:0/4250642321 >> 172.21.2.112:0/4250642321 conn(0x7f22840ce420 msgr2=0x7f22840d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:49.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:49.790 7f228aa2c700 1 -- 172.21.2.112:0/4250642321 shutdown_connections 2021-02-10T11:48:49.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:49.790 7f228aa2c700 1 -- 172.21.2.112:0/4250642321 wait complete. 2021-02-10T11:48:49.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:49.805 7f228aa2c700 1 Processor -- start 2021-02-10T11:48:49.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:49.805 7f228aa2c700 1 -- start start 2021-02-10T11:48:49.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:49.805 7f228aa2c700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f22841a79a0 0x7f22841a7da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:49.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:49.805 7f228aa2c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f22840bdb40 0x7f22840bdf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:49.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:49.805 7f228aa2c700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2284082ca0 0x7f22840830a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:49.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:49.805 7f228aa2c700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f228419d990 con 0x7f22840bdb40 2021-02-10T11:48:49.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:49.805 7f228aa2c700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f228419dbb0 con 0x7f2284082ca0 2021-02-10T11:48:49.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:49.805 7f228aa2c700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22840836c0 con 0x7f22841a79a0 2021-02-10T11:48:49.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:49.806 7f2283fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f22840bdb40 0x7f22840bdf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:49.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:49.806 7f2283fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f22840bdb40 0x7f22840bdf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:33298/0 (socket says 172.21.2.112:33298) 2021-02-10T11:48:49.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:49.806 7f2283fff700 1 -- 172.21.2.112:0/3017995571 learned_addr learned my addr 172.21.2.112:0/3017995571 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:48:49.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:49.806 7f2288848700 1 --2- 172.21.2.112:0/3017995571 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f22841a79a0 0x7f22841a7da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:49.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:49.806 7f2283fff700 1 -- 172.21.2.112:0/3017995571 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f22841a79a0 msgr2=0x7f22841a7da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:49.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:49.806 7f2283fff700 1 --2- 172.21.2.112:0/3017995571 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f22841a79a0 0x7f22841a7da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:49.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:49.806 7f2283fff700 1 -- 172.21.2.112:0/3017995571 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2284082ca0 msgr2=0x7f22840830a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:49.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:49.806 7f2283fff700 1 --2- 172.21.2.112:0/3017995571 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2284082ca0 0x7f22840830a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:49.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:49.806 7f2283fff700 1 -- 172.21.2.112:0/3017995571 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22840838e0 con 0x7f22840bdb40 2021-02-10T11:48:49.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:49.806 7f2283fff700 1 --2- 172.21.2.112:0/3017995571 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f22840bdb40 0x7f22840bdf40 secure :-1 s=READY pgs=3215 cs=0 l=1 rev1=1 rx=0x7f2274009fc0 tx=0x7f227400bcc0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:49.815 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:49.806 7f22817fa700 1 -- 172.21.2.112:0/3017995571 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2274000d10 con 0x7f22840bdb40 2021-02-10T11:48:49.815 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:49.806 7f22817fa700 1 -- 172.21.2.112:0/3017995571 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f227400c2f0 con 0x7f22840bdb40 2021-02-10T11:48:49.815 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:49.806 7f228aa2c700 1 -- 172.21.2.112:0/3017995571 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22840838e0 con 0x7f22840bdb40 2021-02-10T11:48:49.815 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:49.806 7f228aa2c700 1 -- 172.21.2.112:0/3017995571 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22840838e0 con 0x7f22840bdb40 2021-02-10T11:48:49.815 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:49.806 7f22817fa700 1 -- 172.21.2.112:0/3017995571 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f227400c740 con 0x7f22840bdb40 2021-02-10T11:48:49.815 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:49.807 7f22817fa700 1 -- 172.21.2.112:0/3017995571 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f227400c4d0 con 0x7f22840bdb40 2021-02-10T11:48:49.815 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:49.807 7f22817fa700 1 --2- 172.21.2.112:0/3017995571 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2268018730 0x7f226801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:49.815 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:49.807 7f22817fa700 1 -- 172.21.2.112:0/3017995571 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f2274032b60 con 0x7f22840bdb40 2021-02-10T11:48:49.815 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:49.807 7f2288848700 1 --2- 172.21.2.112:0/3017995571 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2268018730 0x7f226801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:49.815 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:49.808 7f2288848700 1 --2- 172.21.2.112:0/3017995571 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2268018730 0x7f226801ac10 crc :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:49.840 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:49.837 7f228aa2c700 1 -- 172.21.2.112:0/3017995571 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f225c002fd0 con 0x7f22840bdb40 2021-02-10T11:48:49.843 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:49.840 7f22817fa700 1 -- 172.21.2.112:0/3017995571 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f227400be00 con 0x7f22840bdb40 2021-02-10T11:48:49.991 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:49.987 7f228aa2c700 1 -- 172.21.2.112:0/3017995571 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f225c002ef0 con 0x7f22840bdb40 2021-02-10T11:48:49.995 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:49.991 7f22817fa700 1 -- 172.21.2.112:0/3017995571 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v48)=0 v48) v1 ==== 122+0+0 (secure 0 0 0) 0x7f227401f8a0 con 0x7f22840bdb40 2021-02-10T11:48:49.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:49.996 7f228aa2c700 1 -- 172.21.2.112:0/3017995571 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2268018730 msgr2=0x7f226801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:50.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:49.996 7f228aa2c700 1 --2- 172.21.2.112:0/3017995571 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2268018730 0x7f226801ac10 crc :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:50.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:49.996 7f228aa2c700 1 -- 172.21.2.112:0/3017995571 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f22840bdb40 msgr2=0x7f22840bdf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:50.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:49.996 7f228aa2c700 1 --2- 172.21.2.112:0/3017995571 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f22840bdb40 0x7f22840bdf40 secure :-1 s=READY pgs=3215 cs=0 l=1 rev1=1 rx=0x7f2274009fc0 tx=0x7f227400bcc0).stop 2021-02-10T11:48:50.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:49.996 7f228aa2c700 1 -- 172.21.2.112:0/3017995571 shutdown_connections 2021-02-10T11:48:50.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:49.996 7f228aa2c700 1 --2- 172.21.2.112:0/3017995571 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2268018730 0x7f226801ac10 unknown :-1 s=CLOSED pgs=1448 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:50.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:49.996 7f228aa2c700 1 --2- 172.21.2.112:0/3017995571 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2284082ca0 0x7f22840830a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:50.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:49.996 7f228aa2c700 1 --2- 172.21.2.112:0/3017995571 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f22840bdb40 0x7f22840bdf40 unknown :-1 s=CLOSED pgs=3215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:50.001 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:49.996 7f228aa2c700 1 --2- 172.21.2.112:0/3017995571 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f22841a79a0 0x7f22841a7da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:50.001 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:49.996 7f228aa2c700 1 -- 172.21.2.112:0/3017995571 >> 172.21.2.112:0/3017995571 conn(0x7f22840bef00 msgr2=0x7f228409f200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:50.001 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:49.997 7f228aa2c700 1 -- 172.21.2.112:0/3017995571 shutdown_connections 2021-02-10T11:48:50.001 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:49.997 7f228aa2c700 1 -- 172.21.2.112:0/3017995571 wait complete. 2021-02-10T11:48:50.008 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:48:50.009 DEBUG:teuthology.orchestra.run.gibba012:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:48:50.355 DEBUG:teuthology.orchestra.run.gibba020:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:48:50.699 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:48:50.700 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:48:50.700 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:48:50.700 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:48:51.036 INFO:teuthology.orchestra.run.gibba012.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.0’: No such file or directory 2021-02-10T11:48:51.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:48:51.036 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2021-02-10T11:48:51.036 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:48:51.037 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:48:51.037 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:48:51.373 INFO:teuthology.orchestra.run.gibba012.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: No such file or directory 2021-02-10T11:48:51.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:48:51.373 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2021-02-10T11:48:51.373 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:48:51.374 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:48:51.374 DEBUG:teuthology.orchestra.run.gibba020:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:48:51.707 INFO:teuthology.orchestra.run.gibba020.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:48:51.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:48:51.707 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:48:51.708 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:48:51.708 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:48:51.708 DEBUG:teuthology.orchestra.run.gibba020:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:48:52.042 INFO:teuthology.orchestra.run.gibba020.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:48:52.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:48:52.043 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:48:52.043 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json-pretty 2021-02-10T11:48:52.473 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:52.453 7f817158b700 1 Processor -- start 2021-02-10T11:48:52.473 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:52.454 7f817158b700 1 -- start start 2021-02-10T11:48:52.473 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:52.454 7f817158b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f816c1b0470 0x7f816c1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:52.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:52.454 7f817158b700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f816c0ef610 0x7f816c0efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:52.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:52.454 7f817158b700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f816c0f0030 0x7f816c0f0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:52.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:52.454 7f817158b700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f816c0e5ea0 con 0x7f816c0ef610 2021-02-10T11:48:52.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:52.454 7f817158b700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f816c0f0b50 con 0x7f816c0f0030 2021-02-10T11:48:52.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:52.454 7f817158b700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f816c0e52e0 con 0x7f816c1b0470 2021-02-10T11:48:52.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:52.454 7f816aabe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f816c0ef610 0x7f816c0efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:52.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:52.454 7f816aabe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f816c0ef610 0x7f816c0efa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:33304/0 (socket says 172.21.2.112:33304) 2021-02-10T11:48:52.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:52.454 7f816aabe700 1 -- 172.21.2.112:0/4231612927 learned_addr learned my addr 172.21.2.112:0/4231612927 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:48:52.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:52.454 7f816bac0700 1 --2- 172.21.2.112:0/4231612927 >> v2:172.21.2.120:3300/0 conn(0x7f816c0f0030 0x7f816c0f0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:52.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:52.454 7f816b2bf700 1 --2- 172.21.2.112:0/4231612927 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f816c1b0470 0x7f816c1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:52.474 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:52.455 7f816bac0700 1 -- 172.21.2.112:0/4231612927 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f816c1b0470 msgr2=0x7f816c1b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:52.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:52.455 7f816bac0700 1 --2- 172.21.2.112:0/4231612927 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f816c1b0470 0x7f816c1b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:52.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:52.455 7f816bac0700 1 -- 172.21.2.112:0/4231612927 >> v2:172.21.2.112:3300/0 conn(0x7f816c0ef610 msgr2=0x7f816c0efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:52.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:52.455 7f816bac0700 1 --2- 172.21.2.112:0/4231612927 >> v2:172.21.2.112:3300/0 conn(0x7f816c0ef610 0x7f816c0efa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:52.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:52.455 7f816bac0700 1 -- 172.21.2.112:0/4231612927 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f816c0f10d0 con 0x7f816c0f0030 2021-02-10T11:48:52.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:52.455 7f816bac0700 1 --2- 172.21.2.112:0/4231612927 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f816c0f0030 0x7f816c0f0430 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 rx=0x7f8164006e50 tx=0x7f8164009e70).ready entity=mon.1 client_cookie=5f4640e47d85dce server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:52.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:52.455 7f816a2bd700 1 -- 172.21.2.112:0/4231612927 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8164006a20 con 0x7f816c0f0030 2021-02-10T11:48:52.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:52.455 7f816a2bd700 1 -- 172.21.2.112:0/4231612927 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8164006790 con 0x7f816c0f0030 2021-02-10T11:48:52.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:52.455 7f816a2bd700 1 -- 172.21.2.112:0/4231612927 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f816400b700 con 0x7f816c0f0030 2021-02-10T11:48:52.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:52.455 7f817158b700 1 -- 172.21.2.112:0/4231612927 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f816c0f0030 msgr2=0x7f816c0f0430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:52.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:52.455 7f817158b700 1 --2- 172.21.2.112:0/4231612927 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f816c0f0030 0x7f816c0f0430 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 rx=0x7f8164006e50 tx=0x7f8164009e70).stop 2021-02-10T11:48:52.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:52.456 7f817158b700 1 -- 172.21.2.112:0/4231612927 shutdown_connections 2021-02-10T11:48:52.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:52.456 7f817158b700 1 --2- 172.21.2.112:0/4231612927 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f816c0f0030 0x7f816c0f0430 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:52.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:52.456 7f817158b700 1 --2- 172.21.2.112:0/4231612927 >> v2:172.21.2.112:3300/0 conn(0x7f816c0ef610 0x7f816c0efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:52.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:52.456 7f817158b700 1 --2- 172.21.2.112:0/4231612927 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f816c1b0470 0x7f816c1b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:52.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:52.456 7f817158b700 1 -- 172.21.2.112:0/4231612927 >> 172.21.2.112:0/4231612927 conn(0x7f816c0ce3e0 msgr2=0x7f816c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:52.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:52.456 7f817158b700 1 -- 172.21.2.112:0/4231612927 shutdown_connections 2021-02-10T11:48:52.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:52.456 7f817158b700 1 -- 172.21.2.112:0/4231612927 wait complete. 2021-02-10T11:48:52.477 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:52.470 7f817158b700 1 Processor -- start 2021-02-10T11:48:52.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:52.470 7f817158b700 1 -- start start 2021-02-10T11:48:52.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:52.471 7f817158b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f816c108ce0 0x7f816c09da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:52.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:52.471 7f817158b700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f816c1090e0 0x7f816c09e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:52.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:52.471 7f817158b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f816c09eee0 0x7f816c09f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:52.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:52.471 7f817158b700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f816c1094e0 con 0x7f816c108ce0 2021-02-10T11:48:52.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:52.471 7f817158b700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f816c09e960 con 0x7f816c1090e0 2021-02-10T11:48:52.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:52.471 7f817158b700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f816c09e690 con 0x7f816c09eee0 2021-02-10T11:48:52.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:52.471 7f816b2bf700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f816c108ce0 0x7f816c09da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:52.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:52.471 7f816b2bf700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f816c108ce0 0x7f816c09da50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:33312/0 (socket says 172.21.2.112:33312) 2021-02-10T11:48:52.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:52.471 7f816b2bf700 1 -- 172.21.2.112:0/3831850318 learned_addr learned my addr 172.21.2.112:0/3831850318 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:48:52.478 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:52.471 7f816bac0700 1 --2- 172.21.2.112:0/3831850318 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f816c09eee0 0x7f816c09f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:52.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:52.471 7f816aabe700 1 --2- 172.21.2.112:0/3831850318 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f816c1090e0 0x7f816c09e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:52.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:52.471 7f816b2bf700 1 -- 172.21.2.112:0/3831850318 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f816c09eee0 msgr2=0x7f816c09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:52.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:52.471 7f816b2bf700 1 --2- 172.21.2.112:0/3831850318 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f816c09eee0 0x7f816c09f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:52.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:52.471 7f816b2bf700 1 -- 172.21.2.112:0/3831850318 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f816c1090e0 msgr2=0x7f816c09e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:52.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:52.471 7f816b2bf700 1 --2- 172.21.2.112:0/3831850318 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f816c1090e0 0x7f816c09e070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:52.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:52.471 7f816b2bf700 1 -- 172.21.2.112:0/3831850318 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f816c09f900 con 0x7f816c108ce0 2021-02-10T11:48:52.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:52.471 7f816b2bf700 1 --2- 172.21.2.112:0/3831850318 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f816c108ce0 0x7f816c09da50 secure :-1 s=READY pgs=3216 cs=0 l=1 rev1=1 rx=0x7f815c006f00 tx=0x7f815c00b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:52.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:52.472 7f8163fff700 1 -- 172.21.2.112:0/3831850318 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f815c00bd30 con 0x7f816c108ce0 2021-02-10T11:48:52.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:52.472 7f8163fff700 1 -- 172.21.2.112:0/3831850318 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f815c002030 con 0x7f816c108ce0 2021-02-10T11:48:52.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:52.472 7f8163fff700 1 -- 172.21.2.112:0/3831850318 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f815c002590 con 0x7f816c108ce0 2021-02-10T11:48:52.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:52.472 7f817158b700 1 -- 172.21.2.112:0/3831850318 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f816c09f900 con 0x7f816c108ce0 2021-02-10T11:48:52.479 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:52.472 7f817158b700 1 -- 172.21.2.112:0/3831850318 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f816c09f900 con 0x7f816c108ce0 2021-02-10T11:48:52.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:52.472 7f8163fff700 1 -- 172.21.2.112:0/3831850318 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f815c002210 con 0x7f816c108ce0 2021-02-10T11:48:52.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:52.472 7f8163fff700 1 --2- 172.21.2.112:0/3831850318 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f81500186c0 0x7f815001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:52.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:52.472 7f8163fff700 1 -- 172.21.2.112:0/3831850318 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f815c032550 con 0x7f816c108ce0 2021-02-10T11:48:52.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:52.473 7f816aabe700 1 --2- 172.21.2.112:0/3831850318 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f81500186c0 0x7f815001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:52.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:52.473 7f816aabe700 1 --2- 172.21.2.112:0/3831850318 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f81500186c0 0x7f815001abc0 crc :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:52.507 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:52.504 7f817158b700 1 -- 172.21.2.112:0/3831850318 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f816c0f0b50 con 0x7f816c108ce0 2021-02-10T11:48:52.510 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:52.506 7f8163fff700 1 -- 172.21.2.112:0/3831850318 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f815c002560 con 0x7f816c108ce0 2021-02-10T11:48:52.636 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:52.633 7f817158b700 1 -- 172.21.2.112:0/3831850318 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f816c0f0b50 con 0x7f816c108ce0 2021-02-10T11:48:52.637 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:52.634 7f8163fff700 1 -- 172.21.2.112:0/3831850318 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v189) v1 ==== 81+0+30374 (secure 0 0 0) 0x7f815c030d00 con 0x7f816c108ce0 2021-02-10T11:48:52.638 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:48:52.638 INFO:teuthology.orchestra.run.gibba012.stdout:{ 2021-02-10T11:48:52.638 INFO:teuthology.orchestra.run.gibba012.stdout: "epoch": 189, 2021-02-10T11:48:52.638 INFO:teuthology.orchestra.run.gibba012.stdout: "fsid": "f14f004e-0699-4584-ad81-3754258493de", 2021-02-10T11:48:52.638 INFO:teuthology.orchestra.run.gibba012.stdout: "created": "2021-02-10 11:13:26.501424", 2021-02-10T11:48:52.638 INFO:teuthology.orchestra.run.gibba012.stdout: "modified": "2021-02-10 11:48:11.483156", 2021-02-10T11:48:52.639 INFO:teuthology.orchestra.run.gibba012.stdout: "last_up_change": "2021-02-10 11:13:52.780316", 2021-02-10T11:48:52.639 INFO:teuthology.orchestra.run.gibba012.stdout: "last_in_change": "2021-02-10 11:13:52.780316", 2021-02-10T11:48:52.639 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-02-10T11:48:52.639 INFO:teuthology.orchestra.run.gibba012.stdout: "flags_num": 5799936, 2021-02-10T11:48:52.639 INFO:teuthology.orchestra.run.gibba012.stdout: "flags_set": [ 2021-02-10T11:48:52.639 INFO:teuthology.orchestra.run.gibba012.stdout: "pglog_hardlimit", 2021-02-10T11:48:52.639 INFO:teuthology.orchestra.run.gibba012.stdout: "purged_snapdirs", 2021-02-10T11:48:52.639 INFO:teuthology.orchestra.run.gibba012.stdout: "recovery_deletes", 2021-02-10T11:48:52.639 INFO:teuthology.orchestra.run.gibba012.stdout: "sortbitwise" 2021-02-10T11:48:52.639 INFO:teuthology.orchestra.run.gibba012.stdout: ], 2021-02-10T11:48:52.639 INFO:teuthology.orchestra.run.gibba012.stdout: "crush_version": 4, 2021-02-10T11:48:52.639 INFO:teuthology.orchestra.run.gibba012.stdout: "full_ratio": 0.94999998807907104, 2021-02-10T11:48:52.639 INFO:teuthology.orchestra.run.gibba012.stdout: "backfillfull_ratio": 0.89999997615814209, 2021-02-10T11:48:52.640 INFO:teuthology.orchestra.run.gibba012.stdout: "nearfull_ratio": 0.85000002384185791, 2021-02-10T11:48:52.640 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_snapshot": "", 2021-02-10T11:48:52.640 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_max": 41, 2021-02-10T11:48:52.640 INFO:teuthology.orchestra.run.gibba012.stdout: "max_osd": 8, 2021-02-10T11:48:52.640 INFO:teuthology.orchestra.run.gibba012.stdout: "require_min_compat_client": "jewel", 2021-02-10T11:48:52.640 INFO:teuthology.orchestra.run.gibba012.stdout: "min_compat_client": "jewel", 2021-02-10T11:48:52.640 INFO:teuthology.orchestra.run.gibba012.stdout: "require_osd_release": "nautilus", 2021-02-10T11:48:52.640 INFO:teuthology.orchestra.run.gibba012.stdout: "pools": [ 2021-02-10T11:48:52.640 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:48:52.640 INFO:teuthology.orchestra.run.gibba012.stdout: "pool": 1, 2021-02-10T11:48:52.640 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_name": "rbd", 2021-02-10T11:48:52.640 INFO:teuthology.orchestra.run.gibba012.stdout: "create_time": "2021-02-10 11:13:57.508758", 2021-02-10T11:48:52.640 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:48:52.641 INFO:teuthology.orchestra.run.gibba012.stdout: "flags_names": "hashpspool", 2021-02-10T11:48:52.641 INFO:teuthology.orchestra.run.gibba012.stdout: "type": 1, 2021-02-10T11:48:52.641 INFO:teuthology.orchestra.run.gibba012.stdout: "size": 2, 2021-02-10T11:48:52.641 INFO:teuthology.orchestra.run.gibba012.stdout: "min_size": 1, 2021-02-10T11:48:52.641 INFO:teuthology.orchestra.run.gibba012.stdout: "crush_rule": 0, 2021-02-10T11:48:52.641 INFO:teuthology.orchestra.run.gibba012.stdout: "object_hash": 2, 2021-02-10T11:48:52.641 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:48:52.641 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num": 8, 2021-02-10T11:48:52.641 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num": 8, 2021-02-10T11:48:52.641 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num_target": 8, 2021-02-10T11:48:52.641 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_target": 8, 2021-02-10T11:48:52.641 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_pending": 8, 2021-02-10T11:48:52.641 INFO:teuthology.orchestra.run.gibba012.stdout: "last_pg_merge_meta": { 2021-02-10T11:48:52.642 INFO:teuthology.orchestra.run.gibba012.stdout: "source_pgid": "0.0", 2021-02-10T11:48:52.642 INFO:teuthology.orchestra.run.gibba012.stdout: "ready_epoch": 0, 2021-02-10T11:48:52.642 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_started": 0, 2021-02-10T11:48:52.642 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_clean": 0, 2021-02-10T11:48:52.642 INFO:teuthology.orchestra.run.gibba012.stdout: "source_version": "0'0", 2021-02-10T11:48:52.642 INFO:teuthology.orchestra.run.gibba012.stdout: "target_version": "0'0" 2021-02-10T11:48:52.642 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:48:52.642 INFO:teuthology.orchestra.run.gibba012.stdout: "last_change": "16", 2021-02-10T11:48:52.642 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend": "0", 2021-02-10T11:48:52.642 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:48:52.642 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:48:52.642 INFO:teuthology.orchestra.run.gibba012.stdout: "auid": 0, 2021-02-10T11:48:52.642 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:48:52.643 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_seq": 0, 2021-02-10T11:48:52.643 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_epoch": 0, 2021-02-10T11:48:52.643 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_snaps": [], 2021-02-10T11:48:52.643 INFO:teuthology.orchestra.run.gibba012.stdout: "removed_snaps": "[]", 2021-02-10T11:48:52.643 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_bytes": 0, 2021-02-10T11:48:52.643 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_objects": 0, 2021-02-10T11:48:52.643 INFO:teuthology.orchestra.run.gibba012.stdout: "tiers": [], 2021-02-10T11:48:52.643 INFO:teuthology.orchestra.run.gibba012.stdout: "tier_of": -1, 2021-02-10T11:48:52.643 INFO:teuthology.orchestra.run.gibba012.stdout: "read_tier": -1, 2021-02-10T11:48:52.643 INFO:teuthology.orchestra.run.gibba012.stdout: "write_tier": -1, 2021-02-10T11:48:52.643 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_mode": "none", 2021-02-10T11:48:52.643 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_bytes": 0, 2021-02-10T11:48:52.643 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_objects": 0, 2021-02-10T11:48:52.644 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:48:52.644 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:48:52.644 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:48:52.644 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_flush_age": 0, 2021-02-10T11:48:52.644 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_evict_age": 0, 2021-02-10T11:48:52.644 INFO:teuthology.orchestra.run.gibba012.stdout: "erasure_code_profile": "", 2021-02-10T11:48:52.644 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_params": { 2021-02-10T11:48:52.644 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "none" 2021-02-10T11:48:52.644 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:48:52.644 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_period": 0, 2021-02-10T11:48:52.644 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_count": 0, 2021-02-10T11:48:52.644 INFO:teuthology.orchestra.run.gibba012.stdout: "use_gmt_hitset": true, 2021-02-10T11:48:52.644 INFO:teuthology.orchestra.run.gibba012.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:48:52.645 INFO:teuthology.orchestra.run.gibba012.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:48:52.645 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:48:52.645 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:48:52.645 INFO:teuthology.orchestra.run.gibba012.stdout: "grade_table": [], 2021-02-10T11:48:52.645 INFO:teuthology.orchestra.run.gibba012.stdout: "stripe_width": 0, 2021-02-10T11:48:52.645 INFO:teuthology.orchestra.run.gibba012.stdout: "expected_num_objects": 0, 2021-02-10T11:48:52.645 INFO:teuthology.orchestra.run.gibba012.stdout: "fast_read": false, 2021-02-10T11:48:52.645 INFO:teuthology.orchestra.run.gibba012.stdout: "options": {}, 2021-02-10T11:48:52.645 INFO:teuthology.orchestra.run.gibba012.stdout: "application_metadata": { 2021-02-10T11:48:52.645 INFO:teuthology.orchestra.run.gibba012.stdout: "rbd": {} 2021-02-10T11:48:52.645 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:48:52.645 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:48:52.645 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:48:52.646 INFO:teuthology.orchestra.run.gibba012.stdout: "pool": 40, 2021-02-10T11:48:52.646 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_name": "cephfs_metadata", 2021-02-10T11:48:52.646 INFO:teuthology.orchestra.run.gibba012.stdout: "create_time": "2021-02-10 11:48:08.454038", 2021-02-10T11:48:52.646 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:48:52.646 INFO:teuthology.orchestra.run.gibba012.stdout: "flags_names": "hashpspool", 2021-02-10T11:48:52.646 INFO:teuthology.orchestra.run.gibba012.stdout: "type": 1, 2021-02-10T11:48:52.646 INFO:teuthology.orchestra.run.gibba012.stdout: "size": 2, 2021-02-10T11:48:52.646 INFO:teuthology.orchestra.run.gibba012.stdout: "min_size": 1, 2021-02-10T11:48:52.646 INFO:teuthology.orchestra.run.gibba012.stdout: "crush_rule": 0, 2021-02-10T11:48:52.646 INFO:teuthology.orchestra.run.gibba012.stdout: "object_hash": 2, 2021-02-10T11:48:52.646 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:48:52.646 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num": 8, 2021-02-10T11:48:52.646 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num": 8, 2021-02-10T11:48:52.647 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num_target": 8, 2021-02-10T11:48:52.647 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_target": 8, 2021-02-10T11:48:52.647 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_pending": 8, 2021-02-10T11:48:52.647 INFO:teuthology.orchestra.run.gibba012.stdout: "last_pg_merge_meta": { 2021-02-10T11:48:52.647 INFO:teuthology.orchestra.run.gibba012.stdout: "source_pgid": "0.0", 2021-02-10T11:48:52.647 INFO:teuthology.orchestra.run.gibba012.stdout: "ready_epoch": 0, 2021-02-10T11:48:52.647 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_started": 0, 2021-02-10T11:48:52.647 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_clean": 0, 2021-02-10T11:48:52.647 INFO:teuthology.orchestra.run.gibba012.stdout: "source_version": "0'0", 2021-02-10T11:48:52.647 INFO:teuthology.orchestra.run.gibba012.stdout: "target_version": "0'0" 2021-02-10T11:48:52.647 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:48:52.647 INFO:teuthology.orchestra.run.gibba012.stdout: "last_change": "188", 2021-02-10T11:48:52.647 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend": "0", 2021-02-10T11:48:52.648 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:48:52.648 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:48:52.648 INFO:teuthology.orchestra.run.gibba012.stdout: "auid": 0, 2021-02-10T11:48:52.648 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:48:52.648 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_seq": 0, 2021-02-10T11:48:52.648 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_epoch": 0, 2021-02-10T11:48:52.648 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_snaps": [], 2021-02-10T11:48:52.648 INFO:teuthology.orchestra.run.gibba012.stdout: "removed_snaps": "[]", 2021-02-10T11:48:52.648 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_bytes": 0, 2021-02-10T11:48:52.648 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_objects": 0, 2021-02-10T11:48:52.648 INFO:teuthology.orchestra.run.gibba012.stdout: "tiers": [], 2021-02-10T11:48:52.648 INFO:teuthology.orchestra.run.gibba012.stdout: "tier_of": -1, 2021-02-10T11:48:52.649 INFO:teuthology.orchestra.run.gibba012.stdout: "read_tier": -1, 2021-02-10T11:48:52.649 INFO:teuthology.orchestra.run.gibba012.stdout: "write_tier": -1, 2021-02-10T11:48:52.649 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_mode": "none", 2021-02-10T11:48:52.649 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_bytes": 0, 2021-02-10T11:48:52.649 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_objects": 0, 2021-02-10T11:48:52.649 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:48:52.649 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:48:52.649 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:48:52.649 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_flush_age": 0, 2021-02-10T11:48:52.649 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_evict_age": 0, 2021-02-10T11:48:52.649 INFO:teuthology.orchestra.run.gibba012.stdout: "erasure_code_profile": "", 2021-02-10T11:48:52.649 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_params": { 2021-02-10T11:48:52.649 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "none" 2021-02-10T11:48:52.650 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:48:52.650 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_period": 0, 2021-02-10T11:48:52.650 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_count": 0, 2021-02-10T11:48:52.650 INFO:teuthology.orchestra.run.gibba012.stdout: "use_gmt_hitset": true, 2021-02-10T11:48:52.650 INFO:teuthology.orchestra.run.gibba012.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:48:52.650 INFO:teuthology.orchestra.run.gibba012.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:48:52.650 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:48:52.650 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:48:52.650 INFO:teuthology.orchestra.run.gibba012.stdout: "grade_table": [], 2021-02-10T11:48:52.650 INFO:teuthology.orchestra.run.gibba012.stdout: "stripe_width": 0, 2021-02-10T11:48:52.650 INFO:teuthology.orchestra.run.gibba012.stdout: "expected_num_objects": 0, 2021-02-10T11:48:52.651 INFO:teuthology.orchestra.run.gibba012.stdout: "fast_read": false, 2021-02-10T11:48:52.651 INFO:teuthology.orchestra.run.gibba012.stdout: "options": { 2021-02-10T11:48:52.651 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_autoscale_bias": 4, 2021-02-10T11:48:52.651 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_min": 16, 2021-02-10T11:48:52.651 INFO:teuthology.orchestra.run.gibba012.stdout: "recovery_priority": 5 2021-02-10T11:48:52.651 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:48:52.651 INFO:teuthology.orchestra.run.gibba012.stdout: "application_metadata": { 2021-02-10T11:48:52.651 INFO:teuthology.orchestra.run.gibba012.stdout: "cephfs": { 2021-02-10T11:48:52.651 INFO:teuthology.orchestra.run.gibba012.stdout: "metadata": "cephfs" 2021-02-10T11:48:52.651 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:48:52.651 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:48:52.651 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:48:52.651 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:48:52.652 INFO:teuthology.orchestra.run.gibba012.stdout: "pool": 41, 2021-02-10T11:48:52.652 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_name": "cephfs_data", 2021-02-10T11:48:52.652 INFO:teuthology.orchestra.run.gibba012.stdout: "create_time": "2021-02-10 11:48:09.474627", 2021-02-10T11:48:52.652 INFO:teuthology.orchestra.run.gibba012.stdout: "flags": 1, 2021-02-10T11:48:52.652 INFO:teuthology.orchestra.run.gibba012.stdout: "flags_names": "hashpspool", 2021-02-10T11:48:52.652 INFO:teuthology.orchestra.run.gibba012.stdout: "type": 1, 2021-02-10T11:48:52.652 INFO:teuthology.orchestra.run.gibba012.stdout: "size": 2, 2021-02-10T11:48:52.652 INFO:teuthology.orchestra.run.gibba012.stdout: "min_size": 1, 2021-02-10T11:48:52.652 INFO:teuthology.orchestra.run.gibba012.stdout: "crush_rule": 0, 2021-02-10T11:48:52.652 INFO:teuthology.orchestra.run.gibba012.stdout: "object_hash": 2, 2021-02-10T11:48:52.652 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:48:52.653 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num": 8, 2021-02-10T11:48:52.653 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num": 8, 2021-02-10T11:48:52.653 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_placement_num_target": 8, 2021-02-10T11:48:52.653 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_target": 8, 2021-02-10T11:48:52.653 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_num_pending": 8, 2021-02-10T11:48:52.653 INFO:teuthology.orchestra.run.gibba012.stdout: "last_pg_merge_meta": { 2021-02-10T11:48:52.653 INFO:teuthology.orchestra.run.gibba012.stdout: "source_pgid": "0.0", 2021-02-10T11:48:52.653 INFO:teuthology.orchestra.run.gibba012.stdout: "ready_epoch": 0, 2021-02-10T11:48:52.653 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_started": 0, 2021-02-10T11:48:52.653 INFO:teuthology.orchestra.run.gibba012.stdout: "last_epoch_clean": 0, 2021-02-10T11:48:52.653 INFO:teuthology.orchestra.run.gibba012.stdout: "source_version": "0'0", 2021-02-10T11:48:52.653 INFO:teuthology.orchestra.run.gibba012.stdout: "target_version": "0'0" 2021-02-10T11:48:52.654 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:48:52.654 INFO:teuthology.orchestra.run.gibba012.stdout: "last_change": "189", 2021-02-10T11:48:52.654 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend": "0", 2021-02-10T11:48:52.654 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:48:52.654 INFO:teuthology.orchestra.run.gibba012.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:48:52.654 INFO:teuthology.orchestra.run.gibba012.stdout: "auid": 0, 2021-02-10T11:48:52.654 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:48:52.654 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_seq": 0, 2021-02-10T11:48:52.654 INFO:teuthology.orchestra.run.gibba012.stdout: "snap_epoch": 0, 2021-02-10T11:48:52.654 INFO:teuthology.orchestra.run.gibba012.stdout: "pool_snaps": [], 2021-02-10T11:48:52.654 INFO:teuthology.orchestra.run.gibba012.stdout: "removed_snaps": "[]", 2021-02-10T11:48:52.654 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_bytes": 0, 2021-02-10T11:48:52.654 INFO:teuthology.orchestra.run.gibba012.stdout: "quota_max_objects": 0, 2021-02-10T11:48:52.655 INFO:teuthology.orchestra.run.gibba012.stdout: "tiers": [], 2021-02-10T11:48:52.655 INFO:teuthology.orchestra.run.gibba012.stdout: "tier_of": -1, 2021-02-10T11:48:52.655 INFO:teuthology.orchestra.run.gibba012.stdout: "read_tier": -1, 2021-02-10T11:48:52.655 INFO:teuthology.orchestra.run.gibba012.stdout: "write_tier": -1, 2021-02-10T11:48:52.655 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_mode": "none", 2021-02-10T11:48:52.655 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_bytes": 0, 2021-02-10T11:48:52.655 INFO:teuthology.orchestra.run.gibba012.stdout: "target_max_objects": 0, 2021-02-10T11:48:52.655 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:48:52.655 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:48:52.655 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:48:52.655 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_flush_age": 0, 2021-02-10T11:48:52.655 INFO:teuthology.orchestra.run.gibba012.stdout: "cache_min_evict_age": 0, 2021-02-10T11:48:52.656 INFO:teuthology.orchestra.run.gibba012.stdout: "erasure_code_profile": "", 2021-02-10T11:48:52.656 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_params": { 2021-02-10T11:48:52.656 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "none" 2021-02-10T11:48:52.656 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:48:52.656 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_period": 0, 2021-02-10T11:48:52.656 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_count": 0, 2021-02-10T11:48:52.656 INFO:teuthology.orchestra.run.gibba012.stdout: "use_gmt_hitset": true, 2021-02-10T11:48:52.656 INFO:teuthology.orchestra.run.gibba012.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:48:52.656 INFO:teuthology.orchestra.run.gibba012.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:48:52.656 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:48:52.656 INFO:teuthology.orchestra.run.gibba012.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:48:52.656 INFO:teuthology.orchestra.run.gibba012.stdout: "grade_table": [], 2021-02-10T11:48:52.656 INFO:teuthology.orchestra.run.gibba012.stdout: "stripe_width": 0, 2021-02-10T11:48:52.657 INFO:teuthology.orchestra.run.gibba012.stdout: "expected_num_objects": 0, 2021-02-10T11:48:52.657 INFO:teuthology.orchestra.run.gibba012.stdout: "fast_read": false, 2021-02-10T11:48:52.657 INFO:teuthology.orchestra.run.gibba012.stdout: "options": {}, 2021-02-10T11:48:52.657 INFO:teuthology.orchestra.run.gibba012.stdout: "application_metadata": { 2021-02-10T11:48:52.657 INFO:teuthology.orchestra.run.gibba012.stdout: "cephfs": { 2021-02-10T11:48:52.657 INFO:teuthology.orchestra.run.gibba012.stdout: "data": "cephfs" 2021-02-10T11:48:52.657 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:48:52.657 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:48:52.657 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:48:52.657 INFO:teuthology.orchestra.run.gibba012.stdout: ], 2021-02-10T11:48:52.657 INFO:teuthology.orchestra.run.gibba012.stdout: "osds": [ 2021-02-10T11:48:52.657 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:48:52.657 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 0, 2021-02-10T11:48:52.657 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "95444b8f-6370-4a1d-ae5f-df62651b77da", 2021-02-10T11:48:52.658 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:48:52.658 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:48:52.658 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:48:52.658 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:48:52.658 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:48:52.658 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:48:52.658 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:48:52.658 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 185, 2021-02-10T11:48:52.658 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:48:52.658 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:48:52.658 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:48:52.658 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:48:52.659 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:48:52.659 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:48:52.659 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6817", 2021-02-10T11:48:52.659 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:48:52.659 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:48:52.659 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:48:52.659 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:48:52.659 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6818", 2021-02-10T11:48:52.659 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:48:52.659 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:48:52.659 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:48:52.659 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:48:52.659 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:48:52.660 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:48:52.660 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:48:52.660 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:48:52.660 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6820", 2021-02-10T11:48:52.660 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:48:52.660 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:48:52.660 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:48:52.660 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:48:52.660 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6822", 2021-02-10T11:48:52.660 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:48:52.660 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:48:52.660 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:48:52.660 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:48:52.660 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:48:52.661 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:48:52.661 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:48:52.661 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:48:52.661 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6828", 2021-02-10T11:48:52.661 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:48:52.661 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:48:52.661 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:48:52.661 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:48:52.661 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6830", 2021-02-10T11:48:52.661 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:48:52.661 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:48:52.661 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:48:52.661 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:48:52.662 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:48:52.662 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:48:52.662 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:48:52.662 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:48:52.662 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6824", 2021-02-10T11:48:52.662 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:48:52.662 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:48:52.662 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:48:52.662 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:48:52.662 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6826", 2021-02-10T11:48:52.662 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36030 2021-02-10T11:48:52.663 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:48:52.663 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:48:52.663 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:48:52.663 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.112:6818/36030", 2021-02-10T11:48:52.663 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.112:6822/36030", 2021-02-10T11:48:52.663 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.112:6830/36030", 2021-02-10T11:48:52.663 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.112:6826/36030", 2021-02-10T11:48:52.663 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:48:52.663 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:48:52.663 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:48:52.663 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:48:52.663 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:48:52.663 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:48:52.664 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 1, 2021-02-10T11:48:52.664 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "a23e217e-0d8e-486e-be07-712adfabd262", 2021-02-10T11:48:52.664 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:48:52.664 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:48:52.664 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:48:52.664 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:48:52.664 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:48:52.664 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:48:52.664 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:48:52.664 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 185, 2021-02-10T11:48:52.664 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:48:52.664 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:48:52.664 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:48:52.665 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:48:52.665 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:48:52.665 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:48:52.665 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6800", 2021-02-10T11:48:52.665 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:48:52.665 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:48:52.665 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:48:52.665 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:48:52.665 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6802", 2021-02-10T11:48:52.665 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:48:52.665 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:48:52.665 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:48:52.665 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:48:52.666 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:48:52.666 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:48:52.666 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:48:52.666 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:48:52.666 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6804", 2021-02-10T11:48:52.666 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:48:52.666 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:48:52.666 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:48:52.666 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:48:52.666 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6806", 2021-02-10T11:48:52.666 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:48:52.666 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:48:52.667 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:48:52.667 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:48:52.667 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:48:52.667 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:48:52.667 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:48:52.667 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:48:52.667 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6812", 2021-02-10T11:48:52.667 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:48:52.667 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:48:52.667 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:48:52.667 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:48:52.667 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6814", 2021-02-10T11:48:52.667 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:48:52.667 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:48:52.668 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:48:52.668 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:48:52.668 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:48:52.668 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:48:52.668 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:48:52.668 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:48:52.668 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6808", 2021-02-10T11:48:52.668 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:48:52.668 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:48:52.668 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:48:52.668 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:48:52.668 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6810", 2021-02-10T11:48:52.668 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36029 2021-02-10T11:48:52.669 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:48:52.669 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:48:52.669 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:48:52.669 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.112:6802/36029", 2021-02-10T11:48:52.669 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.112:6806/36029", 2021-02-10T11:48:52.669 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.112:6814/36029", 2021-02-10T11:48:52.669 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.112:6810/36029", 2021-02-10T11:48:52.669 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:48:52.669 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:48:52.669 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:48:52.669 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:48:52.669 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:48:52.669 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:48:52.669 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 2, 2021-02-10T11:48:52.670 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "026afa8a-c0f1-4b35-9a76-53c00fe0c2fc", 2021-02-10T11:48:52.670 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:48:52.670 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:48:52.670 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:48:52.670 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:48:52.670 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:48:52.670 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:48:52.670 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:48:52.670 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 187, 2021-02-10T11:48:52.670 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:48:52.670 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:48:52.670 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:48:52.670 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:48:52.671 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:48:52.671 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:48:52.671 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6801", 2021-02-10T11:48:52.671 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:48:52.671 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:48:52.671 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:48:52.671 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:48:52.671 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6803", 2021-02-10T11:48:52.671 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:48:52.671 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:48:52.671 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:48:52.671 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:48:52.672 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:48:52.672 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:48:52.672 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:48:52.672 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:48:52.672 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6805", 2021-02-10T11:48:52.672 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:48:52.672 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:48:52.672 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:48:52.672 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:48:52.672 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6807", 2021-02-10T11:48:52.672 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:48:52.672 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:48:52.672 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:48:52.673 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:48:52.673 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:48:52.673 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:48:52.673 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:48:52.673 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:48:52.673 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6813", 2021-02-10T11:48:52.673 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:48:52.673 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:48:52.673 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:48:52.673 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:48:52.673 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6815", 2021-02-10T11:48:52.673 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:48:52.673 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:48:52.673 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:48:52.674 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:48:52.674 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:48:52.674 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:48:52.674 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:48:52.674 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:48:52.674 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6809", 2021-02-10T11:48:52.674 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:48:52.674 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:48:52.674 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:48:52.674 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:48:52.674 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6811", 2021-02-10T11:48:52.674 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36031 2021-02-10T11:48:52.675 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:48:52.675 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:48:52.675 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:48:52.675 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.112:6803/36031", 2021-02-10T11:48:52.675 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.112:6807/36031", 2021-02-10T11:48:52.675 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.112:6815/36031", 2021-02-10T11:48:52.675 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.112:6811/36031", 2021-02-10T11:48:52.675 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:48:52.675 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:48:52.675 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:48:52.675 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:48:52.675 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:48:52.675 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:48:52.676 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 3, 2021-02-10T11:48:52.676 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "58795682-e00d-4173-bb84-f7075ca89a11", 2021-02-10T11:48:52.676 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:48:52.676 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:48:52.676 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:48:52.676 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:48:52.676 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:48:52.676 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:48:52.676 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:48:52.676 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 187, 2021-02-10T11:48:52.676 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:48:52.676 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:48:52.676 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:48:52.677 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:48:52.677 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:48:52.677 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:48:52.677 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6816", 2021-02-10T11:48:52.677 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:48:52.677 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:48:52.677 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:48:52.677 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:48:52.677 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6819", 2021-02-10T11:48:52.677 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:48:52.677 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:48:52.677 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:48:52.678 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:48:52.678 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:48:52.678 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:48:52.678 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:48:52.678 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:48:52.678 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6821", 2021-02-10T11:48:52.678 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:48:52.678 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:48:52.678 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:48:52.678 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:48:52.678 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6823", 2021-02-10T11:48:52.678 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:48:52.679 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:48:52.679 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:48:52.679 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:48:52.679 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:48:52.679 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:48:52.679 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:48:52.679 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:48:52.679 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6829", 2021-02-10T11:48:52.679 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:48:52.679 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:48:52.679 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:48:52.679 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:48:52.679 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6831", 2021-02-10T11:48:52.680 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:48:52.680 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:48:52.680 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:48:52.680 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:48:52.680 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:48:52.680 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:48:52.680 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:48:52.680 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:48:52.680 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6825", 2021-02-10T11:48:52.680 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:48:52.680 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:48:52.680 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:48:52.680 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:48:52.681 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:6827", 2021-02-10T11:48:52.681 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36028 2021-02-10T11:48:52.681 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:48:52.681 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:48:52.681 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:48:52.681 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.112:6819/36028", 2021-02-10T11:48:52.681 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.112:6823/36028", 2021-02-10T11:48:52.681 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.112:6831/36028", 2021-02-10T11:48:52.681 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.112:6827/36028", 2021-02-10T11:48:52.681 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:48:52.681 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:48:52.681 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:48:52.681 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:48:52.682 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:48:52.682 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:48:52.682 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 4, 2021-02-10T11:48:52.682 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "6763f8d6-53dc-4e45-bdf6-a6eabcf81e39", 2021-02-10T11:48:52.682 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:48:52.682 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:48:52.682 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:48:52.682 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:48:52.682 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:48:52.682 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:48:52.682 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:48:52.682 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 187, 2021-02-10T11:48:52.682 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:48:52.683 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:48:52.683 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:48:52.683 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:48:52.683 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:48:52.683 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:48:52.683 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6810", 2021-02-10T11:48:52.683 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:48:52.683 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:48:52.683 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:48:52.683 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:48:52.683 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6813", 2021-02-10T11:48:52.683 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:48:52.683 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:48:52.684 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:48:52.684 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:48:52.684 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:48:52.684 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:48:52.684 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:48:52.684 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:48:52.684 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6816", 2021-02-10T11:48:52.684 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:48:52.684 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:48:52.684 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:48:52.684 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:48:52.684 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6819", 2021-02-10T11:48:52.684 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:48:52.685 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:48:52.685 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:48:52.685 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:48:52.685 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:48:52.685 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:48:52.685 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:48:52.685 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:48:52.685 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6825", 2021-02-10T11:48:52.685 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:48:52.685 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:48:52.685 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:48:52.685 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:48:52.685 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6827", 2021-02-10T11:48:52.685 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:48:52.686 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:48:52.686 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:48:52.686 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:48:52.686 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:48:52.686 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:48:52.686 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:48:52.686 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:48:52.686 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6821", 2021-02-10T11:48:52.686 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:48:52.686 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:48:52.686 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:48:52.686 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:48:52.686 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6823", 2021-02-10T11:48:52.687 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36388 2021-02-10T11:48:52.687 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:48:52.687 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:48:52.687 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:48:52.687 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.120:6813/36388", 2021-02-10T11:48:52.687 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.120:6819/36388", 2021-02-10T11:48:52.687 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.120:6827/36388", 2021-02-10T11:48:52.687 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.120:6823/36388", 2021-02-10T11:48:52.687 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:48:52.687 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:48:52.687 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:48:52.687 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:48:52.687 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:48:52.688 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:48:52.688 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 5, 2021-02-10T11:48:52.688 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "5d0e4dd6-0cb0-4bf8-8314-626acaa1b544", 2021-02-10T11:48:52.688 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:48:52.688 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:48:52.688 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:48:52.688 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:48:52.688 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:48:52.688 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:48:52.688 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:48:52.688 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 178, 2021-02-10T11:48:52.688 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:48:52.688 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:48:52.688 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:48:52.689 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:48:52.689 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:48:52.689 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:48:52.689 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6804", 2021-02-10T11:48:52.689 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:48:52.689 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:48:52.689 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:48:52.689 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:48:52.689 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6806", 2021-02-10T11:48:52.689 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:48:52.689 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:48:52.689 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:48:52.690 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:48:52.690 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:48:52.690 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:48:52.690 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:48:52.690 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:48:52.690 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6809", 2021-02-10T11:48:52.690 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:48:52.690 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:48:52.690 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:48:52.690 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:48:52.690 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6812", 2021-02-10T11:48:52.691 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:48:52.691 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:48:52.691 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:48:52.691 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:48:52.691 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:48:52.691 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:48:52.691 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:48:52.691 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:48:52.691 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6820", 2021-02-10T11:48:52.691 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:48:52.691 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:48:52.691 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:48:52.691 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:48:52.692 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6822", 2021-02-10T11:48:52.692 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:48:52.692 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:48:52.692 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:48:52.692 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:48:52.692 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:48:52.692 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:48:52.692 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:48:52.692 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:48:52.692 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6815", 2021-02-10T11:48:52.692 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:48:52.692 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:48:52.692 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:48:52.692 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:48:52.693 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6818", 2021-02-10T11:48:52.693 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36387 2021-02-10T11:48:52.693 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:48:52.693 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:48:52.693 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:48:52.693 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.120:6806/36387", 2021-02-10T11:48:52.693 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.120:6812/36387", 2021-02-10T11:48:52.693 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.120:6822/36387", 2021-02-10T11:48:52.693 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.120:6818/36387", 2021-02-10T11:48:52.693 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:48:52.693 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:48:52.693 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:48:52.694 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:48:52.694 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:48:52.694 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:48:52.694 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 6, 2021-02-10T11:48:52.694 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "b010e187-4541-4b00-80dd-f948f6ab8f15", 2021-02-10T11:48:52.694 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:48:52.694 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:48:52.694 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:48:52.694 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:48:52.694 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:48:52.694 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:48:52.694 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:48:52.695 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 187, 2021-02-10T11:48:52.695 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:48:52.695 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:48:52.695 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:48:52.695 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:48:52.695 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:48:52.695 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:48:52.695 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6824", 2021-02-10T11:48:52.695 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:48:52.695 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:48:52.695 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:48:52.695 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:48:52.696 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6826", 2021-02-10T11:48:52.696 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:48:52.696 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:48:52.696 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:48:52.696 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:48:52.696 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:48:52.696 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:48:52.696 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:48:52.696 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:48:52.696 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6828", 2021-02-10T11:48:52.696 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:48:52.696 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:48:52.696 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:48:52.697 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:48:52.697 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6829", 2021-02-10T11:48:52.697 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:48:52.697 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:48:52.697 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:48:52.697 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:48:52.697 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:48:52.697 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:48:52.697 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:48:52.697 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:48:52.697 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6832", 2021-02-10T11:48:52.697 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:48:52.698 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:48:52.698 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:48:52.698 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:48:52.698 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6833", 2021-02-10T11:48:52.698 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:48:52.698 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:48:52.698 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:48:52.698 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:48:52.698 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:48:52.698 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:48:52.698 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:48:52.698 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:48:52.699 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6830", 2021-02-10T11:48:52.699 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:48:52.699 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:48:52.699 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:48:52.699 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:48:52.699 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6831", 2021-02-10T11:48:52.699 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36385 2021-02-10T11:48:52.699 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:48:52.699 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:48:52.699 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:48:52.699 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.120:6826/36385", 2021-02-10T11:48:52.699 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.120:6829/36385", 2021-02-10T11:48:52.699 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.120:6833/36385", 2021-02-10T11:48:52.700 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.120:6831/36385", 2021-02-10T11:48:52.700 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:48:52.700 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:48:52.700 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:48:52.700 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:48:52.700 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:48:52.700 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:48:52.700 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 7, 2021-02-10T11:48:52.700 INFO:teuthology.orchestra.run.gibba012.stdout: "uuid": "671e2772-33b7-497e-8e44-2178bd850bcf", 2021-02-10T11:48:52.700 INFO:teuthology.orchestra.run.gibba012.stdout: "up": 1, 2021-02-10T11:48:52.700 INFO:teuthology.orchestra.run.gibba012.stdout: "in": 1, 2021-02-10T11:48:52.700 INFO:teuthology.orchestra.run.gibba012.stdout: "weight": 1, 2021-02-10T11:48:52.700 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_affinity": 1, 2021-02-10T11:48:52.701 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_begin": 0, 2021-02-10T11:48:52.701 INFO:teuthology.orchestra.run.gibba012.stdout: "last_clean_end": 0, 2021-02-10T11:48:52.701 INFO:teuthology.orchestra.run.gibba012.stdout: "up_from": 13, 2021-02-10T11:48:52.701 INFO:teuthology.orchestra.run.gibba012.stdout: "up_thru": 187, 2021-02-10T11:48:52.701 INFO:teuthology.orchestra.run.gibba012.stdout: "down_at": 0, 2021-02-10T11:48:52.701 INFO:teuthology.orchestra.run.gibba012.stdout: "lost_at": 0, 2021-02-10T11:48:52.701 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addrs": { 2021-02-10T11:48:52.701 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:48:52.701 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:48:52.701 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:48:52.701 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6802", 2021-02-10T11:48:52.701 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:48:52.701 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:48:52.701 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:48:52.702 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:48:52.702 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6803", 2021-02-10T11:48:52.702 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:48:52.702 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:48:52.702 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:48:52.702 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:48:52.702 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addrs": { 2021-02-10T11:48:52.702 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:48:52.702 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:48:52.702 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:48:52.702 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6805", 2021-02-10T11:48:52.702 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:48:52.703 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:48:52.703 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:48:52.703 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:48:52.703 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6807", 2021-02-10T11:48:52.703 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:48:52.703 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:48:52.703 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:48:52.703 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:48:52.703 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addrs": { 2021-02-10T11:48:52.703 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:48:52.703 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:48:52.703 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:48:52.703 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6814", 2021-02-10T11:48:52.704 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:48:52.704 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:48:52.704 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:48:52.704 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:48:52.704 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6817", 2021-02-10T11:48:52.704 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:48:52.704 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:48:52.704 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:48:52.704 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:48:52.704 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addrs": { 2021-02-10T11:48:52.704 INFO:teuthology.orchestra.run.gibba012.stdout: "addrvec": [ 2021-02-10T11:48:52.704 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:48:52.704 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v2", 2021-02-10T11:48:52.705 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6808", 2021-02-10T11:48:52.705 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:48:52.705 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:48:52.705 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:48:52.705 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:48:52.705 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.120:6811", 2021-02-10T11:48:52.705 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 36386 2021-02-10T11:48:52.705 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:48:52.705 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:48:52.705 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:48:52.705 INFO:teuthology.orchestra.run.gibba012.stdout: "public_addr": "172.21.2.120:6803/36386", 2021-02-10T11:48:52.705 INFO:teuthology.orchestra.run.gibba012.stdout: "cluster_addr": "172.21.2.120:6807/36386", 2021-02-10T11:48:52.705 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_back_addr": "172.21.2.120:6817/36386", 2021-02-10T11:48:52.706 INFO:teuthology.orchestra.run.gibba012.stdout: "heartbeat_front_addr": "172.21.2.120:6811/36386", 2021-02-10T11:48:52.706 INFO:teuthology.orchestra.run.gibba012.stdout: "state": [ 2021-02-10T11:48:52.706 INFO:teuthology.orchestra.run.gibba012.stdout: "exists", 2021-02-10T11:48:52.706 INFO:teuthology.orchestra.run.gibba012.stdout: "up" 2021-02-10T11:48:52.706 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:48:52.706 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:48:52.706 INFO:teuthology.orchestra.run.gibba012.stdout: ], 2021-02-10T11:48:52.706 INFO:teuthology.orchestra.run.gibba012.stdout: "osd_xinfo": [ 2021-02-10T11:48:52.706 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:48:52.706 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 0, 2021-02-10T11:48:52.706 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:48:52.706 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:48:52.706 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:48:52.707 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:48:52.707 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:48:52.707 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:48:52.707 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:48:52.707 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 1, 2021-02-10T11:48:52.707 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:48:52.707 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:48:52.707 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:48:52.707 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:48:52.707 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:48:52.707 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:48:52.707 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:48:52.708 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 2, 2021-02-10T11:48:52.708 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:48:52.708 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:48:52.708 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:48:52.708 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:48:52.708 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:48:52.708 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:48:52.708 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:48:52.708 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 3, 2021-02-10T11:48:52.708 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:48:52.708 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:48:52.708 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:48:52.708 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:48:52.709 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:48:52.709 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:48:52.709 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:48:52.709 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 4, 2021-02-10T11:48:52.709 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:48:52.709 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:48:52.709 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:48:52.709 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:48:52.709 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:48:52.709 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:48:52.709 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:48:52.709 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 5, 2021-02-10T11:48:52.709 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:48:52.710 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:48:52.710 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:48:52.710 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:48:52.710 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:48:52.710 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:48:52.710 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:48:52.710 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 6, 2021-02-10T11:48:52.710 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:48:52.710 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:48:52.710 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:48:52.710 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:48:52.710 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:48:52.711 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:48:52.711 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:48:52.711 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": 7, 2021-02-10T11:48:52.711 INFO:teuthology.orchestra.run.gibba012.stdout: "down_stamp": "0.000000", 2021-02-10T11:48:52.711 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_probability": 0, 2021-02-10T11:48:52.711 INFO:teuthology.orchestra.run.gibba012.stdout: "laggy_interval": 0, 2021-02-10T11:48:52.711 INFO:teuthology.orchestra.run.gibba012.stdout: "features": 4611087854035861503, 2021-02-10T11:48:52.711 INFO:teuthology.orchestra.run.gibba012.stdout: "old_weight": 0 2021-02-10T11:48:52.711 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:48:52.711 INFO:teuthology.orchestra.run.gibba012.stdout: ], 2021-02-10T11:48:52.711 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_upmap": [], 2021-02-10T11:48:52.711 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_upmap_items": [], 2021-02-10T11:48:52.711 INFO:teuthology.orchestra.run.gibba012.stdout: "pg_temp": [], 2021-02-10T11:48:52.711 INFO:teuthology.orchestra.run.gibba012.stdout: "primary_temp": [], 2021-02-10T11:48:52.712 INFO:teuthology.orchestra.run.gibba012.stdout: "blacklist": {}, 2021-02-10T11:48:52.712 INFO:teuthology.orchestra.run.gibba012.stdout: "erasure_code_profiles": { 2021-02-10T11:48:52.712 INFO:teuthology.orchestra.run.gibba012.stdout: "default": { 2021-02-10T11:48:52.712 INFO:teuthology.orchestra.run.gibba012.stdout: "crush-failure-domain": "osd", 2021-02-10T11:48:52.712 INFO:teuthology.orchestra.run.gibba012.stdout: "k": "2", 2021-02-10T11:48:52.712 INFO:teuthology.orchestra.run.gibba012.stdout: "m": "1", 2021-02-10T11:48:52.712 INFO:teuthology.orchestra.run.gibba012.stdout: "plugin": "jerasure", 2021-02-10T11:48:52.712 INFO:teuthology.orchestra.run.gibba012.stdout: "ruleset-failure-domain": "osd", 2021-02-10T11:48:52.712 INFO:teuthology.orchestra.run.gibba012.stdout: "technique": "reed_sol_van" 2021-02-10T11:48:52.712 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:48:52.712 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:48:52.712 INFO:teuthology.orchestra.run.gibba012.stdout: "removed_snaps_queue": [], 2021-02-10T11:48:52.712 INFO:teuthology.orchestra.run.gibba012.stdout: "new_removed_snaps": [], 2021-02-10T11:48:52.713 INFO:teuthology.orchestra.run.gibba012.stdout: "new_purged_snaps": [], 2021-02-10T11:48:52.713 INFO:teuthology.orchestra.run.gibba012.stdout: "crush_node_flags": {}, 2021-02-10T11:48:52.713 INFO:teuthology.orchestra.run.gibba012.stdout: "device_class_flags": {} 2021-02-10T11:48:52.713 INFO:teuthology.orchestra.run.gibba012.stdout:} 2021-02-10T11:48:52.714 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:52.635 7f817158b700 1 -- 172.21.2.112:0/3831850318 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f81500186c0 msgr2=0x7f815001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:52.714 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:52.635 7f817158b700 1 --2- 172.21.2.112:0/3831850318 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f81500186c0 0x7f815001abc0 crc :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:52.714 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:52.635 7f817158b700 1 -- 172.21.2.112:0/3831850318 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f816c108ce0 msgr2=0x7f816c09da50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:52.714 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:52.635 7f817158b700 1 --2- 172.21.2.112:0/3831850318 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f816c108ce0 0x7f816c09da50 secure :-1 s=READY pgs=3216 cs=0 l=1 rev1=1 rx=0x7f815c006f00 tx=0x7f815c00b940).stop 2021-02-10T11:48:52.714 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:52.636 7f817158b700 1 -- 172.21.2.112:0/3831850318 shutdown_connections 2021-02-10T11:48:52.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:52.636 7f817158b700 1 --2- 172.21.2.112:0/3831850318 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f81500186c0 0x7f815001abc0 unknown :-1 s=CLOSED pgs=1449 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:52.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:52.636 7f817158b700 1 --2- 172.21.2.112:0/3831850318 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f816c1090e0 0x7f816c09e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:52.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:52.636 7f817158b700 1 --2- 172.21.2.112:0/3831850318 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f816c09eee0 0x7f816c09f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:52.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:52.636 7f817158b700 1 --2- 172.21.2.112:0/3831850318 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f816c108ce0 0x7f816c09da50 unknown :-1 s=CLOSED pgs=3216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:52.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:52.636 7f817158b700 1 -- 172.21.2.112:0/3831850318 >> 172.21.2.112:0/3831850318 conn(0x7f816c0bc6c0 msgr2=0x7f816c0b1f70 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:52.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:52.636 7f817158b700 1 -- 172.21.2.112:0/3831850318 shutdown_connections 2021-02-10T11:48:52.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:52.636 7f817158b700 1 -- 172.21.2.112:0/3831850318 wait complete. 2021-02-10T11:48:52.716 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:48:53.145 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.124 7f70421fe700 1 Processor -- start 2021-02-10T11:48:53.145 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.124 7f70421fe700 1 -- start start 2021-02-10T11:48:53.145 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.125 7f70421fe700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f703c0d0340 0x7f703c0ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:53.145 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.125 7f70421fe700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f703c0d1c70 0x7f703c0efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:53.145 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.125 7f70421fe700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f703c0f0b10 con 0x7f703c0d1870 2021-02-10T11:48:53.145 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.125 7f70421fe700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f703c0e5ed0 con 0x7f703c0d1c70 2021-02-10T11:48:53.145 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.125 7f70421fe700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f703c1b0be0 con 0x7f703c0d0340 2021-02-10T11:48:53.146 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.125 7f704081b700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f703c0d1c70 0x7f703c0efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:53.146 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.125 7f703bfff700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7f703c0d1870 0x7f703c1b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:54570/0 (socket says 172.21.2.112:54570) 2021-02-10T11:48:53.146 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.125 7f703bfff700 1 -- 172.21.2.112:0/788800873 learned_addr learned my addr 172.21.2.112:0/788800873 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:48:53.146 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.125 7f703b7fe700 1 --2- 172.21.2.112:0/788800873 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f703c0d0340 0x7f703c0ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:53.146 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.126 7f703b7fe700 1 -- 172.21.2.112:0/788800873 >> v2:172.21.2.120:3300/0 conn(0x7f703c0d1c70 msgr2=0x7f703c0efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:53.146 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.126 7f703b7fe700 1 --2- 172.21.2.112:0/788800873 >> v2:172.21.2.120:3300/0 conn(0x7f703c0d1c70 0x7f703c0efc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:53.146 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.126 7f703affd700 1 -- 172.21.2.112:0/788800873 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4170171052 0 0) 0x7f702c0010c0 con 0x7f703c0d1870 2021-02-10T11:48:53.146 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.126 7f703b7fe700 1 -- 172.21.2.112:0/788800873 >> v1:172.21.2.120:6789/0 conn(0x7f703c0d1870 legacy=0x7f703c1b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:53.146 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.126 7f703b7fe700 1 -- 172.21.2.112:0/788800873 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f703c0f20a0 con 0x7f703c0d0340 2021-02-10T11:48:53.146 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.126 7f704081b700 1 --2- 172.21.2.112:0/788800873 >> v2:172.21.2.120:3300/0 conn(0x7f703c0d1c70 0x7f703c0efc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:48:53.147 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.126 7f703b7fe700 1 --2- 172.21.2.112:0/788800873 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f703c0d0340 0x7f703c0ef640 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7f7030006650 tx=0x7f7030009a80).ready entity=mon.2 client_cookie=df8691b67979f806 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:53.147 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.127 7f703affd700 1 -- 172.21.2.112:0/788800873 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f70300009b0 con 0x7f703c0d0340 2021-02-10T11:48:53.147 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.127 7f703affd700 1 -- 172.21.2.112:0/788800873 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f703000c0c0 con 0x7f703c0d0340 2021-02-10T11:48:53.147 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.127 7f703affd700 1 -- 172.21.2.112:0/788800873 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f703000c5e0 con 0x7f703c0d0340 2021-02-10T11:48:53.147 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.127 7f70421fe700 1 -- 172.21.2.112:0/788800873 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f703c0d0340 msgr2=0x7f703c0ef640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:53.147 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.127 7f70421fe700 1 --2- 172.21.2.112:0/788800873 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f703c0d0340 0x7f703c0ef640 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7f7030006650 tx=0x7f7030009a80).stop 2021-02-10T11:48:53.147 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.127 7f70421fe700 1 -- 172.21.2.112:0/788800873 shutdown_connections 2021-02-10T11:48:53.147 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.127 7f70421fe700 1 --2- 172.21.2.112:0/788800873 >> v2:172.21.2.120:3300/0 conn(0x7f703c0d1c70 0x7f703c0efc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:53.147 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.127 7f70421fe700 1 --2- 172.21.2.112:0/788800873 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f703c0d0340 0x7f703c0ef640 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:53.147 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.127 7f70421fe700 1 -- 172.21.2.112:0/788800873 >> 172.21.2.112:0/788800873 conn(0x7f703c0ce450 msgr2=0x7f703c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:53.147 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.127 7f70421fe700 1 -- 172.21.2.112:0/788800873 shutdown_connections 2021-02-10T11:48:53.147 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.127 7f70421fe700 1 -- 172.21.2.112:0/788800873 wait complete. 2021-02-10T11:48:53.149 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.142 7f70421fe700 1 Processor -- start 2021-02-10T11:48:53.149 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.142 7f70421fe700 1 -- start start 2021-02-10T11:48:53.149 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.142 7f70421fe700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f703c0c33c0 0x7f703c0ec230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:53.149 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.142 7f70421fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f703c0b5e30 0x7f703c19ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:53.149 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.142 7f70421fe700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f703c19f260 0x7f703c0ed010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:53.149 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.142 7f70421fe700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f703c0c3810 con 0x7f703c0b5e30 2021-02-10T11:48:53.150 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.142 7f70421fe700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f703c0eca90 con 0x7f703c19f260 2021-02-10T11:48:53.150 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.142 7f70421fe700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f703c0b2790 con 0x7f703c0c33c0 2021-02-10T11:48:53.150 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.143 7f703b7fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f703c0b5e30 0x7f703c19ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:53.150 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.143 7f703b7fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f703c0b5e30 0x7f703c19ec40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:33324/0 (socket says 172.21.2.112:33324) 2021-02-10T11:48:53.150 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.143 7f703b7fe700 1 -- 172.21.2.112:0/504304902 learned_addr learned my addr 172.21.2.112:0/504304902 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:48:53.150 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.143 7f704081b700 1 --2- 172.21.2.112:0/504304902 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f703c19f260 0x7f703c0ed010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:53.150 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.143 7f703bfff700 1 --2- 172.21.2.112:0/504304902 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f703c0c33c0 0x7f703c0ec230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:53.150 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.143 7f703b7fe700 1 -- 172.21.2.112:0/504304902 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f703c0c33c0 msgr2=0x7f703c0ec230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:53.150 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.143 7f703b7fe700 1 --2- 172.21.2.112:0/504304902 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f703c0c33c0 0x7f703c0ec230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:53.150 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.143 7f703b7fe700 1 -- 172.21.2.112:0/504304902 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f703c19f260 msgr2=0x7f703c0ed010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:53.150 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.143 7f703b7fe700 1 --2- 172.21.2.112:0/504304902 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f703c19f260 0x7f703c0ed010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:53.150 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.143 7f703b7fe700 1 -- 172.21.2.112:0/504304902 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f703c19f660 con 0x7f703c0b5e30 2021-02-10T11:48:53.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.143 7f703b7fe700 1 --2- 172.21.2.112:0/504304902 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f703c0b5e30 0x7f703c19ec40 secure :-1 s=READY pgs=3217 cs=0 l=1 rev1=1 rx=0x7f7030009630 tx=0x7f703000b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:53.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.144 7f7038ff9700 1 -- 172.21.2.112:0/504304902 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f70300093a0 con 0x7f703c0b5e30 2021-02-10T11:48:53.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.144 7f7038ff9700 1 -- 172.21.2.112:0/504304902 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f703000bd70 con 0x7f703c0b5e30 2021-02-10T11:48:53.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.144 7f7038ff9700 1 -- 172.21.2.112:0/504304902 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7030005560 con 0x7f703c0b5e30 2021-02-10T11:48:53.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.144 7f70421fe700 1 -- 172.21.2.112:0/504304902 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f703c19f660 con 0x7f703c0b5e30 2021-02-10T11:48:53.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.144 7f70421fe700 1 -- 172.21.2.112:0/504304902 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f703c19f660 con 0x7f703c0b5e30 2021-02-10T11:48:53.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.144 7f7038ff9700 1 -- 172.21.2.112:0/504304902 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7030005770 con 0x7f703c0b5e30 2021-02-10T11:48:53.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.144 7f7038ff9700 1 --2- 172.21.2.112:0/504304902 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f701c0186c0 0x7f701c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:53.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.144 7f7038ff9700 1 -- 172.21.2.112:0/504304902 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f7030032620 con 0x7f703c0b5e30 2021-02-10T11:48:53.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.145 7f703bfff700 1 --2- 172.21.2.112:0/504304902 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f701c0186c0 0x7f701c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:53.151 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.145 7f703bfff700 1 --2- 172.21.2.112:0/504304902 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f701c0186c0 0x7f701c01abc0 crc :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:53.178 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.175 7f70421fe700 1 -- 172.21.2.112:0/504304902 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f703c0ec850 con 0x7f703c0b5e30 2021-02-10T11:48:53.181 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.177 7f7038ff9700 1 -- 172.21.2.112:0/504304902 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7030005560 con 0x7f703c0b5e30 2021-02-10T11:48:53.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.306 7f70421fe700 1 -- 172.21.2.112:0/504304902 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f703c0ec850 con 0x7f703c0b5e30 2021-02-10T11:48:53.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.306 7f7038ff9700 1 -- 172.21.2.112:0/504304902 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 166 v166) v1 ==== 95+0+2107 (secure 0 0 0) 0x7f703000bd70 con 0x7f703c0b5e30 2021-02-10T11:48:53.311 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 166 2021-02-10T11:48:53.311 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:48:53.311 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":166,"default_fscid":20,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7763,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/1478301417","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":1478301417},{"type":"v1","addr":"172.21.2.112:6833","nonce":1478301417}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":161}],"filesystems":[{"mdsmap":{"epoch":165,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:48:10.481032","modified":"2021-02-10 11:48:12.490392","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7760},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7760":{"gid":7760,"name":"b","rank":0,"incarnation":162,"state":"up:active","state_seq":3,"addr":"172.21.2.120:6835/2943198956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":2943198956},{"type":"v1","addr":"172.21.2.120:6835","nonce":2943198956}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[41],"metadata_pool":40,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":20}]} 2021-02-10T11:48:53.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.308 7f70421fe700 1 -- 172.21.2.112:0/504304902 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f701c0186c0 msgr2=0x7f701c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:53.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.308 7f70421fe700 1 --2- 172.21.2.112:0/504304902 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f701c0186c0 0x7f701c01abc0 crc :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:53.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.308 7f70421fe700 1 -- 172.21.2.112:0/504304902 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f703c0b5e30 msgr2=0x7f703c19ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:53.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.308 7f70421fe700 1 --2- 172.21.2.112:0/504304902 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f703c0b5e30 0x7f703c19ec40 secure :-1 s=READY pgs=3217 cs=0 l=1 rev1=1 rx=0x7f7030009630 tx=0x7f703000b980).stop 2021-02-10T11:48:53.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.308 7f70421fe700 1 -- 172.21.2.112:0/504304902 shutdown_connections 2021-02-10T11:48:53.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.308 7f70421fe700 1 --2- 172.21.2.112:0/504304902 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f701c0186c0 0x7f701c01abc0 unknown :-1 s=CLOSED pgs=1450 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:53.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.308 7f70421fe700 1 --2- 172.21.2.112:0/504304902 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f703c19f260 0x7f703c0ed010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:53.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.308 7f70421fe700 1 --2- 172.21.2.112:0/504304902 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f703c0b5e30 0x7f703c19ec40 unknown :-1 s=CLOSED pgs=3217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:53.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.308 7f70421fe700 1 --2- 172.21.2.112:0/504304902 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f703c0c33c0 0x7f703c0ec230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:53.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.308 7f70421fe700 1 -- 172.21.2.112:0/504304902 >> 172.21.2.112:0/504304902 conn(0x7f703c0ebe30 msgr2=0x7f703c08a880 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:53.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.309 7f70421fe700 1 -- 172.21.2.112:0/504304902 shutdown_connections 2021-02-10T11:48:53.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.309 7f70421fe700 1 -- 172.21.2.112:0/504304902 wait complete. 2021-02-10T11:48:53.321 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs fail cephfs 2021-02-10T11:48:53.750 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.728 7fd34d54c700 1 Processor -- start 2021-02-10T11:48:53.750 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.728 7fd34d54c700 1 -- start start 2021-02-10T11:48:53.750 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.729 7fd34d54c700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd3480d0340 0x7fd3480ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:53.750 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.729 7fd34d54c700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3480f0b10 con 0x7fd3480d1c70 2021-02-10T11:48:53.751 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.729 7fd34d54c700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3481b0be0 con 0x7fd3480d1870 2021-02-10T11:48:53.751 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.729 7fd34d54c700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3480e5ed0 con 0x7fd3480d0340 2021-02-10T11:48:53.751 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.729 7fd347ac0700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fd3480d1c70 0x7fd3480efc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:32798/0 (socket says 172.21.2.112:32798) 2021-02-10T11:48:53.751 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.729 7fd347ac0700 1 -- 172.21.2.112:0/1757535008 learned_addr learned my addr 172.21.2.112:0/1757535008 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:48:53.751 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.729 7fd3462bd700 1 -- 172.21.2.112:0/1757535008 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1870294303 0 0) 0x7fd340001500 con 0x7fd3480d1c70 2021-02-10T11:48:53.751 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.729 7fd3462bd700 1 -- 172.21.2.112:0/1757535008 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd334002fe0 con 0x7fd3480d1c70 2021-02-10T11:48:53.751 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.729 7fd346abe700 1 --2- 172.21.2.112:0/1757535008 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd3480d0340 0x7fd3480ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:53.751 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.730 7fd3462bd700 1 -- 172.21.2.112:0/1757535008 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3335536271 0 0) 0x7fd340001500 con 0x7fd3480d1c70 2021-02-10T11:48:53.751 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.730 7fd3462bd700 1 -- 172.21.2.112:0/1757535008 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd3480d0340 msgr2=0x7fd3480ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:53.751 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.730 7fd3462bd700 1 --2- 172.21.2.112:0/1757535008 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd3480d0340 0x7fd3480ef640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:53.751 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.730 7fd3462bd700 1 -- 172.21.2.112:0/1757535008 >> v1:172.21.2.120:6789/0 conn(0x7fd3480d1870 legacy=0x7fd3481b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:53.751 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.730 7fd3462bd700 1 -- 172.21.2.112:0/1757535008 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3480f0450 con 0x7fd3480d1c70 2021-02-10T11:48:53.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.730 7fd3462bd700 1 -- 172.21.2.112:0/1757535008 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7fd340001a00 con 0x7fd3480d1c70 2021-02-10T11:48:53.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.730 7fd3462bd700 1 -- 172.21.2.112:0/1757535008 >> v1:172.21.2.112:6789/0 conn(0x7fd3480d1c70 legacy=0x7fd3480efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:53.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.730 7fd3462bd700 1 --2- 172.21.2.112:0/1757535008 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd334003a60 0x7fd334005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:53.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.730 7fd3462bd700 1 --2- 172.21.2.112:0/1757535008 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd334006790 0x7fd334008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:53.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.730 7fd3462bd700 1 --2- 172.21.2.112:0/1757535008 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd334009590 0x7fd33400ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:53.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.730 7fd3462bd700 1 -- 172.21.2.112:0/1757535008 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd33400c2d0 con 0x7fd334003a60 2021-02-10T11:48:53.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.730 7fd3462bd700 1 -- 172.21.2.112:0/1757535008 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd33400c4f0 con 0x7fd334006790 2021-02-10T11:48:53.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.730 7fd3462bd700 1 -- 172.21.2.112:0/1757535008 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd33400c710 con 0x7fd334009590 2021-02-10T11:48:53.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.730 7fd346abe700 1 --2- 172.21.2.112:0/1757535008 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd334006790 0x7fd334008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:53.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.730 7fd3472bf700 1 --2- 172.21.2.112:0/1757535008 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd334003a60 0x7fd334005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:53.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.730 7fd347ac0700 1 --2- 172.21.2.112:0/1757535008 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd334009590 0x7fd33400ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:53.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.731 7fd3472bf700 1 -- 172.21.2.112:0/1757535008 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd334009590 msgr2=0x7fd33400ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:53.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.731 7fd3472bf700 1 --2- 172.21.2.112:0/1757535008 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd334009590 0x7fd33400ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:53.752 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.731 7fd3472bf700 1 -- 172.21.2.112:0/1757535008 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd334006790 msgr2=0x7fd334008c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:53.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.731 7fd3472bf700 1 --2- 172.21.2.112:0/1757535008 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd334006790 0x7fd334008c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:53.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.731 7fd3472bf700 1 -- 172.21.2.112:0/1757535008 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd334050950 con 0x7fd334003a60 2021-02-10T11:48:53.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.731 7fd346abe700 1 --2- 172.21.2.112:0/1757535008 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd334006790 0x7fd334008c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:48:53.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.731 7fd3472bf700 1 --2- 172.21.2.112:0/1757535008 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd334003a60 0x7fd334005ef0 secure :-1 s=READY pgs=3219 cs=0 l=1 rev1=1 rx=0x7fd330006ad0 tx=0x7fd330009ac0).ready entity=mon.0 client_cookie=58741ae47b5414e1 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:53.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.731 7fd3462bd700 1 -- 172.21.2.112:0/1757535008 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd33000c0c0 con 0x7fd334003a60 2021-02-10T11:48:53.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.731 7fd3462bd700 1 -- 172.21.2.112:0/1757535008 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd33000c3c0 con 0x7fd334003a60 2021-02-10T11:48:53.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.731 7fd3462bd700 1 -- 172.21.2.112:0/1757535008 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd33000c810 con 0x7fd334003a60 2021-02-10T11:48:53.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.731 7fd34d54c700 1 -- 172.21.2.112:0/1757535008 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd334003a60 msgr2=0x7fd334005ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:53.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.731 7fd34d54c700 1 --2- 172.21.2.112:0/1757535008 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd334003a60 0x7fd334005ef0 secure :-1 s=READY pgs=3219 cs=0 l=1 rev1=1 rx=0x7fd330006ad0 tx=0x7fd330009ac0).stop 2021-02-10T11:48:53.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.731 7fd347ac0700 1 -- 172.21.2.112:0/1757535008 reap_dead start 2021-02-10T11:48:53.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.732 7fd34d54c700 1 -- 172.21.2.112:0/1757535008 shutdown_connections 2021-02-10T11:48:53.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.732 7fd34d54c700 1 -- 172.21.2.112:0/1757535008 >> 172.21.2.112:0/1757535008 conn(0x7fd3480ce450 msgr2=0x7fd3480d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:53.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.732 7fd34d54c700 1 -- 172.21.2.112:0/1757535008 shutdown_connections 2021-02-10T11:48:53.754 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.732 7fd34d54c700 1 -- 172.21.2.112:0/1757535008 wait complete. 2021-02-10T11:48:53.754 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.746 7fd34d54c700 1 Processor -- start 2021-02-10T11:48:53.754 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.747 7fd34d54c700 1 -- start start 2021-02-10T11:48:53.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.747 7fd34d54c700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd34819ffa0 0x7fd3480eda10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:53.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.747 7fd34d54c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd3481a03a0 0x7fd3480ee030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:53.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.747 7fd34d54c700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd3480ee650 0x7fd3480ebe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:53.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.747 7fd34d54c700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3480ed490 con 0x7fd3481a03a0 2021-02-10T11:48:53.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.747 7fd34d54c700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3480eea50 con 0x7fd3480ee650 2021-02-10T11:48:53.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.747 7fd34d54c700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3480eec30 con 0x7fd34819ffa0 2021-02-10T11:48:53.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.747 7fd346abe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd3481a03a0 0x7fd3480ee030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:53.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.747 7fd346abe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd3481a03a0 0x7fd3480ee030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:33348/0 (socket says 172.21.2.112:33348) 2021-02-10T11:48:53.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.747 7fd346abe700 1 -- 172.21.2.112:0/1454344136 learned_addr learned my addr 172.21.2.112:0/1454344136 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:48:53.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.748 7fd3472bf700 1 --2- 172.21.2.112:0/1454344136 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd34819ffa0 0x7fd3480eda10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:53.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.748 7fd347ac0700 1 --2- 172.21.2.112:0/1454344136 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd3480ee650 0x7fd3480ebe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:53.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.748 7fd346abe700 1 -- 172.21.2.112:0/1454344136 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd34819ffa0 msgr2=0x7fd3480eda10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:53.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.748 7fd346abe700 1 --2- 172.21.2.112:0/1454344136 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd34819ffa0 0x7fd3480eda10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:53.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.748 7fd346abe700 1 -- 172.21.2.112:0/1454344136 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd3480ee650 msgr2=0x7fd3480ebe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:53.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.748 7fd346abe700 1 --2- 172.21.2.112:0/1454344136 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd3480ee650 0x7fd3480ebe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:53.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.748 7fd346abe700 1 -- 172.21.2.112:0/1454344136 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3480ec4a0 con 0x7fd3481a03a0 2021-02-10T11:48:53.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.748 7fd346abe700 1 --2- 172.21.2.112:0/1454344136 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd3481a03a0 0x7fd3480ee030 secure :-1 s=READY pgs=3220 cs=0 l=1 rev1=1 rx=0x7fd33800b610 tx=0x7fd33800b9b0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:53.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.748 7fd33ffff700 1 -- 172.21.2.112:0/1454344136 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd3380009b0 con 0x7fd3481a03a0 2021-02-10T11:48:53.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.748 7fd33ffff700 1 -- 172.21.2.112:0/1454344136 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd33800bda0 con 0x7fd3481a03a0 2021-02-10T11:48:53.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.748 7fd34d54c700 1 -- 172.21.2.112:0/1454344136 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3480ec4a0 con 0x7fd3481a03a0 2021-02-10T11:48:53.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.748 7fd33ffff700 1 -- 172.21.2.112:0/1454344136 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd338002220 con 0x7fd3481a03a0 2021-02-10T11:48:53.757 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.748 7fd34d54c700 1 -- 172.21.2.112:0/1454344136 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3480eca30 con 0x7fd3481a03a0 2021-02-10T11:48:53.757 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.749 7fd33ffff700 1 -- 172.21.2.112:0/1454344136 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd338000c90 con 0x7fd3481a03a0 2021-02-10T11:48:53.757 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.749 7fd33ffff700 1 --2- 172.21.2.112:0/1454344136 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd32c0186c0 0x7fd32c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:53.757 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.749 7fd33ffff700 1 -- 172.21.2.112:0/1454344136 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fd338032430 con 0x7fd3481a03a0 2021-02-10T11:48:53.757 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.750 7fd3472bf700 1 --2- 172.21.2.112:0/1454344136 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd32c0186c0 0x7fd32c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:53.757 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.750 7fd3472bf700 1 --2- 172.21.2.112:0/1454344136 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd32c0186c0 0x7fd32c01abc0 crc :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:53.782 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.779 7fd34d54c700 1 -- 172.21.2.112:0/1454344136 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3480ebc50 con 0x7fd3481a03a0 2021-02-10T11:48:53.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.781 7fd33ffff700 1 -- 172.21.2.112:0/1454344136 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd3380009b0 con 0x7fd3481a03a0 2021-02-10T11:48:53.913 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:53.910 7fd34d54c700 1 -- 172.21.2.112:0/1454344136 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fd3480ebc50 con 0x7fd3481a03a0 2021-02-10T11:48:54.311 INFO:tasks.ceph.mon.a.gibba012.stderr:2021-02-10 11:48:54.308 7fc3fe6e2700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-02-10T11:48:54.314 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:54.311 7fd33ffff700 1 -- 172.21.2.112:0/1454344136 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v167) v1 ==== 161+0+0 (secure 0 0 0) 0x7fd338000c90 con 0x7fd3481a03a0 2021-02-10T11:48:54.330 INFO:teuthology.orchestra.run.gibba012.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2021-02-10T11:48:54.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:54.329 7fd34d54c700 1 -- 172.21.2.112:0/1454344136 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd32c0186c0 msgr2=0x7fd32c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:54.332 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:54.329 7fd34d54c700 1 --2- 172.21.2.112:0/1454344136 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd32c0186c0 0x7fd32c01abc0 crc :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:54.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:54.329 7fd34d54c700 1 -- 172.21.2.112:0/1454344136 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd3481a03a0 msgr2=0x7fd3480ee030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:54.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:54.329 7fd34d54c700 1 --2- 172.21.2.112:0/1454344136 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd3481a03a0 0x7fd3480ee030 secure :-1 s=READY pgs=3220 cs=0 l=1 rev1=1 rx=0x7fd33800b610 tx=0x7fd33800b9b0).stop 2021-02-10T11:48:54.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:54.329 7fd34d54c700 1 -- 172.21.2.112:0/1454344136 shutdown_connections 2021-02-10T11:48:54.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:54.329 7fd34d54c700 1 --2- 172.21.2.112:0/1454344136 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fd32c0186c0 0x7fd32c01abc0 unknown :-1 s=CLOSED pgs=1451 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:54.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:54.329 7fd34d54c700 1 --2- 172.21.2.112:0/1454344136 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fd3480ee650 0x7fd3480ebe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:54.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:54.329 7fd34d54c700 1 --2- 172.21.2.112:0/1454344136 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fd3481a03a0 0x7fd3480ee030 unknown :-1 s=CLOSED pgs=3220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:54.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:54.329 7fd34d54c700 1 --2- 172.21.2.112:0/1454344136 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fd34819ffa0 0x7fd3480eda10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:54.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:54.329 7fd34d54c700 1 -- 172.21.2.112:0/1454344136 >> 172.21.2.112:0/1454344136 conn(0x7fd348094d90 msgr2=0x7fd34808a420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:54.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:54.330 7fd34d54c700 1 -- 172.21.2.112:0/1454344136 shutdown_connections 2021-02-10T11:48:54.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:54.330 7fd34d54c700 1 -- 172.21.2.112:0/1454344136 wait complete. 2021-02-10T11:48:54.342 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:48:54.366 INFO:tasks.ceph.mds.b.gibba020.stdout:starting mds.b at 2021-02-10T11:48:54.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:54.749 7f119f700700 1 Processor -- start 2021-02-10T11:48:54.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:54.749 7f119f700700 1 -- start start 2021-02-10T11:48:54.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:54.749 7f119f700700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f11980d0340 0x7f11980ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:54.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:54.749 7f119f700700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f11980d1c70 0x7f11980efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:54.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:54.749 7f119f700700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11980f0b10 con 0x7f11980d1870 2021-02-10T11:48:54.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:54.749 7f119f700700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11980e5ed0 con 0x7f11980d0340 2021-02-10T11:48:54.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:54.749 7f119f700700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11981b0be0 con 0x7f11980d1c70 2021-02-10T11:48:54.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:54.750 7f119dd1d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f11980d1c70 0x7f11980efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:54.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:54.750 7f119cd1b700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f11980d0340 0x7f11980ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:54.771 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:54.750 7f119d51c700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7f11980d1870 0x7f11981b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:54604/0 (socket says 172.21.2.112:54604) 2021-02-10T11:48:54.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:54.750 7f119dd1d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f11980d1c70 0x7f11980efc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3301/0 says I am v2:172.21.2.112:33038/0 (socket says 172.21.2.112:33038) 2021-02-10T11:48:54.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:54.750 7f119d51c700 1 -- 172.21.2.112:0/3786040758 learned_addr learned my addr 172.21.2.112:0/3786040758 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:48:54.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:54.751 7f119dd1d700 1 -- 172.21.2.112:0/3786040758 >> v2:172.21.2.120:3300/0 conn(0x7f11980d0340 msgr2=0x7f11980ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:54.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:54.751 7f119dd1d700 1 --2- 172.21.2.112:0/3786040758 >> v2:172.21.2.120:3300/0 conn(0x7f11980d0340 0x7f11980ef640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:54.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:54.751 7f119dd1d700 1 -- 172.21.2.112:0/3786040758 >> v1:172.21.2.120:6789/0 conn(0x7f11980d1870 legacy=0x7f11981b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:54.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:54.751 7f119dd1d700 1 -- 172.21.2.112:0/3786040758 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11980f20a0 con 0x7f11980d1c70 2021-02-10T11:48:54.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:54.751 7f119cd1b700 1 --2- 172.21.2.112:0/3786040758 >> v2:172.21.2.120:3300/0 conn(0x7f11980d0340 0x7f11980ef640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:48:54.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:54.751 7f119dd1d700 1 --2- 172.21.2.112:0/3786040758 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f11980d1c70 0x7f11980efc60 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7f1190006e50 tx=0x7f1190009e70).ready entity=mon.2 client_cookie=58f1f5f786d021e0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:54.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:54.751 7f118bfff700 1 -- 172.21.2.112:0/3786040758 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f119000b510 con 0x7f11980d1c70 2021-02-10T11:48:54.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:54.751 7f118bfff700 1 -- 172.21.2.112:0/3786040758 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1190006790 con 0x7f11980d1c70 2021-02-10T11:48:54.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:54.751 7f118bfff700 1 -- 172.21.2.112:0/3786040758 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f119000b9c0 con 0x7f11980d1c70 2021-02-10T11:48:54.772 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:54.751 7f119f700700 1 -- 172.21.2.112:0/3786040758 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f11980d1c70 msgr2=0x7f11980efc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:54.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:54.751 7f119f700700 1 --2- 172.21.2.112:0/3786040758 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f11980d1c70 0x7f11980efc60 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7f1190006e50 tx=0x7f1190009e70).stop 2021-02-10T11:48:54.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:54.752 7f119f700700 1 -- 172.21.2.112:0/3786040758 shutdown_connections 2021-02-10T11:48:54.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:54.752 7f119f700700 1 --2- 172.21.2.112:0/3786040758 >> v2:172.21.2.120:3300/0 conn(0x7f11980d0340 0x7f11980ef640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:54.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:54.752 7f119f700700 1 --2- 172.21.2.112:0/3786040758 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f11980d1c70 0x7f11980efc60 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:54.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:54.752 7f119f700700 1 -- 172.21.2.112:0/3786040758 >> 172.21.2.112:0/3786040758 conn(0x7f11980ce450 msgr2=0x7f11980d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:54.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:54.752 7f119f700700 1 -- 172.21.2.112:0/3786040758 shutdown_connections 2021-02-10T11:48:54.773 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:54.752 7f119f700700 1 -- 172.21.2.112:0/3786040758 wait complete. 2021-02-10T11:48:54.774 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:54.766 7f119f700700 1 Processor -- start 2021-02-10T11:48:54.774 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:54.766 7f119f700700 1 -- start start 2021-02-10T11:48:54.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:54.767 7f119f700700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f11980936a0 0x7f119819fe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:54.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:54.767 7f119f700700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1198092e10 0x7f11980ae870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:54.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:54.767 7f119f700700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f11980aee90 0x7f1198093b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:54.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:54.767 7f119f700700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1198093210 con 0x7f11980aee90 2021-02-10T11:48:54.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:54.767 7f119f700700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1198094190 con 0x7f11980936a0 2021-02-10T11:48:54.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:54.767 7f119f700700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11980943b0 con 0x7f1198092e10 2021-02-10T11:48:54.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:54.767 7f119dd1d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f11980aee90 0x7f1198093b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:54.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:54.767 7f119dd1d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f11980aee90 0x7f1198093b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:33358/0 (socket says 172.21.2.112:33358) 2021-02-10T11:48:54.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:54.767 7f119dd1d700 1 -- 172.21.2.112:0/1374742699 learned_addr learned my addr 172.21.2.112:0/1374742699 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:48:54.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:54.767 7f119dd1d700 1 -- 172.21.2.112:0/1374742699 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1198092e10 msgr2=0x7f11980ae870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:54.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:54.767 7f119dd1d700 1 --2- 172.21.2.112:0/1374742699 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1198092e10 0x7f11980ae870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:54.775 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:54.767 7f119dd1d700 1 -- 172.21.2.112:0/1374742699 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f11980936a0 msgr2=0x7f119819fe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:54.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:54.767 7f119dd1d700 1 --2- 172.21.2.112:0/1374742699 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f11980936a0 0x7f119819fe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:54.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:54.767 7f119dd1d700 1 -- 172.21.2.112:0/1374742699 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1198191aa0 con 0x7f11980aee90 2021-02-10T11:48:54.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:54.768 7f119dd1d700 1 --2- 172.21.2.112:0/1374742699 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f11980aee90 0x7f1198093b70 secure :-1 s=READY pgs=3222 cs=0 l=1 rev1=1 rx=0x7f11900091d0 tx=0x7f119000bd60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:54.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:54.768 7f1189ffb700 1 -- 172.21.2.112:0/1374742699 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1190006ca0 con 0x7f11980aee90 2021-02-10T11:48:54.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:54.768 7f1189ffb700 1 -- 172.21.2.112:0/1374742699 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f119000c2f0 con 0x7f11980aee90 2021-02-10T11:48:54.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:54.768 7f119f700700 1 -- 172.21.2.112:0/1374742699 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1198191aa0 con 0x7f11980aee90 2021-02-10T11:48:54.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:54.768 7f119f700700 1 -- 172.21.2.112:0/1374742699 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1198192030 con 0x7f11980aee90 2021-02-10T11:48:54.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:54.768 7f1189ffb700 1 -- 172.21.2.112:0/1374742699 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f119000c750 con 0x7f11980aee90 2021-02-10T11:48:54.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:54.768 7f1189ffb700 1 -- 172.21.2.112:0/1374742699 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f11900056b0 con 0x7f11980aee90 2021-02-10T11:48:54.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:54.769 7f1189ffb700 1 --2- 172.21.2.112:0/1374742699 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f11780186c0 0x7f117801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:54.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:54.769 7f1189ffb700 1 -- 172.21.2.112:0/1374742699 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(190..190 src has 1..190) v4 ==== 5659+0+0 (secure 0 0 0) 0x7f1190032920 con 0x7f11980aee90 2021-02-10T11:48:54.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:54.769 7f119d51c700 1 --2- 172.21.2.112:0/1374742699 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f11780186c0 0x7f117801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:54.776 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:54.770 7f119d51c700 1 --2- 172.21.2.112:0/1374742699 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f11780186c0 0x7f117801abc0 crc :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:54.804 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:54.800 7f119f700700 1 -- 172.21.2.112:0/1374742699 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11980ebc60 con 0x7f11980aee90 2021-02-10T11:48:54.806 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:54.802 7f1189ffb700 1 -- 172.21.2.112:0/1374742699 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f119002d700 con 0x7f11980aee90 2021-02-10T11:48:54.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:54.930 7f119f700700 1 -- 172.21.2.112:0/1374742699 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f11980f0b10 con 0x7f11980aee90 2021-02-10T11:48:54.935 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:54.930 7f1189ffb700 1 -- 172.21.2.112:0/1374742699 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 167 v167) v1 ==== 95+0+1762 (secure 0 0 0) 0x7f119002d700 con 0x7f11980aee90 2021-02-10T11:48:54.935 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:48:54.935 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":167,"default_fscid":20,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7763,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/1478301417","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":1478301417},{"type":"v1","addr":"172.21.2.112:6833","nonce":1478301417}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":161}],"filesystems":[{"mdsmap":{"epoch":167,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:48:10.481032","modified":"2021-02-10 11:48:54.308310","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":190,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[41],"metadata_pool":40,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":20}]} 2021-02-10T11:48:54.936 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 167 2021-02-10T11:48:54.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:54.932 7f119f700700 1 -- 172.21.2.112:0/1374742699 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f11780186c0 msgr2=0x7f117801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:54.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:54.932 7f119f700700 1 --2- 172.21.2.112:0/1374742699 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f11780186c0 0x7f117801abc0 crc :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:54.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:54.932 7f119f700700 1 -- 172.21.2.112:0/1374742699 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f11980aee90 msgr2=0x7f1198093b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:54.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:54.932 7f119f700700 1 --2- 172.21.2.112:0/1374742699 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f11980aee90 0x7f1198093b70 secure :-1 s=READY pgs=3222 cs=0 l=1 rev1=1 rx=0x7f11900091d0 tx=0x7f119000bd60).stop 2021-02-10T11:48:54.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:54.932 7f119f700700 1 -- 172.21.2.112:0/1374742699 shutdown_connections 2021-02-10T11:48:54.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:54.932 7f119f700700 1 --2- 172.21.2.112:0/1374742699 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f11780186c0 0x7f117801abc0 unknown :-1 s=CLOSED pgs=1452 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:54.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:54.932 7f119f700700 1 --2- 172.21.2.112:0/1374742699 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f11980aee90 0x7f1198093b70 unknown :-1 s=CLOSED pgs=3222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:54.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:54.932 7f119f700700 1 --2- 172.21.2.112:0/1374742699 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1198092e10 0x7f11980ae870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:54.938 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:54.932 7f119f700700 1 --2- 172.21.2.112:0/1374742699 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f11980936a0 0x7f119819fe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:54.938 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:54.932 7f119f700700 1 -- 172.21.2.112:0/1374742699 >> 172.21.2.112:0/1374742699 conn(0x7f11981aeec0 msgr2=0x7f11981a37b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:54.938 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:54.933 7f119f700700 1 -- 172.21.2.112:0/1374742699 shutdown_connections 2021-02-10T11:48:54.938 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:54.933 7f119f700700 1 -- 172.21.2.112:0/1374742699 wait complete. 2021-02-10T11:48:54.946 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2021-02-10T11:48:55.379 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:55.358 7f91facb5700 1 Processor -- start 2021-02-10T11:48:55.379 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:55.358 7f91facb5700 1 -- start start 2021-02-10T11:48:55.379 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:55.359 7f91facb5700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f91f40d1870 0x7f91f41b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:55.380 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:55.359 7f91facb5700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91f40f0b10 con 0x7f91f41b0a10 2021-02-10T11:48:55.380 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:55.359 7f91facb5700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91f40d0310 con 0x7f91f40f0650 2021-02-10T11:48:55.380 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:55.359 7f91facb5700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91f40e5ed0 con 0x7f91f40d1870 2021-02-10T11:48:55.380 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:55.359 7f91f3fff700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f91f41b0a10 0x7f91f40cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:32828/0 (socket says 172.21.2.112:32828) 2021-02-10T11:48:55.380 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:55.359 7f91f3fff700 1 -- 172.21.2.112:0/2546461567 learned_addr learned my addr 172.21.2.112:0/2546461567 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:48:55.380 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:55.359 7f91f8ad1700 1 --2- 172.21.2.112:0/2546461567 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f91f40d1870 0x7f91f41b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:55.380 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:55.359 7f91f37fe700 1 -- 172.21.2.112:0/2546461567 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2075750059 0 0) 0x7f91e4001180 con 0x7f91f41b0a10 2021-02-10T11:48:55.380 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:55.359 7f91f37fe700 1 -- 172.21.2.112:0/2546461567 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91e0002fe0 con 0x7f91f41b0a10 2021-02-10T11:48:55.380 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:55.359 7f91f8ad1700 1 -- 172.21.2.112:0/2546461567 >> v1:172.21.2.120:6789/0 conn(0x7f91f40f0650 legacy=0x7f91f40cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:55.380 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:55.359 7f91f8ad1700 1 -- 172.21.2.112:0/2546461567 >> v1:172.21.2.112:6789/0 conn(0x7f91f41b0a10 legacy=0x7f91f40cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:55.380 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:55.360 7f91f8ad1700 1 -- 172.21.2.112:0/2546461567 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91f40d0530 con 0x7f91f40d1870 2021-02-10T11:48:55.380 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:55.360 7f91f8ad1700 1 --2- 172.21.2.112:0/2546461567 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f91f40d1870 0x7f91f41b03f0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0x7f91dc006650 tx=0x7f91dc009a80).ready entity=mon.2 client_cookie=3c324d3b241086bf server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:55.380 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:55.360 7f91f37fe700 1 -- 172.21.2.112:0/2546461567 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f91dc0009b0 con 0x7f91f40d1870 2021-02-10T11:48:55.381 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:55.360 7f91f37fe700 1 -- 172.21.2.112:0/2546461567 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91dc00c0c0 con 0x7f91f40d1870 2021-02-10T11:48:55.381 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:55.360 7f91f37fe700 1 -- 172.21.2.112:0/2546461567 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f91dc00c520 con 0x7f91f40d1870 2021-02-10T11:48:55.381 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:55.360 7f91facb5700 1 -- 172.21.2.112:0/2546461567 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f91f40d1870 msgr2=0x7f91f41b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:55.381 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:55.360 7f91facb5700 1 --2- 172.21.2.112:0/2546461567 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f91f40d1870 0x7f91f41b03f0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0x7f91dc006650 tx=0x7f91dc009a80).stop 2021-02-10T11:48:55.381 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:55.361 7f91facb5700 1 -- 172.21.2.112:0/2546461567 shutdown_connections 2021-02-10T11:48:55.381 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:55.361 7f91facb5700 1 --2- 172.21.2.112:0/2546461567 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f91f40d1870 0x7f91f41b03f0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:55.381 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:55.361 7f91facb5700 1 -- 172.21.2.112:0/2546461567 >> 172.21.2.112:0/2546461567 conn(0x7f91f40ce450 msgr2=0x7f91f40d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:55.381 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:55.361 7f91facb5700 1 -- 172.21.2.112:0/2546461567 shutdown_connections 2021-02-10T11:48:55.381 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:55.361 7f91facb5700 1 -- 172.21.2.112:0/2546461567 wait complete. 2021-02-10T11:48:55.382 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:55.375 7f91facb5700 1 Processor -- start 2021-02-10T11:48:55.383 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:55.375 7f91facb5700 1 -- start start 2021-02-10T11:48:55.383 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:55.376 7f91facb5700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f91f40a2fb0 0x7f91f40a33b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:55.383 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:55.376 7f91facb5700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f91f40a39d0 0x7f91f40a4860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:55.383 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:55.376 7f91facb5700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f91f40a3dd0 0x7f91f40a4e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:55.383 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:55.376 7f91facb5700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91f40a2620 con 0x7f91f40a39d0 2021-02-10T11:48:55.383 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:55.376 7f91facb5700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91f40a2840 con 0x7f91f40a3dd0 2021-02-10T11:48:55.383 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:55.376 7f91facb5700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91f40a2b80 con 0x7f91f40a2fb0 2021-02-10T11:48:55.383 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:55.376 7f91f3fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f91f40a39d0 0x7f91f40a4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:55.383 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:55.376 7f91f3fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f91f40a39d0 0x7f91f40a4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:33372/0 (socket says 172.21.2.112:33372) 2021-02-10T11:48:55.383 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:55.376 7f91f3fff700 1 -- 172.21.2.112:0/1349804611 learned_addr learned my addr 172.21.2.112:0/1349804611 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:48:55.383 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:55.376 7f91f8ad1700 1 --2- 172.21.2.112:0/1349804611 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f91f40a2fb0 0x7f91f40a33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:55.383 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:55.376 7f91f92d2700 1 --2- 172.21.2.112:0/1349804611 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f91f40a3dd0 0x7f91f40a4e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:55.384 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:55.376 7f91f3fff700 1 -- 172.21.2.112:0/1349804611 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f91f40a2fb0 msgr2=0x7f91f40a33b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:55.384 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:55.376 7f91f3fff700 1 --2- 172.21.2.112:0/1349804611 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f91f40a2fb0 0x7f91f40a33b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:55.384 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:55.376 7f91f3fff700 1 -- 172.21.2.112:0/1349804611 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f91f40a3dd0 msgr2=0x7f91f40a4e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:55.384 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:55.376 7f91f3fff700 1 --2- 172.21.2.112:0/1349804611 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f91f40a3dd0 0x7f91f40a4e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:55.384 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:55.376 7f91f3fff700 1 -- 172.21.2.112:0/1349804611 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91f40a54a0 con 0x7f91f40a39d0 2021-02-10T11:48:55.384 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:55.376 7f91f92d2700 1 --2- 172.21.2.112:0/1349804611 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f91f40a3dd0 0x7f91f40a4e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:48:55.384 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:55.376 7f91f3fff700 1 --2- 172.21.2.112:0/1349804611 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f91f40a39d0 0x7f91f40a4860 secure :-1 s=READY pgs=3224 cs=0 l=1 rev1=1 rx=0x7f91e400b530 tx=0x7f91e400b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:55.384 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:55.376 7f91f17fa700 1 -- 172.21.2.112:0/1349804611 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f91e400bcc0 con 0x7f91f40a39d0 2021-02-10T11:48:55.384 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:55.376 7f91f17fa700 1 -- 172.21.2.112:0/1349804611 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91e4006200 con 0x7f91f40a39d0 2021-02-10T11:48:55.384 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:55.376 7f91f17fa700 1 -- 172.21.2.112:0/1349804611 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f91e4004590 con 0x7f91f40a39d0 2021-02-10T11:48:55.384 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:55.376 7f91facb5700 1 -- 172.21.2.112:0/1349804611 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91f40a54a0 con 0x7f91f40a39d0 2021-02-10T11:48:55.384 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:55.376 7f91facb5700 1 -- 172.21.2.112:0/1349804611 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91f40a5a30 con 0x7f91f40a39d0 2021-02-10T11:48:55.384 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:55.377 7f91f17fa700 1 -- 172.21.2.112:0/1349804611 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f91e4009fb0 con 0x7f91f40a39d0 2021-02-10T11:48:55.384 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:55.377 7f91f17fa700 1 --2- 172.21.2.112:0/1349804611 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f91d80186c0 0x7f91d801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:55.385 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:55.377 7f91f17fa700 1 -- 172.21.2.112:0/1349804611 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(190..190 src has 1..190) v4 ==== 5659+0+0 (secure 0 0 0) 0x7f91e40327b0 con 0x7f91f40a39d0 2021-02-10T11:48:55.385 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:55.378 7f91f8ad1700 1 --2- 172.21.2.112:0/1349804611 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f91d80186c0 0x7f91d801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:55.385 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:55.378 7f91f8ad1700 1 --2- 172.21.2.112:0/1349804611 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f91d80186c0 0x7f91d801abc0 crc :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:55.413 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:55.408 7f91facb5700 1 -- 172.21.2.112:0/1349804611 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91f40ee060 con 0x7f91f40a39d0 2021-02-10T11:48:55.415 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:55.411 7f91f17fa700 1 -- 172.21.2.112:0/1349804611 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f91e40043a0 con 0x7f91f40a39d0 2021-02-10T11:48:55.543 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:55.539 7f91facb5700 1 -- 172.21.2.112:0/1349804611 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f91f40ce450 con 0x7f91f40a39d0 2021-02-10T11:48:56.326 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:56.321 7f91f17fa700 1 -- 172.21.2.112:0/1349804611 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v168) v1 ==== 104+0+0 (secure 0 0 0) 0x7f91e4009fb0 con 0x7f91f40a39d0 2021-02-10T11:48:56.366 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:56.361 7f91facb5700 1 -- 172.21.2.112:0/1349804611 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f91d80186c0 msgr2=0x7f91d801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:56.366 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:56.361 7f91facb5700 1 --2- 172.21.2.112:0/1349804611 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f91d80186c0 0x7f91d801abc0 crc :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:56.366 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:56.361 7f91facb5700 1 -- 172.21.2.112:0/1349804611 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f91f40a39d0 msgr2=0x7f91f40a4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:56.366 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:56.361 7f91facb5700 1 --2- 172.21.2.112:0/1349804611 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f91f40a39d0 0x7f91f40a4860 secure :-1 s=READY pgs=3224 cs=0 l=1 rev1=1 rx=0x7f91e400b530 tx=0x7f91e400b8d0).stop 2021-02-10T11:48:56.367 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:56.362 7f91facb5700 1 -- 172.21.2.112:0/1349804611 shutdown_connections 2021-02-10T11:48:56.367 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:56.362 7f91facb5700 1 --2- 172.21.2.112:0/1349804611 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f91d80186c0 0x7f91d801abc0 unknown :-1 s=CLOSED pgs=1453 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:56.367 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:56.362 7f91facb5700 1 --2- 172.21.2.112:0/1349804611 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f91f40a3dd0 0x7f91f40a4e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:56.367 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:56.362 7f91facb5700 1 --2- 172.21.2.112:0/1349804611 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f91f40a39d0 0x7f91f40a4860 unknown :-1 s=CLOSED pgs=3224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:56.367 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:56.362 7f91facb5700 1 --2- 172.21.2.112:0/1349804611 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f91f40a2fb0 0x7f91f40a33b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:56.367 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:56.362 7f91facb5700 1 -- 172.21.2.112:0/1349804611 >> 172.21.2.112:0/1349804611 conn(0x7f91f408f030 msgr2=0x7f91f40b5e80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:56.367 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:56.362 7f91facb5700 1 -- 172.21.2.112:0/1349804611 shutdown_connections 2021-02-10T11:48:56.367 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:56.362 7f91facb5700 1 -- 172.21.2.112:0/1349804611 wait complete. 2021-02-10T11:48:56.374 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2021-02-10T11:48:56.809 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:56.788 7f399efc7700 1 Processor -- start 2021-02-10T11:48:56.809 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:56.788 7f399efc7700 1 -- start start 2021-02-10T11:48:56.809 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:56.788 7f399efc7700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f39980d1c80 0x7f39980cef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:56.810 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:56.788 7f399efc7700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f39980cf590 0x7f39980cf990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:56.810 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:56.788 7f399efc7700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39980fe1d0 con 0x7f39980d1880 2021-02-10T11:48:56.810 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:56.788 7f399efc7700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39981b0cc0 con 0x7f39980d1c80 2021-02-10T11:48:56.810 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:56.788 7f399efc7700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3998109a70 con 0x7f39980cf590 2021-02-10T11:48:56.810 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:56.789 7f399cde3700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7f39980d1880 0x7f39981b04d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:54632/0 (socket says 172.21.2.112:54632) 2021-02-10T11:48:56.810 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:56.789 7f399cde3700 1 -- 172.21.2.112:0/1509404624 learned_addr learned my addr 172.21.2.112:0/1509404624 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:48:56.810 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:56.789 7f3997fff700 1 --2- 172.21.2.112:0/1509404624 >> v2:172.21.2.120:3300/0 conn(0x7f39980d1c80 0x7f39980cef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:56.810 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:56.789 7f399d5e4700 1 --2- 172.21.2.112:0/1509404624 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f39980cf590 0x7f39980cf990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:56.810 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:56.789 7f3997fff700 1 -- 172.21.2.112:0/1509404624 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f39980cf590 msgr2=0x7f39980cf990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:56.810 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:56.789 7f3997fff700 1 --2- 172.21.2.112:0/1509404624 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f39980cf590 0x7f39980cf990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:56.810 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:56.789 7f3997fff700 1 -- 172.21.2.112:0/1509404624 >> v1:172.21.2.120:6789/0 conn(0x7f39980d1880 legacy=0x7f39981b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:56.810 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:56.790 7f3997fff700 1 -- 172.21.2.112:0/1509404624 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3998088070 con 0x7f39980d1c80 2021-02-10T11:48:56.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:56.790 7f3997fff700 1 --2- 172.21.2.112:0/1509404624 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f39980d1c80 0x7f39980cef70 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 rx=0x7f3988006650 tx=0x7f3988009a80).ready entity=mon.1 client_cookie=39cb79cf54272310 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:56.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:56.790 7f39977fe700 1 -- 172.21.2.112:0/1509404624 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f398800c0c0 con 0x7f39980d1c80 2021-02-10T11:48:56.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:56.790 7f39977fe700 1 -- 172.21.2.112:0/1509404624 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f398800c340 con 0x7f39980d1c80 2021-02-10T11:48:56.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:56.790 7f39977fe700 1 -- 172.21.2.112:0/1509404624 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f398800c820 con 0x7f39980d1c80 2021-02-10T11:48:56.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:56.790 7f399efc7700 1 -- 172.21.2.112:0/1509404624 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f39980d1c80 msgr2=0x7f39980cef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:56.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:56.790 7f399efc7700 1 --2- 172.21.2.112:0/1509404624 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f39980d1c80 0x7f39980cef70 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 rx=0x7f3988006650 tx=0x7f3988009a80).stop 2021-02-10T11:48:56.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:56.791 7f399efc7700 1 -- 172.21.2.112:0/1509404624 shutdown_connections 2021-02-10T11:48:56.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:56.791 7f399efc7700 1 --2- 172.21.2.112:0/1509404624 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f39980d1c80 0x7f39980cef70 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:56.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:56.791 7f399efc7700 1 --2- 172.21.2.112:0/1509404624 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f39980cf590 0x7f39980cf990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:56.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:56.791 7f399efc7700 1 -- 172.21.2.112:0/1509404624 >> 172.21.2.112:0/1509404624 conn(0x7f39980ce430 msgr2=0x7f39980d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:56.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:56.791 7f399efc7700 1 -- 172.21.2.112:0/1509404624 shutdown_connections 2021-02-10T11:48:56.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:56.791 7f399efc7700 1 -- 172.21.2.112:0/1509404624 wait complete. 2021-02-10T11:48:56.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:56.805 7f399efc7700 1 Processor -- start 2021-02-10T11:48:56.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:56.805 7f399efc7700 1 -- start start 2021-02-10T11:48:56.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:56.806 7f399efc7700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f39981a4a10 0x7f3998187e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:56.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:56.806 7f399efc7700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f39980913a0 0x7f39980a8e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:56.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:56.806 7f399efc7700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f39980a9470 0x7f39980a7170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:56.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:56.806 7f399efc7700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39980a8950 con 0x7f39980913a0 2021-02-10T11:48:56.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:56.806 7f399efc7700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39980917a0 con 0x7f39980a9470 2021-02-10T11:48:56.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:56.806 7f399efc7700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39980a9870 con 0x7f39981a4a10 2021-02-10T11:48:56.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:56.806 7f3997fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f39980913a0 0x7f39980a8e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:56.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:56.806 7f3997fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f39980913a0 0x7f39980a8e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:33386/0 (socket says 172.21.2.112:33386) 2021-02-10T11:48:56.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:56.806 7f3997fff700 1 -- 172.21.2.112:0/4189645536 learned_addr learned my addr 172.21.2.112:0/4189645536 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:48:56.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:56.806 7f3997fff700 1 -- 172.21.2.112:0/4189645536 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f39981a4a10 msgr2=0x7f3998187e80 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:48:56.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:56.806 7f3997fff700 1 --2- 172.21.2.112:0/4189645536 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f39981a4a10 0x7f3998187e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:56.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:56.806 7f3997fff700 1 -- 172.21.2.112:0/4189645536 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f39980a9470 msgr2=0x7f39980a7170 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:48:56.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:56.806 7f3997fff700 1 --2- 172.21.2.112:0/4189645536 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f39980a9470 0x7f39980a7170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:56.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:56.806 7f3997fff700 1 -- 172.21.2.112:0/4189645536 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39980a9a90 con 0x7f39980913a0 2021-02-10T11:48:56.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:56.806 7f3997fff700 1 --2- 172.21.2.112:0/4189645536 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f39980913a0 0x7f39980a8e50 secure :-1 s=READY pgs=3225 cs=0 l=1 rev1=1 rx=0x7f3988009630 tx=0x7f398800b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:56.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:56.806 7f39957fa700 1 -- 172.21.2.112:0/4189645536 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f398800bd70 con 0x7f39980913a0 2021-02-10T11:48:56.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:56.806 7f39957fa700 1 -- 172.21.2.112:0/4189645536 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3988000960 con 0x7f39980913a0 2021-02-10T11:48:56.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:56.806 7f399efc7700 1 -- 172.21.2.112:0/4189645536 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39980a9a90 con 0x7f39980913a0 2021-02-10T11:48:56.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:56.807 7f399efc7700 1 -- 172.21.2.112:0/4189645536 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39980a7a80 con 0x7f39980913a0 2021-02-10T11:48:56.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:56.807 7f39957fa700 1 -- 172.21.2.112:0/4189645536 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3988005560 con 0x7f39980913a0 2021-02-10T11:48:56.815 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:56.807 7f39957fa700 1 -- 172.21.2.112:0/4189645536 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f398800a000 con 0x7f39980913a0 2021-02-10T11:48:56.815 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:56.807 7f39957fa700 1 --2- 172.21.2.112:0/4189645536 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f397c0186d0 0x7f397c01abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:56.815 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:56.807 7f39957fa700 1 -- 172.21.2.112:0/4189645536 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(190..190 src has 1..190) v4 ==== 5659+0+0 (secure 0 0 0) 0x7f3988032840 con 0x7f39980913a0 2021-02-10T11:48:56.815 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:56.808 7f399cde3700 1 --2- 172.21.2.112:0/4189645536 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f397c0186d0 0x7f397c01abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:56.815 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:56.808 7f399cde3700 1 --2- 172.21.2.112:0/4189645536 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f397c0186d0 0x7f397c01abd0 crc :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:56.842 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:56.838 7f399efc7700 1 -- 172.21.2.112:0/4189645536 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3998082aa0 con 0x7f39980913a0 2021-02-10T11:48:56.845 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:56.840 7f39957fa700 1 -- 172.21.2.112:0/4189645536 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3988000940 con 0x7f39980913a0 2021-02-10T11:48:56.972 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:56.968 7f399efc7700 1 -- 172.21.2.112:0/4189645536 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f39980a88d0 con 0x7f39980913a0 2021-02-10T11:48:57.330 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:57.325 7f39957fa700 1 -- 172.21.2.112:0/4189645536 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v191) v1 ==== 185+0+0 (secure 0 0 0) 0x7f398800a000 con 0x7f39980913a0 2021-02-10T11:48:57.338 INFO:teuthology.orchestra.run.gibba012.stderr:pool 'cephfs_metadata' removed 2021-02-10T11:48:57.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:57.335 7f399efc7700 1 -- 172.21.2.112:0/4189645536 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f397c0186d0 msgr2=0x7f397c01abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:57.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:57.335 7f399efc7700 1 --2- 172.21.2.112:0/4189645536 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f397c0186d0 0x7f397c01abd0 crc :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:57.340 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:57.335 7f399efc7700 1 -- 172.21.2.112:0/4189645536 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f39980913a0 msgr2=0x7f39980a8e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:57.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:57.335 7f399efc7700 1 --2- 172.21.2.112:0/4189645536 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f39980913a0 0x7f39980a8e50 secure :-1 s=READY pgs=3225 cs=0 l=1 rev1=1 rx=0x7f3988009630 tx=0x7f398800b980).stop 2021-02-10T11:48:57.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:57.335 7f399efc7700 1 -- 172.21.2.112:0/4189645536 shutdown_connections 2021-02-10T11:48:57.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:57.335 7f399efc7700 1 --2- 172.21.2.112:0/4189645536 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f397c0186d0 0x7f397c01abd0 unknown :-1 s=CLOSED pgs=1454 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:57.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:57.335 7f399efc7700 1 --2- 172.21.2.112:0/4189645536 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f39980a9470 0x7f39980a7170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:57.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:57.335 7f399efc7700 1 --2- 172.21.2.112:0/4189645536 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f39980913a0 0x7f39980a8e50 unknown :-1 s=CLOSED pgs=3225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:57.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:57.335 7f399efc7700 1 --2- 172.21.2.112:0/4189645536 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f39981a4a10 0x7f3998187e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:57.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:57.335 7f399efc7700 1 -- 172.21.2.112:0/4189645536 >> 172.21.2.112:0/4189645536 conn(0x7f3998087360 msgr2=0x7f39980b5980 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:57.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:57.336 7f399efc7700 1 -- 172.21.2.112:0/4189645536 shutdown_connections 2021-02-10T11:48:57.341 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:57.336 7f399efc7700 1 -- 172.21.2.112:0/4189645536 wait complete. 2021-02-10T11:48:57.348 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2021-02-10T11:48:57.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:57.764 7f3087470700 1 Processor -- start 2021-02-10T11:48:57.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:57.764 7f3087470700 1 -- start start 2021-02-10T11:48:57.786 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:57.765 7f3087470700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f30800d1c80 0x7f30800cef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:57.786 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:57.765 7f3087470700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f30800cf590 0x7f30800cf990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:57.786 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:57.765 7f3087470700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30800fe1d0 con 0x7f30800d1880 2021-02-10T11:48:57.786 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:57.765 7f3087470700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30801b0cc0 con 0x7f30800d1c80 2021-02-10T11:48:57.786 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:57.765 7f3087470700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3080109a70 con 0x7f30800cf590 2021-02-10T11:48:57.786 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:57.765 7f3084a8b700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f30800d1c80 0x7f30800cef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:57.786 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:57.765 7f308528c700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f30800d1880 0x7f30801b04d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:32856/0 (socket says 172.21.2.112:32856) 2021-02-10T11:48:57.786 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:57.765 7f308528c700 1 -- 172.21.2.112:0/388379768 learned_addr learned my addr 172.21.2.112:0/388379768 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:48:57.786 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:57.765 7f3085a8d700 1 --2- 172.21.2.112:0/388379768 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f30800cf590 0x7f30800cf990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:57.786 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:57.765 7f3073fff700 1 -- 172.21.2.112:0/388379768 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2216832209 0 0) 0x7f307c000ce0 con 0x7f30800d1880 2021-02-10T11:48:57.787 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:57.765 7f3073fff700 1 -- 172.21.2.112:0/388379768 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f306c002fe0 con 0x7f30800d1880 2021-02-10T11:48:57.787 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:57.765 7f3085a8d700 1 -- 172.21.2.112:0/388379768 >> v2:172.21.2.112:3300/0 conn(0x7f30800d1c80 msgr2=0x7f30800cef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:57.787 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:57.765 7f3085a8d700 1 --2- 172.21.2.112:0/388379768 >> v2:172.21.2.112:3300/0 conn(0x7f30800d1c80 0x7f30800cef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:57.787 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:57.765 7f3085a8d700 1 -- 172.21.2.112:0/388379768 >> v1:172.21.2.112:6789/0 conn(0x7f30800d1880 legacy=0x7f30801b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:57.787 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:57.765 7f3085a8d700 1 -- 172.21.2.112:0/388379768 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3080088070 con 0x7f30800cf590 2021-02-10T11:48:57.787 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:57.765 7f3085a8d700 1 --2- 172.21.2.112:0/388379768 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f30800cf590 0x7f30800cf990 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7f3078006e50 tx=0x7f3078009df0).ready entity=mon.2 client_cookie=b497d6eee6e33485 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:57.787 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:57.765 7f3073fff700 1 -- 172.21.2.112:0/388379768 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f307800b510 con 0x7f30800cf590 2021-02-10T11:48:57.787 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:57.765 7f3073fff700 1 -- 172.21.2.112:0/388379768 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3078003030 con 0x7f30800cf590 2021-02-10T11:48:57.787 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:57.765 7f3073fff700 1 -- 172.21.2.112:0/388379768 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3078000f90 con 0x7f30800cf590 2021-02-10T11:48:57.787 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:57.766 7f3087470700 1 -- 172.21.2.112:0/388379768 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f30800cf590 msgr2=0x7f30800cf990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:57.787 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:57.766 7f3087470700 1 --2- 172.21.2.112:0/388379768 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f30800cf590 0x7f30800cf990 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7f3078006e50 tx=0x7f3078009df0).stop 2021-02-10T11:48:57.788 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:57.766 7f3087470700 1 -- 172.21.2.112:0/388379768 shutdown_connections 2021-02-10T11:48:57.788 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:57.766 7f3087470700 1 --2- 172.21.2.112:0/388379768 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f30800cf590 0x7f30800cf990 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:57.788 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:57.766 7f3087470700 1 --2- 172.21.2.112:0/388379768 >> v2:172.21.2.112:3300/0 conn(0x7f30800d1c80 0x7f30800cef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:57.788 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:57.766 7f3087470700 1 -- 172.21.2.112:0/388379768 >> 172.21.2.112:0/388379768 conn(0x7f30800ce430 msgr2=0x7f30800d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:57.788 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:57.766 7f3087470700 1 -- 172.21.2.112:0/388379768 shutdown_connections 2021-02-10T11:48:57.788 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:57.766 7f3087470700 1 -- 172.21.2.112:0/388379768 wait complete. 2021-02-10T11:48:57.790 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:57.781 7f3087470700 1 Processor -- start 2021-02-10T11:48:57.790 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:57.781 7f3087470700 1 -- start start 2021-02-10T11:48:57.790 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:57.782 7f3087470700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3080188400 0x7f3080188800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:57.790 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:57.782 7f3087470700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3080188e20 0x7f3080189220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:57.790 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:57.782 7f3087470700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f308009cbc0 0x7f308009cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:57.790 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:57.782 7f3087470700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3080187e80 con 0x7f3080188e20 2021-02-10T11:48:57.790 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:57.782 7f3087470700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30800b27c0 con 0x7f3080188400 2021-02-10T11:48:57.790 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:57.782 7f3087470700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30800b29a0 con 0x7f308009cbc0 2021-02-10T11:48:57.790 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:57.782 7f3084a8b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3080188e20 0x7f3080189220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:57.790 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:57.782 7f3084a8b700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3080188e20 0x7f3080189220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:33402/0 (socket says 172.21.2.112:33402) 2021-02-10T11:48:57.791 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:57.782 7f3084a8b700 1 -- 172.21.2.112:0/2625216325 learned_addr learned my addr 172.21.2.112:0/2625216325 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:48:57.791 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:57.782 7f308528c700 1 --2- 172.21.2.112:0/2625216325 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3080188400 0x7f3080188800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:57.791 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:57.782 7f3085a8d700 1 --2- 172.21.2.112:0/2625216325 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f308009cbc0 0x7f308009cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:57.791 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:57.782 7f3084a8b700 1 -- 172.21.2.112:0/2625216325 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f308009cbc0 msgr2=0x7f308009cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:57.791 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:57.782 7f3084a8b700 1 --2- 172.21.2.112:0/2625216325 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f308009cbc0 0x7f308009cfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:57.791 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:57.782 7f3084a8b700 1 -- 172.21.2.112:0/2625216325 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3080188400 msgr2=0x7f3080188800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:57.791 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:57.782 7f3084a8b700 1 --2- 172.21.2.112:0/2625216325 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3080188400 0x7f3080188800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:57.791 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:57.782 7f3084a8b700 1 -- 172.21.2.112:0/2625216325 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f308009d5e0 con 0x7f3080188e20 2021-02-10T11:48:57.791 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:57.782 7f3084a8b700 1 --2- 172.21.2.112:0/2625216325 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3080188e20 0x7f3080189220 secure :-1 s=READY pgs=3227 cs=0 l=1 rev1=1 rx=0x7f3074006f60 tx=0x7f307400b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:57.791 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:57.782 7f3071ffb700 1 -- 172.21.2.112:0/2625216325 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f30740009b0 con 0x7f3080188e20 2021-02-10T11:48:57.791 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:57.783 7f3071ffb700 1 -- 172.21.2.112:0/2625216325 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f307400bdf0 con 0x7f3080188e20 2021-02-10T11:48:57.791 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:57.783 7f3071ffb700 1 -- 172.21.2.112:0/2625216325 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3074002030 con 0x7f3080188e20 2021-02-10T11:48:57.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:57.783 7f3087470700 1 -- 172.21.2.112:0/2625216325 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f308009d5e0 con 0x7f3080188e20 2021-02-10T11:48:57.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:57.783 7f3087470700 1 -- 172.21.2.112:0/2625216325 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f308009d5e0 con 0x7f3080188e20 2021-02-10T11:48:57.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:57.783 7f3071ffb700 1 -- 172.21.2.112:0/2625216325 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3074006b00 con 0x7f3080188e20 2021-02-10T11:48:57.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:57.783 7f3071ffb700 1 --2- 172.21.2.112:0/2625216325 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f30600186c0 0x7f306001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:57.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:57.783 7f3071ffb700 1 -- 172.21.2.112:0/2625216325 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(191..191 src has 1..191) v4 ==== 5221+0+0 (secure 0 0 0) 0x7f3074032420 con 0x7f3080188e20 2021-02-10T11:48:57.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:57.783 7f308528c700 1 --2- 172.21.2.112:0/2625216325 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f30600186c0 0x7f306001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:57.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:57.784 7f308528c700 1 --2- 172.21.2.112:0/2625216325 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f30600186c0 0x7f306001abc0 crc :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:57.818 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:57.814 7f3087470700 1 -- 172.21.2.112:0/2625216325 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f308009db70 con 0x7f3080188e20 2021-02-10T11:48:57.821 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:57.816 7f3071ffb700 1 -- 172.21.2.112:0/2625216325 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3074006340 con 0x7f3080188e20 2021-02-10T11:48:57.950 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:57.946 7f3087470700 1 -- 172.21.2.112:0/2625216325 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f3080082a70 con 0x7f3080188e20 2021-02-10T11:48:58.197 DEBUG:teuthology.orchestra.run.gibba012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:48:58.200 DEBUG:teuthology.orchestra.run.gibba020:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:48:58.334 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:58.329 7f3071ffb700 1 -- 172.21.2.112:0/2625216325 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v192) v1 ==== 173+0+0 (secure 0 0 0) 0x7f3074006b00 con 0x7f3080188e20 2021-02-10T11:48:58.367 INFO:teuthology.orchestra.run.gibba012.stderr:pool 'cephfs_data' removed 2021-02-10T11:48:58.369 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:58.364 7f3087470700 1 -- 172.21.2.112:0/2625216325 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f30600186c0 msgr2=0x7f306001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:58.369 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:58.364 7f3087470700 1 --2- 172.21.2.112:0/2625216325 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f30600186c0 0x7f306001abc0 crc :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:58.369 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:58.364 7f3087470700 1 -- 172.21.2.112:0/2625216325 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3080188e20 msgr2=0x7f3080189220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:58.369 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:58.364 7f3087470700 1 --2- 172.21.2.112:0/2625216325 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3080188e20 0x7f3080189220 secure :-1 s=READY pgs=3227 cs=0 l=1 rev1=1 rx=0x7f3074006f60 tx=0x7f307400b970).stop 2021-02-10T11:48:58.370 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:58.365 7f3087470700 1 -- 172.21.2.112:0/2625216325 shutdown_connections 2021-02-10T11:48:58.370 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:58.365 7f3087470700 1 --2- 172.21.2.112:0/2625216325 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f30600186c0 0x7f306001abc0 unknown :-1 s=CLOSED pgs=1455 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:58.370 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:58.365 7f3087470700 1 --2- 172.21.2.112:0/2625216325 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f308009cbc0 0x7f308009cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:58.370 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:58.365 7f3087470700 1 --2- 172.21.2.112:0/2625216325 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3080188e20 0x7f3080189220 unknown :-1 s=CLOSED pgs=3227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:58.370 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:58.365 7f3087470700 1 --2- 172.21.2.112:0/2625216325 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3080188400 0x7f3080188800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:58.370 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:58.365 7f3087470700 1 -- 172.21.2.112:0/2625216325 >> 172.21.2.112:0/2625216325 conn(0x7f30800c2cd0 msgr2=0x7f30800b5950 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:58.370 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:58.365 7f3087470700 1 -- 172.21.2.112:0/2625216325 shutdown_connections 2021-02-10T11:48:58.370 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:58.365 7f3087470700 1 -- 172.21.2.112:0/2625216325 wait complete. 2021-02-10T11:48:58.377 INFO:tasks.ceph.mds.a:Restarting daemon 2021-02-10T11:48:58.378 INFO:tasks.ceph.mds.a:Stopping old one... 2021-02-10T11:48:58.378 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2021-02-10T11:48:58.378 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:48:58.378 INFO:tasks.ceph.mds.b:Restarting daemon 2021-02-10T11:48:58.378 INFO:tasks.ceph.mds.b:Stopping old one... 2021-02-10T11:48:58.378 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2021-02-10T11:48:58.378 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:48:58.384 INFO:tasks.ceph.mds.a:Stopped 2021-02-10T11:48:58.384 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2021-02-10T11:48:58.385 INFO:tasks.ceph.mds.b:Stopped 2021-02-10T11:48:58.385 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2021-02-10T11:48:58.426 INFO:tasks.ceph.mds.a:Started 2021-02-10T11:48:58.426 DEBUG:teuthology.parallel:result is None 2021-02-10T11:48:58.427 INFO:tasks.ceph.mds.b:Started 2021-02-10T11:48:58.427 DEBUG:teuthology.parallel:result is None 2021-02-10T11:48:58.427 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd blacklist clear 2021-02-10T11:48:58.833 INFO:tasks.ceph.mds.b.gibba020.stdout:starting mds.b at 2021-02-10T11:48:58.836 INFO:tasks.ceph.mds.a.gibba012.stdout:starting mds.a at 2021-02-10T11:48:58.858 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:58.837 7feb2debe700 1 Processor -- start 2021-02-10T11:48:58.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:58.837 7feb2debe700 1 -- start start 2021-02-10T11:48:58.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:58.837 7feb2debe700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7feb280d1870 0x7feb281b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:58.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:58.837 7feb2debe700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7feb280f0650 0x7feb280cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:58.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:58.837 7feb2debe700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb280f0b10 con 0x7feb281b0a10 2021-02-10T11:48:58.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:58.837 7feb2debe700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb280e5ed0 con 0x7feb280d1870 2021-02-10T11:48:58.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:58.837 7feb2debe700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb280d1c70 con 0x7feb280f0650 2021-02-10T11:48:58.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:58.837 7feb26ffd700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7feb281b0a10 0x7feb280cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:32882/0 (socket says 172.21.2.112:32882) 2021-02-10T11:48:58.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:58.837 7feb26ffd700 1 -- 172.21.2.112:0/705132737 learned_addr learned my addr 172.21.2.112:0/705132737 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:48:58.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:58.838 7feb277fe700 1 --2- 172.21.2.112:0/705132737 >> v2:172.21.2.120:3300/0 conn(0x7feb280d1870 0x7feb281b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:58.859 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:58.838 7feb27fff700 1 --2- 172.21.2.112:0/705132737 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7feb280f0650 0x7feb280cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:58.860 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:58.838 7feb267fc700 1 -- 172.21.2.112:0/705132737 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2471129329 0 0) 0x7feb18000ce0 con 0x7feb281b0a10 2021-02-10T11:48:58.860 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:58.838 7feb267fc700 1 -- 172.21.2.112:0/705132737 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb14002fe0 con 0x7feb281b0a10 2021-02-10T11:48:58.860 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:58.838 7feb277fe700 1 -- 172.21.2.112:0/705132737 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7feb280f0650 msgr2=0x7feb280cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:58.860 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:58.838 7feb277fe700 1 --2- 172.21.2.112:0/705132737 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7feb280f0650 0x7feb280cfb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:58.860 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:58.838 7feb277fe700 1 -- 172.21.2.112:0/705132737 >> v1:172.21.2.112:6789/0 conn(0x7feb281b0a10 legacy=0x7feb280cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:58.860 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:58.838 7feb277fe700 1 -- 172.21.2.112:0/705132737 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb280d1e90 con 0x7feb280d1870 2021-02-10T11:48:58.860 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:58.838 7feb27fff700 1 --2- 172.21.2.112:0/705132737 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7feb280f0650 0x7feb280cfb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:48:58.860 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:58.838 7feb277fe700 1 --2- 172.21.2.112:0/705132737 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7feb280d1870 0x7feb281b03f0 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 rx=0x7feb10006650 tx=0x7feb10009a80).ready entity=mon.1 client_cookie=e9fb175a9f8bd1cd server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:58.860 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:58.838 7feb267fc700 1 -- 172.21.2.112:0/705132737 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feb1000c0c0 con 0x7feb280d1870 2021-02-10T11:48:58.860 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:58.838 7feb267fc700 1 -- 172.21.2.112:0/705132737 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb1000c340 con 0x7feb280d1870 2021-02-10T11:48:58.860 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:58.838 7feb267fc700 1 -- 172.21.2.112:0/705132737 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feb1000c820 con 0x7feb280d1870 2021-02-10T11:48:58.860 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:58.838 7feb2debe700 1 -- 172.21.2.112:0/705132737 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7feb280d1870 msgr2=0x7feb281b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:58.860 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:58.838 7feb2debe700 1 --2- 172.21.2.112:0/705132737 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7feb280d1870 0x7feb281b03f0 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 rx=0x7feb10006650 tx=0x7feb10009a80).stop 2021-02-10T11:48:58.861 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:58.839 7feb2debe700 1 -- 172.21.2.112:0/705132737 shutdown_connections 2021-02-10T11:48:58.861 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:58.839 7feb2debe700 1 --2- 172.21.2.112:0/705132737 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7feb280f0650 0x7feb280cfb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:58.861 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:58.839 7feb2debe700 1 --2- 172.21.2.112:0/705132737 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7feb280d1870 0x7feb281b03f0 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:58.861 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:58.839 7feb2debe700 1 -- 172.21.2.112:0/705132737 >> 172.21.2.112:0/705132737 conn(0x7feb280ce450 msgr2=0x7feb280d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:58.861 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:58.839 7feb2debe700 1 -- 172.21.2.112:0/705132737 shutdown_connections 2021-02-10T11:48:58.861 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:58.839 7feb2debe700 1 -- 172.21.2.112:0/705132737 wait complete. 2021-02-10T11:48:58.861 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:58.854 7feb2debe700 1 Processor -- start 2021-02-10T11:48:58.861 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:58.854 7feb2debe700 1 -- start start 2021-02-10T11:48:58.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:58.854 7feb2debe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7feb2819ffa0 0x7feb280eda10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:58.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:58.854 7feb2debe700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7feb281a03a0 0x7feb280ee030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:58.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:58.854 7feb2debe700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7feb280ee650 0x7feb280ebe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:58.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:58.854 7feb2debe700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb280ed490 con 0x7feb2819ffa0 2021-02-10T11:48:58.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:58.854 7feb2debe700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb280eea50 con 0x7feb280ee650 2021-02-10T11:48:58.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:58.854 7feb2debe700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb280eec30 con 0x7feb281a03a0 2021-02-10T11:48:58.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:58.854 7feb277fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7feb2819ffa0 0x7feb280eda10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:58.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:58.854 7feb277fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7feb2819ffa0 0x7feb280eda10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:33424/0 (socket says 172.21.2.112:33424) 2021-02-10T11:48:58.863 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:58.855 7feb277fe700 1 -- 172.21.2.112:0/376980677 learned_addr learned my addr 172.21.2.112:0/376980677 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:48:58.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:58.855 7feb277fe700 1 -- 172.21.2.112:0/376980677 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7feb281a03a0 msgr2=0x7feb280ee030 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:48:58.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:58.855 7feb277fe700 1 --2- 172.21.2.112:0/376980677 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7feb281a03a0 0x7feb280ee030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:58.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:58.855 7feb277fe700 1 -- 172.21.2.112:0/376980677 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7feb280ee650 msgr2=0x7feb280ebe80 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:48:58.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:58.855 7feb277fe700 1 --2- 172.21.2.112:0/376980677 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7feb280ee650 0x7feb280ebe80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:58.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:58.855 7feb277fe700 1 -- 172.21.2.112:0/376980677 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb280ec4a0 con 0x7feb2819ffa0 2021-02-10T11:48:58.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:58.855 7feb277fe700 1 --2- 172.21.2.112:0/376980677 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7feb2819ffa0 0x7feb280eda10 secure :-1 s=READY pgs=3231 cs=0 l=1 rev1=1 rx=0x7feb10006650 tx=0x7feb1000b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:58.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:58.855 7feb1ffff700 1 -- 172.21.2.112:0/376980677 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feb1000bd50 con 0x7feb2819ffa0 2021-02-10T11:48:58.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:58.855 7feb1ffff700 1 -- 172.21.2.112:0/376980677 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb10006030 con 0x7feb2819ffa0 2021-02-10T11:48:58.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:58.855 7feb1ffff700 1 -- 172.21.2.112:0/376980677 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feb10005560 con 0x7feb2819ffa0 2021-02-10T11:48:58.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:58.855 7feb2debe700 1 -- 172.21.2.112:0/376980677 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb280ec4a0 con 0x7feb2819ffa0 2021-02-10T11:48:58.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:58.855 7feb2debe700 1 -- 172.21.2.112:0/376980677 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb280ec4a0 con 0x7feb2819ffa0 2021-02-10T11:48:58.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:58.855 7feb1ffff700 1 -- 172.21.2.112:0/376980677 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7feb10009320 con 0x7feb2819ffa0 2021-02-10T11:48:58.864 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:58.855 7feb1ffff700 1 --2- 172.21.2.112:0/376980677 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7feb0c0186c0 0x7feb0c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:58.865 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:58.856 7feb1ffff700 1 -- 172.21.2.112:0/376980677 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(192..192 src has 1..192) v4 ==== 4839+0+0 (secure 0 0 0) 0x7feb10032280 con 0x7feb2819ffa0 2021-02-10T11:48:58.865 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:58.856 7feb26ffd700 1 --2- 172.21.2.112:0/376980677 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7feb0c0186c0 0x7feb0c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:58.865 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:58.856 7feb26ffd700 1 --2- 172.21.2.112:0/376980677 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7feb0c0186c0 0x7feb0c01abc0 crc :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:58.890 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:58.886 7feb2debe700 1 -- 172.21.2.112:0/376980677 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb280ebc50 con 0x7feb2819ffa0 2021-02-10T11:48:58.892 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:58.889 7feb1ffff700 1 -- 172.21.2.112:0/376980677 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7feb1002ef10 con 0x7feb2819ffa0 2021-02-10T11:48:59.033 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:59.029 7feb2debe700 1 -- 172.21.2.112:0/376980677 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7feb280ebc50 con 0x7feb2819ffa0 2021-02-10T11:48:59.374 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:59.369 7feb1ffff700 1 -- 172.21.2.112:0/376980677 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v193) v1 ==== 97+0+0 (secure 0 0 0) 0x7feb1002ef10 con 0x7feb2819ffa0 2021-02-10T11:48:59.399 INFO:teuthology.orchestra.run.gibba012.stderr: removed all blacklist entries 2021-02-10T11:48:59.401 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:59.396 7feb2debe700 1 -- 172.21.2.112:0/376980677 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7feb0c0186c0 msgr2=0x7feb0c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:59.401 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:59.396 7feb2debe700 1 --2- 172.21.2.112:0/376980677 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7feb0c0186c0 0x7feb0c01abc0 crc :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:59.401 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:59.396 7feb2debe700 1 -- 172.21.2.112:0/376980677 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7feb2819ffa0 msgr2=0x7feb280eda10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:59.401 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:59.396 7feb2debe700 1 --2- 172.21.2.112:0/376980677 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7feb2819ffa0 0x7feb280eda10 secure :-1 s=READY pgs=3231 cs=0 l=1 rev1=1 rx=0x7feb10006650 tx=0x7feb1000b960).stop 2021-02-10T11:48:59.401 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:59.396 7feb2debe700 1 -- 172.21.2.112:0/376980677 shutdown_connections 2021-02-10T11:48:59.401 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:59.396 7feb2debe700 1 --2- 172.21.2.112:0/376980677 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7feb0c0186c0 0x7feb0c01abc0 unknown :-1 s=CLOSED pgs=1456 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:59.401 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:59.396 7feb2debe700 1 --2- 172.21.2.112:0/376980677 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7feb280ee650 0x7feb280ebe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:59.401 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:59.396 7feb2debe700 1 --2- 172.21.2.112:0/376980677 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7feb281a03a0 0x7feb280ee030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:59.401 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:59.396 7feb2debe700 1 --2- 172.21.2.112:0/376980677 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7feb2819ffa0 0x7feb280eda10 unknown :-1 s=CLOSED pgs=3231 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:59.401 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:59.396 7feb2debe700 1 -- 172.21.2.112:0/376980677 >> 172.21.2.112:0/376980677 conn(0x7feb28094d90 msgr2=0x7feb2808a420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:59.402 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:59.397 7feb2debe700 1 -- 172.21.2.112:0/376980677 shutdown_connections 2021-02-10T11:48:59.402 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:59.397 7feb2debe700 1 -- 172.21.2.112:0/376980677 wait complete. 2021-02-10T11:48:59.409 INFO:tasks.cephfs.cephfs_test_case:['0', '1', '2', '3'] 2021-02-10T11:48:59.410 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth ls --format=json-pretty 2021-02-10T11:48:59.839 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:59.818 7f2035f75700 1 Processor -- start 2021-02-10T11:48:59.839 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:59.818 7f2035f75700 1 -- start start 2021-02-10T11:48:59.839 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:59.819 7f2035f75700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f20300f05f0 0x7f20300f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:59.840 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:59.819 7f2035f75700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20301b0c70 con 0x7f20301b0470 2021-02-10T11:48:59.840 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:59.819 7f2035f75700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20300f0b50 con 0x7f20301b0870 2021-02-10T11:48:59.840 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:59.819 7f2035f75700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20300e5ea0 con 0x7f20300f05f0 2021-02-10T11:48:59.840 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:59.819 7f202ffff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f20300f05f0 0x7f20300f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:59.840 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:59.819 7f202ffff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f20300f05f0 0x7f20300f20e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:33436/0 (socket says 172.21.2.112:33436) 2021-02-10T11:48:59.840 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:59.819 7f202ffff700 1 -- 172.21.2.112:0/4030566740 learned_addr learned my addr 172.21.2.112:0/4030566740 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:48:59.840 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:59.819 7f202ffff700 1 -- 172.21.2.112:0/4030566740 >> v1:172.21.2.120:6789/0 conn(0x7f20301b0870 legacy=0x7f20300efe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:59.840 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:59.819 7f202ffff700 1 -- 172.21.2.112:0/4030566740 >> v1:172.21.2.112:6789/0 conn(0x7f20301b0470 legacy=0x7f20300ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:59.840 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:59.819 7f202ffff700 1 -- 172.21.2.112:0/4030566740 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20300e52e0 con 0x7f20300f05f0 2021-02-10T11:48:59.840 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:59.819 7f202ffff700 1 --2- 172.21.2.112:0/4030566740 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f20300f05f0 0x7f20300f20e0 secure :-1 s=READY pgs=3233 cs=0 l=1 rev1=1 rx=0x7f2028006f00 tx=0x7f2028009f10).ready entity=mon.0 client_cookie=c802bc3ab71fb60f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:59.840 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:59.819 7f202e7fc700 1 -- 172.21.2.112:0/4030566740 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f202800b3b0 con 0x7f20300f05f0 2021-02-10T11:48:59.841 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:59.819 7f202e7fc700 1 -- 172.21.2.112:0/4030566740 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f202800b730 con 0x7f20300f05f0 2021-02-10T11:48:59.841 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:59.820 7f202e7fc700 1 -- 172.21.2.112:0/4030566740 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f202800bc10 con 0x7f20300f05f0 2021-02-10T11:48:59.841 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:59.820 7f2035f75700 1 -- 172.21.2.112:0/4030566740 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f20300f05f0 msgr2=0x7f20300f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:59.841 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:59.820 7f2035f75700 1 --2- 172.21.2.112:0/4030566740 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f20300f05f0 0x7f20300f20e0 secure :-1 s=READY pgs=3233 cs=0 l=1 rev1=1 rx=0x7f2028006f00 tx=0x7f2028009f10).stop 2021-02-10T11:48:59.841 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:59.820 7f2035f75700 1 -- 172.21.2.112:0/4030566740 shutdown_connections 2021-02-10T11:48:59.841 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:59.820 7f2035f75700 1 --2- 172.21.2.112:0/4030566740 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f20300f05f0 0x7f20300f20e0 unknown :-1 s=CLOSED pgs=3233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:59.841 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:59.820 7f2035f75700 1 -- 172.21.2.112:0/4030566740 >> 172.21.2.112:0/4030566740 conn(0x7f20300ce3e0 msgr2=0x7f20300e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:59.841 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:59.820 7f2035f75700 1 -- 172.21.2.112:0/4030566740 shutdown_connections 2021-02-10T11:48:59.841 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:59.820 7f2035f75700 1 -- 172.21.2.112:0/4030566740 wait complete. 2021-02-10T11:48:59.841 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:59.835 7f2035f75700 1 Processor -- start 2021-02-10T11:48:59.841 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:59.835 7f2035f75700 1 -- start start 2021-02-10T11:48:59.844 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:59.835 7f2035f75700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2030092e30 0x7f20301c6320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:59.844 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:59.835 7f2035f75700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f20300964a0 0x7f20300968a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:59.844 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:59.835 7f2035f75700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2030096ec0 0x7f2030084f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:59.844 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:59.835 7f2035f75700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20300932b0 con 0x7f2030092e30 2021-02-10T11:48:59.844 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:59.835 7f2035f75700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20301b5f70 con 0x7f20300964a0 2021-02-10T11:48:59.844 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:59.835 7f2035f75700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20301b6150 con 0x7f2030096ec0 2021-02-10T11:48:59.844 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:59.836 7f202f7fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2030092e30 0x7f20301c6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:59.844 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:59.836 7f202f7fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2030092e30 0x7f20301c6320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:33438/0 (socket says 172.21.2.112:33438) 2021-02-10T11:48:59.844 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:59.836 7f202f7fe700 1 -- 172.21.2.112:0/1873045739 learned_addr learned my addr 172.21.2.112:0/1873045739 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:48:59.844 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:59.836 7f202effd700 1 --2- 172.21.2.112:0/1873045739 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f20300964a0 0x7f20300968a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:59.844 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:59.836 7f202ffff700 1 --2- 172.21.2.112:0/1873045739 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2030096ec0 0x7f2030084f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:59.845 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:59.836 7f202f7fe700 1 -- 172.21.2.112:0/1873045739 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2030096ec0 msgr2=0x7f2030084f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:59.845 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:59.836 7f202f7fe700 1 --2- 172.21.2.112:0/1873045739 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2030096ec0 0x7f2030084f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:59.845 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:59.836 7f202f7fe700 1 -- 172.21.2.112:0/1873045739 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f20300964a0 msgr2=0x7f20300968a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:59.845 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:59.836 7f202f7fe700 1 --2- 172.21.2.112:0/1873045739 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f20300964a0 0x7f20300968a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:59.845 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:59.836 7f202f7fe700 1 -- 172.21.2.112:0/1873045739 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20301c6970 con 0x7f2030092e30 2021-02-10T11:48:59.845 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:59.836 7f202effd700 1 --2- 172.21.2.112:0/1873045739 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f20300964a0 0x7f20300968a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:48:59.845 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:59.836 7f202f7fe700 1 --2- 172.21.2.112:0/1873045739 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2030092e30 0x7f20301c6320 secure :-1 s=READY pgs=3234 cs=0 l=1 rev1=1 rx=0x7f2020006ed0 tx=0x7f202000b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:59.845 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:59.836 7f2017fff700 1 -- 172.21.2.112:0/1873045739 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2020006240 con 0x7f2030092e30 2021-02-10T11:48:59.845 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:59.837 7f2017fff700 1 -- 172.21.2.112:0/1873045739 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20200054b0 con 0x7f2030092e30 2021-02-10T11:48:59.845 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:59.837 7f2035f75700 1 -- 172.21.2.112:0/1873045739 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20301c6970 con 0x7f2030092e30 2021-02-10T11:48:59.845 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:59.837 7f2017fff700 1 -- 172.21.2.112:0/1873045739 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2020005990 con 0x7f2030092e30 2021-02-10T11:48:59.845 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:59.837 7f2035f75700 1 -- 172.21.2.112:0/1873045739 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2030085580 con 0x7f2030092e30 2021-02-10T11:48:59.845 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:59.837 7f2017fff700 1 -- 172.21.2.112:0/1873045739 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2020006030 con 0x7f2030092e30 2021-02-10T11:48:59.845 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:59.837 7f2017fff700 1 --2- 172.21.2.112:0/1873045739 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2010018730 0x7f201001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:59.846 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:59.837 7f2017fff700 1 -- 172.21.2.112:0/1873045739 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f2020032440 con 0x7f2030092e30 2021-02-10T11:48:59.846 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:59.838 7f202effd700 1 --2- 172.21.2.112:0/1873045739 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2010018730 0x7f201001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:59.846 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:59.838 7f202effd700 1 --2- 172.21.2.112:0/1873045739 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2010018730 0x7f201001ac10 crc :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:59.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:59.868 7f2035f75700 1 -- 172.21.2.112:0/1873045739 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f203007ed50 con 0x7f2030092e30 2021-02-10T11:48:59.875 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:48:59.872 7f2017fff700 1 -- 172.21.2.112:0/1873045739 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2020006030 con 0x7f2030092e30 2021-02-10T11:49:00.007 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:00.003 7f2035f75700 1 -- 172.21.2.112:0/1873045739 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f20300ce3e0 con 0x7f2030092e30 2021-02-10T11:49:00.007 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:00.004 7f2017fff700 1 -- 172.21.2.112:0/1873045739 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v119) v1 ==== 80+0+4513 (secure 0 0 0) 0x7f202000bd50 con 0x7f2030092e30 2021-02-10T11:49:00.008 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:49:00.008 INFO:teuthology.orchestra.run.gibba012.stdout:{ 2021-02-10T11:49:00.008 INFO:teuthology.orchestra.run.gibba012.stdout: "auth_dump": [ 2021-02-10T11:49:00.008 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:49:00.008 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "mds.a", 2021-02-10T11:49:00.008 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCKvyNgPkV4AxAAEmVNaghdccP6lWd8hvQnxw==", 2021-02-10T11:49:00.009 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:49:00.009 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:49:00.009 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:49:00.009 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:49:00.009 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:49:00.009 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:49:00.009 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:49:00.009 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:49:00.009 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "mds.b", 2021-02-10T11:49:00.009 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCKvyNguxONLhAAHIpry+R89MAwdhFJyEhVCw==", 2021-02-10T11:49:00.009 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:49:00.009 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:49:00.009 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:49:00.010 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:49:00.010 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:49:00.010 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:49:00.010 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:49:00.010 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:49:00.010 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.0", 2021-02-10T11:49:00.010 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCSvyNgXcKPLhAAP/Bj2F+06DCtPCzQDNq+cQ==", 2021-02-10T11:49:00.010 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:49:00.010 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:49:00.010 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:49:00.010 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:49:00.010 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:49:00.011 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:49:00.011 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:49:00.011 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.1", 2021-02-10T11:49:00.011 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCVvyNghfbcLhAA66PrRwR2M7o6VTF5UhU/Fg==", 2021-02-10T11:49:00.011 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:49:00.011 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:49:00.011 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:49:00.011 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:49:00.011 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:49:00.011 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:49:00.011 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:49:00.011 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.2", 2021-02-10T11:49:00.011 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCYvyNgwiPqLhAA66BVkRsgaUlGpAsQ15Nf8Q==", 2021-02-10T11:49:00.012 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:49:00.012 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:49:00.012 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:49:00.012 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:49:00.012 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:49:00.012 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:49:00.012 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:49:00.012 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.3", 2021-02-10T11:49:00.012 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCbvyNgqBZrLxAAERPe/1L3SXAEzI3d9ReHHg==", 2021-02-10T11:49:00.012 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:49:00.012 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:49:00.012 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:49:00.012 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:49:00.013 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:49:00.013 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:49:00.013 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:49:00.013 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.4", 2021-02-10T11:49:00.013 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCkvyNgDQsxIhAAHmVjVYnx20N4SDmKqvbcuA==", 2021-02-10T11:49:00.013 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:49:00.013 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:49:00.013 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:49:00.013 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:49:00.013 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:49:00.013 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:49:00.013 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:49:00.013 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.5", 2021-02-10T11:49:00.014 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCnvyNg8ITFIhAADd4aKybAV7btoGPx3kCg3A==", 2021-02-10T11:49:00.014 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:49:00.014 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:49:00.014 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:49:00.014 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:49:00.014 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:49:00.014 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:49:00.014 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:49:00.014 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.6", 2021-02-10T11:49:00.014 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCqvyNgxOhDIxAAzXixEa/nlDc1xXrg0aeLwg==", 2021-02-10T11:49:00.014 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:49:00.014 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:49:00.015 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:49:00.015 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:49:00.015 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:49:00.015 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:49:00.015 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:49:00.015 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "osd.7", 2021-02-10T11:49:00.015 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCtvyNgxSqGIxAACxJQlwP7DpiVVCZHA3zn1A==", 2021-02-10T11:49:00.015 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:49:00.015 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:49:00.015 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:49:00.015 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:49:00.015 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:49:00.015 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:49:00.015 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:49:00.016 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.0", 2021-02-10T11:49:00.016 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCLvyNgDkNiHRAAygU8Za4PBusTqCgwAYWTiQ==", 2021-02-10T11:49:00.016 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:49:00.016 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:49:00.016 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow r", 2021-02-10T11:49:00.016 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:49:00.016 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:49:00.016 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:49:00.016 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:49:00.016 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.1", 2021-02-10T11:49:00.016 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCLvyNgxmLqMxAA1MlSDJ9LH9u4f2sL7mm8/Q==", 2021-02-10T11:49:00.016 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:49:00.016 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:49:00.017 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow r", 2021-02-10T11:49:00.017 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:49:00.017 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:49:00.017 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:49:00.017 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:49:00.017 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.2", 2021-02-10T11:49:00.017 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCMvyNgp8veDhAAniSVHYPtwwTog/pjbaRQmw==", 2021-02-10T11:49:00.017 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:49:00.017 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:49:00.017 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow r", 2021-02-10T11:49:00.017 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:49:00.017 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:49:00.017 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:49:00.017 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:49:00.018 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.3", 2021-02-10T11:49:00.018 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCMvyNgQEVoJRAApHcWdm1sldIxu+heyD4HdQ==", 2021-02-10T11:49:00.018 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:49:00.018 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow", 2021-02-10T11:49:00.018 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow r", 2021-02-10T11:49:00.018 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:49:00.018 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:49:00.018 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:49:00.018 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:49:00.018 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "client.admin", 2021-02-10T11:49:00.018 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCGvyNg7jjJBRAAtot+ypUZOiTo1GbY+oVhVw==", 2021-02-10T11:49:00.018 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:49:00.019 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow *", 2021-02-10T11:49:00.019 INFO:teuthology.orchestra.run.gibba012.stdout: "mgr": "allow *", 2021-02-10T11:49:00.019 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow *", 2021-02-10T11:49:00.019 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:49:00.019 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:49:00.019 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:49:00.019 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:49:00.019 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "mgr.x", 2021-02-10T11:49:00.019 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCJvyNgTPVwKBAAjduiBE8+RSXBv82tdPecWg==", 2021-02-10T11:49:00.019 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:49:00.019 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow *", 2021-02-10T11:49:00.019 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow profile mgr", 2021-02-10T11:49:00.019 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:49:00.020 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:49:00.020 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:49:00.020 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:49:00.020 INFO:teuthology.orchestra.run.gibba012.stdout: "entity": "mgr.y", 2021-02-10T11:49:00.020 INFO:teuthology.orchestra.run.gibba012.stdout: "key": "AQCJvyNgQ1tTDhAASwBssDy+Rz31ujw8fNlWow==", 2021-02-10T11:49:00.020 INFO:teuthology.orchestra.run.gibba012.stdout: "caps": { 2021-02-10T11:49:00.020 INFO:teuthology.orchestra.run.gibba012.stdout: "mds": "allow *", 2021-02-10T11:49:00.020 INFO:teuthology.orchestra.run.gibba012.stdout: "mon": "allow profile mgr", 2021-02-10T11:49:00.020 INFO:teuthology.orchestra.run.gibba012.stdout: "osd": "allow *" 2021-02-10T11:49:00.020 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:49:00.020 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:49:00.020 INFO:teuthology.orchestra.run.gibba012.stdout: ] 2021-02-10T11:49:00.020 INFO:teuthology.orchestra.run.gibba012.stdout:} 2021-02-10T11:49:00.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:00.006 7f2035f75700 1 -- 172.21.2.112:0/1873045739 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2010018730 msgr2=0x7f201001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:00.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:00.006 7f2035f75700 1 --2- 172.21.2.112:0/1873045739 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2010018730 0x7f201001ac10 crc :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:00.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:00.006 7f2035f75700 1 -- 172.21.2.112:0/1873045739 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2030092e30 msgr2=0x7f20301c6320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:00.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:00.006 7f2035f75700 1 --2- 172.21.2.112:0/1873045739 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2030092e30 0x7f20301c6320 secure :-1 s=READY pgs=3234 cs=0 l=1 rev1=1 rx=0x7f2020006ed0 tx=0x7f202000b960).stop 2021-02-10T11:49:00.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:00.006 7f2035f75700 1 -- 172.21.2.112:0/1873045739 shutdown_connections 2021-02-10T11:49:00.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:00.006 7f2035f75700 1 --2- 172.21.2.112:0/1873045739 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f2010018730 0x7f201001ac10 unknown :-1 s=CLOSED pgs=1457 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:00.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:00.006 7f2035f75700 1 --2- 172.21.2.112:0/1873045739 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f20300964a0 0x7f20300968a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:00.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:00.006 7f2035f75700 1 --2- 172.21.2.112:0/1873045739 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f2030096ec0 0x7f2030084f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:00.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:00.006 7f2035f75700 1 --2- 172.21.2.112:0/1873045739 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f2030092e30 0x7f20301c6320 unknown :-1 s=CLOSED pgs=3234 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:00.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:00.006 7f2035f75700 1 -- 172.21.2.112:0/1873045739 >> 172.21.2.112:0/1873045739 conn(0x7f20300c3bd0 msgr2=0x7f2030089210 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:00.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:00.006 7f2035f75700 1 -- 172.21.2.112:0/1873045739 shutdown_connections 2021-02-10T11:49:00.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:00.006 7f2035f75700 1 -- 172.21.2.112:0/1873045739 wait complete. 2021-02-10T11:49:00.023 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:49:00.453 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:00.433 7ff4da773700 1 Processor -- start 2021-02-10T11:49:00.453 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:00.433 7ff4da773700 1 -- start start 2021-02-10T11:49:00.453 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:00.433 7ff4da773700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7ff4d40d1c70 0x7ff4d40f20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:00.453 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:00.433 7ff4da773700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4d40f0b10 con 0x7ff4d40d1870 2021-02-10T11:49:00.454 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:00.433 7ff4da773700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4d41b0be0 con 0x7ff4d40d0340 2021-02-10T11:49:00.454 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:00.433 7ff4da773700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4d40e5ed0 con 0x7ff4d40d1c70 2021-02-10T11:49:00.454 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:00.434 7ff4d8d90700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7ff4d40d1c70 0x7ff4d40f20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:00.454 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:00.434 7ff4d3fff700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7ff4d40d1870 0x7ff4d41b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:32908/0 (socket says 172.21.2.112:32908) 2021-02-10T11:49:00.454 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:00.434 7ff4d8d90700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7ff4d40d1c70 0x7ff4d40f20a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:33448/0 (socket says 172.21.2.112:33448) 2021-02-10T11:49:00.454 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:00.434 7ff4d8d90700 1 -- 172.21.2.112:0/2863074128 learned_addr learned my addr 172.21.2.112:0/2863074128 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:49:00.454 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:00.434 7ff4d8d90700 1 -- 172.21.2.112:0/2863074128 >> v1:172.21.2.120:6789/0 conn(0x7ff4d40d0340 legacy=0x7ff4d40ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:00.454 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:00.434 7ff4d8d90700 1 -- 172.21.2.112:0/2863074128 >> v1:172.21.2.112:6789/0 conn(0x7ff4d40d1870 legacy=0x7ff4d41b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:00.454 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:00.434 7ff4d2ffd700 1 -- 172.21.2.112:0/2863074128 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1847973196 0 0) 0x7ff4c8000c60 con 0x7ff4d40d0340 2021-02-10T11:49:00.454 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:00.434 7ff4d8d90700 1 -- 172.21.2.112:0/2863074128 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4d40efe30 con 0x7ff4d40d1c70 2021-02-10T11:49:00.455 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:00.434 7ff4d8d90700 1 --2- 172.21.2.112:0/2863074128 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff4d40d1c70 0x7ff4d40f20a0 secure :-1 s=READY pgs=3236 cs=0 l=1 rev1=1 rx=0x7ff4cc006f00 tx=0x7ff4cc009e90).ready entity=mon.0 client_cookie=9f8b7101f9251fc6 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:00.455 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:00.435 7ff4d2ffd700 1 -- 172.21.2.112:0/2863074128 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff4cc00b300 con 0x7ff4d40d1c70 2021-02-10T11:49:00.455 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:00.435 7ff4d2ffd700 1 -- 172.21.2.112:0/2863074128 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4cc00b640 con 0x7ff4d40d1c70 2021-02-10T11:49:00.455 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:00.435 7ff4d2ffd700 1 -- 172.21.2.112:0/2863074128 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff4cc00ba90 con 0x7ff4d40d1c70 2021-02-10T11:49:00.455 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:00.435 7ff4da773700 1 -- 172.21.2.112:0/2863074128 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff4d40d1c70 msgr2=0x7ff4d40f20a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:00.455 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:00.435 7ff4da773700 1 --2- 172.21.2.112:0/2863074128 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff4d40d1c70 0x7ff4d40f20a0 secure :-1 s=READY pgs=3236 cs=0 l=1 rev1=1 rx=0x7ff4cc006f00 tx=0x7ff4cc009e90).stop 2021-02-10T11:49:00.455 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:00.435 7ff4da773700 1 -- 172.21.2.112:0/2863074128 shutdown_connections 2021-02-10T11:49:00.455 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:00.435 7ff4da773700 1 --2- 172.21.2.112:0/2863074128 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff4d40d1c70 0x7ff4d40f20a0 unknown :-1 s=CLOSED pgs=3236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:00.455 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:00.435 7ff4da773700 1 -- 172.21.2.112:0/2863074128 >> 172.21.2.112:0/2863074128 conn(0x7ff4d40ce450 msgr2=0x7ff4d40d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:00.455 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:00.435 7ff4da773700 1 -- 172.21.2.112:0/2863074128 shutdown_connections 2021-02-10T11:49:00.455 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:00.435 7ff4da773700 1 -- 172.21.2.112:0/2863074128 wait complete. 2021-02-10T11:49:00.457 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:00.450 7ff4da773700 1 Processor -- start 2021-02-10T11:49:00.457 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:00.450 7ff4da773700 1 -- start start 2021-02-10T11:49:00.457 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:00.450 7ff4da773700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff4d41947b0 0x7ff4d40a69a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:00.457 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:00.450 7ff4da773700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff4d40a6fc0 0x7ff4d41a15d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:00.458 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:00.450 7ff4da773700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff4d41a1bf0 0x7ff4d41a1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:00.458 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:00.450 7ff4da773700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4d40a73c0 con 0x7ff4d41947b0 2021-02-10T11:49:00.458 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:00.450 7ff4da773700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4d418f170 con 0x7ff4d40a6fc0 2021-02-10T11:49:00.458 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:00.450 7ff4da773700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4d4194bb0 con 0x7ff4d41a1bf0 2021-02-10T11:49:00.458 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:00.451 7ff4d3fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff4d41947b0 0x7ff4d40a69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:00.458 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:00.451 7ff4d3fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff4d41947b0 0x7ff4d40a69a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:33454/0 (socket says 172.21.2.112:33454) 2021-02-10T11:49:00.458 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:00.451 7ff4d3fff700 1 -- 172.21.2.112:0/1383076667 learned_addr learned my addr 172.21.2.112:0/1383076667 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:49:00.458 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:00.451 7ff4d37fe700 1 --2- 172.21.2.112:0/1383076667 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff4d40a6fc0 0x7ff4d41a15d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:00.458 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:00.451 7ff4d8d90700 1 --2- 172.21.2.112:0/1383076667 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff4d41a1bf0 0x7ff4d41a1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:00.458 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:00.451 7ff4d3fff700 1 -- 172.21.2.112:0/1383076667 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff4d41a1bf0 msgr2=0x7ff4d41a1ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:00.458 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:00.451 7ff4d3fff700 1 --2- 172.21.2.112:0/1383076667 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff4d41a1bf0 0x7ff4d41a1ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:00.458 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:00.451 7ff4d3fff700 1 -- 172.21.2.112:0/1383076667 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff4d40a6fc0 msgr2=0x7ff4d41a15d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:00.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:00.451 7ff4d3fff700 1 --2- 172.21.2.112:0/1383076667 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff4d40a6fc0 0x7ff4d41a15d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:00.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:00.451 7ff4d3fff700 1 -- 172.21.2.112:0/1383076667 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4d418ec70 con 0x7ff4d41947b0 2021-02-10T11:49:00.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:00.451 7ff4d3fff700 1 --2- 172.21.2.112:0/1383076667 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff4d41947b0 0x7ff4d40a69a0 secure :-1 s=READY pgs=3237 cs=0 l=1 rev1=1 rx=0x7ff4c400b530 tx=0x7ff4c400b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:00.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:00.451 7ff4d0ff9700 1 -- 172.21.2.112:0/1383076667 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff4c4006020 con 0x7ff4d41947b0 2021-02-10T11:49:00.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:00.452 7ff4d0ff9700 1 -- 172.21.2.112:0/1383076667 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4c40062a0 con 0x7ff4d41947b0 2021-02-10T11:49:00.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:00.452 7ff4da773700 1 -- 172.21.2.112:0/1383076667 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4d418ec70 con 0x7ff4d41947b0 2021-02-10T11:49:00.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:00.452 7ff4da773700 1 -- 172.21.2.112:0/1383076667 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4d418ec70 con 0x7ff4d41947b0 2021-02-10T11:49:00.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:00.452 7ff4d0ff9700 1 -- 172.21.2.112:0/1383076667 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff4c4004620 con 0x7ff4d41947b0 2021-02-10T11:49:00.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:00.452 7ff4d0ff9700 1 -- 172.21.2.112:0/1383076667 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff4c400a000 con 0x7ff4d41947b0 2021-02-10T11:49:00.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:00.452 7ff4d0ff9700 1 --2- 172.21.2.112:0/1383076667 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff4b40186c0 0x7ff4b401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:00.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:00.452 7ff4d0ff9700 1 -- 172.21.2.112:0/1383076667 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 4753+0+0 (secure 0 0 0) 0x7ff4c4032430 con 0x7ff4d41947b0 2021-02-10T11:49:00.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:00.453 7ff4d37fe700 1 --2- 172.21.2.112:0/1383076667 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff4b40186c0 0x7ff4b401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:00.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:00.453 7ff4d37fe700 1 --2- 172.21.2.112:0/1383076667 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff4b40186c0 0x7ff4b401abc0 crc :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:00.487 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:00.484 7ff4da773700 1 -- 172.21.2.112:0/1383076667 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4d40ee0e0 con 0x7ff4d41947b0 2021-02-10T11:49:00.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:00.486 7ff4d0ff9700 1 -- 172.21.2.112:0/1383076667 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff4c4006b90 con 0x7ff4d41947b0 2021-02-10T11:49:00.614 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:00.610 7ff4da773700 1 -- 172.21.2.112:0/1383076667 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff4d40ce450 con 0x7ff4d41947b0 2021-02-10T11:49:00.614 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:00.611 7ff4d0ff9700 1 -- 172.21.2.112:0/1383076667 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v193) v1 ==== 74+0+11154 (secure 0 0 0) 0x7ff4c400a000 con 0x7ff4d41947b0 2021-02-10T11:49:00.615 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:49:00.615 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":193,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:48:59.368565","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":41,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:49:00.616 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:00.612 7ff4da773700 1 -- 172.21.2.112:0/1383076667 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff4b40186c0 msgr2=0x7ff4b401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:00.616 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:00.612 7ff4da773700 1 --2- 172.21.2.112:0/1383076667 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff4b40186c0 0x7ff4b401abc0 crc :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:00.616 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:00.612 7ff4da773700 1 -- 172.21.2.112:0/1383076667 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff4d41947b0 msgr2=0x7ff4d40a69a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:00.616 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:00.612 7ff4da773700 1 --2- 172.21.2.112:0/1383076667 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff4d41947b0 0x7ff4d40a69a0 secure :-1 s=READY pgs=3237 cs=0 l=1 rev1=1 rx=0x7ff4c400b530 tx=0x7ff4c400b8d0).stop 2021-02-10T11:49:00.617 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:00.612 7ff4da773700 1 -- 172.21.2.112:0/1383076667 shutdown_connections 2021-02-10T11:49:00.617 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:00.612 7ff4da773700 1 --2- 172.21.2.112:0/1383076667 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff4b40186c0 0x7ff4b401abc0 unknown :-1 s=CLOSED pgs=1458 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:00.617 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:00.612 7ff4da773700 1 --2- 172.21.2.112:0/1383076667 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff4d40a6fc0 0x7ff4d41a15d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:00.617 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:00.612 7ff4da773700 1 --2- 172.21.2.112:0/1383076667 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff4d41a1bf0 0x7ff4d41a1ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:00.617 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:00.612 7ff4da773700 1 --2- 172.21.2.112:0/1383076667 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff4d41947b0 0x7ff4d40a69a0 unknown :-1 s=CLOSED pgs=3237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:00.617 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:00.612 7ff4da773700 1 -- 172.21.2.112:0/1383076667 >> 172.21.2.112:0/1383076667 conn(0x7ff4d41a38c0 msgr2=0x7ff4d41a2f00 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:00.617 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:00.613 7ff4da773700 1 -- 172.21.2.112:0/1383076667 shutdown_connections 2021-02-10T11:49:00.617 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:00.613 7ff4da773700 1 -- 172.21.2.112:0/1383076667 wait complete. 2021-02-10T11:49:00.625 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:13:57.508758', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2021-02-10T11:49:00.625 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:49:01.059 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.038 7fe807024700 1 Processor -- start 2021-02-10T11:49:01.059 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.038 7fe807024700 1 -- start start 2021-02-10T11:49:01.059 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.039 7fe807024700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fe8000d1870 0x7fe8001b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:01.059 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.039 7fe807024700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe8001b0ae0 0x7fe800108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:01.059 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.039 7fe807024700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fe8000d1c70 0x7fe800108f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:01.059 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.039 7fe807024700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8000fe1c0 con 0x7fe8000d1870 2021-02-10T11:49:01.059 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.039 7fe807024700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe800109a60 con 0x7fe8000d1c70 2021-02-10T11:49:01.059 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.039 7fe807024700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe800088070 con 0x7fe8001b0ae0 2021-02-10T11:49:01.059 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.039 7fe804e40700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fe8000d1870 0x7fe8001b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:01.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.039 7fe805641700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fe8000d1c70 0x7fe800108f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:01.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.039 7fe805641700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fe8000d1c70 0x7fe800108f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3300/0 says I am v2:172.21.2.112:35738/0 (socket says 172.21.2.112:35738) 2021-02-10T11:49:01.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.039 7fe804e40700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fe8000d1870 0x7fe8001b04c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:33458/0 (socket says 172.21.2.112:33458) 2021-02-10T11:49:01.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.039 7fe805641700 1 -- 172.21.2.112:0/3667571127 learned_addr learned my addr 172.21.2.112:0/3667571127 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:49:01.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.039 7fe7fffff700 1 --2- 172.21.2.112:0/3667571127 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe8001b0ae0 0x7fe800108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:01.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.040 7fe804e40700 1 -- 172.21.2.112:0/3667571127 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe8001b0ae0 msgr2=0x7fe800108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:49:01.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.040 7fe804e40700 1 --2- 172.21.2.112:0/3667571127 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe8001b0ae0 0x7fe800108940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:01.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.040 7fe804e40700 1 -- 172.21.2.112:0/3667571127 >> v2:172.21.2.120:3300/0 conn(0x7fe8000d1c70 msgr2=0x7fe800108f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:49:01.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.040 7fe804e40700 1 --2- 172.21.2.112:0/3667571127 >> v2:172.21.2.120:3300/0 conn(0x7fe8000d1c70 0x7fe800108f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:01.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.040 7fe804e40700 1 -- 172.21.2.112:0/3667571127 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe800109580 con 0x7fe8000d1870 2021-02-10T11:49:01.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.040 7fe805641700 1 --2- 172.21.2.112:0/3667571127 >> v2:172.21.2.120:3300/0 conn(0x7fe8000d1c70 0x7fe800108f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:49:01.060 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.040 7fe804e40700 1 --2- 172.21.2.112:0/3667571127 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe8000d1870 0x7fe8001b04c0 secure :-1 s=READY pgs=3238 cs=0 l=1 rev1=1 rx=0x7fe7e8006650 tx=0x7fe7e8009a80).ready entity=mon.0 client_cookie=3e67097b43a3b382 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:01.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.040 7fe7ff7fe700 1 -- 172.21.2.112:0/3667571127 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe7e80009b0 con 0x7fe8000d1870 2021-02-10T11:49:01.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.040 7fe7ff7fe700 1 -- 172.21.2.112:0/3667571127 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7e800c0c0 con 0x7fe8000d1870 2021-02-10T11:49:01.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.040 7fe7ff7fe700 1 -- 172.21.2.112:0/3667571127 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe7e800c5e0 con 0x7fe8000d1870 2021-02-10T11:49:01.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.040 7fe807024700 1 -- 172.21.2.112:0/3667571127 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe8000d1870 msgr2=0x7fe8001b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:01.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.040 7fe807024700 1 --2- 172.21.2.112:0/3667571127 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe8000d1870 0x7fe8001b04c0 secure :-1 s=READY pgs=3238 cs=0 l=1 rev1=1 rx=0x7fe7e8006650 tx=0x7fe7e8009a80).stop 2021-02-10T11:49:01.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.041 7fe807024700 1 -- 172.21.2.112:0/3667571127 shutdown_connections 2021-02-10T11:49:01.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.041 7fe807024700 1 --2- 172.21.2.112:0/3667571127 >> v2:172.21.2.120:3300/0 conn(0x7fe8000d1c70 0x7fe800108f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:01.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.041 7fe807024700 1 --2- 172.21.2.112:0/3667571127 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe8001b0ae0 0x7fe800108940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:01.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.041 7fe807024700 1 --2- 172.21.2.112:0/3667571127 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe8000d1870 0x7fe8001b04c0 unknown :-1 s=CLOSED pgs=3238 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:01.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.041 7fe807024700 1 -- 172.21.2.112:0/3667571127 >> 172.21.2.112:0/3667571127 conn(0x7fe8000ce420 msgr2=0x7fe8000d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:01.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.041 7fe807024700 1 -- 172.21.2.112:0/3667571127 shutdown_connections 2021-02-10T11:49:01.061 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.041 7fe807024700 1 -- 172.21.2.112:0/3667571127 wait complete. 2021-02-10T11:49:01.063 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.056 7fe807024700 1 Processor -- start 2021-02-10T11:49:01.063 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.056 7fe807024700 1 -- start start 2021-02-10T11:49:01.063 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.056 7fe807024700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe800085bc0 0x7fe8000863e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:01.063 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.056 7fe807024700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe8000954c0 0x7fe800096300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:01.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.056 7fe807024700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe8000958c0 0x7fe800096920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:01.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.056 7fe807024700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe800095d80 con 0x7fe8000954c0 2021-02-10T11:49:01.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.056 7fe807024700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8000bfa60 con 0x7fe8000958c0 2021-02-10T11:49:01.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.056 7fe807024700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe800085fc0 con 0x7fe800085bc0 2021-02-10T11:49:01.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.056 7fe7fffff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe8000954c0 0x7fe800096300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:01.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.056 7fe7fffff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe8000954c0 0x7fe800096300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:33462/0 (socket says 172.21.2.112:33462) 2021-02-10T11:49:01.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.056 7fe7fffff700 1 -- 172.21.2.112:0/1719702495 learned_addr learned my addr 172.21.2.112:0/1719702495 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:49:01.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.057 7fe7fffff700 1 -- 172.21.2.112:0/1719702495 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe800085bc0 msgr2=0x7fe8000863e0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:49:01.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.057 7fe7fffff700 1 --2- 172.21.2.112:0/1719702495 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe800085bc0 0x7fe8000863e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:01.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.057 7fe7fffff700 1 -- 172.21.2.112:0/1719702495 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe8000958c0 msgr2=0x7fe800096920 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:49:01.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.057 7fe7fffff700 1 --2- 172.21.2.112:0/1719702495 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe8000958c0 0x7fe800096920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:01.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.057 7fe7fffff700 1 -- 172.21.2.112:0/1719702495 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe800086c10 con 0x7fe8000954c0 2021-02-10T11:49:01.064 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.057 7fe7fffff700 1 --2- 172.21.2.112:0/1719702495 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe8000954c0 0x7fe800096300 secure :-1 s=READY pgs=3239 cs=0 l=1 rev1=1 rx=0x7fe7f00099e0 tx=0x7fe7f000bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:01.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.057 7fe7fd7fa700 1 -- 172.21.2.112:0/1719702495 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe7f000c2f0 con 0x7fe8000954c0 2021-02-10T11:49:01.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.057 7fe7fd7fa700 1 -- 172.21.2.112:0/1719702495 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7f000c570 con 0x7fe8000954c0 2021-02-10T11:49:01.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.057 7fe7fd7fa700 1 -- 172.21.2.112:0/1719702495 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe7f000ca90 con 0x7fe8000954c0 2021-02-10T11:49:01.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.057 7fe807024700 1 -- 172.21.2.112:0/1719702495 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe800086c10 con 0x7fe8000954c0 2021-02-10T11:49:01.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.057 7fe807024700 1 -- 172.21.2.112:0/1719702495 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe800097230 con 0x7fe8000954c0 2021-02-10T11:49:01.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.057 7fe7fd7fa700 1 -- 172.21.2.112:0/1719702495 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe7f000a020 con 0x7fe8000954c0 2021-02-10T11:49:01.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.058 7fe7fd7fa700 1 --2- 172.21.2.112:0/1719702495 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe7e40186c0 0x7fe7e401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:01.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.058 7fe7fd7fa700 1 -- 172.21.2.112:0/1719702495 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 4753+0+0 (secure 0 0 0) 0x7fe7f0032600 con 0x7fe8000954c0 2021-02-10T11:49:01.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.058 7fe804e40700 1 --2- 172.21.2.112:0/1719702495 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe7e40186c0 0x7fe7e401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:01.065 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.059 7fe804e40700 1 --2- 172.21.2.112:0/1719702495 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe7e40186c0 0x7fe7e401abc0 crc :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:01.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.089 7fe807024700 1 -- 172.21.2.112:0/1719702495 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8000fe1c0 con 0x7fe8000954c0 2021-02-10T11:49:01.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.092 7fe7fd7fa700 1 -- 172.21.2.112:0/1719702495 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe7f0000d30 con 0x7fe8000954c0 2021-02-10T11:49:01.222 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.219 7fe807024700 1 -- 172.21.2.112:0/1719702495 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7fe8000fe1c0 con 0x7fe8000954c0 2021-02-10T11:49:01.223 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.219 7fe7fd7fa700 1 -- 172.21.2.112:0/1719702495 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v193) v1 ==== 92+0+10 (secure 0 0 0) 0x7fe7f0009fe0 con 0x7fe8000954c0 2021-02-10T11:49:01.224 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:49:01.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.221 7fe807024700 1 -- 172.21.2.112:0/1719702495 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe7e40186c0 msgr2=0x7fe7e401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:01.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.221 7fe807024700 1 --2- 172.21.2.112:0/1719702495 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe7e40186c0 0x7fe7e401abc0 crc :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:01.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.221 7fe807024700 1 -- 172.21.2.112:0/1719702495 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe8000954c0 msgr2=0x7fe800096300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:01.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.221 7fe807024700 1 --2- 172.21.2.112:0/1719702495 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe8000954c0 0x7fe800096300 secure :-1 s=READY pgs=3239 cs=0 l=1 rev1=1 rx=0x7fe7f00099e0 tx=0x7fe7f000bd40).stop 2021-02-10T11:49:01.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.221 7fe807024700 1 -- 172.21.2.112:0/1719702495 shutdown_connections 2021-02-10T11:49:01.225 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.221 7fe807024700 1 --2- 172.21.2.112:0/1719702495 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fe7e40186c0 0x7fe7e401abc0 unknown :-1 s=CLOSED pgs=1459 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:01.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.221 7fe807024700 1 --2- 172.21.2.112:0/1719702495 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fe8000958c0 0x7fe800096920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:01.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.221 7fe807024700 1 --2- 172.21.2.112:0/1719702495 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fe8000954c0 0x7fe800096300 unknown :-1 s=CLOSED pgs=3239 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:01.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.221 7fe807024700 1 --2- 172.21.2.112:0/1719702495 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fe800085bc0 0x7fe8000863e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:01.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.221 7fe807024700 1 -- 172.21.2.112:0/1719702495 >> 172.21.2.112:0/1719702495 conn(0x7fe8000bef00 msgr2=0x7fe80009d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:01.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.222 7fe807024700 1 -- 172.21.2.112:0/1719702495 shutdown_connections 2021-02-10T11:49:01.226 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.222 7fe807024700 1 -- 172.21.2.112:0/1719702495 wait complete. 2021-02-10T11:49:01.234 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --format=json-pretty osd lspools 2021-02-10T11:49:01.665 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.644 7fac73faa700 1 Processor -- start 2021-02-10T11:49:01.665 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.644 7fac73faa700 1 -- start start 2021-02-10T11:49:01.665 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.644 7fac73faa700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fac6c0d0340 0x7fac6c0ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:01.665 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.644 7fac73faa700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fac6c0d1c70 0x7fac6c0efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:01.665 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.644 7fac73faa700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac6c0f0b10 con 0x7fac6c0d1870 2021-02-10T11:49:01.665 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.644 7fac73faa700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac6c0e5ed0 con 0x7fac6c0d1c70 2021-02-10T11:49:01.665 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.644 7fac73faa700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac6c1b0be0 con 0x7fac6c0d0340 2021-02-10T11:49:01.665 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.645 7fac71dc6700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fac6c0d1870 0x7fac6c1b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:32932/0 (socket says 172.21.2.112:32932) 2021-02-10T11:49:01.666 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.645 7fac71dc6700 1 -- 172.21.2.112:0/63882352 learned_addr learned my addr 172.21.2.112:0/63882352 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:49:01.666 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.645 7fac70dc4700 1 -- 172.21.2.112:0/63882352 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1760378543 0 0) 0x7fac680010b0 con 0x7fac6c0d1870 2021-02-10T11:49:01.666 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.645 7fac70dc4700 1 -- 172.21.2.112:0/63882352 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac58002fe0 con 0x7fac6c0d1870 2021-02-10T11:49:01.666 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.645 7fac725c7700 1 --2- 172.21.2.112:0/63882352 >> v2:172.21.2.120:3300/0 conn(0x7fac6c0d1c70 0x7fac6c0efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:01.666 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.645 7fac715c5700 1 --2- 172.21.2.112:0/63882352 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fac6c0d0340 0x7fac6c0ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:01.666 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.645 7fac70dc4700 1 -- 172.21.2.112:0/63882352 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1927630655 0 0) 0x7fac68001010 con 0x7fac6c0d1870 2021-02-10T11:49:01.666 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.645 7fac70dc4700 1 -- 172.21.2.112:0/63882352 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fac6c0d0340 msgr2=0x7fac6c0ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:49:01.666 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.645 7fac70dc4700 1 --2- 172.21.2.112:0/63882352 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fac6c0d0340 0x7fac6c0ef640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:01.666 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.645 7fac70dc4700 1 -- 172.21.2.112:0/63882352 >> v2:172.21.2.120:3300/0 conn(0x7fac6c0d1c70 msgr2=0x7fac6c0efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:49:01.666 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.645 7fac70dc4700 1 --2- 172.21.2.112:0/63882352 >> v2:172.21.2.120:3300/0 conn(0x7fac6c0d1c70 0x7fac6c0efc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:01.666 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.645 7fac70dc4700 1 -- 172.21.2.112:0/63882352 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac6c0f20a0 con 0x7fac6c0d1870 2021-02-10T11:49:01.666 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.645 7fac70dc4700 1 -- 172.21.2.112:0/63882352 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7fac68001530 con 0x7fac6c0d1870 2021-02-10T11:49:01.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.645 7fac70dc4700 1 -- 172.21.2.112:0/63882352 >> v1:172.21.2.112:6789/0 conn(0x7fac6c0d1870 legacy=0x7fac6c1b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:01.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.645 7fac70dc4700 1 --2- 172.21.2.112:0/63882352 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fac58003a70 0x7fac58005ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:01.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.645 7fac70dc4700 1 --2- 172.21.2.112:0/63882352 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fac58006740 0x7fac58008c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:01.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.645 7fac70dc4700 1 --2- 172.21.2.112:0/63882352 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fac58009500 0x7fac5800ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:01.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.645 7fac70dc4700 1 -- 172.21.2.112:0/63882352 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac580502e0 con 0x7fac58006740 2021-02-10T11:49:01.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.645 7fac70dc4700 1 -- 172.21.2.112:0/63882352 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac58050490 con 0x7fac58009500 2021-02-10T11:49:01.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.645 7fac70dc4700 1 -- 172.21.2.112:0/63882352 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac580506b0 con 0x7fac58003a70 2021-02-10T11:49:01.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.645 7fac715c5700 1 --2- 172.21.2.112:0/63882352 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fac58006740 0x7fac58008c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:01.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.646 7fac725c7700 1 --2- 172.21.2.112:0/63882352 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fac58009500 0x7fac5800ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:01.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.646 7fac715c5700 1 -- 172.21.2.112:0/63882352 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fac58003a70 msgr2=0x7fac58005ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:49:01.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.646 7fac715c5700 1 --2- 172.21.2.112:0/63882352 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fac58003a70 0x7fac58005ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:01.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.646 7fac715c5700 1 -- 172.21.2.112:0/63882352 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fac58009500 msgr2=0x7fac5800ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:49:01.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.646 7fac715c5700 1 --2- 172.21.2.112:0/63882352 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fac58009500 0x7fac5800ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:01.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.646 7fac715c5700 1 -- 172.21.2.112:0/63882352 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac58050950 con 0x7fac58006740 2021-02-10T11:49:01.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.646 7fac725c7700 1 --2- 172.21.2.112:0/63882352 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fac58009500 0x7fac5800ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:49:01.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.646 7fac715c5700 1 --2- 172.21.2.112:0/63882352 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fac58006740 0x7fac58008c60 secure :-1 s=READY pgs=3241 cs=0 l=1 rev1=1 rx=0x7fac5c006ad0 tx=0x7fac5c009b40).ready entity=mon.0 client_cookie=83ce25f19703df0d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:01.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.646 7fac70dc4700 1 -- 172.21.2.112:0/63882352 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fac5c00c0c0 con 0x7fac58006740 2021-02-10T11:49:01.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.646 7fac70dc4700 1 -- 172.21.2.112:0/63882352 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac5c00c300 con 0x7fac58006740 2021-02-10T11:49:01.669 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.646 7fac70dc4700 1 -- 172.21.2.112:0/63882352 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fac5c00c7e0 con 0x7fac58006740 2021-02-10T11:49:01.669 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.646 7fac73faa700 1 -- 172.21.2.112:0/63882352 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fac58006740 msgr2=0x7fac58008c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:01.669 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.646 7fac73faa700 1 --2- 172.21.2.112:0/63882352 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fac58006740 0x7fac58008c60 secure :-1 s=READY pgs=3241 cs=0 l=1 rev1=1 rx=0x7fac5c006ad0 tx=0x7fac5c009b40).stop 2021-02-10T11:49:01.669 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.646 7fac725c7700 1 -- 172.21.2.112:0/63882352 reap_dead start 2021-02-10T11:49:01.669 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.647 7fac73faa700 1 -- 172.21.2.112:0/63882352 shutdown_connections 2021-02-10T11:49:01.670 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.647 7fac73faa700 1 -- 172.21.2.112:0/63882352 >> 172.21.2.112:0/63882352 conn(0x7fac6c0ce450 msgr2=0x7fac6c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:01.670 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.647 7fac73faa700 1 -- 172.21.2.112:0/63882352 shutdown_connections 2021-02-10T11:49:01.670 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.647 7fac73faa700 1 -- 172.21.2.112:0/63882352 wait complete. 2021-02-10T11:49:01.670 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.661 7fac73faa700 1 Processor -- start 2021-02-10T11:49:01.670 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.661 7fac73faa700 1 -- start start 2021-02-10T11:49:01.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.662 7fac73faa700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fac6c195650 0x7fac6c0b5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:01.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.662 7fac73faa700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fac6c0b5b80 0x7fac6c1a2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:01.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.662 7fac73faa700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fac6c0acfa0 0x7fac6c1a3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:01.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.662 7fac73faa700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac6c0b6010 con 0x7fac6c0b5b80 2021-02-10T11:49:01.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.662 7fac73faa700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac6c1a3a60 con 0x7fac6c195650 2021-02-10T11:49:01.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.662 7fac73faa700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac6c1a3c40 con 0x7fac6c0acfa0 2021-02-10T11:49:01.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.662 7fac715c5700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fac6c0b5b80 0x7fac6c1a2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:01.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.662 7fac715c5700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fac6c0b5b80 0x7fac6c1a2e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:33482/0 (socket says 172.21.2.112:33482) 2021-02-10T11:49:01.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.662 7fac715c5700 1 -- 172.21.2.112:0/880320561 learned_addr learned my addr 172.21.2.112:0/880320561 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:49:01.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.662 7fac715c5700 1 -- 172.21.2.112:0/880320561 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fac6c0acfa0 msgr2=0x7fac6c1a3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:01.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.662 7fac715c5700 1 --2- 172.21.2.112:0/880320561 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fac6c0acfa0 0x7fac6c1a3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:01.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.662 7fac715c5700 1 -- 172.21.2.112:0/880320561 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fac6c195650 msgr2=0x7fac6c0b5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:01.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.662 7fac715c5700 1 --2- 172.21.2.112:0/880320561 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fac6c195650 0x7fac6c0b5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:01.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.662 7fac715c5700 1 -- 172.21.2.112:0/880320561 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac6c0bcae0 con 0x7fac6c0b5b80 2021-02-10T11:49:01.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.662 7fac715c5700 1 --2- 172.21.2.112:0/880320561 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fac6c0b5b80 0x7fac6c1a2e20 secure :-1 s=READY pgs=3242 cs=0 l=1 rev1=1 rx=0x7fac5c00b580 tx=0x7fac5c00b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:01.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.663 7fac627fc700 1 -- 172.21.2.112:0/880320561 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fac5c00bd10 con 0x7fac6c0b5b80 2021-02-10T11:49:01.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.663 7fac627fc700 1 -- 172.21.2.112:0/880320561 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac5c000c90 con 0x7fac6c0b5b80 2021-02-10T11:49:01.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.663 7fac627fc700 1 -- 172.21.2.112:0/880320561 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fac5c0022b0 con 0x7fac6c0b5b80 2021-02-10T11:49:01.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.663 7fac73faa700 1 -- 172.21.2.112:0/880320561 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac6c0bcae0 con 0x7fac6c0b5b80 2021-02-10T11:49:01.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.663 7fac73faa700 1 -- 172.21.2.112:0/880320561 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac6c0bcae0 con 0x7fac6c0b5b80 2021-02-10T11:49:01.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.663 7fac627fc700 1 -- 172.21.2.112:0/880320561 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fac5c0052e0 con 0x7fac6c0b5b80 2021-02-10T11:49:01.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.663 7fac627fc700 1 --2- 172.21.2.112:0/880320561 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fac540186c0 0x7fac5401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:01.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.663 7fac627fc700 1 -- 172.21.2.112:0/880320561 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 4753+0+0 (secure 0 0 0) 0x7fac5c032240 con 0x7fac6c0b5b80 2021-02-10T11:49:01.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.664 7fac71dc6700 1 --2- 172.21.2.112:0/880320561 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fac540186c0 0x7fac5401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:01.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.664 7fac71dc6700 1 --2- 172.21.2.112:0/880320561 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fac540186c0 0x7fac5401abc0 crc :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:01.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.694 7fac73faa700 1 -- 172.21.2.112:0/880320561 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac6c0b6010 con 0x7fac6c0b5b80 2021-02-10T11:49:01.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.697 7fac627fc700 1 -- 172.21.2.112:0/880320561 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fac5c006d60 con 0x7fac6c0b5b80 2021-02-10T11:49:01.825 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.821 7fac73faa700 1 -- 172.21.2.112:0/880320561 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fac6c0b6010 con 0x7fac6c0b5b80 2021-02-10T11:49:01.825 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.822 7fac627fc700 1 -- 172.21.2.112:0/880320561 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v193) v1 ==== 84+0+64 (secure 0 0 0) 0x7fac5c00bd10 con 0x7fac6c0b5b80 2021-02-10T11:49:01.826 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:49:01.826 INFO:teuthology.orchestra.run.gibba012.stdout:[ 2021-02-10T11:49:01.826 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:49:01.826 INFO:teuthology.orchestra.run.gibba012.stdout: "poolnum": 1, 2021-02-10T11:49:01.826 INFO:teuthology.orchestra.run.gibba012.stdout: "poolname": "rbd" 2021-02-10T11:49:01.826 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:49:01.826 INFO:teuthology.orchestra.run.gibba012.stdout:] 2021-02-10T11:49:01.827 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.824 7fac73faa700 1 -- 172.21.2.112:0/880320561 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fac540186c0 msgr2=0x7fac5401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:01.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.824 7fac73faa700 1 --2- 172.21.2.112:0/880320561 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fac540186c0 0x7fac5401abc0 crc :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:01.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.824 7fac73faa700 1 -- 172.21.2.112:0/880320561 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fac6c0b5b80 msgr2=0x7fac6c1a2e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:01.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.824 7fac73faa700 1 --2- 172.21.2.112:0/880320561 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fac6c0b5b80 0x7fac6c1a2e20 secure :-1 s=READY pgs=3242 cs=0 l=1 rev1=1 rx=0x7fac5c00b580 tx=0x7fac5c00b920).stop 2021-02-10T11:49:01.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.824 7fac73faa700 1 -- 172.21.2.112:0/880320561 shutdown_connections 2021-02-10T11:49:01.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.824 7fac73faa700 1 --2- 172.21.2.112:0/880320561 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fac540186c0 0x7fac5401abc0 unknown :-1 s=CLOSED pgs=1460 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:01.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.824 7fac73faa700 1 --2- 172.21.2.112:0/880320561 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fac6c0acfa0 0x7fac6c1a3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:01.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.824 7fac73faa700 1 --2- 172.21.2.112:0/880320561 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fac6c0b5b80 0x7fac6c1a2e20 unknown :-1 s=CLOSED pgs=3242 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:01.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.824 7fac73faa700 1 --2- 172.21.2.112:0/880320561 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fac6c195650 0x7fac6c0b5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:01.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.824 7fac73faa700 1 -- 172.21.2.112:0/880320561 >> 172.21.2.112:0/880320561 conn(0x7fac6c0b3040 msgr2=0x7fac6c0c3250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:01.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.824 7fac73faa700 1 -- 172.21.2.112:0/880320561 shutdown_connections 2021-02-10T11:49:01.828 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:01.824 7fac73faa700 1 -- 172.21.2.112:0/880320561 wait complete. 2021-02-10T11:49:01.836 INFO:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2021-02-10T11:49:01.836 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_metadata 8 2021-02-10T11:49:02.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:02.249 7f984080d700 1 Processor -- start 2021-02-10T11:49:02.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:02.249 7f984080d700 1 -- start start 2021-02-10T11:49:02.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:02.250 7f984080d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f983c0d1870 0x7f983c1b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:02.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:02.250 7f984080d700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f983c109a60 con 0x7f983c1b0ae0 2021-02-10T11:49:02.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:02.250 7f984080d700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f983c088070 con 0x7f983c0cff70 2021-02-10T11:49:02.270 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:02.250 7f984080d700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f983c0e5ef0 con 0x7f983c0d1870 2021-02-10T11:49:02.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:02.250 7f9839e1b700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f983c1b0ae0 0x7f983c108940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:32950/0 (socket says 172.21.2.112:32950) 2021-02-10T11:49:02.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:02.250 7f9839e1b700 1 -- 172.21.2.112:0/2376327127 learned_addr learned my addr 172.21.2.112:0/2376327127 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:49:02.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:02.250 7f983a61c700 1 --2- 172.21.2.112:0/2376327127 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f983c0d1870 0x7f983c1b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:02.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:02.250 7f983961a700 1 -- 172.21.2.112:0/2376327127 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2031715558 0 0) 0x7f98300010b0 con 0x7f983c1b0ae0 2021-02-10T11:49:02.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:02.250 7f983961a700 1 -- 172.21.2.112:0/2376327127 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9824002fe0 con 0x7f983c1b0ae0 2021-02-10T11:49:02.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:02.250 7f983961a700 1 -- 172.21.2.112:0/2376327127 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 708802855 0 0) 0x7f9834001400 con 0x7f983c0cff70 2021-02-10T11:49:02.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:02.250 7f983a61c700 1 -- 172.21.2.112:0/2376327127 >> v1:172.21.2.120:6789/0 conn(0x7f983c0cff70 legacy=0x7f983c109130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:02.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:02.250 7f983a61c700 1 -- 172.21.2.112:0/2376327127 >> v1:172.21.2.112:6789/0 conn(0x7f983c1b0ae0 legacy=0x7f983c108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:02.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:02.250 7f983a61c700 1 -- 172.21.2.112:0/2376327127 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f983c0d1c70 con 0x7f983c0d1870 2021-02-10T11:49:02.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:02.251 7f983a61c700 1 --2- 172.21.2.112:0/2376327127 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f983c0d1870 0x7f983c1b04c0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7f982c006650 tx=0x7f982c009a80).ready entity=mon.2 client_cookie=4aaeb83efecad26a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:02.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:02.251 7f983961a700 1 -- 172.21.2.112:0/2376327127 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f982c00c0c0 con 0x7f983c0d1870 2021-02-10T11:49:02.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:02.251 7f983961a700 1 -- 172.21.2.112:0/2376327127 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f982c00c340 con 0x7f983c0d1870 2021-02-10T11:49:02.271 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:02.251 7f983961a700 1 -- 172.21.2.112:0/2376327127 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f982c00c860 con 0x7f983c0d1870 2021-02-10T11:49:02.272 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:02.251 7f984080d700 1 -- 172.21.2.112:0/2376327127 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f983c0d1870 msgr2=0x7f983c1b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:02.272 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:02.251 7f984080d700 1 --2- 172.21.2.112:0/2376327127 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f983c0d1870 0x7f983c1b04c0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7f982c006650 tx=0x7f982c009a80).stop 2021-02-10T11:49:02.272 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:02.252 7f984080d700 1 -- 172.21.2.112:0/2376327127 shutdown_connections 2021-02-10T11:49:02.272 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:02.252 7f984080d700 1 --2- 172.21.2.112:0/2376327127 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f983c0d1870 0x7f983c1b04c0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:02.272 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:02.252 7f984080d700 1 -- 172.21.2.112:0/2376327127 >> 172.21.2.112:0/2376327127 conn(0x7f983c0ce420 msgr2=0x7f983c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:02.272 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:02.252 7f984080d700 1 -- 172.21.2.112:0/2376327127 shutdown_connections 2021-02-10T11:49:02.272 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:02.252 7f984080d700 1 -- 172.21.2.112:0/2376327127 wait complete. 2021-02-10T11:49:02.272 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:02.266 7f984080d700 1 Processor -- start 2021-02-10T11:49:02.272 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:02.266 7f984080d700 1 -- start start 2021-02-10T11:49:02.273 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:02.267 7f984080d700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f983c0872c0 0x7f983c1a7580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:02.273 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:02.267 7f984080d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f983c1a7ba0 0x7f983c0a6860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:02.274 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:02.267 7f984080d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f983c1a7fa0 0x7f983c0a6e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:02.274 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:02.267 7f984080d700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f983c086e20 con 0x7f983c1a7fa0 2021-02-10T11:49:02.274 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:02.267 7f984080d700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f983c190c60 con 0x7f983c0872c0 2021-02-10T11:49:02.274 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:02.267 7f984080d700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f983c0a74a0 con 0x7f983c1a7ba0 2021-02-10T11:49:02.274 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:02.267 7f983ae1d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f983c1a7fa0 0x7f983c0a6e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:02.274 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:02.267 7f983ae1d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f983c1a7fa0 0x7f983c0a6e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:33492/0 (socket says 172.21.2.112:33492) 2021-02-10T11:49:02.274 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:02.267 7f983ae1d700 1 -- 172.21.2.112:0/2466160317 learned_addr learned my addr 172.21.2.112:0/2466160317 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:49:02.274 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:02.267 7f983ae1d700 1 -- 172.21.2.112:0/2466160317 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f983c1a7ba0 msgr2=0x7f983c0a6860 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:49:02.274 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:02.267 7f983ae1d700 1 --2- 172.21.2.112:0/2466160317 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f983c1a7ba0 0x7f983c0a6860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:02.274 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:02.267 7f983ae1d700 1 -- 172.21.2.112:0/2466160317 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f983c0872c0 msgr2=0x7f983c1a7580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:49:02.274 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:02.267 7f983ae1d700 1 --2- 172.21.2.112:0/2466160317 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f983c0872c0 0x7f983c1a7580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:02.274 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:02.267 7f983ae1d700 1 -- 172.21.2.112:0/2466160317 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f983c0a76c0 con 0x7f983c1a7fa0 2021-02-10T11:49:02.275 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:02.267 7f983ae1d700 1 --2- 172.21.2.112:0/2466160317 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f983c1a7fa0 0x7f983c0a6e80 secure :-1 s=READY pgs=3244 cs=0 l=1 rev1=1 rx=0x7f9834009940 tx=0x7f983400bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:02.275 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:02.267 7f982affd700 1 -- 172.21.2.112:0/2466160317 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9834000f40 con 0x7f983c1a7fa0 2021-02-10T11:49:02.275 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:02.267 7f982affd700 1 -- 172.21.2.112:0/2466160317 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f983400c2f0 con 0x7f983c1a7fa0 2021-02-10T11:49:02.275 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:02.267 7f982affd700 1 -- 172.21.2.112:0/2466160317 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f983400c810 con 0x7f983c1a7fa0 2021-02-10T11:49:02.275 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:02.267 7f984080d700 1 -- 172.21.2.112:0/2466160317 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f983c0a76c0 con 0x7f983c1a7fa0 2021-02-10T11:49:02.275 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:02.267 7f984080d700 1 -- 172.21.2.112:0/2466160317 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f983c0a76c0 con 0x7f983c1a7fa0 2021-02-10T11:49:02.275 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:02.268 7f982affd700 1 -- 172.21.2.112:0/2466160317 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9834003080 con 0x7f983c1a7fa0 2021-02-10T11:49:02.275 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:02.268 7f982affd700 1 --2- 172.21.2.112:0/2466160317 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f98200186c0 0x7f982001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:02.275 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:02.268 7f982affd700 1 -- 172.21.2.112:0/2466160317 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f98340325e0 con 0x7f983c1a7fa0 2021-02-10T11:49:02.276 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:02.269 7f983a61c700 1 --2- 172.21.2.112:0/2466160317 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f98200186c0 0x7f982001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:02.276 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:02.269 7f983a61c700 1 --2- 172.21.2.112:0/2466160317 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f98200186c0 0x7f982001abc0 crc :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:02.303 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:02.300 7f984080d700 1 -- 172.21.2.112:0/2466160317 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f983c086e20 con 0x7f983c1a7fa0 2021-02-10T11:49:02.306 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:02.302 7f982affd700 1 -- 172.21.2.112:0/2466160317 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f983400be30 con 0x7f983c1a7fa0 2021-02-10T11:49:02.435 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:02.431 7f984080d700 1 -- 172.21.2.112:0/2466160317 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"} v 0) v1 -- 0x7f983c086e20 con 0x7f983c1a7fa0 2021-02-10T11:49:02.841 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:02.838 7f982affd700 1 -- 172.21.2.112:0/2466160317 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"}]=0 pool 'cephfs_metadata' created v194) v1 ==== 133+0+0 (secure 0 0 0) 0x7f9834003060 con 0x7f983c1a7fa0 2021-02-10T11:49:02.851 INFO:teuthology.orchestra.run.gibba012.stderr:pool 'cephfs_metadata' created 2021-02-10T11:49:02.853 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:02.849 7f984080d700 1 -- 172.21.2.112:0/2466160317 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f98200186c0 msgr2=0x7f982001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:02.853 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:02.849 7f984080d700 1 --2- 172.21.2.112:0/2466160317 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f98200186c0 0x7f982001abc0 crc :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:02.853 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:02.849 7f984080d700 1 -- 172.21.2.112:0/2466160317 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f983c1a7fa0 msgr2=0x7f983c0a6e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:02.853 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:02.849 7f984080d700 1 --2- 172.21.2.112:0/2466160317 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f983c1a7fa0 0x7f983c0a6e80 secure :-1 s=READY pgs=3244 cs=0 l=1 rev1=1 rx=0x7f9834009940 tx=0x7f983400bcf0).stop 2021-02-10T11:49:02.853 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:02.849 7f984080d700 1 -- 172.21.2.112:0/2466160317 shutdown_connections 2021-02-10T11:49:02.854 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:02.849 7f984080d700 1 --2- 172.21.2.112:0/2466160317 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f98200186c0 0x7f982001abc0 unknown :-1 s=CLOSED pgs=1461 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:02.854 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:02.849 7f984080d700 1 --2- 172.21.2.112:0/2466160317 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f983c1a7fa0 0x7f983c0a6e80 unknown :-1 s=CLOSED pgs=3244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:02.854 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:02.849 7f984080d700 1 --2- 172.21.2.112:0/2466160317 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f983c1a7ba0 0x7f983c0a6860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:02.854 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:02.849 7f984080d700 1 --2- 172.21.2.112:0/2466160317 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f983c0872c0 0x7f983c1a7580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:02.854 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:02.849 7f984080d700 1 -- 172.21.2.112:0/2466160317 >> 172.21.2.112:0/2466160317 conn(0x7f983c0be980 msgr2=0x7f983c092cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:02.854 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:02.849 7f984080d700 1 -- 172.21.2.112:0/2466160317 shutdown_connections 2021-02-10T11:49:02.854 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:02.849 7f984080d700 1 -- 172.21.2.112:0/2466160317 wait complete. 2021-02-10T11:49:02.861 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_data 8 2021-02-10T11:49:03.293 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:03.273 7f4429f74700 1 Processor -- start 2021-02-10T11:49:03.293 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:03.273 7f4429f74700 1 -- start start 2021-02-10T11:49:03.293 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:03.273 7f4429f74700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f442410aff0 0x7f44240cf750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:03.293 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:03.273 7f4429f74700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44240fe1c0 con 0x7f44240d1870 2021-02-10T11:49:03.293 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:03.273 7f4429f74700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44241b0cb0 con 0x7f44240d1c70 2021-02-10T11:49:03.293 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:03.273 7f4429f74700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4424109a60 con 0x7f442410aff0 2021-02-10T11:49:03.293 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:03.273 7f44237fe700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f44240d1870 0x7f44241b04c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:32962/0 (socket says 172.21.2.112:32962) 2021-02-10T11:49:03.293 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:03.273 7f44237fe700 1 -- 172.21.2.112:0/1535922943 learned_addr learned my addr 172.21.2.112:0/1535922943 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:49:03.293 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:03.274 7f4423fff700 1 --2- 172.21.2.112:0/1535922943 >> v2:172.21.2.120:3300/0 conn(0x7f442410aff0 0x7f44240cf750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:03.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:03.274 7f44227fc700 1 -- 172.21.2.112:0/1535922943 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 751083060 0 0) 0x7f44180009b0 con 0x7f44240d1870 2021-02-10T11:49:03.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:03.274 7f44227fc700 1 -- 172.21.2.112:0/1535922943 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f440c002fe0 con 0x7f44240d1870 2021-02-10T11:49:03.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:03.274 7f44227fc700 1 -- 172.21.2.112:0/1535922943 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4071460201 0 0) 0x7f4414000ff0 con 0x7f44240d1c70 2021-02-10T11:49:03.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:03.274 7f44227fc700 1 -- 172.21.2.112:0/1535922943 --> v1:172.21.2.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f440c003210 con 0x7f44240d1c70 2021-02-10T11:49:03.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:03.274 7f4423fff700 1 -- 172.21.2.112:0/1535922943 >> v1:172.21.2.120:6789/0 conn(0x7f44240d1c70 legacy=0x7f44240cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:03.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:03.274 7f4423fff700 1 -- 172.21.2.112:0/1535922943 >> v1:172.21.2.112:6789/0 conn(0x7f44240d1870 legacy=0x7f44241b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:03.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:03.274 7f4423fff700 1 -- 172.21.2.112:0/1535922943 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f442410b3f0 con 0x7f442410aff0 2021-02-10T11:49:03.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:03.274 7f4423fff700 1 --2- 172.21.2.112:0/1535922943 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f442410aff0 0x7f44240cf750 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 rx=0x7f441c006e50 tx=0x7f441c009e70).ready entity=mon.1 client_cookie=ec643d3b2052a8ab server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:03.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:03.274 7f44227fc700 1 -- 172.21.2.112:0/1535922943 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f441c006a20 con 0x7f442410aff0 2021-02-10T11:49:03.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:03.274 7f44227fc700 1 -- 172.21.2.112:0/1535922943 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f441c006790 con 0x7f442410aff0 2021-02-10T11:49:03.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:03.274 7f44227fc700 1 -- 172.21.2.112:0/1535922943 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f441c00b700 con 0x7f442410aff0 2021-02-10T11:49:03.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:03.274 7f4429f74700 1 -- 172.21.2.112:0/1535922943 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f442410aff0 msgr2=0x7f44240cf750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:03.295 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:03.274 7f4429f74700 1 --2- 172.21.2.112:0/1535922943 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f442410aff0 0x7f44240cf750 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 rx=0x7f441c006e50 tx=0x7f441c009e70).stop 2021-02-10T11:49:03.295 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:03.275 7f4429f74700 1 -- 172.21.2.112:0/1535922943 shutdown_connections 2021-02-10T11:49:03.295 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:03.275 7f4429f74700 1 --2- 172.21.2.112:0/1535922943 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f442410aff0 0x7f44240cf750 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:03.295 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:03.275 7f4429f74700 1 -- 172.21.2.112:0/1535922943 >> 172.21.2.112:0/1535922943 conn(0x7f44240ce420 msgr2=0x7f44240d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:03.295 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:03.275 7f4429f74700 1 -- 172.21.2.112:0/1535922943 shutdown_connections 2021-02-10T11:49:03.295 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:03.275 7f4429f74700 1 -- 172.21.2.112:0/1535922943 wait complete. 2021-02-10T11:49:03.296 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:03.290 7f4429f74700 1 Processor -- start 2021-02-10T11:49:03.297 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:03.290 7f4429f74700 1 -- start start 2021-02-10T11:49:03.297 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:03.290 7f4429f74700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f44240a7950 0x7f44240b8df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:03.297 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:03.290 7f4429f74700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f44241a7490 0x7f44241a7890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:03.297 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:03.290 7f4429f74700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f44241a7eb0 0x7f44240ac630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:03.297 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:03.290 7f4429f74700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44240a7fb0 con 0x7f44240a7950 2021-02-10T11:49:03.297 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:03.290 7f4429f74700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44240acc50 con 0x7f44241a7490 2021-02-10T11:49:03.297 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:03.290 7f4429f74700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44240ace30 con 0x7f44241a7eb0 2021-02-10T11:49:03.297 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:03.290 7f44237fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f44240a7950 0x7f44240b8df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:03.297 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:03.290 7f44237fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f44240a7950 0x7f44240b8df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:33506/0 (socket says 172.21.2.112:33506) 2021-02-10T11:49:03.297 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:03.290 7f44237fe700 1 -- 172.21.2.112:0/3556674093 learned_addr learned my addr 172.21.2.112:0/3556674093 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:49:03.297 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:03.290 7f4422ffd700 1 --2- 172.21.2.112:0/3556674093 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f44241a7490 0x7f44241a7890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:03.297 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:03.290 7f4423fff700 1 --2- 172.21.2.112:0/3556674093 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f44241a7eb0 0x7f44240ac630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:03.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:03.291 7f44237fe700 1 -- 172.21.2.112:0/3556674093 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f44241a7eb0 msgr2=0x7f44240ac630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:03.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:03.291 7f44237fe700 1 --2- 172.21.2.112:0/3556674093 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f44241a7eb0 0x7f44240ac630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:03.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:03.291 7f44237fe700 1 -- 172.21.2.112:0/3556674093 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f44241a7490 msgr2=0x7f44241a7890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:03.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:03.291 7f44237fe700 1 --2- 172.21.2.112:0/3556674093 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f44241a7490 0x7f44241a7890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:03.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:03.291 7f44237fe700 1 -- 172.21.2.112:0/3556674093 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44240ad050 con 0x7f44240a7950 2021-02-10T11:49:03.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:03.291 7f44237fe700 1 --2- 172.21.2.112:0/3556674093 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f44240a7950 0x7f44240b8df0 secure :-1 s=READY pgs=3246 cs=0 l=1 rev1=1 rx=0x7f4418001c60 tx=0x7f441800a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:03.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:03.291 7f440bfff700 1 -- 172.21.2.112:0/3556674093 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f441800acc0 con 0x7f44240a7950 2021-02-10T11:49:03.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:03.291 7f440bfff700 1 -- 172.21.2.112:0/3556674093 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4418004140 con 0x7f44240a7950 2021-02-10T11:49:03.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:03.291 7f440bfff700 1 -- 172.21.2.112:0/3556674093 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4418004660 con 0x7f44240a7950 2021-02-10T11:49:03.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:03.291 7f4429f74700 1 -- 172.21.2.112:0/3556674093 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44240ad050 con 0x7f44240a7950 2021-02-10T11:49:03.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:03.291 7f4429f74700 1 -- 172.21.2.112:0/3556674093 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44240a8530 con 0x7f44240a7950 2021-02-10T11:49:03.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:03.292 7f440bfff700 1 -- 172.21.2.112:0/3556674093 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4418004320 con 0x7f44240a7950 2021-02-10T11:49:03.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:03.292 7f440bfff700 1 --2- 172.21.2.112:0/3556674093 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f44040186c0 0x7f440401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:03.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:03.292 7f440bfff700 1 -- 172.21.2.112:0/3556674093 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(195..195 src has 1..195) v4 ==== 5107+0+0 (secure 0 0 0) 0x7f4418032350 con 0x7f44240a7950 2021-02-10T11:49:03.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:03.292 7f4422ffd700 1 --2- 172.21.2.112:0/3556674093 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f44040186c0 0x7f440401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:03.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:03.293 7f4422ffd700 1 --2- 172.21.2.112:0/3556674093 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f44040186c0 0x7f440401abc0 crc :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:03.326 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:03.323 7f4429f74700 1 -- 172.21.2.112:0/3556674093 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4424082a70 con 0x7f44240a7950 2021-02-10T11:49:03.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:03.325 7f440bfff700 1 -- 172.21.2.112:0/3556674093 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f441802dfe0 con 0x7f44240a7950 2021-02-10T11:49:03.457 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:03.454 7f4429f74700 1 -- 172.21.2.112:0/3556674093 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"} v 0) v1 -- 0x7f4424082a70 con 0x7f44240a7950 2021-02-10T11:49:03.848 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:03.845 7f440bfff700 1 -- 172.21.2.112:0/3556674093 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"}]=0 pool 'cephfs_data' created v196) v1 ==== 125+0+0 (secure 0 0 0) 0x7f441802dfe0 con 0x7f44240a7950 2021-02-10T11:49:03.873 INFO:teuthology.orchestra.run.gibba012.stderr:pool 'cephfs_data' created 2021-02-10T11:49:03.875 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:03.871 7f4429f74700 1 -- 172.21.2.112:0/3556674093 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f44040186c0 msgr2=0x7f440401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:03.875 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:03.871 7f4429f74700 1 --2- 172.21.2.112:0/3556674093 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f44040186c0 0x7f440401abc0 crc :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:03.875 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:03.871 7f4429f74700 1 -- 172.21.2.112:0/3556674093 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f44240a7950 msgr2=0x7f44240b8df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:03.875 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:03.871 7f4429f74700 1 --2- 172.21.2.112:0/3556674093 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f44240a7950 0x7f44240b8df0 secure :-1 s=READY pgs=3246 cs=0 l=1 rev1=1 rx=0x7f4418001c60 tx=0x7f441800a8d0).stop 2021-02-10T11:49:03.875 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:03.871 7f4429f74700 1 -- 172.21.2.112:0/3556674093 shutdown_connections 2021-02-10T11:49:03.875 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:03.871 7f4429f74700 1 --2- 172.21.2.112:0/3556674093 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f44040186c0 0x7f440401abc0 unknown :-1 s=CLOSED pgs=1462 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:03.876 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:03.871 7f4429f74700 1 --2- 172.21.2.112:0/3556674093 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f44241a7490 0x7f44241a7890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:03.876 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:03.871 7f4429f74700 1 --2- 172.21.2.112:0/3556674093 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f44241a7eb0 0x7f44240ac630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:03.876 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:03.871 7f4429f74700 1 --2- 172.21.2.112:0/3556674093 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f44240a7950 0x7f44240b8df0 unknown :-1 s=CLOSED pgs=3246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:03.876 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:03.871 7f4429f74700 1 -- 172.21.2.112:0/3556674093 >> 172.21.2.112:0/3556674093 conn(0x7f44240a02d0 msgr2=0x7f442409f200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:03.876 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:03.872 7f4429f74700 1 -- 172.21.2.112:0/3556674093 shutdown_connections 2021-02-10T11:49:03.876 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:03.872 7f4429f74700 1 -- 172.21.2.112:0/3556674093 wait complete. 2021-02-10T11:49:03.884 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2021-02-10T11:49:04.316 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.296 7f3c900ad700 1 Processor -- start 2021-02-10T11:49:04.316 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.296 7f3c900ad700 1 -- start start 2021-02-10T11:49:04.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.297 7f3c900ad700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f3c880d1880 0x7f3c881b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:04.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.297 7f3c900ad700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c880fe1d0 con 0x7f3c881b0af0 2021-02-10T11:49:04.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.297 7f3c900ad700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c88109a70 con 0x7f3c880cff80 2021-02-10T11:49:04.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.297 7f3c900ad700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c88088070 con 0x7f3c880d1880 2021-02-10T11:49:04.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.297 7f3c8dec9700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f3c880d1880 0x7f3c881b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:04.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.297 7f3c8dec9700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f3c880d1880 0x7f3c881b04d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:33514/0 (socket says 172.21.2.112:33514) 2021-02-10T11:49:04.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.297 7f3c8d6c8700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f3c881b0af0 0x7f3c88108950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:32976/0 (socket says 172.21.2.112:32976) 2021-02-10T11:49:04.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.297 7f3c8dec9700 1 -- 172.21.2.112:0/3079512178 learned_addr learned my addr 172.21.2.112:0/3079512178 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:49:04.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.297 7f3c8dec9700 1 -- 172.21.2.112:0/3079512178 >> v1:172.21.2.120:6789/0 conn(0x7f3c880cff80 legacy=0x7f3c88109140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:49:04.317 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.297 7f3c8dec9700 1 -- 172.21.2.112:0/3079512178 >> v1:172.21.2.112:6789/0 conn(0x7f3c881b0af0 legacy=0x7f3c88108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:04.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.297 7f3c8dec9700 1 -- 172.21.2.112:0/3079512178 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c88088260 con 0x7f3c880d1880 2021-02-10T11:49:04.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.297 7f3c8dec9700 1 --2- 172.21.2.112:0/3079512178 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3c880d1880 0x7f3c881b04d0 secure :-1 s=READY pgs=3248 cs=0 l=1 rev1=1 rx=0x7f3c84006700 tx=0x7f3c84009b00).ready entity=mon.0 client_cookie=ab1d5d117d1f9845 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:04.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.298 7f3c8cec7700 1 -- 172.21.2.112:0/3079512178 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3c8400c0c0 con 0x7f3c880d1880 2021-02-10T11:49:04.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.298 7f3c8cec7700 1 -- 172.21.2.112:0/3079512178 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c8400c400 con 0x7f3c880d1880 2021-02-10T11:49:04.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.298 7f3c8cec7700 1 -- 172.21.2.112:0/3079512178 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3c8400c920 con 0x7f3c880d1880 2021-02-10T11:49:04.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.298 7f3c900ad700 1 -- 172.21.2.112:0/3079512178 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3c880d1880 msgr2=0x7f3c881b04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:04.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.298 7f3c900ad700 1 --2- 172.21.2.112:0/3079512178 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3c880d1880 0x7f3c881b04d0 secure :-1 s=READY pgs=3248 cs=0 l=1 rev1=1 rx=0x7f3c84006700 tx=0x7f3c84009b00).stop 2021-02-10T11:49:04.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.298 7f3c900ad700 1 -- 172.21.2.112:0/3079512178 shutdown_connections 2021-02-10T11:49:04.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.298 7f3c900ad700 1 --2- 172.21.2.112:0/3079512178 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3c880d1880 0x7f3c881b04d0 unknown :-1 s=CLOSED pgs=3248 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:04.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.298 7f3c900ad700 1 -- 172.21.2.112:0/3079512178 >> 172.21.2.112:0/3079512178 conn(0x7f3c880ce430 msgr2=0x7f3c880d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:04.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.298 7f3c900ad700 1 -- 172.21.2.112:0/3079512178 shutdown_connections 2021-02-10T11:49:04.318 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.298 7f3c900ad700 1 -- 172.21.2.112:0/3079512178 wait complete. 2021-02-10T11:49:04.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.313 7f3c900ad700 1 Processor -- start 2021-02-10T11:49:04.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.313 7f3c900ad700 1 -- start start 2021-02-10T11:49:04.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.313 7f3c900ad700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3c8809fbb0 0x7f3c881b5a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:04.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.313 7f3c900ad700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3c8809ffb0 0x7f3c8808a930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:04.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.313 7f3c900ad700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3c8808af50 0x7f3c8809c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:04.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.313 7f3c900ad700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c8809f6b0 con 0x7f3c8809fbb0 2021-02-10T11:49:04.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.313 7f3c900ad700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c8808b390 con 0x7f3c8809ffb0 2021-02-10T11:49:04.320 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.313 7f3c900ad700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c881b6060 con 0x7f3c8808af50 2021-02-10T11:49:04.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.314 7f3c8dec9700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3c8809fbb0 0x7f3c881b5a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:04.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.314 7f3c8dec9700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3c8809fbb0 0x7f3c881b5a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:33522/0 (socket says 172.21.2.112:33522) 2021-02-10T11:49:04.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.314 7f3c8dec9700 1 -- 172.21.2.112:0/692606319 learned_addr learned my addr 172.21.2.112:0/692606319 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:49:04.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.314 7f3c8dec9700 1 -- 172.21.2.112:0/692606319 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3c8808af50 msgr2=0x7f3c8809c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:04.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.314 7f3c8dec9700 1 --2- 172.21.2.112:0/692606319 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3c8808af50 0x7f3c8809c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:04.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.314 7f3c8dec9700 1 -- 172.21.2.112:0/692606319 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3c8809ffb0 msgr2=0x7f3c8808a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:04.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.314 7f3c8dec9700 1 --2- 172.21.2.112:0/692606319 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3c8809ffb0 0x7f3c8808a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:04.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.314 7f3c8dec9700 1 -- 172.21.2.112:0/692606319 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c8809ca30 con 0x7f3c8809fbb0 2021-02-10T11:49:04.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.314 7f3c8dec9700 1 --2- 172.21.2.112:0/692606319 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3c8809fbb0 0x7f3c881b5a40 secure :-1 s=READY pgs=3249 cs=0 l=1 rev1=1 rx=0x7f3c8400b660 tx=0x7f3c8400ba00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:04.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.314 7f3c7a7fc700 1 -- 172.21.2.112:0/692606319 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3c84009440 con 0x7f3c8809fbb0 2021-02-10T11:49:04.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.314 7f3c7a7fc700 1 -- 172.21.2.112:0/692606319 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c84000c90 con 0x7f3c8809fbb0 2021-02-10T11:49:04.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.314 7f3c7a7fc700 1 -- 172.21.2.112:0/692606319 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3c84002030 con 0x7f3c8809fbb0 2021-02-10T11:49:04.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.314 7f3c900ad700 1 -- 172.21.2.112:0/692606319 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c8809ca30 con 0x7f3c8809fbb0 2021-02-10T11:49:04.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.315 7f3c900ad700 1 -- 172.21.2.112:0/692606319 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c8809ca30 con 0x7f3c8809fbb0 2021-02-10T11:49:04.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.315 7f3c7a7fc700 1 -- 172.21.2.112:0/692606319 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3c840093e0 con 0x7f3c8809fbb0 2021-02-10T11:49:04.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.315 7f3c7a7fc700 1 --2- 172.21.2.112:0/692606319 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3c700186c0 0x7f3c7001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:04.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.315 7f3c7a7fc700 1 -- 172.21.2.112:0/692606319 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 5457+0+0 (secure 0 0 0) 0x7f3c84032570 con 0x7f3c8809fbb0 2021-02-10T11:49:04.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.315 7f3c8d6c8700 1 --2- 172.21.2.112:0/692606319 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3c700186c0 0x7f3c7001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:04.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.316 7f3c8d6c8700 1 --2- 172.21.2.112:0/692606319 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3c700186c0 0x7f3c7001abc0 crc :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:04.350 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.347 7f3c900ad700 1 -- 172.21.2.112:0/692606319 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c880802b0 con 0x7f3c8809fbb0 2021-02-10T11:49:04.353 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.349 7f3c7a7fc700 1 -- 172.21.2.112:0/692606319 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3c8402df40 con 0x7f3c8809fbb0 2021-02-10T11:49:04.481 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.477 7f3c900ad700 1 -- 172.21.2.112:0/692606319 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"} v 0) v1 -- 0x7f3c880ce430 con 0x7f3c8809fbb0 2021-02-10T11:49:04.485 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.481 7f3c7a7fc700 1 -- 172.21.2.112:0/692606319 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"}]=0 new fs with metadata pool 42 and data pool 43 v171) v1 ==== 189+0+0 (secure 0 0 0) 0x7f3c840093e0 con 0x7f3c8809fbb0 2021-02-10T11:49:04.489 INFO:teuthology.orchestra.run.gibba012.stderr:new fs with metadata pool 42 and data pool 43 2021-02-10T11:49:04.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.486 7f3c900ad700 1 -- 172.21.2.112:0/692606319 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3c700186c0 msgr2=0x7f3c7001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:04.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.486 7f3c900ad700 1 --2- 172.21.2.112:0/692606319 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3c700186c0 0x7f3c7001abc0 crc :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:04.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.486 7f3c900ad700 1 -- 172.21.2.112:0/692606319 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3c8809fbb0 msgr2=0x7f3c881b5a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:04.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.486 7f3c900ad700 1 --2- 172.21.2.112:0/692606319 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3c8809fbb0 0x7f3c881b5a40 secure :-1 s=READY pgs=3249 cs=0 l=1 rev1=1 rx=0x7f3c8400b660 tx=0x7f3c8400ba00).stop 2021-02-10T11:49:04.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.487 7f3c900ad700 1 -- 172.21.2.112:0/692606319 shutdown_connections 2021-02-10T11:49:04.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.487 7f3c900ad700 1 --2- 172.21.2.112:0/692606319 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f3c700186c0 0x7f3c7001abc0 unknown :-1 s=CLOSED pgs=1465 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:04.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.487 7f3c900ad700 1 --2- 172.21.2.112:0/692606319 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f3c8809ffb0 0x7f3c8808a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:04.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.487 7f3c900ad700 1 --2- 172.21.2.112:0/692606319 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f3c8808af50 0x7f3c8809c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:04.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.487 7f3c900ad700 1 --2- 172.21.2.112:0/692606319 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f3c8809fbb0 0x7f3c881b5a40 unknown :-1 s=CLOSED pgs=3249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:04.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.487 7f3c900ad700 1 -- 172.21.2.112:0/692606319 >> 172.21.2.112:0/692606319 conn(0x7f3c8809bad0 msgr2=0x7f3c880804e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:04.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.487 7f3c900ad700 1 -- 172.21.2.112:0/692606319 shutdown_connections 2021-02-10T11:49:04.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.487 7f3c900ad700 1 -- 172.21.2.112:0/692606319 wait complete. 2021-02-10T11:49:04.499 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:49:04.929 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.909 7f137ebdd700 1 Processor -- start 2021-02-10T11:49:04.929 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.909 7f137ebdd700 1 -- start start 2021-02-10T11:49:04.929 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.909 7f137ebdd700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f13780d1870 0x7f13781b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:04.929 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.909 7f137ebdd700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f13781b0a10 0x7f13780cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:04.930 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.909 7f137ebdd700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f13780d1c70 0x7f13780cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:04.930 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.909 7f137ebdd700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13780e5ed0 con 0x7f13781b0a10 2021-02-10T11:49:04.930 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.909 7f137ebdd700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13780f0b10 con 0x7f13780d1870 2021-02-10T11:49:04.930 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.909 7f137ebdd700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13780cff70 con 0x7f13780d1c70 2021-02-10T11:49:04.930 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.909 7f1377fff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f13781b0a10 0x7f13780cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:04.930 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.910 7f1377fff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f13781b0a10 0x7f13780cf330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:33528/0 (socket says 172.21.2.112:33528) 2021-02-10T11:49:04.930 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.910 7f1377fff700 1 -- 172.21.2.112:0/951331470 learned_addr learned my addr 172.21.2.112:0/951331470 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:49:04.930 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.910 7f137c9f9700 1 --2- 172.21.2.112:0/951331470 >> v2:172.21.2.120:3300/0 conn(0x7f13780d1870 0x7f13781b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:04.930 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.910 7f137d1fa700 1 --2- 172.21.2.112:0/951331470 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f13780d1c70 0x7f13780cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:04.930 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.910 7f1377fff700 1 -- 172.21.2.112:0/951331470 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f13780d1c70 msgr2=0x7f13780cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:49:04.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.910 7f1377fff700 1 --2- 172.21.2.112:0/951331470 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f13780d1c70 0x7f13780cf950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:04.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.910 7f1377fff700 1 -- 172.21.2.112:0/951331470 >> v2:172.21.2.120:3300/0 conn(0x7f13780d1870 msgr2=0x7f13781b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:49:04.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.910 7f1377fff700 1 --2- 172.21.2.112:0/951331470 >> v2:172.21.2.120:3300/0 conn(0x7f13780d1870 0x7f13781b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:04.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.910 7f1377fff700 1 -- 172.21.2.112:0/951331470 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13780d0150 con 0x7f13781b0a10 2021-02-10T11:49:04.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.910 7f1377fff700 1 --2- 172.21.2.112:0/951331470 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f13781b0a10 0x7f13780cf330 secure :-1 s=READY pgs=3250 cs=0 l=1 rev1=1 rx=0x7f1368006700 tx=0x7f1368009b80).ready entity=mon.0 client_cookie=46d810308abecec9 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:04.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.910 7f13777fe700 1 -- 172.21.2.112:0/951331470 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f136800c0c0 con 0x7f13781b0a10 2021-02-10T11:49:04.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.910 7f13777fe700 1 -- 172.21.2.112:0/951331470 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f136800c340 con 0x7f13781b0a10 2021-02-10T11:49:04.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.910 7f13777fe700 1 -- 172.21.2.112:0/951331470 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f136800c820 con 0x7f13781b0a10 2021-02-10T11:49:04.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.910 7f137ebdd700 1 -- 172.21.2.112:0/951331470 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f13781b0a10 msgr2=0x7f13780cf330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:04.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.910 7f137ebdd700 1 --2- 172.21.2.112:0/951331470 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f13781b0a10 0x7f13780cf330 secure :-1 s=READY pgs=3250 cs=0 l=1 rev1=1 rx=0x7f1368006700 tx=0x7f1368009b80).stop 2021-02-10T11:49:04.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.911 7f137ebdd700 1 -- 172.21.2.112:0/951331470 shutdown_connections 2021-02-10T11:49:04.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.911 7f137ebdd700 1 --2- 172.21.2.112:0/951331470 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f13780d1c70 0x7f13780cf950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:04.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.911 7f137ebdd700 1 --2- 172.21.2.112:0/951331470 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f13781b0a10 0x7f13780cf330 unknown :-1 s=CLOSED pgs=3250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:04.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.911 7f137ebdd700 1 --2- 172.21.2.112:0/951331470 >> v2:172.21.2.120:3300/0 conn(0x7f13780d1870 0x7f13781b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:04.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.911 7f137ebdd700 1 -- 172.21.2.112:0/951331470 >> 172.21.2.112:0/951331470 conn(0x7f13780ce450 msgr2=0x7f13780d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:04.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.911 7f137ebdd700 1 -- 172.21.2.112:0/951331470 shutdown_connections 2021-02-10T11:49:04.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.911 7f137ebdd700 1 -- 172.21.2.112:0/951331470 wait complete. 2021-02-10T11:49:04.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.925 7f137ebdd700 1 Processor -- start 2021-02-10T11:49:04.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.925 7f137ebdd700 1 -- start start 2021-02-10T11:49:04.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.926 7f137ebdd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f13781942b0 0x7f13780b3550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:04.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.926 7f137ebdd700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1378195070 0x7f1378195470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:04.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.926 7f137ebdd700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f13780bb8e0 0x7f13780bbce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:04.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.926 7f137ebdd700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13781946b0 con 0x7f13781942b0 2021-02-10T11:49:04.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.926 7f137ebdd700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13780b3b70 con 0x7f13780bb8e0 2021-02-10T11:49:04.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.926 7f137ebdd700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13780b3d90 con 0x7f1378195070 2021-02-10T11:49:04.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.926 7f137c9f9700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f13781942b0 0x7f13780b3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:04.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.926 7f137c9f9700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f13781942b0 0x7f13780b3550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:33534/0 (socket says 172.21.2.112:33534) 2021-02-10T11:49:04.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.926 7f137c9f9700 1 -- 172.21.2.112:0/3515019747 learned_addr learned my addr 172.21.2.112:0/3515019747 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:49:04.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.926 7f1377fff700 1 --2- 172.21.2.112:0/3515019747 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1378195070 0x7f1378195470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:04.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.926 7f137d1fa700 1 --2- 172.21.2.112:0/3515019747 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f13780bb8e0 0x7f13780bbce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:04.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.926 7f137c9f9700 1 -- 172.21.2.112:0/3515019747 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1378195070 msgr2=0x7f1378195470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:04.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.927 7f137c9f9700 1 --2- 172.21.2.112:0/3515019747 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1378195070 0x7f1378195470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:04.934 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.927 7f137c9f9700 1 -- 172.21.2.112:0/3515019747 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f13780bb8e0 msgr2=0x7f13780bbce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:04.935 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.927 7f137c9f9700 1 --2- 172.21.2.112:0/3515019747 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f13780bb8e0 0x7f13780bbce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:04.935 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.927 7f137c9f9700 1 -- 172.21.2.112:0/3515019747 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13781a49e0 con 0x7f13781942b0 2021-02-10T11:49:04.935 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.927 7f137d1fa700 1 --2- 172.21.2.112:0/3515019747 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f13780bb8e0 0x7f13780bbce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:49:04.935 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.927 7f137c9f9700 1 --2- 172.21.2.112:0/3515019747 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f13781942b0 0x7f13780b3550 secure :-1 s=READY pgs=3251 cs=0 l=1 rev1=1 rx=0x7f1360006f00 tx=0x7f136000b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:04.935 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.927 7f13757fa700 1 -- 172.21.2.112:0/3515019747 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f136000bd30 con 0x7f13781942b0 2021-02-10T11:49:04.935 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.927 7f13757fa700 1 -- 172.21.2.112:0/3515019747 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1360002030 con 0x7f13781942b0 2021-02-10T11:49:04.935 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.927 7f13757fa700 1 -- 172.21.2.112:0/3515019747 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1360002480 con 0x7f13781942b0 2021-02-10T11:49:04.935 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.927 7f137ebdd700 1 -- 172.21.2.112:0/3515019747 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13781a49e0 con 0x7f13781942b0 2021-02-10T11:49:04.935 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.927 7f137ebdd700 1 -- 172.21.2.112:0/3515019747 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13781a49e0 con 0x7f13781942b0 2021-02-10T11:49:04.935 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.928 7f13757fa700 1 -- 172.21.2.112:0/3515019747 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1360000c90 con 0x7f13781942b0 2021-02-10T11:49:04.935 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.928 7f13757fa700 1 --2- 172.21.2.112:0/3515019747 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f135c0186c0 0x7f135c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:04.935 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.928 7f13757fa700 1 -- 172.21.2.112:0/3515019747 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f1360032560 con 0x7f13781942b0 2021-02-10T11:49:04.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.928 7f1377fff700 1 --2- 172.21.2.112:0/3515019747 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f135c0186c0 0x7f135c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:04.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.929 7f1377fff700 1 --2- 172.21.2.112:0/3515019747 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f135c0186c0 0x7f135c01abc0 crc :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:04.962 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.959 7f137ebdd700 1 -- 172.21.2.112:0/3515019747 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13780bc5f0 con 0x7f13781942b0 2021-02-10T11:49:04.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:04.962 7f13757fa700 1 -- 172.21.2.112:0/3515019747 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f136002f540 con 0x7f13781942b0 2021-02-10T11:49:05.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:05.086 7f137ebdd700 1 -- 172.21.2.112:0/3515019747 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f13780bc5f0 con 0x7f13781942b0 2021-02-10T11:49:05.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:05.087 7f13757fa700 1 -- 172.21.2.112:0/3515019747 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v197) v1 ==== 74+0+14020 (secure 0 0 0) 0x7f136002f540 con 0x7f13781942b0 2021-02-10T11:49:05.091 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:49:05.091 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":197,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:49:04.478749","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":43,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":42,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:49:02.450970","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"197","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":43,"pool_name":"cephfs_data","create_time":"2021-02-10 11:49:03.473275","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"197","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":196,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":196,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":196,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":196,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":196,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:49:05.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:05.088 7f137ebdd700 1 -- 172.21.2.112:0/3515019747 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f135c0186c0 msgr2=0x7f135c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:05.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:05.088 7f137ebdd700 1 --2- 172.21.2.112:0/3515019747 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f135c0186c0 0x7f135c01abc0 crc :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:05.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:05.088 7f137ebdd700 1 -- 172.21.2.112:0/3515019747 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f13781942b0 msgr2=0x7f13780b3550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:05.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:05.088 7f137ebdd700 1 --2- 172.21.2.112:0/3515019747 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f13781942b0 0x7f13780b3550 secure :-1 s=READY pgs=3251 cs=0 l=1 rev1=1 rx=0x7f1360006f00 tx=0x7f136000b940).stop 2021-02-10T11:49:05.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:05.089 7f137ebdd700 1 -- 172.21.2.112:0/3515019747 shutdown_connections 2021-02-10T11:49:05.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:05.089 7f137ebdd700 1 --2- 172.21.2.112:0/3515019747 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f135c0186c0 0x7f135c01abc0 unknown :-1 s=CLOSED pgs=1466 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:05.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:05.089 7f137ebdd700 1 --2- 172.21.2.112:0/3515019747 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f13780bb8e0 0x7f13780bbce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:05.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:05.089 7f137ebdd700 1 --2- 172.21.2.112:0/3515019747 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f1378195070 0x7f1378195470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:05.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:05.089 7f137ebdd700 1 --2- 172.21.2.112:0/3515019747 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f13781942b0 0x7f13780b3550 unknown :-1 s=CLOSED pgs=3251 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:05.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:05.089 7f137ebdd700 1 -- 172.21.2.112:0/3515019747 >> 172.21.2.112:0/3515019747 conn(0x7f13780b3040 msgr2=0x7f13780c3250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:05.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:05.089 7f137ebdd700 1 -- 172.21.2.112:0/3515019747 shutdown_connections 2021-02-10T11:49:05.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:05.089 7f137ebdd700 1 -- 172.21.2.112:0/3515019747 wait complete. 2021-02-10T11:49:05.101 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:49:05.532 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:05.512 7f6e06108700 1 Processor -- start 2021-02-10T11:49:05.533 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:05.512 7f6e06108700 1 -- start start 2021-02-10T11:49:05.533 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:05.513 7f6e06108700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6e000d1870 0x7f6e001b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:05.533 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:05.513 7f6e06108700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f6e001b0a10 0x7f6e000cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:05.533 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:05.513 7f6e06108700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f6e000d1c70 0x7f6e000cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:05.533 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:05.513 7f6e06108700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e000fe230 con 0x7f6e001b0a10 2021-02-10T11:49:05.533 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:05.513 7f6e06108700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e000e5ed0 con 0x7f6e000d1c70 2021-02-10T11:49:05.533 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:05.513 7f6e06108700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e000f0b10 con 0x7f6e000d1870 2021-02-10T11:49:05.533 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:05.513 7f6dfeffd700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f6e001b0a10 0x7f6e000cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:05.533 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:05.513 7f6dfffff700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f6e000d1c70 0x7f6e000cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:05.534 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:05.513 7f6dfeffd700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f6e001b0a10 0x7f6e000cf330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:33542/0 (socket says 172.21.2.112:33542) 2021-02-10T11:49:05.534 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:05.513 7f6dfffff700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f6e000d1c70 0x7f6e000cf950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3300/0 says I am v2:172.21.2.112:35822/0 (socket says 172.21.2.112:35822) 2021-02-10T11:49:05.534 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:05.513 7f6dfeffd700 1 -- 172.21.2.112:0/2176737096 learned_addr learned my addr 172.21.2.112:0/2176737096 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:49:05.534 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:05.513 7f6dff7fe700 1 --2- 172.21.2.112:0/2176737096 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6e000d1870 0x7f6e001b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:05.534 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:05.513 7f6dfeffd700 1 -- 172.21.2.112:0/2176737096 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6e000d1870 msgr2=0x7f6e001b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:49:05.534 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:05.513 7f6dfeffd700 1 --2- 172.21.2.112:0/2176737096 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6e000d1870 0x7f6e001b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:05.534 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:05.513 7f6dfeffd700 1 -- 172.21.2.112:0/2176737096 >> v2:172.21.2.120:3300/0 conn(0x7f6e000d1c70 msgr2=0x7f6e000cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:49:05.534 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:05.513 7f6dfeffd700 1 --2- 172.21.2.112:0/2176737096 >> v2:172.21.2.120:3300/0 conn(0x7f6e000d1c70 0x7f6e000cf950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:05.534 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:05.513 7f6dfeffd700 1 -- 172.21.2.112:0/2176737096 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e000cff70 con 0x7f6e001b0a10 2021-02-10T11:49:05.534 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:05.513 7f6dfeffd700 1 --2- 172.21.2.112:0/2176737096 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6e001b0a10 0x7f6e000cf330 secure :-1 s=READY pgs=3252 cs=0 l=1 rev1=1 rx=0x7f6df0006700 tx=0x7f6df0009b80).ready entity=mon.0 client_cookie=97dd9352487234ab server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:05.534 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:05.514 7f6dfe7fc700 1 -- 172.21.2.112:0/2176737096 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6df000c0c0 con 0x7f6e001b0a10 2021-02-10T11:49:05.534 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:05.514 7f6dfe7fc700 1 -- 172.21.2.112:0/2176737096 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6df000c440 con 0x7f6e001b0a10 2021-02-10T11:49:05.534 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:05.514 7f6dfe7fc700 1 -- 172.21.2.112:0/2176737096 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6df000c920 con 0x7f6e001b0a10 2021-02-10T11:49:05.535 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:05.514 7f6e06108700 1 -- 172.21.2.112:0/2176737096 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6e001b0a10 msgr2=0x7f6e000cf330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:05.535 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:05.514 7f6e06108700 1 --2- 172.21.2.112:0/2176737096 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6e001b0a10 0x7f6e000cf330 secure :-1 s=READY pgs=3252 cs=0 l=1 rev1=1 rx=0x7f6df0006700 tx=0x7f6df0009b80).stop 2021-02-10T11:49:05.535 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:05.514 7f6e06108700 1 -- 172.21.2.112:0/2176737096 shutdown_connections 2021-02-10T11:49:05.535 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:05.514 7f6e06108700 1 --2- 172.21.2.112:0/2176737096 >> v2:172.21.2.120:3300/0 conn(0x7f6e000d1c70 0x7f6e000cf950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:05.535 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:05.514 7f6e06108700 1 --2- 172.21.2.112:0/2176737096 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6e001b0a10 0x7f6e000cf330 unknown :-1 s=CLOSED pgs=3252 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:05.535 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:05.514 7f6e06108700 1 --2- 172.21.2.112:0/2176737096 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6e000d1870 0x7f6e001b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:05.535 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:05.514 7f6e06108700 1 -- 172.21.2.112:0/2176737096 >> 172.21.2.112:0/2176737096 conn(0x7f6e000ce450 msgr2=0x7f6e000d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:05.535 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:05.514 7f6e06108700 1 -- 172.21.2.112:0/2176737096 shutdown_connections 2021-02-10T11:49:05.535 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:05.514 7f6e06108700 1 -- 172.21.2.112:0/2176737096 wait complete. 2021-02-10T11:49:05.535 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:05.529 7f6e06108700 1 Processor -- start 2021-02-10T11:49:05.535 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:05.529 7f6e06108700 1 -- start start 2021-02-10T11:49:05.537 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:05.529 7f6e06108700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6e001a4a60 0x7f6e000c1c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:05.537 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:05.529 7f6e06108700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6e000c2290 0x7f6e000c2690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:05.537 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:05.529 7f6e06108700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6e000c2cb0 0x7f6e000bfd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:05.537 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:05.529 7f6e06108700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e000f05b0 con 0x7f6e000c2290 2021-02-10T11:49:05.537 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:05.529 7f6e06108700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e000f07d0 con 0x7f6e001a4a60 2021-02-10T11:49:05.537 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:05.529 7f6e06108700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e000c0340 con 0x7f6e000c2cb0 2021-02-10T11:49:05.537 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:05.530 7f6dfeffd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6e000c2290 0x7f6e000c2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:05.537 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:05.530 7f6dfeffd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6e000c2290 0x7f6e000c2690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:33552/0 (socket says 172.21.2.112:33552) 2021-02-10T11:49:05.537 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:05.530 7f6dfeffd700 1 -- 172.21.2.112:0/3898329080 learned_addr learned my addr 172.21.2.112:0/3898329080 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:49:05.537 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:05.530 7f6dff7fe700 1 --2- 172.21.2.112:0/3898329080 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6e001a4a60 0x7f6e000c1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:05.537 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:05.530 7f6dfffff700 1 --2- 172.21.2.112:0/3898329080 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6e000c2cb0 0x7f6e000bfd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:05.537 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:05.530 7f6dfeffd700 1 -- 172.21.2.112:0/3898329080 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6e000c2cb0 msgr2=0x7f6e000bfd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:05.538 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:05.530 7f6dfeffd700 1 --2- 172.21.2.112:0/3898329080 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6e000c2cb0 0x7f6e000bfd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:05.538 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:05.530 7f6dfeffd700 1 -- 172.21.2.112:0/3898329080 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6e001a4a60 msgr2=0x7f6e000c1c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:05.538 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:05.530 7f6dfeffd700 1 --2- 172.21.2.112:0/3898329080 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6e001a4a60 0x7f6e000c1c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:05.538 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:05.530 7f6dfeffd700 1 -- 172.21.2.112:0/3898329080 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e000c0560 con 0x7f6e000c2290 2021-02-10T11:49:05.538 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:05.530 7f6dfeffd700 1 --2- 172.21.2.112:0/3898329080 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6e000c2290 0x7f6e000c2690 secure :-1 s=READY pgs=3253 cs=0 l=1 rev1=1 rx=0x7f6df000b5e0 tx=0x7f6df000b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:05.538 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:05.530 7f6df7fff700 1 -- 172.21.2.112:0/3898329080 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6df000bd70 con 0x7f6e000c2290 2021-02-10T11:49:05.538 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:05.530 7f6df7fff700 1 -- 172.21.2.112:0/3898329080 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6df0002030 con 0x7f6e000c2290 2021-02-10T11:49:05.538 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:05.530 7f6e06108700 1 -- 172.21.2.112:0/3898329080 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e000c0560 con 0x7f6e000c2290 2021-02-10T11:49:05.538 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:05.530 7f6df7fff700 1 -- 172.21.2.112:0/3898329080 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6df0002550 con 0x7f6e000c2290 2021-02-10T11:49:05.538 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:05.530 7f6e06108700 1 -- 172.21.2.112:0/3898329080 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e000c0560 con 0x7f6e000c2290 2021-02-10T11:49:05.538 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:05.531 7f6df7fff700 1 -- 172.21.2.112:0/3898329080 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6df0002210 con 0x7f6e000c2290 2021-02-10T11:49:05.538 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:05.531 7f6df7fff700 1 --2- 172.21.2.112:0/3898329080 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6de4018730 0x7f6de401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:05.539 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:05.531 7f6df7fff700 1 -- 172.21.2.112:0/3898329080 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f6df0032650 con 0x7f6e000c2290 2021-02-10T11:49:05.539 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:05.531 7f6dff7fe700 1 --2- 172.21.2.112:0/3898329080 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6de4018730 0x7f6de401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:05.539 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:05.531 7f6dff7fe700 1 --2- 172.21.2.112:0/3898329080 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6de4018730 0x7f6de401ac10 crc :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:05.566 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:05.562 7f6e06108700 1 -- 172.21.2.112:0/3898329080 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e000f0af0 con 0x7f6e000c2290 2021-02-10T11:49:05.568 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:05.565 7f6df7fff700 1 -- 172.21.2.112:0/3898329080 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6df0002240 con 0x7f6e000c2290 2021-02-10T11:49:05.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:05.688 7f6e06108700 1 -- 172.21.2.112:0/3898329080 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6e000f0af0 con 0x7f6e000c2290 2021-02-10T11:49:05.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:05.689 7f6df7fff700 1 -- 172.21.2.112:0/3898329080 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v198) v1 ==== 74+0+14007 (secure 0 0 0) 0x7f6df0002030 con 0x7f6e000c2290 2021-02-10T11:49:05.693 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:49:05.693 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":198,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:49:05.481103","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":43,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":42,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:49:02.450970","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"197","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":43,"pool_name":"cephfs_data","create_time":"2021-02-10 11:49:03.473275","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"198","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":196,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":196,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":196,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":196,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":196,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:49:05.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:05.691 7f6e06108700 1 -- 172.21.2.112:0/3898329080 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6de4018730 msgr2=0x7f6de401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:05.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:05.691 7f6e06108700 1 --2- 172.21.2.112:0/3898329080 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6de4018730 0x7f6de401ac10 crc :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:05.695 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:05.691 7f6e06108700 1 -- 172.21.2.112:0/3898329080 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6e000c2290 msgr2=0x7f6e000c2690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:05.695 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:05.691 7f6e06108700 1 --2- 172.21.2.112:0/3898329080 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6e000c2290 0x7f6e000c2690 secure :-1 s=READY pgs=3253 cs=0 l=1 rev1=1 rx=0x7f6df000b5e0 tx=0x7f6df000b980).stop 2021-02-10T11:49:05.695 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:05.691 7f6e06108700 1 -- 172.21.2.112:0/3898329080 shutdown_connections 2021-02-10T11:49:05.695 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:05.691 7f6e06108700 1 --2- 172.21.2.112:0/3898329080 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6de4018730 0x7f6de401ac10 unknown :-1 s=CLOSED pgs=1467 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:05.695 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:05.691 7f6e06108700 1 --2- 172.21.2.112:0/3898329080 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6e000c2cb0 0x7f6e000bfd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:05.695 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:05.691 7f6e06108700 1 --2- 172.21.2.112:0/3898329080 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6e000c2290 0x7f6e000c2690 unknown :-1 s=CLOSED pgs=3253 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:05.695 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:05.691 7f6e06108700 1 --2- 172.21.2.112:0/3898329080 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6e001a4a60 0x7f6e000c1c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:05.695 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:05.691 7f6e06108700 1 -- 172.21.2.112:0/3898329080 >> 172.21.2.112:0/3898329080 conn(0x7f6e000ebe30 msgr2=0x7f6e000bf200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:05.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:05.691 7f6e06108700 1 -- 172.21.2.112:0/3898329080 shutdown_connections 2021-02-10T11:49:05.696 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:05.691 7f6e06108700 1 -- 172.21.2.112:0/3898329080 wait complete. 2021-02-10T11:49:05.703 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2021-02-10T11:49:06.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.116 7f5b3739a700 1 Processor -- start 2021-02-10T11:49:06.137 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.116 7f5b3739a700 1 -- start start 2021-02-10T11:49:06.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.116 7f5b3739a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5b3010aff0 0x7f5b300cf750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:06.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.116 7f5b3739a700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b300fe1c0 con 0x7f5b300d1870 2021-02-10T11:49:06.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.116 7f5b3739a700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b301b0cb0 con 0x7f5b300d1c70 2021-02-10T11:49:06.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.116 7f5b3739a700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b30109a60 con 0x7f5b3010aff0 2021-02-10T11:49:06.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.117 7f5b351b6700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f5b300d1870 0x7f5b301b04c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:33018/0 (socket says 172.21.2.112:33018) 2021-02-10T11:49:06.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.117 7f5b351b6700 1 -- 172.21.2.112:0/4007000229 learned_addr learned my addr 172.21.2.112:0/4007000229 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:49:06.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.117 7f5b23fff700 1 -- 172.21.2.112:0/4007000229 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2577030157 0 0) 0x7f5b24001180 con 0x7f5b300d1870 2021-02-10T11:49:06.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.117 7f5b23fff700 1 -- 172.21.2.112:0/4007000229 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b1c002fe0 con 0x7f5b300d1870 2021-02-10T11:49:06.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.117 7f5b359b7700 1 --2- 172.21.2.112:0/4007000229 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5b3010aff0 0x7f5b300cf750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:06.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.117 7f5b23fff700 1 -- 172.21.2.112:0/4007000229 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4050796920 0 0) 0x7f5b24000ce0 con 0x7f5b300d1870 2021-02-10T11:49:06.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.117 7f5b23fff700 1 -- 172.21.2.112:0/4007000229 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5b3010aff0 msgr2=0x7f5b300cf750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:49:06.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.117 7f5b23fff700 1 --2- 172.21.2.112:0/4007000229 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5b3010aff0 0x7f5b300cf750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:06.138 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.117 7f5b23fff700 1 -- 172.21.2.112:0/4007000229 >> v1:172.21.2.120:6789/0 conn(0x7f5b300d1c70 legacy=0x7f5b300cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:06.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.117 7f5b23fff700 1 -- 172.21.2.112:0/4007000229 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b3010b3f0 con 0x7f5b300d1870 2021-02-10T11:49:06.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.117 7f5b23fff700 1 -- 172.21.2.112:0/4007000229 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f5b240017f0 con 0x7f5b300d1870 2021-02-10T11:49:06.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.117 7f5b23fff700 1 -- 172.21.2.112:0/4007000229 >> v1:172.21.2.112:6789/0 conn(0x7f5b300d1870 legacy=0x7f5b301b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:06.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.117 7f5b23fff700 1 --2- 172.21.2.112:0/4007000229 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5b1c003a70 0x7f5b1c005f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:06.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.117 7f5b23fff700 1 --2- 172.21.2.112:0/4007000229 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5b1c0067a0 0x7f5b1c008ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:06.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.117 7f5b23fff700 1 --2- 172.21.2.112:0/4007000229 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5b1c0095a0 0x7f5b1c00ba90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:06.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.117 7f5b23fff700 1 -- 172.21.2.112:0/4007000229 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b1c00c2e0 con 0x7f5b1c003a70 2021-02-10T11:49:06.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.117 7f5b23fff700 1 -- 172.21.2.112:0/4007000229 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b1c00c500 con 0x7f5b1c0067a0 2021-02-10T11:49:06.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.117 7f5b23fff700 1 -- 172.21.2.112:0/4007000229 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b1c00c720 con 0x7f5b1c0095a0 2021-02-10T11:49:06.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.117 7f5b351b6700 1 --2- 172.21.2.112:0/4007000229 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5b1c003a70 0x7f5b1c005f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:06.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.118 7f5b349b5700 1 --2- 172.21.2.112:0/4007000229 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5b1c0067a0 0x7f5b1c008ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:06.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.118 7f5b359b7700 1 --2- 172.21.2.112:0/4007000229 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5b1c0095a0 0x7f5b1c00ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:06.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.118 7f5b351b6700 1 -- 172.21.2.112:0/4007000229 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5b1c0095a0 msgr2=0x7f5b1c00ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:49:06.139 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.118 7f5b351b6700 1 --2- 172.21.2.112:0/4007000229 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5b1c0095a0 0x7f5b1c00ba90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:06.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.118 7f5b351b6700 1 -- 172.21.2.112:0/4007000229 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5b1c0067a0 msgr2=0x7f5b1c008ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:49:06.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.118 7f5b351b6700 1 --2- 172.21.2.112:0/4007000229 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5b1c0067a0 0x7f5b1c008ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:06.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.118 7f5b351b6700 1 -- 172.21.2.112:0/4007000229 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b1c00c9c0 con 0x7f5b1c003a70 2021-02-10T11:49:06.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.118 7f5b351b6700 1 --2- 172.21.2.112:0/4007000229 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5b1c003a70 0x7f5b1c005f00 secure :-1 s=READY pgs=3255 cs=0 l=1 rev1=1 rx=0x7f5b240018d0 tx=0x7f5b2404bcc0).ready entity=mon.0 client_cookie=ac9de0c7afabce6b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:06.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.118 7f5b23fff700 1 -- 172.21.2.112:0/4007000229 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5b240014e0 con 0x7f5b1c003a70 2021-02-10T11:49:06.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.118 7f5b23fff700 1 -- 172.21.2.112:0/4007000229 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b24000ce0 con 0x7f5b1c003a70 2021-02-10T11:49:06.140 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.118 7f5b23fff700 1 -- 172.21.2.112:0/4007000229 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5b2404e500 con 0x7f5b1c003a70 2021-02-10T11:49:06.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.118 7f5b3739a700 1 -- 172.21.2.112:0/4007000229 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5b1c003a70 msgr2=0x7f5b1c005f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:06.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.118 7f5b3739a700 1 --2- 172.21.2.112:0/4007000229 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5b1c003a70 0x7f5b1c005f00 secure :-1 s=READY pgs=3255 cs=0 l=1 rev1=1 rx=0x7f5b240018d0 tx=0x7f5b2404bcc0).stop 2021-02-10T11:49:06.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.118 7f5b359b7700 1 -- 172.21.2.112:0/4007000229 reap_dead start 2021-02-10T11:49:06.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.119 7f5b3739a700 1 -- 172.21.2.112:0/4007000229 shutdown_connections 2021-02-10T11:49:06.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.119 7f5b3739a700 1 -- 172.21.2.112:0/4007000229 >> 172.21.2.112:0/4007000229 conn(0x7f5b300ce420 msgr2=0x7f5b300d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:06.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.119 7f5b3739a700 1 -- 172.21.2.112:0/4007000229 shutdown_connections 2021-02-10T11:49:06.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.119 7f5b3739a700 1 -- 172.21.2.112:0/4007000229 wait complete. 2021-02-10T11:49:06.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.134 7f5b3739a700 1 Processor -- start 2021-02-10T11:49:06.141 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.134 7f5b3739a700 1 -- start start 2021-02-10T11:49:06.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.134 7f5b3739a700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5b301a79a0 0x7f5b301a7da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:06.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.134 7f5b3739a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5b300bdb40 0x7f5b300bdf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:06.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.134 7f5b3739a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5b30082ca0 0x7f5b300830a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:06.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.134 7f5b3739a700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b3019d990 con 0x7f5b30082ca0 2021-02-10T11:49:06.143 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.134 7f5b3739a700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b3019dbb0 con 0x7f5b301a79a0 2021-02-10T11:49:06.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.134 7f5b3739a700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b300836c0 con 0x7f5b300bdb40 2021-02-10T11:49:06.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.135 7f5b359b7700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5b30082ca0 0x7f5b300830a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:06.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.135 7f5b359b7700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5b30082ca0 0x7f5b300830a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:33568/0 (socket says 172.21.2.112:33568) 2021-02-10T11:49:06.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.135 7f5b359b7700 1 -- 172.21.2.112:0/3843960601 learned_addr learned my addr 172.21.2.112:0/3843960601 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:49:06.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.135 7f5b359b7700 1 -- 172.21.2.112:0/3843960601 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5b300bdb40 msgr2=0x7f5b300bdf40 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:49:06.144 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.135 7f5b359b7700 1 --2- 172.21.2.112:0/3843960601 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5b300bdb40 0x7f5b300bdf40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:06.145 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.135 7f5b359b7700 1 -- 172.21.2.112:0/3843960601 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5b301a79a0 msgr2=0x7f5b301a7da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:06.145 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.135 7f5b359b7700 1 --2- 172.21.2.112:0/3843960601 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5b301a79a0 0x7f5b301a7da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:06.145 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.135 7f5b359b7700 1 -- 172.21.2.112:0/3843960601 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b300838e0 con 0x7f5b30082ca0 2021-02-10T11:49:06.145 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.135 7f5b359b7700 1 --2- 172.21.2.112:0/3843960601 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5b30082ca0 0x7f5b300830a0 secure :-1 s=READY pgs=3256 cs=0 l=1 rev1=1 rx=0x7f5b280099e0 tx=0x7f5b2800bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:06.145 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.135 7f5b21ffb700 1 -- 172.21.2.112:0/3843960601 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5b28006030 con 0x7f5b30082ca0 2021-02-10T11:49:06.146 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.135 7f5b21ffb700 1 -- 172.21.2.112:0/3843960601 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b280062b0 con 0x7f5b30082ca0 2021-02-10T11:49:06.146 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.135 7f5b3739a700 1 -- 172.21.2.112:0/3843960601 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b300838e0 con 0x7f5b30082ca0 2021-02-10T11:49:06.146 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.135 7f5b3739a700 1 -- 172.21.2.112:0/3843960601 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b30084420 con 0x7f5b30082ca0 2021-02-10T11:49:06.146 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.135 7f5b21ffb700 1 -- 172.21.2.112:0/3843960601 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5b280056b0 con 0x7f5b30082ca0 2021-02-10T11:49:06.146 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.136 7f5b21ffb700 1 -- 172.21.2.112:0/3843960601 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5b280056b0 con 0x7f5b30082ca0 2021-02-10T11:49:06.146 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.136 7f5b21ffb700 1 --2- 172.21.2.112:0/3843960601 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5b10018730 0x7f5b1001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:06.146 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.136 7f5b21ffb700 1 -- 172.21.2.112:0/3843960601 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f5b28032bb0 con 0x7f5b30082ca0 2021-02-10T11:49:06.146 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.136 7f5b351b6700 1 --2- 172.21.2.112:0/3843960601 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5b10018730 0x7f5b1001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:06.146 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.137 7f5b351b6700 1 --2- 172.21.2.112:0/3843960601 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5b10018730 0x7f5b1001abc0 crc :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:06.170 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.166 7f5b3739a700 1 -- 172.21.2.112:0/3843960601 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b30082a70 con 0x7f5b30082ca0 2021-02-10T11:49:06.172 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.169 7f5b21ffb700 1 -- 172.21.2.112:0/3843960601 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5b28006bc0 con 0x7f5b30082ca0 2021-02-10T11:49:06.311 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.307 7f5b3739a700 1 -- 172.21.2.112:0/3843960601 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"} v 0) v1 -- 0x7f5b300838e0 con 0x7f5b30082ca0 2021-02-10T11:49:06.492 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.488 7f5b21ffb700 1 -- 172.21.2.112:0/3843960601 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"}]=0 v174) v1 ==== 118+0+0 (secure 0 0 0) 0x7f5b28006290 con 0x7f5b30082ca0 2021-02-10T11:49:06.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.524 7f5b3739a700 1 -- 172.21.2.112:0/3843960601 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5b10018730 msgr2=0x7f5b1001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:06.528 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.524 7f5b3739a700 1 --2- 172.21.2.112:0/3843960601 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5b10018730 0x7f5b1001abc0 crc :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:06.528 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.524 7f5b3739a700 1 -- 172.21.2.112:0/3843960601 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5b30082ca0 msgr2=0x7f5b300830a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:06.528 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.524 7f5b3739a700 1 --2- 172.21.2.112:0/3843960601 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5b30082ca0 0x7f5b300830a0 secure :-1 s=READY pgs=3256 cs=0 l=1 rev1=1 rx=0x7f5b280099e0 tx=0x7f5b2800bcf0).stop 2021-02-10T11:49:06.528 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.524 7f5b3739a700 1 -- 172.21.2.112:0/3843960601 shutdown_connections 2021-02-10T11:49:06.529 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.524 7f5b3739a700 1 --2- 172.21.2.112:0/3843960601 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5b10018730 0x7f5b1001abc0 unknown :-1 s=CLOSED pgs=1468 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:06.529 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.524 7f5b3739a700 1 --2- 172.21.2.112:0/3843960601 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5b30082ca0 0x7f5b300830a0 unknown :-1 s=CLOSED pgs=3256 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:06.529 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.524 7f5b3739a700 1 --2- 172.21.2.112:0/3843960601 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5b300bdb40 0x7f5b300bdf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:06.529 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.524 7f5b3739a700 1 --2- 172.21.2.112:0/3843960601 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5b301a79a0 0x7f5b301a7da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:06.529 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.524 7f5b3739a700 1 -- 172.21.2.112:0/3843960601 >> 172.21.2.112:0/3843960601 conn(0x7f5b300bef00 msgr2=0x7f5b3009f200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:06.529 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.525 7f5b3739a700 1 -- 172.21.2.112:0/3843960601 shutdown_connections 2021-02-10T11:49:06.529 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.525 7f5b3739a700 1 -- 172.21.2.112:0/3843960601 wait complete. 2021-02-10T11:49:06.536 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:49:06.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.944 7fa5f960e700 1 Processor -- start 2021-02-10T11:49:06.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.944 7fa5f960e700 1 -- start start 2021-02-10T11:49:06.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.944 7fa5f960e700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa5f40d1c70 0x7fa5f40f20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:06.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.944 7fa5f960e700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5f40f0b10 con 0x7fa5f40d0340 2021-02-10T11:49:06.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.944 7fa5f960e700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5f41b0be0 con 0x7fa5f40d1870 2021-02-10T11:49:06.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.944 7fa5f960e700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5f40e5ed0 con 0x7fa5f40d1c70 2021-02-10T11:49:06.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.944 7fa5f2abe700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fa5f40d0340 0x7fa5f40ef640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:33038/0 (socket says 172.21.2.112:33038) 2021-02-10T11:49:06.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.944 7fa5f2abe700 1 -- 172.21.2.112:0/3798552873 learned_addr learned my addr 172.21.2.112:0/3798552873 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:49:06.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.945 7fa5f22bd700 1 -- 172.21.2.112:0/3798552873 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1858836976 0 0) 0x7fa5e8000ce0 con 0x7fa5f40d0340 2021-02-10T11:49:06.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.945 7fa5f22bd700 1 -- 172.21.2.112:0/3798552873 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5dc002fe0 con 0x7fa5f40d0340 2021-02-10T11:49:06.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.945 7fa5f3ac0700 1 --2- 172.21.2.112:0/3798552873 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa5f40d1c70 0x7fa5f40f20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:06.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.945 7fa5f22bd700 1 -- 172.21.2.112:0/3798552873 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1462531173 0 0) 0x7fa5e8001180 con 0x7fa5f40d0340 2021-02-10T11:49:06.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.945 7fa5f22bd700 1 -- 172.21.2.112:0/3798552873 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa5f40d1c70 msgr2=0x7fa5f40f20a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:49:06.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.945 7fa5f22bd700 1 --2- 172.21.2.112:0/3798552873 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa5f40d1c70 0x7fa5f40f20a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:06.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.945 7fa5f22bd700 1 -- 172.21.2.112:0/3798552873 >> v1:172.21.2.120:6789/0 conn(0x7fa5f40d1870 legacy=0x7fa5f41b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:06.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.945 7fa5f22bd700 1 -- 172.21.2.112:0/3798552873 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5f40efe30 con 0x7fa5f40d0340 2021-02-10T11:49:06.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.945 7fa5f22bd700 1 -- 172.21.2.112:0/3798552873 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7fa5e8001680 con 0x7fa5f40d0340 2021-02-10T11:49:06.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.945 7fa5f22bd700 1 -- 172.21.2.112:0/3798552873 >> v1:172.21.2.112:6789/0 conn(0x7fa5f40d0340 legacy=0x7fa5f40ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:06.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.945 7fa5f22bd700 1 --2- 172.21.2.112:0/3798552873 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa5dc003b10 0x7fa5dc005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:06.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.945 7fa5f22bd700 1 --2- 172.21.2.112:0/3798552873 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa5dc006840 0x7fa5dc008d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:06.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.945 7fa5f22bd700 1 --2- 172.21.2.112:0/3798552873 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa5dc009620 0x7fa5dc00bb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:06.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.945 7fa5f22bd700 1 -- 172.21.2.112:0/3798552873 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5dc00c360 con 0x7fa5dc003b10 2021-02-10T11:49:06.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.945 7fa5f22bd700 1 -- 172.21.2.112:0/3798552873 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5dc00c580 con 0x7fa5dc006840 2021-02-10T11:49:06.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.945 7fa5f22bd700 1 -- 172.21.2.112:0/3798552873 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5dc00c7a0 con 0x7fa5dc009620 2021-02-10T11:49:06.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.945 7fa5f3ac0700 1 --2- 172.21.2.112:0/3798552873 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa5f40d1c70 0x7fa5f40f20a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:49:06.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.945 7fa5f32bf700 1 --2- 172.21.2.112:0/3798552873 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa5dc003b10 0x7fa5dc005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:06.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.946 7fa5f2abe700 1 --2- 172.21.2.112:0/3798552873 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa5dc006840 0x7fa5dc008d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:06.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.946 7fa5f3ac0700 1 --2- 172.21.2.112:0/3798552873 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa5dc009620 0x7fa5dc00bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:06.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.946 7fa5f32bf700 1 -- 172.21.2.112:0/3798552873 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa5dc009620 msgr2=0x7fa5dc00bb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:49:06.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.946 7fa5f32bf700 1 --2- 172.21.2.112:0/3798552873 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa5dc009620 0x7fa5dc00bb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:06.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.946 7fa5f32bf700 1 -- 172.21.2.112:0/3798552873 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa5dc006840 msgr2=0x7fa5dc008d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:49:06.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.946 7fa5f32bf700 1 --2- 172.21.2.112:0/3798552873 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa5dc006840 0x7fa5dc008d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:06.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.946 7fa5f32bf700 1 -- 172.21.2.112:0/3798552873 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5dc00ca40 con 0x7fa5dc003b10 2021-02-10T11:49:06.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.946 7fa5f32bf700 1 --2- 172.21.2.112:0/3798552873 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa5dc003b10 0x7fa5dc005fa0 secure :-1 s=READY pgs=3258 cs=0 l=1 rev1=1 rx=0x7fa5e4006ad0 tx=0x7fa5e404db50).ready entity=mon.0 client_cookie=c574603e20e6e6f3 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:06.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.946 7fa5f22bd700 1 -- 172.21.2.112:0/3798552873 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa5e40066d0 con 0x7fa5dc003b10 2021-02-10T11:49:06.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.946 7fa5f22bd700 1 -- 172.21.2.112:0/3798552873 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5e4002280 con 0x7fa5dc003b10 2021-02-10T11:49:06.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.946 7fa5f22bd700 1 -- 172.21.2.112:0/3798552873 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa5e40027a0 con 0x7fa5dc003b10 2021-02-10T11:49:06.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.946 7fa5f960e700 1 -- 172.21.2.112:0/3798552873 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa5dc003b10 msgr2=0x7fa5dc005fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:06.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.946 7fa5f960e700 1 --2- 172.21.2.112:0/3798552873 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa5dc003b10 0x7fa5dc005fa0 secure :-1 s=READY pgs=3258 cs=0 l=1 rev1=1 rx=0x7fa5e4006ad0 tx=0x7fa5e404db50).stop 2021-02-10T11:49:06.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.946 7fa5f3ac0700 1 -- 172.21.2.112:0/3798552873 reap_dead start 2021-02-10T11:49:06.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.947 7fa5f960e700 1 -- 172.21.2.112:0/3798552873 shutdown_connections 2021-02-10T11:49:06.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.947 7fa5f960e700 1 -- 172.21.2.112:0/3798552873 >> 172.21.2.112:0/3798552873 conn(0x7fa5f40ce450 msgr2=0x7fa5f40d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:06.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.947 7fa5f960e700 1 -- 172.21.2.112:0/3798552873 shutdown_connections 2021-02-10T11:49:06.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.947 7fa5f960e700 1 -- 172.21.2.112:0/3798552873 wait complete. 2021-02-10T11:49:06.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.962 7fa5f960e700 1 Processor -- start 2021-02-10T11:49:06.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.962 7fa5f960e700 1 -- start start 2021-02-10T11:49:06.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.962 7fa5f960e700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa5f41a4a60 0x7fa5f40c1c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:06.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.962 7fa5f960e700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa5f40c2290 0x7fa5f40c2690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:06.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.962 7fa5f960e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa5f40c2cb0 0x7fa5f40bfd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:06.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.962 7fa5f960e700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5f40f05b0 con 0x7fa5f40c2cb0 2021-02-10T11:49:06.970 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.962 7fa5f960e700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5f40f07d0 con 0x7fa5f41a4a60 2021-02-10T11:49:06.971 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.962 7fa5f960e700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5f40c0340 con 0x7fa5f40c2290 2021-02-10T11:49:06.971 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.962 7fa5f3ac0700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa5f40c2cb0 0x7fa5f40bfd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:06.971 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.962 7fa5f3ac0700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa5f40c2cb0 0x7fa5f40bfd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:33588/0 (socket says 172.21.2.112:33588) 2021-02-10T11:49:06.971 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.962 7fa5f3ac0700 1 -- 172.21.2.112:0/2504772503 learned_addr learned my addr 172.21.2.112:0/2504772503 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:49:06.971 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.963 7fa5f3ac0700 1 -- 172.21.2.112:0/2504772503 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa5f40c2290 msgr2=0x7fa5f40c2690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:06.971 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.963 7fa5f3ac0700 1 --2- 172.21.2.112:0/2504772503 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa5f40c2290 0x7fa5f40c2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:06.971 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.963 7fa5f3ac0700 1 -- 172.21.2.112:0/2504772503 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa5f41a4a60 msgr2=0x7fa5f40c1c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:06.971 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.963 7fa5f3ac0700 1 --2- 172.21.2.112:0/2504772503 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa5f41a4a60 0x7fa5f40c1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:06.971 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.963 7fa5f3ac0700 1 -- 172.21.2.112:0/2504772503 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5f40c0560 con 0x7fa5f40c2cb0 2021-02-10T11:49:06.971 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.963 7fa5f3ac0700 1 --2- 172.21.2.112:0/2504772503 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa5f40c2cb0 0x7fa5f40bfd20 secure :-1 s=READY pgs=3259 cs=0 l=1 rev1=1 rx=0x7fa5ec0099e0 tx=0x7fa5ec00bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:06.971 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.963 7fa5dbfff700 1 -- 172.21.2.112:0/2504772503 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa5ec006030 con 0x7fa5f40c2cb0 2021-02-10T11:49:06.971 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.963 7fa5dbfff700 1 -- 172.21.2.112:0/2504772503 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5ec0062b0 con 0x7fa5f40c2cb0 2021-02-10T11:49:06.971 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.963 7fa5dbfff700 1 -- 172.21.2.112:0/2504772503 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa5ec005930 con 0x7fa5f40c2cb0 2021-02-10T11:49:06.972 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.963 7fa5f960e700 1 -- 172.21.2.112:0/2504772503 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5f40c0560 con 0x7fa5f40c2cb0 2021-02-10T11:49:06.972 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.963 7fa5f960e700 1 -- 172.21.2.112:0/2504772503 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5f40c0af0 con 0x7fa5f40c2cb0 2021-02-10T11:49:06.972 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.964 7fa5dbfff700 1 -- 172.21.2.112:0/2504772503 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa5ec0056b0 con 0x7fa5f40c2cb0 2021-02-10T11:49:06.972 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.964 7fa5dbfff700 1 --2- 172.21.2.112:0/2504772503 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa5d40186c0 0x7fa5d401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:06.972 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.964 7fa5dbfff700 1 -- 172.21.2.112:0/2504772503 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fa5ec0328f0 con 0x7fa5f40c2cb0 2021-02-10T11:49:06.972 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.964 7fa5f32bf700 1 --2- 172.21.2.112:0/2504772503 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa5d40186c0 0x7fa5d401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:06.972 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.965 7fa5f32bf700 1 --2- 172.21.2.112:0/2504772503 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa5d40186c0 0x7fa5d401abc0 crc :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:06.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.994 7fa5f960e700 1 -- 172.21.2.112:0/2504772503 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5f40f0b10 con 0x7fa5f40c2cb0 2021-02-10T11:49:07.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:06.997 7fa5dbfff700 1 -- 172.21.2.112:0/2504772503 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa5ec005bf0 con 0x7fa5f40c2cb0 2021-02-10T11:49:07.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:07.125 7fa5f960e700 1 -- 172.21.2.112:0/2504772503 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa5f40f0b10 con 0x7fa5f40c2cb0 2021-02-10T11:49:07.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:07.125 7fa5dbfff700 1 -- 172.21.2.112:0/2504772503 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 174 v174) v1 ==== 95+0+2108 (secure 0 0 0) 0x7fa5ec0056b0 con 0x7fa5f40c2cb0 2021-02-10T11:49:07.129 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:49:07.129 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":174,"default_fscid":21,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":13800,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/2202619473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":2202619473},{"type":"v1","addr":"172.21.2.112:6833","nonce":2202619473}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":170}],"filesystems":[{"mdsmap":{"epoch":174,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:49:04.478835","modified":"2021-02-10 11:49:06.484559","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9715},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9715":{"gid":9715,"name":"b","rank":0,"incarnation":171,"state":"up:active","state_seq":3,"addr":"172.21.2.120:6835/3826510568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":3826510568},{"type":"v1","addr":"172.21.2.120:6835","nonce":3826510568}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[43],"metadata_pool":42,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":21}]} 2021-02-10T11:49:07.129 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 174 2021-02-10T11:49:07.131 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:07.127 7fa5f960e700 1 -- 172.21.2.112:0/2504772503 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa5d40186c0 msgr2=0x7fa5d401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:07.131 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:07.127 7fa5f960e700 1 --2- 172.21.2.112:0/2504772503 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa5d40186c0 0x7fa5d401abc0 crc :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:07.131 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:07.127 7fa5f960e700 1 -- 172.21.2.112:0/2504772503 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa5f40c2cb0 msgr2=0x7fa5f40bfd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:07.131 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:07.127 7fa5f960e700 1 --2- 172.21.2.112:0/2504772503 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa5f40c2cb0 0x7fa5f40bfd20 secure :-1 s=READY pgs=3259 cs=0 l=1 rev1=1 rx=0x7fa5ec0099e0 tx=0x7fa5ec00bcf0).stop 2021-02-10T11:49:07.131 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:07.127 7fa5f960e700 1 -- 172.21.2.112:0/2504772503 shutdown_connections 2021-02-10T11:49:07.131 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:07.127 7fa5f960e700 1 --2- 172.21.2.112:0/2504772503 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa5d40186c0 0x7fa5d401abc0 unknown :-1 s=CLOSED pgs=1469 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:07.131 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:07.127 7fa5f960e700 1 --2- 172.21.2.112:0/2504772503 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa5f40c2cb0 0x7fa5f40bfd20 unknown :-1 s=CLOSED pgs=3259 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:07.131 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:07.127 7fa5f960e700 1 --2- 172.21.2.112:0/2504772503 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa5f40c2290 0x7fa5f40c2690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:07.131 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:07.127 7fa5f960e700 1 --2- 172.21.2.112:0/2504772503 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa5f41a4a60 0x7fa5f40c1c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:07.131 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:07.127 7fa5f960e700 1 -- 172.21.2.112:0/2504772503 >> 172.21.2.112:0/2504772503 conn(0x7fa5f40ebe30 msgr2=0x7fa5f40bf200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:07.131 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:07.127 7fa5f960e700 1 -- 172.21.2.112:0/2504772503 shutdown_connections 2021-02-10T11:49:07.132 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:07.127 7fa5f960e700 1 -- 172.21.2.112:0/2504772503 wait complete. 2021-02-10T11:49:07.139 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:49:07.568 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:07.548 7f01c212d700 1 Processor -- start 2021-02-10T11:49:07.568 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:07.548 7f01c212d700 1 -- start start 2021-02-10T11:49:07.568 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:07.548 7f01c212d700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f01bc0d1870 0x7f01bc1b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:07.568 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:07.548 7f01c212d700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01bc0f0b10 con 0x7f01bc0f0650 2021-02-10T11:49:07.568 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:07.548 7f01c212d700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01bc0d0310 con 0x7f01bc1b0a10 2021-02-10T11:49:07.568 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:07.548 7f01c212d700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01bc0d0530 con 0x7f01bc0d1870 2021-02-10T11:49:07.569 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:07.548 7f01bbfff700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f01bc0f0650 0x7f01bc0cfb20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:33058/0 (socket says 172.21.2.112:33058) 2021-02-10T11:49:07.569 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:07.549 7f01bbfff700 1 -- 172.21.2.112:0/487512605 learned_addr learned my addr 172.21.2.112:0/487512605 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:49:07.569 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:07.549 7f01bb7fe700 1 --2- 172.21.2.112:0/487512605 >> v2:172.21.2.120:3300/0 conn(0x7f01bc0d1870 0x7f01bc1b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:07.569 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:07.549 7f01ba7fc700 1 -- 172.21.2.112:0/487512605 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1514696217 0 0) 0x7f01b4001500 con 0x7f01bc0f0650 2021-02-10T11:49:07.569 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:07.549 7f01ba7fc700 1 -- 172.21.2.112:0/487512605 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01a4002fe0 con 0x7f01bc0f0650 2021-02-10T11:49:07.569 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:07.549 7f01bb7fe700 1 -- 172.21.2.112:0/487512605 >> v1:172.21.2.120:6789/0 conn(0x7f01bc1b0a10 legacy=0x7f01bc0cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:07.569 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:07.549 7f01bb7fe700 1 -- 172.21.2.112:0/487512605 >> v1:172.21.2.112:6789/0 conn(0x7f01bc0f0650 legacy=0x7f01bc0cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:07.569 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:07.549 7f01bb7fe700 1 -- 172.21.2.112:0/487512605 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01bc0d1c70 con 0x7f01bc0d1870 2021-02-10T11:49:07.569 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:07.549 7f01bb7fe700 1 --2- 172.21.2.112:0/487512605 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f01bc0d1870 0x7f01bc1b03f0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 rx=0x7f01ac006650 tx=0x7f01ac009a80).ready entity=mon.1 client_cookie=835cda41fc892f5e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:07.569 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:07.550 7f01ba7fc700 1 -- 172.21.2.112:0/487512605 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f01ac00c0c0 con 0x7f01bc0d1870 2021-02-10T11:49:07.569 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:07.550 7f01ba7fc700 1 -- 172.21.2.112:0/487512605 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01ac00c340 con 0x7f01bc0d1870 2021-02-10T11:49:07.569 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:07.550 7f01ba7fc700 1 -- 172.21.2.112:0/487512605 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f01ac00c860 con 0x7f01bc0d1870 2021-02-10T11:49:07.570 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:07.550 7f01c212d700 1 -- 172.21.2.112:0/487512605 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f01bc0d1870 msgr2=0x7f01bc1b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:07.570 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:07.550 7f01c212d700 1 --2- 172.21.2.112:0/487512605 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f01bc0d1870 0x7f01bc1b03f0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 rx=0x7f01ac006650 tx=0x7f01ac009a80).stop 2021-02-10T11:49:07.570 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:07.550 7f01c212d700 1 -- 172.21.2.112:0/487512605 shutdown_connections 2021-02-10T11:49:07.570 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:07.550 7f01c212d700 1 --2- 172.21.2.112:0/487512605 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f01bc0d1870 0x7f01bc1b03f0 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:07.570 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:07.550 7f01c212d700 1 -- 172.21.2.112:0/487512605 >> 172.21.2.112:0/487512605 conn(0x7f01bc0ce450 msgr2=0x7f01bc0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:07.570 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:07.551 7f01c212d700 1 -- 172.21.2.112:0/487512605 shutdown_connections 2021-02-10T11:49:07.570 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:07.551 7f01c212d700 1 -- 172.21.2.112:0/487512605 wait complete. 2021-02-10T11:49:07.572 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:07.565 7f01c212d700 1 Processor -- start 2021-02-10T11:49:07.572 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:07.565 7f01c212d700 1 -- start start 2021-02-10T11:49:07.572 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:07.565 7f01c212d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f01bc0b0ad0 0x7f01bc091f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:07.572 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:07.565 7f01c212d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f01bc1a8850 0x7f01bc1a8c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:07.572 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:07.565 7f01c212d700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f01bc1a6fa0 0x7f01bc1a73a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:07.572 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:07.565 7f01c212d700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01bc1abfe0 con 0x7f01bc1a8850 2021-02-10T11:49:07.572 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:07.565 7f01c212d700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01bc1a9c40 con 0x7f01bc1a6fa0 2021-02-10T11:49:07.572 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:07.565 7f01c212d700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01bc1a9730 con 0x7f01bc0b0ad0 2021-02-10T11:49:07.572 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:07.565 7f01baffd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f01bc1a8850 0x7f01bc1a8c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:07.572 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:07.565 7f01baffd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f01bc1a8850 0x7f01bc1a8c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:33598/0 (socket says 172.21.2.112:33598) 2021-02-10T11:49:07.572 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:07.565 7f01baffd700 1 -- 172.21.2.112:0/3096215513 learned_addr learned my addr 172.21.2.112:0/3096215513 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:49:07.573 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:07.566 7f01baffd700 1 -- 172.21.2.112:0/3096215513 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f01bc0b0ad0 msgr2=0x7f01bc091f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:07.573 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:07.566 7f01baffd700 1 --2- 172.21.2.112:0/3096215513 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f01bc0b0ad0 0x7f01bc091f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:07.573 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:07.566 7f01baffd700 1 -- 172.21.2.112:0/3096215513 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f01bc1a6fa0 msgr2=0x7f01bc1a73a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:07.573 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:07.566 7f01baffd700 1 --2- 172.21.2.112:0/3096215513 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f01bc1a6fa0 0x7f01bc1a73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:07.573 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:07.566 7f01baffd700 1 -- 172.21.2.112:0/3096215513 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01bc1a9270 con 0x7f01bc1a8850 2021-02-10T11:49:07.573 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:07.566 7f01baffd700 1 --2- 172.21.2.112:0/3096215513 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f01bc1a8850 0x7f01bc1a8c50 secure :-1 s=READY pgs=3261 cs=0 l=1 rev1=1 rx=0x7f01b0006ed0 tx=0x7f01b000b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:07.573 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:07.566 7f01a3fff700 1 -- 172.21.2.112:0/3096215513 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f01b0006240 con 0x7f01bc1a8850 2021-02-10T11:49:07.573 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:07.566 7f01a3fff700 1 -- 172.21.2.112:0/3096215513 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01b00009b0 con 0x7f01bc1a8850 2021-02-10T11:49:07.573 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:07.566 7f01a3fff700 1 -- 172.21.2.112:0/3096215513 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f01b0005730 con 0x7f01bc1a8850 2021-02-10T11:49:07.573 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:07.566 7f01c212d700 1 -- 172.21.2.112:0/3096215513 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01bc1a9270 con 0x7f01bc1a8850 2021-02-10T11:49:07.573 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:07.566 7f01c212d700 1 -- 172.21.2.112:0/3096215513 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01bc1a7cb0 con 0x7f01bc1a8850 2021-02-10T11:49:07.573 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:07.567 7f01a3fff700 1 -- 172.21.2.112:0/3096215513 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f01b0006030 con 0x7f01bc1a8850 2021-02-10T11:49:07.573 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:07.567 7f01a3fff700 1 --2- 172.21.2.112:0/3096215513 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f019c0186c0 0x7f019c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:07.573 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:07.567 7f01a3fff700 1 -- 172.21.2.112:0/3096215513 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f01b00327d0 con 0x7f01bc1a8850 2021-02-10T11:49:07.574 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:07.567 7f01bb7fe700 1 --2- 172.21.2.112:0/3096215513 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f019c0186c0 0x7f019c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:07.574 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:07.568 7f01bb7fe700 1 --2- 172.21.2.112:0/3096215513 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f019c0186c0 0x7f019c01abc0 crc :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:07.602 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:07.598 7f01c212d700 1 -- 172.21.2.112:0/3096215513 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01bc1abfe0 con 0x7f01bc1a8850 2021-02-10T11:49:07.604 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:07.601 7f01a3fff700 1 -- 172.21.2.112:0/3096215513 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f01b0000de0 con 0x7f01bc1a8850 2021-02-10T11:49:07.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:07.726 7f01c212d700 1 -- 172.21.2.112:0/3096215513 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f01bc1abfe0 con 0x7f01bc1a8850 2021-02-10T11:49:07.730 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:07.727 7f01a3fff700 1 -- 172.21.2.112:0/3096215513 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v198) v1 ==== 74+0+14007 (secure 0 0 0) 0x7f01b0006030 con 0x7f01bc1a8850 2021-02-10T11:49:07.731 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:49:07.731 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":198,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:49:05.481103","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":43,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":42,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:49:02.450970","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"197","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":43,"pool_name":"cephfs_data","create_time":"2021-02-10 11:49:03.473275","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"198","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":196,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":196,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":196,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":196,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":196,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:49:07.732 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:07.728 7f01c212d700 1 -- 172.21.2.112:0/3096215513 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f019c0186c0 msgr2=0x7f019c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:07.732 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:07.728 7f01c212d700 1 --2- 172.21.2.112:0/3096215513 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f019c0186c0 0x7f019c01abc0 crc :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:07.732 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:07.728 7f01c212d700 1 -- 172.21.2.112:0/3096215513 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f01bc1a8850 msgr2=0x7f01bc1a8c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:07.732 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:07.728 7f01c212d700 1 --2- 172.21.2.112:0/3096215513 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f01bc1a8850 0x7f01bc1a8c50 secure :-1 s=READY pgs=3261 cs=0 l=1 rev1=1 rx=0x7f01b0006ed0 tx=0x7f01b000b960).stop 2021-02-10T11:49:07.733 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:07.729 7f01c212d700 1 -- 172.21.2.112:0/3096215513 shutdown_connections 2021-02-10T11:49:07.733 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:07.729 7f01c212d700 1 --2- 172.21.2.112:0/3096215513 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f019c0186c0 0x7f019c01abc0 unknown :-1 s=CLOSED pgs=1470 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:07.733 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:07.729 7f01c212d700 1 --2- 172.21.2.112:0/3096215513 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f01bc1a6fa0 0x7f01bc1a73a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:07.733 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:07.729 7f01c212d700 1 --2- 172.21.2.112:0/3096215513 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f01bc1a8850 0x7f01bc1a8c50 unknown :-1 s=CLOSED pgs=3261 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:07.733 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:07.729 7f01c212d700 1 --2- 172.21.2.112:0/3096215513 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f01bc0b0ad0 0x7f01bc091f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:07.733 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:07.729 7f01c212d700 1 -- 172.21.2.112:0/3096215513 >> 172.21.2.112:0/3096215513 conn(0x7f01bc19f8f0 msgr2=0x7f01bc1a5f80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:07.733 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:07.729 7f01c212d700 1 -- 172.21.2.112:0/3096215513 shutdown_connections 2021-02-10T11:49:07.733 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:07.729 7f01c212d700 1 -- 172.21.2.112:0/3096215513 wait complete. 2021-02-10T11:49:07.741 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:49:08.184 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.163 7fc41455f700 1 Processor -- start 2021-02-10T11:49:08.184 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.163 7fc41455f700 1 -- start start 2021-02-10T11:49:08.184 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.164 7fc41455f700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fc40c0d1880 0x7fc40c1b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:08.184 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.164 7fc41455f700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc40c1b0af0 0x7fc40c108950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:08.184 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.164 7fc41455f700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc40c0fe1d0 con 0x7fc40c0d1c80 2021-02-10T11:49:08.185 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.164 7fc41455f700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc40c109a70 con 0x7fc40c0d1880 2021-02-10T11:49:08.185 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.164 7fc41455f700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc40c109760 con 0x7fc40c1b0af0 2021-02-10T11:49:08.185 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.164 7fc412b7c700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7fc40c0d1c80 0x7fc40c108f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:54858/0 (socket says 172.21.2.112:54858) 2021-02-10T11:49:08.185 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.164 7fc412b7c700 1 -- 172.21.2.112:0/316626333 learned_addr learned my addr 172.21.2.112:0/316626333 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:49:08.185 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.164 7fc41237b700 1 --2- 172.21.2.112:0/316626333 >> v2:172.21.2.120:3300/0 conn(0x7fc40c0d1880 0x7fc40c1b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:08.185 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.164 7fc411b7a700 1 --2- 172.21.2.112:0/316626333 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc40c1b0af0 0x7fc40c108950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:08.185 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.165 7fc411379700 1 -- 172.21.2.112:0/316626333 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3673392938 0 0) 0x7fc404001510 con 0x7fc40c0d1c80 2021-02-10T11:49:08.185 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.165 7fc411379700 1 -- 172.21.2.112:0/316626333 --> v1:172.21.2.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3f8002fe0 con 0x7fc40c0d1c80 2021-02-10T11:49:08.185 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.165 7fc41237b700 1 -- 172.21.2.112:0/316626333 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc40c1b0af0 msgr2=0x7fc40c108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:49:08.185 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.165 7fc41237b700 1 --2- 172.21.2.112:0/316626333 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc40c1b0af0 0x7fc40c108950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:08.185 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.165 7fc41237b700 1 -- 172.21.2.112:0/316626333 >> v1:172.21.2.120:6789/0 conn(0x7fc40c0d1c80 legacy=0x7fc40c108f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:08.185 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.165 7fc41237b700 1 -- 172.21.2.112:0/316626333 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc40c088070 con 0x7fc40c0d1880 2021-02-10T11:49:08.185 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.165 7fc411b7a700 1 --2- 172.21.2.112:0/316626333 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc40c1b0af0 0x7fc40c108950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:49:08.186 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.165 7fc41237b700 1 --2- 172.21.2.112:0/316626333 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc40c0d1880 0x7fc40c1b04d0 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 rx=0x7fc3fc006650 tx=0x7fc3fc009a80).ready entity=mon.1 client_cookie=56a6fa167e3f2d7c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:08.186 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.166 7fc411379700 1 -- 172.21.2.112:0/316626333 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc3fc0009b0 con 0x7fc40c0d1880 2021-02-10T11:49:08.186 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.166 7fc411379700 1 -- 172.21.2.112:0/316626333 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3fc00c0c0 con 0x7fc40c0d1880 2021-02-10T11:49:08.186 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.166 7fc411379700 1 -- 172.21.2.112:0/316626333 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc3fc00c5e0 con 0x7fc40c0d1880 2021-02-10T11:49:08.186 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.166 7fc41455f700 1 -- 172.21.2.112:0/316626333 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc40c0d1880 msgr2=0x7fc40c1b04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:08.186 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.166 7fc41455f700 1 --2- 172.21.2.112:0/316626333 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc40c0d1880 0x7fc40c1b04d0 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 rx=0x7fc3fc006650 tx=0x7fc3fc009a80).stop 2021-02-10T11:49:08.186 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.166 7fc41455f700 1 -- 172.21.2.112:0/316626333 shutdown_connections 2021-02-10T11:49:08.186 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.166 7fc41455f700 1 --2- 172.21.2.112:0/316626333 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc40c1b0af0 0x7fc40c108950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:08.186 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.166 7fc41455f700 1 --2- 172.21.2.112:0/316626333 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc40c0d1880 0x7fc40c1b04d0 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:08.186 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.166 7fc41455f700 1 -- 172.21.2.112:0/316626333 >> 172.21.2.112:0/316626333 conn(0x7fc40c0ce430 msgr2=0x7fc40c0d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:08.186 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.166 7fc41455f700 1 -- 172.21.2.112:0/316626333 shutdown_connections 2021-02-10T11:49:08.186 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.166 7fc41455f700 1 -- 172.21.2.112:0/316626333 wait complete. 2021-02-10T11:49:08.186 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.181 7fc41455f700 1 Processor -- start 2021-02-10T11:49:08.186 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.181 7fc41455f700 1 -- start start 2021-02-10T11:49:08.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.181 7fc41455f700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc40c090e50 0x7fc40c1a44b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:08.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.181 7fc41455f700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc40c1a4ad0 0x7fc40c083200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:08.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.181 7fc41455f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc40c083820 0x7fc40c083c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:08.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.181 7fc41455f700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc40c091250 con 0x7fc40c083820 2021-02-10T11:49:08.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.181 7fc41455f700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc40c084a40 con 0x7fc40c1a4ad0 2021-02-10T11:49:08.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.181 7fc41455f700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc40c084c60 con 0x7fc40c090e50 2021-02-10T11:49:08.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.181 7fc412b7c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc40c083820 0x7fc40c083c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:08.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.181 7fc412b7c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc40c083820 0x7fc40c083c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:33610/0 (socket says 172.21.2.112:33610) 2021-02-10T11:49:08.189 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.181 7fc412b7c700 1 -- 172.21.2.112:0/1302001223 learned_addr learned my addr 172.21.2.112:0/1302001223 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:49:08.189 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.182 7fc412b7c700 1 -- 172.21.2.112:0/1302001223 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc40c090e50 msgr2=0x7fc40c1a44b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:08.189 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.182 7fc412b7c700 1 --2- 172.21.2.112:0/1302001223 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc40c090e50 0x7fc40c1a44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:08.189 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.182 7fc412b7c700 1 -- 172.21.2.112:0/1302001223 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc40c1a4ad0 msgr2=0x7fc40c083200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:08.189 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.182 7fc412b7c700 1 --2- 172.21.2.112:0/1302001223 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc40c1a4ad0 0x7fc40c083200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:08.189 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.182 7fc412b7c700 1 -- 172.21.2.112:0/1302001223 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc40c084e80 con 0x7fc40c083820 2021-02-10T11:49:08.189 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.182 7fc412b7c700 1 --2- 172.21.2.112:0/1302001223 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc40c083820 0x7fc40c083c20 secure :-1 s=READY pgs=3262 cs=0 l=1 rev1=1 rx=0x7fc404001ba0 tx=0x7fc40400ac80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:08.189 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.182 7fc402ffd700 1 -- 172.21.2.112:0/1302001223 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc4040070d0 con 0x7fc40c083820 2021-02-10T11:49:08.189 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.182 7fc402ffd700 1 -- 172.21.2.112:0/1302001223 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc404007390 con 0x7fc40c083820 2021-02-10T11:49:08.189 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.182 7fc41455f700 1 -- 172.21.2.112:0/1302001223 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc40c084e80 con 0x7fc40c083820 2021-02-10T11:49:08.189 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.182 7fc41455f700 1 -- 172.21.2.112:0/1302001223 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc40c085410 con 0x7fc40c083820 2021-02-10T11:49:08.189 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.182 7fc402ffd700 1 -- 172.21.2.112:0/1302001223 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc404004b30 con 0x7fc40c083820 2021-02-10T11:49:08.189 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.183 7fc402ffd700 1 -- 172.21.2.112:0/1302001223 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc4040046a0 con 0x7fc40c083820 2021-02-10T11:49:08.189 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.183 7fc402ffd700 1 --2- 172.21.2.112:0/1302001223 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc3f4018730 0x7fc3f401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:08.190 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.183 7fc402ffd700 1 -- 172.21.2.112:0/1302001223 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fc404032b50 con 0x7fc40c083820 2021-02-10T11:49:08.190 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.183 7fc41237b700 1 --2- 172.21.2.112:0/1302001223 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc3f4018730 0x7fc3f401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:08.190 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.184 7fc41237b700 1 --2- 172.21.2.112:0/1302001223 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc3f4018730 0x7fc3f401ac10 crc :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:08.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.213 7fc41455f700 1 -- 172.21.2.112:0/1302001223 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc40c084290 con 0x7fc40c083820 2021-02-10T11:49:08.219 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.216 7fc402ffd700 1 -- 172.21.2.112:0/1302001223 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc4040046a0 con 0x7fc40c083820 2021-02-10T11:49:08.351 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.347 7fc41455f700 1 -- 172.21.2.112:0/1302001223 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fc40c084290 con 0x7fc40c083820 2021-02-10T11:49:08.354 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.350 7fc402ffd700 1 -- 172.21.2.112:0/1302001223 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v120) v1 ==== 184+0+0 (secure 0 0 0) 0x7fc404007390 con 0x7fc40c083820 2021-02-10T11:49:08.358 INFO:teuthology.orchestra.run.gibba012.stderr:updated caps for client.0 2021-02-10T11:49:08.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.356 7fc41455f700 1 -- 172.21.2.112:0/1302001223 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc3f4018730 msgr2=0x7fc3f401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:08.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.356 7fc41455f700 1 --2- 172.21.2.112:0/1302001223 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc3f4018730 0x7fc3f401ac10 crc :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:08.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.356 7fc41455f700 1 -- 172.21.2.112:0/1302001223 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc40c083820 msgr2=0x7fc40c083c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:08.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.356 7fc41455f700 1 --2- 172.21.2.112:0/1302001223 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc40c083820 0x7fc40c083c20 secure :-1 s=READY pgs=3262 cs=0 l=1 rev1=1 rx=0x7fc404001ba0 tx=0x7fc40400ac80).stop 2021-02-10T11:49:08.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.356 7fc41455f700 1 -- 172.21.2.112:0/1302001223 shutdown_connections 2021-02-10T11:49:08.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.356 7fc41455f700 1 --2- 172.21.2.112:0/1302001223 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fc3f4018730 0x7fc3f401ac10 unknown :-1 s=CLOSED pgs=1471 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:08.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.356 7fc41455f700 1 --2- 172.21.2.112:0/1302001223 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fc40c1a4ad0 0x7fc40c083200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:08.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.356 7fc41455f700 1 --2- 172.21.2.112:0/1302001223 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fc40c083820 0x7fc40c083c20 unknown :-1 s=CLOSED pgs=3262 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:08.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.356 7fc41455f700 1 --2- 172.21.2.112:0/1302001223 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fc40c090e50 0x7fc40c1a44b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:08.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.356 7fc41455f700 1 -- 172.21.2.112:0/1302001223 >> 172.21.2.112:0/1302001223 conn(0x7fc40c0a1100 msgr2=0x7fc40c0b5080 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:08.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.356 7fc41455f700 1 -- 172.21.2.112:0/1302001223 shutdown_connections 2021-02-10T11:49:08.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.356 7fc41455f700 1 -- 172.21.2.112:0/1302001223 wait complete. 2021-02-10T11:49:08.368 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:49:08.809 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.789 7f89367eb700 1 Processor -- start 2021-02-10T11:49:08.809 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.789 7f89367eb700 1 -- start start 2021-02-10T11:49:08.809 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.789 7f89367eb700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f89300d1880 0x7f89301b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:08.809 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.789 7f89367eb700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f89301b0af0 0x7f8930108950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:08.809 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.789 7f89367eb700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89300fe1d0 con 0x7f89300d1c80 2021-02-10T11:49:08.809 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.789 7f89367eb700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8930109a70 con 0x7f89301b0af0 2021-02-10T11:49:08.810 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.789 7f89367eb700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8930109760 con 0x7f89300d1880 2021-02-10T11:49:08.810 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.790 7f892f7fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f89301b0af0 0x7f8930108950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:08.810 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.790 7f892f7fe700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f89301b0af0 0x7f8930108950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:33616/0 (socket says 172.21.2.112:33616) 2021-02-10T11:49:08.810 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.790 7f892f7fe700 1 -- 172.21.2.112:0/405275556 learned_addr learned my addr 172.21.2.112:0/405275556 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:49:08.810 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.790 7f892ffff700 1 --2- 172.21.2.112:0/405275556 >> v2:172.21.2.120:3300/0 conn(0x7f89300d1880 0x7f89301b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:08.810 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.790 7f892f7fe700 1 -- 172.21.2.112:0/405275556 >> v2:172.21.2.120:3300/0 conn(0x7f89300d1880 msgr2=0x7f89301b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:49:08.810 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.790 7f892f7fe700 1 --2- 172.21.2.112:0/405275556 >> v2:172.21.2.120:3300/0 conn(0x7f89300d1880 0x7f89301b04d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:08.810 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.790 7f892f7fe700 1 -- 172.21.2.112:0/405275556 >> v1:172.21.2.112:6789/0 conn(0x7f89300d1c80 legacy=0x7f8930108f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:08.810 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.790 7f892f7fe700 1 -- 172.21.2.112:0/405275556 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8930088070 con 0x7f89301b0af0 2021-02-10T11:49:08.810 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.790 7f892ffff700 1 --2- 172.21.2.112:0/405275556 >> v2:172.21.2.120:3300/0 conn(0x7f89300d1880 0x7f89301b04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:49:08.810 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.790 7f892f7fe700 1 --2- 172.21.2.112:0/405275556 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f89301b0af0 0x7f8930108950 secure :-1 s=READY pgs=3264 cs=0 l=1 rev1=1 rx=0x7f8924006700 tx=0x7f8924009b80).ready entity=mon.0 client_cookie=8cae865a79358182 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:08.810 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.790 7f892effd700 1 -- 172.21.2.112:0/405275556 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f892400c0c0 con 0x7f89301b0af0 2021-02-10T11:49:08.810 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.790 7f892effd700 1 -- 172.21.2.112:0/405275556 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f892400c440 con 0x7f89301b0af0 2021-02-10T11:49:08.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.790 7f892effd700 1 -- 172.21.2.112:0/405275556 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f892400c960 con 0x7f89301b0af0 2021-02-10T11:49:08.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.790 7f89367eb700 1 -- 172.21.2.112:0/405275556 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f89301b0af0 msgr2=0x7f8930108950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:08.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.790 7f89367eb700 1 --2- 172.21.2.112:0/405275556 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f89301b0af0 0x7f8930108950 secure :-1 s=READY pgs=3264 cs=0 l=1 rev1=1 rx=0x7f8924006700 tx=0x7f8924009b80).stop 2021-02-10T11:49:08.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.791 7f89367eb700 1 -- 172.21.2.112:0/405275556 shutdown_connections 2021-02-10T11:49:08.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.791 7f89367eb700 1 --2- 172.21.2.112:0/405275556 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f89301b0af0 0x7f8930108950 unknown :-1 s=CLOSED pgs=3264 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:08.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.791 7f89367eb700 1 --2- 172.21.2.112:0/405275556 >> v2:172.21.2.120:3300/0 conn(0x7f89300d1880 0x7f89301b04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:08.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.791 7f89367eb700 1 -- 172.21.2.112:0/405275556 >> 172.21.2.112:0/405275556 conn(0x7f89300ce430 msgr2=0x7f89300d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:08.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.791 7f89367eb700 1 -- 172.21.2.112:0/405275556 shutdown_connections 2021-02-10T11:49:08.811 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.791 7f89367eb700 1 -- 172.21.2.112:0/405275556 wait complete. 2021-02-10T11:49:08.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.806 7f89367eb700 1 Processor -- start 2021-02-10T11:49:08.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.806 7f89367eb700 1 -- start start 2021-02-10T11:49:08.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.806 7f89367eb700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f89300aedc0 0x7f8930095430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:08.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.806 7f89367eb700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f893009ac10 0x7f8930095a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:08.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.806 7f89367eb700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f89301af300 0x7f8930099030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:08.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.806 7f89367eb700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89301aed80 con 0x7f89301af300 2021-02-10T11:49:08.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.806 7f89367eb700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8930086a60 con 0x7f893009ac10 2021-02-10T11:49:08.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.806 7f89367eb700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8930096070 con 0x7f89300aedc0 2021-02-10T11:49:08.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.806 7f8934e08700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f89301af300 0x7f8930099030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:08.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.806 7f8934e08700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f89301af300 0x7f8930099030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:33622/0 (socket says 172.21.2.112:33622) 2021-02-10T11:49:08.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.806 7f8934e08700 1 -- 172.21.2.112:0/2676370310 learned_addr learned my addr 172.21.2.112:0/2676370310 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:49:08.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.807 7f892f7fe700 1 --2- 172.21.2.112:0/2676370310 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f893009ac10 0x7f8930095a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:08.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.807 7f8934e08700 1 -- 172.21.2.112:0/2676370310 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f89300aedc0 msgr2=0x7f8930095430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:08.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.807 7f8934e08700 1 --2- 172.21.2.112:0/2676370310 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f89300aedc0 0x7f8930095430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:08.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.807 7f8934e08700 1 -- 172.21.2.112:0/2676370310 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f893009ac10 msgr2=0x7f8930095a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:08.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.807 7f8934e08700 1 --2- 172.21.2.112:0/2676370310 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f893009ac10 0x7f8930095a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:08.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.807 7f8934e08700 1 -- 172.21.2.112:0/2676370310 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8930099650 con 0x7f89301af300 2021-02-10T11:49:08.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.807 7f8934e08700 1 --2- 172.21.2.112:0/2676370310 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f89301af300 0x7f8930099030 secure :-1 s=READY pgs=3265 cs=0 l=1 rev1=1 rx=0x7f8928009980 tx=0x7f892800bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:08.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.807 7f892cff9700 1 -- 172.21.2.112:0/2676370310 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8928005870 con 0x7f89301af300 2021-02-10T11:49:08.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.807 7f892cff9700 1 -- 172.21.2.112:0/2676370310 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8928005a70 con 0x7f89301af300 2021-02-10T11:49:08.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.807 7f892cff9700 1 -- 172.21.2.112:0/2676370310 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8928006030 con 0x7f89301af300 2021-02-10T11:49:08.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.807 7f89367eb700 1 -- 172.21.2.112:0/2676370310 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8930099650 con 0x7f89301af300 2021-02-10T11:49:08.815 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.807 7f89367eb700 1 -- 172.21.2.112:0/2676370310 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8930099be0 con 0x7f89301af300 2021-02-10T11:49:08.815 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.808 7f892cff9700 1 -- 172.21.2.112:0/2676370310 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8928006030 con 0x7f89301af300 2021-02-10T11:49:08.815 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.808 7f892cff9700 1 --2- 172.21.2.112:0/2676370310 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f89100186c0 0x7f891001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:08.815 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.808 7f892cff9700 1 -- 172.21.2.112:0/2676370310 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f8928032980 con 0x7f89301af300 2021-02-10T11:49:08.815 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.808 7f892ffff700 1 --2- 172.21.2.112:0/2676370310 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f89100186c0 0x7f891001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:08.815 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.809 7f892ffff700 1 --2- 172.21.2.112:0/2676370310 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f89100186c0 0x7f891001abc0 crc :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:08.842 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.839 7f89367eb700 1 -- 172.21.2.112:0/2676370310 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89300fe1d0 con 0x7f89301af300 2021-02-10T11:49:08.845 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.841 7f892cff9700 1 -- 172.21.2.112:0/2676370310 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8928006240 con 0x7f89301af300 2021-02-10T11:49:08.975 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.971 7f89367eb700 1 -- 172.21.2.112:0/2676370310 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f89300fe1d0 con 0x7f89301af300 2021-02-10T11:49:08.979 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.975 7f892cff9700 1 -- 172.21.2.112:0/2676370310 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v121) v1 ==== 184+0+0 (secure 0 0 0) 0x7f8928005df0 con 0x7f89301af300 2021-02-10T11:49:08.982 INFO:teuthology.orchestra.run.gibba012.stderr:updated caps for client.1 2021-02-10T11:49:08.983 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.980 7f89367eb700 1 -- 172.21.2.112:0/2676370310 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f89100186c0 msgr2=0x7f891001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:08.983 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.980 7f89367eb700 1 --2- 172.21.2.112:0/2676370310 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f89100186c0 0x7f891001abc0 crc :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:08.984 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.980 7f89367eb700 1 -- 172.21.2.112:0/2676370310 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f89301af300 msgr2=0x7f8930099030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:08.984 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.980 7f89367eb700 1 --2- 172.21.2.112:0/2676370310 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f89301af300 0x7f8930099030 secure :-1 s=READY pgs=3265 cs=0 l=1 rev1=1 rx=0x7f8928009980 tx=0x7f892800bcd0).stop 2021-02-10T11:49:08.984 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.980 7f89367eb700 1 -- 172.21.2.112:0/2676370310 shutdown_connections 2021-02-10T11:49:08.984 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.980 7f89367eb700 1 --2- 172.21.2.112:0/2676370310 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f89100186c0 0x7f891001abc0 unknown :-1 s=CLOSED pgs=1472 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:08.984 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.980 7f89367eb700 1 --2- 172.21.2.112:0/2676370310 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f893009ac10 0x7f8930095a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:08.984 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.980 7f89367eb700 1 --2- 172.21.2.112:0/2676370310 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f89301af300 0x7f8930099030 unknown :-1 s=CLOSED pgs=3265 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:08.984 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.980 7f89367eb700 1 --2- 172.21.2.112:0/2676370310 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f89300aedc0 0x7f8930095430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:08.984 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.980 7f89367eb700 1 -- 172.21.2.112:0/2676370310 >> 172.21.2.112:0/2676370310 conn(0x7f89300845b0 msgr2=0x7f89301ac340 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:08.984 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.980 7f89367eb700 1 -- 172.21.2.112:0/2676370310 shutdown_connections 2021-02-10T11:49:08.985 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:08.980 7f89367eb700 1 -- 172.21.2.112:0/2676370310 wait complete. 2021-02-10T11:49:08.992 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.2 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:49:09.424 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:09.404 7faebc842700 1 Processor -- start 2021-02-10T11:49:09.425 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:09.404 7faebc842700 1 -- start start 2021-02-10T11:49:09.425 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:09.404 7faebc842700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7faeb80d1c80 0x7faeb80cef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:09.425 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:09.404 7faebc842700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7faeb80cf590 0x7faeb80cf990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:09.425 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:09.404 7faebc842700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faeb80fe1d0 con 0x7faeb80d1880 2021-02-10T11:49:09.425 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:09.404 7faebc842700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faeb81b0cc0 con 0x7faeb80cf590 2021-02-10T11:49:09.425 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:09.404 7faebc842700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faeb8109a70 con 0x7faeb80d1c80 2021-02-10T11:49:09.425 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:09.405 7faeb6e1d700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7faeb80cf590 0x7faeb80cf990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:09.425 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:09.405 7faeb5e1b700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7faeb80d1c80 0x7faeb80cef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:09.425 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:09.405 7faeb661c700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7faeb80d1880 0x7faeb81b04d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:54880/0 (socket says 172.21.2.112:54880) 2021-02-10T11:49:09.425 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:09.405 7faeb661c700 1 -- 172.21.2.112:0/493163049 learned_addr learned my addr 172.21.2.112:0/493163049 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:49:09.425 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:09.406 7faeb561a700 1 -- 172.21.2.112:0/493163049 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1029368161 0 0) 0x7faea0001190 con 0x7faeb80d1880 2021-02-10T11:49:09.425 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:09.406 7faeb561a700 1 -- 172.21.2.112:0/493163049 --> v1:172.21.2.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faea4002fe0 con 0x7faeb80d1880 2021-02-10T11:49:09.426 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:09.406 7faeb6e1d700 1 -- 172.21.2.112:0/493163049 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7faeb80d1c80 msgr2=0x7faeb80cef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:49:09.426 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:09.406 7faeb6e1d700 1 --2- 172.21.2.112:0/493163049 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7faeb80d1c80 0x7faeb80cef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:09.426 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:09.406 7faeb6e1d700 1 -- 172.21.2.112:0/493163049 >> v1:172.21.2.120:6789/0 conn(0x7faeb80d1880 legacy=0x7faeb81b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:09.426 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:09.406 7faeb6e1d700 1 -- 172.21.2.112:0/493163049 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faeb8088070 con 0x7faeb80cf590 2021-02-10T11:49:09.426 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:09.406 7faeb5e1b700 1 --2- 172.21.2.112:0/493163049 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7faeb80d1c80 0x7faeb80cef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:49:09.426 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:09.406 7faeb6e1d700 1 --2- 172.21.2.112:0/493163049 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7faeb80cf590 0x7faeb80cf990 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 rx=0x7faeb0006e50 tx=0x7faeb0009e70).ready entity=mon.1 client_cookie=7efc3a87c6de4dd0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:09.426 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:09.406 7faeb561a700 1 -- 172.21.2.112:0/493163049 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faeb00066f0 con 0x7faeb80cf590 2021-02-10T11:49:09.426 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:09.406 7faeb561a700 1 -- 172.21.2.112:0/493163049 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faeb0003030 con 0x7faeb80cf590 2021-02-10T11:49:09.426 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:09.406 7faebc842700 1 -- 172.21.2.112:0/493163049 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7faeb80cf590 msgr2=0x7faeb80cf990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:09.426 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:09.406 7faebc842700 1 --2- 172.21.2.112:0/493163049 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7faeb80cf590 0x7faeb80cf990 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 rx=0x7faeb0006e50 tx=0x7faeb0009e70).stop 2021-02-10T11:49:09.426 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:09.406 7faeb561a700 1 -- 172.21.2.112:0/493163049 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faeb000b700 con 0x7faeb80cf590 2021-02-10T11:49:09.426 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:09.407 7faebc842700 1 -- 172.21.2.112:0/493163049 shutdown_connections 2021-02-10T11:49:09.426 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:09.407 7faebc842700 1 --2- 172.21.2.112:0/493163049 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7faeb80cf590 0x7faeb80cf990 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:09.427 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:09.407 7faebc842700 1 --2- 172.21.2.112:0/493163049 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7faeb80d1c80 0x7faeb80cef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:09.427 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:09.407 7faebc842700 1 -- 172.21.2.112:0/493163049 >> 172.21.2.112:0/493163049 conn(0x7faeb80ce430 msgr2=0x7faeb80d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:09.427 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:09.407 7faebc842700 1 -- 172.21.2.112:0/493163049 shutdown_connections 2021-02-10T11:49:09.427 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:09.407 7faebc842700 1 -- 172.21.2.112:0/493163049 wait complete. 2021-02-10T11:49:09.429 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:09.421 7faebc842700 1 Processor -- start 2021-02-10T11:49:09.429 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:09.421 7faebc842700 1 -- start start 2021-02-10T11:49:09.429 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:09.422 7faebc842700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7faeb8095600 0x7faeb8095a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:09.429 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:09.422 7faebc842700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7faeb8096020 0x7faeb8096420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:09.429 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:09.422 7faebc842700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faeb8092a10 0x7faeb8092e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:09.430 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:09.422 7faebc842700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faeb8094100 con 0x7faeb8092a10 2021-02-10T11:49:09.430 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:09.422 7faebc842700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faeb819e9e0 con 0x7faeb8095600 2021-02-10T11:49:09.430 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:09.422 7faebc842700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faeb8093430 con 0x7faeb8096020 2021-02-10T11:49:09.430 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:09.422 7faeb661c700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7faeb8095600 0x7faeb8095a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:09.430 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:09.422 7faeb6e1d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faeb8092a10 0x7faeb8092e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:09.431 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:09.422 7faeb661c700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7faeb8095600 0x7faeb8095a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3300/0 says I am v2:172.21.2.112:35912/0 (socket says 172.21.2.112:35912) 2021-02-10T11:49:09.431 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:09.422 7faeb661c700 1 -- 172.21.2.112:0/191347495 learned_addr learned my addr 172.21.2.112:0/191347495 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:49:09.431 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:09.422 7faeb5e1b700 1 --2- 172.21.2.112:0/191347495 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7faeb8096020 0x7faeb8096420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:09.431 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:09.422 7faeb661c700 1 -- 172.21.2.112:0/191347495 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7faeb8096020 msgr2=0x7faeb8096420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:09.431 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:09.422 7faeb661c700 1 --2- 172.21.2.112:0/191347495 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7faeb8096020 0x7faeb8096420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:09.431 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:09.422 7faeb661c700 1 -- 172.21.2.112:0/191347495 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faeb8092a10 msgr2=0x7faeb8092e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:09.431 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:09.422 7faeb661c700 1 --2- 172.21.2.112:0/191347495 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faeb8092a10 0x7faeb8092e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:09.431 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:09.422 7faeb661c700 1 -- 172.21.2.112:0/191347495 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faeb8093650 con 0x7faeb8095600 2021-02-10T11:49:09.431 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:09.423 7faeb661c700 1 --2- 172.21.2.112:0/191347495 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7faeb8095600 0x7faeb8095a00 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 rx=0x7faea00078d0 tx=0x7faea000a950).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:09.431 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:09.423 7faeaeffd700 1 -- 172.21.2.112:0/191347495 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faea000ad40 con 0x7faeb8095600 2021-02-10T11:49:09.431 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:09.423 7faeaeffd700 1 -- 172.21.2.112:0/191347495 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faea0007480 con 0x7faeb8095600 2021-02-10T11:49:09.431 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:09.423 7faebc842700 1 -- 172.21.2.112:0/191347495 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faeb8093650 con 0x7faeb8095600 2021-02-10T11:49:09.431 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:09.423 7faebc842700 1 -- 172.21.2.112:0/191347495 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faeb8093650 con 0x7faeb8095600 2021-02-10T11:49:09.432 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:09.423 7faeaeffd700 1 -- 172.21.2.112:0/191347495 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faea00045a0 con 0x7faeb8095600 2021-02-10T11:49:09.432 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:09.423 7faeaeffd700 1 -- 172.21.2.112:0/191347495 <== mon.1 v2:172.21.2.120:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7faea0000c10 con 0x7faeb8095600 2021-02-10T11:49:09.432 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:09.424 7faeaeffd700 1 --2- 172.21.2.112:0/191347495 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fae9c018730 0x7fae9c01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:09.432 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:09.424 7faeaeffd700 1 -- 172.21.2.112:0/191347495 <== mon.1 v2:172.21.2.120:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 5573+0+0 (secure 0 0 0) 0x7faea00327d0 con 0x7faeb8095600 2021-02-10T11:49:09.432 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:09.424 7faeb5e1b700 1 --2- 172.21.2.112:0/191347495 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fae9c018730 0x7fae9c01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:09.432 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:09.424 7faeb5e1b700 1 --2- 172.21.2.112:0/191347495 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fae9c018730 0x7fae9c01ac10 crc :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:09.458 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:09.454 7faebc842700 1 -- 172.21.2.112:0/191347495 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faeb819e9e0 con 0x7faeb8095600 2021-02-10T11:49:09.461 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:09.457 7faeaeffd700 1 -- 172.21.2.112:0/191347495 <== mon.1 v2:172.21.2.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7faea0007050 con 0x7faeb8095600 2021-02-10T11:49:09.594 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:09.590 7faebc842700 1 -- 172.21.2.112:0/191347495 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7faeb819e9e0 con 0x7faeb8095600 2021-02-10T11:49:09.599 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:09.595 7faeaeffd700 1 -- 172.21.2.112:0/191347495 <== mon.1 v2:172.21.2.120:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.2 v122) v1 ==== 184+0+0 (secure 0 0 0) 0x7faea0000c10 con 0x7faeb8095600 2021-02-10T11:49:09.601 INFO:teuthology.orchestra.run.gibba012.stderr:updated caps for client.2 2021-02-10T11:49:09.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:09.599 7faebc842700 1 -- 172.21.2.112:0/191347495 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fae9c018730 msgr2=0x7fae9c01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:09.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:09.599 7faebc842700 1 --2- 172.21.2.112:0/191347495 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fae9c018730 0x7fae9c01ac10 crc :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:09.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:09.599 7faebc842700 1 -- 172.21.2.112:0/191347495 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7faeb8095600 msgr2=0x7faeb8095a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:09.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:09.599 7faebc842700 1 --2- 172.21.2.112:0/191347495 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7faeb8095600 0x7faeb8095a00 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 rx=0x7faea00078d0 tx=0x7faea000a950).stop 2021-02-10T11:49:09.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:09.599 7faebc842700 1 -- 172.21.2.112:0/191347495 shutdown_connections 2021-02-10T11:49:09.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:09.599 7faebc842700 1 --2- 172.21.2.112:0/191347495 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fae9c018730 0x7fae9c01ac10 unknown :-1 s=CLOSED pgs=1473 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:09.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:09.599 7faebc842700 1 --2- 172.21.2.112:0/191347495 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7faeb8092a10 0x7faeb8092e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:09.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:09.599 7faebc842700 1 --2- 172.21.2.112:0/191347495 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7faeb8096020 0x7faeb8096420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:09.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:09.599 7faebc842700 1 --2- 172.21.2.112:0/191347495 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7faeb8095600 0x7faeb8095a00 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:09.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:09.599 7faebc842700 1 -- 172.21.2.112:0/191347495 >> 172.21.2.112:0/191347495 conn(0x7faeb81a6190 msgr2=0x7faeb80949b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:09.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:09.599 7faebc842700 1 -- 172.21.2.112:0/191347495 shutdown_connections 2021-02-10T11:49:09.603 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:09.599 7faebc842700 1 -- 172.21.2.112:0/191347495 wait complete. 2021-02-10T11:49:09.611 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.3 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:49:10.051 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.031 7fae94acb700 1 Processor -- start 2021-02-10T11:49:10.051 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.031 7fae94acb700 1 -- start start 2021-02-10T11:49:10.051 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.031 7fae94acb700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fae900d1c80 0x7fae900cef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:10.051 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.031 7fae94acb700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fae900cf590 0x7fae900cf990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:10.051 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.031 7fae94acb700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae900fe1d0 con 0x7fae900d1880 2021-02-10T11:49:10.052 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.031 7fae94acb700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae901b0cc0 con 0x7fae900d1c80 2021-02-10T11:49:10.052 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.031 7fae94acb700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae90109a70 con 0x7fae900cf590 2021-02-10T11:49:10.052 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.031 7fae8e05d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fae900d1c80 0x7fae900cef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:10.052 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.031 7fae8e85e700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fae900d1880 0x7fae901b04d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:33102/0 (socket says 172.21.2.112:33102) 2021-02-10T11:49:10.052 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.031 7fae8e85e700 1 -- 172.21.2.112:0/3608294796 learned_addr learned my addr 172.21.2.112:0/3608294796 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:49:10.052 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.032 7fae8e05d700 1 -- 172.21.2.112:0/3608294796 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fae900cf590 msgr2=0x7fae900cf990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:49:10.052 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.032 7fae8e05d700 1 --2- 172.21.2.112:0/3608294796 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fae900cf590 0x7fae900cf990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:10.052 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.032 7fae8e05d700 1 -- 172.21.2.112:0/3608294796 >> v1:172.21.2.112:6789/0 conn(0x7fae900d1880 legacy=0x7fae901b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:10.052 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.032 7fae8d85c700 1 -- 172.21.2.112:0/3608294796 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 712358184 0 0) 0x7fae78000ce0 con 0x7fae900d1880 2021-02-10T11:49:10.052 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.032 7fae8e05d700 1 -- 172.21.2.112:0/3608294796 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae90088070 con 0x7fae900d1c80 2021-02-10T11:49:10.052 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.032 7fae8e05d700 1 --2- 172.21.2.112:0/3608294796 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fae900d1c80 0x7fae900cef70 secure :-1 s=READY pgs=3267 cs=0 l=1 rev1=1 rx=0x7fae80006650 tx=0x7fae80009a00).ready entity=mon.0 client_cookie=b39b8de08493a9d1 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:10.053 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.032 7fae8d85c700 1 -- 172.21.2.112:0/3608294796 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fae800009b0 con 0x7fae900d1c80 2021-02-10T11:49:10.053 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.032 7fae8d85c700 1 -- 172.21.2.112:0/3608294796 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae8000c0c0 con 0x7fae900d1c80 2021-02-10T11:49:10.053 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.032 7fae8d85c700 1 -- 172.21.2.112:0/3608294796 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fae8000c5a0 con 0x7fae900d1c80 2021-02-10T11:49:10.053 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.032 7fae94acb700 1 -- 172.21.2.112:0/3608294796 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fae900d1c80 msgr2=0x7fae900cef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:10.053 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.032 7fae94acb700 1 --2- 172.21.2.112:0/3608294796 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fae900d1c80 0x7fae900cef70 secure :-1 s=READY pgs=3267 cs=0 l=1 rev1=1 rx=0x7fae80006650 tx=0x7fae80009a00).stop 2021-02-10T11:49:10.053 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.033 7fae94acb700 1 -- 172.21.2.112:0/3608294796 shutdown_connections 2021-02-10T11:49:10.053 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.033 7fae94acb700 1 --2- 172.21.2.112:0/3608294796 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fae900cf590 0x7fae900cf990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:10.053 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.033 7fae94acb700 1 --2- 172.21.2.112:0/3608294796 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fae900d1c80 0x7fae900cef70 unknown :-1 s=CLOSED pgs=3267 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:10.053 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.033 7fae94acb700 1 -- 172.21.2.112:0/3608294796 >> 172.21.2.112:0/3608294796 conn(0x7fae900ce430 msgr2=0x7fae900d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:10.053 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.033 7fae94acb700 1 -- 172.21.2.112:0/3608294796 shutdown_connections 2021-02-10T11:49:10.053 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.033 7fae94acb700 1 -- 172.21.2.112:0/3608294796 wait complete. 2021-02-10T11:49:10.055 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.048 7fae94acb700 1 Processor -- start 2021-02-10T11:49:10.055 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.048 7fae94acb700 1 -- start start 2021-02-10T11:49:10.055 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.048 7fae94acb700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fae901b9470 0x7fae901ae6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:10.055 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.048 7fae94acb700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fae901aecf0 0x7fae900957f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:10.055 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.048 7fae94acb700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fae901af0f0 0x7fae90095e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:10.055 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.048 7fae94acb700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae90096d70 con 0x7fae901b9470 2021-02-10T11:49:10.055 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.048 7fae94acb700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae900a7780 con 0x7fae901aecf0 2021-02-10T11:49:10.056 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.048 7fae94acb700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae901b9870 con 0x7fae901af0f0 2021-02-10T11:49:10.056 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.048 7fae8e85e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fae901b9470 0x7fae901ae6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:10.056 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.049 7fae8e85e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fae901b9470 0x7fae901ae6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:33646/0 (socket says 172.21.2.112:33646) 2021-02-10T11:49:10.056 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.049 7fae8e85e700 1 -- 172.21.2.112:0/1276857586 learned_addr learned my addr 172.21.2.112:0/1276857586 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:49:10.056 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.049 7fae8e05d700 1 --2- 172.21.2.112:0/1276857586 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fae901aecf0 0x7fae900957f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:10.056 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.049 7fae8e85e700 1 -- 172.21.2.112:0/1276857586 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fae901af0f0 msgr2=0x7fae90095e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:10.056 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.049 7fae8f05f700 1 --2- 172.21.2.112:0/1276857586 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fae901af0f0 0x7fae90095e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:10.056 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.049 7fae8e85e700 1 --2- 172.21.2.112:0/1276857586 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fae901af0f0 0x7fae90095e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:10.056 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.049 7fae8e85e700 1 -- 172.21.2.112:0/1276857586 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fae901aecf0 msgr2=0x7fae900957f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:10.056 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.049 7fae8e85e700 1 --2- 172.21.2.112:0/1276857586 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fae901aecf0 0x7fae900957f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:10.056 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.049 7fae8e85e700 1 -- 172.21.2.112:0/1276857586 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae9019e6b0 con 0x7fae901b9470 2021-02-10T11:49:10.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.049 7fae8e05d700 1 --2- 172.21.2.112:0/1276857586 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fae901aecf0 0x7fae900957f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:49:10.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.049 7fae8f05f700 1 --2- 172.21.2.112:0/1276857586 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fae901af0f0 0x7fae90095e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:49:10.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.049 7fae8e85e700 1 --2- 172.21.2.112:0/1276857586 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fae901b9470 0x7fae901ae6d0 secure :-1 s=READY pgs=3268 cs=0 l=1 rev1=1 rx=0x7fae7800a590 tx=0x7fae7800a930).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:10.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.049 7fae877fe700 1 -- 172.21.2.112:0/1276857586 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fae78000ef0 con 0x7fae901b9470 2021-02-10T11:49:10.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.049 7fae877fe700 1 -- 172.21.2.112:0/1276857586 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae7800ad20 con 0x7fae901b9470 2021-02-10T11:49:10.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.049 7fae94acb700 1 -- 172.21.2.112:0/1276857586 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae9019e6b0 con 0x7fae901b9470 2021-02-10T11:49:10.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.050 7fae877fe700 1 -- 172.21.2.112:0/1276857586 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fae78004620 con 0x7fae901b9470 2021-02-10T11:49:10.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.050 7fae94acb700 1 -- 172.21.2.112:0/1276857586 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae90096720 con 0x7fae901b9470 2021-02-10T11:49:10.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.050 7fae877fe700 1 -- 172.21.2.112:0/1276857586 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fae78008000 con 0x7fae901b9470 2021-02-10T11:49:10.057 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.050 7fae877fe700 1 --2- 172.21.2.112:0/1276857586 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fae740186c0 0x7fae7401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:10.058 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.050 7fae877fe700 1 -- 172.21.2.112:0/1276857586 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fae78032580 con 0x7fae901b9470 2021-02-10T11:49:10.058 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.051 7fae8e05d700 1 --2- 172.21.2.112:0/1276857586 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fae740186c0 0x7fae7401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:10.058 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.051 7fae8e05d700 1 --2- 172.21.2.112:0/1276857586 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fae740186c0 0x7fae7401abc0 crc :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:10.085 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.081 7fae94acb700 1 -- 172.21.2.112:0/1276857586 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae900829a0 con 0x7fae901b9470 2021-02-10T11:49:10.088 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.084 7fae877fe700 1 -- 172.21.2.112:0/1276857586 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fae78000ef0 con 0x7fae901b9470 2021-02-10T11:49:10.217 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.214 7fae94acb700 1 -- 172.21.2.112:0/1276857586 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fae900a7780 con 0x7fae901b9470 2021-02-10T11:49:10.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.218 7fae877fe700 1 -- 172.21.2.112:0/1276857586 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.3 v123) v1 ==== 184+0+0 (secure 0 0 0) 0x7fae78008000 con 0x7fae901b9470 2021-02-10T11:49:10.225 INFO:teuthology.orchestra.run.gibba012.stderr:updated caps for client.3 2021-02-10T11:49:10.227 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.223 7fae94acb700 1 -- 172.21.2.112:0/1276857586 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fae740186c0 msgr2=0x7fae7401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:10.227 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.223 7fae94acb700 1 --2- 172.21.2.112:0/1276857586 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fae740186c0 0x7fae7401abc0 crc :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:10.227 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.223 7fae94acb700 1 -- 172.21.2.112:0/1276857586 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fae901b9470 msgr2=0x7fae901ae6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:10.227 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.223 7fae94acb700 1 --2- 172.21.2.112:0/1276857586 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fae901b9470 0x7fae901ae6d0 secure :-1 s=READY pgs=3268 cs=0 l=1 rev1=1 rx=0x7fae7800a590 tx=0x7fae7800a930).stop 2021-02-10T11:49:10.227 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.223 7fae94acb700 1 -- 172.21.2.112:0/1276857586 shutdown_connections 2021-02-10T11:49:10.227 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.223 7fae94acb700 1 --2- 172.21.2.112:0/1276857586 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fae740186c0 0x7fae7401abc0 unknown :-1 s=CLOSED pgs=1474 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:10.227 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.223 7fae94acb700 1 --2- 172.21.2.112:0/1276857586 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fae901aecf0 0x7fae900957f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:10.228 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.223 7fae94acb700 1 --2- 172.21.2.112:0/1276857586 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fae901af0f0 0x7fae90095e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:10.228 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.223 7fae94acb700 1 --2- 172.21.2.112:0/1276857586 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fae901b9470 0x7fae901ae6d0 unknown :-1 s=CLOSED pgs=3268 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:10.228 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.223 7fae94acb700 1 -- 172.21.2.112:0/1276857586 >> 172.21.2.112:0/1276857586 conn(0x7fae900845b0 msgr2=0x7fae9018c290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:10.228 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.223 7fae94acb700 1 -- 172.21.2.112:0/1276857586 shutdown_connections 2021-02-10T11:49:10.228 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.223 7fae94acb700 1 -- 172.21.2.112:0/1276857586 wait complete. 2021-02-10T11:49:10.235 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:49:10.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.643 7f013cdf2700 1 Processor -- start 2021-02-10T11:49:10.665 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.643 7f013cdf2700 1 -- start start 2021-02-10T11:49:10.665 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.644 7f013cdf2700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f01380d1870 0x7f01381b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:10.665 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.644 7f013cdf2700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01380f0b10 con 0x7f01381b0a10 2021-02-10T11:49:10.665 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.644 7f013cdf2700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01380d0310 con 0x7f01380f0650 2021-02-10T11:49:10.665 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.644 7f013cdf2700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01380e5ed0 con 0x7f01380d1870 2021-02-10T11:49:10.665 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.644 7f0136312700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f01381b0a10 0x7f01380cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:33116/0 (socket says 172.21.2.112:33116) 2021-02-10T11:49:10.665 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.644 7f0136312700 1 -- 172.21.2.112:0/558061941 learned_addr learned my addr 172.21.2.112:0/558061941 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:49:10.665 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.644 7f0136b13700 1 --2- 172.21.2.112:0/558061941 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f01380d1870 0x7f01381b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:10.665 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.644 7f0135b11700 1 -- 172.21.2.112:0/558061941 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1181840585 0 0) 0x7f0128000ce0 con 0x7f01381b0a10 2021-02-10T11:49:10.665 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.644 7f0135b11700 1 -- 172.21.2.112:0/558061941 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0124002fe0 con 0x7f01381b0a10 2021-02-10T11:49:10.666 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.644 7f0135b11700 1 -- 172.21.2.112:0/558061941 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3764457051 0 0) 0x7f0130000f90 con 0x7f01380f0650 2021-02-10T11:49:10.666 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.644 7f0135b11700 1 -- 172.21.2.112:0/558061941 --> v1:172.21.2.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0124003210 con 0x7f01380f0650 2021-02-10T11:49:10.666 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.644 7f0135b11700 1 -- 172.21.2.112:0/558061941 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1682027991 0 0) 0x7f0128000ed0 con 0x7f01381b0a10 2021-02-10T11:49:10.666 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.644 7f0135b11700 1 -- 172.21.2.112:0/558061941 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f01380d1870 msgr2=0x7f01381b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:49:10.666 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.644 7f0135b11700 1 --2- 172.21.2.112:0/558061941 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f01380d1870 0x7f01381b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:10.666 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.645 7f0135b11700 1 -- 172.21.2.112:0/558061941 >> v1:172.21.2.120:6789/0 conn(0x7f01380f0650 legacy=0x7f01380cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:10.666 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.645 7f0135b11700 1 -- 172.21.2.112:0/558061941 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01380d0530 con 0x7f01381b0a10 2021-02-10T11:49:10.666 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.645 7f0135b11700 1 -- 172.21.2.112:0/558061941 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f0128001600 con 0x7f01381b0a10 2021-02-10T11:49:10.666 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.645 7f0135b11700 1 -- 172.21.2.112:0/558061941 >> v1:172.21.2.112:6789/0 conn(0x7f01381b0a10 legacy=0x7f01380cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:10.666 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.645 7f0135b11700 1 --2- 172.21.2.112:0/558061941 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0124003c30 0x7f0124006040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:10.666 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.645 7f0135b11700 1 --2- 172.21.2.112:0/558061941 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f01240067d0 0x7f0124008c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:10.666 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.645 7f0135b11700 1 --2- 172.21.2.112:0/558061941 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0124009520 0x7f012400ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:10.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.645 7f0135b11700 1 -- 172.21.2.112:0/558061941 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01240502e0 con 0x7f0124009520 2021-02-10T11:49:10.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.645 7f0135b11700 1 -- 172.21.2.112:0/558061941 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01240504c0 con 0x7f0124003c30 2021-02-10T11:49:10.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.645 7f0135b11700 1 -- 172.21.2.112:0/558061941 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01240506e0 con 0x7f01240067d0 2021-02-10T11:49:10.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.645 7f0137314700 1 --2- 172.21.2.112:0/558061941 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0124009520 0x7f012400ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:10.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.645 7f0136b13700 1 --2- 172.21.2.112:0/558061941 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0124003c30 0x7f0124006040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:10.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.645 7f0136312700 1 --2- 172.21.2.112:0/558061941 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f01240067d0 0x7f0124008c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:10.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.645 7f0137314700 1 -- 172.21.2.112:0/558061941 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f01240067d0 msgr2=0x7f0124008c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:49:10.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.645 7f0137314700 1 --2- 172.21.2.112:0/558061941 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f01240067d0 0x7f0124008c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:10.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.645 7f0137314700 1 -- 172.21.2.112:0/558061941 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0124003c30 msgr2=0x7f0124006040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:49:10.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.645 7f0137314700 1 --2- 172.21.2.112:0/558061941 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0124003c30 0x7f0124006040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:10.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.645 7f0137314700 1 -- 172.21.2.112:0/558061941 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0124050980 con 0x7f0124009520 2021-02-10T11:49:10.667 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.645 7f0137314700 1 --2- 172.21.2.112:0/558061941 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0124009520 0x7f012400ba20 secure :-1 s=READY pgs=3270 cs=0 l=1 rev1=1 rx=0x7f01300013c0 tx=0x7f0130009e20).ready entity=mon.0 client_cookie=16e0ab3b94914ad1 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:10.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.645 7f0135b11700 1 -- 172.21.2.112:0/558061941 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0130001400 con 0x7f0124009520 2021-02-10T11:49:10.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.645 7f0135b11700 1 -- 172.21.2.112:0/558061941 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0130006a60 con 0x7f0124009520 2021-02-10T11:49:10.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.645 7f0135b11700 1 -- 172.21.2.112:0/558061941 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0130001980 con 0x7f0124009520 2021-02-10T11:49:10.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.645 7f013cdf2700 1 -- 172.21.2.112:0/558061941 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0124009520 msgr2=0x7f012400ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:10.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.645 7f013cdf2700 1 --2- 172.21.2.112:0/558061941 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0124009520 0x7f012400ba20 secure :-1 s=READY pgs=3270 cs=0 l=1 rev1=1 rx=0x7f01300013c0 tx=0x7f0130009e20).stop 2021-02-10T11:49:10.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.645 7f0137314700 1 -- 172.21.2.112:0/558061941 reap_dead start 2021-02-10T11:49:10.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.646 7f013cdf2700 1 -- 172.21.2.112:0/558061941 shutdown_connections 2021-02-10T11:49:10.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.646 7f013cdf2700 1 -- 172.21.2.112:0/558061941 >> 172.21.2.112:0/558061941 conn(0x7f01380ce450 msgr2=0x7f01380d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:10.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.646 7f013cdf2700 1 -- 172.21.2.112:0/558061941 shutdown_connections 2021-02-10T11:49:10.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.646 7f013cdf2700 1 -- 172.21.2.112:0/558061941 wait complete. 2021-02-10T11:49:10.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.661 7f013cdf2700 1 Processor -- start 2021-02-10T11:49:10.668 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.661 7f013cdf2700 1 -- start start 2021-02-10T11:49:10.670 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.661 7f013cdf2700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f01381a49e0 0x7f0138095ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:10.670 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.661 7f013cdf2700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f01380960e0 0x7f01380a1980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:10.670 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.661 7f013cdf2700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f01380a1fa0 0x7f01380c1ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:10.670 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.661 7f013cdf2700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01380a23a0 con 0x7f01380a1fa0 2021-02-10T11:49:10.670 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.661 7f013cdf2700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01380c24f0 con 0x7f01381a49e0 2021-02-10T11:49:10.670 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.661 7f013cdf2700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01380c2710 con 0x7f01380960e0 2021-02-10T11:49:10.670 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.661 7f0137314700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f01380a1fa0 0x7f01380c1ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:10.670 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.661 7f0137314700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f01380a1fa0 0x7f01380c1ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:33668/0 (socket says 172.21.2.112:33668) 2021-02-10T11:49:10.670 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.661 7f0137314700 1 -- 172.21.2.112:0/2577965422 learned_addr learned my addr 172.21.2.112:0/2577965422 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:49:10.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.662 7f0136b13700 1 --2- 172.21.2.112:0/2577965422 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f01381a49e0 0x7f0138095ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:10.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.662 7f0136312700 1 --2- 172.21.2.112:0/2577965422 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f01380960e0 0x7f01380a1980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:10.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.662 7f0137314700 1 -- 172.21.2.112:0/2577965422 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f01380960e0 msgr2=0x7f01380a1980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:10.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.662 7f0137314700 1 --2- 172.21.2.112:0/2577965422 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f01380960e0 0x7f01380a1980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:10.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.662 7f0137314700 1 -- 172.21.2.112:0/2577965422 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f01381a49e0 msgr2=0x7f0138095ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:10.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.662 7f0137314700 1 --2- 172.21.2.112:0/2577965422 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f01381a49e0 0x7f0138095ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:10.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.662 7f0137314700 1 -- 172.21.2.112:0/2577965422 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01380c2930 con 0x7f01380a1fa0 2021-02-10T11:49:10.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.662 7f0137314700 1 --2- 172.21.2.112:0/2577965422 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f01380a1fa0 0x7f01380c1ed0 secure :-1 s=READY pgs=3271 cs=0 l=1 rev1=1 rx=0x7f0130006f40 tx=0x7f013000bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:10.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.662 7f012f7fe700 1 -- 172.21.2.112:0/2577965422 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0130001390 con 0x7f01380a1fa0 2021-02-10T11:49:10.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.662 7f012f7fe700 1 -- 172.21.2.112:0/2577965422 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0130006c50 con 0x7f01380a1fa0 2021-02-10T11:49:10.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.662 7f012f7fe700 1 -- 172.21.2.112:0/2577965422 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0130001950 con 0x7f01380a1fa0 2021-02-10T11:49:10.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.662 7f013cdf2700 1 -- 172.21.2.112:0/2577965422 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01380c2930 con 0x7f01380a1fa0 2021-02-10T11:49:10.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.662 7f013cdf2700 1 -- 172.21.2.112:0/2577965422 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01380c30e0 con 0x7f01380a1fa0 2021-02-10T11:49:10.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.663 7f012f7fe700 1 -- 172.21.2.112:0/2577965422 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0130001180 con 0x7f01380a1fa0 2021-02-10T11:49:10.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.663 7f012f7fe700 1 --2- 172.21.2.112:0/2577965422 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f011c0186c0 0x7f011c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:10.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.663 7f012f7fe700 1 -- 172.21.2.112:0/2577965422 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f0130032900 con 0x7f01380a1fa0 2021-02-10T11:49:10.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.663 7f0136b13700 1 --2- 172.21.2.112:0/2577965422 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f011c0186c0 0x7f011c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:10.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.664 7f0136b13700 1 --2- 172.21.2.112:0/2577965422 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f011c0186c0 0x7f011c01abc0 crc :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:10.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.694 7f013cdf2700 1 -- 172.21.2.112:0/2577965422 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01380ebc80 con 0x7f01380a1fa0 2021-02-10T11:49:10.700 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.696 7f012f7fe700 1 -- 172.21.2.112:0/2577965422 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0130001c10 con 0x7f01380a1fa0 2021-02-10T11:49:10.830 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.826 7f013cdf2700 1 -- 172.21.2.112:0/2577965422 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f01380f0b10 con 0x7f01380a1fa0 2021-02-10T11:49:10.830 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.826 7f012f7fe700 1 -- 172.21.2.112:0/2577965422 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 174 v174) v1 ==== 95+0+2108 (secure 0 0 0) 0x7f0130001180 con 0x7f01380a1fa0 2021-02-10T11:49:10.831 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:49:10.831 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":174,"default_fscid":21,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":13800,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/2202619473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":2202619473},{"type":"v1","addr":"172.21.2.112:6833","nonce":2202619473}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":170}],"filesystems":[{"mdsmap":{"epoch":174,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:49:04.478835","modified":"2021-02-10 11:49:06.484559","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9715},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9715":{"gid":9715,"name":"b","rank":0,"incarnation":171,"state":"up:active","state_seq":3,"addr":"172.21.2.120:6835/3826510568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":3826510568},{"type":"v1","addr":"172.21.2.120:6835","nonce":3826510568}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[43],"metadata_pool":42,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":21}]} 2021-02-10T11:49:10.831 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 174 2021-02-10T11:49:10.832 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.828 7f013cdf2700 1 -- 172.21.2.112:0/2577965422 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f011c0186c0 msgr2=0x7f011c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:10.832 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.828 7f013cdf2700 1 --2- 172.21.2.112:0/2577965422 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f011c0186c0 0x7f011c01abc0 crc :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:10.833 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.828 7f013cdf2700 1 -- 172.21.2.112:0/2577965422 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f01380a1fa0 msgr2=0x7f01380c1ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:10.833 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.828 7f013cdf2700 1 --2- 172.21.2.112:0/2577965422 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f01380a1fa0 0x7f01380c1ed0 secure :-1 s=READY pgs=3271 cs=0 l=1 rev1=1 rx=0x7f0130006f40 tx=0x7f013000bd40).stop 2021-02-10T11:49:10.833 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.829 7f013cdf2700 1 -- 172.21.2.112:0/2577965422 shutdown_connections 2021-02-10T11:49:10.833 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.829 7f013cdf2700 1 --2- 172.21.2.112:0/2577965422 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f011c0186c0 0x7f011c01abc0 unknown :-1 s=CLOSED pgs=1475 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:10.833 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.829 7f013cdf2700 1 --2- 172.21.2.112:0/2577965422 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f01380a1fa0 0x7f01380c1ed0 unknown :-1 s=CLOSED pgs=3271 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:10.833 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.829 7f013cdf2700 1 --2- 172.21.2.112:0/2577965422 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f01380960e0 0x7f01380a1980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:10.833 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.829 7f013cdf2700 1 --2- 172.21.2.112:0/2577965422 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f01381a49e0 0x7f0138095ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:10.833 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.829 7f013cdf2700 1 -- 172.21.2.112:0/2577965422 >> 172.21.2.112:0/2577965422 conn(0x7f01380a2b90 msgr2=0x7f0138190660 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:10.833 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.829 7f013cdf2700 1 -- 172.21.2.112:0/2577965422 shutdown_connections 2021-02-10T11:49:10.833 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:10.829 7f013cdf2700 1 -- 172.21.2.112:0/2577965422 wait complete. 2021-02-10T11:49:11.042 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 174, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:49:04.478835', 'modified': '2021-02-10 11:49:06.484559', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9715}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9715': {'gid': 9715, 'name': 'b', 'rank': 0, 'incarnation': 171, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.120:6835/3826510568', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.120:6834', 'nonce': 3826510568}, {'type': 'v1', 'addr': '172.21.2.120:6835', 'nonce': 3826510568}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [43], 'metadata_pool': 42, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:49:11.043 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:49:11.043 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:49:11.440 INFO:teuthology.orchestra.run.gibba020.stdout:{ 2021-02-10T11:49:11.441 INFO:teuthology.orchestra.run.gibba020.stdout: "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", 2021-02-10T11:49:11.441 INFO:teuthology.orchestra.run.gibba020.stdout: "whoami": 0, 2021-02-10T11:49:11.441 INFO:teuthology.orchestra.run.gibba020.stdout: "id": 9715, 2021-02-10T11:49:11.441 INFO:teuthology.orchestra.run.gibba020.stdout: "want_state": "up:active", 2021-02-10T11:49:11.441 INFO:teuthology.orchestra.run.gibba020.stdout: "state": "up:active", 2021-02-10T11:49:11.441 INFO:teuthology.orchestra.run.gibba020.stdout: "rank_uptime": 6.9560201299999997, 2021-02-10T11:49:11.441 INFO:teuthology.orchestra.run.gibba020.stdout: "mdsmap_epoch": 174, 2021-02-10T11:49:11.441 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch": 198, 2021-02-10T11:49:11.441 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch_barrier": 198, 2021-02-10T11:49:11.441 INFO:teuthology.orchestra.run.gibba020.stdout: "uptime": 12.606188877999999 2021-02-10T11:49:11.441 INFO:teuthology.orchestra.run.gibba020.stdout:} 2021-02-10T11:49:11.444 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", "whoami": 0, "id": 9715, "want_state": "up:active", "state": "up:active", "rank_uptime": 6.9560201299999997, "mdsmap_epoch": 174, "osdmap_epoch": 198, "osdmap_epoch_barrier": 198, "uptime": 12.606188877999999 } 2021-02-10T11:49:11.445 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:49:11.876 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:11.856 7f31d3c05700 1 Processor -- start 2021-02-10T11:49:11.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:11.856 7f31d3c05700 1 -- start start 2021-02-10T11:49:11.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:11.857 7f31d3c05700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f31cc0d1870 0x7f31cc1b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:11.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:11.857 7f31d3c05700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f31cc1b0a10 0x7f31cc0cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:11.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:11.857 7f31d3c05700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31cc0f0b10 con 0x7f31cc0d1c70 2021-02-10T11:49:11.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:11.857 7f31d3c05700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31cc0e5ed0 con 0x7f31cc0d1870 2021-02-10T11:49:11.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:11.857 7f31d3c05700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31cc0f0650 con 0x7f31cc1b0a10 2021-02-10T11:49:11.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:11.857 7f31d1a21700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f31cc0d1870 0x7f31cc1b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:11.877 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:11.857 7f31d1a21700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f31cc0d1870 0x7f31cc1b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:33674/0 (socket says 172.21.2.112:33674) 2021-02-10T11:49:11.878 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:11.857 7f31d1a21700 1 -- 172.21.2.112:0/2903907101 learned_addr learned my addr 172.21.2.112:0/2903907101 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:49:11.878 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:11.857 7f31d1a21700 1 -- 172.21.2.112:0/2903907101 >> v2:172.21.2.120:3300/0 conn(0x7f31cc1b0a10 msgr2=0x7f31cc0cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:49:11.878 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:11.857 7f31d1a21700 1 --2- 172.21.2.112:0/2903907101 >> v2:172.21.2.120:3300/0 conn(0x7f31cc1b0a10 0x7f31cc0cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:11.878 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:11.857 7f31d1a21700 1 -- 172.21.2.112:0/2903907101 >> v1:172.21.2.120:6789/0 conn(0x7f31cc0d1c70 legacy=0x7f31cc0cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:49:11.878 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:11.857 7f31d1a21700 1 -- 172.21.2.112:0/2903907101 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31cc0f0840 con 0x7f31cc0d1870 2021-02-10T11:49:11.878 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:11.857 7f31d1a21700 1 --2- 172.21.2.112:0/2903907101 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f31cc0d1870 0x7f31cc1b03f0 secure :-1 s=READY pgs=3272 cs=0 l=1 rev1=1 rx=0x7f31c8006700 tx=0x7f31c8009b00).ready entity=mon.0 client_cookie=59a4cb6a3dbbaed6 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:11.878 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:11.858 7f31d0a1f700 1 -- 172.21.2.112:0/2903907101 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f31c800c0c0 con 0x7f31cc0d1870 2021-02-10T11:49:11.878 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:11.858 7f31d0a1f700 1 -- 172.21.2.112:0/2903907101 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31c800c400 con 0x7f31cc0d1870 2021-02-10T11:49:11.878 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:11.858 7f31d0a1f700 1 -- 172.21.2.112:0/2903907101 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f31c800c920 con 0x7f31cc0d1870 2021-02-10T11:49:11.878 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:11.858 7f31d3c05700 1 -- 172.21.2.112:0/2903907101 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f31cc0d1870 msgr2=0x7f31cc1b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:11.879 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:11.858 7f31d3c05700 1 --2- 172.21.2.112:0/2903907101 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f31cc0d1870 0x7f31cc1b03f0 secure :-1 s=READY pgs=3272 cs=0 l=1 rev1=1 rx=0x7f31c8006700 tx=0x7f31c8009b00).stop 2021-02-10T11:49:11.879 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:11.858 7f31d3c05700 1 -- 172.21.2.112:0/2903907101 shutdown_connections 2021-02-10T11:49:11.879 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:11.858 7f31d3c05700 1 --2- 172.21.2.112:0/2903907101 >> v2:172.21.2.120:3300/0 conn(0x7f31cc1b0a10 0x7f31cc0cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:11.879 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:11.858 7f31d3c05700 1 --2- 172.21.2.112:0/2903907101 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f31cc0d1870 0x7f31cc1b03f0 unknown :-1 s=CLOSED pgs=3272 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:11.879 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:11.858 7f31d3c05700 1 -- 172.21.2.112:0/2903907101 >> 172.21.2.112:0/2903907101 conn(0x7f31cc0ce450 msgr2=0x7f31cc0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:11.879 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:11.859 7f31d3c05700 1 -- 172.21.2.112:0/2903907101 shutdown_connections 2021-02-10T11:49:11.879 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:11.859 7f31d3c05700 1 -- 172.21.2.112:0/2903907101 wait complete. 2021-02-10T11:49:11.881 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:11.873 7f31d3c05700 1 Processor -- start 2021-02-10T11:49:11.881 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:11.873 7f31d3c05700 1 -- start start 2021-02-10T11:49:11.881 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:11.874 7f31d3c05700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f31cc0b4f30 0x7f31cc18b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:11.881 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:11.874 7f31d3c05700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f31cc093f40 0x7f31cc094340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:11.881 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:11.874 7f31d3c05700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f31cc094960 0x7f31cc187e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:11.881 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:11.874 7f31d3c05700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31cc094e90 con 0x7f31cc094960 2021-02-10T11:49:11.882 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:11.874 7f31d3c05700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31cc18bff0 con 0x7f31cc0b4f30 2021-02-10T11:49:11.882 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:11.874 7f31d3c05700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31cc08f780 con 0x7f31cc093f40 2021-02-10T11:49:11.882 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:11.874 7f31d2222700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f31cc094960 0x7f31cc187e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:11.882 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:11.874 7f31d2222700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f31cc094960 0x7f31cc187e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:33680/0 (socket says 172.21.2.112:33680) 2021-02-10T11:49:11.882 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:11.874 7f31d2222700 1 -- 172.21.2.112:0/2452250160 learned_addr learned my addr 172.21.2.112:0/2452250160 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:49:11.882 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:11.874 7f31d1a21700 1 --2- 172.21.2.112:0/2452250160 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f31cc0b4f30 0x7f31cc18b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:11.882 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:11.874 7f31d1220700 1 --2- 172.21.2.112:0/2452250160 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f31cc093f40 0x7f31cc094340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:11.882 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:11.874 7f31d2222700 1 -- 172.21.2.112:0/2452250160 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f31cc093f40 msgr2=0x7f31cc094340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:11.882 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:11.874 7f31d2222700 1 --2- 172.21.2.112:0/2452250160 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f31cc093f40 0x7f31cc094340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:11.882 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:11.874 7f31d2222700 1 -- 172.21.2.112:0/2452250160 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f31cc0b4f30 msgr2=0x7f31cc18b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:11.883 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:11.874 7f31d2222700 1 --2- 172.21.2.112:0/2452250160 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f31cc0b4f30 0x7f31cc18b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:11.883 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:11.874 7f31d2222700 1 -- 172.21.2.112:0/2452250160 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31cc188470 con 0x7f31cc094960 2021-02-10T11:49:11.883 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:11.874 7f31d1a21700 1 --2- 172.21.2.112:0/2452250160 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f31cc0b4f30 0x7f31cc18b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:49:11.883 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:11.874 7f31d2222700 1 --2- 172.21.2.112:0/2452250160 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f31cc094960 0x7f31cc187e50 secure :-1 s=READY pgs=3273 cs=0 l=1 rev1=1 rx=0x7f31c4009940 tx=0x7f31c400bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:11.883 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:11.875 7f31be7fc700 1 -- 172.21.2.112:0/2452250160 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f31c4000f90 con 0x7f31cc094960 2021-02-10T11:49:11.883 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:11.875 7f31be7fc700 1 -- 172.21.2.112:0/2452250160 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31c400c2f0 con 0x7f31cc094960 2021-02-10T11:49:11.883 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:11.875 7f31be7fc700 1 -- 172.21.2.112:0/2452250160 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f31c400c750 con 0x7f31cc094960 2021-02-10T11:49:11.883 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:11.875 7f31d3c05700 1 -- 172.21.2.112:0/2452250160 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31cc188470 con 0x7f31cc094960 2021-02-10T11:49:11.883 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:11.875 7f31d3c05700 1 -- 172.21.2.112:0/2452250160 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31cc188470 con 0x7f31cc094960 2021-02-10T11:49:11.883 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:11.875 7f31be7fc700 1 -- 172.21.2.112:0/2452250160 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f31c4001220 con 0x7f31cc094960 2021-02-10T11:49:11.883 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:11.875 7f31be7fc700 1 --2- 172.21.2.112:0/2452250160 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f31b40186c0 0x7f31b401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:11.884 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:11.875 7f31be7fc700 1 -- 172.21.2.112:0/2452250160 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f31c4032940 con 0x7f31cc094960 2021-02-10T11:49:11.884 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:11.876 7f31d1a21700 1 --2- 172.21.2.112:0/2452250160 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f31b40186c0 0x7f31b401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:11.884 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:11.876 7f31d1a21700 1 --2- 172.21.2.112:0/2452250160 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f31b40186c0 0x7f31b401abc0 crc :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:11.910 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:11.906 7f31d3c05700 1 -- 172.21.2.112:0/2452250160 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31cc0ce450 con 0x7f31cc094960 2021-02-10T11:49:11.912 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:11.909 7f31be7fc700 1 -- 172.21.2.112:0/2452250160 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f31c400be30 con 0x7f31cc094960 2021-02-10T11:49:12.037 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:12.033 7f31d3c05700 1 -- 172.21.2.112:0/2452250160 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f31cc0ce450 con 0x7f31cc094960 2021-02-10T11:49:12.037 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:12.034 7f31be7fc700 1 -- 172.21.2.112:0/2452250160 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v198) v1 ==== 74+0+14007 (secure 0 0 0) 0x7f31c401f8c0 con 0x7f31cc094960 2021-02-10T11:49:12.038 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:49:12.038 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":198,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:49:05.481103","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":43,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":42,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:49:02.450970","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"197","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":43,"pool_name":"cephfs_data","create_time":"2021-02-10 11:49:03.473275","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"198","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":196,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":196,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":196,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":196,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":196,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:49:12.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:12.036 7f31d3c05700 1 -- 172.21.2.112:0/2452250160 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f31b40186c0 msgr2=0x7f31b401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:12.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:12.036 7f31d3c05700 1 --2- 172.21.2.112:0/2452250160 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f31b40186c0 0x7f31b401abc0 crc :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:12.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:12.036 7f31d3c05700 1 -- 172.21.2.112:0/2452250160 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f31cc094960 msgr2=0x7f31cc187e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:12.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:12.036 7f31d3c05700 1 --2- 172.21.2.112:0/2452250160 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f31cc094960 0x7f31cc187e50 secure :-1 s=READY pgs=3273 cs=0 l=1 rev1=1 rx=0x7f31c4009940 tx=0x7f31c400bcf0).stop 2021-02-10T11:49:12.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:12.036 7f31d3c05700 1 -- 172.21.2.112:0/2452250160 shutdown_connections 2021-02-10T11:49:12.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:12.036 7f31d3c05700 1 --2- 172.21.2.112:0/2452250160 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f31b40186c0 0x7f31b401abc0 unknown :-1 s=CLOSED pgs=1476 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:12.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:12.036 7f31d3c05700 1 --2- 172.21.2.112:0/2452250160 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f31cc094960 0x7f31cc187e50 unknown :-1 s=CLOSED pgs=3273 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:12.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:12.036 7f31d3c05700 1 --2- 172.21.2.112:0/2452250160 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f31cc093f40 0x7f31cc094340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:12.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:12.036 7f31d3c05700 1 --2- 172.21.2.112:0/2452250160 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f31cc0b4f30 0x7f31cc18b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:12.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:12.036 7f31d3c05700 1 -- 172.21.2.112:0/2452250160 >> 172.21.2.112:0/2452250160 conn(0x7f31cc19f8f0 msgr2=0x7f31cc1a5f80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:12.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:12.036 7f31d3c05700 1 -- 172.21.2.112:0/2452250160 shutdown_connections 2021-02-10T11:49:12.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:12.036 7f31d3c05700 1 -- 172.21.2.112:0/2452250160 wait complete. 2021-02-10T11:49:12.049 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:13:57.508758', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 42, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:49:02.450970', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '197', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 43, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:49:03.473275', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '198', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:49:12.049 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:49:12.481 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:12.461 7fa9453bd700 1 Processor -- start 2021-02-10T11:49:12.481 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:12.461 7fa9453bd700 1 -- start start 2021-02-10T11:49:12.481 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:12.462 7fa9453bd700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fa94010aff0 0x7fa9400cf750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:12.481 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:12.462 7fa9453bd700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9400fe1c0 con 0x7fa9400d1c70 2021-02-10T11:49:12.481 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:12.462 7fa9453bd700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9401b0cb0 con 0x7fa9400d1870 2021-02-10T11:49:12.481 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:12.462 7fa9453bd700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa940109a60 con 0x7fa94010aff0 2021-02-10T11:49:12.481 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:12.462 7fa93f87e700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fa94010aff0 0x7fa9400cf750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:12.481 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:12.462 7fa93f87e700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fa94010aff0 0x7fa9400cf750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:33688/0 (socket says 172.21.2.112:33688) 2021-02-10T11:49:12.481 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:12.462 7fa93f87e700 1 -- 172.21.2.112:0/1069858328 learned_addr learned my addr 172.21.2.112:0/1069858328 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:49:12.481 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:12.462 7fa93f87e700 1 -- 172.21.2.112:0/1069858328 >> v1:172.21.2.120:6789/0 conn(0x7fa9400d1870 legacy=0x7fa9401b04c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:49:12.481 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:12.462 7fa93f87e700 1 -- 172.21.2.112:0/1069858328 >> v1:172.21.2.112:6789/0 conn(0x7fa9400d1c70 legacy=0x7fa9400cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:12.482 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:12.462 7fa93f87e700 1 -- 172.21.2.112:0/1069858328 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa94010b3f0 con 0x7fa94010aff0 2021-02-10T11:49:12.482 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:12.462 7fa93f87e700 1 --2- 172.21.2.112:0/1069858328 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa94010aff0 0x7fa9400cf750 secure :-1 s=READY pgs=3275 cs=0 l=1 rev1=1 rx=0x7fa938006f00 tx=0x7fa938009f10).ready entity=mon.0 client_cookie=74535d45c8d587cd server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:12.482 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:12.462 7fa93e07b700 1 -- 172.21.2.112:0/1069858328 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa93800b3b0 con 0x7fa94010aff0 2021-02-10T11:49:12.482 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:12.463 7fa93e07b700 1 -- 172.21.2.112:0/1069858328 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa93800b630 con 0x7fa94010aff0 2021-02-10T11:49:12.482 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:12.463 7fa93e07b700 1 -- 172.21.2.112:0/1069858328 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa93800bb50 con 0x7fa94010aff0 2021-02-10T11:49:12.482 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:12.463 7fa9453bd700 1 -- 172.21.2.112:0/1069858328 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa94010aff0 msgr2=0x7fa9400cf750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:12.482 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:12.463 7fa9453bd700 1 --2- 172.21.2.112:0/1069858328 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa94010aff0 0x7fa9400cf750 secure :-1 s=READY pgs=3275 cs=0 l=1 rev1=1 rx=0x7fa938006f00 tx=0x7fa938009f10).stop 2021-02-10T11:49:12.482 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:12.463 7fa9453bd700 1 -- 172.21.2.112:0/1069858328 shutdown_connections 2021-02-10T11:49:12.482 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:12.463 7fa9453bd700 1 --2- 172.21.2.112:0/1069858328 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa94010aff0 0x7fa9400cf750 unknown :-1 s=CLOSED pgs=3275 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:12.482 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:12.463 7fa9453bd700 1 -- 172.21.2.112:0/1069858328 >> 172.21.2.112:0/1069858328 conn(0x7fa9400ce420 msgr2=0x7fa9400d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:12.482 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:12.463 7fa9453bd700 1 -- 172.21.2.112:0/1069858328 shutdown_connections 2021-02-10T11:49:12.482 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:12.463 7fa9453bd700 1 -- 172.21.2.112:0/1069858328 wait complete. 2021-02-10T11:49:12.484 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:12.477 7fa9453bd700 1 Processor -- start 2021-02-10T11:49:12.484 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:12.477 7fa9453bd700 1 -- start start 2021-02-10T11:49:12.484 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:12.478 7fa9453bd700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa9400c0740 0x7fa9401c5580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:12.484 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:12.478 7fa9453bd700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa9401c5ba0 0x7fa9400a5c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:12.484 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:12.478 7fa9453bd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa9400b6e60 0x7fa9400a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:12.484 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:12.478 7fa9453bd700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9400a6880 con 0x7fa9400b6e60 2021-02-10T11:49:12.484 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:12.478 7fa9453bd700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9400a6aa0 con 0x7fa9401c5ba0 2021-02-10T11:49:12.484 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:12.478 7fa9453bd700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9400a6cc0 con 0x7fa9400c0740 2021-02-10T11:49:12.485 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:12.478 7fa93f87e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa9400b6e60 0x7fa9400a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:12.485 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:12.478 7fa93f87e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa9400b6e60 0x7fa9400a6260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:33694/0 (socket says 172.21.2.112:33694) 2021-02-10T11:49:12.485 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:12.478 7fa93f87e700 1 -- 172.21.2.112:0/2882443368 learned_addr learned my addr 172.21.2.112:0/2882443368 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:49:12.485 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:12.478 7fa93f87e700 1 -- 172.21.2.112:0/2882443368 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa9400c0740 msgr2=0x7fa9401c5580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:49:12.485 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:12.478 7fa93f87e700 1 --2- 172.21.2.112:0/2882443368 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa9400c0740 0x7fa9401c5580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:12.485 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:12.478 7fa93f87e700 1 -- 172.21.2.112:0/2882443368 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa9401c5ba0 msgr2=0x7fa9400a5c40 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:49:12.485 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:12.478 7fa93f87e700 1 --2- 172.21.2.112:0/2882443368 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa9401c5ba0 0x7fa9400a5c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:12.485 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:12.478 7fa93f87e700 1 -- 172.21.2.112:0/2882443368 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9400a6ee0 con 0x7fa9400b6e60 2021-02-10T11:49:12.485 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:12.478 7fa93f87e700 1 --2- 172.21.2.112:0/2882443368 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa9400b6e60 0x7fa9400a6260 secure :-1 s=READY pgs=3276 cs=0 l=1 rev1=1 rx=0x7fa93800b9b0 tx=0x7fa93800bd50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:12.485 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:12.479 7fa937fff700 1 -- 172.21.2.112:0/2882443368 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa938005820 con 0x7fa9400b6e60 2021-02-10T11:49:12.485 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:12.479 7fa937fff700 1 -- 172.21.2.112:0/2882443368 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa938005aa0 con 0x7fa9400b6e60 2021-02-10T11:49:12.485 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:12.479 7fa937fff700 1 -- 172.21.2.112:0/2882443368 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa938006030 con 0x7fa9400b6e60 2021-02-10T11:49:12.485 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:12.479 7fa9453bd700 1 -- 172.21.2.112:0/2882443368 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9400a6ee0 con 0x7fa9400b6e60 2021-02-10T11:49:12.486 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:12.479 7fa9453bd700 1 -- 172.21.2.112:0/2882443368 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9400b5650 con 0x7fa9400b6e60 2021-02-10T11:49:12.486 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:12.479 7fa937fff700 1 -- 172.21.2.112:0/2882443368 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa938005c80 con 0x7fa9400b6e60 2021-02-10T11:49:12.486 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:12.479 7fa937fff700 1 --2- 172.21.2.112:0/2882443368 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa9240186c0 0x7fa92401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:12.486 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:12.479 7fa937fff700 1 -- 172.21.2.112:0/2882443368 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fa9380328a0 con 0x7fa9400b6e60 2021-02-10T11:49:12.486 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:12.480 7fa93f07d700 1 --2- 172.21.2.112:0/2882443368 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa9240186c0 0x7fa92401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:12.486 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:12.480 7fa93f07d700 1 --2- 172.21.2.112:0/2882443368 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa9240186c0 0x7fa92401abc0 crc :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:12.515 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:12.511 7fa9453bd700 1 -- 172.21.2.112:0/2882443368 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9400807d0 con 0x7fa9400b6e60 2021-02-10T11:49:12.517 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:12.514 7fa937fff700 1 -- 172.21.2.112:0/2882443368 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa93802df10 con 0x7fa9400b6e60 2021-02-10T11:49:12.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:12.643 7fa9453bd700 1 -- 172.21.2.112:0/2882443368 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7fa940088070 con 0x7fa9400b6e60 2021-02-10T11:49:12.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:12.644 7fa937fff700 1 -- 172.21.2.112:0/2882443368 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v198) v1 ==== 92+0+10 (secure 0 0 0) 0x7fa93802df10 con 0x7fa9400b6e60 2021-02-10T11:49:12.648 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:49:12.649 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:12.646 7fa9453bd700 1 -- 172.21.2.112:0/2882443368 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa9240186c0 msgr2=0x7fa92401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:12.649 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:12.646 7fa9453bd700 1 --2- 172.21.2.112:0/2882443368 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa9240186c0 0x7fa92401abc0 crc :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:12.649 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:12.646 7fa9453bd700 1 -- 172.21.2.112:0/2882443368 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa9400b6e60 msgr2=0x7fa9400a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:12.649 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:12.646 7fa9453bd700 1 --2- 172.21.2.112:0/2882443368 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa9400b6e60 0x7fa9400a6260 secure :-1 s=READY pgs=3276 cs=0 l=1 rev1=1 rx=0x7fa93800b9b0 tx=0x7fa93800bd50).stop 2021-02-10T11:49:12.650 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:12.646 7fa9453bd700 1 -- 172.21.2.112:0/2882443368 shutdown_connections 2021-02-10T11:49:12.650 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:12.646 7fa9453bd700 1 --2- 172.21.2.112:0/2882443368 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa9240186c0 0x7fa92401abc0 unknown :-1 s=CLOSED pgs=1477 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:12.650 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:12.646 7fa9453bd700 1 --2- 172.21.2.112:0/2882443368 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa9401c5ba0 0x7fa9400a5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:12.650 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:12.646 7fa9453bd700 1 --2- 172.21.2.112:0/2882443368 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa9400b6e60 0x7fa9400a6260 unknown :-1 s=CLOSED pgs=3276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:12.650 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:12.646 7fa9453bd700 1 --2- 172.21.2.112:0/2882443368 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa9400c0740 0x7fa9401c5580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:12.650 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:12.646 7fa9453bd700 1 -- 172.21.2.112:0/2882443368 >> 172.21.2.112:0/2882443368 conn(0x7fa9400c4120 msgr2=0x7fa9400ffd60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:12.650 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:12.646 7fa9453bd700 1 -- 172.21.2.112:0/2882443368 shutdown_connections 2021-02-10T11:49:12.650 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:12.646 7fa9453bd700 1 -- 172.21.2.112:0/2882443368 wait complete. 2021-02-10T11:49:12.658 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:49:13.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.070 7f4e3c545700 1 Processor -- start 2021-02-10T11:49:13.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.070 7f4e3c545700 1 -- start start 2021-02-10T11:49:13.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.070 7f4e3c545700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4e340d1870 0x7f4e341b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:13.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.071 7f4e3c545700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f4e341b0ae0 0x7f4e34108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:13.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.071 7f4e3c545700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e34109a60 con 0x7f4e340d1c70 2021-02-10T11:49:13.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.071 7f4e3c545700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e340e5ef0 con 0x7f4e341b0ae0 2021-02-10T11:49:13.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.071 7f4e3c545700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e34109750 con 0x7f4e340d1870 2021-02-10T11:49:13.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.071 7f4e39b60700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f4e341b0ae0 0x7f4e34108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:13.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.071 7f4e39b60700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f4e341b0ae0 0x7f4e34108940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3300/0 says I am v2:172.21.2.112:35980/0 (socket says 172.21.2.112:35980) 2021-02-10T11:49:13.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.071 7f4e3ab62700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7f4e340d1c70 0x7f4e34108f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:54952/0 (socket says 172.21.2.112:54952) 2021-02-10T11:49:13.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.071 7f4e3a361700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4e340d1870 0x7f4e341b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:13.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.072 7f4e39b60700 1 -- 172.21.2.112:0/2925655377 learned_addr learned my addr 172.21.2.112:0/2925655377 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:49:13.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.072 7f4e3a361700 1 -- 172.21.2.112:0/2925655377 >> v2:172.21.2.120:3300/0 conn(0x7f4e341b0ae0 msgr2=0x7f4e34108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:49:13.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.072 7f4e3a361700 1 --2- 172.21.2.112:0/2925655377 >> v2:172.21.2.120:3300/0 conn(0x7f4e341b0ae0 0x7f4e34108940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:13.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.072 7f4e3a361700 1 -- 172.21.2.112:0/2925655377 >> v1:172.21.2.120:6789/0 conn(0x7f4e340d1c70 legacy=0x7f4e34108f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:13.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.072 7f4e3a361700 1 -- 172.21.2.112:0/2925655377 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e34088070 con 0x7f4e340d1870 2021-02-10T11:49:13.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.072 7f4e3a361700 1 --2- 172.21.2.112:0/2925655377 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4e340d1870 0x7f4e341b04c0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 rx=0x7f4e24006650 tx=0x7f4e24009a80).ready entity=mon.2 client_cookie=da4d34751c2cbae1 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:13.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.072 7f4e3935f700 1 -- 172.21.2.112:0/2925655377 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4e240009b0 con 0x7f4e340d1870 2021-02-10T11:49:13.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.072 7f4e3935f700 1 -- 172.21.2.112:0/2925655377 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e2400c0c0 con 0x7f4e340d1870 2021-02-10T11:49:13.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.072 7f4e3935f700 1 -- 172.21.2.112:0/2925655377 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4e2400c520 con 0x7f4e340d1870 2021-02-10T11:49:13.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.073 7f4e3c545700 1 -- 172.21.2.112:0/2925655377 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4e340d1870 msgr2=0x7f4e341b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:13.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.073 7f4e3c545700 1 --2- 172.21.2.112:0/2925655377 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4e340d1870 0x7f4e341b04c0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 rx=0x7f4e24006650 tx=0x7f4e24009a80).stop 2021-02-10T11:49:13.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.073 7f4e3c545700 1 -- 172.21.2.112:0/2925655377 shutdown_connections 2021-02-10T11:49:13.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.073 7f4e3c545700 1 --2- 172.21.2.112:0/2925655377 >> v2:172.21.2.120:3300/0 conn(0x7f4e341b0ae0 0x7f4e34108940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:13.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.073 7f4e3c545700 1 --2- 172.21.2.112:0/2925655377 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4e340d1870 0x7f4e341b04c0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:13.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.073 7f4e3c545700 1 -- 172.21.2.112:0/2925655377 >> 172.21.2.112:0/2925655377 conn(0x7f4e340ce420 msgr2=0x7f4e340d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:13.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.073 7f4e3c545700 1 -- 172.21.2.112:0/2925655377 shutdown_connections 2021-02-10T11:49:13.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.073 7f4e3c545700 1 -- 172.21.2.112:0/2925655377 wait complete. 2021-02-10T11:49:13.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.088 7f4e3c545700 1 Processor -- start 2021-02-10T11:49:13.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.088 7f4e3c545700 1 -- start start 2021-02-10T11:49:13.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.088 7f4e3c545700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4e340a58c0 0x7f4e340a4d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:13.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.088 7f4e3c545700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4e340b1ba0 0x7f4e340b1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:13.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.088 7f4e3c545700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4e340b25c0 0x7f4e340b3270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:13.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.088 7f4e3c545700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e340b2a40 con 0x7f4e340b25c0 2021-02-10T11:49:13.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.088 7f4e3c545700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e340a5340 con 0x7f4e340a58c0 2021-02-10T11:49:13.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.088 7f4e3c545700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e340a5560 con 0x7f4e340b1ba0 2021-02-10T11:49:13.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.088 7f4e3ab62700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4e340b25c0 0x7f4e340b3270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:13.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.088 7f4e3ab62700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4e340b25c0 0x7f4e340b3270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:33708/0 (socket says 172.21.2.112:33708) 2021-02-10T11:49:13.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.088 7f4e3ab62700 1 -- 172.21.2.112:0/2654471194 learned_addr learned my addr 172.21.2.112:0/2654471194 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:49:13.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.088 7f4e3a361700 1 --2- 172.21.2.112:0/2654471194 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4e340a58c0 0x7f4e340a4d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:13.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.089 7f4e39b60700 1 --2- 172.21.2.112:0/2654471194 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4e340b1ba0 0x7f4e340b1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:13.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.089 7f4e3ab62700 1 -- 172.21.2.112:0/2654471194 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4e340b1ba0 msgr2=0x7f4e340b1fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:13.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.089 7f4e3ab62700 1 --2- 172.21.2.112:0/2654471194 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4e340b1ba0 0x7f4e340b1fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:13.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.089 7f4e3ab62700 1 -- 172.21.2.112:0/2654471194 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4e340a58c0 msgr2=0x7f4e340a4d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:13.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.089 7f4e3ab62700 1 --2- 172.21.2.112:0/2654471194 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4e340a58c0 0x7f4e340a4d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:13.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.089 7f4e3ab62700 1 -- 172.21.2.112:0/2654471194 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e340b3890 con 0x7f4e340b25c0 2021-02-10T11:49:13.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.089 7f4e3a361700 1 --2- 172.21.2.112:0/2654471194 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4e340a58c0 0x7f4e340a4d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:49:13.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.089 7f4e3ab62700 1 --2- 172.21.2.112:0/2654471194 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4e340b25c0 0x7f4e340b3270 secure :-1 s=READY pgs=3277 cs=0 l=1 rev1=1 rx=0x7f4e30001450 tx=0x7f4e3000a930).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:13.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.089 7f4e2affd700 1 -- 172.21.2.112:0/2654471194 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4e3000ad20 con 0x7f4e340b25c0 2021-02-10T11:49:13.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.089 7f4e2affd700 1 -- 172.21.2.112:0/2654471194 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e30007590 con 0x7f4e340b25c0 2021-02-10T11:49:13.099 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.089 7f4e2affd700 1 -- 172.21.2.112:0/2654471194 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4e30004510 con 0x7f4e340b25c0 2021-02-10T11:49:13.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.089 7f4e3c545700 1 -- 172.21.2.112:0/2654471194 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e340b3890 con 0x7f4e340b25c0 2021-02-10T11:49:13.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.089 7f4e3c545700 1 -- 172.21.2.112:0/2654471194 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e340b3890 con 0x7f4e340b25c0 2021-02-10T11:49:13.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.090 7f4e2affd700 1 -- 172.21.2.112:0/2654471194 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4e30004530 con 0x7f4e340b25c0 2021-02-10T11:49:13.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.090 7f4e2affd700 1 --2- 172.21.2.112:0/2654471194 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4e1c0186d0 0x7f4e1c01abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:13.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.090 7f4e2affd700 1 -- 172.21.2.112:0/2654471194 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f4e30032620 con 0x7f4e340b25c0 2021-02-10T11:49:13.100 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.090 7f4e3a361700 1 --2- 172.21.2.112:0/2654471194 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4e1c0186d0 0x7f4e1c01abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:13.101 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.091 7f4e3a361700 1 --2- 172.21.2.112:0/2654471194 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4e1c0186d0 0x7f4e1c01abd0 crc :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:13.124 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.120 7f4e3c545700 1 -- 172.21.2.112:0/2654471194 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e34082980 con 0x7f4e340b25c0 2021-02-10T11:49:13.126 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.123 7f4e2affd700 1 -- 172.21.2.112:0/2654471194 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4e30004320 con 0x7f4e340b25c0 2021-02-10T11:49:13.257 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.253 7f4e3c545700 1 -- 172.21.2.112:0/2654471194 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f4e340b3890 con 0x7f4e340b25c0 2021-02-10T11:49:13.258 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.254 7f4e2affd700 1 -- 172.21.2.112:0/2654471194 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v198) v1 ==== 104+0+10 (secure 0 0 0) 0x7f4e30004530 con 0x7f4e340b25c0 2021-02-10T11:49:13.258 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:49:13.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.256 7f4e3c545700 1 -- 172.21.2.112:0/2654471194 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4e1c0186d0 msgr2=0x7f4e1c01abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:13.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.256 7f4e3c545700 1 --2- 172.21.2.112:0/2654471194 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4e1c0186d0 0x7f4e1c01abd0 crc :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:13.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.256 7f4e3c545700 1 -- 172.21.2.112:0/2654471194 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4e340b25c0 msgr2=0x7f4e340b3270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:13.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.256 7f4e3c545700 1 --2- 172.21.2.112:0/2654471194 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4e340b25c0 0x7f4e340b3270 secure :-1 s=READY pgs=3277 cs=0 l=1 rev1=1 rx=0x7f4e30001450 tx=0x7f4e3000a930).stop 2021-02-10T11:49:13.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.256 7f4e3c545700 1 -- 172.21.2.112:0/2654471194 shutdown_connections 2021-02-10T11:49:13.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.256 7f4e3c545700 1 --2- 172.21.2.112:0/2654471194 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4e1c0186d0 0x7f4e1c01abd0 unknown :-1 s=CLOSED pgs=1478 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:13.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.256 7f4e3c545700 1 --2- 172.21.2.112:0/2654471194 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4e340b25c0 0x7f4e340b3270 unknown :-1 s=CLOSED pgs=3277 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:13.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.256 7f4e3c545700 1 --2- 172.21.2.112:0/2654471194 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4e340b1ba0 0x7f4e340b1fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:13.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.256 7f4e3c545700 1 --2- 172.21.2.112:0/2654471194 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4e340a58c0 0x7f4e340a4d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:13.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.256 7f4e3c545700 1 -- 172.21.2.112:0/2654471194 >> 172.21.2.112:0/2654471194 conn(0x7f4e340b5dc0 msgr2=0x7f4e34095960 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:13.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.256 7f4e3c545700 1 -- 172.21.2.112:0/2654471194 shutdown_connections 2021-02-10T11:49:13.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.256 7f4e3c545700 1 -- 172.21.2.112:0/2654471194 wait complete. 2021-02-10T11:49:13.268 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:49:13.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.680 7f31a79e6700 1 Processor -- start 2021-02-10T11:49:13.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.680 7f31a79e6700 1 -- start start 2021-02-10T11:49:13.701 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.681 7f31a79e6700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f31a00d1870 0x7f31a01b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:13.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.681 7f31a79e6700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f31a00cff70 0x7f31a0109130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:13.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.681 7f31a79e6700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31a0109a60 con 0x7f31a01b0ae0 2021-02-10T11:49:13.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.681 7f31a79e6700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31a0109750 con 0x7f31a00d1870 2021-02-10T11:49:13.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.681 7f31a79e6700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31a0088070 con 0x7f31a00cff70 2021-02-10T11:49:13.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.681 7f31a5802700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f31a00d1870 0x7f31a01b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:13.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.681 7f31a5001700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f31a01b0ae0 0x7f31a0108940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:33174/0 (socket says 172.21.2.112:33174) 2021-02-10T11:49:13.702 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.681 7f31a5001700 1 -- 172.21.2.112:0/3683281348 learned_addr learned my addr 172.21.2.112:0/3683281348 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:49:13.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.681 7f31a4800700 1 -- 172.21.2.112:0/3683281348 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1164296726 0 0) 0x7f31940010b0 con 0x7f31a01b0ae0 2021-02-10T11:49:13.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.681 7f31a4800700 1 -- 172.21.2.112:0/3683281348 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f318c002fe0 con 0x7f31a01b0ae0 2021-02-10T11:49:13.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.681 7f31a5802700 1 -- 172.21.2.112:0/3683281348 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f31a00cff70 msgr2=0x7f31a0109130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:49:13.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.682 7f31a5802700 1 --2- 172.21.2.112:0/3683281348 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f31a00cff70 0x7f31a0109130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:13.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.682 7f31a5802700 1 -- 172.21.2.112:0/3683281348 >> v1:172.21.2.112:6789/0 conn(0x7f31a01b0ae0 legacy=0x7f31a0108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:13.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.682 7f31a5802700 1 -- 172.21.2.112:0/3683281348 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31a00d1c70 con 0x7f31a00d1870 2021-02-10T11:49:13.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.682 7f31a5802700 1 --2- 172.21.2.112:0/3683281348 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f31a00d1870 0x7f31a01b04c0 secure :-1 s=READY pgs=3279 cs=0 l=1 rev1=1 rx=0x7f319c006650 tx=0x7f319c009a80).ready entity=mon.0 client_cookie=7a388a14c3026eaf server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:13.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.682 7f31a4800700 1 -- 172.21.2.112:0/3683281348 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f319c00c0c0 con 0x7f31a00d1870 2021-02-10T11:49:13.703 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.682 7f31a4800700 1 -- 172.21.2.112:0/3683281348 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f319c00c340 con 0x7f31a00d1870 2021-02-10T11:49:13.704 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.682 7f31a4800700 1 -- 172.21.2.112:0/3683281348 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f319c00c860 con 0x7f31a00d1870 2021-02-10T11:49:13.704 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.682 7f31a79e6700 1 -- 172.21.2.112:0/3683281348 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f31a00d1870 msgr2=0x7f31a01b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:13.704 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.682 7f31a79e6700 1 --2- 172.21.2.112:0/3683281348 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f31a00d1870 0x7f31a01b04c0 secure :-1 s=READY pgs=3279 cs=0 l=1 rev1=1 rx=0x7f319c006650 tx=0x7f319c009a80).stop 2021-02-10T11:49:13.704 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.683 7f31a79e6700 1 -- 172.21.2.112:0/3683281348 shutdown_connections 2021-02-10T11:49:13.704 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.683 7f31a79e6700 1 --2- 172.21.2.112:0/3683281348 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f31a00cff70 0x7f31a0109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:13.704 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.683 7f31a79e6700 1 --2- 172.21.2.112:0/3683281348 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f31a00d1870 0x7f31a01b04c0 unknown :-1 s=CLOSED pgs=3279 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:13.704 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.683 7f31a79e6700 1 -- 172.21.2.112:0/3683281348 >> 172.21.2.112:0/3683281348 conn(0x7f31a00ce420 msgr2=0x7f31a00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:13.704 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.683 7f31a79e6700 1 -- 172.21.2.112:0/3683281348 shutdown_connections 2021-02-10T11:49:13.704 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.683 7f31a79e6700 1 -- 172.21.2.112:0/3683281348 wait complete. 2021-02-10T11:49:13.704 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.697 7f31a79e6700 1 Processor -- start 2021-02-10T11:49:13.704 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.698 7f31a79e6700 1 -- start start 2021-02-10T11:49:13.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.698 7f31a79e6700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f31a01a79a0 0x7f31a01a7da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:13.705 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.698 7f31a79e6700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f31a00bdb40 0x7f31a00bdf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:13.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.698 7f31a79e6700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f31a0082ca0 0x7f31a00830a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:13.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.698 7f31a79e6700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31a019d990 con 0x7f31a00bdb40 2021-02-10T11:49:13.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.698 7f31a79e6700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31a019dbb0 con 0x7f31a0082ca0 2021-02-10T11:49:13.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.698 7f31a79e6700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31a00836c0 con 0x7f31a01a79a0 2021-02-10T11:49:13.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.698 7f31a5001700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f31a00bdb40 0x7f31a00bdf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:13.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.698 7f31a5001700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f31a00bdb40 0x7f31a00bdf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:33720/0 (socket says 172.21.2.112:33720) 2021-02-10T11:49:13.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.698 7f31a5001700 1 -- 172.21.2.112:0/2436197978 learned_addr learned my addr 172.21.2.112:0/2436197978 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:49:13.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.698 7f31a5001700 1 -- 172.21.2.112:0/2436197978 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f31a01a79a0 msgr2=0x7f31a01a7da0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:49:13.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.698 7f31a5001700 1 --2- 172.21.2.112:0/2436197978 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f31a01a79a0 0x7f31a01a7da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:13.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.698 7f31a5001700 1 -- 172.21.2.112:0/2436197978 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f31a0082ca0 msgr2=0x7f31a00830a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:49:13.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.698 7f31a5001700 1 --2- 172.21.2.112:0/2436197978 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f31a0082ca0 0x7f31a00830a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:13.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.698 7f31a5001700 1 -- 172.21.2.112:0/2436197978 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31a00838e0 con 0x7f31a00bdb40 2021-02-10T11:49:13.707 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.698 7f31a5001700 1 --2- 172.21.2.112:0/2436197978 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f31a00bdb40 0x7f31a00bdf40 secure :-1 s=READY pgs=3280 cs=0 l=1 rev1=1 rx=0x7f3194001c60 tx=0x7f319400a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:13.707 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.699 7f31927fc700 1 -- 172.21.2.112:0/2436197978 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f319400acc0 con 0x7f31a00bdb40 2021-02-10T11:49:13.707 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.699 7f31927fc700 1 -- 172.21.2.112:0/2436197978 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3194004140 con 0x7f31a00bdb40 2021-02-10T11:49:13.707 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.699 7f31927fc700 1 -- 172.21.2.112:0/2436197978 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3194004660 con 0x7f31a00bdb40 2021-02-10T11:49:13.707 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.699 7f31a79e6700 1 -- 172.21.2.112:0/2436197978 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31a00838e0 con 0x7f31a00bdb40 2021-02-10T11:49:13.707 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.699 7f31a79e6700 1 -- 172.21.2.112:0/2436197978 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31a00838e0 con 0x7f31a00bdb40 2021-02-10T11:49:13.707 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.699 7f31927fc700 1 -- 172.21.2.112:0/2436197978 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3194004320 con 0x7f31a00bdb40 2021-02-10T11:49:13.707 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.699 7f31927fc700 1 --2- 172.21.2.112:0/2436197978 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f31880186c0 0x7f318801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:13.707 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.699 7f31927fc700 1 -- 172.21.2.112:0/2436197978 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f3194032780 con 0x7f31a00bdb40 2021-02-10T11:49:13.707 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.700 7f31a5802700 1 --2- 172.21.2.112:0/2436197978 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f31880186c0 0x7f318801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:13.707 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.701 7f31a5802700 1 --2- 172.21.2.112:0/2436197978 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f31880186c0 0x7f318801abc0 crc :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:13.734 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.730 7f31a79e6700 1 -- 172.21.2.112:0/2436197978 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31a0082a70 con 0x7f31a00bdb40 2021-02-10T11:49:13.736 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.733 7f31927fc700 1 -- 172.21.2.112:0/2436197978 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3194004350 con 0x7f31a00bdb40 2021-02-10T11:49:13.867 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.863 7f31a79e6700 1 -- 172.21.2.112:0/2436197978 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f31a0082a70 con 0x7f31a00bdb40 2021-02-10T11:49:13.867 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.864 7f31927fc700 1 -- 172.21.2.112:0/2436197978 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v198) v1 ==== 100+0+10 (secure 0 0 0) 0x7f3194004140 con 0x7f31a00bdb40 2021-02-10T11:49:13.868 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:49:13.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.866 7f31a79e6700 1 -- 172.21.2.112:0/2436197978 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f31880186c0 msgr2=0x7f318801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:13.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.866 7f31a79e6700 1 --2- 172.21.2.112:0/2436197978 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f31880186c0 0x7f318801abc0 crc :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:13.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.866 7f31a79e6700 1 -- 172.21.2.112:0/2436197978 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f31a00bdb40 msgr2=0x7f31a00bdf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:13.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.866 7f31a79e6700 1 --2- 172.21.2.112:0/2436197978 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f31a00bdb40 0x7f31a00bdf40 secure :-1 s=READY pgs=3280 cs=0 l=1 rev1=1 rx=0x7f3194001c60 tx=0x7f319400a8d0).stop 2021-02-10T11:49:13.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.866 7f31a79e6700 1 -- 172.21.2.112:0/2436197978 shutdown_connections 2021-02-10T11:49:13.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.866 7f31a79e6700 1 --2- 172.21.2.112:0/2436197978 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f31880186c0 0x7f318801abc0 unknown :-1 s=CLOSED pgs=1479 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:13.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.866 7f31a79e6700 1 --2- 172.21.2.112:0/2436197978 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f31a0082ca0 0x7f31a00830a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:13.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.866 7f31a79e6700 1 --2- 172.21.2.112:0/2436197978 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f31a00bdb40 0x7f31a00bdf40 unknown :-1 s=CLOSED pgs=3280 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:13.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.866 7f31a79e6700 1 --2- 172.21.2.112:0/2436197978 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f31a01a79a0 0x7f31a01a7da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:13.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.866 7f31a79e6700 1 -- 172.21.2.112:0/2436197978 >> 172.21.2.112:0/2436197978 conn(0x7f31a00bef00 msgr2=0x7f31a009f200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:13.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.866 7f31a79e6700 1 -- 172.21.2.112:0/2436197978 shutdown_connections 2021-02-10T11:49:13.871 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:13.866 7f31a79e6700 1 -- 172.21.2.112:0/2436197978 wait complete. 2021-02-10T11:49:13.879 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:49:13.879 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:49:14.308 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:14.288 7f32825e7700 1 Processor -- start 2021-02-10T11:49:14.308 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:14.288 7f32825e7700 1 -- start start 2021-02-10T11:49:14.308 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:14.289 7f32825e7700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f327c0d1870 0x7f327c1b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:14.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:14.289 7f32825e7700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f327c0f0650 0x7f327c0cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:14.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:14.289 7f32825e7700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f327c0f0b10 con 0x7f327c1b0a10 2021-02-10T11:49:14.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:14.289 7f32825e7700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f327c0e5ed0 con 0x7f327c0d1870 2021-02-10T11:49:14.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:14.289 7f32825e7700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f327c0d1c70 con 0x7f327c0f0650 2021-02-10T11:49:14.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:14.289 7f327bfff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f327c0d1870 0x7f327c1b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:14.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:14.289 7f327b7fe700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f327c1b0a10 0x7f327c0cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:33188/0 (socket says 172.21.2.112:33188) 2021-02-10T11:49:14.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:14.289 7f327b7fe700 1 -- 172.21.2.112:0/4030096280 learned_addr learned my addr 172.21.2.112:0/4030096280 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:49:14.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:14.289 7f3280c04700 1 --2- 172.21.2.112:0/4030096280 >> v2:172.21.2.120:3300/0 conn(0x7f327c0f0650 0x7f327c0cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:14.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:14.289 7f327bfff700 1 -- 172.21.2.112:0/4030096280 >> v2:172.21.2.120:3300/0 conn(0x7f327c0f0650 msgr2=0x7f327c0cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:49:14.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:14.289 7f327bfff700 1 --2- 172.21.2.112:0/4030096280 >> v2:172.21.2.120:3300/0 conn(0x7f327c0f0650 0x7f327c0cfb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:14.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:14.289 7f327bfff700 1 -- 172.21.2.112:0/4030096280 >> v1:172.21.2.112:6789/0 conn(0x7f327c1b0a10 legacy=0x7f327c0cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:14.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:14.289 7f327bfff700 1 -- 172.21.2.112:0/4030096280 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f327c0d1e90 con 0x7f327c0d1870 2021-02-10T11:49:14.309 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:14.289 7f3280c04700 1 --2- 172.21.2.112:0/4030096280 >> v2:172.21.2.120:3300/0 conn(0x7f327c0f0650 0x7f327c0cfb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:49:14.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:14.290 7f327bfff700 1 --2- 172.21.2.112:0/4030096280 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f327c0d1870 0x7f327c1b03f0 secure :-1 s=READY pgs=3282 cs=0 l=1 rev1=1 rx=0x7f326c006650 tx=0x7f326c009a80).ready entity=mon.0 client_cookie=37c3b568c467e7da server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:14.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:14.290 7f327affd700 1 -- 172.21.2.112:0/4030096280 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f326c00c0c0 con 0x7f327c0d1870 2021-02-10T11:49:14.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:14.290 7f327affd700 1 -- 172.21.2.112:0/4030096280 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f326c00c340 con 0x7f327c0d1870 2021-02-10T11:49:14.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:14.290 7f327affd700 1 -- 172.21.2.112:0/4030096280 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f326c00c860 con 0x7f327c0d1870 2021-02-10T11:49:14.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:14.290 7f32825e7700 1 -- 172.21.2.112:0/4030096280 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f327c0d1870 msgr2=0x7f327c1b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:14.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:14.290 7f32825e7700 1 --2- 172.21.2.112:0/4030096280 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f327c0d1870 0x7f327c1b03f0 secure :-1 s=READY pgs=3282 cs=0 l=1 rev1=1 rx=0x7f326c006650 tx=0x7f326c009a80).stop 2021-02-10T11:49:14.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:14.290 7f32825e7700 1 -- 172.21.2.112:0/4030096280 shutdown_connections 2021-02-10T11:49:14.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:14.290 7f32825e7700 1 --2- 172.21.2.112:0/4030096280 >> v2:172.21.2.120:3300/0 conn(0x7f327c0f0650 0x7f327c0cfb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:14.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:14.290 7f32825e7700 1 --2- 172.21.2.112:0/4030096280 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f327c0d1870 0x7f327c1b03f0 unknown :-1 s=CLOSED pgs=3282 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:14.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:14.290 7f32825e7700 1 -- 172.21.2.112:0/4030096280 >> 172.21.2.112:0/4030096280 conn(0x7f327c0ce450 msgr2=0x7f327c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:14.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:14.290 7f32825e7700 1 -- 172.21.2.112:0/4030096280 shutdown_connections 2021-02-10T11:49:14.310 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:14.290 7f32825e7700 1 -- 172.21.2.112:0/4030096280 wait complete. 2021-02-10T11:49:14.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:14.305 7f32825e7700 1 Processor -- start 2021-02-10T11:49:14.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:14.305 7f32825e7700 1 -- start start 2021-02-10T11:49:14.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:14.305 7f32825e7700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f327c0f0280 0x7f327c0a2310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:14.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:14.305 7f32825e7700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f327c0f0680 0x7f327c0a2930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:14.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:14.305 7f32825e7700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f327c0ec180 0x7f327c0ec580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:14.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:14.305 7f32825e7700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f327c0efd00 con 0x7f327c0ec180 2021-02-10T11:49:14.312 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:14.305 7f32825e7700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f327c0ecba0 con 0x7f327c0f0680 2021-02-10T11:49:14.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:14.305 7f32825e7700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f327c0ecd80 con 0x7f327c0f0280 2021-02-10T11:49:14.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:14.306 7f3280c04700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f327c0ec180 0x7f327c0ec580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:14.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:14.306 7f3280c04700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f327c0ec180 0x7f327c0ec580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:33730/0 (socket says 172.21.2.112:33730) 2021-02-10T11:49:14.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:14.306 7f3280c04700 1 -- 172.21.2.112:0/1845953063 learned_addr learned my addr 172.21.2.112:0/1845953063 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:49:14.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:14.306 7f327b7fe700 1 --2- 172.21.2.112:0/1845953063 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f327c0f0680 0x7f327c0a2930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:14.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:14.306 7f327bfff700 1 --2- 172.21.2.112:0/1845953063 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f327c0f0280 0x7f327c0a2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:14.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:14.306 7f3280c04700 1 -- 172.21.2.112:0/1845953063 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f327c0f0280 msgr2=0x7f327c0a2310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:14.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:14.306 7f3280c04700 1 --2- 172.21.2.112:0/1845953063 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f327c0f0280 0x7f327c0a2310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:14.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:14.306 7f3280c04700 1 -- 172.21.2.112:0/1845953063 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f327c0f0680 msgr2=0x7f327c0a2930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:14.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:14.306 7f3280c04700 1 --2- 172.21.2.112:0/1845953063 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f327c0f0680 0x7f327c0a2930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:14.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:14.306 7f3280c04700 1 -- 172.21.2.112:0/1845953063 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f327c0ecfa0 con 0x7f327c0ec180 2021-02-10T11:49:14.313 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:14.306 7f3280c04700 1 --2- 172.21.2.112:0/1845953063 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f327c0ec180 0x7f327c0ec580 secure :-1 s=READY pgs=3283 cs=0 l=1 rev1=1 rx=0x7f327400b970 tx=0x7f327400bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:14.314 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:14.307 7f3278ff9700 1 -- 172.21.2.112:0/1845953063 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3274000f90 con 0x7f327c0ec180 2021-02-10T11:49:14.314 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:14.307 7f3278ff9700 1 -- 172.21.2.112:0/1845953063 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3274006aa0 con 0x7f327c0ec180 2021-02-10T11:49:14.314 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:14.307 7f3278ff9700 1 -- 172.21.2.112:0/1845953063 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3274001680 con 0x7f327c0ec180 2021-02-10T11:49:14.314 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:14.307 7f32825e7700 1 -- 172.21.2.112:0/1845953063 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f327c0ecfa0 con 0x7f327c0ec180 2021-02-10T11:49:14.314 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:14.307 7f32825e7700 1 -- 172.21.2.112:0/1845953063 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f327c0ed530 con 0x7f327c0ec180 2021-02-10T11:49:14.314 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:14.307 7f3278ff9700 1 -- 172.21.2.112:0/1845953063 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f32740011f0 con 0x7f327c0ec180 2021-02-10T11:49:14.314 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:14.307 7f3278ff9700 1 --2- 172.21.2.112:0/1845953063 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f325c0186c0 0x7f325c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:14.314 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:14.307 7f3278ff9700 1 -- 172.21.2.112:0/1845953063 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f32740328f0 con 0x7f327c0ec180 2021-02-10T11:49:14.314 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:14.308 7f327bfff700 1 --2- 172.21.2.112:0/1845953063 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f325c0186c0 0x7f325c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:14.314 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:14.308 7f327bfff700 1 --2- 172.21.2.112:0/1845953063 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f325c0186c0 0x7f325c01abc0 crc :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:14.345 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:14.339 7f32825e7700 1 -- 172.21.2.112:0/1845953063 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f327c0ebc60 con 0x7f327c0ec180 2021-02-10T11:49:14.346 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:14.341 7f3278ff9700 1 -- 172.21.2.112:0/1845953063 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f327401ca60 con 0x7f327c0ec180 2021-02-10T11:49:14.473 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:14.469 7f32825e7700 1 -- 172.21.2.112:0/1845953063 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f327c0f0b10 con 0x7f327c0ec180 2021-02-10T11:49:14.473 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:14.469 7f3278ff9700 1 -- 172.21.2.112:0/1845953063 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 174 v174) v1 ==== 95+0+2108 (secure 0 0 0) 0x7f327401ca60 con 0x7f327c0ec180 2021-02-10T11:49:14.474 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:49:14.474 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":174,"default_fscid":21,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":13800,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/2202619473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":2202619473},{"type":"v1","addr":"172.21.2.112:6833","nonce":2202619473}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":170}],"filesystems":[{"mdsmap":{"epoch":174,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:49:04.478835","modified":"2021-02-10 11:49:06.484559","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9715},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9715":{"gid":9715,"name":"b","rank":0,"incarnation":171,"state":"up:active","state_seq":3,"addr":"172.21.2.120:6835/3826510568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":3826510568},{"type":"v1","addr":"172.21.2.120:6835","nonce":3826510568}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[43],"metadata_pool":42,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":21}]} 2021-02-10T11:49:14.474 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 174 2021-02-10T11:49:14.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:14.471 7f32825e7700 1 -- 172.21.2.112:0/1845953063 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f325c0186c0 msgr2=0x7f325c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:14.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:14.471 7f32825e7700 1 --2- 172.21.2.112:0/1845953063 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f325c0186c0 0x7f325c01abc0 crc :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:14.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:14.471 7f32825e7700 1 -- 172.21.2.112:0/1845953063 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f327c0ec180 msgr2=0x7f327c0ec580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:14.475 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:14.471 7f32825e7700 1 --2- 172.21.2.112:0/1845953063 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f327c0ec180 0x7f327c0ec580 secure :-1 s=READY pgs=3283 cs=0 l=1 rev1=1 rx=0x7f327400b970 tx=0x7f327400bd10).stop 2021-02-10T11:49:14.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:14.471 7f32825e7700 1 -- 172.21.2.112:0/1845953063 shutdown_connections 2021-02-10T11:49:14.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:14.471 7f32825e7700 1 --2- 172.21.2.112:0/1845953063 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f325c0186c0 0x7f325c01abc0 unknown :-1 s=CLOSED pgs=1480 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:14.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:14.471 7f32825e7700 1 --2- 172.21.2.112:0/1845953063 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f327c0f0680 0x7f327c0a2930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:14.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:14.471 7f32825e7700 1 --2- 172.21.2.112:0/1845953063 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f327c0ec180 0x7f327c0ec580 unknown :-1 s=CLOSED pgs=3283 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:14.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:14.471 7f32825e7700 1 --2- 172.21.2.112:0/1845953063 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f327c0f0280 0x7f327c0a2310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:14.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:14.471 7f32825e7700 1 -- 172.21.2.112:0/1845953063 >> 172.21.2.112:0/1845953063 conn(0x7f327c0c2130 msgr2=0x7f327c09e3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:14.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:14.472 7f32825e7700 1 -- 172.21.2.112:0/1845953063 shutdown_connections 2021-02-10T11:49:14.476 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:14.472 7f32825e7700 1 -- 172.21.2.112:0/1845953063 wait complete. 2021-02-10T11:49:14.687 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 174, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:49:04.478835', 'modified': '2021-02-10 11:49:06.484559', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9715}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9715': {'gid': 9715, 'name': 'b', 'rank': 0, 'incarnation': 171, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.120:6835/3826510568', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.120:6834', 'nonce': 3826510568}, {'type': 'v1', 'addr': '172.21.2.120:6835', 'nonce': 3826510568}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [43], 'metadata_pool': 42, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:49:14.687 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:49:14.687 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:49:15.087 INFO:teuthology.orchestra.run.gibba020.stdout:{ 2021-02-10T11:49:15.087 INFO:teuthology.orchestra.run.gibba020.stdout: "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", 2021-02-10T11:49:15.087 INFO:teuthology.orchestra.run.gibba020.stdout: "whoami": 0, 2021-02-10T11:49:15.087 INFO:teuthology.orchestra.run.gibba020.stdout: "id": 9715, 2021-02-10T11:49:15.087 INFO:teuthology.orchestra.run.gibba020.stdout: "want_state": "up:active", 2021-02-10T11:49:15.088 INFO:teuthology.orchestra.run.gibba020.stdout: "state": "up:active", 2021-02-10T11:49:15.088 INFO:teuthology.orchestra.run.gibba020.stdout: "rank_uptime": 10.602183303, 2021-02-10T11:49:15.088 INFO:teuthology.orchestra.run.gibba020.stdout: "mdsmap_epoch": 174, 2021-02-10T11:49:15.088 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch": 198, 2021-02-10T11:49:15.088 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch_barrier": 198, 2021-02-10T11:49:15.089 INFO:teuthology.orchestra.run.gibba020.stdout: "uptime": 16.252351401999999 2021-02-10T11:49:15.089 INFO:teuthology.orchestra.run.gibba020.stdout:} 2021-02-10T11:49:15.091 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", "whoami": 0, "id": 9715, "want_state": "up:active", "state": "up:active", "rank_uptime": 10.602183303, "mdsmap_epoch": 174, "osdmap_epoch": 198, "osdmap_epoch_barrier": 198, "uptime": 16.252351401999999 } 2021-02-10T11:49:15.091 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:49:15.092 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2021-02-10T11:49:15.092 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@gibba012.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2021-02-10T11:49:15.092 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:49:15.092 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:49:15.429 DEBUG:teuthology.orchestra.run.gibba012:> sudo modprobe fuse 2021-02-10T11:49:15.775 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:49:15.775 DEBUG:teuthology.orchestra.run.gibba012:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:49:16.119 INFO:teuthology.orchestra.run.gibba012.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:49:16.120 INFO:teuthology.orchestra.run.gibba012.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:49:16.121 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:49:16.121 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:49:16.121 DEBUG:teuthology.orchestra.run.gibba012:> ls /sys/fs/fuse/connections 2021-02-10T11:49:16.457 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2021-02-10T11:49:16.457 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:49:16.460 DEBUG:teuthology.orchestra.run.gibba012:> sudo modprobe fuse 2021-02-10T11:49:16.804 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:49:16.804 DEBUG:teuthology.orchestra.run.gibba012:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:49:16.870 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba012.stderr:2021-02-10 11:49:16.867 7f8424059f80 -1 init, newargv = 0x557c58518ba0 newargc=9 2021-02-10T11:49:16.870 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba012.stderr:ceph-fuse[196573]: starting ceph client 2021-02-10T11:49:16.875 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba012.stderr:ceph-fuse[196573]: starting fuse 2021-02-10T11:49:17.146 INFO:teuthology.orchestra.run.gibba012.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:49:17.147 INFO:teuthology.orchestra.run.gibba012.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:49:17.147 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:49:17.148 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:49:17.148 DEBUG:teuthology.orchestra.run.gibba012:> ls /sys/fs/fuse/connections 2021-02-10T11:49:17.483 INFO:teuthology.orchestra.run.gibba012.stdout:42 2021-02-10T11:49:17.483 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2021-02-10T11:49:17.483 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:49:17.484 DEBUG:teuthology.orchestra.run.gibba012:> sudo python3 -c ' 2021-02-10T11:49:17.484 DEBUG:teuthology.orchestra.run.gibba012:> import glob 2021-02-10T11:49:17.484 DEBUG:teuthology.orchestra.run.gibba012:> import re 2021-02-10T11:49:17.484 DEBUG:teuthology.orchestra.run.gibba012:> import os 2021-02-10T11:49:17.484 DEBUG:teuthology.orchestra.run.gibba012:> import subprocess 2021-02-10T11:49:17.484 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:49:17.484 DEBUG:teuthology.orchestra.run.gibba012:> def find_socket(client_name): 2021-02-10T11:49:17.484 DEBUG:teuthology.orchestra.run.gibba012:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2021-02-10T11:49:17.484 DEBUG:teuthology.orchestra.run.gibba012:> files = glob.glob(asok_path) 2021-02-10T11:49:17.484 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:49:17.484 DEBUG:teuthology.orchestra.run.gibba012:> # Given a non-glob path, it better be there 2021-02-10T11:49:17.485 DEBUG:teuthology.orchestra.run.gibba012:> if "*" not in asok_path: 2021-02-10T11:49:17.485 DEBUG:teuthology.orchestra.run.gibba012:> assert(len(files) == 1) 2021-02-10T11:49:17.485 DEBUG:teuthology.orchestra.run.gibba012:> return files[0] 2021-02-10T11:49:17.485 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:49:17.485 DEBUG:teuthology.orchestra.run.gibba012:> for f in files: 2021-02-10T11:49:17.485 DEBUG:teuthology.orchestra.run.gibba012:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:49:17.485 DEBUG:teuthology.orchestra.run.gibba012:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:49:17.485 DEBUG:teuthology.orchestra.run.gibba012:> return f 2021-02-10T11:49:17.485 DEBUG:teuthology.orchestra.run.gibba012:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:49:17.485 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:49:17.485 DEBUG:teuthology.orchestra.run.gibba012:> print(find_socket("client.0")) 2021-02-10T11:49:17.485 DEBUG:teuthology.orchestra.run.gibba012:> ' 2021-02-10T11:49:17.847 INFO:teuthology.orchestra.run.gibba012.stdout:/var/run/ceph/ceph-client.0.196573.asok 2021-02-10T11:49:17.850 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.196573.asok 2021-02-10T11:49:17.850 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:49:17.850 DEBUG:teuthology.orchestra.run.gibba012:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.196573.asok status 2021-02-10T11:49:18.241 INFO:teuthology.orchestra.run.gibba012.stdout:{ 2021-02-10T11:49:18.242 INFO:teuthology.orchestra.run.gibba012.stdout: "metadata": { 2021-02-10T11:49:18.242 INFO:teuthology.orchestra.run.gibba012.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:49:18.242 INFO:teuthology.orchestra.run.gibba012.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:49:18.242 INFO:teuthology.orchestra.run.gibba012.stdout: "entity_id": "0", 2021-02-10T11:49:18.242 INFO:teuthology.orchestra.run.gibba012.stdout: "hostname": "gibba012", 2021-02-10T11:49:18.242 INFO:teuthology.orchestra.run.gibba012.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2021-02-10T11:49:18.242 INFO:teuthology.orchestra.run.gibba012.stdout: "pid": "196573", 2021-02-10T11:49:18.242 INFO:teuthology.orchestra.run.gibba012.stdout: "root": "/" 2021-02-10T11:49:18.242 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:49:18.242 INFO:teuthology.orchestra.run.gibba012.stdout: "dentry_count": 0, 2021-02-10T11:49:18.242 INFO:teuthology.orchestra.run.gibba012.stdout: "dentry_pinned_count": 0, 2021-02-10T11:49:18.242 INFO:teuthology.orchestra.run.gibba012.stdout: "id": 13962, 2021-02-10T11:49:18.243 INFO:teuthology.orchestra.run.gibba012.stdout: "inst": { 2021-02-10T11:49:18.243 INFO:teuthology.orchestra.run.gibba012.stdout: "name": { 2021-02-10T11:49:18.243 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "client", 2021-02-10T11:49:18.243 INFO:teuthology.orchestra.run.gibba012.stdout: "num": 13962 2021-02-10T11:49:18.243 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:49:18.243 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": { 2021-02-10T11:49:18.243 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:49:18.243 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:0", 2021-02-10T11:49:18.243 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 855028981 2021-02-10T11:49:18.243 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:49:18.243 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:49:18.243 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": { 2021-02-10T11:49:18.243 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:49:18.244 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:0", 2021-02-10T11:49:18.244 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 855028981 2021-02-10T11:49:18.244 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:49:18.244 INFO:teuthology.orchestra.run.gibba012.stdout: "inst_str": "client.13962 172.21.2.112:0/855028981", 2021-02-10T11:49:18.244 INFO:teuthology.orchestra.run.gibba012.stdout: "addr_str": "172.21.2.112:0/855028981", 2021-02-10T11:49:18.244 INFO:teuthology.orchestra.run.gibba012.stdout: "inode_count": 1, 2021-02-10T11:49:18.244 INFO:teuthology.orchestra.run.gibba012.stdout: "mds_epoch": 174, 2021-02-10T11:49:18.244 INFO:teuthology.orchestra.run.gibba012.stdout: "osd_epoch": 198, 2021-02-10T11:49:18.244 INFO:teuthology.orchestra.run.gibba012.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:49:18.244 INFO:teuthology.orchestra.run.gibba012.stdout: "blacklisted": false 2021-02-10T11:49:18.244 INFO:teuthology.orchestra.run.gibba012.stdout:} 2021-02-10T11:49:18.246 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:49:18.246 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:49:18.246 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:49:18.583 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:49:18.583 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:49:18.583 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:49:18.583 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:49:18.930 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:49:19.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.339 7f404c9a2700 1 Processor -- start 2021-02-10T11:49:19.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.339 7f404c9a2700 1 -- start start 2021-02-10T11:49:19.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.340 7f404c9a2700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f40480d1870 0x7f40481b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:19.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.340 7f404c9a2700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f40481b0a10 0x7f40480cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:19.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.340 7f404c9a2700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40480f0b10 con 0x7f40480d1c70 2021-02-10T11:49:19.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.340 7f404c9a2700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40480fe230 con 0x7f40481b0a10 2021-02-10T11:49:19.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.340 7f404c9a2700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40480e5ed0 con 0x7f40480d1870 2021-02-10T11:49:19.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.340 7f4045e1b700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f40481b0a10 0x7f40480cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:19.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.340 7f4045e1b700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f40481b0a10 0x7f40480cf330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:33750/0 (socket says 172.21.2.112:33750) 2021-02-10T11:49:19.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.340 7f4045e1b700 1 -- 172.21.2.112:0/1496459680 learned_addr learned my addr 172.21.2.112:0/1496459680 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:49:19.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.341 7f4045e1b700 1 -- 172.21.2.112:0/1496459680 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f40480d1870 msgr2=0x7f40481b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:49:19.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.341 7f4045e1b700 1 --2- 172.21.2.112:0/1496459680 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f40480d1870 0x7f40481b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:19.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.341 7f4045e1b700 1 -- 172.21.2.112:0/1496459680 >> v1:172.21.2.120:6789/0 conn(0x7f40480d1c70 legacy=0x7f40480cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:49:19.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.341 7f4045e1b700 1 -- 172.21.2.112:0/1496459680 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40480f0650 con 0x7f40481b0a10 2021-02-10T11:49:19.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.341 7f4045e1b700 1 --2- 172.21.2.112:0/1496459680 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f40481b0a10 0x7f40480cf330 secure :-1 s=READY pgs=3287 cs=0 l=1 rev1=1 rx=0x7f403c006700 tx=0x7f403c009b80).ready entity=mon.0 client_cookie=1537786684a5988b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:19.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.341 7f404561a700 1 -- 172.21.2.112:0/1496459680 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f403c00c0c0 con 0x7f40481b0a10 2021-02-10T11:49:19.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.341 7f404561a700 1 -- 172.21.2.112:0/1496459680 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f403c00c340 con 0x7f40481b0a10 2021-02-10T11:49:19.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.341 7f404561a700 1 -- 172.21.2.112:0/1496459680 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f403c00c860 con 0x7f40481b0a10 2021-02-10T11:49:19.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.341 7f404c9a2700 1 -- 172.21.2.112:0/1496459680 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f40481b0a10 msgr2=0x7f40480cf330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:19.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.341 7f404c9a2700 1 --2- 172.21.2.112:0/1496459680 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f40481b0a10 0x7f40480cf330 secure :-1 s=READY pgs=3287 cs=0 l=1 rev1=1 rx=0x7f403c006700 tx=0x7f403c009b80).stop 2021-02-10T11:49:19.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.342 7f404c9a2700 1 -- 172.21.2.112:0/1496459680 shutdown_connections 2021-02-10T11:49:19.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.342 7f404c9a2700 1 --2- 172.21.2.112:0/1496459680 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f40481b0a10 0x7f40480cf330 unknown :-1 s=CLOSED pgs=3287 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:19.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.342 7f404c9a2700 1 --2- 172.21.2.112:0/1496459680 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f40480d1870 0x7f40481b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:19.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.342 7f404c9a2700 1 -- 172.21.2.112:0/1496459680 >> 172.21.2.112:0/1496459680 conn(0x7f40480ce450 msgr2=0x7f40480d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:19.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.342 7f404c9a2700 1 -- 172.21.2.112:0/1496459680 shutdown_connections 2021-02-10T11:49:19.362 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.342 7f404c9a2700 1 -- 172.21.2.112:0/1496459680 wait complete. 2021-02-10T11:49:19.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.357 7f404c9a2700 1 Processor -- start 2021-02-10T11:49:19.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.357 7f404c9a2700 1 -- start start 2021-02-10T11:49:19.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.357 7f404c9a2700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f40480b4f30 0x7f404818b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:19.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.357 7f404c9a2700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4048093f40 0x7f4048094340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:19.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.357 7f404c9a2700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4048094960 0x7f4048187e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:19.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.357 7f404c9a2700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4048094e90 con 0x7f40480b4f30 2021-02-10T11:49:19.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.357 7f404c9a2700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f404818bff0 con 0x7f4048094960 2021-02-10T11:49:19.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.357 7f404c9a2700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f404808f780 con 0x7f4048093f40 2021-02-10T11:49:19.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.357 7f404661c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f40480b4f30 0x7f404818b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:19.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.357 7f404661c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f40480b4f30 0x7f404818b9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:33754/0 (socket says 172.21.2.112:33754) 2021-02-10T11:49:19.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.357 7f404661c700 1 -- 172.21.2.112:0/2843690306 learned_addr learned my addr 172.21.2.112:0/2843690306 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:49:19.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.358 7f4046e1d700 1 --2- 172.21.2.112:0/2843690306 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4048094960 0x7f4048187e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:19.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.358 7f4045e1b700 1 --2- 172.21.2.112:0/2843690306 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4048093f40 0x7f4048094340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:19.365 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.358 7f404661c700 1 -- 172.21.2.112:0/2843690306 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4048093f40 msgr2=0x7f4048094340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:19.365 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.358 7f404661c700 1 --2- 172.21.2.112:0/2843690306 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4048093f40 0x7f4048094340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:19.365 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.358 7f404661c700 1 -- 172.21.2.112:0/2843690306 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4048094960 msgr2=0x7f4048187e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:19.365 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.358 7f404661c700 1 --2- 172.21.2.112:0/2843690306 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4048094960 0x7f4048187e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:19.365 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.358 7f404661c700 1 -- 172.21.2.112:0/2843690306 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4048188470 con 0x7f40480b4f30 2021-02-10T11:49:19.365 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.358 7f404661c700 1 --2- 172.21.2.112:0/2843690306 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f40480b4f30 0x7f404818b9d0 secure :-1 s=READY pgs=3288 cs=0 l=1 rev1=1 rx=0x7f4038006f00 tx=0x7f403800b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:19.365 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.358 7f4036ffd700 1 -- 172.21.2.112:0/2843690306 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4038006350 con 0x7f40480b4f30 2021-02-10T11:49:19.365 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.358 7f4036ffd700 1 -- 172.21.2.112:0/2843690306 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f403800bdc0 con 0x7f40480b4f30 2021-02-10T11:49:19.365 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.358 7f404c9a2700 1 -- 172.21.2.112:0/2843690306 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4048188470 con 0x7f40480b4f30 2021-02-10T11:49:19.365 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.358 7f404c9a2700 1 -- 172.21.2.112:0/2843690306 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4048188a00 con 0x7f40480b4f30 2021-02-10T11:49:19.365 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.358 7f4036ffd700 1 -- 172.21.2.112:0/2843690306 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4038002020 con 0x7f40480b4f30 2021-02-10T11:49:19.365 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.359 7f4036ffd700 1 -- 172.21.2.112:0/2843690306 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4038005310 con 0x7f40480b4f30 2021-02-10T11:49:19.365 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.359 7f4036ffd700 1 --2- 172.21.2.112:0/2843690306 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f402c018730 0x7f402c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:19.366 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.359 7f4036ffd700 1 -- 172.21.2.112:0/2843690306 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f4038032780 con 0x7f40480b4f30 2021-02-10T11:49:19.366 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.360 7f4045e1b700 1 --2- 172.21.2.112:0/2843690306 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f402c018730 0x7f402c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:19.366 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.361 7f4045e1b700 1 --2- 172.21.2.112:0/2843690306 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f402c018730 0x7f402c01abc0 crc :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:19.394 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.390 7f404c9a2700 1 -- 172.21.2.112:0/2843690306 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40480ce450 con 0x7f40480b4f30 2021-02-10T11:49:19.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.393 7f4036ffd700 1 -- 172.21.2.112:0/2843690306 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4038000940 con 0x7f40480b4f30 2021-02-10T11:49:19.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.517 7f404c9a2700 1 -- 172.21.2.112:0/2843690306 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f40480ce450 con 0x7f40480b4f30 2021-02-10T11:49:19.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.518 7f4036ffd700 1 -- 172.21.2.112:0/2843690306 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v198) v1 ==== 74+0+14007 (secure 0 0 0) 0x7f403800bd30 con 0x7f40480b4f30 2021-02-10T11:49:19.522 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:49:19.522 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":198,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:49:05.481103","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":43,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":42,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:49:02.450970","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"197","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":43,"pool_name":"cephfs_data","create_time":"2021-02-10 11:49:03.473275","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"198","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":196,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":196,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":196,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":196,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":196,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:49:19.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.519 7f404c9a2700 1 -- 172.21.2.112:0/2843690306 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f402c018730 msgr2=0x7f402c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:19.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.519 7f404c9a2700 1 --2- 172.21.2.112:0/2843690306 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f402c018730 0x7f402c01abc0 crc :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:19.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.519 7f404c9a2700 1 -- 172.21.2.112:0/2843690306 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f40480b4f30 msgr2=0x7f404818b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:19.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.519 7f404c9a2700 1 --2- 172.21.2.112:0/2843690306 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f40480b4f30 0x7f404818b9d0 secure :-1 s=READY pgs=3288 cs=0 l=1 rev1=1 rx=0x7f4038006f00 tx=0x7f403800b940).stop 2021-02-10T11:49:19.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.519 7f404c9a2700 1 -- 172.21.2.112:0/2843690306 shutdown_connections 2021-02-10T11:49:19.524 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.519 7f404c9a2700 1 --2- 172.21.2.112:0/2843690306 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f402c018730 0x7f402c01abc0 unknown :-1 s=CLOSED pgs=1481 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:19.524 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.519 7f404c9a2700 1 --2- 172.21.2.112:0/2843690306 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4048094960 0x7f4048187e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:19.524 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.519 7f404c9a2700 1 --2- 172.21.2.112:0/2843690306 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4048093f40 0x7f4048094340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:19.524 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.519 7f404c9a2700 1 --2- 172.21.2.112:0/2843690306 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f40480b4f30 0x7f404818b9d0 unknown :-1 s=CLOSED pgs=3288 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:19.524 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.519 7f404c9a2700 1 -- 172.21.2.112:0/2843690306 >> 172.21.2.112:0/2843690306 conn(0x7f404819f8f0 msgr2=0x7f40481a5f80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:19.524 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.520 7f404c9a2700 1 -- 172.21.2.112:0/2843690306 shutdown_connections 2021-02-10T11:49:19.524 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.520 7f404c9a2700 1 -- 172.21.2.112:0/2843690306 wait complete. 2021-02-10T11:49:19.532 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:13:57.508758', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 42, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:49:02.450970', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '197', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 43, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:49:03.473275', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '198', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:49:19.532 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:49:19.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.945 7f9c9eda0700 1 Processor -- start 2021-02-10T11:49:19.965 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.945 7f9c9eda0700 1 -- start start 2021-02-10T11:49:19.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.946 7f9c9eda0700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9c980d1870 0x7f9c981b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:19.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.946 7f9c9eda0700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c98109a60 con 0x7f9c981b0ae0 2021-02-10T11:49:19.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.946 7f9c9eda0700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c98088070 con 0x7f9c980cff70 2021-02-10T11:49:19.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.946 7f9c9eda0700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c980e5ef0 con 0x7f9c980d1870 2021-02-10T11:49:19.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.946 7f9c97fff700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f9c981b0ae0 0x7f9c98108940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:33224/0 (socket says 172.21.2.112:33224) 2021-02-10T11:49:19.966 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.946 7f9c97fff700 1 -- 172.21.2.112:0/735881667 learned_addr learned my addr 172.21.2.112:0/735881667 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:49:19.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.946 7f9c9cbbc700 1 --2- 172.21.2.112:0/735881667 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9c980d1870 0x7f9c981b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:19.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.946 7f9c9cbbc700 1 -- 172.21.2.112:0/735881667 >> v1:172.21.2.120:6789/0 conn(0x7f9c980cff70 legacy=0x7f9c98109130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:19.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.946 7f9c977fe700 1 -- 172.21.2.112:0/735881667 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3180543915 0 0) 0x7f9c90001400 con 0x7f9c980cff70 2021-02-10T11:49:19.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.946 7f9c9cbbc700 1 -- 172.21.2.112:0/735881667 >> v1:172.21.2.112:6789/0 conn(0x7f9c981b0ae0 legacy=0x7f9c98108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:19.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.946 7f9c9cbbc700 1 -- 172.21.2.112:0/735881667 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c980d1c70 con 0x7f9c980d1870 2021-02-10T11:49:19.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.947 7f9c9cbbc700 1 --2- 172.21.2.112:0/735881667 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9c980d1870 0x7f9c981b04c0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7f9c88006650 tx=0x7f9c88009a80).ready entity=mon.2 client_cookie=57dcb2dbd553a851 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:19.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.947 7f9c977fe700 1 -- 172.21.2.112:0/735881667 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9c880009b0 con 0x7f9c980d1870 2021-02-10T11:49:19.967 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.947 7f9c977fe700 1 -- 172.21.2.112:0/735881667 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c8800c0c0 con 0x7f9c980d1870 2021-02-10T11:49:19.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.947 7f9c977fe700 1 -- 172.21.2.112:0/735881667 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9c8800c5e0 con 0x7f9c980d1870 2021-02-10T11:49:19.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.947 7f9c9eda0700 1 -- 172.21.2.112:0/735881667 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9c980d1870 msgr2=0x7f9c981b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:19.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.947 7f9c9eda0700 1 --2- 172.21.2.112:0/735881667 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9c980d1870 0x7f9c981b04c0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7f9c88006650 tx=0x7f9c88009a80).stop 2021-02-10T11:49:19.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.947 7f9c9eda0700 1 -- 172.21.2.112:0/735881667 shutdown_connections 2021-02-10T11:49:19.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.947 7f9c9eda0700 1 --2- 172.21.2.112:0/735881667 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9c980d1870 0x7f9c981b04c0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:19.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.947 7f9c9eda0700 1 -- 172.21.2.112:0/735881667 >> 172.21.2.112:0/735881667 conn(0x7f9c980ce420 msgr2=0x7f9c980d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:19.968 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.947 7f9c9eda0700 1 -- 172.21.2.112:0/735881667 shutdown_connections 2021-02-10T11:49:19.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.947 7f9c9eda0700 1 -- 172.21.2.112:0/735881667 wait complete. 2021-02-10T11:49:19.971 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.962 7f9c9eda0700 1 Processor -- start 2021-02-10T11:49:19.971 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.962 7f9c9eda0700 1 -- start start 2021-02-10T11:49:19.971 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.962 7f9c9eda0700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9c980c2370 0x7f9c9818f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:19.971 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.962 7f9c9eda0700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9c98084cb0 0x7f9c981a7470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:19.972 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.962 7f9c9eda0700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9c981a7a90 0x7f9c98085700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:19.972 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.962 7f9c9eda0700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c980be840 con 0x7f9c980c2370 2021-02-10T11:49:19.972 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.962 7f9c9eda0700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c980c29a0 con 0x7f9c98084cb0 2021-02-10T11:49:19.972 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.962 7f9c9eda0700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c981a7e90 con 0x7f9c981a7a90 2021-02-10T11:49:19.972 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.963 7f9c9cbbc700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9c980c2370 0x7f9c9818f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:19.972 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.963 7f9c9cbbc700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9c980c2370 0x7f9c9818f870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:33768/0 (socket says 172.21.2.112:33768) 2021-02-10T11:49:19.972 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.963 7f9c9d3bd700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9c981a7a90 0x7f9c98085700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:19.973 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.963 7f9c9cbbc700 1 -- 172.21.2.112:0/575705925 learned_addr learned my addr 172.21.2.112:0/575705925 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:49:19.973 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.963 7f9c97fff700 1 --2- 172.21.2.112:0/575705925 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9c98084cb0 0x7f9c981a7470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:19.973 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.963 7f9c9cbbc700 1 -- 172.21.2.112:0/575705925 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9c981a7a90 msgr2=0x7f9c98085700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:19.973 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.963 7f9c9cbbc700 1 --2- 172.21.2.112:0/575705925 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9c981a7a90 0x7f9c98085700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:19.973 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.963 7f9c9cbbc700 1 -- 172.21.2.112:0/575705925 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9c98084cb0 msgr2=0x7f9c981a7470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:19.973 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.963 7f9c9cbbc700 1 --2- 172.21.2.112:0/575705925 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9c98084cb0 0x7f9c981a7470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:19.974 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.963 7f9c9cbbc700 1 -- 172.21.2.112:0/575705925 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c981a80b0 con 0x7f9c980c2370 2021-02-10T11:49:19.974 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.963 7f9c9cbbc700 1 --2- 172.21.2.112:0/575705925 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9c980c2370 0x7f9c9818f870 secure :-1 s=READY pgs=3290 cs=0 l=1 rev1=1 rx=0x7f9c88009650 tx=0x7f9c8800b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:19.974 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.963 7f9c957fa700 1 -- 172.21.2.112:0/575705925 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9c8800bd50 con 0x7f9c980c2370 2021-02-10T11:49:19.974 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.963 7f9c957fa700 1 -- 172.21.2.112:0/575705925 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c88006030 con 0x7f9c980c2370 2021-02-10T11:49:19.974 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.963 7f9c957fa700 1 -- 172.21.2.112:0/575705925 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9c88005560 con 0x7f9c980c2370 2021-02-10T11:49:19.974 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.963 7f9c9eda0700 1 -- 172.21.2.112:0/575705925 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c981a80b0 con 0x7f9c980c2370 2021-02-10T11:49:19.975 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.963 7f9c9eda0700 1 -- 172.21.2.112:0/575705925 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c981a80b0 con 0x7f9c980c2370 2021-02-10T11:49:19.975 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.964 7f9c957fa700 1 -- 172.21.2.112:0/575705925 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9c880052e0 con 0x7f9c980c2370 2021-02-10T11:49:19.975 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.964 7f9c957fa700 1 --2- 172.21.2.112:0/575705925 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9c78018730 0x7f9c7801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:19.975 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.964 7f9c957fa700 1 -- 172.21.2.112:0/575705925 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f9c880327b0 con 0x7f9c980c2370 2021-02-10T11:49:19.975 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.964 7f9c97fff700 1 --2- 172.21.2.112:0/575705925 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9c78018730 0x7f9c7801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:19.975 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.965 7f9c97fff700 1 --2- 172.21.2.112:0/575705925 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9c78018730 0x7f9c7801abc0 crc :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:19.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.995 7f9c9eda0700 1 -- 172.21.2.112:0/575705925 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c980be840 con 0x7f9c980c2370 2021-02-10T11:49:20.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:19.998 7f9c957fa700 1 -- 172.21.2.112:0/575705925 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9c880009c0 con 0x7f9c980c2370 2021-02-10T11:49:20.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:20.126 7f9c9eda0700 1 -- 172.21.2.112:0/575705925 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f9c980be840 con 0x7f9c980c2370 2021-02-10T11:49:20.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:20.126 7f9c957fa700 1 -- 172.21.2.112:0/575705925 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v198) v1 ==== 92+0+10 (secure 0 0 0) 0x7f9c8800bd50 con 0x7f9c980c2370 2021-02-10T11:49:20.131 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:49:20.132 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:20.128 7f9c9eda0700 1 -- 172.21.2.112:0/575705925 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9c78018730 msgr2=0x7f9c7801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:20.133 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:20.128 7f9c9eda0700 1 --2- 172.21.2.112:0/575705925 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9c78018730 0x7f9c7801abc0 crc :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:20.133 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:20.128 7f9c9eda0700 1 -- 172.21.2.112:0/575705925 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9c980c2370 msgr2=0x7f9c9818f870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:20.133 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:20.128 7f9c9eda0700 1 --2- 172.21.2.112:0/575705925 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9c980c2370 0x7f9c9818f870 secure :-1 s=READY pgs=3290 cs=0 l=1 rev1=1 rx=0x7f9c88009650 tx=0x7f9c8800b960).stop 2021-02-10T11:49:20.133 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:20.128 7f9c9eda0700 1 -- 172.21.2.112:0/575705925 shutdown_connections 2021-02-10T11:49:20.133 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:20.128 7f9c9eda0700 1 --2- 172.21.2.112:0/575705925 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9c78018730 0x7f9c7801abc0 unknown :-1 s=CLOSED pgs=1482 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:20.133 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:20.128 7f9c9eda0700 1 --2- 172.21.2.112:0/575705925 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f9c98084cb0 0x7f9c981a7470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:20.134 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:20.128 7f9c9eda0700 1 --2- 172.21.2.112:0/575705925 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f9c981a7a90 0x7f9c98085700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:20.134 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:20.128 7f9c9eda0700 1 --2- 172.21.2.112:0/575705925 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f9c980c2370 0x7f9c9818f870 unknown :-1 s=CLOSED pgs=3290 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:20.134 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:20.128 7f9c9eda0700 1 -- 172.21.2.112:0/575705925 >> 172.21.2.112:0/575705925 conn(0x7f9c980b5dc0 msgr2=0x7f9c980c2d90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:20.134 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:20.129 7f9c9eda0700 1 -- 172.21.2.112:0/575705925 shutdown_connections 2021-02-10T11:49:20.134 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:20.129 7f9c9eda0700 1 -- 172.21.2.112:0/575705925 wait complete. 2021-02-10T11:49:20.141 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:49:20.574 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:20.553 7fa754e5a700 1 Processor -- start 2021-02-10T11:49:20.574 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:20.554 7fa754e5a700 1 -- start start 2021-02-10T11:49:20.575 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:20.554 7fa754e5a700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fa7500d1c60 0x7fa7500cef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:20.575 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:20.554 7fa754e5a700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fa7500cf570 0x7fa7500cf970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:20.575 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:20.554 7fa754e5a700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7500fe1b0 con 0x7fa7500d1860 2021-02-10T11:49:20.575 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:20.554 7fa754e5a700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7501b0ca0 con 0x7fa7500d1c60 2021-02-10T11:49:20.575 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:20.554 7fa754e5a700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa750109a50 con 0x7fa7500cf570 2021-02-10T11:49:20.575 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:20.554 7fa746312700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fa7500d1c60 0x7fa7500cef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:20.575 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:20.554 7fa746312700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fa7500d1c60 0x7fa7500cef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:33776/0 (socket says 172.21.2.112:33776) 2021-02-10T11:49:20.575 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:20.554 7fa746312700 1 -- 172.21.2.112:0/928558384 learned_addr learned my addr 172.21.2.112:0/928558384 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:49:20.575 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:20.554 7fa74eb13700 1 --1- 172.21.2.112:0/928558384 >> v1:172.21.2.112:6789/0 conn(0x7fa7500d1860 0x7fa7501b04b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:33236/0 (socket says 172.21.2.112:33236) 2021-02-10T11:49:20.575 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:20.554 7fa74f314700 1 --2- 172.21.2.112:0/928558384 >> v2:172.21.2.120:3300/0 conn(0x7fa7500cf570 0x7fa7500cf970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:20.575 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:20.554 7fa746312700 1 -- 172.21.2.112:0/928558384 >> v2:172.21.2.120:3300/0 conn(0x7fa7500cf570 msgr2=0x7fa7500cf970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:49:20.576 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:20.554 7fa746312700 1 --2- 172.21.2.112:0/928558384 >> v2:172.21.2.120:3300/0 conn(0x7fa7500cf570 0x7fa7500cf970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:20.576 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:20.554 7fa746312700 1 -- 172.21.2.112:0/928558384 >> v1:172.21.2.112:6789/0 conn(0x7fa7500d1860 legacy=0x7fa7501b04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:20.576 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:20.555 7fa746312700 1 -- 172.21.2.112:0/928558384 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa750088070 con 0x7fa7500d1c60 2021-02-10T11:49:20.576 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:20.555 7fa746312700 1 --2- 172.21.2.112:0/928558384 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa7500d1c60 0x7fa7500cef50 secure :-1 s=READY pgs=3292 cs=0 l=1 rev1=1 rx=0x7fa738006700 tx=0x7fa738009b00).ready entity=mon.0 client_cookie=bac07aedfa8e1f8c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:20.576 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:20.555 7fa74e312700 1 -- 172.21.2.112:0/928558384 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa73800c0c0 con 0x7fa7500d1c60 2021-02-10T11:49:20.576 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:20.555 7fa74e312700 1 -- 172.21.2.112:0/928558384 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa73800c340 con 0x7fa7500d1c60 2021-02-10T11:49:20.576 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:20.555 7fa74e312700 1 -- 172.21.2.112:0/928558384 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa73800c820 con 0x7fa7500d1c60 2021-02-10T11:49:20.576 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:20.555 7fa754e5a700 1 -- 172.21.2.112:0/928558384 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa7500d1c60 msgr2=0x7fa7500cef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:20.576 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:20.555 7fa754e5a700 1 --2- 172.21.2.112:0/928558384 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa7500d1c60 0x7fa7500cef50 secure :-1 s=READY pgs=3292 cs=0 l=1 rev1=1 rx=0x7fa738006700 tx=0x7fa738009b00).stop 2021-02-10T11:49:20.576 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:20.555 7fa754e5a700 1 -- 172.21.2.112:0/928558384 shutdown_connections 2021-02-10T11:49:20.576 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:20.555 7fa754e5a700 1 --2- 172.21.2.112:0/928558384 >> v2:172.21.2.120:3300/0 conn(0x7fa7500cf570 0x7fa7500cf970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:20.576 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:20.555 7fa754e5a700 1 --2- 172.21.2.112:0/928558384 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa7500d1c60 0x7fa7500cef50 unknown :-1 s=CLOSED pgs=3292 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:20.576 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:20.555 7fa754e5a700 1 -- 172.21.2.112:0/928558384 >> 172.21.2.112:0/928558384 conn(0x7fa7500ce410 msgr2=0x7fa7500d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:20.576 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:20.555 7fa754e5a700 1 -- 172.21.2.112:0/928558384 shutdown_connections 2021-02-10T11:49:20.577 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:20.555 7fa754e5a700 1 -- 172.21.2.112:0/928558384 wait complete. 2021-02-10T11:49:20.577 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:20.570 7fa754e5a700 1 Processor -- start 2021-02-10T11:49:20.577 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:20.571 7fa754e5a700 1 -- start start 2021-02-10T11:49:20.579 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:20.571 7fa754e5a700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa750085270 0x7fa750085670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:20.579 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:20.571 7fa754e5a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa750085c90 0x7fa750086090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:20.579 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:20.571 7fa754e5a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa7500c2740 0x7fa7500c2b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:20.579 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:20.571 7fa754e5a700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa750084cf0 con 0x7fa750085c90 2021-02-10T11:49:20.579 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:20.571 7fa754e5a700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7500be1b0 con 0x7fa750085270 2021-02-10T11:49:20.579 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:20.571 7fa754e5a700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7500be390 con 0x7fa7500c2740 2021-02-10T11:49:20.579 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:20.571 7fa746312700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa750085c90 0x7fa750086090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:20.579 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:20.571 7fa746312700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa750085c90 0x7fa750086090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:33782/0 (socket says 172.21.2.112:33782) 2021-02-10T11:49:20.579 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:20.571 7fa746312700 1 -- 172.21.2.112:0/731744057 learned_addr learned my addr 172.21.2.112:0/731744057 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:49:20.579 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:20.571 7fa74f314700 1 --2- 172.21.2.112:0/731744057 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa7500c2740 0x7fa7500c2b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:20.580 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:20.571 7fa74eb13700 1 --2- 172.21.2.112:0/731744057 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa750085270 0x7fa750085670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:20.580 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:20.571 7fa746312700 1 -- 172.21.2.112:0/731744057 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa7500c2740 msgr2=0x7fa7500c2b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:20.580 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:20.572 7fa746312700 1 --2- 172.21.2.112:0/731744057 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa7500c2740 0x7fa7500c2b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:20.580 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:20.572 7fa746312700 1 -- 172.21.2.112:0/731744057 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa750085270 msgr2=0x7fa750085670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:20.580 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:20.572 7fa746312700 1 --2- 172.21.2.112:0/731744057 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa750085270 0x7fa750085670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:20.580 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:20.572 7fa746312700 1 -- 172.21.2.112:0/731744057 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7500c3160 con 0x7fa750085c90 2021-02-10T11:49:20.580 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:20.572 7fa74f314700 1 --2- 172.21.2.112:0/731744057 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa7500c2740 0x7fa7500c2b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:49:20.580 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:20.572 7fa746312700 1 --2- 172.21.2.112:0/731744057 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa750085c90 0x7fa750086090 secure :-1 s=READY pgs=3293 cs=0 l=1 rev1=1 rx=0x7fa738006f20 tx=0x7fa73800b9b0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:20.580 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:20.572 7fa747fff700 1 -- 172.21.2.112:0/731744057 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa738009100 con 0x7fa750085c90 2021-02-10T11:49:20.580 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:20.572 7fa747fff700 1 -- 172.21.2.112:0/731744057 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa73800bda0 con 0x7fa750085c90 2021-02-10T11:49:20.580 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:20.572 7fa747fff700 1 -- 172.21.2.112:0/731744057 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa7380054d0 con 0x7fa750085c90 2021-02-10T11:49:20.580 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:20.572 7fa754e5a700 1 -- 172.21.2.112:0/731744057 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7500c3160 con 0x7fa750085c90 2021-02-10T11:49:20.580 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:20.572 7fa754e5a700 1 -- 172.21.2.112:0/731744057 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7500c36c0 con 0x7fa750085c90 2021-02-10T11:49:20.581 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:20.573 7fa747fff700 1 -- 172.21.2.112:0/731744057 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa738009410 con 0x7fa750085c90 2021-02-10T11:49:20.581 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:20.573 7fa747fff700 1 --2- 172.21.2.112:0/731744057 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa7340186c0 0x7fa73401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:20.581 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:20.573 7fa747fff700 1 -- 172.21.2.112:0/731744057 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fa7380325a0 con 0x7fa750085c90 2021-02-10T11:49:20.581 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:20.573 7fa74eb13700 1 --2- 172.21.2.112:0/731744057 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa7340186c0 0x7fa73401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:20.581 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:20.573 7fa74eb13700 1 --2- 172.21.2.112:0/731744057 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa7340186c0 0x7fa73401abc0 crc :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:20.607 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:20.604 7fa754e5a700 1 -- 172.21.2.112:0/731744057 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa750082a20 con 0x7fa750085c90 2021-02-10T11:49:20.610 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:20.607 7fa747fff700 1 -- 172.21.2.112:0/731744057 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa738009100 con 0x7fa750085c90 2021-02-10T11:49:20.742 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:20.738 7fa754e5a700 1 -- 172.21.2.112:0/731744057 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7fa750082a20 con 0x7fa750085c90 2021-02-10T11:49:20.742 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:20.738 7fa747fff700 1 -- 172.21.2.112:0/731744057 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v198) v1 ==== 104+0+10 (secure 0 0 0) 0x7fa738009410 con 0x7fa750085c90 2021-02-10T11:49:20.743 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:49:20.745 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:20.740 7fa754e5a700 1 -- 172.21.2.112:0/731744057 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa7340186c0 msgr2=0x7fa73401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:20.745 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:20.740 7fa754e5a700 1 --2- 172.21.2.112:0/731744057 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa7340186c0 0x7fa73401abc0 crc :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:20.745 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:20.740 7fa754e5a700 1 -- 172.21.2.112:0/731744057 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa750085c90 msgr2=0x7fa750086090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:20.746 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:20.740 7fa754e5a700 1 --2- 172.21.2.112:0/731744057 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa750085c90 0x7fa750086090 secure :-1 s=READY pgs=3293 cs=0 l=1 rev1=1 rx=0x7fa738006f20 tx=0x7fa73800b9b0).stop 2021-02-10T11:49:20.746 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:20.741 7fa754e5a700 1 -- 172.21.2.112:0/731744057 shutdown_connections 2021-02-10T11:49:20.746 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:20.741 7fa754e5a700 1 --2- 172.21.2.112:0/731744057 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fa7340186c0 0x7fa73401abc0 unknown :-1 s=CLOSED pgs=1483 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:20.746 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:20.741 7fa754e5a700 1 --2- 172.21.2.112:0/731744057 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fa7500c2740 0x7fa7500c2b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:20.746 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:20.741 7fa754e5a700 1 --2- 172.21.2.112:0/731744057 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fa750085c90 0x7fa750086090 unknown :-1 s=CLOSED pgs=3293 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:20.747 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:20.741 7fa754e5a700 1 --2- 172.21.2.112:0/731744057 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fa750085270 0x7fa750085670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:20.747 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:20.741 7fa754e5a700 1 -- 172.21.2.112:0/731744057 >> 172.21.2.112:0/731744057 conn(0x7fa7500b5f10 msgr2=0x7fa75009f200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:20.747 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:20.741 7fa754e5a700 1 -- 172.21.2.112:0/731744057 shutdown_connections 2021-02-10T11:49:20.747 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:20.741 7fa754e5a700 1 -- 172.21.2.112:0/731744057 wait complete. 2021-02-10T11:49:20.753 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:49:21.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.167 7fb4310de700 1 Processor -- start 2021-02-10T11:49:21.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.167 7fb4310de700 1 -- start start 2021-02-10T11:49:21.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.167 7fb4310de700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb42c0d1870 0x7fb42c1b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:21.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.167 7fb4310de700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fb42c1b0ae0 0x7fb42c108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:21.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.167 7fb4310de700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb42c109a60 con 0x7fb42c0d1c70 2021-02-10T11:49:21.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.167 7fb4310de700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb42c0e5ef0 con 0x7fb42c1b0ae0 2021-02-10T11:49:21.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.167 7fb4310de700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb42c109750 con 0x7fb42c0d1870 2021-02-10T11:49:21.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.168 7fb42a59c700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fb42c1b0ae0 0x7fb42c108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:21.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.168 7fb42ad9d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb42c0d1870 0x7fb42c1b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:21.188 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.168 7fb42b59e700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7fb42c0d1c70 0x7fb42c108f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:55044/0 (socket says 172.21.2.112:55044) 2021-02-10T11:49:21.189 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.168 7fb42ad9d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb42c0d1870 0x7fb42c1b04c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3301/0 says I am v2:172.21.2.112:33470/0 (socket says 172.21.2.112:33470) 2021-02-10T11:49:21.189 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.168 7fb42a59c700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fb42c1b0ae0 0x7fb42c108940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3300/0 says I am v2:172.21.2.112:36070/0 (socket says 172.21.2.112:36070) 2021-02-10T11:49:21.189 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.168 7fb42b59e700 1 -- 172.21.2.112:0/2888046337 learned_addr learned my addr 172.21.2.112:0/2888046337 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:49:21.189 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.169 7fb42ad9d700 1 -- 172.21.2.112:0/2888046337 >> v2:172.21.2.120:3300/0 conn(0x7fb42c1b0ae0 msgr2=0x7fb42c108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:49:21.189 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.169 7fb42ad9d700 1 --2- 172.21.2.112:0/2888046337 >> v2:172.21.2.120:3300/0 conn(0x7fb42c1b0ae0 0x7fb42c108940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:21.189 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.169 7fb42ad9d700 1 -- 172.21.2.112:0/2888046337 >> v1:172.21.2.120:6789/0 conn(0x7fb42c0d1c70 legacy=0x7fb42c108f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:21.189 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.169 7fb429d9b700 1 -- 172.21.2.112:0/2888046337 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4040202550 0 0) 0x7fb424001440 con 0x7fb42c0d1c70 2021-02-10T11:49:21.189 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.169 7fb42ad9d700 1 -- 172.21.2.112:0/2888046337 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb42c088070 con 0x7fb42c0d1870 2021-02-10T11:49:21.189 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.169 7fb42a59c700 1 --2- 172.21.2.112:0/2888046337 >> v2:172.21.2.120:3300/0 conn(0x7fb42c1b0ae0 0x7fb42c108940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:49:21.189 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.169 7fb42ad9d700 1 --2- 172.21.2.112:0/2888046337 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb42c0d1870 0x7fb42c1b04c0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 rx=0x7fb41c006650 tx=0x7fb41c009a80).ready entity=mon.2 client_cookie=d5a57ec171b7d91 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:21.189 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.169 7fb429d9b700 1 -- 172.21.2.112:0/2888046337 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb41c00c0c0 con 0x7fb42c0d1870 2021-02-10T11:49:21.189 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.169 7fb429d9b700 1 -- 172.21.2.112:0/2888046337 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb41c00c340 con 0x7fb42c0d1870 2021-02-10T11:49:21.189 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.169 7fb429d9b700 1 -- 172.21.2.112:0/2888046337 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb41c00c790 con 0x7fb42c0d1870 2021-02-10T11:49:21.190 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.169 7fb4310de700 1 -- 172.21.2.112:0/2888046337 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb42c0d1870 msgr2=0x7fb42c1b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:21.190 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.169 7fb4310de700 1 --2- 172.21.2.112:0/2888046337 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb42c0d1870 0x7fb42c1b04c0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 rx=0x7fb41c006650 tx=0x7fb41c009a80).stop 2021-02-10T11:49:21.190 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.170 7fb4310de700 1 -- 172.21.2.112:0/2888046337 shutdown_connections 2021-02-10T11:49:21.190 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.170 7fb4310de700 1 --2- 172.21.2.112:0/2888046337 >> v2:172.21.2.120:3300/0 conn(0x7fb42c1b0ae0 0x7fb42c108940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:21.190 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.170 7fb4310de700 1 --2- 172.21.2.112:0/2888046337 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb42c0d1870 0x7fb42c1b04c0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:21.190 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.170 7fb4310de700 1 -- 172.21.2.112:0/2888046337 >> 172.21.2.112:0/2888046337 conn(0x7fb42c0ce420 msgr2=0x7fb42c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:21.190 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.170 7fb4310de700 1 -- 172.21.2.112:0/2888046337 shutdown_connections 2021-02-10T11:49:21.190 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.170 7fb4310de700 1 -- 172.21.2.112:0/2888046337 wait complete. 2021-02-10T11:49:21.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.184 7fb4310de700 1 Processor -- start 2021-02-10T11:49:21.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.184 7fb4310de700 1 -- start start 2021-02-10T11:49:21.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.185 7fb4310de700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb42c0abd60 0x7fb42c0ac160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:21.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.185 7fb4310de700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb42c0ac780 0x7fb42c0acb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:21.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.185 7fb4310de700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb42c083910 0x7fb42c083d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:21.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.185 7fb4310de700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb42c087180 con 0x7fb42c0abd60 2021-02-10T11:49:21.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.185 7fb4310de700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb42c18fcd0 con 0x7fb42c083910 2021-02-10T11:49:21.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.185 7fb4310de700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb42c084330 con 0x7fb42c0ac780 2021-02-10T11:49:21.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.185 7fb42b59e700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb42c083910 0x7fb42c083d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:21.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.185 7fb42b59e700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb42c083910 0x7fb42c083d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3300/0 says I am v2:172.21.2.112:36074/0 (socket says 172.21.2.112:36074) 2021-02-10T11:49:21.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.185 7fb42ad9d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb42c0abd60 0x7fb42c0ac160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:21.192 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.185 7fb42b59e700 1 -- 172.21.2.112:0/3541180532 learned_addr learned my addr 172.21.2.112:0/3541180532 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:49:21.193 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.185 7fb42a59c700 1 --2- 172.21.2.112:0/3541180532 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb42c0ac780 0x7fb42c0acb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:21.193 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.186 7fb42ad9d700 1 -- 172.21.2.112:0/3541180532 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb42c0ac780 msgr2=0x7fb42c0acb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:21.193 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.186 7fb42ad9d700 1 --2- 172.21.2.112:0/3541180532 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb42c0ac780 0x7fb42c0acb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:21.193 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.186 7fb42ad9d700 1 -- 172.21.2.112:0/3541180532 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb42c083910 msgr2=0x7fb42c083d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:21.193 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.186 7fb42ad9d700 1 --2- 172.21.2.112:0/3541180532 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb42c083910 0x7fb42c083d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:21.193 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.186 7fb42ad9d700 1 -- 172.21.2.112:0/3541180532 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb42c084550 con 0x7fb42c0abd60 2021-02-10T11:49:21.193 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.186 7fb42b59e700 1 --2- 172.21.2.112:0/3541180532 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb42c083910 0x7fb42c083d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:49:21.193 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.186 7fb42ad9d700 1 --2- 172.21.2.112:0/3541180532 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb42c0abd60 0x7fb42c0ac160 secure :-1 s=READY pgs=3294 cs=0 l=1 rev1=1 rx=0x7fb41c0008e0 tx=0x7fb41c00b8e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:21.193 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.186 7fb4237fe700 1 -- 172.21.2.112:0/3541180532 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb41c009300 con 0x7fb42c0abd60 2021-02-10T11:49:21.193 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.186 7fb4237fe700 1 -- 172.21.2.112:0/3541180532 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb41c000d90 con 0x7fb42c0abd60 2021-02-10T11:49:21.193 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.186 7fb4237fe700 1 -- 172.21.2.112:0/3541180532 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb41c0022b0 con 0x7fb42c0abd60 2021-02-10T11:49:21.193 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.186 7fb4310de700 1 -- 172.21.2.112:0/3541180532 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb42c084550 con 0x7fb42c0abd60 2021-02-10T11:49:21.193 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.186 7fb4310de700 1 -- 172.21.2.112:0/3541180532 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb42c084550 con 0x7fb42c0abd60 2021-02-10T11:49:21.194 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.186 7fb4237fe700 1 -- 172.21.2.112:0/3541180532 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb41c009fd0 con 0x7fb42c0abd60 2021-02-10T11:49:21.194 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.187 7fb4237fe700 1 --2- 172.21.2.112:0/3541180532 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb410018730 0x7fb41001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:21.194 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.187 7fb4237fe700 1 -- 172.21.2.112:0/3541180532 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fb41c032560 con 0x7fb42c0abd60 2021-02-10T11:49:21.194 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.187 7fb42a59c700 1 --2- 172.21.2.112:0/3541180532 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb410018730 0x7fb41001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:21.194 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.187 7fb42a59c700 1 --2- 172.21.2.112:0/3541180532 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb410018730 0x7fb41001ac10 crc :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:21.221 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.217 7fb4310de700 1 -- 172.21.2.112:0/3541180532 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb42c082980 con 0x7fb42c0abd60 2021-02-10T11:49:21.223 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.220 7fb4237fe700 1 -- 172.21.2.112:0/3541180532 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb41c0067b0 con 0x7fb42c0abd60 2021-02-10T11:49:21.353 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.349 7fb4310de700 1 -- 172.21.2.112:0/3541180532 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7fb42c109a60 con 0x7fb42c0abd60 2021-02-10T11:49:21.353 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.349 7fb4237fe700 1 -- 172.21.2.112:0/3541180532 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v198) v1 ==== 100+0+10 (secure 0 0 0) 0x7fb41c009fd0 con 0x7fb42c0abd60 2021-02-10T11:49:21.354 INFO:teuthology.orchestra.run.gibba012.stdout:pg_num: 8 2021-02-10T11:49:21.355 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.351 7fb4310de700 1 -- 172.21.2.112:0/3541180532 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb410018730 msgr2=0x7fb41001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:21.355 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.351 7fb4310de700 1 --2- 172.21.2.112:0/3541180532 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb410018730 0x7fb41001ac10 crc :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:21.355 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.351 7fb4310de700 1 -- 172.21.2.112:0/3541180532 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb42c0abd60 msgr2=0x7fb42c0ac160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:21.355 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.351 7fb4310de700 1 --2- 172.21.2.112:0/3541180532 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb42c0abd60 0x7fb42c0ac160 secure :-1 s=READY pgs=3294 cs=0 l=1 rev1=1 rx=0x7fb41c0008e0 tx=0x7fb41c00b8e0).stop 2021-02-10T11:49:21.356 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.352 7fb4310de700 1 -- 172.21.2.112:0/3541180532 shutdown_connections 2021-02-10T11:49:21.356 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.352 7fb4310de700 1 --2- 172.21.2.112:0/3541180532 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb410018730 0x7fb41001ac10 unknown :-1 s=CLOSED pgs=1484 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:21.356 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.352 7fb4310de700 1 --2- 172.21.2.112:0/3541180532 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb42c083910 0x7fb42c083d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:21.356 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.352 7fb4310de700 1 --2- 172.21.2.112:0/3541180532 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb42c0ac780 0x7fb42c0acb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:21.356 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.352 7fb4310de700 1 --2- 172.21.2.112:0/3541180532 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb42c0abd60 0x7fb42c0ac160 unknown :-1 s=CLOSED pgs=3294 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:21.356 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.352 7fb4310de700 1 -- 172.21.2.112:0/3541180532 >> 172.21.2.112:0/3541180532 conn(0x7fb42c0a2b30 msgr2=0x7fb42c09ce60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:21.356 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.352 7fb4310de700 1 -- 172.21.2.112:0/3541180532 shutdown_connections 2021-02-10T11:49:21.356 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.352 7fb4310de700 1 -- 172.21.2.112:0/3541180532 wait complete. 2021-02-10T11:49:21.364 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:49:21.364 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:49:21.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.772 7fbe1fee2700 1 Processor -- start 2021-02-10T11:49:21.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.772 7fbe1fee2700 1 -- start start 2021-02-10T11:49:21.792 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.773 7fbe1fee2700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbe180d1870 0x7fbe181b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:21.793 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.773 7fbe1fee2700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fbe181b0a10 0x7fbe180cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:21.793 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.773 7fbe1fee2700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fbe180d1c70 0x7fbe180cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:21.793 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.773 7fbe1fee2700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe180fe230 con 0x7fbe181b0a10 2021-02-10T11:49:21.793 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.773 7fbe1fee2700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe180e5ed0 con 0x7fbe180d1c70 2021-02-10T11:49:21.793 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.773 7fbe1fee2700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe180f0b10 con 0x7fbe180d1870 2021-02-10T11:49:21.793 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.773 7fbe1d4fd700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fbe181b0a10 0x7fbe180cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:21.793 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.773 7fbe1d4fd700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fbe181b0a10 0x7fbe180cf330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:33802/0 (socket says 172.21.2.112:33802) 2021-02-10T11:49:21.793 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.773 7fbe1d4fd700 1 -- 172.21.2.112:0/313564283 learned_addr learned my addr 172.21.2.112:0/313564283 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:49:21.793 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.773 7fbe1d4fd700 1 -- 172.21.2.112:0/313564283 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbe180d1870 msgr2=0x7fbe181b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:49:21.793 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.773 7fbe1d4fd700 1 --2- 172.21.2.112:0/313564283 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbe180d1870 0x7fbe181b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:21.793 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.773 7fbe1d4fd700 1 -- 172.21.2.112:0/313564283 >> v2:172.21.2.120:3300/0 conn(0x7fbe180d1c70 msgr2=0x7fbe180cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:49:21.793 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.773 7fbe1d4fd700 1 --2- 172.21.2.112:0/313564283 >> v2:172.21.2.120:3300/0 conn(0x7fbe180d1c70 0x7fbe180cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:21.793 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.773 7fbe1d4fd700 1 -- 172.21.2.112:0/313564283 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe180cff70 con 0x7fbe181b0a10 2021-02-10T11:49:21.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.773 7fbe1d4fd700 1 --2- 172.21.2.112:0/313564283 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbe181b0a10 0x7fbe180cf330 secure :-1 s=READY pgs=3295 cs=0 l=1 rev1=1 rx=0x7fbe0c006700 tx=0x7fbe0c009b80).ready entity=mon.0 client_cookie=8b3ce9366af4a79f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:21.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.774 7fbe1ccfc700 1 -- 172.21.2.112:0/313564283 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbe0c00c0c0 con 0x7fbe181b0a10 2021-02-10T11:49:21.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.774 7fbe1ccfc700 1 -- 172.21.2.112:0/313564283 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe0c00c440 con 0x7fbe181b0a10 2021-02-10T11:49:21.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.774 7fbe1ccfc700 1 -- 172.21.2.112:0/313564283 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbe0c00c8a0 con 0x7fbe181b0a10 2021-02-10T11:49:21.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.774 7fbe1fee2700 1 -- 172.21.2.112:0/313564283 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbe181b0a10 msgr2=0x7fbe180cf330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:21.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.774 7fbe1fee2700 1 --2- 172.21.2.112:0/313564283 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbe181b0a10 0x7fbe180cf330 secure :-1 s=READY pgs=3295 cs=0 l=1 rev1=1 rx=0x7fbe0c006700 tx=0x7fbe0c009b80).stop 2021-02-10T11:49:21.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.774 7fbe1fee2700 1 -- 172.21.2.112:0/313564283 shutdown_connections 2021-02-10T11:49:21.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.774 7fbe1fee2700 1 --2- 172.21.2.112:0/313564283 >> v2:172.21.2.120:3300/0 conn(0x7fbe180d1c70 0x7fbe180cf950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:21.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.774 7fbe1fee2700 1 --2- 172.21.2.112:0/313564283 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbe181b0a10 0x7fbe180cf330 unknown :-1 s=CLOSED pgs=3295 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:21.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.774 7fbe1fee2700 1 --2- 172.21.2.112:0/313564283 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbe180d1870 0x7fbe181b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:21.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.774 7fbe1fee2700 1 -- 172.21.2.112:0/313564283 >> 172.21.2.112:0/313564283 conn(0x7fbe180ce450 msgr2=0x7fbe180d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:21.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.774 7fbe1fee2700 1 -- 172.21.2.112:0/313564283 shutdown_connections 2021-02-10T11:49:21.794 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.774 7fbe1fee2700 1 -- 172.21.2.112:0/313564283 wait complete. 2021-02-10T11:49:21.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.789 7fbe1fee2700 1 Processor -- start 2021-02-10T11:49:21.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.789 7fbe1fee2700 1 -- start start 2021-02-10T11:49:21.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.789 7fbe1fee2700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbe181b7cf0 0x7fbe1819cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:21.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.789 7fbe1fee2700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fbe181b80f0 0x7fbe181b84f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:21.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.789 7fbe1fee2700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbe1819da20 0x7fbe1819de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:21.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.789 7fbe1fee2700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe180a77c0 con 0x7fbe1819da20 2021-02-10T11:49:21.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.789 7fbe1fee2700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe1819d4a0 con 0x7fbe181b80f0 2021-02-10T11:49:21.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.789 7fbe1fee2700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe180954f0 con 0x7fbe181b7cf0 2021-02-10T11:49:21.796 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.790 7fbe1e4ff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbe1819da20 0x7fbe1819de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:21.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.790 7fbe1e4ff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbe1819da20 0x7fbe1819de20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:33810/0 (socket says 172.21.2.112:33810) 2021-02-10T11:49:21.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.790 7fbe1e4ff700 1 -- 172.21.2.112:0/3331686984 learned_addr learned my addr 172.21.2.112:0/3331686984 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:49:21.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.790 7fbe1e4ff700 1 -- 172.21.2.112:0/3331686984 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbe181b7cf0 msgr2=0x7fbe1819cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:21.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.790 7fbe1e4ff700 1 --2- 172.21.2.112:0/3331686984 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbe181b7cf0 0x7fbe1819cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:21.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.790 7fbe1e4ff700 1 -- 172.21.2.112:0/3331686984 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fbe181b80f0 msgr2=0x7fbe181b84f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:21.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.790 7fbe1e4ff700 1 --2- 172.21.2.112:0/3331686984 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fbe181b80f0 0x7fbe181b84f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:21.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.790 7fbe1e4ff700 1 -- 172.21.2.112:0/3331686984 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe181b8d60 con 0x7fbe1819da20 2021-02-10T11:49:21.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.790 7fbe1e4ff700 1 --2- 172.21.2.112:0/3331686984 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbe1819da20 0x7fbe1819de20 secure :-1 s=READY pgs=3296 cs=0 l=1 rev1=1 rx=0x7fbe10009980 tx=0x7fbe1000bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:21.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.790 7fbe0a7fc700 1 -- 172.21.2.112:0/3331686984 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbe10005af0 con 0x7fbe1819da20 2021-02-10T11:49:21.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.790 7fbe0a7fc700 1 -- 172.21.2.112:0/3331686984 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe10005d70 con 0x7fbe1819da20 2021-02-10T11:49:21.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.790 7fbe0a7fc700 1 -- 172.21.2.112:0/3331686984 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbe100062b0 con 0x7fbe1819da20 2021-02-10T11:49:21.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.790 7fbe1fee2700 1 -- 172.21.2.112:0/3331686984 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe181b8d60 con 0x7fbe1819da20 2021-02-10T11:49:21.797 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.791 7fbe1fee2700 1 -- 172.21.2.112:0/3331686984 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe181b8d60 con 0x7fbe1819da20 2021-02-10T11:49:21.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.791 7fbe0a7fc700 1 -- 172.21.2.112:0/3331686984 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fbe10006030 con 0x7fbe1819da20 2021-02-10T11:49:21.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.791 7fbe0a7fc700 1 --2- 172.21.2.112:0/3331686984 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fbe000186c0 0x7fbe0001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:21.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.791 7fbe0a7fc700 1 -- 172.21.2.112:0/3331686984 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fbe10032770 con 0x7fbe1819da20 2021-02-10T11:49:21.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.792 7fbe1dcfe700 1 --2- 172.21.2.112:0/3331686984 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fbe000186c0 0x7fbe0001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:21.798 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.792 7fbe1dcfe700 1 --2- 172.21.2.112:0/3331686984 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fbe000186c0 0x7fbe0001abc0 crc :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:21.826 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.822 7fbe1fee2700 1 -- 172.21.2.112:0/3331686984 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe180ebc00 con 0x7fbe1819da20 2021-02-10T11:49:21.829 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.825 7fbe0a7fc700 1 -- 172.21.2.112:0/3331686984 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fbe10005af0 con 0x7fbe1819da20 2021-02-10T11:49:21.958 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.954 7fbe1fee2700 1 -- 172.21.2.112:0/3331686984 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbe180f0b10 con 0x7fbe1819da20 2021-02-10T11:49:21.958 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.955 7fbe0a7fc700 1 -- 172.21.2.112:0/3331686984 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 174 v174) v1 ==== 95+0+2108 (secure 0 0 0) 0x7fbe10006030 con 0x7fbe1819da20 2021-02-10T11:49:21.959 INFO:teuthology.orchestra.run.gibba012.stderr:dumped fsmap epoch 174 2021-02-10T11:49:21.959 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:49:21.959 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":174,"default_fscid":21,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":13800,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.112:6833/2202619473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6832","nonce":2202619473},{"type":"v1","addr":"172.21.2.112:6833","nonce":2202619473}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":170}],"filesystems":[{"mdsmap":{"epoch":174,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:49:04.478835","modified":"2021-02-10 11:49:06.484559","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9715},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9715":{"gid":9715,"name":"b","rank":0,"incarnation":171,"state":"up:active","state_seq":3,"addr":"172.21.2.120:6835/3826510568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6834","nonce":3826510568},{"type":"v1","addr":"172.21.2.120:6835","nonce":3826510568}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[43],"metadata_pool":42,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":21}]} 2021-02-10T11:49:21.960 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.956 7fbe1fee2700 1 -- 172.21.2.112:0/3331686984 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fbe000186c0 msgr2=0x7fbe0001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:21.960 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.956 7fbe1fee2700 1 --2- 172.21.2.112:0/3331686984 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fbe000186c0 0x7fbe0001abc0 crc :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:21.960 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.956 7fbe1fee2700 1 -- 172.21.2.112:0/3331686984 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbe1819da20 msgr2=0x7fbe1819de20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:21.961 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.956 7fbe1fee2700 1 --2- 172.21.2.112:0/3331686984 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbe1819da20 0x7fbe1819de20 secure :-1 s=READY pgs=3296 cs=0 l=1 rev1=1 rx=0x7fbe10009980 tx=0x7fbe1000bcd0).stop 2021-02-10T11:49:21.961 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.957 7fbe1fee2700 1 -- 172.21.2.112:0/3331686984 shutdown_connections 2021-02-10T11:49:21.961 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.957 7fbe1fee2700 1 --2- 172.21.2.112:0/3331686984 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fbe000186c0 0x7fbe0001abc0 unknown :-1 s=CLOSED pgs=1485 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:21.961 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.957 7fbe1fee2700 1 --2- 172.21.2.112:0/3331686984 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fbe181b80f0 0x7fbe181b84f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:21.961 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.957 7fbe1fee2700 1 --2- 172.21.2.112:0/3331686984 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbe1819da20 0x7fbe1819de20 unknown :-1 s=CLOSED pgs=3296 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:21.961 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.957 7fbe1fee2700 1 --2- 172.21.2.112:0/3331686984 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbe181b7cf0 0x7fbe1819cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:21.961 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.957 7fbe1fee2700 1 -- 172.21.2.112:0/3331686984 >> 172.21.2.112:0/3331686984 conn(0x7fbe180ed810 msgr2=0x7fbe18094980 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:21.961 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.957 7fbe1fee2700 1 -- 172.21.2.112:0/3331686984 shutdown_connections 2021-02-10T11:49:21.961 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:21.957 7fbe1fee2700 1 -- 172.21.2.112:0/3331686984 wait complete. 2021-02-10T11:49:22.170 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 174, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:49:04.478835', 'modified': '2021-02-10 11:49:06.484559', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9715}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9715': {'gid': 9715, 'name': 'b', 'rank': 0, 'incarnation': 171, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.120:6835/3826510568', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.120:6834', 'nonce': 3826510568}, {'type': 'v1', 'addr': '172.21.2.120:6835', 'nonce': 3826510568}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [43], 'metadata_pool': 42, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:49:22.170 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:49:22.170 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:49:22.569 INFO:teuthology.orchestra.run.gibba020.stdout:{ 2021-02-10T11:49:22.570 INFO:teuthology.orchestra.run.gibba020.stdout: "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", 2021-02-10T11:49:22.570 INFO:teuthology.orchestra.run.gibba020.stdout: "whoami": 0, 2021-02-10T11:49:22.570 INFO:teuthology.orchestra.run.gibba020.stdout: "id": 9715, 2021-02-10T11:49:22.570 INFO:teuthology.orchestra.run.gibba020.stdout: "want_state": "up:active", 2021-02-10T11:49:22.570 INFO:teuthology.orchestra.run.gibba020.stdout: "state": "up:active", 2021-02-10T11:49:22.570 INFO:teuthology.orchestra.run.gibba020.stdout: "rank_uptime": 18.085121879999999, 2021-02-10T11:49:22.570 INFO:teuthology.orchestra.run.gibba020.stdout: "mdsmap_epoch": 174, 2021-02-10T11:49:22.570 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch": 198, 2021-02-10T11:49:22.570 INFO:teuthology.orchestra.run.gibba020.stdout: "osdmap_epoch_barrier": 198, 2021-02-10T11:49:22.570 INFO:teuthology.orchestra.run.gibba020.stdout: "uptime": 23.735291635999999 2021-02-10T11:49:22.571 INFO:teuthology.orchestra.run.gibba020.stdout:} 2021-02-10T11:49:22.574 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "f14f004e-0699-4584-ad81-3754258493de", "whoami": 0, "id": 9715, "want_state": "up:active", "state": "up:active", "rank_uptime": 18.085121879999999, "mdsmap_epoch": 174, "osdmap_epoch": 198, "osdmap_epoch_barrier": 198, "uptime": 23.735291635999999 } 2021-02-10T11:49:22.574 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:49:22.574 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2021-02-10T11:49:22.574 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.1 at ubuntu@gibba012.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2021-02-10T11:49:22.575 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:49:22.575 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:49:22.912 DEBUG:teuthology.orchestra.run.gibba012:> sudo modprobe fuse 2021-02-10T11:49:23.256 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:49:23.257 DEBUG:teuthology.orchestra.run.gibba012:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:49:23.601 INFO:teuthology.orchestra.run.gibba012.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:49:23.601 INFO:teuthology.orchestra.run.gibba012.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:49:23.602 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:49:23.602 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:49:23.603 DEBUG:teuthology.orchestra.run.gibba012:> ls /sys/fs/fuse/connections 2021-02-10T11:49:23.938 INFO:teuthology.orchestra.run.gibba012.stdout:42 2021-02-10T11:49:23.938 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [42] 2021-02-10T11:49:23.938 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.1 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:49:23.941 DEBUG:teuthology.orchestra.run.gibba012:> sudo modprobe fuse 2021-02-10T11:49:24.286 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:49:24.286 DEBUG:teuthology.orchestra.run.gibba012:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:49:24.353 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba012.stderr:ceph-fuse[197213]: starting ceph client 2021-02-10T11:49:24.353 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba012.stderr:2021-02-10 11:49:24.349 7fb1f4420f80 -1 init, newargv = 0x55b6473c81f0 newargc=9 2021-02-10T11:49:24.358 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba012.stderr:ceph-fuse[197213]: starting fuse 2021-02-10T11:49:24.629 INFO:teuthology.orchestra.run.gibba012.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:49:24.629 INFO:teuthology.orchestra.run.gibba012.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:49:24.629 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:49:24.630 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:49:24.630 DEBUG:teuthology.orchestra.run.gibba012:> ls /sys/fs/fuse/connections 2021-02-10T11:49:24.966 INFO:teuthology.orchestra.run.gibba012.stdout:42 2021-02-10T11:49:24.966 INFO:teuthology.orchestra.run.gibba012.stdout:43 2021-02-10T11:49:24.966 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42, 43] 2021-02-10T11:49:24.967 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:49:24.967 DEBUG:teuthology.orchestra.run.gibba012:> sudo python3 -c ' 2021-02-10T11:49:24.967 DEBUG:teuthology.orchestra.run.gibba012:> import glob 2021-02-10T11:49:24.967 DEBUG:teuthology.orchestra.run.gibba012:> import re 2021-02-10T11:49:24.967 DEBUG:teuthology.orchestra.run.gibba012:> import os 2021-02-10T11:49:24.967 DEBUG:teuthology.orchestra.run.gibba012:> import subprocess 2021-02-10T11:49:24.967 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:49:24.967 DEBUG:teuthology.orchestra.run.gibba012:> def find_socket(client_name): 2021-02-10T11:49:24.967 DEBUG:teuthology.orchestra.run.gibba012:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2021-02-10T11:49:24.967 DEBUG:teuthology.orchestra.run.gibba012:> files = glob.glob(asok_path) 2021-02-10T11:49:24.967 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:49:24.967 DEBUG:teuthology.orchestra.run.gibba012:> # Given a non-glob path, it better be there 2021-02-10T11:49:24.968 DEBUG:teuthology.orchestra.run.gibba012:> if "*" not in asok_path: 2021-02-10T11:49:24.968 DEBUG:teuthology.orchestra.run.gibba012:> assert(len(files) == 1) 2021-02-10T11:49:24.968 DEBUG:teuthology.orchestra.run.gibba012:> return files[0] 2021-02-10T11:49:24.968 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:49:24.968 DEBUG:teuthology.orchestra.run.gibba012:> for f in files: 2021-02-10T11:49:24.968 DEBUG:teuthology.orchestra.run.gibba012:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:49:24.968 DEBUG:teuthology.orchestra.run.gibba012:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:49:24.968 DEBUG:teuthology.orchestra.run.gibba012:> return f 2021-02-10T11:49:24.968 DEBUG:teuthology.orchestra.run.gibba012:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:49:24.968 DEBUG:teuthology.orchestra.run.gibba012:> 2021-02-10T11:49:24.968 DEBUG:teuthology.orchestra.run.gibba012:> print(find_socket("client.1")) 2021-02-10T11:49:24.968 DEBUG:teuthology.orchestra.run.gibba012:> ' 2021-02-10T11:49:25.331 INFO:teuthology.orchestra.run.gibba012.stdout:/var/run/ceph/ceph-client.1.197213.asok 2021-02-10T11:49:25.422 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.197213.asok 2021-02-10T11:49:25.422 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:49:25.423 DEBUG:teuthology.orchestra.run.gibba012:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.197213.asok status 2021-02-10T11:49:25.813 INFO:teuthology.orchestra.run.gibba012.stdout:{ 2021-02-10T11:49:25.814 INFO:teuthology.orchestra.run.gibba012.stdout: "metadata": { 2021-02-10T11:49:25.814 INFO:teuthology.orchestra.run.gibba012.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:49:25.814 INFO:teuthology.orchestra.run.gibba012.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:49:25.814 INFO:teuthology.orchestra.run.gibba012.stdout: "entity_id": "1", 2021-02-10T11:49:25.814 INFO:teuthology.orchestra.run.gibba012.stdout: "hostname": "gibba012", 2021-02-10T11:49:25.814 INFO:teuthology.orchestra.run.gibba012.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2021-02-10T11:49:25.814 INFO:teuthology.orchestra.run.gibba012.stdout: "pid": "197213", 2021-02-10T11:49:25.814 INFO:teuthology.orchestra.run.gibba012.stdout: "root": "/" 2021-02-10T11:49:25.814 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:49:25.815 INFO:teuthology.orchestra.run.gibba012.stdout: "dentry_count": 0, 2021-02-10T11:49:25.815 INFO:teuthology.orchestra.run.gibba012.stdout: "dentry_pinned_count": 0, 2021-02-10T11:49:25.815 INFO:teuthology.orchestra.run.gibba012.stdout: "id": 14001, 2021-02-10T11:49:25.815 INFO:teuthology.orchestra.run.gibba012.stdout: "inst": { 2021-02-10T11:49:25.815 INFO:teuthology.orchestra.run.gibba012.stdout: "name": { 2021-02-10T11:49:25.815 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "client", 2021-02-10T11:49:25.815 INFO:teuthology.orchestra.run.gibba012.stdout: "num": 14001 2021-02-10T11:49:25.815 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:49:25.815 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": { 2021-02-10T11:49:25.815 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:49:25.815 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:0", 2021-02-10T11:49:25.815 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 3566675864 2021-02-10T11:49:25.815 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:49:25.816 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:49:25.816 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": { 2021-02-10T11:49:25.816 INFO:teuthology.orchestra.run.gibba012.stdout: "type": "v1", 2021-02-10T11:49:25.816 INFO:teuthology.orchestra.run.gibba012.stdout: "addr": "172.21.2.112:0", 2021-02-10T11:49:25.816 INFO:teuthology.orchestra.run.gibba012.stdout: "nonce": 3566675864 2021-02-10T11:49:25.816 INFO:teuthology.orchestra.run.gibba012.stdout: }, 2021-02-10T11:49:25.816 INFO:teuthology.orchestra.run.gibba012.stdout: "inst_str": "client.14001 172.21.2.112:0/3566675864", 2021-02-10T11:49:25.816 INFO:teuthology.orchestra.run.gibba012.stdout: "addr_str": "172.21.2.112:0/3566675864", 2021-02-10T11:49:25.816 INFO:teuthology.orchestra.run.gibba012.stdout: "inode_count": 1, 2021-02-10T11:49:25.817 INFO:teuthology.orchestra.run.gibba012.stdout: "mds_epoch": 174, 2021-02-10T11:49:25.817 INFO:teuthology.orchestra.run.gibba012.stdout: "osd_epoch": 198, 2021-02-10T11:49:25.817 INFO:teuthology.orchestra.run.gibba012.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:49:25.817 INFO:teuthology.orchestra.run.gibba012.stdout: "blacklisted": false 2021-02-10T11:49:25.817 INFO:teuthology.orchestra.run.gibba012.stdout:} 2021-02-10T11:49:25.818 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:49:25.818 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:49:25.818 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:49:26.154 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:49:26.155 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:49:26.155 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:49:26.155 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:49:26.502 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2021-02-10T11:49:26.935 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:26.915 7f6ca1456700 1 Processor -- start 2021-02-10T11:49:26.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:26.915 7f6ca1456700 1 -- start start 2021-02-10T11:49:26.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:26.916 7f6ca1456700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6c9c0d1c80 0x7f6c9c0cef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:26.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:26.916 7f6ca1456700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f6c9c0cf590 0x7f6c9c0cf990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:26.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:26.916 7f6ca1456700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c9c0fe1d0 con 0x7f6c9c0d1880 2021-02-10T11:49:26.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:26.916 7f6ca1456700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c9c1b0cc0 con 0x7f6c9c0cf590 2021-02-10T11:49:26.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:26.916 7f6ca1456700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c9c109a70 con 0x7f6c9c0d1c80 2021-02-10T11:49:26.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:26.916 7f6c9b07d700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f6c9c0d1880 0x7f6c9c1b04d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:33288/0 (socket says 172.21.2.112:33288) 2021-02-10T11:49:26.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:26.916 7f6c9b07d700 1 -- 172.21.2.112:0/919580432 learned_addr learned my addr 172.21.2.112:0/919580432 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:49:26.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:26.916 7f6c9b87e700 1 --2- 172.21.2.112:0/919580432 >> v2:172.21.2.112:3300/0 conn(0x7f6c9c0cf590 0x7f6c9c0cf990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:26.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:26.916 7f6c9a07b700 1 -- 172.21.2.112:0/919580432 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3170651996 0 0) 0x7f6c94001500 con 0x7f6c9c0d1880 2021-02-10T11:49:26.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:26.916 7f6c9b87e700 1 -- 172.21.2.112:0/919580432 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6c9c0d1c80 msgr2=0x7f6c9c0cef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:49:26.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:26.916 7f6c9b87e700 1 --2- 172.21.2.112:0/919580432 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6c9c0d1c80 0x7f6c9c0cef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:26.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:26.916 7f6c9b87e700 1 -- 172.21.2.112:0/919580432 >> v1:172.21.2.112:6789/0 conn(0x7f6c9c0d1880 legacy=0x7f6c9c1b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:26.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:26.916 7f6c9b87e700 1 -- 172.21.2.112:0/919580432 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c9c088070 con 0x7f6c9c0cf590 2021-02-10T11:49:26.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:26.916 7f6c9b87e700 1 --2- 172.21.2.112:0/919580432 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6c9c0cf590 0x7f6c9c0cf990 secure :-1 s=READY pgs=3301 cs=0 l=1 rev1=1 rx=0x7f6c8c006ab0 tx=0x7f6c8c009b40).ready entity=mon.0 client_cookie=845f77441da298fb server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:26.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:26.917 7f6c9a07b700 1 -- 172.21.2.112:0/919580432 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6c8c0022e0 con 0x7f6c9c0cf590 2021-02-10T11:49:26.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:26.917 7f6c9a07b700 1 -- 172.21.2.112:0/919580432 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c8c0066d0 con 0x7f6c9c0cf590 2021-02-10T11:49:26.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:26.917 7f6c9a07b700 1 -- 172.21.2.112:0/919580432 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6c8c0028a0 con 0x7f6c9c0cf590 2021-02-10T11:49:26.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:26.917 7f6ca1456700 1 -- 172.21.2.112:0/919580432 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6c9c0cf590 msgr2=0x7f6c9c0cf990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:26.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:26.917 7f6ca1456700 1 --2- 172.21.2.112:0/919580432 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6c9c0cf590 0x7f6c9c0cf990 secure :-1 s=READY pgs=3301 cs=0 l=1 rev1=1 rx=0x7f6c8c006ab0 tx=0x7f6c8c009b40).stop 2021-02-10T11:49:26.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:26.917 7f6ca1456700 1 -- 172.21.2.112:0/919580432 shutdown_connections 2021-02-10T11:49:26.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:26.917 7f6ca1456700 1 --2- 172.21.2.112:0/919580432 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6c9c0cf590 0x7f6c9c0cf990 unknown :-1 s=CLOSED pgs=3301 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:26.938 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:26.917 7f6ca1456700 1 --2- 172.21.2.112:0/919580432 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6c9c0d1c80 0x7f6c9c0cef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:26.938 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:26.917 7f6ca1456700 1 -- 172.21.2.112:0/919580432 >> 172.21.2.112:0/919580432 conn(0x7f6c9c0ce430 msgr2=0x7f6c9c0d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:26.938 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:26.917 7f6ca1456700 1 -- 172.21.2.112:0/919580432 shutdown_connections 2021-02-10T11:49:26.938 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:26.917 7f6ca1456700 1 -- 172.21.2.112:0/919580432 wait complete. 2021-02-10T11:49:26.940 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:26.932 7f6ca1456700 1 Processor -- start 2021-02-10T11:49:26.940 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:26.932 7f6ca1456700 1 -- start start 2021-02-10T11:49:26.940 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:26.933 7f6ca1456700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6c9c190df0 0x7f6c9c1b7ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:26.940 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:26.933 7f6ca1456700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6c9c19ba20 0x7f6c9c1b87c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:26.940 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:26.933 7f6ca1456700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6c9c19be20 0x7f6c9c18f830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:26.940 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:26.933 7f6ca1456700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c9c1b8240 con 0x7f6c9c190df0 2021-02-10T11:49:26.940 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:26.933 7f6ca1456700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c9c1911f0 con 0x7f6c9c19be20 2021-02-10T11:49:26.940 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:26.933 7f6ca1456700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c9c1908a0 con 0x7f6c9c19ba20 2021-02-10T11:49:26.940 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:26.933 7f6c9b07d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6c9c190df0 0x7f6c9c1b7ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:26.940 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:26.933 7f6c9b07d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6c9c190df0 0x7f6c9c1b7ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:33832/0 (socket says 172.21.2.112:33832) 2021-02-10T11:49:26.940 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:26.933 7f6c9b07d700 1 -- 172.21.2.112:0/67313297 learned_addr learned my addr 172.21.2.112:0/67313297 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:49:26.941 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:26.933 7f6c9b87e700 1 --2- 172.21.2.112:0/67313297 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6c9c19be20 0x7f6c9c18f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:26.941 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:26.933 7f6c9a87c700 1 --2- 172.21.2.112:0/67313297 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6c9c19ba20 0x7f6c9c1b87c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:26.941 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:26.933 7f6c9b07d700 1 -- 172.21.2.112:0/67313297 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6c9c19ba20 msgr2=0x7f6c9c1b87c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:26.941 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:26.933 7f6c9b07d700 1 --2- 172.21.2.112:0/67313297 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6c9c19ba20 0x7f6c9c1b87c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:26.941 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:26.933 7f6c9b07d700 1 -- 172.21.2.112:0/67313297 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6c9c19be20 msgr2=0x7f6c9c18f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:26.941 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:26.933 7f6c9b07d700 1 --2- 172.21.2.112:0/67313297 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6c9c19be20 0x7f6c9c18f830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:26.941 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:26.933 7f6c9b07d700 1 -- 172.21.2.112:0/67313297 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c9c1b8de0 con 0x7f6c9c190df0 2021-02-10T11:49:26.941 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:26.933 7f6c9b87e700 1 --2- 172.21.2.112:0/67313297 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6c9c19be20 0x7f6c9c18f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:49:26.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:26.933 7f6c9a87c700 1 --2- 172.21.2.112:0/67313297 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6c9c19ba20 0x7f6c9c1b87c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:49:26.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:26.933 7f6c9b07d700 1 --2- 172.21.2.112:0/67313297 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6c9c190df0 0x7f6c9c1b7ac0 secure :-1 s=READY pgs=3302 cs=0 l=1 rev1=1 rx=0x7f6c9400a940 tx=0x7f6c9400ace0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:26.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:26.934 7f6c93fff700 1 -- 172.21.2.112:0/67313297 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6c940070d0 con 0x7f6c9c190df0 2021-02-10T11:49:26.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:26.934 7f6c93fff700 1 -- 172.21.2.112:0/67313297 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c940072d0 con 0x7f6c9c190df0 2021-02-10T11:49:26.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:26.934 7f6c93fff700 1 -- 172.21.2.112:0/67313297 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6c94004720 con 0x7f6c9c190df0 2021-02-10T11:49:26.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:26.934 7f6ca1456700 1 -- 172.21.2.112:0/67313297 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c9c1b8de0 con 0x7f6c9c190df0 2021-02-10T11:49:26.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:26.934 7f6ca1456700 1 -- 172.21.2.112:0/67313297 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c9c1b8de0 con 0x7f6c9c190df0 2021-02-10T11:49:26.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:26.934 7f6c93fff700 1 -- 172.21.2.112:0/67313297 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6c940074b0 con 0x7f6c9c190df0 2021-02-10T11:49:26.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:26.934 7f6c93fff700 1 --2- 172.21.2.112:0/67313297 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6c800186c0 0x7f6c8001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:26.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:26.934 7f6c93fff700 1 -- 172.21.2.112:0/67313297 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f6c940328e0 con 0x7f6c9c190df0 2021-02-10T11:49:26.943 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:26.935 7f6c9a87c700 1 --2- 172.21.2.112:0/67313297 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6c800186c0 0x7f6c8001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:26.943 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:26.935 7f6c9a87c700 1 --2- 172.21.2.112:0/67313297 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6c800186c0 0x7f6c8001abc0 crc :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:26.969 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:26.966 7f6ca1456700 1 -- 172.21.2.112:0/67313297 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c9c082aa0 con 0x7f6c9c190df0 2021-02-10T11:49:26.972 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:26.968 7f6c93fff700 1 -- 172.21.2.112:0/67313297 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6c9402ab60 con 0x7f6c9c190df0 2021-02-10T11:49:27.109 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:27.105 7f6ca1456700 1 -- 172.21.2.112:0/67313297 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f6c9c082aa0 con 0x7f6c9c190df0 2021-02-10T11:49:27.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:27.321 7f6c93fff700 1 -- 172.21.2.112:0/67313297 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v175) v1 ==== 136+0+0 (secure 0 0 0) 0x7f6c9402ab60 con 0x7f6c9c190df0 2021-02-10T11:49:27.375 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:27.371 7f6ca1456700 1 -- 172.21.2.112:0/67313297 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6c800186c0 msgr2=0x7f6c8001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:27.375 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:27.371 7f6ca1456700 1 --2- 172.21.2.112:0/67313297 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6c800186c0 0x7f6c8001abc0 crc :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:27.376 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:27.371 7f6ca1456700 1 -- 172.21.2.112:0/67313297 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6c9c190df0 msgr2=0x7f6c9c1b7ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:27.376 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:27.371 7f6ca1456700 1 --2- 172.21.2.112:0/67313297 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6c9c190df0 0x7f6c9c1b7ac0 secure :-1 s=READY pgs=3302 cs=0 l=1 rev1=1 rx=0x7f6c9400a940 tx=0x7f6c9400ace0).stop 2021-02-10T11:49:27.376 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:27.371 7f6ca1456700 1 -- 172.21.2.112:0/67313297 shutdown_connections 2021-02-10T11:49:27.376 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:27.371 7f6ca1456700 1 --2- 172.21.2.112:0/67313297 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6c800186c0 0x7f6c8001abc0 unknown :-1 s=CLOSED pgs=1486 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:27.376 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:27.371 7f6ca1456700 1 --2- 172.21.2.112:0/67313297 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6c9c19be20 0x7f6c9c18f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:27.376 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:27.371 7f6ca1456700 1 --2- 172.21.2.112:0/67313297 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6c9c19ba20 0x7f6c9c1b87c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:27.376 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:27.371 7f6ca1456700 1 --2- 172.21.2.112:0/67313297 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6c9c190df0 0x7f6c9c1b7ac0 unknown :-1 s=CLOSED pgs=3302 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:27.376 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:27.371 7f6ca1456700 1 -- 172.21.2.112:0/67313297 >> 172.21.2.112:0/67313297 conn(0x7f6c9c19f940 msgr2=0x7f6c9c18c290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:27.376 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:27.372 7f6ca1456700 1 -- 172.21.2.112:0/67313297 shutdown_connections 2021-02-10T11:49:27.376 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:27.372 7f6ca1456700 1 -- 172.21.2.112:0/67313297 wait complete. 2021-02-10T11:49:27.384 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs volume ls 2021-02-10T11:49:27.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:27.792 7f5e8329a700 1 Processor -- start 2021-02-10T11:49:27.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:27.792 7f5e8329a700 1 -- start start 2021-02-10T11:49:27.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:27.793 7f5e8329a700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f5e7c0d1870 0x7f5e7c1b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:27.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:27.793 7f5e8329a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5e7c1b0a10 0x7f5e7c0cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:27.812 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:27.793 7f5e8329a700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f5e7c0d1c70 0x7f5e7c0cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:27.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:27.793 7f5e8329a700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e7c0fe200 con 0x7f5e7c0d1c70 2021-02-10T11:49:27.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:27.793 7f5e8329a700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e7c0e5f10 con 0x7f5e7c0d1870 2021-02-10T11:49:27.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:27.793 7f5e8329a700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e7c0f0b50 con 0x7f5e7c1b0a10 2021-02-10T11:49:27.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:27.793 7f5e818b7700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f5e7c0d1c70 0x7f5e7c0cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:27.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:27.793 7f5e818b7700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f5e7c0d1c70 0x7f5e7c0cf950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:33840/0 (socket says 172.21.2.112:33840) 2021-02-10T11:49:27.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:27.793 7f5e818b7700 1 -- 172.21.2.112:0/1328433585 learned_addr learned my addr 172.21.2.112:0/1328433585 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:49:27.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:27.793 7f5e810b6700 1 --2- 172.21.2.112:0/1328433585 >> v2:172.21.2.120:3300/0 conn(0x7f5e7c0d1870 0x7f5e7c1b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:27.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:27.793 7f5e808b5700 1 --2- 172.21.2.112:0/1328433585 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5e7c1b0a10 0x7f5e7c0cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:27.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:27.793 7f5e818b7700 1 -- 172.21.2.112:0/1328433585 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5e7c1b0a10 msgr2=0x7f5e7c0cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:49:27.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:27.793 7f5e818b7700 1 --2- 172.21.2.112:0/1328433585 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5e7c1b0a10 0x7f5e7c0cf330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:27.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:27.793 7f5e818b7700 1 -- 172.21.2.112:0/1328433585 >> v2:172.21.2.120:3300/0 conn(0x7f5e7c0d1870 msgr2=0x7f5e7c1b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:49:27.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:27.793 7f5e818b7700 1 --2- 172.21.2.112:0/1328433585 >> v2:172.21.2.120:3300/0 conn(0x7f5e7c0d1870 0x7f5e7c1b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:27.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:27.793 7f5e818b7700 1 -- 172.21.2.112:0/1328433585 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e7c0cff70 con 0x7f5e7c0d1c70 2021-02-10T11:49:27.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:27.793 7f5e810b6700 1 --2- 172.21.2.112:0/1328433585 >> v2:172.21.2.120:3300/0 conn(0x7f5e7c0d1870 0x7f5e7c1b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:49:27.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:27.793 7f5e818b7700 1 --2- 172.21.2.112:0/1328433585 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5e7c0d1c70 0x7f5e7c0cf950 secure :-1 s=READY pgs=3303 cs=0 l=1 rev1=1 rx=0x7f5e74006f00 tx=0x7f5e74009f10).ready entity=mon.0 client_cookie=161349874033eff7 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:27.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:27.793 7f5e6ffff700 1 -- 172.21.2.112:0/1328433585 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5e7400b3b0 con 0x7f5e7c0d1c70 2021-02-10T11:49:27.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:27.794 7f5e6ffff700 1 -- 172.21.2.112:0/1328433585 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e7400b730 con 0x7f5e7c0d1c70 2021-02-10T11:49:27.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:27.794 7f5e8329a700 1 -- 172.21.2.112:0/1328433585 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5e7c0d1c70 msgr2=0x7f5e7c0cf950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:27.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:27.794 7f5e8329a700 1 --2- 172.21.2.112:0/1328433585 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5e7c0d1c70 0x7f5e7c0cf950 secure :-1 s=READY pgs=3303 cs=0 l=1 rev1=1 rx=0x7f5e74006f00 tx=0x7f5e74009f10).stop 2021-02-10T11:49:27.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:27.794 7f5e6ffff700 1 -- 172.21.2.112:0/1328433585 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5e7400bc10 con 0x7f5e7c0d1c70 2021-02-10T11:49:27.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:27.794 7f5e8329a700 1 -- 172.21.2.112:0/1328433585 shutdown_connections 2021-02-10T11:49:27.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:27.794 7f5e8329a700 1 --2- 172.21.2.112:0/1328433585 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5e7c0d1c70 0x7f5e7c0cf950 unknown :-1 s=CLOSED pgs=3303 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:27.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:27.794 7f5e8329a700 1 --2- 172.21.2.112:0/1328433585 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5e7c1b0a10 0x7f5e7c0cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:27.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:27.794 7f5e8329a700 1 --2- 172.21.2.112:0/1328433585 >> v2:172.21.2.120:3300/0 conn(0x7f5e7c0d1870 0x7f5e7c1b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:27.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:27.794 7f5e8329a700 1 -- 172.21.2.112:0/1328433585 >> 172.21.2.112:0/1328433585 conn(0x7f5e7c0ce450 msgr2=0x7f5e7c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:27.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:27.794 7f5e8329a700 1 -- 172.21.2.112:0/1328433585 shutdown_connections 2021-02-10T11:49:27.815 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:27.794 7f5e8329a700 1 -- 172.21.2.112:0/1328433585 wait complete. 2021-02-10T11:49:27.815 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:27.808 7f5e8329a700 1 Processor -- start 2021-02-10T11:49:27.815 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:27.808 7f5e8329a700 1 -- start start 2021-02-10T11:49:27.816 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:27.809 7f5e8329a700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5e7c1a06a0 0x7f5e7c09f380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:27.816 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:27.809 7f5e8329a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5e7c09f9a0 0x7f5e7c08b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:27.816 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:27.809 7f5e8329a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5e7c08bb80 0x7f5e7c08bf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:27.816 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:27.809 7f5e8329a700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e7c08c780 con 0x7f5e7c08bb80 2021-02-10T11:49:27.816 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:27.809 7f5e8329a700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e7c09fda0 con 0x7f5e7c1a06a0 2021-02-10T11:49:27.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:27.809 7f5e8329a700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e7c08cd00 con 0x7f5e7c09f9a0 2021-02-10T11:49:27.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:27.809 7f5e818b7700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5e7c08bb80 0x7f5e7c08bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:27.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:27.809 7f5e818b7700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5e7c08bb80 0x7f5e7c08bf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:33848/0 (socket says 172.21.2.112:33848) 2021-02-10T11:49:27.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:27.809 7f5e818b7700 1 -- 172.21.2.112:0/3225859515 learned_addr learned my addr 172.21.2.112:0/3225859515 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:49:27.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:27.809 7f5e818b7700 1 -- 172.21.2.112:0/3225859515 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5e7c09f9a0 msgr2=0x7f5e7c08b560 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:49:27.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:27.809 7f5e818b7700 1 --2- 172.21.2.112:0/3225859515 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5e7c09f9a0 0x7f5e7c08b560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:27.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:27.809 7f5e818b7700 1 -- 172.21.2.112:0/3225859515 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5e7c1a06a0 msgr2=0x7f5e7c09f380 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:49:27.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:27.809 7f5e818b7700 1 --2- 172.21.2.112:0/3225859515 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5e7c1a06a0 0x7f5e7c09f380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:27.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:27.809 7f5e818b7700 1 -- 172.21.2.112:0/3225859515 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e7c08cf20 con 0x7f5e7c08bb80 2021-02-10T11:49:27.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:27.809 7f5e818b7700 1 --2- 172.21.2.112:0/3225859515 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5e7c08bb80 0x7f5e7c08bf80 secure :-1 s=READY pgs=3304 cs=0 l=1 rev1=1 rx=0x7f5e74009970 tx=0x7f5e7400bcc0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:27.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:27.809 7f5e6dffb700 1 -- 172.21.2.112:0/3225859515 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5e74009450 con 0x7f5e7c08bb80 2021-02-10T11:49:27.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:27.810 7f5e6dffb700 1 -- 172.21.2.112:0/3225859515 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e74000cc0 con 0x7f5e7c08bb80 2021-02-10T11:49:27.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:27.810 7f5e8329a700 1 -- 172.21.2.112:0/3225859515 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e7c08cf20 con 0x7f5e7c08bb80 2021-02-10T11:49:27.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:27.810 7f5e8329a700 1 -- 172.21.2.112:0/3225859515 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e7c08cf20 con 0x7f5e7c08bb80 2021-02-10T11:49:27.818 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:27.810 7f5e6dffb700 1 -- 172.21.2.112:0/3225859515 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5e7400c4e0 con 0x7f5e7c08bb80 2021-02-10T11:49:27.818 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:27.810 7f5e6dffb700 1 -- 172.21.2.112:0/3225859515 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5e7400c2f0 con 0x7f5e7c08bb80 2021-02-10T11:49:27.818 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:27.810 7f5e6dffb700 1 --2- 172.21.2.112:0/3225859515 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5e5c018730 0x7f5e5c01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:27.818 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:27.810 7f5e6dffb700 1 -- 172.21.2.112:0/3225859515 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f5e740329a0 con 0x7f5e7c08bb80 2021-02-10T11:49:27.818 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:27.811 7f5e810b6700 1 --2- 172.21.2.112:0/3225859515 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5e5c018730 0x7f5e5c01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:27.818 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:27.811 7f5e810b6700 1 --2- 172.21.2.112:0/3225859515 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5e5c018730 0x7f5e5c01ac10 crc :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:27.846 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:27.842 7f5e8329a700 1 -- 172.21.2.112:0/3225859515 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e7c0ebc50 con 0x7f5e7c08bb80 2021-02-10T11:49:27.848 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:27.845 7f5e6dffb700 1 -- 172.21.2.112:0/3225859515 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5e74009450 con 0x7f5e7c08bb80 2021-02-10T11:49:27.982 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:27.979 7f5e8329a700 1 -- 172.21.2.112:0/3225859515 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs volume ls", "target": ["mgr", ""]}) v1 -- 0x7f5e7c0ebc50 con 0x7f5e5c018730 2021-02-10T11:49:27.983 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:27.980 7f5e6dffb700 1 -- 172.21.2.112:0/3225859515 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+41 (crc 0 0 0) 0x7f5e70000940 con 0x7f5e5c018730 2021-02-10T11:49:27.986 INFO:teuthology.orchestra.run.gibba012.stdout:[ 2021-02-10T11:49:27.986 INFO:teuthology.orchestra.run.gibba012.stdout: { 2021-02-10T11:49:27.986 INFO:teuthology.orchestra.run.gibba012.stdout: "name": "cephfs" 2021-02-10T11:49:27.986 INFO:teuthology.orchestra.run.gibba012.stdout: } 2021-02-10T11:49:27.986 INFO:teuthology.orchestra.run.gibba012.stdout:] 2021-02-10T11:49:27.987 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:27.983 7f5e8329a700 1 -- 172.21.2.112:0/3225859515 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5e5c018730 msgr2=0x7f5e5c01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:27.987 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:27.983 7f5e8329a700 1 --2- 172.21.2.112:0/3225859515 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5e5c018730 0x7f5e5c01ac10 crc :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:27.987 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:27.983 7f5e8329a700 1 -- 172.21.2.112:0/3225859515 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5e7c08bb80 msgr2=0x7f5e7c08bf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:27.987 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:27.983 7f5e8329a700 1 --2- 172.21.2.112:0/3225859515 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5e7c08bb80 0x7f5e7c08bf80 secure :-1 s=READY pgs=3304 cs=0 l=1 rev1=1 rx=0x7f5e74009970 tx=0x7f5e7400bcc0).stop 2021-02-10T11:49:27.988 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:27.984 7f5e8329a700 1 -- 172.21.2.112:0/3225859515 shutdown_connections 2021-02-10T11:49:27.988 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:27.984 7f5e8329a700 1 --2- 172.21.2.112:0/3225859515 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f5e5c018730 0x7f5e5c01ac10 unknown :-1 s=CLOSED pgs=1487 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:27.988 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:27.984 7f5e8329a700 1 --2- 172.21.2.112:0/3225859515 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f5e7c08bb80 0x7f5e7c08bf80 unknown :-1 s=CLOSED pgs=3304 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:27.988 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:27.984 7f5e8329a700 1 --2- 172.21.2.112:0/3225859515 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5e7c09f9a0 0x7f5e7c08b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:27.988 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:27.984 7f5e8329a700 1 --2- 172.21.2.112:0/3225859515 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f5e7c1a06a0 0x7f5e7c09f380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:27.988 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:27.984 7f5e8329a700 1 -- 172.21.2.112:0/3225859515 >> 172.21.2.112:0/3225859515 conn(0x7f5e7c0b83b0 msgr2=0x7f5e7c0a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:27.988 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:27.984 7f5e8329a700 1 -- 172.21.2.112:0/3225859515 shutdown_connections 2021-02-10T11:49:27.988 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:27.984 7f5e8329a700 1 -- 172.21.2.112:0/3225859515 wait complete. 2021-02-10T11:49:27.995 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:49:28.430 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.410 7fb2ecc1a700 1 Processor -- start 2021-02-10T11:49:28.431 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.410 7fb2ecc1a700 1 -- start start 2021-02-10T11:49:28.431 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.411 7fb2ecc1a700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fb2e810aff0 0x7fb2e80cf750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:28.431 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.411 7fb2ecc1a700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2e80fe1c0 con 0x7fb2e80d1870 2021-02-10T11:49:28.431 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.411 7fb2ecc1a700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2e81b0cb0 con 0x7fb2e80d1c70 2021-02-10T11:49:28.431 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.411 7fb2ecc1a700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2e8109a60 con 0x7fb2e810aff0 2021-02-10T11:49:28.431 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.411 7fb2e685e700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7fb2e80d1870 0x7fb2e81b04c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:33314/0 (socket says 172.21.2.112:33314) 2021-02-10T11:49:28.431 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.411 7fb2e705f700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fb2e810aff0 0x7fb2e80cf750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:28.431 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.411 7fb2e685e700 1 -- 172.21.2.112:0/3355916688 learned_addr learned my addr 172.21.2.112:0/3355916688 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:49:28.431 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.411 7fb2e705f700 1 -- 172.21.2.112:0/3355916688 >> v1:172.21.2.120:6789/0 conn(0x7fb2e80d1c70 legacy=0x7fb2e80cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:28.431 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.411 7fb2e705f700 1 -- 172.21.2.112:0/3355916688 >> v1:172.21.2.112:6789/0 conn(0x7fb2e80d1870 legacy=0x7fb2e81b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:28.431 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.411 7fb2e705f700 1 -- 172.21.2.112:0/3355916688 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2e810b3f0 con 0x7fb2e810aff0 2021-02-10T11:49:28.431 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.411 7fb2e705f700 1 --2- 172.21.2.112:0/3355916688 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb2e810aff0 0x7fb2e80cf750 secure :-1 s=READY pgs=3306 cs=0 l=1 rev1=1 rx=0x7fb2e0006e30 tx=0x7fb2e0009e60).ready entity=mon.0 client_cookie=728873a56ea0dc11 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:28.432 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.412 7fb2e585c700 1 -- 172.21.2.112:0/3355916688 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb2e0006a50 con 0x7fb2e810aff0 2021-02-10T11:49:28.432 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.412 7fb2e585c700 1 -- 172.21.2.112:0/3355916688 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2e0005820 con 0x7fb2e810aff0 2021-02-10T11:49:28.432 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.412 7fb2e585c700 1 -- 172.21.2.112:0/3355916688 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb2e0005d00 con 0x7fb2e810aff0 2021-02-10T11:49:28.432 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.412 7fb2ecc1a700 1 -- 172.21.2.112:0/3355916688 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb2e810aff0 msgr2=0x7fb2e80cf750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:28.432 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.412 7fb2ecc1a700 1 --2- 172.21.2.112:0/3355916688 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb2e810aff0 0x7fb2e80cf750 secure :-1 s=READY pgs=3306 cs=0 l=1 rev1=1 rx=0x7fb2e0006e30 tx=0x7fb2e0009e60).stop 2021-02-10T11:49:28.432 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.412 7fb2ecc1a700 1 -- 172.21.2.112:0/3355916688 shutdown_connections 2021-02-10T11:49:28.432 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.412 7fb2ecc1a700 1 --2- 172.21.2.112:0/3355916688 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb2e810aff0 0x7fb2e80cf750 unknown :-1 s=CLOSED pgs=3306 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:28.432 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.412 7fb2ecc1a700 1 -- 172.21.2.112:0/3355916688 >> 172.21.2.112:0/3355916688 conn(0x7fb2e80ce420 msgr2=0x7fb2e80d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:28.432 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.413 7fb2ecc1a700 1 -- 172.21.2.112:0/3355916688 shutdown_connections 2021-02-10T11:49:28.432 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.413 7fb2ecc1a700 1 -- 172.21.2.112:0/3355916688 wait complete. 2021-02-10T11:49:28.435 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.427 7fb2ecc1a700 1 Processor -- start 2021-02-10T11:49:28.435 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.427 7fb2ecc1a700 1 -- start start 2021-02-10T11:49:28.435 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.427 7fb2ecc1a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb2e80a25d0 0x7fb2e80a29d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:28.435 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.427 7fb2ecc1a700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb2e80a2ff0 0x7fb2e80a33f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:28.435 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.427 7fb2ecc1a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb2e80b5650 0x7fb2e80b5a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:28.435 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.427 7fb2ecc1a700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2e81c3780 con 0x7fb2e80b5650 2021-02-10T11:49:28.435 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.427 7fb2ecc1a700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2e80a20c0 con 0x7fb2e80a2ff0 2021-02-10T11:49:28.435 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.427 7fb2ecc1a700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2e80b60a0 con 0x7fb2e80a25d0 2021-02-10T11:49:28.435 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.428 7fb2e705f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb2e80b5650 0x7fb2e80b5a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:28.436 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.428 7fb2e705f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb2e80b5650 0x7fb2e80b5a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:33858/0 (socket says 172.21.2.112:33858) 2021-02-10T11:49:28.436 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.428 7fb2e705f700 1 -- 172.21.2.112:0/587352916 learned_addr learned my addr 172.21.2.112:0/587352916 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:49:28.436 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.428 7fb2e605d700 1 --2- 172.21.2.112:0/587352916 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb2e80a2ff0 0x7fb2e80a33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:28.436 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.428 7fb2e685e700 1 --2- 172.21.2.112:0/587352916 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb2e80a25d0 0x7fb2e80a29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:28.436 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.428 7fb2e705f700 1 -- 172.21.2.112:0/587352916 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb2e80a25d0 msgr2=0x7fb2e80a29d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:28.436 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.428 7fb2e705f700 1 --2- 172.21.2.112:0/587352916 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb2e80a25d0 0x7fb2e80a29d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:28.436 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.428 7fb2e705f700 1 -- 172.21.2.112:0/587352916 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb2e80a2ff0 msgr2=0x7fb2e80a33f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:28.436 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.428 7fb2e705f700 1 --2- 172.21.2.112:0/587352916 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb2e80a2ff0 0x7fb2e80a33f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:28.436 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.428 7fb2e705f700 1 -- 172.21.2.112:0/587352916 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2e80b62c0 con 0x7fb2e80b5650 2021-02-10T11:49:28.436 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.428 7fb2e705f700 1 --2- 172.21.2.112:0/587352916 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb2e80b5650 0x7fb2e80b5a50 secure :-1 s=READY pgs=3307 cs=0 l=1 rev1=1 rx=0x7fb2e000b970 tx=0x7fb2e000bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:28.436 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.428 7fb2df7fe700 1 -- 172.21.2.112:0/587352916 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb2e00093d0 con 0x7fb2e80b5650 2021-02-10T11:49:28.436 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.428 7fb2df7fe700 1 -- 172.21.2.112:0/587352916 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2e0000fa0 con 0x7fb2e80b5650 2021-02-10T11:49:28.437 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.428 7fb2ecc1a700 1 -- 172.21.2.112:0/587352916 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2e80b62c0 con 0x7fb2e80b5650 2021-02-10T11:49:28.437 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.428 7fb2ecc1a700 1 -- 172.21.2.112:0/587352916 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2e80b6850 con 0x7fb2e80b5650 2021-02-10T11:49:28.437 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.428 7fb2df7fe700 1 -- 172.21.2.112:0/587352916 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb2e0005b60 con 0x7fb2e80b5650 2021-02-10T11:49:28.437 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.429 7fb2df7fe700 1 -- 172.21.2.112:0/587352916 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb2e0005820 con 0x7fb2e80b5650 2021-02-10T11:49:28.437 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.429 7fb2df7fe700 1 --2- 172.21.2.112:0/587352916 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb2cc018730 0x7fb2cc01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:28.437 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.429 7fb2df7fe700 1 -- 172.21.2.112:0/587352916 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fb2e0032b30 con 0x7fb2e80b5650 2021-02-10T11:49:28.437 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.429 7fb2e685e700 1 --2- 172.21.2.112:0/587352916 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb2cc018730 0x7fb2cc01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:28.437 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.430 7fb2e685e700 1 --2- 172.21.2.112:0/587352916 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb2cc018730 0x7fb2cc01ac10 crc :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:28.464 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.461 7fb2ecc1a700 1 -- 172.21.2.112:0/587352916 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2e80807d0 con 0x7fb2e80b5650 2021-02-10T11:49:28.467 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.463 7fb2df7fe700 1 -- 172.21.2.112:0/587352916 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb2e00093d0 con 0x7fb2e80b5650 2021-02-10T11:49:28.566 DEBUG:teuthology.orchestra.run.gibba012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:49:28.568 DEBUG:teuthology.orchestra.run.gibba020:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:49:28.623 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.619 7fb2ecc1a700 1 -- 172.21.2.112:0/587352916 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fb2e8088070 con 0x7fb2e80b5650 2021-02-10T11:49:28.624 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.620 7fb2df7fe700 1 -- 172.21.2.112:0/587352916 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v48)=0 v48) v1 ==== 122+0+0 (secure 0 0 0) 0x7fb2e0005820 con 0x7fb2e80b5650 2021-02-10T11:49:28.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.622 7fb2ecc1a700 1 -- 172.21.2.112:0/587352916 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb2cc018730 msgr2=0x7fb2cc01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:28.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.622 7fb2ecc1a700 1 --2- 172.21.2.112:0/587352916 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb2cc018730 0x7fb2cc01ac10 crc :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:28.626 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.622 7fb2ecc1a700 1 -- 172.21.2.112:0/587352916 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb2e80b5650 msgr2=0x7fb2e80b5a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:28.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.622 7fb2ecc1a700 1 --2- 172.21.2.112:0/587352916 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb2e80b5650 0x7fb2e80b5a50 secure :-1 s=READY pgs=3307 cs=0 l=1 rev1=1 rx=0x7fb2e000b970 tx=0x7fb2e000bd10).stop 2021-02-10T11:49:28.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.622 7fb2ecc1a700 1 -- 172.21.2.112:0/587352916 shutdown_connections 2021-02-10T11:49:28.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.622 7fb2ecc1a700 1 --2- 172.21.2.112:0/587352916 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fb2cc018730 0x7fb2cc01ac10 unknown :-1 s=CLOSED pgs=1488 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:28.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.622 7fb2ecc1a700 1 --2- 172.21.2.112:0/587352916 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fb2e80a2ff0 0x7fb2e80a33f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:28.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.622 7fb2ecc1a700 1 --2- 172.21.2.112:0/587352916 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fb2e80b5650 0x7fb2e80b5a50 unknown :-1 s=CLOSED pgs=3307 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:28.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.622 7fb2ecc1a700 1 --2- 172.21.2.112:0/587352916 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fb2e80a25d0 0x7fb2e80a29d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:28.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.622 7fb2ecc1a700 1 -- 172.21.2.112:0/587352916 >> 172.21.2.112:0/587352916 conn(0x7fb2e80c4120 msgr2=0x7fb2e80ffd60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:28.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.623 7fb2ecc1a700 1 -- 172.21.2.112:0/587352916 shutdown_connections 2021-02-10T11:49:28.627 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.623 7fb2ecc1a700 1 -- 172.21.2.112:0/587352916 wait complete. 2021-02-10T11:49:28.634 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000215179 2021-02-10T11:49:29.001 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.981 7f0e04cc7700 1 Processor -- start 2021-02-10T11:49:29.001 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.981 7f0e04cc7700 1 -- start start 2021-02-10T11:49:29.001 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.981 7f0e04cc7700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0e000d1870 0x7f0e001b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:29.001 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.981 7f0e04cc7700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e00109a60 con 0x7f0e000cff70 2021-02-10T11:49:29.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.981 7f0e04cc7700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e00088070 con 0x7f0e001b0ae0 2021-02-10T11:49:29.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.981 7f0e04cc7700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e000e5ef0 con 0x7f0e000d1870 2021-02-10T11:49:29.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.981 7f0dff05f700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f0e000cff70 0x7f0e00109130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:33328/0 (socket says 172.21.2.112:33328) 2021-02-10T11:49:29.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.981 7f0dff05f700 1 -- 172.21.2.112:0/965037430 learned_addr learned my addr 172.21.2.112:0/965037430 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:49:29.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.982 7f0dfe85e700 1 --2- 172.21.2.112:0/965037430 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0e000d1870 0x7f0e001b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:29.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.982 7f0dfe05d700 1 -- 172.21.2.112:0/965037430 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3450661677 0 0) 0x7f0df8001500 con 0x7f0e000cff70 2021-02-10T11:49:29.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.982 7f0dfe05d700 1 -- 172.21.2.112:0/965037430 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0dec002fe0 con 0x7f0e000cff70 2021-02-10T11:49:29.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.982 7f0dfe05d700 1 -- 172.21.2.112:0/965037430 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3323369186 0 0) 0x7f0de8000ff0 con 0x7f0e001b0ae0 2021-02-10T11:49:29.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.982 7f0dfe05d700 1 -- 172.21.2.112:0/965037430 --> v1:172.21.2.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0dec0042c0 con 0x7f0e001b0ae0 2021-02-10T11:49:29.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.982 7f0dfe05d700 1 -- 172.21.2.112:0/965037430 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1406404626 0 0) 0x7f0df8001460 con 0x7f0e000cff70 2021-02-10T11:49:29.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.982 7f0dfe85e700 1 -- 172.21.2.112:0/965037430 >> v1:172.21.2.120:6789/0 conn(0x7f0e001b0ae0 legacy=0x7f0e00108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:29.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.982 7f0dfe85e700 1 -- 172.21.2.112:0/965037430 >> v1:172.21.2.112:6789/0 conn(0x7f0e000cff70 legacy=0x7f0e00109130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:29.002 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.982 7f0dfe85e700 1 -- 172.21.2.112:0/965037430 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e000d1c70 con 0x7f0e000d1870 2021-02-10T11:49:29.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.982 7f0dfe85e700 1 --2- 172.21.2.112:0/965037430 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0e000d1870 0x7f0e001b04c0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7f0df0006650 tx=0x7f0df0009a80).ready entity=mon.2 client_cookie=7c7856750f08e663 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:29.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.983 7f0dfe05d700 1 -- 172.21.2.112:0/965037430 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0df000c0c0 con 0x7f0e000d1870 2021-02-10T11:49:29.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.983 7f0dfe05d700 1 -- 172.21.2.112:0/965037430 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0df000c340 con 0x7f0e000d1870 2021-02-10T11:49:29.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.983 7f0dfe05d700 1 -- 172.21.2.112:0/965037430 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0df000c860 con 0x7f0e000d1870 2021-02-10T11:49:29.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.983 7f0e04cc7700 1 -- 172.21.2.112:0/965037430 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0e000d1870 msgr2=0x7f0e001b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:29.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.983 7f0e04cc7700 1 --2- 172.21.2.112:0/965037430 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0e000d1870 0x7f0e001b04c0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7f0df0006650 tx=0x7f0df0009a80).stop 2021-02-10T11:49:29.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.983 7f0e04cc7700 1 -- 172.21.2.112:0/965037430 shutdown_connections 2021-02-10T11:49:29.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.983 7f0e04cc7700 1 --2- 172.21.2.112:0/965037430 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0e000d1870 0x7f0e001b04c0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:29.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.983 7f0e04cc7700 1 -- 172.21.2.112:0/965037430 >> 172.21.2.112:0/965037430 conn(0x7f0e000ce420 msgr2=0x7f0e000d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:29.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.983 7f0e04cc7700 1 -- 172.21.2.112:0/965037430 shutdown_connections 2021-02-10T11:49:29.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.983 7f0e04cc7700 1 -- 172.21.2.112:0/965037430 wait complete. 2021-02-10T11:49:29.005 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.998 7f0e04cc7700 1 Processor -- start 2021-02-10T11:49:29.005 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.998 7f0e04cc7700 1 -- start start 2021-02-10T11:49:29.005 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.998 7f0e04cc7700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0e000c1850 0x7f0e000c1f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:29.005 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.998 7f0e04cc7700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0e000c2810 0x7f0e000c2c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:29.005 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.998 7f0e04cc7700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0e000c3230 0x7f0e000c3630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:29.005 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.998 7f0e04cc7700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e000c1c50 con 0x7f0e000c3230 2021-02-10T11:49:29.005 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.998 7f0e04cc7700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e00084920 con 0x7f0e000c2810 2021-02-10T11:49:29.005 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.998 7f0e04cc7700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e00084b40 con 0x7f0e000c1850 2021-02-10T11:49:29.005 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.998 7f0df605d700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0e000c2810 0x7f0e000c2c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:29.005 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.998 7f0dff05f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0e000c3230 0x7f0e000c3630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:29.005 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.998 7f0dff05f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0e000c3230 0x7f0e000c3630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:33874/0 (socket says 172.21.2.112:33874) 2021-02-10T11:49:29.006 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.998 7f0dff05f700 1 -- 172.21.2.112:0/1008912524 learned_addr learned my addr 172.21.2.112:0/1008912524 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:49:29.006 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.999 7f0dfe85e700 1 --2- 172.21.2.112:0/1008912524 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0e000c1850 0x7f0e000c1f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:29.006 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.999 7f0dff05f700 1 -- 172.21.2.112:0/1008912524 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0e000c1850 msgr2=0x7f0e000c1f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:29.006 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.999 7f0dff05f700 1 --2- 172.21.2.112:0/1008912524 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0e000c1850 0x7f0e000c1f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:29.006 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.999 7f0dff05f700 1 -- 172.21.2.112:0/1008912524 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0e000c2810 msgr2=0x7f0e000c2c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:29.006 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.999 7f0dff05f700 1 --2- 172.21.2.112:0/1008912524 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0e000c2810 0x7f0e000c2c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:29.006 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.999 7f0dff05f700 1 -- 172.21.2.112:0/1008912524 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e00084d60 con 0x7f0e000c3230 2021-02-10T11:49:29.006 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.999 7f0dff05f700 1 --2- 172.21.2.112:0/1008912524 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0e000c3230 0x7f0e000c3630 secure :-1 s=READY pgs=3309 cs=0 l=1 rev1=1 rx=0x7f0df8009960 tx=0x7f0df800bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:29.006 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.999 7f0df7fff700 1 -- 172.21.2.112:0/1008912524 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0df80060c0 con 0x7f0e000c3230 2021-02-10T11:49:29.006 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.999 7f0df7fff700 1 -- 172.21.2.112:0/1008912524 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0df8006400 con 0x7f0e000c3230 2021-02-10T11:49:29.006 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.999 7f0df7fff700 1 -- 172.21.2.112:0/1008912524 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0df8004720 con 0x7f0e000c3230 2021-02-10T11:49:29.006 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.999 7f0e04cc7700 1 -- 172.21.2.112:0/1008912524 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e00084d60 con 0x7f0e000c3230 2021-02-10T11:49:29.006 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:28.999 7f0e04cc7700 1 -- 172.21.2.112:0/1008912524 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e000852f0 con 0x7f0e000c3230 2021-02-10T11:49:29.007 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:29.000 7f0df7fff700 1 -- 172.21.2.112:0/1008912524 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0df8000d30 con 0x7f0e000c3230 2021-02-10T11:49:29.007 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:29.000 7f0df7fff700 1 --2- 172.21.2.112:0/1008912524 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0de40186c0 0x7f0de401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:29.007 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:29.000 7f0df7fff700 1 -- 172.21.2.112:0/1008912524 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f0df8032940 con 0x7f0e000c3230 2021-02-10T11:49:29.007 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:29.000 7f0dfe85e700 1 --2- 172.21.2.112:0/1008912524 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0de40186c0 0x7f0de401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:29.007 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:29.001 7f0dfe85e700 1 --2- 172.21.2.112:0/1008912524 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0de40186c0 0x7f0de401abc0 crc :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:29.035 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:29.031 7f0e04cc7700 1 -- 172.21.2.112:0/1008912524 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e00109a60 con 0x7f0e000c3230 2021-02-10T11:49:29.037 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:29.034 7f0df7fff700 1 -- 172.21.2.112:0/1008912524 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0df8006300 con 0x7f0e000c3230 2021-02-10T11:49:29.163 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:29.160 7f0e04cc7700 1 -- 172.21.2.112:0/1008912524 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000215179", "target": ["mgr", ""]}) v1 -- 0x7f0e000852f0 con 0x7f0de40186c0 2021-02-10T11:49:29.168 INFO:tasks.ceph.mgr.x.gibba020.stderr:2021-02-10 11:49:29.164 7f6382eee700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:49:29.168 INFO:tasks.ceph.mgr.x.gibba020.stderr:2021-02-10 11:49:29.164 7f6382eee700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:49:29.168 INFO:tasks.ceph.mgr.x.gibba020.stderr:2021-02-10 11:49:29.164 7f6382eee700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:49:29.168 INFO:tasks.ceph.mgr.x.gibba020.stderr:2021-02-10 11:49:29.164 7f6382eee700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:49:29.168 INFO:tasks.ceph.mgr.x.gibba020.stderr:2021-02-10 11:49:29.164 7f6382eee700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:49:29.198 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:29.194 7f0df7fff700 1 -- 172.21.2.112:0/1008912524 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f0df0006420 con 0x7f0de40186c0 2021-02-10T11:49:29.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:29.225 7f0e04cc7700 1 -- 172.21.2.112:0/1008912524 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0de40186c0 msgr2=0x7f0de401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:29.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:29.225 7f0e04cc7700 1 --2- 172.21.2.112:0/1008912524 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0de40186c0 0x7f0de401abc0 crc :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:29.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:29.225 7f0e04cc7700 1 -- 172.21.2.112:0/1008912524 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0e000c3230 msgr2=0x7f0e000c3630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:29.229 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:29.225 7f0e04cc7700 1 --2- 172.21.2.112:0/1008912524 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0e000c3230 0x7f0e000c3630 secure :-1 s=READY pgs=3309 cs=0 l=1 rev1=1 rx=0x7f0df8009960 tx=0x7f0df800bcf0).stop 2021-02-10T11:49:29.230 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:29.226 7f0e04cc7700 1 -- 172.21.2.112:0/1008912524 shutdown_connections 2021-02-10T11:49:29.230 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:29.226 7f0e04cc7700 1 --2- 172.21.2.112:0/1008912524 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f0de40186c0 0x7f0de401abc0 unknown :-1 s=CLOSED pgs=1489 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:29.230 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:29.226 7f0e04cc7700 1 --2- 172.21.2.112:0/1008912524 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0e000c2810 0x7f0e000c2c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:29.230 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:29.226 7f0e04cc7700 1 --2- 172.21.2.112:0/1008912524 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0e000c3230 0x7f0e000c3630 unknown :-1 s=CLOSED pgs=3309 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:29.231 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:29.226 7f0e04cc7700 1 --2- 172.21.2.112:0/1008912524 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0e000c1850 0x7f0e000c1f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:29.231 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:29.226 7f0e04cc7700 1 -- 172.21.2.112:0/1008912524 >> 172.21.2.112:0/1008912524 conn(0x7f0e000af690 msgr2=0x7f0e00094c40 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:29.231 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:29.226 7f0e04cc7700 1 -- 172.21.2.112:0/1008912524 shutdown_connections 2021-02-10T11:49:29.231 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:29.226 7f0e04cc7700 1 -- 172.21.2.112:0/1008912524 wait complete. 2021-02-10T11:49:29.237 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000215179 2021-02-10T11:49:29.670 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:29.650 7f6b97e4e700 1 Processor -- start 2021-02-10T11:49:29.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:29.650 7f6b97e4e700 1 -- start start 2021-02-10T11:49:29.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:29.650 7f6b97e4e700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f6b900d1c70 0x7f6b900cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:29.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:29.650 7f6b97e4e700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f6b900cf580 0x7f6b900cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:29.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:29.650 7f6b97e4e700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b900fe1c0 con 0x7f6b900d1870 2021-02-10T11:49:29.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:29.650 7f6b97e4e700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b901b0cb0 con 0x7f6b900cf580 2021-02-10T11:49:29.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:29.650 7f6b97e4e700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b90109a60 con 0x7f6b900d1c70 2021-02-10T11:49:29.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:29.650 7f6b95c6a700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f6b900d1870 0x7f6b901b04c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:33340/0 (socket says 172.21.2.112:33340) 2021-02-10T11:49:29.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:29.650 7f6b9646b700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f6b900cf580 0x7f6b900cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:29.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:29.650 7f6b95c6a700 1 -- 172.21.2.112:0/3499085421 learned_addr learned my addr 172.21.2.112:0/3499085421 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:49:29.671 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:29.651 7f6b95469700 1 --2- 172.21.2.112:0/3499085421 >> v2:172.21.2.120:3300/0 conn(0x7f6b900d1c70 0x7f6b900cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:29.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:29.651 7f6b9646b700 1 -- 172.21.2.112:0/3499085421 >> v2:172.21.2.120:3300/0 conn(0x7f6b900d1c70 msgr2=0x7f6b900cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:49:29.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:29.651 7f6b9646b700 1 --2- 172.21.2.112:0/3499085421 >> v2:172.21.2.120:3300/0 conn(0x7f6b900d1c70 0x7f6b900cef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:29.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:29.651 7f6b9646b700 1 -- 172.21.2.112:0/3499085421 >> v1:172.21.2.112:6789/0 conn(0x7f6b900d1870 legacy=0x7f6b901b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:29.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:29.651 7f6b9646b700 1 -- 172.21.2.112:0/3499085421 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b90088070 con 0x7f6b900cf580 2021-02-10T11:49:29.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:29.651 7f6b95469700 1 --2- 172.21.2.112:0/3499085421 >> v2:172.21.2.120:3300/0 conn(0x7f6b900d1c70 0x7f6b900cef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:49:29.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:29.651 7f6b9646b700 1 --2- 172.21.2.112:0/3499085421 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6b900cf580 0x7f6b900cf980 secure :-1 s=READY pgs=3311 cs=0 l=1 rev1=1 rx=0x7f6b88006e30 tx=0x7f6b88009e60).ready entity=mon.0 client_cookie=774d8dbd2bf4b3e6 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:29.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:29.651 7f6b94c68700 1 -- 172.21.2.112:0/3499085421 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6b88006a50 con 0x7f6b900cf580 2021-02-10T11:49:29.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:29.651 7f6b94c68700 1 -- 172.21.2.112:0/3499085421 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b8800b300 con 0x7f6b900cf580 2021-02-10T11:49:29.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:29.651 7f6b94c68700 1 -- 172.21.2.112:0/3499085421 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6b8800b750 con 0x7f6b900cf580 2021-02-10T11:49:29.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:29.652 7f6b97e4e700 1 -- 172.21.2.112:0/3499085421 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6b900cf580 msgr2=0x7f6b900cf980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:29.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:29.652 7f6b97e4e700 1 --2- 172.21.2.112:0/3499085421 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6b900cf580 0x7f6b900cf980 secure :-1 s=READY pgs=3311 cs=0 l=1 rev1=1 rx=0x7f6b88006e30 tx=0x7f6b88009e60).stop 2021-02-10T11:49:29.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:29.652 7f6b97e4e700 1 -- 172.21.2.112:0/3499085421 shutdown_connections 2021-02-10T11:49:29.672 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:29.652 7f6b97e4e700 1 --2- 172.21.2.112:0/3499085421 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6b900cf580 0x7f6b900cf980 unknown :-1 s=CLOSED pgs=3311 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:29.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:29.652 7f6b97e4e700 1 --2- 172.21.2.112:0/3499085421 >> v2:172.21.2.120:3300/0 conn(0x7f6b900d1c70 0x7f6b900cef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:29.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:29.652 7f6b97e4e700 1 -- 172.21.2.112:0/3499085421 >> 172.21.2.112:0/3499085421 conn(0x7f6b900ce420 msgr2=0x7f6b900d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:29.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:29.652 7f6b97e4e700 1 -- 172.21.2.112:0/3499085421 shutdown_connections 2021-02-10T11:49:29.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:29.652 7f6b97e4e700 1 -- 172.21.2.112:0/3499085421 wait complete. 2021-02-10T11:49:29.674 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:29.667 7f6b97e4e700 1 Processor -- start 2021-02-10T11:49:29.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:29.667 7f6b97e4e700 1 -- start start 2021-02-10T11:49:29.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:29.668 7f6b97e4e700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6b900c3820 0x7f6b90084cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:29.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:29.668 7f6b97e4e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6b900a0290 0x7f6b900852d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:29.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:29.668 7f6b97e4e700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6b90086270 0x7f6b90086670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:29.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:29.668 7f6b97e4e700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b900c0700 con 0x7f6b900a0290 2021-02-10T11:49:29.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:29.668 7f6b97e4e700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b900858f0 con 0x7f6b900c3820 2021-02-10T11:49:29.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:29.668 7f6b97e4e700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b90085b10 con 0x7f6b90086270 2021-02-10T11:49:29.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:29.668 7f6b95469700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6b900a0290 0x7f6b900852d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:29.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:29.668 7f6b95469700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6b900a0290 0x7f6b900852d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:33884/0 (socket says 172.21.2.112:33884) 2021-02-10T11:49:29.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:29.668 7f6b95469700 1 -- 172.21.2.112:0/2886591641 learned_addr learned my addr 172.21.2.112:0/2886591641 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:49:29.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:29.668 7f6b9646b700 1 --2- 172.21.2.112:0/2886591641 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6b90086270 0x7f6b90086670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:29.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:29.668 7f6b95469700 1 -- 172.21.2.112:0/2886591641 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6b90086270 msgr2=0x7f6b90086670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:29.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:29.668 7f6b95469700 1 --2- 172.21.2.112:0/2886591641 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6b90086270 0x7f6b90086670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:29.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:29.668 7f6b95469700 1 -- 172.21.2.112:0/2886591641 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6b900c3820 msgr2=0x7f6b90084cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:29.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:29.668 7f6b95469700 1 --2- 172.21.2.112:0/2886591641 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6b900c3820 0x7f6b90084cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:29.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:29.668 7f6b95469700 1 -- 172.21.2.112:0/2886591641 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b90086c90 con 0x7f6b900a0290 2021-02-10T11:49:29.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:29.668 7f6b9646b700 1 --2- 172.21.2.112:0/2886591641 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6b90086270 0x7f6b90086670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:49:29.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:29.668 7f6b95469700 1 --2- 172.21.2.112:0/2886591641 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6b900a0290 0x7f6b900852d0 secure :-1 s=READY pgs=3312 cs=0 l=1 rev1=1 rx=0x7f6b8c006f20 tx=0x7f6b8c00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:29.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:29.669 7f6b827fc700 1 -- 172.21.2.112:0/2886591641 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6b8c00bd80 con 0x7f6b900a0290 2021-02-10T11:49:29.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:29.669 7f6b827fc700 1 -- 172.21.2.112:0/2886591641 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b8c000c90 con 0x7f6b900a0290 2021-02-10T11:49:29.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:29.669 7f6b827fc700 1 -- 172.21.2.112:0/2886591641 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6b8c0022b0 con 0x7f6b900a0290 2021-02-10T11:49:29.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:29.669 7f6b97e4e700 1 -- 172.21.2.112:0/2886591641 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b90086c90 con 0x7f6b900a0290 2021-02-10T11:49:29.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:29.669 7f6b97e4e700 1 -- 172.21.2.112:0/2886591641 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b90086c90 con 0x7f6b900a0290 2021-02-10T11:49:29.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:29.669 7f6b827fc700 1 -- 172.21.2.112:0/2886591641 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6b8c00a000 con 0x7f6b900a0290 2021-02-10T11:49:29.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:29.669 7f6b827fc700 1 --2- 172.21.2.112:0/2886591641 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6b780186c0 0x7f6b7801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:29.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:29.669 7f6b827fc700 1 -- 172.21.2.112:0/2886591641 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f6b8c0323e0 con 0x7f6b900a0290 2021-02-10T11:49:29.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:29.670 7f6b95c6a700 1 --2- 172.21.2.112:0/2886591641 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6b780186c0 0x7f6b7801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:29.677 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:29.670 7f6b95c6a700 1 --2- 172.21.2.112:0/2886591641 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6b780186c0 0x7f6b7801abc0 crc :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:29.704 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:29.700 7f6b97e4e700 1 -- 172.21.2.112:0/2886591641 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b90082a70 con 0x7f6b900a0290 2021-02-10T11:49:29.706 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:29.703 7f6b827fc700 1 -- 172.21.2.112:0/2886591641 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6b8c006bb0 con 0x7f6b900a0290 2021-02-10T11:49:29.835 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:29.831 7f6b97e4e700 1 -- 172.21.2.112:0/2886591641 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000215179", "target": ["mgr", ""]}) v1 -- 0x7f6b90082a70 con 0x7f6b780186c0 2021-02-10T11:49:29.838 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:29.835 7f6b827fc700 1 -- 172.21.2.112:0/2886591641 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f6b84001530 con 0x7f6b780186c0 2021-02-10T11:49:29.843 INFO:teuthology.orchestra.run.gibba012.stdout:/volumes/_nogroup/subvolume_0000000000215179/8daf014a-0e15-4322-8266-c0f1c83b066e 2021-02-10T11:49:29.845 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:29.841 7f6b97e4e700 1 -- 172.21.2.112:0/2886591641 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6b780186c0 msgr2=0x7f6b7801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:29.845 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:29.841 7f6b97e4e700 1 --2- 172.21.2.112:0/2886591641 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6b780186c0 0x7f6b7801abc0 crc :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:29.845 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:29.841 7f6b97e4e700 1 -- 172.21.2.112:0/2886591641 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6b900a0290 msgr2=0x7f6b900852d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:29.845 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:29.841 7f6b97e4e700 1 --2- 172.21.2.112:0/2886591641 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6b900a0290 0x7f6b900852d0 secure :-1 s=READY pgs=3312 cs=0 l=1 rev1=1 rx=0x7f6b8c006f20 tx=0x7f6b8c00b990).stop 2021-02-10T11:49:29.845 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:29.841 7f6b97e4e700 1 -- 172.21.2.112:0/2886591641 shutdown_connections 2021-02-10T11:49:29.845 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:29.841 7f6b97e4e700 1 --2- 172.21.2.112:0/2886591641 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6b780186c0 0x7f6b7801abc0 unknown :-1 s=CLOSED pgs=1490 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:29.845 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:29.841 7f6b97e4e700 1 --2- 172.21.2.112:0/2886591641 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6b90086270 0x7f6b90086670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:29.845 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:29.841 7f6b97e4e700 1 --2- 172.21.2.112:0/2886591641 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6b900a0290 0x7f6b900852d0 unknown :-1 s=CLOSED pgs=3312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:29.845 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:29.841 7f6b97e4e700 1 --2- 172.21.2.112:0/2886591641 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6b900c3820 0x7f6b90084cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:29.845 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:29.841 7f6b97e4e700 1 -- 172.21.2.112:0/2886591641 >> 172.21.2.112:0/2886591641 conn(0x7f6b900872c0 msgr2=0x7f6b90095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:29.846 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:29.841 7f6b97e4e700 1 -- 172.21.2.112:0/2886591641 shutdown_connections 2021-02-10T11:49:29.846 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:49:29.841 7f6b97e4e700 1 -- 172.21.2.112:0/2886591641 wait complete. 2021-02-10T11:49:29.853 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000215179 with 64 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000215179/8daf014a-0e15-4322-8266-c0f1c83b066e 2021-02-10T11:49:29.853 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:49:29.853 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:49:29.853 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:49:30.188 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:49:30.189 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:49:30.189 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000215179/8daf014a-0e15-4322-8266-c0f1c83b066e/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:49:30.575 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:49:30.576 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:49:30.576 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0297085 s, 35.3 MB/s 2021-02-10T11:49:30.577 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:49:30.577 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:49:30.577 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:49:30.912 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:49:30.912 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:49:30.912 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000215179/8daf014a-0e15-4322-8266-c0f1c83b066e/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:49:31.286 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:49:31.286 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:49:31.286 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.025403 s, 41.3 MB/s 2021-02-10T11:49:31.287 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:49:31.287 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:49:31.287 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:49:31.623 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:49:31.623 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:49:31.623 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000215179/8daf014a-0e15-4322-8266-c0f1c83b066e/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:49:31.999 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:49:32.000 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:49:32.000 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0259991 s, 40.3 MB/s 2021-02-10T11:49:32.000 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:49:32.001 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:49:32.001 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:49:32.336 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:49:32.337 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:49:32.337 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000215179/8daf014a-0e15-4322-8266-c0f1c83b066e/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:49:32.716 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:49:32.716 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:49:32.716 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0269635 s, 38.9 MB/s 2021-02-10T11:49:32.717 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:49:32.717 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:49:32.717 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:49:33.054 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:49:33.054 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:49:33.054 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000215179/8daf014a-0e15-4322-8266-c0f1c83b066e/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:49:33.435 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:49:33.435 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:49:33.435 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.027437 s, 38.2 MB/s 2021-02-10T11:49:33.436 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:49:33.436 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:49:33.436 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:49:33.772 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:49:33.773 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:49:33.773 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000215179/8daf014a-0e15-4322-8266-c0f1c83b066e/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:49:34.147 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:49:34.147 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:49:34.148 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.025696 s, 40.8 MB/s 2021-02-10T11:49:34.148 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:49:34.149 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:49:34.149 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:49:34.484 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:49:34.484 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:49:34.484 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000215179/8daf014a-0e15-4322-8266-c0f1c83b066e/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:49:34.860 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:49:34.860 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:49:34.860 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.023877 s, 43.9 MB/s 2021-02-10T11:49:34.861 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:49:34.861 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:49:34.861 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:49:35.198 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:49:35.198 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:49:35.198 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000215179/8daf014a-0e15-4322-8266-c0f1c83b066e/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:49:35.575 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:49:35.576 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:49:35.576 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.026365 s, 39.8 MB/s 2021-02-10T11:49:35.576 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:49:35.577 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:49:35.577 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:49:35.913 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:49:35.913 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:49:35.914 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000215179/8daf014a-0e15-4322-8266-c0f1c83b066e/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:49:36.319 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:49:36.319 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:49:36.320 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0251094 s, 41.8 MB/s 2021-02-10T11:49:36.320 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:49:36.321 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:49:36.321 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:49:36.679 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:49:36.679 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:49:36.679 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000215179/8daf014a-0e15-4322-8266-c0f1c83b066e/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:49:37.063 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:49:37.064 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:49:37.064 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0232057 s, 45.2 MB/s 2021-02-10T11:49:37.065 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:49:37.065 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:49:37.065 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:49:37.410 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:49:37.412 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:49:37.412 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000215179/8daf014a-0e15-4322-8266-c0f1c83b066e/subvolume_file.10 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:49:37.792 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:49:37.793 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:49:37.793 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0239079 s, 43.9 MB/s 2021-02-10T11:49:37.793 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:49:37.794 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:49:37.794 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:49:38.130 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:49:38.130 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:49:38.130 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000215179/8daf014a-0e15-4322-8266-c0f1c83b066e/subvolume_file.11 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:49:38.504 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:49:38.504 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:49:38.504 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0233601 s, 44.9 MB/s 2021-02-10T11:49:38.505 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:49:38.505 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:49:38.506 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:49:38.842 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:49:38.842 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:49:38.842 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000215179/8daf014a-0e15-4322-8266-c0f1c83b066e/subvolume_file.12 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:49:39.215 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:49:39.215 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:49:39.215 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0247568 s, 42.4 MB/s 2021-02-10T11:49:39.216 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:49:39.216 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:49:39.216 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:49:39.552 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:49:39.552 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:49:39.552 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000215179/8daf014a-0e15-4322-8266-c0f1c83b066e/subvolume_file.13 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:49:39.927 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:49:39.927 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:49:39.927 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0251353 s, 41.7 MB/s 2021-02-10T11:49:39.928 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:49:39.929 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:49:39.929 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:49:40.263 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:49:40.264 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:49:40.264 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000215179/8daf014a-0e15-4322-8266-c0f1c83b066e/subvolume_file.14 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:49:40.637 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:49:40.637 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:49:40.637 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0231887 s, 45.2 MB/s 2021-02-10T11:49:40.638 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:49:40.638 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:49:40.638 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:49:40.973 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:49:40.973 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:49:40.974 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000215179/8daf014a-0e15-4322-8266-c0f1c83b066e/subvolume_file.15 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:49:41.346 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:49:41.347 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:49:41.347 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0224943 s, 46.6 MB/s 2021-02-10T11:49:41.347 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:49:41.348 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:49:41.348 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:49:41.684 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:49:41.684 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:49:41.684 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000215179/8daf014a-0e15-4322-8266-c0f1c83b066e/subvolume_file.16 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:49:42.058 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:49:42.059 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:49:42.059 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0220956 s, 47.5 MB/s 2021-02-10T11:49:42.060 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:49:42.060 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:49:42.060 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:49:42.395 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:49:42.396 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:49:42.396 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000215179/8daf014a-0e15-4322-8266-c0f1c83b066e/subvolume_file.17 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:49:42.775 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:49:42.775 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:49:42.775 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0267518 s, 39.2 MB/s 2021-02-10T11:49:42.776 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:49:42.776 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:49:42.776 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:49:43.111 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:49:43.111 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:49:43.111 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000215179/8daf014a-0e15-4322-8266-c0f1c83b066e/subvolume_file.18 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:49:43.486 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:49:43.486 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:49:43.486 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0230301 s, 45.5 MB/s 2021-02-10T11:49:43.487 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:49:43.487 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:49:43.487 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:49:43.824 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:49:43.824 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:49:43.824 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000215179/8daf014a-0e15-4322-8266-c0f1c83b066e/subvolume_file.19 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:49:44.196 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:49:44.196 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:49:44.196 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0235218 s, 44.6 MB/s 2021-02-10T11:49:44.197 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:49:44.197 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:49:44.198 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:49:44.533 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:49:44.533 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:49:44.533 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000215179/8daf014a-0e15-4322-8266-c0f1c83b066e/subvolume_file.20 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:49:44.907 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:49:44.907 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:49:44.907 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0238304 s, 44.0 MB/s 2021-02-10T11:49:44.908 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:49:44.908 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:49:44.908 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:49:45.243 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:49:45.244 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:49:45.244 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000215179/8daf014a-0e15-4322-8266-c0f1c83b066e/subvolume_file.21 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:49:45.619 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:49:45.619 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:49:45.619 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0244799 s, 42.8 MB/s 2021-02-10T11:49:45.620 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:49:45.620 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:49:45.620 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:49:45.956 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:49:45.956 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:49:45.956 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000215179/8daf014a-0e15-4322-8266-c0f1c83b066e/subvolume_file.22 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:49:46.330 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:49:46.330 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:49:46.330 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0232693 s, 45.1 MB/s 2021-02-10T11:49:46.331 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:49:46.331 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:49:46.332 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:49:46.667 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:49:46.667 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:49:46.667 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000215179/8daf014a-0e15-4322-8266-c0f1c83b066e/subvolume_file.23 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:49:47.041 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:49:47.042 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:49:47.042 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0230127 s, 45.6 MB/s 2021-02-10T11:49:47.043 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:49:47.043 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:49:47.043 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:49:47.378 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:49:47.379 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:49:47.379 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000215179/8daf014a-0e15-4322-8266-c0f1c83b066e/subvolume_file.24 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:49:47.755 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:49:47.755 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:49:47.756 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0251777 s, 41.6 MB/s 2021-02-10T11:49:47.756 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:49:47.757 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:49:47.757 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:49:48.092 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:49:48.092 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:49:48.093 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000215179/8daf014a-0e15-4322-8266-c0f1c83b066e/subvolume_file.25 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:49:48.472 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:49:48.472 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:49:48.473 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0281962 s, 37.2 MB/s 2021-02-10T11:49:48.473 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:49:48.474 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:49:48.474 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:49:48.809 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:49:48.809 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:49:48.809 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000215179/8daf014a-0e15-4322-8266-c0f1c83b066e/subvolume_file.26 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:49:49.183 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:49:49.183 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:49:49.183 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0236879 s, 44.3 MB/s 2021-02-10T11:49:49.184 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:49:49.185 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:49:49.185 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:49:49.521 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:49:49.522 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:49:49.522 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000215179/8daf014a-0e15-4322-8266-c0f1c83b066e/subvolume_file.27 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:49:49.898 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:49:49.898 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:49:49.898 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0248804 s, 42.1 MB/s 2021-02-10T11:49:49.899 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:49:49.899 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:49:49.899 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:49:50.236 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:49:50.236 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:49:50.237 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000215179/8daf014a-0e15-4322-8266-c0f1c83b066e/subvolume_file.28 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:49:50.610 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:49:50.610 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:49:50.610 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0230798 s, 45.4 MB/s 2021-02-10T11:49:50.611 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:49:50.611 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:49:50.612 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:49:50.947 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:49:50.947 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:49:50.947 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000215179/8daf014a-0e15-4322-8266-c0f1c83b066e/subvolume_file.29 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:49:51.322 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:49:51.322 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:49:51.322 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.024217 s, 43.3 MB/s 2021-02-10T11:49:51.323 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:49:51.323 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:49:51.323 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:49:51.658 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:49:51.659 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:49:51.659 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000215179/8daf014a-0e15-4322-8266-c0f1c83b066e/subvolume_file.30 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:49:52.033 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:49:52.034 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:49:52.034 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.023809 s, 44.0 MB/s 2021-02-10T11:49:52.035 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:49:52.035 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:49:52.035 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:49:52.370 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:49:52.371 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:49:52.371 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000215179/8daf014a-0e15-4322-8266-c0f1c83b066e/subvolume_file.31 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:49:52.747 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:49:52.747 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:49:52.747 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0235379 s, 44.5 MB/s 2021-02-10T11:49:52.748 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:49:52.748 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:49:52.748 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:49:53.083 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:49:53.132 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:49:53.132 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000215179/8daf014a-0e15-4322-8266-c0f1c83b066e/subvolume_file.32 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:49:53.512 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:49:53.512 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:49:53.512 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.025459 s, 41.2 MB/s 2021-02-10T11:49:53.514 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:49:53.514 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:49:53.514 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:49:53.849 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:49:53.850 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:49:53.850 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000215179/8daf014a-0e15-4322-8266-c0f1c83b066e/subvolume_file.33 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:49:54.225 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:49:54.225 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:49:54.225 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0254555 s, 41.2 MB/s 2021-02-10T11:49:54.226 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:49:54.226 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:49:54.226 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:49:54.562 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:49:54.562 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:49:54.562 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000215179/8daf014a-0e15-4322-8266-c0f1c83b066e/subvolume_file.34 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:49:54.935 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:49:54.935 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:49:54.935 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0227077 s, 46.2 MB/s 2021-02-10T11:49:54.936 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:49:54.937 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:49:54.937 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:49:55.273 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:49:55.273 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:49:55.274 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000215179/8daf014a-0e15-4322-8266-c0f1c83b066e/subvolume_file.35 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:49:55.649 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:49:55.649 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:49:55.650 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0257395 s, 40.7 MB/s 2021-02-10T11:49:55.650 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:49:55.651 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:49:55.651 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:49:55.986 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:49:55.987 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:49:55.987 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000215179/8daf014a-0e15-4322-8266-c0f1c83b066e/subvolume_file.36 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:49:56.364 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:49:56.365 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:49:56.365 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0268282 s, 39.1 MB/s 2021-02-10T11:49:56.365 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:49:56.366 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:49:56.366 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:49:56.701 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:49:56.701 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:49:56.701 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000215179/8daf014a-0e15-4322-8266-c0f1c83b066e/subvolume_file.37 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:49:57.077 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:49:57.077 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:49:57.077 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0256506 s, 40.9 MB/s 2021-02-10T11:49:57.078 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:49:57.079 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:49:57.079 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:49:57.429 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:49:57.430 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:49:57.431 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000215179/8daf014a-0e15-4322-8266-c0f1c83b066e/subvolume_file.38 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:49:57.809 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:49:57.809 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:49:57.809 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0229267 s, 45.7 MB/s 2021-02-10T11:49:57.810 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:49:57.810 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:49:57.810 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:49:58.148 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:49:58.148 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:49:58.148 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000215179/8daf014a-0e15-4322-8266-c0f1c83b066e/subvolume_file.39 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:49:58.528 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:49:58.529 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:49:58.529 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0225025 s, 46.6 MB/s 2021-02-10T11:49:58.529 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:49:58.529 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:49:58.529 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:49:58.866 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:49:58.866 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:49:58.867 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000215179/8daf014a-0e15-4322-8266-c0f1c83b066e/subvolume_file.40 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:49:59.067 DEBUG:teuthology.orchestra.run.gibba012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:49:59.071 DEBUG:teuthology.orchestra.run.gibba020:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:49:59.237 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:49:59.237 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:49:59.237 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0231166 s, 45.4 MB/s 2021-02-10T11:49:59.238 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:49:59.239 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:49:59.239 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:49:59.574 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:49:59.574 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:49:59.574 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000215179/8daf014a-0e15-4322-8266-c0f1c83b066e/subvolume_file.41 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:49:59.948 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:49:59.948 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:49:59.948 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.021879 s, 47.9 MB/s 2021-02-10T11:49:59.949 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:49:59.949 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:49:59.949 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:50:00.284 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:50:00.285 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:50:00.285 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000215179/8daf014a-0e15-4322-8266-c0f1c83b066e/subvolume_file.42 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:50:00.659 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:50:00.659 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:50:00.659 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.02291 s, 45.8 MB/s 2021-02-10T11:50:00.660 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:50:00.660 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:50:00.660 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:50:00.995 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:50:00.996 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:50:00.996 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000215179/8daf014a-0e15-4322-8266-c0f1c83b066e/subvolume_file.43 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:50:01.371 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:50:01.371 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:50:01.371 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0236578 s, 44.3 MB/s 2021-02-10T11:50:01.372 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:50:01.372 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:50:01.372 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:50:01.707 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:50:01.708 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:50:01.708 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000215179/8daf014a-0e15-4322-8266-c0f1c83b066e/subvolume_file.44 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:50:02.085 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:50:02.085 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:50:02.085 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0232548 s, 45.1 MB/s 2021-02-10T11:50:02.085 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:50:02.085 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:50:02.085 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:50:02.428 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:50:02.428 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:50:02.428 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000215179/8daf014a-0e15-4322-8266-c0f1c83b066e/subvolume_file.45 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:50:02.808 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:50:02.808 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:50:02.808 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0261461 s, 40.1 MB/s 2021-02-10T11:50:02.809 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:50:02.809 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:50:02.809 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:50:03.146 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:50:03.146 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:50:03.146 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000215179/8daf014a-0e15-4322-8266-c0f1c83b066e/subvolume_file.46 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:50:03.522 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:50:03.522 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:50:03.522 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0234932 s, 44.6 MB/s 2021-02-10T11:50:03.523 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:50:03.523 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:50:03.523 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:50:03.856 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:50:03.856 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:50:03.857 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000215179/8daf014a-0e15-4322-8266-c0f1c83b066e/subvolume_file.47 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:50:04.231 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:50:04.231 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:50:04.231 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0256523 s, 40.9 MB/s 2021-02-10T11:50:04.232 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:50:04.233 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:50:04.233 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:50:04.568 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:50:04.568 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:50:04.569 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000215179/8daf014a-0e15-4322-8266-c0f1c83b066e/subvolume_file.48 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:50:04.944 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:50:04.944 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:50:04.944 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0228387 s, 45.9 MB/s 2021-02-10T11:50:04.945 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:50:04.945 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:50:04.945 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:50:05.282 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:50:05.282 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:50:05.282 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000215179/8daf014a-0e15-4322-8266-c0f1c83b066e/subvolume_file.49 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:50:05.658 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:50:05.658 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:50:05.658 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0240911 s, 43.5 MB/s 2021-02-10T11:50:05.659 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:50:05.660 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:50:05.660 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:50:05.996 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:50:05.996 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:50:05.996 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000215179/8daf014a-0e15-4322-8266-c0f1c83b066e/subvolume_file.50 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:50:06.372 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:50:06.373 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:50:06.373 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0237233 s, 44.2 MB/s 2021-02-10T11:50:06.374 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:50:06.374 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:50:06.374 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:50:06.710 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:50:06.710 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:50:06.710 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000215179/8daf014a-0e15-4322-8266-c0f1c83b066e/subvolume_file.51 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:50:07.083 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:50:07.084 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:50:07.084 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0236536 s, 44.3 MB/s 2021-02-10T11:50:07.085 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:50:07.085 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:50:07.085 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:50:07.420 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:50:07.420 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:50:07.420 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000215179/8daf014a-0e15-4322-8266-c0f1c83b066e/subvolume_file.52 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:50:07.795 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:50:07.796 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:50:07.796 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0230673 s, 45.5 MB/s 2021-02-10T11:50:07.797 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:50:07.797 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:50:07.797 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:50:08.132 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:50:08.132 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:50:08.132 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000215179/8daf014a-0e15-4322-8266-c0f1c83b066e/subvolume_file.53 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:50:08.507 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:50:08.507 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:50:08.507 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0225808 s, 46.4 MB/s 2021-02-10T11:50:08.508 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:50:08.508 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:50:08.508 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:50:08.844 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:50:08.844 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:50:08.845 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000215179/8daf014a-0e15-4322-8266-c0f1c83b066e/subvolume_file.54 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:50:09.217 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:50:09.217 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:50:09.217 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.022368 s, 46.9 MB/s 2021-02-10T11:50:09.218 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:50:09.219 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:50:09.219 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:50:09.554 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:50:09.554 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:50:09.554 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000215179/8daf014a-0e15-4322-8266-c0f1c83b066e/subvolume_file.55 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:50:09.928 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:50:09.929 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:50:09.929 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0222701 s, 47.1 MB/s 2021-02-10T11:50:09.930 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:50:09.930 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:50:09.930 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:50:10.265 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:50:10.266 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:50:10.266 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000215179/8daf014a-0e15-4322-8266-c0f1c83b066e/subvolume_file.56 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:50:10.641 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:50:10.641 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:50:10.642 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0241569 s, 43.4 MB/s 2021-02-10T11:50:10.642 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:50:10.643 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:50:10.643 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:50:10.979 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:50:10.979 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:50:10.980 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000215179/8daf014a-0e15-4322-8266-c0f1c83b066e/subvolume_file.57 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:50:11.356 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:50:11.356 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:50:11.356 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0236314 s, 44.4 MB/s 2021-02-10T11:50:11.357 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:50:11.357 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:50:11.358 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:50:11.694 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:50:11.694 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:50:11.694 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000215179/8daf014a-0e15-4322-8266-c0f1c83b066e/subvolume_file.58 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:50:12.069 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:50:12.069 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:50:12.069 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0231588 s, 45.3 MB/s 2021-02-10T11:50:12.070 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:50:12.070 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:50:12.070 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:50:12.405 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:50:12.406 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:50:12.406 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000215179/8daf014a-0e15-4322-8266-c0f1c83b066e/subvolume_file.59 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:50:12.782 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:50:12.783 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:50:12.783 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0253149 s, 41.4 MB/s 2021-02-10T11:50:12.784 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:50:12.784 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:50:12.784 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:50:13.119 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:50:13.119 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:50:13.119 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000215179/8daf014a-0e15-4322-8266-c0f1c83b066e/subvolume_file.60 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:50:13.495 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:50:13.496 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:50:13.496 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0242985 s, 43.2 MB/s 2021-02-10T11:50:13.497 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:50:13.497 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:50:13.497 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:50:13.833 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:50:13.833 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:50:13.833 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000215179/8daf014a-0e15-4322-8266-c0f1c83b066e/subvolume_file.61 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:50:14.204 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:50:14.204 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:50:14.205 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0229953 s, 45.6 MB/s 2021-02-10T11:50:14.205 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:50:14.206 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:50:14.206 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:50:14.541 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:50:14.541 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:50:14.542 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000215179/8daf014a-0e15-4322-8266-c0f1c83b066e/subvolume_file.62 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:50:14.921 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:50:14.921 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:50:14.921 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0260304 s, 40.3 MB/s 2021-02-10T11:50:14.922 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:50:14.923 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:50:14.923 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:50:15.257 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:50:15.257 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:50:15.257 DEBUG:teuthology.orchestra.run.gibba012:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000215179/8daf014a-0e15-4322-8266-c0f1c83b066e/subvolume_file.63 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:50:15.635 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records in 2021-02-10T11:50:15.636 INFO:teuthology.orchestra.run.gibba012.stderr:1+0 records out 2021-02-10T11:50:15.636 INFO:teuthology.orchestra.run.gibba012.stderr:1048576 bytes (1.0 MB) copied, 0.0255463 s, 41.0 MB/s 2021-02-10T11:50:15.637 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume snapshot create cephfs subvolume_0000000000215179 snapshot_0000000000779647 2021-02-10T11:50:16.069 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.049 7f9794485700 1 Processor -- start 2021-02-10T11:50:16.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.049 7f9794485700 1 -- start start 2021-02-10T11:50:16.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.049 7f9794485700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f978c0d1c80 0x7f978c0cef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:16.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.049 7f9794485700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f978c0cf590 0x7f978c0cf990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:16.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.049 7f9794485700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f978c0fe1d0 con 0x7f978c0d1880 2021-02-10T11:50:16.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.049 7f9794485700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f978c1b0cc0 con 0x7f978c0d1c80 2021-02-10T11:50:16.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.049 7f9794485700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f978c109a70 con 0x7f978c0cf590 2021-02-10T11:50:16.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.049 7f97922a1700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f978c0d1880 0x7f978c1b04d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:33364/0 (socket says 172.21.2.112:33364) 2021-02-10T11:50:16.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.049 7f97922a1700 1 -- 172.21.2.112:0/2966297211 learned_addr learned my addr 172.21.2.112:0/2966297211 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:50:16.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.049 7f9791aa0700 1 --2- 172.21.2.112:0/2966297211 >> v2:172.21.2.112:3300/0 conn(0x7f978c0d1c80 0x7f978c0cef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:16.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.050 7f9792aa2700 1 --2- 172.21.2.112:0/2966297211 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f978c0cf590 0x7f978c0cf990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:16.070 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.050 7f979129f700 1 -- 172.21.2.112:0/2966297211 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3950645854 0 0) 0x7f97800010b0 con 0x7f978c0d1880 2021-02-10T11:50:16.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.050 7f9791aa0700 1 -- 172.21.2.112:0/2966297211 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f978c0cf590 msgr2=0x7f978c0cf990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:50:16.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.050 7f9791aa0700 1 --2- 172.21.2.112:0/2966297211 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f978c0cf590 0x7f978c0cf990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:16.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.050 7f9791aa0700 1 -- 172.21.2.112:0/2966297211 >> v1:172.21.2.112:6789/0 conn(0x7f978c0d1880 legacy=0x7f978c1b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:16.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.050 7f9791aa0700 1 -- 172.21.2.112:0/2966297211 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f978c088070 con 0x7f978c0d1c80 2021-02-10T11:50:16.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.050 7f9791aa0700 1 --2- 172.21.2.112:0/2966297211 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f978c0d1c80 0x7f978c0cef70 secure :-1 s=READY pgs=3314 cs=0 l=1 rev1=1 rx=0x7f9788006650 tx=0x7f9788009a00).ready entity=mon.0 client_cookie=eee20c86b9c4398c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:16.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.050 7f979129f700 1 -- 172.21.2.112:0/2966297211 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f97880009b0 con 0x7f978c0d1c80 2021-02-10T11:50:16.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.050 7f979129f700 1 -- 172.21.2.112:0/2966297211 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f978800c0c0 con 0x7f978c0d1c80 2021-02-10T11:50:16.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.050 7f979129f700 1 -- 172.21.2.112:0/2966297211 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f978800c5a0 con 0x7f978c0d1c80 2021-02-10T11:50:16.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.050 7f9794485700 1 -- 172.21.2.112:0/2966297211 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f978c0d1c80 msgr2=0x7f978c0cef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:16.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.050 7f9794485700 1 --2- 172.21.2.112:0/2966297211 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f978c0d1c80 0x7f978c0cef70 secure :-1 s=READY pgs=3314 cs=0 l=1 rev1=1 rx=0x7f9788006650 tx=0x7f9788009a00).stop 2021-02-10T11:50:16.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.051 7f9794485700 1 -- 172.21.2.112:0/2966297211 shutdown_connections 2021-02-10T11:50:16.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.051 7f9794485700 1 --2- 172.21.2.112:0/2966297211 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f978c0cf590 0x7f978c0cf990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:16.071 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.051 7f9794485700 1 --2- 172.21.2.112:0/2966297211 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f978c0d1c80 0x7f978c0cef70 unknown :-1 s=CLOSED pgs=3314 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:16.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.051 7f9794485700 1 -- 172.21.2.112:0/2966297211 >> 172.21.2.112:0/2966297211 conn(0x7f978c0ce430 msgr2=0x7f978c0d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:16.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.051 7f9794485700 1 -- 172.21.2.112:0/2966297211 shutdown_connections 2021-02-10T11:50:16.072 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.051 7f9794485700 1 -- 172.21.2.112:0/2966297211 wait complete. 2021-02-10T11:50:16.073 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.066 7f9794485700 1 Processor -- start 2021-02-10T11:50:16.073 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.066 7f9794485700 1 -- start start 2021-02-10T11:50:16.073 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.066 7f9794485700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f978c0aef00 0x7f978c099c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:16.074 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.066 7f9794485700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f978c09a260 0x7f978c09a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:16.074 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.066 7f9794485700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f978c09ac80 0x7f978c19a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:16.074 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.066 7f9794485700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f978c19bdd0 con 0x7f978c09ac80 2021-02-10T11:50:16.074 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.066 7f9794485700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f978c19de70 con 0x7f978c0aef00 2021-02-10T11:50:16.074 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.066 7f9794485700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f978c19a2d0 con 0x7f978c09a260 2021-02-10T11:50:16.074 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.066 7f9792aa2700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f978c09ac80 0x7f978c19a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:16.074 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.066 7f9792aa2700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f978c09ac80 0x7f978c19a850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:33906/0 (socket says 172.21.2.112:33906) 2021-02-10T11:50:16.074 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.066 7f9792aa2700 1 -- 172.21.2.112:0/1764076594 learned_addr learned my addr 172.21.2.112:0/1764076594 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:50:16.074 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.067 7f97922a1700 1 --2- 172.21.2.112:0/1764076594 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f978c0aef00 0x7f978c099c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:16.074 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.067 7f9792aa2700 1 -- 172.21.2.112:0/1764076594 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f978c09a260 msgr2=0x7f978c09a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:16.074 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.067 7f9792aa2700 1 --2- 172.21.2.112:0/1764076594 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f978c09a260 0x7f978c09a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:16.074 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.067 7f9792aa2700 1 -- 172.21.2.112:0/1764076594 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f978c0aef00 msgr2=0x7f978c099c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:16.074 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.067 7f9792aa2700 1 --2- 172.21.2.112:0/1764076594 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f978c0aef00 0x7f978c099c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:16.075 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.067 7f9792aa2700 1 -- 172.21.2.112:0/1764076594 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f978c199f50 con 0x7f978c09ac80 2021-02-10T11:50:16.075 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.067 7f97922a1700 1 --2- 172.21.2.112:0/1764076594 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f978c0aef00 0x7f978c099c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:50:16.075 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.067 7f9792aa2700 1 --2- 172.21.2.112:0/1764076594 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f978c09ac80 0x7f978c19a850 secure :-1 s=READY pgs=3315 cs=0 l=1 rev1=1 rx=0x7f978400b970 tx=0x7f978400bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:16.075 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.067 7f977effd700 1 -- 172.21.2.112:0/1764076594 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9784000f90 con 0x7f978c09ac80 2021-02-10T11:50:16.075 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.067 7f977effd700 1 -- 172.21.2.112:0/1764076594 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9784006bb0 con 0x7f978c09ac80 2021-02-10T11:50:16.075 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.067 7f9794485700 1 -- 172.21.2.112:0/1764076594 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f978c199f50 con 0x7f978c09ac80 2021-02-10T11:50:16.075 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.067 7f9794485700 1 -- 172.21.2.112:0/1764076594 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f978c19b160 con 0x7f978c09ac80 2021-02-10T11:50:16.075 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.067 7f977effd700 1 -- 172.21.2.112:0/1764076594 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9784001400 con 0x7f978c09ac80 2021-02-10T11:50:16.075 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.068 7f977effd700 1 -- 172.21.2.112:0/1764076594 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f97840011f0 con 0x7f978c09ac80 2021-02-10T11:50:16.075 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.068 7f977effd700 1 --2- 172.21.2.112:0/1764076594 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9774018730 0x7f977401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:16.075 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.068 7f977effd700 1 -- 172.21.2.112:0/1764076594 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f9784032ae0 con 0x7f978c09ac80 2021-02-10T11:50:16.075 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.068 7f97922a1700 1 --2- 172.21.2.112:0/1764076594 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9774018730 0x7f977401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:16.075 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.068 7f97922a1700 1 --2- 172.21.2.112:0/1764076594 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9774018730 0x7f977401ac10 crc :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:16.103 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.099 7f9794485700 1 -- 172.21.2.112:0/1764076594 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f978c082a70 con 0x7f978c09ac80 2021-02-10T11:50:16.105 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.101 7f977effd700 1 -- 172.21.2.112:0/1764076594 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9784000f40 con 0x7f978c09ac80 2021-02-10T11:50:16.235 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.231 7f9794485700 1 -- 172.21.2.112:0/1764076594 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume snapshot create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000215179", "target": ["mgr", ""], "snap_name": "snapshot_0000000000779647"}) v1 -- 0x7f978c19b160 con 0x7f9774018730 2021-02-10T11:50:16.248 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.245 7f977effd700 1 -- 172.21.2.112:0/1764076594 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f97800009b0 con 0x7f9774018730 2021-02-10T11:50:16.253 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.249 7f9794485700 1 -- 172.21.2.112:0/1764076594 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9774018730 msgr2=0x7f977401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:16.253 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.249 7f9794485700 1 --2- 172.21.2.112:0/1764076594 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9774018730 0x7f977401ac10 crc :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:16.253 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.249 7f9794485700 1 -- 172.21.2.112:0/1764076594 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f978c09ac80 msgr2=0x7f978c19a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:16.253 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.249 7f9794485700 1 --2- 172.21.2.112:0/1764076594 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f978c09ac80 0x7f978c19a850 secure :-1 s=READY pgs=3315 cs=0 l=1 rev1=1 rx=0x7f978400b970 tx=0x7f978400bd10).stop 2021-02-10T11:50:16.254 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.249 7f9794485700 1 -- 172.21.2.112:0/1764076594 shutdown_connections 2021-02-10T11:50:16.254 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.249 7f9794485700 1 --2- 172.21.2.112:0/1764076594 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9774018730 0x7f977401ac10 unknown :-1 s=CLOSED pgs=1491 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:16.254 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.249 7f9794485700 1 --2- 172.21.2.112:0/1764076594 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f978c09ac80 0x7f978c19a850 unknown :-1 s=CLOSED pgs=3315 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:16.254 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.249 7f9794485700 1 --2- 172.21.2.112:0/1764076594 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f978c09a260 0x7f978c09a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:16.254 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.249 7f9794485700 1 --2- 172.21.2.112:0/1764076594 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f978c0aef00 0x7f978c099c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:16.254 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.249 7f9794485700 1 -- 172.21.2.112:0/1764076594 >> 172.21.2.112:0/1764076594 conn(0x7f978c18e810 msgr2=0x7f978c18c290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:16.254 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.249 7f9794485700 1 -- 172.21.2.112:0/1764076594 shutdown_connections 2021-02-10T11:50:16.254 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.249 7f9794485700 1 -- 172.21.2.112:0/1764076594 wait complete. 2021-02-10T11:50:16.261 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume snapshot clone cephfs subvolume_0000000000215179 snapshot_0000000000779647 clone_0000000000168834 2021-02-10T11:50:16.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.671 7f558ca09700 1 Processor -- start 2021-02-10T11:50:16.692 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.671 7f558ca09700 1 -- start start 2021-02-10T11:50:16.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.672 7f558ca09700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f55880dce80 0x7f55880d14f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:16.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.672 7f558ca09700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f558810b2c0 0x7f55880eb620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:16.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.672 7f558ca09700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f558810a550 con 0x7f55880eb220 2021-02-10T11:50:16.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.672 7f558ca09700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55880d1fa0 con 0x7f558810b2c0 2021-02-10T11:50:16.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.672 7f558ca09700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55880881d0 con 0x7f55880dce80 2021-02-10T11:50:16.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.672 7f5585e1b700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f55880eb220 0x7f558810aad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:33376/0 (socket says 172.21.2.112:33376) 2021-02-10T11:50:16.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.672 7f5585e1b700 1 -- 172.21.2.112:0/2519228326 learned_addr learned my addr 172.21.2.112:0/2519228326 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:50:16.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.672 7f5586e1d700 1 --2- 172.21.2.112:0/2519228326 >> v2:172.21.2.112:3300/0 conn(0x7f558810b2c0 0x7f55880eb620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:16.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.672 7f558661c700 1 --2- 172.21.2.112:0/2519228326 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f55880dce80 0x7f55880d14f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:16.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.672 7f558561a700 1 -- 172.21.2.112:0/2519228326 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2302075360 0 0) 0x7f55780010b0 con 0x7f55880eb220 2021-02-10T11:50:16.693 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.672 7f558561a700 1 -- 172.21.2.112:0/2519228326 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5574002fe0 con 0x7f55880eb220 2021-02-10T11:50:16.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.672 7f5586e1d700 1 -- 172.21.2.112:0/2519228326 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f55880dce80 msgr2=0x7f55880d14f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:50:16.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.672 7f5586e1d700 1 --2- 172.21.2.112:0/2519228326 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f55880dce80 0x7f55880d14f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:16.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.672 7f5586e1d700 1 -- 172.21.2.112:0/2519228326 >> v1:172.21.2.112:6789/0 conn(0x7f55880eb220 legacy=0x7f558810aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:16.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.672 7f5586e1d700 1 -- 172.21.2.112:0/2519228326 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55880dd280 con 0x7f558810b2c0 2021-02-10T11:50:16.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.673 7f5586e1d700 1 --2- 172.21.2.112:0/2519228326 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f558810b2c0 0x7f55880eb620 secure :-1 s=READY pgs=3317 cs=0 l=1 rev1=1 rx=0x7f5580006e30 tx=0x7f5580009e60).ready entity=mon.0 client_cookie=670b420c0c0eed69 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:16.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.673 7f558561a700 1 -- 172.21.2.112:0/2519228326 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5580006a50 con 0x7f558810b2c0 2021-02-10T11:50:16.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.673 7f558561a700 1 -- 172.21.2.112:0/2519228326 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f558000b300 con 0x7f558810b2c0 2021-02-10T11:50:16.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.673 7f558561a700 1 -- 172.21.2.112:0/2519228326 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f558000b820 con 0x7f558810b2c0 2021-02-10T11:50:16.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.673 7f558ca09700 1 -- 172.21.2.112:0/2519228326 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f558810b2c0 msgr2=0x7f55880eb620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:16.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.673 7f558ca09700 1 --2- 172.21.2.112:0/2519228326 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f558810b2c0 0x7f55880eb620 secure :-1 s=READY pgs=3317 cs=0 l=1 rev1=1 rx=0x7f5580006e30 tx=0x7f5580009e60).stop 2021-02-10T11:50:16.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.674 7f558ca09700 1 -- 172.21.2.112:0/2519228326 shutdown_connections 2021-02-10T11:50:16.694 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.674 7f558ca09700 1 --2- 172.21.2.112:0/2519228326 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f558810b2c0 0x7f55880eb620 unknown :-1 s=CLOSED pgs=3317 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:16.695 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.674 7f558ca09700 1 --2- 172.21.2.112:0/2519228326 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f55880dce80 0x7f55880d14f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:16.695 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.674 7f558ca09700 1 -- 172.21.2.112:0/2519228326 >> 172.21.2.112:0/2519228326 conn(0x7f55880ea630 msgr2=0x7f55880d0800 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:16.695 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.674 7f558ca09700 1 -- 172.21.2.112:0/2519228326 shutdown_connections 2021-02-10T11:50:16.695 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.674 7f558ca09700 1 -- 172.21.2.112:0/2519228326 wait complete. 2021-02-10T11:50:16.695 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.688 7f558ca09700 1 Processor -- start 2021-02-10T11:50:16.695 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.688 7f558ca09700 1 -- start start 2021-02-10T11:50:16.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.689 7f558ca09700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5588092c60 0x7f5588098bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:16.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.689 7f558ca09700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f55880addc0 0x7f55880991f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:16.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.689 7f558ca09700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f55880970f0 0x7f55880974f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:16.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.689 7f558ca09700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5588098650 con 0x7f55880970f0 2021-02-10T11:50:16.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.689 7f558ca09700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5588099810 con 0x7f55880addc0 2021-02-10T11:50:16.697 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.689 7f558ca09700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55880999f0 con 0x7f5588092c60 2021-02-10T11:50:16.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.689 7f558661c700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5588092c60 0x7f5588098bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:16.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.689 7f5585e1b700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f55880addc0 0x7f55880991f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:16.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.689 7f558661c700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5588092c60 0x7f5588098bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.120:3301/0 says I am v2:172.21.2.112:33600/0 (socket says 172.21.2.112:33600) 2021-02-10T11:50:16.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.689 7f558661c700 1 -- 172.21.2.112:0/3128669389 learned_addr learned my addr 172.21.2.112:0/3128669389 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:50:16.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.689 7f5586e1d700 1 --2- 172.21.2.112:0/3128669389 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f55880970f0 0x7f55880974f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:16.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.689 7f558661c700 1 -- 172.21.2.112:0/3128669389 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f55880addc0 msgr2=0x7f55880991f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:16.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.689 7f558661c700 1 --2- 172.21.2.112:0/3128669389 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f55880addc0 0x7f55880991f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:16.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.689 7f558661c700 1 -- 172.21.2.112:0/3128669389 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f55880970f0 msgr2=0x7f55880974f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:16.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.689 7f558661c700 1 --2- 172.21.2.112:0/3128669389 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f55880970f0 0x7f55880974f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:16.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.689 7f558661c700 1 -- 172.21.2.112:0/3128669389 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5588097b40 con 0x7f5588092c60 2021-02-10T11:50:16.698 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.689 7f558661c700 1 --2- 172.21.2.112:0/3128669389 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5588092c60 0x7f5588098bd0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7f5570006f60 tx=0x7f557000b970).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:16.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.690 7f557effd700 1 -- 172.21.2.112:0/3128669389 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f557000bd60 con 0x7f5588092c60 2021-02-10T11:50:16.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.690 7f557effd700 1 -- 172.21.2.112:0/3128669389 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5570000c90 con 0x7f5588092c60 2021-02-10T11:50:16.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.690 7f558ca09700 1 -- 172.21.2.112:0/3128669389 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5588097b40 con 0x7f5588092c60 2021-02-10T11:50:16.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.690 7f558ca09700 1 -- 172.21.2.112:0/3128669389 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5588097b40 con 0x7f5588092c60 2021-02-10T11:50:16.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.690 7f557effd700 1 -- 172.21.2.112:0/3128669389 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f55700009b0 con 0x7f5588092c60 2021-02-10T11:50:16.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.690 7f557effd700 1 -- 172.21.2.112:0/3128669389 <== mon.2 v2:172.21.2.120:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5570006b00 con 0x7f5588092c60 2021-02-10T11:50:16.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.690 7f557effd700 1 --2- 172.21.2.112:0/3128669389 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f556c018730 0x7f556c01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:16.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.690 7f557effd700 1 -- 172.21.2.112:0/3128669389 <== mon.2 v2:172.21.2.120:3301/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f55700327a0 con 0x7f5588092c60 2021-02-10T11:50:16.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.691 7f5585e1b700 1 --2- 172.21.2.112:0/3128669389 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f556c018730 0x7f556c01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:16.699 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.691 7f5585e1b700 1 --2- 172.21.2.112:0/3128669389 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f556c018730 0x7f556c01ac10 crc :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:16.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.722 7f558ca09700 1 -- 172.21.2.112:0/3128669389 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f558810a550 con 0x7f5588092c60 2021-02-10T11:50:16.728 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.724 7f557effd700 1 -- 172.21.2.112:0/3128669389 <== mon.2 v2:172.21.2.120:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f55700009b0 con 0x7f5588092c60 2021-02-10T11:50:16.857 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.852 7f558ca09700 1 -- 172.21.2.112:0/3128669389 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "subvolume_0000000000215179", "target": ["mgr", ""], "snap_name": "snapshot_0000000000779647", "target_sub_name": "clone_0000000000168834"}) v1 -- 0x7f558810a550 con 0x7f556c018730 2021-02-10T11:50:16.904 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.900 7f557effd700 1 -- 172.21.2.112:0/3128669389 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f5578000ec0 con 0x7f556c018730 2021-02-10T11:50:16.923 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.918 7f558ca09700 1 -- 172.21.2.112:0/3128669389 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f556c018730 msgr2=0x7f556c01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:16.923 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.918 7f558ca09700 1 --2- 172.21.2.112:0/3128669389 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f556c018730 0x7f556c01ac10 crc :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:16.923 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.918 7f558ca09700 1 -- 172.21.2.112:0/3128669389 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5588092c60 msgr2=0x7f5588098bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:16.923 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.918 7f558ca09700 1 --2- 172.21.2.112:0/3128669389 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5588092c60 0x7f5588098bd0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7f5570006f60 tx=0x7f557000b970).stop 2021-02-10T11:50:16.923 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.918 7f558ca09700 1 -- 172.21.2.112:0/3128669389 shutdown_connections 2021-02-10T11:50:16.923 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.918 7f558ca09700 1 --2- 172.21.2.112:0/3128669389 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f556c018730 0x7f556c01ac10 unknown :-1 s=CLOSED pgs=1492 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:16.923 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.918 7f558ca09700 1 --2- 172.21.2.112:0/3128669389 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f55880addc0 0x7f55880991f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:16.923 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.918 7f558ca09700 1 --2- 172.21.2.112:0/3128669389 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f55880970f0 0x7f55880974f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:16.923 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.918 7f558ca09700 1 --2- 172.21.2.112:0/3128669389 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f5588092c60 0x7f5588098bd0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:16.923 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.918 7f558ca09700 1 -- 172.21.2.112:0/3128669389 >> 172.21.2.112:0/3128669389 conn(0x7f55880a1390 msgr2=0x7f55880a8e10 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:16.923 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.918 7f558ca09700 1 -- 172.21.2.112:0/3128669389 shutdown_connections 2021-02-10T11:50:16.924 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:16.918 7f558ca09700 1 -- 172.21.2.112:0/3128669389 wait complete. 2021-02-10T11:50:16.931 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs clone_0000000000168834 2021-02-10T11:50:17.366 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:17.345 7efc76a32700 1 Processor -- start 2021-02-10T11:50:17.366 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:17.345 7efc76a32700 1 -- start start 2021-02-10T11:50:17.366 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:17.345 7efc76a32700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7efc700d1870 0x7efc701b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:17.367 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:17.345 7efc76a32700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7efc701b0ae0 0x7efc70108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:17.367 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:17.345 7efc76a32700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efc700d1c70 0x7efc70108f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:17.367 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:17.345 7efc76a32700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc700fe1c0 con 0x7efc701b0ae0 2021-02-10T11:50:17.367 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:17.345 7efc76a32700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc70109a60 con 0x7efc700d1870 2021-02-10T11:50:17.367 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:17.345 7efc76a32700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc70088070 con 0x7efc700d1c70 2021-02-10T11:50:17.367 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:17.346 7efc6ffff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7efc701b0ae0 0x7efc70108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:17.367 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:17.346 7efc6ffff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7efc701b0ae0 0x7efc70108940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:33926/0 (socket says 172.21.2.112:33926) 2021-02-10T11:50:17.367 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:17.346 7efc6ffff700 1 -- 172.21.2.112:0/1327127341 learned_addr learned my addr 172.21.2.112:0/1327127341 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:50:17.367 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:17.346 7efc7484e700 1 --2- 172.21.2.112:0/1327127341 >> v2:172.21.2.120:3300/0 conn(0x7efc700d1870 0x7efc701b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:17.367 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:17.346 7efc7504f700 1 --2- 172.21.2.112:0/1327127341 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efc700d1c70 0x7efc70108f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:17.367 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:17.346 7efc6ffff700 1 -- 172.21.2.112:0/1327127341 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efc700d1c70 msgr2=0x7efc70108f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:50:17.367 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:17.346 7efc6ffff700 1 --2- 172.21.2.112:0/1327127341 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efc700d1c70 0x7efc70108f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:17.368 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:17.346 7efc6ffff700 1 -- 172.21.2.112:0/1327127341 >> v2:172.21.2.120:3300/0 conn(0x7efc700d1870 msgr2=0x7efc701b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:50:17.368 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:17.346 7efc6ffff700 1 --2- 172.21.2.112:0/1327127341 >> v2:172.21.2.120:3300/0 conn(0x7efc700d1870 0x7efc701b04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:17.368 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:17.346 7efc6ffff700 1 -- 172.21.2.112:0/1327127341 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc70109580 con 0x7efc701b0ae0 2021-02-10T11:50:17.368 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:17.346 7efc6ffff700 1 --2- 172.21.2.112:0/1327127341 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efc701b0ae0 0x7efc70108940 secure :-1 s=READY pgs=3318 cs=0 l=1 rev1=1 rx=0x7efc5c006700 tx=0x7efc5c009b00).ready entity=mon.0 client_cookie=14a455e48f751860 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:17.368 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:17.346 7efc6f7fe700 1 -- 172.21.2.112:0/1327127341 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efc5c00c0c0 con 0x7efc701b0ae0 2021-02-10T11:50:17.368 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:17.346 7efc6f7fe700 1 -- 172.21.2.112:0/1327127341 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc5c00c380 con 0x7efc701b0ae0 2021-02-10T11:50:17.368 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:17.346 7efc6f7fe700 1 -- 172.21.2.112:0/1327127341 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efc5c00c860 con 0x7efc701b0ae0 2021-02-10T11:50:17.368 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:17.346 7efc76a32700 1 -- 172.21.2.112:0/1327127341 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efc701b0ae0 msgr2=0x7efc70108940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:17.368 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:17.346 7efc76a32700 1 --2- 172.21.2.112:0/1327127341 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efc701b0ae0 0x7efc70108940 secure :-1 s=READY pgs=3318 cs=0 l=1 rev1=1 rx=0x7efc5c006700 tx=0x7efc5c009b00).stop 2021-02-10T11:50:17.368 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:17.347 7efc76a32700 1 -- 172.21.2.112:0/1327127341 shutdown_connections 2021-02-10T11:50:17.368 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:17.347 7efc76a32700 1 --2- 172.21.2.112:0/1327127341 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efc700d1c70 0x7efc70108f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:17.368 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:17.347 7efc76a32700 1 --2- 172.21.2.112:0/1327127341 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efc701b0ae0 0x7efc70108940 unknown :-1 s=CLOSED pgs=3318 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:17.368 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:17.347 7efc76a32700 1 --2- 172.21.2.112:0/1327127341 >> v2:172.21.2.120:3300/0 conn(0x7efc700d1870 0x7efc701b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:17.368 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:17.347 7efc76a32700 1 -- 172.21.2.112:0/1327127341 >> 172.21.2.112:0/1327127341 conn(0x7efc700ce420 msgr2=0x7efc700d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:17.369 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:17.347 7efc76a32700 1 -- 172.21.2.112:0/1327127341 shutdown_connections 2021-02-10T11:50:17.369 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:17.347 7efc76a32700 1 -- 172.21.2.112:0/1327127341 wait complete. 2021-02-10T11:50:17.369 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:17.362 7efc76a32700 1 Processor -- start 2021-02-10T11:50:17.369 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:17.362 7efc76a32700 1 -- start start 2021-02-10T11:50:17.370 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:17.363 7efc76a32700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efc700b22c0 0x7efc700b26c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:17.371 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:17.363 7efc76a32700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7efc700a6df0 0x7efc700a71f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:17.371 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:17.363 7efc76a32700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efc700a5cd0 0x7efc700a60d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:17.371 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:17.363 7efc76a32700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc700c23f0 con 0x7efc700a5cd0 2021-02-10T11:50:17.371 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:17.363 7efc76a32700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc700a66f0 con 0x7efc700a6df0 2021-02-10T11:50:17.371 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:17.363 7efc76a32700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc700a68e0 con 0x7efc700b22c0 2021-02-10T11:50:17.371 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:17.363 7efc7504f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efc700a5cd0 0x7efc700a60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:17.371 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:17.363 7efc7504f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efc700a5cd0 0x7efc700a60d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:33936/0 (socket says 172.21.2.112:33936) 2021-02-10T11:50:17.371 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:17.363 7efc7504f700 1 -- 172.21.2.112:0/1503209650 learned_addr learned my addr 172.21.2.112:0/1503209650 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:50:17.371 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:17.363 7efc7484e700 1 --2- 172.21.2.112:0/1503209650 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efc700b22c0 0x7efc700b26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:17.371 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:17.363 7efc6ffff700 1 --2- 172.21.2.112:0/1503209650 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7efc700a6df0 0x7efc700a71f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:17.371 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:17.363 7efc7504f700 1 -- 172.21.2.112:0/1503209650 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efc700b22c0 msgr2=0x7efc700b26c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:17.371 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:17.363 7efc7504f700 1 --2- 172.21.2.112:0/1503209650 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efc700b22c0 0x7efc700b26c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:17.371 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:17.363 7efc7504f700 1 -- 172.21.2.112:0/1503209650 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7efc700a6df0 msgr2=0x7efc700a71f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:17.372 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:17.363 7efc7504f700 1 --2- 172.21.2.112:0/1503209650 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7efc700a6df0 0x7efc700a71f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:17.372 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:17.363 7efc7504f700 1 -- 172.21.2.112:0/1503209650 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc700c0e40 con 0x7efc700a5cd0 2021-02-10T11:50:17.372 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:17.363 7efc6ffff700 1 --2- 172.21.2.112:0/1503209650 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7efc700a6df0 0x7efc700a71f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:50:17.372 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:17.363 7efc7504f700 1 --2- 172.21.2.112:0/1503209650 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efc700a5cd0 0x7efc700a60d0 secure :-1 s=READY pgs=3319 cs=0 l=1 rev1=1 rx=0x7efc68006f40 tx=0x7efc6800bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:17.372 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:17.363 7efc6d7fa700 1 -- 172.21.2.112:0/1503209650 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efc680011f0 con 0x7efc700a5cd0 2021-02-10T11:50:17.372 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:17.363 7efc6d7fa700 1 -- 172.21.2.112:0/1503209650 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc68003030 con 0x7efc700a5cd0 2021-02-10T11:50:17.372 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:17.363 7efc6d7fa700 1 -- 172.21.2.112:0/1503209650 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efc680016a0 con 0x7efc700a5cd0 2021-02-10T11:50:17.372 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:17.363 7efc76a32700 1 -- 172.21.2.112:0/1503209650 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc700c0e40 con 0x7efc700a5cd0 2021-02-10T11:50:17.372 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:17.363 7efc76a32700 1 -- 172.21.2.112:0/1503209650 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc700c0e40 con 0x7efc700a5cd0 2021-02-10T11:50:17.372 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:17.364 7efc6d7fa700 1 -- 172.21.2.112:0/1503209650 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7efc68001470 con 0x7efc700a5cd0 2021-02-10T11:50:17.372 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:17.364 7efc6d7fa700 1 --2- 172.21.2.112:0/1503209650 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7efc50018730 0x7efc5001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:17.372 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:17.364 7efc6d7fa700 1 -- 172.21.2.112:0/1503209650 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 5573+0+0 (secure 0 0 0) 0x7efc68032a20 con 0x7efc700a5cd0 2021-02-10T11:50:17.372 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:17.364 7efc7484e700 1 --2- 172.21.2.112:0/1503209650 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7efc50018730 0x7efc5001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:17.372 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:17.364 7efc7484e700 1 --2- 172.21.2.112:0/1503209650 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7efc50018730 0x7efc5001ac10 crc :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:17.398 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:17.394 7efc76a32700 1 -- 172.21.2.112:0/1503209650 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc70082a70 con 0x7efc700a5cd0 2021-02-10T11:50:17.401 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:17.397 7efc6d7fa700 1 -- 172.21.2.112:0/1503209650 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7efc68000f40 con 0x7efc700a5cd0 2021-02-10T11:50:17.538 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:17.534 7efc76a32700 1 -- 172.21.2.112:0/1503209650 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "clone_0000000000168834", "target": ["mgr", ""]}) v1 -- 0x7efc700a66f0 con 0x7efc50018730 2021-02-10T11:50:17.540 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:17.537 7efc6d7fa700 1 -- 172.21.2.112:0/1503209650 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+78 (crc 0 0 0) 0x7efc64006a70 con 0x7efc50018730 2021-02-10T11:50:17.541 INFO:teuthology.orchestra.run.gibba012.stdout:/volumes/_nogroup/clone_0000000000168834/c7e2554e-daa6-4020-9d17-cc2b12c3e0b0 2021-02-10T11:50:17.543 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:17.538 7efc76a32700 1 -- 172.21.2.112:0/1503209650 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7efc50018730 msgr2=0x7efc5001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:17.543 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:17.538 7efc76a32700 1 --2- 172.21.2.112:0/1503209650 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7efc50018730 0x7efc5001ac10 crc :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:17.543 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:17.538 7efc76a32700 1 -- 172.21.2.112:0/1503209650 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efc700a5cd0 msgr2=0x7efc700a60d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:17.543 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:17.538 7efc76a32700 1 --2- 172.21.2.112:0/1503209650 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efc700a5cd0 0x7efc700a60d0 secure :-1 s=READY pgs=3319 cs=0 l=1 rev1=1 rx=0x7efc68006f40 tx=0x7efc6800bd40).stop 2021-02-10T11:50:17.543 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:17.538 7efc76a32700 1 -- 172.21.2.112:0/1503209650 shutdown_connections 2021-02-10T11:50:17.543 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:17.538 7efc76a32700 1 --2- 172.21.2.112:0/1503209650 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7efc50018730 0x7efc5001ac10 unknown :-1 s=CLOSED pgs=1493 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:17.543 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:17.538 7efc76a32700 1 --2- 172.21.2.112:0/1503209650 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7efc700a6df0 0x7efc700a71f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:17.543 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:17.538 7efc76a32700 1 --2- 172.21.2.112:0/1503209650 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7efc700a5cd0 0x7efc700a60d0 unknown :-1 s=CLOSED pgs=3319 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:17.543 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:17.538 7efc76a32700 1 --2- 172.21.2.112:0/1503209650 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7efc700b22c0 0x7efc700b26c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:17.543 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:17.538 7efc76a32700 1 -- 172.21.2.112:0/1503209650 >> 172.21.2.112:0/1503209650 conn(0x7efc7009ffd0 msgr2=0x7efc70082ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:17.543 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:17.538 7efc76a32700 1 -- 172.21.2.112:0/1503209650 shutdown_connections 2021-02-10T11:50:17.544 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:17.538 7efc76a32700 1 -- 172.21.2.112:0/1503209650 wait complete. 2021-02-10T11:50:17.551 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:50:17.551 DEBUG:teuthology.orchestra.run.gibba012:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:50:17.898 DEBUG:teuthology.orchestra.run.gibba020:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:50:18.247 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:50:18.247 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:50:18.247 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:50:18.247 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:50:18.583 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:50:18.583 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:50:18.583 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba012.front.sepia.ceph.com... 2021-02-10T11:50:18.584 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:50:18.584 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:50:18.939 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba012.stderr:ceph-fuse[196573]: fuse finished with error 0 and tester_r 0 2021-02-10T11:50:18.940 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:50:18.940 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:50:18.940 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:50:19.277 INFO:teuthology.orchestra.run.gibba012.stdout:ext2/ext3 2021-02-10T11:50:19.278 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:50:19.278 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:50:19.278 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:50:19.614 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:50:19.614 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:50:19.614 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:50:19.951 INFO:teuthology.orchestra.run.gibba012.stdout:fuseblk 2021-02-10T11:50:19.951 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:50:19.951 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba012.front.sepia.ceph.com... 2021-02-10T11:50:19.951 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:50:19.952 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:50:20.305 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba012.stderr:ceph-fuse[197213]: fuse finished with error 0 and tester_r 0 2021-02-10T11:50:20.306 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:50:20.307 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:50:20.307 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:50:20.641 INFO:teuthology.orchestra.run.gibba012.stdout:ext2/ext3 2021-02-10T11:50:20.642 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:50:20.642 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:50:20.642 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:50:20.978 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:50:20.979 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:50:20.979 DEBUG:teuthology.orchestra.run.gibba020:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:50:21.314 INFO:teuthology.orchestra.run.gibba020.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:50:21.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:50:21.315 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:50:21.315 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:50:21.315 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:50:21.650 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:50:21.650 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:50:21.650 DEBUG:teuthology.orchestra.run.gibba020:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:50:21.985 INFO:teuthology.orchestra.run.gibba020.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:50:21.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:50:21.985 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:50:21.986 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:50:21.986 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:50:22.320 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config rm mon mon_allow_pool_delete 2021-02-10T11:50:22.754 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:22.732 7fbd34c4a700 1 Processor -- start 2021-02-10T11:50:22.754 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:22.732 7fbd34c4a700 1 -- start start 2021-02-10T11:50:22.754 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:22.733 7fbd34c4a700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fbd300d0340 0x7fbd300ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:22.754 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:22.733 7fbd34c4a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbd300d1c70 0x7fbd300efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:22.754 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:22.733 7fbd34c4a700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd300f0b10 con 0x7fbd300d1870 2021-02-10T11:50:22.754 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:22.733 7fbd34c4a700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd300e5ed0 con 0x7fbd300d0340 2021-02-10T11:50:22.754 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:22.733 7fbd34c4a700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd301b0be0 con 0x7fbd300d1c70 2021-02-10T11:50:22.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:22.733 7fbd2e05d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fbd300d0340 0x7fbd300ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:22.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:22.733 7fbd2e05d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fbd300d0340 0x7fbd300ef640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:33940/0 (socket says 172.21.2.112:33940) 2021-02-10T11:50:22.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:22.733 7fbd2e05d700 1 -- 172.21.2.112:0/2478064455 learned_addr learned my addr 172.21.2.112:0/2478064455 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:50:22.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:22.733 7fbd2f05f700 1 --2- 172.21.2.112:0/2478064455 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbd300d1c70 0x7fbd300efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:22.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:22.733 7fbd2e05d700 1 -- 172.21.2.112:0/2478064455 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbd300d1c70 msgr2=0x7fbd300efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:50:22.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:22.733 7fbd2e05d700 1 --2- 172.21.2.112:0/2478064455 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbd300d1c70 0x7fbd300efc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:22.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:22.733 7fbd2e05d700 1 -- 172.21.2.112:0/2478064455 >> v1:172.21.2.120:6789/0 conn(0x7fbd300d1870 legacy=0x7fbd301b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:50:22.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:22.733 7fbd2e05d700 1 -- 172.21.2.112:0/2478064455 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd300f20a0 con 0x7fbd300d0340 2021-02-10T11:50:22.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:22.733 7fbd2f05f700 1 --2- 172.21.2.112:0/2478064455 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbd300d1c70 0x7fbd300efc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:50:22.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:22.734 7fbd2e05d700 1 --2- 172.21.2.112:0/2478064455 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbd300d0340 0x7fbd300ef640 secure :-1 s=READY pgs=3320 cs=0 l=1 rev1=1 rx=0x7fbd24006700 tx=0x7fbd24009b80).ready entity=mon.0 client_cookie=453bbfc69669507e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:22.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:22.734 7fbd2d85c700 1 -- 172.21.2.112:0/2478064455 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbd2400c0c0 con 0x7fbd300d0340 2021-02-10T11:50:22.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:22.734 7fbd2d85c700 1 -- 172.21.2.112:0/2478064455 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd2400c340 con 0x7fbd300d0340 2021-02-10T11:50:22.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:22.734 7fbd2d85c700 1 -- 172.21.2.112:0/2478064455 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbd2400c860 con 0x7fbd300d0340 2021-02-10T11:50:22.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:22.734 7fbd34c4a700 1 -- 172.21.2.112:0/2478064455 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbd300d0340 msgr2=0x7fbd300ef640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:22.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:22.734 7fbd34c4a700 1 --2- 172.21.2.112:0/2478064455 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbd300d0340 0x7fbd300ef640 secure :-1 s=READY pgs=3320 cs=0 l=1 rev1=1 rx=0x7fbd24006700 tx=0x7fbd24009b80).stop 2021-02-10T11:50:22.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:22.735 7fbd34c4a700 1 -- 172.21.2.112:0/2478064455 shutdown_connections 2021-02-10T11:50:22.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:22.735 7fbd34c4a700 1 --2- 172.21.2.112:0/2478064455 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbd300d0340 0x7fbd300ef640 unknown :-1 s=CLOSED pgs=3320 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:22.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:22.735 7fbd34c4a700 1 --2- 172.21.2.112:0/2478064455 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbd300d1c70 0x7fbd300efc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:22.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:22.735 7fbd34c4a700 1 -- 172.21.2.112:0/2478064455 >> 172.21.2.112:0/2478064455 conn(0x7fbd300ce450 msgr2=0x7fbd300d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:22.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:22.735 7fbd34c4a700 1 -- 172.21.2.112:0/2478064455 shutdown_connections 2021-02-10T11:50:22.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:22.735 7fbd34c4a700 1 -- 172.21.2.112:0/2478064455 wait complete. 2021-02-10T11:50:22.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:22.749 7fbd34c4a700 1 Processor -- start 2021-02-10T11:50:22.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:22.749 7fbd34c4a700 1 -- start start 2021-02-10T11:50:22.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:22.750 7fbd34c4a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbd3009f1c0 0x7fbd3009f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:22.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:22.750 7fbd34c4a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbd301a4510 0x7fbd301a4910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:22.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:22.750 7fbd34c4a700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fbd3009ddb0 0x7fbd3009e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:22.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:22.750 7fbd34c4a700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd3018fb60 con 0x7fbd301a4510 2021-02-10T11:50:22.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:22.750 7fbd34c4a700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd3018fd80 con 0x7fbd3009ddb0 2021-02-10T11:50:22.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:22.750 7fbd34c4a700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd3009e7d0 con 0x7fbd3009f1c0 2021-02-10T11:50:22.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:22.750 7fbd2e05d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbd301a4510 0x7fbd301a4910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:22.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:22.750 7fbd2e05d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbd301a4510 0x7fbd301a4910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:33944/0 (socket says 172.21.2.112:33944) 2021-02-10T11:50:22.758 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:22.750 7fbd2e05d700 1 -- 172.21.2.112:0/777088235 learned_addr learned my addr 172.21.2.112:0/777088235 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:50:22.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:22.750 7fbd2f05f700 1 --2- 172.21.2.112:0/777088235 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fbd3009ddb0 0x7fbd3009e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:22.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:22.750 7fbd2e85e700 1 --2- 172.21.2.112:0/777088235 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbd3009f1c0 0x7fbd3009f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:22.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:22.750 7fbd2e05d700 1 -- 172.21.2.112:0/777088235 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbd3009f1c0 msgr2=0x7fbd3009f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:22.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:22.750 7fbd2e05d700 1 --2- 172.21.2.112:0/777088235 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbd3009f1c0 0x7fbd3009f5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:22.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:22.750 7fbd2e05d700 1 -- 172.21.2.112:0/777088235 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fbd3009ddb0 msgr2=0x7fbd3009e1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:22.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:22.750 7fbd2e05d700 1 --2- 172.21.2.112:0/777088235 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fbd3009ddb0 0x7fbd3009e1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:22.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:22.750 7fbd2e05d700 1 -- 172.21.2.112:0/777088235 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd3009e9b0 con 0x7fbd301a4510 2021-02-10T11:50:22.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:22.750 7fbd2f05f700 1 --2- 172.21.2.112:0/777088235 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fbd3009ddb0 0x7fbd3009e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:50:22.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:22.750 7fbd2e85e700 1 --2- 172.21.2.112:0/777088235 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbd3009f1c0 0x7fbd3009f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:50:22.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:22.750 7fbd2e05d700 1 --2- 172.21.2.112:0/777088235 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbd301a4510 0x7fbd301a4910 secure :-1 s=READY pgs=3321 cs=0 l=1 rev1=1 rx=0x7fbd2400b5e0 tx=0x7fbd2400b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:22.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:22.750 7fbd1f7fe700 1 -- 172.21.2.112:0/777088235 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbd2400bd70 con 0x7fbd301a4510 2021-02-10T11:50:22.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:22.751 7fbd1f7fe700 1 -- 172.21.2.112:0/777088235 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd24002030 con 0x7fbd301a4510 2021-02-10T11:50:22.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:22.751 7fbd34c4a700 1 -- 172.21.2.112:0/777088235 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd3009e9b0 con 0x7fbd301a4510 2021-02-10T11:50:22.759 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:22.751 7fbd1f7fe700 1 -- 172.21.2.112:0/777088235 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbd24002550 con 0x7fbd301a4510 2021-02-10T11:50:22.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:22.751 7fbd34c4a700 1 -- 172.21.2.112:0/777088235 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd3009e9b0 con 0x7fbd301a4510 2021-02-10T11:50:22.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:22.751 7fbd1f7fe700 1 -- 172.21.2.112:0/777088235 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fbd24000c90 con 0x7fbd301a4510 2021-02-10T11:50:22.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:22.751 7fbd1f7fe700 1 --2- 172.21.2.112:0/777088235 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fbd140186c0 0x7fbd1401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:22.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:22.751 7fbd1f7fe700 1 -- 172.21.2.112:0/777088235 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fbd240326e0 con 0x7fbd301a4510 2021-02-10T11:50:22.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:22.752 7fbd2e85e700 1 --2- 172.21.2.112:0/777088235 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fbd140186c0 0x7fbd1401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:22.760 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:22.752 7fbd2e85e700 1 --2- 172.21.2.112:0/777088235 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fbd140186c0 0x7fbd1401abc0 crc :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:22.788 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:22.782 7fbd34c4a700 1 -- 172.21.2.112:0/777088235 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd300f0b10 con 0x7fbd301a4510 2021-02-10T11:50:22.790 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:22.785 7fbd1f7fe700 1 -- 172.21.2.112:0/777088235 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fbd2401f5a0 con 0x7fbd301a4510 2021-02-10T11:50:22.939 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:22.934 7fbd34c4a700 1 -- 172.21.2.112:0/777088235 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fbd300f0b10 con 0x7fbd301a4510 2021-02-10T11:50:22.942 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:22.937 7fbd1f7fe700 1 -- 172.21.2.112:0/777088235 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v49) v1 ==== 104+0+0 (secure 0 0 0) 0x7fbd24000c90 con 0x7fbd301a4510 2021-02-10T11:50:22.948 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:22.942 7fbd34c4a700 1 -- 172.21.2.112:0/777088235 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fbd140186c0 msgr2=0x7fbd1401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:22.948 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:22.942 7fbd34c4a700 1 --2- 172.21.2.112:0/777088235 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fbd140186c0 0x7fbd1401abc0 crc :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:22.948 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:22.942 7fbd34c4a700 1 -- 172.21.2.112:0/777088235 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbd301a4510 msgr2=0x7fbd301a4910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:22.948 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:22.942 7fbd34c4a700 1 --2- 172.21.2.112:0/777088235 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbd301a4510 0x7fbd301a4910 secure :-1 s=READY pgs=3321 cs=0 l=1 rev1=1 rx=0x7fbd2400b5e0 tx=0x7fbd2400b980).stop 2021-02-10T11:50:22.948 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:22.942 7fbd34c4a700 1 -- 172.21.2.112:0/777088235 shutdown_connections 2021-02-10T11:50:22.948 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:22.943 7fbd34c4a700 1 --2- 172.21.2.112:0/777088235 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fbd140186c0 0x7fbd1401abc0 unknown :-1 s=CLOSED pgs=1494 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:22.948 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:22.943 7fbd34c4a700 1 --2- 172.21.2.112:0/777088235 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fbd3009ddb0 0x7fbd3009e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:22.948 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:22.943 7fbd34c4a700 1 --2- 172.21.2.112:0/777088235 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbd301a4510 0x7fbd301a4910 unknown :-1 s=CLOSED pgs=3321 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:22.948 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:22.943 7fbd34c4a700 1 --2- 172.21.2.112:0/777088235 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbd3009f1c0 0x7fbd3009f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:22.948 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:22.943 7fbd34c4a700 1 -- 172.21.2.112:0/777088235 >> 172.21.2.112:0/777088235 conn(0x7fbd3009fd30 msgr2=0x7fbd30190100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:22.949 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:22.943 7fbd34c4a700 1 -- 172.21.2.112:0/777088235 shutdown_connections 2021-02-10T11:50:22.949 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:22.943 7fbd34c4a700 1 -- 172.21.2.112:0/777088235 wait complete. 2021-02-10T11:50:22.957 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestVolumes.test_subvolume_clone_in_progress_getpath' 2021-02-10T11:50:23.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:23.365 7f00b7cb2700 1 Processor -- start 2021-02-10T11:50:23.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:23.365 7f00b7cb2700 1 -- start start 2021-02-10T11:50:23.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:23.366 7f00b7cb2700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f00b00d1870 0x7f00b01b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:23.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:23.366 7f00b7cb2700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f00b01b0a10 0x7f00b00cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:23.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:23.366 7f00b7cb2700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00b00f0b10 con 0x7f00b00d1c70 2021-02-10T11:50:23.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:23.366 7f00b7cb2700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00b00e5ed0 con 0x7f00b00d1870 2021-02-10T11:50:23.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:23.366 7f00b7cb2700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00b00f0650 con 0x7f00b01b0a10 2021-02-10T11:50:23.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:23.366 7f00b62cf700 1 --1- >> v1:172.21.2.120:6789/0 conn(0x7f00b00d1c70 0x7f00b00cf950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.120:6789/0 says I am v1:172.21.2.112:55204/0 (socket says 172.21.2.112:55204) 2021-02-10T11:50:23.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:23.366 7f00b62cf700 1 -- 172.21.2.112:0/46087165 learned_addr learned my addr 172.21.2.112:0/46087165 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:50:23.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:23.366 7f00b52cd700 1 --2- 172.21.2.112:0/46087165 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f00b01b0a10 0x7f00b00cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:23.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:23.366 7f00b5ace700 1 --2- 172.21.2.112:0/46087165 >> v2:172.21.2.120:3300/0 conn(0x7f00b00d1870 0x7f00b01b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:23.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:23.367 7f00b52cd700 1 -- 172.21.2.112:0/46087165 >> v2:172.21.2.120:3300/0 conn(0x7f00b00d1870 msgr2=0x7f00b01b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:50:23.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:23.367 7f00b52cd700 1 --2- 172.21.2.112:0/46087165 >> v2:172.21.2.120:3300/0 conn(0x7f00b00d1870 0x7f00b01b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:23.389 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:23.367 7f00b52cd700 1 -- 172.21.2.112:0/46087165 >> v1:172.21.2.120:6789/0 conn(0x7f00b00d1c70 legacy=0x7f00b00cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:23.389 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:23.367 7f00b52cd700 1 -- 172.21.2.112:0/46087165 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00b00f0870 con 0x7f00b01b0a10 2021-02-10T11:50:23.389 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:23.367 7f00b52cd700 1 --2- 172.21.2.112:0/46087165 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f00b01b0a10 0x7f00b00cf330 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 rx=0x7f00a4006650 tx=0x7f00a4009a80).ready entity=mon.2 client_cookie=eaa32dee53afd39c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:23.389 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:23.367 7f00b4acc700 1 -- 172.21.2.112:0/46087165 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f00a40009b0 con 0x7f00b01b0a10 2021-02-10T11:50:23.389 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:23.367 7f00b4acc700 1 -- 172.21.2.112:0/46087165 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00a400c0c0 con 0x7f00b01b0a10 2021-02-10T11:50:23.389 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:23.367 7f00b4acc700 1 -- 172.21.2.112:0/46087165 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f00a400c5e0 con 0x7f00b01b0a10 2021-02-10T11:50:23.389 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:23.368 7f00b7cb2700 1 -- 172.21.2.112:0/46087165 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f00b01b0a10 msgr2=0x7f00b00cf330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:23.389 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:23.368 7f00b7cb2700 1 --2- 172.21.2.112:0/46087165 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f00b01b0a10 0x7f00b00cf330 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 rx=0x7f00a4006650 tx=0x7f00a4009a80).stop 2021-02-10T11:50:23.389 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:23.368 7f00b7cb2700 1 -- 172.21.2.112:0/46087165 shutdown_connections 2021-02-10T11:50:23.389 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:23.368 7f00b7cb2700 1 --2- 172.21.2.112:0/46087165 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f00b01b0a10 0x7f00b00cf330 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:23.389 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:23.368 7f00b7cb2700 1 --2- 172.21.2.112:0/46087165 >> v2:172.21.2.120:3300/0 conn(0x7f00b00d1870 0x7f00b01b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:23.389 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:23.368 7f00b7cb2700 1 -- 172.21.2.112:0/46087165 >> 172.21.2.112:0/46087165 conn(0x7f00b00ce450 msgr2=0x7f00b00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:23.389 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:23.368 7f00b7cb2700 1 -- 172.21.2.112:0/46087165 shutdown_connections 2021-02-10T11:50:23.390 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:23.368 7f00b7cb2700 1 -- 172.21.2.112:0/46087165 wait complete. 2021-02-10T11:50:23.391 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:23.382 7f00b7cb2700 1 Processor -- start 2021-02-10T11:50:23.391 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:23.383 7f00b7cb2700 1 -- start start 2021-02-10T11:50:23.391 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:23.383 7f00b7cb2700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f00b0093f10 0x7f00b0094310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:23.391 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:23.383 7f00b7cb2700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f00b00ae9e0 0x7f00b00aede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:23.391 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:23.383 7f00b7cb2700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f00b0092d90 0x7f00b0093190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:23.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:23.383 7f00b7cb2700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00b019ebc0 con 0x7f00b0093f10 2021-02-10T11:50:23.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:23.383 7f00b7cb2700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00b01b8760 con 0x7f00b00ae9e0 2021-02-10T11:50:23.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:23.383 7f00b7cb2700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00b00937b0 con 0x7f00b0092d90 2021-02-10T11:50:23.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:23.383 7f00b5ace700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f00b0093f10 0x7f00b0094310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:23.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:23.383 7f00b5ace700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f00b0093f10 0x7f00b0094310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:33956/0 (socket says 172.21.2.112:33956) 2021-02-10T11:50:23.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:23.383 7f00b5ace700 1 -- 172.21.2.112:0/769728490 learned_addr learned my addr 172.21.2.112:0/769728490 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:50:23.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:23.383 7f00b52cd700 1 --2- 172.21.2.112:0/769728490 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f00b00ae9e0 0x7f00b00aede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:23.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:23.383 7f00b62cf700 1 --2- 172.21.2.112:0/769728490 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f00b0092d90 0x7f00b0093190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:23.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:23.384 7f00b5ace700 1 -- 172.21.2.112:0/769728490 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f00b0092d90 msgr2=0x7f00b0093190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:23.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:23.384 7f00b5ace700 1 --2- 172.21.2.112:0/769728490 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f00b0092d90 0x7f00b0093190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:23.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:23.384 7f00b5ace700 1 -- 172.21.2.112:0/769728490 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f00b00ae9e0 msgr2=0x7f00b00aede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:23.392 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:23.384 7f00b5ace700 1 --2- 172.21.2.112:0/769728490 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f00b00ae9e0 0x7f00b00aede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:23.393 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:23.384 7f00b5ace700 1 -- 172.21.2.112:0/769728490 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00b00939d0 con 0x7f00b0093f10 2021-02-10T11:50:23.393 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:23.384 7f00b5ace700 1 --2- 172.21.2.112:0/769728490 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f00b0093f10 0x7f00b0094310 secure :-1 s=READY pgs=3322 cs=0 l=1 rev1=1 rx=0x7f00ac006f60 tx=0x7f00ac00b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:23.393 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:23.384 7f00a27fc700 1 -- 172.21.2.112:0/769728490 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f00ac0009b0 con 0x7f00b0093f10 2021-02-10T11:50:23.393 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:23.384 7f00a27fc700 1 -- 172.21.2.112:0/769728490 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00ac00bdf0 con 0x7f00b0093f10 2021-02-10T11:50:23.393 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:23.384 7f00a27fc700 1 -- 172.21.2.112:0/769728490 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f00ac002030 con 0x7f00b0093f10 2021-02-10T11:50:23.393 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:23.384 7f00b7cb2700 1 -- 172.21.2.112:0/769728490 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00b00939d0 con 0x7f00b0093f10 2021-02-10T11:50:23.393 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:23.384 7f00b7cb2700 1 -- 172.21.2.112:0/769728490 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00b00939d0 con 0x7f00b0093f10 2021-02-10T11:50:23.393 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:23.385 7f00a27fc700 1 -- 172.21.2.112:0/769728490 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f00ac006b00 con 0x7f00b0093f10 2021-02-10T11:50:23.393 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:23.385 7f00a27fc700 1 --2- 172.21.2.112:0/769728490 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f00980186c0 0x7f009801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:23.393 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:23.385 7f00a27fc700 1 -- 172.21.2.112:0/769728490 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f00ac032400 con 0x7f00b0093f10 2021-02-10T11:50:23.393 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:23.385 7f00b52cd700 1 --2- 172.21.2.112:0/769728490 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f00980186c0 0x7f009801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:23.393 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:23.386 7f00b52cd700 1 --2- 172.21.2.112:0/769728490 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f00980186c0 0x7f009801abc0 crc :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:23.421 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:23.416 7f00b7cb2700 1 -- 172.21.2.112:0/769728490 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00b00ebc60 con 0x7f00b0093f10 2021-02-10T11:50:23.423 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:23.418 7f00a27fc700 1 -- 172.21.2.112:0/769728490 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f00ac006340 con 0x7f00b0093f10 2021-02-10T11:50:23.554 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:23.549 7f00b7cb2700 1 -- 172.21.2.112:0/769728490 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_subvolume_clone_in_progress_getpath"]} v 0) v1 -- 0x7f00b00ebc60 con 0x7f00b0093f10 2021-02-10T11:50:24.333 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:24.328 7f00a27fc700 1 -- 172.21.2.112:0/769728490 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_subvolume_clone_in_progress_getpath"]}]=0 v931) v1 ==== 157+0+0 (secure 0 0 0) 0x7f00ac006b00 con 0x7f00b0093f10 2021-02-10T11:50:24.376 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:24.371 7f00b7cb2700 1 -- 172.21.2.112:0/769728490 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f00980186c0 msgr2=0x7f009801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:24.376 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:24.371 7f00b7cb2700 1 --2- 172.21.2.112:0/769728490 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f00980186c0 0x7f009801abc0 crc :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:24.376 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:24.371 7f00b7cb2700 1 -- 172.21.2.112:0/769728490 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f00b0093f10 msgr2=0x7f00b0094310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:24.376 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:24.371 7f00b7cb2700 1 --2- 172.21.2.112:0/769728490 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f00b0093f10 0x7f00b0094310 secure :-1 s=READY pgs=3322 cs=0 l=1 rev1=1 rx=0x7f00ac006f60 tx=0x7f00ac00b970).stop 2021-02-10T11:50:24.376 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:24.371 7f00b7cb2700 1 -- 172.21.2.112:0/769728490 shutdown_connections 2021-02-10T11:50:24.376 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:24.371 7f00b7cb2700 1 --2- 172.21.2.112:0/769728490 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f00980186c0 0x7f009801abc0 unknown :-1 s=CLOSED pgs=1495 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:24.376 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:24.371 7f00b7cb2700 1 --2- 172.21.2.112:0/769728490 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f00b00ae9e0 0x7f00b00aede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:24.377 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:24.371 7f00b7cb2700 1 --2- 172.21.2.112:0/769728490 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f00b0092d90 0x7f00b0093190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:24.377 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:24.371 7f00b7cb2700 1 --2- 172.21.2.112:0/769728490 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f00b0093f10 0x7f00b0094310 unknown :-1 s=CLOSED pgs=3322 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:24.377 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:24.371 7f00b7cb2700 1 -- 172.21.2.112:0/769728490 >> 172.21.2.112:0/769728490 conn(0x7f00b019f8a0 msgr2=0x7f00b0094980 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:24.377 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:24.371 7f00b7cb2700 1 -- 172.21.2.112:0/769728490 shutdown_connections 2021-02-10T11:50:24.377 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:24.371 7f00b7cb2700 1 -- 172.21.2.112:0/769728490 wait complete. 2021-02-10T11:50:24.386 INFO:tasks.cephfs_test_runner:test_subvolume_clone_in_progress_getpath (tasks.cephfs.test_volumes.TestVolumes) ... ERROR 2021-02-10T11:50:24.386 INFO:tasks.cephfs_test_runner: 2021-02-10T11:50:24.386 INFO:tasks.cephfs_test_runner:====================================================================== 2021-02-10T11:50:24.387 INFO:tasks.cephfs_test_runner:ERROR: test_subvolume_clone_in_progress_getpath (tasks.cephfs.test_volumes.TestVolumes) 2021-02-10T11:50:24.387 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2021-02-10T11:50:24.387 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2021-02-10T11:50:24.387 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph-c_962fac13b1a781b8c7d25c1eed50d93813b8f5f0/qa/tasks/cephfs/test_volumes.py", line 4052, in test_subvolume_clone_in_progress_getpath 2021-02-10T11:50:24.387 INFO:tasks.cephfs_test_runner: raise RuntimeError("expected fetching path of an pending clone to fail") 2021-02-10T11:50:24.387 INFO:tasks.cephfs_test_runner:RuntimeError: expected fetching path of an pending clone to fail 2021-02-10T11:50:24.387 INFO:tasks.cephfs_test_runner: 2021-02-10T11:50:24.387 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2021-02-10T11:50:24.387 INFO:tasks.cephfs_test_runner:Ran 21 tests in 2116.992s 2021-02-10T11:50:24.387 INFO:tasks.cephfs_test_runner: 2021-02-10T11:50:24.387 INFO:tasks.cephfs_test_runner:FAILED (errors=1, skipped=1) 2021-02-10T11:50:24.387 INFO:tasks.cephfs_test_runner: 2021-02-10T11:50:24.387 INFO:tasks.cephfs_test_runner:====================================================================== 2021-02-10T11:50:24.387 INFO:tasks.cephfs_test_runner:ERROR: test_subvolume_clone_in_progress_getpath (tasks.cephfs.test_volumes.TestVolumes) 2021-02-10T11:50:24.388 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2021-02-10T11:50:24.388 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2021-02-10T11:50:24.388 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph-c_962fac13b1a781b8c7d25c1eed50d93813b8f5f0/qa/tasks/cephfs/test_volumes.py", line 4052, in test_subvolume_clone_in_progress_getpath 2021-02-10T11:50:24.388 INFO:tasks.cephfs_test_runner: raise RuntimeError("expected fetching path of an pending clone to fail") 2021-02-10T11:50:24.388 INFO:tasks.cephfs_test_runner:RuntimeError: expected fetching path of an pending clone to fail 2021-02-10T11:50:24.388 INFO:tasks.cephfs_test_runner: 2021-02-10T11:50:24.388 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/run_tasks.py", line 94, in run_tasks manager.__enter__() File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_962fac13b1a781b8c7d25c1eed50d93813b8f5f0/qa/tasks/cephfs_test_runner.py", line 207, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_subvolume_clone_in_progress_getpath (tasks.cephfs.test_volumes.TestVolumes) 2021-02-10T11:50:24.671 ERROR:teuthology.run_tasks: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=9b39ed5cb94d4706938f1211e2cb43b9 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/run_tasks.py", line 94, in run_tasks manager.__enter__() File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_962fac13b1a781b8c7d25c1eed50d93813b8f5f0/qa/tasks/cephfs_test_runner.py", line 207, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_subvolume_clone_in_progress_getpath (tasks.cephfs.test_volumes.TestVolumes) 2021-02-10T11:50:24.672 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner 2021-02-10T11:50:24.681 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2021-02-10T11:50:24.706 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2021-02-10T11:50:24.706 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:50:24.707 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:50:24.707 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:50:25.061 INFO:teuthology.orchestra.run.gibba012.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.0’: No such file or directory 2021-02-10T11:50:25.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:50:25.061 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2021-02-10T11:50:25.062 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:50:25.062 DEBUG:teuthology.orchestra.run.gibba012:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:50:25.062 DEBUG:teuthology.orchestra.run.gibba012:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:50:25.398 INFO:teuthology.orchestra.run.gibba012.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: No such file or directory 2021-02-10T11:50:25.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:50:25.399 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2021-02-10T11:50:25.399 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:50:25.399 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:50:25.400 DEBUG:teuthology.orchestra.run.gibba020:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:50:25.735 INFO:teuthology.orchestra.run.gibba020.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:50:25.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:50:25.735 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:50:25.735 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:50:25.736 DEBUG:teuthology.orchestra.run.gibba020:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:50:25.736 DEBUG:teuthology.orchestra.run.gibba020:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:50:26.070 INFO:teuthology.orchestra.run.gibba020.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:50:26.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:50:26.070 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:50:26.071 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2021-02-10T11:50:26.090 INFO:tasks.ceph.ceph_manager.ceph:Canceling any pending splits or merges... 2021-02-10T11:50:26.091 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:50:26.520 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:26.499 7f76b7bd4700 1 Processor -- start 2021-02-10T11:50:26.520 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:26.499 7f76b7bd4700 1 -- start start 2021-02-10T11:50:26.520 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:26.500 7f76b7bd4700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f76b00d1870 0x7f76b01b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:26.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:26.500 7f76b7bd4700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f76b01b0a10 0x7f76b00cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:26.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:26.500 7f76b7bd4700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76b00f0b10 con 0x7f76b00d1c70 2021-02-10T11:50:26.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:26.500 7f76b7bd4700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76b00e5ed0 con 0x7f76b01b0a10 2021-02-10T11:50:26.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:26.500 7f76b7bd4700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76b00f0650 con 0x7f76b00d1870 2021-02-10T11:50:26.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:26.500 7f76b51ef700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f76b01b0a10 0x7f76b00cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:26.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:26.500 7f76b61f1700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f76b00d1c70 0x7f76b00cf950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:33424/0 (socket says 172.21.2.112:33424) 2021-02-10T11:50:26.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:26.500 7f76b61f1700 1 -- 172.21.2.112:0/2161727894 learned_addr learned my addr 172.21.2.112:0/2161727894 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:50:26.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:26.500 7f76b51ef700 1 -- 172.21.2.112:0/2161727894 >> v2:172.21.2.120:3300/0 conn(0x7f76b00d1870 msgr2=0x7f76b01b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:50:26.521 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:26.500 7f76b51ef700 1 --2- 172.21.2.112:0/2161727894 >> v2:172.21.2.120:3300/0 conn(0x7f76b00d1870 0x7f76b01b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:26.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:26.500 7f76b51ef700 1 -- 172.21.2.112:0/2161727894 >> v1:172.21.2.112:6789/0 conn(0x7f76b00d1c70 legacy=0x7f76b00cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:26.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:26.500 7f76b49ee700 1 -- 172.21.2.112:0/2161727894 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1651552430 0 0) 0x7f76a8001500 con 0x7f76b00d1c70 2021-02-10T11:50:26.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:26.500 7f76b51ef700 1 -- 172.21.2.112:0/2161727894 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76b00f0840 con 0x7f76b01b0a10 2021-02-10T11:50:26.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:26.500 7f76b51ef700 1 --2- 172.21.2.112:0/2161727894 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f76b01b0a10 0x7f76b00cf330 secure :-1 s=READY pgs=3324 cs=0 l=1 rev1=1 rx=0x7f76a0006650 tx=0x7f76a0009a80).ready entity=mon.0 client_cookie=e7abf1b7f46a0f4 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:26.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:26.500 7f76b49ee700 1 -- 172.21.2.112:0/2161727894 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f76a000c0c0 con 0x7f76b01b0a10 2021-02-10T11:50:26.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:26.501 7f76b49ee700 1 -- 172.21.2.112:0/2161727894 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76a000c340 con 0x7f76b01b0a10 2021-02-10T11:50:26.522 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:26.501 7f76b49ee700 1 -- 172.21.2.112:0/2161727894 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f76a000c860 con 0x7f76b01b0a10 2021-02-10T11:50:26.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:26.501 7f76b7bd4700 1 -- 172.21.2.112:0/2161727894 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f76b01b0a10 msgr2=0x7f76b00cf330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:26.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:26.501 7f76b7bd4700 1 --2- 172.21.2.112:0/2161727894 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f76b01b0a10 0x7f76b00cf330 secure :-1 s=READY pgs=3324 cs=0 l=1 rev1=1 rx=0x7f76a0006650 tx=0x7f76a0009a80).stop 2021-02-10T11:50:26.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:26.501 7f76b7bd4700 1 -- 172.21.2.112:0/2161727894 shutdown_connections 2021-02-10T11:50:26.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:26.501 7f76b7bd4700 1 --2- 172.21.2.112:0/2161727894 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f76b01b0a10 0x7f76b00cf330 unknown :-1 s=CLOSED pgs=3324 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:26.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:26.501 7f76b7bd4700 1 --2- 172.21.2.112:0/2161727894 >> v2:172.21.2.120:3300/0 conn(0x7f76b00d1870 0x7f76b01b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:26.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:26.501 7f76b7bd4700 1 -- 172.21.2.112:0/2161727894 >> 172.21.2.112:0/2161727894 conn(0x7f76b00ce450 msgr2=0x7f76b00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:26.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:26.501 7f76b7bd4700 1 -- 172.21.2.112:0/2161727894 shutdown_connections 2021-02-10T11:50:26.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:26.501 7f76b7bd4700 1 -- 172.21.2.112:0/2161727894 wait complete. 2021-02-10T11:50:26.523 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:26.516 7f76b7bd4700 1 Processor -- start 2021-02-10T11:50:26.524 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:26.516 7f76b7bd4700 1 -- start start 2021-02-10T11:50:26.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:26.516 7f76b7bd4700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f76b01b7cf0 0x7f76b019cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:26.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:26.516 7f76b7bd4700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f76b01b80f0 0x7f76b01b84f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:26.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:26.516 7f76b7bd4700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f76b019da20 0x7f76b019de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:26.525 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:26.516 7f76b7bd4700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76b00a77c0 con 0x7f76b019da20 2021-02-10T11:50:26.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:26.516 7f76b7bd4700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76b019d4a0 con 0x7f76b01b80f0 2021-02-10T11:50:26.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:26.516 7f76b7bd4700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76b00954f0 con 0x7f76b01b7cf0 2021-02-10T11:50:26.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:26.516 7f76b61f1700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f76b019da20 0x7f76b019de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:26.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:26.516 7f76b61f1700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f76b019da20 0x7f76b019de20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:33968/0 (socket says 172.21.2.112:33968) 2021-02-10T11:50:26.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:26.516 7f76b61f1700 1 -- 172.21.2.112:0/618061716 learned_addr learned my addr 172.21.2.112:0/618061716 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:50:26.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:26.517 7f76b61f1700 1 -- 172.21.2.112:0/618061716 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f76b01b7cf0 msgr2=0x7f76b019cdb0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:50:26.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:26.517 7f76b61f1700 1 --2- 172.21.2.112:0/618061716 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f76b01b7cf0 0x7f76b019cdb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:26.526 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:26.517 7f76b61f1700 1 -- 172.21.2.112:0/618061716 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f76b01b80f0 msgr2=0x7f76b01b84f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:26.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:26.517 7f76b61f1700 1 --2- 172.21.2.112:0/618061716 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f76b01b80f0 0x7f76b01b84f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:26.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:26.517 7f76b61f1700 1 -- 172.21.2.112:0/618061716 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76b01b8d60 con 0x7f76b019da20 2021-02-10T11:50:26.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:26.517 7f76b61f1700 1 --2- 172.21.2.112:0/618061716 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f76b019da20 0x7f76b019de20 secure :-1 s=READY pgs=3325 cs=0 l=1 rev1=1 rx=0x7f76a8001c10 tx=0x7f76a800ac60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:26.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:26.517 7f76a67fc700 1 -- 172.21.2.112:0/618061716 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f76a8000f90 con 0x7f76b019da20 2021-02-10T11:50:26.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:26.517 7f76a67fc700 1 -- 172.21.2.112:0/618061716 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76a8007020 con 0x7f76b019da20 2021-02-10T11:50:26.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:26.517 7f76a67fc700 1 -- 172.21.2.112:0/618061716 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f76a8007540 con 0x7f76b019da20 2021-02-10T11:50:26.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:26.517 7f76b7bd4700 1 -- 172.21.2.112:0/618061716 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76b01b8d60 con 0x7f76b019da20 2021-02-10T11:50:26.527 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:26.518 7f76b7bd4700 1 -- 172.21.2.112:0/618061716 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76b01b8d60 con 0x7f76b019da20 2021-02-10T11:50:26.528 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:26.518 7f76a67fc700 1 -- 172.21.2.112:0/618061716 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f76a8007200 con 0x7f76b019da20 2021-02-10T11:50:26.528 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:26.518 7f76a67fc700 1 --2- 172.21.2.112:0/618061716 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f76980186c0 0x7f769801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:26.528 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:26.518 7f76a67fc700 1 -- 172.21.2.112:0/618061716 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f76a80328d0 con 0x7f76b019da20 2021-02-10T11:50:26.528 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:26.518 7f76b59f0700 1 --2- 172.21.2.112:0/618061716 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f76980186c0 0x7f769801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:26.528 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:26.519 7f76b59f0700 1 --2- 172.21.2.112:0/618061716 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f76980186c0 0x7f769801abc0 crc :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:26.554 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:26.550 7f76b7bd4700 1 -- 172.21.2.112:0/618061716 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76b00ebc00 con 0x7f76b019da20 2021-02-10T11:50:26.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:26.552 7f76a67fc700 1 -- 172.21.2.112:0/618061716 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f76a8000cc0 con 0x7f76b019da20 2021-02-10T11:50:26.681 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:26.677 7f76b7bd4700 1 -- 172.21.2.112:0/618061716 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f76b00f0b10 con 0x7f76b019da20 2021-02-10T11:50:26.682 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:26.678 7f76a67fc700 1 -- 172.21.2.112:0/618061716 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v198) v1 ==== 74+0+14007 (secure 0 0 0) 0x7f76a8007200 con 0x7f76b019da20 2021-02-10T11:50:26.683 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:50:26.683 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":198,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:49:05.481103","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":43,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":42,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:49:02.450970","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"197","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":43,"pool_name":"cephfs_data","create_time":"2021-02-10 11:49:03.473275","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"198","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":196,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":196,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":196,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":196,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":196,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:50:26.684 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:26.680 7f76b7bd4700 1 -- 172.21.2.112:0/618061716 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f76980186c0 msgr2=0x7f769801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:26.684 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:26.680 7f76b7bd4700 1 --2- 172.21.2.112:0/618061716 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f76980186c0 0x7f769801abc0 crc :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:26.684 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:26.680 7f76b7bd4700 1 -- 172.21.2.112:0/618061716 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f76b019da20 msgr2=0x7f76b019de20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:26.684 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:26.680 7f76b7bd4700 1 --2- 172.21.2.112:0/618061716 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f76b019da20 0x7f76b019de20 secure :-1 s=READY pgs=3325 cs=0 l=1 rev1=1 rx=0x7f76a8001c10 tx=0x7f76a800ac60).stop 2021-02-10T11:50:26.685 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:26.680 7f76b7bd4700 1 -- 172.21.2.112:0/618061716 shutdown_connections 2021-02-10T11:50:26.685 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:26.680 7f76b7bd4700 1 --2- 172.21.2.112:0/618061716 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f76980186c0 0x7f769801abc0 unknown :-1 s=CLOSED pgs=1496 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:26.685 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:26.680 7f76b7bd4700 1 --2- 172.21.2.112:0/618061716 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f76b01b80f0 0x7f76b01b84f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:26.685 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:26.680 7f76b7bd4700 1 --2- 172.21.2.112:0/618061716 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f76b019da20 0x7f76b019de20 unknown :-1 s=CLOSED pgs=3325 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:26.685 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:26.680 7f76b7bd4700 1 --2- 172.21.2.112:0/618061716 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f76b01b7cf0 0x7f76b019cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:26.685 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:26.680 7f76b7bd4700 1 -- 172.21.2.112:0/618061716 >> 172.21.2.112:0/618061716 conn(0x7f76b00ed810 msgr2=0x7f76b0094980 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:26.686 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:26.680 7f76b7bd4700 1 -- 172.21.2.112:0/618061716 shutdown_connections 2021-02-10T11:50:26.686 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:26.680 7f76b7bd4700 1 -- 172.21.2.112:0/618061716 wait complete. 2021-02-10T11:50:26.693 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph pg dump --format=json 2021-02-10T11:50:27.124 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.103 7f356b62a700 1 Processor -- start 2021-02-10T11:50:27.124 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.103 7f356b62a700 1 -- start start 2021-02-10T11:50:27.124 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.104 7f356b62a700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f35641b0470 0x7f35641b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:27.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.104 7f356b62a700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f35640ef610 0x7f35640efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:27.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.104 7f356b62a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f35640f0030 0x7f35640f0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:27.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.104 7f356b62a700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35640e5ea0 con 0x7f35640ef610 2021-02-10T11:50:27.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.104 7f356b62a700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35640f0b50 con 0x7f35641b0470 2021-02-10T11:50:27.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.104 7f356b62a700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35640e52e0 con 0x7f35640f0030 2021-02-10T11:50:27.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.104 7f3568c45700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f35640ef610 0x7f35640efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:27.125 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.104 7f3568c45700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f35640ef610 0x7f35640efa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:33974/0 (socket says 172.21.2.112:33974) 2021-02-10T11:50:27.126 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.104 7f3568c45700 1 -- 172.21.2.112:0/4031496411 learned_addr learned my addr 172.21.2.112:0/4031496411 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:50:27.126 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.104 7f3569c47700 1 --2- 172.21.2.112:0/4031496411 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f35640f0030 0x7f35640f0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:27.126 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.104 7f3569446700 1 --2- 172.21.2.112:0/4031496411 >> v2:172.21.2.120:3300/0 conn(0x7f35641b0470 0x7f35641b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:27.126 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.104 7f3568c45700 1 -- 172.21.2.112:0/4031496411 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f35640f0030 msgr2=0x7f35640f0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:50:27.126 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.104 7f3568c45700 1 --2- 172.21.2.112:0/4031496411 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f35640f0030 0x7f35640f0430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:27.126 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.104 7f3568c45700 1 -- 172.21.2.112:0/4031496411 >> v2:172.21.2.120:3300/0 conn(0x7f35641b0470 msgr2=0x7f35641b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:50:27.126 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.104 7f3568c45700 1 --2- 172.21.2.112:0/4031496411 >> v2:172.21.2.120:3300/0 conn(0x7f35641b0470 0x7f35641b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:27.126 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.104 7f3568c45700 1 -- 172.21.2.112:0/4031496411 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35640f10d0 con 0x7f35640ef610 2021-02-10T11:50:27.127 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.104 7f3568c45700 1 --2- 172.21.2.112:0/4031496411 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f35640ef610 0x7f35640efa10 secure :-1 s=READY pgs=3326 cs=0 l=1 rev1=1 rx=0x7f3554006700 tx=0x7f3554009b80).ready entity=mon.0 client_cookie=e640fdeb50bc653f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:27.127 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.105 7f355bfff700 1 -- 172.21.2.112:0/4031496411 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f355400c0c0 con 0x7f35640ef610 2021-02-10T11:50:27.127 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.105 7f355bfff700 1 -- 172.21.2.112:0/4031496411 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f355400c340 con 0x7f35640ef610 2021-02-10T11:50:27.127 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.105 7f355bfff700 1 -- 172.21.2.112:0/4031496411 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f355400c820 con 0x7f35640ef610 2021-02-10T11:50:27.127 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.105 7f356b62a700 1 -- 172.21.2.112:0/4031496411 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f35640ef610 msgr2=0x7f35640efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:27.127 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.105 7f356b62a700 1 --2- 172.21.2.112:0/4031496411 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f35640ef610 0x7f35640efa10 secure :-1 s=READY pgs=3326 cs=0 l=1 rev1=1 rx=0x7f3554006700 tx=0x7f3554009b80).stop 2021-02-10T11:50:27.127 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.105 7f356b62a700 1 -- 172.21.2.112:0/4031496411 shutdown_connections 2021-02-10T11:50:27.127 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.105 7f356b62a700 1 --2- 172.21.2.112:0/4031496411 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f35640f0030 0x7f35640f0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:27.127 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.105 7f356b62a700 1 --2- 172.21.2.112:0/4031496411 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f35640ef610 0x7f35640efa10 unknown :-1 s=CLOSED pgs=3326 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:27.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.105 7f356b62a700 1 --2- 172.21.2.112:0/4031496411 >> v2:172.21.2.120:3300/0 conn(0x7f35641b0470 0x7f35641b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:27.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.105 7f356b62a700 1 -- 172.21.2.112:0/4031496411 >> 172.21.2.112:0/4031496411 conn(0x7f35640ce3e0 msgr2=0x7f35640e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:27.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.106 7f356b62a700 1 -- 172.21.2.112:0/4031496411 shutdown_connections 2021-02-10T11:50:27.128 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.106 7f356b62a700 1 -- 172.21.2.112:0/4031496411 wait complete. 2021-02-10T11:50:27.129 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.120 7f356b62a700 1 Processor -- start 2021-02-10T11:50:27.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.120 7f356b62a700 1 -- start start 2021-02-10T11:50:27.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.121 7f356b62a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f35640bba80 0x7f3564108e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:27.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.121 7f356b62a700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f35640bbe80 0x7f35640bd5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:27.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.121 7f356b62a700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f35641094b0 0x7f35640bdbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:27.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.121 7f356b62a700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35641089f0 con 0x7f35640bba80 2021-02-10T11:50:27.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.121 7f356b62a700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35640c2ec0 con 0x7f35640bbe80 2021-02-10T11:50:27.130 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.121 7f356b62a700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35640c30e0 con 0x7f35641094b0 2021-02-10T11:50:27.131 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.121 7f3569446700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f35640bba80 0x7f3564108e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:27.131 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.121 7f3569446700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f35640bba80 0x7f3564108e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:33978/0 (socket says 172.21.2.112:33978) 2021-02-10T11:50:27.131 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.121 7f3569446700 1 -- 172.21.2.112:0/2636455518 learned_addr learned my addr 172.21.2.112:0/2636455518 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:50:27.131 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.121 7f3569446700 1 -- 172.21.2.112:0/2636455518 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f35641094b0 msgr2=0x7f35640bdbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:27.131 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.121 7f3569446700 1 --2- 172.21.2.112:0/2636455518 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f35641094b0 0x7f35640bdbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:27.131 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.121 7f3569446700 1 -- 172.21.2.112:0/2636455518 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f35640bbe80 msgr2=0x7f35640bd5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:27.131 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.121 7f3569446700 1 --2- 172.21.2.112:0/2636455518 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f35640bbe80 0x7f35640bd5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:27.131 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.121 7f3569446700 1 -- 172.21.2.112:0/2636455518 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35640be200 con 0x7f35640bba80 2021-02-10T11:50:27.132 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.121 7f3569446700 1 --2- 172.21.2.112:0/2636455518 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f35640bba80 0x7f3564108e90 secure :-1 s=READY pgs=3327 cs=0 l=1 rev1=1 rx=0x7f3560006f00 tx=0x7f356000b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:27.132 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.122 7f3559ffb700 1 -- 172.21.2.112:0/2636455518 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f356000bd30 con 0x7f35640bba80 2021-02-10T11:50:27.132 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.122 7f3559ffb700 1 -- 172.21.2.112:0/2636455518 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3560002030 con 0x7f35640bba80 2021-02-10T11:50:27.132 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.122 7f356b62a700 1 -- 172.21.2.112:0/2636455518 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35640be200 con 0x7f35640bba80 2021-02-10T11:50:27.132 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.122 7f3559ffb700 1 -- 172.21.2.112:0/2636455518 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3560002550 con 0x7f35640bba80 2021-02-10T11:50:27.132 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.122 7f356b62a700 1 -- 172.21.2.112:0/2636455518 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35640be200 con 0x7f35640bba80 2021-02-10T11:50:27.132 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.122 7f3559ffb700 1 -- 172.21.2.112:0/2636455518 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3560000c90 con 0x7f35640bba80 2021-02-10T11:50:27.132 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.122 7f3559ffb700 1 --2- 172.21.2.112:0/2636455518 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f35440186c0 0x7f354401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:27.133 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.122 7f3559ffb700 1 -- 172.21.2.112:0/2636455518 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f3560032840 con 0x7f35640bba80 2021-02-10T11:50:27.133 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.123 7f3568c45700 1 --2- 172.21.2.112:0/2636455518 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f35440186c0 0x7f354401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:27.133 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.123 7f3568c45700 1 --2- 172.21.2.112:0/2636455518 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f35440186c0 0x7f354401abc0 crc :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:27.158 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.153 7f356b62a700 1 -- 172.21.2.112:0/2636455518 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35640f0b50 con 0x7f35640bba80 2021-02-10T11:50:27.160 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.156 7f3559ffb700 1 -- 172.21.2.112:0/2636455518 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f356000bd30 con 0x7f35640bba80 2021-02-10T11:50:27.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.289 7f356b62a700 1 -- 172.21.2.112:0/2636455518 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "pg dump", "target": ["mgr", ""], "format": "json"}) v1 -- 0x7f35640f0b50 con 0x7f35440186c0 2021-02-10T11:50:27.297 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.293 7f3559ffb700 1 -- 172.21.2.112:0/2636455518 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 dumped all) v1 ==== 18+0+101101 (crc 0 0 0) 0x7f3554000960 con 0x7f35440186c0 2021-02-10T11:50:27.301 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:50:27.302 INFO:teuthology.orchestra.run.gibba012.stderr:dumped all 2021-02-10T11:50:27.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.298 7f356b62a700 1 -- 172.21.2.112:0/2636455518 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f35440186c0 msgr2=0x7f354401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:27.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.298 7f356b62a700 1 --2- 172.21.2.112:0/2636455518 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f35440186c0 0x7f354401abc0 crc :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:27.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.298 7f356b62a700 1 -- 172.21.2.112:0/2636455518 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f35640bba80 msgr2=0x7f3564108e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:27.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.298 7f356b62a700 1 --2- 172.21.2.112:0/2636455518 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f35640bba80 0x7f3564108e90 secure :-1 s=READY pgs=3327 cs=0 l=1 rev1=1 rx=0x7f3560006f00 tx=0x7f356000b940).stop 2021-02-10T11:50:27.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.299 7f356b62a700 1 -- 172.21.2.112:0/2636455518 shutdown_connections 2021-02-10T11:50:27.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.299 7f356b62a700 1 --2- 172.21.2.112:0/2636455518 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f35440186c0 0x7f354401abc0 unknown :-1 s=CLOSED pgs=1497 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:27.304 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.299 7f356b62a700 1 --2- 172.21.2.112:0/2636455518 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f35640bbe80 0x7f35640bd5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:27.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.299 7f356b62a700 1 --2- 172.21.2.112:0/2636455518 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f35641094b0 0x7f35640bdbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:27.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.299 7f356b62a700 1 --2- 172.21.2.112:0/2636455518 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f35640bba80 0x7f3564108e90 unknown :-1 s=CLOSED pgs=3327 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:27.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.299 7f356b62a700 1 -- 172.21.2.112:0/2636455518 >> 172.21.2.112:0/2636455518 conn(0x7f35640bc6c0 msgr2=0x7f35640b1f70 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:27.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.299 7f356b62a700 1 -- 172.21.2.112:0/2636455518 shutdown_connections 2021-02-10T11:50:27.305 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.299 7f356b62a700 1 -- 172.21.2.112:0/2636455518 wait complete. 2021-02-10T11:50:27.311 INFO:teuthology.orchestra.run.gibba012.stdout:{"pg_ready":true,"pg_map":{"version":1298,"stamp":"2021-02-10 11:50:26.271383","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":134996368,"num_objects":154,"num_object_clones":1,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":154,"num_whiteouts":0,"num_read":65,"num_read_kb":65537,"num_write":422,"num_write_kb":131927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":407,"ondisk_log_size":407,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"kb":754974720,"kb_used":8672404,"kb_used_data":283720,"kb_used_omap":0,"kb_used_meta":8388608,"kb_avail":746302316,"statfs":{"total":773094113280,"available":764213571584,"internally_reserved":8589934592,"allocated":290529280,"data_stored":278796512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8589934592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":70640950,"num_objects":70,"num_object_clones":1,"num_object_copies":140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70,"num_whiteouts":0,"num_read":65,"num_read_kb":65537,"num_write":192,"num_write_kb":69027,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002380"},"pg_stats":[{"pgid":"43.6","version":"198'22","reported_seq":"40","reported_epoch":"198","state":"active+clean","last_fresh":"2021-02-10 11:50:17.478190","last_change":"2021-02-10 11:49:04.486122","last_active":"2021-02-10 11:50:17.478190","last_peered":"2021-02-10 11:50:17.478190","last_clean":"2021-02-10 11:50:17.478190","last_became_active":"2021-02-10 11:49:04.485856","last_became_peered":"2021-02-10 11:49:04.485856","last_unstale":"2021-02-10 11:50:17.478190","last_undegraded":"2021-02-10 11:50:17.478190","last_fullsized":"2021-02-10 11:50:17.478190","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:49:03.842471","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:49:03.842471","last_clean_scrub_stamp":"2021-02-10 11:49:03.842471","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":23068672,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":9,"num_read_kb":9216,"num_write":22,"num_write_kb":22528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.7","version":"198'8","reported_seq":"19","reported_epoch":"198","state":"active+clean","last_fresh":"2021-02-10 11:49:05.908221","last_change":"2021-02-10 11:49:03.849288","last_active":"2021-02-10 11:49:05.908221","last_peered":"2021-02-10 11:49:05.908221","last_clean":"2021-02-10 11:49:05.908221","last_became_active":"2021-02-10 11:49:03.849100","last_became_peered":"2021-02-10 11:49:03.849100","last_unstale":"2021-02-10 11:49:05.908221","last_undegraded":"2021-02-10 11:49:05.908221","last_fullsized":"2021-02-10 11:49:05.908221","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:49:02.834883","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:49:02.834883","last_clean_scrub_stamp":"2021-02-10 11:49:02.834883","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":566,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"43.7","version":"198'14","reported_seq":"30","reported_epoch":"198","state":"active+clean","last_fresh":"2021-02-10 11:50:17.482787","last_change":"2021-02-10 11:49:04.486203","last_active":"2021-02-10 11:50:17.482787","last_peered":"2021-02-10 11:50:17.482787","last_clean":"2021-02-10 11:50:17.482787","last_became_active":"2021-02-10 11:49:04.485927","last_became_peered":"2021-02-10 11:49:04.485927","last_unstale":"2021-02-10 11:50:17.482787","last_undegraded":"2021-02-10 11:50:17.482787","last_fullsized":"2021-02-10 11:50:17.482787","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:49:03.842471","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:49:03.842471","last_clean_scrub_stamp":"2021-02-10 11:49:03.842471","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14680064,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":7,"num_read_kb":7168,"num_write":14,"num_write_kb":14336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.6","version":"198'16","reported_seq":"27","reported_epoch":"198","state":"active+clean","last_fresh":"2021-02-10 11:50:18.838786","last_change":"2021-02-10 11:49:03.848744","last_active":"2021-02-10 11:50:18.838786","last_peered":"2021-02-10 11:50:18.838786","last_clean":"2021-02-10 11:50:18.838786","last_became_active":"2021-02-10 11:49:03.848469","last_became_peered":"2021-02-10 11:49:03.848469","last_unstale":"2021-02-10 11:50:18.838786","last_undegraded":"2021-02-10 11:50:18.838786","last_fullsized":"2021-02-10 11:50:18.838786","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:49:02.834883","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:49:02.834883","last_clean_scrub_stamp":"2021-02-10 11:49:02.834883","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":35,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"43.5","version":"198'23","reported_seq":"41","reported_epoch":"198","state":"active+clean","last_fresh":"2021-02-10 11:50:17.521238","last_change":"2021-02-10 11:49:04.485844","last_active":"2021-02-10 11:50:17.521238","last_peered":"2021-02-10 11:50:17.521238","last_clean":"2021-02-10 11:50:17.521238","last_became_active":"2021-02-10 11:49:04.485608","last_became_peered":"2021-02-10 11:49:04.485608","last_unstale":"2021-02-10 11:50:17.521238","last_undegraded":"2021-02-10 11:50:17.521238","last_fullsized":"2021-02-10 11:50:17.521238","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:49:03.842471","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:49:03.842471","last_clean_scrub_stamp":"2021-02-10 11:49:03.842471","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16777506,"num_objects":18,"num_object_clones":1,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":10,"num_read_kb":10240,"num_write":20,"num_write_kb":16387,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"42.4","version":"198'213","reported_seq":"224","reported_epoch":"198","state":"active+clean","last_fresh":"2021-02-10 11:50:18.943802","last_change":"2021-02-10 11:49:03.848271","last_active":"2021-02-10 11:50:18.943802","last_peered":"2021-02-10 11:50:18.943802","last_clean":"2021-02-10 11:50:18.943802","last_became_active":"2021-02-10 11:49:03.848058","last_became_peered":"2021-02-10 11:49:03.848058","last_unstale":"2021-02-10 11:50:18.943802","last_undegraded":"2021-02-10 11:50:18.943802","last_fullsized":"2021-02-10 11:50:18.943802","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:49:02.834883","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:49:02.834883","last_clean_scrub_stamp":"2021-02-10 11:49:02.834883","log_size":213,"ondisk_log_size":213,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":777352,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":213,"num_write_kb":821,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"43.2","version":"198'20","reported_seq":"37","reported_epoch":"198","state":"active+clean","last_fresh":"2021-02-10 11:50:17.460794","last_change":"2021-02-10 11:49:04.486157","last_active":"2021-02-10 11:50:17.460794","last_peered":"2021-02-10 11:50:17.460794","last_clean":"2021-02-10 11:50:17.460794","last_became_active":"2021-02-10 11:49:04.485916","last_became_peered":"2021-02-10 11:49:04.485916","last_unstale":"2021-02-10 11:50:17.460794","last_undegraded":"2021-02-10 11:50:17.460794","last_fullsized":"2021-02-10 11:50:17.460794","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:49:03.842471","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:49:03.842471","last_clean_scrub_stamp":"2021-02-10 11:49:03.842471","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20971520,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":8,"num_read_kb":8192,"num_write":20,"num_write_kb":20480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.3","version":"198'6","reported_seq":"17","reported_epoch":"198","state":"active+clean","last_fresh":"2021-02-10 11:49:05.486065","last_change":"2021-02-10 11:49:03.848638","last_active":"2021-02-10 11:49:05.486065","last_peered":"2021-02-10 11:49:05.486065","last_clean":"2021-02-10 11:49:05.486065","last_became_active":"2021-02-10 11:49:03.848441","last_became_peered":"2021-02-10 11:49:03.848441","last_unstale":"2021-02-10 11:49:05.486065","last_undegraded":"2021-02-10 11:49:05.486065","last_fullsized":"2021-02-10 11:49:05.486065","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:49:02.834883","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:49:02.834883","last_clean_scrub_stamp":"2021-02-10 11:49:02.834883","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"43.3","version":"198'25","reported_seq":"42","reported_epoch":"198","state":"active+clean","last_fresh":"2021-02-10 11:50:17.526323","last_change":"2021-02-10 11:49:04.486243","last_active":"2021-02-10 11:50:17.526323","last_peered":"2021-02-10 11:50:17.526323","last_clean":"2021-02-10 11:50:17.526323","last_became_active":"2021-02-10 11:49:04.485844","last_became_peered":"2021-02-10 11:49:04.485844","last_unstale":"2021-02-10 11:50:17.526323","last_undegraded":"2021-02-10 11:50:17.526323","last_fullsized":"2021-02-10 11:50:17.526323","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:49:03.842471","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:49:03.842471","last_clean_scrub_stamp":"2021-02-10 11:49:03.842471","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18874505,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":8,"num_read_kb":8192,"num_write":25,"num_write_kb":18436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.2","version":"197'1","reported_seq":"12","reported_epoch":"197","state":"active+clean","last_fresh":"2021-02-10 11:49:04.488456","last_change":"2021-02-10 11:49:03.849353","last_active":"2021-02-10 11:49:04.488456","last_peered":"2021-02-10 11:49:04.488456","last_clean":"2021-02-10 11:49:04.488456","last_became_active":"2021-02-10 11:49:03.849140","last_became_peered":"2021-02-10 11:49:03.849140","last_unstale":"2021-02-10 11:49:04.488456","last_undegraded":"2021-02-10 11:49:04.488456","last_fullsized":"2021-02-10 11:49:04.488456","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:49:02.834883","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:49:02.834883","last_clean_scrub_stamp":"2021-02-10 11:49:02.834883","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"43.0","version":"198'19","reported_seq":"38","reported_epoch":"198","state":"active+clean","last_fresh":"2021-02-10 11:50:17.436525","last_change":"2021-02-10 11:49:04.487368","last_active":"2021-02-10 11:50:17.436525","last_peered":"2021-02-10 11:50:17.436525","last_clean":"2021-02-10 11:50:17.436525","last_became_active":"2021-02-10 11:49:04.487045","last_became_peered":"2021-02-10 11:49:04.487045","last_unstale":"2021-02-10 11:50:17.436525","last_undegraded":"2021-02-10 11:50:17.436525","last_fullsized":"2021-02-10 11:50:17.436525","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:49:03.842471","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:49:03.842471","last_clean_scrub_stamp":"2021-02-10 11:49:03.842471","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18874368,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":10,"num_read_kb":10240,"num_write":18,"num_write_kb":18432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"42.1","version":"198'2","reported_seq":"14","reported_epoch":"198","state":"active+clean","last_fresh":"2021-02-10 11:50:18.496010","last_change":"2021-02-10 11:49:03.849478","last_active":"2021-02-10 11:50:18.496010","last_peered":"2021-02-10 11:50:18.496010","last_clean":"2021-02-10 11:50:18.496010","last_became_active":"2021-02-10 11:49:03.849304","last_became_peered":"2021-02-10 11:49:03.849304","last_unstale":"2021-02-10 11:50:18.496010","last_undegraded":"2021-02-10 11:50:18.496010","last_fullsized":"2021-02-10 11:50:18.496010","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:49:02.834883","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:49:02.834883","last_clean_scrub_stamp":"2021-02-10 11:49:02.834883","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":101,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"43.1","version":"198'11","reported_seq":"26","reported_epoch":"198","state":"active+clean","last_fresh":"2021-02-10 11:50:17.501741","last_change":"2021-02-10 11:49:04.487450","last_active":"2021-02-10 11:50:17.501741","last_peered":"2021-02-10 11:50:17.501741","last_clean":"2021-02-10 11:50:17.501741","last_became_active":"2021-02-10 11:49:04.487096","last_became_peered":"2021-02-10 11:49:04.487096","last_unstale":"2021-02-10 11:50:17.501741","last_undegraded":"2021-02-10 11:50:17.501741","last_fullsized":"2021-02-10 11:50:17.501741","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:49:03.842471","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:49:03.842471","last_clean_scrub_stamp":"2021-02-10 11:49:03.842471","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":10485760,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":6,"num_read_kb":6144,"num_write":10,"num_write_kb":10240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.0","version":"197'8","reported_seq":"19","reported_epoch":"197","state":"active+clean","last_fresh":"2021-02-10 11:49:04.490252","last_change":"2021-02-10 11:49:03.848900","last_active":"2021-02-10 11:49:04.490252","last_peered":"2021-02-10 11:49:04.490252","last_clean":"2021-02-10 11:49:04.490252","last_became_active":"2021-02-10 11:49:03.848142","last_became_peered":"2021-02-10 11:49:03.848142","last_unstale":"2021-02-10 11:49:04.490252","last_undegraded":"2021-02-10 11:49:04.490252","last_fullsized":"2021-02-10 11:49:04.490252","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:49:02.834883","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:49:02.834883","last_clean_scrub_stamp":"2021-02-10 11:49:02.834883","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":"180","reported_epoch":"197","state":"active+clean","last_fresh":"2021-02-10 11:49:04.484090","last_change":"2021-02-10 11:13:58.963167","last_active":"2021-02-10 11:49:04.484090","last_peered":"2021-02-10 11:49:04.484090","last_clean":"2021-02-10 11:49:04.484090","last_became_active":"2021-02-10 11:13:58.962972","last_became_peered":"2021-02-10 11:13:58.962972","last_unstale":"2021-02-10 11:49:04.484090","last_undegraded":"2021-02-10 11:49:04.484090","last_fullsized":"2021-02-10 11:49:04.484090","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:13:57.948661","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:13:57.948661","last_clean_scrub_stamp":"2021-02-10 11:13:57.948661","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":"180","reported_epoch":"197","state":"active+clean","last_fresh":"2021-02-10 11:49:04.484174","last_change":"2021-02-10 11:13:58.964263","last_active":"2021-02-10 11:49:04.484174","last_peered":"2021-02-10 11:49:04.484174","last_clean":"2021-02-10 11:49:04.484174","last_became_active":"2021-02-10 11:13:58.964050","last_became_peered":"2021-02-10 11:13:58.964050","last_unstale":"2021-02-10 11:49:04.484174","last_undegraded":"2021-02-10 11:49:04.484174","last_fullsized":"2021-02-10 11:49:04.484174","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:13:57.948661","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:13:57.948661","last_clean_scrub_stamp":"2021-02-10 11:13:57.948661","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"187","reported_epoch":"197","state":"active+clean","last_fresh":"2021-02-10 11:49:04.483150","last_change":"2021-02-10 11:13:58.964235","last_active":"2021-02-10 11:49:04.483150","last_peered":"2021-02-10 11:49:04.483150","last_clean":"2021-02-10 11:49:04.483150","last_became_active":"2021-02-10 11:13:58.963951","last_became_peered":"2021-02-10 11:13:58.963951","last_unstale":"2021-02-10 11:49:04.483150","last_undegraded":"2021-02-10 11:49:04.483150","last_fullsized":"2021-02-10 11:49:04.483150","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:13:57.948661","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:13:57.948661","last_clean_scrub_stamp":"2021-02-10 11:13:57.948661","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":"183","reported_epoch":"197","state":"active+clean","last_fresh":"2021-02-10 11:49:04.483420","last_change":"2021-02-10 11:13:58.962265","last_active":"2021-02-10 11:49:04.483420","last_peered":"2021-02-10 11:49:04.483420","last_clean":"2021-02-10 11:49:04.483420","last_became_active":"2021-02-10 11:13:58.961934","last_became_peered":"2021-02-10 11:13:58.961934","last_unstale":"2021-02-10 11:49:04.483420","last_undegraded":"2021-02-10 11:49:04.483420","last_fullsized":"2021-02-10 11:49:04.483420","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:13:57.948661","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:13:57.948661","last_clean_scrub_stamp":"2021-02-10 11:13:57.948661","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"179","reported_epoch":"197","state":"active+clean","last_fresh":"2021-02-10 11:49:04.482926","last_change":"2021-02-10 11:13:58.964324","last_active":"2021-02-10 11:49:04.482926","last_peered":"2021-02-10 11:49:04.482926","last_clean":"2021-02-10 11:49:04.482926","last_became_active":"2021-02-10 11:13:58.964097","last_became_peered":"2021-02-10 11:13:58.964097","last_unstale":"2021-02-10 11:49:04.482926","last_undegraded":"2021-02-10 11:49:04.482926","last_fullsized":"2021-02-10 11:49:04.482926","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:13:57.948661","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:13:57.948661","last_clean_scrub_stamp":"2021-02-10 11:13:57.948661","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"43.4","version":"198'10","reported_seq":"25","reported_epoch":"198","state":"active+clean","last_fresh":"2021-02-10 11:50:17.445438","last_change":"2021-02-10 11:49:04.485490","last_active":"2021-02-10 11:50:17.445438","last_peered":"2021-02-10 11:50:17.445438","last_clean":"2021-02-10 11:50:17.445438","last_became_active":"2021-02-10 11:49:04.485246","last_became_peered":"2021-02-10 11:49:04.485246","last_unstale":"2021-02-10 11:50:17.445438","last_undegraded":"2021-02-10 11:50:17.445438","last_fullsized":"2021-02-10 11:50:17.445438","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:49:03.842471","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:49:03.842471","last_clean_scrub_stamp":"2021-02-10 11:49:03.842471","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":10485760,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":6,"num_read_kb":6144,"num_write":10,"num_write_kb":10240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"42.5","version":"198'9","reported_seq":"20","reported_epoch":"198","state":"active+clean","last_fresh":"2021-02-10 11:50:18.493972","last_change":"2021-02-10 11:49:03.848519","last_active":"2021-02-10 11:50:18.493972","last_peered":"2021-02-10 11:50:18.493972","last_clean":"2021-02-10 11:50:18.493972","last_became_active":"2021-02-10 11:49:03.848285","last_became_peered":"2021-02-10 11:49:03.848285","last_unstale":"2021-02-10 11:50:18.493972","last_undegraded":"2021-02-10 11:50:18.493972","last_fullsized":"2021-02-10 11:50:18.493972","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:49:02.834883","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:49:02.834883","last_clean_scrub_stamp":"2021-02-10 11:49:02.834883","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":160,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"188","reported_epoch":"197","state":"active+clean","last_fresh":"2021-02-10 11:49:04.484033","last_change":"2021-02-10 11:13:58.964040","last_active":"2021-02-10 11:49:04.484033","last_peered":"2021-02-10 11:49:04.484033","last_clean":"2021-02-10 11:49:04.484033","last_became_active":"2021-02-10 11:13:58.963814","last_became_peered":"2021-02-10 11:13:58.963814","last_unstale":"2021-02-10 11:49:04.484033","last_undegraded":"2021-02-10 11:49:04.484033","last_fullsized":"2021-02-10 11:49:04.484033","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:13:57.948661","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:13:57.948661","last_clean_scrub_stamp":"2021-02-10 11:13:57.948661","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"179","reported_epoch":"197","state":"active+clean","last_fresh":"2021-02-10 11:49:04.482937","last_change":"2021-02-10 11:13:58.964469","last_active":"2021-02-10 11:49:04.482937","last_peered":"2021-02-10 11:49:04.482937","last_clean":"2021-02-10 11:49:04.482937","last_became_active":"2021-02-10 11:13:58.964040","last_became_peered":"2021-02-10 11:13:58.964040","last_unstale":"2021-02-10 11:49:04.482937","last_undegraded":"2021-02-10 11:49:04.482937","last_fullsized":"2021-02-10 11:49:04.482937","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:13:57.948661","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:13:57.948661","last_clean_scrub_stamp":"2021-02-10 11:13:57.948661","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"180","reported_epoch":"197","state":"active+clean","last_fresh":"2021-02-10 11:49:04.484108","last_change":"2021-02-10 11:13:58.962834","last_active":"2021-02-10 11:49:04.484108","last_peered":"2021-02-10 11:49:04.484108","last_clean":"2021-02-10 11:49:04.484108","last_became_active":"2021-02-10 11:13:58.962533","last_became_peered":"2021-02-10 11:13:58.962533","last_unstale":"2021-02-10 11:49:04.484108","last_undegraded":"2021-02-10 11:49:04.484108","last_fullsized":"2021-02-10 11:49:04.484108","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:13:57.948661","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:13:57.948661","last_clean_scrub_stamp":"2021-02-10 11:13:57.948661","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":43,"num_pg":8,"stat_sum":{"num_bytes":134218155,"num_objects":131,"num_object_clones":1,"num_object_copies":262,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":131,"num_whiteouts":0,"num_read":64,"num_read_kb":65536,"num_write":139,"num_write_kb":131079,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":268460032,"data_stored":268436310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":144,"ondisk_log_size":144,"up":16,"acting":16,"num_store_stats":8},{"poolid":42,"num_pg":8,"stat_sum":{"num_bytes":778213,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":283,"num_write_kb":848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1622016,"data_stored":1556426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":263,"ondisk_log_size":263,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575289,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1087964,"kb_used_data":39380,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93283876,"statfs":{"total":96636764160,"available":95522689024,"internally_reserved":1073741824,"allocated":40325120,"data_stored":38850165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 11:49:53 2021","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.82499999999999996,"15min":0.79000000000000004},"min":{"1min":0.37,"5min":0.37,"15min":0.36499999999999999},"max":{"1min":1.1850000000000001,"5min":1.248,"15min":1.4259999999999999},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.82599999999999996,"15min":0.80600000000000005},"min":{"1min":0.40699999999999997,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.1379999999999999,"5min":1.3340000000000001,"15min":1.3939999999999999},"last":0.77300000000000002}]},{"osd":1,"last update":"Wed Feb 10 11:49:53 2021","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.80200000000000005,"15min":0.80500000000000005},"min":{"1min":0.44700000000000001,"5min":0.436,"15min":0.42199999999999999},"max":{"1min":1.161,"5min":1.3839999999999999,"15min":1.3839999999999999},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.83499999999999996,"5min":0.84999999999999998,"15min":0.81599999999999995},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.371},"max":{"1min":1.1140000000000001,"5min":1.3979999999999999,"15min":1.3979999999999999},"last":0.874}]},{"osd":2,"last update":"Wed Feb 10 11:49:53 2021","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":0.91400000000000003,"15min":0.88100000000000001},"min":{"1min":0.432,"5min":0.42899999999999999,"15min":0.40699999999999997},"max":{"1min":1.151,"5min":1.4510000000000001,"15min":1.4510000000000001},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.94199999999999995,"5min":1.0029999999999999,"15min":0.96999999999999997},"min":{"1min":0.54900000000000004,"5min":0.51400000000000001,"15min":0.47399999999999998},"max":{"1min":1.2250000000000001,"5min":1.5329999999999999,"15min":1.5329999999999999},"last":0.85499999999999998}]},{"osd":3,"last update":"Wed Feb 10 11:49:53 2021","interfaces":[{"interface":"back","average":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.80400000000000005},"min":{"1min":0.51200000000000001,"5min":0.46700000000000003,"15min":0.40699999999999997},"max":{"1min":1.099,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.89800000000000002,"5min":0.90600000000000003,"15min":0.85999999999999999},"min":{"1min":0.5,"5min":0.48099999999999998,"15min":0.42599999999999999},"max":{"1min":1.1259999999999999,"5min":1.4339999999999999,"15min":1.4339999999999999},"last":0.80500000000000005}]},{"osd":4,"last update":"Wed Feb 10 11:49:53 2021","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.69699999999999995,"15min":0.69299999999999995},"min":{"1min":0.46400000000000002,"5min":0.40999999999999998,"15min":0.34300000000000003},"max":{"1min":0.89000000000000001,"5min":1.3320000000000001,"15min":1.3320000000000001},"last":0.82399999999999995},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.70999999999999996,"15min":0.69399999999999995},"min":{"1min":0.42599999999999999,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.93700000000000006,"5min":1.127,"15min":1.149},"last":0.60599999999999998}]},{"osd":5,"last update":"Wed Feb 10 11:49:53 2021","interfaces":[{"interface":"back","average":{"1min":0.86399999999999999,"5min":0.81399999999999995,"15min":0.80800000000000005},"min":{"1min":0.61599999999999999,"5min":0.48399999999999999,"15min":0.45700000000000002},"max":{"1min":1.1259999999999999,"5min":1.3140000000000001,"15min":1.3140000000000001},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.85799999999999998,"5min":0.88200000000000001,"15min":0.878},"min":{"1min":0.59399999999999997,"5min":0.50700000000000001,"15min":0.45200000000000001},"max":{"1min":1.1220000000000001,"5min":1.3260000000000001,"15min":1.377},"last":0.69099999999999995}]},{"osd":6,"last update":"Wed Feb 10 11:49:53 2021","interfaces":[{"interface":"back","average":{"1min":0.80600000000000005,"5min":0.81699999999999995,"15min":0.82299999999999995},"min":{"1min":0.55700000000000005,"5min":0.52900000000000003,"15min":0.50700000000000001},"max":{"1min":0.97599999999999998,"5min":1.2709999999999999,"15min":1.327},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.80200000000000005,"15min":0.80300000000000005},"min":{"1min":0.55400000000000005,"5min":0.53800000000000003,"15min":0.53200000000000003},"max":{"1min":1.0469999999999999,"5min":1.276,"15min":1.276},"last":0.503}]}]},{"osd":6,"up_from":13,"seq":55834575290,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1070284,"kb_used_data":21700,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93301556,"statfs":{"total":96636764160,"available":95540793344,"internally_reserved":1073741824,"allocated":22220800,"data_stored":20752226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 11:50:07 2021","interfaces":[{"interface":"back","average":{"1min":0.94199999999999995,"5min":0.83199999999999996,"15min":0.81299999999999994},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.34699999999999998},"max":{"1min":1.169,"5min":1.1859999999999999,"15min":1.4259999999999999},"last":0.878},{"interface":"front","average":{"1min":0.879,"5min":0.78700000000000003,"15min":0.78600000000000003},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41299999999999998},"max":{"1min":1.131,"5min":1.2050000000000001,"15min":1.4419999999999999},"last":0.86499999999999999}]},{"osd":1,"last update":"Wed Feb 10 11:50:07 2021","interfaces":[{"interface":"back","average":{"1min":0.90800000000000003,"5min":0.80800000000000005,"15min":0.79800000000000004},"min":{"1min":0.46300000000000002,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.147,"5min":1.147,"15min":1.504},"last":0.89900000000000002},{"interface":"front","average":{"1min":0.95999999999999996,"5min":0.83699999999999997,"15min":0.83399999999999996},"min":{"1min":0.48399999999999999,"5min":0.45600000000000002,"15min":0.45300000000000001},"max":{"1min":1.2549999999999999,"5min":1.2549999999999999,"15min":1.3959999999999999},"last":0.97999999999999998}]},{"osd":2,"last update":"Wed Feb 10 11:50:07 2021","interfaces":[{"interface":"back","average":{"1min":1.0149999999999999,"5min":0.93400000000000005,"15min":0.93500000000000005},"min":{"1min":0.623,"5min":0.49299999999999999,"15min":0.438},"max":{"1min":1.339,"5min":1.339,"15min":1.3799999999999999},"last":0.82399999999999995},{"interface":"front","average":{"1min":1.054,"5min":0.996,"15min":0.98699999999999999},"min":{"1min":0.61399999999999999,"5min":0.61299999999999999,"15min":0.55200000000000005},"max":{"1min":1.3320000000000001,"5min":1.3320000000000001,"15min":1.4950000000000001},"last":0.77000000000000002}]},{"osd":3,"last update":"Wed Feb 10 11:50:07 2021","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":0.90700000000000003,"15min":0.90200000000000002},"min":{"1min":0.498,"5min":0.47199999999999998,"15min":0.39900000000000002},"max":{"1min":1.272,"5min":1.272,"15min":1.272},"last":0.96399999999999997},{"interface":"front","average":{"1min":1.0129999999999999,"5min":0.96699999999999997,"15min":0.95699999999999996},"min":{"1min":0.76700000000000002,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.246,"5min":1.258,"15min":1.361},"last":0.95299999999999996}]},{"osd":4,"last update":"Wed Feb 10 11:50:07 2021","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.66900000000000004,"15min":0.71599999999999997},"min":{"1min":0.48699999999999999,"5min":0.441,"15min":0.31900000000000001},"max":{"1min":0.92800000000000005,"5min":1.0960000000000001,"15min":1.399},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.70099999999999996,"15min":0.71399999999999997},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.35599999999999998},"max":{"1min":0.88700000000000001,"5min":1.004,"15min":1.23},"last":0.62}]},{"osd":5,"last update":"Wed Feb 10 11:50:07 2021","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.68000000000000005,"15min":0.71899999999999997},"min":{"1min":0.45200000000000001,"5min":0.42199999999999999,"15min":0.40899999999999997},"max":{"1min":0.85199999999999998,"5min":1.113,"15min":1.327},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.75,"5min":0.77500000000000002,"15min":0.79600000000000004},"min":{"1min":0.58699999999999997,"5min":0.57699999999999996,"15min":0.40699999999999997},"max":{"1min":0.97799999999999998,"5min":1.163,"15min":1.5},"last":0.65600000000000003}]},{"osd":7,"last update":"Wed Feb 10 11:50:07 2021","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.75900000000000001,"15min":0.78000000000000003},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.41599999999999998},"max":{"1min":0.92700000000000005,"5min":1.149,"15min":1.2829999999999999},"last":0.754},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.79300000000000004,"15min":0.81499999999999995},"min":{"1min":0.58399999999999996,"5min":0.53900000000000003,"15min":0.505},"max":{"1min":1.1160000000000001,"5min":1.1699999999999999,"15min":1.3600000000000001},"last":0.70599999999999996}]}]},{"osd":5,"up_from":13,"seq":55834575289,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1083868,"kb_used_data":35272,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93287972,"statfs":{"total":96636764160,"available":95526883328,"internally_reserved":1073741824,"allocated":36118528,"data_stored":34655064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 11:49:55 2021","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":0.83199999999999996,"15min":0.81999999999999995},"min":{"1min":0.35799999999999998,"5min":0.35699999999999998,"15min":0.34999999999999998},"max":{"1min":1.3580000000000001,"5min":1.3580000000000001,"15min":1.581},"last":1.123},{"interface":"front","average":{"1min":0.84099999999999997,"5min":0.79800000000000004,"15min":0.80100000000000005},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.35799999999999998},"max":{"1min":1.288,"5min":1.288,"15min":1.3939999999999999},"last":1.1579999999999999}]},{"osd":1,"last update":"Wed Feb 10 11:49:55 2021","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.81200000000000006,"15min":0.81799999999999995},"min":{"1min":0.375,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.2490000000000001,"5min":1.2490000000000001,"15min":1.3540000000000001},"last":1.212},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.81599999999999995,"15min":0.82499999999999996},"min":{"1min":0.376,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":1.097,"5min":1.2509999999999999,"15min":1.319},"last":1.2030000000000001}]},{"osd":2,"last update":"Wed Feb 10 11:49:55 2021","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.80700000000000005,"15min":0.81499999999999995},"min":{"1min":0.42399999999999999,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.1439999999999999,"5min":1.258,"15min":1.2729999999999999},"last":1.0389999999999999},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.83199999999999996,"15min":0.84499999999999997},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.24,"5min":1.2769999999999999,"15min":1.2769999999999999},"last":1.274}]},{"osd":3,"last update":"Wed Feb 10 11:49:55 2021","interfaces":[{"interface":"back","average":{"1min":0.83599999999999997,"5min":0.85499999999999998,"15min":0.873},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.264,"5min":1.323,"15min":1.464},"last":1.254},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.85299999999999998,"15min":0.875},"min":{"1min":0.57599999999999996,"5min":0.52400000000000002,"15min":0.44900000000000001},"max":{"1min":1.0960000000000001,"5min":1.2549999999999999,"15min":1.3520000000000001},"last":1.103}]},{"osd":4,"last update":"Wed Feb 10 11:49:55 2021","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.68400000000000005,"15min":0.70899999999999996},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.117,"5min":1.1910000000000001,"15min":1.1910000000000001},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.67800000000000005,"15min":0.70899999999999996},"min":{"1min":0.45200000000000001,"5min":0.45000000000000001,"15min":0.40799999999999997},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":1.2490000000000001},"last":0.628}]},{"osd":6,"last update":"Wed Feb 10 11:49:55 2021","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.72499999999999998,"15min":0.73099999999999998},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.38900000000000001},"max":{"1min":1.0169999999999999,"5min":1.22,"15min":1.22},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.76000000000000001,"15min":0.78600000000000003},"min":{"1min":0.52800000000000002,"5min":0.48699999999999999,"15min":0.40500000000000003},"max":{"1min":1.0009999999999999,"5min":1.2210000000000001,"15min":1.2210000000000001},"last":0.94199999999999995}]},{"osd":7,"last update":"Wed Feb 10 11:49:55 2021","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.755,"15min":0.78300000000000003},"min":{"1min":0.48799999999999999,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.93300000000000005,"5min":1.1459999999999999,"15min":1.1699999999999999},"last":0.81399999999999995},{"interface":"front","average":{"1min":0.748,"5min":0.78600000000000003,"15min":0.81599999999999995},"min":{"1min":0.52800000000000002,"5min":0.47099999999999997,"15min":0.46899999999999997},"max":{"1min":1.161,"5min":1.355,"15min":1.4710000000000001},"last":0.71399999999999997}]}]},{"osd":4,"up_from":13,"seq":55834575290,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1074376,"kb_used_data":25792,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93297464,"statfs":{"total":96636764160,"available":95536603136,"internally_reserved":1073741824,"allocated":26411008,"data_stored":24946496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 11:50:10 2021","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":0.81000000000000005,"15min":0.81100000000000005},"min":{"1min":0.40100000000000002,"5min":0.32300000000000001,"15min":0.317},"max":{"1min":1.212,"5min":1.3660000000000001,"15min":1.5840000000000001},"last":0.94699999999999995},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.79600000000000004,"15min":0.78800000000000003},"min":{"1min":0.46899999999999997,"5min":0.40799999999999997,"15min":0.38100000000000001},"max":{"1min":1.0740000000000001,"5min":1.2909999999999999,"15min":1.2909999999999999},"last":0.82899999999999996}]},{"osd":1,"last update":"Wed Feb 10 11:50:10 2021","interfaces":[{"interface":"back","average":{"1min":0.81100000000000005,"5min":0.76000000000000001,"15min":0.77800000000000002},"min":{"1min":0.47699999999999998,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.095,"5min":1.2110000000000001,"15min":1.2689999999999999},"last":0.88300000000000001},{"interface":"front","average":{"1min":0.84399999999999997,"5min":0.85299999999999998,"15min":0.84699999999999998},"min":{"1min":0.55700000000000005,"5min":0.48399999999999999,"15min":0.40000000000000002},"max":{"1min":1.091,"5min":1.3180000000000001,"15min":1.319},"last":0.86399999999999999}]},{"osd":2,"last update":"Wed Feb 10 11:50:10 2021","interfaces":[{"interface":"back","average":{"1min":1.024,"5min":1.0089999999999999,"15min":1.0069999999999999},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.46899999999999997},"max":{"1min":1.353,"5min":1.4350000000000001,"15min":1.635},"last":1.054},{"interface":"front","average":{"1min":0.98399999999999999,"5min":0.91200000000000003,"15min":0.92200000000000004},"min":{"1min":0.67800000000000005,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.3120000000000001,"5min":1.327,"15min":1.4690000000000001},"last":1.0629999999999999}]},{"osd":3,"last update":"Wed Feb 10 11:50:10 2021","interfaces":[{"interface":"back","average":{"1min":1.03,"5min":1.008,"15min":0.998},"min":{"1min":0.67000000000000004,"5min":0.53800000000000003,"15min":0.48099999999999998},"max":{"1min":1.355,"5min":1.46,"15min":1.5569999999999999},"last":1.0029999999999999},{"interface":"front","average":{"1min":1.04,"5min":0.97099999999999997,"15min":0.94399999999999995},"min":{"1min":0.755,"5min":0.498,"15min":0.48899999999999999},"max":{"1min":1.325,"5min":1.379,"15min":1.3979999999999999},"last":0.96799999999999997}]},{"osd":5,"last update":"Wed Feb 10 11:50:10 2021","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.73599999999999999,"15min":0.73399999999999999},"min":{"1min":0.438,"5min":0.436,"15min":0.36799999999999999},"max":{"1min":1.095,"5min":1.1799999999999999,"15min":1.292},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.745,"5min":0.73199999999999998,"15min":0.70899999999999996},"min":{"1min":0.52500000000000002,"5min":0.46899999999999997,"15min":0.40999999999999998},"max":{"1min":1.3540000000000001,"5min":1.3540000000000001,"15min":1.3540000000000001},"last":0.53400000000000003}]},{"osd":6,"last update":"Wed Feb 10 11:50:10 2021","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.71999999999999997,"15min":0.73599999999999999},"min":{"1min":0.503,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.105,"5min":1.105,"15min":1.22},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.81299999999999994,"5min":0.82299999999999995,"15min":0.82099999999999995},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.43099999999999999},"max":{"1min":1.1479999999999999,"5min":1.321,"15min":1.5149999999999999},"last":0.70999999999999996}]},{"osd":7,"last update":"Wed Feb 10 11:50:10 2021","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.79900000000000004,"15min":0.77300000000000002},"min":{"1min":0.61199999999999999,"5min":0.56499999999999995,"15min":0.40300000000000002},"max":{"1min":1.256,"5min":1.256,"15min":1.256},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.80300000000000005,"15min":0.80200000000000005},"min":{"1min":0.628,"5min":0.58099999999999996,"15min":0.52100000000000002},"max":{"1min":1.163,"5min":1.169,"15min":1.169},"last":0.63}]}]},{"osd":0,"up_from":13,"seq":55834575289,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1061324,"kb_used_data":12740,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93310516,"statfs":{"total":96636764160,"available":95549968384,"internally_reserved":1073741824,"allocated":13045760,"data_stored":11586266,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Feb 10 11:49:56 2021","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.49299999999999999,"15min":0.48399999999999999},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":1,"5min":1,"15min":1},"last":0.628},{"interface":"front","average":{"1min":0.496,"5min":0.48299999999999998,"15min":0.47999999999999998},"min":{"1min":0.34599999999999997,"5min":0.32500000000000001,"15min":0.313},"max":{"1min":0.68000000000000005,"5min":0.73999999999999999,"15min":0.88800000000000001},"last":0.46700000000000003}]},{"osd":2,"last update":"Wed Feb 10 11:49:56 2021","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.64500000000000002,"15min":0.622},"min":{"1min":0.40999999999999998,"5min":0.317,"15min":0.317},"max":{"1min":0.871,"5min":0.95999999999999996,"15min":1.1719999999999999},"last":0.95399999999999996},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.64900000000000002,"15min":0.64400000000000002},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":1.125},"last":0.77300000000000002}]},{"osd":3,"last update":"Wed Feb 10 11:49:56 2021","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.52200000000000002,"15min":0.51900000000000002},"min":{"1min":0.38400000000000001,"5min":0.35299999999999998,"15min":0.34200000000000003},"max":{"1min":0.71099999999999997,"5min":0.83599999999999997,"15min":0.97199999999999998},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.625,"15min":0.60899999999999999},"min":{"1min":0.46400000000000002,"5min":0.41899999999999998,"15min":0.318},"max":{"1min":0.77200000000000002,"5min":0.97099999999999997,"15min":0.99199999999999999},"last":0.81399999999999995}]},{"osd":4,"last update":"Wed Feb 10 11:49:56 2021","interfaces":[{"interface":"back","average":{"1min":0.98199999999999998,"5min":0.90200000000000002,"15min":0.92200000000000004},"min":{"1min":0.56399999999999995,"5min":0.52400000000000002,"15min":0.45400000000000001},"max":{"1min":1.3100000000000001,"5min":1.3320000000000001,"15min":1.365},"last":1.0249999999999999},{"interface":"front","average":{"1min":0.96899999999999997,"5min":0.94299999999999995,"15min":0.94699999999999995},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.46600000000000003},"max":{"1min":1.3,"5min":1.3,"15min":1.393},"last":1.073}]},{"osd":5,"last update":"Wed Feb 10 11:49:56 2021","interfaces":[{"interface":"back","average":{"1min":1.0720000000000001,"5min":1.0229999999999999,"15min":1.071},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.5169999999999999,"5min":1.5169999999999999,"15min":1.8660000000000001},"last":1.04},{"interface":"front","average":{"1min":1.123,"5min":1.048,"15min":1.0940000000000001},"min":{"1min":0.64800000000000002,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":1.5389999999999999,"5min":1.6000000000000001,"15min":1.6000000000000001},"last":1.2110000000000001}]},{"osd":6,"last update":"Wed Feb 10 11:49:56 2021","interfaces":[{"interface":"back","average":{"1min":1.046,"5min":1.0409999999999999,"15min":1.077},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.4399999999999999,"5min":1.4399999999999999,"15min":1.542},"last":1.1930000000000001},{"interface":"front","average":{"1min":1.099,"5min":1.046,"15min":1.093},"min":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.61699999999999999},"max":{"1min":1.548,"5min":1.548,"15min":1.792},"last":1.222}]},{"osd":7,"last update":"Wed Feb 10 11:49:56 2021","interfaces":[{"interface":"back","average":{"1min":1.087,"5min":1.054,"15min":1.095},"min":{"1min":0.58299999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.494,"5min":1.548,"15min":1.6559999999999999},"last":1.1359999999999999},{"interface":"front","average":{"1min":1.101,"5min":1.0489999999999999,"15min":1.0820000000000001},"min":{"1min":0.65500000000000003,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":1.4339999999999999,"5min":1.4990000000000001,"15min":1.4990000000000001},"last":1.1220000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575288,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1092052,"kb_used_data":43468,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93279788,"statfs":{"total":96636764160,"available":95518502912,"internally_reserved":1073741824,"allocated":44511232,"data_stored":43043903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 11:50:08 2021","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.32700000000000001,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.876},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.47999999999999998,"15min":0.49199999999999999},"min":{"1min":0.374,"5min":0.308,"15min":0.308},"max":{"1min":0.75800000000000001,"5min":0.82499999999999996,"15min":1.1279999999999999},"last":0.36499999999999999}]},{"osd":2,"last update":"Wed Feb 10 11:50:08 2021","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.61799999999999999,"15min":0.63},"min":{"1min":0.44600000000000001,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.753,"5min":0.96399999999999997,"15min":1.4319999999999999},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.57799999999999996,"15min":0.59399999999999997},"min":{"1min":0.39000000000000001,"5min":0.378,"15min":0.313},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.95099999999999996},"last":0.44500000000000001}]},{"osd":3,"last update":"Wed Feb 10 11:50:08 2021","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.54700000000000004,"15min":0.54200000000000004},"min":{"1min":0.38800000000000001,"5min":0.315,"15min":0.315},"max":{"1min":0.68799999999999994,"5min":1,"15min":1},"last":0.499},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.61299999999999999,"15min":0.62},"min":{"1min":0.34399999999999997,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.83299999999999996,"5min":1.0429999999999999,"15min":1.22},"last":0.57999999999999996}]},{"osd":4,"last update":"Wed Feb 10 11:50:08 2021","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.875,"15min":0.90300000000000002},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":1.1910000000000001,"5min":1.498,"15min":1.498},"last":1.0229999999999999},{"interface":"front","average":{"1min":0.85999999999999999,"5min":0.89200000000000002,"15min":0.91700000000000004},"min":{"1min":0.59199999999999997,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.2310000000000001,"5min":1.3380000000000001,"15min":1.3540000000000001},"last":0.89300000000000002}]},{"osd":5,"last update":"Wed Feb 10 11:50:08 2021","interfaces":[{"interface":"back","average":{"1min":0.99099999999999999,"5min":1.0209999999999999,"15min":1.052},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.48199999999999998},"max":{"1min":1.544,"5min":1.544,"15min":1.6399999999999999},"last":1.0389999999999999},{"interface":"front","average":{"1min":0.98799999999999999,"5min":1.0429999999999999,"15min":1.071},"min":{"1min":0.65100000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.4279999999999999,"5min":1.446,"15min":1.4870000000000001},"last":1.1020000000000001}]},{"osd":6,"last update":"Wed Feb 10 11:50:08 2021","interfaces":[{"interface":"back","average":{"1min":0.92300000000000004,"5min":0.95699999999999996,"15min":0.95599999999999996},"min":{"1min":0.66700000000000004,"5min":0.51800000000000002,"15min":0.496},"max":{"1min":1.2549999999999999,"5min":1.421,"15min":1.5349999999999999},"last":0.86899999999999999},{"interface":"front","average":{"1min":0.871,"5min":0.91400000000000003,"15min":0.93200000000000005},"min":{"1min":0.60899999999999999,"5min":0.56100000000000005,"15min":0.498},"max":{"1min":1.3029999999999999,"5min":1.3029999999999999,"15min":1.3919999999999999},"last":1.1890000000000001}]},{"osd":7,"last update":"Wed Feb 10 11:50:08 2021","interfaces":[{"interface":"back","average":{"1min":1.008,"5min":1.0389999999999999,"15min":1.0449999999999999},"min":{"1min":0.751,"5min":0.59099999999999997,"15min":0.52100000000000002},"max":{"1min":1.444,"5min":1.444,"15min":1.4830000000000001},"last":1.2070000000000001},{"interface":"front","average":{"1min":1.0529999999999999,"5min":1.079,"15min":1.097},"min":{"1min":0.81000000000000005,"5min":0.627,"15min":0.627},"max":{"1min":1.4330000000000001,"5min":1.4670000000000001,"15min":1.615},"last":1.238}]}]},{"osd":2,"up_from":13,"seq":55834575289,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1122780,"kb_used_data":74196,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93249060,"statfs":{"total":96636764160,"available":95487037440,"internally_reserved":1073741824,"allocated":75976704,"data_stored":74501655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 11:49:52 2021","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.46899999999999997,"15min":0.47799999999999998},"min":{"1min":0.35499999999999998,"5min":0.308,"15min":0.308},"max":{"1min":0.70799999999999996,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.502,"15min":0.51000000000000001},"min":{"1min":0.34399999999999997,"5min":0.316,"15min":0.316},"max":{"1min":0.70799999999999996,"5min":0.92600000000000005,"15min":1.0329999999999999},"last":0.48699999999999999}]},{"osd":1,"last update":"Wed Feb 10 11:49:52 2021","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.51600000000000001,"15min":0.51700000000000002},"min":{"1min":0.38900000000000001,"5min":0.34000000000000002,"15min":0.32300000000000001},"max":{"1min":0.60299999999999998,"5min":0.94499999999999995,"15min":1.133},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.57299999999999995,"15min":0.57699999999999996},"min":{"1min":0.40799999999999997,"5min":0.375,"15min":0.375},"max":{"1min":0.84499999999999997,"5min":0.84699999999999998,"15min":0.93400000000000005},"last":0.63400000000000001}]},{"osd":3,"last update":"Wed Feb 10 11:49:52 2021","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.58399999999999996,"15min":0.58799999999999997},"min":{"1min":0.48299999999999998,"5min":0.40300000000000002,"15min":0.39900000000000002},"max":{"1min":0.76000000000000001,"5min":0.90500000000000003,"15min":0.96099999999999997},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.54600000000000004,"15min":0.56399999999999995},"min":{"1min":0.42199999999999999,"5min":0.36599999999999999,"15min":0.34599999999999997},"max":{"1min":0.77300000000000002,"5min":0.89900000000000002,"15min":0.94699999999999995},"last":0.70399999999999996}]},{"osd":4,"last update":"Wed Feb 10 11:49:52 2021","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":0.91700000000000004,"15min":0.94099999999999995},"min":{"1min":0.68999999999999995,"5min":0.53100000000000003,"15min":0.52400000000000002},"max":{"1min":1.278,"5min":1.331,"15min":1.3919999999999999},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.95599999999999996,"5min":0.94499999999999995,"15min":0.96699999999999997},"min":{"1min":0.60499999999999998,"5min":0.57799999999999996,"15min":0.53300000000000003},"max":{"1min":1.304,"5min":1.304,"15min":1.4430000000000001},"last":0.76800000000000002}]},{"osd":5,"last update":"Wed Feb 10 11:49:52 2021","interfaces":[{"interface":"back","average":{"1min":0.95599999999999996,"5min":0.94799999999999995,"15min":0.97199999999999998},"min":{"1min":0.54400000000000004,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.206,"5min":1.393,"15min":1.4059999999999999},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.96599999999999997,"5min":0.94199999999999995,"15min":0.95399999999999996},"min":{"1min":0.65700000000000003,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.3300000000000001,"5min":1.359,"15min":1.369},"last":0.75700000000000001}]},{"osd":6,"last update":"Wed Feb 10 11:49:52 2021","interfaces":[{"interface":"back","average":{"1min":0.97799999999999998,"5min":0.94699999999999995,"15min":0.98199999999999998},"min":{"1min":0.62,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.361,"5min":1.361,"15min":1.462},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.96299999999999997,"5min":0.97199999999999998,"15min":0.98799999999999999},"min":{"1min":0.68200000000000005,"5min":0.57099999999999995,"15min":0.54900000000000004},"max":{"1min":1.288,"5min":1.296,"15min":1.4319999999999999},"last":0.68500000000000005}]},{"osd":7,"last update":"Wed Feb 10 11:49:52 2021","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":0.98099999999999998,"15min":0.999},"min":{"1min":0.73199999999999998,"5min":0.64900000000000002,"15min":0.58699999999999997},"max":{"1min":1.2290000000000001,"5min":1.4079999999999999,"15min":1.4079999999999999},"last":0.81599999999999995},{"interface":"front","average":{"1min":1.012,"5min":1.01,"15min":1.012},"min":{"1min":0.72899999999999998,"5min":0.64200000000000002,"15min":0.63800000000000001},"max":{"1min":1.3420000000000001,"5min":1.395,"15min":1.4350000000000001},"last":0.84299999999999997}]}]},{"osd":3,"up_from":13,"seq":55834575290,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1079756,"kb_used_data":31172,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93292084,"statfs":{"total":96636764160,"available":95531094016,"internally_reserved":1073741824,"allocated":31920128,"data_stored":30460737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 11:50:09 2021","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.498,"15min":0.496},"min":{"1min":0.35199999999999998,"5min":0.311,"15min":0.311},"max":{"1min":0.66300000000000003,"5min":1,"15min":1},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.498,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.40400000000000003,"5min":0.315,"15min":0.307},"max":{"1min":0.66900000000000004,"5min":0.83099999999999996,"15min":1.216},"last":0.45200000000000001}]},{"osd":1,"last update":"Wed Feb 10 11:50:09 2021","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.49199999999999999,"15min":0.502},"min":{"1min":0.34999999999999998,"5min":0.32900000000000001,"15min":0.30299999999999999},"max":{"1min":0.63800000000000001,"5min":0.98399999999999999,"15min":1.0740000000000001},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.55900000000000005,"15min":0.56799999999999995},"min":{"1min":0.39800000000000002,"5min":0.34499999999999997,"15min":0.28299999999999997},"max":{"1min":0.76400000000000001,"5min":0.85799999999999998,"15min":1.02},"last":0.52900000000000003}]},{"osd":2,"last update":"Wed Feb 10 11:50:09 2021","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.624,"15min":0.64000000000000001},"min":{"1min":0.42899999999999999,"5min":0.41499999999999998,"15min":0.33500000000000002},"max":{"1min":0.85599999999999998,"5min":0.88,"15min":1.081},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.77800000000000002,"15min":0.77400000000000002},"min":{"1min":0.54600000000000004,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.075,"5min":1.107,"15min":1.2030000000000001},"last":0.73199999999999998}]},{"osd":4,"last update":"Wed Feb 10 11:50:09 2021","interfaces":[{"interface":"back","average":{"1min":0.90900000000000003,"5min":0.90100000000000002,"15min":0.90000000000000002},"min":{"1min":0.53600000000000003,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.181,"5min":1.3839999999999999,"15min":1.4830000000000001},"last":1.052},{"interface":"front","average":{"1min":0.92700000000000005,"5min":0.91500000000000004,"15min":0.92700000000000005},"min":{"1min":0.51300000000000001,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.3620000000000001,"5min":1.4890000000000001,"15min":1.4890000000000001},"last":1.0660000000000001}]},{"osd":5,"last update":"Wed Feb 10 11:50:09 2021","interfaces":[{"interface":"back","average":{"1min":1.095,"5min":1.0760000000000001,"15min":1.0649999999999999},"min":{"1min":0.73999999999999999,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.5069999999999999,"5min":1.508,"15min":1.538},"last":1.181},{"interface":"front","average":{"1min":1.0860000000000001,"5min":1.0589999999999999,"15min":1.048},"min":{"1min":0.66400000000000003,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.6419999999999999,"5min":1.6419999999999999,"15min":1.6419999999999999},"last":1.2250000000000001}]},{"osd":6,"last update":"Wed Feb 10 11:50:09 2021","interfaces":[{"interface":"back","average":{"1min":0.95299999999999996,"5min":0.93500000000000005,"15min":0.95499999999999996},"min":{"1min":0.60599999999999998,"5min":0.501,"15min":0.501},"max":{"1min":1.2509999999999999,"5min":1.3939999999999999,"15min":1.452},"last":1.121},{"interface":"front","average":{"1min":0.96199999999999997,"5min":0.95799999999999996,"15min":0.97599999999999998},"min":{"1min":0.57899999999999996,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.323,"5min":1.323,"15min":1.4079999999999999},"last":1.0780000000000001}]},{"osd":7,"last update":"Wed Feb 10 11:50:09 2021","interfaces":[{"interface":"back","average":{"1min":1.0589999999999999,"5min":1.0549999999999999,"15min":1.0409999999999999},"min":{"1min":0.58999999999999997,"5min":0.58699999999999997,"15min":0.55100000000000005},"max":{"1min":1.5309999999999999,"5min":1.5309999999999999,"15min":1.5309999999999999},"last":1.147},{"interface":"front","average":{"1min":1.1040000000000001,"5min":1.0629999999999999,"15min":1.071},"min":{"1min":0.68700000000000006,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.518,"5min":1.556,"15min":1.556},"last":1.171}]}]}]}} 2021-02-10T11:50:27.314 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:50:27.747 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.725 7fbf9ee17700 1 Processor -- start 2021-02-10T11:50:27.748 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.726 7fbf9ee17700 1 -- start start 2021-02-10T11:50:27.748 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.726 7fbf9ee17700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fbf980d1870 0x7fbf981b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:27.748 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.726 7fbf9ee17700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fbf980f0650 0x7fbf980cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:27.748 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.726 7fbf9ee17700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf980f0b10 con 0x7fbf981b0a10 2021-02-10T11:50:27.748 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.726 7fbf9ee17700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf980e5ed0 con 0x7fbf980f0650 2021-02-10T11:50:27.748 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.726 7fbf9ee17700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf980d1c70 con 0x7fbf980d1870 2021-02-10T11:50:27.748 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.726 7fbf9d434700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fbf980f0650 0x7fbf980cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:27.748 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.726 7fbf9d434700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fbf980f0650 0x7fbf980cfb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:33988/0 (socket says 172.21.2.112:33988) 2021-02-10T11:50:27.749 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.726 7fbf9d434700 1 -- 172.21.2.112:0/1629757624 learned_addr learned my addr 172.21.2.112:0/1629757624 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:50:27.749 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.726 7fbf97fff700 1 --1- 172.21.2.112:0/1629757624 >> v1:172.21.2.112:6789/0 conn(0x7fbf981b0a10 0x7fbf980cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:33448/0 (socket says 172.21.2.112:33448) 2021-02-10T11:50:27.749 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.726 7fbf9cc33700 1 --2- 172.21.2.112:0/1629757624 >> v2:172.21.2.120:3300/0 conn(0x7fbf980d1870 0x7fbf981b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:27.749 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.727 7fbf977fe700 1 -- 172.21.2.112:0/1629757624 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2473022797 0 0) 0x7fbf88001060 con 0x7fbf981b0a10 2021-02-10T11:50:27.749 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.727 7fbf9d434700 1 -- 172.21.2.112:0/1629757624 >> v2:172.21.2.120:3300/0 conn(0x7fbf980d1870 msgr2=0x7fbf981b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:50:27.749 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.727 7fbf9d434700 1 --2- 172.21.2.112:0/1629757624 >> v2:172.21.2.120:3300/0 conn(0x7fbf980d1870 0x7fbf981b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:27.749 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.727 7fbf9d434700 1 -- 172.21.2.112:0/1629757624 >> v1:172.21.2.112:6789/0 conn(0x7fbf981b0a10 legacy=0x7fbf980cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:27.749 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.727 7fbf9d434700 1 -- 172.21.2.112:0/1629757624 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf980d1e90 con 0x7fbf980f0650 2021-02-10T11:50:27.750 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.727 7fbf9d434700 1 --2- 172.21.2.112:0/1629757624 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbf980f0650 0x7fbf980cfb20 secure :-1 s=READY pgs=3329 cs=0 l=1 rev1=1 rx=0x7fbf90006b80 tx=0x7fbf90009b90).ready entity=mon.0 client_cookie=e19c75cc47852a86 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:27.750 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.727 7fbf977fe700 1 -- 172.21.2.112:0/1629757624 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbf900022e0 con 0x7fbf980f0650 2021-02-10T11:50:27.750 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.727 7fbf977fe700 1 -- 172.21.2.112:0/1629757624 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf90002620 con 0x7fbf980f0650 2021-02-10T11:50:27.750 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.727 7fbf977fe700 1 -- 172.21.2.112:0/1629757624 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbf90002b00 con 0x7fbf980f0650 2021-02-10T11:50:27.750 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.728 7fbf9ee17700 1 -- 172.21.2.112:0/1629757624 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbf980f0650 msgr2=0x7fbf980cfb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:27.750 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.728 7fbf9ee17700 1 --2- 172.21.2.112:0/1629757624 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbf980f0650 0x7fbf980cfb20 secure :-1 s=READY pgs=3329 cs=0 l=1 rev1=1 rx=0x7fbf90006b80 tx=0x7fbf90009b90).stop 2021-02-10T11:50:27.750 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.728 7fbf9ee17700 1 -- 172.21.2.112:0/1629757624 shutdown_connections 2021-02-10T11:50:27.750 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.728 7fbf9ee17700 1 --2- 172.21.2.112:0/1629757624 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbf980f0650 0x7fbf980cfb20 unknown :-1 s=CLOSED pgs=3329 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:27.751 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.728 7fbf9ee17700 1 --2- 172.21.2.112:0/1629757624 >> v2:172.21.2.120:3300/0 conn(0x7fbf980d1870 0x7fbf981b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:27.751 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.728 7fbf9ee17700 1 -- 172.21.2.112:0/1629757624 >> 172.21.2.112:0/1629757624 conn(0x7fbf980ce450 msgr2=0x7fbf980d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:27.751 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.728 7fbf9ee17700 1 -- 172.21.2.112:0/1629757624 shutdown_connections 2021-02-10T11:50:27.751 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.728 7fbf9ee17700 1 -- 172.21.2.112:0/1629757624 wait complete. 2021-02-10T11:50:27.751 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.743 7fbf9ee17700 1 Processor -- start 2021-02-10T11:50:27.751 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.743 7fbf9ee17700 1 -- start start 2021-02-10T11:50:27.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.744 7fbf9ee17700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbf9819ab70 0x7fbf98096ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:27.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.744 7fbf9ee17700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbf98095cb0 0x7fbf980960b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:27.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.744 7fbf9ee17700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fbf980966d0 0x7fbf980b16f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:27.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.744 7fbf9ee17700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf98096be0 con 0x7fbf98095cb0 2021-02-10T11:50:27.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.744 7fbf9ee17700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf980b1d10 con 0x7fbf980966d0 2021-02-10T11:50:27.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.744 7fbf9ee17700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf980b1f30 con 0x7fbf9819ab70 2021-02-10T11:50:27.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.744 7fbf97fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbf98095cb0 0x7fbf980960b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:27.754 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.744 7fbf97fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbf98095cb0 0x7fbf980960b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:33990/0 (socket says 172.21.2.112:33990) 2021-02-10T11:50:27.754 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.744 7fbf97fff700 1 -- 172.21.2.112:0/3874715940 learned_addr learned my addr 172.21.2.112:0/3874715940 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:50:27.754 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.744 7fbf9d434700 1 --2- 172.21.2.112:0/3874715940 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fbf980966d0 0x7fbf980b16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:27.754 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.744 7fbf9cc33700 1 --2- 172.21.2.112:0/3874715940 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbf9819ab70 0x7fbf98096ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:27.754 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.744 7fbf97fff700 1 -- 172.21.2.112:0/3874715940 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbf9819ab70 msgr2=0x7fbf98096ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:27.754 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.744 7fbf97fff700 1 --2- 172.21.2.112:0/3874715940 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbf9819ab70 0x7fbf98096ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:27.754 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.744 7fbf97fff700 1 -- 172.21.2.112:0/3874715940 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fbf980966d0 msgr2=0x7fbf980b16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:27.754 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.744 7fbf97fff700 1 --2- 172.21.2.112:0/3874715940 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fbf980966d0 0x7fbf980b16f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:27.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.744 7fbf97fff700 1 -- 172.21.2.112:0/3874715940 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf980b2150 con 0x7fbf98095cb0 2021-02-10T11:50:27.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.745 7fbf97fff700 1 --2- 172.21.2.112:0/3874715940 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbf98095cb0 0x7fbf980960b0 secure :-1 s=READY pgs=3330 cs=0 l=1 rev1=1 rx=0x7fbf88009980 tx=0x7fbf8800bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:27.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.745 7fbf957fa700 1 -- 172.21.2.112:0/3874715940 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbf88006340 con 0x7fbf98095cb0 2021-02-10T11:50:27.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.745 7fbf957fa700 1 -- 172.21.2.112:0/3874715940 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf88006600 con 0x7fbf98095cb0 2021-02-10T11:50:27.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.745 7fbf957fa700 1 -- 172.21.2.112:0/3874715940 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbf88004910 con 0x7fbf98095cb0 2021-02-10T11:50:27.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.745 7fbf9ee17700 1 -- 172.21.2.112:0/3874715940 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf980b2150 con 0x7fbf98095cb0 2021-02-10T11:50:27.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.745 7fbf9ee17700 1 -- 172.21.2.112:0/3874715940 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf981a4b60 con 0x7fbf98095cb0 2021-02-10T11:50:27.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.746 7fbf957fa700 1 -- 172.21.2.112:0/3874715940 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fbf8800a000 con 0x7fbf98095cb0 2021-02-10T11:50:27.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.746 7fbf957fa700 1 --2- 172.21.2.112:0/3874715940 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fbf7c0186c0 0x7fbf7c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:27.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.746 7fbf957fa700 1 -- 172.21.2.112:0/3874715940 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fbf880328f0 con 0x7fbf98095cb0 2021-02-10T11:50:27.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.746 7fbf9cc33700 1 --2- 172.21.2.112:0/3874715940 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fbf7c0186c0 0x7fbf7c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:27.756 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.747 7fbf9cc33700 1 --2- 172.21.2.112:0/3874715940 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fbf7c0186c0 0x7fbf7c01abc0 crc :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:27.780 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.775 7fbf9ee17700 1 -- 172.21.2.112:0/3874715940 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf980ebc00 con 0x7fbf98095cb0 2021-02-10T11:50:27.782 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.778 7fbf957fa700 1 -- 172.21.2.112:0/3874715940 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fbf88004720 con 0x7fbf98095cb0 2021-02-10T11:50:27.907 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.903 7fbf9ee17700 1 -- 172.21.2.112:0/3874715940 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbf980f0b10 con 0x7fbf98095cb0 2021-02-10T11:50:27.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.904 7fbf957fa700 1 -- 172.21.2.112:0/3874715940 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v198) v1 ==== 74+0+14007 (secure 0 0 0) 0x7fbf8800a000 con 0x7fbf98095cb0 2021-02-10T11:50:27.908 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:50:27.909 INFO:teuthology.orchestra.run.gibba012.stdout:{"epoch":198,"fsid":"f14f004e-0699-4584-ad81-3754258493de","created":"2021-02-10 11:13:26.501424","modified":"2021-02-10 11:49:05.481103","last_up_change":"2021-02-10 11:13:52.780316","last_in_change":"2021-02-10 11:13:52.780316","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":43,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:13:57.508758","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":42,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:49:02.450970","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"197","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":43,"pool_name":"cephfs_data","create_time":"2021-02-10 11:49:03.473275","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"198","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"95444b8f-6370-4a1d-ae5f-df62651b77da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6817","nonce":36030},{"type":"v1","addr":"172.21.2.112:6818","nonce":36030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6820","nonce":36030},{"type":"v1","addr":"172.21.2.112:6822","nonce":36030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6828","nonce":36030},{"type":"v1","addr":"172.21.2.112:6830","nonce":36030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6824","nonce":36030},{"type":"v1","addr":"172.21.2.112:6826","nonce":36030}]},"public_addr":"172.21.2.112:6818/36030","cluster_addr":"172.21.2.112:6822/36030","heartbeat_back_addr":"172.21.2.112:6830/36030","heartbeat_front_addr":"172.21.2.112:6826/36030","state":["exists","up"]},{"osd":1,"uuid":"a23e217e-0d8e-486e-be07-712adfabd262","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":196,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6800","nonce":36029},{"type":"v1","addr":"172.21.2.112:6802","nonce":36029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6804","nonce":36029},{"type":"v1","addr":"172.21.2.112:6806","nonce":36029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6812","nonce":36029},{"type":"v1","addr":"172.21.2.112:6814","nonce":36029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6808","nonce":36029},{"type":"v1","addr":"172.21.2.112:6810","nonce":36029}]},"public_addr":"172.21.2.112:6802/36029","cluster_addr":"172.21.2.112:6806/36029","heartbeat_back_addr":"172.21.2.112:6814/36029","heartbeat_front_addr":"172.21.2.112:6810/36029","state":["exists","up"]},{"osd":2,"uuid":"026afa8a-c0f1-4b35-9a76-53c00fe0c2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":196,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6801","nonce":36031},{"type":"v1","addr":"172.21.2.112:6803","nonce":36031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6805","nonce":36031},{"type":"v1","addr":"172.21.2.112:6807","nonce":36031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6813","nonce":36031},{"type":"v1","addr":"172.21.2.112:6815","nonce":36031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6809","nonce":36031},{"type":"v1","addr":"172.21.2.112:6811","nonce":36031}]},"public_addr":"172.21.2.112:6803/36031","cluster_addr":"172.21.2.112:6807/36031","heartbeat_back_addr":"172.21.2.112:6815/36031","heartbeat_front_addr":"172.21.2.112:6811/36031","state":["exists","up"]},{"osd":3,"uuid":"58795682-e00d-4173-bb84-f7075ca89a11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":196,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6816","nonce":36028},{"type":"v1","addr":"172.21.2.112:6819","nonce":36028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6821","nonce":36028},{"type":"v1","addr":"172.21.2.112:6823","nonce":36028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6829","nonce":36028},{"type":"v1","addr":"172.21.2.112:6831","nonce":36028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.112:6825","nonce":36028},{"type":"v1","addr":"172.21.2.112:6827","nonce":36028}]},"public_addr":"172.21.2.112:6819/36028","cluster_addr":"172.21.2.112:6823/36028","heartbeat_back_addr":"172.21.2.112:6831/36028","heartbeat_front_addr":"172.21.2.112:6827/36028","state":["exists","up"]},{"osd":4,"uuid":"6763f8d6-53dc-4e45-bdf6-a6eabcf81e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":36388},{"type":"v1","addr":"172.21.2.120:6813","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":36388},{"type":"v1","addr":"172.21.2.120:6819","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6825","nonce":36388},{"type":"v1","addr":"172.21.2.120:6827","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":36388},{"type":"v1","addr":"172.21.2.120:6823","nonce":36388}]},"public_addr":"172.21.2.120:6813/36388","cluster_addr":"172.21.2.120:6819/36388","heartbeat_back_addr":"172.21.2.120:6827/36388","heartbeat_front_addr":"172.21.2.120:6823/36388","state":["exists","up"]},{"osd":5,"uuid":"5d0e4dd6-0cb0-4bf8-8314-626acaa1b544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":196,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":36387},{"type":"v1","addr":"172.21.2.120:6806","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6809","nonce":36387},{"type":"v1","addr":"172.21.2.120:6812","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":36387},{"type":"v1","addr":"172.21.2.120:6822","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6815","nonce":36387},{"type":"v1","addr":"172.21.2.120:6818","nonce":36387}]},"public_addr":"172.21.2.120:6806/36387","cluster_addr":"172.21.2.120:6812/36387","heartbeat_back_addr":"172.21.2.120:6822/36387","heartbeat_front_addr":"172.21.2.120:6818/36387","state":["exists","up"]},{"osd":6,"uuid":"b010e187-4541-4b00-80dd-f948f6ab8f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":196,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6824","nonce":36385},{"type":"v1","addr":"172.21.2.120:6826","nonce":36385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6828","nonce":36385},{"type":"v1","addr":"172.21.2.120:6829","nonce":36385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6832","nonce":36385},{"type":"v1","addr":"172.21.2.120:6833","nonce":36385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":36385},{"type":"v1","addr":"172.21.2.120:6831","nonce":36385}]},"public_addr":"172.21.2.120:6826/36385","cluster_addr":"172.21.2.120:6829/36385","heartbeat_back_addr":"172.21.2.120:6833/36385","heartbeat_front_addr":"172.21.2.120:6831/36385","state":["exists","up"]},{"osd":7,"uuid":"671e2772-33b7-497e-8e44-2178bd850bcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":36386},{"type":"v1","addr":"172.21.2.120:6803","nonce":36386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6805","nonce":36386},{"type":"v1","addr":"172.21.2.120:6807","nonce":36386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":36386},{"type":"v1","addr":"172.21.2.120:6817","nonce":36386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":36386},{"type":"v1","addr":"172.21.2.120:6811","nonce":36386}]},"public_addr":"172.21.2.120:6803/36386","cluster_addr":"172.21.2.120:6807/36386","heartbeat_back_addr":"172.21.2.120:6817/36386","heartbeat_front_addr":"172.21.2.120:6811/36386","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:50:27.910 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.905 7fbf9ee17700 1 -- 172.21.2.112:0/3874715940 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fbf7c0186c0 msgr2=0x7fbf7c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:27.910 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.905 7fbf9ee17700 1 --2- 172.21.2.112:0/3874715940 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fbf7c0186c0 0x7fbf7c01abc0 crc :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:27.910 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.905 7fbf9ee17700 1 -- 172.21.2.112:0/3874715940 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbf98095cb0 msgr2=0x7fbf980960b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:27.910 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.905 7fbf9ee17700 1 --2- 172.21.2.112:0/3874715940 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbf98095cb0 0x7fbf980960b0 secure :-1 s=READY pgs=3330 cs=0 l=1 rev1=1 rx=0x7fbf88009980 tx=0x7fbf8800bcd0).stop 2021-02-10T11:50:27.910 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.906 7fbf9ee17700 1 -- 172.21.2.112:0/3874715940 shutdown_connections 2021-02-10T11:50:27.910 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.906 7fbf9ee17700 1 --2- 172.21.2.112:0/3874715940 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fbf7c0186c0 0x7fbf7c01abc0 unknown :-1 s=CLOSED pgs=1498 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:27.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.906 7fbf9ee17700 1 --2- 172.21.2.112:0/3874715940 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fbf980966d0 0x7fbf980b16f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:27.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.906 7fbf9ee17700 1 --2- 172.21.2.112:0/3874715940 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fbf98095cb0 0x7fbf980960b0 unknown :-1 s=CLOSED pgs=3330 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:27.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.906 7fbf9ee17700 1 --2- 172.21.2.112:0/3874715940 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fbf9819ab70 0x7fbf98096ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:27.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.906 7fbf9ee17700 1 -- 172.21.2.112:0/3874715940 >> 172.21.2.112:0/3874715940 conn(0x7fbf981b1e40 msgr2=0x7fbf981a71e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:27.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.906 7fbf9ee17700 1 -- 172.21.2.112:0/3874715940 shutdown_connections 2021-02-10T11:50:27.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:27.906 7fbf9ee17700 1 -- 172.21.2.112:0/3874715940 wait complete. 2021-02-10T11:50:28.919 INFO:tasks.ceph:Scrubbing osd.0 2021-02-10T11:50:28.920 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2021-02-10T11:50:29.355 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.332 7f02b1e02700 1 Processor -- start 2021-02-10T11:50:29.355 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.332 7f02b1e02700 1 -- start start 2021-02-10T11:50:29.355 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.333 7f02b1e02700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f02ac10b000 0x7f02ac0cf760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:29.355 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.333 7f02b1e02700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02ac0fe1d0 con 0x7f02ac0d1c80 2021-02-10T11:50:29.356 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.333 7f02b1e02700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02ac1b0cc0 con 0x7f02ac0d1880 2021-02-10T11:50:29.356 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.333 7f02b1e02700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02ac109a70 con 0x7f02ac10b000 2021-02-10T11:50:29.356 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.333 7f02aaffd700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f02ac0d1c80 0x7f02ac0cef70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:33460/0 (socket says 172.21.2.112:33460) 2021-02-10T11:50:29.356 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.333 7f02aaffd700 1 -- 172.21.2.112:0/3031924068 learned_addr learned my addr 172.21.2.112:0/3031924068 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:50:29.356 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.333 7f02abfff700 1 --2- 172.21.2.112:0/3031924068 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f02ac10b000 0x7f02ac0cf760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:29.356 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.333 7f02aa7fc700 1 -- 172.21.2.112:0/3031924068 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 915233753 0 0) 0x7f02a0001180 con 0x7f02ac0d1c80 2021-02-10T11:50:29.356 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.334 7f02aa7fc700 1 -- 172.21.2.112:0/3031924068 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0294002fe0 con 0x7f02ac0d1c80 2021-02-10T11:50:29.357 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.334 7f02aa7fc700 1 -- 172.21.2.112:0/3031924068 <== mon.1 v1:172.21.2.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 323357841 0 0) 0x7f029c001080 con 0x7f02ac0d1880 2021-02-10T11:50:29.357 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.334 7f02aa7fc700 1 -- 172.21.2.112:0/3031924068 --> v1:172.21.2.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0294003210 con 0x7f02ac0d1880 2021-02-10T11:50:29.357 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.334 7f02aa7fc700 1 -- 172.21.2.112:0/3031924068 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3056915380 0 0) 0x7f02a0001180 con 0x7f02ac0d1c80 2021-02-10T11:50:29.357 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.334 7f02aa7fc700 1 -- 172.21.2.112:0/3031924068 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f02ac10b000 msgr2=0x7f02ac0cf760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:50:29.357 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.334 7f02aa7fc700 1 --2- 172.21.2.112:0/3031924068 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f02ac10b000 0x7f02ac0cf760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:29.357 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.334 7f02aa7fc700 1 -- 172.21.2.112:0/3031924068 >> v1:172.21.2.120:6789/0 conn(0x7f02ac0d1880 legacy=0x7f02ac1b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:29.357 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.334 7f02aa7fc700 1 -- 172.21.2.112:0/3031924068 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02ac10b400 con 0x7f02ac0d1c80 2021-02-10T11:50:29.358 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.334 7f02aa7fc700 1 -- 172.21.2.112:0/3031924068 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f02a0001680 con 0x7f02ac0d1c80 2021-02-10T11:50:29.358 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.334 7f02aa7fc700 1 -- 172.21.2.112:0/3031924068 >> v1:172.21.2.112:6789/0 conn(0x7f02ac0d1c80 legacy=0x7f02ac0cef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:29.358 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.334 7f02aa7fc700 1 --2- 172.21.2.112:0/3031924068 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0294003bb0 0x7f0294005fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:29.358 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.334 7f02aa7fc700 1 --2- 172.21.2.112:0/3031924068 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0294006840 0x7f0294008d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:29.358 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.334 7f02aa7fc700 1 --2- 172.21.2.112:0/3031924068 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0294009660 0x7f029400bb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:29.358 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.334 7f02aa7fc700 1 -- 172.21.2.112:0/3031924068 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f029400c380 con 0x7f0294003bb0 2021-02-10T11:50:29.358 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.334 7f02aa7fc700 1 -- 172.21.2.112:0/3031924068 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0294050570 con 0x7f0294006840 2021-02-10T11:50:29.358 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.334 7f02aa7fc700 1 -- 172.21.2.112:0/3031924068 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0294050790 con 0x7f0294009660 2021-02-10T11:50:29.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.334 7f02abfff700 1 --2- 172.21.2.112:0/3031924068 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f02ac10b000 0x7f02ac0cf760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:50:29.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.334 7f02ab7fe700 1 --2- 172.21.2.112:0/3031924068 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0294003bb0 0x7f0294005fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:29.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.334 7f02aaffd700 1 --2- 172.21.2.112:0/3031924068 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0294006840 0x7f0294008d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:29.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.335 7f02ab7fe700 1 -- 172.21.2.112:0/3031924068 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0294009660 msgr2=0x7f029400bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:50:29.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.335 7f02ab7fe700 1 --2- 172.21.2.112:0/3031924068 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f0294009660 0x7f029400bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:29.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.335 7f02ab7fe700 1 -- 172.21.2.112:0/3031924068 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0294006840 msgr2=0x7f0294008d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:50:29.359 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.335 7f02ab7fe700 1 --2- 172.21.2.112:0/3031924068 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f0294006840 0x7f0294008d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:29.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.335 7f02ab7fe700 1 -- 172.21.2.112:0/3031924068 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0294050a30 con 0x7f0294003bb0 2021-02-10T11:50:29.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.335 7f02ab7fe700 1 --2- 172.21.2.112:0/3031924068 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0294003bb0 0x7f0294005fc0 secure :-1 s=READY pgs=3332 cs=0 l=1 rev1=1 rx=0x7f029c006ad0 tx=0x7f029c009b40).ready entity=mon.0 client_cookie=edd86f4b018752c9 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:29.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.335 7f02aa7fc700 1 -- 172.21.2.112:0/3031924068 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f029c000c10 con 0x7f0294003bb0 2021-02-10T11:50:29.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.335 7f02aa7fc700 1 -- 172.21.2.112:0/3031924068 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f029c00b2e0 con 0x7f0294003bb0 2021-02-10T11:50:29.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.335 7f02aa7fc700 1 -- 172.21.2.112:0/3031924068 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f029c00b800 con 0x7f0294003bb0 2021-02-10T11:50:29.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.335 7f02b1e02700 1 -- 172.21.2.112:0/3031924068 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0294003bb0 msgr2=0x7f0294005fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:29.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.335 7f02b1e02700 1 --2- 172.21.2.112:0/3031924068 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f0294003bb0 0x7f0294005fc0 secure :-1 s=READY pgs=3332 cs=0 l=1 rev1=1 rx=0x7f029c006ad0 tx=0x7f029c009b40).stop 2021-02-10T11:50:29.360 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.335 7f02abfff700 1 -- 172.21.2.112:0/3031924068 reap_dead start 2021-02-10T11:50:29.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.336 7f02b1e02700 1 -- 172.21.2.112:0/3031924068 shutdown_connections 2021-02-10T11:50:29.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.336 7f02b1e02700 1 -- 172.21.2.112:0/3031924068 >> 172.21.2.112:0/3031924068 conn(0x7f02ac0ce430 msgr2=0x7f02ac0d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:29.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.336 7f02b1e02700 1 -- 172.21.2.112:0/3031924068 shutdown_connections 2021-02-10T11:50:29.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.336 7f02b1e02700 1 -- 172.21.2.112:0/3031924068 wait complete. 2021-02-10T11:50:29.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.350 7f02b1e02700 1 Processor -- start 2021-02-10T11:50:29.361 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.350 7f02b1e02700 1 -- start start 2021-02-10T11:50:29.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.351 7f02b1e02700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f02ac1a3570 0x7f02ac1a3970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:29.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.351 7f02b1e02700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f02ac09c9f0 0x7f02ac09cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:29.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.351 7f02b1e02700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f02ac09d410 0x7f02ac098170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:29.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.351 7f02b1e02700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02ac09c470 con 0x7f02ac09d410 2021-02-10T11:50:29.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.351 7f02b1e02700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02ac0b2840 con 0x7f02ac09c9f0 2021-02-10T11:50:29.363 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.351 7f02b1e02700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02ac0b2a20 con 0x7f02ac1a3570 2021-02-10T11:50:29.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.351 7f02abfff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f02ac09d410 0x7f02ac098170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:29.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.351 7f02abfff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f02ac09d410 0x7f02ac098170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:34014/0 (socket says 172.21.2.112:34014) 2021-02-10T11:50:29.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.351 7f02abfff700 1 -- 172.21.2.112:0/1633816704 learned_addr learned my addr 172.21.2.112:0/1633816704 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:50:29.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.351 7f02abfff700 1 -- 172.21.2.112:0/1633816704 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f02ac1a3570 msgr2=0x7f02ac1a3970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:29.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.351 7f02abfff700 1 --2- 172.21.2.112:0/1633816704 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f02ac1a3570 0x7f02ac1a3970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:29.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.351 7f02abfff700 1 -- 172.21.2.112:0/1633816704 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f02ac09c9f0 msgr2=0x7f02ac09cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:29.364 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.351 7f02abfff700 1 --2- 172.21.2.112:0/1633816704 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f02ac09c9f0 0x7f02ac09cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:29.365 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.351 7f02abfff700 1 -- 172.21.2.112:0/1633816704 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02ac09d880 con 0x7f02ac09d410 2021-02-10T11:50:29.365 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.352 7f02abfff700 1 --2- 172.21.2.112:0/1633816704 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f02ac09d410 0x7f02ac098170 secure :-1 s=READY pgs=3333 cs=0 l=1 rev1=1 rx=0x7f02a400b970 tx=0x7f02a400bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:29.365 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.352 7f0293fff700 1 -- 172.21.2.112:0/1633816704 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f02a4008030 con 0x7f02ac09d410 2021-02-10T11:50:29.365 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.352 7f0293fff700 1 -- 172.21.2.112:0/1633816704 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02a4008270 con 0x7f02ac09d410 2021-02-10T11:50:29.365 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.352 7f0293fff700 1 -- 172.21.2.112:0/1633816704 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f02a4008790 con 0x7f02ac09d410 2021-02-10T11:50:29.365 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.352 7f02b1e02700 1 -- 172.21.2.112:0/1633816704 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02ac09d880 con 0x7f02ac09d410 2021-02-10T11:50:29.365 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.352 7f02b1e02700 1 -- 172.21.2.112:0/1633816704 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02ac098a80 con 0x7f02ac09d410 2021-02-10T11:50:29.366 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.353 7f0293fff700 1 -- 172.21.2.112:0/1633816704 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f02a4008450 con 0x7f02ac09d410 2021-02-10T11:50:29.366 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.353 7f0293fff700 1 --2- 172.21.2.112:0/1633816704 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f028c0186c0 0x7f028c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:29.366 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.353 7f0293fff700 1 -- 172.21.2.112:0/1633816704 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f02a4032890 con 0x7f02ac09d410 2021-02-10T11:50:29.366 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.353 7f02ab7fe700 1 --2- 172.21.2.112:0/1633816704 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f028c0186c0 0x7f028c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:29.366 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.354 7f02ab7fe700 1 --2- 172.21.2.112:0/1633816704 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f028c0186c0 0x7f028c01abc0 crc :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:29.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.383 7f02b1e02700 1 --2- 172.21.2.112:0/1633816704 >> [v2:172.21.2.112:6817/36030,v1:172.21.2.112:6818/36030] conn(0x7f02ac0c4620 0x7f02ac099250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:29.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.383 7f02b1e02700 1 -- 172.21.2.112:0/1633816704 --> [v2:172.21.2.112:6817/36030,v1:172.21.2.112:6818/36030] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f02ac0b2840 con 0x7f02ac0c4620 2021-02-10T11:50:29.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.383 7f02aaffd700 1 --2- 172.21.2.112:0/1633816704 >> [v2:172.21.2.112:6817/36030,v1:172.21.2.112:6818/36030] conn(0x7f02ac0c4620 0x7f02ac099250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:29.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.383 7f02aaffd700 1 --2- 172.21.2.112:0/1633816704 >> [v2:172.21.2.112:6817/36030,v1:172.21.2.112:6818/36030] conn(0x7f02ac0c4620 0x7f02ac099250 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:29.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.384 7f0293fff700 1 -- 172.21.2.112:0/1633816704 <== osd.0 v2:172.21.2.112:6817/36030 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+5151 (crc 0 0 0) 0x7f02a000b050 con 0x7f02ac0c4620 2021-02-10T11:50:29.395 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.391 7f02b1e02700 1 -- 172.21.2.112:0/1633816704 --> [v2:172.21.2.112:6817/36030,v1:172.21.2.112:6818/36030] -- command(tid 2: {"prefix": "config set", "key": "osd_debug_deep_scrub_sleep", "value": "0"}) v1 -- 0x7f02ac098a80 con 0x7f02ac0c4620 2021-02-10T11:50:29.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.392 7f0293fff700 1 -- 172.21.2.112:0/1633816704 <== osd.0 v2:172.21.2.112:6817/36030 2 ==== command_reply(tid 2: 0 Set osd_debug_deep_scrub_sleep to 0) v1 ==== 43+0+0 (crc 0 0 0) 0x7f02a0006e00 con 0x7f02ac0c4620 2021-02-10T11:50:29.396 INFO:teuthology.orchestra.run.gibba012.stderr:Set osd_debug_deep_scrub_sleep to 0 2021-02-10T11:50:29.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.393 7f02b1e02700 1 -- 172.21.2.112:0/1633816704 >> [v2:172.21.2.112:6817/36030,v1:172.21.2.112:6818/36030] conn(0x7f02ac0c4620 msgr2=0x7f02ac099250 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:29.398 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.393 7f02b1e02700 1 --2- 172.21.2.112:0/1633816704 >> [v2:172.21.2.112:6817/36030,v1:172.21.2.112:6818/36030] conn(0x7f02ac0c4620 0x7f02ac099250 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:29.398 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.393 7f02b1e02700 1 -- 172.21.2.112:0/1633816704 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f028c0186c0 msgr2=0x7f028c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:29.398 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.393 7f02b1e02700 1 --2- 172.21.2.112:0/1633816704 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f028c0186c0 0x7f028c01abc0 crc :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:29.398 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.393 7f02b1e02700 1 -- 172.21.2.112:0/1633816704 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f02ac09d410 msgr2=0x7f02ac098170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:29.398 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.393 7f02b1e02700 1 --2- 172.21.2.112:0/1633816704 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f02ac09d410 0x7f02ac098170 secure :-1 s=READY pgs=3333 cs=0 l=1 rev1=1 rx=0x7f02a400b970 tx=0x7f02a400bd10).stop 2021-02-10T11:50:29.398 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.393 7f02abfff700 1 -- 172.21.2.112:0/1633816704 reap_dead start 2021-02-10T11:50:29.399 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.393 7f02b1e02700 1 -- 172.21.2.112:0/1633816704 shutdown_connections 2021-02-10T11:50:29.399 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.393 7f02b1e02700 1 -- 172.21.2.112:0/1633816704 >> 172.21.2.112:0/1633816704 conn(0x7f02ac0c2cd0 msgr2=0x7f02ac0b5950 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:29.399 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.393 7f02b1e02700 1 -- 172.21.2.112:0/1633816704 shutdown_connections 2021-02-10T11:50:29.399 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.393 7f02b1e02700 1 -- 172.21.2.112:0/1633816704 wait complete. 2021-02-10T11:50:29.405 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd deep-scrub 0 2021-02-10T11:50:29.436 DEBUG:teuthology.orchestra.run.gibba012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:50:29.451 DEBUG:teuthology.orchestra.run.gibba020:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:50:29.836 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.816 7f56a6477700 1 Processor -- start 2021-02-10T11:50:29.837 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.816 7f56a6477700 1 -- start start 2021-02-10T11:50:29.837 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.816 7f56a6477700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f56a00d1870 0x7f56a01b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:29.837 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.816 7f56a6477700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f56a01b0a10 0x7f56a00cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:29.837 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.816 7f56a6477700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56a00f0b10 con 0x7f56a00d1c70 2021-02-10T11:50:29.837 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.816 7f56a6477700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56a00e5ed0 con 0x7f56a00d1870 2021-02-10T11:50:29.837 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.816 7f56a6477700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56a00f0650 con 0x7f56a01b0a10 2021-02-10T11:50:29.837 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.817 7f569ffff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f56a00d1870 0x7f56a01b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:29.838 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.817 7f569ffff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f56a00d1870 0x7f56a01b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:34020/0 (socket says 172.21.2.112:34020) 2021-02-10T11:50:29.838 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.817 7f569ffff700 1 -- 172.21.2.112:0/4225083168 learned_addr learned my addr 172.21.2.112:0/4225083168 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:50:29.838 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.817 7f569ffff700 1 -- 172.21.2.112:0/4225083168 >> v2:172.21.2.120:3300/0 conn(0x7f56a01b0a10 msgr2=0x7f56a00cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:50:29.838 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.817 7f569ffff700 1 --2- 172.21.2.112:0/4225083168 >> v2:172.21.2.120:3300/0 conn(0x7f56a01b0a10 0x7f56a00cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:29.838 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.817 7f569ffff700 1 -- 172.21.2.112:0/4225083168 >> v1:172.21.2.120:6789/0 conn(0x7f56a00d1c70 legacy=0x7f56a00cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:50:29.838 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.817 7f569ffff700 1 -- 172.21.2.112:0/4225083168 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56a00f0840 con 0x7f56a00d1870 2021-02-10T11:50:29.838 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.817 7f569ffff700 1 --2- 172.21.2.112:0/4225083168 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f56a00d1870 0x7f56a01b03f0 secure :-1 s=READY pgs=3334 cs=0 l=1 rev1=1 rx=0x7f5690006700 tx=0x7f5690009b00).ready entity=mon.0 client_cookie=d683b24ee653645d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:29.838 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.817 7f569effd700 1 -- 172.21.2.112:0/4225083168 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f569000c0c0 con 0x7f56a00d1870 2021-02-10T11:50:29.839 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.817 7f569effd700 1 -- 172.21.2.112:0/4225083168 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f569000c400 con 0x7f56a00d1870 2021-02-10T11:50:29.839 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.817 7f569effd700 1 -- 172.21.2.112:0/4225083168 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f569000c860 con 0x7f56a00d1870 2021-02-10T11:50:29.839 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.817 7f56a6477700 1 -- 172.21.2.112:0/4225083168 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f56a00d1870 msgr2=0x7f56a01b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:29.839 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.817 7f56a6477700 1 --2- 172.21.2.112:0/4225083168 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f56a00d1870 0x7f56a01b03f0 secure :-1 s=READY pgs=3334 cs=0 l=1 rev1=1 rx=0x7f5690006700 tx=0x7f5690009b00).stop 2021-02-10T11:50:29.839 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.818 7f56a6477700 1 -- 172.21.2.112:0/4225083168 shutdown_connections 2021-02-10T11:50:29.839 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.818 7f56a6477700 1 --2- 172.21.2.112:0/4225083168 >> v2:172.21.2.120:3300/0 conn(0x7f56a01b0a10 0x7f56a00cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:29.839 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.818 7f56a6477700 1 --2- 172.21.2.112:0/4225083168 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f56a00d1870 0x7f56a01b03f0 unknown :-1 s=CLOSED pgs=3334 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:29.839 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.818 7f56a6477700 1 -- 172.21.2.112:0/4225083168 >> 172.21.2.112:0/4225083168 conn(0x7f56a00ce450 msgr2=0x7f56a00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:29.839 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.818 7f56a6477700 1 -- 172.21.2.112:0/4225083168 shutdown_connections 2021-02-10T11:50:29.840 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.818 7f56a6477700 1 -- 172.21.2.112:0/4225083168 wait complete. 2021-02-10T11:50:29.840 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.832 7f56a6477700 1 Processor -- start 2021-02-10T11:50:29.841 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.833 7f56a6477700 1 -- start start 2021-02-10T11:50:29.841 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.833 7f56a6477700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f56a00bdda0 0x7f56a018b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:29.842 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.833 7f56a6477700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f56a018b870 0x7f56a00c0030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:29.842 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.833 7f56a6477700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f56a018bc70 0x7f56a00c0650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:29.842 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.833 7f56a6477700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56a019d390 con 0x7f56a018b870 2021-02-10T11:50:29.842 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.833 7f56a6477700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56a00bd740 con 0x7f56a00bdda0 2021-02-10T11:50:29.842 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.833 7f56a6477700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56a00be1a0 con 0x7f56a018bc70 2021-02-10T11:50:29.842 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.833 7f569f7fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f56a018b870 0x7f56a00c0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:29.842 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.833 7f569f7fe700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f56a018b870 0x7f56a00c0030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:34028/0 (socket says 172.21.2.112:34028) 2021-02-10T11:50:29.842 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.833 7f569f7fe700 1 -- 172.21.2.112:0/259731930 learned_addr learned my addr 172.21.2.112:0/259731930 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:50:29.843 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.833 7f569ffff700 1 --2- 172.21.2.112:0/259731930 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f56a00bdda0 0x7f56a018b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:29.843 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.833 7f56a4a94700 1 --2- 172.21.2.112:0/259731930 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f56a018bc70 0x7f56a00c0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:29.843 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.834 7f569f7fe700 1 -- 172.21.2.112:0/259731930 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f56a018bc70 msgr2=0x7f56a00c0650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:29.843 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.834 7f569f7fe700 1 --2- 172.21.2.112:0/259731930 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f56a018bc70 0x7f56a00c0650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:29.843 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.834 7f569f7fe700 1 -- 172.21.2.112:0/259731930 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f56a00bdda0 msgr2=0x7f56a018b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:29.844 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.834 7f569f7fe700 1 --2- 172.21.2.112:0/259731930 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f56a00bdda0 0x7f56a018b250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:29.844 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.834 7f569f7fe700 1 -- 172.21.2.112:0/259731930 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56a00c0c70 con 0x7f56a018b870 2021-02-10T11:50:29.844 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.834 7f569f7fe700 1 --2- 172.21.2.112:0/259731930 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f56a018b870 0x7f56a00c0030 secure :-1 s=READY pgs=3335 cs=0 l=1 rev1=1 rx=0x7f5694006f00 tx=0x7f569400b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:29.844 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.834 7f569cff9700 1 -- 172.21.2.112:0/259731930 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f569400bd30 con 0x7f56a018b870 2021-02-10T11:50:29.844 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.834 7f569cff9700 1 -- 172.21.2.112:0/259731930 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5694000c90 con 0x7f56a018b870 2021-02-10T11:50:29.844 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.834 7f56a6477700 1 -- 172.21.2.112:0/259731930 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56a00c0c70 con 0x7f56a018b870 2021-02-10T11:50:29.844 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.834 7f569cff9700 1 -- 172.21.2.112:0/259731930 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f56940022a0 con 0x7f56a018b870 2021-02-10T11:50:29.844 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.834 7f56a6477700 1 -- 172.21.2.112:0/259731930 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56a00c1200 con 0x7f56a018b870 2021-02-10T11:50:29.845 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.835 7f569cff9700 1 -- 172.21.2.112:0/259731930 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5694005310 con 0x7f56a018b870 2021-02-10T11:50:29.845 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.835 7f569cff9700 1 --2- 172.21.2.112:0/259731930 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f56800186c0 0x7f568001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:29.845 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.835 7f569cff9700 1 -- 172.21.2.112:0/259731930 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f5694032580 con 0x7f56a018b870 2021-02-10T11:50:29.845 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.835 7f569ffff700 1 --2- 172.21.2.112:0/259731930 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f56800186c0 0x7f568001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:29.845 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.835 7f569ffff700 1 --2- 172.21.2.112:0/259731930 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f56800186c0 0x7f568001abc0 crc :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:29.870 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.866 7f56a6477700 1 -- 172.21.2.112:0/259731930 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56a00ee0b0 con 0x7f56a018b870 2021-02-10T11:50:29.873 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.869 7f569cff9700 1 -- 172.21.2.112:0/259731930 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5694000960 con 0x7f56a018b870 2021-02-10T11:50:30.000 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.996 7f56a6477700 1 -- 172.21.2.112:0/259731930 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mgr", ""]}) v1 -- 0x7f56a00c1200 con 0x7f56800186c0 2021-02-10T11:50:30.001 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.997 7f569cff9700 1 -- 172.21.2.112:0/259731930 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (crc 0 0 0) 0x7f5690000c90 con 0x7f56800186c0 2021-02-10T11:50:30.001 INFO:teuthology.orchestra.run.gibba012.stderr:instructed osd(s) 0 to deep-scrub 2021-02-10T11:50:30.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.998 7f56a6477700 1 -- 172.21.2.112:0/259731930 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f56800186c0 msgr2=0x7f568001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:30.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.998 7f56a6477700 1 --2- 172.21.2.112:0/259731930 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f56800186c0 0x7f568001abc0 crc :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:30.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.998 7f56a6477700 1 -- 172.21.2.112:0/259731930 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f56a018b870 msgr2=0x7f56a00c0030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:30.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.998 7f56a6477700 1 --2- 172.21.2.112:0/259731930 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f56a018b870 0x7f56a00c0030 secure :-1 s=READY pgs=3335 cs=0 l=1 rev1=1 rx=0x7f5694006f00 tx=0x7f569400b940).stop 2021-02-10T11:50:30.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.998 7f56a6477700 1 -- 172.21.2.112:0/259731930 shutdown_connections 2021-02-10T11:50:30.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.998 7f56a6477700 1 --2- 172.21.2.112:0/259731930 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f56800186c0 0x7f568001abc0 unknown :-1 s=CLOSED pgs=1500 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:30.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.998 7f56a6477700 1 --2- 172.21.2.112:0/259731930 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f56a018bc70 0x7f56a00c0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:30.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.998 7f56a6477700 1 --2- 172.21.2.112:0/259731930 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f56a018b870 0x7f56a00c0030 unknown :-1 s=CLOSED pgs=3335 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:30.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.998 7f56a6477700 1 --2- 172.21.2.112:0/259731930 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f56a00bdda0 0x7f56a018b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:30.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.998 7f56a6477700 1 -- 172.21.2.112:0/259731930 >> 172.21.2.112:0/259731930 conn(0x7f56a00a6eb0 msgr2=0x7f56a0098ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:30.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.999 7f56a6477700 1 -- 172.21.2.112:0/259731930 shutdown_connections 2021-02-10T11:50:30.005 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:29.999 7f56a6477700 1 -- 172.21.2.112:0/259731930 wait complete. 2021-02-10T11:50:30.012 INFO:tasks.ceph:Scrubbing osd.1 2021-02-10T11:50:30.012 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2021-02-10T11:50:30.446 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.425 7f78d481c700 1 Processor -- start 2021-02-10T11:50:30.447 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.425 7f78d481c700 1 -- start start 2021-02-10T11:50:30.447 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.426 7f78d481c700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f78d00d1880 0x7f78d01b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:30.447 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.426 7f78d481c700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f78d01b0af0 0x7f78d0108950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:30.447 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.426 7f78d481c700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78d00fe1d0 con 0x7f78d00d1c80 2021-02-10T11:50:30.447 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.426 7f78d481c700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78d0109a70 con 0x7f78d00d1880 2021-02-10T11:50:30.447 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.426 7f78d481c700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78d0109760 con 0x7f78d01b0af0 2021-02-10T11:50:30.448 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.426 7f78ce61c700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f78d00d1880 0x7f78d01b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:30.448 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.426 7f78cee1d700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f78d00d1c80 0x7f78d0108f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:33494/0 (socket says 172.21.2.112:33494) 2021-02-10T11:50:30.448 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.426 7f78cee1d700 1 -- 172.21.2.112:0/125586210 learned_addr learned my addr 172.21.2.112:0/125586210 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:50:30.448 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.427 7f78ce61c700 1 -- 172.21.2.112:0/125586210 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f78d01b0af0 msgr2=0x7f78d0108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:50:30.448 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.427 7f78ce61c700 1 --2- 172.21.2.112:0/125586210 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f78d01b0af0 0x7f78d0108950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:30.448 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.427 7f78ce61c700 1 -- 172.21.2.112:0/125586210 >> v1:172.21.2.112:6789/0 conn(0x7f78d00d1c80 legacy=0x7f78d0108f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:30.448 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.427 7f78ce61c700 1 -- 172.21.2.112:0/125586210 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78d0088070 con 0x7f78d00d1880 2021-02-10T11:50:30.448 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.427 7f78ce61c700 1 --2- 172.21.2.112:0/125586210 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f78d00d1880 0x7f78d01b04d0 secure :-1 s=READY pgs=3337 cs=0 l=1 rev1=1 rx=0x7f78c0006650 tx=0x7f78c0009a80).ready entity=mon.0 client_cookie=708c533d572a43f8 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:30.449 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.427 7f78cd61a700 1 -- 172.21.2.112:0/125586210 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f78c000c0c0 con 0x7f78d00d1880 2021-02-10T11:50:30.449 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.427 7f78cd61a700 1 -- 172.21.2.112:0/125586210 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78c000c340 con 0x7f78d00d1880 2021-02-10T11:50:30.449 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.427 7f78cd61a700 1 -- 172.21.2.112:0/125586210 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f78c000c860 con 0x7f78d00d1880 2021-02-10T11:50:30.449 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.427 7f78d481c700 1 -- 172.21.2.112:0/125586210 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f78d00d1880 msgr2=0x7f78d01b04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:30.449 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.427 7f78d481c700 1 --2- 172.21.2.112:0/125586210 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f78d00d1880 0x7f78d01b04d0 secure :-1 s=READY pgs=3337 cs=0 l=1 rev1=1 rx=0x7f78c0006650 tx=0x7f78c0009a80).stop 2021-02-10T11:50:30.449 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.428 7f78d481c700 1 -- 172.21.2.112:0/125586210 shutdown_connections 2021-02-10T11:50:30.449 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.428 7f78d481c700 1 --2- 172.21.2.112:0/125586210 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f78d01b0af0 0x7f78d0108950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:30.449 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.428 7f78d481c700 1 --2- 172.21.2.112:0/125586210 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f78d00d1880 0x7f78d01b04d0 unknown :-1 s=CLOSED pgs=3337 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:30.450 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.428 7f78d481c700 1 -- 172.21.2.112:0/125586210 >> 172.21.2.112:0/125586210 conn(0x7f78d00ce430 msgr2=0x7f78d00d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:30.450 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.428 7f78d481c700 1 -- 172.21.2.112:0/125586210 shutdown_connections 2021-02-10T11:50:30.450 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.428 7f78d481c700 1 -- 172.21.2.112:0/125586210 wait complete. 2021-02-10T11:50:30.451 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.442 7f78d481c700 1 Processor -- start 2021-02-10T11:50:30.452 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.443 7f78d481c700 1 -- start start 2021-02-10T11:50:30.452 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.443 7f78d481c700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f78d018d050 0x7f78d01b3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:30.452 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.443 7f78d481c700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f78d00ac100 0x7f78d00ac500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:30.452 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.443 7f78d481c700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f78d018b400 0x7f78d018b800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:30.452 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.443 7f78d481c700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78d018d450 con 0x7f78d018b400 2021-02-10T11:50:30.452 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.443 7f78d481c700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78d018be20 con 0x7f78d018d050 2021-02-10T11:50:30.452 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.443 7f78d481c700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78d018c040 con 0x7f78d00ac100 2021-02-10T11:50:30.453 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.443 7f78cee1d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f78d018b400 0x7f78d018b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:30.453 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.443 7f78cee1d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f78d018b400 0x7f78d018b800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:34038/0 (socket says 172.21.2.112:34038) 2021-02-10T11:50:30.453 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.443 7f78cee1d700 1 -- 172.21.2.112:0/1503288417 learned_addr learned my addr 172.21.2.112:0/1503288417 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:50:30.453 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.444 7f78cee1d700 1 -- 172.21.2.112:0/1503288417 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f78d00ac100 msgr2=0x7f78d00ac500 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:50:30.453 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.444 7f78cee1d700 1 --2- 172.21.2.112:0/1503288417 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f78d00ac100 0x7f78d00ac500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:30.453 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.444 7f78cee1d700 1 -- 172.21.2.112:0/1503288417 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f78d018d050 msgr2=0x7f78d01b3600 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:50:30.453 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.444 7f78cee1d700 1 --2- 172.21.2.112:0/1503288417 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f78d018d050 0x7f78d01b3600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:30.453 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.444 7f78cee1d700 1 -- 172.21.2.112:0/1503288417 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78d0193c40 con 0x7f78d018b400 2021-02-10T11:50:30.454 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.444 7f78cee1d700 1 --2- 172.21.2.112:0/1503288417 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f78d018b400 0x7f78d018b800 secure :-1 s=READY pgs=3338 cs=0 l=1 rev1=1 rx=0x7f78c800a940 tx=0x7f78c800ace0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:30.454 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.444 7f78beffd700 1 -- 172.21.2.112:0/1503288417 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f78c8001270 con 0x7f78d018b400 2021-02-10T11:50:30.454 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.444 7f78beffd700 1 -- 172.21.2.112:0/1503288417 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78c8007020 con 0x7f78d018b400 2021-02-10T11:50:30.454 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.444 7f78beffd700 1 -- 172.21.2.112:0/1503288417 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f78c8007500 con 0x7f78d018b400 2021-02-10T11:50:30.454 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.444 7f78d481c700 1 -- 172.21.2.112:0/1503288417 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78d0193c40 con 0x7f78d018b400 2021-02-10T11:50:30.454 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.444 7f78d481c700 1 -- 172.21.2.112:0/1503288417 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78d01941a0 con 0x7f78d018b400 2021-02-10T11:50:30.454 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.445 7f78beffd700 1 -- 172.21.2.112:0/1503288417 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f78c8007200 con 0x7f78d018b400 2021-02-10T11:50:30.454 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.445 7f78beffd700 1 --2- 172.21.2.112:0/1503288417 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f78b40186c0 0x7f78b401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:30.455 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.445 7f78beffd700 1 -- 172.21.2.112:0/1503288417 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f78c80328f0 con 0x7f78d018b400 2021-02-10T11:50:30.455 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.445 7f78ce61c700 1 --2- 172.21.2.112:0/1503288417 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f78b40186c0 0x7f78b401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:30.455 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.446 7f78ce61c700 1 --2- 172.21.2.112:0/1503288417 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f78b40186c0 0x7f78b401abc0 crc :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:30.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.476 7f78d481c700 1 --2- 172.21.2.112:0/1503288417 >> [v2:172.21.2.112:6800/36029,v1:172.21.2.112:6802/36029] conn(0x7f78d018a330 0x7f78d018a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:30.481 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.476 7f78d481c700 1 -- 172.21.2.112:0/1503288417 --> [v2:172.21.2.112:6800/36029,v1:172.21.2.112:6802/36029] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f78d01941a0 con 0x7f78d018a330 2021-02-10T11:50:30.481 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.476 7f78cde1b700 1 --2- 172.21.2.112:0/1503288417 >> [v2:172.21.2.112:6800/36029,v1:172.21.2.112:6802/36029] conn(0x7f78d018a330 0x7f78d018a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:30.481 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.476 7f78cde1b700 1 --2- 172.21.2.112:0/1503288417 >> [v2:172.21.2.112:6800/36029,v1:172.21.2.112:6802/36029] conn(0x7f78d018a330 0x7f78d018a730 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:30.481 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.477 7f78beffd700 1 -- 172.21.2.112:0/1503288417 <== osd.1 v2:172.21.2.112:6800/36029 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+5151 (crc 0 0 0) 0x7f78c4006960 con 0x7f78d018a330 2021-02-10T11:50:30.487 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.483 7f78d481c700 1 -- 172.21.2.112:0/1503288417 --> [v2:172.21.2.112:6800/36029,v1:172.21.2.112:6802/36029] -- command(tid 2: {"prefix": "config set", "key": "osd_debug_deep_scrub_sleep", "value": "0"}) v1 -- 0x7f78d018be20 con 0x7f78d018a330 2021-02-10T11:50:30.487 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.483 7f78beffd700 1 -- 172.21.2.112:0/1503288417 <== osd.1 v2:172.21.2.112:6800/36029 2 ==== command_reply(tid 2: 0 Set osd_debug_deep_scrub_sleep to 0) v1 ==== 43+0+0 (crc 0 0 0) 0x7f78c4006960 con 0x7f78d018a330 2021-02-10T11:50:30.488 INFO:teuthology.orchestra.run.gibba012.stderr:Set osd_debug_deep_scrub_sleep to 0 2021-02-10T11:50:30.489 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.484 7f78d481c700 1 -- 172.21.2.112:0/1503288417 >> [v2:172.21.2.112:6800/36029,v1:172.21.2.112:6802/36029] conn(0x7f78d018a330 msgr2=0x7f78d018a730 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:30.489 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.484 7f78d481c700 1 --2- 172.21.2.112:0/1503288417 >> [v2:172.21.2.112:6800/36029,v1:172.21.2.112:6802/36029] conn(0x7f78d018a330 0x7f78d018a730 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:30.489 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.484 7f78d481c700 1 -- 172.21.2.112:0/1503288417 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f78b40186c0 msgr2=0x7f78b401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:30.489 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.484 7f78d481c700 1 --2- 172.21.2.112:0/1503288417 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f78b40186c0 0x7f78b401abc0 crc :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:30.489 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.484 7f78d481c700 1 -- 172.21.2.112:0/1503288417 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f78d018b400 msgr2=0x7f78d018b800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:30.489 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.484 7f78d481c700 1 --2- 172.21.2.112:0/1503288417 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f78d018b400 0x7f78d018b800 secure :-1 s=READY pgs=3338 cs=0 l=1 rev1=1 rx=0x7f78c800a940 tx=0x7f78c800ace0).stop 2021-02-10T11:50:30.489 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.484 7f78cee1d700 1 -- 172.21.2.112:0/1503288417 reap_dead start 2021-02-10T11:50:30.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.484 7f78d481c700 1 -- 172.21.2.112:0/1503288417 shutdown_connections 2021-02-10T11:50:30.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.484 7f78d481c700 1 -- 172.21.2.112:0/1503288417 >> 172.21.2.112:0/1503288417 conn(0x7f78d019f950 msgr2=0x7f78d0082ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:30.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.484 7f78d481c700 1 -- 172.21.2.112:0/1503288417 shutdown_connections 2021-02-10T11:50:30.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.485 7f78d481c700 1 -- 172.21.2.112:0/1503288417 wait complete. 2021-02-10T11:50:30.497 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd deep-scrub 1 2021-02-10T11:50:30.925 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.904 7f382fe01700 1 Processor -- start 2021-02-10T11:50:30.925 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.904 7f382fe01700 1 -- start start 2021-02-10T11:50:30.925 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.904 7f382fe01700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f38280d1c70 0x7f38280f20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:30.925 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.904 7f382fe01700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38280f0b10 con 0x7f38280d1870 2021-02-10T11:50:30.925 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.904 7f382fe01700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38281b0be0 con 0x7f38280d0340 2021-02-10T11:50:30.925 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.904 7f382fe01700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38280e5ed0 con 0x7f38280d1c70 2021-02-10T11:50:30.926 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.904 7f382dc1d700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f38280d1870 0x7f38281b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:33508/0 (socket says 172.21.2.112:33508) 2021-02-10T11:50:30.926 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.905 7f382dc1d700 1 -- 172.21.2.112:0/690698214 learned_addr learned my addr 172.21.2.112:0/690698214 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:50:30.926 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.905 7f382e41e700 1 --2- 172.21.2.112:0/690698214 >> v2:172.21.2.120:3300/0 conn(0x7f38280d1c70 0x7f38280f20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:30.926 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.905 7f382cc1b700 1 -- 172.21.2.112:0/690698214 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 325551354 0 0) 0x7f38240010b0 con 0x7f38280d1870 2021-02-10T11:50:30.926 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.905 7f382e41e700 1 -- 172.21.2.112:0/690698214 >> v1:172.21.2.120:6789/0 conn(0x7f38280d0340 legacy=0x7f38280ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:30.926 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.905 7f382e41e700 1 -- 172.21.2.112:0/690698214 >> v1:172.21.2.112:6789/0 conn(0x7f38280d1870 legacy=0x7f38281b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:30.926 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.905 7f382e41e700 1 -- 172.21.2.112:0/690698214 --> v2:172.21.2.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38280efe30 con 0x7f38280d1c70 2021-02-10T11:50:30.927 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.905 7f382e41e700 1 --2- 172.21.2.112:0/690698214 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f38280d1c70 0x7f38280f20a0 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 rx=0x7f3820006ad0 tx=0x7f3820009ad0).ready entity=mon.1 client_cookie=3efe788ced3fc3ac server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:30.927 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.905 7f382cc1b700 1 -- 172.21.2.112:0/690698214 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f38200022e0 con 0x7f38280d1c70 2021-02-10T11:50:30.927 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.905 7f382cc1b700 1 -- 172.21.2.112:0/690698214 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3820006410 con 0x7f38280d1c70 2021-02-10T11:50:30.927 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.905 7f382cc1b700 1 -- 172.21.2.112:0/690698214 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3820002620 con 0x7f38280d1c70 2021-02-10T11:50:30.927 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.905 7f382fe01700 1 -- 172.21.2.112:0/690698214 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f38280d1c70 msgr2=0x7f38280f20a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:30.927 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.905 7f382fe01700 1 --2- 172.21.2.112:0/690698214 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f38280d1c70 0x7f38280f20a0 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 rx=0x7f3820006ad0 tx=0x7f3820009ad0).stop 2021-02-10T11:50:30.927 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.906 7f382fe01700 1 -- 172.21.2.112:0/690698214 shutdown_connections 2021-02-10T11:50:30.927 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.906 7f382fe01700 1 --2- 172.21.2.112:0/690698214 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f38280d1c70 0x7f38280f20a0 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:30.928 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.906 7f382fe01700 1 -- 172.21.2.112:0/690698214 >> 172.21.2.112:0/690698214 conn(0x7f38280ce450 msgr2=0x7f38280d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:30.928 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.906 7f382fe01700 1 -- 172.21.2.112:0/690698214 shutdown_connections 2021-02-10T11:50:30.928 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.906 7f382fe01700 1 -- 172.21.2.112:0/690698214 wait complete. 2021-02-10T11:50:30.929 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.921 7f382fe01700 1 Processor -- start 2021-02-10T11:50:30.929 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.921 7f382fe01700 1 -- start start 2021-02-10T11:50:30.930 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.921 7f382fe01700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f38280b3c90 0x7f38280b4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:30.930 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.921 7f382fe01700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f38280b46b0 0x7f38280b2100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:30.930 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.921 7f382fe01700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f38280b4ab0 0x7f38280b2720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:30.930 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.921 7f382fe01700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38280b3790 con 0x7f38280b4ab0 2021-02-10T11:50:30.930 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.921 7f382fe01700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38280a2bb0 con 0x7f38280b3c90 2021-02-10T11:50:30.930 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.921 7f382fe01700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38280a2590 con 0x7f38280b46b0 2021-02-10T11:50:30.930 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.922 7f382e41e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f38280b4ab0 0x7f38280b2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:30.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.922 7f382dc1d700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f38280b3c90 0x7f38280b4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:30.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.922 7f382e41e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f38280b4ab0 0x7f38280b2720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:34050/0 (socket says 172.21.2.112:34050) 2021-02-10T11:50:30.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.922 7f382e41e700 1 -- 172.21.2.112:0/4285028911 learned_addr learned my addr 172.21.2.112:0/4285028911 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:50:30.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.922 7f382d41c700 1 --2- 172.21.2.112:0/4285028911 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f38280b46b0 0x7f38280b2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:30.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.922 7f382dc1d700 1 -- 172.21.2.112:0/4285028911 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f38280b46b0 msgr2=0x7f38280b2100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:30.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.922 7f382dc1d700 1 --2- 172.21.2.112:0/4285028911 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f38280b46b0 0x7f38280b2100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:30.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.922 7f382dc1d700 1 -- 172.21.2.112:0/4285028911 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f38280b4ab0 msgr2=0x7f38280b2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:30.931 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.922 7f382dc1d700 1 --2- 172.21.2.112:0/4285028911 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f38280b4ab0 0x7f38280b2720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:30.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.922 7f382dc1d700 1 -- 172.21.2.112:0/4285028911 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38280b2d40 con 0x7f38280b3c90 2021-02-10T11:50:30.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.922 7f382dc1d700 1 --2- 172.21.2.112:0/4285028911 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f38280b3c90 0x7f38280b4090 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 rx=0x7f3824001c60 tx=0x7f382400a870).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:30.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.922 7f381e7fc700 1 -- 172.21.2.112:0/4285028911 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f38240009b0 con 0x7f38280b3c90 2021-02-10T11:50:30.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.922 7f381e7fc700 1 -- 172.21.2.112:0/4285028911 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f382400ac60 con 0x7f38280b3c90 2021-02-10T11:50:30.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.922 7f381e7fc700 1 -- 172.21.2.112:0/4285028911 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f38240045d0 con 0x7f38280b3c90 2021-02-10T11:50:30.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.922 7f382fe01700 1 -- 172.21.2.112:0/4285028911 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38280b2d40 con 0x7f38280b3c90 2021-02-10T11:50:30.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.923 7f382fe01700 1 -- 172.21.2.112:0/4285028911 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38280b2d40 con 0x7f38280b3c90 2021-02-10T11:50:30.932 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.923 7f381e7fc700 1 -- 172.21.2.112:0/4285028911 <== mon.1 v2:172.21.2.120:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3824007fd0 con 0x7f38280b3c90 2021-02-10T11:50:30.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.923 7f381e7fc700 1 --2- 172.21.2.112:0/4285028911 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f38080186c0 0x7f380801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:30.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.923 7f381e7fc700 1 -- 172.21.2.112:0/4285028911 <== mon.1 v2:172.21.2.120:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f3824032430 con 0x7f38280b3c90 2021-02-10T11:50:30.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.923 7f382d41c700 1 --2- 172.21.2.112:0/4285028911 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f38080186c0 0x7f380801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:30.933 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.924 7f382d41c700 1 --2- 172.21.2.112:0/4285028911 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f38080186c0 0x7f380801abc0 crc :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:30.959 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.954 7f382fe01700 1 -- 172.21.2.112:0/4285028911 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38280ee0f0 con 0x7f38280b3c90 2021-02-10T11:50:30.963 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:30.959 7f381e7fc700 1 -- 172.21.2.112:0/4285028911 <== mon.1 v2:172.21.2.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f382402df00 con 0x7f38280b3c90 2021-02-10T11:50:31.094 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:31.089 7f382fe01700 1 -- 172.21.2.112:0/4285028911 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mgr", ""]}) v1 -- 0x7f38280ce450 con 0x7f38080186c0 2021-02-10T11:50:31.095 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:31.090 7f381e7fc700 1 -- 172.21.2.112:0/4285028911 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (crc 0 0 0) 0x7f3818000ce0 con 0x7f38080186c0 2021-02-10T11:50:31.095 INFO:teuthology.orchestra.run.gibba012.stderr:instructed osd(s) 1 to deep-scrub 2021-02-10T11:50:31.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:31.092 7f382fe01700 1 -- 172.21.2.112:0/4285028911 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f38080186c0 msgr2=0x7f380801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:31.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:31.092 7f382fe01700 1 --2- 172.21.2.112:0/4285028911 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f38080186c0 0x7f380801abc0 crc :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:31.096 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:31.092 7f382fe01700 1 -- 172.21.2.112:0/4285028911 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f38280b3c90 msgr2=0x7f38280b4090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:31.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:31.092 7f382fe01700 1 --2- 172.21.2.112:0/4285028911 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f38280b3c90 0x7f38280b4090 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 rx=0x7f3824001c60 tx=0x7f382400a870).stop 2021-02-10T11:50:31.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:31.092 7f382fe01700 1 -- 172.21.2.112:0/4285028911 shutdown_connections 2021-02-10T11:50:31.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:31.092 7f382fe01700 1 --2- 172.21.2.112:0/4285028911 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f38080186c0 0x7f380801abc0 unknown :-1 s=CLOSED pgs=1502 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:31.097 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:31.092 7f382fe01700 1 --2- 172.21.2.112:0/4285028911 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f38280b4ab0 0x7f38280b2720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:31.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:31.092 7f382fe01700 1 --2- 172.21.2.112:0/4285028911 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f38280b46b0 0x7f38280b2100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:31.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:31.092 7f382fe01700 1 --2- 172.21.2.112:0/4285028911 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f38280b3c90 0x7f38280b4090 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:31.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:31.092 7f382fe01700 1 -- 172.21.2.112:0/4285028911 >> 172.21.2.112:0/4285028911 conn(0x7f38280946f0 msgr2=0x7f3828095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:31.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:31.092 7f382fe01700 1 -- 172.21.2.112:0/4285028911 shutdown_connections 2021-02-10T11:50:31.098 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:31.092 7f382fe01700 1 -- 172.21.2.112:0/4285028911 wait complete. 2021-02-10T11:50:31.105 INFO:tasks.ceph:Scrubbing osd.2 2021-02-10T11:50:31.105 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2021-02-10T11:50:31.538 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:31.517 7f4fcf464700 1 Processor -- start 2021-02-10T11:50:31.538 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:31.517 7f4fcf464700 1 -- start start 2021-02-10T11:50:31.539 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:31.518 7f4fcf464700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f4fc80d1880 0x7f4fc81b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:31.539 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:31.518 7f4fcf464700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f4fc80cff80 0x7f4fc8109140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:31.539 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:31.518 7f4fcf464700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fc80fe1d0 con 0x7f4fc81b0af0 2021-02-10T11:50:31.539 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:31.518 7f4fcf464700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4fc8109ae0 con 0x7f4fc80d1880 2021-02-10T11:50:31.539 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:31.518 7f4fcf464700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4fc8109760 con 0x7f4fc80cff80 2021-02-10T11:50:31.539 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:31.518 7f4fcd280700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f4fc80d1880 0x7f4fc81b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:31.539 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:31.518 7f4fcca7f700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f4fc81b0af0 0x7f4fc8108950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:33518/0 (socket says 172.21.2.112:33518) 2021-02-10T11:50:31.539 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:31.518 7f4fcca7f700 1 -- 172.21.2.112:0/1978459506 learned_addr learned my addr 172.21.2.112:0/1978459506 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:50:31.540 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:31.519 7f4fcda81700 1 --2- 172.21.2.112:0/1978459506 >> v2:172.21.2.120:3300/0 conn(0x7f4fc80cff80 0x7f4fc8109140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:31.540 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:31.519 7f4fcd280700 1 -- 172.21.2.112:0/1978459506 >> v2:172.21.2.120:3300/0 conn(0x7f4fc80cff80 msgr2=0x7f4fc8109140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:50:31.540 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:31.519 7f4fcd280700 1 --2- 172.21.2.112:0/1978459506 >> v2:172.21.2.120:3300/0 conn(0x7f4fc80cff80 0x7f4fc8109140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:31.540 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:31.519 7f4fcd280700 1 -- 172.21.2.112:0/1978459506 >> v1:172.21.2.112:6789/0 conn(0x7f4fc81b0af0 legacy=0x7f4fc8108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:31.540 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:31.519 7f4fcd280700 1 -- 172.21.2.112:0/1978459506 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fc8088070 con 0x7f4fc80d1880 2021-02-10T11:50:31.540 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:31.519 7f4fcd280700 1 --2- 172.21.2.112:0/1978459506 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4fc80d1880 0x7f4fc81b04d0 secure :-1 s=READY pgs=3341 cs=0 l=1 rev1=1 rx=0x7f4fbc006650 tx=0x7f4fbc009a00).ready entity=mon.0 client_cookie=caf103237e49736b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:31.540 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:31.519 7f4fbbfff700 1 -- 172.21.2.112:0/1978459506 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4fbc009df0 con 0x7f4fc80d1880 2021-02-10T11:50:31.540 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:31.519 7f4fbbfff700 1 -- 172.21.2.112:0/1978459506 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fbc0009b0 con 0x7f4fc80d1880 2021-02-10T11:50:31.541 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:31.519 7f4fbbfff700 1 -- 172.21.2.112:0/1978459506 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4fbc00c2b0 con 0x7f4fc80d1880 2021-02-10T11:50:31.541 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:31.519 7f4fcf464700 1 -- 172.21.2.112:0/1978459506 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4fc80d1880 msgr2=0x7f4fc81b04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:31.541 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:31.519 7f4fcf464700 1 --2- 172.21.2.112:0/1978459506 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4fc80d1880 0x7f4fc81b04d0 secure :-1 s=READY pgs=3341 cs=0 l=1 rev1=1 rx=0x7f4fbc006650 tx=0x7f4fbc009a00).stop 2021-02-10T11:50:31.541 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:31.520 7f4fcf464700 1 -- 172.21.2.112:0/1978459506 shutdown_connections 2021-02-10T11:50:31.541 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:31.520 7f4fcf464700 1 --2- 172.21.2.112:0/1978459506 >> v2:172.21.2.120:3300/0 conn(0x7f4fc80cff80 0x7f4fc8109140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:31.541 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:31.520 7f4fcf464700 1 --2- 172.21.2.112:0/1978459506 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4fc80d1880 0x7f4fc81b04d0 unknown :-1 s=CLOSED pgs=3341 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:31.541 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:31.520 7f4fcf464700 1 -- 172.21.2.112:0/1978459506 >> 172.21.2.112:0/1978459506 conn(0x7f4fc80ce430 msgr2=0x7f4fc80d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:31.541 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:31.520 7f4fcf464700 1 -- 172.21.2.112:0/1978459506 shutdown_connections 2021-02-10T11:50:31.542 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:31.520 7f4fcf464700 1 -- 172.21.2.112:0/1978459506 wait complete. 2021-02-10T11:50:31.543 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:31.534 7f4fcf464700 1 Processor -- start 2021-02-10T11:50:31.543 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:31.534 7f4fcf464700 1 -- start start 2021-02-10T11:50:31.543 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:31.535 7f4fcf464700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4fc81b58d0 0x7f4fc80aa4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:31.543 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:31.535 7f4fcf464700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4fc80aaaf0 0x7f4fc81b80a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:31.544 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:31.535 7f4fcf464700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4fc81b86c0 0x7f4fc819de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:31.544 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:31.535 7f4fcf464700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fc81b8ac0 con 0x7f4fc81b86c0 2021-02-10T11:50:31.544 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:31.535 7f4fcf464700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fc81b8e20 con 0x7f4fc81b58d0 2021-02-10T11:50:31.544 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:31.535 7f4fcf464700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fc81b5cd0 con 0x7f4fc80aaaf0 2021-02-10T11:50:31.544 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:31.535 7f4fcda81700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4fc81b86c0 0x7f4fc819de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:31.544 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:31.535 7f4fcda81700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4fc81b86c0 0x7f4fc819de70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:34060/0 (socket says 172.21.2.112:34060) 2021-02-10T11:50:31.544 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:31.535 7f4fcda81700 1 -- 172.21.2.112:0/4133697839 learned_addr learned my addr 172.21.2.112:0/4133697839 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:50:31.544 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:31.535 7f4fcda81700 1 -- 172.21.2.112:0/4133697839 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4fc80aaaf0 msgr2=0x7f4fc81b80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:31.545 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:31.536 7f4fcda81700 1 --2- 172.21.2.112:0/4133697839 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f4fc80aaaf0 0x7f4fc81b80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:31.545 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:31.536 7f4fcda81700 1 -- 172.21.2.112:0/4133697839 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4fc81b58d0 msgr2=0x7f4fc80aa4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:31.545 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:31.536 7f4fcda81700 1 --2- 172.21.2.112:0/4133697839 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f4fc81b58d0 0x7f4fc80aa4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:31.545 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:31.536 7f4fcda81700 1 -- 172.21.2.112:0/4133697839 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fc8095710 con 0x7f4fc81b86c0 2021-02-10T11:50:31.545 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:31.536 7f4fcda81700 1 --2- 172.21.2.112:0/4133697839 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4fc81b86c0 0x7f4fc819de70 secure :-1 s=READY pgs=3342 cs=0 l=1 rev1=1 rx=0x7f4fc0009980 tx=0x7f4fc000bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:31.545 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:31.536 7f4fb9ffb700 1 -- 172.21.2.112:0/4133697839 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4fc00011f0 con 0x7f4fc81b86c0 2021-02-10T11:50:31.545 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:31.536 7f4fb9ffb700 1 -- 172.21.2.112:0/4133697839 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fc0003030 con 0x7f4fc81b86c0 2021-02-10T11:50:31.545 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:31.536 7f4fb9ffb700 1 -- 172.21.2.112:0/4133697839 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4fc0001770 con 0x7f4fc81b86c0 2021-02-10T11:50:31.546 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:31.536 7f4fcf464700 1 -- 172.21.2.112:0/4133697839 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fc8095710 con 0x7f4fc81b86c0 2021-02-10T11:50:31.546 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:31.536 7f4fcf464700 1 -- 172.21.2.112:0/4133697839 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fc819e780 con 0x7f4fc81b86c0 2021-02-10T11:50:31.546 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:31.537 7f4fb9ffb700 1 -- 172.21.2.112:0/4133697839 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4fc00011f0 con 0x7f4fc81b86c0 2021-02-10T11:50:31.546 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:31.537 7f4fb9ffb700 1 --2- 172.21.2.112:0/4133697839 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4fa80186c0 0x7f4fa801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:31.546 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:31.537 7f4fb9ffb700 1 -- 172.21.2.112:0/4133697839 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f4fc0032920 con 0x7f4fc81b86c0 2021-02-10T11:50:31.546 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:31.537 7f4fcd280700 1 --2- 172.21.2.112:0/4133697839 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4fa80186c0 0x7f4fa801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:31.546 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:31.538 7f4fcd280700 1 --2- 172.21.2.112:0/4133697839 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4fa80186c0 0x7f4fa801abc0 crc :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:31.572 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:31.568 7f4fcf464700 1 --2- 172.21.2.112:0/4133697839 >> [v2:172.21.2.112:6801/36031,v1:172.21.2.112:6803/36031] conn(0x7f4fc81b8cb0 0x7f4fc80627c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:31.572 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:31.568 7f4fcf464700 1 -- 172.21.2.112:0/4133697839 --> [v2:172.21.2.112:6801/36031,v1:172.21.2.112:6803/36031] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4fc8082aa0 con 0x7f4fc81b8cb0 2021-02-10T11:50:31.572 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:31.568 7f4fcca7f700 1 --2- 172.21.2.112:0/4133697839 >> [v2:172.21.2.112:6801/36031,v1:172.21.2.112:6803/36031] conn(0x7f4fc81b8cb0 0x7f4fc80627c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:31.573 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:31.568 7f4fcca7f700 1 --2- 172.21.2.112:0/4133697839 >> [v2:172.21.2.112:6801/36031,v1:172.21.2.112:6803/36031] conn(0x7f4fc81b8cb0 0x7f4fc80627c0 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:31.573 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:31.569 7f4fb9ffb700 1 -- 172.21.2.112:0/4133697839 <== osd.2 v2:172.21.2.112:6801/36031 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+5151 (crc 0 0 0) 0x7f4fc4006e30 con 0x7f4fc81b8cb0 2021-02-10T11:50:31.581 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:31.576 7f4fcf464700 1 -- 172.21.2.112:0/4133697839 --> [v2:172.21.2.112:6801/36031,v1:172.21.2.112:6803/36031] -- command(tid 2: {"prefix": "config set", "key": "osd_debug_deep_scrub_sleep", "value": "0"}) v1 -- 0x7f4fc8082aa0 con 0x7f4fc81b8cb0 2021-02-10T11:50:31.581 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:31.577 7f4fb9ffb700 1 -- 172.21.2.112:0/4133697839 <== osd.2 v2:172.21.2.112:6801/36031 2 ==== command_reply(tid 2: 0 Set osd_debug_deep_scrub_sleep to 0) v1 ==== 43+0+0 (crc 0 0 0) 0x7f4fc4006c20 con 0x7f4fc81b8cb0 2021-02-10T11:50:31.582 INFO:teuthology.orchestra.run.gibba012.stderr:Set osd_debug_deep_scrub_sleep to 0 2021-02-10T11:50:31.582 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:31.578 7f4fcf464700 1 -- 172.21.2.112:0/4133697839 >> [v2:172.21.2.112:6801/36031,v1:172.21.2.112:6803/36031] conn(0x7f4fc81b8cb0 msgr2=0x7f4fc80627c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:31.583 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:31.578 7f4fcf464700 1 --2- 172.21.2.112:0/4133697839 >> [v2:172.21.2.112:6801/36031,v1:172.21.2.112:6803/36031] conn(0x7f4fc81b8cb0 0x7f4fc80627c0 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:31.583 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:31.578 7f4fcf464700 1 -- 172.21.2.112:0/4133697839 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4fa80186c0 msgr2=0x7f4fa801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:31.583 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:31.578 7f4fcf464700 1 --2- 172.21.2.112:0/4133697839 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f4fa80186c0 0x7f4fa801abc0 crc :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:31.583 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:31.578 7f4fcf464700 1 -- 172.21.2.112:0/4133697839 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4fc81b86c0 msgr2=0x7f4fc819de70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:31.583 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:31.578 7f4fcf464700 1 --2- 172.21.2.112:0/4133697839 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f4fc81b86c0 0x7f4fc819de70 secure :-1 s=READY pgs=3342 cs=0 l=1 rev1=1 rx=0x7f4fc0009980 tx=0x7f4fc000bcd0).stop 2021-02-10T11:50:31.583 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:31.578 7f4fcda81700 1 -- 172.21.2.112:0/4133697839 reap_dead start 2021-02-10T11:50:31.583 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:31.578 7f4fcf464700 1 -- 172.21.2.112:0/4133697839 shutdown_connections 2021-02-10T11:50:31.583 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:31.578 7f4fcf464700 1 -- 172.21.2.112:0/4133697839 >> 172.21.2.112:0/4133697839 conn(0x7f4fc819f940 msgr2=0x7f4fc818c290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:31.584 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:31.579 7f4fcf464700 1 -- 172.21.2.112:0/4133697839 shutdown_connections 2021-02-10T11:50:31.584 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:31.579 7f4fcf464700 1 -- 172.21.2.112:0/4133697839 wait complete. 2021-02-10T11:50:31.591 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd deep-scrub 2 2021-02-10T11:50:32.019 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:31.999 7f114b58d700 1 Processor -- start 2021-02-10T11:50:32.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:31.999 7f114b58d700 1 -- start start 2021-02-10T11:50:32.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:31.999 7f114b58d700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f11440d1870 0x7f11441b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:32.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:31.999 7f114b58d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f11441b0a10 0x7f11440cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:32.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.000 7f114b58d700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11440f0b10 con 0x7f11440d1c70 2021-02-10T11:50:32.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.000 7f114b58d700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11440e5ed0 con 0x7f11441b0a10 2021-02-10T11:50:32.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.000 7f114b58d700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11440f0650 con 0x7f11440d1870 2021-02-10T11:50:32.020 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.000 7f1148ba8700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f11441b0a10 0x7f11440cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:32.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.000 7f1148ba8700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f11441b0a10 0x7f11440cf330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:34070/0 (socket says 172.21.2.112:34070) 2021-02-10T11:50:32.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.000 7f1148ba8700 1 -- 172.21.2.112:0/1791677562 learned_addr learned my addr 172.21.2.112:0/1791677562 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:50:32.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.000 7f11493a9700 1 --2- 172.21.2.112:0/1791677562 >> v2:172.21.2.120:3300/0 conn(0x7f11440d1870 0x7f11441b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:32.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.000 7f1148ba8700 1 -- 172.21.2.112:0/1791677562 >> v2:172.21.2.120:3300/0 conn(0x7f11440d1870 msgr2=0x7f11441b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:50:32.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.000 7f1148ba8700 1 --2- 172.21.2.112:0/1791677562 >> v2:172.21.2.120:3300/0 conn(0x7f11440d1870 0x7f11441b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:32.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.000 7f1148ba8700 1 -- 172.21.2.112:0/1791677562 >> v1:172.21.2.120:6789/0 conn(0x7f11440d1c70 legacy=0x7f11440cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:50:32.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.000 7f1148ba8700 1 -- 172.21.2.112:0/1791677562 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11440f0840 con 0x7f11441b0a10 2021-02-10T11:50:32.021 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.000 7f1148ba8700 1 --2- 172.21.2.112:0/1791677562 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f11441b0a10 0x7f11440cf330 secure :-1 s=READY pgs=3343 cs=0 l=1 rev1=1 rx=0x7f1138006700 tx=0x7f1138009b80).ready entity=mon.0 client_cookie=cb0c18cbe232efbc server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:32.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.000 7f1137fff700 1 -- 172.21.2.112:0/1791677562 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f113800c0c0 con 0x7f11441b0a10 2021-02-10T11:50:32.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.000 7f1137fff700 1 -- 172.21.2.112:0/1791677562 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f113800c400 con 0x7f11441b0a10 2021-02-10T11:50:32.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.000 7f1137fff700 1 -- 172.21.2.112:0/1791677562 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f113800c920 con 0x7f11441b0a10 2021-02-10T11:50:32.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.000 7f114b58d700 1 -- 172.21.2.112:0/1791677562 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f11441b0a10 msgr2=0x7f11440cf330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:32.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.000 7f114b58d700 1 --2- 172.21.2.112:0/1791677562 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f11441b0a10 0x7f11440cf330 secure :-1 s=READY pgs=3343 cs=0 l=1 rev1=1 rx=0x7f1138006700 tx=0x7f1138009b80).stop 2021-02-10T11:50:32.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.001 7f114b58d700 1 -- 172.21.2.112:0/1791677562 shutdown_connections 2021-02-10T11:50:32.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.001 7f114b58d700 1 --2- 172.21.2.112:0/1791677562 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f11441b0a10 0x7f11440cf330 unknown :-1 s=CLOSED pgs=3343 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:32.022 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.001 7f114b58d700 1 --2- 172.21.2.112:0/1791677562 >> v2:172.21.2.120:3300/0 conn(0x7f11440d1870 0x7f11441b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:32.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.001 7f114b58d700 1 -- 172.21.2.112:0/1791677562 >> 172.21.2.112:0/1791677562 conn(0x7f11440ce450 msgr2=0x7f11440d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:32.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.001 7f114b58d700 1 -- 172.21.2.112:0/1791677562 shutdown_connections 2021-02-10T11:50:32.023 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.001 7f114b58d700 1 -- 172.21.2.112:0/1791677562 wait complete. 2021-02-10T11:50:32.024 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.015 7f114b58d700 1 Processor -- start 2021-02-10T11:50:32.024 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.015 7f114b58d700 1 -- start start 2021-02-10T11:50:32.024 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.016 7f114b58d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f11440ebe80 0x7f11440b8730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:32.025 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.016 7f114b58d700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f11440afa30 0x7f11440afe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:32.025 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.016 7f114b58d700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f11440b0450 0x7f11440ec8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:32.025 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.016 7f114b58d700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11440b8d50 con 0x7f11440ebe80 2021-02-10T11:50:32.025 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.016 7f114b58d700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f114409df70 con 0x7f11440afa30 2021-02-10T11:50:32.025 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.016 7f114b58d700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11440b0850 con 0x7f11440b0450 2021-02-10T11:50:32.025 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.016 7f11493a9700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f11440ebe80 0x7f11440b8730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:32.025 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.016 7f11493a9700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f11440ebe80 0x7f11440b8730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:34076/0 (socket says 172.21.2.112:34076) 2021-02-10T11:50:32.025 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.016 7f11493a9700 1 -- 172.21.2.112:0/2085184487 learned_addr learned my addr 172.21.2.112:0/2085184487 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:50:32.026 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.016 7f1148ba8700 1 --2- 172.21.2.112:0/2085184487 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f11440afa30 0x7f11440afe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:32.026 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.016 7f1149baa700 1 --2- 172.21.2.112:0/2085184487 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f11440b0450 0x7f11440ec8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:32.026 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.017 7f11493a9700 1 -- 172.21.2.112:0/2085184487 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f11440b0450 msgr2=0x7f11440ec8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:32.026 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.017 7f11493a9700 1 --2- 172.21.2.112:0/2085184487 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f11440b0450 0x7f11440ec8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:32.026 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.017 7f11493a9700 1 -- 172.21.2.112:0/2085184487 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f11440afa30 msgr2=0x7f11440afe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:32.026 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.017 7f11493a9700 1 --2- 172.21.2.112:0/2085184487 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f11440afa30 0x7f11440afe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:32.026 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.017 7f11493a9700 1 -- 172.21.2.112:0/2085184487 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11440ecef0 con 0x7f11440ebe80 2021-02-10T11:50:32.026 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.017 7f11493a9700 1 --2- 172.21.2.112:0/2085184487 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f11440ebe80 0x7f11440b8730 secure :-1 s=READY pgs=3344 cs=0 l=1 rev1=1 rx=0x7f1140006f00 tx=0x7f114000b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:32.027 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.017 7f1135ffb700 1 -- 172.21.2.112:0/2085184487 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f114000bd30 con 0x7f11440ebe80 2021-02-10T11:50:32.027 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.017 7f1135ffb700 1 -- 172.21.2.112:0/2085184487 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1140002030 con 0x7f11440ebe80 2021-02-10T11:50:32.027 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.017 7f114b58d700 1 -- 172.21.2.112:0/2085184487 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11440ecef0 con 0x7f11440ebe80 2021-02-10T11:50:32.027 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.017 7f1135ffb700 1 -- 172.21.2.112:0/2085184487 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1140002480 con 0x7f11440ebe80 2021-02-10T11:50:32.027 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.017 7f114b58d700 1 -- 172.21.2.112:0/2085184487 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11440ed480 con 0x7f11440ebe80 2021-02-10T11:50:32.027 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.018 7f1135ffb700 1 -- 172.21.2.112:0/2085184487 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1140000c90 con 0x7f11440ebe80 2021-02-10T11:50:32.027 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.018 7f1135ffb700 1 --2- 172.21.2.112:0/2085184487 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f11240186c0 0x7f112401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:32.027 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.018 7f1135ffb700 1 -- 172.21.2.112:0/2085184487 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f11400323f0 con 0x7f11440ebe80 2021-02-10T11:50:32.028 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.018 7f1148ba8700 1 --2- 172.21.2.112:0/2085184487 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f11240186c0 0x7f112401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:32.028 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.018 7f1148ba8700 1 --2- 172.21.2.112:0/2085184487 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f11240186c0 0x7f112401abc0 crc :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:32.054 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.049 7f114b58d700 1 -- 172.21.2.112:0/2085184487 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11440ebc50 con 0x7f11440ebe80 2021-02-10T11:50:32.056 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.052 7f1135ffb700 1 -- 172.21.2.112:0/2085184487 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1140023190 con 0x7f11440ebe80 2021-02-10T11:50:32.186 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.182 7f114b58d700 1 -- 172.21.2.112:0/2085184487 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mgr", ""]}) v1 -- 0x7f11440ed480 con 0x7f11240186c0 2021-02-10T11:50:32.187 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.182 7f1135ffb700 1 -- 172.21.2.112:0/2085184487 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (crc 0 0 0) 0x7f1138000940 con 0x7f11240186c0 2021-02-10T11:50:32.187 INFO:teuthology.orchestra.run.gibba012.stderr:instructed osd(s) 2 to deep-scrub 2021-02-10T11:50:32.189 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.184 7f114b58d700 1 -- 172.21.2.112:0/2085184487 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f11240186c0 msgr2=0x7f112401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:32.189 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.184 7f114b58d700 1 --2- 172.21.2.112:0/2085184487 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f11240186c0 0x7f112401abc0 crc :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:32.189 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.184 7f114b58d700 1 -- 172.21.2.112:0/2085184487 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f11440ebe80 msgr2=0x7f11440b8730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:32.189 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.184 7f114b58d700 1 --2- 172.21.2.112:0/2085184487 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f11440ebe80 0x7f11440b8730 secure :-1 s=READY pgs=3344 cs=0 l=1 rev1=1 rx=0x7f1140006f00 tx=0x7f114000b940).stop 2021-02-10T11:50:32.190 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.185 7f114b58d700 1 -- 172.21.2.112:0/2085184487 shutdown_connections 2021-02-10T11:50:32.190 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.185 7f114b58d700 1 --2- 172.21.2.112:0/2085184487 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f11240186c0 0x7f112401abc0 unknown :-1 s=CLOSED pgs=1504 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:32.190 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.185 7f114b58d700 1 --2- 172.21.2.112:0/2085184487 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f11440afa30 0x7f11440afe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:32.190 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.185 7f114b58d700 1 --2- 172.21.2.112:0/2085184487 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f11440b0450 0x7f11440ec8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:32.190 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.185 7f114b58d700 1 --2- 172.21.2.112:0/2085184487 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f11440ebe80 0x7f11440b8730 unknown :-1 s=CLOSED pgs=3344 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:32.190 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.185 7f114b58d700 1 -- 172.21.2.112:0/2085184487 >> 172.21.2.112:0/2085184487 conn(0x7f11440a6b90 msgr2=0x7f11440af100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:32.191 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.185 7f114b58d700 1 -- 172.21.2.112:0/2085184487 shutdown_connections 2021-02-10T11:50:32.191 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.185 7f114b58d700 1 -- 172.21.2.112:0/2085184487 wait complete. 2021-02-10T11:50:32.198 INFO:tasks.ceph:Scrubbing osd.3 2021-02-10T11:50:32.198 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2021-02-10T11:50:32.630 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.610 7f041e0be700 1 Processor -- start 2021-02-10T11:50:32.630 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.610 7f041e0be700 1 -- start start 2021-02-10T11:50:32.630 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.610 7f041e0be700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f041810aff0 0x7f04180cf750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:32.630 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.610 7f041e0be700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04180fe1c0 con 0x7f04180d1870 2021-02-10T11:50:32.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.610 7f041e0be700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04181b0cb0 con 0x7f04180d1c70 2021-02-10T11:50:32.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.610 7f041e0be700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0418109a60 con 0x7f041810aff0 2021-02-10T11:50:32.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.610 7f0417fff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f041810aff0 0x7f04180cf750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:32.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.610 7f0417fff700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f041810aff0 0x7f04180cf750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:34088/0 (socket says 172.21.2.112:34088) 2021-02-10T11:50:32.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.610 7f0417fff700 1 -- 172.21.2.112:0/851540857 learned_addr learned my addr 172.21.2.112:0/851540857 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:50:32.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.610 7f0417fff700 1 -- 172.21.2.112:0/851540857 >> v1:172.21.2.120:6789/0 conn(0x7f04180d1c70 legacy=0x7f04180cef60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:50:32.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.610 7f0417fff700 1 -- 172.21.2.112:0/851540857 >> v1:172.21.2.112:6789/0 conn(0x7f04180d1870 legacy=0x7f04181b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:32.631 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.610 7f0417fff700 1 -- 172.21.2.112:0/851540857 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f041810b3f0 con 0x7f041810aff0 2021-02-10T11:50:32.632 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.611 7f0417fff700 1 --2- 172.21.2.112:0/851540857 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f041810aff0 0x7f04180cf750 secure :-1 s=READY pgs=3346 cs=0 l=1 rev1=1 rx=0x7f0410006f00 tx=0x7f0410009f10).ready entity=mon.0 client_cookie=6ca608ec5e91518e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:32.632 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.611 7f04167fc700 1 -- 172.21.2.112:0/851540857 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f041000b3b0 con 0x7f041810aff0 2021-02-10T11:50:32.632 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.611 7f04167fc700 1 -- 172.21.2.112:0/851540857 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f041000b730 con 0x7f041810aff0 2021-02-10T11:50:32.632 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.611 7f04167fc700 1 -- 172.21.2.112:0/851540857 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f041000bc10 con 0x7f041810aff0 2021-02-10T11:50:32.632 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.611 7f041e0be700 1 -- 172.21.2.112:0/851540857 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f041810aff0 msgr2=0x7f04180cf750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:32.632 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.611 7f041e0be700 1 --2- 172.21.2.112:0/851540857 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f041810aff0 0x7f04180cf750 secure :-1 s=READY pgs=3346 cs=0 l=1 rev1=1 rx=0x7f0410006f00 tx=0x7f0410009f10).stop 2021-02-10T11:50:32.632 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.611 7f041e0be700 1 -- 172.21.2.112:0/851540857 shutdown_connections 2021-02-10T11:50:32.632 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.611 7f041e0be700 1 --2- 172.21.2.112:0/851540857 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f041810aff0 0x7f04180cf750 unknown :-1 s=CLOSED pgs=3346 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:32.632 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.611 7f041e0be700 1 -- 172.21.2.112:0/851540857 >> 172.21.2.112:0/851540857 conn(0x7f04180ce420 msgr2=0x7f04180d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:32.633 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.611 7f041e0be700 1 -- 172.21.2.112:0/851540857 shutdown_connections 2021-02-10T11:50:32.633 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.611 7f041e0be700 1 -- 172.21.2.112:0/851540857 wait complete. 2021-02-10T11:50:32.634 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.626 7f041e0be700 1 Processor -- start 2021-02-10T11:50:32.634 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.626 7f041e0be700 1 -- start start 2021-02-10T11:50:32.635 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.626 7f041e0be700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f04181c6a40 0x7f04180aa760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:32.635 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.626 7f041e0be700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f04180ab440 0x7f04180c0860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:32.635 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.626 7f041e0be700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f04180bfa90 0x7f04180bfe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:32.635 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.626 7f041e0be700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04180c0e80 con 0x7f04180bfa90 2021-02-10T11:50:32.635 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.626 7f041e0be700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04180c04b0 con 0x7f04180ab440 2021-02-10T11:50:32.635 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.626 7f041e0be700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f041808b410 con 0x7f04181c6a40 2021-02-10T11:50:32.635 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.627 7f0417fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f04180bfa90 0x7f04180bfe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:32.635 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.627 7f0417fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f04180bfa90 0x7f04180bfe90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:34094/0 (socket says 172.21.2.112:34094) 2021-02-10T11:50:32.636 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.627 7f0417fff700 1 -- 172.21.2.112:0/3435562951 learned_addr learned my addr 172.21.2.112:0/3435562951 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:50:32.636 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.627 7f04177fe700 1 --2- 172.21.2.112:0/3435562951 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f04181c6a40 0x7f04180aa760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:32.636 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.627 7f0416ffd700 1 --2- 172.21.2.112:0/3435562951 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f04180ab440 0x7f04180c0860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:32.636 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.627 7f0417fff700 1 -- 172.21.2.112:0/3435562951 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f04181c6a40 msgr2=0x7f04180aa760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:32.636 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.627 7f0417fff700 1 --2- 172.21.2.112:0/3435562951 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f04181c6a40 0x7f04180aa760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:32.636 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.627 7f0417fff700 1 -- 172.21.2.112:0/3435562951 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f04180ab440 msgr2=0x7f04180c0860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:32.636 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.627 7f0417fff700 1 --2- 172.21.2.112:0/3435562951 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f04180ab440 0x7f04180c0860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:32.636 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.627 7f0417fff700 1 -- 172.21.2.112:0/3435562951 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f041808b630 con 0x7f04180bfa90 2021-02-10T11:50:32.637 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.627 7f04177fe700 1 --2- 172.21.2.112:0/3435562951 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f04181c6a40 0x7f04180aa760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:50:32.637 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.627 7f0416ffd700 1 --2- 172.21.2.112:0/3435562951 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f04180ab440 0x7f04180c0860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:50:32.637 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.627 7f0417fff700 1 --2- 172.21.2.112:0/3435562951 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f04180bfa90 0x7f04180bfe90 secure :-1 s=READY pgs=3347 cs=0 l=1 rev1=1 rx=0x7f041000b9b0 tx=0x7f041000bd50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:32.637 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.627 7f040ffff700 1 -- 172.21.2.112:0/3435562951 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0410005820 con 0x7f04180bfa90 2021-02-10T11:50:32.637 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.627 7f040ffff700 1 -- 172.21.2.112:0/3435562951 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0410005a60 con 0x7f04180bfa90 2021-02-10T11:50:32.637 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.627 7f040ffff700 1 -- 172.21.2.112:0/3435562951 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0410006030 con 0x7f04180bfa90 2021-02-10T11:50:32.637 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.627 7f041e0be700 1 -- 172.21.2.112:0/3435562951 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f041808b630 con 0x7f04180bfa90 2021-02-10T11:50:32.637 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.628 7f041e0be700 1 -- 172.21.2.112:0/3435562951 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f041808bbc0 con 0x7f04180bfa90 2021-02-10T11:50:32.638 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.628 7f040ffff700 1 -- 172.21.2.112:0/3435562951 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0410005c40 con 0x7f04180bfa90 2021-02-10T11:50:32.638 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.628 7f040ffff700 1 --2- 172.21.2.112:0/3435562951 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f03fc0186c0 0x7f03fc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:32.638 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.628 7f040ffff700 1 -- 172.21.2.112:0/3435562951 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f0410032910 con 0x7f04180bfa90 2021-02-10T11:50:32.638 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.629 7f04177fe700 1 --2- 172.21.2.112:0/3435562951 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f03fc0186c0 0x7f03fc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:32.638 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.629 7f04177fe700 1 --2- 172.21.2.112:0/3435562951 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f03fc0186c0 0x7f03fc01abc0 crc :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:32.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.660 7f041e0be700 1 --2- 172.21.2.112:0/3435562951 >> [v2:172.21.2.112:6816/36028,v1:172.21.2.112:6819/36028] conn(0x7f04180c45d0 0x7f04180627c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:32.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.660 7f041e0be700 1 -- 172.21.2.112:0/3435562951 --> [v2:172.21.2.112:6816/36028,v1:172.21.2.112:6819/36028] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f041808bbc0 con 0x7f04180c45d0 2021-02-10T11:50:32.664 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.660 7f0416ffd700 1 --2- 172.21.2.112:0/3435562951 >> [v2:172.21.2.112:6816/36028,v1:172.21.2.112:6819/36028] conn(0x7f04180c45d0 0x7f04180627c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:32.665 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.661 7f0416ffd700 1 --2- 172.21.2.112:0/3435562951 >> [v2:172.21.2.112:6816/36028,v1:172.21.2.112:6819/36028] conn(0x7f04180c45d0 0x7f04180627c0 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:32.665 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.661 7f040ffff700 1 -- 172.21.2.112:0/3435562951 <== osd.3 v2:172.21.2.112:6816/36028 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+5151 (crc 0 0 0) 0x7f0400000960 con 0x7f04180c45d0 2021-02-10T11:50:32.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.669 7f041e0be700 1 -- 172.21.2.112:0/3435562951 --> [v2:172.21.2.112:6816/36028,v1:172.21.2.112:6819/36028] -- command(tid 2: {"prefix": "config set", "key": "osd_debug_deep_scrub_sleep", "value": "0"}) v1 -- 0x7f04180800b0 con 0x7f04180c45d0 2021-02-10T11:50:32.673 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.669 7f040ffff700 1 -- 172.21.2.112:0/3435562951 <== osd.3 v2:172.21.2.112:6816/36028 2 ==== command_reply(tid 2: 0 Set osd_debug_deep_scrub_sleep to 0) v1 ==== 43+0+0 (crc 0 0 0) 0x7f0400006bc0 con 0x7f04180c45d0 2021-02-10T11:50:32.674 INFO:teuthology.orchestra.run.gibba012.stderr:Set osd_debug_deep_scrub_sleep to 0 2021-02-10T11:50:32.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.670 7f041e0be700 1 -- 172.21.2.112:0/3435562951 >> [v2:172.21.2.112:6816/36028,v1:172.21.2.112:6819/36028] conn(0x7f04180c45d0 msgr2=0x7f04180627c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:32.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.670 7f041e0be700 1 --2- 172.21.2.112:0/3435562951 >> [v2:172.21.2.112:6816/36028,v1:172.21.2.112:6819/36028] conn(0x7f04180c45d0 0x7f04180627c0 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:32.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.670 7f041e0be700 1 -- 172.21.2.112:0/3435562951 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f03fc0186c0 msgr2=0x7f03fc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:32.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.670 7f041e0be700 1 --2- 172.21.2.112:0/3435562951 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f03fc0186c0 0x7f03fc01abc0 crc :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:32.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.670 7f041e0be700 1 -- 172.21.2.112:0/3435562951 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f04180bfa90 msgr2=0x7f04180bfe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:32.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.670 7f041e0be700 1 --2- 172.21.2.112:0/3435562951 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f04180bfa90 0x7f04180bfe90 secure :-1 s=READY pgs=3347 cs=0 l=1 rev1=1 rx=0x7f041000b9b0 tx=0x7f041000bd50).stop 2021-02-10T11:50:32.675 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.671 7f0417fff700 1 -- 172.21.2.112:0/3435562951 reap_dead start 2021-02-10T11:50:32.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.671 7f041e0be700 1 -- 172.21.2.112:0/3435562951 shutdown_connections 2021-02-10T11:50:32.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.671 7f041e0be700 1 -- 172.21.2.112:0/3435562951 >> 172.21.2.112:0/3435562951 conn(0x7f0418080cb0 msgr2=0x7f04180950a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:32.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.671 7f041e0be700 1 -- 172.21.2.112:0/3435562951 shutdown_connections 2021-02-10T11:50:32.676 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:32.671 7f041e0be700 1 -- 172.21.2.112:0/3435562951 wait complete. 2021-02-10T11:50:32.683 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd deep-scrub 3 2021-02-10T11:50:33.112 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.091 7ff85a269700 1 Processor -- start 2021-02-10T11:50:33.112 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.091 7ff85a269700 1 -- start start 2021-02-10T11:50:33.112 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.092 7ff85a269700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff8540d0340 0x7ff8540ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:33.112 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.092 7ff85a269700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7ff8540d1c70 0x7ff8540efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:33.113 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.092 7ff85a269700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8540f0b10 con 0x7ff8540d1870 2021-02-10T11:50:33.113 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.092 7ff85a269700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8540e5ed0 con 0x7ff8540d1c70 2021-02-10T11:50:33.113 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.092 7ff85a269700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8541b0be0 con 0x7ff8540d0340 2021-02-10T11:50:33.113 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.092 7ff858886700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7ff8540d1c70 0x7ff8540efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:33.113 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.092 7ff853fff700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7ff8540d1870 0x7ff8541b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:33562/0 (socket says 172.21.2.112:33562) 2021-02-10T11:50:33.113 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.092 7ff858886700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7ff8540d1c70 0x7ff8540efc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:34102/0 (socket says 172.21.2.112:34102) 2021-02-10T11:50:33.113 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.092 7ff853fff700 1 -- 172.21.2.112:0/1415073409 learned_addr learned my addr 172.21.2.112:0/1415073409 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:50:33.113 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.092 7ff858886700 1 -- 172.21.2.112:0/1415073409 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff8540d0340 msgr2=0x7ff8540ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:50:33.114 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.092 7ff858886700 1 --2- 172.21.2.112:0/1415073409 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff8540d0340 0x7ff8540ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:33.114 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.092 7ff858886700 1 -- 172.21.2.112:0/1415073409 >> v1:172.21.2.112:6789/0 conn(0x7ff8540d1870 legacy=0x7ff8541b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:33.114 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.092 7ff858886700 1 -- 172.21.2.112:0/1415073409 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8540f20a0 con 0x7ff8540d1c70 2021-02-10T11:50:33.114 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.092 7ff858886700 1 --2- 172.21.2.112:0/1415073409 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff8540d1c70 0x7ff8540efc60 secure :-1 s=READY pgs=3349 cs=0 l=1 rev1=1 rx=0x7ff84c006e30 tx=0x7ff84c009e60).ready entity=mon.0 client_cookie=325ad301083b383d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:33.114 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.093 7ff852ffd700 1 -- 172.21.2.112:0/1415073409 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff84c006a50 con 0x7ff8540d1c70 2021-02-10T11:50:33.114 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.093 7ff852ffd700 1 -- 172.21.2.112:0/1415073409 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff84c00b300 con 0x7ff8540d1c70 2021-02-10T11:50:33.114 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.093 7ff852ffd700 1 -- 172.21.2.112:0/1415073409 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff84c00b750 con 0x7ff8540d1c70 2021-02-10T11:50:33.114 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.093 7ff85a269700 1 -- 172.21.2.112:0/1415073409 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff8540d1c70 msgr2=0x7ff8540efc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:33.114 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.093 7ff85a269700 1 --2- 172.21.2.112:0/1415073409 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff8540d1c70 0x7ff8540efc60 secure :-1 s=READY pgs=3349 cs=0 l=1 rev1=1 rx=0x7ff84c006e30 tx=0x7ff84c009e60).stop 2021-02-10T11:50:33.115 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.093 7ff85a269700 1 -- 172.21.2.112:0/1415073409 shutdown_connections 2021-02-10T11:50:33.115 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.093 7ff85a269700 1 --2- 172.21.2.112:0/1415073409 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff8540d1c70 0x7ff8540efc60 unknown :-1 s=CLOSED pgs=3349 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:33.115 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.093 7ff85a269700 1 --2- 172.21.2.112:0/1415073409 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff8540d0340 0x7ff8540ef640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:33.115 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.093 7ff85a269700 1 -- 172.21.2.112:0/1415073409 >> 172.21.2.112:0/1415073409 conn(0x7ff8540ce450 msgr2=0x7ff8540d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:33.115 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.093 7ff85a269700 1 -- 172.21.2.112:0/1415073409 shutdown_connections 2021-02-10T11:50:33.115 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.093 7ff85a269700 1 -- 172.21.2.112:0/1415073409 wait complete. 2021-02-10T11:50:33.117 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.108 7ff85a269700 1 Processor -- start 2021-02-10T11:50:33.117 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.108 7ff85a269700 1 -- start start 2021-02-10T11:50:33.117 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.108 7ff85a269700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff8541908b0 0x7ff8541a4440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:33.117 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.108 7ff85a269700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff8540bd920 0x7ff8540bdd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:33.117 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.108 7ff85a269700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff85408d510 0x7ff85408d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:33.117 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.108 7ff85a269700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff85408cf90 con 0x7ff8541908b0 2021-02-10T11:50:33.117 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.108 7ff85a269700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8540be340 con 0x7ff8540bd920 2021-02-10T11:50:33.117 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.108 7ff85a269700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8540be680 con 0x7ff85408d510 2021-02-10T11:50:33.118 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.109 7ff853fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff8541908b0 0x7ff8541a4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:33.118 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.109 7ff853fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff8541908b0 0x7ff8541a4440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:34108/0 (socket says 172.21.2.112:34108) 2021-02-10T11:50:33.118 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.109 7ff853fff700 1 -- 172.21.2.112:0/1199572606 learned_addr learned my addr 172.21.2.112:0/1199572606 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:50:33.118 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.109 7ff853fff700 1 -- 172.21.2.112:0/1199572606 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff85408d510 msgr2=0x7ff85408d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:33.118 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.109 7ff853fff700 1 --2- 172.21.2.112:0/1199572606 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff85408d510 0x7ff85408d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:33.118 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.109 7ff853fff700 1 -- 172.21.2.112:0/1199572606 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff8540bd920 msgr2=0x7ff8540bdd20 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:50:33.118 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.109 7ff853fff700 1 --2- 172.21.2.112:0/1199572606 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff8540bd920 0x7ff8540bdd20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:33.118 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.109 7ff853fff700 1 -- 172.21.2.112:0/1199572606 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8541a4a60 con 0x7ff8541908b0 2021-02-10T11:50:33.119 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.109 7ff853fff700 1 --2- 172.21.2.112:0/1199572606 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff8541908b0 0x7ff8541a4440 secure :-1 s=READY pgs=3350 cs=0 l=1 rev1=1 rx=0x7ff84400a4f0 tx=0x7ff84400a890).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:33.119 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.109 7ff850ff9700 1 -- 172.21.2.112:0/1199572606 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff84400ac80 con 0x7ff8541908b0 2021-02-10T11:50:33.119 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.109 7ff850ff9700 1 -- 172.21.2.112:0/1199572606 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8440009b0 con 0x7ff8541908b0 2021-02-10T11:50:33.119 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.109 7ff850ff9700 1 -- 172.21.2.112:0/1199572606 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff844004590 con 0x7ff8541908b0 2021-02-10T11:50:33.119 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.109 7ff85a269700 1 -- 172.21.2.112:0/1199572606 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8541a4a60 con 0x7ff8541908b0 2021-02-10T11:50:33.119 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.109 7ff85a269700 1 -- 172.21.2.112:0/1199572606 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8541a4a60 con 0x7ff8541908b0 2021-02-10T11:50:33.119 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.110 7ff850ff9700 1 -- 172.21.2.112:0/1199572606 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff844008000 con 0x7ff8541908b0 2021-02-10T11:50:33.119 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.110 7ff850ff9700 1 --2- 172.21.2.112:0/1199572606 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff8340186c0 0x7ff83401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:33.120 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.110 7ff850ff9700 1 -- 172.21.2.112:0/1199572606 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 5573+0+0 (secure 0 0 0) 0x7ff8440325b0 con 0x7ff8541908b0 2021-02-10T11:50:33.120 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.110 7ff8537fe700 1 --2- 172.21.2.112:0/1199572606 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff8340186c0 0x7ff83401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:33.120 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.111 7ff8537fe700 1 --2- 172.21.2.112:0/1199572606 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff8340186c0 0x7ff83401abc0 crc :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:33.146 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.142 7ff85a269700 1 -- 172.21.2.112:0/1199572606 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8540be680 con 0x7ff8541908b0 2021-02-10T11:50:33.148 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.144 7ff850ff9700 1 -- 172.21.2.112:0/1199572606 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff844030de0 con 0x7ff8541908b0 2021-02-10T11:50:33.276 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.272 7ff85a269700 1 -- 172.21.2.112:0/1199572606 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mgr", ""]}) v1 -- 0x7ff854190d00 con 0x7ff8340186c0 2021-02-10T11:50:33.277 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.272 7ff850ff9700 1 -- 172.21.2.112:0/1199572606 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (crc 0 0 0) 0x7ff848006970 con 0x7ff8340186c0 2021-02-10T11:50:33.277 INFO:teuthology.orchestra.run.gibba012.stderr:instructed osd(s) 3 to deep-scrub 2021-02-10T11:50:33.278 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.274 7ff85a269700 1 -- 172.21.2.112:0/1199572606 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff8340186c0 msgr2=0x7ff83401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:33.278 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.274 7ff85a269700 1 --2- 172.21.2.112:0/1199572606 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff8340186c0 0x7ff83401abc0 crc :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:33.278 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.274 7ff85a269700 1 -- 172.21.2.112:0/1199572606 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff8541908b0 msgr2=0x7ff8541a4440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:33.279 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.274 7ff85a269700 1 --2- 172.21.2.112:0/1199572606 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff8541908b0 0x7ff8541a4440 secure :-1 s=READY pgs=3350 cs=0 l=1 rev1=1 rx=0x7ff84400a4f0 tx=0x7ff84400a890).stop 2021-02-10T11:50:33.279 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.274 7ff85a269700 1 -- 172.21.2.112:0/1199572606 shutdown_connections 2021-02-10T11:50:33.279 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.274 7ff85a269700 1 --2- 172.21.2.112:0/1199572606 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff8340186c0 0x7ff83401abc0 unknown :-1 s=CLOSED pgs=1506 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:33.279 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.274 7ff85a269700 1 --2- 172.21.2.112:0/1199572606 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff8540bd920 0x7ff8540bdd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:33.280 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.274 7ff85a269700 1 --2- 172.21.2.112:0/1199572606 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff85408d510 0x7ff85408d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:33.280 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.274 7ff85a269700 1 --2- 172.21.2.112:0/1199572606 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff8541908b0 0x7ff8541a4440 unknown :-1 s=CLOSED pgs=3350 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:33.280 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.274 7ff85a269700 1 -- 172.21.2.112:0/1199572606 >> 172.21.2.112:0/1199572606 conn(0x7ff85408f030 msgr2=0x7ff8540b5e80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:33.280 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.275 7ff85a269700 1 -- 172.21.2.112:0/1199572606 shutdown_connections 2021-02-10T11:50:33.280 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.275 7ff85a269700 1 -- 172.21.2.112:0/1199572606 wait complete. 2021-02-10T11:50:33.287 INFO:tasks.ceph:Scrubbing osd.4 2021-02-10T11:50:33.288 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2021-02-10T11:50:33.719 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.699 7f2154ab3700 1 Processor -- start 2021-02-10T11:50:33.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.699 7f2154ab3700 1 -- start start 2021-02-10T11:50:33.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.700 7f2154ab3700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f21500d1c70 0x7f21500cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:33.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.700 7f2154ab3700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f21500cf580 0x7f21500cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:33.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.700 7f2154ab3700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21500fe1c0 con 0x7f21500d1870 2021-02-10T11:50:33.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.700 7f2154ab3700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21501b0cb0 con 0x7f21500d1c70 2021-02-10T11:50:33.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.700 7f2154ab3700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2150109a60 con 0x7f21500cf580 2021-02-10T11:50:33.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.700 7f214e05d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f21500d1c70 0x7f21500cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:33.720 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.700 7f214e05d700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f21500d1c70 0x7f21500cef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:34116/0 (socket says 172.21.2.112:34116) 2021-02-10T11:50:33.721 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.700 7f214e05d700 1 -- 172.21.2.112:0/3826116078 learned_addr learned my addr 172.21.2.112:0/3826116078 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:50:33.721 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.700 7f214e05d700 1 -- 172.21.2.112:0/3826116078 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f21500cf580 msgr2=0x7f21500cf980 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:50:33.721 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.700 7f214e05d700 1 --2- 172.21.2.112:0/3826116078 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f21500cf580 0x7f21500cf980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:33.721 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.700 7f214e05d700 1 -- 172.21.2.112:0/3826116078 >> v1:172.21.2.120:6789/0 conn(0x7f21500d1870 legacy=0x7f21501b04c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:50:33.721 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.700 7f214e05d700 1 -- 172.21.2.112:0/3826116078 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2150088070 con 0x7f21500d1c70 2021-02-10T11:50:33.721 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.700 7f214e05d700 1 --2- 172.21.2.112:0/3826116078 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f21500d1c70 0x7f21500cef60 secure :-1 s=READY pgs=3351 cs=0 l=1 rev1=1 rx=0x7f2140006700 tx=0x7f2140009b80).ready entity=mon.0 client_cookie=b836021be2b3ba3d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:33.721 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.700 7f214d85c700 1 -- 172.21.2.112:0/3826116078 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f214000c0c0 con 0x7f21500d1c70 2021-02-10T11:50:33.721 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.700 7f214d85c700 1 -- 172.21.2.112:0/3826116078 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f214000c3c0 con 0x7f21500d1c70 2021-02-10T11:50:33.721 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.700 7f214d85c700 1 -- 172.21.2.112:0/3826116078 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f214000c810 con 0x7f21500d1c70 2021-02-10T11:50:33.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.700 7f2154ab3700 1 -- 172.21.2.112:0/3826116078 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f21500d1c70 msgr2=0x7f21500cef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:33.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.700 7f2154ab3700 1 --2- 172.21.2.112:0/3826116078 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f21500d1c70 0x7f21500cef60 secure :-1 s=READY pgs=3351 cs=0 l=1 rev1=1 rx=0x7f2140006700 tx=0x7f2140009b80).stop 2021-02-10T11:50:33.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.701 7f2154ab3700 1 -- 172.21.2.112:0/3826116078 shutdown_connections 2021-02-10T11:50:33.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.701 7f2154ab3700 1 --2- 172.21.2.112:0/3826116078 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f21500d1c70 0x7f21500cef60 unknown :-1 s=CLOSED pgs=3351 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:33.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.701 7f2154ab3700 1 --2- 172.21.2.112:0/3826116078 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f21500cf580 0x7f21500cf980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:33.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.701 7f2154ab3700 1 -- 172.21.2.112:0/3826116078 >> 172.21.2.112:0/3826116078 conn(0x7f21500ce420 msgr2=0x7f21500d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:33.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.701 7f2154ab3700 1 -- 172.21.2.112:0/3826116078 shutdown_connections 2021-02-10T11:50:33.722 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.701 7f2154ab3700 1 -- 172.21.2.112:0/3826116078 wait complete. 2021-02-10T11:50:33.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.715 7f2154ab3700 1 Processor -- start 2021-02-10T11:50:33.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.715 7f2154ab3700 1 -- start start 2021-02-10T11:50:33.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.716 7f2154ab3700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2150086440 0x7f21500b7890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:33.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.716 7f2154ab3700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f21500b7eb0 0x7f21501a0230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:33.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.716 7f2154ab3700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f21500b82b0 0x7f21501a0850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:33.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.716 7f2154ab3700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2150187e70 con 0x7f21500b7eb0 2021-02-10T11:50:33.724 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.716 7f2154ab3700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21501880e0 con 0x7f2150086440 2021-02-10T11:50:33.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.716 7f2154ab3700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21501a0e70 con 0x7f21500b82b0 2021-02-10T11:50:33.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.716 7f214e05d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f21500b7eb0 0x7f21501a0230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:33.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.716 7f214e05d700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f21500b7eb0 0x7f21501a0230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:34124/0 (socket says 172.21.2.112:34124) 2021-02-10T11:50:33.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.716 7f214e05d700 1 -- 172.21.2.112:0/1301312643 learned_addr learned my addr 172.21.2.112:0/1301312643 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:50:33.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.716 7f214e05d700 1 -- 172.21.2.112:0/1301312643 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f21500b82b0 msgr2=0x7f21501a0850 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:50:33.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.716 7f214e05d700 1 --2- 172.21.2.112:0/1301312643 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f21500b82b0 0x7f21501a0850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:33.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.716 7f214e05d700 1 -- 172.21.2.112:0/1301312643 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2150086440 msgr2=0x7f21500b7890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:50:33.725 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.716 7f214e05d700 1 --2- 172.21.2.112:0/1301312643 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f2150086440 0x7f21500b7890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:33.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.716 7f214e05d700 1 -- 172.21.2.112:0/1301312643 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21501a1050 con 0x7f21500b7eb0 2021-02-10T11:50:33.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.716 7f214e05d700 1 --2- 172.21.2.112:0/1301312643 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f21500b7eb0 0x7f21501a0230 secure :-1 s=READY pgs=3352 cs=0 l=1 rev1=1 rx=0x7f214000b650 tx=0x7f214000b9f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:33.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.716 7f21477fe700 1 -- 172.21.2.112:0/1301312643 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f21400090b0 con 0x7f21500b7eb0 2021-02-10T11:50:33.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.716 7f21477fe700 1 -- 172.21.2.112:0/1301312643 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f214000bde0 con 0x7f21500b7eb0 2021-02-10T11:50:33.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.716 7f21477fe700 1 -- 172.21.2.112:0/1301312643 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2140002220 con 0x7f21500b7eb0 2021-02-10T11:50:33.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.716 7f2154ab3700 1 -- 172.21.2.112:0/1301312643 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21501a1050 con 0x7f21500b7eb0 2021-02-10T11:50:33.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.717 7f2154ab3700 1 -- 172.21.2.112:0/1301312643 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2150188950 con 0x7f21500b7eb0 2021-02-10T11:50:33.726 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.717 7f21477fe700 1 -- 172.21.2.112:0/1301312643 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f21400052e0 con 0x7f21500b7eb0 2021-02-10T11:50:33.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.717 7f21477fe700 1 --2- 172.21.2.112:0/1301312643 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f21340186c0 0x7f213401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:33.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.717 7f21477fe700 1 -- 172.21.2.112:0/1301312643 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f2140032650 con 0x7f21500b7eb0 2021-02-10T11:50:33.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.718 7f214e85e700 1 --2- 172.21.2.112:0/1301312643 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f21340186c0 0x7f213401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:33.727 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.718 7f214e85e700 1 --2- 172.21.2.112:0/1301312643 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f21340186c0 0x7f213401abc0 crc :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:33.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.749 7f2154ab3700 1 --2- 172.21.2.112:0/1301312643 >> [v2:172.21.2.120:6810/36388,v1:172.21.2.120:6813/36388] conn(0x7f21501891e0 0x7f21501895e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:33.753 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.749 7f2154ab3700 1 -- 172.21.2.112:0/1301312643 --> [v2:172.21.2.120:6810/36388,v1:172.21.2.120:6813/36388] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2150188950 con 0x7f21501891e0 2021-02-10T11:50:33.754 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.749 7f214f05f700 1 --2- 172.21.2.112:0/1301312643 >> [v2:172.21.2.120:6810/36388,v1:172.21.2.120:6813/36388] conn(0x7f21501891e0 0x7f21501895e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:33.754 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.750 7f214f05f700 1 --2- 172.21.2.112:0/1301312643 >> [v2:172.21.2.120:6810/36388,v1:172.21.2.120:6813/36388] conn(0x7f21501891e0 0x7f21501895e0 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:33.755 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.751 7f21477fe700 1 -- 172.21.2.112:0/1301312643 <== osd.4 v2:172.21.2.120:6810/36388 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+5151 (crc 0 0 0) 0x7f2148006d10 con 0x7f21501891e0 2021-02-10T11:50:33.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.758 7f2154ab3700 1 -- 172.21.2.112:0/1301312643 --> [v2:172.21.2.120:6810/36388,v1:172.21.2.120:6813/36388] -- command(tid 2: {"prefix": "config set", "key": "osd_debug_deep_scrub_sleep", "value": "0"}) v1 -- 0x7f2150188950 con 0x7f21501891e0 2021-02-10T11:50:33.762 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.758 7f21477fe700 1 -- 172.21.2.112:0/1301312643 <== osd.4 v2:172.21.2.120:6810/36388 2 ==== command_reply(tid 2: 0 Set osd_debug_deep_scrub_sleep to 0) v1 ==== 43+0+0 (crc 0 0 0) 0x7f2148006db0 con 0x7f21501891e0 2021-02-10T11:50:33.763 INFO:teuthology.orchestra.run.gibba012.stderr:Set osd_debug_deep_scrub_sleep to 0 2021-02-10T11:50:33.764 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.759 7f2154ab3700 1 -- 172.21.2.112:0/1301312643 >> [v2:172.21.2.120:6810/36388,v1:172.21.2.120:6813/36388] conn(0x7f21501891e0 msgr2=0x7f21501895e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:33.764 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.759 7f2154ab3700 1 --2- 172.21.2.112:0/1301312643 >> [v2:172.21.2.120:6810/36388,v1:172.21.2.120:6813/36388] conn(0x7f21501891e0 0x7f21501895e0 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:33.764 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.759 7f2154ab3700 1 -- 172.21.2.112:0/1301312643 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f21340186c0 msgr2=0x7f213401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:33.764 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.759 7f2154ab3700 1 --2- 172.21.2.112:0/1301312643 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f21340186c0 0x7f213401abc0 crc :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:33.765 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.759 7f2154ab3700 1 -- 172.21.2.112:0/1301312643 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f21500b7eb0 msgr2=0x7f21501a0230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:33.765 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.759 7f2154ab3700 1 --2- 172.21.2.112:0/1301312643 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f21500b7eb0 0x7f21501a0230 secure :-1 s=READY pgs=3352 cs=0 l=1 rev1=1 rx=0x7f214000b650 tx=0x7f214000b9f0).stop 2021-02-10T11:50:33.765 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.759 7f214f05f700 1 -- 172.21.2.112:0/1301312643 reap_dead start 2021-02-10T11:50:33.765 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.760 7f2154ab3700 1 -- 172.21.2.112:0/1301312643 shutdown_connections 2021-02-10T11:50:33.765 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.760 7f2154ab3700 1 -- 172.21.2.112:0/1301312643 >> 172.21.2.112:0/1301312643 conn(0x7f2150098ab0 msgr2=0x7f21501974b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:33.765 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.760 7f2154ab3700 1 -- 172.21.2.112:0/1301312643 shutdown_connections 2021-02-10T11:50:33.765 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:33.760 7f2154ab3700 1 -- 172.21.2.112:0/1301312643 wait complete. 2021-02-10T11:50:33.772 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd deep-scrub 4 2021-02-10T11:50:34.202 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.181 7f5283c29700 1 Processor -- start 2021-02-10T11:50:34.202 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.181 7f5283c29700 1 -- start start 2021-02-10T11:50:34.202 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.182 7f5283c29700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f527c0d0340 0x7f527c0ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:34.202 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.182 7f5283c29700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f527c0d1c70 0x7f527c0efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:34.203 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.182 7f5283c29700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f527c0f0b10 con 0x7f527c0d1870 2021-02-10T11:50:34.203 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.182 7f5283c29700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f527c0e5ed0 con 0x7f527c0d0340 2021-02-10T11:50:34.203 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.182 7f5283c29700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f527c1b0be0 con 0x7f527c0d1c70 2021-02-10T11:50:34.203 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.182 7f5281244700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f527c0d0340 0x7f527c0ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:34.203 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.182 7f5281244700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f527c0d0340 0x7f527c0ef640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:34130/0 (socket says 172.21.2.112:34130) 2021-02-10T11:50:34.203 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.182 7f5281244700 1 -- 172.21.2.112:0/188378571 learned_addr learned my addr 172.21.2.112:0/188378571 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:50:34.203 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.182 7f5282246700 1 --2- 172.21.2.112:0/188378571 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f527c0d1c70 0x7f527c0efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:34.204 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.182 7f5281244700 1 -- 172.21.2.112:0/188378571 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f527c0d1c70 msgr2=0x7f527c0efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:50:34.204 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.182 7f5281244700 1 --2- 172.21.2.112:0/188378571 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f527c0d1c70 0x7f527c0efc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:34.204 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.182 7f5281244700 1 -- 172.21.2.112:0/188378571 >> v1:172.21.2.120:6789/0 conn(0x7f527c0d1870 legacy=0x7f527c1b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:34.204 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.182 7f5281244700 1 -- 172.21.2.112:0/188378571 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f527c0f20a0 con 0x7f527c0d0340 2021-02-10T11:50:34.204 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.182 7f5281244700 1 --2- 172.21.2.112:0/188378571 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f527c0d0340 0x7f527c0ef640 secure :-1 s=READY pgs=3353 cs=0 l=1 rev1=1 rx=0x7f5278006700 tx=0x7f5278009b80).ready entity=mon.0 client_cookie=dd74d268cb740100 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:34.204 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.182 7f5280a43700 1 -- 172.21.2.112:0/188378571 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f527800c0c0 con 0x7f527c0d0340 2021-02-10T11:50:34.204 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.183 7f5280a43700 1 -- 172.21.2.112:0/188378571 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f527800c340 con 0x7f527c0d0340 2021-02-10T11:50:34.204 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.183 7f5280a43700 1 -- 172.21.2.112:0/188378571 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f527800c860 con 0x7f527c0d0340 2021-02-10T11:50:34.204 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.183 7f5283c29700 1 -- 172.21.2.112:0/188378571 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f527c0d0340 msgr2=0x7f527c0ef640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:34.205 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.183 7f5283c29700 1 --2- 172.21.2.112:0/188378571 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f527c0d0340 0x7f527c0ef640 secure :-1 s=READY pgs=3353 cs=0 l=1 rev1=1 rx=0x7f5278006700 tx=0x7f5278009b80).stop 2021-02-10T11:50:34.205 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.183 7f5283c29700 1 -- 172.21.2.112:0/188378571 shutdown_connections 2021-02-10T11:50:34.205 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.183 7f5283c29700 1 --2- 172.21.2.112:0/188378571 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f527c0d0340 0x7f527c0ef640 unknown :-1 s=CLOSED pgs=3353 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:34.205 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.183 7f5283c29700 1 --2- 172.21.2.112:0/188378571 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f527c0d1c70 0x7f527c0efc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:34.205 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.183 7f5283c29700 1 -- 172.21.2.112:0/188378571 >> 172.21.2.112:0/188378571 conn(0x7f527c0ce450 msgr2=0x7f527c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:34.205 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.183 7f5283c29700 1 -- 172.21.2.112:0/188378571 shutdown_connections 2021-02-10T11:50:34.205 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.183 7f5283c29700 1 -- 172.21.2.112:0/188378571 wait complete. 2021-02-10T11:50:34.207 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.198 7f5283c29700 1 Processor -- start 2021-02-10T11:50:34.207 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.198 7f5283c29700 1 -- start start 2021-02-10T11:50:34.207 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.198 7f5283c29700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f527c19ffa0 0x7f527c0eda10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:34.208 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.198 7f5283c29700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f527c1a03a0 0x7f527c0ee030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:34.208 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.198 7f5283c29700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f527c0ee650 0x7f527c0ebe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:34.208 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.198 7f5283c29700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f527c0ed490 con 0x7f527c0ee650 2021-02-10T11:50:34.208 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.198 7f5283c29700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f527c0eea50 con 0x7f527c1a03a0 2021-02-10T11:50:34.208 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.198 7f5283c29700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f527c0eec30 con 0x7f527c19ffa0 2021-02-10T11:50:34.208 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.199 7f5282246700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f527c0ee650 0x7f527c0ebe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:34.208 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.199 7f5282246700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f527c0ee650 0x7f527c0ebe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:34140/0 (socket says 172.21.2.112:34140) 2021-02-10T11:50:34.208 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.199 7f5282246700 1 -- 172.21.2.112:0/26254612 learned_addr learned my addr 172.21.2.112:0/26254612 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:50:34.209 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.199 7f5281a45700 1 --2- 172.21.2.112:0/26254612 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f527c19ffa0 0x7f527c0eda10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:34.209 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.199 7f5281244700 1 --2- 172.21.2.112:0/26254612 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f527c1a03a0 0x7f527c0ee030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:34.209 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.199 7f5282246700 1 -- 172.21.2.112:0/26254612 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f527c19ffa0 msgr2=0x7f527c0eda10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:34.209 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.199 7f5282246700 1 --2- 172.21.2.112:0/26254612 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f527c19ffa0 0x7f527c0eda10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:34.209 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.199 7f5282246700 1 -- 172.21.2.112:0/26254612 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f527c1a03a0 msgr2=0x7f527c0ee030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:34.209 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.199 7f5282246700 1 --2- 172.21.2.112:0/26254612 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f527c1a03a0 0x7f527c0ee030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:34.209 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.199 7f5282246700 1 -- 172.21.2.112:0/26254612 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f527c0ec4a0 con 0x7f527c0ee650 2021-02-10T11:50:34.209 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.199 7f5281244700 1 --2- 172.21.2.112:0/26254612 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f527c1a03a0 0x7f527c0ee030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:50:34.210 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.199 7f5282246700 1 --2- 172.21.2.112:0/26254612 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f527c0ee650 0x7f527c0ebe80 secure :-1 s=READY pgs=3354 cs=0 l=1 rev1=1 rx=0x7f527400b970 tx=0x7f527400bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:34.210 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.200 7f526e7fc700 1 -- 172.21.2.112:0/26254612 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5274000f90 con 0x7f527c0ee650 2021-02-10T11:50:34.210 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.200 7f526e7fc700 1 -- 172.21.2.112:0/26254612 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52740066f0 con 0x7f527c0ee650 2021-02-10T11:50:34.210 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.200 7f5283c29700 1 -- 172.21.2.112:0/26254612 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f527c0ec4a0 con 0x7f527c0ee650 2021-02-10T11:50:34.210 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.200 7f526e7fc700 1 -- 172.21.2.112:0/26254612 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f52740015f0 con 0x7f527c0ee650 2021-02-10T11:50:34.210 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.200 7f5283c29700 1 -- 172.21.2.112:0/26254612 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f527c0ec4a0 con 0x7f527c0ee650 2021-02-10T11:50:34.210 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.200 7f526e7fc700 1 -- 172.21.2.112:0/26254612 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5274000df0 con 0x7f527c0ee650 2021-02-10T11:50:34.210 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.200 7f526e7fc700 1 --2- 172.21.2.112:0/26254612 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f52640186d0 0x7f526401abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:34.211 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.200 7f526e7fc700 1 -- 172.21.2.112:0/26254612 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f5274032b40 con 0x7f527c0ee650 2021-02-10T11:50:34.211 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.201 7f5281a45700 1 --2- 172.21.2.112:0/26254612 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f52640186d0 0x7f526401abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:34.211 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.201 7f5281a45700 1 --2- 172.21.2.112:0/26254612 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f52640186d0 0x7f526401abd0 crc :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:34.236 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.232 7f5283c29700 1 -- 172.21.2.112:0/26254612 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f527c0ebc50 con 0x7f527c0ee650 2021-02-10T11:50:34.238 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.234 7f526e7fc700 1 -- 172.21.2.112:0/26254612 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f52740066f0 con 0x7f527c0ee650 2021-02-10T11:50:34.367 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.362 7f5283c29700 1 -- 172.21.2.112:0/26254612 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mgr", ""]}) v1 -- 0x7f527c0ebc50 con 0x7f52640186d0 2021-02-10T11:50:34.367 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.363 7f526e7fc700 1 -- 172.21.2.112:0/26254612 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (crc 0 0 0) 0x7f5270000960 con 0x7f52640186d0 2021-02-10T11:50:34.367 INFO:teuthology.orchestra.run.gibba012.stderr:instructed osd(s) 4 to deep-scrub 2021-02-10T11:50:34.369 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.365 7f5283c29700 1 -- 172.21.2.112:0/26254612 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f52640186d0 msgr2=0x7f526401abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:34.369 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.365 7f5283c29700 1 --2- 172.21.2.112:0/26254612 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f52640186d0 0x7f526401abd0 crc :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:34.370 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.365 7f5283c29700 1 -- 172.21.2.112:0/26254612 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f527c0ee650 msgr2=0x7f527c0ebe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:34.370 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.365 7f5283c29700 1 --2- 172.21.2.112:0/26254612 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f527c0ee650 0x7f527c0ebe80 secure :-1 s=READY pgs=3354 cs=0 l=1 rev1=1 rx=0x7f527400b970 tx=0x7f527400bd10).stop 2021-02-10T11:50:34.370 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.365 7f5283c29700 1 -- 172.21.2.112:0/26254612 shutdown_connections 2021-02-10T11:50:34.370 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.365 7f5283c29700 1 --2- 172.21.2.112:0/26254612 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f52640186d0 0x7f526401abd0 unknown :-1 s=CLOSED pgs=1508 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:34.370 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.365 7f5283c29700 1 --2- 172.21.2.112:0/26254612 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f527c1a03a0 0x7f527c0ee030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:34.370 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.365 7f5283c29700 1 --2- 172.21.2.112:0/26254612 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f527c0ee650 0x7f527c0ebe80 unknown :-1 s=CLOSED pgs=3354 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:34.370 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.365 7f5283c29700 1 --2- 172.21.2.112:0/26254612 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f527c19ffa0 0x7f527c0eda10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:34.370 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.365 7f5283c29700 1 -- 172.21.2.112:0/26254612 >> 172.21.2.112:0/26254612 conn(0x7f527c094d90 msgr2=0x7f527c08a420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:34.371 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.365 7f5283c29700 1 -- 172.21.2.112:0/26254612 shutdown_connections 2021-02-10T11:50:34.371 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.365 7f5283c29700 1 -- 172.21.2.112:0/26254612 wait complete. 2021-02-10T11:50:34.378 INFO:tasks.ceph:Scrubbing osd.5 2021-02-10T11:50:34.378 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2021-02-10T11:50:34.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.789 7f6a53c79700 1 Processor -- start 2021-02-10T11:50:34.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.789 7f6a53c79700 1 -- start start 2021-02-10T11:50:34.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.790 7f6a53c79700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7f6a4c0d1870 0x7f6a4c1b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:34.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.790 7f6a53c79700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6a4c1b0ae0 0x7f6a4c108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:34.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.790 7f6a53c79700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a4c0fe1c0 con 0x7f6a4c0d1c70 2021-02-10T11:50:34.813 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.790 7f6a53c79700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a4c109a60 con 0x7f6a4c0d1870 2021-02-10T11:50:34.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.790 7f6a53c79700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a4c109750 con 0x7f6a4c1b0ae0 2021-02-10T11:50:34.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.790 7f6a52296700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f6a4c0d1c70 0x7f6a4c108f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:33606/0 (socket says 172.21.2.112:33606) 2021-02-10T11:50:34.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.790 7f6a52296700 1 -- 172.21.2.112:0/1549164638 learned_addr learned my addr 172.21.2.112:0/1549164638 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:50:34.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.790 7f6a50a93700 1 -- 172.21.2.112:0/1549164638 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4060985820 0 0) 0x7f6a44001500 con 0x7f6a4c0d1c70 2021-02-10T11:50:34.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.791 7f6a50a93700 1 -- 172.21.2.112:0/1549164638 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a38002fe0 con 0x7f6a4c0d1c70 2021-02-10T11:50:34.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.791 7f6a51294700 1 --2- 172.21.2.112:0/1549164638 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6a4c1b0ae0 0x7f6a4c108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:34.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.791 7f6a51a95700 1 --2- 172.21.2.112:0/1549164638 >> v2:172.21.2.120:3300/0 conn(0x7f6a4c0d1870 0x7f6a4c1b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:34.814 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.791 7f6a50a93700 1 -- 172.21.2.112:0/1549164638 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 155072655 0 0) 0x7f6a44001460 con 0x7f6a4c0d1c70 2021-02-10T11:50:34.815 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.791 7f6a50a93700 1 -- 172.21.2.112:0/1549164638 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6a4c1b0ae0 msgr2=0x7f6a4c108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:50:34.815 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.791 7f6a50a93700 1 --2- 172.21.2.112:0/1549164638 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6a4c1b0ae0 0x7f6a4c108940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:34.815 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.791 7f6a50a93700 1 -- 172.21.2.112:0/1549164638 >> v2:172.21.2.120:3300/0 conn(0x7f6a4c0d1870 msgr2=0x7f6a4c1b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:50:34.815 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.791 7f6a50a93700 1 --2- 172.21.2.112:0/1549164638 >> v2:172.21.2.120:3300/0 conn(0x7f6a4c0d1870 0x7f6a4c1b04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:34.815 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.791 7f6a50a93700 1 -- 172.21.2.112:0/1549164638 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a4c088070 con 0x7f6a4c0d1c70 2021-02-10T11:50:34.815 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.791 7f6a50a93700 1 -- 172.21.2.112:0/1549164638 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f6a44001980 con 0x7f6a4c0d1c70 2021-02-10T11:50:34.815 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.791 7f6a50a93700 1 -- 172.21.2.112:0/1549164638 >> v1:172.21.2.112:6789/0 conn(0x7f6a4c0d1c70 legacy=0x7f6a4c108f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:34.816 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.791 7f6a50a93700 1 --2- 172.21.2.112:0/1549164638 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6a38003a40 0x7f6a38005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:34.816 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.791 7f6a50a93700 1 --2- 172.21.2.112:0/1549164638 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6a3804a7a0 0x7f6a3804cca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:34.816 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.791 7f6a50a93700 1 --2- 172.21.2.112:0/1549164638 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6a3804d5a0 0x7f6a3804fa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:34.816 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.791 7f6a50a93700 1 -- 172.21.2.112:0/1549164638 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a380502e0 con 0x7f6a3804a7a0 2021-02-10T11:50:34.816 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.791 7f6a50a93700 1 -- 172.21.2.112:0/1549164638 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a38050500 con 0x7f6a38003a40 2021-02-10T11:50:34.816 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.791 7f6a50a93700 1 -- 172.21.2.112:0/1549164638 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a38050720 con 0x7f6a3804d5a0 2021-02-10T11:50:34.816 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.791 7f6a51294700 1 --2- 172.21.2.112:0/1549164638 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6a3804a7a0 0x7f6a3804cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:34.816 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.791 7f6a52296700 1 --2- 172.21.2.112:0/1549164638 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6a3804d5a0 0x7f6a3804fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:34.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.791 7f6a51a95700 1 --2- 172.21.2.112:0/1549164638 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6a38003a40 0x7f6a38005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:34.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.792 7f6a51294700 1 -- 172.21.2.112:0/1549164638 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6a3804d5a0 msgr2=0x7f6a3804fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:50:34.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.792 7f6a51294700 1 --2- 172.21.2.112:0/1549164638 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6a3804d5a0 0x7f6a3804fa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:34.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.792 7f6a51294700 1 -- 172.21.2.112:0/1549164638 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6a38003a40 msgr2=0x7f6a38005ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:50:34.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.792 7f6a51294700 1 --2- 172.21.2.112:0/1549164638 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6a38003a40 0x7f6a38005ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:34.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.792 7f6a51294700 1 -- 172.21.2.112:0/1549164638 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a380509c0 con 0x7f6a3804a7a0 2021-02-10T11:50:34.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.792 7f6a51a95700 1 --2- 172.21.2.112:0/1549164638 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6a38003a40 0x7f6a38005ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:50:34.817 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.792 7f6a51294700 1 --2- 172.21.2.112:0/1549164638 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6a3804a7a0 0x7f6a3804cca0 secure :-1 s=READY pgs=3356 cs=0 l=1 rev1=1 rx=0x7f6a40006ac0 tx=0x7f6a40009a30).ready entity=mon.0 client_cookie=ea83712c8cec3f0d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:34.818 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.792 7f6a50a93700 1 -- 172.21.2.112:0/1549164638 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6a4000c0c0 con 0x7f6a3804a7a0 2021-02-10T11:50:34.818 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.792 7f6a50a93700 1 -- 172.21.2.112:0/1549164638 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a40009e20 con 0x7f6a3804a7a0 2021-02-10T11:50:34.818 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.792 7f6a50a93700 1 -- 172.21.2.112:0/1549164638 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6a4000c0c0 con 0x7f6a3804a7a0 2021-02-10T11:50:34.818 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.792 7f6a53c79700 1 -- 172.21.2.112:0/1549164638 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6a3804a7a0 msgr2=0x7f6a3804cca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:34.818 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.792 7f6a53c79700 1 --2- 172.21.2.112:0/1549164638 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6a3804a7a0 0x7f6a3804cca0 secure :-1 s=READY pgs=3356 cs=0 l=1 rev1=1 rx=0x7f6a40006ac0 tx=0x7f6a40009a30).stop 2021-02-10T11:50:34.818 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.792 7f6a52296700 1 -- 172.21.2.112:0/1549164638 reap_dead start 2021-02-10T11:50:34.818 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.792 7f6a53c79700 1 -- 172.21.2.112:0/1549164638 shutdown_connections 2021-02-10T11:50:34.818 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.792 7f6a53c79700 1 -- 172.21.2.112:0/1549164638 >> 172.21.2.112:0/1549164638 conn(0x7f6a4c0ce420 msgr2=0x7f6a4c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:34.819 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.793 7f6a53c79700 1 -- 172.21.2.112:0/1549164638 shutdown_connections 2021-02-10T11:50:34.819 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.793 7f6a53c79700 1 -- 172.21.2.112:0/1549164638 wait complete. 2021-02-10T11:50:34.819 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.807 7f6a53c79700 1 Processor -- start 2021-02-10T11:50:34.819 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.807 7f6a53c79700 1 -- start start 2021-02-10T11:50:34.819 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.808 7f6a53c79700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6a4c0a8f60 0x7f6a4c0b2250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:34.819 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.808 7f6a53c79700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6a4c1a7ae0 0x7f6a4c1a7ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:34.819 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.808 7f6a53c79700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6a4c0b50a0 0x7f6a4c0b54a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:34.819 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.808 7f6a53c79700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a4c0b5b40 con 0x7f6a4c1a7ae0 2021-02-10T11:50:34.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.808 7f6a53c79700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a4c0b5d60 con 0x7f6a4c0a8f60 2021-02-10T11:50:34.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.808 7f6a53c79700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a4c0b5f80 con 0x7f6a4c0b50a0 2021-02-10T11:50:34.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.808 7f6a51294700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6a4c1a7ae0 0x7f6a4c1a7ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:34.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.808 7f6a51294700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6a4c1a7ae0 0x7f6a4c1a7ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:34156/0 (socket says 172.21.2.112:34156) 2021-02-10T11:50:34.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.808 7f6a51294700 1 -- 172.21.2.112:0/252293846 learned_addr learned my addr 172.21.2.112:0/252293846 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:50:34.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.808 7f6a51a95700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6a4c0a8f60 0x7f6a4c0b2250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:34.820 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.808 7f6a52296700 1 --2- 172.21.2.112:0/252293846 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6a4c0b50a0 0x7f6a4c0b54a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:34.822 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.809 7f6a51294700 1 -- 172.21.2.112:0/252293846 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6a4c0b50a0 msgr2=0x7f6a4c0b54a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:34.822 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.809 7f6a51294700 1 --2- 172.21.2.112:0/252293846 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6a4c0b50a0 0x7f6a4c0b54a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:34.822 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.809 7f6a51294700 1 -- 172.21.2.112:0/252293846 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6a4c0a8f60 msgr2=0x7f6a4c0b2250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:34.822 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.809 7f6a51294700 1 --2- 172.21.2.112:0/252293846 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6a4c0a8f60 0x7f6a4c0b2250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:34.822 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.809 7f6a51294700 1 -- 172.21.2.112:0/252293846 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a4c0b6be0 con 0x7f6a4c1a7ae0 2021-02-10T11:50:34.822 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.809 7f6a51294700 1 --2- 172.21.2.112:0/252293846 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6a4c1a7ae0 0x7f6a4c1a7ee0 secure :-1 s=READY pgs=3357 cs=0 l=1 rev1=1 rx=0x7f6a400066a0 tx=0x7f6a4000b9a0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:34.823 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.809 7f6a3e7fc700 1 -- 172.21.2.112:0/252293846 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6a4000bd90 con 0x7f6a4c1a7ae0 2021-02-10T11:50:34.823 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.809 7f6a3e7fc700 1 -- 172.21.2.112:0/252293846 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a40006030 con 0x7f6a4c1a7ae0 2021-02-10T11:50:34.823 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.809 7f6a53c79700 1 -- 172.21.2.112:0/252293846 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a4c0b6be0 con 0x7f6a4c1a7ae0 2021-02-10T11:50:34.823 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.809 7f6a3e7fc700 1 -- 172.21.2.112:0/252293846 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6a400052e0 con 0x7f6a4c1a7ae0 2021-02-10T11:50:34.823 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.809 7f6a53c79700 1 -- 172.21.2.112:0/252293846 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a4c0b6be0 con 0x7f6a4c1a7ae0 2021-02-10T11:50:34.823 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.810 7f6a3e7fc700 1 -- 172.21.2.112:0/252293846 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6a40006210 con 0x7f6a4c1a7ae0 2021-02-10T11:50:34.823 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.810 7f6a3e7fc700 1 --2- 172.21.2.112:0/252293846 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6a34018730 0x7f6a3401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:34.823 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.810 7f6a3e7fc700 1 -- 172.21.2.112:0/252293846 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f6a400327d0 con 0x7f6a4c1a7ae0 2021-02-10T11:50:34.823 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.810 7f6a51a95700 1 --2- 172.21.2.112:0/252293846 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6a34018730 0x7f6a3401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:34.824 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.811 7f6a51a95700 1 --2- 172.21.2.112:0/252293846 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6a34018730 0x7f6a3401ac10 crc :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:34.844 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.840 7f6a53c79700 1 --2- 172.21.2.112:0/252293846 >> [v2:172.21.2.120:6804/36387,v1:172.21.2.120:6806/36387] conn(0x7f6a4c0c45d0 0x7f6a4c0b77d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:34.844 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.840 7f6a53c79700 1 -- 172.21.2.112:0/252293846 --> [v2:172.21.2.120:6804/36387,v1:172.21.2.120:6806/36387] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6a4c0825d0 con 0x7f6a4c0c45d0 2021-02-10T11:50:34.845 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.841 7f6a52296700 1 --2- 172.21.2.112:0/252293846 >> [v2:172.21.2.120:6804/36387,v1:172.21.2.120:6806/36387] conn(0x7f6a4c0c45d0 0x7f6a4c0b77d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:34.845 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.841 7f6a52296700 1 --2- 172.21.2.112:0/252293846 >> [v2:172.21.2.120:6804/36387,v1:172.21.2.120:6806/36387] conn(0x7f6a4c0c45d0 0x7f6a4c0b77d0 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:34.846 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.842 7f6a3e7fc700 1 -- 172.21.2.112:0/252293846 <== osd.5 v2:172.21.2.120:6804/36387 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+5151 (crc 0 0 0) 0x7f6a44007a70 con 0x7f6a4c0c45d0 2021-02-10T11:50:34.853 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.849 7f6a53c79700 1 -- 172.21.2.112:0/252293846 --> [v2:172.21.2.120:6804/36387,v1:172.21.2.120:6806/36387] -- command(tid 2: {"prefix": "config set", "key": "osd_debug_deep_scrub_sleep", "value": "0"}) v1 -- 0x7f6a4c0b6be0 con 0x7f6a4c0c45d0 2021-02-10T11:50:34.854 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.850 7f6a3e7fc700 1 -- 172.21.2.112:0/252293846 <== osd.5 v2:172.21.2.120:6804/36387 2 ==== command_reply(tid 2: 0 Set osd_debug_deep_scrub_sleep to 0) v1 ==== 43+0+0 (crc 0 0 0) 0x7f6a44007a70 con 0x7f6a4c0c45d0 2021-02-10T11:50:34.854 INFO:teuthology.orchestra.run.gibba012.stderr:Set osd_debug_deep_scrub_sleep to 0 2021-02-10T11:50:34.855 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.850 7f6a53c79700 1 -- 172.21.2.112:0/252293846 >> [v2:172.21.2.120:6804/36387,v1:172.21.2.120:6806/36387] conn(0x7f6a4c0c45d0 msgr2=0x7f6a4c0b77d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:34.855 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.850 7f6a53c79700 1 --2- 172.21.2.112:0/252293846 >> [v2:172.21.2.120:6804/36387,v1:172.21.2.120:6806/36387] conn(0x7f6a4c0c45d0 0x7f6a4c0b77d0 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:34.855 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.850 7f6a53c79700 1 -- 172.21.2.112:0/252293846 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6a34018730 msgr2=0x7f6a3401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:34.855 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.850 7f6a53c79700 1 --2- 172.21.2.112:0/252293846 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6a34018730 0x7f6a3401ac10 crc :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:34.855 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.850 7f6a53c79700 1 -- 172.21.2.112:0/252293846 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6a4c1a7ae0 msgr2=0x7f6a4c1a7ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:34.856 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.850 7f6a53c79700 1 --2- 172.21.2.112:0/252293846 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6a4c1a7ae0 0x7f6a4c1a7ee0 secure :-1 s=READY pgs=3357 cs=0 l=1 rev1=1 rx=0x7f6a400066a0 tx=0x7f6a4000b9a0).stop 2021-02-10T11:50:34.856 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.850 7f6a52296700 1 -- 172.21.2.112:0/252293846 reap_dead start 2021-02-10T11:50:34.856 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.850 7f6a53c79700 1 -- 172.21.2.112:0/252293846 shutdown_connections 2021-02-10T11:50:34.856 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.850 7f6a53c79700 1 -- 172.21.2.112:0/252293846 >> 172.21.2.112:0/252293846 conn(0x7f6a4c0bc070 msgr2=0x7f6a4c0ba770 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:34.856 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.850 7f6a53c79700 1 -- 172.21.2.112:0/252293846 shutdown_connections 2021-02-10T11:50:34.856 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:34.850 7f6a53c79700 1 -- 172.21.2.112:0/252293846 wait complete. 2021-02-10T11:50:34.863 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd deep-scrub 5 2021-02-10T11:50:35.292 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.271 7f9170bb8700 1 Processor -- start 2021-02-10T11:50:35.292 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.271 7f9170bb8700 1 -- start start 2021-02-10T11:50:35.293 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.272 7f9170bb8700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f916c0d1870 0x7f916c1b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:35.293 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.272 7f9170bb8700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f916c1b0a10 0x7f916c0cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:35.293 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.272 7f9170bb8700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f916c0f0b10 con 0x7f916c0d1c70 2021-02-10T11:50:35.293 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.272 7f9170bb8700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f916c0e5ed0 con 0x7f916c0d1870 2021-02-10T11:50:35.293 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.272 7f9170bb8700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f916c0f0650 con 0x7f916c1b0a10 2021-02-10T11:50:35.293 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.272 7f916a85e700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f916c0d1870 0x7f916c1b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:35.293 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.272 7f916a85e700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f916c0d1870 0x7f916c1b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:34164/0 (socket says 172.21.2.112:34164) 2021-02-10T11:50:35.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.272 7f916a85e700 1 -- 172.21.2.112:0/2320724945 learned_addr learned my addr 172.21.2.112:0/2320724945 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:50:35.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.272 7f916985c700 1 -- 172.21.2.112:0/2320724945 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2435992464 0 0) 0x7f9164000f90 con 0x7f916c0d1c70 2021-02-10T11:50:35.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.272 7f916a85e700 1 -- 172.21.2.112:0/2320724945 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f916c1b0a10 msgr2=0x7f916c0cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:50:35.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.272 7f916a85e700 1 --2- 172.21.2.112:0/2320724945 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f916c1b0a10 0x7f916c0cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:35.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.272 7f916a85e700 1 -- 172.21.2.112:0/2320724945 >> v1:172.21.2.112:6789/0 conn(0x7f916c0d1c70 legacy=0x7f916c0cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:35.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.272 7f916a85e700 1 -- 172.21.2.112:0/2320724945 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f916c0f0870 con 0x7f916c0d1870 2021-02-10T11:50:35.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.273 7f916a85e700 1 --2- 172.21.2.112:0/2320724945 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f916c0d1870 0x7f916c1b03f0 secure :-1 s=READY pgs=3359 cs=0 l=1 rev1=1 rx=0x7f915c006700 tx=0x7f915c009b80).ready entity=mon.0 client_cookie=f1c8eadab4457cce server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:35.294 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.273 7f916985c700 1 -- 172.21.2.112:0/2320724945 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f915c00c0c0 con 0x7f916c0d1870 2021-02-10T11:50:35.295 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.273 7f916985c700 1 -- 172.21.2.112:0/2320724945 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f915c00c340 con 0x7f916c0d1870 2021-02-10T11:50:35.295 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.273 7f916985c700 1 -- 172.21.2.112:0/2320724945 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f915c00c860 con 0x7f916c0d1870 2021-02-10T11:50:35.295 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.273 7f9170bb8700 1 -- 172.21.2.112:0/2320724945 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f916c0d1870 msgr2=0x7f916c1b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:35.295 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.273 7f9170bb8700 1 --2- 172.21.2.112:0/2320724945 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f916c0d1870 0x7f916c1b03f0 secure :-1 s=READY pgs=3359 cs=0 l=1 rev1=1 rx=0x7f915c006700 tx=0x7f915c009b80).stop 2021-02-10T11:50:35.295 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.274 7f9170bb8700 1 -- 172.21.2.112:0/2320724945 shutdown_connections 2021-02-10T11:50:35.295 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.274 7f9170bb8700 1 --2- 172.21.2.112:0/2320724945 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f916c1b0a10 0x7f916c0cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:35.295 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.274 7f9170bb8700 1 --2- 172.21.2.112:0/2320724945 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f916c0d1870 0x7f916c1b03f0 unknown :-1 s=CLOSED pgs=3359 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:35.296 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.274 7f9170bb8700 1 -- 172.21.2.112:0/2320724945 >> 172.21.2.112:0/2320724945 conn(0x7f916c0ce450 msgr2=0x7f916c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:35.296 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.274 7f9170bb8700 1 -- 172.21.2.112:0/2320724945 shutdown_connections 2021-02-10T11:50:35.296 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.274 7f9170bb8700 1 -- 172.21.2.112:0/2320724945 wait complete. 2021-02-10T11:50:35.297 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.288 7f9170bb8700 1 Processor -- start 2021-02-10T11:50:35.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.288 7f9170bb8700 1 -- start start 2021-02-10T11:50:35.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.289 7f9170bb8700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f916c1908b0 0x7f916c1a4440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:35.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.289 7f9170bb8700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f916c0bd920 0x7f916c0bdd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:35.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.289 7f9170bb8700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f916c08d510 0x7f916c08d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:35.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.289 7f9170bb8700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f916c08cf90 con 0x7f916c1908b0 2021-02-10T11:50:35.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.289 7f9170bb8700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f916c0be340 con 0x7f916c08d510 2021-02-10T11:50:35.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.289 7f9170bb8700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f916c0be680 con 0x7f916c0bd920 2021-02-10T11:50:35.298 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.289 7f916a85e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f916c1908b0 0x7f916c1a4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:35.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.289 7f916a85e700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f916c1908b0 0x7f916c1a4440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:34174/0 (socket says 172.21.2.112:34174) 2021-02-10T11:50:35.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.289 7f916a85e700 1 -- 172.21.2.112:0/1129490077 learned_addr learned my addr 172.21.2.112:0/1129490077 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:50:35.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.289 7f916a85e700 1 -- 172.21.2.112:0/1129490077 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f916c0bd920 msgr2=0x7f916c0bdd20 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:50:35.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.289 7f916a85e700 1 --2- 172.21.2.112:0/1129490077 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f916c0bd920 0x7f916c0bdd20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:35.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.289 7f916a85e700 1 -- 172.21.2.112:0/1129490077 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f916c08d510 msgr2=0x7f916c08d910 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:50:35.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.289 7f916a85e700 1 --2- 172.21.2.112:0/1129490077 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f916c08d510 0x7f916c08d910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:35.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.289 7f916a85e700 1 -- 172.21.2.112:0/1129490077 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f916c1a4a60 con 0x7f916c1908b0 2021-02-10T11:50:35.299 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.289 7f916a85e700 1 --2- 172.21.2.112:0/1129490077 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f916c1908b0 0x7f916c1a4440 secure :-1 s=READY pgs=3360 cs=0 l=1 rev1=1 rx=0x7f915c00b5e0 tx=0x7f915c00b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:35.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.289 7f91637fe700 1 -- 172.21.2.112:0/1129490077 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f915c00bd70 con 0x7f916c1908b0 2021-02-10T11:50:35.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.290 7f91637fe700 1 -- 172.21.2.112:0/1129490077 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f915c002030 con 0x7f916c1908b0 2021-02-10T11:50:35.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.290 7f9170bb8700 1 -- 172.21.2.112:0/1129490077 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f916c1a4a60 con 0x7f916c1908b0 2021-02-10T11:50:35.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.290 7f9170bb8700 1 -- 172.21.2.112:0/1129490077 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f916c1a4a60 con 0x7f916c1908b0 2021-02-10T11:50:35.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.290 7f91637fe700 1 -- 172.21.2.112:0/1129490077 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f915c002550 con 0x7f916c1908b0 2021-02-10T11:50:35.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.290 7f91637fe700 1 -- 172.21.2.112:0/1129490077 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f915c000c90 con 0x7f916c1908b0 2021-02-10T11:50:35.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.290 7f91637fe700 1 --2- 172.21.2.112:0/1129490077 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9150018730 0x7f915001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:35.300 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.290 7f91637fe700 1 -- 172.21.2.112:0/1129490077 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f915c032860 con 0x7f916c1908b0 2021-02-10T11:50:35.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.291 7f916a05d700 1 --2- 172.21.2.112:0/1129490077 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9150018730 0x7f915001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:35.301 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.291 7f916a05d700 1 --2- 172.21.2.112:0/1129490077 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9150018730 0x7f915001ac10 crc :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:35.326 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.322 7f9170bb8700 1 -- 172.21.2.112:0/1129490077 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f916c0be680 con 0x7f916c1908b0 2021-02-10T11:50:35.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.325 7f91637fe700 1 -- 172.21.2.112:0/1129490077 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f915c01f5a0 con 0x7f916c1908b0 2021-02-10T11:50:35.457 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.452 7f9170bb8700 1 -- 172.21.2.112:0/1129490077 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mgr", ""]}) v1 -- 0x7f916c190d00 con 0x7f9150018730 2021-02-10T11:50:35.457 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.453 7f91637fe700 1 -- 172.21.2.112:0/1129490077 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (crc 0 0 0) 0x7f9154006970 con 0x7f9150018730 2021-02-10T11:50:35.458 INFO:teuthology.orchestra.run.gibba012.stderr:instructed osd(s) 5 to deep-scrub 2021-02-10T11:50:35.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.455 7f9170bb8700 1 -- 172.21.2.112:0/1129490077 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9150018730 msgr2=0x7f915001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:35.459 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.455 7f9170bb8700 1 --2- 172.21.2.112:0/1129490077 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9150018730 0x7f915001ac10 crc :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:35.460 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.455 7f9170bb8700 1 -- 172.21.2.112:0/1129490077 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f916c1908b0 msgr2=0x7f916c1a4440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:35.460 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.455 7f9170bb8700 1 --2- 172.21.2.112:0/1129490077 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f916c1908b0 0x7f916c1a4440 secure :-1 s=READY pgs=3360 cs=0 l=1 rev1=1 rx=0x7f915c00b5e0 tx=0x7f915c00b980).stop 2021-02-10T11:50:35.460 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.455 7f9170bb8700 1 -- 172.21.2.112:0/1129490077 shutdown_connections 2021-02-10T11:50:35.460 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.455 7f9170bb8700 1 --2- 172.21.2.112:0/1129490077 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f9150018730 0x7f915001ac10 unknown :-1 s=CLOSED pgs=1510 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:35.460 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.455 7f9170bb8700 1 --2- 172.21.2.112:0/1129490077 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f916c08d510 0x7f916c08d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:35.460 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.455 7f9170bb8700 1 --2- 172.21.2.112:0/1129490077 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f916c0bd920 0x7f916c0bdd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:35.460 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.455 7f9170bb8700 1 --2- 172.21.2.112:0/1129490077 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f916c1908b0 0x7f916c1a4440 unknown :-1 s=CLOSED pgs=3360 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:35.460 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.455 7f9170bb8700 1 -- 172.21.2.112:0/1129490077 >> 172.21.2.112:0/1129490077 conn(0x7f916c08f030 msgr2=0x7f916c0b5e80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:35.461 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.455 7f9170bb8700 1 -- 172.21.2.112:0/1129490077 shutdown_connections 2021-02-10T11:50:35.461 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.455 7f9170bb8700 1 -- 172.21.2.112:0/1129490077 wait complete. 2021-02-10T11:50:35.468 INFO:tasks.ceph:Scrubbing osd.6 2021-02-10T11:50:35.468 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2021-02-10T11:50:35.902 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.881 7f82f47d6700 1 Processor -- start 2021-02-10T11:50:35.903 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.881 7f82f47d6700 1 -- start start 2021-02-10T11:50:35.903 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.882 7f82f47d6700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f82ec0d1880 0x7f82ec1b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:35.903 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.882 7f82f47d6700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82ec0fe1d0 con 0x7f82ec0cff80 2021-02-10T11:50:35.903 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.882 7f82f47d6700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82ec109a70 con 0x7f82ec1b0af0 2021-02-10T11:50:35.903 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.882 7f82f47d6700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82ec088070 con 0x7f82ec0d1880 2021-02-10T11:50:35.903 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.882 7f82f25f2700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f82ec0d1880 0x7f82ec1b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:35.903 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.882 7f82f2df3700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f82ec0cff80 0x7f82ec109140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:33640/0 (socket says 172.21.2.112:33640) 2021-02-10T11:50:35.904 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.882 7f82f2df3700 1 -- 172.21.2.112:0/4139780303 learned_addr learned my addr 172.21.2.112:0/4139780303 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:50:35.904 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.883 7f82f25f2700 1 -- 172.21.2.112:0/4139780303 >> v1:172.21.2.120:6789/0 conn(0x7f82ec1b0af0 legacy=0x7f82ec108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:35.904 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.883 7f82f25f2700 1 -- 172.21.2.112:0/4139780303 >> v1:172.21.2.112:6789/0 conn(0x7f82ec0cff80 legacy=0x7f82ec109140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:35.904 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.883 7f82f25f2700 1 -- 172.21.2.112:0/4139780303 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82ec088260 con 0x7f82ec0d1880 2021-02-10T11:50:35.904 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.883 7f82f25f2700 1 --2- 172.21.2.112:0/4139780303 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f82ec0d1880 0x7f82ec1b04d0 secure :-1 s=READY pgs=3362 cs=0 l=1 rev1=1 rx=0x7f82e8006650 tx=0x7f82e8009a80).ready entity=mon.0 client_cookie=87e6fcd5fc28eece server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:35.904 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.883 7f82f15f0700 1 -- 172.21.2.112:0/4139780303 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f82e800c0c0 con 0x7f82ec0d1880 2021-02-10T11:50:35.904 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.883 7f82f15f0700 1 -- 172.21.2.112:0/4139780303 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82e800c340 con 0x7f82ec0d1880 2021-02-10T11:50:35.904 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.883 7f82f15f0700 1 -- 172.21.2.112:0/4139780303 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f82e800c860 con 0x7f82ec0d1880 2021-02-10T11:50:35.905 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.883 7f82f47d6700 1 -- 172.21.2.112:0/4139780303 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f82ec0d1880 msgr2=0x7f82ec1b04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:35.905 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.883 7f82f47d6700 1 --2- 172.21.2.112:0/4139780303 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f82ec0d1880 0x7f82ec1b04d0 secure :-1 s=READY pgs=3362 cs=0 l=1 rev1=1 rx=0x7f82e8006650 tx=0x7f82e8009a80).stop 2021-02-10T11:50:35.905 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.884 7f82f47d6700 1 -- 172.21.2.112:0/4139780303 shutdown_connections 2021-02-10T11:50:35.905 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.884 7f82f47d6700 1 --2- 172.21.2.112:0/4139780303 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f82ec0d1880 0x7f82ec1b04d0 unknown :-1 s=CLOSED pgs=3362 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:35.905 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.884 7f82f47d6700 1 -- 172.21.2.112:0/4139780303 >> 172.21.2.112:0/4139780303 conn(0x7f82ec0ce430 msgr2=0x7f82ec0d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:35.905 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.884 7f82f47d6700 1 -- 172.21.2.112:0/4139780303 shutdown_connections 2021-02-10T11:50:35.905 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.884 7f82f47d6700 1 -- 172.21.2.112:0/4139780303 wait complete. 2021-02-10T11:50:35.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.898 7f82f47d6700 1 Processor -- start 2021-02-10T11:50:35.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.898 7f82f47d6700 1 -- start start 2021-02-10T11:50:35.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.899 7f82f47d6700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f82ec09fbb0 0x7f82ec1b5a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:35.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.899 7f82f47d6700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f82ec09ffb0 0x7f82ec08a930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:35.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.899 7f82f47d6700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f82ec08af50 0x7f82ec09c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:35.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.899 7f82f47d6700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82ec09f6b0 con 0x7f82ec09fbb0 2021-02-10T11:50:35.908 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.899 7f82f47d6700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82ec08b390 con 0x7f82ec09ffb0 2021-02-10T11:50:35.909 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.899 7f82f47d6700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82ec1b6060 con 0x7f82ec08af50 2021-02-10T11:50:35.909 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.899 7f82f25f2700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f82ec09fbb0 0x7f82ec1b5a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:35.909 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.899 7f82f2df3700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f82ec08af50 0x7f82ec09c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:35.909 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.899 7f82f25f2700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f82ec09fbb0 0x7f82ec1b5a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:34180/0 (socket says 172.21.2.112:34180) 2021-02-10T11:50:35.909 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.899 7f82f25f2700 1 -- 172.21.2.112:0/3002507090 learned_addr learned my addr 172.21.2.112:0/3002507090 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:50:35.909 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.899 7f82f1df1700 1 --2- 172.21.2.112:0/3002507090 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f82ec09ffb0 0x7f82ec08a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:35.909 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.899 7f82f2df3700 1 -- 172.21.2.112:0/3002507090 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f82ec09ffb0 msgr2=0x7f82ec08a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:35.909 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.899 7f82f2df3700 1 --2- 172.21.2.112:0/3002507090 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f82ec09ffb0 0x7f82ec08a930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:35.910 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.899 7f82f2df3700 1 -- 172.21.2.112:0/3002507090 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f82ec09fbb0 msgr2=0x7f82ec1b5a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:35.910 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.899 7f82f2df3700 1 --2- 172.21.2.112:0/3002507090 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f82ec09fbb0 0x7f82ec1b5a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:35.910 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.899 7f82f2df3700 1 -- 172.21.2.112:0/3002507090 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82ec09ca30 con 0x7f82ec08af50 2021-02-10T11:50:35.910 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.899 7f82f25f2700 1 --2- 172.21.2.112:0/3002507090 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f82ec09fbb0 0x7f82ec1b5a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:50:35.910 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.900 7f82f2df3700 1 --2- 172.21.2.112:0/3002507090 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f82ec08af50 0x7f82ec09c410 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7f82e4007f70 tx=0x7f82e400ac20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:35.910 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.900 7f82deffd700 1 -- 172.21.2.112:0/3002507090 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f82e4007020 con 0x7f82ec08af50 2021-02-10T11:50:35.910 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.900 7f82deffd700 1 -- 172.21.2.112:0/3002507090 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82e4007800 con 0x7f82ec08af50 2021-02-10T11:50:35.910 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.900 7f82f47d6700 1 -- 172.21.2.112:0/3002507090 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82ec09ca30 con 0x7f82ec08af50 2021-02-10T11:50:35.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.900 7f82deffd700 1 -- 172.21.2.112:0/3002507090 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f82e4004720 con 0x7f82ec08af50 2021-02-10T11:50:35.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.900 7f82f47d6700 1 -- 172.21.2.112:0/3002507090 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82ec09ca30 con 0x7f82ec08af50 2021-02-10T11:50:35.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.900 7f82deffd700 1 -- 172.21.2.112:0/3002507090 <== mon.2 v2:172.21.2.120:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f82e40072a0 con 0x7f82ec08af50 2021-02-10T11:50:35.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.901 7f82deffd700 1 --2- 172.21.2.112:0/3002507090 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f82d40186d0 0x7f82d401abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:35.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.901 7f82deffd700 1 -- 172.21.2.112:0/3002507090 <== mon.2 v2:172.21.2.120:3301/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f82e4032a80 con 0x7f82ec08af50 2021-02-10T11:50:35.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.901 7f82f25f2700 1 --2- 172.21.2.112:0/3002507090 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f82d40186d0 0x7f82d401abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:35.911 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.901 7f82f25f2700 1 --2- 172.21.2.112:0/3002507090 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f82d40186d0 0x7f82d401abd0 crc :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:35.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.932 7f82f47d6700 1 --2- 172.21.2.112:0/3002507090 >> [v2:172.21.2.120:6824/36385,v1:172.21.2.120:6826/36385] conn(0x7f82ec0c4620 0x7f82ec0627c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:35.936 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.932 7f82f47d6700 1 -- 172.21.2.112:0/3002507090 --> [v2:172.21.2.120:6824/36385,v1:172.21.2.120:6826/36385] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f82ec09d2b0 con 0x7f82ec0c4620 2021-02-10T11:50:35.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.933 7f82f1df1700 1 --2- 172.21.2.112:0/3002507090 >> [v2:172.21.2.120:6824/36385,v1:172.21.2.120:6826/36385] conn(0x7f82ec0c4620 0x7f82ec0627c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:35.937 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.933 7f82f1df1700 1 --2- 172.21.2.112:0/3002507090 >> [v2:172.21.2.120:6824/36385,v1:172.21.2.120:6826/36385] conn(0x7f82ec0c4620 0x7f82ec0627c0 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:35.938 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.934 7f82deffd700 1 -- 172.21.2.112:0/3002507090 <== osd.6 v2:172.21.2.120:6824/36385 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+5151 (crc 0 0 0) 0x7f82e00054b0 con 0x7f82ec0c4620 2021-02-10T11:50:35.944 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.940 7f82f47d6700 1 -- 172.21.2.112:0/3002507090 --> [v2:172.21.2.120:6824/36385,v1:172.21.2.120:6826/36385] -- command(tid 2: {"prefix": "config set", "key": "osd_debug_deep_scrub_sleep", "value": "0"}) v1 -- 0x7f82ec0ce430 con 0x7f82ec0c4620 2021-02-10T11:50:35.945 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.941 7f82deffd700 1 -- 172.21.2.112:0/3002507090 <== osd.6 v2:172.21.2.120:6824/36385 2 ==== command_reply(tid 2: 0 Set osd_debug_deep_scrub_sleep to 0) v1 ==== 43+0+0 (crc 0 0 0) 0x7f82e0006a50 con 0x7f82ec0c4620 2021-02-10T11:50:35.945 INFO:teuthology.orchestra.run.gibba012.stderr:Set osd_debug_deep_scrub_sleep to 0 2021-02-10T11:50:35.946 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.942 7f82f47d6700 1 -- 172.21.2.112:0/3002507090 >> [v2:172.21.2.120:6824/36385,v1:172.21.2.120:6826/36385] conn(0x7f82ec0c4620 msgr2=0x7f82ec0627c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:35.946 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.942 7f82f47d6700 1 --2- 172.21.2.112:0/3002507090 >> [v2:172.21.2.120:6824/36385,v1:172.21.2.120:6826/36385] conn(0x7f82ec0c4620 0x7f82ec0627c0 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:35.947 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.942 7f82f47d6700 1 -- 172.21.2.112:0/3002507090 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f82d40186d0 msgr2=0x7f82d401abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:35.947 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.942 7f82f47d6700 1 --2- 172.21.2.112:0/3002507090 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f82d40186d0 0x7f82d401abd0 crc :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:35.947 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.942 7f82f47d6700 1 -- 172.21.2.112:0/3002507090 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f82ec08af50 msgr2=0x7f82ec09c410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:35.947 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.942 7f82f47d6700 1 --2- 172.21.2.112:0/3002507090 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f82ec08af50 0x7f82ec09c410 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7f82e4007f70 tx=0x7f82e400ac20).stop 2021-02-10T11:50:35.947 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.942 7f82f2df3700 1 -- 172.21.2.112:0/3002507090 reap_dead start 2021-02-10T11:50:35.947 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.942 7f82f47d6700 1 -- 172.21.2.112:0/3002507090 shutdown_connections 2021-02-10T11:50:35.947 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.942 7f82f47d6700 1 -- 172.21.2.112:0/3002507090 >> 172.21.2.112:0/3002507090 conn(0x7f82ec09bad0 msgr2=0x7f82ec0804e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:35.947 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.942 7f82f47d6700 1 -- 172.21.2.112:0/3002507090 shutdown_connections 2021-02-10T11:50:35.948 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:35.942 7f82f47d6700 1 -- 172.21.2.112:0/3002507090 wait complete. 2021-02-10T11:50:35.954 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd deep-scrub 6 2021-02-10T11:50:36.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.365 7fce8f967700 1 Processor -- start 2021-02-10T11:50:36.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.365 7fce8f967700 1 -- start start 2021-02-10T11:50:36.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.365 7fce8f967700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fce880d0330 0x7fce880ef630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:36.387 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.365 7fce8f967700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7fce880d1c60 0x7fce880efc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:36.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.365 7fce8f967700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce880f0b00 con 0x7fce880d1860 2021-02-10T11:50:36.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.365 7fce8f967700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce880e5ec0 con 0x7fce880d0330 2021-02-10T11:50:36.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.365 7fce8f967700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce881b0bd0 con 0x7fce880d1c60 2021-02-10T11:50:36.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.365 7fce8cf82700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fce880d0330 0x7fce880ef630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:36.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.366 7fce8cf82700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7fce880d0330 0x7fce880ef630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:34192/0 (socket says 172.21.2.112:34192) 2021-02-10T11:50:36.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.366 7fce8cf82700 1 -- 172.21.2.112:0/146194144 learned_addr learned my addr 172.21.2.112:0/146194144 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:50:36.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.366 7fce8df84700 1 --2- 172.21.2.112:0/146194144 >> v2:172.21.2.120:3300/0 conn(0x7fce880d1c60 0x7fce880efc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:36.388 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.366 7fce8cf82700 1 -- 172.21.2.112:0/146194144 >> v2:172.21.2.120:3300/0 conn(0x7fce880d1c60 msgr2=0x7fce880efc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:50:36.389 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.366 7fce8cf82700 1 --2- 172.21.2.112:0/146194144 >> v2:172.21.2.120:3300/0 conn(0x7fce880d1c60 0x7fce880efc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:36.389 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.366 7fce8cf82700 1 -- 172.21.2.112:0/146194144 >> v1:172.21.2.120:6789/0 conn(0x7fce880d1860 legacy=0x7fce881b03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:36.389 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.366 7fce8cf82700 1 -- 172.21.2.112:0/146194144 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce880f2090 con 0x7fce880d0330 2021-02-10T11:50:36.389 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.366 7fce8cf82700 1 --2- 172.21.2.112:0/146194144 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fce880d0330 0x7fce880ef630 secure :-1 s=READY pgs=3363 cs=0 l=1 rev1=1 rx=0x7fce78006700 tx=0x7fce78009b80).ready entity=mon.0 client_cookie=ede4fb4ef0cdd7fe server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:36.389 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.366 7fce7ffff700 1 -- 172.21.2.112:0/146194144 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fce7800c0c0 con 0x7fce880d0330 2021-02-10T11:50:36.389 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.366 7fce7ffff700 1 -- 172.21.2.112:0/146194144 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce7800c440 con 0x7fce880d0330 2021-02-10T11:50:36.389 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.366 7fce7ffff700 1 -- 172.21.2.112:0/146194144 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fce7800c920 con 0x7fce880d0330 2021-02-10T11:50:36.389 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.366 7fce8f967700 1 -- 172.21.2.112:0/146194144 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fce880d0330 msgr2=0x7fce880ef630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:36.390 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.366 7fce8f967700 1 --2- 172.21.2.112:0/146194144 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fce880d0330 0x7fce880ef630 secure :-1 s=READY pgs=3363 cs=0 l=1 rev1=1 rx=0x7fce78006700 tx=0x7fce78009b80).stop 2021-02-10T11:50:36.390 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.367 7fce8f967700 1 -- 172.21.2.112:0/146194144 shutdown_connections 2021-02-10T11:50:36.390 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.367 7fce8f967700 1 --2- 172.21.2.112:0/146194144 >> v2:172.21.2.120:3300/0 conn(0x7fce880d1c60 0x7fce880efc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:36.390 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.367 7fce8f967700 1 --2- 172.21.2.112:0/146194144 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fce880d0330 0x7fce880ef630 unknown :-1 s=CLOSED pgs=3363 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:36.390 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.367 7fce8f967700 1 -- 172.21.2.112:0/146194144 >> 172.21.2.112:0/146194144 conn(0x7fce880ce440 msgr2=0x7fce880d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:36.390 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.367 7fce8f967700 1 -- 172.21.2.112:0/146194144 shutdown_connections 2021-02-10T11:50:36.390 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.367 7fce8f967700 1 -- 172.21.2.112:0/146194144 wait complete. 2021-02-10T11:50:36.393 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.383 7fce8f967700 1 Processor -- start 2021-02-10T11:50:36.393 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.383 7fce8f967700 1 -- start start 2021-02-10T11:50:36.393 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.383 7fce8f967700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fce881a35c0 0x7fce881a39c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:36.393 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.383 7fce8f967700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fce8809d0e0 0x7fce8809d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:36.393 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.383 7fce8f967700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fce8809db00 0x7fce8809df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:36.394 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.383 7fce8f967700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce8809cb60 con 0x7fce881a35c0 2021-02-10T11:50:36.394 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.383 7fce8f967700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce880c3730 con 0x7fce8809db00 2021-02-10T11:50:36.394 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.383 7fce8f967700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce880c3910 con 0x7fce8809d0e0 2021-02-10T11:50:36.394 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.384 7fce8d783700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fce881a35c0 0x7fce881a39c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:36.394 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.384 7fce8d783700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fce881a35c0 0x7fce881a39c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:34194/0 (socket says 172.21.2.112:34194) 2021-02-10T11:50:36.395 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.384 7fce8d783700 1 -- 172.21.2.112:0/1126444865 learned_addr learned my addr 172.21.2.112:0/1126444865 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:50:36.395 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.384 7fce8cf82700 1 --2- 172.21.2.112:0/1126444865 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fce8809d0e0 0x7fce8809d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:36.395 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.384 7fce8df84700 1 --2- 172.21.2.112:0/1126444865 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fce8809db00 0x7fce8809df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:36.395 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.384 7fce8cf82700 1 -- 172.21.2.112:0/1126444865 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fce8809db00 msgr2=0x7fce8809df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:36.395 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.384 7fce8cf82700 1 --2- 172.21.2.112:0/1126444865 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fce8809db00 0x7fce8809df00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:36.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.384 7fce8cf82700 1 -- 172.21.2.112:0/1126444865 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fce881a35c0 msgr2=0x7fce881a39c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:36.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.384 7fce8cf82700 1 --2- 172.21.2.112:0/1126444865 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fce881a35c0 0x7fce881a39c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:36.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.384 7fce8cf82700 1 -- 172.21.2.112:0/1126444865 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce88093e60 con 0x7fce8809d0e0 2021-02-10T11:50:36.396 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.384 7fce8cf82700 1 --2- 172.21.2.112:0/1126444865 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fce8809d0e0 0x7fce8809d4e0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 rx=0x7fce780096d0 tx=0x7fce7800b8c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:36.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.384 7fce7dffb700 1 -- 172.21.2.112:0/1126444865 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fce780009b0 con 0x7fce8809d0e0 2021-02-10T11:50:36.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.384 7fce7dffb700 1 -- 172.21.2.112:0/1126444865 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce7800bcb0 con 0x7fce8809d0e0 2021-02-10T11:50:36.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.384 7fce8f967700 1 -- 172.21.2.112:0/1126444865 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce88093e60 con 0x7fce8809d0e0 2021-02-10T11:50:36.397 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.384 7fce7dffb700 1 -- 172.21.2.112:0/1126444865 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fce780009b0 con 0x7fce8809d0e0 2021-02-10T11:50:36.398 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.384 7fce8f967700 1 -- 172.21.2.112:0/1126444865 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce88093e60 con 0x7fce8809d0e0 2021-02-10T11:50:36.398 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.385 7fce7dffb700 1 -- 172.21.2.112:0/1126444865 <== mon.2 v2:172.21.2.120:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fce780064f0 con 0x7fce8809d0e0 2021-02-10T11:50:36.398 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.385 7fce7dffb700 1 --2- 172.21.2.112:0/1126444865 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fce68018730 0x7fce6801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:36.398 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.385 7fce7dffb700 1 -- 172.21.2.112:0/1126444865 <== mon.2 v2:172.21.2.120:3301/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fce78032560 con 0x7fce8809d0e0 2021-02-10T11:50:36.399 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.385 7fce8d783700 1 --2- 172.21.2.112:0/1126444865 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fce68018730 0x7fce6801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:36.399 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.386 7fce8d783700 1 --2- 172.21.2.112:0/1126444865 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fce68018730 0x7fce6801ac10 crc :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:36.418 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.413 7fce8f967700 1 -- 172.21.2.112:0/1126444865 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce880ebbe0 con 0x7fce8809d0e0 2021-02-10T11:50:36.420 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.416 7fce7dffb700 1 -- 172.21.2.112:0/1126444865 <== mon.2 v2:172.21.2.120:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fce78000940 con 0x7fce8809d0e0 2021-02-10T11:50:36.553 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.548 7fce8f967700 1 -- 172.21.2.112:0/1126444865 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mgr", ""]}) v1 -- 0x7fce8809cb60 con 0x7fce68018730 2021-02-10T11:50:36.554 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.549 7fce7dffb700 1 -- 172.21.2.112:0/1126444865 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) v1 ==== 41+0+0 (crc 0 0 0) 0x7fce84000960 con 0x7fce68018730 2021-02-10T11:50:36.554 INFO:teuthology.orchestra.run.gibba012.stderr:instructed osd(s) 6 to deep-scrub 2021-02-10T11:50:36.555 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.551 7fce8f967700 1 -- 172.21.2.112:0/1126444865 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fce68018730 msgr2=0x7fce6801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:36.555 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.551 7fce8f967700 1 --2- 172.21.2.112:0/1126444865 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fce68018730 0x7fce6801ac10 crc :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:36.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.551 7fce8f967700 1 -- 172.21.2.112:0/1126444865 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fce8809d0e0 msgr2=0x7fce8809d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:36.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.551 7fce8f967700 1 --2- 172.21.2.112:0/1126444865 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fce8809d0e0 0x7fce8809d4e0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 rx=0x7fce780096d0 tx=0x7fce7800b8c0).stop 2021-02-10T11:50:36.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.551 7fce8f967700 1 -- 172.21.2.112:0/1126444865 shutdown_connections 2021-02-10T11:50:36.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.551 7fce8f967700 1 --2- 172.21.2.112:0/1126444865 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7fce68018730 0x7fce6801ac10 unknown :-1 s=CLOSED pgs=1512 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:36.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.551 7fce8f967700 1 --2- 172.21.2.112:0/1126444865 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7fce8809db00 0x7fce8809df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:36.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.551 7fce8f967700 1 --2- 172.21.2.112:0/1126444865 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7fce8809d0e0 0x7fce8809d4e0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:36.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.551 7fce8f967700 1 --2- 172.21.2.112:0/1126444865 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7fce881a35c0 0x7fce881a39c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:36.556 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.551 7fce8f967700 1 -- 172.21.2.112:0/1126444865 >> 172.21.2.112:0/1126444865 conn(0x7fce880a6b00 msgr2=0x7fce88095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:36.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.551 7fce8f967700 1 -- 172.21.2.112:0/1126444865 shutdown_connections 2021-02-10T11:50:36.557 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.551 7fce8f967700 1 -- 172.21.2.112:0/1126444865 wait complete. 2021-02-10T11:50:36.564 INFO:tasks.ceph:Scrubbing osd.7 2021-02-10T11:50:36.564 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2021-02-10T11:50:36.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.977 7f68efefd700 1 Processor -- start 2021-02-10T11:50:36.998 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.977 7f68efefd700 1 -- start start 2021-02-10T11:50:36.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.978 7f68efefd700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f68e80d1880 0x7f68e81b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:36.999 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.978 7f68efefd700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f68e80cff80 0x7f68e8109140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:37.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.978 7f68efefd700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68e8109a70 con 0x7f68e81b0af0 2021-02-10T11:50:37.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.978 7f68efefd700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68e8109760 con 0x7f68e80d1880 2021-02-10T11:50:37.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.978 7f68efefd700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68e8088070 con 0x7f68e80cff80 2021-02-10T11:50:37.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.978 7f68edd19700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f68e80d1880 0x7f68e81b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:37.003 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.978 7f68edd19700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f68e80d1880 0x7f68e81b04d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:34200/0 (socket says 172.21.2.112:34200) 2021-02-10T11:50:37.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.978 7f68edd19700 1 -- 172.21.2.112:0/971935958 learned_addr learned my addr 172.21.2.112:0/971935958 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:50:37.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.978 7f68ee51a700 1 --2- 172.21.2.112:0/971935958 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f68e80cff80 0x7f68e8109140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:37.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.978 7f68edd19700 1 -- 172.21.2.112:0/971935958 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f68e80cff80 msgr2=0x7f68e8109140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:50:37.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.978 7f68edd19700 1 --2- 172.21.2.112:0/971935958 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f68e80cff80 0x7f68e8109140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:37.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.978 7f68edd19700 1 -- 172.21.2.112:0/971935958 >> v1:172.21.2.120:6789/0 conn(0x7f68e81b0af0 legacy=0x7f68e8108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:37.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.978 7f68edd19700 1 -- 172.21.2.112:0/971935958 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68e80d1c80 con 0x7f68e80d1880 2021-02-10T11:50:37.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.978 7f68ee51a700 1 --2- 172.21.2.112:0/971935958 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f68e80cff80 0x7f68e8109140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:50:37.004 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.978 7f68edd19700 1 --2- 172.21.2.112:0/971935958 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f68e80d1880 0x7f68e81b04d0 secure :-1 s=READY pgs=3364 cs=0 l=1 rev1=1 rx=0x7f68d8006700 tx=0x7f68d8009b80).ready entity=mon.0 client_cookie=2d7f3fbcc46693f4 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:37.005 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.979 7f68ecd17700 1 -- 172.21.2.112:0/971935958 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f68d800c0c0 con 0x7f68e80d1880 2021-02-10T11:50:37.005 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.979 7f68ecd17700 1 -- 172.21.2.112:0/971935958 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68d800c440 con 0x7f68e80d1880 2021-02-10T11:50:37.005 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.979 7f68ecd17700 1 -- 172.21.2.112:0/971935958 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f68d800c8a0 con 0x7f68e80d1880 2021-02-10T11:50:37.005 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.979 7f68efefd700 1 -- 172.21.2.112:0/971935958 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f68e80d1880 msgr2=0x7f68e81b04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:37.005 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.979 7f68efefd700 1 --2- 172.21.2.112:0/971935958 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f68e80d1880 0x7f68e81b04d0 secure :-1 s=READY pgs=3364 cs=0 l=1 rev1=1 rx=0x7f68d8006700 tx=0x7f68d8009b80).stop 2021-02-10T11:50:37.005 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.979 7f68efefd700 1 -- 172.21.2.112:0/971935958 shutdown_connections 2021-02-10T11:50:37.005 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.979 7f68efefd700 1 --2- 172.21.2.112:0/971935958 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f68e80cff80 0x7f68e8109140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:37.005 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.979 7f68efefd700 1 --2- 172.21.2.112:0/971935958 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f68e80d1880 0x7f68e81b04d0 unknown :-1 s=CLOSED pgs=3364 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:37.006 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.979 7f68efefd700 1 -- 172.21.2.112:0/971935958 >> 172.21.2.112:0/971935958 conn(0x7f68e80ce430 msgr2=0x7f68e80d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:37.006 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.980 7f68efefd700 1 -- 172.21.2.112:0/971935958 shutdown_connections 2021-02-10T11:50:37.006 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.980 7f68efefd700 1 -- 172.21.2.112:0/971935958 wait complete. 2021-02-10T11:50:37.006 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.994 7f68efefd700 1 Processor -- start 2021-02-10T11:50:37.006 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.994 7f68efefd700 1 -- start start 2021-02-10T11:50:37.008 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.994 7f68efefd700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f68e80ad520 0x7f68e80ad920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:37.008 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.994 7f68efefd700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f68e80adf40 0x7f68e8097c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:37.008 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.994 7f68efefd700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f68e8098250 0x7f68e8098650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:37.008 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.994 7f68efefd700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68e8098c70 con 0x7f68e8098250 2021-02-10T11:50:37.008 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.994 7f68efefd700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68e80910f0 con 0x7f68e80adf40 2021-02-10T11:50:37.008 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.994 7f68efefd700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68e8091310 con 0x7f68e80ad520 2021-02-10T11:50:37.009 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.995 7f68ee51a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f68e8098250 0x7f68e8098650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:37.009 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.995 7f68ee51a700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f68e8098250 0x7f68e8098650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:34206/0 (socket says 172.21.2.112:34206) 2021-02-10T11:50:37.009 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.995 7f68ee51a700 1 -- 172.21.2.112:0/400114091 learned_addr learned my addr 172.21.2.112:0/400114091 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:50:37.009 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.995 7f68edd19700 1 --2- 172.21.2.112:0/400114091 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f68e80ad520 0x7f68e80ad920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:37.009 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.995 7f68ed518700 1 --2- 172.21.2.112:0/400114091 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f68e80adf40 0x7f68e8097c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:37.009 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.995 7f68edd19700 1 -- 172.21.2.112:0/400114091 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f68e80adf40 msgr2=0x7f68e8097c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:37.009 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.995 7f68edd19700 1 --2- 172.21.2.112:0/400114091 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f68e80adf40 0x7f68e8097c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:37.009 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.995 7f68edd19700 1 -- 172.21.2.112:0/400114091 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f68e8098250 msgr2=0x7f68e8098650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:37.010 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.995 7f68edd19700 1 --2- 172.21.2.112:0/400114091 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f68e8098250 0x7f68e8098650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:37.010 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.995 7f68edd19700 1 -- 172.21.2.112:0/400114091 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68e819a3b0 con 0x7f68e80ad520 2021-02-10T11:50:37.010 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.995 7f68edd19700 1 --2- 172.21.2.112:0/400114091 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f68e80ad520 0x7f68e80ad920 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7f68d80096d0 tx=0x7f68d800b8c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:37.010 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.995 7f68de7fc700 1 -- 172.21.2.112:0/400114091 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f68d80009b0 con 0x7f68e80ad520 2021-02-10T11:50:37.010 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.995 7f68de7fc700 1 -- 172.21.2.112:0/400114091 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68d800bcb0 con 0x7f68e80ad520 2021-02-10T11:50:37.010 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.995 7f68de7fc700 1 -- 172.21.2.112:0/400114091 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f68d80022b0 con 0x7f68e80ad520 2021-02-10T11:50:37.010 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.995 7f68efefd700 1 -- 172.21.2.112:0/400114091 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68e819a3b0 con 0x7f68e80ad520 2021-02-10T11:50:37.010 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.996 7f68efefd700 1 -- 172.21.2.112:0/400114091 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68e819a3b0 con 0x7f68e80ad520 2021-02-10T11:50:37.011 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.996 7f68de7fc700 1 -- 172.21.2.112:0/400114091 <== mon.2 v2:172.21.2.120:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f68d80064f0 con 0x7f68e80ad520 2021-02-10T11:50:37.011 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.996 7f68de7fc700 1 --2- 172.21.2.112:0/400114091 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f68d00186d0 0x7f68d001abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:37.011 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.996 7f68de7fc700 1 -- 172.21.2.112:0/400114091 <== mon.2 v2:172.21.2.120:3301/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f68d8000940 con 0x7f68e80ad520 2021-02-10T11:50:37.011 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.996 7f68ed518700 1 --2- 172.21.2.112:0/400114091 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f68d00186d0 0x7f68d001abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:37.011 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:36.997 7f68ed518700 1 --2- 172.21.2.112:0/400114091 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f68d00186d0 0x7f68d001abd0 crc :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:37.032 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:37.027 7f68efefd700 1 --2- 172.21.2.112:0/400114091 >> [v2:172.21.2.120:6802/36386,v1:172.21.2.120:6803/36386] conn(0x7f68e819afa0 0x7f68e819b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:37.032 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:37.028 7f68efefd700 1 -- 172.21.2.112:0/400114091 --> [v2:172.21.2.120:6802/36386,v1:172.21.2.120:6803/36386] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f68e8082a70 con 0x7f68e819afa0 2021-02-10T11:50:37.032 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:37.028 7f68ee51a700 1 --2- 172.21.2.112:0/400114091 >> [v2:172.21.2.120:6802/36386,v1:172.21.2.120:6803/36386] conn(0x7f68e819afa0 0x7f68e819b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:37.033 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:37.029 7f68ee51a700 1 --2- 172.21.2.112:0/400114091 >> [v2:172.21.2.120:6802/36386,v1:172.21.2.120:6803/36386] conn(0x7f68e819afa0 0x7f68e819b3a0 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:37.034 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:37.029 7f68de7fc700 1 -- 172.21.2.112:0/400114091 <== osd.7 v2:172.21.2.120:6802/36386 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+5151 (crc 0 0 0) 0x7f68e4000bc0 con 0x7f68e819afa0 2021-02-10T11:50:37.040 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:37.036 7f68efefd700 1 -- 172.21.2.112:0/400114091 --> [v2:172.21.2.120:6802/36386,v1:172.21.2.120:6803/36386] -- command(tid 2: {"prefix": "config set", "key": "osd_debug_deep_scrub_sleep", "value": "0"}) v1 -- 0x7f68e8082a70 con 0x7f68e819afa0 2021-02-10T11:50:37.041 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:37.037 7f68de7fc700 1 -- 172.21.2.112:0/400114091 <== osd.7 v2:172.21.2.120:6802/36386 2 ==== command_reply(tid 2: 0 Set osd_debug_deep_scrub_sleep to 0) v1 ==== 43+0+0 (crc 0 0 0) 0x7f68e40090c0 con 0x7f68e819afa0 2021-02-10T11:50:37.041 INFO:teuthology.orchestra.run.gibba012.stderr:Set osd_debug_deep_scrub_sleep to 0 2021-02-10T11:50:37.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:37.038 7f68efefd700 1 -- 172.21.2.112:0/400114091 >> [v2:172.21.2.120:6802/36386,v1:172.21.2.120:6803/36386] conn(0x7f68e819afa0 msgr2=0x7f68e819b3a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:37.043 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:37.038 7f68efefd700 1 --2- 172.21.2.112:0/400114091 >> [v2:172.21.2.120:6802/36386,v1:172.21.2.120:6803/36386] conn(0x7f68e819afa0 0x7f68e819b3a0 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:37.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:37.038 7f68efefd700 1 -- 172.21.2.112:0/400114091 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f68d00186d0 msgr2=0x7f68d001abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:37.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:37.038 7f68efefd700 1 --2- 172.21.2.112:0/400114091 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f68d00186d0 0x7f68d001abd0 crc :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:37.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:37.038 7f68efefd700 1 -- 172.21.2.112:0/400114091 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f68e80ad520 msgr2=0x7f68e80ad920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:37.044 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:37.038 7f68efefd700 1 --2- 172.21.2.112:0/400114091 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f68e80ad520 0x7f68e80ad920 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7f68d80096d0 tx=0x7f68d800b8c0).stop 2021-02-10T11:50:37.045 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:37.038 7f68ee51a700 1 -- 172.21.2.112:0/400114091 reap_dead start 2021-02-10T11:50:37.045 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:37.038 7f68efefd700 1 -- 172.21.2.112:0/400114091 shutdown_connections 2021-02-10T11:50:37.045 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:37.038 7f68efefd700 1 -- 172.21.2.112:0/400114091 >> 172.21.2.112:0/400114091 conn(0x7f68e818e810 msgr2=0x7f68e818c290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:37.045 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:37.038 7f68efefd700 1 -- 172.21.2.112:0/400114091 shutdown_connections 2021-02-10T11:50:37.045 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:37.038 7f68efefd700 1 -- 172.21.2.112:0/400114091 wait complete. 2021-02-10T11:50:37.051 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd deep-scrub 7 2021-02-10T11:50:37.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:37.458 7f700fc64700 1 Processor -- start 2021-02-10T11:50:37.480 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:37.459 7f700fc64700 1 -- start start 2021-02-10T11:50:37.481 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:37.459 7f700fc64700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f70080d1870 0x7f70081b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:37.481 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:37.459 7f700fc64700 1 -- --> v1:172.21.2.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70080f0b10 con 0x7f70080f0650 2021-02-10T11:50:37.481 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:37.459 7f700fc64700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70080d0310 con 0x7f70081b0a10 2021-02-10T11:50:37.481 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:37.459 7f700fc64700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70080e5ed0 con 0x7f70080d1870 2021-02-10T11:50:37.481 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:37.459 7f700e281700 1 --1- >> v1:172.21.2.112:6789/0 conn(0x7f70080f0650 0x7f70080cfb20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.112:6789/0 says I am v1:172.21.2.112:33676/0 (socket says 172.21.2.112:33676) 2021-02-10T11:50:37.481 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:37.459 7f700e281700 1 -- 172.21.2.112:0/2719233481 learned_addr learned my addr 172.21.2.112:0/2719233481 (peer_addr_for_me v1:172.21.2.112:0/0) 2021-02-10T11:50:37.481 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:37.459 7f700da80700 1 --2- 172.21.2.112:0/2719233481 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f70080d1870 0x7f70081b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:37.482 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:37.459 7f700ca7e700 1 -- 172.21.2.112:0/2719233481 <== mon.0 v1:172.21.2.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3446971192 0 0) 0x7f7000001500 con 0x7f70080f0650 2021-02-10T11:50:37.482 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:37.459 7f700ca7e700 1 -- 172.21.2.112:0/2719233481 --> v1:172.21.2.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ff4002fe0 con 0x7f70080f0650 2021-02-10T11:50:37.482 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:37.460 7f700ca7e700 1 -- 172.21.2.112:0/2719233481 <== mon.0 v1:172.21.2.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2107786219 0 0) 0x7f7000001270 con 0x7f70080f0650 2021-02-10T11:50:37.482 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:37.460 7f700ca7e700 1 -- 172.21.2.112:0/2719233481 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f70080d1870 msgr2=0x7f70081b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:50:37.482 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:37.460 7f700ca7e700 1 --2- 172.21.2.112:0/2719233481 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f70080d1870 0x7f70081b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:37.482 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:37.460 7f700ca7e700 1 -- 172.21.2.112:0/2719233481 >> v1:172.21.2.120:6789/0 conn(0x7f70081b0a10 legacy=0x7f70080cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:37.482 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:37.460 7f700ca7e700 1 -- 172.21.2.112:0/2719233481 --> v1:172.21.2.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70080d0530 con 0x7f70080f0650 2021-02-10T11:50:37.483 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:37.460 7f700ca7e700 1 -- 172.21.2.112:0/2719233481 <== mon.0 v1:172.21.2.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 2223367685 0 0) 0x7f7000001b50 con 0x7f70080f0650 2021-02-10T11:50:37.483 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:37.460 7f700ca7e700 1 -- 172.21.2.112:0/2719233481 >> v1:172.21.2.112:6789/0 conn(0x7f70080f0650 legacy=0x7f70080cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:37.483 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:37.460 7f700ca7e700 1 --2- 172.21.2.112:0/2719233481 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6ff4003b10 0x7f6ff4005fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:37.483 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:37.460 7f700ca7e700 1 --2- 172.21.2.112:0/2719233481 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6ff4006770 0x7f6ff4008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:37.483 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:37.460 7f700ca7e700 1 --2- 172.21.2.112:0/2719233481 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6ff4009510 0x7f6ff400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:37.483 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:37.460 7f700ca7e700 1 -- 172.21.2.112:0/2719233481 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff400c2c0 con 0x7f6ff4006770 2021-02-10T11:50:37.483 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:37.460 7f700ca7e700 1 -- 172.21.2.112:0/2719233481 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff400c4e0 con 0x7f6ff4009510 2021-02-10T11:50:37.483 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:37.460 7f700ca7e700 1 -- 172.21.2.112:0/2719233481 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff400c700 con 0x7f6ff4003b10 2021-02-10T11:50:37.483 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:37.460 7f700d27f700 1 --2- 172.21.2.112:0/2719233481 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6ff4006770 0x7f6ff4008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:37.484 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:37.460 7f700da80700 1 --2- 172.21.2.112:0/2719233481 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6ff4003b10 0x7f6ff4005fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:37.484 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:37.460 7f700e281700 1 --2- 172.21.2.112:0/2719233481 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6ff4009510 0x7f6ff400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:37.484 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:37.460 7f700d27f700 1 -- 172.21.2.112:0/2719233481 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6ff4003b10 msgr2=0x7f6ff4005fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:50:37.484 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:37.460 7f700d27f700 1 --2- 172.21.2.112:0/2719233481 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f6ff4003b10 0x7f6ff4005fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:37.484 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:37.460 7f700d27f700 1 -- 172.21.2.112:0/2719233481 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6ff4009510 msgr2=0x7f6ff400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:50:37.484 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:37.460 7f700d27f700 1 --2- 172.21.2.112:0/2719233481 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6ff4009510 0x7f6ff400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:37.484 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:37.460 7f700d27f700 1 -- 172.21.2.112:0/2719233481 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ff4050970 con 0x7f6ff4006770 2021-02-10T11:50:37.484 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:37.460 7f700e281700 1 --2- 172.21.2.112:0/2719233481 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f6ff4009510 0x7f6ff400ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:50:37.485 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:37.460 7f700d27f700 1 --2- 172.21.2.112:0/2719233481 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6ff4006770 0x7f6ff4008c90 secure :-1 s=READY pgs=3366 cs=0 l=1 rev1=1 rx=0x7f7004006ac0 tx=0x7f7004009a70).ready entity=mon.0 client_cookie=229e1b4b69abeb7c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:37.485 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:37.460 7f700ca7e700 1 -- 172.21.2.112:0/2719233481 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7004007500 con 0x7f6ff4006770 2021-02-10T11:50:37.485 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:37.460 7f700ca7e700 1 -- 172.21.2.112:0/2719233481 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7004007780 con 0x7f6ff4006770 2021-02-10T11:50:37.485 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:37.460 7f700ca7e700 1 -- 172.21.2.112:0/2719233481 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7004007ca0 con 0x7f6ff4006770 2021-02-10T11:50:37.485 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:37.460 7f700fc64700 1 -- 172.21.2.112:0/2719233481 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6ff4006770 msgr2=0x7f6ff4008c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:37.485 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:37.460 7f700fc64700 1 --2- 172.21.2.112:0/2719233481 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f6ff4006770 0x7f6ff4008c90 secure :-1 s=READY pgs=3366 cs=0 l=1 rev1=1 rx=0x7f7004006ac0 tx=0x7f7004009a70).stop 2021-02-10T11:50:37.485 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:37.460 7f700e281700 1 -- 172.21.2.112:0/2719233481 reap_dead start 2021-02-10T11:50:37.485 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:37.461 7f700fc64700 1 -- 172.21.2.112:0/2719233481 shutdown_connections 2021-02-10T11:50:37.486 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:37.461 7f700fc64700 1 -- 172.21.2.112:0/2719233481 >> 172.21.2.112:0/2719233481 conn(0x7f70080ce450 msgr2=0x7f70080d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:37.486 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:37.461 7f700fc64700 1 -- 172.21.2.112:0/2719233481 shutdown_connections 2021-02-10T11:50:37.486 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:37.461 7f700fc64700 1 -- 172.21.2.112:0/2719233481 wait complete. 2021-02-10T11:50:37.486 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:37.476 7f700fc64700 1 Processor -- start 2021-02-10T11:50:37.486 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:37.476 7f700fc64700 1 -- start start 2021-02-10T11:50:37.488 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:37.476 7f700fc64700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f70081a49b0 0x7f7008094a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:37.488 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:37.476 7f700fc64700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7008095080 0x7f70080c0370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:37.488 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:37.476 7f700fc64700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f70080c0990 0x7f70080c0d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:37.488 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:37.476 7f700fc64700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70080c13b0 con 0x7f7008095080 2021-02-10T11:50:37.488 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:37.476 7f700fc64700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70080c15d0 con 0x7f70080c0990 2021-02-10T11:50:37.488 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:37.476 7f700fc64700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70080c1d80 con 0x7f70081a49b0 2021-02-10T11:50:37.489 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:37.477 7f700d27f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7008095080 0x7f70080c0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:37.489 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:37.477 7f700d27f700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7008095080 0x7f70080c0370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:34222/0 (socket says 172.21.2.112:34222) 2021-02-10T11:50:37.489 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:37.477 7f700d27f700 1 -- 172.21.2.112:0/3247221781 learned_addr learned my addr 172.21.2.112:0/3247221781 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:50:37.489 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:37.477 7f700d27f700 1 -- 172.21.2.112:0/3247221781 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f70081a49b0 msgr2=0x7f7008094a60 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:50:37.489 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:37.477 7f700d27f700 1 --2- 172.21.2.112:0/3247221781 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f70081a49b0 0x7f7008094a60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:37.489 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:37.477 7f700d27f700 1 -- 172.21.2.112:0/3247221781 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f70080c0990 msgr2=0x7f70080c0d90 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:50:37.489 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:37.477 7f700d27f700 1 --2- 172.21.2.112:0/3247221781 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f70080c0990 0x7f70080c0d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:37.489 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:37.477 7f700d27f700 1 -- 172.21.2.112:0/3247221781 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70080c1fa0 con 0x7f7008095080 2021-02-10T11:50:37.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:37.477 7f700d27f700 1 --2- 172.21.2.112:0/3247221781 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7008095080 0x7f70080c0370 secure :-1 s=READY pgs=3367 cs=0 l=1 rev1=1 rx=0x7f700400bea0 tx=0x7f700400d940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:37.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:37.477 7f6ffa7fc700 1 -- 172.21.2.112:0/3247221781 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7004000de0 con 0x7f7008095080 2021-02-10T11:50:37.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:37.477 7f6ffa7fc700 1 -- 172.21.2.112:0/3247221781 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f700400b0d0 con 0x7f7008095080 2021-02-10T11:50:37.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:37.477 7f6ffa7fc700 1 -- 172.21.2.112:0/3247221781 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7004009800 con 0x7f7008095080 2021-02-10T11:50:37.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:37.477 7f700fc64700 1 -- 172.21.2.112:0/3247221781 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70080c1fa0 con 0x7f7008095080 2021-02-10T11:50:37.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:37.477 7f700fc64700 1 -- 172.21.2.112:0/3247221781 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70080c1fa0 con 0x7f7008095080 2021-02-10T11:50:37.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:37.478 7f6ffa7fc700 1 -- 172.21.2.112:0/3247221781 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f700400b2b0 con 0x7f7008095080 2021-02-10T11:50:37.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:37.478 7f6ffa7fc700 1 --2- 172.21.2.112:0/3247221781 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6ff00186c0 0x7f6ff001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:37.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:37.478 7f6ffa7fc700 1 -- 172.21.2.112:0/3247221781 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f70040325a0 con 0x7f7008095080 2021-02-10T11:50:37.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:37.478 7f700da80700 1 --2- 172.21.2.112:0/3247221781 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6ff00186c0 0x7f6ff001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:37.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:37.479 7f700da80700 1 --2- 172.21.2.112:0/3247221781 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6ff00186c0 0x7f6ff001abc0 crc :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:37.513 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:37.509 7f700fc64700 1 -- 172.21.2.112:0/3247221781 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70080ebc60 con 0x7f7008095080 2021-02-10T11:50:37.515 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:37.511 7f6ffa7fc700 1 -- 172.21.2.112:0/3247221781 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f700402dc70 con 0x7f7008095080 2021-02-10T11:50:37.645 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:37.640 7f700fc64700 1 -- 172.21.2.112:0/3247221781 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mgr", ""]}) v1 -- 0x7f70080f0b10 con 0x7f6ff00186c0 2021-02-10T11:50:37.645 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:37.641 7f6ffa7fc700 1 -- 172.21.2.112:0/3247221781 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) v1 ==== 41+0+0 (crc 0 0 0) 0x7f6ffc0068c0 con 0x7f6ff00186c0 2021-02-10T11:50:37.646 INFO:teuthology.orchestra.run.gibba012.stderr:instructed osd(s) 7 to deep-scrub 2021-02-10T11:50:37.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:37.643 7f700fc64700 1 -- 172.21.2.112:0/3247221781 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6ff00186c0 msgr2=0x7f6ff001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:37.647 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:37.643 7f700fc64700 1 --2- 172.21.2.112:0/3247221781 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6ff00186c0 0x7f6ff001abc0 crc :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:37.648 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:37.643 7f700fc64700 1 -- 172.21.2.112:0/3247221781 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7008095080 msgr2=0x7f70080c0370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:37.648 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:37.643 7f700fc64700 1 --2- 172.21.2.112:0/3247221781 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7008095080 0x7f70080c0370 secure :-1 s=READY pgs=3367 cs=0 l=1 rev1=1 rx=0x7f700400bea0 tx=0x7f700400d940).stop 2021-02-10T11:50:37.648 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:37.643 7f700fc64700 1 -- 172.21.2.112:0/3247221781 shutdown_connections 2021-02-10T11:50:37.648 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:37.643 7f700fc64700 1 --2- 172.21.2.112:0/3247221781 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f6ff00186c0 0x7f6ff001abc0 unknown :-1 s=CLOSED pgs=1514 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:37.648 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:37.643 7f700fc64700 1 --2- 172.21.2.112:0/3247221781 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f70080c0990 0x7f70080c0d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:37.648 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:37.643 7f700fc64700 1 --2- 172.21.2.112:0/3247221781 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f7008095080 0x7f70080c0370 unknown :-1 s=CLOSED pgs=3367 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:37.648 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:37.643 7f700fc64700 1 --2- 172.21.2.112:0/3247221781 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f70081a49b0 0x7f7008094a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:37.648 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:37.643 7f700fc64700 1 -- 172.21.2.112:0/3247221781 >> 172.21.2.112:0/3247221781 conn(0x7f70080c3e80 msgr2=0x7f70080f00d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:37.649 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:37.643 7f700fc64700 1 -- 172.21.2.112:0/3247221781 shutdown_connections 2021-02-10T11:50:37.649 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:37.643 7f700fc64700 1 -- 172.21.2.112:0/3247221781 wait complete. 2021-02-10T11:50:37.656 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph pg dump --format=json 2021-02-10T11:50:38.084 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:38.064 7ff9e6708700 1 Processor -- start 2021-02-10T11:50:38.085 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:38.064 7ff9e6708700 1 -- start start 2021-02-10T11:50:38.085 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:38.065 7ff9e6708700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff9e00d0340 0x7ff9e00ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:38.085 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:38.065 7ff9e6708700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7ff9e00d1c70 0x7ff9e00efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:38.085 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:38.065 7ff9e6708700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9e00f0b10 con 0x7ff9e00d1870 2021-02-10T11:50:38.085 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:38.065 7ff9e6708700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9e00e5ed0 con 0x7ff9e00d1c70 2021-02-10T11:50:38.085 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:38.065 7ff9e6708700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e01b0be0 con 0x7ff9e00d0340 2021-02-10T11:50:38.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:38.065 7ff9e4d25700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7ff9e00d1c70 0x7ff9e00efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:38.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:38.065 7ff9e4d25700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7ff9e00d1c70 0x7ff9e00efc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:34234/0 (socket says 172.21.2.112:34234) 2021-02-10T11:50:38.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:38.065 7ff9e4d25700 1 -- 172.21.2.112:0/1750673522 learned_addr learned my addr 172.21.2.112:0/1750673522 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:50:38.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:38.065 7ff9e4d25700 1 -- 172.21.2.112:0/1750673522 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff9e00d0340 msgr2=0x7ff9e00ef640 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:50:38.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:38.065 7ff9e4d25700 1 --2- 172.21.2.112:0/1750673522 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff9e00d0340 0x7ff9e00ef640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:38.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:38.065 7ff9e4d25700 1 -- 172.21.2.112:0/1750673522 >> v1:172.21.2.120:6789/0 conn(0x7ff9e00d1870 legacy=0x7ff9e01b03f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:50:38.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:38.065 7ff9e4d25700 1 -- 172.21.2.112:0/1750673522 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9e00f20a0 con 0x7ff9e00d1c70 2021-02-10T11:50:38.086 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:38.065 7ff9e4d25700 1 --2- 172.21.2.112:0/1750673522 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff9e00d1c70 0x7ff9e00efc60 secure :-1 s=READY pgs=3368 cs=0 l=1 rev1=1 rx=0x7ff9d8006f00 tx=0x7ff9d8009f10).ready entity=mon.0 client_cookie=f8397637a81ea4f1 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:38.087 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:38.065 7ff9deffd700 1 -- 172.21.2.112:0/1750673522 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff9d800b3b0 con 0x7ff9e00d1c70 2021-02-10T11:50:38.087 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:38.066 7ff9deffd700 1 -- 172.21.2.112:0/1750673522 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9d800b630 con 0x7ff9e00d1c70 2021-02-10T11:50:38.087 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:38.066 7ff9deffd700 1 -- 172.21.2.112:0/1750673522 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff9d800bb10 con 0x7ff9e00d1c70 2021-02-10T11:50:38.087 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:38.066 7ff9e6708700 1 -- 172.21.2.112:0/1750673522 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff9e00d1c70 msgr2=0x7ff9e00efc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:38.087 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:38.066 7ff9e6708700 1 --2- 172.21.2.112:0/1750673522 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff9e00d1c70 0x7ff9e00efc60 secure :-1 s=READY pgs=3368 cs=0 l=1 rev1=1 rx=0x7ff9d8006f00 tx=0x7ff9d8009f10).stop 2021-02-10T11:50:38.087 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:38.066 7ff9e6708700 1 -- 172.21.2.112:0/1750673522 shutdown_connections 2021-02-10T11:50:38.087 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:38.066 7ff9e6708700 1 --2- 172.21.2.112:0/1750673522 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff9e00d1c70 0x7ff9e00efc60 unknown :-1 s=CLOSED pgs=3368 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:38.087 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:38.066 7ff9e6708700 1 --2- 172.21.2.112:0/1750673522 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff9e00d0340 0x7ff9e00ef640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:38.088 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:38.066 7ff9e6708700 1 -- 172.21.2.112:0/1750673522 >> 172.21.2.112:0/1750673522 conn(0x7ff9e00ce450 msgr2=0x7ff9e00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:38.088 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:38.066 7ff9e6708700 1 -- 172.21.2.112:0/1750673522 shutdown_connections 2021-02-10T11:50:38.088 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:38.066 7ff9e6708700 1 -- 172.21.2.112:0/1750673522 wait complete. 2021-02-10T11:50:38.089 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:38.081 7ff9e6708700 1 Processor -- start 2021-02-10T11:50:38.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:38.081 7ff9e6708700 1 -- start start 2021-02-10T11:50:38.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:38.081 7ff9e6708700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff9e01abb30 0x7ff9e0187e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:38.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:38.081 7ff9e6708700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff9e00b48b0 0x7ff9e008c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:38.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:38.081 7ff9e6708700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff9e00b4cb0 0x7ff9e008ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:38.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:38.081 7ff9e6708700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e01ac060 con 0x7ff9e00b4cb0 2021-02-10T11:50:38.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:38.081 7ff9e6708700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e008d280 con 0x7ff9e01abb30 2021-02-10T11:50:38.090 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:38.081 7ff9e6708700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e018cb00 con 0x7ff9e00b48b0 2021-02-10T11:50:38.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:38.081 7ff9e4d25700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff9e00b4cb0 0x7ff9e008ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:38.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:38.081 7ff9e4d25700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff9e00b4cb0 0x7ff9e008ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:34236/0 (socket says 172.21.2.112:34236) 2021-02-10T11:50:38.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:38.081 7ff9e4d25700 1 -- 172.21.2.112:0/248016459 learned_addr learned my addr 172.21.2.112:0/248016459 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:50:38.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:38.082 7ff9dffff700 1 --2- 172.21.2.112:0/248016459 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff9e01abb30 0x7ff9e0187e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:38.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:38.082 7ff9df7fe700 1 --2- 172.21.2.112:0/248016459 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff9e00b48b0 0x7ff9e008c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:38.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:38.082 7ff9e4d25700 1 -- 172.21.2.112:0/248016459 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff9e00b48b0 msgr2=0x7ff9e008c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:38.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:38.082 7ff9e4d25700 1 --2- 172.21.2.112:0/248016459 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff9e00b48b0 0x7ff9e008c460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:38.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:38.082 7ff9e4d25700 1 -- 172.21.2.112:0/248016459 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff9e01abb30 msgr2=0x7ff9e0187e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:38.091 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:38.082 7ff9e4d25700 1 --2- 172.21.2.112:0/248016459 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff9e01abb30 0x7ff9e0187e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:38.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:38.082 7ff9e4d25700 1 -- 172.21.2.112:0/248016459 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9e018cce0 con 0x7ff9e00b4cb0 2021-02-10T11:50:38.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:38.082 7ff9df7fe700 1 --2- 172.21.2.112:0/248016459 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff9e00b48b0 0x7ff9e008c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:50:38.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:38.082 7ff9dffff700 1 --2- 172.21.2.112:0/248016459 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff9e01abb30 0x7ff9e0187e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:50:38.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:38.082 7ff9e4d25700 1 --2- 172.21.2.112:0/248016459 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff9e00b4cb0 0x7ff9e008ca80 secure :-1 s=READY pgs=3369 cs=0 l=1 rev1=1 rx=0x7ff9d8009390 tx=0x7ff9d800bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:38.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:38.082 7ff9dcff9700 1 -- 172.21.2.112:0/248016459 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff9d8003030 con 0x7ff9e00b4cb0 2021-02-10T11:50:38.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:38.082 7ff9dcff9700 1 -- 172.21.2.112:0/248016459 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9d8006c80 con 0x7ff9e00b4cb0 2021-02-10T11:50:38.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:38.082 7ff9dcff9700 1 -- 172.21.2.112:0/248016459 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff9d8001480 con 0x7ff9e00b4cb0 2021-02-10T11:50:38.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:38.082 7ff9e6708700 1 -- 172.21.2.112:0/248016459 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9e018cce0 con 0x7ff9e00b4cb0 2021-02-10T11:50:38.092 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:38.082 7ff9e6708700 1 -- 172.21.2.112:0/248016459 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9e008daf0 con 0x7ff9e00b4cb0 2021-02-10T11:50:38.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:38.083 7ff9dcff9700 1 -- 172.21.2.112:0/248016459 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff9d8001270 con 0x7ff9e00b4cb0 2021-02-10T11:50:38.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:38.083 7ff9dcff9700 1 --2- 172.21.2.112:0/248016459 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff9c00186c0 0x7ff9c001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:38.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:38.083 7ff9dcff9700 1 -- 172.21.2.112:0/248016459 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 5573+0+0 (secure 0 0 0) 0x7ff9d80328f0 con 0x7ff9e00b4cb0 2021-02-10T11:50:38.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:38.083 7ff9dffff700 1 --2- 172.21.2.112:0/248016459 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff9c00186c0 0x7ff9c001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:38.093 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:38.084 7ff9dffff700 1 --2- 172.21.2.112:0/248016459 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff9c00186c0 0x7ff9c001abc0 crc :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:38.119 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:38.114 7ff9e6708700 1 -- 172.21.2.112:0/248016459 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9e018cce0 con 0x7ff9e00b4cb0 2021-02-10T11:50:38.121 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:38.117 7ff9dcff9700 1 -- 172.21.2.112:0/248016459 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff9d802b020 con 0x7ff9e00b4cb0 2021-02-10T11:50:38.254 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:38.249 7ff9e6708700 1 -- 172.21.2.112:0/248016459 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "pg dump", "target": ["mgr", ""], "format": "json"}) v1 -- 0x7ff9e008daf0 con 0x7ff9c00186c0 2021-02-10T11:50:38.256 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:38.252 7ff9dcff9700 1 -- 172.21.2.112:0/248016459 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 dumped all) v1 ==== 18+0+101202 (crc 0 0 0) 0x7ff9d0006bc0 con 0x7ff9c00186c0 2021-02-10T11:50:38.258 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:50:38.258 INFO:teuthology.orchestra.run.gibba012.stderr:dumped all 2021-02-10T11:50:38.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:38.255 7ff9e6708700 1 -- 172.21.2.112:0/248016459 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff9c00186c0 msgr2=0x7ff9c001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:38.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:38.255 7ff9e6708700 1 --2- 172.21.2.112:0/248016459 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff9c00186c0 0x7ff9c001abc0 crc :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:38.259 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:38.255 7ff9e6708700 1 -- 172.21.2.112:0/248016459 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff9e00b4cb0 msgr2=0x7ff9e008ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:38.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:38.255 7ff9e6708700 1 --2- 172.21.2.112:0/248016459 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff9e00b4cb0 0x7ff9e008ca80 secure :-1 s=READY pgs=3369 cs=0 l=1 rev1=1 rx=0x7ff9d8009390 tx=0x7ff9d800bd40).stop 2021-02-10T11:50:38.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:38.255 7ff9e6708700 1 -- 172.21.2.112:0/248016459 shutdown_connections 2021-02-10T11:50:38.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:38.255 7ff9e6708700 1 --2- 172.21.2.112:0/248016459 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7ff9c00186c0 0x7ff9c001abc0 unknown :-1 s=CLOSED pgs=1515 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:38.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:38.255 7ff9e6708700 1 --2- 172.21.2.112:0/248016459 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7ff9e00b4cb0 0x7ff9e008ca80 unknown :-1 s=CLOSED pgs=3369 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:38.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:38.255 7ff9e6708700 1 --2- 172.21.2.112:0/248016459 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7ff9e00b48b0 0x7ff9e008c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:38.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:38.255 7ff9e6708700 1 --2- 172.21.2.112:0/248016459 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7ff9e01abb30 0x7ff9e0187e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:38.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:38.255 7ff9e6708700 1 -- 172.21.2.112:0/248016459 >> 172.21.2.112:0/248016459 conn(0x7ff9e019f880 msgr2=0x7ff9e00ebe30 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:38.260 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:38.255 7ff9e6708700 1 -- 172.21.2.112:0/248016459 shutdown_connections 2021-02-10T11:50:38.261 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:38.255 7ff9e6708700 1 -- 172.21.2.112:0/248016459 wait complete. 2021-02-10T11:50:38.267 INFO:teuthology.orchestra.run.gibba012.stdout:{"pg_ready":true,"pg_map":{"version":1303,"stamp":"2021-02-10 11:50:36.273588","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":135153129,"num_objects":154,"num_object_clones":1,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":154,"num_whiteouts":0,"num_read":65,"num_read_kb":65537,"num_write":432,"num_write_kb":132086,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":417,"ondisk_log_size":417,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"kb":754974720,"kb_used":8672708,"kb_used_data":284024,"kb_used_omap":0,"kb_used_meta":8388608,"kb_avail":746302012,"statfs":{"total":773094113280,"available":764213260288,"internally_reserved":8589934592,"allocated":290840576,"data_stored":279110034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8589934592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":156761,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002852"},"pg_stats":[{"pgid":"43.6","version":"198'22","reported_seq":"40","reported_epoch":"198","state":"active+clean","last_fresh":"2021-02-10 11:50:17.478190","last_change":"2021-02-10 11:49:04.486122","last_active":"2021-02-10 11:50:17.478190","last_peered":"2021-02-10 11:50:17.478190","last_clean":"2021-02-10 11:50:17.478190","last_became_active":"2021-02-10 11:49:04.485856","last_became_peered":"2021-02-10 11:49:04.485856","last_unstale":"2021-02-10 11:50:17.478190","last_undegraded":"2021-02-10 11:50:17.478190","last_fullsized":"2021-02-10 11:50:17.478190","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:49:03.842471","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:49:03.842471","last_clean_scrub_stamp":"2021-02-10 11:49:03.842471","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":23068672,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":9,"num_read_kb":9216,"num_write":22,"num_write_kb":22528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.7","version":"198'8","reported_seq":"19","reported_epoch":"198","state":"active+clean","last_fresh":"2021-02-10 11:49:05.908221","last_change":"2021-02-10 11:49:03.849288","last_active":"2021-02-10 11:49:05.908221","last_peered":"2021-02-10 11:49:05.908221","last_clean":"2021-02-10 11:49:05.908221","last_became_active":"2021-02-10 11:49:03.849100","last_became_peered":"2021-02-10 11:49:03.849100","last_unstale":"2021-02-10 11:49:05.908221","last_undegraded":"2021-02-10 11:49:05.908221","last_fullsized":"2021-02-10 11:49:05.908221","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:49:02.834883","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:49:02.834883","last_clean_scrub_stamp":"2021-02-10 11:49:02.834883","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":566,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"43.7","version":"198'14","reported_seq":"30","reported_epoch":"198","state":"active+clean","last_fresh":"2021-02-10 11:50:17.482787","last_change":"2021-02-10 11:49:04.486203","last_active":"2021-02-10 11:50:17.482787","last_peered":"2021-02-10 11:50:17.482787","last_clean":"2021-02-10 11:50:17.482787","last_became_active":"2021-02-10 11:49:04.485927","last_became_peered":"2021-02-10 11:49:04.485927","last_unstale":"2021-02-10 11:50:17.482787","last_undegraded":"2021-02-10 11:50:17.482787","last_fullsized":"2021-02-10 11:50:17.482787","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:49:03.842471","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:49:03.842471","last_clean_scrub_stamp":"2021-02-10 11:49:03.842471","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14680064,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":7,"num_read_kb":7168,"num_write":14,"num_write_kb":14336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.6","version":"198'16","reported_seq":"27","reported_epoch":"198","state":"active+clean","last_fresh":"2021-02-10 11:50:18.838786","last_change":"2021-02-10 11:49:03.848744","last_active":"2021-02-10 11:50:18.838786","last_peered":"2021-02-10 11:50:18.838786","last_clean":"2021-02-10 11:50:18.838786","last_became_active":"2021-02-10 11:49:03.848469","last_became_peered":"2021-02-10 11:49:03.848469","last_unstale":"2021-02-10 11:50:18.838786","last_undegraded":"2021-02-10 11:50:18.838786","last_fullsized":"2021-02-10 11:50:18.838786","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:49:02.834883","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:49:02.834883","last_clean_scrub_stamp":"2021-02-10 11:49:02.834883","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":35,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"43.5","version":"198'23","reported_seq":"41","reported_epoch":"198","state":"active+clean","last_fresh":"2021-02-10 11:50:17.521238","last_change":"2021-02-10 11:49:04.485844","last_active":"2021-02-10 11:50:17.521238","last_peered":"2021-02-10 11:50:17.521238","last_clean":"2021-02-10 11:50:17.521238","last_became_active":"2021-02-10 11:49:04.485608","last_became_peered":"2021-02-10 11:49:04.485608","last_unstale":"2021-02-10 11:50:17.521238","last_undegraded":"2021-02-10 11:50:17.521238","last_fullsized":"2021-02-10 11:50:17.521238","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:49:03.842471","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:49:03.842471","last_clean_scrub_stamp":"2021-02-10 11:49:03.842471","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16777506,"num_objects":18,"num_object_clones":1,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":10,"num_read_kb":10240,"num_write":20,"num_write_kb":16387,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"42.4","version":"198'223","reported_seq":"234","reported_epoch":"198","state":"active+clean","last_fresh":"2021-02-10 11:50:28.839246","last_change":"2021-02-10 11:49:03.848271","last_active":"2021-02-10 11:50:28.839246","last_peered":"2021-02-10 11:50:28.839246","last_clean":"2021-02-10 11:50:28.839246","last_became_active":"2021-02-10 11:49:03.848058","last_became_peered":"2021-02-10 11:49:03.848058","last_unstale":"2021-02-10 11:50:28.839246","last_undegraded":"2021-02-10 11:50:28.839246","last_fullsized":"2021-02-10 11:50:28.839246","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:49:02.834883","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:49:02.834883","last_clean_scrub_stamp":"2021-02-10 11:49:02.834883","log_size":223,"ondisk_log_size":223,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":934113,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":223,"num_write_kb":980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"43.2","version":"198'20","reported_seq":"37","reported_epoch":"198","state":"active+clean","last_fresh":"2021-02-10 11:50:17.460794","last_change":"2021-02-10 11:49:04.486157","last_active":"2021-02-10 11:50:17.460794","last_peered":"2021-02-10 11:50:17.460794","last_clean":"2021-02-10 11:50:17.460794","last_became_active":"2021-02-10 11:49:04.485916","last_became_peered":"2021-02-10 11:49:04.485916","last_unstale":"2021-02-10 11:50:17.460794","last_undegraded":"2021-02-10 11:50:17.460794","last_fullsized":"2021-02-10 11:50:17.460794","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:49:03.842471","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:49:03.842471","last_clean_scrub_stamp":"2021-02-10 11:49:03.842471","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20971520,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":8,"num_read_kb":8192,"num_write":20,"num_write_kb":20480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.3","version":"198'6","reported_seq":"17","reported_epoch":"198","state":"active+clean","last_fresh":"2021-02-10 11:49:05.486065","last_change":"2021-02-10 11:49:03.848638","last_active":"2021-02-10 11:49:05.486065","last_peered":"2021-02-10 11:49:05.486065","last_clean":"2021-02-10 11:49:05.486065","last_became_active":"2021-02-10 11:49:03.848441","last_became_peered":"2021-02-10 11:49:03.848441","last_unstale":"2021-02-10 11:49:05.486065","last_undegraded":"2021-02-10 11:49:05.486065","last_fullsized":"2021-02-10 11:49:05.486065","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:49:02.834883","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:49:02.834883","last_clean_scrub_stamp":"2021-02-10 11:49:02.834883","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"43.3","version":"198'25","reported_seq":"42","reported_epoch":"198","state":"active+clean","last_fresh":"2021-02-10 11:50:17.526323","last_change":"2021-02-10 11:49:04.486243","last_active":"2021-02-10 11:50:17.526323","last_peered":"2021-02-10 11:50:17.526323","last_clean":"2021-02-10 11:50:17.526323","last_became_active":"2021-02-10 11:49:04.485844","last_became_peered":"2021-02-10 11:49:04.485844","last_unstale":"2021-02-10 11:50:17.526323","last_undegraded":"2021-02-10 11:50:17.526323","last_fullsized":"2021-02-10 11:50:17.526323","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:49:03.842471","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:49:03.842471","last_clean_scrub_stamp":"2021-02-10 11:49:03.842471","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18874505,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":8,"num_read_kb":8192,"num_write":25,"num_write_kb":18436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.2","version":"197'1","reported_seq":"12","reported_epoch":"197","state":"active+clean","last_fresh":"2021-02-10 11:49:04.488456","last_change":"2021-02-10 11:49:03.849353","last_active":"2021-02-10 11:49:04.488456","last_peered":"2021-02-10 11:49:04.488456","last_clean":"2021-02-10 11:49:04.488456","last_became_active":"2021-02-10 11:49:03.849140","last_became_peered":"2021-02-10 11:49:03.849140","last_unstale":"2021-02-10 11:49:04.488456","last_undegraded":"2021-02-10 11:49:04.488456","last_fullsized":"2021-02-10 11:49:04.488456","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:49:02.834883","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:49:02.834883","last_clean_scrub_stamp":"2021-02-10 11:49:02.834883","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"43.0","version":"198'19","reported_seq":"38","reported_epoch":"198","state":"active+clean","last_fresh":"2021-02-10 11:50:17.436525","last_change":"2021-02-10 11:49:04.487368","last_active":"2021-02-10 11:50:17.436525","last_peered":"2021-02-10 11:50:17.436525","last_clean":"2021-02-10 11:50:17.436525","last_became_active":"2021-02-10 11:49:04.487045","last_became_peered":"2021-02-10 11:49:04.487045","last_unstale":"2021-02-10 11:50:17.436525","last_undegraded":"2021-02-10 11:50:17.436525","last_fullsized":"2021-02-10 11:50:17.436525","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:49:03.842471","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:49:03.842471","last_clean_scrub_stamp":"2021-02-10 11:49:03.842471","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18874368,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":10,"num_read_kb":10240,"num_write":18,"num_write_kb":18432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"42.1","version":"198'2","reported_seq":"14","reported_epoch":"198","state":"active+clean","last_fresh":"2021-02-10 11:50:18.496010","last_change":"2021-02-10 11:49:03.849478","last_active":"2021-02-10 11:50:18.496010","last_peered":"2021-02-10 11:50:18.496010","last_clean":"2021-02-10 11:50:18.496010","last_became_active":"2021-02-10 11:49:03.849304","last_became_peered":"2021-02-10 11:49:03.849304","last_unstale":"2021-02-10 11:50:18.496010","last_undegraded":"2021-02-10 11:50:18.496010","last_fullsized":"2021-02-10 11:50:18.496010","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:49:02.834883","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:49:02.834883","last_clean_scrub_stamp":"2021-02-10 11:49:02.834883","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":101,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"43.1","version":"198'11","reported_seq":"26","reported_epoch":"198","state":"active+clean","last_fresh":"2021-02-10 11:50:17.501741","last_change":"2021-02-10 11:49:04.487450","last_active":"2021-02-10 11:50:17.501741","last_peered":"2021-02-10 11:50:17.501741","last_clean":"2021-02-10 11:50:17.501741","last_became_active":"2021-02-10 11:49:04.487096","last_became_peered":"2021-02-10 11:49:04.487096","last_unstale":"2021-02-10 11:50:17.501741","last_undegraded":"2021-02-10 11:50:17.501741","last_fullsized":"2021-02-10 11:50:17.501741","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:49:03.842471","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:49:03.842471","last_clean_scrub_stamp":"2021-02-10 11:49:03.842471","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":10485760,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":6,"num_read_kb":6144,"num_write":10,"num_write_kb":10240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.0","version":"197'8","reported_seq":"19","reported_epoch":"197","state":"active+clean","last_fresh":"2021-02-10 11:49:04.490252","last_change":"2021-02-10 11:49:03.848900","last_active":"2021-02-10 11:49:04.490252","last_peered":"2021-02-10 11:49:04.490252","last_clean":"2021-02-10 11:49:04.490252","last_became_active":"2021-02-10 11:49:03.848142","last_became_peered":"2021-02-10 11:49:03.848142","last_unstale":"2021-02-10 11:49:04.490252","last_undegraded":"2021-02-10 11:49:04.490252","last_fullsized":"2021-02-10 11:49:04.490252","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:49:02.834883","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:49:02.834883","last_clean_scrub_stamp":"2021-02-10 11:49:02.834883","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":"180","reported_epoch":"197","state":"active+clean","last_fresh":"2021-02-10 11:49:04.484090","last_change":"2021-02-10 11:13:58.963167","last_active":"2021-02-10 11:49:04.484090","last_peered":"2021-02-10 11:49:04.484090","last_clean":"2021-02-10 11:49:04.484090","last_became_active":"2021-02-10 11:13:58.962972","last_became_peered":"2021-02-10 11:13:58.962972","last_unstale":"2021-02-10 11:49:04.484090","last_undegraded":"2021-02-10 11:49:04.484090","last_fullsized":"2021-02-10 11:49:04.484090","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:13:57.948661","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:13:57.948661","last_clean_scrub_stamp":"2021-02-10 11:13:57.948661","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":"180","reported_epoch":"197","state":"active+clean","last_fresh":"2021-02-10 11:49:04.484174","last_change":"2021-02-10 11:13:58.964263","last_active":"2021-02-10 11:49:04.484174","last_peered":"2021-02-10 11:49:04.484174","last_clean":"2021-02-10 11:49:04.484174","last_became_active":"2021-02-10 11:13:58.964050","last_became_peered":"2021-02-10 11:13:58.964050","last_unstale":"2021-02-10 11:49:04.484174","last_undegraded":"2021-02-10 11:49:04.484174","last_fullsized":"2021-02-10 11:49:04.484174","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:13:57.948661","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:13:57.948661","last_clean_scrub_stamp":"2021-02-10 11:13:57.948661","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"187","reported_epoch":"197","state":"active+clean","last_fresh":"2021-02-10 11:49:04.483150","last_change":"2021-02-10 11:13:58.964235","last_active":"2021-02-10 11:49:04.483150","last_peered":"2021-02-10 11:49:04.483150","last_clean":"2021-02-10 11:49:04.483150","last_became_active":"2021-02-10 11:13:58.963951","last_became_peered":"2021-02-10 11:13:58.963951","last_unstale":"2021-02-10 11:49:04.483150","last_undegraded":"2021-02-10 11:49:04.483150","last_fullsized":"2021-02-10 11:49:04.483150","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:13:57.948661","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:13:57.948661","last_clean_scrub_stamp":"2021-02-10 11:13:57.948661","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":"183","reported_epoch":"197","state":"active+clean","last_fresh":"2021-02-10 11:49:04.483420","last_change":"2021-02-10 11:13:58.962265","last_active":"2021-02-10 11:49:04.483420","last_peered":"2021-02-10 11:49:04.483420","last_clean":"2021-02-10 11:49:04.483420","last_became_active":"2021-02-10 11:13:58.961934","last_became_peered":"2021-02-10 11:13:58.961934","last_unstale":"2021-02-10 11:49:04.483420","last_undegraded":"2021-02-10 11:49:04.483420","last_fullsized":"2021-02-10 11:49:04.483420","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:13:57.948661","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:13:57.948661","last_clean_scrub_stamp":"2021-02-10 11:13:57.948661","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"179","reported_epoch":"197","state":"active+clean","last_fresh":"2021-02-10 11:49:04.482926","last_change":"2021-02-10 11:13:58.964324","last_active":"2021-02-10 11:49:04.482926","last_peered":"2021-02-10 11:49:04.482926","last_clean":"2021-02-10 11:49:04.482926","last_became_active":"2021-02-10 11:13:58.964097","last_became_peered":"2021-02-10 11:13:58.964097","last_unstale":"2021-02-10 11:49:04.482926","last_undegraded":"2021-02-10 11:49:04.482926","last_fullsized":"2021-02-10 11:49:04.482926","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:13:57.948661","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:13:57.948661","last_clean_scrub_stamp":"2021-02-10 11:13:57.948661","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"43.4","version":"198'10","reported_seq":"25","reported_epoch":"198","state":"active+clean","last_fresh":"2021-02-10 11:50:17.445438","last_change":"2021-02-10 11:49:04.485490","last_active":"2021-02-10 11:50:17.445438","last_peered":"2021-02-10 11:50:17.445438","last_clean":"2021-02-10 11:50:17.445438","last_became_active":"2021-02-10 11:49:04.485246","last_became_peered":"2021-02-10 11:49:04.485246","last_unstale":"2021-02-10 11:50:17.445438","last_undegraded":"2021-02-10 11:50:17.445438","last_fullsized":"2021-02-10 11:50:17.445438","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:49:03.842471","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:49:03.842471","last_clean_scrub_stamp":"2021-02-10 11:49:03.842471","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":10485760,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":6,"num_read_kb":6144,"num_write":10,"num_write_kb":10240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"42.5","version":"198'9","reported_seq":"20","reported_epoch":"198","state":"active+clean","last_fresh":"2021-02-10 11:50:18.493972","last_change":"2021-02-10 11:49:03.848519","last_active":"2021-02-10 11:50:18.493972","last_peered":"2021-02-10 11:50:18.493972","last_clean":"2021-02-10 11:50:18.493972","last_became_active":"2021-02-10 11:49:03.848285","last_became_peered":"2021-02-10 11:49:03.848285","last_unstale":"2021-02-10 11:50:18.493972","last_undegraded":"2021-02-10 11:50:18.493972","last_fullsized":"2021-02-10 11:50:18.493972","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:49:02.834883","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:49:02.834883","last_clean_scrub_stamp":"2021-02-10 11:49:02.834883","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":160,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"188","reported_epoch":"197","state":"active+clean","last_fresh":"2021-02-10 11:49:04.484033","last_change":"2021-02-10 11:13:58.964040","last_active":"2021-02-10 11:49:04.484033","last_peered":"2021-02-10 11:49:04.484033","last_clean":"2021-02-10 11:49:04.484033","last_became_active":"2021-02-10 11:13:58.963814","last_became_peered":"2021-02-10 11:13:58.963814","last_unstale":"2021-02-10 11:49:04.484033","last_undegraded":"2021-02-10 11:49:04.484033","last_fullsized":"2021-02-10 11:49:04.484033","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:13:57.948661","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:13:57.948661","last_clean_scrub_stamp":"2021-02-10 11:13:57.948661","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"179","reported_epoch":"197","state":"active+clean","last_fresh":"2021-02-10 11:49:04.482937","last_change":"2021-02-10 11:13:58.964469","last_active":"2021-02-10 11:49:04.482937","last_peered":"2021-02-10 11:49:04.482937","last_clean":"2021-02-10 11:49:04.482937","last_became_active":"2021-02-10 11:13:58.964040","last_became_peered":"2021-02-10 11:13:58.964040","last_unstale":"2021-02-10 11:49:04.482937","last_undegraded":"2021-02-10 11:49:04.482937","last_fullsized":"2021-02-10 11:49:04.482937","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:13:57.948661","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:13:57.948661","last_clean_scrub_stamp":"2021-02-10 11:13:57.948661","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"183","reported_epoch":"198","state":"active+clean","last_fresh":"2021-02-10 11:50:31.812840","last_change":"2021-02-10 11:50:31.812874","last_active":"2021-02-10 11:50:31.812840","last_peered":"2021-02-10 11:50:31.812840","last_clean":"2021-02-10 11:50:31.812840","last_became_active":"2021-02-10 11:13:58.962533","last_became_peered":"2021-02-10 11:13:58.962533","last_unstale":"2021-02-10 11:50:31.812840","last_undegraded":"2021-02-10 11:50:31.812840","last_fullsized":"2021-02-10 11:50:31.812840","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:50:31.812837","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:50:31.812837","last_clean_scrub_stamp":"2021-02-10 11:50:31.812837","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":43,"num_pg":8,"stat_sum":{"num_bytes":134218155,"num_objects":131,"num_object_clones":1,"num_object_copies":262,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":131,"num_whiteouts":0,"num_read":64,"num_read_kb":65536,"num_write":139,"num_write_kb":131079,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":268460032,"data_stored":268436310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":144,"ondisk_log_size":144,"up":16,"acting":16,"num_store_stats":8},{"poolid":42,"num_pg":8,"stat_sum":{"num_bytes":934974,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293,"num_write_kb":1007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1933312,"data_stored":1869948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":273,"ondisk_log_size":273,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575291,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1087964,"kb_used_data":39380,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93283876,"statfs":{"total":96636764160,"available":95522689024,"internally_reserved":1073741824,"allocated":40325120,"data_stored":38850165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 11:49:53 2021","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.82499999999999996,"15min":0.79000000000000004},"min":{"1min":0.37,"5min":0.37,"15min":0.36499999999999999},"max":{"1min":1.1850000000000001,"5min":1.248,"15min":1.4259999999999999},"last":0.628},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.82599999999999996,"15min":0.80600000000000005},"min":{"1min":0.40699999999999997,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.1379999999999999,"5min":1.3340000000000001,"15min":1.3939999999999999},"last":0.60599999999999998}]},{"osd":1,"last update":"Wed Feb 10 11:49:53 2021","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.80200000000000005,"15min":0.80500000000000005},"min":{"1min":0.44700000000000001,"5min":0.436,"15min":0.42199999999999999},"max":{"1min":1.161,"5min":1.3839999999999999,"15min":1.3839999999999999},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.83499999999999996,"5min":0.84999999999999998,"15min":0.81599999999999995},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.371},"max":{"1min":1.1140000000000001,"5min":1.3979999999999999,"15min":1.3979999999999999},"last":0.80800000000000005}]},{"osd":2,"last update":"Wed Feb 10 11:49:53 2021","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":0.91400000000000003,"15min":0.88100000000000001},"min":{"1min":0.432,"5min":0.42899999999999999,"15min":0.40699999999999997},"max":{"1min":1.151,"5min":1.4510000000000001,"15min":1.4510000000000001},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.94199999999999995,"5min":1.0029999999999999,"15min":0.96999999999999997},"min":{"1min":0.54900000000000004,"5min":0.51400000000000001,"15min":0.47399999999999998},"max":{"1min":1.2250000000000001,"5min":1.5329999999999999,"15min":1.5329999999999999},"last":0.75900000000000001}]},{"osd":3,"last update":"Wed Feb 10 11:49:53 2021","interfaces":[{"interface":"back","average":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.80400000000000005},"min":{"1min":0.51200000000000001,"5min":0.46700000000000003,"15min":0.40699999999999997},"max":{"1min":1.099,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.89800000000000002,"5min":0.90600000000000003,"15min":0.85999999999999999},"min":{"1min":0.5,"5min":0.48099999999999998,"15min":0.42599999999999999},"max":{"1min":1.1259999999999999,"5min":1.4339999999999999,"15min":1.4339999999999999},"last":0.82899999999999996}]},{"osd":4,"last update":"Wed Feb 10 11:49:53 2021","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.69699999999999995,"15min":0.69299999999999995},"min":{"1min":0.46400000000000002,"5min":0.40999999999999998,"15min":0.34300000000000003},"max":{"1min":0.89000000000000001,"5min":1.3320000000000001,"15min":1.3320000000000001},"last":0.88400000000000001},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.70999999999999996,"15min":0.69399999999999995},"min":{"1min":0.42599999999999999,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.93700000000000006,"5min":1.127,"15min":1.149},"last":0.79900000000000004}]},{"osd":5,"last update":"Wed Feb 10 11:49:53 2021","interfaces":[{"interface":"back","average":{"1min":0.86399999999999999,"5min":0.81399999999999995,"15min":0.80800000000000005},"min":{"1min":0.61599999999999999,"5min":0.48399999999999999,"15min":0.45700000000000002},"max":{"1min":1.1259999999999999,"5min":1.3140000000000001,"15min":1.3140000000000001},"last":0.95299999999999996},{"interface":"front","average":{"1min":0.85799999999999998,"5min":0.88200000000000001,"15min":0.878},"min":{"1min":0.59399999999999997,"5min":0.50700000000000001,"15min":0.45200000000000001},"max":{"1min":1.1220000000000001,"5min":1.3260000000000001,"15min":1.377},"last":0.98699999999999999}]},{"osd":6,"last update":"Wed Feb 10 11:49:53 2021","interfaces":[{"interface":"back","average":{"1min":0.80600000000000005,"5min":0.81699999999999995,"15min":0.82299999999999995},"min":{"1min":0.55700000000000005,"5min":0.52900000000000003,"15min":0.50700000000000001},"max":{"1min":0.97599999999999998,"5min":1.2709999999999999,"15min":1.327},"last":1.05},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.80200000000000005,"15min":0.80300000000000005},"min":{"1min":0.55400000000000005,"5min":0.53800000000000003,"15min":0.53200000000000003},"max":{"1min":1.0469999999999999,"5min":1.276,"15min":1.276},"last":0.85799999999999998}]}]},{"osd":6,"up_from":13,"seq":55834575292,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1070436,"kb_used_data":21852,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93301404,"statfs":{"total":96636764160,"available":95540637696,"internally_reserved":1073741824,"allocated":22376448,"data_stored":20908987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 11:50:07 2021","interfaces":[{"interface":"back","average":{"1min":0.94199999999999995,"5min":0.83199999999999996,"15min":0.81299999999999994},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.34699999999999998},"max":{"1min":1.169,"5min":1.1859999999999999,"15min":1.4259999999999999},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.879,"5min":0.78700000000000003,"15min":0.78600000000000003},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41299999999999998},"max":{"1min":1.131,"5min":1.2050000000000001,"15min":1.4419999999999999},"last":0.84599999999999997}]},{"osd":1,"last update":"Wed Feb 10 11:50:07 2021","interfaces":[{"interface":"back","average":{"1min":0.90800000000000003,"5min":0.80800000000000005,"15min":0.79800000000000004},"min":{"1min":0.46300000000000002,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.147,"5min":1.147,"15min":1.504},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.95999999999999996,"5min":0.83699999999999997,"15min":0.83399999999999996},"min":{"1min":0.48399999999999999,"5min":0.45600000000000002,"15min":0.45300000000000001},"max":{"1min":1.2549999999999999,"5min":1.2549999999999999,"15min":1.3959999999999999},"last":0.80000000000000004}]},{"osd":2,"last update":"Wed Feb 10 11:50:07 2021","interfaces":[{"interface":"back","average":{"1min":1.0149999999999999,"5min":0.93400000000000005,"15min":0.93500000000000005},"min":{"1min":0.623,"5min":0.49299999999999999,"15min":0.438},"max":{"1min":1.339,"5min":1.339,"15min":1.3799999999999999},"last":0.81899999999999995},{"interface":"front","average":{"1min":1.054,"5min":0.996,"15min":0.98699999999999999},"min":{"1min":0.61399999999999999,"5min":0.61299999999999999,"15min":0.55200000000000005},"max":{"1min":1.3320000000000001,"5min":1.3320000000000001,"15min":1.4950000000000001},"last":0.97299999999999998}]},{"osd":3,"last update":"Wed Feb 10 11:50:07 2021","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":0.90700000000000003,"15min":0.90200000000000002},"min":{"1min":0.498,"5min":0.47199999999999998,"15min":0.39900000000000002},"max":{"1min":1.272,"5min":1.272,"15min":1.272},"last":0.90000000000000002},{"interface":"front","average":{"1min":1.0129999999999999,"5min":0.96699999999999997,"15min":0.95699999999999996},"min":{"1min":0.76700000000000002,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.246,"5min":1.258,"15min":1.361},"last":0.92200000000000004}]},{"osd":4,"last update":"Wed Feb 10 11:50:07 2021","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.66900000000000004,"15min":0.71599999999999997},"min":{"1min":0.48699999999999999,"5min":0.441,"15min":0.31900000000000001},"max":{"1min":0.92800000000000005,"5min":1.0960000000000001,"15min":1.399},"last":0.85999999999999999},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.70099999999999996,"15min":0.71399999999999997},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.35599999999999998},"max":{"1min":0.88700000000000001,"5min":1.004,"15min":1.23},"last":0.67400000000000004}]},{"osd":5,"last update":"Wed Feb 10 11:50:07 2021","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.68000000000000005,"15min":0.71899999999999997},"min":{"1min":0.45200000000000001,"5min":0.42199999999999999,"15min":0.40899999999999997},"max":{"1min":0.85199999999999998,"5min":1.113,"15min":1.327},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.75,"5min":0.77500000000000002,"15min":0.79600000000000004},"min":{"1min":0.58699999999999997,"5min":0.57699999999999996,"15min":0.40699999999999997},"max":{"1min":0.97799999999999998,"5min":1.163,"15min":1.5},"last":0.91400000000000003}]},{"osd":7,"last update":"Wed Feb 10 11:50:07 2021","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.75900000000000001,"15min":0.78000000000000003},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.41599999999999998},"max":{"1min":0.92700000000000005,"5min":1.149,"15min":1.2829999999999999},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.79300000000000004,"15min":0.81499999999999995},"min":{"1min":0.58399999999999996,"5min":0.53900000000000003,"15min":0.505},"max":{"1min":1.1160000000000001,"5min":1.1699999999999999,"15min":1.3600000000000001},"last":0.871}]}]},{"osd":5,"up_from":13,"seq":55834575291,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1083868,"kb_used_data":35272,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93287972,"statfs":{"total":96636764160,"available":95526883328,"internally_reserved":1073741824,"allocated":36118528,"data_stored":34655064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 11:49:55 2021","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":0.83199999999999996,"15min":0.81999999999999995},"min":{"1min":0.35799999999999998,"5min":0.35699999999999998,"15min":0.34999999999999998},"max":{"1min":1.3580000000000001,"5min":1.3580000000000001,"15min":1.581},"last":0.84899999999999998},{"interface":"front","average":{"1min":0.84099999999999997,"5min":0.79800000000000004,"15min":0.80100000000000005},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.35799999999999998},"max":{"1min":1.288,"5min":1.288,"15min":1.3939999999999999},"last":0.88500000000000001}]},{"osd":1,"last update":"Wed Feb 10 11:49:55 2021","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.81200000000000006,"15min":0.81799999999999995},"min":{"1min":0.375,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.2490000000000001,"5min":1.2490000000000001,"15min":1.3540000000000001},"last":0.86599999999999999},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.81599999999999995,"15min":0.82499999999999996},"min":{"1min":0.376,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":1.097,"5min":1.2509999999999999,"15min":1.319},"last":0.75600000000000001}]},{"osd":2,"last update":"Wed Feb 10 11:49:55 2021","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.80700000000000005,"15min":0.81499999999999995},"min":{"1min":0.42399999999999999,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.1439999999999999,"5min":1.258,"15min":1.2729999999999999},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.83199999999999996,"15min":0.84499999999999997},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.24,"5min":1.2769999999999999,"15min":1.2769999999999999},"last":0.81799999999999995}]},{"osd":3,"last update":"Wed Feb 10 11:49:55 2021","interfaces":[{"interface":"back","average":{"1min":0.83599999999999997,"5min":0.85499999999999998,"15min":0.873},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.264,"5min":1.323,"15min":1.464},"last":0.80100000000000005},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.85299999999999998,"15min":0.875},"min":{"1min":0.57599999999999996,"5min":0.52400000000000002,"15min":0.44900000000000001},"max":{"1min":1.0960000000000001,"5min":1.2549999999999999,"15min":1.3520000000000001},"last":0.77200000000000002}]},{"osd":4,"last update":"Wed Feb 10 11:49:55 2021","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.68400000000000005,"15min":0.70899999999999996},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.117,"5min":1.1910000000000001,"15min":1.1910000000000001},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.67800000000000005,"15min":0.70899999999999996},"min":{"1min":0.45200000000000001,"5min":0.45000000000000001,"15min":0.40799999999999997},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":1.2490000000000001},"last":0.64800000000000002}]},{"osd":6,"last update":"Wed Feb 10 11:49:55 2021","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.72499999999999998,"15min":0.73099999999999998},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.38900000000000001},"max":{"1min":1.0169999999999999,"5min":1.22,"15min":1.22},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.76000000000000001,"15min":0.78600000000000003},"min":{"1min":0.52800000000000002,"5min":0.48699999999999999,"15min":0.40500000000000003},"max":{"1min":1.0009999999999999,"5min":1.2210000000000001,"15min":1.2210000000000001},"last":0.55100000000000005}]},{"osd":7,"last update":"Wed Feb 10 11:49:55 2021","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.755,"15min":0.78300000000000003},"min":{"1min":0.48799999999999999,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.93300000000000005,"5min":1.1459999999999999,"15min":1.1699999999999999},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.748,"5min":0.78600000000000003,"15min":0.81599999999999995},"min":{"1min":0.52800000000000002,"5min":0.47099999999999997,"15min":0.46899999999999997},"max":{"1min":1.161,"5min":1.355,"15min":1.4710000000000001},"last":0.67900000000000005}]}]},{"osd":4,"up_from":13,"seq":55834575292,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1074528,"kb_used_data":25944,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93297312,"statfs":{"total":96636764160,"available":95536447488,"internally_reserved":1073741824,"allocated":26566656,"data_stored":25103257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 11:50:10 2021","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":0.81000000000000005,"15min":0.81100000000000005},"min":{"1min":0.40100000000000002,"5min":0.32300000000000001,"15min":0.317},"max":{"1min":1.212,"5min":1.3660000000000001,"15min":1.5840000000000001},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.79600000000000004,"15min":0.78800000000000003},"min":{"1min":0.46899999999999997,"5min":0.40799999999999997,"15min":0.38100000000000001},"max":{"1min":1.0740000000000001,"5min":1.2909999999999999,"15min":1.2909999999999999},"last":0.41499999999999998}]},{"osd":1,"last update":"Wed Feb 10 11:50:10 2021","interfaces":[{"interface":"back","average":{"1min":0.81100000000000005,"5min":0.76000000000000001,"15min":0.77800000000000002},"min":{"1min":0.47699999999999998,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.095,"5min":1.2110000000000001,"15min":1.2689999999999999},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.84399999999999997,"5min":0.85299999999999998,"15min":0.84699999999999998},"min":{"1min":0.55700000000000005,"5min":0.48399999999999999,"15min":0.40000000000000002},"max":{"1min":1.091,"5min":1.3180000000000001,"15min":1.319},"last":0.379}]},{"osd":2,"last update":"Wed Feb 10 11:50:10 2021","interfaces":[{"interface":"back","average":{"1min":1.024,"5min":1.0089999999999999,"15min":1.0069999999999999},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.46899999999999997},"max":{"1min":1.353,"5min":1.4350000000000001,"15min":1.635},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.98399999999999999,"5min":0.91200000000000003,"15min":0.92200000000000004},"min":{"1min":0.67800000000000005,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.3120000000000001,"5min":1.327,"15min":1.4690000000000001},"last":0.78000000000000003}]},{"osd":3,"last update":"Wed Feb 10 11:50:10 2021","interfaces":[{"interface":"back","average":{"1min":1.03,"5min":1.008,"15min":0.998},"min":{"1min":0.67000000000000004,"5min":0.53800000000000003,"15min":0.48099999999999998},"max":{"1min":1.355,"5min":1.46,"15min":1.5569999999999999},"last":0.73699999999999999},{"interface":"front","average":{"1min":1.04,"5min":0.97099999999999997,"15min":0.94399999999999995},"min":{"1min":0.755,"5min":0.498,"15min":0.48899999999999999},"max":{"1min":1.325,"5min":1.379,"15min":1.3979999999999999},"last":0.65500000000000003}]},{"osd":5,"last update":"Wed Feb 10 11:50:10 2021","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.73599999999999999,"15min":0.73399999999999999},"min":{"1min":0.438,"5min":0.436,"15min":0.36799999999999999},"max":{"1min":1.095,"5min":1.1799999999999999,"15min":1.292},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.745,"5min":0.73199999999999998,"15min":0.70899999999999996},"min":{"1min":0.52500000000000002,"5min":0.46899999999999997,"15min":0.40999999999999998},"max":{"1min":1.3540000000000001,"5min":1.3540000000000001,"15min":1.3540000000000001},"last":0.55400000000000005}]},{"osd":6,"last update":"Wed Feb 10 11:50:10 2021","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.71999999999999997,"15min":0.73599999999999999},"min":{"1min":0.503,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.105,"5min":1.105,"15min":1.22},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.81299999999999994,"5min":0.82299999999999995,"15min":0.82099999999999995},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.43099999999999999},"max":{"1min":1.1479999999999999,"5min":1.321,"15min":1.5149999999999999},"last":0.63600000000000001}]},{"osd":7,"last update":"Wed Feb 10 11:50:10 2021","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.79900000000000004,"15min":0.77300000000000002},"min":{"1min":0.61199999999999999,"5min":0.56499999999999995,"15min":0.40300000000000002},"max":{"1min":1.256,"5min":1.256,"15min":1.256},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.80300000000000005,"15min":0.80200000000000005},"min":{"1min":0.628,"5min":0.58099999999999996,"15min":0.52100000000000002},"max":{"1min":1.163,"5min":1.169,"15min":1.169},"last":0.68300000000000005}]}]},{"osd":0,"up_from":13,"seq":55834575291,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1061324,"kb_used_data":12740,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93310516,"statfs":{"total":96636764160,"available":95549968384,"internally_reserved":1073741824,"allocated":13045760,"data_stored":11586266,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Feb 10 11:49:56 2021","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.49299999999999999,"15min":0.48399999999999999},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":1,"5min":1,"15min":1},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.496,"5min":0.48299999999999998,"15min":0.47999999999999998},"min":{"1min":0.34599999999999997,"5min":0.32500000000000001,"15min":0.313},"max":{"1min":0.68000000000000005,"5min":0.73999999999999999,"15min":0.88800000000000001},"last":0.60299999999999998}]},{"osd":2,"last update":"Wed Feb 10 11:49:56 2021","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.64500000000000002,"15min":0.622},"min":{"1min":0.40999999999999998,"5min":0.317,"15min":0.317},"max":{"1min":0.871,"5min":0.95999999999999996,"15min":1.1719999999999999},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.64900000000000002,"15min":0.64400000000000002},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":1.125},"last":0.73299999999999998}]},{"osd":3,"last update":"Wed Feb 10 11:49:56 2021","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.52200000000000002,"15min":0.51900000000000002},"min":{"1min":0.38400000000000001,"5min":0.35299999999999998,"15min":0.34200000000000003},"max":{"1min":0.71099999999999997,"5min":0.83599999999999997,"15min":0.97199999999999998},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.625,"15min":0.60899999999999999},"min":{"1min":0.46400000000000002,"5min":0.41899999999999998,"15min":0.318},"max":{"1min":0.77200000000000002,"5min":0.97099999999999997,"15min":0.99199999999999999},"last":0.71999999999999997}]},{"osd":4,"last update":"Wed Feb 10 11:49:56 2021","interfaces":[{"interface":"back","average":{"1min":0.98199999999999998,"5min":0.90200000000000002,"15min":0.92200000000000004},"min":{"1min":0.56399999999999995,"5min":0.52400000000000002,"15min":0.45400000000000001},"max":{"1min":1.3100000000000001,"5min":1.3320000000000001,"15min":1.365},"last":1.2689999999999999},{"interface":"front","average":{"1min":0.96899999999999997,"5min":0.94299999999999995,"15min":0.94699999999999995},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.46600000000000003},"max":{"1min":1.3,"5min":1.3,"15min":1.393},"last":1.143}]},{"osd":5,"last update":"Wed Feb 10 11:49:56 2021","interfaces":[{"interface":"back","average":{"1min":1.0720000000000001,"5min":1.0229999999999999,"15min":1.071},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.5169999999999999,"5min":1.5169999999999999,"15min":1.8660000000000001},"last":1.3560000000000001},{"interface":"front","average":{"1min":1.123,"5min":1.048,"15min":1.0940000000000001},"min":{"1min":0.64800000000000002,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":1.5389999999999999,"5min":1.6000000000000001,"15min":1.6000000000000001},"last":1.4099999999999999}]},{"osd":6,"last update":"Wed Feb 10 11:49:56 2021","interfaces":[{"interface":"back","average":{"1min":1.046,"5min":1.0409999999999999,"15min":1.077},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.4399999999999999,"5min":1.4399999999999999,"15min":1.542},"last":1.4870000000000001},{"interface":"front","average":{"1min":1.099,"5min":1.046,"15min":1.093},"min":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.61699999999999999},"max":{"1min":1.548,"5min":1.548,"15min":1.792},"last":1.5189999999999999}]},{"osd":7,"last update":"Wed Feb 10 11:49:56 2021","interfaces":[{"interface":"back","average":{"1min":1.087,"5min":1.054,"15min":1.095},"min":{"1min":0.58299999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.494,"5min":1.548,"15min":1.6559999999999999},"last":1.2929999999999999},{"interface":"front","average":{"1min":1.101,"5min":1.0489999999999999,"15min":1.0820000000000001},"min":{"1min":0.65500000000000003,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":1.4339999999999999,"5min":1.4990000000000001,"15min":1.4990000000000001},"last":1.327}]}]},{"osd":1,"up_from":13,"seq":55834575290,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1092052,"kb_used_data":43468,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93279788,"statfs":{"total":96636764160,"available":95518502912,"internally_reserved":1073741824,"allocated":44511232,"data_stored":43043903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 11:50:08 2021","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.32700000000000001,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.876},"last":0.443},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.47999999999999998,"15min":0.49199999999999999},"min":{"1min":0.374,"5min":0.308,"15min":0.308},"max":{"1min":0.75800000000000001,"5min":0.82499999999999996,"15min":1.1279999999999999},"last":0.46000000000000002}]},{"osd":2,"last update":"Wed Feb 10 11:50:08 2021","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.61799999999999999,"15min":0.63},"min":{"1min":0.44600000000000001,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.753,"5min":0.96399999999999997,"15min":1.4319999999999999},"last":0.621},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.57799999999999996,"15min":0.59399999999999997},"min":{"1min":0.39000000000000001,"5min":0.378,"15min":0.313},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.95099999999999996},"last":0.64700000000000002}]},{"osd":3,"last update":"Wed Feb 10 11:50:08 2021","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.54700000000000004,"15min":0.54200000000000004},"min":{"1min":0.38800000000000001,"5min":0.315,"15min":0.315},"max":{"1min":0.68799999999999994,"5min":1,"15min":1},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.61299999999999999,"15min":0.62},"min":{"1min":0.34399999999999997,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.83299999999999996,"5min":1.0429999999999999,"15min":1.22},"last":0.52700000000000002}]},{"osd":4,"last update":"Wed Feb 10 11:50:08 2021","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.875,"15min":0.90300000000000002},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":1.1910000000000001,"5min":1.498,"15min":1.498},"last":0.85399999999999998},{"interface":"front","average":{"1min":0.85999999999999999,"5min":0.89200000000000002,"15min":0.91700000000000004},"min":{"1min":0.59199999999999997,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.2310000000000001,"5min":1.3380000000000001,"15min":1.3540000000000001},"last":0.97199999999999998}]},{"osd":5,"last update":"Wed Feb 10 11:50:08 2021","interfaces":[{"interface":"back","average":{"1min":0.99099999999999999,"5min":1.0209999999999999,"15min":1.052},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.48199999999999998},"max":{"1min":1.544,"5min":1.544,"15min":1.6399999999999999},"last":1.0449999999999999},{"interface":"front","average":{"1min":0.98799999999999999,"5min":1.0429999999999999,"15min":1.071},"min":{"1min":0.65100000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.4279999999999999,"5min":1.446,"15min":1.4870000000000001},"last":1.103}]},{"osd":6,"last update":"Wed Feb 10 11:50:08 2021","interfaces":[{"interface":"back","average":{"1min":0.92300000000000004,"5min":0.95699999999999996,"15min":0.95599999999999996},"min":{"1min":0.66700000000000004,"5min":0.51800000000000002,"15min":0.496},"max":{"1min":1.2549999999999999,"5min":1.421,"15min":1.5349999999999999},"last":1.0289999999999999},{"interface":"front","average":{"1min":0.871,"5min":0.91400000000000003,"15min":0.93200000000000005},"min":{"1min":0.60899999999999999,"5min":0.56100000000000005,"15min":0.498},"max":{"1min":1.3029999999999999,"5min":1.3029999999999999,"15min":1.3919999999999999},"last":0.94899999999999995}]},{"osd":7,"last update":"Wed Feb 10 11:50:08 2021","interfaces":[{"interface":"back","average":{"1min":1.008,"5min":1.0389999999999999,"15min":1.0449999999999999},"min":{"1min":0.751,"5min":0.59099999999999997,"15min":0.52100000000000002},"max":{"1min":1.444,"5min":1.444,"15min":1.4830000000000001},"last":1.0920000000000001},{"interface":"front","average":{"1min":1.0529999999999999,"5min":1.079,"15min":1.097},"min":{"1min":0.81000000000000005,"5min":0.627,"15min":0.627},"max":{"1min":1.4330000000000001,"5min":1.4670000000000001,"15min":1.615},"last":1.159}]}]},{"osd":2,"up_from":13,"seq":55834575291,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1122780,"kb_used_data":74196,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93249060,"statfs":{"total":96636764160,"available":95487037440,"internally_reserved":1073741824,"allocated":75976704,"data_stored":74501655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 11:49:52 2021","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.46899999999999997,"15min":0.47799999999999998},"min":{"1min":0.35499999999999998,"5min":0.308,"15min":0.308},"max":{"1min":0.70799999999999996,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.502,"15min":0.51000000000000001},"min":{"1min":0.34399999999999997,"5min":0.316,"15min":0.316},"max":{"1min":0.70799999999999996,"5min":0.92600000000000005,"15min":1.0329999999999999},"last":0.53400000000000003}]},{"osd":1,"last update":"Wed Feb 10 11:49:52 2021","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.51600000000000001,"15min":0.51700000000000002},"min":{"1min":0.38900000000000001,"5min":0.34000000000000002,"15min":0.32300000000000001},"max":{"1min":0.60299999999999998,"5min":0.94499999999999995,"15min":1.133},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.57299999999999995,"15min":0.57699999999999996},"min":{"1min":0.40799999999999997,"5min":0.375,"15min":0.375},"max":{"1min":0.84499999999999997,"5min":0.84699999999999998,"15min":0.93400000000000005},"last":0.65300000000000002}]},{"osd":3,"last update":"Wed Feb 10 11:49:52 2021","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.58399999999999996,"15min":0.58799999999999997},"min":{"1min":0.48299999999999998,"5min":0.40300000000000002,"15min":0.39900000000000002},"max":{"1min":0.76000000000000001,"5min":0.90500000000000003,"15min":0.96099999999999997},"last":0.627},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.54600000000000004,"15min":0.56399999999999995},"min":{"1min":0.42199999999999999,"5min":0.36599999999999999,"15min":0.34599999999999997},"max":{"1min":0.77300000000000002,"5min":0.89900000000000002,"15min":0.94699999999999995},"last":0.75}]},{"osd":4,"last update":"Wed Feb 10 11:49:52 2021","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":0.91700000000000004,"15min":0.94099999999999995},"min":{"1min":0.68999999999999995,"5min":0.53100000000000003,"15min":0.52400000000000002},"max":{"1min":1.278,"5min":1.331,"15min":1.3919999999999999},"last":0.88500000000000001},{"interface":"front","average":{"1min":0.95599999999999996,"5min":0.94499999999999995,"15min":0.96699999999999997},"min":{"1min":0.60499999999999998,"5min":0.57799999999999996,"15min":0.53300000000000003},"max":{"1min":1.304,"5min":1.304,"15min":1.4430000000000001},"last":0.83699999999999997}]},{"osd":5,"last update":"Wed Feb 10 11:49:52 2021","interfaces":[{"interface":"back","average":{"1min":0.95599999999999996,"5min":0.94799999999999995,"15min":0.97199999999999998},"min":{"1min":0.54400000000000004,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.206,"5min":1.393,"15min":1.4059999999999999},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.96599999999999997,"5min":0.94199999999999995,"15min":0.95399999999999996},"min":{"1min":0.65700000000000003,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.3300000000000001,"5min":1.359,"15min":1.369},"last":1.026}]},{"osd":6,"last update":"Wed Feb 10 11:49:52 2021","interfaces":[{"interface":"back","average":{"1min":0.97799999999999998,"5min":0.94699999999999995,"15min":0.98199999999999998},"min":{"1min":0.62,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.361,"5min":1.361,"15min":1.462},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.96299999999999997,"5min":0.97199999999999998,"15min":0.98799999999999999},"min":{"1min":0.68200000000000005,"5min":0.57099999999999995,"15min":0.54900000000000004},"max":{"1min":1.288,"5min":1.296,"15min":1.4319999999999999},"last":0.80300000000000005}]},{"osd":7,"last update":"Wed Feb 10 11:49:52 2021","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":0.98099999999999998,"15min":0.999},"min":{"1min":0.73199999999999998,"5min":0.64900000000000002,"15min":0.58699999999999997},"max":{"1min":1.2290000000000001,"5min":1.4079999999999999,"15min":1.4079999999999999},"last":0.94899999999999995},{"interface":"front","average":{"1min":1.012,"5min":1.01,"15min":1.012},"min":{"1min":0.72899999999999998,"5min":0.64200000000000002,"15min":0.63800000000000001},"max":{"1min":1.3420000000000001,"5min":1.395,"15min":1.4350000000000001},"last":0.86299999999999999}]}]},{"osd":3,"up_from":13,"seq":55834575292,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1079756,"kb_used_data":31172,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93292084,"statfs":{"total":96636764160,"available":95531094016,"internally_reserved":1073741824,"allocated":31920128,"data_stored":30460737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 11:50:09 2021","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.498,"15min":0.496},"min":{"1min":0.35199999999999998,"5min":0.311,"15min":0.311},"max":{"1min":0.66300000000000003,"5min":1,"15min":1},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.498,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.40400000000000003,"5min":0.315,"15min":0.307},"max":{"1min":0.66900000000000004,"5min":0.83099999999999996,"15min":1.216},"last":0.38600000000000001}]},{"osd":1,"last update":"Wed Feb 10 11:50:09 2021","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.49199999999999999,"15min":0.502},"min":{"1min":0.34999999999999998,"5min":0.32900000000000001,"15min":0.30299999999999999},"max":{"1min":0.63800000000000001,"5min":0.98399999999999999,"15min":1.0740000000000001},"last":0.443},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.55900000000000005,"15min":0.56799999999999995},"min":{"1min":0.39800000000000002,"5min":0.34499999999999997,"15min":0.28299999999999997},"max":{"1min":0.76400000000000001,"5min":0.85799999999999998,"15min":1.02},"last":0.55600000000000005}]},{"osd":2,"last update":"Wed Feb 10 11:50:09 2021","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.624,"15min":0.64000000000000001},"min":{"1min":0.42899999999999999,"5min":0.41499999999999998,"15min":0.33500000000000002},"max":{"1min":0.85599999999999998,"5min":0.88,"15min":1.081},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.77800000000000002,"15min":0.77400000000000002},"min":{"1min":0.54600000000000004,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.075,"5min":1.107,"15min":1.2030000000000001},"last":0.60699999999999998}]},{"osd":4,"last update":"Wed Feb 10 11:50:09 2021","interfaces":[{"interface":"back","average":{"1min":0.90900000000000003,"5min":0.90100000000000002,"15min":0.90000000000000002},"min":{"1min":0.53600000000000003,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.181,"5min":1.3839999999999999,"15min":1.4830000000000001},"last":1.0589999999999999},{"interface":"front","average":{"1min":0.92700000000000005,"5min":0.91500000000000004,"15min":0.92700000000000005},"min":{"1min":0.51300000000000001,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.3620000000000001,"5min":1.4890000000000001,"15min":1.4890000000000001},"last":1.075}]},{"osd":5,"last update":"Wed Feb 10 11:50:09 2021","interfaces":[{"interface":"back","average":{"1min":1.095,"5min":1.0760000000000001,"15min":1.0649999999999999},"min":{"1min":0.73999999999999999,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.5069999999999999,"5min":1.508,"15min":1.538},"last":1.1839999999999999},{"interface":"front","average":{"1min":1.0860000000000001,"5min":1.0589999999999999,"15min":1.048},"min":{"1min":0.66400000000000003,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.6419999999999999,"5min":1.6419999999999999,"15min":1.6419999999999999},"last":1.198}]},{"osd":6,"last update":"Wed Feb 10 11:50:09 2021","interfaces":[{"interface":"back","average":{"1min":0.95299999999999996,"5min":0.93500000000000005,"15min":0.95499999999999996},"min":{"1min":0.60599999999999998,"5min":0.501,"15min":0.501},"max":{"1min":1.2509999999999999,"5min":1.3939999999999999,"15min":1.452},"last":1.0089999999999999},{"interface":"front","average":{"1min":0.96199999999999997,"5min":0.95799999999999996,"15min":0.97599999999999998},"min":{"1min":0.57899999999999996,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.323,"5min":1.323,"15min":1.4079999999999999},"last":1.028}]},{"osd":7,"last update":"Wed Feb 10 11:50:09 2021","interfaces":[{"interface":"back","average":{"1min":1.0589999999999999,"5min":1.0549999999999999,"15min":1.0409999999999999},"min":{"1min":0.58999999999999997,"5min":0.58699999999999997,"15min":0.55100000000000005},"max":{"1min":1.5309999999999999,"5min":1.5309999999999999,"15min":1.5309999999999999},"last":1.2689999999999999},{"interface":"front","average":{"1min":1.1040000000000001,"5min":1.0629999999999999,"15min":1.071},"min":{"1min":0.68700000000000006,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.518,"5min":1.556,"15min":1.556},"last":1.175}]}]}]}} 2021-02-10T11:50:38.272 INFO:tasks.ceph:pgid 43.6 last_scrub_stamp 2021-02-10 11:49:03.842471 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=49, tm_sec=3, tm_wday=2, tm_yday=41, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=50, tm_sec=27, tm_wday=2, tm_yday=41, tm_isdst=0) 2021-02-10T11:50:38.272 INFO:tasks.ceph:pgid 42.7 last_scrub_stamp 2021-02-10 11:49:02.834883 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=49, tm_sec=2, tm_wday=2, tm_yday=41, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=50, tm_sec=27, tm_wday=2, tm_yday=41, tm_isdst=0) 2021-02-10T11:50:38.273 INFO:tasks.ceph:pgid 43.7 last_scrub_stamp 2021-02-10 11:49:03.842471 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=49, tm_sec=3, tm_wday=2, tm_yday=41, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=50, tm_sec=27, tm_wday=2, tm_yday=41, tm_isdst=0) 2021-02-10T11:50:38.273 INFO:tasks.ceph:pgid 42.6 last_scrub_stamp 2021-02-10 11:49:02.834883 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=49, tm_sec=2, tm_wday=2, tm_yday=41, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=50, tm_sec=27, tm_wday=2, tm_yday=41, tm_isdst=0) 2021-02-10T11:50:38.273 INFO:tasks.ceph:pgid 43.5 last_scrub_stamp 2021-02-10 11:49:03.842471 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=49, tm_sec=3, tm_wday=2, tm_yday=41, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=50, tm_sec=27, tm_wday=2, tm_yday=41, tm_isdst=0) 2021-02-10T11:50:38.273 INFO:tasks.ceph:pgid 42.4 last_scrub_stamp 2021-02-10 11:49:02.834883 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=49, tm_sec=2, tm_wday=2, tm_yday=41, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=50, tm_sec=27, tm_wday=2, tm_yday=41, tm_isdst=0) 2021-02-10T11:50:38.273 INFO:tasks.ceph:pgid 43.2 last_scrub_stamp 2021-02-10 11:49:03.842471 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=49, tm_sec=3, tm_wday=2, tm_yday=41, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=50, tm_sec=27, tm_wday=2, tm_yday=41, tm_isdst=0) 2021-02-10T11:50:38.273 INFO:tasks.ceph:pgid 42.3 last_scrub_stamp 2021-02-10 11:49:02.834883 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=49, tm_sec=2, tm_wday=2, tm_yday=41, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=50, tm_sec=27, tm_wday=2, tm_yday=41, tm_isdst=0) 2021-02-10T11:50:38.274 INFO:tasks.ceph:pgid 43.3 last_scrub_stamp 2021-02-10 11:49:03.842471 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=49, tm_sec=3, tm_wday=2, tm_yday=41, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=50, tm_sec=27, tm_wday=2, tm_yday=41, tm_isdst=0) 2021-02-10T11:50:38.274 INFO:tasks.ceph:pgid 42.2 last_scrub_stamp 2021-02-10 11:49:02.834883 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=49, tm_sec=2, tm_wday=2, tm_yday=41, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=50, tm_sec=27, tm_wday=2, tm_yday=41, tm_isdst=0) 2021-02-10T11:50:38.274 INFO:tasks.ceph:pgid 43.0 last_scrub_stamp 2021-02-10 11:49:03.842471 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=49, tm_sec=3, tm_wday=2, tm_yday=41, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=50, tm_sec=27, tm_wday=2, tm_yday=41, tm_isdst=0) 2021-02-10T11:50:38.274 INFO:tasks.ceph:pgid 42.1 last_scrub_stamp 2021-02-10 11:49:02.834883 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=49, tm_sec=2, tm_wday=2, tm_yday=41, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=50, tm_sec=27, tm_wday=2, tm_yday=41, tm_isdst=0) 2021-02-10T11:50:38.274 INFO:tasks.ceph:pgid 43.1 last_scrub_stamp 2021-02-10 11:49:03.842471 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=49, tm_sec=3, tm_wday=2, tm_yday=41, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=50, tm_sec=27, tm_wday=2, tm_yday=41, tm_isdst=0) 2021-02-10T11:50:38.274 INFO:tasks.ceph:pgid 42.0 last_scrub_stamp 2021-02-10 11:49:02.834883 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=49, tm_sec=2, tm_wday=2, tm_yday=41, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=50, tm_sec=27, tm_wday=2, tm_yday=41, tm_isdst=0) 2021-02-10T11:50:38.275 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2021-02-10 11:13:57.948661 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=13, tm_sec=57, tm_wday=2, tm_yday=41, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=50, tm_sec=27, tm_wday=2, tm_yday=41, tm_isdst=0) 2021-02-10T11:50:38.275 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2021-02-10 11:13:57.948661 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=13, tm_sec=57, tm_wday=2, tm_yday=41, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=50, tm_sec=27, tm_wday=2, tm_yday=41, tm_isdst=0) 2021-02-10T11:50:38.275 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2021-02-10 11:13:57.948661 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=13, tm_sec=57, tm_wday=2, tm_yday=41, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=50, tm_sec=27, tm_wday=2, tm_yday=41, tm_isdst=0) 2021-02-10T11:50:38.275 INFO:tasks.ceph:pgid 1.4 last_scrub_stamp 2021-02-10 11:13:57.948661 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=13, tm_sec=57, tm_wday=2, tm_yday=41, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=50, tm_sec=27, tm_wday=2, tm_yday=41, tm_isdst=0) 2021-02-10T11:50:38.275 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2021-02-10 11:13:57.948661 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=13, tm_sec=57, tm_wday=2, tm_yday=41, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=50, tm_sec=27, tm_wday=2, tm_yday=41, tm_isdst=0) 2021-02-10T11:50:38.275 INFO:tasks.ceph:pgid 43.4 last_scrub_stamp 2021-02-10 11:49:03.842471 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=49, tm_sec=3, tm_wday=2, tm_yday=41, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=50, tm_sec=27, tm_wday=2, tm_yday=41, tm_isdst=0) 2021-02-10T11:50:38.275 INFO:tasks.ceph:pgid 42.5 last_scrub_stamp 2021-02-10 11:49:02.834883 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=49, tm_sec=2, tm_wday=2, tm_yday=41, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=50, tm_sec=27, tm_wday=2, tm_yday=41, tm_isdst=0) 2021-02-10T11:50:38.276 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2021-02-10 11:13:57.948661 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=13, tm_sec=57, tm_wday=2, tm_yday=41, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=50, tm_sec=27, tm_wday=2, tm_yday=41, tm_isdst=0) 2021-02-10T11:50:38.276 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2021-02-10 11:13:57.948661 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=13, tm_sec=57, tm_wday=2, tm_yday=41, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=50, tm_sec=27, tm_wday=2, tm_yday=41, tm_isdst=0) 2021-02-10T11:50:38.276 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2021-02-10T11:50:58.277 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph pg dump --format=json 2021-02-10T11:50:58.708 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:58.687 7f284eff2700 1 Processor -- start 2021-02-10T11:50:58.708 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:58.687 7f284eff2700 1 -- start start 2021-02-10T11:50:58.708 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:58.688 7f284eff2700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f28480d0340 0x7f28480ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:58.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:58.688 7f284eff2700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f28480d1c70 0x7f28480efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:58.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:58.688 7f284eff2700 1 -- --> v1:172.21.2.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28480f0b10 con 0x7f28480d1870 2021-02-10T11:50:58.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:58.688 7f284eff2700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28480e5ed0 con 0x7f28480d1c70 2021-02-10T11:50:58.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:58.688 7f284eff2700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28481b0be0 con 0x7f28480d0340 2021-02-10T11:50:58.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:58.688 7f284d60f700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f28480d1c70 0x7f28480efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:58.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:58.688 7f284d60f700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7f28480d1c70 0x7f28480efc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:34248/0 (socket says 172.21.2.112:34248) 2021-02-10T11:50:58.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:58.688 7f284d60f700 1 -- 172.21.2.112:0/790191190 learned_addr learned my addr 172.21.2.112:0/790191190 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:50:58.709 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:58.688 7f284d60f700 1 -- 172.21.2.112:0/790191190 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f28480d0340 msgr2=0x7f28480ef640 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:50:58.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:58.688 7f284d60f700 1 --2- 172.21.2.112:0/790191190 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f28480d0340 0x7f28480ef640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:58.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:58.688 7f284d60f700 1 -- 172.21.2.112:0/790191190 >> v1:172.21.2.120:6789/0 conn(0x7f28480d1870 legacy=0x7f28481b03f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:50:58.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:58.688 7f284d60f700 1 -- 172.21.2.112:0/790191190 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28480f20a0 con 0x7f28480d1c70 2021-02-10T11:50:58.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:58.688 7f284d60f700 1 --2- 172.21.2.112:0/790191190 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f28480d1c70 0x7f28480efc60 secure :-1 s=READY pgs=3370 cs=0 l=1 rev1=1 rx=0x7f2840006f00 tx=0x7f2840009f10).ready entity=mon.0 client_cookie=f9c82c9d22b55061 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:58.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:58.689 7f28477fe700 1 -- 172.21.2.112:0/790191190 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f284000b3b0 con 0x7f28480d1c70 2021-02-10T11:50:58.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:58.689 7f28477fe700 1 -- 172.21.2.112:0/790191190 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f284000b730 con 0x7f28480d1c70 2021-02-10T11:50:58.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:58.689 7f28477fe700 1 -- 172.21.2.112:0/790191190 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f284000bc50 con 0x7f28480d1c70 2021-02-10T11:50:58.710 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:58.689 7f284eff2700 1 -- 172.21.2.112:0/790191190 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f28480d1c70 msgr2=0x7f28480efc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:58.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:58.689 7f284eff2700 1 --2- 172.21.2.112:0/790191190 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f28480d1c70 0x7f28480efc60 secure :-1 s=READY pgs=3370 cs=0 l=1 rev1=1 rx=0x7f2840006f00 tx=0x7f2840009f10).stop 2021-02-10T11:50:58.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:58.689 7f284eff2700 1 -- 172.21.2.112:0/790191190 shutdown_connections 2021-02-10T11:50:58.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:58.689 7f284eff2700 1 --2- 172.21.2.112:0/790191190 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f28480d1c70 0x7f28480efc60 unknown :-1 s=CLOSED pgs=3370 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:58.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:58.689 7f284eff2700 1 --2- 172.21.2.112:0/790191190 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f28480d0340 0x7f28480ef640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:58.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:58.689 7f284eff2700 1 -- 172.21.2.112:0/790191190 >> 172.21.2.112:0/790191190 conn(0x7f28480ce450 msgr2=0x7f28480d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:58.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:58.689 7f284eff2700 1 -- 172.21.2.112:0/790191190 shutdown_connections 2021-02-10T11:50:58.711 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:58.689 7f284eff2700 1 -- 172.21.2.112:0/790191190 wait complete. 2021-02-10T11:50:58.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:58.704 7f284eff2700 1 Processor -- start 2021-02-10T11:50:58.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:58.704 7f284eff2700 1 -- start start 2021-02-10T11:50:58.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:58.704 7f284eff2700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f28481abb30 0x7f2848187e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:58.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:58.704 7f284eff2700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f28480b48b0 0x7f284808c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:58.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:58.704 7f284eff2700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f28480b4cb0 0x7f284808ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:58.713 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:58.704 7f284eff2700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28481ac060 con 0x7f28480b48b0 2021-02-10T11:50:58.714 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:58.704 7f284eff2700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f284808d280 con 0x7f28480b4cb0 2021-02-10T11:50:58.714 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:58.704 7f284eff2700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f284818cb00 con 0x7f28481abb30 2021-02-10T11:50:58.714 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:58.705 7f2847fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f28480b48b0 0x7f284808c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:58.714 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:58.705 7f2847fff700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f28480b48b0 0x7f284808c460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:34250/0 (socket says 172.21.2.112:34250) 2021-02-10T11:50:58.714 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:58.705 7f2847fff700 1 -- 172.21.2.112:0/477972902 learned_addr learned my addr 172.21.2.112:0/477972902 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:50:58.714 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:58.705 7f284d60f700 1 --2- 172.21.2.112:0/477972902 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f28480b4cb0 0x7f284808ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:58.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:58.705 7f284ce0e700 1 --2- 172.21.2.112:0/477972902 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f28481abb30 0x7f2848187e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:58.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:58.705 7f2847fff700 1 -- 172.21.2.112:0/477972902 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f28481abb30 msgr2=0x7f2848187e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:58.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:58.705 7f2847fff700 1 --2- 172.21.2.112:0/477972902 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f28481abb30 0x7f2848187e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:58.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:58.705 7f2847fff700 1 -- 172.21.2.112:0/477972902 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f28480b4cb0 msgr2=0x7f284808ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:58.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:58.705 7f2847fff700 1 --2- 172.21.2.112:0/477972902 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f28480b4cb0 0x7f284808ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:58.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:58.705 7f2847fff700 1 -- 172.21.2.112:0/477972902 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f284818cce0 con 0x7f28480b48b0 2021-02-10T11:50:58.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:58.705 7f284ce0e700 1 --2- 172.21.2.112:0/477972902 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f28481abb30 0x7f2848187e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:50:58.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:58.705 7f284d60f700 1 --2- 172.21.2.112:0/477972902 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f28480b4cb0 0x7f284808ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:50:58.715 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:58.705 7f2847fff700 1 --2- 172.21.2.112:0/477972902 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f28480b48b0 0x7f284808c460 secure :-1 s=READY pgs=3371 cs=0 l=1 rev1=1 rx=0x7f283800b5c0 tx=0x7f283800b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:58.716 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:58.705 7f28457fa700 1 -- 172.21.2.112:0/477972902 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2838000c40 con 0x7f28480b48b0 2021-02-10T11:50:58.716 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:58.705 7f28457fa700 1 -- 172.21.2.112:0/477972902 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f283800bd50 con 0x7f28480b48b0 2021-02-10T11:50:58.716 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:58.705 7f28457fa700 1 -- 172.21.2.112:0/477972902 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f28380054d0 con 0x7f28480b48b0 2021-02-10T11:50:58.716 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:58.705 7f284eff2700 1 -- 172.21.2.112:0/477972902 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f284818cce0 con 0x7f28480b48b0 2021-02-10T11:50:58.716 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:58.705 7f284eff2700 1 -- 172.21.2.112:0/477972902 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f284818cce0 con 0x7f28480b48b0 2021-02-10T11:50:58.716 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:58.706 7f28457fa700 1 -- 172.21.2.112:0/477972902 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2838006240 con 0x7f28480b48b0 2021-02-10T11:50:58.716 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:58.706 7f28457fa700 1 --2- 172.21.2.112:0/477972902 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f282c0186c0 0x7f282c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:58.716 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:58.706 7f28457fa700 1 -- 172.21.2.112:0/477972902 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f2838032410 con 0x7f28480b48b0 2021-02-10T11:50:58.716 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:58.707 7f284ce0e700 1 --2- 172.21.2.112:0/477972902 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f282c0186c0 0x7f282c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:58.717 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:58.708 7f284ce0e700 1 --2- 172.21.2.112:0/477972902 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f282c0186c0 0x7f282c01abc0 crc :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:58.742 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:58.738 7f284eff2700 1 -- 172.21.2.112:0/477972902 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28480ebc00 con 0x7f28480b48b0 2021-02-10T11:50:58.745 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:58.740 7f28457fa700 1 -- 172.21.2.112:0/477972902 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f283802ef10 con 0x7f28480b48b0 2021-02-10T11:50:58.879 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:58.874 7f284eff2700 1 -- 172.21.2.112:0/477972902 --> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] -- command(tid 0: {"prefix": "pg dump", "target": ["mgr", ""], "format": "json"}) v1 -- 0x7f284808d280 con 0x7f282c0186c0 2021-02-10T11:50:58.882 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:58.878 7f28457fa700 1 -- 172.21.2.112:0/477972902 <== mgr.4100 v2:172.21.2.120:6800/34990 1 ==== command_reply(tid 0: 0 dumped all) v1 ==== 18+0+101087 (crc 0 0 0) 0x7f2830000960 con 0x7f282c0186c0 2021-02-10T11:50:58.883 INFO:teuthology.orchestra.run.gibba012.stdout: 2021-02-10T11:50:58.884 INFO:teuthology.orchestra.run.gibba012.stderr:dumped all 2021-02-10T11:50:58.885 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:58.879 7f284eff2700 1 -- 172.21.2.112:0/477972902 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f282c0186c0 msgr2=0x7f282c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:58.885 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:58.879 7f284eff2700 1 --2- 172.21.2.112:0/477972902 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f282c0186c0 0x7f282c01abc0 crc :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:58.885 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:58.879 7f284eff2700 1 -- 172.21.2.112:0/477972902 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f28480b48b0 msgr2=0x7f284808c460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:58.885 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:58.879 7f284eff2700 1 --2- 172.21.2.112:0/477972902 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f28480b48b0 0x7f284808c460 secure :-1 s=READY pgs=3371 cs=0 l=1 rev1=1 rx=0x7f283800b5c0 tx=0x7f283800b960).stop 2021-02-10T11:50:58.885 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:58.880 7f284eff2700 1 -- 172.21.2.112:0/477972902 shutdown_connections 2021-02-10T11:50:58.886 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:58.880 7f284eff2700 1 --2- 172.21.2.112:0/477972902 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7f282c0186c0 0x7f282c01abc0 unknown :-1 s=CLOSED pgs=1516 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:58.886 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:58.880 7f284eff2700 1 --2- 172.21.2.112:0/477972902 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7f28480b4cb0 0x7f284808ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:58.886 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:58.880 7f284eff2700 1 --2- 172.21.2.112:0/477972902 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7f28480b48b0 0x7f284808c460 unknown :-1 s=CLOSED pgs=3371 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:58.886 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:58.880 7f284eff2700 1 --2- 172.21.2.112:0/477972902 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7f28481abb30 0x7f2848187e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:58.886 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:58.880 7f284eff2700 1 -- 172.21.2.112:0/477972902 >> 172.21.2.112:0/477972902 conn(0x7f284819f880 msgr2=0x7f28480ebe30 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:58.886 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:58.880 7f284eff2700 1 -- 172.21.2.112:0/477972902 shutdown_connections 2021-02-10T11:50:58.886 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:58.880 7f284eff2700 1 -- 172.21.2.112:0/477972902 wait complete. 2021-02-10T11:50:58.892 INFO:teuthology.orchestra.run.gibba012.stdout:{"pg_ready":true,"pg_map":{"version":1314,"stamp":"2021-02-10 11:50:58.279875","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":135153129,"num_objects":154,"num_object_clones":1,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":154,"num_whiteouts":0,"num_read":65,"num_read_kb":65537,"num_write":432,"num_write_kb":132086,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5006,"num_omap_keys":18,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":505,"ondisk_log_size":505,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"kb":754974720,"kb_used":8672708,"kb_used_data":284024,"kb_used_omap":0,"kb_used_meta":8388608,"kb_avail":746302012,"statfs":{"total":773094113280,"available":764213260288,"internally_reserved":8589934592,"allocated":290840576,"data_stored":279110034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8589934592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003735"},"pg_stats":[{"pgid":"43.6","version":"198'44","reported_seq":"65","reported_epoch":"198","state":"active+clean","last_fresh":"2021-02-10 11:50:50.765779","last_change":"2021-02-10 11:50:50.765807","last_active":"2021-02-10 11:50:50.765779","last_peered":"2021-02-10 11:50:50.765779","last_clean":"2021-02-10 11:50:50.765779","last_became_active":"2021-02-10 11:49:04.485856","last_became_peered":"2021-02-10 11:49:04.485856","last_unstale":"2021-02-10 11:50:50.765779","last_undegraded":"2021-02-10 11:50:50.765779","last_fullsized":"2021-02-10 11:50:50.765779","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"198'44","last_scrub_stamp":"2021-02-10 11:50:50.765775","last_deep_scrub":"198'44","last_deep_scrub_stamp":"2021-02-10 11:50:50.765775","last_clean_scrub_stamp":"2021-02-10 11:50:50.765775","log_size":44,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":23068672,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":9,"num_read_kb":9216,"num_write":22,"num_write_kb":22528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.7","version":"198'12","reported_seq":"26","reported_epoch":"198","state":"active+clean","last_fresh":"2021-02-10 11:50:42.051005","last_change":"2021-02-10 11:50:42.051027","last_active":"2021-02-10 11:50:42.051005","last_peered":"2021-02-10 11:50:42.051005","last_clean":"2021-02-10 11:50:42.051005","last_became_active":"2021-02-10 11:49:03.849100","last_became_peered":"2021-02-10 11:49:03.849100","last_unstale":"2021-02-10 11:50:42.051005","last_undegraded":"2021-02-10 11:50:42.051005","last_fullsized":"2021-02-10 11:50:42.051005","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"198'12","last_scrub_stamp":"2021-02-10 11:50:42.050988","last_deep_scrub":"198'12","last_deep_scrub_stamp":"2021-02-10 11:50:42.050988","last_clean_scrub_stamp":"2021-02-10 11:50:42.050988","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":566,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"43.7","version":"198'21","reported_seq":"40","reported_epoch":"198","state":"active+clean","last_fresh":"2021-02-10 11:50:38.365569","last_change":"2021-02-10 11:50:38.365635","last_active":"2021-02-10 11:50:38.365569","last_peered":"2021-02-10 11:50:38.365569","last_clean":"2021-02-10 11:50:38.365569","last_became_active":"2021-02-10 11:49:04.485927","last_became_peered":"2021-02-10 11:49:04.485927","last_unstale":"2021-02-10 11:50:38.365569","last_undegraded":"2021-02-10 11:50:38.365569","last_fullsized":"2021-02-10 11:50:38.365569","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"198'21","last_scrub_stamp":"2021-02-10 11:50:38.365565","last_deep_scrub":"198'21","last_deep_scrub_stamp":"2021-02-10 11:50:38.365565","last_clean_scrub_stamp":"2021-02-10 11:50:38.365565","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14680064,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":7,"num_read_kb":7168,"num_write":14,"num_write_kb":14336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.6","version":"198'17","reported_seq":"31","reported_epoch":"198","state":"active+clean","last_fresh":"2021-02-10 11:50:43.654235","last_change":"2021-02-10 11:50:43.654259","last_active":"2021-02-10 11:50:43.654235","last_peered":"2021-02-10 11:50:43.654235","last_clean":"2021-02-10 11:50:43.654235","last_became_active":"2021-02-10 11:49:03.848469","last_became_peered":"2021-02-10 11:49:03.848469","last_unstale":"2021-02-10 11:50:43.654235","last_undegraded":"2021-02-10 11:50:43.654235","last_fullsized":"2021-02-10 11:50:43.654235","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"198'17","last_scrub_stamp":"2021-02-10 11:50:43.654231","last_deep_scrub":"198'17","last_deep_scrub_stamp":"2021-02-10 11:50:43.654231","last_clean_scrub_stamp":"2021-02-10 11:50:43.654231","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":35,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":386,"num_omap_keys":8,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"43.5","version":"198'33","reported_seq":"54","reported_epoch":"198","state":"active+clean","last_fresh":"2021-02-10 11:50:38.931592","last_change":"2021-02-10 11:50:38.931644","last_active":"2021-02-10 11:50:38.931592","last_peered":"2021-02-10 11:50:38.931592","last_clean":"2021-02-10 11:50:38.931592","last_became_active":"2021-02-10 11:49:04.485608","last_became_peered":"2021-02-10 11:49:04.485608","last_unstale":"2021-02-10 11:50:38.931592","last_undegraded":"2021-02-10 11:50:38.931592","last_fullsized":"2021-02-10 11:50:38.931592","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"198'33","last_scrub_stamp":"2021-02-10 11:50:38.931589","last_deep_scrub":"198'33","last_deep_scrub_stamp":"2021-02-10 11:50:38.931589","last_clean_scrub_stamp":"2021-02-10 11:50:38.931589","log_size":33,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16777506,"num_objects":18,"num_object_clones":1,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":10,"num_read_kb":10240,"num_write":20,"num_write_kb":16387,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"42.4","version":"198'226","reported_seq":"240","reported_epoch":"198","state":"active+clean","last_fresh":"2021-02-10 11:50:38.950848","last_change":"2021-02-10 11:50:38.950887","last_active":"2021-02-10 11:50:38.950848","last_peered":"2021-02-10 11:50:38.950848","last_clean":"2021-02-10 11:50:38.950848","last_became_active":"2021-02-10 11:49:03.848058","last_became_peered":"2021-02-10 11:49:03.848058","last_unstale":"2021-02-10 11:50:38.950848","last_undegraded":"2021-02-10 11:50:38.950848","last_fullsized":"2021-02-10 11:50:38.950848","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"198'226","last_scrub_stamp":"2021-02-10 11:50:38.950844","last_deep_scrub":"198'226","last_deep_scrub_stamp":"2021-02-10 11:50:38.950844","last_clean_scrub_stamp":"2021-02-10 11:50:38.950844","log_size":226,"ondisk_log_size":226,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":934113,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":223,"num_write_kb":980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"43.2","version":"198'27","reported_seq":"47","reported_epoch":"198","state":"active+clean","last_fresh":"2021-02-10 11:50:36.375465","last_change":"2021-02-10 11:50:36.375497","last_active":"2021-02-10 11:50:36.375465","last_peered":"2021-02-10 11:50:36.375465","last_clean":"2021-02-10 11:50:36.375465","last_became_active":"2021-02-10 11:49:04.485916","last_became_peered":"2021-02-10 11:49:04.485916","last_unstale":"2021-02-10 11:50:36.375465","last_undegraded":"2021-02-10 11:50:36.375465","last_fullsized":"2021-02-10 11:50:36.375465","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"198'27","last_scrub_stamp":"2021-02-10 11:50:36.375461","last_deep_scrub":"198'27","last_deep_scrub_stamp":"2021-02-10 11:50:36.375461","last_clean_scrub_stamp":"2021-02-10 11:50:36.375461","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20971520,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":8,"num_read_kb":8192,"num_write":20,"num_write_kb":20480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.3","version":"198'9","reported_seq":"23","reported_epoch":"198","state":"active+clean","last_fresh":"2021-02-10 11:50:42.250034","last_change":"2021-02-10 11:50:42.250072","last_active":"2021-02-10 11:50:42.250034","last_peered":"2021-02-10 11:50:42.250034","last_clean":"2021-02-10 11:50:42.250034","last_became_active":"2021-02-10 11:49:03.848441","last_became_peered":"2021-02-10 11:49:03.848441","last_unstale":"2021-02-10 11:50:42.250034","last_undegraded":"2021-02-10 11:50:42.250034","last_fullsized":"2021-02-10 11:50:42.250034","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"198'9","last_scrub_stamp":"2021-02-10 11:50:42.250032","last_deep_scrub":"198'9","last_deep_scrub_stamp":"2021-02-10 11:50:42.250032","last_clean_scrub_stamp":"2021-02-10 11:50:42.250032","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4620,"num_omap_keys":10,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"43.3","version":"198'29","reported_seq":"49","reported_epoch":"198","state":"active+clean","last_fresh":"2021-02-10 11:50:37.369014","last_change":"2021-02-10 11:50:37.369044","last_active":"2021-02-10 11:50:37.369014","last_peered":"2021-02-10 11:50:37.369014","last_clean":"2021-02-10 11:50:37.369014","last_became_active":"2021-02-10 11:49:04.485844","last_became_peered":"2021-02-10 11:49:04.485844","last_unstale":"2021-02-10 11:50:37.369014","last_undegraded":"2021-02-10 11:50:37.369014","last_fullsized":"2021-02-10 11:50:37.369014","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"198'29","last_scrub_stamp":"2021-02-10 11:50:37.369009","last_deep_scrub":"198'29","last_deep_scrub_stamp":"2021-02-10 11:50:37.369009","last_clean_scrub_stamp":"2021-02-10 11:50:37.369009","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18874505,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":8,"num_read_kb":8192,"num_write":25,"num_write_kb":18436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.2","version":"197'1","reported_seq":"15","reported_epoch":"198","state":"active+clean","last_fresh":"2021-02-10 11:50:34.368827","last_change":"2021-02-10 11:50:34.368861","last_active":"2021-02-10 11:50:34.368827","last_peered":"2021-02-10 11:50:34.368827","last_clean":"2021-02-10 11:50:34.368827","last_became_active":"2021-02-10 11:49:03.849140","last_became_peered":"2021-02-10 11:49:03.849140","last_unstale":"2021-02-10 11:50:34.368827","last_undegraded":"2021-02-10 11:50:34.368827","last_fullsized":"2021-02-10 11:50:34.368827","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"197'1","last_scrub_stamp":"2021-02-10 11:50:34.368824","last_deep_scrub":"197'1","last_deep_scrub_stamp":"2021-02-10 11:50:34.368824","last_clean_scrub_stamp":"2021-02-10 11:50:34.368824","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"43.0","version":"198'27","reported_seq":"49","reported_epoch":"198","state":"active+clean","last_fresh":"2021-02-10 11:50:44.304760","last_change":"2021-02-10 11:50:44.304790","last_active":"2021-02-10 11:50:44.304760","last_peered":"2021-02-10 11:50:44.304760","last_clean":"2021-02-10 11:50:44.304760","last_became_active":"2021-02-10 11:49:04.487045","last_became_peered":"2021-02-10 11:49:04.487045","last_unstale":"2021-02-10 11:50:44.304760","last_undegraded":"2021-02-10 11:50:44.304760","last_fullsized":"2021-02-10 11:50:44.304760","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"198'27","last_scrub_stamp":"2021-02-10 11:50:44.304757","last_deep_scrub":"198'27","last_deep_scrub_stamp":"2021-02-10 11:50:44.304757","last_clean_scrub_stamp":"2021-02-10 11:50:44.304757","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18874368,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":10,"num_read_kb":10240,"num_write":18,"num_write_kb":18432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"42.1","version":"198'2","reported_seq":"17","reported_epoch":"198","state":"active+clean","last_fresh":"2021-02-10 11:50:35.806017","last_change":"2021-02-10 11:50:35.806050","last_active":"2021-02-10 11:50:35.806017","last_peered":"2021-02-10 11:50:35.806017","last_clean":"2021-02-10 11:50:35.806017","last_became_active":"2021-02-10 11:49:03.849304","last_became_peered":"2021-02-10 11:49:03.849304","last_unstale":"2021-02-10 11:50:35.806017","last_undegraded":"2021-02-10 11:50:35.806017","last_fullsized":"2021-02-10 11:50:35.806017","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"198'2","last_scrub_stamp":"2021-02-10 11:50:35.806015","last_deep_scrub":"198'2","last_deep_scrub_stamp":"2021-02-10 11:50:35.806015","last_clean_scrub_stamp":"2021-02-10 11:50:35.806015","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":101,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"43.1","version":"198'17","reported_seq":"35","reported_epoch":"198","state":"active+clean","last_fresh":"2021-02-10 11:50:44.703655","last_change":"2021-02-10 11:50:44.703683","last_active":"2021-02-10 11:50:44.703655","last_peered":"2021-02-10 11:50:44.703655","last_clean":"2021-02-10 11:50:44.703655","last_became_active":"2021-02-10 11:49:04.487096","last_became_peered":"2021-02-10 11:49:04.487096","last_unstale":"2021-02-10 11:50:44.703655","last_undegraded":"2021-02-10 11:50:44.703655","last_fullsized":"2021-02-10 11:50:44.703655","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"198'17","last_scrub_stamp":"2021-02-10 11:50:44.703652","last_deep_scrub":"198'17","last_deep_scrub_stamp":"2021-02-10 11:50:44.703652","last_clean_scrub_stamp":"2021-02-10 11:50:44.703652","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":10485760,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":6,"num_read_kb":6144,"num_write":10,"num_write_kb":10240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.0","version":"198'12","reported_seq":"26","reported_epoch":"198","state":"active+clean","last_fresh":"2021-02-10 11:50:40.729488","last_change":"2021-02-10 11:50:40.729528","last_active":"2021-02-10 11:50:40.729488","last_peered":"2021-02-10 11:50:40.729488","last_clean":"2021-02-10 11:50:40.729488","last_became_active":"2021-02-10 11:49:03.848142","last_became_peered":"2021-02-10 11:49:03.848142","last_unstale":"2021-02-10 11:50:40.729488","last_undegraded":"2021-02-10 11:50:40.729488","last_fullsized":"2021-02-10 11:50:40.729488","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"198'12","last_scrub_stamp":"2021-02-10 11:50:40.729484","last_deep_scrub":"198'12","last_deep_scrub_stamp":"2021-02-10 11:50:40.729484","last_clean_scrub_stamp":"2021-02-10 11:50:40.729484","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":"183","reported_epoch":"198","state":"active+clean","last_fresh":"2021-02-10 11:50:41.264777","last_change":"2021-02-10 11:50:41.264811","last_active":"2021-02-10 11:50:41.264777","last_peered":"2021-02-10 11:50:41.264777","last_clean":"2021-02-10 11:50:41.264777","last_became_active":"2021-02-10 11:13:58.962972","last_became_peered":"2021-02-10 11:13:58.962972","last_unstale":"2021-02-10 11:50:41.264777","last_undegraded":"2021-02-10 11:50:41.264777","last_fullsized":"2021-02-10 11:50:41.264777","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:50:41.264774","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:50:41.264774","last_clean_scrub_stamp":"2021-02-10 11:50:41.264774","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":"183","reported_epoch":"198","state":"active+clean","last_fresh":"2021-02-10 11:50:39.185635","last_change":"2021-02-10 11:50:39.185664","last_active":"2021-02-10 11:50:39.185635","last_peered":"2021-02-10 11:50:39.185635","last_clean":"2021-02-10 11:50:39.185635","last_became_active":"2021-02-10 11:13:58.964050","last_became_peered":"2021-02-10 11:13:58.964050","last_unstale":"2021-02-10 11:50:39.185635","last_undegraded":"2021-02-10 11:50:39.185635","last_fullsized":"2021-02-10 11:50:39.185635","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:50:39.185632","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:50:39.185632","last_clean_scrub_stamp":"2021-02-10 11:50:39.185632","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"190","reported_epoch":"198","state":"active+clean","last_fresh":"2021-02-10 11:50:36.947029","last_change":"2021-02-10 11:50:36.947060","last_active":"2021-02-10 11:50:36.947029","last_peered":"2021-02-10 11:50:36.947029","last_clean":"2021-02-10 11:50:36.947029","last_became_active":"2021-02-10 11:13:58.963951","last_became_peered":"2021-02-10 11:13:58.963951","last_unstale":"2021-02-10 11:50:36.947029","last_undegraded":"2021-02-10 11:50:36.947029","last_fullsized":"2021-02-10 11:50:36.947029","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:50:36.947027","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:50:36.947027","last_clean_scrub_stamp":"2021-02-10 11:50:36.947027","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":"186","reported_epoch":"198","state":"active+clean","last_fresh":"2021-02-10 11:50:34.566603","last_change":"2021-02-10 11:50:34.566637","last_active":"2021-02-10 11:50:34.566603","last_peered":"2021-02-10 11:50:34.566603","last_clean":"2021-02-10 11:50:34.566603","last_became_active":"2021-02-10 11:13:58.961934","last_became_peered":"2021-02-10 11:13:58.961934","last_unstale":"2021-02-10 11:50:34.566603","last_undegraded":"2021-02-10 11:50:34.566603","last_fullsized":"2021-02-10 11:50:34.566603","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:50:34.566600","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:50:34.566600","last_clean_scrub_stamp":"2021-02-10 11:50:34.566600","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"182","reported_epoch":"198","state":"active+clean","last_fresh":"2021-02-10 11:50:38.134005","last_change":"2021-02-10 11:50:38.134035","last_active":"2021-02-10 11:50:38.134005","last_peered":"2021-02-10 11:50:38.134005","last_clean":"2021-02-10 11:50:38.134005","last_became_active":"2021-02-10 11:13:58.964097","last_became_peered":"2021-02-10 11:13:58.964097","last_unstale":"2021-02-10 11:50:38.134005","last_undegraded":"2021-02-10 11:50:38.134005","last_fullsized":"2021-02-10 11:50:38.134005","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:50:38.134002","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:50:38.134002","last_clean_scrub_stamp":"2021-02-10 11:50:38.134002","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"43.4","version":"198'16","reported_seq":"34","reported_epoch":"198","state":"active+clean","last_fresh":"2021-02-10 11:50:36.650210","last_change":"2021-02-10 11:50:36.650234","last_active":"2021-02-10 11:50:36.650210","last_peered":"2021-02-10 11:50:36.650210","last_clean":"2021-02-10 11:50:36.650210","last_became_active":"2021-02-10 11:49:04.485246","last_became_peered":"2021-02-10 11:49:04.485246","last_unstale":"2021-02-10 11:50:36.650210","last_undegraded":"2021-02-10 11:50:36.650210","last_fullsized":"2021-02-10 11:50:36.650210","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"198'16","last_scrub_stamp":"2021-02-10 11:50:36.650207","last_deep_scrub":"198'16","last_deep_scrub_stamp":"2021-02-10 11:50:36.650207","last_clean_scrub_stamp":"2021-02-10 11:50:36.650207","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":10485760,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":6,"num_read_kb":6144,"num_write":10,"num_write_kb":10240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"42.5","version":"198'12","reported_seq":"26","reported_epoch":"198","state":"active+clean","last_fresh":"2021-02-10 11:50:41.732527","last_change":"2021-02-10 11:50:41.732565","last_active":"2021-02-10 11:50:41.732527","last_peered":"2021-02-10 11:50:41.732527","last_clean":"2021-02-10 11:50:41.732527","last_became_active":"2021-02-10 11:49:03.848285","last_became_peered":"2021-02-10 11:49:03.848285","last_unstale":"2021-02-10 11:50:41.732527","last_undegraded":"2021-02-10 11:50:41.732527","last_fullsized":"2021-02-10 11:50:41.732527","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"198'12","last_scrub_stamp":"2021-02-10 11:50:41.732523","last_deep_scrub":"198'12","last_deep_scrub_stamp":"2021-02-10 11:50:41.732523","last_clean_scrub_stamp":"2021-02-10 11:50:41.732523","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":160,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"191","reported_epoch":"198","state":"active+clean","last_fresh":"2021-02-10 11:50:38.782612","last_change":"2021-02-10 11:50:38.782641","last_active":"2021-02-10 11:50:38.782612","last_peered":"2021-02-10 11:50:38.782612","last_clean":"2021-02-10 11:50:38.782612","last_became_active":"2021-02-10 11:13:58.963814","last_became_peered":"2021-02-10 11:13:58.963814","last_unstale":"2021-02-10 11:50:38.782612","last_undegraded":"2021-02-10 11:50:38.782612","last_fullsized":"2021-02-10 11:50:38.782612","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:50:38.782609","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:50:38.782609","last_clean_scrub_stamp":"2021-02-10 11:50:38.782609","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"182","reported_epoch":"198","state":"active+clean","last_fresh":"2021-02-10 11:50:41.067707","last_change":"2021-02-10 11:50:41.067739","last_active":"2021-02-10 11:50:41.067707","last_peered":"2021-02-10 11:50:41.067707","last_clean":"2021-02-10 11:50:41.067707","last_became_active":"2021-02-10 11:13:58.964040","last_became_peered":"2021-02-10 11:13:58.964040","last_unstale":"2021-02-10 11:50:41.067707","last_undegraded":"2021-02-10 11:50:41.067707","last_fullsized":"2021-02-10 11:50:41.067707","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:50:41.067704","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:50:41.067704","last_clean_scrub_stamp":"2021-02-10 11:50:41.067704","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"183","reported_epoch":"198","state":"active+clean","last_fresh":"2021-02-10 11:50:31.812840","last_change":"2021-02-10 11:50:31.812874","last_active":"2021-02-10 11:50:31.812840","last_peered":"2021-02-10 11:50:31.812840","last_clean":"2021-02-10 11:50:31.812840","last_became_active":"2021-02-10 11:13:58.962533","last_became_peered":"2021-02-10 11:13:58.962533","last_unstale":"2021-02-10 11:50:31.812840","last_undegraded":"2021-02-10 11:50:31.812840","last_fullsized":"2021-02-10 11:50:31.812840","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:50:31.812837","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:50:31.812837","last_clean_scrub_stamp":"2021-02-10 11:50:31.812837","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":43,"num_pg":8,"stat_sum":{"num_bytes":134218155,"num_objects":131,"num_object_clones":1,"num_object_copies":262,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":131,"num_whiteouts":0,"num_read":64,"num_read_kb":65536,"num_write":139,"num_write_kb":131079,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":268460032,"data_stored":268436310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":214,"ondisk_log_size":214,"up":16,"acting":16,"num_store_stats":8},{"poolid":42,"num_pg":8,"stat_sum":{"num_bytes":934974,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293,"num_write_kb":1007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5006,"num_omap_keys":18,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1933312,"data_stored":1869948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":291,"ondisk_log_size":291,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575295,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1087964,"kb_used_data":39380,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93283876,"statfs":{"total":96636764160,"available":95522689024,"internally_reserved":1073741824,"allocated":40325120,"data_stored":38850165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 11:49:53 2021","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.82499999999999996,"15min":0.79000000000000004},"min":{"1min":0.37,"5min":0.37,"15min":0.36499999999999999},"max":{"1min":1.1850000000000001,"5min":1.248,"15min":1.4259999999999999},"last":1.081},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.82599999999999996,"15min":0.80600000000000005},"min":{"1min":0.40699999999999997,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.1379999999999999,"5min":1.3340000000000001,"15min":1.3939999999999999},"last":1.288}]},{"osd":1,"last update":"Wed Feb 10 11:49:53 2021","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.80200000000000005,"15min":0.80500000000000005},"min":{"1min":0.44700000000000001,"5min":0.436,"15min":0.42199999999999999},"max":{"1min":1.161,"5min":1.3839999999999999,"15min":1.3839999999999999},"last":0.93899999999999995},{"interface":"front","average":{"1min":0.83499999999999996,"5min":0.84999999999999998,"15min":0.81599999999999995},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.371},"max":{"1min":1.1140000000000001,"5min":1.3979999999999999,"15min":1.3979999999999999},"last":1.1739999999999999}]},{"osd":2,"last update":"Wed Feb 10 11:49:53 2021","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":0.91400000000000003,"15min":0.88100000000000001},"min":{"1min":0.432,"5min":0.42899999999999999,"15min":0.40699999999999997},"max":{"1min":1.151,"5min":1.4510000000000001,"15min":1.4510000000000001},"last":1.2410000000000001},{"interface":"front","average":{"1min":0.94199999999999995,"5min":1.0029999999999999,"15min":0.96999999999999997},"min":{"1min":0.54900000000000004,"5min":0.51400000000000001,"15min":0.47399999999999998},"max":{"1min":1.2250000000000001,"5min":1.5329999999999999,"15min":1.5329999999999999},"last":1.3400000000000001}]},{"osd":3,"last update":"Wed Feb 10 11:49:53 2021","interfaces":[{"interface":"back","average":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.80400000000000005},"min":{"1min":0.51200000000000001,"5min":0.46700000000000003,"15min":0.40699999999999997},"max":{"1min":1.099,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":1.115},{"interface":"front","average":{"1min":0.89800000000000002,"5min":0.90600000000000003,"15min":0.85999999999999999},"min":{"1min":0.5,"5min":0.48099999999999998,"15min":0.42599999999999999},"max":{"1min":1.1259999999999999,"5min":1.4339999999999999,"15min":1.4339999999999999},"last":1.224}]},{"osd":4,"last update":"Wed Feb 10 11:49:53 2021","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.69699999999999995,"15min":0.69299999999999995},"min":{"1min":0.46400000000000002,"5min":0.40999999999999998,"15min":0.34300000000000003},"max":{"1min":0.89000000000000001,"5min":1.3320000000000001,"15min":1.3320000000000001},"last":0.748},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.70999999999999996,"15min":0.69399999999999995},"min":{"1min":0.42599999999999999,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.93700000000000006,"5min":1.127,"15min":1.149},"last":0.70899999999999996}]},{"osd":5,"last update":"Wed Feb 10 11:49:53 2021","interfaces":[{"interface":"back","average":{"1min":0.86399999999999999,"5min":0.81399999999999995,"15min":0.80800000000000005},"min":{"1min":0.61599999999999999,"5min":0.48399999999999999,"15min":0.45700000000000002},"max":{"1min":1.1259999999999999,"5min":1.3140000000000001,"15min":1.3140000000000001},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.85799999999999998,"5min":0.88200000000000001,"15min":0.878},"min":{"1min":0.59399999999999997,"5min":0.50700000000000001,"15min":0.45200000000000001},"max":{"1min":1.1220000000000001,"5min":1.3260000000000001,"15min":1.377},"last":0.87}]},{"osd":6,"last update":"Wed Feb 10 11:49:53 2021","interfaces":[{"interface":"back","average":{"1min":0.80600000000000005,"5min":0.81699999999999995,"15min":0.82299999999999995},"min":{"1min":0.55700000000000005,"5min":0.52900000000000003,"15min":0.50700000000000001},"max":{"1min":0.97599999999999998,"5min":1.2709999999999999,"15min":1.327},"last":1.014},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.80200000000000005,"15min":0.80300000000000005},"min":{"1min":0.55400000000000005,"5min":0.53800000000000003,"15min":0.53200000000000003},"max":{"1min":1.0469999999999999,"5min":1.276,"15min":1.276},"last":0.85199999999999998}]}]},{"osd":6,"up_from":13,"seq":55834575296,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1070436,"kb_used_data":21852,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93301404,"statfs":{"total":96636764160,"available":95540637696,"internally_reserved":1073741824,"allocated":22376448,"data_stored":20908987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 11:50:07 2021","interfaces":[{"interface":"back","average":{"1min":0.94199999999999995,"5min":0.83199999999999996,"15min":0.81299999999999994},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.34699999999999998},"max":{"1min":1.169,"5min":1.1859999999999999,"15min":1.4259999999999999},"last":1.147},{"interface":"front","average":{"1min":0.879,"5min":0.78700000000000003,"15min":0.78600000000000003},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41299999999999998},"max":{"1min":1.131,"5min":1.2050000000000001,"15min":1.4419999999999999},"last":1.022}]},{"osd":1,"last update":"Wed Feb 10 11:50:07 2021","interfaces":[{"interface":"back","average":{"1min":0.90800000000000003,"5min":0.80800000000000005,"15min":0.79800000000000004},"min":{"1min":0.46300000000000002,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.147,"5min":1.147,"15min":1.504},"last":1.105},{"interface":"front","average":{"1min":0.95999999999999996,"5min":0.83699999999999997,"15min":0.83399999999999996},"min":{"1min":0.48399999999999999,"5min":0.45600000000000002,"15min":0.45300000000000001},"max":{"1min":1.2549999999999999,"5min":1.2549999999999999,"15min":1.3959999999999999},"last":0.96899999999999997}]},{"osd":2,"last update":"Wed Feb 10 11:50:07 2021","interfaces":[{"interface":"back","average":{"1min":1.0149999999999999,"5min":0.93400000000000005,"15min":0.93500000000000005},"min":{"1min":0.623,"5min":0.49299999999999999,"15min":0.438},"max":{"1min":1.339,"5min":1.339,"15min":1.3799999999999999},"last":1.2170000000000001},{"interface":"front","average":{"1min":1.054,"5min":0.996,"15min":0.98699999999999999},"min":{"1min":0.61399999999999999,"5min":0.61299999999999999,"15min":0.55200000000000005},"max":{"1min":1.3320000000000001,"5min":1.3320000000000001,"15min":1.4950000000000001},"last":1.099}]},{"osd":3,"last update":"Wed Feb 10 11:50:07 2021","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":0.90700000000000003,"15min":0.90200000000000002},"min":{"1min":0.498,"5min":0.47199999999999998,"15min":0.39900000000000002},"max":{"1min":1.272,"5min":1.272,"15min":1.272},"last":1.125},{"interface":"front","average":{"1min":1.0129999999999999,"5min":0.96699999999999997,"15min":0.95699999999999996},"min":{"1min":0.76700000000000002,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.246,"5min":1.258,"15min":1.361},"last":1.157}]},{"osd":4,"last update":"Wed Feb 10 11:50:07 2021","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.66900000000000004,"15min":0.71599999999999997},"min":{"1min":0.48699999999999999,"5min":0.441,"15min":0.31900000000000001},"max":{"1min":0.92800000000000005,"5min":1.0960000000000001,"15min":1.399},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.70099999999999996,"15min":0.71399999999999997},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.35599999999999998},"max":{"1min":0.88700000000000001,"5min":1.004,"15min":1.23},"last":0.54500000000000004}]},{"osd":5,"last update":"Wed Feb 10 11:50:07 2021","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.68000000000000005,"15min":0.71899999999999997},"min":{"1min":0.45200000000000001,"5min":0.42199999999999999,"15min":0.40899999999999997},"max":{"1min":0.85199999999999998,"5min":1.113,"15min":1.327},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.75,"5min":0.77500000000000002,"15min":0.79600000000000004},"min":{"1min":0.58699999999999997,"5min":0.57699999999999996,"15min":0.40699999999999997},"max":{"1min":0.97799999999999998,"5min":1.163,"15min":1.5},"last":0.68600000000000005}]},{"osd":7,"last update":"Wed Feb 10 11:50:07 2021","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.75900000000000001,"15min":0.78000000000000003},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.41599999999999998},"max":{"1min":0.92700000000000005,"5min":1.149,"15min":1.2829999999999999},"last":0.624},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.79300000000000004,"15min":0.81499999999999995},"min":{"1min":0.58399999999999996,"5min":0.53900000000000003,"15min":0.505},"max":{"1min":1.1160000000000001,"5min":1.1699999999999999,"15min":1.3600000000000001},"last":0.79100000000000004}]}]},{"osd":5,"up_from":13,"seq":55834575295,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1083868,"kb_used_data":35272,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93287972,"statfs":{"total":96636764160,"available":95526883328,"internally_reserved":1073741824,"allocated":36118528,"data_stored":34655064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 11:49:55 2021","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":0.83199999999999996,"15min":0.81999999999999995},"min":{"1min":0.35799999999999998,"5min":0.35699999999999998,"15min":0.34999999999999998},"max":{"1min":1.3580000000000001,"5min":1.3580000000000001,"15min":1.581},"last":1.0269999999999999},{"interface":"front","average":{"1min":0.84099999999999997,"5min":0.79800000000000004,"15min":0.80100000000000005},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.35799999999999998},"max":{"1min":1.288,"5min":1.288,"15min":1.3939999999999999},"last":1.044}]},{"osd":1,"last update":"Wed Feb 10 11:49:55 2021","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.81200000000000006,"15min":0.81799999999999995},"min":{"1min":0.375,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.2490000000000001,"5min":1.2490000000000001,"15min":1.3540000000000001},"last":0.95399999999999996},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.81599999999999995,"15min":0.82499999999999996},"min":{"1min":0.376,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":1.097,"5min":1.2509999999999999,"15min":1.319},"last":1.0900000000000001}]},{"osd":2,"last update":"Wed Feb 10 11:49:55 2021","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.80700000000000005,"15min":0.81499999999999995},"min":{"1min":0.42399999999999999,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.1439999999999999,"5min":1.258,"15min":1.2729999999999999},"last":1.153},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.83199999999999996,"15min":0.84499999999999997},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.24,"5min":1.2769999999999999,"15min":1.2769999999999999},"last":1.0780000000000001}]},{"osd":3,"last update":"Wed Feb 10 11:49:55 2021","interfaces":[{"interface":"back","average":{"1min":0.83599999999999997,"5min":0.85499999999999998,"15min":0.873},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.264,"5min":1.323,"15min":1.464},"last":1.143},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.85299999999999998,"15min":0.875},"min":{"1min":0.57599999999999996,"5min":0.52400000000000002,"15min":0.44900000000000001},"max":{"1min":1.0960000000000001,"5min":1.2549999999999999,"15min":1.3520000000000001},"last":1.173}]},{"osd":4,"last update":"Wed Feb 10 11:49:55 2021","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.68400000000000005,"15min":0.70899999999999996},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.117,"5min":1.1910000000000001,"15min":1.1910000000000001},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.67800000000000005,"15min":0.70899999999999996},"min":{"1min":0.45200000000000001,"5min":0.45000000000000001,"15min":0.40799999999999997},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":1.2490000000000001},"last":0.746}]},{"osd":6,"last update":"Wed Feb 10 11:49:55 2021","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.72499999999999998,"15min":0.73099999999999998},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.38900000000000001},"max":{"1min":1.0169999999999999,"5min":1.22,"15min":1.22},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.76000000000000001,"15min":0.78600000000000003},"min":{"1min":0.52800000000000002,"5min":0.48699999999999999,"15min":0.40500000000000003},"max":{"1min":1.0009999999999999,"5min":1.2210000000000001,"15min":1.2210000000000001},"last":0.76000000000000001}]},{"osd":7,"last update":"Wed Feb 10 11:49:55 2021","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.755,"15min":0.78300000000000003},"min":{"1min":0.48799999999999999,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.93300000000000005,"5min":1.1459999999999999,"15min":1.1699999999999999},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.748,"5min":0.78600000000000003,"15min":0.81599999999999995},"min":{"1min":0.52800000000000002,"5min":0.47099999999999997,"15min":0.46899999999999997},"max":{"1min":1.161,"5min":1.355,"15min":1.4710000000000001},"last":0.60699999999999998}]}]},{"osd":4,"up_from":13,"seq":55834575296,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1074528,"kb_used_data":25944,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93297312,"statfs":{"total":96636764160,"available":95536447488,"internally_reserved":1073741824,"allocated":26566656,"data_stored":25103257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 11:50:10 2021","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":0.81000000000000005,"15min":0.81100000000000005},"min":{"1min":0.40100000000000002,"5min":0.32300000000000001,"15min":0.317},"max":{"1min":1.212,"5min":1.3660000000000001,"15min":1.5840000000000001},"last":0.85999999999999999},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.79600000000000004,"15min":0.78800000000000003},"min":{"1min":0.46899999999999997,"5min":0.40799999999999997,"15min":0.38100000000000001},"max":{"1min":1.0740000000000001,"5min":1.2909999999999999,"15min":1.2909999999999999},"last":1.079}]},{"osd":1,"last update":"Wed Feb 10 11:50:10 2021","interfaces":[{"interface":"back","average":{"1min":0.81100000000000005,"5min":0.76000000000000001,"15min":0.77800000000000002},"min":{"1min":0.47699999999999998,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.095,"5min":1.2110000000000001,"15min":1.2689999999999999},"last":1.0509999999999999},{"interface":"front","average":{"1min":0.84399999999999997,"5min":0.85299999999999998,"15min":0.84699999999999998},"min":{"1min":0.55700000000000005,"5min":0.48399999999999999,"15min":0.40000000000000002},"max":{"1min":1.091,"5min":1.3180000000000001,"15min":1.319},"last":1.0900000000000001}]},{"osd":2,"last update":"Wed Feb 10 11:50:10 2021","interfaces":[{"interface":"back","average":{"1min":1.024,"5min":1.0089999999999999,"15min":1.0069999999999999},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.46899999999999997},"max":{"1min":1.353,"5min":1.4350000000000001,"15min":1.635},"last":1.2070000000000001},{"interface":"front","average":{"1min":0.98399999999999999,"5min":0.91200000000000003,"15min":0.92200000000000004},"min":{"1min":0.67800000000000005,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.3120000000000001,"5min":1.327,"15min":1.4690000000000001},"last":1.131}]},{"osd":3,"last update":"Wed Feb 10 11:50:10 2021","interfaces":[{"interface":"back","average":{"1min":1.03,"5min":1.008,"15min":0.998},"min":{"1min":0.67000000000000004,"5min":0.53800000000000003,"15min":0.48099999999999998},"max":{"1min":1.355,"5min":1.46,"15min":1.5569999999999999},"last":1.1399999999999999},{"interface":"front","average":{"1min":1.04,"5min":0.97099999999999997,"15min":0.94399999999999995},"min":{"1min":0.755,"5min":0.498,"15min":0.48899999999999999},"max":{"1min":1.325,"5min":1.379,"15min":1.3979999999999999},"last":1.1479999999999999}]},{"osd":5,"last update":"Wed Feb 10 11:50:10 2021","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.73599999999999999,"15min":0.73399999999999999},"min":{"1min":0.438,"5min":0.436,"15min":0.36799999999999999},"max":{"1min":1.095,"5min":1.1799999999999999,"15min":1.292},"last":0.623},{"interface":"front","average":{"1min":0.745,"5min":0.73199999999999998,"15min":0.70899999999999996},"min":{"1min":0.52500000000000002,"5min":0.46899999999999997,"15min":0.40999999999999998},"max":{"1min":1.3540000000000001,"5min":1.3540000000000001,"15min":1.3540000000000001},"last":0.79600000000000004}]},{"osd":6,"last update":"Wed Feb 10 11:50:10 2021","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.71999999999999997,"15min":0.73599999999999999},"min":{"1min":0.503,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.105,"5min":1.105,"15min":1.22},"last":0.78600000000000003},{"interface":"front","average":{"1min":0.81299999999999994,"5min":0.82299999999999995,"15min":0.82099999999999995},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.43099999999999999},"max":{"1min":1.1479999999999999,"5min":1.321,"15min":1.5149999999999999},"last":0.96499999999999997}]},{"osd":7,"last update":"Wed Feb 10 11:50:10 2021","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.79900000000000004,"15min":0.77300000000000002},"min":{"1min":0.61199999999999999,"5min":0.56499999999999995,"15min":0.40300000000000002},"max":{"1min":1.256,"5min":1.256,"15min":1.256},"last":0.90900000000000003},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.80300000000000005,"15min":0.80200000000000005},"min":{"1min":0.628,"5min":0.58099999999999996,"15min":0.52100000000000002},"max":{"1min":1.163,"5min":1.169,"15min":1.169},"last":0.89900000000000002}]}]},{"osd":0,"up_from":13,"seq":55834575295,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1061324,"kb_used_data":12740,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93310516,"statfs":{"total":96636764160,"available":95549968384,"internally_reserved":1073741824,"allocated":13045760,"data_stored":11586266,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Feb 10 11:49:56 2021","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.49299999999999999,"15min":0.48399999999999999},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":1,"5min":1,"15min":1},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.496,"5min":0.48299999999999998,"15min":0.47999999999999998},"min":{"1min":0.34599999999999997,"5min":0.32500000000000001,"15min":0.313},"max":{"1min":0.68000000000000005,"5min":0.73999999999999999,"15min":0.88800000000000001},"last":0.47999999999999998}]},{"osd":2,"last update":"Wed Feb 10 11:49:56 2021","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.64500000000000002,"15min":0.622},"min":{"1min":0.40999999999999998,"5min":0.317,"15min":0.317},"max":{"1min":0.871,"5min":0.95999999999999996,"15min":1.1719999999999999},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.64900000000000002,"15min":0.64400000000000002},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":1.125},"last":0.67000000000000004}]},{"osd":3,"last update":"Wed Feb 10 11:49:56 2021","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.52200000000000002,"15min":0.51900000000000002},"min":{"1min":0.38400000000000001,"5min":0.35299999999999998,"15min":0.34200000000000003},"max":{"1min":0.71099999999999997,"5min":0.83599999999999997,"15min":0.97199999999999998},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.625,"15min":0.60899999999999999},"min":{"1min":0.46400000000000002,"5min":0.41899999999999998,"15min":0.318},"max":{"1min":0.77200000000000002,"5min":0.97099999999999997,"15min":0.99199999999999999},"last":0.60599999999999998}]},{"osd":4,"last update":"Wed Feb 10 11:49:56 2021","interfaces":[{"interface":"back","average":{"1min":0.98199999999999998,"5min":0.90200000000000002,"15min":0.92200000000000004},"min":{"1min":0.56399999999999995,"5min":0.52400000000000002,"15min":0.45400000000000001},"max":{"1min":1.3100000000000001,"5min":1.3320000000000001,"15min":1.365},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.96899999999999997,"5min":0.94299999999999995,"15min":0.94699999999999995},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.46600000000000003},"max":{"1min":1.3,"5min":1.3,"15min":1.393},"last":0.71099999999999997}]},{"osd":5,"last update":"Wed Feb 10 11:49:56 2021","interfaces":[{"interface":"back","average":{"1min":1.0720000000000001,"5min":1.0229999999999999,"15min":1.071},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.5169999999999999,"5min":1.5169999999999999,"15min":1.8660000000000001},"last":0.83499999999999996},{"interface":"front","average":{"1min":1.123,"5min":1.048,"15min":1.0940000000000001},"min":{"1min":0.64800000000000002,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":1.5389999999999999,"5min":1.6000000000000001,"15min":1.6000000000000001},"last":0.77400000000000002}]},{"osd":6,"last update":"Wed Feb 10 11:49:56 2021","interfaces":[{"interface":"back","average":{"1min":1.046,"5min":1.0409999999999999,"15min":1.077},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.4399999999999999,"5min":1.4399999999999999,"15min":1.542},"last":0.78400000000000003},{"interface":"front","average":{"1min":1.099,"5min":1.046,"15min":1.093},"min":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.61699999999999999},"max":{"1min":1.548,"5min":1.548,"15min":1.792},"last":0.98299999999999998}]},{"osd":7,"last update":"Wed Feb 10 11:49:56 2021","interfaces":[{"interface":"back","average":{"1min":1.087,"5min":1.054,"15min":1.095},"min":{"1min":0.58299999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.494,"5min":1.548,"15min":1.6559999999999999},"last":0.93000000000000005},{"interface":"front","average":{"1min":1.101,"5min":1.0489999999999999,"15min":1.0820000000000001},"min":{"1min":0.65500000000000003,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":1.4339999999999999,"5min":1.4990000000000001,"15min":1.4990000000000001},"last":0.92100000000000004}]}]},{"osd":1,"up_from":13,"seq":55834575294,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1092052,"kb_used_data":43468,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93279788,"statfs":{"total":96636764160,"available":95518502912,"internally_reserved":1073741824,"allocated":44511232,"data_stored":43043903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 11:50:08 2021","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.32700000000000001,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.876},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.47999999999999998,"15min":0.49199999999999999},"min":{"1min":0.374,"5min":0.308,"15min":0.308},"max":{"1min":0.75800000000000001,"5min":0.82499999999999996,"15min":1.1279999999999999},"last":0.41099999999999998}]},{"osd":2,"last update":"Wed Feb 10 11:50:08 2021","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.61799999999999999,"15min":0.63},"min":{"1min":0.44600000000000001,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.753,"5min":0.96399999999999997,"15min":1.4319999999999999},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.57799999999999996,"15min":0.59399999999999997},"min":{"1min":0.39000000000000001,"5min":0.378,"15min":0.313},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.95099999999999996},"last":0.46000000000000002}]},{"osd":3,"last update":"Wed Feb 10 11:50:08 2021","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.54700000000000004,"15min":0.54200000000000004},"min":{"1min":0.38800000000000001,"5min":0.315,"15min":0.315},"max":{"1min":0.68799999999999994,"5min":1,"15min":1},"last":0.625},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.61299999999999999,"15min":0.62},"min":{"1min":0.34399999999999997,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.83299999999999996,"5min":1.0429999999999999,"15min":1.22},"last":0.70799999999999996}]},{"osd":4,"last update":"Wed Feb 10 11:50:08 2021","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.875,"15min":0.90300000000000002},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":1.1910000000000001,"5min":1.498,"15min":1.498},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.85999999999999999,"5min":0.89200000000000002,"15min":0.91700000000000004},"min":{"1min":0.59199999999999997,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.2310000000000001,"5min":1.3380000000000001,"15min":1.3540000000000001},"last":0.72599999999999998}]},{"osd":5,"last update":"Wed Feb 10 11:50:08 2021","interfaces":[{"interface":"back","average":{"1min":0.99099999999999999,"5min":1.0209999999999999,"15min":1.052},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.48199999999999998},"max":{"1min":1.544,"5min":1.544,"15min":1.6399999999999999},"last":0.92600000000000005},{"interface":"front","average":{"1min":0.98799999999999999,"5min":1.0429999999999999,"15min":1.071},"min":{"1min":0.65100000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.4279999999999999,"5min":1.446,"15min":1.4870000000000001},"last":0.95599999999999996}]},{"osd":6,"last update":"Wed Feb 10 11:50:08 2021","interfaces":[{"interface":"back","average":{"1min":0.92300000000000004,"5min":0.95699999999999996,"15min":0.95599999999999996},"min":{"1min":0.66700000000000004,"5min":0.51800000000000002,"15min":0.496},"max":{"1min":1.2549999999999999,"5min":1.421,"15min":1.5349999999999999},"last":0.84599999999999997},{"interface":"front","average":{"1min":0.871,"5min":0.91400000000000003,"15min":0.93200000000000005},"min":{"1min":0.60899999999999999,"5min":0.56100000000000005,"15min":0.498},"max":{"1min":1.3029999999999999,"5min":1.3029999999999999,"15min":1.3919999999999999},"last":0.64700000000000002}]},{"osd":7,"last update":"Wed Feb 10 11:50:08 2021","interfaces":[{"interface":"back","average":{"1min":1.008,"5min":1.0389999999999999,"15min":1.0449999999999999},"min":{"1min":0.751,"5min":0.59099999999999997,"15min":0.52100000000000002},"max":{"1min":1.444,"5min":1.444,"15min":1.4830000000000001},"last":1.0249999999999999},{"interface":"front","average":{"1min":1.0529999999999999,"5min":1.079,"15min":1.097},"min":{"1min":0.81000000000000005,"5min":0.627,"15min":0.627},"max":{"1min":1.4330000000000001,"5min":1.4670000000000001,"15min":1.615},"last":1.1020000000000001}]}]},{"osd":2,"up_from":13,"seq":55834575295,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1122780,"kb_used_data":74196,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93249060,"statfs":{"total":96636764160,"available":95487037440,"internally_reserved":1073741824,"allocated":75976704,"data_stored":74501655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 11:49:52 2021","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.46899999999999997,"15min":0.47799999999999998},"min":{"1min":0.35499999999999998,"5min":0.308,"15min":0.308},"max":{"1min":0.70799999999999996,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.433},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.502,"15min":0.51000000000000001},"min":{"1min":0.34399999999999997,"5min":0.316,"15min":0.316},"max":{"1min":0.70799999999999996,"5min":0.92600000000000005,"15min":1.0329999999999999},"last":0.53800000000000003}]},{"osd":1,"last update":"Wed Feb 10 11:49:52 2021","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.51600000000000001,"15min":0.51700000000000002},"min":{"1min":0.38900000000000001,"5min":0.34000000000000002,"15min":0.32300000000000001},"max":{"1min":0.60299999999999998,"5min":0.94499999999999995,"15min":1.133},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.57299999999999995,"15min":0.57699999999999996},"min":{"1min":0.40799999999999997,"5min":0.375,"15min":0.375},"max":{"1min":0.84499999999999997,"5min":0.84699999999999998,"15min":0.93400000000000005},"last":0.73299999999999998}]},{"osd":3,"last update":"Wed Feb 10 11:49:52 2021","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.58399999999999996,"15min":0.58799999999999997},"min":{"1min":0.48299999999999998,"5min":0.40300000000000002,"15min":0.39900000000000002},"max":{"1min":0.76000000000000001,"5min":0.90500000000000003,"15min":0.96099999999999997},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.54600000000000004,"15min":0.56399999999999995},"min":{"1min":0.42199999999999999,"5min":0.36599999999999999,"15min":0.34599999999999997},"max":{"1min":0.77300000000000002,"5min":0.89900000000000002,"15min":0.94699999999999995},"last":0.629}]},{"osd":4,"last update":"Wed Feb 10 11:49:52 2021","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":0.91700000000000004,"15min":0.94099999999999995},"min":{"1min":0.68999999999999995,"5min":0.53100000000000003,"15min":0.52400000000000002},"max":{"1min":1.278,"5min":1.331,"15min":1.3919999999999999},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.95599999999999996,"5min":0.94499999999999995,"15min":0.96699999999999997},"min":{"1min":0.60499999999999998,"5min":0.57799999999999996,"15min":0.53300000000000003},"max":{"1min":1.304,"5min":1.304,"15min":1.4430000000000001},"last":0.85999999999999999}]},{"osd":5,"last update":"Wed Feb 10 11:49:52 2021","interfaces":[{"interface":"back","average":{"1min":0.95599999999999996,"5min":0.94799999999999995,"15min":0.97199999999999998},"min":{"1min":0.54400000000000004,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.206,"5min":1.393,"15min":1.4059999999999999},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.96599999999999997,"5min":0.94199999999999995,"15min":0.95399999999999996},"min":{"1min":0.65700000000000003,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.3300000000000001,"5min":1.359,"15min":1.369},"last":0.88400000000000001}]},{"osd":6,"last update":"Wed Feb 10 11:49:52 2021","interfaces":[{"interface":"back","average":{"1min":0.97799999999999998,"5min":0.94699999999999995,"15min":0.98199999999999998},"min":{"1min":0.62,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.361,"5min":1.361,"15min":1.462},"last":0.92700000000000005},{"interface":"front","average":{"1min":0.96299999999999997,"5min":0.97199999999999998,"15min":0.98799999999999999},"min":{"1min":0.68200000000000005,"5min":0.57099999999999995,"15min":0.54900000000000004},"max":{"1min":1.288,"5min":1.296,"15min":1.4319999999999999},"last":0.751}]},{"osd":7,"last update":"Wed Feb 10 11:49:52 2021","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":0.98099999999999998,"15min":0.999},"min":{"1min":0.73199999999999998,"5min":0.64900000000000002,"15min":0.58699999999999997},"max":{"1min":1.2290000000000001,"5min":1.4079999999999999,"15min":1.4079999999999999},"last":0.97799999999999998},{"interface":"front","average":{"1min":1.012,"5min":1.01,"15min":1.012},"min":{"1min":0.72899999999999998,"5min":0.64200000000000002,"15min":0.63800000000000001},"max":{"1min":1.3420000000000001,"5min":1.395,"15min":1.4350000000000001},"last":0.80000000000000004}]}]},{"osd":3,"up_from":13,"seq":55834575296,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1079756,"kb_used_data":31172,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93292084,"statfs":{"total":96636764160,"available":95531094016,"internally_reserved":1073741824,"allocated":31920128,"data_stored":30460737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 11:50:09 2021","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.498,"15min":0.496},"min":{"1min":0.35199999999999998,"5min":0.311,"15min":0.311},"max":{"1min":0.66300000000000003,"5min":1,"15min":1},"last":0.501},{"interface":"front","average":{"1min":0.498,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.40400000000000003,"5min":0.315,"15min":0.307},"max":{"1min":0.66900000000000004,"5min":0.83099999999999996,"15min":1.216},"last":0.63400000000000001}]},{"osd":1,"last update":"Wed Feb 10 11:50:09 2021","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.49199999999999999,"15min":0.502},"min":{"1min":0.34999999999999998,"5min":0.32900000000000001,"15min":0.30299999999999999},"max":{"1min":0.63800000000000001,"5min":0.98399999999999999,"15min":1.0740000000000001},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.55900000000000005,"15min":0.56799999999999995},"min":{"1min":0.39800000000000002,"5min":0.34499999999999997,"15min":0.28299999999999997},"max":{"1min":0.76400000000000001,"5min":0.85799999999999998,"15min":1.02},"last":0.72699999999999998}]},{"osd":2,"last update":"Wed Feb 10 11:50:09 2021","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.624,"15min":0.64000000000000001},"min":{"1min":0.42899999999999999,"5min":0.41499999999999998,"15min":0.33500000000000002},"max":{"1min":0.85599999999999998,"5min":0.88,"15min":1.081},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.77800000000000002,"15min":0.77400000000000002},"min":{"1min":0.54600000000000004,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.075,"5min":1.107,"15min":1.2030000000000001},"last":0.82399999999999995}]},{"osd":4,"last update":"Wed Feb 10 11:50:09 2021","interfaces":[{"interface":"back","average":{"1min":0.90900000000000003,"5min":0.90100000000000002,"15min":0.90000000000000002},"min":{"1min":0.53600000000000003,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.181,"5min":1.3839999999999999,"15min":1.4830000000000001},"last":1.0960000000000001},{"interface":"front","average":{"1min":0.92700000000000005,"5min":0.91500000000000004,"15min":0.92700000000000005},"min":{"1min":0.51300000000000001,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.3620000000000001,"5min":1.4890000000000001,"15min":1.4890000000000001},"last":0.97699999999999998}]},{"osd":5,"last update":"Wed Feb 10 11:50:09 2021","interfaces":[{"interface":"back","average":{"1min":1.095,"5min":1.0760000000000001,"15min":1.0649999999999999},"min":{"1min":0.73999999999999999,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.5069999999999999,"5min":1.508,"15min":1.538},"last":1.194},{"interface":"front","average":{"1min":1.0860000000000001,"5min":1.0589999999999999,"15min":1.048},"min":{"1min":0.66400000000000003,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.6419999999999999,"5min":1.6419999999999999,"15min":1.6419999999999999},"last":1.206}]},{"osd":6,"last update":"Wed Feb 10 11:50:09 2021","interfaces":[{"interface":"back","average":{"1min":0.95299999999999996,"5min":0.93500000000000005,"15min":0.95499999999999996},"min":{"1min":0.60599999999999998,"5min":0.501,"15min":0.501},"max":{"1min":1.2509999999999999,"5min":1.3939999999999999,"15min":1.452},"last":1.034},{"interface":"front","average":{"1min":0.96199999999999997,"5min":0.95799999999999996,"15min":0.97599999999999998},"min":{"1min":0.57899999999999996,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.323,"5min":1.323,"15min":1.4079999999999999},"last":0.90100000000000002}]},{"osd":7,"last update":"Wed Feb 10 11:50:09 2021","interfaces":[{"interface":"back","average":{"1min":1.0589999999999999,"5min":1.0549999999999999,"15min":1.0409999999999999},"min":{"1min":0.58999999999999997,"5min":0.58699999999999997,"15min":0.55100000000000005},"max":{"1min":1.5309999999999999,"5min":1.5309999999999999,"15min":1.5309999999999999},"last":1.27},{"interface":"front","average":{"1min":1.1040000000000001,"5min":1.0629999999999999,"15min":1.071},"min":{"1min":0.68700000000000006,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.518,"5min":1.556,"15min":1.556},"last":1.179}]}]}]}} 2021-02-10T11:50:58.895 DEBUG:teuthology.orchestra.run.gibba012:> sudo ceph --cluster ceph --mon-client-directed-command-retry 5 tell mon.* injectargs -- --no-mon-health-to-clog 2021-02-10T11:50:59.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:59.299 7feb65873700 1 Processor -- start 2021-02-10T11:50:59.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:59.299 7feb65873700 1 -- start start 2021-02-10T11:50:59.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:59.299 7feb65873700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7feb600e53a0 0x7feb601b0550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:59.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:59.299 7feb65873700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7feb601b0b70 0x7feb601089d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:59.321 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:59.299 7feb65873700 1 --2- >> v2:172.21.2.120:3300/0 conn(0x7feb60108ff0 0x7feb601093f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:59.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:59.299 7feb65873700 1 -- --> v2:172.21.2.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb600fe2b0 con 0x7feb601b0b70 2021-02-10T11:50:59.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:59.299 7feb65873700 1 -- --> v2:172.21.2.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb600e5f80 con 0x7feb60108ff0 2021-02-10T11:50:59.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:59.299 7feb65873700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb60109ac0 con 0x7feb600e53a0 2021-02-10T11:50:59.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:59.299 7feb5ed73700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7feb601b0b70 0x7feb601089d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:59.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:59.299 7feb5ed73700 1 --2- >> v2:172.21.2.112:3300/0 conn(0x7feb601b0b70 0x7feb601089d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:34258/0 (socket says 172.21.2.112:34258) 2021-02-10T11:50:59.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:59.299 7feb5ed73700 1 -- 172.21.2.112:0/165999516 learned_addr learned my addr 172.21.2.112:0/165999516 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:50:59.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:59.300 7feb5fd75700 1 --2- 172.21.2.112:0/165999516 >> v2:172.21.2.120:3300/0 conn(0x7feb60108ff0 0x7feb601093f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:59.322 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:59.300 7feb5ed73700 1 -- 172.21.2.112:0/165999516 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7feb600e53a0 msgr2=0x7feb601b0550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:50:59.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:59.300 7feb5f574700 1 --2- 172.21.2.112:0/165999516 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7feb600e53a0 0x7feb601b0550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:59.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:59.300 7feb5ed73700 1 --2- 172.21.2.112:0/165999516 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7feb600e53a0 0x7feb601b0550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:59.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:59.300 7feb5ed73700 1 -- 172.21.2.112:0/165999516 >> v2:172.21.2.120:3300/0 conn(0x7feb60108ff0 msgr2=0x7feb601093f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:50:59.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:59.300 7feb5ed73700 1 --2- 172.21.2.112:0/165999516 >> v2:172.21.2.120:3300/0 conn(0x7feb60108ff0 0x7feb601093f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:59.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:59.300 7feb5ed73700 1 -- 172.21.2.112:0/165999516 --> v2:172.21.2.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb600e57a0 con 0x7feb601b0b70 2021-02-10T11:50:59.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:59.300 7feb5fd75700 1 --2- 172.21.2.112:0/165999516 >> v2:172.21.2.120:3300/0 conn(0x7feb60108ff0 0x7feb601093f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:50:59.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:59.300 7feb5f574700 1 --2- 172.21.2.112:0/165999516 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7feb600e53a0 0x7feb601b0550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:50:59.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:59.300 7feb5ed73700 1 --2- 172.21.2.112:0/165999516 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7feb601b0b70 0x7feb601089d0 secure :-1 s=READY pgs=3372 cs=0 l=1 rev1=1 rx=0x7feb50006700 tx=0x7feb50009b80).ready entity=mon.0 client_cookie=59455af24b97c3c4 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:59.323 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:59.300 7feb5e572700 1 -- 172.21.2.112:0/165999516 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feb5000c0c0 con 0x7feb601b0b70 2021-02-10T11:50:59.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:59.300 7feb5e572700 1 -- 172.21.2.112:0/165999516 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb5000c400 con 0x7feb601b0b70 2021-02-10T11:50:59.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:59.300 7feb5e572700 1 -- 172.21.2.112:0/165999516 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feb5000c8e0 con 0x7feb601b0b70 2021-02-10T11:50:59.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:59.300 7feb65873700 1 -- 172.21.2.112:0/165999516 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7feb601b0b70 msgr2=0x7feb601089d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:59.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:59.300 7feb65873700 1 --2- 172.21.2.112:0/165999516 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7feb601b0b70 0x7feb601089d0 secure :-1 s=READY pgs=3372 cs=0 l=1 rev1=1 rx=0x7feb50006700 tx=0x7feb50009b80).stop 2021-02-10T11:50:59.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:59.301 7feb65873700 1 -- 172.21.2.112:0/165999516 shutdown_connections 2021-02-10T11:50:59.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:59.301 7feb65873700 1 --2- 172.21.2.112:0/165999516 >> v2:172.21.2.120:3300/0 conn(0x7feb60108ff0 0x7feb601093f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:59.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:59.301 7feb65873700 1 --2- 172.21.2.112:0/165999516 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7feb601b0b70 0x7feb601089d0 unknown :-1 s=CLOSED pgs=3372 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:59.324 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:59.301 7feb65873700 1 --2- 172.21.2.112:0/165999516 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7feb600e53a0 0x7feb601b0550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:59.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:59.301 7feb65873700 1 -- 172.21.2.112:0/165999516 >> 172.21.2.112:0/165999516 conn(0x7feb600ce4b0 msgr2=0x7feb600e45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:59.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:59.301 7feb65873700 1 -- 172.21.2.112:0/165999516 shutdown_connections 2021-02-10T11:50:59.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:59.301 7feb65873700 1 -- 172.21.2.112:0/165999516 wait complete. 2021-02-10T11:50:59.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:59.315 7feb65873700 1 Processor -- start 2021-02-10T11:50:59.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:59.315 7feb65873700 1 -- start start 2021-02-10T11:50:59.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:59.316 7feb65873700 1 --2- >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7feb600ab040 0x7feb600ab440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:59.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:59.316 7feb65873700 1 --2- >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7feb600aba60 0x7feb601aa150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:59.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:59.316 7feb65873700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7feb601aa770 0x7feb601aab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:59.325 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:59.316 7feb65873700 1 -- --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb601a7d90 con 0x7feb601aa770 2021-02-10T11:50:59.326 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:59.316 7feb65873700 1 -- --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb600abe60 con 0x7feb600ab040 2021-02-10T11:50:59.326 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:59.316 7feb65873700 1 -- --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb6018f9b0 con 0x7feb600aba60 2021-02-10T11:50:59.326 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:59.316 7feb5fd75700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7feb601aa770 0x7feb601aab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:59.326 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:59.316 7feb5fd75700 1 --2- >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7feb601aa770 0x7feb601aab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.112:3300/0 says I am v2:172.21.2.112:34266/0 (socket says 172.21.2.112:34266) 2021-02-10T11:50:59.326 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:59.316 7feb5fd75700 1 -- 172.21.2.112:0/4184437144 learned_addr learned my addr 172.21.2.112:0/4184437144 (peer_addr_for_me v2:172.21.2.112:0/0) 2021-02-10T11:50:59.326 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:59.316 7feb5f574700 1 --2- 172.21.2.112:0/4184437144 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7feb600ab040 0x7feb600ab440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:59.326 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:59.316 7feb5ed73700 1 --2- 172.21.2.112:0/4184437144 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7feb600aba60 0x7feb601aa150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:59.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:59.317 7feb5fd75700 1 -- 172.21.2.112:0/4184437144 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7feb600aba60 msgr2=0x7feb601aa150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:59.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:59.317 7feb5fd75700 1 --2- 172.21.2.112:0/4184437144 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7feb600aba60 0x7feb601aa150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:59.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:59.317 7feb5fd75700 1 -- 172.21.2.112:0/4184437144 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7feb600ab040 msgr2=0x7feb600ab440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:59.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:59.317 7feb5fd75700 1 --2- 172.21.2.112:0/4184437144 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7feb600ab040 0x7feb600ab440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:59.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:59.317 7feb5fd75700 1 -- 172.21.2.112:0/4184437144 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb60091480 con 0x7feb601aa770 2021-02-10T11:50:59.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:59.317 7feb5fd75700 1 --2- 172.21.2.112:0/4184437144 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7feb601aa770 0x7feb601aab70 secure :-1 s=READY pgs=3373 cs=0 l=1 rev1=1 rx=0x7feb58009980 tx=0x7feb5800bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:59.328 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:59.317 7feb57fff700 1 -- 172.21.2.112:0/4184437144 <== mon.0 v2:172.21.2.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feb5800c2f0 con 0x7feb601aa770 2021-02-10T11:50:59.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:59.317 7feb57fff700 1 -- 172.21.2.112:0/4184437144 <== mon.0 v2:172.21.2.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb58006790 con 0x7feb601aa770 2021-02-10T11:50:59.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:59.317 7feb57fff700 1 -- 172.21.2.112:0/4184437144 <== mon.0 v2:172.21.2.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feb5800c870 con 0x7feb601aa770 2021-02-10T11:50:59.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:59.317 7feb65873700 1 -- 172.21.2.112:0/4184437144 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb60091480 con 0x7feb601aa770 2021-02-10T11:50:59.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:59.317 7feb65873700 1 -- 172.21.2.112:0/4184437144 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb60091480 con 0x7feb601aa770 2021-02-10T11:50:59.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:59.318 7feb57fff700 1 -- 172.21.2.112:0/4184437144 <== mon.0 v2:172.21.2.112:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7feb580011f0 con 0x7feb601aa770 2021-02-10T11:50:59.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:59.318 7feb57fff700 1 --2- 172.21.2.112:0/4184437144 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7feb440186d0 0x7feb4401abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:59.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:59.318 7feb57fff700 1 -- 172.21.2.112:0/4184437144 <== mon.0 v2:172.21.2.112:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 5573+0+0 (secure 0 0 0) 0x7feb58032a50 con 0x7feb601aa770 2021-02-10T11:50:59.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:59.318 7feb5f574700 1 --2- 172.21.2.112:0/4184437144 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7feb440186d0 0x7feb4401abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:59.329 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:59.318 7feb5f574700 1 --2- 172.21.2.112:0/4184437144 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7feb440186d0 0x7feb4401abd0 crc :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:59.354 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:59.349 7feb65873700 1 -- 172.21.2.112:0/4184437144 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7feb601026e0 con 0x7feb601aa770 2021-02-10T11:50:59.354 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:59.349 7feb57fff700 1 -- 172.21.2.112:0/4184437144 <== mon.0 v2:172.21.2.112:3300/0 6 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+968 (secure 0 0 0) 0x7feb5800be10 con 0x7feb601aa770 2021-02-10T11:50:59.355 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:59.350 7feb65873700 1 -- 172.21.2.112:0/4184437144 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb60109ac0 con 0x7feb601aa770 2021-02-10T11:50:59.357 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:59.353 7feb57fff700 1 -- 172.21.2.112:0/4184437144 <== mon.0 v2:172.21.2.112:3300/0 7 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7feb580066e0 con 0x7feb601aa770 2021-02-10T11:50:59.490 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:59.485 7feb65873700 1 -- 172.21.2.112:0/4184437144 --> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] -- mon_command({"injected_args": ["--no-mon-health-to-clog"], "prefix": "injectargs"} v 0) v1 -- 0x7feb601026e0 con 0x7feb601aa770 2021-02-10T11:50:59.491 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:59.486 7feb57fff700 1 -- 172.21.2.112:0/4184437144 <== mon.0 v2:172.21.2.112:3300/0 8 ==== mon_command_ack([{"injected_args": ["--no-mon-health-to-clog"], "prefix": "injectargs"}]=0 injectargs:mon_health_to_clog = 'false' v0) v1 ==== 144+0+0 (secure 0 0 0) 0x7feb5800be10 con 0x7feb601aa770 2021-02-10T11:50:59.493 INFO:teuthology.orchestra.run.gibba012.stderr:mon.a: injectargs:mon_health_to_clog = 'false' 2021-02-10T11:50:59.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:59.489 7feb65873700 1 -- 172.21.2.112:0/4184437144 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7feb601aa770 msgr2=0x7feb601aab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:59.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:59.489 7feb65873700 1 --2- 172.21.2.112:0/4184437144 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7feb601aa770 0x7feb601aab70 secure :-1 s=READY pgs=3373 cs=0 l=1 rev1=1 rx=0x7feb58009980 tx=0x7feb5800bcd0).stop 2021-02-10T11:50:59.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:59.489 7feb65873700 1 --2- 172.21.2.112:0/4184437144 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7feb601901a0 0x7feb60091ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:59.493 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:59.489 7feb65873700 1 -- 172.21.2.112:0/4184437144 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb600abe60 con 0x7feb601901a0 2021-02-10T11:50:59.494 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:59.490 7feb5ed73700 1 --2- 172.21.2.112:0/4184437144 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7feb601901a0 0x7feb60091ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:59.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:59.490 7feb5ed73700 1 -- 172.21.2.112:0/4184437144 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7feb60109ac0 con 0x7feb601901a0 2021-02-10T11:50:59.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:59.490 7feb5ed73700 1 -- 172.21.2.112:0/4184437144 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb5000b630 con 0x7feb601901a0 2021-02-10T11:50:59.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:59.490 7feb5ed73700 1 --2- 172.21.2.112:0/4184437144 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7feb601901a0 0x7feb60091ff0 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 rx=0x7feb5000bd20 tx=0x7feb5000bc30).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:59.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:59.490 7feb57fff700 1 -- 172.21.2.112:0/4184437144 <== mon.1 v2:172.21.2.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feb50009420 con 0x7feb601901a0 2021-02-10T11:50:59.495 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:59.490 7feb57fff700 1 -- 172.21.2.112:0/4184437144 <== mon.1 v2:172.21.2.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb500020d0 con 0x7feb601901a0 2021-02-10T11:50:59.496 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:59.491 7feb57fff700 1 -- 172.21.2.112:0/4184437144 <== mon.1 v2:172.21.2.120:3300/0 3 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7feb50009420 con 0x7feb601901a0 2021-02-10T11:50:59.497 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:59.493 7feb57fff700 1 -- 172.21.2.112:0/4184437144 <== mon.1 v2:172.21.2.120:3300/0 4 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7feb50009420 con 0x7feb601901a0 2021-02-10T11:50:59.634 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:59.629 7feb65873700 1 -- 172.21.2.112:0/4184437144 --> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] -- mon_command({"injected_args": ["--no-mon-health-to-clog"], "prefix": "injectargs"} v 0) v1 -- 0x7feb60109ac0 con 0x7feb601901a0 2021-02-10T11:50:59.635 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:59.631 7feb57fff700 1 -- 172.21.2.112:0/4184437144 <== mon.1 v2:172.21.2.120:3300/0 5 ==== mon_command_ack([{"injected_args": ["--no-mon-health-to-clog"], "prefix": "injectargs"}]=0 injectargs:mon_health_to_clog = 'false' v0) v1 ==== 144+0+0 (secure 0 0 0) 0x7feb50000c90 con 0x7feb601901a0 2021-02-10T11:50:59.636 INFO:teuthology.orchestra.run.gibba012.stderr:mon.b: injectargs:mon_health_to_clog = 'false' 2021-02-10T11:50:59.637 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:59.632 7feb65873700 1 -- 172.21.2.112:0/4184437144 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7feb601901a0 msgr2=0x7feb60091ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:59.637 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:59.632 7feb65873700 1 --2- 172.21.2.112:0/4184437144 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7feb601901a0 0x7feb60091ff0 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 rx=0x7feb5000bd20 tx=0x7feb5000bc30).stop 2021-02-10T11:50:59.637 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:59.632 7feb65873700 1 --2- 172.21.2.112:0/4184437144 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7feb601aa150 0x7feb600ab040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:59.637 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:59.632 7feb65873700 1 -- 172.21.2.112:0/4184437144 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb601aa550 con 0x7feb601aa150 2021-02-10T11:50:59.638 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:59.633 7feb5ed73700 1 --2- 172.21.2.112:0/4184437144 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7feb601aa150 0x7feb600ab040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:59.638 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:59.633 7feb5ed73700 1 -- 172.21.2.112:0/4184437144 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7feb60109ac0 con 0x7feb601aa150 2021-02-10T11:50:59.638 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:59.633 7feb5ed73700 1 -- 172.21.2.112:0/4184437144 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb5001fc10 con 0x7feb601aa150 2021-02-10T11:50:59.638 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:59.633 7feb5ed73700 1 --2- 172.21.2.112:0/4184437144 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7feb601aa150 0x7feb600ab040 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 rx=0x7feb50020300 tx=0x7feb50022af0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:59.638 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:59.634 7feb57fff700 1 -- 172.21.2.112:0/4184437144 <== mon.2 v2:172.21.2.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feb50000c90 con 0x7feb601aa150 2021-02-10T11:50:59.639 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:59.634 7feb57fff700 1 -- 172.21.2.112:0/4184437144 <== mon.2 v2:172.21.2.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb500027d0 con 0x7feb601aa150 2021-02-10T11:50:59.639 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:59.634 7feb57fff700 1 -- 172.21.2.112:0/4184437144 <== mon.2 v2:172.21.2.120:3301/0 3 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7feb5000c040 con 0x7feb601aa150 2021-02-10T11:50:59.640 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:59.636 7feb57fff700 1 -- 172.21.2.112:0/4184437144 <== mon.2 v2:172.21.2.120:3301/0 4 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7feb5002e5f0 con 0x7feb601aa150 2021-02-10T11:50:59.777 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:59.773 7feb65873700 1 -- 172.21.2.112:0/4184437144 --> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] -- mon_command({"injected_args": ["--no-mon-health-to-clog"], "prefix": "injectargs"} v 0) v1 -- 0x7feb601aa550 con 0x7feb601aa150 2021-02-10T11:50:59.779 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:59.775 7feb57fff700 1 -- 172.21.2.112:0/4184437144 <== mon.2 v2:172.21.2.120:3301/0 5 ==== mon_command_ack([{"injected_args": ["--no-mon-health-to-clog"], "prefix": "injectargs"}]=0 injectargs:mon_health_to_clog = 'false' v0) v1 ==== 144+0+0 (secure 0 0 0) 0x7feb50025ab0 con 0x7feb601aa150 2021-02-10T11:50:59.780 INFO:teuthology.orchestra.run.gibba012.stderr:mon.c: injectargs:mon_health_to_clog = 'false' 2021-02-10T11:50:59.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:59.778 7feb65873700 1 -- 172.21.2.112:0/4184437144 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7feb440186d0 msgr2=0x7feb4401abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:59.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:59.778 7feb65873700 1 --2- 172.21.2.112:0/4184437144 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7feb440186d0 0x7feb4401abd0 crc :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:59.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:59.778 7feb65873700 1 -- 172.21.2.112:0/4184437144 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7feb601aa150 msgr2=0x7feb600ab040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:59.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:59.778 7feb65873700 1 --2- 172.21.2.112:0/4184437144 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7feb601aa150 0x7feb600ab040 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 rx=0x7feb50020300 tx=0x7feb50022af0).stop 2021-02-10T11:50:59.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:59.778 7feb65873700 1 -- 172.21.2.112:0/4184437144 shutdown_connections 2021-02-10T11:50:59.783 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:59.778 7feb65873700 1 --2- 172.21.2.112:0/4184437144 >> [v2:172.21.2.120:3300/0,v1:172.21.2.120:6789/0] conn(0x7feb601901a0 0x7feb60091ff0 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:59.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:59.778 7feb65873700 1 --2- 172.21.2.112:0/4184437144 >> [v2:172.21.2.120:6800/34990,v1:172.21.2.120:6801/34990] conn(0x7feb440186d0 0x7feb4401abd0 unknown :-1 s=CLOSED pgs=1517 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:59.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:59.778 7feb65873700 1 --2- 172.21.2.112:0/4184437144 >> [v2:172.21.2.120:3301/0,v1:172.21.2.120:6790/0] conn(0x7feb601aa150 0x7feb600ab040 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:59.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:59.778 7feb65873700 1 --2- 172.21.2.112:0/4184437144 >> [v2:172.21.2.112:3300/0,v1:172.21.2.112:6789/0] conn(0x7feb601aa770 0x7feb601aab70 unknown :-1 s=CLOSED pgs=3373 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:59.784 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:59.778 7feb65873700 1 -- 172.21.2.112:0/4184437144 >> 172.21.2.112:0/4184437144 conn(0x7feb6019d660 msgr2=0x7feb601b1a80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:59.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:59.779 7feb65873700 1 -- 172.21.2.112:0/4184437144 shutdown_connections 2021-02-10T11:50:59.785 INFO:teuthology.orchestra.run.gibba012.stderr:2021-02-10 11:50:59.779 7feb65873700 1 -- 172.21.2.112:0/4184437144 wait complete. 2021-02-10T11:50:59.791 INFO:teuthology.misc:Shutting down mds daemons... 2021-02-10T11:50:59.792 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2021-02-10T11:50:59.792 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:50:59.798 INFO:tasks.ceph.mds.a:Stopped 2021-02-10T11:50:59.798 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2021-02-10T11:50:59.798 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:50:59.805 INFO:tasks.ceph.mds.b:Stopped 2021-02-10T11:50:59.805 INFO:teuthology.misc:Shutting down osd daemons... 2021-02-10T11:50:59.805 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2021-02-10T11:50:59.805 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:50:59.821 DEBUG:teuthology.orchestra.run.gibba012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:50:59.840 DEBUG:teuthology.orchestra.run.gibba020:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:50:59.865 INFO:tasks.ceph.osd.0:Stopped 2021-02-10T11:50:59.865 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2021-02-10T11:50:59.865 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:50:59.929 INFO:tasks.ceph.osd.1:Stopped 2021-02-10T11:50:59.930 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2021-02-10T11:50:59.930 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:50:59.997 INFO:tasks.ceph.osd.2:Stopped 2021-02-10T11:50:59.997 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2021-02-10T11:50:59.997 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:51:00.059 INFO:tasks.ceph.osd.3:Stopped 2021-02-10T11:51:00.059 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2021-02-10T11:51:00.059 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:51:00.083 INFO:tasks.ceph.osd.4:Stopped 2021-02-10T11:51:00.083 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2021-02-10T11:51:00.083 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:51:00.152 INFO:tasks.ceph.osd.5:Stopped 2021-02-10T11:51:00.152 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2021-02-10T11:51:00.152 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:51:00.217 INFO:tasks.ceph.osd.6:Stopped 2021-02-10T11:51:00.217 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2021-02-10T11:51:00.217 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:51:00.280 INFO:tasks.ceph.osd.7:Stopped 2021-02-10T11:51:00.281 INFO:teuthology.misc:Shutting down mgr daemons... 2021-02-10T11:51:00.281 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2021-02-10T11:51:00.281 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:51:00.294 INFO:tasks.ceph.mgr.y:Stopped 2021-02-10T11:51:00.295 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2021-02-10T11:51:00.295 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:51:00.340 INFO:tasks.ceph.mgr.x:Stopped 2021-02-10T11:51:00.340 INFO:teuthology.misc:Shutting down mon daemons... 2021-02-10T11:51:00.340 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2021-02-10T11:51:00.340 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:51:00.350 INFO:tasks.ceph.mon.a:Stopped 2021-02-10T11:51:00.351 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2021-02-10T11:51:00.351 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:51:00.387 INFO:tasks.ceph.mon.b:Stopped 2021-02-10T11:51:00.388 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2021-02-10T11:51:00.388 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:51:00.435 INFO:tasks.ceph.mon.c:Stopped 2021-02-10T11:51:00.435 INFO:tasks.ceph:Checking cluster log for badness... 2021-02-10T11:51:00.436 DEBUG:teuthology.orchestra.run.gibba012:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'OSD full dropping all updates' | egrep -v 'OSD near full' | egrep -v 'pausewr flag' | egrep -v 'failsafe engaged, dropping updates' | egrep -v 'failsafe disengaged, no longer dropping' | egrep -v 'is full \(reached quota' | egrep -v POOL_FULL | egrep -v POOL_BACKFILLFULL | head -n 1 2021-02-10T11:51:00.785 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@gibba012.front.sepia.ceph.com 2021-02-10T11:51:00.786 DEBUG:teuthology.orchestra.run.gibba012:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2021-02-10T11:51:01.313 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@gibba012.front.sepia.ceph.com 2021-02-10T11:51:01.313 DEBUG:teuthology.orchestra.run.gibba012:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2021-02-10T11:51:01.713 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@gibba012.front.sepia.ceph.com 2021-02-10T11:51:01.713 DEBUG:teuthology.orchestra.run.gibba012:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2021-02-10T11:51:02.111 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@gibba012.front.sepia.ceph.com 2021-02-10T11:51:02.111 DEBUG:teuthology.orchestra.run.gibba012:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2021-02-10T11:51:02.522 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@gibba020.front.sepia.ceph.com 2021-02-10T11:51:02.523 DEBUG:teuthology.orchestra.run.gibba020:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2021-02-10T11:51:03.016 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@gibba020.front.sepia.ceph.com 2021-02-10T11:51:03.016 DEBUG:teuthology.orchestra.run.gibba020:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2021-02-10T11:51:03.423 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@gibba020.front.sepia.ceph.com 2021-02-10T11:51:03.424 DEBUG:teuthology.orchestra.run.gibba020:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2021-02-10T11:51:03.832 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@gibba020.front.sepia.ceph.com 2021-02-10T11:51:03.832 DEBUG:teuthology.orchestra.run.gibba020:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2021-02-10T11:51:04.248 INFO:tasks.ceph:Archiving mon data... 2021-02-10T11:51:04.250 DEBUG:teuthology.misc:Transferring archived files from gibba012:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/khiremat-2021-02-10_10:13:45-fs-wip-khiremat-mgr-authid-bp-7-nautilus-distro-basic-gibba/5872720/data/mon.a.tgz 2021-02-10T11:51:04.250 DEBUG:teuthology.orchestra.run.gibba012:> mktemp 2021-02-10T11:51:04.587 INFO:teuthology.orchestra.run.gibba012.stdout:/tmp/tmp.ixrMyxtCfL 2021-02-10T11:51:04.587 DEBUG:teuthology.orchestra.run.gibba012:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.ixrMyxtCfL 2021-02-10T11:51:05.131 DEBUG:teuthology.orchestra.run.gibba012:> sudo chmod 0666 /tmp/tmp.ixrMyxtCfL 2021-02-10T11:51:05.813 DEBUG:teuthology.orchestra.remote:gibba012:/tmp/tmp.ixrMyxtCfL is 951KB 2021-02-10T11:51:06.209 DEBUG:teuthology.orchestra.run.gibba012:> rm -fr /tmp/tmp.ixrMyxtCfL 2021-02-10T11:51:06.543 DEBUG:teuthology.misc:Transferring archived files from gibba020:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/khiremat-2021-02-10_10:13:45-fs-wip-khiremat-mgr-authid-bp-7-nautilus-distro-basic-gibba/5872720/data/mon.b.tgz 2021-02-10T11:51:06.543 DEBUG:teuthology.orchestra.run.gibba020:> mktemp 2021-02-10T11:51:06.878 INFO:teuthology.orchestra.run.gibba020.stdout:/tmp/tmp.RgvKKrloqg 2021-02-10T11:51:06.879 DEBUG:teuthology.orchestra.run.gibba020:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.RgvKKrloqg 2021-02-10T11:51:07.463 DEBUG:teuthology.orchestra.run.gibba020:> sudo chmod 0666 /tmp/tmp.RgvKKrloqg 2021-02-10T11:51:08.141 DEBUG:teuthology.orchestra.remote:gibba020:/tmp/tmp.RgvKKrloqg is 963KB 2021-02-10T11:51:08.498 DEBUG:teuthology.orchestra.run.gibba020:> rm -fr /tmp/tmp.RgvKKrloqg 2021-02-10T11:51:08.831 DEBUG:teuthology.misc:Transferring archived files from gibba020:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/khiremat-2021-02-10_10:13:45-fs-wip-khiremat-mgr-authid-bp-7-nautilus-distro-basic-gibba/5872720/data/mon.c.tgz 2021-02-10T11:51:08.832 DEBUG:teuthology.orchestra.run.gibba020:> mktemp 2021-02-10T11:51:09.166 INFO:teuthology.orchestra.run.gibba020.stdout:/tmp/tmp.U3j5Fprcj2 2021-02-10T11:51:09.166 DEBUG:teuthology.orchestra.run.gibba020:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.U3j5Fprcj2 2021-02-10T11:51:09.712 DEBUG:teuthology.orchestra.run.gibba020:> sudo chmod 0666 /tmp/tmp.U3j5Fprcj2 2021-02-10T11:51:10.390 DEBUG:teuthology.orchestra.remote:gibba020:/tmp/tmp.U3j5Fprcj2 is 963KB 2021-02-10T11:51:10.785 DEBUG:teuthology.orchestra.run.gibba020:> rm -fr /tmp/tmp.U3j5Fprcj2 2021-02-10T11:51:11.119 INFO:tasks.ceph:Cleaning ceph cluster... 2021-02-10T11:51:11.119 DEBUG:teuthology.orchestra.run.gibba012:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2021-02-10T11:51:11.124 DEBUG:teuthology.orchestra.run.gibba020:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2021-02-10T11:51:11.468 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2021-02-10T11:51:11.468 DEBUG:teuthology.orchestra.run.gibba012:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2021-02-10T11:51:11.471 DEBUG:teuthology.orchestra.run.gibba020:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2021-02-10T11:51:11.813 INFO:teuthology.orchestra.run.gibba012.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2021-02-10T11:51:11.815 INFO:teuthology.orchestra.run.gibba020.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2021-02-10T11:51:11.820 INFO:tasks.ceph:Archiving crash dumps... 2021-02-10T11:51:11.822 DEBUG:teuthology.misc:Transferring archived files from gibba012:/var/lib/ceph/crash to /home/teuthworker/archive/khiremat-2021-02-10_10:13:45-fs-wip-khiremat-mgr-authid-bp-7-nautilus-distro-basic-gibba/5872720/remote/gibba012/crash 2021-02-10T11:51:11.823 DEBUG:teuthology.orchestra.run.gibba012:> sudo tar cz -f - -C /var/lib/ceph/crash -- . 2021-02-10T11:51:12.169 DEBUG:teuthology.misc:Transferring archived files from gibba020:/var/lib/ceph/crash to /home/teuthworker/archive/khiremat-2021-02-10_10:13:45-fs-wip-khiremat-mgr-authid-bp-7-nautilus-distro-basic-gibba/5872720/remote/gibba020/crash 2021-02-10T11:51:12.170 DEBUG:teuthology.orchestra.run.gibba020:> sudo tar cz -f - -C /var/lib/ceph/crash -- . 2021-02-10T11:51:12.515 INFO:tasks.ceph:Shutting down logrotate 2021-02-10T11:51:12.516 DEBUG:teuthology.orchestra.run.gibba012:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:51:12.519 DEBUG:teuthology.orchestra.run.gibba020:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:51:12.884 DEBUG:teuthology.orchestra.run.gibba012:> sudo rm /etc/logrotate.d/ceph-test.conf 2021-02-10T11:51:13.227 DEBUG:teuthology.orchestra.run.gibba020:> sudo rm /etc/logrotate.d/ceph-test.conf 2021-02-10T11:51:13.571 INFO:tasks.ceph:Compressing logs... 2021-02-10T11:51:13.572 DEBUG:teuthology.orchestra.run.gibba012:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2021-02-10T11:51:13.575 DEBUG:teuthology.orchestra.run.gibba020:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2021-02-10T11:51:33.343 INFO:tasks.ceph:Archiving logs... 2021-02-10T11:51:33.344 DEBUG:teuthology.misc:Transferring archived files from gibba012:/var/log/ceph to /home/teuthworker/archive/khiremat-2021-02-10_10:13:45-fs-wip-khiremat-mgr-authid-bp-7-nautilus-distro-basic-gibba/5872720/remote/gibba012/log 2021-02-10T11:51:33.345 DEBUG:teuthology.orchestra.run.gibba012:> sudo tar cz -f - -C /var/log/ceph -- . 2021-02-10T11:51:37.490 DEBUG:teuthology.misc:Transferring archived files from gibba020:/var/log/ceph to /home/teuthworker/archive/khiremat-2021-02-10_10:13:45-fs-wip-khiremat-mgr-authid-bp-7-nautilus-distro-basic-gibba/5872720/remote/gibba020/log 2021-02-10T11:51:37.491 DEBUG:teuthology.orchestra.run.gibba020:> sudo tar cz -f - -C /var/log/ceph -- . 2021-02-10T11:51:41.309 DEBUG:teuthology.run_tasks:Unwinding manager install 2021-02-10T11:51:41.320 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits... 2021-02-10T11:51:41.320 DEBUG:teuthology.orchestra.run.gibba012:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits 2021-02-10T11:51:41.324 DEBUG:teuthology.orchestra.run.gibba020:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits 2021-02-10T11:51:41.721 DEBUG:teuthology.run_tasks:Unwinding manager clock 2021-02-10T11:51:41.731 INFO:teuthology.task.clock:Checking final clock skew... 2021-02-10T11:51:41.731 DEBUG:teuthology.orchestra.run.gibba012:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-02-10T11:51:42.077 INFO:teuthology.orchestra.run.gibba012.stdout: remote refid st t when poll reach delay offset jitter 2021-02-10T11:51:42.077 INFO:teuthology.orchestra.run.gibba012.stdout:============================================================================== 2021-02-10T11:51:42.077 INFO:teuthology.orchestra.run.gibba012.stdout:+hv01.front.sepi 216.240.36.24 3 u 28 128 377 0.101 0.744 2.629 2021-02-10T11:51:42.077 INFO:teuthology.orchestra.run.gibba012.stdout:*hv02.front.sepi 163.237.218.18 2 u 40 128 377 0.052 1.121 2.663 2021-02-10T11:51:42.077 INFO:teuthology.orchestra.run.gibba012.stdout:+hv03.front.sepi 74.6.168.72 3 u 20 128 377 0.074 0.953 2.608 2021-02-10T11:51:42.078 INFO:teuthology.orchestra.run.gibba012.stdout:-hv04.front.sepi 144.172.126.201 3 u 23 128 377 0.073 2.005 2.799 2021-02-10T11:51:42.078 DEBUG:teuthology.orchestra.run.gibba020:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-02-10T11:51:42.423 INFO:teuthology.orchestra.run.gibba020.stdout: remote refid st t when poll reach delay offset jitter 2021-02-10T11:51:42.424 INFO:teuthology.orchestra.run.gibba020.stdout:============================================================================== 2021-02-10T11:51:42.424 INFO:teuthology.orchestra.run.gibba020.stdout:+hv01.front.sepi 216.240.36.24 3 u 73 128 377 0.094 -4.222 4.644 2021-02-10T11:51:42.424 INFO:teuthology.orchestra.run.gibba020.stdout:*hv02.front.sepi 163.237.218.18 2 u 88 128 377 0.085 -2.042 3.601 2021-02-10T11:51:42.424 INFO:teuthology.orchestra.run.gibba020.stdout:+hv03.front.sepi 74.6.168.72 3 u 26 128 377 0.061 0.797 3.070 2021-02-10T11:51:42.424 INFO:teuthology.orchestra.run.gibba020.stdout:+hv04.front.sepi 144.172.126.201 3 u 28 128 377 0.105 -2.043 3.896 2021-02-10T11:51:42.425 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2021-02-10T11:51:42.435 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2021-02-10T11:51:42.436 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2021-02-10T11:51:42.468 DEBUG:teuthology.orchestra.run.gibba012:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2021-02-10T11:51:42.837 DEBUG:teuthology.orchestra.run.gibba020:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2021-02-10T11:51:43.210 DEBUG:teuthology.orchestra.run.gibba012:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\)' 2021-02-10T11:51:43.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:51:43.557 DEBUG:teuthology.orchestra.run.gibba020:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\)' 2021-02-10T11:51:43.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:51:43.903 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2021-02-10T11:51:43.916 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2021-02-10T11:51:43.945 INFO:teuthology.task.internal:Duration was 2950.045896 seconds 2021-02-10T11:51:43.946 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2021-02-10T11:51:43.987 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2021-02-10T11:51:43.987 DEBUG:teuthology.orchestra.run.gibba012:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2021-02-10T11:51:43.991 DEBUG:teuthology.orchestra.run.gibba020:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2021-02-10T11:51:44.349 INFO:teuthology.orchestra.run.gibba012.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2021-02-10T11:51:44.349 INFO:teuthology.orchestra.run.gibba020.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2021-02-10T11:51:44.365 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2021-02-10T11:51:44.366 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@gibba012.front.sepia.ceph.com 2021-02-10T11:51:44.366 DEBUG:teuthology.orchestra.run.gibba012:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/*.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2021-02-10T11:51:44.711 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@gibba020.front.sepia.ceph.com 2021-02-10T11:51:44.711 DEBUG:teuthology.orchestra.run.gibba020:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/*.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2021-02-10T11:51:45.052 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2021-02-10T11:51:45.052 DEBUG:teuthology.orchestra.run.gibba012:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2021-02-10T11:51:45.056 DEBUG:teuthology.orchestra.run.gibba020:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2021-02-10T11:51:45.417 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2021-02-10T11:51:45.430 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2021-02-10T11:51:45.430 DEBUG:teuthology.orchestra.run.gibba012:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2021-02-10T11:51:45.777 DEBUG:teuthology.orchestra.run.gibba020:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2021-02-10T11:51:46.121 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2021-02-10T11:51:46.176 DEBUG:teuthology.orchestra.run.gibba012:> sudo sysctl -w kernel.core_pattern=core && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2021-02-10T11:51:46.180 DEBUG:teuthology.orchestra.run.gibba020:> sudo sysctl -w kernel.core_pattern=core && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2021-02-10T11:51:46.521 INFO:teuthology.orchestra.run.gibba012.stdout:kernel.core_pattern = core 2021-02-10T11:51:46.523 INFO:teuthology.orchestra.run.gibba020.stdout:kernel.core_pattern = core 2021-02-10T11:51:46.525 DEBUG:teuthology.orchestra.run.gibba012:> test -e /home/ubuntu/cephtest/archive/coredump 2021-02-10T11:51:46.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:51:46.859 DEBUG:teuthology.orchestra.run.gibba020:> test -e /home/ubuntu/cephtest/archive/coredump 2021-02-10T11:51:47.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:51:47.194 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2021-02-10T11:51:47.207 INFO:teuthology.task.internal:Transferring archived files... 2021-02-10T11:51:47.208 DEBUG:teuthology.misc:Transferring archived files from gibba012:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/khiremat-2021-02-10_10:13:45-fs-wip-khiremat-mgr-authid-bp-7-nautilus-distro-basic-gibba/5872720/remote/gibba012 2021-02-10T11:51:47.208 DEBUG:teuthology.orchestra.run.gibba012:> sudo tar cz -f - -C /home/ubuntu/cephtest/archive -- . 2021-02-10T11:51:47.567 DEBUG:teuthology.misc:Transferring archived files from gibba020:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/khiremat-2021-02-10_10:13:45-fs-wip-khiremat-mgr-authid-bp-7-nautilus-distro-basic-gibba/5872720/remote/gibba020 2021-02-10T11:51:47.568 DEBUG:teuthology.orchestra.run.gibba020:> sudo tar cz -f - -C /home/ubuntu/cephtest/archive -- . 2021-02-10T11:51:47.920 INFO:teuthology.task.internal:Removing archive directory... 2021-02-10T11:51:47.921 DEBUG:teuthology.orchestra.run.gibba012:> rm -rf -- /home/ubuntu/cephtest/archive 2021-02-10T11:51:47.925 DEBUG:teuthology.orchestra.run.gibba020:> rm -rf -- /home/ubuntu/cephtest/archive 2021-02-10T11:51:48.257 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2021-02-10T11:51:48.270 INFO:teuthology.task.internal:Not uploading archives. 2021-02-10T11:51:48.270 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2021-02-10T11:51:48.307 INFO:teuthology.task.internal:Tidying up after the test... 2021-02-10T11:51:48.307 DEBUG:teuthology.orchestra.run.gibba012:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2021-02-10T11:51:48.312 DEBUG:teuthology.orchestra.run.gibba020:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2021-02-10T11:51:48.643 INFO:teuthology.orchestra.run.gibba012.stdout:393221 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 10 11:51 /home/ubuntu/cephtest 2021-02-10T11:51:48.646 INFO:teuthology.orchestra.run.gibba020.stdout:393217 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 10 11:51 /home/ubuntu/cephtest 2021-02-10T11:51:48.647 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2021-02-10T11:51:48.685 INFO:teuthology.nuke:Checking targets against current locks 2021-02-10T11:51:48.786 CRITICAL:teuthology:Uncaught exception Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/virtualenv/bin/teuthology", line 33, in sys.exit(load_entry_point('teuthology', 'console_scripts', 'teuthology')()) File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/scripts/run.py", line 38, in main teuthology.run.main(args) File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/run.py", line 412, in main report_outcome(config, archive, fake_ctx.summary, fake_ctx) File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/run.py", line 269, in report_outcome nuke(fake_ctx, fake_ctx.lock) File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/nuke/__init__.py", line 266, in nuke for unnuked in p: File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/nuke/__init__.py", line 280, in nuke_one keep_logs = ctx.keep_logs File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/config.py", line 255, in __getattr__ raise AttributeError(name) AttributeError: keep_logs